Computational Intelligence
Machine motivation and
cognitive neuroscience
approach
Janusz Starzyk
www.ent.ohiou.edu/~starzyk
EE141
Outline









Traditional Artificial Intelligence
Embodied Intelligence (EI)
Embodiment of Mind
EI Interaction with Environment
How to Motivate a Machine
Goal Creation Hierarchy
GCS Experiment
Motivated Learning
Challenges of EI
 We need to know how to organize it
 We need means to implement it
 We need resources to build and
sustain its operation

Promises of EI
 To economy
EE141 To society
http://www.referenceforbusiness.com
Intelligence
http://lifeboat.com

AI’s holy grail From Pattie Maes MIT Media Lab
“…Perhaps the last frontier of science – its
ultimate challenge- is to understand the biological
basis of consciousness and the mental process by
which we perceive, act, learn and remember..”
from Principles of Neural Science by E. R. Kandel et al.
 E. R. Kandel won Nobel Price in 2000 for his work on physiological
basis of memory storage in neurons.
 “…The question of intelligence is the last great
terrestrial frontier of science...” from Jeff Hawkins On
Intelligence.
 Jeff Hawkins founded the Redwood Neuroscience Institute devoted
to brain research
EE141
Is what is intelligence?
http://www.redorbit.com/news/science/
EE141
Various Definitions of Intelligence









The American Heritage Dictionary:
 The capacity to acquire and apply knowledge.
 The faculty of thought and reason.
Webster Dictionary:
 The act or state of knowing; the exercise of the understanding.
 The capacity to know or understand; readiness of comprehension;
Wikipedia – The Free Encyclopedia:
 The capacity to reason, plan, solve problems, think abstractly, comprehend ideas
and language, and learn.
Kaplan & Sadock:
 The ability to learn new things, recall information, think rationally, apply knowledge
and solve problems.
On line dictionary dict.die.net
 The ability to comprehend; to understand and profit from experience
The classical behavioral/biologists:
 The ability to adapt to new conditions and to successfully cope with life situations.
Dr. C. George Boeree, professor in the Psychology Department at Shippensburg University:
 A person's capacity to (1) acquire knowledge (i.e. learn and understand), (2) apply
knowledge (solve problems), and (3) engage in abstract reasoning.
Stanford University Professor of Computer Science Dr. John McCarthy, a pioneer in AI:
 The computational part of the ability to achieve goals in the world.
Scientists in Psychology:
 Ability to remember and use what one has learned, in order to solve problems, adapt
to new situations, and understand and manipulate one’s reality.
EE141
Intelligence
Mainstream Science on Intelligence
December 13, 1994:
An Editorial With 52 Signatories, by
Linda S. Gottfredson, University of
Delaware
Intelligence is a very general
mental capability that, among
other things, involves the ability
to reason, plan, solve problems,
think abstractly, comprehend
complex ideas, learn quickly and
learn from experience.
From http://www.indiana.edu/~intell/map.shtml
EE141
Animals’ Intelligence
 Defining intelligence
through humans is not
appropriate to design
intelligent machines:
– Animals are intelligent too
 Dog IQ test:
 Dogs can learn 165 words (similar to 2 year olds)
 Average dog has the mental abilities of a 2-year-old child (or better)
 They would beat a 3- or 4-year-old in basic arithmetic,
 Dogs show some basic emotions, such as happiness, anger and disgust
 “The social life of dogs is very complex - more like human teenagers interested in who is moving up in the pack, who is sleeping with who etc,“
says professor Stanleay Coren from University of British Columbia
 Border collies, poodles, and german shepards are the smartest dogs
EE141
Traditional AI

Abstract intelligence
Embodied Intelligence

 attempt to simulate
“highest” human faculties:
– language, discursive
reason, mathematics,
abstract problem solving

Environment model
 Condition for problem
solving in abstract way
 “brain in a vat”
EE141
Embodiment
 knowledge is implicit in the
fact that we have a body
– embodiment supports brain
development

Intelligence develops
through interaction with
environment
 Situated in environment
 Environment is its best model
Design principles of intelligent systems
from Rolf Pfeifer “Understanding of Intelligence”
Design principles
synthetic methodology
time perspectives
emergence
diversity/compliance
frame-of-reference
complete agent
principle
From: www.spectrum.ieee.org/.../biorobot11f-thumb.jpg
EE141
Design principles of intelligent systems
from Rolf Pfeifer “Understanding of Intelligence”, 1999








EE141
Interaction with
complex environment
ecological balance
redundancy principle
parallel, loosely
coupled processes
asynchronous
sensory-motor
coordination
value principle
cheap design
Agent
Drawing by Ciarán O’Leary- Dublin Institute of Technology
The principle of “cheap design”

intelligent agents: “cheap”
 exploitation of ecological
niche
 economical (but redundant)
 exploitation of specific
physical properties of
interaction with real world
EE141
Principle of “ecological balance”

balance / task distribution
between
 morphology
 neuronal processing (nervous
system)
 materials
 environment

balance in complexity
 given task environment
 match in complexity of sensory,
motor, and neural system
EE141
The redundancy principle



redundancy prerequisite for
adaptive behavior
partial overlap of
functionality in different
subsystems
sensory systems: different
physical processes with
“information overlap”
http://www.enspire.com/wp-content
EE141
Generation of sensory stimulation
through interaction with environment
multiple modalities
 constraints from
morphology and
materials
 generation of
correlations through
physical process
 basis for crossmodal associations

EE141
The principle of sensory-motor
coordination
Holk Cruse



self-structuring of
sensory data through
interaction with
environment
physical process —
not „computational“
prerequisite for
learning
EE141
•no central control
•only local
neuronal
communication
•global
communication
through
environment
neuronal
connections
The principle of parallel, loosely
coupled processes
Intelligent behavior emergent
from agent-environment
interaction
 Large number of parallel,
loosely coupled processes
 Asynchronous
 Coordinated through agent’s
–sensory-motor system
–neural system
–interaction with environment

http://edwardrippen.com
EE141
So what is an Embodied
Intelligence ?
EE141
Embodied Intelligence
 Definition
 Embodied Intelligence (EI) is a mechanism that learns
how to survive in a hostile environment
– Mechanism: biological, mechanical or virtual agent
with embodied sensors and actuators
– EI acts on environment and perceives its actions
– Environment hostility is persistent and stimulates EI to act
– Hostility: direct aggression, pain, scarce resources, etc
– EI learns so it must have associative self-organizing memory
– Knowledge is acquired by EI
EE141
Embodied Intelligence

EI mimics biological intelligent systems,
extracting general principles of intelligent
behavior and applying them to intelligent agents.

Knowledge is not entered into such systems, but
rather is a result of their successful interaction
with the environment.

Embodied intelligent systems adapt to
unpredictable and dynamic situations in the
environment by learning, which gives them a
high degree of autonomy.

Learning in such systems is incremental, with
continuous prediction of the input associations
based on the emerging models - only new
information is registered in the memory.
EE141
What is Embodiment of a Mind?
http://elainewintman.com
EE141
manwithoutqualities.com
Embodiment of a Mind





Embodiment of a mind is a
part of environment under
control of the mind
It contains intelligence core
and sensory motor interfaces
to interact with environment
It is necessary for
development of intelligence
It is not necessarily constant
or in the form of a physical
body
Boundary of embodiment
transforms modifying brain’s
self-determination
EE141
Embodiment
Sensors
channel
Environment
Intelligence
core
Actuators
channel
Embodiment of Mind




Brain learns own body’s dynamic
Self-awareness is a result of
identification with own embodiment
Embodiment can be extended by
using tools and machines
Successful operation is a function
of correct perception of
environment and own embodiment
EE141
Requirements for Embodied Intelligence

State oriented
 Learns spatio-temporal patterns
 Situated in time and space

Learning
 Perpetual learning
 Screening for novelty

Value driven
 Pain detection
 Pain management
 Goal creation
 Competing goals

Emergence
 artificial evolution
 self-organization
EE141
EI Interaction with Environment
Agent Architecture
Reason
Short-term Memory
Perceive
Act
RETRIEVAL
LEARNING
Long-term Memory
INPUT
OUTPUT
Task
Environment
Simulation or
Real-World System
EE141
From
Randolph M. Jones, P : www.soartech.com
Sensory Inputs Coding
Kandel Fig. 30-1
Kandel Fig. 23-5
Richard Axel, 1995
Visual, auditory, olfactory,
tactile, smell -> motor
How do we process and
represent sensory
information?
EE141
Hip
Trunk
Arm
Hand
Foot
Face
Tongue
Larynx
Challenges of Embodied Intelligence
http://cdn.grin.com
EE141
Challenges of Embodied Intelligence

Development of sensory interfaces
 Active vision
 Speech processing
 Tactile, smell, taste, temperature, pressure
sensing
 Additional sensing
– Infrared, radar, lidar, ultrasound, GPS, etc.
– Can too many senses be less useful?

Development of pain sensors
 Energy, temperature, pressure, acceleration
level
 Teacher input

Development of motor interfaces
 Arms, legs, fingers, eye movement
EE141
Embodiment
Sensors Intelligence
core
Actuators
Environment
Challenges of Embodied Intelligence (cont.)

Finding algorithmic solutions for
 Association, memory, sequence
learning, invariance building,
representation, anticipation, value
learning, goal creation, planning

Development of circuits for neural
computing
 Determine organization of artificial
minicolumn
 Self-organized hierarchy of
minicolumns for sensing and motor
control
 Self-organization of goal creation
pathway
EE141
Human Intelligence – Cortex Uniform Structure




V. Mountcastle argues that all
regions of the brain perform the
same computational algorithm
Groups of neurons (minicolumns)
connected in a pseudorandom way
Same structure
Minicolumns organized in
macrocolumns
VB Mountcastle (2003). Introduction [to a special issue of
Cerebral Cortex on columns]. Cerebral Cortex, 13, 2-4.
EE141
How to Motivate a Machine ?
A fundamental question is what
motivates an agent to do anything,
and in particular, to enhance its
own complexity?
What drives an agent to explore
the environment and learn ways to
effectively interact with it?
EE141
How to Motivate a Machine ?

Pfeifer claims that an agent’s motivation should emerge
from the developmental process.
 He called this the “motivated complexity” principle.
 Chicken and egg problem? An agent must have a motivation to
develop while motivation comes from development?

Steels suggested equipping an agent with self-motivation.
 “Flow” experienced when people perform their expert activity well
would motivate to accomplish even more complex tasks.
 Humans get internal reward for activities that are slightly above their
level of development (Csikszentmihalyi).
 But what is the mechanism of “flow”?

Oudeyer proposed an intrinsic motivation system.
 Motivation comes from a desire to minimize the prediction error.
 Similar to “artificial curiosity” presented by Schmidhuber.
EE141
How to Motivate a Machine ?
Can a machine that only implements
externally given goals be intelligent?
If not how these goals can be created?
•There is a need for a hierarchy of values.
•Not all values can be predetermined by
the designer.
•There is a need for motivation to act,
explore and learn.
•As machine makes new observations
about the environment, there is a need to
relate them to goals and values and create
new goals and values.
EE141
How to Motivate a Machine ?

Although artificial curiosity helps to
explore the environment, it leads to
learning without a specific purpose.
 It may be compared to exploration in
reinforcement learning.
 internal reward motivates the machine to
perform exploration.

Exploration is needed in order to learn and to model the
environment.
 But is this mechanism the only motivation we need to develop
intelligence?
 Can “flow” ideas explain goal oriented learning?
 Can we find a more efficient mechanism for learning?

I suggest a simpler mechanism to motivate a machine.
EE141
How to Motivate a Machine ?

I suggest that it is the hostility of the environment, in the
definition of EI that is the most effective motivational factor.
 It is the pain we receive that moves us.
 It is our intelligence determined to reduce this pain that motivates us
to act, learn, and develop.

Both are needed - hostility of the environment and
intelligence that learns how to reduce the pain.
 Thus pain is good.
 Without pain there would be no intelligence.
 Without pain we would not be motivated to develop.
Fig. englishteachermexico.wordpress.com/
EE141
Motivated Learning

I suggest a goal-driven mechanism to motivate
a machine to act, learn, and develop.




A simple pain based goal creation system is
explained next.
It uses externally defined pain signals that are
associated with primitive pains.
Machine is rewarded for minimizing the primitive
pain signals.
Definition: Motivated learning (ML) is learning based on the
self-organizing system of goal creation in embodied agent.



Machine creates higher level (abstract) goals based on the primitive
pain signals.
It receives internal rewards for satisfying its goals (both primitive and
abstract).
ML applies to EI working in a hostile environment.
EE141
EI Interaction with Environment
EI Architecture
Pain
Perceive
Goal
Creation
Competing
goals
Act
Planning
INPUT
OUTPUT
Task
Environment
Simulation or
Real-World System
EE141
EI machine
interacts with environment using its three pathways
Pain-center and Goal Creation for ML





Simple Mechanism
Creates hierarchy of
Dual pain level
Pain increase
values
(-)
Leads to formulation of
+
(-)
complex goals
(+)
Pain comparators release
(+)
reinforcement
(+)
neurotransmitter:
(-)
• Pain increase Pain level
Pain decrease
inhibitory
• Pain decrease Excitation
excitatory
Forces exploration
EE141
Sensor
Environment
Motor
Abstract Goal Creation for ML
 The goal is to reduce
the primitive pain level
 Abstract goals are
created if they satisfy
the primitive goals
Sensory pathway
(perception, sense)
Motor pathway
(action, reaction)
refrigerator
Open
-
+
food”becomes a
“
sensory input to
abstract pain center
Abstract pain
(Delayed memory of pain)
Food
Eat
-
Association
Inhibition
Reinforcement
Connection
Planning
Expectation
EE141
Level II
Level I
+
Dual pain
Pain
Primitive
Level
Stomach
Abstract Goal Hierarchy
Sensory pathway
(perception, sense)
Motor pathway
(action, reaction)
Job
 Hierarchy of abstract
goals is created if they
satisfy the primitive goals
-
Activation
Stimulation
Inhibition
Reinforcement
Echo
Need
Expectation
EE141
Sugar level
Spend
Level II
Eat
Level I
+
Food
-
Level III
+
Money
-
Work
+
Primitive
Level
How can we make human level
intelligence?
 We
need to know how
 We need means to
implement it
 We need resources to
build and sustain its
operation
EE141
Resources – Evolution of Electronics
EE141
From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006
EE141
By Gordon E. Moore
EE141
Clock Speed (doubles every 2.7 years)
EE141
From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006
Doubling (or Halving) times








EE141
Dynamic RAM Memory “Half Pitch” Feature Size
Dynamic RAM Memory (bits per dollar)
Average Transistor Price
5.4 years
1.5 years
1.6 years
Microprocessor Cost per Transistor Cycle
Total Bits Shipped
Processor Performance in MIPS
Transistors in Intel Microprocessors
Microprocessor Clock Speed
1.1 years
1.1 years
1.8 years
2.0 years
2.7 years
From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006
EE141
From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006
Software or hardware?
Software





Sequential
Error prone
Require programming
Low cost
Well developed
programming methods
EE141
Hardware





Concurrent
Robust
Require design
Significant cost
Hardware prototypes
hard to build
Promises of embodied intelligence

To society
 Advanced use of technology
– Robots
– Tutors
– Intelligent gadgets
 Intelligence age follows
– Industrial age
– Technological age
– Information age
 Society of minds
– Superhuman intelligence
– Progress in science
– Solution to societies’ ills

To industry
 Technological development
 New markets
 Economical growth
EE141
ISAC, a Two-Armed Humanoid Robot
Vanderbilt University
Sounds like science fiction


EE141
If you’re trying to look far
ahead, and what you see
seems like science fiction,
it might be wrong.
But if it doesn’t seem like
science fiction, it’s
definitely wrong.
From presentation by Feresight Institute
Embodied Artificial Intelligence
Based on:
[1] E. R. Kandel et al. Principles of Neural Science,
McGraw-Hill/Appleton & Lange; 4 edition, 2000.
[2] F. Inda, R. Pfeifer, L. Steels, Y. Kuniyoshi, “Embodied Artificial
Intelligence,” International seminar, Germany, July 2003.
[3] R. Chrisley, “Embodied artificial intelligence, ” Artificial
Intelligence, vol. 149, pp.131-150, 2003.
[4] R. Pfeifer and C. Scheier, Understanding Intelligence, MIT
Press, Cambridge, MA, 1999.
[5] R. A. Brooks, “Intelligence without reason,” In Proc. IJCAI-91.
(1991) 569-595 .
[6] R. A. Brooks, Flesh and Machines: How Robots Will Change Us,
(Pantheon, 2002).
[7] R. Kurzweil The Age of Spiritual Machines: When Computers
Exceed Human Intelligence, (Penguin, 2000).
EE141
Questions?
EE141