Frictionless Brains:
Evolution and Analysis of
Brain-Body-Environment Systems
Randall D. Beer
Cognitive Science Program
Dept. of Computer Science
Dept. of Informatics
Indiana University
rdbeer@indiana.edu
http://mypage.iu.edu/~rdbeer/
How can we understand the
neural mechanisms underlying
animal behavior?
How can we design artificial
systems with the flexibility and
robustness of animals?
Beer/Keck/2004
The Artificial Insect
Beer (1989)
Research Overview
Biology
Robotics
Theoretical Challenges
Theoretical Challenges
Nervous systems are complex networks of
heterogeneous nonlinear elements
Nervous systems co-evolved with the bodies and
environments in which they are embedded
Environment
Body
Nervous
System
Nervous systems were evolved, not designed
A Perspective on Behavioral Systems
Situatedness
Embodiment
Dynamics
Environment
Body
Nervous
System
Behavior is a property of the entire brain-body-environment system, not of
any individual component
It can only be understood properly in this broader context
Evolutionary Robotics
...
Environment
Dynamical
Analysis
Body
Nervous
System
0.17
0.001
0.21
0.09
0.12
Selection
Copy
Mutation
...
Crossover
...
CTRNN Dynamics
Continuous-Time Recurrent Neural Networks
1


N
#
1 
ẏi =
−yi +
wji σ (yj + θj ) + Ii 
τi
j=1
0.8
σ (x) =
1
1 + e−x
0.6
0.4
0.2
0
-10
-5
0
• Simplest nonlinear continuous-time dynamical neural model
• Biological interpretations
•
•
•
•
➪ Mean firing rate model
➪ Model of nonspiking neuron with input nonlinearities
Computationally and analytically tractable
(Continuous) Hopfield networks, but
➪ Connections need not be symmetric
➪ Self-connections allowed
Universal approximator of smooth dynamics
Model neurons or convenient basis dynamics?
5
10
CTRNN Dynamics
Phase Portraits
3.5
y2 10
7.5
5
3
y2
2.5
2.5
2
0
2
10
7.5
y3
5
y3
1
2.5
0
0
0
0
2.5
5
y1
2
7.5
y1
10
2
0
6
4
8
6
10
10
y2 4
2
8
6
y2
0
-2.6
4
6
-2.55
-2.5
Θ2
2
4
2
-2.45
0
-2.4
y1
0
2
3
4
5
w
6
Parameter Charts
Bifurcation Diagrams
y1
4
6
7
Beer (1995)
CTRNN Parameter Space Structure
BS
FS
BS
BS
FS
FT
FT
FT
8 Parameters
15 Parameters
...
INT
INT1
3 Parameters
FS
24 Parameters
INT2
35 Parameters
What is the structure of the (N 2+2N)-dimensional space CCTRNN (N ) ?
CTRNN Parameter Space Structure
Varying a self-weight
Varying a cross-weight
-1
-1.5
-2
Θ2
-2.5
-3
-3.5
-4.5
-4
-3.5
-3
Θ1
-2.5
-2
0
-2
-4
-6
-8
-8
-6
Θ2
-6
-4
-4
-2
-2
00
Beer (2006)
Θ1
Θ3
CTRNN Design & aVLSI Implementation
2.2 mm
Current [!A]
Neuron 1 output
4
2
0
Current [!A]
Neuron 2 output
4
2
0
Current [!A]
Neuron 3 output
4
2
0
R. Kier and R. Harrison
University of Utah
Current [!A]
Neuron 4 output
4
2
0
2
4
6
8
10
with Kier, Ames and Harrison (2006)
12
Time [s]
14
16
18
20
22
Walking
Legged Locomotion
AS
BS
FS
FT
BS
FS
BS
BS
FS
FT
FT
FT
INT
INT1
15 Free Parameters
FS
24 Free Parameters
with Gallagher (1992)
INT2
35 Free Parameters
Multiple Instantiability
*
BS
FS
+
*
BS
+
*
FS
FT
+
FT
+
*
BS
+
*
FS
-
*
BS
+
FS
+
*
FT
*
FS
-
*
BS
+
FT
+
*
FS
-
BS
+
FT
-
Differ by less than 2% in performance
with Chiel and Gallagher (1999)
*
FS
+
*
BS
FT
+
FT
+
*
FS
*
BS
+
*
FS
+
*
BS
+
*
FT
+
*
FT
BS
+
FS
+
FT
+
Failure of Averaging
!FT
!BS
!FS
10
20
wFS
0
10
wBS
-10
0
wFT
-10
-20
-20
20
Best CPG
FT
BS
FS
0
"FT
10
"BS
wFT#FS
wBS#FT
wBS#FS
wFS#FT
wFS#BS
30
40
Average CPG
"FS
wFT#BS
20
time
FT
BS
FS
0
10
with Chiel and Gallagher (1999)
20
time
30
40
50
Sensitivity and Robustness
*
BS
129-467% coordinated
change in 3 parameters
Nominal
6% change in 1 parameter
*
BS
FS
+
*
BS
+
FS
+
*
FT
-
*
FT
-
*
FT
-
FT
BS
FS
0
10
20
time
FS
+
30
with Chiel and Gallagher (1999)
40
Walking:
Neuromechanical Analysis
Biomechanical Analysis
D* ≈
0.627
V* = *
T
π/6
-π/6
DC
TSwing
0
T*
⎩
⎜
⎜
TStance
0
Foot
Down
⎧
⎨
⎩
TC
Swing
⎧
⎨
⎩
TP
Foot
Up
⎜
⎜
⎜
⎜
⎜
⎜
⎜
Stance
Coast
⎧⎧ ⎧
⎜⎜ ⎨
⎜
⎩
⎜⎜
⎨
⎜⎜ ⎧
⎜⎜ ⎨
⎜⎩ ⎩
Stance
Power
⎨
π/6
D*
-π/6
⎧
⎨
⎩
DP
20
with Chiel and Gallagher (1999)
Explaining Motor Pattern Variability
FT
BS
FS
INT 1
INT 2
0
10
20
30
Time
with Chiel and Gallagher (1999)
40
Walking:
Fitness Space Structure
CPG Fitness Space Structure
15
20
20
10
10
0
0
-10
-10
10
5
0
-5
-10
-15
-15
-10
-5
0
5
-10
15
10
0
10
20
20
20
20
10
10
10
0
0
0
-10
-10
-10
-10
0
10
20
-10
0
10
Beer (2002)
20
-10
0
10
20
-10
0
10
20
RPG Fitness Space Structure
w = 16
10
8
6
τ
4
2
-9
-8
-7
-5
-6
θ
τ = 0.5
-4
-3
τ=5
8
τ = 10
6
4
2
0
θ
θ
-2
-4
-6
w
w
Beer (2002)
θ
-5
-4
-3
-2
-1
w
0
Walking:
The Impact of Circuit
Architecture
The Impact of Circuit Architecture
BS
FS
BS
BS
FS
FT
FT
FT
INT
INT1
64 Architectures
(x 300 exps)
FS
4,096 Architectures
(x 200 exps)
INT2
528,384 Architectures
(~1% sample x 100 exps)
~2.2 million experiments!
with Psujek and Ames (2006)
Fitness Classes
FT
3-Neuron
0.6
Class 1
BS
Best Fitness
0.5
FS
FT
Class 2
0.4
BS
FS
0.3
0.2
FT
0.1
Class 3
0
4-Neuron
BS
5-Neuron
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
0
with Psujek and Ames (2006)
FS
Evolvability
High
Evolvability
Subgroup
Average Fitness
0.4
0.3
Class 1
Class 2
0.2
Class 3
0.1
Low
Evolvability
Subgroup
0
0
0.1
0.2
0.3
0.4
Best Fitness
0.5
0.6
Log Frequency
-1
-2
High Evolvability
-3
Low Evolvability
-4
-5
-6
Frequency x 10-5
0
8
6
4
2
-7
0
0.1
0.2
0.3
0.4
0.5
0.6
Fitness
with Psujek and Ames (2006)
3 Neuron
4 Neuron
5 Neuron
Learning
Associative Learning
4
Environment A
Smell
Action
Before Training
Reinforcement
An E
Circuit Architecture
Action
After Training
Mouth
–
+
Re
N1
+
–
N2
N3
Environment B
Smell
Action
Before Training
Reinforcement
Action
After Training
N4
+
–
N5
Reinforcement Smell
–
+
Evolution
with Phattanasri and Chiel (2002)
0.8
0.6
Circuit Behavior
Environment A
Environment B
Environment A
1
R
0
-1
1
S
0
-1
(N1)
M
N2
N3
1
0
1
0
1
0
0
100
200
300
400
500
time
with Phattanasri and Chiel (2002)
600
700
The Dynamics of a Trial
1
5
9
P!
P!
P!
2
6
10
P0
P0
P0
3
7
11
P+
P"
P+
4
8
12
P0
P0
P0
Circuit Dynamics
y2
-10
10
y2
0
B4
B2
A4
25
y3
B3
0
B1
-25
A3
A1
-50
-40
0
-10
A2
50
10
50
y3
0
-50
-50
-25
-20
0
0
y1
20
y1
40
–
A3
+
25
50
–
!
B3
!
!
!
A4
A1
B4
B1
"
+
+
"
A2
"
Environment A
"
–
–
B2
+
Environment B
Minimally-Cognitive Behavior
Visually-Guided Behavior
Catching Objects
Short-Term Memory
Perception of Body-Scaled Affordances
Selective Attention
An Object Discrimination Task
x
o9
150
0.75
0.5
0.25
0
25
-50
-25
0
150
0.75
0.5
0.25
0
A
25
150
-50
-25
0
150
y
100
100
50
50
1
2
3
4
5
6
175
7
175
150
150
+
+
8
8
9
9
10
10
+
+
11
11
12
12
125
125
100
100
75
75
50
50
13
-60 -40 -20
0
20
40
60
14
-60 -40 -20
Beer (1996)
0
20
40
60
Evolution of
Genetic Regulatory Networks
Model: Genomes and Proteins
Model
Model
Bind
x
y
z
x
Start codon
!
Coding region codons are transcribed/translated
Promotor into
! Environment
amino acids to form a protein
!
!
!
Cell Enhancer
Genomes
Repressor
Proteins
Domain Function
Data Segment
with Drennan (2004, 2006)
The Repressilator
Evolution task: Repressilator
Elowitz and Leibler (2000)
with Drennan (2004, 2006)
Fitness = 34
Evolving a Repressilator
Results:
An Interesting Example
esults: An Interesting
Results:
Example
An Interesting
Example
Fitness = 0
Fitness = 0
Fitness =
esults: An Interesting
Results:
Example
An Interesting
Example
Results:
An Interesting Example
! Crossover forms a 3-cycle
! Crossover destroys several genes
mber of initial population
! Other crossed-over genes
form a fourth
! Deletion
of extra regulatory node
ntained 3 regulatory nodes
regulatory node
Fitness = 12
Fitness = 30
Fitness =
with Drennan (2004, 2006)
sover destroys yet!more
! Crossover
removes last excitatory
Moregenes
genes culled
over
32 generations
connection
to reveal the 3-cycle
moval of self-inhibiting !gene
causes
3-cycle
almost completely
unencumbered
nzero fitness to be achieved
Interplay of
Development Bias and Evolution
Figure 33
Figure
Model Gene
aa
Model gene
gene
Model
Regulatory section
section
Regulatory
Coding section
section
Coding
CCAAA CCCAC
CCCAC AAACA
AAACA CAACA
CAACA AAAAAACAACCACACACAACACACA
AAAAAACAACCACACACAACACACA
CCAAA
Repression11
Repression22
Repression
Repression
Activation11
Activation 22
Activation
Activation
bb
Codingsection
section sequences
sequences
Coding
Threshold
Threshold
Protein types
types
Protein
CACCA
CACCA
AAAAAAC
AAAAAAC
Type
Type
Modifier
Modifier
Protein
Protein
Protein shapes
shapes
Protein
i)i)
CCxxx
CCxxx
xx
xx
Intra-cellularsignal/TF
signal/TF (Int)
(Int)
Intra-cellular
...
ii)
ii)
CAxxx
CAxxx
xx
xx
Extra-cellularsignal/TF
signal/TF (Ext)
(Ext)
Extra-cellular
...
iii)
iii)
ACxxx
ACxxx
xx
xx
Pre-synapticmarker
marker (Pre)
(Pre)
Pre-synaptic
...
iv)
iv)
AAxxx
AAxxx
xx
xx
Post-synaptic marker
marker (Post)
(Post)
Post-synaptic
...
with Psujekto(2006)
Fig. 3.
3. Model
Model gene
gene and
and sequences
sequences corresponding
corresponding
protein types.
types. A.
A. Genes
Genes are
are divided
divided into
into aa regulatory
regulatory
Fig.
to protein
and aa coding
coding section.
section. The
The regulatory
regulatory section
section is
is further
further subdivided
subdivided into
into regions
regions that
that determine
determine ifif aa gene
gene is
is
and
activated or
or repressed.
repressed. The
The coding
coding section
section isis divided
divided into
into two
two regions
regions that
that determine
determine aa protein’s
protein’s function
function
activated
and its binding region. Two bases, ‘A’ and ‘C’, are used to generate genes. B. The four protein types in the
Cell 1
S. Psujek and R. D.1 Beer
Fig. 1. A radially-represented uniform distribution of genotypes leading to a non-uniform distribution of
phenotypes
through a developmental process. Each of these rays represents a unique genotype or
Figure
5
aphenotype. Under phenotypes, the length of each ray indicates the number of genotypes resulting in that
Cell process.
1
phenotype
through the developmental
In the figure, some phenotypes are produced by multiple
1
genotypes; whereas, other phenotypes are not generated at all.
CCCCC
1 CCCCC
Developmental Model
1 P re
CCCCC
CCCCC
2 CCCCC
1 Pos t CCCAC
CCCCA
CCCCC
3 CCCCC
CCCCC
1 CCCCC
1
1 P re
2
CCCCC
2 CCCCC
2
1 Pos t CCCAC
CCCCA
CCCCC
a
3 CCCCC
21
1 Ext
b
CCCCA
Cell 1
3
Cell 2
1
b
1 P re
4 CCCCA
CCCCC
Cell 2
1 Pos t CCCAC
Cell 3
12
CCCCA
4 CCCCA
13 P re
5 CCCCA
14 Pos t CCCCC
5 CCCCA
1) Cell 12
placement
CCCCA
CCCAC
2
4 CCCCA
3
Figure 2
13 P re
CCCCA 15
15
b
6 CCCCA
21
15
13 Ext
CCCCA
CCCAC
1
4 CCCCA
c
13 P re
CCCCA 15
5 CCCCA
1
12
15
2
Cell
1 t
14 Pos
CCCCA 15
6 CCCCA
13 Ext
CCCAC
CCCAC
CCCAC 15
3
7 CCCAC
20 P re
CCCAC 15
5 CCCCA
13 Ext
1
CCCCA
CCCAC
6 CCCCA
13 P re
Cell 1
1
15
12
14 Pos t CCCCC
12
13 Ext
CCCAC
CCCCA
7 CCCAC
12
31
13 Ext
CCCAC
31
20 P re
20 P re
CCCAC
CCCAC
7 CCCAC
20 Pos t CCCCA
8 CCCAC
CCCAC 15
12
6 CCCCA
1
Cell 3
CCCAC 15
CCCCA
5 CCCCACell
14 3
Pos t CCCCC
CCCAC
CCCAC
CCCCA
3) Target identification
CCCAC
12
4 CCCCA
7 CCCAC
13 Ext
CCCCA
13 P re
3
2) Differentiation
12
c
CCCAC
CCCAC
2
CCCCA
CCCCA
6 CCCCA
CCCCA
CCCCC
14 Pos t CCCCC
12
14 Pos t CCCCC
CCCCA
CCCCA
4 CCCCA
CCCCA 15
6 CCCCA
Cell 3
CCCCA 15
CCCCA
CCCCA
12
Cell 2
3
15
1 Ext
12
15
CCCCA
CCCCA
3 CCCCC
5 CCCCA
CCCCA
CCCCC
CCCCA 15
3
2 CCCCC
13 P re
CCCCA 15
CCCCC
1
Cell 3
CCCCA 15
CCCCC
1 CCCCC
CCCCA
3
S. Psujek and R. D. Beer
CCCCC
Figure 5
1 Ext
2
8 CCCAC
CCCAC
31
20 P re
CCCAC
31
20 Pos t CCCCA
3
CCCAC
31
Fig. 5. The differentiation steps as generated by the genome in Figure 4. Only the relevant genes are shown
at each time step. Note that expressed surface proteins are bound in the cell’s membrane and do not degrade
c
over the growth period. However, signaling proteins act for only one time step before being degraded. This
Cell 1
Cell 3
compresses the time for the developmental sequence to play out. Instead of incorporating the time to set up
21
3
a chemical gradient, we chose an exponential decay function that calculates concentration based on
distance. This generates a gradient with a sharp decay that allows gene thresholds to match the distances in
15
of our two-dimensional growth surface. Concentration in this model is therefore a function of distance and
31
CCCAC 15
Fig. 5. The differentiation steps as generated
by the genome in Figure 4. CCCAC
Only the
relevant not
genes
are shown
of both
distance and time.
7 CCCAC 20 P re CCCAC
7 CCCAC 20 P re CCCAC
at each time step. Note that expressed surface proteins are bound in the cell’s membrane and do not degrade
over the growth period. However, signaling
proteins act for only one time
step before being degraded. This
CCCAC 15
CCCAC 31
compresses the time for the developmental
sequence to play out. Instead
of incorporating the time to set up
8 CCCAC 20 Pos t CCCCA
8 CCCAC 20 Pos t CCCCA
1
unit
a chemical gradient, we 31
chose an exponential decay function that calculates concentration based on
18
distance.
a gradient
with a sharp
that allows genein
thresholds
to match the distances
Fig. This
2. generates
The three
stages
of decay
development
the formation
of ainthree-neuron circuit with three synaptic
of our two-dimensional
2
3growth surface. Concentration in this model is therefore a function of distance and
connections.
not
of both distance andIn
time.stage 1 cells are placed at fixed distances in a planar environment. In stage 2, cells
31
2
8 CCCAC
3
20 Pos t CCCCA
8 CCCAC
20 Pos t CCCCA
2
3
with Psujek (2006)
2
3
Developmental Bias
Global Bias
Local Bias
*
with Psujek (2006)
Summary
•
•
•
•
•
Research Program: Dynamics of brain-body-environment systems
Methodology: Evolution and analysis of model agents
Results
➪ Dynamics of neural circuits
➪ Evolution and analysis of sensorimotor control
➪ Evolution and analysis of learning
➪ Evolution and analysis of minimally cognitive behavior
“Frictionless Brains” (and Bodies and Environments)
Theoretical/Computational Biology