Hierarchical Bayes and free energy
Karl Friston, University College London
Abstract: How much about our interactions with – and experience of – our world can be deduced from basic
principles? This talk reviews recent attempts to understand the self-organised behaviour of embodied agents, like
ourselves, as satisfying basic imperatives for sustained exchanges with the environment. In brief, one simple driving
force appears to explain many aspects of action and perception. This driving force is the minimisation of free energy,
surprise or prediction error that – in the context of perception – corresponds to Bayes-optimal predictive coding. We
will look at some of the phenomena that emerge from this principle; such as hierarchical message passing in the brain
and the perceptual inference that ensues. I hope to illustrate the ensuing dynamics using illustrations of action, action
observation and the active (sensing) of the visual world. I hope to conclude with an illustration of the same principles
at a microscopic level, using morphogenesis as an example.
Key words: active inference ∙ autopoiesis ∙ cognitive ∙ dynamics ∙ free energy ∙ epistemic value ∙ self-organization
.
Overview
Prediction and life
Markov blankets and ergodic systems
Simulations of a primordial soup
The anatomy of inference
Graphical models and predictive coding
Canonical microcircuits
Action and perception
Knowing your place
Multi-agent prediction
Morphogenesis
“How can the events in space and time which take place within the spatial
boundary of a living organism be accounted for by physics and chemistry?”
(Erwin Schrödinger 1943)
The Markov blanket as a statistical boundary
(parents, children and parents of children)
Internal states
External states
Sensory states
Active states
The Markov blanket in biotic systems
Sensory states
s  f s ( , s, a)  s
  f ( , s, a)  
External states
  f  ( s , a,  )
a  f a (s, a,  )
Active states
Internal states
lemma: any (ergodic random) dynamical system (m) that possesses a Markov
blanket will appear to engage in active inference
x  f ( x)  
p ( x | m)
The Fokker-Planck equation
p( x | m)    (  f ) p
And its solution in terms of curl-free and divergence-free components
p( x | m)  0  f ( x)  (  Q) ln p( x | m)
But what about the Markov blanket?
s ( s ,a ,  )
ln p ( s | m) 
f  ( s )  (  Q)  ln p( s | m)
Perception
f a ( s )  (  Q)a ln p( s | m)
Action
Value
Reinforcement learning, optimal control
and expected utility theory
F   ln p ( s | m) 
Surprise
Information theory and minimum
redundancy
Et [ ln p ( s | m)] 
Entropy
Self-organisation, synergetics and
homoeostasis
p ( s | m) 
Pavlov
Barlow
Haken
Model evidence
Bayesian brain, active inference and
predictive coding
Helmholtz
Overview
Prediction and life
Markov blankets and ergodic systems
Simulations of a primordial soup
The anatomy of inference
Graphical models and predictive coding
Canonical microcircuits – in the brain
Saccadic searches and salience
Knowing your place
Multi-agent prediction
Morphogenesis – in other animals
Position
Simulations of a (prebiotic)
primordial soup
Short-range
forces
Strong repulsion
Weak electrochemical attraction
Finding the (principal) Markov blanket
Markov blanket matrix: encoding the children, parents and parents of children
B  A  AT  AT A
Markov Blanket = [B · [eig(B) > τ]]
Adjacency matrix
Markov Blanket
20
Hidden states
40
A
60
80
Sensory states
Active states
Internal states
100
120
20
40
60
Element
80
100
120
Does action maintain the structural and functional integrity of the Markov blanket (autopoiesis) ?
Do internal states appear to infer the hidden causes of sensory states (active inference) ?
Active lesion
Markov blanket
8
8
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-8
-6
-4
-2
0
2
4
6
8
-8
-8
-6
-4
-2
Position
0
2
4
6
8
Position
Autopoiesis, oscillator death and simulated brain lesions
Sensory lesion
Internal lesion
8
8
6
6
4
4
2
2
0
0
-2
-2
-4
-4
-6
-6
-8
-8
-6
-4
-2
0
Position
2
4
6
8
-8
-8
-6
-4
-2
0
Position
2
4
6
8
Decoding through the Markov blanket and simulated brain activation
True and predicted motion
8
Motion of external state
0
Predictability
6
Christiaan Huygens
-0.1
-0.2
-0.3
-0.4
100
2
200
0
-2
-6
-8
300
Time
400
500
Internal states
-4
5
-5
0
Position
5
10
Modes
Position
4
15
20
25
30
100
200
300
Time
400
500
Interim summary
The existence of a Markov blanket necessarily implies a partition of states into
internal states, their Markov blanket (sensory and active states) and external or
hidden states.
Because active states change – but are not changed by – external states they
minimize the entropy of internal states and their Markov blanket. This means action
will appear to maintain the structural and functional integrity of the Markov blanket
(autopoiesis).
Internal states appear to infer the hidden causes of sensory states (by maximizing
Bayesian evidence) and influence those causes though action (active inference)
Overview
Prediction and life
Markov blankets and ergodic systems
Simulations of a primordial soup
The anatomy of inference
Graphical models and predictive coding
Canonical microcircuits
Action and perception
Knowing your place
Multi-agent prediction
Morphogenesis
“Objects are always imagined as being present in the field of
vision as would have to be there in order to produce the same
impression on the nervous mechanism” - von Helmholtz
Hermann von Helmholtz
Richard Gregory
Geoffrey Hinton
The Helmholtz machine and the
Bayesian brain
Thomas Bayes
Richard Feynman
“Objects are always imagined as being present in the field of
vision as would have to be there in order to produce the same
impression on the nervous mechanism” - von Helmholtz
Hermann von Helmholtz
Richard Gregory
Impressions on the Markov blanket…
sS
Bayesian filtering and predictive coding
f  ( s )  (Q  )F ( s ,  )
 D      
prediction
update

  s  g ( )
prediction error
Making our own sensations
sensations – predictions
Prediction error
Action
Perception
Changing
sensations
Changing
predictions
Hierarchical generative models
what
A simple hierarchy
(3)
 v(3)

v
Ascending
prediction errors
the 
(2)
v
(2)
 x(2)

x
 x(2)
(2)
 v(2)

v
 v(1)
(1)
 x(1)

x
 x(1)
(1)
 v(1)

v
v(0)
where
Descending
predictions
  D      
Sensory
fluctuations
Overview
Prediction and life
Markov blankets and ergodic systems
Simulations of a primordial soup
The anatomy of inference
Graphical models and predictive coding
Canonical microcircuits – in the brain
Action and perception
Knowing your place
Multi-agent prediction
Morphogenesis – in other animals
Action with point
attractors
 x(1)
 v(2)
 v(1)
visual input
V 
s    
J 
(0, 0)
Descending
proprioceptive predictions
x1
J1
proprioceptive input

(1)
v
a   a s     (1)
 x1 
s    
 x2 
x2
J2
V  (v1 , v2 , v3 )
Action with heteroclinic orbits
 x(1)
 x(2)
Descending
proprioceptive predictions
action
observation
0.4
position (y)
0.6
0.8
1
1.2
1.4
0
a   a s    
(1)
0.2
0.4
0.6
0.8
position (x)
1
1.2
1.4
0
0.2
0.4
0.6
0.8
position (x)
1
1.2
1.4
F ( s, )  Eq [ ln p( s |  , m)  ln p( | u )  ln p(u | m)  ln q( , u)]
Free energy
Likelihood
Empirical priors
Prior beliefs
Entropy
Energy
“I am [ergodic] therefore I think [I will minimise free energy]”
ln p  u | m   Eq ( s |u ) [ D[q(  | s , u ) || q(  | u )]]   (u)
Epistemic value or information gain
 (t )  
s (t )  S
𝜎(𝑢
Extrinsic value
stimulus
visual input
salience
Sampling the world to resolve uncertainty
sampling
 x, p
Parietal (where)
Frontal eye fields
v
xp
x
v
Visual cortex
v ,q
sq
x
Pulvinar salience map
 x ,q
Fusiform (what)
 (u )
xp
a
oculomotor reflex arc
v, p
sp
Superior colliculus
Saccadic eye movements
Saccadic fixation and salience maps
Action (EOG)
2
Hidden (oculomotor) states
0
-2
200
400
600
800
time (ms)
1000
1200
1400
Visual samples
Posterior belief
5
Conditional expectations
about hidden (visual) states
vs.
0
-5
200
And corresponding percept
400
600
800
time (ms)
1000
1200
1400
vs.
Overview
Prediction and life
Markov blankets and ergodic systems
Simulations of a primordial soup
The anatomy of inference
Graphical models and predictive coding
Canonical microcircuits
Action and perception
Knowing your place
Multi-agent prediction
Morphogenesis
Knowing your place: multi-agent games and morphogenesis
Intercellular signaling
li (y x ,y c )= t × å jy cj ×exp(|y xi -y xj |)
é s
ê c
s = ê sx
ê
êë s l
ù é
yc
ú ê
ú=ê
yx
ú ê
úû êë l(y x ,y c )
ù
Intrinsic signals
ú
ú + w Exogenous signals
ú
Endogenous signals
úû
| xi  xj |
Generative model
  c 


g( )    x   ( )
( x , c )


 ( i ) 
exp( i )
 i exp( i )
(Genetic) encoding of target morphology
( x , c )
4
3
Target form
position
2
Position (place code)
 x 
1
4
 9 5 5 1 1 3 7 11
 0 4 4 2 2 0 0 0 


1
0
-1
-2
-3
-4
-4
-2
Signal expression (genetic code)
 

c
1
1

0

0
1 1 1 1 1 1 1
1 1 1 1 0 0 0 
0 0 1 1 0 0 1

0 0 0 0 1 1 0
0
2
4
 ( x , c )
4
Extracellular target signal
3
2
1
0
-1
-2
-3
-4
-4
-2
0
position
2
4
Morphogenesis
Expectations
3
Morphogenesis
Action
2.5
4
2.5
2
2
1.5
1.5
1
1
0.5
0.5
0
0
-0.5
-0.5
-1
-2
-1
-1.5
-3
-1.5
5
10
15
20
25
30
3
location
2
0
-1
-2
5
10
15
time
20
25
30
-4
0
time
5
10
25
30
35
Solution
3
2
2
3
-360
20
4
1
-340
15
time
Softmax expectations
Free energy
1
cell
Free energy
1
-380
4
0
5
-1
6
-400
-2
7
-3
8
-420
5
10
15
20
time
25
30
-4
1
2
3
4
5
cell
6
7
8
-4
-2
0
location
2
4
Dysmorphogenesis
Gradient
TARGET
sx  12  x
Intrinsic
ss  2  s
Gradient
sx1  2  x1
Extrinsic
sc 2  14  c 2
Extrinsic
sc3  14  c3
Regeneration of head
location
morphogenesis
4
4
3
3
2
2
1
1
0
0
-1
-1
-2
-2
-3
-3
-4
-4
0
5
10
15
20
25
30
35
30
35
time
morphogenesis
4
3
2
location
Regeneration
1
0
-1
-2
-3
-4
0
5
10
15
20
time
Regeneration of tail
25
-4
-2
0
location
2
4
Hermann von Helmholtz
“Each movement we make by which we alter the appearance of
objects should be thought of as an experiment designed to test
whether we have understood correctly the invariant relations of
the phenomena before us, that is, their existence in definite
spatial relations.”
‘The Facts of Perception’ (1878) in The Selected Writings of Hermann von
Helmholtz, Ed. R. Karl, Middletown: Wesleyan University Press, 1971 p. 384
Thank you
And thanks to collaborators:
And colleagues:
Rick Adams
Ryszard Auksztulewicz
Andre Bastos
Sven Bestmann
Harriet Brown
Jean Daunizeau
Mark Edwards
Chris Frith
Thomas FitzGerald
Xiaosi Gu
Stefan Kiebel
James Kilner
Christoph Mathys
Jérémie Mattout
Rosalyn Moran
Dimitri Ognibene
Sasha Ondobaka
Will Penny
Giovanni Pezzulo
Lisa Quattrocki Knight
Francesco Rigoli
Klaas Stephan
Philipp Schwartenbeck
Micah Allen
Felix Blankenburg
Andy Clark
Peter Dayan
Ray Dolan
Allan Hobson
Paul Fletcher
Pascal Fries
Geoffrey Hinton
James Hopkins
Jakob Hohwy
Mateus Joffily
Henry Kennedy
Simon McGregor
Read Montague
Tobias Nolte
Anil Seth
Mark Solms
Paul Verschure
And many others
A (Markovian) generative model
, 
P  o, s , u ,  | a , m   P  o | s  P  s | a  P  u |   P   | m 
P  o | s   P (o0 | s0 ) P (o1 | s1 )

P (ot | st )
P  ot | st   A
P  s | a   P ( st | st 1 , at )
P  st 1 | st , ut   B(ut )
P  u   |     (   F )
Likelihood
Control states
P ( s1 | s0 , a1 ) P ( s0 | m)
ut ,
C
, uT
Empirical priors – hidden states
B
– control states
F   EQ ( o , s | ) [ln P (o , s |  )  ln Q( s |  )]
Hidden states
st 1
st
st 1
P  o | m   C
P  s0 | m   D
P   | m   ( ,  )
A
Full priors
ot 1
ot
Prior beliefs about policies
ln P  u |      F : F   F ( , )
Quality of a policy = (negative) expected free energy
 F ( , )  EQ ( o , s | ) [ln P(o , s |  )]  H [Q( s |  )]
 EQ ( o , s | ) [ln Q( s | o ,  )  ln P(o | m)  ln Q( s |  )]
 EQ ( o | ) [ln P(o | m)]  EQ ( o | ) [ D[Q( s | o ,  ) || Q( s |  )]]
Extrinsic value
Extrinsic value
Epistemic value
Epistemic value or information gain
Bayesian surprise and Infomax
KL or risk-sensitive control
Expected utility theory
In the absence of prior beliefs about outcomes:
In the absence of ambiguity:
In the absence of uncertainty or risk:
 EQ ( s | ) [ln P( s |  )  ln Q( s |  )]
 EQ ( o | ) [ln P(o | m)]
 EQ ( o | ) [ D[Q ( s | o ,  ) || Q ( s |  )]]
Bayesian Surprise
Bayesian surprise
 D[Q ( s , o |  ) || Q( s |  )Q(o |  )]
Predictive mutual information
Predicted mutual information
  D[Q( s |  ) || P( s |  )]
KL divergence
Predicted divergence
Extrinsic value
Extrinsic value
Generative models
Discrete states
Continuous states
v(3)
, 
 v(2)

 x(2)
 x(2)
Control states


(2)
v
ut ,
(1)
v
C
, uT
B
 x(1)
 x(1)
 v(1)
 v(0)
Hidden states

(i )
ot 1
ot
st 1
Variational Bayes
(belief updating)
  D     
(i )
st
A
Bayesian filtering
(predictive coding)
(i )
st 1
(i )
(i )
(i )
 ( i ) 0

st
st   (A  ot  Bt 1st 1  Bt  st 1 )
Variational updates
Perception
Action selection
Incentive salience
Functional anatomy
st   ( A  ot  Bt 1st 1  Bt  st 1 )
at
π   ( γ  F)
motor Cortex
β    π  ln π
   arg min F (o1 ,
st
, ot ,  )
occipital Cortex
striatum
prefrontal Cortex
F
ot
π
s
β
Performance
FE
KL
RL
success rate (%)
80
DA
60
40
20
0
0
0.2
0.4
0.6
0.8
Prior preference
Simulated behaviour
1
VTA/SN
hippocampus
Variational updating
Perception
Action selection
Incentive salience
Functional anatomy
st   ( A  ot  Bt 1st 1  Bt  st 1  ln st )
at
π   ( γ  F)
motor Cortex
β    π  ln π  β
  F (o1 ,
st
, ot ,  )
occipital Cortex
striatum
prefrontal Cortex
F
ot
π
s
β
30
25
20
15
10
VTA/SN
5
0.2
0.4
0.6
0.8
1
1.2
1.4
0.08
0.06
0.04
0.02
0
-0.02
0.25
0.5
0.75
1
1.25
1.5
0.2
0.15
0.1
0.05
0
-0.05
10
20
30
40
50
60
70
80
90
Simulated neuronal behaviour
hippocampus
Continuous or discrete state-space models or both?
, 

Control states
ut ,
st 1
st
 v(2)
 x(2)
 x(2)
v(2)
 v(1)
 x(1)
 x(1)
 v(1)
 v(0)
C
, uT
B
Hidden states
v(3)
st 1
A
ot 1
ot