Activity of a single neuron in the cortex
one of the learned
stimuli
new
stimulus
Hebbian plasticity
“When an axon of cell A is near enough to
excite cell B and repeatedly or persistently
takes part in firing it, some growth process
or metabolic change takes place in one or
both cells such that A's efficiency, as one
of the cells firing B, is increased”
Donald Hebb, 1949
“Neurons that fire together wire together”
A cortical network
A cortical network
A cortical network
A cortical network
A cortical network
A cortical network
Hebbian plasticity
A cortical network
Network can sustain
activity even in the
absence of input
Specificity of sustained activity
Specificity of sustained activity
Specificity of sustained activity
Specificity of sustained activity
Specificity of sustained activity
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
A model of associative memory
‘Biological’ memories
• Associative: recall is based on content
rather than on the address
• A transient cue induces a sustained recall
• Robust to minor failures of the hardware
• Distributed
The mathematical model
I will use a slightly different
model than the one presented
in the last 10 minutes of
Wednesday’s class
The mathematical model
Neurons are binary:
The activity of neuron i, Si = 0,1
S i  t  1 
1  sgn  h  t   
at time t+1
i
2
input to neuron
i at time t
The mathematical model
hi 
J
ij
Sj
j
5
J51
4
1
J21
2
J32
3
The mathematical model
A memory pattern is a vector of desired
neural activities
5
For example:
4
1
p   1, 0, 0,1,1 
2
3
The Hopfield model
J ij  n  1   J ij  n  
trial n +1
1
N
p
n 1
i
 0.5   p
n 1
j
 0.5 
The Hopfield model
J ij  n  1   J ij  n  
1
N
p
n 1
i
 0.5   p
n 1
j
 0.5 
The Hopfield model
J ij  n  1   J ij  n  
1
p
N
n 1
i
J ii  0
 0.5   p
n 1
j
 0.5 
5
J51
4
1
• local learning rule
• incremental, on-line
J21
2
J32
3
“Neurons that fire together wire together”
The Hopfield model
Network connections are symmetrical. It can be
shown that with asynchronous updating, the
dynamics necessarily converge to a fixed point.
Questions:
1) What are the fixed points of the dynamics?
2) What is their relation with the memory pattern?
Hopfield.m
The Hopfield model
Memory patterns:
If Activities of neurons within and between patterns are
independently chosen by tossing an unbiased coin then in the
limit of large number o neurons, N the network can store ~N
memory patterns
The Perceptron
Afferents
What does a neuron do?
spike
no spike
Vthr
V rest
K ( Dt )
0
tmax T
0 t
Afferents
We consider a simplified case: input is synchronous
Null
Vthr
0
tmax
T
Afferents
Alternatively, input is constant
The perceptron
Y 
Y 
1
2
1
1  sgn  h  
h
W
j
1  sgn  W  X  

2
X1
X2
X3
W4
W1
Y
X4
j
X
j
Geometrical interpretation
W  X  W1  X 1  W 2  X 2
X1
X2
W1
W2
Y
W2
W
X
X2
W1
X1
Geometrical interpretation
W  X  W1  X 1  W 2  X 2
 W cos   W   X cos   X   W sin   W   X sin   X
 W X   cos   W  cos   X   sin   W  sin   X
 W X  cos   W   X

W2
W
X
X2
W1
X1


Geometrical interpretation
Y 

1  sgn  W  X  

2
1
X1
X2
W1
sgn  cos     

2
1
W2
Y
W
W2
X2
X

W1
X1
The perceptron
The Perceptron categorizes the space of
inputs into inputs that should evoke a
response and inputs that should not evoke
a response
Constraints on possible categorizations
Y 
1  sgn  W  X  

2
1
1

  1  sgn   W i  X i
2
 i
X2
X1



Constraints on possible categorizations
Y 
1  sgn  W  X  

2
1
1

  1  sgn   W i  X i
2
 i
X2
X1



Constraints on possible categorizations
Y 
1  sgn  W  X  

2
1
1

  1  sgn   W i  X i
2
 i
X2
X1



Constraints on possible categorizations
X2  X
2
1
0
X2
X1