I. LNLN model of MT
(an elaboration of Tony’s description)
II. Bayes motion estimation
linear (L)
simple nonlinearity (LN)
subunits (LNL)
[from EJ’s talk]
linear (L)
simple nonlinearity (LN)
subunits (LNL)
[from EJ’s talk]
linear (L)
simple nonlinearity (LN)
subunits (LNL)
[from EJ’s talk]
LGN model
+ constant
[from Matteo’s talk]
V1 models
Generalized LNP response model
E1
...
Simple cell
w1
√Σ
En
wn
S1
w1
Complex cell
+
...
Stimulus
Sn
E
N(E,S)
S
√Σ
wn
[from Nicole’s talk]
V1 models
Generalized LNP response model
E1
...
Simple cell
w1
√Σ
En
wn
S1
w1
Complex cell
+
...
Stimulus
Sn
E
N(E,S)
S
√Σ
wn
[from Nicole’s talk]
The linear model of simple cells
Firing
rate
Retinal image
The normalization model of simple cells
Firing
rate
Retinal image
Other cortical cells
RC circuit implementation
Firing
rate
Retinal image
Other cortical cells
Carandini, Heeger, and Movshon, 1996
Models so far....
Models so far....
• Basic cell properties (e.g., tuning/invariance)
captured by linear model
Models so far....
• Basic cell properties (e.g., tuning/invariance)
captured by linear model
• Simple nonlinearities:
-
“Point” (a.k.a. memoryless or static) nonlinearities
Modulatory: Gain control / Divisive normalization
Models so far....
• Basic cell properties (e.g., tuning/invariance)
captured by linear model
• Simple nonlinearities:
-
“Point” (a.k.a. memoryless or static) nonlinearities
Modulatory: Gain control / Divisive normalization
• Cascades
Models so far....
• Basic cell properties (e.g., tuning/invariance)
captured by linear model
• Simple nonlinearities:
-
“Point” (a.k.a. memoryless or static) nonlinearities
Modulatory: Gain control / Divisive normalization
• Cascades
• This “toolbox” is surprisingly effective. How
far can we go with it?
Obviously missing:
Recurrent processing
• Within single neurons: Spike generation (e.g.,
refractoriness, spike rate adaptation)
• Lateral connectivity
• Feedback connectivity
Obviously missing:
Recurrent processing
• Within single neurons: Spike generation (e.g.,
refractoriness, spike rate adaptation)
• Lateral connectivity
• Feedback connectivity
Note: despite their existence and potential functional
importance, these are much harder to understand and model,
and are often underconstrained by existing experimental data
(my opinion).
A striking nonlinearity
V1 cell
Movshon etal, 1985
A striking nonlinearity
V1 cell
MT pattern cell
Movshon etal, 1985
Retinal
image
Moving
image
Other
MT
cells
Other
V1
cells
+
-
+
-
+
+
-
Vy
-
Linear
operator
Vx
Normalization
Output
nonlinearity
Linear
operator
Normalization
Output
nonlinearity
[from Tony’s talk]
the “trick”
Find a way to describe pattern motion
selectivity as a linear problem
Motion is space-time orientation
Adelson & Bergen, 1985
t
x
White background,
Gray bar,
translating left
x
Bar translating left,
occluding a striped
background moving right
t
x
Striped background,
looming
x
Transparently overlayed
stripe patterns, moving
in opposit directions
t
x
White background,
Gray bar,
translating left
x
Bar translating left,
occluding a striped
background moving right
t
x
Striped background,
looming
x
Transparently overlayed
stripe patterns, moving
in opposit directions
t
x
White background,
Gray bar,
translating left
x
Bar translating left,
occluding a striped
background moving right
t
x
Striped background,
looming
x
Transparently overlayed
stripe patterns, moving
in opposit directions
t
x
White background,
Gray bar,
translating left
x
Bar translating left,
occluding a striped
background moving right
t
x
Striped background,
looming
x
Transparently overlayed
stripe patterns, moving
in opposit directions
t
x
White background,
Gray bar,
translating left
x
Bar translating left,
occluding a striped
background moving right
t
x
Striped background,
looming
x
Transparently overlayed
stripe patterns, moving
in opposit directions
t
x
t
x
Population code
0
preferred speed
XYT
[video’s c/o Tim Saint]
[video’s c/o Tim Saint]
[video’s c/o Tim Saint]
[video’s c/o Tim Saint]
Watson & Ahumada, 1985
Spectral power of translating power lies on a
plane in the Fourier domain
Watson & Ahumada, 1985
Direction-selective filter
!"#$% &
!'#(%)*+,% -%+$."+/$
0123+%3 *2/+/$
!#
#
"
!
!"
!!
41,5+/%6 +,#$% +/"%/6+"+%6 7+# 6'#(%)"+,% 13+%/"%8 9+/%#3 !9"%36:
Loosely speaking, V1 cells represent balls of
*2/+/$; 91(#9+<%8 +/ ".% 0123+%3 81,#+/:
spectral energy
= '1'29#"+1/ 1> ".%6% ,%(.#/+6,6 '317+8%6 # 8+6"3+52"%8 3%'3%6%/)
!"#$% & '( !"# )%*+,'"-./01%2
Filter is not velocity-selective
!#
!"
!!
3+4%5 (6%,"507 +8 # "5#1(*#"'1$ 9: 6#""%51
*'%(
Simoncelli,
1993+1 # "
Linear velocity selectivity
WT
Wt
WY
WX
Wy
Wx
Construction of MT pattern cell velocity selectivity via combination
ofAdd
V1 complex
afferents,
shown in the Fourier domain.
spectralcell
energy
on plane
Subtract spectral energy off plane
Simoncelli, 1993
V1
!"#$% &
Linear weighting
!'#(%)*+,%
-%+$."+/$
Tuning
0123+%3
*2/+/$
!#
#
"
!
!"
!!
41,5+/%6 +,#$% +/"%/6+"+%6 7+# 6'#(%)"+,% 13+%/"%8 9+/%#3 !9"%36:
*2/+/$; 91(#9+<%8 +/ ".% 0123+%3 81,#+/:
Simoncelli, 1993
MT
!"#$% &&
Linear
weighting
!'( )%*$+"*,$
!#
Tuning
-%./0*"1
'2,*,$
!"
!!
$"
$!
3/45*,%6 /2"72"6 /8 -9 2,*"6 "2,%: 8/; :*88%;%," /;*%,"#"*/,6<
!
'2,*,$= ./0#.*>%:
*, "+% *4#$%?@%./0*"1 :/4#*,<
Simoncelli,
1993
A 7/72.#"*/, /8 "+%6% 4%0+#,*646 7;/@*:%6 # :*6";*52"%:
;%7;%6%,?
Input: image intensities
Linear
Receptive
Field
Half-squaring
Rectification
Input: V1 afferents
1
...
...
...
2
...
1
+
+
Divisive
Normalization
...
Output: V1 neurons tuned for
spatio-temporal orientation
2
...
Output: MT neurons tuned for
local image velocity
Simoncelli and Heeger, 1998
Input: image intensities
Linear
Receptive
Field
Half-squaring
Rectification
Input: V1 afferents
1
...
...
...
2
...
1
+
+
Divisive
Normalization
...
Output: V1 neurons tuned for
spatio-temporal orientation
2
...
Output: MT neurons tuned for
local image velocity
Simoncelli and Heeger, 1998
Component cell
Input: V1 afferents
Linear
Receptive
Field
...
Half-squaring
Rectification
...
2
+
Divisive
Normalization
...
Output: neurons tuned for
speed and orientation
Simoncelli and Heeger, 1998
Movshon et al., 1983
A
80
B
80
Model
C
.70
D
.70
40
40
.35
.35
24
24
.50
.50
V1
E
12
F
12
G
H
.25
.25
MT
Gratings
Plaids
Gratings
Plaids
Simoncelli and Heeger, 1998
Rodman & Albright (1987)
30
A
B
.75
20
.50
10
.25
0
0
-10
-.25
2
8
32
128
30
Response
Model
2
8
32
128
16
32
64
.75
20
.50
C
10
0
0
-10
-.25
8
30
D
.25
16
32
64
E
8
F
.75
20
.50
10
.25
0
0
-10
-.25
1
4
16
64
1
Speed (deg/sec)
4
16
64
Britten et al. (1993)
A
150
Response
Model
100
.5
50
.25
0
0
0
B
.75
25
50
75
100
0
25
Stimulus Coherence (%)
50
75
100
Direction-tuning predictions
stimulus \ cell type
component
pattern
grating
unimodal
bimodal at
low speeds
dots
bimodal at
high speeds
unimodal
!"#$"%&%' !&(( ) *+,-',%. *"'/
!&((
0"1&(
79:;1)/&8
/$&&1
<96:;1)/&8
397<;1)/&8
5943;1)/&8
59=;1)/&8
2345
2365
275
5
75
1,+&8',"%
365
345
2345 2365
275
5
75
365
345
1,+&8',"%
Simoncelli, Bair, Cavanaugh, Movshon, ARVO-96
>,#"1?(,'@ ?' A,.A /$&&1/ B?/ $+&1,8'&1C9
!"##$%& '$(( ) *+&$ ,%"#+&'$((
./0$(
67689:
#
276=89:
473<89:
47;289:
475289:
1234
1254
164
4
64
0+%$>#+/&
254
234
1234
1254
164
4
64
254
234
0+%$>#+/&
Simoncelli, Bair, Cavanaugh, Movshon, ARVO-96
?+@/0"(+#A "# (/B CD$$0C E"C D%$0+>#$0F7
Stochastic spectral stimuli
Component
a)
b)
!t
Plaid
c)
!y
!y
!x
!x
!x
t
t
t
y
y
x
!t
!t
!y
y
Planar
x
x
Schrater, Knill, and Simoncelli, 2000
Stochastic spectral stimuli
Component
a)
b)
!t
Plaid
c)
!y
!y
!x
!x
!x
t
t
t
y
y
x
!t
!t
!y
y
Planar
x
x
Schrater, Knill, and Simoncelli, 2000
Stochastic spectral stimuli
Component
a)
b)
!t
Plaid
c)
!y
!y
!x
!x
!x
t
t
t
y
y
x
!t
!t
!y
y
Planar
x
x
Schrater, Knill, and Simoncelli, 2000
Stochastic spectral stimuli
Component
a)
b)
!t
Plaid
c)
!y
!y
!x
!x
!x
t
t
t
y
y
x
!t
!t
!y
y
Planar
x
x
Schrater, Knill, and Simoncelli, 2000
Stochastic spectral stimuli
Component
a)
b)
!t
Plaid
c)
!y
!y
!x
!x
!x
t
t
t
y
y
x
!t
!t
!y
y
Planar
x
x
Schrater, Knill, and Simoncelli, 2000
Stochastic spectral stimuli
Component
a)
b)
!t
Plaid
c)
!y
!y
!x
!x
!x
t
t
t
y
y
x
!t
!t
!y
y
Planar
x
x
Schrater, Knill, and Simoncelli, 2000
Stochastic spectral stimuli
Component
a)
b)
!t
Plaid
c)
!y
!y
!x
!x
!x
t
t
t
y
y
x
!t
!t
!y
y
Planar
x
x
Schrater, Knill, and Simoncelli, 2000
Detection thresholds
1.5
Component
Plaid
Planar
Ideal Observer
Threshold SNR
Threshold SNR
2
Planar
3 gratings, non-coplanar
Plaid1
Model prediction
Plaid2
3 gratings, coplanar
1
0.5
0
0.6
0.4
0.2
PS
ML
Subject
AS
0
PS
Subject
ML
Schrater, Knill, and Simoncelli, 2000