Latency, duration and codes for objects
in inferior temporal cortex
Gabriel Kreiman, Chou Hung, Tomaso Poggio, James DiCarlo
Center for Biological and Computational Learning
Computation and Systems Biology Initiative
McGovern Institute for Brain Research
Massachusetts Institute of Technology
Recognition can be very fast
Stimulus presentation
Passive viewing
(fixation task)
time
100 ms
100 ms
5 objects presented
per second
•Recordings of spiking activity from macaque monkeys
•Recordings in an area involved in object recognition (inferior temporal cortex)
•10-20 repetitions per stimulus
• Presentation order randomized
• 77 stimuli drawn from 8 pre-defined categories
Recordings made by Chou Hung and James DiCarlo
Reading the neuronal code
1
2
3
4
5
Neuron 1
Neuron 2
Neuron 3
Object
Yes
Yes
Yes
No
No
Yes
No
No
No
1
1
2
Yes
Yes
Yes
3
Mind reading Æ Neuronal reading
Can we read out what the monkey is seeing?
Learning
from (x,y)
pairs
x
y ∈ {1,…,8}
Input to the classifier
0 ms
100 ms
w
MUA: spike counts in each bin
LFP: power in each bin
MUA+LFP: concatenation of MUA and LFP
200 ms
300 ms
Accurate read-out of object category and identity
from a small population
Hung*, Kreiman*, Poggio, DiCarlo. Science 2005
Decoding requires very few spikes
w = 12.5 ms
Hung*, Kreiman*, Poggio, DiCarlo. Science 2005
Local field potentials (the “input”) also show selectivity
MUA: multi-unit spiking activity
SUA: single-unit spiking activity
LFP: local field potentials
MUA & LFP: MUA combined with LFP
Kreiman*, Hung*, Kraskov, Quiroga, Poggio, DiCarlo. Neuron 2006
The classifier extrapolates to new scales and positions
0.7
0.7
n=31
n=31
n=24
n=24
n=9
n=9
n=9
n=9
n=17
n=17
n=10
n=10
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
00
TRAIN
TRAIN
TEST
TEST
chance
chance
chance
Other observations
•
We can decode information from local field potentials. MUA+LFP > MUA > LFP
•
Feature selection significantly improves performance. Choosing the “best” neurons >>
randomly selecting neurons
•
We can decode the time of stimulus onset
•
We can also read out coarse “where” information
•
Decoding is robust to internal and external perturbations
•
The population can extrapolate to novel pictures within known categories
We can decode object information from the model units
Serre, Kouh, Cadieu, Knoblich, Kreiman, Poggio. MIT AI Memo 2005
The model
shows scale
and position
invariance
Latency, duration and codes for objects
in inferior temporal cortex
Gabriel Kreiman, Chou Hung, Tomaso Poggio, James DiCarlo
Center for Biological and Computational Learning
Computation and Systems Biology Initiative
McGovern Institute for Brain Research
Massachusetts Institute of Technology