CHAPTER 6
What Has fMRI ThughtUs
About Object Recognition?
Kalanit Grill-Spector
recogn
researc
t-ocust
that ar
recogr
Int
about
tt'rpic
u: ed I
Fttr e:
oblec
i
[1r\\
6.L lntroduction
Humanscaneffortlesslyrecognizeobjectsin a fractionof a seconddespitelargevariability in the appearanceof objects (Thorpe et al. 1996). What are the underlying
representations
and computationsthat enablethis remarkablehumanability? One wa1
to answerthesequestionsis to investigatethe neuralmechanisms
of objectrecognition
in the humanbrain.With the adventof functionalmagneticresonance
imaging(fMRI I
about 15 yearsago, neuroscientists
and psychologistsbeganto examinethe neural
basesof objectrecognitionin humans.Functionalmagneticresonance
imaging(fMRI I
is an attractivemethodbecauseit is a noninvasivetechniquethat allowsmultiple measurements
of brain activationin the sameawakebehavinghuman.Among noninvasive
techniques,it providesthe best spatialresolutioncurrently available,enablingus to
localizecortical activationsin the spatialresolutionof millimeters(as fine as I mm I
andat a reasonable
time scale(in the orderof seconds).
Before the adventof fMRI, knowledgeaboutthe function of the ventral streamwas
basedon single-unitelectrophysiology
measurements
in monkeysandon lesionstudies.
Thesestudiesshowedthat neuronsin the monkeyinferotemporal(IT) cortexrespondto
shapes(Fujitaetal.1992)andcomplexobjectssuchasfaces(Desimoneet al. 1984),and
thatlesionsto theventralstreamcanproducespecificdeficitsin objectrecognition,such
(inability to recognize
as agnosia(inability to recognizeobjects)and prosopagnosia
faces,Farah 1995).However,interpretinglesiondatais complicatedbecauselesions
aretypically diffuse(usuallymorethanoneregionis damaged),typically disruptboth
a corticalregionandits connectivity,andarenot replicableacrosspatients.Therefore.
the primary knowledge gained from fMRI researchwas which cortical sites in the
normalhumanbrain are involvedin objectrecognition.The first set of fMRI studies
of objectandfacerecognitionin humansidentifiedthe regionsin the humanbrainthat
respondselectivityto objectsand faces(Malachet al. 1995;Kanwisheret al. 1997
Grill-Spectoret al. 1998b).Then a seriesof studiesdemonstrated
that activationin
object- and face-selective
regionscorrelateswith successat recognizingobject and
providingstrikingevidencefor the involvementof theseregionsin
faces,respectively,
102
rrrtru:
oh1e,
rcco!
.rf ttx
rr her
, lloP
rmpl
lr CO
..,fF
lr
:iur.\
of
rI
I
I rrf
:cl.:
::E
cio
:NU
.-rf
-rJ
3l
-tl
T1
.i
JI
wHAT nls flrnr TAUGHTus ABour oBJEcr REcocNrrron?
tUs
ion?
rddespitelargevariare the underlying
an ability?Oneway
rf objectrecognition
nceimaging(fMRI)
examinethe neural
nceimaging(fMRI)
llowsmultiplemea\mong noninvasive
ble. enablingus to
; (asfine as I mm)
ventralstreamwas
d on lesionstudies.
) cortexrespondto
neetal. 1984),and
t recognition,such
rility to recognize
d because
lesions
cally disruptboth
rtients.Therefore,
nical sitesin the
t of fMRI studies
humanbrainthat
isheret al. 1997;
:hatactivationin
izing objectand
'theseregions
in
103
recognition(Grill-Spectoret al. 2000;Bar et al.200l; Grill-Spectoret al. 2004).Once
researchersfound which regionsin the cortex are involved in object recognition,the
and computations
focus of researchshifted to examiningthe natureof representations
that are implementedin theseregionsto understandhow they enableefficient object
recognitionin humans.
"'
In this chapter I will review fMRI researchthat provided important knowledge
in the humanbrain. I choseto focus on this
aboutthe natureof object representations
topic becauseresults from theseexperimentsprovide important insights that can be
usedby computerscientistswhen they designartificial object recognitionsystems.
For example,one of the fundamentalproblemsin recognitionis how to recognizean
(invariantobjectrecognition).Understanding
objectacrossvariationsin its appearance
how a biological systemhas solvedthis problem may give cluesfor how to build a
robust artificial recognition system.Funher, fMRI is more adequatefor measuring
than the temporal sequenceof computationsen route to object
object representations
is longerthan the time scale
recognitionbecausethe time scaleof fMRI measurements
of the recognitionprocess(the temporalresolutionof fMRI is in the orderof seconds,
combiningpsywhereasobject recognitiontakesabout 100-250 ms). Nevertheless,
chophysicswith fMRI may give us somecluesaboutwhat kind of visualprocessingis
implementedin distinct cortical regions.For example,finding regionswhoseactivation
is correlatedwith successat sometasks,but not others,may suggestthe involvement
of particularcortical regionsin one computation,but not another.
In discussinghow fMRI hasimpactedour currentunderstandingof objectrepresentations,I will focuson resultspertainingto two aspectsof objectrepresentation:
. How do the underlyingrepresentations
providefor invariantobjectrecognition?
. How is categoryinformationrepresented
in the ventralstream?
I havechosenthesetopicsbecause(i) they are centraltopicsin objectrecognition
for which fMRI has substantiallyadvancedour understanding.(ii) Some findings
relatedto thesetopics stirred considerabledebate(see sect.6.7), and (iii) someof
the fMRI findings in humansare surprisinggiven prior knowledgefrom single-unit
electrophysiologyin monkeys.In organizingthe chapter,I will begin with a brief
introductionof the functionalorganizationof the humanventralstreamanda definition
of object-selectivecortex. Then I will describeresearchthat elucidatedthe properties
of theseregionswith respectto basiccodingprinciples.I will continuewith findings
relatedto invariant object recognition, and end with researchand theoriesregarding
categoryrepresentationand specializationin the humanventral stream.
6.2 The Functional Organization of the Human Ventral Stream
The first set of fMRI studieson object and face recognitionin humanswas devotedto
Electrophysiology
identifyingtheregionsin thebrainthatareobject-andface-selective.
that neuronsin higher-levelregionsrespondto shapes
researchin monkeyssuggested
and objectsmore than simple stimuli such as lines, edges,and patterns(Desimone
et al. 1984;Fujita et al. 1992;Logothetiset al. 1995).Basedon thesefindings,fMRI
studiesmeasuredbrain activationwhen peopleviewedpicturesof objectscomparedto
t04
KALANIT
GRILL-SPECTOR
I
I
;
;
I
I
I
I
(b)
Eto
t
Ea
x
I
lo
E
-..traa|raalrahlJla
6a r o
*rftdrt
f'
+Ut
I
a
I
I
Puonlrtlon Dwrlbo (mr)
Figure6.1. Object-,face-,and place-selective
cortex.(a) Data of one representive
subjeo
shownon her partiallyinflatedright hemisphere:
lateralview (/eft);ventralview (right).Dark
gray indicatessulci. Lightgray indicatesgyri. Black /inesdelineateretinotopicregions.Blue
regionsdelineateobject-selective
regions(objects> scrambledobjects),includingLO and
pFus/OTSventrallyas well as dorsalfoci along the intraparietalsulcus(lPS).Redregions
delineateface-selective
regions(faces> non-facesobjects),includingthe FFA,a regionin
LOS and a region in posteriorSTS.Magentaregionsdelineateoverlap betweenface- and
regions(places> objectst,
objects-selective
regions.Creenregionsdelineateplace-selective
includingthe PPAand a dorsalregionlateralto the lPS.Dark greenregionsdelineateoverlap
betweenplace-and object-selective
regions.All mapsthresholdedat P < 0.001,voxel level.
(b)LO and pFus/OTS
(butnot Vl) responses
(Grillarecorrelatedwith recognitionperformance
recognitionperformanceand fMRl signalson the same
Spectoret al. 2000).To superimpose
plot, all valueswere normalizedrelativeto the maximumresponsefor the 500-msduration
s ti m u l u s .
"hsignal(condition)
ForfMRf signars(brue,red, and orarrge lines):
For recognitionperforffi.
o/ocorrect(condition).
(seecolor plate6.1.)
mance(bfack)"hcorrect(5O0ms)
scrambled objects (have the same local information and statistics, but do not contain
an object) or texture patterns (e.9., checkerboards, which are robust visual stimuli,
but do not elicit a percept of a global form). These studies found a constellation of
regions in the lateral occipital cortex (or the lateral occipital complex, LOC), beginning
around the lateral occipital sulcus, posterior to MT, and extending ventrally into the
occipitotemporal sulcus (OTS) and the fusiform gyrus (Fus), that respond more to
objects than controls. The LOC is located lateral and anterior to early visual areas (VlV4), (Grill-Spector et al. t998a; Grill-Spector et al. 1998b) and is typically divided
into two subregions: LO, a region in the lateral occipital cortex, adjacent and posterior
to MT; and pFus/OTS; a ventral region overlapping the OTS and posterior fusiform
gyrus (Fig. 6.1).
The lateral occipital complex (LOC) responds similarly to many kinds of objects and
object categories (including novel objects) and is thought to be in the intermediate- or
high-level stages of the visual hierarchy. Importantly, LOC activations are correlated
with subjects' object recognition perfonnance. High LOC responses correlate with
I
I
I
I
I
I
I
I
I
WHAT HAS fMRI TAUGHT
ntive subject
'rishtt. Dark
regions. B/ue
ding LO and
Red regions
. a region in
ln face- and
s > objects\.
teateoverlap
, r , o x e ll e v e l .
r t a n c er C r i l i on the same
-ms duration
t i : r c l np e n o r -
not contatn
ral stimulr.
te llat iono i
. b e ginn i n g
ll 1 int o rh e
td more tt-r
areas( \'l ll r divide d
d posterior
,r fusiform
US ABOUT OBJECT RECOGNITION?
105
successfulobject recognition(hits), and low LOC responsescorrelatewith trials in
which objectsare presentbut not recognized(misses)(Fig. 6.1(b)).Object-selective
regionsare also found in the dorsal stream(Grill-Spector2003; Grill-Spectorand
Malach2004),but activationof theseregionsdoesnot correlatewith objectrecognition
performance(Fang and He 2005). Theseregionsmay be involved in computations
relatedto visually guided actionstoward objects(Culhamet al. 2003). However,a
comprehensive
discussionof the role of the dorsal streamin object perceptionis
beyondthe scopeof this chapter.
In addition to the LOC, researchersfound severalventral regionsthat showpreferential responsesto specificobject categories.The searchfor regionswith categorical
preferencewas motivatedby repons that suggestedthat lesionsto the ventral stream
canproducevery specificdeficits,suchasthe inability to recognizefacesor the inability to readwords,while othervisual (andrecognition)facultiesremainpreserved.By
contrastingactivationsto different kinds of objects,researchersfound ventral regions
that show higher responsesto specificobject categories:including lateral fusiform
regionsthat respondmore to animalsthan tools, and medial fusiform regionsthat
respondto tools more than animals(Martin et al. 1996;Chao et al. 1999)),a region
in the left OTS that respondsmore strongly to letters than textures(the visual word
form area,or VWFA) (Cohenet al. 2000); severalfoci that respondmore strongly to
facesthan to other objects(Kanwisheret al. 1997;Haxby et al. 2000; Hoffman and
Haxby 2000; Grill-Spector et al. 2004), including the well known fusiform face area
(FFA) (Kanwisheret al. 1997),regionsthatrespondmorestronglyto housesandplaces
gyrus,the parahipthanfacesand objects(includinga regionin the parahippocampal
pocampalplacearea(PPA)(Epsteinand Kanwisher1998);regionsthat respondmore
stronglyto body partsthan facesand objects,including a regionnearMT called the
extrastriate
body area(EBA) (Downinget al. 2001);anda regionin the fusiformgyrus
(the fusiform body area,or FBA) (Schwarzloseet al. 2005)).Nevertheless,
many of
regionsrespondto more than one object
theseobject-selective
and category-selective
categoryand also respondstronglyto object fragments(Grill-Spectoret al. 1998b;
Lerneret al. 200I; Lerneret al. 2008).This suggeststhat one mustbe cautiouswhen
interpretingthe natureof the selectiveresponses.It is possiblethat the underlying
representationis perhapsof object parts,features,and/orfragmentsand not of whole
objectsor objectcategories.
regionsin the humanbrain initiateda fiercedebate
Findingsof category-selective
abouttheprinciplesof functional organizationin the ventralstream.Are thereregionsin
the cortexthat are specializedfor any objectcategory?How abstractis the information
in theseregions,
represented
in theseregions(e.g.,is categoryinformationrepresented
I
will
addressthese
or low-levelvisual featuresthat are associatedwith categories)?
questionsin detailin section6.7.
in the LOC
6.3 Cue-InvariantResponses
lbjectsand
rediate-or
correlated
e l a t e u'i th
responsesin the humanbrain were suggestive
Although findingsof object-selective
of the involvementof theseregion in processingobjects,there are many differences
betweenobjectsand scrambledobjects(or objectsandtexturepatterns).Objectshave
106
KALANIT
GRILL.SPECTOR
I Ob;c
!r
,
I
Figure
responses
visual
6.2.Selective
to objects
across
multiple
cuesacross
theLOC.Statistical
mapsof selective
response
to objectfrom luminance,
stereoandmotioninformation
in a
representative
subject.
All mapswerethresholded
atP < 0.005,voxellevel,andareshownon
(a)Luminance
the inflated
righthemisphere
of a representative
> scrambled
subject.
objects
(b)Objectsgenerated
objects.
fromrandomdot stereograms
versus
structureless
randomdot
(perceived
streograms
fromdot motionversus
asa cloudof dots).(c)Objectsgenerated
the
same
dotsmoving
randomly.
Visual
meridians
arerepresented
byred(upper),
blue(horizontal),
andgreen(lower)lines.Whitecontourindicates
region,or MT.Adapted
a motion-selective
fromVinberg
andCrill-Spector
2008.(Seecolorplate6.2.)
shapes,surfaces,and contours;they are associatedwith a meaning and semantic
information;and they are generallymore interestingthan texturepatterns.Each of
thesefactorsmay affectthe higherfMRI responseto objectsthancontrols.Differences
in low-levelvisualpropertiesacrossobjectsandcontrolsmay be driving someof these
effectsaswell.
Convergingevidencefrom severalstudiesrevealedan importantaspectof coding
in the LOC: it respondsto object shape,not low-levelvisual features.Severalstudies
showedthatall LOC subregions(LO andpFus/OTS)areactivatedmorestronglywhen
subjectsview objectsindependentof the type of visual informationthat definesthe
object form (Grill-Spectoret al. 1998a;Kastneret al. 2000; Kourtzi and Kanwisher
2000, 2001; Gilaie-Dotanet al. 2002; Vinberg and Grill-Spector2008) (Fig. 6.2).
The lateral occipital complex (LOC) respondsmore strongly to (1) objectsdefined
by luminancethan to luminancetextures;(2) objects generatedfrom random dot
(3) objectsgeneratedfrom
stereograms
than to structureless
randomdot stereograms;
structurefrom motionthanto random(structureless)
motion;and(4) objectsgenerated
from texturesthanto texturepatterns.Theresponse
of theLOC to objectsis alsosimilar
acrossobject formatl (gray-scale,line drawings,and silhouettes),and it respondsto
objectsdelineatedby both real and illusory contours(Mendolaet al. 1999;Stanley
and Rubin 2003).Kourtzi and Kanwisher(Kourtzi and Kanwisher2001)also showed
thattherewasfMRl-adaptation(fMRI-A, indicatinga commonneuralsubstrate),
when
objectshadthe sameshapebut differentcontours,but that therewasno fMRI-A when
the sharedcontourswere identicalbut the perceivedshapewas different,suggesting
that the LOC respondsto global shaperatherthan to local contours(seealso Lerner
etal.2002;Kourtzi etal.2003).Overall,thesestudiesprovidedfundamentalknowledge
!
1
,
i
t'Jf
::€
.'f
a
!
J
t
.t
I Selectiveresponsesto faces and housesacross stimulus format (photographs,line drawings, and silhouettes)
have also been shown for the FFA and PPA, respectively.
WHAT HAS fMRI
(a)object
TAUGHT
Hole
US ABOUT OBJECT RECOGNITION?
107
Random
Edges
ffi
(b) vl
W trw
v4
LO
pFus/OTS
:c i
f; r.sl
t:
(.,r
F:
l' ,'
i'*3:
li
rf
e
Ei
910
E os ,
1
-9
onn
-
.s"- O H E S R
Condition
figure 6.3. Responses
to shape,edges,and surfacesacrossthe ventralstream.(a)Schematic
illustration
from eithermotionor stereo
of experimental
conditions.Stimuliwere generated
rnformation
alone and had no luminanceedgesor surfaces(exceptfor the screenborder,
n'hichwas presentduringthe entireexperiment,
includingblank baselineblocks).Forillustrationpurposes,darkerregionsindicatefront surfaces.Fromleft to right Object on the front
surfacein front of a flat backgroundplane.Hole on the front surfacein front of a flat backqround.Disconnected
by scrambling
edgesin frontof a flat background.Edgesweregenerated
the shapecontours.Surfaces,
flat surfacesat differentdepths.Random
Two semitransparent
stimuliwith no coherentstructure,
or globalshape.Randomstimuli
edges,globalsurfaces,
had the samerelativedisparityor depthrangeas otherconditions.Seeexamplesof stimuli:
(b) Responses
http://www-psych.stanford.edu/-kalanit/jnpstim/.
to objects,holes,edges,and
mean+ SEMacross
hierarchy.
Responses:
acrossthe visualventral-processing
elobalsurfaces
eightsubjects.O, object; H, hole; 5, surfaces;F, edges;R, random;diamonds,significantly
differentthanrandomat P < 0.05.Adaptedfrom Vinbergand Crill-Spector
2008.
by showing that activation of the LOC is driven by shape rather than by the low-level
visual information that generatesform.
In a recent study, we examined whether the LOC response to objects is driven
by their global shape or their surface information and whether LOC subregions are
sensitive to border ownership. One open question in object recognition is whether
the region in the image that belongs to the object is first segmented from the rest of
the image (figure-ground segmentation)and then recognized, or whether knowing the
shapeof an object aids its segmentation(Petersonand Gibson 1994b,|994a;Nakayama
et al. 1995). To addressthese questions,we scannedsubjectswhen they viewed stimuli
that were matched for their low-level information and generated different percepts:
( I ) a percept of an object in front of a background object, (2) a shaped hole (same
shape as the object) in front of a background, (3) two flat surfaces without shapes,
(4) local edges (created by scrambling the object contour) in front of a background,
or (5) random dot stimuli with no structure (Fig. 6.3(a)) (Vinberg and Grill-Spector
2008). We repeatedthe experiment twice, once with random dots that were presented
stereoscopicallyand once with random dots that moved. We found that LOC responses
(both LO and pFus/OTS) were higher for objects and shaped holes than for surfaces,
local edges, or random stimuli (Fig. 6.3(b)). These results were observed for both
108
KALANIT
GRILL.SPECTOR
motion and stereocues.In contrast,LOC responseswere not higher for surfacesthan
for randomstimuli and were not higher for local edgesthan for randomstimuli. Thus.
adding either local edge information or global surfaceinformation doesnot increase
LOC response.Adding a gl66al shape,however,doesproducea significant increase
in LOC response.Theseresultsprovideclearevidencethat cue-invariantresponses
in
the LOC are driven by object shape,ratherthan by global surfaceinformation or local
edgeinformation.
Interestingly, recent evidence shows that the LOC is sensitive to border
ownership/figure-groundsegmentation)(Appelbaumet al. 2006; Vinberg and GrillSpector2008).We found that the LO andpFus/OTSresponses
werehigherfor objects
(shapespresentedin the foreground)than for the sameshapeswhen they had defined
holes in the foreground.The only differencebetweenthe objects and the holes was
the assignmentof the figure region (or border ownershipof the contour defining the
shape).This higherresponseto objectsthanto holeswasa uniquecharacteristic
of LOC
(Fig.
subregionsand did not occur in other visual regions
6.3). This result suggests
that the LOC prefers shapes(and contours)when they define the figure region. One
implication of this result is that the samebrain machinerymay be involved in both
recognizingobjectsand in determiningwhich region in the visual input containsthe
figure region.Thus,oneconsiderationfor computerscientistsis that an effectiveobject
recognitionalgorithmshoulddetermineboth what is the the objectin the sceneaswell
aswhich regionin the scenecorresponds
to the object.
6.4 Neural Bases of Invariant Object Recognition
The literature reviewed so far provides evidencethat the LOC is involved in the
recognitionandprocessingof objectform. Giventhe LOC's role in objectperception.
onemay consider,how it dealswith variability in an object'sappearance.
Many factors
canaffectthe appearance
of objects.Changesin an object'sappearance
canoccurasa
resultof the objectbeing at differentlocationsrelativeto the observer,which will affect
it's retinalprojectionof objectsin termsof sizeandposition.Also, the 2-D projection
of a 3-D object on the retina varies considerablyowing to changesin its rotation
and viewpoint relative to the observer.Other changesin appearanceoccur because
of differential illumination conditions, which affect an object's color, contrast,and
shadowing.Nevertheless,humansare able to recognizeobjectsacrosslarge changes
in their appearance,
which is referredto as invariant object recognition.
A central topic of researchin the study of object recognition is understanding
how invariant recognition is accomplished.One view suggeststhat invariant object
recognitionis accomplishedbecausetheunderlyingneuralrepresentations
areinvariant
to the appearance
of objects.Thus,this view suggeststhat thereneuralresponses
will
remain similar even when the appearanceof an object changesconsiderably.One
meansby which this can be achievedis by extractingfrom the visual input featuresor
fundamentalelements(suchas geons(Biederman1987)that are relatively insensitive
to changesin objects' appearance.
According to one influential model (recognitionby
components,
or RBC; Biederman1987),objectsarerepresented
by a library of geons,
whichareeasyto detectin manyviewingconditions,andby theirspatialrelations.Other
theoriessuggestthat invariancemay be generatedthrougha sequenceof computations
WHAT HAS fMRI
TAUGHT
US ABOUT OBJECT RECOGNITION?
109
aro\s a hierarchically organizedprocessingstreamin which the level of invariance
n'reases from one level of the processingto the next. For example,at the lowestlevel
,r theprocessingstream,neuronscodelocal features;at higherlevelsof theprocessing
rtrcam,neuronsrespondto more complexshapesand are ldsi sensitiveto changesin
andPoggio 1999).
axition and size(Riesenhuber
\euroimaging studiesof invariantobjectrecognitionfound differentialsensitivity
aross the ventralstreamto objecttransformations
suchas size,position,illumination,
aJ r iewpoint. Intermediateregions, such as LO, show higher sensitivity to image
anstormationsthan higher-levelregions,suchas pFus/OTS.Notably,evidencefrom
o.m) studiessuggeststhat at no point in the ventral streamare neuralrepresentations
csctrelyinvariantto objecttransformations.
Theseresultssupportan accountin which
ryrariantrecognitionis supportedby a pooledresponseacrossneuralpopulationsthat
r:
One way in which this canbe accomplishedis
"ensitiveto objecttransformations.
Qt a neuralcodethat containsindependentsensitivityto object information and object
firnrlormation (DiCarlo andCox 2007);for example,neuronsmay be sensitiveto both
ro.!eL-t
categoryand object position. As long as the categoricalpreferenceis retained
rrLrsSobject transformations,invariantobject information can be extracted.
6.5 Object and PositionInformation in the LOC
tfnc r ariation that the object recognitionsystemneedsto deal with is variation in the
ruc andpositionof objects.Sizeandpositioninvariancearethoughtto be accomplished
x Franby an increasein the size of neural receptivefields along the visual hierarchy
: e..a-soneascends
the visualhierarchy,neuronsrespondto stimuli acrossa largerpart
;t the visualfield). At the sametime, a more complexvisual stimulusis necessaryto
ir*rt si,enificant
in neurons(e.g.,shapesinsteadof orientedlines).Findings
responses
-'m electrophysiology
suggestthat evenat the higheststagesof the visualhierarchy,
Eurons retainsomesensitivityto objectlocationand sizealthoughelectrophysiology
rn)rt\ r'arysignificantlyaboutthe degreeof positionsensitivityof IT neurons(Op De
Bsclk and Vogels2000;Rolls 2000;DiCarlo and Maunsell2003).A relatedissueis
rirether positionsensitivityof neuronsin highervisual areasmanifestsas an orderly,
rrgrglnphic representation
have examinedposition
of the visual field. Researchers
(such
rize
as PPA and FFA) using
srJ
sensitivityin the LOC and nearby cortex
nr&\urementsof the mean responseacrossa region of interest;fMRI-A, in which
trr measuredsensitivityto changesin objectsizeor position;andexaminationof the
.fo'rnbutedresponseacrossthe ventral streamto the sameobject or object category
rrr-r\\ sizesandpositions.
Sereralstudiesdocumented
sensitivityto botheccentricityandpolaranglein distinct
",rntral streamregions.Both object-selectiveand category-selective
regions in the
r<ntralstreamrespondto objectspresentedat multiple positionsand sizes.However,
- amplitudeof responseto object variesacrossdifferentretinal positions.The LO
regions(e.g.,FFA, PPA) respondmore
rnt pFus/OTSas well as category-selective
ist-'ngl) to objectspresentedin the contralateralversusipsilateralvisual field (Grill>,lcr'roret al. 1998b;Hemondet al. 2007;McKyton andZohary2007).Someregions
LO andEBA) alsorespondmorestronglyto objectspresentedin the lower visualfield
l+,rres and Grill-Spector2008; Schwarzloseet al. 2008).Responses
also vary with
110
KALANIT
GRILL.SPECTOR
eccentricity:LO, FFA, and the VWFA respondmore stronglyto centrallypresented
stimuli, and the PPA respondsmore stronglyto peripherallypresentedstimuli (Lerr
et al., 2001;Hassonet al. 2002;Hassonet al. 2003;SayresandGrill-Spector2008).
UsingfMRI-A, my colleaguesandl haveshownthatpFus/OTS,but not LO, exhibitr
somedegreeof insensitivityto an object'ssizeandposition(Grill-Spectoret al.1999t.
The fMRI-A methodallowsoneto characterizethe
sensitivityof neuralrepresentation:
to stimulustransformations
at a subvoxelresolution.fMRI-A is basedon findingsfronr
single-unitelectrophysiologythat showthat when objectsrepeat,thereis a stimulusspecificdecreasein the responseof IT cells to the repeatedimage but not to other
object images(Miller et al. l99l; Sawamuraet al. 2006). Similarly, fMRI signals
in higher visual regions show a stimulus-specificreduction(fMRI-A) in response
to repetitionof identicalobject images(Grill-Spectoret al. 1999;Grill-Spectorand
Malach 2001; Grill-Spectoret al. 2006a).We showedthat fMRI-A can be usedto
test the sensitivityof neural responsesto object transformationby adaptingcortex
with a repeatedpresentationof an identicalstimulusand then examiningadaptation
effectswhenthe stimulusis changedalongan objecttransformation(e.g.,changingits
position).If the responseremainsadapted,it indicatesthat neuronsare insensitiveto
the change;however,if responses
returnto the initial level (recoverfrom adaptation).
it indicatessensitivityto the change(Grill-SpectorandMalach2001).
Using fMRI-A we found that repeatedpresentationof the sameface or object
at the sameposition and size producesreducedfMRI activationor fMRI-A. This
is thoughtto reflect stimulus-specificneuraladaptation.Presentingthe sameface or
object in different positionsin the visual field or at different sizes also produces
fMRI-A in pFus/OTSand FEA, indicating insensitivityto object size and position
(Grill-Spectoret aI. 1999;seealso Vuilleumieret al. 2002).This resultis consistent
with electrophysiology
findingsthat showedthat IT neuronsthat respondsimilarly to
stimuli at differentpositionsin the visual field also show adaptationwhen the same
objectis shownin differentpositions(Lueschowet al. 1994).Incontrast,LO recovers
from fMRI-A to imagesof the samefaceor objectwhenpresentedat differentsizesor
positions.This indicatesthat LO is sensitiveto objectpositionand size.
Recently,severalgroupsexaminedthe sensitivityof the distributedresponseacross
the visual streamto object categoryand object position (Sayresand Grill-Spector
2008; Schwarzloseet al. 2008) and also object identity and object position (Eger
et al. 2008).Thesestudiesusedmulti-voxel patternanalysesand classifiermethods
developedin machinelearningto examinewhatinformationis presentin thedistributed
responses
acrossvoxelsin a corticalregion.Thedistributedresponse
cancarrydifferent
informationfrom themeanresponse
of a regionof interestwhenthereis variationacross
voxels'responses.
In order to examinesensitivityto position information,severalstudiesexamined
whetherdistributedresponsepatternsto sameobject category(or object exemplar)is
the same(or different) when the samestimulusis presentedin a different position
in the visual field. In multi-voxelpatternanalysesresearchers
typically split the data
into two independentsetsand examinethe cross-correlation
betweenthe distributed
responses
to the same(or different)stimulusin the same(or different)positionacross
the two datasets.This gives a measureof the sensitivityof distributedresponsesto
object informationand position. If responsesare position-invariant,there will be a
\
- l
WHAT HAS fMRI
TAUGHT
US ABOUT OBJECT RECOGNITION?
Etttttl
9o Left
4o Left
Fovea
4'Right
4o Up
111
4o Down
rllllll
fflltlll
2-1
012
Amplitude
versusmeanresponse
% SignalChange
Qure 5.4. LO distributedresponsepatternsto differentobject categoriesand stimuluspo;i tlns. Data are shown on the lateralaspectof the right hemispherecorticalsurfacefor a
-Dreentativesubject.Eachpanelshowsthe distributedLO fMRl amplitudesaftersubtracting
r-.-nreachvoxel its meanresponse.Redand yellow,Responses
that are higherthan the voxel's
-ean response.Blue and cyan, Responses
that are lower than the voxel'smean response.
r---:. Inflatedrightcorticalhemisphere,
with red squareindicatingthe zoomedregion.Note
:a: patternchangessignificantlyacrosscolumns(positions)
and to a lesserextentacrossrows
.a:eqories).
Adaptedfrom Sayresand Crill-Spector
2008. (Seecolor plate6.4.)
heh correlation between the distributed responses to the same object category (or
<remplar) at different positions. If responsesare sensitive to position, there will be a
Lur correlation between responsesto the same object category (or exemplar) at different
poritions.
Figure 6.4 illustrates distributed LO responsesto three categories: houses, limbs,
uld faces at six visual locations (fovea; 4o up, right, down, or left from the fovea;
9 leti of fovea). Activation patterns for the same object category presented at diffcrent positions vary considerably (compare the response patterns in the same row
LToss the different columns in Fig. 6.4). There is also variation (but to a lesser extent)
\Toss different object categories when presented at the same retinal position (same
.arlumn, different rows, in Fig. 6.4). Surprisingly, position effects in LO were larger
dren category effects - that is, showing objects from the same category, but at a diff<rent position, reduced the correlation between activation patterns (Fig. 6.5, flrst vs.
drrrd bars) more signiflcantly than changing the object category in the same position
"Fig. 6.5, first vs. second bar). Importantly, position and category effects were inJcpendent, as there were no significant interactions between position and category
112
KALANIT
WHAT
GRILL-SPECTOR
A related recen
ysis more broadlt
hierarchical organ
zlose and colleae
(r
c
.9
0.3-
(faces, bodY Part'
(e.g., PFus/OTS
EBA and LOr' T
(u
o
v
0.1 :
Cl
((l:;
o:
=
o:
{}'1 '
:
:
4.2'
r
8$:tU,"ffio 8$[fr
"ffio
DifferentPosition
SamePosition
Figure
LOdistributed
6.5. Meancross-correlations
between
responses
across
twoindependent
halvesof the datafor the sameor different
positionin the
category
at the sameor different
visualfield.Positioneffects:
patterns
weresubstantially
LO response
to the samecategory
(firstand
moresimilarif theywerepresented
positions
at the samepositionversus
different
Themeancorrelation
washigherfor same-category
thirdbars,P < 10-7).Category
effects:
response
patterns
in the same
thanfor different-category
response
patterns
whenpresented
(first
retinotopic
position
twobars,P < 10-o).Error
subjects.
Adapted
barsindicate
SEMacross
fromSayres
andCrill-Spector
2008.
(all F values< 1.02, all P values> 0.31).Thus,changingbothobjectcategoryandpo(Fig. 6.5, fourth
sition producedmaximaldecorrelationbetweendistributedresponses
bar).
We also examinedwhetherposition sensitivityin LO is manifestedas an orderly
topographicmap (similar to retinotopicorganizationin lower visual areas),by measuring retinotopic mapsin LO using standardtraveling wave paradigms(Wandell 1999;
Sayresand Grill-Spector2008).We found a continuousmappingof the visualfield in
LO both in terms of eccentricityand polar angle.This topographicmap containedan
over-representation
of the contralateraland lower visual field (more voxels preferred
thesevisual field positionsthan the ipsilateraland uppervisual fields).Although we
did not consistentlyfind a single visual field map (a singlehemifieldor quarterfield
representation)
in LO, this analysissuggeststhat thereis preservedretinotopicinformation in LO that may underliethe position effectsobservedin analysesof distributed
LO responses.
Overall, our data show that different object categoriesproduce relatively small
changesto both the meanand distributedresponseacrossLO (categoricaleffectsare
larger in the fusiform and parahippocampalgyri). In comparison,a modest4o change
in an object'spositionproducessignalchangesin LO thatareaslargeor largerthanthe
categorymodulation.This 4" displacement
is well within the rangefor which humans
can categonzeand detect objects (Thorpe et al. 2001). This indicates a difference
betweenthe positionsensitivityof recognitionbehaviorandthat of neuralpopulations
in LO. However,it is possiblethat performancein recognitiontasksthat requirefinegrain discriminationbetweenexemplarsis morepositionsensitive,and limited by the
degreeof positionsensitivityin LO.
s$dies suggest:
which rePresent
hierarchY.
6-5
It is imPofiant tt'
tions of object' '
'
dependson thc'
ral rePresentatl
fullY invariant rt
indePendentof
'
the context trf
to be a gradc'd1
degree of st'nt
stimulus 'st'/t't1994; Roll: en
growing litcrat
tributed neural
(Dill and Edclr
maintaining {
maintain infot
man and Intra
object-basedt
and Grill-SPr
structural L'nc
robust u'ar ie
model. oblec
ttt
PoPulation
for fast deci'
to deternrinc
2007)' Finall
ization and n
w h e r ei t i : t '
6't
Another :t-t
change ucr
WHAT HAS fMRI
TAUGHT
US ABOUT OBJECT RECOGNITION?
113
A relatedrecentstudyexaminedpositionsensitivityusingmulti-voxelpatternanalvsis more broadly acrossthe ventral streamand providedadditionalevidencefor a
hierarchicalorganizationacrossthe ventralstream(Schwarzlose
et al. 2008).Schwarzloseand colleaguesfound that distributedresponsesto'ri particularobject category
tfaces,body parts,or scenes)was similar acrosspositionsin ventrotemporalregions
(e.9.,pFus/OTSand FBA) but changedacrosspositionsin occipital regions (e.9.,
EBA and LO). Thus, evidencefrom both fMRI-A and multi-voxel patternanalysis
rtudiessuggestsa hierarchyof representations
in the humanventral streamthrough
r,rhichrepresentations
thevisual
becomelesssensitiveto objectpositionasoneascends
hierarchy.
6.5.L lmplications for Theories of Object Recognition
It is importantto relateimagingresultsto the conceptof position-invariant
representationsof objectsandobjectcategories.What exactlyis implied by the term "invariance"
dependson the scientificcontext.In someinstances,this term is takento reflecta neural representation
that is abstractedso as to be independentof viewing conditions.A
iullv invariantrepresentation,
in this meaningof the term,is expectedto be completely
rndependent
of informationaboutretinal position (Biedermanand Cooper 1991).In
the contextof studiesof visual cortex, however,the term is more often considered
ttr [rs a gradedphenomenon,in which neuralpopulationsare expectedto retain some
.lesreeof sensitivityto visual transformations(like position changes)but in which
.timulus selectivityis preservedacrossthesetransformations(Kobatakeand Tanaka
199-l;Rolls and Milward 2000; DiCarlo and Cox 2007).In supportof this view, a
rrowing literaturesuggeststhat maintaininglocal position informationwithin a distnbutedneuralrepresentation
may actuallyaid invariantrecognitionin severalways
,Dill andEdelman2001;DiCarlo andCox 2007;SayresandGrill-Spector2008).First,
maintainingseparableinformationaboutpositionand categorymay also allow on to
naintain information about the structuralrelationshipsbetweenobject parts (Edelnan and Intrator2000).Indeedsomeexperimentssuggestthat LO may containboth
.*rject-based(McKyton and Zohary 2007) and retinal-basedreferenceframes(Sayres
end Grill-Spector2008). The object-basedreferenceframe may provide a basisfor
.tructur&lencoding.Second,separablepositionandobjectinformationmay providea
nrbustway for generatingpositioninvarianceby using a populationcode.Under this
model,objectsarerepresented
asmanifoldsin a high-dimensionalspacespannedby a
gx-rpulation
of neurons.The separabilityof position and object information may allow
ior fast decisionsbasedon linear computations(e.g., linear discriminantfunctions)
to determinethe object identity (or category)acrosspositionssee(DiCarlo and Cox
lm7). Finally,separableobjectand positioninformationmay allow concurrentlocalrzationandrecognitionof objects(i.e.,recognizingwhat the objectis anddetermining
u hereit is).
6.6 Evidencefor ViewpointSensitivityAcrossthe LOC
Another source of change in object appearancethat merits separateconsideration is
change across rotation in depth. In contrast to position or size changes, in which
rt4
KALANIT
GRILL.SPECTOR
invariancemay be achievedby a linear transformation,the shapeof objectschange:
with depthrotation.This is becausethe visual systemreceives2-D rettnalprojections
of 3-D objects.Sometheoriessuggestthat view-invariantrecognitionacrossobject
rotationsis accomplishedby largelyvfek-invariantrepresentations
of objects(generalizedcylinders',
Marr 1980;recognitionbycomponents',
orRBC', Biederman19871:
that is, the underlyingneuralrepresentations
respondsimilarly to an objectacrossits
views.Othertheories,however,suggestthatobjectrepresentations
areview-dependent.
thatis, theyconsistof several2-D viewsof an object(Ullman 1989;PoggioandEdelman 1990;Bulthoff and Edelman,1992;Edelmanand Bulthoff 1992;Bulthoff et al.
1995;Tarr and Bulthoff 1995).Invariantobjectrecognitionis accomplished
by interpolationacrosstheseviews(Ullman 1989;PoggioandEdelman1990;Logothetis
et al. 1995)or by a distributedneuralcodeacrossview-tunedneurons(Penettet al.
r998).
Single-unitelectrophysiology
studiesin primatesindicatethat the majority of neurons in monkey inferotemporalcortex are view-dependent(Desimoneet al. 1984:
Logothetiset al. 1995;Perrett1996:Wanget al. 1996;Vogelsand Biederrnan2002)
with a small minority (5-l0%o)of neuronsshowingview-invariantresponses
across
(Logothetis
objectrotations
et al. 1995;BoothandRolls 1998).
In humans,resultsvary considerably.Short-laggedfMRI-A experiments,in which
the test stimulusis presentedimmediatelyafter the adaptingstimulus(Grill-Spector
(Fang
et al. 2006a),suggestthatobjectrepresentations
in the LOC areview-dependent
et al. 2007;Gauthieret al. 2002;Grill-Spectoret al.1999;but seeValyearet al. 2006).
However,long-laggedfMRI-A experiments,
in which manyinterveningstimuli occur
betweenthe test and adaptingstimulus (Grill-Spectoret al. 2006a),have provided
some evidencefor view-invariantrepresentations
in ventral LOC, especiallyin the
left hemisphere(Jameset al. 2002; Vuilleumier et al. 2002) and the PPA (Epstein
et al. 2008).Also, a recentstudyshowedthat the distributedLOC responses
to objects
remainedstableacross60orotations(Egeret al. 2008).Presently,thereis no consensus
acrossexperimentalfindingsin the degreeto which ventralstreamrepresentations
are
view-dependentor view-invariant.Thesevariableresultsmay reflect differencesin
the neuralrepresentations
dependingon object categoryand cortical region, and/or
methodologicaldifferencesacrossstudies(e.g.,level of object rotation and fMRI-A
paradigmused).
We addressedthesedifferential findings in a recent study in which we used a
parametricapproachto investigatesensitivityto object rotation and a computational
modelto link putativeneuraltuning andresultantfI\4RI signals(Andresenet al. 2008,
2009).The parametricapproachallowsa richercharacterization
of rotationsensitivity
because,rather than characteizing representations
as invariantor not invariant,it
measures
the degreeof sensitivityto rotations.Using fMRI-A we measuredviewpoint
sensitivityas a function of the rotationlevel for two objectcategories- animalsand
vehicles.Overall,we found sensitivityto object rotationin the LOC, but therewere
differencesacrosscategoriesandregions.First, therewashighersensitivityto vehicle
rotationthan to animalrotation.Rotationsof 60' produceda completerecoveryfrom
adaptationfor vehicles,but rotationsof 120' werenecessary
to producerecoveryfrom
(Fig.
adaptationfor animals
6.6). Second,we found evidencefor overrepresentation
of the front view of animalsin the right pFus/OTS:its responsesto animalswere
. 16rr
' , .
I
;
-r --i[
r-:'
-i:
l -j.
-J
wHAT HAs fMRr TAUGHT us ABour oBJECT REcocNruon?
8c'
115
(a) Vehicles
}n!
R-l
ET.
o
P 1.3
au
.tr
o
It.
NI
cid
n-
E
tr
.9
Right
pFUS/OTS
$,tr"*{
0.8
o
s o.g
Adapt frontview
Adapt backview
00 60"120"180'
;{;il;il;.
Rotation from adapting view
It.
ri
(b) Animals
LO
[t.
t.
o
cr, 1.3
tr
r!
s
o
h
f
F
E 0.8
.9
o
s
0.3
k.4
Adapt frontview
Adapt baeJ<
view
|-|-r-.._..-|
00 60" 120"180"
0' 60" 120' 190"
Rotation from adapting view
Figure6.6. LO responses
Eachlinerepresents
of rotationsensitivity.
duringfMRl-Aexperiments
response
afteradaptingwith a front (dashedblack)or backview (solidgray)of an object.The
ronadaptedresponseis indicatedby the diamonds(blackfor front view, and grayfor back
r rerv).The open circlesindicatesignificantadaptation,lower than nonadapted,
P < 0.05,
oairedt-testacrosssubjects.(a)Vehicledata.(b) Animal data.Responses
are plottedrelative
:o a blank fixationbaseline.Errorbars indicateSEMacrosseight subjects.Adaptedfrom
(2009).
{ndreson,Vinberg,and Grill-Spector
higher for the front view than the back view (compare black and gray circles in
Fig. 6.6(b) righ|. In addition fMRI-A effects across rotation varied according to the
adapting view (Fig. 6(b) righ|. When adapting with the back view of animals, we
tound recovery from adaptation for rotations of 120o or larger, but when adapting with
the front view of animals, there was no significant recovery from adaptation across
rotations. One interpretation is that there is less sensitivity to rotation when adapting
s'ith front views than back views of animals. However, the behavioral performance of
rubjects in a discrimination task across object rotations showed that they are equally
\ensitive to rotations (performance decreaseswith rotation level) whether rotations are
relative to the front or back of an animal (Andresen et al. 2008), which suggeststhat this
interpretation is unlikely. Alternatively, the apparent adaptation across a 180" rotation
relative to a front animal view, may just reflect lower responses to a back animal
view.
To better characterize the underlying representations and examine which representations may lead to our observed results, we simulated putative neural responsesand
predicted the resultant fMRI responsesin a voxel. In the model, each voxel contains
a mixture of neural populations, each of which is tuned to a different object view
116
KALANIT
GRILL.SPECTOR
(a) View dependent- equal distribution
Front
o
c
g,
Vlew
Populatlon vlewtuning wldth (o)
Front
Vlew
?"a:
*
AdTtlmlt'b
-
Adftbd(lr|I
T
CD
g
tr
OffectVlew
a0 60 t20 180
Rotation from adapting view
h voxel
(b) Viewdependent- unequaldistribution
Front
g
!g
t
tD
c
tr
o
c
t I I f f i T
I * I I I *
-105'.05' 15' 75' 135' 195'
OUgr;tWew
II l l l l
x
LIIIII
$-
dlsttbufron
in voxel
Populatlon view tuning width (o)
*
^frrffitu
"9
o
o
J
o
@
1.4
a0 6012018{t
Rotation from adapting view
Figure6.7. SimulationspredictingfMRl responses
of putativevoxelscontaininga mixture
of view-dependentneural populations.Left,Schematicillustrationof the view tuning ard
distributionof neuralpopulationstunedto differentviews in a voxel.Forillustrationpurposes
we show a putativevoxel with 4 neural populations.Right,Resultof model simulations
illustratingthe predictedfMRl-A data. In all panels,the model includes6 Gaussians
tuned
to specificviews aroundthe viewing circle, separated60' apart.Acrosscolumns,the vierr
tuning width varies.Acrossrows, the distributionof neuralpopulationspreferringspecific
views varies.Diamond, Responses
without adaptation(blackfor back view, and grayfor front
view). Lines,Response
afteradaptationwith a frontview (dashedgrayline)or backview (solid
black line).(a) Mixtureof view-dependent
neuralpopulationsthat are equallydistributedin
a voxel. Narrowertuning (left)showsrecoveryfrom fMRl-A for smallerrotationsthan wider
view tuning (right).This model predictsthe samepatternof recoveryfrom adaptationwhen
neuralpopulationsin a
adaptingwith the front or backview. (b) Mixtureof view-dependent
voxelwith a higherproportionof neuronsthat preferthe front view The numberon the right
indicatesthe ratio betweenthe percentagesneuronstuned to the front versusthe back view.
Top row, ratio - 1.2. Bottomrow, ratio : 1.4. Becausethereare more neuronstunedto the
frontview in thismodel,it predictshigherBOLDresponses
to frontalviewswithoutadaptation
(grayvs. blackdiamonds)and a flatterprofileof fMRl-Aacrossrotationswhen adaptingwith
the front view.Adaptedfrom Andreson,Vinberg,and Crill-Spector(in press).
(Fig. 6.7 and Andresen et al. 2008, in press). fMRI responses were modeled to be
proportional to the sum of responses across all neural populations in a voxel. We simulated the fMRI responses in fMRI-A experiments for a set of putative voxels that
varied in the view-tuning width of neural populations, the preferred view of different
neural populations, the number of different neural populations, and the distribution of
populations tuned to different views within a voxel. Results of the simulations indicate
that two main parameters affected the pattern of fMRI data: (l) the view-tuning width
of the neural population, and (2) the proportion of neurons in a voxel that prefer a
specific object view.
WHAT HAS fMRI
TAUGHT
US ABOUT OBJECT RECOGNITION?
Lt7
Figure6.7(a) showsthe responsecharacteristicsof a model of a putativevoxel that
containsa mixture of view-dependentneural populationsthat are tuned to different
object views, in which the distributionof neuronstunEdto different views is uniform.
ln this model,niurowertuning (left) showsrecoveryf";fi fMRI-A for smallerrotations
than wider view tuning (right). Responsesto front and back views are identical when
thereis no adaptation(Fig. 6.7(a),diamonds),andthe patternof adaptationasa function
of rotation is similar when adaptingwith the front or back views (Fig. 6.7(a)). Such
e model provides an accountof responsesto vehicles acrossobject-selectivecortex
r.s measuredwith fMRI) and for animalsin LO. Thus, this model suggeststhat the
differencebetweenthe representationof animals and vehiclesin LO is likely due to
e smaller population view tuning for vehicles than for animals (a tuning width of
o < 40o producescompleterecovery from adaptationfor rotations larger than 60o,
rs observedfor vehicles).Figure 6.7(b) showsthe effects of neural distributionson
fllRl signals.Simulationspredictthat when thereare more neuronsin a voxel that are
uned to the front view, therewill be higher BOLD responsesto frontal views without
daptation (gray vs. black diamonds)and a flatter profile of fMRI-A acrossrotations
rben adaptingwith the front view, as observedin pFus/OTSfor animals.
6.6.1Implications
for Theories of Object Recognition
Orerall, our resultsprovide empirical evidencefor view-dependentobject representson acrosshumanobject-selectivecortexthat is evidentwith standardfMRI aswell as
fIIRI-A measurements.
Thesedataprovideimportantempiricalconstraintsfor theories
d object recognition and highlight the importanceof parametricmanipulationsfor
qturing neuralselectivityto any type of stimulustransformation.
Given the evidencefor neuralsensitivityto objectview, how is view-invariantobject
reognition accomplished?One appealingmodel for view-invariantobjectrecognition
rr that objectsare representedby a populationcode in which single neuronsmay be
r{cctive to a particularview, but the distributedrepresentationacrossthe entireneural
gopulationis robustto changesin objectview (Perrettet al. 1998).
Doesthe view-specificapproachnecessitatea downstreamview-invariantneuron?
ftc possibilityis thatperceptualdecisionsmay be performedby neuronsoutsidevisual
cutter and theseneuronsare indeedview-invariant.Examplesof such view-invariant
-urons havebeenfound in the hippocampus,perirhinal cortex,and prefrontalcortex
tfi,aedmanet al. 200I,2003; Quirogaet al. 2005; Quirogaet al. 2008). Alternatively,
ogalationsbasedon the population code (or a distributed code) acrossview-tuned
:urons may be sufficientfor view-invariantdecisionsbasedon view-sensitiveneural
Ipresentations.
6.7 DebatesAbout the Nature of Functional Organization
in the Human Ventral Stream
So far we haveconsideredgeneralcomputationalprinciples that are requiredby any
Sycct recognition system. Nevertheless,it is possible that some object classesor
&srains requirespecializedcomputations.The restof this chapterexaminesfunctional
118
KALANIT
GRILL-SPECTOR
specializationin the ventral stream that may be linked to these putative "domainspecific"computations.
As illustratedin Figure6.1, severalregionsin the ventralstreamexhibit higherresponsesto particularobjectcategories,.such
asplaces,faces,andbody parts,compared
to otherobjectcategories.
The discoveryof category-selective
regionsinitiateda fierce
debateaboutthe principles of functional organizationin the ventral stream.Are there
regionsin the cortex that are specializedfor any object category?Is there something
specialaboutcomputationsrelevantto specificcategoriesthat generatesspecialized
cortical regions for thesecomputations?In other words, perhapssome generalprocessingis appliedto all objects,but somecomputationsmay be specificto certain
domainsandmay requireadditionalbrainresources.
We may alsoaskabouthow these
category-selective
regionscome about:Are they innate,or do they requireexperience
to develop?
Four prominentviews haveemergedto explainingthe patternof functional selectivity in the ventral stream.The main debatecenterson the questionof whetherregions
that elicit maximal responsefor a categoryarea module for the representationof that
category,or whetherthey are part of a more generalobject recognitionsystem.
6.7.1 Limited Category-Specific Modules and a General Area
for All Other Objects
Kanwisherandcoworkers(Kanwisher2000;Op de Beecket al. 2008) suggested
that
ventraltemporalcortexcontainsa limited numberof modulesspecializedfor the recognition of specialobjectcategoriessuchas faces(in the FFA), places(in the PPA),and
bodyparts(in the EBA andFBA). The remainingobject-selective
cortex(LOC), which
showslittle selectivityfor particularobject categories,is a general-purpose
mechanism for perceivingany kind of visually presentedobject or shape.The underlying
hypothesisis that there are few domain-specificmodulesthat perform computations
that are specific to theseclassesof stimuli beyond what would be required from a
generalobjectrecognitionsystem.For example,faces,like other objects,needto be
(a domain-generalprocess).However,
recognizedacrossvariationsin their appearance
giventhe importanceof faceprocessingfor socialinteractions,thereareaspectsof face
processingthat areunique.Specializedfaceprocessingmay includeidentifyingfaces
at the individual level (e.9.,John vs. Harry), extractinggenderinformation,evaluating gazeand expression,and so on. Theseunique, face-relatedcomputationsmay be
implementedin face-selective
regions.
6.7.2 ProcessMaps
Tarr and Gauthier(2000) proposedthat object representations
are clusteredaccording
to the type of processingthat is required,ratherthanaccordingto their visual attributes.
It is possiblethat differentlevelsof processingmay requirededicatedcomputations
that are performedin localized cortical regions.For example,faces are usually recognized at the individual level (e.9., "That is Bob Jacobs"),but many objectsare
typically recognizedat the categorylevel (e.9.,"That is a horse").Followingthis reasoning,andevidencethat objectsof expertiseactivatethe FFA morethan otherobjects
(Gauthi
sugges
ordinat
et al. l(
Haxbl
cipitott
attribut
across
Haxbl
were n
distritr
that it
region
exclud
reprev
On'
binato
Given
(Biedt
a largr
consi
as mu
Mala
nizat
COTTC
and t
ities
hous
fove
to fc
et al
bias
ana
recc
tion
alsc
fielr
wit
I
fun
wHATHAsfMRrTAUGHTus ABour oBJEcr REcocNlrrox?
r.e "domainrit higher ret-s.compared
iated a fierce
m. Are there
e something
; specialized
general proic to ceftain
ut how these
e experience
rnal selectiv:ther regions
ation of that
stem.
nea
ggested that
)r the recogre PPA), and
.OC), which
ose mechaunderlying
rmputations
ired from a
need to be
). However,
ects of face
fying faces
rn. evaluatrns may be
according
attributes.
nputations
;ually recrbjects are
_ethis rearer objects
119
(Gauthieret al. 1999;Gauthieret al. 2000), Gauthier,Tarr, and their colleagueshave
suggestedthat the FFA is not a region for facerecognition,but rathera region for subordinateidentificationof any object categorythat is automatedby expertise(Gauthier
et al. 1999:Gauthieret al. 2000: Tarr ahtl Gauthier2000).
6.7.3 Distributed Object-Form Topography
Haxby and colleagues(2001)positedan "object-formtopography,"in which the occipitotemporal cortex contains a topographicallyorganizedrepresentationof shape
attributes.The representationof an object is reflectedby a distinct patternof response
acrossall ventralcortex,and this distributedactivationproducesthe visual perception.
Haxby and colleaguesshowedthat the activationpatternsfor eight object categories
were replicableand that the responseto a given categorycould be determinedby the
distributedpatternof activationacrossall ventrotemporalcortex.Further,they showed
that it is possibleto predict what object categorythe subjectsviewed, even when
regions that show maximal activation to a particular category (e.9., the FFA) were
thatthe ventrotemporal
cortex
excluded(Haxbyet al. 2001).Thus,this modelsuggests
representsobject categoryinformation in an overlappingand distributedfashion.
One of the reasonsthat this view is appealingis that a distributedcodeis a combinatorial code that allows representationof a large number of object categories.
Given Biederman'srough estimatethat humanscan recognizeabout30,000categories
(Biederman1987),this providesa neuralsubstratethathasa capacityto representsuch
a largenumberof categories.Second,this model positeda provocativeview that when
consideringinformation in the ventral stream,one needsto considerthe weak signals
as much asthe strongsignals,becauseboth conveyuseful information.
6.7.4 Topographic Representation
Malach and colleagues(2002) suggestedthat eccentricity biasesunderlie the organization of ventral and dorsal streamobject-selectiveregions becausethey found a
correlationbetweencategorypreference(higherresponseto one categoryover others)
andeccentricitybias (higherresponseto a specificeccentricitythan to othereccentricities (Levy et al. 200I; Hassonet al. 2002; Hassonet al. 2003). Regionsthat prefer
housesto other objects also respondmore strongly to peripheralstimulation than to
foveal stimulation.In contrast,regionsthatpreferfacesor lettersrespondmore strongly
to foveal stimulation than to peripheralstimulation.Malach and colleagues(Malach
et al. 2002)proposedthat the correlationbetweencategoryselectivityand eccentricity
needs.Thus,objectswhoserecognitiondependson
biasis drivenby spatial-resolution
and objectswhose
analysisof fine detailsare associatedwith foveal representations,
peripheral
representawith
recognitionrequireslarge-scaleintegrationare associated
tions.To date,however,thereis no clearevidencethateccentricitybiasesin the FFA are
also coupledwith better representationof high spatialfrequencyor smallerreceptive
fields or, conversely,that the PPA preferslow spatialfrequenciesor containsneurons
with largerreceptivefields.
Presently,there is no consensusin the field about which accountbest explainsthe
functional organizationof the ventralstream.Much of the debatecenterson the degree
r20
KALANIT
GRILL.SPECTOR
to which object processingis constrainedto discretemodulesor involves distributed
computationsacrosslarge stretchesof the ventral stream(Op de Beeck et al. 2008).
The debateis about the spatial scale on which computationsfor object recognition
occur as well as the fundamenthlprinciples that underlie specializationin the ventral
stream.
On the one hand,domain-specifictheoriesneedto addressfindingsof multiple foci
that show selectivity.For example,multiple foci in the ventral streamrespondmore
strongly to faces than to objects; thus, a strong modular accountof a single "face
module" for face recognition is unlikely. Also, the spatial extent of these putative
modulesis undetermined,and it is unclear whether eachof thesecategory-selective
regionscorrespondsto a visual area.Further,high-resolutionfMRI (1-2 mm on a side)
showsthat the spatialextentof category-selective
regionsis smallerthanthat estimated
with standardfMRI (34 mm on a side) and that theseregions appearmore patchy
(Schwarzlose
et aL.2005;Grill-Spectoret al. 2006b).
On the otherhand,a potentialproblemwith very distributedandoverlappingaccount
of object representationin the ventral stream is that, in order to resolve category
information, the brain may need to read out information present acrossthe entire
ventral stream(which is inefficient). Further,the fact that there is information in the
distributedresponsedoesnot meanthat the brain usesthe information in the sameway
that an independentclassifier does.It is possiblethat activationin localized regions
is more informative for perceptualdecisionsthan the information availableacrossthe
entire ventral stream(Grill-Spector et al. 2004; Williams et al. 2007). For example,
FFA responsespredictwhen subjectsrecognizefacesandbirds but do not predictwhen
subjectsrecognizehouses,guitars,or flowers (Grill-Spectoret aI.2004).
6.8 Differential Developmentof CategorySelectivity
from Childhood to Adulthood
One researchdirection that can shedlight on thesedebatesis an examinationof the
developmentof ventral streamfunctional organization.What is the role of experience
in shapingcategoryselectivityin the ventral stream?
6.8.L fMRI Measurements of the Development of the Ventral Stream
To addressthesequestions,our lab (Golarai et al. 2007) identified face-, place-, and
object-selective
regionswithin individualchildren(7-11yearsold), adolescents
(12-14
years old), and adult subjects(18-35 years old) while subjectsfixated and reported
infrequent eventswhen two consecutiveimageswere identical (one-backtask). We
found a prolonged developmentof the right FFA (rFFA) and left PPA (IPPA) that
manifestedas an expansionof the spatialextent of theseregionsacrossdevelopment
from age7 to adulthood(Fig. 6.8). The rFFA and IPPA were significantly larger in
adults than in children, with an intermediatesize of these regions in adolescents.
Notably, the rFFA of children was about a third of the adult size, but it was still
evidentin 85Voof children. Thesedevelopmentalchangescould not be explainedby
smaller anatomicalcortical volumes of the fusiform gyrus or the parahippocampal
L2l
SHAT H^4sfunr TAUGHTus ABour oBJEcr REcocNlrrox?
f'olvesdistributed
:eck et al. 200gt.
bject recognition
ion in the ventral
; of multiplefoci
m respondmore
,f a single ..face
,f theseputative
fiegory-selective
? mm on a side)
n rhatestimated
ar more patchy
2,500
;. r.600
3 ' -8oo
I
!
I
:4m
ffi
Ll
Match€d
PPA size
Left
Right
€ ,,*
Right
ni'iJ-'d'"#t$
Ll
2,500
2,000
1,500
1,000
500
0
Matched .
€ r,soo
E r,ooo
€ soo
0
LOC size
Right
Left
ffi z-t t vear-old
I
f
o,ooo1
to1-J,9t"*-o'o
.
,
6.OOO
g;.qffi
4,500
3,000
appingaccount
solve category
ross the entire
)rmationin the
t the sameway
'alized
regions
tble acrossthe
For example,
t predictwhen
ty
narionof the
rf experience
itream
place-,and
e n B( 1 2 _ 1 4
nd reported
c task). We
IIPPA)that
:r'elopment
y' larger in
lolescents.
t was sdll
llained by
pocampal
1,500
0
andadults.
adolescents,
children,
STS,andLOCacross
Figure6.8. Volumeof therFFA,IPPA,
10adoles20 children,
whichinclude
Fiiiedbarsindicate
all subjects,
volumeacross
average
that
of subjects
for thesubset
volumes
cents,and15 adults.Openbarsindicatethe average
g adolescents,
and13
andinilude10 children,
nerematched
confounds
for BOlD-related
than
different
indicatesignificantly
Asterisks
adults.ErrorbarsindicateSEMacrosssubjects.
from
on the y-axis.Adapted
scales
panelshavedifferent
adult,P < 0.05.Notethatdifferent
Colarai
et al.2007.
gyrus (Golarai et al. 2007),which were similar acrosschildren and adults,or higher
BOlD-related confoundsin children (i.e., larger fMRl-related noise or larger subject
motion;seeGolaraiet al. Z}}7;Grill-Spectoret al. 2008)becauseresultsremainedthe
sarnefor a subsetof subjectsthat were matchedfor fMRl-related confoundsacross
ages.Thesedevelopmentalchangeswere specificto the rFFA and IPPA, becausewe
found no differencesacrossagesin the size of the LOC or the size of the pSTS faceselectiveregion. Finally, within the functionally defined FFA, PPA, and LOC, there
were no differencesacrossagesin the level of responseamplitudesto faces,objects,
andplaces.
We alsomeasuredrecognitionmemoryoutsidethe scannerandfound that that faceand place-recognitionmemory increasedfrom childhood to adulthood(Golarai et al.
2007).Further,face-recognitionmemory was significantly correlatedwith rFFA size
(but not the sizeof otherregions)in childrenandadolescents(butnot adults),andplacerecognitionmemorywassignificantlycorrelatedwith IPPAsize(but not the sizeof other
regions)in eachof the agegroups(Fig. 6.9). Thesedatasuggestthat improvementsin
face-and place-recognitionmemory during childhood and adolescenceare correlated
*'ith increasesin the sizeof the rFFA and IPPA,respectively.
Another recent study (Scherf et al. 2007) examinedthe developmentof the venrral stream in children (5-8 years old), adolescents(ll-14 years old), and adults
usingmovieclips that containedfaces,objects,buildings,andnavigation.Using group
analysismethods,they reportedthe absenceof face-selectiveactivations(vs. objects,
buildings,and navigation)in 5- to 8-year-oldchildren in both the fusiform gyrus and
rhe pSTS.The lack of FFA in young children in the group analysismay be due to the
smallerand more variableFFA location in children, which would affect its detection
122
KALANIT
(a) Face recognition
memory vs. rFFA size
z"
1.0
o
E
o
c
.9
?_
It
GRILL.SPECTOR
(b) Place recognition
memory vs. IPPA size
(c) Object recognition
memory vs. LOC size
0.8
0.7
1.0
0.7
0.6
0.8
0.7
.c inrplcrtlcl:
.lJ\C\ 1g$ tr
0.4
0.6
j\[Cn\l\ C r'\f
0.3
0.4
0.2
o.2
0.1
0.0
0.6
0.4
c
o)
o 0.2
o
o 0.0
E.
4.2
0 1000
3000
5000
RightFFA Size
[mmst
O t-tl years old
n=20
:hc lir:t h t,'
'rr Jtthn:rrll;
1.2
0.8
:cgton:. Tht'
r tcs ing t'l l-r
Thc rcl"'
\n\r\\ n. lrrrp
0
I
Left PPA Size
[mmst
lz-t6 years old
n=10
2000
6000 10000
RightLOC size
[mmgt
# 18-35years old
n=15
Figure6.9. Recognition
memoryversussizeof rFFA,iPPA,and LOC.(a)Recognitionmemory
accuracyfor differentcategories
acrossagegroups.Recognitionmemoryfor faceswassignific a n t l y b e t t e r i n a d u l t s t h a n c h i l d r e n<(0*.P0 0 0 1 ) o r a d o l e s c e n t s ( *<*0P. 0 3 ) . A d o l e s c e n t s '
(**P < 0.03).Recognition
memoryfor faceswasbetterthanchildren's
memoryfor placeswas
betterin adultsthanin children(tP . 0.0001).Adolescents'
memoryfor placeswasbetterthan
children's(+P < 0.01).Recognition
accuracyfor objectswas not differentacrossagegroups.
Errorbars indicateSEM.(b) Recognitionmemoryfor facesversusFFAsize.Correlations
are
(r > O.49,P < 0.03),but not adults.(c) Recogwithin childrenand adolescents
significant
nition memoryof placesversusPPAsize.Correlations
within eachagegroup
are significant
(r > 0.59, P < 0.03).(d) Recognition
memoryfor objectsversusLOC size.No correlations
(P > O.4).Adaptedfrom Colaraiet al. 2OO7.
weresignificant
in a group analysis. Indeed when Scherf and colleagues performed individual subject
analysis, they found face-selective activations in 807o of their child subjects, but the
extent of activations was smaller and more variable in location compared to those in
adults. Like Golarai and colleagues, they found no difference in the spatial extent or
level of response amplitudes to objects in the LOC. Unlike Golorai and colleagues,
they reported no developmental changes in the PPA. The variant results may be due to
differences in stimuli (pictures vs. movies), task (one-back task vs. passive viewing),
and analysis methods (single subject vs. group analysis) across the two studies. For
example, Golarai and colleagues instructed subjects to fixate and perforrn a one-back
task. In this task, children had the same accuracy as adults but were overall slower
in their responses,with no differences across categories.In the Scherf study, subjects
watched movies, and perfonnance was not measuredduring the scan.It is possible that
the children had differential different eye movements or levels of attention than the
adults, and this affected their findings.
6.8.2 Implications of Differential Development of Visual Cortex
Overall,fMRI findingssuggestdifferentialdevelopmentaltrajectoriesacrossthehuman
ventral visual stream.Surprisingly,thesedata suggestthat more than a decadeis
necessaryfor the developmentof an adult-like rFFA and IPPA. This suggeststhat
experienceovera prolongedtime may be necessaryfor thenormaldevelopmentof these
lf trlt;'tlll lllCJ
r''th urc' llkc
: C p f L ' \ CI l t J l I \ '
3 \ [ ' r . - f l r " ' 0 s Cl r
:'lJtUrC .!l il
lla:tlr-ttr
[:Ii.\
rat
rc-[r'n'
]rx x tr. [:t,ufi
i r f f c ' r J I ' lr \ ' ' ! l :
wHAT HAS fMRI TAUGHT us ABour
ognition
OC size
'0@0
stze
old
I memory
a ss ignifi llescents'
laceswas
etterthan
e groups.
r t i o nsare
l, Recog8e group
're
lat ion s
subject
but the
those in
llent or
eagues.
: due to
eu'ing).
es. For
te-back
slower
ubjects
rle that
ran the
luman
ade is
ts that
f these
oBJEcr
REcocNrrrox?
L23
regions.This resultis surprising,especiallybecausethereis evidencefor preferential
viewing of face-like in newbornbabiesand evidencefor face-selectiveERPs within
the first 6 to 12 months(for a review,seeJohnson2001).One possibility suggested
by Johnsonand colleaguesis that face processinghas an innatecomponentthat may
be implementedin subcorticalpathways(e.g.,throughthe superiorcolliculus),which
biasesnewbornsto look at faces.However,cortical processingof facesmay require
extensiveexperience(GauthierandNelson2001;Nelson2001)andmay developlater.
The reasonsfor differential developmentacross ventral stream regions are unknown.Importantly,it is difficult to disentanglematurationalcomponents(genetically
programmeddevelopmentalchanges)from experience-related
components,because
both are likely to play a role during development.One possibility is that the type of
representationsand computationsin the rFFA and IPPA may require more time and
experienceto maturethan thosein the LOC. Second,differentcortical regionsmay
mature at different rates owing to genetic factors. Third, the FFA may retain more
plasticity (evenin adulthood)than the LOC, as suggestedby studiesthat show that
FFA responsesare modulatedby expertise(Gauthier et al. 1999; Tarr and Gauthier
2000).Fourth,the neuralmechanisms
underlyingexperience-dependent
changesmay
differ amongLOC, FFA, and PPA.
6.9 Conclusion
In sum,neuroimagingresearchin the pastdecadehasadvancedour understanding
of
objectrepresentations
in the humanbrain.Thesestudieshaveidentifiedthe functional
organizationof the human ventral stream,shown the involvementof ventral stream
regionsin objectrecognition,and laid fundamentalsteppingstonesin understanding
the neuralmechanisms
that underlieinvariantobjectrecognition.
Many questionsremain,however.First,whatis the relationshipbetweenneuralsensitivity to object transformationsand behavioralsensitivityto object transformations?
producebiasesin performance?
For example,emDo biasesin neuralrepresentations
pirical evidenceshowsoverrepresentation
of the lower visual field in LO. Does this
lead to betterrecognitionin the lower visual field than in the upper visual field? A
secondquestionis relatedto the developmentof the ventral stream:To what extent
is experience(vs. genes)necessaryfor shapingfunctional selectivityin the ventral
remainplasticin adulthood?If so,what is the
stream?Third, do objectrepresentations
changeslong-lasting?Fourth,
temporalscaleof plasticity,andareexperience-induced
what computationsareimplementedin the distinctcorticalregionsthat areinvolvedin
in
objectrecognition?Doesthe"aha"momentin recognitioninvolvea speciflcresponse
a particularbrain region,or doesit involve a distributedresponseacrossa largecortical
expanse?
CombiningexperimentalmethodssuchasfMRI andMEG will provideboth
this question.Fifth,
high spatialandtemporalresolution,which is criticalto addressing
what is the patternof connectivitybetweenventralstreamvisual regions?Although
the connectivityin monkeyvisual cortex has beenexploredextensively(Van Essen
et al. 1990;Moelleret al. 2008),we know little aboutthe connectivitybetweencortical
visual areasin the human ventral stream.This knowledgeis necessaryfor building a model of hierarchicalprocessingin humansand any neuralnetwork model of
lu
KALANIT
GRILL-SPECTOR
object recognition.Approachesthat combine methodologies,such as psychophysics
with fMRI, MEG with fMRI, or DTI with fMRI, will be instrumentalin addressing
thesefundamentalquestions.
I
I
I
Acknowledgments
I thank David Andresen,Golijeh Golarai, Rory Sayres,and Joakim Vinberg for their
contributionsto the researchsummarizedin this chapter.David Andresen,for his contributions to understandingviewpoint-sensitiverepresentationsin the ventral stream.
Golijeh Golarai,for spearheading
the researchon the developmentof the ventralstudy.
Rory Sayres,for his contributionsto researchon position invarianceand distributed
analysesof the ventral stream.Joakim Vinberg, for his dedicationto researchingcueinvariant and figure-groundprocessingin the ventral stream.I thank David Remus,
Kevin Weiner, Nathan Witthotft, and two anonymousreviewersfor commentson this
manuscript.This work was supportedby National Eye Institute Grants 5 R21 EY016199-02andWhitehallFoundationGrant2005-05-III-RES.
Bibliography
Andresen DR, Vinberg J, Grill-Spector K. 20[19.The representationof object viewpoint in the human
visual cortex. Neuroimage 45(2):522-536. E pub Nov, 25, 2008.
Appelbaum LG, Wade AR, Vildavski VY, Pettet MW, Norcia AM. 200f, Cue-invariant networks for
figure and background processing in human visual cortex. J Neurosci 26:11,695-1 1,708.
Bar M, Tootell RB, Schacter DL, Greve DN, Fischl B, Mendola JD, Rosen BR, Dale AM. 2001.
Cortical mechanismsspecific to explicit visual object recognition. Neuron29:529-535.
Berl MM, Vaidya CJ, Gaillard WD.2006. Functional imaging of developmentaland adaptivechanges
in neurocognitton. N euroitnage 30:67 949 l.
Biederman I. 1987. Recognition-by-components: a theory of human image understanding. Psychol
Rev 94:115-147.
Biederman I, Cooper EE. 1991. Evidence for complete translational and reflectional invariance in
visual object priming. Perception 20 :585-593.
Booth MC, Rolls ET. 1998. View-invariant representationsof familiar objects by neurons in the
inferior temporal visual cortex. Cereb Cortex 8:510-523.
Bulthoff HH, Edelman S. 1992. Psychophysical support for a two-dimensional view interpolation
theory of object recognition. Proc Natl Acad Sci USA 89:60{4.
Bulthoff HH, Edelman SY, Tarr MJ. 1995. How are three-dimensional objects representedin the
brain? Cereb Conex 5:247-2ffi.
Chao LL, Haxby JV, Martin A. 1999. Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nat Neurosci2:913-919.
Cohen L, Dehaene S, Naccache L, lrhericy S, Dehaene-LambertzG, Henaff MA, Michel F. 2000.
The visual word form area: spatial and temporal characteization of an initial stage of reading in
normal subjects and posterior split-brain patients. Brain 123 (Pt 2):291-307.
Culham JC, Danckert SL, DeSouzaJF, Gati JS, Menon RS, Goodale MA. 2003. Visually guided
grasping produces fMRI activation in dorsal but not ventral stream brain areas. Exp Brain Res
153:18f189.
Desimone R, Albright TD, Gross CG, Bruce C. 1984. Stimulus-selective properties of inferior
temporal neurons in the macaque.J Neurosci 4:2051-2M2.
WHAT HAS fMRI
TAUGHT
US ABOUT OBJECT RECOGNITION?
L25
DiCarlo JJ, Cox DD.2007. Untangling invariant object recognition. TrendsCogn Sci ll:333-341.
DiCarlo JJ, Maunsell JH. 2003. Anterior inferotemporal neurons of monkeys engaged in object
recognition can be highly sensitive to object retinal position. J Neurophysiol S9:3264- 3278.
Dill M, Edelman S. 2001. Imperfect invarianceto object translati6tiin the discrimination of complex
shapes.Perception 3O:7O7-7 24.
Downing PE, Jiang Y, ShumanM, Kanwisher N. 2001. A cortical areaselectivefor visual processing
of the human body. Science 293:247V2473.
Fielman S, Bulthoff HH. 1992. Orientation dependence in the recognition of familiar and novel
views of three-dimensional objects. Vision Res32:2385-24ffi.
Edelman S, Intrator N. 2000. (Coarse coding of shape fragments) * (retinotopy) approximately :
representationof structure. Spat Vis 13:255-264.
Eger E, Ashburner J, Haynes JD, Dolan RI, Rees G. 2008. fMRI activity patternsin human LOC
carry information about object exemplars within category.J Cogn Neurosci 20:35G370.
Epstein R, Kanwisher N. 1998. A cortical representationof the local visual environment.Nature
392:598-601.
EpsteinRA, ParkerWE, Feiler AM. 2008. Two kinds of fMRI repetition suppression?Evidencefor
di ssociable neural mechanisms. J N europ hysio I 99(6) :2877 -2886.
FangF, He S. 2005. Cortical responsesto invisible objectsin the human dorsal and ventral pathways.
Nat Neurosci8: 1380-1385.
FangF, Murray SO, He S. 2007. Duration-dependentFMRI adaptationand distributed viewer-centered
face representationin human visual cortex. Cereb Cortex 17:14O2-1411.
FarahMJ. 1995. Visualagnosia.Cambridge:MIT Press.
FreedmanDJ, RiesenhuberM, PoggioT, Miller EK. 2001. Categoricalrepresentationof visual stimuli
in the primate prefrontal cortex. Science291312-316.
FreedmanDJ, RiesenhuberM, Poggio T, Miller EK. 2003. A comparison of primate prefrontal and
inferior temporal cortices during visual categoization. J Neurosci 23:5235-5246.
Fujita I, Tanaka K, Ito M, Cheng K. 1992. Columns for visual features of objects in monkey
i nferotemporal cortex. N ature 360:343-346.
Gauthier I, Hayward WG, Tarr MJ, Anderson AW, Skudlarski P, Gore JC.2002. BOLD activity
during mental rotation and viewpoint-dependentobject recognition. Neuron 34:16l-17l.
GauthierI, Nelson CA. 2001. The developmentof face expertise.Curr Opin Neurobiol lI:219-224Gauthier I, Skudlarski P, Gore JC, Anderson AW. 2000. Expertise for cars and birds recruits brain
areasinvolved in face recognition.Nat Neurosci 3:l9l-197 .
Gauthier I, Tarr MJ, Anderson AW, Skudlarski P, Gore JC. 1999. Activation of the middle fusiform
'face
area' increaseswith expertisein recognizingnovel objects.Nat Neurosci 2:568-573.
Gilaie-Dotan S, Ullman S, Kushnir T, Malach R. 2002. Shape-selectivestereoprocessingin human
object-related visual areas.Hum Brain Mapp 15:67--79.
Golarai G, Ghahremani DG, Whitfield-Gabrieli S, Reiss A, Eberhardt JL, Gabrieli JD, Grill-Spector
K. 2001. Differential development of high-level visual cortex correlates with category-specific
recognition memory. Nat Neurosci lO:512-522.
Grill-Spector K. 2003. The neural basisof object perception.Curr Opin Neurobiol l3:159-166.
Grill-Spector K, Golarai G, Gabrieli J. 2008. Developmental neuroimaging of the human ventral
visualcortex.TrendsCogn Sci 12:152-162.
Grill-Spector K, Henson R, Martin A. 2006a. Repetition and the brain: neural models of stimulusspecific effects. Trends Cogn Sci 10:14-23.
Grill-Spector K, Knouf N, Kanwisher N. 2ffi4. The fusiform face area subservesface perception, not
genericwithin-categoryidentification. N at N eurosci 7 :555-562.
Grill-SpectorK, Kushnir I EdelmanS, Avidan G, Itzchak Y, Malach R. 1999.Differential processing
of objectsunder variousviewing conditionsin the humanlateraloccipital complex.Neuron24:187203.
126
KALANIT
GRILL.SPECTOR
Grill-Spector K, Kushnir T, Edelman S, Itzchak X Malach R. 1998a. Cue-invariant activation in
object-related areasof the human occipital lobe. Neuron 2l:l9l-202.
Grill-SpectorK, Kushnir T, Hendler T, EdelmanS, Itzchak Y Malach R. 1998b.A sequenceof objectprocessing stagesrevealed by fMRI'iri the human occipital lobe. Hum Brain Mapp 6:316-328.
Grill-Spector K, Kushnir T, Hendler T, Malach R. 2000. The dynamics of object-selectiveactivation
correlate with recognition performance in humans. Nat Neurosci 3:837-843.
Grill-Spector K, Malach R. 2001. fMR-adaptation: a tool for studying the functional properties of
human cortical neurons.Acta Psychol (Amst) 107:293-321.
Grill-Spector K, Malach R. 2fiX. The human visual cortex.Annu RevNeurosci 27:649477.
Grill-Spector K, SayresR, RessD. 2006b.High-resolutionimaging revealshighly selectivenonface
clustersin the fusiform face area.Nat Neurosci 9:1177-1185.
Hasson U, Harel M, Levy I, Malach R. 2003. Large-scale mirror-symmetry organization of human
occipito-temporalobject areas.N euron 3l :1027-1041.
Hasson U, Levy I, Behrmann M, Hendler T, Malach R. 2002. Eccentricity bias as an organizing
principle for human high-order object areas.Neuron 34:479490.
Haxby JV Gobbini MI, Furey ML, Ishai A, SchoutenJL, Pietrini P. 2001. Distributed and overlapping
representationsof faces and objects in ventral temporal cortex. Science 293:2425-2430.
Haxby JV Hoffman EA, Gobbini MI. 2000. The distributed human neural systemfor face perception.
TrendsCogn Sci4:223-233.
Hemond CC, Kanwisher NG, Op de Beeck HP. 2007. A preferencefor contralateral stimuli in human
object- and face-selectivecortex. PlnS ONE 2:e574.
Hoffman EA, Haxby JV. 2000. Distinct representationsof eye gaze and identity in the distributed
human neural system for face perception. Nat Neurosci 3:8G-84.
JamesTW, Humphrey GK, Gati JS, Menon RS, Goodale MA. 2002. Differential effects of viewpoint
on object-drivenactivationin dorsal and ventral streams.Neuron 35:793-801.
JohnsonMH. 2001. Functional brain developmentin humans.Nat Rev Neurosci 2:475483.
Kanwisher N. 2000. Domain specificity in face perception.Nat Neurosci 3:759-763.
Kanwisher N, McDermott J, Chun MM. 1997.The fusiform face area:a module in human extrastriate
cortex specializedfor face perception.J Neurosci 17:43024311.
Kastner S, De Weerd P, Ungerleider LG. 2000. Texture segregation in the human visual cortex: a
functional MRI study. J N europhysiol 83:2453-2457.
Kobatake E, Tanaka K. 1994. Neuronal selectivities to complex object features in the venffal visual
pathway of the macaquecerebralcortex. J NeurophysiolTl:85G867.
Kourtzi Z, Kanwisher N. 2000. Cortical regions involved in perceiving object shape. J Neurosci
20:3310-3318.
Kourtzi Z, Kanwisher N. 200 1. Representationof perceivedobject shapeby the human lateral occipital
complex. Science 293: I 506-1 509.
Kourtzi Z, Tolias AS, Altmann CF, Augath M, Logothetis NK. 2W3. Integration of local features
into global shapes:monkey and human FMRI studies.Neuron3T:333-346.
Lerner I Epshtein B, Ullman S, Malach R. 2008. Class information predicts activation by object
fragments in human object areas.J Cogn Neurosci 20:1189-1206.
Lerner Y HendlerT, Ben-BashatD, Harel M, Malach R. 2001.A hierarchicalaxis of objectprocessing
stagesin the human visual cortex. Cereb Cortex II:287-297.
Lerner Y, Hendler T, Malach R. 2002. Object-completion effects in the human lateral occipital
complex.CerebCortex 12:163-177.
Levy I, Hasson U, Avidan G, Hendler I Malach R. 2001. Center-periphery organization of human
object areas.Nat Neurosci4:533-539.
Logothetis NK, Pauls J, Poggio T. 1995. Shape representation in the inferior temporal cortex of
monkeys.Curc Biol 5:552-563.
WHAT HAS fMRI TAUGHT US ABOUT OBJECT RECOGNITION?
nant activation in
cquenceofobjecf
tpp 6:316-328.
electiveactivation
onal propertiesof
':619477.
selectivenonface
lrzationof human
as an organizing
127
Lueschow A, Miller EK, Desimone R. 1994. Inferior temporal mechanismsfor invariant object
recognition. Cereb Cortex 4:523-53I.
Malach R, Levy I, HassonU.2002. The topographyof high-order human object areas.TrendsCogn
Sci 6: l7Gl84.
Malach R, ReppasJB, BensonRR, Kwong KK, Jiang H, KennedyWA, Ledden PJ, Brady TJ, Rosen
BR, Tootell RB. 1995.Object-relatedactivity revealedby functional magneticresonanceimaging
in human occipital cortex. Proc Natl Acad Sci USA 92:8135-8139.
Man D. 1980. Visual information processing: the structure and creation of visual representations.
Philos TransR Soc Innd B Biol Sci 29O:199-218.
Martin A, Wiggs CL, UngerleiderLG, Haxby JV. 1996.Neural correlatesof category-specificknowledge.Nature 379:649452.
McKyton A, Zohary 8. 2007. Beyond retinotopic mapping: the spatial representationof objects in
the human lateral occipital complex. Cereb Cortex l7:1164-1112.
Mendola JD, Dale AM, Fischl B, Liu AK, Tootell RB. 1999. The representationof illusory and
real contours in human cortical visual areasrevealed by functional magnetic resonanceimaging.
d and overlapping
-i-t-t30.
rr lace perception.
J Neurosci I 9:8560-8572.
Miller EK, Li L, Desimone R. 1991. A neural mechanismfor working and recognition memory in
inferior temporal cortex. Science 254:1377-1379.
Moeller S, Freiwald WA, Tsao DY. 2008. Patcheswith links: a unified systemfor processingfacesin
stimuli in human
the macaquetemporal lobe. Science320:7355-1359.
NakayamaK, He ZJ, Shimojo S. 1995.Visual surfacerepresentation:a critical link betweenlow-level
and high-level vision. ln An invitation to cognitive sciences:visual cognition, ed. SM Kosslyn and
DN Osherson.Cambridge:MIT Press.
Nelson CA. 2001. The development and neural bases of face recognition. Infant Child Develop
rn rhe distributed
iectsof viewpoint
l0:3-18.
{7-5-.183.
Op de Beeck HP, Haushofer J, Kanwisher NG. 2008. Interpreting fMRI data: maps, modules and
dimensions.Nat RevNeurosci 9:123-135.
u*un extrastriate
Op De Beeck H, Vogels R. 2000. Spatial sensitivity of macaqueinferior temporal neurons. J Comp
Neurol426:505-518.
r risual cortex:a
Perrett DI. 1996. View-dependent coding in the ventral stream and its consequencefor recognition.
ln Vision and movement mechanisms in the cerebral cortex, ed. R Camaniti, KP Hoffmann, and
:he \ entral visual
AJ Lacquaniti, 142-151. Strasbourg:HFSP.
Perrett DI, Oram MW Ashbridge E. 1998. Evidence accumulationin cell populations responsive
to faces: an account of generalisation of recognition without mental transformations. Cognition
67:llI-145.
tape. ./ Neurosci
n lateraloccipital
PetersonMA, Gibson BS. 1994a.Must shape recognition follow figure-ground organization?An
assumptionin peril. Psychol Sci5:253-259.
of local features
Peterson MA, Gibson BS. 1994b. Object recognition contributions to figure-ground organization:
operationson outlines and subjectivecontours.PerceptPsychophys56:551-564.
Poggio T, Edelman S. 1990. A network that learns to recognize three-dimensional objects. Nature
ration by object
343:263-266.
,bjectprocessing
lateral occipital
.ation of human
rporal cortex of
Quiroga RQ, Mukamel R, Isham EA, Malach R, Fried I. 2008. Human single-neuronresponsesat
the thresholdof consciousrecognition.Proc Natl Acad Sci USA 105:3599-3604.
Quiroga RQ, Reddy L, Kreiman G, Koch C, Fried I. 2005. Invariant visual representationby single
neuronsin the human brain. Nature 435:1102-1107.
RiesenhuberM, Poggio T. 1999. Hierarchical models of object recognition in cortex. Nat Neurosci
2:1019-1025.
Rolls ET. 2000. Functions of the primate temporal lobe cortical visual areasin invariant visual object
and face recognition. Neuron 27:205-218.
KALANIT
GRILL.SPECTOR
Rolls ET, Milward T. 2000. A model of invariant object recognition in the visual system: learning
rules, activation functions, lateral inhibition, and information-based performance measures.Neural
Comput 12:2547-2572.
SawamuraH, Orban GA, Vogels R. 2006. Selectivity ofhburonal adaptationdoes not match response
selectivity: a single-cell study of the FMRI adaptation paradigm. Neuron 49:307-318.
SayresR, Grill-Spector K. 2008. Relating retinotopic and object-selectiveresponsesin human lateral
Re
occipital cortex. J Neurophysiol 100(1):249-267,Epub May 7,2008.
Scherf KS, BehrmannM, Humphreys K, Luna 8.2N7. Visual category-selectivityfor faces,places
J-
and objects emergesalong different developmental trajectories. Dev Sci l0:F15-30.
SchwarzloseRF, Baker CI, Kanwisher NK. 2005. Separateface and body selectivity on the fusiform
gyrus.J Neurosci25:11055-11059.
Kt
SchwarzloseRF, Swisher JD, Dang S, Kanwisher N. 2008. The distribution of category and location
information across object-selective regions in human visual cortex. Proc Natl Acad Sci USA
lO5:4447 4452.
Stanley DA, Rubin N. 2003. fMRI activation in responseto illusory contours and salient regions in
the human lateral occipital complex. Neuron3T:323-331.
Tarr MJ, BulthoffHH. 1995. Is human object recognition better describedby geon structural descriptions orby multiple views?Comment on Biedermanand Gerhardstein(1993).J Exp PsycholHum
Percept Perfo rm 2l : | 49Ll 505.
Tarr MJ, Gauthier I. 2000. FFA: a flexible fusiform area for subordinate-level visual processing
automatized by expertise.Nat Neurosci 3:764-769.
Thorpe S, Fize D, Marlot C. 1996.Speedof processingin the human visual system.Nature 381:520-
MinskY t I $
necessit\ of
522.
...it i: nt
Thorpe SJ, Gegenfurtner KR, Fabre-Thorpe M, Bulthoff HH. 2001. Detection of animals in natural
images using far peripheral vision. Eur J Neurosci 14:869-876.
structurc'
an intcntt
for onc Pu
account tt
Ullman S. 1989.Aligning pictorial descriptions:an approachto object recognition. Cognition32:193254.
Ungerleider LG, Mishkin M, Macko KA. 1983. Object vision and spatial vision: two cortical path-
(and idca
achierc. I
ways. TrendsNeurosci 6:414417.
Valyear KF, Culham JC, Sharif N, Westwood D, Goodale MA. 2006. A double dissociation between
sensitivity to changes in object identity and object orientation in the ventral and dorsal visual
sitting rnif ther .hi
we nc'cd
internal i'
streams:a human fMRI study. Neuropsychologia 44:218-228.
Van Essen DC, Felleman DJ, DeYoe EA, Olavarria J, Knierim J. 1990. Modular and hierarchical
organization of extrastriate visual cortex in the macaque monkey. Cold Spring Harb Symp Quant
Biol55:679496.
Vinberg J, Grill-Spector K. 2008. Representationof shapes,edges,and surfacesacrossmultiple cues
in the human visual cortex. J Neurophysiol99:1380-1393.
The earl
structurt'()1
functions t'
Vogels R, Biederman l. 2ffi2. Effects of illumination intensity and direction on object coding in
macaqueinferior temporal cortex. Cereb Cortex l2:75G766.
been rePre
reasonins ;
Vuilleumier P, Henson RN, Driver J, Dolan RI. 2002. Multiple levels of visual object constancy
revealed by event-relatedfMRI of repetition priming. Nat Neurosci 5:491499.
Wandell BA. 1999. Computational neuroimaging of human visual cortex. Annu Rev Neurosci 22;
with thc' ai
quote illu:
depicts a '
145-173.
Wang G, Tanaka K, Tanifuji M. 1996. Optical imaging of functional organization in the monkey
inferotemporal cortex. Science 272: 1665-1668.
Williams MA, Dang S, Kanwisher NG. 2m7. Only some spatial patterns of fMRI responseare read
out in task performance.Nat Neurosci 10:685-686.
*
t.,.
I
research ir
Using Ft'r
point that I
how that o
persp€cti
(b)
Eto
J
Eo
T*
E
go
I,
0
10m
Prtrntrllon Dundm{mr}
Plate 6.1. Object-, face-, and place-selectivecortex. (a) Data of one representivesubject
shown on her partially inflated right hemisphere:lateralview (/eft);ventral view (right).Dark
gray indicates sulci. Light gray indicates gyri. Black /ines delineate retinotopic regions. Blue
regions delineate object-selectiveregions (objects > scrambled objects), including LO and
pFus/OTSventrally as well as dorsal foci along the intraparietalsulcus (lPS). Red regions
delineate face-selectiveregions (faces > non-facesobjects), including the FFA, a region in
LOS and a region in posterior STS. Magenta regions delineate overlap between face- and
objects-selective
regions.Creen regionsdelineate place-selectiveregions(places> objects),
including the PPAand a dorsal region lateralto the lPS.Dark green regionsdelineateoverlap
between place- and object-selectiveregions.All maps thresholdedat P < 0.001, voxel level.
(b) LO and pFus/OTS(but not Vl) responsesare correlatedwith recognitionperformance(CrillSpectoret al. 2000). To superimposerecognition performanceand fMRl signalson the same
plot, all values were normalized relativeto the maximum responsefor the 500-ms duration
stimulus.
%sien*#!ti:ll.
For fMRl signals(blue, red, and orange ltnes) : -o/orignal(Soo
ms)
"h correct(condition)
(black)
:
mance
For recognition perfor-
ms)
"/"correct(S00
Pfr
Plate6.2. Selective
responses
to objectsacrossmultiplevisualcuesacrossthe LOC.Statistical
maps of selectiveresponseto object from luminance,stereoand motion informationin a
representative
subject.All mapswerethresholded
at P < 0.005,voxellevel,and areshownon
the inflatedrighthemisphere
of a representative
subject.(a)Luminance
objects> scrambled
objects.(b) Objectsgeneratedfrom randomdot stereograms
versusstructureless
randomdot
(perceivedas a cloud of dots).(c) Objectsgeneratedfrom dot motion versusthe
streograms
samedotsmovingrandomly.Visualmeridiansarerepresented
by red (upper),blue(horizontal),
andgreen(lower)lines.Whitecontour indicatesa motion-selective
region,or MT. Adapted
fromVinbergand Crill-Spector
2008.
;ubject
,. Dark
s. Blue
O and
egions
;ion in
e- and
rjects),
rverlap
I level.
,Crilll same
lration
rerlor-
rltltlt
9o Left
4'Lefr
Fovea 4o Right
4oup
4o Down
rllllll
ffllllll
-2-1012
Amplitude
versusmeanrosponse
% SignalChange
Plate 6.4. LO distributedresponsepatternsto differentobject categoriesand stimuluspositions.Data are shown on the lateralaspectof the right hemispherecorticalsurfacefor a
representative
subject.Eachpanelshowsthe distributedLO fMRl amplitudesaftersubtracting
from eachvoxelits meanresponse.Redand yellow,Responses
that are higherthanthe voxel's
mean response.Blue and cyan, Responses
that are lower than the voxel'smean response.
lnset,Inflated
with red squareindicatingthe zoomedregion.Note
rightcorticalhemisphere,
that patternchangessignificantlyacrosscolumns(positions)
and to a lesserextentacrossrows
(categories).
Adaptedfrom Sayresand Grill-Spector
2008.