Facial identity
and expression
perception in
the human
visual system
NANOS 2011 Vancouver
Human Vision and Eye Movement Laboratory
Medicine (Neurology)
Ophthalmology and visual sciences
Psychology
University of British Columbia
DISCLOSURES:
NO COMMERCIAL/PROPRIETARY INTERESTS
Canada Research
Chairs
FUNDING SUPPORT:
NIMH 1R01 MH069898
CIHR MOP-77615, MOP-85004, MOP-102567, MOP-106511
NSERC RGPIN 355879-08
Michael Smith Foundation for Health Research CI-SSH-035(03-1)
Canada Research Chairs 950-202111
Student support from MSFHR, CIHR, Alzheimer’s Foundation
COMMON CONCEPT: perception of identity and expression emerge from
divergent streams of face processing
Bruce and Young (1986) cognitive model
COMMON CONCEPT: perception of identity and expression emerge from
divergent streams of face processing
EXPRESSION
IDENTITY
Encode facial dynamics
Encode static structure
Invariant for static structure
Invariant for dynamic changes
• allow generalization across people
•allow stable recognition despite
changes in expression and age
• Are expression and identity functionally and structurally independent?
If so, this would imply that the human brain has evolved highly specific
systems for extracting different structural properties of the same object.
• 1. Behavioural studies
• 2. Anatomic fMRI studies
• 3. Patient studies
1. BEHAVIOURAL INVESTIGATION
use FACE-ADAPTATION to probe the functional
relationship between the perception of facial
expression and perception of facial identity
1. Behavioural adaptation
a. Expression adaptation: are these invariant for the identity of the face?
Method - PERCEPTUAL BIAS TECHNIQUE
- create a morph continuum between two expressions:
anger
fear
90/10
70/30
50/50
30/70
10/90
ambiguous stimuli
- stare at one ‘end-face’ for 5 sec..
..then flash a 300ms ambiguous morph face..
..and ask a question:
“Did you see
fear or anger?”
“EXPRESSION AFTER-EFFECT”:
Subjects are more likely to perceive the other expression
1. Behavioural adaptation
5-second Adapting stimulus
300 ms Probe stimulus
(*three expression pairs:
Angry/afraid;
happy/sad;
disgusted/surprised)
Aftereffect magnitude (%)
1. Behavioural adaptation
• Changing identity reduces but does
not eliminate the expression aftereffect.
}
• Expression representations may
have two components:
}
identity-dependent component
Same Different
person person,
Same
gender
Different
person,
Different
gender
identity-invariant component
1. Behavioural adaptation
b. Identity adaptation: is it invariant for the expression of the face?
Method - create a morph continuum between two different people’s faces:
“Joe”
“Bob”
90/10
70/30
50/50
30/70
10/90
- after adaptation to BOB, subjects are more likely to respond that a following
ambiguous face is more like JOE.
1. Behavioural adaptation
Aftereffect magnitude
Is the identity aftereffect partially dependent on expression?
Same expression
Different
expression
Answer: NO.
In contrast to expression aftereffects, there is no expression-dependent
component of identity aftereffects.
1. Behavioural adaptation
Would we find expression-dependency in the identity after-effect for more
familiar faces?
Aftereffect magnitude
Answer: NO
not
familiar
minimally
familiar
culturally
familiar
very
familiar
1. Behavioural adaptation
There is an interesting asymmetric relationship between identity and expression!
One possible (speculative) interpretation, influenced by network models….
identity
adaptation
Both
contribute to
expression
adaptation
1. STRUCTURAL INVESTIGATION
Use fMRI-ADAPTATION to probe the structural relationship
between perception of facial expression vs. identity
Fusiform face area
Superior temporal sulcus
2-D Morph Matrix with “categorical boundaries”:
2. fMRI adaptation
Expression morphing
Identity morphing
EXPRESSION BOUNDARY
IDENTITY BOUNDARY
• Ultimately, what mattered most in this experiment was not the stimulus but the perceptual
decision - whether the subject said that they saw the face as similar or different.
2. Rapid event-related fMRI adaptation study
Subject sees pairs of faces:
identical
establishes
baseline of
full adaptation
identity is identical (held constant):
similar expression
different expression
expression is identical (held constant):
similar identity
different identity
In all of these, face 2 is 33%
morph different from face 1.
Do FFA and STS differ in
adaptation effects induced by:
• physical changes in expression
vs identity?
• perception of difference in
expression vs identity?
Subjects perform 2 runs with
the same stimulus set:
•Expression task
•Identity task
2. fMRI adaptation
FUSIFORM FACE AREA
(FFA)
Release from adaptation (area ‘detects’ a difference):
Baseline =
adaptation for
repetition of
identical face
* FFA is sensitive to
any structural change
same diff
identity
same diff
expression
(as reported by subject)
# FFA adaptation
release is more when
subject perceives a
difference in EITHER
identity or expression
SUPERIOR TEMPORAL SULCUS
(STS)
2. fMRI adaptation
identity task
• STS is modulated by TASK - differences
only seen when subject is actively
processing expression
same diff
identity
same diff
expression
DURING EXPRESSION TASK:
expression task
* STS is sensitive to any stimulus change,
# STS adaptation release is more when
subject perceives a difference in EITHER
identity or expression
same diff
identity
same
diff
expression
2. fMRI adaptation
Conclusions of fMRI experiment
• FFA and STS activity is sensitive to physical properties of face,
• Correlate with perceptual report of subject, for both identity
AND expression
• STS is recruited during expression task
• Identity and expression specific signals not seen until later in
cortical hierarchy:
Identity – precuneus
Expression – middle STS
1. NEUROPSYCHOLOGY
Differential impact of LESIONS on the perception of
facial expression and perception of facial identity
fMRI
localization
data in
patients
Region of interest
prosopagnosic
Subject
prosopagnosic
prosopagnosic
3. Lesion study
3. Lesion study: are expression and identity differentially impaired?
Using 2-D morph matrix, create 4 oddity paradigm tests:
Identity varies
Identity fixed
Expression varies
Expression fixed
identity
expression
Which face
is the
different
expression?
Which face
is the
different
person?
We vary the level of difficulty by choosing
morph faces of varying distance in the
matrix
3. Lesion study
Other dimension fixed
Impaired
Identity only
normal
Impaired
expression
AND
Impaired
identity
Identity task
Impaired
expression
only
• Identity but not expression
impaired by lesions of:
OFA/FFA
Ant temporal lobe
Medial fusiform
• Expression but not identity
impaired by lesion of:
STS
Expression task
identity
varying
expression
varying
expression fixed
identity fixed
• Expression
constancy of
identity
judgments
impaired by
STS lesion!
Summary
Behavioural adaptation results:
• identity representations are expression-invariant
• expression representations have identity-dependent and identityinvariant components
fMRI-adaptation results:
• effects of perceptual decision for both expression and identity are
REFLECTED in the activity of both FFA and pSTS (does NOT mean that
they are encoded there).
Lesion results:
• “fusiform - anterior temporal stream” lesions impair identity perception
but NOT expression perception
• STS lesion impaired expression perception and impedes identity
judgments when expression must be discounted.
1. Behavioural adaptation
Precuneus?
Anterior
temporal
Insula
Amygdala
Middle
STS
FFA
Other
fusiform
regions
Posterior
STS
Bicycle tour of Vancouver! WEDNESDAY
meet at registration desk 1200h
Behavioural adaptation:
• Fox CJ, Barton JJS. What is adapted in face adaptation? The neural
representations of expression in the human visual system. Brain Res 2007; 1127: 809.
• Butler A, Oruc I, Fox CJ, Barton JJS. Factors contributing to the adaptation
aftereffects of facial expression. Brain Res 2008; 1191: 116-26.
• Fox CJ, Oruc I, Barton JJS. It doesn’t matter how you feel. The facial identity
aftereffect is invariant to changes in facial expression. J Vision 2008; 8(3): 11.1-13.
fMRI-adaptation:
• Fox CJ, Moon S-Y, Iaria G, Barton JJS. The correlates of subjective perception of
identity and expression in the face network: an fMRI adaptation study. Neuroimage
2009; 44: 569-80.
Lesions:
• Fox CJ, Iaria G, Duchaine BC, Barton JJS. Behavioral and fMRI studies of identity
and expression perception in acquired prosopagnosia. Vision Sciences Society,
Naples 2008
PERSONNEL
Andrea Butler
Chris Fox
So Young Moon
Brad Duchaine
Giuseppe Iaria
Ipek Oruç
CONTROL DATA:
Desired test
properties:
• equivalent
perceptual difficulty,
• controls perform
well but not at
ceiling
• low variance
3. Lesion study