Cues to Emotion: Face, Body,
Brain
Bob Coyne (cs6998)
1
Outline
• Emotion in voice and brain
– The voice of emotion: an FMRI study of neural responses to
angry and happy vocal expressions – (2006) Tom
Johnstone et al
• Facial Expressions and emotion
–
–
–
–
–
http://en.wikipedia.org/wiki/Paul_Ekman
Facial Action Coding System (FACS) – Ekman & Friesen
Microexpressions - Ekman
Poker tells: The Body Language of Poker (1984) Mike Caro
3D graphics facial expressions – FaceGen software
• www.facegen.com
2
The voice of emotion: an FMRI study of neural
responses to angry and happy vocal expressions –
(2006) Tom Johnstone et al
• Range of issues:
–
–
–
–
–
–
–
Emotions in general vs specific emotions
General regions of brain (left/right) vs localized
Perception vs production
Prosody vs semantic content in speech
Facial expressions vs speech
Attention vs inattention
Type of experiment, controls, and measurement
(FRMI, MEG, PET, EEG, other)
– What function does vocal affect have and how to test
3
Importance of response to vocal affect
• From the earliest stages of development, infants
respond to affect-laden vocal expressions from their
mothers (Fernald, 1989; Fernald and Morikawa, 1993)
– Why might this be?
– Relevance to the 4 approaches (darwinian, jamesian…)?
• Still important/relevant after?
– When?
– Emoticons?
• Emotion in voice recognized as well as faces across
cultures (Scherer and Wallbott, 1994)
– True?
4
Early Prior work
• Production/comprehension experiments
– Production vs perception. Makes a difference?
– Production in Right hemisphere (Hughlings-Jackson - 1915)
• How did they know?
– Neurological evidence for comprehension (Tucker et al., 1977)
– Reduced affective speech comprehension in right-hemispheredamaged listeners (bowers 1987, Heilman et al., 1984; Peper
and Irle, 1997; Ross, 1981)
• What’s missing?
– Where processed?
– Controlled experiments?
– How else to test/identify region?
5
More recent Prior work
• Specific regions for facial emotion processing (e.g.
amygdala for fear, insula for disgust)
• Little prior work on localized regions for emotional speech
for specific emotions.
• Propositional content vs emotional prosody
– Right prefrontal activation (PET - George et al. 1996)
– EEG scalp electroencephalography Pihan et al. 1997
• Accented syllable duration and fundamental frequency
• Are those correlated with emotional speech?
• Magnetoencephalography (MEG) (Imaizumi et al. 1998)
– Also found some left-hemisphere processing of propositional
and prosodic content
6
Recent Prior work
•
•
•
•
non-differentiated per emotion
Selective activation
Congruence effect with facial and vocal
Experiments
– Mitchell et al. (2003) found areas of posterior middle temporal gyrus (MTG) and superior
temporal sulcus (STS) that activated more when attending to affective prosody as compared
with semantic content of spoken words
– Grandjean et al. (2005) and Sander et al. (2005) have reported FMRI data that revealed a
region in STS that showed greater activation in response to angry speech as compared with
neutral speech
– Wildgruber et al. (2005) identified a right hemisphere network consisting of posterior STS, and
dorsolateral and orbitobasal prefrontal cortex that showed selective activation during an
emotion recognition task. Differential activations for the five emotions were not observed.
– Ethofer et al. (2006) identified regions in the right posterior MTG and STS and bilateral
inferior/middle frontal gyrus that activated more when individuals identified affective prosody
than when identifying the emotional content of the spoken words. No distinction was made
between responses to the different expressed emotions studied
– Pourtois et al. (2005) demonstrated an area in left MTG that showed heightened activation
when congruent vocal expressions and facial expressions of happiness or fear were
simultaneously presented, as compared with when only one expressive modality was
presented
7
The Brain
Whole Brain Atlas Top 100 Brain Structures:
http://www.med.harvard.edu/AANLIB/cases/caseM/case.html
8
Loci of Sulci and Gyri
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Central sulcus
Precentral gyrus
Postcentral gyrus
Precentral sulcus
Postcentral sulcus
Middle frontal gyrus
Superior frontal gyrus
Superior parietal lobule
Occipital gyri
Longitudinal fissure
9
Highly specialized brain function?
10
Hypotheses
• The question remains, therefore, of whether specific neural regions are
more engaged in the processing of some emotions than others.
• Based on the hypothesis that affiliative social vocal signals are prevalent
throughout our lives and serve a fundamental purpose in social binding we
hypothesize that emotional expressions of happiness would preferentially
engage parts of the temporal cortex and inferior frontal regions previously
shown to be involved in the processing of affective prosody.
• A further question concerning the perception of vocal expressions of
emotion is how directed attention towards or away from the expressed
emotion affects the associated neural response.
– How to test this?
• Activation for face versus voice – most studies on face
• What types of emotional speech detected? All?
11
Experimental approach
•
•
•
FMRI to detect localized regions for processing happiness vs anger in speech
Why those two emotions?
Reasons
– Control for similar acoustic signal.
– Common/basic. Easier to characterize.
– social utility hypothesis regarding happiness
•
•
•
Which theory of emotion (jamesian etc)
What was experimental setup?
Some factors
– 40 subjects
– Facial stimuli consisted of 16 grayscale images of posed expressions of anger and 16
expressions of happiness, half of them females.
– Vocal stimuli consisted of short phrases (dates and numbers) lasting on average 1 s, spoken
with either angry or happy prosody.
– Half the participants make their decision on the basis of the facial expression, while the other
half on the basis of the vocal expression
– FMRI scan during happy/angry speech
•
With corresponding or opposite facial expression
12
Experiment Issues/problems?
– Brain regions necessarily responding to emotion?
– Quality of audio? Affects results?
– FMRI measures blood flow. Relevant to neural?
– Right handed subjects. Relevant?
– Faces not correlated to the speaker….
– Why normalized audio amplitude?
– Why no semantic emotional content?
– Why short phrases?
13
FRMI
Functional MRI or functional Magnetic Resonance Imaging (fMRI) is a type of specialized
MRI scan. It measures the haemodynamic response related to neural activity in the brain
or spinal cord of humans or other animals. It is one of the most recently developed forms
of neuroimaging. Since the early 1990s, fMRI has come to dominate the brain mapping
field due to its low invasiveness, lack of radiation exposure, and relatively wide availability.
14
Experiment - results
• Happy voices elicited significantly more activation than
angry voices in right anterior and posterior middle
temporal gyrus (MTG), left posterior MTG and right inferior
frontal gyrus.
• Attention-dependent results: In contrast with brain regions
showing a main effect of vocal emotion regardless of
attentional focus, a network of brain regions including the
left insula, left amygdala and hippocampus, and rostral ACC
responded more to happy voices than to angry voices
when attending to the voice, but showed either no
difference or greater activation to angry voices than to
happy voices when attending to the face.
• Do these results support the social binding hypothesis?
15
Results - RT and accuracy
•Why longer RT for “voice” case?
•Why measure RT and accuracy at all?
16
Results
How do these graphs
support the stated results?
17
Facial Expressions and Emotion - Paul Ekman
http://en.wikipedia.org/wiki/Paul_Ekman
•
Found facial expressions of emotion are not culturally determined, but universal
across human cultures and thus biological in origin.
– How did he determine this?
– Fully true? Examples?
– How does this square with the four approaches: Darwinian? Jamesian? Social constructivist?
Appraisal?
•
Expressions he found to be universal included those indicating anger, disgust, fear,
joy, sadness, and surprise. Findings on contempt are less clear, though there is at
least some preliminary evidence that this emotion and its expression are
universally recognized.
– examples of non-universal emotions? of culture-dependent expression of same emotion? Of
individual-dependent expression of same emotion?
•
Facial Action Coding System (FACS) taxonomize all human facial expression.
(Ekman and Friesen)
– 48 action descriptors, which are a contraction or relaxation of one or more muscles.
– http://en.wikipedia.org/wiki/Facial_Action_Coding_System
– http://www.cs.cmu.edu/afs/cs/project/face/www/facs.htm
•
Microexpressions – detect lying
– How does this phenomenon square with the four approaches?
18
FACS – action descriptors
1 Inner Brow Raiser -- Frontalis (pars medialis)
2 Outer Brow Raiser -- Frontalis (pars lateralis)
4 Brow Lowerer -- Depressor glabellae, Depressor supercilii,
Corrugator supercilii
5 Upper Lid Raiser -- Levator palpebrae superioris
6 Cheek Raiser -- Orbicularis oculi (pars orbitalis)
7 Lid Tightener -- Orbicularis oculi (pars palpebralis)
9 Nose Wrinkler -- Levator labii superioris alaeque nasi
10 Upper Lip Raiser -- Levator labii superioris, caput
infraorbitalis
11 Nasolabial Deepener -- Zygomaticus minor
12 Lip Corner Puller -- Zygomaticus major
13 Cheek Puffer -- Levator anguli oris (also known as Caninus)
14 Dimpler -- Buccinator
15 Lip Corner Depressor -- Depressor anguli oris (also known
as Triangularis)
16 Lower Lip Depressor -- Depressor labii inferioris
17 Chin Raiser -- Mentalis
18 Lip Puckerer -- Incisivii labii superioris and Incisivii labii
inferioris
19 Tongue Out
20 Lip stretcher -- Risorius w/ platysma
21 Neck Tightener -- Platysma
22 Lip Funneler -- Orbicularis oris
23 Lip Tightener -- Orbicularis oris
24 Lip Pressor -- Orbicularis oris
24 Lip Pressor -- Orbicularis oris
25 Lips part -- Depressor labii inferioris or relaxation of
Mentalis, or Orbicularis oris
26 Jaw Drop -- Masseter, relaxed Temporalis and internal
pterygoid
27 Mouth Stretch -- Pterygoids, Digastric
28 Lip Suck -- Orbicularis oris
29 Jaw Thrust
30 Jaw Sideways
31 Jaw Clencher -- Masseter
32 Lip Bite
33 Cheek Blow
34 Cheek Puff
35 Cheek Suck
36 Tongue Bulge
37 Lip Wipe
38 Nostril Dilator
39 Nostril Compressor
41 Lid Droop
42 Slit
43 Eyes Closed -- Relaxation of Levator palpebrae superioris;
Orbicularis oculi (pars palpebralis)
44 Squint
45 Blink -- Relaxation of Levator palpebrae superioris;
Orbicularis oculi (pars palpebralis)
46 Wink -- Relaxation of Levator palpebrae superioris;
Orbicularis oculi (pars palpebralis)
19
Facial Action Coding System - FACS
FACS can be used to distinguish two types of smiles:
• insincere and voluntary Pan American smile:
contraction of zygomatic major alone
•
sincere and involuntary Duchenne smile:
contraction of zygomatic major and inferior part of orbicularis oculi.
20
Pictures of Facial Affect (resource) –
Ekman and Friesen
• 110 pictures of posed facial emotions
– 10 second display to subjects (male and female 50%
mix)
– 70% accurately identified
• Mark each example as one of:
–
–
–
–
–
–
Happy
Sad
Fear
Anger
Surprise
Disgust
21
89
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
22
56
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
23
79
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
24
102
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
25
54
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
26
76
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
27
52
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
28
34
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
29
94
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
30
108
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
31
Tally your results!
• Should match most of the time
32
89
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
33
56
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
34
79
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
35
102
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
36
54
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
37
76
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
38
52
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
39
34
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
40
94
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
41
108
•Happy
•Sad
•Fear
•Anger
•Surprise
•Disgust
42
Ekman - Microexpressions
• Play video
• Impact on Darwinian, jamesian, etc?
• What is deception w/r emotion?
43
Poker Tells – The Body Language of
Poker (1984) Mike Caro
• Poker players exhibit “tells” that reveal valuable
information about their hands or intentions to
bet
• Some tells are unconscious and directly related to
the internal emotion (excitement at a good hand,
disappointment at a bad one)
• Some tells are the result of “acting the opposite”
to try to deceive the opponents.
• Tells involve facial expressions, gestures, hand
positions, body posture, and other behaviors
44
Poker Tells – instant reaction
The Body Language of Poker (1984) Mike Caro
45
Poker Tells – acting the opposite
46
Poker Tells – guarding the hand
47
Expressions in virtual 3D faces – FaceGen
Neutral Pose
www.facegen.com
48
Expressions in virtual 3D faces – FaceGen
Anger
49
Expressions in virtual 3D faces – FaceGen
Disgust+smile(open)
50
Expressions in virtual 3D faces – FaceGen
blend other expression, phonemes, etc.
51
Next Week
• Hand in either:
– A list of possible topics for your course project,
with a paragraph on each, or (if you have already
discussed the project with me and decided on
one)
– A one page project specification with:
•
•
•
•
Project goal
What you will do to achieve
What resources you will need
What prior work you can draw on
52