Sensing Valence and Confusion
with Facial EMG
Phil Davis and Hsin-Ni Ho
Outline
I.
II.
Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Outline
I.
II.
Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Experimental Setup
 Electrodes attached to:
 Corrugator supercilii (“brow”)
 Zygomaticus major (“cheek”)
 Grounded on arm
Outline
I.
II.
Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Outline
I.
II.
Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Problems Addressed
 Can we sense valence from facial
electromyographic (EMG) activity?
 Intensity?
 Can we sense valence from mouse
behavior (pressure in particular)?
 In general, what types of activity can
we sense with facial EMG?
Relevant Prior Work
 Recent study by Larsen, Norris, Cacioppo
 Change in mean “cheek” EMG activity
correlated positively with valence
 Change in mean “brow” EMG activity
negatively correlated with valence
(CS = “brow” muscle, ZM = “cheek” muscle)
Outline
I.
II.
Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Experimental Procedure
 6 subjects
 General procedure:
1. Subject watches a short film clip
2. Subject answers questions about film
content and experienced affective state
 Subject uses pressure sensitive mouse
 EMG output is recorded during all
stages
Film Clip Sequence
 Each film clip was roughly 3 minutes
1. Golf Instruction (Neutral)
2. Clip from “The Champ” (Negative)
 Sadness in 94.2% (Gross & Levenson)
3. Golf Instruction (Neutral)
4. Clip from “Robin Williams Live” (Pos)
 Amusement in 84.1% (Gross & Levenson)
 Did not vary order
Outline
I.
II.
Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Self-reported Valence
Consistent with Expectations
 Neutral Valence: 4 subjects indicated
“neutral” affect, 2 subjects indicated
low intensity positive affect
 Negative Valence: All 6 reported
sadness
 Positive Valence: 5 of 6 reported
amusement
 The sixth subject did not like Robin
Williams
Example of Output
 Output normalized due to large scale
differences
Mean Output Consistent with
Prior Work
Individual Results Varied
Other Observations
 For some subjects, brow activity was
a good indicator of web form activity
Outline
I.
II.
Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Discussion
 Highly sensitive to individual subject differences and
electrode positioning
 Mean output consistent with Larsen, Norris, Cacioppo
 Do results extend to other positive/negative states?
 Unable to correlate EMG output with self-reported mood
intensity
 EMG output may be useful as an activity recognition
sensor
 Unable to correlate mouse pressure or velocity with
valence
 Dynamic model of mouse behavior may produce better
results?
Outline
I.
II.
Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Outline
I.
II.
Introduction and Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Problem Addressed
 Does the facial expression ‘frowning’ represent the
feeling of confusion?
 Can we recognize the feeling of confusion with the EMG
outputs?
 Does the importance of understanding influence the
feeling of confusion?
Outline
I.
II.
Introduction and Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Experimental Procedure
 6 subjects
 General procedure:
1. Subject listens to an audio clip
2. Subject answers questions about audio
content and experienced affective state
 EMG output is recorded during all
stages
Eliciting Confusion with Audio
Recordings
Two levels of the importance of understanding:
Low level importance of understanding.
High level importance of understanding.
Audio clips to induce different levels of confusion:
A
B
Mood control
40 s classic music
Control
spoken with standard American English
Confused by accent
spoken with accent
Confused by
meaning
A brief paragraph with confusing content spoken
with American English
Ratings and Measurements for
feeling of Confusion
 Subjective rating
 5 scale Self-report confusion rating
 Objective rating
 Test of understanding
 Measurement
 EMG responses
Outline
I.
II.
Introduction and Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Result: EMG responses
Brow
Cheek
Mean
210.14
2654
Std
214.44
5876
Difference in
importance of
understanding
NO
DF=1, P=0.28
NO
DF=1, P=0.24
Result: Average EMG responses
For brow
Brow
More EMG activity when
0.8
0.7
Normalized EMG
0.6
mood control
1L
0.5
1W
0.4
listening to clips with accent
2L
2W
0.3
3L
EMG in ‘web-filling’ parts
increase with level of confusion in
high importance of understanding
3W
0.2
0.1
0
important
not important
Cheek
For cheek
0.8
0.7
Normalized EMG
0.6
0
1L
0.5
1W
0.4
2L
2W
0.3
3L
3W
0.2
0.1
0
important
filling the web form
not important
EMG activity increase with
degree of confusion ->Subject
started to laugh
Result:
Self confusion rating and test score
Average plot
Rating and Score
Confusion and Test Score
6.00
5.00
4.00
3.00
2.00
1.00
0.00
Confusion
Test
1
2
3
4
Audio section
5
6
Self rating
confusion
Test score
Mean
3.81
2.38
Std
1.51
1.87
Difference
for 6 clips
difference
among
subjects
create the feeling of confusion consistently
Yes
DF=5, P <
0.0001
NO
DF=5,
P=0.42
Yes
DF=5, P <
0.0001
NO
DF=5,
P=0.33
Outline
I.
II.
Introduction and Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Discussion
 Huge individual difference in EMG responses ->
normalize the data
 More Brow activity when filling web forms ->
It is more correlated to ‘the process of thinking
about confusion thing’
 More Brow activity when listening to clip with
accent
 Cheek activity increase with level of confusion ->
So confused that subjects were giving up
 Difference in importance of understanding
Outline
I.
II.
Introduction and Experimental Setup
Valence
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
III.
Confusion
A.
Problems Addressed
B.
Experimental Procedure
C.
Results
D.
Discussion
IV.
Conclusions
Conclusions
 Baseline difference in EMG for different
muscles
 Individual difference in EMG response
 EMG response sensitive to electrode positioning
 Brow EMG is negatively correlated with valence
and positively correlated with feeling of
confusion
 Cheek EMG is positively correlated with valence
 Facial EMG may be useful for activity
recognition