Lecture 3: Theories of Emotion
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Affective Computing in the news
Dacher Keltner
Berkeley
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Outline

Review why emotion theory useful
–

Give some positive and negative examples
Introduce some features that distinguish different theories
–
–
–
Emotions as discrete or continuous
Emotions as “atoms”, “molecules”, or “mxtures”
Emotions as a consequence or antecedent of emotion

Review some specific influential theories

In-class “experiment” (3-unit students welcome to depart)
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Why should we care about emotion theory?
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
What is a Theory

Theory explains how some aspect of human behavior
or performance is organized. It thus enables us to make
predictions about that behavior.
– Provides a set of interrelated concepts, definitions, and propositions
that explains or predicts aspects of human behavior by specifying
relations among variables.
– Allows us to explain what we see and to figure out how to bring about
change.
– Is a tool that enables us to identify a problem and to plan a means for
altering the situation.
– Create a basis for future research. Researchers use theories to form
hypotheses that can then be tested.
– Creates a basis for building software: suggests what variables are
important to measure and how they relate to each other
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Does a learned model count as a theory?

We’ll learn about machine learning approaches
–
–
Collect bunch of data
Look at lots of features and try to predict some outcome
Input
Features
(events in a
video game)

Allow us to make predictions?
–

Give insight into underlying mechanism?
–

Yes
Not typically (black box). But can answer what features are relevant
Indicate how to bring about change?
–
Not typically
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Predicted
output
And can be misleading: Famous example

In ‘80s, the Pentagon wanted to harness computer technology to make
their tanks harder to attack.

The research team went out and took 100 photographs of tanks hiding
behind trees, and then took 100 photographs of trees - with no tanks.

They trained a neural network. It reached near-perfect accuracy

Independent testing showed all “no tank” photos taken on sunny day
and all “tank” photos taken on cloudy day

Because neural network was “black box”, this not easy to discover
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Affective Computing Example

Last week, showed you system that
tries to recognize nonverbal signs of depression and PTSD

We collected data from two populations
tIn
– Craigslist (and online job-recruiting service)
– US Vets: organization that provides mental-health service for former soldiers

Tried machine learning approach

Discovered vocal pitch a strong predictor of depression
– Lower pitch predicted depression severity
– Not predicted by existing theory

Turns out there was big imbalance in our data
– US Vets had highest rates of depression
– US Vets also had highest rates of Males (most soldiers are male)
– We actually “discovered” that men speak with a lower pitch
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Advantages of building on theory

Theory makes explicit the mechanisms that (are
claimed to) underlie some behavior
– Allows us to explain what we see and to figure out how to bring about
change

Theories (typical) have good empirical support
– The theories we will discuss are supported by dozens of empirical
studies
– They may still be incorrect of insufficient but are unlikely to suffer the
sort of mistakes we just discussed
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Example: Appraisal Theory
Mental State
World Events
Argues for importance of three
interrelated concepts
• World events
• Mental state (e.g. goals)
• Emotional Response
(beliefs, goals)
Body
Expression
If we know two of these
variables, we can make
predictions about the third
Response= f(Env., Mind)
Action Tendency
Physiological Response
10
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
E.G.: Generating Emotional Response
R=f(E,M)
Beliefs,
Goals
Environment
COMPUTER PREDICITONS:
Emotional Response
Expression
Action Tendency
Physiological Response
• Computer could predict what
emotion a person might hold
• Computer could generate a
believable emotion to user
11
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
E.G.: inferring emotional antecedents
M=f-1(E,R)
Reverse Appraisal
Beliefs,
Goals
Environment
COMPUTER PREDICITONS:
Emotional Response
Expression
Action Tendency
• Computer could predict what
goal person has (i.e., what
team are they rooting for
Physiological Response
12
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Influential theory: Galen’s 4-process model of emotion
“Arousal”
Posits 4 “prototypical emotions”.
Emotions organized in 2-dimensional space (valence, arousal)
Argues emotions tend to transition along arrows.
“Angry”
“Happy”
“Sad”
“Apathetic”
“Valence”
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Galen’s 4-process model of emotion
Prototypical emotions associated with 4 specific physiological systems
“Arousal”
YB
Bl
“Angry”
“Happy”
BB
Ph
“Apathetic”
“Sad”
“Valence”
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Popular 2-dimensional model of emotion
Each prototypical state associated with a characteristic expression
YB
Bl
“Arousal”
C
S
Angry
BB
Happy
Ph
M
P
Sad
“Valence”
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Apathetic
Again, this theory affords implementation and
prediction

YB
Dimension 1
– 4 “prototypical” emotion labels but
– 2 dimensions
Bl
C
S
Angry
Happy

BB
Ph
M
Recognition “language”
Predictions
– If we recognize Anger expect YB is active
– If recognized Anger, don’t expect transition to
Apathy
– If BB active, expect sad expressions and selfreport of Sadness
P
Apathetic
Sad
Dimension 2

Control
– Can create Apathy by activating Ph system or
suppressing other systems
– Can’t control Ph by activating Apathy
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
A Hippocratic physician would prescribe treatment to void the
body of imbalanced humor. if it was a fever -- a hot, dry disease -the culprit was yellow bile or blood. So, the doctor could reduce
this by, e.g., bleeding the patient, or increase its opposite,
by Galen of Pergamun (c. 180AD)
phlegm, by prescribing cold baths.
Theory of the
Four Humours
Temperature
Hot
(Fire)
Yellow
Bile
Blood
Choleric
Black
Bile
Sanguine
Phlegm
Melancholy
Phlegmatic
Cold
(Air)
Dry
(Earth)
Wetness
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Wet
(Water)
Proof of the Theory of the Four Humours
by Galen of Pergamun (c. 180AD)

1793 an epidemic of yellow fever struck Philadelphia

Benjamin Rush (signer of Declaration of Independence)
treated by vigorous bloodletting

Each patient that recovered and survived served to prove the
theory

Each patient that died confirmed that the patient was too ill for
the treatment to work

Any issue with this?
–
Example of confirmation bias: a common decision-making bias
–
Another example: “proof that aliens have landed on earth”
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Falsifiability

A good theory is falsifiable
– Falsifiability or refutability of a theory is an inherent possibility to
prove it to be false.
– Theories that are so vague they can explain anything (ex. Psychic
readings) are not falsifiable
– The more specific a theory it is, the more likely it is falsifiable
Karl Popper: on falsifiability, testability
 Galen’s theory is falsifiable (and has be falsified)
‘What
characterises
the
empirical
method
its manner of
– Even
if Benjamin
Rush
failed to
test it is
properly
exposing to falsification, in every conceivable way, the
system to be tested. Its aim is not to save the lives of
untenable systems but, on the contrary, to select the one
which is by comparison the fittest, by exposing them all to
the fiercest struggle for survival’
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Why should we care about emotion theory

Provides a definition of “emotion” and other related concepts
that influence, or are influenced by emotion, and thus a
starting point for affective computing

Unfortunately, psychology hasn’t sorted it all out yet
– Different theories suggest different concepts and relationships between them

E.g., Say we want to recognize emotion
– Give labeled data to machine learning algorithm
– But what are the labels?



Joy vs. Hope vs. Fear?
Positive vs. Negative?
Affective computing researchers must make educated guess
about which theory to use
– But their success or failure can inform research in the social sciences
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Affective computing is interdisciplinary science
Theory
•
•
•
•
e.g.,
Rapport (positive,
Psychology
contingent,
nonverbal feedback)
Linguistics
facilitates
conflict
Neuroscience
resolution
Economics
Data
Human
Behavior
Embed capability
within interactive
virtual human
testbed
Rapport
Integrated
“Test bed”
MRE
SASO-ST
Gunslinger
DCAPS
Rapport
21
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Affective computing is interdisciplinary science
Inform theoretical
debate in social
science
Verify Implementation
• Consistent with prior
findings?
• Treated “as if” real
Human
Behavior
Test theoretical predictions
Human
Studies
Integrated
“Test bed”
MRE
SASO-ST
Gunslinger
DCAPS
Rapport
22
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
For us,
A theory should answer “What is emotion?”

Emotion is a feeling

Emotion is a state (of physiological arousal)

A brain process that computes the value of an experience --- Le Doux

A word we assign to certain configuration of bodily states, thoughts, and
situational factors – Feldman Barrett.

God’s punishment for disobedience -- St Augustine
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
But also, what is emotion NOT?
From Scherer (optional reading)
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
So what is the accepted theory of emotion?
Unfortunately, none exists
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Why no grand unified theory of emotion?
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
What is an emotion?

Components of emotion
Emphasizes that emotion potentially impacts several aspects
– Cognitive: influences or influenced by thinking
– Physiological: related to hormones, heart-rate, sweating…
– Expressive: relates to facial expressions, posture, vocal features
– Motivation: relates to goals and drives
– Feeling: relates to conscious awareness being in an emotional state
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
What is an emotion?

Phases of emotion: Emphasizes that emotions have “stages”
–
–
–
–
–
–
–
Low-level: automatic cognitive processes (e.g., reflexes)
Hi-level: deliberate, conscious cognitive processes
Goals/need setting
Examining action alternative: decision-making/action-selection
Behavior preparation
Behavior execution
Communication with other
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
What is an emotion?

Different theories emphasize different aspects:
– Appraisal theories emphasize cognitive antecedents of emotion
– Discrete emotion theories emphasize physiological and expressive
consequences of emotion

Affective computing researchers tend to draw on different
theories depending on the aspects they focus on
– E.g : emotion recognition techniques often draw upon discrete
emotion theory and avoid appraisal models
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
What is an emotion: theoretical disagreements

Different theories can be distinguished by how they
chose to define emotion with respect to the
previously-mentioned components and phases
– Is emotion discrete or continuous?
– Is emotion an “atom” or “molecule”? (Barrett)
– Is emotion an antecedent or consequent of cognition?
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Emotions as discrete categories,
biologically fixed, universal to all humans
and many animals
Emotions are a combination of several
psychological dimensions
Basic Emotions: Anger, disgust, fear,
happiness, sadness, surprise
Rene Decartes, Silvan Tomkins, Paul
Ekman
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Wilhelm Wundt, James Russell, Lisa
Feldman Barrett
Emotions as discrete categories,
biologically fixed, universal to all humans
and many animals
Emotions are a combination of several
psychological dimensions
Basic Emotions: Anger, disgust, fear,
happiness, sadness, surprise
Rene Decartes, Silvan Tomkins, Paul
Ekman
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Wilhelm Wundt, James Russell, Lisa
Feldman Barrett
Some discrete emotion theories

Tomkins
– Excitement, joy, surprise, distress,
anger, fear, shame,
dissmell (reaction to bad smell),
disgust (reaction to bad taste)

Ekman
– Sadness, happiness, anger, fear,
disgust, and surprise,
sometimes includes contempt
E.g. Le Doux fear
circuit
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Some Dimensional models
Mania ↔ depression
High self-control ↔
“letting go”
Russell & Mehrabian’s ‘77 PAD model (pleasure,
arousal, dominance)
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Russell’s ‘80 circumplex model
Implications for classification / measurement
Continuous
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Implications for classification / measurement
Discrete
Disgust
Continuous
Fear
Surprise
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Atom
Molecule or Mixture
Emotion components are tightly-coupled
and can be treated as a circuit linking
stimuli and response
Emotions are defined by loose
configuration of different components
Jaak Panksepp, Joseph LeDoux, Paul
Ekman
Phoebe Ellsworth, Klaus Scherer, Lisa
Feldman Barrett
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Atom
Molecule or Mixture
Emotion components are tightly-coupled
and can be treated as a circuit linking
stimuli and response
Emotions are defined by loose
configuration of different components
Jaak Panksepp, Joseph LeDoux, Paul
Ekman
Phoebe Ellsworth, Klaus Scherer, Lisa
Feldman Barrett
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Implications for classification / measurement

If emotion is atomic circuit, all components should be aligned
– i.e., Facial expressions, physiological response and felt emotion
should be consistently-aligned with each other
– “Emotion” can refer to the overall circuit but can be measured by any
of the components
– Measured expressions should predict physiology and felt emotion
– Multi-modal recognition should perform the best
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Implications for classification / measurement

If emotion is atomic circuit, all components should be aligned
– i.e., Facial expressions, physiological response and felt emotion
should be consistently-aligned with each other
– “Emotion” can refer to the overall circuit but can be measured by any
of the components
– Measured expressions should predict physiology and felt emotion
– Multi-modal recognition should perform the best

If emotion a molecule or mixture, components not aligned
– Allow that components influence each other but may be out of sync
– Expressions need not accurately reflect physiology and felt emotion
– Constructivist Theories (Feldman Barrett): Emotion is a label we
assign to our sensed physiological state
– Appraisal theories (Scherer & Ellsworth): Emotion is a label a scientist
might apply when different components align in a prototypical way
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Atom or Mixture

Discrete: redundancy across channels
–
Multimodal should be strictly better than unimodal
Predicted
output
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Atom or Mixture

Discrete: redundancy across channels
–
–
Multimodal should be strictly better than unimodal
Late fusion should be great
Predicted
output

Mixture: not so fast…
–
Or at least, association between modalities and predicted emotion is complex
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Top down
Behav. Drives emotion
(e.g. Appraisal Theory)
(e.g., constructivist theories)
Thought precedes emotion. Emotion
precedes and motivates behavior
Behavioral response precedes our
labelling the situation as emotional
Walter Cannon, Phoebe Ellsworth, Klaus
Scherer
William James, Stanley Schachter, Lisa
Feldman Barrett
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Top down
Bottom up
(e.g. Appraisal Theory)
(e.g., constructivist theories)
Thought precedes emotion. Emotion
precedes and motivates behavior
Behavior and body response precedes
and motivates emotion and cognition
Walter Cannon, Phoebe Ellsworth, Klaus
Scherer
William James, Stanley Schachter, Lisa
Feldman Barrett
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Appraisal Theory

“Bottom up” theories argue “seeing the bear” produces fear-like
reactions automatically

What if we knew the bear was friendly?

What if we knew the bear was chained up?
Magda
Arnold
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Appraisal Theory

Appraisal models emphasize the prior beliefs and goals determine
shape emotional responses

Explain this by arguing that cognitive processes ESSENTIAL in
initiating emotional responses
World events are “appraised” along a number of dimensions:
–
–
–
–
Is the event good or bad with respect to my goals
Did I expect the event
Can I control the event
Who do I blame for the event
Different patterns of appraisal will lead to different emotions
– I blame someone else for something bad  Anger
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Some Appraisal Theories

Ortony, Clore and Collins (OCC)
Appraisal Variables
• desirability
•appealingness
•praiseworthyness
•certainty
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Some Appraisal Models

Scherer sequential checking theory
Appraisal Variables
• Relevance
• Implication
• Coping potential
• Normative significance
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Appraisal theory takeaway

Emotions arise from appraisal of goals and beliefs
–
Emphasizes centrality of beliefs, desires and intentions to emotion elicitation

Event has no meaning in of itself

Emotion arises from how event impacts goals and beliefs

Same event will have different meaning to different people
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Classic example
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
But not always so simple

Belief: I’m standing in a room
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
But not always so simple

Belief: I’m standing in a room

Does this contradict appraisal theory?
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Example (Constructivist Theory)

Argues first step in the experience of emotion is
physiological arousal
– Seeing the bear triggers low-level automatic reactions such as arousal
and running away

We next try to find a label to explain our feelings,
usually by looking at what we are doing (behavior)
and what else is happening at the time of arousal
(environment)

Thus, we don’t just feel angry, happy, etc. We
experience general feeling and then decide what
they mean (a specific emotion)
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Schachter 2-factor theory
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Practical implication
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Basic emotion
Theory
Constructivist
Theory
Appraisal
Theory
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Lesson: Definitions matter

Geocentricity
– Placing earth at center of universe makes it difficult to predict motion
of the planets

Alchemy
– All substance can be decomposed into earth, water, air and fire
making it difficult to predict consequences of chemical reactions

Point:
– Theory important: allows us make specific predictions and explain
variance
– Important steps on way to deeper understanding
– Recognize that technological choices depend (implicitly or explicitly)
on (folk or scientific) theoretical assumptions and failure of the
technology may reflect problems with theory, not software
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
End of main
lecture
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
In-class exercise
Split class into 3 groups
• Need group of four volunteers to watch a video
• Most of class will stay put and watch this group
• Need one more group of four to watch the class
I’ll give out some handouts
• First group will mark down how they feel watching the video
• Class will guess what the first group is feeling based on their reactions
• Last group will guess what the first group is feeling based on class’s
reactions
NO TALKING
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
Discussion

Classification
– What featured did you use to identify the felt emotion

Dimensions vs. Basic emotions
– Which framework best captured the “meaning” of the interaction

Observers
– Why were (or weren’t) the 3rd group able to infer what is going on

Mirroring

How do you think a computer would do?
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch
65
CSCI 534(Affective Computing) – Lecture by Jonathan Gratch