Affective Computing: Machines
with Emotional Intelligence
Hyung-il Ahn
MIT Media Laboratory
…doesn’t notice you are annoyed.
[Doesn’t recognize your emotion]
You express more annoyance. He ignores it.
[Stupid about handling your emotion]
He winks, and does a happy little dance before exiting.
[Stupid about expressing emotion.]
Skills of Emotional Intelligence:
•
•
•
•
•
Expressing emotions
Recognizing emotions
Handling another’s emotions
Regulating emotions \
if “have emotion”
Utilizing emotions
/
(Salovey and Mayer 90, Goleman 95)
Research Areas
• Robotic Computer
- Recognizing another’s emotions
- Expressing emotions
- Handling another’s emotions
• Affective and Cognitive Decision Making
- Regulating and utilizing emotions
- Affect as a self-adapting control system
Affect changes the operating characteristics of other three domains
(cognition, motivation, behavior)
Recognizing Emotions
Recognition of three “basic” states:
Future “teacher for every learner”
Can we teach a chair to recognize
behaviors indicative of interest and
boredom? (Mota and Picard)
Sit upright
Lean Forward
Slump Back
Side Lean
Boredom
Interest
What can the sensor chair
contribute toward inferring the
student’s state: Bored vs. interested?
Results (on children not in training data, Mota and Picard, 2003):
9-state Posture Recognition: 89-97% accurate
High Interest, Low interest, Taking a Break: 69-83% accurate
Detecting,
tracking, and
recognizing
facial
expressions
from video
(IBM BlueEyes camera
with MIT algorithms)
Autism Spectrum Conditions
Center for Disease Control and Prevention (2005)
– 1 child in 166 has ASC
Mind-Read > Act > Persuade
hmm … Roz
looks busy. Its
probably not a
good time to
bring this up
Analysis of nonverbal cues
Inference and reasoning
about mental states
Modify one’s actions
Persuade others
Real time Mental State Inference
El Kaliouby and Robinson (2005)
Facial feature
extraction
Head & facial
action unit
recognition
Head & facial
display
recognition
Mental state
inference
Head pose
estimation
Feature point
tracking*
* Nevenvision face-tracker
hmm … Let
me think
about this
Affective-Cognitive Mental States
Baron-Cohen et al.
AUTISM RESEARCH CENTRE,
Agreeing
CAMBRIDGE
Complex
Mental
States
(subset)
Concentrating
Absorbed
Concentrating
Vigilant
Disagreeing
Interested
Thinking
Unsure
Asking
Curious
Impressed
Interested
Baffled
Confused
Undecided
Unsure
Assertive
Committed
Persuaded
Sure
Disapproving
Discouraging
Disinclined
Brooding
Choosing
Thinking
Thoughtful
Physically animated Robotic Computer
(joint with Prof. Cynthia Breazeal)
Goal: increase user movement without distraction and annoyance,
further social-rapport building
Robotic Computer (RoCo): A physically
animated computer
Learning: the user can guide RoCo’s behavior
by explicit and implicit rewards and punishments
(Reinforcement Learning)
tracking
attentive (+)
pleased (+)
rewarding (+)
punishing (-)
attentive
entertaining
distracted
stretching
responding to pleasure
responding to
displeasure
tracking
tracking
slumped
pleased
displeased
rewarding
punishing
RoCo’s postures congruous to the user affect
“Stoop to Conquer” : Posture and affect interact to influence
computer users’ comfort and persistence in problem solving tasks
People tend to be more persistent and feel more comfortable
when RoCo’s posture is congruous to their affective state
N=(17)
Procedure and Tasks
Tracing Task:
a solvable and an unsolvable puzzle
Decision-making Task (in Experiment 2):
to make subjects keep the target posture longer
Affective Cognitive Decision Making
(Example 1) Two-armed bandit gambling tasks
Inspired by Bechara & Damasio’s IOWA gambling tasks (Bechara et al. 1997)
The left arm has ‘Negative Valence’
Arousal (uncertainty) as ‘feeling uneasy’
The right arm has ‘Positive Valence’
Arousal (uncertainty) as ‘feeling lucky’
(Example 2) Decision making under risk
Loss aversion: People strongly prefer avoiding losses than acquiring gains
• ‘Risk-Averse’ choices in the domain of ‘Likely Gains’
$3000 (Pr=1)
>
Option 1
Expected value = $3000 (Gain)
$4000 (Pr=0.8)
$ 0 (Pr=0.2)
Option 2
<
Expected value = $4000 * 0.8 + $0 * 0.2 = $3200 (Gain)
• ‘Risk-Seeking’ choices in the domain of ‘Likely Losses’
- $3000 (Pr=1)
<
Option 1
Expected value = - $3000 (Loss)
- $4000 (Pr=0.8)
$ 0 (Pr=0.2)
Option 2
>
Expected value = - $4000 * 0.8 + $0 * 0.2 = - $3200 (Loss)
The PT (Prospect Theory) value function
( x  xref ) , x  xref  0
v( x  xref )  

 (( x  xref )) , x  xref  0
0    1, 0    1,   1
- Reference Dependence: gains and losses
are defined relative to the reference point ( xref
- Concave above the reference point
- Convex below the reference point
(Tversky & Kahneman)
)
- Diminishing sensitivity:
less sensitive to outliers for both gains
and losses
- Loss aversion: the function is
steeper in the negative (loss) domain
Endowment Effect
• people place a higher value on objects they own relative to
objects they do not.
• In one experiment, people demanded a higher price for a coffee
mug that had been given to them but put a lower price on one
they did not yet own.
• The endowment effect was described as inconsistent with
standard economic theory which asserts that a person's
willingness to pay (WTP) for a good should be equal to their
willingness to accept (WTA) compensation to be deprived of the
good. This hypothesis underlies consumer theory and
indifference curves.
• The effect is related to loss aversion and status quo bias in
prospect theory.
(Example 3) Effects of mood on decision making
(Lerner & Keltner 2000, 2001, 2004)
Happiness
Optimistic about judgments of
future events
Pessimistic judgments of
future events,
Risk-Aversive choices
Fear
Anger
Optimistic judgments of
future events,
Risk-Seeking choices
Reverse Endowment Effect
Sadness
Subjective Value Function
(mood influences decision making)
Affective Cognitive Learning and Decision Making
• A new computational framework for learning and decision making
inspired by the neural basis of motivations and the role of emotions in
human behaviors
• A motivational value (reward)-based learning theory:
decision value = extrinsic (cognitive) value + intrinsic (affective) value
extrinsic value from the cognitive (deliberative and analytic) systems
intrinsic value from multiple affective systems such as Seeking, Fear, Rage, and other circuits.
• Probabilistic models: Cognition (cognitive state transition), Multiple affect
circuits (Seeking, Joy, Anger, Fear, …), and Decision making model
• Any prior and learned knowledge can be incorporated for expecting the
consequences of decisions (or computing the cognitive value)
Choice 1
Effort (r = -80)
Fearless/
Neutral / Fearful
Mood
To destroy the ring
in Mordor with less effort
Prob
Reward
Incidental Emotions
-30 0 20
70
Expected Values
Cognitive Expectations
choice 1 = 20, choice 2 = 20
Choice 2
Effort (r = -30)
Valenced Uncertainty Values
Anticipatory Emotions
from the Seeking Circuit
choice 1 = positive, choice 2 = negative
Pr = 0.5
Fear
Anticipatory Emotions
from Other Circuits
• Success (r = 100)
• Fail (r = 0)