Cornerstones of Social
Robotics in HRI
User Studies,
Psychology &
Social Development
Lasting Relationship
Perspective Taking
Social Intelligence
Cognitive
Compatibility
Teamwork
Transparent
Communication
Social Learning
Interdependence
Human Social Development
Being a body with a mind in a world of like bodies
with like minds yields multi-modal associations
the self and the appearance of other agents
the behaviors (events in time) of the self and
the behaviors of other agents
the internal workings of oneself and other
agents
Simulation Theory
& Mindreading
We use our own cognitive system “offline” to simulate others (R. Gordon)
 Cognitive
processes are dual-use
Generate own actions from our mental states
 Infer the mental states responsible other’s
actions by “stepping into their shoes”

Neural Mechanisms of Mindreading
Dual Use SELF-TOM
TOM-SELF-
TOM+SELF-
 Evidence of overlapping
brain regions involved in
SELF and TOM
TOM+SELF+
TOM-SELF+
Vogeley et al, Neuro Image 14, 170-181 (2001)
 SELF: metarepresentational cognitive
capacity to apply a “self
perspective”
 TOM: mindreading capacity
to model someone else’s
state of mind
Simulation Theory
& Social Learning

Andrew Meltzoff posits that we LEARN to
simulate via early infant imitation-based
interactions with adults

Social Learning Implications:
The experience of others can be mapped to
self --- enabling the development of learning
by observation, imitation, social referencing,
etc.
Architecture
Imitation & Mirror Systems
Dual Use Recognition/Production
Interpret Observed Actions wrt Motor
Repertoire
Vision In
Visual- Motor In
Motor
Xform
Motor
Knowledge
Motor Out
Meltzoff&Moore AIM Model
Synthesize Action from Motor Repertoire
Breazeal et al, Artificial Life (2005)
Social Structure of
Imitation
Human Engages robot in imitation game
by mimicking the robot’s facial
expressions
Social Interaction Affords Learning
Body Maps: how robot’s face (body) maps
onto social others
Mirror System: Dual use of motor
representations for recognition of action in
others and production of own action
Ability to mimic others actions
Development of Social
Referencing
 Social Referencing (~12 mos)
 Understand meaning of affective
signal from adult (~6 mos)
 Shared attention to understand
referent of adult (~9 mos)
 Associate that appraisal with beliefs
and memory (attitudes) toward
that referent
 Interact with novel object
accordingly
 Learn how to appraise novel objects
(~18 mos)
Social Referencing
Challenges
#1 Understand the affective meaning of
another’s expression
Evoke Affect from Observed
Expression
Bodyaffect
loop
Imitate Facial
expression
 Exploit Bi-Directional Body-Affect Pathways to
learn affective meaning of observed facial
expression
Empathic Mechanism
 Learn association to evoke empathic response in
robot
Recognition of Vocal Affective
Intent
Exaggerated
prosody matched
to infant’s innate
responses
pitch, f (kHz)
o
pitch, fo (kHz)
time (ms)
approval
Can you
get it?
Can you
get it?
prohibition
MMMM Oh, honey.
pitch, f (kHz)
o
A. Fernald
No no baby.
time (ms)
pitch, f (kHz)
o
Four cross-cultural
contours of infantdirected speech
That’s a good bo-o-y!
time (ms)
attention
time (ms)
comfort
Evidence for Fernald-like Contours
for (Cute) Robot Directed Speech
Valence and Arousal in Feature
Space
prohibition &
high-energy neutral
attention & approval
soothing & low-energy neutral
Breazeal & Aryananda, Autonomous Robots (2002)
Results, Multiple Languages
Test set
Strength
Class
Test
Size
Classification Result
%
Approval Attention Prohibition Soothing Neutral
Correctly
Objective scorer classifies as
strong, medium, weak
Classified
Caregivers
Naive
Strong
speakers
Medium
Weak
Approval
84
64
15
0
5
0
76.19
Attention
77
21
55
0
0
1
74.32
Prohibition
80
0
1
78
0
1
97.5
Soothing
68
0
0
0
55
13
80.88
Random perf. = 20%
Neutral
62
3
4
0
3
52
83.87
very good for caregivers
Approval
18
14
4
0
0
0
72.2
Attention
20
10
8
1
0
1
40
Prohibition
23
0
1
20
0
2
86.96
Soothing
26
0
1
0
16
10
61.54
Approval
20
8
6
0
1
5
40
Attention
24
10
14
0
0
0
58.33
Prohibition
36
0
5
12
0
18
33.33
Soothing
16
0
0
0
8
8
50
Approval
14
1
3
0
0
10
7.14
Attention
16
7
7
0
0
2
43.75
Prohibition
20
0
4
6
0
10
30
Soothing
4
0
0
0
0
4
0
Neutral
29
0
1
0
4
24
82.76
Good overall performance for
strong instances
good for naive subjects
Acceptable misclassifications
minimal confusion of valence
some confusion of arousal
Responding to Vocal Affect
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
Social Referencing
Challenges
#1 Understand the affective meaning of
another’s expression
#2 Understand the referent that their
emotive reaction is about
Saliency & Visual Attention
(Adapted from J. Wolfe VGS 2.0)
Frame Grabber
motion
w
w
attention
Top down,
task-driven
influences
habituation
w
reset
w
color
inhibit
skin tone
Eye Motor
Control
 Visual attention allows robot to look at salient objects/events
around it --- sets focus of robot’s attention
Looking Preference
“Seek toy” –
“Seek face” –
low skin gain, high saturated-color gain
Looking time 28% face, 72% block
high skin gain, low color saliency gain
Looking time 80% face, 20% block
 Internal influences bias how salience is measured
 The robot is not a slave to its environment
 Prefers behaviorally relevant stimuli
Directing Attention in
Interaction
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
Breazeal & Scassellati, IJCAI 1999
Socially Directed Attention
 Add social cues as stimuli
that explicitly contributes
to saliency in addition to
environmental “pop outs”
Gaze to Pointing
Gaze to Head Pose
Shared Attention:
Aboutness
 Extend attention model to
distinguish focus of attention
(what is salient right now)
verses referential focus (what
this interaction is about)
 Keep track of relative-looking-time
of objects in scene that robot and
human look at
 Hypothesis: object with highest
relative looking time is the shared
object referent.
QuickTime™ and a
decompressor
are needed to see this picture.
Social Referencing
Challenges
#1 Understand the affective meaning of
another’s expression
#2 Understand the referent that their
emotive reaction is about
#3 Use their appraisal to bootstrap its own
appraisal of novel objects
Object Beliefs
 Incoming perceptual features are bound into “beliefs”
about objects
 Object beliefs are tracked perceptual histories over
time:
“What this object was like recently...” +
“What this object is like right now...” +
“What we expect this object to be like soon.”
Object Beliefs &
Working Memory
Object templates are long-term
prototypical representations of objects
 Sets expectations of what this thing typically is like…
Learning Associations
 Bind labels to object templates to learn names of objects
 Bind affect to object templates to learn attitude toward
objects (i.e., somatic markers)
Social Referencing
QuickTime™ and a
YUV420 codec decompressor
are needed to see this picture.
Thomaz et al, Ro-Man 2005
Summary: HRI meets ML
Taking Learning Experience Seriously
Goal: robots that can learn in the real-world from anyone
Most people don’t have experience with Machine Learning
techniques, but they bring a lifetime of experience with social
learning interactions
Social interaction and socioaffective-cognitive skills does
“heavy lifting” of framing the
learning problem as a
collaborative process.
Social-cognitive skills
If done correctly, improves
learning process and
performance for human and
robot. Robot learns what is
intended in a transparent way.
Thank You!
Contributors
Students
Matt Berlin
Andrew Brooks
Jesse Gray
Guy Hoffman
Cory Kidd
Jeff Lieberman
Andrea Thomaz
Dan Stiehl
Our collaborators
Stan Winston Studio
Funding
TTT & DL Media Lab Consortia
ONR YIP
DARPA MARS, BICA
Toyota
For more info
 www.media.mit.edu/~cynthiab
 robotic.media.mit.edu
 Designing Sociable Robots (2002) MIT
Press