Complementing Embodied Conversational
Agents by Implementing Compliments
Measuring the Effect of Compliments in Embodied Conversational Agents
Group 23
Marieke Agterbos | Tim van Bremen | Carel Jansen | Lisa Oud | Justin Post
Research Question
To what extent do compliments have an impact on the
user experience of embodied conversational agents?
Questions
1. Are the ECA’s compliments recognized as compliments? (Part I)
2. Experience with vs. without compliments (Part II)
A. Perceived liking of the agent
B. Perceived level of comfort
C. Perceived information quality
Background theory
 Making machines more human (Picard, 2000)
 Minimize gap between real life contact and digital contact
 Lifelike ECA (Louwerse et al., 2009)
 Is sensitive to moods and sentiments
 Politeness and variation in language (Behnam & A. Niloufar, 2011)
 Add value and believability
 Function of Compliments (Manes en Wolfson, 1981)
 Goodwill and solidarity
 Three types of personal compliments (Jucker, 2009)
 Implicit, explicit or indirect
 Behavioral realism might be more important than photorealism (Groom, 2009)
 Generating positive response
 More productive conversation
Pre-test
1. Bot: ‘Hello, how are you?’
2. User: ‘Great, and you?’
3. Bot: ‘Good, so am I.’
4. User: ‘Good.’
5. Bot: ‘What was the name of the lead guitar
player of the band called “A band of Gypsies?”’
6. User: ‘Jimi Hendrix.’
7. Bot: ‘That is correct.’
8. Bot: ‘Someone told me you are very bright.’
 Are the ECA’s compliments recognized as compliments?
 Participants read a transcript of an ECA conversation
 Questionnaire on different parts of the conversation:
Pre-test results
 84 participants (53, 31)
‘Yes, this is a significant difference (p<0.01)’
Pre-test results
Explicit
Implicit
Indirect
‘You are such a nice
person!’
‘People like you make
this world a better
place.’
‘Someone told me you
are very bright.’
‘I wish everyone was
as nice as you.’
‘It seems like I’m
‘My colleague told me
having a conversation
you have a great
with an intellectual.’
personality.’
Non-contextual
‘You are smarter than
most respondents!’
Contextual
‘You answered really
fast!’
‘You seem to know
your instruments!’
‘According to our
system, you score
better than most
people!’
‘According to our
‘It seems like you know
database, you know a
your politics!’
lot about music!’
Experiment Design
Contextual
Non-contextual
Based on input data
Based on ‘nice-to-hear’statements
Related to conversational content
Unrelated to conversational content
‘You seem to know your instruments!’ ‘People like you make this world a
better place.’
 Measured variables
 Liking
 Comfort
 Information Quality
Experiment
 53 Participants
 Conversation with bot
 Quiz setting
 Questionnaire
 Perceived liking
 Perceived level of comfort
 Information quality
Results
Effects
Results
 H1. Contextual compliments have a positive effect on the user experience of an ECA.
 H1-a. perceived liking of the agent.
 H1-b. perceived level of comfort.
 H1-c. perceived quality of information.
 H2. Non-Contextual compliments have a positive effect on the user experience of an ECA.
 H2-a. perceived liking of the agent.
 H2-b. perceived level of comfort.
 H2-c. perceived quality of information.
 H3. Contextual compliments have a more positive effect than non-contextual
compliments.
 H3-a. perceived liking of the agent.
 H3-b. perceived quality of information.
 H3-c. perceived level of comfort.
Results
 H1. Contextual compliments have a positive effect on non-contextual
compliments on the user experience of an ECA.
 H2. Non-Contextual compliments have a positive effect on the user
experience of an ECA.
 H3. Contextual compliments have a more positive effect than noncontextual compliments.
 H4. Compliments have a positive effect on the user experience of an ECA.
 H4-a. perceived liking of the agent.
 H4-b. perceived quality of information.
 H4-c. perceived level of comfort.
Conclusion
 Compliments have a positive effect on the user experience of an
ECA.
 However, not on likingness.
 The type of compliment matters
 Non-contextual > contextual
Discussion
Pros:
 Pretest, kind of compliments accounted for.
 Pretest in line with experiment outcome.
 All experiments under supervision of a test leader.
Improvements:
 Sample too small
 Language, not native language of all subjects
 Variable mount of compliments given
 Compliments when correctly answered
 Definition of comfort, persuasiveness, likeliness might differ individually
Questions?
Measuring - Liking

‘I would like to be friends with her’

‘If I was upset, she would be able to cheer me up.’

‘She is a bad listener’

Rated on Likert scale 1-7
Measuring - Level of comfort

During the conversation, I felt:

Comfortable

Involved

Open

Relaxed

Warm
 Rated on Likert scale 1-7
Measuring - Information quality

The extent to which the information was:

Correct

Helpful

Persuasive

Insightful

Relevant
 Rated on Likert scale 1-7
Context
 Controlled environment
 Participants from same population