Affect Reflection Technology in Face-to-Face Service Encounters
by
Kyunghee Kim
B.S., Korea Advanced Institute of Science and Technology (2007)
Submitted to the Program in Media Arts and Sciences,
School of Architecture and Planning,
in partial fulfillment of the requirements for the degree of
ARCHIVES
Master of Media Arts and Sciences
MASSACHUSETTS INSTITiUTEi
OF TECHNOLOGY
at the
OCT 2 6 2009
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
LIBRARIES
September, 2009
© Massachusetts Institute of Technology 2009. All Rights Reserved.
Signature of Author: ...............................
Certified by:...........
.....................
..............
Kyunghee Kim
/ y Kyunghee Kim
Program in Media Arts and Sciences
August 7, 2009
. ....................................
Rosalind W. Picard
Professor of Media Arts and Sciences
Program in Media Arts and Sciences
Accepted by: ..................
.....
. . . ......./ - .
Deb Roy
Chairman, Departmental Committee on Gaduate Studies
Program in Media Arts and Sciences
Affect Reflection Technology in Face-to-Face Service Encounters
by
Kyunghee Kim
Submitted to the Program in Media Arts and Sciences,
School of Architecture and Planning,
on August 7, 2009 in partial fulfillment of the
requirements for the degree of
Master of Science in Media Arts and Sciences
Abstract
This thesis examines the role of facial expressions in dyadic interactions between a banking
service provider and customer. We conduct experiments in which service providers manipulate their
facial expressions while interacting with customers in one of three conditions: In the neutral
condition the banker tried to maintain a neutral facial expression; in the smiling condition the banker
tried to smile throughout the interaction; in the empathetic condition the banker tried to respond with
the same or complementary facial expressions. Results show that the customers (n=46) were more
satisfied with the interaction when they perceived the service provider was empathetic. More
significantly, the service provider and customer shared synchronized facial expressions with many
prolonged smiles, when customers said the service provider was empathetic. We suggested three
different criteria to investigate customer satisfaction as follows; according to what the service
provider tried to convey, what the customer perceived and what was actually detected in their
interactions. According to the analysis of the interactions, smiling bankers who shared smiles were
evaluated as the best while smiling bankers who did not share smiles with customers were appraised
similar to non-smiling bankers.
Thesis Supervisor: Rosalind W. Picard
Title: Professor of Media Arts and Sciences
Affect Reflection Technology in Face-to-Face Service Encounters
by
Kyunghee Kim
Thesis Reader: .....................
...
Deb Roy
Associate Professor of Media Arts and Sciences
MIT Program in Media Arts and Sciences
..............................................
Affect Reflection Technology in Face-to-Face Service Encounters
by
Kyunghee Kim
Thesis Reader:......
Dan Ariely
James B. Duke Professor of Behavior Economics
Duke University
Affect Reflection Technology in Face-to-Face Service Encounters
by
Kyunghee Kim
Thesis Reader: ....................
David Price
Executive-in-Residence, Bank of America
MIT Media Lab
10
Acknowledgements
I would like to thank my advisor, Rosalind W. Picard, who supported me running interesting
psychology experiments with her constant interest and feedback over a long period and encouraged
me not to give up anything. I also would like to thank my colleagues in the Affective Computing
group; Jackie Lee who always listened to my words and gave me good advice; Ming-Zher Poh who
patiently and cheerfully waited for me for our heart phones project; Micah Eckhardt who showed me
MIT is not only the place for technology but also the place full of cool potters; Rob Morris who did
not hesitate to volunteer his girlfriend to help me label a lot of DVDs; and I appreciate my other lab
mates for having been good colleagues: Hyungil Ahn, Shaundra Daily, Elliot Hedman, Ehsan Hoque,
Rana el Kaliouby.
I could not thank my friend, Nandi Bugg, who was working with me as an undergraduate
researcher, more or less than others for her hard work to run the experiments together and clean up of
the data. Without her exact time keeping habits, this thesis would not have been able to be born. I
thank and miss Robin Glancy who spent lots of his time off-work interacting with customers. I hope
to see his father's masterpieces at the Museum of Fine Arts some day. I also enjoyed working with
Adnai Mendez who amused me with his funny jokes and stories about his first baby. I thank
Stephanie Carpenter who helped me to reserve the lab space and recruit an ethnically diverse group
of people so that the experiment did not have just my Korean friends. I bet she will do a great job in
the Ph.D management department at the University of Michigan.
I love my Korean friends who entertained me with their hilarious lives and bugged me to
have boba tea at Allston while I was exhausted. Especially Sungmin Kim, you have been the best
roommate ever.
I have had a wonderful summer in Cambridge because of all of you. And to my family; Even
if we are away I always listen to your love and I love you the most.
To my family
Contents
1. Introduction.................................................................20
1.1 B ackground ................. . ................ ...........................................
.................. 21
1.2 Overview .........................
.....................
2. Experiment Design................................................
.............
23
................... 24
2.1 General Description of the Experiment .................................................................. 24
2.2 Participants - Hired Bankers ..............................................
......
..... 26
2.3 Participants - Customers ..............................................................
27
.................................
27
2.4 Procedures ....................
2.5 Facial Expressions Coding .................. ................................................... . .......28
3. M easures ........................................................................................................
3.1 Facial Expression Measures ...................
.
......................
31
31
3.2 Customer Satisfaction Measures .................. ....................................................... 32
4. Banker M anipulation...........................................................34
4.1 Manipulation Checks .....................................................
.. ............
...........
34
4.2 Banker and Customer Perspective Differences ......................................................... 44
5. First Perspective Labeling and Second Perspective Labeling......................................46
...............................
5.1 Data Format
....
...
. .
...............
.......47
5.2 Agreement between First and Second Perspective Labeling ........................................ 50
6. Customer Satisfaction Data Analyses...................................................................
6.1 Hypotheses.............................
55
55
......................................
6.2 Analysis 1. Cognitively Reported Customer Perception .............................................. 56
..........
........................... ....
6.3 Analysis 2. Banker Manipulation
57
6.4 Analysis 3. Affectively Measured Synchronized Facial Expressions ................................. 58
6.5 Summary ....................
.....
.....................
66
7. Dyadic Communication Data Analyses................................................................68
............
7.1 Hypotheses................. ....................................
7.2 Analysis 1. Cognitively Reported Customer Perception ...........................
7.3 Analysis 2. Banker M anipulation ....................................
...
68
69
...................... 73
7.4 Analysis 3. Affectively Measured Synchronized Facial Expressions ................................. 76
7.5 Summary
...........................................................................................
8. Specific Classification and Analysis of Smiles ........................................
79
..... 80
8.1 Smiles in Context ......................................................................... 80
8.2 Validation of Social, Genuine and Greeting Smile .................. ................................. 81
9. Conclusions and Future Work ............................................................................. 83
9.1 Summary .................................
9.2 Suggested Future Work
...................
...............
..........
83
.............................. .......................................... 84
Appendix A: 361 Banker and Customer Facial Expressions Synchrony IDs ........................ 86
Bibliography.................................................
......................................... 87
List of Figures
Figure 2-1. nonreciprocal one-with-many design ........................................................... 25
Figure 2-2. Setting where banker and customer interact ..................................................
26
Figure 2-3. Labeling interface used by banker and customer .............................................
29
Figure 2-4. Labeling interface used by four outside coders .............................
30
Figure 4-1. Banker Manipulation Checks ......................................
....................... 35
Figure 4-2. Histogram of Measurement 1.The percent of smile .......................................... 39
Figure 4-3. Histogram of Measurement 2.The percent of neutral facial expressions ................... 40
Figure 4-4. Histogram of Measurement 3.The number of banker's transitions ........................... 41
Figure 4-5. Histogram of Measurement 4.The difference between banker's transitions and customer's
transitions ......................................................................................... 42
Figure 4-6. The number of smile and The percent of smile .....................
..................... 43
Figure 4-7. The number of neutral facial expressions and the percent of neutral facial
expressions .....................................
...............................
Figure 5-1. Banker's Labeling of Banker's DVD ..................................
43
48
Figure 5-2. Customer's Labeling of Banker's DVD ..................................................... 48
Figure 5-3. Customer's Labeling of Customer's DVD ..................................
Figure 5-4. Banker's Labeling of Customer's DVD .....................
.................... 49
.................... 49
Figure 5-5. Banker's Labeling of Banker's DVD and Customer's Labeling of Banker's DVD.......50
Figure 6-1. Interaction satisfaction, Information satisfaction and Empathy score ...................... 56
Figure 6-2. Interaction satisfaction, Information satisfaction, Empathy score ............................ 57
Figure 6-3. Classification Rules ............................................................ 59
Figure 6-4. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 361 ................
61
Figure 6-5. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 19 (Y-axis: from 0 to
61
1) ...................................................................................
Figure 6-6. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 19 (Y-axis: from 0 to
0 .2) .................................................................................................................. 6 2
Figure 6-7. Distribution of Facial Expressions Synchrony IDs from ID 20 to ID 38.....................62
Figure 6-8. Distribution of Facial Expressions Synchrony IDs from ID 39 to ID 57.....................63
Figure 6-9. Distribution of Facial Expressions Synchrony IDs from ID 97 to ID 115..................63
Figure 6-10. Distribution of the Sum of Facial Expressions Synchrony IDs from ID 1 to ID 19......64
Figure 6-11. Distribution of the Sum of Facial Expressions Synchrony IDs from ID 115 to ID
64
133 ...........................................................
Figure 6-12. Interaction satisfaction, Information satisfaction and Empathy score .................... 66
Figure 7-1. Ten Synchronized Facial Expressions that take up the largest percentage in the entire
interaction in each group ..........................................................
..
71
Figure 7-2. Ten most frequent synchronized facial expressions in each group ........................... 72
Figure 7-3. Ten Synchronized Facial Expressions that take up the largest percentage of the entire
interaction in each group ..............................................................
74
Figure 7-4. Ten most frequent synchronized facial expressions in each group ........................... 75
18
Figure 7-5. Ten Synchronized Facial Expressions that take up largest percentage in the entire
interaction in each group ..........................................................
..................
77
Figure 7-6. Ten most frequent synchronized facial expressions in each group ........................... 78
Figure 8-1. The examples of social, genuine, and greeting smiles ................................... 81
Figure 8-2. Cohen's Kappa Coefficient among three coders ..........................
82
Chapter 1
Introduction
Imagine you went out to a local restaurant for dinner. You would probably enjoy the dinner more
if the waitress is friendly, kind and smiling rather than salty and disgruntled. Hochschild(1983)
showed how service providers, especially flight attendants, have to manage their emotion and display
positive affect when they interact with their passengers. This work is the first notable work in the
emotional labor field of organization behavior area that has influenced other work thereafter such as
exploring the inner feelings of service providers (Rafaeli and Sutton, 1989) and selecting a service
provider who would get less psychologically stressed (Morris & Feldman, 1996). The methodology
in their research focused on examining whether a smiling service provider affected the customer to
feel better or whether smiles were contagious between the service provider and customer by looking
at whether both of them smiled or not. In this thesis, we examine how nonverbal communication
between a service provider and a customer affects the customers' perceived satisfaction. Among
nonverbal communication features such as facial expressions, eye contact, postures and body
20
gestures, we primarily analyze the dynamics of facial expressions between people in face-to-face
conversations. Specifically, we look at both the service provider's smiles and the customer's smiles
as well as customers' reported feeling about the service. Previous studies have investigated
behavioral mimicry in dyadic interaction by having human coders at the location of the experiment to
check whether a certain behavior, such as smiling, happened over the entire interaction (Tsai, 2001).
Other approaches have recorded the interactions and had human coders review the film and count the
number of times a behavior occurred (Chartrand & Bargh, 1999; Barsade, 2002). We propose
quantitatively more accurate ways of measuring mimicry behavior by measuring the number of times
when it occurred and the percentage of time it takes up in the entire interaction. In addition, we
explore empathetic communication not only through shared smile behavior but also with other
complementary facial expressions such as "caring-upset" and "smile-interested". Since emotion is
subjective and emotions such as caring are hard to recognize, we also have people label their videos
after the interaction and compare the labels of what they said they felt with the labels of what others
perceive in their video. With regard to the analysis of smiles, we classify smiles into three different
kinds depending on the intent and context in which the smile happened to elicit its authenticity.
Grandey(2005) classified authentic and inauthentic smiles by their intensity and facial muscles, but
we would rather take into account the intent and the context.
1.1 Background
During conversations people tend to mimic one another's facial and body gestures, such as
smiling together, nodding their heads in unison, or each putting their hand on their chin. Research has
shown that synchronized nonverbal cues can influence face-to-face communication in a positive
manner (Kendon, 1970). Many aspects of mimicry behavior have been studied by social scientists. In
21
Chartrand's work, he demonstrated the chameleon effect, showing that those participants that
interacted with a confederate who imitated the participant's behavior, compared with the case in
which the confederate did not imitate the participant, felt the interaction was more pleasing
(Chartrand & Bargh, 1999). Other research has also investigated the positive influence of mimicry
behavior in varying situations. In student-teacher interaction, the rapport between student and teacher
was stronger when the students were copying the teacher's behavior (LaFrance, 1982). In counselorclient conversation clients preferred those counselors who mimicked the clients' expressions to those
who did not (Maurer & Tindall, 1983).
There are many arguments, and growing evidence, to account for human behavioral mimicry.
According to the common-coding theory (Prinz, 1997; Knoblich & Flach, 2003), the representations
of generated action are affected by the representations of perceived action and vice versa. Decety
claims that people have similar representations of action (Decety J & Sommerville, 2003) and that
people mimic the physical movements of one another because they are projecting the other person's
situation to their own (Decety, 2004). Empathy produces this "kinesthetic" imitation (Lipps, 1903),
which induces people to think that they are sharing similar affective states and experiences with their
conversation partners (Decety, 2004). The result is that people feel as if they "connect" with others,
which influences the building and sustaining of relationships with others (Chartrand, 2005). As a
result people create positive social and emotional qualities including affiliation and rapport by
unconsciously mimicking the physical movements and expressions of one another when they interact
(Chartrand, 2005).
Mimicry behavior can also be advantageous in establishing successful business relationships.
In a study of facial mimicry between a service provider and a customer, at a coffee shop, there was a
positive correlation between mutually similar facial expressions and positive customer
evaluation(Barger, 2006). Moreover, a waitress received higher tips when she mimicked the
customers by repeating the order that she was told (Baaren, 2003). In service-oriented businesses that
22
rely on face-to-face interaction, such as coffee shops, restaurants, banks or hotels, it is vital to
establish and maintain good relationships with the customers. Although research in business
management has shed light on the importance of employee-customer interactions, little has been
done with respect to the analysis of dyadic interactions that focus on the behavior, and subsequent
influences, of one person's actions on the other. Notable examples by Pugh and Tsai demonstrate that
positive affect, smiling, and engaging eye contact can positively influence the customer's experience
(Pugh 2001; Tsai 2001).
1.2 Overview
The remainder of this thesis is organized as follows: chapter two explains how the experiment is
designed to have a banker interact with a customer altering his facial expression; chapter three
elaborates on measures that we quantify to understand customer satisfaction and facial expressions;
chapter four and chapter five emphasize the perspective differences between banker and customer;
chapter six and chapter seven analyze customer satisfaction and dyadic communication; chapter eight
is one of the most interesting parts of this thesis that investigates different kinds of smiles; In chapter
nine, we conclude our findings.
Chapter 2
Experiment Design
2.1 General Description of the Experiment
The general design of the experimental interaction is that of a professional banker interacting
with a customer interested in learning about financial services (Figure 2-1). The banker provides two
kinds of financial services, which are similar to real world services provided at a retail branch. The
first service is to cash a $5 voucher from the customer participant as compensation for participating
in the study. This part is designed to simulate a cashing a check scenario. The participant was told up
front that they would get $10 for compensation, but the banker told them they would have to fill out
more paperwork after the study to get the rest of the money they were owed and could only get $5
....
....
....
....
..
now. This manipulation was made to instill a slightly negative state in the customer in order to
approximate more accurately the situation where a customer might be going to a real bank assistant
for help. After the experiment ended the participant received the rest of the money without additional
paperwork. The second service is to explain one of the financial services that a customer chose to
learn more about: Home Equity Line of Credit (HELOC), Individual Retirement Arrangement (IRA),
Certificate of Deposit (CD), mortgages, credit card, or student financial plans (529 plans & CDs).
This part is to simulate the situation in which bank customers ask questions and receive information
about the financial product they are interested in. There were two male bankers and one of them
interacted with forty-six participants, while the other interacted with forty-five participants. There is
one banker and one customer in each experiment and the experiment was nonreciprocal one-withmany design (David, 2007) (Figure 2-1). The experiment was conducted in a room equipped with a
desk, two chairs, bank service advertising pamphlets and two cameras to make the appearance alike
to a personal banking service section at banks (Figure 2-2). One camera was used to record the
banker's facial expressions and the other was used to record the participant's facial expressions.
Customer46, Neutral facialexpressions
Customer 1, Netralfacialexpressions
Benlrl
C toCrer2, Empathetic facial eKpressions
Customer 47, Empathetic facialepresions
Customer 3, Awayssmilin facial expressions
Customer 48,Alwyssmilin
4,Neutral facial expressions
Customer
Customer49 Neutrlfacial eprssions
ICustomer,Empathetic facialexpressions
Customer 6, Anws
flig facial expressions
Snied
Bnflaeticlear2si
facialexpressions
CustomerSO,
Customer 51, Always smng facial expressiom
O, A waymilins acial expresio
SCustomer
smiligfacial expressions
45, Always
Customer
Customer9l, Neutral facial
Figure 2-1. nonreciprocal one-with-many design
pressions
Figure 2-2. Setting where banker and customer interact
2.2 Participants - Hired Bankers
We hired two professional personal bankers, each with over two years of career experience as
a personal banker, to do what they usually do at work - explain financial services. During the hiring,
we asked them if they would be willing and able to manipulate the type of facial expressions
displayed during interaction with the customer. Each banker agreed to alter his facial expressions in
three different ways, following these exact instructions:
Manipulation 1 - Neutral facial expressions: Please try to sustain neutral facial expressions
regardless of the changes in the customer's facial expressions over the entire interaction.
Manipulation 2 - Always smiling: Please try to keep smiling regardless of the changes in the
customer's facial expressions over the entire interaction.
Manipulation 3 - Complementary facial expressions i.e., empathetic : Please try to
understand the customer's feeling and respond to it appropriately by smiling when the customer
seems to feel good, showing caring facial expressions when the customer expresses concern, showing
neutral facial expressions when you need to express that you are listening to the customer sincerely
and carefully, etc.
Throughout the experiment, the bankers interacted with the customer as they would normally
do in a banking setting aside from the expression manipulation. This included greeting a customer,
providing proper information, and thanking the customer for their time. The facial expressions of the
bankers were unobtrusively videotaped and audio-recorded from the moment they met and greeted
the customer to the end when the customer left the seat.
2.3 Participants - Customers
Thirty males and sixteen females (n=46) were recruited through flyers who were interested in
receiving information about different financial services. Before the experiment started, they were told
that their face and voice would be recorded as banks normally do for security reasons. However, they
were not told that their facial expressions would be analyzed. This was to prevent them being aware
of the purpose of the study. Afterward, they were told about the expressions and helped to label them.
2.4 Procedures
Prior to the participant entering the room the banker was told which expression manipulation
to conduct. The participant was then allowed into the experiment room where they would interact
with the banker and learn about specific financial services. At the end of the experimental interaction,
which took about 10 minutes, both the banker and participant filled out 9-point Likert scale surveys
evaluating the quality of the service based on the most comprehensive and popular instrument
SERVQUAL(Parasuraman, 1985 & 1988) and the attitude of the banker. While the banker and
participant completed the surveys the experimenter transferred the video recorded experimental
session to DVDs. After the banker and participant were finished with surveys they were asked to
label the video data for their facial expressions and emotions. After labeling their own video
information they labeled the videos containing the person they interacted with.
2.5 Facial Expressions Coding
In this study, the banker labeled his own video data and the participants also labeled their
own data. Then, the banker labeled the participant's video data and the participants labeled the
banker's video data. Lastly, human coders not involved in the study labeled both the bankers' and the
participants' data. In the interface of the labeling software that the banker and the participant used,
"VideoLAN-VLC media player" plays the DVD and the labeling software provides an entity to enter
the time when a certain facial expression was observed and seven emotion labels to select (Figure 23). These seven labels are: smile, concerned, caring, confused, upset, sorry, and neutral. If there was
no proper label to choose from, the user could press "Other" and enter another label that they think is
appropriate for the expression. The labelers were instructed to stop playing the video and click on the
label button when they saw a facial expression, and then to continue to play the video until they saw
a change in the facial expression. On the right side of the user interface, there was a text box
displaying the time and the labeling result and it was editable so that the user could annotate the
reason for each facial expression, e.g. "smile - he made me laugh". By providing the space to type
the comment, we could learn more about the intent behind the facial expressions, especially the
smiles, where we would later categorize them into "Greeting Smile," "Social Smile," and "Genuine
28
Smile." For example, when the annotation was "smiling to greet the customer", it was classified as
"Greeting Smile" and when the annotation was "smiling to be polite to the customer", the smile was
classified as "Social Smile". "Genuine Smile" can be annotated as "hearing about getting the cash",
"glad to hear about tax deductions", "the banker told me a funny joke", etc. There were four human
coders not involved in the actual experiments. These labelers used an online video annotation
program, VidL, developed by the experimenters to label the video data (Figure 2-4). The interface
contains nine labeling buttons for facial expressions, which are composed of smile, concerned, caring,
confused, upset, neutral, satisfied, surprised and other and four labels for gestures including head nod,
headshake, chin on hand, open arms. The last three buttons are to record the points when the
participant started filling out a survey asking about a demographic profile at the very beginning of
the interaction and the last scene of the interaction since we could not observe the participants face
during this period.
Figure 2-3. Labeling interface used by banker and customer
Figure 2-4. Labeling interface used by four outside coders; The labels differ slightly here and
these data turned out to not be used in this thesis.
Chapter 3
Measures
3.1 Facial Expression Measures
3.1.1 Percent synchrony time of the facial expressions
We measured the duration of each of the facial expressions studied for both the banker and
participant. In total there are nineteen facial expressions that can be assigned by the human coder
(Table 3-1). Initially, we provided eight facial expressions labels shown in Figure 2-3 and the human
coders entered other emotion labels such as "Surprised", "Interested", "Annoyed" when they pressed
the button "Other". The labels in Table 1 include the seven labels provided in the labeling software
as well as the labels that came out of the "Other" category. Therefore, there are 19* 19 = 361 possible
pairs of synchronized facial expressions between a banker and participant and each synchrony is
assigned a unique identification. For example, "Banker: Smile - Customer: Smile" is assigned to
"Synchrony ID : 1" and "Banker: Concerned - Customer: Smile" is assigned to "Synchrony ID : 20".
In this context, synchronization means a pair of facial expressions between the banker and the
customer that overlaps in time. The percent synchrony time of the facial expressions is defined as the
percentage of the time each synchrony takes up in the entire interaction; the length of the synchrony
is divided by the entire interaction time length.
ID
Facial Expressions
ID
Facial Expressions
1
Smile
11
Relieved
2
Concerned
12
Interested
3
Caring
13
Bored
4
Confused
14
Enthusiastic
5
Upset
15
Persuasive
6
Sorry
16
Annoyed
7
Neutral
17
Survey
8
Other
18
The end
9
Satisfied
19
Missing Label
10
Surprised
Table 3-1. Twenty-one labels for the facial expressions
3.1.2. Frequency of each facial expression
This measure counts how many times each facial expression happened over the entire
interaction.
3.2 Customer Satisfaction Measures
3.2.1. Interaction satisfaction
The participants answered the question, "How satisfied are you with the interaction overall?",
with a 9-point Likert scale rating.
3.2.2. Information satisfaction
The participants also evaluated information satisfaction with the question, "How satisfied are
you with the financial information provided?", using a 9-point Likert scale rating.
3.2.3. Empathy of the bankers
A survey was given to ascertain the customer's interpretation of the service provider's
empathetic attitude. The survey was based on the most commonly used instrument SERVQUAL
(Parasuraman, 1985 & 1988) that investigates five aspects of the service;"Tangibles, Reliability,
Responsiveness, Assurance, Empathy." The four items in the "Empathy" section of SERVQUAL
were adopted and the rating used a 9-point Likert scale.
3.2.4. Customer perception
The customers were asked to choose one of three attitudes for the banker: neutral, always
smiling, and empathetic. We measured this to see whether customer perception did or did not agree
with what the banker intended to convey.
Chapter 4
Banker Manipulation
4.1 Manipulation Checks
Figure 4-1 shows three labeled interactions illustrating interactions where the banker
successfully performed the facial expressions for each condition. In Figure 4-1(a), "Neutral," the
banker is showing neutral facial expressions most of the time, visualized with light blue color, and
other facial expressions such as smile and concerned are rarely observed. In Figure 4-1(b) "Always
Smiling", the banker is smiling throughout the interaction, which is visualized with orange bars. In
Figure 4-1(c), "Complementary facial expressions", we can see more dynamics in the banker's facial
expressions, i.e., transitions between different facial expressions, and we observe where the banker
,~:::::::::::~;;;;;;;~
and customer are smiling together. Additionally we see that the banker was responding with "caring"
facial expressions when the customer was expressing "confused" or "other".
-
Smile
-Concerned
Banker
- -- Caring
-Confused
Upset
Customer
-
Sorry
Neutral
-Other
0:00:48 0:01:52 0:02:57 0:04:02 0:05:07 0:06:12 0-07:16 0:08:21 0-09:26 0:10:31 0:11:36 0:12:40 0:13:45 0:14:50 Time(hr:mm:sec)
(a)Neutral manipulation
-
Smile
-Concerned
Banker
-
Caring
-Confused
-
Customer
Upset
Sorry
Neutral
-
Other
0:00:26 0:01:09 0:01:2 0:02:36 0:05:19 0:04:02 0:04:45 0:05:28 0-06:12 006:55 0:07:38 0:08:21 0:09:04 Time(hr:mm:sec)
(b)Always Smiling manipulation
Banker
*
m
a
a
an
e
.
Ii
-
Smile
-
Concerned
-
Caring
-Confused
Customer
-
Upset
sorry
Neutral
-Other
0:00:43 0:02:10 0:03:36 0:05:02 0:06:29 0:07:55 0:09:22 0:10:48 0:12:14 0:13:41 0:15:07 0:16:34 0:18:00 0:19:26 Time(hrmm:sec)
(c) Complementary facial expressions manipulation
Figure 4-1. Banker Manipulation Checks
We analyzed banker 2's data which has forty-six pairs of interactions in this section as well
as in the remainder of the thesis since banker 1's data still needs to be cleaned up in a right format.
Table 4-1 shows six quantitative measurements to distinguish three different manipulations. These
six measurements are as follows:
1. The percent ofsmiles is defined as the time during which the banker was smiling during
the entire interaction time. For example, in Figure 4-1 (c), the sum of each smile segment colored
with yellow bar is 3 minutes and 14 seconds and the total interaction time is 19 minutes at 33 seconds.
Therefore, the percent of smiles is
3 minutes 14 seconds
3 minutes 14 seconds=
19 minsutes 33 seconds
0.165 = 16.5%. The first row in Table 4-1
summarizes the average of the percent of smile in 46 interactions for each manipulation.
2. The percent of neutralfacial expressions is similar to the first measurement, the percent of
smiles, except that this measurement calculates how much percentage the neutral facial expressions
takes up in each interaction. The values in Table 4-1 are averaged over 46 interactions.
3. The number of banker transitionscounts how many times the banker changed his facial
expressions from one to another. For example, in Figure 4-1(b) , the banker changes twice,
smile(yellow bar) -* concemed(red bar) - smile(yellow bar).
4. The difference between banker's transitionsand customer's transitionsis calculated by
subtracting the number of customer's facial expressions transitions from the number of banker
transitions. For example, in Figure 4-1(b), the number of banker's transitions is 3 and the number of
customer's transitions is 13. Therefore, the difference between the banker's transitions and
customer's transitions is 3-13 = -10 in this case. The negative value means the customer had more
transitions and was more dynamic. The positive value implies that the banker made more transitions.
If this value is zero, both the banker and the customer made the same number of transitions.
5. The number of smiles counts how many times the banker smiled in the interaction. If a
banker starts to smile and stops smiling and his facial expressions changes into another but smiling
....
then it is counted as one smile. For example, in Figure 4-1(b), the number of smiles is 2 and, in
Figure 4-1(c), the number of smiles is 11.
6. The number of neutralfacialexpressions is similar to the number of smiles except that it
counts how many times the banker showed neutral facial expressions.
As we can see from Table 4-1, the banker was smiling most of the time throughout the
"always smiling" manipulation, which was 96.18% of the interaction time on average, while he was
smiling only 1.01% of the interaction time in "neutral" manipulation. The banker was also
successfully following the manipulation instruction by maintaining neutral facial expressions 98.75%
of the interaction time.
Banker
Manipulation
Neutral Neutral
Manipulation
Manipulation
Always Smiling
Manipulation
Complementary
Facial Expressions
Manipulation
Measurement
The percent of smile
1.0%
96.2%
25.6%
The percent of Neutral
98.8%
3.7%
68.0%
0.8
0.9
13.4
The difference between
banker's transitions and
customer's transitions
-14.3
-12.9
-4.3
The number of smiles
0.5
1.4
6.6
The number of neutral
facial expressions
1.3
0.4
6
Facial Expressions
The number of banker
transitions
Table 4-1. Banker smiled most of the time in the "Always Smiling" manipulation, showed
neutral facial expressions for 98.75% of the interaction time in "Neutral" manipulation and was
more dynamic in "Complementary Facial Expressions" manipulation.
Figure 4-2 shows the histogram of these six measurements. From the histogram of Measurement
3,the number of banker transitions, we can see that the banker made more transitions during the
complementary facial expressions manipulation averaging 13.4, which is much more dynamic than
the other two groups averaging close to 0.8. In addition, the dynamics between the banker and the
customer tend to become similar in "Complementary Facial Expressions" manipulations. We can
confirm this tendency from the histogram of Measurement 4,"The Difference Between Banker
Transitions and Customer Transitions" since their difference was lower in "Complementary Facial
Expressions" manipulations averaging -4.3 compared to the other two cases that were -14.3 and -12.9.
Neutral Manipulation (N=15)
16
14
S 12
10
8
0
6
4
0
0-10
10-20
20-30
30-40
40-50
50-60
60-70
70-80
80-90
90-100
(%)
(a) Neutral Manipulation
Always Smiling Manipulation (N=16)
0-10
10-20
20-30
30-40
40-50
50-60
60-70
70-80
80-90 90-100
(%)
80-90 90-100
(%)
(b) Always Smiling Manipulation
Empathetic Manipulation (N=15)
0-10
10-20
20-30
30-40
40-50
50-60
60-70
70-80
(c) Empathetic Manipulation
Figure 4-2. Histogram of Measurement 1.The percent of smile
-I
Neutral Manipulation (N=15)
16
14
S 12
A
8
6
2
0
0-10
10-20
20-30
30-40
40-50
50-60
60-70
70-80
90-100
(%)
80-90 90-100
(%)
80-90 90-100
(%)
80-90
(a) Neutral Manipulation
Always Smiling Manipulation (N=16)
14
12
0
10
8
0o
6
2
0
0-10
10-20
20-30
30-40
40-50
50-60
60-70
70-80
(b) Always Smiling Manipulation
Empathetic Manipulation (N=15)
4.5
2.5
.
2
.o
1.5
0.5
0
0-10
10-20
20-30
30-40
40-50
50-60
60-70
70-80
(c)Empathetic Manipulation
Figure 4-3. Histogram of Measurement 2.The percent of neutral facial expressions
-I
Neutral Manipulation (N=15)
14
, 12
S
2
o
0-2
3-5
6-8
9-11
12-14
15-17
18-20
21-23
The number of banker's transitions
(a)Neutral Manipulation
Always Smiling Manipulation (N=16)
16
14
12
10
8
a
4
2
o
0-2
3-5
6-8
9-11
12-14
15-17
18-20
21-23
The number of banker's transitions
(b) Always Smiling Manipulation
Empathetic Manipulation (N=15)
4.5
l"
4
3.5
3
2.5
S
2
S1.5
1
0.5
n
0-2
3-5
6-8
9-11
12-14
15-17
18-20
21-23
The number of banker's transitions
(c)Empathetic Manipulation
Figure 4-4. Histogram of Measurement 3.The number of banker's transitions
r-
......................
...........
.................
...........
.....
.......................
Neutral Manipulation (N=15)
6
5
4
3
2
0
(-)50-55 (-)45-49 (-)40--44 (-)35-39 (-)30-34 (-)25-29 (-)20-24 (-)15-19 (-)10-14 (-)5-9
0-(-)4
(+)1-5 (+)6-10 (+)11-15
The difference between banker's transitions and customer's transitions
(a)Neutral Manipulation
Always Smiling Manipulation (N=16)
6
5
o
4
3
2
0
(-)50-55 (-)45-49 (-)40-44 (-)35-39 (-)30-34 (-)25-29 (-)20-24 (-)15-19 (-)10-14 (-)5-9
0-(-)4
(+)1-5 (+)6-10 (+)11-15
The difference between banker's transitions and customer's transitions
(b)Always Smiling Manipulation
Empathetic Manipulation (N=15)
7
6
I
5
3
2
(-)50-55 (-)45-49 (-)40-44 (-)35-39 (-)30-34 (-)25-29 (-)20-24 (-)15-19 (-)10-14 (-)5-9
0-(-)4
(+)1-5 (+)6-10 (+)11-15
The difference between banker's transitions and customer's transitions
(c)Empathetic Manipulation
Figure 4-5. Histogram of Measurement 4.The difference between banker's transitions and customer's
transitions
-~
I----
---
--
I-
I
I
I
II
I
I
----
I
I--t--~-
I
I
N
---------
---r---
---
0.8 ------
- --
*
*
------ -I-
-
0.2-----
Ahvays Smilng
--
- -
-
--
-)--
I
0
2
1
4
3
6
5
8
7
Empathetic
9
Mean of Neutral
SMean of Empathetic
I Mean of Always Smiling
10
The number of smile
Figure 4-6. The number of smiles and the percent of smiles
I
I
I
II
S0.8------ ---
-
I
I
------------
I
I
I
I
I
--------
---
-----
0.6 -------
I
-
-
0.4------0---0
Neutral
I
SI
Empathetic
I
I I Always Smiling
0.2 -----
0
A
A
i
1
2
-
- --
-
-
i
3
4
5
6
7
*
- -.I
I
|
8
10
9
*
Mean ofNeutral
MeanofEmpathetic
Mean ofAlways Smiling
The number of neutral facial expressions
Figure 4-7. The number of neutral facial expressions and the percent of neutral facial expressions
Measurements 5 and 6 can be observed more clearly when we combine these measures with the
percentage measurements, i.e., measurement 1 and measurement 2 (Figure 4-6 & Figure 4-7). That is,
there are a smaller number of smiles or neutral facial expressions in "always smiling" and "neutral
facial expressions" manipulations because the banker was told to keep the same facial expressions,
while the percent is high because the banker was trying to keep the same facial expressions all the
time. Meanwhile, in the "complementary facial expressions" manipulation, there are more smiles or
neutral facial expressions, but the percentage is not the highest one among the groups. From this
measurement, we can confirm there are three distinct forms of manipulations taking place.
4.2 Banker and Customer Perspective Differences
Interestingly, despite the differences between the three facial manipulations acted by the
banker (Table 4-1), a customer's perception of the banker's attitude did not always agree with a
banker's manipulation.
ustomer Perception
Neutral
Always Smiling
Empathetic
6
= 0.4
15
2
= 0.1
15
8
6
- = 0.4
Banker Manipulation
7
= 0.5
15
Neutral
-
Always Smiling
-
Complementary
Facial Expressions
2
16
3
15
= 0.1
-
-
16
= 0.5
5
15
16
7
15
Table 4-2. Banker manipulation vs. customer perception
Table 4-2 shows that when the banker tried to maintain a smile throughout the interaction,
only half of sixteen customers reported that they thought the banker was always smiling, while two
of sixteen customers thought the banker's attitude was neutral. Surprisingly, six out of fifteen
customers in the neutral manipulation sessions reported that they perceived the bankers attitude as
"Always Smiling". This may be due to voice tone and the typical notion that bankers usually smile at
their customers. Meanwhile, the probability that the customer would feel the banker was empathetic
was highest when the banker was trying to respond with complementary facial expressions and the
probability of the customer perception to be "Always Smiling" was highest when the banker intended
to always smile. Since a customer's perception does not always match a banker's intent, we needed
something more objective for our analysis. We therefore conducted three kinds of analysis,
classifying the data into three conditions, i.e., neutral, always smiling, empathetic (complementary
facial expressions), according to three different measures: the cognitively reported customer
perception, a banker's intent, and the measured synchrony labeled using their facial expressions. We
present these three analyses in chapter six and chapter seven after discussing the differences in
perspective in chapter five.
Chapter 5
First Perspective Labeling and Second
Perspective Labeling
After filling out the survey asking about their interaction satisfaction, both the banker and the
customer reviewed their own video data recorded during their interaction. Firstly, they labeled their
own facial expressions from the DVD that recorded them. Then the experimenter switched DVDs so
that the banker could label the customer's facial expressions and the customer could label the
banker's facial expressions. The first perspective labeling refers to the facial expressions labeling that
was done while each person was watching their own DVD and the second perspective labeling
corresponds to the facial expressions labeling that was done while they were watching their partner's
DVD. In sections 5.1 and 5.2, we compare these two different perspectives to observe the facial
expressions and list which facial expressions tend to have relatively more agreement between these
two perspectives and which facial expressions are ambiguous.
5.1 Data Format
Since both the banker and the customer reviewed their own DVD and their partner's DVD, there
were four labeling files in each experiment; banker's labeling of his own DVD, banker's labeling of
the customer's DVD, customer's labeling of his/her own DVD and customer's labeling of the
banker's DVD. Figure 5-1, Figure 5-2, Figure 5-3 and Figure 5-4 show one data sample with these
four labeling results. Figure 5-1 is the banker's labeling of his own DVD. The first line starts with the
time when the banker meets and greets the customer and the last line is the time when the banker
finishes the conversation and the customer leaves the room. In this sample, the banker labeled that he
was smiling when he was greeting to meet the customer and recapitulating his explanation session.
The first line and the last line have to be the same in all four files, which means their interaction time
is equal. In Figure 5-2, Figure 5-3 and Figure 5-4, the time of the first line is the same as the one in
Figure 5-1, yet the label is "Missing Label", which means the coder did not start labeling at that point.
In addition, the coders annotated the reason why they were making a certain facial expression at that
moment. We make use of these comments to understand the intent and context of the facial
expressions and classify smiles into three categories in chapter eight. By comparing the banker's
labeling in Figure 5-1 with the customer's labeling of banker's DVD in Figure 5-2, we notice that
there exists some agreement in smile labeling and there exists some disagreements in "Neutral" and
"Caring" facial expressions. In the following section, we calculate the agreement between banker's
labeling and customer's labeling for each facial expression.
........
..............
Figure 5-1. Banker's Labeling of Banker's DVD
Figure 5-2. Customer's Labeling of Banker's DVD
Figure 5-3. Customer's Labeling of Customer's DVD
Figure 5-4. Banker's Labeling of Customer's DVD
5.2 Agreement between First and Second Perspective
Labeling
We have used three measurements to compare first and second perspective labeling. In all three
measurements, the numerator is the length of the time during which the banker's labeling and the
customer labeling agree. The difference between these three measurements is that measurement 1
compares the agreement between the banker and the customer with banker's labeling, measurement 2
compares it with customer's labeling, and measurement 3 compares it with the length of the
interaction. To explain these three measurements specifically, let's take an example with the data
shown in Figure 5-1, Figure 5-2, Figure 5-3 and Figure 5-4 of section 5.1 above.
Banker's Labeling
6:46:42 Smile
6:46:47 Smile
6:47:11 Smile
6:47:25 Smile
6:47:32 Smile
6:47:52 Smile
6:48:03 Smile
6:48:26 Smile
6:49:15 Smile
6:50:19 Smile
6:50:33 Smile
6:51:23 Smile
6:51:30 Smile
6:51:59 Smile
6:52:05 End of the interaction
Duration
Customer's Labeling
5 seconds
24 seconds
14 seconds .
7 seconds
20 seconds
11 seconds
23 seconds
49 seconds
1minutes 4 seconds
14 seconds
50 seconds
7 seconds
29 seconds
6 seconds
6:46:42 Missing Label
6:46:44 Smile
6:46:55 Smile
6:47:07 Caring
6:47:22 Smile
6:47:27 Smile
6:47:44 Neutral
6:47:52 Concerned
6:47:57 Smile
6:48:24 Smile
6:48:34 Concerned
6:48:40 Smile
6:48:46 Neutral
6:48:49 Smile
6:48:55 Concerned
6:49:15 Smile
6:49:23 Concerned
6:49:29 Smile
6:49:44 Caring
6:50:17 Smile
6:50:24 Caring
6:50:34 Smile
6:50:40 Caring
6:51:08 Smile
6:51:12 Caring
6:51:25 Smile
6:51:34 Caring
6:52:00 Smile
6:52:05 End of interaction
Duration
2 seconds
1 seconds
12 seconds
15 seconds
5 seconds
17 seconds
8 seconds
5 seconds
27 seconds
10 seconds
6 seconds
6 seconds
3 seconds
6 seconds
20 seconds
8 seconds
6 seconds
15 seconds
33 seconds
7 seconds
10 seconds
6 seconds
28 seconds
4 seconds
13 seconds
9 seconds
26 seconds
5 seconds
Figure 5-5. Banker's Labeling of Banker's DVD and Customer's Labeling of Banker's DVD
Figure 5-5 compares the banker's labeling of the banker's DVD, which is the same as Figure 5-1,
with the customer's labeling of the banker's DVD, which is the same as Figure 5-2. As we can see on
the left side of Figure 5-5, according to the banker's self labeling the banker was smiling during the
entire interaction time, 5minutes 23 seconds. Meanwhile, according to the customer's labeling the
banker was smiling only 2 minutes and 23 seconds, which is the summation of all of the smiling
segments(11 + 12 + 5 + 17 + 27 + 10 + 6 + 6 + 8 + 15 + 7 + 6 + 4 + 9 seconds = 143 seconds = 2
minutes and 23 seconds). In this case, the value of measurement 1 of smiles is 0.443, which can be
calculated like the following:
The duration of banker and customer labeling agreement of smile
The duration of banker's labeling of smile
-
2 minutes 23 seconds
5minutes 23 seconds
143seconds
323seconds
-=
0.443
Therefore, measurement 1 is the ratio between the banker and customer labeling agreement and
banker's labeling. Measurement 2 can be calculated similar to measurement 1 except that the
denominator is the duration of the customer's labeling of smile this time. Measurement 2 can be
calculated like the following:
The duration of banker and customer labeling agreement of smile
The duration of customer's labeling of smile
2 minutes 23 seconds
2minutes 23 seconds
143seconds
143seconds
Measurement 3 is used to calculate how much ratio the banker and customer labeling agreement of
smile takes up during the entire interaction time.
The duration of banker and customer labeling agreement of smile_ 2 minutes 23 seconds_ 143seconds
5minutes 23 seconds 323seconds
The Interaction Time
Table 5-1 shows these three measurements for all the facial expressions IDs, from ID 1, smile, to ID
19, missing label. The values listed here are the average of forty-six data of banker2 in the
experiment. For example, if we calculate measurement 1 for facial expressions ID 2= concerned =
with the data provided in Figure 5-5, the value was ignored when the average was calculated since
the duration of the banker's labelingof concerned is zero in this data. Table 5-1 shows the
comparison between the banker's self labeling of the banker's DVD and the customer's labeling of
the banker's DVD, which means the banker's labeling is the first person perspective labeling and the
customer's labeling is the second person perspective labeling. As we can see from Table 5-1, "Facial
Expressions ID 1: Smile" made a fairly high ratio of agreement between first person and second
person perspectives labeling, which takes up 45.2% among the banker's smile labeling and 43.4%
among the customer's labeling. "Facial Expressions ID 7: Neutral" was also relatively higher than
others, which was 43.3% among the banker's labeling and 67.6% of the customer's labeling.
Measurement 2.
Measurement 3.
Banker and Customer agreement
Banker and Customer agreement
Banker's Labeling
Customer's Labeling
Interaction Time
1 : Smile
0.452
0.434
0.152
2 : Concerned
0
0
0
3 : Caring
0.222
0.011
0.003
4: Confused
0
0
0
5 : Upset
Not Applicable
0
0
6: Sorry
0
0
0
7 : Neutral
0.433
0.676
0.282
8 : Other
0
0
0
9 : Satisfied
Not Applicable
0
0
10 : Surprised
Not Applicable
0
0
11 : Relieved
Not Applicable
0
0
12 : Interested
Not Applicable
0
0
13 : Bored
Not Applicable
0
0
14 : Enthusiastic
Not Applicable
0
0
15 : Persuasive, Convincing
Not Applicable
0
0
Measurement
Measurementl.
Banker and Customer agreement
Facial Expressions ID
16 : Annoyed
Not Applicable
0
0
17 : Survey
Not Applicable
Not Applicable
Not Applicable
18 : The end of interaction
0
0
0
19 : Missing Label
0
0
0
Table 5-1. Comparison between banker self labeling and the customer's labeling of banker's
DVD for each facial expression ID; banker's self labeling is the first person perspective labeling
and customer's labeling is the second person perspective labeling
Table 5-2 shows the same three measurements for the customer's self labeling of the customer's
DVD and the banker's labeling of customer's DVD. Since the customer's labeling is the first person
perspective labeling and the banker's labeling is the second person perspective labeling in this case,
the denominator of measurement 1 is the duration of the customer's labeling of each facial
expression and the denominator of measurement 2 is the duration of the banker's labeling of each
facial expression. As is shown in Table 5-2, "Facial Expressions ID 7: Neutral" had the highest
agreement, which agreed with 89.9% of customer's self labeling and 52.7% of the banker's labeling
of customer's facial expressions.
Measurement
Measurement
2.
Measurement
3.
Banker and Customer agreement
Banker and Customer agreement
Banker and Customer agreement
Customer's Labeling
Banker's Labeling
Interaction Time
Measurementl.
Facial Expressions ID
1: Smile
0.311
0.510
0.038
2 : Concerned
0.014
0.199
0.003
3 :Caring
0
0
0
4 Confused
0.033
0.172
0.001
5 : Upset
0
0
0
6: Sorry
0
0
0
7 : Neutral
0.899
0.527
0.410
8 : Other
0.064
0.286
0.001
9 : Satisfied
Not Applicable
Not Applicable
Not Applicable
10 : Surprised
0
0
0
11 :Relieved
0
0
0
12 : Interested
0
0
0
13 : Bored
0
0
0
14 : Enthusiastic
0
0
0
15 : Persuasive, Convincing
0
0
0
16: Annoyed
0
0
0
17 : Survey
0.8592
0.765
0.098
18 : The end of interaction
0
0
0
19: Missing Label
0.3366
0.643
0.005
Table 5-2. Comparison between customer self labeling and the banker's labeling of customer's
DVD for each facial expression ID; the customer's self labeling is the first person perspective
labeling and the banker's labeling is the second person perspective labeling
"Facial Expressions ID 1: Smile" also showed more than 50% agreement in the banker's labeling and
31.1% of customer's labeling. We can conclude from the results in Table 5-1 and Table 5-2, smiles
and neutral facial expressions were quite obviously recognizable between different coders, while
caring, concerned, and confused were less obvious in comparison. More interestingly, other facial
expressions except for these ones i.e., smile, neutral, caring, concerned and confused, were very
dependent on the coder's perspective. Therefore, to analyze and understand dyadic communication
not only through shared smiles but also with other complementary facial expressions pairs such as
"Smile-Satisfied", "Caring-Upset", and "Caring-Bored", we adopt both the first person perspective
labeling and the second person perspective labeling in chapters six and seven.
Chapter 6
Customer Satisfaction Data Analyses
6.1 Hypotheses
H1. Customers are more satisfied with the interaction and the information when a customer
feels the service provider is empathetic or always smiling.
H2. There are three distinctive and valuable ways to investigate customer satisfaction and
these are what the service provider is trying to do, what the customer perceives and what actually
happens in their interaction.
:~""""""""""""""""~
The remainder of this chapter analyzes the customer satisfaction data according to
cognitively reported customer perception, banker manipulation and affectively measured
synchronized facial expressions.
6.2 Analysis 1. Cognitively Reported Customer Perception
Interaction satisfaction and information satisfaction were highest in the "Empathetic" group
and SERVQUAL empathy score of a service provider measured with surveys was highest in the
"Always Smiling" group, yet these two groups showed similar mean of ratings in all of these three
measurements (Figure 6-1). An ANOVA was run on each of Interaction Satisfaction, Information
Satisfaction and SERVQUAL empathy score. In each ANOVA, "Interaction Satisfaction" (F =
12.855, p = 0.000042) and "Information Satisfaction" (F = 7.036, p = 0.0023) and "Empathy
Score"(F = 8.672, p = 0.0007) were significantly lower in the "Neutral" group than "Empathetic" and
"Always Smiling" group.
I Neutral
Always Smiling U Empathetic
Neutral
Interaction : Mean =6.083, SD = 1.165
Information : Mean = 6.667, SD = 1.371
Empathy : Mean = 4.958, SD = 1.339
Always Smiling
Interaction : Mean = 7.737, SD = 1.147
Information : Mean =7.947, SO = 1.079
Empathy : Mean = 6.868, SD = 1.311
Empathetic
Interaction : Mean 8.067, SD = 0.884
Information : Mean = 8.067, SD = 0.704
Empathy : Mean = 6.7833, SD = 1.379
ANOVA
Interaction :F = 12855, p = 4.2E-05
Information : F = 7.036, p = 0.0023
Empathy : F= 8.672, p = 0.0007
Interacton Satisfacton
EmpathyScore
Information Satisfaction
Figure 6-1. Interaction Satisfaction, Information Satisfaction and Empathy Score
56
............
........
r.
6.3 Analysis 2. Banker Manipulation
While there are obvious differences among the three measures, Interaction Satisfaction,
Information Satisfaction, and SERVQUAL Empathy Score, when the data is classified according to
customer perception, these measures are similar when we classify the data according to the banker's
manipulation. As it is shown in Figure 6-2, Interaction Satisfaction, Information Satisfaction and
Empathy Score scored highest in "Always Smiling" group (when the banker tried to always smile)
and scored lowest in "Neutral" group(when the banker appeared neutral). However, all of the Pvalues, p(Interaction, Information, Empathy Score) = (0.126, 0.287, 0.424), are higher than 0.05 in
ANOVA, which indicates that the difference is not statistically significant.
MNeutral
1s
* Akamys Smiling
|
Cnmprenwntary
Firial Fressinni
Netral
Interactior : Mean = 7, SD = 1254
Inbrmetioi : Mear = 7.267, SD= 1.163
Epathy : Mean = 6.05, SD= 1.203
-
Always Sm ling
Interactior : Mean = 7.938, SD= 0.998
5Infortio : Mea = 7.938, SD = 1.181
4 lpaty : Mean =b.1, W = 1./ZZ
Interactior : Mean = 7.267, SD = 1.580
nfrrrtio : Memn= 7.733, SD= 1.223
eIpathy : Mean = 6.2, SD= 1678
2
1
Interacion satlstactlOu
Information atisfaction
Epathyscore
ANOVA
Interactior : F= 2.178, p = 0.126
hnr rmtlon : F= 1.284, p = 0287
Lmpathy :1 = U.8 /b, p = 0.424
Figure 6-2. Interaction Satisfaction, Information Satisfaction, Empathy Score
6.4 Analysis 3. Affectively Measured Synchronized Facial
Expressions
We classified the data into three groups by directly measuring the synchronized facial expressions: If
there were more than 10% of shared smiles in the interaction, then the data was placed into the
"Empathetic" group. We also classified not only smile pairs but also other complementary facial
expressions pairs in which the banker was expressing "caring" and the customer was "upset", the
banker was smiling while the customer was satisfied or interested as part of the "Empathetic" group.
With regard to the "Banker : caring - Customer : upset" pair, "Banker : Smile - Customer : Satisfied"
pair, and "Banker : Smile - Customer : Interested" pair, if the percent of one of these pairs is more
than 0%; then the data was classified as part of the "Empathetic" category. These criteria were
chosen by looking at the quantitatively measured percentage of each synchrony pair and comparing
this measurement among the data set. As we can see from Figure 6-6, there is a distinct boundary at
10% in "Banker: smile - Customer: smile" synchrony, which means the data with 10% or more
shared smiles have a relatively large percentage of shared smiles compared to other data in this data
set. If there were fewer than 10% of shared smiles throughout the interaction and more than 40% of
the time the banker smiled, then the data were classified into the "Always Smiling" group. There are
more rules to classify the data into three groups: neutral, always smiling and empathetic. These rules
are explained in Figure 6-3 below.
.
An interaction ->
data
Empathetic
no
no
yes
> Neutral
Ino
no
Always
Smiling
Always Smiling
Figure 6-3. Classification Rules
Figures 6-4 to 6-6 plot the distribution of each synchrony. In these figures, the x-axis corresponds to
the 361 facial expressions synchrony IDs listed in Appendix A and the y-axis indicates the
percentage of each synchrony in each interaction. Therefore, Figure 6-5 shows the distribution of 361
facial expressions synchrony IDs of the forty-six pairs of interactions of Banker 2. The decision
boundaries in the classification rules above (Figure 6-3) were chosen by hand based on the
distribution of these facial expressions synchronies. We can see a lot of data at the range from
Synchrony ID 1 to Synchrony ID 50 and at the range from Synchrony ID 100 to Synchrony ID 150 in
Figure 6-4. To observe this region in detail, the range from Synchrony ID 1 to Synchrony ID 19 is
plotted in Figure 6-5. Synchrony ID 1 is the smile pair i.e., Banker: Smile - Customer: Smile and it is
less than 40% in general. Therefore, we scaled down the y-axis to take a look at the data closely,
which is redrawn in Figure 6-6. The red line in Figure 6-6 indicates there is a gap between the data
.. .......
above the red line and the data below the red line, which means the red line could possibly be a
boundary to divide the data into two groups. That is how the first classification rule is taken in Figure
6-3. There is another red line on Synchrony ID 12 (Banker : Smile - Customer : Interested) in Figure
6-3. This synchrony is chosen to classify the data into the empathetic group if the data does not meet
the criteria to belong to the always smiling group, which has more than 40% of banker's smiles
without customer's shared smiles. Synchrony ID 12 can also be the banker's one-sided smile but it is
different from others in such that this synchrony is a positive one whereas others such as Synchrony
ID 2 (Banker : Smile - Customer : Concerned), Synchrony ID 4 (Banker : Smile - Customer :
Confused) or Synchrony ID 5 (Banker : Smile - Customer : Upset) can be considered as a negative
pair and a lack of empathy. There is one more positive pair which is the not shared smile pair and it
is Synchrony 9 (Banker: smile - Customer - satisfied). However, as is shown in Figure 6-6, this pair
was not found among the dyadic data so that criterion was not chosen to select the empathetic group.
There were rarely found data between Synchrony ID 20 (Banker: concerned - Customer : smile) and
Synchrony ID 38 (Banker: concerned - Customer : missing label) having only two points (Figure 6-7).
-------- -------- -------
1 ------
0.6
0.8
,
0
-------- --------
--------
6
-
----
0.4-
--
- - -
- - --
-
- ------ -------
------ --- --- -I- -
-
I------
-I---
- - - - ----
0'
---
0.2
0.8
------
6-4.
Figure
I
0 2
I-
---
--
150
--
-350
---
300
ID
Synchrony
Expressions
Facial-- Expressions
I----- ID ----------r------Synchrony
-------
Facial
I---I-
L-
-"-------
*
I
T
0.6
o
-
--
-
--
I,
I
III
I
I
-----
------- ---------------------
-----
0.4
250
200
T-T----------------------------------------- -------I
;
---
100
Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 361
t-----
--
,x0.8
r
-
c-
1
0.2
-I
--
i
........ I__
I
0
0
2
4
6
8
10
12
14
16
18
20
Facial Expressions Synchrony ID
Figure 6-5. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 19 (Y-axis: from 0 to 1)
0.2
----
I
I
I
0.18
-----
-----------
0.16 -- 1- ----II-II
I --
....
0.14
.
-I--------II
iI
II
II
I
I
I
---
--------;--- -----1-- ---- ----------
0.1
I
-------
-
I
0.08
i
--
-
- -
----
--------
I
I
0.06
--
I
I
i
----
-t----- --------
----
0.12
--
-
,
--- ----
,
I
I~3
_ _ __
--- -
I
II
I1
I
I
0.04
---r
0.02
0
2
0
4
6
8
10
14
12
16
20
18
Facial Expressions Synchrony ID
Figure 6-6. Distribution of Facial Expressions Synchrony IDs from ID 1 to ID 19 (Y-axis: from 0 to 0.2)
-
0.9 -- ----
0.8 --
-------------
0.7
--
0.
-
----
i--
- - - - -
- - - - -
- --
0.6 - -------- ----- --0
Figure
0.5
-
0.2
-----
6-7.
-- - -
Distribution
-- - - - - T - - - - -
- - - - - - - - - -
----
- - --
---- -
- - - -
Facial
Expressions
Facial
Expressions
-- - - - - - - - - -
--
I--
---
----
-
-
- -
- - - -
r -- -- -- - - -I --- - - -- ---
----- --- --
62
--
---------- --- -
--------- ----- ------
------
--------
-----
----
-----
- - ---t -- -- -- --
- -t
of
___---
--------
---
-----------
Synchrony
ID
Synchrony
ID
from
ID
20
to
ID
38
--
-- --
0.9
-
0.
- -
- -
-
- -
-
-
- -- L
- - -
- - I
- -
-
-
- -
-
0.5-------
0
0.3
- ------------0.7 -------------------------------
-
----
--
4-
- -
- - I
-
I
-I I
1--- --.-
...
---------------------. - -
-,
------------
---
-----
0.1
- -
- -I - -
-
0
-----
I-----r-----------------------
I
I
I
SI
0.5
40
38
------
42
-----
-----
44
48
46
52
50
-- ---
------------------------
---------
-----------------r--------- - - - - - - -----
- - - - -- T - - - - -t ------ - - - r----00.9
.8- ------
T
- - -
-
.8
--T- -- --
---------------0.3
------------- --- - 0.4
0.1--
L --------------------------
0.1 ------------ni
0
96
--
-,
- - - -- - - - - - - - -- --- --- -- - -- -- - - ---- - --------------
o.4-----------+
0.1
0.2
- -
- - -- --
-'-
L ----
----I ----I ----- +---------------
- i -----
i
--------
------
----
--
- -
- - -
- - - -
63
0.8----r-----------------S
j
- - - -
-
58
56
54
-----
i
-I...
-
...
- ------ -i ----------------------
-----------i
I1
'
-----
--
*---------
+
98
100
102
106
104
108
110
112
114
Facial Expressions Synchrony ID
Figure 6-98. Distribution of Facial Expressions Synchrony IDs from ID 97 to ID 115
63
IIIIIIIIIIIIIIII
1---------------0.9 -----
------ --------- ---------------- -----
I
I
--
-- -- ----
II
I
- -
- IS0.
-
- - - -
I
--
0.8
-
0.2--
0.3
0
I
- - - - --i
I
I
0
-
I
t------
I
--
I
I
- - - - -
----
I
-L------
---
- -- ---
+
T------I--~
2
- - - -
--- J -----
------
--
-
--
----
0.6 --
I
----
---
---
0.
0.7
111-
-
4
--
----
T
6
---
r----
8
10
Facial Expressions
--
12
r----- I
14
----
T
16
----
18
I
20
Synchrony ID
Figure 6-10. Distribution of the Sum of Facial Expressions Synchrony IDs from ID 1 to ID 19
0.
-----
0.8---0.5
0)
I
--
-
0. --
0.2
I --------
--
0.1 -
114
------
-
- - I-
-
--
---
118
j
-----
I
- -I
I
-
120
I
------------------
--
---- -
_
-
I
- ---
124
122
- -----
- -
-
-----1
-
-
I
I
- ---....- ,------,
----- ---
-
-
116
-----
I
I
i
-
--
II -
------
128
I
----
, ----
-------
126
----
I
-
I
----
1
----
I
......
130
132
134
Facial Expressions SynchronyID
Figure 6-11. Distribution of the Sum of Facial Expressions Synchrony IDs from ID 115 to ID 133
S 0.4
_I_
64
----..----..-----64
,--
nk.
Synchrony ID 43 (Banker : Caring - Customer : Upset) was also chosen to place the data into the
empathetic category as it is a complementary facial expression pair. The boundary value of this
synchrony is marked as 0% in Figure 6-3. There were seldom data between facial expressions
synchrony ID 96 and ID 115 (Figure 6-4). The x-axes in Figure 6-5 and Figure 6-6 have different
meanings from the ones in Figures 5-1, 5-2, 5-3, 5-4 and 6-3. Figure 6-5 plots the sum of banker-only
smiles without the customer returning a smile. The x-axis in Figure 6-5 means the synchrony ID
takes up the most portion of the sum. For example, if the sum is 0.44 and Synchrony ID 7 takes up
0.38 and the sum of other Synchrony IDs between Synchrony ID 2 and Synchrony ID 19 accounts
for the remaining 0.06%, then Synchrony ID 7 is the x-coordinate of the data and their sum 0.44 is
the y-coordinate of the data. The x-axis and y-axis in Figure 6-6 also have the same concept except
that neutral facial expressions are counted in this case. When we classified the data into neutral,
always smiling and empathetic groups following the classification rules in Figure 6-3, the number of
data in each group was quite evenly distributed, assigning fifteen to neutral, sixteen to empathetic
and fifteen to the always smiling group. Figure 6-12 shows the customer satisfaction analysis results
when the data was classified along with these affectively observed rules. The major difference
between Analysis 1 and Analysis 3 is that the Empathy Score is the highest in "Empathetic" group in
Analysis 3 i.e., affectively measured classification, whereas the Empathy Score is the highest in
"Always Smiling" group in Analysis 1 i.e., cognitively reported classification. In addition, in terms
of interaction satisfaction, the mean of the ratings of "Always Smiling" was close to the mean of the
"Empathetic" group in Analysis 1 yet, the mean of the ratings of "Always Smiling" was close to the
mean of the "Neutral" group in Analysis 3. Meanwhile, the "Empathetic" group scored highest in
"Information Satisfaction" in both analyses. However, in Analysis 3 the significance is not enough to
make strong differences between the three groups, which have p-values larger than 0.05 in ANOVA.
As is the result in this section, since there is a difference between the cognitively reported way to
investigate customer satisfaction and the affectively observed way, we need to analyze the customer
65
......
satisfaction data with both methods. Moreover, there might be more than one way to classify the data
according to the affectively measured method depending on how we describe the criteria. For
example, Synchrony ID 42 (Banker : Caring- Customer : Confused) could have been another
criterion to classify the data into empathetic group. Therefore more ways to select those criteria need
to be explored.
lO
SNeutral
I Always Smiling
MEmpathetic
9Neutral
Interactio : Mean = 7.2, SD = 1.082
Infomrnian : Mean =7533, SD= 0.990
7Empathy : Mean = 6217, SD = 1.168
8
Always Smlng
Interaction : Mean = 7188, SD= 1.797
Infortin : Mean = 7.6, SD= 1.549
Empathy: Mean= 6.183, SD= 2.145
6
5
4
"EmpathetI
: Mean = 7867,D = 0.834
: Mean
Maformwtm
= 7813, SD= 1.047
Empathy: Mean= 6.609, SD = 1242
Interacti
3
2
ANOVA
nteructio- :F = 1.321, p= 0278
Irnatn :F = 0.224, p = 0.800
pathy : F =0354, p = 0.704
1
-
0
Interadon Satisfaction
Information Satisfaction
EmpathyScore
Figure 6-12. Interaction Satisfaction, Information Satisfaction and Empathy Score
6.5 Summary
From the result in Analysis 1, we could confirm hypothesis 1 that claims the customer
satisfaction is the best when the customer think the service provider is empathetic. This is a typical
and traditional way to investigate customer satisfaction. We suggested two more ways to analyze
customer satisfaction and these are to classify the data according to what the service provider tried to
convey and what actually happened in their nonverbal communication. When the data were classified
according to the banker's manipulation the customer satisfaction scored highest in the "Always
Smiling" manipulation in Analysis 2 and when the data were classified according to their detected
66
facial expressions in Analysis 3 the customer satisfaction was the highest in the empathetic group,
while in the other two groups i.e., Neutral and Always Smiling groups, the ratings were similar. This
result agrees with the result mentioned in Chapter 4 that people's intent and perception is subjective.
Therefore, these three methods are valuable, and should be furthur investigated and kept individually
to complement each other.
Chapter 7
Dyadic Communication Data Analyses
7.1 Hypotheses
H3. The banker and the customer share smiles for a longer period of time when the customer
perceives the banker is empathetic.
H4. The banker and the customer share smiles more often when the customer perceives the
banker is empathetic.
As we classified the data into neutral, always smiling and empathetic category according to
three different criteria i.e., cognitively reported customer perception, banker manipulation and
affectively measured facial expressions synchrony in chapter six, we analyze the dyadic
communication with the same way in the following section 7.2, 7.3 and 7.4.
7.2 Analysis 1. Cognitively Reported Customer Perception
We collected forty-six sets of interaction data and in each pair the customer appraised
whether the banker's attitude was neutral, always smiling or empathetic. This section examines when
the data are grouped into these three categories based on using the customer's perceptions of the
banker. Figure 7-1 illustrates the ten kinds of facial expression synchrony between the banker and the
customer that took up the longest duration among 361synchrony pairs in each group. The x-axis
indicates the synchrony ID and the y-axis indicates the fraction of the synchronized facial
expressions. The fraction of each synchrony is computed across the participants in each condition. In
the perceived as "Neutral" condition, the "neutral-caring" pair (ID = 117) occurred most of the time,
taking up 45.1% of the entire interaction time on average. In the perceived as "Always Smiling"
condition, we found more than 50% of one-sided smiles from the banker i.e., 30.8% of the pair
"smile-neutral" (ID = 7), 12.6% of the pair "smile-other" (ID = 8), 11.9% of the pair " smileconcerned" (ID = 2) and 11.7% of the "smile-surprised" (ID = 10). In these banker-only smiles, the
customer was showing neutral facial expressions rather than smiling, which may be inferred as the
customer not enjoying the interaction. The most frequently occurring synchrony pairs are shown in
Figure 7-2. The x-axis indicates the synchrony ID and the y-axis indicates the number of the
synchronized facial expressions. In the perceived as "Neutral" group, the banker was expressing
neutral facial expressions most of the time, regardless of the customer's facial expressions. And
"neutral-neutral" pair (ID = 121) was the most frequent pair. In "Empathetic", the "neutral-neutral"
pair (ID = 121) occurred 5.6 times on average, which was higher than the value of this pair in the
"Neutral" group, which wa 5. "Smile-smile" pair (ID = 1) was also much more frequent, scoring 6
compared to the other two groups ("Neutral" = 2.8,"Always Smiling" = 3.5). This result implies that
the "Empathetic" group is more dynamic than the other two groups. Therefore, we can conclude that
when the customer perceived the banker as empathetic, they smiled together genuinely more often
and for longer.
Neutral (N=12)
60
ID : Banker - Customer
117: neutral - caring
121: neutral- neutral
7 : smile - neutral
8 : smile - other
118 : neutral - confused
122 : neutral - other
124: neutral - surprised
115: neutral - smile
123 : neutral - satisfied
2 : smile - concerned
50
40
30
20
10
117
121
7
8
118
122
124
115
123
2
Synchronized facial expressions ID
Always Smile (N=19)
(%) 60
ID : Banker - Customer
121 : neutral - neutral
7 : smile- neutral
116 : neutral - concerned
127 : neutral - bored
8 : smile - other
2: smile - concerned
126 : neutral - interested
45 : caring - neutral
10 : smile - surprised
122: neutral - other
50
40
30
20
10
0
121
7
116
127
8
2
126
45
10
122
Synchronized facial expressions ID
Empathetic (N= 15)
(%) 60
ID : Banker - Customer
121: neutral - neutral
7 : smile- neutral
102 : sorry - neutral
8 : smile - other
1 : smile - smile
122 : neutral- other
10 : smile - surprised
115: neutral- smile
4: smile - confused
119: neutral - upset
50
40
30
20
10
0
121
7
102
8
1
122
10
115
4
119
Synchronized facial expressions ID
Figure 7-1. Ten Synchronized Facial Expressions that take up the largest percentage in the
entire interaction in each group
--
Neutral (N=12)
7
ID : Banker - Customer
121: neutral - neutral
115: neutral - smile
124: neutral - surprised
1 : smile - smile
117 :neutral -caring
118 : neutral - confused
7 : smile - neutral
122: neutral - other
4 : smile - confused
8 : smile- other
6
5
4
3
2
1
0
121
115
124
1
117
118
7
122
4
8
Synchronized facial expressions ID
Always Smile (N=19)
7
ID : Banker - Customer
7 : smile- neutral
117: neutral - caring
121: neutral - neutral
116: neutral - concerned
I : smile - smile
neutral - smile
127 :neutral - bored
122 : neutral - other
2 : smile- concerned
45 : caring- neutral
6
5
4
3 -115:
2
1
0
7
117
121
116
1
115
127
122
2
45
Synchronized facial expressions ID
Empathetic (N= 15)
7
ID : Banker - Customer
7 : smile- neutral
1 : smile - smile
121 : neutral - neutral
102 : sorry - neutral
6
5
4
115:neutral - smile
3
2
122 : neutral - other
8 : smile - other
96 : sorry - smile
116: neutral - concerned
2 : smile - concerned
1
0
7
1
121
102
115
122
8
96
116
2
Synchronized facial expressions ID
Figure 7-2. Ten most frequent synchronized facial expressions in each group
7.3 Analysis 2. Banker Manipulation
This section examines when the data are grouped into these three conditions based on using
the banker's manipulation and what he was trying to convey.
Figure 7-3 illustrates the ten kinds of facial expression synchrony between the banker and the
customer that took up the longest duration among 361synchrony pairs in each group. The x-axis
indicates the synchrony ID and the y-axis indicates the fraction of the synchronized facial
expressions. The fraction of each synchrony is computed across the participants in each condition. In
the "Neutral" condition, all of the ten longest synchronies were paired with the banker's neutral facial
expressions. The pair "smile-smile" (ID = 1) happened longer in the "Always Smiling" condition
taking up 16.8% than in "Empathetic" condition, 8.2%. The shared smile pair of "Always Smiling"
condition was observed nearly twice as long as that of "Empathetic" condition. This result can be
accounted for with the facts found in chapter four that explain the difference between what the
banker was trying to do and what the customer was responding with and thinking. In the "Always
Smiling" group, there are more than 50% "smile-neutral" pairs along with other banker-only smile
pairs such as "smile-other"(ID=8), "smile-concerned"(ID=2), "smile-confused"(ID=4), and "smileupset"(ID=5). In Figure 7-4, "Empathetic" group shows more diversity in banker's facial
expressions including neutral, smile and sorry. The other two groups do not have "sorry" in banker's
facial expressions, which implies the banker is trying to respond to customer's concerns. Moreover,
the values of the pairs "neutral-neutral"(ID=121), "smile-neutral"(ID=7) and "smile-smile"(ID=1)
seem to be similar between groups; the pair "neutral-neutral"(ID=121) in the "Empathetic" group is
5.2 and in the "Neutral" group is 4.9; the pair "smile-neutral"(ID=7) is 5.4 in the "Always Smiling"
group and 4.9 in the "Empathetic" group; the pair "smile-neutral"(ID=1) is 4.7 in the "Always
Smiling" group and 4.5 in the "Empathetic" group.
Neutral (N=15)
(%) 60
ID : Banker - Customer
121: neutral - neutral
116 : neutral - concerned
117: neutral - caring
127 : neutral - bored
118 : neutral - confused
126 : neutral - interested
122 : neutral - other
124: neutral - surprised
123 : neutral - satisfied
115: neutral- smile
50
40
30
20
10
121
116
117
118
127
126
122
124
123
115
Synchronized facial expressions ID
Always Smile (N=16)
)60
ID : Banker - Customer
7 : smile - neutral
10 : smile - surprised
8 : smile - other
1 : smile - smile
2 : smile - concerned
4: smile - confused
116 : neutral - concerned
121 : neutral - neutral
115: neutral- smile
5 : smile - upset
50
40
30
20
10
7
10
8
1
2
4
116
121
115
5
Synchronized facial expressions ID
Empathetic (N=15)
(%) 60
ID : Banker - Customer
121 : neutral - neutral
102 : sorry - neutral
122 : neutral - other
116 : neutral - concerned
8 : smile - other
115 : neutral - smile
45 : caring - neutral
I : smile - smile
7: smile - neutral
10: smile- surprised
50
40
30
20
10
0
121
102
122
116
8
115
45
7
10
Synchronized facial expressions ID
Figure 7-3. Ten Synchronized Facial Expressions that take up the largest percentage of the
entire interaction in each group
Neutral (N=15)
7
ID : Banker - Customer
121: neutral - neutral
115 : neutral - smile
117 : neutral - caring
127 : neutral - bored
116: neutral - concerned
2: smile - concerned
122 : neutral - other
124 : neutral - surprised
118 : neutral - confused
119: neutral - upset
6
5
4
3
2
0
121
115
117
127
116
2
122
124
118
119
Synchronized facial expressions ID
Always Smile (N= 16)
7
ID : Banker - Customer
7: smile - neutral
1 : smile - smile
8 : smile - other
121 : neutral - neutral
2: smile - concerned
4: smile - confused
5 : smile - upset
3 : smile - caring
115 : neutral - smile
116: neutral - concerned
6
5
4
3 2
0
7
1
8
121
2
4
5
3
115
116
Synchronized facial expressions ID
Empathetic (N= 15)
7
ID : Banker- Customer
121: neutral - neutral
102: sorry - neutral
7 : smile - neutral
1 : smile - smile
115 : neutral - smile
122 : neutral - other
116 : neutral - concerned
96 : sorry - smile
2 : smile - concerned
118 : neutral - confused
6
5
4
3
2
0
121
102
7
1
115
122
116
96
2
118
Synchronized facial expressions ID
Figure 7-4. Ten most frequent synchronized facial expressions in each group
7.4 Analysis 3. Affectively Measured Synchronized Facial
Expressions
This section examines when the data are grouped into these three conditions based on what
was really observed in their dyadic communication through facial expressions. The classification
rules are described in section 6.4.
Figure 7-5 illustrates the ten kinds of facial expression synchrony between the banker and the
customer that took up the longest duration among 361 synchrony pairs in each group. The x-axis
indicates the synchrony ID and the y-axis indicates the fraction of the synchronized facial
expressions. The fraction of each synchrony is computed across the participants in each condition. In
the "Neutral" condition, except for the "smile-surprised" pair (ID = 10), most of the pairs are
banker's neutral facial expressions. In the "Empathetic" group, the smile pair (ID =1) at 20.2%
appeared the longest compared to the other two groups; there is no shared smile in "the Neutral"
group or in the "Always Smiling" group. In the "Always Smiling" group there is more than 50% of
banker-only smiles, combining 29.0% of "smile-neutral"(ID=7) and 14.0% of "smile-other"(ID=8).
This value is smaller than the total of banker-only smiles in the "Empathetic" group, adding up 27.5%
of "smile-neutral"(ID=7), 18.8% of "smile-other"(ID=8), 13.9% of "smile-surprised"(ID=10), 12.6%
of "smile-concerned"(ID=2) and 8.5% of "smile-confused"(ID=4), yet there is no shared smile that
appears among the ten longest pairs in "Always Smiling". The pair "smile-smile"(ID = 1) is most
frequent in the "Empathetic" group compared to other two groups (Figure 7-6).
slr~
Neutral (N=15)
(%) 60
ID : Banker - Customer
121: neutral - neutral
116 : neutral - concerned
127 : neutral - bored
117: neutral - caring
122 : neutral - other
118: neutral - confused
126 : neutral - interested
10 : smile - surprised
123 : neutral - satisfied
115: neutral- smile
50
40
30
20
0
121
116
127
122
117
118
126
10
123
115
Synchronized facial expressions ID
Always Smile (N=15)
(%) 6o
ID : Banker -Customer
121: neutral - neutral
7: smile - neutral
102: sorry - neutral
8 : smile - other
122: neutral - other
116: neutral - concerned
124: neutral- surprised
115: neutral -smile
118: neutral- confused
45: caring -neutral
50
40
30
I0
20
10
0
121
7
102
8
122
116
124
115
118
45
Synchronized facial expressions ID
Empathetic (N= 16)
(%)60
ID : Banker - Customer
7 : smile- neutral
45 : caring - neutral
1 : smile - smile
8 : smile - other
121: neutral - neutral
10 : smile - surprised
2: smile - concerned
115 : neutral - smile
4: smile- confused
122 : neutral - other
50
40
30
20
10
7
45
1
8
121
10
2
115
4
122
Synchronized facial expressions ID
Figure 7-5. Ten Synchronized Facial Expressions that take up largest percentage in the entire
interaction in each group
Neutral (N= 15)
7
ID : Banker - Customer
121: neutral- neutral
116: neutral - concerned
115 : neutral - smile
117: neutral-caring
127 : neutral - bored
122: neutral- other
2: smile-concerned
7 : smile - neutral
118: smile - confused
1 : smile - smile
6
4
3
2
1
0
121
116
115
117
127
122
2
7
118
1
Synchronized facial expressions ID
Always Smile (N= 15)
7
6
5
4
3
2
0
1
121
102
115
7
116
96
122
1
118
ID : Banker - Customer
121: neutral - neutral
102 :sorry - neutral
115: neutral - smile
7 : smile - neutral
116 : neutral - concerned
96 : sorry - smile
122 : neutral - other
1 : smile - smile
118 : neutral - confused
8 : smile - other
8
Synchronized facial expressions ID
Empathetic (N= 16)
7
ID : Banker - Customer
7: smile - neutral
1 : smile - smile
45 : caring - neutral
121: neutral- neutral
115 : neutral - smile
122 : neutral - other
8: smile - other
2 : smile - concerned
3: smile - caring
4: smile - confused
6
5
4
3
2
1
0
7
1
45
121
115
122
8
2
3
4
Synchronized facial expressions ID
Figure 7-6. Ten most frequent synchronized facial expressions in each group
7.5 Summary
We measured the percent of each facial expression synchrony pair as well as the number of
each facial expression synchrony pair. Both measures of the smile-smile pair appeared highest when
the customer perceived the banker was empathetic, which confirms the hypotheses three and four.
Note that the shared smile pairs were more distinctively found when the data were classified according
to what actually happened in the interaction compared to the analyses done by customer perception
and banker manipulation. In the next chapter, chapter eight, we suggest a more novel way to
understand smiles by their context and intent and classify them into four kinds of smiles.
Chapter 8
Specific Classification and Analysis of
Smiles
8.1 Smiles with Context
As is explained in section 5.1, the data gathered for each participant includes the time, facial
expressions and a short comment about the facial expressions. From these comments we can infer
whether the person was smiling just for greeting, laughing at a joke, satisfied with the information or
trying to be polite. Therefore we classified the smile labeling into three groups; social, genuine, and
greeting smile. Here are some examples of the comments that can be classified into these three kinds
of smile (Figure 6-4). Interestingly, there are positive reasons for genuine smiles but also negative
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;~
reasons for genuine smiles such that the customer smiled when he disagreed with the banker,
probably politely, or when he encountered an unexpected situation where did not get the full amount
of compensation for the study, probably to make the banker feel it is okay.
Figure 8-1. The examples of social, genuine, and greeting smiles
8.2 Validation of Social, Genuine and Greeting Smiles
Three undergraduate students labeled three kinds of smiles as explained in section 8.1. Three of them
sat down together until they were trained to get agreement between these three kinds of examples and
...i.
.. ...!
.... ......
... .....
....
...
......
..........................................
then they were separated and labeled the smiles of the full set of data. There are forty-six interactions
and four labeled files for each interaction. There are 46*4 = 184 labeled files in total and 60 of them
did not have any smiles labeled in them. Therefore there were 124 smile labeled files and three
coders labeled these files with social, genuine and greeting smiles. Figure 8-2 shows the inter-coder
agreement using Cohen's Kappa coefficient. Cohen's Kappa coefficient is a commonly used measure
in social science analysis to take into account the effect that human coders can show agreement by
chance when there are two coders (Cohen, 1960; Fleiss, 1971; Galton, 1892; Gwet, 2001; Landis,
1977). From this result, 67 files among 124 smile labeling files got 0.81-1.00 of Cohen's Kappa
coefficient, which means the three coders labeling was pretty much in agreement.
80
70
60
t9
50 40
K
Interpretation
No agreement
<0
0.0-0.20 Slight agreement
0.21-0.40 Fair agreement
-
-30
0.61-0.80 Substantial agreement
0.81-1.00 Almost perfect agreement
i2
U
I1
0-
<0.0
0.0-0.20
0.21-0.40
0.41-0.60
0.61-0.80
0.81-1.00
Cohen's Kappa Coefficient (K)
Figure 8-2. Cohen's Kappa Coefficient among three coders
Chapter 9
Conclusions and Future Work
9.1 Summary
"Service with a smile" (Pugh, 2001 & Grandey, 2005) is a common motto in the service
provider industries. It is assumed that a smiling employee is the best representative for customer
interaction. We examined the effects that a service provider's facial expression manipulation during
customer interaction had on customer satisfaction. We measured interaction satisfaction, information
satisfaction, and the perceived empathetic attitude of the service provider, to estimate customer
satisfaction. The results show that customer satisfaction was not significantly affected by the
banker's intent to portray a particular quality of interaction. Rather, customer satisfaction was
dependent on how the customer perceived the service provider's attitude. When the customer
perceived the service provider as empathetic, customer satisfaction was greater than the cases in
which the customer thought the service provider was neutral or always smiling. We looked more
specifically at the data when the customer felt the service provider was empathetic, and measured
that they shared smiles together more often and for a longer time. In contrast to the common notion
of "Service with a Smile", the "Always Smiling" attitude of the service provider was not effective in
making the service provider or experience more enjoyable. Based on measuring smiles, the important
aspect of smiles in service appears to be when conditions are such that both the service provider and
the customer smile, together, and genuinely. This principle also applies to other complementary
facial expressions.
9.2 Suggested Future Work
Dyadic analysis of forty-six face-to-face customer service encounters reveals that affective
social signals through facial expressions have a significant influence on how customers perceive the
service. We obtained this result using the facial expression labels provided by the experimental
participants. However, the labeling depends on the labeler's perspective. To improve the reliability of
the data labeling, we could compare the three different labelings done by the participants themselves,
their interaction partners and four independent outside coders. In addition, we classified the smiles
into three kinds of smiles; specifically it becomes four kinds when the genuine smile is divided into
positive and negative ones, in chapter eight and we suggest to analyze customer satisfaction and
dyadic communication with these labeled data.
The current state of computer vision technology suggests that computers can be trained to
recognize a number of human facial expressions automatically (el Kaliouby, 2005 & Zaman, 2006).
We visualized the dyadic pattern of facial expressions between a service provider and a customer by
assigning each facial expression to a specific color. A future application might provide real-time
feedback (e.g., Kim et al., 2008) about affective information during a service interaction. With regard
to data analysis, other measures such as the duration of each facial expression and the number of
transitions of between expressions, may be useful. In addition, for more complete understanding,
voice analysis can be combined with facial analysis to explain such topics as why a customer
perceives a service provider was smiling even when the service provider intended to be neutral.
While sharing genuine smiles appears to be very important to interaction, the way to best achieve this
sharing is more challenging than simply asking the banker to act empathetically.
Appendix A: 361 Banker and Customer
Facial Expressions Synchrony IDs
uCstomet
smile concemed caring confused
Missing
upset
sorry
neutral
other Satisfied Surprised Relieved Interested Bored Enthusiatic ersuasiv Annoyed Survey The end
ankeLabel
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
Confused 58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
Upset
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
Sorry
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
Satisfied153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
Surprised172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
Relieved 191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
Interested
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
Enthusiastic
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
Persuasive
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
Annoyed 286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
Survey
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
Theend
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
349
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
Smile
1
Concerned 20
Caring
39
Neutral 115
Other
Bored
Missing
Label
Bibliography
[1] Alford, Bruce L. and Daniel Sherrell. (1996). The Role of Affect in Consumer Satisfaction
Judgments of Credence-Based Services. Journalof Business Research, Vol. 37, 71-84.
[2] Baaren, R. (2003). Mimicry for money : Behavioral consequences of imitation. Journalof
Exprimental Social Psychology, Vol. 39, Issue 4, 393-398
[3] Bagozzi, Richard P., Mahesh Gopinath, and Prashanth U. Nyer. (1999). The Role of Emotions in
Marketing. Journalof the Academy of Marketing Science, Vol. 27, No. 2, 184-206.
[4] Barger, P. (2006). Service With a Smile and Encounter Satisfaction: Emotional Contagion and
Appraisal Mechanisms. Academy ofManagement Journal,Vol. 49, No. 6, 1229-1238
[5] Barsade, Sigal G. (2002). "The Ripple Effect: Emotional Contagion and its Influence on Group
Behavior." Administrative Science Quarterly, 47, 644-675
[6] Chartrand, T.L. & Bargh, J.A. (1999). The chameleon effect: The perception-behavior link and
social interaction. JournalofPersonality and Social Psychology, Vol. 76, 893-910.
[7] Chartrand, T.L., Maddux, W, & Lakin, J. (2005). Beyond the perception-behaviorlink: The
ubiquitous utility and motivationalmoderators ofnonconscious mimicry. (R. Hassin, J. Uleman, &
J.A. Bargh, Eds.), The New Unconscious (pp. 334-361). New York: Oxford University Press.
[8] Cohen, Jacob (1960). A coefficient of agreement for nominal scales, Educationaland
PsychologicalMeasurement Vol.20, No. 1, 37-46.
[9] Decety, J. (2004). The Functional Architecture of Human Empathy. Behavioraland Cognitive
NeuroscienceReviews, Vol. 3, No. 2, 71-100
[ 10] Decety, J. (2007). A social cognitive neuroscience model of human empathy. (E. Harmon-Jones
& P. Winkielman, Eds.), Social Neuroscience: Integrating Biological and Psychological Explanations
of Social Behavior (pp. 246-270). New York: Guilford Publications.
[11] Decety, J. & Sommerville, J.A. (2003). Shared representations between self and others: A social
cognitive neuroscience view. Trends in Cognitive Sciences, Vol. 7, 527-533.
[12] el Kaliouby, R., Robinson, P. (2005). Real-time inference of complex mental states from facial
expressions and head gestures. Real-time vision for HCI, 181-200
[13] Fleiss, J.L. (1971). "Measuring nominal scale agreement among many raters." Psychological
Bulletin, Vol. 76, No. 5, 378-382
[ 14] Fleiss, J. L. (1981) Statisticalmethods for rates andproportions. 2nd ed. (New York: John
Wiley); 38-46
[15] Fleiss, J.L. and Cohen, J. (1973) "The equivalence of weighted kappa and the intraclass
correlation coefficient as measures of reliability" in Educationaland PsychologicalMeasurement,
Vol. 33, 613-619
[16] Galton, F. (1892). FingerPrints Macmillan, London.
[17] Grandey, A.A., Fisk, G.M., Mattila, A.S., Jansen, K.J., Sideman, L.A. (2005), "Is 'service with a
smile' enough? Authenticity of positive displays during service encounters", Organizational
Behavior andHuman Decision Processes,Vol. 96 No. 1, pp. 3 8 -5 5 .
[18] Gwet, K. (2001) StatisticalTablesfor Inter-RaterAgreement. (Gaithersburg : StatAxis
Publishing)
[19] Hatfield, E., Cacioppo, J. T., & Rapson, R. L. (1994). Emotional contagion. Cambridge:
Cambridge University Press.
[20] Hochschild, A. R. (1983). The managedheart. Los Angeles: University of California Press.
88
[21] Kendon, A. (1970). Movement coordinatioon in social interactions. Acta Psychologica,Vol. 32,
101-125.
[22] Kenny, D.A., Kashy, D.A., and Cook, W.L. (2007). Dyadic DataAnalysis, Vol. 61, 263-295,
Guilford Pubn
[23] Kim, T. Chang, A. & Pentland, A. (2008). Meeting Mediator: Enhancing Group Collaboration
with Sociometric Feedback, In Proceedings of Conference on Computer Supported Collaborative
Work, 457-466
[24] Knoblich, G. & Flach, R. (2001). Predicting the effects of actions: interactions of perception and
action. PsychologicalScience, Vol. 12, 467-472.
[25] Knoblich, G., & Flach, R. (2003). Action identity: Evidence from selfrecognition,prediction, and
coordination. Consciousness and Cognition, 12, 620-632.
[26] LaFrance, M. (1982). Posture mirroring and rapport. (M. Davis, Ed.). Interactionrhythms:
Periodicityin communicative behavior (pp. 279-298). New York: Human Sciences Press.
[27] Lakin, J. L., Jefferis, V. E., Cheng, C. M., & Chartrand, T. L. (2003). The Chameleon Effect as
social glue: Evidence for the evolutionary significance of nonconscious mimicry. Journalof
Nonverbal Behavior, Vol. 27, 145-162.
[28] Landis, J.R. and Koch, G. G. (1977) "The measurement of observer agreement for categorical
data" in Biometrics. Vol. 33, 159-174
[29] Lipps, T. (1903). Grundlegung der Asthetik. Hamburg/Leipzig, Germany:Leopold Voss.
[30] Maurer, R. E., & Tindall, J. H. (1983). Effect of postural congruence on client's perception of
counselor empathy. Journalof CounselingPsychology, Vol. 30, 158-163.
[31] Oliva, Terence A., Richard L. Oliver, and Ian C. MacMillan (1992), A Catastrophe Model for
Developing Service Satisfaction Strategies, Journalof Marketing, 56 (July), 83-95.
[32] Oliver, Richard L. (1980), A Cognitive Model of the Antecedents and Consequences of
Satisfaction Decisions, Journalof Marketing Research, 42 (November), 460-69.
89
[33] Oliver, R. L. (1997). Satisfaction:A BehavioralPerspectiveon the Consumer. New York: The
McGraw Hill.
[34] Parasuraman, Zeithaml & Berry (1985). "A Conceptual Model of Service Quality and Its
Implications for Future Research," Journalof Marketing, 41-50.
[35] Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). SERVQUAL: A multiple-Item scale
for measuring consumer perceptions of service quality. Journalof Retailing,Vol. 4, 12-39.
[36] Prinz,W. (1997). Perception and action planning. EuropeanJournalof CognitivePsychology, 9,
129-154.
[37] Pugh, S. D. (2001). Service with a smile: Emotional contagion in the service encounter.
Academy of ManagementJournal,Vol. 44, 1018-1027
[38] Rafaeli, A. (1989). When clerks meet customers: A test of variables related to emotional
expression on the job. JournalofApplied Psychology, Vol. 74, 385-393.
[39] Rafaeli, A., & Sutton, R. I. (1989). The expression of emotion in organizationallife. In L. L.
Cummings and B. M. Staw (Eds.), Research in organizational behavior, vol. 11: 1-42. Academy of
Management Journal- In Press 24 Greenwich, CT: JAI Press.
[40] Rafaeli, A., & Sutton, R. I. (1990). Busy stores and demanding customers: How do they affect
the display of positive emotion? Academy ofManagementJournal,Vol. 33, 623-637.
[41] Tsai, W. (2001). Determinants and consequences of employee displayed positive emotions,
Journalof Management, Vol. 27, 497-512
[42] Tsai, W.-C., & Huang, Y.-M. (2002). Mechanisms linking employee affective delivery and
customer behavioral intentions. Journalof Applied Psychology, Vol. 87, 1001-1008.
[43] Wilk, S.L. & Moynihan, L.M. (2005). Display rule "regulators": The relationship between
supervisors and worker emotional exhaustion. JournalofApplied Psychology, Vol. 90, 918- 927.
[44] Zajonc, R. B. (1985). Emotion and facial efference: An ignored theory reclaimed. Science, 5:
15-21.
[45] Zaman, B. (2006). "The FaceReader: Measuring instant fun of use," NordiCHI2006