Using Software to Code Facial
Expressions of Emotion
Kristin Smith-Crowe | University of Utah
Jaime M. I. Potter and Sigal G. Barsade | University of Pennsylvania
Technical Assistance: Robert Botto, Programmer/Analyst, Senior IT Project Leader
2014 Academy of Management Conference | Unobtrusive Measures Professional Development Workshop
Background on Study
 We are conducting research on emotional contagion.
 We video recorded participants’ faces while they watched a
video stimulus.
 The study took place in a computer lab equipped with desktop
computers.
 Participants were recorded via a webcam.
 The video stimulus was embedded in a Qualtrics survey.
 The audio was delivered via headphones.
2014 Academy of Management Conference | Unobtrusive Measures Professional Development Workshop
Alignment
 An important issue was the alignment of the start of the video
stimulus with the start of the recording of participants.
 We needed to know exactly when the participant saw what
so that we could test hypotheses about the reactions of
participants to particular content.
 We hoped to find a way to automate the simultaneous start of
both, but we weren’t able to do so.
 Instead, we found a low-fi solution that entailed collecting the
data in such a way as to allow us to manage the alignment
post data collection.
2014 Academy of Management Conference | Unobtrusive Measures Professional Development Workshop
Alignment
 During the Data Collection
 We inserted a page in our Qualtrics survey telling participants to
raise their hands to call over an RA.
 The RA then began a screen capture program, the webcam, and
then the video stimulus.
 We used Free Screen to Video V 2.0 to create a video of the computer
screen. This allowed us to record when the video recording of the
participant began and when the video stimulus began.
 Post Data Collection
 For each participant, we watched the screen capture videos in
Aegisub to mark the times that the video stimulus began and the
video recording of the participant began.
 The precision of the timer in Ageisub allowed for greater precision in
alignment.
2014 Academy of Management Conference | Unobtrusive Measures Professional Development Workshop
Analysis of Emotion Data
 Once alignment was achieved, we used Noldus
FaceReader 5 to analyze participants’ facial expressions.
 This software can analyze live feeds or videos.
 It detects the type and intensity of seven categories of
expressions.
 Happy, sad, angry, surprised, scared, disgusted, and neutral
 Click on the pictures below to see examples of FaceReader
analyses:
2014 Academy of Management Conference | Unobtrusive Measures Professional Development Workshop
Output Files
 An example of the output appears below. FaceReader produces
one text file per participant.
 Setting FaceReader to analyze 30 frames per second produces a
lot of data. In this case, there are 720 rows of data per participant.
Note: These data are fictional.
2014 Academy of Management Conference | Unobtrusive Measures Professional Development Workshop
Output Files Converted to Excel
 Once we had all of the output files, we merged them into an Excel
file. Each row is a point in time and the numbers (e.g., Neutral1)
refer to a participant (e.g., Participant 1).
 We are using the Excel file to figure out how to aggregate the data
as we have 5,040 data points per participant.
Note: These data are fictional.
2014 Academy of Management Conference | Unobtrusive Measures Professional Development Workshop
Output Files Converted to SPSS
 Once we aggregate the data, we will create an SPSS file that will
look something like this (where T1 = time 1 and so forth). This type
of format will allow us to test our hypotheses.
Note: These data are fictional.
2014 Academy of Management Conference | Unobtrusive Measures Professional Development Workshop
Tips
 Collect race and age data.
 The general model does not work as well for East Asian people,
the elderly, and children. There are models that are particular to
these populations.
 If you have such data, you can make sure that you aren’t seeing
systematic missing data due to race and age.
 FaceReader 6 is now available.
 It features an improved East Asian model; the capacity to detect
contempt; and the capacity to analyze expressions based on the
circumplex model of affect.
2014 Academy of Management Conference | Unobtrusive Measures Professional Development Workshop
Technology Used
Screen Capture Program
Free Screen to Video V 2.0,
Koyote Software
Survey Platform
Qualtrics
Video Playback Program
Aegisub
Emotion Analysis Program
Noldus FaceReader 5
+white paper
Note: We also used Excel and SPSS.
2014 Academy of Management Conference | Unobtrusive Measures Professional Development Workshop