HSU
Access International Academy Ningbo
RESEARCH PAPER
SUBJECT: Science
TITLE: The Facial Expression
NAME: Steven Hsu
INSTRUCTOR: Mr. Arboleda
DATE: January 2014
WORD COUNT: 3277
i
HSU
Abstract
Purpose of the Research
The primary goal of this paper is to study facial expression in depth. In comparison, the
study of facial expression is a relatively new science. There are many questions in this
field; the principal question is “Can people trust facial expressions?”
Procedures Used
With the combination of Emotions Revealed by Paul Ekman, Reading Facial Expressions
of Emotion by David Matsumoto and Hyi Sung Hwang, and other resources, this paper
will guide the reader to find out answers of the principle question and other questions:
“how does the study of facial expression start?,” “how facial expression is formed?,”
“how people recognize facial expression,” and “can people trust facial expressions?”
Observation/Data/Results
In this paper, there is a section called the Experiments. This section will include
observations and data of the duplications and modification of Paul Ekman’s experiment,
which is carried out by the author of this paper. The Facial Action Coding System
(FACS) is also mentioned in this paper and it will be explained.
Conclusions
After reading this paper, the reader will be able to understand a brief history of facial
expression, what is facial recognition system, how does facial expression formed, how
people recognize facial expressions, are facial expressions similar in most country or
culture, do facial expressions have a standard and can we trust facial expressions?
ii
HSU
Table of Contents
Introduction
1
History of Facial Expression
1
Paul Ekman
1
Formation
2
Facial Action Coding System (FACS)
3
Function
3
Experiment (Primary Research)
4
Accuracy
6
Modern Social Influences
7
Conclusion
8
Bibliography
9
iii
HSU
1. Introduction
Have you ever seen a smiley face? How about an angry face? Both anger and
smiles are examples of facial expression. “Facial expressions are complex
muscular patterns that carry complex social signals” (Jarlier et al.). In other
words, facial expressions are basically made by different muscle contractions or
different muscle patterns. Moreover, they do contain complex social signals; i.e.
convey emotions.
A face usually has 43 muscles and can make up to 10,000 facial
expressions. “There are 10,000 different facial expressions. About 3000 of these
facial expressions are relevant to emotion and most people use only 50-60 in
normal conversation (Steve). In other words, people have ability to make both
emotional and non-emotional facial expressions. For instance, angry face is a
type of emotional facial expressions, because the expression is triggered by
strong emotion; on the other hand, weird face is a type of non-emotional facial
expression, because the expression is not triggered by emotion. According to a
scientific explanation, “[a typical] facial expression results from one or more
motions or positions of the muscles of the face” (Jarlier et al.).
2. History of Facial Expression
In comparison, the study of facial expression is a relatively young science; it
started in 1970s by Paul Ekman and Wallance V. Friesen, Ekman's assistant.
They conducted primary research, such as experiments, to get other scientists to
believe the existence of this field. In order for scientists to accept a brand-new
concept or belief in science, one needs to provide a lot data to support the fresh
idea.
After the field of facial expression begins to be more popular, many more
scientists and psychologists starts to join the field. Some other famous
contributors and pioneers are Izard and Fridlund. Among all the contributors and
pioneers, Paul Ekman is the one that done the most contribution to the field of
facial expression. He established many foundations in this particular field and
provided opportunities for others to develop their ideas and knowledge, in order
to enrich the field of facial expression.
3. Paul Ekman
Paul Ekman is a person who devoted himself to the field of facial expression. He
did many tremendous contributions to the field and be recognized as the pioneer
of the facial expression.
Paul Ekman studied facial expression for more than 50 years, and he got
many honors; one of them was, "the best human lie detector in the world." In
addition, he also worked in Langley Porter Psychiatric Institute for a few years
and helped special institutions. For instance, the Federal Bureau of Investigation
– the FBI- and the Central Intelligence Agency –the CIA- (Ekman). He did not just
carry out experiments, but also wrote some books about the facial expression
and lie detection techniques. The two most famous works among all are the
Emotions Revealed and Telling Lies. Ekman also shared some of his
experiences and perceptions in both books. For instance, “I feared that
-1-
HSU
professional lie catchers… who worked for the government might be overly
optimistic about their ability to tell when someone is lying from behavioral clues”
(Ekman).
Simultaneously, he also invented the Facial Action Coding System (FACS)
- a system that can recognize facial movements and determine emotion. He also
found out some important principles. One of the significant principles is that
"When emotion is weak or come up slow, the expression may not trigger"; this
explains why sometime there is only the existence of emotion and not the
existence of both emotion and expression. Furthermore, he was also recognized
as the founder of “Micro-expression.”
Paul Ekman conducted many experiments in his lifetime. The most
famous experiment he conducted was the experiments that determining the facial
recognition pattern around different cultures and countries. “Dr Paul Ekman has
spent 50 years studying facial micro expressions of people around the world,
including the isolated Fore tribesmen in Papua New Guinea” (DelBene).
After many experiments, Ekman identified the seven universal emotions
and he also found out that those universal facial expressions are universal in
most countries and cultures; even in the tribes that are isolated from most of the
cultures. “[T]he universal facial expressions of seven emotions – anger,
contempt, disgust, fear, joy, sadness, and surprise” (Matsumoto). Aside from
different cultures and countries, he also conducted experiments on facial
movements. With the facial movements he observed and recorded, he made a
collection of different facial movements’ pictures; on top of that, with each one of
the pictures he had an explanation of what factors made up this particular facial
expression.
After Paul Ekman, there are many other scientists carrying out
experiments on facial expression. This led to the opening of the facial expression
researches. Present day scientists did more experiments and developed the
thoughts of Paul Ekman deeper and wider.
4. Formation
The facial expressions are formed by the different muscle patterns and these
patterns can be formed intentionally and unintentionally. In other words, humans
can control most of the muscle movements but some muscle movements are
controlled by our unconsciousness. In addition, there are some muscle
movements that humans are not aware of. The most understandable example is
the movement of the intestines. The intestines are moving but the movements
are rarely aware.
The facial expressions are not just formed by muscle movements, but also
formed by the nervous systems. Therefore, each facial expression is formed by
the collaboration of muscles and nerves. In addition, each facial expression also
has the ability of producing different hormones. For instance, a smile face is
capable of producing mood enhancing hormone; like endorphin (Gutman).
-2-
HSU
5. Facial Action Coding System (FACS)
Facial Action Coding System is a system developed by Paul Ekman and
Wallance V. Friesen. It is a system that is widely used and set the foundation of
many facial detection tools in modern society. “Facial Action Coding System
(FACS) is the most widely used and versatile method for measuring and
describing facial behaviors” (Facial Action Coding System (FACS) and the FACS
Manual).
Paul Ekman and Wallance V. Friesen developed the system by
determining the influences of each muscle contraction. In other words, they
looked at muscle movements and the impacts on facial appearances. “FACS
measurement units are Action Units (AUs), not muscles, for two reasons. First,
for a few appearances, more than one muscle was combined into a single AU...
Second, the appearance changes produced by one muscle were sometimes
separated into two or more AUs …” (“Facial Action Coding System (FACS) and
the FACS Manual”). In simpler terms, AUs are not codes for muscles. This is
because an AU may be caused by more than one muscle movements and
sometime a single muscle movement may create more than one AU.
With each AU, there is also a description of muscle movements. Moreover,
different combinations of AUs can provide indications of a person’s inner status
and inner feelings; meaningful emotions. For instance, AU 1 is coded for “inner
brow raiser” ("FACS - Facial Action Coding System") and the combination of AU
6 and AU 12 indicates the inner status of happy (Heller).
As humans, we recognized facial expression in a similar way as FACS.
We also identified the characteristics of different facial expressions, however, we
did it unconsciously. In other words, we recognized different faces through
special characteristics but we did not know we done that. Furthermore, we do not
have concrete data base like FACS does. We identify different faces through our
personal experiences and environmental influences.
6. Functions
Have you ever thought of the purpose of having facial expression; make facial
expressions to deceive other gamblers in the casino for personal benefits? Or
just make facial expressions for fun?
Modern scientists found out that some of the facial expressions, such as
anger and surprise, are shaped by sensory contraction and expansion action
tendencies. With this new finding, scientists suspected that the actual function of
facial expression was for adaptive sensory function and not for communication
purposes. “More evident in the temporal dynamics of spontaneous facial
expression sequences” (Susskind and Adam). In other words, the facial
expressions that still exist today are to improve the sensory function when we
make the face.
In their researches and experiments Susskind and Adam found out that
facial expression can alter the biomechanical properties of the face. For example,
the facial expression of fear can increase sensory exposure, which makes one
reacts faster, and the facial expression disgust can decrease sensory exposure.
In addition, they found out that this kind of adaptation is not only present in
-3-
HSU
human; they also found it in other primates. The ability to alter the biochemical
properties of face allows primates to protect their sensory organ from harm and
locate the stimulator or source.
Aside from sensory functions, scientists also had a struggle with what is
the basic function of facial expressions. There are three types of explanation for
this struggle.
First type is from Izard. “The point of contention remains in whether the
face also says something about a person's internal state. Some, such as Izard,
say, ‘Absolutely’ ” (Azar). In other words, Izard believed people show their inner
feeling and status of mind through their facial expressions. This means, one of
the purposes of the facial expression is for communication.
Second type is from Ekman, he believed that facial expressions can show
important messages of a person’s status of mind, such as emotion, but it is only
one of the tools and should not be used as a standard. Furthermore, Ekman
believed facial expressions should be described from the expresser’s point of
view; prediction purpose. In other words, what caused the expresser to do this
expression and what action is going to happen? (Ekman)
Other type is from Fridlund. “…Detractors, such as Alan Fridlund, PhD, of
the University of California, Santa Barbara, say an adamant ‘No’ ” (Azar). In other
words, Fridlund does not believe facial expressions contain any evidences of
cues of a person’s inner status or feelings. Moreover, Fridlund also believed that
people should describe the facial expression from the receiver’s point of view;
communicational purpose. In other words, what is the effect on receiver when
expresser makes the expression. “The face is like a switch on a railroad track, it
affects the trajectory of the social interaction the way the switch would affect the
path of the train” (Fridlund).
As the study of facial expression progressed, the two types of explanation
are combined. The facial expressions were existed for these two reasons;
communication and prediction. In other words, people used facial expression to
convey emotion and some people used facial expression to predict others’ action.
7. Experiment (Primary Research)
This section included an experiment planned and conducted by the author of this
research paper. This will provide detail method of the experiment- for those who
wanted to conduct a same experiment to compare result- and a clear list of
findings. Due to participants’ privacy, actual pictures cannot be shared; the
findings and results are only in the form of description.
Experiment Report (Facial Expression and Culture)
Project: Facial Expression and Culture
Purpose: Determine the relationship between culture and facial expression
Background: This project is inspired by Paul Ekman’s experiment. He did a
research on how Papua New Guinea tribes interpret different facial expressions.
However, this experiment focuses on the influences of culture on facial
-4-
HSU
expression, because culture might be a significant factor that influences the
expression.
Dependent Variable: Facial Expression
Independent Variable: Culture
Constant: Types of facial expression, amount of picture taken, and order of
facial expressions presented.
*Extraneous variables are reduced by random sampling
*Random sampling is conducted by random replies
Hypothesis: If participants are from the same culture, then the differences on
facial expressions will not be significant, because all participants have similar
cultural background and experience.
Materials: Camera and writing utensil
Methods:
1. Determine the facial expressions that participants are required to make
2. Ask for participants (Random sampling is necessary)
3. Arrange dates for taking pictures
4. Determine cultural background before taking pictures
5. Ask participant to demonstrate different facial expressions
6. Take pictures for each facial expression
7. Repeat step 4 to step 6 for every participant (Order of facial expression
taken is important)
Data:
 Total Participants: 13 people
 Gender Ratio (Male: Female): 4:9
 Cultures Tested: Western Culture
 Dates: November 20, 2013 – November 21, 2013
 Expression Tested: smile/happy, angry, sad/depressed, surprised, scared,
confused and contempt
Observation:
 As the facial expression gets harder to mimic, participants take longer time
to make the face
 Positive expressions are presented faster than negative expressions
 Most participants have similar, almost identical, “surprised” and “scared”
expressions
 Some participants have similar, almost identical, “angry” and
“sad/depressed” expressions
 Some participants involve gestures
 Most participants have difficulties mimicking “contempt”
 All participants can present “smile/happy” easily
 All participants mimic expressions through personal experiences
Conclusion:
There is no direct linkage between culture and facial expression. Lacking linkage
is due to the fact that participants with same or similar culture do not have similar
expressions in most expressions. Further experiments need to be conducted in
order to eliminate the factor of “experiences.” However, there are other
findings…. (Order is not related to importance)
-5-
HSU
1) Facial expression is also influenced by experiences. (Maybe even greater
influence than culture)
2) Participants’ speed of recalling the expression is correlated with how often
participants make the expression.
3) Most participants need to recall the mood/feeling before they can mimic
the complex facial expressions.
4) Often seen expression, i.e. smile/happy, is easiest to present
5) Gestures help forming facial expression
6) Reaction time of positive and negative expressions can help determine the
overall mood of the participant
7) Complex expressions are harder to present (May be the participants need
more time to recall expression)
8) “Smile/happy” expression is similar, almost identical, in every participant’s
picture.
9) “Surprised” and “scared” expressions are difficult to identify
10) “Angry” and “sad/depressed” expressions have many common traits
Further Experimental Goals:
1. Change “culture” to “experience” (To narrow the factors)
2. Reduce the testing from culture to region
3. Increase participants
8. Accuracy
Many scientists and psychologists had wondered the accuracy of lie detections
and the reliabilities of the special characteristics in each facial expression.
However, this question is like the “nature versus nurture” question in psychology.
Some experts believe people can trust the reliabilities in facial expressions and
some do not.
8.1 Reliable
“Research has shown that only 7% of emotional meaning is conveyed through
the actual words we speak, while the remaining 93% is communicated through
nonverbal means, with 55% through facial expression, body language and
gestures and 38% through tone of voice. (Mehrabian, 1987)” (Crissey). This
result also been known as the “7%-38%-55%” rule found by Albert Mehrabian or
the Mehrabian’s rule. In other words, most of emotion in the verbal messages are
not correctly convey or are hidden.
“We conclude that reliable AUs may indeed convey trustworthy
information” (M.) In other words, the facial expression is reliable because the
FACS is reliable. It is able to detect and recognize different AU, which means
precise AU recognition equal to precise indication of inner status and feelings.
Therefore, the idea of facial expression can reveal the inner status and inner
feelings of a person is true.
8.2 Unreliable
A typical expert who does not believe the facial expression is reliable will likely to
criticize the fact that Mehrabian’s rule is often being overly interpreted. “Total
-6-
HSU
Liking = 7% Verbal Liking + 38% Vocal Liking + 55% Facial Liking. Please note
that this and other equations regarding relative importance of verbal and
nonverbal messages were derived from experiments dealing with
communications of feelings and attitudes. Unless a communicator is talking
about their feelings or attitudes, these equations are not applicable” (Mehrabian).
In other words, this particular equation can only be applied in a conversation of
like or dislike. This result is so narrow and precise to this situation because this
rule is derived from Mehrabian’s other works; the “Decoding of Inconsistent
Communications” (Mehrabian and Wiener) and “Inference of Attitudes from
Nonverbal Communication in Two Channels” (Mehrabian and Ferris).
“‘There’s some sense in which faces express emotion, but only in the
sense that everything expresses emotion,’ says psychologist James Russell,
PhD, of the University of British Columbia, a long-time critic of the expressionemotion link. ‘Music does, posture does, words do, tone of voice does, your
behavior does. The real question is, “Is there anything special about faces?” And
there we really don't know much.’” (Azar). In other words, we cannot pinpoint or
identify the exact relationship of facial expression and emotion. With the modern
knowledge and technology we can only guess the causation and correlation.
Therefore, the field of facial expression still needs more evidences and data to
support the relationship.
9. Modern Social Influences
The idea of facial expression did not just stay as an idea or as a form of science.
This idea also existed in modern media, arts, and perceptions. In media and arts,
the idea of facial expression becomes a form of mood. The media creators and
artists use the significant characteristics of each facial expression to create the
mood of an environment. For instance, an artist may draw many happy faces to
indicate the scene in happy and people are feeling happy. Even though the
audiences may not know the specific characteristic of each facial expression, but
when they see the expression usually the audiences can get the idea or feeling
that the artist is trying to convey. This idea can be seen in many publications of
basic drawings manual. One of the most detail drawing manuals is Gary Faigin’s
The Artist's Complete Guide to Facial Expression. This book shows an extremely
precise and scientific way of drawing different facial expressions.
Besides from media and arts, the idea of facial expression also influences
the perception. When the idea of facial expression is used as a theme of a
popular show, audiences are likely to think this field is interesting and will
established a perception of wanting to learn more about this field or pursuing a
career in the field of facial expression. For instance, a famous television series
called “Lie to Me” is based on the research of Paul Ekman. “In Lie to Me, Tim
Roth plays Dr. Cal Lightman, a deception consultant and expert. But Lightman
doesn't rely on some futuristic mind-reading tricks cooked up by TV writers.
Instead, his character is based on clinical psychologist Paul Ekman, a leading
expert on lie detection.” ("The (Real!) Science Behind Fox's Lie to Me.").
-7-
HSU
10. Conclusion
Even though facial expression is a new form of science, but with the contributions
from many brilliant scientists and psychologists, the understanding of facial
expression improve exponentially.
With the helps from different contributors, we are able to understand two
significant principles in the field of facial expression. First, there are seven
universal facial expressions across the globe. They are anger, contempt, disgust,
fear, joy, sadness, and surprise. Second, if the inner status, emotion, is not
strong or reacts not as fast, then the expression may not appear or may look like
a fake expression.
With the improvement of modern technology, we are able to provide
different forms of facial recognition systems and pinpoint different muscle
movements. The reliability, however, is still a debate for many scientists and
psychologists. There are many experiments and data that can prove both sides
of the claim. Therefore, the best resolution for now is that try to stay in the middle
of the claims. “When we look at people's expressions, [Ekman] says, we don't
receive direct information about their heart rate or other physiological changes
that accompany emotions. We might even think, ‘He's going to whack me’ rather
than ‘He's angry,’ says Ekman” (Azar).
-8-
HSU
Bibliography
Azar, Beth. "What's in a Face?" Http://www.apa.org. N.p., Jan. 2000. Web. 23
Oct. 2013. <http://www.apa.org/monitor/jan00/sc1.aspx>.
Crissey, Pat. "Learning to Read Facial Expressions." Autism Aspergers Digest
RSS. N.p., July-Aug. 2008. Web. 23 Oct. 2013.
<http://autismdigest.com/learning-to-read-facial-expressions/>.
DelBene, Andrea. "The Secret behind the Best Human Lie Detector in the
World." Lost At E Minor: For Creative People. N.p., 22 Aug. 2013. Web. 23
Oct. 2013. <http://www.lostateminor.com/2013/08/22/the-secret-behindthe-best-human-lie-detector-in-the-world/?utm_medium=email>.
Ekman, Paul. Telling Lies: Clues to Deceit in the Marketplace, Politics, and
Marriage. New York [etc.: W.W. Norton, 1992. Print.
Ekman, Paul. Emotions Revealed: Recognizing Faces and Feelings to Improve
Communication and Emotional Life. New York: Times, 2003. Print.
Fridlund, Alan J. Human Facial Expression: An Evolutionary View. San Diego:
Academic, 1994. Print.
Faigin, Gary. The Artist's Complete Guide to Facial Expression. New York:
Watson-Guptill Publications, 1990. Print.
Gutman, Ron. "Ron Gutman: The Hidden Power of Smiling." TED: Ideas worth
Spreading. N.p., May 2011. Web. 23 Aug. 2013.
<http://www.ted.com/talks/ron_gutman_the_hidden_power_of_smiling.htm
l>.
Heller, Michael Coster, and Dr. Phil. "SADNESS AND FACIAL ACTION CODING
SYSTEM (FACS)." N.p., n.d. Web. 18 Dec. 2013.
<www.aqualide.com/upload/texte/text57.ppt>.
Jarlier, Sophie, Didier Grandjean, Sylvain Delplanque, Karim N’Diaye, Isabelle
Cayeux, Maria Ine´s Velazco, David Sander, Patrik Vuilleumier, and Klaus
R. Scherer. "Thermal Analysis of Facial Muscles Contractions." N.p.,
Spring 2011. Web. 23
Oct.2013.<http://labnic.unige.ch/nic/papers/SJ_DG_SD_KND_IC_MIV_DS
_PV_KRS_IEEETransac11.pdf>.
Matsumoto, David, and Hyi Sung Hwang. "Reading Facial Expressions of
Emotion."Http://www.apa.org. N.p., May 2011. Web. 23 Oct. 2013.
<http://www.apa.org/science/about/psa/2011/05/facial-expressions.aspx>.
Mehrabian, Albert; Ferris, Susan R. (1967). "Inference of Attitudes from
Nonverbal Communication in Two Channels". Journal of Consulting
Psychology 31 (3): 248–252. Print.
Mehrabian, Albert. Silent Messages: Implicit Communication of Emotions and
Attitudes. Belmont, CA: Wadsworth Pub., 1981. Print.
Mehrabian, Albert; Wiener, Morton (1967). "Decoding of Inconsistent
Communications". Journal of Personality and
Social Psychology 6 (1): 109–114. Print.
M., Mehu, Mortillaro M., Bänziger T., and Scherer KR.. "Reliable Facial Muscle
Activation Enhances Recognizability and Credibility of Emotional
Expression." National Center for Biotechnology Information. U.S. National
-9-
HSU
Library of Medicine, 28 May 2012. Web. 18 Dec. 2013.
<http://www.ncbi.nlm.nih.gov/pubmed/22642350>.
Roesler, Steve. "Recognize the Seven Universal Emotions." 'All Things
Workplace' N.p., n.d. Web. 23 Oct. 2013.
<http://www.allthingsworkplace.com/2009/06/recognizethe-seven-universal-emotions.html>.
Susskind, Joshua M., and Adam K. Anderson. "Facial Expression Form and
Function."National Center for Biotechnology Information. N.p., Winter
2008. Web. 23 Oct. 2013.
<http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2686004/>.
"Facial Action Coding System (FACS) and the FACS Manual." Data Face. N.p.,
n.d. Web. 18 Dec. 2013. <http://face-and
emotion.com/dataface/facs/description.jsp>.
"FACS - Facial Action Coding System." FACS (Facial Action Coding
System). N.p., n.d. Web. 18 Dec. 2013.
<http://www.cs.cmu.edu/~face/facs.htm>.
"The (Real!) Science Behind Fox's Lie to Me." Popular Mechanics. N.p., 1 Oct.
2009. Web. 18 Dec. 2013.
<http://www.popularmechanics.com/science/4300722>.
- 10 -