MICRO EXPRESSION DETECTION USING STRAIN
PATTERNS
-SRIDHAR GODAVARTHY (U32086851)
1 INTRODUCTION
1.1 THE BLINK EFFECT
The human decision making capability works in two modes.

The most common mode is one where decisions are pondered upon, all factors
weighed and a decision reached - usually after long periods.
The other, more subtle, mode is where decisions are taken instantaneously after
considering all the immediately available evidences. In this mode, commonly
referred to as the subconscious mode, one does not consciously weigh the
various contributing factors. Instead, the human mind, in a blink, recognizes that
there is something right or wrong about the subject in question. Some of these
‘visible cues’ may include- but are not limited to- fast blinking of the eyes, a
shrug of the shoulders, raising of the eyebrows wrinkling of the forehead etc.
This phenomenon by which the human mind captures subtle clues is known as
the blink effect.

1.2 FACIAL EXPRESSIONS
Expressions of the human face can be classified based on the amount of the expression
and the meaning conveyed by the expression. In general they are classified into Macro
expressions and Micro expressions.
1.2.1 MACRO EXPRESSION
A macro expression can be defined as one where the expression on the face conveys a
human emotion (Anger, laughter etc.)
1.2.2 MICRO EXPRESSION
A micro expression can be defined as one where there are significant expressions on the
human face, but not enough to convey an emotion. In other words, facial expressions
which do not fall under the umbrella of a macro expression can be termed as micro
expressions. These are subtle expressions and usually involve only small portions of the
human face such as the eyebrows, eyes forehead etc.
1
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
The classification of an expression into a micro expression is not clearly defined.
However, for the sake of this project, we assume that an expression can be classified as
a micro expression if
a) It is not a macro expression and
b) The strain measured during such an expression is significantly lesser than the
strain measured during a macro expression.
Later in this report, we will also bring in the discussion that an expression can also be
classified in terms of the duration of the expression. Usually, a micro expression lasts
over a second and hence over a 30-40 frame span. A micro expression on the other
hand does not exceed 5-10 frames.
In this project we attempt to replace the human mind by a vision system that observes
the human face and detects such micro expressions. To achieve this goal, we capture
the expressions of the human face, measure the strain on the face and use the amount
and duration of the strain to classify an expression into a micro or a macro expression.
2 BACKGROUND
2.1 STRAIN
To understand strain, we first describe elasticity of a material. Elasticity is the physical
property of a material when it deforms under stress, but returns to its original shape
when the stress is removed. The relative amount of deformation of the material is
known as strain. The elasticity of a material is a constant and can be modeled as follows:
𝐸𝑙𝑎𝑠𝑡𝑖𝑐𝑖𝑡𝑦 𝜆 =
𝑆𝑡𝑟𝑒𝑠𝑠
𝑆𝑡𝑟𝑎𝑖𝑛
If the stress applied on a material is constant, we can measure the strain on the
material.
2.2 FACIAL STRAIN
Facial strain is the strain on the soft tissue of the face when expressions are made. It can
be measured based on the above. The strain thus measured will be nearly a constant for
the same expression irrespective of the number of times it is repeated. Also facial strain
is unique to each expression and to each person. However, not all parts of the human
2
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
face follow the same strain pattern. For example, the eyes and the hair (eyebrows) do
not have a uniform elasticity and cannot be included as part of this exercise.
2.3 MEASUREMENT OF FACIAL STRAIN
There are several contact strain measurement equipment available. However, these are
intrusive. We use an anatomical method to measure the facial strain of a face when an
expression is being performed. The method involves the use of two frames – One frame
showing a neutral expression and another showing the expression used to measure the
deformation between the two frames. The procedure for measurement involves two
steps:
i.
ii.
Obtaining motion field between the two frames
Computing strain image from the above motion field.
2.3.1 MOTION FIELD BETWEEN TWO FRAMES
There are two approaches to calculating the motion field between any two frames:
I.
II.
Feature Based
Flow Based
2.3.1.1 F EATURE B ASED
These methods involve identifying the features in a frame, segmenting them and
determining the motion vectors by measuring the displacement of the features.
This method produces good results, but has a lot of disadvantages in that it is
difficult to identify the features correctly. Further, features may be ill defined (if the
face is camouflaged) and usually requires manual intervention. Also, this method is
more applicable to large motion fields and does not yield effective results in the
case of micro movements which is our concentration in this project.
2.3.1.2 O PTICAL F LOW B ASED
This method involves Pixel tracking to determine the motion vector. This method
can be completely automated and produces a dense motion field. The only
constraint on this method is that it requires a constant illumination. As our project
involves videos captured in a controlled environment, the illumination remains
fairly constant and hence we follow this approach.
2.3.2 OPTICAL FLOW
Optical flow is the pattern of apparent motion of objects or surfaces in a visual scene.
The motion can be caused either by movement of the object or the camera. Optical flow
3
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
is a very popular method and is used in every significant video coding technique.
Following is an illustration of optical flow in a video sequence involving a moving ball.
FIGURE 1 DEMONSTRATION OF OPTICAL FLOW
Despite its advantages, optical flow can only be used between frames which have a
constant illumination. The optical flow is given by the following equation popularly
known as the Optical Flow equation:
If we solve the partial differential equations above, we will obtain a set of motion
vectors. An example motion vector for a frame involving rotating/revolving of an object
is as shown below
FIGURE 2A SPINNING DISK
FIGURE 2B MOTION VECTORS FOR VIDEO IN 2A
2.3.3 STRAIN COMPUTATION
Strain can be of two types: 3D and 2D strain. 3D strain would be ideal to calculate, but
requires sophisticated high speed equipment. We do not compute this as it involves
higher complexity. However, it has been proven as part of another project that there is
not much difference in accuracy between 2D and 3D strain.
2D strain involves measurement of the variation of displacement values obtained from
the optical flow and is measured by calculating the derivative of each pixel. The
derivative can be calculated using any of two approaches –
4
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
i.
The Finite Element Method – This method is an approximation to the solution
and involves forward modeling when the Dirchlet conditions are satisfied. This is
good at handling irregular shapes but is computationally expensive.
The Finite Difference Method – Strain is a tensor and can be expressed as
derivatives of the displacement vector. This can hence be expressed as a finite
difference. This method is an approximation to the actual equations against the
FEM which is an approximation to the solution. This is method is much better at
handling regular shapes.
ii.
We follow the finite difference approach for our solution adapting a Cauchy tensor to
model the strain. The Cauchy tensor can be described by:
∈=
1
3
2
4
1
[∇𝑢 + (∇𝑢)𝑇 ]
2
1. Neutral Frame
2. Frame with Expression
3. Optical Flow between the two frames
4. Calculated optical Strain from the optical flow in (3)
FIGURE 3 SAMPLE OPTICAL FLOW AND CORRESPONDING OPTICAL STRAIN
5
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
3 ALGORITHM AND PROGRAM
The algorithm for the micro expression can be expressed as follows
1. Identify training video
2. Decode training video into frames
3. Ask user to identify the frames with neutral expressions, micro and macro
expressions.
4. Use the above frames to calculate the optical strain for micro and macro
expressions.
 Use the above values to setup threshold strain values for micro and
macro expressions.
5. Identify test video
6. Decode test video into frames
7. Ask user to identify neutral frame.
8. For each frame in the test video, calculate optical strain between the neutral
frame and the target frame.
9. If the optical strain falls between the thresholds calculated earlier, classify frame
as micro expression.
The code for determining the Optical Flow was obtained from [5].
The entire process can be represented as a flowchart:
6
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
Start
Divide videos into
Training and Testing
Sequences
Training Sequence
Read
Training
Video
Decode Video
Read
Neutral
Micro Exp
Macro Exp
Frames
Read Test
Video
Decode Video
Read
Neutral
Frames
Stop
Testing Sequence
Calculate Optical
Flow between
Neutral frame and
each frame
OF within
desired
range?
Yes
Calculate Optical
Strain between
Neutral frame and
this frame
OS within
desired
range?
MICRO
EXPRESSION
No
No
Reject Frame
Yes
Reject Frame
FIGURE 4 FLOW CHART FOR IDENTIFYING MICRO EXPRESSIONS
7
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
3.1 SAMPLE OUTPUT
FIGURE 5 SAMPLE OUTPUT FROM THE PROGRAM
4 DATA SET
The dataset comprises of three videos of subjects performing micro and macro
expressions. The subjects were also asked to task and move their head to introduce
noise to test the robustness of the system. Following are some frames of the dataset
which show some of the expressions in the video. The same datasets were used as test
and training sets. Each dataset consists of about 15 seconds of video i.e approximately
450 frames.
8
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
FIGURE 6 SAMPLE FRAMES FROM THE THREE DATASETS
9
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
5 RESULTS AND DISCUSSIONS
This section discusses the results obtained from this project. We have some promising
results.
In the following sequence, the system was able to clearly identify the micro expression.
The neutral expression and the matching micro expression are shown. The micro
expression involves raising an eyebrow. The optical flow clearly shows that our system
did indeed capture the motion correctly.
1
2
3
4
1. Neutral Frame
2. Frame with Expression
3. Optical flow computed with Black method.
4. Normalized optical flow
FIGURE 7 SAMPLE FRAMES FROM THE TRAINING DATASET
Following is the test sequence in which too, the system was able to identify the micro
expression. The neutral expression and the matching micro expression are shown. The
micro expression involves raising both eyebrows. As can be seen, even though the
optical flow captures the motion all over the face, by normalizing the optical flow, we
are able to identify the motion.
10
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
1
2
3
4
1. Neutral Frame
2. Frame with Expression
3. Optical flow computed with Black method.
4. Normalized optical flow
FIGURE 8 SAMPLE FRAMES FROM TEST DATASET
The following is a negative case in which the system reports a false positive. On a closer
look at the optical flow, we can see that the opening of the mouth caused a large optical
flow near the mouth. The movement of the mouth involves a lot of strain and is
considered a macro expression. Our system should have rejected this frame, but it
reports it as a micro expression. This problem could have been easily resolved by
segmenting the image and considering individual parts separately instead of considering
the face as a whole.
11
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
1
2
3
4
1. Neutral Frame
2. Frame with Expression
3. Optical flow computed with Black method.
4. Normalized optical flow
FIGURE 9 FALSE POSITIVE
The following is a sequence in which our system correctly rejects the frame as much
more than a micro expression. This can be expected as the optical flow is very complex.
This frame sequence would have been rejected based on the Optical Flow itself and
would never have made it to the Optical Strain comparison.
12
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
1
2
3
4
1. Neutral Frame
2. Frame with Expression
3. Optical flow computed with Black method.
4. Normalized optical flow
FIGURE 10 CORRECTLY CLASSIFIED (REJECTED) FRAME SEQUENCE
One of major drawbacks of this approach was that an entire sequence was reported
instead of a single frame. This was a direct consequence of using the strain as the sole
factor for determining the expression. Another drawback was that this method was
unable to identify a micro expression if the face also consisted of a macro expression.
This was because the macro expression caused the strain to exceed thresholds and as
we were only considering the maximum strain in the image, the entire frame was
rejected based on the macro expression.
6 SUGGESTIONS AND PLAUSIBLE RECOMMENDATIONS
Several suggestions were made during the presentation of the project, the most
significant of which are being presented here. These suggestions have been
incorporated as part of the section titled Future work, but are also being presented
separately to present the contribution of the audience.
6.1
DURATION OF THE EXPRESSION
This is probably, the most significant of the recommendations. By utilizing the
assumption that micro expressions are of much shorter duration than macro
expressions, and ignoring all frames that match the optical strain criteria for micro
expressions and are part of a longer sequence of matches, we can eliminate many of the
false positives. Adapting this approach also provides the additional advantage of
reporting only one expression for a sequence of matches.
13
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
6.2 SEGMENTATION OF THE FACE
An important property of micro expressions is that they do not involve the entire face.
Instead they are localized to small regions such as the eyebrows, eyes, the region above
the eyes etc. Hence, it can be expected that we would get better performance if, instead
of measuring the strain over the entire face, we segment the face into regions of
interest and measure strain only in these regions. For example, consider a scenario
where a person is talking and expresses a shock at the same time. His mouth will be
moving, but at the same time, his eyebrows (and the regions above them) would have
risen and fallen. If we were to consider the entire face, the expression would have been
dominated by the mouth and our classifier would have rejected it as a macro expression
as it is a part of a larger sequence. However, if were to individually consider the
eyebrows, it would form a micro expression as this specific movement is for a very short
duration.
7 FUTURE WORK
The future work will mostly comprise of the suggestions obtained during the
presentation. We will attempt to classify the frames based on time rather than amount
of strain. This will help solve several problems:
1. We will no longer have to report an entire sequence as micro expression. We can
consider the entire sequence and report only the frame with the maximum
strain within the sequence.
2. Any computation errors in the strain caused due to noise will be eliminated as
we average out the strain and an odd frame in the middle of a sequence would
be rejected.
The approach would be similar to the current one except that instead of reporting
immediately after calculating the strain, we will introduce a memory component which
will record the frame numbers from the frame where the strain exceeds the lower
threshold until the frame where the strain goes below the lower threshold. If, in
between these frames the strain exceeds the upper threshold for a significant number
of frames, we can classify this as a macro expression and reject the entire sequence. If it
is entirely within the allowed range, we choose the frame with the highest strain and
declare it as the micro expression.
14
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
The next work would be to apply some sort of segmentation to divide the image into 3
to 4 major regions:
a.
b.
c.
d.
e.
Forehead
Eyebrows and skin between eyes and brows.
Eyes
Cheeks
Mouth
The first three parts are the major contributors for micro expressions and the last two
mostly correspond to macro expressions. By performing this segmentation, we could try
out various combinations such as
i.
ii.
Reject cheeks and mouth and consider strain only in the first three
Add a negative weight to the mouth and cheeks and a greater positive
weight to the forehead, eyebrows and eyes.
One thing to be noted here is that the eyes do not follow the same strain pattern as the
skin. Hence it would be wrong to measure the strain in this region of the face. This
brings out with a third possibility
iii.
Black out the eyes and do not use it in strain computation
Eyes, however form a critical part of the human expression. Hence, instead of simply
ignoring the eyes, we could design an algorithm which could measure the amount of
blinks in the eye. A significant increase in the number of blinks could signal a micro
expression.
8 CONCLUSIONS
In this project, we have designed a system which reads a video of the human face and
classifies the various expressions exhibited by the face into micro and macro
expressions. The initial neutral expression is identified by the user and the system
measures the strain between the neutral frame and every other frame. It then uses the
measured strain values to perform the classification. The system performed fairly
accurately in the presence of movement of the face. However, there was a high false
alarm rate. Several suggestions were received as part of the project presentations, the
most significant of which was the usage of the duration of the expression as another
feature to be used in classification. Another significant recommendation was to segment
the face into regions of interest and measure the strain only in these regions instead of
15
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy
using the face as a whole. These should produce better results. These suggestions will
be taken up as part of my future work on this project.
9 REFERENCES
[1]V.Manohar, D.B. Goldgof, S.Sarkar, Y. Zhang, "Facial Strain Pattern as a Soft Forensic
Evidence", IEEE Workshop on Applications of Computer Vision (WACV'07), pp 42-42
[2]Vasant Manohar, Matthew Shreve, Dmitry Goldgof and Sudeep Sarkar, "Finite
Element Modeling of Facial Deformation in Videos for Computing Strain Pattern",
International Conference on Pattern Recognition, Dec. 2008
[3]Matthew A. Shreve, Shaun J. Canavan, Yong Zhang, John R. Sullins, and Rupali Patil,
"Imaging And Characterization Of Facial Strain In Long Video Sequences",xxxx
[4]Malcolm Gladwell,” Blink: The Power of Thinking Without Thinking”, Back Bay Books
(April 3, 2007)
[5] http://www.cs.brown.edu/~dqsun/research/software.html
16
Micro Expression Detection Using Strain Patterns
Sridhar Godavarthy