+
Defense of a Masters Thesis
Computer Science and Engineering
University of South Florida
Sridhar Godavarthy
July 01, 2010
Defense of a Masters Thesis
Computer Science and Engineering
University of South Florida
Microexpression Spotting in Video Using
Optical Strain
Sridhar Godavarthy
Examining Committee
Dmitry B. Goldgof, Ph.D. – Major Professor
Sudeep Sarkar, Ph.D.
Rangachar Kasturi, Ph.D.
July 01, 2010
+ Minutes of the presentation
 Microexpressions
- “micro” expressions.
 Goal: Detect “interesting” sequences containing
μE.
 Approach:

optical flow + strain thresholding.
Result:
True positive detection as high as 80%.
 Good performance on real time videos.

 Conclusion:
Novel system. Scope for improvement.
Need more datasets.
+
Introduction
+ Expressions
5
 Social emotion conveyance
 Non verbal
 Voluntary or involuntary
 6 primary expressions
Anger
Disgust
Fear
Happiness Sadness
Surprise
+ Microexpressions – What?
6
 Subtle movements of the human body
 Observable
 Insufficient to convey emotion
 Masking an expression
 1/25th to 1/5th of a second
 Almost impossible to suppress
07/01/2010
+ Why?
7
 Lie Detection
 Pain detection for autistic and anaesthetized patients
 Social signal processing( boredom/ concentration
detection)
 Psychological
counseling.
07/01/2010
+ State of the art
8
Microexpression Research
Psychology
Vision
Intent
FACS classification
Optical Flow
Gabor Filters
ANNs
Rule Based
07/01/2010
+
Objective
 Design
 Spots
9
a preprocessing system that
microexpressions.
 Handles
small translational and rotational
motion
 Improves
 Greater
performance of existing systems
weight to true positives.
07/01/2010
+
Some Fundamentals
 Optic Flow : Vector representation of temporal
changes
 Strain: Relative deformation of material (skin)
 Haar Classifier / Viola-Jones Face detector
Cascade of weak classifiers
 OpenCV implementation
 Uses Haar rectangular features

+ Brief overview of algorithm
11
( Main Idea)
 Skin deforms during an expression.
Strain Magnitude 
Peak
Thresholding
Detection
 Deformation peaks at peak of expression
Micro
of increased strain corresponds
to
Expression
microexpressionMacro
 Duration
Expression
~5 frames
~22 frames
Frames 
07/01/2010
+
Algorithm
+
System Flow
Split Frames
Strain patterns and
Thresholding
Face Detection & Alignment
Split into regions
Optical Flow
Strain Map
+ Face detection
Viola-Jones face detector
-OpenCV implementation
+ Face Alignment: Rotation
15
07/01/2010
+
System Flow
Split Frames
Strain patterns and
Thresholding
Face Detection & Alignment
Split into regions
Optical Flow
Strain Map
+ Optical flow
17
• Black and Anandan
• Dense
07/01/2010
MJ Black’s Matlab imlementation of OF
+
System Flow
Split Frames
Strain patterns and
Thresholding
Face Detection & Alignment
Split into regions
Optical Flow
Strain Map
+ Facial strain
19
07/01/2010
+
System Flow
Split Frames
Strain patterns and
Thresholding
Face Detection & Alignment
Split into regions
Optical Flow
Strain Map
+ Region Splitting
left cheek (lc)
forehead(fh)
right eye(re)
right cheek (rc)
right mouth(rm)
AUs not covered
left mouth(lm)
below mouth(bm)
Automated. Manual intervention if classifier fails
• Blink
• Close eyes
• Neck tightening
•Nostril flare
+
Datasets & Results
+
Datasets
USF
(100)
Canal9 (24)
USF: IRB
Canal9: EULA
Found (4)
Found Videos: Fair use act
Threshold Determination
Threshold as
percentage of
% True positives % False positives
peak strain
Threshold Selection
Sl. No.
1
100
90
2
3
4
75
31.8
0
50
50
0
35
77.2
22.7
30
54.5
36.4
25
13.6
0
80
% True Positive
+
24
70
60
50
40
30
20
5
10
0
75
50
35
30
25
% Peak Strain
07/01/2010
+
25
Thresholded Strain Maps – Sample 1 / 3
07/01/2010
+
26
Thresholded Strain Maps – Sample 2 / 3
07/01/2010
+
27
Thresholded Strain Maps – Sample 3 / 3
 False Positive ( Indicative only)
07/01/2010
+
29
Microexpression Spotting
07/01/2010
+
30
Negative Test Case
– Rejects Expressions
07/01/2010
+
Concluding Remarks
+
Contributions and Conclusions
 Automated thresholding
 Automated alignment (Partial)
 Region wise detection
 Up to 80% true detection
 Microexpressions with expressions are
detected.
+
Constraints
 Constant illumination
 Neutral face
 Some expressions may be falsely detected

Talking
+ Future Work
34
 Dataset Collection
 Real time questioning videos
 Fully automated face alignment
 By matching optical flow vectors
 Automatic identification of neutral face
 Automatic portioning of faces
 Anthropomorphic landmark identification
07/01/2010
+
35
Related Publications

Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., Sarkar, S.,
"Towards macro- and micro-expression spotting in video using
strain patterns," Workshop on Applications of Computer Vision,
2009 pp:1-6
07/01/2010
+ Other Publications by author

Candamo, J., Kasturi, R., Goldgof, D., Godavarthy, S., "Detecting Wires in
Cluttered Urban Scenes Using a Gaussian Model, " to appear in Proceedings
of International Conference on Pattern Recognition(ICPR 2010), Turkey,
2010

Godavarthy, S., Roomi, M. Md., “Adaptive Contrast Based Unsharp
Masking,” in Proceedings of the National Workshop on Computer Vision,
Graphics and Image Processing, Feb 2002

Godavarthy, S., Pandian, A., Roomi, M. Md., “Histogram Equalization by
Measure of Enhancement,” in Proceedings of the National Workshop on
Computer Vision, Graphics and Image Processing, Feb 2002

Godavarthy, S., Shankar, A., Roomi, M. Md., “Adaptive Watermarking-a FFT
Approach”, Proceedings of International Conference on Advances in
Telecommunication and Information Technology "Asia - Pacific Telecom
2000" (14th, 15th December 2000), Vellore
36
07/01/2010
+
Dr. Ekman on A-Rod
http://www.nytimes.com/2009/02/15/weekinreview/15marsh.html
+
38
THANK YOU
+
Index
Presentation:
Additional Slides

Minutes

Evolutionary psychology

System Flow

Detailed Flow Chart

Thresholding

Optical Flow

Sample Strain Maps

Elasticity and Strain

Results

FACS

Negative Test Case

OF Vs OS

Conclusion and Future Work

Dataset Details
+
Additional Slides
+ Evolutionary Psychology
 Study of everything we discussed
 The child of ONE man






until now
- Paul Ekman.
Over thirty years of research
One of the world’s leading experts on lying.
About 2 dozen books and innumerable articles
Developed FACS
Scientific Advisor to “Lie to Me”
Co creator of Microexpression Training Tool (METTx)
+
42
Flow Chart
Input video sequence
Split into individual frames
Detect and Crop face
Translate/Rotate to align with
neutral mage
Calculate Robust Optical Flow
Viola-Jones Face
Detector
Haar Cascade,
Skin Detection
Black and Anandan
Method
Displacement
Vectors
Calculate Optical Strain
Finite Difference
Method
Strain per pixel
Split into Regions
Eight regions or less
depending on visibility
Compute and normalize
strain/region
Determine threshold for Strain
Magnitude
Threshold
Thresholding for magnitude,
duration and spatial locality
Microexpression
07/01/2010
+ Motion Estimation: Optical Flow
Method

Add: OF
Reflects the changes in the image due to motion
Computation is based on the following assumptions:
 observed brightness of any object point is constant over time
 nearby points in the image plane move in a similar manner
E 2 ( x, y)  ( f xu  f y v  ft )2   (ux2  u y2  vx2  v y2 )
 Minimization problem:

(brightness const.) (smoothness const.)

Robust estimation framework (Black and Anandan, 1996)
 Recast the least squared formulations with a different error-norm function instead of
quadratic
 Coarse-to-fine strategy
 Construct a pyramid of spatially filtered and sub-sampled images
 Compute flow values at lowest resolution and project to next level in the
pyramid
10/29/2009
43
+ Optical Flow
•
Def: Optical Flow is the apparent motion of brightness
patterns in the image
•
Ideally,
the motion field
• same
Key as
assumptions
•
• Brightness constancy: projection of the same
Have to be
careful: apparent motion can be caused by
point looks the same in every frame
lighting changes without any actual motion
• Small motion: points do not move very far
• Spatial coherence: points move like their
neighbors
Elasticity

Different materials have different elasticity

Elasticity can be modeled
stress
Elasticity   
strain
Known
Calculate
+
Facial Strain

What is Facial Strain?

Strain on soft tissue when expressions are made.

Anatomical method

Uses a pair of frames to measure deformation
+
Strain Measurement

Finite Difference Method

Compute spatial derivatives from discrete points.

Forward Difference Method

Central Difference Method

Richardson extrapolation
+
48
Thresholding

Threshold Strain Maps to segment out μE
07/01/2010
+
The Facial Action Coding System
(FACS)

Coding of human expressions

Observational and Anatomical

32 Action Units and 14 Action Descriptors

Encode any possible [facial] expression.

Also used for facial expression simulations
+ Name
Method
Type
Conte
nt
FAST
O
T
S
Early method
FACS
O
A
C
MAX
O
T
S
 All muscles
 Allows for
discovery
Faster
performance
EMG
Obt
A
C
EMFACS
O
T
S
Advantages
Muscular
activity
invisible to
naked eye
Faster
performance.
Disadvantages
Only negative
emotions
-
Only pre
defined
configurations.
Interference
from nearby
muscles
Only certain
emotion
expressions.
O - Observational Obt - Obtrusive A – Anatomical
T – Theoretical S – Selective C – Comprehensive
+
51
FACS examples
07/01/2010
+
52
Why Optical Strain?
07/01/2010
+ Datasets
Dataset
Name
USF –
feigned
USF –
questioning
Canal9
dataset
Found videos
53
Microexp
Approximat
No. of
ressions
e Duration
Sequence
per
per
s
sequenc
sequence(s)
e
Total
Resolution
12
140
8
96
SD / HD
4
65
1
4
HD
6
300-400
4
24
HD
3
30-40
1
4
Very Low
07/01/2010
+
Index
Presentation:
Additional Slides

Minutes

Evolutionary psychology

System Flow

Detailed Flow Chart

Thresholding

Optical Flow

Sample Strain Maps

Elasticity and Strain

Results

FACS

Negative Test Case

OF Vs OS

Conclusion and Future Work

Dataset Details