Computer vision approaches to identifying
people and possible malfeasant behavior.
Dimitris N. Metaxas
Director, Computational Biomedicine
Imaging & Modeling Center
Mark G. Frank
School of Communication,
Information & Library Studies
with special thanks to:
Paul Ekman, UCSF;
Sinuk Kang, Amy Marie Keller, Anastacia Kurylo, Maggie Herbasz,
Belida Uckun, Rutgers;
Jeff Cohn, Pitt; Takeo Kanade, CMU;
Javier Movellan & Marni Bartlett, UCSD.
David Dinges, UPENN
Also thanks to: Office of Naval Research, National Science Foundation
(ITR program), AFOSR, DARPA
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Signals relevant to counter-terror.
•
•
•
•
Identification of bad guys
Changes in gait with loads as small as 1 kg
Anger prior to imminent attack
Fear/distress when lying
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Take a closer look…
Kim Philby, 1960’s
(Frank & Ekman, 2003)
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
How lies are betrayed.
Lie
Cognitive clues
Emotional clues
- Contradictory statements
- Hesitations
-Speech errors
-Reduced illustrators
-Contradictory emblems
-Reduced detail
-Etc.
Lying about feelings
Feelings about lying
-Look for
reliable
signs of
emotion
-Duping delight
-Guilt
-Detection
apprehension
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
The Facial Action Coding System
(FACS)
Ekman & Friesen, 1978
46 Action Units
Action code: 1, 2, 4, 5, 7, 20, 26
1 Inner brow raise
2 Outer brow raise
4 Brow lower
5 Upper lid raise
7 Lid tighten
20 Lip stretch
26 Jaw drop
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Challenges facing behavioral science:
• Advantages:
- reliably identify people & behaviors
- non-obtrusive
- non-inferential, allows for discovery
• Disadvantage:
- laborious
- mistaken identity via cognitive capacity, disguise, etc
• Solution
- automatic computer vision techniques
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Gait recognition
• Identify people from the way they walk
• Important for surveillance and intrusion
detection
• What are good features for identifying a
person?
– i.e., what features are person-specific?
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Background
Sagittal plane - divides body into left and right
halves
Limb segment - a vector between two sites on a
particular limb
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Elevation Angles
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
The trajectories of the
sagittal elevation angles
are invariant across
different subjects.
As a consequence,
person-independent gait
recognition will require
less training data.
(Borghese et al., 1996)
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
The cyclogram
• Elevation angles trace curve in a 4D space
• Curve is called “cyclogram”
• Cyclogram lies in a 2D plane
– Well, almost
• Hypothesis: deviation of cyclogram from
plane is person-specific
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Cyclogram example
Curve is cyclogram
projected into best-fit
plane
Green points are real
points of cyclogram
Red lines trace the
deviation of points from
plane (exaggerated
scale)
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Cyclogram sequence
• Deviation from cyclogram plane can be
represented as a sequence
• e.g., CCCGTTTTATATTTTTAAAAGCCGGTAAATTAGGGG
• Compare sequences between people via
longest common subsequence (LCS)
matching
– Well-known dynamic programming algorithm,
used in computational biology
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Examples of People Detection
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Examples of Gait Analysis
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Face: Tracking: Stress
Recognition
• Identify which Facial Features (space and
time) are important to recognize stress
• Assymetries and Movements around the
mouth and eyebrows
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Slope = Asymmetry
A horizontal line
would indicate no
asymmetry. This
facial expression,
however, is
generally slanted
upward and to the
left.
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
asymmetry
Plots of high and low stress
time
Expression of high stress in form of
asymmetrical facial expression
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
asymmetry
In contrast, low stress
time
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Some more high stress from
different subjects
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Face: Tracking: Stress
Recognition
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Face: Tracking: Stress
Recognition
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Subtle brow changes important.
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Technical Challenges.
• Pose
• Head motion
• Occlusion from glasses, facial hair, rotation,
hands
• Talking
• Video quality
• Frame rate (blinks)
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003
Conclusions.
• There are reliable means to identify people as
well as behaviors associated with deception and
hostile intent.
• We can detect these behaviors.
• We can represent them digitally.
• Can this make us more secure?
M. Frank & D. Metaxas - Rutgers
HMS Symposium 2003