AAM based Face Tracking with
Temporal Matching and Face
Segmentation
Dalong Du
Outline
•
•
•
•
•
Author Introduction
AAM Introduction
Abstract
Method and Theory
Experiment
Author Introduction
• Mingcai Zhou
– Institute of Automation Chinese Academy of
Sciences
• Lin Liang
– Microsoft Research Asia
–
Author Introduction
• Jian Sun
– Microsoft Research Asia
• joined in July, 2003.
– Educational background
• BS degree, MS degree and Ph.D degree from Xian
Jiaotong University in 1997, 2000 and 2003
– Current research interests
• Interactive compute vision (user interface + vision)
• Internet compute vision (large image collection + vision)
• stereo matching and computational photography
Author Introduction
• Yangsheng Wang
– Director of Digital Interactive
Media Lab, Institute of Automation
Chinese Academy of Sciences
– Educational background
• BS degree, MS degree and Ph.D degree from Huazhong
University of Science and Technology
AAM Introduction
• Shape Model
• Appearance (Texture) Model
• AAM Model Search
AAM—Shape Model
• Face Q consists of N landmark points
– x = (x1,y1, … , xn, yn)T
– The geometry information of Q decouples into two parts:
• A shape S
– Shape is the geometric information
invariant to a particular class of
transformations
– e.g. x  u   b Or other
linear or nonlinear methods
b
• A transformation
– θ
– e.g. similarity s, R, t Or
Affine or others.
– Similarity
»
Same shape
Different shape
θ
AAM—Shape Model
• Shape Model Building
– Given a set of shapes
– Align shapes into common frame
• Procrustes analysis
– Estimate shape distribution p(x)
• Use PCA
The aligned shapes
AAM—Shape Model
• Shape Model Building, continued
– Given aligned shapes, {x i}
– Apply PCA
• Compute mean and eigenvectors of covar.
x  x  Pb
– P – First t eigenvectors of covar. matrix
– b – Shape model parameters
AAM—Texture Model
• Building Texture Models
– For each example, extract texture vector
Warp to
mean
shape
Texture, g
– Normalise vectors (as for eigenfaces)
– Build eigen-model g  g  Pg b g
 2 1
b1
2 1
 2 2
b2
2 2
AAM—Texture Model
• Warp method
Warpedpoints: ( xi ' , yi ' )
Controlpoints: ( xi , yi )
c
c'
x
a
a'
b
x  a  b  c
x'
b'
x'  a' b'c'       1
x is inside t he t riangleif
0  α  1 and 0  β  1
AAM—Texture Model
• Warp method, continued
c
 (c  a)
a
x  a   (b  a)   (c  a)
 (1     )a  b  c
 a  b  c
x
 (b  a)
x  a  b  c
    1
b
 x   ax
  
 y    ay
1  1
  
bx
by
1
c x   
 
c y   
1   
AAM—Model Search
• Find the optimal shape parameters p and appearance
parameters  to minimize the difference between the
warped-back appearance I (W ( p)) and synthesized
appearance
Computed by the inverse
Compositional parameter
Update technique
W ( x, p) map every pixel x in the model coordinate to its corresponding image point
I (W ( x, p))
W ( x, p)
s0
Abstract
• Problems
– Generalization problem
– images with cluttered background
• How to do?
– A temporal matching constraint in AAM fitting
• Enforce an inter-frame local appearance constraint
between frames
– Introduce color-based face segmentation as a soft
constraint
Method and Theory
• Extend basic AAM to Multi-band AAM
– The texture(appearance) is a concatenation of three
texture band values
• The intensity (b)
• X-direction gradient strength (c)
• Y-direction gradient strength (d)
Method and Theory
• Temporal Matching Constraint
– Select feature points with salient local appearances at
previous frame
– Optimize the shape parameters to match the local
appearances at current frame
Method and Theory
• Temporal Matching Constraint, continued
–
t
: a set of feature points
• Selected by a corner detector and some semantic points
– At 1 : the face appearance of frame t-1
– R j : the local patch corresponding to the j-th feature
point
Normalize the illuminations of two patches
–
: the average intensity of j-th patches of
frame t-1 and t respectively
Method and Theory
• Temporal Matching Constraint, continued
– Add a new term to the AAM cost function
Can be efficiently minimized based
on inverse compositional algorithm
• Empirically,
Method and Theory
• Temporal Matching Constraint, continued
– Be resistant to global illumination changes
• Match local patches
– Do not suffer from the mismatched points
• Feature matching is continuously refined by updating
the shape parameters during AAM fitting
Method and Theory
• Initialize shape
– Good initial parameters -> good AAM fitting
– Method
• Selected feature points at frame t-1
• Matched feature points at frame t
• Remaining feature points after main direction filter
Method and Theory
• Initialize shape, continued
 zi i 1
M
– M matched points
– Estimate the initial shape parameters
p0
Gauss-Newton
algorithm
•
represents the consistency of feature
points I’s direction
•
is the estimated position of the point I given
the shape parameters p
•
are the vertex coordinate of the triangle
•
are the triangle coordinate
Method and Theory
• Face Segmentation Constrained AAM
– Problem: AAM tends to fit the face outline to the
background edges
– Method: segment the face region using an
adaptive color model and constrain AAM fitting
Method and Theory
• Formalization
Wc = 0.01
– Where {xk } are the locations of the selected
outline points in the model coordinate
Experiments
• RI: robust initialization
• TO: temporal matching constraint
• FS: face segmentation
Experiments
Thank you