AAM based Face Tracking with Temporal Matching and Face Segmentation Dalong Du Outline • • • • • Author Introduction AAM Introduction Abstract Method and Theory Experiment Author Introduction • Mingcai Zhou – Institute of Automation Chinese Academy of Sciences • Lin Liang – Microsoft Research Asia – Author Introduction • Jian Sun – Microsoft Research Asia • joined in July, 2003. – Educational background • BS degree, MS degree and Ph.D degree from Xian Jiaotong University in 1997, 2000 and 2003 – Current research interests • Interactive compute vision (user interface + vision) • Internet compute vision (large image collection + vision) • stereo matching and computational photography Author Introduction • Yangsheng Wang – Director of Digital Interactive Media Lab, Institute of Automation Chinese Academy of Sciences – Educational background • BS degree, MS degree and Ph.D degree from Huazhong University of Science and Technology AAM Introduction • Shape Model • Appearance (Texture) Model • AAM Model Search AAM—Shape Model • Face Q consists of N landmark points – x = (x1,y1, … , xn, yn)T – The geometry information of Q decouples into two parts: • A shape S – Shape is the geometric information invariant to a particular class of transformations – e.g. x u b Or other linear or nonlinear methods b • A transformation – θ – e.g. similarity s, R, t Or Affine or others. – Similarity » Same shape Different shape θ AAM—Shape Model • Shape Model Building – Given a set of shapes – Align shapes into common frame • Procrustes analysis – Estimate shape distribution p(x) • Use PCA The aligned shapes AAM—Shape Model • Shape Model Building, continued – Given aligned shapes, {x i} – Apply PCA • Compute mean and eigenvectors of covar. x x Pb – P – First t eigenvectors of covar. matrix – b – Shape model parameters AAM—Texture Model • Building Texture Models – For each example, extract texture vector Warp to mean shape Texture, g – Normalise vectors (as for eigenfaces) – Build eigen-model g g Pg b g 2 1 b1 2 1 2 2 b2 2 2 AAM—Texture Model • Warp method Warpedpoints: ( xi ' , yi ' ) Controlpoints: ( xi , yi ) c c' x a a' b x a b c x' b' x' a' b'c' 1 x is inside t he t riangleif 0 α 1 and 0 β 1 AAM—Texture Model • Warp method, continued c (c a) a x a (b a) (c a) (1 )a b c a b c x (b a) x a b c 1 b x ax y ay 1 1 bx by 1 c x c y 1 AAM—Model Search • Find the optimal shape parameters p and appearance parameters to minimize the difference between the warped-back appearance I (W ( p)) and synthesized appearance Computed by the inverse Compositional parameter Update technique W ( x, p) map every pixel x in the model coordinate to its corresponding image point I (W ( x, p)) W ( x, p) s0 Abstract • Problems – Generalization problem – images with cluttered background • How to do? – A temporal matching constraint in AAM fitting • Enforce an inter-frame local appearance constraint between frames – Introduce color-based face segmentation as a soft constraint Method and Theory • Extend basic AAM to Multi-band AAM – The texture(appearance) is a concatenation of three texture band values • The intensity (b) • X-direction gradient strength (c) • Y-direction gradient strength (d) Method and Theory • Temporal Matching Constraint – Select feature points with salient local appearances at previous frame – Optimize the shape parameters to match the local appearances at current frame Method and Theory • Temporal Matching Constraint, continued – t : a set of feature points • Selected by a corner detector and some semantic points – At 1 : the face appearance of frame t-1 – R j : the local patch corresponding to the j-th feature point Normalize the illuminations of two patches – : the average intensity of j-th patches of frame t-1 and t respectively Method and Theory • Temporal Matching Constraint, continued – Add a new term to the AAM cost function Can be efficiently minimized based on inverse compositional algorithm • Empirically, Method and Theory • Temporal Matching Constraint, continued – Be resistant to global illumination changes • Match local patches – Do not suffer from the mismatched points • Feature matching is continuously refined by updating the shape parameters during AAM fitting Method and Theory • Initialize shape – Good initial parameters -> good AAM fitting – Method • Selected feature points at frame t-1 • Matched feature points at frame t • Remaining feature points after main direction filter Method and Theory • Initialize shape, continued zi i 1 M – M matched points – Estimate the initial shape parameters p0 Gauss-Newton algorithm • represents the consistency of feature points I’s direction • is the estimated position of the point I given the shape parameters p • are the vertex coordinate of the triangle • are the triangle coordinate Method and Theory • Face Segmentation Constrained AAM – Problem: AAM tends to fit the face outline to the background edges – Method: segment the face region using an adaptive color model and constrain AAM fitting Method and Theory • Formalization Wc = 0.01 – Where {xk } are the locations of the selected outline points in the model coordinate Experiments • RI: robust initialization • TO: temporal matching constraint • FS: face segmentation Experiments Thank you