Enforcing Convexity for
Improved Alignment with
Constrained Local Models
Authors: Yang Wang,
Simon Lucey,
Jeffrey F. Cohn
讲解人: 赵小伟
文章出处:CVPR’08
提纲







作者信息
文章信息
背景知识/(Constrained Local Models)
拟解决的问题与采用的思路
实验
结论
Demos
第一作者

Yang Wang





Research Areas


Computer Vision, Graphics, Medical Image Analysis, Biometrics, Machine
Learning, Computer Animation, and Augmented Reality
Publication


Research scientist at Siemens
Robotics Institute, Carnegie Mellon University
Ph.D. in Computer Science, August 2000 - December 2006
Stony Brook University, New York, USA
B.S. in Computer Science, September 1993 - July 1998
Tsinghua University, Beijing, China
PAMI(07, 09) , IJCV(08), IVC(08, 09, 10), ECCV(02, 08), CVPR(04, 06, 07,
08, 10), ICCV(05, 07, 09), FG(08)
Homepage

http://www.cs.cmu.edu/~wangy/
第二作者

Simon Lucey


Research Interest


I am passionate about gaining a deeper understanding of
perception, learning and intelligence. My practical interests are in
analyzing faces, biometrics and human event recognition. From an
academic perspective I am extremely interested in computer
vision, machine learning and how these evolving topics relate to
deeper questions concerning Artificial Intelligence (AI).
Publication


PhD Student, Universitat Pompeu Fabra
PAMI’10 , IVC’10, IJCV’08, PRL’07, Multimedia’05, ICCV, CVPR
Homepage

http://www.cs.cmu.edu/~slucey/Main.html
第三作者

Jeffrey F. Cohn


Research Interest


He has led interdisciplinary and inter-institutional efforts to develop
advanced methods of automatic analysis of facial expression and prosody;
and applied those tools to research in human emotion, social
development, non-verbal communication, psychopathology, and
biomedicine.
Database




Jeffrey Cohn is Professor of Psychology at the University of Pittsburgh and
Adjunct Faculty at the Robotics Institute at Carnegie Mellon University.
Cohn-Kanade AU-Coded Facial Expression Database. CK
Cohn-Kanade Expanded. CK+
CMU MultiPie.
Homepage

http://www.pitt.edu/~jeffcohn/
提纲







作者信息
文章信息
背景知识/(Constrained Local Models)
拟解决的问题与采用的思路
实验
结论
Demos
文章信息

文章出处


CVPR 2008
相关文献



[7] D. Cristinacce and T.F. Cootes. Feature detection and tracking with
constrained local models. In BMVC, pages 929-938, 2006
[3] S. Baker and I. Matthews. Lucas-Kanade 20 years on: A unifying
framework: Part 1: The quantity approximated, the warp update rule, and
the gradient descent approximation. IJCV, 2004.
[19] Y. Wang, S. Lucey, and J. Cohn. Non-rigid object alignment with a
mismatch template based on exhaustive local search. In IEEE Workshop
on Non-rigid Registration and Tracking through Learning, 2007.
Abstract


Constrained local models (CLMs) have recently demonstrated
good performance in non-rigid object alignment/tracking in
comparison to leading holistic approaches (e.g., AAMs). A
major problem hindering the development of CLMs further, for
non-rigid object alignment/tracking, is how to jointly optimize
the global warp update across all local search responses.
Previous methods have either used general purpose optimizers
(e.g., simplex methods) or graph based optimization
techniques. Unfortunately, problems exist with both these
approaches when applied to CLMs.
In this paper, we propose a new approach for optimizing the
global warp update in an efficient manner by enforcing
convexity at each local patch response surface.
Abstract


Furthermore, we show that the classic Lucas-Kanade approach
to gradient descent image alignment can be viewed as a special
case of our proposed framework.
Finally, we demonstrate that our approach receives improved
performance for the task of non-rigid face alignment/tracking
on the MultiPIE database and the UNBC-McMaster archive.
摘要



与基于全局的方法相比(例如AAM)，带有局部约束的模型(CLMs:
Constrained Local Models)在非刚性物体的对齐和跟踪方面展示了更
好的性能。对于非刚性物体的对齐和跟踪，一个主要的阻碍CLMs进
一步发展的问题是:“如何根据局部搜索的响应，对全局形变的更新参
数(Global warp update)进行联合优化？”之前的方法要么采用
general的优化方式(例如单纯形法),要么采用基于图的优化技术。不幸
的是，当应用于CLMs时，这些方法都存在问题。
本文提出了一种新的方法，强制每个局部patch的响应曲面为凸，这
样就可以以一种高效的方式对全局形状更新进行优化。进一步，我们
证明经典的基于Lucas-Kanade方法进行梯度下降的图像对齐可以看做
本文提出的框架的一个特例。
最后，在非刚性的人脸对齐和跟踪方面，我们的方法在Multi-PIE和
UNBC-McMaster数据库上取得了更好的性能。
提纲







作者信息
文章信息
背景知识/(Constrained Local Models)
拟解决的问题与采用的思路
实验
结论
Demos
Overview of Constrained Local
Models
(i) an exhaustive local search for feature locations to get the response maps
{ p(li  aligned | I , x)}in1
(ii) an optimization strategy to maximize the responses of the PDM
constrained landmarks.
Saragih, J.M.; Lucey, S.; Cohn, J.F.; , “Face alignment through subspace constrained mean-shifts,”
ICCV, 2009, pp.1034-1041
Key Steps of CLMs




Estimating patch/region experts
Obtaining local responses
Estimating PDM(point distribution model)
Constrained local model fitting
Key Steps of CLMs




Estimating patch/region experts
Obtaining local responses
Estimating PDM(point distribution model)
Constrained local model fitting
Estimating patch experts

Arbitrary classifier can be employed to learn patch experts
within a CLM framework



boosting schemes (e.g., AdaBoost, GentleBoost, etc.)
relevance vector machine (RVMs)
A linear SVM classifier was chosen, due to

computational advantages
Key Steps of CLMs




Estimating patch/region experts
Obtaining local responses
Estimating PDM(point distribution model)
Constrained local model fitting
Obtain local responses
(a) is the source
image to be
aligned, while the
black box stand
for the search
window (25*25),
the red cross
illustrate the
ground truth
alignment.
P ( y  1| f ( x)) 
(b) shows the local search responses
using patch experts trained by 125
positive examples and 15k negative
examples.
(b) shows the local search responses
using patch experts trained by 125
positive examples and 8k negative
examples.
1
1  e a f ( x ) b
where f( x) is the match-score
(d) and (e) show the
for the
patch-export
estimated
logistic regression
weight values of (b) and (c),
respectively.
Key Steps of CLMs




Estimating patch/region experts
Obtaining local responses
Estimating PDM(point distribution model)
Constrained local model fitting
Estimating PDM



A point distribution model (PDM) is used for a parametric
representation of the non-rigid shape variation in the
CLM.
The non-rigid warp function can be described as
, where
, p is a parametric vector describing the
non-rigid warp, andV is the matrix of concatenated
eigenvectors. N is the number of patch-experts.
Principal component analysis (PCA) is then employed to
obtain shape eigenvectors V that preserved 95% of the
similarity normalized shape variation in the train set.
Key Steps of CLMs




Estimating patch/region experts
Obtaining local responses
Estimating PDM(point distribution model)
Constrained local model fitting
CLM fitting

Based on the patch experts, non-rigid alignment as be posed as the
following optimization problem:
arg min  Ek {Y ( xk  Vk p)}
p



k
where Ek () is the inverted classifier score function obtained from
applying the k th patch expert to the source image patch intensity Yk ( xk  xk )
The displacement x is constrained to be consistent with the PDM
The matrix V can be decomposed into submatrices Vk for each patch
expert, i.e.
V  [V1T ,...,VNT ]T
提纲







作者信息
文章信息
背景知识/(Constrained Local Models)
拟解决的问题与采用的思路
实验
结论
Demos
拟解决的问题

How to jointly optimize global warp update across all local
search responses?
arg min  Ek {Y ( xk  Vk p)}
p

k
In general, it is difficult to solve for p, as Ek () is a discrete
function due to x only taking on integral values and there is no
guarantee for Ek () being convex.
A Sub-optimal Approach

Exhaustive Local Search (ELS)

Instead of optimizing for the holistic warp update p directly, ELS
optimizes for N local translation updates by exhaustively searching local
regions of the object
xk  arg minEk {Y ( xk  x)}
x

Where xk is the local warp update displacement of the kth region/patch
(k=1,…N) within a local search region. Then
p  (VWV )1VW z

Where V is the matrix of concatenated eigenvectors. W is weighting
matrix,
w  diag{wx1 , wy1 , , wxN , wyN }
Y. Wang, S. Lucey, and J. Cohn., Non-rigid object alignment with a mismatch template based on
exhaustive local search, In IEEE Workshop on Non-rigid Registration and Tracking through
learning, 2007.
本文的解决思路
Learning from Lucas-Kanade
(1/3)

Let us assume that we are attempting to solve for N local
translation updates as in the following equation
xk  arg minEk {Y ( xk  x)}
x

When a sum of squared differences (SSD) error function is
employed:
xk  arg min T ( xk )  Y ( xk  x)
2
x
where T is an arbitrary defined template. We no longer have to
exhaustively search a local region around xk .
本文的解决思路
Learning from Lucas-Kanade(2/3)

Equation
xk  arg min T ( xk )  Y ( xk  x)
2
x
can be rewritten by employing a first Taylor series
approximation at Y ( xk ) .
xk  arg min D( xk )  GT ( xk )x
2
x
which can be expressed generically in the form of a quadratic,
given,
xT Ak x  2bkT x  ck
Ak  G( xk )GT ( xk ), bk  G( xk ) D( xk ), ck  DT ( xk ) D( xk )
where D( xk )  T ( xk )  Y ( xk ),and G( xk )is the2  P 2 localgradient
matrix
Y ( xk )
for eachset of P 2 intensitycenteredaround xk .
xk
本文的解决思路
Learning from Lucas-Kanade(3/3)

Since Ak is virtually always guaranteed of being positive
definite, this implies the quadratic is convex, and has a unique
minima.
Since the summation of N convex functions is still a convex
p for the local translation
function, it is possible to solve not only
undates but the entire warp update
explicitly,
T
p  (VAV )Vb
T
T T
 [bmatrix
,
,
b
where V isbthe
1
N ]of concatenated eigenvectors describing
the PDM,
and the
matrix
A has the form
A
0
 1
A

 0



AN 
本文的解决思路
Generic Convex Quadratic Curve
Fitting

When Ek () is not a SSD classifier, but any function that gives a
low value for correct alignment,
arg min Ak ,bk ,ck

subject to Ak
0
Ek (x)  x Ak x  2b x  ck
T
x
T
k
2
where Ek (x)  Ek {Y ( xk  x)}.

For 2D image alignment, the problem can be further simplified
as
where
本文的解决思路
Generic Convex Quadratic Curve
Fitting


The above optimization is a quadratically constrained quadratic
program (QCQP) and in general costly to be solved directly.
So, Ak is enforced to be a diagonal matrix with non-negative
diagonal elements. More specially,
 a11
Ak  
 0

0 
, where a11 , a22  0

a22 
So,
Convex quadratic
fitting (CQF), which
can be solved efficiently.
本文的解决思路
Generic Convex Quadratic Curve
Fitting

Algorithm outline
进一步的改进

Robust error function

In particular, the robust error function can be defined as
提纲







作者信息
文章信息
背景知识/(Constrained Local Models)
拟解决的问题与采用的思路
实验
结论
Demos
Example Fits
Examples of fitting local search responses:
(a) is the local search responses in Figure 1
(d) using patch experts trained by a linear
support vector machine (SVM).
(b-d) show the surface fitting results. More
specifically,
(b) picks the local displacement with the
minimum response value in the search
window, while (c) and (d) fit the local search
response surface by a quadratic kernel
in Equation 15 and a quadratic kernel with a
robust error function in Equation 16,
respectively. The brighter intensity means
the smaller matching error between the
template and the source image patch. In
each search window, the red cross illustrates
the ground truth location. As we can see, in
most cases, the above three methods can all
achieve good performance, while the
proposed convex quadratic fitting (CQF) (c)
and the robust convex quadratic fitting
(RCQF) (d) methods are less sensitive to
local minima than the exhaustive local
search (ELS) method (b).
提纲







作者信息
文章信息
背景知识/(Constrained Local Models)
拟解决的问题与采用的思路
实验
结论
Demos
本文方法与已有方法的对比
本文方法与已有方法的对比
提纲







作者信息
文章信息
背景知识/(Constrained Local Models)
拟解决的问题与采用的思路
实验
结论
Demos
本文可以借鉴的地方

Formulation
Demos

CLMs Demo:
http://web.mac.com/jsaragih/iWeb/FaceTrac
ker/FaceTracker.html
谢谢！
附录
ASM(1/7)

一个物体的几何描述分为两部分：


相似变换（旋转、缩放、平移）
形状
ASM(2/7)

ASM的任务：


得到姿态参数
得到形状的低维表示，即参数b


x  u  b
ASM匹配的基本过程：

搜索


在马氏距离下搜索与相应灰度梯度分布模型最匹配的特征点
调整

对搜索得到的形状进行调整，以确保获得的形状是可用的
ASM(3/7)

ASM模型：


要完成ASM搜索与匹配的过程，必须要有相应的统计模型做
支撑
ASM模型分为：


每个标注点的灰度梯度分布模型
点分布模型
ASM(4/7)

每个标注点的灰度梯度模型的构建

取标注点 i,1  i  N的profile，并计算得到该profile的归一化的
灰度梯度向量



Profile灰度采样：(v1, v2 ..., vPr ofileLength )
梯度 gi  (v2  v1, v3  v2 ..., vPr ofileLength  vPr ofileLength1 )
g
gi'  Pr ofileLength 1
归一化

j 1

| gij |
若训练集中有M个形状，每个形状有N个标注点，那么对于
i,1  i  N，有协方差矩阵
每个标注点
M M
1
VARi 
( gij  gavgi )( gik  gavgi ，这N个协方差矩阵就构成
)

M  1 j 1 k 1
了灰度梯度模型
ASM(5/7)
•
点分布模型的构建


对于单个形状，每个标注点的坐标为 ( xi , yi ),1  i  N
将所有标注点的坐标串接起来，就得到一个形状向量
( x1 , y1 ,..., xN , yN )T

若训练集中有M个形状，那么我们就有M个形状向量
M
( x , y ,..., x
1

1
N
, yN )Tj 
j 1
于是就可以训练形做PCA，得到能描述
训练集形状变化的特征值与特征向量，
即得到了点分布模型
ASM(6/7)

ASM搜索


对沿标注点i,1  i  N 的profile线的每个像素点 j 取梯度向量 gij
gij  gavgi )VAR1 ( gij  gavgi )
搜索点为：arg min(
j
ASM(7/7)

ASM调整

假定经过一步搜索之后得到形状


将形状进行PCA投影得到参数b


S  ( x1, y1,..., xN , yN )T
b  T (S  Smean )
利用b重新计算形状

Snew  Smean  b