Facial Point Detection
using Boosted Regression
and Graph Models
Authors: Michel Valstar,Brais
Martinez, Xavier Binefa, Maja
Pantic
讲解人: 赵小伟
提纲







作者信息
文章信息
背景知识
拟解决的问题与采用的思路
实现细节
实验
结论
第一作者

Michel Valstar


Research Interest


Automatically recognize facial expressions from face video
Publication


Research associate in Maja Pantic's HCI^2 lab at the
Computing Department of Imperial College London, UK
CVPR’06, CVPR’10
Homepage

http://www.doc.ic.ac.uk/~mvalstar/index.html
第二作者

Brais Martinez


Research Interest




Object Tracking
Facial Feature Detection and Tracking
Thermal Imagery
Publication


PhD Student, Universitat Pompeu Fabra
2 CVPR’10 , PR’08, ICIP’06
Homepage

http://cmtech.upf.edu/?page_id=90
第三作者

Xavier Binefa Valls


Research Interest




Associate Professor, Information Technology and
Telecommunication Department of the Universitat Pompeu
Fabra
Motion Detection and tracking, Machine Learning
Face and Gesture recognition, Digital Libraries
Human computer interaction, Sensor Fusion
Homepage

http://cmtech.upf.edu/?page_id=84
第四作者

Maja Pantic



Research Interest




Imperial College London: Reader in Multimodal HumanComputer Interaction
University of Twente: Professor in
Affective Behavioural Computing
Face and body gesture recognition,
Human-computer interaction (HCI),
Affective computing, Educational software, E-learning tools,
Intelligent systems, Machine learning
HomePage

http://www.doc.ic.ac.uk/~maja/
提纲







作者信息
文章信息
背景知识
拟解决的问题与采用的思路
实现细节
实验
结论
文章信息

文章出处


CVPR 2010
相关文献

[23] D. Vukadinovic and M. Pantic, “Fully automatic facial
feature point detection using gabor feature based boosted
classifiers,” In Proc. Systems, Man and Cybernetics, vol. 2, pp.
1692–1698, 2005.
Abstract



Finding fiducial facial points in any frame of a video showing
rich naturalistic facial behavior is an unsolved problem. Yet this
is a crucial step for geometric-feature-based facial expression
analysis, and methods that use appearance-based features
extracted at fiducial facial point locations.
In this paper we present a method based on a combination of
Support Vector Regression and Markov Random Fields to
drastically reduce the time needed to search for a point’s
location and increase the accuracy and robustness of the
algorithm.
Using Markov Random Fields allows us to constrain the search
space by exploiting the constellations that facial points can
form.
Abstract


The regressors on the other hand learn a mapping between the
appearance of the area surrounding a point and the positions of
these points, which makes detection of the points very fast and
can make the algorithm robust to variations of appearance due
to facial expression and moderate changes in head pose.
The proposed point detection algorithm was tested on 1855
images, the results of which showed we outperform current
state of the art point detectors.
摘要





在具有丰富的自然面部行为的视频帧中进行面部关键特征点的定位是
一个尚未解决的问题。然而，对基于几何特征的面部表情分析以及需
要从面部关键特征点提取表观特征的方法而言，面部关键特征点的定
位是一个很重要的步骤。
本文提出了一种结合SVR和MRF的面部关键特征点定位方法。该方法
大大降低了搜索特征点的时间，并且提高了算法的精度和鲁棒性。
一方面，使用MRF对面部关键特征点的分布进行建模，以此来限制特
征点的搜索范围。
另一方面，通过SVR学习到了特征点周围区域的表观信息与特征点位
置的映射关系。该方法可以更快的检测特征点，并且对由面部表情和
头部姿态的适度变化引起的表观变化比较鲁棒。
我们在1855幅图像上测试了提出的面部特征点检测算法，实验表明，
本文的算法超越了当前state-of-the-art的算法。
提纲







作者信息
文章信息
背景知识
拟解决的问题与采用的思路
实现细节
实验
结论
AdaBoost-based Facial
Landmark Localization
Preparing Samples
Negative Samples
Face Detection &
Normalization
Candidate Points
Search Region
Determination
Candidate Points
Fusion
...
...
...
Multi-Scales
Detection
Real AdaBoost
Classisiers
...
Real AdaBoost Learning
Positive Samples
Feature Extraction
提纲







作者信息
文章信息
背景知识
拟解决的问题与采用的思路
实现细节
实验
结论
拟解决的问题



None but [23] is able to detect all 20 facial points necessary for
automatic expression recognition
No previous work has reported to be able to robustly handle
large occlusions such as glasses, beards, and hair that covers
part of the eyebrows and eyes
None have reported to detect facial points robustly in the
presentence of facial expressions
22 fiducial facial feature points (including pupils)
本文的主要思想


Iteratively using Support Vector Regression and local
appearance based features to provide an initial predictions of
22 points
Then, the Markov Network is applied to ensure the new
locations predicted by SVR regressors form correct point
constellations
SVR regression
The output of the SVRs to detect an pupil
MRF points model
文章结构


Introduction
BoRMaN point detection







A priori probability
Regression prediction
Spatial relations
Point detection algorithm
Local appearance based features and AdaBoost feature selection
Experiments
Conclusions and future work
提纲







作者信息
文章信息
背景知识
拟解决的问题与采用的思路
实现细节
实验
结论
实现细节





A priori probability
Regression prediction
Local appearance based features and
AdaBoost feature selection
Spatial Relations
Point detection algorithm
Regression prediction



The localization problem is formulated as finding the vector v
that relates a patch location L to the target point T.
This problem is decomposed into two separate regression
problem
 Regressor R is tasked with finding the angle  of v
 Regressor R is tasked with finding the length  of v
As we can see, the regressors give a good yet not a perfect
indication of where the target point is.

Note that although the location of the pupil is a global minimum,
the predicted distance at that location is not zero.
Regression prediction
Such errors can be removed
by using a iterative procedure.

The error of the estimates


Impression of the regressors output
Great errors which are not merely impressions
Spatial restrictions on the location
of each facial point depending on
the other facial points are applied
to solve this problem.
The output of the SVRs to detect an pupil
实现细节





A priori probability
Regression prediction
Local appearance based features and
AdaBoost feature selection
Spatial Relations
Point detection algorithm
Local appearance based features
and AdaBoost feature selection

Haar-like filters are adopted as the descriptors of local
appearance

The reason for this is




Show that the success of the proposed approach is due to the idea of
tuning the point detection problem from a classification procedure
into a regression procedure, and not due to asome highly descriptive
appearance feature
Exploring the integral image
The regression performance decrease when the dimensionality
of the training set is too large
AdaBoost is used to select features
实现细节





A priori probability
Regression prediction
Local appearance based features and
AdaBoost feature selection
Spatial Relations
Point detection algorithm
Spatial Relations


Each relative position of a pair of points {i, j} is a vector ri , j
pointing from one facial point to another
The relation between two vectors ri , j and rk ,l is described by two
parameters
 The relation between their angles R   i , j   k ,l

 The relation between their lengths R  i , j 

k ,l
(0,0)
Relation between two vectors
Spatial Relations

Variables such as R and R are modeled as a Sigmoid
function. If a variable takes its value in [m , m ] , then
S ( x)  Psigm (min( x  m ,  x  m )), whereS (m )  S (m )  0.5
Illustration of Sigmoid function, cited from Wiki
Spatial Relations



Once the pairwise relations are defined, the joint probability of
a configuration is modeled by a Markov Random Field.
The nodes correspond to each of the relative positions ri , j
Relation between ri , j  ( i , j , i , j ) and rk ,l  (k ,l , k ,l ) is modeled
as
S ang ( i , j ,  k ,l )  Sdist ( i , j ,  k ,l )
实现细节





A priori probability
Regression prediction
Local appearance based features and
AdaBoost feature selection
Spatial Relations
Point detection algorithm
Point detection algorithm

Flow of algorithm
提纲







作者信息
文章信息
背景知识
拟解决的问题与采用的思路
实现细节
实验
结论
本文方法与已有方法的对比
Distance Metric:
实验结果
提纲







作者信息
文章信息
背景知识
拟解决的问题与采用的思路
实现细节
实验
结论
本文可以借鉴的地方



Regression instead of classification
Markov Random Field to model the
constellation of facial points
Select features by AdaBoost
谢谢！
附录
Introduction of AdaBoost(1/6)

AdaBoost

AdaBoost通过对一些弱分类器(weak classifier)的组合
来形成一个强分类器(strong classifier), “提升(boost)”弱
分类器得到一个分类性能好的强分类器

每一个弱分类器都对前一个分类器错误分类的样本给与
更多的重视
Introduction of AdaBoost(2/6)

AdaBoost 弱分类器

其中，h表示弱分类器的响应值，θ为正例反例判别
阈值，f表示特征响应值
Negative
1, if f j (x)   j
h j ( x)  
 0, otherwise
Positive
Introduction of AdaBoost(3/6)

AdaBoost训练过程

输入



样本集合 (x1,y1), (x2,y2), ..., (xn,yn)
训练参数：样本权值wi、分类器层数T等等
输出

一个由很多弱分类器线性组合得到的强分类器
Introduction of AdaBoost(4/6)
是
分类错误率是否达到？
否
遍历所有特征，分别计算以每个特征
作为弱分类器的分类错误率
选择错误率最小的弱分类器
更新强分类器
样本权值更新，分类正确的样本权值减小
输出强分类器
Introduction of AdaBoost(5/6)

AdaBoost训练过程

For t=1,...,T

1. 归一化权重，使得wt为一个概率分布：
wt ,i 
wt ,i
n
w
j 1

t, j
2. 对每个特征j, 训练一个弱分类器hj, 计算其带权重的错误率
n
 j   wt , i | hj ( xi )  yi |
i 1

3. 选择误差最小的弱分类器ht加入强分类器

4. 更新每个样本的权重
wt ,i  wt ,i  t
1-ei
t
, t 
1 t
Introduction of AdaBoost(6/6)

AdaBoost强分类器

1
H ( x)  
0

T
if
T
 log h ( x)  0.5 log
t 1
t t
Otherwise
t 1
t
Haar-like Feature(1/2)

Haar-like feature

白色矩形像素和减去黑色矩形像素和
Haar-like Feature(2/2)

Haar-like feature


计算矩形内部像素灰度值的和
定义积分图 ii( x, y )   i( x, y )
x x , y  y

计算D内部像素灰度和

4+1-2-3