Automatic Face Recognition under
Component-Based Manifolds
CVGIP 2006
Wen-Sheng Chu (朱文生) and Jenn-Jier James Lien (連震杰)
Robotics Lab. CSIE NCKU
Motivation
• Face recognition is hard due to several image
variations:
illumination
pose
expression
Objective
• Recognize faces using multiple face patterns
rather than a single one.
Person A
Person B
Single input pattern
Person A
Person B
Multiple input patterns
Automatic Acquisition of Facial Components
Training
Data of
Features
2-Class
SVM
Cropped Face I Classifiers
Original Image
Face
Detection
Feature Point
Detection
Facial
Components
Extraction
Extracted Facial
Components
Rejected Non-face
Detected
Features
Face +ve
Removal
Registration by
Affine Warping
Band-pass
Filtering
Normalized
Illumination IB
P. Viola and M. Jones, “Robust Real-Time Face Detection”, IJCV 2004.
Normalized
Pose IR
Automatic Acquisition of Facial Components
Training
Data of
Features
2-Class
SVM
Cropped Face I Classifiers
Original Image
Face
Detection
Feature Point
Detection
Facial
Components
Extraction
Extracted Facial
Components
Rejected Non-face
Detected
Features
Face +ve
Removal
Registration by
Affine Warping
Band-pass
Filtering
Normalized
Illumination IB
Normalized
Pose IR
Facial Feature Detector
• 2-class SVM with feature vector v:
 vA
v  
 vG



v A ( Ny  x )  I ( x , y )
v G ( Ny  x )   I ( x , y )
o
o
o
o
• Reject false positives
x
x
x
Automatic Acquisition of Facial Components
Training
Data of
Features
2-Class
SVM
Cropped Face I Classifiers
Original Image
Face
Detection
Feature Point
Detection
Facial
Components
Extraction
Extracted Facial
Components
Rejected Non-face
Detected
Features
Face +ve
Removal
Registration by
Affine Warping
Band-pass
Filtering
Normalized
Illumination IB
Normalized
Pose IR
Registration & Illumination Normalization
I
Registration
Affine warping
IR
Illumination
Normalization
Band-pass filtering I  I  G
B
R
  0 .5  I R  G   8
IB
Automatic Acquisition of Facial Components
Training
Data of
Features
2-Class
SVM
Cropped Face I Classifiers
Original Image
Face
Detection
Feature Point
Detection
Facial
Components
Extraction
Extracted Facial
Components
Rejected Non-face
Detected
Features
Face +ve
Removal
Registration by
Affine Warping
Band-pass
Filtering
Normalized
Illumination IB
Normalized
Pose IR
Facial Components Extraction
• Effects of pose and illumination are smaller in
each local region compared with those in the
holistic face image.
T. K. Kim, H. Kim, W. Hwang and J. Kittler, “Independent Component Analysis in A Local Facial
Residue Space for Face Recognition”, PR, 2004.
Constrained Mutual Subspace Method
(CMSM)
• Similarity between i and j == θc
• Use the variation of dissimilarity between subjects
subspace j
subspace i
project
θ
project
constrained
subspace
ic
θc
jc
K. Fukui and O. Yamaguchi, “Face Recognition Using Multi-viewpoint Patterns for Robot
Vision”, ISRR 2003.
Constrained Subspace Generation
• Take nose for explanation:
n o se
Ci
nose
1
(P
 1/ N
P
nose
2

N
k 1
xkxk
 ...  P
nose
L
T
 Bi
n o se
i
n o se
)w   w, P i
n o se
T
Bi
nose
 Bi
nose
nose
T
Bi
The eigenvectors, w, selected in ascending order, are
the basis of the constrained subspace, Snose.
Constraint subspace basis
PCA basis
Projection onto Constrained Subspace
nose
Bi
B
nose
j
Snose
1. Projection
basis vectors  constrained subspace Snose
2. Normalization
length(projected vector)  1
3. Orthogonalization
applying Gram-Schmidt process to orthogonalize the
normalized vectors
Comparison between Normalized
Manifolds
• The similarity of nose between subject i and
subject j:
similarity ( B Si , B Sj )  1 / t
nose
where
matrix
i
nose

t
i 1
cos  i
2
nose
n o se
are defined as the eigenvalues of
n o se
n o se
B Si B Si
T
n o se
n o se
B Sj B Sj
T
n o se
n o se
B Si B Si
T

.
NN
• Similarity(i, j) == summing up the five canonical
correlations
Experiment Setup
#individuals
16
#sequence/individual
5
#second/sequence
10
#frame/second
10
Resolution
320x240
Size of Registration Template
100x125
Eye-braw
Size of Facial Eyes
Components Nose
Mouth
40x15
28x18
44x21
28x40
Typical Samples in 3D Principal Component
Space – Holistic Image
subject 1 (․)
subject 2 (․)
subject 3 (․)
subject 4 (․)
Original v.s. Projected Subspaces
– Eye-braw
Original v.s. Projected Subspaces
– Left Eye
Original v.s. Projected Subspaces
– Right Eye
Original v.s. Projected Subspaces
– Nose
Original v.s. Projected Subspaces
– Mouth
Comparison
Methods
Recognition
Rate (%)
NNfacial
component
82.3
MSMfacial
component
90.8
CMSMholistic
face
CMSM-facial
component
95.2
98.6
End
F&Q and thanks!