Face Recognition
Image Understanding
Xuejin Chen
Face Recogntion
• Good websites
– http://www.face-rec.org/
• Eigenface [Turk & Pentland]
Image Understanding, Xuejin Chen
Eigenface
• Projecting a new image into the subspace
spanned by the eigenfaces (face space)
• Classifying the face by comparing its position
in face space with the positions of known
individuals
Image Understanding, Xuejin Chen
• Face image used
as the training set
Image Understanding, Xuejin Chen
• Average face
• 7 eigenfaces
calculated from the
training images
Image Understanding, Xuejin Chen
Calculating Eigenfaces
N 2 -D vector
• Face image I ( x, y ) : N  N intensity matrix
• PCA: to find the vectors that best account for the distribution
of face images within the entire image space
– Training images 1 , 2 , 3 ,...,  M
– Average face Ψ  1  nM1  n
M
– Each face differs from the average by Φi  i  Ψ
– Seeks a set of M orthogonal vectors u n that best describe the
distribution of the data
• The kth vector is chosen such that
1
k 
M
 u Φ 
M
n 1
T
k
n
2
1, if l  k
uTk u k   lk  
0, otherwise
Image Understanding, Xuejin Chen
Calculating Eigenfaces
• The vector u k and scalar k are the eigenvectors and
eigenvalues, respectively, of the covariance matrix
1
M
T
T
C
Φ
Φ

AA
 n1 n n
M
A  Φ1 , Φ 2 ,..., Φ M 
N 2  N 2 Intractable task for typical image size
Need a computationally feasible method to find these eigenvectors
Image Understanding, Xuejin Chen
Calculating Eigenfaces
• If image number M < N^2, only M meaningful
eigenvectors
Solve MxM matrix
16x16  16384*16384
(an 128x128 image)
• Consider eigenvectors vi of A’A
AT A such that
AT Avi  i vi
ui  Avi
AAT Avi  i Avi
i  i
Image Understanding, Xuejin Chen
Calculating Eigenfaces
• Construct MxM matrix L  AT A, Lmn  ΦTmΦn
• Compute M eigenvectors vl of L
• The M eigenfaces is the linear combination of
the M eigenvectors
ui  Avi
i  i
M
ul   v lk Φ k , l  1,..., M
k 1
The computations are greatly reduced
Image Understanding, Xuejin Chen
• Average face
• 7 eigenfaces
calculated from the
training images
Image Understanding, Xuejin Chen
Classify a Face Image
• 40 eigenfaces are sufficient for a good description for
an ensemble of M=115 images [Sirovich and Kirby 1987]
• A smaller M’ is sufficient for identification
• C
• hoose M’ significant eigenvectors of L matrix (M=16,
M’=7)
Image Understanding, Xuejin Chen
Classify a Face Image
• New image Γ : transformed into its eigenface
components
T
T

 1 , 2 ,..., M ' 
k  u k ( Γ - Ψ )
Image Understanding, Xuejin Chen
Classify a Face Image
•
T  1 , 2 ,..., M ' 
describe the contribution of each
eigenface in representing the input image, treating
the eigenfaces as a basis set for face image
• Find a class of the face
– A simple distance
    k
2
k
2
Average of a class
or individual
Image Understanding, Xuejin Chen
Classify a Face Image
• Creating the weight vector is: projecting the
original face image onto the low-D face space
• Distance from image to face space
Φ  ΓΨ
Φ f  i 1iui
  ΦΦf
2
M'
2
Image Understanding, Xuejin Chen
Four possibilities for an image and pattern vector
• Near face space and near a face
class
• Near face space but not near a
known face class
• Distant from face space but near
a face class
• Distant from face space and not
near a known face class
Image Understanding, Xuejin Chen
Distance to face space
(a) 29.8
(b) 58.5
(c) 5217.4
Image Understanding, Xuejin Chen
Summary of Eigenface Recognition
1.
Collect a set of characteristic face image of the known individuals.
–
The set should include a number of images for each person, with
some variation in expression and lighting (M=40)
2.
Calculate the 40x40 matrix L, find its eigenvalues and eigenvectors,
choose M’(~10) eigenvectors with highest eigenvalues
3.
Compute eigenfaces u k
4.
For each known individual, calculate the class vector  k by
averaging the eigenface pattern vector  .
–
–
Choose a threshold  that defines the maximum allowable distance
from any face class and
a threshold  f that defines the maximum allowable distance from
face space
Image Understanding, Xuejin Chen
Summary of Eigenface Recognition
5.
For each new face image to be identified, calculate
–
–
–
–
•
–
•
•
•
its pattern vector  ,
the distance to face space  ,
the distance to each known class  i .
if the minimum distance  k   ,    f
Classify the input face as the individual associated with class vector
If the minimum distance  k   ,    f
The image may be classified as unknown, and
Optionally used to begin a new face class
Optionally, if the new image is classified as a known individual, the
image may be added to the original set of familiar face image, and
the eigenfaces may be recalculated (steps 1-4)
Image Understanding, Xuejin Chen
Locating and Detecting Faces
• Assume a centered face image, the same size
as the training images and the eigenfaces
• Using face space to locate the face in image
• Images of faces do not change radically when
projected into the face space, while the
projection of nonface images appears quite
different
– > detect faces in a scene
Image Understanding, Xuejin Chen
Use face space to locate face
• At every location in the image, calculate the distance
between the local subimage and face space, which is
used as a measure of ‘faceness’  a face map
Expensive calculation
Image Understanding, Xuejin Chen
Face Map
• A subimage at (x,y) 
Φf
Φ = Γ-Ψ
 = ΦΦf
2
2
 Φ  Φ f  Φ  Φ f 
 ΦT Φ  ΦT Φ f  ΦTf  Φ  Φ f
T
 ΦTf Φ f

 0, Φ f   Φ  Φ f
 ΦT Φ  ΦTf Φ f
Image Understanding, Xuejin Chen

Face Map
• Φ f  i 1iui : linear combination of
orthogonal eigenface vectors
L
Φ Φ f  i 1i2
T
f
L
 ( x, y)  Φ ( x, y)Φ( x, y)  i 1i2 ( x, y)
2
L
T
Image Understanding, Xuejin Chen
Face Map
 ( x, y)  Φ ( x, y)Φ( x, y)  i 1i2 ( x, y)
2
L
T
T

(
x
,
y
)


i1
i1 ( x, y)ui
L
2
i
L
 i 1 Γ( x, y)  Ψ ui
L
T
  i 1  ΓT ( x, y )ui  ΨT ui 
L
  i 1  I ( x, y )  ui  ΨT ui 
L
Correlation operator
Image Understanding, Xuejin Chen
Face Map
 ( x, y)  Φ ( x, y)Φ( x, y)  i 1i2 ( x, y)
2
L
T
Φ ( x, y )Φ( x, y )   Γ( x, y )  Ψ   Γ( x, y )  Ψ 
T
T
 Γ( x, y)T Γ( x, y)  2ΨT Γ( x, y )  ΨT Ψ
 Γ( x, y)T Γ( x, y)  2Γ( x, y)  Ψ  ΨT Ψ
 2 ( x, y)  ΓT ( x, y)Γ( x, y)  2Γ( x, y)  Ψ  ΨT Ψ  i 1[Γ( x, y)  ui  Ψ  ui ]
L
L+1 correlations
Precomputed
Can be implemented by a simple neural networks
Image Understanding, Xuejin Chen
Learning to Recognize New Faces
• An image is close to face space, but not classified as
one of the familiar faces, labeled as “unknown”
• If a collection of “unknown” pattern vectors cluster in
the pattern space, a new face is postulated
• Check similarity: the distance from each image to the
mean is smaller than a threshold
• Add the new face to database (optionally)
Image Understanding, Xuejin Chen
Background Issue
• Eigenface analysis can not distinguish the face
from background
• Segmentation?
• Multiply the image by a 2D Gaussian window
centered on the face
– Deemphasize the outside of the face
– Also practical for hairstyle changing
Image Understanding, Xuejin Chen
Scale and Orientation Issue
• Recognition performance decreases quickly as
the size is misjudged
• Motion estimation?
• Multiscale eigenfaces / multiscale input image
• Non-upright faces
– Orientation estimation using symmetry operators
Image Understanding, Xuejin Chen
Distribution in Face Space
• Nearest-neighbor classification assumes Gaussian
distribution of an individual feature vector
• No prior to assume any distribution
• Nonlinear networks to learn the distribution by
example [Fleming and Cottrell, 1990]
Image Understanding, Xuejin Chen
Multiple Views
• Define a number of face classes for each
person
– Frontal view
– Side view at ± 45°
– Right and left profile views
Image Understanding, Xuejin Chen
Experiments
• Database
– Over 2500 face images under controlled
conditions
– 16 subjects
• All combinations of 3 head orientations, 3 head sizes, 3
lighting conditions
• Construct 6-level Gaussian pyramid from 512x512 to
16x16
Image Understanding, Xuejin Chen
Variation of face images
for one individual
Image Understanding, Xuejin Chen
Experiments with Lighting, Size, Orientation
• Training sets
– One image of each person, under the same lighting
condition, size, orientation
– Use seven eigenfaces
• Mean accuracy as the difference between the
training conditions, test conditions
–
–
–
–
Difference in illumination
Image size,
Head orientation
Combinations of illumination, size, orientation
Image Understanding, Xuejin Chen
• Changing lighting
conditions --- few errors
(a) Lighting 96%
(b) Size 85%
(c) Orientation 64%
• Image size changing -performance dramatically
drops
– Need multiscale approach
(d) Orientation & lighting
(e) Orienation & Size 1
(g) Size & Lighting 1
(f) Orientation & Size 2
(h) Size & Lighting 2
Image Understanding, Xuejin Chen
Experiments with varying thresholds
• Smaller threshold:
– Few errors, but more false negative
• Larger threshold
– More errors
• To achieve 100% accurate recognition, boost unknown rate to
– 19% while varying lighting
– 39% for orientation
– 60% for size
• Set the unknown rates to 20%, the correct recognition rate
– 100% for lighting
– 94% for orientation
– 74% for size
Image Understanding, Xuejin Chen
Neural Networks
• Can be implemented using parallel computing elements
Image Understanding, Xuejin Chen
Collection of networks to implement computation of the pattern
vector, projection into face space, distance from face space, and
identification
Image Understanding, Xuejin Chen
Image Understanding, Xuejin Chen
Conclusion
•
•
•
•
Not general recognition algorithm
Practical and well fitted to face recognition
Fast and simple
Do not require perfect identification
– Low false-positive rate
– A small set of likely matches for user-interaction
Image Understanding, Xuejin Chen
Eigenface
• Tutorial
Image Understanding, Xuejin Chen
Bayesian Face Recognition
Baback Moghaddam, Tony Jebara
and Alex Pentland
Pattern Recognition
33(11), Nov. 2000
Novelty
•
•
•
•
A direct visual matching of face images
Probabilistic measure of similarity
Bayesian (MAP) analysis of image differences
Simple computation of nonlinear Bayesian
similarity
A Bayesian Approach
• Many face recognition systems rely on
similarity metrics
– nearest-neighbor, cross-correlation
– Template matching
• Which types of variation are critical in
expressing similarity?
Probabilistic Similarity Measure
• Intensity difference
  I1  I 2
• Two classes of facial image variations
– Intrapersonal variations  I
– Extrapersonal variations E
Non-Euclidean
similarity measure
• Similarity measure
S ( I1 , I 2 )  P(I )  P(I | )
Can be estimated using likelihoods given by Bayes rule
A Bayesian Approach
• First instance of non-Euclidean similarity
measure for face recognition
• A generalized extension of
– Linear Discriminant Analysis
– FisherFace
• Has computational and storage advantages
over most linear methods for large database
Probabilistic Similarity Measures
• Previous Bayesian analysis of facial
appearance
• 3 different inter-image representations were
analyzed using the binary formulation
– XYI-warp modal deformation spectra
– XY-warp optical flow fields
– Simplified I-(intensity)-only image-based
difference
  I1  I 2
Probabilistic Similarity Measures
• Intrapersonal variations
– Images of the same individual with different expression,
lighting conditions..
• Extrapersonal variations
– Variations when matching two individuals
• Both are Gaussian-distributed, learn the likelihoods
P( | I ), P( | E )
Probabilistic Similarity Measures
• Similarity score
S ( I1 , I 2 ) 
P (  |  I ) P ( I )
P (  |  I ) P ( I )  P (  |  E ) P ( E )
– The priors can be set as the portion of image number in
the database or specified knowledge
• Maximum Posterior (MAP)
P(I | )  P(E | )
1
S ( I1 , I 2 ) 
2
Probabilistic Similarity Measures
• M individuals: M classes
– Many classification -> binary pattern classification
• Maximum likelihood measure
S '  P( | I )
• Almost as effective as MAP in most cases
Subspace Density Estimation
• Intensity difference vector: high dimensional
– No sufficient independent training examples
– Computational cost is very large
– Intrinsic dimensionality or major degree-of-freedom of
intensity difference is significantly smaller than N
• PCA
– Divides the vector space R^N into two complementary
subspaces [Moghaddam & Pentaland]
Subspace Density Estimation
• Two complementary subspaces
A typical eigenvalue
spectrum
Subspace Density Estimation
• Likelihood estimation

 1 M yi2   
  2 ()  
 exp       exp  

2

2

i

1
i






Pˆ ( | )  
M
( N M ) / 2

  PF ( | ) PˆF ( | )
M /2
1/ 2    2 
  2   i  

i 1


 
True marginal
yi : the principal components
density in F
2
 (): the residual (DFFS)
: the weighting parameter
found by minimizing the cross-entropy
N
1

i

N  M i  M 1
Estimated marginal
density in F’
Dual Eigenfaces
Variations mostly due to expression
changes
Intrapersonal
Extrapersonal
Variations such as hair, facial hair, glasses…
Dual Eigenfaces
• Intensity differences of extrapersonal type
span a larger vector space
• Intrapersonal eigenfaces corresponds to a
more tightly constrained subspace
• The key idea of probability similarity measure
Efficient Similarity Computation
• Two classes: one intrapersonal and the other
as extrapersonal variations with Gaussian
distribution P( |  )  e   

E
T
1
E
(2 ) D / 2  E

P( |  I ) 
1
2
1/ 2
1
2
e T  I 1
(2 )
D/2
I
Zero-mean since the pair   I k  I j
Use the principal
components
1/ 2
  I j  Ik
Bayesian Face Recognition
CSE 576, Spring 2008
Face Recognition and Detection
55
Computation
• To get the similarity
– Subtracting
– Project to principal eigenfaces of both
extrapersonal and intrapersonal Gaussians
– Expotentials for likelihood
– Iterated all the operations over all members of the
database (many I-k images) until find the
maximum score
Offline Transformations
• Preprocess the I_k images with whitening
transformations
• Consequently every image is stored as two
vectors of whitened subspace coefficients

1
2

1
2
i j =Λ I VI I j : for intrapersonal
e j =Λ E VE I j : for extrapersonal
Λ V are the matrics of the largest eigenvalues and eigenvectors of E or I
Offline Transformations
• Euclidean distances are computed
for each similarity
times
• Likelihood
• Avoid unnecessary and repeated image differencing
and online projection
Experiments
• ARPA FERET database
– Images taken at different times, location, imaging
conditions (clothes, lighting)
• Training Set
– 74 pairs of images (2/person)
• Test set
– 38 pairs of images
(a) Training Set
(b) Test Set
• Differences in clothing, hair, lighting..
Face Alignment (Detection)
Bayesian Matching
• Training data
– 74 intrapersonal differences
– 296 extrapersonal differences
– Separate PCA analysis on each
• How they distribute?
– Completely enmeshed distributions with the same
principle components
Hard to distinguish low-amplitude extrapersonal difference from
intrapersonal difference
Separate PCA
• Dealing with low-D hyper-ellipsoids with are
intersecting near the origin of a very high-D space
• Key distinguishing factor is their relative orientation
Dual Eigenfaces
Variations mostly due to expression
changes
Intrapersonal
Extrapersonal
Variations such as hair, facial hair, glasses…
Dual Eigenfaces
• Intensity differences of extrapersonal type
span a larger vector space
• Intrapersonal eigenfaces corresponds to a
more tightly constrained subspace
• The key idea of probability similarity measure
Dual Eigenfaces
• Compute two sets of , compute likelihood
estimates P( |  ), P( |  )
• Use principal dimensions M  10, M  30
• Set the priors as equal P( )  P( )
I
E
I
I
E
E
Performance
• Improvement over the accuracy obtained with
a standard eigenface nearest-neighbor
matching rule
• Maximum likelihood gets a similar result with
MAP
– 2~3% deficit on recognition rate
– Computational cost is cut by a factor of 2
Performance
Computation Simplification
• Exact mapping of the probabilistic similarity
score without requiring repeated imagedifferencing and eigenface projections
• Nonlinar matching  simple Euclidean norms
of their whitened feature vectors which can be
precomputed offline
Discussion
• Model larger variations in facial appearance?
– Pose, facial decorations?
• Regular glasses
• Sunglasses, significant changes in beards, hair
– Add more variation in interpersonal training? …
– Views
• View-based multiple model
Bayesian Face Recognition
CSE 576, Spring 2008
Face Recognition and Detection
72
Conclusions
• Good performance of probabilistic matching
• Advantageous in intra/extra density estimates explicitly
characterize the type of appearance variations
– Discovering the principle modes of variations
• Optimal non-linear decision rule
– Do not need to compute and store eigenfaces for each individual
– One or two global eigenfaces are sufficient
• Maximum Likelihood vs. MAP
View-Based and Modular Eigenspaces
for Face Recognition
Alex Pentland, Baback Moghaddam and
Thad Starner
CVPR’94
Part-based eigenfeatures
• Learn a separate eigenspace
for each face feature
• Boosts performance of
regular eigenfaces
CSE 576, Spring 2008
Face Recognition and Detection
75
Morphable Face Model
• Use subspace to model elastic 2D or 3D shape
variation (vertex positions), in addition to
appearance variation
Shape S
Appearance T
CSE 576, Spring 2008
Face Recognition and Detection
76
Morphable Face Model
m
S model   ai S i
i 1
m
Tmodel   bi Ti
i 1
• 3D models from Blanz and Vetter ‘99
CSE 576, Spring 2008
Face Recognition and Detection
77
Project 3 Eigenfaces
• Given a skeleton, you need to fill the functions
– PCA to compute eigenfaces
– Projection into the face space
– Determining if a vector represents a face
– Verifying a user based on a face, finding a face
match given a set of user face information
– Finding the size and position of a face in an image
Image Understanding, Xuejin Chen
Project 3 Eigenfaces
• Skeleton code is large, please take time to get
familiar with the classes and methods
– Vector
– Image operations
– Minimum modification: faces.cpp, eigenfaces.cpp
Image Understanding, Xuejin Chen