Last tuesday, you talked about active
shape models
• Data set of 1,500 hand-labeled faces
• 20 facial features (eyes, eye brows, nose, mouth, chin)
• Train 40 individual regressors (x and y positions for
each facial feature)
1
Problems/Solutions
• Our regressors aren’t perfect,
but they’re often wrong in
conflicting ways
• Facial feature positions aren’t
independent of each other
– eg. knowing something about
the position of the eyes ought
to provide a clue about the
position of the nose
• Face alignment needs to be
more than the sum of its parts
– Active shape model
Shape model
• Given many examples of face shapes (and
ignoring the image data)
• Find a lower dimensional model that can still
represent the high dimensional examples
3
The Active Shape Model framework
• Build a model of face shapes to go along with our facial
feature models, then combine the two into one output
Shape
Features
Alignment
4
Shape model
• Problem: We tried to describe our face using a
40-dimensional vector. That gives us a little bit
too much flexibility:
Face
Dog (?)
• Solution: If 40 dimensions are too many, let’s
build a low-dimensional model
Learning a shape model
• Use Principal Component Analysis (PCA) on
the feature positions
– Represent each example as a 40-dimensional
vector, (x1, y1, x2, y2, …, x20, y20)
– Subtract the mean from each vector
– Find the most important (principal) axes
• PCA gives us a total of 40 axes, but if we only
select a few of them (say, the top six), we’ll get
a low dimensional parameterization
Learning a shape model
• Compute mean face shape
Learning a shape model
• Compute principal components
8
ACTIVE APPEARANCE MODELS
• “Interpretation through synthesis”
• Form a model of the object/image (Learnt from the training
dataset)
I. Matthews and S. Baker, "Active Appearance Models Revisited," International Journal of Computer Vision, Vol. 60, No. 2, November, 2004, pp. 135 - 164.
Model
“Shape”
“texture”
Face appearance has two
components:
Active Shape Models
training set
(Hard to automate)
•
Shape Vector provides alignment
=
42
Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt
Random Aside (can’t escape structure!)
Antonio Torralba & Aude Oliva
(2002)
Averages: Hundreds of images
containing a person are
averaged to reveal regularities
in the intensity patterns across
all the images.
Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt
Random Aside (can’t escape structure!)
Tomasz Malisiewicz, http://www.cs.cmu.edu/~tmalisie/pascal/trainval_mean_large.png
“100 Special
Moments” by
Jason Salavon
Jason Salavon, http://salavon.com/PlayboyDecades/PlayboyDecades.shtml
Given Shape Vectors (positions of set of keypoints)
•
Shape Vector provides alignment
Collect keypoints from
training data in Ms whose
columns are training
shapes.
=
Subtract mean shape (s),
then run SVD to get
(Ms = Us S Vs’).
42
si = s + Usai
Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt
“Shape free” Texture Models
warp to mean shape
Once you have the shape model for each image,
warp the model to the mean configuration:
Now, all faces are the same size
and shape and place. So you can
create a column vector out of the
relevant pixels. Put all those
columns together (one for each
training image) into Mt
Subtract mean texture (t),
then run SVD to get
Mt = Ut S Vt’.
ti = t + Utbi
Face model
• All faces parameterized by mean texture t and mean
shape s.
• Each face has shape parameter ai, and texture
paramter bi.
• Given a shape and a texture, we construct a face as:
– Texture = t + Utb
– Shape = s + Usa
– Make the texture, then warp it onto the new shape.
– Awkward Mathematical notion.
– Warp(Image, shape parameters)
– W(t + Utb, s + Usa)
PCA Craziness
• The complete description of the face is [ai,bi].
• Let’s put those into one column, make a matrix
M for all those columns and run PCA.
• Why the heck would we do this?
Average faces.
•
Alignment is the key!
1. Warp to mean
shape
2. Average pixels
Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt
Random Aside
•
Enhancing Gender
more same original androgynous more
opposite
D. Rowland, D. Perrett. “Manipulating Facial Appearance through Shape and Color”, IEEE Computer Graphics and Applications, Vol. 15, No. 5: September 1995, pp. 70-76
Playing with the Parameters
First two modes of shape variation
First two modes of gray-level variation
First four
modes of
appearance
variation
Essence of the Idea (cont.)
•
Explain a new example in terms of the model parameters
Active Appearance Model Search (Results)
Final Projects
• 1 person: Implement any recent paper from
CVPR/ ECCV/ ICCV, and apply to some new
data.
• 2-3 people, small research project; perhaps
starting with recent paper and substantial
extension, or one of the following ideas
Example 1 person projects…
• Single Image Haze Removal
Using Dark Channel Prior
– http://research.microsoft.com/enus/um/people/jiansun/papers/deh
aze_cvpr2009.pdf
Example 2-3 person projects
Matching historical and modern photos
• http://www.thirdview.org/3v/rephotos/index.
html
What year was this photo taken?