COMPARING FACE IMAGES
A Project
Presented to the faculty of the Department of Electrical and Electronic Engineering
California State University, Sacramento
Submitted in partial satisfaction of
the requirements for the degree of
MASTER OF SCIENCE
in
Computer Engineering
by
Rohit Kumar Bhadury
SUMMER
2012
COMPARING FACE IMAGES
A Project
by
Rohit Kumar Bhadury
Approved by:
__________________________________, Committee Chair
Dr. Fethi Belkhouche
__________________________________, Second Reader
Dr. Preetham Kumar
____________________________
Date
ii
Student: Rohit Kumar Bhadury
I certify that this student has met the requirements for format contained in the University
format manual, and that this project is suitable for shelving in the Library and credit is to
be awarded for the project.
__________________________, CPE Graduate Coordinator
Dr. Suresh Vadhva
___________________
Date
Department of Electrical and Electronic Engineering
iii
Abstract
of
COMPARING FACE IMAGES
by
Rohit Kumar Bhadury
Two methods for recognizing face images have been implemented and compared. The
methods are edge map, and Eigenfaces. Various obstacles like changing illumination,
facial expression, pose, and background greatly affect the performance of the methods.
The effects of these parameters on these two methods have been studied. To achieve this,
different databases containing face images obtained under different conditions, have been
used. The databases used were face94 and grimace provided by Dr. Libor Spacek from
the University of Essex UK and the Yale B extended database from Yale University.
Matlab was used to execute the codes and implement the algorithms of the two methods.
After conducting numerous face-matching tests, it has been concluded that edge map
using M2HD is more robust when it comes to variation in illumination and works
considerably well with severe changes in facial expression and pose. However, at the
same time it is computationally very expensive. On the other hand, Eigenfaces method,
iv
using principal component analysis (PCA) is fast, less complex, and requires very little
storage space. Nevertheless, performance under variations in illumination is poor.
_______________________, Committee Chair
Dr. Fethi Belkhouche
_______________________
Date
v
TABLE OF CONTENTS
Page
List of Tables ............................................................................................................. vii
List of Figures ............................................................................................................ viii
Software Specifications .............................................................................................. xi
Chapter
1. INTRODUCTION ................................................................................................. 1
1.1
Overview .......................................................................................................... 1
1.2
Background and Related Work ........................................................................ 2
1.3
Goals of the Project .......................................................................................... 6
1.4
Organization of the Project .............................................................................. 7
2. EIGENFACES FOR RECOGNITION .................................................................. 8
2.1
Overview of Eigenfaces Method ..................................................................... 8
2.2
Eigenfaces Algorithm ...................................................................................... 9
2.3
Detailed Analysis of the Eigenfaces Method ................................................... 9
3. EDGE MAP ......................................................................................................... 13
3.1
The Hausdorff Distance ................................................................................. 13
3.2
MHD and M2HD ........................................................................................... 15
3.3
Face Recognition Using M2HD ..................................................................... 16
4.
EXPERIMENTAL RESULTS AND ANALYSIS ..............................................18
4.1
Experiment 1 .................................................................................................. 18
4.2
Experiment 2 .................................................................................................. 23
4.3
Experiment 3 .................................................................................................. 31
5
CONCLUSION AND FUTURE WORK ........................................................... 37
Appendix
MATLAB Code ...................................................................................... 39
References ................................................................................................................... 44
vi
LIST OF TABLES
Tables
Page
1.
Results of experiment 1 ……………………….………………………………. 23
2.
Results of experiment 2 ………….……………………………………………. 30
3.
Results of experiment 3 …………....……….…………………………………. 35
vii
LIST OF FIGURES
Figures
Page
2.1 Mean Face ..............................................................................................................12
2.2 Six Eigenfaces ........................................................................................................12
3.1 Step 1 .....................................................................................................................13
3.2 Step 2 .....................................................................................................................13
3.3 Step 3 .....................................................................................................................14
3.4 Step 4 .....................................................................................................................14
3.5 Step 5 .....................................................................................................................14
3.6 Step 6 .....................................................................................................................14
3.7 Step 7 .....................................................................................................................14
3.8 Step 8 .....................................................................................................................14
3.9 Example of three – step image processing in M2HD ............................................17
4.1 1st test subject .........................................................................................................20
4.2 2nd test subject ........................................................................................................20
4.3 3rd test subject ........................................................................................................20
4.4 4th test subject.........................................................................................................20
4.5 5th test subject.........................................................................................................21
4.6 6th test subject.........................................................................................................21
viii
4.7 7th test subject.........................................................................................................21
4.8 8th test subject.........................................................................................................21
4.9 9th test subject.........................................................................................................21
4.10 10th test subject.....................................................................................................21
4.11 11th test subject.....................................................................................................22
4.12 12th test subject.....................................................................................................22
4.13 13th test subject.....................................................................................................22
4.14 14th test subject.....................................................................................................22
4.15 1st test subject compared with the database .........................................................24
4.16 2nd test subject compared with the database ........................................................25
4.17 3rd test subject compared with the database .........................................................25
4.18 4th test subject compared with the database .........................................................25
4.19 5th test subject compared with the database .........................................................26
4.20 6th test subject compared with the database .........................................................26
4.21 7th test subject compared with the database .........................................................26
4.22 8th test subject compared with the database .........................................................27
4.23 9th test subject compared with the database .........................................................27
4.24 10th test subject compared with the database .......................................................27
4.25 11th test subject compared with the database .......................................................28
4.26 12th test subject compared with the database .......................................................28
4.27 13th test subject compared with the database .......................................................28
ix
4.28 14th test subject compared with the database .......................................................29
4.29 15th test subject compared with the database .......................................................29
4.30 16th test subject compared with the database .......................................................29
4.31 17th test subject compared with the database .......................................................30
4.32 18th test subject compared with the database .......................................................30
4.33 Quick glimpse of the subset database ..................................................................31
4.34 Relative Positions with 0 being the frontal pose ..................................................33
4.35 illumination rig.....................................................................................................33
4.36 Input images (left) their corresponding ‘Sobel’ edge and binary image and the
match from the database (right) ..........................................................................34
x
SOFTWARE SPECIFICATIONS
1. MATLAB R2011b (7.13.0.564) 32-bit(win32) August 14, 2011
xi
1
CHAPTER 1: INTRODUCTION
1.1 Overview
Comparing face images is one of the many biometric ways of recognizing an individual.
It has many uses starting with searching for an individual in a database, retrieving
personal information about the individual, gaining access to data based on one’s unique
facial identity etc. It has many applications in forensics, tracking and monitoring, counter
terrorism, prevention of identity theft, IT industries, and financial services. Face images
and their expressions are also used for evaluating a person’s mood and determining the
behavior of the individual. For example in casinos, an individual’s face is monitored to
determine cheating and fraud. One particularity about face recognition is that it is
nonintrusive unlike retinal scan, finger print scan and other biometric methods. Face
comparison is based on image comparison. Different regions and aspects of one’s face,
like eyes, ears, nose etc, can be used as vital parameters in distinguishing one individual
from another. Many techniques can be used to process the data provided by the image of
a face. Once processed, this data can be compared with existing data in a database of
other images, and determine whether the face in question matches that of an individual in
the database or if the image is of a person in the database but taken ten years earlier.
In brief, face recognition is a case of pattern recognition and involves any one of the
following:
a) Authentication in which the individual’s identity is confirmed by comparing with a
uniquely stored identity for example the ATM machine or your home pc login. This
is a one – to – one match.
2
b) Recognition in which a subject is compared to a database of multiple subjects for
identification purposes. For example comparing a mug shot in a forensic database.
This is a one – to – many comparisons.
c) Surveillance in which a subject from a video frame is compared to a small group of
individuals. For example cameras at an airport terminal scanning for possible known
terrorists. This is a case of comparison of one – to – a – list of selected few.
1.2 Background and Related Work
The earliest work to partially automate face recognition was done by Bledsoe (1966a, b).
It was a hybrid human – computer system that classified faces based on fiducial marks
entered on photographs manually. Distance between points such as the corners of the
mouth, eyes, tips of the nose and chin were normalized and their ratios were used as
parameters for classification.
Since then there has been significant advancement in the field of face recognition. There
are many approaches to frontal face recognition. Among these, some of the most
successful ones are Eigenfaces, Neural network, Dynamic link architecture [23], Hidden
Markov model, Geometrical feature matching, and template matching.
Face recognition with Eigenfaces is the most thoroughly researched among these.
According to Sirovich and Kirby [1] principal component or eigenvectors can be used to
represent face images. Their research showed that a face could be approximately
reconstructed by a collection of weights describing each face, and a standard
eigenpicture. The weights of each face are obtained by projecting the test image on the
3
eigenpicture. Turk and Pentland [2] used this idea of eigenface for face detection and
identification. Mathematically eigenfaces are the principal components of the distribution
of faces or the eigenvectors of the covariance matrix of the set of face images (this is
explained in chapter 2). The eigenvectors account for the variation between the images in
a set. A face can be represented by a linear combination of the eigenvectors. Later it was
shown that illumination normalization [6] is required for the eigenface approach.
Pentland et al [3] used the eigenface recognition technique and extended it to facial
features such as eyes, nose, and mouth which can be called eigenfeatures. This modular
eigenspace approach comprising of the eigenfeatures namely eigeneyes, eigennose and
eigenmouth was found to be less sensitive to changes in appearance than the standard
eigenface method. To summarize eigenface method is not invariant to illumination and
changes in scale.
Neural network as mentioned previously is another approach in face recognition. The
difference from eigenface method lies in the fact that it is nonlinear. Due to the non linear
nature of the network the feature extraction is more efficient than eigenface. WISARD
(Wilkie, Stonham and Aleksander’s Recognition Device) was one of the first artificial
neural network (ANN) schemes. It was a single layer adaptive network and each
individual [4] stored in it had a separate network. Neural network is application oriented.
For face detection multilayer perceptron and convolutional neural network [12] has been
applied. Again for face verification Cresceptron [5] which is a multiresolution pyramidal
structure is used. The Convolutional network provides partial invariance to translation,
4
rotation, scale, and deformation. Lin’s [6] probabilistic decision-based neural network
(PDBNN) has a modular structure and a decision based neural network. PDBNN can be
applied to face detection, eye localization and face recognition. PDBNN has the merits of
both neural networks and statistical approaches and its distributed computing principle is
relatively easy to implement on a parallel computer. However when the number of people
increases, the need for more computation also increases. This is because the training time
increases. Also, it is not suitable for single model image recognition as multiple model
images per person are required for the optimal training of the system.
Dynamic link architecture [23] is another method for face recognition. It is an extension
of the classical artificial neural network. In this method, memorized objects are
represented by sparse graphs, whose vertices are labeled with a multi-resolution
description in terms of a local power spectrum and whose edges are labeled with
geometrical distance vectors. However in this method the matching process is
computationally expensive. Wiskott and Von der Malsburg [7] extended the technique
and matched faces against a gallery of 112 neutral frontal view faces. They reported 86.5
percent and 66.4 percent for matching 111 faces of 15 degree and 110 faces of 30 degree
rotation to a gallery of 112 neutral frontal faces. In general, dynamic link architecture is
superior to other face recognition techniques in terms of rotation invariance but the
matching process is computationally expensive.
5
Hidden Markov model is another approach to face recognition. Samaria and Fallside [8]
applied this method to human face recognition. Faces were divided into regions like the
eyes, nose, mouth which could be associated with the states of the hidden Markov model.
Since HMM requires 1 dimensional observation sequence and the images are in 2D , the
images must be converted to either 1D temporal sequences or 1D spatial sequences . An
unknown test image is first sampled to an observation sequence. Then it is matched
against every HMM’s in the model face database. The match with the highest likelihood
is considered the best match and the relevant model reveals the identity of the test face.
The recognition rate is 87 percent using ORL database. The training time is usually high
for this approach.
The geometrical feature matching technique is based on the computation of the
geometrical features of the face. The overall configuration is described by a vector
representing the position and size of the facial features like eyes, nose, mouth and shape
of the outline of the face. Kanade’s [9] face recognition based on this approach reached a
75 percent recognition rate. Bruneli’s and Poggio’s [10] method extracted 35 features
from the face to form a 35 dimensional vector. The recognition was performed with a
Bayes classifier. They reported a recognition rate of 90 percent on a database of 47
people. In these methods, typically 35 – 45 feature points per face were generated.
Geometrical feature matching depends on precisely measured distances between features,
which in turn depend on the accuracy of the feature location algorithms.
6
In template matching, a test image is represented as a two-dimensional array of intensity
values and is compared using a metric such as Euclidean distance with a single template
representing the whole face. In this method, each viewpoint of an individual can be used
to generate a template. Thus, multiple templates can represent each person in the
database. A single template corresponding to a viewpoint can be made up of smaller
distinctive templates [24] [10]. Bruneli and Poggio [10] extracted four features namely
the eyes, nose, mouth and the entire face and compared the performance with their
geometrical method with the template method. The template matching method was found
to be superior to the geometrical matching technique. Drawbacks of template matching
include the complexity of the computation and the description of the templates. As the
recognition system is tolerant to certain discrepancies between the test image and the
template, the tolerance might average out the differences that make a face unique.
1.3 Goals of the Project
In this project, various algorithms for frontal face recognition in two dimensions have
been tested on different databases varying in criteria’s like illumination, pose, expression,
and background. The results of these tests have been compared, and the pros and cons of
various algorithms described and explained.
The major approaches investigated are Eigenfaces, and edge map based methods namely
the doubly modified Hausdorff distance M2HD based on the paper by Takács [13]. The
Eigenfaces approach uses The Principal Component Analysis (PCA) on the vector array
formed by the images of the database. Mathematically it involves finding the
eigenvectors of the covariance matrix of the image set. The M2HD on the other hand is
7
based on edge map encoding with pixel – wise Hausdorff distance template matching.
Matlab has been used to implement this project.
1.4 Organization of the Project
The project report is organized as follows:
CHAPTER 1 INTRODUCTION
1.1
Overview
1.2
Background and Related work
1.3
Goals of the project
1.4
Organization of the project
CHAPTER 2 EIGENFACES FOR RECOGNITION
2.1 Overview of Eigenfaces method
2.2 Eigenfaces algorithm
2.3 Detailed analysis of the Eigenfaces method
CHAPTER 3 EDGE MAP
3.1 Hausdorff Distance
3.2 MHD and M2HD
3.3 Face Recognition Using M2HD
CHAPTER 4 EXPERIMENTAL RESULTS, AND ANALYSIS
CHAPTER 5 CONCLUSION AND FUTURE WORK
Appendix
References
8
CHAPTER 2: EIGENFACES FOR RECOGNITION
2.1 Overview of Eigenfaces Method
The Eigenfaces method proposed by Turk and Pentland in 1991 is the first successful
automated appearance - based system for face recognition. It takes into account the whole
face and not its individual features like the eyes, eyebrows, nose, lips etc. The face image
is represented in a low dimension subspace formed by the Principal Component Analysis
of the image database. In other words, this subspace is made up of eigenvectors or
principle components of the array formed by the image database. Mathematically
eigenfaces are the eigenvectors of the covariance matrix of the image set. These
eigenfaces or eigenvectors represent the significant variations amongst the faces of the
database. Each image in the training set has a unique position in this subspace. In this
method each test image is projected onto this feature space and recognition is performed
based on the distance of this projection and known faces or points.
It has been shown previously that variation in faces caused due to different illumination
levels is much more than the variation caused by identity [14]. Since Eigenface method
uses PCA for reduction in the dimension, it yields projection directions which maximize
the total scatter across all images of the faces. In doing so it retains the unwanted
variations due to changes in illumination. This in turn causes errors when it comes to
discrimination between faces. However it is mentioned [15] that discarding the three
most significant principal components help in reducing the effects caused by illumination
assuming that these variations were in the first place captured by these components.
9
2.2 Eigenfaces Algorithm
The process is based on the proposed method by Turk and Pentland [2] to initially train
the system were
a) The images of all the different individuals were collected in a set called the
training set.
b) The Eigenfaces were calculated from the training set but only M Eigenvectors
corresponding to M largest eigenvalues were retained. These comprised the face
space of the entire image database.
c) Next each image in the training set is projected onto this image space and
corresponding weights are calculated. Now each image is represented by M
weights. Thus the entire database is very compactly represented.
d) The test image now undergoes the same process and is now represented by M
weights.
e) The next step is to compute the distance of the M weights to the M weights of
each image in the training set.
The minimum of these distances is calculated and the image from the database is chosen
as the match or the closest match with which it has this minimum distance.
2.3 Detailed Analysis of the Eigenfaces Method
Let us consider an image I(x,y) in a two dimensional N by N matrix of intensity values.
Each x,y pair represents a pixel in the image. When these images are converted into
vectors, they form a vector of dimension N2. In order to obtain the Eigenfaces or
10
Eigenvectors, the mean vector, the deviation from the mean vector and the covariance
matrix of the training set need to be determined. Images in the training set are represented
as 1 , 2 , 3 , …, M , where ‘M’ is the total number of images.
The mean vector is defined as:

1
M
M

n 1
(1)
n
This mean vector represents the common portions of all the images in the test set which is
why when you convert this vector back to a matrix or an image you see a face that has
common features belonging to most of the images in the database. The deviation vector is
defined as:
 n  n  
(2)
The Eigenvectors or Eigenfaces are the set of principal components of the training set. To
obtain the principal components, Principal Component Analysis (PCA) is performed on
the training set.
The principal components are the Eigenvectors of the covariance matrix of the training
set. The covariance matrix is defined by:
C
1
M
M

n 1
n
 Tn  AAT
(3)
11
where the matrix A  [1 2 ... M ] . However, it is clear that C is a N2 by N2 matrix and
to find N2 eigenvectors and eigenvalues is not practical when dealing with typical image
sizes. Furthermore, the purpose of PCA is to represent the training set in a low dimension
and using N2 eigenvectors negates this. The solution to this problem was provided by
Turk and Pentland.
Consider the eigenvectors  n of matrix AT A such that
AT A n   n n
(4)
Multiplying both sides by A, we have
AAT A n  n A n
(5)
from which we see that A n are the eigenvectors of C  AAT . Thus this reduces the
dimensions from the range of N2 to only M XM and M<<N2. Following this analysis, we
construct the M by M matrix L  AT A , where Lmn   Tm  n , and find the M eigenvectors
 n of L. The first M eigenvectors of the covariance matrix can be found by A n . Here we
should normalize A n .We choose only M'<< M, the number of most significant principal
components. The value of M' can be varied based on the intended accuracy. Each face in
the training set is projected onto the ‘face space’ bounded by the eigenvectors, to obtain
the projection of each image by the following operation
 n   n (  ) , where µn = A n
(6)
Thus, each of the projected images is made up of M' weights. With each image
represented by a set of weights, standard pattern recognition methods can be used to
12
compare input images with images in the training set. Either Euclidean distance or
Mahalanobis distance can be used to determine the closest match.
The Euclidean distance is defined as:
Er min ||'||
(7)
The Mahalanobis distance is defined as:
M'
||||= M' )2
(8)
where i is the eigenvalue. Figure 2.1 and figure 2.2 show the mean face, and the six
significant principal components, respectively of the training set from experiment 3.
Figure 2.1 Mean Face
Figure 2.2 Six Eigenfaces
13
CHAPTER 3: EDGE MAP
3.1The Hausdourff Distance
Named after Felix Hausdorff (1868-1942), it is the maximum distance of a set to the
nearest point in the other set. Let set A = { a1,a2,a3,…, ap }, and B = { b1,b2,b3,…, bq } then
the Hausdroff distance between these sets A and B is defined as:
H(A,B) = max (h(A,B), h(B,A))
(9)
where h(A,B) = max{min{d(a,b)}}
(10)
where d(a,b) can be any metric between these point sets for example the Euclidean
distance. Figures 3.1 through 3.8 represent steps on how to calculate the directed
Hausdorff distance.
Figure 3.1 Step1 [25]
Figure 3.2 Step 2 [25]
14
Figure 3.3 Step 3 [25]
Figure 3.4 Step 4 [25]
Figure 3.5 Step 5 [25]
Figure 3.6 Step 6 [25]
Figure 3.7 Step 7 [25]
Figure 3.8 Step 8 [25]
15
The generic Hausdorff distance measures the dissimilarities between two sets of points. It
can be applied to edge maps. Huttenlocher and colleagues [16] were the first to use
Hausdorff distance to compare binary images and computer vision. In their paper, they
stated that it is more tolerant to perturbations in location of points as it measures
proximity and not superposition unlike correlation techniques. Thus it can be calculated
without explicit pairing of points between data sets. The function h(A,B) is called the
forward Hausdorff distance and h(B,A) is called the reverse Hausdorff distance.
3.2 MHD and M2HD
Dubuisson and Jain [17] redefined the Hausdorff distance as
h(A,B) =
1
𝑁
∑𝑎∈𝐴 min‖𝑎 − 𝑏‖
𝑏∈𝐵
(11)
where N = p is the number of points in set A .They called it the modified Hausdorff
distance and which was less sensitive to noise. It obeys metric properties and operates on
binarized images.
Takács [13] made some improvements to the modified distance by introducing the
concept of a neighborhood function (𝑁𝐵𝑎 ) and associated penalties P. It was assumed that
for each point in A, the corresponding point in B should fall within a range of a given
diameter. These assumptions were valid under the conditions that
a) The input and reference images are normalized
b) The non-rigid transformation is small and localized. An indicator I is introduced
which is equal to 1 if there is a point b ∈ 𝑁𝐵𝑎 and 0 otherwise.
16
Thus M2HD [13] can be defined as:
d(a,B) = max (I min𝑎‖𝑎 − 𝑏‖, (1 – I) P)
b∈𝑁𝐵
h(A,B) =
1
𝑁
∑𝑎∈𝐴 d(a, B)
H(A,B) = max(h(A,B), h(B,A))
(12)
(13)
(14)
According to the above, all matching points should fall within a given neighborhood of
each other. If there is no matching pair, a penalty ensures that images with large overlaps
are distinguished.
3.3 Face Recognition Using M2HD
The process of face recognition consists of three stages of preprocessing
i.
Face detection.
ii.
Normalization of images: In this stage the normalized gray-scale face image is
converted to a binary template by edge detection techniques like Sobel operator
followed by thresholding.
iii.
In the final step the template is matched with a database of faces and the results
are ranked accordingly.
Figure 3.9 shows three subjects from the Yale B Extended database, where A is
the gray – scale image, B the edge image generated by using the ‘Sobel’ operator,
and finally C is the binary image.
In the experiments, the grey scale images were converted to edge maps using ‘Sobel’
operator. They were then converted to binary images using Matlab built in function
‘im2bw’ with the appropriate threshold value resulting in the maximum number of
17
matches. An example of the command is as follows: image = im2bw(image,value) . Here
all luminance above this threshold ‘value’ is converted to 1(or white) and below it to 0
(or black). This value of threshold varied in the experiments with different databases. The
Neighborhood function N was taken as 2 and associated penalty as 2. The inherent
invariance of edges to changes in illumination and lack of point to point matching make
the edge map a good contender versus the Eigenface method. The experiments that
follow compare the two methods under different varying conditions like changes in
illumination, facial expression, and pose.
A
B
C
Figure 3.9 Example of three - step image processing in M2HD. A) Gray Scale B) Output
of Sobel edge Detection C) Binary Image
18
CHAPTER 4: EXPERIMENTAL RESULTS AND ANALYSIS
In this project, the experiments were carried out to compare the Eigenfaces method of
Turk and Pentland to the edge map method of Takács. The face recognition process was
carried out to cover different aspects of recognition like a) changes in illumination b)
changes in facial expression c) changes in pose. Different Databases were used namely a
subset of the cropped images from the Yale B extended database, the University of Essex
**face 94 [18] and **grimace [19] databases. The experiments were classified into three
broad categories. The first set of experiments used The University of Essex face94
database with no changes in illumination. The second was carried out using The
University of Essex grimace database which had slight changes in illumination and the
third was carried out under major illumination changes using the Yale B extended
database [20] [21] [22].
4.1 Experiment 1
The detailed description of the face94 database is as follows:
Acquisition conditions
The subjects sit at fixed distance from the camera and are asked to speak, whilst a
sequence of images is taken. The speech is used to introduce facial expression variation.
**Due to copyright issues only results computed by using this database can be published .The images themselves
cannot be published
19
Database Description
Number of individuals: 153

Image resolution: 180 by 200 pixels (portrait format)

directories: female (20), male (113), male staff (20)

Contains images of male and female subjects in separate directories
Variation of individual's images
Backgrounds: the background is plain green

Head Scale: none

Head turn, tilt and slant: very minor variation in these attributes

Position of face in image: minor changes

Image lighting variation: none

Expression variation: considerable expression changes

Additional comment: there is no individual hairstlyle variation as the images were
taken in a single session.
A subset of this database consisting of 14 individuals out of which half were male
and the other half female was used in the experiment.
The following figures show the results for the edge map and Eigenfaces method:
20
The following figures show subjects on the x-axis and their Euclidean distances on the yaxis. The squares in the graph represent the relative positions of the subjects with respect
to their Euclidean distances in pixels. The square with the least Euclidean distance is the
closest match in the database to the input.
Figure 4.1 1st test subject
Figure 4.2 2nd test subject
Figure 4.3 3rd test subject
Figure 4.4 4th test subject
21
Figure 4.5 5th test subject
Figure 4.6 6th test subject
Figure 4.7 7th test subject
Figure 4.8 8th test subject
Figure 4.9 9th test subject
Figure 4.10 10th test subject
22
Figure 4.11 11th test subject
Figure 4.13 13th test subject
Figure 4.12 12th test subject
Figure 4.14 14th test subject
From the above graphs, the Euclidean distance fails to identify the correct individual
twice, with the 6th and the 10th test subjects. Still in both cases, the correct individual is in
the top two matches.
23
TABLE 1
Results of experiment 1 (High expression variation, low pose changes, and no
illumination changes)
Total Subjects
14
Eigenface
Total
Mismatch
2
Total Matches
ACCURACY
12
85.714 %
Edgemap
2
12
85.714 %
The edge map also fails twice to identify the correct individuals. The comparison in
accuracy of recognition is displayed in Table 1.
In this experiment, there is considerable variation in the facial expression, very little
change with respect to the pose, and no variation in the illumination. We find that under
these conditions, both edge map and Eigenface methods perform equally well.
4.2 Experiment 2
The detailed description of the database for the next experiment is as follows:
Database Description

Number of individuals: 18

Image resolution: 180x200 pixels (portrait format)

Contains images of male and female subjects
24
Variation of individual's images

Backgrounds: the background is plain

Head Scale: small head scale variation

Head turn, tilt and slant: considerable variation in these attributes

Position of face in image : some translation

Image lighting variation: very little

Expression Variation: major expression variation

Additional comment: there is no hairstyle variation as the images were taken in a
single session.
The figures below show the results of the eigenfaces and edge map methods on 18
individuals. In the eigenfaces method, both Euclidean and Mahalanobis distances were
used to see which one is more effective. The graphs display the subjects from 1 – 18 on
the x-axis and the respective distances on the y – axis.
Figure 4.15 1st test subject compared with the database
25
Figure 4.16 2nd test subject compared with the database
Figure 4.17 3rd test subject compared with the database
Figure 4.18 4th test subject compared with the database
26
Figure 4.19 5th test subject compared with the database
Figure 4.20 6th test subject compared with the database
Figure 4.21 7th test subject compared with the database
27
Figure 4.22 8th test subject compared with the database
Figure 4.23 9th test subject compared with the database
Figure 4.24 10th test subject compared with the database
28
Figure 4.25 11th test subject compared with the database
Figure 4.26 12th test subject compared with the database
Figure 4.27 13th test subject compared with the database
29
Figure 4.28 14th test subject compared with the database
Figure 4.29 15th test subject compared with the database
Figure 4.30 16th test subject compared with the database
30
Figure 4.31 17th test subject compared with the database
Figure 4.32 18th test subject compared with the database
TABLE 2
Results of experiment 2 (High expression variation, considerable pose changes, and slight
illumination changes)
Total Subjects 18
EDGE MAP
EIGENFACE
Euclidean Distance
EIGENFACE
Mahalanobis Distance
Total Mismatch
5
3
Total Matches
13
15
ACCURACY
72.22%
83.33%
2
16
88.888%
31
A match is considered when the individual is recognized from the database. From the
graphs above, we find that under slight changes in illumination and considerable changes
in face expression and in the pose, Eigenfaces method outperforms the edge map method
using M2HD by 16.66% if we consider the Mahalanobis distance.
4.3 Experiment 3
In this third experiment, cropped images of a subset of the Yale B extended database
shown in figure 4.33 have been used. The extended Yale Face Database B contains 16128
images of 28 human subjects in addition to the original Yale B database that contained 10
subjects making the total 38 subjects.
Figure 4.33 Quick glimpse of the subset database
The acquired images are 8-bit (gray scale) captured with a Sony XC-75 camera (with a
linear response function) and stored in PGM raw format. Each subject is seen under 576
viewing conditions (9 poses x 64 illumination conditions). An ambient illumination was
also captured making the total different illumination to 65. However in this experiment
32
we consider only one pose that is frontal. The file names are in the
followingformat'yaleB03_P06A+035E+40.pgm' where #3 belongs to the individual in
question, P06 implies the pose number, A+035 signifies that the light source direction
with respect to the camera is at 35 degrees azimuth. A positive azimuth implies that the
light source was to the right and a negative means it was to the left of the subject. E
signifies the elevation which is above horizontal when positive and below when negative.
We conducted three different kinds of testing based on the three directions of light on the
subject and three training sets were formed namely right, left, and frontal corresponding
to right, left and frontal illumination of the subjects. The right set contained three images
per individual with positive azimuths of A+15 E +20, A +60 E+20, and A+70 E+00
while the left had A-15 E +20, A-60 E+20, and A-70 E+00 and the frontal had A+00
E+20, A+00 E-20 and A+00 E=+00. We should note here that the test subjects were not
present in the database. The subjects for each test were A+85 E+20, A-85 E+20, and
A+00 E+45 corresponding to right, left, and frontal. Figure 4.32 shows the relative
position of the nine different poses with ‘0’ being the frontal pose. Figure 4.33 shows the
illumination rig used to capture the images.
It is fitted with 64 computer controlled strobes. The 64 images of each subject in a
particular pose were acquired at camera frame rate (30 frames/second) in about 2
seconds, so there is only a small change in head pose and facial expression for those 64
(+1 ambient) images. The image with ambient illumination was captured without a strobe
going off. Some of the database subjects along with their Sobel edge images and
33
corresponding binary images are shown in figure 3.9.The following images show some of
the results of the Edge map.
Figure 4.34 Relative Positions with
0 being the frontal pose [26]
Fig 4.35 illumination rig [27]
34
Figure 4.36 Input images (left) their corresponding ‘Sobel’ edge and binary image and
the match from the database (right).
The figure above shows the input on the left side and the corresponding match on the
right side. It also shows the Sobel edge image and the corresponding binary images of
three test subjects.
35
The Table below gives a summary of the tests.
TABLE 3
Results of experiment 3 (High illumination changes, and no pose changes)
Test Face
EDGE MAP using
M2HD
Eigenface using
6 eigenvectors
Right
Illumination
90%
10%
Left
Illumination
70%
10%
50%
20%
Frontal
Illumination
From the table above, we find that the recognition rate falls to 70% and 50 % for left and
frontal illumination respectively in the edge map process. Eigenface method performs at
only 10% for right and left, and 20% for frontal illumination. These findings are similar
to those of Yongsheng, K.H Leung [11] in their paper “Face Recognition Using Line
Edge Map” where the Purdue University database was used in their experiments to
account for changes in face illumination. However it was suggested in [15] that if we
were to remove the three most significant principal components it will be possible to
36
reduce the errors generated due to variation in eigenfaces caused under different
illumination. So another test was carried out with the right illumination of the subject
where in this time only one image per test subject was kept in the database. The images in
the databae were A+60 E+20 and the test images were A+85 E+20. This time the first
three significant principal components were removed hoping that they will account for
the maximum variation caused by illumination. But the test results did not show any
improvements. The recognition accuracy remained at 10%. Again the same test was
performed with the least three significant components removed but the scores still
remained the same. We can conclude that the three most significant principal components
may or may not account for the variations caused by illumination which is also consistent
with [15] where the authors clearly mentioned that they hoped it did.
It should be noted that tests were also carried out on the entire database of 38 individuals
for right illumination condition and the recognition rate dropped to 47.368%. This
probably because human faces are very similar in structure and with the increase in the
size of the database the dissimilarity between the faces is diminished.
37
CHAPTER 5: CONCLUSION AND FUTURE WORK
In conclusion, we can see that in the presence of severe illumination conditions and
distortions, edge map performs better than Eigenfaces. This is due to the fact that the
edge map method does not rely on matching explicit points, and edge images are
inherently tolerant towards changes in illumination. However, it is sensitive to large
variations in pose as shown in experiment 2, where it underperforms compared to the
Eigenfaces method. This conclusion is also consistent with the findings of [13]. Moving
on to the Eigenfaces method, it is robust when changes in facial expression and pose are
considered but as concluded from experiment 3 it is very sensitive to changes in
illumination. This is due to the fact that it uses the principal components analysis to
reduce the dimension of the image subspace yielding projection direction that maximize
the total scatter across all the images. But in doing so it also retains unwanted variations
due to changes in illumination. As previously mentioned the variations due to changes in
illumination and viewing direction are far greater than variations due to change in face
identity.
Similar research on face recognition using line edge map and Line Segment Hausdorff
distance [11] have been reported to achieve considerable improvement over edge map.
Specially Weighted Hausdorff Distance SWHD and Doubly Spatially Weighted
Hausdorff Distance SW2HD have attained better comparison results by assigning weights
to the points based on spatial information. In these methods face feature points like eyes,
mouth, nose etc, have more priority. Methods that combine the Eigenfaces methods and
the Hausdorff distance are Spatially Eigen Weighted Hausdroff Distance (SEWHD) and
38
Doubly Spatially Eigen Weighted Hausdroff Distance (SEW2HD) where weighing
functions are used in the Eigenfaces.
39
Appendix
Matlab code for generating edge images based on Sobel operator:
function[B]= Sobel_operator(C)
for i=1:size(C,1)-2
for j=1:size(C,2)-2
% Sobel mask for x-direction:
Gx=((2*C(i+2,j+1)+C(i+2,j)+C(i+2,j+2))-(2*C(i,j+1)+C(i,j)+C(i,j+2)));
%Sobel mask for y-direction:
Gy=((2*C(i+1,j+2)+C(i,j+2)+C(i+2,j+2))-(2*C(i+1,j)+C(i,j)+C(i+2,j)));
%The gradient of the image
%B(i,j)=abs(Gx)+abs(Gy);
B(i,j)=sqrt(Gx.^2+Gy.^2);
end
end
Matlab Code for calculating M2HD :
dir1='Z:\CPE500\Data_Base\';
dir2='Z:\CPE500\Input\';
% Use ‘.pgm or .jpg based on the extension of the input image
data_base=dir(fullfile(dir1,'*.pgm'));
input=dir(fullfile(dir2,'*.pgm'));
input_image=(fullfile(dir2,input(1).name));
c=imread(input_image);
% No of Subjects in the Database
40
Total_Sub=numel(data_base);
% Holds HDD values of the subjects
FA=[];
% Comparing the input image with each image in the database and calculating the
Hausdorff Distance
for n=1:Total_Sub
database_image=fullfile(dir1,data_base(n).name);
% RAW IMAGE
a=imread(database_image);
b=c;
% CONVERT TO double type from unint type
a=im2double(a);
b=im2double(b);
% CONVERTING TO SOBEL EDGE
a=Sobel_operator(a);
b=Sobel_operator(b);
% CONVERTING TO BINARY WITH APPROPRIATE THRESHOLH LEVEL
a=im2bw(a,0.55);
b=im2bw(b,0.55);
% TOTAL ROWS
a_rows=size(a,1);
b_rows=size(b,1);
FHD=0;
RHD=0;
HDD=0;
NF=2;
P=5;
% CALCULATING FORWARD HAUSDORFF DISTANCE
for i=1:a_rows
dist_ab=inf;
for j=1:b_rows
temp=norm(a(i,:)-b(j,:));
if dist_ab>temp
dist_ab=temp;
41
end
end
if dist_ab>NF
dist_ab=P;
end
FHD=FHD+dist_ab;
end
FHD=FHD/a_rows;
% CALCULATING REVERSE HAUSDORFF DISTANCE
for i=1:b_rows
dist_ba=inf;
for j=1:a_rows
temp=norm(b(i,:)-a(j,:));
if dist_ba>temp
dist_ba=temp;
end
end
if dist_ba>NF
dist_ba=P;
end
RHD=RHD+dist_ba;
end
RHD=RHD/b_rows;
HDD=max(FHD,RHD);
FA(:,n)=HDD;
End
S=sort(FA);
[match_score,match_idx]=min(FA);
Correct_image=fullfile(dir1,data_base(match_idx).name);
z=imread(Correct_image);
figure, imshow([c z]);
Matlab Code for Eigenvector method for face recognition using ‘Euclidean’ and
‘Mahalanobis’ Distance:
dir1='Z:\CPE500\Input\';
dir2='Z:\CPE500\Data_Base\';
filenames1=dir(fullfile(dir1,'*.pgm'));
42
input=fullfile(dir1,filenames1(1).name);
filenames2=dir(fullfile(dir2,'*.pgm'));
no_subs=numel(filenames2);
no_images=no_subs;
% Array of Images
t_array=[];
% Creating array of images by converting them to vector
for n=1:no_subs
sub=fullfile(dir2,filenames2(n).name);
img=imread(sub);
img=im2double(img);
img=imresize(img,[180 180]);
t_array(:,n)=img(:);
end
% Code for creating the Projected Images:
% Mean face image vector
mean_face=mean(t_array,2);
% Shifted images of all the images
shifted_images_array=t_array-repmat(mean_face,1,no_images);
% Matrix with reduced eigen vectors
new_matrix=shifted_images_array'*shifted_images_array;
% Eigen vectors of the new matrix taking 6 out of 10
no_eigen_vectrs=6;
[eigen_vect,eigen_val]=eigs(new_matrix,no_eigen_vectrs);
% Eigen vector of the covariance matrix(matirx array formed by image set "t_array")
eigen_vect_covmat=shifted_images_array*eigen_vect;
% Normalize Eigenvector of Covariance
matrixeigen_vect_covmat=mat2gray(eigen_vect_covmat);
% Array of Weights of the projected images of every image
omega_array=eigen_vect_covmat'*shifted_images_array;
% Code for creating the test input:
input_img=imread(input);
input_img=im2double(input_img);
input_img=imresize(input_img,[180 180]);
input_vect=input_img(:);
input_shift=input_vect-mean_face;
input_omega=eigen_vect_covmat'*input_shift;
43
% Code for Calculating the Euclidean Distance and plotting the graph:
% Find the similarity score
ss=arrayfun(@(n) norm(omega_array(:,n)-input_omega),1:no_images);
% Find image with highest similarity or min value of ss
[match_score,match_idx]=min(ss);
% Display Image and Statistics
figure, imshow([input_img reshape(t_array(:,match_idx),180,180)]);
plot(ss,'- gs','MarkerEdgeColor','k','MarkerFaceColor','r'),xlabel('Subjects 1 through 18'),
ylabel('Euclidean Distance');
% Code for Calculating the Mahalanobis Distance and plotting the graph:% Mahalanobis
Distance
inverse_eigen_val=inv(eigen_val);
omega=omega_array-repmat(input_omega,1,no_images);
omega=omega.^2;
Mahal=mean(inverse_eigen_val*omega);
[match_score,match_idx]=min(Mahal);
figure, imshow([input_img reshape(t_array(:,match_idx),180,180)]);
plot(Mahal,'- gs','MarkerEdgeColor','k','MarkerFaceColor','r'),xlabel('Subjects 1 through
18'), ylabel('Mahalanobis Distance');
44
References
[1] L. Sirovich and M. Kirby, Low-Dimensional Procedure for the Characterisation of
Human Faces, J. Optical Soc. of Am., vol. 4, pp. 519-524, 1987.
[2] M. Turk and A. Pentland,Eigenfaces for Recognition, J. Cognitive euroscience, vol.
3, pp. 71-86, 1991.
[3] A. Pentland, B. Moghaddam, and T. Starner, View-Based and Modular Eigenspaces
for Face Recognition, Proc. IEEE CS Conf. Computer Vision and Pattern Recognition,
pp. 84-91, 1994.
[4] T.J. Stonham, Practical Face Recognition and Verification with WISARD, Aspects of
Face Processing, pp. 426-441, 1984.
[5] J. Weng, J.S. Huang, and N. Ahuja, Learning Recognition and Segmentation of 3D
objects from 2D images, Proc. IEEE Int'l Conf. Computer Vision, pp. 121-128, 1993.
[6] S.H. Lin, S.Y. Kung, and L.J. Lin,Face Recognition/Detection by Probabilistic
Decision-Based Neural Network, IEEE Trans. Neural Networks, vol. 8, pp. 114-132,
1997.
[7] L. Wiskott and C. von der Malsburg, Recognizing Faces by Dynamic Link Matching,
Neuroimage 4, pp. S14-S18, 1996.
[8] F. Samaria and F. Fallside,Face Identification and Feature Extraction Using Hidden
Markov Models, Image Processing:Theory and Application, G. Vernazza, ed., Elsevier,
1993.
[9] T. Kanade,Picture Processing by Computer Complex and Recognition of Human
Faces, technical report, Dept. Information Science, Kyoto Univ., 1973.
45
[10] R. Bruneli and T. Poggio, Face Recognition: Features versus Templates, IEEE
Trans. Pattern Analysis and Machine Intelligence, vol. 15, pp. 1042-1052, 1993.
[11] Face Recognition Using Line Edge Map Yongsheng Gao, Member, IEEE, and
Maylor K.H. Leung, Member, IEEE .IEEE TRANSACTIONS ON PATTERN
ANALYSIS AND MACHINE INTELLIGENCE, VOL. 24, NO. 6, JUNE 2002.
[12] S. Lawrence, C.L. Giles, A.C. Tsoi, and A.D. Back, Face Recognition: A
Convolutional Neural-Network Approach, IEEE Trans. Neural Networks, vol. 8, pp. 98113, 1997.
[13] B. Takács, Comparing Face Images Using the Modified Hausdorff Distance, Pattern
Recognition, vol. 31, pp. 1873-1881, 1998.
[14] Y. Moses, Y. Adini, and S. Ullman, “Face Recognition: The Problem of
Compensating for Changes in Illumination Direction,”European Conf. Computer Vision,
1994, pp. 286-296.
[15] P.N. Belhumeur, J.P. Hespanha, and D.J. Kriegman, ªEigenfaces vs. Fisherfaces:
Recognition Using Class Specific Linear Projection, IEEE Trans. Pattern Analysis and
Machine Intelligence, vol. 19, pp. 711-720, 1997.
[16] D. P. Huttenlocher, G. A. Klanderman and W. J. Rucklidge Comparing Images
Using the Hausdor Distance, IEEE Trans. Pattern Anal. Machine Intelligence 15(9),
850-863 (1993).
[17] M. Dubuisson and A. K. Jain, A modified Hausdorff distance for object Matching,
Proc. 12th Int. Conf.on Pattern Recognition (ICPR), Jerusalem, Israel,
(1994).
46
[18] Image Database http://cswww.essex.ac.uk/mv/allfaces/faces94.html 2nd Jan 2012
[19] Image Database http://cswww.essex.ac.uk/mv/allfaces/grimace.html 2nd Jan 2012
[20] Image Database Yale B extended database 2nd Jan 2012
[21] From Few to Many: Illumination Cone Models for Face Recognition under Variable
Lighting and Pose Athinodoros S. Georghiades, Student Member, IEEE, Peter N.
Belhumeur, Member, IEEE, and David J. Kriegman, Senior Member, IEEE June 2001,
VOL. 23,NO. 6, pp. 643-660
[22] Acquiring Linear Subspaces for Face Recognition under Variable Lighting
Kuang-Chih Lee, Student Member, IEEE, Jeffrey Ho, Member, IEEE, and David J.
Kriegman, Senior Member, IEEE. IEEE Trans. Pattern Anal. Mach. Intelligence VOL.
27, NO. 5, MAY 2005
[23] M. Lades, J.C. Vorbrüggen, J. Buhmann, J. Lange, C. von der Malsburg, R.P. Würtz,
and M. Konen, Distortion Invariant Object Recognition in the Dynamic Link
Architecture, IEEE Trans.Computers, vol. 42, pp. 300-311, 1993.
[24] R.J. Baron, Mechanism of Human Facial Recognition, Int'l J. Man Machine Studies,
vol. 15, pp. 137-178, 1981.
[25] http://cgm.cs.mcgill.ca/~godfried/teaching/cg-projects/98/normand/main.html 15th
Feb 2012
[26] http://vision.ucsd.edu/~leekc/ExtYaleDatabase/Yale%20Face%20Database.htm
[27] http://cvc.yale.edu/projects/yalefacesB/domeNew2b.jpg 15th Feb 2012
[28] Bledsoe, W. W . The model method in facial recognition. Panoramic Research Inc,
Palo Alto, CA, Rep. PRI: 15, August 1996a.
47
[29] Bledsoe, W. W). Man-machine facial recognition. Panoramic Research Inc, Palo
Alto, CA, Rep. PRI: 22, August 1996b.