Biometric Based User Authentication through
Facial Expression
A
Dissertation
Submitted in the partial fulfillment of the requirement
For the Award of Degree
Of
Master of Technology
In Computer Science and Engineering
Supervised By:
Submitted By:
Dr. Chander Kant
Assistant Professor
Annu
M. Tech (FinalYear)
Roll No. 49035
Department of Computer Science & Applications
Kurukshetra University, Kurukshetra
2013
1
DEPARTMENT OF COMPUTER SCIENCE &APPLICATIONS
KURUKSHETRA UNIVERSITY KURUKSHETRA
HARYANA (INDIA)
No: ____________
Dated: __________
DECLARATION
I, Annu, a student of Master of Technology (Computer Science & Engineering), in the
Department
of
Computer
Science
and
Applications,
Kurukshetra
University,
Kurukshetra, Roll no. 49035, for the session 2011-2013, hereby, declare that the
dissertation entitled “Biometric Based User Authentication through Facial
Expression” has been completed after the theory examination of 3rd semester.
The matter embodied in this Dissertation has not been submitted to any other institute or
university for the award of any degree to the best of my knowledge and belief.
Date:
Annu
2
DEPARTMENT OF COMPUTER SCIENCE & APPLICATIONS
KURUKSHETRA UNIVERSITY KURUKSHETRA
HARYANA (INDIA)
Dr. Chander Kant
No: ____________
(Assistant Professor)
Dated: __________
CERTIFICATE
This is to certify that the dissertation entitled “Biometric Based User Authentication
through Facial Expression” submitted by Ms. Annu for the award of degree of Master
of Technology in Computer Science and Engineering Roll No. 49035, Kurukshetra
University, Kurukshetra is a record of bonafide research work carried by her, under my
supervision and guidance. The dissertation, in my opinion worthy for consideration for
the award of Degree of Master of Technology in accordance with the regulations of
Kurukshetra University, Kurukshetra. The matter entitled in this dissertation has not been
submitted to any other institute or university for the award of any degree to the best of
my knowledge and belief.
(Dr. Chander Kant)
Assistant Professor,
Department of Computer Science & Applications
Kurukshetra University, Kurukshetra
3
DEPARTMENT OF COMPUTER SCIENCE & APPLICATIONS
KURUKSHETRA UNIVERSITY KURUKSHETRA
HARYANA (INDIA)
Prof. Suchita Upadhyaya
No: ____________
(Chairperson)
Dated: __________
CERTIFICATE
It is certified that Ms. Annu is a bonafide student of Master of Technology in Computer
Science and Engineering, Roll no. 49035, Kurukshetra University, Kurukshetra. He has
undertaken the dissertation entitled “Biometric Based User Authentication through
Facial Expression” under the supervision of Dr. Chander Kant.
(Prof. Suchita Upadhyaya)
Chairperson Department of Computer Science & Applications
Kurukshetra University, Kurukshetra
4
Acknowledgement
No one can do everything all without someone’s help and guidance. Here, I would like to
take a moment to thank those who have helped and inspired me throughout this journey
and without their guidance and support I might not have been able to complete this
research work.
First of all I would like to express my sincere gratitude to my supervisor
Dr. Chander Kant, Assistant Professor, Department of Computer Science &
Applications, Kurukshetra University, Kurukshetra, for his constant support, inspiration
and guidance in both academic and personal life. I am extremely grateful to him for
being an excellent advisor and a wonderful teacher. His attention to detail and
completeness to every aspect of the work, including research, presentation and technical
writing has helped me tremendously to improve my skills. In the course of my master
study, he has helped me with helpful ideas, guidance, comments and criticisms. His
constant motivation, support and infectious enthusiasm have guided me throughout my
dissertation work. I have been privileged to work under his supervision, and I truly
appreciate this help. His encouraging words have often pushed me to put in my best
possible efforts.
I express my sincere thanks to Dr. Suchita Upadhyaya, Chairperson, Department of
Computer Science & Application, Kurukshetra University, Kurukshetra and other staff
members for their support and encouragement.
Last, but not least, I would like to express my gratitude and appreciation to all of my
family for their support, encouragement, faith, understanding and help. Without them,
this work would not have been possible.
Annu
5
List of Publications
Publications in Conference:
-
“Challenges and Scope of Face Recognition” published in the proceedings of All
India Seminar on Information Security on February 25-26, 2013 by DCRUST,
Murthal, Sonepat, under the aegis of Computer Engineering Division Board, IEI.
Publications in the International Journal:
-
“Liveness Detection in Face recognition using Euclidean Distances” published
in “International Journal for advance Research in Engineering and Technology
ISSN: 2320-6802” Vol. 1, Issue IV, May 2013, pp 1-5.
-
“A Novel Approach for Facial Expression Recognition Using Euclidean
Distances” is accepted in “International Journal of Engineering and Advanced
Technology (IJEAT) ISSN: 2249-8958”, Vol. 2, Issue V, June 2013.
6
Table of Contents
List of Figures……………………………………………………………….........10
List of Tables …………………………………………………………………….11
Abstract………………………………………………………...............................12
Dissertation Outline ……………………………………………….………..……13
Chapter1 Introduction to Biometrics………………………………………………….14
1.1 Biometric Technology……………………………………………………………….14
1.2 Biometric Techniques………………………………………………………………..18
1.2.1 Physical Characteristics Based Techniques………………………………………..18
1.2.2 Behavioral Characteristics Based Techniques……………………………………..23
1.3 Comparison of Various Biometric Technologies……………………………………25
1.4 Advantages of using Biometrics…………………………………………………….27
1.5 Challenges of Biometric System…………………………………………………….27
1.6 Multimodal Biometrics………………………………………………………………28
1.7 Soft Biometrics………………………………………………………………………29
Chapter 2 Face Recognition System…………………………………………………..31
2.1. Motivation…………………………………………………………………………...32
2.2 Why face Recognition is difficult……………………………………………………33
2.3 Automatic Face Detection……………………………………………………………35
2.4 Face Detection Approaches………………………………………………………….35
2.4.1 Face Detection by Template Matching……………………………………….……36
2.4.2 Skin Distribution Model………………………………………………….………..36
2.4.3 Face Detection by Neural Network………………………………………………..36
2.4.4 Face Detection by Eigen Face Method………………………………………….…37
2.5 Image Processing………………………………………………………………….....38
7
2.6 Fusion of Face with Other Biometric Traits……………………………………........40
2. 7 Various Biometric Fusion Techniques………………………………………………41
2.7.1 Fusion at the Image Level………………………………………………………….42
2.7.2 Fusion at the Feature Extraction Level…………………………………………….43
2.7.3 Fusion at the Rank Level………………………………………………………….44
2.7.4 Fusion at the Matching Score Level……………………………………………….45
2.7.5 Fusion at the Decision Level……………………………………………………….47
2.8 Summary……………………………………………………………………………..48
Chapter 3 Literature Survey…………………………………………………………. 49
3.1 Security and Privacy Issues in Biometric Authentication…………………………..68
3.1.2 Biometric System Concerns……………………………………………………….68
3.1.2 Vulnerability of Biometric Authentication System……………………………….69
Chapter 4 Conceptual Models for Face Recognition…………………………………74
4.1 Spoofing in Biometric system……………………………………………………….76
4.1.1 Spoofing in face Recognition System………………………………………….….76
4.2 Face Image Acquisition……………………………………………………………...77
4.3 Proposed face Recognition Algorithm for Image Acquisition………………………78
4.3.1 Architecture of the proposed approach…………………………………………….78
4.3.2 Euclidean Distance Test……………………………………………………………80
4.3.2.1 Euclidean Distance………………………………………………….……………81
4.3.2.2 Algorithm of the proposed approach………………………….…………………81
4.3.2.3 Comparison………………………………………………………………………82
4.4 Summary………………………………………………………………….………….83
Chapter 5 Facial Expression Recognition…………………………………………….84
5.1 Face Detection / Localization…………………………………………………….….84
5.1.1 Color models for skin color classification…………………………………………85
8
5.1.2 Algorithm for detecting a face……………………………………………………..88
5.2 Facial Expression Recognition………………………………………………………90
5.2.1 Facial Action Coding System……………………………………………………...90
5.2.2 Proposed Work…………………………………………………………………….93
5.2.2.1 Algorithm of the proposed approach…………………………………………….93
5.2.2.2 Eigen Face Method………………………………………………………………95
5.2.2.3 Calculating Eigen Faces………………………………………………………….97
5.3 Summary……………………………………………………………………………..98
Chapter 6 Face Recognition Applications…………………………………………….99
6.1 Government use applications……………………………………………….………..99
6.1.1 Law enforcement…………………………………………………………………..99
6.1.2 Security / counterterrorism……………………………………………………….100
6.1.3 Immigration……………………………………………………………………….101
6.1.4 Correctional institutions/prisons …………………………………………………101
6.1.5 Legislature ………………………………………………………………………..101
6.2 Commercial use…………………………………………………………………….102
6.2.1 Banking……………………………………………………………………...……102
6.2.2 Pervasive computing…………………………………………………………...…101
6.2.3 Voter verification…………………………………………………………………103
6.2.4 Healthcare………………………………………………………………...………104
6.2.5 Day care…………………………………………………………………..………104
6.2.6 Missing children/runaways……………………………………………………….104
6.2.7 Residential security……………………………………………………………….104
6.2.8 Internet, E-commerce…………………………………………………..…………104
6.2.9 Benefit payment……………………………………………………..……………105
6.3 Other areas where face recognition used………………………………...…………105
Chapter 7 Conclusion and Future Scope…………………………………………….106
Bibliography…………………………………………...………………………………108
9
List of Figures
Figure 1.1 Enrollment Process in Biometric System
Figure 1.2 Verification and Identification Process
Figure 1.3 Minutiae point of a fingerprint
Figure 1.4 Stricture of hand geometry
Figure 1.5 Structure of iris
Figure 1.6 Image of a face
Figure 1.7 Image of retina
Figure 1.8 Image of voice rhythm
Figure 1.9 Keystroke pattern
Figure 1.10 Example of a signature
Figure 2.1 Steps for face recognition system applications
Figure 2.2 Methods for face detection
Figure 2.3 Biometric fusion systems
Figure 2.4 Authentication Process Flow
Figure 2.5 Fusion at the Image level
Figure 2.6 Fusion at the Feature Extraction level
Figure 2.7 Fusion at the Matching Score level
Figure 2.8 Fusions at the Decision Level
Figure 3.1 Block diagram of a typical biometric authentication system
Figure 4.1 Techniques used in face recognition
Figure 4.2 Block Diagram of image acquisition using CCD camera
Figure 4.3 Architecture of the proposed approach
Figure 5.1 RGB color space
Figure 5.2 The RGB color space within the YCbCr color space
Figure 5.3 HSI color space
Figure 5.4 Steps of a face detection algorithm
Figure 5.5 Facial action coding system
Figure 5.6 Flow Chart of the proposed approach
10
List of Tables
Table 1.1 Performance Comparison of various Biometric Traits
Table 5.1 FACS Action Units
Table 5.2 Miscellaneous Actions Units
11
Abstract
Biometric Based
Expression
User
Authentication
through
Facial
The dissertation addresses certain special concepts of face recognition system. Compared
with other biometric technologies, such as iris, speech, fingerprint recognition, face
recognition can be easily considered as the most reliable biometric trait in all. Face
recognition is a particular type of biometric system that can be used to reliably identify a
person by analyzing the facial features of a person’s image. A biometric system verifies
user identity through comparison of a certain behavioral or psychological characteristic
possessed by the user. Face recognition is used worldwide for the security purposes.
User authentication systems that are based on knowledge such as password or physical
tokens such as ID card are not able to meet strict security performance requirements of a
number of modern computer applications. These applications generally use computer
networks (e.g., Internet), affect a large portion of population, and control financially
valuable tasks (e.g., e-commerce). Biometrics-based authentication systems are good
alternatives to the traditional methods. These systems are more reliable as biometric data
cannot be lost, forgotten, or guessed and more user-friendly because we don’t need to
remember or carry anything. The main functional component of the existing face
recognition system consists of image capturing, face detection, and then features
extraction and finally matching. If it matches correctly then accept the user otherwise
rejected. This dissertation gives a general presentation of the face recognition technology
and also the facial expression recognition with its advantages as well as challenges.
12
Dissertation Outline
Chapter 1
Chapter 1 gives the introduction part of the biometric authentication system, parameter
characteristics, advantages of biometric system, and challenges of biometric system,
multimodal biometric system, soft biometrics and image processing.
Chapter 2
Chapter 2 gives the introduction of the face recognition and detection, face detection
approaches, difficulties in face recognition and fusion of face with other biometric traits.
Chapter 3
Chapter 3 gives the literature survey, security and privacy issues related to the use of
biometrics. Solutions and previous work in this field are presents as well.
Chapter 4
Chapter 4 will discussed about existing face recognition techniques and the proposed
work which includes a method to check the liveness of a person.
Chapter 5
Chapter 5 will discuss about the facial expression recognition, Facial action coding
system, and the proposed work to recognize a person’s expression and then
authenticating the user.
Chapter 6
Chapter 6 presents current applications of face recognition, face recognition survey and
some areas where face recognition is used.
Chapter 7
Chapter 7 gives the conclusion and future scope of face recognition technology.
13
Chapter 1
Introduction to Biometrics
Now-a-days, biometric recognition is a common and reliable way to authenticate any
human being based on his physiological or behavioral biometrics [1]. A physiological
biometric traits is stable in their biometric like fingerprint, iris pattern, facial feature,
hand geometry, gait pattern etc. whereas behavioral biometric traits is related to the
behavior of person such as signature, speech pattern, keystroke pattern. Facial recognition
system is a computer application for automatically identifying or verifying a person from
a digital image or a video frame from a video source.
Face recognition is not a new idea but it has received substantial attention over the last
three decades due to its value both in understanding how FR process works in humans as
well as in addressing many challenging real-world applications, including de-duplication
of identity documents (e.g. passport, driver license), access control and video
surveillance. While face recognition in controlled conditions (frontal face of cooperative
users and controlled indoor illumination) has already achieved impressive performance
over large-scale galleries [2], there still exist many challenges for face recognition in
uncontrolled environments, such as partial occlusions, large pose variations, and extreme
ambient illumination. Local facial features have played an important role in forensic
applications for matching face images. These features include any salient skin region that
appears on the face. Scars, moles, and freckles are representative examples of the local
facial features [3].
1.1 Biometric Technology
Biometric refers to the automatic identification of a person based on his or her physical or
behavioral characteristics. This identification method is preferred over traditional
methods involving passwords and PINs (personal identification numbers) for several
14
reasons, including the person to be identified is required to be physically present at the
point of identification and identification based on biometric techniques avoids the need to
remember a password or carry a token. Now-a-days, with the increased use of computers
as the means of transportation of information technology, restrict access to sensitive or
personal data is essential. Biometric authentication has seen considerable progress in
reliability and accuracy, with some of the traits offering good performance.
The terms “Biometrics” and “Biometry” have been used since the early 20th century to
refer to the field of improvement of arithmetical and mathematical methods applicable to
data analysis problems in the biological sciences [4]. The need for biometrics can be
found in central, state and local governments, in the military and in commercial
applications. Now-a-days, worldwide network security infrastructures, government IDs,
secure electronic banking, investing and other financial communication, retail sales, law
enforcement and health and social services are already benefiting from these
technologies. Biometric based authentication applications include workstation, network
and domain access, single sign in, application login, data protection from illegal access,
remote access to resources, transaction security and web security. Trust in this type of
electronic communication is the necessary to the successful growth of the global financial
system. Utilizing biometrics for personal authentication is appropriate, well-situated and
considerably more accurate than current methods such as utilization of passwords or
PINs.
A biometric system may operate either in an ‘identification’ system or a ‘verification’
(authentication) system. Before the system can be put into verification or identification
mode, a system database consisting of biometric templates must be created through the
process of enrollment.
Enrollment- is the process where a user’s initial biometric sample(s) are collected,
assessed, processed, and stored for ongoing use in a biometric system as shown in Figure
1.1. Basically, user enrollment is a process that is responsible for registering individuals
in the biometric system storage. During the enrollment process, the biometric
characteristics of a person are first captured by a biometric scanner to produce a sample.
Some systems collect multiple samples of a user and then either select the best image or
15
fuse multiple images or create a composite template. If users are experiencing problems
with a biometric system then they have to re-enroll to gather higher quality data.
Figure1.1 Enrollment Process in Biometrics System
Biometric system provides two main functionalities viz. verification and identification.
Figure 1.2 shows the flow of information in verification and identification systems.
Identification- One-to-Many Correspondence: Biometrics can be used to determine a
person’s identity even without his knowledge or permission. The user’s input is
compared with the templates of all the persons enrolled in the database and the identity of
the person whose template has the highest degree of similarity with the user’s input is
output by the biometric system. Typically, if the highest similarity between the input and
all the templates is less than a fixed minimum threshold, the system rejects the input,
which implies that the user presenting the input is not one among the enrolled users. For
example, scanning a crowd with a camera and using biometric recognition technology,
one can determine matches against a known database. Identification is the initial stage to
identify the user through his biometric trait. The data of user stored for ongoing use in a
biometric system permanently.
Verification- One-to-One correspondence: Biometrics can also be used to verify a
person’s identity and the system verifies whether the claim is genuine. If the user’s input
and the template of the claimed identity have a high degree of similarity, then the claim is
accepted as “genuine”. Otherwise, the claim is rejected and the user is considered as
“fraud”. For example, one can grant physical access to a secure area in a building by
16
using finger scans or can grant access to a bank account at an ATM by using retinal scan.
Figure 1.2 shows the flow of information in verification and identification systems.
Figure1.2 Verification and Identification Process
Biometric is necessary due to these characteristics:

Links the event to a particular user ( a password or symbol may be used by
somebody other than the approved user)

Is convenient (nothing to carry or memorize)

Accurate ( it provides for positive authentication)

Fast and scalable

Easy to use and easily understandable

Is becoming socially acceptable and

Cost effective
17
1.2 Biometric Techniques
There are many different techniques available to identify/verify a person based on their
biometric characteristics as suggested by U.K. Biometric Working Group [UKBWG,
2003]. These techniques can be divided into physical characteristics and behavioral
characteristics based techniques.
1.2.1 Physical characteristics based Techniques
Biometrics techniques based on physical characteristics of human being such as finger
print, hand geometry; palm print etc are called physical characteristics based techniques.
Following are examples of biometric techniques based on physical characteristics.
Fingerprint Recognition: Among all the biometric techniques, fingerprints based
identification is the oldest method which has been successfully used in several
applications. The fingerprint itself consists of patterns originate on the tip of the finger,
thus making it a physical biometric. Fingerprints are known to be unique and absolute for
each person and the basic characteristics of fingerprints do not change with time. The
distinctiveness of a fingerprint can be determined by the patterns of ridges and minutiae
points on the surface of the finger. These unique patterns of lines can either be in a loop,
whorl or arch pattern. The most common method involves recording and comparing the
fingerprint’s “minutiae points”. Minutiae points can be considered the uniqueness of an
individual’s fingerprint. The major Minutia points in fingerprint are: ridge ending,
bifurcation, and short ridge or dot as shown in Figure 1.3.
(a) Ridges Ending
(b) Ridges Bifurcation
Figure 1.3 Minutiae points in fingerprint
18
(c) Dot
The ridge ending is the point at which a ridge terminates (see Figure 1.3 a). Bifurcations
are points at which a single ridge splits into two ridges (see Figure 1.3 b). Short ridges or
dots are ridges which are significantly shorter than the average ridge length on the
fingerprint (see Figure 1.3 c). Some examples of the use of fingerprint devices in general
areas are:

Fight the abuse of social services like social security.

Permitting logins based on fingerprints.

Fight against criminal immigration.

Attendance management system in industry, colleges or companies.
Hand Geometry: This biometric approach uses the geometric form of the hand for
confirming the user’s identity. Specific features of a hand must be combined to assure
dynamic verification, since human hands are not unique. Individual hand features are not
descriptive enough for identification. Characteristics such as finger curves, thickness and
length, the height and width of the back of the hand, the distances between joints and the
overall bone structure are usually extracted. Those characteristics are pretty much
determined and mostly do not change in a range of years. The basic structure of hand
geometry is shown in figure 1.4.
Figure 1.4 Structure of hand geometry
As the hand geometry readers are big, rugged, quick and handy, they are well suited to
use in warehouses, manufacturing facilities and other industrial locations that have the
space to comfortably house them. Hand geometry readers are good for time-andattendance applications (replacing punch clocks) where their simplicity and rapid cycle
times are big assets and their lackluster accuracy rates are not major liabilities. The list
19
showing below following examples in real world areas where hand scan identification is
or was used:

Users at the Olympic Games 1996 were identified with hand scans.

In a lot of cases access to military plants is granted upon successful hand scan
identification.

Airport personnel at the San Francisco Airport are identified by hand scans.
Iris Recognition: Biometrics is the science of measuring human characteristics that are
stable and unique among users. Iris recognition is the process of verify a human being by
the pattern of iris. The iris is the area of the eye where the pigmented or colored circle,
usually black, brown or blue rings the dark pupil of the eye. As compared to other
biometric traits the iris is more secured and protected. Iris recognition is an approach to
biometric based verification and identification of people [5]. The future of the iris
recognition system is better in fields that demand rapid identification of the users in a
dynamic environment [6]. Iris patterns are extremely complex. The basic structure of iris
is shown below in figure 1.5.
Figure 1.5 Structure of iris
In this technique, the user places him so that he can see his own eye's reflection in the
device. The user may be able to do this from up to 2 feet away or may need to be as close
as a couple of inches depending on the device. Verification time is generally less than 5
seconds, though the user will only need to look into the device for a couple of moments.
To prevent a fake eye from being used to fool the system, these devices may vary the
light shone into the eye and watch for pupil dilation.
20
Face Recognition: Face recognition (FR) is the problem of verifying or identifying a
face from its image. User face detection plays an important role in applications such as
video observation, human computer interfaces, face recognition and face image databases
[7]. Local facial features have played an important role in forensic applications for
matching face images. The use of local features has become more popular due to the
development of higher resolution sensors, an increase in face image database size, and
improvements in image processing and computer vision algorithms. Local features
provide a unique capability to investigate, annotate, and exploit face images in forensic
applications by improving both the accuracy and the speed of face-recognition systems.
This information is also necessary for forensic experts to give testimony in courts of law
where they are expected to conclusively identify suspects [8]. The image of face is shown
in figure 1.6.
Figure 1.6 Face Recognition
To enable this biometric technology it requires having at least one video camera, PC
camera or a single-image camera. On the other hand, this biometric technique still has to
deal with a lot of problems. Finding a face in a picture where the location, the direction,
the background and the size of a face is variable is a very hard task and many algorithms
have been worked on to solve this problem.
Retina Recognition: Along with iris recognition technology, retina scan is possibly the
most accurate and reliable biometric technology. It is also among the most difficult to use
and requires well trained and is supposed as being quite to highly invasive. The users
have to be cooperative and patient to achieve an accurate performance.
21
Figure 1.7 Retina Recognition
Basically the retina is a thin curve on the back of the eye which senses light and transmits
impulses through the optic nerve to the brain as shown in figure 1.7. Retinal scanning
analyses the layer of blood vessels at the back of the eye. Blood vessels used for
biometric identification are located along the neural retina which is the outermost of the
retina’s four cell layers. Research has proven that the patterns of blood vessels on the
back of the human eye were unique from person to person. It even been proven that these
patterns, even between twins, were indeed unique. This pattern also does not change over
the course of a lifetime. The retinal scanner requires the user to place their eye into some
sort of device and then asks the user to look at a particular mark so that the retina can be
clearly imaged. Scanning involves using a low-intensity light source and an optical
coupler and can read the patterns at a great level of accuracy. This process takes about 10
to 15 seconds in total. There is no known way to replicate a retina, and a retina from a
dead person would deteriorate too fast to be useful, so no extra precautions have been
taken with retinal scans to be sure the user is a living human being.
Vein pattern recognition: Vascular patterns are best described as a picture of the veins
in a person's hand or face. The thickness and location of these veins are believed to be
unique enough to an individual to be used to verify a person's identity. The most common
form of vascular pattern readers is hand-based, requiring the user to place their hand on a
curved reader that takes an infrared scan. This scan creates a picture that can then be
compared to a database to verify the user's stated identity.
22
1.2.2 Behavioral Characteristics based Techniques:
Those Biometrics techniques which are based on the behavior of human being such as
voice, signature, gait, keystroke etc. are called behavioral characteristics based
techniques. Following are examples of biometric techniques based on behavioral
characteristics.
Voice Recognition: Mostly, the voice biometric solutions can be used through a typical
telephone or microphone equipment to the computer. In order to identify or authenticate
users, most voice biometric solution creates a voice print of the user, a template of the
person’s unique voice characteristics created when the user enrolls with the system.
During enrollment the user has to select a passphrase or repeat a sequence of numbers or
words. The passphrase should be in the length of 1 to 5 seconds. The problem with short
passphrases is that they have not enough data for identification. Longer passphrases have
too much information and takes too much time. The user has to repeat the passphrase or
the sequence of numbers several times. This makes the enrollment process long lasting
than with other biometric technologies. All successive attempts to access the system
require the user to speak, so that their live voice sample may be compared against the
pre-recorded template. Voice rhythm is shown below in figure 1.8.
Figure 1.8 Voice rhythm pattern
A voice biometric sample is a numerical model of the sound, pattern and rhythm of a
user’s voice. The main problem occurring in the voice is that the user’s voice changes
over time along with the growth of a user or when someone has got a cold or another
disease. Background noise can also be a disturbing factor which does not gives the
accurate result.
23
Keystroke Dynamics: Keystroke dynamics uses the manner and rhythm in which a user
types characters/password or phrase on the keyboard or keypad. The system then records
the timing of the typing and compares the password itself and the timing to its database.
Here, verification takes less than 5 seconds. Keystroke dynamics is the process of
analyzing the way a user types at a terminal by monitoring the keyboard inputs thousands
of times per second in an attempt to identify users based on normal typing rhythm
patterns. The keystroke pattern shown as below in figure 1.9.
Figure 1.9 Keystroke pattern
Signature Recognition: Signature verification is the process that is used to recognize a
user’s handwritten signature. Dynamic signature verification uses behavioral biometrics
of a handwritten signature to validate the identity of a person. This can be achieved by
analyzing the shape, speed, stroke, and pen pressure and timing information during the
act of signing. The example of signature is shown below in figure 1.10.
Figure 1.10 Signature Recognition
On the other hand, there is the simple signature comparison which only takes into
account what the signature looks like. So with dynamic signature verification, it is not the
shape or look of the signature that is meaningful, there are the changes in the speed,
pressure and timing that occur during the act of signing, thus making it virtually
impossible to duplicate those features. The main difficulty with this technology is to
distinguish between the reliable part of a signature, these are the characteristics of the
24
static image, and the behavioral parts of a signature, which vary with each signing
process. Comparing many signatures made by one user reveals the fact that a user’s
signature is never completely the same and can vary considerably over a user lifetime.
Allowing these variations in the system, while providing the best protection against fake
is a big problem faced by this biometric technology. The financial industry sometimes
uses signature verification for money transactions.
1.3 Comparison of Various Biometric Technologies:
Performance of a biometric measure is usually referred to in terms of the false accept rate
(FAR), the false non match or reject rate (FRR), and the failure to enroll rate (FTE or
FER). The FAR measures the percent of invalid users who are incorrectly accepted as
genuine users, while the FRR measures the percent of valid users who are rejected as
impostors. A number of biometric characteristics may be captured in the first phase of
processing. However, automated capturing and automated comparison with previously
stored data requires that the biometric characteristics satisfy the following characteristics:
1. Universal: Every person must possess the characteristic or attribute. The attribute must
be one that is universal and rarely lost to accident or disease.
2. Uniqueness/singularity: Each expression of the attribute must be unique to the
individual. The characteristics should have sufficient unique properties to distinguish one
person from any other. Height, weight, hair and eye color are all attributes that are unique
assuming a particularly precise measure, but do not offer enough points of differentiation
to be useful for more than categorizing.
3. Permanence/Invariance of Properties: They should be constant over a long period of
time. The attribute should not be subject to significant differences based on age either
episodic or chronic disease.
4. Collectability/Measurability: The properties should be suitable for capturing without
waiting time and must be easy to gather the attribute data inactively.
25
5. Performance: it is the measurement of accuracy, speed, and robustness of technology
used.
6. Acceptability: The capturing should be possible in a way acceptable to a large
percentage of the population. Excluded are particularly invasive technologies, i.e.
technologies which require a part of the human body to be taken or which apparently
damage the human body.
7. Circumvention: Ease of use of a substitute.
There are also some other parameters which are very important during the analysis of a
biometric trait. These are:
Reducibility: The captured data should be capable of being reduced to a file which is
easy to handle.
Reliability and Tamper-resistance: The attribute should be impractical to mask or
manipulate. The process should ensure high reliability and reproducibility.
Privacy: The process should not violate the privacy of the person.
Comparable: Should be able to reduce the attribute to a state that makes it digitally
comparable to others. The less probabilistic the matching involved, the more authoritative
the identification.
Inimitable: The attribute must be irreproducible by other means. The less reproducible
the attribute, the more likely it will be reliable.
Table 1.1 below shows a comparison of various biometric systems in terms of above
mentioned parameters. A. K. Jain ranks each biometric based on the categories as being
low, medium or high. A low ranking indicates poor performance in the evaluation
criterion whereas a high ranking indicates a very good performance.
26
Table 1.1 Performance Comparison of various Biometric Traits
1.4 Advantages of Using Biometrics:

Easier fraud detection

Better than password/PIN or smart cards

No need to memorize passwords

Requires physical presence of the person to be identified

Unique Physical or behavioral characteristic

Cannot be borrowed, stolen or forgotten

Cannot leave it at home
1.5
Challenges with Biometric System
The human face is not a unique rigid object. There are billions of different faces and each
of them can assume a variety of deformations. These variations can be classified as
follows:
a) Inter-personal variations due to
1. Race: It signifies the complexion of the person according to which a person can be
distinguished from other.
27
2. Identity: It tells about the name, address, phone no. of the person which distinguishes
him from the other person.
3. Genetics: Genetic code of each person is different. Because of the genes every
person has different face and gender.
b) Intra-personal variations due to
1. Deformations: These are the result of injury or accident on the face.
2. Expressions: This shows the mood of the person. From expressions it is easy to
determine whether the person is happy or sad.
3. Aging: With age wrinkles appears on the face. The wrinkles change the formation of
the face to a great extent.
4. Facial hairs: Man have moustaches and beard which change the look of the face when
shaven.
5. Cosmetics: Cosmetic surgery has become one of the widely used techniques to
enhance your facial features.
Each of these variations can give rise to a new field of research. Among each of these
facial expressions is a most widely studied technique from past many years.
1.6 Multimodal Biometrics
To enhance the security of the system Biometric follow a new techniques called as
Multimodal biometric. Multimodal Biometric means Biometric authentication by
scanning more number of characteristics.
A multimodal biometric system uses multiple modalities to capture different types of
biometrics. This allows the integration of two or more types of biometric recognition and
verification systems in order to meet stringent performance requirements or combine
several weak biometric measurements to engineer a strong optionally continuous
biometric system. The multimodal system could be, for instance, a combination of
fingerprint verification, face recognition, voice verification and keystroke dynamics or
any other combination of biometrics. This enhanced structure takes advantage of the
28
proficiency of each individual biometric and can be used to overcome some of the
limitations of a single biometric. A multimodal system can combine any number of
independent biometrics and overcome some of the limitations presented by using just one
biometric as the verification tool. This is important if the quality scores from individual
systems are not very high. A multimodal system, which combines the conclusions made
by a number of unrelated biometrics indicators, can overcome many of these limitations.
Also it is more difficult to forge multiple biometric characteristics than to forge a single
biometric characteristic.
1.7 Soft Biometrics
Soft biometric traits are the characteristics of human being that provide some information
about the user, but lack of the distinctiveness and permanence to sufficiently differentiate
any two users. Soft biometric traits embedded in a face (e.g., gender and facial marks) are
ancillary information and are not fully distinctive by themselves in the facial recognition
tasks. However, this information can be explicitly combined with facial matching score to
improve the overall facial recognition accuracy and efficiency and helps the forensic
investigators. The characteristics of human being like gender, height, weight and age can
also be used for the identification purpose. Although these characteristics are not unique
and reliable, yet they provide some useful information about the user. These
characteristics are known as soft biometric traits and these can be integrated with the
primary biometric identifiers like fingerprint, face, iris, signature for the identification
purpose. Unimodal biometric systems make just use of a single biometric trait for
individual recognition. It is difficult to achieve very high recognition rates using
unimodal systems due to problems like noisy sensor data and non-universality or lack of
distinctiveness of the selected biometric trait. Multimodal biometric systems address
some of these problems by combining evidence obtained from multiple sources. A
multimodal biometric system that utilizes a number of different biometric identifiers like
face, fingerprint, hand-geometry, and iris can be more robust to noise and minimize the
problem of non-universality and lack of distinctiveness. However, the problem with
multimodal system is that it will require a longer verification time thereby causing some
inconvenience to the users. A possible solution to the problem of designing a trustworthy
29
and user-friendly biometric system is to use additional information about the user like
height, weight, age, gender, ethnicity, and eye color to improve the performance of the
primary biometric system.
The rest of the dissertation is divided into the six Chapters.
Chapter 2 discusses the face recognition and detection system, face detection approaches,
difficulties of face recognition system, fusion of face with other biometric traits.
Chapter 3 discusses the literature survey and analyses the existing work in the field of
face recognition and detection in general.
Chapter 4 will discuss about the proposed works, existing face recognition and detection
techniques and to check the liveness of a user.
Chapter 5 will discuss about facial expression recognition system, facial action coding
system and the proposed method for recognizing the facial expression and the Eigen face
method.
Chapter 6 presents the current applications of face recognition, face recognition survey
and some areas where face recognition is used.
At last, Chapter 7 presents the conclusion of dissertation and gives the scope of future
work.
30
Chapter 2
Face Recognition System
Face recognition systems are part of facial image processing applications and their
significance as a research area is increasing recently. These systems use biometric
information of the humans and are applicable easily instead of fingerprint, iris, signature
etc., because these types of biometrics are not much suitable for non-collaborative
people. Face detection is essential front end for a face recognition system. Face detection
locates and segments face regions from cluttered images, either obtained from video or
still image. Detection application is used to find position of the faces in a given image
[9].Numerous techniques have been developed to detect faces in a single image[10][11].It
has numerous applications in areas like surveillance and security control systems, content
based image retrieval, video conferencing and intelligent human computer interfaces.
Most of the current face recognition systems presume that faces are readily available for
processing. However, we do not typically get images with just faces. We need a system
that will segment faces in cluttered images. With a portable system, we can sometimes
ask the user to pose for the face identification task.
Figure 2.1 Steps for face recognition system applications
The first step for face recognition system is to acquire an image from a camera. Second
step is face detection from the acquired image. As a third step, face recognition that takes
the face images from output of detection part. Final step is person identity as a result of
recognition part. An illustration of the steps for the face recognition system is shown in
figure 2.1.
31
In addition to creating a more cooperative target, we can interact with the system in order
to improve and monitor its detection. With a portable system, detection seems easier. The
task of face detection is seemingly trivial for the human brain, yet it still remains a
challenging and difficult problem to enable a computer /mobile phone/PDA to do face
detection. This is because the human face changes with respect to internal factors like
facial expression, beard, mustache glasses etc and it is also affected by external factors
like scale, lightning conditions, and contrast between face, background and orientation of
face.
Face detection remains an open problem. Many researchers have proposed different
methods addressing the problem of face detection. In a recent survey face detection
technique is classified in to feature based and image based. The feature based techniques
use edge information, skin color, motion and symmetry measures, feature analysis,
snakes, deformable templates and point distribution. Image based techniques include
neural networks, linear subspace method like Eigen faces, fisher faces etc. The problem
of face detection in still images is more challenging and difficult when compared to the
problem of face detection in video since emotion information can lead to probable
regions where face could be located.
2.1 Motivation
Face detection plays an important role in today’s world. They have many real world
applications like human/computer interface, surveillance, authentication and video
indexing. However research in this field is still young. Face recognition depends heavily
on the particular choice of features used by the classifier One usually starts with a given
set of features and then attempts to derive an optimal subset (under some criteria) of
features leading to high classification performance with the expectation that similar
performance can also be displayed on future trials using novel (unseen) test data.
Interactive Face Recognition is divided in to several phases. It includes:

Creating drivers for the handheld device that link with the application with the
captured image.
32

A face detection program is run inside the handheld device which detects the face
from the image.

The obtained face is transmitted through wireless network.

The server performs the face recognition and is transmitted back.
2.2 Why Face Recognition is Difficult?
The greatest difficulty of face recognition, compared to other biometrics, stems from the
immense variability of the human face. The facial appearance depends heavily on
environmental factors .for example, the lighting conditions, background scene and head
pose. It also depends on facial hair, the use of cosmetics, jewellery and piercing.

Pose: Variation due to the relative camera-face pose (frontal, 45 degree, profile,
upside down), and some facial features such as Facial Expression or the nose may
become partially or wholly occluded.

Presence or absence of structural components: Facial features such as beards,
mustaches, and glasses may or may not be present, and there is a great deal of
variability amongst these components including shape, color, and size. Local
facial mark features such scars, moles, and freckles play an important role for
matching face images in forensic applications [12]. Local facial mark features
provide a unique capability to investigate, annotate, and exploit face images in
forensic applications by improving both the accuracy and the matching speed of
face-recognition systems. This information is also necessary for forensic experts
to give testimony in courts of law where they are expected to conclusively
identify suspects.

Facial expression: The appearance of faces is directly affected by a person's
facial expression.

Occlusion: Faces may be partially occluded by other objects. In an image with a
group of people, some faces may partially occlude other faces.
33

Image orientation: Face images directly vary for different rotations about the
camera's optical axis.

Imaging conditions: When the image is formed, factors such as lighting (spectra,
source distribution and intensity) and camera characteristics (sensor response,
lenses) affect the appearance of a face.

Facial Aging: Many face recognition scenarios exhibit a significant age
difference between the probe and gallery images of a subject. Facial aging is a
complex process that affects both the shape and texture (e.g., skin tone or
wrinkles) of a face. This aging process also appears in different manifestations in
different age groups. In addition to facial aging, there are other factors that
influence facial appearance as well (e.g. pose, lighting, expression, occlusion)
which makes it difficult to study the aging pattern using these two public domain
longitudinal face databases.

Forensic Sketch Recognition: When no photograph of a suspect is available, a
forensic sketch is often generated. Forensic sketches are an artist rendition of a
person’s facial appearance that is derived from an eye witness description.
Forensic sketches have a long history of use, where traditionally they have been
disseminated to media outlets and law enforcement agencies in the hopes that
someone will recognize the person in the sketch. Forensic sketches can be
misleading due to errors in witness memory recall that cause inaccuracies in the
sketch drawn by a forensic artist. Because a significant amount of time is needed
to generate a single forensic sketch, they generally represent culprits who
committed the most heinous crimes (e.g. murder and sexual assault). Thus, the
ability to match forensic sketches to mug shot databases is of great importance.

Face Recognition in Video: Face recognition in video has gained importance due
to the widespread deployment of surveillance cameras. The ability to
automatically recognize faces in video streams will facilitate a method of human
identification using the existing networks of surveillance cameras. However, face
images in video often contain non-frontal poses of the face and may undergo
severe lighting changes.
34
2.3 Automatic Face Detection
a) Facial measures have historically been of interest to a small group of scientists and
clinicians. The main reason is that pain assessment by facial expression is burdensome.
The coding still consumes many multiples of the real times involved in the behavior.
b) The other limitation include facial coding technique: Anyone who has performed facial
measurement realizes that there are elements of subjectivity in deciding when an action
has occurred or when it meets the minimum requirements for coding. In addition, the
human observer has inherent limitations in his or her capacity to make precise
discriminations at the margins. Alternatives exist or are in development.
c) Some faces are often falsely read as expressing some emotion, even when they are
neutral, because their proportions naturally resemble those another face would
temporarily assume when emoting.
Automatic face detection is a complex problem in image processing. Many methods exist
to solve this problem such as template matching, Fisher Linear Discriminant, Neural
Networks, EIGENFACE, and MRC. Success has been achieved with each method to
varying degrees and complexities.
2.4 Face Detection Approaches
Face detection is the first step of face recognition system. Output of the detection can be
location of face region as a whole, and location of face region with facial features (i.e.
eyes, mouth, eyebrow, nose etc.). Mainly, detection can be classified into two groups as
Knowledge-Based Methods and Image-Based Methods. The methods for detection are
given in Figure 2.2.
Knowledge-Based methods use information about Facial Features, Skin Color or
Template Matching. Facial Features are used to find eyes, mouth, nose or other facial
features to detect the human faces. Skin color is different from other colors and unique,
and its characteristics do not change with respect to changes in pose and occlusion. Skin
color is modeled in each color spaces like RGB (Red-Green-Blue), YCbCr (Luminance35
Blue Difference Chroma-Red Difference Chroma), HSV (Hue- Saturation-Value), YUV
(Luminance-Blue Luminance Difference-Red Luminance Difference), and in statistical
models. Face has a unique pattern to differentiate from other objects and hence a template
can be generated to scan and detect faces.
Figure 2.2 Methods for face detection
2.4.1 Face detection by Template Matching
Once individual candidate face images are separated, template matching is used as not
only a final detection scheme for faces, but also for locating the centroid of the face. The
idea of template matching is to perform cross co-variances with the given image and a
template that is representative of the image. Therefore, in application to face detection,
the template should be a representative face - being either an average face of all the faces
in the training images, or an Eigen face. In our case, both templates were created. The
first step was to crop out the faces from each training image posted on the website. Using
these faces as our set of training images was justified since the same people would be
present in the actual test image- otherwise a larger and more diverse set of training faces
images.
36
2.4.2 Skin color distribution model
In conventional methods, all visible colors are divided into two groups: One is the “skin
color” and the other is not. However, consider two colors near the boundary of the skin
part. Although the difference between them is almost unnoticeable by a human viewer,
one is regarded as “skin color” and the other is not. This is unnatural, and is considered as
one of the reasons of instability in conventional methods for skin color detection.
SCDM is a fuzzy set of skin color. We use a large image set containing to investigate the
distribution of color of the human skin region in order to build the SCDM.
The procedure to build the SCDM is as follows:
1.
Manually select skin regions in each image.
2.
Prepare a table of 92 × 140 entries to record the 2-dimensional chromatic
histogram of skin regions, and initialize all the entries with zero.
3.
Convert the chromaticity value of each pixel in the skin regions UCS, and then
increase the entry of the chromatic histogram corresponding to it by one.
4.
Normalize the table by dividing all entries with the greatest entry in the table.
2.4.3 Face detection by Neural Network
Neural Nets are essentially networks of simple neural processors, arranged and
interconnected in parallel. Neural Networks are based on our current level of knowledge
of the human brain, and attract interest from both engineers, who can use Neural Nets to
solve a wide range of problems, and scientists who can use them to help further our
understanding of the human brain. Since the early stages of development in the 1970’s,
interest in neural networks has spread through many fields, due to the speed of processing
and ability to solve complex problems. As with all techniques though, there are
limitations. They can be slow for complex problems, are often susceptible to noise, and
can be too dependent on the training set used, but these effects can be minimized through
careful design. Neural Nets can be used to construct systems that are able to classify data
into a given set or class, in the case of face detection, a set of images containing one or
37
more face, and a set of images that contains no faces. Neural Networks consist of parallel
interconnections of simple neural processors. Neurons have many weighted inputs, that is
to say each input (p1, p2, p3… pm) has a related weighting (w1, w2, w3… wm)
according to its importance. Each of these inputs is a scalar, representing the data. In the
case of face detection, the shade of GRAY of each pixel could be presented to the neuron
in parallel (thus for a 10x10 pixel image, there would be 100 input lines p1 to p100, with
respective weightings w1 to w100, corresponding to the 100 pixels in the input image).
2.4.4 Face detection by Eigen Face Method
The motivation behind Eigen faces is that it reduces the dimensionality of the training set,
leaving only those features that are critical for face recognition.
Definition
1. The Eigen faces method looks at the face as a whole.
2. In this method, a collection of face images is used to generate a 2-D gray-scale image
to produce the biometric template.
3. Here, first the face images are processed by the face detector. Then we calculate the
Eigen faces from the training set, keeping only the highest Eigen values.
4. Finally we calculate the corresponding location in weight space for each known
individual, by projecting their face images onto the “face space”.
2.5 Image Processing
Image processing is any form of signal processing for which the input is an image, such
as a photograph or video frame; the output of image processing may be either an image
or, a set of characteristics or parameters related to the image. Most image-processing
techniques involve treating the image as a two-dimensional signal and applying standard
signal-processing techniques to it. Image processing usually refers to digital image
processing, but optical and analog image processing also are possible. The acquisition of
images (producing the input image in the first place) is referred to as imaging. Image
processing is a physical process used to convert an image signal into a physical image.
38
The image signal can be either digital or analog. The actual output itself can be an actual
physical image or the characteristics of an image. The most common type of image
processing is photography. In this process, an image is captured using a camera to create
a digital or analog image. In order to produce a physical picture, the image is processed
using the appropriate technology based on the input source type. In digital photography,
the image is stored as a computer file. This file is translated using photographic software
to generate an actual image. The colors, shading, and nuances are all captured at the time
the photograph is taken the software translates this information into an image. When
creating images using analog photography, the image is burned into a film using a
chemical reaction triggered by controlled exposure to light. The image is processed in a
darkroom, using special chemicals to create the actual image. This process is decreasing
in popularity due to the advent of digital photography, which requires less effort and
special training to product images. In addition to photography, there are a wide range of
other image processing operations. The field of digital imaging has created a whole range
of new applications and tools that were previously impossible. Face recognition
software, medical image processing and remote sensing are all possible due to the
development of digital image processing. Specialized computer programs are used to
enhance and correct images. These programs apply algorithms to the actual data and are
able to reduce signal distortion, clarify fuzzy images and add light to an underexposed
image. Animations are series of single images put together into a movie. The images
might be a volume view, a projection, a slice, a time point. The animation is done by just
playing the sequential data set, or by rotating 3D models or volume representations, by
zoom-in & fly through motions, changing of surfaces and transparencies, etc. Today’s
computer allows for calculating and representing animated sequences reasonably fast.
Movie files can be published i.e. in power point or on the web. Also interactive file
formats are possible. If one or more of the images in a data set is taken through a filter
that allows radiation that lies outside the human vision span to pass – i.e. it records
radiation invisible to us - it is of course not possible to make a natural color image. But it
is still possible to make a color image that shows important information about the object.
This type of image is called a representative color image. Normally one would assign
colors to these exposures in chromatic order with blue assigned to the shortest
39
wavelength, and red to the longest. In this way it is possible to make color images from
electromagnetic radiation far from the human vision area, for example x-rays. Most often
it is either infrared or ultraviolet radiation that is used.
2.6 Fusion of Face with Other Biometric Traits
In recent years, biometric authentication has seen considerable improvement in reliability
and accuracy, with some of the biometric traits offering practically good performance
[13] for a comparative survey of up to date biometric authentication technology. On the
other hand, even the best biometrics up to date is at rest facing several problems, some of
them natural to the technology itself. Biometric systems that use a single trait are called
unimodal systems, whereas those that integrate two or more traits are referred to as
multimodal biometric systems. A multimodal biometric system requires an integration
scheme to fuse the information obtained from the individual modalities. The multimodal
biometric systems [14] are found to be extremely useful and exhibit robust performance
over the unimodal biometric systems in terms of several constraints. The aim of any
multimodal system is to acquire multiple sources of information from different modalities
and minimize the error prone effect of mono modal systems. In particular, biometric
authentication systems normally suffer from enrollment problems due to non-universal
biometric traits, vulnerability to biometric spoofing or unsatisfactory accuracy caused by
noisy data acquisition in certain environments. Although some biometrics has gained
more popularity than other biometrics, but each such biometric has its own strengths and
limitations, and no single biometric is expected to meet the desired performance of the
authentication applications. Multi biometrics is relatively a new approach to overcome all
these problems. Driven by lower hardware costs, a multi biometric system uses multiple
sensors for data achievement. In 2002 the research paper “Multi Modal Technology
makes Biometrics Work” states the title of a recent press release from Aurora Defense
[15]. Certainly, multi biometric systems ensure significant improvements over single
biometric systems, for example, higher accuracy and increased resistance to spoofing.
They also claim to be more universal by enabling a user who does not have a particular
biometric identifier to still enroll and authenticate using other biometric traits, thus
eliminating enrollment problems. But can multi biometrics live up to the promotion? At a
40
first quick look, incorporating multiple biometrics into one system appears to be a very
sensitive and understandable concept. There are many different ways to actually merge
various sources of information to make a final authentication conclusion. Information
fusion strategies range from simple Boolean combination to complicated numerical
modeling.
Figure 2.3 Biometric fusion systems
The goal of information fusion, therefore, is to determine the best set of experts in a given
problem domain and devise an appropriate function that can optimally combine the
decisions rendered by the individual experts. The recognition process itself may be
viewed as the reconciliation of evidence pertaining to these multiple modalities. Each
modality on its own cannot always be reliably used to perform recognition. However, the
consolidation of information presented by these multiple experts can result in the accurate
determination or verification of identity. Several approaches can be adopted for
combining the different modalities [16]. Biometric fusion system is shown in figure 2.3.
2.7 Various biometric fusion techniques
In a multi modal biometric system, the information fusion can occur at any of the module
of feature extraction, matcher and decision. Multi biometric systems can be categorized
into five system architectures according to the strategies used for information fusion:

Fusion at image level

Fusion at Feature Extraction Level

Fusion at the Rank Level

Fusion at the Matching score Level
41

Fusion at the Decision Level
That is we classify the systems depending on how early in the authentication process the
information from the different sensors is combined. Biometric authentication is a chain
process, as depicted in Figure 2.4 for a more detailed explanation:
Figure 2.4 Authentication Process Flow
Fusion at the feature extraction level stands for immediate data mixing at the beginning
of the processing chain, while fusion at the decision level represents late integration at the
end of the process. The following section describes each of these architectures in detail:
2.7.1 Fusion at the Image Level
In practice, it is possible to combine only “compatible images”. Therefore, in the context
of facial images, image level combination is used only to combine multiple images of the
same face. Facial images first need to be registered with each other before fusion. In case
facial images from multiple sensors are combined, the sensors must be pre-registered.
The objective of such a fusion is to improve the quality of acquired facial images. The
basis of this improvement is that the useful signal in the facial images captured with
different sensing technologies will be independent because different imaging conditions
capture different surface and sub-surface properties of the skin. Another reason to
conduct the image level fusion of facial images is to increase the acquired fingerprint area
wherein each individual image has captured only a portion of the face.
42
Figure 2.5 Fusion at the Image level
2.7.2 Fusion at the Feature Extraction Level
In this architecture, the information extracted from the different sensors is encoded into a
joint feature vector, which is then compared to an enrollment template which itself is a
joint feature via stored in a database and assigned a matching score as in a single
biometric system.
Figure 2.6 Fusion at the Feature Extraction level
However, feature-level fusion may be preferred because features are more compact than
image, leading to efficient fusion algorithms. But fusion at the feature extraction level is
much less preferable than the other next strategies. Two main problems with this
approach are used to identify the problems:

The feature vectors to be joined might be incompatible due to numerical
problems, or some of them might even be unavailable in cases where the users
possess all biometric identifiers. While the first issue might be resolved by
43
careful system design, leading to a very tightly coupled system, the second one
will cause the enrollment problems we already know from single biometric
systems.

Score generation is problematic: even in a single biometric system, it is difficult
to find a good classifier, i.e. to generate a representative score based on the
matching of feature vector and enrollment template. But for the high-dimensional
joint feature vectors in a multi biometric system, it is even more complicated. As
pointed out in [17], the relationship between the different components of the joint
feature vector may not be linear.
2.7.3 Fusion at the Rank Level
Rank-level fusion is used only in identification systems and is applicable when the
matcher output is a ranking of the “candidates” in the template database. The system is
expected to assign a higher rank to a template that is more similar to the query. Most
identification system actually provides the matching score associated with the candidates.
Therefore, the rank level fusion is widely used in other fields such as pattern recognition
and data mining. This method involves combining identification ranks obtained from
multiple unimodal biometrics. It consolidates a rank that is used for making final
decision. In multimodal biometric system, rank level fusion is the method of
consolidating ranks from different biometric modalities (fingerprints, facial features, iris,
retina etc.) to recognize a user. Ho et al., 1994 describe three methods to combine the
ranks assigned by different matchers. In the highest rank method, each possible identity is
assigned the best (minimum) of all ranks computed by different systems. Ties are broken
randomly to arrive at a strict ranking order and the final decision is made based on the
consolidated ranks. The Borda count method uses the sum of the ranks assigned by the
individual systems to a particular identity in order to calculate the fused rank. The logistic
regression method is a generalization of the Borda count method where a weighted sum
of the individual ranks is used. The weights are determined using logistic regression.
44
2.7.4 Fusion at the Matching Score Level
Each system provides a matching score indicating the proximity of the feature vector
with the template vector. These scores can be combined to assert the veracity of the
claimed identity. Techniques such as logistic regression may be used to combine the
scores reported by the two sensors.
Figure 2.7 Fusion at the Matching Score level
In a multi biometric system built on this architecture, feature vectors are created in
parallel for each sensor and then compare it with the enrollment templates, which are
stored independently for each biometric sample. Based on the closeness of feature vector
and template, each subsystem now computes its own matching score. These users’ scores
are finally combined into a total score, which is handed over to the decision module. The
complete process is shown in figure 2.7.
Score level fusion is widely recognized to offer the best tradeoff between the
effectiveness of fusion and the ease of fusion. While the information contained in
matching scores is not as rich as in images or features, it is much richer than ranks and
decisions. Further, while score-level fusion is not as easy or intuitive as rank-level or
decision-level fusion, it is easier to study and implement than image-level and featurelevel fusion. Scores are typically more accessible and available than images or features
but it does not contain rich information than images or features. Also, fusion at the score
level requires some care. But the main difficulties in the score level fusion may emanate
45
from the non-homogeneity of scores from different matchers, differences in the
distributions of the scores, correlation among the scores, and differences in the accuracies
of different matchers. All scores were mapped to the range [0; 100]. A very welldesigned example for this fusion strategy has recently been presented by Ross and Jain in
two research papers [18]. They include facial scan, iris verification and hand geometry
scan into a common authentication system, and using well known methods for each
identifier (Eigen faces for the facial scan, patterns for the iris system, and commonly used
hand geometry features). A score vector - (x1; x2; x3) - represents the scores of multiple
matchers, with x1, x2 and x3 corresponding to the scores obtained from the 3 modalities.
Matching scores for the three different modalities are then normalized and combined
using one of the following strategies.
1. Sum Rule: The sum rule method of integration takes the weighted average of the
individual score values. This strategy was applied to all possible combinations of the
three modalities. Equal weights were assigned to each modality as the bias of each
matcher was not available.
2. Decision Tree: The C5:0 programs was used to generate a decision tree from the
training set of genuine and impostor score vectors. The training set consisted of 11; 025
impostor score vectors and 225 genuine score vectors. The test set consisted of the same
number of independent impostor and genuine score vectors. This strategy uses a sequence
of threshold comparisons on the different scores to make an authentication decision.
3. Linear Discriminant Function: Linear discriminant analysis of the training set helps
in transforming the 3-dimensional score vectors into a new subspace in which the
separation between the classes of genuine user scores and imposter scores is maximized.
The test set vectors are classified by using the minimum Mahalanobis distance rule (with
the assumption that the two classes have unequal covariance matrices). The optimal
parameters for this transformation are calculated in advance based on a training set. The
output score is defined as the minimum distance to the centroids of the two classes, using
a special metric, the Mahalanobis distance.
46
Based on the experimental results, the authors make the observation that the sum rule
achieves the best performance. Most importantly, they further extend the sum rule using a
really new approach: they suggest applying user-specific weights to the user biometric
samples to be combined as well as using user-specific threshold levels for making the
final authentication decision.
2.7.5 Fusion at the Decision Level
In this fusion strategy, a separate authentication decision is made for each biometric trait.
These decisions are then combined into a final vote, as shown below in figure 2.8.
Decision-level fusion is not as popular as score-level or rank-level fusion. Still, it may be
the only feasible approach if the commercial biometric systems involved in fusion
provide only the final match decision. Similar to the rank-level and score-level fusions,
decision-level fusion is conducted at a high-level, therefore it does not matter which
entities are being combined. Each sensor can capture multiple biometric data and the
resulting feature vectors individually classified into the two classes––accept or reject. A
majority vote scheme, such as that employed in (Zuev and Ivanon, 1996) can be used to
make the final decision.
Figure 2.8 Fusions at the Decision Level
Many different strategies are available to combine the dissimilar decisions into a final
authenticated decision. They range from majority votes to complicated statistical
47
methods. In practice, however, developers seem to prefer the easiest method: Boolean
Conjunctions. Fusion at the decision level occurs at a very late stage of the authentication
process.
2.8 Summary
In this chapter we have discussed all about the face recognition and detection system.
And we also discussed the face detection approaches and also about the image
processing. In this chapter, we compared the fusion of face with other biometric traits.
Multi modal biometric systems integrate the information presented by multiple biometric
indicators. The information can be consolidated at various levels such as image level,
feature extraction level, matching score level, rank level and decision level.
Fusion at the image level: In this level of fusion, we combine only compatible images
because they can be easily combined. Also, we capture the multiple images of the same
biometric traits and then fuse all the images to get a single image of higher quality. This
method is used to increase the acquired facial image area wherein each individual image
has captured only a portion of the face.
Fusion at the feature extraction level: The data obtained from each biometric modality
is used to compute a feature vector. If the features extracted from one biometric indicator
are independent of those extracted from the other, it is reasonable to concatenate the two
vectors into a single new vector, provided the features from different biometric indicators
are in the same type of measurement scale.
Fusion at the rank level: Rank level fusion involves combining identification ranks
obtained from multiple unimodal biometrics. It consolidates a rank that is used for
making final decision. In multi modal biometric system, rank level fusion is the method
of consolidating ranks from different biometric modalities to recognize a user.
Fusion at the matching scores level: Each biometric matcher provides a similarity score
indicating the proximity of the input feature vector with the template feature vector.
These scores can be combined to assert the veracity of the claimed identity.
Fusion at the decision level: each biometric system makes its own recognition decision
based on its own feature vector. A majority vote scheme can be used to make the final
recognition decision.
48
Chapter 3
Literature Survey
For the purpose of this dissertation, the literature survey covers a period of 2000 to 2013.
The dissertation work on “Face and Facial Expression Recognition” is divided into five
main categories:
1. Technology, Application, Challenge and computational Intelligence Solutions.
2. Security issues in biometric Authentication.
3. Emerging methods of Biometric user Authentication.
4. Performance issues in Biometric Authentication Systems.
5. Comparison of various Biometric Authentication Systems.
Anil K. Jain, Arun Ross and Salil Prabhakar in their paper “An Introduction to
Biometrics Recognition” designed a Biometric Recognition system using the four main
modules [19]:
1. Sensor module, which captures the biometric data of an individual.
2. Feature extraction module, in which the acquired biometric data is processed to
extract a set of salient or discriminatory features.
3. Matcher module, in which the features during recognition are compared against the
stored templates to generate matching scores.
4. System database module, which is used store the biometric templates of the enrolled
users.
Weicheng Shen and Tieniu Tan in their paper “Automated Biometrics-based personal
Identification”, proposed a typical automated biometrics-based identification/verification
system that consists of the following major components [20]:
1. Data acquisition component that acquires the biometric data in digital format by
using a sensor.
49
2. Feature extraction component that uses an algorithm to produce a feature vector in
which the components are numerical characterizations of the underlying biometrics.
The feature vectors are designed to characterize the biometrics so that biometric
data collected from one individual, at different times, are “similar”, while those
collected from different individuals are “dissimilar”. In general, the larger the size
of a feature vector (without much redundancy), the higher will be its discrimination
power which is defined as the difference between a pair of feature vectors
representing two different individuals.
3. Matcher component that compares feature vectors obtained from the feature
extraction algorithm to produce a similarity score. This score indicates the degree of
similarity between a pair of biometrics data under consideration.
4. Decision-maker component is the last component of the system that finally
provides/rejects access to the user based on some pre-determined criterion.
Michal Chora in his paper “Emerging Methods of Biometric Human Identification” gives
the idea that Human identification biometrics system may originate from real-life
criminal and forensic applications. Some methods, such as fingerprinting and face
recognition, already proved to be very efficient in computer vision based human
recognition systems. In this paper he focuses on emerging methods of biometrics human
identification also used in the forensic and criminal practice. He says it is our motivation
to design computer vision system based on innovative computing and image processing
that would be used to identify humans on the basis of their ear, palm and lips images
[21]. Those methods have been successfully used by the police and forensic experts and
now gain attention of computer science community. In the article he proposed innovative
methods of ear, lips and palm print biometrics to identify humans.
Qinghan Xiao in his paper “Biometrics Technology, Application, Challenge and
Computational Intelligence Solutions” states that an increasing number of countries have
decided to adopt biometric systems for national security and identity theft prevention
[22]. This trend makes biometrics an important component in security related
applications such as: logical and physical access control, forensic investigation, IT
security, identity fraud detection and terrorist prevention or detection. This paper aims to
50
assist readers as they consider biometric solutions by examining common biometric
technologies, introducing different biometric applications.
Arun Ross, Anil Jain in their paper “Information Fusion in Biometrics” proposed a MultiBiometric system which seeks to alleviate some of drawbacks of single biometric system
by providing multiple evidences of the same identity [23]. These systems help to achieve
an increase in performance that may not be possible using a single biometric indicator.
Multi-Biometric systems provide anti-spoofing measures by making it difficult for an
intruder to spoof multiple biometric traits simultaneously. The authors proposed Fusion
in biometrics with certain levels.
Sanjay Kr. Singh, D. S. Chauhan, Mayank Vatsa, Richa Singh in their paper “A Robust
Skin Color Based Face Detection Algorithm” described a detailed experimental study of
face detection algorithms based on “Skin Color” has been made [24]. The colors of the
human skin represent a special category of colors, because they are distinctive from the
colors of other natural objects. This category is found as a cluster in color spaces, and the
skin color variations between people are mostly due to the intensity differences. Besides,
the face detection based on skin color detection is a faster method as compared to other
techniques. In this paper, the author present a system to track faces by carrying out skin
color detection in four different color spaces: HSI, YCbCr, YES and RGB. Once some
skin color regions have been detected for each color space, author labeled each region
and gets some characteristics such as size and position. The authors were supposing that a
face is located in one the detected regions. After that they compare and employ a polling
strategy between labeled regions to determine the final region where the face effectively
has been detected and located. The authors have compared the algorithms based on these
color spaces and have combined them to get a new skin color based face detection
algorithm which gives higher accuracy. In this paper, the author includes some
experimental results which show that the proposed algorithm is good enough to localize a
human face in an image with an accuracy of 95.18%.
George Chellin, Chandran J and Dr. Rajesh R.S in their paper “Performance Analysis of
Multimodal Biometric System Authentication” states that Traditional identity verification
in computer systems are done which is based on the Knowledge based and Token based
51
identification are leading to fraud. Unfortunately, these may often be forgotten, disclosed
or changed [25]. A reliable and accurate identification/verification technique may be
designed using biometric technologies. Biometric authentication employs unique
combinations of measurable physical characteristics- fingerprint, facial features, iris of
the eye, voice print, hand geometry, vein patterns and so on that cannot be readily
imitated or forged by others. Unimodal biometric systems have variety of problems such
as noisy data, intra-class variations, restricted degree of freedom, non-universality, spoof
attacks and unacceptable error rates. Multimodal biometrics refers the combination of
two or more biometric modalities in a single identification system. The purpose of this
paper is to identify whether the integration of iris and fingerprint biometrics overcome
the hurdles of unimodal biometric system. This paper discuss the various scenarios that
are possible to improve the performance of multimodal biometric systems using the
combined characteristics such as iris and fingerprint, the level of fusion (multimodal
fusion) is applied to that are possible and the integration strategies that can be adopted in
order to increase the overall system performance. Information from multiple sources can
be consolidated in three distinct levels: (i) feature extraction level; (ii) match score level;
(iii) measurement level; and (iv) decision level.
J. D. Woodward in his paper “Biometrics: Privacy’s Foe or Privacy’s Friend?” states that
both the public and private sectors are making extensive use of biometrics for human
recognition [26]. As this technology becomes more economically viable and technically
perfected, and thus more commonplace, the field of biometrics will spark legal and policy
concerns. Critics inevitably compare biometrics to Big Brother and the loss of individual
privacy. The biometric lobby generally stresses the greater security and improved service
that the technology provides. Is biometrics privacy’s friend or privacy’s foe? This paper
explains the various arguments for and against biometrics and contends that while
biometrics may pose legitimate privacy concerns, these issues can be adequately
addressed. In the final analysis, biometrics emerges as privacy’s friend.
L. Hong and A. K. Jain in their paper “Integrating Faces and Fingerprints for Personal
Identification” describes that an automatic personal identification system based only on
the fingerprints or faces is often not able to meet the requirements of the system’s
52
performance i.e. response time requirements as well as the accuracy requirements [27].
Face recognition is fast technology but not extremely reliable, while fingerprint
verification is reliable but inefficient in database retrieval. In this paper, the authors
developed a prototype biometric system which integrates faces and fingerprints. The
system overcomes the limitations of face recognition systems as well as fingerprint
verification systems. The integrated prototype system operates in the identification mode
with an admissible response time. The author describes that the identity established by
the system is more trustworthy than the identity established by a face recognition system.
In addition, the proposed decision fusion scheme makes the performance improvement by
integrating multiple cues with different confidence measures. Experimental results
demonstrate that this system performs very well. It meets the response time as well as the
accuracy requirements. Experimental results demonstrate that the system performs very
well. It meets the response time as well as the accuracy requirements.
Jun Ou, Xiao-Bo Bai Yun Pei ,Liang Ma, Wei Liu [28] in their paper “Automatic Facial
Expression Recognition Using Gabor Filter And Expression Analysis” presents a system
that uses 28 facial feature key-points in images detection and Gabor wavelet filter
provided with 5 frequencies, and 8 orientations. In according to actual demand, the
system can actually extract the feature of low quality facial expression image target, and
is much robust for automatic facial expression recognition. The authors shows some
experimental results in their paper which shows that the performance of the proposed
method achieved excellent average recognition rates, when it is applied to facial
expression recognition system.
Ming Hu in his paper “Application of Rough Sets to Image Pre-processing for Face
Detection” explains that both of face detection and face recognition play a very important
role in the technology of biological character identity [29]. During the processing of face
image, it is indispensable to pre-processing. In this paper, a Pre-processing method for
face color image based on rough sets theory is put forward. In this method, the image is
divided as foreground and background by condition attribution. In order to extract the
skin color regions, foreground is segmented by two kinds of the skin color model, one is
53
YCbCr and the other is HSV color spaces. Then, the opening and closing morphological
operation are used to remove the small regions.
Jiying Wu in his paper “Multi-Scale Preprocessing Model for Face Recognition”
proposed a novel multi-scale preprocessing model (MSPM) for face recognition [30].
MSPM removes lighting effects and enhances the image feature in two scales
simultaneously. It decomposes the original image using Total Variation model. Then the
lighting effects are normalized by self-quotient in the small scale part and equalized in
the large scale part.
Bongjin Jun in his paper “Statistical Face Image Preprocessing and Non-statistical Face
Representation for Practical Face Recognition” states that recognizing face images in real
environment is still a challenging problem since there are many illumination changes
[31]. In this paper, the author proposed a practical face recognition method that combines
statistical global illumination transformation and non-statistical local face representation
method. When a new face image is given, it is transformed into a number of face images
exhibiting different illuminations using a statistical bilinear model-based indirect
illumination transformation. Each illumination transformed image is then represented by
a histogram sequence that concatenates the histograms of the non-statistical multiresolution uniform local Gabor binary patterns (MULGBP) for all the local regions.
Yong Zhang in his paper, “Hand-Drawn Face Sketch Recognition by Humans and a
PCA-Based Algorithm for Forensic Applications” describes that face sketches represent
the original faces in a very concise manner but they are still in a recognizable form,
therefore they play an important role in criminal investigations, human visual perception,
and face biometrics [32]. In this paper, the author compared the performances of humans
and a principle component analysis (PCA)-based algorithm in recognizing face sketches.
A total of 250 sketches of 50 subjects were involved. All of the sketches were drawn
manually by five artists (each artist drew 50 sketches, one for each subject).
Anil K. Jain, Arun Ross, Umut Uludag in their paper “Biometric Template Security:
Challenges and Solutions” explains their ideas that a biometric system is vulnerable to a
variety of attacks aimed at breaking the integrity of the authentication process [33]. These
54
attacks are intended to either circumvent the security afforded by the system or to check
the normal functioning of the system. In this paper, the authors describe the various
threats that can be faced by a biometric system. Therefore, the authors specifically focus
on attacks designed to elicit information about the original biometric data of an individual
from the stored template. Cancelable biometrics may be used to “reset” the biometric
template of a user in the event that the user’s template is compromised. Also, biometric
cryptosystems can also provide contributions to the template security scheme by
supporting biometric matching in secure cryptographic domains.
Jian Yang in his paper “Constructing PCA Baseline Algorithms to Reevaluate ICA-Based
Face-Recognition Performance” states that an independent component analysis (ICA)based face recognition generally evaluates its performance using standard principal
component analysis (PCA) within two architectures, ICA Architecture I and ICA
Architecture II [34]. In this paper, the author analyzes these two ICA architectures and
find that ICA Architecture I involves a vertically centered PCA process (PCA I), while
ICA Architecture II involves a whitened horizontally centered PCA process (PCA II).
Thus, it makes sense to use these two PCA versions as baselines to re-evaluate the
performance of ICA-based face-recognition systems.
Walid Riad Boukabou in his paper “An Improved LDA Approach with DFB Processing
for Face Recognition” states that recently many face recognition systems were developed
[35]. However changing in pose and illumination conditions remains largely unsolved.
Therefore in this paper, the author presents a new method for face recognition that was
based on Linear Discriminant Analysis with a directional filter bank pre-processing. In
this paper, the author shows some experimental results which shows that the DFB preprocessing step improves the original LDA method and a very good recognition rate is
obtained which leads to performance improvement.
Bruce A. Draper in his paper, “Recognizing Faces with PCA and ICA” compares the
principal component analysis (PCA) and independent component analysis (ICA) in the
context of a baseline face recognition system, and a comparison is motivated by
contradictory claims in the literature [36]. This paper shows how the relative performance
of PCA and ICA depends on the task statement, the ICA architecture, the ICA algorithm,
55
and (for PCA) the subspace distance metric. It then also explains the space of PCA/ICA
comparisons by systematically testing two ICA algorithms and two ICA architectures
against PCA with four different distance measures on two tasks (facial identity and facial
expression). In the process, this paper verifies the results of many of the previous
comparisons in the literature, and relates them to each other and to this work. Author are
able to show that the Fast ICA algorithm configured according to ICA architecture II
yields the highest performance for identifying faces, while the Info Max algorithm
configured according to ICA architecture II is better for recognizing facial actions. In
both cases, PCA performs well but not as well as ICA.
Rabia Jafri and Hamid R. Arabnia in their paper “A Survey of Face Recognition
Techniques” stated that Facial recognition system presents a very challenging problem in
the field of the image analysis and computer vision [37], and it also has been received a
great deal of attention over the last few decades because of its many applications in the
various fields. According to them, Face recognition techniques can be broadly divided
into three main categories based on the facial data acquisition methodology. 1) Methods
that operate on intensity images. 2) Those that deal with video sequences, 3) and those
that require other sensory data such as 3D information of images or infra-red imagery.
This paper also defines the need of using the face recognition technology. In this paper,
an overview of some of the well-known methods in each of these categories is provided
and some of the benefits and drawbacks of the schemes mentioned are also examined.
Furthermore, a discussion outlining the benefits for using face recognition, the
applications of this technology, and some of the difficulties plaguing current systems
with regard to this task has also been provided. This paper also discusses some of the
most recent algorithms that was developed for this purpose and tries to give an idea of the
state of the art for the face recognition technology.
Hu Han and Anil K. Jain in their paper “3D Face Texture Modeling from Uncalibrated
Frontal and Profile Images” reveals that 3D face modeling from 2D face images is of
significant importance for facial image analysis, animation and also for recognition [38].
Study tells that previous research on this topic mainly focused on 3D face modeling from
a single 2D face image; however, a single face image can only provide a limited
56
description of a 3D face. In many applications, for example, law enforcement, multi-view
face images are usually captured for a subject during enrollment, which makes it
optimistically build a 3D face texture model, given a pair of frontal and profile facial
images. The author first calculated the relationship between the uncalibrated frontal and
profile face images through facial landmark alignment. An initial 3D face shape is then
reconstructed from the frontal face image, followed by shape refinement utilizing the
depth information provided by the profile image. Finally, face texture is extracted by
mapping the frontal face image on the recovered 3D face shape. In this paper, the
proposed method is utilized for 2D face recognition in two scenarios: one is the
normalization of probe image, and the other one is enhancing the representation
capability of gallery set. Experimental results comparing the proposed method with a
state-of-the-art commercial face matcher and densely sampled LBP on a subset of the
FERET database show the effectiveness of the proposed 3D face texture model.
Hu Han, Charles Otto, and Anil K. Jain in their paper “Age Estimation from Face
Images: Human vs. Machine Performance” states that there has been a growing interest in
automatic age estimation from facial images due to a variety of potential applications in
law enforcement agencies, security control applications, and human computer interaction
(HCI) [39]. However, rather than of these advances in automatic age estimation, it still
remains a challenging problem. This is because the facial aging process is determined not
only by internal factors, e.g. genetic factors, but also some external factors, for e.g.
lifestyle, expression, and environment contributes in the determination process.
Therefore, as a result of it, different people which are of the same age can have quite
different appearances due to different rates of facial aging. In this paper, the authors
proposed a hierarchical approach for automatic age estimation, and provide an analysis of
how aging influences individual facial components. In this paper, the author also shows
some experimental results on the FG-NET, MORPH Album2, and PCSO databases
which show that eyes and nose contains much more information than the other
components and these are also the more stable components than the other facial
components in automatic age estimation. They also studies the ability of humans to
estimate the age using data collected via crowd sourcing or by their ability to remember,
57
and shows that the cumulative score (CS) within 5-year mean absolute error (MAE) of
their method is better than the age estimates provided by human beings.
Brendan F. Klare and Anil K. Jain in their paper “Face Recognition: Impostor-based
Measures of Uniqueness and Quality” presents a framework, named uniqueness-based
non-match estimates (UNE), which explores the ability to improve the face recognition
performance of any face matcher [40]. The first goal of this framework is that a novel
metric that measures the uniqueness of a given user, called the impostor-based
uniqueness measure (IUM). The UNE maps the facial matching scores from any facial
matcher into non-match probability estimates that are conditionally dependent on the
probe image’s IUM. In this paper by using this framework the author justifies that (i) an
improved generalization of matching thresholds, (ii) a score normalization technique that
improves the interoperability for users of different face matchers, and (iii) the predictive
ability of IUM towards the facial recognition accuracy. Studies are conducted on an
operational dataset with 16,000 subjects using three different face matchers in which two
are the commercial, and one is proprietary matcher to illustrate the effectiveness of the
proposed framework.
Zhifeng Li, Unsang Park and Anil K. Jain, in their paper “A Discriminative Model for
Age Invariant Face Recognition” stated that aging variation puts a very challenging
problem to the automatic face recognition systems [41]. Most of the face recognition
studies that have addressed the aging problem are mainly focused on the age estimation
or aging simulation. Designing an appropriate feature representation and an effective
matching framework for age invariant face recognition still remains a big challenging
problem. In this paper, the author proposed a discriminative model to address face
matching in the presence of age variability. In this framework, the author first represent
each face by designing a densely sampled local feature description scheme, in which
scale invariant feature transform (SIFT) and multi-scale local binary patterns (MLBP)
serve as the local descriptors. By densely sampling the two kinds of local descriptors
from the entire facial image, sufficient discriminatory information, including the
distribution of the edge direction in the face image (that is expected to be age invariant)
can be extracted for further analysis. Since both SIFT-based local features and MLBP58
based local features span a high-dimensional feature space, to avoid the over fitting
problem, the author developed an algorithm, called multi-feature discriminant analysis
(MFDA) to process these two local feature spaces in a unified framework. The MFDA is
an extension and improvement of the LDA using multiple features combined with two
different random sampling methods in feature and sample space. By random sampling the
training set as well as the feature space, multiple LDA-based classifiers are constructed
and then combined to generate a robust decision via a fusion rule. The author shows some
experimental results which show that this approach outperforms a state-of-the-art
commercial face recognition engine on two public domain facial aging data sets:
MORPH and FG-NET. They also compared the performance of the proposed
discriminative model with a generative aging model. A fusion of discriminative and
generative models further improves the facial matching accuracy still in the presence of
aging.
Volker Blanz and Thomas Vetter in their paper “Face Recognition Based on Fitting a 3D
Morphable Model” presents a method for face recognition across the many variations in
pose, ranging from frontal to profile views, and across a wide range of illuminations and
changing lighting conditions, including cast shadows and peculiar reflections [42]. To
account for these variations, the algorithm simulates the process of image formation in
3D space, using computer graphics, and it estimates 3D shape and texture of faces from
single images. These estimations are achieved by fitting a statistical, morph able model of
3D faces to images. This model is then trained from using a set of textured 3D scans of
heads. The author describes the construction of the morph able model, an algorithm to fit
the model to images, and a framework for face identification. In this framework, faces are
represented by model parameters for 3D shape and texture. The author also presents the
results obtained with 4,488 images from the publicly available CMU-PIE database and
1,940 images from the FERET database.
Zahid Riaz, Christoph Mayer, Matthias Wimmer, and Bernd Radig in their paper “Model
Based Face Recognition across Facial Expressions” describes a novel idea of face
recognition across facial expression variations using a model-based approach [43]. This
approach is followed in 1) modeling an active appearance model (AAM) for the face
59
image, 2) using optical flow based temporal features for facial expression variability
estimation, 3) and finally applying binary decision trees as a classifier for facial
identification. The novelty lies not only in the generation of appearance models which is
obtained by fitting active shape model (ASM) to the face image using objective but also
using a feature vector which is the combination of shape, texture and temporal parameters
that is robust against facial expression variability. Many experiments have been
performed on Cohn-Kanade facial expression database using 61 subjects of the database
with image sequences consisting of more than 4000 images. The author shows that this
approach has been achieved successful recognition rate up to 91.17% using decision tree
as a classifier in the presence of six different facial expressions.
A.A. Salah, M. Biceg, L. Akarun, E. Grosso, M. Tistarelli in their paper “Hidden Markov
Model-based face recognition using selective attention” describes that the sequential
methods for face recognition relies upon the analysis of local facial features in a
sequential manner, typically with a raster scan [44]. However, the distribution of
discriminative information is not uniform over the facial surface. For example, the nose
and the mouth contain much more information than the cheek or other facial components.
In this paper, the author proposed an extension to the sequential approach, where they
takes into account the local facial features saliency, and replace the raster scan with a
guided scan that mimics the scan path of the human eye. The selective attention
mechanism that guides the human eye operates by coarsely detecting salient locations,
and directing more resources at some interesting or informative parts. The author
simulates this idea by employing a computationally cheap saliency scheme, based on
Gabor wavelet filters. Hidden Markov models are used for classification, and the
observations, i.e. features obtained with the simulation of the scan path, are modeled with
Gaussian distributions at each state of the model. In this paper, the author shows that by
firstly visiting the important locations, this method is able to reach high accuracy with
much shorter feature sequences. The author compared the several features in observation
sequences, among which DCT coefficients results in the highest accuracy.
60
Congyong Su and Li Huang in their paper “Spatio-Temporal Graphical-Model-Based
Multiple Facial Feature Tracking” proposed a spatio-temporal graphical model for
multiple facial feature tracking [45]. Here the graphical model that is described above is
not 2D or 3D facial mesh model. Nonparametric belief propagation is used to infer facial
feature’s interrelationships in a part-based face model, allowing positions and states of
some features in clutter to be recovered. Facial structure is also taken into account,
because facial features have spatial position constraints. The problem is to track multiple
facial features simultaneously when rich expressions are presented on a face. In this
paper, the author proposed a two step solution. In the first step, several independent
condensation-style particle filters are utilized to track each facial feature in the temporal
domain. Particle filters are very effective for visual tracking problems; however multiple
independent trackers ignore the spatial constraints and the natural relationships among
facial features. In the second step, the author use the Bayesian inference belief
propagation to infer each facial feature’s contour in the spatial domain, in which the
author explains the relationships among contours of facial features beforehand with the
help of a large facial expression database. The tracking results in this paper can be used
as motion capture data. Author has planned to use these data to derive a 3D face model
and generate facial animations. The ultimate purpose of multiple facial feature tracking is
for facial animation.
Yan Tong, Wenhui Liao and Qiang Ji in their paper “Facial Action Unit Recognition by
Exploiting their Dynamic and Semantic Relationships” describes that a system could
automatically analyze the facial actions in real time has applications in a wide range of
different fields [46]. However, developing such a system is always challenging due to the
richness, ambiguity, and dynamic nature of facial actions. Although a number of research
groups attempt to recognize facial action units (AUs) by improving either the facial
feature extraction techniques or the AU classification techniques, these methods often
recognize AUs or certain AU combinations individually and statically, ignoring the
semantic relationships among AUs and the dynamics of AUs. Hence, these approaches
cannot always recognize AUs reliably, robustly, and consistently. In this paper, we
propose a novel approach that systematically accounts for the relationships among AUs
and their temporal evolutions for AU recognition. Specifically, we use a dynamic
61
Bayesian network (DBN) to model the relationships among different AUs. The DBN
provides a coherent and unified hierarchical probabilistic framework to represent
probabilistic relationships among various AUs and to account for the temporal changes in
facial action development. Within the system developed by the author, robust computer
vision techniques are used to obtain AU measurements. Such AU measurements are then
applied as evidence to the DBN for inferring various AUs. The author includes some
experiments in his paper which shows that the integration of AU relationships and AU
dynamics with AU measurements yields significant improvement of AU recognition,
especially for spontaneous facial expressions and under more realistic environment
including illumination variation, face pose variation, and occlusion.
Maja Pantic, and Leon J. M. Rothkrantz in their paper “Facial Action Recognition for
Facial Expression Analysis from Static Face Images” discuss the automatic recognition
of facial gestures for e.g., facial muscle activity is rapidly becoming an area of intense
interest in the research field of machine vision [47]. In this paper, the author present an
automated system that was developed to recognize facial gestures in static, frontal- and/or
profile-view color face images. A multi detector approach to facial feature localization is
utilized to spatially sample the profile contour and the contours of the facial components
such as the eyes and the mouth. From the extracted contours of the facial features, we
extract ten profile-contour fiducial points and 19 fiducial points of the contours of the
facial components. Based on these, 32 individual facial muscle actions (AUs) occurring
alone or in combination are recognized using rule-based reasoning. With each scored AU,
the utilized algorithm associates a factor denoting the probability with which the pertinent
AU has been scored and a recognition rate of 86% is achieved.
Michael A. Sayette, Jeffrey F. Cohn, Joan M. Wertz, Michael A. Perrott, and Dominic J.
Parrott in their paper “A Psychometric Evaluation of the Facial Action Coding System
for Assessing Spontaneous Expression” describes that the Facial Action Coding System
(FACS) (Ekman & Friesen, 1978) is a comprehensive and widely used method of
objectively describing the facial activities [48]. Little is known, however, about interobserver reliability in coding the occurrence, intensity, and timing of individual FACS
action units. The present study evaluated the reliability of these measures. Observational
62
data came from three independent laboratory studies designed to elicit a wide range of
spontaneous expressions of emotion. Emotion challenges included olfactory stimulation,
social stress, and cues related to nicotine craving. Facial behavior was video-recorded and
independently scored by two FACS-certified coders. Overall, we found good to excellent
reliability for the occurrence, intensity, and timing of individual action units and for
corresponding measures of more global emotion-specified combinations.
Ying-li Tian, Takeo Kanade and Jeffrey F. Cohn in their paper “Recognizing Action
Units for Facial Expression Analysis” describes that most automatic expression analysis
systems attempt to recognize a small set of prototypic expressions, such as happiness,
anger, disgust, sadness, surprise, and fear [49]. Such prototypic expressions, however,
occur rather infrequently. Human emotions and intentions are more often communicated
by changes in one or a few discrete facial features. In this paper, the authors developed an
Automatic Face Analysis (AFA) system to analyze facial expressions based on both
permanent facial features like brows, eyes, mouth and transient facial features such as
deepening of facial furrows in a nearly frontal-view face image sequence. The AFA
system recognizes fine-grained changes in facial expression into action units (AUs) of the
Facial Action Coding System (FACS), instead of a few prototypic expressions. Multistate face and facial component models are proposed for tracking and modeling the
various facial features, including lips, eyes, brows, cheeks, and furrows. During tracking,
detailed parametric descriptions of the facial features are extracted. With these
parameters as the inputs, a group of action units (neutral expression, 6 upper face AUs,
and 10 lower face AUs) are recognized whether they occur alone or in combinations. The
system has achieved average recognition rates of 96.4% (95.4% if neutral expressions are
excluded) for upper face AUs and 96.7% (95.6% with neutral expressions excluded) for
lower face AUs. The generalizability of the system has been tested by using independent
image databases collected and FACS-coded for ground-truth by different research teams.
Alexander M. Bronstein, Michael M. Bronstein and Ron Kimmel in their paper “ThreeDimensional Face Recognition” presented an expression-invariant 3D face recognition
approach [50]. In this paper the basic assumption was that the facial expressions can be
modeled as isometrics of the facial surface. This allows constructing an expression63
invariant representation of faces using the canonical forms approach. The result is an
efficient and accurate face recognition algorithm, robust to facial expressions that can
distinguish between identical twins. The author explains a prototyping system that was
based on the proposed algorithm and compares its performance to classical face
recognition methods. The numerical methods employed by our approach do not require
the facial surface explicitly. The surface gradients field, or the surface metric, is sufficient
for constructing the expression-invariant representation of any given face. It allows us to
perform the 3D face recognition task while avoiding the surface reconstruction stage.
Claude C. Chibelushi, Fabrice Bourel in their paper “Facial Expression Recognition: A
Brief Tutorial Overview” describes that facial expressions convey non-verbal cues,
which play an important role in interpersonal relationships [51]. Automatic facial
expressions recognitions can be an important component of natural human-machine
interfaces; it may also be used in behavioral science and in clinical practice.
Although humans recognize facial expressions virtually without effort or delay,
reliable expression recognition by machine is still a challenging problem. This
paper presents a high-level overview of automatic expression recognition; it
highlights the main system components and also some research challenges.
Liwei Wang, Yan Zhang, Jufu Feng in their paper “On the Euclidean Distance of
Images” presents a new Euclidean distance for images, which we call IMage Euclidean
Distance (IMED) [52]. Unlike the traditional Euclidean distance, IMED takes into
account the inter-relationships of the pixels. Therefore it is robust to small perturbation of
images. In this paper, the author argues that that IMED is the only intuitively reasonable
Euclidean distance for images. IMED is then applied to image recognition. The key
advantage of this distance measure is that it can be embedded in most image
classification techniques such as SVM, LDA and PCA. The embedding is rather efficient
by involving a transformation referred to as Standardizing Transform (ST). We show that
ST is a transform domain smoothing. Using the Face Recognition Technology (FERET)
database and two state-of-the-art face identification algorithms, the author justifies that a
consistent performance improvement of the algorithms embedded with the new metric
over their original versions.
64
Zhen Leiy, Shengcai Liaoz, Anil K. Jain, and Stan Z. Liy in their paper “Coupled
Discriminant Analysis for Heterogeneous Face Recognition” describes that coupled space
learning is an effective framework for heterogeneous face recognition [53]. In this paper,
the author proposed a novel coupled discriminant analysis method to improve the
heterogeneous face recognition performance. In this paper the author describes the two
main advantages of the proposed method. First, all samples from different modalities are
used to represent the coupled projections, so that sufficient discriminative information
could be extracted. Second, the locality information in kernel space is incorporated into
the coupled discriminant analysis as a constraint to improve the generalization ability. In
particular, two implementations of locality constraint in kernel space (LCKS) based
coupled discriminant analysis methods, namely LCKS- coupled discriminant analysis
(LCKS-CDA) and LCKS- coupled spectral regression (LCKS-CSR) are presented.
Extensive experiments on three cases of heterogeneous face matching (high vs. low
image resolution, digital photo vs. video image and visible light vs. near infrared)
validate the efficacy of the proposed method.
Farrukh Sayeed, M. Hanmandlu, and A.Q. Ansari in their paper “Face Recognition using
Segmental Euclidean Distance” made an attempt to detect the face using the combination
of integral image along with the cascade structured classifier which is built using Adaboost learning algorithm [54]. The detected faces are then passed through a filtering
process for discarding the non face regions. They are individually split up into five
segments consisting of forehead, eyes, nose, mouth and chin. Each segment is considered
as a separate image and Eigen face also called principal component analysis (PCA)
features of each segment is computed. The faces having a slight pose are also aligned for
proper segmentation. The test image is also segmented similarly and its PCA features are
found. The segmental Euclidean distance classifier is used for matching the test image
with the stored one. The success rate comes out to be 88 per cent on the CG database
created from the databases of California Institute and Georgia Institute. However the
performance of this approach on ORL database with the same features is only 70 per
cent. For the sake of comparison, discrete cosine transforms (DCT) and fuzzy features are
tried on CG and ORL databases but using a well known classifier, support vector
machine (SVM). Results of recognition rate with DCT features on SVM classifier are
65
increased by 3 per cent over those due to PCA features and Euclidean distance classifier
on the CG database. The author justifies the results of recognition are improved up to 96
per cent with fuzzy features on ORL database with SVM.
Shalini Gupta, Mia K. Markey, J. K. Aggarwal, Alan C. Bovik in their paper “Three
dimensional face recognition based on geodesic and Euclidean distances” proposed a
novel method to improve the performance of existing three dimensional (3D) human face
recognition algorithms that employ Euclidean distances between facial fiducially points
as features [55]. In this paper, the author further investigated a novel 3D face recognition
algorithm that employs geodesic and Euclidean distances between facial fiducial points.
The author demonstrates that this algorithm was robust enough to changes in facial
expression. Geodesic and Euclidean distances were calculated between pairs of 25 facial
fiducial points. For the proposed algorithm, geodesic distances and “global curvature”
characteristics, defined as the ratio of geodesic to Euclidean distance between a pairs of
points, were employed as features. The most discriminatory features were selected using
stepwise linear discriminant analysis (LDA). These were projected onto 11 LDA
directions, and face models were matched using the Euclidean distance metric. With a
gallery set containing one image each of 105 subjects and a probe set containing 663
images of the same subjects, the algorithm produced EER=1.4% and a rank 1
RR=98.64%. It performed significantly better than the existing algorithms that were
based on the principal component analysis and LDA applied to face range images. This
method’s verification performance for expressive faces was also significantly better than
an algorithm that employed Euclidean distances between facial fiducial points as features.
Haitao Zhao in their paper, “A Novel Incremental Principal Component Analysis and its
Application for Face Recognition” described the major limitations of existing IPCA
methods is that there is no guarantee on the approximation error [56]. In view of this
limitation, this paper proposed a new IPCA method based on the idea of a singular value
decomposition (SVD) updating algorithm, namely an SVD updating-based IPCA
(SVDU-IPCA) algorithm. In the proposed SVDU-IPCA algorithm, Author has
mathematically proved that the approximation error is bounded. A complexity analysis on
the proposed method is also presented. Another characteristic of the proposed SVDU66
IPCA algorithm is that it can be easily extended to a kernel version. The proposed
method has been evaluated using available public databases, namely FERET, AR, and
Yale B, and applied to existing face-recognition algorithms.
Ayan Chakrabarti in their paper, “Super-Resolution of Face Images Using Kernel PCABased Prior” presented a learning-based method to super-resolve face images using a
kernel principal component analysis-based prior model [57]. A prior probability is
formulated based on the energy lying outside the span of principal components identified
in a higher-dimensional feature space. This is used to regularize the reconstruction of the
high-resolution image.
Braun in his paper, “Machine Learning and Recognition of Faces” described a learning
machine has been tested in a closed-loop system in which a teaching machine performs
the function of error detector [58]. The learning curve for a set of ten photographs of
faces is presented. All the faces are recognized correctly after 250 presentations.
Takeo Kanade, Jeffrey F. Cohn and Yingli Tian in their paper “Comprehensive Database
for Facial Expression Analysis” described that within the past decade, a significant effort
has occurred in developing methods of facial expression analysis [59]. Because most
investigators have used relatively limited data sets, the generalizability of these various
methods remains unknown. In this paper, the author described the problem space for
facial expression analysis, which included the level of description, transitions among
expression, eliciting conditions, reliability and validity of training and test data,
individual differences in subjects, head orientation and scene complexity, image
characteristics, and relation to non-verbal behavior. The author then present the CMUPittsburgh AU-Coded Face Expression Image Database, which currently includes 2105
digitized image sequences from 182 adult subjects of varying ethnicity, performing
multiple tokens of most primary FACS action units. This database is the most
comprehensive test-bed to date for comparative studies of facial expression analysis.
Gaurav Mittal in his paper, “Robust Preprocessing Algorithm for Face Recognition”
discussed that the face recognition includes enhancement and segmentation of face
image, detection of face boundary and facial features, matching of extracted features
67
against the features in a database, and finally recognition of the face [60]. The face
detection algorithms are based on either gray level template matching or computation of
geometric relationships among facial features. Though a number of algorithms are
devised for face recognition, the technology is not matured enough to recognize the face
of a person with and without beard to be the same. The author’s research proposed a
robust algorithm for preprocessing the face image with beard for face recognition.
3.1 Security and Privacy Issues in Biometric Authentication
User authentication is fundamental to the protection of information systems. Biometric
authentication comes in play to deal with these difficulties with traditional password
systems. Potentially, biometric systems can be employed in all applications that need
authentication mechanism and so in all applications that today use passwords, PINs, ID
cards or the like [61].
The security concerns related to biometric authentication can be organized into two
categories: concern about the theoretical basis of biometrics and vulnerability of
biometric authentication system as discussed below:
3.1.1 Biometric System Concerns
An evident class of biometric authentication vulnerabilities is those faced by the system
user, which impacts user’s privacy and may show the way to identify theft or system
compromise [62].
Biometrics is not secret: Technology is willingly available to image faces, fingerprints,
and irises and make recording of voice or signature – without subject’s approval or
understanding. From this point of view, Biometrics is not secret. On the other hand, from
a cryptography or privacy’s point of view, biometric data are often considered to be
private or secret.
Biometrics cannot be revoked: A biometric feature is permanently associated with a
user and a compromised biometric model will compromise all applications that make use
of that biometric. But the user cannot change his/her fingerprint or retinal patterns.
68
Biometrics has secondary uses: If a user uses the same biometric feature in multiple
applications, then the user can be tracked if the organization shares biometric data.
Because of the amount and type of information that are also collected along with the
biometrics record, many end users distinguish biometric authentication as an intrusive
process and express concerns about how the information will be used beyond the original
purpose [63].
How reliably unique the biometrics are? Many people like to think of biometrics as
‘unique’, but they are not, at least not with the level of data we can measure [64].
How universal the biometric are? Now all biometric traits are truly universal. The
National Institute of Standards and Technology (NIST) reported that it is not possible to
obtain a good quality fingerprint from approximately two percent of population due to
disabilities, cuts, bruises, wounds etc. [65].
Biometric traits are not always invariant: The biometric data acquired from a user
during verification will not be identical to the data used for generating the user’s template
during enrollment [66]. Even under the same equipment and same environmental
conditions biometric data collected from the same user are likely not to be identical.
Biometric traits may vary due to fatigue, sickness etc., for example, a person’s voice may
change if he/she catches cold, children’s face, and gait change as they grow up.
3.1.2 Vulnerability of Biometric Authentication System
The security of biometric authentication depends on the vulnerability of underlying
biometric system. Since biometric systems are implemented on server computers, they
are at risk to all cryptographic, viruses and other attacks which affect modern computer
systems [67].
To better understand security issues concerned with biometric authentication, it should be
useful to study user components of a typical biometric system, communication channel
among the components, and their vulnerabilities as shown in figure 3.1.
Figure 3.1 shows the major functional components of a standard biometric system, where
major steps in the process of authentication are marked as A, B, C, and so on. Usually,
69
each presented sample (B) is acquired by a sensor (C) processed via segmentation and
feature extraction (D) algorithms.
Figure 3.1 Block diagram of a typical biometric authentication system
If available a sample quality assessment (E) algorithm is used to indicate a need to
reacquire the sample. Biometric features are encoded into template, which is stored (H) in
database or any secure hardware. For biometric encryption systems, a code or token is
combined with the biometric feature in the template. During enrollment, biometric
samples are connected to a claimed identity (A), and during successive verification or
identification, samples are compared with enrolled samples using matching algorithm (I),
and an identity decision (J) is made either automatically, or by a human being reviewing
biometric system outputs. Andy Adler [68] points out the security issues at each of these
components or steps in a typical biometric authentication system, which is discussed
below in short.
Sample presentation (B): The attacker may inject the false biometric sample into the
system. Such attacks are mounted to avoid detection or masquerade/fraud as another
person. The later attack is typically called spoofing.
Sensor (C): Noise can appear in the acquired biometric data due to environmental factors
such as lights, sound, humidity etc. as well as defective or not properly maintained
sensors [68]. Attacks on the biometric sensor may weaken or replace the sensor
hardware. In many cases, an attack on the sensor would take the form of repeat. Unlike
control or knowledge based authentication, accuracy of biometric authentication is much
70
dependent on sensor device used. For example, a computer keyboard does not shows the
hardware specific characteristic in the typed text. However, keyboard characteristics used
for biometric, key-stroke-based authentication significantly affects the resulting sampled
signal due to physical properties like attenuation, pressure sensitivity, and others.
Biometric signals are exposed to distortions based on sensor characteristics. The problem
of unavailability of identical sensor may be applicable for applications in large areas, as
well as long term considerations, where specific hardware may be no longer available
after some time.
Segmentation (D): Biometric segmentation extracts the image or signal of interest from
the background, and a failure means the system does not detect the existence of
appropriate biometric feature. Segmentation attacks may be used to escape observation or
to generate denial of service (DoS) attack.
Feature extraction and Quality assessment (E): Knowledge of feature extraction or
quality assessment algorithms can used in the biometric authentication system that may
be exploited to design some special features in presented biometric samples to cause
wrong features to be considered.
Template creation (G): One of the common claims is that, since template creation is a
one-way function, it is impossible or infeasible to regenerate the image or signal from the
templates. However, recent research has been shown that regeneration of biometric
samples from templates to be feasible.
Data storage (H): For biometric authentication the size of reference data may become
very large, and due to natural inconsistency of biometric information, it is impossible to
apply discrete mathematical techniques like cryptographic hashes to secure the reference
data. Vulnerabilities of template storage concern modifying the storage (adding,
modifying, or removing templates), copying template data for secondary usage (identity
theft), or tampering the identity to which the biometric is assigned. “The biometric
dilemma is that although biometrics can initially improve security, as traditional
biometric databases become widespread, compromises will finally destroy the
biometrics’ value and usefulness for security”.
71
Matching (I): A biometric matcher calculates the matching score related to the
probability that the two biometric samples are from the same individual. For multimodal
or biometric fusion systems, excessive score in one biometric modality may override the
authority of other modalities. Besides the concern of finding methods to increase the
overall accuracy of multi-modal authentication systems, for example, with respect to the
degree of correlation between different modalities, question of finding a meaningful set
of modalities. Biometric system matchers which are based on Fisher discriminant
strategies calculate the global thresholds based on the between cross-covariance relation,
which may be modified by enrolling particularly created biometric samples.
Decision (J): Biometric decisions are often reviewed by human operator. Such operators
are well known to be susceptible to fatigue or monotony. One of the goals of the DoS
attacks is to force operators to discard the biometric system, or not to completely trust its
output by causing it to produce sufficiently large number of errors. All biometric
authentication methods are based on some statistical measurement of matching and
threshold, which makes the process subject to false classification error. Biometric
authentication still is based on probability of matching and so cannot give complete
verification about certain authentication. Many biometric authentication systems allow
the administrator to configure a threshold level that determines the false acceptance rate
and the false denial rate.
Maltoni and et al. [69] classify vulnerability of biometric authentication system as
follows:
Circumvention is an attack which gains access to the protected resources by a technical
measure to undermine the biometric system. Such an attack may undermine the
underlying computer systems (overriding matcher decision, or replacing database
templates) or may involve replay of valid data. This threat can be cast as a privacy attack,
where the attacker accesses the data for that he was not authorized to access (e.g.,
accessing the medical records of another user) or, as a subversive attack, where the
72
attacker manipulates the system (e.g., changing those records, submitting bogus insurance
claims, etc.). It is possible to avoid a biometric system using spoofed traits. For example,
it is possible to construct “gummy fingers” using lifted fingerprint impressions and utilize
them to circumvent a fingerprint based authentication system.
Covert acquisition (contamination) is use of biometric information captured from the
legal users to access a system. Examples include spoofing via capture and playback of
voice password, and lifting latent fingerprints to construct a mold. Further, the biometric
data associated with a specific application can be used in another unintended application.
For e.g., using a fingerprint for accessing medical records instead of the intended use of
office door access control.
Collusion and Coercion are biometric system vulnerabilities from the legal system
users. The main difference is that, in collusion the legal user is will perhaps by bribe,
while the coerced user is forced through threat or blackmail.
Denial of Service (DoS) is an attack which prevents the legal use of the biometric
system. This can take the form of slowing or stopping the system via overload of requests
or by degrading the performance of the system.
Repudiation is the case where the attacker denies to accesses the system. A corrupt user
may deny his/her actions by claiming that his/her biometric data were stolen. For a
biometric authentication system, an online authentication server that processes access
requests via retrieving templates from a database and performing matching with the
transferred biometric data can be sending with many bogus access requests, to a point
where the server’s computational resources cannot handle valid requests to any more.
73
Chapter 4
Conceptual models for face recognition
A complete face recognition system is composed of five parts: image acquisition,
detection of a face, normalization, feature extraction and matching. The image acquisition
step captures the face images. Face detection step is used to find the position of faces in a
given image. Output of the detection can be location of face region as a whole, and
location of face region with facial features (i.e. eyes, mouth, eyebrow, nose etc.). Face
normalization step is used to eliminate non-face objects in the images to save the
computation time. Feature extraction is used for extracting the information that can be
used to distinguished different subjects, creating a template that represents the most
discriminated features of the face, typically it uses texture information. The
corresponding matching stage calculates the distance between image codes and decides
whether it is a match or recognizes the submitted image from the subjects in the data set.
The process of face recognition is usually divided into five steps:
1: Image acquisition: image acquisition step captures the images of a face including the
front profile, left profile or the right profile of the same person. Image acquisition is
usually done using the CCD camera.
2: Detection of a face: this step is used to find the position of faces in a given image.
Face detection performs locating and extracting face image operations for face
recognition system. Output of the detection can be the location of face region as a whole,
and location of face region with facial features (i.e. eyes, mouth, eyebrow, nose etc.). The
goal of face detection is to determine whether or not there are any faces in the image and,
if present, return the image location and extent of each face.
3: Normalization: create the dimensionality consistent representation of the face image
and also eliminate the non- face objects in the images to save the computation time.
74
4: Feature extraction: this stage is used for extracting the information that can be used
to distinguished different subjects, creating a template that represents the most
discriminant features of the face. Some filtering operations are applied to extract feature
candidates and steps are listed below:

Laplacian of Gaussian Filter on Red channel of candidate.

Contrast correction to improve visibility of filter result.

Average filtering to eliminate small noises.
5: Matching: the features vectors are compared using a similarity measure.
Finally, a decision of high confidence level is made to identify whether the user is an
authentic or not. But here, the main problem is that we are not able to distinguish between
a dummy/fake image and a real image. An illustration of the steps for the face recognition
is shown in figure 4.1.
Figure 4.1 Techniques used in face recognition
75
4.1 Spoofing in Biometric System
Attacker can attack any of the above mentioned stage for unauthorized access to the
system; some of those attacks are like biometric sensor attack which happens at the
beginning of the process. In this type of attack fake biometric data like artificial finger, a
mask over a face or a contact lens on an eye etc. may be presented to the sensor and puts
previously stored genuine biometric information into proper place in the processing
chain. Following are some attacks that can be happen on biometric system [70].

Fake biometric data at sensor: In this type of attack fake biometric trait like
fake gummy finger, an iris printout or a face mask is used for unauthorized
access.

Resubmitting previously stored digitized biometric signals (replay attack): A
digitized biometric signal, which has been previously enrolled and stored in the
database, is again send to the system.

Tampering biometric feature: In this attack extracted features of given
biometric trait changed so that it can be accepted by matcher.

Attack
on
enrollment:
An
enrollment
database
used
in
the
verification/identification process can also be altered like by providing fake
biometric traits.
4.1.1 Spoofing in Face Recognition System
Generally speaking, there are three ways to spoof face recognition:
a. Photograph of a valid user
b. Video of a valid user
c. 3D model of a valid user
Photo attack is the cheapest and easiest spoofing approach, since one's facial image is
usually very easily available for the public, for example, downloaded from the web,
captured unknowingly by a camera. The imposter can rotate, shift and bend the photo
before the camera like a live person to fool the authentication system. It is still a
76
challenging task to detect whether an input face image is from a live person or from a
photograph.
Video spoofing is another big threat to face recognition systems, because it is very
similar to live face and can be shot in front of legal user’s face by a needle camera. It has
many physiological clues that photo does not have, such as head movement, facial
expression.
3D model has 3D information of face, however, it is rigid and lack of physiological
information. It is also not very easy to be realistic with live person who the 3D model
imitates. So photo and video are most common spoofing ways to attack face recognition
system.
In general, human is able to distinguish a live face and a photograph without any effort,
since human can very easily recognize many physiological clues of liveness, for example,
facial expression variation, mouth movement, head rotation, eye change. However, the
tasks of computing these clues are often complicated for computer, even impossible for
some clues under the unconstrained environment. These are some common attack mainly
happened on biometric system. Some of attack can be resolved by maintaining some
security like securing database from alteration. But some attacks like fake biometric data
to the sensor cannot be covered by strong security. So, for this type of attacks Liveness
detection technique is used to ensure that the given biometric sample originates from a
living person and is not artificial.
4.2 Face Image Acquisition
The acquisition module captures a series of ocular images; uses a scheme to evaluate
image quality; and selects one image with sufficient facial features information, which
then undergoes additional processing. One of the major challenges of face recognition
system is to capture a high quality image of the overall person. Image acquisition is
considered the most critical step in the project since all subsequent stages highly depends
on the image quality. In order to accomplish this, we use a CCD camera. We set the
resolution to 640×480, the type of the image to jpeg, and the mode to white or black for
77
greater details. Block diagram of face acquisition using CCD camera is shown below in
figure 4.2.
Figure 4.2 Block Diagram of image acquisition using CCD camera
The camera is situated normally between half a meters to one meter from the subject. The
CCD-cameras job is to take the image from the optical system and convert it into the
electronic data. CCD camera then transfers the value of the different pixels out of the
CCD chip. Read out the voltages from the CCD-chip. Thereafter, the signals of each data
are amplified and sent to an ADC (Analog to Digital Converter).
4.3
Proposed
Face
Recognition
Algorithm
for
Image
Acquisition
4.3.1 Architecture of the proposed approach
A block diagram of the proposed face recognition system is shown in figure 4.3.
The architecture of the proposed approach is divided into two steps:
1. Enrollment: User enrollment is a process that is responsible for registering individuals
in the biometric system storage. During the enrollment process, the biometric
characteristics of a person are first captured by a biometric scanner to produce a sample.
Some systems collect multiple samples of a user and then either select the best image or
fuse multiple images or create a composite template. Then the Euclidean distance test is
78
performed to check the Liveness of a person. If the person is live then the enrollment
process takes the enrollment template and stores it in the system storage along with the
demographic information about the user.
2. Authentication: In this process, the subject does not explicitly claim an identity and
the system compares the feature set against the templates of all the subjects in the system
storage; the output is a candidate list that may be empty or contain one or more identifiers
of matching enrollment templates.
Figure 4.3 Architecture of the proposed approach
The modules of the proposed approach are explained as follows:
1. Image Acquisition: An important and complex step of face recognition is image
acquisition of a very high quality of a person. It is difficult to acquire clear images using
the standard CCD camera with ordinary lighting. In this module, a biometric camera is
79
used to capture the user’s face images. Due to distance or illumination changes,
recognition process may produce different recognition rates. A study on illumination
effects on face recognition showed that lighting the face bottom up makes face
recognition a hard task [71]. Also, a study on the direction of illumination [72] showed
the importance of top lighting; it is easier for humans to recognize faces illuminated from
top to bottom than the faces illuminated from bottom to top. May be, the same genuine
user’s facial features may slightly vary at sunlight and twilight. However, the system
confines the Euclidean distance of the same candidate features to the face discriminator
phase. The image acquisition phase should consider three main aspects, that is to say, the
lighting system, the positioning system, and the physical capturing system. The face
recognition system can work both in outdoor and indoor conditions without any hot spot
of lighting intensities.
2. Euclidean Distance Test: This module is used for checking the person’s liveness and
is further described in the section 4.3.2.
3. Face Normalization: Create the dimensionality consistent representation of the face
image and also eliminate the non- face objects in the images to save the computation
time.
4. Feature Extraction: This stage is used for extracting the information that can be used
to distinguish different subjects, creating a template that represents the most
discriminated features of the face. This is done with the help of the PCA technique.
5. Matching: This stage takes a feature set and an enrollment template as inputs and
computes the similarity between them in terms of a matching score. The matching score
is compared to a system threshold to make the final decision; if the match score is higher
than the threshold, the person is recognized, otherwise not.
4.3.2 Euclidean Distance Test
Biometric features may be fake and illegally used. This is a crucial weakness of the
biometric system. This section aims to ensure that an input image actually originates from
a user instead of face photographs or any other artificial sources. In the proposed work,
80
Euclidean distance test is suggested to overcome this problem. So, here we use the
concept of Euclidean distance.
4.3.2.1 Euclidean distance
Euclidean distance between two points in p-dimensional space is a geometrically shortest
distance on the straight line passing through both the points [73].
For a distance between two p-dimensional features x=(x1, x2,………,xp) and y=(y1,
y2,……yp) the Euclidean distance is defined as follows:
d (x, y)=[ ∑𝑝𝑖=1(xi - yi)²]½
So, if the images are of the same person then their Euclidean distance will be minimum
otherwise they come from the different sources.
4.3.2.2 Algorithm of the proposed approach
The algorithm of this method is elaborated as follows:
Step 1: Capture the same person’s face images under the three different profiles- left,
right and front (at least one from each profile) in any random order. For example, LRF or
it may be FLR etc.
Step 2: Measure the Euclidean distance from the captured face images. If these values
are dissimilar then the image is actually coming from a real source (user), otherwise
artificial sources may have been used.
Euclidean distance is given by:
d (x, y)={∑𝑛𝑖=1(xi - yi)²}½
(1)
Using eq’n (1), we can calculate the difference between the Euclidean distances of the
two different images of the same person.
Td= ∑𝑛𝑖=1 | d (xi, yi) – d (xi+1,yi+1) |
EDT= ∫
𝑇𝑟𝑢𝑒,
𝐹𝑎𝑙𝑠𝑒,
(2)
𝑖𝑓 𝑇𝑑 ≠ 0
𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
(3)
81
Here, EDT is the Euclidean distance test parameter, n is the no of face images taken in
any of the following order i.e., it may be LFR, LRF, FLR, RLF, RFL, FRL whereas L
denotes the left profile and R denotes the right profile and F denotes the front profile of a
person. d (xi, yi) and d(xi+1,yi+1) are also the Euclidean distance of the person under
different profiles. Td is the difference between the Euclidean distances of the two images.
If the Td is not equal to zero that means the person image is real and not coming from a
fake biometric system or any other artificial sources and responding to Euclidean distance
test.
4.3.2.3 Comparison
The proposed work presents a technique for liveness detection which uses the concept of
Euclidean distance. This technique ensures the detection of fake/dummy images and also
verifies that the person is actually alive or not. But the traditional method does not use the
concept of Euclidean distance. Therefore, this method is not able to check that the person
is actually alive or not. May be the person’s image is fake or generated from artificial
sources. The main advantage of the proposed work is that if the person is found fake then
it is detected in the Euclidean distance test module and the further computations are not
done. But in the traditional method, there is no provision for liveness detection and thus it
was not able to detect the liveness of the person.
82
4.4 Summary
This chapter has presented a face recognition system which was tested using two
databases of grayscale face images in order to verify the claimed performance of the face
recognition technology.
The system is to be composed of a number of subsystems, which corresponds to each
stage of a face recognition system. These stages mainly includes image acquisition - to
acquire the image of a face using a biometric camera, face detection - determine whether
or not there are any faces in the image and, if present, return the image location and
extent of each face, normalization – creating a dimensionally consistent representation of
the face region and feature extraction – creating a template containing only the most
discriminated features of the face. The input to the system will be a face image and the
output will be a face template. Spoofing in the biometric system was also discussed in
this chapter. Then, an automatic acquisition algorithm for face image using Euclidean
distances was presented which checks the liveness of the person whether the person is
live or not and we can also check whether the person is fake or real. Finally, a
comparison is also made between the proposed approach and the existing approach to
show the advantages of the proposed approach.
83
Chapter 5
Facial Expression Recognition
5.1 Face Detection/Localization:
The ultimate goal of the face detection/localization is to find an object in an image whose
shape resembles the shape of a face. Many algorithms have been proposed for face
localization and detection, which are based on using shape [74], color information [75],
and also on motion [76]. These methods can be classified into the following four
categories [77]:
A. Knowledge-based methods: These rule-based methods encode human knowledge of
what constitutes a typical face. Usually, the rules capture the relationships between facial
features. These methods were mainly designed for face detection/localization.
B. Feature invariant approaches: These algorithms aim to find structural features that
exist even when the pose, viewpoint, or lighting conditions vary, and then use these to
locate faces. These methods are designed mainly for face detection/localization.
C. Template matching methods: Several standard patterns of a face are stored to
describe the face as a whole or the facial features separately. The correlations between an
input image and the stored patterns are computed for detection. These methods have been
used for both face localization and detection.
D. Appearance-based methods: In contrast to template matching, the models (or
templates) are learned from a set of training images, which should capture the
representative variability of facial appearance. These learned models are then used for
detection. These methods are designed mainly for face detection.
84
5.1.1 Color Models for Skin Color Classification:
The study on skin color classification has gained increasing attention in recent years due
to the active research in content-based image representation. For instance, the ability to
locate image object as a face can be exploited for image coding, editing, indexing or
other user interactivity purposes. Moreover, face localization also provides a good
stepping stone in facial expression studies. It would be fair to say that the most popular
algorithm to face localization is the use of color information, whereby estimating areas
with skin color is often the first vital step of such strategy. Hence, skin color
classification has become an important task. There are four color models namely RGB,
YCbCr, YES and HSI. The same process is applied to each color model to obtain at most
four skin regions. Next, we apply a polling strategy to determine the final region where
the face is located. There are some advantages and disadvantages in each of the
algorithms. Therefore, we have taken the combination of all the four color models to find
the skin region and then from the skin region facial features have been extracted to get
the face from the skin region.
a) RGB Color Space
The RGB color space consists of the three additive primaries: red, green and blue.
Spectral components of these colors combine additively to produce a resultant color. The
RGB model is represented by a 3-dimensional cube with red green and blue at the corners
on each axis (Figure 4.4). Black is at the origin. White is at the opposite end of the cube.
The gray scale follows the line from black to white. In a 24-bit color graphics system
with 8 bits per color channel, red is (255, 0, and 0). On the color cube, it is (1, 0, and 0).
The RGB model simplifies the design of computer graphics systems but is not ideal for
all the applications. The red, green and blue color components are highly correlated. This
makes it difficult to execute some image processing algorithms. Many processing
techniques, such as histogram equalization, work on the intensity component of an image
only. RGB color space is shown in figure 5.1 below:
85
Figure 5.1 RGB color space
b) YCbCr Color Space
YCbCr color space has been defined in response to increasing demands for digital
algorithms in handling video information, and has since become a widely used model in a
digital video. It belongs to the family of television transmission color spaces. The family
includes others such as YUV and YIQ. YCbCr is a digital color system, while YUV and
YIQ are analog spaces for the respective PAL and NTSC systems. These color spaces
separate RGB (Red-Green-Blue) into luminance and chrominance information and are
useful in compression applications however the specification of colors is somewhat
unintuitive. The Recommendation 601 specifies 8 bit (i.e. 0 to 255) coding of YCbCr,
whereby the luminance component Y has an excursion of 219 and an offset of +16.
Figure 5.2 The RGB color space within the YCbCr color space. Colors outside the RGB
cube are not valid.
This coding places the black color at code 16 and places the white color at code 235.
Therefore, by doing so, it reserves the extremes of the range for signal processing foot
86
room and headroom. On the other hand, the chrominance components Cb and Cr have
excursions of +112 and offset of +128, producing a range from 16 to 240 inclusively.
There are some points within the YCbCr color cube that cannot be represented in the
corresponding RGB domain. This causes some difficulty in determining how to correctly
interpret and display some YCbCr signals. To figure out the relationship between the
RGB and YCbCr color space cubes, Fig. 5.2 shows the RGB color space contained
within the YCbCr color space where the color outside the RGB cube are not valid as
mentioned.
c) HSI Color Space
Since hue, saturation and intensity are three properties used to describe color, it seems
logical that there be a corresponding color model, HSI. When using the HSI color space,
you don't need to know what percentage of blue or green is required to produce a color.
You simply adjust the hue to get the color you wish. To change a deep red to pink, adjust
the saturation. To make it darker or lighter, alter the intensity. HIS color space is shown
in figure 5.3 below:
Figure 5.3 HSI color space
87
Many applications use the HSI color model. Machine vision uses HSI color model in
identifying the color of different objects. Image processing applications such as
histogram operations, intensity transformations and convolutions operate only on an
intensity image. These operations are performed with much ease on an image in the HIS
color model.
d) YES color Space
The YES color space is defined as the Y channel is the luminance component, and E, S
channels are the chrominance. The YES color space constitutes a linear transformation
from RGB, free of singularities and provides some computational efficiency. The E and S
channels can be computed from RGB by shifting bits rather than multiplying. However, it
is generally agreed that there does not exist a single color space that is good for all
images [78]. The E and S channels provide a suitable space for recognition of the classes
under consideration based on color in classification applications. The Y channel will be
mainly utilized to model the texture information.
5.1.2 Algorithm for detecting a face
There are four steps for detecting a face in a single image using the color model as
described below:
1. Segmentation: An important task in several vision applications is the image pixel
classification in a discrete number of classes. The objective of segmentation is providing
an effective and real time classification. The first step for segmentation is obtaining a set
of pixel values, which corresponds to the color skin in the images. The set is obtained
manually, selecting a small region directly from a frame where the face is located. Then
this set to define the maximum and minimum skin pixel values for each channel of the
color space. This vector will be employed in subsequent classifications due the skin color
constitutes a regular cluster in the color space. Next, we are classifying the pixels by
applying the thresholding operation to the frame. Thresholding involves the use of the
88
vector values in the corresponding three-dimensional color space. Every vector values
can be considered as a class.
2. Regions: After segmentation is performed, then connect all pixels to produce regular
regions or blobs. This process is exhaustive and can affect the real time performance. The
threshold values have a large influence on the segmentation results. A small threshold
value leads to a large number of small regions while with a large threshold value few
large regions are calculated. Steps of a face detection algorithm are shown in figure 5.4
below:
Figure 5.4 Steps of a face detection algorithm
3. Localization: In this stage firstly detect and locate the face in the frame. Then search
in all labeled regions for the one that satisfies a specific area (an approximately number
of pixels to form a face) and size (high and width considering the region is rectangular).
After the region of interest is located, we obtain the position of the region center.
Dimension and center position of the face region is computed for each color model. In
89
this way, now we get the many regions as color spaces where the detection was
successful for the polling process.
4. Polling: Finally, validate the face detection and position in the frame by polling.
Simply, then examine all the regions, and if there is a region common to at least three
colors spaces, then considered such region as the detected face, in the other way, if the
mentioned condition is not satisfied, the region is discarded and the process is performed
to the next region or the next frame.
5.2 Facial Expression Recognition
Since last three decades of face recognition technology, there exists many commercially
available systems to identify human faces, however face recognition is still an
outstanding challenging problem. This challenge can be attributed to (i) large intrasubject variations such as pose, illumination, expression, and aging is commonly
encountered in face recognition and (ii) large inter-user similarity. Meanwhile this
technology has extended its role to Human-Computer-Interaction (HCI) and HumanRobot- Interaction (HRI). Person identity is one of the key tasks while interacting with
the robots, exploiting the un-attentional system security and authentication of the human
interacting with the system. This problem has been addressed in various scenarios by
researchers resulting in commercially available face recognition systems [79, 80].
However, other higher level applications like facial expression recognition and face
tracking still remain outstanding problem along with person identity. This gives rise to an
idea for generating a framework suitable for solving these issues together.
5.2.1 Facial Action Coding System
A fully automatic facial action coding system was developed using action units which is a
user independent fully automatic system for real time recognition of facial actions.
Ekman and Friesen (1976, 1978) were pioneers in the development of measurement
systems for facial expression. Their system, known as the Facial Action Coding System
90
or FACS, was developed based on a discrete emotions theoretical perspective and is
designed to measure specific facial muscle movements which is shown in figure 4.8. A
second system, EMFACS, is an abbreviated version of FACS that assesses only those
muscle movements believed to be associated with emotional expressions. In developing
these systems, Ekman importantly distinguishes between two different types of
judgments: those made about behavior (measuring sign vehicles) and those that make
inferences about behavior (message judgments). Ekman has argued that measuring
specific facial muscle movements (referred to as action units in FACS) is a descriptive
analysis of behavior, whereas measuring facial expressions such as anger or happiness is
an inferential process whereby assumptions about underlying psychological states are
made.
Figure 5.5 Facial action coding system
In this system following steps are considered to extract the expressions1. Firstly, automatic system is used to detect the face and then the Pain Expression from
an image.
2. Then Gabor filter is applied on the extracted face and Pain Expression. This filter is
used in image processing for edge detection.
3. The selected edges are passed through adaboost (adaptive boosting) to select the
desired features.
4. Then Eigen face (support vector machine) is used to classify the features and arrange
them into action units.
5. The action units are then used to classify different features.
91
The Facial Action Coding System is a human-observer based system designed to detect
subtle changes in facial features. Viewing videotaped facial behavior in slow motion,
trained observers can manually FACS code all possible facial displays, which are referred
to as action units (AU) and may occur individually or in combinations. FACS consists of
44 action units. Thirty are anatomically related to contraction of a specific set of facial
muscles (Table 4.1) [81]. The anatomic basis of the remaining 14 is unspecified (Table
4.2). These 14 are referred to in FACS as miscellaneous actions. Many action units may
be coded as symmetrical or asymmetrical. For action units that vary in intensity, a 5point ordinal scale is used to measure the degree of muscle contraction.
Table 5.1 FACS Action Units
92
FACS itself is purely descriptive and includes no inferential labels. By converting FACS
codes to EMFACS or similar systems, face images may be coded for emotion-specified
expressions (e.g., joy or anger) as well as for more molar categories of positive or
negative emotion [82].
Table 5.2 Miscellaneous Actions Units
5.2.2 Proposed Work
In our proposed work, Face and Facial Expression detection is presented by combining
the Eigen value and Eigen face methods which perform almost as fast as the Eigen face
method but with a significant improved speed. Eigenvectors and Eigen values are
dependent on the concept of orthogonal linear transformation. An Eigenvector is
basically a non-zero vector. The dominant Eigenvector of a matrix is the one
corresponding to the largest Eigen value of that matrix. This dominant Eigenvector is
important for many real world applications.
5.2.2.1 Algorithm of the proposed approach
The proposed work is divided into 3 main stages:
1. Image Preprocessing
2. Training the Database
3. Matching
93
The image preprocessing work mainly consists of four different modules, namely,
histogram equalization, edge detection, thinning, and token generation. The face image is
taken as an input and tokens are produced as output. It includes the conversion of image
to the normalized image as well as to extract the features from the image. The filtration
process includes the adjustment of brightness, contrast, low pass, high pass filtration etc.
The filtration will be done in two phases one for face and other for facial expression. To
enhance the image quality, histogram equalization has been performed.
Figure 5.6 Flow Chart of the proposed approach
It is then followed by the edge detection process. Edge detection plays an important role
in finding out the tokens. It is a terminology in image processing, particularly in the areas
of feature detection and feature extraction, which aims at identifying points in a digital
image at which the image brightness changes sharply or more formally have
discontinuities. After the edge detection, thinning has to be performed. It is applied to
94
reduce the width of an edge to single line. After the thinning process, tokens have been
generated. Tokens divide a data set in to the smallest unit of information used for
subsequent processing. Flow chart of the proposed approach is shown in figure 5.6.
Another task is performed on the available database. The database will be trained by
using Neural Network. The training module stores the token information that comes from
the image pre-processing module. It can be done using any method. Here, neural network
is used. The power of neural networks is realized when a pattern of tokens, during testing,
is given as an input and it identifies the matching pattern it has already learned during
training. The training process will be done only once and after training a feature analysis
based database will be created. The actual match will be performed on this Eigen Face
trained dataset. Again we have to generate two datasets one for the face itself and the
other for the Facial Expression.
Third and the final step are to perform the matching with the database. The match will be
performed on both the face image and the Facial Expression image separately. If both
matches give the results better than the expected value then the person will be
recognized.
5.2.2.2 Eigen Face Method
The motivation behind Eigen faces is that it reduces the dimensionality of the training set,
leaving only those features that are critical for face recognition.
Definition
1. The Eigen faces method looks at the face as a whole.
2. In this method, a collection of face images is used to generate a 2-D gray-scale image
to produce the biometric template.
3. Here, first the face images are processed by the face detector. Then we calculate the
Eigen faces from the training set, keeping only the highest Eigen values.
4. Finally we calculate the corresponding location in weight space for each known
individual, by projecting their face images onto the “face space”.
95
In mathematical terms, the objective is to find the principal components of the
distribution of faces, or the eigenvectors of the covariance matrix of the set of face
images. These eigenvectors can be thought of as a set of features which together
characterize the variation between face images. Each image location contributes more or
less to each eigenvector, so that we can display the eigenvector as a sort of ghostly face
called an Eigen face.
Each face image in the training set can be represented exactly in terms of a linear
combination of the Eigen faces. The number of possible Eigen faces is equal to the
number of face images in the training set. However, the faces can also be approximated
using only the “best” Eigen faces-those that have the largest Eigen values, and which
therefore account for the most variance within the set of face images. The primary reason
for using fewer Eigen faces is computational efficiency. The most meaningful M Eigen
faces span an M-dimensional subspace -“face space”- of all possible images. The Eigen
faces are essentially the basis vectors of the Eigen face decomposition. The idea of using
Eigen faces was motivated by a technique for efficiently representing pictures of faces
using principal component analysis.
The Eigen faces approach for face recognition involves the following initialization
operations:
1. Acquire a set of training images.
2. Calculate the Eigen faces from the training set, keeping only the best M images with
the highest Eigen values. These M images define the “face space”. As new faces are
experienced, the Eigen faces can be updated.
3. Calculate the corresponding distribution in M-dimensional weight space for each
known individual (training image), by projecting their face images onto the face space.
Having initialized the system, the following steps are used to recognize new face images:
1. Given an image to be recognized, calculate a set of weights of the M Eigen faces by
projecting it onto each of the Eigen faces.
96
2. Determine if the image is a face at all by checking to see if the image is sufficiently
close to the face space. If it is a face, classify the weight pattern as either a known person
or as unknown.
3. (Optional) Update the Eigen faces and/or weight patterns.
4. (Optional) Calculate the characteristic weight pattern of the new face image, and
incorporate into the known faces.
5.2.2.3 Calculating Eigen faces
Let a face image (x,y) be a two-dimensional N by N array of intensity values. An image
may also be considered as a vector of dimension N 2 , so that a typical image of size 256
by 256 becomes a vector of dimension 65,536, or equivalently, a point in 65,536dimensional space. An ensemble of images, then, maps to a collection of points in this
huge space.
Images of faces, being similar in overall configuration, will not be randomly distributed
in this huge image space and thus can be described by a relatively low dimensional
subspace. The main idea of the principal component analysis is to find the vector that best
account for the distribution of face images within the entire image space. These vectors
define the subspace of face images, which we call “face space”. Each vector is of length
N 2 , describes an N by N image, and is a linear combination of the original face images.
Because these vectors are the eigenvectors of the covariance matrix corresponding to the
original face images, and because they are face-like in appearance, they are referred to as
“Eigen faces”.
1. The first step is to obtain a set X with M face images. Each image is transformed into a
vector of size N and placed into the set. Therefore, we get a single matrix X of M×N.
2. Read the Input Image I.
3. Normalized the Image Set X as to get it in required format.
4. Calculate the Mean Image from Set S as follows:
µx=1/N ∑𝑁
𝑖=1 𝑋 [m, n]
Where µx is the mean image of the complete matrix set X and m, n are the indices of the
matrix and m=1, 2……..M and n=1, 2……..N.
97
5. Find the Difference between the Mean Image and the Input Image I.
Difference=Mean (µx)-Input Image
6. Calculate the Covariance Matrix for the image.
Cov (x) = ∑𝑁
𝑖=1[µx-Input image]² / N-1
7. Calculate the Eigen Values and Eigen vectors for the image.
8. Calculate the Image Set with lower Euclidean distance.
9. Repeat the Operation on this image set with lower Euclidean distance value to get the
image having the expression closer to the defined image.
10. The image with the lowest Euclidean distance in expression images will be
represented as the resultant expression image.
5.3 Summary
In this chapter, firstly we discussed the existing face detection methods which includes
like Knowledge-based methods, Feature invariant approaches, Template matching
methods, Appearance-based methods. We also discussed the color space for skin color
classification such as RGB color space, YCbCr color space; YES color space, HIS color
space and the algorithm to detect the face region in an image.
Next, a system to recognize the facial expressions was also presented in which the Eigen
face and Principal Component Analysis (PCA) method was used together to find the
principal components of the distribution of faces, or the Eigen vectors of the covariance
matrix of the set of face images. Then the Euclidean distance metric was used as a
decision classifier to recognize the expression of a face like sadness, happy, pain, disgust,
fear etc. And finally, matching is performed on both the face image and the facial
expression image separately. If both the matches give the results better than the expected
value then the person will be recognized.
98
Chapter 6
Face Recognition Applications
There are numerous applications for the face recognition technology. Many applications
for face recognition have been envisaged. Commercial applications have so far only
scratched the surface of the potential. The further sections describe the applications
where the technology is currently being deployed and where it shows some future
potential and these applications are divided into two sections as described below:
6.1 Government Use applications
6.1.1 Law Enforcement: Minimizing victim trauma by narrowing mug-shot searches,
verifying identify for court records, and comparing school surveillance camera images to
known child molesters.
a) Identification Systems
Two US States (Massachusetts and Connecticut [83]) are testing face recognition for the
policing of Welfare benefits. This is an identification task, where any new applicant
being enrolled must be compared against the entire database of previously enrolled
claimants, to ensure that they are not claiming under more than one identity.
Unfortunately face recognition is not currently able to reliably identify one person among
the millions enrolled in a single state’s database, so demographic information such as zip
code, age, name, gender etc. are used to narrow the search (thus limiting its
effectiveness), and human intervention is required to review the false alarms that such a
system will produce. Here a more accurate system such as fingerprint or iris-based person
recognition is more technologically appropriate, but face recognition is chosen because it
is more acceptable and less intrusive. In Connecticut, face recognition is the secondary
biometric added to an existing fingerprint identification system. Several US States,
including Illinois, have also instituted face recognition for ensuring that people do not
obtain multiple driving licenses.
99
6.1.2 Security/Counterterrorism: Access control, comparing surveillance images to
known terrorists. To identify and verify terrorists at airports, railway stations and malls
the face recognition technology will be the best choice in India as compared with other
biometric technologies since other technologies cannot be helpful in crowdy places.
a) Access Control
Face verification, matching a face against a single enrolled template, is well within the
capabilities of current Personal Computer hardware. Since PC cameras have become
widespread, their use for face-based PC logon has become feasible, though take-up seems
to be very limited. Increased ease-of-use over password protection is hard to argue with
today’s somewhat unreliable and unpredictable systems, and for few domains there are
motivations to progress beyond the combinations of password and physical security that
protect most enterprise computers. As biometric systems tend to be third party, software
add-ons the systems do not yet have full access to the greater hardware security
guarantees afforded by boot-time and hard disk passwords. Visionics’ face-based screen
lock is one example, bundled with PC cameras. Naturally such PC-based verification
systems can be extended to control authorization for single-sign-on to multiple networked
services, for access to encrypted documents and transaction authorization, though again
uptake of the technology has been slow. Face verification is being used in kiosk
applications, notably in Mr. Payroll’s (now Innoventry) check-cashing kiosk with no
human supervision. Innoventry claims to have one million enrolled customers. Physical
access control is another domain where face recognition is attractive (e.g. Cognitec’s
Face VACS, Miros’ True Face) and here it can even be used in combination with other
biometrics. Bioid [84] is a system which combines face recognition with speaker
identification and lip motion.
b) Surveillance
The application domain where most interest in face recognition is being shown is
probably surveillance. Video is the medium of choice for surveillance because of the
richness and type of information that it contains and naturally, for applications that
require identification, face recognition is the best biometric for video data. Although gait
or lip motion recognition has some potential. Face recognition can be applied without the
100
subject’s active participation, and indeed without the subject’s knowledge. Automated
face recognition can be applied ‘live’ to search for a watch-list of ‘interesting’ people, or
after the fact using surveillance footage of a crime to search through a database of
suspects. The deployment of face-recognition surveillance systems has already begun,
though the technology is not accurate enough yet [85]. The US government is investing
in improving this technology [86] and while useful levels of recognition accuracy may
take some time to achieve, technologies such as multiple steerable zoom cameras, nonvisible wavelengths and advanced signal processing are likely to bring about superhuman perception in the data gathering side of surveillance systems.
6.1.3 Immigration
The U.S. government has recently begun a program called US-VISIT (United States
Visitor and Immigrant Status Indicator Technology), aimed at foreign travelers gaining
entry to the United States. When a foreign traveler receives his visa, he will submit
fingerprints and have his photograph taken. The fingerprints and photograph are checked
against a database of known criminals and suspected terrorists. When the traveler arrives
in the United States at the port of entry, those same fingerprints and photographs will be
used to verify that the person who received the visa is the same person attempting to gain
entry.
6.1.4 Correctional institutions/prisons
It includes the inmate tracking, and employee access. Keeping track of inmates within a
prison or jail is a constant challenge, especially as they move from one part of the facility
to another. Monitoring their movements requires corrections officers to accurately
identify individual prisoners by sight as they pass through security posts. It also requires
frequent telephone and radio communications between officers at two or more security
posts, paper passes authorizing inmates’ movements, and dry-erase or clip boards with
handwritten records to note when prisoners left one area and entered another.
6.1.5 Legislature
It includes the verification of the identity of congressman prior to vote.
101
6.2 Commercial Use
6.2.1 Banking
It includes the minimizing the fraud by verifying the identity of the person. Banks have
been very conservative in deploying biometrics as they risk losing far more through
customers disaffected by being falsely rejected than they might gain in fraud prevention.
Customers themselves are reluctant to incur burdensome additional security measures
when their personal liability is already limited by law. For better acceptance, robust
passive acquisition systems with very low false rejection probabilities are necessary.
Automated Teller Machines, already often equipped with a camera, have also been an
obvious candidate for face recognition systems for e.g. Face PIN, but development seems
not to have got beyond pilot schemes. In order to prevent the frauds of ATM in India, it is
recommended to prepare the database of all ATM customers with the banks in India &
deployment of high resolution camera and face recognition software at all ATMs. So,
whenever user will enter in ATM his photograph will be taken to permit the access after
it is being matched with stored photo from the database.
6.2.2 Pervasive Computing
Another domain where face recognition is expected to become very important, although
it is not yet commercially feasible, is in the area of pervasive or ubiquitous computing.
Many people are envisaging the pervasive deployment of information devices.
Computing devices, many already equipped with sensors, are already found throughout
our cars and in many appliances in our homes, though they will become ever more
widespread. All of these devices are just now beginning to be networked together. We
can envisage a future where many everyday objects have some computational power,
allowing them to adapt their behavior —to time, user, user control and a host of other
factors. The communications infrastructures permitting such devices to communicate to
one another are being defined and developed for e.g. Bluetooth, IEEE 802.11. So while it
is easy to see that the devices will be able to have a well-understood picture of the virtual
world with information being shared among many devices, it is less clear what kind of
information these devices will have about the real physical world. Most devices today
have a simple user interface with inputs controlled only by active commands on the part
102
of the user. Some simple devices can sense the environment, but it will be increasingly
important for such pervasive, networked computing devices to know about the physical
world and the people within their region of interest. Only by making the pervasive
infrastructure ‘human aware’ can we really reap the benefits of productivity, control and
ease-of-use that pervasive computing promises. One of the most important parts of
human awareness is to know the identity of the users close to a device, and while there
are other biometrics that can contribute to such knowledge, face recognition is the most
appropriate because of its passive nature. There are many examples of pervasive face
recognition tasks: Some devices such as Personal Digital Assistants (PDAs) may already
contain cameras for other purposes, and in good illumination conditions will be able to
identify their users. A domestic message centre may have user personalization that
depends on identification driven by a built-in camera. Some pervasive computing
environments may need to know about users when not directly interacting with a device,
and may be made ‘human aware’ by a network of cameras able to track the people in the
space and identify each person, as well as have some understanding of the person’s
activities. Thus a video conference room could steer the camera and generate a labeled
transcript of the conference; an automatic lobby might inform workers of specific
visitors; and mobile workers could be located and kept in touch by a system that could
identify them and redirect phone calls.
6.2.3 Voter verification
It includes the minimizing the fraud by verifying the identity of a person. Duplicate voter
are being reported in India. To prevent this, a database of all voters, of course, of all
constituencies, is recommended to be prepared. Then at the time of voting the resolution
camera and face recognition equipped of voting site will accept a subject face 100% and
generates the recognition for voting if match is found. Some government agencies have
also been using the systems for security and to eliminate voter fraud. The registration
process requires the scanning of all 10 fingerprints and capturing of pictures of
applicants. The voter list includes pictures of all voters. Stakeholders have requested the,
in addition to the biometric voter registration, the commission should procure biometric
voter verification equipment for the Election Day. It will:

Prevent multiple voting.
103

Prevent voter impersonation.

Prevent ballet stuffing.
6.2.4 Healthcare
Minimize fraud by verifying identity. A New Jersey health care provider in a high-crime
area uses facial recognition technology at the entrance to its emergency room to
determine if a person entering the building has a criminal history and may pose a danger
to staff. It pays a subscription fee to access several national databases to verify the
identities of people in its database. Additionally, it reportedly uses the same technology
to screen vendors coming onto the premises.
6.2.5 Day Care
It includes the verification of the identity of the individuals picking up the children. It is
mainly used in schools in which the some person has to pick the children to bring them
back at home. Therefore, in the database the children parents photograph is maintained to
verify the identity of the person.
6.2.6 Missing Children/Runaways
Search surveillance images and the internet for missing children and runaways. The
photograph a children is given to the police station and to be published in the newspapers
to verify the identity of the person.
6.2.7 Residential Security
This application alerts the homeowners of about approaching the personnel at their home.
In includes the CCTV cameras which are used worldwide like in the malls,
colleges/university campus to provide the security and to prevent the thefts and many
other crimes.
6.2.8 Internet, E-commerce
This application verifies the identity of a person for Internet purchases. There are many
sites which includes the online shopping in which the person has to pay the money with
internet banking. Therefore, to secure the transaction that the person made, face
recognition technology is used to verify the identity of the person.
104
6.2.9 Benefit payment
It minimizes the fraud by verifying the identity of the person. Attackers may attack on
someone’s privacy things like their PIN no and credit card no and then withdraw the
money from their bank account and bankrupt the person whose PIN no is stolen.
6.3 Other areas where face recognition used

Passport and Visa verification

Driving License verification

To identify and verify terrorists at airports, railway stations and malls

In defense ministry and all other important places

Credit card authentication

Entitlements and benefits authorization

Automobile ignition and unlocking; anti-theft devices

Secure financial transactions

Internet security, control of access to privileged information

Ticketless travels, authentication of rights to services

Premises access control (home, office, laboratory, etc.)

In various important examinations such as SSC, HSC, Medical, Engineering,
MCA, MBA, B- Pharmacy, Nursing courses etc. The examinee can be identified
and verified using Face Recognition Technique.

In police station to identify and verify the criminals.

Vaults and lockers in banks for access control verification and identification of
authentic users.

Present bar code system could be completely replaced with the face recognition
technology as it is a better choice for access & security since the barcode could be
stolen by anybody else.
105
Chapter 7
Conclusion and Future Scope
Face recognition has proven to be a very useful and multipurpose security measure. It is a
quick and accurate way of identifying an individual with no scope for human error. Face
recognition is widely used in banking and in law enforcement applications and also have
many applications in other fields where security is necessary. In face recognition field,
there are many problems have been solved, but, there are also many problems need us to
solve. A major difficulty is the design of face image acquisition system to ensure that
whether a person is actually live or not. Therefore, in the proposed work, a Euclidean
distance test (EDT) is generated. Euclidean distance test (EDT) is used for checking a
person’s aliveness which ensures the detection of fake/dummy images. The proposed
liveness detection approach not only verifies that the given biometric sample is from an
authorized person but also verifies that the given biometric sample is real or fake.
Initially, it checks the liveness of the person. If the person is found alive, only then the
further calculations are performed. This proposed approach is suitable for any real time
applications such as e-voting, employee management, terrorist identification, passport
verification etc. EDT opens a new path in face recognition research work. In further
development, this system can be improved to identify a person at a few meters distance
apart. Next, in this dissertation we presented a model for facial expression recognition. In
this model, the presented system is a multimodal architecture in which the detection of a
person is performed on the basis of face and the Facial Expression values. We have
maintained a dataset of face and Facial Expression images. The user will pass the input in
the form of face image and the comparison will be performed on both the face and the
Facial Expression images. To perform the recognition process the PCA and the Eigen
face approach is used collectively. The basic steps of recognition are same as some
existing approach. Here we have providing an approach to use combination of
approaches to identify the face and Facial Expression images. To do this, we used the
106
concept of Euclidean distance. On the basis of Euclidean distance we recognize the
person’s expression. In the proposed work, we compare the Euclidean distance between
the actual image of a person and the expression image of a person. Therefore, on the basis
of the range of Euclidean distance we recognize the person’s expression. The present
work can be extended by researchers in different direction. The foremost process is to
improve the identification approach by using some neural or the fuzzy based analysis.
Another improvement can be done to replace the face recognition using facial feature
recognition and facial Expression detection by radial curvature detection. The proposed
work is about to detect the face from the still image and to recognize it we can future
enhance this work in different cases. First of all we can include the concept of face
recognition from the multiple person images. It means its work will be to detect the
person from the set of face on single images and to recognize his expression. Another
extension can be done in area to include the moving image i.e. to capture the image from
the webcam and to perform the recognition.
Based on the performance, applications, reliability, ease of use, software and hardware
devices that currently support it, face recognition technology has potential for widespread
use. Face recognition technology cost comparably much better with many other biometric
traits.
107
Bibliography:
[1] Jain, A.K., Bolle, R. and Pankanti, S., Biometrics: Personal Identification in a
networked Society, Kluwer Academic Publishers, p.37, (2000).
[2] G. Hua, M.-H. Yang, E. Learned-Miller, Y. Ma, M. Turk, D. J. Kriegman, and T. S.
Huang, “Introduction to the special section on real-world face recognition,” IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 10, pp. 1921–
1924, 2011.
[3] N. A. Spaun, “Forensic biometrics from images and video at the Federal Bureau of
Investigation,” in Proc. BTAS, 2007, pp. 1–3.
[4] Chander Kant, Sheetal Verma “Biometric Recognition System: An introduction”
published in National level seminar on Convergence of IT and Management on 24 Nov.
2007 at TIMT, Yamunanagar, (2007).
[5] Arun Ross, “Iris recognition: the path forward”, IEEE Computer Society, (2010).
[6] Makram Nabti, Ahmed Bouridane, “An effective and fast iris recognition system
based on a combined multiscale feature extraction technique”, Pattern Recognition, 41,
pp.868-879, (2008).
[7] rhien-Lien Hsu, Mohamed Abdel-Mottabel, Anil K. Jain “Face Detection in color
Images”, IEEE trans. Pattern Analysis and machine Intelligence, Vol. 24, no. 5, pp.696706, (May2002).
[8] N. A. Spaun, “Facial comparisons by subject matter experts: Their role in biometrics
and their training,” in Proc. ICB, 2009, pp. 161–168.
[9]
Wang, C., and Brandstein, M.S., “A hybrid real-time face tracking system” Proc.
ICASSP 1998, Seattle, WA, May, 1998, pp. 3636-3740.
[10] A. King, “A survey of methods for face detection,” Technical Report. McGill Unv.,
Mar. 2003.
108
[11] M. Yang, D.J. Kriegman and N. Ahuja, “Detecting faces in images: a survey,” IEEE
Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 1, pp. 34-58, Jan
2002. [12] N. A. Spaun, “Forensic Biometrics from Images and Video at the Federal
Bureau of Investigation,” in Proc. BTAS, 2007, pp. 1–3.
[13] Anil K. Jain, Arun Ross and SalilPrabhakar2, “An Introduction to Biometrics
Recognition”, IEEE Transactions on Circuits and Systems for Video Technology, Special
issue on Image-and-Video-Based Biometrics, Vol. 14, No. 1, January 2004.
[14] Bubeck, U. M. and Sanchez D., “Biometric Authentication: Technology and
Evaluation”, 2003.
[15] Ross, A., Nandakumar, K., Jain, A.K.: Handbook of Multibiometrics. Springer
Verlag (2006).
[16] Aurora Defense “Multi Modal Technology makes Biometrics Work”. PR Web Press
Release, LLC, (2002).
[17] R. A. Wasniowski, “Data Fusion for Biometrics Authentication”. RAW99-SR-320,
(2002).
[18] Prabhakar, S. and Jain, A. K. “Decision-level Fusion in Biometric Verification” to
appear in Pattern Recognition Vol. 35, No. 4, (2002).
[19] Ross, A. and Jain, A. K., “Information Fusion in Biometrics” to appear in Pattern
Recognition Letters, (2003).
[20] Weicheng Shen and Tieniu Tan, “Automated Biomertics-based personal
Identification”, Arnold and Mabel Beckman Center of the National Academics of
Sciences and Engineering in Irvine, CA, August 1998.
[21] Michal Chora, “Emerging Methods of Biometrics Human Identification” Image
Processing Group, Institute of Telecommunications, 695-882, IEEE, (2007).
[22] Qinghan Xiao, “Biometrics Technology, Application, Challenge, and Computational
Intelligence Solutions” IEEE Computational Intelligence Magazine, (May 2007).
109
[23] Arun Ross, Anil K. Jain, “Information Fusion in Biometrics”, Department of
Computer Science and Engineering, 2002.
[24] Sanjay Kr. Singh, D. S. Chauhan, Mayank Vatsa, and Richa Singh, “A Robust Skin
Color Based Face Detection Algorithm” Tamkang Journal of Science and Engineering,
Vol. 6, No. 4, pp. 227-234 (2003).
[25] George Chellin Chandran, J. and Dr. Rajesh, R.S. “Performance Analysis of
Multimodal Biometric System Authentication” IJCSNS International Journal of
Computer Science and Network Security, Vol. 9, No. 3, (March 2009).
[26] J. D. Woodward, “Biometrics: Privacy’s foe or privacy’s friend?” in the Proceedings
of the IEEE (Special Issue on Automated Biometrics), vol. 85, pp. 1480–1492,
September 1997.
[27] L. Hong and A. K. Jain, “Integrating faces and fingerprints for personal
identification,”
IEEE Trans. Pattern Anal. Machine Intel, vol. 20, pp. 1295–1307, Dec.
1998.
[28] Jun Ou, Xiao-Bo Bai, “Automatic Facial Expression Recognition Using Gabor Filter
and Expression Analysis”, 2010 Second International Conference on Computer Modeling
and Simulation 978-0-7695-3941-6/10© 2010 IEEE.
[29] Ming Hu, “Application of Rough Sets to Image Pre-processing for Face Detection”,
Proceedings of the 2008 IEEE International Conference on Information and Automation
978-1-4244-2184-8/08© 2008 IEEE.
[30] Jiying Wu, “Multi-Scale Preprocessing Model for Face Recognition”, in the ICSP
2008 Proceedings 978-1-4244-2179-4/08©2008 IEEE.
[31] Bongjin Jun, “Statistical Face Image Preprocessing and Non-statistical Face
Representation for Practical Face Recognition”, 978-1-4244-5950-6/09©2009 IEEE.
[32]Yong Zhang, “Hand-Drawn Face Sketch Recognition by Humans and a PCA-Based
Algorithm for Forensic Applications”, IEEE Transactions on Systems, Man, and
Cybernetics—PART A: SYSTEMS AND HUMANS 1083-4427© 2010 IEEE.
110
[33] Jain A.K, Ross Arun and Uludag.U. “Biometrics Template security: Challenges and
solutions” in Proceedings of European Signal Processing Conference September 2005.
[34] Jian Yang,” Constructing PCA Baseline Algorithms to Reevaluate ICA-Based FaceRecognition Performance”, IEEE Transactions on Systems, Man, and Cybernetics—
PART B: CYBERNETICS 1083-4419© 2007 IEEE.
[35] Walid Riad Boukabou, “An Improved LDA Approach With DFB Preprocessing for
Face Recognition”, 2007 ECSIS Symposium on Bio-inspired, Learning, and Intelligent
Systems for Security 0-7695-2919-4/07© 2007 IEEE.
[36] Bruce A. Draper, “Recognizing Faces with PCA and ICA”.
[37] Rabia Jafri and Hamid R. Arabnia, “A Survey of Face Recognition Techniques”
Journal of Information Processing Systems, Vol.5, No.2, June 2009.
[38] Hu Han and Anil K. Jain, “3D Face Texture Modeling from Uncalibrated Frontal
and Profile Images” published in the 6th IAPR International Conference on Biometrics
(ICB), June 4 - 7, 2013, Madrid, Spain.
[39] Hu Han, Charles Otto, and Anil K. Jain, “Age Estimation from Face Images: Human
vs. Machine Performance” published in the 6th IAPR International Conference on
Biometrics (ICB), June 4 - 7, 2013, Madrid, Spain.
[40] Brendan F. Klare and Anil K. Jain, “Face Recognition: Impostor-based Measures of
Uniqueness and Quality” in the Proceedings of the IEEE Conference on Biometrics:
Theory, Applications, and Systems, 2012.
[41] Zhifeng Li, Unsang Park and Anil K. Jain, “A Discriminative Model for Age
Invariant Face Recognition” in the proceedings of IEEE Transactions on Information
Forensics and Security.
[42] Volker Blanz and Thomas Vetter, “Face Recognition Based on fitting a 3D Morph
able Model” published in the IEEE Transactions on Pattern Analysis and Machine
Intelligence, Vol. 25, No. 9, September 2003.
111
[43] Zahid Riaz, Christoph Mayer, Matthias Wimmer, and Bernd Radig “Model Based
Face Recognition across Facial Expressions” in the proceedings of IEEE transactions.
[44] A.A. Salah, M. Biceg, L. Akarun, E. Grosso, M. Tistarelli, “Hidden Markov Modelbased face recognition using selective attention”.
[45] Congyong Su and Li Huang, “Spatio-Temporal Graphical-Model-Based Multiple
Facial Feature Tracking”.
[46] Yan Tong, Wenhui Liao and Qiang Ji, “Facial Action Unit Recognition by
Exploiting their Dynamic and Semantic Relationships” published in IEEE Transactions
on Pattern Analysis and Machine Intelligence, Vol. 29, No. 10, October 2007.
[47] Maja Pantic, and Leon J. M. Rothkrantz “Facial Action Recognition for Facial
Expression Analysis from Static Face Images” published in the IEEE Transactions
Systems, Man, and Cybernetics—Part B: Cybernetics, Vol. 34, No. 3, June 2004.
[48] Michael A. Sayette, Jeffrey F. Cohn, Joan M. Wertz, Michael A. Perrott, and
Dominic J. Parrot “A Psychometric Evaluation of the Facial Action Coding System for
Assessing Spontaneous Expression” published in the Journal of Nonverbal Behavior
25(3), 2001 Human Sciences Press, Inc.
[49] Ying-li Tian, Takeo Kanade and Jeffrey F. Cohn, “Recognizing Action Units for
Facial Expression Analysis”.
[50] Alexander M. Bronstein, Michael M. Bronstein and Ron Kimmel “ThreeDimensional Face Recognition” published in the International Journal of IJCV - second
review - 3".tex; 14 December, 2004; 20:14; p.1-44.
[51] Claude C. Chibelushi, Fabrice Bourel “Facial Expression Recognition: A Brief
Tutorial Overview”.
[52] Liwei Wang, Yan Zhang, Jufu Feng, “On the Euclidean Distance of Images”.
112
[53] Zhen Leiy, Shengcai Liaoz, Anil K. Jainz, and Stan Z. Liy, “Coupled Discriminant
Analysis for Heterogeneous Face Recognition” in the proceedings of IEEE.
[54] Farrukh Sayeed, M. Hanmandlu, and A.Q. Ansari “Face Recognition using
Segmental Euclidean Distance” Defence Science Journal, Vol. 61, No. 5, September
2011, pp. 431-442 @ 2011, DESIDOC.
[55] Shalini Gupta, Mia K. Markey, J. K. Aggarwal, Alan C. Bovik “Three dimensional
face recognition based on geodesic and Euclidean distances” in the Proceedings of
SPIEIS& T Electronic Imaging, SPIE Vol. 6499, 64990D, © 2007 SPIEIS& T ∙
0277786X/ 07/$15.
[56] Haitao Zhao, “A Novel Incremental Principal Component Analysis and its
Application for Face Recognition” published in the IEEE Transactions on Systems, Man,
and Cybernetics—Part B: Cybernetics 1083-4419© 2006 IEEE.
[57] Ayan Chakrabarti,” Super-Resolution of Face Images Using Kernel PCA-Based
Prior”, IEEE Transactions on Multimedia 1520-9210© 2007 IEEE.
[58] J.Braun, “Machine Learning and Recognition of Faces”.
[59] Takeo Kanade, Jeffrey F. Cohn and Yingli Tian, “Comprehensive Database for
Facial Expression Analysis”.
[60] Gaurav Mittal, “Robust Preprocessing Algorithm for Face Recognition” in the
Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV’06)
0-7695-2542-3/06© 2006 IEEE.
[61] R. D. Madalina Baltatu and R. D’Amico, “Towards ubiquitous Acceptance of
Biometric Authentication: Template Protection Techniques, vol. 3087/2004 Springer
Berlin / Heidelberg, (2004).
[62] R. M. B. N. K. Ratha, J. H. Connel and S. Chikkerur, “Cancelable biometrics: A
case study in fingerprints” published in the IEEE International Conference on Pattern
Recognition, volume 4, pages 370–373, Hong Kong, China, 2006 IEEE.
113
[63] M. Boatwright and X. Luo, “What do we know about biometrics authentication?” In
InfoSecCD ’07: Proceedings of the 4th annual conference on Information security
curriculum development, pages 1–5, New York, NY, USA, 2007.
[64] T. E. Boult and R.Woodworth, “Privacy and Security Enhancements in Biometrics”
published in Springer, US, 2005.
[65] A. K. Jain, “Biometric Recognition: How Do I Know Who You Are?” published in
Springer, US, 2005.
[66] E. Kindt, “Biometric applications and the data protection legislation” published in
Springer, US, 2007.
[67] A. Adler, “Biometric System Security” published in Springer, US, 2007.
[68] C. Vielhauer, “Biometric User Authentication for IT Security: from Fundamentals to
Handwriting” published in Springer, US, 2005.
[69] A. K., J. D. Maltoni, D. Miao and S. Prabhakar, “Handbook of Fingerprint
Recognition” Springer-Verlag, 2003.
[70] Galbally, J., Fierrez, J., Ortega-Garcia, J.: Vulnerabilities in biometric systems:
Attacks and Recent Advances in Liveness Detection, Biometrics Recognition Group,
Madrid, Spain, 2007, p.8.
[71] W. Zhao, R. Chellappa, A. Rosenfeld, and P.J. Phillips, “Face Recognition:
AliteratureSurvey”, Technical Report, Univ. of Maryland, 2000.
[72] V. Bruce, “Identification of Human Faces”, pp. 615-619, Image Processing and Its
Applications, Conference Publication No. 465, IEEE, 1999.
[73] ] J. Li, G. Chen, and Z. Chi, “A Fuzzy Image Metric with Application to Fractal
Coding,” IEEE Trans. Image Processing, vol. 11, no. 6, pp. 636-643, June 2002.
[74] N. Ahuja, “A Transform for Multiscale Image Segmentation by Integrated Edge and
Region Detection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no.
9, pp. 1211-1235, Sept. 1996.
114
[75] Y. Amit, D. Geman, and B. Jedynak, “Efficient Focusing and Face Detection,” Face
Recognition: From Theory to Applications,H. Wechsler, P.J. Phillips, V. Bruce, F.
Fogelman-Soulie, and T.S. Huang, eds., vol. 163, pp. 124-156, 1998.
[76] R. Chellappa, C.L. Wilson, and S. Sirohey, “Human and Machine Recognition of
Faces: A Survey,” Proc. IEEE, vol. 83, no. 5, pp. 705-740, 1995.
[77] M.Yang, D.J. Kriegman, and N.Ahuja, Detecting Faces in Images: A Survey, IEEE
Trans. on Pattern Analysis and Machine Intelligence, VOL. 24, NO. 1, Jan. 2002.
[78] J. Liu and Y. H. Yang, “Multiresolution color image segmentation”, IEEE Trans. on
PAMI, vol. 16, pp. 689-700, July 1994.
[79] W. Zhao, R. Chellapa, A. Rosenfeld and P.J. Philips, “Face Recognition: A
Literature Survey”, UMD CFAR Technical Report CAR-TR-948, 2000.
[80] William A. Barrett, “A Survey of Face Recognition Algorithms and Testing
Results”, Proceeding of IEEE, 1998.
[81] A.J. Zlochower, Deciphering emotion from the face: An evaluation of FACS, EMG,
and computer-vision based approaches to facial expression analysis. Unpublished
manuscript, University of Pittsburgh, 1997.
[82] R. Matias, J.F. Cohn, S. Ross, A comparison of two systems to code infants'
affective expression. Developmental Psychology, 25: 483-489, 1989.
[83]
Biometrics
in
Human
Services
User
Group.
URL:
http://www.dss.state.ct.us/digital.htm.
[84] Ana Orubeondo, “A New Face for Security” in InfoWorld.com, May 2001.
[85] Lee Gomes. Can Facial Recognition Help Snag Terrorists? The Wall Street Journal,
September 21 2001.
[86] Defense Advanced Research Projects Agency, “Human Identification at a Distance”,
BAA00-29 edition, Feb 2000.
115
116
117
118