Multimodal User Authentication: From Theory to Practice
Multimodal User Authentication:
From Theory to Practice
TUTORIAL

Conference
IEEE ICME 2003

Speakers
Jean-Luc DUGELAY
Jean-Claude JUNQUA

Location
Baltimore

Date & Time
Sunday, 6 July 2003, 13:30 - 17:00
1
Multimodal User Authentication: From Theory to Practice
Who we are…

Jean-Luc DUGELAY




Ph.D. 92
Professor at Eurécom
Sophia Antipolis, France
Security Imaging



Watermarking
Biometrics
Jean-Claude JUNQUA
 Ph.D. 89
 Director of PSTL (Panasonic Speech
Technology Laboratory)
 Santa Barbara, California, U.S.A.
 Speech




Recognition
Synthesis
Multimodal Dialogue
Speaker Verification
2
Multimodal User Authentication: From Theory to Practice
Outline (1/3)

Multimodal user authentication: Background
 Introduction
 Is there a universal biometric identifier?
 What are the factors influencing the reliability of biometric
systems?
 Why are there still very few biometric systems in use today?
 Physiological versus behavioral biometrics
 Why multimodal biometrics? Can multimodal biometrics improve
performance?
 Tradeoffs between robustness (security) and convenience
3
Multimodal User Authentication: From Theory to Practice
Outline (2/3)

Main individual modalities





Signature
Voice
Hand
Fingerprint
Face
 Frontal
 Specific acquisition (infra red, profile, dynamic, range)
 Specific parts (eyes: iris & retina, ears, teeth)
[Specificities, Pros & cons, Open problems, Sensing technologies, Major
algorithms, Database examples, …]

Towards Multimodal Biometric Systems
 Sequence
 Fusion
4
Multimodal User Authentication: From Theory to Practice
Outline (3/3)

Applications, Standards and Evaluation











Main application areas
Biometrics and privacy
Important criteria to deploy multimodal authentication systems
Biometric standards
Multimodal databases
Best practices in testing biometric systems
Examples of multimodal user authentication systems
Perspectives and future challenges
Demonstrations
Forthcoming events
Bibliography
5
Multimodal User Authentication: From Theory to Practice
Outline (1/3)

Multimodal user authentication: Background
 Introduction
 Is there a universal biometric identifier?
 What are the factors influencing the reliability of biometric
systems?
 Why are there still very few biometric systems in use today?
 Physiological versus behavioral biometrics
 Why multimodal biometrics? Can multimodal biometrics improve
performance?
 Tradeoffs between robustness (security) and convenience
6
Multimodal User Authentication: From Theory to Practice
Introduction
Drawbacks of traditional identification
(knowledge- or token- based):

 PIN may be forgotten or guessed by imposter
 Physical keys may be misplaced or lost
 It is not possible to differentiate between an authorized person and an
imposter

Biometric system
 Pattern recognition system which establishes the authenticity of a specific
physiological or behavioral user’s characteristic
 Relies on “who you are or what you do” to make a positive personal
identification
 Comprises an enrollment stage and an identification/verification stage
 Identification (1:N matching, who am I?) & Verification (1:1 matching, Am I
who I claim I am?)
→ Well-known example: login + passwd
7
Multimodal User Authentication: From Theory to Practice
Seven
types of authentication:
Something you know (1)
e.g. PIN code, mother’s maiden name, birthday
Something you have (2)
 e.g. Card, key
Something you know + something you have (3)
e.g. ATM card + PIN
7
Security
level
Something you are – Biometrics (4)
5,6
no PIN to remember, no PIN to forget
4
Something you have + something you are (5)
Smart Card
Something you know + something you have (6)
Something you know + something you have + something
you are (7)
8
1, 2
3
Types
Multimodal User Authentication: From Theory to Practice
A Generic Biometric System
Sensor (e.g.
Microphone,
Camera)
Feature
Extraction
Enrollment and
Template Storage
Enrollment
Template
Adaptation
Identification/Verification
Sensor
Feature
Extraction
Identification/
Verification
Action
9
Multimodal User Authentication: From Theory to Practice

How to measure performance


Biometric systems are not perfect. They make errors in identifying or
true claimants and in rejecting imposters
The probability of committing these two types of errors are called
False Rejection Rate (FRR)
False Acceptance Rate (FAR)
ROC: Receiver Operating Characteristic
FRR is user-dependent
10
Multimodal User Authentication: From Theory to Practice
Performance & Evaluation
11
Multimodal User Authentication: From Theory to Practice
Martin et al., « The DET curve in assessment of detection task performance »
Proc. EuroSpeech 1997.
Performance & Evaluation
100%
10%
FRR
1%
Better performance
0.1%
0.0001% 0.001% 0.01% 0.1%
1%
10%
100%
FAR
Detection error trade-off (DET) curves
(uniform treatment of both types of error)
12
Multimodal User Authentication: From Theory to Practice
Table adapted from http://akhisar.sdsu.edu/abut/BC2002talk_jain.pdf
State-of-the-art error rates
Test data
FRR
FAR
Fingerprint
20 years
(average age)
0.2%
0.2%
Face
11 to 13 months
spaced
10-20%
0.1-20%
Textdependent
Speaker
verification
Text-dependent
(entrance door, 3
months period )
1-3%
1-3%
Textindependent
speaker
verification
TextIndependent
(NIST 2000)
10-20%
2-5%
13
Multimodal User Authentication: From Theory to Practice
Some other performance criteria
Failure and difficulties* to
enroll (e.g. amount of data)
Failure and difficulties
to acquire
Performance
False rejection rate
False acceptance rate
* 4% of fingerprints are of poor quality
14
Multimodal User Authentication: From Theory to Practice
Is there a universal biometric identifier?
There
are many biometric identifiers:
Fingerprint
Voice
Image
Hand geometry
In theory many of these biometric identifiers
should be universal. However, in practice
this is not the case.
Retina
Iris
Signature
Keystroke dynamics
Gait
DNA (requires physical sample)
Wrist/hand veins
Brain activity
etc.
Ideally, a biometric identifier should be universal, unique, permanent and measurable
However, in practice each biometric identifier depends on factors such as users’ attitudes,
Personality, operational environment, etc.
15
Multimodal User Authentication: From Theory to Practice
Adapted from Source: http://www.computer.org/itpro/homepage/Jan_Feb/Security3.html
* Also, www.biometricsmi.com (Vol. 1, Issue 01) & ** www.engr.sjsu.deu/biometrics (Dr. J. Wayman)
Each biometric identifier has its strengths and weaknesses
Characteristics
Fingerprints
Hand
Geometry
Retina
Iris
Face
Signature
Voice
Ease to use
High
High
Low
Medium
Medium
High
High
Error incidence
Dryness, dirt,
age
Hand injury,
age
Glasses
Poor Lighting
Lighting, age,
glasses, hair
Changing
signatures
(inconsistencies)
Noise,
Colds, weather
Accuracy
High
High
Very high
Very high
High
High
High
Cost
Medium
High
Medium
High
Medium
Medium
Low
User acceptance
Medium
Medium
Medium
Medium
Medium
Medium
High
Required security
level
High
Medium
High
Very high
Medium
Medium
Medium
Long-term
stability
High
Medium
High
High
Medium
Medium
Medium
Template size *
(bytes)
200+
9
96
512
History of
automatic ID **
(1880)
1963/1974
1972
(1935)
1976
1994
84 (1:n)
1300 (1:1)
3.5 k
(1888)
1972/1987
…
16
500+
(1929)
1983
1964
Multimodal User Authentication: From Theory to Practice
What are the factors influencing the
reliability of biometric systems


The factors influencing the reliability of biometric systems
depend on the biometric identifier used. Understanding the
requirements, the users and the environment is the key
However, some general factors can be identified







User behavior/cooperativeness
Stability (time and environment) of the biometric identifier
How easy is it to use the system?
Is the user accustomed to the use of the biometric sensor?
Quality of the enrollment
Population demographics
User interface
17
Multimodal User Authentication: From Theory to Practice
Influence of the user and the
environment

The user






Behavior
Consistency
Physiology
Appearance
Familiarity with the equipment
The environment
 Lighting
 Background noise
 Weather (e.g. humidity)
18
Multimodal User Authentication: From Theory to Practice
Influence of time

As the time between enrollment and testing increases, the
biometric features enrolled are generally becoming less
reliable
e.g. - 50% decrease in performance after a period of 1 year for a biometric system based on
faces.

However, as the user keeps using the biometric system he/she
tends to adapt to the biometric system

Supervised or unsupervised adaptation helps dealing with the
mismatch between enrollment and testing
19
Multimodal User Authentication: From Theory to Practice
Why are there still very few biometric
systems in use today?

Main reasons










Accuracy
User acceptance & social factors (e.g. lack of familiarity, privacy)
Standards
Every biometric has its limitations
Cost of deployment
Ease of use
Ease of development (e.g. standards)
Lack of understanding on how to combine biometric identifiers
Difficulties to enroll a large set of individuals
Lack of large scale deployments
20
Multimodal User Authentication: From Theory to Practice
Why are there still very few biometric
systems in use today?

Social factors
 Informational privacy (collection, storage and use of the user
information)
 Personal privacy (how invasive or intrusive is the biometric
identifier used?)
 Political will/cultural climate
 User acceptance/familiarity of the technology
21
From: The Role of Dynamics in Visual Speech Biometrics
Presentation by John Mason and Jason Brand, ICASSP 2002
Multimodal User Authentication: From Theory to Practice
Physiological versus behavioral
biometrics
•
Physiological: what we Are or Have
• Behavioral:
what we
Do
Thus Physiological is static, Behavioral is dynamic
Combinations provide potential for robustness
22
From: The Role of Dynamics in Visual Speech Biometrics
Presentation by John Mason and Jason Brand, ICASSP 2002
Multimodal User Authentication: From Theory to Practice
Examples of physiological and
behavioral biometrics
Physiological:
what we Are/Have
 Eye scans
are
 Fingerprints
are
 Face
are
Behavioral:
what we Do
 Handwriting
do
 Gait
do
 Speech (audio)
do
 Speech (visual)
are
23
do
From: The Role of Dynamics in Visual Speech Biometrics
Presentation by John Mason and Jason Brand, ICASSP 2002
Multimodal User Authentication: From Theory to Practice
Behavioral & Physiological
Biometrics

Audio
Behavioral

Visual
Physiological
Instantaneous ‘snap-shots’ inherently
Physiological
(information signal is not a function of time,
Instantaneous ‘snap-shots’
inherently Behavioral
(information signal is a function of time)
though with speech it can become so)
24
From: The Role of Dynamics in Visual Speech Biometrics
Presentation by John Mason and Jason Brand, ICASSP 2002
Multimodal User Authentication: From Theory to Practice
Dynamics in Biometrics
Behavioral: what we
Physiological: what we
Implications
Movement
Signature
Variation
(undesirable)
System
Enrollment/
Training
Do
Are or Have
Possible / maybe
detrimental
Essential
Slow/small/nil
Inherent/ unavoidable
(nuisance)
One-off possible
Multi-session / adaptive to
capture inherent variation
Of course, ‘signatures’ must always relate to physical properties, some less than
others, e.g. gait – highly so, speech or handwriting – perhaps less so
25
Multimodal User Authentication: From Theory to Practice
From: The Role of Dynamics in Visual Speech Biometrics
Presentation by John Mason and Jason Brand, ICASSP 2002
Speech as a Biometric
Speech
Physiological:
Behavioral:
Are / Have
Visual
(lips)
Do (Dynamics)


Acoustic
Error rates

Quantity/quality of data
26
From: The Role of Dynamics in Visual Speech Biometrics
Presentation by John Mason and Jason Brand, ICASSP 2002
Multimodal User Authentication: From Theory to Practice
Visual Speech Biometric
Instantaneous lip contours from a series of frames
in speaking mode
27
Multimodal User Authentication: From Theory to Practice
Why Multimodal Biometrics?

Introduction
 No single biometric is generally considered sufficiently accurate and useracceptable for any given application
 Authentication systems that are robust in natural environments (e.g. in the
presence of noise and illumination changes) cannot rely on a single modality
 Multimodal user authentication can provide a more balanced solution to the
security and convenience requirements of many applications
 There is a clear requirement for the system to be able to adapt to the user
needs and conditions and, especially, to be able to determine and maintain
an acceptable balance between confidence and convenience for its users.
 Each individual biometric can operate in either a verification mode or an
identification mode
28
Multimodal User Authentication: From Theory to Practice
Multimodal Biometrics

Generic architecture
Biometric feature 1
Biometric
sensor 1
Decision
fusion
Biometric feature N
Biometric
sensor N
Biometric
database
29
Claimed identity
accepted or rejected
Multimodal User Authentication: From Theory to Practice
Can multimodal biometrics improve
performance?

Introduction
 Multimodal user authentication provides a practically viable approach for
overcoming the performance and acceptability barriers to the widespread
adoption of authentication systems
 Integration of multimodal biometric modalities is strongly based on a
thorough understanding of each of the modalities and the different sensing
technologies
 A fully successful multimodal fusion can only be obtained through a careful
investigation of these technologies and their interaction
 Multimodal biometrics can improve accuracy or speed (e.g. face recognition,
can be used to index a template database and fingerprint verification can be
used to ensure the overall accuracy)
 There is the perception that if a a strong test is combined with a weaker
test, the resulting decision is averaged. However, the performance
improvement comes from a well-designed fusion algorithm which should take
advantage of additional information
30
Multimodal User Authentication: From Theory to Practice
Can multimodal biometrics improve
performance?
Sequential
A
B
Fusion
The FAR is determined by the FAR of both
systems
A
Parallel
Fusion
A
B
If A rejects then B is used
B
A and B provides separate scores
The fusion algorithm decides
Can produce a very low FAR as well as a very low FRR
The FRR is determined by both systems
31
Multimodal User Authentication: From Theory to Practice
Pros and Cons of Multimodal
Biometrics

Pros
 Can overcome weaknesses of individual biometric identifiers
 Can extend the operation range to a larger target user population
 Can increase the reliability of the decision made by a single
biometric system
 Is generally more robust to fraudulent technologies (it is more
difficult to forge multiple biometric characteristics)
 If well-designed can improve performance and speed

Cons
 Can make the interaction longer
 Cost of deployment is generally higher
 Integration of multiple biometrics is more complex (score
normalization, etc.)
32
Multimodal User Authentication: From Theory to Practice
Trade-off between robustness (security)
and convenience
 A very secure system will have a higher rejection rate
 Or it will have different passes to increase security at the expense
of user convenience
 For some modalities (e.g. voice) the amount of enrollment data is
directly related to this tradeoff
Robustness
(Security)
Convenience
33
Multimodal User Authentication: From Theory to Practice
Outline (2/3)

Main individual modalities





Signature
Voice
Hand
Fingerprint
Face
 Frontal
 Specific acquisition (infra red, profile, dynamic, range)
 Specific parts (eyes: iris & retina, ears)
[Specificities, Pros & cons, Open problems, Sensing technologies, Major
algorithms, Database examples, …]

Towards Multimodal Biometric Systems
 Sequence
 Fusion
34
Multimodal User Authentication: From Theory to Practice
Modality by modality





Signature
Voice
Hand
Fingerprint
Face
35
Multimodal User Authentication: From Theory to Practice
M. EL Yassa et al., « ETAT DE L'ART SUR LA VÉRIFICATION
OFF-LINE DE SIGNATURES MANUSCRITES» SETIT 2003, Tunis.
Signatures: introduction

Several types of signatures
European
 American
 Close to recursive handwritten
 European
 Includes a graphical component
 Arabic
 etc.

Is a signature authentic or not?

Main difficulty –
Intra-class variations
(a signature of one individual)

Easier to forge than other
biometric attributes
American
36
Multimodal User Authentication: From Theory to Practice
Signatures: static vs. dynamic

Off-line versus on-line signature
 Off-line
Only spatial information is
available
Static: shape of the signature
 On-line
Add. features: velocity, pressure,
etc.
Dynamic: the dynamics of how you sign
+ time
(such as speed, pressure, and timing)
37
Multimodal User Authentication: From Theory to Practice
Signatures: acquisitions

Acquisition
 Off-line

“e-pad”
Scans
 On-line

Special hardware:
 Digitizing tablet
 Pressure sensitive pen
38
Multimodal User Authentication: From Theory to Practice
Signatures: Forgeries

Forgeries
 Random forgeries
The forger has either no knowledge about the original signature,
or does not try to imitate the shape of the signature.
 Zero-effort forgeries
 Skilled forgeries
 Others:
 Counter-drawing
 Disguise
39
Multimodal User Authentication: From Theory to Practice
Signatures: Classical criteria

Classical criteria





Alignments: baseline & envelope
Drawing characteristics: upward / downward drawing
Speed
Proportions
Pressure
40
Multimodal User Authentication: From Theory to Practice
Signatures: Example of features
Features: centroid, baseline, top/down envelops
41
Multimodal User Authentication: From Theory to Practice
Modality by modality





Signature
Voice
Hand
Fingerprint
Face
 Frontal
 Specific acquisition (infra red, profile, dynamic, range)
 Specific parts (eyes: iris & retina, ears, teeth)
42
Multimodal User Authentication: From Theory to Practice
Types of Speaker Recognition Systems
Three types of speech-based authentication:
• Text-dependent systems. User enrolls & authenticates with same
password (template approach)
Advantage: low memory, low processing power, low cost.
Disadvantage: impostor can record client’s voice
• Prompted phrases/passwords system (HMM phoneme models)
• Advantage: combination of recognition and adaptation can improve
performance
• Disadvantage: less natural
• Text-independent systems. Enrollment speech and authentication
speech are different (single state HMM, GMM)
• Advantage: very secure
• Disadvantage: lower accuracy, need more resources, more enrollment
and authentication speech
43
Multimodal User Authentication: From Theory to Practice
Text-Dependent Authentication
Speaker Enrollment
Enrollment speech
“Santa Barbara, California”
Simple Model for P
Rightful user P
Speaker Verification
Simple Model for P
Impostor Model
Test speech S
“Santa Barbara, California”
Compare scores
Claimant C
Reject C
Accept C
44
Multimodal User Authentication: From Theory to Practice
Text-Independent Authentication
Speaker Enrollment
Enrollment speech
“April is the
cruellest
month”
Model for P (GMM)
Rightful user P
Speaker Verification
Model for P
Impostor model I
Test speech S
“Do I dare to eat a peach?”
log p(S|P)-log p(S|I) > T?
yes
Claimant C
no
Reject C
Accept C
45
Multimodal User Authentication: From Theory to Practice
Modality by modality





Signature
Voice
Hand
Fingerprint
Face
 Frontal
 Specific acquisition (infra red, profile, dynamic, range)
 Specific parts (eyes: iris & retina, ears, teeth)
46
Multimodal User Authentication: From Theory to Practice
Hand geometry: acquisition
“Hand punch”





Fairly simple and accurate
But human hand is not unique; only used for verification (not
descriptive enough for identification)
Usage: Some people are reluctant to put their hands on the same
support used previously by others
Special hardware; the hand is properly aligned by the pegs
fingers
47
Multimodal User Authentication: From Theory to Practice
Reference: A prototype Hand Geometry-based
Verification System, Arun Ross
Hand geometry
14 axes along which features
values are computed.
(5 pegs serve as control points
and assist in choosing these
axes).
 Ps and Pe refer to the end points
(using gray scale profile)
→ feature vector

48
Multimodal User Authentication: From Theory to Practice
Reference: Biometric Identification through
Hand Geometry Measurements
R. Sanchez-Reillonet al.
IEEE PAMI Vol. 22 no. 10, 2000.
Hand/Finger geometry
Measurements (4 categories)




Widths: Palm, plus each of the four fingers is measured in different heights
Heights: middle, little and palm.
Deviations: distance between a middle point of the finger and the middle point of the straight
line between the interfinger point and the last height where the finger width is measured.
Angles: between the interfinger points and the horizontal.
Classification and Verification
 Euclidean, Hamming, Gaussian Mixture Models (GMMs), Radial Basis Function
Neural Networks (RBF).
49
Multimodal User Authentication: From Theory to Practice
Modality by modality





Signature
Voice
Hand
Fingerprint
Face
 Frontal
 Specific acquisition (infra red, profile, dynamic, range)
 Specific parts (eyes: iris & retina, ears, teeth)
50
Multimodal User Authentication: From Theory to Practice
Fingerprints: acquisition



Inked fingerprints: finger is rolled or
dabbed on a sheet of paper
Live-scan fingerprints: no need of an
intermediate medium like paper;
systems are optical, thermal,
electromagnetic or ultrasound based
Quality fingerprint acquisition is
extremely challenging:
 elastic distortion of the finger on the
acquisition surface dry/wet skin scars,
cuts, presence of dirt/ grease, etc.
 Exact position of the finger on the
scanner machine (i.e. slight rotation
are possible)
 Pressure of the finger on the surface
of the acquisition machine
 Degree of finger moisture at the
contact area
Source:
51
Multimodal User Authentication: From Theory to Practice
Subset of ST Microelectronics’ Fingerprint Image Database
Fingerprints: image examples
52
Multimodal User Authentication: From Theory to Practice
Fingerprints: global classification
Global patterns of ridges and furrows form special configurations in the
central region of fingerprints
Arch
Left Loop
Right Loop
Whorl
6%
34%
32%
28%
• Class information is not sufficient to carry out recognition
• Can be used for clustering: once a fingerprint is classified, it can be
matched only with a subset of the database
53
(Courtesy of ST Microelectronics)
Multimodal User Authentication: From Theory to Practice
Fingerprints: local classification
Local ridge characteristics determine the uniqueness of a fingerprint
bifurcation
ending
bridge
lake
island
• Ridge endings and bifurcations are usually used for their robustness
and stability
•Most automatic fingerprint matching algorithms mimic the process
used by forensic experts to perform recognition:
minutiae extraction
template matching
54
(Courtesy of ST Microelectronics)
Reference: On-Line Fingerprint Verification
Anil Jain & Ruud Bolle
IEEE T-PAMI, Vol. 19, No. 4, April 1997
Multimodal User Authentication: From Theory to Practice
Fingerprints:
Typical algorithm for Minutiae extraction

Minutiae Extraction






Smoothing Filter
Oriented Field Estimation
Fingerprint Region Localization
Ridge Extraction
Thinning
Minutiae extraction
55
(Courtesy of ST Microelectronics)
Multimodal User Authentication: From Theory to Practice
Fingerprints: Matching


Alignment stage (global)
Adjustment (local)
56
(Courtesy of ST Microelectronics)
Reference:
Human and Machine Recognition of Faces: A Survey
R. Chellappa et al.
Proc. of the IEEE Vol. 83., No. 5, May 1995
Multimodal User Authentication: From Theory to Practice
Modality by modality





Signature
Voice
Hand
Fingerprint
Face
 Frontal
 Specific acquisition (infra red, profile, dynamic, range)
 Specific parts (eyes: iris & retina, ears)
57
Multimodal User Authentication: From Theory to Practice
Face: frontal face recognition




Face detection - Is there a face?
Face segmentation - Where?
Face tracking (if video)
Face size and position. In practice, it is very difficult to control the
position of the subject with respect to the camera
→ “Normalize” inter-ocular distance…
 Changes in illumination. If a spotlight is not used, lighting variations
occur. For example, close to a window, the lighting depends strongly
on the time of the day and the weather
→ “Normalize” gray-scale histogram…
 Facial expressions. In practice, it is almost impossible to control the
mood of the subject. The smile causes probably the largest variation
of facial expressions
 Others. Glasses, Hats, Facial hair, etc.
58
Multimodal User Authentication: From Theory to Practice
Subset of AT&T’s Face Image Database
Face (frontal): Image Examples
database: ORL, FERET, M2VTS & XM2VTS
59
Reference:
Detecting Faces in Images: A Survey
M.-H. Yang at al.
IEEE t-PAMI, Vol. 21, No. 1, Jan. 2002
Multimodal User Authentication: From Theory to Practice
Face detection & segmentation
60
Multimodal User Authentication: From Theory to Practice
Face: frontal face recognition
Two successful classes of algorithms
According to the last round of NIST evaluations, current best solutions
are derived either from Eigenfaces or Elastic Graph Matching
approaches.
Projection-based approaches: Eigenfaces
→ Fisherfaces.
 Deformable models: Elastic Graph Matching (EGM)
→ Elastic Bunch Graph Matching.

61
Reference: Face Recognition Using Eigenfaces
M. Turk & A. Pentland
IEEE 1991
Multimodal User Authentication: From Theory to Practice
Eigenfaces
(eigeneyes, eigenmouths, eigenvoices, eigenears, etc.)
62
Reference: Face Recognition Using Eigenfaces
M. Turk & A. Pentland
IEEE 1991
Multimodal User Authentication: From Theory to Practice
Eigenfaces
(Originally designed for compression, not recognition)
Eigenspace can be built using the clients (higher performances but less flexible) or not
I(x,y) can be considered as a two-dimensional NxN array of pixels
(if N=256; can be seen as a point in a 65,536 dimensional space)
“face space”
The space of variation between photographs of human faces with the same
orientation and scale lit in the same way can be described by a relatively low
dimensional subspace.
Individual face image ≈ linear combination of a small number of face components
I1, I2 … IN: set of reference or training faces
1
E0 
N
(eigenface 0)
63
N
I
i 1
i
Multimodal User Authentication: From Theory to Practice
Reference: Face Recognition Using Eigenfaces
M. Turk & A. Pentland
IEEE 1991
Eigenfaces
Each face differs from the other faces by the vector Di = Ii – E0.
Covariance matrix C:
1
C
N
N
D D
i 1
i
T
i
Eigenvectors of C (variation between face images) → Eigenfaces Ek
Dimensionality Reduction Technique (DRT)
Principal Component Analysis (PCA)
→ eigenvectors ordered by the magnitude of their contribution to the variation
between the training images


Extract the R eigenvalues;
Order them from largest to smallest, 1, 2, …r
Order corresponding eigenvectors E1, E2, …Er
→ « principal components »

K
Weighted sums of a small collection of characteristic images
64
I i  E 0    ki E k   k
k 1
Reference: Face Recognition Using Eigenfaces
M. Turk & A. Pentland
IEEE 1991
Multimodal User Authentication: From Theory to Practice
Eigenfaces
PCA in a 2-D space
Euclidean distances between
the K coordinates representing the new face
and each of the K-dimensional vectors
representing the stored faces,
→ the stored image yielding the smallest
distance
65
Multimodal User Authentication: From Theory to Practice
Eigenfaces vs. Fisherfaces
•
Eigenfaces [Kirby & Sirovich, Turk & Pentland]:
-
no distinction between inter- and intra-class variabilities
Average face (eigenface 0) and first four eigenfaces
•
Fisherfaces [Belhumeur, Hespanha & Kriegman]:
- discriminative approach: find a sub-space which maximizes the
-
ratio of inter-class and intra-class variability
same intra-class variability for all classes
Average face (Fisherface 0) and first four Fisherfaces
66
Distortion Invariant Object Recognition
In the Dynamic Link Architecture
M. Lades et al.
IEEE Trans. On Computers, Vol. 42, no. 3 1993.
Courtesy of Thessaloniki Univ.
Multimodal User Authentication: From Theory to Practice
EGM
Elastic Graph Matching
(a)
(b)
(c)
(a) Model grid for person A (1 feature vector / node)
(b) Best grid for test person A after elastic graph matching with the model grid.
(c) Best grid for test person B after elastic graph matching with the model grid for person A.
• Vertex labels (local mappings costs)
C total  C e  C v
• Edge labels (local distortions costs)
• λ controls the rigidity of the image graph
67
Multimodal User Authentication: From Theory to Practice
Modality by modality





Signature
Voice
Hand
Fingerprint
Face
 Frontal
 Specific acquisition (infra red, profile, dynamic, range)
 Specific parts (eyes: iris & retina, ears, dental)
68
Multimodal User Authentication: From Theory to Practice
Profile

distance and angle between
fiducial points
Source:
69
Multimodal User Authentication: From Theory to Practice
Dynamic Video Biometrics
Higher potential of video w.r.t. still images


More clues (Abundant data)
Face / Facial feature tracking
New opportunities
Visual speech data
→ correlation between speech and lip motion)
 Dynamic facial Expression
→ behavior (not physical only)
 Shape/Structure from motion

Useful for covert surveillance (but non-cooperative with low resolution)
70
- Face Recognition using range images B. Achermann et al., VSMM 97.
- Face Identification by Fitting a 3D Morphable Model using Linear Shape
and Texture Error Functions, S. Romdhani et al., ECCV 2002
- Face Recognition based on depth maps and surface curvature G. Gordon,
SPIE Proc., vol. 1570, 1991.
Multimodal User Authentication: From Theory to Practice
3-D Faces, range data
Advantages:
 Access to the depth information (i.e. shape)
 Pose and light conditions may be compensated
 Higher robustness
(e.g. the system cannot be trapped by an impostor using a 2D picture of
someone else)
Disadvantages:
 Acquisition process is slow and highly expensive, e.g. 3-D Scanner, 2 calibrated
video cameras
Cooperation of the users is required
Little literature available on the topic (novel facial biometrics)
Some published works…
 Extension of existing algorithms from 2-D to 3-D (e.g. eigenfaces)
 Adaptation of a generic 3-D Deformable Model to 2-D images of users to
provide a set of parameters associated with a person
 Segmentation of range data into 4 surface regions; Normalization based on
the location of eyes, nose and mouth; Distance computed from the volume
between surfaces
 …
71
- Thermal pattern recoginition systems faces security challenges head on
M. Lawlor, Signal Magazine Nov. 97
- Comparison of visible and infra-red imagery for face recognition
J. Wilder et al., Int. Conf. On Automatic Face and Gesture Recognition, Oct. 96
Multimodal User Authentication: From Theory to Practice
Facial Thermogram (IR imaging)
The facial heat emission patterns can
used to characterize a person
Patterns depend on 9 factors including:
-
Location of major blood vessels
Skeleton thickness
Amount of tissue
Muscle
Fat
Advantages:




Unique (even for identical twins)
Stable over time
Cannot be altered through plastic surgery
Independent of the lighting conditions
Disadvantages:


Source: Other Biometric Techniques
Chapter 10. D. Baik and I. Kim
IR imagery depend on the temperature
Opaque to glass
72
Multimodal User Authentication: From Theory to Practice
Modality by modality





Signature
Voice
Hand
Fingerprint
Face
 Frontal
 Specific acquisition (infra red, profile, dynamic, range)
 Specific parts (eyes: iris & retina, ears, dental)
73
Multimodal User Authentication: From Theory to Practice
How Iris Recognition Works
J. Daugmann http://www.cl.cam.ac.uk/users/jgd1000/
Iris
74
Multimodal User Authentication: From Theory to Practice
How Iris Recognition Works
J. Daugmann http://www.cl.cam.ac.uk/users/jgd1000/
Iris
4 steps
1. Acquisition ( < 1 meter)
2. Find iris in the image (edge detection)
3. Features extraction:
- Local regions of an iris are projected
onto quadrature 2D Gabor wavelets,
generating complex-valued coefficients
whose real and imaginary parts specify
the coordinates of a phasor in the
complex plane
- The angle of each phasor is quantized to
one of the four quadrants, setting two
bits of phase information
- This process is repeated all across the
iris with many wavelet sizes, frequencies
& orientations
→ the Iris-Code (1024 phase bits are
computed)
4. Verification
75
Multimodal User Authentication: From Theory to Practice
Coutesy of J. Leroux des Jardins
Retina

Unique
 Number/Pattern of blood vessels,
that emanate from the optic nerve
and disperse throughout the retina
 Relative angle w.r.t. optical nerve
 Bifurcations
 No two retinas are the same
even in identical twins
 Vascular pattern does not change over
the course of life
 Glasses, contact lenses, existing
medical conditions (e.g. cataracts) do
not interfere
At the moment, identification based on
retina is used for animals (bovines)

‘Uncomfortable’ acquisition
 Eye has to fix a lighting point
 Projected lighting source on the center
of the optical nerve
 Light is absorbed by red vessels but
reflected by retina tissues
76
Multimodal User Authentication: From Theory to Practice
Source: http://www.retinaltech.com
Retina


77
Extracting Intensity Profiles
 Performing Scan
Locating Blood Vessels
 Generating Circular Bar Code
Multimodal User Authentication: From Theory to Practice
Sources
- On the use of Outer Ear Images for Personal Identification in Security Applications
B. Moreno et al., IEEE 1999.
- http://www.dcs.shef.ac.uk/~miguel/papers/msc-thesis.html
- Ear Biometrics for Machine Vision M. Burge and W. Burger
Ear

Advantages of ears over faces

Uniform distribution of colors
Reduced surface
Less variability vs. pose/expressions,
Shape and appearance fixes.

Passive identification (≠ fingerprints)

Algo.



 Eigenears

78
Feature points
Multimodal User Authentication: From Theory to Practice
Source: Ear Biometrics for Machine Vision M. Burge and W. Burger
Iannarelli’s Ear Biometrics

Iannarelli System (1949) is based on 12 measurements.
(External anatomy)
1 Helix Rim,
2 Lobule,
3 Antihelix,
4 Concha,
5 Tragus,
6 Antigrus
7 Crus of Helix
8 Triangular Fossa
9 Incisure Intertragica


Distance between each
of the numbered areas
79
Segmented outer ear
Segmented inner ear
Sources:
- On the use of Outer Ear Images
For Personal Identification in Security Applications
B. Moreno et al., IEEE 1999.
- http://www.dcs.shef.ac.uk/~miguel/papers/msc-thesis.html
Multimodal User Authentication: From Theory to Practice
Eigenears
80
Multimodal User Authentication: From Theory to Practice
« Liveness » and countermeasure
Impostors may use a fake biometric,
 Photography of a face
 Recorded voice
 Plaster hand
 etc.
Countermeasure: To use a « liveness » test to check the
presence of a “real” biometric, e.g. cardiac activity, heart rate
81
Multimodal User Authentication: From Theory to Practice
Source: Ph. D., Fingerprint classification and matching using a filterbank,
S. Prabhakar
Diverse facets of Multimodal
82
Multimodal User Authentication: From Theory to Practice
Multimodality & fusion
e.g. Some possible scenarios in Faces
Face
By default
If needed only
less comfortable
but more accurate
profile
By default
Visible
Iris
If case of darkness
IR
By default
If needed only
less comfortable
but more accurate
Ear
frontal
In case of different shots available
Fusion
profile
83
Multimodal User Authentication: From Theory to Practice
Person Identification Using Multiple Cues
R. Brunelli & D. Falavigna
IEEE T-PAMI, Vol. 17, no. 10, pp. 955-966, Oct. 95
Fusion

At 3 possible levels:
 Abstract level
→ Output of each module is a list of labels without any confidence
information,
Identification: ID of the person
Verification: binary response
 Rank level
Output of each module is a set of possible labels ranked by
decreasing confidence values
 Measurement level
A measure of confidence is associated with each label
84
Multimodal User Authentication: From Theory to Practice
Person Identification Using Multiple Cues
J. Kittler et al.
IEEE T-PAMI, Vol. 20, no. 3, pp. 226-239, 98.
Fusion
{wi} i=1..N
the set of possible classes

max iN1 max Mj1 P( w i x j )
 Identification: Number of persons
present inside the database;
 Verification: Authentic and Impostor.
max iN1 min Mj1 P( w i x j )
{xj} j=1..M
the set of biometrics

max iN1 med Mj1 P( w i x j )
Abstract:
Vote based on majority

Ranks:
Maximum, minimum and median

M
1
max iN1  P(w i x j )
M j1
Scores:
Averaging & Weighed averaging

M
max
85
N
i 1
  P( w
j1
i
i
x j)
http://www.cl.ac.uk/users/jgd1000/combine/combine.html
Multimodal User Authentication: From Theory to Practice
A Tutorial on Support Vector Machines for Pattern Recognition
C. Burges (burges@lucent.com)
(hard/soft) Fusion: AND/OR
Operator AND
Operator OR
Arithmetic Operator

Score #2
(mean)
Score #1

Accepted User

Advanced fusion: SVM (Support Vector Machine)
86
Multimodal User Authentication: From Theory to Practice
Integrating Faces and Fingerprints for Personal Identification
L. Hong & A. Jain
IEEE T-PAMI, Vol. 20, No.12, pp. 1295-1307, 1998.
Example: Face + Fingerprint
Goal: To overcome the limitations of both systems
 i- Pre-selection of N persons using Face Recognition (top 5 matches)
 ii- Fingerprint Verification only performed on pre-selected persons


Reminder
Face recognition is fast but not reliable while fingerprint verification is
reliable but inefficient in database retrieval
Reported results
FAR
1%
0.01%
face
15.8%
61.2%
(0.9 sec.)
FRR
fingerprint
3.9%
10.6
(3.2 sec.)
87
integration
1.8%
6.6%
(4.1 sec.)
Multimodal User Authentication: From Theory to Practice
KITTLER et al. « On Combining Classifiers »
IEEE T-PAMI, Vol. 20, No. 3, March 98
Example: Frontal + Profile + Speech
Method
EER (%)
Frontal
12.2
Profile
8.5
Speech
1.4
Sum
0.7
Product
1.4
Maximum
12.2
Median
1.2
Minimum
4.5
88
Multimodal User Authentication: From Theory to Practice
- B. Duc et al., « Fusion of audio and video information for multimodal person authentication »
Pattern recognition Letters 18 (1997) 835-843
- S. Ben-Yacoub, « Multi-Modal Data Fusion for Person Authentication using SVM »
IDIAP RR 98-07
Example: Face + Speech
Supervisor
FA(%)
FR(%)
TE(%)
Face
3.6
7.4
11.0
6.7
0.0
6.7
Arithmetic mean
1.2
2.1
3.3
Bayesian
conciliation
0.54
0.0
0.54
Linear-SVM
0.07
0.0
0.07
Polynomial-SVM
0.21
0.0
0.21
RBF-SVM
0.12
0.0
0.12
MLP-SVM
0.15
0.0
0.15
(EGM)
Speech
(Text-dependent)
89
Multimodal User Authentication: From Theory to Practice
J. Kittler et al., « Enhancing the performance of personal identity authentication systems by fusion of face verification experts”
Example: Faces
Expert
Evaluation Test
Test Set
N. Experts
DT
BKS
UniS-gdm
97.83
97.15
2
97.02
97.92
UniS-noc
96.46
96.90
Unis-eucl
88.80
91.15
3
97.51
98.15
UCL-lda
96.11
96.68
4
96.46
98.21
UCL-pm1
94.43
95.34
5
96.12
98.34
UCL-pm2
95.29
96.14
6
95.81
98.43
N. Experts
DT
BKS
2
A posteriori
87.79
87.79
97.39
97.39
3
A posteriori
97.7
97.7
97.78
97.78
4
A posteriori
97.28
97.8
97.68
98.05
5
A posteriori
96.6
96.6
98.08
98.19
6
A posteriori
96.24
96.33
98.49
98.49




DATABASE: XM2VTS
DT: decision Templates
BKS: Behavior Knowledge Space
ASR: Average Success Rate of client
acceptance and impostor rejection
on the Evaluation set (top), on the Test set (down).



BKS >> DT
By adding experts, the performance of the
multimodal system will not be degraded.
For a sufficient number of experts, optimal
configuration selected on the evaluation
set, also a posteriori optimal on the test
set.
90
- Information Hiding, S. Katzenbeisser and F. Petitcolas, Eds. 2000 Artech House
- Hide a Face in a Fingerprint Image Jain et al.
Multimodal User Authentication: From Theory to Practice
Note on Biometrics vs. Data Hiding
Goal: To combine watermarking and biometrics, for example by hiding the
minutia of a passport’s owner inside his/her id picture present on the document
→ In order to enforce the security of documents (harder to falsify thanks to crosssecurity)

Embedding of eigenface data (associated with a face image) in a fingerprint image
(cover) of a given person
Problem: Relevant characteristics of host image must remain unchanged (e.g.
location and nature of minutia), i.e. the same map of minutia must be obtained
either from the original fingerprint image or from its watermarked version
Basic recall on watermarking
The aim of digital watermarking is to include a subliminal information (i.e.
imperceptible) in a multimedia document for security purpose (e.g. copyright)
It would then be possible to recover the embedded message using a secret key,
at any time, even if the document was altered
Trade-off: capacity, visibility and robustness
91
Multimodal User Authentication: From Theory to Practice
Outline (3/3)

Applications, Standards and Evaluation











Main application areas
Biometrics and privacy
Important criteria to deploy multimodal authentication systems
Biometric standards
Multimodal databases
Best practices in testing biometric systems
Examples of multimodal user authentication systems
Perspectives and future challenges
Demonstrations
Forthcoming events
Bibliography
92
Multimodal User Authentication: From Theory to Practice
S. Nanavati, M. Thieme, and R. Nanavati: Biometrics, Identity Verification
in a Networked World, Wiley Computer Publishing, 2002
Main application areas

Biometric applications can be classified as follows:
 Forensics: criminal investigation and prison security
 Retail/ATM/point of sale
 Civilian applications: electronic commerce and electronic banking (e.g. Visa
cards)
 Information system/ computer network security: user authentication, remote
access to databases
 Physical access/time and attendance (e.g. cellular phone, workstations, door
entrance, automobile)
 Citizen identification (e.g. interaction with government agencies)
 Surveillance (identify or verify the identity of individuals present in a given
space/area, e.g. airport)
Different Markets require different biometric levels of security
93
Multimodal User Authentication: From Theory to Practice
Biometrics in Airports (USA)
Airport
Biometric
First Installed,
Trial
Population
Staff, travellers
Supplier
Charlotte/Douglas
Iris
US Airways employees entering secure ares
Chicago
Finger
Cargo truck drivers who deliver to the airport
Fresno
Face
Idaho
Hand
JFK
Hand
Lincoln
Finger
Logan
Iris
LAX
Hand
Manchester
Face
Miami
Hand
Mineta San Jose
Trial
Identix
Reco. Systems Inc.
Identix
Nov. 02
Trier Technologies
Hand
Trial
Reco. Systems Inc.
Portland
Hand
Trial
Reco. Systems Inc.
Salt Lake City
Hand
SF
Hand
Springfield
Finger
St. Petersburg
Face
St. PetersburgClearwater
Finger
Reco. Systems Inc.
Identix
Identix
…
94
Multimodal User Authentication: From Theory to Practice
Biometrics in Airports (EU, Others)
Airport
Biometric
First Installed,
Trial
Charles de Gaulle
Trial
Orly
Trial
Population
Staff, passengers
Supplier
Country
France
Jan. 03
ZN Vision Techn. AG
Germany
Iris
Oct. 01
Joh. Enschede BV
Netherlands
London Heathrow
Iris
Aug. 02
Keflavik
1-n Face
Jun. 01
Crowd
Identix Inc., Visionics
FaceIt Software
Iceland
Ben Gurion
1-1 Hand
1998
Passenger
moving through
custom
Reco. Syst. Inc.
Israel
Narita
Face and Iris
King Abdul Aziz
Iris
Singapore
1-1 Finger
Thunder Bay
Face
Toronto
Hand
Canada
Vancouver
Finger and Hand
Canada
Berlin
1-1 Face reco.
Frankfurt
Iris
Amsterdam
Schiphol
UK
NTT DoCoMo
Saudi Arabia
Feb. 02
Trial
NExus AcSys
…
95
Canada
Multimodal User Authentication: From Theory to Practice
In practice…
INPASS program
 Enrollment procedure about 30
min.
 Inpass benefits: insignificant (?)
Palm Beach Airport
Airport Face scanner failed…
Error rate of 53%
(455 success out of 958 attempts)
Vendor argues system was not
used properly (i.e. incorrect
lighting)
96
Multimodal User Authentication: From Theory to Practice
S. Nanavati, M. Thieme, and R. Nanavati: Biometrics, Identity Verification
in a Networked World, Wiley Computer Publishing, 2002
Vertical Markets

Law enforcement

Government sector

Financial sector

Healthcare

Travel and immigration
However, biometric deployments in these markets are not always very different
97
Multimodal User Authentication: From Theory to Practice
Published September 2001, International Biometrics Group
Fingerprint technology is the only biometric that has been implemented
within a large scale (IAFIS)
98
Multimodal User Authentication: From Theory to Practice
Smart Cards and Biometrics

A Smart Card is a portable secure storage
(can contain computer chip)

Smart Cards are excellent support for privacy

Smart Cards can verify the biometric identity

Smart cards can update the biometric template

Smart Cards prevent the need for a big centralized database (support
privacy)
99
Multimodal User Authentication: From Theory to Practice
Privacy
“your privacy is important to us. How much
would you pay to preserve it?”
The Wall Street Journal, November 14th, 2001
100
Multimodal User Authentication: From Theory to Practice
Privacy
Definition
by Alan Westin
“Privacy is the claim of individuals, groups, or institutions
to determine for themselves, when, how and to what extent
information about them is communicated to others”
101
Multimodal User Authentication: From Theory to Practice
Privacy
General

requirements
Use biometric data in accordance with privacy needs
Technical




requirements
Do not store biometric raw data in a database
Do not use the biometric data outside the specified purpose
Do not collect unnecessary personal data
Use adequate algorithms for the calculation of biometric signatures
102
Multimodal User Authentication: From Theory to Practice
Privacy Concerns
Factors
affecting privacy
High
Very High
Low
High
Sensitivity of the data
Privacy is becoming an increasingly important issue especially
in large systems
103
Multimodal User Authentication: From Theory to Practice
Important criteria to deploy multimodal
user authentication systems








Enrollment
User acceptance
Privacy/Civil liberties
ID management/ID theft
Database management/Integrity
Political and cultural environment
System complexity
Cost
104
Multimodal User Authentication: From Theory to Practice
Important criteria to deploy multimodal
user authentication systems
Before
introducing this technology to customers, a number of
fundamental questions about consumer understanding, expectations
and concerns need to be answered. The answers to these questions
will help the development of solutions that are accepted by the
consumers
Understanding
consumer attitudes towards this technology is
essential to business managers as they study the ROI and design
the user interface“
105
Multimodal User Authentication: From Theory to Practice
Biometric standards

What are standards and what are they good for?
 Standards (a general set of rules to which all complying procedures, products or
research must adhere) offer a myriad of benefits. They reduce differences between
products and promote an aura of stability, maturity and quality to both consumers
and potential investors (http://www.biometrics.org/html/standards.html)

Who establishes them?
 Standard Bodies, e.g.
 American National Standards Institute (ANSI)
 International Standards Organization (ISO)
 National Institute of Standards and Technology (NIST)

What biometric standards are available?
 Several already exist (http://www.biometrics.org/html/standards.html)
106
Multimodal User Authentication: From Theory to Practice
Biometric standards

BioAPI
 BioAPI (March 2002: BioAPI Version 1.1 was approved as
ANSI/INCITS 358-2002)
 Biometric Consortium took the lead to merge the efforts of several
vendors under BioAPI with strong support from NIST
 Defines a generic way of interfacing to a broad range of biometric
technologies
 Founded in 1988 by Compaq, Microsoft, Novell, IBM, Identicator,
Miros. Merged with other efforts in 1999
 Purpose: Development of a standard biometric API to bring
platform and device independence to application developers,
integrators and end-users
 Benefits
Easy substitution of biometric technologies
Use of biometric technologies across multiple applications
Easy integration of multiple biometrics using the same interface
Rapid application development – increased competition (tends to lower
costs)
 Application compatibility / interoperability
 www.bioapi.org
107




Multimodal User Authentication: From Theory to Practice
Biometric standards

CBEFF
 CBEFF (NISTIR 6529, Jan.3, 2001)
 Common Biometric Exchange File Format
 Describes a set of data elements necessary to support biometric
technologies in a common way
 Features
 Facilitate biometric data interchange between different system
components or systems
 Promotes interoperability of biometric-based application programs and
systems
 Provides forward compatibility for technology improvements
 Simplifies the hardware and software integration process
 www.itl.nist.gov/div895/isis/cbeff
108
Multimodal User Authentication: From Theory to Practice
Multimodal databases






There are very few
They are costly to record
Many parameters need to be taken into account (e.g. for one modality
such as voice: speaker population, environment, age, text to say, etc.)
Realistic data (from real-world application is very difficult to collect
and it is generally difficult to control the different factors)
Nature of the imposters is an issue,
etc.
→ One possibility could consist in building a multimodal database by
artificially combining different unimodal ones.
There is no correlation between most of biometrics
109
Multimodal User Authentication: From Theory to Practice
Multimodal databases: XM2VTS




http://www.ee.surrey.ac.uk/Research/VSSP/xm2vtsdb/
The database was recorded within the M2VTS project (Multimodal
Verification for Teleservices and Security applications), a part of the
EU ACTS program, which deals with access control by the use of
multimodal identification of human faces
The goal of using a multimodal recognition scheme is to improve the
recognition efficiency by combining single modalities, namely face and
voice features
The XM2VTSDB contains four recordings of 295 subjects taken over a
period of four months. Each recording contains a speaking head shot
and a rotating head shot. Sets of data taken from this database are
available including high quality colour images, 32 KHz 16-bit sound
files, video sequences and a 3D Model
110
Multimodal User Authentication: From Theory to Practice
Multimodal databases: BT-DAVID





http://galilee.swan.ac.uk
The BT-DAVID (Digital Audio-Visual Integrated Database) audio-visual
database is designed for undertaking research in speech or person
recognition, as well as synthesis and communication of audio-visual
signals
Expected areas of application are: automatic speech/person recognition
for terminal interfaces or automated transaction machines, voice control
of video-conferencing resources, speech-assisted video coding, and
synthesis of talking heads
The BT-DAVID database contains full-motion video, showing a full-face and
a profile view of talking subjects, together with the associated
synchronous sound. BT-DAVID includes audio-visual material from more
than 100 subjects including 30 clients recorded on 5 sessions spaced over
several months
The BT-DAVID database was compiled by the Speech and Image Research
Group at University of Wales Swansea under a contract to BT Labs
111
Multimodal User Authentication: From Theory to Practice
Best practices in testing biometric
systems

Fact:
 It is still very difficult to predict real-world error rates

Besides performance (which includes both false positive and
false negative decisions along with failure to enroll and failure
to acquire rates across the test population) the following
criteria should also be taken into account







Reliability, availability and maintainability
Vulnerability
Security
User acceptance
Human factors
Cost/benefit
Privacy regulation compliance
112
A.J. Mansfield and J.L. Wayman: Best Practices in Texting and
Reporting Performance of Biometric Devices, NPL Report CMSC 14/02
P.J. Phillips, A. Martin, C.L. Wilson, and M. Przybocki: An introduction
to evaluating biometric systems. Computer, (Feb. 2000), 56-63
Multimodal User Authentication: From Theory to Practice
Best practices in testing biometric
systems

Biometric technical performance testing can be of three types:
 Technology (database evaluation)
 Scenario (overall system performance in a prototype or simulated
application. Could be a combination of offline and online testings)
 Operational evaluation (performance of a complete system in a
specific application environment with a specific target population.
In general not repeatable)

Each type of test requires a different protocol and produces
different results

The nature of impostors is an important part of the testing of
biometric systems
113
Multimodal User Authentication: From Theory to Practice
K. Jain, L. Hong, and Y. Kulkarni: A Multimodal Biometric System Using Fingerprint,
Face, and Speech, Technical Report MSU-CPS-98-32, Department of Computer Science,
Michigan State University
Acceptance rate (%)
Example
False acceptance rate (%)
114
Multimodal User Authentication: From Theory to Practice
R.W. Frischholz and U. Dieckmann: “BioID: A Multimodal Biometric Identification
System”, IEEE Computer, vol. 33, no. 2, pp. 64-68, February 2000
Example

BioID SDK by HumanScan (Germany)
 http://www.humanscan.de/products/bioid/index.php
 BioID SDK offers multimodal biometrics in the form of a software
development kit
 BioID SDK offers three biometrics:
• Face recognition
• Voice recognition
• Lip movement recognition
 Since BioID uses true multimodality, the preferred way of using it is by using all of the
three biometrics together. But BioID can be easily configured (e.g. using the Control
Panel) to use any of the above three biometrics alone or in any combination
 Basic features include:
• User enrollment wizard
• User recognition (verification or identification)
• User template and authorization management
• Enrollment management
• Template storage to database, local PC or Smart Card
• Support BioAPI
115
Multimodal User Authentication: From Theory to Practice
Perspectives and future challenges

Multimodal biometrics will play vital roles in the next generation of
automatic identification systems

Future challenges in multimodal biometric systems










Accuracy is still an issue for most of existing biometrics
Feature extraction
Dealing with dynamic information using a small amount of training data
How to combine information (fusion) and make use of the strengths of each
modality
Collection of a multimodal and realistic database (most of the existing
databases are unimodal)
Integrating higher level of information (e.g. for speech, prosodic modeling,
word/phrase usage)
Scalability
Establishment of common standards along the lines of GSM in the mobile world
Dealing with privacy concerns
Ease of use and development
116
Multimodal User Authentication: From Theory to Practice
Demonstrations

Iris recognition

Speaker and Fingerprint recognition for door entrance system
117
Multimodal User Authentication: From Theory to Practice
Forthcoming events

IEE PROCEEDINGS VISION, IMAGE AND SIGNAL PROCESSING,
Special Issue on BIOMETRICS ON THE INTERNET, Aladdin
Ariyaeeinia, University of Hertfordshire, UK, Guest Editor
(http://www.iee.org/Publish/Support/Auth/Authproc.cfm)

Multimodal User Authentication workshop, Santa Barbara, CA,
U.S.A. December 2003 (http://mmua.cs.ucsb.edu )

International Conference on Biometric Authentication, Hong
Kong, January 2004 (http://www4.comp.polyu.edu.hk/~icba/)

EURASIP, Applied Signal Processing,
Special Issue on Biometric Signal Processing (4th quarter
2003) (http://asp.hindawi.com)
118
Multimodal User Authentication: From Theory to Practice
Acknowledgments
(for inputs, fruitful discussions and help)



Institut Eurécom (Florent Perronnin)
Panasonic Speech Technology Laboratory
University of Thessaloniki
119
Multimodal User Authentication: From Theory to Practice
Bibliography
BOOKS




S. Nanavati, M. Thieme, and R. Nanavati: “Biometrics, Identity Verification in a Networked
World”, Wiley Computer Publishing, 2002.
J. Ashbourn: “Biometrics, Advanced Identity Verification, The Complete Guide”, Springer,
2000.
L.C. Jain, U. Halici, I. Hayashi, S.B. Lee, and S. Tsutsui, editors: “Intelligent Biometric
Techniques in Fingerprint and Face Recognition”, The CRC Press International Series on
Computational Intelligence, 1999.
A. Jain, R. Bolle, and S. Pankanti, editors: “Biometrics, Personal Identification in Networked
Society”, Kluwer Academic Publishers, 1998.
120
Multimodal User Authentication: From Theory to Practice
Bibliography
INTRODUCTION

Y. W. Yun The ‘123’ of Biometric Technology.
OVERVIEW

J.-L. Dugelay, J.-C. Junqua, C. Kotropoulos, R. Kuhn, F. Perronnin, and I. Pitas: “Recent
Advances in Biometric Person Authentication”, ICASSP 2002, pp. IV 4060-IV 4063.
COURSE (slides)

J. Wayman (San José State University) Biometrics & How they Work.
121
Multimodal User Authentication: From Theory to Practice
Bibliography
MULTIMODAL



A.J. Mansfield and J.L. Wayman: “Best Practices in Texting and Reporting Performance of
Biometric Devices”, NPL Report CMSC 14/02.
K. Jain, L. Hong, and Y. Kulkarni: “A Multimodal Biometric System Using Fingerprint, Face,
and Speech”, Technical Report MSU-CPS-98-32, Department of Computer Science, Michigan
State University.
L. Hong and A. Jain: “Integrating Faces and Fingerprints for Personal Identification”, IEEE
Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 12, pp. 1295-1307, December
1998.
122
Multimodal User Authentication: From Theory to Practice
Bibliography
FUSION





R. Brunelli and D. Falavigna: “Person Identification Using Multiple Cues”, IEEE Trans. Pattern
Analysis and Machine Intelligence, vol. 17, no. 10, pp. 955-966, October 1995.
J. Kittler, M. Hatef, R.P.W. Duin, and J. Matas: “On Combining Classifiers”, IEEE Trans. Pattern
Analysis and Machine Intelligence, vol. 20, no. 3, pp. 226-239, March 1998.
R.W. Frischholz and U. Dieckmann: “BioID: A Multimodal Biometric Identification System”,
IEEE Computer, vol. 33, no. 2, pp. 64-68, February 2000.
V. Chatzis, A.G. Bors, and I. Pitas: “Multimodal Decision-Level Fusion for Person
Authentication”, IEEE Trans. Systems, Man and Cybernetics, Part A, vol. 29, pp. 674-680,
November 1999.
C. Burges: “A Tutorial on Support Vector Machines For Pattern Recognition”.
123
Multimodal User Authentication: From Theory to Practice
Bibliography
LIPREADING – audio/video


B. Duc, E.S. Bigun, J. Bigun, G. Maitre, and S. Fischer: “Fusion of Audio and Video Information
for Multi-Modal Person Authentication”, Pattern Recognition Letters, vol. 18, pp. 835-843,
1997.
S. Ben-Yacoub, Y. Abdeljaoued, and E. Mayoraz: “Fusion of Face and Speech Data for Person
Identity Verification”, IEEE Trans. Neural Networks, vol. 10, no. 5, pp. 1065-1074, September
1999.
124
Multimodal User Authentication: From Theory to Practice
Bibliography
SPEECH



D.A. Reynolds and L.P. Heck: “Speaker Verification: From Research to Reality”, Tutorial,
ICASSP, Salt Lake City, Utah, May 7, 2001.
G. Doddington: “Speaker Recognition Based on Idiolectal Differences between Speakers”,
Eurospeech 2001, V. 4, pp. 2521-2524, Aalborg, Denmark, Sept. 3-7, 2001.
O. Thyes, R. Kuhn, P. Nguyen, and J-C. Junqua: “Speaker Identification and Verification Using
Eigenvoices”, ICSLP-2000, V. 2, pp. 242-245, Beijing China, Oct. 2000.
125
Multimodal User Authentication: From Theory to Practice
Bibliography
FACE



R. Chellappa, C.L. Wilson, and S. Sirohey: “Human and Machine Recognition of Faces: A
Survey”, Proceedings of the IEEE, vol. 83, no. 5, pp. 705-740, May 1995.
M. Turk and A. Pentland: “Eigenfaces for Recognition”, J. Cognitive Neuroscience, vol. 3, no.
1, pp. 71-86, 1991.
S. Pigeon and L. Vandendorpe: “Image-based Multi-Modal Face Authentication”, Signal
Processing, vol. 69, pp. 59-79, August 1998.
IRIS

J. Daugman: “Recognizing Persons by their Iris Patterns”, in Biometrics, Personal
Identification in Networked Society, pp. 103-121, A. Jain, R. Bolle and S. Pankanti, editors,
Kluwer Academic Publishers, 1998.
RETINA

http://www.retinaltech.com.
EAR


B. Moreno et al.: « On the Use of Outer Ear Images for Personal Identification in Security
Applications » 1999 IEEE.
M. Burge and W. Burge: « Ear Biometrics for Machine Vision ».
126
Multimodal User Authentication: From Theory to Practice
Bibliography
FINGERPRINT


R. Adhami and P. Meenen: “Fingerprinting for Security”, IEEE Potentials, Vol. 20, no. 3, pp. 3338, Aug.-Sept. 2001.
A. Jain and S. Pankanti: “Automated Fingerprint Identification and Imaging Systems”,
Advances in Fingerprint Technology, 2nd Ed., Elsevier Science, New York, 2001.
127
Multimodal User Authentication: From Theory to Practice
Bibliography
HAND GEOMETRY

R. Sanchez-Reillonet et al. : “Biometric Identification through Hand Geometry
Measurements”, IEEE PAMI Vol. 22, No. 10, 2000.
SIGNATURE

A. Jain et al. : “On-line Fingerprint Verification”, IEEE T-PAMI Vol. 19, No. 4, April 97.
128
Multimodal User Authentication: From Theory to Practice
Bibliography
DATABASE


S. Pigeon and L.Vandendorpe: “The M2VTS multimodal Face Database”, Lecture Notes in
Computer Science: Audio- and Video-Based Biometric Person Authentication (J. Bigun, G.
Chollet, and G. Borgefors Eds.), vol. 1206, pp. 403-409, 1997.
K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre: “XM2VTSDB: The Extended M2VTS
Database”, in Proc. 2nd Int. Conf. on Audio- and Video-Based Biometric Person Authentication,
March 1999.
EVALUATION

P. J. Philips, et al.: « The Feret evaluation methodology for face-recognition algorithms », IEEE
T-PAMI, Vol. 22, No. 10, Oct. 2000.
129
Multimodal User Authentication: From Theory to Practice
End.


jcj@research.panasonic.com (Jean-Claude JUNQUA)
jld@eurecom.fr (Jean-Luc DUGELAY)
130