以區域二元圖樣與部分比對為基礎之
人臉辨識
Face Recognition with Local Binary
Patterns and Partial Matching
Presenter: 施佩汝
Advisor: 歐陽明 教授
1
Outlines
•
•
•
•
Motivation
Implementation
Result
Conclusion
2
MOTIVATION
3
Motivation
4
Publication
• Che-Hua Yeh, Pei-Ruu Shih, Kuan-Ting Liu, Yin-Tzu
Lin, Huang-Ming Chang, Ming Ouhyoung. A
Comparison of Three Methods of Face
Recognition for Home Photos. ACM Siggraph,
poster, 2009.
5
Problem Statement
6
Main Contribution
• Improve Local Binary Patterns by using Partial
Matching Metric
• Better Performance in Home Photos
7
IMPLEMENTATION
8
System Overview
Images
Descriptors
Build Descriptor
Pre-Processing
Clustering
Calculate
LBP
Build
Descriptor
9
System Overview
Images
Descriptors
Build Descriptor
Pre-Processing
Clustering
Calculate
LBP
Build
Descriptor
10
Pre-Processing
11
System Overview
Images
Descriptors
Build Descriptor
Pre-Processing
Clustering
Calculate
LBP
Build
Descriptor
12
System Overview
Images
Descriptors
Build Descriptor
Pre-Processing
Clustering
Calculate
LBP
Build
Descriptor
13
Local Binary Patterns [PAMI2006]
• An operator to encode the relationship of a
pixel and its neighbors.
Z
LBP   s Z 0  Z p  2 p
1
8
Z2 Z3
Z8 Z0 Z4
p 1
Z7 Z6 Z5
 1, if x  0
sx   
0, ot herwise
58
99
21
54
54
86
67
12
13
1
14
Local Binary Patterns [PAMI2006]
• An operator to encode the relationship of a
pixel and its neighbors.
Z
LBP   s Z 0  Z p  2 p
1
8
Z2 Z3
Z8 Z0 Z4
p 1
Z7 Z6 Z5
 1, if x  0
sx   
0, ot herwise
58
99
21
54
54
86
67
12
13
1
1
15
Local Binary Patterns [PAMI2006]
• An operator to encode the relationship of a
pixel and its neighbors.
Z
LBP   s Z 0  Z p  2 p
1
8
Z2 Z3
Z8 Z0 Z4
p 1
Z7 Z6 Z5
 1, if x  0
sx   
0, ot herwise
58
99
21
54
54
86
67
12
13
1
1
0
16
Local Binary Patterns [PAMI2006]
• An operator to encode the relationship of a
pixel and its neighbors.
Z
LBP   s Z 0  Z p  2 p
1
8
Z2 Z3
Z8 Z0 Z4
p 1
Z7 Z6 Z5
 1, if x  0
sx   
0, ot herwise
58
99
21
54
54
86
67
12
13
1
1
0
1
17
Local Binary Patterns [PAMI2006]
• An operator to encode the relationship of a
pixel and its neighbors.
Z
LBP   s Z 0  Z p  2 p
1
8
Z2 Z3
Z8 Z0 Z4
p 1
Z7 Z6 Z5
 1, if x  0
sx   
0, ot herwise
58
99
21
1
1
0
54
54
86
1
67
12
13
0
18
Local Binary Patterns [PAMI2006]
• An operator to encode the relationship of a
pixel and its neighbors.
Z
LBP   s Z 0  Z p  2 p
1
8
Z2 Z3
Z8 Z0 Z4
p 1
Z7 Z6 Z5
 1, if x  0
sx   
0, ot herwise
58
99
21
54
54
86
67
12
13
1
1
0
1
0
0
19
Local Binary Patterns [PAMI2006]
• An operator to encode the relationship of a
pixel and its neighbors.
Z
LBP   s Z 0  Z p  2 p
1
8
Z2 Z3
Z8 Z0 Z4
p 1
Z7 Z6 Z5
 1, if x  0
sx   
0, ot herwise
58
99
21
54
54
86
67
12
13
1
1
0
1
1
0
0
20
Local Binary Patterns [PAMI2006]
• An operator to encode the relationship of a
pixel and its neighbors.
Z
LBP   s Z 0  Z p  2 p
1
8
Z2 Z3
Z8 Z0 Z4
p 1
Z7 Z6 Z5
 1, if x  0
sx   
0, ot herwise
58
99
21
1
54
54
86
1
67
12
13
1
1
0
1
0
0
21
Local Binary Patterns [PAMI2006]
• An operator to encode the relationship of a
pixel and its neighbors.
Z
LBP   s Z 0  Z p  2 p
1
8
Z2 Z3
Z8 Z0 Z4
p 1
Z7 Z6 Z5
 1, if x  0
sx   
0, ot herwise
58
99
21
1
54
54
86
1
67
12
13
1
1
0
1
0
LBP = 11010011
0
22
System Overview
Images
Descriptors
Build Descriptor
Prepared-Works
Clustering
Calculate
LBP
Build
Descriptor
23
Facial Image Descriptor
• They use Spatially Enhanced Histogram in
original Local Binary Pattern. [PAMI2006]
24
Local Patches
• We sample a m m patch for every s pixels.
• There are S patches for one image.
s s
25
Spatial Block
[CVPR2007]
• We use three concentric circles to describe a
patch.
26
Descriptor
• Build a descriptor for one face.
 f11

 f 21
 

 f K 1
f12
f 22

fK 2
 f1K 

 f2K 
  

 f KK 
27
System Overview
Images
Descriptors
Build Descriptor
Pre-Processing
Clustering
Calculate
LBP
Build
Descriptor
28
System Overview
Images
Descriptors
Build Descriptor
Complete-Linkage Clustering
Pre-Processing
Calculate
LBP
Build
Descriptor
Compute all
the similarities
Clustering
29
System Overview
Images
Descriptors
Build Descriptor
Complete-Linkage Clustering
Pre-Processing
Calculate
LBP
Build
Descriptor
Compute all
the similarities
Clustering
30
Similarity
• They use the weighted Chi-Square distance in
original Local Binary Pattern. [PAMI2006]
2

• 2
S ij  M ij 
 w S , M    wi 
S M
i
j
ij
ij
31
,
Partial Matching [ICCV2009]
• Step1:
– Compute the similarity of each patch from one
image with the nearby patches in another image.
   min
d f
(1)
ij
k ,l:|is ks|r ;| js ls|r
Image 1:
I(1)
Image 2:
I(2)

(1)
ij
sim f , f
( 2)
kl

32
Partial Matching [ICCV2009]
• Step2:
– Sort the similarities of all patches.
d1 , d 2 ,...,dS ,...,d s   Sortd  f i, j 1
(1)
ij
K
– dαS is the similarity of I(1) to I(2).
d (I
(1)
 I )  dS
( 2)
33
Partial Matching [ICCV2009]
• Step3:
– Calculate the similarity of I(2) to I(1)
d ( I ( 2)  I (1) )
34
Partial Matching [ICCV2009]
• Step4:
– Use the maximum of two similarity


D( I (1) , I ( 2) )  max d ( I (1)  I ( 2) ), d ( I ( 2)  I (1) )
35
System Overview
Images
Descriptors
Build Descriptor
Complete-Linkage Clustering
Pre-Processing
Calculate
LBP
Build
Descriptor
Compute all
the similarities
Clustering
36
Hierarchical Clustering
• Build a tree-based hierarchical taxonomy
(dendrogram) from a set of documents.
Material Selected from Introduction to
Information Retrieval, Christopher D.
Manning, Prabhakar Raghavan and
37
Hinrich Schütze, 2008.
Hierarchical Clustering
• Clustering obtained by cutting the dendrogram
at a desired level: each connected connected
component forms a cluster.
Material Selected from Introduction to
Information Retrieval, Christopher D.
Manning, Prabhakar Raghavan and
38
Hinrich Schütze, 2008.
Hierarchical Complete-Linkage
Clustering
• Similarity of the “furthest” points.
• Makes “tighter,” spherical clusters that are
typically preferable.
Material Selected from Introduction to
Information Retrieval, Christopher D.
Manning, Prabhakar Raghavan and
39
Hinrich Schütze, 2008.
Performance Optimization
• 4 threads in Quad-Core system
Images
Descriptors
Build Descriptor
Complete-Link Clustering
Pre-Processing
Calculate
LBP
Build
Descriptor
Compute all
the similarities
Clustering
40
Performance Optimization
• 4 threads in Quad-Core system
– 3 times faster than single thread.
• 73 minutes to 24 minute for 309 images.
Images
Descriptors
Build Descriptor
Complete-Link Clustering
Pre-Processing
Calculate
LBP
Build
Descriptor
Compute all
the similarities
Clustering
41
RESULT
42
FERET Result
•
•
•
•
fa: gallery, 994 images
fb: alternative facial expression, 992 images
dup1: the photos taken after later, 736 images
dup2: the photos taken at least one year after
the gallery, 228 images
43
FERET Result
Accuracy
fb
dup1
dup2
LBP [PAMI2006]
95.67%
59.92%
45.61%
Our Result
98.89%
71.33%
68.42%
Time※
Register
fb
dup1
dup2
LBP [PAMI2006]
46 seconds
102 seconds
67 seconds
21 seconds
Our Result
4 minutes
224 minutes
168 minutes
5 minutes
※ The time results are computed in multithreads version.
44
Experiments
• Home Photo Dataset I
– 309 images, 5 subjects
• Home Photo Dataset II
– 838 images, 8 subjects
45
Evaluation
•
•
•
•
•
Cluster Number
Unknown Number
Pair-wise Precision
Pair-wise Rand Index
Executing time
46
Unknown Number
• The number of clusters which contain only one
component.
47
Precision/Rand Index
precision 
Assigned
Ground
Truth
Same
Cluster
Different
Clusters
Same
Cluster
Different
Clusters
tp
fn
tp
recall 
tp  fn
rand index 
fp
tn
tp
tp  fp
tp  tn
tp  fn  fp  tn
48
Dataset-I Result
#Clusters
#Unknown
Precision
Rand
Index
Time ※
Picasa Web
94
73
99.92%
0.829816
10 seconds
Picasa PC
99
75
100%
0.865002
3 minutes
LBP [PAMI2006]
100
31
90.39%
0.811856
4 seconds
LID_PM
99
37
98.78%
-
11 minutes
Our Result
100
39
99.46%
0.816290
24 minutes
LID+PM
100
43
99.24%
-
25 minutes
[ICCV2009]
[Chang2010]
LBP: Local Binary Pattern, PM: Partial Matching, SB: Spatial Block
※ The time results are computed in multithreads version.
49
Dataset-I Result
LBP
Our Result
50
Dataset-I Result
• Wrong clustering result in LBP
• The clusters in our result
51
Dataset-II Result
#Clusters
#Unknown
Precision
Rand
Index
Time ※
Picasa Web
195
47
99.49%
0.876229
10 seconds
Picasa PC
253
150
100%
0.888492
10 minutes
LBP [PAMI2006]
253
45
91.70%
0.70261
17 seconds
LID_PM
253
79
97.88%
-
51 minutes
Our Result
253
64
99.59%
0.871187
163 minutes
LID+PM
253
89
99.82%
-
136 minutes
[ICCV2009]
[Chang2010]
LBP: Local Binary Pattern, PM: Partial Matching, SB: Spatial Block
※ The time results are computed in multithreads version.
52
Dataset-II Result
LBP
Our Result
53
Dataset-II Result
• Wrong clustering result in LBP
• Clusters in our result
54
Demo
• Face Recognition with web camera
55
CONCLUSION
56
Conclusion
• LBP is an efficient algorithm for face
recognition.
• Partial Matching is good for the different facial
expression or different illumination in facial
images.
• Our system has better performance than LBP.
57
Future Works
• Improve performance by GPU.
• Use other extension of LBP combined with
Partial Matching.
58
Thank You for Your
Attention!
59
Result
Home Photo Dataset II
1
1
0.95
0.95
0.9
0.9
Precision
Precision
Home Photo Dataset I
0.85
0.85
0.8
0.8
0.75
0.75
0.7
0.7
0
0.05
0.1
0.15
0
0.2
0.05
LBP+SB
LBP+PM
0.15
0.2
Recall
Recall
LBP
0.1
LBP+LID+SB
LBP
LBP+SB
LBP+PM
LBP+SB+PM
LBP: Local Binary Pattern, PM: Partial Matching, SB: Spatial Block
60