Tagging of digital historical
images
Authors:
A. N. Talbonen (antal@sampo.ru)
A. A. Rogov (rogov@psu.karelia.ru)
Petrozavodsk state university
General tagging model
Tag DB
Object
selection
Image
collection
Tag
attribution
Object
DB
Indexing
File
Tags
I1
……
I2
……
Full-text
index
General research features
 Research is based on analysis of
image collection of White Sea-Baltic
Sea Canal provided by National
museum of Karelia
 Collection consists of about 8k images
with resolution 75 dpi.
1. Face tagging
General features
 Predominance of small-sized objects (width is less than
40 pixels)
 No database
 Available expert
Distribution of object’s size
1. Face tagging
General algorithm
 Object (face) detection.
 Computing of pairwise distances
between objects.
 Tagging (for each object):
 The system displays a list of the most
similar objects.
 The expert determines a relationship
between objects
 Object tags are specified
1. Face tagging
Face detection features
 There is OpenCV library (OpenCvSharp in
C#) and it’s method
cv::CascadeClassifier::detectMultiScale
(haarDetectObject in C#) (Viola-Jones
implementation) being used for face
detection
 Viola-Jones method parameters are used to
affect on precision and recall on face
detection results
 There is face recognition method based on
Local Binary Patterns being used to
improve the quality of Viola-Jones results
1. Face tagging
Face detection diagram
Source image
Face detection
Object
Face
objects
Fake
objects
Training set
Recognition
Object is
a face
Yes
Insert in result
collection
1. Face tagging
Local binary patterns (LBP)
Original LBP filter
Advanced LBP filters
1. Face tagging
Local binary patterns
Uniform codes
(patterns)
Rotation invariant
codes
1. Face tagging
Local binary patterns
Computing of face object histogram
Weight matrix
1. Face tagging
Face detection experiment
 The purpose is to find the LBP modification with the
best detection rates
 Experiment features:
 Sample of 1070 images
 Assessing features
 Fake object when:



Object is not a face
Faces are recognized weakly
Faces turned at an angle greater than 90 degrees
 Face object when:


Object is a face
Object is an image of people: portraits, paintings,
sculptures
 12 different LBP modifications were used
1. Face tagging
Face detection experiment results
1. Face tagging
Face recognition experiment
 Purpose is to find the LBP
modification with the best face
recognition rates
 Experiment features
 Training set contains 19 objects including
3 relevant pairs of face objects and 1
relevant pair of fake objects
 10 LBP modifications were used
1. Face tagging
Face recognition experiment
1)
2)
3)
4)
5)
6)
7)
8)
9)
10)
11)
12)
13)
14)
15)
16)
17)
18)
19)
Pairs: {1, 15}, {3, 14}, {4, 13}, {7, 9}
1. Face tagging
Face recognition experiment results
Modification
Precision
LBP8,1
0,38
LBP16,1
0,25
LBP8,2
0,50
LBP16,2
0,50
LBP8,3
0,50
LBP16,3
0,75
ri
Взвешенный LBP16,3
0,50
riu
Взвешенный LBP16,3
0,38
u
Взвешенный LBP16,3
0,63
Взвешенный LBP16,3
1,00
1. Face tagging
Face comparing
Training set object’s histograms:
Objects at position (row, col): (1,1) and (3, 4) correspond to
fake objects and have similar histograms very different from the rest
2. Texture tagging
General features
 The classifier with tags based on
moments is built
 Texture searching is based on the
built classifier
 Search involves finding the segments
corresponding to different textures
 Minimal segment size to be include in
result is 100 pixels
2. Texture tagging
Moment-based segmentation
Moment calculation function:
Source image I
Moment image M00
Moment image M10
Moment image M01
2. Texture tagging
Moment-based segmentation
Moment feature calculation function:
F00
Binary segmentation example
F10
Precision: 96,7 %
F01
2. Texture segmentation
Implementation features
 Each moment is an image
 Moment computing is based on
library OpenCV and it’s method
cv::filter2D
 Parameter seek is based on
developed experiment
2. Texture tagging
Parameter seek example
Moment
window size
Moment feature
Window size
Sigma
Precision
9
49
0,01
95,285
9
39
0,005
95,1782
9
39
0,02
95,1752
9
44
0,005
95,1355
9
49
0,015
95,1324
14
14
0,02
93,8416
14
14
0,005
93,7103
14
19
0,005
92,7826
14
19
0,015
92,7826
14
34
0,015
92,5293
14
29
0,015
92,395
14
34
0,02
92,3248
24
24
0,02
87,9639
39
19
0,01
87,9639
2. Texture tagging
Classifier features
 Set of textures of several classes is
given
 Each class is assigned a set of tags
 Each image is subjected to a
separate texture search
 Each texture found adds appropriate
set of tags to the source image
2. Texture tagging
Example
Source image
2. Texture tagging
Example
Classifier example
Classifier textures example
2. Texture tagging
Experiment
 Purpose is to evaluate the search quality
 Experiment features
 Sample of 100 images
 Classifier contains 2 textures
House roof
House wall
2. Texture tagging
Search quality evaluate method
Single texture estimations:
R

F
ij
Prj
i
ij
i
R

E
ij
; Re j
ij
General estimations:
R
Pr 
F
ij
i, j
ij
i, j
i
i
R
; Re 
E
ij
i, j
ij
i, j
Eij
Fij
- Flag of belonging to
assessed collection
- Flag of belonging to
search result
Rij  Eij  Rij
Flag of relevance
2. Texture tagging
Experiment results
Thanks for your attention!