Pattern Recognition and Image Analysis Group (RFAI)
Document (Image) Analysis related work
Laboratory of Computer Science (LI)
François Rabelais University
Tours city, France
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
6.7. Graph matching and embedding
2
Tours city
Paris
Tours
- 137 046 people (2009)
- 204 km southwest of Paris
- Region « centre - Indre et loire »
- 1h20 from Paris by high speed train
- Direct train connection to Charles de
Gaulle -Orly airport in 2h00
3
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
6.7. Graph matching and embedding
4
François-Rabelais University,
les deux lions / Portalis
François Rabelais University
Faculties
Art & human sciences,
economy, business & management, health, information and
technology
Students
21 207 (2 500 foreign students)
Teachers
1 300
Support staff 1000
Laboratories
40
Place
5
François Rabelais
i.e. a famous French
writer of XV° Century
5
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
6.7. Graph matching and embedding
6
School of Engineering Polytech
- 12 schools in France
(Grenoble, Lille, Marseille,
Montpellier, Nantes, NiceSophia, Paris-UPMC, Paris
ORSAY, Savoie, Orléans,
Tours, Clermont-Ferrand)
- 12 000 students
-720 students
- 5 departments (with Labs)
Urban Planning
CITERES
Mechanics
LMR
Electronics
LMP
Computer Science
Embedded computing
LI
7
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
8
Laboratory of Computer Science
77 people, 5 research groups (2009)
Pattern Recognition
and Image Analysis
Data Bases and Natural
Language Processing
Visual Data Mining and
Biomimetic Algorithms
Scheduling and Control
Handicap and New Technologies
9
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
10
Pattern Recognition and Image Analysis (RFAI) (1)
Medical Imaging
- Image segmentation (ultrasound, MRI)
- Video analysis, 3D reconstruction
Document Image Analysis
- Layout analysis & document recognition
- OCR, word spotting & signature verification
- Symbol recognition & spotting
- Content based Image Retrieval
- Camera based Recognition
- Graph matching and embedding
Machine learning for time series prediction
200
R é e lle
180
E s tim é e
160
140
120
100
80
60
40
11
20
0
1921 1926 1931 1936 1941 1946 1951 1956 1961 1966 1971 1976
Professors
Pattern Recognition and Image Analysis (RFAI) (2)
PhD
Romuald
Boné
Alireza
Alaei
Pascal
Makris
Hubert
Cardot
Sabine
Barrat
Julien
Olivier
Jean-Yves
Ramel
Thierry
Brouard
Nicolas
Ragot
Mathieu
Delalandre
Partha
Roy
Muzzamil
Luqman
Nicolas
Sidere
Romain
Raveaux
Gilles
Verley
12
PhD Students & engineers
Pattern Recognition and Image Analysis (RFAI) (3)
Fareed
Ahmed
Anh Khoi
Ngo ho
Ahmed Ben
Salah
The Anh
Pham
Aymen
Cherif
Cyrille
Faucheux
Frédéric
Rayar
13
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
14
Partnership contracts
Scholarships
Local government
projects
National projects
International projects
2003-2006

Navidomass 2006-2009

EPEIRES
2004-2007

BVH
2004-today

ATOS
2005-today

PIVOAN
2008-2009
Madonne
HEC
2005-2011
SNECMA
2008-2011
AAP
2010-2011
VIED
2010-2013
Bnf
2010-2013
Digidoc
2011-2014
Google
2011-2012
ISRC2011
2011-2012
DOD
2011-2015
IndoFrench
2012-2015
SPD
2012-2015
Projects
& partners outline (1)

DIA people plot
20



16
12



8
4

0
2005 2006 2007 2008 2009 2010 2011 2012 2013 2014




15
Projects & partners outline (2)
National projects
People
Institutes
Length
Funding
ACI MADONNE
2003-2006
55
8
2 years
110 k€
ANR Navidomass
2006-2009
40
7
3 years
443 k€
Technovision ÉPEIRES
2004-2007
30
7
3 years
100 k€
ANR Digidoc
2011-2014
18
7
3 years
866 k€
Institut de Recherche en
Informatique et Systèmes
Aléatoires (Rennes)
Laboratoire d'Informatique
de Traitement
de l'Information (Rouen)
Laboratoire d’informatique
image et interaction
(La Rochelle)
Centre de Recherche en
Informatique de Paris 5
(Paris)
Laboratoire Lorrain de
Recherche en Informatique
et ses Applications (Nancy)
Laboratoire Informatique
(Tours)
Centre d’Etude Supérieures de la
Renaissance (Tours)
Laboratoire d'InfoRmatique
en Image et Systèmes
d'information (Lyon)
Laboratoire Bordelais de
Recherche en Informatique
(Bordeaux)
16
Projects & partners outline (3)
Local government projects (i.e. projets region centre)
Length
Funding
Institutes
People
PIVOAN
3
1
1 year
33 k€
AAP
3
1
1 year
38 k€
M. Luqman
T.H. Pham
So famous !
Centre des études supérieures de
la renaissance – bibliothèque
virtuelle humaniste
Maison des Sciences de l'Homme
PhD Scholarships
R.J. Qureshi
Partnership contracts
Higher Education Commission
(HEC) of Pakistan
Vietnam International Education
Development (VIED)
international high-technology
group in aerospace, defense and
security
Bibliothèque Nationale de
France - portail Gallica
Bilateral program
2
3 year
Funding
Length
3
Institutes
People
IndoFrench
70 k€
Digitalisation company
capturing, automatically
processing, and managing all
company’s incoming
documents
Atos Origin is a leading
international IT services
provider for business solutions
17
Projects & partners outline (4)
Computer Vision Center
Document Analysis Group
Barcelona - Spain
“J. Llados, E. Valveny”
Dept. of Computer Science and IS
Osaka Prefecture University
Osaka - Japan
“K. Kise”
Indian Statistical Institute
Kolkata - India
“U. Pal”
Computational Intelligence Laboratory
Athens - Greece
“B. Gatos”
18
Camera based
Recognition
CBIR
Graph matching &
embedding
Symbol recognition &
spotting
OCR, word spotting &
signature verification
Layout analysis &
document recognition
Projects &
partners outline (5)

Madonne
2003-2006
Navidomass
2006-2009
EPEIRES
2004-2007
BVH
2005-today
ATOS
2005-2009
PIVOAN
2008-2009
HEC
2005-2011
SNECMA
2008-2011
AAP
2010-2011
VIED
2010-2013
Bnf
2010-2013

Digidoc
2011-2014

Google
2011-2012

ISRC2011
2011-2012
DOD
2011-2015
IndoFrench
2012-2015

SPD
2012-2015


















19
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
20
Layout analysis & document recognition
“AGORA (1)”
People
Jean-Yves Ramel
Funding CESR partnership, Madonne, PIVOAN
Starting 2005
Ref
J.Y. Ramel and al. User-driven Page
Layout Analysis of historical printed
Books. IJDAR, 2007.
(1) Text/graphics separation
Foreground map: adaptive binarization
[Saul2000] with connected component
labeling, text/graphics separation is done
in terms of size of connected components
(2) Line/word segmentation
1. Background map: statistical
distribution of white and black pixel
on horizontal and vertical scanline
2. Fusion: word segmentation (i.e.
connected components grouping) is
done in terms of thresholding on the
background map.
Layout analysis & document recognition
“AGORA (2)”
People
Jean-Yves Ramel
(3) Interactive system (i.e. user driven analysis)
Funding CESR partnership, Madonne, PIVOAN
Starting 2005
Ref
J.Y. Ramel and al. User-driven Page
Layout Analysis of historical printed
Books. IJDAR, 2007.
Vertical position
average = 0,46 std deviation = 0,41
Horizontal position
average = 0,51 std deviation = 0,07
(4) Results, since 2005: 300 books (50 000 pages)
http://www.bvh.univ-tours.fr/
22
Layout analysis & document recognition
“Document image characterization (1)”
People
Nicholas Journet
(1) Descriptor based on five features:
Funding CESR partnership & Madonne project
Starting 2006
Ref
N. Journet and al. Document Image
Characterization Using a
Multiresolution Analysis of the
Texture: Application to Old
Documents. IJDAR, 2008.
(1) main direction
Directional
rose
(2) isotropy
(3) standard deviation
Spatial
(4)
ink/paper transition
(5)
white spaces separating
the collateral elements
23
Layout analysis & document recognition
“Document image characterization (2)”
People
Nicholas Journet
Funding CESR partnership & Madonne project
Starting 2006
Ref
N. Journet and al. Document Image
Characterization Using a
Multiresolution Analysis of the
Texture: Application to Old
Documents. IJDAR, 2008.
(2) Segmentation:
-features are extracted at four different resolution (45 = 20 features)
- features are then processed with the clustering algorithm CLARA
(Clustering LARge Applications) [Kaufman1990] to achieve automatic
segmentation in text/graphics/background
24
Layout analysis & document recognition
“Document image characterization (3)”
People
Nicholas Journet
Funding CESR partnership & Madonne project
(3) Indexing applied on two different problems
-Layout retrieval, distance is based on a contingency table [Younes2004]
-Graphics retrieval, distance based on a dissimilarity function
Starting 2006
Ref
N. Journet and al. Document Image
Characterization Using a
Multiresolution Analysis of the
Texture: Application to Old
Documents. IJDAR, 2008.
Handmade dataset I (400 images)
Handmade dataset II (400 images)
25
Layout analysis & document recognition
“Cognitive digitalization”
People
A.K. Ngo ho, N. Ragot,
J.Y. Ramel
Funding Digidoc project
Starting 10/2011
Ref
Na
Topic: Incremental and interactive learning for document image, application for
intelligent cognitive scanning of old documents.
Problematic:
- Estimate the scan parameters according to usage, past experience.
- Improve the scan parameters for a document during the scanning.
- Detect the default settings for a document, a collection, a work.
26
Layout analysis & document recognition
“Document classification”
Form
People
Mathieu Delalandre
Funding DOD project
Starting 12/2011
Ref
Na
Publicity
Topic: Recognition of administrative
forms for companies
Problematic:
- high variability “600 to 800 classes”
- binary images at 300 dpi
- time constraint:  to 1,5 s per image
- commercial systems can’t outperform
a 60% recognition rate
Goals:
1. To gain in robustness (set of adapted
and robust specialists)
2. To gain in flexibility (self
learning, content adaptation)
Free letter
Acknowledge reply
27
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
28
OCR, word spotting and signature verification
“Robust OCR-I (1)”
People
Key idea : improving OCR robustness by using similar technics as those used
for handwriting recognition:
- Hidden Markov Models without explicit segmentation
- Adapting a polyfont OCR to specificities of pages (fonts/noise)
Kamel Ait-Mohand
Funding Navidomass project
Starting 2006
Ref
K. Ait-Mohand and al. Structure
Adaptation of HMM Applied to OCR.
ICPR, 2010.
(1) Feature extraction is based on a sliding window and HoG features (no
word/character segmentation)
Sliding window
initial model
Structure
adaptation
training
New model
(2) HMM classification and training
- HMM characters models are learnt on a synthetic dataset (numerous
fonts, degradations possible, no limits in the number of samples per
character) = > polyfont OCR system
- Each character model can be adapted to a specific font/book using only
few lines of transcriptions. The HMM model is adapted at the structure
level (number of states) and at the parameter level (Gaussian MAP
adaptation).
29
OCR, word spotting and signature verification
“Robust OCR-I (2)”
People
Kamel Ait-Mohand
Experiments done using 100 fonts with the degradation model of Baird
Funding Navidomass project
Starting 2006
Ref
K. Ait-Mohand and al. Structure
Adaptation of HMM Applied to OCR.
ICPR, 2010.
Training Testing
Degradation models
Baird
thresholding
12
sparse pixels
Font size
Resolution
300 dpi
Fonts
Image sets (lines)
70
30
10000
15000
Commerc Polyfont
ial OCR
Average (30 fonts)
blurred
Adapted
88.72%
91.70% 97.37%
69.21%
98.33% 98.68%
44.78%
85.46% 98.09%
52.73%
39.73% 74.89%
30
OCR, word spotting and signature verification
“Robust OCR-II”
People
J.Y. Ramel, N. Ragot,
S. Barrat
Funding SDP
Starting 10/20012
Ref
Na
Topic: digitalization and indexing of a military document database for retired pay
Problematic:
- large amount of data (800 000 applications every 3 years)
- large heterogeneity:
from XIX° century “middle” to today,
handwritten and typographic documents,
different languages,
no common layout,
different colors,
etc.
31
OCR, word spotting and signature verification
“Performances prediction/control of OCR”
People
Ahmed Ben Salah, K.
Ait-Mohand
Funding BNF
Starting 10/2010
Ref
Na
Problematic: control and cost reduction of the digitization service to know which
collection/document/part of document is OCRisable and at which quality
Select only adequate documents to be sent to the private service provider in
charge of the digitization and OCRisation
• Studies of relationships between meta-data information (date, format, …)
and OCR results => difficult without deep analysis of the pages
• Characterization of image content with SIFT+LBP; regression towards
OCR results
Real OCR Result 
0-70%
70-80%
80-85%
85-90%
90-100%
Good (≥ 88%)
0%
0%
6,1%
23,11%
77,12%
Undecidable (82%-88%)
5,5%
16%
53,43%
59,11%
20,34%
Bad (<82%)
94,55%
84%
40,47%
17,78%
2,54%
Perfomance expected 
Control of OCR quality assessed by the service provider
• Detection of text zones forgotten by OCR using correct detection performed
(contextual information) => in progress
• Verification of OCR result by matching with image (=> in a near futur)
32
OCR, word spotting and signature verification
“Semi-automatic transcription (1)”
People
Jean-Yves Ramel, Frédéric Rayar
Topic: user driven transcription of character in historical books
Funding CESR partnership, PIVOAN, Google
Starting 2008
Ref
S. Hocquet and al. Analyse de classes
de formes pour la transcription de
textes imprimés anciens. CIFED, 2010
(1) Segmentation process based on
Agora
 Standardized output (e.g. Alto)
(3) Transcription & Typography studies
(2) Clustering process
Finer description of shapes
Features extraction and selection
33
OCR, word spotting and signature verification
“Semi-automatic transcription (2)”
People
Jean-Yves Ramel, Frédéric Rayar
Funding CESR partnership, PIVOAN, Google
Experiments, The Vésales book
Pages
Starting 2008
Ref
connected
components
Custers (i.e. Classes)
40 000
S. Hocquet and al. Analyse de classes
de formes pour la transcription de
textes imprimés anciens. CIFED, 2010
150
1 062 081
10%
90%
> 10 occurrences
< 10 occurrences
93% of the text
7% of the text
0.5% (top 200)
89.5%
85 % of text
8% of text
Reasons is noise
spot
character
on verso
touching
characters
split
character
34
OCR, word spotting and signature verification
“Word Spotting (1)”
People
Topic: Word Retrieval in Historical Documents
P.P. Roy, J.Y. Ramel, F. Rayar
Funding AAP , Renon
Starting 10/2010
Ref
P. P. Roy and al. An efficient coarse-to-fine
indexing technique for fast text retrieval in
historical documents", DAS 2012.
Text Extraction
AGORA
Transcription by codebook
Manuscripts
Codebook of Primitives
Query word
Sequence of primitives
PrimitiveString
Matching
Word Detection
35
OCR, word spotting and signature verification
“Word Spotting (2)”
People
P.P. Roy, J.Y. Ramel,
F. Rayar
Funding AAP , Renon
Starting 10/2010
Ref
P. P. Roy, F.Rayar and
J.Y.Ramel. An efficient
coarse-to-fine indexing
technique for fast text
retrieval in historical
documents", DAS 2012.
Topic: Word Retrieval in Historical Documents
The codebook is created using a clustering
algorithm by template matching of
similarity
Overcoming of segmentation problems are
solved by the Water reservoir method.
Query word is thus converted into a string
of primitives. Approximate string
matching algorithm is used for string
matching
Tests done from
- 24 pages
-corresponding to 57324 primitives
-clustered in 183 representative primitives
-P/R computed with 20 query word images
36
OCR, word spotting and signature verification
“Multlingual Word Spotting”
People
N. Ragot, J. Y. Ramel, U.
Pal
Funding IFCPAR
Starting 04/2012
Ref
Na
Topic: Robust multilingual word spotting:
Problematic:
- Query by text/image
- Partial matching allowed (for occlusion, special characters)
- Matching in two steps : global (shape context) / local (HMM)
37
OCR, word spotting and signature verification
“Online signature verification (1)”
People
Nicolas Ragot
Funding ATOS project
Starting 2005
Ref
N. Ragot and al. Study of
Temporal Variability in
On-Line Signature
Verification. ICPR, 2008
Problematic: to evaluate impact of temporality (i.e. time evolution) on
signature, for performance evaluation of signature verification algorithms.
(1) Database acquisition
signers
18
sessions
12
200
180
160
signatures/
session
10
140
120
mean time
interval
2 weeks
100
80
60
40
20
total
signatures
2160 total
duration
25
months
0
0
50
1 00
150
200
250
300
Training
loop if rough
differences (based on
length, duration,
speed)
Enrollment
“5 signatures”
Final
acquisition
“5 signatures”
38
OCR, word spotting and signature verification
“Online signature verification (2)”
People
Nicolas Ragot
Funding ATOS project
Starting 2005
Ref
N. Ragot and al. Study of
Temporal Variability in
On-Line Signature
Verification. ICPR, 2008
variation
(2) Statistical analysis
(2.1) Global
i.e. without temporal variability
Speed
variation
Length
variation
Duration
correlation
yes
no
stable
(2.1) With temporal variability
Total duration per
signer/ session
Total length per signer/ session
(3) Performance evaluation
Authentication (i.e. recognition) algorithm
based on a Coarse to fine approach
- Coarse step on “basic” features
(length, duration)
- Fine step based on DTW
Proposed dataset
Dataset without
temporality
39
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
40
Symbol recognition & spotting
“Vectorization and GbR (1)”
People
Funding
Starting
Ref
Jean-Yves Ramel
(1) Contour detection, chaining and
polygonalisation [Wall1984]
Na
J.Y. Ramel. A Structural
Representation for
Understanding Line-Drawing
Images. IJDAR, 2000.
(2) Quadrilateral building
(2.1.) Matching
(2.2.) Sorting
(2.3.) Merging
41
Symbol recognition & spotting
“Vectorization and GbR (2)”
People
Funding
Starting
Ref
Jean-Yves Ramel
(3) Graph based representation
Na
J.Y. Ramel. A Structural
Representation for
Understanding Line-Drawing
Images. IJDAR, 2000.
(3) Pros and cons
Cons - lost of connectivity
Cons - parasite quadrilaterals
Pro - better representation of filled
& crossed areas
Symbol recognition & spotting
“Generation of synthetic documents (1)”
People
Funding
Starting
Ref
Mathieu Delalandre
Key idea
To use a same
background layer with
different symbol layers
Na
M. Delalandre and. Generation
of Synthetic Documents for
Performance Evaluation of
Symbol Recognition & Spotting
Systems. IJDAR, 2010.
Graphical documents are
composed of two layers
(1) Constraint model
M1
C1
M2
C2
M3
M4
C3
C4
p
L
c1
c2
symbol
model
loaded symbol

L1
p1
θ1
p2
bounding box and
control point
alignment
  0 , 2   L  0 ,1
L2
43
θ2
Symbol recognition & spotting
“Generation of synthetic documents (2)”
People
Funding
Starting
Ref
Mathieu Delalandre
Na
M. Delalandre and. Generation
of Synthetic Documents for
Performance Evaluation of
Symbol Recognition & Spotting
Systems. IJDAR, 2010.
(2) Building engine and user interaction
(2) run
(1) edit
(3) display
Symbol
Models
Building
Engine
Background
Image
44
Symbol recognition & spotting
“Generation of synthetic documents (3)”
People
Funding
Starting
Ref
Mathieu Delalandre
(3) Datasets
Na
M. Delalandre and. Generation
of Synthetic Documents for
Performance Evaluation of
Symbol Recognition & Spotting
Systems. IJDAR, 2010.
(4) Performance evaluation
- Goal is to evaluate variability impact of
produced datasets on spotting system(s)
- Experiments have been done from
the spotting system of R. Qureshi
Background sets
Mean localization results
Symbol recognition & spotting
“Graph scoring for symbol spotting (1)”
People
Rashid Qureshi
Funding HEC scholarship
(1) Graph based representation: based
on the Jean-Yves Ramel’s work
Starting 2005
Ref
R. Qureshi and al. Spotting
Symbols in Line Drawing Images
Using Graph Representations.
GREC, 2008.
(2) Seeds detection in graph: a set of scoring functions is
computed from all nodes and edges
Scoring functions
Edges
PE1
parallel segments
PE2
 junctions
PE3
comparable length segments
Nodes PN2
PN3
2-3 connection
short length segments
(3) Score propagation: based on a shortest path algorithm, a
global score is normalized from individual score of edge/node
46
Symbol recognition & spotting
“Graph scoring for symbol spotting (2)”
People
Rashid Qureshi
Funding HEC scholarship
(4) Results &
performance evaluation
1
Starting 2005
R. Qureshi and al. Spotting
Symbols in Line Drawing Images
Using Graph Representations.
GREC, 2008.
0
SESYD dataset
Precision
Ref
Recall
47
Symbol recognition & spotting
“Bayesian based system for symbol spotting (1)”
People
Muzzamil Luqman
Funding HEC scholarship
Starting 2008
Ref
M.M. Luqman. A Content Spotting
System for Line Drawing Graphic
Document Images. ICPR, 2010.
(1) Representation phase: used the
graph based representation of JeanYves Ramel
(2) Description phase: approach based on
attributes (of nodes and edges)
(3) Learning and classification phases base on Bayesian network
(3.1.) Discretization step: based on the Akaike Information Criterion
(3.2.) Learning step:
- network topology is done from a genetic algorithm
- parameters conditional probabilities is done from a
maximum likelihood estimation
(3.3.) Classification step:
48
Symbol recognition & spotting
“Bayesian based system for symbol spotting (2)”
Funding HEC scholarship
(4) Performance evaluation at recognition level
ISRC 2003 dataset
Starting 2008
M.M. Luqman. A Content
Spotting System for Line
Drawing Graphic Document
Images. ICPR, 2010.
Clean
level1
level2
level3
Binary degrade
Hand
drawn
Scalability
clean
yes Yes 100% 100% 100% 100%
no no 99% 96% 93% 92%
no no 98% 94% 92% 91%
no no 91% 77% 71% 69%
no no 98% 95% 93% 92%
Hand
drawn
Binary
degrade
(5) Improvements of Rashid Qureshi’s results
SESYD dataset
Precision
Ref
Scaling
Muzzamil Luqman
Rotation
People
49
Recall
Symbol recognition & spotting
“Graph Embedding”
People
N. Sidere, J.Y. Ramel
Topic: Topological Graph Embedding
Funding Navidomass
Starting 2007
Ref
(1) A lexicon is generated from the network of non-isomorphic graphs
N. Sidère et al. Vector Representation of
Graphs : Application to the Classification
of Symbols and Letters. ICDAR 2009.
(2) The embedding is based on occurrences of the patterns
50
Symbol recognition & spotting
“International Symbol Recognition Contest 2011” (1)
Workshop
GREC 2011
Funding Support of TC10
Contest starting
March 2011
Starting
2011
Training datasets
6th of April 2011
Ref
E. Valveny and al. Report on
the Symbol Recognition and
Spotting Contest. GRECLNCS
2012.
Call of participation
2sd of May 2011
Final datasets
25th of July
People
M. Delalandre, R. Raveaux
Recognition
Tests
Localization
Tests
Contest slot
25th July - 1st August 2011
http://iapr-tc10.univ-lr.fr/index.php/symbol-contest-2011
id
domain
models symbols
distortion
Training
#1-#7
Technical
36-150
16650
Rotation, Scaling,
Kanungo, Context
Final
#1-#4
Technical
36-150
16800
Rotation, Scaling,
Kanungo, Context
id
domain
models
Training
#8-#15
Electrical
Architectural
16-21
40
835
None,
Kanungo
Final
#5-#12
Electrical
Architectural
16-21
120
3463
None,
Kanungo
images symbols distortion
Symbol recognition & spotting
“International Symbol Recognition Contest 2011” (2)
People
The participant: Nayef, N. & Breuel, T. On the Use of Geometric
Matching for Both: Isolated Symbol Recognition and Symbol Spotting
Workshop on Graphics Recognition (GREC), 2011
M. Delalandre, R. Raveaux
Funding Support of TC10
Starting
2011
Ref
E. Valveny and al. Report on
the Symbol Recognition and
Spotting Contest. GRECLNCS
2012.
set name models
Connected
components
filtering
noise
Contour
detection and
sampling
Geometric
matching
recognition rate
groundtruth A ( Prel )
Recognition
Tests
Localization
Tests
Final #1
50
Kanungo
94.76%
Final #3
150
Kanungo
85.88%
Final #4
36
Context
96.22%
set name
domain
models
noise
precision / recall
Final #5
Architectural
16
None
0.62 / 0.99
Final #6
Architectural
16
Kanungo
0.64 / 0.98
Final #9
Electrical
21
None
0.37 / 0.56
Final #10
Electrical
21
Kanungo
0.44 / 0.63
intersection
P 
results A ( Pret )
A ( Pret  Prel )
A ( Pret  Prel )
A ( Pret )
R 
A ( Pret  Prel )
A ( Prel )
52
Symbol recognition & spotting
“Performance characterization of symbol localization (1)”
People
Mathieu Delalandre
Open problem
Funding Na
Starting 2008
M. Delalandre and al. A
Performance Characterization
Algorithm for Symbol
Localization. GREC, 2010.
The characterization method must do some rejection, ways to solve are...
1. To define and apply manual threshold (bad ...)
2. To propose a method for adaptative tthresholding, how to do ?
Key idea, characterization method based on context
Groundtruth, gravity
centers, contours
p2
p1
Result points
detection rate
Ref
how to make the difference
between segmentation errors of
background with segmentation
errors of objects
Highest probabilities
p3
Lowest probabilities
probability error
53
Symbol recognition & spotting
“Performance characterization of symbol localization (2)”
Mathieu Delalandre
(1) The method
Funding Na
Starting 2008
M. Delalandre and al. A
Performance Characterization
Algorithm for Symbol
Localization. GREC, 2010.
3.1 Localization
comparison
(1.2) Probability scores are computed from a groundtruth
point gi and the result point r, considering the neighboring
groundtruth point gj. Final result is computed considering
all the groundtruth using a probability score function
3.2 Probability
scores
3.3 Matching
algorithm
3.4 Transform
function
1
g
si
ε
0
score error (ε) 1
  1 p
single (Ts)
1
detection rates
(1.4) Transform function make results context independent,
making difference with self-matching of groundtruth, to
achieve coherent comparison of methods on different datasets
0
Results
(1.1) Localization comparison moves results
from Euclidean to a scale space, to deal with the
scale invariance
(1.3) Matching algorithm looks for statistical distribution of
single, miss, merge and split cases, in an decreasing order of
precision using a bipartite list.
single detections Ts
Ref
Groundtruth
multiple (Tm)
alarm (Tf)
ε
detection rate
People
0
0
probability error
score error
  1 p
54
1
Symbol recognition & spotting
“Performance characterization of symbol localization (3)”
0.60
Mathieu Delalandre
Funding Na
floorplans i(1) = 0.496
0.40
Starting 2008
i(ε)
M. Delalandre and al. A
Performance Characterization
Algorithm for Symbol
Localization. GREC, 2010.
0.20
0.00
0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90
SESYD dataset
Drawing level
Symbol level
1,00
-0.20
floorplans
score error (ε)
diagrams
Ref
electrical diagrams i(1) = 0.529
(2) Results obtained from the
Rashid Qureshi’s system
  1 p
Setting
backgrounds
Dataset
images
Setting
backgrounds
Dataset
images
5
models
16
100
symbols
2521
5
models
17
100
symbols
1340
0.50
0.40
floorplans
i(ε)
People
0.30
electrical diagrams
0.20
0.10
0.00
0.00
0.10
score error (ε)
  1 p
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
56
Content based Image Retrieval
“Robust key points detection for document image retrieval”
People
The Anh Pham
Funding VEID scholarship
Starting 10/2010
Ref
Na
The work focuses on robustness-keypoint detection, robustness in DIA
including: 2D and illumination, noises, artifacts, blurred.
Keypoints detection takes part of the general features extraction,
some typical features used as key points and their characteristics are:
Level of detail
(primitive)
Robustness
Regions
Like-blob points
Edges, contours
Corners, salient points
Patterns
Semantic
So, some ideas of new robust keypoint detection may be:
- Detecting robust features first, then extracting salient points, or
- Using robust methods (i.e Machine learning, Modelbased, Parametric-based) to extract salient points, or
- Combination of above methods
57
Content based Image Retrieval
“Logo recognition and spotting”
People
Mathieu Delalandre
Funding DOD project
Starting 04/2012
Ref
Problematic
- document classification can be supported by logo recognition
- a meta engine will manage the decision rules
- spotting will depend of a previous stage of page segmentation
Na
Page
segmentation
Document
classification
Logo spotting
Logo
recognition
Meta engine
- binary images at 300 dpi
- time constraint:  to 1,5 s per image
High variability and
scalability of logos
Logo are rich graphical parts, 300
dpi and binarization could result
in a high level of degradation
Some logo recognition case looks
like OCR only
58
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
6.7. Graph matching and embedding
59
Camera based recognition
“Robust OCR for video text recognition”
People
Thierry brouard
Funding SNECMA project
Starting 2008
Ref
International patent
Problematic
-Automatic routing of input letters by digitization and
OCR of input documents received from customers
-Response time < 3 s, Recognition rate > 80%, Precision
Equal to 100%, Java environment on mutualized servers
Approach based on cognitive vision and knowledge based systems
(blackboard & mathematical theory of evidence), to achieve robust
segmentation & OCR
60
Camera based recognition
“Real time logos recognition in urban environment”
People
Mathieu Delalandre
Funding Na (JSPS grant)
Starting 2010
Ref
Na
video
a set of images
Problematic
- Logo detection from video capture using some
handled interactions, to display context based
information (tourist check points, bus
stop, meal, etc.).
- Hard points are the real time constraints and
the complexity of the recognition task.
First goal of the project is to support the real time recognition. We start from the
hypothesis than logo appear in a static way in video. We propose to achieve an
automatic control/selection of image capture to reduce the amount of data to
process.
frequency, resolution
parameters
processing
charge
Workflow
management
size
frame
Video
capture
Frame
stack
stored
frame
Frame
selection
tag/remove
motion data
Detection/
Recognition
3D
Mapping
overlapping
Gyro
access
61
Talk workplan
1.
2.
3.
4.
5.
6.
Tours city
François-Rabelais University, les deux lions / Portalis
School of Engineering Polytech’Tours
Laboratory of Computer Science
RFAI group
DIA related work
6.1. Projects & partners outline
6.2. Layout analysis and document recognition
6.3. OCR, word spotting and signature verification
6.4. Symbol recognition & spotting
6.5. Content Based Image Retrieval
6.6. Camera based recognition
62