Handwritten Word Recognition:
A New CAPTCHA Challenge
Amalia Rusu and Venu Govindaraju
CEDAR
University at Buffalo
CAPTCHA
 Completely Automatic Public Turing test to tell Computers and Humans
Apart
 An automated test that humans can pass but current computer programs
fail – beyond the state-of-the-art
 Exploits the difference in abilities between humans and machines
(i.e. text, speech or facial features recognition)
 A new formulation of the Alan Turing’s test - “Can machines think?”
Objective
Please enter the handwritten word as it is shown below:
If you cannot read this image click here
Example of interface and handwritten CAPTCHA to confirm registration.
User Authentication Steps using HCAPTCHA
The user initiate the
dialog and has to be
authenticated by server
Authentication Server
User
Challenge
Response
Internet
User authentication
Automatic Authentication Session for Web Services.
i.
ii.
iii.
iv.
Initialization
Handwritten CAPTCHA Challenge
User Response
Verification
Desirable Properties




CAPTCHA should be automatically generated and graded
Test can be taken quickly and easily by human users
Test will accept virtually all human users and reject software agents
Test will resist automatic attack for many years despite the
technology advances and prior knowledge of algorithms
Previous Work







First CAPTCHA designed in 1997 (for AltaVista website URL filter)
CMU
Gimpy, EZ-Gimpy, Gimpy-R, Bongo, Pix, Eco
PARC
BaffleText
UCB & PARC
PessimalPrint
Microsoft
ARTiFACIAL
Bell Labs
Reverse Turing test using speech
GIT
Character morphing
CAPTCHA Tests
AltaVista URL filter uses isolated
random characters and digits on a
cluttered background.
BaffleText uses pronounceable character strings that are not
in the English dictionary and render the character string
using a font into an image (without physics-based
degradations); then generate a mask image as shown above.
PessimalPrint uses a degradation
model simulating physical defects
caused by copying and scanning of
printed text.
CAPTCHA Tests
EZ-Gimpy uses real English words.
Gimpy
Type 3 different English words
appearing in the picture above.
Gimpy-R uses nonsense words.
Character morphing algorithm that transforms a string
into its graphical form.
Why Handwritten CAPTCHA?
 No handwritten text based CAPTCHA exists - so far!!!
 Several machine printed text based CAPTCHA already broken
 Greg Mori and Jitendra Malik of the UCB have written a program that can solve
Ez-Gimpy with accuracy 83%
 Thayananthan, Stenger, Torr, and Cipolla of the Cambridge vision group have
written a program that can achieve 93% correct recognition rate against EzGimpy
 Gabriel Moy, Nathan Jones, Curt Harkless, and Randy Potter of Areté
Associates have written a program that can achieve 78% accuracy against
Gimpy-R




Machine recognition of handwriting is more difficult than printed text
Handwriting recognition is a task that humans perform easily and reliably
Research is in the early stages - a promising field
Handwritten CAPTCHAs will challenge the KBCS community!
State-of-the-art
Lexicon Lexicon Driven
size
Grapheme Model
time
(secs)
accuracy
time
(secs)
accuracy
Top 1
Top 2
Top 1
Top 2
10
0.027
96.53
98.73
0.021
96.56
98.77
100
0.044
89.22
94.13
0.031
89.12
94.06
1000
0.144
75.38
86.29
0.089
75.38
86.29
20000
1.827
58.14
66.56
0.994
58.14
66.49
Speed and accuracy of a HR. Feature extraction time is excluded. Testing platform is an Ultra-SPARC.
Source of Errors for HW Recognizers
 Image quality
Background noise, printing surface, writing styles
 Image features
Variable stroke width, slope, rotations, stretching, compressing
 Segmentation errors
Over-segmentation, merging, fragmentation, ligatures, scrawls
 Recognition errors
Confusion with similar lexicon entries, large lexicons
Creating H-CAPTCHAS
 Use handwritten word images that current recognizers cannot read
 Controlled “distortion” of existing handwritten word images
 Create handwritten images by concatenating handwritten character
images
 Use handwritten US city name images (4,000 from CEDAR CDROM)
 Character images were discretely printed to begin with
 Character images are automatically segmented out of handwritten word
images
 Use set of 20,000 handwritten character images (extracted by program)
 Synthesize sentence images by gluing together isolated upper and lower
case handwritten characters or word images
H-CAPTCHA Generation Algorithm
Input.
 Original (random) handwritten image (existing US city name image or
synthetic word image with length 5 to 8 characters or meaningful sentence).
 Lexicon containing the image’s truth word.
Output.
 H-CAPTCHA image.
Method.
 Randomly choose a number of transformations
 Randomly establish the transformations corresponding to the given number
from: add lines, circles, grids, arcs, background noise (multiplicative or
impulse), random convolution masks, blur, wave, spread, median filters,
thick or thin characters on vertical or horizontal fashion, etc.
 A priori order is assigned to each transformation based on experimental
results. Sort the list of chosen transformations based on their priority order
and apply them in sequence, so that the effect is cumulative.
Handwritten text images
Examples of handwritten characters used to generate random words.
Examples of handwritten US city name images used as a base for
transformations.
Examples of synthetic handwritten sentence images.
H-CAPTCHA by Image Quality Transforms
Add lines, grids, arcs, background noise, convolution masks and special
filters
H-CAPTCHA by Image Features Transforms
Variable stroke
width, slope,
rotations, stretching,
compressing
H-CAPTCHA by Segmentation Transform
Delete ligatures, use
touching letters/digits,
merge characters for
over segmentation or to
be unable to segment
H-CAPTCHA by Lexicon Transform
Truth
Orlando
Lexicon challenges: size,
density, availability
Lackawanna
Clarence
Buffalo
WMR
results
(Top choice
first)
ovlando
ovlavdo
onlando
orlanolo
orlaudo
oviando
orlahdo
arlando
orlando
ovlanao
lackaevana
lackawawa
lackawaua
lackowana
lackawana
lackawanna
lackawarna
lackawanra
lackamama
lactawana
clarlncl
clarlnce
clarencl
cearence
clarence
cbarence
clorence
clahence
aarence
clawce
buffaio
buffalo
butfalo
buifalo
buffrio
ruffalo
bulfalo
bufialo
buefaio
bullalo
Accuscript
results
(Top choice
first)
ollando
ovlando
orlanolo
orlando
ovlanao
ovlavdo
onlando
oviando
orlanda
arlando
lackawarna
lactawana
lackawarra
lackawawa
lackawana
lackawaua
lackawanna
lackowana
locrawara
lackawanra
claience
clarence
clatence
clarlnce
cearence
clavence
clarenxe
clasence
clorence
claiexce
ruffalo
buffalo
buffrlo
buffaio
buffrio
bulfalo
buifalo
butfalo
buefalo
bufialo
Image
H-CAPTCHA Evaluation
 No risk of image repetition
 Image generation completely automated: words, images and distortions
chosen at random
 The transformed images cannot be easily normalized or rendered
noise free by present computer programs, although original
images must be public knowledge
 Deformed images do not pose problems to humans
 Human subjects succeeded on our test images
 Test against state-of-the-art: WMR, Accuscript
 CAPTCHAs unbroken by CEDAR recognizers
H-CAPTCHAs
Handwritten US city name images that defeat both WMR and Accuscript recognizers.
H-CAPTCHA Challenge
Word Recognizers
Number of
Recognized Images
Accuracy
WMR
383
9.28%
Accuscript
182
4.41%
Low accuracy of handwriting recognizers. The lexicons are created so as to contain all the
truths of test images. Total number of tested images is 4,127 (and so is the lexicon size)
Number of
Students
Number of
Test Images
Humans
Accuracy
WMR
Accuracy
Accuscript
Accuracy
12
15
82%
0%
0%
Low accuracy of handwriting recognizers vs. humans on a subset of test images.
CAPTCHA using Gestalt Psychology
 Gestalt psychology is based on the observation that we often experience things
that are not a part of our simple sensations
 What we are seeing is an effect of the whole event, not contained in the sum of
the parts (holistic approach)
OXXXXXX
XOXXXXX
 Organizing principles - Gestalt laws:
XXOXXXX







law of closure
law of similarity
law of proximity
law of symmetry
law of continuity
law of familiarity
figure and ground
 Not restricted to perception
 memory
[
][
][
]
**********
**********
**********
XXXOXXX
XXXXOXX
XXXXXOX
XXXXXXO
H-CAPTCHA based on Gestalt Laws
Gestalt laws: law of proximity, symmetry, familiarity, continuity
Methods: create horizontal or vertical overlaps - for same words smaller distance overlaps
- for different words bigger distance overlaps
H-CAPTCHA based on Gestalt Laws
Gestalt laws: law of closure, proximity, continuity
Methods: create occlusions by circles, rectangles, lines with random angles
H-CAPTCHA based on gestalt laws
Gestalt laws: law of closure, proximity, continuity
Methods: add occlusions by waves from left to right on entire image, with
various amplitudes / wavelength or rotate them by an angle
H-CAPTCHA based on Gestalt Laws
Gestalt laws: law of closure, proximity, continuity, background
Methods: use empty letters, broken letters, edgy contour, fragmentation
H-CAPTCHA based on Gestalt Laws
Gestalt laws: memory, internal metrics, familiarity of letters
vertical mirror – difficult for
humans
horizontal mirror – difficult for
humans
flip-flop –OK for humans!!
Methods: change word orientation entirely, or the orientation for few letters only
Gestalt H-CAPTCHA Results
Word
Recognizers
Horizontal
Overlap
(Small)
Horizontal
Overlap
(Large)
Vertical
Overlap
Occlusion
by waves
Occlusion
by circles
Empty
Letters
Less
Fragmentation
More
Fragmentation
Old
Transforms
WMR
24.35%
12.93%
27.88%
15.43%
35.93%
0.89%
0%
0.48%
9.28%
Accuscript
2.93%
2.42%
12.64%
10.56%
32.34%
0.06%
0.18%
0%
4.41%
Tested images is 4,127 for each type of transformation.
Future Work
Personalizing Email Addresses
Original Email
Address
Apply Image
Transformation
Transformed Email
Address

Creates transformed alias e-mail addresses to prevent mining by software agents
Future Work
Adult vs. Child vs. Machine

Few methods to differentiate between
adult vs. child
o
o
o
o

Asking a question that has the answer in
the handwritten sentence
Giving an incomplete handwritten
sentence and asking to imply the missing
word
Comparing the handwritten text with a
standard word list
Using
longer,
more
complicated
handwritten sentences, using advanced
topics from technical fields such as math,
physics, or financial
Useful on Internet services due to
expansion of harmful minor websites
Reading abilities delimitation:
Machine vs. 1st grade child
Adult vs. 7th grade child
Future Work
 HCAPTCHA based on Handwritten Sentence Reading and Understanding
 Incorporate and adjust the image complexity factor as a parameter of error
 Try out more image transformations and compare results against humans





performance
Cognitive aspects of HCAPTCHA for adult vs. child protocol
HCAPTCHA as a Challenge Response Protocol for Security Systems
Online-Handwriting CAPTCHA
HCAPTCHA as a Biometric?
HCAPTCHA normalization concerns based on future technology development
Thank You
Power of Context
Context
Ranked
Lexicon