Pen-Centric
Shorthand Interfaces
Charles C. Tappert
Seidenberg School of CSIS, Pace University
1
Themes of Presentation



Online Handwriting Recognition and Pen
Computing Tutorial
Historical Research – undertaken for the
Palm-Xerox Patent Infringement Lawsuit
Recent Research - Enhanced Pen-Centric
Shorthand Interfaces can have benefits

DPS dissertation could extend M.S. thesis
2
Enhanced Pen-Centric
Shorthand Interfaces




Can use word/phrase shorthand to speed
text input
Can provide critical infrastructure for many
pen-centric applications
Can enhance natural pen-centric
interactions for many applications
Will have greatest impact on the utility of
applications running on small mobile devices
3
Part 1: Online (Pen-Centric)
Handwriting Recognition








Written Languages and Handwriting Properties
The Fundamental Property of Writing
Handwriting Recognition Difficulties
Online (Pen-Centric) Handwriting Recognition
Online more accurate than Offline Recognition
Online Info Can Complicate Recognition Process
Design Tradeoffs / Design Decisions
Computer Problems in English
4
Written Language
and Handwriting Properties

Alphabet


Letters, digits, punctuation, special symbols
Writing is a time sequence of strokes



Stroke – writing from pen down to pen up
Usually complete one character before
beginning the next
Spatial order – e.g., in English left to right
5
Fundamental Property of Writing



Differences between different characters
are more significant than differences
between different drawings of the same
character
This makes handwritten communication
possible
Can there be exceptions – say, different
characters written identically?
6
Fundamental Property of Writing
in English



Property holds within subalphabets of uppercase,
lowercase, and digits, but not across them
“I”, “l”, and “1” written with single vertical stroke
“O” and “0” written similarly with an oval
7
Handwriting Recognition Difficulties



Shape, size, and slant variation
Similarly shaped characters – U and V
Careless writing


in the extreme, almost illegible writing
Resolving difficult ambiguities requires
sophisticated recognition algorithms,
syntax/semantics
8
Online (Pen-Centric)
Handwriting Recognition

Electronic tablets invented in late 1950s


Digitizer and display in separate surfaces
Pen Computers arrived in 1980s




Combined digitizer and display
Brought input and output into one surface
Immediate feedback via electronic ink
Created paper-like interface
9
Tablet Digitizers – Dynamic Information


Pen down – indication of inking
X-Y coordinates as function of time


Sampling rate: 100 points per second
Resolution: 200 points per inch
10
Early Pen-Centric Interface


Different surfaces for
input and output
Rand system, about 1959
11
Pen Computers

IBM vision


Paper-like interface, 1992
Microsoft Tablet PC

Launched, 2001
12
Pen-Centric PDAs


Early Palm Pilot
Palm Tungsten T3 and
Sony Clié TH55
13
Online (Pen-Centric)
Handwriting Recognition


Machine recognizes the writing as the user writes
Digitizer equipment captures the dynamic
information of the writing


Stroke number, order, direction, speed
A stroke is the writing from pen down to pen up
14
Online (Pen-Centric) more accurate
than Offline (Static) Recognition


Can use both dynamic and static information
Can often distinguish between similarly shaped
characters
 E.g., 5 versus S where the 5 is usually written
with two strokes and the S with one stroke
15
Online Information Can Complicate
Recognition Process

Large number of possible variations


E can be written with one, two, three, or four
strokes, and with various stroke orders and directions
A four-stroke E has 384 variations (4! stroke orders x
24 stroke directions)
16
Online Information Can Complicate
Recognition Process

Other variations
17
Online Information Can Complicate
Recognition Process

Segmentation ambiguities


character-within-character problem
lowercase d might be recognized as a cl if drawn
with two strokes that are somewhat separated
from one another
18
Design Tradeoffs/Decisions

No constraints on the user


Machine recognizes user's normal writing
User severely constrained


Must write in particular style such as handprint
Must write strokes in particular order,
direction, and graphical specification
19
English Writing Styles

Handprint



Uppercase – about 2 strokes per letter
Lowercase- about 1 stroke per letter
Cursive Script


Usually less than 1 stroke per letter
Delayed crossing and dotting strokes
20
Computer Problems in English

Constrained Handprint



Unconstrained Handprint



Printing one symbol per box – form filling
Printing on lines – symbols can touch or
overlap
No lines and symbols can touch or overlap
Cursive Script
Mixed Printing and Cursive
21
Computer Problems in English
22
Pencept Commercial Product 1980s
23
Part 2
Shorthand in Pen-Centric PDAs





Famous Uses of Shorthand
Historical Shorthand Alphabets
Pen-Centric Shorthand Alphabets
Pen-Centric Word/Phrase Shorthand
Allegro/Chatroom Shorthand System

M.S. thesis that could be extended into a
DPS dissertation
24
Background

Famous writings throughout history were
effectively written in a style of shorthand





Cicero’s orations
Martin Luther’s sermons
Shakespeare’s and George Bernard Shaw’s
plays
Samuel Pepys’ diary
Sir Isaac Newton’s notebooks
25
Historical Shorthand Alphabets



We first review the history of shorthand
systems prior to pen computing
Shorthand is “a method of writing rapidly
by substituting characters, abbreviations,
or symbols for letters, words, or
phrases”
Shorthand can be traced back to the
Greeks in 400 B.C.
26
Historical Shorthand Alphabets


We focus on shorthand alphabets that
might be appropriate for PDAs
We review two types of shorthand

Geometric shorthand



Small number of basic shapes
Shapes reused in multiple orientations
Non-geometric shorthand shorthand
27
Historical Shorthand Alphabets





Ancient Greeks – 400 BC
Tironian Alphabet – 63 BC
John Willis’s Stenography – 1602
Gabelsberger Alphabet – 1834
Moon Alphabet – 1845
28
Tironian Alphabet, 63 B.C.
Non-Geometric
29
Stenography Alphabet, 1602
30
Stenography Alphabet, 1602

Basic Shapes and Orientations
31
Gabelsberger Cursive-Style, 1834
Non-Geometric Alphabet
32
Moon Geometric Alphabet, 1845
33
Other Historical Shorthand Systems

Phonetic alphabets



Pitman (1837), was popular in UK
Gregg (1888), was popular in USA
Systems for the blind

Braille (1821)
34
Pen-Centric Shorthand Alphabets

Some of the earliest were for CAD/CAM



symbols represent graphical items and
commands
Others developed for text input on small
consumer devices like PDAs that have
limited computing power
We review geometric and non-geometric
shorthands appropriate for small devices
35
Pen-Centric Shorthand Alphabets

Historical alphabets presented above could
be used for machine recognition
symbols drawn with a single stroke
(except “K” in Tironian and “+” in Stenography)


In addition to shape and orientation,
online systems can use stroke direction to
differentiate among symbols
36
Pen-Centric Shorthand Alphabets

Geometric Pen-Centric Shorthands




Organek – 1991
Allen – filed 1991, patent 1993
Goldberg (Xerox) – filed 1993, patent 1997
Non-Geometric Pen-Centric Shorthands


Graffiti (Palm Computing) – 1995
Allegro (Papyrus) – 1995
37
Organek Alphabet, 1991
38
Organic Alphabet, 1991
Basic Shapes and Orientations
One shape in 4 orientations.
This gives 8 directions that
together with 3 lengths
provide 24 symbols.
A second wheel provides
additional symbols.
39
Allen patent, filed 1991
40
Allen patent, filed 1991
Basic Shapes and Orientations
41
Goldberg patent, filed 1993
(“unistroke symbols”)
42
Goldberg patent, filed 1993
Basic Shapes and Orientations
43
Goldberg patent, filed 1993

5 Basic shapes
4 Orientations
2 Stroked Directions
40 Possible Symbols
Designed for Speed of Input and
Maximum Symbol Separation
44
Shorthand Alphabet Design


How would you design a shorthand
alphabet?
What would be the design criteria?
45
Design of Graffiti Alphabet
for the Palm Pilot

Small alphabet



Uppercase, digits, special symbols
One stroke per symbol to avoid
segmentation difficulty
Separate writing areas for letters and
digits to avoid same-shape confusions
46
Graffiti Alphabet, 1995
47
Graffiti Mimics Keyboard Input


Character by character input
Mode shifts for



Uppercase
Special characters
Eyes can focus on application’s insertion
point rather than on input area
48
Graffiti Alphabet Design

What was the additional design criterion?
49
Graffiti Alphabet Design

Designed for ease of learning


20 letters exactly match the Roman
alphabet
6 remaining ones match partially
50
Graffiti Alphabet: 11 of 26 characters
have alternate variations
51
Frequently Confused Characters
52
Other Low Performance Characters
53
Symbol Overlap Comparison
54
Graffiti Recognition Accuracy Study
55
Allegro Alphabet (Papyrus), 1995
(now Microsoft)
56
Simplified Design Tradeoffs/Decisions
for Graffiti and Allegro PDA Alphabets

Small alphabet


One stroke per character (character = stroke)


preferably only one
Separate writing areas for letters and digits


allows machine to recognize each character upon pen lift
Small number of writing variations per letter


one case rather than both upper and lowercase
avoids confusion of similarly shaped letters and digits
High correspondence to Roman alphabet for ease
of learning

non-geometric, might not actually qualify as shorthand
57
Commercially Successful Shorthands

Similar to the Roman alphabet

Easy to learn
Graffiti used in Palm OS devices
 notably the Palm Pilot & Handspring models

Allegro used in Microsoft Windows devices


Geometric alphabets not successful
58
Current Commercial Systems
Company/System
Writing Style
Palm Computing/Graffiti*
Special Shorthand Alphabet
Microsoft/Papyrus Allegro
Special Shorthand Alphabet
CIC/Jot
Relatively Unconstrained Handprint
Microsoft
Relatively Unconstrained Handprint
and Cursive
*A few years ago Palm switched from Graffiti to Graffiti2,
Graffiti2 is basically Jot licensed from CIC.
59
Jeff Hawkins, innovator












1979 BSEE Cornell, 1979-1986 worked at Intel and GRiD
1986-1987 ABD BioPhysics doctoral program, U.C. Berkeley
1987- back at GRiD he created GRiDPAD, first pen computer
1992 formed Palm Computing, 1993 created first handwriting reco software
product for a mobile handheld - Casio’s Zoomer
1995 Palm Computing bought by U.S. Robotics
1996 created PalmPilot, first PDA with Graffiti shorthand alphabet (over a
million shipped in 18 months, a 66% market share, and the fastest growth
of any computing product in history, faster than the TV and the VCR)
1997 U.S. Robotics bought by 3Com (sued by Xerox for patent infringement)
1998 left Palm to form Handspring, 1999 launched the Visor handheld
2000 Palm Computing spun off by 3Com
2002 created what is now the Redwood Center for Theoretical Neuroscience
2003 Handspring (with Hawkins, et al.) acquired by Palm Computing
2005 Founded Numenta to build the ultimate brain-like machine
60
Palm-Xerox
Patent Infringement Lawsuit



The nine-year old battle between Palm and Xerox over
handwriting recognition ends in 2006, see article.
Palm pays Xerox $22.5 million for a fully paid-up license
for Xerox patents covering its text input Unistrokes
technology
Xerox first sued Palm predecessor Palm Computing back
in April 1997, claiming that the Graffiti text-entry system
used in its PDAs infringed on patents for Unistrokes,
which allows users to input letters and numbers into
personal data units with basic, one stroke movements.
61
Conclusions
Palm-Xerox Patent Infringement Lawsuit

Invalidity



Infringement


Historical research showed that Goldberg alphabet
not so unique
Even though the patent was accepted as valid,
these arguments narrowed the scope of the patent
Analyses and comparisons of the Goldberg and
Graffiti alphabets showed major differences
Result was favorable settlement for Palm
62
Pen-Centric Word/Phrase Shorthand
such as Chatroom Shorthand


Further increase speed of text entry
Potential applications
Where input speed important
 Where word/phrase abbreviations
occur frequently – e.g., email

63
Chatroom Shorthand Examples
CU
CM
@TEOTD
^5
2nite
LOL
ASAP
B/C or BC
See you, or Cracking up
Call me
At the end of the day
High five
Tonight
Laughing out loud
As soon as possible
Because
64
Allegro/Chatroom Shorthand System

Developed for M.S. dissertation


Student was hearing impaired
Developed as output component of
communication system


Handwriting to text to speech
Two input writing areas


One for Allegro (all-purpose)
One for chatroom-like or user-defined words/phrases
65
Allegro/Chatroom Shorthand System
Stroke acquisition GUI
a single stroke
is it
word/phrase
character
allegro stroke
library
allegro stroke
recognition
other stroke
recognition
alphabet
user-defined
stroke library
meaning
sentence accumulator
done?
no
yes
Sentence display
and spoken output
66
Allegro/Chatroom Shorthand System
67
Allegro/Chatroom Shorthand System
M.S. Thesis Experimental Results

Allegro/Chatroom pen-centric
shorthand input is faster than typing
text and is comparable to typing text
and chatroom shorthand characters
68
Conclusions
Pen-Centric Shorthands


Pen-centric interfaces should use shorthand,
and especially word/phrase shorthand where
appropriate, for fast text input
Benefit of shorthand interfaces



Provides critical infrastructure for many pen-centric
applications
Enhances natural pen-centric interactions for many
applications
Has greatest impact on the utility of applications
running on small mobile devices
69
Conclusions
Handwriting Recognition


Graffiti and Allegro greatly simplified the
recognition problem
Handprint problem not completely solved


Even with IBM’s ThinkWrite, CIC’s Jot, and
Microsoft products
Cursive script problem clearly not solved
70
References





W.B. Huber, S.-H. Cha, C.C. Tappert, and V.L. Hanson, "Use of
Chatroom Abbreviations and Shorthand Symbols in Pen Computing,"
Proc. 9th Int. Workshop on Frontiers in Handwriting Recognition,
IWFHR 2004, Tokyo, Japan, October 2004.
W. Huber, V. Hanson, S. Cha, and C.C. Tappert, "Common Chatroom
Abbreviations Speed Pen Computing," Proc. 11th Int. Conf. HumanComputer Interaction, Las Vegas, NV, July 2005.
C.C. Tappert and S. Cha, "Handwriting Recognition Interfaces," Chapter
6, pp. 123-137, in Text Entry Systems, Scott MacKenzie and Kumiko
Tanaka-Ishii (Eds.), Morgan Kaufmann, 2007.
C.C. Tappert, C.Y. Suen, and T. Wakahara, "The state-of-the-art in online handwriting recognition," IEEE Trans. Pattern Analysis Machine
Intelligence, Vol. PAMI-12, pp. 787-808, August 1990.
C.C. Tappert and J.R. Ward, "Pen-Centric Shorthand Handwriting
Recognition Interfaces," Proc. 1st Int. Workshop on Pen-Based
Learning Technologies, Catania, Italy, May 2007.
71