Using Mobile Phones To
Write In Air
Chris Coykendall – ODU CS495
Introduction


Researchers from Duke University and University
of Illinois collaborated to develop a prototype
system called PhonePoint Pen on the Nokia N95
platform
The PhonePoint Pen system aims to allow
humans to write messages or diagrams in the air
by using a phone as a pen.
The Nokia N95 device
Source: Engadget
Some Motivations




Make “post-it” notes on the fly
Assistive communications for disabled persons
and/or the elderly
Sketching complex ideas, equations and nonstandard information
Emergency operations and first responders
So Why A New System?



User experience on existing mobile technologies
leaves much to be desired
Mobile keyboards difficult to use for motor
impaired individuals
Voice recording technologies are cumbersome to
search through (and speech-to-text software can
be hit or miss)
Primary Goals



To explore the viability of using mobile phone
accelerometers to write in the air.
To understand and overcome limitations with
character recognition.
To develop a prototype on the Nokia N95 platform
and perform a test study.
Challenges

Lack of a gyroscope


Many systems such as the Wii, Kinect and others are
more resourceful in hardware and have a gyroscope
for filtering rotation. (This study was done shortly
before many phones with gyroscopes, such as the
iPhone 4, were released.)
In this study, users were constrained to using the
phone by holding it in a steady, non-rotating grip like a
pen. They were also asked to write in slow, deliberate
motions and make each character around 12 inches
square.
Challenges

Suppressing background vibration “noise”


Jitter is caused by natural hand vibrations and the
accelerometer error itself.
The researchers approached this issue by
implementing a moving ∆velocity over last 7 readings.
Any acceleration samples < .05m/s2 were ignored.
Challenges

Computing displacement of device


The physical position in the air is necessary to know
the size of the characters and their relative positions.
This was overcome by looking at the velocities
measured suppressing the background vibrations. If n
readings were measured as all background noise, it is
likely a pause and the velocity is reset to zero.
(From source article)
Challenges

Multi-stroke characters and transitions



Letters such as the letter ‘A’ require two distinct strokes
to write. The letter ‘B’ looks remarkably similar to
writing ‘13’. With no frame of reference for position, this
is difficult to accommodate.
The study exploited user natural motion of “picking” up
the pen on the Z-axis.
They also relied on a combination of heuristics such as
delimiters and grammar algorithms they developed to
predict what the next stroke was intended to be.
(From source article)
Implementation



Prototypedon Nokia N95 phone platform.
A server-side implementation was developed in
MATLAB, which they were able to use basic
libraries for the signal processing.
The on-phone processing version was
implemented in Python, but stripped-down to only
one character at a time and simpler signal
processing methods.
User Evaluation



Tests were conducted with 10 average users to write the
English alphabet: mainly CS and Engineering students.
4 subjects were trained on the system, while the other 6
“novice” subjects had writing less than ten characters.
Another 5 individuals took part in 8-character tests at Duke
University Hospital under supervision. These patients
either suffered from cognitive disorders or motion
impairments.
Result Highlights


Human readability accuracy (HRA) of the characters was
around 83-85% for the student test subjects. Character
recognition accuracy (CRA) was 91.9% for trained users
and 78.2% for novices
The hospital patient 8-character HRA test results were:
User 1 – 1/8
User 2 – 1/8
User 3 – 1/8
User 4 – 5/8
User 5 – 0/8 (could not operate button)
(From source article)
Patient Barriers




The PhonePoint Pen required a button to be
pressed to begin and end the writing.
Shoulder, elbow, and wrist coordination for large
12in characters can be difficult.
Familiarity with mobile devices.
IRB restrictions make it difficult to exhaustively test
patient use.
Improvements To Be Made






Faster writing (only 3.02s per letter on average)
Writing longer words/drawings
Cursive handwriting
Writing while moving
More diverse test subjects (CS/Enginering majors
likely more technologically-inclined.)
More advanced algorithms (Bayesian Networks,
Hdden Markov Models, etc.)
Conclusions and Related
Research






PhonePoint Pen compares well to other research
in this area considering its limited processing
power and sensor hardware (just the
accelerometer)
Air-gestures with 3D accelerometers (uWave)
Vision-based gestures (Microsoft write-in-the-air)
Stylus-based sketch recognition (SketchREAD)
Wiimote, Logitech Air-Mouse, Nokia NiiMe
Smart Pen and SmartQuill
Source

S. Agarwal et. al., "Using Mobile Phones to Write
in Air", MobiSys'11