Using Mobile Phones To Write In Air
Sandip Agrawal and Romit Roy Choudhury
Dept. of Electrical and Computer Engineering, Duke University
Ionut Constandache
Dept. of Computer Science, Duke University
Shravan Gaonkar
Dept. of Computer Science, University of Illnois
Kevin Caves and Frank DeRuyter
Speech Pathology & Audiology, Duke Medical School
Proceeding MobiSys '11
Proceedings of the 9th international conference on
Mobile systems, applications, and services
Speaker: Yi-Ting Chen
Outline
Introduction
Use Cases
Core Challenges
System Design and Algorithms
Implementation and Evaluation
Limitation
Conclusion
2
A scenario
While driving to office, Leslie stops at a traffic light. She
remembers that she needs to call her friend, Jane, very soon.
She decides to make a note of this task.
She reaches for her phone from the pocket, and by holding it
like a pen, she writes “JANE” in the air. She also gestures a
checkmark to email the written note to herself.
Once in her office, she finds an email in her mailbox that
reads “PhonePoint Pen – JANE”.
3
Introduction
Typing an SMS, while popular among the youth, has been
unpopular among a moderate section of society.
Even if future keyboards improve the typing experience,
some problems may still persist.
While walking, or with one hand occupied, typing in
information may be inconvenient.
The ability to write without having to look at the phone
keypad may offer an added advantage.
4
Introduction
This paper proposes to use the in-built accelerometer in
modern mobile phones to capture (short) written information,
called PhonePoint Pen.
The PhonePoint Pen system aims to allow
humans to write messages or diagrams
in the air by using a phone as a pen.
Platform: Nokia N95
5
Use Case
Assistive Communications for Impaired Patients
One-handed Use
Equations and Sketching
Emergency Operations and First Responders
Other devices and applications
(e.g., a TV remote control could allow users to write number
to switch channel)
6
Core Challenge
1. Filtering Hand Rotation without Gyroscope
2. Coping with Background Vibration (Noise)
3. Computing Displacement of Phone
4. Differentiating an “A" from a Triangle
5. Identifying Character Transitions
7
Filtering Hand Rotation without Gyroscope
Issue:
Nokia N95 phones can measure linear movement along each
axis, but cannot detect rotation.
Proposed Approach:
(i) The simpler one is to pretend that one of the corners of
the phone is the pen tip, and to hold it in a non-rotating grip.
(ii) An accelerometer reading at this paused timepoint can
identify the components of gravity on each axes, and hence,
the angular orientation θ can be determined.
8
Filtering Hand Rotation without Gyroscope
9
Coping with Background Vibration
(Noise)
Issue: Accelerometers are sensitive to small vibrations.
10
Coping with Background Vibration
(Noise)
Proposed Approach:
First, we smooth the accelerometer readings by applying a
moving average over the last n readings (in our current
prototype, n=7).
11
Coping with Background Vibration
(Noise)
Proposed Approach:
Next, we mark as noise all acceleration values less than
0.5m/𝑠 2 .
12
Computing Displacement of Phone
Proposed Approach:
The displacement
we look at consecutive accelerometer readings marked as
noise in the previous step.
13
Differentiating an “A" from a Triangle
Issue:
While writing character “A”, the pen does not know the last
stroke “–” should be added in the center (to indicate an “A”),
or at the bottom(to indicate a triangle, ∆)
Proposed Approach:
The lifting of the pen generates an identifiable impulse on
the Z axis. Set a certain threshold, label that stroke as a
“lifting of the pen”.
It will be going off the record, not get included in the final
output, but still monitored for pen repositioning.
14
Identifying Character Transitions
Issue:
For instance, “B” and “13” may have the exact same handmovement, including the pen-lift.
Proposed Approach:
The simplest approach is to require the user to include a
special gesture between characters, like a “dot” or a
relatively longer pause.
15
System Design and Algorithm
The geometric version can be displayed or emailed as an
image.
Conversion to text is likely to be more useful (for browsing
and searching).
This section develops the algorithmic components towards
this goal.
16
Stroke Detection
Characters can be viewed as a sequence of strokes.
We have analyzed the English alphabets and characterized
the basic set of strokes.
17
Stroke Detection
To identify the strokes, computes a running variance of the
accelerometer readings. When this variance falls below a
threshold, the system marks those regions as a pause of the
hand.
18
Character Recognition
19
Character Recognition
The stroke grammar presents a number of ambiguities.
“O” and “S” :
If the hand is moving upwards, computed from the sign of
the Y-axis acceleration, the alphabet is declared as an “O”.
“D” and “P”:
If the ranges are comparable (second stroke being greater
than 0.75 of the first), the alphabet is deemed as a “D”.
“X” and “Y”
20
Word Recognition
First, we make the observation that while transitioning from
one alphabet to the next, users have a naturally longer pause.
Second, we observe that if the hand moves in a leftward
horizontal direction, it may be a hint about the start of a new
character.
Third, we ask the users to gesture a “dot” between characters.
For example, “13” should be written as “|.ƆƆ”, while the “B”
should be “|ƆƆ”.
21
Control Gestures
The PhonePoint Pen assigns a unique gesture:
The space is denoted by a long horizontal movement or two dots.
The deletion is like using an eraser – the users shakes her hand
at least four times briskly.
To email, the user must draw a check mark in the air.
22
Implementation and Evaluation
We prototyped the PhonePoint Pen on a Nokia N95 mobile
phone.
The system equips with a software accessible 3-axes
accelerometer.
The phone accelerometer was programmed to obtain 30-35
instantaneous acceleration readings per second.
We developed a server side implementation first on
MATLAB and ported this code on Python for on-phone
processing.
23
Evaluation Methodology
We define accuracy as the fraction of successful recognitions,
when a user writes individual alphabets/characters or words.
Compute character recognition accuracy
Compute word-recognition accuracy
24
Performance Evaluation
We present the evaluation results from the Duke University student
group.
25
Systematic Evaluation -- performance
We asked 2 expert users to draw each English alphabet 10 times.
The performances are comparable, except in some cases (D, G, Q, Y)
90.15%
77%
26
Systematic Evaluation -- performance
27
Systematic Evaluation
-- inexperienced users
Requested 4 trained users and 4 novice users to write all 26 alphabets
only one trial.
28
Systematic Evaluation – Speed to write
On average, correctly decoded characters were written in
around 4.5 seconds.
29
Systematic Evaluation – Word recognition
Randomly selected humans were able to read the words with
40% error.
30
Energy Measurements
We sampled the accelerometer at the same rate as
PhonePoint Pen on a fully charged Nokia N95 8GB phone.
The phone exhibited an average battery-lifetime of 40 hours.
We conclude that PhonePoint Pen’s impact on battery
lifetime is marginal, especially in light of its occasional
usage pattern.
31
Limitations
Drawing and Writing Long Words still Primitive
Writing while Moving
Cursive Handwriting
Smaller Alphabets and Quicker Writing
Realtime Character Display
Comment on Survey and Testing Population
32
Conclusion
This paper attempts to exploit the accelerometer in mobile
phones to develop a new input technology.
By holding the phone like a pen, the user should be able to
write short messages in the air.
The written message is displayed on the phone-screen, and
may also be emailed to the user if she desires.
33
Thanks for your listening!
34