Decoding Keypresses
using Mobile Phone
Accelerometers
Woodams Clark
Jesse Freeman
Christopher Weeden
Mobile Phones
Mobile phones come with a wide array of sensors that allow them to interface
with the world around them.
This can also mean they can be used in unintended ways.
Many OS now have users identify permissions for apps to use sensors
- However many are still not tightly controlled
Sensors in a SmartPhone
Accelerometer
Heart Rate Monitor
Gyroscope
Fingerprint Sensor
Magnetometer
Pedometer
Proximity Sensor
Light Sensor
Barometer
Accelerometer
Cannot set permissions for apps to use accelerometer
By using the accelerometer, vibrations can be recorded in order to decipher
what a person is typing on their keyboard.
Many studies have been done in the past using things that can record vibrations,
or electromagnetic waves, but this equipment is obviously more sensitive and
stronger than a standard consumer device
Related Work
●
●
●
●
●
Emanations of electrical and mechanical devices have been used to expose
user’s activities as far back as 1940s
TEMPEST - Referring to spying on information systems through leaking
emanations, including unintentional radio or electrical signals, sounds and
vibrations
Techniques have been used to capture electromagnetic
signals generated by teletype terminal in the 1940s.
Used to detect Viet Cong trucks from distances up to 10
miles away.
Techniques used on CRT and LCD screens in order to recover
their contents. Using electromagnetic emanation
Optical Emanation
Optical emanation has been used to reveal contents displayed on CRT monitors
by using diffuse reflections of the monitors.
Work was later extended to capture reflections from a target’s eye.
These attacks are effective at distances up to 30 meters, but are much more
difficult to execute with an obstructed line of sight.
Acoustic Emanations
High technical devices are not needed when it comes to acoustic emanations. A
consumer grade microphone will suffice.
One can be used to figure out the contents of a dot-matrix printer with high
accuracy.
Requires the microphone to always be near the target.
Attack and Threat Model
In order for the previous types of attack to be possible, it must be easy to
exploit the user’s surroundings.
Using a phone increases the chance of attacks being possible.
●
●
Come with a lot of sensors that can be exploited
But can the sensors be used to actually get decipherable information and
patterns from the target?
Attack Model Continued
Many users place their mobile devices on their desk when doing work.
Model works as follows:
●
●
●
adversary installs malicious application on the phone
application will record accelerometer data by periodically sampling activity
exfiltrate the collected data via connecting to the internet and uploading
the data to a remote server
Application of previous techniques
There are two key problems associated with this work:
●
●
Given that the accelerometers found in current mobile phones sample at
rates that are orders of magnitude smaller than previous acoustic and
electromagnetic attacks, can keypresses even be detected by these
sensors?
if such events can be observed can previously developed methodology such
as identifying individual keys using neural networks be applied to identify
keystrokes?
Comparing two different mobile phones
Comparing two different mobile phones cont.
The iPhone 3GS provides very noisy output from the accelerometer.
●
You can’t differentiate between specific key presses because of all the
output
The iPhone 4 has a much more exact accelerometer
●
Gives a better reading for individual key presses
What does this mean?
Accelerometer-based eavesdropping could potentially be implemented in
phones!
With the accuracy of the iPhone4’s accelerometer there’s a potential to
eavesdrop simply by putting the phone next to a keyboard
Possibly with newer technology using an iPhone 6s there could be better
results?
The experiments set-up
The researchers used the following materials to test the potential of
eavesdropping:
●
●
●
iPhone4
Wooden Desk
Apple Blue-tooth keyboard
Each key was then pressed 150 times to provide training data
Single key press results
Using machine learning algorithms to determine single key results the accuracy was very poor
●
Achieved only a 25.89% accuracy with the iPhone4
●
This was only a third of the results of previous work 78.85%
●
The iPhone’s accelerometer only samples data at a rate of 100Hz
●
Previous work also had an accelerometer that sampled at a rate that was 441 times higher
Acoustic accuracies vs Seismic Accuracies
Due to the Implications
Due to the extremely low sample rates of each key this required that a pair wise system be put in
place to improve accuracy:
●
Pi and Pj are two sequential keypresses
The relation between the two keypresses have two features:
●
●
Horizontal Orientation: The location of each keypress event relative to a central line that divides
the keyboard in half
Distance between Consecutive keypresses: for a threshold of distance in keys there is a distance
where two keys are either classified as either “far” from each other or “near” to each other
The Central line is defined as the line between the keys t,g,b (the left side) and y,h,n (the right side)
The Word Canoe
This two keypress scheme is used to identify more features of the word that was typed with relation
to each of the keys pressed to narrow down the possibilities.
Each word of length n will have n-1 pairs that are output
the word canoe can be broken down into these pairs of keypresses:
“ca” “an” “no” “oe”
which would be represented as:
LLN. LRF. RRF. RLF
since “ca” are both on the left side of the keyboard and were classified as “near” each other the
above results shows that it is represented as LLN
Results
L/R
Distinguishing pairs that are L/R
and N/F is easy, even by eye.
Left/Right pairs have large GForce difference, whereas a
Near/Far pair has minimal GForce difference
N/F
Results
Left
Ultimately this means that
distinguishing individual keys is
hard, but detecting “Regions”
of keys is easy.
L/R classifier was able to
correctly identify 91% of the
individual keypresses as right or
left, and 70% of the keypress
pairs as near or far.
Right
Results Conjunctions
The frequency of correctly matched
words declined greatly due to 2 and
3 letter words.
In english, 2 and 3 letter words tend
to be conjunctions (and, the, to, or,
an…) and can be easily inferred when
the rest of the sentence is decoded.
Typed Text:
The birch canoe slid on the smooth planks
Recovered Text:
*** punch canoe slid ** *** smooth planks
Typed Text:
Glue the sheet to the dark blue background
Recovered Text:
Glue *** sheet ** *** dark blue background
Results Accuracy
Experiment 1
First experiment attempted to decode one of
the first 10 Harvard sentences, whereas the
second experiment attempted to decode all 10.
Both used a dictionary of all words in the first
10 Harvard sentences.
The accuracy of first-word-choice was only
46%, much worse than 80% from experiment 1.
However, when looking at second-word-choice
accuracy rises to nearly 73%
Experiment 2
Results - Comparison
While the results don’t appear incredible, it’s
important to keep perspective.
●
●
Using accelerometers has a much lower sample
frequency than most other methods such as Acoustic
Dictionary Techniques proposed by Berger.
Able to work on words of length 4+ characters
○
●
Bergers’ uses 7-13 and has best results with repetitive
words
Despite this, accuracy is still rather competitive to
these more robust methods.
Limitations
The largest limitation to this method is distance, noise and surface material.
The ability to distinguish keypresses from noise drops with respect to the
Inverse Square Law. A phone more than 2 feet away cannot distinguish
keystrokes from noise.
People walking by, an air-conditioner running, or finger tapping can also
completely disrupt the ability to decode keystrokes. And having the phone on
any soft surface (napkin, cloth, phone-case) will also diminish acoustic
capabilities.
Conclusions
While preventing this attack is trivial (keep phone away from keyboards), the
premise attack exploits secondary sensors in mobile devices, which the user
assumes trust of.
These sensors are assumingly innocuous, but this paper shows they can be taken
advantage of to attack the user.
A better way to prevent these attacks is to ensure access to all sensors of a
device are restricted in an appropriate manner, handled by the OS.
References
Research Paper Link:
- http://www.cc.gatech.edu/fac/traynor/papers/traynor-ccs11.pdf