SurroundSense:
Mobile Phone Localization
via Ambience Fingerprinting
Martin Azizyan Duke University
Ionut Constandache Duke University
Romit Roy Choudhury Duke University
Abstract
• Mobile computing applications center around
user’s location
• Term- Physical Location
(coordinates Latitude and Longitude)
• Term- Logical Location
(like Starbucks or McDonalds)
• Lots of Research available for physical location
• Few attempts recognizing logical location
Abstract
• Ambient sound, light, and color from phone’s
camera and microphone
• Accelerometers for user-motion
• Adjacent stores can be separated logically
• They propose SurroundSense, a mobile phone
based system explores ambience fingerprinting
• 51 different stores - average accuracy of 87%
when all sensing modalities are employed
Introduction
•
•
•
•
•
Starbucks -coffee machines and microwaves
Restaurants -forks and spoons clinking
Target – red colors
Panera Breads- yellow colors
Floors with carpets, ceramic tiles, or wooden
strips
• Bars -dim yellow lights
• BlockBuster - bright white light
Introduction
• Wal-Mart - walking up and down aisles
• Barnes and Noble -relaxed stroll with long
pauses
• Restaurants - short queuing followed by a long
duration of sitting
• May not be unique based on any one attribute,
the combination of all exhibits diversity
SurroundSense Architecture
SurroundSense Architecture
•
•
•
•
•
•
•
•
Mobile phone user visits an unknown store.
The phone senses the ambience
Values forwarded to the fingerprinting factory
Types of sensor data identified
Phone’s (GSM-based) physical coordinates
Geographical database
Fingerprint database and Fingerprint matching
The matching module -best match for test
fingerprint
System Design
•
•
•
•
•
Fingerprinting Sound
Fingerprinting Motion using Accelerometers
Fingerprinting Color/Light using Cameras
Fingerprinting Wi-Fi
Fingerprint Matching
Fingerprinting Sound
• Recorded ambient sound for one
• Time domain, a simple fingerprinting scheme
based on signal amplitude
• Acoustic fingerprints - computed the pair-wise
distances.
• Use sound only as a filter
• The output is fed to the accelerometer filter
Sound fingerprints from 3 adjacent stores
Fingerprinting
Motion using Accelerometers
•
•
•
•
•
Human movements in a location
Restaurants -stationary for long durations
Grocery store - more mobile
Accelerometer readings - stationary vs in motion
Vector machines (SVM), a popular data
classification tool
• User movement is prone to fluctuation
• Clothing store - browse long time or purchase in
haste
• Accelerometers as a filtering mechanism too
Sample Accelerometers Traces
Fingerprinting
Color/Light using Cameras
•
•
•
•
•
•
The wall and floor colors contribute to theme
Use automatically-taken phone pictures
Only floor-facing pictures are used
Color/light extraction
Why picture of the floor?
Colors of carpets, tiles, marble, and wooden
floors
• HSL - hue-saturation-lightness
• Clusters of color color-light fingerprint
Color/light fingerprint in the HSL space
Fingerprinting Wi-Fi
•
•
•
•
•
WiFi fingerprinting no good for logical places
WiFi based fingerprinting -fifth sensor
MAC addresses of visible APs
MAC addresses recorded every 5 seconds
Computing the fraction of times each unique
MAC address was seen over all recordings
• A tuple of fractions forms the WiFi fingerprint of
that place
Fingerprint Matching
• SurroundSense uses 4 filtering/matching
• The (WiFi, sound, and accelerometer) filters are
applied first
• Candidate set fed to the color/light-based
matching scheme
• Use the color/light based matching scheme last
• The final output is an ordered list of candidates
– the top ranked candidate is declared to be the
location of the phone
ProtoType Implementation
• Client and Server
• Populating the Fingerprint Database
Special Note: SurroundSense was implemented on Nokia N95
using Python platform. The server -MATLAB, Python code, data
mining tools
Client and Server
•
•
•
•
•
•
•
•
•
Sensor runs on threads and execute API calls
The accelerometer samples - 4 readings per second.
The audio sampling rate is 8 kHz.
Pictures are taken every 5 seconds
A meta file -stores date, time, GSM, camera mode
The server - several modules.
A Data Manager formats raw data appropriately.
The formatted data sent to Fingerprinting Factory
A MATLAB/ Python based Filtering/Matching
Module -computes the top-ranked match.
Populating the Fingerprint Database
•
•
•
•
•
•
How did they build a fingerprint database?
46 business locations
5 locations in India
Students visited 51 stores
Stores visited multiple times
Design location labeling games. The person with
a best match may win a prize.
• More people play larger fingerprint database
Evaluation
• Partially Controlled Experimentation
• Performance
Pre-Cluster Accuracy
Per-Shop Accuracy
Per-User Accuracy
Per-Sensor Accuracy
Partially Controlled Experimentation
• Not performed with a real user base
• Mobile phones in our hand (and not in our
pockets)
• Phones took pictures for color and light
fingerprinting.
• In uncontrolled environments, phones in pocket
• New wearable mobile phones
• Wrist watches and Necklaces
Mimicking Customer Behavior
•
•
•
•
•
•
•
Groups of 2 people
Went to different stores -time-separated
Fingerprinted every store in cluster
Behave like normal customers
Purchase coffee and food
Mimic the movement of another customers
Atypical behavior -picking up pre-ordered food
shopping very quickly.
Performance: Per-Cluster Accuracy
•
•
•
•
•
Evaluate 4 modes
1. WiFi-only
2. Sound, Accelerometer, Light and Color
3. Sound, Accelerometer
4. SurroundSense (SS) combined all modes
of ambience fingerprinting
• SurroundSense average accuracy of 87%
• All the sensors 90%...
Performance: Per-Shop Accuracy
• 47% of the shops can be localized perfectly using
SurroundSense.
• WiFi displays bimodal behavior – it’s either high
accuracy, or seriously suffers
• Clearly, the combination of multi-modal
fingerprinting is best
Performance: Per-User Accuracy
• User assigned to a random set of stores
• Report the average accuracy
• SurroundSense users achieve between 73% and
75% accuracy
• The accuracy grows to an average of 83% or
more for 80% of the users
• The median accuracy is around 88%, while 10%
users experience 96% accuracy or more
Performance: Per-Sensor Accuracy
• hand-picked 6 examples -merits and demerits of
each sensor
• Whenever the accelerometer is used-accuracy is
always 100%
• Only the camera, -100% accuracy in this
location
• Only color gives average accuracy of 91%. When
sound is added, -66%
• If the correct location is filtered out, the final
match incorrect
Limitations and Future Work
Energy Considerations
• Energy efficient localization and sensing
• Simple sensing mechanisms – when outdoors
• Variation in GSM signal strengths
• Temperature sensing
• An accelerometer trace requires time
• Faster methods of localization without
compromising accuracy
Conclusion
• Logical location, vs physical coordinates
• The main idea -ambient sound, light, color, RF,
luser movement
• Fingerprint identifies user’s location
• SurroundSense not a stand-alone technique- use
with GSM location
• SurroundSense step towards indoor localization
• Further research - better energy management
• SurroundSense a viable solution of the future
Questions/Comments?