SurroundSense: Mobile Phone Localization
via Ambience Fingerprinting
1
Context
Pervasive wireless connectivity
+
Localization technology
=
Location-based applications
2
Location-Based Applications (LBAs)
 For Example:
 GeoLife shows grocery list when near Walmart
 MicroBlog queries users at a museum
 Location-based ad: Phone gets coupon at Starbucks
 iPhone AppStore: 3000 LBAs, Android: 500 LBAs
3
Most emerging location based apps
do not care about the physical location
GPS: Latitude, Longitude
4
Most emerging location based apps
do not care about the physical location
GPS: Latitude, Longitude
Instead, they need the user’s logical location
Starbucks, RadioShack,
Museum, Library
5
Physical Vs Logical
 Unfortunately, most existing solutions are physical




GPS
GSM based
SkyHook
Google Latitude
 RADAR
 Cricket
 …
6
Given this rich literature,
Why not convert from
Physical to Logical Locations?
7
Physical Location
Error
8
Starbucks
Pizza Hut
Physical Location
Error
9
Starbucks
Pizza Hut
Physical Location
Error
The dividing-wall problem
10
SurroundSense:
A Logical Localization Solution
11
Hypothesis
It is possible to localize phones by
sensing the ambience
such as sound, light, color, movement, WiFi …
12
Hypothesis
It is possible to localize phones by
sensing the ambience
such as sound, light, color, movement, WiFi …
13
Multi-dimensional sensing extracts more
ambient information
Any one dimension may not be unique,
but put together, they may provide a
unique fingerprint
14
SurroundSense
 Multi-dimensional fingerprint
 Based on ambient sound/light/color/movement/WiFi
Starbucks
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
Pizza Hut
Wall
15
Should Ambiences be Unique Worldwide?
H
J
P
I
A
K CB D
E
Q
R
O F
Q
L
N M
G
16
GSM Ambiences
provides macro
location
(strip mall)
Should
be Unique
Worldwide?
SurroundSense refines to Starbucks
H
J
P
I
A
K CB D
E
Q
R
O F
Q
L
N M
G
17
Why does it work?
The Intuition:
Economics forces nearby businesses to be diverse
Not profitable to have 3 adjascent coffee shops
with same lighting, music, color, layout, etc.
SurroundSense exploits this ambience diversity
18
SurroundSense Architecture
Ambience Fingerprinting
Matching
Sound
Color/Light
+
Test
Fingerprint
Acc.
=
WiFi
GSM Macro
Location
Logical
Location
Fingerprint
Database
Candidate Fingerprints
19
Acoustic fingerprint
(amplitude distribution)
Fingerprints
0.14
 Sound:
(via phone
microphone)
Normalized Count
0.12
0.1
0.08
0.06
0.04
0.02
0
-1
-0.8 -0.6 -0.4 -0.2
0
0.2
0.4 0.6 0.8 1
Amplitude Values
1
Lightnes
s
 Color:
(via phone
camera)
Color and light fingerprints on HSL space
0.5
0
0
0.5
Hue
1 0
0.2
0.4
0.6
0.8
1
Saturation
20
Fingerprints
 Movement: (via phone accelerometer)
Cafeteria
Clothes Store
Grocery Store
Moving
Static
21
Fingerprints
 Movement: (via phone accelerometer)
Cafeteria
Clothes Store
Grocery Store
Moving
Static
Queuing
Seated
22
Fingerprints
 Movement: (via phone accelerometer)
Cafeteria
Clothes Store
Grocery Store
Moving
Static
Pause for product
browsing
Short walks between
product browsing
23
Fingerprints
 Movement: (via phone accelerometer)
Cafeteria
Clothes Store
Grocery Store
Moving
Static
Walk more
Quicker stops
24
Fingerprints
 Movement: (via phone accelerometer)
Cafeteria
Clothes Store
Grocery Store
Moving
Static
 WiFi: (via phone wireless card)
ƒ(overheard WiFi APs)
25
Discussion
 Time varying ambience
 Collect ambience fingerprints over different time windows
 What if phones are in pockets?
 Use sound/WiFi/movement
 Opportunistically take pictures
 Fingerprint Database
 War-sensing
26
Evaluation Methodology
 51 business locations
 46 in Durham, NC
 5 in India
 Data collected by 4 people
 12 tests per location
 Mimicked customer behavior
27
Evaluation: Per-Cluster Accuracy
Cluster
1
2
3
4
5
6
7
8
9
10
No. of Shops
4
7
3
7
4
5
5
6
5
5
Accuracy (%)
Localization accuracy per cluster
Cluster
28
Evaluation: Per-Cluster Accuracy
Cluster
1
2
3
4
5
6
7
8
9
10
No. of Shops
4
7
3
7
4
5
5
6
5
5
Localization accuracy per cluster
Accuracy (%)
Fault tolerance
Cluster
29
Evaluation: Per-Cluster Accuracy
Cluster
1
2
3
4
5
6
7
8
9
10
No. of Shops
4
7
3
7
4
5
5
6
5
5
Accuracy (%)
Sparse WiFi
Localization accuracy
perAPs
cluster
Cluster
30
Evaluation: Per-Cluster Accuracy
Cluster
1
2
3
4
5
6
7
8
9
10
No. of Shops
4
7
3
7
4
5
5
6
5
5
Accuracy (%)
Localization accuracy per cluster
No WiFi APs
Cluster
31
Evaluation: Per-Scheme Accuracy
Mode
WiFi
Snd-Acc-WiFi
Snd-Acc-Lt-Clr
SS
Accuracy
70%
74%
76%
87%
32
Evaluation: User Experience
Random Person Accuracy
1
WiFI
Snd-Acc-WiFi
Snd-Acc-Clr-Lt
SurroundSense
0.9
0.8
0.7
CDF
0.6
0.5
0.4
0.3
0.2
0.1
0
0
10
20
30 40 50 60 70
Average Accuracy (%)
80
90 100
33
Limitations and Future Work
 Energy-Efficiency
 Continuous sensing likely to have a large energy draw
34
Limitations and Future Work
 Energy-Efficiency
 Continuous sensing likely to have a large energy draw
 Localization in Real Time
 User’s movement requires time to converge
35
Limitations and Future Work
 Energy-Efficiency
 Continuous sensing likely to have a large energy draw
 Localization in Real Time
 User’s movement requires time to converge
 Non-business locations
 Ambiences may be less diverse
36
Limitations and Future Work
 Electroic compasses can fingerprint layout
 Tables and shelves laid out in different orientations
 Users forced to orient in those ways
Quick Time™ and a
TIFF ( Uncompr ess ed) decompr ess or
ar e needed to s ee this pic ture.
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
a d na ™emi Tkci uQ
ro sser pmo ced ) des serp mocn U( F FIT
.erut cip s iht e es ot ded een era
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
37
Conclusion
Ambience can be a great clue about location
Ambient Sound, light, color, movement …
None of the individual sensors good enough
Combined they may be unique
Uniqueness facilitated by economic incentive
Businesses benefit if they are mutually diverse in ambience
Ambience diversity helps SurroundSense
Current accuracy of 89%
38
 Questions?
39
Summary
SurroundSense evaluated in 51 business locations
Achieved 89% accuracy
But perhaps more importantly,
SurroundSense scales to any part of the world
40
SurroundSense
 Today’s technologies cannot provide logical localization
 Ambience contains information for logical localization
 Mobile Phones can harness the ambience via sensors
 More sensors, beter reliability
 Evaluation results:
 51 business locations,
 87% accuracy
SurroundSense can scale to any part of the world
41
42
Evaluation: Per-Shop Accuracy
Localization Accuracy per Shop
1
WiFi
Snd-Acc-WiFi
Snd-Acc-Clr-Lt
SurroundSense
0.9
0.8
0.7
CDF
0.6
0.5
0.4
0.3
0.2
0.1
0
0
10
20
30 40 50 60 70
Average Accuracy (%)
80
90 100
43
Architecture: Filtering & Matching
Candidate Fingerprints
44
Fingerprinting Sound
 Fingerprint generation : Signal amplitude
 Amplitude values divided in 100 equal intervals
 Sound Fingerprint = 100 normalized values
•
valueX = # of samples in interval x / total # of samples
 Filter Metric: Euclidean distance
 Discard candidate fingerprint if metric > threshold г
 Threshold г
 Multiple 1 minute recordings at the same location
 di = max dist ( any two recordings )
 г = max ( di of candidate locations )
45
Fingerprinting Motion
 Fingerprint generation: stationary/moving periods
 Sitting (restaurants, cafes, haircutters )
 Slow Browsing (bookstores, music stores, clothing)
 Speed-Walking (groceries)
 Filter Metric:
R
tmoving
t static
0.0 ≤ R ≤ 0.2
0.2 ≤ R ≤ 2.0
2.0 ≤ R ≤ ∞
sitting
slow browsing
speed-walking
 Discard candidate fingerprints with different classification
46
Fingerprinting WiFi
 Fingerprint generation: fraction of time each unique
address was overheard
 Filter/Ranking Metric
 Discard candidate fingerprints which do not have similar
MAC frequencies
47
Fingerprinting Color
 Floor Pictures
 Rich diversity across different locations
 Uniformity at the same location
 Fingerprint generation: pictures in HSL space
 K-means clustering algorithm
 Cluster’s centers + sizes
 Ranking metric
48
Thinking about Localization
from an application perspective…
49
Emerging location based apps need
place of user, not physical location
Starbucks, RadioShack,
Museum, Library
Latitude, Longitude
50
Emerging location based apps need
place of user, not physical location
Starbucks, RadioShack,
Museum, Library
Latitude, Longitude
We call this Logical Localization …
51
Can we convert from
Physical to Logical Localization?
52
Can we convert from
Physical to Logical Localization?
State of the Art in Physical Localization:
1. GPS
Accuracy: 10m
2. GSM
Accuracy: 100m
3. Skyhook (WiFi+GPS+GSM)
Accuracy: 10m-100m
53
Can we convert from
Physical to Logical Localization?
State of the Art in Physical Localization:
1. GPS
Accuracy: 10m
2. GSM
Accuracy: 100m
3. Skyhook (WiFi+GPS+GSM)
Accuracy: 10m-100m
Widely-deployable localization technologies
have errors in the range of several meters
54
Physical Vs Logical
 Lot of work in physical localization




GPS (5m accuracy, outdoor only)
GSM based (high coverage, 200m accuracy)
SkyHook (40m accuracy, need war-driving)
Google Latitude (similar)
 RADAR (5m accuracy, need indoor mapping)
 Cricket (install per-room devices)
 …
Each of them has distinct
problems …
55
Physical Vs Logical
 Lot of work in physical localization




GPS (5m accuracy, outdoor only)
GSM based (high coverage, 200m accuracy)
SkyHook (40m accuracy, need war-driving)
Google Latitude (similar)
 RADAR (5m accuracy, need indoor mapping)
 Cricket (install per-room devices)
 …
Each of them has distinct
problems …
56
Even if we assume the best of all solutions
(5m accuracy indoors)
logical localization still remains a challenge.
57
Contents
 SurroundSense
 Evaluation
 Limitations and Future Work
 Conclusion
58
Contents
 SurroundSense
 Evaluation
 Limitations and Future Work
 Conclusion
59
Contents
 SurroundSense
 Evaluation
 Limitations and Future Work
 Conclusion
60
Contents
 SurroundSense
 Evaluation
 Limitations and Future Work
 Conclusion
61
Contents
 SurroundSense
 Evaluation
 Limitations and Future Work
 Conclusion
62
Evaluation: Per-Cluster Accuracy
Cluster
1
2
3
4
5
6
7
8
9
10
No. of Shops
4
7
3
7
4
5
5
6
5
5
Accuracy (%)
Localization accuracy per cluster
Multidimensional
sensing
Cluster
63
Limitations and Future Work
 Energy-Efficiency
 Localization in Real Time
 Non-business locations
64
Limitations and Future Work
 Energy-Efficiency
 Continuous sensing likely to have a large energy draw
 Localization in Real Time
 Non-business locations
65
Limitations and Future Work
 Energy-Efficiency
 Continuous sensing likely to have a large energy draw
 Localization in Real Time
 User’s movement requires time to converge
 Non-business locations
66
Limitations and Future Work
 Camera inside pockets
 Detect phone when out of pocket
 Takes pictures when camera pointing downward
67