Design and Implementation of
Smartphone-based Systems and
Networking
Dong Xuan
Department of Computer Science and Engineering
The Ohio State University, USA
Dong Xuan (CSE/OSU) / 2009
Outline

Smartphones Basics

Mobile Social Networks

E-Commerce

E-Health

Safety Monitoring

Future Research Directions
Dong Xuan (CSE/OSU) / 2010
2
Smartphone Basics

A smartphone is a mobile phone offering advanced
capabilities, often with PC-like functionality

Hardware (Apple iPhone 3GS as an example)





CPU at 600MHz, 256MB of RAM
16GB or 32GB of flash ROM
Wireless: 3G/2G, WiFi, Bluetooth
Sensors: camera, acceleration, proximity, light
Functionalities

Communication
News & Information
Socializing
Gaming

Schedule Management etc.



Dong Xuan (CSE/OSU) / 2010
3
Smartphone Popularity

Smartphones are popular and will become more popular
Dong Xuan (CSE/OSU) / 2010
4
Smartphone Accessories
Dong Xuan (CSE/OSU) / 2010
5
Smartphone Features

Communication/Sensing/Computation

Inseparable from our human life
Dong Xuan (CSE/OSU) / 2010
6
Our Smartphone Systems

E-SmallTalker [IEEE ICDCS10]:
senses information published by
Bluetooth to help potential friends find
each other (written in Java)

E-Shadow [IEEE ICDCS11]: enables
rich local social interactions with
local profiles and mobile phone
based local social networking tools

P3-Coupon [IEEE Percom11]:
automatically distributes electronic
coupons based on an probabilistic
forwarding algorithm
Dong Xuan (CSE/OSU) / 2010
7
Our Smartphone Systems

Drunk Driving Detection [PerHealth10]: uses smartphone (Google
G1) accelerometer and orientation
sensor to detect

Stealthy Video Capturer [ACM
WiSec09]: secretly senses its
environment and records video via
smartphone camera and sends it to a
third party (Windows Mobile
application)
Download & Run
Dong Xuan (CSE/OSU) / 2010
Video sent by Email
Captured Video
8
Exemplary System I:E-SmallTaker

Small Talk

A Naïve Approach

Challenges

System Design

Implementation and Experiments

Remarks
Dong Xuan (CSE/OSU) / 2010
9
Small Talk


People come into contact opportunistically
Face-to-face interaction





Crucial to people's social networking
Immediate non-verbal communication
Helps people get to know each other
Provides the best opportunity to expand social network
Small talk is an important social lubricant


Difficult to identify significant topics
Superficial
Dong Xuan (CSE/OSU) / 2010
10
A Naive Approach of Smartphonebased Small Talk




Store all user’s information, including each user’s full contact
list
User report either his own geo-location or a collection of
phone IDs in his physical proximity to the server using internet
connection or SMS
Server performs profile matching, finds out small talk topics
(mutual contact, common interests, etc.)
Results are pushed to or retrieved by users
Dong Xuan (CSE/OSU) / 2010
11
However……




Require costly data services (phone’s internet
connection, SMS)
Require report and store sensitive personal
information in 3rd party
Trusted server may not exist
Server is a bottleneck, single point of failure, target of
attack
Dong Xuan (CSE/OSU) / 2010
12
E-SmallTalker – A Fully
Distributed Approach






No Internet connection required
No trusted 3rd party
No centralized server
Information stored locally on mobile phones
Original personal data never leaves a user’s phone
Communication only happens in physical proximity
Dong Xuan (CSE/OSU) / 2010
13
Two Challenges

How to exchange information without establishing a Bluetooth
connection

Available data communication channels on mobile phones




Cellular network (internet, SMS, MMS), Bluetooth, WiFi, IrDA
Bluetooth is a natural choice
Bluetooth connection needs user’s interaction due to security reasons
How to find out common topics while preserving users privacy


No pre-shared secret for strangers
Bluetooth Service Discovery Protocol can only transfer limited service
information
Dong Xuan (CSE/OSU) / 2010
14
System Architecture




Context exchange
Context encoding and matching
Context data store
User Interface
Dong Xuan (CSE/OSU) / 2010
15
Context Exchange
 Exploit Bluetooth service discovery protocol

No Bluetooth connection needed
 Publish encoded contact data (non-service related) as (virtual) service attributes

Limited size and number( e.g. 128 bytes max each attribute)
Dong Xuan (CSE/OSU) / 2010
16
Context Encoding



Example of Alice’s Bloom
filter
Alice has multiple contacts,
such as Bob, Tom, etc.
Encode contact strings,
Firstname.lastname@phone
_number, such as
“Bob.Johnson@5555555555
” and
“Tom.Mattix@6141234567”
Dong Xuan (CSE/OSU) / 2010
17
Implementation

J2ME
 about

40 java classes, 127Kb jar file
On real phones
 Sony
Ericsson (W810i), Nokia (5610xm, 6650, N70, N75,
N82)
Dong Xuan (CSE/OSU) / 2010
18
Experiments

Settings


6 phones, n=150, k=7, m=1024 bits, default distance=4m, average of
10 runs
Performance Metrics

Discovery time: the period from the time of starting a search to the time of
finding someone with common interest, if there is any
 Discovery rate: percentage of successful discoveries among all attempts


Power consumption
Factors



Bluetooth search interval
Number of users
Distance
Dong Xuan (CSE/OSU) / 2010
19
Experiment Results



Minimum, average and maximum discovery time are
13.39, 20.04 and 59.11 seconds respectively
Always success if repeat searching, 90% overall if
only search once
Nokia N82 last 29 hours when discovery interval is
60 seconds
Dong Xuan (CSE/OSU) / 2010
20
Related Work

Social network applications on mobile phones

Social Serendipity


PeopleTones, Hummingbird, Just-for-Us, MobiLuck, P3 Systems, Micro-Blog,
and Loopt


Centralized, GPS location matching, Internet, existing friends
Nokia Sensor and PeopleNet


Centralized, Bluetooth MAC and profile matching, SMS, strangers
Distributed, profile, Bluetooth / Wifi connection, existing friends
Private matching and set intersection protocols

Homomorphic encryption based
 Too much computation and message overhead for mobile phone

Limitations
Require costly data services (phone’s internet connection, SMS)
 Require report and store sensitive personal information
 Bottleneck, single point of failure, target of attack

Dong Xuan (CSE/OSU) / 2010
21
Remarks

Propose, design, implement and evaluate the E-SmallTalker
system which helps strangers initialize a conversation



Leveraged Bluetooth SDP to exchange these topics without
establishing a connection
Customized service attributes to publish non-service related
information.
Proposed a new iterative commonality discovery protocol based on
Bloom filters that encodes topics to fit in SDP attributes to achieve a
low false positive rate
Dong Xuan (CSE/OSU) / 2010
22
Exemplary System II:E-Shadow

Concept

Application Scenario

Goals and Challenges

System Design

Implementation and Experiments

Remarks
Dong Xuan (CSE/OSU) / 2010
23
Concept


Motivation
 Importance of Face-to-Face Interaction
 Prevalence of mobile phones
Distributed mobile phone-based local social
networking system
 Local profiles
 Mobile phone based local social interaction tools
Dong Xuan (CSE/OSU) / 2010
Application Scenario: Conference
Dong Xuan (CSE/OSU) / 2010
Goals and Challenges

Design Goals
 Far-reaching and Unobtrusive
 Privacy and Security
 Auxiliary Support for Further Interactions
 Broad Adoption

Challenges
 Lack of Communication Support
 Power and Computation Limitation
 Non-pervasive
Dong Xuan (CSE/OSU) / 2010
Localization Service
Layered Publishing

Spatial Layering
 WiFi

SSID
at least 40-50 meters, 32 Bytes
 Bluetooth

20 meters, 2k Bytes
 Bluetooth


Device (BTD) Name
Service (BTS) Name
10 meters, 1k Bytes
Temporal Layering
 For
people being together long or repeatedly
 Erasure Code
Dong Xuan (CSE/OSU) / 2010
E-Shadow Publishing Procedure
Sensor
Feedback
Valve
Generator
Filter
User Online
Maual Data
Input Mining
Information
BT
Device
BT
Service
WiFi
Dong Xuan (CSE/OSU) / 2010
Database
Matching E-Shadow with its Owner
Intuitive Approach: Localization
 However, imprecision beyond 20-25 meters

Dong Xuan (CSE/OSU) / 2010
Human Direction-driven Localization

Direction more important than distance
 Human

observation
A new range-free localization technique
 RSSI
comparison: Less prone to errors
 Space partitioning: Tailored for direction decision
Dong Xuan (CSE/OSU) / 2010
Walking Route and Localization

We allow users to walk a distance




Triangular route: A->B->C in (a), for illustration purposes
Semi-octogonal route: A->B->C->D->E in (c), more natural
Take measurements on turning points
Calculate the direction through RSSI comparison and space
partitioning
Dong Xuan (CSE/OSU) / 2010
Implementation

Information
Publishing Module






Database
Generator
Buffers
Control Valve
Broadcasting
Interfaces
Retrieval &
Matching Module

Receivers
 Localization
 Decoding & Storage


Sensing Module
User Interface
Dong Xuan (CSE/OSU) / 2010
Evaluations (1)-Time & Energy

E-Shadow Collection Time




E-Shadow Power
Consumption


Dong Xuan (CSE/OSU) / 2010
WiFi SSID: 2 seconds
BTD: 12-18 seconds
BTS: 25-35 seconds
3 hours in full performance
operation
>12 hours in typical situation
Evaluations (2)-Localization
3 Outdoor Experiments:
Open field campus
2 Indoor Experiments:
Large classroom
Dong Xuan (CSE/OSU) / 2010
Evaluation (3)-Simulations
Large-Scale Simulations:
Angle deviation CDFs
12 times of exemplary
direction decisions
Dong Xuan (CSE/OSU) / 2010
Related Work

Centralized mobile phones applications


Social Serendipity
 Centralized, Bluetooth MAC and profile matching, SMS, strangers
Decentralized mobile phone applications

Nokia Sensor
 Distributed, profile, Bluetooth / Wifi connection, existing friends
 E-Smalltalker
 Distributed, no Bluetooth / Wifi connection, strangers

Localization techniques for mobile phones applications

GPS
 Virtual Compass
 peer-based relative positioning system using Wi-Fi and Bluetooth radios

Limitations

Privacy compromise
 Unable to capture the dynamics of surroundings
 No mapping between electronic ID and human face
 Localization techniques either not pervasive or not accurate for long range
Dong Xuan (CSE/OSU) / 2010
36
Remarks

Propose, design, implement and evaluate the E-Shadow
system which lubricates local social interactions




E-Shadow concept
Layered publishing to capture the dynamics of surroundings
Human-assisted matching that works for mapping E-Shadow with its
owner in a fairly large distance
Implementing and evaluating E-Shadow on real world mobile phones
Dong Xuan (CSE/OSU) / 2010
37
Exemplary System III:P3-Coupon

Coupon Distribution

A Naïve Approach

Challenges

System Design

Implementation and Experiments

Remarks
Dong Xuan (CSE/OSU) / 2010
38
Electronic Coupon Distribution

Electronic coupons



Similar to paper coupons
Can be stored on mobile phones
Two distribution methods

Downloading from Internet websites




Need to define target group
Limited coverage
Hard to maintain dynamic preferences lists on central databases
Peer to Peer Distribution



No special destination/target group
More coverage
More flexible user-maintained preferences list
Dong Xuan (CSE/OSU) / 2010
39
A Naive Approach of Peer-to-Peer
Coupon Distribution


A store periodically broadcast the coupon
Users within broadcast range receive the coupon


Users forward the coupon to others in physical proximity



User can decide whether to use, forward or discard the coupon
Forwarder’s IDs are recorded in a dynamically expanding list
The coupon is used by some user
The store reward all users who have forwarded the coupon
Dong Xuan (CSE/OSU) / 2010
40
However……

Require manually establishing wireless connections




Cumbersome
Not prompt
Not possible for coupon forwarding among strangers
Require recording the entire forwarding path


Potential privacy leakage
Discourage user’s forwarding incentives
Dong Xuan (CSE/OSU) / 2010
41
Challenge

How to design a prompt coupon distribution
mechanism that
 Incentivize coupon
forwarder appropriately for keeping the
coupons circulating
 Preserve the privacy of coupon forwarders
Dong Xuan (CSE/OSU) / 2010
42
P3-Coupon – A Probabilistic
Coupon Forwarding Approach

Probabilistic sampling on forwarding path

Keep only one forwarder for each coupon: NO privacy leakage
 Probabilistically flip ownership at each hop

Accurate approximation of coupon rewards



plenty of chances of interpersonal encounters
Accurate bonus distribution with 50 coupons and 5000 people
Adaptive to different promotion strategies

Flip-once model
 Always-flip model

No manual connection establishment

Connectionless information exchange via Bluetooth SDP
Dong Xuan (CSE/OSU) / 2010
43
System Architecture

Store Side


A central server for broadcasting and redeeming coupons
Client side

Coupon forwarding manager, coupon exchange, coupon data store, user
interface
Dong Xuan (CSE/OSU) / 2010
44
Probabilistic Forwarding Algorithm

Always-Flip Model

The coupon ownership keeps flipping with certain probability at each hop.
 Good at assigning relative bonuses affected by the whole path lengths


E.g. the parent forwarder receives k times the bonus given to children forwarders
The flip probability can be calculated in advance by the store, once k is fixed, using
the following formula
Dong Xuan (CSE/OSU) / 2010
45
Probabilistic Forwarding Algorithm

Extension: Flip-Once Model
Once flipped, a coupon’s ownership remain the same in a forwarding path.
 Good at assigning absolute bonuses irrelevant of the number of following
forwarders



E.g. hop 1 user gets 10%, hop 2 user gets 5%, etc.
The flip probability can be calculated in advance by the store using the following
formula
Dong Xuan (CSE/OSU) / 2010
46
Coupon Format

Coupon description






Coupon forwarder information


Product description
Discounts
Coupon issuer
Coupon code
Start/end date
The current owner
Digital signature

Prevent forging fraud coupons
Dong Xuan (CSE/OSU) / 2010
47
Implementation

J2ME
 about

17 java classes, 1390Kb jar file
On real phones
 Samsung
Dong Xuan (CSE/OSU) / 2010
(SGH-i550), Nokia (N82, 6650, N71x)
48
Experiments

Experimental evaluations



Simulation evaluation





Coupon forwarding time
Power consumption
Number of Coupon holders vs. Time
Distribution saturation time vs. Number of Seeds
Coupon ownership distribution for probabilistic sampling
Deviation between theoretical and actual bonus (Always-Flip, FlipOnce)
Factors



Number of coupons
Number of users
Number of initial coupon holders
Dong Xuan (CSE/OSU) / 2010
49
Experiment Results




Average coupon forwarding time is 33.52 seconds
Nokia N82 last 25 hours with P3-Coupon running in
background
One coupon could be delivered to 5000 people within 32 hours
Very small deviation between theoretical and actual bonus
distribution with 50 coupons circulating among 5000 people
Dong Xuan (CSE/OSU) / 2010
50
Remarks

Propose, design, implement and evaluate the P3-Coupon
system which helps prompt and privacy preserving coupon
distribution




Probabilistic one-ownership coupon forwarding algorithm
Implement the system on various types of mobile phones
Extensive experiments and evaluations show that our approach
accurately approximate the theoretical coupon distribution in which the
whole forwarding path needs to be recorded
Practical for real-world deployment
Dong Xuan (CSE/OSU) / 2010
51
52
Exemplary System IV – Drunk Driving
Detection
Motivation
Our
Contributions
Detection Criteria
Our System
Related Work
Implementation and Evaluation
Remarks
Dong Xuan (CSE/OSU) / 2010
52
53
Motivation
Crashes
caused by alcohol-impaired driving pose a
serious danger to the general public safety and health
13,041
and 11,773 driving fatalities happened in 2007 and
2008*
32% of the total fatalities in these two years*
Drunk
driving also imposes a heavy financial burden on
the whole society
Annual
cost of alcohol-related crashes totals more than $51
billion**
* Data from U.S. NHTSA (National Highway Traffic Safety Administration)
** Data from U.S. CDC (Central of Disease Control)
Dong Xuan (CSE/OSU) / 2010
53
54
Motivation
 Detection
of drunk driving so far still relies on visual
observation by patrol officers
Drunk drivers usually make
certain types of dangerous maneuvers
NHTSA researchers identify cues of typical drunk driving behavior
 Visual
observation is insufficient to prevent drunk driving
The
number of patrol officers is far from enough
The guidelines are only descriptive and qualitative
Usually, it is too late when drunk drivers are stopped by officers
 It
is essential to develop systems actively monitoring drunk
driving and to prevent accidents
Dong Xuan (CSE/OSU) / 2010
54
55
Our Contributions
Propose
utilizing mobile phones as a platform for
active drunk driving detection system
Design a real-time algorithm for drunk driving
detection system using mobile phones
Simple

sensors required only
i.e., accelerometers and orientation sensors
Design
and implement a mobile phone-based active
drunk driving detection system
Reliable, Non-intrusive, Lightweight and
power efficient, and
No extra hardware and service cost
Dong Xuan (CSE/OSU) / 2010
55
56
Cues for Drunk Driving Detection
 Cues
related to lane position maintenance problems
E.g.,
 Cues
related to speed control problems
E.g.,
 Cues
weaving, drifting, swerving and turning with a wide radius
accelerating or decelerating suddenly, and braking erratically
related to judgment and vigilance problems
E.g.,
driving with tires on lane marker, slow response to traffic signals
Dong Xuan (CSE/OSU) / 2010
56
57
Drunk Driving Detection Criteria
Extract
fundamental detection criteria from these
cues
Capture
the acceleration features
E.g., for the lane position maintenance problems
Dong Xuan (CSE/OSU) / 2010
57
58
Drunk Driving Detection Criteria
Focus
on the first two categories of cues
They
correspond to higher probabilities of drunk driving
Map them into patterns of acceleration
Driver’s
problems in
maintaining lane
position
Driver’s
problems in
controlling
speed
Abnormal lateral
movements
Patterns of
lateral
acceleration of
vehicles
Abrupt speed
variations
Patterns of
longitudinal
acceleration of
vehicles
Probability
of drunk driving detection goes higher while
the number of observed cues increases
Dong Xuan (CSE/OSU) / 2010
58
59
Our System
Dong Xuan (CSE/OSU) / 2010
59
60
Implementation
 Develop
the prototype system on Android G1 phone with
accelerometer and orientation sensor
 Implement
the prototype in Java, with Eclipse and Android 1.6
SDK
 The
whole prototype system can be divided into five major
components
☆ User interface ☆ System configuration ☆ Monitoring daemon
☆ Data processing ☆ Alert notification
Dong Xuan (CSE/OSU) / 2010
60
61
Evaluation - Testing Data Collection
 Test
72
-
22
-
data
sets of data with simulated drunk driving related behaviors
Weaving, swerving, turning with a wide radius
Changing speed erratically (accelerating or decelerating)
sets of data for regular driving
Each one for 5 to 10 minutes
 Mobile
phone positions in the vehicle
Dong Xuan (CSE/OSU) / 2010
61
62
Evaluation - Detection Performance
 Study
 In
the accuracy of detecting drunk driving related behaviors
terms of false negative and false positive
 Study
performance in the special case, such as the phone slides in the vehicle
during driving
 Slides
has obvious impacts on detection accuracy
 May add additional calibration procedure to solve it (future work)
Dong Xuan (CSE/OSU) / 2010
62
63
Evaluation – Energy Efficiency
 Curves
of battery level states during mobile phone running
Phone runs without drunk driving detection system
Monitoring daemon
of system keeps running, sensing and doing the pattern
matching on the monitoring results
Dong Xuan (CSE/OSU) / 2010
63
64
Related Work
 Driver
vigilance monitoring and driver fatigue prevention
 Monitoring the
visual cues of drivers to detect fatigue in driving
× Installed cameras just in front of drivers are potential safety hazard
 Monitoring
through vehicle-human interface
 Capture
fatigued or drunk driving through monitoring interactions
× Low compatibility, vehicles need to couple with auxiliary add-ons
 Detect
abnormal driving through GPS and acceleration data
 Pattern matching
with GPS and acceleration data
× However, GPS data are not always available
Dong Xuan (CSE/OSU) / 2010
64
65
Remarks
 First
to propose utilizing mobile phones as a platform for
developing active drunk driving detection system
 Design and implement an efficient detection system based on
mobile phone platforms
 Experimental results show our system achieves good detection
performance and power efficiency
 In the future work, to improve the system with additional
calibration procedure and by integrating all available sensing
data on a mobile phone such as camera image
Dong Xuan (CSE/OSU) / 2010
65
Exemplary System V: Stealthy Video
Capturer
Background
 SVC Overview
 Challenges
 Our Approaches
 Experimental Evaluations
 Remarks

Dong Xuan (CSE/OSU) / 2010
66
Background

More and more private information is entrusted to
our friend, the 3G Smartphone, which is getting
more and more powerful in performance and
diversified in functionality.
Dong Xuan (CSE/OSU) / 2010
67
SVC Overview



Almost every 3G Smartphone is equipped with a
camera and the wireless options, such as 3G
networks, BlueTooth, WiFi or IrDA.
These wireless connections are good enough to
handle certain types of video transmission.
We turn 3G Smartphones into an online stealthy
video-recorder.
Dong Xuan (CSE/OSU) / 2010
68
System Architecture
Dong Xuan (CSE/OSU) / 2010
69
Challenges

Stealthily install SVC into 3G Smartphones
 Windows
Hiding
 Infection Method

Collect the video information from 3G
Smartphones
 DirectShow
Controls
 Data Compressing

Send the video file to the SVC intender
 File
Sending
Dong Xuan (CSE/OSU) / 2010
70
Infection Method
To embed SVC in a 3G Smartphone is called a
infection process.
 We employ Trojan horse for downloads as the
infection approach.
 Our experimental SVC is hidden in the game
of ”tic-tac-toe” that we develop in Windows
Mobile environment.

Dong Xuan (CSE/OSU) / 2010
71
The Scenario of Tic-Tac-Toe
Dong Xuan (CSE/OSU) / 2010
72
Triggering Schemes


Triggering Algorithm is designed to determine when
to turn on the video capture process and send the
captured video to make SVC stealthier and get more
useful information.
Three scenarios are under consideration.
 The
first scenario is tracking.
 The second scenario is related with political or business
espionage.
 The third scenario is a hybrid one, where SVC is used for
much diversified everyday purposes.
Dong Xuan (CSE/OSU) / 2010
73
Applications


Suspects tracking
Kids care
Dong Xuan (CSE/OSU) / 2010
74
Kids tracking
Dong Xuan (CSE/OSU) / 2010
75
Implementation
Dong Xuan (CSE/OSU) / 2010
76
Experimental Evaluations:
Power Consumption

Power curve
Dong Xuan (CSE/OSU) / 2010
77
Experimental Evaluations:
CPU and Memory Usage

CPU and Memory
Dong Xuan (CSE/OSU) / 2010
78
Remarks


The initial study exploited from SVC will draw wide
attentions on 3G Smartphone’s privacy protection and
open a new horizon on 3G Smartphones security
research and applications.
We are currently investigating the modeling of smart
spyware from the study of ”spear and shield”.
Dong Xuan (CSE/OSU) / 2010
79
A Summary
Dong Xuan (CSE/OSU) / 2010
80
Future Research Directions

Smartphone-based Systems and Networking
 Mobile
social networking, e-commerce, e-health, safety
monitoring etc.
 Easy to start and exciting but too many competitors, lack of
scientific depth

Smartphone Core Improvement
 Multitasking, power
management, efficient local
communication protocol, accurate localization,
security/privacy protection
 Deep but hard to start
Dong Xuan (CSE/OSU) / 2010
81
Final Remarks
Smartphones have brought significant impacts
to our daily life.
 We present five exemplary systems on mobile
social networking, e-commerce, e-health and
safety.
 Research and development on smartphones
will be hot.

Dong Xuan (CSE/OSU) / 2010
82