Anatomizing Application Performance
Differences on Smartphones
Junxian Huang , Qiang Xu, Birjodh Tiwana, Z. Morley Mao
University of Michigan
Ming Zhang, Victor Bahl
Microsoft Research
University of Michigan
Electrical Engineering and Computer Science


Smartphones are popular
Various smartphone platforms
 Many applications



High speed network access
3G (UMTS & EVDO)
4G (LTE)
University of Michigan
Electrical Engineering and Computer Science
1

Understanding application performance is important
AT&T
T-Mobile
Verizon
University of Michigan
Electrical Engineering and Computer Science
2

Design tradeoff in smartphone measurement
 User coverage vs. collected information
Liu, MobiCom 2008
Zhong, HotMetrics 2010
More info
Fewer users
Our study
Collected information
Willkomm, DySPAN 2008
User coverage
Less info
More users
Speedtest.net
FCC’s broadband test
University of Michigan
Electrical Engineering and Computer Science
3

Our contributions



3GTest
Widely deployed
Provide new insights into cellular network performance





A systematic methodology to understand application performance
Isolate individual factors
Enable fair comparisons
Identify performance bottlenecks
Offer concrete recommendations for improvements
University of Michigan
Electrical Engineering and Computer Science
4

Outline



Cellular network performance study
3GTest design
3GTest results



Web performance analysis
Experimental methodology & setup
Web performance results
 Streaming video & VoIP performance study
 Conclusion
University of Michigan
Electrical Engineering and Computer Science
5

Key challenges of 3GTest





Adoptability
User privacy
Running time
Ease of use
Energy consumption


Data usefulness
Metrics relevant to network applications
University of Michigan
Electrical Engineering and Computer Science
6

How to make 3GTest easily adoptable?
 Avoid personal identifiable information http://www.eecs.umich.edu/3gtest
University of Michigan
Electrical Engineering and Computer Science
7

What information is most relevant for 3GTest?
Web server
Local DNS server
Smartphone
University of Michigan
Electrical Engineering and Computer Science
Ping to
1 st hop
8
End-end ping

66K+ users 148K+ runs 84 carriers 147 countries over 10 months
University of Michigan
Electrical Engineering and Computer Science
9

Throughput comparison
T-
Mobile:600 kbps
CDF
AT&T
1Mbps
University of Michigan
Electrical Engineering and Computer Science
10

RTT comparison
CDF
AT&T
350ms
University of Michigan
Electrical Engineering and Computer Science
11
T-Mobile
450ms

Retransmission rate comparison
AT&T
CDF
T-Mobile
University of Michigan
Electrical Engineering and Computer Science
12

Insights from 3GTest results


Local DNS server performance is good
200 ms comparable to ping to 1 st hop


Carrier network delay dominates end-to-end RTT
End-end ping close to ping to 1 st hop


Time of day effect
AT&T’s downlink throughput
 Signal strength correlates with TCP performance
University of Michigan
Electrical Engineering and Computer Science
13

Smartphone web browsing is complicated
Local DNS server
1
DNS lookup Content customization
Client configuration Server configuration
HTML rendering &
JavaScript execution
5
Server processing delay
3
Network condition
Smartphone
2
TCP handshake
University of Michigan
Electrical Engineering and Computer Science
14
4
TCP data transfer Web server

Key challenges of web performance analysis





Multiple factors have impact
Network condition
Client settings
Server configurations
Content differences




Lack of low-level information and access
No visibility into link layer
Difficult to instrument client OS and application
No access to public web servers
University of Michigan
Electrical Engineering and Computer Science
15

How to isolate impact of individual factor?





Vary one factor at a time
Network condition
 Simulate cellular network with WiFi
Client settings
 Control TCP concurrency
Server configurations
 Change compression mode and server parallelism
Content difference
 Host static website snapshots locally
University of Michigan
Electrical Engineering and Computer Science
16

How to identify performance bottlenecks?





TCP flow analysis technique to break down page load time
DNS lookup time
TCP handshake time
TCP idle time
TCP transfer time index.html
logo.png
Time
University of Michigan
Electrical Engineering and Computer Science
17

Controlled experiment setup
 Platform
 Web browsing
 Other applications
University of Michigan
Electrical Engineering and Computer Science
18

Simple website page load time break down
TCP transfer time TCP idle time
University of Michigan
Electrical Engineering and Computer Science http://m.ebay.com
19

Content-rich website page load time break down
TCP transfer time TCP idle time
University of Michigan
Electrical Engineering and Computer Science http://mapquest.com
20

JavaScript execution comparison
University of Michigan
Electrical Engineering and Computer Science
21

Lessons from application performance study







Performance bottlenecks of web browsing
Client execution
Data download
Higher concurrency significantly improves performance
Compression is usually helpful



Content customization is beneficial
Data URL scheme
Minification
Minimal redirections

Video & VoIP
Platform specific application behaviors
University of Michigan
Electrical Engineering and Computer Science
22

Conclusion



3GTest for measuring cellular network performance
Deployed widely
Gain insights into cellular network performance




A systematic methodology to understand application performance
Isolate impact of individual factor
Identify performance bottlenecks
Offer recommendations on performance improvement
University of Michigan
Electrical Engineering and Computer Science
23

Visit 3GTest website http://www.eecs.umich.edu/3gtest iPhone
University of Michigan
Electrical Engineering and Computer Science
Android
24
Windows Mobile