Software Talks
Are you listening?
Post Conference edition (05 Dec 2014)
Creative Commons License
How to design your mobile apps by Julian Harty is licensed under a
Creative Commons Attribution-ShareAlike 3.0 Unported License.
http://creativecommons.org/licenses/by-sa/3.0/deed.en_US
Julian Harty
Value
Why bother?
Are you working too hard, on the wrong things?
We struggle to decide what to test,
how much testing is enough, etc.
110% effort doesn’t cut it…
www.eurostarconferences.com
Most of our work is wasted effort
How much of our testing is on target?
www.eurostarconferences.com
Most of our work is wasted effort
How much of our testing is on target?
80% of reported bugs not addressed
Automated Tests 
www.eurostarconferences.com
Know your users
Custom drink feature removed[1]
=> 1 star feedback ratings
Parallel Kingdom[2]
Regular users generate 2.5x daily revenues
Logos © respective owners
www.eurostarconferences.com
[1] example from App Quality book
[2] example from Tale of Two Apps
Understand the effects
• Battery drain varied by 3x for similar hardware specifications
• Custom code added for Kindle Fire to reduce brightness
– 40% less battery drain
• Higher network latencies reduced interactivity by 40%
• Users preferred Wi-Fi
– 69% for Parallel Kingdom, 58% for StudyBlue
• Tablets 2x usage
• Pull-out keyboard also increased usage
www.eurostarconferences.com
Analytics can augment our work
• Help us to correct and improve what we do
• Reduce waste, reduce latency,
• Increase value
How
• Insights into the app’s behaviour in-the-wild
• Feedback loops
www.eurostarconferences.com
Volume
Volume
Network profiling
Time (of day)
Discover
Time (of day)
Use
www.eurostarconferences.com
Volume
Volume
Network profiling
Time (of day)
Time (of day)
Transform
www.eurostarconferences.com
Raw ingredients / context
How’s it all work anyway? Why do I care? What’s
involved? What’s special about mobile apps?
Some history
Web server logs & analysis
Web App
App Logs
Web Server
Web Logs
W3C format
One ecosystem
www.eurostarconferences.com
Graphs
What’s different about mobile?
Mobile
App
App Logs
Common
System Log
•
•
•
•
Logs isolated on the device
Connection not guaranteed
Many more sensors
Much more variation
www.eurostarconferences.com
Data Collector
TOPOLOGY
Mobile Apps sending
Analytics data
Database
Filter(s)
Analytics
WebServer
Overview of Mobile Analytics
Each step may be delayed
www.eurostarconferences.com
Business view
Types of Events
Mobile app
Analytics
Library
Internet
connection
Analytics
Collector
Events
1:1 App-initiated
m:1 App-initiated
Library-initiated
E
E1 … E4
E
E
Ea
Ea
L
L
www.eurostarconferences.com
Analytics
Database
Analytical Questions: Past
Trends,
Defect Reports
Information
Insight
What’s Happened?
(Reporting)
How & why did it happen?
(Factor analysis)
Software quality models,
bottleneck analysis
www.eurostarconferences.com
Analytical Questions: Present
Engineering Activity,
Benchmarking, Testing
Information
Insight
What’s Happened?
(Alerts)
What’s the best / worst
that can
happen?(Modeling /
Simulation)
Specification refinement,
asset reallocation
www.eurostarconferences.com
Analytical Questions: Future
Extrapolation
What will Happen?
(Forecasting)
What’s the best / worst
that can
happen?(Modeling /
Simulation)
Failure prediction models
www.eurostarconferences.com
Information
Insight
Analytics for Software Development
Trends,
Defect Reports
Information
Insight
Engineering Activity,
Benchmarking, Testing
Extrapolation
Past
Present
Future
What’s Happened?
(Reporting)
What’s Happened?
(Alerts)
What will Happen?
(Forecasting)
How and why did it
happen?
What is the next
best action?
(Factor analysis)
(Recommendation)
What’s the best / worst
that can
happen?(Modeling /
Simulation)
Software quality
models,
bottleneck analysis
Specification
refinement,
asset reallocation
http://research.microsoft.com/pubs/136974/foser-2010-buse.pdf
www.eurostarconferences.com
Failure prediction
models
Implementation Flow
Feasible
Practical
www.eurostarconferences.com
Useful
Gaining confidence
Accuracy & precision
Precision & accuracy
Precision: repeatability
Accuracy: on target
https://en.wikipedia.org/wiki/Accuracy_and_precision
www.eurostarconferences.com
Precision & accuracy
• Add images here:
precision
• And here: accuracy
www.eurostarconferences.com
Divergent answers increase doubt
• A tale of two three mobile analytics libraries
(and what happens when bonuses are on the line…)
• Where were the testers (part one)?
www.eurostarconferences.com
The Dark Side
Of Mobile Analytics
DO NO HARM TO A NEIGHBOUR
Bad stuff happened;
• Location data collected
• Excessive traffic
Where were the testers (part two)?
A Study of Third-Party Tracking by Mobile Apps in the Wild
http://www.vam.ac.uk/users/node/1777
ftp://ftp.cs.washington.edu/tr/2012/03/UW-CSE-12-03-01.PDF
www.eurostarconferences.com
Reducing precision to protect privacy
10km x 10km squares
From: Capturing Mobile Experience in the Wild: A Tale of Two Apps
Figure © ACM
www.eurostarconferences.com
Beware the automation bias
“When presented with
an automated solution
40% of pilots reasoned
less or none at all”
“Automation bias occurs in decision-making because
humans have a tendency to disregard or not search
for contradictory information in light of a computergenerated solution that is accepted as correct and
can be exacerbated in time critical domains.”
Automation Bias in Intelligent Time Critical Decision Support Systems
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.2634&rep=rep1&type=pdf
www.eurostarconferences.com
Necessary but not sufficient
Listening is a means to an end
Complementary Feedback
Polychrome, richer tones while avoiding cacophony
•
•
•
•
•
•
System logs
Crash logs
App Store ratings
In-App feedback
Speaking to humans.
Oh, and software testing 
www.eurostarconferences.com
Developer Console (Google Play)
www.eurostarconferences.com
Instrument the Ecosystem
Application
Analytics
Change Me!
Ecosystem
www.eurostarconferences.com
Observe the Behaviours
Application
Ecosystem
Keys & Gestures
Network Traffic
www.eurostarconferences.com
Correlation? Causation? Appropriate?
From: User Interaction-based Profiling System for Android Application Tuning
Figure © ACM
www.eurostarconferences.com
Breaking-up is hard to do
From one thing, to another
Divorce can be messy, even for software
Changing the code is the easy part…
•What about the data?
•And the systems & processes that rely on the data?
Act in Haste, Repent at Leisure
www.eurostarconferences.com
Two ears to listen
Are you willing to try?
Willing to try?
• Become “one” with the data [Rob Lambert]
• Instrumenting their code, need to learn how to
understand and use the data [Rob Lambert]
• Bringing DevOps to Mobile Apps?
• Internet-of-Things is coming [Andy Stanford-Clark]
• Be aware of what can go wrong [Isabel Evans]
www.eurostarconferences.com
Further reading
These books available at:
[1,2] http://wip.org/
[3] http://www.appqualitybook.com/
[4] http://www.howtomeasureanything.com/
www.eurostarconferences.com
Academic References
Capturing Mobile Experience in the Wild: A Tale of Two Apps
http://doi.acm.org/10.1145/2535372.2535391
User Interaction-based Profiling System for Android Application Tuning
http://doi.acm.org/10.1145/2632048.2636091
Automation Bias in Intelligent Time Critical Decision Support Systems
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.91.2634&rep=rep1&type=pdf
A Study of Third-Party Tracking by Mobile Apps in the Wild
ftp://ftp.cs.washington.edu/tr/2012/03/UW-CSE-12-03-01.PDF
All the papers are freely available online, e.g. using Google Scholar
www.eurostarconferences.com
Q&A Now?
Later: julianharty@gmail.com
Creative Commons License
How to design your mobile apps by Julian Harty is licensed under a
Creative Commons Attribution-ShareAlike 3.0 Unported License.
http://creativecommons.org/licenses/by-sa/3.0/deed.en_US