HUMAN
TESTING OF
MOBILE APPS
FOR VISTACON 2013 CONFERENCE
25 APRIL 2013
Creative Commons License
How to design your mobile apps by Julian Harty is licensed under a
Creative Commons Attribution-ShareAlike 3.0 Unported License.
http://creativecommons.org/licenses/by-sa/3.0/deed.en_US
Rev: 29 April 2013
JULIAN HARTY
Contact me: julianharty@gmail.com
DRIVE FOR EFFICIENCY
AUTOMATION
STATE OF AFFAIRS
The industry is driving towards automated tests
• ‘Manual testing’ is downgraded
• Agile projects need automated tests to provide fast &
frequent feedback
• Development environments for Mobile Apps (SDKs)
include Test Automation frameworks
AUTOMATED TESTS
Unit Tests
Continuous Builds
UI Test Automation
Scale out to run tests on multiple devices
• Virtual devices, remote devices, local devices
Complex, sometimes fragile, ecosystems
• Challenges: device security models, limitations in
frameworks, connectivity, immature domain, lack of
support for multi-modal interfaces, etc.
HUMAN-CENTRIC TESTING
PEOPLE
NEED
PEOPLE TOO
BEYOND THE REACH OF
AUTOMATED TESTS
Test frameworks focus on code, API, or GUI,
Not:
•
Movement: acceleration, rotation, location
•
Tactile, Haptic, audio Inputs or Outputs
•
Aesthetics: look & feel, transitions
•
User Experience (UX)
•
Highly dynamic updates: video, audio output
•
Complex inputs: Gestures, multi-touch, shaking
•
Light: e.g. camera as an input device
So how do we test these?
QOX
Quality Of Experience (QOX)
From the standpoint of a user’s holistic
experience of using the mobile app.
Essentially subjective
http://en.wikipedia.org/wiki/Quality_of_experience
UX Research, Evaluation, and
Testing: strategies, tips and tools
Mark J. Williams
I’VE GOT FEELINGS
TOO
“An inconsistency between the product and the
document might be surprising, frustrating, or
annoying to that person, but what really matters is
inconsistency between the product and the desire.”
I've Got a Feeling: Emotions in Testing
http://www.stickyminds.com/s.asp?F=S17922_COL_2
6 hats of software testing
• The ‘red’ hat – emotions, intuitions, feelings
Here's an article I wrote a few years ago on the six hats of software testing
http://archive.newsweaver.com/qualtech/newsweaver.ie/qualtech/e_article000666359.html
and if you really have lots of time on your hands, here's a video of a talk I gave on the same topic at
StarWest 2008 http://www.stickyminds.com/Media/Video/Detail.aspx?WebPage=117
SO…
WHAT DO
WE NEED
TO DO?
FAST, EFFICIENT
INTERACTIVE TESTING
Bugfests
Move and rotate the phone
Be mobile
Bugfest: A group of people get together to test in parallel and in
common. Developers on-hand for direct feedback.
LUNCH OUT
TESTING A VIDEO APP
• Around 2 hours
• Around 10 testers
• Test the app while going out for lunch
Examples of what the testers did
• Tourist trip on the top of a double-decker bus
• London Underground
• Out for a walk with sandwiches
ESTABLISH STRONG
FOUNDATIONS
Learn:
• The platforms, the norms, what users expect
• How the platform was designed to be used
• The application lifecycle
• The devices
• Common, and unusual problems
INSIGHTFUL TESTING
RAPID software
•
Heuristics
•
Mnemonics to guide the testing
Session-Based Test Management techniques
•
Charter
•
Time-boxed
•
Simple, lightweight test reporting
•
Debrief
http://moolya.com/blog/2012/04/18/test-mobile-applications-with-cop-who-flung-gun/
EXTENDED
TESTING
EXTENDED TESTING
Remote devices
Remote testing
REMOTE TESTERS
Credit: "Ken Banks, kiwanja.net"
http://www.kiwanja.net/gallery/texting/kiwanja_kenya_texting_17.jpg
http://qz.com/49916/indian-smartphone-users-are-gobbling-up-mobile-data/
http://www.imedicalapps.com/2012/09/researchers-smartphone-spirometer-mobile-app/
CHALLENGES OF
EXTENDED TESTING
Loss of information
• Sensory deprivation
• Missing or incomplete data
Loss of fidelity
• Lags, delays
How do we test rich interactive apps unless
you have the device in your hand?
Loss of control
“In theory, theory and practice are the same.
In practice, they are not.”
A device in your hand is worth 2 in the cloud,
and 100 virtual devices.
MORE
CONTEXT
MORE CONTEXT
Additional considerations
•
Cultural norms & differences
•
Generations
•
What’s important to them?
•
How can we empathize?
Mobile Apps & Platforms
www.guardian.co.uk
Culture?
Culture?
Culture?
International
A South Korean woman using her
mobile phone to buy a product at the
virtual retail shop in Seoul.
GSMA REVEALS FEARS OVER MOBILE PRIVACY
ARE HOLDING BACK THE GROWTH OF MOBILE
APPS IN LATIN AMERICA
http://www.gsma.com/newsroom/gsma-reveals
EXAMPLES OF WHAT
CAN GO WRONG
“What this means is that unlike the iPhone, which is
most certainly the closest competitor on the market
to this phone, the Storm's UI is not custom built for
touch navigation -- touch navigation is added after
the fact. Things which flow naturally on an
iPhone -- flicking through lists, scrolling for a
contact, moving around in a webpage or
looking through photos -- feel inelegant and
uncomfortable on the Storm. There's no inertia to
movement, no assurance that your finger is the
lynchpin to control of the device. The screen is
sensitive enough, surely, but how its software
reacts to those touches makes all the
difference, and here the feeling is that you're
never completely in charge of the phone.”
http://www.engadget.com/2008/11/19/blackberry-storm-review/
Image: http://worldwide.blackberry.com/gettingstarted/storm-9500.png
PRACTICAL OUTCOMES
!
[*]
XOR
[*] Not Exclusive OR
COMBINING
STRENGTHS
Automated Tests
• Fast feedback, technology and developer centric
Interactive Tests
• Exploratory, rich interactions and dynamics
• Remote testing e.g. in other countries
Combined
• Augmented, Cyborg testing e.g. automated preparation, data
collection, etc.
Learn to delegate & learn when to take control
SOME FURTHER
READINGS
Testing and Test
Automation for
Mobile Apps
Julian Harty
Summer 2013
CRC Press
http://www.wipconnector.com/download/GuideToTheParallelUniverse_3rdEdition.pdf
http://www.enough.de/fileadmin/uploads/dev_guide_pdfs/Guide_12thEdition_WEB.pdf
Q&A?
IF YOU WANT THE
DRAFT BOOK
Draft book available for review & feedback
Email me julianharty@gmail.com
• Agree not to share or distribute
• Comment to get the next available draft