Testing Requirements for Mobile
Applications
Dennis Soh
Article authored by: Valéria Lelli Leitão
Dantas, Fabiana Gomes Marinho, Aline
Luiza da Costa, Rossana M. C. Andrade
Background
• Explosion of mobile technology
– 60 million smartphone users in US in 2009
– Expected to grow to 142 million by end of 2011
• iPad, other tablets growing in popularity
Background
• Android, iOS, Palm WebOS, Windows 7 among
mobile OSes
• Each has unique properties, including unique
development environment, programming
language support, etc.
Problems
• Testing software on mobile devices has unique
challenges
• These must be considered early in
development
Key factors
• Mobile Context
– Environment limitations; e.g. screen, (lack of)
keyboard, etc.
• Mobile User
– On-the-go; not sitting at a desk
• Mobile Application
– Different aspects of the device that the software
can interact with; network operator, I/O
interfaces, etc.
Any test plan needs to consider
these factors!
Research methodology
• 2 questionnaires, sent to 40 professionals at
15 companies
– Professionals worked on >1 mobile app, >3
months experience in mobile app dev.
– Questionnaire 1: for developers
– Questionnaire 2: for QA
Proposed Testing Requirements
Testing Requirements
• Development process model must focus on
testing process
– Incremental process most popular; used by 70% of
professionals surveyed
– Test process should consider mobile context; 85%
of professionals not using test process designed
for mobile
Testing Requirements
• Mobile applications must be tested on both
emulator and real devices
Emulator
Mobile Device
Functionality
Device variations (keyboard,
screen size, etc.)
Usability issues, design
Usability issues specific to
device
User behavior
Performance, resources, realworld problems
Testing Requirements
• Test reports must document details about the
test environment (emulator/device, device
model, OS version, etc.)
• Testers must verify that application does not
harm any existing applications
– Interruptions (e.g. someone’s calling you) must be
handled appropriately
Testing Requirements
• Applications must be tested according to their
targeted mobile context limitations
– Memory, CPU, screen size/resolution, power, etc.
• Test process must differentiate between
features to be tested in controlled
environment and features to be tested “in the
field.”
Testing Requirements
• Usability testing specific to mobile
applications must be included in testing phase
– 56% of professionals perform usability testing,
only 18% do so in field
• Mobile user must have a clear sense of the
state of the application.
– What have you already done? What can you do
now? Can you undo actions?
Testing Requirements
• Application must support change in device
orientation
– Portrait/landscape format
• Application must ask for permission before
making connections
– What if user doesn’t have unlimited data?
Lawsuit!
Testing Requirements
• UI layout must be appropriate for mobile
devices
– Minimize scrolling, appropriate contrast in
background/text, appropriate text size, etc.
• Important/irreversible actions should require
confirmation
– Easy for user to make mistakes due to screen
size/imprecise input
Testing Requirements
• Sound (if present) should be useful and be
integrated with device’s sound controls
• Buttons should act in manner appropriate to
the device
– E.g. don’t bind the “exit” or “back” key to positive
actions such as select and “OK”
And the Survey Says…
• No real consensus on the “right” way to test
mobile
• Very limited adoption of automated test tools
(only 10% of professionals surveyed)
Testing the Tests
• Researchers applied test requirements to a
mobile shopping app
• Test requirements were found to be beneficial
in highlighting areas of concern specific to
mobile apps
Conclusion
• Testing for mobile has unique challenges
– Vastly different spread in hardware configs
– Mobility
– Hardware constraints
• Recent proliferation of mobile devices -> lack
of consensus regarding development
– Companies are “winging it”
Conclusion
• Proposed requirements offer a “starting
point”
– No comparison point because this is presumed to
be better than “winging it”
• In future..
– Requirements should be refined and prioritized
– Requirements specific to certain categories of
applications should be defined