Tap into Mobile App
Testing
Andreea Haras
11/09/2013
Agenda
•
Introduction
•
Mobile Testing vs. Traditional Web-Based Testing
•
Challenges in Mobile Testing
•
Guidelines & Best Practices
•
Conclusion
•
Q&A
About me
•
Andreea Haras – andreea.haras@gmail.com
•
4+ years experience as software tester
•
Mobile testing enthusiastic
•
Interested in new technology
Why mobile testing?
Why mobile testing?
•
“The mobile market is no longer just communication. It’s internet, music,
video, web, social networks, navigation and everything else... The mobile
ecosystem now embraces anything a consumer wishes to do either on the
move or on a handheld device at home” - Forrester
Why mobile testing?
•
•
“The mobile market is no longer just communication. It’s internet, music,
video, web, social networks, navigation and everything else... The mobile
ecosystem now embraces anything a consumer wishes to do either on the
move or on a handheld device at home” - Forrester
Applications running on mobile devices (e.g., mobile applications running on
smart phones or new generation tablets) are becoming so popular that they
are representing a revolution in the IT sector.
Why mobile testing?
•
•
•
“The mobile market is no longer just communication. It’s internet, music,
video, web, social networks, navigation and everything else... The mobile
ecosystem now embraces anything a consumer wishes to do either on the
move or on a handheld device at home” - Forrester
Applications running on mobile devices (e.g., mobile applications running on
smart phones or new generation tablets) are becoming so popular that they
are representing a revolution in the IT sector.
Over 1.6 million mobile applications have been developed, available in
Apple App Store and Google Play (www.mobithinking.com)
Why mobile testing?
•
•
•
•
“The mobile market is no longer just communication. It’s internet, music,
video, web, social networks, navigation and everything else... The mobile
ecosystem now embraces anything a consumer wishes to do either on the
move or on a handheld device at home” - Forrester
Applications running on mobile devices (e.g., mobile applications running on
smart phones or new generation tablets) are becoming so popular that they
are representing a revolution in the IT sector.
Over 1.6 million mobile applications have been developed, available in
Apple App Store and Google Play (www.mobithinking.com)
Estimates for downloads of apps in 2013 range from 56 to 82 billion
(www.idc.com) and a prediction of 200 billion global downloads in 2017
that will be worth US$63.5 billion (www.portioresearch.com)
Native, Traditional, Hybrid, Oh My!
•
Native Apps: coded with a specific programming language. They are fast, reliable,
and powerful, but are tied to a mobile platform and downloaded from a central portal
by OS (i.e. Apple App Store, Google Play). That means you must duplicate them
using the appropriate programming language in order to target another mobile
platform. Netflix, Pandora, Shazam.
Native, Traditional, Hybrid, Oh My!
•
Native Apps: coded with a specific programming language. They are fast, reliable,
and powerful, but are tied to a mobile platform and downloaded from a central portal
by OS (i.e. Apple App Store, Google Play). That means you must duplicate them
using the appropriate programming language in order to target another mobile
platform. Netflix, Pandora, Shazam.
•
Traditional Mobile Web Apps: mobile web sites designed to match every webenabled phone. They’re accessed through the mobile device’s web browser and do
not need to be downloaded/installed. They re-direct to the site’s M Site. Amazon,
eBay, MSN are good examples.
Native, Traditional, Hybrid, Oh My!
•
Native Apps: coded with a specific programming language. They are fast, reliable,
and powerful, but are tied to a mobile platform and downloaded from a central portal
by OS (i.e. Apple App Store, Google Play). That means you must duplicate them
using the appropriate programming language in order to target another mobile
platform. Netflix, Pandora, Shazam.
•
Traditional Mobile Web Apps: mobile web sites designed to match every webenabled phone. They’re accessed through the mobile device’s web browser and do
not need to be downloaded/installed. They re-direct to the site’s M Site. Amazon,
eBay, MSN are good examples.
•
Hybrid Apps: rely on development frameworks and are written with the same
technology used for websites and mobile web implementations. They are hosted or
run inside a native container on a mobile device. They ensure cross-platform
compatibility and can access the phone’s hardware (camera, GPS, user’s contacts,
etc.). Foursquare, Twitter, Wikipedia and Facebook are good examples.
Pop Quiz
There are seven (yes, seven!) mobile
operating systems that are recognized in
the industry, other than the top 2 (iOS and
Android), can you name one or more of
the other five?
Can you name the differences between
Mobile Testing
and
Traditional Web-Based Testing?
Mobile Testing vs. Traditional Web-Based Testing
•
Integration testing
 Within areas of the Device Hardware
 Battery, light, ringer, sleep, wake, usage, connectivity loss or change in
speeds (“elevator test”), camera
 Integration with other Apps on the device
 Facebook, Twitter, etc.
Mobile Testing vs. Traditional Web-Based Testing
•
Integration testing
 Within areas of the Device Hardware
 Battery, light, ringer, sleep, wake, usage, connectivity loss or change in
speeds (“elevator test”), camera
 Integration with other Apps on the device
 Facebook, Twitter, etc.
•
Interruption Testing
 How the App behaves with user-controlled and automatic interruptions
Mobile Testing vs. Traditional Web-Based Testing
•
Integration testing
 Within areas of the Device Hardware
 Battery, light, ringer, sleep, wake, usage, connectivity loss or change in
speeds (“elevator test”), camera
 Integration with other Apps on the device
 Facebook, Twitter, etc.
•
Interruption Testing
 How the App behaves with user-controlled and automatic interruptions
•
Device-specific Gesture Testing
 Pinching, Zooming, Shaking, Orientation, Eye Scrolling, Eye Pausing,
Tilting, Swiping, Press and Hold, Tapping
Pop quiz
What mobile testing
challenges do you think there
are?
Challenges in Mobile Testing
•
The time frames
•
Variety of Devices & OS Versions




Display density
Different browsers
Network connections
Installation issues
Challenges in Mobile Testing
•
The time frames
•
Variety of Devices & OS Versions




•
Display density
Different browsers
Network connections
Installation issues
Simulating real life scenarios




Call interrupts
Low battery
Alarms
Network issues
More challenges in Mobile Testing
•
Usability
 Accessibility
 Intuitive
 Mobile experience
More challenges in Mobile Testing
•
Usability
 Accessibility
 Intuitive
 Mobile experience
•
Automation
 What to automate?
 Tools
 Effort vs Value
More challenges in Mobile Testing
•
Usability
 Accessibility
 Intuitive
 Mobile experience
•
Automation
 What to automate?
 Tools
 Effort vs Value
•
Non functional testing
 Performance
 Security
 Load testing
Guidelines & Best Practices
•
Determining your Mobile Test Matrix
•
Setting up your Lab: Phones
•
Setting up your Lab: Tablets
•
Checklist
•
Key types of Mobile QA Testing: Manual or Automated Testing?
•
Things to remember
•
Test Automation Tool Selection – Few factors to consider
Determining your Mobile Test Matrix
The most difficult facet of the testing matrix is device fragmentation.
There are nearly 4000 separate Android device models! How do you
make sure that all users get a consistent experience?
Device manufacturer market share (Q1 2013)
Determining your Mobile Test Matrix
Android device fragmentation
Source: BGR
Determining your Mobile Test Matrix
Multiple approaches to build a “Mobile Test Matrix”
•
•
•
•
Group devices based on the hardware and/or software specification and prioritize
them
User base – Build a test matrix based on the device and software version the user
uses
Use statistics from vendor market place (especially useful for native apps like games,
utilities, etc.)
Risk based and functionality based matrix
Determining your Mobile Test Matrix
Multiple approaches to build a “Mobile Test Matrix”
•
•
•
•
Group devices based on the hardware and/or software specification and prioritize
them
User base – Build a test matrix based on the device and software version the user
uses
Use statistics from vendor market place (especially useful for native apps like games,
utilities, etc.)
Risk based and functionality based matrix
Example:
•
•
•
Group 1, priority C: Small devices with a small CPU, RAM and low resolution. Older
software versions and older browsers.
Group 2, priority B: Mid-range devices with an avg. CPU, RAM (<512 MB), good
screen size and resolution. The software is not the latest.
Group 3, priority A: High-end devices with a dual/quad-core CPU, RAM (>512 MB)
and a high screen resolution. Latest software versions.
Setting up your Lab: Phones
•
•
•
•
•
Newest, highest resolution, high density, normal sized phone
running latest version of OS. (Samsung Galaxy S4, iPhone 5S)
Oldest, smallest, low density small sized phone running oldest
supported version of OS. (iPhone 3, HTC Tattoo)
Middle of the range phone matches highest usage, normal size, high
density, normal resolution, highest usage OS (Samsung Galaxy S2)
Phone with Highest Market Share (Samsung Galaxy S3, iPhone 4S)
Budget smartphone - normal size, large screen, mdpi, old OS (Dell
Venue)
Setting up your Lab: Tablets
•
XLarge size high end tablet (10”), xhdpi, latest OS. (Nexus 10)
•
Large size Tablet (5-7”), med density, legacy OS (Kindle Fire)
•
Phablet (Phone/Tablet Hybrid) (5”) xhdpi, large size, latest OS,
strange aspect ratio (LG Intuition, Samsung Note)
•
iOS running iOS 5 or lower (iPad 2)
•
iOS running iOS 6 (iPad mini, iPad 3)
Checklist
Key types of Mobile QA Testing: Manual or Automated
Testing?
Functional testing
Performance testing
Memory Leakage Testing
Interrupt testing
Usability testing
Installation testing
Upgrade testing
Things to remember
•
•
•
•
Start testing early. Use emulators / simulators
Use the app on different carriers and network connections like 3G,
4G, WiFi, or LTE
Combine testing tools and integrate them into a continuous
integration system
When possible, have a group of BETA testers to help with real-world
testing
•
Consider the problem space: go mobile to test mobile
•
Don’t assume that because it works on one device, it will work on all
Test Automation Tool Selection – Few factors to
consider
•
•
•
•
•
•
•
•
•
•
Supported devices
Emulator support
Availability of integrated simulators
Types of connection supported
Multi-device support (this is important to run same test on multiple devices)
Compatibility with existing automation tools
Security - VPN connections, data sharing, etc.
Functionality supported - swipe, drag and drop,
zoom, scrolling, pinching, rotating, etc
Image / text recognition
Debugging and test reporting
Conclusions
•
Despite the challenges in mobile application testing, careful
selection of target devices, using statistical techniques for test
optimization and tools that maximize automation can ensure a cost
effective mobile testing process.
Conclusions
•
•
Despite the challenges in mobile application testing, careful
selection of target devices, using statistical techniques for test
optimization and tools that maximize automation can ensure a cost
effective mobile testing process.
Given the stiff competition in the mobile apps market and variety of
apps available for users to download, it is vital that you thoroughly
test each and every aspect of the app before releasing it into the
market, to ensure that the product meets the quality standard
demanded by users.