Combinatorial Approach to Testing
Mobile Applications:
An Experimental Investigation
Brandi Amstutz
Advisor: Dr. Sergiy Vilkomir
This research is supported by the National Science Foundation REU Site Award 1262933
Introduction
Importance of Research
Mobile testing is a critical step in the development process where
errors in an application can be discovered and corrected to increase
reliability and functionality.
Traditional software testing strategies can be modified and
implemented to solve the challenges associated with mobile device
selection.
The amount of devices required to exhaustively represent today’s
mobile market is extremely high due to a wide range of variability
between available devices.
This investigative research analyzes the efficiency of mobile
testing based on a combinatorial approach to device selection.
Introduction
Research Questions Addressed
Which device characteristics have the potential to
influence application reliability?
Both hardware and software components have the potential to influence an
application’s functionality.
▫ Hardware components considered for this approach include:
 Mobile device type
 Screen resolution
 Hardware manufacturer
▫ Software attributes included:
 Android OS version
 RAM memory
Introduction
Research Questions Addressed
How should device selection be executed to ensure an
optimal set of devices?
This approach suggests parameter value specification based on user profile
data reflecting the current mobile market. Less popular parameter values
can be grouped to reduce the total possible values without compromising
Huawei, 1.2%
proportionality
Pantech, 1.8%
Motorola, 2.4%
Other (tvdpi and
xxhigh), 5.9%
Other,
9.5%
Google, 2.5%
Xiaomi, 2.5%
Low
10%
Amazon, 3.6%
LG, 6.1%
Extra High,
24.6%
Medium
23%
High
36%
Figure 1. Screen Resolution Distribution
HTC, 6.3%
Samsung, 55.9%
Sony,
8.1%
Figure 2. Device Manufacturer Distribution
Source: Unity 3D Mobile Statistics
Introduction
Research Questions Addressed
 How many devices are sufficient for mobile testing?
Exhaustive testing strategies are limited by resource constraints and contain
redundancy between device attributes. This research suggests a set of five
devices selected based on device parameterization and combinatorial testing
techniques for maximum coverage
Materials and Methods: Step One
Device Parameterization
Based on user profile data we suggest to use the following
device characteristics and their parameter values:
Type
Device
Manufacturer
Android OS
Version
Smartphone
HTC
Honeycomb (3.0-3.2)
Tablet
Motorola
Ice Cream Sandwich
(4.0)
Samsung
Jelly Bean (4.1-4.3)
Other
Other
Screen Resolution
Low
less than 426 dp x 320 dp
Medium
at least 470 dp x 320 dp
High
at least 640 dp x480 dp
Extra High
at least 960 dp x 720 dp
Table 1. Selected Parameter Values
RAM
Memory
< 768 MB
≥ 768 MB
Materials and Methods: Step Two
Device Selection
Through Each-Choice coverage techniques and consideration of current market
statistics we suggest the following two sets of selected devices:
Table 2. Selected Device Sets
Based on user profile data the following devices were included due to their high level of popularity:
Set A: Galaxy S2 i9100 and Galaxy Tab 2, 7.0
Set B: Galaxy S2 i9100 and Galaxy Tab 10.1
All other devices have been added to provide total coverage of device characteristics
Materials and Methods: Step Three
Application Selection
Three applications with similar functionality were selected
for testing. The following applications used provide access
to search, book and review travel accommodations ranging
from car rental to discount vacation packages:
Hotwire
Travelocity
BookIt.com
Materials and Methods: Step Four
Device Access
Perfecto Mobile cloud services were used for remote access to devices for
testing purposes. Tests were recorded and screen shots were saved for later
analysis. http://www.perfectomobile.com/
Remote testing procedure using SaaS cloud-based device access:
User
provides
test input
Input is
sent to
devices
selected by
user
Audio &
video of
live test
execution is
recorded
Test results
& video are
sent to user
for analysis
Materials and Methods: Step Four
Device Access
Figure 3. Perfecto Mobile Dashboard
Materials and Methods: Step Five
Additional Devices
For comparison, two sets of random devices equivalent in
size to the Each-Choice sets were selected. The same
testing procedures were executed on the ten additional
devices:
Table 3. Random Device Sets
Results
Screenshot Analysis
Screenshots were analyzed from all sets of devices for fault
identification
Figure 4 (left) illustrates Fault H5 for the
Hotwire application. The expected result
is that the number of rooms should update
on the main page when a user selects a
new number of desired hotel rooms from
the drop down menu. The fault occurs
when the room quantity is not updated
appropriately
Results
Screenshot Analysis
Figure 5 (right) illustrates
fault
T1
for
the
Travelocity.com application.
The expectation is that the
Travelocity menu bar with
their gnome icon is displayed
at the top of the home screen.
The fault occurs when the
menu bar is missing from
view.
Results
Screenshot Analysis
Expected
Fault
Figure 6 (left) illustrates fault B1
of the Bookit.com application.
The expectation is that a menu bar
with current query's information
and reference number is displayed
at the top of the screen. The fault
occurs when this menu bar is
missing from view.
Results
Fault Detection
Fault detection totals are as follows:
Hotwire: 7
 2 device independent
 5 device dependent
Travelocity: 7
 1 device independent
 6 device dependent
Bookit.com: 4
 1 device independent
 3 device dependent
Results
Fault Detection
Table 4. Fault Detection
In 7 out of 12 cases (or 58%) the effectiveness of our approach was greater than the
results of random testing.
In 4 out of 12 cases (or 33%) the effective ness of our approach was equivalent to
the results of random testing.
Conclusion
Conclusions
The quantity of application faults detected by the selected set of
devices was equal to or greater than the quantity detected by the
random set of devices. Additionally, the random set of devices
did not yield a fault that the selected set was not able to identify.
This suggests that the device attributes selected were suitable
based on their expected influence on application reliability.
Combinatorial testing strategies provided sufficient parameter
value coverage with a test suite of five devices. This research
suggests that this small set of devices is effective for mobile
testing while also reasonable enough for resource
constraints.
Questions?