Automated Concolic Testing
of Smartphone Apps
Saswat Anand
Mayur Naik
Stanford Univ.
Georgia Tech.
Hongseok Yang Mary Jean Harrold
Univ. of Oxford
Georgia Tech.
Motivation
Motivation
Problems with Smartphone Apps
Problem
Automatically generate test inputs
for bounded exhaustive testing of
smartphone apps
Test Inputs for Apps
•
•
•
Whole-program testing
Test input is a sequence of events e1, e2…,
en
Types of events: a tap on the screen,
change in geo-location, arrival of a SMS
message, etc.
Bounded Exhaustive Testing of Apps
S, the set of all
event sequences*
s.t. each sequence
takes a unique path
Goal: cover
these
Set of
covered
branches
*of bounded-length
Two subproblems
1. Generate individual events
2. Generate sequences of events
Generating Individual Events
•
An event is associated with data
X & Y coordinates of a tap event
o geo-location of a change-in-geo-location event
o content of an incoming SMS event
o etc.
o
•
Data determine which program path is taken
Challenge: Generate the “right” data
for events
Example: Music Player App
Play
Pause
Rewind
Skip
Stop
Eject
Example: Music Player App
tap(136, 351)
public void onClick(View target) {
if (target == play)
startService(new Intent(ACTION_PLAY));
else if (target == pause)
startService(new Intent(ACTION_PAUSE));
else if (target == skip)
startService(new Intent(ACTION_SKIP));
else if (target == rewind)
startService(new Intent(ACTION_REWIND));
else if (target == stop)
startService(new Intent(ACTION_STOP));
else if (target == eject)
showUrlDialog();
}
Example: Music Player App
tap(248, 351)
public void onClick(View target) {
if (target == play)
startService(new Intent(ACTION_PLAY));
else if (target == pause)
startService(new Intent(ACTION_PAUSE));
else if (target == skip)
startService(new Intent(ACTION_SKIP));
else if (target == rewind)
startService(new Intent(ACTION_REWIND));
else if (target == stop)
startService(new Intent(ACTION_STOP));
else if (target == eject)
showUrlDialog();
}
Example: Music Player App
tap(360, 351)
public void onClick(View target) {
if (target == play)
startService(new Intent(ACTION_PLAY));
else if (target == pause)
startService(new Intent(ACTION_PAUSE));
else if (target == skip)
startService(new Intent(ACTION_SKIP));
else if (target == rewind)
startService(new Intent(ACTION_REWIND));
else if (target == stop)
startService(new Intent(ACTION_STOP));
else if (target == eject)
showUrlDialog();
}
Example: Music Player App
tap(24, 351)
public void onClick(View target) {
if (target == play)
startService(new Intent(ACTION_PLAY));
else if (target == pause)
startService(new Intent(ACTION_PAUSE));
else if (target == skip)
startService(new Intent(ACTION_SKIP));
else if (target == rewind)
startService(new Intent(ACTION_REWIND));
else if (target == stop)
startService(new Intent(ACTION_STOP));
else if (target == eject)
showUrlDialog();
}
Example: Music Player App
tap(136, 493)
public void onClick(View target) {
if (target == play)
startService(new Intent(ACTION_PLAY));
else if (target == pause)
startService(new Intent(ACTION_PAUSE));
else if (target == skip)
startService(new Intent(ACTION_SKIP));
else if (target == rewind)
startService(new Intent(ACTION_REWIND));
else if (target == stop)
startService(new Intent(ACTION_STOP));
else if (target == eject)
showUrlDialog();
}
Example: Music Player App
tap(305, 544)
public void onClick(View target) {
if (target == play)
startService(new Intent(ACTION_PLAY));
else if (target == pause)
startService(new Intent(ACTION_PAUSE));
else if (target == skip)
startService(new Intent(ACTION_SKIP));
else if (target == rewind)
startService(new Intent(ACTION_REWIND));
else if (target == stop)
startService(new Intent(ACTION_STOP));
else if (target == eject)
showUrlDialog();
}
Generating Individual Events
Existing alternatives
• Random Testing
oCannot perform systematic/exhaustive testing
• Platform-specific tools (e.g., hierarchy
viewer in Android)
oLimited to GUI Events
oCannot handle third-party GUI widgets
Generating Individual Events
Our solution
Use concolic execution to generate data
associated with events
Generating Individual Tap Events
1
2
3
4
5
6
7
tap(int x, int y){
if (x>2 && x<4){
if (y>1 && y<3)
W1_clicked();
else
W2_clicked();
}else
W3_clicked();
}
x>2 && x<4
T
y>1 && y<3
1
2
T
3
F
5
F
7
Generating Individual Tap Events
x>2 && x<4
T
y>1 && y<3
1
2
T
3
tap(1, 5)
F
7
F
5
Generating Individual Tap Events
x>2 && x<4
T
y>1 && y<3
1
2
T
3
F
7
F
5
(x>2 && x<4)
tap(1, 5)
F1 !(x>2 && x<4)
W3_clicked()
tap(3, 5)
Generating Individual Tap Events
x>2 && x<4
T
y>1 && y<3
1
2
T
3
tap(1, 5)
F
7
F
5
(x>2 && x<4)
(x>2 && x<4)
(y>1 && y<3)
tap(3, 5)
tap(3, 2)
T1 (x>2 && x<4)
F2 !(y>1 && y<3)
W2_clicked()
Generating Individual Tap Events
x>2 && x<4
T
y>1 && y<3
1
2
T
3
tap(1, 5)
F
7
F
5
(x>2 && x<4)
(y>1 && y<3)
tap(3, 5)
tap(3, 2)
T1
T2
(x>2 && x<4)
(y>1 && y<3)
W1_clicked()
Example: Music Player App
❖
❖
❖
❖
❖
❖
❖
❖
❖
❖
❖
Two subproblems
1. Generate individual events
2. Generate sequences of events
Generating Sequences of Events
Concatenate individual events generated
by concolic execution.
Baseline Algorithm
Baseline
algorithm
Goal: cover
these
S, Set of all event
sequences s.t. each
sequence takes a
unique path
Set of
covered
branches
Baseline Algorithm Suffers from Path Explosion
25000
20000
15000
10000
5000
0
1
2
3
4
Number of sequences generated
for Music Player app by baseline algorithm
ACTEve Algorithm
ACTEve: Automated Concolic
Testing of Event-driven programs
ACTEve Algorithm
Baseline
algorithm
S, Set of all event
sequences s.t. each
sequence takes a
unique path
ACTEve
algorithm
R s.t. R ⊆ S
Goal: cover
these
Set of
covered
branches
ACTEve is
relatively sound
Path Subsumption
Program state in concolic execution
Maps memory
location to
values
(symbolic or
concrete)
< γ, C >
Path
constraint
Path Subsumption
Note
γ - memory map
C – path constraint
Program entry
Path 𝑝1
< γ1 , 𝐶1 >
𝑝1 subsumes 𝑝2
1. 𝐶2 ⇒ 𝐶1
2. 𝛾1 = 𝛾2
Path 𝑝2
< γ2 , 𝐶2 >
Path Subsumption
Note
γ - memory map
C – path constraint
Program entry
Path 𝑝1
< γ1 , 𝐶1 >
Path 𝑝2
< γ2 , 𝐶2 >
𝑝1 subsumes 𝑝2
1.
𝐶2 ⇒generate
𝐶1
- Don’t
test corresponding to
2.
𝛾
=
𝛾
1
2
any path that
is an extension of 𝑝
2
- Only generate tests corresponding to
paths that are extension of 𝑝1
Path Subsumption
• Checking path subsumption is very
expensive in general
o Constraint implication check
o Matching memory map
• But, path subsumption can be checked
cheaply in special cases
o Read-only events
o Events whose mutual ordering does not matter
o etc.
Read-only Events
Program
Entry
event 𝑒 is does not
𝑛
𝑞 corresponds to
write to any memory
𝑒1 , … , 𝑒𝑛−1
location.
corresponds to 𝑒𝑛
Path 𝑝 executed for event
sequence 𝑒1 , … , 𝑒𝑛
𝑝 is subsumed by q
Read-only Events
❖
❖
❖
❖
❖
❖
❖
❖
❖
❖
❖
Read-only events are represented as ❖
ACTEve System Architecture
Empirical Study
• Apply ACTEve and baseline algorithms
o event sequences of length up to 4
o 16 concurrently running emulators
o time budget of 12 hours
• Measured three metrics
o running time
o number of feasible paths
o number of satisfiability checks
Empirical Results
Future Work
Widget
Explosion
Main Contributions
1. Concolic execution to generate individual
events
2. ACTEve: an efficient algorithm for
bounded exhaustive testing of eventdriven programs
o Requires only a small fraction (5-36%) of time
compared to baseline algorithm
3. Implementation for Android
Backup slides
Read-only Events
Program
Entry
′ ⇒𝐶
1.
𝐶
∧
𝐶
1
1
corresponds to
′
2.
𝛾
=
𝛾
because
𝑝
1
2
event sequence
𝑒1 , … , 𝑒𝑛−1
< γ1, 𝐶1 >
corresponds to 𝑒𝑛
in 𝑒1 , … , 𝑒𝑛−1 , 𝑒𝑛
< γ2, 𝐶1 ∧ 𝐶 ′ >
Path 𝑝 executed for input
event sequence 𝑒1 , … , 𝑒𝑛
does not write to any
memory location.
A Solution: Use Platform-specific Knowlege
Output of Android’s “Hierarchy Viewer” tool
A Solution: Use Platform-specific Knowlege
Output of Android’s “Hierarchy Viewer” tool
void onTouchEvent(MotionEvent e) {
int rawX = (int) e.getX();
int rawY = (int) e.getY();
int x = (rawX – MARGIN) / SIZE;
int y = (rawY – MARGIN) / SIZE;
if (x >= 0 && x < 3 && y >= 0 & y < 3) {
int cell = x + 3 * y;
…
}
Path Subsumption
Program
Entry
Program
Entry
same program location
Path 𝑝1
{𝑝′ 𝑠. 𝑡. 𝑝1 ; 𝑝′ is feasible}
Covered
branches
Path 𝑝2
{𝑝′ 𝑠. 𝑡. 𝑝2 ; 𝑝′ is feasible}
Covered
branches
Path Subsumption
Program
Entry
Program
Entry
same program location
Path 𝑝1
′
′
{𝑝 𝑠. 𝑡. 𝑝1 ; 𝑝 is feasible}
Covered
branches
Path 𝑝2
if we′ explore
all paths
′
𝑠. 𝑡. 𝑝2 ; 𝑝 is feasible}
that{𝑝 extends
𝑝1 , then no
need to explore any
path that extends 𝑝2
because no additional
branch coverage will be
obtained. Covered
branches
Example: Music Player App
Path constraint when PAUSE
button is tapped on