Hybrid Concolic Testing
Rupak Majumdar
UC Los Angeles
Koushik Sen
UC Berkeley
Automated Test Generation

Studied since 70’s



King 76, Myers 79
30 years have passed, and yet no effective
solution
What Happened???
Automated Test Generation

Studied since 70’s



King 76, Myers 79
30 years have passed, and yet no effective
solution
What Happened???


Program-analysis techniques were expensive
Automated theorem proving and constraint
solving techniques were not efficient
Automated Test Generation

Studied since 70’s



30 years have passed, and yet no effective
solution
What Happened???



King 76, Myers 79
Program-analysis techniques were expensive
Automated theorem proving and constraint
solving techniques were not efficient
In the recent years we have seen
remarkable progress in static programanalysis and constraint solving

SLAM, BLAST, ESP, Bandera, Saturn, MAGIC
Automated Test Generation

Studied since 70’s



Question:
Can
we
combine
program
30 years have passed, and yet no effective
analysis with classical testing
solution
techniques
to Scale Automated
What
Happened???
Test Generation?
 Program-analysis
techniques were expensive


King 76, Myers 79
Automated theorem proving and constraint
solving techniques were not efficient
In the recent years we have seen
remarkable progress in static programanalysis and constraint solving

SLAM, BLAST, ESP, Bandera, Saturn, MAGIC
Our Approach
Concolic Testing:
1. Combines Dynamic and
Static Program Analysis
2. Exhaustive
3. Fails to scale
+
Random Testing:
1. Fast
2. Non-exhaustive
3. Redundant Executions
and poor coverage
=
Hybrid Concolic Testing
Goals of Test Generation (Simplified)





Generate test inputs
Execute program on generated test inputs
Catch assertion violations
Problem: how to ensure that all reachable
statements are executed
Solution:

Explore all feasible execution paths
Execution of Programs

All Possible Execution
Paths




Binary tree
 Computation tree
Internal node !
conditional statement
execution
Edge ! execution of a
sequence of nonconditional statements
Each path in the tree
represents an
equivalence class of
inputs
NonConditional
Statements
Conditional
Statements
T
F
F
T
T
T
F
F
T
T
F
T
Fuzz (Random) Testing

Random testing








Random Testing [Bird and
Munoz 83]
Fuzz testing Windows NT
[Forrester and Miller 00]
QuickCheck [Claessen &
Hughes 01]
JCrasher [Csallner and
Smaragdakis 04]
RUTE-J [Andrews et al. 06]
Randoop [Pacheco et al. 07]
Very low probability
of reaching an error
Problematic for
complex data
structures
Fuzz (Random) Testing

Random testing








Random Testing [Bird and
Munoz 83]
Fuzz testing Windows NT
[Forrester and Miller 00]
QuickCheck [Claessen &
Hughes 01]
JCrasher [Csallner and
Smaragdakis 04]
RUTE-J [Andrews et al. 06]
Randoop [Pacheco et al. 07]
Example ( ) {
s = readString();
if (s[0]==‘I’ && s[1]==‘C’ &&
s[2]==‘S’ && s[3]==‘E’ &&
s[4]==‘2’ && s[5]==‘0’ &&
s[6]==‘0’ && s[7]==‘7’) {
printf(“Am I here?”);
}
Very low probability
}
of reaching an error
Problematic for
Input domain = {‘0’, ‘2’, ‘7’, ‘C’, ‘E’, ‘I’, ‘S’}
complex data
Probability of reaching printf = 7-8 » 10-7
structures
Fuzz (Random) Testing

Random testing








Random Testing [Bird and
Munoz 83]
Fuzz testing Windows NT
[Forrester and Miller 00]
QuickCheck [Claessen &
Hughes 01]
JCrasher [Csallner and
Smaragdakis 04]
RUTE-J [Andrews et al. 06]
Randoop [Pacheco et al. 07]
Example ( ) {
s = readString();
if (s[0]==‘I’ && s[1]==‘C’ &&
s[2]==‘S’ && s[3]==‘E’ &&
s[4]==‘2’ && s[5]==‘0’ &&
s[6]==‘0’ && s[7]==‘7’) {
printf(“Am I here?”);
}
Very low probability
}
of reaching an error
Problematic for
Input domain = {‘0’, ‘2’, ‘7’, ‘C’, ‘E’, ‘I’, ‘S’}
complex data
Probability of reaching printf = 7-8 » 10-7
structures
Fast and Inexpensive
Concolic Testing

Combine concrete testing (concrete execution)
and symbolic testing (symbolic execution)
[PLDI’05, FSE’05, FASE’06, CAV’06, HVC’06]
Concrete + Symbolic = Concolic
Example
int double (int v) {
}
return 2*v;
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
}
}
}
ERROR;
Example
int double (int v) {
}
return 2*v;
void testme (int x, int y) {
2*y == x
Y
N
z = double (y);
if (z == x) {
x > y+10
N
Y
if (x > y+10) {
}
}
}
ERROR;
ERROR
Concolic Testing Approach
int double (int v) {
}
return 2*v;
Concrete
Execution
Symbolic
Execution
concrete
state
symbolic
state
x = 22, y = 7
x = x0, y = y0
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
}
}
}
ERROR;
path
condition
Concolic Testing Approach
int double (int v) {
}
return 2*v;
Concrete
Execution
concrete
state
Symbolic
Execution
symbolic
state
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
}
}
}
ERROR;
x = 22, y = 7,
z = 14
x = x0, y = y0,
z = 2*y0
path
condition
Concolic Testing Approach
int double (int v) {
}
return 2*v;
Concrete
Execution
concrete
state
Symbolic
Execution
symbolic
state
path
condition
void testme (int x, int y) {
z = double (y);
if (z == x) {
2*y0 != x0
if (x > y+10) {
}
}
}
ERROR;
x = 22, y = 7,
z = 14
x = x0, y = y0,
z = 2*y0
Concolic Testing Approach
int double (int v) {
}
return 2*v;
void testme (int x, int y) {
z = double (y);
Concrete
Execution
concrete
state
Symbolic
Execution
symbolic
state
Solution: x0 = 2, y0 = 1
2*y0 != x0
if (x > y+10) {
}
}
condition
Solve: 2*y0 == x0
if (z == x) {
}
path
ERROR;
x = 22, y = 7,
z = 14
x = x0, y = y0,
z = 2*y0
Concolic Testing Approach
int double (int v) {
}
return 2*v;
Concrete
Execution
concrete
state
Symbolic
Execution
symbolic
state
void testme (int x, int y) {
x = 2, y = 1
z = double (y);
if (z == x) {
if (x > y+10) {
}
}
}
ERROR;
x = x0, y = y0
path
condition
Concolic Testing Approach
int double (int v) {
}
return 2*v;
Concrete
Execution
concrete
state
Symbolic
Execution
symbolic
state
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
}
}
}
ERROR;
x = 2, y = 1,
z=2
x = x0, y = y0,
z = 2*y0
path
condition
Concolic Testing Approach
int double (int v) {
}
return 2*v;
Concrete
Execution
concrete
state
Symbolic
Execution
symbolic
state
path
condition
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
}
}
}
ERROR;
2*y0 == x0
x = 2, y = 1,
z=2
x = x0, y = y0,
z = 2*y0
Concolic Testing Approach
int double (int v) {
}
return 2*v;
Concrete
Execution
concrete
state
Symbolic
Execution
symbolic
state
path
condition
void testme (int x, int y) {
z = double (y);
if (z == x) {
2*y0 == x0
if (x > y+10) {
}
}
}
x0 · y0+10
ERROR;
x = 2, y = 1,
z=2
x = x0, y = y0,
z = 2*y0
Concolic Testing Approach
int double (int v) {
}
return 2*v;
void testme (int x, int y) {
z = double (y);
Concrete
Execution
concrete
state
Symbolic
Execution
symbolic
state
Solution: x0 = 30, y0 = 15
2*y0 == x0
if (x > y+10) {
}
}
condition
Solve: (2*y0 == x0) Æ (x0 > y0 + 10)
if (z == x) {
}
path
x0 · y0+10
ERROR;
x = 2, y = 1,
z=2
x = x0, y = y0,
z = 2*y0
Concolic Testing Approach
int double (int v) {
}
return 2*v;
Concrete
Execution
Symbolic
Execution
concrete
state
symbolic
state
x = 30, y = 15
x = x0, y = y0
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
}
}
}
ERROR;
path
condition
Concolic Testing Approach
int double (int v) {
}
return 2*v;
void testme (int x, int y) {
Concrete
Execution
concrete
state
Symbolic
Execution
symbolic
state
path
condition
Program Error
z = double (y);
if (z == x) {
2*y0 == x0
if (x > y+10) {
}
}
}
ERROR;
x0 > y0+10
x = 30, y = 15
x = x0, y = y0
Explicit Path (not State) Model Checking

Traverse all execution
paths one by one to
detect errors

assertion violations
program crash

uncaught exceptions


combine with valgrind
to discover memory
errors
T
F
F
T
T
T
F
F
T
T
F
T
Explicit Path (not State) Model Checking

Traverse all execution
paths one by one to
detect errors

assertion violations
program crash

uncaught exceptions


combine with valgrind
to discover memory
errors
T
F
F
T
T
T
F
F
T
T
F
T
Explicit Path (not State) Model Checking

Traverse all execution
paths one by one to
detect errors

assertion violations
program crash

uncaught exceptions


combine with valgrind
to discover memory
errors
T
F
F
T
T
T
F
F
T
T
F
T
Explicit Path (not State) Model Checking

Traverse all execution
paths one by one to
detect errors

assertion violations
program crash

uncaught exceptions


combine with valgrind
to discover memory
errors
T
F
F
T
T
T
F
F
T
T
F
T
Explicit Path (not State) Model Checking

Traverse all execution
paths one by one to
detect errors

assertion violations
program crash

uncaught exceptions


combine with valgrind
to discover memory
errors
T
F
F
T
T
T
F
F
T
T
F
T
Explicit Path (not State) Model Checking

Traverse all execution
paths one by one to
detect errors

assertion violations
program crash

uncaught exceptions


combine with valgrind
to discover memory
errors
T
F
F
T
T
T
F
F
T
T
F
T
Limitations

Path Space of a Large Program is Huge
 Path Explosion Problem
Entire Computation
Tree
Limitations

Path Space of a Large Program is Huge
 Path Explosion Problem
Entire Computation
Tree
Explored by
Concolic Testing
Limitations: A Comparative View
Concolic: Broad, shallow
Random: Narrow, deep
Limitations: Example
Example ( ) {
1: state = 0;
2: while(1) {
3:
s = input();
4:
c = input();
5:
if(c==‘:’ && state==0)
state=1;
6:
else if(c==‘\n’ && state==1)
state=2;
7:
else if (s[0]==‘I’ &&
s[1]==‘C’ &&
s[2]==‘S’ &&
s[3]==‘E’ &&
state==2) {
COVER_ME:;
}
}
}
•Want to hit COVER_ME
•input() denotes external input
•Can be hit on an input sequence
s = “ICSE”
c : ‘:’ ‘\n’
Similar code in
•Text editors (vi)
•Parsers (lexer)
•Event-driven
programs (GUI)
Limitations: Example
Example ( ) {
1: state = 0;
2: while(1) {
3:
s = input();
4:
c = input();
5:
if(c==‘:’ && state==0)
state=1;
6:
else if(c==‘\n’ && state==1)
state=2;
7:
else if (s[0]==‘I’ &&
s[1]==‘C’ &&
s[2]==‘S’ &&
s[3]==‘E’ &&
state==2) {
COVER_ME:;
}
}
}
•Pure random testing can get to
state = 2
But difficult to get ‘ICSE’ as a
Sequence
Probability 1/(28)6 » 3X10-15
•Conversely, concolic testing
can generate ‘ICSE’ but explores
many paths to get to state = 2
Hybrid Concolic Testing

Interleave Random Testing and Concolic Testing to
increase coverage
Motivated by similar idea used in VLSI design validation:
Ganai et al. 1999, Ho et al. 2000
Hybrid Concolic Testing

Interleave Random Testing and Concolic Testing to
increase coverage
while (not required coverage) {
while (not saturation)
perform random testing;
Checkpoint;
while (not increase in coverage)
perform concolic testing;
Restore;
}
Hybrid Concolic Testing

Interleave Random Testing and Concolic Testing to
increase coverage
while (not required coverage) {
while (not saturation)
perform random testing;
Checkpoint;
while (not increase in coverage)
perform concolic testing;
Restore;
}
Deep, broad search
Hybrid Search
Hybrid Concolic Testing
Example ( ) {
1: state = 0;
2: while(1) {
3:
s = input();
4:
c = input();
5:
if(c==‘:’ && state==0)
state=1;
6:
else if(c==‘\n’ && state==1)
state=2;
7:
else if (s[0]==‘I’ &&
s[1]==‘C’ &&
s[2]==‘S’ &&
s[3]==‘E’ &&
state==2) {
COVER_ME:;
}
}
}

Random Phase

‘$’, ‘&’, ‘-’, ‘6’, ‘:’, ‘%’, ‘^’,
‘\n’, ‘x’, ‘~’ …



Saturates after many
(~10000) iterations
In less than 1 second
COVER_ME is not
reached
Hybrid Concolic Testing
Example ( ) {
1: state = 0;
2: while(1) {
3:
s = input();
4:
c = input();
5:
if(c==‘:’ && state==0)
state=1;
6:
else if(c==‘\n’ && state==1)
state=2;
7:
else if (s[0]==‘I’ &&
s[1]==‘C’ &&
s[2]==‘S’ &&
s[3]==‘E’ &&
state==2) {
COVER_ME:;
}
}
}

Random Phase

‘$’, ‘&’, ‘-’, ‘6’, ‘:’, ‘%’, ‘^’,
‘\n’, ‘x’, ‘~’ …




Saturates after many
(~10000) iterations
In less than 1 second
COVER_ME is not
reached
Concolic Phase

s[0]=‘I’, s[1]=‘C’,
s[2]=‘S’, s[3]=‘E’

Reaches COVER_ME
Hybrid Concolic Testing


4x more coverage than random testing
2x more coverage than concolic testing
Results
Results: Red Black Tree
test_driver()
RedBlackTree rb = new RedBlackTree();
while(1) {
choice = input();
data = input();
switch(choice) {
case 1: rb.add(data); break;
case 2: rb.remove(data); break;
case 3: rb.find(data); break;
default: rb.add_if_not_member(data); break;
}
}
}
Results
Summary
Concolic
Testing
Random
Testing
Summary
Concolic
Testing
Random
Testing
Hybrid Concolic
Testing
Thank You!