Enhancing Symbolic Execution
with Veritesting
Thanassis Avgerinos, Alexandre Rebert, Sang Kil Cha
and David Brumley
Carnegie Mellon University
ICSE 2014
1
Background
Symbolic Execution
• Use symbols to represent variables
x = y + 1
z = x * 2 + 3
• Concrete execution
y = 1
 z = 7
• Symbolic execution
y = in_y  z = (in_y + 1) * 2 + 3
2
Background
Symbolic Execution (2)
x = input()
if(x > 0)
y = x;
else
y = -x;
z = y;
x = input()
x>0?
T
F
y=x
y = -x
z=y
Test case generation
x > 0  SMT solver  input
3
Background
Symbolic Execution (3)
x = input()
if(x > 0)
y = x;
else
y = -x;
z = y;
x = input()
x>0?
T
F
y=x
y = -x
z=y
Program verification: z = |x|
z = ite(x>0, x, -x)  SMT solver  Valid?
4
Problem
Approaches
• Dynamic symbolic execution (DSE) - testing
– Path-based formulas
– Easy-to-solve
– Hard-to-generate (Path explosion)
• Static symbolic execution (SSE) - verification
– Property-based formulas
– Hard-to-solve (solver blowup)
– Easy-to-generate
• Easy-to-generate & Easy-to-solve ?
5
Method
Veritesting
• Alternates between SSE and DSE.
• Twice as many Bugs
DSE
SSE
DSE
SSE
DSE
• Orders of magnitude
more paths
• Higher code coverage
6
Method
DSE w/o Veritesting
x = input()
S <- Ø
x>0?
T
F
y=x
y = -x
z=y
……………
7
Method
• CFGRecovery
• CFGReduce
• StaticSymbolic
• Finalize
8
Method(1)
CFGRecovery
• Generate a partial CFG
• (S) Symbolic branch
• (E) Any hard-to-handle inst
– ret
– syscall
– unknown
 exit node
9
Method(2)
CFGReduce
• Transition points
– Immediate postdominator
of entry node
– Predecessors of Exit
• Unrolling loops
– Switch to concrete value
– User-defined bound
10
Method(3)
StaticSymbolic
if(x > 1) y = 1; else if(x < 42) y = 17;
11
Method(4)
Finalize
• Create new executor
– For each distinct transition point
• CFG accurate
– Overestimation
– Underestimation
• Incremental Deployment
x = input()
x>0?
T
F
y=x
y = -x
z=y
……………
12
Implementation
MergePoint
13
Evaluation
Evaluation
• Metrics
– Number of bugs
– Node coverage
– Path coverage
• Benchmarks
– GNU coreutils
– BIN suite (1,023 programs)
– Debian packages (33,248 programs)
14
Evaluation (1)
Bug finding
• BIN: 63 + 85
• coreutils: 2 new bugs
• 9 years old, time zone parser in Gnulib
15
Evaluation (2)
Node Coverage
•
• 27% more coverage than S2E on coreutils
•
16
Evaluation (3)
Path Coverage
• Three estimations
– Time to complete test
• 46 programs, 73% faster
– Multiplicity
• For bin, 1.4 x 10290 (average), 1.8 x 1012 (median)
• For coreutils, 1.4 x 10199 (average), 4.4 x 1011 (median)
– Fork rate
• Reduce average by 65%
• Reduce median by 44%
17
Evaluation (4)
Debian benchmark
18
Conclusion
Conclusion
• Veritesting: enhance the DSE with SSE
• MergePonit: infrastructure testing programs
• Large value evaluation and results
19
Discussion
• Why is it faster?
– SSE introduces overhead for formula-solving
– Reduces the number of duplicated paths
 Benefits > cost
– Insight into the SMT solver
• Exploit generation
• Other bugs
20
Thanks
21