A Case for White-box Testing Using Declarative Specification
Danhua Shao
Sarfraz Khurshid
Dewayne E. Perry
Department of Electrical and Computer Engineering
The University of Texas at Austin,
Austin, TX 78712
{dshao, khurshid, perry}@ece.utexas.edu
Introduction
• White-box testing and black-box testing are two
commonly used and complementary techniques in
automatic testing.
• The specification-based black-box testing, such as
TestEra, is not efficient in generating tests for code
coverage.
• We propose Whispec to extend the specificationbased testing to support white-box testing.
• Key idea: integration of both preconditions and path
conditions with relational logic so that they are
solved together for test generation.
Framework
• In Whispec, the constraints are written in Alloy, a
first-order language based on relations.
• Given the precondition constraint, we first solve it
using the Alloy Analyzer and concretize the
solution into a test input.
• We execute the method on that input and build
path conditions for the resulting execution path.
• After abstraction, the path condition is translated
into new path conditions representing previously
unexplored paths.
Contributions
• Specification-based white-box approach: Under
the relational logic model, conjunct preconditions
with path conditions in test generation.
• Implementation: During the execution of one test
case, Whispec does Breadth First Search to cover
different branches as early as possible.
• Evaluation: Compared with TestEra, Whispec will
generate fewer test cases to achieve the same
branch coverage.
Experiment
• We compared the Whispec with TestEra using a
variety of data structure implementations from the
Java libraries.
• Given the limit on the size of data structures and
path conditions, we run Whispec and TestEra to
generate tests to maximize branch coverage.
• The results show that, under the same branch
coverage, Whispec generates significantly smaller
test suites than TestEra.
10000
• We run the analyzer on a conjunction of the
precondition and a new path condition.
964
148
Precondition
Run Code
148
12
10
55
53
4
6
6
55
12
6
6
3
3
6
3
3
1
Li
nk
ed
Li
Test
Output
148
st
.c
Li
on
nk
ta
ed
in
Li
ts
st
()
Li
.I
nk
n
de
Li
ed
xO
nk
Li
f(
ed
st
)
Li
.R
st
em
.L
ov
as
e(
Tr
tI
)
ee
nd
Ma
ex
pF
Of
ir
()
Tr
st
ee
En
Ma
tr
p.
y(
L
Tr
as
)
ee
tE
Ma
nt
p.
ry
Bi
s
()
u
na
cc
ry
es
Se
so
T
ar
re
r(
Bi
ch
eM
)
na
Tr
ap
ry
ee
.p
Se
.F
ut
ar
ir
()
Bi
ch
st
na
Tr
En
ry
ee
tr
Se
.L
y(
ar
as
)
ch
tE
Tr
nt
ee
ry
.s
()
uc
ce
ss
or
()
Test Driver
148
100
• The iterative execution of Whispec can
systematically enumerate inputs that maximize
code coverage.
Wispec
Concretization
1344
1000
• The solution is concretize into a new test input
and triggers next round test generation.
Alloy
Analyzer
78819
100000
Abstraction
Whispec
Alloy
Instance
Java
Input
Path
Condition
New Path
Condition
TestEra