Today
 Executions, paths, and states
 Testing via model checking
1
Executions, Paths, and States
Control flow graph (CFG):
structure of a program’s code
An execution is simply a
run of the program – the result
of some set of commands
Some executions, with different operations:
path is the
a particular
way
AAstate
program’s
location
the
CFG – ignoring
+through
memory
contents
at a point in
the commands – where the
time
program “goes”
(in the executions, color only shows
the location in the CFG, not the
contents of memory, which may be
different even if colors are the same)
Same state, different executions,
different
Same
path, paths
different executions
Same paths, different states
2
A Question
 Testing can focus on exploring lots of
•
Executions
•
•
Paths
•
•
Random testing
Concolic testing
States?
•
•
Model checking
But first: does this gain us anything?
3
A Question
 First question: does it make the problem
more manageable
•
That is, are there cases where exploring all
states is more feasible than exploring all
paths?
4
Paths vs. States
int G = 0;
int foo (int x, int y) {
if (x < y) 2
G++;
if (x == 10)
4
G++;
int main () {
foo(x, y);
bar(x, y);
if (x + y == 40)
G++;
32
}
}
int bar (int x, int y) {
if (x < 50)
8
G++;
if (y > 8)
16
G++;
Once more, how many paths through this
program?
}
5
Paths vs. States
int G = 0;
int foo (int x, int y) {
if (x < y)
G++;
if (x == 10)
G++;
int main () {
foo(x, y);
bar(x, y);
if (x + y == 40)
G++;
}
}
int bar (int x, int y) {
if (x < 50)
G++;
if (y > 8)
G++;
What if we consider the number of states?
If we include x and y’s values in the state,
the number is much larger than the number
of paths… so it’s certainly not always an
improvement!
}
6
Paths vs. States
pick(chunk, 0, 10);
pick(nbytes, 0, 10);
Less clear which is better in this
case: lots of states, but also lots
of paths
write(chunk, wbuf, nbytes);
pick(chunk, 0, 10);
pick(nbytes, 0, 10);
Paths lets us consider inputs that
produce the same sequence of writes
as “the same”
read(chunk, rbuf, nbytes);
pick(chunk, 0, 10);
pick(nbytes, 0, 10);
write(chunk, wbuf, nbytes);
States means we don’t have to worry
about error returns in the past, if that
error return doesn’t change the system
state
7
Paths vs. States
 So, considering states rather than paths
seems to be a trade-off without a clear
winner – often at least as un-manageable
as complete path exploration
 Which is better for error detection?
•
Hard to say, but for a class of errors
(reachability), we can say that complete
state coverage would definitely expose them
8
Model Checking
 A model checker is a tool for exploring a
state space
 Basic idea: generate every reachable
state of a transition system
•
•
•
Think of states as nodes in a graph
Directed edges mean “from this state, this is
a possible next state of the program”
Multiple outgoing edges where there is
input/thread scheduling/other
nondeterminism
9
Model Checking
 Model checking is a huge research field
•
Clarke, Emerson, and Sifakis received this
year’s Turing Award for coming up with the
basic idea
•
Some major subfields:
•
•
Hardware and software
Symbolic vs. explicit-state
10
Model Checking
 Symbolic model checkers “explore” a
state space by manipulating logical
expressions representing many states
 In explicit-state model checking (part of)
the graph is actually enumerated
• Especially useful for software, where
complex data structures are hard to
handle symbolically
• Closer to testing
Gerard’s SPIN is the most popular
explicit-state model checker
11
Model Checking
 We will be looking at one particular kind
of explicit-state model checking
• Using SPIN
• To explore the state spaces of C
programs
 Other model checkers used for this
approach to testing
• Java PathFinder 2 (NASA Ames)
• Bogor (U Kansas/Nebraska)
12
Model Checking: State-Based Testing
 Model-checking by executing the program
• Backtracking search for all states
Will explore, as a side-effect,
many executions and many paths,
but the goal is to explore states
CFG
State already visited!
Backtrack and try a
different operation
Done with test!
State already visited!
Backtrack and try a
Backtrack and try a
different operation
different operation
13
Many Software Model Checkers
BLAST
CRunner SPIN
CBMC
JPF2
SLAM
Bogor
CMC
MAGIC
VeriSoft
14
Two Approaches
Execution of actual code
(dynamic: like testing)
BLAST
CRunner SPIN
CBMC
JPF2
SLAM
Bogor
CMC
MAGIC
Our focus in this class
VeriSoft
Analysis of derived transition system
(static: like a complete analysis)
15
SPIN and Model-Driven Verification
 SPIN 4.0 introduced model-driven
verification
•
•
Can embed C code in a PROMELA model
Code is executed as part of a transition during
depth-first search
 SPIN compiles a PROMELA model into a C
program: it’s a model checker generator
•
•
Embed C code in transitions by executing the
compiled C code – at native speed, with real
compiled behavior (including optimizations)
Take advantage of all SPIN features –
hashing, multicore exploration, etc.
16
SPIN and Model-Driven Verification
 Used to “model check” significant pieces of
implementation code at JPL
 Very useful for checking rich properties that
automated abstraction engines often fail to
handle
•
•
Functional correctness (differential checking
with a reference implementation)
Invariants of rich data structures
 Using SPIN for testing C programs
17
SPIN and Model-Driven Verification
 When SPIN backtracks,
it uses information on
how to restore the state
of the C program:
•
•
•
Tracked memory is
restored on backtrack
Matched memory is also
used to determine if a
state has been visited
before
All other memory is not
restored on backtrack
or stored in state table
Execute C code
until control
returns to SPIN
Push tracked & matched
state on stack
Has state been
visited before?
Y
Backtrack:
pop stack &
restore tracked &
matched state
N
Store matched
state in state
table
18
SPIN and Model-Driven Verification
 Use state that is tracked but not matched to
introduce abstractions
•
•
•
•
•
E.g., symmetry reduction: a property that does
not depend on the order of items in a linked list
Track the original version of the list
Copy and sort the list after each transition
Match on the sorted list only
SPIN will backtrack when it visits a state with
same list contents, even in a different order
19
Testing via Model Checking
 We do not expect to cover all states
• There are far too many (trillions, at least)
 Use abstraction (“consider this state and
that state to be the same”) to limit the
exploration – may miss errors
 Use techniques that have a small
probability of missing some states (hashing
schemes)
20
Testing via Model Checking
 Unsound abstractions:
• For example, in the flight flash file
systems we’ve checked, the state of the
flash device
Used page
Free page
Dirty page
Bad block
live pages? (0-1) x dirty pages (0-1) x block state (bad, free, current)
21
Testing via Model Checking
 Does not guarantee that states (or errors)
won’t be missed, but provides a better
understanding of which states have been
explored, based on the program’s purpose
 Manages state space size
Used page
Free page
Dirty page
Bad block
live pages? (0-1) x dirty pages (0-1) x block state (bad, free, current)
22
Unsound Abstractions
 “Abstraction” can mean a lot of things
 What I’ll be talking about in this class:
•
•
•
One operation: from a concrete state, c, we
can compute an abstract state A(c)
Backtrack according to abstract state matching
Goal: we would like to explore every reachable
abstract state of a C program
23
Unsound Abstractions
 For some abstractions, the goal is reached
if we simply match on abstract state – if
 c1, c2 . A(c1) = A(c2) 
c3 . T(c1, c3) 
c4 . T(c2, c4) 
A(c3) = A(c4)
24
Unsound Abstractions
 Otherwise:
These states abstract the same
So do these
And these
These, however, do not abstract equivalently,
and can only be reached through their parents
(and are the only concrete representatives of
their respective abstract states).
Whichever of these concrete states we explore
first, we will miss one of the children.
25
Unsound Abstractions
 We usually don’t have soundness for any
abstractions that reduce the state-space to
a manageable size
 If not, unsound in our context
(underapproximating abstract successors
by abstracting concrete states)
 That’s ok
• It’s just testing, we don’t expect to be
complete
26
Testing via Model Checking
 Requires the ability to restore a running
program to an earlier execution state
• Difficult engineering problem, handled by
applying automatic code instrumentation
•
•
Rewrite the program to check that the
backtracking works properly
As a side-benefit, can simulate flight
software resets and hardware faults
27
Simple PROMELA Code
int x;
int y;
active proctype main () {
1
This model has 7 states
if
2 :: x = 1
3 :: x = 2
fi;
5
}
7
Start simple
What are they?
State = (PC, x, y)
assert (x == y);
SPIN’s nondeterministic choice construct
Picks any one of the choices that is enabled
How do we guard a choice?
Not mutually
exclusive!
if
:: (x < 10) -> y = 1
:: (x < 5) -> y = 3
:: (x > 1) -> y = 4
fi;
28
Simple PROMELA Code
int x;
int y;
active proctype main () {
1
if
2 :: x = 1
3 :: x = 2
5
fi;
if
7 :: y = 1
9 :: y = 2
fi;
13
if
14:: x > y -> x = y
15:: y > x -> y = x
17:: else -> skip
fi;
assert (x == y);
}
This model has 17 states
What are they?
State = (PC, x, y)
Er…
Don’t worry about state-counting too
much – SPIN has various automatic
reductions and atomicity choices that
can make that difficult
29
Simple PROMELA Code
int x;
active proctype main () {
do
:: (x < 10) -> x++
Only a couple more PROMELA
constructs to learn for building test
harnesses: the do loop
:: break
od
Like if, except it introduces
a loop to the top – break choice
can exit the loop
This nondeterministically assigns x
a value in the range 0…9
30
Simple PROMELA Code
int x;
inline pick (var, MAX)
var = 0;
inline gives us a macro facility
do
:: (var < MAX) -> var++
:: break
As you can imagine, this is a
useful macro for building a
test harness!
od
31
Testing via Model Checking
 Basic idea:
• We’ll write a test harness in PROMELA
• Use SPIN to backtrack and explore all
input choices
• Call C code we’re testing
• Use abstraction to limit the number of
states we consider
•
We can even “trick” SPIN into doing pure
random testing!
32
Testing via Model Checking
 Rest of class:
•
•
We’ll look at the idea of “model-driven verification” that
drives all this
I’ll show how to write a simple PROMELA harness to
test C code
•
Thursday: we’ll investigate automatic code
instrumentation, some novel coverage-based code
strategies, and see how model checking and random
testing compare
•
Download SPIN yourself, and play with it!
•
•
http://www.spinroot.com
Examples available on class website
33