Strauss:
A Specification Miner
Glenn Ammons
Department of Computer Sciences
University of Wisconsin-Madison
How should programs behave?
• Strauss mines temporal specifications
• for example, always call free after malloc
• Why?
• To debug
• To verify
• To test
• To modify
• To understand
2
How should programs behave?
• Strauss mines temporal specifications
• for example, always call free after malloc
• Why?
• To debug
• To verify
• To test
• To modify
• To understand
• Specs constrain programs
• like structured programming, types, etc.
3
Strauss output
For all calls Y := accept(sock = X):
start
X := socket()
read(sock = Y)
Y := accept(sock = X)
write(sock = Y)
close(sock = Y)
end
Spec. says what programs should do:
•What order of calls?
•What values in calls?
4
Strauss inputs
Interface (e.g., sockets)
Library
Client programs
Traces
...
...
socket(domain
= 2,
...
socket(domain
type
= 1,
proto == 2,
7
:=
socket(domain
= 1,= proto
=
0,type
return
7))
=
2,
type
=
1,
0, return = 7))
...
proto = 0)
...
...
Strauss
5
Strauss inputs
Interface (e.g., sockets)
Library
Client programs
State transition model (STM)
Traces
def socket()
close(def use sock)
def accept(use sock)
read(use sock)
write(use sock)
...
...
socket(domain
= 2,
...
socket(domain
type
= 1,
proto == 2,
7
:=
socket(domain
= 1,= proto
=
0,type
return
7))
=
2,
type
=
1,
0, return = 7))
...
proto = 0)
...
...
Scenario criterion
accept
Strauss
6
My contributions
• A dynamic approach to mining
• analyze communication, not source code
• A tool for mining specifications
• practical: scalable and mostly automatic
• temporal specs may refer to multiple data values
• requires little info about code
• New method for debugging specifications
• Birds of a feather flock together (concept analysis)
• Found specifications and bugs
7
Summary of results
• Mined 17 specs. for X11
• example: XInternAtom should only be called
during initialization, not in the event loop.
• Found 199 bugs
• In widely distributed programs
• wish (Tk), xpdf, xpilot, ...
• Included races, leaks, logic errors, and
performance bugs
• By verifying dynamically (on traces)
8
Related work: obtaining specs.
• By hand
• From programmers
• Types
• Specification languages and annotations
• From analysts, testers
• Abstraction tools (Bandera [Dwyer et al.])
• With specification mining
9
Comparing spec. miners
• Strauss
• Specs: temporal, multiple objects, arbitrary NFAs
• Mining: from traces, local, no code necessary
• Goal: specs. for interfaces
• Other work
• Concurrent work
• FSMs for Java objects [Whaley, Martin, and Lam]
• Simple templates [Metacompilation]
• Loop invariants [ESC/Java]
• Earlier work
• Arithmetic properties [Daikon]
• Cliches [Programmer’s Apprentice]
• Communicating processes [Verisoft]
10
Overview of Strauss
Programs and interface
Instrumented
programs
Run on
test
inputs
Preprocess
Strauss
Tracer
Mine
Scenario
extractor
Training
traces
Unvalidated
specification
Learner
Scenarios
Debug
STM and
scenario
criterion
Cable
Developer’s
judgement
Specification
11
Trace + STM = Semantic trace
State transition model (STM)
1 7 := socket(domain = 2,
type = 1,
def socket()
proto = 0)
close(def use sock)
2 0x100 := malloc(size = 23)
def accept(use sock)
3 8 := accept(sock = 7,
read(use sock)
addr = 0x40,
write(use sock)
addr_len = 0x50)
4 23 := write(sock = 8,
Semantic trace
buf = 0x100,
Vars: v7, v8
len = 23)
5 0 := close(sock = 8)
1 v7 := socket()
...
2 skip
3 v8 := accept(sock = v7)
4 write(sock = v8)
5 v8 := close(sock = v8)
...
12
Trace + STM = Semantic trace
State transition model (STM)
1 7 := socket(domain = 2,
type = 1,
def socket()
proto = 0)
close(def use sock)
2 0x100 := malloc(size = 23)
def accept(use sock)
3 8 := accept(sock = 7,
read(use sock)
addr = 0x40,
write(use sock)
addr_len = 0x50)
4 23 := write(sock = 8,
Semantic trace
buf = 0x100,
Vars: v7, v8
len = 23)
5 0 := close(sock = 8)
1 v7 := socket()
...
2 skip
3 v8 := accept(sock = v7)
4 write(sock = v8)
5 v8 := close(sock = v8)
...
13
Trace + STM = Semantic trace
State transition model (STM)
1 7 := socket(domain = 2,
type = 1,
def socket()
proto = 0)
close(def use sock)
2 0x100 := malloc(size = 23)
def accept(use sock)
3 8 := accept(sock = 7,
read(use sock)
addr = 0x40,
write(use sock)
addr_len = 0x50)
4 23 := write(sock = 8,
Semantic trace
buf = 0x100,
Vars: v7, v8
len = 23)
5 0 := close(sock = 8)
1 v7 := socket()
...
2 skip
3 v8 := accept(sock = v7)
4 write(sock = v8)
5 v8 := close(sock = v8)
...
14
A different STM
1 7 := socket(domain = 2,
type = 1,
proto = 0)
2 0x100 := malloc(size = 23)
3 8 := accept(sock = 7,
addr = 0x40,
addr_len = 0x50)
4 23 := write(sock = 8,
buf = 0x100,
len = 23)
5 0 := close(sock = 8)
...
State transition model (STM)
def malloc()
free(def p)
read(use buf)
write(use buf)
Semantic trace
Var: v0x100
1 skip
2 v0x100 := malloc()
3 skip
4 write(buf = v0x100)
5 skip
...
15
A different STM
1 7 := socket(domain = 2,
type = 1,
proto = 0)
2 0x100 := malloc(size = 23)
3 8 := accept(sock = 7,
addr = 0x40,
addr_len = 0x50)
4 23 := write(sock = 8,
buf = 0x100,
len = 23)
5 0 := close(sock = 8)
...
State transition model (STM)
def malloc()
free(def p)
read(use buf)
write(use buf)
Semantic trace
Var: v0x100
1 skip
2 v0x100 := malloc()
3 skip
4 write(buf = v0x100)
5 skip
...
16
From dependences to scenarios
Pick a seed
Slice forwards
Slice backwards
Chop each pair
17
From dependences to scenarios
Pick a seed
Slice forwards
Slice backwards
Chop each pair
18
From dependences to scenarios
Pick a seed
Slice forwards
Slice backwards
Chop each pair
19
From dependences to scenarios
Pick a seed
Slice forwards
Slice backwards
Chop each pair
20
From dependences to scenarios
Pick a seed
Slice forwards
Slice backwards
Chop each pair
21
Extracting scenarios: example
Vars: v7, v8
1 v7 := socket
2 v8 := accept(sock = v7)
3 write(sock = v8)
4 v8 := close(sock = v8)
5 v7 := close(sock = v7)
22
Extracting scenarios: example
Vars: v7, v8
Pick a seed
1 v7 := socket
2 v8 := accept(sock = v7)
3 write(sock = v8)
4 v8 := close(sock = v8)
5 v7 := close(sock = v7)
23
Extracting scenarios: example
Vars: v7, v8
Pick a seed
Slice forwards
1 v7 := socket
2 v8 := accept(sock = v7)
3 write(sock = v8)
4 v8 := close(sock = v8)
5 v7 := close(sock = v7)
24
Extracting scenarios: example
Vars: v7, v8
1 v7 := socket
Pick a seed
Slice forwards
Slice backwards
2 v8 := accept(sock = v7)
3 write(sock = v8)
4 v8 := close(sock = v8)
5 v7 := close(sock = v7)
25
Extracting scenarios: example
Vars: v7, v8
1 v7 := socket
Pick a seed
Slice forwards
Slice backwards
Chop each pair
2 v8 := accept(sock = v7)
3 write(sock = v8)
4 v8 := close(sock = v8)
5 v7 := close(sock = v7)
26
Overview of Strauss
Programs and interface
Instrumented
programs
Run on
test
inputs
Preprocess
Strauss
Tracer
Mine
Scenario
extractor
Training
traces
Unvalidated
specification
Learner
Scenarios
Debug
STM and
scenario
criterion
Cable
Developer’s
judgement
Specification
27
Overview of learning
Strings
Scenarios
(dep. graphs)
standardize
A
C
E
G
B
Scenario
acceptor (SA)
Scenario
A
C
E
G
B
A
C
E
G
B
standardize
NFA
String
NFA
Learn NFA
Yes or no
Specification = (SA, STM, scenario criterion)
28
Standardizing scenarios
v7 := socket()
v2 := socket()
v8 := accept(sock = v7)
v3 := accept(sock = v2)
write(sock = v8)
v2 := close(sock = v2)
read(sock = v8)
read(sock = v3)
v8 := close(sock = v8)
write(sock = v3)
v7 := close(sock = v7)
v3 := close(sock = v3)
29
Standardizing scenarios
v7 := socket()
v2 := socket()
v8 := accept(sock = v7)
v3 := accept(sock = v2)
v7 := close(sock = v7)
v2 := close(sock = v2)
read(sock = v8)
read(sock = v3)
write(sock = v8)
write(sock = v3)
v8 := close(sock = v8)
v3 := close(sock = v3)
30
Standardizing scenarios
X := socket()
X := socket()
Y := accept(sock = X)
Y := accept(sock = X)
X := close(sock = X)
X:= close(sock = X)
read(sock = Y)
read(sock = Y)
write(sock = Y)
write(sock = Y)
Y := close(sock = Y)
Y := close(sock = Y)
31
Standard scenario = string
X := socket()
A
X := socket()
Y := accept(sock = X)
B
Y := accept(sock = X)
X := close(sock = X)
C
X:= close(sock = X)
read(sock = Y)
D
read(sock = Y)
write(sock = Y)
E
write(sock = Y)
Y := close(sock = Y)
F
Y := close(sock = Y)
32
Overview of learning
Strings
Scenarios
(dep. graphs)
standardize
A
C
E
G
B
Scenario
acceptor (SA)
Scenario
A
C
E
G
B
A
C
E
G
B
standardize
NFA
String
NFA
Learn NFA
Yes or no
Specification = SA + STM + scenario criterion
33
Raman’s PFSA algorithm
1. Build a weighted retrieval tree
2. Merge similar states
34
Raman’s PFSA algorithm
1. Build a weighted retrieval tree
2. Merge similar states
socket
100
accept
5
95
read
25
25
read
read
70
write
write 5
70
write
25
70
25
end
write
write
5
write
5
35
Raman’s PFSA algorithm
1. Build a weighted retrieval tree
2. Merge similar states
socket
100
accept
5
95
read
25
25
read
read
70
write
write 5
70
write
25
70
25
end
write
write
5
write
5
36
Raman’s PFSA algorithm
1. Build a weighted retrieval tree
2. Merge similar states
socket
100
accept
5
95
read
25
25
read
read
25
write 5
70
write
5
70
write
100
end
write
37
Raman’s PFSA algorithm
1. Build a weighted retrieval tree
2. Merge similar states
socket
95
read
25
25
read
read
25
100
accept
5
70
5
write
75
write
100
end
38
Raman’s PFSA algorithm
1. Build a weighted retrieval tree
2. Merge similar states
socket
25
95
read
25
read
25
100
accept
5
70
5
write
75
write
100
end
39
Overview of Strauss
Programs and interface
Instrumented
programs
Run on
test
inputs
Preprocess
Strauss
Tracer
Mine
Scenario
extractor
Training
traces
Unvalidated
specification
Learner
Scenarios
Debug
STM and
scenario
criterion
Cable
Developer’s
judgement
Specification
40
Overview of Debugging
Scenarios
Cable
Classified
scenarios
Learner
A
C
E
G
B
A
C
E
G
B
Specification
A
C
E
G
B
Debugged specification
41
Classifying scenarios
b2
b1
g0 G0 G0
b0 g1 G0
G0
G0
42
Classifying scenarios
b2
b2
b1
b0
b1
g0 G0 G0
b0 g1 G0
G0
G0
g0 G0 G0
g1 G0
G0
G0
43
Classifying scenarios
b2
b1
g0 G0 G0
b0 g1 G0
G0
G0
g0
g1
b2
b1
b0
g0 G0 G0
g1 G0
G0
G0
G0 G0
G0
G0
G0
44
Concept analysis: mammals
4-legged
hairy
cats
yes
yes
dogs
yes
yes
dolphins
gibbons
yes
smart
marine
yes
yes
thumbed
yes
yes
humans
yes
yes
whales
yes
yes
45
Concept analysis: mammals
4-legged
hairy
cats
yes
yes
dogs
yes
yes
dolphins
gibbons
yes
smart
marine
yes
yes
thumbed
yes
yes
humans
yes
yes
whales
yes
yes
46
Cable method: good pets
Good pets:
Bad pets:
47
Cable method: good pets
cats
gibbons
dogs
dolphins
humans
whales
{}
Good pets:
Bad pets:
48
Cable method: good pets
cats
gibbons
dogs
hairy
Good pets:
Bad pets:
49
Cable method: good pets
cats
4-legged
hairy
dogs
Good pets:
Bad pets:
50
Cable method: good pets
cats
4-legged
hairy
dogs
Good pets:
cats, dogs
Bad pets:
51
Cable method: good pets
gibbons
smart
dolphins
humans
whales
Good pets:
cats, dogs
Bad pets:
52
Cable method: good pets
gibbons
smart
dolphins
humans
whales
Good pets:
cats, dogs
Bad pets:
gibbons
dolphins
humans
whales
53
Concept analysis: scenarios
Takes
Takes
Takes
Takes
Takes
transition transition transition transition transition
2
3
4
0
1
Scen. 0
yes
yes
Scen. 1
yes
yes
Scen. 2
Scen. 3
yes
yes
yes
yes
yes
Scen. 4
yes
yes
Scen. 5
yes
yes
54
Cable method: good scenarios
Good scenarios:
Bad scenarios:
55
Cable method: good scenarios
scen.
scen.
scen.
scen.
scen.
scen.
0
1
2
3
4
5
{}
Good scenarios:
Bad scenarios:
56
Cable method: good scenarios
scen. 0
scen. 1
scen. 2
Takes transition 1
Good scenarios:
Bad scenarios:
57
Cable method: good scenarios
scen. 0
Takes transition 0
Takes transition 1
scen. 2
Good scenarios:
Bad scenarios:
58
Cable method: good scenarios
scen. 0
Takes transition 0
Takes transition 1
scen. 2
Good scenarios:
0, 2
Bad scenarios:
59
Cable method: good scenarios
scen. 1
scen. 3
scen. 4
scen. 5
Takes transition 2
Takes transition 3
Takes transition 4
Good scenarios:
0, 2
Bad scenarios:
60
Cable method: good scenarios
scen. 1
scen. 3
scen. 4
scen. 5
Takes transition 2
Takes transition 3
Takes transition 4
Good scenarios:
0, 2
Bad scenarios:
1, 3, 4, 5
61
Experimental results
• Where to find bugs?
• in programs (static verification)?
• or in traces (dynamic verification)?
• Specifications and bugs
62
How Strauss verifies a spec
Traces
...
socket(domain
=
...
2, socket(domain
type = 1,
=
...
proto
= 0, =return
2, socket(domain
type
1,
=
= 7))
proto
= 0, =return
2, type
1,
...= 7))
proto = 0, return
...= 7))
...
Scenarios
(dep. graphs)
extract scenarios
Positive specifications:
Accept trace iff every seed has
a scenario that the SA accepts.
Negative specifications:
Accept trace iff every seed has
no scenario that the SA accepts.
socket(...)
socket(...)
socket(...)
accept(...)
accept(...)
write(...)
read(...) accept(...)
read(...) write(...)
close(...) write(...)
read(...)
close(...)
close(...)
Scenario acceptor
Yes or no
63
Experimental results
Rule
PrsAccelTbl
PrsTransTbl
Quarks
RegionsAlloc
RegionsBig
RmvTimeOut
XFreeGC
XGContextFromGC
XGetSelOwner
XInternAtom
XPutImage
XSaveContext
XSetFont
XSetSelOwner
XtFree
XtOwnSel
XtRealizeProc
STM States Edges Bugs
9
5
4
4
33
3
3
4
3
17
9
8
4
16
4
16
4
3
3
63
8
370
3
10
22
5
3
3
989
20
3
95
4
22
10
1
4
0
93
0
14
16
604
12
3
0
13
44
36
44
6
9
11
42
7 2+2
1519
33
37 44+1
8
7
171
45
23
1
36
0
•90 traces
•72 programs
•7 “book” rules
•10 “graph” rules
64
Bugs found
1.
X11 selection protocol specs.:
•
•
2.
Performance: XInternAtom should not be called
in the event loop
•
3.
42 buggy programs (out of 72); degree varied
XPutImage spec. (more on this later)
•
4.
Found 17 bugs total.
English spec is buggy!
2 different bugs! But, also 2 false positives.
Other bugs:
•
138 total (resource errors)
65
Cable was useful
Specification
PrsAccelTbl
PrsTransTbl
Quarks
RegionsAlloc
RegionsBig
RmvTimeOut
XFreeGC
XGContextFromGC
XGetSelOwner
XInternAtom
XPutImage
XSaveContext
XSetFont
XSetSelOwner
XtFree
XtOwnSel
XtRealizeProc
Cost of labeling
Base
Optimal Expert
18
4
8
6
2
2
16
2
2
20
8
16
540
X
149
4
2
2
20
6
12
50
14
24
4
4
5
20
4
5
12
6
10
184
X
42
56
X
53
12
4
5
224
X
28
14
4
5
76
12
13
Cost =
concepts visited
+ concepts labeled
66
XSetSelectionOwner
English: Pass XSetSelectionOwner the timestamp
from the last event.
(event.time = X) := XNextEvent()
XFilterEvent(event.time = X)
XtDispatchEvent(event.time = X)
(event.time = X) := XCheckWindowEvent()
cb_XtActionProc(event.time = X)
XSetSelectionOwner(time = X)
For all calls XSetSelectionOwner(time = X)
67
XInternAtom
English: Don’t call XInternAtom in the event loop.
cb_XtEventHandler(event.display = X)
cb_XtActionProc(event.display = X)
(event.display = X) := XtAppNextEvent()
(event.display = X) := XNextEvent()
XFilterEvent(event.display = X)
XtCallActionProc(event.display = X)
(event.display = X) := XWindowEvent
XtDispatchEvent(event.display = X)
(event.display = X) := XCheckWindowEvent()
XInternAtom(display = X)
For no calls XInternAtom(display = X)
68
XPutImage
English: The image and GC passed to XPutImage
must have been created on the same display.
Z := XCreateGC(display = X)
Y := XCreateImage(display = X)
Y := XShmCreateImage(display = X)
XPutImage(display = X, image = Y, gc = Z)
For all calls
XPutImage(display = X, image = Y, gc = Z)
69
Conclusions
• Strauss is
• local
• scalable
• dynamic
• Strauss finds
• real specs.
• real bugs
• Does Strauss
• generalize well?
• work for others?
• do too much or too little?
70
Suggestions for future work
• Increasing automation
• Exploiting patterns
• Exploiting labels
• Beyond mining from traces
• Static; hybrids (profile-based)
• More expressive models
• Structured heaps; arithmetic properties
• Other domains
• Program understanding; testing
71
My publications
• On Strauss.
• Debugging temporal specifications with concept
analysis. With Rastislav Bodik, James R. Larus, and
David Mandelin. PLDI03.
• Mining specifications. With Rastislav Bodik and James
R. Larus. POPL02.
• Path profiling
• Improving data-flow analysis with path profiles. With
James R. Larus. PLDI98. Selected for 20 Years of
PLDI (1979-1999).
• Exploiting hardware performance counters with flow
and context sensitive profiling. With Tom Ball and
James R. Larus. PLDI97.
72
End of talk
73
Classifying scenarios
b2
b1
g0 G0 G0
b0 g1 G0
G0
G0
g0
g1
b2
b1
b0
g0 G0 G0
g1 G0
G0
G0
G0 G0
G0
G0
G0
74
Overview of Strauss
Traces and STM
apply STM
Seed pattern
Scenarios
extract scenarios
Trace programs
(with PDGs)
socket(...)
socket(...)
socket(...)
accept(...)
accept(...)
write(...)
read(...) accept(...)
read(...) write(...)
close(...) write(...)
read(...)
close(...)
close(...)
Classify and learn
start
X := socket()
Y := accept(so = X)
close(so = X)
end
read(so = Y)
.
.
write(so = Y)
close(so = Y)
Specification
75
Overview of Strauss
Traces and STM
apply STM
Seed pattern
Scenarios
extract scenarios
Trace programs
(with PDGs)
socket(...)
socket(...)
socket(...)
accept(...)
accept(...)
write(...)
read(...) accept(...)
read(...) write(...)
close(...) write(...)
read(...)
close(...)
close(...)
Classify and learn
start
X := socket()
Y := accept(so = X)
close(so = X)
end
read(so = Y)
.
.
write(so = Y)
close(so = Y)
Specification
76
Strauss inputs
Interface (e.g., sockets)
Library
Client programs
State transition model (STM)
Traces
def socket()
close(def use sock)
def accept(use sock)
read(use sock)
write(use sock)
...
...
socket(domain
= 2,
...
socket(domain
type
= 1,
proto == 2,
7
:=
socket(domain
= 1,= proto
=
0,type
return
7))
=
2,
type
=
1,
0, return = 7))
...
proto = 0)
...
...
Strauss
77