Software Analysis via Data Analysis
Matthias F. Stallmann, 2006/04/04, DIMACS EAA:ISC planning meeting
Based on
joint work
with:
2006/04/04
Franc Brglez
Raleigh, NC, USA
DIMACS EAA:ICS
Xiao Yu Li
Seattle, WA, USA
1
Software versus Algorithms
o
o
Problems are NP-hard; require heuristics or
(worst case) exponential algorithms.
Simple algorithms must be compared with
•
•
o
cplex and other ILP solvers
metaheuristics (SA, GA, “particle swarm”, “ants…”,
etc.) with lots of adjustable parameters
Want “black box” comparison with (in many
cases) no prior understanding of (some of) the
algorithms
2006/04/04
DIMACS EAA:ICS
2
Data Presentations
based on CPLEX runs for (permutations of) a single instance
o
Descriptive Statistics
•
mean / median / stdev
-
600.4 / 25.3 / 1767.3
o
Histogram
o
Percent solved
2006/04/04
DIMACS EAA:ICS
3
Stretching the Truth
or making it clearer?
•
more information here, but we need to look carefully
2006/04/04
DIMACS EAA:ICS
4
A more “normal” distribution
CPLEX with different settings under the same conditions
2006/04/04
DIMACS EAA:ICS
5
uf250..87: QT2/QT1 vs UW2/UW1 (1)
What about random 3-SAT instances?
1
uf250..87-uw1
exp. d.
16.7/17.2
0.8
solvability
0.6
0.4
0.2
0
0.01
0.1
1
10
100
runtime (seconds)
2006/04/04
DIMACS EAA:ICS
6
uf250..87: QT2/QT1 vs UW2/UW1 (2)
UW1 performs the same as QT1 (t-test: t = 1.88 > 1.97)
1
exp. d.
16.7/17.2
uf250..87-uw1
solvability
0.8
uf250..87-qt1
exp. d.
12.3/12.5
0.6
0.4
0.2
0
0.01
2006/04/04
0.1
1
runtime (seconds)
DIMACS EAA:ICS
10
100
7
uf250..87: QT2/QT1 vs UW2/UW1 (3)
UW1 performs the same as QT1 (t-test: t = 1.88 > 1.97)
UW2 outperforms UW1 by a factor of 31 ...
1
exp. d.
16.7/17.2
uf250..87-uw1
solvability
0.8
0.6
uf250..87-qt1
uf250..87_uw2
exp. d.
12.3/12.5
exp. d.
0.39/0.29
0.4
0.2
0
0.01
0.1
1
10
100
runtime (seconds)
2006/04/04
DIMACS EAA:ICS
8
uf250..87: QT2/QT1 vs UW2/UW1 (4)
UW1 performs the same as QT1 (t-test: t = 1.88 > 1.97)
UW2 outperforms UW1 by a factor of 31 ...
QT2 outperforms UW2 slightly (t-test: t = 2.24 > 1.97)
1
exp. d.
16.7/17.2
solvability
uf250..87-uw1
0.8
0.6
0.4
0.2
0
0.01
2006/04/04
uf250..87-qt1
uf250..87_uw2
exp. d.
uf250..87_qt2
exp. d.
12.3/12.5
exp. d.
0.31/0.28
exp. d.
0.39/0.29
exp. d.
0.21/0.19
0.1
1
runtime (seconds)
DIMACS EAA:ICS
10
100
9
Sources of Data Distributions
first, 100 random instances (SAT)
median
4.8
mean
21.2
stdev
42.9
Heavy tail
2006/04/04
DIMACS EAA:ICS
10
Not all instances are equal:
here’s an easy one (128 permutations + original)
median
4.4
mean
6.7
stdev
6.5
Exponential
2006/04/04
DIMACS EAA:ICS
11
…and here’s a harder one
median
84.8
mean
126.7
stdev
117.6
Exponential
2006/04/04
DIMACS EAA:ICS
12
Another wrinkle: stochastic search
first, same seed and 32 permuted instances + original
number of flips
median
42484
mean
62457
stdev
86551
slightly worse than
Exponential
2006/04/04
DIMACS EAA:ICS
13
…versus 33 different seeds; same distribution?
number of flips
median
36637
mean
50185
stdev
83226
slightly worse than
Exponential
2006/04/04
DIMACS EAA:ICS
14
Things get strange when the solver is not completely
stochastic (e.g. B&B with stochastic search)
Bi-modal:
Stochastic search
either finds
optimum at root,
or at first branch.
Lower bound
“finds” optimum
at root.
No randomness
in LB method.
2006/04/04
DIMACS EAA:ICS
15
Lower bound method is extremely sensitive to
input ordering
Heavy tail (and one
instance times out):
Lower bound method
finds optimum at root
or in an early branch
only if input order
is “friendly”.
2006/04/04
DIMACS EAA:ICS
16
Another Data Analysis Application:
Instance Profiling
o
o
sao2.b is very easy to solve (variables easy to distinguish)
e64.b is very difficult (lots of variables occur equally often)
2006/04/04
DIMACS EAA:ICS
17