Master Class on
Experimental Study of Algorithms
Scientific Use of Experimentation
Carla P. Gomes
Cornell University
CPAIOR
Bologna , Italy
2010
Big Picture of Topics Covered in this talk
Part I
• Understanding computational complexity beyond worst-case
complexity
– Benchmarks: The role of Random Distributions Random SAT
– Typical Case Analysis vs. Worst Case Complexity analysis – phase
transition phenomena
• Part II
• Understanding runtime distributions of complete search
methods
– Heavy and Fat-Tailed Phenomena in combinatorial search and Restart
strategies
• Understanding tractable sub-structure
– Backdoors and Tractable sub-structure
– Formal Models of Heavy-tails and Backdoors
– Performance of current state-of-the art solvers on real-world structured
problems exploiting backdoors
II - Understanding runtime distributions of complete
search methods
Outline
• Complete randomized backtrack search methods
• Runtime distributions of complete randomized backtrack
search methods
Complete Randomized Backtrack search methods
Exact / Complete Backtrack Methods
Main Underlying (Search) Mechanisms in:
Mathematical Programming (MP)
Constraint Programming (CP)
Satisfiability
 Backtrack Search;
Branch & Bound;
Branch & Cut;
Branch & Price;
Davis-Putnam-Logemann-Lovelan Proc.(DPLL)
…
1
44
x1 = 0
44
2
3
4
44
x3 = 0
43 88
44
x2 = 1
x2 = 0
x2 = 1
x2 = 0
42
x1 = 1
5
44
x3 = 1
43 99
x3 = 0
10
43 10
6
44 7
x3 = 1
11
43 11
x3 = 0
44 12
44 14
maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6
subject to 5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14
xj binary for j = 1 to 6
44 16
38 18
x3 = 1
- 13
15 -
17 -
19
-
Backtrack Search - Satisfiability
( a OR
NOT b
OR
NOT c ) AND ( b OR NOT c) AND ( a OR c)
State-of-the-art complete solvers are based on backtrack search procedures
(typically with unit-propagation, learning, randomization, restarts);
Randomization in Complete
Backtrack Search Methods
Motivation: Randomization in Local Search
The use of randomization has been very successful in the
area of local search or meta heuristics.
Simulated annealing
Genetic algorithms
Tabu Search
Gsat, Walksat and variants.
Limitation: inherent incomplete nature of local search
methods – cannot prove optimality or inconsistency.
Randomized Backtrack Search
What if the we introduce an element of randomness
into a complete backtrack search method?
Goal:
Explore the addition of a stochastic element into
a systematic search procedure without
losing completeness.
Randomized Backtrack Search
Several ways of introducing randomness into a backtrack search
method:
simple way  randomly breaking ties in variable and/or
value selection.
general framework  imposing a probability distribution for
value/value selection or other search parameters;
Compare with standard lexicographic tie-breaking.
Note: with simple book-keeping we can maintain the completeness
of the backtrack search method;
Notes on Randomizing Backtrack Search
Lots of opportunities to introduce randomization  basically
at different decisions points of backtrack search:
– Variable/value selection
– Look-ahead / look-back procedures
– E.g.:
• When and how to perform domain
reduction/propagation
• What cuts to add;
– Target backtrack points
– Restarts
Not necessarily tie breaking only  more generally we can define a
probability distribution over the set of possible choices at a given decision point
Notes on Randomizing Backtrack Search (cont).
• Can we replay a “randomized” run?  yes since we use pseudo random
numbers; if we save the “seed”, we can then repeat the run with the
same seed;
• “Deterministic randomization” (Wolfram 2002) – the behavior of some
very complex deterministic systems is so unpredictable that it actually
appears to be random (e.g., adding nogoods or cutting constraints
between restarts  used in the satisfiability community)
• What if we cannot randomized the code?
Randomize the input –
Randomly rename the variables
(Motwani and Raghavan 95)
(Walsh (99) applied this technique to study
the runtime distributions of graph-coloring
using a deterministic algorithm based on
DSATUR implemented by Trick)
Walsh 99
Runtime Distributions of Complete Randomized
Backtrack search methods
Backtrack Search
Two Different Executions
( a OR
NOT b
OR
NOT c ) AND ( b OR NOT c) AND ( a OR c)
Size of Search Trees in Backtrack Search
• The size of the search tree varies dramatically ,
• depending on the order in which we pick the
variables to branch on
•  Important to choose good heuristics for
variable/value selection;
Runtime distributions of Complete
Randomized Backtrack search methods
When solving instances of a combinatorial problem
such as the Satisfiability problem or an Integer Program
using a complete randomized search method such as
backtrack search or branch and bound
- the run time of the randomized backtrack search method,
running on single individual instances
(i.e.,several runs of the same complete randomized procedure on
the same instance) exhibits very high variance.
Randomized Backtrack Search
Latin Square
(Order 4)
Time:
11
(*) no solution found - reached cutoff: 2000
30
(*)
(*)
Erratic Behavior of Sample Mean
3500!
sample
mean
2000
Median = 1!
500
number of runs
Heavy-Tailed Distributions
… infinite variance … infinite mean
Introduced by Pareto in the 1920’s
--- “probabilistic curiosity.”
Mandelbrot established the use of heavy-tailed distributions to
model real-world fractal phenomena.
Examples: stock-market, earth-quakes, weather,...
The Pervasiveness of Heavy-Tailed Phenomena in
Economics. Science, Engineering, and Computation
Annual meeting (2005).b
Tsunami 2004
Blackout of
August 15th 2003
> 50 Million People Affected
Financial Markets
with huge crashes
Backtrack
search
… there are
a few billionaires
Power Law Decay
Exponential Decay
Standard Distribution
(finite mean & variance)
Decay of Heavy-tailed Distributions
Standard --- Exponential Decay
e.g. Normal:
Pr[ X  x] Ce  x 2, for some C  0, x 1
Heavy-Tailed --- Power Law Decay
e.g. Pareto-Levy:
Pr[ X  x] Cx  , x  0
Normal, Cauchy, and Levy
Cauchy -Power law Decay
Levy -Power law Decay
Normal - Exponential Decay
Tail Probabilities
(Standard Normal, Cauchy, Levy)
c
Normal
0
0.5
1
0.1587
2
0.0228
3
0.001347
4 0.00003167
Cauchy Levy
0.5
1
0.25
0.6827
0.1476
0.5205
0.1024
0.4363
0.078
0.3829
Fat tailed distributions

4
Kurtosis =
2
2
Normal distribution 
kurtosis is 3
Fat tailed distribution 
when kurtosis > 3
(e.g., exponential, lognormal)

2
second central moment (i.e., variance)

4
fourth central moment
Fat and Heavy-tailed distributions
Exponential decay for
standard distributions, e.g. Normal, Logonormal,
exponential:
Normal
Pr[ X  x]  Ce x2,
for some C  0
Heavy-Tailed
Power Law Decay
e.g. Pareto-Levy:
Pr[ X  x] Cx  , x  0
How to Visually Check
for Heavy-Tailed Behavior
Log-log plot of tail of distribution
exhibits linear behavior.
How to Check for “Heavy Tails”?
Log-Log plot of tail of distribution
should be approximately linear.
Slope gives value of
 1

infinite mean and infinite variance
1 2 infinite variance
Pareto =1
Lognormal 1,1
Lognormal(1,1)
Pareto(1)
f(x)
X
Infinite mean and infinite variance.
Survival Function:
Pareto and Lognormal
Example of Heavy Tailed Model
Random Walk:
Start at position 0
Toss a fair coin:
with each head take a step up (+1)
with each tail take a step down (-1)
X --- number of steps the random walk takes
to return to position 0.
Zero crossing
Long periods without
zero crossing
The record of 10,000 tosses of an ideal coin
(Feller)
1-F(x)
Unsolved fraction
Heavy-tails vs. Non-Heavy-Tails
50%
Random Walk
Median=2
Normal
(2,1000000)
O,1%>200000
Normal
(2,1)
2
X - number of steps the walk takes to return to zero (log scale)
(1-F(x))(log)
Unsolved fraction
Heavy-Tailed Behavior in
Quasigroup Completion Problem Domain
  0.153
  0.319
18%
unsolved
  0.466
 1 => Infinite mean
Number backtracks (log)
0.002%
unsolved
To Be or Not To Be
Heavy-Tailed
Gomes, Fernandez, Selman, Bessiere – CP 04
(1-F(x))(log)
Unsolved fraction
Heavy-Tailed Behavior in
Quasigroup Completion Problem Domain
  0.153
  0.319
18%
unsolved
  0.466
 1 => Infinite mean
Number backtracks (log)
0.002%
unsolved
Research Questions:
Concrete CSP Models
Complete Randomized Backtrack Search
1. Can we provide a characterization of heavy-tailed
behavior: when it occurs and it does not occur?
2. Can we identify different tail regimes across
different constrainedness regions?
3. Can we get further insights into the tail regime by
analyzing the concrete search trees produced by the
backtrack search method?
Scope of Study
•
•
•
•
•
Random Binary CSP Models
Encodings of CSP Models
Randomized Backtrack Search Algorithms
Search Trees
Statistical Tail Regimes Across Constrainedness
Regions
– Empirical Results
– Theoretical Model
Binary Constraint Networks
• A finite binary constraint network
P = (X, D,C)
– a set of n variables X = {x1, x2, …, xn}
– For each variable, set of finite domains
D = { D(x1), D(x2), …, D(xn)}
– A set C of binary constraints between pairs of variables;
a constraint Cij, on the ordered set of variables (xi, xj) is a
subset of the Cartesian product D(xi) x D(xj) that specifies the
allowed combinations of values for the variables xi and xj.
– Solution to the constraint network
instantiation of the variables such that all constraints are satisfied.
Random Binary CSP Models
Model B < N, D, c, t >
N – number of variables; D – size of the domains;
c – number of constrained pairs of variables;
p1 – proportion of binary constraints included in network ;
c = p1 N ( N-1)/ 2;
t – tightness of constraints;
p2 - proportion of forbidden tuples; t = p2 D2
Model E <N, D, p>
N – number of variables; D – size of the domains:
p – proportion of forbidden pairs (out of D2N ( N-1)/ 2)
(Gent et al 1996)
N – from 15 to 50;
(Achlioptas et al 2000)
(Xu and Li 2000)
Computational Cost (Mean)
% of solvable instances
Typical Case Analysis:
Beyond NP-Completeness
Phase Transition
Phenomenon:
Discriminating
“easy” vs.
“hard”
instances
Constrainedness
Hogg et al 96
Encodings
• Direct CSP Binary Encoding
• Satisfiability Encoding (direct encoding)
Backtrack Search Algorithms
• Look-ahead performed::
– no look-ahead (simple backtracking BT);
– removal of values directly inconsistent with the last instantiation
performed (forward-checking FC);
– arc consistency and propagation (maintaining arc consistency, MAC).
• Different heuristics for variable selection (the next variable to instantiate):
– Random (random);
– variables pre-ordered by decreasing degree in the constraint graph (deg);
– smallest domain first, ties broken by decreasing degree (dom+deg)
• Different heuristics for variable value selection:
– Random
– Lexicographic
• For the SAT encodings we used the simplified Davis-Putnam-LogemannLoveland procedure: Variable/Value static and random
Inconsistent Subtrees
Distributions
• Runtime distributions of the backtrack
search algorithms;
• Distribution of the depth of the
inconsistency trees found during the search;
All runs were performed without censorship.
Main Results
1 - Runtime distributions
2 – Inconsistent Sub-tree Depth
Distributions
Dramatically different statistical
regimes across the constrainedness
regions of CSP models;
Runtime distributions
Distribution of Depth of
Inconsistent Subtrees
Depth of Inconsistent Search Tree vs.
Runtime Distributions
Other Models and More Sophisticated
Consistency Techniques
BT
MAC
Model B
Heavy-tailed and non-heavy-tailed regions.
As the “sophistication” of the algorithm increases the heavy-tailed
region extends to the right, getting closer to the phase transition
SAT encoding:
DPLL
To Be or Not To Be Heavy-tailed:
Summary of Results
1 As constrainedness increases change
from heavy-tailed to a non-heavy-tailed
regime
Both models (B and E), CSP and SAT
encodings, for the different backtrack
search strategies:
To Be or Not To Be Heavy-tailed:
Summary of Results
2 Threshold from the heavy-tailed to non-heavytailed regime
– Dependent on the particular search procedure;
– As the efficiency of the search method increases, the
extension of the heavy-tailed region increases: the
heavy-tailed threshold gets closer to the phase
transition.
To Be or Not To Be Heavy-tailed:
Summary of Results
3 Distribution of the depth of inconsistent search sub-trees
Exponentially distributed inconsistent sub-tree depth
(ISTD) combined with exponential growth of the search
space as the tree depth increases implies heavy-tailed
runtime distributions.
As the ISTD distributions move away from the exponential
distribution, the runtime distributions become non-heavytailed.
Theoretical model fits data nicely!
Theoretical Model
Depth of Inconsistent Search Tree vs.
Runtime Distributions
Theoretical Model
X – search cost (runtime);
ISTD – depth of an inconsistent sub-tree;
Pistd [ISTD = N]– probability of finding an inconsistent sub-tree of
depth N during search;
P[X>x | ISTD=N] – probability of the search cost being larger x,
given an inconsistent tree of depth N
Depth of Inconsistent Search Tree vs.
Runtime Distributions:Theoretical Model
See paper for proof
details
Regressions for B1, B2, K
Regression for B1 and B2
Regression for k
Validation:
Theoretical Model vs. Runtime Data
α= 0.27 using runtime data;
α= 0.26  using the model;
Exploiting Heavy-Tailed Behavior:
Restarts
Fat and Heavy Tailed behavior has been observed in several
domains:
Quasigroup Completion Problems;
Graph Coloring;
Planning;
Scheduling;
Circuit synthesis;
Decoding, etc.
How to avoid the long runs?
Use restarts or parallel / interleaved runs to
exploit the extreme variance performance.
Restarts provably eliminate
heavy-tailed behavior.
(Gomes et al. 97,98,2000)
Restarts
1-F(x)
Unsolved fraction
no restarts
70%
unsolved
restart every 4 backtracks
0.001%
unsolved
250 (62 restarts)
Number backtracks (log)
1000000
100000
log ( backtracks )
Number backtracks (log)
Example of Rapid Restart Speedup
(planning)
100000
~10 restarts
10000
~100 restarts
2000
1000
1
20
10
100
1000
log( cutoff )
Cutoff (log)
10000
100000
1000000
Sketch of proof of elimination of heavy tails
X  number of backtracks to solve the problem
pm  Pr[ X  m]
Let’s truncate the search procedure after m backtracks.
Probability of solving problem with truncated version:
Run the truncated procedure and restart it repeatedly.
Y totalnumberbacktrackswithrestarts
Number of Re starts  Y / m ~ Geometric( pm)




F  Pr[Y  y]  (1 pm)












Y /m
 c1ec2 y
Y - does not have Heavy Tails
Current state art sat solvers
use restarts!!!
Restart Strategies
• Restart with increasing cutoff - e.g., used by the
Satisfiability community; cutoff increases linearly:
• Randomized backtracking – (Lynce et al 2001) 
randomizes the target decision points when backtracking
(several variants)
• Random jumping (Zhang 2002)  the solver randomly
jumps to unexplored portions of the search space; jumping
decisions are based on analyzing the ratio between the
space searched vs. the remaining search space; solved
several open problems in combinatorics;
• Geometric restarts – (Walsh 99) – cutoff is increased
geometrically;
• Learning restart strategies – (Kautz et al 2001 and Ruan et.
al 2002) – results on optimal policies for restarts under
particular scenarios. Huge area for further research.
• Universal restart strategies (Luby et al 93) – seminal paper
on optimal restart strategies for Las Vegas algorithms
(theoretical paper)
III - Understanding Tractable Sub-Structure in
Combinatorial Problems
Backdoors
Defying NP-Completeness
Current state of the art complete or exact solvers can handle
very large problem instances of hard combinatorial :
 We are dealing with formidable search spaces of
exponential size --- to prove optimality we have to
implicitly search the entire search ;
 the problems we are able to solve are much larger than
would predict given that such problems are in general NP
complete or harder
Example – a random unsat 3-SAT formula in
the phase transition region with over 1000
variables cannot be solved while real-world
sat and unsat instances with over 100,000
variables are solved in a few minutes.
A “real world” example
Bounded Model Checking instance:
i.e. ((not x1) or x7)
and ((not x1) or x6)
and … etc.
10 pages later:
…
(x177 or x169 or x161 or x153 …
or x17 or x9 or x1 or (not x185))
clauses / constraints are getting more interesting…
4000 pages later:
!!!
a 59-cnf
clause…
…
Finally, 15,000 pages later:
What makes this possible?
Note that:
… !!!
The Chaff SAT solver (Princeton) solves
this instance in less than one minute.
Inference and Search
–
• Inference at each node of search tree:
– MIP uses LP relaxations and cutting planes;
– CP and SAT - domain reduction constraint propagation
and no-good learning.
• Search
Different search enhancements in terms of variable and value
selection strategies, probing, randomization etc, while
guaranteeing the completeness of the search procedure.
Tractable Problem
Sub-structure
Real World Problems are also characterized by
Hidden tractable substructure in real-world
problems.
Can we make this more precise?
We consider particular structures we call
backdoors.
Backdoors
Backdoors: intuitions
Real World Problems are characterized
by Hidden Tractable Substructure
BACKDOORS
Subset of “critical” variables such
that once assigned a value the instance simplifies to a
tractable class.
Explain how a solver can get “lucky” and solve
very large instances
Backdoors to tractability
Informally:
A backdoor to a given problem is a subset of its
variables such that, once assigned values, the remaining
instance simplifies to a tractable class (not necessarily
syntactically defined).
Formally:
We define notion of a “sub-solver” (handles tractable
substructure of problem instance)
Backdoors and strong backdoors
Defining a sub-solver
Note on Definition of Sub-solver
•Definition is general enough to encompass any polynomial
time propagation methods used by state of the art solvers:
–Unit propagation
–Arc consistency
–ALLDIFF
–Linear programming
–…
–Any polynomial time solver
• Definition is also general to include even polytime solvers
for which there does not exist a clean syntactical
characterization of the tractable subclass.
•Applies to CSP, SAT, MIP, etc
Defining backdoors
Given a combinatorial problem C
Backdoors (for satisfiable instances):
Strong backdoors (apply to satisfiable or inconsistent instances):
Example: Cycle-cutset
• Given an undirected graph, a cycle cutset is a subset of nodes in
the graph whose removal results in a graph without cycles
• Once the cycle-cutset variables are instantiated, the remaining
problem is a tree  solvable in polynomial time using arc
consistency;
• A constraint graph whose graph has a cycle-cutset of size c can
be solved in time of O((n-c) k (c+2) )
• Important: verifying that a set of nodes is a cutset (or a bcuteset) can be done in polynomial time (in number of nodes).
(Dechter 93)
B
Clique of size k  cutset of size k-2;
Cutset variable
Backdoors
•Can be viewed as a generalization of cutsets;
•Backdoors use a general notion of tractability based on a polytime
sub-solver --- backdoors do not require a syntactic characterization
of tractability.
•Backdoors factor in the semantics of the constraints wrt sub-solver and
values of the variables;
•Backdoors apply to different representations, including different
semantics for graphs, e.g., network flows --- CSP, SAT, MIP, etc;
Note: Cutsets and W-cutsets – tractability based solely on the structure of the constraint graph,
independently of the semantics of the constraints; (Dechter 93)
Backdoors --- “seeing is believing”
Logistics_b.cnf planning formula.
843 vars, 7,301 clauses, approx min backdoor 16
Logistics.b.cnf after setting 5 backdoor vars
(result after propagation; large cutsets);
After setting just 12 (out of 800+) backdoor vars – problem almost solved.
Inductive inference problem --- ii16a1.cnf. 1650 vars, 19,368 clauses.
Backdoor size 40.
After setting 6 backdoor vars.
Some other intermediate stages:
After setting 38 (out of 1600+)
backdoor vars:
So: Real-world structure
hidden in the network.
Related to small-world
networks etc.
Backdoors:
How the concept came about
Backdoors –
The notion came about from an abstract formal model built to explain the high
variance in performance of current state-of-the-art solvers in particular heavy-tailed
behavior and in our quest to understand the behavior of real solvers (propagation
mechanisms, “sub-solvers” are key);
Emphasis not so much on proving that a set of variables is a backdoor
(or that it's easy to find), but rather on the fact that if we have a (small) set
of variables that is a backdoor set, then, once the variables are assigned a
value, the polytime solver will solve the resulting formula it in polytime.
Surprisingly, real solvers are very good at finding small backdoors!
Backdoors:
Quick detection of inconsistencies
• Detecting inconsistencies quickly --- in logical reasoning the
ability to detect global inconsistency based on local
information is very important, in particular in backtrack
search (global solution);
• Tractable substructure helps in recognizing quickly global
inconsistency --- backdoors exploit the existence of substructures that are sufficient to proof global inconsistency
properties;
• How does this help in solving sat instances? By combining it
with backtrack search, as we start setting variables the subsolver quickly recognizes inconsistencies and backtracks.
Formal Models:
On the connections between backdoors and
heavy-tailedness
Fat and Heavy-tailed distributions
Explain very long runs of complete solvers;
But also imply the existence of a wide range
of solution times, often from very short runs to
very long
How to explain short runs?
Backdoors
Formal Model Yielding
Heavy-Tailed Behavior
T - the number of leaf nodes visited up to and including
the successful node; b - branching factor
P[T bi ] (1 p) pi i  0
Trade-off: exponential decay in making wrong branching
decisions with exponential growth in cost of mistakes.
(inspired by work in information theory, Berlekamp et al.
1972)
p –probability of not finding the backdoor
1 backdoor
b=2
(Gomes 01; Chen, Gomes, and Selman 01)
Expected Run Time
p 1
b
E[T ] 
(infinite expected time)
Variance
p  1 V [T ]
b2
(infinite variance)
Tail
p  1 P[T  L ] C L    2
2
b
p –probability of not finding the backdoor
(heavy-tailed)
More than 1 backdoor
(Williams, Gomes, Selman 03)
Backdoors provide detailed formal model for heavy-tailed search behavior.
Can formally relate size of backdoor and strength of heuristics (captured
by its failure probability to identify backdoor variables) to occurrence
of heavy tails in backtrack search.
Backdoors in real-world problems instances
Backdoors can be surprisingly small:
Backdoors explain how a solver can get “lucky” on certain
runs, when the backdoors are identified early on in the
search.
(large cutsets)
Synthetic Plannnig Domains
Synthetic domains, carefully crafted families of
formulas:
•as simple as possible enabling a full rigorous
analysis
•rich enough to provide insights into real-world
domains
Research questions – the relationship between
problem structure, semantics of backdoors,
backdoors size, and problem hardness.
Hoffmann, Gomes, Selman 2005
Synthetic Planning Domains
Three Synthetic Domains:
Structured Pigeon Hole (SPHnk);
Synthetic Logistics Map Domain (MAPnk);
Synthetic Blocks World (SBWnk);
 backdoor set O(n)
 backdoor set O(log n)
 backdoor set O(log n)
Each family is characterized by size (n) and a structure parameter (k);
Focus
DPLL – unit propagation;
Strong backdoors (for proving unsatisfiability)
Hoffmann, Gomes, Selman 2005
L113
(...)
L21
Ln1
L10
…
L11
MAP813
Number of Variables O(n2)
L17
backdoor set O(log n)
L16
backdoor set O(n2)
lim n AsymRatio  0
…
Number of Variables O(n2)
Cutset (n2)
L13
L12
lim n AsymRatio  1
Cutset (n2)
L11
Note: the topology of the constraint graphs is
identical for both cases. Size of cutset is of same
order for both cases.
L21
L10
Semantics of Backdoors
• Consider G the set of
goals in the planning
problem; let’s define:
AsymRatio 
max gG cos t (G)
cos t (G)
AsymRatio (0,1]
Intuition – if there is a sub-goal that requires more
resources than the other sub-goals
main reason for unsatisfiability
the larger the ratio the easier it is to detect inconsistency
Hoffmann, Gomes, Selman 2005
Asym Ratio – “Rovers” Domain
(Simplified version of a NASA space application)
As asymRatio increases, the hardness decreases
(Conjecture - Smaller backdoors)
Similar results for other domains: Depots, Driverlog, Freecell,Zenotravel
MAP-6-7.cnf infeasible planning instances. Strong backdoor of size 3.
392 vars, 2,578 clauses.
Map 5 Top: running without backdoor
Map 5 Top: running with “a” backdoor
(size 9 – not minimum)
Map 5 Top: running with minimum backdoor
(size 3)
Graph after setting 2 backdoor variables
After setting
three1backdoor
Graph
afterInitial
setting
backdoorvariables
variable
Graph
In this graph one single
variable is enough
to proof inconsistency
(with unit propagation)
Map 5 Top: running with backdoor
(minimum – size 3)
Initial Graph
After setting two backdoors
After setting one backdoor
After setting three backdoors
Exploiting Backdoors
Williams, Gomes, Selman 03/04
Algorithms
We cover three kinds of strategies for dealing with
backdoors:
A complete deterministic algorithm
A complete randomized algorithm
Provably better performance over the deterministic
one
A heuristicly guided complete randomized algorithm
Assumes existence of a good heuristic for choosing
variables to branch on
We believe this is close to what happens in practice
Deterministic
Generalized Iterative Deepening
Generalized Iterative Deepening
x1 = 0
x1 = 1
x2 = 0
(…)
xn = 0
xn = 1
All possible trees of depth 1
x2 = 1
Generalized Iterative Deepening
Level 2
x1 = 0
x2 = 0
x2 = 1
x1 = 1
x2 = 0
All possible trees of depth 2
x2 = 1
Generalized Iterative Deepening
Level 2
xn-1 = 0
xn = 0
xn= 1
Xn-1 = 1
xn = 0
All possible trees of depth 2
Level 3, level 4, and so on …
xn = 1
Randomized
Generalized Iterative Deepening
Assumption:
There exists a backdoor whose size is bounded by a function of n (call
it B(n))
Idea:
Repeatedly choose random subsets of variables that are slightly larger than
B(n), searching these subsets for the backdoor
Deterministic Versus Randomized
Suppose variables have 2 possible values
(e.g. SAT)
For B(n) = n/k, algorithm
runtime is cn
c
Deterministic strategy
Randomized
strategy
k
Det. algorithm outperforms
brute-force search for k > 4.2
Complete Randomized
Depth First Search with Heuristic
Assume we have the following.
DFS, a generic depth first search randomized
backtrack search solver with:
• (polytime) sub-solver A
• Heuristic H that (randomly) chooses variables to branch on, in
polynomial time
 H has probability 1/h of choosing a
backdoor variable (h is a fixed constant)
Call this ensemble (DFS, H, A)
Polytime Restart Strategy for
(DFS, H, A)
Essentially:
If there is a small backdoor, then (DFS, H, A)
has a restart strategy that runs in polytime.
Runtime Table for Algorithms
DFS,H,A
B(n) = upper bound on the size of a backdoor, given n variables
When the backdoor is a constant fraction of n, there is an
exponential improvement between the randomized and
deterministic algorithm
Exploiting Structure using Randomization:
Summary
Over the past few years, randomization has become
a powerful tool to boost performance of complete (
exact ) solvers;
Very exciting new research area with successful
stories 
E.g., state of the art complete Sat solvers use
randomization.
Very effective when combined with no-good
learning
Exploiting Randomization in Backtrack Search:
Summary
•Stochastic search methods (complete and incomplete) have
been shown very effective.
•Restart strategies and portfolio approaches can lead to
substantial improvements in the expected runtime and
variance, especially in the presence of fat and heavy-tailed
phenomena – a way of taking advantage of backdoors and
tractable sub-structure.
•Randomization is therefore a tool to improve algorithmic
performance and robustness.
Summary
Research questions:
Should we consider dramatically different
algorithm design strategies leading to highly
asymmetric distributions, with a good chance of
short runs (even if that means also a good chance of
long runs), that can be effectively exploited with
restarts?
Summary
Notion of a “backdoor” set of variables.
Captures the combinatorics of a problem instance, as
dealt with in practice.
Provides insight into restart strategies.
Backdoors can be surprisingly small in practice.
Search heuristics + randomization can be used to find
them, provably efficiently.
Research Issues
Understanding the semantics of backdoors
Scientific Use of
Experimentation:
Take Home Message
Talk: described scientific experimentation applied to the study
constrained problems has led us to the discovery of and understanding of
interesting computational phenomena which in turn allowed us to better
algorithm design.
Unlikely that we would have discover such phenomena by pure mathematical
thinking / modeling.
Take home message:
In order to understand real-world constrained problems and scale up solutions the
principled experimentation plays a role as important as formal models – the
empirical study of phenomena is a sine qua non for the advancement of the field.
The End
!
www.cs.cornell.edu/gomes