Explorations in Artificial Intelligence
Prof. Carla P. Gomes
gomes@cs.cornell.edu
Module 7
Part 3
Integer Programming
Divisibility
Decision variables in an LP model are allowed to have any values,
including noninteger values, that satisfy the functional and
nonnegativity constraints. i.e., activities can be run at fractional levels.
What to do when divisibility assumption violated:
realm of integer programming!!!
Revisiting the TBA Airlines Problem
An Example where Integrality Matters
The TBA Airlines Problem
•
TBA Airlines is a small regional company that specializes in short flights in small
airplanes.
•
The company has been doing well and has decided to expand its operations.
•
The basic issue facing management is whether to purchase more small airplanes to add
some new short flights, or start moving into the national market by purchasing some
large airplanes, or both.
Question: How many airplanes of each type should be purchased to maximize their
total net annual profit?
Data for the TBA Airlines Problem
Small
Airplane
Large
Airplane
Net annual profit per airplane
$1 million
$5 million
Purchase cost per airplane
5 million
50 million
2
—
Maximum purchase quantity
Capital
Available
$100 million
Linear Programming Formulation
Let S = Number of small airplanes to purchase
L = Number of large airplanes to purchase
Maximize Profit = S + 5L ($millions)
subject to
Capital Available:
5S + 50L ≤ 100 ($millions)
Max Small Planes: S ≤ 2
and
S ≥ 0, L ≥ 0.
Graphical Method for Linear Programming
Number
of large
airplanes
purchased
L
3
2
(2, 1.8) = Optimal solution
Profit = 11 = S + 5 L
1
0
Feasible
region
(2, 1) = Rounded solution
(Profit = 7)
1
2
3
Number of small airplanes purchased
S
Violates Divisibility Assumption of LP
•
Divisibility Assumption of Linear Programming: Decision variables in a linear
programming model are allowed to have any values, including fractional values, that
satisfy the functional and nonnegativity constraints. Thus, these variables are not
restricted to just integer values.
•
Since the number of airplanes purchased by TBA must have an integer value, the
divisibility assumption is violated.
Integer Programming Formulation
Let S = Number of small airplanes to purchase
L = Number of large airplanes to purchase
Maximize Profit = S + 5L ($millions)
subject to
Capital Available:
5S + 50L ≤ 100 ($millions)
Max Small Planes: S ≤ 2
and
S ≥ 0, L ≥ 0
S, L are integers.
Graphical Method for Integer Programming
L
Number
of large
3
airplanes
purchased
2
1
0
(0, 2) = Optimal solution for the integer programming problem (Profit = 10)
(2, 1.8) = Optimal solution for the
LP relaxation (Profit = 11)
Profit = 10 = S + 5 L
(2, 1) = Rounded solution (Profit = 7)
1
2
3
Number of small airplanes purchased
S
Graphical Method for Integer Programming
•
When an integer programming problem has just two decision variables, its optimal solution can be found
by applying the graphical method for linear programming with just one change at the end.
•
We begin as usual by graphing the feasible region for the LP relaxation, determining the slope of the
objective function lines, and moving a straight edge with this slope through this feasible region in the
direction of improving values of the objective function.
•
However, rather than stopping at the last instant the straight edge passes through this feasible region, we
now stop at the last instant the straight edge passes through an integer point that lies within this feasible
region.
•
This integer point is the optimal solution.
Why integer programs?
•
Advantages of restricting variables to take on integer values
– More realistic
– More flexibility
•
Disadvantages
– More difficult to model
– Can be much more difficult to solve
Integer Programming
•
When are “non-integer” solutions okay?
– Solution is naturally divisible
• e.g., $, pounds, hours
– Solution represents a rate
• e.g., units per week
– Solution only for planning purposes
•
When is rounding okay?
– When numbers are large
• e.g., rounding 114.286 to 114 is probably okay.
•
When is rounding not okay?
– When numbers are small
• e.g., rounding 2.6 to 2 or 3 may be a problem.
– Binary variables
• yes-or-no decisions
Types of Integer Programming Problems
•
Pure integer programming problems are those where all the decision
variables must be integers.
•
Mixed integer programming problems only require some of the variables
(the “integer variables”) to have integer values so the divisibility assumption
holds for the rest (the “continuous variables”).
•
Binary variables are variables whose only possible values are 0 and 1.
•
Binary integer programming (BIP) problems are those where all the
decision variables restricted to integer values are further restricted to be binary
variables.
– Such problems can be further characterized as either pure BIP problems or mixed
BIP problems, depending on whether all the decision variables or only some of
them are binary variables.
Examples of Applications of Binary Variables
•
Making “yes-or-no” type decisions
–
–
–
–
•
Logical constraints
–
–
•
Alternative constraints
Conditional constraints
Representing non-linear functions
–
–
–
–
•
Build a factory?
Manufacture a product?
Do a project?
Assign a person to a task?
Fixed Charge Problem
• If a product is produced, must incur a fixed setup cost.
• If a warehouse is operated, must incur a fixed cost.
Piecewise linear representation
Diseconomies of scale
Approximation of nonlinear functions
Set-covering, and set partitioning
–
–
Make a set of assignments that “cover” a set of requirements.
Partition a set into subsets meeting given requirements
StockCompany Example
Capital Budgeting Allocation Problem
StockCompany is considering 6 investments. The cash required from each
investment as well as the NPV of the investment is given next. The cash available
for the investments is $14,000. Stockco wants to maximize its NPV. What is the
optimal strategy?
An investment can be selected or not. One cannot select a fraction of an investment.
Data for the StockCompany Problem
Investment budget = $14,000
Investment 1
Cash
Required
(1000s)
NPV
added
(1000s)
$5
2
3
4
5
6
$7
$4
$3
$4
$6
$16 $22 $12 $8
$11 $19
Integer Programming Formulation
What are the decision variables?
1, if we invest in i  1,...,6,
xi  
else
0,
Objective and Constraints?
Max 16x1+ 22x2+ 12x3+ 8x4+ 11x5+ 19x6
5x1+ 7x2+ 4x3+ 3x4+ 4x5+ 6x6  14
xj e {0,1} for each j = 1 to 6
Capital Budgeting Allocation Problem (one resource)
Knapsack Problem
• Why is a problem with the characteristics of the previous problem
called the Knapsack Problem?
• It is an abstraction, considering the simple problem:
A hiker trying to fill her knapsack to maximum total value.
Each item she considers taking with her has a certain value and a
certain weight. An overall weight limitation gives the single
constraint.
Practical applications:
Project selection and capital budgeting allocation problems
Storing a warehouse to maximum value given the indivisibility of
goods and space limitations
Sub-problem of other problems e.g., generation of columns for a given
model in the course of optimization – cutting stock problem (beyond
the scope of this course)
• The previous constraints represent “economic indivisibilities”, either a
project is selected, or it is not. There is no selecting of a fraction of a
project.
• Similarly, integer variables can model logical requirements (e.g., if
stock 2 is selected, then so is stock 1.)
How to model “logical” constraints
•
Exactly 3 stocks are selected.
•
If stock 2 is selected, then so is stock 1.
•
If stock 1 is selected, then stock 3 is not selected.
•
Either stock 4 is selected or stock 5 is selected, but not both.
Formulating Constraints
•
Exactly 3 stocks are selected
x1+ x2+ x3+ x4+ x5+ x6=3
If stock 2 is selected then so is stock 1
The integer programming constraint:
A 2-dimensional
representation
x1  x2
Stock 2
Stock 1
If stock 1 is selected then stock 3 is not selected
The integer programming constraint:
A 2-dimensional
representation
x1 + x3  1
Stock 3
Stock 1
Either stock 4 is selected or stock 5 is selected, but
not both.
The integer programming constraint:
A 2-dimensional
representation
x4 + x5 = 1
stock 5
stock 4
California Manufacturing Company
•
The California Manufacturing Company is a diversified company with several
factories and warehouses throughout California, but none yet in Los Angeles
or San Francisco.
•
A basic issue is whether to build a new factory in Los Angeles or San
Francisco, or perhaps even both.
•
Management is also considering building at most one new warehouse, but will
restrict the choice to a city where a new factory is being built.
Question: Should the California Manufacturing Company expand with
factories and/or warehouses in Los Angeles and/or San Francisco?
Data for California Manufacturing
Decision
Number
Yes-or-No
Question
Decision
Variable
Net Present
Value
(Millions)
Capital
Required
(Millions)
1
Build a factory in Los Angeles?
x1
$8
$6
2
Build a factory in San Francisco?
x2
5
3
3
Build a warehouse in Los Angeles?
x3
6
5
4
Build a warehouse in San Francisco?
x4
4
2
Capital Available: $10 million
Binary Decision Variables
Decision
Number
Decision
Variable
Possible
Value
Interpretation
of a Value of 1
Interpretation
of a Value of 0
1
x1
0 or 1
Build a factory in
Los Angeles
Do not build
this factory
2
x2
0 or 1
Build a factory in
San Francisco
Do not build
this factory
3
x3
0 or 1
Build a warehouse in
Los Angeles
Do not build
this warehouse
4
x4
0 or 1
Build a warehouse in
San Francisco
Do not build
this warehouse
Algebraic Formulation
Let x1 = 1 if build a factory in L.A.; 0 otherwise
x2 = 1 if build a factory in S.F.; 0 otherwise
x3 = 1 if build a warehouse in Los Angeles; 0 otherwise
x4 = 1 if build a warehouse in San Francisco; 0 otherwise
Maximize NPV = 8x1 + 5x2 + 6x3 + 4x4 ($millions)
subject to
Capital Spent:
6x1 + 3x2 + 5x3 + 2x4 ≤ 10 ($millions) Resource Availability
Max 1 Warehouse:
x3 + x4 ≤ 1 Mutually exclusive decisions
Warehouse only if Factory:
x3 ≤ x1
Contingent decisions
x4 ≤ x2
and
x1, x2, x3, x4 are binary variables.
Mpl Model
Max 8x1 + 5x2 + 6x3 + 4x4;
subject to
6x1 + 3x2 + 5x3 + 2x4 <= 10;
x3 + x4 <= 1;
x3 <= x1;
x4 <= x2;
BINARY
x1;
x2;
x3;
x4;
Modeling Fixed Charge Problems
If a product is produced, must incur a fixed setup cost.
If a warehouse is operated, must incur a fixed cost.
 The problem is non-linear.
x – quantity of product to be manufactured
x = 0  cost =0;
x > 0  cost = C1x + C2
 How to model it? Using an indicator variable y
y = 1  x is produced; y = 0  x is not produced
Objective function becomes  C1x + C2y
Additional Constraint  x ≤ My
Wyndor with Setup Costs (Variation 1)
Suppose that two changes are made to the original Wyndor problem:
1. For each product, producing any units requires a substantial one-time setup
cost for setting up the production facilities.
2. The production runs for these products will be ended after one week, so D and
W in the original model now represent the total number of doors and windows
produced, respectively, rather than production rates. Therefore, these two
variables need to be restricted to integer values.
Graphical Solution to Original Wyndor Problem
W
Production rate
for windows
8
Optimal solution
(2, 6)
6
4
Feasible
Region
P = 3,600 = 300 D + 500 W
2
0
4
2
Production rate for doors
6
8
10 D
Net Profit for Wyndor Problem with Setup Costs
Net Profit ($)
Number of
Units Produced
Doors
Windows
0
0(300) – 0 = 0
0 (500) – 0 = 0
1
1(300) – 700 = –400
1(500) – 1,300 = –800
2
2(300) – 700 = –100
2(500) – 1,300 = –300
3
3(300) – 700 = 200
3(500) – 1,300 = 200
4
4(300) – 700 = 500
4(500) – 1,300 = 700
5
Not feasible
5(500) – 1,300 = 1,200
6
Not feasible
6(500) – 1,300 = 1,700
Feasible Solutions for Wyndor with Setup Costs
W
Production
quantity for
windows
Optimal solution
8
(0, 6) gives P = 1700
6
(2, 6) gives P = -100 + 1700
= 1600
4
(4, 3) gives P = 500 + 200
= 700
2
(0, 0)
gives P = 0
(4, 0) gives P = 500
0
6
2
4
Production quantity for doors
8 D
Algebraic Formulation
Let D = Number of doors to produce,
W = Number of windows to produce,
y1 = 1 if perform setup to produce doors; 0 otherwise,
y2 = 1 if perform setup to produce windows; 0 otherwise .
Maximize P = 300D + 500W – 700y1 – 1,300y2
subject to
Original Constraints:
Plant 1:
D≤4
Plant 2:
2W ≤ 12
Plant 3:
3D + 2W ≤ 18
Produce only if Setup:
Doors:
D ≤ My1
Windows: W ≤ My2
and
D ≥ 0, W ≥ 0, y1 and y2 are binary.
Wyndor with Mutually Exclusive Products
(Variation 2)
Suppose that now the only change from the original Wyndor problem is:
•
The two potential new products (doors and windows) would compete for the
same customers. Therefore, management has decided not to produce both of
them together.
–
At most one can be chosen for production, so either D = 0 or W = 0, or both.
Feasible Solution for
Wyndor with Mutually Exclusive Products
(for non-binary variables)
W
Production
rate for
windows
P = 300 D + 500 W
Either D = 0 or W = 0
8
(0, 6) gives P = 3,000
6
4
2
(0, 0) gives P = 0
0
(4, 0) gives P = 1,200
2
4
Production rate for doors
6
8 D
Algebraic Formulation
Let D = Number of doors to produce,
W = Number of windows to produce,
y1 = 1 if produce doors; 0 otherwise,
y2 = 1 if produce windows; 0 otherwise.
Maximize P = 300D + 500W
subject to
Original Constraints:
Plant 1:
D≤4
Plant 2:
2W ≤ 12
Plant 3:
3D + 2W ≤ 18
Auxiliary variables must =1 if produce any:
Doors:
D ≤ My1
Windows:
W ≤ My2
Mutually Exclusive: y1 + y2 ≤ 1
and
D ≥ 0, W ≥ 0, y1 and y2 are binary.
Wyndor with Either-Or Constraints
(Variation 3)
Suppose that now the only change from the original Wyndor problem is:
•
The company has just opened a new plant (plant 4) that is similar to plant 3, so
the new plant can perform the same operations as plant 3 to help produce the
two new products (doors and windows).
•
However, management wants just one of the plants to be chosen to work on
these new products. The plant chosen should be the one that provides the most
profitable product mix.
Data for Wyndor with Either-Or Constraints
(Variation 3)
Production Time Used for
Each Unit Produced (Hours)
Plant
Doors
Windows
Production Time
Available
per Week (Hours)
1
1
0
4
2
0
2
12
3
3
2
18
4
2
4
28
Unit Profit
$300
$500
Graphical Solution with Plant 3 or Plant 4
(a) Choose Plant 3
(b) Choose Plant 4
W
W
8
8
(2, 6) gives
P = 3,600
6
6
4
4
Feasible
region
Feasible
region
2
0
(4, 5) gives
P = 3,700
2
2
4
D
0
2
4
D
Algebraic Formulation
Let D = Number of doors to produce,
W = Number of windows to produce,
y = 1 if plant 4 is used; 0 if plant 3 is used
Maximize P = 300D + 500W
subject to
Plant 1:
D≤4
Plant 2:
2W ≤ 12
Plant 3:
Plant 4:
3D + 2W ≤ 18 + My
2D + 4W ≤ 28 + M(1 – y)
and
D ≥ 0, W ≥ 0, y is binary.
Applications of Binary Variables
•
Making “yes-or-no” type decisions
–
–
–
–
•
Build a factory?
Manufacture a product?
Do a project?
Assign a person to a task?
Fixed costs
– If a product is produced, must incur a fixed setup cost.
– If a warehouse is operated, must incur a fixed cost.
•
Either-or constraints
– Production must either be 0 or ≥ 100.
•
Subset of constraints
– meet 3 out of 4 constraints.
Special Kinds of Integer Programming Models
• Knapsack Problem
• Set Covering Problem
• Set Partitioning Problem
• Set Packing Problem
• The Traveling Salesman Problem
• The Quadratic Assignment Problem
Set Covering Problem
• We are given a set of objects S = {1, 2, 3, …, n}.
• We are also given a set of subsets of S, S. Each subset has
a cost associated with it.
• Problem:
– to “cover” all the members of S at the minimum cost using
members of S.
• Properties:
– The problem is a minimization and all constraints are >=;
– All RHS coefficients are 1;
– All other matrix coefficients are 0 or 1.
Fire Station Problem
Set Covering Problem
1
2
5
Locate fire stations
so that each district
has a fire station in
it, or next to it.
3
6
7
4
8
11
Minimize the
number of fire
stations needed.
9
12
13
10
14
15
16
Representation as Set Covering Problem
1
2
5
7
8
11
Covers
1
1, 2, 4, 5
2
1, 2, 3, 5, 6
3
2, 3, 6, 7
16
13, 15, 16
3
6
4
Set
9
12
13
10
14
15
16
Representation as Integer program
11
2
xj = 1 if node j is selected
xj = 0 otherwise
3
Minimize x1 + x2 + … + x16
5
6
4
8
11
7
x1 + x2 + x3 + x5 + x6  1
9
12
s.t. x1 + x2 + x4 + x5  1
13
10
14
15
16
x13 + x15 + x16  1
xj  {0, 1} for each j.
Southwestern Airways Crew Scheduling
•
Southwestern Airways needs to assign crews to cover all its upcoming flights.
•
We will focus on assigning 3 crews based in San Francisco (SFO) to 11 flights.
Question: How should the 3 crews be assigned 3 sequences of flights so that
every one of the 11 flights is covered?
Southwestern Airways Flights
Seat tl e
(SEA)
San Francis co
(SFO)
Los Angel es
(LAX)
Denver
(DEN)
Chi cago
ORD)
Data for the Southwestern Airways Problem
Feasible Sequence of Flights (pairings)
Flights
1
1. SFO–LAX
1
2. SFO–DEN
2
3
5
6
1
1
3. SFO–SEA
1
2
3
3
3
4
9. DEN–ORD
3
4
4
4
6
7
3
5
5
3
3
4
5
7
2
4
2
3
2
2
2
11. SEA–LAX
1
5
2
10. SEA–SFO
12
4
7. ORD–SEA
2
11
1
3
3
10
1
1
2
2
9
1
2
8. DEN–SFO
8
1
1
6. ORD–DEN
Cost, $1,000s
7
1
4. LAX–ORD
5. LAX–SFO
4
8
5
2
4
4
2
9
9
8
9
Algebraic Formulation
Let xj = 1 if flight sequence (paring) j is assigned to a crew; 0 otherwise. (j = 1, 2, … , 12).
Minimize Cost = 2x1 + 3x2 + 4x3 + 6x4 + 7x5 + 5x6 + 7x7 + 8x8 + 9x9 + 9x10 + 8x11 + 9x12
(in $thousands)
pairings
subject to
Flight 1 covered:
Flight 2 covered:
:
Flight 11 covered:
x1 + x4 + x7 + x10 ≥ 1
x2 + x5 + x8 + x11 ≥ 1
:
x6 + x9 + x10 + x11 + x12 ≥ 1
Three Crews:
x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11 + x12 ≤ 3
and
xj are binary (j = 1, 2, … , 12).
Set Covering Problem
• We are given a set of objects S = {1, 2, 3, …, n}.
• We are also given S, a set of subsets of S. Each subset has
a cost associated with it.
• Problem:
– to “cover” all the members of S at the minimum cost using
members of S.
• Properties:
– The problem is a minimization and all constraints are >=;
– All RHS coefficients are 1;
– All other matrix coefficients are 0 or 1.
Some Comments on IP models
• There are often multiple ways of modeling the
same integer program.
• Solvers for integer programs are extremely
sensitive to the formulation. (not true for LPs)
Summary on Integer Programming
• Dramatically improves the modeling capability
– Economic indivisibilities
– Logical constraints
– Modeling nonlinearities (e.g., fixed cost)
– classical problems in capital budgeting and in
supply chain management
– Lots of other applications and models
• Not as easy to model
• Not as easy to solve.
• Solving LP
The Challenges of Rounding
•
Rounded Solution may not be
feasible.
•
Rounded solution may not be
close to optimal.
•
There can be many rounded
solutions.
– Example: Consider a problem
with 30 variables that are noninteger in the LP-solution.
How many possible rounded
solutions are there?
x2
5
4
3
2
1
1
2
3
4
5
x1
Overview of Techniques for Solving Integer
Programs
• Enumeration Techniques
– Complete Enumeration
• list all “solutions” and choose the best
– Branch and Bound
• Implicitly search all solutions, but cleverly eliminate the vast majority
before they are even searched
– Implicit Enumeration
• Branch and Bound applied to binary variables
• Cutting Plane Techniques
– Use LP to solve integer programs by adding constraints to eliminate the
fractional solutions.
How Integer Programs are Solved
x2
5
4
3
2
1
1
2
3
4
5
x1
How Integer Programs are Solved
x2
5
4
3
2
1
1
2
3
4
5
x1
Branch and Bound
– It is the starting point for all solution techniques for
integer programming
– Lots of research has been carried out over the past 40
years to make it more and more efficient
– But, it is an art form to make it efficient. (We shall get
a sense why.)
– Integer programming is intrinsically difficult.
Capital Budgeting Example
Investment budget = $14,000
Investment
Cash
Required
(1000s)
NPV
added
(1000s)
1
2
3
4
5
6
$5
$7
$4
$3
$4
$6
$16 $22 $12 $8
$11 $19
maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6
subject to
5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14
xj binary for j = 1 to 6
Complete Enumeration
• Systematically considers all possible values of the decision
variables.
– If there are n binary variables, there are 2n different
ways.
• Usual idea: iteratively break the problem in two. At the
first iteration, we consider separately the case that x1 = 0
and x1 = 1.
An Enumeration Tree
Original
problem
x1 = 0
x2 = 0
x3 = 0
x3 = 1
x1 = 1
x3 = 0
x2 = 1
x2 = 0
x2 = 1
x3 = 1
x3 = 0
x3 = 1
x3 = 0
x3 = 1
On complete enumeration
• Suppose that we could evaluate 1 billion solutions per
second.
• Let n = number of binary variables
• Solutions times (approx.)
– n = 30,
1 second
– n = 40,
18 minutes
– n = 50
13 days
– n = 60
31 years
On complete enumeration
• Suppose that we could evaluate 1 trillion solutions per
second, and instantaneously eliminate 99.9999999% of all
solutions as not worth considering
• Let n = number of binary variables
• Solutions times
– n = 70,
– n = 80,
– n = 90
– n = 100
1 second
17 minutes
11.6 days
31 years
Branch and Bound
The essential idea: search the enumeration tree, but at
each node
1.
2.
Solve the linear program at the node
Eliminate the subtree (fathom it) if
1. The solution is integer (there is no need to go furtherwhy?) or
2. The best solution in the subtree cannot be as good as
the best available solution (the incumbent- how does
that happen?) or
3. There is no feasible solution
Branch and Bound
1
44 3/7
Node 1 is the
original LP
Relaxation
maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6
subject to
5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14
0  xj  1 for j = 1 to 6
Solution at node 1:
x1 =1 x2 = 3/7
x3 = x4 = x5 = 0 x6 =1
z = 44 3/7
The IP cannot have value higher than 44 3/7.
Branch and Bound
1
44 3/7
x1 = 0
44
Node 2 is the
original LP
Relaxation plus
the constraint
x1 = 0.
2
maximize 16x1 + 22x2 + 12x3 + 8x4 +11x5 + 19x6
subject to
5x1 + 7x2 + 4x3 + 3x4 +4x5 + 6x6  14
0  xj  1 for j = 1 to 6, x1 = 0
Solution at node 2:
x1 = 0 x2 = 1 x3 = 1/4 x4 = x5 = 0 x6 = 1
z = 44
Branch and Bound
1
x1 = 0
44
Node 3 is the
original LP
Relaxation plus
the constraint
x1 = 1.
44 3/7
x1 = 1
3
2
44 3/7
The solution at node 1 was
x1 =1 x2 = 3/7
x3 = x4 = x5 = 0
x6 =1
z = 44 3/7
Note: it was the best solution with no constraint on x1.
So, it is also the solution for node 3. (If you add a
constraint, and the old optimal solution is feasible, then
it is still optimal.)
Branch and Bound
44 3/7
1
x1 = 0
44
Node 4 is the
original LP
Relaxation plus
the constraints
x1 = 0, x2 = 0.
x1 = 1
2
3
44 3/7
x2 = 0
42
44
Solution at node 4:
0
0
1
0
1
Our first incumbent solution!
No further searching from node 4 because
there cannot be a better integer solution.
1
z = 42
No solution in
the subtree can
have a value
better than 42.
Branch and Bound
The incumbent
is the best
solution on
hand.
44
1
44
44 3/7
x1 = 0
x1 = 1
2
3
x2 = 0
x2 = 1
x2 = 0
42
The
incumbent
solution has
value 42
44
5
44
6
44 3/7
x2 = 1
44 1/3 7
We next solved the LP’s associated with nodes 5, 6, and 7
No new integer solutions were found.
We would eliminate (fatham) a subtree if we were guaranteed
that no solution in the subtree were better than the incumbent.
Branch and Bound
1
44 3/7
x1 = 0
44
44
x1 = 1
2
3
x2 = 0
x2 = 1
x2 = 0
42
The
incumbent
solution has
value 42
44
x3 = 0
43.75 8
5
44
x3 = 1
43.5 9
x3 = 0
43.25 10
6
44 3/7
x2 = 1
44 1/3 7
x3 = 1
43.8 11
x3 = 0
44.3 12
x3 = 1
-
We next solved the LP’s associated with nodes 8 -13
13
13
Summary so far
• We have solved 13 different linear programs so far.
– One integer solution found
– One subtree fathomed (pruned) because the solution
was integer (node 4)
– One subtree fathomed because the solution was
infeasible (node 13)
– No subtrees fathomed because of the bound
Branch and Bound
1
x1 = 0
44
4
x1 = 1
2
3
x2 = 0
x2 = 1
x2 = 0
42
The
incumbent
solution has
value 42
44 3/7
5
44
x3 = 0
43.75 8
44
x3 = 1
43.5 9
x3 = 0
43.25 10
6
44 3/7
x2 = 1
44 1/3 7
x3 = 1
43.8 11
x3 = 0
44.3 12
x3 = 1
- 13
43.75 42.66
We next solved the LP’s associated with the next nodes.
We can fathom the node with z = 42.66. Why?
Getting a better bound
• The bound at each node is obtained by solving an LP.
• But all costs are integer, and so the objective value of each
integer solution is integer. So, the best integer solution has an
integer objective value.
• If the best integer valued solution for a node is at most 42.66,
then we know the best bound is at most 42.
• Other bounds can also be rounded down.
Branch and Bound
1
x1 = 0
44
4
x1 = 1
2
3
x2 = 0
x2 = 1
x2 = 0
42
The
incumbent
solution has
value 42
44 3/7
5
44
x3 = 0
43.75 8
44
x3 = 1
43.5 9
43.75 42.66
x3 = 0
43.25 10
6
44 3/7
x2 = 1
44 1/3 7
x3 = 1
43.8 11
x3 = 0
44.3 12
x3 = 1
-. 13
Branch and Bound
1
x1 = 0
44
x1 = 1
2
3
4
44
5
43 8
43
x3 = 1
43 9
42
43
x2 = 1
6
44
x3 = 0
44
x2 = 0
x2 = 1
x2 = 0
42
The
incumbent
solution has
value 42
44
x3 = 0
x3 = 1
43 10
42
43
44 7
43 11
x3 = 0
44 12
x3 = 1
- 13
43
We found a new incumbent solution!
x1 = 1, x2 = x3 = 0, x4 = 1, x5 = 0, x6 = 1 z = 43
Branch and Bound
1
x1 = 0
44
x1 = 1
2
3
4
44
5
43 88
43
x3 = 1
43 99
42
43
x2 = 1
6
44
x3 = 0
44
x2 = 0
x2 = 1
x2 = 0
42
The new
incumbent
solution has
value 43
44
x3 = 0
x3 = 1
10
43 10
42
43
44 7
11
43 11
x3 = 0
44 12
x3 = 1
- 13
43
We found a new incumbent solution!
x1 = 1, x2 = x3 = 0, x4 = 1, x5 = 0, x6 = 1 z = 43
Branch and Bound
1
x1 = 0
44
4
x1 = 1
2
3
44
x3 = 0
43 88
44
x2 = 1
x2 = 0
x2 = 1
x2 = 0
42
The new
incumbent
solution has
value 43
44
5
44
x3 = 1
43 99
x3 = 0
10
43 10
6
44 7
x3 = 1
11
43 11
x3 = 0
44 12
x3 = 1
- 13
If we had found this incumbent earlier, we could have
saved some searching.
Finishing Up
1
x1 = 0
44
4
x1 = 1
2
3
44
x3 = 0
43 88
44
x2 = 1
x2 = 0
x2 = 1
x2 = 0
42
The new
incumbent
solution has
value 43
44
5
44
x3 = 1
43 99
x3 = 0
10
43 10
6
44 7
x3 = 1
11
43 11
x3 = 0
44 12
44 14
44 16
38 18
x3 = 1
- 13
15 -
17 -
19
-
Lessons Learned
• Branch and Bound can speed up the search
• Only 25 nodes (linear programs) were evaluated
• Other nodes were fathomed
• Obtaining a good incumbent earlier can be valuable
• only 19 nodes would have been evaluated.
• Solve linear programs faster, because we start with an excellent or
optimal solution
• uses a technique called the dual simplex method
• Obtaining better bounds can be valuable.
Branch and Bound
Notation:
– z* = optimal integer solution value
– Subdivision: a node of the B&B Tree
– Incumbent: the best solution on hand
– zI: value of the incumbent
– zLP: value of the LP relaxation of the current node
– Children of a node: the two problems created for a
node, e.g., by saying xj = 1 or xj = 0.
– LIST: the collection of active (not fathomed) nodes,
with no active children.
NOTE: zI  z*
Illustrating the definitions
1
44 3/7
x1 = 0
44
x1 = 1
2
3
x2 = 0
x2 = 1
x2 = 0
42 44
The incumbent
solution has
value 42
44
x3 = 0
43.75 8
5
44
x3 = 1
43.5 9
x3 = 0
43.25 10
6
44 3/7
x2 = 1
44 1/3 7
x3 = 1
43.8 11
x3 = 0
44.3 12
x3 = 1
13
- 13
z* = 43 = optimal integer solution value. (We
found it later in the search)
Incumbent is 0 0 1 0 1 1 zI = 42.
It is the optimal solution for the subdivision 4.
The zLP values for each subdivision are next to the nodes.
Illustrating the definitions
1
44 3/7
x1 = 0
44
x1 = 1
2
3
x2 = 0
x2 = 1
x2 = 0
42 44
The incumbent
solution has
value 42
44
x3 = 0
43.75 8
5
44
x3 = 1
43.5 9
x3 = 0
43.25 10
6
44 3/7
x2 = 1
44 1/3 7
x3 = 1
43.8 11
x3 = 0
44.3 12
x3 = 1
13
- 13
The children of node (subdivision) 1 are nodes 2 and 3.
The children of node 3 are nodes 6 and 7.
LIST = { 8, 9, 10, 11, 12 } = unfathomed nodes with no
active children
Branch and Bound Algorithm
INITIALIZE LIST =
{original problem}
Incumbent: = 
zI = -
SELECT:
If LIST = , then the Incumbent is optimal if it exists,
and the problem is infeasible if no incumbent exists;
else, let S be a node (subdivision) from LIST.
Let xLP be the optimal solution to S
Let zLP = its objective value
e.g., S = {1}
44 3/7 1
e.g., S = {13}
- 13
CASE 1. zLP = - (the LP is infeasible)
Remove S from LIST (fathom it)
Return to SELECT
Branch and Bound Algorithm
INITIALIZE
SELECT:
If LIST = , then the Incumbent is optimal (if it exists),
and the problem is infeasible if no incumbent exists;
else, let S be a node from LIST.
Let xLP be the optimal solution to S
Let zLP = its objective value
CASE 2. - < zLP  zI.
That is, the LP is dominated by the incumbent.
Then remove S from LIST (fathom it)
Return to SELECT
43
43 14
e.g., the incumbent has value 43, and node
14 is selected. zLP = 43.
88
42
Branch and Bound Algorithm
INITIALIZE
SELECT:
If LIST = , then the Incumbent is optimal (if it exists),
and the problem is infeasible if no incumbent exists;
else, let S be a subdivision from LIST.
Let xLP be the optimal solution to S
Let zLP = its objective value
CASE 3. zI < zLP and xLP is integral.
That is, the LP solution is integral and
dominates the incumbent.
e.g., node 4 was selected,
and the solution to the LP
was integer-valued.
Then Incumbent := xLP;
zI := zLP
Remove S from LIST (fathomed by integrality)
Return to SELECT
42
4
Branch and Bound Algorithm
INITIALIZE
SELECT:
If LIST = , then the Incumbent is optimal (if it exists),
and the problem is infeasible if no incumbent exists;
else, let S be a subdivision from LIST.
Let xLP be the optimal solution to S
Let zLP = its objective value
e.g., select node 3.
I
LP
LP
CASE 4. z < z and x is not integral.
There is not enough information to fathom S
x1 = 1
3
Remove S from LIST
Add the children of S to LIST
Return to SELECT
x2 = 0
44 3/7
x2 = 1
6
List := List – 3 + {6,7}
7
Different Selection Rules are Possible
• Rule of Thumb 1: Don’t let LIST get too big (the solutions
must be stored). So, prefer nodes that are further down in
the tree.
• Rule of Thumb 2: Pick a node of LIST that is likely to lead
to an improved incumbent. Sometimes special heuristics
are used to come up with a good incumbent.
Branching
One does not have to have the B&B
tree be symmetric, and one does not
select subtrees by considering variables
in order.
X1
X2
=0
X2
X
X3 =
. .
. .
. .
0
=0
X3 =
. .
. .
. .
3= 0
X1
=1
X8 = 0
X8 = 1
X
X3=
1
X 3=
1
. .
. .
. .
=1
. .
. .
. .
0
X 3=
. .
. .
. .
. .
. .
. .
3
=0
1
X3
.
.
.
.
.
.
.
.
.
.
.
.
=1
Choosing how to
branch so as to
reduce running
time is largely
“art” and based
on experience.
Different Branching Rules are Possible
• Branching: determining children for a node. There are
many choices.
• Rule of thumb 1: if it appears clear that xj = 1 in an
optimal solution, it is often good to branch on xj = 0 vs xj =
1.
– The hope is that a subdivision with xj = 0 can be
pruned.
• Rule of thumb 2: branching on important variables is
worthwhile
Different Bounding Techniques are Possible
• We use the bound obtained by dropping the integrality
constraints (LP relaxation). There are other choices.
• Key tradeoff for bounds: time to obtain a bound vs quality
of the bound.
• If one can obtain a bound much quicker, sometimes we
would be willing to get a bound that is worse
• It usually is worthwhile to get a bound that is better, so
long as it doesn’t take too long.
What if the variables are general integer variables?
• One can choose children as follows:
– child 1: x1  3 (or xj  k)
– child 2 x1  4 (or xj  k+1)
• How would one choose the variable j and the value k
– A common choice would be to take a fractional value
from xLP. e.g., if x7 = 5.62, then we may branch on x7 
5 and x7  6.
– Other choices are also possible.
•
Branch and Bound is the standard way of solving IPs to optimality.
•
There is art to making it work well in practice.
•
Much of the art is built into state-of-the-art solvers such as CPLEX.
New exciting area
combining LP based branch-and-bound based
techniques with constraint programming
techniques - forward checking; arc-consistency
and other forms of consistency checking and
propagation!
A* - well-known AI algorithm that is a
generalization of branch-and-bound with
LP relaxations – notion of admissible
heuristic that overestimates
(underestimates) the objective function for
a maximization (minimization) problem
(analogously to what a relaxation does)
Used in e.g. MapQuest and Darpa Challenge
It was fun to teach INFO 372!
Hope you had fun too!
THE END
!!!