CS6234: Lecture 4
 Linear Programming
 LP and Simplex Algorithm [PS82]-Ch2
 Duality
[PS82]-Ch3
 Primal-Dual Algorithm
[PS82]-Ch5
 Additional topics:
Reading/Presentation by students
Lecture notes adapted from Comb Opt course by
Jorisk, Math Dept, Maashricht Univ,
Hon Wai Leong, NUS
(CS6234, Spring 2009) Page 1
Copyright © 2009 by Leong Hon Wai
Combinatorial Optimization
Chapter 3 of [PS82]
Duality
7/1/2016
Combinatorial Optimization
Masters OR
LP in general form
min
s.t.
c´ x
a´i x
a´i x
xj ≥
xj
=
bi
≤
bi
0
free
jεN
iεM
i ε M´
j ε N´
1. Introduce surplus variables for inequality
constraints
2. Replace free variables by two non-negative
variables.
7/1/2016
Combinatorial Optimization
Masters OR
In standard form…
min
s.t.
c ~´ x ~
A~ x~ = b
x~ ≥ 0
where A~ has extra columns for x´s in N´, and
for slack variables,
x~ has extra variables for x´s in N´ and
for slack variables,
c~ has extra elements for x´s in N´.
7/1/2016
Combinatorial Optimization
Masters OR
Starting the dual
The simplex method gives an optimal solution x0~
to the LP in standard form, with a basis Q,
satisfying
c~´ – (cB~´ B~-1)A~ ≥ 0.
Thus, π´ = cB~´ B~-1 is a feasible solution to the
problem
π´ A~ ≤ c~´ where π ε Rm.
7/1/2016
Combinatorial Optimization
Masters OR
The dual
1. π´ Aj ≤ cj, for j ε N,
2. π´ Aj ≤ cj, for j ε N´, and
π´ Aj ≤ –cj, for j ε N´,
and hence π´ Aj = cj, for j ε N´.
3. –πi ≤ 0, for i ε M´, or
πi ≥ 0, for i ε M´.
Objective: max π´ b.
7/1/2016
Combinatorial Optimization
Masters OR
Definition of the Dual
Definition 3.1: Given an LP in general form, called
the primal, the dual is defined as follows
Primal
Dual
Min c´x
Max π´b
7/1/2016
a ´i x = b i
iεM
πi free
a´i x ≥ bi
i ε M´
πi ≥ 0
xj ≥ 0
jεN
π´ Aj ≤ cj
xj free
j ε N’
π´ Aj = cj
Combinatorial Optimization
Masters OR
Strong duality
Theorem 3.1: If an LP has an optimal
solution, so does its dual, and at optimality
their costs are equal.
Proof. Let x and π be optimal solutions to
the primal and the dual resp., then
c´x ≥ π´Ax ≥ π´b.
From the
dual
7/1/2016
(*)
From the
primal
Combinatorial Optimization
Masters OR
Strong duality proof
Thus the primal has cost at least as high as the dual.
Then, if the primal has a feasible solution,
the cost of the dual cannot be unbounded.
It has feasible solution π´, and since it is not unbounded,
it must have an optimum.
The cost of π´ is
π´ b = cB~´ B~-1 b = cB~´ x0~
which is the optimal solution of the primal.
Together with (*) this suffices to prove the Theorem.
7/1/2016
Combinatorial Optimization
Masters OR
The dual of the dual
Theorem 3.2. The dual of the dual is the primal.
Proof. Left as an exercise.
Theorem 3.3: Given a primal-dual pair, either of
the following holds:
1. Both have a finite optimum
2. Both are infeasible
3. One is unbounded the other is infeasible
Proof. Skip.
7/1/2016
Combinatorial Optimization
Masters OR
Complementary slackness
Theorem 3.4:
A pair, x, π, respectively feasible in a
primal-dual pair is optimal if and only if:
ui = πi (a´i x - bi) = 0
vj = (cj - π´ Aj) xj = 0
7/1/2016
for all i
for all j.
Combinatorial Optimization
Masters OR
(1)
(2)
Complementary slackness (2)
Proof:
Recall ui = πi (a´i x - bi) & vj = (cj - π´Aj) xj
Define u =  ui (≥ 0)
& v =  vj (≥ 0)
Thus, u = 0 if and only if (1) holds (each ui = 0) and
v = 0 if and only if (2) holds (each vj = 0).
Now, notice that (u + v) = c´x – π´b.
Thus, if (1) and (2) hold, (u + v) = 0, and c´x = π´b.
This implies that x and π have the same objective value,
and hence are both optimal.
7/1/2016
Combinatorial Optimization
Masters OR
Complementary slackness (3)
Consequences:
If an inequality constraint in the dual is not binding,
then the corresponding primal variable must
have value zero and vice versa.
Likewise, if a nonnegative variable has strictly
positive value, then the corresponding constraint
must be binding.
7/1/2016
Combinatorial Optimization
Masters OR
Tableau (for example 2.6)
Min Z = x1
s.t.
3x1 + 2x2 +
5x1 + 2x2 +
2 x1 + 5 x 2 +
–z
7/1/2016
0
1
3
4
+ x2 + x3 + x4 + x5
x3
x3
x3
+ x4
+ x5
=1
=3
=4
x1
x2
x3
x4
x5
1
3
5
2
1
2
1
5
1
1
1
1
1
0
1
0
1
0
0
1
Combinatorial Optimization
Masters OR
Simplex Algorithm (Tableau Form)
–z
x1
x2
x3
x4
x5
0
1
1
1
1
1
1
3
2
1
0
0
3
5
1
1
1
0
4
2
5
1
0
1
x1
x2
x3
x4
x5
–z
-6
-3
-3
0
0
0
x3 =
1
3
2
1
0
0
x4 =
2
2
-1
0
1
0
x5 =
3
-1
3
0
0
1
7/1/2016
Diagonalize to
get bfs { x3, x4, x5 }
x1
x2
x3
x4
x5
–z
0
1
1
1
1
1
x3=
1
3
2
1
0
0
x4=
2
2
-1
0
1
0
x5 =
3
-1
3
0
0
1
Determine reduced cost by
making c~j = 0 for all basic col
Combinatorial Optimization
Masters OR
7/1/2016
Combinatorial Optimization
Masters OR
Farkas Lemma
Skip 3.3
7/1/2016
Combinatorial Optimization
Masters OR
The shortest path problem
and its dual
Definition 3.3 Given a directed graph G=(V,E)
and a nonnegative weight cj ≥ 0 associated with
each arc ej ε E, an instance of the shortest path
problem (SP) is the problem of finding a directed
path from a distinguished source node s to a
distinguished terminal node t, with the minimum
total weight.
7/1/2016
Combinatorial Optimization
Masters OR
Shortest path problem
Feasible set F is given by
F = {P = (ej1,..,ejk) : P is a directed path from s to t in G}.
Cost c(P) = Σi=1k cji .
Formulation as LP uses node-arc incidence matrix A:
aij = {
7/1/2016
1
-1
0
if arc j leaves node i
-f arc j enters node i
otherwise
Combinatorial Optimization
Masters OR
SP Example
e1 e2 e 3 e4 e5
a
e1
3
1
t
e3
2
1
0
0
0
A
=
t
0
0
0
-1 -1
a
-1
0
1
1
0
b
0
-1 -1
0
1
e5
1
e2
1
e4
2
s
s
b
7/1/2016
Combinatorial Optimization
Masters OR
LP formulation
Min e1+
2e2+
2e3 +
3e4 + e5
s.t.
e1 + e2
- e1
e3 +
- e3 +
- e2
0 ≤ ei ≤ 1, i =1…5
7/1/2016
Combinatorial Optimization
Masters OR
- e4 e4
e5
e5
= 1
= -1
= 0
= 0
Redundancy
Claim: Any solution that satisfies 3 of the four
constraints, satisfies all 4 of them.
Easily checked.
Thus, we omit the constraint for the sink t.
Next, we consider the tableau of a feasible solution
7/1/2016
Combinatorial Optimization
Masters OR
a
e1
Starting Tableau
3
1
e4
2
s
t
e3
2
e5
1
e2
b
e1
e2
e3
e4
e5
-z
0
1
2
2
3
1
s
1
1
1
0
0
0
a
0
-1
0
1
1
0
b
0
0
-1
-1
0
1
7/1/2016
Combinatorial Optimization
Masters OR
Summary of LP for SP
• Simplex algorithm in tableau form
– Cost vector c is also included as a constraint
(Read details in [PS82]-Ch-2)
• Each basis has (n-1) edges;
Clever idea!
– Not all are positive;
– Edges with +ve flows are on the (s-t) path
• Pivoting – bring in a new edge;
– Based on relative cost (row0 of tableau)
7/1/2016
Combinatorial Optimization
Masters OR
a
e1
Starting Tableau
3
1
e4
2
s
t
e3
2
e5
1
e2
b
e1
e2
e3
e4
e5
-z
0
1
2
2
3
1
s
1
1
1
0
0
0
a
0
-1
0
1
1
0
b
0
0
-1
-1
0
1
To prepare basis { e1, e4, e5 }: row2 row2 + row1;
7/1/2016
Combinatorial Optimization
Masters OR
a
e1
Initial feasible bfs (1)
After we have row-2  row-1 + row-2;
3
1
e4
2
s
t
e3
2
e5
1
e2
b
-z
s
a
b
0
1
1
0
e1
e2
e3
e4
e5
1
1
0
0
2
1
1
-1
2
0
1
-1
3
0
1
0
1
0
0
1
Basis { e1, e4, e5 }: e1=1, e4=1, e5 =0, this bfs is degenerate
Now, to adjust reduced costs; First, row-0  row-0 - row-1
7/1/2016
Combinatorial Optimization
Masters OR
a
e1
Initial feasible bfs (2)
After we have row 2  row 1 + row 2;
After, row 0  row 0 - row 1;
3
1
e4
2
s
t
e3
2
e5
1
e2
b
-z
s
a
b
-1
1
1
0
e1
e2
e3
e4
e5
0
1
0
0
1
1
1
-1
2
0
1
-1
3
0
1
0
1
0
0
1
Then, row-0  row-0 – 3*(row-2);
7/1/2016
Combinatorial Optimization
Masters OR
a
e1
Initial feasible bfs (3)
After we have row 2  row 1 + row 2;
Then, row 0  row 0 + row 1;
Then, row 0  row 0 – 3*(row 2);
-z
s
a
b
-4
1
1
0
e4
2
s
t
e3
2
e5
1
e2
b
e1
e2
e3
e4
e5
0
1
0
0
-2
1
1
-1
-1
0
1
-1
0
0
1
0
1
0
0
1
Then, row-0  row-0 – row-3;
7/1/2016
3
1
Combinatorial Optimization
Masters OR
a
e1
Initial feasible bfs (4)
After we have row 2  row 1 + row 2;
Then, row 0  row 0 + row 1;
Then, row 0  row 0 – 3*(row 2);
Then, row 0  row 0 + *(row 3)
-z
s
a
b
-4
1
1
0
3
1
2
s
t
e3
2
e5
1
e2
b
e1
e2
e3
e4
e5
0
1
0
0
-1
1
1
-1
0
0
1
-1
0
0
1
0
0
0
0
1
Basis { e1, e4, e5 }: e1=1, e4=1, e5 =0, this bfs is degenerate
7/1/2016
e4
Combinatorial Optimization
Masters OR
a
e1
Pivoting (1)
-4
1
1
0
e4
2
s
Choose col with -ve relative-cost;
(bring col e2 into the basis)
Choose pivot from that column;
-z
s
a
b
3
1
t
e3
2
e5
1
e2
b
e1
e2
e3
e4
e5
0
1
0
0
-1
1
1
-1
0
0
1
-1
0
0
1
0
0
0
0
1
Now, perform: (a) row1  row1 – row2; (b) row3  row3 + row1;
7/1/2016
Combinatorial Optimization
Masters OR
a
e1
Pivoting (2)
3
1
2
s
Now, to bring e2 into the basis;
After (a) and (b);
e4
t
e3
2
e5
1
e2
b
-z
s
a
b
-4
0
1
1
e1
e2
e3
e4
e5
0
1
0
0
-1
0
1
0
0
-1
1
0
0
-1
1
1
0
0
0
1
Now, get reduced cost: (c) row0  row0 + row2;
7/1/2016
Combinatorial Optimization
Masters OR
a
e1
Pivoting (3)
-3
0
1
1
t
e3
2
e5
1
e2
b
e1
e2
e3
e4
e5
0
1
0
0
0
0
1
0
1
-1
1
0
1
-1
1
1
0
0
0
1
This solution is optimal. All relative cost (row0) are positive!
7/1/2016
e4
2
s
Now, to bring e2 into the basis;
After (a) and (b);
And (c);
-z
s
a
b
3
1
Combinatorial Optimization
Masters OR
The dual of SP
Max πs – πt
a
e1
3
1
s.t.
π ´A ≤ c ´.
π free.
2
s
t
e3
2
b
Combinatorial Optimization
Masters OR
e5
1
e2
In case of SP:
the constraints of the dual are: πi – πj ≤ cij.
7/1/2016
e4
Complementary slackness for SP
A path f as defined by primal variables, and an
assignment of dual variables π are jointly
optimal if and only if:
1. A primal variable with positive value
corresponds to a dual constraint that is
satisfied at equality.
2. A dual constraint not satisfied at equality
corresponds to a primal variable with value
zero.
7/1/2016
Combinatorial Optimization
Masters OR
Min e1 + 2e2 + 2e3 + 3e4 + e5
s.t.
e1 + e2
- e4 - e5
- e1
+ e3 + e 4
- e2 - e3
+ e5
0 ≤ ei ≤ 1, for i =1…5
a
a
e1
= 1
= -1
= 0
= 0
3
1
e4
2
s
t
3
e3
2
0
e5
1
e2
b
1
s
t
e3
2
Masters OR
e5
1
e2
Min πs – πt
s.t.
πs - πa  1
πs - πb  2
πa - πb  2
πa - πt  3
πb - πt  1
each πi  0
Compare dual variables π with label used
in 7/1/2016
Dijkstra algorithm for shortest path!
Combinatorial Optimization
e4
2
b
?
e1
3
1
Read up the following sections
on your own.
Sect 3.5: Dual Info in the Tableau
Sect 3.6: Dual Simplex Algorithm
Sect 3.7: Intrepretation of Dual Simplex Algorithm
7/1/2016
Combinatorial Optimization
Masters OR
Some Suggested Exercises:
All from [PS82]
•
•
•
•
•
•
Theorem 3.3
3
11
12
13
17
7/1/2016
Combinatorial Optimization
Masters OR
Thank you.
Q &A
Hon Wai Leong, NUS
(CS6234, Spring 2009) Page 38
Copyright © 2009 by Leong Hon Wai