Linear Programming and
Simplex Algorithm
Reference:
Numerical Recipe Sec. 10.8
Content
•
•
•
•
Linear programming problems
Simplex algorithm
QSOpt solver
LP Applications
– Collision detection
– Melodic similarity
– Underdetermined linear systems
2
The Diet Problem
• Dietician preparing a diet consisting of two
foods, A and B.
A
B
Cost
0.60
0.40
Protein
20
30
Fat Carbohydrate
12
30
6
15
Minimum requirement: protein 60g; fat 24g; carbohydrate 30g
Looking for minimum cost diet
3
Linear Programming
min C  0.60x1  0.40x2
s.t. 20x1  30x2  60
12x1  6 x2  24
30x1  15x2  30
where
x1 , x2  0
Cp. NLP
min f ( x)
subject to
h( x )  0
g ( x)  0
4
The Problem
Maximize
Subject to
N: dimension of x
M: number of constraints
(M = m1+m2+m3)
5
Theory of Linear Optimization
Terminology:
Feasible vector
Feasible basic vector
Optimal feasible vector
6
Theory (cont)
• Feasible region is convex and bounded by
hyperplanes
• Feasible vectors that satisfy N of the original
constraints as equalities are termed feasible basic
vectors.
• Optimal occur at boundary (gradient vector of
objective function is always nonzero)
• Combinatorial problem: determining which N
constraints (out of the N+M constraints) would be
satisfied by the optimal feasible vector
7
Simplex Algorithm (Dantzig 1948)
• A series of combinations is tried to increase the
objective function
• Number of iterations less than O(max(M,N))
• Worst case complexity is exponential (in number
of variables); yet has polynomial smoothed
complexity (and works well in practice)
8
The Canonical Problem
Maximize
Subject to
N: dimension of x
M: number of constraints
(M = m1+m2+m3)
I.
II.
III.
9
QSopt [ref]
• Contains a GUI
and a callable
library
11
LP Format Example
min C  0.60x1  0.40x2
s.t. 20x1  30x2  60
12x1  6 x2  24
30x1  15x2  30
where
x1 , x2  0
Somehow
x1 >= 0
x2 >= 0
Doesn’t work!?
Problem smallExample
Minimize
obj: 0.60x1 + 0.40x2
Subject to
c1: 12x1 + 6x2 >= 24
30x1 + 15x2 >= 30
Bounds
0 <= x1
0 <= x2
End
BNF for QSOpt scripts
12
QSOpt Examples
13
Duality
• Primal problem
• Dual problem
14
15
Duality
• If a linear program has a finite optimal
solution then so does the dual. Furthermore,
the optimal values of the two programs are
the same.
• If either problem has an unbounded optimal
solution, then the other problem has no
feasible solutions.
16
Diet Problem
min C  0.60x1  0.40x2
maxC  0.60x1  0.40x2
s.t. 20x1  30x2  60
s.t. 20x1  30x2  s1  60
12x1  6 x2  24
12x1  6 x2  s2  24
30x1  15x2  30
30x1  15x2  s3  30
where
x1 , x2  0
where
x1 , x2 , s1 , s2 , s3  0
17
Applications
The Problem
Ax  b
min x
Under-determined system
(infinite number of solutions)
Find the solution with minimum
magnitude (i.e., closest to origin)
19
Example
 2
 x1  x2

x2  x3  0

Approach 1: find complete solution
1 1 0
0 1 1

 2
0 
1
  2
 
 
xn  c  1, x p   0 
1
 0 
 
 
xc  x p  xn
(-2,0,0) subtract its
projection along (1,-1,1)
(-2,0,0)
(1,-1,1)
1
 1 1 1 
1   T 1
24

q
  1, qq   1 1  1
9
3
3 
 1  1 1 
1
  2
 1  1 1   2    2   23   34 
  1
 0    0     2     2 
0


1
1

1
 

     3   3 
3
 0 
 1  1 1  0   0   23   23 
20
 
d
Linear Programming Solution
 Ax  b

min a where    xi  
 ,x
  0

• Most simplex algorithm assume nonnegative
variables
• Set xi  yi  zi
where yi  0 and zi  0 xi  R 
21
Example
min 3
subject to:
z1  y1   z2  y2   2
 z 2  y 2    z 3  y3   0
zi  yi    , i  1,2,3
zi  yi    , i  1,2,3
  0, yi  0, zi  0
xi  zi  yi
 2
 x1  x2

x2  x3  0

min 3
 y1  z1    0
 y2  z 2    0
 y3  z 3    0
y1  z1    0
y2  z 2    0
y3  z 3    0
 y1  z1    y2  z2   2
 y 2  z 2    y3  z 3   0
22
Solved by QSopt
d 3
The real minimum (-4/3, -2/3, 2/3)
The LP constraint is not exactly ||x||2
(but ||x||)
This problem can also be solved by SVD (singular
value decomposition)
23
LP in CD (narrow phase, ref)
• A pair of convex objects: each facet
represented by the plane inequality
aix+biy+ciz  di
• If two sets of inequality (from two
convex objects) define a feasible region,
then a collision is detected
• The type of optimization (min|max) and
the objective function used is irrelevant.
24
LP for CD
Convert to dual problem
for better efficiency(?!)
25
Transportation Distance as a
Measure to Melodic Similarity
Ref: Typke et al. 2003
Example
Partially
matched
27
Earth Mover Distance
A  a1 ,, am  weightedpointset
ai  xi , wi , xi  R 2 , wi  R   {0}
W   wi
B  b1 ,, bn  weightedpointset
b j   y j , u j , y j  R 2 , u j  R   {0}
U  u j
f ij : flow of weight from xi to y j over thedistanced ij
28
EMD as Linear Program
Solve by
simplex
algorithm
29
EMD for Melodic Similarity
• Ground distance: Euclidean distance
dij 
t  t    p  p 
2
i
j
2
i
j
• Scale time coordinate (so that they are
comparable)
• Transposition: transpose one of the
melodies so that the weighted average pitch
is equal (not optimum but acceptable)
30
HW: Verify planar CD using LP
Warning:
Non-negative assumption
(for some solvers)!!
31
Verify planar CD using LP.
5
2
1. y  3
[II] y  s1  3
2. x  3
[II] x  s2  3
3. x  y  11[I]  x  y  s3  11
6
4. y  5
5. x  10
4
1
[II] y  s4  5
[I]  x  s5  10
6. x  y  0[II] x  y  s5  0
3
32
LP Solvers
• Lp_solve: MILP
• QSopt
• GLPK
33
QSopt [ref]
• Contains a GUI
and a callable
library
35
LP Format Example
min C  0.60x1  0.40x2
s.t. 20x1  30x2  60
12x1  6 x2  24
30x1  15x2  30
where
x1 , x2  0
[weird] Somehow
x1 >= 0
x2 >= 0
Doesn’t work!?
Problem smallExample
Minimize
obj: 0.60x1 + 0.40x2
Subject to
c1: 12x1 + 6x2 >= 24
30x1 + 15x2 >= 30
Bounds
0 <= x1
0 <= x2
End
36
LP format BNF definition1
37
LP format BNF definition2
38
QSOpt API
Scenario: Read and Solve
40
Accessing a Solution
• value: objective
function
• x: solution vector
(columns)
• pi: dual variables
• slack: slack variables
• rc: reduced cost
• In our applications, we
don’t really care about
pi, slack, and rc.
• Set them to NULL
41
QS_MAX
or
QS_MIN
cm at
3.1 2.3 1.4
5.0 1.1 0 


cmatind
0 0 0
1 1
# of nonzeros in jth column
Location of start entry
42
Modify an LP Problem
•
•
•
•
•
•
•
•
•
•
QSnew_row
QSadd_rows
QSnew_col
QSadd_cols
QSdelete_row
QSdelete_col
QSchange_coef
QSchange_objcoef
QSchange_rhscoef
QSchange_sense
• See reference manuals
for more details (Ch.5)
• Use QSwrite_prob to
verify the change
43