Linear programming
Thomas S. Ferguson
University of California at Los Angeles
Compressive Sensing Tutorial PART 3
Svetlana Avramov-Zamurovic
January 29, 2009.
Definitions







A linear programming: problem of
maximizing or minimizing a linear
function subject to linear constraints.
xi ≥ 0: nonnegativity constraints.
Ax≤b: main constraints
(inequalities).
A vector x or y feasible if it satisfies
the corresponding constraints.
Constraint set is set of feasible
vectors.
LP problem is feasible if constraint set
is not empty.
The function to be maximized (or
minimized) is called the objective
function cTx.


Objective function always takes on its
maximum (minimum) value at a corner
point of the constraint set if the
constraint set is bounded.
A feasible problem is unbounded if
the objective function can assume
arbitrarily large values at feasible
vectors. Otherwise it is bounded.
STANDARD MAXIMUM PROBLEM
Find an n-vector, x  ( x1...xn )T ,
to maximize cT x  c1 x1  ...  cn xn
subject to constraints Ax  b and x1  0,..., xn  0
STANDARD MINIMUM PROBLEM
Find an m-vector, y  ( y1... ym ),
to maximize yT b  b1 y1  ...  bm ym
subject to constraints yT A  cT and y1  0,..., ym  0
(a) Some constraints may be equalities.
Solve the equality, eliminate the variable.
(b) Some variables may not be
restricted to nonnegatives.
x=u-v (u,v are nonnegative, u>0, v>0).
This adds extra variable to the problem
Duality
MAX cTx, due to Ax≤b, x>0
MIN yTb, due to yT A≥cT, y>0
x1
x2
…
xn
y1
a11
a12
…
a1n
y2
a21
a22
…
a2n
MIN variables

ym


am1
am2
…
amn
≥ c1
≥ c2
…
≥ cn
Objective function
MAX variable

≤ b1
≤ b2

≤ bm
Constraints
Duality: If you conjecture (guess) a solution to one problem,
you may solve the dual using the complementary slackness
conditions and than see if your conjecture is correct.

Thm 1:
If x is feasible for the standard MAX problem and if y is
feasible for its dual then cTx ≤ yTb.

C1:
If a standard problem and its dual are both feasible, then
both are bounded feasible.

C2:
If there exists feasible x* and y* for a standard MAX problem
and its dual such that cTx* = y* Tb than both are optimal for their
respective problems.

The Duality Thm:
If a standard LP problem is bounded feasible, than
so its dual, their values are equal, and there exist optimal vectors for both
problems.

Equilibrium Thm:
Let x* and y* be feasible vectors for a standard MAX
problem and its dual respectively. Then x* and y* are optimal if and only if
y  0 for all i for which
*
i
n
a
j 1
ij
x  bi and x  0 for all j for which
*
j
*
j
m
y a
i 1
*
j ij
 cj
Complementary slackness conditions:
strict inequality (slackness) in a standard problem constraint implies that
complementary constraint in the dual be satisfied with equality
The Pivot Operation
y1a11    yi ai1   ym am1  s1
The realtionsh ip between aij (pivot) and aˆij is

y1a1 j    yi aij   ym amj  s j
aˆij 

y1a1n    yi ain   ym amn  sm
Interchang e indipenden t variable yi with s j
y1aˆ11    s j aˆi1   ym aˆm1  s1

y1aˆ1 j    s j aˆij   ym aˆmj  yi

y1aˆ1n    s j aˆin   ym aˆmn  sm
1
aij
aˆhj  
aˆik 
(at pivot location)
ahj
aij
for h  i
aik
for k  j
aij
aˆik  ahk 
aik ahj
aij
(elements in the pivot column)
(elements in the pivot row)
for k  j and h  i (all other elements)
Pivoting results in matrix inverse
(special case)
yT A= sT => yT =A-1 sT
Linear programming
A system of linear
inequalities
defines a polytope as
a feasible region.
The simplex algorithm
begins at a starting vertex
and moves along the
edges of the polytope
until it reaches
the vertex of
the optimum solution.
LP MIN: Find y to minimize yTb subject to y≥0 and yTA ≥ cT
Simplex method
Find y and s to minimize yTb subject to y≥0, s≥0 and sT = yTA-cT
Slack variable
s1
s2
…
sn
y1
a11
a12
…
a1n
y2
a21
a22
…
a2n
MIN variables




b1
b2

ym
am1
am2
…
amn
bm
1
-c1
-c2
…
-cn
0
Objective function
Simplex Tableau sT = yTA-cT
Simplex method
is pivot madly method that tells you which points to choose
as pivots to approach the solution systematically.
 If -c≥0 and b≥0, there is an obvious solution to the problem:
the minimum occurs at y=0; and s=-c and the minimum
value yTb=0.
 Assume: there is at least one nonnegative entry in the last
column or last row (exclusive of the corners). If we pivot
about a11, including the last column and the last row we have
r=(y1, s2,…, sn) and t=(s1,y2, …,ym)

Find vectors y and s to minimize tTb subject to y≥0, s≥0 and
r=tTA^-c^
 The Pivot Madly method:
Pivot madly until you suddenly find -c^≥0 and b^≥0
Then, obtain a feasible point by setting r=-c and t=0
The value is the lower right corner.
r
t
A
b
-c
v
LP Max problem (slack variable u=b-Ax, same tableau if –u=Ax-b)
Pivot rules for simplex method
from the point of the MAX problem

b≥0. Take any column with last entry negative, say -cj0<0.
Among those i for which ai,j0>0, choose that i0 for which the
ratio bj/ai,j0 is smallest. If there are ties, choose any such
i0.Pivot around ai0,j0.


After pivoting the b column stays nonnegative, so you still have a
feasible point for the maximum problem.
The value of the new tableau is never less than the old (generally
greater.)



b^i=bj-ai,j0bj0/ ai0,j0 and b^i=bj0/ ai0,j0
v^=v-(cj0)bj0/ ai0,j0
Some bi are negative. Take the first negative bj, say bk<0
(where b1, … bk-1≥0). Find any negative entry in row k, ak,j0<0.
The pivot will be column j0. Compare bk/ak,j0 and bj/ai,j0 for which
bj>0 and ai,j0>0 and choose i0 for which this ratio is smallest (i0
may be equal to k). You may choose any such i0 if there are
ties. Pivot on ai0,j0 .
George Dantzig in 1947
The algorithm moves along the edges of the polyhedron defined by the constraints, from one
vertex to another, while decreasing the value of the objective function, cTx at each step.
Pivot rules for simplex method
from the point of the MIN problem
 -c≥0. Take any column with last entry negative, say
bi0<0. Among those j for which ai0,j<0, choose that j0 for
which the ratio -cj/ai0,j is closest to 0. If there are ties,
choose any such j0.Pivot around ai0,j0.
 Some -ci are negative. Take the first negative -ci , say ck<0 (where -c1, … -ck-1≥0). Find any positive entry in
coulmn k, ai0,k>0. Compare -ck/ai0,k and -cj/ai0,j for which
-cj>0 and ai0,j<0 and choose j0 for which this ratio is
clossest to zero (j0 may be equal to k). Pivot on ai0,j0 .