EE/Econ 458
The Simplex Method
J. McCalley
1
An approach
5
2
4
2
6
2
100
90
3
2
7
2
80
70
60
8
2
2
9
1
10
2
50
40
30
20
5
10
2
Definitions
Adjacent corner points are connected by a single line
segment on the boundary of the feasible region.
One corner point is better than another if it has a
higher value of the objective function f.
3
An approach
1. Pick a corner point at random.
2. Move to an adjacent corner point that is better.
a. If there are two that are better, move to the one that is best.
b. If there are no better adjacent corner points, the current
corner point is the solution to the problem.
Optimality (stopping) condition: If a corner point feasible solution is
equal to or better than all its adjacent corner point feasible solutions,
then it is equal to or better than all other corner point feasible
solutions, i.e., it is optimal.
Main ideas of proof:
1. If objective function monotonically increases (decreases) in some direction within the
decision-vector space, then each adjacent corner point will become progressively better in
the direction of objective function increase (decrease) such that the last corner point must
have two adjacent corner points that are worse.
2. The monotonicity of objective function increase (decrease) is guaranteed by its linearity.
4
Standard Form – non-negativity constraints
Many problems require non-negativity
constraints on decision variables. SCED
is like that due to requirement that
generation & demand be positive.
min  f ( x)
s.t. h( x )  c
g ( x)b
Simplex method requires non-negativity on
decision variables to ensure a bounded region.
When decision variable must allow negativity, they can be converted
into one, or two decision variables with non-negativity constraints.
x j  10  x j  xj 10, xj  0
x j    xj  xj  xj , xj  0, xj  0,
min f ( x)
s.t. h( x )  c
g ( x)b
x0
5
Standard Form – eliminating equalities
min  f ( x)
s.t. h( x )  c
g ( x)b
Equality constraints:
h( x )  c
h( x )  c 
h( x )  c
h( x )  c 
h( x )  c
 h( x )  c
x0
min  f ( x)
g ( x ) b
x0
6
Slack Variables
max F  3 x1  5 x2
s.t. x1
4
x3=4-x1
2 x2  12
3 x1  2 x2  18
x1  0, x2  0
x3 is the slack variable.
It gives “slack” between two
sides of the inequality x1<4.
We can replace the first
inequality x1<4 with
x1  x3  4,
x3  0
If “slack” is zero, then
inequality is satisfied with
equality.
And we can do that with all
inequalities, leading to…
Observe that slack variables
cannot be negative, because
then the inequality would be
violated.
7
Equality Form and Augmented Solution
max F  3x1  5 x2
s.t .
 x3
x1
3x1
2 x2
 2 x2

 x4
 x5
4
 12
 18
Here, we have introduced slack
variables within all inequalities
x1  0, x2  0, x3  0, x4  0, x5  0
EQUALITY FORM: Has all inequality constraints converted
to equality constraints via introduction of slack variables.
AUGMENTED SOLUTION: A solution to the LP that
includes values of the decision and slack variables.
Solution:
(x1,x2)=(3,2)
Augmented solution:
(x1,x2, x3, x4, x5)=(3,2,1,8,5).
8
max F  3x1  5 x2
s.t .
 x3
x1
3x1
2 x2
 2 x2

 x4
 x5
Basic & basic feasible
solutions
4
 12
 18
x1  0, x2  0, x3  0, x4  0, x5  0
Basic solution: An augmented
corner-point solution, e.g.,
(x1,x2,x3,x4,x5)=(4,6,0,0,-6).
Basic solutions may be
feasible or infeasible.
Basic Feasible Solution: A
feasible augmented
corner-point solution, e.g.,
(x1,x2,x3,x4,x5)=(0,6,4,0,6).
9
Positive right-hand-sides
min  f ( x)
g ( x ) b
We require b ≥ 0
x0
If there is a bi<0, then we can multiply the corresponding
constraint by -1 and introduce a slack variable. See end of
LPSimplex1 notes for an example.
10
High-Level Version of Simplex Method
1. Initialization: Start at a corner point solution.
2. Iterative step: Move to a better adjacent corner point
feasible solution.
3. Optimality test: Determine if the current feasible
corner point is optimal using our optimality test (if
none of its adjacent feasible corner points are better,
then the current feasible corner point is optimal).
a. If the current feasible corner point is optimal, the
solution has been found, and the method
terminates.
b. If the current feasible corner point is not optimal,
then go to 2.
11
Initialization
max F  3x1  5 x2
s.t .
 x3
x1
3x1
2 x2
 2 x2

 x4
 x5
4
 12
 18
x1  0, x2  0, x3  0, x4  0, x5  0
xn+k is the slack variable.
a1x1  a2 x2  ...  an xn  xn  k  bk
Above equation is satisfied by
below basic solution
x1  0, x2  0,..., xn  0, xn k  bk
Because bk ≥0, we are assured
xn+k≥0, which satisfies variable
non-negativity.
And so an initial BFS is found by
letting all decision variables be
zero (the origin!).
(x1, x2, x3, x4, x5)=(0,0,4,12,18)
is the initial BFS.
12
Basic & nonbasic variables
max F  3x1  5 x2
s.t .
 x3
x1
3x1
2 x2
 2 x2

 x4
 x5
4
 12
 18
Our initial BFS:
(x1, x2, x3, x4, x5)=(0,0,4,12,18)
x1  0, x2  0, x3  0, x4  0, x5  0
For any BFS
Basic variables: not 0.
Nonbasic variables: 0
n: number of decision variables.
m: number of constraints.
We will always have n+m variables
in a BFS:
n are non-basic
m are basic
13
High-Level Version of Simplex Method
1. Initialization: Start at a corner point solution.
2. Iterative step: Move to a better adjacent corner point
feasible solution.
3. Optimality test: Determine if the current feasible
corner point is optimal using our optimality test (if
none of its adjacent feasible corner points are better,
then the current feasible corner point is optimal).
a. If the current feasible corner point is optimal, the
solution has been found, and the method
terminates.
b. If the current feasible corner point is not optimal,
then go to 2.
14
Iterative step
Move to a better adjacent corner point feasible solution.
Two feasible corner points are
said to be adjacent if
• They satisfy all constraint
equations.
• The constraint equations which
define them are the same, with
exception of just one
they share n-1 constraint
equations, with one
constraint equation for one
corner point differing from
one constraint equation for
the other corner point.
A feasible corner point is the
simultaneous solution of a set
of n constraint equations that
does not violate any
constraint equations.
15
Iterative step
Crnr pnt feasible soltns
(0,0)
1
2
3
4
1
2
(0,6)
3
(2,6)
4
(4,3)
5
(4,0)
5
Defining cnstrnt eqtns
x1=0
x2=0
x1=0
2x2=12
2x2=12
3x1+2x2=18
3x1+2x2=18
x1=4
x1=4
x2=0
5 feasible corner points
16
Iterative step
Move from 1 to 2.
Crnr pnt feasible soltns
(0,0)
1
2
3
4
1
5
2
(0,6)
3
(2,6)
4
(4,3)
5
(4,0)
Defining cnstrnt eqtns
x1=0
x2=0
x1=0
2x2=12
2x2=12
3x1+2x2=18
3x1+2x2=18
x1=4
x1=4
x2=0
Start at (0,0).
1. The x2=0 constraint becomes inactive.
2. Move along the x1=0 constraint until you reach the next
corner point, which is (0,6).
3. The 2x2=12 constraint becomes active.
17
Iterative step
Move from 2 to 3.
Crnr pnt feasible soltns
(0,0)
1
2
3
4
1
5
2
(0,6)
3
(2,6)
4
(4,3)
5
(4,0)
Defining cnstrnt eqtns
x1=0
x2=0
x1=0
2x2=12
2x2=12
3x1+2x2=18
3x1+2x2=18
x1=4
x1=4
x2=0
1. The x1=0 constraint becomes inactive.
2. Move along the 2x2=12 constraint until you reach the
next corner point, which is (2,6).
3. The 3x1+2x2=18 constraint becomes active.
18
Iterative step
Move from 3 to 4.
Crnr pnt feasible soltns
(0,0)
1
2
3
4
1
5
2
(0,6)
3
(2,6)
4
(4,3)
5
(4,0)
Defining cnstrnt eqtns
x1=0
x2=0
x1=0
2x2=12
2x2=12
3x1+2x2=18
3x1+2x2=18
x1=4
x1=4
x2=0
1. The 2x2=12 constraint becomes inactive.
2. Move along the 3x1+2x2=18 constraint until you reach
the next corner point, which is (4,3).
3. The x1=4 constraint becomes active.
19
Iterative step
Move from 4 to 5.
Crnr pnt feasible soltns
(0,0)
1
2
3
4
1
5
2
(0,6)
3
(2,6)
4
(4,3)
5
(4,0)
Defining cnstrnt eqtns
x1=0
x2=0
x1=0
2x2=12
2x2=12
3x1+2x2=18
3x1+2x2=18
x1=4
x1=4
x2=0
1. The 3x1+2x2=18 constraint becomes inactive.
2. Move along the x1=4 constraint until you reach the next
corner point, which is (4,0).
3. The x2=0 constraint becomes active.
20
Iterative step
Basic feasible solutions (BFSs)
Basic feasible soltns
(0,0,4,12,18)
1
2
3
2
(0,6,4,0,6)
3
(2,6,2,0,0)
4
(4,3,0,6,0)
5
(4,0,0,12,6)
4
1
5
Defining cnstrnt eqtns
x1=0
x2=0
x1=0
2x2+x4 =12
2x2+x4 =12
3x1+2x2+x5=18
3x1+2x2+x5=18
x1+x3=4
x1+x3=4
x2=0
Observation 1:
The slack variable, for a particular corner point, is the weighted
distance between that corner point and the slack variable’s
constraint (weights are the coefficients from constraint equation).
For the constraint corresponding to x1<4 (or x1+x3=4), because the
coefficient for x1 is 1, the slack variable is exactly the “distance”
between the given corner point and this constraint (from Table 2,
this would be 4, 4, 2, 0, 0 for points 1, 2, 3, 4, 5, respectively).
21
Iterative step
Basic feasible solutions (BFSs)
2
Basic feasible soltns
(0,0,4,12,18)
1
3
4
1
2
(0,6,4,0,6)
3
(2,6,2,0,0)
4
(4,3,0,6,0)
5
(4,0,0,12,6)
5
Defining cnstrnt eqtns
x1=0
x2=0
x1=0
2x2+x4 =12
2x2+x4 =12
3x1+2x2+x5=18
3x1+2x2+x5=18
x1+x3=4
x1+x3=4
x2=0
Observation 2:
All BFSs have exactly n variables equal to 0 because any BFS is the
simultaneous solution of a system of n constraint equations. There are
2 types of such constraint equations that can define a BFS:
• Nonnegativity constraints on decision variablesDV is 0 in the BFS
• Constraints with slack variablesSV is 0 in the BFS
Any BFS has n variables 0. Any BFS has n non-basic variables.
22
Iterative step
Basic feasible solutions (BFSs)
Basic feasible soltns
(0,0,4,12,18)
1
2
3
2
(0,6,4,0,6)
3
(2,6,2,0,0)
4
(4,3,0,6,0)
5
(4,0,0,12,6)
4
1
5
Observation 3:
The way we move from one BFS
to another is by exchanging
exactly one zeroed variable with
a non-zeroed variable.
Defining cnstrnt eqtns
x1=0
x2=0
x1=0
2x2+x4 =12
2x2+x4 =12
3x1+2x2+x5=18
3x1+2x2+x5=18
x1+x3=4
x1+x3=4
x2=0
The way we move from one
BFS to another is by
exchanging exactly one nonbasic variable with a basic
variable.
23
Iterative step
One constraint equation
becomes inactive while
one constraint equation
becomes active.
One variable enters the
basis (becomes nonzero)
while one variable leaves
the basis (becomes zero).
Three questions:
1. How is the entering basic variable identified?
2. How is the leaving basic variable identified?
3. How is the new basic feasible solution found?
24
Iterative step
1. How is the entering basic variable identified?
The candidates for the entering
basic variable are the n nonbasic
variables. In the first step of our
example (solution 1 to 2), the
candidates are x1 and x2.
Criterion is:
Select the one that improves the
objective at the highest rate (i.e.,
the largest amount of objective
per unit change in variable).
Basic feasible soltns
(0,0,4,12,18)
1
2
(0,6,4,0,6)
3
(2,6,2,0,0)
4
(4,3,0,6,0)
5
(4,0,0,12,6)
Objective
Defining cnstrnt eqtns
x1=0
x2=0
x1=0
2x2+x4 =12
2x2+x4 =12
3x1+2x2+x5=18
3x1+2x2+x5=18
x1+x3=4
x1+x3=4
x2=0
F  3x1  5x2
Increasing either variable, x1 or x2,
increases the objective, but x2
increases it the most for a given unit
change, since 5>3; so x2 is the
entering variable.
25
Iterative step
2. How is the leaving basic variable identified?
Basic feasible soltns
(0,0,4,12,18)
1
Defining cnstrnt eqtns
x1=0
x2=0
x1=0
2x2+x4 =12
2x2+x4 =12
3x1+2x2+x5=18
3x1+2x2+x5=18
x1+x3=4
x1+x3=4
x2=0
The candidates for the leaving basic
(0,6,4,0,6)
variable are the m basic variables.
2
In the first step of our example
(2,6,2,0,0)
3
(solution 1 to 2), the candidates are
x3, x4, and x5. The balance between 4 (4,3,0,6,0)
how much the entering variable
(4,0,0,12,6)
5
may increase without pushing any
current basic variable negative is
Recall initial point, origin, has all n DV zero and all m SV non-zero. There is 1
controlled by our m constraints. SV per constraint, so each constraint contains exactly 1 basic variable at
Criterion is: Choose the
leaving variable to be the
basic variable that hits 0
first as the entering
variable is increased, as
dictated by one of the m
constraint equations.
initialization. Since iterations always cause each constraint to exchange 1 basic
and 1 non-basic variables, each constraint always has just 1 basic variable.
Recall x2 is our entering variable. We want to identify the
constraint, when its basic variable is 0, most limits x2.
Basic
Constraint
variable
x3
x1+x3=4
x4
2x2+x4=12
x5
3x1+2x2+x5=18
Upper bound for x2
No limit imposed
x2=(12-0)/2=6
x2=(18-3(0)-0)/2=9
26
Iterative step
2. How is the leaving basic variable identified?
Basic
Constraint
variable
x3
x1+x3=4
x4
2x2+x4=12
x5
3x1+2x2+x5=18
Upper bound for x2
No limit imposed
x2=(12-0)/2=6
x2=(18-3(0)-0)/2=9
The above table tells us the following:
• We can increase x2 up to 6, but if we go further, x4 will become negative.
• We can increase x2 up to 9, but if we go further, x5 will become negative.
So we cannot increase x2 beyond 6, and x4 becomes the leaving variable
x4 becomes 0 in the new BFS.
(x4 becomes non-basic in the new BFS)
and of course, x2 becomes non-zero
(x2 becomes basic).
27
Iterative step
3. How is the new basic feasible solution found?
Basic
Constraint
variable
x3
x1+x3=4
x4
2x2+x4=12
x5
3x1+2x2+x5=18
Upper bound for x2
No limit imposed
x2=(12-0)/2=6
x2=(18-3(0)-0)/2=9
The new BFS could be found from:
• we know x1 is still non-basic and therefore x1=0
• x4 just became non-basic and therefore x4=0
• the most limiting constraint in the above table, x2=6
• the other constraints:
Row 1 of table: x3=4-x1=4-0=4
Row 3 of table: x5=18-3x1-2x2=18-3(0)-2(6)=6
(x1,x2,x3,x4,x5)=(0,6,4,0,6), and we are done with this iteration.
Then check for optimality.
28
Iterative step
3. How is the new basic feasible solution found?
But we need a more structured way (that we can teach to a computer).
Write the objective function like this:
F  3x1  5x2  0
F  3x1  5x2
For the initial solution, we can write F together with constraint equations:
F
 3x1
x1
3x1
 5 x2
 x3
2 x2
 2 x2
 x4
 x5
 0
 4
 12
 18
(7)
(8)
(9)
(10)
The above is set up for our initial solution: (x1, x2, x3, x4, x5)=(0,0,4,12,18). We
observe values of the basic variables are the right-hand-sides of (8), (9), (10).
Because x1 and x2 are non-basic (0), the right-hand-side of (7) is the objective
value at this solution.
29
Iterative step
3. How is the new basic feasible solution found?
F
 3x1
x1
3x1
 5 x2
 x3
2 x2
 2 x2
 x4
 x5
 0
 4
 12
 18
(7)
(8)
(9)
(10)
Goal: Have x2 enter the basis & x4 leave the basis so that, at the new
solution, our system of equations is in the same form as above where:
• the values of the basic variables at the solution can be directly read
off as the right-hand-sides of those equations and
• the value of the objective at that solution can be directly read off as
the right-hand-side of the objective equation.
To accomplish this…
• Because x2 is going to be non-zero, it must show up in only one
constraint equation, with a 1 as its coefficient (so it can be directly read off),
and it must not appear in the objective equation (since it will not be 0).
• Because x4 is going to be 0, it must appear in the objective equation
(so the right-hand-side of the objective equation will directly give F).
30
Iterative step
3. How is the new basic feasible solution found?
F
 3x1
x1
3x1
Entering variable: x2
Leaving variable: x4
 5 x2
 x3
2 x2
 2 x2
1  3  5
0 1
0

0 0
2

2
0 3
 x4
 x5
0 0
1 0
0 1
0 0
 0
 4
 12
 18
(7)
(8)
(9)
(10)
F 
0  x1   0 
0  x2   4 
 
0  x3  12

 
1  x4  18
 
 x5 
We want to eliminate (make 0) the coefficient of x2 (-5) in row 1.
We will use row 3 as our “pivot row” (and a33 our pivot) so that the
leaving variable (x4) appears in row 1 after eliminating x2.
31
Iterative step
3. How is the new basic feasible solution found?
Join the matrix with the right-hand-side vector, like this
1  3  5
0 1
0

0 0
2

2
0 3
0 0
1 0
0 1
0 0
F 
0  x1   0 
0  x2   4 
 
0  x3  12

 
1  x4  18
 
 x5 
1  3  5
0 1
0

0 0
2

2
0 3
0
1
0
0
Pivot
0
0
1
0
0 0
0 4 
0 12  Pivot
row

1 18
Use Gaussian Elimination! For an n × n matrix:
Choose the row from which the leaving variable was identified to be
the pivot row. Let this be row k so that akk is the pivot.
1. Divide row k by akk.
2. Eliminate all ajk, j=1,…,n except j=k. This means to make all elements
directly above and beneath the pivot equal to 0 by adding an
appropriate multiple of the pivot row to each row above or beneath
32
the pivot.
Iterative step
3. How is the new basic feasible solution found?
1  3  5
0 1
0

0 0
2

2
0 3
0 0
1 0
0 1
0 0
1  3  5
0 1
0

0 0
1

2
0 3
1  3
0 1

0 0

0 3
0
0
1
0
0
1
0
0
F 
0  x1   0 
0  x2   4 
 

0  x3  12

 
1  x4  18
 
 x5 
0 0 0
0 0 4 
0.5 0 6  

0 1 18
0 2.5 0 30
1 0 0 4 
0 0.5 0 6   2 

0 1 1 6  
1  3  5
0 1
0

0 0
2

2
0 3
1  3
0 1

0 0

0 3
0
0
1
2
0
1
0
0
0
0
1
0
0 0
0 4 
0 12 

1 18
0 2.5 0 30 
1 0 0 4 
0 0.5 0 6   5 

0 0 1 18
Recalling (x2, x3, x5) are basic variables
(with non-zero values) and (x1, x4) are
non-basic variables (with zero values),
one can immediately read off
x3=4 (from second row)
x2=6 (from third row)
x5=6 (from fourth row)
33
Iterative step
3. How is the new basic feasible solution found?
1  3
0 1

0 0

0 3
0
0
1
0
0 2.5 0 30
1 0 0 4 
0 0.5 0 6   2 

0 1 1 6  
F  3x1
x1
x2
3x1
 2.5 x4
 x3
 0.5 x4
 x4
 x5
 30
 4
 6
 6
(12)
(13)
(14)
(15)
Recalling (x2, x3, x5) are basic variables
(with non-zero values) and (x1, x4) are
non-basic variables (with zero values),
one can immediately read off
x3=4 (from second row)
x2=6 (from third row)
x5=6 (from fourth row)
34
Iterative step – Key ideas
How to identify the entering variable?
Select the variable that improves the objective
at the highest rate (i.e., the largest amount
of objective per unit change in variable).
How to identify the leaving variable?
Choose the leaving variable to be the one that
hits 0 first as the entering variable is increased,
as dictated by one of the m constraint equations.
How is the new BFS found?
Using the equation used to identify the leaving variable as
the pivot row, eliminate the entering variable from all
other equations.
35
Optimality test
If there are any remaining variables with positive coefficients
in current objective function expression, then the current
solution may still be improved; take another iteration.
If not, the current solution may not be further improved, and
it is therefore optimal.
From example:
F
 3x1
x1
 2.5 x4
 x3
x2
3x1
 0.5 x4
 x4
 x5
 30
 4
 6
 6
(12)
(13)
(14)
(15)
In the right-hand-form, optimality occurs
when all coefficients are negative.
In the left-hand-form, optimality occurs
when all coefficients are negative.
F  30  3x1  2.5x4
Since x1 has a positive
coefficient, F may still be
improved. Therefore this
solution is not optimal.
36