Initial feasible origin:
1. Set values of original variables to zero.
2. Set values of slack variables according
to the dictionary.
The problems we have solved so far always
had an initial feasible origin.
1. Make up one problem which does NOT
have an initial feasible origin.
2.What do you look at in the initial
problem to tell if there is an initial
feasible origin or not?
1
Office hours:
MWR 4:20-5:30 inside or just outside Elliott 162tell me in class that you would like to attend.
For those of you who cannot stay:
MWR: 1:30-2:30pm. But let me know 24 hours in
advance you would like me to come in early. We will meet
outside Elliott 162- Please estimate the length of time
you require.
I am also very happy to provide e-mail help:
(wendym@csc.UVic.ca).
Important note: to ensure your e-mail gets through the
spam filter, use your UVic account. I answer all e-mails
that I receive from my students.
2
George Dantzig: Founder of the Simplex
method.
http://wiki.hsc.com/wiki/Main/ConvexOptimization
3
In Dantzig’s own words: During my first year at Berkeley I
arrived late one day to one of Neyman's classes. On the
blackboard were two problems which I assumed had been
assigned for homework. I copied them down. A few days
later I apologized to Neyman for taking so long to do the
homework - the problems seemed to be a little harder to
do than usual.
I asked him if he still wanted the
work. He told me to throw it on
his desk. I did so reluctantly
because his desk was covered
with such a heap of papers that I
feared my homework would be
lost there forever.
4
About six weeks later, one Sunday morning about eight
o'clock, Anne and I were awakened by someone banging
on our front door. It was Neyman. He rushed in with
papers in hand, all excited: "I've just written an
introduction to one of your papers. Read it so I can send it
out right away for publication." For a minute I had no idea
what he was talking about. To make a long story short, the
problems on the blackboard which I had solved thinking
they were homework were in fact two famous unsolved
problems in statistics.
5
John von Neumannn established the theory of
duality also in 1947.
He made major contributions to a vast
number of fields, including mathematics (set
theory, functional analysis, ergodic theory,
geometry, numerical analysis, and many
other mathematical fields), physics (quantum
mechanics, hydrodynamics, and fluid
dynamics), economics (game theory),
computer science (linear programming), and
statistics. He is generally regarded as one of
the greatest mathematicians in modern
history.
http://en.wikipedia.org/wiki/John_von_Neumann
John von Neumann: Dec. 28, 1903 – Feb. 8, 1957.
6
His contributions to computer science:
• mergeSort.
• established game theory as a mathematical
discipline.
• contributions to mathematics of economics.
• introduced ideas leading to Karmarker’s
algorithm.
• developed a fast method for making
pseudorandom numbers.
• first to describe a computer architecture where
the data and the program are both stored in the
computer's memory in the same address space.
• designed first template of a computer virus.
7
Other contributions:
• helped design nuclear bomb.
• member of committee responsible
Hiroshima and Nagasaki as the first
targets of the atomic bomb.
• oversaw computations related to the
expected size of the bomb blasts,
estimated death tolls, and the distance
above the ground at which the bombs
should be detonated for optimum shock
wave propagation and thus maximum
effect.
8
• Popularity sky rocketed with it was realized it
could be used to solve problems in production
management formerly tackled by hit-or-miss or
intuitive approaches.
• Awareness grew of advantages of stating
decision problems in well-defined, clear cut
terms.
• The Nobel prize in economics was awarded in
1975 to the mathematician Leonid Kantorovich
(USSR) and the economist Tjalling Koopmans
(USA) for their contributions to the theory of
optimal allocation of resources.
9
The Nobel prize in economics was awarded in 1975
to the mathematician Leonid Kantorovich (USSR)
and the economist Tjalling Koopmans (USA) for
their contributions to the theory of optimal
allocation of resources.
Kantorovich: Jan. 19, 1912,April 7, 1986.
Koopmans: Aug. 28, 1910Feb. 26, 1985.
10
Leonid Khachiyan: May 3,
1952- April 29, 2005.
The Simplex method runs very
fast in practice but has
exponential worst case time. In
1979, Khachiyan presented the
first polynomial time algorithm
(the ellipsoid method) to solve
linear programming problems
(but it was not efficient in
practice).
Leonid Khachiyan was a Soviet
mathematician of Armenian descent who
taught Computer Science at Rutgers
University.
11
Karmarker in 1984 presented the first reasonably
efficient algorithm for solving linear programs in
polynomial time.
Our text: copyright 1983.
At the time he discovered the algorithm, Narendra Karmarkar
was employed by AT&T. AT&T applied for a patent on
Karmarkar's algorithm. This left many mathematicians
uneasy, such as Ronald Rivest (himself one of the holders of
the patent on the RSA algorithm), who expressed the opinion
that research proceeded on the basis that algorithms should
be free. The patent was eventually granted but proved to be
of limited commercial value.
Narendra Karmarkar
1957http://tetrahedral.blogspot.ca/2010/10/karmarkar-algorithm-algorithms-should.html
12
The Simplex Method (algorithm we are using)
How can we solve problems which are not in
standard form?
How can we prove that the solution is optimal at
the end?
How can this be implemented in the computer?
How can numerical round off errors be mitigated?
How long does it take in the worst case?
What can we do if we do not have an initial
feasible solution?
13
Can we choose an entering variable to make the
algorithm terminate faster? How should we try
to do that?
Can we prove that there always exists a basic
feasible solution for feasible problems?
If there is more than one optimal solution, what
can the solution space look like?
How can we analyze problems given that there
could be small changes to the constraints
(without starting from scratch)?
14
How long does it take in the worst case?
Problem:As described, the Simplex
method could end up in an infinite loop!
For the following linear programming
problem, the pivot variable is chosen to
be the one with the largest positive
coefficient in the z row. After 6 pivots,
the dictionary is the same as the one we
started with. This results in an infinite
loop. The tightest constraint
corresponding to the variable with
smallest subscript is chosen to enter.
15
Input file:
4 3
10
-57
-9
0 0.5 -5.5 -2.5
0 0.5 -1.5 -0.5
1
1
0
0
-24
9
1
0
*************** Problem
Phase 1: Input dictionary.
1 ***************
X5 =
0.00- 0.50 X1 + 5.50 X2 + 2.50 X3 - 9.00 X4
X6 =
0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X4
X7 =
1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X4
--------------------------------------------------z =
0.00+ 10.00 X1 -57.00 X2 - 9.00 X3 -24.00 X4
16
The initial dictionary:
X5 = 0.00- 0.50 X1 + 5.50 X2 + 2.50 X3 - 9.00 X4
X6 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X4
X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X4
-----------------------------------------------------Z = -0.00+ 10.00 X1 - 57.00 X2 - 9.00 X3 - 24.00 X4
X1 enters.
X5 leaves. z = -0.000000
After 1 pivot:
X1 = 0.00+ 11.00 X2 + 5.00 X3 - 18.00 X4 - 2.00 X5
X6 = 0.00- 4.00 X2 - 2.00 X3 + 8.00 X4 + 1.00 X5
X7 = 1.00- 11.00 X2 - 5.00 X3 + 18.00 X4 + 2.00 X5
------------------------------------------------------z = -0.00+ 53.00 X2 + 41.00 X3 - 204.00 X4 - 20.00 X5
X2 enters.
X6 leaves. z = -0.000000
17
After
2 pivots:
X1 = 0.00- 0.50 X3 + 4.00 X4 + 0.75 X5 - 2.75 X6
X2 = 0.00- 0.50 X3 + 2.00 X4 + 0.25 X5 - 0.25 X6
X7 = 1.00+ 0.50 X3 - 4.00 X4 - 0.75 X5 + 2.75 X6
-----------------------------------------------------z = -0.00+ 14.50 X3 - 98.00 X4 - 6.75 X5 - 13.25 X6
X3 enters.
After
X1 leaves. z = -0.000000
3 pivots:
X3 = 0.00- 2.00 X1 + 8.00 X4 + 1.50 X5 - 5.50 X6
X2 = 0.00+ 1.00 X1 - 2.00 X4 - 0.50 X5 + 2.50 X6
X7 = 1.00- 1.00 X1 + 0.00 X4 + 0.00 X5 + 0.00 X6
-----------------------------------------------------z = -0.00- 29.00 X1 + 18.00 X4 + 15.00 X5 - 93.00 X6
X4 enters.
X2 leaves. z = -0.000000
18
After
4 pivots:
X3 = 0.00+ 2.00 X1 - 4.00 X2 - 0.50 X5 + 4.50 X6
X4 = 0.00+ 0.50 X1 - 0.50 X2 - 0.25 X5 + 1.25 X6
X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X5 + 0.00 X6
----------------------------------------------z = -0.00- 20.00 X1 - 9.00 X2 + 10.50 X5 - 70.50 X6
X5 enters.
After
X3 leaves. z = -0.000000
5 pivots:
X5 = 0.00+ 4.00 X1 - 8.00 X2 - 2.00 X3 + 9.00 X6
X4 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X6
X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X6
----------------------------------------------z = -0.00+ 22.00 X1 - 93.00 X2 - 21.00 X3 + 24.00 X6
X6 enters.
X4 leaves. z = -0.000000
19
After
6 pivots:
X5 = 0.00- 0.50 X1 + 5.50 X2 + 2.50 X3 - 9.00 X4
X6 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X4
X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X4
-----------------------------------------------------z = -0.00+ 10.00 X1 - 57.00 X2 - 9.00 X3 - 24.00 X4
This is the same as:
The initial dictionary:
X5 = 0.00- 0.50 X1 + 5.50 X2 + 2.50 X3 - 9.00 X4
X6 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X4
X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X4
-----------------------------------------------------z = -0.00+ 10.00 X1 - 57.00 X2 - 9.00 X3 - 24.00 X4
20
After
5 pivots:
X5 = 0.00+ 4.00 X1 - 8.00 X2 - 2.00 X3 + 9.00 X6
X4 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X6
X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X6
------------------------------------------------------z = -0.00+ 22.00 X1 - 93.00 X2 - 21.00 X3 + 24.00 X6
Using smallest subscript rule instead for the
entering variable:
X1 enters.
X4 leaves. z = -0.000000
After 6 pivots:
X5 = 0.00+ 4.00 X2 + 2.00 X3 - 8.00 X4 + 1.00 X6
X1 = 0.00+ 3.00 X2 + 1.00 X3 - 2.00 X4 - 2.00 X6
X7 = 1.00- 3.00 X2 - 1.00 X3 + 2.00 X4 + 2.00 X6
---------------------------------------------------z = -0.00-27.00 X2 + 1.00 X3 - 44.00 X4 - 20.00 X6
X3 enters.
X7 leaves. z = -0.000000
21
After 7 pivots:
X5 = 2.00- 2.00 X2 - 4.00 X4 + 5.00 X6 - 2.00 X7
X1 = 1.00+ 0.00 X2 + 0.00 X4 + 0.00 X6 - 1.00 X7
X3 = 1.00- 3.00 X2 + 2.00 X4 + 2.00 X6 - 1.00 X7
---------------------------------------------------z = 1.00-30.00 X2 -42.00 X4 -18.00 X6 - 1.00 X7
The optimal solution: 1.000000
X1 =
1.0000
X2 =
0.0000
X3 =
1.0000
X4 =
0.0000
X5 =
2.0000
X6 =
0.0000
X7 =
0.0000
22
What caused the infinite loop?
For each basis, there is a unique dictionary and
value for z. If the value for z increases at each
iteration, no dictionary can be repeated and
hence, there will be no infinite loop.
Degenerate solution: has at least one basic
variable with value 0.
Degenerate pivot: pivot that does not increase
the value for z.
Infinite loop: sequence of degenerate pivots.
23
How many different bases can there be if the
original problem has n variables and m equations?
Example given: n=4, m=3.
How can we prove that for each choice of basis,
the dictionary for that choice of basis has the
same equations?
Or equivalently, if the rows/columns are listed in
sorted order according to the subscripts of the
variables then there is a unique dictionary for
each basis.
24
Theorem: Any two dictionaries with the same
choice of basis must be the same equations.
Proof (by contradiction).
Assume not.
Consider two dictionaries D and D* which have
the same basis.
B is the set of subscripts of the basic variables.
25
Dictionary D:
xi = bi for each i ∈ B.
𝑗∉𝐵 𝑎𝑖𝑗 𝑥𝑗
----------------------------------------------------------z = v + 𝑗∉𝐵 𝑐𝑗 𝑥𝑗
Dictionary D* with the same basis B:
bi*
∗
xi =
- 𝑗∉𝐵 𝑎𝑖𝑗 𝑥𝑗 for each i ∈ B.
--------------------------------------------------------∗
*
z = v + 𝑗∉𝐵 𝑐𝑗 𝑥𝑗
26
We get from one dictionary to another by:
1. Adding the same thing to both sides of an
equation.
2. Multiplying both sides of an equation by a
non-zero constant.
3. Adding a constant multiple of one equation
to another one.
If you do these operations to a set of
equations, the set of solutions is preserved.
27
Operation 3:
(1)f(x) = b1
(2)g(x) = b2
(3)g(x) + c * f(x) = c* b1 + b2
Equations (1) and (2) have exactly the same
solutions as equations (1) and (3).
If x is a solution to (1) and (2) then clearly it
satisfies (3) and hence is a solution to (1) and(3).
If x is a solution to (1) and (3), then by (1),
c * f(x) = c * b1
and hence from (3)
g(x) = b2 so it also satisfies (1) and (2).
Equations (2) and (3) do not imply (1) if c=0.
28
Choose one non-basic variable xk and set xk=t
and set other non-basic variables to zero:
Dictionary D:
xi = bi for each i ∈ B.
𝑗∉𝐵 𝑎𝑖𝑗 𝑥𝑗
----------------------------------------------------------z = v + 𝑗∉𝐵 𝑐𝑗 𝑥𝑗
Dictionary D* with the same basis B:
bi*
∗
xi =
- 𝑗∉𝐵 𝑎𝑖𝑗 𝑥𝑗 for each i ∈ B.
--------------------------------------------------------∗
*
z = v + 𝑗∉𝐵 𝑐𝑗 𝑥𝑗
29
Choose one non-basic variable xk and set xk=t
and set other non-basic variables to zero:
Dictionary D:
xi = bi - aik t for each i ∈ B.
----------------------------------------------------------z = v + ck t
Dictionary D* with the same basis B:
xi = bi* - aik* t for each i ∈ B.
----------------------------------------------------------z = v* + ck* t
30
X5 = 1 - 2.5 X1 + 5.5 X2 - 2.5 X3 - 9.0 X4
X6 = 2 - 1.5 X1 - 1.5 X2 + 0.5 X3 - 1.0 X4 D
X7 = 3 - 1.0 X1 + 0.0 X2 + 0.0 X3 + 0.0 X4
-------------------------------------------Z =
5 +10.0 X1 - 7.0 X2 + 9.0 X3 - 4.0 X4
X5 = b5 - a5,1 X1 - a5,2 X2 - a5,3 X3 - a5,4 X4
X6 = b6 - a6,1 X1 - a6,2 X2 - a6,3 X3 - a6,4 X4 D*
X7 = b7 - a7,1 X1 - a7,2 X2 - a7,3 X3 - a7,4 X4
--------------------------------------------Z =
v +
c1 X1 + c2 X2 + c3 X3 + c4 X4
If D and D* both have X1=X2=X3=X4=0,
X5=1, X6=2 and X7=3 as solutions, what
does this say about D*?
What if they both also have X2=X3=X4=0,
X1=1 and X5= -1.5, X6= 0.5, and X7= 2? 31
What can we conclude from setting t=0?
Dictionary D:
xi = bi - aik t for each i ∈ B.
----------------------------------------------------------z = v + ck t
Dictionary D* with the same basis B:
xi = bi* - aik* t for each i ∈ B.
----------------------------------------------------------z = v* + ck* t
32
What can we conclude from setting t=1?
Dictionary D:
xi = bi - aik t for each i ∈ B.
----------------------------------------------------------z = v + ck t
Dictionary D* with the same basis B:
xi = bi* - aik* t for each i ∈ B.
----------------------------------------------------------z = v* + ck* t
33
What can we conclude from trying all choices for
a non-basic variable xk?
Dictionary D:
xi = bi - aik t for each i ∈ B.
----------------------------------------------------------z = v + ck t
Dictionary D* with the same basis B:
xi = bi* - aik* t for each i ∈ B.
----------------------------------------------------------z = v* + ck* t
34
What can we do to stop our programs from
going into an infinite loop?
35