Introduction
5-Steps Method :
1. Question
2. Approach
3. Formulate
4. Solve
5. Interpret
Example #1
A PC manufacturer currently sells 10,000 units/ month of a certain model.
The manufacturing cost is $ 700/ unit. The whole sale price is $950/ unit. In a
few test markets, the price was discounted by $100, with result of 50% sales increase. The advertising cost at present are $50,000/ month. Research by the ad
agency shows an increase of advertising budget by $10,000 would increase sales by
200/month. (The company will not increase budget above $100,000).
Variables:
n = 10, 000 units/ month - current sales
m = $700/ unit - manufacturing cost
p = $950/ unit - price
D = units of discount ($100/unit)
r = 50% increase in sales a = $50, 000/ month - advertising
b - ad budget increase = b · $10, 000
c = 200/month - increase in sales
Goal:
Maximize P the total profit
Assumptions:
N = total number of units sold
N = n · (1 + D(0.50)) + 200
| {z· b}
|
{z
}
sold by increasing discount
q = the profit/unit
1
increase due to advertising
q = (p − m) − D
| ·{z100}
| {z }
net profit
Case A: No ad increase
The total profit is:
discount
P = 10, 000 · (1 + D · (0.50)) · (250 − D · 100) −50, 000
|
{z
} |
N
{z
q
}
Maximize the profit:
dP
= n · (0.50) (250 − D · 100) − 100 · n · (1 + D · (0.50) ) = 0
| {z }
| {z }
dD
r
r
(1)
125 − 100 − 50 · D − 50 · D = 0
So 100D = 25 =⇒ D = 0.25
d2 P
< 0 =⇒ Max.
dD 2
Interpret: The price has to be discounted by $25 to optimize the total profit P .
Note: P = 10, 000(1.125)(225) − 50, 000 = 11, 250(225) − 50, 000 = 2, 481, 250.
dP
From
= 0 we have
dD
−200rD + 250r − 100 = 0
250
1
250r − 100
=
−
is
Sensitivity to r (rate of increase due to discount) of D =
200r
200 2r
Note:
r dD
r
r
1
(250r − 100) · 200 250 · (r · 200)
1
S(D, r) =
=
+
=
−
= 2·
2
2
D dr
D
(200r)
(200r)
2r D
2rD
S(D, r) =
(FIGURE)
∆D
D
∆r
r
= (normalized changes in D)/ (normalized changes in r)
2
Note for
r = 0.4 =⇒ D = 0
r = 0.6 =⇒ D = 0.4167
dP
= nDq = 10, 000 · D · (250 − 100D) = 2500 · 225
dr
So the sensitivity of P to r is
r
· 2500 · 225 = 0.1
S(P, r) =
P
(FIGURE)
Case B: Include the effect of advertising
P (b, D) = [n(1 + rD) + |{z}
200 ·b] · [250 − D · 100] − b · 10, 000 − 50, 000
c
Maximize P (b, D) by using Lagrange multipliers:
∂P
= nr[250 − D · 100] − 100[n(1 + rD) + cb]
∂D
∂P
= 200[250 − 100D] − 10, 000 = 50, 000 − 100Dc
∂b
∂P
> 0 for 0 ≤ D ≤ 2.5.
∂b
We also know that the company will not increase budget above $100,000, so 0 ≤
b ≤ 10.
First find the critical points that satisfy the inequalities by solving the system:
Note that
∂P
=0
∂D
,
3
∂P
=0
∂b
Equivalently we have:
100rn + 100rn 100c
100c
0
!
D
b
!
=
250nr − 100n
200 · 250 − 10, 000
!
So, from
100cD = 200 · 250 − 10, 000
we get
D=
250 · 2 − 100
400
=
=2
c
200
From the second equation
200nrD + 100cb = 250nr − 100n
we get
n
[−150r − 100] = −75r − 50
100c
Note that the point (b, D) = (−75r − 50, 2) is a saddle point.
Definition of saddle point
b=
Next, we proceed with the Lagrange multipliers and we treat the constraints as
equalities instead of inequalities i.e.
b=0
, b = 10
and
D=0
, D = 2.5
We solve the following system
∂P
= λ1
∂D
,
∂P
= λ2
∂b
for D and b in terms of λ1 and λ2 , then substitute in the above constraint equations.
(FIGURE)
4
Why does this make sense? (review)
General example for Lagrange multipliers(with equality constraints)
minf (x̄)
s.t.
x̄ = (x1 , x2 , · · · , xn )
g1 (x̄) = c1
..
.
gm (x̄) = cm
For linear constraints
min f (x̄)
Ax = b
Equivalently :
a11 x1 + a12 x2 + · · · a1n xn = b1
..
.
am1 x1 + am2 x2 + · · · amn xn = bm
Then append the constraints (from optimization theory)


f x1


 f x2 
 .  = ∇f = λ1 ∇g1 + · · · + λm ∇gm
 . 
| {z }
|{z}
 . 
f xn
n-vector
n-vector
Note

a11
 .
.
∇ḡ = 
 .
a21
···
a1n a2n · · ·
am1
amn


 = AT

So we have n + m equations with x̄ = (x1 , x2 , · · · , xn ) and λ̄ = (λ1 , λ2 , · · · , λm )
unknowns.
5
Note: Assume ∇gk with k = 1, 2, · · · , m are vectors linearly independent.
Let’s consider a simple example with linear constraints:
min(x21 + x22 )
s.t.
2x1 + x2 = 3
f at x = x*
f at x = x*
g is
g=3
Clearly ∇f 6= 0 for feasible solutions. Suppose we find the minimum:
Note: g1 = 3 =⇒ 2x1 + x2 = 3 so ∇g1 = (2, 1)
To determine x1 , x2 we solve the following system:
2x1 = 2λ
2x2 = λ
=⇒ λ = x1 and λ = x2 . Substitute x1 and x2 in the constraint:
2λ +
λ
=3
2
6
6
3
and get λ = . Therefore x1 = and x2 = . For these values f (x) = x21 + x22 =
5
5
5
36 + 9
9
=
25
5
6
We can check this result by plugging the constraint into f (x) and minimize.
f (x) = x21 + (3 − 2x1 )2 = 5x21 − 12x1 + 9
Now minimize this function:
so from
f
= 10x1 − 12
x1
f
6
= 0 =⇒ x1 = which confirms our previous result.
x1
5
General example for Lagrange multipliers (with inequality constraints)
minf (x̄)
s.t.
x̄ = (x1 , x2 , · · · , xn )
g1 (x̄) ≤ c1
..
.
gm (x̄) ≤ cm
(FIGURE)
Reconsider now, the previous example with inequality constraints:
min(x21 + x22 )
s.t.
2x1 + x2 ≥ 3
x1 ≤ 2
x1 − x2 ≥ −3
(FIGURE)
7
Note that ∇f 6= 0 on the region. By looking at the graph we expect minimum on
6
2x1 + x2 = 3 =⇒ λ =
5
What if we guess wrong by looking for the minimum on x 1 = 2?
Then the equations are: ∇f = (2x1 , 2x2 ) and g1 = x1 = 2. So ∇g1 = (1, 0)
Equivalently
2x1 = λ
2x2 = 0
so λ = 4. But this says ∇f = λ∇g > 0 (an increasing function) =⇒ cannot be a
minimum.
Now return to the profit problem.
The goal is to maximize the profit function P (b, D) on 0 ≤ b ≤ 10 with respect to
D and b. So the problem is:
maxP (b, D)
s.t.
b ≤ 10
b≥0
Note ∇P 6= 0 on the interval [0, 10].
First, try g1 (b, D) = b = 10 constraint.
∇P = λ(0, 1).
So from here
∂P
= 0 = nr[250 − D · 100] − 100[n(1 + rD) + cb]
∂D
∂P
= λ = 200[250 − 100D] − 10, 000
∂b
Therefore D =
250 λ + 10, 000
λ + 10, 000
−
=⇒ 250 − 100D =
100
200 · 100
200
Plug this in the first equation and get
λ + 10, 000
nr
− 100[n(1 + rD) + cb] = 0
200
From these equations λ = 39, 000 > 0 and D = 0.05
8
Next, try g1 (b, D) = b = 0 constraint. From the above equations when b = 0 we get
D = 0.25 and λ > 0
∂P
Note, in this case λ =
> 0.
∂b
Since λ > 0, maximum must be at top constraint. If the maximum was at b = 0,
∂P
< 0 at b = 0, so λ < 0.
then
∂b
What conclusions can we draw?
D = 0.25 and b = 10 give the maximum.
Is D as significant as b?
Conclusions about dependence of P on D.
(FIGURE)
What about sensitivity?
As before:
r ∂P
P ∂r
r ∂D
S(D, r) =
D ∂r
Now we can also test sensitivity to constraint by solving three equations:
S(P, r) =
∂P
=0
∂D
∂P
=λ
∂b
b = bmax
9
We have
40, 000 − 20, 000D = λbmax
λbmax
2−D =
20, 000
λbmax
D =2−
20, 000
Now, we study the sensitivity of P to b max .
Recall the problem: P is maximized under the constraint 0 ≤ b ≤ 10
∇P = (
∂P ∂P
,
)
∂D ∂b
, ∇P = λ∇g
, g(b, D) = b = 10
Note also:
∂P
∂P ∂D
∂P ∂b
=
+
∂bmax
∂D ∂bmax
∂b ∂bmax
∂b
∂D
,
)
= ∇P · (
∂bmax ∂bmax
∂D
∂b
= λ∇g · (
,
)
∂bmax ∂bmax
∂D
∂b
= λ(0, 1) · (
,
)=λ
∂bmax ∂bmax
Therefore:
∂P
= λ = 39, 000
∂bmax
bmax
So S(P, bmax ) = λ
P
λ = ”Shadow price”, amount P increases with change in constraint.
In general, to maximize
f (x̄)
x̄ = (x1 , x2 , · · · , xn )
10
with constraints:
g1 (x1 , x2 , · · · , xn ) = c1
g2 (x1 , x2 , · · · , xn ) = c2
..
.
gk (x1 , x2 , · · · , xn ) = ck
∇f = λ∇g
, λ = (λ1 , λ2 , · · · , λk )
where
fxj = λ1 g1xj + λ2 g2xj + · · · + λk gkxj
, j = 1, 2, · · · , n
For c ”variable”(parameter) x̄ = x̄(c). Then
∂f ∂x1
∂f ∂x2
∂f ∂xn
∂f
=
+
+ ··· +
∂c1
∂x1 ∂c1
∂x2 ∂c1
∂xn ∂c1
= ∇f · (x̄0 (c1 ))
= λ∇ · · · x̄0 (c1 )
∂x1
∂c1
∂x2
+ · · · + λk gkx2 )
∂c1
= (λ1 g1x1 + λ2 g2x1 + · · · + λk gkx1 )
+ (λ1 g1x2 + λ2 g2x2
..
.
+ (λ1 g1xn + λ2 g2xn + · · · + λk gkxn )
= λ1
n
X
∂g1 ∂xj
j=1
= λ1
So
∂xj ∂c1
+ λ2
n
X
∂g2 ∂xj
j=1
∂xj ∂c1
∂g1
∂g2
∂gk
+ λ2
+ · · · + λk
∂c1
∂c1
∂c1
∂xn
∂c1
+ · · · λk
n
X
∂gk ∂xj
j=1
∂xj ∂c1
∂f
= λ1
∂c1
which means that the Lagrange multiplier is the rate of change of the objective
function with respect to constraints.
Problem
A local newspaper is looking for ways to increase profits. Currently, the paper is $
11
1.50/week with circulation of 80,000. Advertising is $ 250/page, and 350 pages/week
are sold.
It is estimated that an increase of 10 cents/week in the subscription price will cause
a drop in circulation of 5,000 subscribers.
Increasing ad prices by $100/page will cause the paper to lose approximately 50
pages of ads/week. The loss of 50 advertising pages will reduce circulation by 1,000
subscriptions.
Find the weekly subscription price that will maximize profit, keeping advertising
costs the same.
How do the assumptions affect your result?
What is the optimal strategy if both subscription price and advertising price are
changed?
Variables:
w = $1.50 : initial price/week
s0 = 80, 000 : # of subscribers (initially)
a = $250 : ad price/page (initially)
p = 350 : initial pages /week
c = $0.10 increase in subscription price/week (units of price increase)
b : increase in ad price/page/week (units of $100)
r = 50 : loss of ad pages/unit due to increase in ad price
x = 1, 000 : loss of subscribers due to r ad pages lost
k = 5, 000 : loss of subscribers per $ 0.10 price increase
The profit P ( = revenue )
Goal:
Maximize P the total profit
Step 1:
s = 80, 000 − ck − bx = # of subscribers
q = w + cl = profit/ subscriber
z = a + 100b = profit/ ad page
n = 350 − br = # of ad pages
P = profit = sq + zn
Look at 5-step method:
1. Ask/Identify variables, units sensible, assumptions identified.
2. Modelling approach
3. Formulate according to 2 - using 1
4. Solve 3
12
5. Answer - using 4 and 1 (criticize)
The profit function is given by :
P = (s0 − ck − bx)(w + c(0.10)) + (a + 100b)(350 − br)
where the variables are the ad price (b) and subscription price (c).
Case 1
First fix b = 0
dP
= −k(w + cl) + l(80, 000 − ck − bx) = 0
dc
−5, 000(1.50 + c(0.10)) + 0.10(80, 000 − 5, 000c) = 0
From here
1
2
Then q = w + cl = 1.55 and s = 80, 000 − 2, 500 = 77, 500.
c=
Sensitivity analysis: What if lost sales is not k = 5, 000? How sensitive is the result
to this assumption?
dP
= −k(w + cl) + l(80, 000 − ck − bx) = −k(1.50 + 0.10c) + 8, 000 − 0.10ck = 0
dc
Solve this equation for c, c∗ = c∗ (k). Optimal c∗ is a function of k which depends
on assumption about k. So from the above equation:
c∗ =
From here
5 · 8, 000
− 7.5
k
dc∗
40, 000
=−
dk
k2
The sensitivity is given by
S(c∗ , k) =
dc∗ k
40, 000 k
40, 000
· =−
· =−
= −16
2
dk c
k
c
2, 500
which represents 16% increase in c∗ for a drop of 1% in k.
13
f(c) k
f(c) k = (k c2 ) / 3
worse
f(c) k = k
15000
better
c
Nonlinear relationship
What if the assumption about k depends on a survey asking about a $. 30 increase?
Perhaps we expect better results for c < 0.30 and worse results for c > 0.30
Question
What is the sensitivity of the optimal price to assumption about k?
Linear case
P = (s0 − c∗ k − bx)(w + c∗ (0.10)) + (a + 100b)(350 − br)
The sensitivity is:
S(P ∗ , k) =
dP ∗ k
k
0.5 · (1.55) · 5, 000
· ∗ = −c∗ (w + c∗ (0.10)) · ∗ =
dk P
P
P∗
(FIGURE)
14
Quadratic case
P ∗ = (s0 −
c∗ 2
k − bx)(w + c∗ (0.10)) + (a + 100b)(350 − br)
3
The sensitivity is:
S(P ∗ , k) =
c∗2
k
1.42
5, 000
dP ∗ k
· ∗ =−
(w + c∗ (0.10)) · ∗ = −
(1.5 + 1.4) ·
dk P
3
P
3
P∗
(FIGURE)
In both cases the sensitivity gives changes of ≈ $10, 000/week for 1% change in k.
Then
c2
k − bx)(w + c(0.10)) + (a + 100b)(350 − br)
3
2ck
c2
2ck
dP
=−
(w + c(0.10)) + l[80, 000 − k − bx] = −c2 kl −
w + 8, 000 = 0
dc
3
3
3
Solve this equation for c and get:
P = (s0 −
−k +
∗
c =
p
k 2 + 3, 200k
2(0.10k)
or
∗
c =
−k −
p
k 2 + 3, 200k
2(0.10k)
Thus
r
3, 200
c = −5 + 5 1 +
k
∗
For k = 5, 000 we get c∗ = 1.4
Then
5
dc∗
= q
dk
2 1+
The sensitivity is
S(c∗ , k) =
3,200
k
−
or
∗
r
c = −5 − 5 1 +
3, 200
k
16, 000
3, 200
=− p 4
k2
2 k + 3, 200k 3
1
−16, 000
16, 000
dc∗ k
q
= p 2
=−
∗
dk c∗
c
2 k + 3, 200k
2 · 5, 000 1 +
15
3,200
5,000
1
≈ 0.9
1.4
(FIGURE)
Case 2- 2-D optimization
Optimize the profit P (c, b) with respect to subscription cost and ad cost.
Find the critical point:
∂P
= 0 = −k(w + cl) + l(80, 000 − ck − bx)
∂c
∂P
= 0 = −x(w + cl) + 100(350 − br) − r(250 + 100b)
∂b
This is a linear system of two equations and two unknowns.
Equivalently we have:
2lk −xl
xl 200r
!
c
b
!
=
80, 000
35, 000 − xw − 250r
!
By substituting the values of the parameters and solving this system we get: (c ∗0 , b∗0 ) =
(0.2903, 2.0971).
Note that
∂2P
= −2kl
∂c2
∂2P
= −200r
∂b2
∂2P
= −lx
∂c∂b
∂2P 2
∂2P ∂2P
·
−(
) > 0, so (c∗0 , b∗0 ) = (0.2903, 2.0971) is not a saddle
Therefore
2
2
∂c
∂b
∂c∂b
point.
Note that if there is a cap on the amount of increase in price, i.e. b = 1.5, then it
can not achieve this maximum.
(FIGURE)
16
Results for the problem for parameters as given, b ∗ = 1.5 and c∗ = 0.35
The goal is to maximize the profit function P (b, c) on b ≤ 1.5 with respect to c and
b. So the problem is:
maxP (b, c)
s.t.
b ≤ 1.5
There is no local optimum obtained for b < 1.5 , therefore it must be obtained on
the boundary b = 1.5. So the problem becomes:
maxP (b, c)
s.t.
b = 1.5
Note that ∇P = λ(0, 1).
So from here
∂P
= 0 = −k(w + cl) + l(80, 000 − ck − bx)
∂c
∂P
= λ = −x(w + cl) + 100(350 − br) − r(250 + 100b)
∂b
For b = 1.5 substitute the other known variables (w, x, r, k) and solve the system
for λ and c.
We can test the sensitivity to assumptions:
∂b r
= 0,
∂r b
∂b k
=0
S(b, k) =
∂k b
S(b, r) =
b is constant at r = 50
at
(FIGURES)
Next, we look at sensitivity of P to r.
17
k = 5, 000
(FIGURE)
Finally, we look at sensitivity of P to constraint b = c 1 .
S(P, c1 ) =
c1
∂P c1
=λ
∂c1 P
P
So we get an increase of 0.0393 % of P for 1% increase in c 1
If we include the constraint at b = 0 we repeat the analysis with b = c 2 = 0
constraint.
∂P
=0
∂c
,
From here λ2 < 0 which indicates that
maximum there
∂P
= λ2
∂b
∂P
< 0 on this boundary =⇒ not a
∂c2
Problem
Determine population levels that maximize population growth.
Variable population model: Logistic model:
x0 = r1 x(1 −
x
)
k1
Population decays (grows) for x > (<)k 1 .
We can solve for x(t).
x
dx
= r1 x(1 − )
dt
k1
=⇒
Z
dx
r1 x(1 −
x =
)
k1
Z
dt
1
x
[ln x + ln(1 − )] = t + C
r1
k1
x
x
ln x(1 − ) = r1 t + C =⇒ x(1 − ) = Cer1 t
k1
k1
18
Model for two competing species - The whale problem
x
) − α1 xy
k1
y
y 0 = r2 y(1 − ) − α2 xy
k2
x0 = r1 x(1 −
where
ri = the intrinsic growth (Lotka - Voltera)
ki = maximum sustainable population
αi = competition
Question: At what population levels is the total growth largest?
Therefore we have to determine population levels that are feasible (x, y ≥ 0) and
sustainable (x0 , y 0 ≥ 0) which maximize the total population growth. Consider
effect of α1 = α2 = α.
r
k
α
Blue
Fin
0.05
1.5 · 105
10−8
0.08
4 · 105
10−8
The problem becomes:
max(x0 + y 0 )
s.t. x ≥ 0
y≥0
x0 ≥ 0
y0 ≥ 0
(FIGURES)
19
We initially consider the optimization problem with equality constraints.
max(x0 + y 0 )
s.t. B1 : x = 0
B2 : y = 0
B3 : x 0 = 0
0
B4 : y = 0
(x 6= 0)
(y 6= 0)
(FIGURES)
For this, first of all we find the critical point for
f = x0 + y 0 = r1 x(1 −
x
y
) − α1 xy + r2 y(1 − ) − α2 xy
k1
k2
which satisfies the equality constraints:
∂f
=0
∂x
,
∂f
=0
∂y
Equivalently
r1
x
) − α1 y − α2 y − x = 0
k1
k1
r2
y
r2 (1 − ) − α1 x − α2 x − y = 0
k2
k2
r1 (1 −
Substituting the values for r1 , r2 , k1 , k2 , α1 = α2 = α and solving the system we
get x ≈ 69104 and y ≈ 1.96540 · 105 .
If we substitute these values in the equations of derivatives, we get: x 0 + y 0 =
9.5887 · 103 > 0
20
4 x 105
B1
B4
2
B3
0.5
1.5 x 105
B2
Next, we proceed with the Lagrange multipliers and we treat the constraints as
equalities instead of inequalities.
Note on:
B1 &B2 we have x0 + y 0 = 0
B1 &B4 we have x0 + y 0 = 0
B2 &B3 we have x0 + y 0 = 0
Thus we can eliminate the corners.
Note on :
B1 , x = c (= 0)
dP
dc =
∂f
∂f
=
= λ, if decreasing, then moving into domain =⇒ f decreasing =⇒
∂x
∂c
optimize on boundary
B2 , y = c (= 0)
21
dP
dc
> 0
=
∂f
∂f
=
= λ > 0 =⇒ optimize by moving into domain
∂y
∂c
B3 , x 0 = c
dP =
dc
>0
∂f
∂f
=
= λ > 0 . As c increases, the boundary moves to the left.
∂y
∂c
If λ > 0 =⇒ f increases, then moving into the domain.
If λ < 0 =⇒ f decreases, then moving out of the domain, but for this x < 0.
Similarly for B4 .
Next consider the constraint B1 : g1 (x, y) = x = 0. From here ∇g1 = (1, 0) so the
constraints with Lagrange multipliers become:
fx = λ
fy = 0
x=0
x
If we consider the constraint B3 : g3 (x, y) = x0 = r1 x(1− )−α1 xy = 0. From here
k1
x
r1
∇g3 = (r1 (1 − ) − α1 y − x, −α1 x) so the constraints with Lagrange multipliers
k1
k1
22
become:
fx = λ[r1 (1 −
x
r1
) − α1 y − x]
k1
k1
fy = λ(−α1 x)
x0 = 0
Problem 2.42
max(x + y)
s.t. x ≥ 0
y≥0
x0 ≥ 0
y0 ≥ 0
Note ∇(x + y) = (1, 1) =⇒ cannot have internal maxima.
Consider the problem with the equality constraints:
max(x + y)
s.t. x = 0
0
x =0
or y = 0 or
or y 0 = 0
Newton’s Method
Case A - (1-D)
Find the root of f (x) i.e find x∗ such that f (x∗ ) = 0.
Suppose you make an estimate xe of x∗ . Then
f (x∗ ) = f (xe ) + f 0 (xe )(x∗ − xe )
We want f (x∗ ) = 0, so f (xe ) = f 0 (xe )(xe − x∗ ) =⇒
x∗ ≈ x e −
f (xe )
.Let’s see how this works:
f 0 (xe )
23
f (xe )
≈ xe − x∗ . So, from here
f 0 (xe )
f(x)
f(xe)
xe
xe + c
x*
f(xe + c)
If we guess xe , then we can improve the guess by
f (xe )
= c > 0 =⇒ the new guess
f 0 (xe )
will be xe + c. Now replace xe with xe + c.
Update:
f (xe + c)
xe + c − 0
f (xe + c)
So guesses to left, correct to the right and guesses to right correct to the left.
If we want to find minimum(maximum) of a function F (x) we have to find the
critical points first. So we have to find the roots of F 0 (x) = f (x) = 0. Start with
an initial guess x0 and improve it:
xn+1 = xn −
F 0 (xn )
F 00 (xn )
where xn+1 is the updated (improved) guess.
Case B - (2-D)
Now we want to find the maximum (minimum) of function F (x, y). First of all we
have to find the critical points by solving the following system:
f (x∗ , y ∗ ) = Fx = 0
g(x∗ , y ∗ ) = Fy = 0
24
Start with an initial guess (x0 , y0 ). Then
0 = f (x∗ , y ∗ ) = f (x0 , y0 ) + fx (x0 , y0 )(x∗ − x0 ) + fy (x0 , y0 )(y ∗ − y0 )
0 = g(x∗ , y ∗ ) = g(x0 , y0 ) + gx (x0 , y0 )(x∗ − x0 ) + gy (x0 , y0 )(y ∗ − y0 )
Or in the matrix form:
fx fy
gx gy
!
x∗ − x 0
y ∗ − y0
!
=
−f (x0 , y0 )
−g(x0 , y0 )
!
Therefore
f gy − gfy
fx gy − g x fy
fx g − g x f
y ∗ − y0 = −
fx gy − g x fy
x∗ − x 0 = −
In general
xn+1 = xn −
Fx Fyy − Fy Fxy
Fxx Fyy − Fxy Fyx
, yn+1 = yn −
Fy Fxx − Fx Fxy
Fxx Fyy − Fxy Fyx
Linear Programs
We want to solve the following problem:
maxf (x̄) = c1 x1 + c2 x2 + · · · + cn xn where
s.t.
c1 , c2 , · · · , cn - constants
g1 (x̄) ≤ b1
..
.
gm (x̄) ≤ bm
We’ve already seen a linear program, even though it didn’t appear to be. Let’s
reconsider the whale model
max(x + y)
s.t. x ≥ 0
y≥0
x0 ≥ 0
y0 ≥ 0
25
where
x
) − α1 xy ≥ 0
k1
y
y 0 = r2 y(1 − ) − α2 xy ≥ 0
k2
x0 = r1 x(1 −
Note that these constraints are linear but in general they would be nonlinear (quadratic).
We saw that the maximum was obtained at a corner which is geometrically ”obvious”.
The idea behind solving a linear program is that if moving along a boundary is
optimal, then continue until corner is reached.
Many software packages exist for solving linear programs, based on the ”simplex”
method.
We consider a simple example to see how the simplex method works.
max(x1 + 2x2 )
s.t. − 2x1 + x2 ≤ 2
− x1 + 2x2 ≤ 7
x1 ≤ 3
8
C
5 D
3.5
-2 x1 + x2 = 2
2
B
x1 = 3
x1 = 0
A
x2 = 0
E
It can be proved that the optimal point is an extreme point(corner).
Idea: If moving along the constraint improves solution, continue until you hit the
next constraint.
26
Standard form: Introduce ”slack” variables - slack variables pick up the slack in the
inequality and convert to equality constraints.
Standard form:
minfˆ
s.t. Ax̂ = b
x̂ ≥ 0
We have our problem in terms of maximum, so we can take −f and look for
min(−f ).
min(−x1 − 2x2 )
s.t. − 2x1 + x2 + s3 = 2
− x1 + 2x2 + s4 = 7
x1 + s 5 = 3
x1 , x 2 ≥ 0
s3 , s 4 , s 5 ≥ 0
sj = slack variables Replace s3 , s4 , s5 to x3 , x4 , x5 .
We know that we will find the maximum at an extreme point. So we choose one
feasible solution (the one that satisfies the constraints ) which is an extreme point,
and see if can do better.
So we start with something simple: if we take x 3 , x4 , x5 , the slack variables, to equal
the constraints, the x1 , x2 = 0.
Note: we need (at least) 3 of x0i s 6= 0(basic variables) to satisfy the constraints (3
equations and 3 unknowns)=⇒ x3 = 2, x4 = 7, x5 = 3 .
This means that we start at A, where x 1 , x2 = 0 (non-basic variables). Now we
look to see if we can improve the objective function f by changing the non-basic
variables x1 or x2 .
YES! In fact if we increase either one, f is improved. Which variable should we
choose ? We have to choose the one which improves f more. Note that the objective
function is f = −x1 − 2x2 , so it will be improved more if we choose to change x 2 .
Do not choose both variables, because we know that we should stay on constraint.
By making x2 6= 0, we can now satisfy the constraints with either x 2 , x4 , x5 ,
x2 , x3 , x5 , or x2 , x3 , x4 , etc.
Moving on constraint x1 = 0 (increasing x2 ), which constraint do we hit first by
27
decreasing x3 or x4 ?
(−2 · 0) + x2 + x3 = 2
(−0) + 2x2 + x4 = 7
x1 + s5 = 3 (it doesn’t change)
Therefore x2 = 2, x3 = 0, and x4 ≥ 0. So, move along constraint until you hit
x2 = 2, then x3 = 0. Note that this is the point B.
We have changed our ”basic” variables; before we had x 3 , x4 , x5 6= 0 to make up
nontrivial variables satisfying the constraints.
Now we have new ”basic” variables:
x2 , x4 , x5 6= 0
x1 , x 3 = 0
(−2 · 0) + x2 + 0 = 2
=⇒
x2 = 2
(−0) + 2x2 + x4 = 7
=⇒
x4 = 3
x1 + s 5 = 3
=⇒
x5 = 3
This step is called a pivot .
We repeat the procedure until we can no longer improve f .
1. Check to see possible improvements to f by changing non-basic variables
2. Choose direction to search
3. Pivot (switch basic, non-basic variables) and repeat.
First Step (x1 = 0, x2 = 0)
Basic variables
x3
x4
x5
Non-basic variables
x1
x2
The problem is
min(−x1 − 2x2 )
s.t. − 2x1 + x2 + x3 = 2
− x1 + 2x2 + x4 = 7
x1 + x 5 = 3
28
We changed x2 because the coefficient of x2 in f is greater. So the new basic
variables are x2 , x4 , x5 from increasing x2 until x3 = 0 and x4 ≥ 0
Second Step(x1 = 0, x2 = 2)
Basic variables
x2
x4
x5
Non-basic variables
x1
x3
We change the objective function in terms of non-basic variables
min(−x1 − 4x1 + 2x3 − 4) = min(−5x1 + 2x3 − 4)
s.t. − 2x1 + x2 + x3 = 2
− x1 + 2x2 + x4 = 7
x1 + x 5 = 3
Note that the coefficient of x1 in the objective function is greater, so we change
x1 =⇒ it affects x2 , x4 , x5 .
At x1 = 3, x5 = 0 which is off constraint.
−x2
, so from here
Moving along the first constraint (x 3 = 0) we have x1 = 1 +
2
−x1 + 4x1 + 4 + x4 = 7. Therefore x1 = 1, x4 = 0, x2 = 4.
Third Step(x1 = 1, x2 = 4)
Basic variables
x1
x2
x5
Non-basic variables
x4
x3
Can we improve f by increasing one of the non-basic variables? Write f in terms
of non-basic variables x3 and x4 .
From the first constraint we have x2 = 2 + 2x1 − x3 and by substituting this into
the third constraint we get −x1 + 2(2 + 2x1 − x3 ) + x1 = 7.
So the objective function in terms of non-basic variables will be:
2
1
2
1
5
4
f = −[1 + x3 − x4 ] − 2[2 + 2(1 + x3 − x4 ) − x3 ] = −9 + x4 − x3
3
3
3
3
3
3
29
Thus the problem will be:
4
5
min(−9 + x4 − x3 )
3
3
s.t. − 2x1 + x2 + x3 = 2
− x1 + 2x2 + x4 = 7
x1 + x 5 = 3
Note that by increasing x3 we improve the objective function =⇒ it affects x 1 , x2 , x5 .
7 − x1
We follow the constraint −x1 + 2x2 = 7 since x4 =0. From here x2 =
.
2
7 − x1
By substituting this into the second constraint we get: −2x 1 +
+ x3 = 2.
2
5
7 − (3 − x5 )
+ x3 = 2. So from here x5 + x3 = 3. If
Equivalently: −2(3 − x5 ) +
2
2
x3 = 3 =⇒ x2 = 5. Therefore x1 = 3 and x2 = 5
Fourth Step(x1 = 3, x2 = 5)
Basic variables
x1
x2
x3
Non-basic variables
x4
x5
From the constraints we have: x1 = 3 − x5 and 2x2 = 7 − x4 + 3 − x5 . So the
objective function in terms of non-basic variables will be:
f = −(3 − x5 ) − (7 − x4 + 3 − x5 ) = 2x5 + x4 − 13
At this point the objective function cannot be improved anymore by increasing the
non-basic variables.
Therefore x4 = 0, x5 = 0 =⇒ −x1 + 2x2 = 7 and x1 = 3. The ”binding” conditions
give the extreme point.
Example in 4-D
f (x1 , x2 , x3 , x4 ) = ax1 + bx2 + cx3 + dx4 . f (x) is a linear function in all variables.
xj = # of certain product which is limited by resources, materials, work.
30
The problem will be:
max(ax1 + bx2 + cx3 + dx4 )
s.t.
X
X
cj xj ≤
dj xj ≤
xj ≥ 0
So , there are 6 constraints and 4 variables.
Where do we expect to find maximum?
On edge! - WHY?
Let’s pick up an interior point. The linear function f (x) can always be increased
by increasing xj (or if a, b, c, d < 0 by decreasing xj ) =⇒ eventually hit the edge.
Further,we can always increase the function f by increasing along the edge: follow
constraint until the direction changes =⇒ eventually hit the corner =⇒ optimize
over corners.
Dual Problem
Primal (Original) Problem
maxf (x) = c1 x1 + c2 x2 + · · · + cn xn

a11
a12
···
 .
s.t. 
 ..
am1 am2 · · ·




b
x
a1n  1   1 
x
  2   b2 
 .  ≤  . 
 .   . 
 .   . 
amn
bm
xn

Dual Problem
ˆ = b 1 y1 + b 2 y2 + · · · + b m ym
maxf(y)

a11
 .
s.t. 
 ..
a21
···
am1
a1n a2n · · ·
amn




c1
y1
   
  y2   c2 
 .  ≥  . 
 .   . 
 .  .

ym
cn
Note: The dual variables are the same as the shadow prices from Lagrange multipliers.
31
∇f = λ∇g. Then the equations are:
c1 = a11 λ1 + a21 λ2 + · · · am1 λm
..
.
cn = a1n λ1 + a2n λ2 + · · · amn λm
Then max fˆ = min f (the objective functions are equal). If we increase b i by 1
∂ fˆ
= yi .
unit, fˆ increases by yi units i.e.
∂bi
Let’s consider again the whale problem.
Primal Problem
min(−x1 − x2 )


!
r1 αk1
.
 x1

s.t. 
≤
 ..
 x
2
r2 αk2
r1 k1
r2 k2
!
− x1 ≤ 0
− x2 ≤ 0
Dual Problem
min(r1 k1 y1 + r2 k2 y2 )
s.t. r1 y1 + r2 y2 ≤ −1
αk1 y1 + αk2 y2 ≤ −1
y1 ≤ 0
y2 ≤ 0
Next, let’s reconsider the whale problem as linear programming problem.
max(x + y)
s.t. x ≥ 0
y≥0
x0 ≥ 0
y0 ≥ 0
32
where
x
) − α1 xy ≥ 0
k1
y
y 0 = r2 y(1 − ) − α2 xy ≥ 0
k2
x0 = r1 x(1 −
We can solve this problem in MATLAB by using the following function:
[X, F V AL, EXIT F LAG, OU T P U T, LAM BDA] = linprog(f, A, b)
which solves the general problem
minf
s.t. Ax ≥ b
Now we put the whale problem into ”matlab” form:
min(−x0 − y 0 )
x
s.t. − r1 x(1 − ) + α1 xy ≤ 0
k1
y
− r2 y(1 − ) + α2 xy ≤ 0
k2
−x≤0
−y ≤0
So
f = −1 −1

r1
 .

A =  ..
αk1
αk2
r2





Example
Bookshelf
With doors
With drawers
Custom
Wood
Labor
Revenue
10
12
25
20
2
4
8
12
100
150
200
400
33

r1 k1
r k 


b =  2 2
 0 
0
There are 500 units of wood available, and 150 units of labor available. Which types
of book shelves should be produced to maximize revenue?
f (x) = 100x1 + 150x2 + 200x3 + 400x4 , where xi are the number of units of each
type of bookshelves. The problem becomes:
max(100x1 + 150x2 + 200x3 + 400x4 )
s.t. 10x1 + 12x2 + 25x3 + 20x4 ≤ 500
2x1 + 4x2 + 8x3 + 12x4 ≤ 150
x1 , x 2 , x 3 , x 4 ≥ 0
For the given parameters, the ”matlab” form of this problem is:
min(−100x1 − 150x2 − 200x3 − 400x4 )
s.t. 10x1 + 12x2 + 25x3 + 20x4 ≤ 500
2x1 + 4x2 + 8x3 + 12x4 ≤ 150
− x1 , −x2 , −x3 , −x4 ≤ 0
Using linear programming in MATLAB we get x 1 =37.5 and x4 =6.25. So the revenue
is 6250.
Note: linear programming gives approximation.
In reality we can only make integer multiples of products =⇒ x 1 =38 and x4 =6 and
therefore the revenue is 6200. In this case there were 500 units of wood used and
only 148 units of labor (not all consumed).
Case 1
If increase the constraint of wood by 1, i.e. 10x 1 + 12x2 + 25x3 + 20x4 ≤ 501, we
get $5 increase in the revenue =⇒ the revenue = 6255.
Case 2
If increase the constraint of wood by 2, i.e. 10x 1 + 12x2 + 25x3 + 20x4 ≤ 502 we get
x1 =37.8 and x4 =6.2.
For x1 ≈ 38 and x4 ≈ 6 =⇒ the revenue = 6200.
For x1 ≈ 37 and x4 ≈ 6 =⇒ the revenue = 6250 which is the maximum.
Case 3
If increase the constraint of wood by 8, i.e. 10x 1 + 12x2 + 25x3 + 20x4 ≤ 508 the
maximum is given by x1 =39 and x4 =6.
34
Integer programming
For the previous problem suppose that x i can take only integer values, i.e.
x1 = 1, 2, · · · , 50
x2 = 1, 2, · · · , 34
x3 = 1, 2, · · · , 18
x4 = 1, 2, · · · , 8
The actual optimal solution changes discontinuity with constraints. If we have n
bookshelves, then we have n! combinations.
This is an example of knapsack problem: compare all possible combinations of items
to be put in knapsack with different values, weights, sizes,, weight restriction.
Possible solution methods:
• Combinatorial methods (polyhedral considerations)
• Dynamic programming
Take one item, then proceed according to some algorithm until restriction is hit compare all possible end results, take maximum, and go back recursively.
• Tree - exponential time algorithm.
Ways to reduce time
• Use linear programming as approximation to reduce state space.
For the wood problem the linear programming gave x 1 = 37.5 so we could choose
x1 = 37 or 38. Similarly for x4 =6.25, we could choose either x4 = 6 or 7.
•Greedy algorithm.
For the wood problem - make as many of most expensive items as possible if resources
are available.
Make 12 custom units and 1 unit with drawers =⇒ the revenue = 4,800+200=5,000
and the left over wood is 500-240=260.
Improve by replacing some custom units with items not requiring much labor, but
using wood.
Shortest path problem - Travelling salesperson problem
The sales person must visit n cities. It cost him (time) to go between different cities.
What is the minimum cost to visit all 5 cities once?
35
(FIGURE)
We can do the calculation by hand.
C(1,4,1)=min 3 + C(2, 3, 1) = 3 + 10 = 13
6 + C(3, 3, 1) = 6 + 7 = 13
5 + C(4, 3, 1) = 5 + 8 = 13
C(2,3,1)=min 2 + C(3, 2, 1) = 2 + 8 = 10
1 + C(4, 2, 1) = 1 + 9 = 10
C(3,3,1)=min 2 + 6 = 8
3+4=7
C(4,3,1)=min 3 + 5 = 8
1+8=9
1 −→ 2 −→ 3 −→ 4 −→ 1
1 −→ 2 −→ 4 −→ 3 −→ 1
What about ”greedy algorithm”?
Do the best you can do on the first step, then do the best you can do on the next
step.
1 −→ 2 −→ 3 −→ 4 −→ 1 does give the optimal.
36