Constrained Optimization
●
●
Unconstrained Optimization (Review)
Constrained Optimization
Approach
— Equality constraints
—
—
*
Lagrangeans
*
Shadow prices
Inequality constraints
*
*
Kuhn-Tucker conditions
Complementary slackness
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 1 of 23
Unconstrained Optimization (1)
●
Definitions:
—
Optimization = Maximum of desired quantity
= Minimum of undesired quantity
—
—
Objective Function = Expression to be
optimized
= Z(X)
Decision Variables = Variables about which
we can make decisions
= X = (X1….Xn)
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 2 of 23
Page 1
Unconstrained Optimization (2)
D
B
F(X)
A
C
E
X
●
By calculus:
If F(X) continuous, analytic:
Condition for maxima and minima
∂F(X) / ∂Xi = 0
∀i
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 3 of 23
Unconstrained Optimization (3)
●
Secondary conditions:
∂2F(X) / ∂Xi2 < 0
∂2F(X) / ∂Xi2 > 0
= > Max
= > Min
(B,D)
(A,C,E)
These define whether point of no change
in Z is a maximum or a minimum
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 4 of 23
Page 2
Unconstrained Optimization (4)
●
Example: Housing insulation
F(x) = K1 / x + K2x
Total Cost = Fuel cost + Insulation cost
x = Thickness of insulation
∂F(x) / ∂x = 0 = -K1 / x2 + K2
=> x* = {K1 / K2} 1/2
(starred quantities are optimal)
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 5 of 23
Unconstrained Optimization (5)
●
K1 = 500
K2 = 24
X* = 4.56
Optimizing Cost Example
600
Cost
500
400
Fuel
Insulation
Total
300
200
100
0
1
3
5
7
9
11
Inches of Insulation
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 6 of 23
Page 3
Constained Optimization
●
●
“Constrained Optimization” involves the
optimization of a process subject to
constraints
Constraints have two basic types
Equality Constraints -- some factors have to
equal constraints
— Inequality Constraints -- some factors have to
be less less than or greater than the
constraints (these are “upper” and “lower”
bounds
—
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 7 of 23
Equality Constraints
●
●
●
Example: Best use of budget
Maximize: Output = Z(X) = aox1a1x2a2
Subject to (s.t.):
Total costs = Budget = p1x1 + p2x2
Z(x)
Z*
X
Budget
Note: ∂Z(X) / ∂X ≠ 0 at optimum
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 8 of 23
Page 4
Constrained Optimization
●
●
Approach
To solve situations of increasing
complexity, (for example, those with
equality, inequality constraints) ...
Transform more difficult situation into
one we know how to deal with
In this case, transform optimization of a
“constrained” situation to optimization of
“unconstrained” situation
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 9 of 23
Lagrangean Method (1)
●
●
Transforms equality constraints into
unconstrained problem
Start with:
Opt: F(x)
s.t.: gj(x) = bj => gj(x) - bj = 0
●
Get to:
L = F(x) - Σj λj[gj(x) - bj]
λj = Lagrangean multipliers (lambdas) -- these
are unknown quantities for which we must solve
Note: [gj(x) - bj] = 0 by definition, thus
optimum for F(x) = optimum for L
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 10 of 23
Page 5
Lagrangean Method (2)
●
●
To optimize L:
∂L / ∂xi = 0
∀i
∂L / ∂λj = 0
∀i
Example:
Opt: F(x) = 6x1x2
s.t.: g(x) = 3x1 + 4x2 = 18
L = 6x1x2 - λ(3x1 + 4x2 - 18)
∂L / ∂x1 = 6x2 - 3λ
λ=0
λ=0
∂L / ∂x2 = 6x1 - 4λ
∂L / ∂λi = 3x1 + 4x2 -18 = 0
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 11 of 23
Lagrangean Method (3)
●
●
●
●
Solving as unconstrained problem:
∂L / ∂x1 = 6x2 - 3λ
λ=0
∂L / ∂x2 = 6x1 - 4λ
λ=0
∂L / ∂λi = 3x1 + 4x2 -18 = 0
so that: λ = 2x2 = 1.5x1
x2 = 0.75x1
3x1 + 3x1 - 18 = 0
x1* = 18/3 = 6 x2* = 18/8 = 2.25
λ* = 4.5
F(x)* = 40.5
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 12 of 23
Page 6
Shadow Prices
●
●
Shadow Price is the Rate of change of
objective function per unit change of
constraint
= ∂F(x) / ∂bj
This is meaning of Lagrangean multiplier
SPj = ∂F(x)*/ ∂bj = λj
—
●
●
Naturally, this is an instantaneous rate
The shadow price is extremely important
for system design
It defines value of changing constraints
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 13 of 23
Shadow Prices (2)
●
●
●
Let’s see how this works in example, by
changing constraint by 0.1 units:
Opt: F(x) = 6x1x2
s.t.: g(x) = 3x1 + 4x2 = 18.1
The optimum values of the variables are
x1* = (18.1)/6
x2* = (18.1)/8
Thus F(x)* = 6(18.1/6)(18.1/8) = 40.95
∆F(x) = 40.95 - 40.5 = 0.45 = λ* (0.1)
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 14 of 23
Page 7
Inequality Constraints
●
Example: Housing insulation
Min: Costs = K1 / x + K2x
s.t.: x ≥ 8 (minimum thickness)
O ptimizing Cost Example
600
Cost
500
400
Fuel
Insulation
Total
300
200
100
0
1
3
5
7
9
11
Inches of Insulation
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 15 of 23
Inequality Constraints (2)
●
●
●
Approach: Transform inequalities into
equalities, then proceed as before
Again, introduce new variable -- the
“Slack” variable that defines “slack” or
distance between constraint and amount
used
The resulting equations are known as the
“Kuhn-Tucker conditions”
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 16 of 23
Page 8
Inequality Constraints -- insertion of
slack variables in Lagrangean
●
●
●
●
A “slack variable”, sj , for each inequality
gj(x) ≤ bj => gj(x) + sj2 = bj
gj(x) ≥ bj => gj(x) - sj2 = bj
These are “squared” to be positive
start from:
opt: F(x)
s.t.: gj(x) ≤ bj
get to:
L = F(x) - Σjλj[gj(x) + sj2 - bj]
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 17 of 23
Inequality Constraints -Complementary Slackness Conditions
●
●
The optimality conditions are:
∂L / ∂xi = 0
∂L / ∂λj = 0
plus: ∂L / ∂sj = 2sjλj = 0
These new equations imply:
sj = 0
λj ≠ 0
or
sj ≠ 0
λj = 0
They are the “complementary slackness”
conditions. Either slack or lambda =0 ∀i
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 18 of 23
Page 9
Interpretation of Complementary
Slackness Conditions
●
●
●
Interpretation:
If there is slack on bj,
(i.e. more than enough of it)
=> No value to objective function
to having more: λj = ∂F(x) / ∂bj = 0
If λj ≠ 0, then all available bj used
=> sj = 0
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 19 of 23
Application to Example
●
●
●
●
●
●
●
Min: Costs = K1 / x + K2x
s.t.: x ≥ 8 (minimum thickness)
L = K1/x + K2x - λ[x - s2 - b]
L = 500/x + 24x - λ[x - s2 - 8]
500/x2 + 24 - λ = 0 2 λs
λ = 0 x - s2 = 8
If s = 0, x = 8, λ = 31.8 (at that point)
Max = 254.5
Therefore, worth relaxing (in this case,
lowering) constraint to get maximum
x* = 4.56 Optimum = 221
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 20 of 23
Page 10
Unconstrained Optimization (5)
●
K1 = 500
K2 = 24
X* = 4.56
Optimizing Cost Example
600
Cost
500
400
Fuel
Insulation
Total
300
200
100
0
1
3
5
7
9
11
Inches of Insulation
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 21 of 23
Another application to Example
●
●
●
●
●
●
Min: Costs = K1 / x + K2x
s.t.: x ≥ 4 (NEW MINIMUM)
L = K1/x + K2x - λ[x + s2 - b]
L = 500/x + 24x - λ[x + s2 - 4]
500/x2 + 24 - λ = 0 2 λs
λ = 0 x - s2 = 4
If λ = 0, x = 4.56, slack, s2 = 1.56
Optimum = 221
not worth changing constraint
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 22 of 23
Page 11
Summary of Presentation
●
Important mathematical approaches
Lagreangeans
— Kuhn- Tucker Conditions
—
●
●
Important Concept: Shadow Prices
THESE ANALYSES GUIDE DESIGNERS
TO CHALLENGE CONSTRAINTS
Engineering Systems Analysis for Design
Massachusetts Institute of Technology
Richard de Neufville, Joel Clark, and Frank R. Field
Constrained Optimization
Slide 23 of 23
Page 12