10.34, Numerical Methods Applied to Chemical Engineering
Professor William H. Green
Lecture #17: Constrained Optimization.
Notation
“second derivative of f(x).”: We normally mean
fxx
⎛ ∂2 f
⎜
∂x 2
H = ⎜⎜ 2 1
∂ f
⎜
⎝ ∂x 2 ∂x1
Hessian Matrix
∂2 f
∂x1∂x 2
∂2 f
∂x 22
⎞
⎟
⎟
⎟
⎟
⎠
V2f in BEERS
but second derivative can also mean:
∂2 f ∂2 f
+
Tr{H} =
- scalar
∂x12 ∂x 22
Laplacian
fxx
V2f in Physics Texts
V·(Vf)
Constrained Optimization
Equality Constraints: minx f(x) such that g(x) = 0
May be able to invert this statement as: xN = G(x1, x2, …, xN-1)
Then we can state min as: min f(x1, x2, …, xN-1,G(x1, x2, …, xN-1))
Notice the xN is gone. Constrained becomes unconstrained. Solve with previous methods.
Other way to do this:
Lagrange Multipliers
∂f
∂x n
Unconstrained
-
= 0 at the minimum
x mn
constrained problems do not work that way!
o
Constrained:
BOUNDARIES GET IN THE WAY
∂f
∂x n
=λ
x const . min
∂g
∂x n
Vf|const. min = λVg|x const. min
x const . min
Gradient of f equals 0 in directions parallel to constraint but not perpendicular
Create a new function L(x, λ) = f(x) – λg(x) (λ is unknown before you do the problem)
VxL = 0
∂L
/∂λ = g(x) Æ 0
at constrained min
at constrained min
Second derivatives not necessarily all positive
Cite as: William Green, Jr., course materials for 10.34 Numerical Methods Applied to Chemical
Engineering, Fall 2006. MIT OpenCourseWare (http://ocw.mit.edu), Massachusetts Institute of
Technology. Downloaded on [DD Month YYYY].
Augmented Lagrangian
LA = f(x) – λg(x) +
1
2μ
(0)
( g ( x)) 2
minx LA given initial guess λ[0], μ[0] Æ xmin[0]
VxLA(xmin[0],
[0]
λ ) = Vf|
[0]
xmin
[0]
-λ
Vg(xmin[0])
-
1
μ (0)
[0]
g(x)Vg(x) Î Vf – (λ
-
[0]
g ( x min
)
μ (0)
)Vg(x)
λ[1]
As μ[0] shrinks,
1
gets large, magnifying (g(x))2 term, and thus holding the constraint
2 μ (0)
more strictly. minx LA using λ[1] get a new xmin. In quantum mechanics, λ corresponds to
orbital energies. Most of the time, λ does not have a physical meaning. μ[0] is a
mathematical trick.
More Than One Constraint
Suppose you have >1 constraints:
L = f(x) -
∑
g1(x) = 0
make sure these are compatible
g2(x) = 0
i.e. there is a “feasible space” – set
g3(x) = 0
of x that satisfies all constraints
λigi(x)
i
VL = 0
Vf =
∑
λiVgi
i
Inequality Constraints
very common
min f(x), s.t. g(x) = 0, h(x) ≥ 0
Active inequality constraints: h(xmin) = 0
Inactive inequality constraints: h(xmin) > 0
Usually, we do not know whether h’s are active or inactive before doing a problem, but must
leave in during optimization process, to allow finding of solution:
Vf = KjVhj, K ≥ 0 when hj is active; also hj = 0 and kj ≥ 0.
Kjhj(x
const. min)
= 0 when hj is inactive
10.34, Numerical Methods Applied to Chemical Engineering
Prof. William Green
Lecture 17
Page 2 of 3
Cite as: William Green, Jr., course materials for 10.34 Numerical Methods Applied to Chemical
Engineering, Fall 2006. MIT OpenCourseWare (http://ocw.mit.edu), Massachusetts Institute of
Technology. Downloaded on [DD Month YYYY].
if inactive, hj ≠ 0 and kj = 0. Vhj can be anything; it does not affect the problem
Karash-Kahn-Tucker (KKT) conditions:
L = f(x) - Σλigi(x) – Σkjhj(x)
VL(xmin) = 0
h(xmin) ≥ 0
g(xmin) = 0
Kj ≥ 0
Kjhj = 0
To handle active-inactive constraints, add slack variables:
hj(x) ≥ 0 Æ hj(x) – Sj = 0, Sj ≥ 0
Augmented Method LA:
Optimal Sj = max{hj(x) – μ[k]Kj[k]; 0}
LA = f(x) – Σλigi - Σkjhj – Σμ[k](kj[k])2 + (1/2μ[0])(gi2+hj2+(μ[0]kj)2)
F(x) = VLA = 0
Use Newton’s Method with Broyden to approximate the Hessian matrix.
Trying to solve: JLA *Δx = -VLA Use Newton’s method to find x
Jacobian is messy:
⎛⎛ 2 ⎞
⎜⎜ ∂ L ⎟
⎜ ⎜ ∂x ∂x ⎟
⎜ ⎝ i j ⎠ old
⎜ ⎛
⎞
⎜ ⎜ ∂C n ⎟
⎜ ⎜⎝ ∂x j ⎟⎠
old
⎝
⎛ ∂C m
⎜−
⎜ ∂x
j
⎝
0
⎞
⎟
⎟
⎠
T
⎞
⎟
⎟
⎛ Δx ⎞ ⎛ − ∇ x f x old ⎞
⎟
old ⎟⎜
⎟=⎜
⎟⎜⎝ Δλ ⎟⎠ ⎜ S − C ( x old ) ⎟
⎝
⎠
⎟
⎟
⎠
⎛ ∂2L
⎜
If we want to: minpf(p) = ( /2)p ⎜ ∂x ∂x
⎝ i j
1
such that
∂c m
∑ ∂x
T
⎞
⎟
⎟ p + Vf|x ·p
⎠ x old
old
p j + c m ( x old ) = 0 ∀ m = 1,..., N constraints
j x old
- can easily get p (same as Δx above)
“quadratic program”
Sequential Quadratic Programming (SQP)
In MATLAB: fmincon
10.34, Numerical Methods Applied to Chemical Engineering
Prof. William Green
Lecture 17
Page 3 of 3
Cite as: William Green, Jr., course materials for 10.34 Numerical Methods Applied to Chemical
Engineering, Fall 2006. MIT OpenCourseWare (http://ocw.mit.edu), Massachusetts Institute of
Technology. Downloaded on [DD Month YYYY].