Introduction to Optimization
(ii) Constrained and Unconstrained
Optimization
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Objectives

To study the optimization of functions of multiple variables without
optimization.

To study the above with the aid of the gradient vector and the Hessian matrix.

To discuss the implementation of the technique through examples

To study the optimization of functions of multiple variables subjected to
equality constraints using

2
the method of constrained variation
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Unconstrained optimization
 If a convex function is to be minimized, the stationary point is
the global minimum and analysis is relatively straightforward
 A similar situation exists for maximizing a concave variable
function.
 Necessary and sufficient conditions for the optimization of
unconstrained function of several variables
3
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Necessary condition
 In case of multivariable functions a necessary condition for a stationary
point of the function f(X) is that each partial derivative is equal to zero.
 Each element of the gradient vector defined below must be equal to
zero. i.e. the gradient vector of f(X),
f
at X=X*,x defined
as follows,
must be equal to zero:
 f
* 
 x (  ) 
 1

 f
* 
(

)
 x
2
0
x f  






 f

( * ) 

 dxn

4
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Sufficient condition

For a stationary point X* to be an extreme point, the matrix of second
partial derivatives (Hessian matrix) of f(X) evaluated at X* must be:


positive definite when X* is a point of relative minimum, and

negative definite when X* is a relative maximum point.
When all eigen values are negative for all possible values of X, then X* is
a global maximum, and when all eigen values are positive for all possible
values of X, then X* is a global minimum.

If some of the eigen values of the Hessian at X* are positive and some
negative, or if some are zero, the stationary point, X*, is neither a local
maximum nor a local minimum.
5
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Example
Analyze the function
f ( x)   x12  x22  x32  2 x1 x2  2 x1 x3  4 x1  5 x3  2
and classify the stationary points as maxima, minima and points of inflection
Solution
6
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Example …
Solving these simultaneous equations we get
X*=[1/2, 1/2, -2]
2 f
2 f
2 f
 2; 2  2; 2  2
x12
x2
x3
2 f
2 f

2
x1x2 x2x1
2 f
2 f

0
x2 x3 x3x2
2 f
2 f

2
x3x1 x1x3
7
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Example …
Hessian of f(X) is
 2 f 
H


x

x
 i j 
 2 2 2 
H   2 2 0 
 2 0 2 
2
 I - H  2
2
8
Water Resources Systems Planning and Management: M2L2
2
2
2
0
0
 2
0
D Nagesh Kumar, IISc
Example …
or (  2)(  2)(  2)  2(  2)(2)  2(2)(  2)  0
(  2)[ 2  4  4  4  4]  0
(  2)3  0
or
1  2  3  2
Since all eigen values are negative the function attains a maximum at the point
X*=[1/2, 1/2, -2]
9
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Constrained optimization
A function of multiple variables, f(x), is to be optimized subject to one or
more equality constraints of many variables. These equality constraints,
gj(x), may or may not be linear. The problem statement is as follows:
Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m
where
10
 x1 
x 
 
X   2
 
 xn 
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Constrained optimization…
 With the condition that
m  n ; or else if m > n then the problem
becomes an over defined one and there will be no solution. Of the
many available methods, the method of constrained variation is
discussed
 Another method using Lagrange multipliers will be discussed in the
next lecture.
11
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Solution by Method of Constrained Variation
 For the optimization problem defined above, consider a specific case with n
= 2 and m = 1
 The problem statement is as follows:
Minimize f(x1,x2), subject to g(x1,x2) = 0
 For f(x1,x2) to have a minimum at a point X* = [x1*,x2*], a necessary
condition is that the total derivative of f(x1,x2) must be zero at [x1*,x2*].
df 
12
f
f
dx1 
dx2  0
x1
x2
Water Resources Systems Planning and Management: M2L2
(1)
D Nagesh Kumar, IISc
Method of Constrained Variation…
 Since g(x1*,x2*) = 0 at the minimum point, variations dx1 and dx2 about the
point [x1*, x2*] must be admissible variations, i.e. the point lies on the
constraint:
g(x1* + dx1 , x2* + dx2) = 0
(2)
assuming dx1 and dx2 are small the Taylor’s series expansion of this gives
us
g ( x1 *  dx1 , x2*  dx2 )
 g ( x1*, x2* ) 
13
g * *
g * *
(x1 ,x 2 ) dx1 
(x 1 ,x 2 ) dx2  0
x1
x2
Water Resources Systems Planning and Management: M2L2
(3)
D Nagesh Kumar, IISc
Method of Constrained Variation…
or
dg 
g
g
dx1 
dx2  0
x1
x2
at [x1*,x2*]
(4)
which is the condition that must be satisfied for all admissible
variations.
Assuming g / x2  0 , (4) can be rewritten as
g / x1 * *
dx2  
( x1 , x2 )dx1
g / x2
14
Water Resources Systems Planning and Management: M2L2
(5)
D Nagesh Kumar, IISc
Method of Constrained Variation…
(5) indicates that once variation along x1 (dx1) is chosen arbitrarily, the variation along x2
(dx2) is decided automatically to satisfy the condition for the admissible variation.
Substituting equation (5) in (1) we have:
 f g / x1 f 
df  

dx1  0

 x1 g / x2 x2  (x1* , x 2* )
(6)
The equation on the left hand side is called the constrained variation of f. Equation (5) has
to be satisfied for all dx1, hence we have
 f g f g 

0


 x1 x2 x2 x1  (x1* , x 2* )
(7)
This gives us the necessary condition to have [x1*, x2*] as an extreme point (maximum or
minimum)
15
Water Resources Systems Planning and Management: M2L2
D Nagesh Kumar, IISc
Thank You
Water Resources Systems Planning and Management: M2L2
D. Nagesh Kumar, IISc