A TEXT FOR NONLINEAR PROGRAMMING
Thomas W. Reiland
Department of Statistics
North Carolina State University
Raleigh, NC 27695-8203
Email: reiland@ncsu.edu
Table of Contents
Chapter I: Optimality Conditions..................................................Error! Bookmark not defined.
§ 1.1 Differentiability.................................................................Error! Bookmark not defined.
§ 1.2. Unconstrained Optimization ............................................Error! Bookmark not defined.
§ 1.3 Equality Constrained Optimization.................................................................................... 0
§ 1.3.1 Interpretation of Lagrange Multipliers......................................................................... 10
§ 1.3.2 Second order Conditions – Equality Constraints ......................................................... 11
§ 1.3.3 The General Case ......................................................................................................... 13
§ 1.4 Inequality Constrained Optimization ...............................Error! Bookmark not defined.
§ 1.5 Constraint Qualifications (CQ) ........................................Error! Bookmark not defined.
§ 1.6 Second-order Optimality Conditions ...............................Error! Bookmark not defined.
§ 1.7 Constraint Qualifications and Relationships Among Constraint Qualifications ..... Error!
Bookmark not defined.
Chapter II: Convexity ...................................................................Error! Bookmark not defined.
§ 2.1 Convex Sets ......................................................................Error! Bookmark not defined.
§ 2.2 Convex Functions ............................................................Error! Bookmark not defined.
§ 2.3 Subgradients and differentiable convex functions ............Error! Bookmark not defined.
1.3 Equality Constrained Optimization
page 1.3-1
§ 1.3 Equality Constrained Optimization
Recall that f ( x ) points in the direction from x in which the instantaneous rate of increase of f
is the greatest; the length of f ( x ) is equal to that maximum rate of increase.
Example 1.2 Continued: f ( x)  x12  x22
f ( x)  x12  x22  k provides circular contours
 2x 
f ( x)   1 
 2 x2 
2
 2   4
At x    , f     
 2   4
2
f (2)  4 2
Notice that f ( x) is perpendicular to any direction in
which the instantaneous rate of change of f is zero;
f ( x) is perpendicular to all surfaces of the form
f ( x)  k , a constant.
Equality Constrained Problem:
Max (or min) f ( x) , x  E n
s.t.
gi ( x)  bi , i  1,..., m (m  n)
f, gi’s have continuous 1st partial derivatives
This problem can reduce to the unconstrained problem by solving the constraints for m
variables in terms of the other n-m variables and substituting into the objective function.
Though an important idea, equality constraints have little theoretical impact on the problem as
they only reduce the dimensionality of the problem; they do not establish boundaries.
Example 1-9: Min x12  x22  x32
s.t. x1  x2  x3  10
The feasible region is a plane in E 3 framed by the triangle.
The solution is “constrained to the plane” but not bounded.
Definition 1.3.1 A point x satisfying the constraints gi ( x)  bi is a regular point of the
constraints if g1 ( x ), g 2 ( x ),
, g m ( x ) are linearly independent. In other words, x is a
regular point if the Jacobian matrix J evaluated at x is of full rank (m).
1.3 Equality Constrained Optimization
 g1 ( x )

 x1
J ( x )  


 g m ( x )
 x
1

page 1.3-2
g1 ( x ) 

xn 



g m ( x ) 
xn 
Theorem 1.5 Lagrange Multipliers
Let x  E n be any local maximum or minimum of f ( x) s.t. gi ( x)  bi , i  1,
,m ,

m<n. If x is a regular point of the constraints, the there exists a unique set of Lagrange
multipliers  ,

1
m
,  such that f ( x )   igi (x )  0 .

m
i 1
Remark: Define the Lagrangian function by F ( x,  )  f ( x)   i [ gi ( x)  bi ] . Then the
necessary conditions of Theorem 1.5 become:
 x F ( x ,   )  f ( x )   i gi ( x )  0
  F ( x ,   )  g ( x )  b  0
Or, putting these together yields:
F ( x ,   )  0, where F is an (m  n) 1vector.
Implicit Function Theorem
Let x 0  ( x10 ,..., xn0 )T  E n satisfy the following:
(i) The functions gi are continuously differentiable in a neighborhood of x 0 .
(ii) gi ( x 0 )  bi , i  1,..., m
(iii) The m n Jacobian is nonsingular ( x 0  E n is a regular point of the constraints).
Then there exists a neighborhood of xˆ 0  ( xm0 1 ,..., xn0 )T  E n  m such that for
xˆ  ( xm 1 , xm  2 ,..., xn )T in this neighborhood there exist functions i ( xˆ ), i  1, 2,..., m such that
(I) i are continuously differentiable
(II) xi0  i ( xˆ 0 ), i  1, 2,..., m (the i are implicit functions)
(III) gi (1 ( xˆ ), 2 ( xˆ ), , m ( xˆ ), xˆ )  bi , i  1, 2, , m
Example 1-10: Min f ( x)  x12  x22
s.t. 3x1  4 x2  6
n=2
m=1
Solve for an implicit function: x1 
6  4 x2
3
 6  4 x2 
2
fˆ ( x2 )  
  x2
 3 
2
Substitute into the objective function:
1.3 Equality Constrained Optimization
page 1.3-3
Now the problem can be readily solved using the first derivative (unconstrained).
Example 1-11: Given the constraint x12  x22  0 , n = 2, m = 1, and let x 0  (0, 0)T
Condition (iii) is violated since J ( x0 )  011 which is singular.
Therefore, no  exist such that x1   ( x2 ) .
x1   x22  i x22  ix2
Proof of Theorem 1.5:
Since x is a regular point of the constraints, the Implicit Function Theorem implies that
there exists continuously differentiable 1 , 2 , , m such that
(1)
xi  i ( xˆ  ), i  1,
where xˆ   ( xm 1 , xm  2 ,
(2)
,m
, xn )T , the objective function f ( x ) becomes
fˆ ( xˆ )  f (1 ( xˆ ), 2 ( xˆ ),
, m ( xˆ ), xˆ ) .
This function has an unconstrained local maximum or minimum at xˆ   E n  m so
(3)
fˆ ( xˆ  )
 0, j  m  1, m  2,
x j
,n
Apply the chain rule for partial differentiation to (2):
m
fˆ
f k f
at x̂ for j  m  1, , n .


0
x j k 1 xk x j x j
f
f
Note: Use
instead of
because using (1) they are equivalent.
xk
 k
(4)
By the Implicit Function Theorem, gi (1 ( xˆ  ), 2 ( xˆ  ),
, m ( xˆ  ), xˆ  )  bi , i  1,
, m , at x̂ and
all x̂ in a neighborhood of x̂ ; thus the partial derivatives of gi with respect to the remaining
components xm 1 , , xn vanish at all such points (function in constant in neighborhood):
(5)
gi m gi k

 0, i =1,
x j k 1 xk x j
, m; j  m  1,
,n .
Since x is a regular point, define the unique numbers 1 , 2 ,
in (6).
, m by means of the equations
1.3 Equality Constrained Optimization
(6)
page 1.3-4
f ( x ) m  gi ( x )
  i
, j  1,
x j
x j
i 1
,m
For a specific j, (5) represents m equations (one equation for each i); multiply each equation by
the respective i  and sum over i to obtain (7).
m
(7)
 i
i 1
gi m k

x j k 1 x j
 m  gi
  i
 i 1 x j

  0, j  m  1,

,n .
Evaluating (7) at x and subtracting from (4) gives
 f ( x ) m  gi ( x )  k ( x ) f ( x ) m  gi ( x )
  i

  i
 0, j  m  1,



xk  x j
x j
x j
k 1  xk
i 1
i 1
m
,n .
But from (6), the term in brackets is zero for every k, leaving
f ( x ) m  gi ( x )
  i
 0, j  m  1,
x j
x j
i 1
,n
We combine this with (6) to get the desired result:
m
f ( x )   i gi ( x )  0 . QED
i 1
______________________________________________________________________________
Problem P : Min f ( x) , f has continuous 1st partials on some open subset containing Ω
x   E n
Definition A vector d  E n is a feasible direction at x if there exists a   0 such that
x   d  ,  , 0     .
Theorem If x is a local minimum point of f over Ω, then for any feasible direction d  E n ,
f ( x )T d  0 .
Proof: Suppose f ( x )T d  0 for some feasible d. From Def. of Differentiability,
f ( x   d )  f ( x )  f ( x )T d   d  ( x   d ; x ) for 0     .
f (x  d )  f (x )

 f ( x )T d  d  ( x   d ; x )
1.3 Equality Constrained Optimization
lim
f (x  d )  f (x )
 0

page 1.3-5
 f ( x )T d  0
 f ( x   d )  f ( x )  0 for τ sufficiently small.
 f ( x   d )  f ( x ) for τ sufficiently small. # This contradicts the fact that x is
a local minimum. QED.
Remark: If x  int() or if   E n , then f ( x )T d  0 for d  E n implies f ( x )  0
(Theorem 1.13).
Example 1-12: f ( x)  x 2 ,   [1, 2]
d  E1 is feasible at x  1 if d  0
x  1 is a global minimum on Ω
f ( x )  f ( x )  2 x  2 at x  1
f ( x )T d  f ( x )d  2d  0 for d feasible
Example 1-13: Min f ( x1 , x2 )  x12  x1  x2  x1 x2 ,   ( x1 , x2 )T  E 2 | x1  0, x2  0

1
f has a global minimum at x1  , x2  0
2
0
 2 x1  1  x2 
f ( x )   3 
f ( x)  

 
x

1
1


2
3
d 
T
d   1   E 2 is feasible at x if d 2  0 since f ( x ) d  d 2  0
2
 d2 
( d1 can have any value)
Theorem – 2nd Order Necessary Conditions
Let f have continuous second partial derivatives on an open set containing   E n . If x
is a local minimum of f over Ω, then for any feasible direction d  E n at x we have
(4a)
f ( x )T d  0
(5a) if f ( x )T d  0 , then d T 2 f ( x )d  0 .
Remark: If x  int() or   E n , then (5a) becomes (5) in Theorem 1.3:
d T 2 f ( x )d  0 , d  E n .
Proof: (4a) already proven in previous proof. Suppose d T 2 f ( x )d  0 for some feasible
direction d where f ( x )T d  0 .
1.3 Equality Constrained Optimization
page 1.3-6
f ( x   d )  f ( x )  f ( x )T d   d T  2 f ( x ) d   d
f (x  d )  f (x )

lim
2
 d T  2 f ( x )d  d
f (x  d )  f (x )
 0

2
2
2
 (x  d ; x )
 (x  d ; x )
 d T  2 f ( x )d  0
 f ( x   d )  f ( x )  0 for τ small enough
 f ( x   d )  f ( x ) for τ small enough. # QED.
Example 1-13 Continued: Min f ( x1 , x2 )  x12  x1  x2  x1 x2 ,   ( x1 , x2 )T  E 2 | x1  0, x2  0
T
1

0
Global Minimum occurs at x  
2

3
f ( x )T d  d 2
(5a) applies when d 2  0
2
2 1
 2 f ( x)  
 2 f ( x )

1 0
d T  2 f ( x )d  2d12  2d1d 2  2d12  0 when d 2  0 .
Example 1-14: Min f ( x1 , x2 )  x13  x12 x2  2 x22 ,    x  E 2 | x1  0, x2  0

Suppose solution is in interior of Ω (ignore constraint).
3x12  2 x1 x2 
0 
6

f ( x)   2
  0  x    or x   
0 
9 
  x1  4 x2 
 18 12
2 f ( x )  
 which is not positive definite
 12 4 
 18   12  
det 2 f ( x )   I   det  
   (18   )(4   )  144  0

12
4





2
   22  72  0
22  484  288
772
28

 11 
 11 
2
2
2
2
  f ( x ) is indeterminant and x is not a local minimum.
6 x  2 x2
 2 f ( x)   1
 2 x1
2 x1 
4 
Example 1-15: Max f ( x1 , x2 )  x1 x2
s.t. g ( x1 , x2 )  x1  x2  2
F ( x,  )  x1 x2   ( x1  x2  2)
Set F ( x ,   )  0
1.3 Equality Constrained Optimization
(1)
F ( x ,   )
 x2     0
x1
(2)
F ( x ,   )
 x1     0
x2
page 1.3-7
F ( x ,   )
 ( x1  x2  2)  0


(1) and (2)    x1  x2
(3)
(3)  x1  x2  2   2     1
So  x1  x2     1 satisfies the necessary conditions for a local maximum or
minimum.
f ( x ) is a linear combination of g ( x )
Theorem 1.5 gives only necessary conditions. Consider Example 1.16 below.
Example 1-16: Max or min f ( x)  x13  x1 x2  x 2
s.t. g ( x)  x2  0
F ( x,  )  x13  x1 x2  x 2   x2
 x F ( x ,   )  0
(1)
3 x12  x2  0
(2)
3 x1  1     0
  F ( x ,   )  0  x2  0
From (1), x1  0 . From (2),    1 .
x  (0, 0)T ,    1 is the only candidate but x is neither a local maximum or local
minimum.
f ( x )  0
f (ò,0)  ò3  0 , f (ò, 0)  ò3  0 , ò  0
Note that Max f   at x1  , x2  0
Min f   at x1  , x2  0
_____________________________________________________________________________
2nd Derivation of Lagrange Multipliers
1.3 Equality Constrained Optimization
page 1.3-8
Max (or min) f ( x) x  E n
s.t. gi ( x)  bi , i  1, , m
m < n ; f, gi’s have continuous 1st partial derivatives
Note: gi ( x)  bi , i  1, , m defines surface of dimension n-m (no gi are redundant).
Associated with each point on surface S defined by the constraints is the tangent plan at that
point.
Definition 1.3.2 A curve on a surface S is a family of points x (t )  S continuously
parameterized by t for a  t  b . The curve is differentiable if dx(t ) exists, and is twice
dt
2
differentiable if d x(t ) 2 . A curve x(t ) is said to pass through the point x if x(t  )  x for
dt

some t  , a  t  b . The derivative of the curve at x is dx (t ) .
dt
dx
(
t
)
 1

dt 

dx(t )  

dt 

 dxn (t ) 
dt  n1

Definition 1.3.3 Consider all differentiable curves on S passing through a point x . The
tangent plane at x is the collection of the derivatives at x of all these differentiable curves.
The tangent plan is a subspace of E n . Suppose a surface is definite by the constraints
gi ( x)  bi ; we want to obtain explicit representation for the tangent plane. In particular, we want
to express the tangent plan in terms of gi .
 g1 ( x) 
i  1, m
 , g ( x)   gi ( x) 
Let g ( x)  
( g ( x) is the Jacobian).



j

1,
,
n

x


j

 m n
 g m ( x) 
Define M   y  E n : g ( x ) y  0 is a subspace of E n . We want to investigate the
conditions under which M is the tangent plane at x .
Theorem 1.6 At a regular point x of the surface defined by g ( x)  b the tangent plane is equal
to M   y  E n : g ( x ) y  0 .
Example 1-17: In E 2 , let g1 ( x1 , x2 )  x1 . Then  x  E 2 : g ( x1 , x2 )  0 is the x2 axis. Since
g1 ( x)  1 0 , every point on the x2 axis is a regular point. The tangent plan at any regular
T
1.3 Equality Constrained Optimization
point is the x2 axis:
y  E
2
page 1.3-9

:[1, 0] y  0  x2 axis. M is not a regular point when x is not a
regular point.
Example 1-18: Let g1 ( x1 , x2 )  x12 ;
g1 ( x1 , x2 )   2 x1
x  E
2

: g ( x1 , x2 )  0 is the surface S = x2 axis.
0 so g1   0 0 for any point in S; so no points in S are regular points.
T
T
In this case, M  E 2 , but the tangent plane is the x2 axis.
Theorem 1.7 Let x be a regular point of the constraints g ( x)  b and a local minimum or
maximum of f subject to these constraints. Then all y  E n satisfying g ( x ) y  0 must also
satisfy f ( x )T y  0 .
In other words, f ( x ) is perpendicular to the tangent plane. This implies that f ( x ) is
a linear combination of the gradients of g at x .
Proof: Let y be any vector in the tangent plane at x and let x(t ) be any differentiable
curve on the constraint surface passing through x with derivative y at x .
x(0)  x ,
dx(0)
 y, g ( x(t ))  b for a  t  a for some a  0
dt
Since x is a regular point, the tangent plane is identical with the y ' s  E n such that
g ( x ) y  0 . Since x is a constrained local extremum point of f, we have the following:
d
f ( x(t )) t 0  0
dt
n
f ( x(0)) dx j (0)
 
0
x j
dt
j 1
T
 f ( x )  y  0
QED.
Theorem 1.8 Lagrange Multipliers
Let x be a local minimum or maximum point of f subject to g ( x )  b . Assume also that
x is a regular point of the constraints. Then there exist    E m such that
m
f ( x )   igi ( x )  0 .
i 1
Proof: Consider the LP ( P ) :
s.t.
Max z  cT x
Ax  b , A is m  n, m  n
1.3 Equality Constrained Optimization
page 1.3-10
Assume the objective is bounded. The partial derivatives are
gi
 ai j .
x j
By Theorem 1.5, at the (finite) maximum x0 there exist i , i  1,
f ( x0 ) m gi
  i
 0, j  1,
x j
x j
i 1
, m , such that
,n .
m
 c j   i ai j  0, j  1,
,n
i 1
T
zmax
In matrix form, A   c . The optimal value of the objective function
 cT x0  x0T c  x0T AT   bT  .
Since x0 is feasible, the Lagrange multipliers are just the dual variables.
If x0 is optimal for Problem ( P ), there exist   E n such that AT   c and cT x0  bT  .
For Problem ( P ), we have proved the duality theorem using Lagrange Multipliers.
i is the rate of change of the optimal value of f as the right hand side constraint bi
changes.
§ 1.3.1 Interpretation of Lagrange Multipliers
Consider the following problem:
Max z  f ( x), x  E n
s.t. gi ( x)  b0i , i  1, , m, m  n
The global optimum occurs at x with associated Lagrange multipliers    E m . If we substitute
different bi ’s, the rest of the problem remains unchanged, though we would expect the maximum
point to occur at a different point. Suppose each xj and i are continuously differentiable
functions of b in some neighborhood of b0.
Theorem 1.3.1 Interpretation of Lagrange Multipliers
Let z   f ( x ) . Take the partial derivatives of z   f ( x ) with respect to b in the
neighborhood of b0. Using the chain rule:

(1)
n
z 
f ( x ) x j

bi j 1 x j bi
From g k ( x )  b0 k , we have
(2)

g k ( x ) n g k ( x ) x j 1 k  i


bi
x j bi 0 k  i
j 1
1.3 Equality Constrained Optimization
page 1.3-11
For any specific i, (2) represents m equations (one for each k); multiplying each by the
respective k , k  1, , m , and summing over k gives

n x m
g k x j
g
j
 

k k  i


x j
k 1
j 1 x j bi
j 1 bi k 1
m
(3)

k
n
Substituting the necessary conditions into (3) gives
xj f ( x )
 i

x j
j 1 bi
n
Hence from (1) we have the result below:
i 
z 
, i  1,
bi
,m
The Lagrange multipliers associated with the global optimum point gives the rate of
change of the optimal value of the objective function with respect to changes in the right
hand side of the constraints.
§ 1.3.2 Second order Conditions – Equality Constraints
Consider the problem posed at the beginning of §1.3.1 except now the functions are twice
differentiable.
Theorem 1.9 2nd Order Necessary Conditions
Suppose x is a local maximum of f subject to gi ( x)  bi , i  1,
, m, m  n , and that x
is a regular point of the constraints. There exist a unique    E m such that
m
f ( x )   i  gi ( x )  0
i 1
If we denote by M the tangent plane M   y  E n : g ( x ) y  0 , then
m
 2x F ( x ,   )   2 f ( x )   i 2 gi ( x )
i 1
2
x
is negative semidefinite on M.  y  F ( x ) y  0, y  M
T
Theorem 1.10 2nd Order Sufficient Conditions
1.3 Equality Constrained Optimization
page 1.3-12
Suppose there exists a point x satisfying gi ( x )  bi , i  1,
, m, and a    E m such
m
m


that f ( x )   i  gi ( x )  0 . Suppose also that yT  2 f ( x )   i 2 gi ( x )  y  0 for all
i 1
i 1


n

y  M   y  E : g ( x ) y  0 , y  0 ,
Then x is a strict local maximum of f subject to gi ( x )  bi , i  1,
,m .
Example 1-17: Max x1 x2  x2 x3  x1 x3
s.t. x1  x2  x3  3
f ( x )   g ( x )  0
(1)
x2  x3    0
x1  x3    0
(2)
x1  x2    0
(3)
(1)  x2  x3  
(1) and (2)  x1  x2  0
(1) and (3)  x1  x3  0
 x1  x2  x3  1,    2
0 1 1 
 f ( x)  1 0 1 
1 1 0
 2 g ( x)  0
2
0 1 1 
 F ( x ,  )  1 0 1 
1 1 0 

2
x



 0
M  y  E 3 : g ( x  ) y  0

 y  E 3 : y1  y2  y3
so yT  2x F ( x ,   ) y  y1 ( y2  y3 )  y2 ( y1  y3 )  y3 ( y1  y2 )
 ( y12  y22  y32 )  0 for y  0 so x is at least a local maximum
Example 1-18: Max or min f ( x)  x13  x1 x2  x2
s.t. g ( x)  x2  0
candidate: x   0 0 ,    1
T
1.3 Equality Constrained Optimization
page 1.3-13
3x 2  x2 
f ( x)   1

 x1  1 
0 
g ( x)   
1 
6 x 1 
 2 f ( x)   1

 1 0
0 1 
 2 F ( x ,   )  

1 0 
0 0 
 2 g ( x)  

0 0

y T  2 F ( x ,   ) y  0
 

M  y  E 2 : g ( x  ) y  0  y  E 2 : y 2  0
§ 1.3.3 The General Case
The necessary conditions can be generalized further for cases when the Jacobian is not assumed
to be at full rank.
Consider the problem:
Max or min f ( x)
x  En
s.t. gi ( x)  bi , i  1, , m, m  n
f, gi’s have continuous 1st partial derivatives
Define the Augmented Jacobian to have m+1 rows and n columns:
g1 
 g1
 x
xn 
 1



J  


J f      g m
g m 
f   x
xn 
1


f 
 f
 x1
xn  ( m 1)n
Assume x is a local maximum or minimum and the augmented Jacobian matrix is not
full rank at x . r ( J f )  m  1 at x .
When r ( J f )  m  1 at x then x cannot be a relative maximum or minimum.
Consider the equations:
()
0f ( x )  1g1 ( x )  mg m ( x )  0
Which can be rewritten as:
1.3 Equality Constrained Optimization
page 1.3-14
f ( x0 )
g1 ( x0 )
g m ( x0 )
0
 1
 m
 0, j  1,, n
x j
x j
x j
1
2
m 0  J f  0 (Linear Combination of the rows of Jf)
Since r ( J f )  m  1 at x , there exist nontrivial solutions to (  ) . If r ( J f )  m  1 , then
m+1 rows are linearly independent and the only way a linear combination of the rows is zero is if
all coefficients are zero.
Case (i) r ( J f )  m at x
We can select any m equations (i.e. columns of Jf ) containing a nonsingular submatrix of
order m whose elements are in J, and solve uniquely for 1 ,, m in terms of 0 (m equations
with m + 1 unknowns). The value of 0 can be designed arbitrarily; it cannot be zero, however,
if we want a nontrivial solution. For convenience, set 0  1 .
Case (ii) r ( J f )  m at x , but r ( J )  m  1
Select m equations from (  ); the matrix of whose coefficients contains a nonsingular
matrix of order m. There does not exist a solution to this set of equations which does not have
0  0 . If 0  0 , there exist 1 ,, m , not all zero, which satisfy (  ). The λi are not uniquely
determined, but instead, the solutions span a one dimensional subspace of E m .
1
In the figure below, g1 ( x0 )  3g 2 ( x0 ) . 1  2
3
Example 1-19: Let f ( x ) denote components of f ( x ) chosen in m equations from (  ).
0f  1g1   mgm  0
m

0  0  f ( x )   i gi
i 1 0
1.3 Equality Constrained Optimization
page 1.3-15
 r ( J f )  m if r ( J )  m  1
Example 1-20: There exist i , i  1,, m , not all zero such that 1g1  mg m  0
r ( J )  m  1 , so one gradient, say g1 , is a linear combination of the others.
m
g1    i gi
i 1
 m

 1    i gi   2g 2   mg m  0
 i 2

 (1 2  2 )g2  (13  3 )g3   (1 m  m )gm  0
Since r ( J )  m  1 , all coefficients must equal zero.

 1  2
2
1 
1 
3
3
m
m
Each equation for 1 is a hyperplane (of dimension m 1 ) in m-space; the intersection of
m 1 hyperplanes is a 1-dimensional space.
Case (iii) r ( J f )  r ( J )  r  m at x
Select r equations from (  ) (r columns from Jf ). There exists the matrix of coefficients
that contains a nonsingular submatrix of order r whose elements are in J. Then we can solve for
r of the λ’s in terms of 0 and the other m  r λ’s. 0 is arbitrary; for convenience set 0  1 .
m  r of the λ’s are also arbitrary.
1.3 Equality Constrained Optimization
page 1.3-16
Case (iv) r ( J f )  r  m, r ( J f )  r ( J ) at x
Select r equations from (  ) the matrix of whose coefficients contains a nonsingular
submatrix of order r. As in case (ii), there are no solutions which do not have 0  0 . If we set
0  0 , there exist i , i  1,, m , not all zero, which satisfy (  ). The i are not unique but
span a m  r  1 dimensional subspace of E m .
In the figure below, m  r  1  2 . To specify i :
g1 ( x )  2 g3 ( x )
3

g1 ( x )  2g 2 ( x )
3
 i gi ( x )  0
i 1
 g1 ( x )(1  1 2  3 3 )  0
2
2
3
1

  2 3  1
2 2
Theorem 1.11 Let f and gi have continuous partial derivatives. If x is a local maximum or
minimum of f ( x) subject to gi ( x)  bi , i  1,, m , then there exist m  1 real numbers
0 , 1 ,, m , not all zero such that:
m
0f ( x )   igi (x )  0
i 1
Example 1-21: Min f ( x)   x1
x  E3
s.t. g1 ( x)  (1  x1 )3  x2  0
g 2 ( x)  x2  0
x  (1, 0, x3 )T has minimizing point x  (1,0,0)T .
1.3 Equality Constrained Optimization
 1
f ( x )   0 
 0 
page 1.3-17
0
g1 ( x )   1
 0 


0 
g 2 ( x )  1 
0 

Looking
from above:
We have case (ii). r ( J f )  2 , r ( J )  1
2
0f ( x )   igi (x )  0
i 1
We can see pictorially that 0  0, 1  2
 1
0
0




 
  0   1  1  2 1   0
 0 
 0 
0
  0
  0 
1  2  0

0
 1
Here the objective function f plays no role in the solution. Any f such that f ( x )   0 
 k 
will work, where k  E1 . For example, f ( x)   ln( x1 )  x23 x32  e x3 would also give (1, 0, x3 )T as
a candidate point.

Theorem 1.12 If r ( J f ) at x0 is m  1 , then f ( x) does not take on a local maximum or
minimum at x0 .
Proof: Assume r ( J f )  m  1 at x0 . In particular, let the first m  1 columns of Jf
form an nonsingular submatrix of order m  1 at x0 . Then the Implicit Function Theorem
implies that for the m  1 equations:
1.3 Equality Constrained Optimization
 
page 1.3-18
gi ( x)  bi , i  1,, m, f ( x)  z  0
In n  1 variables x1 ,, xn , z, there exists an ε-neighborhood of  xm0  2 , , xn0 , z0  , where
z0  f ( x0 ) , in which we can solve   explicitly for x1 ,, xm 1 to yield the following:
xi  i ( xm  2 ,, xn , z ), i  1,, m  1
Note that z is one of the variables that can be assigned an arbitrary value in i .
Consequently, if we set xm  2  xm0  2 , , xn  xn0 , then z  z  z0   , there exists x such that
f ( x)  z . Since the functions i are continuous (from the Implicit Function Theorem) it follows
that if we select any δ-neighborhood such that xm  2  xm0  2 , , xn  xn0 with the property that for
some of these points, f ( x)  z0 , while for others f ( x)  z0 . QED.
Result from the Duality Theorem: If the problem is unbounded, no  ’s will exist. If
the problem is infeasible, there will be an infinite number of  ’s.
Example 1-22: Max x1  x2
s.t. x1  x2  1
1  1  0
1  2  0
2  0 No solution for 
Problem is Unbounded.
Example 1-23: Max 0T x (this is trivial with equality constraints)
s.t. x1  x2  x3  1
x1  x2  x3  0
1  2  0

1  2  0  1  2
1  2  0 
There are an infinite number of solutions.
This example is like case (iii). r ( J f )  r ( J )  r  m
1.3 Equality Constrained Optimization
page 1.3-19
Here m = 2, r = 1.
1 1 1
J f  1 1 1 Can solve for r  ’s in terms of the other m  r  ’s and 0 .
0 0 0 
Example 1-24: Examples of cases (i) – (iv)
Min x12  x22  ( x3  1) 2
s.t. x1 x2  x3  1
J   x2
x1 1 , r ( J )  1 full rank
2 x1  1 x2  0
(1)
2 x2  1 x1  0
(2)
2( x3  1)  1  0
(3)
 1  2(x3  1)
(1)  2 x1  2 x2 ( x3  1)  0
(2)  2 x2  2 x1 ( x3  1)  0
 x1  x2 ( x3  1)
x2  x1 ( x3  1)
(5) x1  x2  ( x2  x1 )( x3  1)
(i) if x2  x1 , x1 , x2  0
x3  1  1
x3  0, 1  2
(1) and (2)  2 x1  2 x2  0
(4)  x1 x2  1
This results in the unpleasant x12  1 .
(ii) x2  x1 , x1  0
(5)  x3  0  1  2 x3
(4) x1  0  x3  1  1  0
Also not good.
(iii) x1  x2
(a) x1  x2  0 (4)  x3  1  1  0
(b) x1  x2  0
(4)  x3  1  x12
Case (ii) r ( J f )  m at x0 , r ( J )  m  1 at x0
1.3 Equality Constrained Optimization
page 1.3-20
0  0; i ’s not unique
Case (iii) r ( J f )  r ( J )  r  m
2 0
1 
 1
 2 1 1
2 
Jf  
 0
5 1
4 


 2 x1 0 0 2( x4  1) 
0 
1 
r ( J f )  r ( J )  2 at x0   
1 
 
1 
Can solve for r = 2 of the λ’s in terms of λ0 and the other m – r = 3 – 2 λ’s. For
convenience, let λ0 = 1.
0 
1 
Solving you get x   
1 
 
1 
 1 
 2
     1 
2 
 2

 3 
 1 
Case (iv) r ( J f )  r  m and r ( J f )  r ( J )
0  0 , λi’s not unique but span m – r + 1 dimensional subspace of E m .
Example 1-25: Min x12  ( x4  1)2  x52
s.t. x1  2 x2  x4  2 x5  4
2 x1  x2  x3  x42  x5  4
5 x2  x3  x42  x5  10
x1  1 x2  1 x3  1 x42  1 x5  2
2
2
2
2
n  5, m  4, x0   0 1 1 2 0
T
2
1
 2 1

5
J f  0

1 1 2

0
0
0
1
1 4
1 6
1
2
2
0
2
2
1 
3

1 
2

0
r ( J f )  3  r, r ( J )  2
At x0, 2g1  g2  g3 and 0g1  1 g 2  g 4
2
1.3 Equality Constrained Optimization
()
f ( x0 ) 5
g ( x )
0
  i i 0  0, j  1,, 5 choose j  1,3, 4
x j
x j
i 1
(1)
1  22  4  0
(2)
21  2  53  1 4  0
2
1
2  3  4  0
2
20  1  42  63  24  0
(3)
(4)
(5)
page 1.3-21
21  2  33  1 4  0
2
We can choose r = 3 equations from () (i.e. choose three columns from Jf ) such that
the 5  3 matrix of coefficients has a 3  3 nonsingular submatrix. Chose columns 1, 3, 4 (i.e.
choose equations (1), (3), and (4).
There exist i , i  1,, 4 , not all zero, which satisfy equations (1) – (5). The λi’s are not
unique but span an m  r 1  4  3 1  2 dimensional subspace of E m  E 4 .
(3)  2  3  1 4
(4)  20  0
2
(1)  1  22  4  23
3 , 4 arbitrary