Unconstrained optimization
Thabo Mboweni
March 23, 2021
Thabo Mboweni
Unconstrained optimization
Introduction
Optimization plays a major role in economic theory
The chapter turns from the matrix criteria that specify the
conditions for optimizing a quadratic form to the first and
second order derivative conditions that charectirize the optima
of a general differantiable function.
Just as techniques of calculus play a major role in
optimization problems for functions of one variable, they play
an equally important role for functions of several variables.
The main results for multivariate functions are analogous to
the one dimensional results:
1
2
a necessary consition for x0 to be an interior max of z = F (x)
is that the first derivatives of F at x0 be zero, and
An appropriate condition on the second derivatives of F ,
makes this necessary condition a sufficient condition.
Thabo Mboweni
Unconstrained optimization
First order conditions
The first order condition for a point x∗ to be a max or min of
a function of f of one variable is that f 0 (x ∗ ) = 0, that is, that
x ∗ be a critical point of f . The condition that x∗ be an
interior point of f .
The same first order consition works for a function F of n
variables, all partial derivatives must equal zero,i.e.
∂F /∂xi = 0, at x∗ .
Theorem
Let F : U → Rn be a C 1 function defined on a subset U of Rn . If
x∗ is a local max or min of F in U and if x∗ is an interior point of
U, then
∂F ∗
(x ) = 0 for i = 1, · · · , n.
∂xi
Look at Example 17.1
Thabo Mboweni
Unconstrained optimization
First order conditions
Example (7.1)
To find the local maxs and mins of F (x, y ) = x 3 − y 3 + 9xy , One
computes the first order partial derivatives and sets them equal to
zero:
∂F
= 3x 2 + 9y = 0
∂x
and
∂F
= −3y 2 + 9x = 0
∂y
The first equation yields: y = − 31 x 2 . And if you substitute this ino
the second equation:
1
1 2 2
+ 9x = − x 4 + 9x
0 = −3y + 9x = −3 − x
3
3
2
If you multiply both sides by 3, you get 27x − x 4 = x(27 − x 3 ) = 0
and the solutions are x = 0 and x = 3
Thabo Mboweni
Unconstrained optimization
First order conditions
Example (7.1)
Susititute this solutions into y = − 13 x 2 and the two solutions are
y = 0 if x = 0 and y = −3 if x = 3. Therefore the two solution
points are (0, 0) and (3, −3). At these point we have two
candidates for a local min and local max of F , but we cannot
which is a max or a min.
Thabo Mboweni
Unconstrained optimization
Second order conditions
Definition
We say that the n-vector x∗ is a critical point of a function
F (x1 , · · · , xn ) if x∗ satisfies
∂F ∗
(x ) = 0 for
∂xi
i = 1, · · · , n.
Hessian matrix for a function of n variables evaluated at the
critical point x∗ will give us the second order derivative
condition.
 2

∂2F
∂ F ∗
∗)
(x
)
·
·
·
(x
2
∂x
∂x
n
1

 ∂x1 .
..
..

.
D 2 F (x ∗ ) = 
.
.
.


∂2F
∂2F
∗
∗
(x )
∂x1 ∂xn (x ) · · ·
∂ 2 xn
Since cross-partials are equal for a C 2 function, D 2 F (x ∗ ) is a
symmetric matrix.
Thabo Mboweni
Unconstrained optimization
Sufficient conditions
The second order condition for a critical point x ∗ of a
function on R1 to be a max is that the second derivative
f 00 (x ∗ ) be negative.
For a function of F of n variables, the second derivative
D 2 F (x∗ ) be a negative definite as symmetric matrix at the
critical point x∗ .
Similarly, the second order sufficient condition for a critical
point of a function f of one variable to be a local min is that
f 00 (x ∗ ) be positive.
The analogous second order condition for an n-dimensional
critical point x∗ to be a local min is that the Hessian of F at
x∗ , D 2 F (x∗ ), be positive definite
Thabo Mboweni
Unconstrained optimization
Sufficient conditions
Theorem
Let F : U → R1 be a C 2 function whose domain is an open set U
in Rn . Suppose that x∗ is a critical point of F , then
1
2
3
If the Hessian matrix D 2 F (x∗ ) is a negative definite
symmetric matrix, then x∗ is a strict local max of F .
If the Hessian matrix D 2 F (x∗ ) is a positive definite symmetric
matrix, then x∗ is a strict local min of F .
D 2 F (x∗ ) is indefinite, then x∗ is neither a local max nor a
local min of F .
Thabo Mboweni
Unconstrained optimization
Sufficient conditions
Theorem
Let F : U → R1 be a C 2 function whose domain is an open set U
in Rn . Suppose that
∂F
= 0 for
∂xi
i = 1, · · · , n
and that the n leading principal minors of D 2 F (x∗ ) alternate in sign
Fx1 x1 < 0,
Fx1 x1
Fx1 x2
Fx2 x1
> 0,
Fx2 x2
Fx1 x1
Fx1 x2
Fx1 x3
Fx2 x1
Fx2 x2
Fx2 x3
at x∗ , then x∗ is a strict local max of F .
Thabo Mboweni
Unconstrained optimization
Fx3 x1
Fx3 x2 < 0, · · ·
Fx3 x3
Sufficient conditions
Theorem
Let F : U → R1 be a C 2 function whose domain is an open set U
in Rn . Suppose that
∂F
= 0 for
∂xi
i = 1, · · · , n
and that the n leading principal minors of D 2 F (x∗ ) are all positive.
Fx1 x1 > 0,
Fx1 x1
Fx1 x2
Fx2 x1
> 0,
Fx2 x2
Fx1 x1
Fx1 x2
Fx1 x3
Fx2 x1
Fx2 x2
Fx2 x3
at x∗ , then x∗ is a strict local min of F .
Thabo Mboweni
Unconstrained optimization
Fx3 x1
Fx3 x2 > 0, · · ·
Fx3 x3
Sufficient conditions
Theorem
Let F : U → R1 be a C 2 function whose domain is an open set U
in Rn . Suppose that
∂F
= 0 for
∂xi
i = 1, · · · , n
and that the n leading principal minors of D 2 F (x∗ ) violate the sign
patterns of the previous two theorems. Then x∗ is a saddle point
of F ; it is neither a local max nor min.
Thabo Mboweni
Unconstrained optimization
Necessary conditions
The second order necessary condition for a max or min of a
function of one variable is weaker than the second order
sufficient condition.
The necessary condition of f 00 (x ∗ ) ≤ 0 at a local max and
f 00 (x ∗ ) ≥ 0 at a local min, replaces the strict inequality of the
sufficient condition.
In the case of n-variables, one replaces the negative definite
and postive definite conditions on the Hessian of F in the
sufficient conditions by the requirement that the Hessian must
be negative semidefinite at a local max and positive
semidefinite at a local min.
Thabo Mboweni
Unconstrained optimization
Necessary conditions
Theorem
Let F : U → R1 be a C 2 function whose domain is an open set U
in Rn . Suppose that x∗ is an interior point of U and that x∗ is a
local max of F . Then, DF (x∗ ) = 0 and D 2 F (x∗ ) is negative
semidefinite.
Theorem
Let F : U → R1 be a C 2 of n variables. Suppose that x∗ is an
interior point of U.
1
2
If x∗ is a local min of F , then
(∂F /∂xi ) (x∗ ) = 0 for i = 1, · · · , n and all principal minors
of the Hessian D 2 F (x∗ ) are ≥ 0
If x∗ is a local min of F , then
(∂F /∂xi ) (x∗ ) = 0 for i = 1, · · · , n and all principal minors
of the Hessian D 2 F (x∗ ) of odd order are ≤ 0 and all the
principal minors of Hessian D 2 F (x∗ ) of even order are ≥ 0
Thabo Mboweni
Unconstrained optimization
Necessary conditions
Example
In example 7.1, We computed the critical ponits of
F (x, y ) = x 3 − y 3 + 9xy as (0, 0) and (3, −3), The hessian matrix
is given by
Fxx Fxy
6x
9
2
D F (x, y ) =
=
Fyx Fyy
9 −6y
The first LPM is detFxx = Fxy = 6x and the second LPM is
detD 2 F (x, y ) = −36xy − 81. At (0, 0), these LPMs are 0 and
−81, respectively. F is a saddle point- neither a max or min. At
(3, −3), these two minors are 18 and 243, Since these two numbers
are positive D 2 F (3, −3) is positive definite and (3, −3) is a strict
local min of F .
Thabo Mboweni
Unconstrained optimization
Global maxima and minima
The first and the second order sufficient conditions of the last
section will find all the local maxima and minima of a
differentiable function whose domain is an open set in Rn .
These conditions say nothing about whether or not any of
these local extrema is a global max or min.
The study of one-dimensional optimization problems put forth
two coditions for a critical point x ∗ of f to be a global max
(or min), when f is a C 2 function defined on a connected
interval I of R1 :
1
2
x ∗ is a local max (or min) and it’s the only critical value of f
in I ; or
f 00 ≤ 0 on all of I (or f 00 ≥ 0 on I for a min), that is, f is
concave function on I (or f is a convex function for a min).
Thabo Mboweni
Unconstrained optimization
Global maxima and minima
Theorem
Let F : U → R1 be a C 2 function whose domain is a convex open
subset U of Rn .
(a) The following three conditions are equivalent
(i) F is a concave function on U; and
(ii) F (y) − F (x) ≤ DF (x)(y − x) for all x, y ∈ U; and
(iii) D 2 F (x) is negative semidefinite for all x ∈ U
(b) The following three conditions are equivalent
(i) F is a convex function on U; and
(ii) F (y) − F (x) ≥ DF (x)(y − x) for all x, y ∈ U; and
(iii) D 2 F (x) is positive semidefinite for all x ∈ U
(c) If F is a concave function on U and DF (x∗ ) = 0 for some
x∗ ∈ U, then x∗ is a global max of F on U
(d) If F is a convex function on U and DF (x∗ ) = 0 for some
x∗ ∈ U, then x∗ is a global min of F on U
Tutorial: 17.1 and 17.2
Thabo Mboweni
Unconstrained optimization