Optimization in Continuous Time
Jesús Fernández-Villaverde
University of Pennsylvania
November 9, 2013
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
1 / 28
Motivation
Three Approaches
We are interested in optimization in continuous time, both in
deterministic and stochastic environments.
Elegant and powerful math (di¤erential equations, stochastic
processes...).
Three approaches:
1
Calculus of Variations.
2
Optimal Control.
3
Dynamic Programming.
We will focus on the last two:
1
Optimal control can do everything economists need from calculus of
variations.
2
Dynamic programming is better for the stochastic case.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
2 / 28
Deterministic Case
Maximization Problem I
Basic setup:
V (0, x (0)) = max
Z ∞
x (t ),y (t ) 0
f (t, x (t ) , y (t )) dt
s.t. ẋ = g (t, x (t ) , y (t ))
x (0) = x0 , lim b (t ) x (t )
t !∞
x1
x (t ) 2 X , y (t ) 2 Y
x (t ) is a state variable.
y (t ) is a control variable.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
3 / 28
Deterministic Case
Maximization Problem II
Admissible pair: (x (t ) , y (t )) s.t. the previous conditions are
satis…ed.
Optimal pair: (b
x (t ) , yb (t )) that reach V (0, x (0)) < ∞.
Then:
V (0, x (0)) =
Z ∞
0
f (t, b
x (t ) , yb (t )) dt
Two di¢ culties:
1
We need to …nd a whole function y (t ) of optimal choices.
2
The constraint is in the form of a di¤erential equation.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
4 / 28
Deterministic Case
Optimal Control
Optimal Control
Pontryagin and co-authors.
Principle of Optimality
If (b
x (t ) , yb (t )) is an optimal pair, then:
V (t0 , x (t0 )) =
for all t1
t0 .
Z t1
t0
f (t, b
x (t ) , yb (t )) dt + V (t1 , b
x (t1 ))
We will assume that there is an optimal path.
Proving existence is, however, not a trivial task.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
5 / 28
Deterministic Case
Optimal Control
Hamiltonian
De…ne:
H (t, x (t ) , y (t ) , λ (t )) = f (t, x (t ) , y (t )) + λ (t ) g (t, x (t ) , y (t ))
where λ (t ) is the co-state multiplier.
Necessary conditions:
Hy (t, b
x (t ) , yb (t ) , λ (t )) = 0
Hx (t, b
x (t ) , yb (t ) , λ (t ))
ẋ = Hλ (t, b
x (t ) , yb (t ) , λ (t ))
λ̇ (t ) =
plus x (0) = x0 and limt !∞ b (t ) x (t )
Jesús Fernández-Villaverde (PENN)
x1 .
Optimization in Continuous Time
November 9, 2013
6 / 28
Deterministic Case
Optimal Control
Exponential Discounting Case I
More speci…c form:
V (x (0)) = max
Z ∞
x (t ),y (t ) 0
e
ρt
f (x (t ) , y (t )) dt
s.t. ẋ = g (x (t ) , y (t ))
x (0) = x0 , lim b (t ) x (t )
t !∞
x1
x (t ) 2 IntX , y (t ) 2 IntY
g (x (t ) , y (t )) being autonomous is not needed but it helps to
simplify notation.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
7 / 28
Deterministic Case
Optimal Control
Exponential Discounting Case II
Hamiltonian:
H (t, x (t ) , y (t ) , λ (t )) = e
= e
ρt
f (x (t ) , y (t )) + λ (t ) g (x (t ) , y (t ))
ρt
[f (x (t ) , y (t )) + µ (t ) g (x (t ) , y (t ))]
where
µ (t ) = e ρt λ (t )
Current-Value Hamiltonian:
b (x (t ) , y (t ) , µ (t )) = f (x (t ) , y (t )) + µ (t ) g (x (t ) , y (t ))
H
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
8 / 28
Deterministic Case
Optimal Control
Maximum Principle for Discounted In…nite-Horizon
Problems
Theorem
Under some technical conditions, the optimal pair (b
x (t ) , yb (t )) satis…es
the necessary conditions:
1
2
3
4
b y (x (t ) , y (t ) , µ (t )) = 0 for 8t 2 R+ .
H
b x (x (t ) , y (t ) , µ (t )) = ρµ (t ) µ̇ (t ) for 8t 2 R+ .
H
b µ (x (t ) , y (t ) , µ (t )) = ẋ (t ) for 8t 2 R+ ,x (0) = x0 and
H
limt !∞ x (t ) x1 .
b (x (t ) , y (t ) , µ (t )) = limt !∞ e ρt µ (t ) x̂ (t ) = 0.
limt !∞ e ρt H
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
9 / 28
Deterministic Case
Optimal Control
Su¢ ciency Conditions
Previous theorem only delivers necessary conditions.
However, we also need su¢ cient conditions.
Theorem
Mangasarian Su¢ cient Conditions for Discounted In…nite-Horizon
Problems. The necessary conditions will be su¢ cient if f and g are
continuously di¤erentiable and weakly monotone and
H (t, x (t ) , y (t ) , λ (t )) is jointly concave in x (t ) and y (t ) for 8t 2 R+ .
We will skip su¢ ciency arguments. They will be relevant later in
models of endogenous growth.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
10 / 28
Deterministic Case
Optimal Control
Example I
Consumption-savings problem:
V (a) = max
a,c
Z ∞
e
ρt
u (c ) dt
0
ȧ = ra + w
c
Hamiltonian:
b (a, c, µ) = u (c ) + µ (ra + w
H
c)
Necessary conditions:
b c (a, c, µ) = 0 ) u 0 (c )
H
b a (a, c, µ) = ρµ
H
Jesús Fernández-Villaverde (PENN)
µ = 0 ) u 0 (c ) = µ
µ̇
µ̇ ) r µ = ρµ µ̇ ) = (r
µ
Optimization in Continuous Time
ρ)
November 9, 2013
11 / 28
Deterministic Case
Optimal Control
Example II
Then:
u 00 (c ) ċ = µ̇ )
µ̇
u 00 (c )
ċ = =
0
u (c )
µ
(r
ρ)
Assume, for instance:
u (c ) = log c
and we get:
ċ
=r ρ
c
Comparison with bang-bang solutions (linear returns and bounded
controls).
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
12 / 28
Deterministic Case
Dynamic Programming
Dynamic Programming
Dynamic programming is a more ‡exible approach (for example, later,
to introduce uncertainty).
Instead of searching for an optimal path, we will search for decision
rules.
Cost: we will need to solve for PDEs instead of ODEs.
But at the end, we will get the same solution.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
13 / 28
Deterministic Case
Dynamic Programming
Hamilton-Jacobi-Bellman (HJB) Equation
When V (t, x (t )) is di¤erentiable, (b
x (t ) , yb (t )) satis…es:
f (t, b
x (t ) , yb (t )) + V̇ (t, b
x (t )) + Vx (t, b
x (t )) g (t, b
x (t ) , yb (t )) = 0
Similar the Euler equation from a value function in discrete time.
Other way to write the formula, closer to the Bellman equation:
V̇ (t, b
x (t )) = max f (t, x (t ) , y (t )) + g (t, x (t ) , y (t )) Vx (t, x (t ))
x (t ),y (t )
Tight connection between Vx (t, x (t )) and µ (t ) .
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
14 / 28
Deterministic Case
Dynamic Programming
Solution of the HJB Equation I
The HJB equation allows for easy derivations.
For exponential discount problems:
Note that:
V (t, b
x (t )) =
V (t, b
x (t )) = e
ρt
Z ∞
e
t
Z ∞
t
e
ρs
f (b
x (s ) , yb (s )) ds
ρ (s t )
f (b
x (s ) , yb (s )) ds
and the integral on the right hand side does not depend on t.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
15 / 28
Deterministic Case
Dynamic Programming
Solution of the HJB Equation II
Then:
V̇ (t, b
x (t )) =
=
=
Simply…ng notation:
ρt
ρe
ρ
Z ∞
t
Z ∞
e
ρ (s t )
t
e
ρs
f (b
x (s ) , yb (s )) ds
f (b
x (s ) , yb (s )) ds
ρV (t, b
x (t ))
ρV (x ) = maxf (x, y ) + g (x, y ) V 0 (x )
x ,y
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
16 / 28
Deterministic Case
Dynamic Programming
Solution of the HJB Equation III
Characterized by a necessary condition:
fy (x, y ) + gy (x, y ) V 0 (x ) = 0
and an envelope condition:
(ρ
gx (x, y )) V 0 (x )
Then:
V 0 (x ) =
fx (x, y ) = g (x, y ) V 00 (x )
fy (x, y )
= h (x, y )
gy (x, y )
and
V 00 (x ) = hx (x, y ) + hy (x, y )
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
dy
dx
November 9, 2013
17 / 28
Deterministic Case
Dynamic Programming
Solution of the HJB Equation IV
We can plug these two equations in the envelope condition, to get:
(ρ
=
an ODE on
gx (x, y )) h (x, y ) fx (x, y )
dy
hx (x, y ) + hy (x, y )
g (x, y )
dx
dy
dx .
Analytical solutions?
Standard numerical solution methods.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
18 / 28
Deterministic Case
Dynamic Programming
Solution of the HJB Equation V
With our previous example:
ρV (a) = maxu (c ) + (ra + w
a,c
c ) V 0 (a )
Then:
h (a, c ) = u 0 (c )
and:
(r
ρ) u 0 (c ) = u 00 (c )
dc
ȧ = u 00 (c ) ċ
da
Therefore, as before:
u 00 (c )
ċ =
u 0 (c )
Jesús Fernández-Villaverde (PENN)
(r
Optimization in Continuous Time
ρ)
November 9, 2013
19 / 28
Deterministic Case
Dynamic Programming
Comparison with Discrete Time
HJB versus Bellman equation:
c ) V 0 (a )
ρV (a) = maxu (c ) + (ra + w
a,c
V (a) = maxu (c ) + βV ((1 + r ) a + w
a,c
c)
Optimality conditions:
u 00 (c )
ċ
u 0 (c )
u 0 (c 0 )
u 0 (c )
Jesús Fernández-Villaverde (PENN)
=
(r
ρ)
= β (1 + r )
Optimization in Continuous Time
November 9, 2013
20 / 28
Stochastic Case
Stochastic Case
We move now into the stochastic case.
Handling it with calculus of variations or optimal control is hard.
At the same time, there are many problems in macro with uncertainty
which are easy to formulate in continuous time.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
21 / 28
Stochastic Case
Stochastic Problem
Consider the problem:
V (x (0)) = max E
x (t ),y (t )
Z ∞
e
ρt
f (x (t ) , y (t )) dt
0
s.t. dx (t ) = g (x (t ) , y (t )) dt + σ (x (t )) dW (t )
give some initial conditions.
The evolution of the state is a controlled di¤usion.
If f is continuous and bounded, the integral is well de…ned.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
22 / 28
Stochastic Case
Value Function and a Bellman-Type Property
Given a small interval of time ∆t, we get:
maxf (x (0) , y (0)) ∆t +
V (x (0))
x ,y
1
E [V (x (0 + ∆t ))]
1 + ρ∆t
Multiply by (1 + ρ∆t ) and substrate V (x (0)):
ρV (x (0)) ∆t
maxf (x (0) , y (0)) ∆t (1 + ρ∆t ) + E [∆V ]
x ,y
Divide by ∆t
ρV (x (0))
maxf (x (0) , y (0)) (1 + ρ∆t ) +
x ,y
1
E [∆V ]
∆t
Letting ∆t ! 0 and taking the limit:
ρV (x (0)) = maxf (x (0) , y (0)) +
x ,y
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
1
E [dV ]
dt
November 9, 2013
23 / 28
Stochastic Case
Hamilton-Jacobi-Bellman (HJB) Equation I
Given a small interval of time ∆t, we get:
ρV (x ) = maxf (x, y ) +
x ,y
1
E [dV ]
dt
Applying previous results:
1
1
E [dV ] = gV 0 + σ2 V 00
dt
2
we have
1
ρV (x ) = maxf (x, y ) + g (x, y ) V 0 (x ) + σ2 (x ) V 00 (x ) 8x
x ,y
2
Important observation: thanks to Itō’s lemma, the HJB is not
stochastic.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
24 / 28
Stochastic Case
Hamilton-Jacobi-Bellman (HJB) Equation II
Comparison with deterministic case:
ρV (x ) = maxf (x, y ) + g (x, y ) V 0 (x )
x ,y
1
ρV (x ) = maxf (x, y ) + g (x, y ) V 0 (x ) + σ2 (x ) V 00 (x )
x ,y
2
Extra term 21 σ2 (x ) V 00 (x ) corrects for curvature.
Concentrated HJB:
1
ρV (x ) = f (x, y (x )) + g (x, y (x )) V 0 (x ) + σ2 (x ) V 00 (x )
2
is the Feynman–Kac formula that links parabolic partial di¤erential
equations (PDEs) and stochastic processes.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
25 / 28
Stochastic Case
Hamilton-Jacobi-Bellman (HJB) Equation III
Characterized by a necessary condition:
fy (x, y ) + gy (x, y ) V 0 (x ) = 0
and an envelope condition:
gx (x, y )) V 0 (x ) fx (x, y ) =
1
g (x, y ) V 00 (x ) + σ2 (x ) V 000 (x ) + σ (x ) σ0 (x ) V 00 (x )
2
(ρ
Solutions:
1
Theoretical: classical, viscosity, backward SDE, martingale duality.
2
Numerical.
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
26 / 28
Stochastic Case
Real Business Cycle I
Standard business cycle framework without labor choice:
E0
max
fc (t ),k (t )gt∞=0
Z ∞
ρt
u (c (t )) dt
0
s.t. k̇ = e z k α
dz =
e
δk
c
λzdt + σzdW
HJB:
ρV (k, z ) =
u (c ) + (e z k α
Jesús Fernández-Villaverde (PENN)
δk
c ) V1 (k, z )
λzV2 (k, z ) +
Optimization in Continuous Time
1
(σz )2 V22 (k, z )
2
November 9, 2013
27 / 28
Stochastic Case
Real Business Cycle II
Necessary condition
u 0 (c (t ))
V1 (k, z ) = 0
and envelope condition:
ρ
αe z k α
1
δ
V1 (k, z )
z α
= (e k
δk c ) V11 (k, z ) λzV21 (k, z )
1
+ (σz )2 V221 (k, z ) + σ2 zV22 (k, z )
2
Jesús Fernández-Villaverde (PENN)
Optimization in Continuous Time
November 9, 2013
28 / 28