University of Houston
Economics Department
Practical Dynamic Programming
in
Ljungqvist – Sargent (2004)
Presented by
Edson Silveira Sobrinho
for Dynamic Macro class
Motivation
First approach
Second approach
Conclusions

The curse of dimensionality
Chapter 3 presents some computational methods for Dynamic programs of
the form:
V ( x)= max u {r (x , u)+ βV ( ̃x )}

Subject to:
x̃ = g(x ,u)
given x.

The Euler Equation helps in a variety of cases.

One can also start with a Vo and iterate until convergence.

However, it may be impossible to solve analytically. Then we have to adopt
some numerical approximations.
2
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
Numerical implementation
Conclusions
First approach for obtaining numerical approximations:

The first method replaces the original problem with another problem by
forcing the state vector to live on a finite and discrete grid of points.

Then applies discrete-state dynamic programming to this problem.

Downside is that it requires a small number of discrete states.

Guess what? Markov chains!
3
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
Numerical implementation
Conclusions






Set up: an infinitely lived household likes to consume one good, which it can
acquire by using labor income (wst) or accumulated savings (at).
st is endowment of labor at time t. si is endowment of labor in state i.
m-state Markov chain with transition matrix P. Say m = 2, where si = 0 (if
unemployed) or si = 1 (if employed).
The wage w is fixed over time.
The household can choose to hold a single asset in discrete amount at
where at Є A is a grid [ a1 < a2 < … < an].
The asset bears a gross rate of return r fixed over time.
4
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
Numerical implementation
Conclusions

Household maximization problem (Eq. 4.2.1):
∞
E ∑ t=0 βt u (c t )

Subject to :

Assume that:
c t + a t+1 = (1+ r)a t + ws t
β(1+ r)< 1
5
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
Numerical implementation
Conclusions

Bellman equation:
v (a , s )= Max a '∈ A {u [(1+ r )a + ws− a ' ]+ β E v (a ' , s' | z )}

Which is a matrix m x n (2 states of employment x n choices of a).

Let a' and s' indicate next period's asset and state.


v (a,s) is the optimal value of the objective function, starting from asset and
employment state (a, s).
A solution of this problem is a value function v(a, s) that satisfies the
Bellman Equation and an associated policy function a' = g(a, s) mapping this
period’s (a, s) pair into an optimal choice of assets to carry into next period.
6
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
Numerical implementation
Conclusions



For discrete-state space of small size, it is easy to solve the Bellman
equation numerically by manipulating matrices. Here is how to write a
computer program
Define two n x 1 vectors vj, j=1,2 , i.e. one vector for each employment state.
Each row takes the value of v(i) for a given asset i.
Let 1 be the n x 1 vector consisting entirely of ones
7
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
Numerical implementation
Conclusions
Define two n x n matrices Rj whose (i, h) element is (Eq. 4.3.0):
R j (i , h)= u[(r+ 1)a i+ ws j − a h ], i= 1, . . . , n ; h= 1, . . . , n


This is the current period utility conditional on current asset i and next period
asset h, for a given employment state j.
That is, all the possible utilities that arise from combinations of choices of
assets for this period and the next one.
8
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
Numerical implementation
Conclusions

Define an operator T ([v1, v2]) that maps a pair of vectors [v1, v2] into a pair
of vectors [tv1, tv2] (Eq. 4.3.1) :
tv1= Max {R1+ β P 11 1 v 1 ' + β P 12 1 v 2 ' }

LHS is a column vector n x 1

The term in the Max operator is n x n Matrix.


Intuition: each element brings the lifetime utility for a given pair of asset
choices for current and next period. For each row (say current asset), the
agent chooses the column (next period's asset) that allows max lifetime
utility. Current consumption comes from the budget constraint in R1.
Similarly, for unemployed state:
tv 2= Max {R2 + β P 21 1 v 1 ' + β P 22 1 v 2 ' }
9
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
The production dynamics
Numerical implementation
Conclusions

Compact notation (Eq. 4.3.2):
[ ] {[ ]
tv 1
= Max
tv 2

Then the Bellman Equation can be viewed as:
[v1

[ ]}
R1
tv1 '
+ β( P 1)
R2
tv2 '
And can be solved as:
[v1
v 2 ]= T ([v1 v2 ])
v 2 ]m+ 1= T ([v1 v2]m )
10
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
Numerical implementation
Conclusions




Back in Chapter 3 we saw the Howard Improvement Algorithm.
The idea was to Pick a feasible policy, u = h0(x), and compute the value
associated with operating forever with that policy. Next generate a new policy
u = hj+1(x) that solves the two-period problem. Then iterate.
Now let there be a predetermined class M of (N × N) stochastic matrices P,
which are the objects of choice.
I think that we can associate each stochastic matrix P to a pair (a,a'). That's
why we have n x n matrices P. So choosing P is like choosing (a,a').
11
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
Numerical implementation
Conclusions

We can put the Bellman equation in the following form (Eq. 4.4.3):
(I −β P n)v P = c P
n
n
−1
v P = ( I −βP n ) c P
n

n
For B = T – I , find Pn+1 such that (Eq. 4.4.4):
c Pn+ 1+ (β P n+ 1 − I )v Pn= Bv Pn

Intuition: Pick a feasible policy function and use it to iterate. I believe the
advantage of the algorithm is that it is easier to compute (compare Eq 4.3.0
with 4.5.2):
Bv = Max P∈ M {cP+ βPv }− v
12
Motivation
Discretization of state space
First approachDiscrete-state dynamic programming
Application of Howard Improvement algorithm
Second approach
Numerical implementation
Conclusions
Eq. 4.5.1
Rh (a)= u[(r+ 1)a i+ ws j − g (a , sh )],
Eq. 4.5.2



[ ][ (
v1
P J P12 J 1
= I − β 11 1
v2
P 21 J 2 P 22 J 2
−1
h= 1, . . . , n
)] [ ]
r1
r2
Step 1. For an initial feasible policy function g j (k, j) for j = 1, form the rh
matrices using an equation similar to (4.5.1), then use equation (4.5.2) to
evaluate the vectors of values [vj1, vj2] implied by using that policy forever.
Step 2. Use [vj1, vj2] as the terminal value vectors in equation (4.3.2), and
perform one step on the Bellman equation to find a new policy function
gj+1(k, s) for j + 1 = 2. Use this policy function, update j , and repeat step 1.
Step 3. Iterate.
13
Motivation
model
FirstThe
approach
Implications for saving
and
growth
Second
approach
Conclusions
Polynomial approximations
Second approach for obtaining numerical approximations:



Judd (1998) describes a method for iterating on the Bellman equation using
a polynomial to approximate the value function and a numerical optimizer to
perform the optimization at each iteration.
We describe this method in the context of the Bellman equation for a
particular problem.
Model of optimal unemployment insurance as in chapter 19.
14
Motivation
model
FirstThe
approach
Implications for saving
and
growth
Second
approach
Conclusions



Polynomial approximations
A planner wants to provide incentives to an unemployed worker to search for
a new job while also partially insuring the worker against bad luck in the
search process.
The planner seeks to deliver discounted expected utility V to an unemployed
worker at minimum cost while providing proper incentives to search for work.
Hopenhayn and Nicolini show that the minimum cost C(V) satisfies the
Bellman equation (Eq. 4.7.1).
C (V )= minv {c+ β[1− p(a )]C (V )}
u
u
15
Motivation
model
FirstThe
approach
Implications for saving
and
growth
Second
approach
Conclusions





Polynomial approximations
The method uses a polynomial to approximate the i th iterate Ci(V) of C(V).
This polynomial is stored on the computer in terms of n + 1 coefficients.
Then at each iteration, the Bellman equation is to be solved at a small
number m >= n + 1 values of V .
This procedure gives values of the i th iterate of the value function Ci(V) at
those particular V ’s.
Then we interpolate to fill in the continuous function Ci(V).
The minimum problem on the right side of equation (4.7.1) becomes a
numerical minimizer (doable with simple algorithms in Matlab or Gauss).
16
Motivation
model
FirstThe
approach
Implications for saving
and
growth
Second
approach
Conclusions
Polynomial approximations
Algorithm summary:
1. Choose upper and lower bounds for Vu [Vu, Vu] required to assure
participation and a positive search effort, computed in chapter 19.
1. Choose a degree n for the approximator, a Chebyshev polynomial, and a
number m >= n + 1 of nodes.
1. Generate the m zeros of the Chebyshev polynomial on the set [1,−1] , given
by (4.7.6).
1. By a change of scale, transform the zi’s to corresponding points Vlu in
[Vu, Vu].
17
Motivation
model
FirstThe
approach
Implications for saving
and
growth
Second
approach
Conclusions
Polynomial approximations
5. Choose initial values of the n+1 coefficients. Use these coefficients to define
the function Ci(Vu) for iteration number i = 0.
6. Compute the function
̃Ci(V )≡ c+ β [1 − p (a ) ]C i (V u )
5. For each point Vlu , use a numerical minimization program to find Ci+1(Vlu ).
6. Compute new values of the coefficients by using “least squares”. Return to
step 5 and iterate to convergence.
18
Motivation
First approach
Second approach
Conclusions



This chapter has described two of three standard methods for approximating
solutions of dynamic programs numerically: discretizing the state space and
using polynomials to approximate the value function.
The next chapter describes the third method: making the problem have a
quadratic return function and linear transition law.
This will make solving a dynamic program easy by exploiting stochastic
linear difference equations.
19