advertisement

Optimization in Continuous Time Jesús Fernández-Villaverde University of Pennsylvania November 9, 2013 Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 1 / 28 Motivation Three Approaches We are interested in optimization in continuous time, both in deterministic and stochastic environments. Elegant and powerful math (di¤erential equations, stochastic processes...). Three approaches: 1 Calculus of Variations. 2 Optimal Control. 3 Dynamic Programming. We will focus on the last two: 1 Optimal control can do everything economists need from calculus of variations. 2 Dynamic programming is better for the stochastic case. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 2 / 28 Deterministic Case Maximization Problem I Basic setup: V (0, x (0)) = max Z ∞ x (t ),y (t ) 0 f (t, x (t ) , y (t )) dt s.t. ẋ = g (t, x (t ) , y (t )) x (0) = x0 , lim b (t ) x (t ) t !∞ x1 x (t ) 2 X , y (t ) 2 Y x (t ) is a state variable. y (t ) is a control variable. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 3 / 28 Deterministic Case Maximization Problem II Admissible pair: (x (t ) , y (t )) s.t. the previous conditions are satis…ed. Optimal pair: (b x (t ) , yb (t )) that reach V (0, x (0)) < ∞. Then: V (0, x (0)) = Z ∞ 0 f (t, b x (t ) , yb (t )) dt Two di¢ culties: 1 We need to …nd a whole function y (t ) of optimal choices. 2 The constraint is in the form of a di¤erential equation. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 4 / 28 Deterministic Case Optimal Control Optimal Control Pontryagin and co-authors. Principle of Optimality If (b x (t ) , yb (t )) is an optimal pair, then: V (t0 , x (t0 )) = for all t1 t0 . Z t1 t0 f (t, b x (t ) , yb (t )) dt + V (t1 , b x (t1 )) We will assume that there is an optimal path. Proving existence is, however, not a trivial task. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 5 / 28 Deterministic Case Optimal Control Hamiltonian De…ne: H (t, x (t ) , y (t ) , λ (t )) = f (t, x (t ) , y (t )) + λ (t ) g (t, x (t ) , y (t )) where λ (t ) is the co-state multiplier. Necessary conditions: Hy (t, b x (t ) , yb (t ) , λ (t )) = 0 Hx (t, b x (t ) , yb (t ) , λ (t )) ẋ = Hλ (t, b x (t ) , yb (t ) , λ (t )) λ̇ (t ) = plus x (0) = x0 and limt !∞ b (t ) x (t ) Jesús Fernández-Villaverde (PENN) x1 . Optimization in Continuous Time November 9, 2013 6 / 28 Deterministic Case Optimal Control Exponential Discounting Case I More speci…c form: V (x (0)) = max Z ∞ x (t ),y (t ) 0 e ρt f (x (t ) , y (t )) dt s.t. ẋ = g (x (t ) , y (t )) x (0) = x0 , lim b (t ) x (t ) t !∞ x1 x (t ) 2 IntX , y (t ) 2 IntY g (x (t ) , y (t )) being autonomous is not needed but it helps to simplify notation. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 7 / 28 Deterministic Case Optimal Control Exponential Discounting Case II Hamiltonian: H (t, x (t ) , y (t ) , λ (t )) = e = e ρt f (x (t ) , y (t )) + λ (t ) g (x (t ) , y (t )) ρt [f (x (t ) , y (t )) + µ (t ) g (x (t ) , y (t ))] where µ (t ) = e ρt λ (t ) Current-Value Hamiltonian: b (x (t ) , y (t ) , µ (t )) = f (x (t ) , y (t )) + µ (t ) g (x (t ) , y (t )) H Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 8 / 28 Deterministic Case Optimal Control Maximum Principle for Discounted In…nite-Horizon Problems Theorem Under some technical conditions, the optimal pair (b x (t ) , yb (t )) satis…es the necessary conditions: 1 2 3 4 b y (x (t ) , y (t ) , µ (t )) = 0 for 8t 2 R+ . H b x (x (t ) , y (t ) , µ (t )) = ρµ (t ) µ̇ (t ) for 8t 2 R+ . H b µ (x (t ) , y (t ) , µ (t )) = ẋ (t ) for 8t 2 R+ ,x (0) = x0 and H limt !∞ x (t ) x1 . b (x (t ) , y (t ) , µ (t )) = limt !∞ e ρt µ (t ) x̂ (t ) = 0. limt !∞ e ρt H Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 9 / 28 Deterministic Case Optimal Control Su¢ ciency Conditions Previous theorem only delivers necessary conditions. However, we also need su¢ cient conditions. Theorem Mangasarian Su¢ cient Conditions for Discounted In…nite-Horizon Problems. The necessary conditions will be su¢ cient if f and g are continuously di¤erentiable and weakly monotone and H (t, x (t ) , y (t ) , λ (t )) is jointly concave in x (t ) and y (t ) for 8t 2 R+ . We will skip su¢ ciency arguments. They will be relevant later in models of endogenous growth. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 10 / 28 Deterministic Case Optimal Control Example I Consumption-savings problem: V (a) = max a,c Z ∞ e ρt u (c ) dt 0 ȧ = ra + w c Hamiltonian: b (a, c, µ) = u (c ) + µ (ra + w H c) Necessary conditions: b c (a, c, µ) = 0 ) u 0 (c ) H b a (a, c, µ) = ρµ H Jesús Fernández-Villaverde (PENN) µ = 0 ) u 0 (c ) = µ µ̇ µ̇ ) r µ = ρµ µ̇ ) = (r µ Optimization in Continuous Time ρ) November 9, 2013 11 / 28 Deterministic Case Optimal Control Example II Then: u 00 (c ) ċ = µ̇ ) µ̇ u 00 (c ) ċ = = 0 u (c ) µ (r ρ) Assume, for instance: u (c ) = log c and we get: ċ =r ρ c Comparison with bang-bang solutions (linear returns and bounded controls). Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 12 / 28 Deterministic Case Dynamic Programming Dynamic Programming Dynamic programming is a more ‡exible approach (for example, later, to introduce uncertainty). Instead of searching for an optimal path, we will search for decision rules. Cost: we will need to solve for PDEs instead of ODEs. But at the end, we will get the same solution. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 13 / 28 Deterministic Case Dynamic Programming Hamilton-Jacobi-Bellman (HJB) Equation When V (t, x (t )) is di¤erentiable, (b x (t ) , yb (t )) satis…es: f (t, b x (t ) , yb (t )) + V̇ (t, b x (t )) + Vx (t, b x (t )) g (t, b x (t ) , yb (t )) = 0 Similar the Euler equation from a value function in discrete time. Other way to write the formula, closer to the Bellman equation: V̇ (t, b x (t )) = max f (t, x (t ) , y (t )) + g (t, x (t ) , y (t )) Vx (t, x (t )) x (t ),y (t ) Tight connection between Vx (t, x (t )) and µ (t ) . Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 14 / 28 Deterministic Case Dynamic Programming Solution of the HJB Equation I The HJB equation allows for easy derivations. For exponential discount problems: Note that: V (t, b x (t )) = V (t, b x (t )) = e ρt Z ∞ e t Z ∞ t e ρs f (b x (s ) , yb (s )) ds ρ (s t ) f (b x (s ) , yb (s )) ds and the integral on the right hand side does not depend on t. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 15 / 28 Deterministic Case Dynamic Programming Solution of the HJB Equation II Then: V̇ (t, b x (t )) = = = Simply…ng notation: ρt ρe ρ Z ∞ t Z ∞ e ρ (s t ) t e ρs f (b x (s ) , yb (s )) ds f (b x (s ) , yb (s )) ds ρV (t, b x (t )) ρV (x ) = maxf (x, y ) + g (x, y ) V 0 (x ) x ,y Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 16 / 28 Deterministic Case Dynamic Programming Solution of the HJB Equation III Characterized by a necessary condition: fy (x, y ) + gy (x, y ) V 0 (x ) = 0 and an envelope condition: (ρ gx (x, y )) V 0 (x ) Then: V 0 (x ) = fx (x, y ) = g (x, y ) V 00 (x ) fy (x, y ) = h (x, y ) gy (x, y ) and V 00 (x ) = hx (x, y ) + hy (x, y ) Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time dy dx November 9, 2013 17 / 28 Deterministic Case Dynamic Programming Solution of the HJB Equation IV We can plug these two equations in the envelope condition, to get: (ρ = an ODE on gx (x, y )) h (x, y ) fx (x, y ) dy hx (x, y ) + hy (x, y ) g (x, y ) dx dy dx . Analytical solutions? Standard numerical solution methods. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 18 / 28 Deterministic Case Dynamic Programming Solution of the HJB Equation V With our previous example: ρV (a) = maxu (c ) + (ra + w a,c c ) V 0 (a ) Then: h (a, c ) = u 0 (c ) and: (r ρ) u 0 (c ) = u 00 (c ) dc ȧ = u 00 (c ) ċ da Therefore, as before: u 00 (c ) ċ = u 0 (c ) Jesús Fernández-Villaverde (PENN) (r Optimization in Continuous Time ρ) November 9, 2013 19 / 28 Deterministic Case Dynamic Programming Comparison with Discrete Time HJB versus Bellman equation: c ) V 0 (a ) ρV (a) = maxu (c ) + (ra + w a,c V (a) = maxu (c ) + βV ((1 + r ) a + w a,c c) Optimality conditions: u 00 (c ) ċ u 0 (c ) u 0 (c 0 ) u 0 (c ) Jesús Fernández-Villaverde (PENN) = (r ρ) = β (1 + r ) Optimization in Continuous Time November 9, 2013 20 / 28 Stochastic Case Stochastic Case We move now into the stochastic case. Handling it with calculus of variations or optimal control is hard. At the same time, there are many problems in macro with uncertainty which are easy to formulate in continuous time. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 21 / 28 Stochastic Case Stochastic Problem Consider the problem: V (x (0)) = max E x (t ),y (t ) Z ∞ e ρt f (x (t ) , y (t )) dt 0 s.t. dx (t ) = g (x (t ) , y (t )) dt + σ (x (t )) dW (t ) give some initial conditions. The evolution of the state is a controlled di¤usion. If f is continuous and bounded, the integral is well de…ned. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 22 / 28 Stochastic Case Value Function and a Bellman-Type Property Given a small interval of time ∆t, we get: maxf (x (0) , y (0)) ∆t + V (x (0)) x ,y 1 E [V (x (0 + ∆t ))] 1 + ρ∆t Multiply by (1 + ρ∆t ) and substrate V (x (0)): ρV (x (0)) ∆t maxf (x (0) , y (0)) ∆t (1 + ρ∆t ) + E [∆V ] x ,y Divide by ∆t ρV (x (0)) maxf (x (0) , y (0)) (1 + ρ∆t ) + x ,y 1 E [∆V ] ∆t Letting ∆t ! 0 and taking the limit: ρV (x (0)) = maxf (x (0) , y (0)) + x ,y Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time 1 E [dV ] dt November 9, 2013 23 / 28 Stochastic Case Hamilton-Jacobi-Bellman (HJB) Equation I Given a small interval of time ∆t, we get: ρV (x ) = maxf (x, y ) + x ,y 1 E [dV ] dt Applying previous results: 1 1 E [dV ] = gV 0 + σ2 V 00 dt 2 we have 1 ρV (x ) = maxf (x, y ) + g (x, y ) V 0 (x ) + σ2 (x ) V 00 (x ) 8x x ,y 2 Important observation: thanks to Itō’s lemma, the HJB is not stochastic. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 24 / 28 Stochastic Case Hamilton-Jacobi-Bellman (HJB) Equation II Comparison with deterministic case: ρV (x ) = maxf (x, y ) + g (x, y ) V 0 (x ) x ,y 1 ρV (x ) = maxf (x, y ) + g (x, y ) V 0 (x ) + σ2 (x ) V 00 (x ) x ,y 2 Extra term 21 σ2 (x ) V 00 (x ) corrects for curvature. Concentrated HJB: 1 ρV (x ) = f (x, y (x )) + g (x, y (x )) V 0 (x ) + σ2 (x ) V 00 (x ) 2 is the Feynman–Kac formula that links parabolic partial di¤erential equations (PDEs) and stochastic processes. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 25 / 28 Stochastic Case Hamilton-Jacobi-Bellman (HJB) Equation III Characterized by a necessary condition: fy (x, y ) + gy (x, y ) V 0 (x ) = 0 and an envelope condition: gx (x, y )) V 0 (x ) fx (x, y ) = 1 g (x, y ) V 00 (x ) + σ2 (x ) V 000 (x ) + σ (x ) σ0 (x ) V 00 (x ) 2 (ρ Solutions: 1 Theoretical: classical, viscosity, backward SDE, martingale duality. 2 Numerical. Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 26 / 28 Stochastic Case Real Business Cycle I Standard business cycle framework without labor choice: E0 max fc (t ),k (t )gt∞=0 Z ∞ ρt u (c (t )) dt 0 s.t. k̇ = e z k α dz = e δk c λzdt + σzdW HJB: ρV (k, z ) = u (c ) + (e z k α Jesús Fernández-Villaverde (PENN) δk c ) V1 (k, z ) λzV2 (k, z ) + Optimization in Continuous Time 1 (σz )2 V22 (k, z ) 2 November 9, 2013 27 / 28 Stochastic Case Real Business Cycle II Necessary condition u 0 (c (t )) V1 (k, z ) = 0 and envelope condition: ρ αe z k α 1 δ V1 (k, z ) z α = (e k δk c ) V11 (k, z ) λzV21 (k, z ) 1 + (σz )2 V221 (k, z ) + σ2 zV22 (k, z ) 2 Jesús Fernández-Villaverde (PENN) Optimization in Continuous Time November 9, 2013 28 / 28