/
OEWEV
HD28
.M414
/^-\
0
no. 3<^/D
''
ALFRED
P.
WORKING PAPER
SLOAN SCHOOL OF MANAGEMENT
NfUMERICAL ANALYSIS OF A FREE-BOUNDARY SINGULAR
CONTROL PROBLEM IN FINANCIAL ECONOMICS
Ayman Hindy
Huang
Hang Zhu
Chi-fu
Massachusetts Institute of Technology
WP#3615-93-EFA
August 1993
MASSACHUSETTS
INSTITUTE OF TECHNOLOGY
50 MEMORIAL DRIVE
CAMBRIDGE, MASSACHUSETTS 02139
NUMERICAL ANALYSIS OF A FREE-BOUNDARY SINGULAR
CONTROL PROBLEM IN FINANCIAL ECONOMICS
Ayman Hindy
Huang
Hang Zhu
Chi-fu
Massachusetts Institute of Technology
WP#3615-93-EFA
August 1993
M.f.T.
OCT
LIBRARIES
1 5 1993
RECEIVED
NUMERICAL ANALYSIS OF A
FREE-BOUNDARY SINGULAR CONTROL
PROBLEM IN FINANCIAL ECONOMICS
Ayman
Hindy, Chi-fu Huang, and
Hang Zhu
*
August 1993
Abstract
We
analyze a numerical scheme for solving the consumption and investment allocation
problem studied
in a
companion paper by Hindy, Huang, and Zhu (1993). For
lem, Bellman's equation
is
The
ential equation with gradient constraints.
at
which consumption occurs.
ity solution to
this prob-
a differential inequality involving second order partial
We
Bellman's equation.
solution involves finding a free-boundary
prove that the value function
We
differ-
is
the unique viscos-
describe a numerical analysis technique based
on Markov chain approximation schemes. We approximate the original continuous time
problem by a sequence of discrete parameter Markov chains control problems. We prove
that the value functions and the optimal investment policies in the sequence of the ap-
proximating control problems converge to the value function and the optimal investment
policy,
if it
exists, of the original
and abstinence regions
in
problem.
the sequence of
We
show that the optimal consumption
approximating problems converge to those in
also
the original problem.
Graduate School of Business, Stanford University, Stanford, CA 94305. Huang
Management, Massachusetts Institute of Technology, Cambridge, MA 02139.
Zhu is with the Derivatives Research Division at CitiBank. We acknowledge financial support from the
National Science Foundation under grants NSF SES-9022937 and NSF SES-9022939.
"Hindy
is
is
at the
at the Sloan School of
1
1
1
INTRODUCTION
1
Introduction
We analyze
a numerical scheme for solving the problem of optimal life-time consumption
and investment allocation studied
HH&Z
(1993), denoted
henceforth.
companion paper by Hindy, Huang, and Zhu
in a
HH&Z
study a model that represents,
things, habit forming preferences over the service flows
among
other
from irreversible purchases of a
We provide a brief statement of a special
HH&Z contains the complete formulation, the
durable good whose stock decays over time.
case of the control problem in section
2.
motivation of the study, the economic interpretations of the variables, and a detailed
discussion of the numerical solution.
framework that supports the
We
ainalysis in
In this paper,
we only provide the
theoretical
HH&Z.
use dynamic programming to solve the optimal stochastic control for the contin-
uous time problem formulated
in
HH&Z. The
important feature of this problem
is
that
the admissible controls allow for cumulative consumption processes with possible jumps
and singular sample paths. Furthermore, the controls do not appear directly
As a
jective functional.
sequel, takes the
tial differential
result, the associated
form of a
Bellman's equation, formula
differential inequality involving a nonlinear
equation with gradient constraint.
The
in the ob(7) in the
second order par-
differential inequality reveals the
"free-boundary" nature of the solution. In particular, consumption occurs only at some
boundary surface
is
in the
domain
of definition of the value function. This free-boundary
the area in the state space at which both the second order differential equation and
the gradient constraint are equal to zero. Furthermore, the optimal consumption
amount required
boundaxy.
to keep the trajectory of the controlled process
is
the
from crossing the free-
This typically results in optimal consumption, and associated wealth, with
singular sample paths; hence the term "singular control".
In section 3.1,
we
state the differential inequality that the value function solves.
HH&Z contains the details of the derivation which is
is
assumed
to
be twice continuously
and Huang (1993) and
HH&Z
differentiable.
show that
if
heuristic because the value function
Under the same assumption, Hindy
a function solves the differential inequality,
together with boundary and regularity conditions, then such a function
is
indeed the
INTRODUCTION
J
2
value function. Furthermore,
candidate policy
In general,
it
is
HH&Z
provide a verification theorem to check whether a
optimal.
difficult to establish
is
one can either rely on analytical
smoothness of the value function.
To do
so,
results characterizing the regularity of the solution of
Bellman's equation or produce a closed form expression
regularity can be verified. Neither option
is
for the value function
whose
available in our case. In paxticulax, standard
results in the theory of nonlinear partial difFerenticd equations; see, for
example, Friedman
(1964) and Ladyzenskaja, Solonnikov, and Ural'ceva (1968), do not provide evidence for
the smoothness of the solution to our differential inequality with
We
will
show, however, in section 3 that the
diflferentiaJ
so-caJled viscosity sense. Furthermore, that solution
dynamic program.
In section 4,
We
inequality has a solution in the
indeed the value function of our
is
the details of the numerical scheme used for obtaining the
HH&Z. The
numerical scheme
we use
is
based on the Markov chain
approximation method introduced by Kushner (1977) and described
and Dupuis (1992).
We
in details in
Kushner
approximate the original controlled continuous time processes
by a sequence of controlled Markov
Markov chains by the
gradient constraints.
will define the notion of viscosity solutions in section 3.
we provide
solutions reported in
its
chains.
size of the possible
We
jump
peirameterize the sequence of controlled
in
each chain which we denote by
approximating Markov chains are chosen such that, as
/i
| 0, the
mean and mean
change per step, under any control, resemble more and more closely the local
h.
The
square
drift
and
the local variance, respectively, of the controlled continuous time process. This property
is
called "local consistency".
We
also replace the objective functional for the original
problem by a sequence of appropriate
anailogs for the
Markov
chains.
Given the chosen approximating Meirkov chains and objective functional, the approach
is
to approximate the original continuous time
problem by a sequence of discrete
parameter Markov decision problems. For each decision problem
corresponding value function
satisfies
in the sequence, the
an iterative Bellman's equation.
equations are easy to program <ind solve numerically.
Such
iterative
Furthermore, the sequence of
discrete Bellman's equations can be viewed as finite-difference approximations of the
original
Bellman equation
for the
continuous time case.
1
—
lar,
INTRODUCTION
3
Next, we turn our attention to the convergence of the numerical scheme. In particu-
we
focus on convergence of the value functions in the sequence of the approximating
decision problems to the value function of the original continuous time problem.
We
use
analyticaJ techniques, in particular the notion of viscosity solutions, to address this question.
The argument
runs, roughly, as follows.
We
demonstrate that the vaJue function
the unique viscosity solution of Bellman's equation. This
steps.
We
first
show that the value function
is
is
achieved
a viscosity solution.
comparison result between sub- and super-solutions, defined
establish the uniqueness result.
We
in section 3 in
We
is
two
next develop a
in section 3,
and use
it
to
\
then focus on the sequence of Bellman's equations for the approximating Markov
chain problems and consider those as finite-difference approximations to the original
monotone,
We establish that this sequence of finite-difference approximations is
consistent, aad stable. We will define these notions in section 5.1. Using these
properties,
we show
Bellman's equation.
that the sequence of value functions for the approximating
Markov
decision problems converges to a viscosity solution of the original Bellman's equation.
uniqueness result established in section 3 implies that this limit
the original program.
we
We
is
the value function for
present the convergence analysis in section
5.
also prove that the sequence of optimal investment policies in the
In that section,
approximating
decision problems converges to the optimal investment policy in the original problem
latter indeed exists. Furthermore,
and abstinence
in the
we show
The
if
the
that the sequence of regions of consumption
approximating problems converge to the corresponding ones
in the
original problem.
It is
perhaps worthwhile to point out that the analysis
tion to the possibilities of jumps
policies,
in this
and singular sample paths
paper pays special atten-
in the admissible
consumption
and the special structure of Bellman's equation imposed by the presence of the
gradient constraints. In particular, the proofs of theorems
ments to handle the possible jumps
in
1
to 4 require involved argu-
consumption and the gradient constraints. These
arguments extend the existing literature on viscosity solutions
as reported, for example, in
Fleming and Soner (1993).
in
dynamic programming
STATEMENT OF THE CONTROL PROBLEM
2
4
Statement of the Control Problem
2
In this section,
we
state the control problem which
is
the focus of analysis in this paper.
This problem, with constant investment opportunity set and infinite horizon,
case of the general problem studied by
HH&Z.
a special
is
For a statement of the general formulation,
a complete discussion of the motivations for studying this problem, and for the economic
interpretation of the variables,
Hindy and Huang (1993) and HH&Z.
consider a frictionless securities market with two long lived and continuously traded
securities:
at
to
Financial Market
2.1
We
we refer the reader
time
where
r
The
t.
is
a stock and a bond. The stock
The bond
is riskless,
pays no dividends, and
sells for
B{t)
=
sells for
e'"'
at
S{t)
time
t,
the constant riskless interest rate.
S{t)
= S{0)+
is
a geometric Brownian Motion given by:^
[ fiS{s)ds+
Jo
where
B is one-dimensional standard
space
{Q,^,P) and
bond
the market
is
fi
>
r
cire strictly
given by
F=
and
cr
>
f'
aS{s)dB{s)
are constants.
5
a.s.,
(1)
Brownian Motion defined on a complete probability
{^t;t € [0,oo)} where ^t
<
Vi>0
Jo
Note that the prices of both the
positive with probability one.
J^ with respect to which {5(5);
will
risky,
does not pay dividend, and
price process^ for the stock
stock and
is
<
i} is
is
The
available information in
the smallest sub-sigma-field of
measurable. All processes to be discussed
be adapted to F.^
^
'A process y is a mapping Y:Q x [0,oo) — 3J that is measurable with respect to
<Si B{[0,oo)), the
and the Borel sigma-field of [0, oo). For each u £Q, Y{u, .): [0, oo) —
product sigma-field generated by
^
a sample path and for each t € [0, oo), Y{., <): Q — 3? is a random variable.
^The notation a.s. denotes statements which are true with probability one. We will use weak relations.
Increasing means nondecreasing and positive means nonnegative.
^We assume that Tt is complete in the sense that it contains all the probability zero sets of J-. The
process Y is said to be adapted to F if for each t 6 [0,oo), Y{t) is ^,-measurable.
3t is
2
STATEMENT OF THE CONTROL PROBLEM
Admissible Controls
2.2
The
5
control variables are a consumption plan and an investment strategy. -A- consumption
C
plan
is
with C{u},
a process whose seimple paths are positive, increasing, and right- continuous,
t)
denoting the cumulative consumption from time
investment strategy
is
C
is
t
before consumption and trading.
said to be financed by the investment strategy
= W{0)+
W{t)
/
in state
An
u;.
one-dimensional process A, where A{t) denotes the proportion of
wealth invested in the stock at time
plan
to time
I {rW{s) + A{s){ti-r)W{s)) ds-C{t-)
A
+
^0
if,
A
consumption
a.s.,
f aA{s)W{s)dB{3),
V<
>
0,
Jo
(2)
where W{t)
is
wecdth at time
t
left-continuous sample paths and that H^(f'")
A
admissible
of
t.
C
consumption plan
if
(2)
is
= W{i) —
and the investment strategy
well-defined^ and, for
Denote by C and
Note that the wealth process has
before consumption.
all t,
E[C{t)]
<
AC(<).
A
that finances
Q{t), with Q{t)
it
axe said to be
some polynomial
A the spjice of admissible consumption plans and trading strategies,
respectively.
2.3
The
Objective Functional
objective functional
is
over the stock of durable,
defined as an integraJ, over time, of a utility function defined
z,
and the standard of
living, y,
which are given, respectively,
by
z{t)
y{t)
=
z{0-)e-'^'
=
y(0-)e-^'
f\-'^^'-'UC{s)
+
13
+
A /' e-^('-')ffC(5)
a.s.,
(3)
a.s.,
(4)
Jq-
no-
where 2(0")
>
and the integrjJs
*For this we
and y(0") >
in (3)
mean both
and
are constants,
(4) are defined
/?
cind A,
with
path by path
>
X, are
weighting factors,
in the Lebesgue-Stieltjes sense.
the Lebesgue integral and the ltd integral are well-defined.
When
A(t)
feedback control depending on the state variables and C{t) depends on the history of {W,S), we
there exists a solution
is
a
mean
W to the stochastic differential equation (2) for every pair of feedback controls.
3
ANALYSIS OF THE VALUE FUNCTION
Wealth
managed
is
dynaimicaJly to solve the following program:
C
s.t.
is
financed by
and W{t)
The
Wq
is
initial
function u
•
u
is
is
wealth,
«(y, z)
a fixed
The second
A
e
with W{0)
- AC{t) >
a discount
is
satisfies
< K{y +
V<
fcictor
>
=
Wq,
(5)
0,
and
u(-,-)
is
the felicity function.
the linear growth condition:
Vj/
z)-,
>
0,
z
>
for
0.
some constant
/T
>
0.
(6)
•) is
a strictly increasing, concave, and differentiable function.
z, u(-,z) is
a strictly increasing, concave, and differentiable function.
• For a fixed y, u{y,
• For
>
<5
A
cissumed to satisfy the following conditions:
nonnegative and
<
-
••
E[l^e-''u{y{t),z{t))dt]
supc€C
where
6
constraint in (5)
is
the requirement that wealth after consumption at any
time must be positive. Condition
(6)
guarantees that the
supremum
in (5)
is finite.
Since the interest rate and the parameters of the return on the stock are constant,
the
supremum
call
the
supremum
J(H^, y,z).
The
in (5)
We
is
a function of the state variables
in (5) the value function of the
observe that, in general,
little is
W
,
y,
and
z only.
As
usual,
dynamic program and denote
known about
it
we
by
the differentiability of J.
choice of a technique to analyze the dynamic program in (5)
is
influenced by prior
knowledge of the smoothness of the value function.
3
3.1
Analysis of
Bellman's Equation
In this section,
(5).
The Value Function
Although
we
report Bellman's equation associated with the
little is
known about the
dynamic program
regularity of the value function J,
Bellman's equation heuristicaJly by assuming that J
is
we can
in
derive
twice continuously differentiable.
3
ANALYSIS OF THE VALUE FUNCTION
This assumption allows us to utilize
Ito's
7
lemma, together with the dynamic programming
principle, to derive Bellmaui's equation. Bellman's equation turns out to
be a
difFerentiaJ
inequality involving a nonlinear second order partial differential equation with gradient
constraints. In section 3,
of Bellman's equation
we prove rigorously that the value function
we
derive heuristically here.
solution in the viscosity sense
The
— a notion that we define
Assuming that the value function
tions for optimality are given
is
smooth,
by the following
max|max^[u(y,2) + JwW[r +
A{fi
M+
-
indeed a solution
value function, however,
HH&Z
differential inequjJity in (7)
a
show that the necessary condi-
differential inequality:
+ ^JwwW'^A'^a - Jw} = 0,
/3zJ^
-
can also be viewed,
-
XyJy
SJ]
,
(7)
plus the condition that nontrivial consumption occurs only at states where
Jw- The
is
in section 3.
r)]
AJv
is
in the region of
/3 J,
-}-
XJy
=
no consumption,
as a nonlinear second-order partial differential equation with the gradient constraint that
13 J,
+
XJy
-Jw<0.
In addition to the above necessary conditions,
it
is
clear that
J must
satisfy the
following boundary condition:
^(y,z)
The
tion 2
is
= \unJ{W,y,z) =
j^ e-"u{ye-'\ze-^')ds
differential inequzJity (7) suggests that the
(8)
.
dynamic program formulated
in sec-
a free-boundciry singulcir control problem. Consumption should occur only
when
the state variables {W^y^z) reach a certain boundary surface characterized by the condition that
Jw —
l3Jz
+ XJy.
Furthermore, the optimal consumption
is
the amount required
to keep the trajectory of the controlled processes from crossing this critical boundary.
Since the returns on the stock follow a Brownian Motion, such a consumption policy,
and the associated wealth process, may have singular sample paths
"singular control".
Computing the location
of the free-boundary
—
is
hence the term
the focus of this
paper.
Hindy and Huang (1993) and
HH&Z
show that
if
there
is
a smooth solution to (7)
that satisfies (8) and additional mild regularity conditions, then that solution
is
the
ANALYSIS OF THE VALUE FUNCTION
3
value function for the program
8
These papers also provide verification theorems
(5).
that specify conditions for a candidate feasible consumption and investment policy to
attain the
supremum
in (5).
We
refer the reader to
HH&Z
statements of
(§3) for the
the necessary and sufficient conditions of optimality, the verification theorems, and the
proofs.
Viscosity Solutions of
tion
3.2
The
verification
function
is
theorem presented
in
know whether
useful
is
is
when
the ca^e
Hindy and Huang
it is
known
when
there
(1993).
that the value
a closed form
is
In general,
it is
difficult
the value function has enough regularity for the verification theorem to
To handle the
apply.
HH&Z
twice continuously differentiable. Such
solution of Bellman's equation as in
to
The Dynamic Programming Equa-
the value function,
which there
situations in
we
utilize the
is
no information about the regularity of
notion of weak, or viscosity, solutions.
Viscosity solutions were introduced by Crandall and Lions (1982) for first-order equations,
and by Lions (1983)
theory,
We
we
for
second-order equations.
refer the reader to the User's
We
Guide by Crandall,
Ishii,
and Lions (1991).
are interested in viscosity solutions in the context of proving convergence of the
Markov chain numerical scheme
we
For a general overview of the
to
be presented
To
in the following sections.
that end,
characterize the value function as the unique viscosity solution of Bellman's equation.
prove uniqueness of the solution to Bellman's equation by utilizing a comparison
principle between viscosity subsolutions and viscosity supersolutions.
result has additional significance to our subsequent analysis because
in establishing
Let
Q =
This comparison
it is
a key element
convergence of the numerical scheme.
(0,
oo) X
[0,
oo) x
[0,
oo) and consider a second-order partial differential
equation of the form:
ma.x[-6v
where 7
+ F{d^v, Dv) +
= — l,A,/3), D =
(
u{y, z),
7
•
D^ v) =
{dw,dy,dz),
$
is
0,
on
Q
with u(0, y, z)
some function
of y
and
z,
=
$(y, 2)
,
(9)
and ^ denotes
ANALYSIS OF THE VALUE FUNCTION
3
transpose.
F{dwv, dwv,
Next,
we
F
The function
given by:
is
= max{- VK^A^a^5^u + W[r +
dyV, d^v)
-
A{fi
r)]dwv]
state the notion of viscosity solutions as in Crandall, Ishii,
-
\ydyV -^zd^v.
and Lions (1991).
An upper semicontinuous function v
is a (viscosity) subsolution of (9) on the domain Q if v{0,y,z) < $(j/,z) and
4> G C^'^'^{Q) and any local maximum point (VKo,yo, 2o) ^ Q of v ~
Definition
1
(Viscosity Solution)
(i)
:
Q
—^
R
for every
(j),
max|-(5v(iyo,
I/o,
^o)
+
F{d1y<i>{Wo, yo, zo), 4>w{Woyyo-, 2o),
+u(yo,
zo),
7 D^<i>{Wo,
•
yo, zo)}
<?!'y(VKo, j/o,
>
^o), <i>z{Wo, yo, ^o))
.
(10)
(ii)
A
the
domain
lower semicontinuous function v
Q
if
v{0,y, z)
point (Wo,yo,zo) e
max|-(^u(V^o,
Q
yo, zo)
>
:
Q
—*
R
^{y,z) and for every
is
(f>
a (viscosity) supersolution of (9) on
6
C^'^'^{Q)
and any
local
minim.um
ofv-4>,
+
F{d^(^{Wo, yo,
zo), <f>w{WQ, yo, zq), <f>y{Wo, yo, zo), <f>z{Wo, yo, Zo))
+u(yo,Zo),7--0''<?i(Wo,yo,Zo)}
<0.
(11)
A
continuous function v
and
:
Q
—^
R
is
a viscosity solution of (9) if it
is
both a subsolution
a supersolution of (9).
The main
result of this section
Bellman's equation
(9)
is
that J(W,y,z)
is
the unique viscosity solution of
with boundary condition u(0,y,z)
=
^(y,2) as
in (8). First,
we
state without proof the following proposition
Proposition
programming
1
satisfies the
dynamic
E[e-'^J{WiT),y{T),z{T))+ re-''u{y{t),z{t))dt],
(12)
(Optimality Principle) The value function J
principle
JiW,y,z)=
sup
for any Ft-adapted stopping time r
the stopped processes
ofW,y, and
>
0,
z at r.
where W{t), y{T), and z{t)
are, respectively,
ANALYSIS OF THE VALUE FUNCTION
3
The dynamic programming
example,
for
in
10
principle in proposition
Fleming and Soner (1993,
§5.2).
from the standard proofs because
controls allow for
jumps and
singular sample paths.
Zhu
may
utilize the principle of
The proof
of the possible
Theorem
1
concave
(0,
in all of its
J(H^,0,0).
We
to prove that the value function
of this result
J
is
is
is
a
not standard because
continuous on the domain
WG
all
(0, oo), of
continuous in the region
need to show that J
the doamin Q. Since u
of its elements,
J(Vl^, 0,0).
=
is
increasing,
is
and hence limy_o,z-.o J{^i
Yimy^Q,,^oJ{W,y,z)
>
is
We then
(1975, §14, page 82).
0, 0),
fact,
then follows that J
limy—o,z-.o «^(^) J/,^)
V-,
2)
>
J(W^,0,0).
From the dynamic programming
we have
J{W, 0, 0) >
where (M^«,yj,2j) are the
controls.
we show how one
a viscosity solution of Bellman's equation (9).
It
increasing in
prove that, in
Suppose that
principle,
is
Next,
II).
and using standard reasoning, we can show that J
Holmes
continuous on the boundary (IV,
the value function J
is
arguments.
oo) X (0, oo) X (0, oo); see
however,
proof of the dynamic programming
establish that the vcJue function
Utilizing the concavity of u,
Q.
is
first
is,
singular sample paths in the admissible controls.
The value function J
We
Proof.
jumps and the
1
our formulation the admissible
in
A
what can be found,
of proposition
(1991, Chapter
dynamic programming
viscosity solution of Bellman's equation.
similar to
The proof
slightly different
principle in such a case can be found in
1 is
i;
[e-''J{Wu yu
~zt)
+
e-''u{y„
z,) ds]
f^
trajectories of a continuous state process for
Taking the limit as
<
J,
0,
and
(13)
,
some admissible
utilizing the continuity of the trajectory of the
controlled process, the right-hand side of (13) converges to a value strictly greater than
J{W,0,0)- a
We
(i)
J
is
clear contradiction.
next show that
(^
is
a viscosity solution of Bellman's equation.
a viscosity subsolution.
subsolution, then the
Let
J
be a
We argue
by contradiction and show that
dynamic programming principle stated
C^'*'^((5) function
such that
J—
<l>
has a local
if
in proposition
maximum
at
J
fails
1 is
{Wq,
to be a
violated.
yo, zq)
6 Q.
.
.
U
ANALYSIS OF THE VALUE FUNCTION
3
That
is,
3 a neighborhood A/q of {Wo,yo,zo) such that
- HWo.yo^zo) > J{W,y,z) -
J{Wo,yo,zo)
Without
that
<j>
loss of generality,
we assume
+ '^{yi^) < at
= <f){Wo,yo,Zo),
By
F{<f>ww,4>w,4>y,4>z)
{Wo,yoiZo).
that J{Wo,yo,ZQ)
there exists 7
neighborhood A/ of (Wo,
6<f>{W, y, z)
(14)
= <^(^o? J'o, ^o)- Now suppose
— <f>w < and —SJ +
\<f>y + ^(f>z
continuity,
>
and using the assumption
such that, for {W,y,2) in some
J/0,20),
X<f>y{W,y,z)
-
Afo-
that J(VFo,yo,2o)
In other words, both
violates inequality (10).
y{W,y,z) e
4>{W,y,z),
+ l3<f>,{W,y,z)-4>w{W,y,z)<0
and
(15)
+ F{dU{^, y. ^), 4>w{W, y, z), <f>,{W, y, z), M^, y, z)) +
u{y, z)
< -7
(16)
By
we
the existence result in Zhu (1991, Theorems IV. 2. 2 and IV. 2. 3),
optimal policy {A',C') at (Wo,yo,
does not prescribe an
initial
jump
We now
^o)-^
consider the
establish that the optimal policy
at {Wo,yo,Zo). If not,
then
C
prescribes
C
some jump
Using the dynamic programming principle, we can conclude that
T]o.
<
-^(Vl^o-'?,yo
the property of the local
maximum in
^(Vl^o,yo,2o)
By
+
A7?,Zo
+
VO
/?r/),
<
r/
<
770
(14), the previous inequality also holds for ^,
and hence
<
where
by
T]
<
and
t;
<
7?o is
letting
77
J.
C)
-
r/,
yo
+
At?, zq
sufficiently small so that
0,
<
^From the
(i>{Wo
-
(Wo —
t?,
4>{^o, Vo, Zo)
yo
+
A77, zo
+
0t])
€
A/q.
Dividing
we obtain
X4>y{Wo, yo, Zo)
+ ^4>ziWo, yo, zo) -
Zhu (1991, Chapter
analysis in
/9r?)
+
<i>w{Wo, yo, zq)
IV), the optimal policy {A' ,C')
is
,
known
to exist. However,
a space of processes larger than the space of admissible controls {A, C). The
crucial feature of the analysis is that the expected life time utility associated with {A',C'), starting
from any point [W, y, z), is equal to the viJue function J evaluated at that point. For more details, we
{A'
,
may
be realized
refer the reader to
Zhu
in
(1991).
.
3
ANALYSIS OF THE VALUE FUNCTION
which
We
a contradiction to (15).
is
have now established that
{W*,y*,Zf)
to
12
is
(15) holds, then the optimally controlled process
if
continuous in a neighborhood of (WoiJ/Oi^o).
We
be a subsolution, then the dynamic programming principle
We
define the optional time f
by the established
policy.
Then,
it
fact that
=
{W^,y',z^)
follows from (15)
>
inf{i
and
:
(VF,',^^*, 2j')
next show that
is
K.
Now
take r to be the
^ Ao}.
Clearly, f
>
0, a.s.,
continuous up to r under the optimal
is a.s.
K
ini{t
minimum
fails
violated.
4>z)
+
u](W:, yl z;)dt]
+E[j\-'\\<i>, + ^<t>,-<i>w]{W:,ylz;)dC^]
=
J
(16) that
E ll^ e-*'[-<5<^ + F{dU, 4>w,
Define the optional time f
if
>
of f
:
Jq
and
<
> K},
\AlW^<7\^ds
Applying
f.
(17)
-7E[f].
for
some constant
Ito's rule to
the process
e-*V(M^;,yr,2;),wehave
<i>{^o,yo,zo)
=
E[e-''<f>iw:,y;,z;)]
-E\I^ e-''[-S4> + FidU. 4>w^
-E
[J^^
Note that the expectation of the
integral
is
the local
e-^m, + 134>, -
maximum
combine the
is
<f>.)]iW;, y;, z:)dt]
boimded
property in (14), we can replace
A'W'a dBt
till
(^
vanishes because that
the stopping time r. Again by
by J
in first line of (18)
result with (17) to conclude that
J{Wo,yo,zo)
> E[e-'W{W:,y;,z;)]
-E[j\-''[\<i>y
+
P4>,
> E[J\-''n{ylz:)dt +
(18)
cf>w]{w;,y:, z:)dc:]
stocheistic integral Jq
a martingale as the integrand
<f>y,
- 4>w]{w:,y;,z;)dc:]
e-'^JiW:,y;,z:)]
+
T^H
,
and then
ANALYSIS OF THE VALUE FUNCTION
3
which clearly violates the optimality
ming
of
13
(A',C')
dynamic prograxn-
ajid the principle of
(12).
J
(a)
smooth
4)
is
a viscosity supersolution.
such that J(Wo,yo,2o)
-
By
we must show,
definition,
< J{W,y,z) -
<?^(W^o, J/o,2o)
every C^'^'^-
for
4>{W,y,z) for {W,y,z) G
A/q, that the following holds
<
[-8J + F{4>wwAwAyAz)^-u\{WQ,yQ,zo)
A<?^„(H^o,yo,2o)
First,
we choose
interval [0,t),
W°,
y°,
is
=
zero consumption C,
where r
=
inf{<
<
/3<^z(V^o,yo,2o)-<^w(V^o,yo,2o)
>
and some investment policy
/q |i4,W,a|^<i3
:
> K}
for
the principle of dynamic progrctmming,
(20)
A on
p
and
<
t,
Let
0.
A
on
where p
an optional time,
Replacing
J by
we then apply
<^,
stochastic integral
is
eIJ'
inequality (19).
e-^'[-6<^
To derive
J{Wo,
Since
7;
J —
>
<i>
is
Ito's rule,
.
together with the observation that the
a maxtingaJe, to obtain from the above inequality that
Dividing by p and letting p
yo, ^o)
has a local
+F+
<0
u]{W^,ylz'',)ds\
I 0, using the continuity of
(20), the
<
the integrand,
r
.
we
J,
0,
r/,
yo
+
Ary,
zq
+
^77),
VO <
r/
<
first
Wq.
at (Wo,yo, 20)1 this inequality also holds for
sufficiently small such that
we obtain the second
derive the
dynamic programming principle implies that
> J{Wo -
minimum
VO < p
{Wq —
T],yo
+
\t],zq
+
0t])
€ Ao-
subtracting the right-hand side from the left-hand side and dividing by
77
>
ii'
=
all
J{Wo,yo,zo)>E[e-"'J{W^,y''^,z°)+re-"u{ylz',)ds
if
the stochastic
some constant
follows that, for
it
(19)
0.
and 2° be the state processes corresponding to the controls C,
By
[0,r).
+
and
inequality (20). This completes the proof.
t]
I
<i>
as well
Therefore,
and sending
3
ANALYSIS OF THE VALUE FUNCTION
Comparison of Subsolutions and Supersolutions
3.3
We
14
conclude this section by providing a theorem that compares the subsolutions of Bell-
maji's equation to its supersolutions.
standard results
example,
as, for
The
proof of this result
Fleming and Soner (1993,
in
The comparison
result
viscosity solution of the
result
is
used
later
is
§5.8).
We utilize additional
Jw — ^Jy — ^Jz >
arguments to handle the gradient constraint that
tion.
not a simple extension of
is
in
Bellman's equa-
then used to argue that the value function
dynamic programming equation.
is
the unique
In addition, the comparison
convergence of the Markov chain approximations to be
in the proof of
presented in a subsequent section.
Theorem
2
(Comparison Principle)
a supersolution of (9) on the
domain
Q
Let
V
and
V
be, respectively,
a suhsolution
in the sense of Definition 1, then
V
<V
and
on the
domain Q.
= {W,y,z) and x = {W,y,z). We rewrite
Bellman's equation as ma.x{—6v + F{^^^rV,Dxv) + u{y,z),j Djv} = 0, where 7 =
— 1, A,/3) and Dj is the transpose of {dw,dy,dz). We construct a function U{W,y,z) =
Proof.
Let us denote, for simplicity, x
•
(
/o°°
e~*'u(j/e~^', 2~^') dt
— mo
for
some constant mo >
-SU + F{d^U, DM) + «(l/, 2) >
and
0.
One can
\Uy
+ l3U,-Uw>0,
where the second inequality follows from the assumption that u
both y
easily verify that
is
strictly increasing in
eind z.
Next, we will prove the comparison between
which the proof of the theorem
is
V and
V^ = 6V + {1 — 0)U,
for all 9,
from
immediate. For this purpose, we choose the auxiliary
function
rl;{x,
where a >
and
x)
<
c
=
<
||x 1.
x\'
+
Assume
e{\x\'
+ [x^
V(x, x) e
Q
x
Q
,
to the contrary that:
V\x') > V(i'),
for
some x' G Q.
(21)
3
ANALYSIS OF THE VALUE FUNCTION
By
the linear growth of
attain
of
its
a and
(1991,
maximum at
e.
We
V
and V,
Appendix A: proof
— V{x) —
easy to check that V*(x)
Q^Q- Note
a point (io,io) €
xp{x,x) will
that (lo, xq) depends on the values
dependence in the notation. One can prove, as
will suppress this
+ \xo\) -*
We will show
it is
15
of comparison principle), that a|io
—
io|^
—>
as
a
in
Zhu
| oo,
and
as c i 0.
e(|xo|
From the
that supposition (21) produces the following contradiction.
we
definition of (xo,Xo),
have, for small
e,
=
V\xo)-V{xo)-xl;{xo,Xo)
V\x) - V{x) -
max
4>{x,x)
(z,x)€QxQ
> V\x')-V{x')-2e\x'\^>0.
However, we
will
show next that the
respectively, implies that V*(io)
—
V'(xo)
—
V
V* and
fact that
are sub- and supersolutions,
as e
V'(io,Xo) i
Jind
J,
a
which
t oo,
is
a
clear contradiction.
Applying Definition
1
to the subsolution
V^ and the super-solution V, we conclude
that
rmix{-SV^ + F{d^rp,Dr7P) + u{-),^- Djrp}
+ F(-5^V,-^xV') +
max{-<5V'
Now,
since Dxxk
=
2ex
+
a\x
—
x\
«(•),
=
and D^^
-7- -010}
2ex
—
a\x
—
>
0,
at xq
<
0,,
at xo
x|, it follows
(22)
(23)
.
from inequality
(23) that
-7
Moreover, we claim that
holds. Subtracting
it
contradiction to the
Djxl^
= -7
in (22), 7- [2cxo
from
fcict
•
(24),
•
[2exo
-
+ q(x — xq)] <
we conclude that 2e7
that e(|xo|
+
-
a(xo
|xo|) i
•
xq)]
0. If
(xq
<
0.
(24)
not, the opposite inequality
+ xq) >
as e | 0. Therefore,
0,
Va >
,
which
we must have
and (23) that
-6V\xo) + F(5^V, OrV') +
u(l/o, zo)
>
0,
at xq
-6V{x) + F{-dl,ik,-Dsi^) +
u{yo,zo)
<
0,
at xq
and
is
a
in (22)
,
3
ANALYSIS OF THE VALUE FUNCTION
16
Subtracting one from the other yields the inequality
6[V\xo)
-
V{xo)]
< F(5^V, D^rP) - F(-a^V, -D,^) +
Meanwhile, we calculate (Note:
F{d^w^,
=
D.'iP)
r
=
max{^V^o'^'<7'
-
yo)
+ ^oAti[a{Wo - Wo) +
+
where the maximizers can be
explicitly
j/o)
+
computed
as
T.
_
,
ccWo
< ^cc{Wo -a{X\yo
and the proof
Corollary
1
zq)\
c X
and a
°"
-q(Wo - Wo) + 2tWo
\Ji
aWo
^2
we can
verify that
zoP)
+
Wo)'
- 2e'-^{W^ + W^)
[u(yo, zo)
-
"(yo, ^o)]
This completes the demonstration of a contra-
| 00.
is
Zo)\
- 2ef^iWo -
- yoT + /?ko -
of the theorem.
The value function J
Wo)'
-
^zo{zo
Substituting into (25) and rearranging the terms,
diction
2eWo]}
,
-
- Wo) + ItWo
^°-~<72
which goes to zero as
-
^zq{zq
I
-a[Ayo(yo
S{V\xo) - V{xo)\
(25)
"(yo, ^0).
= max{-^V^o'^'^' - WoAfi[-a{Wo - Wo) + 2eWo]}
A
F{~dlxP, -D,rP)
ol{Wo
-
for simplicity)
-Q:['^yo(yo
\i
u{yo, 20)
I
the
unique viscosity solution of Bellman's equation
(9).
Proof.
V
Suppose that there
is
another viscosity solution V.
and J are sub- and supersolutions,
because
J and V
respectively.
On
We
V<
then have
the other hand,
are also sub- and supersolutions, respectively. Hence,
J because
we have J <
J = V.
I
V
,
4
NUMERICAL APPROACH
17
Numerical Approach
4
In this section
utility
we
describe the numerical approach utilized to solve the infinite- horizon
We
maximization problem.
replace the continuous time processes {W,y,z) by a
sequence of approximating discrete parameter Markov chains. The important feature of
the approximating sequence
mean square change
that, as the step size converges to zero, the
is
mean and
per step, under any control, converge to the local drift and the
local variance, respectively, of the controlled continuous
called "local consistency".
The
original optimization
a sequence of discrete parameter control problems.
each Markov chain control problem
discrete Bellman's equation.
satisfies
time process. This property
problem
We
is
is
then approximated by
show that the value function of
an iterative, and hence easily programmable,
The Bellman's equation
of the discrete control
problem
is
a natural finite-difference approximation to the continuous time Bellman's "nonlinear"
second order partial differential equation with gradient constraint.
4.1
A
Constrained Problem
For numerical implementation,
variables to a finite region
we need
Qm =
to restrict the
domain
[0,Mw] x [0,My] x
of definition of the state
[0,71/^].
For this purpose,
introduce the reflection processes Lt^Rt and Dt at the boundaries of the region.
we
We
hence consider the following modified state variables:
where
Lt, Rt
{W,z,y)
We
dyt
=
—\ytdt
dzt
=
-^Zidt
dWt
=
Wt{r
+
XdCt
—
dLt
,
+ ^dCt - dRt
+ Atifi - r))dt ,
and
dCt
+ WtAtadBt -
dDt
and Dt are nondecreasing and increase only when the gtate processes
hit the cutoff
boundaries
W = Mw, ^ = M^, and y = My,
recall that the stock price process follows
the riskless interest rate
is
constant.
We
will,
resp-^ tively.
a geometric Brownian
I
^tion
and that
hence, apply the Markov.^ hain approach
NUMERICAL APPROACH
4
18
to solve numerically, on the restricted domain, the following infinite horizon program:
/•oo
=
j'^{W,y,z)
.
E
sup
A,CGAC
e-''u{yt,zt)dt
(26)
Jo
subject to the dynamics of z and y and the dynamic budget constraint, where {W, y, z) 6
Qm
is
the initial state.
The Bellman's equation
max{-<5J^ + max{£^ J^} +
Mw]
X
[0,
the operator
£^
on
[0,
addition,
we
My] x
is
[0,
u{y, 2),
given by
Ajf +
£^ = ^W^A^a^-^ + W[r +
—
boundary conditions was suggested
- J^] =
(27)
0,
A{fji
-r)]^- \y§^ - ^zj-^.
In
in
W
=
0,
= Mw,
—
and
y
= My,
=
and
0,
z
=
Kushner and Dupuis (1992,
M^. This type of
§5.7)
and employed
numerical analysis of the one-dimensional financial model studied by Fitzpatrick
and Fleming (1992).
These boundary constraints
reflect the
the value function on the original unbounded domain.
in the theoretical analysis of
It is
asymptotic behavior of
worthwhile to remark that
convergence of the numerical scheme, the value functions
Qm
increases to Q,
at the boundaries.
The boundary
converge pointwise to J as the sequence of restricted domains
regardless of the specification of the behavior of
conditions
we
J^
chose, however, affect the quality of the actual numerical solution.
The Mcirkov Chain Scheme
4.2
We
(iJ^
specify the boundary constraints
respectively at the cutoff boundaries
J^
dynamic program takes the form:
Mz], together with the appropriate boundary conditions, where
__.0,
in the
for this
introdu-e a sequence of grids
Gh =
{{k.
):W = kxh,y = ixh,z=j xh;0<k< Nw,0<i <
= My/h, N^ = M^/h,
and
iV„,0
is
integers.
Each grid point {k,i,j) £ Gh corresponds to a state {W,y,z) with
step size, and where A^y
N,}
Nw = Mw/h
where h
i.ie
<j <
,
are
W=kx
h,
NUMERICAL APPROACH
4
y
=
i
X
h,
=
and z
For a fixed grid,
j x h. For simplicity,
we introduce
ACn =
A
where
an
is
artificial
We
19
{{A,
taJce
the saime step size in
all
three directions.
the spa<;e of discrete strategies
Cy,A =
the step size of control,
bound
we
lxA,AC = OoTh;0<l<NA}
A'^^
= A/ A
to be relaxed in the limit
eis
is
A
total
number
of control steps with
A
and A^^ x
J,
A
f oo.
approximate the continuous time process {W,z,y) by a sequence of discrete pa-
rameter Markov chains {(H^,J,y^, z^);n
=
1,2, •••}
with h denoting the granularity of
the grid and n denoting the steps of the Markov chain. For every h, the chain has the
property that, at each step n, there
is
a choice between investment and consumption. At
any time, a chain can either make an instantaneous jump,
(AC =
walk",
is
of the
domain. Fix a chain and
1.
The
reflected to the interior in a
Markov chain
its
investment policy
I
— (AC =
move from
+
{k
A=
a "random
h.
its
dynamics
The
in the interior
transition probabilities
are specified as follows:
possibly
neighboring states:
consistent with
corresponding grid size
case of no consumption
The chain can
manner
h), or follow
At the cutoff boundaries the
the neighboring states on the grid.
0), to
chain
for this
(AC =
y.
l,i,j), {k
0):
the current state {k,i,j) only to one of the four
—
l,t,j), {k,i
—
l,j),
A, the transition probabilities
and {k,i,j
—
1).
For any
in this case are defined as
P/1MVV2 + kh^r + fiA)
Q'{k,i,j)
fc^/tMV/2 +
fc/t^(M)
Q'{k,i,j)
P^[{Ki,Jlik,i-l,j)]
Pk[{k,iJ),{k,iJ)]
Q'{k,i,jy
= l-Pi^[ik,iJ),{k+hiJ)]-P;^[{k,iJ),ik-hhJ)]
-P;^[{k,i,j),ik,i-lJ)]-Pl^[{k,ijy{k,iJ-l)]
NUMERICAL APPROACH
4
where the normalization
Q''{k,iJ)
The
= ^max
20
fcLctor (5''(^,t,i) is
{Ph''{l X
A)V2 +
recipe for these transition probabiHties
given by
kh^[r
+
I
x A{(x
+
r)]
{\i +. ^j)h''}
.
a slightly modified version of those sug-
is
Note that the normalization factor
gested by Kushner (1977).
+
independent of the
is
investment policy A. This specification implies that, at each step, the Markov-chain has
a strictly positive probability of remaining at the
same node.
Furthermore, we define the incremental difference Ay^
and AWl^
z^J,
= W„\i -
The quantity At^{k,i,j)
W^^.
=
is
j/^^^
—
y^,
Az^ =
z^^j
—
equal to h?IQ^[k,i,i) and
interpreted, following Kusnher and Dupuis (1992), as the "interpolation time" for the
Markov
varies
chain.
Note that the interpolation time At^[k,i,j)
from state to
state.
a random variable that
is
At step n of the chain, with the previously defined time scale
A<^, one can verify that
= -XyAt'^^-ALl
= -^zAt^^-ARl
£„^[Az„^]
K\^K] = ^[r- + Mf^ - r)Wu - A^n,
E^lAWl: E^lAW!:]]'' = H^MVAt;; + O(Aii),
^n'[Ay^]
where E^ denotes expectation conditional on the nth-time state
the reflecting processes
y^, z^,
and Wj^ reach
moments
We
2.
call this
The
The
{k
of the
—
AL^, A/2^, and AZ)^ equal
and where
to the positive value h only
when
and second
of the continuous process {Wt,y,,Zg).
property "local" consistency of the Markov chain.
case of consumption
— (AC =
chain jumps along the direction
\,i
^
(M^,J,y^, z^),
their respective boundaries. This implies that the first
Markov chain approximate those
°^
and
+ X,j + ^).
However, the
except in the trivial case A
the direction vector
{
=
1
— \,\,^)
between three grid points
=
(
h):
— 1, A, ^)
from the current state
later state, {k
— l,i + \,j + 0),
^. For ease of
is
{k, i,j) to
the state
usually not on the grid
programming, we take the intersection of
with the corresponding surface boundary and randomize
axijacent to the intersection point.
way that the expected random increment
will
We
randomize
be along the direction
(
in
such a
— l,A,/3) and
length equal to the distance from the starting state to the point of intersection.
of
NUMERICAL APPROACH
4
-
Without
21
we assume
loss of generality,
hereafter that A
+
and {k —
1),
+
1,1
l,j
+
The
1).
transition probabilities can
P^[{k,i,j),{k,iJ
+
+
l)]
=
+ 1J +
1)]
=
Pk[{k,i,j),ik-'^,ij
we can
In this case, also,
E^fAy^:]
where
AC =
4
+
1),
be defined
as:
E^[Az':i]
= 0AC,
-^,
and
^,
and
the increment in consumption.
4.3
A
AC
E„^[AW^„^]
The
= -AC
(29)
quantities in the right-hand
time process
units of consumption.
The Markov— Chain Decision Problem
{A,C)
policy
is
admissible
conditional on
where
—
'
side of the equalities in (29) are the respective changes in the continuous
corresponding to
{k
verify the property of "local" consistency:
= AAC,
is
1,1
^. In this case, the
0-1
=
l)]
/?
P^[{k,iJ),{k-
<
1
by the three grid points {k,i,j
intersection occurs inside a triangle spanned
1,1, j
<
W
problem
= kh
and
preserves the Mairkov property in that
{)Zvy,T,{~t<^''
W=
for the discrete
if it
k'h, y
=
ih
and
y'
=
parameter Markov chain
J''(^,i,j)
=
rnax
^t.
I
I
i'h, z
is
=
A'^K*.', j),
jh and
z'
(*'.
=
i',
j'h.
j')l,
The
if
AC =
control
then to solve the program:
£ ^""""(yn, ^n)Ai^
(30)
71=0
where
tjj
=
Xro</<n ^'l*- Note, (30)
is
analogous to (26) in the sense that the
former approximates the expected integral in the
latter.
sum
in the
k
,
4
NUMERICAL APPROACH
The
dynamic progreimming equation now takes the
discrete
=
J'{k,i,j)
22
iterative
form
m^x{Zk'„',:'P^[{k,iJ)Ak\i\r)]J'{k',i',j'),
maxo<KN.{e-*^''•('^•'•^•)Ev..^;'^/:*[(t,t,;),(fc^^^;0]^'(fc^^^/^
(31)
+U{iJ)At'{k,i,j)}
G Gn, with
for {k,i,j)
that Ph[*,
*]
D-J{k^i,j)
and P^[*,
*]
are the transition probabilities of the chain.
= nk,r,j)-nk,i-l,j)
h
=
.
Using
this notation,
~
=
Jjk
J*"
C^ =
l)
Furthermore, one of these two
if it
is
l,i,j)
D^ J{k,t,j)
=
-
+
2J{k,i,j)
-
l,ij)
.
U{i,j)
l)
<
0,
<
0,
(32)
where
(33)
W{r + Ati)Dt - WArD; - XyD- - ^zDj.
inequcilities
must hold
as an equality at each {k, i,j)
€ Gh-
optimal not to consume at {k,i,j) then (32) holds as an equality.
I 0.
Clearly,
[1
-
e-*^'''(*''-')]/Af''(Jt,z', j)
This leads to a pair of discrete differential
which gives a finite-difference approximation of Bellman's equation (31)
the constrained maximization problem on the domain
5,
J{k
+ m&x{Ctrik,iJ)] +
approximates the discount factor 6 as h
section
and
,
discrete Bellman's equation (31) in the form:
Otherwise, (33) holds as an equality at {k,ij).
inequalities
us denote
h
+ l3Dp\k,iJ)-D;j''{k,iJ +
Ih/MVD^ +
let
= Ak,iJ + l)-J{k,z,j)^
-^
At^{k,i,j)
L
+
we can express the
XDtJ''{k-l,i,j +
In particular,
D-Jik,i,j)
^
^
,
n2 jf,
L>k J\k,i,j)
Now,
recall
Ak,t,j)-Hk,i,J-l)^
D-Jik^iJ) =
^
Ak,i+hj)-J{k,i,j)
DfJiKrJ) =
DjJ{k,i,j)
where we
iterative reflection® at the artificial boundaries,
we
[0,
Mw] x
[0,My] x [0,Mi].
for
In
outline a proof that the above described sequence of finite-difference ap-
proximations, together with the boundary specifications, converges to the solution of the
continuous time Bellman's equation.
^This specification will be detailed in the end of this section.
4
NUMERICAL APPROACH
Boundary
4.4
23
Specification of
- For numerical implementation,
we attach
The Value Function
a "fictitious" surface to each cutoff boundary
We denote by
G^ the grid including this attached exterior surfaces. The three surfaces {W = 0,z =
0, andy = 0) are natural boundaries whereas the three surfaces {W = Mw + h,z =
Mz + h, and y = My + h) are artificial fictitious boundaries. The value function on the
surrounding the cubic grid Gh in which the Markov chain
artificial
conditions
we turn
boundary
=
W=
0,
the boundajy condition
surfax:e
sumption because of
y
=
infinite
0, it is clecir
marginal
value function at this boundary
= ^J'{k,0,j +
• Similarly, at the
boundary 2
diate consumption
J'{k,i,0)
Next consider the
•
is
is
given by the natural constraint
is
1)
=
utility for y
on that boundary. Hence, the
given by:
+
0,
that the optimal policy calls for con-
^A^ -
1,0,;
+
1)
+ jJ'ik -
1,1,;
because of infinite marginal utility for
z,
+
1).
imme-
optimal. Hence, the value function satisfies:
= i^jf^{k,i,l) +
artificial cutoff
At the top-level boundary
^J'{k - l,i, 1) + ^J'{k -l,i +
result, the iterative
the natural constraint
l, 1).
boundaries.
W = M\y +
h,
that the agent will optimally choose to
As a
These are the
^iih,jh).
At the boundary
J'{k,0,j)
surfaces.
to next.
At the boundary
J'{0,ij)
•
defined.
boundaries will be specified to capture the optimal policy in a manner locally
consistent with that of the continuous processes at the
•
is
scheme
XDf J'' +
we take
it
that the wealth level
consume regardless
~
DtJ*"
=
0.
so high
of the levels of z or y.
of the value function at that
I^Dfj''
is
boundary
is
given by
5
CONVERGENCE OF THE NUMERICAL SCHEME
•
= My +
At the y-side boundaxy y
-
/3D/ J''
•
At the
z-side
boundary
z
5
we
let
DfJ^ =
0,
if
=
0,
if
DtJ''
= M^ +
XDfJ''
h,
/i,
we
24
AC =
AC =
and
h.
let
Dp'' =
0,
if
- DtJ^ =
0,
if
AC =
AC =
and
h.
Convergence of the Numerical Scheme
we analyze the convergence
In this section,
of the value function
approximating Markov chain problems to those
trols in the
We
time problem.
scheme described
first
and the optimal con-
in the original continuous
analyze the properties of the finite-difference approximation
in section 4.
In particular,
we show
that the discrete
Markov chain
approximation approach produces a monotone, stable, and consistent finite-difference
approximation to the Bellman's equation that characterizes the value function.
Armed with
these results,
we proceed
domain
Qm
and the
size of the cubic
discrete
Markov chain problems converges
to prove that, as the
mesh
size goes to zero
increases, the sequence of value functions in the
to the value function of the original
problem
uniformly on any compact subset of the domain Q. Furthermore, convergence obtains
gardless of the
boundary conditions
The proof
domains Qmfunction
J
is
case with a
equation.
is
based on the
artificial
boundaries of the restricted
we already
established, that the value
on the
fact,
the unique viscosity solution of Bellman's equation (27).
smooth value function
follows directly
situation
is
specified
(see, for instance,
from consistency and
different because
We, hence,
we do
utilize the
re-
Strikwerda (1989)), convergence
stability of the finite-difference
not, in general,
In the classical
have a
scheme. Here, the
classical solution of
Bellman's
comparison principle for subsolutions and supersolutions
of Bellman's nonlinear partial differential equation in order to gu£u-antee convergence of
the approximations to the value function of the continuous-time problem.
CONVERGENCE OF THE NUMERICAL SCHEME
5
25
Properties of the Finite— Difference Scheme
5.1
Let B{Gh) denote the spa<:e of real-valued functions on the extended lattice Gh- To study
the discrete Bellman's equation (31), we introduce the mappings
T^: B{Gh)-^B{Gh)
=
{T;^V)ik,i,j)
and
e-'^'"^"''^^
Sh
:
B{Gk)
^ B{Gk)
where
VA
Y: P,[{k,iJ),{k\i'J')]Vik',t'J'),
k',i',j'
=
{S,V){k,iJ)
Yl
PK[{k,i,j)Ak',i\j')]V{k',i',j'),
k',i',j'
for
any
V
G B{Gh)- In the case
Obviously A/,
II
lloo
>
0.
of no consumption,
Define the operator
norm
•
||
||
as
we
set
Ak =
HrH
=
sup^gg
m\nk,i,j
At^{k,i,j).
||Tu||oo/||u||oo5
with
being the L°°-norm of the bounded function space B.
The
discrete Bellman's equation can be rewritten as follows:
J\k,ij) = max{max[T;f J''(fc,t-,j)] + u{iJ)At\k,iJ),SHJHk,iJ)},
V(^,t, ;)
€ Gh.
(34)
It is
clear that the solution of the discrete Bellman's equation
operator equation (34).
The
is
a fixed point of the
properties of the operators T/^ and 5a, which are useful for
proving convergence, are recorded in the following proposition.
Proposition 2
7^*
'5
are strict contraction operators with
a contraction operator with
\\Sh\\
<
1.
\\Tf^\\
<
e"*^*"
<
1
and Sh
is
Thus, for every h, there exists a function J^ that
solves (34)Proof.
We
estimate directly
\T,^V{k,i,j)-T;^V'ik,i,j)\
<
Y: PH[ik,tJ)Ak\i'J')]\V{k\i\j')-V'{k',t\j')\
e-^^'''('=-'"-^)
k'.i'j'
<
max
e-*^"
\V{k,iJ)-V'{k,i,j)\
and
(*,i,»eGh
\ShV{k,iJ)-SHV'ik,iJ)\
<
Y
Ph[{k,iJ),{k',i\j')]\V{k',i'J')-V{k',i'J')\
k',i',j'
<
max
(*,t,j)eGh
\V{k,iJ)-V'ikJ,j)\
,
5
CONVERGENCE OF THE NUMERICAL SCHEME
26
which implies the property of contraction. Given that both Tf^ and Sh are both contractions, the
second assertion follows.
I
Next, we discuss the properties of the finite-difference scheme, (32) and (33), associ-
On
ated with the Markov chain approximation.
finite-difference operator
at all
^hV ^
when v <
equation
differential
We
v.
will that v
v'.
The scheme ChV = /
Cv = f
if,
pointwise at each grid point, as
for ciny
/i
J.
>
such that
there exists
compact domain
T)
and some positive constant
The
scheme we constructed
Theorem
will
each grid point,
result, at
v' \i
is
denote a given
v{W,y,z) < v'{W,y,z)
monotone
said to be
said to be consistent with the partial
we have
<f>,
||t;''l|t»
^
K,
scheme
<
for all
K independent of h,
—
£/,</>
/i
where
<
Second, we
||p
for
>
0,
is
any
denotes
in section 4.2.
stable,
and
scheme for Bellman's equation.
monotonicity of the scheme
<
operators T/^V
/iq,
•
|1
—
= /
C4>
Ch.v^
3 The discTtte Markov chain approximation yields a monotone,
First,
if
following result establishes the properties of the finite-difference
consistent finite-difference
Proof.
we
h,
Finally, a finite-difference
0.
if
/iq
is
smooth function
said to be stable
the L°°('D)-norm.
<
Recall that a finite difference scheme
{W,y,z) € Q.
jC/iu'
by Ch and interpret ChV to be the
scheme to the function
of applying the
a grid of size
ShV < ShV
T;^V' and
Cein verify directly that, for
is
for
implied by monotonicity of the discrete
any V,V' G B{Gh) such that
any smooth function
<^,
V < V.
and with the obvious
notation:
Tl^4>\k,i,j)-<f>\Ki,j)
Sh<t>'{k,i,j)
-
<l>'{k,
i,j)
=
=
-[l-e-'''''^''''^^)]4>'{k,i,j)
[XDr<f>'{k
-
l,i, j
+
1)
+
+ CU'{k.ijW{k,i,j),
- D;4>\k,i,j +
^Dl<f>\k,i,j)
l)]h,
(35)
where
<i>'*{k,i,j)
Df(i>
D;
(i>
=
^{kh,ih,jh).
= 4>w±
=
<i>y
-^
^<f>ww
-4>yy
X h
The
difference terms can be estimated
xh + C>(/i')
+
0{h^)
,
,
a.nd
Dl<f>
=
Dj 4> =
4>ww
(f),
+
+
by
0(/i')
-<f>,,
xh +
0{h^).
5
CONVERGENCE OF THE NUMERICAL SCHEME
Therefore,
£'^<f>
—
=
€*<!>
27
0{h), which implies the consistency of our finite-difference
from Proposition 2 that,
any investment policy A,
approximation. Finally,
it
the set {(T;^,
uniformly bounded in operator norm, which implies the stability
5/,);
V/i}
is
follows
of the finite-difference scheme.
Note that consistency
I
theorem
as used in
3
is
actually equivalent to the "local"
consistency property (28) and (29) of the corresponding
5.2
Markov
chain.
Convergence of The Value Function
In this section,
we
report the result that the sequence of value functions in the approxi-
mating decision problems converges
program. The proof of
in the
for
to the Vcdue function of the original continuous
time
this result requires special attention to the gradient constraints
sequence of Bellman's equations of the approximating control problems.
Theorem
4 (Convergence) The sequence of solutions
J'^ik, i,j)
of the discrete
Markov
chain problems converges to the value function J{W,y,z) of the continuous-time problem
uniformly on any compact subset of
Ny
t oo, in
Proof.
Q =
(0, oo)"'
and
as h [
such a way that hk —* W, hi —* y, and hj —*
Nw
T oo?
^z
T °°> '^^^
z.
For each (W^,y, 2) 6 Q, define
r{W,y,z) =
limsup
J^{W\y\z'),
and
(36)
(W,v',x')-(V*',»,x)
fciO,
J.(iy,y,z)
=
(W',v',i')gG'>
,^,liminf,
j\W\y\z').
(37)
'>10.(W",>i',j')€G''
every point
J'
is
finite at
J'
is
upper semicontinuous
(VF, j/,2) since
ajid J,
is
the finite-difference scheme
is
stable.
lower semicontinuous, and
J>{W,y,z)<J'{W,y,z), on Q.
Note that
for {y^z)
J''(0, y, 2)
=
'I'(y, 2),
G (0,00) X (0,oo).
We
Clearly,
V(y, 2) G G^. Therefore, J'(0, y, 2)
will verify
here that J*
is
(38)
=
^{y, 2)
=
J.(0, y, z)
a viscosity subsolution and J»
CONVERGENCE OF THE NUMERICAL SCHEME
5
is
28
a viscosity supersolution of Bellman's equation
max{-<5J+max{£^J}+u(j/,2),AJy+^Jj-JH'} =0,
on
Q
with J(0,y,z)
=
*(y,-2).
(39)
It
<
then follows from the comparison result in theorem 2 that J*
together with (38), implies that J'
there exists
limit
/i
J.,
and {Nw-, Ny, N^)
I
some ho >
domain spanned by the
t cxd in
such that for h
grid Gh-
By
result,
and hence, by uniqueness, both are equal to J.
Q be a strict local mjiximumof J' ~(f> for some
Let {Wo, yo, zo) G
we take the
=
on Q. This
J,
<
€
(j)
C^'^'^{Q). Since
such a way that h x {Nw, Ny, N^)
ho, {Wo.,yo,2o) is
] oo,
an interior point of the
the property of "limsup", there exists a sequence
indexed by h such that
{Wh,yh',Zh) -^ {Wo,yo,zo),
For h
A/'o
<
ho,
J^ — 4>
hais
aloccJ
maximum at
{Wh.,yh-i Zh),
and hence,
for
some neighborhood
{W,
y, z)
of (VVo,2o,yo),
J\W,, y,, z,) Since
J'' is
4>{Wf,, y,, zh)
> J\W, y, z) -
4>{W, y, z),
if
6
A^o
(40)
.
a solution of the discrete operator equation
j''iWH,yk,ZK)
=
m^x{m^x[T^j''{WH,yH,ZH)]
using (40) and the monotonicity of Tf^ and Sh,
max{max[T^^(^
On
ash 10.
J''{W'h,yh,Zh) -^ J'{Wo,yo,ZQ)
the other hand,
-
<f>]{Wh,
we
recall
y/.,
+
u{yH,ZH)At\S'HJ\WH,yH,Zh)},
we can
^0 + ^Vh, ZK)At\
replace J^ by
[5/.<A
-
<!>
<^]{Wh, yn,
such that
^O} >
from (35) that
T;^<i>-<f>
=
-[l-e-^^''']<P
SH<f>-<f>
=
[XDt4>
+
+ C^4>At\
and
(42)
0Dl<f>-D:<f>]h.
(43)
Therefore, dividing (42) by Af'' and (43) by h, the inequality (41) at {Wh,yh,
max{-(
(41)
0.
^^r— )^ + max[£;f + u(y,, z,), AD.+ ^ + /3Dlcf> .?i]
2/1)
D;4>}
>
becomes
0,
5
CONVERGENCE OF THE NUMERICAL SCHEME
for h
<
ho-
Finally,
29
by the consistency of our fmite-diiference scheme, as we
let
/i
j 0,
we obtain
max\^-S<f>
Thus, J'
is
+
max[£'^(^]
+
u(yo, zq),
X<f>y
+
^(f>^
a viscosity subsolution by Definition
combined with
2, it follows
(38), proves that, for all
Hm
j\W',y',z') exists
that
4>w]
>
0,
at
(Wq,
j/o, -^o)-
1.
Using similar reasoning, we can show that J,
comparison principle, theorem
-
a viscosity supersolution.
is
J'{W,y,z)
<
J,(\V,y,z) on
Q
By
the
which,
{W,y,z) 6 Q,
=
=
J{W,y,z)
J'{W,y,z)
=
J,{W,y,z).
I
hlO, (VV",v',i')6G''
In the
ifications
argument of convergence, we do not require knowledge of the particular specon the cutoff boundaries
for the
Mao-kov chains.
The same
limit for
J^
will
be obtained regardless of the conditions chosen at the cutoff boundaries. This results
from the
mesh
fact that, in the theoretical sequence of approximations, as the
creases, the size of the corresponding
bounded domain
increases.
numerical implementation, the size of the restricted domain
is
However,
size de-
in the actual
kept fixed and hence the
choice of the boundary conditions affects the quality of the numerical solution.
boundary specifications discussed
in the previous sections
The
were chosen to obtain a good
approximation to the solution of the continuous-time problem.
5.3
Convergence of Approximating Policies
Existence of the value function and the feasibility of computing
its
values with high
accuracy do not guarantee existence of optimal policies that achieve the value function.
Contrast this with classical feedback control case. In that case, optimal controls can be
expressed as explicit functions of the value function and
existence of a
its
derivatives.
smooth value function immediately implies existence
In our case, with a free-boundary problem, there
value function and the optimal controls, which,
the value function
is
not
known
to be smooth.
if
is
no
As a consequence,
of optimal controls.
direct relationship
exist, are not feedback.
between the
Furthermore,
CONVERGENCE OF THE NUMERICAL SCHEME
5
30
we prove convergence of the optimal investment
In this section,
policies in the approx-
imating Markov chains to the optimal investment policy of the diffusion control problem
if
We
the latter indeed exists.
will
parameterize the sequence of approximating problems
by h and denote the optimal consumption and investment
and
A''*, respectively.
We
policies in
problem h by C'^
Markov
define an interpolation of the value function for the
chain problem using the following linear aifine interpolation:
=
j\W,y,z)
for
W<
<
kh
{k
-h
+ DtJHk,i + lJ + l)x{W-kh)
+D-J\k,ij + 1) X (y - ih) + D-J\k,i,j) X {z-jh)
j'{k,iJ)
\)h, ih
<
y
<
+
{i
l)h,
and jh
<
extensions at the cutoff boundaries. Clearly, J'*{W, y, z)
[0,
Mw]
X
Lemma
[0,
1
Mj,]
As
W
Proof.
The proof
on
Qm =
\)h; with appropriate
continuous on the cubic region
that
is
nondecreasing in {W,y,z) and con-
M]^.
[0,
J'^ is
as the proof of property
827).
is
+
{j
X [0,Mj]. One can easily show the following
a function of{W,y,z), J^{W,y,z)
cave in
W<
1
nondecreasing in
its
arguments follows the same reasoning
of the value function in Fitzpatrick
Furthermore, the proof of concavity of
(page 829) in the same reference.
J''
in
and Fleming (1991, page
W follows the proof of property 2
I
Next, we utilize the special monotonicity and concavity features of the problem to
prove convergence of the
first
and second order derivatives dJw{k, i,j)
dJwwi^-ihJ) ~^ Jww{^,y,z)
is
as
/i
J.
and {hk,hi,hj)
used to show that the optimal investment
A'^ for
>
>
Jw{W^,
and
for the
Markov chain
control
continuous-time model
latter exists.
Theorem
5 Suppose that there exists an optimal control policy {A',C') for the original
continuos time problem.. Then, the sequence of policies A'^{k,i.,j) converges at
{W.,y.,z)
y, ^)
{W,y,z). This information
the discrete
problem converges to the optimal investment policy A'
when the
—
—
where
Jw and Jww
exist,
which
is
all
points
almost everywhere, to the optimal investment
5
CONVERGENCE OF THE NUMERICAL SCHEME
A'{W,y,z) as h
policy
hk
— W, hi
>
Proof.
—y y, and hj —*
Nw
and
I 0,
oo,Nz
1
1 oo,
and Ny
] oo,
in
such a way that
z.
Using standard arguments, we can show that the value function J
and concave
in its
arguments.
Bellman's equation, J
the maximization in
satisfies that
increasing
result, since
J
is
first
a viscosity solution of
equation in almost everywhere sense. Carrying out
we conclude that A'
(7),
is
then follows; see Royden (1988, §5.5), that J has
It
and second derivatives almost everywhere. As a
We
31
= j—-^'^
almost everywhere.
rewrite the "finite-difference" form of the Bellman's equation for the interpolated
value function J^
_ -SAt"
[—^-^]J'{k,tJ) +
1
in the case of
no consumption
ment policy by the formula A'
grid.
Note that only the
first
0{h)
AC =
0.
{k, i,j)
=
= ^ma^^{£if /"(/:, i,;)} +
From
this,
f/(i,j),
(44)
we can compute the optimal
—
wa^D^J^>^
'
^°^ (^' ^'?)
and second order derivatives
^
^~
invest-
*^^ interior
of J, with respect to
W,
are
used in that formula. Substituting the previous formula back into the difference equation
(44),
and
after a long calculation,
we can
express DlJ'^ as H{Dfj'^,
some continuous function H. Now, using arguments
Fleming (1991, theorem
and
A'^
D~ J^
— A',
>
of the
—^ J^.
As a
and lemma
result,
1
,
we can prove
that DfJ'^ —^
Qm
sumption and abstinence
in the
as
/i
|
and
that
as the size
approximating sequence converge to the corresponding
Specifically,
if
at a given point (iy,y,z), the optimal
jump
in
solutions, except for, at most, a finite
grid points that converge to
calls for
,
Jy,
I
solution of the original problem prescribes a
time problem
W
»
following result that shows that the optimal "regions" of con-
regions in the original problem.
mating optimal
D~ J^ —
D\J^ —+ Jww and consequently we must have
increases to Q.
we provide the
Jw,
for
and
similar to those in Fitzpatrick
at the points of differentiability with respect to
domain
Finally,
4),
D~ J^, Dj J^i J^)
{W
,
y, z). Similarly, if
no consumption
at
consumption, then,
number,
all
also prescribe
the approxi-
jumps
at the
the optimal solution in the continuous
{W,y,z), then almost
all
the approximating
CONCLUDING REMARKS
6
optimal solutions
Theorem
32
no consumption
will also caJl for
6 Suppose that there
exists
converge to
at the grid points that
an optimal control policy (A'^C) for the original
continues time problem. Consider the sequence of approximating problems indexed by h
and
h I
let
0,
and
value function
J
Nw
is
T °°i
^z
T °°> <*"<^
^y
Let x
T oo-
=
Q
{W, y,z) €
twice continuously differentiate in a neighborhood of x. Let
be the size of the optimal
consumption jump
=
Let {x^}
at x.
{(fc,
=
O;, then AC*''(x'')
(AC'\x^) =
>
0), respectively,
for
AC(x)
i,j)} be a sequence
of grid points that converges to x in such a way that J'^{x^) —^ J{i)-
(AC'{x)
be such that the
If
all h,
AC*{x) >
except for, at
most, a finite number.
Consider the case when AC*(i)
Proof.
=
0.
Since
J
is
smooth
in a
neighborhood of
follows from arguments similar to those used for proving the verification result
2—
in
HH&Z
It
it
— theorem
that
pj, -\-\Jy-
Now
x,
suppose that AC*''(x'')
=
for only
Jw <Q
a
at X.
finite
(45)
number
of points in the sequence.
then follows from Bellman's equation for the Markov chain decision problem that
/3D/J''
+
XDfJ''
-
DtJ''
=
a.t
x^, for
but a
all
sequence. However, from the proof of theorem
lim(/3£>t J''
which contradicts
ments and
6
The
(45).
+
XDtJ''
The proof
utilizes the result of
-
5,
DtJ''){x^)
of the case
finite
number
we conclude
=
{^Jz
+
XJy
that
- ^)(x) =
when AC*(x) >
theorem 5 that
of elements in the
follows the
{A'''} converge to A".
same
argu-
I
Concluding Remarks
analysis of this paper achieves its objective of providing technical support for the
numerical solution reported in
HH&Z.
In particular,
we provide
proofs of convergence
of the sequences of both the vaJue functions, the optimal investment policies,
and the
CONCLUDING REMARKS
6
33
optimal consumption regions in the approximating problems to the corresponding entities
in the original
problem. Such results justify the description
There are
results as e-optimal solutions.
We
still
some
We
hope that further research
We
remark that there
is
silent
on the
in this area
HH&Z
of the numerical
interesting theoretical
have not proved existence of an optimal solution
Furthermore, we are also quite
in
in the class of
open questions.
admissible controls.
reguljirity properties of the free-boundary.
sheds some light on these issues.
another method of proving convergence of the sequence of
value functions in the approximating
Markov decision problems. That method
is
based
on probabilistic techniques from the theory of weak convergence of probability measures.
For a detailed exposition of the probabilistic approach,
and Dupuis (1992, §9 — §10) and the references
therein.
we
The
refer the reader to
idea there
is,
Kushner
very roughly, as
follows.
One constructs continuous time interpolations of the optimally controlled Markov
chains.
Then one proves
that the sequence of interpolated processes has a subsequence
that converges weakly to an optimally controlled process of the original problem. Since
the approximating sequence has the same objective functional as the original problem, one
then concludes, using the theory of weak convergence, that the sequence of approximating
value functions converges to the value function of the original problem.
Furthermore, by extending the space of admissible controls to the so-called "relaxed
controls"; see
Kushner and Dupuis (1992,
relaxed control for the original problem.
be
§4.6),
one can show that there exists an optimal
The optimal
in the class of original admissible controls. In
any
relaxed controls, however, need not
case, the
sequence of approximating
optimal controls converges weakly to the optimal relaxed control of the original problem.
We
chose not to present the probabilistic approach here because the notion of weak
convergence does not guarantee that the optimeil relaxed control are adapted to the
information structure in the economy. As a result, the optimal relaxed control
to be feasible
We
and admissible control
may
fail
policies.
remark, however, that using the analytical approach we can prove that the ap-
proximating optimal investment policies and the consumption regions converge pointwise
to their corresponding counterparts in the original
of admissible controls. This
is
problem
if
the latter exists in the class
a stronger result than weak convergence to a relaxed con-
7
REFERENCES
trol,
which
may
34
not be in the class of the original admissible controls, that obtains using
probabilistic techniques.- Finally,
optimaJ policies
we note
that neither approach establishes existence of
in the originad admissible class of controls described in section 2.2. Es-
tablishing such a result requires information about the regularity of the free-boundary.
For more on this issue, we refer the reader to Soner and Shreve (1989) and Hindy and
Huang
(1993).
References
7
1.
M. Crandall and
P. L. Lions, Viscosity
Solutions of Hamilton- Jacobi Equations,
Transactions of the AmericeLD Mathematical Society 277, 1982, pp. 1-42.
2.
M. G. Crandall, H.
Ishii,
ond Order PartiaJ
Differential Equations, Bulletin of the
and
P. L. Lions, User's
Guide
to Viscosity Solutions of Sec-
American Mathematical
Society, 1992.
3.
B. Fitzpatrick and
W.
Fleming, Numerical Methods for Optimal Investment-Consumption,
Mathematics of Operations Research 16, 1991, pp. 823-841.
4.
5.
W. Fleming and
H. Soner, Controlled Markov Processes and Viscosity Solutions,
Springer- Verlag,
New
W. Fleming and
Player,
Zero-Sum
York, 1993.
P. Souganidis,
On The
Stochastic Differential
Existence of Value Functions of
Two-
Games, Indiana University Mathematics
Journal 38, 1989, pp. 293-314.
6.
A. Hindy, C. Huang, and H. Zhu, Optimal Consumption and Portfolio Rules With
Durability and Habit Formation, Working Paper, Stanford University, 1993.
7.
A. Friedman, Partial Differential Equations of Parabolic Type, Prentice-Hall,
New
York, 1964.
Inc.,
7
REFERENCES
8.
35
A. Hindy and C. Huang, Optimal Consumption and Portfolio Rules
With Durability
and Local Substitution, Econometrica 61, 1993, pp. 85-122.
9.
R. Holmes, Geometric Functional Analysis and Its Applications, Springer-Verlag,
New
10.
H.
York, 1975.
J.
Kushner, Probability Methods for Approximations
for Elliptic Equations, Academic Press,
11.
H.
J.
Kushner and
P. G. Dupuis,
New
York, 1977.
New
York, 1992.
O. A. Ladyzenskaja, V. A. Solonnikov, and N. N. Ural'ceva, Linear and Quasilinear
Equations of Parabolic Type, Academic Press,
13.
Control and
Numerical Methods for Stochastic Control Prob-
lems in Continuous Time, Springer-Verlag,
12.
in Stochastic
P. L.
New
York, 1968.
Lions and P. E. Souganidis, Differential Games, Optimal Control, and Direc-
tional Derivatives of Viscosity Solutions of Bellman's
and
Isaac's Equations,
SIAM
Journal on Control 23, 1985, pp. 566-583.
14.
P. L. Lions,
Optimal Control of Diffusion Processes and Hamilton-Jacobi-Bellman
Equations, Part
II:
Viscosity Solutions and Uniqueness,
CommuDications
in
Pure
and Applied Matbematic's 37, 1983, pp. 511-553.
15.
H. L. Royden, Real Analysis, Third Edition, Macmillan Publishing Co.,
New
York,
1988.
16.
H. M. Soner and
S.
E. Shreve, Regularity of the Value Function of a
Singular Stochastic Control Problem,
SIAM
Two-Dimensional
Journal on Control and Optimization
27, 1989, pp. 876-907.
17.
J.
C. Strikwerda, Finite Difference
Wadsworth
&
Schemes and Partial
Brooks, California, 1989.
Differential Equations,
7
REFERENCES
18.
T. Zariphopoulou, Investment-Consumption Models with Transaction Fees and
Markov Chain
-
36
Paramieters,
SIAM
Journai on Control and Optimization 30, 1992,
pp. 613-636.
19.
H. Zhu, Variational Inequalities and Dynamic Programming for Singular Stochastic
Control, Doctoral Dissertation, Division of Applied Mathematics,
sity,
1991.
Brown Univer-
2935
j6i.
MIT LIBRARIES
3
lOaO D0flSbSE4
1
Date Due