Chap 4
Overview of Infinite-Horizon Problems
Infinite-horizon problems differ from finite-horizon ones in two key respects:
(a) The number of stages is infinite.
(b) The system is stationary: the dynamics f , the stage cost g, and the disturbance law do not change over time.
The author notes that while true infinity rarely occurs in practice, it serves as
an excellent approximation when the horizon is very large and/or the parameters vary slowly. Moreover, periodic or nonstationary problems can often be
reformulated as stationary ones (see [?], Sec. 4.6). Stationary infinite-horizon
problems admit elegant analysis, and their optimal policies are typically stationary (time-independent).
Types of Infinite-Horizon Problems
The chapter focuses on two classes of total-cost minimization problems:
1. Discounted problems:
"N −1
#
X
k
Jπ (x0 ) = lim
α g xk , µk (xk ), wk x0 ,
N →∞
0 ≤ α < 1.
k=0
Here α is the discount factor; when α < 1, future costs are geometrically
down-weighted.
2. Stochastic Shortest Path (SSP):
Jπ (x0 ) =
−1
hτX
i
g(xk , µk (xk ), wk ) x0 ,
k=0
where there is a special termination state t with zero cost, α = 1, and τ
is the (random) time of termination. The author assumes termination is
inevitable under any admissible policy.
Value Iteration and Bellman’s Equation
The author introduces the value iteration (VI) recursion for the N -stage cost
functions:
J0 (x) = 0,
(1)
Jk+1 (x) = min g(x, u, w) + Jk f (x, u, w) ,
u∈U (x)
He argues (and later proves) that under suitable conditions:
1
k ≥ 0.
(2)
1. Convergence of costs:
J ∗ (x) = lim JN (x),
N →∞
where J ∗ (x) is the optimal infinite-horizon cost.
2. Bellman’s equation:
h
i
J ∗ (x) = min g(x, u, w) + J ∗ f (x, u, w) .
u∈U (x)
This functional equation uniquely characterizes J ∗ .
3. Stationary optimal policy: any selector
µ∗ (x) ∈ arg min g(x, u, w) + J ∗ (f (x, u, w))
u∈U (x)
yields an optimal policy {µ∗ , µ∗ , . . . }.
Intuitively, the author explains, stationarity of the problem implies that “the
future optimization problem looks the same regardless of the stage at which it
begins.”
Finite–State Formulation
Restricting to a finite state set {1, . . . , n} (plus termination state t for SSP), let
pij (u) denote the transition probability from state i to j under control u ∈ U (i),
with stage cost g(i, u, j). Under a policy π the system is a Markov chain {ik }
and the total expected cost from i is
Jπ (i) = lim
N →∞
−1
hNX
αk g(ik , µk (ik ), ik+1 )
i
i0 = i .
k=0
The optimal cost function is J ∗ (i) = inf π Jπ (i), and the cost of a stationary
policy µ is Jµ (i). The author will show (under mild assumptions) that an
optimal stationary policy always exists and satisfies
Jµ (i) = J ∗ (i),
∀ i.
Key Takeaways
• Infinite–horizon stationary problems admit value iteration and obey Bellman’s functional equation.
• Optimal policies can be taken to be stationary.
• Discounted and SSP problems share essentially the same algorithmic structure, distinguished only by α < 1 versus α = 1 plus explicit termination.
• Finite-state models simplify notation via transition probabilities pij (u)
and allow a clear Markov-chain interpretation.
===============================
2
4.2 Stochastic Shortest Path Problems
Here we set α = 1 and introduce a cost-free absorbing termination state t:
ptt (u) = 1,
g(t, u, t) = 0,
∀ u ∈ U (t).
States 1, . . . , n are all nonterminal states.
Bellman Equation and Value Iteration
For each nonterminal state i, the Bellman equation becomes
∗
h
J (i) = min pit (u) g(i, u, t) +
u∈U (i)
n
X
i
pij (u) g(i, u, j) + J ∗ (j) .
(3)
j=1
The corresponding Value-Iteration (VI) recursion, for any initialization J0 (i), is
n
h
X
i
Jk+1 (i) = min pit (u) g(i, u, t) +
pij (u) g(i, u, j) + Jk (j) .
u∈U (i)
(4)
j=1
The bracketed term in (3) splits into:
• pit (u) g(i, u, t): cost of terminating this step,
P
•
j pij (u) g(i, u, j): cost of nonterminating transitions,
P
∗
•
j pij (u) J (j): optimal cost-to-go from the next state.
Assumption: Inevitable Termination
[4.2.1] There exist integers m ≥ 1 and a constant 0 ≤ ρ < 1 such that
ρ = max max P xm ̸= t | x0 = i, π < 1.
π
i=1,...,n
The author remarks that this ensures the probability of not reaching t by
time km decays as ρk , so the limit in the cost definition exists and is finite for
any policy.
Main Theoretical Results
Under the above assumption, the following propositions hold:
[4.2.1, VI Convergence] For any initial J0 (i), the iterates Jk (i) defined by
(4) converge to J ∗ (i) < ∞ for all i = 1, . . . , n.
[4.2.2, Bellman’s Equation Uniqueness] The vector J ∗ = (J ∗ (1), . . . , J ∗ (n))
is the unique solution of (3).
3
[4.2.3, VI & Bellman for a Fixed Policy] For a stationary policy µ, its cost
Jµ satisfies
n
X
Jµ (i) = pit µ(i) g(i, µ(i), t) +
pij µ(i) g(i, µ(i), j) + Jµ (j) ,
j=1
and the VI recursion with µ converges to Jµ .
[4.2.4, Optimality Condition] A stationary policy µ is optimal if and only if,
for every i, µ(i) attains the min in (3).
Example: Maximum Expected Time to Termination
Set g(i, u, j) = −1. Then minimizing total cost is equivalent to maximizing the
expected time to reach t. Bellman’s equation becomes
n
h
i
X
J ∗ (i) = min −1 +
pij (u) J ∗ (j) ,
u∈U (i)
i = 1, . . . , n,
j=1
and for a fixed µ, −Jµ (i) is the mean first passage time from i to t.
=====================================
Bellman Operators and Contraction
Define for any cost-to-go vector J ∈n :
n
h
X
i
(T J)(i) = min pit (u) g(i, u, t) +
pij (u) g(i, u, j) + J(j) ,
u∈U (i)
j=1
n
X
pij µ(i) g(i, µ(i), j) + J(j) .
(Tµ J)(i) = pit µ(i) g i, µ(i), t +
j=1
Here T and Tµ are the Bellman (DP) operators, and the fixed-point equations
J ∗ = T J ∗ , Jµ = Tµ Jµ are just Bellman’s equations.
Proposition (Contraction). There exists a weight-vector v > 0 and scalar
ρ ∈ (0, 1) such that, with the norm
∥J∥v = max
i
|J(i)|
,
v(i)
the operators satisfy
∥T J − T J ′ ∥v ≤ ρ ∥J − J ′ ∥v ,
∥Tµ J − Tµ J ′ ∥v ≤ ρ ∥J − J ′ ∥v .
Here v(i) is the maximal expected steps to reach t from i, and ρ = maxi v(i)−1
v(i) .
k
Hence VI converges at rate ρ .
4
Q-Factors and Their VI
Define the optimal Q-factor
Q∗ (i, u) = pit (u) g(i, u, t) +
n
X
pij (u) g(i, u, j) + J ∗ (j) ,
j=1
so that an optimal stationary policy is
µ∗ (i) ∈ arg min Q∗ (i, u).
u∈U (i)
The Bellman equation for Q∗ is
Q∗ (i, u) = pit (u) g(i, u, t) +
n
X
h
i
pij (u) g(i, u, j) + min Q∗ (j, v) ,
v∈U (j)
j=1
and VI on Q-factors iterates
Qk+1 (i, u) = pit (u) g(i, u, t) +
n
X
h
i
pij (u) g(i, u, j) + min Qk (j, v) .
v∈U (j)
j=1
Temporal Differences and Cost Shaping
Introduce any baseline vector V ∈n , V (t) = 0, and define
(
g(i, u, j) + V (j) − V (i),
∗
Jˆ = J − V, ĝ(i, u, j) =
g(i, u, t) − V (i),
j ̸= t,
j = t.
Then the “variational” Bellman equation reads
n
h
X
i
ˆ = min pit (u) ĝ(i, u, t) +
ˆ
J(i)
pij (u) ĝ(i, u, j) + J(j)
.
u∈U (i)
j=1
The author observes that this is equivalent to the original SSP, so exact DP
yields the same J ∗ . However, under approximation, choosing V ≈ J ∗ (“cost
shaping” or “reward shaping”) can reduce the range of Jˆ and improve approximation quality.
====================================
4.3 Discounted Problems
Here 0 ≤ α < 1. Using the finite-state notation, for each state i = 1, . . . , n:
5
Bellman Equation and Value Iteration
J ∗ (i) = min
u∈U (i)
Jk+1 (i) = min
u∈U (i)
n
X
j=1
n
X
pij (u) g(i, u, j) + α J ∗ (j) ,
pij (u) g(i, u, j) + α Jk (j) ,
(5)
k ≥ 0.
(6)
j=1
Equivalence to an SSP Formulation
Define an SSP with the same states 1, . . . , n plus a new termination state t,
where
pSSP
ij (u) = α pij (u),
pSSP
it (u) = 1−α,
g SSP (i, u, j) = g(i, u, j),
g SSP (i, u, t) = 0.
Then for any policy, stage k cost in SSP equals αk times the undiscounted cost,
matching the discounted problem. Hence Props. 4.2.1–4.2.5 apply, giving:
• VI convergence: Jk → J ∗ as k → ∞.
• Uniqueness: J ∗ is the unique solution of (5).
• Stationary optimal policies exist, obtained by minimizing the RHS of (5).
• Contraction: Define
(T J)(i) = min
u∈U (i)
X
pij (u) g(i, u, j) + αJ(j) ,
j
then for the sup-norm ∥J∥ = maxi |J(i)|,
∥T J − T J ′ ∥ ≤ α ∥J − J ′ ∥.
Example: Asset-Selling Problem
Offers v1 , . . . , vn occur i.i.d. with prob. pj . If accepted at period k, the discounted
reward is (1 + r)−k vik , so α = (1 + r)−1 . Bellman’s equation becomes
n
n
o
X
J ∗ (i) = max vi , α
pj J ∗ (j) .
j=1
An optimal rule is “sell if vi ≥ c,” where
c=α
n
X
pj J ∗ (j).
j=1
6
Cost Shaping for Discounting
For any baseline V ∈n , define Jˆ = J ∗ − V and
ĝ(i, u, j) = g(i, u, j) + α V (j) − V (i).
Then the variational Bellman equation is
n
X
ˆ = min
J(i)
u∈U (i)
ˆ
pij (u) ĝ(i, u, j) + α J(j)
.
j=1
The author notes that while exact DP is invariant to V , approximate methods
(e.g. TD, policy gradient) can benefit from choosing V ≈ J ∗ (“reward shaping”).
Q-Factors in Discounted Problems
Define
Q∗ (i, u) =
n
X
pij (u) g(i, u, j) + α J ∗ (j) .
j=1
Then
Q∗ (i, u) =
Qk+1 (i, u) =
n
X
j=1
n
X
h
i
pij (u) g(i, u, j) + α min Q∗ (j, v) ,
v∈U (j)
h
i
pij (u) g(i, u, j) + α min Qk (j, v) .
v∈U (j)
j=1
An optimal policy is µ∗ (i) ∈ arg minu Q∗ (i, u). These VI updates form the basis
of Q-learning and other approximate DP/RL methods.
===============================
1
Semi-Markov Discounted Problems
The author extends the standard discrete-time discounted DP framework to
semi-Markov models, where state transitions occur after random intervals. Key
definitions and results:
• Transition distributions: For state i, control u, and next state j, define
ξij (τ, u) = P tk+1 − tk ≤ τ, xk+1 = j | xk = i, uk = u .
Ordinary transition probabilities recover via
pij (u) = lim ξij (τ, u).
τ →∞
The conditional CDF of the holding time τ given (i, j, u) is
ξij (τ, u)
,
pij (u)
7
pij (u) > 0.
(Thus ξij is a “scaled CDF,” modelling both discrete & continuous sojourn
times.)
• Expected sojourn time:
n Z ∞
X
τ i (u) =
τ dξij (τ, u),
j=1
0 < τ i (u) < ∞.
0
• Cost per transition: If g(i, u) is cost rate, the expected present-value
cost of one transition is
n Z ∞
X
1 − e−βτ
dξij (τ, u).
G(i, u) = g(i, u)
β
j=1 0
• Effective discount factors: The author defines
Z ∞
n
X
mij (u) =
e−βτ dξij (τ, u),
mij (u) < 1,
0
j=1
which plays the role of α pij (u) in the discrete-time case.
(Intuitively, mij (u) is the joint probability of transiting to j & surviving discounting.)
• Bellman equation (semi-Markov):
n
i
h
X
J ∗ (i) = min G(i, u) +
mij (u) J ∗ (j) ,
u∈U (i)
i = 1, . . . , n.
j=1
• Value iteration:
n
h
i
X
Jk+1 (i) = min G(i, u) +
mij (u) Jk (j) .
u∈U (i)
j=1
• Variations:
1. Instantaneous cost H(i, u) at decision time adds directly:
X
J ∗ (i) = min H(i, u) + G(i, u) +
mij (u)J ∗ (j) .
u∈U (i)
j
2. Stage-cost depending on next state j:
n Z ∞
X
1 − e−βτ
G(i, u) =
g(i, u, j)
dξij (τ, u).
β
j=1 0
3. Deterministic integer delays d(i, u, j) yield
X
J ∗ (i) = min
pij (u) g(i, u, j) + α d(i,u,j) J ∗ (j) .
u
j
8
2
Asynchronous Distributed Value Iteration
The author then generalizes VI to settings where updates occur non-simultaneously
across states or processors:
• Partitioned state-space: States {1, . . . , n} are split into subsets I1 , . . . , Im ,
each handled by processor ℓ. The usual synchronous update for discounted
costs is
Jℓt+1 (i) = min
u∈U (i)
m X
X
pij (u) g(i, u, j) + α Jst (j) .
s=1 j∈Is
• Asynchronous updates: Processor ℓ updates only at times t ∈ Rℓ , and
τ (t)
may use delayed values Js ℓs (j) from other processors:
Jℓt+1 (i) =

m X
X

 min
pij (u) g(i, u, j) + α J τℓs (t) (j)
u∈U (i)
s
t ∈ Rℓ ,
s=1 j∈Is

 t
Jℓ (i)
t∈
/ Rℓ .
• One-state-at-a-time (Gauss–Seidel style): Let i0 , i1 , . . . be a sequence (e.g. from simulation). Then update only state it at iteration
t:
(
T J t (ℓ) ℓ = it ,
t+1
Jℓ =
Jℓt
ℓ ̸= it .
• Convergence conditions: Under mild assumptions
1. Each processor updates infinitely often: |Rℓ | = ∞,
2. Delays grow unbounded: τℓs (t) → ∞ as t → ∞,
the asynchronous VI iterates converge to the true J ∗ from any initialization. The author’s and Bertsekas–Tsitsiklis’s theory provides the rigorous
underpinning.
=====================================
3
Policy Iteration (PI)
Policy Iteration alternates between policy evaluation and policy improvement,
starting from an initial stationary policy µ0 . The author notes its close relation
to the rollout algorithm (Section 2.4), but with repeated exact evaluations.
9
3.1
Exact PI for SSP and Discounted Problems
1. Policy Evaluation: Given µk , solve the linear Bellman system for Jµk .
• SSP:
n
X
Jµk (i) = pit µk (i) g i, µk (i), t +
pij µk (i) g(i, µk (i), j)+Jµk (j) .
j=1
• Discounted:
Jµk (i) =
n
X
pij µk (i) g(i, µk (i), j) + α Jµk (j) .
j=1
2. Policy Improvement: For each i, choose
(
P
pit (u) g(i, u, t) + j pij (u) g(i, u, j) + Jµk (j) , SSP,
k+1
µ
(i) ∈ arg min P
u∈U (i)
discounted.
j pij (u) g(i, u, j) + α Jµk (j) ,
3. Termination: Stop if Jµk+1 = Jµk (or equivalently the policy does not
change).
4. Convergence (Prop. 4.6.1):
Jµk+1 (i) ≤ Jµk (i) ∀i, k,
and finite termination with an optimal policy is guaranteed.
3.2
Examples
• Treasure Hunting (fast PI):
– States i = 0, . . . , n = treasures left; choose search or stop.
– Reward r(i), cost c per day. Bellman:
i
n
o
X
J ∗ (i) = max 0, r(i) − c +
p(m | i) J ∗ (i − m) .
m=0
– PI from “never search” converges in two iterations to the threshold
policy r(i) > c.
• Deterministic Shortest-Path (slow PI):
– Line graph 1 ↔ · · · ↔ n → 0.
– Initial policy moves right always; optimal moves left.
– PI corrects one state per iteration, requiring n iterations.
10
3.3
Optimistic and Multistep PI
• Optimistic PI:
1. Improvement: µk from one-step lookahead on Jk .
2. Approximate Evaluation: mk VI sweeps under µk :
X
Jˆk,m+1 (i) =
pij µk (i) g(i, µk (i), j) + α Jˆk,m (j) ,
j
starting from Jˆk,0 = Jk , then set Jk+1 = Jˆk,mk .
3. Convergence (Prop. 4.6.2): Jk → J ∗ and eventually µk optimal.
• Multistep Lookahead PI:
1. Exact evaluation of Jµk .
2. Solve an ℓ-stage DP with terminal cost Jµk , yielding policies (µ̂0 , . . . , µ̂ℓ−1 ).
3. Set µk+1 = µ̂0 .
4. Monotonic improvement and finite convergence as in exact PI.
3.4
PI via Q-Factors
Define
Qµ (i, u) =
n
X
pij (u) g(i, u, j) + α Qµ j, µ(j) ,
j=1
then PI proceeds by
Evaluate: Qµk (i, u) ∀i, u,
Improve: µk+1 (i) ∈ arg min Qµk (i, u).
u
This is algebraically equivalent to the standard PI but directly yields actionvalue estimates.
===============================
4
Bellman Operators and Monotonicity
Define for discounted DP
(T J)(i) = min
u∈U (i)
(Tµ J)(i) =
n
X
n
X
pij (u) g(i, u, j) + α J(j) ,
j=1
pij µ(i) g(i, µ(i), j) + α J(j) .
j=1
Both satisfy the monotonicity property
J ≥ J′
=⇒
T J ≥ T J ′ , Tµ J ≥ Tµ J ′ .
Moreover, adding a constant C to J shifts T J and Tµ J by α C (“constant-shift”).
11
5
Convergence of Value Iteration (Prop. 4.2.1 /
4.3.1)
• Key idea: Under SSP or discounted assumptions, the “tail cost” beyond
a horizon of length mK decays geometrically (ρ < 1). Thus finite-horizon
DP limits to infinite-horizon Bellman equation.
• Sketch:
1. Break Jπ (x0 ) into cost over first mK stages plus remainder.
2. Show remainder ≤ C ρK /(1 − ρ) → 0.
3. Compare mK-stage optimal cost with JmK (x0 ) from VI to bound
|JmK − J ∗ | ≤ ϵK → 0.
4. Extend to all iterates Jk by interleaving the m phases.
• Conclusion: Jk → J ∗ pointwise, and J ∗ is finite.
6
Bellman Equation and Uniqueness (Props. 4.2.2, 4.3.2)
• By taking limk→∞ in VI: J ∗ = T J ∗ .
• If any J satisfies J = T G then running VI from J leaves it fixed, so by
convergence J = J ∗ (uniqueness).
7
Contraction of Bellman Operator (Prop. 4.2.5
/ 4.3.5)
• Weighted
norm: ∥J∥v = maxi |J(i)|/v(i), with v(i) > 0 chosen so that
P
p
(µ)
v(j)
≤ ρ v(i) for all µ, ρ < 1.
ij
j
• Proof sketch:
|(Tµ J)(i) − (Tµ J ′ )(i)| ≤
X
pij (µ) |J(j) − J ′ (j)| ≤ ρ v(i) ∥J − J ′ ∥v .
j
Taking min over u yields same for T , so ∥T J − T J ′ ∥ ≤ ρ∥J − J ′ ∥.
8
Convergence of Exact Policy Iteration (Prop. 4.6.1)
• PI alternates:
improve
eval
J µk −−−−−→ µk+1 −−→ J µk+1 .
• Monotonicity: One-step VI under µk+1 from J µk yields a nonincreasing
sequence converging to J µk+1 , so J µk ≥ J µk+1 .
• Finite termination: Only finitely many stationary policies exist, so eventually µk+1 = µk , which by Bellman uniqueness implies optimality.
12
9
Convergence of Optimistic Policy Iteration (Prop. 4.6.2)
• Optimistic PI: At each k, do one-step improvement on Jk , then perform
mk VI sweeps under that policy.
• Bounding argument:
J ∗ ≤ Jk + (shift) ≤ T k (J0 + shift),
and T k (J0 + shift) → J ∗ by VI contraction.
• Result: Jk → J ∗ and for large k the policy µk satisfies Tµk J ∗ = T J ∗ ,
hence is optimal.
13