Proc. Indian Acad. Sci. (Math. SO.), Vol. 106,No. 3, August 1996,pp. 289-300.
t~ Printed in India
Distributed computation of fixed points of ~ - n o n e x p a n s i v e maps
VIVEK S BORKAR
Department of Computer Science and Automation, Indian Institute of Science,
Bangalore 560012, India
MS received 7 August 1995;revised 3 May 1996
Absln~L The distributed implementationof an algorithm for computing fixed points of an
oo-nonexpansivemap is shownto convergeto the set offixedpointsunderverygeneralconditions.
KeywordL Distributed algorithm;fixedpoint computation; oo-nonexpansivemap; tapering
stepsize;controlled Markov chains.
I. Introduction
Many problems in optimization can be cast as problems of finding a fixed point of
a map F : R d ~ R ~ which is nonexpansive with respect to the oo-norm, or a weighted
version thereof. Examples are algorithms for solving dynamic programming equations
for shortest path problems and Markov decision processes, certain network flow
problems, etc. A standard approach then is to use the iteration
x , +1 = F ( x . )
or its 'relaxed' version
xn+ 1 = (1 - a)xn + aF(x~),
a~(0, 1).
(1)
A comprehensive account of synchronous and asynchronous implementations of these
algorithms and their applications to optimization and numerical analysis appears in
[11 along with an extensive bibliography. A continuous time analog is studied in [4].
This work considers a distributed implementation of a variant of (1). We replace 'a'
by a tapering stepsize {a(n)} c (0, 1) as in stochastic approximation theory, satisfying
for some re(O, 1),
1 >~ a(n + 1)/a(n) --, 1,
~, a(n) = oo,
n
~_, a(n) ~+~ <
oc
(2)
PI
for q >/r. An example is a ( n ) = (n + 2)-1, r = (say) 0-5. Our model of distributed
computation is as follows: We postulate a set-valued random process which selects
at each time a subset of { 1. . . . . d} indicating the indices of the components which
are to be updated. The update uses delayed information regarding other components
with the delays being required to satisfy a mild conditional moment bound. We obtain
a very general convergence theorem under these conditions (Theorem 1.1 below)
which does not require F to be a contraction or a pseudocontraction as in the earlier
literature.
Tapering stepsize as a means for suppressing the effects of delays was proposed in [3]
in the case of algorithms whose continuous limits are differential equations admitting
strict Liapunov functions. Equation (1) need not satisfy this condition. Furthermore,
the model of distributed computing is more elaborate than in the above work.
289
290
Vivek S Borkar
The paper is organized as follows. The remainder of this section formulates the
algorithm and states the main result. The next section studies an associated o.d.e, which
this algorithm tracks in the limit. The third section proves the main result using this
'o.d.e. limit'. The last section gives some examples if nonexpansive maps arising in
numerical analysis and optimization. A forthcoming companion paper studies the
asynchronous version of this algorithm (i.e., one without a 'global clock').
Introduce the norms
Ix;Ip}
=
,
1 ~<p< ~ ,
Ixll~ = max Ixil.
i
for x = [x~ .... , xd]. Let F:Ra--. R d be oo-nonexpansive, i.e.,
I l F ( x ) - f(y)ll~ <<.] : x - yl]oo,
x, Ye R~.
In particular, it is Lipschitz. Let G = {xIF(x) = x} denote the (closed) set of fixed points
of F, assumed nonempty.
Let I = { 1, 2 . . . . . d} and S a collection of nonempty subsets o f / t h a t cover I. Let { Y, }
be an S-valued process and for each n, z~j(n), i r
random variables (delays) taking
values {0, 1..... n}. We set z,(n) = 0 Vi, n. The distributed version of(l) we consider is as
follows: Given X(0), compute X ( n ) = [X 1(n)..... Xd(n)] iteratively by
Xi(n + 1) = Xi(n ) + a(n)[Fi(X 1(n - vii (n)) . . . . . Xa(n - Tin(n)))
-- X,(n)] I { i c Y n }
(3)
for iel, n >~O. Let . ~ n = a ( X ( m ) , Y(m), m <~n, ~ii(m), m < n , i , j e I ) and ~ , , = a ( X ( m ) ,
Y(m), "rij(m), m <~n, i,jeI), n >~O. We shall be making the following key assumptions:
(A1) There exists a fi > 0 such that the following holds. For any A, B e S , the quantity
P(Y.+ 1 = B / Y . = A, fgn)
(4)
is either always zero, a.s., or always exceeds 6, a.s. That is, having picked A at time n,
picking B at the next instant is either improbable or probable with a minimum
conditional probability of 3, regardless of n and the 'history' f~..
Furthermore, if we draw a directed graph with node set S and an edge from A to
B when (4) exceeds di a.s., then this graph is irreducible, i.e., there is a directed path from
any A e S to any BeS.
(A2) There exist b >1r/(1 - r) and C > 0 such that
E[(zlj(n))b/.~,,] <~C
a.s.
Vi,j,n.
(A3) If fi = the integer part ofa(n)'- ~, then ~ is o(n) and moreover, lim supn_~o a(n - h)/
a(n) < c~. (Note that this condition is satisfied by our example a(n) = (n + 2)- 1 with
r = 0"5.)
The condition z,(n) = 0 implies that for updating the ith component, its most recent
value is immediately available. The idea is that each component is computed by
a specific processor, and the different processors communicate with each other in
conformity with (A2). We make the following immediate observation for later use.
Fixed point computation
291
Let zeG. If C. = max.,.<. JiX(n) - z,l~, then by (3) and ce-nonexpansivity F, {C.} is
a nonincreasing sequence. Thus
s u p IIX(n) - zll ~o ~< I I S ( 0 ) - z [ l ~ ,
Our main result is the following:
Theorem 1.1. X (n) ~ G a.s.
The proof will use the fact that (3) asymptotically tracks an o.d.e. The next section
studies this o.d.e.
2. An associated o.d.e.
We start with some notation. For A e S , define Fa(.) = [F~(.) .... Faa(')]: Rd--* R a by
F'~(x) = Fj(x)I {yeA} + x f l {j~A}
for x = [ x t , . . . , x ~ ] . For any Polish space X, let ~ ( X ) denote the Polish space of
probability measures on X with the Prohorov topology. In particular, .~(S) is the space
of probability vectors on S. For # e ~ ( S ) , define Fu: R d --, R d by
FU(.) = ~ # ( A ) F A ( . ) .
Clearly, F A, F ~' are oo-nonexpansive. Given cc> O, say that # e ~ ( S ) is a-thick if
mina#(A) i> a and thick if it is ~-thick for some a > O. The following is easily proved.
Lemma 2.1. G is precisely the set of fixed points o f F ~ for thick # and the intersection of
the sets of fixed points o f F A, A e S (resp., F ~, ile~(S)).
Let U denote the space of.~(S)-valued trajectories fi = {#, t/> 0} with the coarsest
T
topology that renders continuous the maps #-...~ So
f ( t ) # f ( A ) d t for T~>0, A e S ,
f e L 2 [0, T]. U then is compact metrizable. Let us say that fie U is ~t-thick for a given
> 0 if #t is for a.e. t, where the 'a.e.' may be dropped by taking an appropriate version.
Say that it is thick if it is 0t-thick for some ~t > 0.
Lemma 2.2. Let fi" -* fib in U and fi" is a-thick, ~ > O, for n = 1, 2 ..... Then fW is ~-thick.
Proof. For any A e S , t > s >~O, n >~ 1,
f ' #~r(A)dy >~ct(t-- s).
The inequality is preserved in the limit n ~ oo. The rest is easy.
[]
Given fie U, consider the o.d.e.
2(0 = FU'(x(t)) -- x(t), x(O) = x.
(5)
Lemma 2.3. The map (fi, x ) e U x Rd ~ x(')eC([O, oc); R d) defined by (5) is continuous.
Proof. Let (fi",x,)--,(fi%x~). For n/> 1, let x"(') satisfy
.~"(t) = FU,(x"(t)) - x"(t), x"(O) = x,.
292
Vivek S Borkar
Using the Gronwall lemma and Arzela-Ascoli theorem, one verifies that {xn(')} is
relatively compact in C ( [0, ~); Rd). By dropping to a subsequence if necessary, suppose
that xn(.)--.x~
Then x~(0) = x~. Also, for t/> 0, n >/1,
x~(t)=x,+f'o(F~;(x~(s))-F"7(x~(s)))ds
j"
+fto(F#,(x~176176176176
(F"~(x ~ (s)) - F"2 (x ~ (s)))ds
+
0
(6)
The first and the fourth integral go to zero as n -~ oc because xn(.) ~ x ~ (.). (We use the
oo-nonexpansivity of F" in the former case.) So does the second integral in view of our
topology on U. Thus x~(') satisfies
~ ( t ) = F',~ ( x ~ 1 7 6
x~(t), x~(O) = x ~ .
The claim follows.
[]
The main result of this section is the following:
Tl~orem 2.1. Given a thick fie U, the solution x(.) of (5) converges to a point in G which
may depend on the initial condition. Furthermore, t -~ IIx(t) - x* II| is nonincreasing for
any x*~G.
The proof will closely follow that of Theorem 3.1 [4], except for the additional
complications caused by the fact that (5) is nonautonomous. We split the proof into
several lemmas. Let x * e G and/2 be ~t-thick for ct > 0.
Lemma 2.4. The map t--* '~lx(O - x* IFo~is nonincreasimj and hence converges to some a > O.
Proof. A straightforward computation shows that for pc(l, oo)
d
dt [Ix(t) - x* lip = - Ilx(t) - x* lip + [Ix(t) - x* 1191-Pg(t),
where
t4 i=1
~< IIx(t) - x , lipp-a
IlF~'(x(t))-F"'(x*)llv.
Thus for t > s,
,lx(t)-x*llr<~,lx(s)-x*l,+fts(-llx(y)-x*]i,
+ IIFU'(x(y)) - FU'(x *) IIp)dy.
Let p ~ ~ to obtain
,[x(t)-x*ll~<~llx(s)-x*,loo+fts(-,lx(y)-x*,,~o
+ IIF~'(x(Y)) - F~'(x *) II~)dy.
(7)
F i x e d point computation
The claim follows in view of the oo-nonexpansivity of F v.
293
[]
I f a = 0, we are done. Suppose a > 0. Define Ba = {xeRd[ [Ix -- x* I[o~ = a}. Introduce
the terminology: An m-face for m ~< d is a set of the type
{x = [x 1 . . . . . x d] Ix~ ~Ea~., b~.], k <~ m, x~ = c,., k > m},
where {i~ . . . . . ia } is a permutation of { 1.... , d} and b k > a k, c k are scalars. Then Ba is the
union of(d - 1)-faces of the type {xlx~ - x* = a (or - a) and Ixj - x~l < a f o r j :# i}. F o r
any (d - 1)-face of this type (say, H), define G n = { x e H l F ( x ) e H } . Then G u is closed,
possibly empty. Since [Ix(t) - x* li ~ --* a, x(t) ~ Bo. N o w the trajectories x ( t + .), t >1 O,
form a relatively compact set in C([0, ~ ) ; Ra). Thus any limit point ~(.) thereof as t ~
must lie in B,. By L e m m a s 2.2 and 2.3, .~(.) satisfies
~(t) = F ~ ( ~ ( t ) ) - ~(t)
for some ~t-thick/~eU. Let {~(.)} = { ~ ( t ) l t e R + } .
Lemma 2.5. { ~ ( - ) } c ~ H c G n.
Proof. If both sets are empty, there is nothing to prove. Suppose {~(-)} c~H ~ 4- F o r
simplicity, let n = {xlx~ - x* = a, lx i - x*[ ~ a,i > 1}. Suppose {~(t)lt~l-0,A-I} c n .
Then Yq (t) = x* + a, te[O, A], leading to 0 = xl (t) = F~'(~(t)) - xl (t). Thus F~t(~(t)) =
~ ( t ) = x * + a = F l ( ~ ( t ) ) in view of ~t-thickness of/~, for t e [ 0 , A ] . Since F is oo-
nonexpansive and x* is a fixed point of it, we also have
l,F(~(t)) - x* ]l~ ~ [I~7(t) - x* II ~ = a.
Thus we must have F ( g ( t ) ) e H , t e [ 0 , A ] , implying ~ ( t ) e G u. It follows that all
connected segments of {~(.)} c~ H that contain more than one point must be in G u. On
the other hand, those containing a single point must clearly be in the relative b o u n d a r y
OH of H, which is the union of its (d - 2)-faces. Let xe{~(')} c~ c~H. It suffices to show
that F ( x ) e d H . If not, F ( x ) - x
and therefore F * ( x ) - x for any thick ~ would be
transversal to c~H at x. This is not possible because ~(.) is a differentiable trajectory
confined to B a and cannot make 'sharp turns' around corners. This completes the proof.
[]
Consider a fixed (d - 1)-face H o f B a for the time being. Let H = { x l x 1 = x* + al,I
xj - x*l <-<a,j > 1 } for simplicity.
L e m m a 2.6. I f Gu q: qS, the map F :G u -~ H can be extended to an oo-nonexpansive map
F:H--* H which has a f i x e d point s
Proof. The second claim follows from the first by the Brouwer fixed point theorem. Fix
1 < i ~< d and define
gi(x) = inf (Fi(y) + [Ix - yll ~), x ~ n .
yeGB
Then gi(x) <~ Fi(x), x e G u. F o r x, y e G n, ~ - n o n e x p a n s i v i t y of F leads to
Fi(y ) + lix - y I[~ >~ Fi(x ).
294
Vivek S Borkar
Thus 9i(x) >1 Fi(x ), implying 9i = Fi on GH" N o w for x, z e H ,
9~(x) <~ inf (F~(y) + ]IY - z]] ~ + Ilz - xl] ~)
y~Gn
<~adz) + liz - xll~o.
(8)
Similarly, 91(z) ~< Oi(x) + [Iz - x I~. Hence
Igdx) - 0r
-< IIz - x l ~.
Let Fi(x) = max(x* - a, min(0i(x), x* + a)). Then
IFi(x) - Fi(z)I ~< Ilx - z l ' ~ .
Let/~l(x) = x* + a, x E H . Then F(.) = [ F I ( ' ) . . . . .
/Ta(') ] is
the desired map.
[]
Proof of Theorem 2.1. The same a r g u m e n t as in L e m m a 2.6 can be used to extend/~ to
an oc-nonexpansive m a p if: R a ~ R a that restricts to ff on H and to F on u A G a. Define
F A, if" in analogy with F A, F ~ using F in place of F. Then 2(.) satisfies
x(t) = F~,(~(t)) -
~(t).
We conclude that t --, II~(t) - 211~ is nonincreasing in t and hence decreases to b >/0. If
b = 0, we are done. If not, let B b = {x] l]x - 2 [I~o = b}. Then if(t) --* B b. Also, it is clear that
no (d - 1)-face H of B b is c o p l a n a r with H. This a r g u m e n t can n o w be repeated for each
( d - D - f a c e H of Bo that intersects {2(.)}, leading to possibly m o r e I!.l!~-spheres
Br n .... defined analogously to B, such that 2(t)--.B~c~B~c~Br
.... T h e a b o v e
remarks also imply that this intersection is a union of m-faces with m at most (d - 2).
N o w consider a limit point x'(') ofY(t + .), t t> 0, in C([-0, 0o); R a) as t--* 0o. Repeat the
a b o v e a r g u m e n t to conclude that x'(t) converges to a union of m-faces with m at most
(d - 3). Iterating this a r g u m e n t at most d times, we are left with a union offinitely m a n y
points to one of which 2('), x'(.).., and hence x(.) must converge and which then must
be a fixed point of F ~ for some thick/~, hence of F. This completes the proof.
[]
C O R O L L A R Y 2.1
Given ~, b > O, there exist T = T(e, b) > O, rl = rl(e, b) > 0 such that for any solution x(') of
(5) satisfyin# I I X ( 0 ) l l ~ <- K < oo and
(i) fi is b-thick,
(ii) {x(t)lte[O, T ] } n G ~ = ch where G ' = {xlinfr~61 x -Y~l < e } , we have
infllx(t)-yll~infllx(O)-yll~-~/
~G
for
t~>T.
~G
Proof. Suppose that the claim is false. Then there exist x " e R d, fi"e U such that for some
b > 0 , {fi"} are b-thick and x"(.) satisfy (i) ~"(t)=FUT(x"(t))--x*(t), x " ( 0 ) = x ~,
Ix"[I ~ ~< K, (ii) x"(t)ddG ~ for te[O,n] and
infll x~ - yll~o >/ sup infb[x"(t)-y,[~ >1 inf I'x~ - y l ] ~ - 1/n,
~n~G
t~[O,n] y~G
~G
for n 1> 1. By d r o p p i n g to a subsequence if necessary, we m a y then suppose that
Fixed point computation
295
x" --, x ~ r G ', ~" ~ / ~ in U and x"(') ~ x ~ (.) in C( [0, ~ ) ; Rn). By L e m m a s 2.2 and 2.3,/]~
is b-thick and x~
satisfies.
.~oo(t) = F . ? (x~~
- xOO(t), x~(O) = x ~.
Also, infy~ I!x ~ - y IIoo = inf~o fix ~ (t) - y il~, t ~> 0. This contradicts T h e o r e m 2.1, establishing the claim.
[]
F o r ~, q, T > 0, call a trajectory y(.): R + --* R ~ an (~, r/, T)-perturbation of (5) if there
exist 0 = T O < T l < T 2 < . . . in [0, oc) with Tj+ 1 - Tj/> T, such that for some xJ(')
satisfying (5) for some ~-thick/7 ~ in place of/~, we have
sup
fly(t) - xJ(t)l] oo < rl,j >~O.
te[75,Ti+ 2]
C O R O L L A R Y 2.2
F o r any ~, e. > 0, T > 0 as in Corollary 2.1 and 7 > 0 sufficiently small, any (:q 7, T)perturbation y(-) of (5) converges to G '.
Proof In view of Corollary 2.1, this is a straightforward a d a p t a t i o n of T h e o r e m 1, p.
339 of [5].
[]
3. P r o o f of T h e o r e m I.I
As a first step towards establishing that {X(n)} tracks (5) asymptotically, we analyze the
S-valued process { Y, }. In particular, we shall show that it m a y be viewed as a controlled
M a r k o v chain.
F o r AeS, let D A = {BeS[ (4) exceeds 6, a.s.} and VA = {u~(DA)Iu(B) >>.6VB~DA}.
Let V = F I A VA. Define p:S • S x V~]-0, 1] by
p(A. B,/.t) = #a(B),
where/aa is the Ath c o m p o n e n t of/a. Define V-valued r a n d o m variables {Z"} as follows.
The Ath c o m p o n e n t of Z", denoted by Z~t is given by
Z] (B) = P( Y.+ , = B/(9.)I { V. = A)+ 7?AI { Y. r A},
where h~A are fixed elements of VA, A~S. Then (4) equals p(A,B,Z") and {Y.) m a y be
viewed as an S-valued controlled M a r k o v chain with action space V and transition
probability function p. It should be kept in mind, however, that this is purely a technical
convenience and it is in no way implied that {Z"} is actually a control process. In
particular, this allows us to conceive of a 'stationary control policy n' associated with
a m a p ~: S--* V wherein Z" = ~z(Y.), n ~> 0. The latter part of(A1) implies that {Y.} will
be an ergodic M a r k o v chain under any stationary policy zr with a corresponding
unique stationary distribution v.e~(S).
Let t o = 0, t. = Z~, = 1a(m), n i> 1. Define y('): [0, ~ ) ~ S by
y ( t ) = Y.,t.<~t <t.+1,n>~O.
Define/~E U by/a, (A) = I {y(t) = A }, t >i 0 and/2se U, s ~> 0, by /at" - #~+~, t~>0.
296
Vivek S Borkar
Lemma 3.1. There exists an ot > 0 such that any limit point of {/2'} as s ~ oc is a-thick, a.s.
Proof. Let AeS. Then
M.=
~ a(m)[I{Y==A}-- ~.,p(B,A,Z"-x)I{Y,,_x=B}-I
m=l
B
is a zero mean bounded increment martingale with respect to {(#,}, whose quadratic
variation process converges a.s. in view of (2). By Proposition VII-2-3 (c), pp. 149-150,
[6-1, {Mn} converges a.s. For n/> 0, let h(s) = min{m > nl Z~=n+ ta(j) >/s}, s > 0. Then
lim._.~(M~,) - M.) = 0 a.s. and Z ~ . a ( m ) t> s together imply
~-f'2~na(m)l{Y==A} - ~ ' ~ " a ( m ) ~ ' s P ( B ' A ' Z " - t ) I { Y ~ - I = B }
.0
(9)
.atm)
a.s. Define O,,,e~(S x V) by
r
• J)=
~ _ ~ .a(m) I { r., e C, Z " e,l }
~,,a(m)
for C c S, J c V Borel. Then from (9), it follows that a.s., any limit point O of O,,, in
~(S x V) as n ~ oo must satisfy
O({A} x V)=
f ,A,
)dO, AeS.
Thus 9 must be of the form
O({A} x J) = v(A)(p.4(J), AeS, d ~ V Borel,
where A -~ (pa:S ~ ( V ) defines a stationary randomized policy and v the corresponding stationary distribution (see e.g. [2], pp. 55-56). By Lemma 1.2, p. 56 and Lemma 2.1,
p. 60 of [2-1, it follows that
rainy(A) >/rain v,(A) & ct > 0.
A,x
A
Thus
liminf ~ " a ( m ) I { Y~ = A}
>/~ a.s.
,~ ~
~.,~2,,a(m)
From our definition of {/~t}, it then follows that
liminf 1 I ' dY#ty(A)>1eta.s.VAeS.
t--* oo
SJo
Fix a sample point outside the zero probability set on which this claim fails for any
AeS, s > 0 rational. For this sample point, the claim follows easily in view of our
topology on U.
[]
Now rewrite (3) as "
X(n + 1) = X(n) + a(n)(W(n) - X(n))
for appropriately defined W(n)= [Wl(n) ..... Wd(n)] and set i f ( n ) = E [ W ( n ) / ~ , ] ,
n/> 0, the conditioning being componentwise. Write if(n) = [Wl(n) ..... ia(n)].
Fixed point computation
297
Lemma 3.2. There exist K > 0, N ~> 1 such that for n >I N,
IIF~"(X(n))- if(n)I,~ < Ka(n)'.
Proof. W.l.o.g., let X(0) be deterministic. Then VA c I, zeG,
IIF A(X(n))II ~ ~< IIX(n) - z II~ + [IF(z)II o~ V IPzll
~< 3[Iz[l| + lIF(z)ll~ + ~lX(O) ![~o A M <
~.
F o r each i, 1 ~< i ~ d, and c = 1 - r,
[Fr'(X (n)) -- l~i(n)[ ~< E [ [Fr~'(X(n))- W~(n)[I {xq(n) ~< a(n)-C for all i,j}/.~,,]
+ E [ [ F r ' ( ( X ( n ) ) - W~(n)[I{zij(n ) > a(n) -~ for some i,j}/.~,].
The second term is bounded by 2MCa(n) ~ in view of (A2) and the conditional
Chebyshev inequality. Let fi be the integer part of a(n) -~. Since a(n) -~ is o(n), we may
pick n large enough so that n > ~. Then for m ~< t~, (A3) leads to
IX(n) - X ( n - m)ii ~ ~ 2 M
~
a(k) <. I~a(n) 1-~
k=n-t~
for a s u i t a b l e / ( > 0. Thus the first term is b o u n d e d by K,a(n)'. Since b >>.r/(1 - r), the
claim follows.
[]
Let T > 0 . Define T o = O , T , = m i n { t , , I t m > > . T , _ l + T }, n~> 1. Then T , = t,,<,) for
a strictly increasing sequence {re(n)}. Let I, = [T,, 7".+ 1], n/> 0. Define ~"(t), t e l , , by
Y~"(T,) = X(m(n)) and
x"(tm~.~+k. 1) = ~"(t.,~.~+ ~) + F r ...... (~"(t,.~.)+~))
-
Y~'(t,.~.)+k))(t.,r §
1 - t,.r
with linear interpolation on each interval It,.<.) + k, t,.~.~+ k + ~]" Define x(t), t >1O, by
x(t.) = X(n) with linear interpolation on each interval [t., t. +~].
Lemma 3.3.
lim sup Ilx(t) - 2"(t)[I ~o = 0 a.s.
n ~
oo
tfivJ'~
Proof. Let n t> 1. F o r i >>.re(n), we have
x(t,+ 1) = x(t,) + a(i)(Fr'(x(ti)) - x(ti)) + a(i)(lTV(i) -
F~'(x(t,)))
+ a(i)(W(i) - if(i)).
Let l~l m = Y~=oa(i)(W(i)- l?V(i)) and ~ = M i - h4,~,) for i>~ re(n). Then {~rm, ~-m} is
a zero mean bounded increment vector martingale and the quadratic variation process
of each of its c o m p o n e n t martingales converges a.s. by virtue of (2). Thus by Proposition VII-2-3(c), pp. 149-150 of [6], {J~m} converges a.s. Fix a sample point for which
this convergence holds and let 6 > 0. Then
sup ~[~i!1~ < ~5/2
i >ire(n)
for n sufficiently large.
298
Vivek S Borkar
Let .,2i + 1 = x(t i + ~) - r i >>.m(n) with 2,,~,) = X,,{.)(i.e., ~,,~,~_ 1 _a__0). T h e n for i >~ re(n),
-~i + 1 = 2i + a(i)(Fr'(2i) - 2"i) + a(i)(Fr'(s + ~i- 1) - (s + ~i- l )
- FY'(s
+ .~) + a(i)(17V(i)- FV'(x(ti)).
Also
s
~) = s
+ a(i)(Fr'(s
- s
S u b t r a c t i n g a n d using the p r e c e d i n g l e m m a , we have, for n sufficiently large,
[I2i* x - 2"(ti + x ) l[o~ ~< (1 + 2a(i)) I 2i - ~"(ti) II~ + 2a(i) II~ - ~ li o~ + Ka(i) l +'.
By increasing n if necessary, we m a y s u p p o s e t h a t
a(i) 1 +" < 6/2.
i>~n
T h e n using the inequality 1 + 2a(i) ~< exp(2a(i)) a n d iterating, we have for n sufficiently
large,
sup
l[2i-~(ti):lo~<<.2e2Cr+l)(K + T + 1)6.
m(n)<~i<~m(n- 1 )
Also
sup
I1~ - x(t,)II ~ <~ 6/2,
m(nJ<~i~m(n + 1}
for sufficiently large n. Since ~5 > 0 was a r b i t r a r y , the claim follows on o b s e r v i n g that
x('), s
are linearly i n t e r p o l a t e d from their values at {ti}.
[]
N e x t define :~'(t), t e l n , by g"(tm(n~ ) = x(tM~fl and
.~(t) = FYt~
- s
tell.
L e m m a 3.4.
lim sup I ~"(t) -
:~n(t)11 ~ = 0 .
Proof. This follows easily f r o m the G r o n w a l l l e m m a .
[]
Let ~ > 0 be as in L e m m a 3.1.
L e m m a 3.5. Almost surely, the followin9 holds. There exists an or-thick sequence
fi"e U, n >~ O, such that if s
t e l , , is defined by s
= x(t,,tn)) and
.in(t) = F;"(s
-- 2(0, t e l , ,
for n >1 1, then
lim sup IIs
- 2~(t)l[ o~ = O.
Proof. This is i m m e d i a t e f r o m L e m m a s 2.3 a n d 3.1.
[]
P r o o f of T h e o r e m 1.1. Let e > 0. Let b = :~ a b o v e in C o r o l l a r y 2.1 a n d pick T = T(e, ~)
accordingly. Pick "/> 0 as in C o r o l l a r y 2.2. C o m b i n i n g L e m m a s 3.3-3.5, we h a v e
Fixed point computation
lim sup
. ~ ~
299
lid'(t) - x(t)'[ ~ = 0 a.s.
tEln
Thus x(t, + -) is a (ct, T, 7)-perturbation of(5) for sutficiently large n. By Corollary 2.2, it
follows that x(t) ~ G~. Since e > 0 is arbitrary, the claim follows.
[]
Observe that the foregoing can be easily extended to the following relaxation of
the latter half of (A2). The directed graph formed therein need not be irreducible,
but each communicating class in it must correspond to elements of S which together
cover I. Also, extension to nonexpansive F with respect to the weighted w - n o r m
is straighforward.
4. Examples
This section sketches some important instances of fixed point problems for oononexpansive maps. A general reference for these is [1].
(i) Shortest path problems: Given d + l locations {0,1 ..... d} and the distances
{dg~,0 ~< i,j <~d, iv~j} between them, the problem is to find the shortest path from
location i r 0 to location 0. Letting V(i) = the length of the shortest path from i to 0, one
has the dynamic programming equations
V(i)=min(dlo, min(d,j+
V(j))),
l<.i<~d.
j~i.o
Letting V = [V(l) ..... V(d)] T, this has the form V = F(V) for an ~-nonexpansive F.
(ii) Markov decision processes: Consider a controlled Markov chain {X.} on a
finite state space S, with a compact metric action space A and a continuous transition probability function p:S x S x A - * [ 0 , 1 ] . The aim is to choose an A-valued
sequence {Z.} that does not anticipate future, to minimize a suitable total expected
cost. Thus
P( X. +1 = J/X i, Zi, i <~n) = p( X .,j, Z.) Vn.
Let B be a proper subset of S and ae(O, 1). Consider two cost functionals: For
x A),
keC(S
(1) cost up to a first passage time:
E
[
~ k(x.,z.)
_~'=0
]
.
where ~ = min {n/> OIX.eB},
(2) infinite horizon discounted cost:
E[~=oa"k(X.,Z.) ]"
Letting V(i) denote the minimum cost when X o = i, the dynamic programming
equations in the two cases are, respectively,
V(i)= min
u
(k(i,u) + ~ p(i,j,u) V(j)),
j~B
ieS\B,
300
Vivek S Borkar
and
V(i)=
min(k(i,u)+a~p(i,j,u) V(j)),
u
i~S.
j
Both can be cast as fixed point equations V = F (V) for ~ - n o n e x p a n s i v e V.
(iii) Systems of linear equations: The problem of solving a system of linear equations
A x = b can be cast as finding the fixed point of F ( x ) = x - a ( A x - b ) , a e ( O , 1). If
[11 - AII ~ ~< 1, F is ~ - n o n e x p a n s i v e .
(iv) Strictly convex network flow problems: The following problem arises in network
flow optimization:
minimize
ao(fi~)
~
(i,j)EA
subject to
~, f i j {~(i,j'~A}
~, fj,=S, ViEN, b,j<~fo<~coY(i,j)~A,
{jI(.],i)EA}
where alj(- ) are strictly convex. This problem can be cast as that of finding a fixed point
of a pseudononexpansive map, i.e., a m a p F satisfying liE(x) - y II~ ~< IIx - Yll ~ whenever y is a fixed point of F. (See (1), w7.2 for details). O u r analysis applies here as well.
Acknowledgements
This research was supported by a grant from the I S R O - I I S c Space T e c h n o l o g y Cell.
The a u t h o r would like to thank Prof. Vinod S h a r m a for pointing out an error in the
preliminary version of this paper.
References
[1] Bertsekas D P and Tsitsiklis J N, Parallelanddistributedcomputation(Prentice Hall, Englewood Cliffs)
(1989)
1-2] Borkar V S, Topics in Controlled Markov Chains, Pitman Research Notes in Maths, No. 240, Longman
Scientific and Technical, H arlow ( 1991)
I-3] Borkar V S and Phansalkar V V, Managing interprocessor delays in distributed recursive algorithms,
Sadhana19 (1994) 995-1003
[4] Borkar V S and Soumyanath K, A new analog parallel scheme for fixed point computation, Part I:
Theory, preprint (1993)
[5] Hirsch M W, Convergent activation dynamics in continuous time networks, NeuralNetworks2 (1987)
331-349
[6] Neveu J, DiscreteParameterMartinoales(North Holland: Amsterdam) (1975)