Proceedings of the 42nd IEEE
Conferenceon Decision and Control
Maui, Hawaii USA, December2003
ThP04-6
Characterization of the Optimal Disturbance Attenuation
for Nonlinear Stochastic Uncertain Systems
Charalambos D. Charalambous, t Farzad Rezaei and ISeddik M. Djouadi
+School of Information Technology and Engineering
University of Ottawa
Ottawa, Ontario, KIN 6N5, Canada
E-mail: {chadcha,frezaei} @ site.uottawa.ca
$System Engineering Department, University of Arkansas at Little rock
Little Rock, AR 72204
E-mail: msdjoudi@ualr.edu
Abstract-This paper is concerned with an abstract formulation of stochastic optimal control systems, in which uncertainty
is described by a relative entropy constraint between the
nominal measure and the uncertain measure, while the payoff is a functional of the uncertain measure. This is a minimax
game in which the controller seeks to minimize the pay-off,
while the disturbance described by a set of measures aims
at maximizing the pay-off. When stochastic dynamical systems
are considered, this problem is known to be equivalent to the
optimal disturbanceattenuationproblem arising in Hm control.
The objective of this paper is to derive several properties of the
optimal solution. One such property is the characterization of
the optimal disturbance attenuation, in terms of the nominal
measure and an estimate of the uncertain measure. The characterization is important for computing, as well as comparing,
the solution of sub-optimal disturbance attenuation problems
to the optimal one.
Key Words: Nonlinear Uncertain Stochastic Systems, Large
Deviations, Relative Entropy, Minimax Games, Duality Properties.
I. INTRODUCTION
Subsequent to the publication of Zames [ l ] seminal paper on H" control, several tools have been introduced to
extend the H" techniques to nonlinear deterministic and
stochastic systems. Three pay-off functionals which received
significant attention are deterministic minimax games [3],
risk-sensitivity pay-offs 141, [51, [61, [71, 181, [91, 1101, [ l l l ,
[12], [13], [15], andistochastic minimax games [ll], [12].
For nonlinear problems robustification is captured through
dissipation inequalities [2].
Another class of problems aiming at robustification are
described through relative entropy constraints, using duality
relations between relative entropy and free energy. The problems are equivalent to risk-sensitive problems [18], [17], [15],
and thus equivalent to the minimax game formulation [15]
of the H m disturbance attenuation problem. For linear fully
observable or partially observable problems, the characterization of the optimal disturbance attenuation is expressed in
terms of certain Ricatti equations. Unfortunately, in nonlinear
0-7803-7924-1/03/$17.00 02003 IEEE
problems such Ricatti equations are not available. Therefore,
one considers the sub-optimal problem by either fixing the
structure of the controller or by considering minimax dynamic games that result in strategies which ensure a certain
dissipation inequality with respect to the supply rate (energy
functional) [3].
Ideally, one would like to have an explicit expression which
describes the optimal disturbance attenuation level for general nonlinear fully observed and partially observed (output
feedback) problems. Such an expression will be useful in
determining how far any sub-optimal solution is from the
optimal one.
The main contributions of this paper are the following. First,
several properties associated with the optimal solution are
derived, Second, the optimal disturbance attenuation level is
characterized explicitly through the relative entropy between
the worst possible estimate of the uncertain measure and
the nominal measure. The results are quite general; they
are applicable to both fully observed and partially observed
stochastic uncertain dynamical systems.
The abstract formulation is the following.
Let (E, d) denote a complete separable metric space (Banach
Space), and (E, B ( C ) ) the corresponding measurable space
in which B ( C ) are identified as the Bore1 sets generated
by open sets in E. Let M ( C ) denote the set of probability
,
the set of admissible controls,
measures on ( C , B ( C ) ) Uad
and let L" : C + R be a real-valued measurable function,
bounded from below, which depends on the control U E U a d .
Here, M (E) denotes the set of all possible measures induced
by the stochastic systems, while Lu denotes the energy
function associated with a given choice of the control law
E Uad.
Given a control law U E Uad and a nominal measure
p" E M(E) induced by the nominal stochastic system, the
pay-off is defined by
4260
J ( u ,v") = E,*
(e")
=
J, Pdv"
0.1)
Using duality relations, it is shown that "s*" is the optimal
disturbance attenuation level associated with H m disturbance
{vu E M ( C ) ; H ( v " l p " ) I
R, vu << p " }
attenuation problems. Thus, computing s* corresponds to
where R E (O,oo),1-1" E M ( C ) and H ( v " ( p " ) denotes the computing the optimal disturbance attenuation level. One of
relative entropy between the uncertain measure vu and the the main contributions of this paper is the characterization
nominal measure p" (see Definition 2.1), and << denotes of the optimal disturbance attenuation s*. In particular,
absolute continuity of v" E M ( C ) with respect to p" E the Lagrange multiplier s*, which corresponds to the
optimal disturbance attenuation is characterized through the
M (E).
For a given control law U E Uad, and nominal measure p" E relative entropy between an estimate of the true measure
M ( C ) ,let vu?* E M ( C )denote the measure which achieves vug* E M ( C ) and the nominal measure p" E M ( C ) . Thus,
since s* corresponds to the optimal disturbance attenuation
the supremum in (1.1) defined by
level associated with the H" disturbance attenuation
J ( u ,v " ~ ' )
(1.2)
problem, this characterization provides an explicit formula
for computing as well as comparing the solution of the
SUP
e"dv"
{ U ~ € M ( C ) ; H ( V ~ ~ ~ " ) ~ R , ~ ~ < <C~ ~ , / L " Esub-optimal
M ( C ) } disturbance attenuation problem to the optimal
one.
The robust control problem is to find a control law U* E i?&d
In addition, several monotonicity properties of the dual
which impacts
functional are derived, including a lower and upper bound
J ( u * )= inf J ( u , vu>*)
on the dual functional (ps'(u,R).
UEUad
subject to fidelity
J
= inf
"EUod
(
11. DUALITY RELATIONS AND TILTED
MEASURES
In this section the basic definitions and duality relations
between relative entropy, free energy, and cumulant moment
generating functions are introduced. These quantities are employed in subsequent sections to derive the results discussed
J ewvu>
{~"EM(C);H(V~I~~)~R,V~<<~".~~€M(C)}
SUP
C
(1.3)
k"
M(C)}
The constraint
E M(C);H(v"lp")
5 R, v" <<
p", p" E
describes the set of all admissible in the introduction.
measures induced by the stochastic system, relative to the
nominal measure. The pay-off EuU P represents average
Definition 2.1: For a given U E Uad, let U", p" E M ( C )
energy with respect to the unknown measure v" E M ( C ) , and C" : C -+ R a measurable function.
1) The moment generating function of e" with respect to p"
such as integral quadratic constraints, tracking errors, etc..
The parameter R describes the admissible level of uncertainty is defined by
associated with a family of stochastic systems with respect
to the nominal one. Since H(v"1p") = 0 if and only if
U" = p", the parameter R is always in the interval ( O , o o ) ,
where s E R.
otherwise the trivial solution vu** = p" will maximize
2) The cumulant generating function of e" with respect to
(1.2). Clearly, larger values of R imply larger uncertainties
p" is defined by
measured by the relative entropy. Such a problem formun
lation is also considered in [18] for the case of discreteQJpu(s)
= logMpy(s)=log
ese"dp" E (--oo,oo] (II.6)
time stochastic systems and in [17] for discrete-time linear
systems. However, the aim of this paper is fundamentally where s E R.
different from the results of [18], [17], in both the general 3) The free energy of e" with respect to p" is defined by
conclusions as well as the application considered.
E,U(C") 5 Sp(1) E (-Oo,oo];
Specifically, the current paper is concerned with computing 4) The relative entropy of vu with respect to p" is defined
the optimal strategies and pay-off by using duality relations
by
between the primal problem 0.1)- (1.3) and its dual unconstrained problem. In the dual space, the problem is described
H(u"1p") 5
(D.7)
by the infimum over s 2 0 of the Lagrangian functional,
namely,
U
L
A
pS*( U , R)= inf
s1°{ v " E M ( C ) ;
SUP
vu<<pU, p " E M ( C ) }
{
P d v " - s ( H ( v " J p " )- R ) }
(1.4)
otherwise
It can be shown that EpU(CU)
as a function of C" is convex,
H(v"Ip") as a function of p", U" E M ( C )is convex in both
4261
arguments, H(v"1p") 2 0, and H(v"1p") = 0, if and only
if ,U" = vu, and Alpu(s),Q p u ( s ) are convex functions of
s E 8.
In addition, the following result, is a generalization of a
theorem found in [ 151.
Theorem 2.2: For every measurable function C" : C + 8
bounded below and p" E M ( C ) and s > 0
(
log L e S d p " ) ' =
Next, for every s E X,define the Lagrangian associated with
the problem of Definition 3.1
A
J S . R ( ~ , v u=) E,=(C")- s(H(v"1p") - R)
and its associated dual functional
J37R(U, VU>*)
<</A",puEM (E)}
JS(u, V U ) (111.11 )
SUP
{ v u EM (E); v u < <pU,
SUP
{vu EM (E);v'
(111.10)
pu EM
(E)}
In addition, define the quantity
Pdv" - sH(v"Ip")}
(11.8)
(III.12)
Moreover, if Cue$ E L 1 ( p u ) ,then the supremum is attained
at
Proof. The derivation is &har
which treats the case s = 1.
io the one found in [15],
The above duality relation is important in solving the dual
problem associated with the primal problems stated in the
introduction.
111. PROPERTIES OF DUALFUNCTIONAL
This section is concerned with reformulating the constrained optimization described in the introduction, as an
unconstrained optimization using the theory of Lagrange
functionah. Thus, the dual functional is introduced and
for a fixed U E Uad several properties of the pay-off are
presented. These include the range of values of the Lagrange
multiplier associated with the dual functional, for which the
constrained and unconstrained problems are equivalent. In
addition, the value of the Lagrange multiplier for which
the dual functional is minimized is explicitly characterized.
Because the Lagrange multiplier corresponds to the attenuation level associated with disturbance attenuation problems,
this characterization is important in computing the optimal
solution of disturbance attenuation problems.
The primal problem stated in the introduction is now defined
precisely as follows.
Dejinition 3.1: Let U E
e" measurable and bounded
below, C" E L l ( p " ) , p" E M ( C ) which is a fixed nominal
measure, and R E (0,CO).
Find vu>* E M ( C ) which achieves the supremum in (I.]),
that is,
J ( u , VU(*)
-
SUP
1
C"dv"
which may or may not exist. In general J s ~ R ( ~ , v Uis* * )
not defined throughout the entire real line with respect to
s E 3,and there might be regions in [ O , o o ) for which
P R ( u ,vu>*)= CO.Next, several properties of the dual functional are derived, and the regions for which J37R(u,vu,*)
is finite, are identified. These results are employed in Theorem 3.3, to show equivalence between the unconstrained
functional (In. 12) (dual functional) and the constrained problem of Definition 3.1 (primal problem). In addition, the value
of s* that achieves the infimum of the dual functional is
characterized, and it is shown that s* occurs on the boundary
of the constraint.
Lemma 3.2: For a given U E Uad. 8" a measurable
function bounded from below, and for some 0 < s 5 s* the
following statements hold.
1) The dual functional J S y R (vU3*)
~ , is related to the cumulant generating function of C" with respect to p" E M ( C )
via
J 3 3 ( U , VU?*)= s
1
-
1
s c
SUP
{ v ' € M ( C ) ; uu<<pu,
pUEM(C)}
+
P d v " - H ( v U I p " ) } sR
= s l o g L e CL"
Tdp"
+ s R = s Q p u ( -S1) + sR
{
(III.13)
(III.14)
Moreover, if cue$ E L 1 ( p U ) the supremum in (III.13) is
attained at vu?* E M ( C ) given by
(111.15)
2) The dual functional P R ( u ,vu") is convex in s > 0.
A
3) The function rpU(s)
= s Q p u ( $ ) is a non-increasing
function of s E (0, CO), that is,
{ ~ u E M ( C ) ; H ( ~ U I ~ u ) I R , u Y < < p U , p L " ECM ( C ) }
"9)
where R E (0, m).
4262
where 0 < s2 I SI.
4) The infimum of the dual functional J s % R ( u , ~ " 9over
*)
s > 0 defined by
and if
E L l ( p u " )then
A
@i,(R)= cp"*(u,R)= inf
s>o
is a concave functional of R 2 0.
Moreover, if there exists q > 0 such that Pe@' E L1(p"),
then
a)
q,(o)
=
{e.}
=
Jc e v p u
(III.18)
where 0
b)
lim s l o g k e q d p " = k P ' d p u = (ai,(O)
S'CC
{e.}
5
( R )= cps*(u,R ) 5 R + E,%{e.} (111.20)
7) If e$ E L1(pcL"),
E Ll(pu")and ( l u ) 2 e $ E
L1(p") then the infimum of the functional JStR(u,vu?*)over
s > 0 is uniquely attained at
H ( ~ " ' * l p . " ) l ~=
= ~R*
(nI.21)
Remark 3.4: According to Theorem 3.3, at s = s* the
maximizing measure vu?* E M ( C ) occur on the boundary of
the relative entropy constraints, and satisfies the monotonicity
property (III.25). If the above abstract formulation is applied
to stochastic dynamical systems, then the value of the Lagrange multiplier s* corresponds to the optimal disturbance
attenuation parameter associated with stochastic optimal H"
disturbance attenuation problems.
Using the results of the previous Lemma 3.2, the equivalence between the unconstrained and constrained problems
is established in the next theorem.
Theorem 3.3: For a given U E Uad,C" a measurable
function bounded from below, e? E L1(pcL")
and for some
0 < s 5 s* the following statements hold.
The primal constrained problem of Definition 3.1 and the
unconstrained dual problem are equivalent, that is,
inf
s>o
{
J S ~ R vu,*)}
( ~ ,
= ps*(u,
R)
(III.22)
J,eudv")
{
= (pa* (21,R) = E,= eu}ls=s*
Proof. The derivation is found &-[20].
where vu?*is given by (III.15).
That is, assuming the constrained and unconstrained problems are equal (which is shown next), a necessary condition
for the infimum of the dual functional J s ~ R ( u , ~ "over
~*)
s > 0 is that s* occurs on the boundary of the relative entropy
constraint.
Proof. The derivation is found in [20].
J S * ~ R vu?*)
(~, =
< s* I:s1 5 s2 and the dual functional is given by
JS*'R(~,
(III.19)
6) If there exists r] > 0 such that t"e@y E L 1 ( p u ) ,then the
) s > 0 is
infimum of the dual functional J s , R ( ~ , v " i *over
bounded above and from below as follows.
E,,
Moreover, if in addition Cue!eu E L1(pu),(eu)2ete" E
L l ( p " ) then the infimum over s > 0 in (III.22) is uniquely
attained at s* > 0 given by
(111.23)
4263
IV. REFERENCES
G. Zames, "Feedback and optimal sensitivity: Model
reference transformations, multiplicative seminoms,
and approximate inverses," IEEE Transactions on Automatic Control, vol. 26, no. 2, pp. 301-320, 1981.
J. Willems, "Dissipative dynamical systems part I:
General theory," Arch. Rational Mech. Anal., vol. 45,
pp. 321-351, 1972.
A. Isidori and A. Astolfi, "Disturbance attenuation and
X m-control via measurement feedback in nonlinear
systems," IEEE Transactions on Automatic Control,
vol. 37, no. 9, pp. 1283-1293, 1992.
D. Jacobson, "Optimal stochastic linear systems with
exponential performance criteria and their relation to
deterministic differential games," IEEE Tramactions on
Automatic Control, vol. 18, no. 2, pp. 124-131, 1973.
J. Speyer, "An adaptive terminal guidance scheme based
on an exponential cost criterion with applications to
homing missile guidance," IEEE Transactions on automatic control, vol. 21, pp. 371-375, 1976.
P. Whittle, "Risk-sensitive Linear/Quadratic/Gaussian
control," Advances in Applied Probability, vol. 13,
pp. 764-777, 1981.
[7] A. Bensoussan and J. H. van Schuppen, “Optimal
control of partially observable stochastic systems with
an exponential-of-integral performance index,” SIAM
Joumal on Control and Optimization, vol. 23, no. 4,
pp. 599-613, 1985.
[8] K. Glover and J. Doyle, “State-space formulae for all
stabilizing controllers that satisfy an H,-norm bound
and relations to risk-sensitivity,” Systems and Control
Letters, vol. 11, pp. 167-172, 1988.
[9] P. Whittle, “A risk-sensitive maximum principle,” Systems and Control Letters, vol. 15, pp. 183-192, 1990.
[lo] P. Whittle, “A risk-sensitive maximum principle: The
case of imperfect state observations,” IEEE Transactions on Automatic Control, vol. 36, no. 7, pp. 793-801,
1991.
[l 11 W. H. Fleming and W. M. McEneaney, “Risk-sensitive
control and differential
games, pp. 185- 197,” in Stochastic Theory and Adaptive
Control, eds., T.E. Duncan and B. Pasik-Duncan, (New
York), Springer-Verlag 1992.
[12] M. James, J. Baras, and R. Elliott, “Risk-sensitive control and dynamic games for partially observed discretetime nonlinear systems,” IEEE Transaction on Auromatic Control, vol. 39, no. 4, pp. 780-792, 1994.
[13] C. Charalambous and J. Hibey, “Minimum principle
for partially observable nonlinear risk-sensitive control problems using measure-valued decompositions,”
Stochastics and Stochastics Reports, vol. 57, 1996.
[14] C. Charalambous, “The role of informations state and
adjoint in relating nonlinear output feedback risksensitive control and dynamic games,” IEEE Transactions on Automatic Control, vol. 42, no. 8, pp. 11631170, 1997.
[15] P. D. Pra, L. Meneghini, and W. Runggaldier, “Some
connections between stochastic control and dynamic
games.”Mathematics of Control, Signals, and Systems
, vol. 9, pp. 303-326, 1996.
[16] R. Elliott, L. Aggoun, and J. Moore, Hidden Markov
Models Estimation and Control. New York: SpringerVerlag, 1995.
[17] V. A. Ugninovskii and I. Petersen, “Minimax LGG control for stochastic partially observed uncertain systems,”
SIAM Joumal on Control and Optimization, vol. 40,
no. 4, pp. 1189-1226, 2001.
[18] I. Petersen and M. R. James and P. Dupuis “Minimax
optimal control of stochastic uncertain systems with
relative entropy constrains,” IEEE Transactions on Automatic Control, vol. 45, pp. 398412, 2000.
[I91 W. H. Fleming and E. Pardoux, “Optimal Control
of Partially Observed Diffusions,” SIAM Journal on
Control and Optimization, vol. 20, no. 2, pp. 261-285,
1982.
[20] C. D. Charalambous and F. Rezaei and M. Djouadi,
“Optimal Disturbance Attenuation for Nonlinear
4264
Stochastic Uncertain Systems: A Measure-Theoretic
Minimax Approach,” Preprint, pages-45, 2003.