This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
$ 2 E -IS 2 C£ E froa [lxL-(s)1 + Iy (5) - y(x' (s))1 + + lyE(s) - y(x£(s))1 3]dS where xE,y£ refer to the system when controlled by the composite feedback, i.e. (6.30). From the estimates (6.31) follows, (6.38) On the other hand, for an admissible control of the class (6.10), which suffi ces, the corresponding states (also noted xE,yE for brevity) satisfy (6.39) 169 HABETS P. [1], Singular Perturbations in Non linear Systems and Optimal Control, in M. ARDEMA (edition, see above). pp. 103-143 HADDAD A.M., KOKOTOVIC P.V. [lJ, Note on Singular perturbations of linear state regulators. IEEE Trans.Auto.Control, AC-16, 3, pp. 279-281, (1971) HADLOCK C.A. [lJ, Existence and dependence on a parameter of solutions of a non linear two point boundary value problem. J. Diff Equat., 14, (1973), pp. 498-517 KOKOTOVIC P.V. [lJ. Applications of Singular perturbation techniques to control problems, SIAM Review, (1984) KOKOTOVIC P.V., Olt~ALLEY Jr R.E., SANNUTI P.• UJ, Singular perturbations and order reduction in Control theory, an overview Automatica, 12, (1976), pp. 123-132 KOKOTOVIC P.V., SAKSENA V.R .• [1], Singular perturbations in Control theory. Survey 1976. 1982 KOKOTOVIC P. V.• YACKEL R.A .• [lJ. Si ngul al" pel turbati on of linear regul ators, IEEE Trans.Auto.Control. (1972). AC-17. pp. 29-37 1 LIONS J.L. , [1], Quelques m!thodes de r~solution des linAaires. Dunod. Paris (1969) probl~mes aux limites non MINTY G.J. [lJ. Monotone (non linear) operators in Hilbert Spaces, Duke Math-Journal, 29 (1962), pp. 341-346 O'MfdlEY Jr R.E. [1], The singularly perturbed linear state regulator problem, SIAM Cant, 10, (1972), pp. 399-413 [2], Singular perturbations of the time invariant linear state regulator protilem, J. Diff equat. (1972) 12, pp. 117-128 [3J, Boundary layer methods for certain non linear singularly perturbed optimal control problems, J. ~1ath anal. App. 45, (1974), pp. 468-484 [4J, Introduction to Singular Perturbations, Academic PI~ess, 1974, N.Y. control. In Mathematical Control 680, Springer, N.Y. O'MALLEY Jr R.E. ,KUNG CF.[llJhe matrix Riccati approach to a singularly perturbed regulator proble~. J. Diff equat. (1974), 17, pp. 413-427 [2J. The singularly perturbed linear state regulator problem, SIAM cant. (1974), 13, pp. 327-337 SAKSENA V.R., O'REILLY J .• KOKOTOVIC P.V., Singular Perturbations and Time-scale methods in Contl'ol Theory. SUI"Vey 1976, 1982 SANNUTI P. [1], Asymptotic Solution of singularly perturbed optimal control problems, Automatica, (1974). 10. pp. 183-194 [2]. Asymptotic expansions of singularly perturbed quasi linear optimal systems, SIAM Cant. (1975). 13, 3 • pp. 572-592 SANNUTI P., KOKOTOVIC P.V., [lJ, Near Optimum design of linear systems by a singular perturbations method, IEEE Trans -Auto-Control (1969). AC-14, pp. 15-22 SINGULAR PERTURBATIONS IN STOCI-IASTIC CONTROL A. Bensoussan t G.L. Blankenshl p tt Abstract: \Ve consider a class of problems in stochastic conl,ral theory Involving stochastic systems with small parameters. Using both analytical and prouablllsUc methods adapLed to Lhe special structures of singularly pcrt.ul'bcd stochastic control problems, we develop a sysl,emal.lc methodology for their analysis. Introduction In this article we address the following class of COil trol problems. \Vc have a system governed by dx = fdy = f (x ,Y ,v ) dt 9 (x.y ,v )dt x (0) + J2 rl w + /2itlb (1 ) x. y (0) = y. where wand bare independen t; vViener processes. The state x (i) represents the slow system, while the state y (t) represents the fast system. The scaling is such that the variaLions of the fast system PCI' unit; of time, in average as well as in variance, are of order The dynamics are controlled via the parameter 'IJ (t). There is full information and IINRIA, Domain!! de Voluccll.u. Rocquencourt, D.P. 105, 7815:1 LE CHESNAY CEDEX FRANCE. Also with Lite Uniufrsile' de Paris - Dauphine. The research of this Il.uthor was sUl"portei.! In pnrl by the U,S. Department or Energy. ttElect.rlcal Engineering Department, University of Maryland, College Park. 1\1D ~Oj.j:!. This fl':'lf;rlrch was support.ed In part. by the U.S. Army Hesc!l.rch OUlce and t.he Army Night Vision and Electro-OptIcs Lauomlury ullder contract DAA.G 2!l-S3-C-OO:!8 with Systems Engineering, Tnc., Greenbelt, MD. 173 One can then consider the Bellman equation of ergodic cont;rol relative to (G). It is defined are follows: pick a constant X (constant wil.h respect to y) and a function
Suppose llhat one can 11 nd a pair x..cP tP + X = H{y ,Dllf/J)· (7) depending parametrically on x x= ,p ; hence. X(x .p ). If we choose u so that - All 7- f3u = X(x ,D;r (8) 11 }, then the pair u ,cP will satisfy (5). Onc can Lhus expect a solution or (8), vanishing on the boundary r of 0 to be the limit; of 1t '. This procedure depends on the possibiliLy of being able (,0 solve ergodic contl'ol problems of the t,ype (7). Tho cont,rol problem iLselr is as follows: Consider dy = G (y ,11 )d T + v?idb. y (0) lim 1 E = () Co) T T-+oo f D (y , I) } if T o then in general x In! ( Nil (v (.»)} independent. v (0) of y. The interpretation of tjJ is more delicate (eL sect. 2.G). Pick a feedback 11 (!I) and consider th e con trolled state ely =G(y,v(y))dr + ~db. y(o):::=y. (10) It seems inevit,able to requ ire crgodicity of the process y. This means that as T - 00, Y (T) behaves like a random variable following a proba- bUity In;:V(')(y), depending on the choice of v (.) l\,nd of the paramet,er entering into the deOnition or G. Suppose, moreover, t;hat, m is a. probability density WiUl respect t,o Lebesgue measure; it is possible to give another inLerpretalioll or \: as follows: 175 case, while in sections 3 and the case of reflected dHTusions is treated. -1 In section 5 we consider a fast system of the form iii dy = G (v)y dt E + III urn 7J db m (t). Y (0) !I. (13) =1 Defining the norm and the angular velocity I y (t) I fl{t ) ~(t ) J y (t ) I y (l ) I = then by linearity of (13). eO) is itself a diffusion taldng place on a sphere; hence, ergodicity will hold for e(t}. \I'Ve assume that the slow system depends separat;ely on the norm and the angular velocity of the fast system. namely dx = I (x ,p.e,u) dt -I- "f2 dw, x(o) = (14) x and we minimize a cost function of the form E Jo ".. - PI I ( x .p ....t:) ,v dt . \Ve treat this model in situations where p -+ 0 as l ---+ co, and ~ is ergodic. In sec- tions 6 and 7 we consider (10) in the whole space. To get an invariant probability on the whole space, we follow the theory developed by Khas'minskii [2] in the case without Note that G troL Fy + COll- cannot be bounded; hence we consider a drift ot' the form G (y ,v (y)). where F is a stable matrix. This n.1't.icle covets most cases where a natural ergodic fast system governs the evolution 01' Lhe sl,atc. There may be many other situatiolls where different techniques of singular perturbations are used. Examples of such situations may be found in the paper of ft. approaches to ergod ic con trol. see Jensen and P.L. Lions [3j. For other Hl. Ack now ledgement The first au thor has bencH ted from useful discussions with H. Brezis, L.C. Evans, and J.L. Lions. The second author would lil~e to thank G.C. Papanicolaou and P.V. Koko- 177 (O.A ,F t .Pt), btJ (t) becomes a Wiener process and the process y (l) is the solution of dy = (l + Vi. db g (t). {y (t ),v (t)) dt y (0) = y. (1.8) Let us consider the cost function 00 J(lI ll'(v{.))=Elje o otl(y(t),v(t) dt (1.Q) \Ve set {1.1O} \Ve shall be interested in the behavior of rPo(Y) as Q' tends to O. 1.2 The Hamilton-Jacobi-Bellman Equation Let us consider the Hamiltonian H(y,q)= In( {l{y,v) + q.g(y,tJ)} v E tJ,2r/ J..nt = ~ I;: {L(y ,q (1'.11) .v)}. ~d From the assumptions (1.3) and (1.5) there exists a Dorcl map Y with values In U ad , such that - lI(y,q) L(y,q, Y(y.q». (1.12) The following is a classical result in stochrusl;ic 'control theory: Theorem 1.1. The Junelion tPa is the uniqlle periodic jlmclion beiongino to (1.13) A1oreover, 1/ we sel v o(Y ) then the process -i} 179 (l - AG{)m =0. (1.19) The number of linearly independent solutions of (1.10) is the same as that of Le .• (1.20) z periodic, ;; E Hl(y). But the solution of (1.20) is more regular, Z E WI~it (lRl'I). \::1 p. 2 ~ 1) ~ 00. The desired result follows. 0 1.4 Fundamental solution Let us consider the Cauchy problem f}z gV.Dz = 0 al Z(y,O) (1.21) tMy) where ¢ is Borel bounded. Then there exists the following represen tation formula z (y ,t) = f P v (y ,l,'I)¢(11}d 11 (1.22) Ei where the funct.ion 1) (y ,t ,11) is the fundamental solution and has the following properties \10> 0, Ii < T, \1 g, pll(x;.,.) E L a(8,T;H 1(lRI'I}) Cn exp( - Q'~ - where 0'1' QI::J > 0, C I' C'J > 0 (1.23) I Y -'I I ~ ] l and they depend only on the bound on g G • (1.2-:1) Tn particular, they do not depend on the particular feedback v (.). This result is due to D. Aronson [5]. Now note that; if ¢ is periodic, z is period ie, and we can write z (y ,t ) = f PD~(Y ,l,1/)¢(11)d'l l' (1.25) 181 I z(y,l) Taking 11 = J4>(t})TI(d'l) y I ~ K Iltbll e p. [l), we deduce I z(y.t) - JrfJ('1)TI (dry) 1 < J( c P 114>11 t! -pl. Y Using the invariant measure m Tn 11 (1.31 ) defined in (1.IG), we also see easily from (1.113) and (1.21) that Jz(y,t) m(y) dy = J~6(y) m(y) dy. y y (1.32) Using (1.31) in (1.32), we deduce J Tn (y ) dy J cP(77) y which proves th at. n (d '1) = J ¢(y) y Jm (y )dy 0, 'Tn (y ) t/y y since is not a. e. O. Normalizing the in tegral to be I, 111 r we see that IT(dy) = m (y)dy and th us {l.31} yields Izt>(y.i) - Jq,(y}mtl{y)dy I < J( ePllrfJlle- Pt • y Now recalling (l.30) Jy ,p(y )m (y }dy = J}' z {y ,1)111 (y )(iy and from (1.25) and (1.27) {j Jy ¢(y }dy ~ < 81 ~ J ¢(Y)1I1 (y )dy < Z (y ,1) Jy rfJ(y )dy. Therefore, 8J q,(y )dy }I" Hence, Y 01 J ,p(y )lly. Y (1.33) 183 JU(y.vo(Y)) - atPo(y))mcx(y)dy =0, y Considering ;];0' = Jy ~Q(Y )r/y we deduce rrom (l.38) that But rrom Theorem 1.3 we can assert that where 0. 01 do not depend on 0'. Therefore, it follows that; J I D tPa I :::; C ely :l I r/J" - ¢a I L!lP") Y ::5 a [ ~ I D ~" I 'dy 1/2 1 By Poincare's inequality we thus obtain the estimate I D rPo I L:l :::; C. (l.30) Let us consider Hence, - rPa is bounded in H 1CV) and Xa is bounded. Note that Jf(y.D ,po} is also bounded Consider a subsequence such that Xa - x' 4>0 - H (y ,D 4>0) r/J in }[l{Jr) wellkly and a.c. -10 e In L 2( Y) weakly. Passing to the limit in (1.13) yields - ~t/> + X= e. 4> periodic. Jy 4>dy o. (lAO) ,185 Considering m ~ thejnvariant pt'obability corresponding to v:!. we deduce by multiplying Iy m~ I D (¢1 I ':!dy - ¢:2) + = 0 tP'J) + = o. There(ore, constant. 0 ¢'J 1.0 The-ergodic control problem \Ve can now interpret the pair (X, ¢). \Ve already know x= = lim Ci tl o(y') 0 ..... 0 11m Ci 0 .... 0 1111 Ky'Y(v (.». In fact, one can,be slightly more ,precise. \Ve have Theorem 1.5. 'Under the assumptions of Theorem 1.4 we have 00 'X = 'in] { 11m 0(0) Ci Ej'Ie - 0-0 = Illf{ 11m v (0) r::;-oo ot I (y, (f).v' (t» tit } (1..12) 0 1 T 1, EoVII(y(t),v(l» lit} 0 'Nloreover, choosing the undetermined constant for ¢ such ,that J~ry(y )11~ (y)i/y =,0, we y have T I t/>Ur)=Inj {l!ill 'E!f(l(y(t),v{l)) - x)ut 7 .... 00 Proof. v(.): E,J't/J(y(T»-o} 0 Let us simply prove (1.-13). For any control 11 (0) we have T ¢(y) < EJ' I( /(y{t),v(t)) o ;Hence, if E~1J¢(y (T)) -+ 0, 'then - X) lit + E~!JtP(y(T» (1..13) 187 = - b/(t) (J tV (/) (x (s ),g t(s ),v (.5 J - »ds (2..1) (2.5) {x (s ),y (.'I ),V (8)) lis Let us consider a bounded smooth domain 0 of m B , and denotes the flrst exit time r=T;: of the process x (t) from the domain O. Since we are not going to consider the process x outside 0, we ma,y without loss of generality assume that JJ OJ and I are bounded functions. Let us define the probability pf (which depends also on the control dp E dP exp ([ [ 1 f'1 + [ - ~ j [~ 1 9 ·1 0 ! 1 (x (s 1 ).v (8)) 12 -}- f For the system (O.A ,FI·P f), and (x ,y)) a (x (. l,y ,(a Lv (, )). db (, 1 ] (x (. l,y ,(, l,v (.)). ,(w (.) ),Y t(1I 11 (.), 1 (1.6) I J (x (8 ),y «s),v (.5» 12]dS } the processes b l(t) ilnd w £(t) become standard independen t \Viener processes and the processes x (t ),Y ,(t ) appear ns the solutions of (2.7) x (0) = x, y,(O) = ]I Our objective is to minimize the pn.y olT function .l/l1(11(.»=E' J l(x(t),Yt(l),v(t»e-t/t ilt (2.8) o wh ere /1 > o. Let us set U ,(x ,Y) = InJ J.'.,(v to (a) •• , (.» Then u, is the unique solution of the H.J.B. equation (2.0) 189 iJ:! Ir = av 0, z is periodic in y and the solu tion is necess(l,rily a constan t. vVe have the following important estimate Lemma 2.1 lYe have (2.16) where 5 and 01 are the same as in (1.34). Proof. Let us consider the fundamental solution of the Cauchy problem. o iJz I r = 0, -8 1.1 . dl em . =. IS perlO z (x ,V ,0) where we h ave set 9 c = (2.17) y 1/J(x tY) 9 (x ,y ,v (x ,y ). 'Ve are going to use the probabilistic interpretation of (2.17). Consider process reflected at the boundary of O. Namely, J2€ iIx = dU! - Xr(x (t» v d ~(t) x(O) = where e is an x increasing process. Consider next dy(t) = V2 db{l), yeo) = y and assume wand b are independent. Let us define the pl'oeesE t bt(t)= bet) - ~ £g((x(S).Y(8).V,(X(s},y(s))) tis and the change of probability t dP' IF" exp{[~ t g(x(,r;),y(,r;)).db - +[ 1!1{I~cl.s} 11 \Viener 191 8¢ at - AtjJ f q,(x ,0) = = a~ ° av I r 0, (~.~1) Jy 1f;(x ,'I) d t] But we notice that JJ 8 III f(X ,1) JJ 111 Ax (x Z ,1) ,1) dxdy ,1) )1/1(X ,1) dxdy = JJ S JJ TIt 111 ,(x 1f;(x ,1) rlxdy ,1) ,(x ,y) '1Nx ,y ) drily However, from (2.15) we have - f. A;t J1" m ((x ,y) dy 8 = 0, -a v Jl' TTl ,(x ,y )dy 0011 r hence Jm f(X ,1) dy = constant = 1. Y We then deduce from (2.22) 8 Jq,(x ,1) dx o ~ JJ 111 s 8 Jr/J(x ,1) dx kr,y) 1f;(x ,V) dxdy 1 0 Bu t i'rom (2.21) J ¢(x ,1) dx o J tjJ(x ,0) dx = JJ 1/J(x o which, used in (2.33). implies the desired properLy ,1) dxdy 0,." (~.lG). D 2.3 A priori estimates \Ve start with Lemma 2.2 The following estimales hold 2 Here we use the positivity of m (/ which follows from ergodic theory. ll.5 III secLloll 1.'1. (2.23) 193 + I I Dz rP I ~ + dx j3 I if/ dz Proof. We multiply (2.25) by m c¢,l and in tegrate. \Ve have (2.20) Now mul!;iply (2.15) by 11 L¢ and integrate. Using t,he fact that tP does not depend on y, we obtain II + .!.. E D" m (DII u! tP dxdy = II 1 f TIl f 9 (2.30) D" u ( rP rlxdy. But (2.31) - II m ( ~¢ Tt ( tlxdy. + II Using (2.30). (2.31) in (2.2\)) yields 2 II Dz u (D z ¢ m (dxdy III f ttl 1::..q) (2.32) dIdy But now (2.33) II + m I I D:: I £ t/J Dz I2 tt ( dx I + 2 clxdy j3 I + .!.. € II m f I D1/ It £ l!l dxdy + I I t/J2 dx Using (2.26) and (2.32) in (2.33). the result (2.28) follows. 0 j3 111 f It (2 dxdy 195 implies f fm o I f (x ,y ) 1jJ( x ) dx tf;( x) fix \;! fjJ ELI. o y Letting e tend to 0 in (2.35), we deduce + jJtP,. ) dxdy I~ u) = + 2 f3 Letting k tend to '}. f + J Du D,pl; dx - 2/3 f ,p/:2 dx + 00, 2 f I D" (h it (4J k - I tt) f I D" rPk f (,p,. dx + 2jJ + dx 2 f I 2 D",p,. + 2jJ f (111: ~ dx I~ dx I dx - :l 1I)~ rl.r.. we deduce Since clearly 1t) I 2 + jJo dxlly JJ (ll c - 11)~ dxd y the strong convergence property is established. 0 "Ve are now going to iden tify the limit. \Ve shall llse the notation or section 1. Let us consider x E 0, and tion 2. with 9 (x 'v ,v) [J E Un as parameters. \Vc shall consider the SitlHltioll or sec- instead or 9 (y ,f}) and I (x ,y • v) -I- p.J (x ,y ,v) instead or I(x.y,v). A feedback now is a Borcl function or y, which mas be indexed by xJp. feedback v consider the invariant. probability m ~ (x ,y), which is the solution of - ~v m In + dlv g {111 9 (x ,Y E HI(y'l, til ,1) ( Y ))) = 0 periodic. For I,p fixed we can consider the scalar X defined in (1.3G), namely For any 197 fJ¢c Ir (2.,13) 0, ¢( periodlc in y This problem is studied using ergodic control theory as in section I. 111; have (recalling that the integral of Ai = 111/ lI(.,.j I0 is ,) m: (x ,y ) II In particular, we (2,-1-1) F(x,y ,v (x,y)) dxlly. Consider now for x rrozen the problem - Q.lJ ¢ H(x ,y .DJI ¢ periodic in y. ¢ E \Ve want to study the limit of A( as Lemma 2.5 ~Ve f q,) - (2A5) A(x) tv'.!·l' (Y), \-J x tends to o. We shall need Lhe have I0 I A,""'" J A( x) rlx (:3..I6) o Proof. Note that A( x ) = Jn / fI(.) J 111 II (x ,11 ) F (x •y , v (y »tl 11 where tn is donned by (2.36). Hence A E L ~(O). Since we can write - - DII ¢.g (x ,1/ , v) - A{x) -Q. v ¢=P(x,y.v) Considering 11''' v, \VC deduce J mil I Dv ¢ 1:1 ely = J 1It II {F (x .1/ ,II) A(x ))(rp - -;j;) dy r l' where ~(x ) = Jy ¢(x ,y) dV· Hence, f I Dy 4) l!.l dy $; CF {x f. }' Therefore, Dy IjJ E L2{O X Y). From the eqUation (2.-15) it also roll0\v8 t.hat 199 hence Letting k -+ 00, we obtain A, I 0 I ::; J A(x) fix which with (2.50) completes the proof of the desired result (2.-16), 0 \Ve can now proceed with t,he Proof of Theorem 2.1 Let. us consider a subsequence of u[ which converges to u in [JI{O X Y) weakly. Such subsequences exist, by virtue of Lemma 2.2. 1vloreover, from Lemma 2A, we can assert that u does not depend on y [wd that the convergence is strong. u E HoI (0). Let,p E Coco (0). ,p ~ O. ~roreover, Consider the relation (2.25) and multiply by cP m (" ActuaIJy, we have already done the calculation in Lemma 2.3 (c.l'. (2.20). a.nd (2,32)). \Vc thus have (c.r. (2.32». 2 JJ Dz uc D z rP 111 (chtly -I- ff III ( It < .6. q, drily (2.51) Now consider F(x,y.v)=q,(x)(I(x,y,v) -I- Duo! (x,y,v)) which satisfies (2."11). Obviously. (2.52) Note that A(x) lIence from Lemma 2.5 X(x ,DIL ) ,p(x ) 201 f D: u D;z ¢ tlx + J :::; f J (I (X ,V ,V (X ,y}) + D:z :3 11 .6.t/J dx + fJ f tl.f (X ,y ,V)) tl (2.56) ¢ elx 111 V qJ tlxdy. Therefore, - Au + fJ u :::; f (l (x .y ,v (x ,V)) + D;: u.J (x ,y ,tJ )) m v dy r' and since v is arbitrary, we deduce - .6. u + fJ u :::; X(x ,Du ). Therefore, u coincides with the solution of (2.38). By uniqueness the desired result obtains. D 2.5 Asymptotic expansion The result of Theorem 2.1 does not give any estimate of Lhe rate of convergence. Bu t its advantage is that it requires no particular reglllariLy assumptions. It is possible to give a rate or convergence and to proceed dHferenLly relying on maximum principle type of arguments, when additionaJ smoothness is available. The situation is somewhat similar to what arises in homogenization (c.f. Bensoussan Lions Papanicolaou [GJ). \Ve shall outline the argument, and in particular we shall not try to present a.':;sumptions under which the regularity reqnil'cmen ts are satisfled (see for til at aspect (71). Consider t/J(x ,y) such that All ¢ + X(x ,Du ) = H (x .D" 1l tY ,D" t/J) (2.57) t/> periodic in y. \Ve define u [ by u!=tt Then 11 l +tt/J-f-1tc; (2.58) satisfies (2.59) -I- H(x 203 V"(x .Du ,y ,DII q,) v (x ,Y ) (2.63) is an optimal feedback for the limit problem. In fact, this is the feedback Lo be applied on the real system as a surrogate for u «(x ,y) den ned in (2.13). One can show by techniques similar to those used in previolls paragraphs to obt.aJll Theorem 2.1, t,hat the corresponding cost function will converge as tends E a to 11 in Hl(O X V). Note t.hat unlike the deterministic situation the optimal feedback for the limit problem is not a function of x only. In fact (2.63) corresponds to the composite feedback ot' Kokotovic [8] (c.r. also 17\ Chow~ in the deterministic case). 3. Ergodic. Control for Reflected Diffusions 3.1 Assumptions and notation Our objective in this section is to describe another class or crgod ic con t,l'ol problems and to consider stochastic can trol problems -with singular perturbations wll kh call be associated to them, as in section 2. \Ve shall consider ditTllSiolls with reOeetion. Let B be a smooth bounded doma.in in nd, whose boundary is denoted by DB. Lct g and I be continuous funct;ions g (y,v): Ii X U -> n. d I (y .v): iJ X U -> nd (3.1 ) where U is as in (1.2) and Uall is as in (1.5). Consider (O,A. ,P ,FI ,6 (l)) as in section 1.1. and let y (/) represent t.he dilrllsion process reflected at the boundary of B ely .J2 db - XlJn (y (l ))/Ld lJ yeo) = (3.2) y. where II is t,he outward unit normal at. the boundary of B, and ,,(l) is an increasing process. Admissiblc con('rols are defined as in section 1.1. Lor; us consider next the process be (l) defined in (1.6). and t.he cbange of probabilit.y dotlned in (1.7). For the system 205 the coeJficicn ts are not smooth. These properties have not, been clearly stated in the literature. \Ve shall proceed differently using some ideas at' Y. Kogan [oj. Consider the parabolic problem 8z 8z a; I IlB = 0, ~z - Z - g II.D:: o (3.7) 4>(1)), ¢ Borel bounded (y ,0) and we slla1l define the operator, as in (1.2S), PI/>(y) = Let us write for r a Borel subset of A...~(r) (3.8) z (y ,1). 11 P Xr(x) - P Xr(Y). Y,z E B (3.0) \Ve have Lemma 3.1 81t 1l { A;:~ (n I t}, X ,y ,f'} < (3.10) 1. Proof. Suppose that (3.10) does not hold. Then there exists a sequence { Ilk ,:fk ,YI: ,r k } such that Hence, Define Zk (y initial data ,t ) to be tIle solution of (3.7) corresponding to 1;lw control Xf't (y) = t'l: (.), and to the tPJ: (y). \Ve then have (3.11 ) Bu t the classical estimates on parabolic equations yield ZI; bounded in W 2 •P ,1(B X(5, T)), \-1 8 > 0 207 Therefore, (1.33) also holds. This estimate implies in particular Lhat In v ;::: 0 and can be taken to be a probability. To prove the estimate (1.3-1), we rely on the foHowin g Lemma. Lemma 3.2. The solution oj (3.5) (normalized as n probab£lilY) satisfies (3.17) wilh a norm bounded in v (.). Proof. Let us consider the problem o If l/J E L' (8) then tP E W~,' (B). :s .s 1 < 00. Moreover, we have Jm tP dy = J111 ,pdy. B (3.18) B We deduce, since m is a probability, :s I Jm ~,dy I B I r/; I o"(li) < ClltPll w''2.' • with - -< C 1 Therefore, mEL '., with 1 1 /1 (3.17) is proved. Suppose d 3, I () ' 8' a d > 2 IV)I L" " . 1>_1_>1_2.. It' d then we have 111 1,2. d &/} E L", 1:S So < the result. 3. \Ve proceed using (3.18) together with a boot strapping argument. \Ve Ilave with 1 ~ 1 < -;-;-:- + 2 d; and (3.17) is proved, If tl = h 1 1 ence, ---;:: > ~ > '1, . . . , 1 8" we proceed in the finite number of steps suffice to imply (3.17). 2 J' Since tl SLime 3. S I is i.1l'bil,rary way. For a.ny value of d, a 209 If h > k > 0, then A (h ) c A (k). and we have < k} (Aleas A (h ))' {ll ff ::; !J(k ){m - k}" dy l' It and from (3.20) it follows that (It - k) (Aleas .A (11 »' J.. < elm I L' (Afeas A (~.))!! or G I m Il: (J.. _ J..),' , Afeas A (h) ::; - - - - (Mens A (k» ~ (Il - k)' Pick s > d. then 1f;(t). ko ::; t < 00 (2.. 2 .!..)s· > s 1. \Ve use the following result from [llJ (p. 63): Let be nonnegative and non increasing, slIch that (3.21) where C ,01, and f3 are positive constants with f3 > 1. Then (3.22) where (3.23) It is clear that this result applies, and thus A1eas A (k) = k = G 1111 I L' I 0 where J.. (3.2-1) (Aleas B) d The second estimate is thus proved. The proof of the second estimate is more involved. \Ve refer t,O [7]. Remark 3.1. The fUllction result in Lions - ~!lagenes 1)1 l12J v E WI,IJ (B). \1 p E (I.OO). T'his follows from a general Teo. 6.1, p. 33. Indeed, we write (3.5) as follows 211 i)z at i)z -a IBB II with tP E L l(B}. tP ~ O. 9~ .6.z = 0, Z .Dz = (y ,0) (3.27) 0 4>(y} Let us assume that (3.28) where Co does nOL depend 011 tP, nor on v (.). v. vVc then have Proposition 3.1. The following estimate holds where c is i'ndependenl of ¢. v (.). and y. Proof. ''\Tc shall prove that lu! {:(y.l) ly,l)(·).I/I~O, ItPlLI l,pk ILl = if this is false. there exists a sequence I/Ik ~ 0, jng =k the solution of (3.27) corresponding- to 1/11:. Vk (.). I} (3.30) 1. Vk. tIl: (.) such that, denot- then one has (3.31 ) \Vriting i)ZI: ~ O. I.IJI lOB = O. =1; (y ,0) tPJ: (y) and making use of (3.28), we can asserL Lhat zdy ,I) is bOll nded in L =(fl X(8. T then ZJ: remains bounded in w~,llr (B X (8,1')). \-f p. 2:::; [J < n. Bu L 00. Reasoning as in Lemma 3.2, we identify a limit fundion z' such that az • -.6.z • • - 9 ".Dz a• ~1/lB ap' •= 0, 0 and as a conseqnence of (3.31) we have z'(y' .1) O. t E [ti.T 1 (3.32) 213 To prove (3.34), we consider (~ ,t ) r (y t}q(y,t) 2 which is the solution of aT - Ar + r g oil dlv (rg ) I DB = q (3.35) = 0 r (y ,.!..) = 0 2 and we know a pr£ori thilt (3.3G) The result of Proposition 3.~ follows from the following result of LadyzenskilYil, SOIOIllli- kov, and Ural'tseva [10]. Lemma 3.3. The solution r oj (9,95) belongs to L =(B X (O,..!:..)). 2 Proof. The argument is in the spirit of Theorem 3.1 or cnce we take T instead of 2-, 2 [111. For notational conveniE L 00(0.1': L 'I-.(B )). by and assume q ;::: o. NOLe thnt r the definition of r. Let k be a constant and vVe easily deduce from (3.5) that t ..!:.. I t](t ) I ~ + f I D '] I ~ 2 0 ds = f ds f(l·g = f(lSf o whel"c X E L 00(0, T ; L >-(B )}. Let us introduce the following nOl'ms + q) n 0 n X 'I tlx t] elx (3.37) 215 (3.40) 2:: For k 1 we can also write T III til II < Ok It then follows from the theorem in [ ).. [ dt CAfe-as ilk (t 1 II I, »~p ] [101 loc. cil., p. 102, ('I'heorem 6.1) that if we can write ).. - 1 ---p' 8' ).. with _1_ ~ > 2.. , then r r 1 =q r + fJ is bounded. Expressing.!:.. = q 1 d ).. - 1 -+ :2 ).. L, ,~ d d 2q -! ).. 1 r • then tl -r. (3.·11) ·1 Now from (3.37) it follows easily that hence [ 1 + d ·1 since).. > .!!:.. . 0 2 4. Singular Perturbations with Reflected Diffusion 4.1 Assumptions and notation One can apply the ergodic theory of Theorem 3.2 to solve some problems oj' singular perturbations in a similar way as in section 2. LeL us consider fUnctions I . !1. and It cOIJLinuOlIs 217 One can also choose a Borel runction v (x ,y) such that (4.8) 4.2 Convergence Now the same theory as the one developed from section 2.2 to 2,4 can be carried over to study the limIt· of' ('-r.7). \\Fe just state the result. For x ,p parameters we solve the ergodic problem of the type (3.213) q, + All X(x.p) = a¢ ap H(x.p .y .DJI t/J), - l o B = 0 (-1.0 ) and the limit problem is given by - A u + f3 u = X(x .Du). 1L r = 0, Tt E W~·P (0) (-1.10) Theorem 4.1. Assume ({. l)(f. 2). Then one has 11 ( - - (-1, ll) u in 1[1(0 X B) strongly The same considerations as in sectlons '2.5 and 2.6 carryover to this case. 5. Singular perturbations in the case of a linear fast system 5.1 Study of a linear system Let us consider the following linear system dy = G(v)y tit + r 'E CI"r Y dbr(t), yeO) = y. (5.1) r=l where G (V): Ua.d - Uaa L (IRa; lR d ): is a continuous bounded function compact subset of a met,ric space U. (5.2) (5,3) br (t ) independent standard scalar \Viener processes on (O.A ,P .Ft ) An admissible control is a process v(l) which is adapced to FI and takes values in UtJd • 219 Hence, It is convenient to introduce the vector -~( t ) ... ( - sinO ) = (5.0) cosO which is orthogonal to e(t). We deduce ~ (€·O'r [e.g e dO(t) = - e) (e·O'r e) ] at (5.]0) We shall assume that E - (e·O'r e)~ ~ cr > 0, \1 O. (5.11) Therefore, O(t) is a non degenerate diffusion, which is periodic with period 211". Remark 5.1 When d > 2. we shall have a local representation Consider then d €( t ) = D r d0 + .!.. D 2r 2 dBd0 hence dO = (D r" D rr L Dr" d Note that the relation I reo) I!! = 1, e- ..!... (D r" D rr 1 Dr· D 2r cl 0 dO. 2 implies Dr'" r = 0, i.e., Dr" e= o. We finally obtain (5.12) In the case (5.10) we have -e = Dr and Dr" Dr = 1, which implies Dr' n'2 r = o. The 221 - -?; E IE, - (ir €( I :l] ell + p£ JE €c (ir €( db r r (5.10) Let T( be the first exit time of x (t) from 0 ; we consider the cost functional /:~p,O (v (.)) = E J e -/31 l(x({i ),Pf(t ),0(1 ),v (t) (5.20) tit o and we set (5.21 ) Remark 5.2 The assumption on the derivatives of ]; G, and I allows us to consider a. strong formulation of the state equations. Consider now an ergodic control problem related to (I, according to the theory developed in section 1. Let us write F(x ,1) ,O,v) I (x ,0,0,1/) + (5.22) l'.f (x ,0,O,t}) where X,p are parameters. Let us also define (5.23) +E a (0) ... III f H (x ,p ,O,q ) = II (€ rir e)2 r [F (x .p ,O.v) + q.g (x ,O,v ) ] E uc:t (here q is a scalar). A feedback is a Borel function v (0) with values in U lJd invariant probability 111 II == 111 11 (x a~ T'o any 11 (.) we associate the ,0), which is the solution of - " (a (0) m to) ao- • a + -ao (IT! v g) 0 (5.26) m periodIc. Since we are considering a one dimensional problem, we have in fact an explicit I'ormula of the form 223 (5.34) He Ir 0, 1l (periodic In 0 where we have set H(x ,p JJ,q ,p,A) = [I (x ,p,O,v) In f II E fl.1l + q 9 (x ,O,v) + + p.f (x ,p,O,v) A k (x ,O,lI) (5.35) 1 and clearly the function defined in (5.35) coincides with H (x ,1' ,O,q ,0,0). Let us set 'It ((x ,0) = 'It Ax ,0,0) which is the solution of - 1 DUe H (x .Du (JO, -; ;)0,0,0) - u, I r = 0, 11 { periodic in (5.30) O. Let us also define 'It !ic(x ,p,O) = r(x ,p,O) - 11 ,0) (5.30) -------- p Then we have the following Lemma 5.1 Assume 'Y stl.ff£ciently large, then one has 1S"(e- JlP I L 2::; I a~c Proof. The function e - .~ C VE., ~p I L::! ::; c a~~ PP IpapeIL::!::;c vt Je. I D )f e - ~p I L!! ::; (5.38) c. rl satisfies (5.30) - 1 - H (x ,Du (00.-; Therefore, for a convenient feedbaclc v r(x .P.O) au ( ;)O,O,O) ]. 225 k - e+ E·G t: I 1 -;:;-E ! a r EI'.:! - r For I sufficiently large we can then deduce that I ~f + I L ~ a~ + :::; a.;e. I P --/;p ILl!:::; C .;e a~ + I +0 I :::; a JE, ID ~c + I :::; (5..11) G . Similarly, we can pick a feedbacl( for which the reverse or (5040) holds. "Multiplying by ~c - e-'.:!I1P and making similar calculations, we obtain the sa.me estimates as (SAl) for ~f - . The desired result follows. 0 \Ve can then state the following Theorem 5.1. Assume (5.1.1), (5.15), (5.16), (5.17) and (5.81) wilh I sllfficiently large. Then we have (Ul(X ,p,O) - 'U (x)) e -I'P __ 0 in Ll! (5 A::!) where Jl > 0 £s arb ilrary. Proof. From Theorem 2.1, we have kr 11 ,0) - u (x) -- 0 in Hl. This result combined with the estimates of Lemma 5.1 implies the desired result (SA::!). Note that JI is arbi- trary > o. 0 5.3 The smooth case \Ve shall make some formal calculations in the smooth case and derive estimates on the rate of convergence. \Ve define tPo{x ,0), - Al! tPl(X ,0) as follows: + [)~tPo (3u - a a2,p, - a alP [)4>o -I'l- [)o- - c = H (x .Du ,0, -;-0 ,0,0). u atPl eo = tPo at/Jo H /X ,Drl ,0, DO .0,0) periodic (5..13) (5.-1-1) 227 we have I u ~(x ,p,O) = E e - PI Xl(X c(t), o[(t). Pe{t» dt a - + E u ~(x f(T!)JO~(Tc)' Pl(Tl )) e - (Jr C and from (5.46), (5.48) it follows that I uc(x,p,O) I ::5 O(e + fP + E Ie-PI p,(tfl[l). o Using the expression for Pl' and assuming that we can conclude that which implies It l (x ,p,O) which justifies the convergence of 1L l to I ::5 0 (e + ~ (J + e p~) (5..10) 11. O. Ergodic Control for diffusions in the whole space 0.1 Assumptions - Notation Let us consider basically the situation of section 1, except Lhat now Lhe periodicity is left out. Of course, additional assumptions are necessary to recover the ergodicity. \Ve consider 9 (y ,v): lR d X U _ lR d I (y ,v): IntI X U lR d -J. (0.1) continuous and bounded Uad compact subset of U a met,ric space. (6.2) For a given feedback v (y), which is a Borel function with values in [lad, we shall solve in a weak sense the stochastic differential equation. 229 In genera1 one can try to flnd 1/1 of the fOfm = t/J{y) + LogQ (y) k (6.0) where ~ Afy • y + Q (y) fit • 2 Y + P (6.10) /\1 symmetric positive definite Q :;::: 0; D is a region containing lhe zeros of Q. The following condition must be achieved to get (6.8) 1 2 ~ for a convenient 11.£ = I. m = - tJ'M ~My'J -I- my } + (Fy + g (y ,1) )) • (My + III) (G.Il) p },fy'1 choice of /\1, + my 111, + and p. \1 y E 117(d - D For instance if d p. 2, we can take 0, andp = 0 and (G.D) is saliisfied provided that for instance F < - (_2:.. 2 6) I (6.12) and D is a neighborhood or 0 suH'icicntly big. 'Ve now follow Khas'mins)rii 12] to introduce a compact; set with an ergodic Ivlarlcov chain as in sect,ion 1..1. \Ve mal,-c the I'ollowing assumption: There exists a bounded smooth domain D fwd a function Ij. is continuolls and 10cal1y bounded on lR d A 1/1 - 9 (y IV) D 1jJ I/J> 0< 1/1 - co as - ?:. ~ 0 which D and 1, Iy I - 'v1. V. Y E IR J co and I D'I/; I. ~ I/J D (6.8) hounded In general, one can try to find 1jJ of the form VJ( Y ) = log Q (y) where + k (6.0) 231 I , T 1- T :1' •••• such that 0 To In! {t > T 11+1 = , T In! {t ~ u ~ 1 fl Tn I y (t) ED}. n ~ 0 The process y (t) in the brackets is the process defined by (G.3), i.e. with initial condition y. Let us set Yn = Y (Tn ). n ~ 1. Then Y n E PI and is a 1vlarkov chain with tmn- sition probability defined by {B. IS) \Ve define the following operator on Borel bounded functions on PI (B.IO) We can give an analytic formula as follows. Consider the problem A ~ - g (y ,v (y)) • D ~ = 0 in D I • d I\ ifJ. (6.20) We first note that Eti= ifJ(y;: (O(a: »)) = Ev;: dy.l (0' (a:) therefore taking account of (6.17) , we have P ¢(a:) = 17(a:) (6.21) where 17 denotes the solution of (6.16) corresponding to the boundary condition It Of course, in (6.21) x E rl =~. are the only relevant points. \'\'e then have Lemma 6.1. The operator P is ergodic. Proof. \Ve proceed as in the proof of Lemma 3.1. Indeed, defining \-f a:.y E r l• B Borel subset of P 1 everything amounts to showing that sup U ,I.l/,e >";:~(B) < L (6.22) 233 From ergodic theory, it follows (c.f. (1.30)) I where J(, pn r/J(y) - J r/J(11) n(d cr) I ::; J( 11 q, II e -P'" E 1\ x (6.25) '. p are uniform with respect to the feedback control v ( • ), and iT = Jr" denotes the invariant probability on r l • It follows that, since we can write I Evll~(Y(TR)) - J t/J(I/)iT(dcr) I '. We can then define a probability on IR. d , II ~ J( II t/J e -PRo (6.20) by the formula 8(.,) J [Evil J A(y,,(t)) dt J A(y) d p(y) = '. I iT(d cr) 0 ---------- J E/' O(JJ)1i(d £1") Rd (6.27) r. \-1 A Borel bounded In lR d • Following Khas'minskii, one can then prove that the invariant probability is unique, has a density with respect to Lebesgue mCU.'SHre, denoted by m = m Ii which is the solution of A • m + dlv (mg ") = J m (y) dy = 0, III > 0, (6.28) 1. (Rd where A - - ~ + div (Fy .). Consider now the Cauchy problem 8z at + A z - 9 v Dz z (y ,0) = t/J(y ) = 0 (G.2{l) 235 Hence, we deduce from (0.34) that 1) f I DIt I 2 tly v~ ltd I) - + 1) - ('2p P Jv 2 J 11 Dv • Dh dy - (2p I) - nd [ - nil Il t - f v ( .:!:.. nd t::..h + - 1 tl" F - Fy • DII '2p DIJ g Dh ) tly v 9 P - '2 p g • Dit 1 tly o. 2p Simplifying, one obtains J I Dv I --:.--..,..-_1 + Jv 2 [ - ht + _P_ _ l t::..h + 2p P JR4 2. - 1 - dy - nd P p 1 Jv nd I Dh I ~ - tr F - Fy • Dh 2p We now fix the function h, setting h(y.l)={ - Q(t)(y - r(i))!! - pel)] and choosing the fUnctions Q(t), r(t) and p(t) in order that ht + t::..h '2 I Dh 12 + Fy • Dh = o. Performing the calculations we obtain F Q =0 . F r r - p + =0 '2 tr Q o. vVe take Q (1) It follows that I, T (0) = E arbitrary, pea) o. Q (t) is the solution of a Riccati cquaLion and satisfies \¥e deduce from (6.35) Dv • y tly (6.35) !1 Dh J dy o 237 I Let us set .!L = 1 2 + O!, v + 2,2 ::; C I then we can assert tha,t • I •. , ::; , \Ve tihen take p = (1 I a)k, P ( ClI I" 1 +a+ q ,r 1 ] k = 1,2 ... and note From (0.30) we deduce the induction relation + a 1 +l)k~1. (6.-10) Note that by (6.30) ,pI ::; C I II I u c II I I + a III < C (although we cannot assert that 4>0 is ft nUe). From Co (1 + 0') a Lemma in ?: [10J ' p. 05, we deduce from (G.-1O) t,ha.t (assuming I), with Therefore, (0.-11) Since we get 239 + 2 J I ~ IP J I DIS' I I ~ dy _-2-!.P~(2.!-P _ _l~) d -dtE:;.p(l) p~ I g D ~ IP p Rd dy + Ra J I S' 1 2P (-2k 0 P - tr F) dy. Rd Choosing koso that 2 k0 + II iJ II - 2 ;:::: tr F we deduce 2p - 1 [J JR.' P I DIS' I P I ~ tty + I I S' l!lp dy j. (BAS) R4 Using the in terpolation inequality I u I;:! ~ applied with IJ I~IP , 0 Ilu I Iu I I we get hence, from (6.'15) the inequality --'---- (E,l (t )) - d"' p or also (6,46) Applying (6,46) with k = I, yields (taking account of (6.44)) and integrating, one obtains Following Besala [14], we deduce by induction that 241 From results on the Dirichlet problem, it follows that p, 1 < p < In particular, 00. m" (y) Tn 2:: 0 belongs to 111 ok > \1 11 E 0, J(. 11 ( • ). \Ve also shall consider the fo1lowing approximation to mR nonempty. Otherwise 0 = 0, which is impossible. radius R, centered at o. Let us consider (5.40) compact Remark 6.2. The assumption ((1.8) cannot be made without D fin ay (JR."). \7 0 is continuous. Therefore, we deduce that where the constant Ok does not depend· on (6.8) and (6.28) yield WI,II '/Il. Let Bil be the baH of defined by (6.50) in which A is sufficiently large so that e. g 0 + (A + ~ tr F }O'.l 2: I eI 2 + C( 02 ) \1 ~ E lR d , 0 E IR 'Moreover, TR (y ) = r( ~} where r( y ) Is smooLh T( 11 ) = 0 for r( 11 ) = 1, for I y I :c:; 2"1 I 11 I 2: and 0 ~ I, ~ L T \Ve have Lemma 6.3. m R - strongly and m the exlension of mil fU~, q by 0 outside B n , converges to m converges monotonically increasing to m. Proof. \Ve compute A - (TR m) + dlv (1'R + D Til hDnce 111 • g g) = 111 - ~TR + D Tn • tTl Fy - 111 2 D 1)~ • Dm In JIl{lR d) 243 f(mR -11I)+g.D(mR -m)+£ly DR + (1 + ~tr F) 2 + A J (m - f - m) + ~ tly (mn DR Tn m) (mn - 111) + dy = O. DR Hence, (rna m) + = 0 in Ea. In a. similar way we have Indeed multiplying by mn- and integrating yields J I DmR- I 2dy + DR f mn- g • Dmn- dy Dn Hence, ma - = o. Let us next prove that mq1 ~ mil!! if q 1 ~ q!.l' - m q ., - Multiplying by (1IIq 1 pletes the proof. 0 o. 18B 2'•1 mq(!)+ and integrating, we deduce (111 91 - m q2 )+ 0, which com- 245 Lemma 6.5. lVe have D (6.58) where the constant does not depend on a, nor y. Proof. We write (13.54) as follows A 4t0! - D I ,paCy) cPa • 9 (y ,v 0:) J cPo(71) il'o(d (1) I ::; (5.50) = T....{y). G1/-O( y ) in IR d { C inD - D (6.58) PI with From Lemma 6.-1. we have (6.60) :Moreover, clea.rly (5.61) Using the sequence of stopping times Tn defined in (O.~), we can write f'N Eel/a ¢,)(y (TN)) - tPo(Y) = E.Yer J lo(Y (l ») tit. (6.£32) Now from (6.26) EvYr:. tPo(Y TN) -" J ,po:(rJ) il'o.(il a) as N - 00. rl (0.03) Let us prove th at rN Ev"a JTa(y) til o Indeed I ::; C' 11'(y} \-! a, \-1 N. \-! y E IR.'f - D . (6.0-1) 247 r" + 1 N J E1a :E n=] To{Y) tIt j I :::; 0 :::; 0 1 independent; of 11 • N. 1"1'1 Moreover, 1"1 I E!CI JTcr(y) tIt o I E!a O(y) I ~ 0 t/;{y) which implies (6,64) and the desired result. 0 Proof of Theorem 6.1. Existence Let us set tPo = tPo J tPo{I1) 1rCl(J u). Then Al II ~I~ IlL"" ~ '0. Moreover, from (6.6) If' we also have (6.68) in which Xa J tPQ(ll)1ra(d17). C¥ At It readily follows from (6.68) that bounded in W!!·" .11(IIl d ). 2 ~ p :::; 00, JL > O. \Ve can extract a subsequence such that XD' -+ X tPOi -+ cP in W 2 ,p 'Jl(IR Ii) weakly. We can assert that tPQ' D t/!OI -+ cPo D rP pointwise. hence, H (y ,D cPo) -- H (y .D ifJ) pointwise, Noting that H(y.D¢a) is bounded in LP.JJ, we can pass to the limit in (0 (iR), and the 249 J a4>o(Y) mtl(y) (ly J I(y,v(y)) mfl{y) dy. ~ Hence. as a tends to 0, x ~ and since 11 ( • ) J I(y,v(y)) mil (y) dy is arbitrary X ~ X'. Therefore, (B.5\)) is proved . . Let v be a feedback associated to tP. where 4> is any solu tion of (6.53). Let us show that (6.73) Indeed call X the right hand side of (6.50). \Ve have A 4> - (} v .. • D 4> + X - X = - f (y ,v) - - and Ji(y) m~ (y) dy = o. \Ve deduce as in (6.62) r_ = - J Ef l(y(t))tlt Q bence, - (X - X ) E;!' TN bounded in N. However. E;!' TN - + 00 as N - since = E; Z (Y:r (II and (x)) I :z 00, X - = I (y) 251 D (rpt - interchanging ,pl 4/2 ) + o. and ,p':l, leads ftnally to D (,p1 - ¢'2) 0, and this completes the proof of the uniqueness. 0 7. Singular perturbations with diffusions in the whole space 7.1 Setting of the problem Again we basically consider the setting of section 2, in which we shall drop the aBsurnptions of periodicity as far as the fast system is concerned. We consider / (x ,y ,v) : JRfl X n 4 X U _ 11111 U (x ,Y ,v) : nil x ill:' X U _ I (x ,y ,v) : nil X n J X U - (7.1) JRd JR continuous bounded Uad compact of U (metrIc space). On a convenient set (O,A ,Ft .PC) (c.f. (7.2) section 2.1), we deftne a dynamic system, composed of a slow and a fast system described by the equations (2.7), with g replaced by Fy + g (x.y.v). The cost function is defined by (2.8). and we are interested in the behavior of the value function u (x ,y). It is given as the solution of the H.J.B. equation (noting Ay = - ~y - Fy • D) A:t tiE - .!.A f II U It, Uf + f3 tl , c = 0 for x = If (x ,D,z tI ,.y. -1 f E r, Dv u ,) (7.3) \1 y E W 2,P,P(O X n d ). 2~p <00 By W 2 •r ,1l{O X ni) we mean in fact, (since 0 is bounded) the set of functions z such that z "p(y) belongs to W 2 ,p (0 X IR d ). We shall denote by v ~(x ,y) the optimal feedback attached to (2.3), as defined in (2.13). The assumption (6.8) is replaced by 253 and we have (cor. (6.15)l if E (x ,V) ~ 1jJ(y). Define the sequence of stopping times l\.1arkov chain Xn = X (Til)' 1'"11 ]I To = O. 1"f1 , T n'I-1 as in section 6.2. and the (Tn) which is a Markov chain on 0 X "'11- define the linear opemtor on Borel bounded functions on 0 X p ( ~(x ,y) = E:tJ fjJ(x (O).y II vVe then by the rcla.tion (O». (7.7) \Ve deduce the analytic formula (c.r. (6.21)) (7.S) where - e.Ll:f 11 + f1 - f 9G All '1 - ~~ ~ • DlI 11 = 0, on 0 X (lll" D) (7.0) 8'1 1'1= f, -1[,=0 8v + Av !: - gv • DII ~ 0 on 0 X DI (7.10) o The ergodicity of P' is proved like that of P (c.l'. Lemma 6.1). Let lI"C{clx ,ll 0') be the corresponding invariant probability on 0 X II. \Ve then define the probability JJ'(dx ,ely) on a x IRd by the formula fI II A(x ,y ) d J'(X ,v ) 8(~.!,) [E/rJ 07l I A{x (l ),y (t)) tit 111"!(rJ e.d 11) 0 JI E/" °R d (7.11) O(e,ll) lI"!(d e,d ") 0'1 for any A Borel bounded on a X JRd Let us note that we can also give an analytic formula for the quantity 8{=.lJ ) aC(x ,'9) Ev:flJ f o namely ~ereE E,;Z 'V for short. A(x (t ).y (t») ill 255 Lemma 7.1. Let Bp be the ball of radius pin IRd J lip and IRd Bp_ Then {7.I8} where 6(p) -J. 0 as p -+ 00. Proof. Consider in (7.12), A(x ,y) Ii" n Dt rjJ, we deduce that" - f~;: + a 0 a C is a solution of and thus a All ()( - gv • D) Dy a = oa ov Ir 0, Let us consider the solu tion Assuming p sufficiently large so that XBp(Y). (7.H) o. It Av u - ko 1 DJI u I = Xii p u I.., (7.20) o. \Ve have (7.21) since clearly - c ~;: (a - 'U) g fI + Du - ko (0' - I All (0:' - u) - g" • D (0:' - u) Du 1 u) I.., = 0, ~ 0 In 0 X (n d a(~~U) 1r D), 0 Note also that since All (u - Du t/J) - ko 1 Du (u As p tends to + 00, (Du - D1jJ) ~ ainIrr d - 1 -1/;) 11'~o. up remains bounded in W::l,1' .11(Irr d - D). D 257 meR -+ m t in LIn fli as R -+!Xl. (7.2G) 7.3 A priori estimate \Ve shall need the approximation of given by tic (7.27) = 'U, 0 on 0(0 X B R ) and U til -+ 1£ (in 'VI~c" weakly and In L 00 wenl.;: sLat' (7.28) where loc is meant only for trhe y variable. \Ve shall need also a similar approximation in the case of' explicit feedbacks; in particular v f Lemma 7.1. The following eslim,ates hold I D;r U f I ll~c < c. I 11 ( !L 00 :::; C (7.20) Proof. Using the feed back v E' equation (7.3) reads - ~;r 1 'U£ - -;AlI 11£ + + ];..r; D11 Similarljr, define - .6.% 11 eR 11 l • g (x.Y." l ) corresponding to (7,30) (7.31) U (R U ( Consider similarly me and meR' obtains (7.30) (ju f = 0 on a(O X Bn ). :t-.1ultiplying (7.31) by nt,R tlcR and int,egrnt.ing. onc 259 7.4 Convergence We have Lemma 7.3. Let us consider a subsequence of 11! such that tL £ -- 11 in 1I,~c (0 X lR d Then u is a function oj x only, belongs to [J 01 (0), ) (7.37) weakly. and fhe convergence (7.37) i,') strong. Proof. Setting u) I:! + dx dy 1 e I I m ( I VII 11 [ I!! dx dy and making use of Lemma 7.1, one can prove as in Lemma 2.-1. thaL lim Xc 0 ' .... 0 and using the eSLimates (7.15) \Ve deduce I oxK I I D"Z (tL, - u) I:! dx fly + 1 e I I I Dy 11 oxK U)2 dx tly -- f I:! r/;r: ely + 0, ror any J( compact subset of IR. d • This proves the desired result. 0 We now identify the limit. Let us recall the clefinit,ion or x(x ,p ) by III v given in (7.5). Define the formula x(x,p)=Inf P (.J I m,,(%,vl(l(x,y,v(y)) + p ·f(:c,y,u(y)})dy IRd (7.38) and consider the Dirichlet problem -~lj +f31t xCx ,Du). 11 Ir = 0, tt E tV!!·I' (0) (7.30) \Ve can then state the following Theorem 7.1. We assume (7.1), (7.2) and (7.16). Then we have (7.40) Part II: LARGE SCALE SYSTEMS SINGULAR PERTURBATION OF MARKOV CHAINS F. Delebecque, O. Muron, J.P. Quadrat INRIA Domaine de Voluceau B. P. 105 78150 LE CHESNAY, FRANCE ABSTRACT. This p8.per f1arkov chains. and shaH how SC;:J.les. studies these chains arise In the second part chai ns to some aspects of perturbation theory applip.d to In the first part we introduce the notion of agregated chain I in the cont8xt of perturbation and time we study some applications of perturbed t1ar\ the ReI iabil i ty of large scale repairable systems. In the third part we give some applications to optimal control. 265 Properties of A Clearly A1 0 i. e. the zero off-diagonal elements of A are non 1s an eigenvalue of A with eigenvector negative. 1. Also In general however there are several eigenvectors associated wi th the zero eigenvalue of A. After reordering the elements of E one can get the following figure for A A state x 1s either recurrent (x € R) or transient (x € T). I f x is recurrent we will denote by x its recurrent class (independent block in the preceding figure), We will denote by m the number of such blocks. The events Ax 1 is absorbed 1n the class th~ x" function defined by q-(x) These x m, funct ions are eigenvalue of A. Aq; ~ O. ~ : ItX n make a partition of E. We will denote by q'X = P right X• . Clearly eigenvectors associated wi th the zero 267 1 P. For a 1 2 in' D.1 2h 1 Markov RO)dA f Residue of R(A) at A (Ai) J (Ai) }"R(}..)dA - Ai P i generator A we have L. 1 A, }P .• (A 1 Do 1 O. This means that the eigenvalue 0 of A is diagonalizable (no Jordan blocks). Moreover one knows an explicit full rank factorisation of P (mean ergodic theorem) lim lR(}..) P }.. . . o So the m right eigenvectors q; make a basis of Ker A 1 the left eigenvectors make the dual basis o y -;t x We have the direct sum decomposition E n With Ker A + ~ (A) respect to R (P) + any basis Ker P. compatible with operator (P - A) is represented by the matrix: Ker A R(A) 1 0 Ker A 1 R(A) 0 (P - A) -A this decomposition the 269 Let us set A' P AlP. 0 Then the resolvent of Al may be written 0 deduce \'le P that th~ spectral projection R for t)le eigenvalue 0 of P may be written A, R pl + (I - P) ,.M where pl Q 1 Q 1 t11 QQ and Q and 11 are obtained from the f·1 M agregate A 1 '" M A1 Q in the same manner as Q and t1 were from Q.M. spectral projection P for A admits the factor i za Uon P 1 0 the number of recurrent classes Df the agregate II ,. the The redL;C0.d resDlvent of Al is : o R where 31 is the reduced resolvent at zero of A l' One ,,1 has .J o • and R{P) may be decomposed into a direct sum : p (2) This decomposltion Is similar to (1), Since pl may be interpreted as the spectral projection for the zero used p1A P elgenv~lue of a Markov chain it may be to aggregate a new generator A2 as done above wi th p, Tn other words 1 2 '" MNA / 2 .R (P') may be interpreted as the generator of a neh' chain QQ. The process may be iterated, Let us eX3mine for insbmce the previous example. 271 we obtain the following system of equations A V 1) 0 o a o ... (1) (I - This system V. PV. 1 0 P) V 0 may solved be easily For S(-A )V .. + thanks instance the 1 o 1 and the second one impl ies -APV P( ii) We etc ... o PA P V + 1 + 0 P 0 f to decompos i tion the first equation gives 0 obtain the aggregate Markov chain PA P of the preceding section 1 and P(li) uniquely defines V ii) also PV o o S{-A )V (I o The above part the determines SA 1 process 1 is V0 + of V, in R because the computation ( 1-P) Sf. interesting of (Va ,V 1 • ... ) is made in a decentral ized-aggregated way : first we compute A 1 and then we solve the aggregate system. r-lore generally the computation of €:k (Ek,\ - {Ao + €: 111 )}-l may be done by recurrence on k. Let us consider the case k=2. To compute Va one has to solve the system (i) AV (ii) AV + {iii} -'wo + o 0 0 a 1 AV 1 0 0 II V a 1 + AV 1 1 + f a 273 \~e will denote by E the matrix E '" [ell ..• I e ] . m 1 The eigenvalue equation may be written where diRg(A~>""X~). II' X By identification of the factors of z, we obtain A Q o The first 1 + A E En 1 1 ~quation is trivially s~tisifed if 8 is obtained from Q by a change a variables E '" QT with T invertible. I f we multiply from the left the second equation by MI we obtain f1 Q T ",' and so vanish. known we if ~ggregate choose T the matrix operator M A1 Q that Is T These in formal which diagonalize (M A1 Q)T '" II 1 theory if the (if possible) the we see that the z-term calculations show the following perturbation described above then -1 important eigenvalues split in fact well the form they are solution of an aggregate eigenvalue problem and. also a bi'ls is QT of eigenvectors of Ao may be constructed made of limit of eigenvectors for the perturbed eigenvalue problem (the first order eigenvalues fix the zero order eigenvectors). For perturbed eigenvalues of Markov chains one is mainly interested in computIng the limit of me Where mE: is the invariant pr'obability measure (left-eigenvector) corresponding to the zero eigenvalue of Ao + E:A,. The most common situ3tion is the case of a generator A0 with a block-diagonal structure with weak interactions between the blocks. In ego [5J it is shown that lim m£ is -':l.Ssociated wi th the eigenvalue zero of the aggregate 1-1 Al Q , th:at is m , perturbed eigenvalues split and lim m£ Invariant probability measure of MAl Q. E-+O ,. mWhere m is the 275 Ql Here Ai eigenvalue is the of A1(O) total = PA,P projection associated wi th the unperturbed for the series A'(z). (When 0 is an eigenvalue of A'(z)/Im Q(z) then one has to define Q1 (z) b o The operators 1 y 2i1r 1 Qi Q () 0 (z) z . (f 0. - (0) are holomorphic subprojections of Q(z) which reduce A(z) and the characteristic polynomial of A(z). (which is p('\,A(z» r ITo i'=" thanks pC'\,A(z) I 1m This to lj) may be factorized once more 1 Q. (z)) P(A.A(z)/Qi (z». 1 "process of reduction" eigenvalue of the first (Kato [9]) may be pursued whenever an , 2 term of the aggregate series A (z). A (z), ..• is diagonali zable. In [2J, [5J. it is shown that this is always o of a perturbed Harkov chain. In particular, if true for the eigenvalue An the n-th aggregate chain admi ts ,\... 0 as eigenvalue with multiplid ty m then there n repeated eigenvalues of A(E) of the form Snh + o(c ). In the next paragraph we will exist m derive a simpler way to compute a perturbed eigenvalue, for systems with particular structure. 277 Assume the components are in "parallel" mode that is the system fails only when the two components are failed. The reliability engineers are interested in three parameters : = Avai13bility Reliability MTT!? = P (the system works at time t) l-P(X P (the system works between 0 and t) = t P(Xs 0) ~ s 0, ~ t} Hean time before the first failure of the system (the ini tial state being the state where both components works). ylhen the system is large (the number of components is more than 10) an exact computation of these parameters is not possible ; a perturbation method will provide approximatIons for then. 2.2. Model of the system The large scale repairable systems involving a repair poli'cy with the constitution of queues can in the exponential case be modelled by a Harkov chain X with the following properties : t There exists a partition of the set E of states in subsets G i E and the following hypothesis are made Hl : the only nonzero transitions occurs between neighbooring subsets (from G i to 01-1 or H2 : from any state e in G t (1 > 0) there exists an nonzero transition to the subset G - • i 1 H3 : the largest of the transi tion rates smallest of the transItion rates G i H4 Any state e in G 1 + -+ G+ 1 1 is much smaller than the G- , i 1 (I ) 0) can be reached from at least one state In G - • 1 1 279 The submatrices constituting A have the following properties a) Ai (i 0, k-l) are of dimension (01+1' their elements are between 0 ni' and 1. b) 8 (i '" 1, k) are of dimension (oi-1' ni' ; their elements are greater than 1 one; moreover the sums of each column is strictly positive by H2. c) The di agonal el emen ts are such tha t the sum of the col umns 1s zero. Di (d O,k) can therefore be expressed as a function of Ai and B , 1 (1 D.(O) Let Di d) 1 (i O,k) The sum of each line of Ai (i O,k-l) is non zero by H4, 2.3. Approximates for availabilities Availabilities can be expressed as function of the stationary probabilities of the Markov chain. The first result gIves approximates for them when E is small. Theorem 1 Let p.(e) 1 (1 O,k) be the column vector of the stationary probabilities for the st.ates in G. 1 and let Ql{E) (1 = O,k) be the columns vectors defined by Q (E) o 1, 0i (E) E i j=1 IT 1=1 (D-: (i J A j-l ) 1 ,k) 281 o Ci 0, k-2) Dt(E) is inversible : moreover From the last equation one can deduct substituting in the preceeding one Let 0' k-2 (E) By induction one gets then D' i+1 (c) P i+1 Ce} (1 o. k-1). An explicit formula for P.(s) is then: 1 P. (IE:) 1 E i (0' -1 j p (r:::) o (i o t k). 283 Let A' be the submatrix of A corresponding to pC the complementary subset of P in E. The structure of A is similar to that of A : D~ (E). B'l EA~, DI, (d, - 81 2 Let Qt(E) be the vector whose components are the mean time spent in the states of Pc between 0 and t. Qt(c) is solution of : Let 11m T t,""rn T is solution of AI T Ti (i = 0, k') is a column vector of dimension n l , Solving the system as in?, i one gets : 1, k') with 0,,-1 1, 1(1) i f1oreover I ( ) '"" C~ as E '"" 0 C' o 285 Let G. be 1 the subset. of states where components are ('?ither awaiting repair or in repair). To define a state in G < ••• < enough to give the list j, (~) 1 of such states is not Horking $ r, i t is i ji of components in repair. The number i • In the subsets G i) i r. it is necessary to know the components in repair and the order of the queue. There are n-r A i-I' H3 such states. The corresponding process is easily seen to satisfy assumptions H , H , 2 1 and Hit and approximates ca.n therefore be found for the avai 13bil tty of the system. Let p us [ 11 deftned 12 by n get for < example < [jl the probabil i ty is not available. I f i ( ( jrJ the ) that the subset r a state e of P is list of the components being repaired and by the order of the queue for the components awaitjng repair. Applying Theorem 1 to each of these states and summing one gets the following approxim3tion for the unavailability of P : Q (i - r)! IT A" j6? J Similar results can be obtained for various repair policies (see [14] fOl' det3ils). Approximat ion easily be obtained for the reI iabil i ty pElrameters of a system can therefore for large scale systems by transition graph of the associated MarkOV Chain. a simple Inspectrion of the 287 [11] R. PHILIPS, P. KOKaroVIC. A singular perturbation approach to rnodclline and control of Harkov, chains IEEE A.C. Bellnan issue, '981. [12J H. SnON, A. ANOO. Aggregation of variables in d}7lamic syster.lS, Econometrica, 29, 111-139, 1961. [13J J. KDtENY J 1.. SNELL. Finite :f-.1arkov chains J Van Nostr:md, 1960. [ 14 J O. lvlURDN. Evaluation de pol i tique 5 de maintenance pour lU1 sys teffie complexe, RIIID, voL 14, n° 3, pp~ 265-282, 1980. [15] S.L. CAHBELL, C.D. l'-1.EYER jr. Generalized inverses of linear transformations. Pitman, London, 1979. [16J TKIOUAT. These Rabat a paraitre. [17J J.P. QUADRAT. Commande Outils et [un ~bdiHes ortir.ale de chaines de ~.farkov perturbces 1'-lath. pour l'automatique ... t3 edition CNRS 1983. J.P. QUADRA..T Optimal control of perturbed, lo:!arkov chain the lTu11titime scale case. Sin::;ular pertubation in systems :md control. CISfv1 courses and lectures n° 280, Springer Verlag 82. [19J F. DELEBECQUE, J.P. QUADRAT. COIltribution of stochastic control, teaJ11 theory and singular perturbation to an example of laY,Ge scale systems Management of hydropower production. IEEE AC avril 1973. 289 PLA..1>.J - Introduction 2 - Notations and statement of the problem 3 Perturbed Markov chains 4 - Reviel'l of controlled l!arkov chains 5 - Control of perturbed Markov chains 6 - Example and application. 1 - INTRODUCfION Stochastic or deterministic control problems can be reduced after discretization to the control of }·'-arkov chains. This approach leads to control of ~,!arkov chains which have a larp:e number of states. An attempt to solve this difficulty is to see the initial ~!arkov chains as the perturbation of a simpler one, and to design algori thtts lvhich use the hierarchical structure of more and more aggregated P.lodels, described in the previous paper of Delebecque, to increase the computation speed of the optmal control. The two time scale control problem (actualization rate of order £) is solved in Delebecque-Quadrat [6J , [7]. The ergodic control problem when the unperturbed chain lUiS no transient classes has been studied in Philips-Kokotovic [19]. In this paper we give the construction of the complete expansion of the optimal cost of the control nroblem in the general multi-tine scale situation. This presentation is a very little improved version of 0;uadrat [17J , [18J. For that, we use three kinds of results: - the Delebecque's result discussed in the previous ~aper. - the realization theory of ~licit systew5 developed by Bernhard [1J. Indeed this method gives a recursive mean of co~putinr the complete cost expansion in the uncontrolled case. the r,lille -Veinott [10J way of constructing the opti!!1al cost expansion of an unperturbed ~!~rkov chain having a small actualization rate. 29' The conditional e).-pected cost knoh'ing X(O,w) is a defined by : wx : E[j (lI.)) I X(O ,w) = xJ, Vx E Ixl- Z vector denoted h' (2.2) The Hawiltonian is the operator IRlxl h lR -+ w Ixl [m - (1 +A)iJ w + c where i denotes the identify of the (2.3) (lxi, Ixl) - matrices set. Then w defined by (2.2) is the unique solution of the KollTDgorov equation hew) o (2.4) 2.2 - In the perturbed situation the n-tuple defining the perturbed ~nrkov chain is (T, X, 8, m(e), c (e), A(c)) - tJ is now the space of the perturbations ; in all the following it is m,+ ; m(E), C(E), A(E) have the same definition as previously but depends on the parameter variable. E E: a , and we suppose that they are polynomials in this l'/e denote by d the degree of a polynomial and by v its valuation (the smallest non zero power of the polynominal). In the follO\>,ing d(m) = 1, vern) = 0, V(A) = v(c) = d(A) =i . From this particular case the general case can be understood. The Hamiltonian of the perturbed problem is denoted by h(w,c) = [m(E) - (1 + A(c)) iJ W + C(E) (2.5) 293 Then with H(W) [lvl-(I+J\)JW+C, (2.11) \vhere c (~, I the identity operator A n t: IN, cn the operator lx' _[:ec~or~) are o i 0 0 0 .... 0 ... . 0 i ~ th IzI-blocti~~ J 1 an e).:pansion of the cost is obtained by solving HeW) o (2. 12) f\brever the sequence (\"i' i E: n.J) can be computed recursively. These two resul ts \."ill be shown in part 4. 2 . .3 - For the control prahl em u (T, X', 'U-' rn, \ole need the introduction of the n-tuple , A) - 'U is the set of control l'Ihich is here a finite set. luI denotes the cardinal of '4- . Its generic element is denoted by u. - m denotes the (I'l..d ,lxi, Ix!) tensor of entries mU xx' go in Xl, starting from x, the control being u. c denotes the control bein~ (lUI, 'Xl) u. matrLx of entries c U x , the probability to the cost to be in x, the 295 h*(W) = min hU (w), Vx £~ W X U (2. 18) x The optimal expected cost w* is the unique solution of the dynamic prograrrnning equation (2. 19) 1m ......... ~.. I ...Ui......... policy is given by x 2.4 - The perturbed control problem is defined by the n-tuple Its interpretation is clear from the previous paragraphs. . u * *E U analogy the notatlon H (W,E), h (W,E) , W ,H 0'1) are clear, but we need a definition of H*(\~. For that let us introduce the lexicographic order, ~ , for sequences of real numbers, tlmt is : By (Yo'y,,···) ~ (yo'y"yZ"") is true (if Yn = yin' Yn < m <=> then Ym ~ ylm) Ym € (2.20) IN. We denote by min the min.imtnn for this order. Then we define H* by : * ..,. H (\\1) = min x Hu.X Oil) (2.21) 297 ' with nu - i a O :: (3.4) ao~ ~~ E ~ml 0 aO ~ ml~ J [! ]}!+1l Gi aO CR. + 1) blocks (3.6) 0 0 G aO [0\\ F (3.5) (.2.+ 1) blocks (3.7) blocb O--OJ (2+ 1) blocks (3.8) Indeed if W is a solution of (3. 1) yn = (Wn , n +1' •••• J Wn +9.) \'of is a solution of (3.3). Conversely if \>Je t~ is a solution of (3.3), by el~ination of the variables y see that W satisfies (3.1). Let us denote by £ the space m.1X'1 x(2 + 1) in 'vhich lives y. 299 Now the fact that ?'l (J) :J?'l (E) implies that the sequence Wn is lIDiquely defined. We have proved the Theorem 1 : The solution of : WE h(W,E) : ::: (r.1(E) - i A(E) \'I + C(E) = a (3.13 ) admits an e:x.)?ansion WeE) lvhich is the unique solution of H(W): = (M-I-A)W + C= 0 ( 3.14 ) Jvbreover W can be computed recursively by constnlcting the implicit system realiz.ation of y -1 o (3.15) \.;here E, F. 0, H are defined in (3.5) to (3.8). This implicit system has an output uniquely defined and it admits a strictly causal realization. A specific algorithm is given in Tkiovat [16J 4 - REVIBV OF CONTROLLED f','lARKOV mAINS U Given the controlled Markov chain n-tuple : (T,X, U , m J C U J A). The opt:i.nal conditional expected w* cost is the unique solution in w of the dynamic programming equation h*(\'I) X u min [(m - 1 - 1.)\<1 + cU]x u O,VX(Z. (4.1) This result can be proved using the Howard algoritrun Z Step 1 : Given a policy s f.U linear equation : , let us compute \"', solving, in w, the 301 The existence and the tmiqueness of a solution in \V of (4.1) folloNS easily from this result. 5 COm-ROL OF PERTURBED J-.1A.RKOV Gl!\INS U Given the perturbed controlled ~larkov chain n-tuple (T,X lU ,a , m (A) J cU(e) , A (E)). TIle optimal cost is the unique solution in \'J of the dynamic programming equation : h *..c\11 ) d :: ':X: u nUn u [(m (E) -1 - A(E) ) \11 + U C (E) ] X 0, Vx IS X 5 ( .. 1) We have the Theorem 2 : The solution of (5.1) denoted by 'II*E admits an eA~ansion in E denoted by W* (E) lvhich is the unique solution in 1'! of the vectorial dyna- mic nrogramming equation : H*.X 0'1) == min [(t-P- I - /1.)1" + U eU ].x = 0, Vx E""" .... (5.2) + Let us remember that min means the minimum for the lexicographic order on the sequence of real numbers. The solution W* can be computed by the vectoriel HO\vard algorithm ~ x Given a policy s IS U ,let us compute tv using the results of part 4 Hos (w) = 0 (5.3) Step 2 : Given a conditional expected cost Wl let us :ir.1prove the policy by computing + U min H U Ne change .x ( 5.4) 0'1) U s (x) only if H .X (W)< O. TIlen we return to step 1 . 303 By (3.15) we know that the order of the matrix E is smller than (v CA) + The entry C .+ is equal to zero for n ~ d (c) - £ - 1 nH 1 1)!:t1. We add IZj new states to z, denoted by z'with : z'n+l = Wn • With the new states (5.8) Zll = (z, z') the second part of (6.6) can be l>.rritten : (5.9) (5.9) has the form J'z" n a (S .10) with J' an observation matrLx of the dynamical system of state z", It n follows by the Cayley-H8..!~lton theorem that if (5.10) is true \In : n ~ n > d (c) then (5.10) is true Vn> dec). The theorem 3 is deduced easily from this result. Remark: In Tkiouat [16] a method is given to compute for each state x a bound q(x) on the size of vectors on which ,,,e have to make the vectorial minimization. 6-Example and application Let us show on a trivial 2 time scale example hO\II these results can be useful to design fast algoritlnn to solve stochastic control problem. Let us take the most simple example 305 using the particular structure of \Ii, and the results explained in Delebecque, that is \'1e compute solution of: and P2 solution of : . P,- ['] 1 1 = Then \ye compute the aggregate \<Jhain transition matrix 0 P, 0 A= 0 0 0 0 A Pz 0 0 and the aggregated cost C, = P, [CC']' z c3 z = Pz rtc C 1 4 307 Then it is possible to improve the strategy by minimizing in u all the entries of the 4 - vector \'Ie have seen that in this process we have never solved a linear of size 4 but three systems of size 2, and 4 minimizations. generaly when the matrice mO has a block diagonal structure, this perturbation method ir.1prove the speed of the 1-I0Wllrd algorithn. In the best situation \<Je can obtain a [Z [2 algorithr.1 to solve the probler.l. ~bre In this discussion we have only compute the first tem. of the expansion "there the vectorial minimization defines completely the control after computing only the two first terms of the expansion Wo and W . 1 The algorithm is complicated to be implemented. It is done inTkiouat [16]. This method can be applied for discrete version of the followinrr diffusion process : dX t = b,(Xt,Yt,Ut ) , dt + dYt = £" b 2 (Xt ,Yy 'Ut ) dt t a 1 (Xt ,Yt , Ut ) ill1 1 + ~ ? aZ(Xt,Yt,l't J d1'J~ that is diffusion process having b~o time scales. Some dam management problems can be described in this fOTQalism see Delebecque-Quadrat [19J 309 [11] R. PHILIPS, P. KOKafOVIC. A singular perturbation approach to modellin!) and control of [12J H. SHDN, A. ~ ANlx)~ chains IEEE A. C. Bellr.tan issue, 1.98 t. Aggregation of variables in dynamic systerns, Econometrica, 29, 111-139, 1961. [13J J. KBIENY, L. SNELL. Finite Markov chains, Van Nostrand, 1960. [14J O. MURON. Evaluation de politiques de maintenance pour un systeme complexe, RlRO, vol. 14, nO 3, pp. 265-282, 1980. [15J S.l. CAJillELL, C.D. ~~R jr. Generalized inverses of linear transfor- mations. Pitman, London, 1979. [16J TKIOUAT. These Rabat a paraitre. [17J J.P. QUADRAT. Corrunande ortir..ale de chaines de r·(arkov perturbees Outils et 1-1odiHes Math. pour I' automatique . .. t3 edition CNRS 1983. [1SJ J.P. QUADRAT Optjrnal control of perturbed, ~~rkov chain the Jnultitime scale case. Sin!:Ular pertubation in systems and control. CISM courses and lectures n° 280, Springer Verlag 82. [19] F. DELEBECQUE, J.P. QUADRAT. Contribution of stochastic control~ teaM theory and singular perturbation to an example of large scale systems }'1anagement of hydropower production. IEEE AC avril 1978. 311 but they words. may weak be the network. assuming forces coupled in determine a the slow time-scale. long-term dynamic In other behavior of The aggregate model captures this long-term behavior by that its the nodes "aggregate" external strongly connections node strength of to their lose connected connections. to The internal other: same connections identity and "aggregate" weak in act nodes external each as a area single through connections weak have a negligible effect on the short-term behavior of the network which is modeled with the areas disconnected from each other. The outlined methodology is a analysis with connections E~O in model two of respect [1]. time-scales. the ne two rk parameter is t ~ and singular [: obta ined tIt. perturbation representing the weak itself groups is the A singularly by transforming the set of aggregate transformation algorithm which scalar of a This asymptotic analysis investigates the limit as variables into a The to a result (slow) and local found nodes areas. r igina 1 s ta te 0 (fast) variables. automatically into perturbed The by a computer detai Is of this grouping algorithm and its applications to large power systems can be found in [6}. A summary of the algorithm is given in Appendix A limi ta tion individual of the external proportional to methodology connection the small presented is assumed parameter c. in to This [1] be is c. tha teach weak. that formulation is, excludes more common situations in which the weakness of external connections is due to their sparsity. In many applications. individual external connections as as. than, are strong connections. In connections. Relating this paper. the but we much allow sparsity sparser strong pattern but of a the internal sparse network external with time scale properties. we extend the asymptotic analysis of larger class of our analysis networks. also [I} its to a From the graph theory point of view [7] contains a new result on the dependence of r the eigenvalues of a graph on its sparsity pattern. We and ~. first and characterize show that the the sparse time-scale properties similar connections. a singular network topology external by two parameters connections induce d the to those of dynamic networks with weak Although the aggregate model depends on two parameters, perturbation model is still obtained by the same decomposition-aggregation transformation. An application of pattern of a large network. our grouping algorithm. system of these is to find We show that the unknown sparsity this can be determined by The network considered represents the Western U.S .• In this case, results but with connections the power treated as 1 or O. the algorithm produces a grouping into nine areas quite 313 £. I min a,i max a, i It is desired (2.:2 ) that a partitioning into areas be such that even the worst node has mare internal than external connections. that is. the .!lode parametet. d =: -E 1 c I£. (2.3) should be small, may violate d«l. this 1n a la rge ne twork., requirement. E can be used and c. al fu.rther discussed be example in Section 5. The a-th area connections if to in the Then a small number of nodes values of c 'I ai parameter d. This wi 11 the large power network define the conjunction node with sparse external has average and internal dense (2. '1 ) where E 'Y a the total number of external connections in area a, Y! the total number of internal connection~ in area The least favorable areas are those with the connections yl, and the densest exter.nal connections Y I mi.n a Cleacly, y I I Y ~ ill £. {Y 1 } , a -E Y max { E} 'Va . 11. sparsest yE, internal where (2.5) 11 is bounded below by ill £. I , that is. I (2.6) where ill min {mal, a m Il the number of nodes in area a. Our goal is to find a partitioning in which even the worst area has more internal than external connections, that is, the "area parameter" 315 , Area 1 /' ( ----- Area 2 /--1---\ 1 1 I I 1 I I I I 1 I I I 1 I I I I 1 I I 1 I I I I I 1 I 1 I I I l, - - - - - _ . / 5 I ./ 1 o Node - - - Connection Figure 2.1 A lO-Node. 2-Area Network Area 1 r---------, 11 l0 Area 2 r---------~ 2 3 4111 2 3 4 0 0 0>--1-1-<0 0 0 0 l 1 '--------_./ '---------_./ o Node - - - Connection Figure 2.2 An B-Node. 2-Area Longitudinal Network is still applicable, but with less accuracy. Additional examples of sparse networks are given in [8]. 2.3 Dynamic Networks Al though clarity we To stress our ana lys is present the it applies in the structural to other context of aspect of types of networks. for electromechanical networks. the problem. we make the 317 where K! is the maxma internal connection matrix of area a. Since an area itself is a dynamic network, the i-th diagonal entry of is K! -c!i' that is the sum of the other entries in row i. off-diagonal entries of Kl is 2yl The sum of the Although the diagonal entries of I . a. a K are much larger ln magnltude than any of the off-diagonal entries. 't . . . . . Ka is not dlagonally dominant because lts rank is rna-I, due to a zero eigenvalue with the eigenvector u [1 a This zero mode disconnected that, (2.11) 1 is from the the equilibrium rest of when isolated, area a the manifold = x~, ] the area when a expresses the fact a = 0, whenever all x = j 1,2, ... ,m . a . . E . 2 . partition of K lnto r block matrlces KE , a, 1 . 'l'h e correspon d lng 1,2, ... ,r, It is at equilibrium, its storage potentials are the same, x~ B of network. i. aB K~B contains the connections between areas is such that a and B. We note that KE is a diagonal matrix whose i-th diagonal E aa entry is -cai' the negative of the sum of the other entries in row i. It is also helpful to observe that the sum of the entries of K!B is equal to the number of external connections between areas a and fl, and t ' I . E . hence d«l and 0«1 imply that Ka lS dense and KaB 16 sparse, for all a and B. 3. TIME SCALE SEPARATION We areas now demonstrate and behavior their of decomposes sparse the how the equilibrium manifold external connections overall dynamic network. The this slow motion all is, the nodes because the dense equilibrium. areas This is connections During negligible due the to fast the motion, sparsity exchange becomes significant over a motion of the two- time-sea Ie variables. motion areas behavior as is the areas. Intuitively. internal potentials in the same area to rapidly equalize. area network During in the same area are "coherent." the areas move as IIrigid bodies." fast the two-time-scale into the fast local motions of the storage elements within the same area and the network-wide slow motion of are property of induce a heavier made allow bodies by a new is the node to reach an exchange external longer period, rigid apparent the local motions that is, the of that with other connections. that is. slow. choice of the This s ta te 319 We have thus defined a transformati.on into the aggregate and local variables y c z G of the original x . state {3. B} The inverse of this transformation is explicitly known x (3.9) where G+ is block diagonal (3.10) L G+ a. m -1 -1 -1 ma. -1 -1 -1 -1 ma.- 1 -1 -1 -1 a. (3.11) ma. -1 In the new variables y and z. the network model (2.9) becomes y All A12 y {3.12} z A21 A22 z where All CKEU A12 CKEG+ (3.13) A21 GKEU A22 G(K 1 + KE}G+ x 321 I All/{£. 0). All (3.21) I A21 -= AZ1/{£ d), so that the norms of All' A12 , 1\21 This scaling reveals that the model (3.12) the parameter 0, and 1\22 are all 0(1).' is singularly pecturbed by that is, (3.22 ) In the slow time-scale t (3.23) S the same model has the 50-called explicit singular perturbation form (3.24) This well-known results. One of model allows them is us the to make following use Theorem of many [1,9,20] existing about the < 0*. time-scale properties of (3.24). Theorem 3.1. a < d },.6 ~ There exist 0* and d* such that for all 0 0 =5 d*. the system (3.24) has r slow eigenvalues = },.[ (All -1 dA12A22A21) + O(6d)] (3.25) and n-c fast eigenvalues },.f and, = }"[A 22 hence. + O(od)]/o I },.61/1},.f I (3.26) O(o}. Furthermore. the slow fast subsystems of (3.24) are y( 0) , (3.27) and 323 is based on Theorem 1 in [211. In this case. the s low-fas t time relation (3.23) is ts vb tt' the eigenvalue approximations (3.25). (3.26) are only O{vOd}. The state approximations (3.27}-(3.30) are also O(vbd). but because of the oscillatory character. are valid only for a finite time interval. Our development sparsity and remainder of thus far time-scale the paper properties we decomposition-aggregation has demonstrated a of dynamic show how these procedure relationship between and networks. properties for a are In used grouping into the in a areas when the partitioning is unknown. 4. DECOMPOSITION AND AGGREGATION The presence of the construction of the node parameter d in Ao several approximate models foe and 1\22 suggests the slow and fast subsystems. 4.1 Aggregate Models Since centers of the states of the inertias, we treat slow subsystem (3.27) as an (3.27) are aggregate the area model. To simplify (3.27). we first express the terms in AO in a clearer form: (4.l) which is derived in [1], and 1 where using (3.2) and (3.l0). (4.3 ) Therefore. the slow SUbsystem can be rewritten in a form similar to that of the original system (2.9). namely, (4. II) First. we note that Ka and K! are symmetric. Second. the row sums of each of these two matrices are zero. This can be verified by post-multiplying Ka and K! by the r-dimensional vector 325 which makes the effect of the internal connections more exp 1 ic itof KI is this simplified form The computation of even a time-consuming for large networks. To avoid it. and also to improve the accuracy of the aggregate (4.?). we propose a compensated aggregate (4. g) where c is a compensation factor chosen to minimize the discrepancies between the The eigenvalues actual [6], slow while require much to a (4.9) and are known eigenvalues Ma' straightforward leads of Ka to and calculate. additional major: the the from TheI:"efore, of the slow the the as CMa Ka of c examples accuracy o( algorithm --1 of choice the eigenvalues. grouping eigenvalues computation and. improvement actual the are does will not show. compensated aggregate over the rigid aggr:egate (4.4). 4.2 Local Models The fast individual subsystem variables reference states. (3.28) in the represents areas the with r:elative motions of the respect to their local The local motions can be simplified by recognizing that (4,10) where (4.11) (4.12) are 0(1). Since A 220 is a blOCk-diagonal matr:ix. the fast subsystem (3.28) decouples i.nto r a. 1.2 •••.• r, areas, and are the obtained frequencies 220 by discarding the of the modes d small. (4.13) where zfa. are the area a. states of Z and A These models then for local. models of = d'lag (A 1 220 connections (4.12) will • . . . • Ar ). 220 between the be somewhat lower than those of the fast modes of the full system [13. p. 331]. A correction for the mode shirt is to include a. blocks. A , of A in the compensated local models 22d 22d the diagonal 327 Table 4.1 Eigenvalues of Slow Models (IO-Node Network) Matrix Eigenvalues Exact K 0.00. -0.61 Complete Aggregate AO 0.00. -0.63 All 0.00, -0.80 Slow Model Rigid Aggregate Table 4.2 Eigenvalues of Fast Models (IO-Node Network) ~atLix Fast Model Eigenvalues Area 2 Area 1 Exact -3.34.-5.00,-5.00,-6.16 -3.00,-5.00.-5.00.-6.90 K Past Subsystem A22 a. A220 + -3.42,-5.00,-5.28.-5.89 -3.00,-5.00,-5.60,-6.00 Compensated Local Models dA and 4.2. Note to 0. 22d A~20 Local Models ass igned 3.00.-5.00,-5.00,-6.89 -3.31,-5.00,-5.00.-6.00 the -3.00,-5.00.-5.00,-5.00 -3.00, 5.00,-5.00,-5.00 that the areas. fast eigenvalues of K and A • while 22 depend on the other areas. The in general, time scale separation is \-3.00\/1-0.611 The relative I: igid errors aggrega te and 4.9 in 0(1/0). the the fas t slow eigenvalue approximations by e igenva lue approxima t ions by the the loca I models are of Oed). the slow 0.61/0.80 the exact aggregates. the and the relative errors of the approximations by By choosing c and fast subsystems are of O(od). 0.76, the eigenvalues of the compensated aggregate are slow The compensated eigenvalues, which is achievable eigenvalue approximations local models are of results aLe verified for this example. of 2 0{d ). for the fast Thus, all 2-area subsystem by the time-scale 329 experience motivates the modification of the theoretical results for practical applications to large networks. The power network considered is a 411-machine, 17S0-bus. 2800-line model of the Western Systems Coordinating Council (WSCC) system (Figure 5.1). The total number of nodes is machines are shown as dots on the map. 411 1750 + preserve allow 2161. the the The overall use of nodes bus sparsity the sparsity-based reduced not are in to order thus network structure, and grouping algorithm, which the of Most of the is summarized in Appendix C for completeness. The WSCC dispersed system consists generation centers. of sites. several major Connecti.ons are load dense centers about with the load The centers are interconnected with a few long transmission lines along system. the Pacific Coast and around forming transmission the lines so-called are the eastern portion of IIdonut " sparse, but pattern [14]. each individually K was constructed the These long strong. and cannot be modeled as E-connections. The 2161x2161 storage. connection matrix using sparse Then all the non-zero Off-diagonal entries of K were set to 1. while the diagonal entries were set to be the negative of the row sums of the oCf-diagonal entries. set to 1. The inertias at the nodes were all The computation time required to calculate the 15 slowest eigenvalues and their eigenvectors using the about 6 minutes on a VAX 11/760 computer. to use partition WSCC the 9-area procedure. into several different partition shown in sparse algorithm was The eigenvectors were used numbers Figure of areas. 5.1 to We will illustrate the The same technique can be applied to other partitions. The algorithm partitioned the 9 areas of WSCC along boundaries of well-derined uti.lity geographical systems. are denser the 9 areas than are Since the regions the connections partitioned approximately connections within between along corresponding the different boundaries of utility uti.lity to systems systems, sparse connections. The 9-area partition will be compared to the II-area partition based on actual connection strength in Section 7 to identify the weak connections. Let us pa rameter now examine Sand d. connections for external Table 5.2. . The numbers o( nodes, and internal and external the areas are shown in Table 5.1, connections between the areas are and the numbers of shown by the Ka matrix in In this system. some internal nodes have . connectlons. that 1S. £ we still have small violates the sparsity pattern using the area and node 1 = 1. b = 20/104 the requirement d«l, Since m = 104 0.192. . 1S single internal E However, with the assumption -E y. much larger than »c E c 2, d ~ 2 is no longer 331 Table 5.1 Numbers of Nodes, and Internal and External connections, for 9-Area Partition of WSCC Number of Average Number Number of Average Number Internal of of External Internal External Number Connections. Connections. Connections, of Nodes, Connections, Area E E I yI m (ca,)ave (ca)ave Yo. a a n 1 274 352 2.57 20 0.073 2 104 124 2.38 13 0.125 3 176 221 2.51 18 0.102 4 234 290 2.48 17 0.073 5 166 208 2.51 3 0.018 6 283 357 2.52 11 0.039 7 405 542 2.68 15 0.037 B 232 306 2.64 5 0.022 9 287 371 2.59 12 0.042 Table 5.2 Aggregate Connection Matrix Ka Column Row 1 2 3 4 5 6 7 8 9 1 -20.0 0.0 1.0 10.0 0.0 9.0 0.0 0.0 0.0 2 0.0 -13.0 3.0 0.0 0.0 0.0 6.0 0.0 4.0 3 1.0 3.0 -18.0 2.0 0.0 1.0 5.0 3.0 3.0 4 10.0 0.0 2.0 -17.0 0.0 0.0 0.0 0.0 5.0 5 0.0 0.0 0.0 0.0 -3.0 0.0 3.0 0.0 0.0 6 9.0 0.0 1.0 0.0 0.0 -11.0 0.0 1.0 0.0 7 0.0 6.0 5.0 0.0 3.0 0.0 -15.0 1.0 0.0 8 0.0 0.0 3.0 0.0 0.0 1.0 1.0 -5.0 0.0 9 0.0 4.0 3.0 5.0 0.0 0.0 0.0 0.0 -12.0 333 6. NE'l'WORKS WI'l.'H SPARSE AND WEAK CONNECTIONS In sparse our presentation of connections, we have the area decomposition simplified the analysis results [or by assuming . k aB . . t h at eac h connect ton ij between node 1. tn area a and node J. 1n area B normalized with respect to the inertias is unity. To be more precise of areas, the actual in determining time-scale decomposition Of connection strength and inertia values have to be used. particular importance are the weak and strong connections. For many practical power systems, while the ucores" o( the areas aLe largely the sparse and dense connection patterns. the determined by boundaries of the areas are determined by the connection strength and inertias. In some cases, weak and strong connections may cause large perturbations in the area partition. from sparse connections may split For example, an area determined into several areas when actual connection strength is considered, because it possesses weak internal connections. As another example, areas determined (rom sparse connections may combine into a single area connection strength between the areas is strong. section is to pLesent because the external The pULpose of this time- scale decomposition of areas taking into account simultaneously sparse and dense connections, and weak and strong connections. We will summarize the weak connection results, analyze a special class of systems with sparse and weak connections, and use the WSCC examp le to discuss a gene I:a 1 a ppI:oach to ana 1 yze practical systems using the grouping algorithm in Appendix C. 6.1 Weak Connections The aI:ea decomposition results for networks with weak connections in [1] can be readily deI:ived [ollowing the analysis in Sections 2, 3 and 4 (or sparse connections. In the weak connection analysis, we use the actual inertia values and connection strength. system are ~ a . where m 15 t The network dynamic equations [or an r-area = 1.2, ... ,I:I and BFa when i, (6.1) . . . the lnert1a or capacltance of the storage element of ~ Xi' and k~~ is the actual connection strength between node i of area a and node j of area B. [-'allowing the notation of (2.9) and (2.10), we write (6.1) in the compact form 335 where (6.9) System (6.8) is a singularly perturbed system in the fast time scale t = t. and is similar to system (3.22) f except that the weak. connection parameter area parameter b and the node parameter d. In the slow time-scale ts = Et , f explicit singular perturbation form for sparse functions £" system connections. as both the the has (6.8) (G.IO) The ti.me-scale results as in Theorem 3.1. are now in terms of that for Theorem 6.1. 'I'here exists £* such system (6.8) has r slow eigenvalues "- s -1 f:A1ZA2ZA21) + "-[ (All 0(£ 2 t: instead all 0 < of t: d and 5. E*, <5 the )] (6.11) "- [All + O{E:)] and n-r fast eigenvalues l..f 2 l..[A 22 + O(E: }]/£ (6.12) I -I l.. [GK G + O(t}]/t and, hence. Il..sl/Il..fl fast subsystems of (6.8) are o (£) . Furthermore. the slow yeO) , (6.13) (6.14) and 337 dy/dt eoA f l1 Y ... COA 12 Z (6.17) dz/dt cdA f where tf 21 ~ y A Z 22 £.1 t and All ci{Eu / [£.1 0 ) , A12 A21 Gi(Eu / (£.1 d ) , A22 ci{EG+1 (£.1' O), (6.18) System fast (6.17) is a scale t time separation and time-scale ts = G(K + ti(E)G+/£.I. singularly perturbed where f the = 1 the product (to)t , cd produc.t denotes system f di weak (6.17) system expressed denotes the coupling. becomes in the' time-scale In the the slow explicit singular perturbation form (6.19) Theorems 3.1 and 6.1 can now be combined to obtain the following result. The~~. o < d 5 d* and There exist &*, d* and c* such 0 < t 5 c*. that for all 0 < 0 ~ 0*, the system (6.19) has r slow eigenvalues (6.20) = O(A l1 + O(ed)] and n-r fast eigenvalues (6.21) and, hence. 1)..6 11 I)..r 1 O(di). fast subsystems of (6.19) are Furthermore, the sloW' and 339 Table 6.2 Eigenvalues of Past Models (lO-Node Network with Weak connections) Fast Model Matrix Eigenvalues Area 2 Area 1 Exact -3.05,-5.00,-5.00.-5.09 -3.00.-5.00, 5.00,-5.18 K Fast Subsysterr A22 Compensated Local Models A + -3.05,-5.00, 5.02,-5.09 -3.00,-5.00,-5.07,-5.07 220 cdA From the the (1 22d A~20 Local Models -3.00.-5.00, 5.00,-5.00 -3.00,-5.00. 5.00,-5.00 results of Theorem 6.2 in which the product co denotes time-scale existence 3.05.-5.00,-5.00,-5.09 -3.00,-5.00,-5.00, 5.1B (1 of separation, multiple we time can readily infer the possible scales. Consider the case when some external connections are sparse, some weak and others both sparse and weak. Assuming that the c, 0« slow time-scale has a three-level hierarchy (6.26) in which 1 In this four time-scale s is the slowest time-scale. 3 system. a nested aggregation starting from t s can be used to obtain subsystems for the different time-scales. t 6.3 A Procedure to Ident i fy Sparse and Weak Connections in Large Scale Networks Large scale connections decomposition. for power power which networks together often contain determine the both time sparse scales and and weak area Information on sparse and weak connections is useful system planning and operation. setting of power system stabilizers and results in 'l.'heorem 6.2 only deal with the the sparse connections are also weak. such as the siting and protective relays. The simplest situation where For more general situations we propose to use the grouping algorithm to identify the weak and sparse connections. The first We will use the WSCC example as an illustration. identification step, the procedure grouping consists algorithm is of three applied to steps. the In the system with 341 Figure 6.1 An II-Area Partition of WSCC Figures S.l and 6.1 of the WSCC system illustrate that our 3-step identification weak approach connections. design of the The WSCC provides results Intertie valuable here would Generator information have on sparse been useful Dropping and large disturbances described in [22]. the connections and weak will also system planning, eKpansion and control design. be helpful and the Controlled Separation scheme for sparse in for Knowing future 343 where x~ is the machine angle. m~ the inertia constant. d~ the damping ~ 1 1 constant. the mechanical input power and v~ the voltage at node i 1 aB in area a. As in the linear case. we assume that the admittance Bij be tween node i in a rea Cl and node in a rea 13 is dense wi thi.n an area and sparse between the areas such that the node parameter d (2.3) and the area parameter b (2.7) defined according to the number of connections are small. Without the external connections. each of the r areas is isolated. The dynamic equation of area d is pi -d~x'?" + p,?" i =1.2 •••• , rna' 111 (7.2) Area d has a continuum of equilibr.ium points -a. (7.3) X a where Xo is an equilibrium point. Cd a scalar. and un as given in (2.11) (7.4) Equation {7.3} implies that if (7.2) is at equilibrium. then (7.2) is still at equilibrium if all the components of xa. are increased or decreased by an equa 1 amount. This is the egu i 1 ibr i urn proper ty and (7.3) defines the equilibrium manifold for area n. On this manifold, the potentials within an area are equalized. A second property is the conservation property that the state ya. m La. C1 n a. mC1 rnixi/rn • i=l m La. m~ 1 (7.S) i=l remains unchanged when m La. P'?" i=l d~1 1 Of O. i (7.6 ) 1.2 •...• rna' (7.7 ) 345 From these results. subsystems. It we is can also derive possible the to nonlinear derive slow various and fast aggregate and ana lys is for local models following the analysis in Section 4. The time-sea 1e decompos i t i on and aggrega tion nonlinear dynamic networks with both sparse and weak connections will a Iso make use of the conservation and equ il i br ium proper ties. The derivation can follow the steps in Section 6. 8. CONCLUSIONS The time-scale sparse dynamic method [1] approach networks for to the developed dynamic decomposit1.on-aggregation here networks extends with and of complements £-connections. the These new results are mote broadly applicable. since most practical large scale networks have the characterized d. Although they can using terms the be a in be readily nested removed of pa t tern an area cesults ar:e transformation. can sparsi ty for extended Furthermore, in a parameter networks of the manner in this .5 are time-scales. time-scales decomposition-aggregation assumption to and node pacameter two with multiple the simi lar paper and a a separation into to application assumed the of linear nonl inea r connections ne two rks wi th £-connections [5.1]. Using the descr ibing revea 1 Some the the of parameter s low and r:oles the networks node played cesults with f as t by also d. new aggregate dynamics the internal improve £-connections. are the The and proposed. local models Thes e models and externa 1 connections. models proposed earlier application of the for results to large networks is illustrated with a 2000-node power system. 'The sparse connection whose both. is connection results time- sea les Only the analyzed. 2000-node power algorithm for to are are a due either: special More results provide to case where general system. more then weak the 0 r with treatment sparse of the weak net'Wor:ks connee t ions or sparse connections are also weak situations A three-step identifying combined general sparse are illustrated procedure using connections the and with the grouping tbe weak connections has been proposed and shown to pr:ovide useful information about the 2000-node system. 347 In the row of GK responding to E to the first node of area « is -eEl' . z~. l corresponding to the variable node i+l of area ~ is c;(i+l), E E the entry car- the entry corresponding and the entry corresponding ~ to E node j in area B lS C«(i+l)Bj-c«lBj' where e«iBj is the number of connections between node i of area « and node j of area E. Then in the row of A21 sponding to = y~ GKEU corresponding to the variable z~, the entry corre- is (A. 6) and the entry corresponding to YB is (A. 7) Hence max [ E «, i (A. B) In corresponding the to the variable the entry corresponding to z~ is 1 (A. 9) a. the entry corresponding to zk' k I: L is (C E - ca.(k+l»/ma. a.l E (A.IO) and the entry corresponding to z~ is J c cE (c E «(i+l)E( j+l) alB (j +1) + alB E - E Ca.(i+l}B)/m,B' (A.ll) 349 APPENDIX B PROOF OF THEOREM 3.1 We and [1]. show the O(od} approximations of ~s and ~f in (3.25) by an iterative upper block triangularization process in Introducing the new fast variables (3.26) ill = Z + we transform dY/dt s = -1 dA 22 A21 y (3.24) (B.I) into 1 AllY +- A12 Til' (B.2 ) where (B. 3) A second transformation using the revised fast variables (B. 4) yields (B. 5) where (B.6) 351 APPENDIX C GROUPING ALGORITHM For a large identifying scale sparsely dynamic connected network, areas is an the automated grouping tool for a 19ori thm in [1,6], which is summarized as follows: step 1: Choose the number of areas. Step 2: Compute a basis matrix V of the eigenspace of the r slowest t. eigenvalues, using either EISPACK [15] for dense matrices or the Lanczos algorithm in [16] for sparse matrices. Step 3: Apply Gaussian elimination with complete obtain the states used for the pivoting pivots as to V and the reference if the row all the reference states. Step 4: Assign a state corresponding i to to reference state a. is. a. state among in V states. closest to that corresponding to state i. When the large r number of for Step 2. of r to find an r and area areas for entries per a is not known, we choose a sufficiently Then Step 3 can be repeated for various values that yields a partition with small node parameter d parameter computer r (). Typical 2000-node row of K eigenvalues/eigenvectors. computation network are with about 4 an times on average minutes of a of VAX 4 CPU 11/780 non-zero per 10 353 16. J. Cul11lm and R.A. Willoughby, IIComputing Ei.genvalues of Very Large Symmetric Matrices," J. Computational Physics. Vol. 44. pp. 329-358. 1981. 17. H.A. Simons. liThe Architecture of Complexity," Proceedings of the American Philosophical Society, Vol. 104. pp. 467-482. 1962. 18. R.M. May, Stability and Complexity in Model Ecosystems, Princeton University Press, 1973. 19. D.G. Feingold and R.S. Varga, "Block Diagonally Dominant Matrices and Generalizations of the Gerschgorin Circle Theorem, II Journal of Mathematics. Vol. 12. pp. 1241-1250, 1962. 20. P.V. Kokotovic, "A Riccati Equation for Black-Diagonalization of Ill-Conditioned Systems.1I IEEE Transactions on Automatic Control, Vol. AC-20, pp. B12-B14, 1975. 21. J.H. Chow. J.J. Allemong and P.V. Kokotovic. Perturba t ion Analys is of Sys terns wi. th Sus tained High Oscillations," Autornatica, Vol. 14, pp. 271--279, 1978. "Singular Frequency 22. L. H. Fi nk, II Emergency Control P rac t ices. II 'I'as k Force on Emergency Control Report. Paper BSWM034-4. IEEE Winter Power Meeting, 1985. STABILITY ANALYSIS OF SINGULARLY PERTURBED SYSTE"NIS HI\.. I\.h alilt The scale decompositJon or a singularly perturbed system into reduced and boundary layer stability analysts. gy~1tems provide::.; a strong for KLIMUSHCHEV and KRASOVSKII (1961) employed Lyapunov functions to show that asymptotic stability of equili- brium of a singularly perturbed system can be established, for sufficently small perturbation parameter E, by investigating equilibria r the reduced and boundary-layer systems. Similar results for linear systems were given in DESOER and SHENSA (1970) and WILDE and KOKOTOVIC (1972). HOPPENSTEADT (1966) gave what are probably the weakest "conceptual" cond:i. t Ions under which unif'o r'm asymptotic stabil:i ty of the equi librium of a singularly perturbed system is confirmed for sufficiently small E. com~rise The conditions uniform asymptotic stability of tHe equilibrium of reduced system, unlf.'orm asymptotic stability f the equilibrium of' the boundary-layer system. wliformly in the frozen slow parameters, and growth concH tions on r'ight-hand side rune tions. Stability investigations in which conditions, guaranteeing those of HOPPENSTEADT (1966), are imposed on Lyapunov functions for the reduced and boundary-layer systems have been pursued by a few researchers. Examples can be found in HAB8TS (1974), CHOW (1978). GRUJIC (lg81) and SABERI and KHALIL (19SQ). Stability results for linear multiparameter singularly perturbed systems are reported in KI-[ALIL and IWKOTOVIC (1979). LADDE and SILJ AJ{ ( ) and ABED (1985), and for a class of nonlinear systems which are linear in 'Elect,rical Engineering Dcparr.ment" hflchigan 51,al-;: lJniver"!ty, East. LaIlSltll!" l\.Jr_ 359 We consider a nonlinear nonautonomous singularly perturbed system f( t , x, Z, E ) , (1.1 ) EZ = g(t,x,Z,E), £>0, where the functions f and g are smooth enough to ensure that, for specified initial conditions, (1.1) has a unique solution. Suppose that (1.1) has an isolated equilibrium point at the origin. Stability of' the origin is investigated by examining the reduced system x where z f(t,x,h(t,x),o) (1. 2) h(t,x) is an isolated root of g(t,x,s,o) and 0 the boundary-layer system dT g(t,x,Z('r),o) , T= tie: where t and x are treated as fixed parameters. (1. 3) Theorem 1 states, essentially, that if x=o is a uniformly asymptotically stable equilibrium of the reduced system (1.2), z = h(t,x) is an asymp- totically stable equilibrium of the boundary-layer system (1.3), uniformly in t and x, and f and g satisfy certain growth conditions, then the origin is a uniformly asymptotically stable equilibrium of the singularly perturbed system (1.1), for sufficiently small E. In Theorem 1, asymptotic stability reqUirements on the reduced and boundary-layer systems are expressed by requiring the existence of Lyapunov functions for each system. The growth requirements on f and g take the form of inequalities satisfied by the Lyapunov functions, which we call interconnection conditions. We assume that the following condltions hold for all 361 where ~(.) is a continuous function of an Rm-vector which vanishes only at zf= O. Interconnection Conditions V and W satisfy the following inequalities: + (i) (x) (1. 9) (ii) (z - h(t,x» + £B;~(X). (1.10) (iii) altJ at + (t,x,Z,£) < Y~t2(z - h(t,x» + B~~(x)¢(z - h(t,x) 2, (1.11) Y2 Y2 For simplicity the constants , ~2' 8 Yl , and are assumed to be nonnegative. The term £Y $ in (1.9) allows for more general l dependence of f on £. It drops cut when f is Jndependent of E. Similarly, (1. 1. 0) drops Oil t Nhsn g is :l ndependent or at the above inequalities, one might have to obtain on £-dependent terms like Kl + an interval [0, £1" E Lhat is why ; E £. In arriving unlfo~m bounds may be restricted to \ boundary-layer systems, V and W, in hand, we consider a Lyapunov function candidate v(t,x,z) del~ined by a weighted sum of V and H v( ,x,z) = (1 - d)V(t,x) + dW(t,x,z), o (1.12) From the properties of V and Wand inequality (1.6), it follows that v (t,x,z) is positive-definite and decrescent. with respect to (1.1) and using (1.7) - \J where < [1 8 2f + [l-d)(a, 2 Sit and Y 2 2 (1.11) we obtain ~(I-d)Bl- - - 1:(l-d)Sl- !d B2 2 = Y2 + It can be easily seen that for all 2" Y . Computing v d e:;(a 2 -£y ) 2 ~df21l:'1 (1.13) 363 [(l-d) + 4(1-d)d (1.18) Several examples of antonomous systems were studied in SABERI and KHALIL (1984), including estimating the region of attraction of a synchronous generator connected to an infinite bus. For the linear time-varying system x l\ll(t) x + (t) (1.19 ) (t) x EZ + (t)z where Aij(t) are continuously differentiable, He ~(A22(t» < - C < 0 and the reduced system b- -1 x = [All(t) (t) are bounded, (t) (t) is uniformly asymptotically A21 (t)]x stable~ Ao(t) x (1.20) the functions V and W can be taken as V(t,x) x Tp (t)x s (1.21) (1. 22) where Ps(t and (t) -=-cIm>o satisfy the I'.,.yapunov equations (t)+A (t) T o (t) o (1. (1. 4) Then A;;;sumptions 5.2 - (Euclidean norm), [I) [12=1, (z-h(t ,x» and Y2 are bounds on the time functions 365 The analysis of Section extended to multiparameter singularly perturbed systems. convenience} we consider an autonomous system x . ( Ill. >0, ~iE.R l)J=I~ ... N , If s are known, they can be represented as known multiples of a single parameter, i.e., E, and the problem reduces to the single parameter case treated in section 1. In same applications it is important to perform stability analysis without assuming the knowledge of the mutual ratIos of the perturbation parameters. In this section we extend the stabl1Lty analysis of section 1 f to the rnult1parametel' system (2.1) 'tJhen the parameters E are unknown. We want to study the stability of an isolated equilibrium at the origin (x=o, zi=o Vi). required to hold for all (x,sl" The following assumptions are .,,)€ x .. xB ~l :";N The origin is the unique equilibrium of (2.1) algebraic equations o (2.2) have a unique root z.=h.(x) such that l l (0 ) o. The reduced system is given by f(x,h (x), ... 1 ,hl~(x))A ~'l == and has equilibrium at x=o. (x) (2.3) 367 Viewing gi as the interconnection term. the ith isolated subsystem is given by (2.10) which has an equilibrium at constant (2.10). Suppose now that we can find a Lyapunov function Hi (x~ ) satisfying 2 01:1 i (2.11) rp i ' dZ 1 hi(x» is a continusous scalar function of (x) which vanishes only at iV i t S Notice that the positive (x). does not affect the stability of the equilibrium of hi(X). Suppose further that satisfy the "spatial" interconnection condition N < i """"" ~ b .. I~ J.;) .Ip J. , l b .. >0 lJ- (2.12) 'faking N L (x, ), o (2.13) i=l with unspecified e I 1 s as a Lyapunov function candidate for (2.9), it can be shown that the derivative of W along the trajectDry of (2.9) satisfies dW \'There (p T <: ¢ T (E N R + RTEN ) ¢ _ ( <11 (2.14) 1 " " , {PN)' E and :j=i R (r.o ) ; 1.•) 369 ties (2.18) and (2.19) are stright-forward extensions or (l.g) and (l.ll) when restricted to the autonomous case. A. Lyapunov function candidate for the singularly perturbed system (2.1) is taken as v(x" ,,'" , ) 'ZN) (2.20) The derivative of v along the trajectory of (2.1) for some do>O. satisfies v [II:: IV ] < - r I L where doct o -u -u 1 S E (u l ' ... ,uN) , Iz(doP i DR + RTD - DE E= diag (~, ... , T r I[~~ I + (2.21 ) diEici)~ E: E D, EN)' and r PH ij ). Since D R + RT D is positive-definite, S is positive-definite for Moreover~ sufficiently small £i' o~ are bounded and for any there exists E.· 1 >0 < 1, since doYo - such that whenever T -1 u S U E E equilibrium of (2.1) is asymptotically stable. su~narized the elements of u >0 as of Thus , v in Theorem 2. Suppose that Assumpt10ns 2.1 - 2.11 hold. origin (x E. -j.-O. l = 0, zi = Vi) 0 Then the an asymptotically stable equilibrium singularly perturbed system (2.1) for sufficiently small Calculating the bourtds numbers £i for whic~ is positive-definite. can be done by determining the largest the matrix on the right-hand side of (2.21) An easier to calculate, yet more conservative, bound can be obtained as follOWS. Using that i'le obtain 371 1. Extension of Theorem 2 to nonautonomous systems is straightforward. 2. For N=l, the bound (2.23) reduces to the bound (1.17) derived in the single parameter case. 3. The analysis is valid for all £i>o including cases IH::e Ei« E j . However, when the perturbation parameters are of different orders of magnituate, less conservative results can be obtained by treating the problem as nested single parameter perturbations. 4. The analysis does not involve bounds on the ratio between parameters, i.e.~ (2.24) as in the earlier work of KHAI,JL and KOKOTOVIC (1979) and KHALIL (1981). This should come as no surprise. The use of the (2.24) bounds in KHALIL and KOKOTOVIC (1979) and KHALIL (1981) was a matter of convenience. It was a way at excluding cases when perturbation parameters are of different orders of magnitude since, as we pointed out in Remark 3, such cases are better treated as nested single parameter perturbations. Actually in KHALIL and KOKOTOVIC (1979) and KHALIL (1981) the numbers lliij and M1j were allowed to take any finite positive values. Recently, ABED (1985) emphasized that bounds like (2.24) are not needed by performing the analysis for the linear case without sueh bounds. 373 Ladde, G.S. and D.D. Siljak (1983), Multiparameter singular perturbations or linear systems with multiple time scales, Automatica~ 19, pp. 385-394. A.N. and R. K. Miller (1977), Qualitative Analysis Large Scale Dynamical Systems, Academic Press. Michel~ or A. (1983), Stability and Control of Nonlinear Singularly urbed Systems, with Application to High-Gain Feedback, Ph.D. Dissertation, Michigan State University. Saberi Saberi, A. and H. Khalil (1984), Quadratic-type Lyapunov functions for singularly perturbed systems, IEEE Trans. on Auto. Control, AC-29, pp. 542-550. Saksena, V.R., J.OtReilly and P.V. Kokotovic (1984), Singular perturbations and time-scale methods in control theory: survey 1976-1983, Automatlca, 20, pp 273-293. Siljak, D.D. (1978), Large-Scale Dynamic Systems: Structure, New York: North-Holland. Stability and Wilde, n.R. and P.V. Kokotovic (1972), Stability of Singularly perturbed systems and networks with ics, IEEE Trans. on Auto. Control, AC-17~ pp. 245-24 Zien s An upper bound for the perturbed system, J. parameter in a Inst.~ 375 side of (1.2). We relax this assumption in Section 2 of this paper. Our theorems are rather different in hypothesis and rather simpler than those of [4.5] in this regard. Another important contribution of Section 2 is to relate the exponential stability of the averaged system (1.2) to the exponential stability of the unaveraged system (1.1). using a converse theorem of Lyapunov. As such, these theorems are a considerable extension of the local stability theorems of Hale. In Section 4. we extend all of these results to tWo-time scale state space systems and the results are generalizations in the sense mentioned above of those of Hale and Sethna. (B) Our development of these theorems on averaging was heavily motivated by recent literature on the application of averaging techniques to adaptive control--notably the work of Krause et al [9], Astrom [10]. Riedle and Kokotovic [11]. Averaging methods have been more prevalent in the stochastic adaptive control literature. ego Ljung [12] and the first attempts to apply averaging were made heuristically in [9]. and increasingly rigorously in [10] and [11]: The primary focus of the efforts in (9-11] is to use averaging to explain instability mechanisms in adaptive control arising from unmodelled dynamics. a phenomenon popularized by Rohrs et al [13]. In this paper, we content ourselves with applying our results on averaging theory along with techniques of generalized harmonic analysis introduced in Boyd and Sastry [14]. We study convergence rates of adaptive identification schemes and linearized adaptive control schemes without unmodelled dynamics and in the presence of persistent excitation. Estimates of convergence rates are of interest in the determination of optimal input signals for identification. In earlier work (Bodson and Sastry [15]). we also showed how persistent excitation guarantees a margin of robustness to unmodelled dynamics and established connections between the rate of convergence of the adaptive schemes and their robustness margins. A more detailed study of instability theorems for averaging and their application to understanding the mechanism of slow drift instability pointed out by Riedle and Kokotovic [16] is an interesting avenue of future work. In adaptive systems. averaging has usually been associated with slow cu1aptation. Since the parameter e appears in the right-hand side of the differential equation governing the adaptive parameters. and since averaging is considered as a perturbation technique. it is frequently understood that the results are valid only for e small (if not infi.nitesimaI), Le. for slow adaptation. However. simulations of adaptive systems show that averaging often provides a good approximation for relatively large (of order 1) values of the parameter • After this manuscript was written, new and related work of KOSUl and Anderson [11] was communica led 10 us for system (1.1) with f (t .X) linear in x . but wilh weaker conditions in the limit in 0.3). €. 377 2. Basic Averaging Theory In this section. we consider differential equations of the form: i = e f (t .x .e) where x ERn. t ~O. O<e ~€o. and f (2.1) is piecewise continuous with respect \0 time. We will concentrate our attention on the behavior of the solutions in some closed ball Bil of radius h . centered at the origin. For small E, the variation of x with time is slow. as compared to the rate of time variation of f. Such systems can be conveniently studied using the method of averaging (see e.g. [1], [3]. [6], [7J). The theory relies on the assumption of the existence of the mean value of f (t .x ,0) defined by the limit: t+T f (n' (x) = lim 1 T-= assuming that the limit exists uniformly in Jf (T .x .0) d (2.2) T t t and x. This is formulated more precisely in the following definition: Deftnition 2.1 The function Mean Value of a Function, Convergence Function f (t .x .0) is said to have mean value f al' ex) if there exists a continuous function y(T): R + ...... R +, strictly decreasing, such that y(T)- 0 as T -+ tXJ, and: 1 t+T Jf (T .x .0) d 1" - f (I" (x ) II ~ Y (T) (2.3) for all t .T ~O, x EBh . The function y(T) will be called the convergence function. Note that the function {(t .x .0) has mean value d(t.x)=j(t.x.O) f ai' (x) if and only if the function: ja,,(x) (2.4) has zero mean value. The following definition ([20]. p 7) will also be useful: Deftnition 2.2 Class K Function A function a(e): R+-R+ belongs to class K (aCe) E K) . if it is continuous. strictly increasing. and a(O)=O. It is common. in the literature on averaging. to assume that the function j (t .x .e) is periodic in 1 , or almost periodic in t. Then. the existence of the mean value is guaranteed. 379 (2.7) for all t ~O. x EB1, • Moreover. w eCO.x )=0. for all x EBI! . If, moreover: y(T)=a / T r for some a ~O, r ECO,l], Then.: The function g(e) can be chosen to be 2a e r • The proof of Lemma 2.1 is provided in the appendix. The construction of the function we(r .x) is identical to that in [2.11 but the proof of (2.6), (2.7) is different. and leads to the relationship between the convergence function yeT) and the function gee). The main point of Lemma 2.1 is that. although the exact integral of d Ct .x) may be an unbounded fUnction of time. there exists a bounded function we(t .x ). whose first partial derivative with respect to t is arbitrarily close to d Cr .x). Although the bound on we(t .x) may increase as e-O. it increases slower than 1/ e, as indicated by (2.6). It is necessary to obtain a function w t:(t ,x). as in Lemma 2.1, that has some addi- tional smoothness properties. A useful lemma is given by Hale in [3] (lemma 5. p 349). For the price of additional assumptions on the function d (l .x ). the following lemma leads to stronger conclusions that are useful in the sequeL Lemma 2.2 Smooth Approximate Integral of a Zero Mean Function If: d (t .x ): R +X Bh - R n is piecewise continuous with respect to t. has bounded and con- tinuous first partial derivatives with respect to x, and d Ct ,O}==O for all t ~O. Moreover. d (t .x ) has zero mean value. with convergence function y(T) II x U. and ad (t .x) has zero ax mean value, with convergence function y(T), Then.: There exists g( e) E K . and a function We (t .x ): R +x Bh - R 11 Uewe(t .x ) fi ~ fCe) II x 0 II awe(t ,x) at U e -d (t.x) II ~ g(eH xU aw E(t ,x) ax II ~ gee) • such that: (2.8) (2.9) (2.10) for all t ~O. x EBn . Moreover. w E(O.x )=0, for all x EBh . If, moreover: y(T )=a / T r for some a ~O, r E(O.l]. Then.: the function fCe) can be chosen to be 2a el' . The proof of Lemrru:t. 2.2 is provided in the appendix. The difference from Lemma 2.1 is in the condition on the partial derivative of W E(l .x) with respect to x in (2.10), and the 381 Comments The proof of Lemma 2.3 is provided in the appendix. A similar lemma can be found in [3J (lemma 3.2. p 192). Inequality (2.17) is a Lipschitz type of condition on p (t .z .e). which is not found in [3], and results from the stronger conclusions of Lemma 2.2. Lemma 2.3 is fundamental to the theory of averaging presented hereafter .. It separates the error in the approximation of the original system by tbe averaged system (x -xu,,) into two components: x-z and Z-XQ1 •• The first component results from a pointwise On time) transformation of variable. This component is guaranteed to be small by inequality (2.8). For e sufficiently small (e ~e 1). the transformation z -x is invertible. nnd as e-O. it tends to the identity transformation. The second component is due to the perturbation term p (t .z .€). Inequality (2.17) guarantees that this perturbation is small as e -+ O. A t this point. we can relate the convergence of the function y(T) to the order of the two components of the error x -xcv in the approximation of the original system by the averaged system. The relationship between the functions yeT) and tee) was indicated in Lemma 2.1. Lemma 2.3 relates the function g(e.) to the error due to the averaging. If d (t .x) has a bounded integral (i.e. y(T }--1/ T), then both x -z and p (t order of e with respect to the main term f QtJ (z). e-+O. but possibly more slowly than linearly ( as ,Z .e) are of the In generaL these terms go to zero as -J'E for example). The proof of Lemma 2.1 provides a direct relationship between the order of the convergence to the mean value, and the order of the error terms. We now focus attention on the approximation of the original system by the averaged system. Consider first the fallowing assumption: (AS) Xo is sufficiently small that. for fixed T. and some h' t E[O.T / e) (tbis is possible. from (A3)). Theorem 2.4 Basic Averaging Theorem If: The original system (2.1). and the averaged system (2.5) satisfy assumptions (Al)- (AS). Then: There exists W(e) as in Lemnw. 2.3 such that. given T ~O: D x (t )-X01 ' (t ) n ~ wCe) br for some br , er >0. and for all t e[O.T/ e]. e~eT' (2.18) 383 II x(t )-xo,,(t) II ~tP(e)b (2.24) for all t ~O. and some b. Consequently. Theorem 2.4 does not allow us to relate the sta.bility of the original and of the averaged system. This relationship is investigated in Theorem 2.5, after a preliminary definition. Definition 2.3 Exponential Stability, Rate of Convergence The equilibrium point x =0 of a differential equation is said to be exponentially stable, with rate of convergence 0/ (01 >0). if: II x (t ) II ~ m II x (t 0) Ue -0(1-1 0) (2.25) for all t ~to~O. x(to)EBho ' and some m ~l. We assume that ho~h/ m. so that all trajectories are guaranteed to remain in B h • Theorem 2.5 Exponential Stability Theorem If: The original and averaged systems satisfy assumptions (A1)-CAS), the function f ov (x) has continuous and bounded :first order partial derivatives in x. and x =0 is an exponentially stable equilibrium point of the averaged system. Then: There exists e2>0 such that the equilibrium paint x =0 of the original system is exponentially stable for all e ~E2' Proof: The proof relies on a converse theorem of Lyapunov for exponentially stable systems (see for example [201. p 273). v (X 01 ' ): Under the hypotheses, there exists a function R" -R +. and strictly positive constants OIl.0!2.0:'3'0:'4. such that. for all XOl' EBh 0: (2.26) (2.27) (2.28) The derivative in (2.27) is to be taken along the trajectories of the averaged system (2.5). The function v is now used to study the stability of the perturbed system (2.16). Considering v (z). inequalities (2.26) and (2.28) are still verified. with z replacing X"", The derivative of v (z ) along the trajectories of (2.16) is given by: V(Z)I(2.16)=V(z)b.s)+( ~~ )(ep(t.z,e)) and. using previous inequalities (including those from Lemma 2.3 ): (2.29) 385 and any such function will provide a bound on the rate of convergence of the original system for e sufficiently small. 3) The conclusion of Theorem 2.5 is quite different from the conclusion of Theorem 2.4. Since both x and X a ,. go to zero exponentially with t . the error x - Xa ,. also goes to zero exponentially with t. Yet. Theorem 2.5 does not relate the bound on the error to E. It is possible, however. to combine Theorem 2.+ and Theorem 2.5 to obtain a uni- form approximation result. with an estimate similar to (2.24). 387 r PLANT (5I-Ar'b v(I) + Fig 3.1 Block diagram of ada.ptive identi6er. 389 in which instance. tbe limit Ru (T) is called the autocovariance of u. It may be verified that the autocovariance matrix of a stationary signal w is a posi- tive semidefinite function RII' (T). and that w is persistently exciting if and only if the autocovariance at 0 is positive definite [14]. Also. R".(T) can be written as the inverse Fourier transform of a positive spectral measure SII' (d v): tQ R". ('T ) j e il'T S", Cd v) (3.16) Further. if the input r is also stationary. S... (d v) can be computed. using the fact that the transfer function from r to w is given by: (3.17) so that: S... Cd v) =qCj v) q' (j v) sr Cd v) (3.18) Using eqns. (3.16) and (3.17). we can conclude that: = R".(O) = jq(jv)q'Civ)sr(dV) >0 This in turn is assured [14] if the support of points (the dimension of w = Sr Cd v) (3.19) is greater than or equal to 2n +1 the number of unknown parameters = 2n +1). With these definitions. the averaged system corresponding to (3.14) is simpJy: (3.20) This system is particularly easy to study. since it is linear, and when w is persistently exciting. R". (0) is a positive definite matrix. A natural Lyapunov function for (3.14) is: (3.21) and: (3.22) where "'min and "'max are respectively the minimum and maximum eigenvalues of RI/.' (0). Thus. the rate of exponential convergence of the averaged system is at least E)..min(R", (0)). and at most €"'maiR", (0)). By the comments after theorem 2.5. we can conclude that the rate of convergence of the unaveraged system for e small enough is close to the interval [EAmi!l(R~. (0)). EAmll,,(R w (0))]. 391 s. Ca) 6. (b) c, ~ , I. (c) Fig 3.2 Trajectories of parameter error fll(= Cl-C:) and ¢lliJl with three different adaptation gains. (a) £=1 (b) (=0.5 (c) £=0.1 393 c. t, 2. I. -2. -\, -£. (a) 6• •• 2. e, ·~I .,. (b) 3,7: 2. : 1.1£ &• . :.~: "'~': (c) If. • : I hi . Fig 3.4 Trajectories or Lyapunov (unction V(¢) and V(¢u) with three adaptation gains (a) l=1 (b) £=0.5 (c) £=0.1 using log ocale. 395 (B3) the function d (t .x) = (B4) A ERm Xm is Hurwitz. (B5) Xo f (t .x .0)- f aI' (X) satisfies the conditions of Lemma 2.2 is sufficiently small that. for T fixed. and some h ' X al , (t) EEn' for all t E[O. TIe] (this is possible. from (B2)). \Ve will also assume that YoEBh '. the corresponding closed ball in Rm. Theorem 4.1 Basic Averaging Theorem for Two-Time Scale System If: The original system (4.1). (4.2). and the averaged system (4.3), satisfy assumptions (B1)-(B5). Then: There exists I/1Ce)EK such that. given T ~O: (4.8) for some br • Er >0. and for all t E[O. TIe]. e ~er. and Yo sufficiently small. Further.l/I(e) is of the order of e+E(e) (as defined in Lemma 2.2). Proof: We first apply Lemmo. 2.2, and obtain a result similar to Lemmo. 2.3. Consider the transformation of variable: x = z + Ew f(t .z) (4.9) with e ~E 1. This transformation leads to: i = (I + e a: uZ € )-le If 01' Cz ) + ( f (t .z .0) - f av (Z ) _ OW £ + Cf Ct .z+ewE.O) +(f (t .z Hw ,.y) - at f ) (t .z ,0)) f (t .z +ew ,.0)) I (4.10) or: Z (O)=XII (4.11) where: (4.12) and: (4.13) 397 Using this estimate in (4.16), and using the Generalized Bellm.aJl-GronwaU Lemma again: (4.23) As in Theorem 2.4, it follows that. for some bT : Ux By assumption. I Xl.ll' (4.24) (t (t) U~h ' let Yo. and Er sufficiently small that. by (4.22). y (t )EBh • for all t e[O.T J e]. It follows. from a simple contradiction argument. that the estimate in (4.24) is valid for all t E [0. T J e]. whenever e ~er . Theorem 4.2 Exponential Stability Theorem for Two-Time Scale Systems If: The original system (4.1). (4.2). and the averaged system (4.3) satisfy assumptions (Bl)-(B4). the function fat.' (x ) has continuous and bounded first partial derivatives in x. and x =0 is an exponentially stable equilibrium point of the averaged system. Then: There exists E4>O such that the equilibrium point x =0 of the original system is exponentially stable for all e ~e4' Proof: Since XU\' =0 is an exponentially stable equilibrium point of the averaged system. satisfying (2.26 )-(2.28). On the other hand. since A is Hurwitz. there exist matrices P,Q >0. such that A r P+PA =-Q. Denote by Pl,P2.tJ l.Q2 there exists a function v (X/.l v ) the minimum and maximum eigenvalues of the P and Q matrices. We now study the stability of the system (4.11). (4.2), and consider the following Lyapunov function: VI (z .y ) = v (z ) a., + _- y T P Y (4.25) P2 so that: (4.26) where Q'I=min(al,~pl)' The derivative of P'2 be estimaled. using the previous results: VI along the trajectories of (4.11). (4.2) can 399 Mixed Time Scales We now discuss a more general class of two-time scale systems. arising in adaptive control: :i sf '(t ,x.y ') y' = Ay + h (t I (4.32) + eg r(t ,x) (4.33) .x .y') \Ve will show that system (4.32)-(4.33) can be transformed into the system described in the previous section. In tbis case. x is a slow variable. but y' has both a fast. and a slow component. The averaged system corresponding to (4.32). (4.33) is obtained as follows. Define the function: t vet ,x) = J eA(l-T)h (r.x)d (4.34) T (I and assume that the following limit exists uniformly in t and x: 1 t+T f al' (x ) =T-c:o lim -T J f ( 4.35) '('1' oX .v ('1' oX )) d T I Intuitively, v (t ,x) represents the steady-state value of the variable y with x frozen nnd E=O in (4.33).* To show that the averaged system of (4.35) is the right one. we transform the system (4.32). (4.33) to the form (4.1), (4.2). using the transformation: y =y , - (4.36) v (t oX ) From (4.34). v (t .x) sat-isfies: :t v (t .x ) = A v (t ,x ) + h (t oX ) l' (4.37) (t .0)=0 Differentiating (4.36), we have that: .= I a y Ay +e - 1' (t aXoX) f '(t ,X .y +v (t oX)) + g '(t oX .y +v (t ,x)) I (4.38) so that system (4.32). (4.33), is of the form (4.1). (4.2), with: f g (t .x .y) = (t ,x ,y) = f 'et.x.y +v (t ,x)) 8v ~:) f '(t .x .y +v (t .x )) + g I(t .x .y +v (t ,x )) 'This choice of transformation was pointed out to us by B. Ril!dJe & P. Kokoto'\'ic. (4.39) (4.40) 401 5. Two-Time Scale Averaging Applied to Model Reference Adaptive Controller To apply the theory of Section 4 to model reference adaptive controllers we review the model reference adaptive system of Narendra. Va]avani [18) for the relative degree 1 case (our notation is however consistent with Sastry [19]). Consider a plant with transfer function (5.1) where np. d p are relatively prime monic polynomials of degree n-l, n respectively and kp is a scalar (the representation in (5.1) is assumed minimal). The following are assumed to be known' about the plant transfer function: dp • np (el) The degrees of the polynomials are known. (e2) The sign of kp is known (say kp >0). (e3) The plant transfer function is assumed to be minimum phase. The objective is to build a compensator so that the plant output asymptotically Tn matches that of a stable reference model (s) with input r (t). output Ym (t) nnd transfer function m(s) where k m >0 and nm • dm are monic polynomials of degree n -1. n respectively (not necessarily relatively prime but both Hurwitz). If we denote the input and output of the plant u (t) and Yp (t) respectively. the objective may be stated as: find u (t) so that Yp (t )- Ym (t ) -0 as t - o:::t By using suitable prefiltering of the reference signal if neces- sary. we may assume that the model m(s) is strictly positive real. The scheme is shown in Figure 5.1. The dynamical compensator blocks F 1 and F 2 (reminiscent of those in Section 3) are identical one input. n-1 output systems. each with transfer function (s1 -A)-lb eigenvalues are the zeros of : A E Rn.-1Xn-l • b E R rt - nm • The pair A 1 where A is chosen so that its b is assumed controllable and. for ease of book-keeping On the algorithm proof alone). we assume that they are in controllable form so that 1 (s1 -A)-l = _1_ nmes) .r 403 The parameters e E R TI zeros: d E R l1 - The parameter 1 1 in the precompensator block serve to tune the closed-loop plant • doE R in the feedback compensator assign the closed loop plant poles. Co adjusts the overall gain of the closed loop plant. Thus. the vector of 2n adjustable parameters denoted B is with the signal vector w E R211 defined by The input to the plant is seen to be and the stnte equations of the plant loop are given by :ip Ap 0 0 Xp 0 A 0 v(l) beT p 0 A V(2) = \~(l) \;(2) bl' + b BTw (5.2) 0 It may be verified that there is a unique constant B· E R '2n such that. when B transfer function of the plant plus controller equals when m. (5). It can B'. the also be shown {18] that is bounded and the parameter update law is given by T (5.3) with r E R2nX2n • a positive definite matrix. all signals in the loop. i.e. u. v. v(l).1'(2),yp .Ym are bounded. In addition. lim e l(t) = 0 so that asymptotically Yp (t) approacbes Ym (t ). 1-.:>:.1 The proof of this fact used the following procedure: represent the model ( in non-minimal form) as the plant loop with B set equal to e' . The state equations for the model loop are given by ' • T oCp .im . (1) Vm • (2) Vm bpC 'T bpd*T Xm hd~cJ A+bc'r \' (J) bc:' 0 bd' r A AI' +bp d = m v (2) m + hI' h 0 Co T (5.4) The 3n-2 X 3n-2 matrix in (5.4) is henceforth referred to as A . and the 3n-2 vector in (5.4) as b. Then. subtracting (5.4) from (5.2) with we have lhat (5.5) and · 405 and the eigenvalues of A). Wm is bounded. Hence it is easy to see that the equations (5.12). (5.13) are of the form of (4.33). (4.32) with the functions f ' and h satisfying the condi- tions of Section 4. To establish the averaging results. we assume that r is stationary. This implies. as has been shown in Boyd and Sastry [14]. that Wm is stationary. Its spectral measure is related to that of r by (5.14) with 1 an exponentially stable transfer function. The function v (t .rp) of Section 4 for the system (5.12), (5.13) is r v(t.rp):=[! eA(t-T)bw!(T)d'i]rp o and the averaged f is given by (5.15) Since wm is stationary, the limit in (5.15) may be shown to exist as follows. Define a filtered version of Wm to be t w l1l! (t) = Since cT (sJ - A )-11; = ].m (s) !o (5.16) is stable. it follows that wm! (t) is also stationary. The Co quantity inside 1.he square bracke1.s in (5.15) is s+T lim T1 T--. ! wm(t s i.e. the cross correlation between Wm )w~j (t )dt = R",m \0' mt (0) (5.17) and wntf evaluated at O. Consequently. we may use (5.14) and (5.16) to obtain a formula for R..,~, III 111/ (0) as (5.18) 407 using equation (5.18) .. With a m =3. k m =3, ap =1. k p =2. a=3. w=2. the two eigenvalues of the averaged system are computed to be -3.10e and -0.43e. both real negative. Figs 5.2. 5.3 show the plots of the parameter errors of Co and do for the original and averaged system. with three different adaptation gains. Fig. 4 corresponds to a higher frequency input signal w = 4 such that the eigenvalues of the matrix R ....m ,,'ml (0) are complex (-0.49±0.30i )e. and explains the oscillatory behavior of the original and averaged systems. Using the results of Boyd and Sastry [14]. it is easy to verify the following facts (i) R"'mwm'(O) is singular unless R",(O) >0. i.e. wet) is persistently exciting. Thus persistent excitation of w is a necessary condition for exponential stability of (5.19). (ii) If m(s) is strictly positive real and w (t ) is persistently exciting, then R".m \4"m! (0) is Hurwitz. Hence m(s) being strictly positive real is a sufficient condition for stabil- ity of (5.19), given that w (t ) is persistently exciting. It is intuitive that if w is persistently exciting and m(s) is close in some sense to being strictly positive real that R".m Ii"tn! (0) will be Hurwitz (in particular. this is the case if Re m. (j v) fails to be posi~ive at frequencies where n (j v) is small enough). More specific results in this context are in [11.17). In view of the results stated at the end of section 5. averaging can also be applied to the nonlinear system described by (5.10)-(5.11). with A {x )=..1 +bx T Q. Consequently. the nonlinear time varying adaptive control scheme can be analyzed through the autonomous averaged system (a generalisation of the ideas of [24]). However. due to the nonlinearity of the system. the frequency domain analysis. and the derivation of guaranteed convergence rates are not straightforward. 409 e. -1.5 -3, ·~,S ·it \ r -7.5 \/ (a) e, -l.:~ -2,~ -l,iS -~. -:.:: I 1 (b) ·7« S ~ O. (c) Fig 5.3 Trajectories of parameter error q,i:= da-do ) and ¢'1J2 with three adapta.tion gaiDs (a) t==l (b) f=0.5 (e) f=O.l using log scale. 411 6. Concluding Remarks We have presented in this paper new stability theorems for averaging analysis of one and two time scale systems. We have applied these techniques to obtain bounds on the rates of convergence of adaptive identifiers and controllers of relative degree L We feel that the techniques presented here can be extended to obtain instability theorems for averaging. Such theorems could be used to study the mechanism of slow drift instability in adaptive schemes in the presence of unmodelled dynamics. in a framework resembling that of [10]. 413 Thjs. in turn. implies that (LIO) Clearly g(e)EK. From (Ll), it follows that: aWE(t,X) ---::-,--- - d (t ,x ) = -e at W E(t .X ) (Lll) so that both (2.6) and (2.7) are satisfied. If y(T )=a / T r , then the right-hand side of (L8) can be computed explicitly: (L12) and. with r denoting the standard gamma function: co Ja e r ('J",)l-r e-r'd T';; a e r r(2-r) ~a e r (L13) o Defining g(e)=2a e r • the second part of the lemma is verified. Proof of Lemma 2.2 Define W E(t..x) as in Lemma 2.1. Consequently, (L14) · ad OX (t .X) IS . zero mean. an d'IS b ound e. d Lemma. 2 1 can be app I"Ie d to ad aX (t .x) • an d Slnce inequality (2.6) of Lemma 2.1 becomes inequality (2.10) of Lemma 2.2. Note that since ad ~t;) is bounded. and d (t ,0)=0 for all t ~O. d (t ..x) is Lipschitz. Since d (t..x) is zero mean, with convergence function yeT) Ox D. the proof of Lemma 2.1 can be extended. with an additional factor Ox H. This leads directly to (2.8) and (2.9) (although the function gee) od ~~,x) • these functions can be replaced by a may be different from that obtained with single gee )). 415 +E ( f (t .z .€)- f (t ,z .0) ) := ef av (z) + Ep '(t (L18) ,x ,z .e) where. using the assumptions. and the results of Lemma. 2.2: (L19) For E ~El' (2.10) implies that (J +E OlY az) E , has a bounded inverse for all t ~O. z EBh . r Consequently. z satisfies the differential equation: i = [I+e = Ef av 1l;:' (z ) (e!,,(z)+ep'(t,z,e)) + E P (t ,z .E) Z (L20) (O)=xo where: p(t.z,E)= ! aWE l+e-_a~ -1 OWe Ip'(t.z.e)-e-_-!a\'(Z) I a~ 1 (L21) and: := .pCe) II z I (L22) Generalized Bellman-Gronwall Lemma (cf. [71 P 169) If: x (t ). a (t ), u (r ) are positive functions satisfying: t x (t ) :£;. Jo a ( T )x ( T )d T + u (t ) (L23) for all t e[O.T). and u(t) is differentiable. Then: t x(t) ~u(O)e ja(a)dCT 0 t + z J ti(r)c o for all t E[O.T]. jaCa)da T dT (L24) 417 [15] Bodson. M. and S. Sastry. liSman Signal 1/0 Stability of Nonlinear Control Systems: Application to the Robustness of n MRAC Scheme." Memorandum No. UCB/ERL M84/70. Electronics Research Laboratory, University of California. Berkeley, 1984. [16] Riedle. B. and P. Kokotovic. "Stability Analysis of Adaptive System with Unmodelled Dynamics." to appear in 1nt. 1. of Control. Vol. 41. 1985. [17] Kosut. R .. B.D.O. Anderson and I. Mnreels. "Stability Theory for Adaptive Systems: Method of Averaging and Persistency of Excitntion/' Preprint. Feb. 1985. [18] Nnrendra, K. and L Valavani ... Stable Adaptive Controller Design-Direct Control,'· IEEE Trans. on Autonuztic Control, Vol. AC-23 (1978). pp 570-583. [19] Sastry. S., "Model Reference Adaptive Control - Stability. Parameter Convergence. and RObustness." IMA Journal of Mathemotical Control & Information, Vol. 1 (1984), pp 27-66. [20] Hahn. W.o Stability of Motion, Springer Verlag. Berlin. 1967. [21] Luders. G. and K.S. Narendra. II An Adaptive Observer and Identifier for a Linear System," IEEE Trans. on Automatic Control, Vol. AC-18 (1973). pp. 496-499. [22] Kreisselmeier. G. \I Adaptive Observers with Exponential Rate of Convergence." IEEE Trans. on Automatic Control, Vol. AC-22 (1977). pp. 2-8. [23] Goodwin. G. C. and R. L Payne. Dynamic System Identification, Academic Press. New York. 1977. [24] Riedle B. and P. Kokotovic. "Integral Manifold Approach to Slow Adaptation," Report DC-80. University of Illinois. March 1985.