Optimal Control: Basics and Beyond

CHAPTER 7

The Pontryagin Maximum Principle The Pontryagin maximum principle states a method of direct trajectQry optimisation which we have in fact already seen in two contexts and by two approaches. These are the approaches which yield both a rapid development of the formalism and a feeling for its meaning. In Section 2.10 we considered perturbations from an optimal trajectory and, in the continuous-time case, deduced the first-order relations (2.76) and (2.77) by dynamic programming methods. .These can be regarded as necessary conditions for optimality of the trajectory, at least when the derivatives invoked exist. Then, in the last chapter, we found Lagrangian methods efficacious for a direct optimisation of trajectory. This was for the LQ case, but perhaps generalises. The maximum principle was first seen as replacing the classical calculus of variations, which it superseded because it could deal with cases for which the optimal control turned out to be discontinuous. One fmds such discontinuous controls in the 'bang-bang' control of the Bush problem (Section 6), rocket thrust programming (Section 7) and even our fishing problem (Section 2.7). The principle also revealed an attractive formalism, a Hamiltonian structure analogous to that of classical mechanics. The reason for this is that a Hamiltonian structure appears whenever an incomplete specification of dynamics is supplemented by an extremal principle. In the control case the dynamical specification is incomplete because the plant equation determines process dynamics for given control values, but gives no guide for the determination of these control values. The guide is supplied by the extremal principle of costminimisation. A fully rigorous derivation of the principle would be, at the present, both lengthy and unattractive. We shall rather adopt a heuristic approach which reveals the structure very quickly and which provides a machinery for the quick generation, in any particular case, of the assertions which one may expect to hold. There is one interesting point, however. Versions of the principle hold in both discrete and continuous time, but in fact under milder conditions in the continuous-time case, for reasons explained in Section 1. Because of this and because of the greater importance of the continuous-time case in the control context we shall devote most of the treatment to this case.

132

THE PONTRYAGIN MA XIM UM PRIN CIPL E

1 TH E PRINCIPLE AS A DIRECT LAGRANGIAN OPTIMISATION We saw in the last chapter how effectiv e direct trajectory optimisation was for the LQ model, especially in the infinite -horizon limit. It is then natural to ask whether the same methods could not be carried over to othe r models. Tha t is, consider a discrete-time model with the state-structure implied by the plant equation (2.2) and the cost function (2.3) and with a vector-valued state variable. Regard the plant equation at time T as a constraint on the {x, u} path with which can be associated a vector Lagrangian multiplier A-r· The Lagrangian form h-1

2)c (x., .,un T)

-r=O

+ >.;(x-r- a(x-r-J,U-r-J,T)] + Ch(xh)

(1)

should then be extremised with resp ect to the {x, u, >.} path, for given initial conditions. We have hitherto suppos ed the horizon poin t h prescribed, with h = +oo as a limit case. It is conceiv able, however, that the Lagrangian approach could be valid und er othe r stopping rules; e.g. that h is the first time at which x first enters some set. For instance, the flight of an aircraft terminates at the moment when it comes to rest on the ground, which may be at a time and in a man ner unscheduled. The formalism that one derives this way is exactly the formalism of the socalled 'discrete maximum principle', the maximum principle in a discrete -time formulation. However, the conclus ions are correct und er only quite rest rictive conditions, because the strong form of Lagrangian methods to which we wish to appeal is valid only und er such con ditions. Briefly, we would like to asse rt that necessary and sufficient conditions for optimality of the {x, u} path are that the Lagrangian form (1) should be minima l with respect to {x, u} and maxima l with respect to {>.}. This assertion is cert ainly true und er the following hyp otheses: that the set of permitted paths {x, u} is convex, that c is convex in its x, u arguments, that a is linear in its x, u arguments, that the horizon is fixe d and finite, plus growth conditions ensu ring boundedness of optimal path and cost. These hypotheses were all satisfied for the LQ model of the last chapter , at least so long as the horizon was held finit e. However, if they are weakened then one can be sure of nothing without further investigation. The continuous-time analogue of the Lagrangian form (1) would be

foh [c(x, u, T)

+ _xT(.x- a(x, u, T))jdT + C(x(h),h)

(2)

in the notation of Section 2.6. One is now has a continuous infinity of variables, and so would expect additional reas ons for possible failure of the Lag rangian approach. Greater care is indeed necessary, but the continuous-tim e case presents one significant simplification. Suppose that the. control variable u is also vector-valued, but with its values rest ricted to a set 01/ which may very well not be

\

__ )

1 THE PRINCIPLE AS A DIRECT LAGRANGIAN OPTIMISATION

133

convex. However, by varying u rapidly relative to the rate at which x is changing one can effectively achieve any value of u which is an average of values in 11/1. That is, 11/1 is effectively replaced by its convex hull, a convex set. This is the intuitive content of the so-called 'chattering lemma: This differing behaviour manifests itself in that the dynamic programming equations yield a version of the maximum principle in continuous time which can only be equalled in strength in discrete time if one makes restrictive assumptions. We shall then associate the principle almost entirely with the continuous-time case. The LQ model remains the outstanding example of a case in which the discrete-time maximum principle is valid and useful; we indicate a couple of others in the exercises. Exercises and comments (1) Economic growth Consider the dynamic allocation problem of Section 3.4, where x 1 is the 'activity vector' at time, t, the vector of intensities at which the various activities are pursued. Thus Xt ~ 0, and resource limitations enforce the plant equation Ax1 ~ b + Bxt-1· Suppose that utility is linear: that one wishes to maximise I:~ cTx 1 , where the kth component of c is the rate at which activity k delivers 'utility~ Application of Lagrangian techniques is valid; the Lagrangian form would be h

L[cT Xt

+ A"f'(b + Bx1- t - Axt- z1)]

I= I

Here z 1 ~ 0 is the margin of inequality in the plant equation; the vector of amounts of resources unused at stage t. The multiplier has the familiar price interpretation; the jth element of At is the effective unit price of resource j at time t. Maximisation with respect to Zt yields the conclusion At ~ 0 with equality in those components for which the corresponding component of Zt is positive. Maximisation with respect to x 1 yields c + BT At+ I -AT At ~ 0, with equality in those components for which the corresponding component of x 1 is positive. If the system is self-sufficient in that it can maintain itself in the absence of external supplies then, at times remote from both the beginning and the end of the optimisation period, it settles on to a maximal growth path (the turnpike of the economists), for which x 1 is of the form p1x for some fixed x and some p ~ 1. The maximal growth rate p and the direction x of the optimal path are determined by

(3) sothatpisjustthemaximal rootoflpA- Bl

= 0.

134

THE PONTRYAGIN MA XIM UM PRINCIPLE

The von Ne um ann -Ga le model (see e.g. Gale, 1960, 1967, 1968) generalises this in that it relaxes the assum ptions of linear dependence of consumption, production and utility on x. Con vexity assumptions must be reta ine d, however, or the two extremal operations in the analogue of (3) no longer commute, and the concept of an optimal growth rate has to be qualified. (2) Optimal dosage Consider the pla nt equation x 1 = Ax _ 1 1 + u1 in sca lar variables, where the controls u are to be chosen to minimise Lj (x r- X,:) 2 subject only to u ~ 0. This would then be an LQ tracking problem but for the fact that control costing has been replaced by the positivity con dition u ~ 0. One might regard x as the concentrat ion of a drug in a patient's bod y, attenuating at rate A in the absence of further administration, but maintaine d by dosage u. The sequence { is the desired concentration pro file. (It is convenient to make couple of temporary changes a of convention: we write Ur rath er tha n Ur- I in the plant equation, and shall change the sign of A.) The Lagrangian form is

!

ua

Ia

L!(xr-X~) 2 + >.,(Axr-1 + u

1-

t=l

One then fmds the conditions >.

1

~

x1)].

0 with equality if u1 > 0 and

(1 :!{. t
h-1

0 :!{.At= '2: ..R x,+ j- '2:..Rx~+ j =X ,- _x::, j=O

say, so that Xh+l =

j=O

Xh+ 1 = 0. Note then that

(1 - Aff)(1 - Af f- 1 )X, = u ~ 0. 1 Complete the argument to show that, if we define d if X 1 = ( 1 - Af f) (1 - Af f- 1 )X, = (1 + A 2 )Xx AX t-l - AXt+l, then the opt imal solution corresponds to the sequence {X 1} which is min ima l sub ject to the conditions X,~_x::,

difX,~O (1:!{.t:!{.h). The course of X correspondin g to the optimal solution wil l consist of 'free' segments for which d if X = 0 and X1 ~ X'f (at which no dos 1 e is administered), interpersed by points at which X1 = X'f and u1 = d if X > 0 1 (at which a dose is administered). One can be more explicit abo ut the algorithm. Denote the times t at which X, = X'f by lt, t2, ... , going bac kwards in time. The n It = h + 1, when Xh+l = 0.

2 THE PONTRYAGIN MAXIMU M PRINCIPLE

The solution in the range

tt+l ~ t ~ t;

135

has the 'free solution' form X 1 = cuA 1+

c2;A- 1 • The coefficients care to be chosen so that Xt =X~ fort= t;, ti+l· Once t 1, t 2 , ..• , !; have been determined then t;+I is determin ed as the smallest value t ( ~ 1) such that the free solution agreeing with at t and t; is not smaller

of than xc at any intermediate point. If this value oft seems to be t = 1 then indeed it is provided that the value of x 0 is such that d i1 X 1 > 0, i.e. such that medicati on should begin immediately. If, on the other hand, d i1 X1 ~ 0, then medicati on begins first at t;. We have used the notation d to indicate potential generalisations of the argument. The problem has something in common with the production problem of Section 4.3. In the case A = 1 it reduces to the so-called 'monotone regression' problem, in which one tries to approximate a sequence {.x<;} as closely as possible by a non-decreasing sequence {Xt}. In this case X 1 is the smallest concave sequence which exceeds and its 'free segments' are straight lines.

xc

x;

2 THE PONTRYAGIN MAXIM UM PRINCIPLE The maximum principle (henceforth abbreviated to MP) is a direct optimality condition on the path of the process. It is a calculation for a fixed initial value :x of state, whereas the DP approach is a calculation for a generic initial value. It can be regarded as both a computational and an analytic technique (and in the second case will then solve the problem for general initial value). The proof of the fact that derivatives etc. exist in the required sense is a very technical and lengthy matter, which we shall not attempt. It is much more importan t to have a feeling for the principle and to understa nd why it holds, coupled with an appreciation that caution may be necessary. We shall give a heuristic derivation based upon the dynamic programming equation, which is certainly the directest and most enlightening way to derive the conclusions which one may expect to be valid. A conjugate variable p will make its appearance. This corresponds to the multiplier vector .A, the identification in fact being p =_AT, so that pis a row vector. The row notation p fits in naturally with gradient and Hamilton ian conventions; the column notation ). is better when, as in equation (6.20), we wish to write all the stationarity conditions as a single equation system. We shall refer top as either the 'conjugate variable' or the 'dual variable'. Note the conventions on derivatives listed in Appendix 1: in particular, that the vector of first derivatives of a scalar variable with respect to a column (row) vector variable is a row (column) vector. Consider first a time-invariant formulation. The state variable x is a column vector of dimension n; the control variable u may take values in a largely arbitrary set illt. We suppose plant equation .X= a(x, u), instantaneous cost function c(x, u), and that the process.~tops when x first enters a prescribed stopping set Y, when a terminal cost IK(:X) is incurred. The value function F(x) then obeys the dynamic programming equation

136

THE PONTRYAGIN MAXIMUM PRIN CIPLE

inf( c + Fxa) = 0 u

(x f. Y'),

(5)

with the term inal cond ition

F(x) = IK(x)

(x E Y').

(6)

The derivative Fx may well not exist if x is close to the bou ndar y of a forbidden region (within which F is effectively infin ite) or even if it is close to the bou ndar y of a highly pena lised but avoidable regi on (when F will be discontinuous at the boundary). We have already seen exam ples of this in Exercise 2.6.2 and shall see others in Section 10. However, let us supp ose for the mom ent that xis on a free orbit, on which any pert urba tion 8x in position changes F only by a term Fx8x + o(8x). Define the conjugate variable p= -Fx

(7)

(a row vector, to be rega rded as a func tion of time p(t) on the path) and the Hamiltonian

H(x , u,p) = pa(x, u) - c(x, u) (a scalar, defined at each poin t on the path

(8)

as a function of curr ent x, u and p).

*Theorem 7.2.1 (The Pontryagin max imu m principle on a free orbit; time-invariant version) ( i) On the optimal path the variables x and p obey the equations

[= a(x, u)]

(9)

andtheoptimalvalue ofu(t) is the valueofu maximising H[x(t), u,p(t)]. (ii) The value ofHis identically zero on this path .

(10)

*Proof Only assertions (9) and (10) need proof; the others follow from the dynam ic prog ram min g equa tion (5) and the definition (7) of p. Ass ertio n (9) is obviously valid. To dem onst rate (10), writ e the dyn ami c prog ram min g equa tion in incremental form as F(x)

= inf[c (x, u)c5t + F(x + a(x, u)8t)] + o(8t). u

( 11)

Differentiation with respect to x yields

-p(t ) = Cxc5t- p(t + 8t)[I +ax c5t] whence (10) follows.

+ o(c5t) 0

2 THE PONTRYAGIN MAXIMUM PRINCIPLE

137

The fact that the principle is such an immedi ate consequence of the dynami c program ming equation may make one wonder what has been gained. What has been gained is that, instead of having to solve the partial differential equation (5) (with its associated extremal condition on u) over the whole continu ation set, one has now merely to solve the two sets of ordinar y differential equations (9) and (10) (with the associat ed extremal condition on u) on the orbit. These conditions on the orbit are indeed those which one would obtain by a formal extremisation of the Lagrangian form (2) with respect to x, u and A, as we leave the reader to verify. Note that the equations (9) and (10) demand <mly stationarity of the Lagrangian form with respect to the A- and x-paths, whereas the conditio n with respect to u makes the stronger demand ofmaximality. It is (9) and (10) which one would regard as characterising Hamilto nian structure; they follow by extremisation of an integral J[px- H(x,p)] dt with respect to the (x,p) path. A substantial question, which we shall defer to the next section, is that of the termina l conditions which hold when x encoun ters!/. Let us first transfer the conclusions above the the time-dependent case, when a, c, !/ and IK may all be !dependent. The DPequa tionfor F(x, t) will now be inf(c + F1 + Fxa) u

=0

(12)

outside.9', withF(x , t) = IK(x, t) for (x, t) in!/. However, wecanr educeth iscase to a formally time-invariant case by augmenting the state variable x by the variable t. We then have the augmented variables P-+ [p Po].

(13)

where the scalar p 0 is to be identified with- F 1• However, we shall still preserve the same definition (8) of Has before, so that, as we see from (12), the relation (14) holds on the optimal orbit. Theorem 72.2 (The Pontryagin maximu m principle on a free orbit) (i) The assertionsofTheorem 7.2.1 (i)still hold, but equation (10) is now augmented by the relation

(15)

(ii) H +Po is identically zero on an optimal orbit. Suppose the system timehomogeneous in that a and care independent oft. Then His constant on an optimal orbit.

138

THE PONTRYAGIN MAXIMUM PRINCIPLE

Proof All assertions save the last are simple translations of the assertions of Theor em 7.2.1. If a and c are indep enden t oft then we see from (15) that p 0 is constant on an optim al orbit, whence the final assertion follow s. 0 However, the essential assertions of the maxim um principle are those expressed in Theor em 7.2.1 (i) which, as we see, transfer to the time-dependent case uncha nged Note that His now a function oft as well as of x( t), u(t) and p( t).

Exercises and comments (1) As indicated above, one can expect H to be identically zero on an optimal orbit when the process is intrinsically time- invariant and the total cost ~) is welldefined. The case of a scalar state variable is then partic ularly amenable. By eliminatingp from the two relations, that His identically zero and that it is maximal with respect to u, one derives the optim al control rule in closed -loop form. (2) Suppose that the process is time-invariant and has a well-d efined average cost 'Y· The total future cost is then F(x, t) = f(x) - -yt plus an 'infinite constant' representing a cost of "' per unit time in perpetuity. We thus have H = -p0 = F1 = --y, so that the const ant value of H can be identi fied with the average reward rate --y. In the scalar case the optimal control rule can be determined, at least implicitly, from H + 'Y = 0 and the u-maximality condition. The equilibrium point is then determ ined from Hx = Hp = 0; 'Y can then be evaluated as the value of-H at this equilibrium point. 3 TERM INAL COND ITION S The most obvious example of depar ture from a free orbit is at termin ation of the path on the stopping set. Since the path is continuous, there is the obvious · matching condition: that the termi nal point is the limit point along the path. However, if one may vary the path so as to choose a favour able termi nal point, then there will also be optimality conditions. The rigorous statement of these termi nal conditions can be quite difficult if one allows rather general stopping sets and terminal costs. We shall give only the assertions which follow readily in the most regular cases. However, even more difficult than termin ation is the case when parts of state space are forbidden, so constraining the path. (For example, an aircraft may be compelled to avoid obstacles, or an industrialist may not be allowed to incur debt, even temporarily.) In such a case the optim al path must presumably skirt the bound ary of the forbidden region for a time before resuming a free path. The special conditions which hold on entry to and exit from such re-stricted phases are terme d transversality conditions; we shall consider them in Section 11. Consi der first the fully time-invariant case. One then has the termi nal condition F(x) = IK(x) for x in the stopping set f/'. Howe ver, one can appeal to

3 TERMINAL CONDITIONS

139

this as a continuity condition, that F(x) - t IK(x) as x (outsideS") approaches x (inside 9'), only if xis the optimal termination point for some free trajectory. Obviously x must lie on the boundary 8Y of Y, since motion is continuous. However, we shall see from the examples of the next section there may be points on this boundary which are so costly that they are not optimal terminal points for any trajectory terminating in Y. Let Y opt denote the set of possible optimal termination points. Partial integration of the Lagrangian expression for cost minimisation throws it into the form

1 1

(px- H) dr + IK(x)

=

-1

1

(jx +H) dr

+ px + IK(x) -

p(O)x(O)

{16)

where the overbar indicates terminal values. Let a be a direction from x into Y opt> in that there is a value x, = x + Ea + o( c) which lies in Y opt for all small enough values of the positive scalar c. If x is an optimal termination point for the trajectory under consideration then we deduce from (16) that px + IK(x) :::; px, + II<(x,). In the limit of small E this yields

(17) where the derivative Kx is evaluated at x. *Theorem 7.3.1 Let x and j5 be the terminal values ofstate and dual variable on an optimal trajectory for a time~invariant problem. Then the terminal optimality condi~ tion (17) holds for all directions a into Y optfrom x. Ifx is interior to !/opt. the derivative IKx is continuous at x and a tangent plane to !/ exists at x then ( 17) can be strengthened to

(p + IKx)a = 0. for all directions in the tangent plane. This can be expressed: the vector (p normal to the boundary of!/ at x.

(18)

+ IKx) Tis

The strengthening of (17) to (18) follows because, under the conditions stated, the inequality (17) holds for all directions in the tangent plane, and in particular holds for -a if it holds for a. If we transfer these conclusions to the problem with state variable~= (x, t) then we obtain the generalisation to the time-variable case. Note that!/ will now be a set of~ values and IK a function of~*Theorem 7.3.2 tory. Then

Let (x, t) and (J5,J5o) be the terminal values on an optimal trajec-

(19)

140

TH E PONTRYAGIN MA XI MU M PR IN CI PL E

for all directions (a, r) into[//opt from (.X, t). If( x, t) is interior to [//o pt• ofU<are continuous the the derivatives ~ re and a tangent plane to[// exists there, then (19)forall (a, r) in thi equality holds in 1 s tangent plane. 4 M IN IM AL TI M E AN D DISTANCE PR OBLEMS It is useful to discuss a problem whose so lution is intuitively ob tackling a substantia vious before l control problem. In this way one gets so application of the ma me feeling for xi mu m principle, an d also for how it shou when the or bit encoun ld be modified ters a constraint. Th e first is an M P tre atm en t of the problem already discussed in Ex Suppose tha t x is the ercise 2.6.3. coordinate of a particle inn -d im en sio na l spac moves at velocity v( x e. The particle 1 bu t its direction of move ment ca n be chosen at plant equation is thus will. The .X= uv(x) where u is a un it vector to be ch that the cost associated osen. Suppose with an orbit is the tim e taken to reach the sto Y plus a function fi< pping x-set (x) of the terminal coordinate x. Th at is, terminal cost functio c = 1 an d a n fi<(x) is defined on Y. Th e problem is thus time-invariant with H = pu v( x) - 1. Sinc positive scalar then e v(x) is a H is ma xim al with res pect to u when u is direction ofpT; i.e. chosen in the

On an optimal free or

The ter mi na tio n cond

bit p obeys

P = -p uv x = -!Pivx. ition (18) becomes

(20)

(21)

p + fi<x .l oY (22) (where the ter mi na tio n values are understoo d). If n< is co ns tan t in simply am ou nts to u. Y then this lo Y; th at the optimal trajectories me et the orthogonally to its bo stopping set undary. Consider now the pa rti cu lar case when the ve locity is constant; we suppose tha t v = 1. W may as well e see then from (21) tha t p is co ns tan t an d so an optimal pa th has a from (20) tha t co ns tan t direction. Th at is, optimal free orbit lines. This is no su s are straight rprise; the minimisa tion of time now am minimisation of distan ou nts to the ce, an d the minimaldistance pa th to Y is which meets the bo un a straight line da ry of Y orthogonally . We can now visualis trajectories very easily e the optimal , even in constraine d cases, if we see th stretched tightly betw em as a string een initial an d termina l points. Th e value function wo uld now have the form F( x) = in f[ lx - x! + fi<(x)] X

4 MINIMAL TIME AND DISTANCE PROBLEMS

141

Figure 1 The minimal-distance path to a stopping set !1', meeting the boundary of !I'

normally.

with x ofcourse constrained to!/. Consider the particular case when the stopping set consists of just two points, x and x', say, with zero terminal cost. Then F(x) = min[lx- x'l, lx- x''IJ,

and one simply moves in a straight line to the nearer terminal point. The interest of the example is that Fx does not exist at either of the possible terminal points or at points equidistant from x' and x''. This does not matter at all; the relevant directional derivative always exists in directions in which it would be optimal to move. The loci of discontinuity in Fx correspond to break-points in optimal control; e.g. points at which the optimal direction of movement changes discontinuously with x. A second example illustrates the effect of a discontinuity in IK(x); the type of behaviour against which the conditions of the last section were intended to guard. Consider a two-dimensional example, x having components x 1 and x 2 , with stopping set x2 = 0 and terminal cost equal to unity for negative x 1 and zero for non-negative x 1• That is, the stopping set is the x 1-axis, and the positive half..axis is preferred to the negative half-axis. The optimal path is a straight line to the nearest point in one of the two half-axes; we leave the reader to show that

Here the three· regions f!(i are marked in Figure 2; their boundaries are determined by continuity of F(x). So, in!!£ 1 (which is just x 1 ~ 0) one moves straight to the nearest point on the XIaxis. In !!£2 one still aims for the nearest point on the non-negative x 1-axis, which is the origin. However, if x 1 is sufficiently large and negative for a given value of x2 (i.e. in f!l 3) one gives up the struggle and goes for the nearest termination point, despite the fact that this will now be on the more heavily penalised negative axis.

142

THE PONTRYAGIN MAXIM UM PRINC IPLE

Figure 2 Optimalpathsfor termination on the x 1 axis when terminat ion on the negative part ofthe axis carries unit penalty. Termination on -I < x < Ofrom a starting 1 point offthe axis is never optimal.

Figure 3 Minimal-distance paths to.'?2 when.'? 1 is effectivelyforbidde n. Thefree parts ofthe optimal trajectory are straight lines; otherwise they hug the boundar y of.'? 1 •

x,

Note that the segme nt -1 < < 0 of f/ is never entere d by an optima l trajectory. Relate d to this is the fact that both Fx and the action rule are discontinuous on the f£ 2/ f£ 3 boundary. The situation for which a part of state space is forbidden is equiva lent to that for which passag e into it is. so heavily penali sed as to be effectiv ely forbidden. So, consid er the example of Figure 3, for which the termin ations set breaks up into two parts: Y' 1, which is so heavily penali sed that it is effectiv ely forbidden, and f/'2, within which IK = 0. Optim al paths will take the shorte st route to f/' 2 which avoids f/ 1; this avoidance may take the form of skirtin g the bound ary off/ 1 for a time, as we see from the diagram.

5 SOME MISCELLANEOUS PROBLEMS The maxim um princip le has found extensive applic ation in econom ics. For what is probably the simplest example, consid er the situation of a monop olist who holds an amoun t x of a non-renewable resource which he releases at a time-

5 SOME MISCELLANEOUS PROBLEMS

143

variable rate u. Because he has a monopoly this release rate determines the current unit price of the commodity, p(x). He wishes to maximise the total discounted return

1

00

e-atup(u) dt.

The plant equation is x = -u. We shall take >. as the conjugate variable (a scalar), since we are already usingp to denote price. Discounting makes the problem time-dependent. (This time-dependence is removed in a dynamic programming approach only by a renormalisation to present value as time advances,) The instantaneous value of u is such as to maximise the Hamiltonian

H = e-atup(u) - >.u.

(23)

We have >. = -Hx = 0, so that >. is constant. If >.o is the conjugate variable associated with time then >.0 + H = 0. At termination we have >.o + K1 = 0. But II( is identically zero, so >.o is zero at termination, which implies that H = 0 at termination. This certainly holds if u = _0 at termination, and this turns out to be the only way of satisfying the terminal condition. The initial condition is

1

00

X=

U

dt.

(24)

Consider ftrst the case p(u) = u-'Y, which in fact makes the problem a version of the consumption problem considered in Section 2.2. Maximisation of H gives the rule u = ke-ath, for some constant k. If t is the termination time then the condition u(t) = 0 is satisfied only for t infinite; the resource is never released completely in fmite time. Condition (24) gives the evaluation k = (a.xh), so that the optimal release rule is u = ( a.xh) e-at h. Here x is the initial stock and this is the open-loop release rule. If xis taken as current stock then the optimal rule in closed-loop form is u = a.xh. If we insert the open-loop expression for u into the reward function above we fmd the maximal return F(x) = ('yfa.)ax 1-'Y. These conclusions are consistent with those of Section 2.2, in the case when infinitehorizon limits existed. Consider now the case p(u) = (1 - u/2)+. The rate of return up(u) thus tends to zero as u l 0; we shall see that this has as consequence that the resource is released in a finite time. The maximal instantaneous rate of return is attained for u = 1, and we certainly may assume that u < 2, since higher values yield zero rate of return. The optimal release rate is u = 1 - >.ea 1• The condition u(t) = 0 then implies that>. = e-ar, and the optimal release rule in open-loop form is

u = 1 - e-a(i-t).

(25)

144

THE PONTRYAGIN MAXIMU M PRINCIPLE

Condition (24) yields the determin ing condition fort

x = t- (1 - e-o.l)fa

(26)

and substitution ofexpression (26) for u into the reward function yields

F(x)

= (1 -

e-o.7) 2 f(2a).

(27)

The determinations (25) and (27) of the optimal u and the value function are only implicit; they are expressed in terms oft which is only implicitly determin ed as a function of x by (26). The dynamic programm ing equation, had we taken that route, could not have been solved more explicitly. Indeed, it would have taken some ingenuity to have spotted this implicit form of the solution. Exercises and comments (1) Zermelo's problem A straight river has a current of speed c(y), where y is the distance from the bank from which a boat is leaving. The boat then crosses the river at a constant speed v relative to the water, so that its position in downstream/cross-stream coordinates (x,y) satisfies = v cos(u) + c(y), y = v sin u where u is the heading angle indicated in Figure 4. (i) Suppose that the boatman wishes to be carried down-stream as little as possible in crossing. Show that he should follow a heading

x

u = cos- 1 ( -

c~)).

(Note the implication, we must have c(y) ~ v for all J! Otherwise the boatman could move upstream in the slack water as far as he likedJ (ii) Suppose the boatman wishes to reach a given point on the opposite bank in minimal time. Show that he should follow the heading U

= COS-I ( - l

+~c(y)),

where pis a constant chosen to make the path pass throught the target point.

y

o(y)

+ X

Flglll'e 4 The Zermelo problem ofEx. 5.1: the optimal crossing ofa stream.

6 THE BUSH PROBLEM

145

6 THE BUSH PROBLEM This is one of the celebrated early problems of optimal control: to bring a mass to rest in a given position in minimal time, using a force of bounded magnitud e. One example would be that of bringing the rollers of a rolling mill to rest in standard position in minimal time, the angular momentu m of the rollers correspo nding to the linear momentu m of the mass. In the one-dimensional case the solution turns out to be simple: to apply maximal force first in one direction and then in the other, the order and duration being such that the mass will come to rest and to the desired position simultaneously. However, to prove the optimality of this manoeuvre, extreme and discontin uous in character, was beyond classical methods of variational calculus. Consider the one-dime nsional case; let x denote the coordinate and v the velocity. Thus (x, v) is the state variable, with the plant equation

x= v,

v= u,

(28) where u can be interpret ed as the force applied per unit mass. We suppose that \u\ : : :; M, c = 1 and that !I' consists of the single point (x, v) = (0, 0). That is, the mass is to be brought to rest at the origin in minimal time. Ifp, q are taken as the variables conjugate to x, v respectively, then

H=pv+ qu-1

(29)

jJ = 0,

(30)

u = M sgn (q)

(31)

If the terminal values ofp and q are denoted a and !3 then (30) implies that

p=a,

q=f3+a s

(32)

in terms of time-to-go s. Since u = M sgn(P) and v = 0 at termination, the relation H = 0 implies that

I.BI = 1/M. Consider first the positive option, !3 = 1/M. If a

~ 0 then q u = M, and backwards integration along the orbit yields

~

0 for s ~ 0, so that

v= -Ms, Thus x, v lie on the parabolic locus

(x

~

0, v:::::; 0)

This is the lower half of the switching locus drawn in Figure 5. If, on the other hand, a is negative, then qchanges sign ats = !3/\a\ =so, say, as then does u. If we follow the optimal path in reverse time then in this case it

146

THE PONTRYAGIN MAXI MUM PRINC IPLE

X

/14

Figure 5 The Bush problem. The two half-parabo/ae through the origin constitute the switching locus; we illustrate a path which begins with maximal deceleration, then switches to maximal acceleration when it meets the locus.

leaves the switching locus at s = so and then follows the lightly-drawn parab ola in Figure 5. In forward time, if one starte d at a point on this path then one would apply maxi mal deceleration u = - M and hold it until the switching locus was reached. One would then apply maxi mal acceleratio n u = M and move along the switching locus to the origin. The case f3 = 1/M leads corre spond ingly to the other half of the switching locus: (x ~ 0, v;;?; 0).

The maxi mum princ iple yields the relations (31) and (32) and from these we have deduc ed the optim al rule in closed-loop form, expressed in terms of the switching locus. If one starts off the locus then one applie s maxi mal deceleration or acceleration depen ding on wheth er one is above or below it. Once the locus is reached one applies maxi mal decel eratio n or accel eration depen ding upon whether one is upon the uppe r or the lower branch. Once the origin is reach ed then the force is of course removed. The control rule we have dedu ced is an example of what is terme d 'bang -bang ' control. That is, the control variable u in general takes extreme values in its perm itted set o/1, sometimes switching between these in a discontinuous fashion, as in this example. Small heatin g systems are usual ly run this way, with the gas flame either fully on or fully off; interm ediate settin gs being impracticable. Fuel economy certainly requires that rocket thrus t should be progr amm ed this way: to operate in general either at full thrus t (in some direct ion) or at zero thrus t. In the Bush example the control force was not casted, merel y limited. In the rocket case

7 ROCKET THRUST PROGRAMME OPTIMISATION

147

there is a linear costing in addition to limitation; this implies the bang-ba ng character of the optimal rule, as we shall see in the next section. The treatmen t of the Bush problem generalises to some degree to the ndimensio nal case. If we work from the vector generalisations of (30) and (31) then we deduce the partial characterisation

u = M (sa. + /3) T !sa.+ /31 of the control law, for fixed a., /3 subject to l/31 = 1j M. As an optimal path is followed we see that the direction of the control force now in general varies all the time, with up to n reversals of direction in some component. One generates all optimal orbits by variation of a. and /3. However, to deduce the closed-loop form of the control in this way seems difficult. Somewhat simpler is the LQ version of the problem, outlined in Exercise 1. Exercises and comments (1) Consider the vector version of the Bush ~roblem with no bound on u, but with an instantan eous cost function (L + Qlul ), penalisin g respectively time taken and control energy consume d during passage to terminati on at x = v = 0. Suppose, to begin with, that we prescribe the terminat ion time t. Show (and we shall treat a more general case in Section 7) that the optimal control is

!

u = _3_(3x+ 2vs) = s2 and that the control cost incurred is

-~2 (x+ vs) s

2

v s'

(33)

(34) where s = t - t is time to go. The first term in the final expression of (33) induces correctio n of final position (with an inference that the effective average velocity over the remainin g time interval is v/2) and the second term induces correctio n offinal velocity. One can now take account of the time-cos t by choosing s to minimise Ls + F(x, v, s). This will give an (x, v)-dependent evaluation of sand so oft, but the evaluation oft is necessarily constant along the optimal orbit.

!

7 ROCKE T THRUST PROGR AMME OPTIMI SATION The solution of the Bush problem of Section 5 is very much what intuition might suggest, but is nevertheless remarkable for its extreme and discontinuous character. It is just for this reason that the problem was so resistant to classical optimisa tion methods, such as as the calculus of variations. The maximum

148

THE PONTRYAGIN MAXIMUM PRINCIPLE

principle has been significant in that it provided a technique for just such cases. A source of significant problems since the 'forties has been the incentive to determine a rocket thrust program me which is optimal in the time/fue l consumption needed to achieve some given manoeuvre. We shall assume the rocket to be effectively a point mass, so that its state is specified by its coordinates in physical space (x ), its velocity (v) and its mass (m ). That is, we neglect rotation, yaw, vibration or any aspect of the rocket other than translation of its centre of mass through space and the wasting of its mass through fuel consumption. Note that x in this case is not the state vector, but simply that part of the state vector (x, v, m) which describes the Euclidea n coordinates of the rocket in physical space. Suppose that the rocket jet has a backward vector velocity k relative to the rocket (so that the absolute velocity of the materia l ~fthe jet is v-k) and that the rocket is subject to external forces of vector magnitude h = h(x, v, m, t). Then the condition of conservation of linear momen tum over a time interval of length 6t gives the equation

(m- 8m)(v +8v) + (v -k)8m = mv+h 8t or

mv=km +h,

(35)

the so-called rocket equation. We shall assume that the direction of the jet vector k can be freely controlled and that the rate mat which mass can be expelled in the rocket jet can also be controlled within limits. If we set

km=u, then we can regard the thrust vector u as the control vector, to be chosen freely subject to lui~M,

(36)

say. With these definitions the collective plant equation for the rocket can be written

x=v mv=u +h. m= -clul

(37)

If conjugate variables p, q, rare associated with x, v, m respectively, then equation s (10) become for this case

. qhXl p=-m

. -p-qh11) q=

m

(38)

7 ROCKET THRUST PROGRAMME OPTIMISATION

149

If we suppose that the cost function is purely terminal then the optimal control u should be such as to maximise the expression qu

--crlul m For a given value of ofqT:

(39)

lui the thrust vector u will then be chosen in the direction

u=qrl.~ lql' and lui will be chosen to maximise x:lul, where

That is,

1•1

~ { ~rermmaw, }

accordmg au { :

~.

(40)

The vector q is often termed the primer; its direction determines the optimal thrust direction and the magnitudes of q and r determine the optimal thrust magnitude. The paths on which lui is respectively M, 0 or intermediate in value are often called maximal thrust, null thrust and intermediate thrust arcs respectively, or simply MT, NTand IT arcs. Equations (38) simplifY somewhat if the rocket is moving in a purely gravitational field, when h = m7, where 'Y is the gravitational vector. One is thus neglecting aerodynamic effects such as drag. The vector 'Y will have the form 7(x) = if the field is a conservative one, V(x) being the potential energy associated with the gravitational field. In this case equations (38) reduce to

v;

p = -q"fx,

q= -p,

; = qufm2

and the primer obeys the equation

q=

q'Yx = qVxx

(41)

where Vxx is the matrix of second differentials of x. Equations (38) become particularly simple if Vmay be assumed quadratic, when Vxx is independent of x. As a very special case, consider the problem of maximising the height reached by a sounding rocket, assuming constant gravity and neglecting (implausibly!) effects such as aerodynamic drag. The problem can be taken as a onedimensional one in which x represents height measured upward from the starting point (the ground). The quantities v, p, q and r will all be scalar, and 'Y = - g, where g is the gravitational constant.

150

THE PONTRYAGIN MA XIM UM PRINCIPLE

The pro blem is one of max imi sing x at termination. Let mo den ote the mas s of the rocket structure, so tha t w = m - mo is the mas s of fuel remaini ng. The term inal con diti ons hol din g are the n p = 1, q = 0, and r is non-po sitive or nonnegative acc ord ing as the fuel rese rve w is positive or zero. We find from equations (38) tha tp = 1, q = s(= tim e to go) so tha t (42) But u, if non-zero, is in the directio n of q = s and so positive (upwar d-directed). Thus k = -m - 1 and l'i. mu st dec rease strictly thro ugh time. We thus see that the thru st pro gra mm e mu st tak e the form of a pha se of max ima l thru st followed by a pha se of null thru st, eith er or bot h of these phases possibly being of zero duration. Den ote the term ina l value of r by r0 • If r0 > 0 (so tha t w = 0 at terminat ion) the n it follows from (40) and (42) tha t thru st is zero for s < cmoro and max ima l for larger s, and the con diti on H = 0 implies tha t v = 0 at terminat ion . Thi s is the usual case, in which an MT pha se which exhausts fuel is followed by an NT pha se dur ing which the rocket coasts to its max ima l height, when its velo city is zero. If r0 ~ 0 the n l'i. > 0 at termination . The re is thus no NT arc; max ima l thru st is applied throughout. If ro = 0 the n it follows aga in from H = 0 tha t v = 0 at termination. If r 0 < 0 the n v < 0 at termination. The se are case s in whi ch the thru st is insufficient to lift the roc ket If initially the rocket hap pen s to be alre ady rising then max ima l thru st is app lied unt il the rocket is on the poi nt of reversing, which is take n as the term ina l inst ant. If the rocket hap pen s to be alre ady falling then term inat ion is immediate. This last discussion illustrates the literal natu re of the analysis. In disc ussing all the possible cases one com es acro ss som e which are ind eed physica lly possible but which one would hardly envisag e in practice. Exercises and comments (1) An app rox ima te reverse of the sou ndi ng rocket pro blem is tha t of soft landing: to lan d a rocket on the surface of a plan et wit h pre scri bed term inal velocity in such a way as to min imi se fuel consumption. It may be ass um ed tha t gravitational forces are vertical and constant, tha t ther e is no atm osp her e and tha t all mo tion is vertical. Not e tha t equation (42) remains valid. Hen ce show tha t the thru st pro gra mm e mu st con sist of a pha se of null thru st foll owed by one of max ima l thru st upwards (the pha ses possibly being of zero dur ation). How is the solution affected ifone also pen alises the tim e taken?

8 PROBLEMS WH ICH ARE PAR TIALLY LQ

LQ models can be equally wel l trea ted by dyn ami c pro gra mm ing or by the max imu m principle; one trea tme nt is in fact only a slightly disguise d version of

1 (

8 PROBLEMS WHICH ARE PARTIALLY LQ

151

the other. However, there is a class of partially LQ models for which the maximum principle quickly reveals some simple conclusions. We shall treat these at some length, since conclusions are both explicit and transfer in an interes ting way to the stochastic case (see Chapter 24). Assume vector state and control variables and a linear plant equation x=Ax +Bu.

(43)

Suppose an instantaneous cost function c(u)

= !uT Qu,

(44) which is quadratic in u and independent of x altogether. We shall suppos e that the only state costs are those incurred at termination. The analysis which follows remains valid if we allow the matrices A, Band Qto depend upon time t, but we assume them constant for simplicity. Howeve r, we shall allow termination rules which are both time-dependent and non-LQ , in that we shall assume that a terminal cost IK( ~) is incurre d upon first entry to a stopping set of fvalues 9', where~ is the combined state/time variable~= (x, t). We assume that any constraint on the path is incorporated in the prescri ption of g and IK, so that ~ values which are 'forbidden' belong to g and carry infinite penalty. The model is thus LQ except at termination. The assumption that state costs are incurred first at termination is realistic under certain circumstances . For example, imagine a missile or an aircraft which is moving through a region of space which (outside the stopping set 9') is uniform in its properties (i.e. in gravitational force and air density). Then no immediate position-dependent cost is incurred. This does not mean to say that spatial position is immate rial, however; one will certainly avoid any configuration of the craft which would take it to the wrong target or (in the case of the aircraft) lead it to crash. In other words, one will try to so maneoeuvre the craft that flight terminates favoura bly, in that the sum of control costs and terminal cost is minimal. This will be the interest of the problem: to chart out a course which is both econom ical and avoids hazards (e.g. mounta in peaks) which would lead to premature termination. The effect of such hazards is even more interesting in the stochas tic case, when even the controlled path of the craft is not completely predict able. It is then not enough to scrape past a hazard; one must allow a safe clearan ce. The analysis of this section has a natural stochastic analogue, which we pursue in Chapte r24. The Hamilt onian for the problem is .

H(x,u, p) = ).T[Ax +BuJ- !uTQu

if we take p = ).T as the multiplier. It thus follows that on a free section of the optimal path (i.e. a section clear of 9')

152

THE PONTRYAGIN MAXIMUM PRINCIPLE

u = Q-!BT>.

(45)

j_=-AT.A.

(46)

Consider now the optimal passage from an initial position~ = ( x, t) to a terminal position ~ = (x, t) by a path which we suppose to be free. We shall correspondingly denote the terminal value of>. by 5, and shall denote time-to-go t - t by s. It follows then from (40) and (41) that the optimal value of u is given by

(47) Inserting this expression for u back into the plant equation and cost function we find the expressions

(48) F(~, ~)

= !,\T V(s)5

{49)

for terminal x and total cost in terms of 5, where

V(s) = los ~r JeATr d-r.

(50)

and J = BQ- 1BT, as ever. In (50) we recognise just the controllability Gramian. Solving for 5 from (48) and substituting in (47) and (49) we deduce

Theorem 7.8.1 Assume the model specified above. Then (i) The minimal cost offree passage from ~to {is F(~, e)= !(x- eAsx?V(s)- 1(x- eA3 x),

(51)

and the open-loop form ofthe optimal control at ~ = (x, t) is u = Q-! BTeATs V(sr!(x- eAsx). where s = t - t. (ii) The minimal cost ofpassage from ~ to the stopping set !/, by an orbit which is free before termination, is F(e)

=

!nf[F(~,~)+K(~)].

~E9'

(52)

ifthis can be attained (53)

Expression (52) still gives the optimal control rule, with ~ determined by the minimisation in (53). This value of~ will be constant along an optimal path. We have in effect used the simple and immediate consequence (47) of the maximum principle to solve the dynamic programming equation. Relation (52) is indeed the closed-loop rule which one would wish, but one would never have

9 CONTROL OF THE INERTIALESS PARTICLE

153

imagined that it would imply the simple course (47) of actual control values along the optimal orbit. Solution of (43) and (47) yields the optimal orbit as x( r)

=eArx + V( r)eAT(I-r) X

(t ~ r ~ T)

where x = x(t) and X is determined by (48). For validity of evaluation (53) it is necessary that this orbit should not meet the stopping set !/before timet. Should it do so, then the orbit will have to break up into more than one free section, these sections being separated by grazing encounters with !/ at which special transition conditions will hold. We shall consider some such cases by example in the following sections. 9 CONTR OL OF THE INERTIA LESS PARTICLE The examples we now consider are grossly simpler than any actual practical problem, but bring out points which are importan t for such problems. We shall be able to generalise these to the stochastic case (see Chapter 24), where they are certainly non-trivial. Let x be a scalar, corresponding to the height of an aircraft above level ground. We shall suppose that the craft is moving with a constant horizontal velocity, which we can normalise to unity, so that time can be equated to horizonta l distance travelled. We suppose that the plant equation is simply

X=U, (54) i.e. that velocity equals control force applied. This would represent the dynamics of a mass moving in treacle: there are no inertial effects, and it is velocity rather than acceleration which is proportio nal to applied force. We shall then refer to the object being controlled as an 'inertialess particle'; inertialess for the reasons stated, and a particle because its dynamic state is supposed specified fully by its position. It is then the lamest possible example of an aircraft; it not merely shows no inertia, but also no directional effects, no angular inertia and no aerodynamic effects such as lift. We shall use the term 'aircraft' for vividness, however, and as a reminder of the physical object towards whose description we aspire by elaboration of the model. We have A = 0, B = 1. We thus see from (45) I (46) that the optimal control value u is constant along a free section of the orbit, whence it follows from the plant equation (54) that such sections of orbit must be straight lines. We find that · V(s) = Q- 1s, so that F(~, ~) = Q(x- x) 2 j2s. Suppose that the stopping set is the solid, level earth, so that the region of free movement is x > 0 and the effective stopping set is the surface of the ground, x = 0. The terminal cost can then be specified as a function IK(t) of time (i.e. distance along the ground). The expressions (53) and (52) for the value function and the closed-loop optimal control rule then become

154

THE PONTRYAGIN MAXIMUM PRINCIPLE

F(x, t) =

!~[~ + IK(t+s)],

(55)

u = -xjs.

(56) Here the time-to-go s must be determined from the minim isation in (55), which determines the optim al landing-point 1 = t + s. The rule (56) is indee d consistent with a const ant rate of descent along the straight-line path joining ~. t) and (0,7). However, suppose there is a sharp moun tain between the startin g point and the desired termin al point 1 determ ined above, sufficiently high that the path determined above will not clear it. That is, if the peak occur s at coordinate t 1 and has height h then we require that x(ti) >h. If the optim al straight-line path determined above does not satisfy this then the path must divide into two straight-line segments as illustrated in Figure 6. The total cost of this comp ound path is

Q(h -x) 2 F1(x, t) = 2 ( ) +F(h ,t1), ti-t

(57)

where Fis the 'free path' value function determined in (55). It is more realistic to regard a crash on the moun tain as carrying a high cost, K 1, say, rather than prohobited. In the stochastic case this is the view that one must necessarily take, becau se then the crash outcome always has positive probability. If one resigns and chooses to crash then there will be no control cost at all and the total cost incurr ed will be just K 1• One will then choose the crash option if one is in a position (x, t) for which expression (57) exceeds K1. i.e. for which

x
(58)

where dis the const ant (2/Q)[K1 - F(h, t 1)]. For K1lar gethis can occur only if the craft is much closer to the moun tain than it is to the peak; see Figure 6. X

t Figure 6 The straight-line segments ofa path surmounting a peak. The curve is the switching locus (determined by equality in (58)) on which one switchesfrom endeavouring to clear the peak to accepting a crash.

10 CONTROL OF THE INERTIAL PARTICLE

155

10 CONTROL OF THE INERTIAL PARTICLE The feature that must most urgently be added to the model above is that of inertia for vertical motion. Let us take the state variable as having compo nents x and v, where xis again height and vis vertical velocity: rate of increas e of height. We a.re thus guilty of an inconsistency in that x denotes a compo nent of the state vector rather than the full vector. However, continuity with the discuss ion above makes this desirable. We assum e then the plant equation

x= v, v= u, (59) and again the instantaneous cost functi on! Qul. Any coeffic ients which might have occurr ed in the second equation have been norma lised to unity by scalechanges in the variables. We have then As=

e

[10

s] 1 ·

It follows then from (45)/ (46) that the optimal control is linear in time, and so from the plant equati on (59) that, along the optimal orbit, the height is cubic in time:

(t

~

T

~

t).

Here a and (3 are coefficients to be determ ined from the termin al conditions. To determ ine the control rule in closed-loop form, we note first the evaluation

V(s) = Q-I [ ~~;

S

t2].

2

Expressions (51) and (52) for minim al cost of passage from and

~then becom e

F(~.~)

optimal control at

= (6Q/; )[x- x-! (v + v)sf + (Q/2s )(v- v) 2, u=(2 jl)[3 (x-x) -2vs- vs].

(60)

(61) If one is consid ering passage to fl' rather than to a prescribed ~then ~is again determ ined by the minimisation in (53). The control rule (61) is in the closed-loop form that one would wish, but indeed one would never have guessed from this that the optima l control in fact varies linearly with time. These evaluations are of course valid only if passage is free. To consider what happens in other cases, suppose we indeed consider an optima l 'landing' in that ~ = (.X, v, t) is prescr ibed as (0, w, t). That is, one is require d to land at timet (corresponding to prescription of a position on the ground) with vertical velocity w (necessarily non-positive). Then a cubic path satisfying initial and terminal conditions (and so the unique optima l free path) could take any of the forms

156

THE PONTRYAGIN MAXIMUM PRINCIPLE

't

(c)

(d)

Figure 7 The various possibilities for the free trajectory ofthe optimally controlled inertial particle.

(i}-{iv) of Figure 7. Cases (i) and (ii) are acceptable, because the orbit meets the stopping set (the ground) flrst at the desired terminal point. In cases (iii) and (iv) the aircraft begins in so violent a dive that it overshoots the plane x = 0 before it finally approaches the desired terminal point in the prescribed fashion. But such an 'Qvershoot' is a crash-a premature encounter with 9'-and is effectively forbidden if the penalty is set high enough. Note that case (iv) could not occur if w < 0; it is possible only if w = 0, and can be regarded as a limit version of case (iii). In either of these cases the actual optimal orbit must break into two parts, as illustrated in Figure 8. First the pilot makes strenuous efforts to come out of the dive, which he does at time t1, say.

Figure 8 The case in which the optimal path to a desired terminal configuration at t breaks into two parts: a grazing escape.from crash at ttfollowed by afree trajectory to termination.

10 CONTROL OF THE INERTIAL PARTICLE

157

Economy of control dictates that he only just escapes a crash, in that as he comes out of the dive (i.e. v = 0) he only just misses the ground (i.e. x = 0+). Having come out of the dive he approaches the desired terminal point by an optimal path which is free. The two sections of the path (those before and after tJ) are indeed distinct in that they are distinct cubic curves. We have to work out how the optimal grazing point t 1 is determined, and the transition conditions obtaining there. Optimisation of t1 means that the pilot has the desired terminal configuration~ in mind and comes out of the dive in such a way as to optimise costs after as well as before t 1• But this is to demand too much. In practice a pilot will simply concentrate on the immediate emergency, knowing that this is the priority. Consideration of what is to happen once the emergency is over is generally deferred, partly because one does not have the processing power to do anything else and partly because later events are secondary in cost terms. We shall find support for the latter assertion. Let us first consider, then, the problem of pulling out of a dive as economically as possible, without consideration of what is to happen afterwards. The cubic path must then take the form of Figure 9, touching x = 0 from above at a point t to be determined. We use the notation t because this is a terminal point as far as this phase of the operation is concerned.

e

Theorem 7.10.1 Suppose the initial configuration is (x, v, t) with v < 0. Then the closed-loop control rule which minimises the cost ofpulling out ofthe dive is

2v2

u = 3x.

(62)

This pullsthecraftoutofth ediveaftera times= t- tequalto -3xjv, and at a cost of

-(2Qv3 j9x). Proof We may as well set t = 0 and t = s. We know from (60), (61) that the minimal cost offree passage from (x, v, 0) to (0, 0, s) is

(63)

Figure 9 The presumedform ofan optimal crash-avoiding trajectory.

158

THE PONTRYAGIN MAXIMUM PRINCIPLE

and that the closed-loop rule which achieves it is u

= -(2j?)(3x + 2vs).

(64)

We can optimise with respect to s. The value of s minimising expression (63) for v negative is s = -3xjv; at this value the cost F and the control u have the evaluations asserted above. We have to confirm that the path thus generated does not in fact cross x = 0 before times. This path x(r) is a cubic with x(O) = x, x!(O) = v, and a double zero at -3xjv. These conditions determine the cubic, and its third zero is found also tolieat-3xjv. 0 Note a consequence of the last observation: that - 3x / v is in fact the largest value which could be chosen for s; any larger and the path would have crossed the axis before time s. At this point one will switch from control rule (62) to a zero control if crash avoidance is all that is demanded. The uncontrolled path now avoids !/, if only just. We can now return to the landing problem, and the question of determining whether the path to landing is free or is broken by a grazing encounter with the ground. In this latter case, we should also determine the point at which break occurs. We shall now use t to denote the time of landing (prescribed) and shall suppose that if there is an earlier grazing then it occurs at t 1• Theorem 7.10.2 Consider the landing problem enunciated above; sets = t - t. Then the optimalpath is a free one unless both the following inequalities hold: 3x+ vs

< 0,

(3x + vs) 2 w > - -'----:---':...._ 4xs

(65)

If these inequalities do indeed both hold then the optimal path suffers a grazing encounter with the ground after a time SJ = t 1 - t determined as the unique positive root of (3x + vsi) 2

w2

sj

(s- s 1) 2

(66)

which is less than -3xjv.

The first condition of (65) states that the prescribed landing time must be later than the time at which it would be optimal to pull out of the dive without consideration of the sequel. The second condition implies that, even if the first holds, the optimal path to the termination point will still be free if the required terminal rate of descent is large enough. That is, if one is willing to accept a crash landing at the destination!

10 CONTROL OF THE INERTIAL PARTICLE

159

Proof If the path is a free one then it has the cubic form given above. We may as well normalise the time origin by setting t = 0 and so t = s in this relation. The coefficients a and f3 in the cubic are then determined by the terminal conditions x(s) = O,x'(s) = w. The cubic then has a root at-r = s, and one finds that theremaining two roots are the roots of the quadratic ~- 2a-r+b

= 0,

(67)

where

2a =

s(x + vs) 2x+ (v+ w)s'

-Slx

b = ::------c;------;2x+ (v+ w)s·

The only case (consistent with x > 0, w < 0) in which the optimal path is not free is the case (iii) of Figure 7, so this is the one we must exclude. This will be the case in which the quadratic (67) has both roots between 0 and s. This in turn is true if and only if the quadratic expression is positive at both 0 and s, and has a turning point inside the interval at which it is negative. That is: b

< 0, ? - 2as + b > 0, 0 < a < s,

c? > b.

We find that the first two of these conditions are both equivalent to

2x+(v+w)s
(68)

This last with the inequality a > 0 implies that

x+vs < 0 The condition a

(69)

< sis equivalent to 3x + vs + 2ws < 0,

(70)

and the final condition, a 2 > b, is equivalent to

(3x + vs) 2 + 4xws > 0.

(71)

The free path is non-optimal if and only ifall of relations (68)-(71) hold. Relations (70) and (71) give the bounds on w

3x + vs (3x + vs) - - < - w < .o._---:----''2

2s

4xs

The upper bound in this relation exceeds the lower bound by (3x + vs)(x + vs)/ (4xs). It follows from (69) that the interval thus determined for w is empty unless 3x + vs < 0, a relation which implies (68), (69) and (70). We are thus left with the pair of conditions (65) asserted in the theorem. In the case that both these conditions are fulfilled the optimal path cannot be free, and is made up of two free segments meeting at time t1 = t + SJ. We choose s 1 to minimise the sum of the cost incurred on the two free segments, as given by expression (60); the stationarity condition (66) emerges immediately. It follows

160

THE PONTRYAGIN MAXIMUM PRINCIPLE

v FigrudO A graphical illustration ofthe solution of equation

the grazing point

(66)for the optimal timing SJ of

from the observation after Theo rem 7.10.1 that the root s 1 must be less than - 3x j v. Indeed, equation (66) has a single such root as we see from Figure 10; the left- and right-hand members of (66) are respectively decreasing and increasing, as functions of s1. in the interval 0 :::::; s1 :::::; - 3xfv. 0 Indeed, we can determine s 1 explicitly. In takin g square roots in (66) we must take the negative optio n on one side, because 3x + vs 1 is positive whereas w is negative. The appropriate root of the resulting quad ratic in s 1 is

31

(3x- vs) + V(3x + vs) 2 - 12xws = 2(w~v) '

atlea stifw - v > 0, whic hwem ayex pect. Thisa ppro

ache s-3xf vasw tends tozer o.

11 AVOIDANCE OF THE SfOPPING SEf: A GEN ERAL RESULT The conclusions of Theo rem 7.10.1 can be generalised , with implications for the

stochastic case. We consider the general mode l of Section 9 and envisage a situation in which the initial conditions are such that the uncontrolled path would meet the stopping set !/'; one's only wish is to avoid this encounter in the most economical fashion. Presumably the optim al avoiding path will graze !/ and then continue in a zero-cost fashion (i.e. subsequently avoid !/' without further control). We shall speak of this grazing point as the 'term inatio n' point, since it indee d marks the end of the controlled phase of the orbit. We then consider the linear system (43) with contr ol cost (44). Suppose also that the stopping set !/ is the half-space

11 AVOIDANCE OF THE STOPPING SET: A GENERAL RESULT

9'={x:aTx::;;;b}

161 (72)

This is not as special as it may appear; if the stopping set is one that is to be avoided rather than sought then its boundary will generally be (n - 1)dimensional, and can be regarded as locally planar under regularity conditions. Let us denote the the linear function aTx of state by z. Let F( ~) be the minimal cost of transition by a free orbit from an initial point = (x, 0) to a terminal point ~ = (x, t), already evaluated in (51). Then we shall find that, under certain conditions, the grazing point ~of the optimal avoiding orbit is determined by minimising F(e, ~)with respect to the free components of ~at termination and maximising it (at least locally) with respect to i That is, the grazing point is, as far as timing goes, the most expensive point on the surface of 9' on which to terminate from a free orbit. The principal condition required is that aTB = 0, implying that the component z is not directly affected by the control. This is equivalent to aTJ = 0. Certainly, if one is to terminate at a given timet then the value of z is prescribed as b, but one should then optimise over all other components of x. Let the value of F(e, ~) thus minimised be denoted G(e, t). The assertion is then that the optimal value oft is that which maximises G( t). We need a preparatory lemma.

e

e,

e,

Lemma 7.11.1 Add to the prescriptions (43), (44) and (72) ofplant equation, cost function and stopping set the conditions that the process be controllable and that aTB =O.Then (73)

e,

Proof Optimisation ofF( ~) with respect to the free components of x will imply that the Xof (47) is proportional to a, and so can be written Oa for some scalar 0. The values of() and tare related by the termination condition z = b, which, in virtue of (48), we can write as aTeA1x

+ ()aTV(t)a =b.

(74)

We further see from (49) that

G(e, t) = !fiaTV(t)a.

(75)

Let us write V(t) and its derivative with respect to t simply as V and V. Controllability then implies that aTVa > 0. By replacing T by s- T in the integrand of (50) we see that we can write Vas

V=J+AV+ VAT.

(76)

Differentiating (74) with respect tot we fmd that

(aTVa)(dOfdt) +aT AeA1a + oaTva = 0,

(77) .

162

THE PONTRYAGIN MAX IMUM PRIN CIPL E

so that

oG = !B2aTVa + OaTVa(dOfdt) =-B aT AeA1x- !01aTVa.

at

Finally, we have

z= ar(Ax + J5..) =aTAx= aTAeA x+ oar AVa = arAeA x +!Oarva. 1

1

The second equality follows beca use aT J = 0 and the fourth by appeal to (76). We thus deduce from the last two relations that 8Gja t = -Bz. Inser ting the evaluation of eimpl ied by (74) we deduce (73). Theorem 7.11.2 The assumptions of Theorem 7.11.1 imply that the grazing poin t ~ ofthe optimal Y' -avoiding orbit is determined by first mini misin g F (.;, ~) with respect to x subject to aTx = band then maxi misin g it, at least locally, with respect tot. Proo f As indic ated above, optimality will require the restricted x-optimisation; we have then to show that the optim al t maxi mises G(~, !). At any value t for which the controlled orbit crosses z = 0 the uncontrolled orbit will lie below z = 0, so that aT eA7x - b < 0. If, on the contr olled orbit, z decreases throu gh zero at time !, then one will increase tin an attem pt to find an orbit which does not enter :/.W e see from (73) that G(x, t) will then increase. Correspondingly, if z crosses zero from below then one will decrease t and G(x, t) will again increase. If the two original controlled orbits are such that the t values ultimately coincide unde r this exercise, then = 0 at the com mon value, so that the orbit grazes :/ locally, and G(x, t) is locally max imal with respect tot. The true grazing poin t (i.e. that for which the orbit meets Y at no othe r point) will be found by repeated elimination of crossings in this fashion. 0

x

One conjectures that G(x, 7) is in fact globally maxi mal with respect to 7 at the grazing point, but this has yet to be demonstra ted. We leave the reader to conf irm that this criterion yields the know n graz ing poin t t = -3xf v for the crash avoidance problem of Theo rem 7.10.1, for which the cond ition aT B = 0 was inde ed satisfied. We should now deter mine the optim al Y -avoi ding control explicitly. Theorem 7.11.3

Defin e

~ = ~(x,s)

= b- aTeAsx,

a= a(s) = JaTV (s)a .

Then, under the conditions of Theorem 7.11.1, the optim

al :/-av oidin g control at xis

(78)

12 CONSTRAINED PATHS: TRANSVERSALITY CONDITIONS

163

where s is given the values which maximises (~I (j) 2 . With this understanding ~I rf2 is constant along the optimal orbit (before the grazing point) and s is the time remaining until grazing.

e

Proof Let us set ~ = (X' 0)' = (X' s) so that X is the value of state when a time s remains before termination. The cost of passage along the optimal free orbit from~to eis (79) where V = V(s) is given by (50) and 8 = x- eAsx. The optimal control at time t = 0 for prescribed is

e

(80) The quantity v- 18 is the terminal value of A. and is consequently invariant along the optimal orbit. That is, if one considers it as a function of initial value ~then its value is the same for any initial point chosen on that orbit. Specialising now to the Y' -avoiding problem, we know from the previous theorem that we determine the optimal grazing point by minimising F (~, e) with respect to x subject to z = b and then maximising it with respect to s. The first minimisation yields (81) so the values of sat the optimal grazing point is that which maximises (~I (j) 2 • Expression (78) for the optimal control now follows from (80) and (81). The identification of ~I dl with v- 18 (with s determined in terms of current x) D demonstrates its invariance along the optimal orbit (before grazing). 12 CONSTRAINED PATHS: TRANSVERSALITY CONDITIONS We should now consider the transition rules which hold when a free orbit enters or emerges from a part of state space in which the orbit is constrained. We shall see that conclusions and argument are very similar to those which we deduced for termination in Section 3. Consider first of all the time-invariant case. Suppose that an optimal path which begins freely meets a set F in state-space which is forbidden. We shall assume that § is open, so that the path can traverse the boundary 8§' ofF for a while, during which time the path is of course constrained. One can ask: what conditions hold at the optimal points of entry to and exit from oF? Let x be an entry point and p and p' be the values of the conjugate variable immediately before and after entry. Just as for the treatment of the terminal problem in Section 3, we can partially integrate the Lagrangian expression for minimal total cost up to the transition point and so deduce an expression whose primary dependence upon the transition value x occurs in a term (p - p')x.

164

TH E PONTRYAGIN MA XIM UM PRI NC IPL E

Suppose we can vary x to x + w· + o( c), a neighbouring poi nt in 8-F . Then, by the same argument as tha t of Th eor em 7.3.1 we deduce tha t (p - p') a is zero if x is an optimal transition value. Th at is, the linear function pa of the conjugate variable is continuous at an opt ima l tran sition point for all directions a tangential to the surface of F at x. Otherwise expressed, the vector (p - p') T is nor ma l to the surface of .F at x. We deduce the sam e conclu sion for optimal exit points by appeal to a timereversed version of the proble m. Transferring these results to the time-varying problem by the usual device of taking an augmented state variable~= (x, t) we thus deduce *Th eor em 7.12.1 Let .F be an open set in (x, t) space whi ch is forbidden. Let (x, t) be an opt ima l transition poi nt (fo r eith er ent ry to or exi t fro m 8-F ) and (p, Po) and p',p the values of the aug me nte d conjugate variable imm edi ate ly before and after transition. The n

0)

(p - p') a + (po -

p~)r =

0 (82) for all directions (a, r) tan gen tial to 8-F at (x, t). In particular, if t can be var ied ind epe nde ntly of x then the Ha mil ton ian H is con tinu ous at the transition. Pro of Th e first assertion follows from the argument bef ore the theorem, as indicated. If we can vary tin bot h directions for fixed x at transiti on then (82) implies tha t p 0 is continuous at transiti on. Bu t we have p 0 + H = 0 on bot h sides of the transition, so the implication is tha t Hi s continuous at the transition. 0 One can also develop conclu sions concerning the form of the optimal pat h during the phase when it lies in 8-F. However, we shall con ten t ourselves with what can be gained from discus sion of a simple example in the next section. An example we have already considered by a direct discus sion of costs is the steering of the inertial particl e in Section 10. For this the sta te variable was (\", v) and ff was x < 0. Th e bou nda ry 8-F is then x = 0, but on this we mu st also require tha t v = 0, or the condition x ;;?: 0 would be vio late d in either the immediate pas t or the imm edi ate future. Suppose tha t we sta rt fro m ~. v) at t = 0 and reach x = 0 first at time t (when necessarily v = 0 as wel l). Sup the variables conjugate to x, pose p, q are v, so tha t the Hamiltonian is H = pv + qu - Qu2 /2, and so equal to pv + q2 j2Q when u is chosen optimally. Continuity of H at a transition point, when vis nec essarily zero, thus amounts to continuity of q2. Let us con firm tha t this con tinuity is consistent with the previously derived condition (66). Ifp and q den ote the values of the conjugate var iables at t 1 -, jus t before transition, then integra tion of equation (46) leads 1 to p(r ) = p, u(r ) = Q- q(r ) = Q- 1(q + ps) , v(r ) = -Q - 1(qs + pil /2) , x(r 1 ) = Q(qi l /2 + ps3/6) , where s = t 1 - r. The values of p and q are determined by the prescribed initial

13 REGULATION OF A RESERVOIR

165

values x and v. We find q proportional to (3x + vt) / t2, and one can find a similar expression for q immediately after transition in terms of the values 0 and w of terminal height and velocity. Assertion of the continuity of q2 at transition thus leads exactly to equation (66). 13 REGULATION OF A RESERVOIR This is a problem which the author, among others, has discussed as 'regulation of a dam'. However, purists are correct in their demur that the term 'dam' can refer only to the retaining wall, and that the object one wishes to control, the mass of water, is more properly referred to as the 'reservoir: Let x denote the amount of water in the reservoir, and suppose that it obeys the plant equation .X = v - u, where v is the inflow rate (a function of time known in advance) and u is the draw-off rate (a quantity at the disposition of the controller). One wishes to maximise a criterion with instantaneous reward rate g( u), where g is concave and monotonic increasing. This concavity will (by Jensen's inequality) discourage variability in u. One also has the natural constraint u ~ 0. The state variable x enters the analysis by the constraint 0 ~ x ~ C, where C is the capacity of the reservoir. We shall describe the situations in which x = C, x = 0 and 0 < x < Cas fun empty and intermediate phases respectively. One would of course wish to extend the analysis to the case for which v (which depends on future rainfall, for example) is imperfectly predictable, and so supposed stochastic. This can be achieved for LQ versions of the model (see Section 2.9) but is difficult if one retains the hard constraints on x and u and a non-quadratic reward rate g( u). We can start from minimisation of the Lagrangian form J[-g(u) + p(xv + u)] dr, so that the Hamiltonian is H(x, u,p) = g(u) + p(v- u). A price interpretation would indeed characterise p as an effective current price for water. We then deduce the following conclusions.

Theorem 7.13.1 An optimal draw-offprogramme shows thefollowingfoatures. (i) The value ofu is the non-negative value maximising g(u)- pu, which then increases with decreasingp. (ii) The value ofu is constant in any one intermediate phase. (iii) The value ofp is decreasing vncreasing) and so the value ofv = u is decreasing vncreasing) in an empty (full) phase. (iv) The value ofu is continuous at transition points. Proof Assertion (i) follows immediately from the form of the Hamiltonian and the nature of g. Assertions (ii) and (iii) follow from extremisation of the Lagrangian form with respect to x. In an intermediate phase x can be perturbed either way, and one deduces that jJ = 0 (the free orbit condition for this particular case). Hence p, and so u, is constant in such a phase. In an empty phase perturba-

166

THE PONTRYAGIN MAX IMU M PRIN CIPLE

tions of x can only be

non-negative, and so one can deduce only that P :::;: 0. Thu s p is decreasing, and so u is increasing. Since u and v are necessarily equal, if x is

being held constant, then v mus t also be increasing. The analogous assertion s for a full phase follow in the sam e way; perturbations can then only be nonpositive. The final assertion follows from continui ty of Hat transition points. With u set equal to its opti mal value H becomes a monotonic, continuous function of p. Continuity of Hat transition points then implies continuity ofp, and so of u. 0 The example is interesting for its appeal to transversality conditions, but also because there is some discussion of opti mal behaviour during the empty and full phases (which constitute the bou nda ry off of the forbidden region!!': the union of x < 0 and x > C). Trivially, one mus t have u = v in these phases. However, one should not regard this as the equation dete rmining u. In the case x = 0 (say) one is always free to take a smaller value of u (and so to let water accumulate and so to move into an intermediate phase). The optimal draw-off rate continues to be determined as the value extremising g( u) - pu; it is the development ofp which is constrained by the condition x = 0. Alth ough the rule u = v is trivial if one is determined not to leave the empty phase, the conclusion that v mus t be increasing during such a phase (for optimali ty) is non-trivial.

Notes Pontryagin is indeed the originator of the principle which bears his name, and whose theory and application has been so developed by himself and others. It is notable that be held the dyn ami c prog ram min g principle in great scorn; M.H.A. Davis describes him memorably as hold ing it up 'like a dead rat by its tail' in the preface to Pontryagin et al. (1962). This was because of the occasional nonexistence of the derivative Fx in the simp lest of cases. However, as we have seen, it is a rat which alive, ingenious, direct, and able to squeeze through where authorities say it cannot. The material of Section 11 is believed to be new.

i

I

1

I \

l

PART 2

Stochastic Models

CHAPTER 8

Stochastic Dynamic Programming A difficulty which must be faced is that of incompleteness of information. That is, one may simply not have all the information needed to make an optimal decision, and which we have hitherto supposed available. For example, it may be impossible or impracticable to observe all aspects of the process variable-tb.e workings of even a moderate-sized plant, or of the patient under anaesthesia which we instanced in Section 5.2, are far too complex. This might matter less if the plant were observable in the technical sense of Chapter 5, so that the observations available nevertheless allowed one to build up a complete picture of tb.e state of affairs in the course of time. However, there are other uncertainties which cannot be resolved in this way. Most systems will have exogenous inputs of some kind: disturbances, reference signals or time-varying parameters such as price or weather. If the future of these is imperfectly predictable, as is usually the case, then the basis for the methods we have used hitherto is lost. There are two approaches which lead to a natural mathematical resolution of this situation. One is to adopt a stochastic formulation. That is, one arrives somehow at a probability model for plant and observations, so that all variables are jointly defined as random variables. The variables which are observable can then be used to make inferences on those which are not. More specifically, one chooses a policy, a control rule in terms of current observables, which minimises the expectation of some criterion based on cost. The other, quite as natural mathematically, is the minimax approach. In this one assumes that all unobservables take the worst values they can take (judged on the optimisation criterion) consistently with the values of observables. The operation of conditional expectation is thus replaced by a conditional maximisation (of cost). The stochastic approach seems to be the one which takes account of average performance in the long run; it has the completer theory and is the one usually adopted. The minimax approach corresponds to a worst-case analysis, and is frankly pessimistic. We shall consider only the stochastic approach, but shall find minimax ideas playing a role when we later develop the idea of risk-sensitivity. Lastly, there is a point which should be made to maintain perspective, even if it cannot be followed up in this volume. The larger the system (i.e. the greater the number of individual variables) then the more unrealistic becomes the picture that there is a central optimiser who uses all currently available information to make all necessary decisions. The physical flow of information and commands

170

STOCHASTIC DYNAMIC PROGRAMMING

would be excessive, as would the central processing load. This is why an economy or a biological organism is partially decentralised: some control decisions are made locally, on the basis of local information plus central commands, leaving only the major decisions to be made centrally, on aggregated information. Indeed, the more complex the system, the greater the premium on trading a loss in optimality for a gain in simplicity-and, perhaps, the greater the possibility of doing so advantageously, and of recognising the essential which is to be optimised. We use the familiar notations E(x) and E(xiy) for expectation and conditional expectation, and shall rarely make the notational distinction (which is only occasionally called for) between random variables and particular values which they may adopt. Correspondingly, P(x) and P(xiy) denote the probability (unconditional and conditional) of a particular value x, at least if x is discretevalued. However, more generally and more loosely, we also use P(x) to denote simply the probability law of x. So, the Markov property of a process {x 1} would be expressed, whatever the nature ofthe state space, by P(xt+1IX1 ) = P(xt+ 1 lx 1), where X 1 is the history {X 7 ; T ~ t }. 1 ONE-STAGE OPTIMISATION A special feature of control optimisation is that it is a multi-stage problem: one makes a sequence of decisions in time, the later decisions being in general based on more information than the earlier ones. For this very reason it is helpful to begin by considering the single-stage case, in which one only has a single decision to inake. For example, suppose that the pollution level of a water supply is being monitored. One observes pollution level y in the sample taken and has then the choice of two actions u: to raise the alarm or to do nothing. In practice,. of course, one might well convert this into a dynamic problem by allowing sampling to continue over a period oftime until there is a more assured basis for action one way or the other. However, suppose that action must be taken on the basis of this single observation. A cost C is incurred; the costs of raising the alarm (perhaps wrongly) or of not doing so (perhaps wrongly). The magnitude of the cost will then depend upon the decision u and upon the unknown 'true state' of affairs. Let us denote the cost incurred if action u is taken by C(u1 a random variable whose distribution depends on u. One assumes a stochastic (probabilistic) model in which the value of the cost C(u) for varying u and of the observable y are jointly defined as random variables. Apolicy prescribes u as function u(y) of the observable u; the policy is to be chosen to minimise E[C(u(y))]. Theorem 8.1.1 The optimal decision function u(y) is determined by choosing u as the value minimising E[C(u)ly]. Proof If a decision rule u(y) is followed then the expected cost is

,

1 ONE-STAGE OPTIMISATION

171

E[C(u(y))] = E{E[C(u(y))iy]} ~ E{inf E[C(u)iy]} u

and the lower bound is attained by the rule suggested in the theorem.

0

The theorem may seem trivial, but the reader should understand its point: the reduction of a constrained minimisation problem to a free one. The initial problem is that of minimising E[C(u(y)] with respect to the jUnction u(y), so that the minimising u is constrained to be a function of y at most. This is reduced to the problem of minimising E[C( u) IY]freely with respect to the parameter u. One might regard u as a variable whose prescription affects the probability distribution of the cost C,just as does that of y, and so write E[C(u)iy] rather as E[qy, u]. However, to do this is to blur a distinction between the variablesy and u. The variable y is a random variable whose specification conditions the distribution of C. The variable u is not initially random, but a variable whose value can be chosen by the optimiser and which parametrises the distribution of C. We discuss the point in Appendix 2, where a distinction is made by writing the expectation as E[qy; u], the semicolon separating parametrising variables from conditioning variables. However, while the distinction is important in some contexts, it is not in this, for reasons explained in Appendix 2. The reader may be uneasy: the formulation of Theorem 81.1 makes no mention of an important physical variable: the 'true state' of affairs. This would be the actual level of pollution in the pollution example. It would be this variable of which the observationy is an imperfect indicator, and which in combination with the decision u determines the cost. Suppose that the problem admits a state variable x which really does express the 'true state' of affairs, in that the cost is in fact a deterministic function C(x, u) ofx and u. So, if one knew x, one would simply choose u to minimise C(x, u). However, one knows only y, which is to be regarded as an imperfect observation on x. The joint distribution of x andy is independent of u, because the values of these random variables, whether observable or not, have been realised before the decision u is taken.

Theorem 8.1.2 Suppose that the problem admits a state variable x, that C(x, u) is the cost jUnction andf(x,y) the joint density ofx andy with respect to a product measure Jl.i (dx)p,2 (dy ). Then the optimal value of u is that minimising J C(x, u) f(x,y)P,i dx. Proof Let us assume for simplicity that x and y are discrete random variables with a joint distribution P{?c, y); ~e formal generalisation is then clear. In this case E[C(u)iy] =

L C(x, u)P(xiy) ex: L C(x, u)P(x,y) = L C(x, u)P(x)P(yJx), X

X

X

{1)

172

STOCHASTIC DYNAMIC PROG RAMM ING

where the proportionality sign indicates a factor P(y) -I, indep enden t of u. The third of these expressions is the analogue of the integr al expression asserted in the theorem. 0 We give the fourth expression in (1) because P(x) and P(yjx) are often specified on fairly distinct grounds. The conditional distributio n P(y lx) of observation on state is supplied by one's statistical model of the observation process, whose mechanism may be fairly clear. The distribution P(x) constitutes the 'prior distribution' of state and its specification may be debat able; see Exercise 1. Exercises and comments (1) The so-called two-hypothesis two-action case is the simplest, but is both illuminating and useful. Cons ider the pollution exam ple of the text, and suppose that serious pollution of the river, if it occurs, can only be due to a catastrophic failure at a factory upstream. There are then only two 'pollu tion states', that this failure has not occurred or that it has, corresponding to x equal to 0 or 1, say. Denote the prior probabilities P(x) of these by ?rx and the probability density of the observation y conditional on the value of x by fx(y ~ Suppose that there are just two actions: to raise the alarm or not. The cost of raising the alarm is Co or zero according as x is 0 or 1; the cost of not raisin g the alarm is zero or C1 according as xis 0 or 1. It follows then from the last form of the criterion that one should raise the alarm if ?ro/o(y)Co < 7rifi(y)CJ. That is, if the likelihood ratio fi(Y)/fo(y) exceeds the threshold value 7roCo/7ri C1.

(2) Risk-sensitivity and hedging. The new effects that a stochastic element can bring can be demonstrated on a one-stage model. Supp ose that, if one divides a sum of money x so that an amou nt x1 is invested in activi ty j, then one receives a total return of r = E1 CjXJ. That is, c1 is the rate of return on activity j. If the c1 are known then one maximises r by investing the whole sum in an activity for which the return rate Cj is greatest. (If there are severa l activities achieving this rate then the investment can be spread over them, but there is no advantage in such diversification). If the c1 are unkn own, and regar ded as rando m variables, then one maximises the expected retur n by inves ting the whole sum in an activity for which the expected rate of return E( c ) is maximal, where this is an 1 expectation conditional on one's information at the time of the investment decision. However, suppose one chooses rathe r to extremise the expected value of exp( Or), maximising or minimising according as the risk-sensitivity parameter() is positive or negative. This criterion takes account ofvar iability of return as well as of expectation, variability being welcome or unwelcome according as () is positive or negative (the risk-seeking and risk-averse cases; see Chap ter 16).

2 MULTI-STAGE OPTIMISATION

173

For simplicity, let us assume (rather unrealistically) that the random variables c1 are independent. Then one chooses the allocation to maximise '2:.1 B- 1Fj(Bx1 ), where Fj(o:) = log{E[exp(o:cJ)]}. The functions Fj(o:) are convex (see Appendix 3). It follows then that the functions e-t Fj(Bx1) are convex or concave according as one is in the risk-seeking or risk-averse case. In the first case will invest the whole sum in an activity j for which e- 1Fj( Bx) is maximal. In the second case one spreads the investment (hedges against uncertainty) by choosing the x1 so that Fj(Bx1) ~>.,with equality if x1 is positive (j = 1, 2, ... ). Here>. is a Lagrange multiplier chosen so that '2:.1 x1 = x. Hedging is a very real feature in investment practice, and we see that it is induced by the two elements of uncertainty and riskaverseness. (3) Follow through the treatment of Exercise 2 in the case (again unrealistic!) of normally distributed CJ> when Fj(o:) = p1a+!v1a 2• Here 1-LJ and v1 are respectively the mean and variance of c1 (conditional on information at the time of decision). 2 MULTI-STAGE OPTIMISATIO N; THE DYNAMIC PROGRAMMI NG EQUATION If we extend the analysis of the last section to the multi-stage case then we are essentially treating a control optimisation problem in discrete time. Indeed, the discussion will link back to that of Section 2.1 in that we arrive at a stochastic version of the dynamic programming principle. There are two points to be made, however. Firstly, the stochastic formulation makes it particularly clear that the dynamic programming principle is valid without the assumption of state structure and, indeed, that state structure is a separate issue best brought in later. Secondly, the temporal structure of the problem implies properties which one often takes for grap.ted: this structure has to be made explicit. Suppose, as in Section 2.1, that the process is to be optimised over the time period t = 0, 1, 2, ... , h. Let W0 indicate all the information available at time 0; it is from this and the stochastic model that one must initially infer plant history up to time t = 0, insofar as this is necessary. Let x 1 denote the value of the process variable at time t, and X 1 the partial process history {x1,x2 , •.. ,x1 }. Correspondingly let y 1 denote the observation which becomes available and u1 the control action taken at time t, with corresponding partial histories Y 1 and Ut. Let W1 denote the information available at time t; i.e. the information on which choice of u1 is to be based. Then we assume that W1 = {Wo, Y, U1-t}. That is, current information consists just of initial information plus current observation history plus previous control history. It is taken for granted, and so not explicitly indicated, that W1 also implies prescription of the stochastic model and knowledge of clock time t.

174

STOCHASTIC DYNAMIC PROGRAMMING

A realisable policy 1ris one which specifies u1 as a function of W 1 fort = 1, 2, ... , One assumes a cost function C. This may be specified as a function of Xh and Uh-l, but is best regarded simply as a random variable whose distribution, jointly with that of the course of the process and observations, is parametrised by the chosen control sequence Uh-I· The aim is then to choose 1rto minimise E,,.(C). Define the total value function uh-I.

G( W 1)

=

inf ,. E1r[CI Wt],

(2)

the minimal expected cost conditional on information at timet. Here E,. is the expectation operator induced by policy 11: We term G the total value function because it refers to total cost, whereas the usual value function F refers only to present and future cost (in the case when cost can indeed be partitioned over time). G is automatically t-dependent, in that W1 takes values in different sets for different t. However, the simple specification of W1 as argument is enough to indicate this dependence. *Theorem 8.2.1 (The dynamic programming principle) The total value function G( W 1) obeys the backward recursion (the dynamic programming or optimality

equation) (t

= 1, 2, ... )h - 1)

(3)

with closing condition

(4) and the minimising value ofu 1 in (3) is the optimal value ofcontrol at timet.

We prove these assertions formally in Appendix 2. They may seem plausible in the light of the discussion of the previous section, but demonstration of even their formal truth requires a consideration of the structure implicit in a temporal optimisation problem. They are in fact rigorously true if all variables take values in finite sets and if the horizon is finite; the theorem is starred only because of possible technical complications in other cases. That some discussion is needed even of formal validity is clear from the facts that the conditioning variables W1 in (2) and ( W1, u1) in (3) are a mixture of random variables Y 1 which are genuinely conditioning and control histories U1_t or U1 which should be seen as parametrising. Further, it is implied that the expectations in (3) and (4) are policy-independent, the justification in the case of (4) being that all decisions lie in the past These points are covered in the discussion of Appendix 2. Relation (3) certainly provides a taut and general expression of the dynamic programming principle, couched, as it must be, in terms of the maximal current observable We.

3 STATE STRUCT URE

175

3 STATE STRUC TURE The two principa l properties which ensure state structur e are exactly the stochastic analogues of those assumed in Chapter 2 (i) Markov dynamics. It is required that the process variable x should have the property (5) where Xr. U1 are now complete histories. That is, if we conside r the distribu tion of Xr+l conditio nal on process history and paramet rised by control history then it is in fact only the values of process and control variables at time t which have any effect. This is the stochastic analogue of the simply-recursive deterministic plant equation (2.2), and specification of the right-hand member of (5) as a function of its three argumen ts amounts to specification of a stochastic plant equation. (ii) Decomposable cost function. It is required that the cost function should break into a sum of instanta neous and closing costs, of the form

c=

h-1

h-1

1=0

1=0

L c(xl, ul, t) + Ch(xh) = L

Cz

+ ch,

(6)

say. This is exactly the assumption (2.3) already made in the deterministic case. We recall the definition of sufficiency of a variable ~~ in Section 2.1 and the characte risation of x as a state variable if (x 1 , t) is sufficient. These definitio ns transfer to the stochastic case, and we shall see, by an argume nt parallel to that of the deterministic case, that assumptions (i) and (ii) do indeed imply that x is a state variable, if only it is observable. A model satisfYing these assumpt ions is often termed a Markov decision process, the point being that they defme a simply recursive structure. However, if one is to reap the maxima l benefit of this structur e then one must make an observational demand . (iii) Perfect state observation. It is required that the current value of state should be observable. That is, x 1 should be known at the time t when u is to be 1 determi ned, so that W1 =(X,, Ut-i)· As we have seen already in the deterministic case, assumpt ion (i) can in principl e be satisfied if there is a description x of the system which is detailed enough that it can be regarded as physically complete. Whethe r this detailed description is immediately observable is another matter, and one to which we return in Chapter s 12 and 15. We follow the pattern of Section 2.1. Define the future cost at time t h-1

C1 = Lcr+C h r=l and the value function

F( W1)

= inf EII"[C1 1Wz] 7r

(7)

l 176

I

STOCHASTIC DYNAMIC PROGRAMMING

so that G{ W1) = I:~~~ Cr + F ( W1). Then the following theorem spells out the sufficiency of et = (xt, t) under the assumptions above. Theorem 8.3.1 Assume conditions ( i)-( iii) above. Then (i) F(W1) is a function ofx1 and t alone. If we write it F(x1 , t) then it obeys the dynamic programming equation

(t

~h)

(8)

with terminal condition

(9) (ii) The minimising value ofu1 in (8) is the optimal value ofcontrol at timet, which is consequently also a function only ofx 1 and t. Proof The value of F(Wh) is Ch(xh), so the asserted reduction of Fis valid at time h. Assume it valid at time t + 1. The general dynamic programming equation (3) then reduces to F( W 1)

c(x1 , u 1, t) + E[F(xt+l, t + l)IX1, U1]} = inf{ u,

(10)

and the minimising value of u1 is optimal. But, by assumption (i), the right-hand member of (10) reduces to the right-hand member of (8). All assertions then D follow by induction. So, again one has the simplification that not all past information need be stored; it is sufficient for purposes of optimisation that one should know the current value of state. The optimal control rule derived by the minimisation in (8) is again in closed-loop form, since the policy before timet has not been specified. It is in the stochastic case that the necessity for closed-loop operation is especially clear, since continuing stochastic disturbance of the dynamics makes use of the most recent information imperative. At least in the time-homogeneous case it is convenient to write (8) simply as F(·,t)

= !l'F(·,t+ I)

(11)

where !l' is the operator with action !l'¢(x)

u) + E[¢(xt+l)lx1 = x, u = u]}. = inf{c(x, u 1

(12)

This is of course just the stochastic version of the forward operator already introduced in Section 3.1. As then, !l'¢(x1) is the minimal cost incurred if one is allowed to choose u1 optimally, in the knowledge that at time t + 1 one will incur a cost of¢( x 1+1). In the discounted case !l' would have the action !l'¢(x)

u) + ,8E[¢(xr+i)lx1 = x, Ur = u]}. = inf{c(x, u

(13)

4 THE EQUATION IN CONTINUOUS TIME

177

4 THE DYNAMIC PROGRAMMING EQUATION IN CONTINUOUS TIME It is convenient to note here the continuous-time analogue of the material of the last section and then to develop some continuous-time formalism in Chapter 9, before progressing to applications in Chapter 10. The analogues of assumptions (i)-(iii) of that section will be plain; we deduce the continuous-time analogues of the conclusions by a formal passage to the limit. It follows by the discrete-time argument that the value function, the infimum of expected remaining cost from timet conditional on previous process and control history, is a function F(x, t) of x(t) = x and t alone. The analogue of the dynamic programming equation (8) for passage from t to t + 8t is F(x, t) = inf{ c(x, u, t) u

+ E[F(x(t + 8t), t + 8t)lx(t) =

x, u(t) = u]}

+ o(5t). (14)

Defme now the infinitesimal generator A(uJ t) of the controlled process by A(u, t)¢(x)

= lim(8t)- 1{E[¢(x(t + 8t))lx(t) 8t!O

= x,u(t) = u]- ¢(x)}.

(15)

That is, there is an assumption that, at least for sufficiently regular ¢( x ), E[¢(x(t + 8t))lx(t) = x, u(t)

= u] = ¢(x) + [A(u, t)¢(x)]8t + o(8t).

The form of the term of order 8t defines the operator A; to write the coefficient of 8t as A(u, t)¢(x) emphasises that the distribution of x(t + 8t) is conditioned by the valuex of x(t) and parametrised by t and the value u of u(t).We shall consider the form of A in some particular cases in the next chapter. *Theorem 8.4.1 Assume the continuous-time analogues of conditions (i)-(iii) of the Section 3. Then x is a state variable and the value function F (x, t) obeys the dynamic programming equation

i~f [c(x, u, t) + BF~~, t) + A(u, t)F(x, t)]

= 0

(t
(16)

The minimising value ofu is the optimal value ofu( t).

Equation (16) follows formally from equation (14) in the limit of small 5t. We have starred the theorem because, to the technical complications which can arise when x and u may vary in infinite sets and when also h may later allowed to be infinite are added those which may appear in the passage to continuous time. However, any invalidity of (16) in particular cases is usually well signalled by one's realisation of some anomaly in the mathematical representation of the physical picture.

178

l

STOCHASTIC DYNAMIC PROGRAM MING

If costs are discounted at rate a then equation (16) becomes

it![c-aF +

0:: +AF] =0

(t
(17)

Here the arguments of the various functions have been left understood , as is often possible without ambiguity.

I

CHAP TER 9

Stochastic Dyna mics in Continuous

Time In general we assume the reader acquainted with basic probability theory. However, we shall devote a short chapter to continuous-time stochastic processes, since these show special features and are of special relevance in control contexts. Some of the material also prepares the way for that of Chapter 22. We assume a state-structured model with state variable x, but shall for simplicity consider only the time-homogeneous uncontrolled case. These limitations are easily removed later, by simply making the various rates which occur depend also upon t and u.

1 JUMP PROCESSES One extreme case is that in which the state space flt is discrete, so that changes in state must be discontinuous. For example, if x is the size of a queue then x can only take the values 0, 1, 2, ... and a transition (from x to x ± 1) takes place whenever a customer arrives or leaves. Similarly, if we consider a model of a population which is recognised as being made up of individuals, then populatio n size can only be integral. Notation is simplified if we assume that the values of state x are integral, taking values j = 0, 1, 2, ... , say. If one later wishes to attach other numerical values to the states then there is no trouble in doing so. Suppose now that

P[x(t + 6t) = kjx(t) = j] = Ajk8t + o(6t)

(k

i= j)

(I)

for small positive &. This is a regularity condition which turns out to be selfconsistent. The quantity >yk is termed the probability intensity of transition from j to k, or simply the transition intensity. The assumption itself implies that the transition has been a direct one: the probability of its having occurred by passage through some other state is of smaller order in 8t (see Exercise 1).

Theorem 9.1.1 The process with transition intensities >yk has infinitesimal generator A with action

A¢(j)

= _E>.ik[¢ (k)- ¢(j)]. k

(2)

180

1

STOCHASTIC DYNAMICS IN CONTINUOUS TIME

i

Proof It is a consequence of (1) that E[¢(x(t + &)) - ¢(x)lx(t) = j) =

L Ajk[¢(k)- ¢(j)]8t + o(6t) k#j

whence (2) follows, by the definition of A To include the case k summation plainly has no effect.

= j in the D

We can return to the controlled time-dependent case by making the transition intensity a function >..1k(u, t) of u and t, when the dynamic programming equation (8.16) becomes

i~f [c(j, u, t) +oF~, t) + ~ Ajk(u, t)[F(k, t)- F(j, t)]l

= 0.

(3)

Exercises and comments (1) It follows from (1) and the Markov character of the process that

P[x(t + 8t) = i,

x(t + 2 8t) = klx(t) = j] = AJiAik(8t) 2 + o[(8tf]

fori distinct from bothj and k. This at least makes it plausible that the probability of multiple transitions in an interval oflength 8t is o( 8t).

2 DETERMINISTIC AND PIECEWISE DETERMINISTIC PROCESSES The deterministic model for which x is a vector obeying the plant equation x = a(x) is indeed a special case of a stochastic model. The rate of change of ¢(x) in time is ¢xa, so that the infinitesimal generator A has the action

A¢(x)

= ¢x(x)a(x)

(4)

where ¢xis the row-vector of differentials of¢ with respect to the components of x. Consider a hybrid of this deterministic process and the jump process of the last section, in which the x-variable follows deterministic dynamics x = aJ(x) in the jth regime, but transition can take place from regime j to regime k with intensity >..1k(x). Such a process is termed a piecewise deterministic process. The study of such processes was initiated and developed by Davis (1984, 1986, 1993). For example, if we consider an animal population, then statistical variability can occur in the population for at least two reasons. One is the intrinsic variability due to the fact that the population consists of a finite number of individuals: demographic stochasticity. Another is that induced by variability of climate, weather etc.: environmental stochasticity. If the population is large then it is the second source which is dominant: the population will behave virtually deterministically under fixed environmental conditions. If we suppose, for

3 THE DERIVATE CHARACTERISTIC FUNCTION

181

simplicity, that the environmental states are discrete, with well-defined transition intensities, then the process is effectively piecewise deterministic. In such a case the state variable consists of the pair (j, x): the regime labelj and the plant variable x. We leave it to the reader to verify that the infinitesimal generator of the process has the action

+ LAik(x)[¢(k,x)- ¢(j,x)].

A¢(j,x) = ¢x(j,x)aj(x)

k

3 THE DERIVATE CHARACTERISTIC FUNCTION Recall that the moment-generating function (abbreviated MGF) of a random column vector xis defined as M(a) = E(e=), where the transform variable a is then a row vector. Some basic properties of MGFs are derived in Appendix 3. One would define M(iO) as the characteristic function of x; this always exists for real 0. The two definitions then differ only by a 90° rotation of the argument in the complex plane, and it is not uncommon to see the two terms loosely confused. · Suppose from now on that the state variable x is vector-valued. We can then define the function (5) of the column vector x and the row vector a It is known as the derivate characteristic function (abbreviated to DCF). We see, from the interpretation of A following its definition (8.15), that H has the corresponding interpretation

E(ea6xlx(t)

=

x)

=

1 + H(x, o:)8t + o(8t),

{6)

where 8x = x(t + 8t) - x(t) is the increment in x over the time interval (t, t + 8t]. The DCF thus determines the MGF of this increment for small 8t; to this fact plus the looseness of terminology mentioned above it owes its name. For example, consider a process for which x can jump to a value x + dj(x) with probability intensity Aj(x) (j = 1, 2, ... ) • For this the infinitesimal generator has the action A¢(x) = L Aj(x)[¢(x + tlj(x)) - ¢(x)]

(7)

j

and the DCF has the evaluation

H(x,a)

=

LAj(x)[eadj(x) -1].

(8)

j

Comparing these last two relations we see that we can make the assertion, at least for processes of this type:

182

STOCHASTIC DYNAMICS IN CONTINUOUS TIME

*Theorem 93.1 Relation (5) between the derivate characteristic function Hand the infinitesimal generator A has the inversion

A= H(x,alax)

(9)

if it is understood that, when we form A¢(x) = H(x, alax)¢(x), the differential operator a 1ax acts only on the x-argument of¢.

This indeed follows from comparison of (7), (8) and appeal to the 'Taylor' formula e(8f8x)d¢(x)

= ¢(x +d),

Here (8/ ax )d we mean the inner product L:k dk8 I Oxk' where dk and Xk are thekth components of the vectors d and x respectively. We shall return to the important identity (9) in Section 22.3. 4 PROCESSES OF INDEPENDEN T INCREMENTS , AND PROCESSES DRIVEN BY THEM A process {x(t)} is one of independent increments if the increments in x over disjoint time intervals are statistically independent. If, in addition, the distribution of the increment depends only upon the length of the interval then it is a homogeneous process of independent increments (abbreviated HPII). Such processes are important because their 'time derivative' (should it exist) provides the continuous-time equivalent of a sequence of liD random variables. The abbreviation HPII is not a standard one, but it will serve us for this and the next section. Let us define the MG F

M(a, t) =

E[e"[x(t)-x(O)l].

Then the HPII property is obviously equivalent to the property

M(a, *Theorem 94.1

t1

+ tz)

= M(a, tt)M(a, tz)

(10)

.lf{x( t)} is an HPJ! then there exists a function '1/J( a) such that M(a, t) = e1'¢(a)_

(11)

Furthermore

{12) for prescribedfunctions a( t)for which the right-hand member is defined.

4 PROCESSES OF INDEPENDENT INCREMENTS

183

*Proof Relation (11) is indeed a consequence of (10). It follows from (11) and the independence of increments that, if { t1} is an increasing sequence of timepoints, then E [exp{

~ aj[x(tJ)- x(tJ-d}] = exp{ ~(tj- tJ-d'!fJ(aJ) }·

Relation (12) then follows by a limiting argument.

0

*Theorem 9.4.2 The process {x( t)} is an HPII ifand only ifH(x, a) is independent ofx, so that A= '!fJ(8/8x), (13) say. One can then identify 1/J with the 1/J of(11). *Proof If the process is an HPII so that (11) holds then one indeed verifies that H (x, a) = 1/J( a), implying (13). To establish the reverse conclusion, define the conditional MGF M(a,x, t) = E[eru:(tllx(O) = xl This obeys the backward equation 8 01 M(a,x, t)

= 'l/J(8/8x)M(a,x, t)

which has a solution

M( a, x, t) = eru:+h/J(a).

(14)

But relation (14) is sufficient to establish both relation (10) and the representa~~

0

So, H (x, a) = 1/J( a) is both necessary and sufficient for the HPII property, with . the 1/J( a) of (11) identified with the DCF. It is natural to ask: what functions 1/J are possible? which are interesting? In fact, the HPII property implies that exp[.,P(a)] must be the MGF of an infmtely divisible distribution. The Uvy-Khintchine theorem then implies that an HPII process must be a superposition of independent Poisson and Wiener processes. It is a quick matter to describe these processes and verifY the HPII property for them. The simplest HPII process is the Poisson process. For this x takes only nonnegative integer values, the only transitions are of the form x --+ x + 1 and these all have the same intensity A. One could regard x(t)- x(O) as the number of events which have taken place in the time interval (0, t], if events take place independently with probability intensity A. These events might be insurance claims, traffic accidents or arrivals of cosmic particles in a chamber. The DCFofthe process is 1/J(a) = A(e0 -1). We see then from (11) that the distribution of the number of events in a time interval of length t is Poisson with expectation At, whence the name of the process. One speaks of the events occurring in a Poisson stream ofrate A.

184

STOCHASTIC DYNAMICS IN CONTINUOUS TIME

We can generalise the process by superimposing Poisson streams. Conside r the scalar process { w( t)} generated by w(t) = Laixi( t)

(15)

j

where the ai are constants and the {xj(t)} are independent Poisson processes of respective rates Aj. For example, the jth stream might be interpretable as a stream of particles each carrying charge ai; the variable w(t) is then the total charge which has accumulated by time t. Such a process is termed a compound Poisson process. It is plainly HPII with DCF 1/J(a)

=L

.Xj(eaaJ- 1).

(16)

j

One obtains an interesting class of models by drivfng a deterministic model by anHPII . Theorem 9.4.3 Consider the vector process {x(t)} generated by the different ial equation ~itten in incrementa/form) 8x = a(x) 8t + b(x) 8w

(17)

where { w( t)} is a vector-valued homogeneous process ofindependent increments with DCF'IjJ(a} Then {x(t)} isaMark ovproce sswithD CF H(x, a) = aa(x)

+ 1/J(ab{x)). •

(18)

The assertions follow immediately if one regards relation (6) as providing the effective definition of the DCFan d inserts expression (17) for 8x into it. Th.e 'plant equation' (17) is interesting as the continuous-time version of a process driven by 'noise' which is totally random, in the sense that its values at different times are statistically independent. One is tempted to rewrite it as

x = a(x) + b(x)e

(19)

where the 'noise' variable e is the time derivative of w. The process {w(t)} generated by (15) plainly does not have a derivative in the usual sense. If one admitte d the idea one would have to regard it as a random sequence of impulse s, those of magnitude ai occurring with intensity Aj, independently of other impulses. This is the 'shot noise' process formulated by physicists early in the century. The concept is proper in that integrals of shot noise are proper; one can write (12) as E[exp {j a(t)e(t) dt}] =exp{ j 1/J(a{t))dt}.

(20)

, .l

5 WIENER PROCESSES (BROWNIA N MOTION)

185

5 WIENER PROCESSES (BROWNIAN MOTION), WHITE NOISE AND DIFFUSION PROCESSES Ifwe consider a sequence of shot noise processes in which the impulses become ever weaker but ever more frequent then we reach the classic limit process associated with Brownian motion and white noise.

*Theorem 9.5.1 Consider the scalar HPII {w(t)} with DCF 'lj;(a.) = (eaa + e-aa)/2a2.

(21) The limit process for small a is then an HPII with DCFand infinitesimal generator

A= 'l/J(ofox) =

!(:x)

2

(22)

Further, since E[exp{/ a.(t)dw(t) }] =E[exp {/ a.(t)€(t)dt }] =exp{!/ a.(t) 2 dt}. (23)

increments ofthe process are jointly normally distributed. The assertions are formally immediate. They constitute in fact a process form of the central limit theorem, given operational meaning by the fact that the argument can be phrased in terms of the statistics of well-behaved functionals such as the integral Ja.(t) dw(t). We recognise in (21) a compound Poisson process consisting of two streams, each of rate 1j2a2 and carrying 'charges' of size a and opposite signs. If a is small then the charge carried is small, but particles arrive fast. The 'derivative' € of the limit process is an infinitely dense sequence of infinitesimal independe nt impulses, whose integral has zero expectation but is normally distributed. The limit process { w( t)} is a time-homogeneous process of independe nt increments which also has the property of being Gaussian. As a function of time it in fact continuous, and is the only continuous HPII. It is in all senses classical: sometimes referred to as Brownian motion and written B(t); sometimes referred to as the Wiener process and written W(t). Its formal differential f(t) is termed 'white noise' by engineers for reasons explained in Section 13.2 It is improper in having infinite variance, but proper in that integrals of white noise are proper, as we see in (23). The argument of Theorem 9.5.1 can be repeated in a vector version. We then arrive at the notion of a vector Wiener process w(t) and vector white noise f(t) with the property that the linear form Jo:(t) dw(t) = Ja.(t)€(t) dt is normally distributed with zero mean and covariance matrix

186

STOCHASTIC DYNAMICS IN CONTINUOUS TIME

cov(/ a(t)e(t) dt)

=

J

a(t)Na(t )T dt

(24)

for some non-negative definite matrix N, at least for functions a(t) such that this last integral is convergent. Relation (24) implies that

( 1

cov (6t)- 1

t+6t

1

•1 .I

)

e(r) dr = Nj6t,

which has an infinite limit as 6t tends to zero. This is an indication that e itselfhas an irregular character; its components have infinite variance. Such a conclusion is inevitable if one requires property (24) for non-zero N. The matrix N indeed gives the statistical scale of e but is not its covariance matrix. It is more appropriately to as the power matrix. As we have seen, one can derive white noise as the 'limit' of realistic physical processes. Mathematically, the concept is regularised in the so-called theory of generalised processes (see e.g. Gihman and Skorohod, 1972, 1979} White noise is to conventional processes what the Dirac 6-function is to conventional functions. Like the 6-function, it can be seen as a limit of conventional processes, and the operational characterisation in terms oflinear functionals above is analogous to the operational characterisation I a(t)6(t) dt = a(O) of the 6-function. Initial discussions ofthe a-function took the circumspect route of writing I a(t)6(t) dt as I a(t)d.H(t) where the Heaviside function H(t) is the formal integral of the 6-:-function; the function which is constant except for a single unit step at the origin. This is now seen as a crutch which can be kicked away, but the same caution leads to the writing of I a(t)e(t) dt as I a(t) dw(t) where {w(t)} is a Wiener process. The noise-driven processes (19) which are most considered, for various reasons, are those for which the driving noise e is white; such a process is termed a diffusion process for reasons which will soon become clear. If one takes a ·deterministic process as a first crude approximation to a given stochastic process, then there are good reasons for taking a diffusion process as the second; see Section 22.3. We can specialise Theorem 9.4.3 to the diffusion case.

J

'I I I

:I

I

J

Il 1

Theorem 95.2 Consider the vector process {x(t)} generated by the differential equation 6x = a(x)t + b(x) 6w

!

(25)

or

x=

a(x) + b(x)e

(26)

where e is a white noise process (wa Wiener process) ofpower matrix N Then {x( t)} is a Markov process with DCF

·I

whereN(x)

5 WIENER PROCESSES (BROWNIAN MOTION)

187

H(x, a) = aa(x) +! aN(x)aT

(27)

= b(x)Nb(x)T.

We see from (13) and (27) that the generator of the process has the action A¢ = ¢xa(x)

+! tr[N(x)¢xx]

(28)

where
+ F + Fxa(x, u, t) +! 1

tr(N(x, u, t)Fxx)]

= 0.

(29)

CHAPTERJO

Some Stochastic Examples 1 LQG REGULATION WITH PLANT NOISE Let us return to the LQ regulation example of Section 2.4 with the sole modification that the plant equation is changed to (1) where e1 is a random disturbance term. This term would often be referred to as plant noise-'plant' because it occurs in the plant equation and 'noise' because in audio-electronic contexts the disturbance manifests itself as a rush of background noise. Plant noise is indeed often to be regarded as random in that it is only partially predictable. The thermal noise of electronic circuits or the gusts to which an aircraft is subject provide examples of disturbances which are scarcely predictable at all. The mention of predictability is a reminder that the stochastic character of the noise must be specified. A common starting assumption is that the noise is as unpredictable as possible; that it is white in that the distribution of e1 conditional on process history before time tis independent ofboth process history and t. This implies that the process has the Markov property (i) of Section 8.3. It also implies that the variables e1 are independently and identically distributed (See Sections 9.5 and 13.2 for a discussion of the special character of white noise in continuous time and of the reason for the description 'white~ Process history before time t is specified by (Xr-1 , Ur-11 since this determines e,. for T < t. Actually, we do not need to demand complete 'whiteness', but simply that

(2) for a constant matrix N That is, that the conditional mean of e1 should be constant (and normalised to zero, by convention) and that the conditional covariance matrix of e1 should also be constant. A stronger assumption which is often made is that the noise should be Gaussian as well as white, i.e. that the noise variables should be jointly normally distributed. The resultant model is then termed an LQG model, indicating linear dynamics, quadratic costs and Gaussian noise. However, as far as discrete-time models are concerned, we need the Gaussian assumption first when we come to consider imperfect state observation in Chapter 12.

190

SOME STOCHASTIC EXAMPLES

Theorem 10.1.1 Assume the stochastic plant equation specified by (1), (2) and the quadratic cost structure specified in equations (2.21) and (2.23). Then the value function has the form F(x, t) = !xTII,x + 6,

(3)

and the optimal control the form

(4) Here the matrices II1 and K 1 have the same evaluations as in Theorem 2.4.2 and 61 = Dt+t

+! tr(NII,+l)

(5)

with terminal condition Dh = 0. That is, the matrix II1 is the same solution of the Riccati equation (2.25) as before, and K1 is determined from it by the same relation (2.28) as before. Indeed, the control rule (4) is exactly what it was in the noise-free case, and the only effect of the noise is to add a term 61 to the cost. The interpretation is that the quadratic term in (3) represents the cost caused by the initial deviation of state x from zero and the term 61 represents the cost caused by the continued disturbance of state by noise. In the infinite-horizon limit (if one exists) the presence of noise induces a cost of! tr( NII) per unit time.

Proof The course of the proof is inductive, as ever. Relation (3) holds at time

h. Assume then that it holds at time t + 1, so that

F(W1)

= inf{c(x, u) + !E[(Ax + Bu + et+t)TIIt+t(Ax + Bu + t:r+t)IX1,Ur]} "

where x, u and e are the values holding at time t. Assumptions (2) imply that the conditional expectation in this relation reduces to

!(Ax+ Bu) TIIt+l (Ax+ Bu) whence all assertions follow.

+! tr(NIIr+l) 0

The admission of plant noise with the 'second-order white' properties (2) thus affects the optimal control rule not at all (at least, if this is expressed in closed-loop form) and increases costs only by a term independent of state or policy. It may seem strange that, for our flrst stochastic example, the optimal control was unaffected (at least in its closed-loop form) by the presence of the stochastic element. This is a consequence of the very special assumptions. The essence of optimal control generally is that one exploits the statistical characteristics of

1 LQG REGULATION WITH PLANT NOISE

191

noise variables to the full. For example, suppose that the plant equation (1) is modified to = Axr-! +Bur-!

+ v1

(6) where v 1 is a stochastic noise signal of some more general form. One will often suppose that it can be generated as the output v1 = G(ff)Er of a linear filter driven by white noise, so that one still has in effect a system driven by white noise, but more complex than (1). This can be coped with in various ways, but the most insightful is that which follows from the certainty equivalence principle of Chapter 12. This principle implies that, under some further assumptions on the linear/Gaussian nature of observations, the optimal control at time t for system (6) would be the same as that for the deterministic system X1

(r > t),

(7)

where v~) is an appropriate linear predictor of v 7 based on information available at timet. That is, at timet one replaces future stochastic noise v7 (r > t) by an 'equivalent' deterministic disturbance v~r) and then applies the methods of Sections 2.9 or 6.3 to deduce the optimal feedback/feedforward control in terms of this predicted disturbance. We shall see in Chapter 12 that similar considerations hold if the state vector xis itself not perfectly observable. It turns out that E~) = 0 (1 > t) for a white-noise input E which has been perfectly observed up to time t. This explains why the closed-loop control rule was unaffected in case (1). Once we drop LQG assumptions then treatment of the stochastic case becomes much more difficult. For general non-linear cases there is not a great deal that can be said. We shall see in Section 7 and in Chapter 24 that one can treat some models for which LQG assumptions hold before termination, but for which rather general termination conditions and costs may be assumed. Some other models which we treat in this chapter are those concerned with the timing of a single definite action, or with the determination of a threshold for action. For systems of a realistic degree of complexity the natural appeal is often to asymptotic considerations: e.g. the 'heavy traffic' approximations for queueing systems or the large-deviation treatment oflarge-scale systems.

Exercises and comments (1) Consider the closed- and open-loop forms of optimal control (2.32) and (2.33) deduced for the simple LQ problem considered there. Show that if the plant equation is driven by white noise of variance N then the additional cost incurred from time t = 0 is D QN E;,;;-ci (Q + sD) -I or hDN according as the closed- or the open-loop rule is used. These then grow as log h or as h with increasing horizon.

192

SOME STOCHASTIC EXAMPLES

2 OPTIM AL EXERCISE OF A STOCK OPTIO N

As a last discrete-time example we shall consider a simple but typical financia l optimisation problem. One has an option, although not an obligation, to buy a share at price p. The option must be exercised by day h. If the option is exercised on day t then one can sell immediately at the current price x 1, realising a profit of x 1 - p. The price sequence obeys the equation Xt+l = x 1 + E1 where the € 1 are independently and identically distributed random variables for which EJEJ < oo. The aim is to exercise the option optimally. The state variable at time t is, strictly speaking, x 1 plus a variable which indicates whether the option has been exercised or not. However, it is only the latter case which is of interest, sox is the effective state variable. If F (x) is the 3 value function (maximal expected profit) with times to go then Fo(x) = max{x - p, 0} = (x- p)+ and

Fs(x) = max{x - p,E[Fs-! (x +E)]}

(s = 1,2, ... ).

The general character of Fs(x) is indicated in Figure 1; one can establish the following properties inductively: (i) Fs(x)- xisnon- increas inginx; (ii) Fs(x) is increasing in x; (iii) Fs(x) is continuous in x; (iv) Fs(x) is non-decreasing ins. For example, (iv) is obvious, since an increase in s amounts to a relaxation of the time constraint. However, for a formal proof: F, (x) = max{x - p, E[Fo(x +E)]};?: max{x - p, 0} = Fo(x),

Figure 1 The value function at horizon sfor the stock option example.

3 A QUEUEING MODE

193

whence Fs is nondecreasing ins, by Theorem 3.1.1. Correspondingly, an inductive proof of (i) follows from

Fs(x)- x =max{ -p, E[Fs-1 (x +e)- (x +e)]+ E(e)}. We then derive

Theorem 10.2.1 There exists a non- decreasing sequence {as} such that an optimal policy is to exercise the option first when x ~ as. where x is the current price and s is the number ofdays to go before expiry ofthe option. Proof From (i) and the fact that Fs(x) ~ x- pit follows that there exists an as such that Fs(x) is greater than x- p if x < as and equals x- p if x ~ as.It follows from (iv) that as is non-decreasing in s. 0 The constant as is then just the supremum ofvalues of x for which Fs(x)

>

x-p.

3 A QUEUEING MODEL Queues and systems of queues provide a rich source of optimisation models in continuous time and with discrete state variable. One must not think simply of the single queue ('line') at a ticket counter; computer and communication systems are examples of queueing models which constitute a fundamental type of stochastic system of great technological importance. However, consideration of queues which feed into each other opens too big a subject; we shall just cover a few of the simplest ideas for single and parallel queues in this section and the next chapter. Consider the case of a so-called M/M/1 queue, with x representing the size of the queue and the control variable u being regarded as something like service effort. If we say that customers arrive at rate A and are served at rate p,(u) then this is a loose way of stating that the transition x -+ x + 1 has intensity Aand the transition x-+ x- 1 has intensity p,(u) if x > 0 (and, of course, intensity zero if x = 0). We assume the process time-homogeneous, so that the dynamic programming equation takes the form

h![c(x,u)+ BF~:,t) +A(u)F(x,t)] =0.

(8)

Here the infinitesimal generator has the action (cf. (9.2))

A(u).p(x) = A[.P(x + 1) - .P(x)] + p,(u, x) [.P(x- 1) - .P(x)] where p,(x, u) equals p,(u) or zero according as xis positive or zero. If we were interested in average-optimisation over an infinite horizon then equation (8) would be replaced by

194

SOME STOCHASTIC EXAMPL ES

1 = inf(c(x, u) u

+ A(ulf(x)]

(9)

where .\ and f(x ) are respectively the average cost and the transien t cost associated with the average-optim al policy. In fact, we shall con cer n ourselves more with the question of optimal allocation of effort or of customers between several queues tha n with optimi sation of a single queue. In preparation for this , it is useful to solve (9) for the unc ontrolled case, when JL(u) reduces to a con stant f.L and c(x, u) to c(x). In fact , we shall assume the instantaneous cost pro portional to the num ber in the que ue, so that c(x) = ax. We leave it to the reader to verify tha t the solution of equation (9) is, in this reduced case,

a>.

!=--,, f.J, -A

f(x ) =

a f.L- .\

x(x + 1) 2

(10)

We have assumed the normalisat ion f(O) = 0. The solution is, of course, valid only if>. < f.L, for it is only then tha t queue size is finite in equilibrium .

4 THE HARVESTING EXAMPLE : A BIRTH-DEATH MODEL

Recall the deterministic harvesting model of Section 1.2, which we sha ll generally associate with fisheries, for definite ness. This had a scalar state variabl e x, the 'biomass' or stock level, which foll owed the equation

.X= a(x )- u.

(11)

Here 1J is the harvesting rate, which (it is supposed) may be varied as des ired. The rate of retu rn is also supposed pro portional to u, and nor mal ised to be equ al to it. (The model thus neglects two ver y imp orta nt elements: the age stru ctur e of the stock and the x-dependence of the cost ofharvesting at rate uJ We suppose again tha t the functio n a(x), the net reproduction rate of the unharvested population, has the form illustrated in Figure 1.1; see also Figure 2. We again denote by Xm and .xo the values at which a(x) is respectively maximal ·an d zero. An unharvested pop ula tion would thus reach an equilibrium at x = xo. We know from the discussion of Section 2.7 tha t the optimal policy has the threshold form: u is zero for x ~ c and takes its maximal value (M, say) for x > c. Here c is the threshold, and one seeks now to determine its optima l value. If a(x) > 0 for x ~ c and a(x )- M < 0 for x > c then the harvested pop ulation has the equilibrium value c and yields a return at rate 1 =a ( c). If we do not discount, and so choose a thresho ld value which maximises this ave rage retu rn A, then the optimal threshold is the valu e Xm which maximises a( c). A threshold policy will still be optimal for a stochastic model und er corresponding assumptions on birt h and death rates. However, there is an effect which at first sight seems remark able. If extinction of the pop ula tion is impossible then one will again choose a threshold value which maximises average

4 THE HARVESTING EXAMPLE: A BIRTH-DEATH MODEL

., I

I

I I I

195

return, and we shall see that, under a variety ofassumptions, this optimal threshold indeed approaches Xm as the stochastic model approaches determinism. However, if extinction is certain under harvesting then a natural criterion (in the absence of discounting) is to maximise the expected total return before extinction. It then turns out that the optimal threshold approaches x0 rather than Xm as the model approache s determinis m. There thus seems to be a radical discontinuity in optimal. policy between the situations in which the time to extinction is finite or infinite (with probability one, in both cases). We explain the apparent inconsistency in Section 6, and are again led to a more informed choice of criterion. We shall consider three distinct stochastic versions of the model; to follow them through at least provides exercise in the various types of continuou s-time dynamics described in the last chapter. The first is a birth-deat h process. Letjbe the actual number offish; we shall set x = j JK where K is a scaling parameter, reflecting the fact that quite usual levels of stock x correspon d to large values of j. (We are forced to assume biomass proportion al to population size, since we have not allowed for an age structure). We shall suppose thatj follows a continuous-time Markov process on the nonnegative integers with possible transitions j --t j + I and j --t j - 1 at respective probability intensities >.i and f..ti· These intensities thus correspond to populatio n birth and death rates. The net reproduction rate >.i - f..ti could be written ai, and correspon ds to Ka(x). Necessarily f..to = 0, but we shall suppose initially that >.o > 0. That is, that a zero population is replenished (by a trickle of immigration, say), so that extinction is impossible. Let 7rj denote the equilibrium distribution of population size; the probability that the population has size j in the steady state. Then the relation 7rjAj = 1ri+1f..ti+ 1 (expressing the balance of probability flux between states j and j + 1 in equilibrium) implies that 7rj ex: Pi> where

Pi =

>.o>.1 ... >-i-1 f..tlf..t2 ••• /-tj

(

j = O, 1, 2, . . .),

(12)

(with Po = 1, which is consistent with the convention that an empty product should be assigned the value unity). A threshold c for the x-process implies a threshold d ~ Kc for the j-process. For simplicity we shall suppose that the harvesting rate M is infinite, alth()ugh the case of a finite rate can be treated almost as easily. Any excess of population over d is then immediately removed and one effectively has Ad = 0 and Pi = 0 for j >d. The average return (i.e. expected rate of return in the steady state) on thexscale is then

(13)

196

SOME STOCHASTIC EXAMPLES

the term 1rd>.d representing the expected rate at which excess of population over c is produced and immediately harvested. Suppose now that the ratio 91 = J.i.JIAJ is effectively constant (and less th~ unity) for j in the neighbourhood of d. The effect of this is that 'lrd-J ~ 7rd()~ (j ~d), so the probability that the population is an amo untj below threshold falls away exponentially fast with increasingj. Form ula (13) then becomes 1 ~ ~~;- 1 ~(1- 9d)

= ~~;- 1 (>.d- JJ.d) = K- 1ad =a( c).

The optimal rule unde r these circumstances is then indeed to choose the threshold c as the level at which the net reproducti on rate is maximal, namely, Xm. This argument can be made precise if we pay attention to the scaling. The nature of the scaling leads one to suppose that the birth and death rates are of the forms >.1 = ~~;)..(jj~~;), J.£J = KJJ.(jf~~;) in terms of functions A(x) and p,(x), corresponding to the deterministic equation x = >.(x) - J.£(x) = a(x) in the limit of large l'i.. The implication is then that A]/ p, varies slowl y withj if~~; is large, with the 1 consequence that the equilibrium distributio n of x falls away virtually exponentially as x decreases from d = cf~~;. The details of this verbal argument are easily completed (although anomalous behaviour atj = 0 can invalidate it in an interesting fashion, as we shall see). The theory of large deviations (Chapter 22) deals precisely with such scaled processes, in the range for which the scale is large but not yet so large that the process has collapsed to determinism. Virtually any physical system whose complexity grows at a smaller rate than its size generates examples of such processes. Suppose now that >.o = 0, so that extinction is possi ble (and indeed certain if, as we shall suppose, passage to 0 is possible from all states and the population is harvested above some ftnite threshold,) Expressio n (12) then yields simply a distribution concentrated onj = 0. Let Fj be the expected total retur n before extinction conditional on an initial population ofj. (It is unde rstoo d that the policy is that ofharvesting at an infinite rate above the prescribed threshold value ~ The dynamic programming equation is then

(O< j
(15)

5 BEHAVIOUR FOR NEAR-DETERMINISTIC MODELS

197

where

(16) Proof We can write (14) as >..p!:l.j+l = p,16.1 where 6.1 = Fj- Ff-t· Using this equation to determine 6.1 in terms of .6.d+1 = I and then summing to determine Fj, we obtain the solution (15). D We see from (15) that the d-dependence of the Fj occurs only through the common factor lid, and the optimal threshold will maximise this. The maximising value will be that at which Ad/ JLd decreases from a value above unity to one below, so that ad = Ad - JLd decreases through zero. That is, the optimal value of c is xo, the equilibrium level of the unharvested population. More exactly, it is less than x 0 by an amount not exceeding K:- 1. This means in fact a very low rate of catch, even while the population is viable. The two cases thus lead to radically different recommendations: that the threshold should be set near to Xm or virtually at x 0 respectively. We shall explain the apparent conflict in the next two sections. It turns out that the issue is not really one of whether extinction is possible or not, but of two criteria which differ fundamentally and are both extreme in their way. A better understanding of the issues reveals the continuum between the two policies. Exercises and comments (1) Consider the naive policy first envisaged in Chapter 1, in which the population was harvested at a flat rate u for all positive x. Suppose that this translates into an extra mortality rate of v per fish in the stochastic model. The equilibrium distribution 1T"J of population size is then again given by expression (12), once normalised, but with JLJ now modified to JLJ + v. The roots Xt and x2 of a(x) = u, indicated in Figure 1.2 and corresponding to unstable and stable equilibria of the deterministic process, now correspond to a local minimum and a local maximum of 1T"j. It is on this local maximum that faith is placed, in that the probability mass is supposed to be concentrated there. However, as v (and so u) increases this local maximum becomes ever feebler, and vanishes altogether when u reaches the critical value Urn = a(xm)· 5 BEHAVIOUR FOR NEAR-DETERMINISTIC MODELS In order to explain the apparent stark contradiction between the two policies derived in Section 4 we need to obtain a feeling for orders of magnitude of the various quantities occurring as K: becomes large, and the process approaches determinism. We shall follow through the analysis just for the birth-death model

198

SOME STOCHASTIC EXAMPLES

of the last section, but it holds equally for the alternative models of Sections 8 and 9. Indeed, all three cases provide exam ples of the large deviation theo ry of Chapter22. Consider then the birth--death mod el in the case when extinction is possible. Since mos t of the time before extinction will be spent near the thre shol d value if r;, is large (an assertion which we shall shor tly justify) we shall cons ider only Ftt. the expected yield before extinction cond ition al on an initial valu ed ofj. Let 'Fj denote the expected time befo re extinction which is spen t in state j (conditional on a star t from d ). Then, by the same meth ods which led to the evaluation (15), we find that

'Fj = rrd

[t k=!

1-"11-"2 .. ·1-"k-1] )qA2 ... Ak-1

[~-"J+IPJ+2 . . ·1-"d]. .\.tAJ+l ... Ad

which is consistent with expression (15) for Fd = AdFd. Whe n we see the process in term s of the scaled variable x Fj = "'F( x) and Tj = T(x). Ifwe defi ne

R(x)

=

1x

log[A(y)l p(y)J dy

(17)

= j I"' we shall write (18)

then we deduce from expression (15) that

F(c) =

ex:R(c)+o(~<)

(19)

for large "'·We thus see that F( c) grows expo nentially fast with"'· Inde ed, the same holds true for the occupation times: T(x) = e~
Tj

j J.LlJ.L2 ••. J.Lk-1 ] = (PJIAo) [ 2.:: AA A ~ Sp11Ao, k=l I 2 • · ·

k-1

say, where

s=

t

k=l

[J.L!J.L2 ... J.Lk-1] A2A2 ... Ak-1

(21)

(22)

In the next section we shall inte rpre t S I Ao as the expected time taken for a popu latio n which has been extinguished to be reseeded and grow to viability. Since the term s in the sum S decline exponentially fast with increasing k, the relative erro r in approximation (21) is expo nentially small.

6 THE UNDERSTANDING OF THRESHOLD CHOICE

199

The scaling argument will again imply that ()j = J.l.j / Aj varies only slowly if"" is large, with the implication from (21) that Td-j "'

Td()~

for fixedj; an analogue of the corresponding assertion for 'Trj in the last section. If we define Text = 1 1), the expected time to extinction, then

I:,f=

(23)

so that 1 - ()d is a measure of the proportion of time before extinction which is actually spent at threshold. Moreover, since Lj ,;; Kx 1j /Text decreases to zero with increasing"" we can just as well interpret Text as the expected time needed to escape permanently from any neighbourhood of d. In other words, the time to extinction is asymptotic to the time at which harvesting becomes commercially nonviable. Suppose that rJ.!I is the actual total yield (on the x-scale) before extinction and ff the actual time to extinction, so that these are random variables with respective expectations F(c) and Text (conditional on a start from x =c). Then a closer analysis shows that in fact rJ.!I j F( c) and ff /Text converge to unity in almost any stochastic sense as K---+ oo. The point is that x recurs to c a great many times before it ultimately drops to zero, and the contributions to rJ.!I or ff from each of these excursions away from c and back are independent and identically distributed random variables. The consequence is that the expectation of the average rate of return before extinction, rJ.!I / ff, can be replaced by the ratio of expectations:

E(OJI/ff)

~

E(OJI)/E(ff) = F(c)/Text,

the relation becoming exact in the limit oflarge K. Relation (23) then implies the evaluation

F( c)/Text =

I'C- 1

Fct/Text ~

I'C- 1

Ad(l - Bd)

= K:- 1(>.d- J.Ld) = >.(c) - p,(c) =a( c) (24)

for the expected average return over the period before extinction. This demonstrates that the expected rate of return averaged over the time before extinction converges to the equilibrium rate of return for the deterministic process asK -+ oo. 6 THE UNDERSTANDING OF THRESHOLD CHOICE

Expressions (16) and (24) are again maximal for c near to xo and Xm respectively. We see that the discrepancy is not a consequence of differing assumptions, because these two evaluations have been made for the same process; one for which ultimate extinction is certain. It is a consequence of differing criteria. To

200

SOM E STOC HAST IC EXAM PLES

ask for max imal total retur n and to ask for maxi mal average retur n over the time to extinction (of stock or, almo st equivalent ly, of com merc ial viability) are two very different things. They differ so because the expo nenti al depe nden ce of Text upon "' mean s that it varies extremely rapid ly with c, wher eas the average retur n (24) varies only moderately. The maxi misa tion of expe cted total retur n then amou nts virtu ally to the maxi misa tion of expe cted survival time, with the rate of retur n playing a role which actually becomes ever less signi fican t as "' increases. Inde ed, the reco mme ndat ion is virtually that one shou ld not harvest. In fact, both crite ria are extreme, one takin g the yield rate before extinction and the othe r the time to extin ction as virtually the sole cons idera tion. A balan ced crite rion would be one which chose the thres hold c to maxi mise yield rate 1 =a( c) subje ct to a presc ribed lowe r boun d on Text· Since this last expression depe nds exponentially upon "' (which is what indu ces the sensitivity of its depe nden ce upon c) one migh t cons ider rathe r the norm alise d expression

L(c) = lim "'- 1 log /t-+00

Text·

This in fact has the evaluation L(c) = { R(c)

R(xo)

(c ~ xo)

(c

~

xo)

(25)

where R is inde ed the func tion defin ed in (18). The two quantities a( c) and L(c) vary joint ly with thres hold cas indic ated in Figure 2; a( c) increases with c up to the value Xm and decli nes there after ; LV:) increases with c up to the value x and 0 is cons tant thereafter. Thus, if one prescribes the value of Text as at least Tmin, then L( c) must be at least K- 1 log Tmin. For smal l enou gh "' one will thus have to take x 0 as thres hold , but as "' increases the reco mme nded thres hold will decre ase quite quickly until it reaches Xm.

Figure 2 The characterisers ofperformance for a stochastic fishing model operating at threshold c in the asymptotic (near-deterministic) case. The graphs ofa(c) the average yieldrate, and of L(c) the normalised logarithm ofexpect ed extinction time, as functions ofc.

7 CONTINUOUS-TIME LQG PROCESSES

201

Interesting~ one can reach this same view by returning to the first stochastic model of Section 4: the birth-death model without extinction. It follows from (21) that we can write expression (13) for the yield rate in the steady state as

A(c)Td "{ ~ (S/'Ao) +Text~ a(c)[l

+ S/'AoText)],

(26)

where the terms neglected are o( 1) in K.. Here Text is now to be interpreted as the time to first extinction for this immortal population. Now, Ao is the rate at which a population which has been reduced to zero is restarted by some external mechanism, so that A(} 1 is the expected 'seeding' time. This seeding does not amount to an effective restart until numbers have been brought up to viability; we can regard (S/'Ao) as the expected time needed for this to occur: the expected restart time. Let us then denote it by Tres, so that expression (26) for 7 becomes

I

-r

I

7 ~ a(c)[l + (Tres/Text)r 1. This is then approximate ly a(c) or a(c)(TextfTres) according as to whether the restart time is small or large relative to the extinction time. Since Tres is independent of c, these two extreme evaluations are equivalent to those obtained for the two extreme criteria in the case when extinction was possible. As one varies the value assumed for Tres one moves monotonical ly between these two extremes, just as one did by varying the prescribed value Tmin in the constrained optimisation ofyield rate. Ifwe consider discounted criteria then the possibility of extinction is seemingly irrelevant, because any accounting horizon is orders of magnitude smaller than a prudent lower bound for the extinction time-the conservation horizon. As above, then, the constraint of such a lower bound must be imposed explicitly. That is, suppose that a is the discount rate and that F(c, a) is the discounted future yield at an operating level (and threshold) of c, calculated on the basis of a deterministic model. Then one should choose c to maximise F(c, a), subject to prescription of a lower bound Tmin on Text· As in the undiscounte d case above, the effect of this constraint will weaken rapidly as the scale parameter Kincreases, and the model approaches determinism .

7 CONTINUO US-TIME LQG PROCESSE S AND PASSAGE TO A STOPPING SET One of the most tractable controlled diffusion processes is, as one might expect, the continuous-t ime version of the LQG regulation problem treated in Section 1. We assume the quadratic cost structure of Section 2.8 but modifY the plant equation to

x=Ax+Bu +e

(27)

202

SOME STOCHASTIC EXAMPLES

where E is white noise of power matrix N. The dynamic progr ammi ng equation is then inf[c(x, u) + F1 + Fx(Ax + Bu) +! tr(NFxx)] = 0 u

(t
(28)

with termin al condition F(x, h) = !xTll( h)x. One readily verifies the solutions

F(x, t) = !xTll (t)x + 8(t),

u(t) = K(t)x (t) (29) for value function and optim al control, where TI and K are exactly the matrices of the deterministic treatm ent of Section 2.8. The term 8( t), representing the future cost attributable to process noise, has the evaluation

8(t) =!

Jh tr[NTI(r)] dr

(30)

in analogue to the discrete-time relation (5). For a modification from the pure LQG form, consider a stochastic version of the first-passage problem of Exercise 2.6.2, in which one attempted to reach one end or the other of the interval [0, 1] with the optimal compromise between economy of time and of control effort. As first-passage proble ms go this is about the simplest, but it raises a numb er of interesting points. We asume the mode l modif ied only in that the plant equation becomes .X = u + E, where E is scalar white noise of power N. One's intentions are then to some extent frustrated by rando m disturbances to the path. The dynamic progr ammi ng equation (28) now becomes inf[! (L + Qtl) + uFx +! NFxx] = 0 u

(x:f:0 ,1).

(31)

This holds up to termin ation, which occurs on passage into either of the values x = 0 or x = 1, with respective termin al costs F(O) =Co,

F(l) =

c1.

We can actually perfo rm the minimisation in (31) to deduc U=

-Q- 1Fx

and the reduced version

(32) e (33)

(34) of the dynamic progr ammi ng equation. The non-linear equati on (34) can in fact be linearised by the transformation

1/J(x) to the form

= e-F(x)fQN

r

7 CONTINUOUS-TIME LQG PROCESSES

203

This has the general solution

'1/J(x)

= c1e"'x + cze-=

(O~x~l)

(35)

where a = J Lj QN2 and the coefficients c; are determined from the boundary conditions (32). In contrast to the piecewise-linear expression ofF (x) for the deterministic case derived in Exercise 2.6.2, F(x) now has a single analytic expression over the whole interval 0 ~ x ~ 1. Mathematically, the fact that F now obeys a secondorder equation means that both boundary conditions can be met by a single analytic expression. Physically, the fact that noise can carry one off the intended course means that one might end up at either boundary point, and induces a blurring of costs which makes F smooth. The evaluation ofF is now both greater and smoother than in the deterministic case, as indicated in Figure 3. In case (a) there was formerly (i.e. in the deterministic case) a change in goal at an intermediate value of x and there still is, at the point at which F is maximal. However, Fis now smoothly differentiable at this break-point, because noise can carry one across it in either direction. In case (b) Fx increases as one approaches the undesirable termination point x = 1, reflecting one's increasing efforts to avoid it. Avoidance is never certain,

(a) 0

X

(b)

0

X

Figure 3 The stochastic version ofFig. 22 The upper curves give the value function F(x)for the stochastic version ofthe optimalfirst-passage problem of Ex. 26.2. The two cases correspond, as previously. to those in which the optimal deterministic policy has a break-point in the interior ofthe continuation region or on its boundary.

204

SOME STOCHASTIC EXAMPLES

however, as reflected in the fact that F is now continuous at this boundary. This might be termed the 'fly-paper' effect, if one regards the moving point as a fly trying to avoid the fate of becomi ng stuck on fly-paper at x = 1. A deterministic fly, whose path is fully under its own control, can actually approach arbitrari ly close to the fly-paper with impunity, knowing that he can avoid entrapment. Hence the discontinuity ofF at x = 1, reflecting the difference between being close and being stuck. A stochastic fly cannot guarantee his excape; the nearer he is to the paper, the more certain it is that he will be carried on to it Hence the continuity ofF at x = 1 in this case. This also explains why the fly tries so much harder in the stochastic case than in the deterministic case to escape the neighbourhood of the fly-paper (as manifested by the greater magnitude in the stochastic case of Fx, and so of -u for x near 1). One may say that the penalty of ending on the fly-paper 'propagates' into the free-flight region in the stochast ic case, causing the fly to take avoiding action while still at a distance from the paper. A rigorous treatme nt of the dynamic programming equation in cases where the postulated differential Fx does not exist everywhere can be based on the concept of a 'viscosity colution' (see Fleming and Soner, 1992). In at least some cases this can be envisaged as obtaine d by the addition of a little stochasticity to a previously deterministic problem. As above, this has the effect of smoothing the solution, and the limit of this stochastic solution as one approaches the determi nistic case is in fact the correct deterministic solution. This approach must be regarded as justificatory rather than constructive, however; we shall see in the large deviation treatment of Chapter 22 that the boot is often on the other foot, and that the stochastic solution is, in a sense, approximated by a deterministic solution . To return to the stochastic fly: it is only the difference in penalties at the two termination points which matters, so we may as well set Co = 0. Suppose we now let C1 tend to +oo, so that absorpt ion at x = 1 carries infinite penalty and is to be avoided at all costs. One would imagine that the controlled process would then become improper in some sense, because there is a contingency, presumably of positive probability, which carries infinite penalty. This is not the case, however . The limit process is perfectly proper with a control rule which, although also perfectly proper, becomes vigorous enough as x = 1 is approached that absorption at this point in fact has zero probability. (Stronger than that: the expectation of cost due to a contingency of infinite cost but zero probability could be anything. In fact, it is zero in this case.) In this limit we must have '!f;(O) = 1, '1/J( 1) = 0, and the solution (35) which satisfies these bounda ry conditions is

e

-F(x)/QN

a{l-x)

= '!f;(x) = e

- e

a{x-1)

ea -e-a

.

The optimal control rule is ea(l-x)

u = -Q-1 Fx = N'I/Jx/'1/J = -VLJQ ea(l-x)

+ ea{x-1) - ea{x-1).

(36)

7 CONTINUOUS-TIME LQG PROCESSES

205

- yfiJQ for x For N small (and so a large) u has approximately the constant value u l - oo, then 1 j x as er, Howev 1. from between zero and a value bound ed away effort is this That I. = x to e passag indicating a strenuous effort to avoid < 1. x for finite is (31), by given as ), x ( F successful is indicated by the fact that we which ses proces lled contro of class a This example is a particular case of shall investigate in Chapter 24. for a genera l There is anothe r point which comes out of this example. Suppose, t)Fxx] in the u, , tr[N(x term final controlled diffusion, that F(x, t) is such that the it can be that so u, and x of ndent dynamic programming equation (28) is indepe differ will t) F(x, n functio value written as a function p(t) of time alone. Then the : oftime n functio a from that for the deterministic case Fctet(x, t) only by

F(x, t) = Fctet(X, t) +

1

p(T) dT,

case. This was and the optimal control will be the same as for the deterministic ourhoo d neighb the in true y imatel true for the LQG example; it would be approx it to be expect may One s. proces of a stable equilibrium of the deterministic large. is Fxx where points at failure approximately true if N is small, but to show al optim The above. le examp ssage This is exactly what we observe in the first-pa inistic determ the for that from policy in the stochastic case deviates seriously corres pond case exactly at those points where Fxx is large for some reason, which These were case. inistic to those points at which Fx failed to exist in the determ forced. or chosen points at which there was a discontinuity in policy, either um principle' These matters are related to the search for a 'stochastic maxim which we shall join in Chapter 23. Exercises and comments

be of the form (1) Note that the solution of (34) for x ~ 0 or x ~ I will equally well ary condit ion at (35), but with the c-coefficients now determined by the bound it. one stopping point and a growth condition at infinity. Determ ine n will satisfY (2) Optimal stopping and tangency conditions. The value functio ter in some charac s change s matching conditions at a bound ary where the proces n of the positio the if sense, and will in addition satisfY optimality conditions forms the of sion bound ary itself can be optimised. Even a non-rigorous discus upon s depend such conditions can take can be quite intricate, as a great deal (see etc. e, possibl continuity of the path, the directions in which movement is Whittle 1983a, pp.l02-108 and 201-207). that of optimal However, an impor tant case which appears in various guises is of continuation or stopping, where the optimiser can choose between the actions The example of termination, so that the stopping set itself can be optimised. which the value Section 2 was one such. Consider a discrete-time formulation in

SOME STOCHASTIC EXAMPLES

206

II< in the stopping function F obeys F = LF in the continuation region C and F = suppose, for shall region£'). The state-variable argum ent xis understood, and we uation/ contin the simplicity, that there are no actions to be optimised other than t forvarian stopping decision. We shall suppose an infinite-horizon time-in mulation. The optimality equation is then F = min(L F, II<), with F

= LF < II<

{~);

on holds. Here ~ where the brackets indicate the set of x within which the asserti by choosing the $ and £') have been chosen optimally and we have closed g set g*; that stoppin stopping option in cases of indifference. Define the accessible of~- The part part of g which can be reached by a single transition from some continued the also essential matching condit ion is then that F = II< in £')*. Define with Fas cost F*; the solution ofF* = LF* (holding everywhere) which agrees then, have s). We much as it can: on~ U £')* (which is the only set of x which matter from the relations above and the definition ofF*, F*
(~);

F* = II<

($*);

LF*

~

LF

(q;*).

cy condition, These relations imply optimality conditions which imply a tangen can go to a we if that F* - II< should have zero derivative at the stopping point, continuous limit in space and time. as Fj etc., and Suppose for example that x takes integer values j; let us write F (x) r) constitute a suppose that LFj = c1 + p1Fj_ 1 + q1Fj + r1Fj+ 1 where (p, q, could thus yield distribution. A continuous-limit version of this assumption the value j = k either a deterministic process or a diffusion process. Suppose that in£'). We suppose marks an optima l stopping boundary, in that k - 1 is in ~and k possible. Then the that r > 0, so that movement in the direction of increasing} is yields ively, respect k and k 1, k at d last set of relations above, asserte which will imply That is, F* - IK has a turnin g point at the stopping boundary, . version imit uous-l contin a zero derivative at this point in cy condition at Note that we have already demonstrated validity of the tangen of Section 2.7. model ting harves inistic the optimal threshold value for the determ L 8 THE HARV ESTIN G EXAM PLE: A DIFFU SION MODE to the stochastic Suppose the deterministic harvesting equation (11) modified differential equation

x = a(x)- u+ E

(37)

so that the model where E is white noise of power v( x) / K. We introduce the factor K, e large. The can be made to approach determinism by allowing K, to becom

207

8 THE HARVESTING EXAMPLE: A DIFFUSION MODEL

as we shall introdu ction of,.. in this way amoun ts to a scaling assump tion which, of the see in Chapte r 22, fits into a general pattern consist ent with the scaling birth-d eath process in Section 4. O) has an If the unharv ested version of the model (i.e. that for which u equilib rium distribu tion then this has probability density

=

(38) where

R(x) = 2

jx

a(y)v(y )- 1 dy.

(39)

is by some (see Exercise 1). We assume that x is kept to the positive half-ax it is best then c > x for ing harvest allow we If 0. 1 x conditi on such as v(x) 1 0 as so that say, M, to equal and finite is rate ing to assume initially that the harvest deduce thus We >c. x a(x) is modifi ed to a(x)- Mfor

Theorem 10.8.1 The average rate ofreturn for the diffusion model (3 7) is _ 'Y-

M

.fc 7r(x)e-~~;MQ(x) dx

00 fo 7r(x) dx + fc 7r(x)e-"MQ(x) dx

c

(40)

where 71" has the evaluation (38) and

Q(x)

=

21x

v(y)- 1 dy.

(41)

Evaluation (10) becomes 'Y =

7r( c)v( c) c 2 fo 7r(x) dx

(42)

in the limit oflarge M. M. These Evalua tion (40) is direct and (42) the limit version of it for large one would expressions must be maxim ised with respect to c. Let us confirm what -+ oo and the expect: that they approa ch the known determ inistic value a (c) as ,.. x < c then model approa ches determ inism. If we assume R(x) increasing in x for (42) ion we find the evaluat ion of express 'Y

rv

!v(c)R '(c) =a( c).

See Exercise 2 for the case of a finite harvest rate. x = 0 is Suppos e however that extinct ion is possible, in that the state value 4. Section of that to similar s absorbi ng for the process (37). We follow an analysi initial an on onal conditi ion If F(x) is the total expecte d return before extinct value x then this obeys the dynam ic progra mming equatio n

SOME STOCHASTIC EXAMPLES

208

(43)

u +(a - u)F' + (2K-)- 1 vF" = 0.

where the harvest rate u takes the values 0 or m according as

x

~

cor x > c.

harvested process. Then Theorem 10.8.2 Suppose that extinction is certain for the the relevant solution of (43) is F(x)

= e~R(c)

(1x

where F' (c) has the evaluation

F'(c) =2M

e-l
dy )F'(c )

(0

~ x ~c),

leo e~fR(y)-R(c)-MQ(y)lv(yr 1

dy.

and R(x), Q(x) are the functions defined in (39) and (41). F'(c) = 1.

(44)

(45)

If M is infinite then

on of(43) for x ~ c. Proof We obtain solution (44) as we obtained (15); by soluti c and appeal to the > x for (43) of tion (45) then follows by solution The evalua this evaluation. 0 condition of certain extinction. The last assertion follows from s. The analogue of the final conclusion of Section 4 then follow ising the expression Theorem 10.8.3 The optimal value of c is that maxim value ofc is then determined by e~
y the value x 0 of That is, if M is infinite, then optimal threshold value is exactl different optimality Figure 2, whatever K-. The phenomenon of the two evaluation (25) still recommendations is explained exactly as in Section 6, and (39). holds, with R now having the evaluation Exercises and comments

of the section must (1) The equilibrium distribution 1r defined at the beginning1 (v7r)" = 0. This insatisfy the Kolmogorov forward equation -(a1r)' + (2K-)the net probability tegrates once, and the integral must be zero since it represents F(O) = 0. flux. The resulting equation has (44) as the solution satisfying ) decreasing for (2) Suppose that R(x) is increasing for x < c and R(x) - MQ(x x > c. Show then that expression (40) has the evaluation Mj(M fZ -R') 1 ~ (1/ R') + 1/(M (l- R')

for large K, where all functions are evaluated at c.

=a,

ENT 9 A MODEL WITH A SWITCHING ENVIRONM

WIT 9 THE HARVESTING EXAMPLE: A MODEL T ENVIRONMEN

209

H A SWI TCH ING

wise-deterministic mod el to M indicated in Section 9.2, we can use a piece Suppose that the mod el has represent the effects of environmental variation. = 1, 2, . . . . In regime i the several environmental regimes, labelled by i , but transition can take place population grows deterministically at net rate ai(x) to regime h with probability intensity Kllih· a different nature to that of This is then a model whose stochasticity is of quite 8. It comes from with out and 4 the birth -deat h or diffusion models of Sections ronmental stochasticity', 'envi rather than within, and represents what is term ed affects conclusions has this ther as distinct from 'demographic stochasticity'. Whe to be deter mine d by equation (11) but with The equivalent deterministic model would be given a(x)

= :~::)iai(x)

(46)

i

m is in regime i. The model where Pi is the steady-state probability that the syste s between regimes take place converges to this deterministic version if transition ge regime'. This occurs in the 'avera so rapidly that one is essentially working in an scaling parameter. al limit oflarge, x:, so that K again appears as the natur for such a multi-regime al A fixed threshold would certainly not be optim hold nature, but with thres of a model. It is likely that the optimal policy would be ion whether the quest a it is a different threshold in each regime. Of course, n of the rate of rvatio . Obse regime is know n at the time decisions must be made e is currently regim mine which change ofx should in principle enable one to deter x itself is of n ver, observatio in force, and so what threshold to apply. Howe . If one error ht with extreme unreliable, and estimation of its rate of change fraug optimal policy would base allowed for such imperfect observation then the bution conditional on curre nt action on a poste rior distribution (i.e. a distri ter 15). An optim al policy observables) of the values of both x and i (see Chap t should be applied only if an would probably take the form that harvesting effor hold dependent on both the estimate of the current value of x exceeded a thres probabilities of the different . precision of this estimate and the current poste rior regimes. desperately crude thoug h it We shall consider only the fixed-threshold policy, threshold value compares with must be in this case, and shall see how the optim al that for the equivalent deterministic mod el able to analysis. A value of x We shall consider a two-regime case, which is amen ot have positive probability in at which a1(x) and a2(x) have the same sign cann ~ 0 and a2(x) = -JJ.(x) :E;; 0 equilibrium. Let us suppose then that a1 (x) = ..\(x) est We shall set 1112 = lit and over an interval which includes all x-values of inter 1121

= 112·

SOME STOCHASTIC EXAMPLES

210

Suppose initially that extinction is impossible, so that the aim is to maximise the expected rate of return 'Y in the steady state. We shall suppose that the maxima l harvest rate M is infinite. For the deterministic equivalent of the process we have, by (46),

a(x)

= z-'2..\(x) .

vw.(x).

(47)

v1 + v2

We shall suppose that this has the character previously assumed, see Figure 2. We also suppose that p,(x) = 0 for x :E;; 0, so that xis indeed confine d to x ;;;::: 0. The question of extinction or non-extinction is more subtle for this model Suppose, for example, that ..\(0) = 0 (so that a population cannot be replenished) and that p,(x) is bounde d away from zero for positive x. Then extinction would be 2 certain, because there is a non-zero probability that the unfavourable regime to on extincti For zero. to down can be held long enough that the populat ion is run be impossible in an isolated populat ion one requires that p,(x) tends to zero sufficiently fast as x decreases to zero; the exact condition will emerge. Let Pi(x) denote the probability/probability-density of the ilx pair in equilibrium. These obey the Kolmogorov forward equations

(0

:E;;

x
If The second equation continues to hold at x = c, but the first does not. is c over excess all since , increase to (i, x) = (1, c) then x cannot continue 2; to 1 from i of n transitio by only left immediately cropped. The state (1, c) can be a discrete event which occurs with intensity 1w1• The effect of this is that an atom of probability forms at (1, c). If this has magnitude q in equilibrium then the balance of probability flux into and out of the state implies the equation

(x =c). · Theorem 10.9.1 PI (x)

(49)

(i) The equilibrium distribution of(i, x) is given by

= k..\(x)- 1 e~(x), P2(x) = kp,(xr 1 e~(x), q = k(~vl)- 1 e~~:R(c)

{50)

where k is a normalising constant and R(x)

= fox[v2p.{yr 1 -

v1..\{yr 1J dy.

(51)

(ii) The average return under the c-thresholdpolicy is

..\(c)

'Y - ---:::::: ----:--- :,.-,---- --:...:-, -----~~~~ exp[~R(x) - ~R(c)]g(x) dx + 1

-

whereg(x)

fo

= ..\(xr 1 + p.(xr 1.

(52)

,

__

'

9 A MODEL WITH A SWITCHING ENVIRONMENT

211

Proof The balance of probability flux at x in equilibrium implies that (53)

(O~x
Using this equation to eliminate p 2 from the first equation of (48) we deduce an equation .Av2 ) tWt Pt + -:\+T --;- PI= 0.

, (X

This leads to the solution for p 1 asserted in (50). The solutions for pz and q then follow from (53) and (49). Assertion (ii) follows from (50) and the relation 0 'Y = qa(c). The condition of non-extinction is just that the distribution (50), when a normalised, should not have all its probability mass at the origin. This sets 1 1 0. x! as grows bound on the rate at which v2 f.L(xr - v1>.(x)-

Theorem10.9.2 ln the deterministic limit expression (57) reduces to 'Y =a( c), with a(x) specified by (52} Proof We may suppose that a(x) ~ Oin the interval [0, c), so that R is increasing in the interval. In the limit oflarge "'expression (52) then becomes

Y"'

,\ (vt/R!)(>.-1 + f.£-1)

+1 0

with the argument c understood throughout.

The optimal value of c thus converges to Xm in the deterministic limit, as one would expect. Suppose now that extinction is possible, in that >.(0) = 0 and that the population can be run down to zero in a bounde d time in the unfavourable regime 2. It will then in fact be certain under communication/boundedness conditions. Let F;(x) denote the expected total return before absorption conditional on a start from {i, x ). We have then the dynamic programming equations

(O<x< c). Since escape from x = 0 by any means is impossible we have F1 (0) = F2(0) However, the real assertion is that

Ft(O+) = ¢,

(54)

= 0. (55)

. where F;(O+) = limxto F;(x) and¢ is an as yet undetermined positive quantity 1 regime in grow to time has it The point is that, if x is small and positive, then and time to decline to zero in regime 2 (before there is a change in regime). The

212

SOME STOCHASTIC EXAMPLES

second equation of (54) continues to hold at x substitute

= c, but for the first we must (56)

Relation (56) follows from the fact that escape from (1, c) is possible only the transition of i from 1 to 2; this takes an expected time (Kv1)- 1 during which return is being built up at rate .A( c). Theoreml0.9.3

The valuefunctions Fi(x) have the evaluations

F1 (x)

=

¢ + (/3/112)

F2(x) = ({3/vi) where¢=

1x

1x

.A(y)-!e-"R(y) dy,

JL(y)-!e-KR(y) dy

(0

~ x < c),

(57)

/3/r;, /3 =

v} 1 .A(c)e~
and R( x) is the function (51) so normalised that R(0)

(58)

= 0.

Proof We deduce from equations (54) that

(59) Using this relation to eliminate F2 from the first equation of (54) we obtain an equation for F: similar to that for PI(x) above, with solution Ff = (r/v2)>..- 1e"R for some constant {3, with the consequence that F~ = ({3/vi)JL- 1e"R. Integrating with end conditions (55) we thus deduce equation (57). Substituting these expressions for the Fi in relation (56) and in the second equation of (54) at x = c we deduce the determinations of¢ and /3 asserted. D We see from solution (57) that optimisation with respect to c amounts to maximisation of {3, and so to maximisation of >..(c)e"R(c). For K large this amounts to the maximisation of R(c), i.e. to the equation a( c)= 0, with a~) having the determination (47). That is, the optimal threshold again approaches thevaluexo. To be exact, the stationarity condition with respect to c is

Ifwe assume that .A(k) is increasing with x then we see that, at least for sufficiently large r;, the optimal threshold c lies somewhat above the value x 0 . For the two previous models it lay below. It is in this that the nature of the stochasticity (environmental rather than demographic) reveals itself. In the previous examples

9 A MODEL WITH A SWITCHI NG ENVIRON MENT

213

there would virtually never have been any harvesting if c had been set above the equilibrium value xo. The effect in this case is that x can indeed rise sufficiently above x 0 during the favourable regime 1, and one waits for this to happen before harvesting. Notes on the literature

The fact that the threshold c should seemingly be set at the unharves ted equilibrium level x 0 if one sought to maximise the expected total return before a point of certain extinction was first observed by Lande, Engen and Saether (1994, 1995), for the case of a diffusion process. The analysis of Section 8 expands this treatment. The material of Sections 4-6 and 9 appears in Whittle and Horwood (1995).

_,.

CHAPTER 11

Policy Improvement: Stochastic Versions and Examples CONCLUSIONS 1 THE TRANSFER OF DETERMINISTIC ite-horizon behaviour and the In Chap ter 3 we considered patte rns of infin ministic case. All conclusions technique of policy improvement for the deter stochastic case if we appropriately reached there transfer as they stand to the L and 2 and their continuousextend the definitions of the forward operators time analogues. state-structured time-homogeneAs in Chap ter 3, attention is restricted to the al expectation operator ous case. In discrete time we define the condition

E(u)cp(x) = E[cp(xt+l)!xt = x, Ut = u].

(1)

policy g(oo) and an optimal policy The forward operators L(g) and .ff' for the respectively are then defined by (2) L(g)cp(x) = c(x,g(x)) + {JE(g(x))cp(x), (x, u) !l'cjJ(x) = inf[c u

+ {JE(u)cp(x)].

(3)

equations for the value functions The corresponding dynamic prog ramm ing s Vs = Vsg((oo)) and Fs then again take the form

Vs = L(g) Vs-1,

Fs = .PFs-1

(s > 0),

(4)

d. with the x-argument of these functions understoo it stands with these extended as valid now is 3 The material of Chap ter the monotonicity of the forward definitions. Explicitly: Theo rem 3.1.1, asserting value functions unde r appr opria te operators and the monotonicity (in s) of the the instantaneous cost function terminal conditions, still holds. If we assume on total cost still hold, as does c(x, u) non-negative then Theorems 3.2.1-3.2.3 Theorem 3.5.1 on policy improvement. (1) is the infinitesimal generator The continuous-time analogue of the oper ator . In terms of this the stochastic A(u) for the controlled process, defined in (8.15) M(g) and~ of Section 3.1 take the versions of the differential forward operators forms

216

POLICY IMPROVEMENT. STOCHASTIC VERSIONS

M(g)<jJ(x) = c(x,g(x ))- a<jJ(x) + A(g(x))
"

The assertions of Chapter 3 for the continuous time case then also transfer bodily.

Exercises and comments (1) We can supplement the example of instability given in Exercise 3.2.1 by the classic stochastic example of the simplest gambling problem. Suppose a gambler has a capital x which takes integral values, positive as well as negative, and has the choice of ceasing play with reward x, or of placing a unit stake and continuing. In this second case he doubles his stake or loses it, each with probability 1/2. If his aim is to maximise expected reward then the dynamic programming equation is

Gs(x) = max{x, ![Gs-1( x- 1) + Gs-1 (x + 1)]}

(s > 0)

where G3 (x) is his maximal expected reward if he has capital x with splays remaining. If at s = 0 he only has the option of retiring then Go(x) = x, and so Gs(x) = x for all s. However, the infinite-horizon version of this equation also has a solution G(x) = +oo. If the retirement reward xis replaced by min (a, x) for integral a then the equation has a solution G(x) =a for x::;:;; a. This corresponds to the policy in which the player continues until he has a capital of a (an event which ultimately occurs with probability one for any prescribed a) and then retires. The solution G(x) = +oo corresponds to an indefinitely large choice ofa. InVestigate how infmite-horizon conclusions are modified if any of the following concessions t6 reality is admitted: (i) debt is forbidden, so that termination is enforced in state 0; (ii) rewards are discounted; (iii) a constan t . positive transaction cost is levied at each play. (2) An int~resting example in positive programming is that of blackmail. Suppose there are two states: those in which the blackmailer's victim is compliant or resistan t Suppose that, if the blackmailer makes a demand of u (0 ::;:;; u ::;:;; 1), then a compliant victim pays it, but becomes resistant with probability zil. A resistant victim pays nothing and stays resistant. If Gs is the maxima l expected amount· the blackmailer can extract from a compliant victim in s further demands, then Go = 0 and

Gs+l = sup[u + (1- u2)Ga] = '1/J(Gs),

" say. Here the optimising value of u is the smaller of l and (2Gs) -I and '1/J( G) is 1 or G + 1/ (4G) according as G is less than or greater than!· Show that Gs grows as s112 and the optimal demand decreases as s- 112 for large s. There is thus no meaningful infmite-horizon limit, either in total reward or in

2 AVERAGE-COST OPTIMALITY

217

optimal policy. The blackmailer becomes ever more careful as his horizon increases, but the limiting policy u = 0 is of course not optimal. (3) Consider the discounted version of the problem, for which the infinite- horizon 1 reward G obeys G = 7/J(f3G). Show that, if! ~ f3 < 1, then G = (2yf/3( 1- {3))- . 2 AVERAGE-COST OPTIM ALITY

The problem of average-cost optimisation is one for which the stochastic model in fact shows significant additional features. Because of its importa nce in applications, it is also one for which we would wish to strengthen the discussion of Chapter 3. We observed in the deterministic contexts of Section 2.9 and 3.3 that one could relatively easily determine the value of control at an optimal equilibrium point, but that the determination of a control which stabilised that point (or, more ambitiously, optimised passage to it) was a distinct and more difficult matter. The r stochastic case is less degenerate, in that this distinction is then blurred. Conside of class a is there that Suppose y. the case of a discrete state space, for simplicit states 9l for which recurrence is certain under a class of control policies which includes the optimal policy. Then all these states will have positive probability in equilibrium (under any policy of the class) and, in minimising the average cost, one also optimises the infinite-horizon control rule at every state value in 91. Otherwise expressed: since equilibrium behaviour still implies continuing variation (within Bl) in the stochastic case, optimisation of average cost also implies optimisation against transient disturbance (within i?ll). These ideas allow us to give the equilibrium dynamic programming equations (3.9) and (3.10) an interpretation and a derivation independent of the sometimes troublesome infinite-horizon limit. Consider the cost recursion for the policy g(oo):

'Y + v = L(g)v,

(5)

where 'Y is the average cost under the policy and v(x) the transient cost from x, suitably normalised. (These are both g-dependent, but we take this as understood, for notational simplicity.) Suppose that the state space is discrete and all states are recurrent under the policy. Then 'Y can be regarded as an average cost over a recurrence cycle to any prescribed state (see Exercise 2). Equation (5) can be very easily derived in this approach, which completely avoids any mention of infinite horizon limits, although it does imply indefinite continuation. The natural normalisation of v is to require that E[v(x)] should be zero, where the expectation is that induced by policy in equilibrium. This is equivalent to requiring that the total transient cost over a recurrence cycle should have zero expectation. If there are several recurrence classes under the policy then there will be a separate equation (5) for each recurrence class. These recurrence classes are

218

POLICY IMPROVEMENT: STOCHASTIC VERSIONS

analogous to the stable equilibrium points of the deterministic case (although see Exercise 1). In the optimised case the equation

1+f=! £'f

(6)

has a similar interpretation, the equation holding for all x in a given recurrence class under the optimal policy, and 1 and f being the minima l average cost and transient cost in this class. Whether this equation is either necessary or sufficient for optimality depends again upon uniqueness questions: on whether the value function could be affected by a notional closing cost, even in the infinite-horizon limit. The supposition of a non-negative cost does indeed imply thatfma y be assumed bounde d below, and so that the relevan tfsoluti on of (6) is the minimal solution exceeding an arbitrary specified bound. However, one must frequently appeal to arguments more specific to the problem if one is to resolve the matter.

Theorem 11.2.1 Suppose that (6) can be strengthened to

1+ f

= 2f = L(g)f.

Then

(7)

(8) where dn = n- 1(J(xo) - E[f(xn) l Wo]), for any policy 1l; with equality if1r = gC 00 l. If dn ---+ 0 with increasing n for any 1r and for any Wo one can then assert that the policy gC 00 l is average-cost optimal. Proof Relation (7) is a strengthening in that the second equality asserts that the infimum with respect to u implied in the evaluation of .Ief(x) is attained by the choice u = g(x). If we denote the value of c(x, u) at timet by c1 then (7) can be written "t+f(xt ) ~E?r[ct+f(xt+l)IWt] for any policy 11; with equality if 1r = gCool. Taking expectations on both sides conditional on Wo and summing over t from 0 to n - 1 we deduce that

n7+ f(xo) 'i:

E,(~c, +f(x"))Wo}

whence the assertions from (8) onwards follow.

0

219

2 AVERAGE-COST OPTIM ALITY

assume the same off, If cis uniformly bound ed in both directions then we may l argum ents are specia cases other and it is then clear that dn has a zero limit. In required to establish the result. Exercises and comments ous to the doma ins of (1) One might think ofthe recurr ence classes as being analog that occup ation notion attraction of the deterministic case, but this is not so. The case this inistic determ of the states should actually recur is impor tant: in the e, the ermor or. Furth would true only of the points of the so-called attract of the many e with transient states in the stochastic formulation may communicat recurrence classes. are not conce med for A different set of ideas altogether (and one with which we that it appro aches a the moment) is encountered if a process is 'scaled' so case states which are in deterministic limit as a param eter ,., is increased. In this feebly as ,., is increa sed, the same recurrence class may comm unica te ever more inistic limit. until they belong to distinct doma ins of attrac tion in the determ v control policy is (2) A controlled Markov process with a statio nary Marko on. Suppose the statesimply a Markov process with an associated cost functi x and thatp( x, y) is the space discrete, that c(x) is the instan taneou s cost in state define a modif ied cost transition probability from state x to state y. Suppose we of modified costs over a function c(x)- "'/and define v(x) as the expected sum ence) to a specified path which starts at state x and ends at first entry (or recurr state, say state 0. Then

v(x)

= c(x) -7 + _Ep(x,y)v(y). y~O

of modified costs has Suppose we choose 1' so that v(O) = 0, i.e. so that the sum on becom es simply zero expectation over a recurr ence cycle. Then this last equati

(9)

v(x) = c(x)- 'Y + _Ep(x,y)v(y). y

which we can identifY with (5). We have 0 = v{O) sum is over a recurrence cycle to state 0. That is,

= E[2: (c(xt )- "'/)],where the 1

e cost over a recurr ence where Tis the recurr ence time; this exhibits~, as the averag = l:x 11'(x)c(x), where 1' that cycle. On the other hand, we deduce from (9) d interp retatio n is secon The {11'(X)} is the equilibrium distribution over states. ence, with no recurr of terms the 'infinite horizo n' one; the first is purely in explicit appea l to limiting operations.

220

POLICY IMPROVEMENT: STOCHAS TIC VERSIONS

(3) The blackmail example of Exercise 1.2 showed infinite total reward, but is not rephrasable as a regular average-reward problem. The maximal expected reward over horizons grows as s112 rather than s, so the average reward is zero and the transient reward infinite. Indeed, an average reward of 1 and a transient reward ofj(x) are associated with a development

F(x)

=

1 2/3 + f(x)

+ o(l- !3)

of the discounted value function for 1 - /3 small and positive. This contrasts with the ( 1 - {3) -I 12 behaviour observed in Exercise 1.2. 3 POLICY IMPROVEMENT FOR THE AVERAGE-COST CASE The average-cost criterion is the natural one in many control contexts, as we have emphasised. It is then desirable that we should obtain an average-cost analogue of Theorem 3.5.1, establishing the efficacy of policy improvement. We assume a policy g~ 001 at stage i. Let us distinguish the corresponding evaluations of {, v(x), L and expectation E by a subscript i. The policyimprovement step determines g;+I by 2v; = L;+I v;. Let us write the consequent equation

(10) in the form

(11) where 8 is some non-negative constant. If 8 can in fact be chosen positive then one has an improvement in a rather strong sense, as we shall see.

Theorem11.11 Relation (11) has the implication

(12) where

dn = n- 1(v;(xo)- Ei+I[vi(xn)lxo]).

If dn --t 0 with increasing n it then follows that: (i) there is a strict improvement in average cost if 8 is positive; (iz) average-optimality has been attained if equality holds in (1 1). The proof follows the same course as that of Theorem 11.2.1. Note that, effectively, the only policies considered are stationar y Markov. However, we expect the optimal policy to lie in this class.

221

4 MACHINE MAIN TENA NCE

In continuous time the optimality equation (7) is replaced by 1 1 = inf[e(x, u) + A(u)f(x)].

= .Af, i.e. (13)

u

4 MACHINE MAINTENANCE service effort over a As an example, consider the optimisation of the allocation of lated in continuous set of n machines. The model will be a very simple one, formu use, passes through time. Consider first a model for a single machine which, with x to x + 1 has states of increasing wear x = 0, 1, 2, . . . . The passage from while in state x, this intensity.>.. for all x, and the machine incurs a cost at a rate ex s the machine representing the effect of wear on operation. A service restore instantaneously to state 0. intensity p,. The Suppose the machine is serviced randomly, with probability be will policy blind dynamic programming equation under this (14) 1 =ex+ .>..[!(x + 1)- f(x)] + p,[f(x) -/(0) ] policy and ft?c) the if costs are undiscounted. Here 1 is the average cost under the transient cost in state x. The equation has the unique solution

= .Xe/Jl. if we make the normalising assumptionf(O) = 0. f(x) = ex/ J.i.,

1

(15)

parameters and Suppose now that we have n such machines and distinguish the whole system the state of the ith machine by a subscript i. The average cost for will then be (16) .X;c;/ Jb; li = I =

L i

L i

with the Jb; constrained by (17) if J.l is the intensity oftotal service effort available. a machine is Specification of the Jli amounts to a random policy in which, when probability Jbd Jb· A to be serviced, the maintenance man selects machine i with stage of policy One {x;}. = x state more intelligent policy will react to system ering policy consid before ver, Howe .~~·""!!J=~~"~improvement will achieve this. ing the Jli to choos by policy m rando improvement, one could simply optimise the that this finds readily One (17). minimise expression (16) subject to constraint leads to an allocation Jli ex ~machine i for Policy improvement will recommend one to next service the which

222

POLICY IMPROVEMENT: STO CHASTIC VERSIONS

:L{CjXJ + >.,[.tj(Xj+!)- jj(x})]} + tt(Ji(O)- f(x;)j j

is minimal; i.e. for which Ji(xi) = c;xd /ti is greatest. If we use the optimi sed value of f.L; derived above the n the recommendation is tha t the next machine i to be serviced is tha t for which x 1 jCJ>:; is greatest. . Note tha t this is an index rule, in tha t an index x 1 ~ is calculat ed for each machine and tha t machine chosen for service whose cur ren t index is greatest. The rule seems sensible: degree of wear, cost of wear and rapidity of wea r are all factors which would cause one to direct attention towards a give n mac hine. However, the rule is counter-intuiti ve in one respect: the index dec reases with increasing >.1, so tha t an increased rate of wear would seem to make the machine need service less urgently. Howeve r, the rate is already taken accoun t of in the state variable x 1 itself, which one wou ld expect to be of ord er At if a give n time has elapsed since the last service. The deflation of x1 by a factor A is a reflection of the fact that one expects a quickly wearing component to be mo re worn, even und er an optimal policy. An alternative arg um ent in Section 14.6 will demonstrate tha t this pol icy is indeed close to optimal. 5 CU STO ME R ALLOCATION BE TW EE N QU EU ES Suppose there are n queues of the type considered in Section 10.3. Quantities defined for the ith queue will be given subscript i, so tha t x 1, A;, f.Li and a1x 1 represent size, arrival rate, service rate and instantaneous cost rate for tha t queue. We suppose initially tha t these que ues operate independently, and use

(18) to denote the total arrival rate of cus tomers into the system. However, suppose that arriving cus tomers can in fact be routed into any of the queues (so tha t the queues are mu tually substitutable alternatives rath er tha n components of a stru ctu red netw ork). We look for a routing policy 1r which minimises the expected average cos t "/1r = E1r[L: 1 a1x 1]. The policy imp lied by the specification above simply sends an ving customer to queue iwi th pro bability >..d >..;the optimal policy will presumarri ably react to the current system stat e { x 1}. The ran dom routing policy achieve s an average cost of

(19)

6 ALLOCATION OF SERVER EFFORT BETWEEN QUEUES

223

(see (10.10)). As in the last section, before applying policy improvement we might ·as well optimise the random policy by choosing the allocation rates At to minimise expression (19) subject to (18), for given A. One readily fmds that the optimal choice is (20) where () is a Lagrange multiplier whose value is chosen to secure equality in (18). So, speed of service and cheapness of occupation both make a queue more attractive; a queue for which a;J JLi > 1j Owill not be used at all. Consider one stage of policy improvement. ·It follows from the form of the dynamic programming equation that, if the current system state is {xi}, then one will send the next arrival to that queue i for which

is minimal. Here the fi, are the transient costs under the random allocation policy, determined in (10.10). That is, one sends the arrival to that queue i for which

fi(xi + 1)- fi(xi) = at(Xt +A1) J.'i-

i

is minimal. If the At have already been optimised by the rule (20) then it follows that one sends the new arrival to the queue for which ((x 1+ 1) J a;J JJ.t) is minimal, although with i restricted to the set of values for which expression (20) is positive. This rule seems sensible: one tends to direct customers to queues which are small, cheap and fast-serving. Note that the rule again has the index form. 6 ALLOCATION OF SERVER EFFORT BETWEEN QUEUES We could have considered the problem of the last section under the assumption that it is server effort rather than customer arrivals which can be switched between queues. The problem could be rephrased in the form in which it often occlirs: that there is a single queue to which customers of different types (indexed by z) arrive, and such that any customer can be chosen from the queue for the next service. Customers of different types may take different times to serve, so it is as well to make a distinction between service effort and service rate. We shall suppose that if one puts service effort u1 into serving customers of type i (i.e. the equivalent of u1 servers working at a standard rate) then the intensity of service completion is the service rate O"tJ.I.i· One may suppose that a customer of type i has an exponentially distributed 'service requirement' of expectation JJ.i 1, and that this is worked off at rate ai if service effort ui is applied.

224

POLICY IMPROVEMENT: STOCHASTIC VERSIONS

As in the last section, we can begin by optimising the fixed service-allocation policy, which one will do by minimising the expression

with respect to the service efforts subject to the constraint

L:u =S,

(22)

1

i

on total service effort. The optimal allocation is

u1 = p.j 1 [A; + ylOa;A;J.L;]

(23) where ()is a Lagrange multipler, chosen to achieve equality in (22). An application of policy improvement leads us, by an argum ent analogous to that of the previous section, to the conclusion that all servic e effort should be allocated to that non-empty queue i for which p.1[.fi(x 1) - Ji(x1)] is minimal; 1 i.e. for which

v1(x1) = p.1[.fi(x1) - Ji(x; - 1)] =

a·J.L·X· 1 ' 'A

O';Jl.;-

i

(24)

is maximal. If the fixed rates p.1 have been given their optim al values (23) then the rule is: all service effort should be concentrated on that non-empty customer class i for which x1J a1p.t/ A; is maximal. It is reasonable that one should direct effort towards queue s whose size x 1 or unit cost a; is large, or for which the response p.; to servic e effort is good. However, again it seems paradoxical that a large arrival rate A; should work in the opposite direction. The explanation is analogous to that of the previous section: this arrival rate is already taken account of in the queue size x1 itself, and the deflation of x1 by a factor JX1 is a reflection of the fact that one expects a busy queue to be larger, even under an optimal policy. Of course, the notion that service effort can be switch ed wholly and instantaneously is unrealistic, and a policy that took accou nt of switching costs could not be a pure index policy. Suppose that to switch an amou nt of service effort u from one queue to anoth er costs c!u!. Suppose that one application of policy improvement to a policy of fixed allocation {u;} of service effort will modifY this to {~}.Then the~ will minimise

E!!c;l~- u;!- ~v;) i

(25)

subject to

(26)

7 REWARDED SERVICE RATHER THAN PENALISED WAITING

225

and non-negativity. Here Vt = v;(x;) is the index defined in (24) and the factor! occurs because the first sum in (25) effectively counts each transfer of effort twice. If we take account of constraint (26) by a Lagrangian multiplier B then the differential of the Lagrangian form L is

oL

8~

{Li+ :=!C-Vt-9 Lt- := -!c - Vj - e =

(cr; > cr1) (cr; < crt)

We must thus have d; equal to cr;., not less than cr1, not greater than cr1 or equal to zero according as L;_ < 0 < Lt+• L;+ = 0, L 1_ = 0 or L 1_ > 0. This leads us to the improved policy. Let 22+ be the set of i for which v 1 is maximal and 22_ the set of i (possibly empty) for which v 1 < max vj - c.

(27)

1

Then all serving effort for members of 22_ should be transferred to members of 22+. The definitions of the v; will then of course be updated by substitution of the new values of service allocation. Such a reallocation of effort will occur whenever discrepancies between queues become so great that (27) holds for some i. The larger c, the less frequently will this occur. 7 REWARDED SERVICE RATHER THAN PENALISED WAITING Suppose the model of the last section is modified in that that a reward r 1 is earned for every customer of type i whose service is completed, no cost being levied for waiting customers. This is then a completely different situation, in that queues can be allowed to build up indefmitely without penalty. It has long been known (Cox and Smith, 1961) that the optimal policy under these circumstances is to serve a customer of that type i for which r1p,1 is maximal among those customers present in the queue. This is intuitively right, and in fact holds for service requirements of general distribution. We shall see that policy improvement leads to what is effectively just this policy. For a queue of a single type served at unit rate we have the dynamic programming equation in the undiscounted case 1

= AA(x + 1) + p,(x)[r- A(x)],

(28)

where xis queue size, A is arrival rate, p,(x) equals the completion rate JL if xis positive and is otherwise zero, r is the reward and A(x) is the increment f (x) - f (x - 1) in transient cost. Equation (28) has the general solution for A A(x)

=I- W +(I- AT) [!!:.]x-1 A-p,

p,-A

A

(29)

If p, > A then finiteness implies that the second term must be absent in (29), whence we deduce that

226

POLIC Y IMPROVEMENT: STOCHASTIC VERS IONS

6(x)

= r.

(30) This is the situation in which it is the arrival rate which limits the rate at which reward can be earned. In the other case, 11 < A, it is the completion rate which is the limiting factor; the queue builds up and we have !=J.l r,

6(x)

= (J.L/>Y- 1•

(31) The total reward rate for a queue of several types unde r a fixed allocation { cr1} of service effort is then 1

=I>~ min[>.;, cr;Jt;]. i

If we rank projects in order of decreasing r;J.l; then an optim al fixed allocation is that for which cr1 = >..;/ J.l; for i = 1, 2, ... , m, where m is as large as possible consistent with (22), to allocate any remaining servic e effort to type m + 1 and zero service effort to remaining types. Now consider a policy improvement step. We should direct the next service to a customer of type i which is present in the queue and for which i maximises J.L;[r;- 6 1(x1)]. It follows from solutions (30), (31) that this expression is zero for i = 1,2, ... ,m and equal to r1tt1[1- (cr;J.L;/>.;)x1- 1 ] fori > m. That is, one will serve any of the first m types if one can. If none of these are present, then in effect one will serve the custo mer type present which maxim ises r;J.l;, because the fact that the traffic intensity for queue m + 1 exceeds unity means that Xm+l will be infinite in equilibrium. It may 'seem strange that the order in which one serve s members of the m queues of highest effective value r;J.l; appears imma terial. The point is that all arrivals of these types will be served ultimately in any case. If there were any discounting at all then one would, of course, always choose the type of highest value among those prese nt for first service. 8 CALL ROUTING IN LOSS NETWORKS Consider a network of telephone exchanges, with the nodes of the network (the exchanges) indexed by a variable j = 1, 2, .... Supp ose that there are mjk lines ('trun ks') on the directed link from exchange j to excha nge k, of which Xjk are busy at a given time. One might think then that the vector ~ = {Xjk} of these occupation numbers would adequately describe the state of the system, but we shall see that this is not quite so. Calls arrive for a jk conne ction in a Poisson stream of rate Ajk. these streams being supposed indep endent. Such calls, once established, terminate with probability intensity /ijk. When a call arrives for ajk-connection, it need not be established on a direc tjk link. There may be no free trunk s on this link, in which case one can either look for an alternative indirect routing (of which there may be many) or simply not

227

8 CALL ROUTING IN LOSS NETWORKS

accept the call. In this latter case we assume that the call is simply lost-no queueing facility is provided, and the caller is assumed to drop back into the population, resigned to disappo intment We see that a full description of the state of the network must indicate how many calls are in progress on each possible route. Let n, be the number of calls in progress on route r. Denote the vector with elements n, by !! and let !! + e, denote the same vector with n, increased by one. Let us suppose that the a establishment of a jk-connection brings in revenue WJk. and that one seeks c dynami routing policy which maximises the average expected revenue. The programming equation would then be 'Y=

2:2:>-1kmax{O,w1k+max[f(!! +e,) -/(!!)] } r

k

j

(32)

+ LnrtLr [f(!!- e,)- /(_!!)], r

where 'Y andfind icate average reward and transient reward respectively. Here the r-maximisation in the first sum is over feasible routes which establish a jkconnection. The zero option in this sum corresponds to rejection ofthe call on the grounds that connection is either impossible or uneconomic. (The difference f (!!) - f (!! + er) can be regarded as the implied cost of establishing an incoming call along router. If this exceeds w1k then the connection uses capacity which could be more profitably used elsewhere. We can take as a convention that this cost is infinite if the route is infeasib le-i.e. requires non-existent capacity.) In the second sum ILr is taken to equaliLJk if route r begins inj and ends ink. The term indicated is included in the sum only if !! - e, ~ 0; i.e. if there is indeed at least one call established on route r. Solution of this equation seems hopeless. However, the value function can be a determined for the simple policy in which one uses only direct routes, accepting stage one apply then shall We trunk. free a is there if only call for this route if and of policy improvement. The average and transient rewards on one such link (for which we shall drop thejk subscripts) are determined by 1 = >.[w + f(x

+ 1)- f(x)]

+ tLX[f(x - 1)- f(x)]

(0 < x < m).

(33)

For x = 0 this equation holds with the term in IL missing; for x = m it holds with the term in >. missing. Let us define the quantity

.6.(x) = f(x)- f(x + 1) which can be interpreted as the cost of accepting an incoming call if x trunks are currently busy. One finds then the elegant solution

B) A( ) = w B(m, B(x, ())

w. X

(0 < X < m ) ;

'Y

= >.w [1 - B(m, e)] )

228

POLICY IMPROVEMENT: STOCHASTIC VERSIONS

where() = )..j J-L is the traffic intensity on the link and B(m, B) is the Erlangfunction

This is the probability tha~ all trunks are busy, so that an incoming call cannot be accepted: the blocking probability. The formula for Wjk thus makes sense. Returning to the full network, define

Then we see from (32) that one stage of policy improvement yields the revised policy: If a call comes in, assign it the feasible route for which the sum of the costs x along the route is minimal, provided that there is such a feasible route and that this minimal cost does not exceed WJk. In other cases, the call must or should be rejected. The policy seems sensible, and is attractive in that the effective cost of a route is just the sum of individual components x. For this latter reason the routing policy is termed separable. Separability ignores the interactions between links, and to that extent misses the next level of sophistication one would wish to reach. The policy tends to assign indirect routings more readily than does a more sophisticated policy. This analysis is taken, with minor modifications, from Krishnan and Ott (1986). Later work has taken the view that, for a large system, policies should be decentralised, assume very limited knowledge of system state and demand little in the way of real-time calculation. One might expect that performance would fall well below the full-information optimum under such constraints, but it seems, quite remarkably, that this need not be the case. Gibbens et al. (1988, 1995) have proposed the dynamic alternative routing policy under which, if a one-step route is not available, two-step routes are tested at random, the call being rejected if this search is not quickly successful. One sees how little information or processing is required, and yet performance has been shown to be close to optimal.

CH AP TE R12

The LQG Model with Imperfect Observation 1 AN INDICATION OF CONCLUSIONS Section 8.3 we assumed that the When we introduced stochastic state structure in a property generally referred to current value of the state variable was observable, same as 'complete information', as 'perfect observation~ Note that this is not the e future course of the process is which describes the situation for which the whol predictable for a given policy. observed (i.e. if the current If the model is state-structured but imperfectly simple recursive treatment of value of state is imperfectly observable) then the in this case, an effective state Section 8.3 fails. We shall see in Chapter 15 that, n state: the distribution of x 1 variable is supplied by the so-called informatio !'£'with which we have to work conditional on W,. This means that the state space butions on!'£', a tremendous has perforce been expanded to the space of distri increase in dimensionality. tion, in that for them these However, LQG processes show a great simplifica and so are parametrised by the conditional distributions are always Gaussian, covariance matrix V1• Indeed, conditional mean value x1 and the conditional develop in time by deterministic matters are even simpler, in that V1 turns out to h we appeal to the observed whic and policy-independent rules. The only point at as an x of n 1• One can regard x1 course of the process is then in the calculatio g estin inter is It • W on 1 mati infor nt estimate of the current state x 1 based on curre even bles; serva unob ate estim to one s that the formulation of a control rule force rule implies criteria for the more interesting that the optimisation of this optimisation of estimation. properties of linear relations LQG processes have already been defined by the noise. We shall come shortly to a between variables, quadratic costs and Gaussian briefly and exactly. If we add definition which expresses their essence even more regulation problem of Section the feature of imperfect observation to the LQG prototype imperfectly observed 10.1 then we obtain what one might regard as the this the plant and observation state-structured LQG process in discrete time. For relations take the form (I) r-l+ Et

x, =

AXt- 1

Yt = CXt-1

+Bu

+ fJt

(2)

230

THE LQG MODEL WITH IMPERFECT OBSERVATION

where y 1 is the observation which becomes available at timet. We suppose that the process noise E and the observation noise 1J jointly constitute a Gaussian white noise process with zero mean and with covariance matrix (3)

Further, we retain the instantaneous and terminal cost functions

c(x,u)=~[~r[~ ~][~],

(4)

of Section 9.1. If we can treat this model then treatment of more general cases (e.g. incorporating tracking of a reference signal) will follow readily enough. We shall see that there are two principal and striking conclusions. One is that, if u1 = K 1x 1 is the optimal control in the case of perfect state information, then the optimal control in the imperfect-observation case is simply u1 = Kr!xr. This is a manifestation of what is termed the certainty equivalence principle (CEP). The CEP states, roughly, that one should proceed by replacing any unobservable by its current estimate, and then behaving as if one were in the perfect-observation case. ltturns outto be a key concept, not limited to LQG models. On the other hand, it cannot hold for models in which policy affects the information which is gained. The other useful conclusion is the recursion for the estimate i 1

(5) known as the Kalman filter. This might be said to be the plant equation for the effective state variable i 1; it takes the form of the original plant equation (1), but, instead of being driven by plant noise Er, is driven by the innovation Yt- Cic-1· The innovation is just the deviation of the observation y 1 from the value E(yrl Wt-i) that one would have predicted for it at timet- 1; hence the name. The matrix H 1 is calculated by rules depending on V1_ 1, , The Kalman filter provides the natural computational tool for the real-time determination of state estimates, a computation which would be realised by either a computational or an analogue model of the plant. Finally, LQG structure has really nothing to with state structure, and essential ideas are indeed obscured if one treats the state-structured case alone. Suppose that X, U and Y denote process, control and observation realisations over the complete course of the process. The cost function C will then be a function C(X, U). Suppose that the probability density of X and Yfor a control sequence U announced in advance is

f(X, Yl; U) =

e-O(X,YI;U)

(6)

(so that U is a parametrising variable). This must be a density relative to an appropriate measure; we return to this point below. We shall term [)

OBSERVATION 2 LQG STRUCTURE AND IMPE RFEC T

231

asing improbability of plan t/ the discrepancy, since it increases with incre The two functions C and I[]) of the observation realisations X and Y for given U. stochastic structure of the problem. variables indicated characterise the cost and cost funct ion C and the discrepancy One can say that the problem is LQG if both the density (6) is relative to Lebesgue []) are quadratic in their arguments when the measure. economically, that dyna mic This characterisation indeed implies, very Gaussian. It also implies that relations are linear, costs quadratic and noise the only effect of controls on policy cann ot affect information, in that r in know n controls, and so can be observations is to add on a term which is linea variables take values freely in a corrected out. It implies in addition that the Lebesgue measure) and that the vector space (since the density is relative to t h (since histories X, Y, U are stopping rule is specification of a horizon poin taken over a prescribed time interval). ntly ofLQ G ideas) that the two It can be asserted quite generally (i.e. independe se the model. One migh t say that quantities C and [j) between them characteri they could be), in that one wishes both C and I[]) should be small (relative to what to make C small and expects IDi to be small. variable; i.e. [J)(x) for x alon e or One can define the discrepancy for any rand om discrepancy is the negative the I[D(xjy) for x conditioned by y. The fact that be normalised in different may h logarithm of a density relative to a measure whic constant. We shall assu me ive addit ways means that it is determined only up to an it so normalised that inf[J)(x) = 0. X

n p. and covariance matrix V, then So, if xis a vector normally distributed with mea (x- p.). Note the general validity of [Jl(x) is just the quad ratic form ! (x- p.) T v-I formulae such as [Jl(x,y) = [Jl(y)

+ [J)(xjy).

is taken in the form (2), with y 1 It is often asked why the observation relation ely previous state rather than on essentially being an observation on immediat work out nicely that way. Probably curre nt state. The weak answer is that things as a joint state variable, then y 1 is a the right answer is that, if one regards (xt,Yt) . This is an alternative expression function of curre nt state, unco rrupt ed by noise deal to recommend it: that one of imperfect observation which has a good error. observes some aspect of current state without

ERVATION 2 LQG STRUCTURE AND IMPERFECT OBS models involves a whole train of The treatment of imperfectly observed LQG as they seek the right development. ideas, which auth ors order in different ways

232

THE LQG MODEL WITH IMPERFE CT OBSERVATION

We shall start from what is surely the most economical characterisation of LQG structure: the assumed quadratic character of the cost C(X, U) and the discrepancy [])(X, YJ; U) as functions of their arguments. We also regard the treatment as constrained by two considerations: it should not appeal to state structure and it should generalise naturally to what is a natural extension of LQG structure: the risk-sensitive models of Chapters 16 and 21. The consequent treatment is indeed a very economical and direct one; completed essentially in this section and the next. Sections 6-8 are added for completeness: to express the results already obtained in the traditional vocabulary oflinear least square estimates, innovations, etc. Section 9 introduces the dual variables, in terms of which the duality of optimal estimation to optimal control finds its natural statement. Note first some general points. The fact that we have written the cost C as C(X, U) implies that the process has been defined generally enough that the pair (X1, U1) includes all arguments entering the cost function Cup timet (such as values of reference signals as well of plant itself). The dynamics of the process and observations are specified by the probability law P(X, YJ; U), which is subject to some natural constraints. We have P(X, YJ; U)

h-1

h-1

t=O

1=0

=IT P(xt+!,Yt+J/Xr, Yt; U) =IT P(xt+!,Yt+I/Xr, Yt; Ur)

(7)

the second equality following from the basic condition of causality; equation (A2.4). Further

[])(xt+l ,yt+I!Xr; Ur) = [])(xt+IIXt, Yr; Ur) + [])(yt+dX r+l, Yt; Ur) = [])(xt+dX t; Ut) + [])(Yt+I!Xt+l• Yt; Ur)

(8)

the second equality expressing the fact that plant is autonomous and observation subsidiary to it, in that the pair (X1 , Y1) is no more informative for the prediction of xt+I than is X 1 alone. . Relations (7) and (8) then imply

Theorem 12.2.1 The discrepancy has the additive decomposition h-1

[ll(X, YJ; U) = L[[])(xr+ J!Xr; Ur)

+ [])(yt+IIXt+l> Yr; Ut)].

(9)

t=O

Consider now the LQG case, when all expressions in (9) are quadratic in their arguments. The dyamics of the process itself are specified by h-1

[])(Xj; U) = L t=O

[])(xt+1IXt; U 1)

(10)

2 LQG STRUCTURE AND IMPERF ECT OBSERVATION

233

and the conditio nal discrepancies will have the specific form [])(xt+tiXt; Ut)

=! {xc+l -

dt+t - AtXt - Bt Uz) TNi+\ (xr+l- dr+l- ArXt - Bt Uz)

(11)

(10) for some vector d and matrices A, B, N, in general time-dependent. Relations and (11) between them imply a stochastic plant equation (12) (0 ~ t t) have been substituted the values which minimise [))(XI; U). The value ofu1 thus determined, Udet(X1 , U1_t ), i.s the optimal value ofutfor the deterministic process in closed-loop form.

Proof All that we have done is to use the determi nistic version of (12) to express future process variables x 7 ( T > t) in the cost function in terms of control variexables U and current process history Xt, and then minimis e the consequ ent r, Howeve pression for C with respect to the as yet underm ined controls u7(T;;;: t). is the point of the theorem is that this future course of the deterministic process fudetermi ned (for given U) by minimis ation of [))(XI; U) with respect to these 0 ture variables. Otherwi se expressed, the 'deterministic future path' is exactly the most probable future path for given X 1 and U. In optimisi ng the 'deterministic process' we have suppose d current plant history X 1 known; a supposition we now drop.

Exercises and comments (1) Note that state structur e would be expressed, as far as plant dynamics go, by P(XI; U) = IlzP(xt+tlxz; ut)· (2) Relation (12) should indeed be seen as a canonic al plant equation rather than necessarily the 'physical' plant equation. The physical plant equation might have the form

234

TH E LQ G MO DEL WIT H IMP ERF ECT OBSERVATION

where the plant noise e* is autoco rrelated. However, we can substit ute e;+I = E( e;+ 1 1X~> Ut) + ft+l· This standar dises the equation to the form (12) expectation is linear in the con , since the ditioning variables and e has the required orthogonality properties. The det erministic forms of the two sets of equations will be equivalent, one being deriva ble from the other by linear operati ons. 3 TH E CERTAINTY EQUIV AL

ENCE PR IN CIP LE

When we later develop the ideas of projection estimates and innova tions then the certainty equivalence principle is rather easily proved in a ver sion which immediately generalises; see Exe rcise 7.2. Many readers may fmd this sufficient. However, we fmd it economical to give a version which does not presume this apparatus, does not assume state-s tructure and which holds for a mo re general optimisation criterion. The LQG criterion is that one cho oses a policy 1r to minimise the exp ected cost E'Jr (C). However, it is actually sim pler and more economical for pre sent purposes to consider the rather more genera l criterion: that 1r should maximise E'Jr [e-BC] for prescribed positive 9. Since this second criterion function is 1 fJE'Jr(C) + o(9) for small 0, we see that the two crit eria agree in the limit of small (). For convenience we shall refer to the two criteria as the LQG-crite rion and the LEQG-criterion respectively, LE QG being an established term in which the EQ stands for 'exponential of quadratic ~ The move to the LEQG criterion induces a measure of 'risk-sensitivity'; of regard for the variability of C as well as its expectation. We shall see in Chapt ers 16 and 21 that LQG theory has a complete LEQG analogue. Indeed, LQG theory appears as almost a degene rate case of LEQG theory, and it is that fact wh ich we exploit in this section: LEQG methods provide the insightful and econom ical treatment of the LQG case. We wish to set up a dynamic progra mming equation for the LEQG mo del. If we defme the somewhat transformed total value function e-G(W,) = f(Y t) supE.r[e-BCIWt], 'If

where f is the Gaussian probab ility density, then the dynamic programming equation takes the convenient form e-G(W,) =SUp Ut

J

dyt+le-G(WHI)

(t
(13)

We lose no generality by assuming perfect information at the horizo n point h; this cannot affect policy since no further actions are to be taken. Thus Wh = (X, Y, U) and the terminal con dition for equation (13) is

CIPLE 3 THE CERTAINTY EQUIVALENCE PRIN

G(Wh) =[]) (X, Y/; U) +BC(X, U). t. The following simple lemma expresses a key resul

235 (14)

function of vectors z andy, positive Lemma 12.3.1 Suppose that Q( z, y) is a quadratic definite in y. Then

j exp[-Q(z,y)Jdy

= exp [q- io/ Q(z,y)]

(27r)' /2 1Qyy 1- 1/ 2], r is the dimension of where q is a constant having the evaluation log[ s of Q( z, y) with respect toy. y and Qyy is the matrix ofsecond-order derivative an integration with respect toy by The point of the lemma is that, if one replaces the result is correct as far as terms a minimisation of Q with respect to y, then erned. dependent on the second argument z are conc in Proof If y is the minimising value (which will one can write Q(z,y) = Q(z,y)

general be z-dependent) then

+!( Y- .YfQ yy(y - y).

ect to the variable y - y. The lemma now follows by integration with resp

0

We can apply the Lemma to the solution of (13). the dynamic programming equation Lemma 12.3.2 Assume LEQG structure. Then (13) with terminal condition (14) has the solution (15) J)(X, Y/; U) + BC(X, U)] + g1 inf inf t inf[[ G( W1) = u,:r :;. t yT:r> X . The value ofUt thus determined is where g1 is a policy-independent function oft alone the LEQG-optimal value ofthe control at timet.

ratic in its arguments at time T = h; Proof We see from (14) that G( Wr) is quad n of the integral in (13) by appeal to suppose this so at time t + 1. Then evaluatio the lemma leads to the conclusion that (16) G(W1) = inf inf G(W1+J) + ... Ur

Yr+l

tion oft alone, and we know the where + ... indicates a policy-independent func nature of G is thus established infimising u1 to be optimal. The quadratic (16). In making this last deduction inductively, and solution (15) follows from (14), they indeed commute because they we have re-ordered the extremal operations; isation occurs because, in the are all infimising operations. The .X-minim

236

THE LQG MOD EL WIT H IMP ERF ECT OBSERVATION

dyn ami c prog ram min g equa tion at time h - 1, the inte grat ion over Yh amounts to . an integration over all variables of the full desc ripti on which are not dete rmin ed . from Wh-1 · 0 The restrictions on the form of[) whic h we dedu ced in the last section imply a strengthening of the last theorem.

Theorem 12.3.3 Assume LEQG structure. Then the dynamic programming equation (13) with terminal condition (I 4) has the solu tion G(W1) =in f{D (X1, Y 1 i; Ut-I) x,

+ u,.:r;; inf .t

inf [D(xt+t. ... ,xh!Xr; U)

~=r>t

+ OC(X, U)]} + g1

(17)

where g1 is a policy-independent/unction oft alone. The value ofu1 thus determined is the LEQG-optimal value of the control at time t. If the value of u1 minimising the square bracket is denoted u(X , u,_,) then the LEQG-optimal value of Ut is u( x, is the value of X dete rmined by thefina l X 1-minimisation. 1 The tran sfor med form (17) of (15) follo ws by appe al to the redu ctio n of [) expressed in (9). The final asse rtion of the theo rem follows from it. This asse rtion is very significant; it is actually the risk-sensitive form of the certainty equivalence principle (CEP), at least for the case 0 ~ 0. We shall develop it in Cha pter 16. The poin t for the mom ent is. that the CEP we seek follows immediately from it.

Theorem 12.3.4 (The Cert aint y Equivalence Principle for the LQG case) Assume LQG structure. Then the optimal value ofu 1 is Udet(X1(t), Ut-t ), where Udet (X1 , U,_I) is the optimal control for the deterministic case determined in Theorem 12.2.2 and x?> isth eva lueo fX1 minimising'D(X, Y1l; U,_,). Proof This is juSt the final asse rtion of the previous theo rem in the limit 0 L0. Suppose 0 small, and cons ider first the extremisations of the square bracket in (17). The values of ~( r ~ t) minimising the square bracket for given U are, to within a term O(O ),jus tthe valu es minimis ing D(xt+ 1, ••• ,xhi X1; U). Tha t is, just the future course of the deterministic vers ion of the process (for given X, U) to within a term 0(0). The se yield a value for D(x1+t. ... Xh!X, U) which is 0(02 ), and the u-minimisations which follow are essentially min imis atio ns of OC(X, U) with future process variables { Xri r > t) given thei r dete rmin istic prediction. The value u(X1, U1-1) of Ut which minimises the squa re bracket is thus Udet ( X 1, U,_ )+ 1 0(0). The min ima l value of the squa re brac ket in (17) is then 0(0), with the consequence that the value of X1 min imis ing the curly brac ket is just x, + 0( 0), with the defmition of x?> mod ified to that state d in the theorem. The fina l

4 APPLICATIONS: THE EXTENDE D CEP

237

determination of the minimising value of Ut, which we know to be LEQGoptimal at 0, is then U
Ut

= Kxr

+ L Ljdt+j+l j=O

(18)

238

THE LQG MOD EL WITH IMPE RFEC T OBSE RVATION

for know n matr ices L1, wher e we have for simpl icity writt en out only the timeinvar iant form of this relation. If one penalises the depa rture of x and u from presc ribed reference signals then we know (see the sectio ns quote d) that we can reduc e the optim al contr ol to the form (18), when d is a linea r comb inatio n of actua l distu rbanc es and reference signals. In practice, the future cours e of distu rbanc es and reference signals will in gene ral be unkn own, but, unde r LQG assum ption s, the CEP will be applicable, and one can asser t that the optim al contr ol rule will be just 00

Ut

(t) = Kx t

~ L d(t) + L-J i t+j+l

j=O

(19)

Here d~ 1 is an effective forecast of the value of dr+J form ed at time t, in that it is the most proba ble value of dt+J cond ition al on infor matio n available at time t. If one does not feel confi dent enou gh of the statis tics of the situa tion to form a probability mode l then the 'mos t proba ble' forec asts in (19) could be repla ced by the best infor mal or subjective forec ast available. One could not say much abou t perfo rman ce of the rule in such a case, but at least one could be sure that the rule was of a form know n to be optim al for a fully speci fied model. Altho ugh we deriv ed the CEP by appe al to the dyna mic prog ramm ing recursi on (in parti cular , in the passa ge from (13) to (17)), it could also be appli ed to a direc t trajec tory optim isatio n (as is indee d impli cit in Theo rem 12.2.2). That is, at tili\e t one just subst itutes cond itiona lly most proba ble estim ates for unob serya bles and then carri es out, by what ever mean s convenient, a direc t optim isatio n of the future trajec tory thus predi cted. The deter mina tion of the curre nt contr ol obtai ned in this way is then optim al. This extended form may be useful if one wishes to resto re the ment al pictu re of the future cours e of the proce ss whic h is largely lost in a recursive form ulation. The values u~l for r > t oflat er decisions form ed in this way are not in general the optim al values of these quantities, but they are indee d the best estim ates one can form at time t of what these optim al future decisions will be (an asser tion we make explicit in Exercise 7.1). This is one advan tage of the exten ded formulation; that optim isatio n is seen as part of a provisiona l forward plan. This plan will of cours e be upda ted as time moves on and one gains new infor matio n, but the fact that it has been form ed gives one much more consc ious appre ciatio n of the cours e of events and the conti ngen cies again st whic h one is guard ing. It is a view of optim isatio n whic h lies close to the intuitive meth ods by whic h we cope with daily life. Agai n, one can imag ine form al estim ates being repla ced by infor mal estimates in cases where one does not have the basis for a comp lete stoch astic model.

a

Of course, the point abou t the recursive appro ach is that it is econ omic al: one does not have to hold a whole future in one's mind (or processors). The chess analo gue we have already given in Secti on 1.3 bring s home the contr ast betw een

239

5 CALCULATION OF ESTIMATES: THE KALMAN FILTER

the two views. To play chess by the extended method is to choose one's current JilOVe after analysing all likely sequences of play some time into the future. To play ; chess by the recursive method is to give every board configuration x a value F(x) and to look for a move which changes the configuration to one of greater (or Jllaximal possible) value. In other words, an ultimate goal (winning) is replaced by an immedia te goal (improvement of configuration). Of course, the chess example is complicated by the fact that there are two players, optimising for different goals. However, there are plenty of other examples. One can say that Nature has allowed for the limited intelligence (and processing power) of her creations by replacing ultimate goals by intermediate goals. An animal does not eat in order that it may not die (a distant goal) or in order that the species may survive (even more distant) but in order that it may not feel hungry (immediate goal). Evolution has implante d a mechanism which both senses relevant aspects of state (a need for food) and evaluates them (so that a condition of extreme hunger is unpleasant). 5 CALCULATION OF ESTIMA TES: THE KALMA N FILTER In order to implement a control rule such as as u1 = K 1x1 for an imp,erfectly observed state-structured system we have to be able to calculate 1 = x/>. LQG assumptions will imply that the distribution of x 1 conditional on W1 is normal, with mean x1 and covariance matrix V, say. The conditional mean value x1 is indeed identifiable with the conditionally most probable value, because it is the value which maximises P(x11W,). If the observation relation is also 'state-structured' in the appropriate sense then one can develop recursions which express and V, in terms of 1. V1_ 1, 1 and u1- 1 . The recursion for 1 is the ubiquitous Kalman filter (5). Such recursions supply the natural way of calculating the estimates, whether control is realised by digital or analogue means. Before developing them we shall clarify notation and some properties of the normal distribution. Suppose that x and yare a pair of random vectors (so defmed as joint random variables). Let us suppose for convenience that they have zero means: E(x) = 0, E(y) = 0. Then the matrix E(xyT) with jkth element E(xjyk) is the covariance matrix between x and y (note that the order is relevant). We shall denote it variously (20) E(xyT) = cov(x, y) = Vxy·

x

x

x,

x,_

y

We shall denote cov(x, x) simply by cov(x); the covariance matrix of the random vector x. If xis scalar then cov(x) is simply the variance of x, which we shall write as var(x). If cov(x, y) = 0 then one says that x andy are mutually uncorrelated, and writes this x.iy. The relation is indeed an orthogonality relation under conventions natural to the context.

240

THE LQG MODEL WITH IMPERFECT OBSERVATION

Lemma 12.5.1 Suppose the vectors x andy are jointly normally distributed with means. Then the distribution of x conditional on y is normal with mean and iance matrix given by

Proof Denote the value asserted for E(xly) in (21) by x; we shall later see this notation as consistent. Then one readily verifies that x - x and y are independently (and normally) distributed, so that the unconditioned distribution of x - .X is the same as its distribution conditional on the value of y. But one again verifies that the unconditioned distribution of x - x is normal with zero mean and covariance matrix equal to the expression for cov(xly) asserted in (21). The conclusion of the theorem thus follows. 0 We have implicitly assumed Vyy non-singular, but the assertions of the theorem remain meaningful even if this is not the case; see Exercise 1. We come now to the principal conclusions.

Theorem 12.5.2 Assume the imperfectly observed state-structured model specified in relations (1)-(4) of Section 1. Suppose initial conditions prescribe xo as normally distributed with mean x0 and covariance matrix V0. The model is then LQG and the distribution of x 1 conditional on W 1 is normal with mean x1 and covariance matrix Vr, say. (i) The estimate x1 obeys the updating relation Xr

= Axt-1 +But-! + Hr(Yr- Cxt-d

(22)

(the Kalman filter), where

(23) (ii) The covariance matrix V1 obeys the updating relation

vi= N + AVt-tAT- (L + AV1-1 cT)(M + cv1-t cTrl (LT + cv1-1AT). (24) Proof There are many proofs, and we shall indeed give one in Section 8 which dispenses with the assumption of normality. However, this assumption has been intrinsic to the LQG formulation and makes for what is by far the most economicalproof. The preliminary assertions of the Theorem follow from previous discussion. If we denote the estimation error x 1 - x1 by b.1 then V1 is exactly the covariance matrix of b. 1. The quantities Xr-1 - Xr-1, Xr- Ax1-1 - But-1, and Yt - Cxt-! are

I

241

: THE KAL MAN FILTER 5 CALCULATION OF ESTIMATES

The se are jointly norm ally with ~t-1. Er and 'IJt respectively. with zero mea n and covariance mat rix

[ v0 0

0

N L1

0

L M

l

.

join t as V. This effectively gives us the where we have written Vt-1 simply is to do t mus nal on W,_t, ur-I · Wh at we distribution of Xt-i . x, and Yr conditio by Xt of tion then con ditio n the distribu integrate out the unobservable Xr-t and -t, (Wr dist ribu tion of x 1 conditional on the value of Yt. so obtaining the the to nt gration over values of x 1_ 1 is equivale Ut-1 ,y 1) = W1, as desired. Inte t-t and -Bu -t Ax 1 Xt= <:; quantities substitution Xr-t = Xc-J + ~t-J· The I and so At+C 'IJt and I Atidentifiable with Et +A ( 1 = y 1 - C.Xr-1 are then rix ted with zero mea n and covariance mat ribu are jointly norm ally dist

VA 1 [ N1 +A L + CVA 1

L +A VC1 ] M + CVC1

(25)

nal 12.5.1 that the dist ribu tion of C con ditio It then follows by an appeal to Lem ma H1 re whe , V rix 1 n H 1( 1 and covariance mat on the value of ( 1 is norm al with mea nt eme stat a and (24). Converting this into and V1 have the values asserted in (23) (i) s rtion asse (and W1_ 1) we deduce the of the dist ribu tion of Xt con ditio nal ony1 0 and (ii) of the Theorem. n of Ric cati type, analogous to the Rec ursi on (24) is a forward recursio a optimisation. The analogy is based on backward Ric cati recursion of control only is it ; istic rmin dete is 9. The recu rsio n real duality, as we shall see in Section V ating. However, we need the value of upd its for which need s the observations the 1, ion Sect in see from (23). As observed to dete rmi ne the updating, as we g the equ atio n with the estimate replacin t Kal man filter (22) is just the plan It is t. e nois t innovation (rep laci ng the plan actual stat e value x and a term in the y1 ion rvat ides a corr ecti on when the obse this latter term which of course prov ch would have bee n predicted. turn s out to differ from the value whi ly con tinu ous- time version. We shall simp All this mat eria l has an imm edia te or by a ined either by analogous arguments quote results, since these can be obta t. formal pass age to the continuous limi ion el specifies the plan t and observat mod ured The stan dard state-struct relations (26) Cx+ "l

x

x

x

.X =A x+ Bu + E,

y

=

n ion noise) are jointly white, with zero mea where f. and "' (plant noise and observat rential form, Note that the plan t relation is in diffe and pow er mat rix [

;!r

i£] .

242

THE LQG MODEL WITH IMPE RFEC T OBSE RVATION

the observation relation in instantaneous form. This is generally regarded as natural formulation, although we shall have occasion to recast it in Chapter 25.

Theorem 12.5.3 Assume the imperftctly observed statestructured model sveczrz,pil above. Suppose that initial conditions prescribe x( 0) as normally distributed with i(O) and covariance matrix V(O). Then the distributio n ofx(t ) conditional on is norma/with mean i(t) and covariance matrix V(t), say, determined as follows. (i) The estimate xobeys the updating relation

i =Ax +Bu + H(y -

Cx)

(the Kalman filter), where H

= (L + VC1 )M- 1 •

(ii) The covariance matrix Vobeys the updating relation

V = N +AV + VA 1

-

(L+ VC1 )M- 1(L1 + CV).

Exercises and comments (1) Suppose that Vyy is singular, so that vectors c exist such that Vyyc = 0. These are elements of the null space %of Vyy· For such vecto rs var( c1 y) = c1 Vyyc = 0, so that c1 y is zero in mean square. Show that VxyC = 0 for such c. An equation Vyya = b for vector a will have a solution, which we shall denote by V_;; 1b, only if b is orthogonal to all c of .JV, and this solution is arbit rary to within addition of an element of% . Show that different evaluations of the expression for E(xjy) in (21) obtained in this way differ by a term c1 y for some c of .JV (and so are zero in mean square) and that expression for cov(xjy) is uniqu ely determined.

6 PROJECTION ESTIMATES Our LQG analysis has generated effective estimates, the most probable values of unobservable random variables conditional on the values of observables, under the assumption that all random variables are jointl y normally distributed. However, the estimates thus generated have distin ctive properties if one makes no distributional assumptions at all, apart from speci fication of first and second moments of the variables concerned. This weakening of hypotheses leads to a new view of the conclusions derived under Gaussian hypotheses in Sections 1-5, and a valuable one. We shall assume notations (for covariance matrices, etc.) and defin itions (of orthogonality etc.) listed in Appendix 1. Suppose that x is a random vector of zero mean and covariance matrix Vxx· Then we can still define the quadratic form

6 PROJECTION ESTIMATES

243

D(x) = !xTV~1 x, norm al density had x which would have occu rred in the expo nent of the then have term ed the norm ally distributed, and which we woul d ugh it is only when x is norm ally iilel'l:Pillll'-'Y· Let us conti nue to do so (altho istrilbutc:a that we can identify D(x) with II) ( x)). non-singular A of appro priat e dime nsion we have

D(x) =!(Ax)T(AVx.xAT)- 1(Ax). A VxxAT is just the covariance matr ix of the trans form

ed vector Ax. We thus

the simple but significant conclusion:

(Ax) T v- 1(Ax). Then necessarily ,,,,, ,, , L,_,,,_J2.6.1 Suppose that D(x) can be written! ·· = cov(Ax).

k,. ·. ·. From this we furth er dedu ce the following lemm L:··: ...... L~mma12.6.2 infxD(x,y) = D(y).

a.

r

! .

linea r function of y. Then Proof Let the minimising value be deno ted x, a D(x, y) can be written as the sum of quad ratic forms (30) (30) and the previous prop ositio n for some Vt, Vz. It then follows from expression l, with V1 = cov( x- x) and that x- x and y are mutu ally ortho gona final form in (261 the V2 = cov(y). Since plainly infxD(x,y) equals the D conclusion then follows. ofx in terms of y with whic h Note that the .X thus deter mine d is just the estimate ble value if distributions are we are familiar, the conditionally most proba estimate. norm al. Let us now term it the minimal discrepancy not, and one would wish to has x and Suppose now that y has been observed lest estimate would be a linear form an estimate of x in term s of y. The simp estimate, ofthe form

x=H y.

(31)

a linear least-square estimate: to and the most imme diate conc ept would be that of re error E[ (x - x? G(x - x) choose the coefficient H in (31) so that the mean -squa readily finds that noth ing is is mini mal for presc ribed positive-definite G. One (i.e. the optim al value of the gaine d by the addit ion of a cons tant term to (31) H yields the condition to ct cons tant is zero) and that optim isatio n with respe (32) X- X ..L y,

244

THE LQG MODEL WITH IMPERFE CT OBSERVATION

independent of G. That is, that the estimation error is orthogonal to the observations. Let us term a linear estimate satisfYing (32) a projection estimate. Then we can assert that condition (32) is sufficient for the linear least-square property as well as necessary, in that a projection estimate also has the linear least-square property. In fact, we can assert a number of equivalences.

Theorem 126.3 The following characterisations ofan estimate x of x in terms ofy are equivalent. (!) It is a projection estimate, i.e. a linear function ofy for which x- x ..L y. qz) It is a linear least-square estimate. qiz) It is the value ofx which freely minimises the discrepancy D~, y). qv) It has the property that cov(x- x)

~

cov(x- x).

(33)

for any linear estimate x. All such estimates are effectively identical, in that they are equal in mean square. Proof Let us abbreviate 'linear least-square' and 'minimal discrepancy' to LLS and MD respectively. We know that the LLS estimate is a projection estimate and it follows from the proof of Lemma U.6.2 that the same is true of the MD estimate. If x = Hy then the orthogonality condition (32) implies that 0 = E[(Hy- x)yTJ = HVyy - Vxy·

(34)

So, if Vyy is non-singular, then the projection estimate has the unique determination X= A

vxy v-• yyY·

(35)

(cf. (21)). If Vyy is singular then the analysis of Exercise 5.1 shows that the solution (35) is still meaningful and different evaluations of it are equivalent in mean square. However, proof that a projection estimate has the stronger LLS property (34) both provides the reverse implications (i) => (iv) => (ii) and demonstrates mean square equivalence. because x - is a linear function of y. We have x ..L x Note first that then

x-

x,

x

cov(x- x) = cov(x- x + x- x) = cov(x- x) +cov(x - x,x- x) + cov(x- x,x- x) = cov(x- x) + cov(x- x)

+ cov(x- x) (36)

Now cov( 6) ~ 0 for any random vector 6, and in particular for the vector 6 = x - x. Thus inequality (33) holds.

6 PROJECTION ESTIMATES

245

the the argu men t above also establishes If x is also a projection estimate then lity. reverse inequality to (33), and so equa square. (x- x) = 0, so that x = x in mea n cov then (33) in s If equality hold als ty then shows that cov (x- x,y) equ An application of Cauchy's inequali 0 sis. Thus xis also a projection estimate. cov (x- x,y), which is zero, by hypothe

, y), re is trouble with the definition of D(x We have glossed over one point. The t join the if ate, estim imu m discrepancy and so with the con cep t of the min nt ifica sign very a ular. The poin t is actually covariance mat rix of x and y is sing the deal with it (see Exercise 1) provide we one, beca use the means by which 9. tion Sec whole con trol problem; see passage to the 'righ t' formulation of the have the erro r covariance mat rix We verifY that all proj ecti on estimates (37) 1 cov (x- x) = Vxx - Vxy V_;;, Vyx, rcise in the mor e circ ums pect sense of Exe immediately if Vyy is non-singular and 5.1 otherwise. on esti mat e x of x in term s of yas We shall som etim es write the projecti (38) jy),

x = c&'(x

of s. Thi s can be regarded as a tran slat ion to emphasise the role of the two variable al, norm were s able vari be app ropr iate if the the expression E(xjy), which would line ar prop ertie s are assumed. c&'(xiy) is a rder to the case whe n only seco nd-o that x. It is a line ar ope rato r in function of y having the dim ensi on of 8(x1 + x2iY) = 8(xJiy) + 8(x2iY) and 8(Axiy) = A8(xiy) is that it ension. However, the essential poin t for x 1 and x2 vectors of the sam e dim erties. prop c risti acte the following char is inde ed a proj ecti on operator, enjoying

Theorem 12.6.4

m8 2 = 8, in that $[8(xiy)iy]

vz) IfYI .l Y2 then

I l

I !

= 8(xJy). (39)

from S(Hyly) = HC(yjy) = Hy. AsProof Ass ertio n (i) follows immediately nd mem ber of (39) is from the observations that the righ t-ha

sert ion (ii) follows late d that the erro r of this estimate is unc orre cert ainl y a line ar esti mat e of x, and with both y1 and y2.

Exercises and comments

nite V 1 .[2..\T x- ..\TV..\] for non-negative defi (1) The mat rix identity xTv - x = sup.> +oo is on is sing ular (in that the evaluati is fundamental, and valid even if V

246

THE LQG MODEL WITH IMPERF ECT OBSERVATION

unless x lies in the orthogo nal complement of the null space of V). It gives us the evaluation of the discrepancy D(x,y)

=

r[

s~y [ATx + pTy- ![~ ~;: ~;;] [~ J]

The minimi sing values of A and p are the differentials of D with respect to x andy. If x has the discrepancy-minimising value then). must be zero, so the extremal equations with respect to A and f-t become x = Vxy/L andy = Vyy/.L· This amoun ts just to the equatio n (35). The analogues of A and f-t in the dynamic context are just the Lagrange multipliers associated with plant and observation equations, and 'it is by their introduction that a direct trajectory optimisation finds its natural completion (see Section 9) as it did already in the case of perfect observation (Sectio n 6.3).

x

7 INNOVATIONS The previous section was concer ned with a one-stage situation. In the temporal context we shall have a multi-stage situation, in which at every instant of time t (= 0, 1, 2, ... ) one receives a new vector observation y 1• The observa tion history at timet is then Y1 = {yr; 0 ~ T ~ t}. Define the innovation ( 1 at time t by (o =Yo- E(yo), (t=Yt-< if(YtlY t_I) (t=l,2 ,3, ... ). (40) The innovation is thus the deviation of the actual observation Yt at timet from the projection forecast one would have made of it the momen t before, at time t - 1. It thus represents the 'new' inform ation gained at time t, in that it is that part of y 1 which could not have been predict ed from earlier information.

Theorem 12.7.1 The innovations are mutually uncorrelated. Proof Certain ly ( 1 l. Yt-l· It follows then that ( 1 l. (r forT < t, becaus e (r is a linear function of Yt-l· 0 Now define

x(t) = .C(xl Yt), the projection estimate of a given random vector x based on inform ation at time t. We shall use this supersc ript notatio n consistently from now on, to denote the optima l determ ination (of the quantity to which the supersc ript is applied , and under a criterion of optimality which may vary) based on inform ation at timet. It follows, by appeal to (39), that

x(tl = C(xl Yt-d

+ tB'(xlCr)

= x(t-i)

+ HtCt

(41)

7 INNOVATIONS

247

where the matrix Hr in fact has the determination

Hr

= cov(x, Ct)[cov((r)r 1.

(42)

Equation (41) shows that the estimate of x based on Yz-l can be updated to the estimate based on Y1 simply by the addition of a term linear in the last innovation. This is elegant conceptually, and turns out to provide the natural algorithm in the dynamic context. Equation (37) of course implies further that X(t)

=

t

t

r=O

r=O

'L: l(xl(r) = L Hr(r-

which is nothing but the development of a projection in terms of an orthogonal basis. However, the innovations are more than just an orthogonalisation of the sequence {y1}; the fact that the orthogonalisation is achieved by the timeordered rule (40) means that ( 1 indeed has the character expressed in the term 'innovation: The innovations themselves are calculated by the forward recursion 1-l

(t

= Yt- L l(yrl(r)·

(43)

r=l

Alternatively, we can take the dual point of view, and calculate them by minimising the discrepancy backwards in time rather than by doing least square calculations forward in time. Theorem 12.7.2 The value ofy 1 minimising D( Y1) is C(y 1 1 Yr-1) and the minimised value is D ( Yt-1). Further,

(44) where Mt = cov( (t). Proof The first pair of assertions follows from the theorems of the last section,

and relation (44) is nothing but relation (30) transferred to this case.

0

We plainly then have the additive decomposition of the discrepancy

D(Yr)

=

n=~=l (~M; 1 Cr·

Exercises and comments

(1) Let u~l be the estimate of the optimal value of uT formed at time t by the application of the extended CEP in Section 4. Show that indeed uVl = l(u~r) IW1 ).

248

THE LQG MODEL WITH IMPERFEC T OBSERVATION

(2) A proof of the certainty equivalence principle. Consider the state-structured

LQG model expressed in equations (1)-(4), the criterion being minimisation of expected cost. Let F(x 1 , t) and u(x1 , t) be the value function and oftimal control rule at time t under perfect state-observation. Note that x~t+l = x~r) + z1 = x1 + z1 where, conditional on ( Wr, ur), the random variable z r has zero expectation and a covariance matrix independent of policy or process history. Hence show, by an inductive argument, that the value function and optimal control in the case of imperfect observation are F(x1, t) + · · · and u(x 11 t) respectively, where+ · · · indicates a term independent of policy or process history. 8 THE KALMAN FILTER REVISITE D We can now give a quick derivation of the Kalman filter under purely secondorder assumptions, when x1 is defined as the projection estimate tf(xrl Wr). The proof is in some ways more enlightening than that of Section 5, although it needs more in the way of preliminary analysis. We have Xr = tf(Axr-l +Bur-l+ =Bur-!+ O"(Axr-!

= A.Xr-1

c:rl We)

+ t:eiWr-1) + tf(AXr-I + t:rl(r)

(45)

+ Bur-! + Hr(t

for appropriate H 1• Here ( 1 is the innovation (46) so the form (22) of the Kalman filter recursion follows from (45) and (46). Further 1 Hr = cov(Axr-1 + er, (r)[cov((r )r 1 = cov((;, (r)[cov((r )r ,

(47)

where

(48) But, as in Section 5, (; and ( 1 can be written as e1 + A~r-1 and 'TJr + C~r-I. respectively, and are jointly normally distributed with zero means and with covariance matrix (25). The expression (23) for H 1 thus follows, as does expression (24) for (49) The quantity ( 1 is certainly the innovation in the observations. The quantity (; equals x 1 - tf(x11 Wr-1 ), and so can be regarded as the plant innovation. 9 ESTIMAT ION AS DUAL TO CONTRO L The recursion (24) for the error covariance matrix V1 stands in obvious parallel to the Riccati recursion (2.25)/(2.26) of LQ control optimisation. The analogy is

r

'';•i>

9 ESTIMATION AS DUAL TO CONTR OL

249

of covariance complete, with only the 'dualising' modifications that transposes d instead of forwar goes ion matrices replace cost matrices and the recurs which is tion estima and l backward. In fact, there is a duality between contro ngian Lagra the of on constantly in evidence, but is revealed cleanly only by adopti methods of Section 6.3. conditions, The stochastic model must be completed by prescription of initial ution distrib the of which in the present state-structured case means prescription . The V matrix 0 of xo conditional on Wo; norma l with mean .Xo and covariance to up tion realisa expression for Dr= [l)(X1, Ytl Wo; Ur- 1 ), the discrepancy for the time t, is then

u by appeal to where A, E and TJ are understood as expressed in terms of x, y and n relatio initial the and (2) the plant and observation relations (1), (51) Ao = .Xo- xo. {x~l; 0:::;; T~ t}, The projection estimate of the course of the process up to timet, x'1'-values. is obtained by minimising [l)r with respect to the corresponding less natural does the , inistic determ being to is model However, the nearer the s appear in (11) inverse whose es matric ance this formulation seem. The covari zero, and this ally identic is nent compo noise and (50) will be singular if any forms. these ising minim and sing plainly presents problems in discus reformulation The resolution turns out to be a continuation of the Lagrangian naturally to over carries indeed which of Section 6.3 for the deterministic case, plant and the view we is, That . the stochastic imperfectly observed model aints constr as (51) ion condit observation equations together with the initial us Let liers. multip ge Lagran of which are best taken care of by the introduction and 0 = T case the in (51) introduce multipliers IT to take account of constraint multiplier mT to the plant equation at time Tin the case T > 0; correspondingly, a the Lagrangian has take account of the observation relation at time T. One then

form r

[l)r-

IQ(xo- .Xo- b.o)- 2)z;( xT- AxT-1 - BuT-! - ET) T=J

+ m;(YT - CxT-1 -TJT)], to be maximised to be minimised with respect to x and the noise variables and a superscript have should liers with respect to the multipliers. Properly, the multip at time t. ation inform on t to indicate that the extremisation is one based when only cript supers this e However, for notational simplicity we shall includ the point needs emphasis.

THE LQG MODEL WITH IMPERFECT OBSERVATION

250

Theorem 12.9.1 If the noise variables are minimised out ofthe Lagrangian form then the problem is reduced to that ofrendering the time integral I

O(l,m, x) =[!lTV/+ zT(x ~ x)Jo + l:[v(lr,mr)

(52)

T=J

+ !J(xr ~ Axr-1 ~ Bur-d + m;(Yr ~ Cxr-I)] maximal with respect to the x variables and minimal with respect to the (/, m) variables. Here

v(l,m)=![~r[~ z][~]

(53)

and the noise estimates are related to the multipliers by T7

/(t) ~ -

r 0 0

(t) A (1) - Xo - Xo , uo

(54)

A

(55)

Proof Equations (54) and (55) are just the stationarity conditions for the Lagrangian form with respect to the noise variables. At the values thus determined the Lagrangian form reduces to -0(/, m, x), and remaining assertions follow. D Note that the transformation clears the form of matrix inverses. Note also that 0 is the analogue of the Lagrangian form (6.19) for the deterministic control case, with(/, m, x) replacing (x, u, >.). Thequadraticform v(l, m) isjustthe information analogue of the instantaneous cost c(x, u) defined in (4). In general, we shall see that the primal and dual variables switch roles when we move from the control problem defined on the future to the estimation problem defined on the past . . The multipliers I and m are to be regarded as differentials of []) 1 with respect to the values of noise variables, which in turn implies the relations (54) and (55) between multipliers and noise estimates. By taking the extremisations of the form 0 in various orders we can obtain the analogues of various familiar assertions. Firstly, by writing down all the stationarity conditions simultaneously we obtain the equation system

[~ ~ !Hfr+[~"] ,~ 0

(1

~ 7 ~

(56)

t).

together with the end conditions uz(t) -

A

r,o - xo

~

(t)

Xo ,

(l,m)~) = 0

(T

> t).

(57)

r

~.·.

9 ESTIMATION AS DUAL TO CONTROL

251

In (56) we have again used the operator notation d=l-A ff,

with il = I - AT ff- 1 etc. Note also that the translation operator acts on the subscript r, not on the superscript t, which is regarded as fixed for the moment. In equation (56) we see the complete dual of the corresponding system (6.20) for the control problem. We can use the operator and operator factorisation approach just as we did in Section 6.3 and 6.5; the analogue of an infmite horizon will now be the passage from a start-up point at r = 0 to r--+ -oo. We shall follow through these ideas explicitly and for a more general case in Chapter 20. Theorem 12.9.2 Ifone extremises out the x-variables in 0then the problem is reduced to that ofminimising the form I

!UTV1)0 + l:)v(l,.,mT ) -z;B~-1

+ m;yT]

(58)

T=1

with respect to the variable (~ m), subject to the backward equation (0

~

r

~

t)

(59)

and end-condition (I, m)T = 0 (r > t). The assertion is immediate. Its point is that exhibits the estimation problem as the complete analogue of the control problem originally posed in Section 2.4 Equation (59) is the analogue of the plant equation, v(!, m) is the analogue of the instantaneous cost, and the initial term ! WV/) 0 is the analogue of the terminal cost. The difference lies only the occurrence of a few terms linear in (l, m) in the sum (58)-the observation and control variables constitute effective input terms which lead to a non-homogeneity. However, we are in general not interested in estimating all of process history, but merely the value of current state x 1• For this we resort to recursive arguments, just as the optimal control was deduced by the recursive argument of the dynamic programming equation. Theorem 12.9.3 The extremum of form (52) for prescribed x 1 and 11 is (! [T VI + [T (x - x) ]1 + · · · , where + · · · indicates terms independent of these variables. Proof Let us drop the !-subscripts for the moment The constrained extremum is certainlyoftheform!JT PI+ zT(x- J.t) +···for someP, J.t. Ifweminimisewith respect to 1 then this becomes - ! (x ~. J.t) Tp- 1(x - J.t) + · · ·. But this must be 0 identical with-! (x- x) Ty- 1(x- x) + ···,whenc e the assertion follows.

252

THE LQG MODEL WITH IMPERF ECT OBSERVATION

Theorem12.9.4

The recursion

holds, with

lt-1 =AT[+ cTm. The Kalman filter (22) and theforward Riccati recursion (24)for Vfollow. Proof It follows from the previous theorem and expression (52) for the time integral that we have the recursion [!/TV/+ /T(x- x)] 1 = ext{(!zTV/ + zT(x- x)] 1_1 + v(l,,m,) + !'[(x,- AXt-1- Bu,_r) + m"f(y,- Cx,-1)}, (62) where the extremum is with respect to /1_ 1 , x 1_ 1 and m1• The extremum with respect to 1 yields the condition (61) and the recursion reduces to (60). The Kalman and Riccati relations then follow immediately. 0

x,_

The reader may ask why we should bother with yet a third derivation of the Kalman filter. Well, the derivation is incidental to the goal; a formalism which is claimed as significant could hardly be regarded as living up to that claim unless it delivered the Kalman filter in passing. On~ point is that the Lagrangian approach, already seen as valuable for the optifiisation of deterministic control, is now seen to extend to this stochastic imperf~ctiY observed case. More sigiuficantly, it achieves the goal which has loomed ever more clearly as desirable: the characterisation of the optimisation problem as the free extremisation of a timeintegral such as (52) with respect to its variables. Constraints can of course not ~e eliminated, but they can be exhibited as natural consequ ences of a free optimisation in a higher-level problem. This higher-level formulation was achieved in two steps. The first was the deduction of the certainty equivalence ,principle, which absorbed the constraint constituted by the realisability condition (that control must depend on current observables alone). The second was the introduction of the dual variables I and m, which absorbed the constraints constituted by plant and observation equations. Comparison of the equation systems (6.20) and (56) demonstrates the complete symmetry between control/future and estimation/past, with primal variables (x, u) and dual variable s(/, m) switching roles in the two cases. There are other byproducts in terms of insight and solution: reduction of these equation systems by the canonical factorisation methods of Section 6.3 solves the stationary optimisation problem for stochastic dynamics of any order (see Chapter 18-21). Remarkably, insight becomes complete first when we consider the risk-sensitive models of Chapters 16 and 21, for which past and future time-integrals are

9 ESTIMATION AS DUAL TO CONT ROL

253

,A. assoc iated in a single integral, and the Lagrange multipliers land d. relate be the plant equation in the past and in the future can

-~~""""

Exercises and comments extrema are taken in the (1) Alternative forms. By varying the order in which the holds with the alternative last equation, demo nstrat e that the Kalm an filter (22) evaluation of Hand recursion for P.' (63) (64) 1 ions (63)/(64) have the Here A= A- LM- 1 C and N = N- LM- LT. Relat I (24), just as was the case same continuous-time limit version (28)/ (29) as do (23) for the corre spond ing control relations.

CH AP TER 13

Stationary Process; Spectral Theory E FUN CTIO N 1 STATIONARITY: THE AUTOCOVARIANC homogeneous and for whic h a Consider a system which is intrinsically timerule is successful, in that it is stationary control rule is adopted. If this control to steady-state behaviour. A stabilising, then the system will eventually settle down viour is itself termed stationary. stochastic process showing such steady-state beha statistically invariant unde r More technically, a process is stationary if it is this, consider first the discretetime-translation. To appreciate what is mean t by xis the process variable, and time case. Deno te the process by {x1}, so that whole process in a parti cular denote a realisation (i.e. the actual course of the tor f7 introduced in Secti on case) by X. Recall the backward translation opera formed sequence f7 X is X 1-t 4.2, which has the effect that the tth term in the trans sequence fl' X is x 1_,, for any rather than x 1• More generally, the tth term in the nary if statio is } integral r. One says then that the process {x 1

E[(fi'X)] = E[(X)]

(1)

of the realisation for which the for any integral rand for any scalar functional ly what is meant by 'statistical right-hand mem ber in (1) is defined. This is exact will in fact hold for all r if it invariance unde r time-translation'. Note that (1) holds for r = ±1. been to optimise in the finite Our approach to control optimisation hithe rto has tended to a stationary form ol horizon initially and then see if the optim al contr itself converge to stationarity in the infinite-horizon limit. The process may then . However, one could move in the course oftim e (i.e. in the infinite-history limit) g for the stationary rule which to the stationary situation immediately, by askin statio nary state. We make some minimises an expected cost per unit time in the clear, in any event, that there is observations on this point in Exercise 2, but it is d, unde r LQ assumptions there interest in the study of the stationary case. Indee with the state-structured case are two principal bodies of technique: one dealing the stationary case by trans form by recursive methods and the other dealing with methods. ption that xis vector-valued (a If we consider LQ models then there is an assum functions of X which we only column vector of dimension n, say) and that the process is Gaussian, then its need consider are linear/quadratic. Indeed, if the and second-order moments. statistics are completely characterised by its first-

STATIONARY PROCESSES; SPECTRAL THEORY

256

As far as frrst-order moments are concerned, the stationarity condition (1) will imply that E(x 1) is independent oft, so that

(2) for some constant n-vector J.L· As far as second-order moments are concerned, (1) will imply that that the covariance between x 1 and x 1_, is a function of r alone:

cov(xr,Xt-r)

=

v,,

(3)

say, for integral r. The quantity v, is termed the autocovariance at lag r of the process. It is an n x n matrix whose jkth element is the covariance between the jth element of x 1 and the kth element of x 1_,. It provides the only measure one has of dependence between values of the process variable at different times, if one is restricted to knowledge of first and second-order moments. It is therefore the only measure there is at all in the Gaussian case. If one regards v, as a function of the lag r then it is termed the autocovariance function. The full expression

v,

= E[(x 1 - J.L)(x 1_ , -

for v, and stationarity imply that v_,

JL)T],

(4)

= v'f, or v,

T = v_,.

(5)

That is, the autocovariance function is invariant under the combined operations of. transposition and lag-reversal. This plus the further property indicated in Exercise 1 are in fact the characterising properties of an autocovariance function. Commonly one supposes that the process has been reduced to zero mean (by adoption of a new variable x- J.L). In this case (4) reduces to v, = E[x1xJ_,]. A degenerate but important stationary process is vector white noise, {E1}, for which v, is zero for non-zero r. One normally supposes that this has zero mean, and, if v0 = V, then one speaks of {E1} as 'white noise of covariance matrix V~ The corresponding assertions for the case of continuous time are largely immediate; one simply allows the time variable t and the lag variable r to take any value on the real line. The one point that does need special discussion is the character of white noise, already covered to some extent in Section 9.5. Discretetime white noise {Er} of covariance matrix Vhas the property that the linear form I>=~ 1 E 1 has zero mean and covariance matrix 'L, 1 at V a'f, at least for all sequences of matrix coefficients {a 1} such that this last sum is convergent. The corresponding characterisation of continuous-time white noise is that the linear form a(t)E~t) dt is normally distributed with zero mean and covariance matrix Ja(t) Va(t) dt. The additional property of normality follows from the assumed independence and stationarity properties. If E had autocovariance function v(r) then the relation

J

DENSITY MATRIX 2 THE ACTION OF FILTERS: THE SPECTRAL

cov (j a(t)E(t) dt)

=

jj

257

a(tJ) v(t1- t2)a(t2)T dt1 dt2

asserted implies then that would hold in regular cases. The evaluation ion; an indication. of the v(r) = Vc5(r), where 6 is the Dirac delta funct noise. The matrix V is the exceptional character of continuous-time white time interval; it is appropriately covariance matrix of the integral oft: over a unit referred to as the power matrix oft:. Exercises and comments

2 0, for any sequence of column (1) Note that 1:1 L:k aJ v1-kak = E(l: 1 aJ Xt) ;;;;: ges. vectors {at} for which the first sum cover programming equa tion that (2) Note that there is no assumption in the dynamic one is considering an assum ed the controlled process is stationary, even when Kx 1). One may say that this is infinite-horizon limit (as with the simple rule Ut = nary, optimises passage to the because the control thus derived, although statio state. If one tries to determine steady state as well as performance in the steady s oneself an instr umen t (the an optimal steady-state rule directly then one denie less effective in dealing with reaction against transients) and may derive a rule al for the deterministic LQ optim such transients. For example, the rule u1 0 is dy been reached. It will alrea has problem of Section 2.4 if the equilibrium x 1 = 0 ising unless x = 0 is a stabil be not be optimal if x is not zero, and will not even stable equilibrium of the uncontrolled process. ge-cost optimal' but not This is an extreme example of a policy which is 'avera (as in Section 10.1, for N > 0) optimal. If non-degenerate plant noise is present st optimal. This is the infinitethen there is only one policy which is average-co rary initial conditions, and so horizon optimal policy, which is optimal for arbit also copes optimally with transients.

=

DENSITY MATRIX 2 THE ACTION OF FILTERS: THE SPECTRAL is as natural in this context as it The use of z-transforms and Fourier transforms se of the translation-invariant was in Chap ter 4, and for the same reason: becau ete-time case and define the character of the process. Consider again the discr ) autocovariance generating function (abbreviated AGF 00

g(z)

=

L

v,t',

(6)

r=-oo

x n matrix whose elements are where z is a complex scalar. The AGF is then ann 1 z- ) T, a relation which we shall functions of z. Property (5) implies that g( z) = g( ons 6.3 and 6.5. It follows then write as g = g, consistent with the usage of Secti

258

STATIONARY PROCESSES; SPECTRAL THEORY

that, if the doubly infinite series (6) converges at a given value of z, then it also converges at z- 1• In particular, if v, decays as p' with increasing r, for some scalar pin [0, 1), then g(z) converges in the annulus p < lzl < p- 1. Note that a white noise process of covariance matrix Vhas a constant AGF: g(z) = V. The AGF may well not converge anywhere. The standard example is the simple sinusoid x 1 = sin(w0 t - '1/J), which defines a stationary process if the phase 'lj; is assumed to be a random variable uniformly distributed over (-1r, 1r). Its autocovariance function is v, = cos(wor), so that the series (6) indeed converges nowhere. A regular class of stationary process is constituted by those for which x is the output of a stable system obeying finite-order linear dynamics and driven by white noise. We shall see below that g(z) is then rational and necessarily convergent in an annulus p < lzl < p- 1• The important property of the AGF is that it transforms very much more pleasantly under the action of a filter on the process than does the autocovariance function itself. Suppose that a process {y1} is derived as the output of a translation-invariant linear filter with stationary input {x 1}:

!

(7) Here we have used the notation of Chapter 4, so that b, is the transient response of the filter and B(z) = Ls b,z' its generating function; the transfer function of a discrete-time filter. If the sum in (7) is convergent (the appropriate sense here being that of convergence in mean square) then the output {yt} is defined and also stationary. Since we are dealing with more than one process then autocovariances etc. must be labelled by the process to which they refer. Let us denote the autocovariance and the AG F for the process {x 1 } by v~x) and g(x) ( z), etc. Theorem 13.2.1 Suppose that the functions g(x) (z) and B(z) are both analytic in an annulus p < lzl < p- 1• Then the sum (7) is mean-square convergent, the AGFg(yl(z) ofthe output is analytic in the same annulus and output statistics are related to input statistics by Vr(y) --

" "" " b·Jvr-j-j+k (x) bTk• L..L..j

(8)

k

(9) Proof Relations (8) and (9) are those which would be established by formal argument: assertion (8) follows from (7) and the definition of an autocovariance; assertion (9) then follows by the formation ofgenerating functions. Convergence of the sum (7) and validity of these manipulations is amply justified by our

DENSITY MATRIX 2 THE ACTION OF FILTERS: THE SPECTRAL

259

erge to zero as plrJ with incre as· very strong assumptions: that v~x) and b, conv 0 ing lrl. strong, but make for simplicity, The assumptions of the theorem are excessively h largely covers our needs. Som e and are justified in the rational case whic tantial relaxation takes one deep relaxation is indicated in Exercise 2, but subs into Fourier theory. form ation rule (8) for the The poin t of the theo rem is that the trans ly pleasing, but the rule (9) for the autocovariances themselves is not particular suggestive. We shall often write it transformation of AGF s is both comp act and simply as g(y) = Bg(x) lJ. a mod el already cons idere d in A prim e example is a statistical version of Section 4.4: A(ff )x = t. y= Cx, noise of covariance V, so that the Here A (ff) is polynomial in ff and t is white rated by a stochastic difference outp ut y is a linear function of a variable x gene have all its zeros strictly outside the equation. Stability requires that lA (z) Ishould esses converge to stationarity with unit circle. Und er these circumstances the proc 1 g(yl(z) = CA(z )- 1 V A(z f cT.

a neig hbou rhoo d of the unit circle. This is certainly rational, and free of poles in , if, as in Section 4.2, we see the The AGF can be given a more physical basis ed define power series (6) as a Four ier series. Let us inde

:2: v,e-irw, 00

f(w)

= g(e-iw) =

(10)

r=-oo

ency'. The transformation (10) with where the variable w is to be regarded as 'frequ unit circle, where we expect it to w real amounts to considering g(z) on the converge if it converges anywhere. Suppose x scalar, for simplicity. We have h-l

0 ~ h-1 Ell : Xte-itwl2 = h-1 t=O

h

h-1 h-1

:2: :2: j=O k=O

Vj-kei(k-j)w =

L (I -

lr!/h)v,e-irw.

r=-h

with increasing h if regularity This last expression will converge to f(w) is that 2::, lv,! < oo. We see then conditions are satisfied; a sufficient condition positive and can be identified with that, unde r such cond ition s,f(w ) is real and trans form of the sequence {x1 }, in the expected squa red mod ulus of the Fourier sense. This argu ment leads to an an appropriately norm alise d and limiting

260

STATIONARY PROCESSES; SPECTR AL THEORY

interpretation off(w) as the density of the expected distribution of 'power' in the process {x 1 } over the frequency spectrum. For this reason it is termed the spectral density function in the scalar case, and the spectral density matrix in the vector case. For the sinusoidal process sin(wot -1{;) mentioned above the Fourier series (10) of course does not converge, but limiting arguments evaluate it as f (w) = [8(w- wo) + 8(w + wo)]. That is, the energy of the process is indicated as being concentrated in the frequencies ±w0 , which is indeed consistent with the nature of the process. Relation (9) becomes, in terms of spectral densities,

!

J(Y) (w)

= f3(w)f(x) (w)f3( -w) T = f3f(x) ,8,

(11)

where f3(w) = L,, b,e-irw is. the frequency response function of the ftlter (see Section 4.2). This could be regarded as the consequence E[(CY>(CY>J = f3E[((x)((x)],8 of a relation (CY>(w) = f3(w)((x)(w), where ((x)(w) is the (random) Fourier amplitude at frequency w in a Fourier decomposition of x into 1 frequencies and ((xl(w) its transposed complex conjugate. In fact, such a decomposition is a delicate matter and one must be circumspect in speaking of a 'Fourier amplitude', but the view is nevertheless a correct one; see Exercise 1. Note that we expect the relation Vr

= -21 11"

1'/f ei""f(w) dw,

{12)

-'If

inverse to (10). 'The continuous-time results are formally analogous, with sums overt replaced by integrals over the whole time axis and integrals over w replaced by integrals over the infinite frequency axis ( -oo, oo). Thus, the mutually inverse pair of relations (10), (12) is replaced by

v(r)

11

= -2

11"

00

eiWTf(w) dw.

-oo

These relations are certainly valid if/(w) exists and is integrable. If we conside r the importa nt special case (in continuous time) of rational f(w), thenf(w ) can certainly have no singularity on the real w-axis, so integrability cannot fail for this reason. However, integrability over the infinite w-axis will also require thatf{w ) should tend to zero with increasing w, and should do so sufficiently quickly. The effect of this condition is to exclude a white noise component, since white noise has a constan t spectral density. If f(w) is rational and tends to zero with increasing w then it must indeed tend to zero at least as w- 2, and so is integrable. The ftlter x--+ y expressed by y = B(!'i)x has transfer function B(s) and frequency response function f3(w) = B(iw). With these understandings, relation (11) holds in regular cases.

I 4

.~

~ ~

i

1 ::;.

ESSIVE REPRESENTATIONS 3 MOVING-AVERAGE AND AUTOREGR

261

the cons tant value V (where Vis the . For a white noise process the SDF f( w) has as to whether the proc ess is in (;()variance matr ix or powe r matr ix, according as an indication that the ener gy discrete or continuous time). One interprets this rang e of meaningful frequencies of the process is uniformly distributed over the cont inuo us time). The proc ess was (i.e. (-11", 11") in discrete time and ( -oo, oo) in belie f that white light has such a then given the nam e 'white' noise in the mist aken even over the visible range, but uniform ener gy spectrum. It certainly does not, the term 'white noise' has stuck. Exercises and comments

onar y process can be unde rstoo d (1) Circulant processes. The struc ture of a stati by assu ming it periodic, with perio d much bette r if one makes the time axis finite of the sequence {xo, x 1 , x2, ... , m. That is, the complete realisation X consists cyclic perm utati on, so that .rx = Xm- 1}, and the shift oper ator ff effects a just means statistical inva rianc e {xm-I,xo,Xt, ... ,Xm-2}· 'Stationarity' then on. Assume stationarity in what unde r .r, and so unde r any cyclic perm utati follows. e with perio d m. Suppose, for Show that cov(xr, x,_,) is a function v, of r alon e matr ix of the m-vector X. rianc simplicity, that x is scalar, and let V be the cova the kth element of the that and Show that V has eigenvalues jj = }:, v,e-ir ) and sums run over 11"i/m exp(2 = corresponding right eigenvector is ()ik, where() a 'spectral density' is value eigen any m consecutive integers. That is, the jth ~ (j is the finite wher El([1 with evaluated at w = 21f'j/m, and can be identified 1 • Furth ermo re, 2 1 0-1 x ~ 1 1 1 (j = mFour ier trans form of the sequence X in that disti nctj, kin for 0 = k) that E((j( these Four ier amplitudes are uncorrelated, in thes etO, l, .. . ,m -1. b,.r ' is to multiply (j by B(()i). The effect of a filter with oper ator B(.r ) = }:, n by white noise, and so with outp ut (2) Cons ider a discrete-time SISO filter drive that this expression should }:, b,e,_,. The necessary and sufficient cond ition ld have finite seco nd-o rder shou ut converge in mean square (and so that the outp therefore sufficient if one is ition moments) is that }:, is fmite. The same cond condition will also be The e. abov considers an inpu t whose SDF is boun ded zero. away from nece ssary if the SDF of the inpu t is boun ded

b;

SSIV E REP RES ENT ATIO NS: 3 MOVING-AVERAGE AND AUT ORE GRE CAN ONI CAL FACTORISATIONS ut of a filter with a white noise Cons ider the process {x 1} obta ined as the outp input:

(13)

262

STATIONARY PROCESSES; SPECTRAL THEORY

In the time-series literature such a process would be referred to as a moving average process, the words 'average' and 'moving' being a loose indication of the properties 'linear' and 'translation-invariant~ If the white-noise process has covariance matrix V then it follows, as a special case of (9), that {x,} has AGF

g(z) = B(z)VB(z)

(14)

If the filter is causal then b, is zero for r negative, so that B(z) is a series in nonnegative powers of z. One then speaks of the moving average relation itself as being 'causal' or 'realisable: Suppose, as in Section 4.4, that the filter is specified as an input-drive n linear difference equation 00

A(ff)x, = L:a,x,_, =

Et

(15)

r=O

We regard (15) as a relation determinin g x 1 in terms of €1 and past x-values, so that a, is indeed zero for r negative and a0 is non-singular. If the relation (15) is stable

then the output {x 1} generated is stationary, and can indeed be represented as a realisable moving average of the input. That is, it can be represente d in the form (13) with B(z) = A(z)- 1, where these expressions are understoo d as power series in non-negative powers of z (see Section 4.4~ Its AGF is then

(16) In the time-series literature a process generated by a relation such as (15) is termed an autoregressive process, the relation itself being an autoregression. The term 'conveys, again loosely, the idea that the variable is linearly dependent upon its own past. The relation (15) is of course just a stochastic difference equation, linear and with constant coefficients, and of order p if a, is zero for r > p. Let us now reverse these deductions. If the AGF has the form (14) then the process could have been generated by a moving average relation (13); if the AGF has the form (16) then the process could have been generated by an autoregressive . relation (15). In other words, one would have, respectively, moving average and autoregressive representations of the process. The point of this manoeuvr e will appear in the next section. Let us make it more explicit.

Theorem 113.1 Suppose that the AGF g(z) ofa process x 1 can befactorised in the form Q4), where both B(z) and B(z)- 1 are analytic in lzl ~ 1. Then the process has a causal moving average representation QJ), with V identified as the covariance matrix ofthe white noise process {E1}. Furthermore, this representation is invertible, in that it can be inverted to the autoregressive relation Q5), with A(z) = B(zr 1• Proof Define a process { E1} by Et

= B(ff)- 1x 1 = A(ff)x1,

(17)

I

.d

ONS 3 MOVING-AVERAGE AND AUTOREGRESSIVE REPRESENTATI

263

of§. It where it is unders tood that the expansions are in non-negative powers fined, so follows from the assumptions of the theorem that this quantity is well-de (15) hold that the process { Er} is stationary and both the relations (13) and that (9) relation to appeal by now see We es. between the two process g(E)

= AgA = B- 1glr 1 = V,

so that the process {€ 1} is white, with covariance matrix V.

D

ation of The factorisation (14) deman ded in the theorem is a canonical factoris we look, essentially, for a g~). This type of factorisation will occur repeatedly as . The factorisation process the of ntation represe ressive moving-average or autoreg pty annulu s non-em a in c analyti is and AGF an is is indeed possible if g(z) p < lzl < p-1. with an We can restate the theorem in what is a completely equivalent form, but ant. signific prove will emphasis whose difference its inverse as Theorem 13.3.2 Suppose that the AGF g(z) ofa process x 1 is such that form a matrix can befactorised in the

(18) has a stable where both A(z) and A(zr 1 are analytic in izi ~ 1. Then the process of the matrix nce covaria the as ed identifi V autoregressive representation (15), with white noise process { € 1}.

. Note, This is indeed just a variation of the stateme nt of the previous theorem 1 rather g(zr of ation factoris cal however, that it is expressed in terms of a canoni factor than of g(z), and that the order of factors is reversed, in that it is the final inside ies propert ity regular have rather than the initial factor which is required to the unit circle. ion Of course, one factorisation immediately implies the other, but the distinct state to comes it when , between the two versions will prove significant. Roughly equivalent estimation (as it will) then factorisations (14) and (18) are respectively square least linear a as to the two characterisations of a projection estimate es process with deal estimate or as a minimu m discrepancy estimate. When we a is (18) ation whose dynamics are of finite order (as we shall) then factoris polynomial factorisation, whereas (14) is not. ing The € 1 defintJd by (17) are in fact just the innovations of the x-process (assum e -averag this to have begun in the infinite past) and the realisable moving own its of representation (13) is just a representation of the process in terms (1938). The innovations. It is a special case of a representation deduced by Wold above (the Wold representation breaks x into the moving-average term deduced ed non-deterministic component) and a term which can be perfectly predict purely

STATIONARY PROCESSES; SPECTRAL THEORY

264

by linear relations. Our factorisation hypotheses exclude the possibility of this second, deterministic, component.

4 THE WIENER FILTER Suppose that one has observed a process {x 1} up to timet, and so knows its partial history X 1 = {Xr; T ::;;;; t}. Suppose that one wishes to use the observations to estimate the value of some unknown random variable f It is natural to consider the projection estimate ~(r) = 8(~JX1 ), which we know, from the last chapter, to have optimality properties. We suppose all variables reduced to zero mean. Theorem 13.4.1 Suppose that the AGFofthe process {x1 } has canonicalfactorisation (13). Then the projection estimate of~ in terms of X 1 is given by 00

c(t)-"'

<,

-

(19)

L....- "frXt-n r=O

where

(20) and

(21)

Proof We can equivalently project~ upon the to-history, so obtaining 00

~(t)

00

00

= LG(~Jt:r-r) = L~~;,V- 1 Et-r = L~~;,.'T'V- 1 B(.r)- 1 xr r=O

r=O

r=O

whence (19) follows. The ftrst equality is a consequence of the orthogonality of thet:,. D Essentially, one obtains the estimate by projecting upon x-innovations and then expressing these innovations in terms ofobserved x. The filter (19) which determines ~(r) from X 1 is the Wiener filter. It is generally stated for the particular case ~ = Xr+m, when one is attempting to predict what the value of the process variable will be m steps in the future. Exercises and comments (1) Show that, for the prediction problem last indicated, solution (20) for the "{coefficients becomes E~ 'Yrz' = [z-m B(z)]+B(z)- 1 •

5 OPTIMISATION CRITER IA IN THE STEADY STATE

265

one could .· (2) Denote the predictor of Xr+m thus determined by x~2m Show that relations the of tion applica by ors predict these ted have e~uivalently have calcula relation, ressive autoreg the to appeal d repeate by is, That 'l:r a,x/l., = 0 (T > t). of the with future E set equal to zero. This is indeed an immediate consequence oise fact that tS'(ETIXr) = 0 forT> t, which is itself a consequence of the white-n assumptions. 5 OPTIMISATION CRITERIA IN THE STEADY STATE spectra l Optimisation criteria will often have a compact expression in terms of best one r Whethe densities in the stationary regime (i.e. the steady-state limit). these of sation goes about steady-state optimisation by a direct minimi ions are expressions with respect to policy is another matter, but the express certainly interesting and have operational significance. n Consider first the LQG criterion. Let us write an instantaneous cost functio such as (2.23) in the 'system' form

(22) , for Here ~ is the vector of 'deviations' which one wishes to penalise (having matrix example, x - r and u - uc as components) and 9t is the associated has auto(which would be just [ ~ ~] in the case (2.23) ). Suppose that { ~t} a given covariance generating function g(z) in the stationary. regime under is to aim the Then f(w). density l spectra stabilising policy 11; and corresponding choose 1r to minimise the average cost (23) tr[9t cov(~)]. !E{tr[9t~~T]} 'Y E(!~T!Jt~]

=!

=

=

and tr(P) Here E denotes an expectation under the stationary regime for policy 1r have not we n notatio of y econom For P matrix the denotes, as ever, the trace of to the ing Appeal 1r. upon g(z) and E "f, of ence explicitly indicated the depend formula cov(~) = 217r f(w) dw,

j

we thus deduce

Theorem 13.5.1 The criterion (23) can be expressed 'Y

= E(!~T9\A] =

4~

J

tr[9lf(w)J dw

(24)

ry where f (w) is the spectral density function of the A-process under the stationa the in 1r] [-1r, interval regime for policy 11; and the integral is is over the rea/frequency case ofdiscrete time and over the whole rea/line in the case ofcontinuous time.

266

STATIONARY PROCESSES; SPECTRA L THEORY

In the discrete-time case it is sometimes useful to see expression (24) in power series rather than Fourier terms, and so to write it

1 7 = !Abs{tr[9tg(z)]} = -47rl.

dz j tr[9tg(z)]-. z

(25)

Here the symbol ~s· denotes the operation of extracting the absolute term in the expansion of the bracketed term in powers of z upon the unit circle, and the integral is taken around the unit circle in an anticlockwise direction. If we had considere d a cost function h-1

C=z=c,

(26)

t=O

up to a horizon h then we could have regarded the average cost (23) as being characterised by the asymptotic relation E(C) = h7 + o(h)

{27)

for large h. Here E is again the expectation operator under policy 1r, but now conditional on the specified initial conditions. The o(h) term reflects the effect of these conditions; it will be zero if the stationary regime has already been reached at timet= 0. Consider now the LEQG criterion introduce d in Section 12.3. We saw already from that section that the LEQG model provided a natural embedding for the LQG model; we shall see in Chapters 16, 17 and 21 that it plays an increasing role as we bring in the concepts of risk-sensitivity, the H 00 criterion and largedeviation evaluations. For this criterion we would expect a relation analogous to(27):

(28) Here 7( 9) is a type of geometric-average cost, depending on both the policy 1r and _ the risk-sensitivity paramete r 9. The least ambitious aim of LEQG-optimisation in the infinite-horizon limit would be to choose the policy 1r to minimise 7(6). ('Least ambitious', because a full-dress dynamic program ming approach would minimise transient costs as well as average costsJ We aim then to derive an expression for 7( 9) analogous to (24). Theorem 13.5.2

The average cost 7( 9) de]med by (48) has the evaluation

!j

7(6) = 4 9

logjl + 99tf(w)l dw

{29)

for values of(} such that the symmetrisation of the matrix I+ 09tf(w) is positive definite for all real w. Here f (w) is the spectral density function of the fl.-process under the stationary regime for the specified policy 1r (assumed stationary and

5 OPTIMISATION CRITERIA IN THE STEADY STATE

267

linear), and the integral is again over the interval [-1r, 1r] or the whole real axis in discrete or continuous time respectively. Here /P/ denotes the determin ant of a matrix P; note that expression (29) indeed reduces to (24) in the limit(} --+ 0. We shall prove an intermediate lemma for the discrete-time case before proving the theorem. Suppose that the AGF of { 6-c} has a canonica l factorisation (16) with A(z) analytic in some annulus /zl ~ 1 + 6 for positive 8. That is, the 6.process has an autoregressive representation. It is also Gaussian, since the policy is linear. Let us further suppose this representation so normalis ed that Ao =I. The probability density of 6. 1 conditional on past values is then

f(6.c/6.r;T < t) = [(27r)m/V/]- 112 exp[-!ci- "V- 1E1] where Ec

= A(Y)6.c and m is the dimension of 6.. This implies that

f(6.o, 6.,' ... '6-h-1/6-r; 'T < 0) = [(27r)ml VIJ-h/Zexp [ = [(27r)'/ VIJ-h/ 2 exp [

-! ~ E; v-'fc]

l

-! ~ 6-'f M(ff)Ac + o(h)

(30)

where M(z) = A(z) v- 1A(z) = g(zf 1 . Since the multivariate density (30) integrates to unity, and since the normalis ation Ao = I implies the relation log/ V/ = dlogjg(z )/, we see that we can write conclusion (30) as follows.

Lemma 13.5.3 Suppose that M(z) is self-conjugate, positive definite on the unit circle and analytic in a neighbourhood ofthe unit circle. Then

II··· Iexr[-l~x;'M(ff)x}xodXI· .dxo-1

(31)

= (27r)hmf2exp{ -(h/2)Ab s[log/M( z)/J + o(h)}. *Proof of Theorem 13.5.2 The expectation in (28) can be expressed as the ratio

multivariate integrals of type (31), with the identifications M(z) = g(z) -I + (}91 and M(z) = g(z)- 1 in numerato r and denomin ator respectively. We thus have

of two

E(e-ec) = exp[-(h/2 )Abs{log jg(z)- 1 + 091/ + log!g(z)i} + o(h)] whence the evaluation

"!(B)

1

= 20 Abs[log/I + 091g(z)l}

.I

268

STATIONARY PROCESSES; SPECTRAL THEORY

and its alternative expression (29) follow. The continuous-time demonstration is analogous. 0 The only reason why we have starred this proof is because the o(h) term in the exponent of the final expression (30) is in fact Ll-dependent, so one should go into more detail to justify the passage to the assertion (31).

r' '

~''~' ,. .· : r '

!

CHA PTE R14

Optimal Allocation; The Multi-armed Bandit 1 AU.OCATION AND CONTROL topic of conThis chapter is somewhat off the principal theme, although on a he wishes. if it bypass may reader the and siderable importance in its own right, resources limited of g sharin the Allocation problems are concerned with problem of kind the not This d. pursue between various activities which are being m if proble l contro a indeed is envisaged in classical control theory, but of course le, examp For ons. conditi ng this allocation is being varied in time to meet changi rk netwo ons unicati comm a the adaptive allocation of transmission capacity in tance. impor logical techno provides just such an example; clearly of fundamental henceforth The classic dynamic allocation problem is the 'multi-armed bandit', ce of sequen a makes er referred to as MAB. This is the situation in which a gambl the choose to plays of any of n gambling machines (the 'bandits'1 and wishes f pay-of ed expect machine which he plays at each stage so as to maximise the total the of ility probab (perhaps discounted, in the infinite-horizon case). The pay-off er, the ith machine is a parameter, 0; say, whose value is unknown. Howev he as gains gambler builds up an estimate of Oi which becomes ever more exact n playing a more experience of the machine. The conflict, then, is betwee g with a machine which is known to have a good value of 0; and experimentin better. It is machine about which little is known, but which just might prove even m as an proble the ates formul one that t in order to resolve this conflic optimisation problem. features As an allocation problem this is quite special, on the one hand, but has ing is allocat is one which ce which confuse the issue, on the other. The resour at a ne machi one only play one's time (or, equivalently, effort) in that one can the split to able be will one s time, and must decide which. In more general model but ting fascina is allocation at a given time. The problem also has a feature which time is not is irrelevant in the first instance: the 'state' of the ith machine at a given of 0;, an value its physical state, but is the state of one's information about the chapter, next 'informational state~ We shall see how to handle this concept in the which is to but this aspect should not divert one from the essential problem, we may decide which machine to use next on the basis of something which indeed term the current 'state' of each machine.

270

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

In order to sideline such irrelevancies it is useful to formulate the MAB problem in greater generality. We shall suppose that one has n 'projects', the ith of which has state value xi. The current state of all projects is assumed known. One can engage only one project at a time; if one engages project i then xi changes to a value ~ by time-homog eneous Markov rules (i.e. the distribution of ~ conditional on all previous state values of all projects is in fact dependent only on xi), the states of unengaged projects do not change and there is a reward whose expectation is a function r;(x;) of x; and i. If one envisages indefinite operation starting from time t = 0 and discounts rewards by a factor f3 then the aim is to choose policy 1r so as to maximise E'lr[L::o {3 1R 1], where R 1 is the reward received at time t. The policy is the rule by which one decides which project to engage at any given time. One can generalise even this formulation so as to make it more realistic in several directions, as we shall see. However, the problem as stated is the best first formalisation, and captures the essential elements of a dynamic allocation problem. The problem in this guise proved frustratingly difficult, and resisted sustained attack from the 'forties to the 'seventies. However, it had in fact been solved by Gittins about 1970; his solution became generally known about 1981, when it opened up wide practical and conceptual horizons. Gittins' solution is simple to a degree which is found amazing by anyone who knew the frustrations of earlier work on the topic. One important feature which emerges is that the optimal policy is an index policy. That is, one can attach a index lli(xi) to the ith project which is a function of the project label i and the current state x; of the project alone. If the index is appropriatel y calculated (the Gittins index), then the optimal policy is simply to choose a project of currently greatest index at each stage. Furthermore , the Gittins index II; is determined by the statistical properties of project i alone. We shall describe this determinatio n, both simple and subtle, in the next section. The MAB formulation must be generalised if one is to approach a problem as complicated as, for example, the routing of telephone traffic through a network of exchanges. One must allow several types of resource; these must be capable of allocation over more than one 'project' at a time; projects which are unengaged may nevertheless be changing state; projects may indeed interact. We shall sketch one direction of generalisatio n in Sections 5-7. The Gittins solution of the MAB problem stands as the exact solution of a 'pure' problem. The inevitable next stage in the analysis is to see how this exact solution of an idealised problem implies a solution, necessarily optimal only in some asymptotic sense, for a large and complex system.

2 THE GITTINS INDEX The Gittins index is defined as follows. Consider the situation in which one has only two alternative actions: either to operate project i or to stop operation and

271

2 THE GITTINS INDE X

then in fact an optim al stopp ing receive a 'retir emen t' reward of M. One has proje ct, its state will not chan ge probl em (since once one ceases to opera te the te the value function for this and there is no reaso n to resum e operation). Deno on the retire ment reward as well probl em by ¢i (Xi, M), to make the depe nden ce dyna mic progr amm ing equa tion as on the proje ct state explicit. This will obey the

(1) a relati on whic h we shall abbreviate to

(2)

of proje ct i before and after one Here Xi and~ are, as above, the values of the state stage of opera tion. the form indic ated in Figu re As a funct ion of M for fixed Xi the funct ion ¢i has for M great er than a critic al value 1: non-d ecrea sing and convex, and equa l to M accep t the retire ment rewa rd M;(xi)· This is the range in whic h it is optim al to over value, at which M is just large rathe r than to continue, and M; (x;) is the cross ng are equally attractive. enou gh that the optio ns of conti nuing or term inati the proje ct when in state xi; it is Note that Mi(xi) is not the fair buy-o ut price for (in state Xi) if an offer is made more subtle than that. It is the price whic h is fair ct opera tor is free to acce pt at which is to rema in open, and so which the proje can be taken as the Gittins index , any time in the future. It is this quantity whic h altho ugh usual ly one scales it to take

(3)

ity that a capit al sum M woul d as the index. One can regard vas the size of an annu choic e betw een the altern ative s of buy, so that one is rephr asing the probl em as a n or of moving to the 'certa in' eithe r opera ting the proje ct with its unce rtain retur proje ct whic h offers a cons tant incom e of v.

M. (x;)

M

X; and Mare respectively the state of Figure 1 The graph of r/J;(x;, M) as afunction ofM Here and
272

OPTI MAL ALLOCATION; THE MULTI-ARMED BANDIT

Note that the index v;(x;) is indeed evaluated in term s of the properties ofproject i alone. The solution of then- proje ct probl em is phra sed in terms of these ; indices: the optim al policy (the Gittins index policy ) is to choose at each stage one of the projects of currently greatest index. One may indee d regard this as a reduction of the problem which is so powerful that one can term it a solution, since solution of the n-project problem has been reduced to solution of a stopping problem for individual projects. .~ We shall prove optimality of this policy in the next section, but let us note now :$ an associated assertion. Let x denote the comp osite state (x1, x2, ... , X 11 ) of the '~; set of all n projects, and let (x, M) denote the value function for the problem of choosing optimally between any one of these proje cts and the additional option of total retirement with a term inal reward M. Let A(l {3) and B(l - {3) be uniform lower and uppe r boun ds (presumed finite ) on the reward rates r;(x;), so that A and B are lower and uppe r boun ds on the total expected discounted reward obtainable if there is no retirement optio n. Then we shall show that (x, M) has the evaluation in term s of the one-p roject value functions c/Ji

;>.0

(4) Solution of this augmented problem for gener al M implies solution of the nproject problem without the retirement option, because if M < A then one will never accept the retirement option. Exercises and comments (1) Prove that ¢;(x;, M) has indee d the chara cter asser ted in the text and illustrated in Figure 1.

3 OPTIMALITY OF THE GITTINS INDEX POL ICY The value function ci> (x, M) will obey the dynamic prog ramm ing equation = max[M,m~xL;] l

(5)

where the opera tor L; defined implicitly by comp arison of (1) and (2) will act only on the x;-ar gume nt of . We shall prove valid ity of (4) and optimality of the Gittins policy by demonstrating that expression (4) is the unique solution of (5) and that the Gittins policy corresponds to the maxi mising options in (5). More explicitly, that one operates a project of maxi mal index v; if this exceeds M(l - {3) and otherwise accepts the retirement option. Many other proofs of optimality have now been offered in the literature which do not depe nd upon dynamic prog ramm ing ideas; one parti cular line is indicated in the exercises

3 OPTIMALITY OF THE GITTINS INDEX POLICY

273

not yield the extra below. However, these are no shorter, most of them do are more insightful. conclusion (4), and it is a matter of opinion as to whether they

Lemma 14.3.1 Expression (4) may alternatively be written

~(x,M) = ¢i(xi, M)Pi (x,M) + Loo ¢i(xi,m) dmPi(x,m) where

P·( ,x, M) .·=

, M) II 8¢j(Xj ;::.M #i

(6)

{7)

u

is non-negative, non-decreasing in M and equal to unity for

(8) dence upon xwhic h Proof Note that quantities such as Mi and M(i) have a depen l integration. Since ¢i, we have suppressed. Equation (6) follows from (4) by partia M for M ~ Mi. then as a function of M, is non-decreasing, convex and equal to for M ~ M;. The unity to equal and 8¢;/ aM is non-negative, non-decreasing 0 properties asserted for Pi thus follow. Consider the quantity

8i(Xi,M) = ¢;(xi ,M)- Li
with equality if M

~ m~xMi, J

\I>(M) - L;\I>(M) with equality if M;

= max Mi

M

(9)

and

= 8;(M)P;(M) + Loo 8;(m)dmPi(m) ~ 0 ~

(10)

M.

J

in m. Here dmPi(m) is the increment in P; (m) for an increment dm ty case follow from Proof Inequality (9) and the characterisations of the equali (6) and the properties of Pi. non-negativity of The first relation of (10) follows immediately from (6). The the non-negative and the expression follows from the non-negativity of 8; and

274

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

non-decreasing nature of P;. We know that 8;(M) = 0 for M ~ M; and that dmP;(m) = 0 form~ M(i)• so that expression (10) will be zero if M ~ M; and M(i) ~ M;. This pair of conditions is equivalent to those asserted in the lemma. 0 Theorem 14.3.3 The value function 4>(x, M) of the augmented problem indeed has the evaluation (4) and the Gittins index policy is optimal. Proof The assertions of Lemma 14.3.2 show both that expression (4) satisfies the dynamic programming equation (2) and that the Gittins index policy (augmented by the recommendatio n of termination if M exceeds max;M;) provides the maximising option in (2). But since (2) has a unique solution and the maximising option indicates the optimal action (see Exercise 3.1.1, valid also for the stochastic case) both assertions are proved.

Exercises and comments

We indicate an alternative line of argument which explains the form of solution (4) and avoids some of the appeal to dynamic programming ideas.

(1) (Whittle, 1980). Consider a policy which is such that project i is terminated as soon as xi enters a write-off setS; (i = 1, 2, ... , N) and retirement with reward M takes place the moment all projects have been written off. We assume that there is some rule for the choice from the set of projects still in use, which we need not specify. Let us term such a policy a write-offpolicy, and denote the value functions under a given such policy for theN-project and one-project situations by F(x, M) andfi(x;, M) respectively. Then

oF

oM= E(,ffjx)), where r is the (random) time taken to drive all N projects into their write-off sets and r; the time taken to drive project i into its write-off set. But r is distributed as L; T; with the r; taken as independently distributed. Hence it follows that

aF

oM

=IT oM' afi i

which would imply the validity of (4) if it could be asserted that the optimal policy was also a write-off policy. (2) (Tsitsiklis-Weber). Denote the value function for the augmented problem if i is restricted to a set I of projects by V (I). This should also show a dependence on M and the project states which we suppress. Then Vhas the sub modular property

V(I)

+ V(J)

~

V(I U J)

+ V(I n J).

(11)

4 EXAMPLES

275

appe al to the fact that choice of a Prove this by induction on the time-to-go sand be seen as a choice of one proje ct one project from each of I U J and I n J can tion is not in general true. from each of I and J, but that the converse asser cts which are written off (in an (3) (Tsitsiklis, 1986). Take I as the set of all proje and J as its complement. Then optim al policy for the full set of n projects) ci> ~ V(J). But plainly the reverse relation (11) becomes M + V(J) ~ ci> + M, or exceed M if and only if J is noninequality holds, so that ci> = V (J), and this will inde ed aban done d once they are empty. Thus, in an optim al policy, projects are projects are written off. Thus the written off and operation continues until all ssion (4) for the value function is optim al policy is a write-off policy, and expre al to something like the argu men t inde ed correct. However, one still has to appe policy. of the text to establish optimality of the Gittins

4 EXAMPLES tion of the index v(x) for an The problem has been reduced to deter mina i. Dete rmin ation of v(x) index ct individual project, so we can drop the proje and one may well have ct, proje that requires solution of the stopping problem for analytic solut ion is that fact t. The to resort to num erica l methods at this poin fact that the MAB the idate inval possible in only relatively few cases does not lem to the oneprob ect -proj of then prob lem is essentially solved by the reduction ples which in exam some list shall project problem (with a retirement option). We fact perm it rapid and transparent treatment. ¢(x( t), M) is necessarily nonLet us say that a project is deteriorating if mach ine whose state is sufficiently increasing in t. One may, for example, have a deteriorates with age. We leave the indicated by its age, and whose perfo rman ce ing equation (2), that v(x) = r(x) read er to show, from the dynamic prog ramm function. simply, where r( x) is the insta ntan eous reward al policy is a one-step lookIf all projects were deteriorating then the optim h the expected immediate reward ahead policy: one chooses the project i for whic situation in which the ri have ri(xi) is maximal. This will ultimately lead to the one then switches projects to keep been roughly equalised for all projects, and the tyres on one's car with the spare them so. That is, it is as if one kept changing of wear. Switching costs will ensu re so as to keep all five tyres in an equal state that one in fact tolerates a degree of inequality. project, for which ¢(x( t),M ) is The opposite situation is that of an improving perfo rman ce of a machine may non-decreasing with time. For example, the lasts, the more likely it is to be a improve with age in the sense that, the longer it in this case, good one. We leave to the reader to conf irm that,

276

OPTIMA L ALLOCATION; THE MULTI-ARMED BANDIT

That is, the index is the discounted constan t income equivalent of the expected discounted return from indefinite operati on of the project. If all projects are improving then, once one adopts a project, one stays with it. However, mentio n of 'lasting' brings home the possibility that a machin e may indeed 'improve' up to the point where it fails, and is thereafter valueles s. Let us denote this failure state by x = 0, presum e it absorbing and that r(O) = 0. Suppose that the machin e is otherwise improving in that ¢(x(t), M) is nondecreasing in t as long as the state x = 0 is avoided. Let a denote the random failure time, the smallest value oft for which x( t) = 0. In this case

v(x) E['E~:J ,81r(x(t))jx(O) = x] 1 - ,8 = I E[,B"ix(O) = x] If all projects follow this 'improving through life' pattern then, once one adopts a project, one will stay with it until it fails. Anothe r tractable example is provided by a diffusion process in continu ous time. Suppose that the state x of the project takes values on the real line, that the project yields reward at rate r(x) = x while it is being operate d, that reward is discounted at rate a, and that x itself follows a diffusion process with drift and diffusion coefficients J1. and N. This conveys the general idea of a project whose return improves with its 'condition', but whose conditi on varies random ly. The equatio n for ¢(x, M) is then X -

a¢ + Jl.¢x + !N ¢xx = 0

(x > ~)

(12)

where~ is the optima l breakp oint for retirem ent reward M. We find the solution of (12) to be

¢(x,M ) = (xja) + (Jl.Ja 2 ) + cePx

(13)

where pis the negative solution of

! Np2 + Jl.P -

a = 0.

and c is an arbitra ry constant. The general solution of (12) would also contain an exponential term corresp onding to the positive root of this last equatio n, but this will be excluded since ¢ cannot grow faster than linearly with increas ing x. The unknow ns c and ~ are determ ined by the bounda ry conditi ons ¢ = M and ¢x = 0 at x = ~(see Exercise 10.7.2~ If we substitute expression (13) into these two equations then the relation between M and ~ which results is equival ent to M = M(e). We leave it to the reader to verify that the calculation yields the determ ination

v(x) = aM(x) = x+

J1. + ..jJ.L 2 + 2aN

la

.

5 RESTLESS BANDITS

277

reward expected from The constant added to x represents the future discounted quence of the fact future change in x. This is positive even if J1. is negat ive-a conse if this occurs, but that one can take advantage of a random surge against trend can retire if it does not.

5 RESTLESS BANDITS be to allow projects to One desirable relaxation of the basic MAB model would by different rules. For course of gh change state even when not engaged, althou ent used to comb at a treatm al medic a example, one's knowledge of the efficacy of actually deteriorate could it ver, particular infection improves as one uses it. Howe virus causing the the le, examp when one ceased to use the treatment if, for infection were mutating. on of an enemy For a similar example, one's information concerning the positi would actually but it, submarine will in general improve as long as one tracks deliberate taking not deteriorate if one ceased tracking. Even if the vessel were evasive action its path would still not be perfectly predictable. of whom exactly As a final example, suppose that one has a pool of n employees employees who are m are to be set to work at a given time. One can imagine that yees who are resting working produce, but at a decreasing rate as they tire. Emplo is thus changing state do not produce, but recover. The 'project' (the employee) whether or not he is at work. tion or not as We shall speak of the phases when a project is in opera a project was static active and passive phases. For the traditional MAB model is not true: the this ms proble many for in its passive phase. As we have seen, space. For state in ents movem ry contra active and passive phases produce ation inform of loss and gain e induc s submarine surveillance the two phase and tiring to pond corres s phase respectively. For the labour force the two recovery. passive phase as a We shall refer to a project which may change even in the rine example. subma the 'restless bandit', the description being a literal one for er respect: one anoth in l The work-force example generalised the MAB mode one. We shall just than was allowed to engage m of then projects at a time rather se suppo that m ( < n) allow this, so that, for the submarine example, we could a matter of allocating aircraft are available to track the n submarines. It is then of all n submarines the surveillance effort of the m aircraft in order to keep track as well as possible. assume rewards We shall specialise the model in one respect: we shall e reward. This averag the ising maxim of undiscounted, so that the problem is that a number of of ns solutio n know the , makes for a much simpler analysis; indeed As we have case. d ounte undisc the standard problems are greatly simpler in n 16.9, the Sectio in ds groun maintained earlier, and argue on structural t. contex l contro undiscounted case is in general the realistic one in the

278

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

Let us label the phases by k; active and passive phases corresponding to k = 1 and k = 2 respectively. If one could operate project i without constraint then it would yield a maximal average reward "f; determined by the dynamic programming equation "fi

+ fi(x;)

= max{rik(x;) k

+ Ekffi(X:)Ix;]}.

(14)

Here r;k(x;) is the expected instantaneous reward for project i in phase k, Ek is the expectation operator in phase k and fi(x;) is the transient reward for the optimised project. We shall write (14) more compactly as

"(; + Ji = max[Lil Ji,Li2fi]

(i=l,2, ... ,n).

(15)

Let m(t) be the number of projects which are active at time t. We wish to optimise operation under the constraint m(t) = m for prescribed m and identically in t; that exactly m projects should be active at all times. Let Ropt(m) be the optimal average return (from the whole population of n projects) under this constraint. However, a more relaxed demand would be simply to require that (16)

E[m(t)] =m,

where the expectation is the equilibrium expectation for the policy adopted. Essentially, then, we wish to maximise ECL_; r;) subject to E(L_; !;) = n- m. Here r; is the reward yielded by project i (dependent on project state and phase) and l; is the indicator function which takes the value 1 or 0 according as to whether project i is in the passive or the active phase. However, this is a constraint we could take account of by maximising E[L_;(r; + 11/;)], where 11 is a Lagrangian multiplier. We are thus effectively solving the modified version of (15) "f;(v)

+Ji = max[L;,Ji, v + Li2fi]

(i = 1,2, ... ,n)

(17)

wherefi is a functionfi(x;, v) of x; and v. An economist would view v as a 'subsidy for passivity', pitched at just the level (which might be positive or negative) which ensures that m projects are active on average. Note that the subsidy is independent of the project; the constraint (16) is one on total activity, not on individual project activity. We thus see that the effect of relaxing the constraint m(t) = m to the averaged version (16) is to decouple the projects; relation (17) involves project i alone. This is also the point of the Gittins solution of the original MAB problem: that thenproject problem was decomposed into n one-project problems. A negative subsidy would usually be termed a 'tax: We shall use the term 'subsidy' under all circumstances, however, and shall refer to the policy induced by the optimality equations (17) as the subsidy policy. This is a policy optimal under the averaged constraint (16). If we wish to be specific about the value of 11 we shall refer to the policy as the v-subsidy policy. For definiteness, we shall close

•

. 110:.

5 RESTLESS BAND ITS

279

v + L,2 J; then project i is to be the passive set. That is, if x; is such that Ln J; = rested. cts are active on average. The value of v must be chosen so that indeed m proje cts. This induces a mild recoupling of the proje constraint (16) is Theorem 14.5.1 The maximal average reward under R(m) =

i~f [~ 'Y;(v)- v(n- m)]

(18)

and the minimising value ofv is the required subsidy level. ing (that, unde r regularity Proof This is a classic assertion in convex prog ramm problems are equal) but best conditions, the extrema attained in prim al and dual square-bracketed expression in seen directly. The average reward is indee d the acted. Since (18), because the average subsidy paid must be subtr

(where 1r denotes policy) then

a

avl; : 'Y;(v) = E1f I

L l; = (n- m). I

ality condition in (18). The This equation relates m and v and yields the minim is convex increasing in v. v) ( 'Y; condition is indeed one of minimality, because 2: mal average reward for maxi The function R(m) is concave, and represents the 0 any min [0, n].

x; as the value ofv which Define now the index v;(x;) of project iwhe n in state value of subsidy which the is it makes Ln J; = v + L;2 J; in (17). In other words, x;. This is an obvious state in i ct makes the two phases equally attractive for proje passive projects when es d reduc analogue of the Gittins index, to which it indee (which Gittins index the that are static and yield no reward. The interest is the Lagrange as seen is now characterised as a fair 'retirement income') is obviously index The ty. activi multiplier associated with a constraint on average project, the a rest to one e meaningful: the greater the subsidy needed to induc more rewarding must it be to operate that project. m(t) = m rigidly. Then a Suppose now that we wish to enforce the constraint to choose the projects to be plausible policy is the index policy; at all times index (i.e. the first m on a list operated as the m projects of currently greatest te the average return from this ranking projects by decreasing index). Let us deno policy by Rinct(m).

280

OPTIMAL ALLOCATION; THE MULTI-ARMED BANDIT

Theorem 14.5.2

Rind(m) ~ Ropt(m) ~ R(m).

(19)

Proof The first inequality holds because Ropt is by definition the optimal average return under the constrain t m(t) = m. The second holds because R(m) is the optimal average return under the relaxed version of this constraint, D E[m(t)] = m.

The question now is: how close are the inequalities (19), i.e. how close is the index policy to optimality? Suppose we reduce rewards to a per project basis in that we divide through by n. The relation (20) Rind(m)/n ~ Ropt(m)fn ~ R(m)/n then expresses inequalities between rewards (under various policies) averaged over both time and projects. One might conjecture that, if we let m and n tend to infinity in constant ratio and hold the populatio n of projects to some fixed composition, then all the quotients in (20) will have limits and equality will hold throughout in this limit This conjecture has in fact been essentially verified in a very ingenious analysis by Weber and Weiss (1990). However, there are a couple of interesting reservations. Let us say that a project is indexable if the set of values of state for which the project is rested increases from the empty set to the set of all states as v increases. This implies that, if the project is rested for a given value of subsidy, then it is rested for all greater values. It also implies that, if all projects are indexable, then the projects i which are active under a a 11-subsidy policy are just those for which

1/i(Xt) > 1/.

One might think that indexability would hold as a matter of course. It does so in the classic MAB case, but not in this. Counter-examples can be found, although they seem to constitute a small proportio n of the set of all examples. An example given in Whittle (1988) shows how non-indexability can come about. Let D(v) be the set of states for which a given project is rested under the v-subsidy policy. Suppose that a given state (x = {, say) enters D as v increases. It can be that paths starting from {with { in D show long excursions from D before they return. This implies a surrende r of subsidy which can become non-opti mal once v increases through some higher value, when {will leave D. Another point is that asymptotic equality in the second inequality of (20) can fail unless a certain stabilit;y condition is satisfied (explicitly, unless the solution of the deterministic version of the equations governing the distribution of index values in the populatio n under the index policy converges to a unique equilibrium). However, the statistics of the matter are interesting. In an investigation of over 20000 randomly generated test problems Weber and Weiss found that about 90% were indexable, but found no counterexamples to average-optimality (i.e. of

281

NTE NAN CE 6 AN EXAMPLE: MAC HIN E MAI

of s). In searching a more specific set instability of the dynamic equation 3 for and , 10in fewer than one case in examples they found counterexamples 5 the orde r of one part in 1o- . of these the mar gin of suboptimality was for average-optimal is then virtually true The assertion that the index policy is dity vali lute an assertion can escape abso all indexable cases; it is remarkable that on that asymptotic optimality can be cati by so little. The result gives some indi le policies. achieved in large systems by quite simp INT ENA NC E 6 AN EXAMPLE: MA CH INE MA d sidered in Section 11.4 constitutes a goo The machine maintenance problem con is it that s of costs rath er then rewards, so first example. Thi s is phrased in term ; action rath er than a subsidy for inaction now natu ral to thin k of v as a cost for 11.4, tion overhaul. In the notation of Sec i.e. to identify it as the cost of a machine atio n for a single machine analogous to equ the dynamic programming equation (17) is then (21) .X[f(x + 1)- f(x) ]. 'Y = min{v +ex + f(O )- f(x) , ex+ ice ecture that the optimal policy is to serv If we norm alis ef(O ) to zero and conj s tion equa the value ~then (21) implies the machine if x ;;::: ~for some critical 'Y + f(x) = v +ex 'Y = ex+ .X[f(x + 1)- f(x) ]

These have solution

(x;;:::

f(x )=c x+ v-' Y f(x) =

f::(ij=O

~)

ex)/ .X= "fX - cx( x- 1) 2.X .A

The identity of the two solutions at x for 'Y in terms of~

equ = .; thus provides the determining

atio n (22)

~ is effect by replacing~(~- 1) bye _ Now Here we have neglected discreteness with l ima min be ld requirement that 'Y shou determined by optimality; i.e. by the that the derivatives with respect to .; of iring respect to f This is equivalent to requ the be equal (see Exercise 10.7.2), i.e. to the two sides of relation (22) should a uce ded we evaluation of 1' into (22) condition .Xc "''Y - e~. Substituting this on uati eval the v "' v( ~). In this way we fmd relation between v and .; equivalent to

cx2

v(x) "' c(x +.X) + li,

282

BANDIT OPTIMAL ALLOCATION; THE MULTI-ARMED

dominant) term in this accurate to within a discreteness effect. The last (and policy improvement in by expression is essentially the index which was deduced Section 11.4.

7 QUEUEING PROBLEMS vement produ ced plausible Despite the fact that the technique of policy impro in Sections 11.5-11.7, the index policies for a numb er of queueing problems restless bandit technique fails to do so. of deducing an index Consider, for example, the costing of service, in the hope cost structure of Section for the allocation of service between queues. With the to (17) would be 11.5 the dynamic progr amm ing equation corresponding 7 =ex + min{A L\(x + 1), v + AA(x + 1)- J.£(x)L\(x)} , L\(x) is the increment Here v is the postulated cost of employing a server zero according as xis f(x) - f(x- 1) in transient cost and J.£(x) equals 1-£ or to be finite. One fmds (and positive or zero. One must assume that J.£ > A if 7 is non-negative v the active we leave this as an exercise to the reader) that for any r-is the whole set x > 0 of set-t he set in which it is optimal to employ a serve dary which changes with positive x. One thus does not have a decision boun the index was based in of ition defin the changing v; the very feature on which Section 5. considering policies for One can see how this comes about: there is no point in for some { > 0. Such ~ = x which, for example, there is no service in a state ever for the queue on as em policies would present the same optimisation probl ting a base-load of { accep the Set Of States X;:;::: {,but With the feature that one is and so incurring an unnecessary constant cost of c{. es of engagement, so One might argue that variation of J.£ allows varying degre sponding proportional that one might allow Jl to vary with state with a corre same conclusion in the s.ervice cost. However, one reaches essentially the states x > 0 at a comm on undiscounted case: that an optimal policy serves all rate. es manifest if one The special character of such queueing problems becom It is assumed that 5. n Sectio in aged envis considers the large-system limit n ___. oo so that there is unity, than less is the traffic intensity for the whole system this capacity is all case that In sufficient service capacity to cover all queues. s are either queue all that so directed to a queue the mom ent it has customers, This rather e. servic of e empty or (momentarily) have one customer in the cours ited by inhib is nse respo if unrealistic immediacy of response can be avoided only er. transf or vation switching costs or slowed down by delay in either obser problem of Section 11.7. We d rewar pure the for hold not do Such considerations on 6 indeed lead to the leave the reader to confi rm that the methods of Secti known optimal policy.

7 QUEUEING PROBLEMS

Notes on the literature

283

ns had arrived at his solution by The subject has a vast literature. Although Gitti s (1974~ His later book (Gittins, . 1970 it was published first in Gittins and Jone subject. The proo f of optimality e 1989) gives a collected exposition of the whol was given in Whittle (1980). tion associated with solution (4) for the value func their analysis completed and ) (1988 Restless bandits were intro duce d in Whittle by Weber and Weiss (1990).

CHA PTE R IS

Imperfect State Observation ent of the case of We saw in Chapt er 12 that, in the LQG case, there was a treatm By 'imperfect it. imperfect observation which was both complete and explic le variab (or of the state observation' we mean that the current value of the process able. In practice, variable, in state-structured cases) is not completely observ observation will seldom be complete. le; this tractability However, the LQG case is an exception in that it is so tractab 16). There are very few . carries no further than to the LEQG case (see Chapt er d both exactly and models with imperfect observation which can be treate case, which one might explicitly. Let us restrict attention to the state-structured l formal result which centra the Then form. regard as the standard normalised there is still a simply that is ed observ fectly emerges if the state variable is imper ent of the value argum the ver, Howe on. recursive dynamic programming equati ational' state 'inform an but le, variab function is no longer the physical state tional on condi state al physic of value variable: the distribution of the current nt of the eleme an rly forme as not current information. This argument is then, of the ality cardin in se increa state space !!l, but a distribution on !!l. This great lt. argument makes analysis very much more difficu distribution is The simplification of the LQG case is that this conditional covariance and x mean 1 t always Gaussian, and so parametrised by its curren and the alone, ents matrix V,. The value function depends then on these argum s even implie ideas validity of the certainty equivalence principle and associated further simplification. lence principle, Certainly, there is no general analogue of the certainty equiva lty. In general the and it must be said that this fact adds interest as well as difficu gains, so that control choice of control will also affect the information one in mind: to control actions must be chosen with two (usually conflicting) goals on aspects of the the system in the conventional sense and to gain information a strange car is of g' steerin the 'feel to le, system for which it is needed. For examp information sary neces gains one that effective as a long-term control measure (in rly, it is the Simila one. term shorton the car's driving characteristics) but not as a of the basis the on on misati conflict between the considerations of profit-maxi base ation inform this ve information one has and the choice of actions to impro ter. charac ular that gives the original multi-armed bandit problem its partic l. (A 'duality' quite Control with this dual goal is often referred to as dual contro and estimation/ e distinct from the mathematical duality between control/futur

286

IMPERFEC T STATE OBSERVATION

past which has been our constant theme.} An associated concept is that of adaptive control: the system may have parameters which are not merely unobservable but also changing, and procedures must be such as to track these changes as well as possible and adapt the control rule to them. A procedure which effectively estimates an unknown parameter will of course also track a changing parameter. The theory of dual and adaptive control requires a completely new set of ideas; it is subtle, technical and, while extensive, is as yet incomplete. For these reasons we shall simply not attempt any account of it, but shall merely outline the basic formalism and give a single tractable example. 1 SUFFICIE NCY OF THE POSTERI OR DISTRIBU TION OF STATE Let us suppose, for simplicity of notation, that all random variables are discretevalued-th e formal extension of conclusions to more general cases in then obvious. We shall use a naive notation, so that, for example P(x11W1) denotes the probability of a value x 1 of the state at time t conditional on the information W1 available at timet. We are thus not making a notational distinction between a random variable and particular values of that random variable, just as the 'P' in the above expression denotes simply 'probability of' rather than a defined function of the bracketed arguments. We shall use a more explicit functional notation when needed. Let us consider the discrete-time case. The structural axioms of Appendix 2 are taken for granted (and so also their implication: that past controlsparametrising variables -can be unequivocally lumped in with the conditioning variables). We assume the following modified version of the state-structure hypotheses of Section 8.3. (i) Markov dynamics It is assumed that process variable x and observation y have the property P(xt+l•Yt+liXt, Y,, U,)

= P(xt+l,Yt+IIx, u,).

(1)

(ii) Decomposable cost function It is assumed that the cost function separates into a sum of instantaneous and terminal costs, of the form

C=

h-1

L {3' c(x, u, t) + phch(xh).

(2)

1=0

(iii) Information It is assumed that W1 = (Wo, Y, U1_ 1 ) and that the information available at time t = 0 implies a prior distribution of initial state P(xoiWo). It is thus implied in (ill) that y, is the observation that becomes available at time t, when the value of u1 is to be determine d Assumption (i) asserts rather more than Markov structure; it states that, for given control values U, the stochastic

RIBUTION OF STATE 1 SUFF ICIEN CY OF THE POSTERIOR DIST

287

ibuti on of x, + 1 conditional on X 1 process {x1} is auto nom ous in that the distr r words, the causal depe nden ce is and Y1 is in fact depe nden t only on X 1• In othe . This is an assu mpti on which coul d one-way; y depe nds upon x but not conversely ded a disco unt factor in the cost be weakened; see Exercise 1. We have inclu function (2) for econ omy of treat men t variables and observations can Assu mpti ons (i) and (iii) imply that both state stochastic treat ment can be start ed be rega rded as rand om variables, and that the is presc ribed for initial state .xo (the up, in that an initia l distribution P(xol Wo) radical than it seems. For prior distribution). The implication may be more it the physical state variable is in fact example, for the origi nal mult i-arm ed band unkn own success probabilities. The the para mete r vecto r = {Oi}; the vecto r of r usua l assumptions, makes it no fact that this does not change with time, unde poin t which allows one to rega rd less a state variable. However, the change in view l. this para mete r as a rand om variable is non-trivia own plan t param eters are to unkn that ies impl e abov on Generally, the formulati rega rded as rand om variables, not be inclu ded in the state variable and are to be ibution. That is, one takes the directly observable but of know n prio r distr ture. The· controversy amo ng Bayesian poin t of view to inference on struc on and its interpretation has been statisticians conc ernin g the Bayesian form ulati take the prag mati c poin t of view a battle not yet cons igne d to histor)t We shall whic h lead to a natu ral recursive that, in this context, the only formulations the Bayesian formulation and its math emat ical analysis of the prob lem are mini max analogue. We shall refer to the distr ibuti on

e

(3) citly, the poste rior distr ibuti on of as the posterior distr ibuti on of state. More expli ition al upon the infor mati on x 1 at time t, in that it is the distribution of x 1 cond ral forward recursion. that has been gath ered by time t. It obeys a natu distribution) Under the asTheorem 15.1.1 (Bayes upda ting of the post erior bution P 1 obeys the updating sumptions (i)-(iii) listed above the posterior distri formula t+I, Yt+IIxr. u,) (4) ) I:x, P,(x,)P(x P ( I ) . ) ( t+l Xt+! = "' L..Jx, P, Xt P(yt+1 Xt, Ut

Proof We have, for fixed Wr+l and variable x 1+h P(xt+tiWt+t) ex: P(xt+IoYt+tiW,u,) = LP(x,,Xt+IoYt+!oiWr.ut) x,

288

IMPERFECT STATE OBSERVATION

= LP(xrl Wr, Ur)P(Xr+J,Yt+d W1,XI> u1) x,

= LPr(Xr)P(xr+l,Yt+dxr, ur)· X

The last step follows from (3) and the implication of causality: that P(x1 W 1 , u1 ) = P(x1 W1). Normalising this expression for the conditional distribution of x 1+1 we deduce recursion (4). 0 1

1

Just as the generic value of Xr is often denoted simply by x, so shall we often denote the generic value of P1 simply by P. We see from (4) that the updating formula for P can be expressed

P(x)--+ P'(x)

:=

L:z P(z)a (x,y, lz, u) L:x L:z P(z)ar(x,y!z, u)

(S)

1

where a1 (x, ylz, u) is the functional form of the conditional probability

P(xt+l = X,Yt+l = ylx1 = z, u1 = u). Recall now our definition of a sufficient variable ~~ in Section 2.1. Theorem 8.3.1 could be expressed as an assertion that, under the assumptions (i)-(iii) of Section 8.3, the pair (x1 , t) is sufficient, where x 1 is the dynamic state variable. What we shall now demonstrate is that, under the imperfect-observation versions of these assumptions expressed in (i)-(iii) above, it is the pair (P1 , t) which is sufficient, where P1 is the posterior distribution of dynamical state Xt· For this reason Pis sometimes referred to as an 'informational state' variable or a 'hyperstate' variable, to distinguish it from x itself, which still remains the underlying physical state variable. Theorem 15.1.2 (The optimality equation under imperfect state observation) Under the assumptions (i)-(iii) listed above, the variable (P1 , t) is sufficient, and the optimality equation for the minimal expected discountedfuture cost takes the form

F(P, t) = i~f [ L P(x)c(x, u, t) X

+f3LLLP(z) a,(x,y!z,u)F (P',t+ X

y

1)]

(6) (t
Z

where P' is defined by (5) and the terminal condition for system (6) is

F(P, h) = L P(x)Ch(x).

(7)

X

The prooffollows by the now-familiar backward induction; confirmation is left to the reader.

2 MACHINE SERVICING AND CHANGE -POINT DETECTION

289

Recursio n (6) may seem unattractive in view of the 'fractional linear' dependence (5) of P' on P. In fact, this reduces effectively to a linear relation. Recall that a function ¢(P) is homogeneous ofdegree r in P if ¢(>.P) = N ¢(P) for any positive scalar -\.We shall find it sometimes convenient to write F(P, t) as F(( {P(x)}, t) if we wish to indicate how P(x) transform s for a given value ofx.

Theorem 15.1.3 The value function F(P, t) can be consistently extended to unnormalised distributions P by the requirement that it be homogeneous ofdegree one in P, when the dynamic programming equation (6) simplifies to the form F(P, t)

=i~ [;; P(x)c(x, u, t)+{3 ~ F ( { ;;= P(z)at(x,ylz, u)}, t+ 1) ]·

(8)

Proof Recursion (6) would certainly reduce to (8) if F(P, t + 1) had the homogeneity property, and F(P, t) would then share this property. But it is evident from D (7) that F ( P, h) has the property. The conclusio n can be regarded as an indicatio n of the fact that it is only the relative values of P(x11W1) (for varying Xt and fixed W1) which matter, and that the normalis ation factor in (5) is then irrelevant. Exercises and comments (1) An alternative and in some ways more natural formulation is to regard (x1 , y 1) as jointly constituting the physical state variable, but of which only the compone nt Yr is observed. The Markov assumpti on (i) of the text will then be weakened to

P(xt+I,Yt+I!Xt, Yt, Ut) = P(xr+I,Yt+dxt,Yt.Ut) consisten t with the previous assumpti on (i) of Section 8.3. Show that the variable (P 1,y1,t) IS sufficient, where Pt = {P1(xt)} = {P(xt!Wt)} is updated by the formula

Pt+l (xt+d ex

Lx, Pt(Xt)P(xt+l, Yt+dxt, Yt, Ut)·

2 EXAMP LES: MACHI NE SERVIC ING AND CHANG E-POINT DETECT ION One might say that the whole of optimal adaptive control theory is latent in equation (8), if one could only extract it! Even somethin g like optimal statistical commun ication theory would be just a special case. However, we shall confine our ambitions to the simplest problem which is at all amenable.

290

IMPERFECT STATE OBSERVATION

Suppose that the dynamic state variable x represents the state of a machine; suppose that this takes integer values j = 0, 1, 2, . . . . Suppose that the only actions available are to let the machine run (in which case x follows a Markov chain with transition probabilities PJk) or to service it (in which case the machine is brought to state 0). To run the machine for unit time in statej costs ci> to service it costs d. At each stage one derives an observation y on machine state. Suppose, for simplicity, that this is discrete-valued, the probability of an observation y conditional on machine state j being Pi (y). Let P = { P1} denote the current posterior distribution of machine state, and let 'Y and f(P) denote the average and transient cost under an optimal policy (presumed stationary). The dynamic programming equation corresponding to (8) is then

where 8(P) is the unnormalised distribution which assigns the entire probability mass 2:1 P1 to state 0. The hope is, of course, to determine the set of P-va1ues for which the option of servicing is indicated. However, even equation (9) offers no obvious purchase for general solution. The trivial special case is that in which there are no observations at all. Then P is a function purely of the time which has elapsed since the last service, and the optimal policy must be to service at regular intervals. The optimal length of interval is easily determined in principle, without recourse to (9). A case which is still special, but less trivial, is that in which the machine can be in only two states, x = 0 or 1, say. We would interpret these as 'satisfactory' and 'faulty' respectively. In this case the informational state can be expressed in terms of the single number 7f=

P, Po+Pt

0

This is the probability (conditional on current information), that the machine is faulty. Let us suppose that Pot = p = 1 - q and Pto = 0. That is, the fault-free machine can develop a fault with probability p, but the faulty machine cannot spontaneously correct itself. If we setf(P) = ¢(1r) and assume the normalisation <;b(O) = 0 then equation (9) becomes, in this special case,

( 10) Here we have assumed that c0

= 0, and have defined

291

DETECTION 2 MACH INE SERVICING AND CHAN GE-PO INT

p(y) = (1- 1r)qpo(y) + (p + 1rq)p1 (y),

'(y) 7r

= (p + 1rq)pl (y) p(y)

.

(11)

optimal decision will Form ula (11) gives the updating rule 1r --+ n'. The old value, this value being presumably be to service when 1r exceeds some thresh in principle determinable from (10). of change-point detection This two-state model can also represent the problem in which a poten tial that mentioned in Section 8.1. Suppose that state 0 is state 1 that in which it and pollution source (say, a factory or a reactor) is inactive, alarm and take antithe give to is active. One must decide whether to 'service' (i.e. g the alarm costs Givin y s pollution measures) or not on the basis of observation time. unit per c d; delaying the alarm when it should be given costs in the literature: those in Two partic ular cases of this model have been analysed les respectively, with variab al norm which the observations y are Poisson or of determining the em probl The j. parameters dependent upon pollution state s is soluble, and is ption assum these critical threshold value of 1r from (10) unde r tively. respec em, probl er referred to as the Poisson or Gaus sian disord

BEYOND PAR T 3

Risk-Sensitive and H 00 Criteria

r~~' 4'0--<-

~'~~

,-,

CH APT ER1 6

Risk-sensitivity: The LEQG Model 1 UTILITY AND RISK-SENSITIVITY whereas economists work Control optimisation has been posed in terms of cost, shall be invoking some we Since cost. largely in terms of reward, i.e. negative of reward for the terms in ssion discu economic concepts, let us conduct the . lation moment, before reverting to the cost formu enterprise. One then Suppose that IR is the net mone tary reward from some as to maximise IR. so ) policy wishes to cond uct the enterprise (i.e. choose a follow that one sarily neces However, if IRis a rando m variable, then it does not e rather to choos t migh one will wish to maximise E,. (IR) with respect to policy 1r; mono tone ly usual on, maximise E,.[U(IR)], where U is some non-linear functi would be this then 1 e Figur increasing. For example, if Uhad the concave form of if IR were it benef be of less an indication that a given increment in reward would already large than if it were small. lly defined by the fact The function U is termed a utility function, and is virtua ed that E,.[U(IR)J is the that, on axiomatic or behavioural grounds, one has decid choice of a utility function quantity one wishes to maximise. The necessity for outcome, but it would also arises because one is averaging over an uncer tain from a reward which was arise if one wished to characterise the benefit derived distributed over time or over many enterprises. of expected cost E,. (C) to In cost terms, one could generalise the minim isatio n disutility function, again a Lis the minimisation of the criterion E,.[L(C)]. Here

u(R)

e increasing form usually conFigure 1 The graph ofa utility function U(IR) of the concav of return of utility with reward, sidered. The concavity expresses a decreasing marginal rate which induces a risk-averseness on the part ofthe optimiser.

296

RISK-SENSITIVITY: THE LEQC MODEL

Figure 1 Aconvexdisutilityfonction, implying an effective risk-averse attitude orpessimism.

presumably monotone increasing. One gains a feeling for the implications of such a generalisation if one considers the two cases of Figures 2 and 3. In the case of Figure 2 L is supposed convex, so that every successive increment in cost is reganied ever more seriously. In this case Jensen's inequality implies that E[L(C)] ~ L[E(C)] so that, for a given value of E(C), a certain outcome is preferred to an uncertain outcome. That is, an optimiser with a convex disutility function is risk-averse, in that he dislikes uncertainty. The concave disutility function of Figure 3 corresponds to the opposite attitude. In this case Lis supposed concave, so that successive increments in cost are regarded ever less seriously. Jensen's inequality is then reversed, with the implication that an optimiser with a concave disutility function is risk-seeking, in that he positively welcomes uncertainty. In the transition case, when L is linear, the optimiser is risk-neutral in that he is concerned only by the expectation of cost and not by its variability. All other cases correspond to a degree of risk-sensitivity on his part. One can interpret risk-seeking and risk-averse attitudes on the part of the optimiser as manifestations of optimism or pessimism respectively, in that they

Fig•re 3 A concave disutility jUnction, implying an effective risk-seeking attitude or optimism.

f ~::.

.·

1 UTILITY AND RISK-SENSITIVITY

297

to his advantage or disadvantage illlply his belief that uncertainties tend er mor e explicitly in the next section. respectively. This conclusion will emerge rath erts the criterion back on to a cost The attitude to risk is revealed also if one conv scale by defining (1) which certainly exists if L is stric tly Here L -I is the function inverse to L, then mini misa tion of x" is of cour se lllonotonic. If L is monotone increasing disutility. equivalent to minimisation of the expected C has expectation m and a sma ll Suppose now that unde r policy 1r the cost of C- m then leads to the conc lusio n variance v. Expansion of L(C) in powers (und er regularity conditions on that E"[L(C)] = L(m) +!L "(m )v + o(v) ies that differentials and moments). This in turn impl

L"(m) v x"= m+ L'(m )l+o (v).

(2)

negatively in the criterion according as That is, variability is weighted positively or the dis utility function is convex or concave. base d on Jensen's inequality, in that This argu men t is less convincing than that , it is illuminating in othe r respects; it makes unne cess ary assumptions. However see Exercise 1. icula r attention to the exponential Ther e are now good reasons for paying part r, the risk-sensitivity parameter. This disutility function e-/JC, where ()is a paramete tive (when it is mon oton e increasing) function should be minimised for () nega mon oton e decreasing). However, if we and max imis ed for () positive (when it is then we find that all cases are cove red norm alise back on to a cost scale as in (1), by the assertion that the norm alise d crite rion

(3) crite rion reduces in the case () = 0 to should be mini mise d with respect to Jr. The it corr espo nds to increasingly riskthe classic risk-neutral criterion E"(C); s through positive values or decreases seeking or risk-averse attitudes as (}increase that relation (2) now becomes through negative values respectively. Note

x"(O) = m- Ov/2 + o(v).

(4)

attractive for two reasons. (i) The The exponential disutility function is on a scale of optim ism- pess imis m. para mete r () places the optimiser naturally alone, the coefficient of v in (2) is (ii) In this case, and essentially in this case in this case there is an appr oxim ate inde pend ent of m (see Exercise 1). Tha t is, variability of cost. decoupling of the aspects of expectation and are what one migh t term math eHowever, the mor e compelling reasons ons have a way of turn ing out to be mati cal/p ragm atic in character; such reas the exponential criterion leads to a fundamental. (iii) Und er LQG assu mpti ons

298

RISK-SENSITIVITY: THE LEQC MODEL

the complete and attractive generalisation of LQG theory. Essentially, n Gaussia a ing resembl ng somethi expectation in (3) is then the integral of 'largeterm might one what by density. (iv) If LQG assump tions are replaced scale' assumpt ions and if an exponential criterion is adopted then large-deviation theory become s immedi ately applicable. It is striking that the econom ic concept of risk-sensitivity, interesting in itself, should mesh so naturall y with the mathematics. We shall explore the LQG a generalisation in the remaind er of this chapter. Large-deviation concept s open complex of ideas to which the final part of the book is devoted.

Exercises and comments (1) Show that L"(m)/ L'(m) is indepen dent of m if and only if L(m) is a linear ntly function of an exponen tial of m. A utility function is of course not significa e commut mations transfor such changed by a linear transfor mation, because with the operatio n of expectation. (2) Note an implica tion of Exercise 8.1.3: that relation (4) is exact (i.e. there is no remaind er term) if L(m) is an exponen tial function of a normal variable.

(3) A classic momen t inequality asserts that, if x is a non-negative scalar random is variable, then (Ex) 1/r is non-dec reasing in r. From this it follows that x1r(O) non-increasing in B.

2 THE RISK-SENSITIVE CERTAINTY-EQUIVALENCE PRINCIPLE tial The combin ation of the LQG hypotheses of Section 12.2 and the exponen to and 12.3, Section in d discusse criterion (3) leads us to the LEQG model already most the that 12.3 Section which this chapter is devoted. In fact, we found in

the economical way of proving the certainty-equivalence principl e (CEP) in or king risk-see LQG case was to do so first for the LEQG model in the there 'optimistic' case (} > 0. Let us slightly rephrase the conclusions summar ised in Lemma 12.3.2 and Theorem 12.3.3. Define the stress

(5) the the linear combin ation of cost and discrepancy which occurs naturally in value st evaluation of expecta tion (3). Let us also define the modifie d total-co function G( W1) as in Section 12.3 by (6) e-eG(W,) = f( Y1)extE7r[e-9Cj Wr]· 7r (} is where the extremisation is a maximi sation or a minimis ation according as 12.3.3 Theorem and 12.3.2 positive or negative. Then the conclusions of Lemma can be rephrase d as follows.

UIVALENCE PRINCIPLE 2 THE RISK-SENSITIVE CERTAINTY-EQ

299

equivalence theorem for the riskTheorem 16.2.1 (The risk-sensitive certainty G structure with B > 0. Then the total seeking (optimistic) case) Assume LEQ value function has the expression inf inf t inf§ G(Wr) = gt + u.,;r;; X . t JT:r> - 1[J)(Xr, Yrli Ur-I) + inf{B X, + inf inf [C(X, U) + e- 1UJ(xr+l) ... )XhiXri U)]}

= gc

(7)

u.,;r;;. t x.,.:r>t

. The value of u1 thus determined is where g1 is a policy-independent function oft alone t. If the value of u1 minimising the the LEQG-optimal value of the control at time the LEQG-optimal value of u1 is square bracket is denoted u(X1 , U1_I) then mined by the final Xrminimisation. u( xit), Ut-1) where x?l is the value ofX 1 deter e function at time t is obtained by The first equality of (7) asserts that the valu ess/observation variables currently minimising the stress with respect to all proc formed, and that the value of u1 deterunobservable and all decisions not already immediately suggest the expression of min ed in this way is optimal. This may not achieves what one might term convera CEP, but it is a powerful assertion which isation with respect of functions sion to free form. By this we mea n that an optim ined minimisation with respect to unu1( W1) has been replaced by an unconstra t is, the constraint that the opti mal observables and undetermined controls. Tha achieved automatically in a free u1 should depe nd only upon W1 has been ing constraints will be taken to its extremisation. This process of effectively relax h the full time-integral formulation. conclusion in Chapters 19-21, when we reac h mor e like a CEP, in that it asserts The final assertion of the theorem looks muc optimal value for known X 1 with X 1 that the optimal value of u1 is just the is often confused with the 1 ). The risk-neutral CEP replaced by an estimate not well agreed) the separateness of separation principle, which asserts (in terms Ther e is certainly no such separation the optimisations of estimation and controL and future) affect the value of the in the LEQ G case. Control costs (both past rol rule (even if the process variable is estimates and noise statistics affect the cont perfectly observed) should now be expressed. If Xr is However, we see how a separation principle uation of the two terms inside the provisionally assumed known then the eval which can be regarded as conc erne d curly brackets in the final expression of (7), can proceed separately. The two with estimation and control respectively, misation with respect to X 1, which evaluations are then coupled by the final mini tiveness of this separation is muc h also yields the final estimate x?l. The effec d below. clearer in the state-structured case considere rs interestingly, and requires mor e The CEP in the risk-averse case, B < 0, diffe the fact that relation (12.16) now careful statement. The distinction arises from becomes

Xi

RISK-SENSITIVITY: THE LEQC MODEL

300

(8)

G(W1) =sup infG(Wt+ 1) + ... u,

Yt+l

and that the order of the two extremal operations cannot in general be reversed. The consequence is that the analogue of Theorem 16.2.1 is Theorem 16.2.2 (The risk-sensitive certainty equivalence theorem for the riskaverse (pessimistic) case) Assume LEQG structure with(}< 0. Then the total value function has the expression

G( W1) = g1 + inf sup ... inf sup§ u,

= gt +

Yt+I

uh-I

stat{fJ- 1[])(Xt,

x,

Yh

(9)

Ytli Ur-i)

stat [C(X, U) + 8- 1[])(xt+l• ... ,xhiXti U)]}

+stat

u.,.:r ~ t x.,.:r>t

where g 1 is a policy-independent function oft alone. The value ofu1 thus determined is the LEQG-optimal value of the control at time t. If the value of u1 extremising the square bracket is denoted u(Xi, U1_!) then the LEQG-optimal value of u1 is is the value of X 1 determined by the final Xrextremisation. u( x?l, Ur-1) where

x?l

Here Yh can be regarded as yielding complete information, and so can be identified with X, and the operator 'stat' simply renders the expression to which it is applied stationary with respect to the variable indicated. The first relation in (9) follows as in the risk-seeking case, but the order of the extremisations must now be observed. However, all that is necessary in applying recursions such as (8) is that quantities being minimised (maximised) should be convex (concave) in the relevant argument The cases in which this requirement fails prove interesting; see Sections 3 and 4. The extremisation conditions in the first expression of (9) will yield linear relations which can be solved (i.e. variables eliminated) in any order. The rearrangement of extremal operations in the final expression of (9) corresponds to just such a reordering, but with the characterisations of maximality or minimality now weakened to stationarity of some kind. Although state-structure plays no part in this formalism (and it is for this reason plus economy that we have dragged the reader through the general case) the situation does indeed become more transparent if state structure is assumed. In such a case cost and discrepancy will have the additive decompositions h-1

h-l

c=L

c(xr, Ut, t) h

D(xt)

L Ct + ch

(10)

t=O

t=O

[]) =

+ Ch(xh) =

h

+ LD(xr,YriXt-l, Yt-!iUt-1) =Do+ LDt 1=1

(11)

t=l

say. Here we have not discounted explicitly, and D(x0 ) is the discrepancy derived from the prior distribution of x 0 conditional on W0 .

UIVA 2 THE RISK-SENSITIVE CERTAINTY-EQ

LENCE PRIN CIPL E

301

and the past stress P1 = P(x , W1) as Defm e now the future stress Fr = F(xr, t) the values of t-1

h-1

I:(c.,.+ e- Dr+t) + Ch 1

and

e-

Do+ L(c .,.+ 8- 1D.,.+t)

1

r=O

d out. Tha t is, all controls from time t with all variables except W1 and x, extremise servables at time t except x 1 itself are onwards are mini mise d out and all unob 8 is positive or negative. mini mise d or max imis ed out, acco rdin g as nce for the state -stru cture d

vale Theorem 16.2.3 (Recursions and certainty-equi Then the future and past stresses l. mode G LEQ d case) Assume a state-structure itions are determined by the recursions and terminal cond ext [c(x1, u, t) + e-t D(xr+tlx,; u,) F(xr, t) = inf Xt+l Ut

(12)

(O~t
+F( xr+ t,t+ 1)] F(xh) = Ch(xh)

(13) (14)

+ P(xr-1, Wr-t)) P(xo, Wo)

(0 < t ~h)

= e-t D(Xo)

(15)

ation according as B is positive or where ext indicates a minimisation or a maximis ted u(x, t) then the optimal value of negative.lftheminimisingvalueofu1in Q2) is deno g P + F1 = P(xr, W1) + P(x1 , W 1 ). u1 isu(x,, t), wherexr isthevalueofx1 extremisin 1 nces of the previous two theo rems All the assertions are imm edia te conseque familiar; the last asse rtion is more h and the definitions. Results now look muc r and in equa tions (12) and (14) we plainly certainty-equivalent in char acte mic prog ramm ing equation and the recognise respectively analogues of the dyna distr ibuti on P(x11W1). Because of the upda ting equa tion for the cond ition al these equa tions will reduce to Ricc ati quad ratic natu re of cost and discrepancy from Sect ions 2.4 and 12.5. and Kalm an relations of the type fami liar s itself. Provisional post ulati on of We see how the sepa ratio n prin ciple manifest evaluations of past and future stress, so the value x 1 of curr ent state decouples the cont rol/f utur e and estim ation /pas t that recursions (12) and (14), conc erne d with functions of observables W1 and the can be solved separately for F1 and P1 as lations are then recoupled by the (generally unobservable) x 1• The calcu ate 1 = x~t) , so yielding the fmal estim mini misa tion of P1 + F1 with resp ect to x 1 rol u(x" t). and the certainty-equivalent opti mal cont to that of the LQG case with, however, close then is rges eme h The patte rn whic disc repa ncy term in (12) enforces a a mixi ng of cost and stochastic effects. The

x

302

RISK-SENSITIVITY: THE LEQC MODEL

stochast ic plant equation but also induces a depende nce of the optimal control upon the covarian ce matrix N of plant noise, even if state observa tion is perfect. The cost term in (14) has an effect upon estimates, while the final estimate 1 shows a depende nce on future costs as well as past costs. The future stress F(x~> t) indeed differs from the normali sed value function for the case of perfect state observa tion only by a term depende nt on t alone. This term represen ts the cost due to plant noise, and will be evaluate d in Section 10. Finally, a word on notation . We shall term 1 the minimal stress estimate of x 1, even if indeed the extremis ation of stress is more complic ated than a simple minimis ation. It is identica l with what we have before written x)tl and shall do again: the best estimate of Xt based upon informa tion at time t. We shall use 1 to denote the estimate of x 1 based upon informa tion at time t which extremises past stress. In the risk-neu tral case the two estimate s coincide d, because future costs had no effect upon state estimates. However, we have now to make a distincti on.

x

x

x

3 SOME SIMPLE EXAMPLES It is helpful to gain a feeling for the effects of risk-sensitivity by examini ng a couple of examples which are simple in that they are concern ed purely with estimati on. Conside r the uncontr olled Markov model x 1 = Axr-1 + E1• We suppose that x is scalar and that the plant noise c is white and Gaussia n with variance N The cost function is C = ~:o R.x;- and it is suppose d that the total informa tion available at time 0 is knowled ge of x 0 • The stress function is consequ ently

!

00

§

= !L[R.x7 + (8Nf 1(xt- Axt-d 2]. t=O

At time 0 this is to be extremis ed with respect to the current unobser vables, which are just x 1 , x 2 , x 3 , • . . • We thus have a risk-sensitive formula tion of a pure predicti on problem from time t = 0. In the risk-neutral case the predicto r is evidently just the projecti on estimate: tS'(x 1 jx0 ) = A 1x . The values 0 x)0l extremis ing stress satisfY

( 1 + A 2 + ONR)x(O ) - A(x(O) t t-1

+ x(O) ) =0 x+l

(t

~

0).

Assume , for simplicity, that 0
(16)

e

If is positive then 0 < a < A, and the forecast is optimist ic in that it tends to zero faster (and so incurs a smaller cost) than does the risk-neu tral predicto r. Better expressed: the forecast er is optimist ic, in that he is effectively assumin g that the noise c will be such as to deflect the path of the process in a favourable direction .

3 SOME SIMPLE EXAMPLES

303

> A and the forecast is pessimistic. prediction as having broken down altogether if (} is so the regard One can the extremal negative that equation (16) has no root smaller than unity, when stress is infinite. This occurs when

If(} is negative then a

a~ e·=- (1- A)2 "'

.

NR

maximised with At the critical point Bthe stress function, which is now being has extremal and respect to unobservables, is no longer concave in its arguments, ster has becom e value +oo. One can regard this as the point at which the foreca his apprehenso pessimistic that he has essentially reached a breakdown point: sions ofloss outweigh the reassurance of statistics. the one-stage For an even simpler example, consider a risk-sensitive version of jointly norma l estimation problem of Section 12.6. Suppose that x and y are random vectors with zero mean and covariance matrix

l

cov[; = [

~:: t~

rl

observed and xis and that there is a cost function~ xT Rx. The variable y has been estimate x by the derives one lation formu to be estimated. In a risk-sensitive minimising

lxx 2(}§=( }xTRX + [X]T[ fyx Y

lxy] [X] fyy y

with respect to x, thus deriving

X= -(fxx + (}R)- 1lxyY· and approaches This estimate of course equals the projection estimate if(} is zero, ation matrix the minimal-cost value 0 as(} increases. The matrix lxx is the inform to lxx +OR. this modifY to is y nsitivit risk-se of x conditional on y; the effect of to let costhappy is one that (in e positiv is () That is, one 'gains' information if value 0 ing inimis cost-m its to nearer pressures supply the 'information' that xis e (in negativ is if(} ation' 'inform loses than the data would have suggested). One that e negativ so es becom () When that fear of costs makes one doubt the data). e becom has ser optimi the her: lxx + ()R is singular then estimation fails altoget this which at B value l critica pessimistic to the point of neurotic collapse. The occurs is the greatest root of llxx + 8R! = 0. in a negative In both examples optimisation failed only for (} sufficiently large by zero in below ed direction. This was because the cost function C was bound infinite an yielded it both cases, and stress-extremisation broke down only when and istic, pessim positive cost. This occurred when the optimiser was sufficiently e · negativ infinite one might term the breakdown that of neurosis. If C can take

LEQC MODEL RISK-SENSITIVITY: TH E

304

ur if() is then breakdown can also occ le) sib pos is ard rew te ini inf values (i.e. if an n can yield an infinite e. Th at is, stress-extremisatio sufficiently large and positiv imiser is sufficiently te positive reward) if the opt ini inf an . (i.e t cos e ativ neg s breakdown tha t of euphoria. optimistic; one might ter m thi n of the one-stage l is supplied by consideratio tro con of le mp exa ple sim A of the multi-stage something of the character es sag pre on uti sol ose wh R( xl - r) , problem, n is c = !uTQu +~(xi- r)T ctio fun t cos the t tha se ppo and u0 . Th e problem. Su u are jus t the values of x 0 and x t tha so £, + Bu £T (BN) - 1£. where x1 =A x+ remise the stress C ext to sen cho be to n the are variables u and £ this as convenient form if we write We obtain results in the mo st 1 2AT(Ax + Bu + € - xc)] . 1 [uT Qu + €T(ON)- £ - >.TR- A+

+!

§

= mq .X

pect to Awe find tha t the t to u and £an d then with res Extremising first with respec are given by optimal u and the extremal § 1 r) , §= !( Ax - r)T P- (A xincreasing 1 + J(O), say. Th e effect of R1 N= +O 1 BT BQ + efinite where P = Rke P larger (in the positive-d ma to eed ind is 0) g sin rea optimism (i.e. inc ponding stress value optimal control and corres sense) and to make bo th the mutation of them) are ns (and our implicit com tio isa rem ext ese Th r. alle sm > 0, where if is the e definite, i.e. as long as () itiv pos is P as g lon gest as ate legitim the critical value if is the lar 1 + J(O)J = 0. Equivalently, jRof t roo ar. largest is singul The n which is correct even if R tio lua eva an 0, = B)I RJ( It is always root of II+ int of 'neurotic' breakdown. po the eed ind is d ine erm value if thus det -negative definite. non-positive if Q and R are non le, in tha t they both ave as controls in this examp Note that u and £ bo th beh in the stress function. sion for x 1 and quadratically res exp the in ly ear lin ear app is chosen to to sen to minimise stress, £ cho ays alw is u as ere wh , However e. Th at is, this ing as () is positive or negativ ord acc it ise xim ma or it as he is minimise strate the optimiser according fru or p hel to sen cho is ol' auX.iliary 'contr in the next section. shall expand up on this point risk-seeking or risk-averse. We (x1 - r) T R(x1 - r) cost function then the ter m! If we modify R to - R in the Th e calculations ard, which we wish to be large. rew a as her rat ed ret erp int be is to of sign of R) is negative P (modified by the change as g lon as id val l stil are B) I = 0. This above the smallest root of II - RJ ( eed exc t no s doe ) as( g lon as er sign. definite, i.e. ric' breakdown. It can be of eith pho 'eu of int po the s ent res up per bo un d rep T STATE OBSERVATION 4 TH E CASE OF PE RF EC ncipal interest: r is devoted to the case of pri pte cha the of der ain rem the omogeneous Most of del in the standard time-h mo n tio ula reg d ure uct str Section 12, the state(12.1)-(12.4). Th e exception is ns atio equ of on lati mu for undiscounted

4 THE CASE OF PERFE CT STATE OBSERVATION

305

look at the question of where we find ourselves forced to take a funda menta l discounting. m reduces then to Let us first of all assume perfect state observation. The proble the dynam ic ially essent ; stress· the solution of the equation (12) for the future progr ammi ng equation. This can be written

F(xr,t)

= inf Ut

ext[c(xt,Ut) +!B- 1 (£TN- 1 c)t+ 1 +F(x t+t,t+ 1)]

(17)

Xt+l

ing as eis positive or where 'ext' denotes a minimisation or a maxim isatio n accord In virtue of this (12.1). ion equat negative, and €t+l and xt+ 1 are related by the plant last fact we can rewrite (12) as Er+l• t + 1)] 1 F(x,, t) = inf ext[c(x,, ut) + !B- 1 ( ET N- c) 1+ 1 + F(Ax, +Bur + Ut


(18) l variable, enteri ng But in this form we see that € can be seen as a subsidiary contro atic cost just as u quadr a ng carryi the plant equat ion linearly just as u does and e him if is oppos to and e does. It is chosen to help the optimiser if (J is positiv d by the wielde are negative. One might regard u and E as the controls which es that assum vely optimiser and by Nature respectively. The optim iser effecti is risk-seeking or riskNatur e is working with him or against him according as he respectively. Note that avers e-a fair characterisation of optim ism or pessimism Natur e makes its move first, at each stage. equation, but only in Of course, € does not appea r as a control in the actual plant In the actual plant s. proces the predicted course of the optimally controlled as a 'control' for the rs equation it is simply rando m process noise, as ever. It appea a stress extremisation; predic ted process because this prediction is generated by current control. the extremisation which determines the optim al value of the familiar Ricca ti The LQ character of recursion (18) implies a solution on on of the risk neutr al lines, which can indeed be expressed in terms of the soluti case.

e

has the quadratic · Theorem 16.4.1 The solution of equation ()8) for the future stress form (19) F(x1, t) =! (xTilx) 1

if it has thisform for t = h, and the optimal control then has the linear form Ut = KtXt·

(20)

(6.31) of the riskHere II1 is determined by either of the alternative forms (2.25) or ative equations neutral Riccati equation and K1 by either of the corresponding altern equations by (2.27) or (6.32) if ITt+ 1 is replaced in the right-hand side ofthese (21)

RISK-SENSITIVITY: THE LEQC MODEL

306

IIM\ Validity of these conclusions is subject to the proviso that the matrix J(B) + controlted augmen the is J(B) where t, should be positive definite for all relevant power matrix (22) 1. We Proof This is inductive, as ever. Suppose that relation holds at time t +

leave the reader to verify the identity ext[(t? (BN)- 1€ +(a+ €liT( a+ €)]

= aTfia

(23)

•

If we perform the €-extremisation in (23) we thus obtain · 1, Ut) F(xt, t) = mf[c(x u,

+ 2I (Axt +But) T-ITt+ I (Axt +But)].

But this is just the inductive relation which held in the risk-neutral case with the g substitution of IT1+1 for II 1+ 1. The question of validity is covered in the followin D discussion.

If we consider the solutions of the risk-neutral case in the alternative forms (6.311 (6.32) then we see that the only effect of risk-sensitivity is simply to replace for the control-power matrix J = BQ- 1BT by the augmented form (22), so that, example, the Riccati equation becomes

(24) if Shas been normali sed to zero. That is, the control-power matrix is augmented as by a multiple of the noise-power matrix. This illustrates again the role of noise control d intende the against or with working an effective auxiliary control, according as eis positive or negative. If the final maximisation with respect to >. of the second. set of displayed of equations in Section 6.4 is now to be valid then we require the final condition 1. the theorem. This sets a lower bound on 8: see Exercise The optimal (i.e. stress-extrernising) values of u1 and €t+I are matrix multiples + of x 1; if they are given these values then the quantities Ax1 + But and Ax1 the as BK =A+ 1 r But+ €t+l can be written r 1x 1 and f' 1x 1• We can regard 1 actual gain matrix for the optimally controlled process and f' 1 as what one might call the predictive gain matrix: the gain matrix that would hold if €1+1 really did by take the role of an auxiliary control variable and take the value predicted for it 6.4 Section stress extremisation. By appealing to the alternative derivations of one finds the evaluations

r1 =A -

J(J(e)

+ rr~.\)- 1 A,

(25)

by if S has been normali sed to zero. In other cases A should be replaced a ily A - sT Q- 1B. If infinite horizon limits exist then it is f' which is necessar

) CASE 5 THE DISTURBED (NON-HOMOGENEOUS

307

is excessively optimistic. Note the stability matr ix; r may not be if the optim iser relation

Exercises and comments of the Ricca ti equation (24) (1) In the scala r case the infin ite-h orizo n form beco mes

A 2 II II= R + 1 + J(8)II that the equa tion has a finite wher eJ(O ) is given by (22). Assu meth atR > 0. Show 2 ~ 1 and J(fJ) > -(1 -IAI )/ R non-negative solution iff J(B) > 0 in the case !AI 2 2 QIN d ii is - B IN Q or - B in the case lA I < 1. That is, the critical lower boun is unsta ble or stable. (l - IAI 2 )/NR accor ding as the unco ntrol led plant US) CAS E 5 THE DIST URB ED (NON -HOM OGE NEO the addit ion of a deter minis tic If the plant equa tion (12.1) is modi fied by distu rbanc e d1 : x, = Axr-1 +Bu r-l+ d, + tr the non-h omog eneo us quad ratic then we shall expe ct the future stress to have form

(26)

can generalise relation (23) to where + · · · indicates terms indep ende nt of x. We obtai n )] = aTfi a- 20'Ta + · · · ext,[(tT (BN)- 1 £+( a+ tlii( a + £) - 2aT (a+£ where + · · · indicates terms indep ende nt of a, the and

matr ix fi is again given by (21),

From this we dedu ce

ministic disturbance d then Theorem 16.5.1 If the plant equation includes a deter modified Riccati equation indithe future stress has the form (26) with Il1 obeying the recursion cated in Theorem 16.4.1 and a 1 obeying the backward (27) Here f\ is the predictive gain matr ix defined in

(25).

RISK-SENSITIVITY: THE LEQC MODE L

308

(27) differs from the Verification is immediate. We see that recursion by the substitution of of corresponding risk-neutral recursion of Section 2.9 only be, since we have f' 1 for the risk-neutral evaluation ofr 1 • This is indeed as it mustoptim isation with u by replaced optimisation with respect to a single control respect to the pair (u, e). al control as The same argument leads to the evaluation of the optim (28) 1 lN)-l (at+ I - IIt+ldt+l)· Ut = KtXt + (Q + BTfrt+IB)- BT (I+ OIIt+ the explicit feedbackThe combination of (28) and the recursion (27) gives one . (2.65) to gous analo feedforward formula for the optimal control more rapidly and much e emerg s As for the risk-neutral case, all these result isation techfactor the adopt cleanly (at least in the stationary case) when we . niques of Sections 6.3; see Section 11 and Chapter 21.

TION OF THE 6 IMPERFECT STATE OBSERVATION: THE SOLU SION P-RECUR the forward recursion In the case of imperfect observation one has also to solve the F-recursion implies (14) for the function P(xtJ W 1). Just as the solution of of perfect observation, solution of the control-optimisation problem in the case the estimation problem. so solution of the P-recursion largely implies solution of estimate x)Jased upon 'Largely', because the P-recursion produces only the state is then quickly derived, past stress.. However, the full minimal-stress estimate x1 as we shall see in the next section. ions (12.1)-(12.4) we For the standard regulation problem as formulated in equat can write the P-recursion (14) as (29) BP(xt, W,) = min[Bct-1 + Dt + BP(xt-1, Wt-d ) Xr-1

where Ct-1

=![~r_J~ ~H~t-~·

plant and observation and f, TJ are to be expressed in terms of x, y, u by the relations (12.1) and (12.2). have a limit, D, say, Now, if we take the risk-neutral limit()----+ 0 then ()p will which satisfies the risk-neutral form of (29) (30) D(xt, W,) = min[Dt + D(xt- l. Wi-1)] .l"t-1

and has the interpretation D(xr, W,)

= D(xr, Yr\; Ur-d

= D(xt\ W,)

+ ···

=! [(x- x? v- (x- .X)] + ··· 1

1

(31)

309

6 IMPE RFEC T STATE OBSERVATION

fact identifiable with D( Y1I; U1_ 1 ) ) , Here + · · ·indicates term s not involving x, (in cova rianc e of x 1 conditional on W1• and 1 and V1 are respectively the mea n and and, as we saw from Section 12.5, In this risk-neutral case is identifiable with of xand Y. the Kalm an filter and the relation (31) implies the recursive upda tings know n solution of (30) to dete rmin e Ricc ati recursion. We can in fact utilise the that of (29). that OP has the quadratic form In the risk-sensitive case we can again establish exhibited in the final mem ber of (31), so that

x

x

x

P(xr. W,) =

;o

[(x- x)

v-

1 (x-

x)Jr + · · ·,

(32)

of x, irrelevant for our purp oses . where + · · · again indicates term s inde pend ent estim ate of XT which extremises past The quantity X1 can now be identified as the ure the precision of this estim ate in stress at time t, and V1 can be said to meas stress as x 1 varies from 1• Rela tion that (OV1)- 1 measures the curvature of past quantities. (29) now deter mine s the upda ting rules for these

x

past stress has the quadratic form Theorem 16.6.1 Under the assumptions above the prescribed mean and variance of xo conditional ~2) with x0 and V1 identified as the those ofx1_ 1and Vr-1 by the Kalon Wo. The values of X1 and Vr are determinedfrom (1 2.24) with the modifications man filter (1 2.22)1 (1 2.23) and the Riccati recursion that x1 is replaced by X1, Vr-1 by Vt-l

-1

= ( Vt-1 +OR )

-1

(33)

andxr-1 by (34)

ofthese recursions it is necessary in the right-hand sides ofthese relations. For validity for all relevant t. that V;=_11 + cr M- 1C+ should be positive definite agai n inductively from (29). ReProof The quad ratic character (32) of P follows · If we assu me from (30) only in the addi tion of the term Ocr-! cursi on (29) differs ted then we fmd that that P(x 1_ 1, W1_ 1) has the quad ratic form asser (Jet-!

+ BP(Xt-1, Wr-1)

=! [(x- x)Tv -I (x- x)]t-1 +ter ms independent of Xr-1

whence it follows that

al case with the subs But this is just the recursion ofth e risk-neutr and Vt-! for Xt-! and Vt-!·

titution of Xr-1

0

310

RISK-SENSITIVITY: THE LEQC MODEL

The modified recursions are again more transparent in the alternative forms (12.59), (12.60). One sees that, as far as updatin g of Vis concern ed, the only effect 1 of risk- sensitivity is to modify CTM- 1 C to CTM- C+BR. That is, to the one adds the matrix ation information matrix associated with a single y-observ or negative (positive BR, reflecting the 'information' implied by cost-pressures according to the sign of B). The passage (34) from 1_ 1 to Xt-1 indicates how the estimate of Xt-1 changes if we add present cost to past stress. In continuous time the two forms of the modified recursions coincide and are more elegant; we set them out explicitly in Section 8.

x

7 IMPERFECT STATE OBSERVATION: RECOUPLING In the risk-neutral case the recipe for the coupling of estimation and control asserted by the CEP is so immedi ate that one scarcely gives it thought: the optimal control is obtaine d by simple substitution of the estimate x1 for Xt in the its perfect-observation form of the optimal rule. It is indeed too simple to suggest the that is This 16.2.3. Theorem from know we which sation, risk-sensitive generali optimal control at time t is u(.Xt, t), where u(x~o t) is the optimal perfectof observation (but risk-sensitive) control and x1 is the minimal-stress estimate t). , F(x + ) W 1 , P(x 1 1 xr: the value of x 1 extremising It was the provisional specification of current state x 1 which allowed us to d decouple the evaluations of past and future stress; the evaluations are recouple ofF (32) and (26) ons evaluati separate the that by the determi nation of 1• Now be and P have been made explicit in the last two sections, the recoupling can also made explicit.

x

Theorem 16.7.1 The optimal control at timet is given by u1 = K/x,, where

.x, = (I+ ev~rr~r\i·, + evto-t)·

(35)

Here II1, K1 and o-1 are the expressions determined in Theorems 16.4.1 and 16.5.1 and V1, xt those determined in Theorem 16.6.1. This follows immediately from the CEP assertion of Theorem 16.2.3 and the Xr evaluations of the last two sections. Note that 1 is an average of the value of which extremises past stress and the value II;- 1a1 which extremises future stress. As one approaches the risk-neutral case then the effect of past stress swamps that of future stress.

x

x,

8 CONTINUOUS TIME to The continuous-time analogue of all the LEQG conclusions follows by passage The . the continuous limit, and is in general simpler than its discrete-time original

7 IMPERFE CT STATE OBSERVATION: RECOUPL ING

311

analogues of the CEP theorems of Section 1 are obvious. Note that the u and t: extremisations of equation (18) are now virtually simultaneous: the optimiser and Nature are playing a differential game, with shared or opposing objectives according as eis positive or negative. The solutions for past and future stress in the state-stru ctured case simplifY in that the two alternative forms coalesce. The solution for the forward stress for the disturbed but otherwise time-homogeneous regulation problem is F(x, t)

= !xTIIx- aT x + · · ·

where II and a are functions of time. The matrix II obeys the backward Riccati equation

This reduces to

IT+ R + ATII +ITA- IIJ(O)II = 0

(37)

if Shas been normalis ed to zero, where the augmented control-power matrix J(O) again has the evaluation

The vector a obeys the backward linear equation

a-+ f'T

(T-

IId

=0

where f' is the predictive gain matrix

The optimal control in the case of perfect state observation is

u = Kx+ Q- 1BTa-

(38)

. where the time-dep endence of u, K, x and a is understo od, and

(39) In the case of imperfec t state observation the past stress has the solution

P(x, W)

= ;e(x- x)TV- 1(x- x) + ...

where the time-dependence of x, W, forward Riccati equation

xand Vis understood. The matrix Vobeys the

RISK-SENSITIVITY: THE LEQC MODEL

312 which reduces to

if L has been normalised to zero. The updating formula for Kalman filter) is

x (the risk-sensitive

dX/dt =Ax+ Bu + d + H(y- Cx)- OV(Rx +STu)

(40)

where H = (L + VC)M- 1 • Recoupling follows the discrete-time pattern exactly. That is, the optimal control is u = Kx + Q- 1BTu where Kis given by (39) and x is the minimal-stress estimate of x:

9 SOME CONTINUOUS-TIME EXAMPLES The simplest example is that of scalar regulation, the continuous-time equivalent of Exercise 4.1. Equation (37) for II can be written

~ = R + 2AIT- J(O)II2 =/(IT),

1

I

I

1

(41)

say, where J(O) = B2 Q- 1 + BN, and s represents time to go. Let us suppose that R > 0. We are interested in the non-negative solutions of f(IT) = 0 which moreover constitute stable equilibrium solutions of (41), in that/' (II) < 0. In the case J(B) > 0 there is only one non-negative root, and it is stable; see Figure 4. In the case J(8) = 0 there is such a root if A < 0 and no non-negative root at all if can be A~ 0. If J(O) < 0, then there is no non-negative root if A~ 0, but there of root one if A is negative and -J (0) not too large; see Figure 5. In fact, there is a the required character if A < 0 and R - A2 / J( 0) < 0. To summarise:

II II

Figure 4 The graph ofjTI in the case J > 0; it has a single positive zero.

9 SOME CONTINUOUS-TIME EXAMPLES

313

II is a positive zero if J exceeds a criFigure 5 The graph of[IT in the case J < 0, A < 0; there value. tical

em set out above, with S = 0 Theorem 1691 Assume the scalar regulation probl itude of K decrease as e inand R and Q both positive. Then both II and the magn creases. down value 0equals If A ~ 0 (i.e. the uncontrolled plant is unstable) then the break value. - B2 / N Q and II becomes infinite as() decreases to this breakdown value 0 equals the then ) stable is plant d If A < 0 (j. e. the uncontrolle solution TI of (41) at()= 0 is -B2 jNQ - A 2 jNR. The non-negative equilibrium finite, but is unstable to positive perturbations. linea rised pend ulum mode l of A secon d-ord er exam ple is provi ded by the on the angle of defle ction a versi Secti on 28 and Exercise 5.1.5. In a stoch astic from the vertic al obeys

a= aa+ bu+ E

icien t a is negative or posit ive wher e E is white noise of powe r N and the coeff ing or the inver ted posit ion. The accor ding as one seeks to stabi lise to the hang 2 Q strictly positive. 2 cost funct ion is! (r 1a 2 + r 2a + Qu ), with r 1 and 2 Q- 1 + BN. It follows from that The analysis of Secti on 2.8 applies with J = b II of the Ricca ti equa tion if and analysis that there is a finite equil ibriu m solut ion . The break down value is thus only if J > 0, and that this solut ion is then stable of the hang ing posit ion 0 = -1/ NQ, whatever the sign of a. The great er stabithelityrelati ve magn itude s of II in comp ared with the inver ted posit ion is reflected only neutr ally stable, rathe r than in the two cases, but the hang ing posit ion is still truly stable. stead y state is provi ded by the Finally, a secon d-ord er exam ple whic h has no solut ion for the optim al contr ol inert ial missile exam ple of Exerc ise 2.8.4. The obtai ned there now beco mes

u=-

Ds(x1 + x2s) Q + D(l + BNQ)s3 /3'

314

RISK-SENSITIVITY: THE LEQC MODEL

where sis time to go. The critical breakdown value is iJ = -1 INQ - 3INDs3, and so increases with s. The longer the time remaining, the more possibility there is for mischance, according to a pessimist. 10 AVERAGE COST The normalised value function F(x, t) defmed by e-BF(x,t)

= extuE[e-8C'Ixr = x,ur = u].

(42)

in the state-structured case should be distinguished from the future stress defined in Section 2, which is in fact the x-dependent part of the value function The neglected term, independent of x, is irrelevant for determination of the optimal control, but has interest in view of its interpretation as the cost due to noise in the risk-sensitive case. Let us make the distinction by provisionally denoting the value function by F,(x, t). Theorem 16.10.1 Consider the LEQG regulation problem in discrete time with perfect state observation and no deterministic disturbance (i.e. d = 0). Then the normalised value function Fv has the evaluation

F,(x, t) = F(x, t) + 8r

(43)

where F(x, t) is the future stress, evaluated in Theorems 16.4.1 and 16.5.1, and (44)

The proof follows by the usual inductive argument, applied now to the explicit form exp[-OF,(x, t)] = e:t(?rrn/2 INI- 1/ 2

j exp[-!eTN- e- Oc(x,u) 1

- OFv(Ax + Bu + e, t + 1)] de of recursion (42). The evaluation (44) of the increment in cost due to noise indeed reduces to the corresponding risk-neutral evaluation (10.5) in the limit of zero 0. It provides the evaluation 1 (45) 'Y(O) = 26 log!/+ ONITI of average error in the steady state, where II is now the equilibrium solution of the Riccati equation (24) (with A and R replaced by A - ST Q- 1B and R - ST Q- 1S if Shas not been normalised to zero). More generally, it provides an evaluation of the average cost for a policy u(t) = Kx(t) which is not necessarily optimal (but stabilising, in an appropriate sense) if II is now the solution of

315

10 AVERAGE COST

BK)T (II- 1 + ON)- 1(A + BK). II= R + KTS + ST K + KT QK +( A+

(46)

methods of rage cost is best evaluated by the For mo re general models the ave uced the expression Section 13.5. Recall tha t we there ded (47) 1(8) = 4 0 log ll + 09if(w)l dw ilising, but which is linear, stat ion ary and stab for the average cost und er a policy deviation the spectral density function of the otherwise arbitrary. Her e f(w) is s expression the associated penalty matrix. Thi vector 6. und er the policy and 9i tion of stat e variant model; the re is no assump is valid for a general linear time-in rem ain s task the d, han ervation. On the oth er structure or of perfect process obs of f(w ) s clas the g the policy and determinin of det erm inin gf( w) in terms of icy is varied. which can be generated as this pol rmined by evaluation (47) reduces to tha t dete It is by no means evident tha t the (supposed cy d regulation case with the poli (45) and (46) in the state-structure straightis iation in the risk-neutral case stabilising) u = Kx. The reconcil ht per hap s e case is less so and, as one mig forward. Tha t for the risk-sensitiv factorisation: follows by appeal to a canonical conjecture from the form of (47), see Exercise 20.2.1. years is to ch has become pop ula r over recent A view ofsystem per form anc e whi urbances dist s iou var system output, and the consider the deviation vector !::. as system as e) nois n atio (e.g. pla nt and observ to which the system may be subject funce ons resp cy characterised by the frequen input. A given control rule is the n rule the ose cho ch it induces. On e wishes to tion G( iw) from inp ut to out put whi sen. cho be ld there are many nor ms which cou to make G small in some nor m, and ut inp em ld be regarded as a collective syst So, suppose tha t the noise inputs cou r filte g lisin wer) mat rix 91, say. (A pre-norma (wh ich is white with covariance (po ion Express inc orp ora ted in the total filter GJ which would achieve this could be (47) the n becomes

!j

1(8) =

4~0 j

!j

log !/+ 09iG(iw)9lG(( -iw ?!d w = 4 0

Iog!I + 09iG9lGI dw (48)

respect to of minimising this expression with and the problem then becomes one of LE QG ion not the m on G, generated by G. Expression (48) is indeed a nor blem in pro on sati me). To phr ase the optimi optimisation (in the stationary regi pter. cha t nex the tain issues, as we shall see in this way helps in discussion of cer fact the with ntial issue: how does one cope However, it also glosses over an esse te qui is ied ns G generated as policy is var tha t the class of response functio es vid t it pro amic pro gra mm ing approach tha restricted? It is a virtue of the dyn on of the cati cifi spe by lied constraints imp an automatic recognition of the n. st be provided in a direct optimisatio system; some equivalent insight mu

316

RISK-SENSITIVITY: THE LEQC MODEL

For reasons which will emerge in the next chapter, one sometimes normalises the matrices if\ and 91 to identity matrices by regarding G as the transfer function from a normalised input to a normalised output (so absorbing 9l and 91 into the definition of G). In such a case (48) reduces to

1(8)

1 = 41re

J

-

logjl + BGGI dw.

(49)

The criterion function (48) or (49) is sometimes termed an entropy criterion, in view of its integrated logarithmic character. However, we should see it for what it is: an average cost under LEQG assumptions. In the risk-neutral case (49) reduces simply to the mean-square norm (47rf 1 Jtr[GG]jdw, also proportional to a mean-square norm for the transient response.

11 TIME-INTEGRAL METHODS FOR THE LEQG MODEL The time-integral methods of Sections 6.3 and 6.5 are equally powerful in the risksensitive case, and equally well cut through detailed calculations to reveal the essential structure. Furthermore, the modification introduced by risk-sensitivity is most interesting. We shall consider this approach more generally in Chapters 21 and 23, and so shall for the moment just consider the perfect observation version of the standard state-structured regulation model (12.1)-(12.4). The stress has the form §

= l_.)c +!ET(ON)- 1E]r +terminal cost. T

This is to be extremised with respect to u and f subject to the constraint of the plant equation (12.1). If we take account of the plant equation at time 7 by a Lagrange multiplier >.r and extremise out E then we are left with an expression

~ = I'::[c(xn ur) + >.;(xr- Axr-1- Bur-l)- !B>.JN>.r] +terminal cost (50) T

to be extremised with respect to x, u and>.. This is the analogue of the Lagrangian expression (6.19) for the deterministic case which we found so useful, with stochastic effects taken care of by the quadratic term in >.. If we have reached time t then stationarity conditions will apply only over the time-interval 7;;;: t. The stationarity condition with respect to fr implies the relation

between the multiplier and the estimate of process noise. In the risk-neutral case 8 0 this latter estimate will be zero, which is indeed the message of Section 10.1. Stationarity of the time-integral (50) with respect to remaining variables at time 7leads to the simple generalisation

317

12 WHY DISCOUNT?

sT

(r

Q

~

t).

(51)

-Bf / case, when deter mini stic distu rban ces of equa tion (6.20) (in the speci al regu latio n . The matr ix oper ator
J

[c(x, u) +

A.T(;~;- a(x, u))- !e.>..TN.\] dr +ter mina l cost,

(52)

linea r a and quad ratic c. This can be whic h can be treat ed in the same way for the form (7.2) assoc iated with the regar ded as a stoch astic gene ralisa tion of in .:\ repre senti ng the stoch astic max imum princ iple, with the quad ratic term ensitivity. We take up these matt ers effect insof ar as it repre sents the effect of risk-s in Part 5.

12 WH Y DISCOUNT? whic h is time- homo gene ous, and has Supp ose we assu me a mode l in discr ete time Supp ose also that it is disco unted , both an infin ite horiz on and an infin ite past. I:~;;, 0 f3T Cn wher e /3 is the disco unt so that the cost func tion from time 0 is§= depe nd explicitly upon time T. The facto r and the insta ntan eous cost cT does not expe cts the optim al polic y also to one form ulati on then seems time -inva riant and ider optim isatio n on an expo nenti al be so (i.e. statio nary) . However, if we cons 1r is chos en to extre mise crite rion from time t ( ~ 0), then the polic y (53) of futur e diSCOUnted COSt at time f. Where Cl =I:~~ I /3'"-IC T iS the prese nt Value rred befor e time t, which will not The prop ortio nalit y facto r involves costs incu the optim al policy cann ot poss ibly affect polic y from time t. We see from (53) that ing-p oint from time 0 to time t we be time -inva riant , beca use in shift ing the start para mete r from (} to {JI6, all othe r have in effect chan ged the risk-sensitivity facto rs rema ining statistically cons tant. -inva riant if correctly viewed, and One feels that the mod el is nevertheless time red. The reme dy is certa inly not to wond ers how the corre ct view can be resto value at each stage; this woul d be reno rmal ise the cost- funct ion to prese nt

318

RISK-SENSITIVITY: THE LEQC MODEL

inconsistent with the principle that an optimisation from a given time can be embedded in an optimisation from an earlier time. It seems necessary to think through the concept of discounting afresh. There seem to be three motivations or justifications for introducing discounting, and some doubt as to whether any of them are applicable in the context of process control. These are (i) mathematical convenience, (ii) the possibility of random termination, and (iii) the growth of capital by compound interest. For point (i), discounting is indeed a convenient way of reducing a total cost over an infinite horizon to something finite, mathematically attractive to handle and reminiscent of the concept of Abel summation. However, convenience is not a reason; any regularising reformulation must have an operational justification. If one expects that the controlled system will settle down to a steady state of some kind then minimisation of average cost in this steady state seems the natural criterion. The only point which should perhaps be emphasised is that already made earlier: the model should be such that transients are excited spontaneously if a policy effective in the steady state can also be guaranteed to be effective against transients. Point (ii) corresponds to the conviction that nothing goes on for ever. The investor of Section 2.2 cannot postpone consumption of his wealth for ever; he must reckon with the fact that he will die at some point. No industrial plant or process will be run for ever; the uncertainties of future economics and technology are at least certain in predicting that. Suppose that, if the plant has survived in its existing form to a given timet, then it has probability f3 of doing so to time t + 1. Then one can regard the discounted term {3F(-, t + 1) in the dynamic programming equation as representing the expectation of future cost over survival prospects. The expectation should also include a term ( 1 ~ {3) times the cost incurred if the plant does not survive to time t + 1. This term will presumably be zero if non-survival simply means termination; it will be something different if non-survival means movement into a new technology, for example. However, while one may have to reckon with such contingencies on a planning time-scale, one scarcely needs to do so on the time-scale of routine control, for a which an average criterion then seems more appropriate. A secondary observation is that a discounting induced in this way would, under LEQG assumptions, nevertheless destroy LEQG structure: see Exercise 1. It is alternative (iii), the effect of compound interest, which is the usual justification. However, a complete model has to make the interest mechanism explicit. The introduction of interest implies the existence of two alternatives: either to use one's money to cover the costs of the enterprise (i.e. of the system being controlled) or to leave it in the bank to grow by compound interest. (More generally, one could consider the allocation of resources between several enterprises competing for support, but the comparison of one uncertain enterprise with the certainty of an interest account is sufficient to make the

l

319

12 WH Y DIS COU NT?

t now the process being controlled we mus point.) To the variable x, describing who ner t-ow plan represents the capital of the add ano ther variable, Zt, say, which n rsio recu the y costs. This capital will obe is run ning the process and paying the (54) 1 at each stage and c, is to be if unused capital grows by a factor {3s net process cost to the plant-owner. Thu

Zh

= p-h ( zo -

L !Y c.,. h-1

)

regarded as the

.

-r=O

ose e terminal capital zh then he will cho If the plant-owner wishes to maximis if or ar line is U If U tion some utility func policy 1r to maximise E... [U(zh)] for the n of this will be equivalent to minimisatio there is no statistical variation then and the not, will it s case er p-r c.,.). In all oth expected discounted cost E... CE~:~ lem. prob the of able the state vari current capital z 1 must figure as part of izon t that one mus t have an infinite-hor poin the still is Of course, there e care is stationary opti mal policy, and som formulation if one is to derive a that because the surplus of capital above needed in finding such a formulation, ld, cou growing exponentially. However, one needed to cover net process costs is at will z ation of the probability of ruin: that for example, consider the minimis some future time fall below zero. of the point is that it is the non-linearity This is in a sense a side-issue. The el as mod the conventional discounted-cost exponential criterion which exposes ete in its state description. essentially incomplete, and incompl Exercises and comments unted one wishes to minimise total undisco (1) Suppose that the criterion is that one ory, hist , conditional on current process cost incu rred over a lifetime, and that ion isat xim g the the next unit time step. (Ma has constant probability f3 of survivin cost into but this can be translated of reward would be more natural, is using optimiser has survived to age t and minimisation) Then, given that the er is eaft ion which he must extremise ther an exponential criterion, the express

- f3)(3j = th, a rand om variable, and (1 Here t is the time of his dea tion is solu mal is invariant in t, and its opti P(t = i + jlt ~ t). This problem amics dyn and s es LQ G assumptions on cost stationary However, even if one mak G LEQ that lies imp of this last expression (apart from those of death), the form structure has been lost.

320

RISK-SENSITIVITY: THE LEQC MODEL

Notes on the literature The LEQG criterion was first investigated by Jacobson (1973, 1977), who solved the perfect-observation state-structured case and saw the role of the noise variable as an independent auxiliary control. This treatment was extended by Speyer (1976) and Speyer, Deyst and Jacobson (1974) and Kumar and van Schuppen (1981). However, the general state-structured case with imperfect observation resisted solution until Whittle (1981) deduced the risk-sensitive certainty equivalence principle; the assumption of state structure was dropped in Whittle (1986). Whittle did not observe the point that the extremal operations in equation (8) do not commute in general; this point was corrected by Fragopoulos (1994). However, it has no consequences if the appropriate saddlepoint exists, and is located by stationarity conditions. The entropy evaluation (47) of the average cost under LEQG assumptions is due to Glover and Doyle (1988). Chang and Sobel (1987) observed that the combination of an LEQG criterion and discounting implied non-stationarity of the optimal policy.

CH AP TE R1 7

The H00 Formulation 1 THE H oo LIMIT In Section 16.10 we quoted the expression "fa( B)=

4~&

J

log[ji + BG(iw)G( -iw)Tj] dw =

4~8

J

log[ji + BGG!] dw, (1)

rate of cost incurred in the stationary derived in Section 13.5, for the average sensitivity parameter e. This relation, regime und er the LEQG criterion with risk8), expresses the average cost very first derived by Glover and Doyle (198 function G induced by a stationary, explicitly in terms of the system transfer onse' we mean the response of a linear stabilising policy. By 'system resp dardised white-noise input. We are standardised deviation output to a stan n G(£i2) and so transfer function G(s). assuming that the equivalent filter has actio in the discrete-and continuous-time The limits of integration are ±n or ±oo cases respectively. by some linear stationary stabilising The class of response functions induced The aim of steady-state optimisation is policy constitutes the class of feasible G. minimal in an appropriate norm , and to choose a G in this class which is on G which is implied by choice of the expression (1) can be regarded as the norm immediately relevant properties of this LEQ G criterion. Let us summarise the norm. rable over the relevant frequency range. Theorem 17.1.1 Assume that GG is integ Then: + BGG is positive definite for all real (i) In the range e > () 0 for which the matrix I increasing in e. w, expression (1) is positive,finite, and nontive, below which ro(B) is no longer nega ily (ii) The critical value () 0 , necessar defined is determined by (2) -B( / = c?(G)

the maximal eigenvalue ofGG. where a1 (G) is the maximal value (over w) of (16.48) to expression (1) implies that Proof The normalisation of expression negative. It follows then that the criter9\ ~ 0, and so that the cost function is nonall real() for which it is defined; we see ion x1r(B) of (16.1.3) is non-negative for

322

THE Hoo FORMULATION

from Exercise 16.1.3 that it is also non-increasing in B. These properti es then transfer to the limit evaluation /G (B). Integrability of GG (which implies that G is proper, in the continuous-time case) implies integrability oflogj/ + BGGI as long as the matrix I + BGG remains positive definite. As 8 is decreased this condition fails first at()= ()G· As we shall see by example, the average cost /G(B) may or may not become infinite at that of value, but its definition certainl y fails from that point, because the derivation 0 ness. -definite evaluation (1) in Section 13.5 required positive Since the class of control rules corresponding to the feasible G includes the optimal stationa ry rules at any given value of fJ we have

Theorem 17.1.2 The breakdown value 8 of() in the stationa ry case can be identified with infGBG, where the infinum is over feasible values of G. Equivalently, (3) -e-l = inf al{G). G

The quantity cr( G), the non-negative square root of c?( G), is also a norm on G. It is known as the Hoo norm of G, otherwise written

IIGJioo = cr(G). ry The conclusion of Theorem 17.1.2 can then be expressed: an optimal stationa s minimise it that in , -optimal H also is 8 00 point wn breakdo LEQG control rule at the . function transfer the HX! norm ofthe effective This is the essential implication of the Glover-Doyle paper, and was found particularly striking because the Hoo criterion had attracte d intense interest since about 1981, for reasons we shall explain in the next two sections. The realisation that it is so closely related to the LEQG criterion, whose investigation had proceeded completely independently, marked a real breakthrough in understanding. We can make the immedi ate assertio n that the Hoo problem is solved for the . state-structured problem of the last chapter by the LEQG-o ptimal policies we deduced in Sections 16.4-16.7, applied at the breakdown point B = 8. However, should make the connect ion more explicit. If the actual white-noise input has covariance matrix m and if the cost replaced by attach~d to an output deviation !::,. is !::,. Tmt::,. then the form GG is evaluation affect not does which mGmG, to within a similarity transfor mation be wcould 9\ and m both of eigenvalues, and so of the Hoo norm. Indeed, n of inclusio and noise te dependent, correspo nding to allowance of non-whi system the r lagged terms in the quadrat ic cost function. Conside

!

dx+fl u

=f

y+~X='17

u = Jf"y

(4)

1 TH E Hoo LIMIT

323

matrix (123) an d the the white with covariance are uts inp ise no the where rate-interpretations in the is given by (12.4) (with ion nct for t-fu cos us eo tan ins tan pu re regulation pro ble m are thus considering a We e). va cas ser e ob , im on s-t ati ou nu equ conti pectively pla nt res te itu nst co (4) of s which the three relation s the solution e. Suppose system (5) ha rul l tro con d an on ati rel tio n erator :Y{ to mi nim ise the tha t which chooses the op is e rul o Bo al tim op the Th en er w) of the ma tri x al over eigenvalues an d ov im ax (m e alu env eig al ma xim G( -iw)T. G(iw) [ [ the rules de du ce d in state-structured case by the in ved sol is m ble breakdown value 8 = 9. Th is pro stationary lim it an d at the the in en e tak , 6.7 4-1 16. s Section linear Markov for the cas ptimal control rules are o-o Bo ter the lat t se tha n the the of s ns It follow uivalent versio ion, an d are certainty-eq ate 1 of pe rfe ct state observat the mi nim al stress est im is, at servation. Th ob te sta ct rfe nic al pe no im 'ca of e the , in the cas However tuted for x 1 in the rule. sti sub is ) see ate m, im for est ed ear tur te-struc at 9 (a lin tpu t form rat he r tha n sta ou utinp in is d en ite giv plo en ex oft e we hav model' is dynamical str uc tur e which the t tha s an me ich wh Section 3, largely lost. is a useful one, since it co nc ep t of B 00-optimality the er eth wh er nd wo y On e ma = 9 at which LEQGyty at the very po int (} ali tim -op QG LE to s 1 an d kn ow alread correspond by example in Exercise see ll sha we As ls. fai n d Kb ec om e inf ini te at optimisatio unstable the n bo th II an is nt pla the if , 9.1 16. m these quantities, bu t fro mT he ore n finite solutions exist for the ble sta is nt pla the If this point. perturbations. in fact unstable to positive its usefulness in the the determination of II is no rm has rat he r be en o Bo the of int po the xt two sections. However, which we develop in the ne rty pe pro a s, nes ust rob an B 2 criterion, tre atm en t of uld be sai d to be ba sed on wo e rul G LQ ry na tio sta Th e optimal . adratic no rm Jtr( GG) dw in tha t it minimises the qu

~ ~]

~

t]

x

Exercises and comments rule in the simplest case: nation of the B 00 -op tim al mi ter de ect dir the r de continuous-t im e case. (1) Consi tion to zero in the scalar ula reg n tio va ser ob ctrfe tha t of pe normalised to zero. If uctured mo de l with S str testa al usu the e the transfer function We ass um ising policy u = Kx the n bil sta a ts op ad e on d an N = wwT pa ir (x, u) is nt noise to the deviation from the normalised pla

G(s) =

[~] s~r

324 where r

THE Hoo FORMULATION

= A + BK. It follows that (with s = I

I

+

OHHI = 1 BN(R + QK2)

+

and so that (}-

iw)

= - [·1UKf sup w

w2

+r2

BN(R + QK2 )] -I ..? rz w-+

The maximising value of w is zero, so the H 00 -optimal value of K minimises (R + QK2 )/(A + BK) 2 subject to stability, i.e. to A+ BK < 0. The freely minimising value is at K = BR /AQ and is acceptable if A< 0; we have then -ON = B2 / Q + A 2 j R. If A ;;;:: 0 the n the minimising value of K (subject to stability) is K infinite and of opp osite sign to B, corresponding to -O N= B2/ Q. We thus confirm the identity of the H00 -optimal rule with the LEQG-optimal rule at the breakdown point, set out in Theorem 16.9.1. 2 CHARACTERISTICS OF TH E Hoo NORM The H 00 nor m was originally intr oduced for considerations which are not at all stochastic in nature, let alone LEQ G. In order to und ers tan d these we should establish some properties of the norm.

Theorem 17.2.1 Consider the case in which G is a constant matrix, so that II Gil~ is the maximal eigenvalue of GGT. We then have the characterisations 2 IIGII co

_ -

jG8j 2 _ tr(GTMG) 2 - sup (M ) o 181 M tr

sup

(5)

where the suprema are respectively over non-zero vectors 8 and non-zer o non-negative definite matrices M ofappropriate dimension. Proof The second expression in (5) is sup6((8T GT G8)/(8T 8)] which is indeed the· maximal eigenvalue of GT G and so also of GGT. If M has spe ctral representation M = L_1 >.AoJ, where the eigenvalues >..1 are necessarily non-negative, then

tr(GTMG) _ L_>..1 IG8/ tr(M ) - L_.>.1 18i which plainly has the same sha rp upp er bou nd as does the sec ond expression 0 One might express the first charac terisation verbally as: II Gil~ is the maximal 'energy' amplification that G can achieve when applied to a vector. Consider now in~

2 CHARACTERISTICS OF THE Hoo NORM

325

response function of a fllter with the dynamic case, when G( iw) is the frequency action G(!'J).

ion G(s) Theorem 17.2.2 In the case ofafilter with transfer funct IIGII2 = su EIG(9))812 p £1812 00

nary vector processes {8(t)} of where the supremum is over the class of statio 2 appropriate dimension for which 0 < Ej8j < oo. distribution matrix F(w). Then Proof Suppose that the 6-process has spectral we can write

EjG(!'J)oj 2 _ ftr(G dF G) Jtr(d F) Ejoj 2 -

definite matrix. We thus see from But the increment dF = dF(w) is a non-negative rem that the sharp uppe r boun d the second characterisation of the previous theo F) is the supremum over w of the to this last expression (under variations of The boun d is attained when 6( t) maximal eigenvalue of GG, which is just II Gil;,. 0 vector amplitude. is a pure sinusoid of appropriate frequency and previous theorem: II Gil;, can be We thus have the dynamic extension of the tion that the filter G can achieve characterised as the maximal 'power' amplifica for a stationary input. on of this last theorem which we shall not Finall~ there is a deterministic versi then give a finite-dimensional prove. We quote only the continuous-time case, in Exercise 2. Suppose that the proo f in Exercise 1 and a counterexample class, in that is is causal as well as response function G(s) belongs to the Hardy closed right half of the complex stable. This implies that G(s) is analytic in the eigenvalue of G(s) G(sl (over plane and (not trivially) that the maximal tive real part) is attained for a value eigenvalues and over complex s with non-nega s = iw on the imaginary axis. One can assert:

class then Theorem 17.2.3 Ifthefilter G belongs to the Hardy IG(!'J)8j2dt II GII2 = su J p J j6j2df I 00

6( t) ofappropriate dimension. where the supremum is over non-zero vector signals in that 6 is a deterministic This differs from the situation of Theorem 17.2.2 y stochastic signal of finite power. signal of finite energy rathe r than a stationar expectation. The assertion is that An integration over time replaces the statistical

THE Hoo FORMULATION

326

IIGII~ is the maximal amplification of 'total energy' that G can achieve when applied to a signal. Since we wish G to be 'small' in the control context, it is apparent from the above that adoption of the HXJ criterion amounts to design for protection against the 'worst case~ This is consistent with the fact that LEQG design is increasingly pessimistic as edecreases, and reaches blackest pessimism at e = 0. The contrast is with the H 2 or risk-neutral case, e = e, when one designs with the average case in mind. However, the importance of the Hoo criterion over the past few years has derived, not from its minimax character as such, but rather from its suitability for the analysis of robustness of design. This suitability stems from the property, easily established from the characterisations of the last two theorems, that

(6) Exercises and comments (1) The output from a discrete-time filter with input 81 and transient response g1 is y 1 = ~rgrDt-r· We wish to find a sequence ~81 } whose 'energy' is amplified maximally by the filter, in that (~ 1 1Ytl 2 / ~~ 181 1 is maximal. Consider the SISO case, and suppose time periodic in that gr has period m and all the sums over time are restricted to m consecutive values. Show (by appeal to Theorem 17.2.1 and Exercise 13.2.1, if desired) that the energy ratio is maximised for 81 = eiw1 with w some multiple of 2rr/m, and that the value of the ratio is then IG(iw)l 2 = I ~Tgre-iwrl2· (2) Consider the realisable SISO continuous-time filter with transient response 1 . Then II Gil;, is just the maximal eat and so transfer function G(s) = (s2 value of IG(iw)l , which is a- 2. Consider a signal 8(t) = e-f3t for t ~ 0, zero otherwise, where (3 is positive. Show that the ratio of integrated squared output to integrated squared input is

ar

2(3(a +

f3r2loo

(e2at- 2e(a-f3)t

+ e-2f3t) dt.

If a< 0 (so that the filter is causal) then this reduces to (a2- a(J)- 1 which is indeed less than a- 2 . If a > 0 (so that the filter is not causal) then the expression is infinite.

3 THE Hoo CRITERION AND ROBUSTNESS Suppose that a control rule is designed for a particular plant, and so presumably behaves well for that plant (in that, primarily, it stabilises adherence to set points or command signals). The rule is robust if it continues to behave well even if the actual plant deviates somewhat from that specified. The concept of robustness

327

N AN D ROBUSTNESS 3 TH E Hoo CR ITE RIO

G

u w K

+

of G(s) corresponding to a ofequation ( 7). lfa pole em syst the of m not be comple· gra dia ck Figure I A blo the controlled system will ed by a zero ofK(s), then plant instability is cancell tely stable.

may never be kn ow n tha t the plant structure t fac the of t un co ac as in statistics an d oth er thus takes time. In control theory, in ry va eed ind y ma d for optimality mu st be exactly, an grown that a co nc ern s ha ion ict nv co the , both qualities mu st be subjects, robustness. Furthermore for ern nc co a by ted en between goals which are complem the right compromise ch rea to is e on if ied quantif nflicting. ou tpu t necessarily somewhat co 1, designed to make plant ure Fig of tem sys the r For an example, conside nsfer functions of pla nt Here G an d K are the tra w. l na sig d an mm co a ed from observed pla nt v follow nt ou tpu t v is distinguish pla l ua act alent d an er, oll ntr an d co e block diagram is equiv is observation noise. Th TJ ere wh J, r ' + v = y t outpu (7) to the relations u = K( w- v- TJ) v= Gu , pressi whence we deduce the ex

on

(8) 1 - GKTJ) v- w =( I+ GK )- (w tem. Fo r stable of the inputs to the sys ms ter in w v or err have a stable causal for the tracking erator I+ GK should op the t tha e uir req all gain theorem operation we holds; the so-called sm ity bil sta t tha e um ass inverse. Let us t IIGKIIoo < l. ndition ensuring this is tha or to co mm an d asserts tha t a sufficient co tra functions of cking err nse po res the t tha (8) We see from vely vation noise are respecti signal an d (negative) obser

norm. They ca nn ot all, in some appropriate sm be to se the of ty of the th bo e One would lik is known as the sensitivi S se 1 + S2 =I . S 1 cau be , ver we ho , bo th be small

328

THE

n)O FORMULATION

system; its norm measures perform ance in that it measures the relative tracking error. S2 is known as the complementary sensitivity, and actually provides a measure· of robustness (or, rather, of lack of robustness) in that it measures the sensitivity of perform ance to a change in plant specification. This is plausible, in that noise-corruption of plant output is a kind of plant perturbation. Howeve r, for an explicit demonstration, suppose the plant operato r perturb ed from G to G + 8G. We see from the correspondingly perturb ed version of equations (7) that

v =(I+ GKr 1GK(w- TJ) +(I+ GKr 1(8G)K( w- v- TJ). The perturb ed system will remain stable if the operator I+ (I+ GK)- 1(8G)K acting on v has a stable causal inverse. It follows from another application of the small gain theorem that this continued stability will hold if

i.e. if the relative perturba tion in plant is less than the reciprocal of the complementary sensitivity, in the Hoo norm. Actually, one should take of the expected scale and dynamics of the inputs to the system. This isaccount achieved by setting w = W1 w and TJ = Wtii, say, where W 1 and W2 are prescribed filters. In the statistical LEQG approach one would regard wand ij as standard vector white noise variables. In the worst-case deterministic approach one would generate the class of typical inputs by letting w and ij vary in the class of signals of unit total energy. Performance and robustness are then measured by the smallness of IIS1 W1lloo and IIS2 W2lloo respectively. Specifically, the upper bound on I!G- 18GIIoo which ensures continued stability is

IIS2W211~1 .

Of course, in a simple minimisation of quadratic costs a compromise will be struck between 'minimisation' of the two operators sl and s2 in some norm. There will be a quadratic term in the tracking error v - w in the cost function , and this will lead to an expression in H 2 norms of S1 and S2. The more observation noise there is, the greater the consideration given to decreasing S 2, so assuring increased robustness. The advantages of an Hoo formulation were demonstrated first in Zames (1981), who began an analysis which has since been brought to a high technical level by Doyle, Francis and Glover, among others; see Francis (1987) and and Doyle, Francis and Tannenbaum (1992). The standard formulation of the problem is the system formulation of equations (6.44)/ (6.45), expressed in the block diagram of Figure 4.4. The design problem is phrased as the choice of K to minimise the response of A to ( in the Hoo norm, subject to the condition that 'K should stabilise the system', i.e. that all system outputs should be stabilised against all system inputs. The analysis of this formulation has generated a large specialist literature. We shall simply list a number of observations.

-~- ._-:• .•.·

:1

ROBUSTNESS 3 THE H 00 CRI TER ION AND

329

man y n to the LEQ G criterion means that (i) The relation of the H 00 criterio d LE QG tion in the now well-establishe Hoo problems already have a solu tion noise The need to take account of observa extension of classical LQ G ideas. criteria, s derived on either LQ G or LEQG ensures a degree of robustness in rule es. ht not have bee n apparent before Zam although this is an insight which mig utp ut ut-o inp in ed stat are in this section (ii) The two problems formulated rath er than simply by its response function G form, in tha t the plant is specified of attempts ber num a n bee e mple. The re hav by a state-structured model, for exa ework, s directly, usually in the LQ G fram over the years to attack such problem dra tic qua a ses imi min ch filter K in (7) whi by, for example, simply seeking the eta!. t Hol 7), (195 ser Kai Newton, Gou ld and (See ~). T9t E(~ as h suc n erio crit e a cer tain rno and Jabr (1976)). One can mak (1960),Whittle (1963),Youla, Bongio roach can app n d-o hea a ed in Section 6.7, amount of progress, but, as we not almost aled reve are ch s two insights whi prove perplexing and tends to mis se are The . blem pro ary lysis of the non-station automatically in a state-space ana and ion mat esti of ects separation of the asp s; (i) certainty-equivalence, with its able vari te juga con of of the introduction control, and (ii) the usefulness the by ted stitu con with the constraints Lagrange multipliers associated ies the ation of these properties simplif loit Exp prescribed dynamic relations. vector case. analysis radically, especially in the without of optimal stationary controls We consider the determination hod s met al tegr e-in pters 18-21, but using tim reduction to state structure in Cha t is wha y isel prec by their natu re and yield which incorporate these insights needed. tegral niques associated with these time-in (iii) The operator factorisation tech ck atta ct dire a in Wie ner -Ho pf methods used methods are distinct from bot h the ded oun exp es mat rix /-factorisation techniqu on the inp ut-o utp ut model and the of the Hoo a and Glover (1990) for solution by e.g. Vidyasagar (1985) and Mustaf problem. ility e nor m will in fact assure overall stab (iv) Simple minimisation of~ in som s of sure mea sically achievable) if Ll includes of the system (provided this is phy eria crit the example, the inclusion of u itself in relevant signals in the system. For ility or avoids the possibility that good stab used through the whole of this text , and ther Fur expense of infinite control forces. tracking could be achieved at the of ree e deg on will automatically achieve som as we have seen, such an optimisati assumed in the model. robustness if observation noise is of an mate guarantee of robustness is use Finally, one might say tha t the ulti n we can tion which opens wider vistas tha adaptive control rule; an observa explore.

I"

PART 4

d Time-integral Methods acnies Optimal Stationary Poli al stationary policy termination of the optim de t ec dir the to ted vo eneous, bu t no t This Pa rt is de LEQG an d time-homog or G LQ ed os pp su is that th e subfor a model which is an interesting one in me the e Th . ed tur uc um principle, necessarily state-str uivalence, the maxim eq y int rta ce ls, gra coalesce themes of time-inte d policy improvement an n tio isa tor fac al nic no eady applied to the transform methods, ca velopment of those alr de a are ds tho me e ralised in Section 6.5. naturally. Th in Section 6.3 and gene se ca ed tur uc str testa uld omit this Part deterministic state-structured case co the ly on r ide ns co to nt for enlightenThe reader conte h passing up a chance ug tho (al s low fol at wh without prejudice to w!). timal ment, in the author's vie deriving a stationary op of t tha is red ide ns co neral neither stateThe problem flrst to be eous LQ G model, in ge en og om e-h tim a for y ally assume the control polic Ou r methods intrinsic . ble va ser ob y ctl rfe pe derived as the structured no r t the optimal policy is tha so , its lim on riz ho on policy. existence of infinitean optimal finite-horiz of ry) na tio sta t fac (in i.e. of infinite-horizon limit of being average-optimal: y ert op pr the ve ha y inl stationary regime. It Such a policy will certa urred pe r un it time in the inc st co ge era av the on to transients, minimising erty of optimising reacti op pr r ge on str the ve ha is such that it will in fact also t have unless plant noise no y ma y lic po al tim which an average-op of the system. policy is can stimulate all modes in formula (13.24), and st co ge era av the for :ft" which one uses to We have an expression lisable linear operator rea t an ari nv e-i tim the e might regard the specified by current observables. On of ms ter in u ol ntr isation of this express the co matter of direct minim a as ply sim m ble optimisation pro

332

TIME-INTEGRAL METHODS

expression with respect to :It. This was the approach taken by a number of auth ors in the early days of control optimisati on (see Newton, Gould and Kaiser (1957), Holt et al. (1960), Whittle (1963)), whe n it seemed a natural developmen t of the techniques employed by Wiener for optimal prediction. However, while the approach yields results rather easily in the case of scalar variables, in the vector case one is led to equations which seem neither tractable nor transparent. In fact, by attacking the problem in this bull-like fashion one is forgoing all the insights of certainty equivalence, the Kalman filter etc. One could could argue that, if these insights had not already been gained, they should be revealed in any natural approach. If so, then this is not one. In fact, the seeming simplif ication of narrowing the problem down to one of average-optimisation blinds one to an even more direct approach. This is an approach which is familia r in the deterministic case and which turns out to be available even in the stochas tic case: the extremisation of a time-integral. We use this term in a rather specific technical sense; by a time-integral we mean a sum or integral over time of a fun ction of current variables of the mo del in which expectations are absent and which is such that the optimal valu es of decisions and estimates can be obt ained by a free and unconstrained extremisation of the integral. In earlier publications on this topi c the author has referred to these as 'pathintegrals', but this is a usage inconsis tent with the quantum-theoretic use of the term. Strictly speaking, a path-int egral is an integral over paths (i.e. an expectation over the many paths whi ch are possible) whereas a time-integr al is an integral along a path. The fact which makes substantial progress possible is that a path-integral can often be express ed as an extremum over time-integr als. For example, we we saw in Chapter 16 that the expectation (i.e. the path-int egral) E[exp( -OC)J could be expressed as an extremum of the str ess §= C + 1 o- [). If one clears matrix inverses from the stress by Legendre transformations (i.e. by introducing Lagrange multipliers to take account of the contraints of pla nt and observation equations) then one has the expectation exactly in the form of the extremum of a time-integral. , It is this reduction which we have exploited in Chapters 6 and 16, sha ll exploit for a general class of LQG and LEQ G models in this Part, and shall exte nd (under scaling assumptions) to the non -LQG case in Part 5. We saw in Section 6.3 that the state-st ructured LQ problem could be conver ted to a time-integral formulation by the introduction of Lagrange multiplier s, and that the powerful technique of can onical factorisation then determi ned the optimal stationary policy almost imm ediatel)t We saw in Sections 6.5 and 6.6 that these techniques extended directly to models which were not state-st ructured. These solutions extend to the stoc hastic and imperfectly observed case by the simple substitution of estimates for unobservables, justified by the cert aintyequivalence principle. We shall see in Chapter 20 that time-integral tech niq ues also take care of the estimation pro blem (the familiar control/estimation duality

TIME-INTEGRAL METHODS

333

methods extend to the finding perfect expression) and, in Chapt er 21, that these to the non-LQG case ion extens LEQG model. All these results are exact, but the ion of the pathximat appro of Part 5 is approximate in the same sense as is the n integrals) of (actio integrals of quant um mechanics by the time-integrals ximation at sensitive classical mechanics, with refmement to a higher-order appro parts of the trajectory. al formalism first, There is a case, then, for developing the general time- integr al pattern, unclu ttered which we do in Chap ter 18. In this way one sees the gener ation to control and applic the in arise by the special features which necessarily estimation. models are no longe r There is one point which should be made. Although our input/ outpu t form. in given state-structured, they are not so general that they are state-structured the in Rather, we assume plant and observation equations, as p. The loss value some case, but allow variables in them to occur to any lag up to mode l to the reduces of an explicit dynamic relationship which occurs when one n 6.7. input/ outpu t form has severe consequences, as we saw in Sectio r's previous work autho the of n versio lined stream This Part gives a somewhat ver, the mater ial of on time-integral methods as set out in Whittle (1990a). Howe edge, new. There knowl r's autho the of Sections 20.3-20.6 and 21.3 is, to the best r would be autho the and er, howev must be points of conta ct in the literature, grateful for notice ofthese.

l

CHAPTER18

The Time-integral Formalism CRETE TIME 1 QUADRATIC INTEGRALS IN DIS the term 'time-integral' even in discrete For uniformity we shall continue to use then a sum. Consider the integral in the time, despite the fact that the 'integral' is variable~

(1) is then a sum over time of a quad ratic with prescribed coefficients G and (. This the function being time-invariant in its function of the vector sequence {~T }, Sections 6.3 and 6.5 ~ is the vector with second-degree part. So, for the models of ying term (T would arise from kno wn sub-vectors x, u and >., and the time-var als r, uc. The matrix coefficients Gj and disturbances d or known com man d sign but no generality is lost by imposition of the vector coefficients (T are specified, the normalisation

(2) then the 'end terms' arise arise from If the sum in (1) runs over h1 < r < h2, h2 respectively. The final term can be contributions at times r ~ h1 and r ?: and the initial term as arising from a regarded as arising from a terminal cost itions. probabilistic specification of initial cond such that we wish to extremise the Suppose the optimisation problem is variable {~T}. We cann ot at the mom ent integral with respect to the course of the see by considering again the models of be more specific than 'extremise', as we respect to x and u and maximised with Cha pter 6, for which one minimised with condition with respect to ~T is easily seen respect to A. In any case, the stationarity to be

(3) and h2 that neither end term involves if r is sufficiently remote from both h1 where

IP(!/) :=

L j

Gj§-J.

~n

(4)

336

THE TIME-INTEGRAL FORMALISM

The normalisation (2) has the implication =(z- 1 onz-transfor ms. Suppose that we regard our optimisation as a 'forward' optimisation in that, if t is the current instant of time, then ~Tis already determined forT< t, and we can optimise only for r ~ t. We shall then write equation (3) rather as

r

(r

~

t)

(6) to emphasise this dependence on t. That is, relation (6) determ ines the optimal course of the process~ from timet onwards for a prescribed past (at timet). This was exactly the situation which prevailed for the control optimi sation of Sections 6.3 and 6.5. If the horizon h2 becomes infinite then we may indeed deman d that (6) should hold for all r ~ t and, if the sequen ce {(T} and the terminal term are sufficiently well-behaved, expect that the semi-i nversion ofthe system (6) analogous to equation (6.27) should be valid. For the estimation problems of Chapter 20 we shall see that the same ideas apply, but with the optimisation applied to the past rather than the future (cf. Sectio n 12.9). To consider the general formulation (1) is then already a simplificatio n, in that we see that these two cases can be unified, and that the operat or
If
(z) = ¢(z)
(7) where both ¢(z) and ¢(zr 1 have expansions in non-negative power z valid in ·lzi ~ 1. We have interposed a constant matrix factor o (necessarilys of symmetric) for generality. It could be normalised to the identity by redefin ition of ¢ (see Exercise 1), but we shall not find this to be the natural norma lisation. If we assume 'dynamics of order p' in that G1 = 0 for ljl > p then it will turn out that ¢( z) is a polynomial in z of degree p at most. As in Chapt er 6, we can semi-invert system (6) to obtain a closer determination of the optimal future course off Theorem 18.1.1 Suppose that
¢(.r){~) =
(r ~ t)

(8)

IN THE 'MARKOV' CASE 2 FACTORISATION AND RECURSION

337

---1

the subscript T and that ¢( f/) is to be (r;ith the understandings that f7 operates on right-hand member of (8) is defined. expanded in non-positive powers of ff) if the ys as p7 for increasing T, where pis less This latter condition will be satisfied if deca than the radius ofconvergence of¢( z)- .

fr

recursion (6), symmetric in past and The passage from (6) to (8) converts the recursion, expressing ~~r) for T ~ t in future in the sense (5), to a stable forward t-hand side of (8) constitutes a kno wn terms of past values ~~l(a < r); the righ d complete the inversion to obta in a driving term for this recursion. One coul is positively undesirable. Relation (8) complete determination, but to do so known ~r for t - p :::;; T < t) and, as we determines ~?) explicitly (in terms of the pter 6, this determination yields the saw for the control application of Cha is both closed-loop and natural for realoptimal control rule in the form which time realisation. orizon limit h2 -> oo for system (6) Of course, if (8) is to hold in the infinite-h The analogue of the controllability/ then cert ain conditions must be satisfied. , which were the basic ones for the sensitivity conditions of Theorem 6.1.3 ly that a canonical factorisation (7) existence of infmite-horizon limits, is simp also be required of the term inal should exist. Regularity conditions will of the path integral (1). contributions implied in the specification ite-horizon solution (8) is formally If such conditions are granted, then the infin e we must find a workable way of immediate. However, to make it workabl We shall achieve this by returning to the deriving the canonical factorisation (7). g and relating the canonical factor ¢ to recursive ideas of dynamic programmin need never be determined; it is enough the value function. This value function e of policy improvement implies a that the dynamic programming techniqu ation of the canonical factorisation. rapidly convergent algorithm for determin rkov' case, for which dynamics are of Let us begin with consideration of the 'Ma order one and recursions are simple.

Exercises and comments ld be invertible in stable form is that it (1) A necessary condition that 4>(z) shou e, which implies that all three factors should be non-singular for z on the unit circl icient condition for canonical factorisin (7) should be non-singular there. A suff tive definite on the unit circle. This latter ability is that 4>(z) should be strictly posi analysis, however, because the quadratic condition will in general not hold in our ed convex/concave character. form constituted by Dhas in general a mix IN THE 'MARKOV' CASE 2 FACTORISATION AND RECURSION (cross-terms) appear in the time-integral If G1 = 0 for Ul > p, so that interactions that we are dealing with pth- orde r only at time lags up to p, we shall say

338

THE TIME-INTEGRAL FORMALISM

dynamics. We shall then refer to the case p = 1 as the Markov or state-structured case. In the Markov case the time-integral can be written U=

L

Cr

+ end terms,

T

where

(9) Consider now a recursive extremisation of the integral, with cr regarded as an 'instantaneous cost', and with the closing cost at the horizon point h assumed to 2 be a quadratic function of ~h2 alone. If we are interested in a forward optimisation then we can define a 'value function' F(~1 , t), this being the extremised 'cost' from timet onwards for given ~1 . This will obey the dynamic program ming equation F(~t,

t) =stat [c1 + F(~t+l, t + 1)]. (t+l

(10)

where 'stat' signifies evaluation of the bracket at a value of ~t+l which renders it stationary. As a dynamic program ming equation this has the simplifying feature that one optimises with respect to the whole vector argume nt ~ rather than just some part of it. In the case under conside ration the value function will have the quadratic form

(11) for t ~ h2 if it does so at hz, with the coefficients (respectively matrix, vector and scalar) obeying the backward recursions lit

= Go - G_tii;;1Gt

at= (t- G_tii;;1at+I

(12)

81 = 81+1- a~ 1 1IH\at+1·

The first of these is just the Riccati recursion; strikingly simpler in this general formulation than it was when written out for the state-str uctured case of the control problem in equations (2.25) I (2.26). The second relation likewise corresponds to the equation before (2.65). The third gives the recursion for the ~­ indepen dent compon ent of cost incurred which we thought too troublesome to derive in Section 2.9. The extremising value of ~t+ 1 in (10) is determined by

(13) Suppose that we are in the infinite horizon limit h 2 ~ +oo, and that the problem is well-behaved in that II 1 has, for fixed t, a limit II. It is under these conditio ns that one expects the forward recursion (13) for ~~ to be stable in that its solution

URSION 3 VALUE FUN CTI ONS AND REC

S

339

a stable t. In othe r words, that II + G1 ff is should tend to zero with increasing to the one s lead s Thi 1 a stability matrix. operator, or that r = - II- G1 is l to ona orti prop be ht mig factor (z) of
(14)

onical from (13) that this is indeed the can is a factorisation of
(,. =

p-1

Cr

e;_ jGp -jer -r = !{;_p+l Go{r-p+l + L j=O

(15)

Ifwe define the value function

(16) then this will obey the optimality (dyn

amic programming) equation

[c, + F({t+l, e~> F({ t,et- 1, ... , {t-p+l; t) = stat (,+I

... ,et-p+2; t + 1)].

(17)

endence purely quadratic with a limited dep We suppose that the term inal cost is F will ord er dynamics then implies that on the past. The assumption of pthfact be ables indicated in (16), and it will in inde ed be a function of just the vari two) in thee-variables: homogeneous quadratic (i.e. of degree

340

THE TIME-INTEGRAL FORMALISM

say. The 'optimal polic y' will then be of a p-lag linea r form in that the extremal criterion in (17) will deter mine ~t+ 1linearly in term s of the p previous values of~: p

I>l! k( t + 1)~t-k+ 1 = k=O

o

(19)

say, with a:0 necessarily symmetric. Our expectation is that the coefficients a:k(t + 1) will beco me inde pend ent oft in the infin

ite-horizon limit (if this exists) and that (19) will then be a stable forw ard recu rsion which, written ¢(ff)~t+l = 0, defines the cano nical factor ¢(z). The argu men t follows. If we substitute the form (19) into the optimality equa tion (17) we obta in a 'superRiccati' equa tion for the p(p + 1) /2 matr ices IT1k which is most forbidding. The relations beco me muc h more com pact in term s of the generating functions

F(w,z; t) =

_L _LII;k(t)wiz'< j


k p

p

}=I

}=I

L G1z1 + L G_1wi

¢(z; t + 1) = F(O, z; t + 1) + GpzP

(20)

(21) (22)

p

a:(z;t+ 1) = _La :k(t+ 1)!'

(23)

k=O

in the complex scalars z and w. We shall refer to F ( w, z : t) as the value function transform. We shall now find it convenient to suppose that F is not necessarily the value function corre spon ding to an optimal polic y, but that corre spon ding to an arbit rary p-lag linea r policy of the form (19). It will then still have the homogeneous quad ratic form (18).

Theorem 18.3.1 (i) The value function transform under the policy (19) obeys the backward recursion (dynamic programming equa tion)

F(w, z; t) = (wz f 1 [(wz)P
t + 1.

{24)

carry the additional argument

RISATION RELATIONSHIP 4 A CENT RAL RESULT: THE RICCATIIFACTO

(ii) The optimal choice ofa is a(z; t + 1) = cjJ(z; t + 1) = F(O, z; t + 1) + GpzP With this choice (24) reduces to the optimality equation 1 ¢(w) T¢01¢(z)]t+ 1. F(w,z; t) = (wz)- 1 [(wz)P(z- 1 , w- ) + F(w ,z)-

341

(25)

(26)

in Whittle (1990a) p. 164. The proo f is by heavy verification; the detail is given the right -han d member of (24) One can see that the terms independent of a in (18) and the translation of the!originate from the presence of the cost term c1 in nate from the fact that ~t+l is subscripts of the ~-arguments. The terms in a origi expressed in terms of earlier ~-values by (19). in (26) we have an elegant Although derivation may be heavy, we see that an elegant expression of what expression of the optimality equation, and in (25) term s of the value function in ¢ r will soon be identified as a canonical facto generating furnction (21), le doub transform F Note the need to introduce the related to (z) by

(27)

(z) = (z- 1,z).

TORISATION 4 A CEN TRA L RESULT: THE RICCATI/FAC REL ATIO NSH IP unde r an optimal policy then If we assume the existence of infinite-horizon limits ler. simp the assertions of Theorem 18.3.1 beco me even

al Theorem 18.4.1 (i) In the infinite-horizon limit the optim has the evaluation

in terms of

cjJ(z) = F(O,z)

+ GpzP.

(ii) The equation ¢(5")~.,.

= 0

(r

~

t)

value function transform

(29)

(30)

holds along the optimalpath. The assertions also begin to have a very clear signi (28) we obtain

ficance. If we set w = z- 1 in

since the forward recur This is nothing but a canonical factorisation of (z), (30) is stable. We thus deduce

(31) sion

342

THE TIME-INTEGRAL FORMALISM

Theorem 18.4.2 (i) If infinite-horizon limit s exist then \P has a canonicalfactorisation (7) with cp(z) apth -ord er polynomial. (ii) This factorisation can be given the spec ial form (31), with cp(z) having identification (29). It then has the specialfeatu res (32) ¢o = Iloo is symmetric o

(33)

= ¢o 1.

(34) These two theorems express the close and elegant relation between the infinitehorizon value function transform F(w, z) and the canonical factor cp(z). The double generating function F ( w, z) is expr essed in terms of the single generating function cp(z) by (28), and ¢ is expressed in terms of Fby (29). The first relation implies the factorisation characterisation (31). Note that we find the fact or¢ with the part icular characteristics (32) and (33) by a very direct route: we simply write dow n the stationarity condition with respect to ~t+I in the optimality equation (17). This yields a linear relation ¢(5" , t + 1) ~t+l = 0 determined by (22) who se coefficients have the properties (32) and (33); these coefficients become the coefficients of¢ (z) in the infinite-horizon limit. We must regard these properties as constitut ing a natural normalisation; a view which will be confirmed.

5 POLICY IMPROVEMENT AND SUC CESSIVE APPROXIMATION TO THE CANONICAL FACTORISATION The analysis of the last two sections is imp orta nt enough in itself. However, it also proves to be crucial for the dedu ction of a good iterative algorithm for the determination of the canonical factorisation. Essentially, the policy improvement algorithm turns out to imp ly a simple iterative algorithm for the determination of ¢ which makes no refer ence to the value function. The policy , improvement algorithm has itself a clea r variational characterisation and shows second-order convergence (see Section 3.5); these properties will transfer to the ¢;- version. We shall again suppose pth- orde r dyna mics. Let the stage of iteration be labelled by i = 0, 1, 2, ... and let p

¢(i)(z)

= L r/Jijzj j=O

denote the approximation to cp(z) at stag e i. Consider the procedure in which is determined from rP(i) by the linear equa tion system u+IJ¢if/¢(iJ + ¢uJ¢ir/¢u+t) = + ¢uJ¢ ir/¢i+I,o¢ir/rP(iJ· (35)

rP(i+i)

·'i

N 5 POLICY IMPROVEMENT AND SUCCESSIVE APPROXIMATIO

343

Theorem18.5.1 Recursion (35) has thefollowingproperties. m to (i) It is the recursion generated by application ofthe Newton-Raphson algorith the factorisation (31), regarded as an equation for¢. (ii) It is also the recursion which would be derived by application of the policyimprovement algorithm. is a (iii) It conserves the normalisation properties (32) and (33) in that, if ¢(i) (z) and the polynomial in z of degree p for which the absolute term ¢i0 is symmetric coefficient ¢ip ofzP equals GP' then the same is true of¢(i+l)(z). uProof Consid er first assertion (i). For notatio nal simplicity, denote the consec in 6.. by¢+ ing¢ Replac ¢+b.. tive trial solutions ¢(i) and ¢(i+I) by¢ and 7/J = obtain we b. (31) and expand ing as far as first-order terms in
= ¢¢()!¢ + ti¢()1¢ + ¢¢()! Ll + ¢¢()1 Llo¢()1¢.

Setting b. = 7/J - ¢ in this relation we deduce (35). can be Asserti on (iii) is easily confirm ed from relation (35). Assertion (ii) of proof the confirm ed by appeal to relations (24) and (25). More economically, LQ control equivalence of the two algorithms established for the state-st ructure d 0 case. general more this to r transfe problem in Section 3.5 will Proper ty (34) is implicitly conserved under the algorithm, in that we as

define ci>io

,+,-! 'l'iO.

be more Equati on (35) constitutes a linear equatio n for ¢(i+I)• but we can which is form a in ed express specific. The solutio n of this equatio n can in fact be ion expans series the explicit to within application of the operati on of truncat ing ion expans of a function. Suppose we have a functio nf(z) with a power series 00

f(z) =

L Jjz 1 J=-00

obtaine d by valid on the unit circle. Then we shall define [f(z))+ as the function ion: expans the retention only of terms for non-negative j in this 00

[f(z))+ =

Ljjzl . j=O

Theorem 18.5.2

The solution ofequation (35) can be expressed
= ¢,u[¢(;{ci>¢(tn+¢uJ·

Proof We can write relation (35) as ¢ii17/J¢-1 + ¢-Iif;¢()1 = ¢-l
(36)

344

THE TIME-INTEGRAL FORMALISM

in the ¢>, '1/J notati on introd uced earlier. Applying the trunca tion throughout, we deduce (36). It is worth while noting a point which is not immediately eviden t from (36): that the right-hand memb er of this equation is indeed a polynomial of degree pin z. We shall refer to (35) and its more specific form (36) as the PI/NR algor ithman abbreviation for 'policy improvement!Newton-Raphso n'. We shall apply the algorithm to partic ular control examples in the next chapt er and shall see then that it is useful, not merely for nume rical conclusions, but also for structural · ones. Explicitly:

Theorem 18.5.3 Suppose that a property of the normalised trial factor ¢>(i) is conserved under the PIINR algorithm, so that it is shared by ¢>U+lr Then the normalised canonicalfactor¢> has this property. Proof If the policy at stage i is non-optimal, so that ¢(i) is not a canonical factor, · then ¢(i) will be modified by the algorithm. The conservatio n ofa prope rty under the algorithm thus implies that it is shared by some canonical factor. But the normalised canonical factorisation is unique, so the property must be shared by ¢>. 0 6 THE INTEGRAL FORMALISM IN CONTINUOUS TIME The continuous-time analogue of the path-integral (1) will now indee d be an integral over time

6(~) =

j c(~) dT +end effects

(37)

where c(e) is a quadratic function of the vector variable ~(t). We shall assume the specific form

(sr)

ckJ

= cik(rs)

( 39 )

However, this normalisation is not the only possible recast ing of the integral (37). Partial integration yields the effective equality c[r]c[s] _ _ c[r-I]c( s+l] ':.j c.,k

':.J ':.k -

(r > 0),

(40)

IN CONTINUOUS TIME 6 THE INTEGRAL FORMALISM

345

the the substitution will change only we mea n by 'effective equality' that rior inte ts poin e tim at ons ity con diti and so does not affect the optimal range of integration. as far as to write the path-integral (37) We can car ry the reduction (40) so 0

=!

J2::: L I: L cj~s) r~~r+s] J ~ J[~ e~(~)~dr -

(-

(,T dr +en d effects,

or, more compactly,

(T (] dr +en d effects.

fi =

rators wit hjk th elem Here q,(~) is ann x n mat rix of ope

\Pjk(~) = LLc};s)(-~)'EPs.

(41)

ent (42)

s

the integral the ~-path for r ~ t should render Theorem 18.6.1 The condition that ·' · - (37) stationary can be written (43) ~ t) ( 1"

rator \P(.@) is rval of integration. The mat rix ope at time points interior to the inte Hermitian in that (44)

first the n te consequence of (39) and (42); the The sec ond asse rtio n is an imm edia as =(s)

= <J>_(s)+(s).

sym met ric ns tha t we can ind eed look for a The Her mit ian character of mea factorisation (45) (s) = ¢(s)oc/J(s) left hal f of e all thei r singularities strictly in the say, where bot h cjJ(s) and cjJ(s) -I hav nite-horizon -horizon limits exist then the infi the complex s-plane. If infinite to the stab le n by the semi-inversion of (43) solution is again effectively give forward equ atio n

(46)

for the optimising ~. ion of the there is a nat ura l normalisat As in the discrete-time case, from the n atio oris by ded ucti on of this fact factorisation (45), dete rmi ned

346-

TH E TIME-INTEGRAL FOR

MALISM

optimality equation of a recursive approach. We cover these matter s in the next section, which reveals both the analogies to and some difference s from the discrete-time material of Sections 4 and 5. 7 RECURSIONS AND FACTOR ISATIONS IN CONTINUOUS TIME In considering the factorisati on question we can again nor malise to the homogeneous ('regulation to zero') case ( =0. There is a difference between the discrete- and continuous-time cas does not seem more than notatio es which nal, but is annoyingly persistent . It refers to degree of dynamics, and has alre ady dynamics were of order p in the disc arisen in Section 4.7. When we said that rete-time case we really meant tha t the of order p at most. If they were in fact of lesser order in some relation y were s then this did not matter; the only require ment was that a recursion such as (19) should really be a determining forward recursion, in that the coefficient ao of at the current time-argument should be non-singular. The corresponding requirement in the continuous-time case is tha t the matrix coefficient of the differentials of highest order should be non-singul ar, and this order may well be different for diff erent components of €. Suppose that, under the normalisation (39), the differe ntials of ej occur in c( €) up to ord er rj exactly (j = 1, 2, ... , n). Then what one might term a minimal-order line ar policy would determine the forward evolution of eby a set oflinear relations

e

(j= l,2 , ... ,n),

(47)

say, where the coefficients l'i. may be time-dependent. The point is that the optimal policy will lie within this class, and that determination of the optimal relation (47) in the infinite-horizon limit implies a determination of the canonical factor if>(s). Let F( €, t) denote the value functio n under policy (47), starting from a known €history at time t. Although we hav e loosely written this as €-depende nt, in fact it will be dependent on the functio n €(r) through the differentials ej'l(r < rj; j = 1, 2, ... , n) at the current instant t. Under appropriate restriction s on the form of tlie terminal cost it follows then tha t the value function under policy (47) is of the homogeneous quadratic form

F(€,t)

=! I:I:I:I:rrj~s>(t)ej'let1 j

and that Fwill obey the backward

k

$

(48)

equation

c + oF + " " e!r+lJ oF = at ~ L.J ') adrl 0 J r
(49)

~th eYJ) expressed interms oflower-o rder derivatives by (47). Under an policy (49) is replaced by the optima optimal lity equation

7 RECURSIONS AND FACTORISATIONS IN CONTINUOUS TIME

.

stat [c +

0: + 2:2: ~J'+l] 8~1] = o t

a~j

r

j

347

(so)

where the stationarity is with respect to the highest·order derivatives ~[r,] (j = 1, 2, ... , n). " Again we introduce transforms, and define the matrices ofgenerating functions

F(s 1,s2)

=

(:L2:rrJ~1 ''2 )s~1 s;2 ) '2

,,

the t·dependence ofF and II being understood. The bracketed expression is the jkth element of the matrix in question, j and k taking values 1, 2, ... , n if ~ has dimension n. We again refer to F(s1, s2) as the value function transform. We shall have occasion to distinguish the parts of these generating functions associated with the highest· degree terms, and so define

H

= (hJk) = (c)2''k)) d(s)

(51)

= diag(s'j).

Ct (s) =

(L c)'Jt sv) v)

s
v)

) () ( "rr(i-l,v FtS= S. L:jk

Theorem 18.7.1 (i) The optimality equation for the value function transform F = F(st,s2) is 1 T 8F (52) c(st,s2) +at+ (st +s2)F- cf>(s!) H- cf>(s2) = 0, where

cf>(s) = Hd(s)

+ Ct (s) + F1 (s).

(53)

( ii) The optima/future course of~ is determined by the forward equation

(54) This is the analogue of Theorem 18.3.1 (ii), and proved analogously; for further details see Whittle (1990a) pp.l94-5. Note that, as previously, the form (53) of the canonical factor is determined simply from the stationarity condition in the dynamic program ming equation. This gives a forward differential equation for ~.

348

THE TIME-INTEGRAL FORMALISM

necessarily stable if infinite-horizon limits exist, and we write it as¢(£'))~= 0 to deduce (52). The way is now open for our principal conclusion; the analogue of Theorem 18.4.2.

Theorem 18.7.2 Suppose infinite-horizon limits exist for the optimal policy. Then the canonicalfactorisation

(55) holds with H equal to the known matrix (51) and cp(s) equal to the infinite-horizon limit ofexpression (53). This factorisation then satisfies the normalisation that both cf> 01 and the matrix coefficient of the highest-degree terms in cp(s) equal the known matrix H. Thisfollowssimplybythesubstitutionofs1 = -s2 = -sintheeq uilibrium form of (52), and appeal to the fact that (54) must be a stable forward recursion. The continuous-time case thus shows one simplification: the matrix H defined in (51) is known before the factorisation is attempted, while the matrix ¢ 0 of (31) is not.

8 THE PIINR ALGORITHM IN CONTINUOUS TIME The same simplification persists in the policy-improvement algorithm which is, as ever, identical with the Newton-R aphson algorithm applied to relation (45) as an equation in ¢. If¢(;) is the determination ofthe canonical factor at stage i then the recursion analogous to (35) is -

1

-

I

I ¢(i+l)n- ¢uJ + ¢uJH- ¢u+Il =+ ¢uJH¢uJ (56) and the normalisa tion indicated in Theorem 18.7.2 is conserved under the algorithm. The more explicit recursion (36) also has an analogue. Suppose a functionf( s) . is representable for purely imaginary s by the integral

f(s) =

L:

est

dg(t).

We then define the truncation operator [ ]+ in continuous time by

[f(s)]+

=

L:

e'1 dg(t).

Then, just as in Section 5, the solution of (56) can be expressed ¢(i+l) = H[¢~l¢~n+¢(i)·

(57)

The derivation of these results is very much as for the discrete-time case; the argument is amplified in Whittle (1990a), pp.197-8.

CHAPTER 19

: Optimal Stationary LQG Policies Perfect Observation EGRAL TION FROM THE TIME-INT 1 DERIVATION OF THE SOLU

e by pathof the perfect-observation cas nt tme trea the in s step ial The essent this section we up in the short Section 6.5. In integral methods are summed sections we two ing that discussion. In the follow simply recapitulate and expand sions. clu con n us chapter can be used to sharpe see how the results of the previo ume a plant equation In the discrete-time case we ass

(1)

{d } is a kno wn iant causal linear operators, 1 where d and f!J are time-invar ite noise with ances and {ft} is Gaussian wh deterministic sequence of disturb dynamics tha n the trix N. If we assume pth -or der zero mean and covariance ma f!J = B( f/) = s d = A(5") = I:~=O A,5 "' and form the e hav l wil rs rato ope and have malising assumption Ao =I , nor the ke ma can We "'. L:~=I B,5 ermined at umption tha t the control u1 det ass the to g din pon res cor 0, set B0 = ble fact that, if u1 e t + 1. This reflects the inevita time t has an effect first at tim sally dependent cau includes x~> then x 1 can not be may depend upo n dat a which function has the 2.4, that the instantaneous cost We shall suppose, as in Section m time-homogeneous quadratic for

OnUt.

Ct=C(Xt,Ut)=![~r[~ ~][~t

(2)

have assumed that, tf make no appearance here. We and r s nal sig nd ma com The zero, by the they have been normalised to n the e, anc adv in wn kno are if they variables and of as redefined process and control adoption of x - r and u - uc 2.9). If they are tion disturbance variable (cf. Sec d - d r - f!Juc as a redefined model, and so presume them generated in the not known in advance, then we s variable x. subsumed as par t of the proces der lags in the the incorporation of higher-or by c We could generalise 1 to start from the equation. However, it is easier nt pla the in e hav we as les, variab s at a later l generalisation becomes obviou ura nat the (2); n ptio um ass r familia point.

350

OPT IMA L STATIONARY LQG POL ICIES: PER FEC T OBSERVATIO N

The development from now on is exactly as in Sec tion 6.3. By the cert ainty equivalence prin cipl e the white-n oise term E will have no effect on control optimisation, and we can for the mom ent delete it (although it will cert ainl y play a role in the next two chapters). Regard the con seq uen t dete rmi nist ic process equ atio n as a con stra int and asso ciate with it a vector Lag rang e mul tiplier .A 1. The con seq uen t Lag rang ian form h-1

H=

I:c( x,., Un T) r=O

h

+ LA ;(d x+ ~u- d)T + ch.

(3)

r=l

constitutes our time-integral. Her e Ch is the closing cos t at tim e h; we could, consistently with othe r assumpt ions, sup pos e it a qua drat ic func tion of Xr (h - p < T ~

h).

We aga in mak e the dist inct ion betw een T, a general run nin g tim e variable , and t, which labels the mom ent 'now : In oth er words, we assu me that ur has alre ady bee n dete rmi ned for T < t, not necessarily optimally, and that Ur is to be dete rmi ned forT;::;: t. Ext rem isat ion of 0 with respect to (x, u, .A)T gives a line ar system of equ atio ns which we can writ e as

(t

~

T

< h- p).

(4)

The corr esp ond ing equations for h - p < r ~ h will be mod ifie d in that they will include con trib utio ns from C11. As in Section 6.5, fJJ and .91 are the ope rato rs conjugate to d and f!J in that, for exam ple, .91 = A(.:T- 1) T = L::=o A"[.:r'. So d operates into the pas t and .91 into the future. Not e that , effectively, Ar = 0 for T >h . The sup ersc ript (t) indicates that the opti mis atio n is one holding from tim et onwards. The values of xVl and u~1 l for r ;::;: t dete rmi ned by (4) and sub sequ ent equ atio ns con stitu te the prediction at tim e t of the future course of the opti mal ly con trol led process. Thi s will coin cide with the actu al cou rse only if ther e is no plan t noise E. However, the dete rmi nati on of u) 1l thus obta ined is opti mal und er all circumstances. If we write the equ atio n system (4) as

(5)

then this can be identified with the equ atio n (18.3) of the general path -int egra l trea tme nt. The mat rix function ( z) thus implicitly defi ned cert ainl y sati sfies the sym met ry requ irem ent ~ = of the general trea tme nt. Suppose that (z) has the can onic al factorisation

(6)

TROL RULE IN DISCRETE TIME 2 EXPRESSION OF THE OPTIMAL CON

351

of (18.311 the natural discrete-time where we have supposed the normalisation normalisation. Then. the semi-inversion

(7)

(r;;:: t)

y solution in the infinite-horizon of (5), if valid, provides the optimal stationar limit. constitutes a stable forward relation It provides the solution in that relation (7) ally-controlled process. However, determining the predicted course of the optim gives an explicit expression for the more importantly, relation (7) for r = t shall make this determination mor e optimal value of u1 in closed-loop form. We well claim it as the most direct, explicit in the following section; one may opti mal stationary policy. economic and natural determination of the ion (6) should exist is just the The condition that a canonical factorisat ld exist in the case ( 0; the case of condition that infmite-horizon limits shou right-hand member of (7) shou ld be 'regulation to zero~ The condition that the future) is just the condition that the finite (with the operator acting into the to cope with the input (; see the optimally controlled system should be able discussion after Theorem 6.3.1. ediate. The operators in the plan t The continuous-time analogue is fairly imm al operator~. so that, for example, equation are now polynomials in the differenti integrals in (3), and relation (4) d = A(!~) = Er Ar~'. Sums are replaced by of conjugacy. In analogue to (5) one holds with the definition il =A ( -~)T writes this system as

=

(8)

(r;;?: t) ~ is unde rstoo d as

where a time argument ris understood and the canonical factorisation

d/dr . lfe)( s) has

(9) of Theorem 18.7.2, where we have supposed the normalisation of the semi-inversion (7) is

(r;;?: t).

I I

I

(10)

TRO L RULE IN DIS CRE TE 2 EXP RES SIO N OF THE OPT IMA L CON TIM E rating function i.P(z) has the form For the LQG control problem the matrix gene

i

I

then the analogue

S(z) A(z )l Q(z) B(z) . 0 A(z) B(z)

R(z)

i.P(z)

= [ S(z)

(11)

352

OPTIMA L STATIONARY LQG POLICIES: PERFEC T OBSERVATION

Here have in fact generalised the cost function somewhat by replacing

the matrix

R by a generating function R(z), etc. This corresp onds to the inclusion oflagg ed variables in the cost functio n c1; the generalisation can be made painles sly at this

point and the conclus ions we now reach remain valid under it. Recall that Bo = 0, andsup poseth enorma lisation Ao =I. This form is special enough that we can say someth ing about the form of the canoni cal factor ¢(z) in the normal ised factorisation (9).

Theorem 19.2.1 The canonicalfactor¢ ofthe matrix generating function (11), under the normalisation of Theorem 18.4.2, has theform ¢xx ¢= [ ¢ux

¢xu /] ¢uu 0 , B 0

A

(12)

the partitioning being the (x, u, >.) partitioning of~. and an argument z being understood throughout. If the cost function involves lags ofsize p - 1 at most then the submatrices with the ¢-labels are polynomials in z ofdegree p - 1 at most. Proof Validity of the form (12) follows by appeal to the recursive algorit hm (18.35). We leave the reader to verify that, if ¢(i) has the form (12), then so does ¢(i+I). So then does¢, by Theore m 18.5.3. The fmal assertio n follows from the fact

that, under the assump tion stated,

¢p

= Gp = [ ~

Ap

0 0] 0

0

Bp

0

j

see (18.32).

0

Theorem 19.22 Suppose that infinite-horizon limits exist. Then the optimal value ofUt is given explicitly and in closed-loop form by --

¢ux(ff)Xt + ¢uu(ff)ut = [¢o¢(ff-)1 Ju.A,

(13)

where the u>. subscript on the bracket indicates the extraction of the corresp onding submatrix in the (x, u, >.)-partitioning of the bracketed matrix. If the cost function involves lags ofsize p - 1 at most then relation (13) expresses the optimal u in terms 1 of(Xr, Xt-I, ... ,Xt-p+I), (ut-I. Ut-2 1 ••• , Ut-p+I)and (dt, dt+l, ... ). Proof Relation (13) follows if we extract the u-subvector relationship from relation (7) at r = t, taking accoun t of the form asserte d for¢ in Theore m 9.2.1. 0 The closed-loop determ ination (13) of the optima l control is explicit to within achievement of the canoni cal factorisation, and is both as neat and as explicit a solution as the general case will allow.

TINUOUS TIME 3 OPTIMAL CONTROL RULE IN CON

353

e insig ht if we intro duce the s:ystem We gain both som e econ omy and som m (4) in the generalised case (11) as nota tion of Sect ion 6.6, and write the syste

) IU(ff)] [x] (t) = [OJ [m(ff d / A 0 21(5")

(14)

T

where then

X=[~],

I!!= [d £!6']

The asse rtion (12) is then that the and the plan t equa tion reduces to lUx =d. norm alise d cano nica l facto r ¢ takes the form

¢(z) _ [ ¢xx(z) IU(z) -

l!lo]

0

(15)

'

ix poly nom ial IU(z). where 910 is the abso lute term in the matr rem 19.2.1, but in the mor e econ omic al We can follow thro ugh the proo f of Theo (15) directly. However, we have to retu rn system nota tion throu ghou t, to dedu ce ion (13) if we wish to disti ngui sh the to the mor e explicit nota tion of relat rule explicitly. part icula r role of u, and dedu ce the cont rol

Exercises and comments

= - Bff and there are no lagg ed term s (1) For the Mar kov case d = I - Aff, £!6' nica l facto r ¢(z) of\II(z) has the form in the cost function. The norm alise d cano

R+A TIIA ¢(z) = [ S +Br iTA

1-A z

sr +AT IIB /] Q + BTIIB

0

-Bz

0

whe re II is the matr ix of the infin ite-h orizo

T

=

[m + ~ IIil11

n value function F (x).

TROL RULE IN 3 EXPRESSION OF THE OPTIMAL CON CONTINUOUS TIME 18.7, that, for re, discussed alrea dy in Sections 4.7 and

We agai n have the featu of the discrete-time case is the matr ix insta nce, the anal ogue of the matr ix Ao atives of com pone nts of x occu rring coefficient, A. say, of the high est-o rder deriv is the high est orde r of deriv ative of r in the plan t equation. Tha t is, supp ose that 1 in the plan t equation. If A, has jkth the jth com pone nt x1 of x that occu rs com pone nt aj~l then

(16)

354

OP TI MA L STATIO NARY LQ G POLICI ES: PE RF EC T OBSE RVATION

This matrix must be no n-singular if the plant equation is to constitut relation which determ e a dynamic ines the forward path of the process. We corre define B. as the matr spondingly ix coefficient in f!J of the highest-order deriv occurring in the plant atives of u equation or the cost fu nction c, and Q* as the these highest-order de matrix in c of rivatives. For the control problem the matrix
(17)

The form of H follows from its general definiti on in (18.51), and the fo the argument of Theore rm of ¢ by m 18.2.1. The fact that B* is in ge neral non-zero makes the solution for the op of u(t) a little more co timal value mplicated than the co rresponding expressio discrete-time case. From n (13) for the relation (10) at r = t we deduce that ¢xx(~)x + ¢xu(~)u

+ AYA = Px>.(~)d Y A= Pu>.(!!))d

ifJux(~)x + ¢uu(~)u +B

whereP(~} = H¢(!!)) -I andPx>.(:!)) etc. indi

cates the corresponding All quantities are evalu submatrix. ated at time t. Eliminati ng A we deduce the relat ion [ifJuxU~)- B* A; 1¢xx(! !))jx + [¢uu(!!))- B*A:;- 1 ¢xu(!!))ju = [Pu>.(!!))- B*A-; 1 (18) Px>.(~)Jd. This must be regarded as a forward differentia l equation for the optim by terms in differentia al u, driven ls of known x. It can, of course, be solved in terms, but would be transform generated in real tim e by a mechanism rea differential equation. lising the The relation simplifie s considerably under certain conditions. Su example, that u does no ppose, for t appear in differentiat ed form in either cost plant equation and that function or the cross-terms betwee n u and x in the cost fu been normalised to zero. nction have Then
=

R(s) [ 0

0

Q A(s) Bo

A(s)

BJ 0

l

(19)

3 OPTIMAL CONTROL RULE IN CONTINUOUS TIME

355

Q is now a simple matrix, as is B0• Let us suppose, for simplicity, that A. has normalised to the identity. Then· one, finds, by the arguments of Theorem ·.· ·1s.1.2, that the expressions (17) reduce further: to

l

0 I Q BJ , I Bo 0

0 H = [0

¢{s) =

[rPxx(s) 0 A(s)

0 Q

Bo

l

I . BJ

0

It follows then that (18) simplifies to

.u = Q- 1{BJ¢xx(!'d)x + [Pu>.(~)- B6Px>.(~)]d}. Exercises and comments (1) The return difference equation Consider again the classic feedback loop of Figure 4.2. The operator -:f{'r§ is the loop operator, giving the effect on a signal which passes successively through the plant and the controller, and ; = ..F + :f{'r§ is the return difference operator, giving the difference in effectis · between no traverse and one traverse of the loop. If a disturbance d superimposed on the control u as it is fed into the plant then u satisfies J u = d. Let J(s) be the continuous-time transfer function of .f. The return difference equation is a version of the inequality JQJ > Q, which holds for LQ-optimal control, at least under state-assumptions, if Q is the penalty matrix for contr<Jl. It expresses the fact that the operation ; -1 attenuates at all frequencies. The relation is very easily proved from the canonical factorisation of with . factors of the known form (17). If the column vector with subvectors x, u and .X is written b.., let the solution of ¢b..= 0 in terms ofu be written b..= Du. Then the return difference equation amounts simply to (20)

h(j)n- 1¢D = hD. Show that in the special case (19) relation (20) implies ]QJ =

Q+BA- 1RA- 1B,

where J = I - Q- 1B"[ rPxxA- 1B. Show that in the case 0 R(s) Q(s) (s) = [ 0 A(s) B(s)

~]

B(s) , 0

special only in that S(s) has been normalised to zero, relation (20) amounts to

JQ.J= Q+BA- 1RA- 1B, where J = Q;:- 1{[¢uu(2)) - B*A;:- 1¢xu]- [¢ux- B.A;:- 1¢xx]A- 1B}. In both cases J is indeed the return difference transfer function for the optimal control.

CHAPT ER20

Optimal Stationary LQG Policies: Imperfect Observation 1 mE PROCESS/OBSERVATION MODEL: APPEAL TO CERTAINTY EQUIVALENCE We assume the linear model of equation (19.1) together with an observa.tion relation: d x + E!lu = d + f

(1)

y+~x= TJ.

(2)

This specification covers both the discrete- and continuous-time formulations; the time variable has not been explicitly indicated The operator ~ is, like d and E!4, a causal translation-invariant linear operator. Let us initially discuss the discrete-time case withpth-ord er dynamics, with CG having the form C(fr) = L:f= 1 C,fr'. As ever, dis a deterministic disturbance term and f and TJ are plant and observation noise respective!~ We suppose that these noise terms jointly constitute Gaussian white noise with zero mean and covariance matrix

In this case of imperfect observation the information W1 available at time t consists of the observation and control histories Y1 and U,_ 1, plus the complete course of the deterministic component of disturbance {d1}. We shall ultimately be passing to the stationary regime, in which the past as well a.s the future is infmite, so that observation and control histories extend into the infinite past. Since the model is totally LQG we can appeal to the certainty equivalence principle of Section 12.3 to deduce the optimal control in this imperfectly observed case from that for perfect observation. We know from the analysis of Section 19.2 that, in the case of perfect observation, the optimal stationary determination of u1 is given in closed-loop form by

(3)

358

OP TI MA L STATIONA RY LQG POLICIES

Th e left-hand member of this relation gives the feedback compon control, expressing u ent of optimal 1 in terms of x, (t p < T ~ t) an d u, right-hand member giv (t p < T < t). Th e es the feedforward ter m, in terms of d, (T ;;::: In the case of imperfe t). ct observation recursio n (3) for the optimal holds, except th at x, mu control still st be replaced, where it occurs, by the curre estimate x~l. We are th nt projection en led to the inference problem; the determi estimates. The duality na tio n of these of estimation an d cont rol has already been for the Markov case in demonstrated Section 12.9; we shall see how this extends to general order. dynamics of

2 PROCESS ESTIM ATION IN TIME-IN TEGRAL FORM (DISCRETE TIME) Th e characterisation th at we shall take of projection estimates least-square property is no t the linear asserted in (ii) of Th eorem 12.6.3, bu t ra th probability-maximisi er the dual ng (or discrepancy-mi nimising) property as the same theorem. It se rte d in (iii) of is this characterisatio n which yields the na integral formulation. tural timeA related po in t is th at canonical factorisa factorisations of some tions are then thing like the reciproc al of an AG F rather th LLS approach associa an (as in the ted with the Wiener fil ter) a factorisation of means that, in the ca an AGF. This se of pt h- or de r dyna mics, the factors are degreep. polynomials of We ca n set up the prob lem rigorously un de r the supposition th at ob began at a finite time servation h1, an d can th en pass to the infinite-history limit ht ju st as we passed to the --> -o o, infinite-horizon limit for control optimisatio will then be restricted n. Histories histories, so th at Xt is {x,; h1 - p ~ T ~ t}, etc negative exponent in th . Then the e Gaussian density of X 1 an d Y1 for prescr discrepancy ibed U1_ 1 is the

i0 1 =priorterms+!t[E]T[~ r=hr

'T] r

ft ]- l[ E ], 'T]

r where E an d ry are ex pressed in terms of x, y and u by appeal to observation relations the plant and (1) an d (2). Th e 'pr io r ter ms' reflect the distribut for relevant system hi ion assumed story before time h 1· In the case ofpt h- or de r will constitute a quad dy namics they ratic function of {x ,y , u,; h 1 - p ~ r < ht}. The projection estimate s x~l are ju st the value s of x,. minimising 10 thus amounts to a back 1 • Estimation ward rather th an a forw ard optimisation prob integral (or a precur lem; a timesor to one) is to be extremised over its co current m om en t t rath ur se before the er th an after. Actually, we would no t regard extremisation of 101 as extremisation integral, because it of a timeis subject to the co nstraints implied by observation relations the pl an t an d (1) an d (2). Le t us eli minate these by the in troduction of

2 PROCESS ESTIMATION IN TIME-INTE GRAL FORM

359

Lagrangian multiplier vectors l.r and m 7 for the constraints constituted by the relations at time T and so extremise a Lagrangian form [l)t

+ L:W (dx +P-lu-d -E) + mT (y +
7

T

with [1) 1 expressed in terms of the noise variables as above. Minimisation of this form with respect to the noise variables reduces the problem to extremisation of the past path integral t

llp(l, m,x)

=

(prior terms)+ L[v(l, m) -IT(dx + P-lu- d)- mT (y +
(4) Here

is the informational analogue of c(x, u). Strictly; we should should give the multipliers I and m superscripts (t), to indicate that they are the multipliers associated with optimisation on the basis of observables at time t. The relation between the multipliers and the noise estimates is

The direct interpretation of the multipliers is the familiar one of sensitivities: of Wl and m~l as respectively the rates of change of the minimal value of [)) 1 (subject to constraints (1) and (2)) with changes in the actual values of Er and TJr· Note one effect of the transformation from [)l, to flp; the expression for the timeintegral has been cleared of all inverses of covariance matrices. Thus one puzzling and seemingly perverse feature associated with a stochastic formulation is removed: that it actually seems to become anomalous if some noise components are zero, and so the noise covariance matrix is singular. One cannot directly relate the multipliers lr and A.r associated with the plant equation at past and future times. This is because the past optimisation takes precedence: one first minimises [)l with respect to unobservables and then C with respect to undetermined controls. This is again a reflection of the degenerate character of the risk-neutral model. When we come to the risk-sensitive formulation in the next chapter we shall see that effectively there is a single timeintegral which spans both past and future, with the consequence that past and future multipliers can be related. The estimates x~t) of process history at timet are obtained by minimising ll)r with respect to the corresponding Xr- The path integral llp must then be

360

OP TIM AL STATIONARY

LQ G POLICIES

ma xim ise d with respect to these variables and mi nim ise d wit h res pec t to cor res pon din g/, m. We ded uce the n the stationarity condition s

[fr

~ ~] [~~] (t) = [d -_:u] (t)

..ci'~O

XT

0

(hr < r ~ t);

the estimation analogue of the set of control optimisat ion relations we rewrote (19.4) in the con den sed for m (19.5), so we rewrite (5) as

(ht say. Th e ter mi nal con dit ion s for

=0

(5) :

T

(19.4~

Just as

< 'T ~ t),

(6)

this equation set are effectiv ely

) =0 (r> t), ' T (7) since ter ms for r > t do no t app ear in Dp. Th e initial con ditions are provided by the stationarity conditions forT~ ht. which dev iate from the pat ter n (5) in they will involve the pri or dis tha t tribution. Note tha t relation s (7) imply tha t (6) has a solution ind epe nde nt of the values of x~> for r > t, bec aus e x~l simply does no t occ ur in the equation system (5) for T > t. We shall refer to thi s feature as the lack of forward coupling. It reflec ts the fact tha t the estima tes of pre sen t and pas t variables do no t dep end upo n the estimates of future var iables. On e can con tra st it with the essential pre sen ce of backward coupling in the cor res pon din g control relation (19.5); pre sen t con tro l is undoubtedly expressed in ter ms of pre sen t and pas t values of the process var iable. Relation (6) plus its mo dif ied version at earlier tim e poi nts determines in principle all pas t estimates x~l. We wish to solve these equations efficiently, but we really only nee d to solve the m for the estimates x~l ( t - p < r ~ t) which are nee ded for im ple me nta tio n of the con tro l rule. In the inf inite-history lim it this poi nt is again me t by app eal to a can oni cal factorisation ; this tim e of '11 (z ). In the infinite-history limit h 1 -oo equations (5) and (6) hol d for all r ~ t. If w~ ass um e tha t '11 (z) has a can oni cal factorisation J(t) T

m
.;~

(8) which we have taken in the nor ma lise d form analogous to (19.6), the n the system (6) can be semi-inverted to (r ~ t). (9) He re the ope rat or 'lj;(!T) act s int o the future, with an effe ctive bou nda ry condition x~> = 0 (r > t) implied by the pro per ty of lack of for ward coupling. Th e ofterator 'lj;(!T)- 1 acts int o the lation (9) for r =!d ete rm ine x/>) explicitly. Th e relations forpasr t.= Re s x~e) (and so t - 1, t - 2 ... the n det erm ine • 1 X1(t)_ 1,x1(t) the _ 2 , ••• recurstve values of y.

\

.·~-1

' .,i

3 THE PARALLEL KALMAN FILTER (DISCRETE TIME)

361

With this one would seem to have estimates of the process variable at the relevant times in the form one would wish. This is not quite true, however. Rather than an expression for x~r) in the form of an infinite sum involving all past observations (which is what (9) yields if we set r = t) we would wish to generate this estimate by some simple updating recursion, as was achieved by the Kalman filter in the state-structured case. The question is, then, whether the backward recursion with respect to r implied by (9) can be converted into a natural forward recursion with respect to t: the pth-order equivalent of the Kalman filter. We shall see in succeeding sections that this conversion is almost immediate. Note one necessary difference between factorisations (19.6) and (8): the order of factors is now reversed, in that the factor 1/J{z) = :L:}=0 1/ljzl, for which both .,P(z) and its matrix inverse have expansions in non-negative powers of z valid in lzl ~ 1, is now the initial factor. Otherwise the factorisation is the complete analogue of (19.6) (e.g., in that 1/Jo is the constant term in 1/J(z) and is symmetric) and can be achieved by the same policy-improvement algorithm. Of course, the improvement is in inference rule rather than control rule. The existence of a canonical factorisation (8) is again the essential condition that infinite-history limits should exist for the estimation rule and the distribution of estimation errors. Exercises and comments (1) For the Markov cased = I - Aff, fJl = - Bff and fl = - Cff the normalised canonical factor '1/J(z) of w(z) has the form N +AVAT L+AVCT '1/J(z) = [ LT + CVAT M + VCVT 0 I

I -Azl -Cz 0

where Vis the limit value of the covariance matrix of the estimation error .X - x (cf. Exercise 19.2.1).

3 THE PARALLEL KALMAN FILTER (DISCRETE TIME) The innovation in the observations now has the form /" _

(t-1) _

.,, - Yr - Yr

rA

(t-1)

- Yr + -.xt

·

(10)

Here the time-translation operator acts, as ever, only on the subscript, so that

rtx~t-l) = :L:~ 1 Crx~~~I). We assume the normalised form (8) of the canonical factorisation.

Theorem 20.3.1 (The parallel Kalman filter) dated by the relations

The estimates x~l (r ~ t) are up-

362

OPTI MAL STATIONARY LQG POLICIES "" d + ( d - I ) X 1(t-1) + VfJUt ~ t + Ho(r, x(tl = x(t-1 ) + H r T T (r < t), f-T<,t

x 1(t)

( 11) (12)

where ( 1 is the innovation (I 0) and the matrix coeff icients Hj are determined by

L HjZ-j = -['1/i(z) -llxm· 00

H(z) :=

J=O

(13)

Equa tion (11) is recognisable as a generalise d form of the Kalm an filter; it must now be supp leme nted by the relations (12) whic h upda te also the estimates of the lagg ed x-values. The real novelty of the theo rem lies in the com pact evaluation (13) of the coefficients Hj in term s of the cano nical factor ¢. The xm factor indicates that we extra ct the corre spon ding subm atrix from the (j, m, x)parti tione d matr ix. We term relat ion (11) plus relations (12) for 0 < j < p the parallel filter because it simultaneously upda tes estimates of the all the comp onen ts of what would in fact be a state vector. It cont rasts with anot her possible version of the Kalm an filter, the serial filter, which emer ges in the next section.

Proof Note the cruc ial difference between relat ions (19.5) and (6) on which we have already comm ented . Relations (19.5) coup le back into the past in that they involve Xr for 7 < t. However, relations (6) do not couple forw ard into the future; future values of x are not involved and future values of land mare zero, as asser ted in (7). This abse nce offo rwar d coupling impl ies an effective boun dary cond ition

(14)

We dedu ce from (6) that

w(.r)(xVlxV- 1l)

= P!- 1 - P~- 1

(15) :aut the right -han d mem ber of (15) is equal to zero for 7 < t and to the colu mn vector 'Yr with parti tion (0, -(1 , 0) forT = t. Premultiplying relat ion (15) by the oper ator '1/1( Y) -I '1/Jo we thus dedu ce that

(r:::; t). This, toge ther with the effective boun dary cond

-h1-T Xr(t) - Xr(t-1 )- - ht-r'Y t-

ition (14), implies that

[~] ~

(r:::; t),

(16)

where h1 is the coefficient of z-i in the expa nsion of -'1/i(z) -I in non-positive powe rs of z. This demo nstra tes the validity of (12) for r :::; t with the evaluation

363

4 THE SERIAL KALMAN FILTER (DISCRETE TIME)

(13) of the coefficients Hi. For the particular case r for x~c-l) from the relation (t-1)

.s;l Xc

= t we deduce an expressi()n

+ !!4ur = d,,

so deducing (11) from (12) for the case r

=

0

t.

Exercises and comments (1) Consider again the Markov case, for which the canonical factor '¢ is given in 1 Exercise 1 of Section 2. The xm submatrix of -i/1( z) - is

H(z)

= H +z- 1 V(I- nTz- 1r 1 cT(M +

cvcT)- 1

(17)

where H = (L + AVCT)(M + CVCT)- 1 and 0 =A- HC. Then Ho indeed has the standard evaluation H. The relations (12) for r < t are of only academic 1 1 interest in this case, but we see from (17) that Hi= V[nTy- cT(M + CVCT)for j > 0. Part of the reason why this formula does not hold at j = 0 is tb.at observation of y 1 gives some informatio n on the value of t:. 1 if L =/:. 0 (i.e. if plant and observatio n noise are correlated). This helps in the estimation of Xc. but not of earlier x-values.

4 THE SERIAL KALMAN FILTER (DISCRETE TIME) Direct operations on the relation (6) yield an interesting alternative form of the higher-order Kalman filter. Let us define Xc as'¢(?/) -l Pt, i.e as the solution of '¢(?/)x

(18)

= p.

Theorem 20.4.1 The vector

(19) can be identified with x) 1l. In particular, current process variable.

x can be identified with xl l, the estimate of 1

1

Proof It follows from (6) and (18) that '¢() 1 '¢(?/)x~l =

Xr

(20)

But, because of the lack of forward couplin~ and the fact that '¢o is the absolute D term in i/1, relation (20) for r = t reduces to x/) = Xr-

364

OPT IMA L STATIONARY LQG POL ICIES

One mig ht define the serial estimate of process hist ory at tim et as {x.,.; r ~ t}3 and the upd ated or revised estimate as {x~lr ~ t}. Correspondingly, the best pred icto r of y 1 at time t- 1 is y~t-! ) = - E:=l C,x;~-;:'l, whereas the seria l predictor is 1 = - L::=l CrXt-r· Cor resp ond ing to the noti on of the inno vati on ( 1 = y 1 - y;t-I ) is then that of the serial innovation

y

1;

s

l

J (z= Yr- Yr= Yt + l?fxt. (21) By the same argu men t as that which proved the special form (19.12) for the ·~ cano nica l factor ¢of iP we ded uce that the cano nica l factor 1j; has the form J

1./J =

[

1./J11 1./Jtm 1./Jmt 1./Jmm

I

where an argu men t f7 or z is und erst

(22) .

0

ood .

Theorem 20A.2 (Th e seri al Kal man filte r) forward pair ofrecursions

The variables x and mobey the stable

dx + !Jiu = d + 1./Jzm(fl)m 1/lmm(fl)m = y so that xis determined recursively by the

(23)

+ l?lx

(24)

serial Kalman filter

dx + ~u = d + 1Ptm(fi)1./Jmm(ffr 1 (,

itselfa stable forward recursion.

(25)

Proof Written in full, relation (18) beco mes

1./Ju 1./Jtm [ 1./Jml 1./Jmm I 0

dl [-=/x ] [a- f?lul 1?1 0

m

'T

-y 0

(26) 'T

From this it follows that l = 0 and that the equation system reduces to (23), (24). J This redu ced system implies the dete rmin atio n m= 1/J;;;~ ( and the Kal man filter recursion (25) for x. Stability of all relations as forward recursio ns is gua rant eed by the cano nica l char acte r of .,P.

0

Relation (25) has inde ed the char acte r of the classical Kal man filter, in that it is equivalent to the driving of a plan t mod el by the serial innovations, or of a plan t/ observation mod el by the observations . The parallel filter of the last section has rath er the character of the driving of a state-reduced plan t mod el by the innovations; the serial filter avoids such a reduction. However, the fact that the serial filter works on serial innovations mea ns that the driving term 1/JJm'I/J;;;~ s (is

365

5 INVALIDITY OF THE SERIAL FILTER s

s

general not a function of current ( alone, but also of past (. Some ~orr111 .,u.,••~u·u is required to take account of the fact that history has been .,~t;m:atea serially. The best way to view relation (25) is in fact to revert to the equation-pair (23), (24) and regard this as a coupled system of plant/observation model and compensator driven by the observations. Of course, the reason why l1 = 1~ 1) is zero for all tis that the plant equation at t constitutes no essential constraint; the variable x 1 appears in this relation alone and can be given the value which satisfies it best, without affecting the estimates of earlier history or their fit. The following result clarifies the character of m1 and relates the two innovations. Theorem 20.4.3

The serial and parallel innovations are related by -1

-1 s

A

(27)

WommC = m = Wmm ( ·

T = t plus the lack of forward coupling implies that Because 1/J has the form (22) this last relation implies that /~ 1) -/(t-l) = 0, which we know, and also that 'l/Jomm(m) 1l - m)t-l)) = (1• Since m;t-!) = 0 the first equality of (27) thus follows; the second we know D from(24).

Proof Relation (16) at

1/Jo(xit) - xir-!))

= It·

Finally, summation of the equation x~"") - x~""-!) over the range T < a ~ t leads to the conclusion

= H
Theorem 20.4.4 The updated estimates of past process values are obtained from the serial estimates .X by the formula

X~) = Xr +

1-T

1-T

j=l

j=l

L Hj(r+j = Xr + L Hj'l/JOmmmr+J

(T~t).

(28)

5 THE CONTINUOUS-TIM E CASE; INVALIDITY OF THE SERIAL FILTER Interesting points arise in the continuous-time case. The transfer from the discrete-time case can not be taken mechanically, and some points are delicate enough to affect implementation. When it comes to estimation then relation (5), written again as (6), still holds, with the boundary condition of lack of forward coupling. We appeal to a canonical factorisation (29) However, it is now sometimes advantageous to vary the normalisation from that indicated in Theorem 18.7.2. In the next section we shall demonstrate that a

366

OPT IMA L STATIONARY LQG POL ICIES

factorisation can be foun d of the form (22); see (35). It will then also follow that , if we agai n define x( t) as the solu tion of (18), then .X( t) can aga in be iden tifie d with the esti mat e of curr ent proc ess valu e x(tl(t). Furt herm ore, it follows from the form of '1jJ that .X and fh obey the anal ogu es of (23), (24)

dx + r16u = d + '1/Jtm(~)fh '1/Jmm(~)fh = Y +~X,

x

so that agai n obeys the seri al Kal man filter s

dx + :18u

(30)

relation, anal ogo us to (25),

= d + 1/JJm(~)1/Jmm(!!))- 1 (s .

(31) Her e (is agai n the seri al inno vati on. The true and seri al inno vati ons now have the definitions s (=y+~x

(32)

respectively. Her e we have used x,( t) to den ote the rth differential!?)' x( t) of x at t and .X,( t) to den ote its proj ecti on esti mat e on info rma tion at t:

x,(t) = lim~'x(tl(-r). Tjt

(33)

Not e that !?) acts, as ever, on the runn ing time argu men t 7 in (33). However, rela tion s (31) and (32) have only a form al validity. The inno vati on (is not differentiable with resp ect to time ~since it has a whi te-n oise com pon ent) . Nei ther in fact is the seri al inno vati on (, and the fact that equ atio n (32) s rela tes differentials of .X to differentials of ( is an indi cati on that som e of the diffe rentials of do not exist either. Equ atio ns (30) are prop er in that they defi ne a stable filter with inpu t y and well-defined outp uts x and fh. However, the equ atio ns themselves con stitu te an imp rope r real isati on of this filter, in that they expr ess relations betw een signals (variables) whi ch are so ill-d efin ed mat hem atic ally as to be hopelessly ill-c ond ition ed phys ically. · In orde r to obta in a set of upd atin g relations in which all variables are well defi ned we have to reve rt to the para llel filter and so, effectively, to a stat e redu ctio n of the model. Thi s goes rath er agai nst our prog ram me, one of who se elements was the refusal to reso rt to stat e reductions. However, the redu ctio n is for purp oses of pro of only, and we shal l see that the parallel filter, gen erat ing all the relevant estim ates .X,, can be deri ved directly from the form of the cano nica l factor'ljJ.

x

6 THE PARALLEL KALMAN FILTER (CONTINUOUS TIME)

The con tinu ous- time case has show n itse lf to differ from the discrete-tim e case in that the seri al Kal man filter can not be imp lem ente d as it stands. Tur ning then to

6 THE PARALLEL KALMAN FILTER (CONTINUOUS TIME)

367

the parallel filter, we see that there is necessarily another difference. Whereas one can well consider the estimation of x at any lag (either in discrete or continuou s time) one cannot consider the estimate of differentials of x of any order, because differentials of componen ts of x will in general exist only up to order one less than the maximal order occurring in the plant equation. (Recall that the highestorder derivative has in general a white-noise component.) This is of course acceptable, in that these are the only estimates of differentials that one needs for purposes of control. However, if one restricts oneself to estimating just these differentials, then one has effectively reverted to a state-reduc ed treatment, which is rather against the spirit of our programm e. We shall in fact appeal to a state-reduction for purposes of proof, but the final conclusions do not appeal to such a reduction. That is, the continuou s-time analogue of the matrices Hj occurring in the parallel filter analogous to (11), (12) will be deduced directly from the canonical factor 'ljJ of w. Let us for the moment omit the input terms d and i?Ju in the plant equation; these can be left out and later restored without affecting estimation. In the Markov case (with d(s) =sf- A, C6'(s) = -C for constant matrices A and C) we find canonical factors

L+ VCT M 0

V

'f/J(s) = [ 0 I

si-Al -C 0

'l/J.

,

=

0 0 [0 M 0 I

/]

0 .

(34)

0

where Vis the stationary covariance matrix of the estimation error x- x. We shall see that this generalises to

'l/J=

[~I !0 ~]0 [~~~I t~:0 ~],'l/J. 0 =

(35)

where 'ljJ and its elements have arguments s or ~ as appropriat e. Let us initially suppose that the dynamics are of order p exactly, so that the matrix coefficient Ap of ~P in d is non-singular, and can be normalised to the identity. The Kalman filter for the state-reduc ed model, which gives the parallel updating for the unreduced model, then has the form (analogous to (11), (12)), (O~r
(36)

p-!

~Xp-1 + LA,x, = Hp-1(

(37)

r=O

where (is the innovation, expressed more explicitly in (32), (33). This constitutes the parallel filter. It is convenient to define the matrices G, = MH, and the generating function

' f

368

OPT IMA L STATIONARY LQG POLICIES

p-I

G(s) =

L G,s-r-I. r=O

The problem is then to determine the coefficie nts H, in (36), (37) directly from the .· . canonical factor '1/J of (35), without reversion to a state-reduced treatment. This is • · resolved as follows. Theorem 20.6.1 The coefficients llj

= GjM- 1 in the parallel Kalman filter

(36), • the canonicalfactorisation (35) by the

(37) are determined in terms ofthe factor '1/J of

relation

'1/J~m(s) =

[A(s)G(s)]+

(39) where the truncation operator [ ]+ retains only the terms in non-negative powers of s. This implies an equation system p

'1/Jrlm =

L

AkGk-r-l

(r = 0, 1, ... ,p- 1)

k=r+1

(40)

which determines Gp_ 1, Gp_ 2 , •.• recursively. Suppose dynamics are of variable order, in that rj is the order of the highest derivative of the jth component ofx which occu rs in the plant equation. Then the jkth elementof'I/Jrlm is zero for r ~ rjandwe may assu me the same ofG,. Relations (40) then again determine the components ofthe matr ices G, recursively from the highest order downwards. Proof The essential conclusion, expressed in (39), is as simple as one could wish. However, the proof is somewhat more exten ded. Consider first the Markov case and define the vector as the solution of '1/J( $'")::( = p, with the canonical factors given by (34). That is,

x

[~ L+f~

2}-t][ ~l] [7] =

It follows then immediately, as in Section 4, that l = 0, that x obeys the Kalman filter relation !'Jx =A x+ H(y - Cx),

and that m= M- 1(y- Cx) = M- 1(. Further, the relation -ifix(tl(r) = '1/J.xT now amounts to

1]0 [-m -1 ] 0

X

(t) T

=

[

~

x ] -Mm 0

(r~t),

(41)

6 THE PARALLEL KALM AN FILTER (CONTINUOUS TIME)

3691

since z(tl(t) = 0, it indeed follows that we can make the identification == xC1l(t) andalsomCtl(t) = m(t). er its state Now consider the model withpth-order dynamics exactly, and consid uishing tilde on '' reduction, using the notation of the Markov model with a disting vector with rth } variables and coefficients. The state vector x will be a column then implies · subcompopent x, = 9J'x(O ~ r < p). The analogue of relation (41) again that I = 0 and that mand the x, satisfY (42) -G,m + 9Jx, - x,+l = o (O~r
-Gp-lm + 9Jxp-l + L:A,x, = o

(43)

r=O

p-1

-Mm+ L:c,x, = -y

(44)

r=O

of L + VC1 • where the vertically partitioned matrix G = (G,) is the analogue H, = G,M- 1, These relations imply the parallel filter relations (36), (37) with x, indeed has and we know from consideration of the state-structured case that of 9)' x. value t curren the the interpretation (33); it is the projection estimate of rather sation factori ical It remains then to determine G, but from the canon ns equatio reduce than by actual calculation ofthe analogue of L+ veT. Ifwe x with xo of ication (42}-(44) to (30) by elimination of x, (0 < r < p) and identif 1/Jmm· and /J1m I ' then we can identifY the resulting operator coefficients of mwith Solution of (42) yields the expression r-1

x, =

P~'x-

L Gk9Jr-k- m= 9J'x- [9J'G{9J)]+.m 1

(45)

k=O

of PI (or s, as where the operator []+annihilates all terms in negative powers (43) and (44) into sion expres appropriate) in the bracket Substitution of this yields relations (30) with the identifications

'1/Jmm(s)

= M + [~(S)G(s)]+.

determining In particular, we have the key assertion (391 which implies the linear relations (40). canonical A continuation of this argument implies the asserted form (35) of the of the tion adapta factorisation. The final assertion of the theorem follows by D argument given to the case of variable order.

CH AP TE R 21

The Risk-sensitive (LEQG) Version optim isati on unde r LQG assu mpThe time-integral characterisation of control one inco rpora tes risk-sensitivity by tions seems to reach its completion first when is beca use the optimisations of generalising to an LEQ G characterisation. This the extremisation of a single time control and estim ation are then combined in the matr ices of operators 1> and \II integral, representing the stress. Furth ermo re, the LQG mod el appe ar inde ed generalise in so pleasing a fashion that they make when it is imbe dded in the class of as a degenerate case which finds its place first LEQ G models. pected that there is one step whose With so pleasing a completion it is then unex recoupling step of Section 16.7. We natural generalisation proves elusive: the final discuss the poin t in Sections 2 and 3.

Assu

1 DEDUCTION OF THE TIME-INTEGRAL me again the process and observation relations

+ :Jiu = d + E

d x

(1)

y+~x = 1J

(E, 'TJ). In discrete time we assu me with jointly white plan t and observation noise ntan eous cost function (12.4). In the noise cova rianc e matr ix (12.3) and insta ssion with a rate interpretation. The continuous time we assu me these same expre ion of risk-sensitivity are com mon struc tural poin ts which arise with the intro duct for definiteness. to both versions, so we shall keep to discrete time and know then from the risk-ac), E.,..(e We assume the risk-sensitive crite rion ion 16.2 that the optimal value of sensitive certa inty equivalence principle of Sect stress the of the control u1 is deter mine d by extremisation §

= c + e- 1 UJ

all currently unde term ined controls with respect to all curre nt unobservables and explicitly the sense in which this (at time t). Theo rems 16.2.1 and 16.2.2 state more extremisation is to be unde rstoo d. the stress as Und er the assu mpti ons above we can then write § = L(c- r + o-l Dr)+ end term s T

372

THE RISK-SENSITIVE (LEQG) VERSION

where

and e, TJ are to be expressed in terms of x, y and u by (1). As emphasis ed in Chapters 12 and 16, the effect of the certainty-equivalence principle is to reduce constrain ed minimisations to unconstr ained minimisations. We still have the constraints of the plant and observation relations, however, which we reduce, as in Sections 6.5 and 129, by the introduct ion of Lagrange multipliers. We thus consider the Lagrangi an form §

+ l:[AT{d x+ ~u- d-e)+ JLT(y+ ctx- TJ)].,.. .,.

{2)

If we extremise this with respect to the noise variables then we are left with the integral 0 = l:[c(x,u ) + AT(dx+ ~u- d)+ JLT(y+ Cx)- 8v(A,JL)].,. +end terms T

(3) where the cost and 'dual cost' terms c( x, u) and v( A, JL) have the expressions C.,.=

I

2

[X]T [R U T

s

ST] [X] Q

U .,.'

(4)

Expression (3) constitutes our time-integral, in that it can extremised freely with respect to every variable except those whose values are currently known. So, if one is determin ing the optimal value of u, then the control and observation histories U,_ 1 and Y, are known, as is also the whole course {d,} of the deterministic disturbance. All other x, y, u, A and JL values are then to be extremised out. Whatever the nature of the extreme, it will be attained at a stationar y point if the optimisation problem is properly posed Extremising values calculated subject to specification of observables at time will be given the superscri pt (t); they are to be regarded as 'minimal stress' determinations of the quantity in question conditional on informat ion at time t. (A 'determination' can then have aspects of both estimation and optimisation, and 'minimal stress' is really 'extremal stress', with the nature of the extremum dependin g on the sign of 8'J The dual variables A and JL have the physical meaning of 'sensitivities': rates of change of minimal stress with respect to changes in the value of plant and observation noise. Their extremising values are related to effective estimates of these noise inputs by

(5) as we see from the values of e and TJ which extremise expression (2}

373

2 THE STATIONARITY CONDITIONS

I ~

!

Note that the time-integral (3) is a linear combination of the time-integrals ion (19.3) and (20.4) arising in the separate considerations of control and estimat also Note y. naturall e for the LQG case. In the LEQG treatment these combin that the passage from (2) to (3) clears all matrix inverses from the integral. 2 THE STATIONARITY CONDITIONS to Extremising expression (3) with respect to all disposable variables, subject ns equatio of specification of the observables at timet, we deduce the two sets

sT

[!

[~

Q

91 L M tj

~f!l J -(}N

n·> = [Rr+S'Wr s.r + Qtf d + ()Lf.t

u

>.

T

=

-y ()STu

(6)

(r~t)

(7)

T

r [a-"·r

-ef.t $rt ] [ -9>. r x -{}R

(r;;;:: t)

r

f.tr for To these can be added the stationarity conditions with respect to Yr and saying t), > r ( 0 = f.tr relation ant import the r > t. The first of these yields the essentially that the observation relation is longer a constraint for r > t (since The . exactly) relation the satisfy to chosen be will estimate of a future observation future second gives a predictive relation for Yr (i.e. for what the values of t interes oflittle usually observations would be) which is and The two equation systems (6) and (7) plainly generalise equations (19.4) s previou these of tion comple natural the te (20.5). They indeed seem to constitu in entry empty the and s, variable n commo versions in that they are now linked by been the bottom right-hand corner of the two matrices of operators has now about bring entries These -eR. by other filled; in the one case by -(}Nan d in the sitive the effect which we have already remarked in Chapter 16: that in the risk-sen costby affected is ion estimat and case control is affected by noise properties ons conditi the them to add we pressures. The systems are complete for given t if

Ur specified (r

Yr specified (r ~ t);

< t).

(8)

As in Sections 18.1 and 19.2 we write the two equation systems (6) and (7) as

= (~t) 'lt(5")x~l = p~l ~( 5"){~)

(r;;;:: t)

(r

~

t)

with the revised definitions of these quantities implied in (6), (7). the Use of the canonical factorisations again reduces these equations almost to of ation factoris al canonic a have required degree. For example, ~{z) will again

374

THE RISK-SENSITIVE (LEQG) VERSION

the form (19.6) with ¢o symmetric, and the expression (19.12) of the canonical factor