1
INTRODUCTION Eugene A. Feinberg Adam Shwartz
This volume deals with the theory of Markov Decision Processes (MDPs) and their applications. Each chapter was written by a leading expert in the respective area. The papers cover major research areas and methodologies, and discuss open questions and future research directions. The papers can be read independently, with the basic notation and concepts of Section 1.2. Most chapters should be accessible by graduate or advanced undergraduate students in elds of operations research, electrical engineering, and computer science. 1.1
AN OVERVIEW OF MARKOV DECISION PROCESSES
The theory of Markov Decision Processes|also known under several other names including sequential stochastic optimization, discrete-time stochastic control, and stochastic dynamic programming|studies sequential optimization of discrete time stochastic systems. The basic object is a discrete-time stochastic system whose transition mechanism can be controlled over time. Each control policy de nes the stochastic process and values of objective functions associated with this process. The goal is to select a \good" control policy. In real life, decisions that humans and computers make on all levels usually have two types of impacts: (i) they cost or save time, money, or other resources, or they bring revenues, as well as (ii) they have an impact on the future, by in uencing the dynamics. In many situations, decisions with the largest immediate pro t may not be good in view of future events. MDPs model this paradigm and provide results on the structure and existence of good policies and on methods for their calculation. MDPs have attracted the attention of many researchers because they are important both from the practical and the intellectual points of view. MDPs provide tools for the solution of important real-life problems. In particular, 1
2
HANDBOOK OF MARKOV DECISION PROCESSES
many business and engineering applications use MDP models. Analysis of various problems arising in MDPs leads to a large variety of interesting mathematical and computational problems. Accordingly, this volume is split into two major categories: theory (Parts I and II) and applications (Part III). The concept of dynamic programming, which is very important for MDPs, was systematically studied by Bellman in many papers and in the book [6]. This concept is natural and several authors used dynamic programming methods in 1940s{early 1950s or probably earlier to approach various problems. Examples include the work on statistical sequential analysis by Wald [50] and by Arrow, Blackwell, and Girshick[3], the work by Arrow, Harris, and Marschack [4] and by Dvoretsky, Kiefer, and Wolfowitz [21] on inventory control, and the work by Bellman and Blackwell [7] and Bellman and LaSalle [8] on games. Shapley's [47] seminal work on stochastic games introduced important definitions and results. The relationship between MDPs and stochastic games is similar to the relationship between a usual game and an optimization problem: a stochastic game with one player is an MDP. Therefore, many experts consider this Shapley's paper as the rst study of MDPs. In addition to the mentioned individuals, Puterman [43, p. 16] refers to Isaacs, Karlin, Masse, and Robbins as major contributors to early breaking work in 1940s and early 50s. The book by Dubins and Savage [20] on gambling theory played an important role. Howard [33] introduced policy iteration algorithms and that book started the systematic study of MDPs. Several seminal contributions were done by Blackwell, Denardo, Derman, Ross, and Veinott in the 1960s (references to their work, and to work of other individuals mentioned by names in this paragraph, can be found in Puterman [43]). Also in the 1960s, three distinguished probabilitists, Dynkin, Krylov, and Shiryaev [48], worked on MDPs in Russia. Hinderer's book [32] was an important contribution. Over the following thirty years, there were many fundamental and exciting developments in MDPs and their applications. Most are either described in this volume or associated with the names of its contributors. Since their introduction in the 1950s, MDPs have become an important research area with a rich and deep theory and various applications. In fact, MDPs became basic tools for the analysis of many problems in operations research, electrical engineering and computer science. Algorithms for inventory control and telecommunications protocols are two examples of such engineering applications. During the rst thirty years of the MDP theory, roughly speaking until early 1980s, most of the research was centered around optimality equations and methods for their solution, namely policy and value iteration. Value iteration algorithms and their various versions are also known under the names of successive approximation, backward induction, and dynamic programming. The dynamic programming principle in its classical form can be applied only to problems with an appropriate single objective function. For example, the dynamic programming algorithm is applicable to optimization of an expected total reward over a nite time horizon. It can usually be applied to singlecriterion in nite horizon problems with a total expected reward or average reward per unit time. For some other objective functions, or when the goal
INTRODUCTION
3
is to optimize one objective function under constraints on other criteria, the problem usually cannot be solved directly by dynamic programming; for an indirect approach see Piunovskiy and Mao [41]. Convex analytic methods, including linear and convex programming in nite and in nite dimensional spaces are usually more natural in these situations. There are many exciting recent developments, especially in applications, in which the dynamic programming principle plays an important role (see e.g. [12] and chapter 14, by Van Roy, in this volume). However, most of the research over the last two decades has been focused on diÆcult problems to which the dynamic programming principle cannot be applied in its direct form. In particular, a signi cant part of current research deals with multiple criteria. 1.2
DEFINITIONS AND NOTATIONS
Let N = f0; 1; : : : g and let IRn be an n-dimensional Euclidean space, IR = IR1 . A Markov Decision Process (MDP) is de ned through the following objects: a state space X; an action space A ; sets A (x) of available actions at states x 2 X; transition probabilities, denoted by p(Y jx; a); reward functions r(x; a) denoting the one-step reward using action a in state x. The above objects have the following meaning. There is a stochastic system with a state space X. When the system is at state x 2 X, a decision-maker selects an action a from the set of actions A (x) available at state x. After an action a is selected, the system moves to the next state according to the probability distribution p(jx; a) and the decision-maker collects a one-step reward r(x; a). The selection of an action a may depend on the current state of the system, the current time, and the available information about the history of the system. At each step, the decision maker may select a particular action or, in a more general way, a probability distribution on the set of available actions A (x). Decisions of the rst type are called nonrandomized and decisions of the second type are called randomized. An MDP is called nite if the state and action sets are nite. We say that a set is discrete if it is nite or countable. An MDP is called discrete if the state and action sets are discrete. A signi cant part of research and applications related to MDPs deals with discrete MDPs. For discrete MDPs, we do not need additional measurability assumptions on the major objects introduced above. Readers who are not familiar with measure theory can still read the papers of this volume, since most of the papers deal with discrete MDPs: for the other papers, the results may be restricted to discrete state and action sets. For a discrete state space X we denote the transition probabilities by p(yjx; a) or pxy (a), and use (in addition to x; y) also the letters i; j; k etc. to denote states. Unless mentioned otherwise, we always assume that p(Xjx; a) = 1. The time parameter is t; s or n 2 N and a trajectory is a sequence x0 a0 x1 a1 : : : The set of all trajectories is H1 = (X A )1 . A trajectory of length n Discrete MDPs.
4
HANDBOOK OF MARKOV DECISION PROCESSES
is called a history, and denoted by hn = x0 a0 : : : xn 1 an 1 xn . Let Hn = X (A X)n be the space of histories up to epoch n 2 N . A nonrandomized policy is a sequence of mappings n , n 2 N , from Hn to A such that n (x0 a0 : : : xn 1 an 1 xn ) 2 A (xn ). If for each n this mapping depends only on xn , then the policy is called Markov. In other words, a Markov policy is de ned by mappings n : X ! A such that n (x) 2 A (x) for all x 2 X, n = 0; 1; : : : . A Markov policy is called stationary if the n do not depend on n. A stationary policy is therefore de ned by a single mapping : X ! A such that (x) 2 A (x) for all x 2 X. We denote by , M ; and S the sets of all nonrandomized, Markov, and stationary policies respectively. We observe that S M . As mentioned above, by selecting actions randomly, it is possible to expand the set of policies. A randomized policy is a sequence of transition probabilities n (an jhn ) from Hn to A , n 2 N , such that n (A (xn )jx0 a0 : : : xn 1 an 1 xn )= 1. A policy is called randomized Markov if n (an jx0 a0 : : : xn 1 an 1 xn ) = n (an jxn ). If m (jx) = n (jx) for all m; n 2 N then the randomized Markov policy is called randomized stationary. A randomized stationary policy is thus de ned by a transition probability from X to A such that (A (x)jx) = 1 for all x 2 X. We denote by R , RM , RS the sets of all randomized, randomized Markov, and randomized stationary policies respectively. We have that RS RM R , and in addition S RS , M RM , and R . Note that, while we try to be consistent with the above de nitions, there is no standard terminology for policies: in particular, there is no general agreement as to whether \stationary" implies nonrandomized or, more generally, whether the \default" should be randomized (the more general case) or nonrandomized. The following additional terms are sometimes also used: pure policy means nonrandomized; deterministic policy means (nonrandomized) stationary. The stochastic process evolves as follows. If at time n the process is in state x, having followed the history hn , then an action is chosen (perhaps randomly) according to the policy . If action a ensued, then at time n + 1 the process will be in the state y with probability p(yjx; a). Given an initial state x and a policy , the \evolution rule" described above de nes all nite-dimensional distributions x0 ; a0 ; : : : ; xn , n 2 N . Kolmogorov's extension theorem guarantees that any initial state x and any policy de ne a stochastic sequence x0 a0 x1 a1 : : : . We denote by IPx and IEx respectively the probabilities and expectations related to this stochastic sequence; IPx fx0 = xg = 1. Any stationary policy de nes for any initial distribution a homogeneous Markov chain with transition probabilities pxy ( ) = p(yjx; (x)) on the state space X. A randomized stationary policy also de nes for each initial distribution a homogeneous Markov chain withPthe state space X: In the latter case, the transition probabilities are pxy () = a2A (x) (a)p(yjx; a). We denote by P () the transition matrix with elements fpxy ()g. The limiting matrix 1
N X1
Q() = lim P n ( ) N !1 N n=0
(1.1)
INTRODUCTION
5
always exists and, when X is nite, this matrix is stochastic; Chung [18, Section 1.6]. Let f be a terminal reward function and be a discount factor. We denote by vN (x; ; ; f ) the expected total reward over the rst n steps, n 2 N: vN (x; ; ; f ) = IE
x
"N 1 X
nr
n=0
(xn ; an ) +
#
N f
(xN ) ;
(1.2)
whenever this expectation is well-de ned. If 2 [0; 1[ then we deal with expected total discounted reward. If = 1, we deal with the expected total undiscounted reward or simply the total reward. For in nite-horizon problems with N = 1, we do not write N explicitly and the expected total rewards do not depend on the terminal reward f . Thus, we de ne by v(x; ; ) the expected total rewards over the in nite horizon. If the discount factor 2 [0; 1] is xed, we usually write v(x; ) instead of v(x; ; ). The expected total reward over an in nite horizon is v(x; ) = v(x; ; ) = v1 (x; ; ; 0) : (1.3) If the reward function r is bounded either from above or from below, the expected total rewards over the in nite horizon are well-de ned when 2 [0; 1[: Additional conditions are required for the expected total reward v(x; ; 1) to be well-de ned. Since this sum may diverge when the discount factor is 1, it is natural to consider the expected reward per unit time 1 (1.4) w(x; ) = lim inf v (x; ; 1; 0): n!1
N
N
If a performance measure g(x; ) is de ned for all policies , we denote G(x) = sup g(x; ): (1.5) R 2
In terms of the performance measures de ned above, this yields the values VN (x; ; f ) , sup vN (x; ; ; f ) ; (1.6) R 2
V (x) = V (x; ) , sup v(x; ; ) ; 2R W (x) , sup w(x; ) : 2R
(1.7) (1.8)
For 0, a policy is called -optimal for criterion g if g(x; ) G(x) for all x 2 X. A 0-optimal policy is called optimal. We introduce the important notions of optimality operators and optimality equations. The conditions when optimality operators are well-de ned and optimality equations hold are considered in appropriate chapters. For a function g on X, we consider the reward operators: P a g(x) , IE[g(x1 ) j x0 = x; a0 = a] ; (1.9) a a T g(x) , r(x; a) + P g(x) (1.10)
6
HANDBOOK OF MARKOV DECISION PROCESSES
and the optimality operators: P g(x) , sup P a g(x) ; a2A (x)
T g(x) , sup T a g(x) : a2A (x)
(1.11) (1.12)
The nite horizon Optimality Equation is VN +1 (x) = T VN (x); x 2 X; N = 0; 1; : : : ; (1.13) with V0 (x) = f (x) for all x 2 X: The discounted reward Optimality Equation is V (x) = T V (x) x 2 X: (1.14) An action a 2 A(x) is called conserving at state x for the (N + 1)-step problem if T aVN (x) = T VN (x): An action a 2 A(x) is called conserving at state x for the total discounted reward if T aV (x) = T V (x): When = 1 we denote T a = T1a and T = T1. In particular, V (x) = T V (x); x 2 X; (1.15) is the Optimality Equation for expected total undiscounted rewards. For total reward criteria, value functions usually satisfy the optimality equation. In addition, the sets of conserving n-step actions, n = 1; : : : ; N + 1 form the sets of optimal actions for (N + 1)-step problems. Under some additional conditions, the sets of conserving actions form the sets of optimal actions for in nite horizon problems. We shall consider these results in appropriate chapters. The average reward Optimality Equations are W (x) = P W (x); x 2 X; (1.16) W (x) + h(x) = sup T ah(x); x 2 X; (1.17) a2A 0 (x)
where A0
(x) = fa 2 A (x) : P a W (x) = P W (x)g ; x 2 X: (1.18) Equation (1.16) is called the First Optimality Equation and equation (1.17) is called the Second Optimality Equation. We remark that W has a meaning of an optimal average reward per unit time and h has a meaning of a terminal reward. Note that if W (x) = W , a constant, then the First Optimality Equation holds and A 0 (x) = A (x). In this case, the Second Optimality Equations transforms into W + h(x) = T h(x); x 2 X; (1.19) which is often referred to simply as the Optimality Equation for average rewards.
INTRODUCTION
7
We allow for the starting point x to be de ned by an initial probability distribution . In this case, we keep the above notation and de nitions but we replace the initial state x with the initial distribution . For example, we use IP , IE , v(; ), V (); w(; ); and W (). We remark that, generally speaking, optimality and -optimality with respect to all initial distributions are stronger notions than the optimality and -optimality with respect to all initial states. However, in many natural cases these de nitions are equivalent. For example, this is true for total reward criteria. A more general problem arises when there are multiple objectives. Suppose there are (K + 1) reward functions rk (x; a); k = 0; : : : ; K . For nite horizon problems, terminal rewards may also depend on k. In this case, we index by k = 0; : : : ; K all functions that describe rewards. For example, we use the notation wk (x; ), fk (x), and Wk (x). For problems with multiple criteria, it is usually natural to x an initial state x. It is also possible to x an initial distribution , with our convention that all de nitions remain the same, but we write instead of x. So, for simplicity, we de ne optimal policies when the initial state x (not a distribution) is xed. If the performance of a policy is evaluated by (K +1) criteria gk (x; ) then one goal may be to optimize criterion g0 subject to constraints on g1; : : : ; gK : Let Ck ; k = 1; : : : ; K; be given numbers. We say that a policy is feasible if gk (x; ) Ck ;
k = 1; : : : ; K :
(1.20)
A policy is called optimal for a constrained optimization problem if it is feasible and g0(x; ) g0 (x; ) for any feasible policy :
(1.21)
general constructions. When a state space X or an action space A are is not discrete, the natural assumption is that they are measurable spaces endowed with - elds X and A respectively. When X or A are discrete, the corresponding - eld is the set of all subsets of the corresponding set. It is also natural to assume that the sets A (x) 2 A of feasible actions are measurable, for all states x 2 X. Of course, this assumption always holds when A is discrete. Unless we specify otherwise, we always consider the Borel - eld B(IR) on IR: this is the minimal eld containing all intervals. For non-discrete MDPs, we also assume that r is a measurable function on (X A ; X A) and p(Y jx; a) is a transition probability from (X A ; X A) to (X; X ). Recall that given two measurable spaces (E1 ; E1 ) and (E2 ; E2 ), we call p a transition probability from E1 to E2 if the following two conditions hold: (i) p(je2 ) is a probability measure on (E1 ; E1 ) for any e2 2 E2 , and (ii) the function p(B j) is measurable on E2 for any B 2 E1 . In order to de ne policies in the general situation, we consider - elds Hn = X (AX )n on the sets of histories Hn = X (A X)n . Nonrandomized and randomized strategies are de ned in a way similar to discrete MDPs, with standard and natural additional measurability conditions: (a) nonrandomized policies are de ned by mappings n which are measurable on (Hn ; Hn ), and
Nondiscrete MDPs:
8
HANDBOOK OF MARKOV DECISION PROCESSES
(b) stationary and Markov policies are de ned by mappings which are measurable on X: Similarly, for randomized policies, n are transition probabilities from (Hn ; Hn ) to (A ; A) and, for randomized Markov and stationary policies, they are transition probabilities from (X; X ) to (A ; A). Let H1 = (X A)1 : Ionescu Tulcea theorem, Neveu [39, Section 5.1], implies that any initial probability measure on X and any policy de ne a unique probability measure on (H1 ; H1 ). In particular, de nes the initial distribution, n de ne transition probabilities from Hn to Hn A , and p de ne transition probabilities from Hn A to Hn+1 , n = 0; 1; : : : : We denote this measure by IP . Sometimes this measure is called a \strategic" measure. We denote by IE expectations with respect to this measure. If (x) = 1 for some x 2 X, we write IPx and IEx instead of respectively IP and IE . We also notice that Ionescu Tulcea theorem implies that IPx is a transition probability from (X; X ) to (H1 ; H1 ) and this implies that the functions vn (x; ; ; f ) and v(x; ; ) are measurable in x for any policy (the terminal function f is also assumed to be measurable). We remark that we use Ionescu Tulcea theorem instead of the better known Kolmogorov's extension theorem primarily because the latter one requires that the process has values in a locally compact metric spaces. For MDPs this means that the state and action spaces are required to be locally compact metric spaces. Since Ionesco Tulcea theorem holds for arbitrary measurable spaces, it is more convenient to apply it to the construction of strategic measures in MDPs, rather than Kolmogorov's extension theorem. At the intuitive level, a randomized decision at any state is a probability measure on the set of nonrandomized decisions. In addition, in order to avoid a trivial situation, an MDP has to have at least one policy. In order to guarantee these two intuitive properties, we always assume the following two mild conditions: (i) all one-point sets fag are elements of A; a 2 A ; (ii) there is at least one measurable function from X to A such that (x) 2 A (x) for all x 2 X. The rst assumption always holds for models with discrete action spaces. The second assumption always holds for models with discrete state spaces. For a measure and a measurable function f we use the equivalent notations (f ) ,
Z
f ( ) d ( ) , f ( ) :
(1.22)
If we denote x () = (jx) for a randomized stationary policy then, similarly to discrete MDPs, this policy de nes a Markov chain with transition probabilities p(dyjx; x ): If X is discrete, this chain has transition matrix P () with elements pxy (x ). Thus, an MDP, strategies, and objective functions can be de ned under very general conditions. However, very little can be done if one tries to analyze MDPs with arbitrary measurable state spaces. The rst complication is that the value functions V may not be measurable even for one-step models. The second complication is that an important step in the analysis of MDPs is to construct an equivalent randomized Markov policy for an arbitrary policy; see Derman-Strauch's theorem which is the rst theorem in chapter 6. This can be done by constructing transition probabilities IPx (dan jxn ) which may not exist
INTRODUCTION
9
for general state and action spaces. These two complications do not exist if the state space is countable. These two complications can be resolved if X and A are Borel spaces. In addition, at the current state of knowledge, there is no clear need to consider MDPs with arbitrary measurable state spaces because there is no clear motivation or practical need for such objects. For example, MDPs with Borel state spaces have applications in statistics, control of models with incomplete information, and inventory management. However we are not aware of possible applications of MDPs with state spaces having higher cardinality than continuum. In this case, the state space X is discrete and the action space is a measurable space (A ; A) such that all one-point sets are measurable. From the de nitions for general MDPs we have that the sets of feasible actions A (x) are also elements of A; reward functions r(x; a) and transition probabilities p(yjx; a) are measurable in a. All constructions described for discrete and general MDPs go through with X being the - eld of all subsets of
Discrete state MDPs.
X:
Though we do not follow any particular text, all de nitions, constructions, and statements, related to Borel spaces we mention in this chapter can be found in Bertsekas and Shreve [11, Chapter 7]; see also Dynkin and Yushkevich [22] and Kechris [35]. Two measurable spaces (E1 ; E1 ) and (E2 ; E2 ) are called isomorphic if there is a one-to-one measurable mapping f of (E1 ; E1 ) onto (E2 ; E2 ) such that f 1 is measurable. A Polish space is a complete separable metric space. Unless we specify otherwise, we always consider a Borel - eld B(E ) on a metric space E ; B(E ) is the minimal - eld containing all open subsets of E . A measurable space (E; E ) is called Borel if it is isomorphic to a Polish space. All Borel spaces are either nite or countable or continuum, and two Borel spaces with the same cardinality are isomorphic. Therefore, uncountable Borel spaces are continuum. They are also isomorphic to each other and to the sets (IR; B(IR)) and ([0; 1]; B([0; 1])). Any measurable subset E 0 of a Polish space forms a Borel space endowed with the Borel - eld which is the intersection of E 0 with Borel subsets of the original space. The assumptions for Borel MDPs are: (i) X and A are Borel spaces and X and A are the corresponding Borel - elds; (ii) the graph Gr(A ) = f(x; a)j x 2 X; a 2 A (x)g is a measurable subset of (X A ; X A) and there exists at least one measurable mapping of X into A such that (x) 2 A (x) for all x 2 A (x); (iii) the reward functions r(x; a) are measurable on X A and the transition probabilities p(jx; a) are transition probabilities from X A to X: Conditions (i) and (iii) are similar to the corresponding assumptions for general models. The measurability of the graph in (ii) implies that the sets Classical Borel MDPs.
10
HANDBOOK OF MARKOV DECISION PROCESSES
(x) are measurable. The existence of a measurable mapping (often called a \selector") implies that A (x) 6= ; for all x. We remark that it is possible that the graph is Borel and all images are non-empty but the graph does not contain a Borel mapping. Therefore, the second assumption in (ii) is essential for the existence of at least one policy. As was discussed above, the rst real complication is that even for one-step problems, the values V may not be Borel measurable functions on X. However, conditions (i)-(iii) imply that these functions are universally measurable for nite and in nite-horizon problems and therefore optimality operators can be de ned. Here we explain the concepts of universally measurable sets and functions. Let (E; E ) be a Borel space. For a given probability measure p on (E; E ), de ne the - eld Ep as the completion of E with respect to the measure p. That is, Ep is the minimal - eld that contains E and all subsets F of E such that F F 0 for some F 0 2 E , and p(F 0 ) = 0. For example, if (E; E ) = ([0; 1]; B([0; 1])) then we can consider the Lebesgue measure m de ned by m([a; b]) = jb aj: Then Em is the so-called Lebesgue - eld. Let P(E ) be the set of all probability measures on E . Then the intersection of all - elds Ep , U (E ) = \fp2P(E)g Ep , is a - eld and it is called the universal - eld. This - eld is also called the - eld of universally measurable sets and its elements are called universally measurable subsets of E . A universally measurable function on E is a measurable mapping from (E; U (E )) to (IR; B(IR)) Of course, any Borel set and any Borel function are universally measurable. Thus, optimality equations can be de ned for Borel MDPs. However, there is another complication for Borel models, which is annoying mostly for aesthetic reasons: -optimal policies may not exist for positive , even for one-step Borel MDPs with bounded reward functions. The example constructed by David Blackwell is based on the observation that the value function is universally measurable but it may not be Borel. However, for any policy, the expected one-step reward is a Borel function of the initial step. Moreover, it is possible to show that for the Borel MDP described above, for any initial measure p on X, and for any > 0 there exists a policy which is p-a.s. -optimal. Such policies are called (p; )-optimal. A
If we expand the set of policies and consider universally measurable policies, -optimal policies exist and the concept of (p; ) optimality is not needed. However, if we expand the set of policies, the results and their proofs hold for assumptions which are broader than (ii) and (iii). Before we give formal de nitions, we explain the concept of analytic sets. Let f be a measurable mapping of a Borel space (E1 ; E1) into a Borel space (E; E ). If F 2 E then by de nition f 1(F ) 2 E1 . However, it is possible that f (E ) 2= E for some Borel set F 2 E1 . A subset F of a Borel space (E; E ) is called analytic if there exists a Borel space (E1 ; E1 ) and a measurable mapping of E1 to E such that F = f (F1 ) for some F1 2 E1 . Since one can select E1 = E and f (e) = e, every Borel set is analytic. It is also possible to show that any analytic set is universally measurable. It is
Universally measurable Borel MDPs.
INTRODUCTION
11
also possible to consider the - eld of analytically measurable sets which is the smallest - eld containing all analytic subsets of an analytic set. We remark that Borel and universally measurable - elds consist respectively of Borel and universally measurable sets. The situation is di erent for analytic sets and - elds of analytically measurable sets. The complement of an analytic set may not be analytic. Therefore, the - eld of analytically measurable sets contains sets other than analytic. We remark that there are many equivalent de nitions of analytic sets. For example, for Polish spaces they can be de ned as continuous images or even as projections of Borel sets. If (E; E ) and (E1 ; E1 ) are two Borel spaces (Borel sets with Borel - elds) then the mapping f : E ! E1 is called universally (analytically) measurable if f 1(B ) belongs to the - eld of universally (analytically) measurable subsets of E for all B in E1 . The assumptions for universally measurable MDPs are: (a) The state and action spaces (X; X ) and (A ; A) are Borel spaces; (b) Gr(A) is an analytic subset of X A and all sets A (x) are not empty; (c) The reward function r(x; a) is an upper analytic function on X A , that is, for any real number c, the set fr cg is an analytic subset of X A ; (d) The transition function p(jx; a) is a transition probability from (X A ; X A) to (X; X ). Assumptions (a) and (d) coincide with similar assumptions for Borel MDPs. According to the Jankov-von Neumann theorem, assumption (b) implies that there is an analytically measurable mapping from X to A such that (x) 2 A (x) for all x 2 X: Of course, any analytically measurable mapping is universally measurable. Assumption (c) is more general than the assumption that r(x; a) is Borel. This assumption on a reward function is considered in the literature mainly because the optimality operator preserves this property. The last important di erence between Borel and universally measurable MDPs is that policies are universally measurable for the latter ones. Nonrandomized policies are universally measurable mappings n of Hn to A such that (hn ) 2 A (xn ) for any hn = x0 an : : : xn 2 Hn : Markov (and stationary) policies are de ned by universally measurable mappings n of X to A such that n (x) 2 A (x) ( (x) 2 A (x)) for all x 2 X: Randomized, randomized Markov, and randomized stationary policies are transition probabilities de ned in the same way as for general models but the sets Hn and X are endowed with elds of universally measurable subsets that play the role of - eld E1 in the de nition of transition probabilities given above. Condition (b) implies that there exists at least one policy. There are other versions of universally measurable MDPs. For example, one can consider analytically measurable policies; see Bertsekas and Shreve [11] for details. The important feature is that all de nitions and notations, given for discrete MDPs, hold also for universally measurable MDPs.
12 1.3
HANDBOOK OF MARKOV DECISION PROCESSES THE SCOPE OF THIS VOLUME
The rst two parts of this book deal with theoretical questions and Part III addresses some applications of MDPs. Part I deals with models with nite state and action spaces, and Part II deals with in nite state problems. The paper by Lodewijk Kallenberg surveys the classical theory for basic criteria including total discounted expected rewards, average expected rewards per unit time, and more sensitive criteria including bias, Blackwell, and ndiscount optimality criteria. The paper by Mark Lewis and Martin Puterman focuses on bias optimality. In real life, parameters of the models may be measured with nite accuracy. An important question is what happens in the case of, say a linear perturbation when the transition probabilities p(yjx; a) are replaced with transition probabilities p(yjx; a)+ d(yjx; a), where > 0 is a small parameter. The survey by Konstantin Avrachenkov, Jerzy Filar, and Moshe Haviv describes research and applications for a nontrivial case of singular perturbation when the above transformation changes the ergodic structure of the system. One important application is an approach to the classical Hamiltonian Cycle and Traveling Salesman Problems via MDPs introduced by Filar and Krass [28]. Part II covers the following major objective criteria for in nite state models: expected total rewards (Eugene Feinberg), average rewards/costs per unit time (Linn Sennott, Armand Makowski and Adam Shwartz, Sean Meyn, and significant parts of chapters written by Vivek Borkar and by Onesimo HernandezLerma and Jean Lasserre), Blackwell optimality (Arie Hordijk and Alexander Yushkevich), and linear combinations of various criteria (Eugene Feinberg and Adam Shwartz). The chapter written by Vivek Borkar concentrates on convex analytic methods and the chapter written by Onesimo Hernandez-Lerma and Jean Lasserre describes the in nite dimensional linear programming approach which is one of the major developments of these methods. Gambling theory, introduced by Dubins and Savage [20], is a close relative of MDPs. The chapter written by Lester Dubins, Ashok Maitra, and William Sudderth, the major contributors to gambling theory over the last three decades (see Maitra and Sudderth [38] for references and many beautiful results on gambling and games), establishes some links between gambling and MDPs. Though gambling theory and MDPs look like close elds, as far as we know, this chapter is only the third major paper that links MDPs and gambling. The other two publications are by Blackwell [15] and Schal [44]. A signi cant part of this volume deals with average reward MDPs. This criterion is very important for applications. In addition, many interesting mathematical questions arise for average reward problems. A major research direction over the last fteen years was to nd ergodicity and other special conditions that hold for broad classes of applications and that ensure the existence of stationary optimal policies. The papers of this volume and many recent publications, including Sennott's and Hernandez-Lerma and Lasserre's books [45, 30], demonstrate signi cant progress in this direction. As we mentioned, these results usually require some ergodicity or other structural assumptions which could be diÆcult to verify. An interesting development is a minimal pair approach, in which the controller selects an initial state in addition to a policy.
INTRODUCTION
13
This approach is described in chapter 12 by Hernandez-Lerma and Lasserre. Theorem 3 in that chapter is a beautiful result that states the existence of optimal policies for the minimal pair approach without any explicit ergodicity or other structural conditions; see also [30] and references therein. If there are no additional structural assumptions, stationary optimal policies may not exist for the standard average reward criterion except in the case of nite state and action sets; see chapter 2, or original contributions [14, 19]. Attempts to expand this result to broader state and action spaces, undertaken between 1960 and 1980, identi ed signi cant diÆculties. For nite state MDPs with compact action sets and continuous transition probabilities and reward functions, stationary optimal policies may not exist [5, 22, 7]. Stationary optimal strategies exist for such models when continuity of reward functions is relaxed to upper-semicontinuity; see [17, 24]. For arbitrary average rewards nite state MDPs, there exist Markov -optimal policies [25, 13], -optimal policies in several other classes of nonstationary policies [27], but stationary -optimal policies may not exist [22]. If the state space is in nite, it is possible that there is no randomized Markov -optimal policy which is -optimal for two given initial states [25]. It is also possible that the supremum of average rewards over all randomized Markov policies is greater than the similar supremum over all (nonrandomized) Markov policies [22]; see also [46, p.91] for a corresponding example for stationary policies. If the initial state is xed, in view of the Derman-Strauch theorem (the rst theorem in Feinberg, chapter 6), for any > 0 there exists an -optimal randomized Markov policy. If lim inf is replaced with lim sup in (1.4), for any give initial state and for any > 0 there exists an -optimal Markov policy [26]. Part III deals with some applications of MDPs. Benjamin van Roy, chapter 14, describes recent trends and directions in neuro-dynamic programming, one of the major relatively recent developments in arti cial intelligence, also known as reinforcement learning, which combines MDPs with approximation and simulations techniques. Manfred Schal, chapter 15, considers MDP applications to nance. Eitan Altman, chapter 16, surveys MDP applications to telecommunications. Bernard Lamond and Abdeslem Boukhtouta, chapter 17, describe water reservoir applications. This book covers most of the major directions in MDPs. It is probably impossible to cover in detail all aspects of MDPs in one book. This book focuses on discrete-time models with complete information. Models with incomplete information [29], continuous-time models, in particular Semi-Markov Decision Processes and Continuous Time MDPs (see [36, 10, 37, 45] and references therein), and risk-sensitive criteria [51] are three important topics which are out of the major scope of this book. Though this book does not have a special chapter on problems with multiple criteria and constraints, it describes the convex analytic approach which is the methodological foundation for studying such problems. Several chapters mention particular results on constrained MDPs and additional details can be found in books by Altman [1], Borkar [6], Kallenberg [37], and Piunovskiy [44]. There are numerous areas of applications of MDPs in addition to areas covered in this book. Some of them are summarized in Puterman's and Bertsekas' books [37, 9, 10]. Here we just mention the
14
HANDBOOK OF MARKOV DECISION PROCESSES
fundamental importance of MDPs for economic dynamics methods [49], transportation science [37], control of queues [36, 45], and production, inventory and supply chain management [9, 31, 42, 2]. All papers of this volume have been refereed. We would like to thank our colleagues for providing the editors and the authors with their comments and suggestions. In addition to the authors of this volume, most of whom served as referees, we would like to thank Igor Evstigneev, Emmanuel FernandezGaucher and, Michael Katehakis, Victor Pestien, Ulrich Rieder, and Chelsea C. White, III for their valuable help. Dimitri P. Bertsekas and Matthew J. Sobel provided us with valuable comments on some literature sources. We are especially grateful to Martin L. Puterman who inspired us on this project. Last, but de nitely not least, we would like to thank Ms. Lesley Price, who served as the de-facto technical editor of this volume. From re-typing a paper to correcting author's errors, Lesley withstood endless interchanges with authors eÆciently and patiently. Her contribution to both form and substance of this volume is much appreciated. Research of the rst editor during his work on this volume was partially supported by NSF grant DMI-9908258. Research of the second editor was supported in part by the fund for promotion of research at the Technion, and in part by the fund for promotion of sponsored research at the Technion. References
[1] E. Altman, Chapman & Hall/CRC, Boca Raton, 1999. [2] R. Anupindi and Y. Bassok, \Supply contracts with quantity commitments and stochastic demand," in ( S. Tayur, R. Ganeshan, M. Magazine, eds.), pp. 197{232, Kluwer, Boston, 1999. [3] K.J. Arrow, D. Blackwell, and M.A. Girshick, \Bayes and minimax solutions of sequential decision processes," 17, pp. 213{244, 1949. [4] K.J. Arrow, T. Harris, and J. Marschak, \Optimal inventory policies," 19, pp. 250{272, 1951. [5] J. Bather, \Optimal decision procedures for nite Markov chains I," 5, pp. 328-339, 1973. [6] R.E. Bellman, Princeton University Press, Princeton, 1957. [7] R.E. Bellman and D. Blackwell, \On a particular non-zero sum game," RM-250, RAND Corp., Santa Monica, 1949. [8] R.E. Bellman and J.P. LaSalle, \On non-zero sum games and stochastic processes," RM-212, RAND Corp., Santa Monica, 1949. [9] D.P. Bertsekas, Athena Scienti c, Bellmont, MA, 2000 (second edition). [10] D.P. Bertsekas, Athena Scienti c, Bellmont, MA, 1995. Constrained
Markov
Decision
Processes,
Quantitative Models for Supply Chain
Management
Econometrica
Econometrica
Adv.
Appl. Prob.
Dynamic
Programming,
Dynamic Programming and Optimal Control: Volume I,
Dynamic Programming and Optimal Control: Volume II,
15
INTRODUCTION
[11] D.P. Bertsekas and S.E. Shreve, Academic Press, New York, 1978 (republished by Athena Scienti c, 1997). [12] D.P. Bertsekas and J.N. Tsitsiklis, Athena Scienti c, Bellmont, MA, 1996. [13] K.-J. Bierth, \An expected average reward criterion", 26, pp. 133{140, 1987. [14] D. Blackwell, \Discrete dynamic programming," 33, pp. 719{726, 1962. [15] D. Blackwell, \The stochastic processes of Borel gambling and dynamic programming," Annals of Statistics 4, pp. 370{374, 1976. [16] V.S. Borkar, Pitman research Notes in Math., 240, Longman Scienti c and Technical, Harlow, 1991. [17] R.Ya. Chitashvili, \A controlled nite Markov chain with an arbitrary set of decisions," 20, pp. 839{846, 1975. [18] K.L. Chung, Springer-Verlag, Berlin, 1960. [19] C. Derman, \On sequential decisions and Markov chains," 9, pp. 16{24, 1962. [20] L.E. Dubins and L.J. Savage, McGraw-Hill, New York, 1965. [21] A. Dvoretsky, J. Kiefer, and J. Wolfowitz, \The inventory problem: I. Case of known distribution of demand," 20, pp. 187{222, 1952. [22] E.B. Dynkin and A.A. Yushkevich, Springer-Verlag, New York, 1979 (translation from 1975 Russian edition). [23] A. Federgruen, \Centralized planning models for multi-echelon inventory systems under inventory," in (S.C. Graves, A.H.G. Rinnooy Kan, and P.H. Zipkin, eds), Handbooks in Operations Research and Management Science, 4, pp. 133{173, North-Holland, Amsterdam, 1993. [24] E.A. Feinberg, \The existence of a stationary -optimal policy for a nite Markov chain," 23, pp. 297{313, 1978. [25] E.A. Feinberg, \An -optimal control of a nite Markov chain with an average reward criterion," 25, pp. 70{81, 1980. [26] E.A. Feinberg, \Controlled Markov processes with arbitrary numerical criteria," 27 pp. 486{503, 1982. [27] E.A. Feinberg and H. Park, \Finite state Markov decision models with average reward criteria," Stoch. Processes Appl., 31 pp. 159{177, 1994. [28] J.A. Filar and D. Krass, \Hamiltonian cycles and Markov chains," Math. Oper. Res. 19, pp. 223-237, 1994. [29] O. Hernandez-Lerma, Springer, New York, 1989. Stochastic
Optimal
Control:
The
Discrete-Time Case,
Neuro-Dynamic Programming,
Stochastic Pro-
cesses and Applications
Ann. Math. Stat.
Topics in Controlled Markov Chains,
SIAM Theory Prob. Appl.
Markov Chains with Stationary Transition Probabilities,
Man. Sci.
How to Gamble if You Must: Inequalities
for Stochastic Processes,
Econometrica
Controlled
Markov
Processes,
Logistics of Production and Inventory,
SIAM Theory Prob. Appl.
SIAM Theory Prob. Appl.
SIAM Theory Prob. Appl.
Adaptive Markov Control Processes,
16
HANDBOOK OF MARKOV DECISION PROCESSES
[30] O. Hernandez-Lerma and J.B. Lasserre, Springer, New York, 1999. [31] D.P. Heyman and M.J. Sobel, McGraw-Hill, New York, 1984. [32] K. Hinderer, , Springer-Verlag, New York, 1970. [33] R.A. Howard , MIT Press, Cambridge, 1960. [34] L.C.M. Kallenberg, , Mathematical Centre Tract 148, Mathematical Centre, Amsterdam, 1983. [35] A.S. Kechris, Springer-Verlag, New York, 1995. [36] M.Yu. Kitaev and V.V. Rykov, CRC Press, Boca Raton, 1995. [37] A.J. Kleywegt and J.D. Papastavrou, \Acceptance and dispatching policies for a distribution problem", 32, pp. 127-141, 1998. [38] A.P. Maitra and W.D. Sudderth, Springer, New York, 1996. [39] J. Neveu, Holden-Day, San Francisco, 1965. [40] A.B. Piunovskiy, Kluwer, Dordrecht, 1997. [41] A.B. Piunovskiy and X. Mao, \Constrained Markovian decision processes: the dynamic programming approach," 27, pp. 119-126, 2000. [42] E.L. Porteus, \Stochastic inventory theory," in (D.P. Heyman and M.J. Sobel, eds), Handbooks in Operations Research and Management Science, 2, pp. 605{652, North-Holland, Amsterdam, 1990. [43] M.L. Puterman, Wiley, New York, 1994. [44] M. Schal, \On stochastic dynamic programming: a bridge between Markov decision processes and gambling." Markov processes and control theory, pp. 178{216, Math. Res. 54, Akademie-Verlag, Berlin, 1989. [45] L. Sennott, Wiley, New York, 1999. [46] S. Ross, Academic Press, New York, 1983. [47] L.S. Shapley, \Stochastic games," , pp. 1095{1100, 1953. Further Topics in Discrete-Time
Markov Control Processes,
Stochastic Methods in Operations Re-
search. Volume II: Stochastic Optimization,
Foundations of Non-stationary Dynamic Programming with
Discrete Time Parameter
Dynamic Programming and Markov Processes
Linear Programming and Finite Markovian Control
Problems
Classical Descriptive Set Theory,
Controlled Queueing Systems,
Transportation Science,
Discrete
Gambling
and
Stochastic
Games,
Mathematical
Foundations
of
the
Calculus
of
Probability,
Optimal Control of Random Sequences in Problems
with Constraints,
Operations Research Letters
Stochastic
Models,
Markov Decision Processes,
Stochastic Dynamic Programming and the Control of Queue-
ing Systems,
Introduction to Stochastic Dynamic Programming,
Proceedings of the National Academy
of Sciences
INTRODUCTION
17
[48] A.N. Shiryaev, \On the theory of decision functions and control by an observation process with incomplete data," 6, pp.162{188, 1966. [49] N.L. Stokey and R.E. Lucas, Jr. Harvard University Press, Cambridge, 1989. [50] A. Wald, Wiley, New York, 1947. [51] P. Whittle, Wiley, NY, 1990 Selected Translations in Math.
Statistics and Probability
Recursive Methods in Economic Dynam-
ics,
Sequential Analysis,
Risk-Sensitive Optimal Control,
Eugene A. Feinberg Department of Applied Mathematics and Statistics SUNY at Stony Brook Stony Brook, 11794-3600, NY, USA
[email protected]
Adam Shwartz Department of Electrical Engineering Technion|Israel Institute of Technology Haifa 32000, Israel
[email protected]
2
FINITE STATE AND ACTION MDPS Lodewijk Kallenberg
In this chapter we study Markov decision processes (MDPs) with nite state and action spaces. This is the classical theory developed since the end of the fties. We consider nite and in nite horizon models. For the nite horizon model the utility function of the total expected reward is commonly used. For the in nite horizon the utility function is less obvious. We consider several criteria: total discounted expected reward, average expected reward and more sensitive optimality criteria including the Blackwell optimality criterion. We end with a variety of other subjects. The emphasis is on computational methods to compute optimal policies for these criteria. These methods are based on concepts like value iteration, policy iteration and linear programming. This survey covers about three hundred papers. Although the subject of nite state and action MDPs is classical, there are still open problems. We also mention some of them. Abstract:
2.1 2.1.1
INTRODUCTION Origin
Bellman's book [13], can be considered as the starting point of Markov decision processes (MDPs). However, already in 1953, Shapley's paper [221] on stochastic games includes as a special case the value iteration method for MDPs, but this was recognized only later on. About 1960 the basics for the other computational methods (policy iteration and linear programming) were developed in publications like Howard [121], De Ghellinck [42], D'Epenoux [55], Manne [164] and Blackwell [27]. Since the early sixties, many results on MDPs are published in numerous journals, monographs, books and proceedings. Thousands of papers were published in scienti c journals. There are about fty books on MDPs. Around 1970 a rst series of books was published. These books (e.g. Derman [58], Hinderer [107], Kushner [148], Mine and Osaki [167] and Ross [198]) contain the fundamentals of the theory of nite MDPs. Since that time nearly 21
22
HANDBOOK OF MARKOV DECISION PROCESSES
every year one or more MDP-books appeared. These books cover special topics (e.g. Van Nunen [250], Van der Wal [246], Kallenberg [134], Federgruen [69], Vrieze [260], Hernandez-Lerma [102], Altman [2] and Sennott [218]) or they deal with the basic and advanced theory of MDPs (e.g. Bertsekas [15], Whittle [289], [290], Ross [200], Dietz and Nollau [63], Bertekas [17], Denardo [50], Heyman and Sobel [106], White [285], Puterman [186], Bertsekas [18], [19], Hernandez-Lerma and Lasserre [103], [104], and Filar and Vrieze [79]. 2.1.2
The model
We will restrict ourselves to discrete, nite Markovian decision problems, i.e. the state space X and the action sets A (i); i 2 X, are nite, and the decision time points t are equidistant, say t = 1; 2; : : : . If, at time point t, the system is in state i and action a 2 A (i) is chosen, then the following happens independently of the history of the process: (1) a reward r(i; a) is earned immediately; (2) the process moves P to state j 2 X with transition probability p(j ji; a), where p(j ji; a)) 0 and j p(j ji; a) = 1 for all i; j and a. The objective is to determine a policy, i.e. a rule at each decision time point, which optimizes the performance of the system. This performance is expressed as a certain utility function. Such utility function may be the expected total (discounted) reward over the planning horizon or the average expected reward per unit time. The decision maker has to nd the optimal balance between immediate reward and future reward: a high immediate reward may bring the process in a bad situation for later rewards. In Chapter 1 several classes of policies are introduced: general policies, Markov policies and stationary policies. There are randomized and nonrandomized (pure) policies. Denote the set of pure stationary policies by F and a particular policy of that set by f. Let X A = f(i; a) j i 2 X; a 2 A (i)g, let the random variables Xt and Yt denote the state and action at time t and let IP ; [Xt = j; Yt = a] be the notation for the probability that at time t the state is j and the action is a, given that policy is used and is the initial distribution. The next theorem shows that for any initial distribution , any sequence of policies 1 ; 2 ; : : : and any convex combination of the marginal distributions of IP ;k ; k 2 N , there exists a Markov policy with the same marginal distribution. Theorem 2.1 Given any initial distribution , any sequence of P policies 1 , 2 ; : : : and any sequence of nonnegative real numbers p1 ; p2 : : : with k pk = 1; there exists a Markov policy such that for every (j; a) 2 X A
IP ; [Xt = j; Yt = a] =
X
p IP ;k [Xt k k
= j; Yt = a]; t 2 N :
(1:1)
Corollary 2.1 For any starting state i and any policy , there exists a Markov policy such that
IPi; [Xt = j; Yt = a] = IPi; [Xt = j; Yt = a]; t 2 N ; (j; a) 2 X A : (1:2)
FINITE STATE AND ACTION MDPS
23
The results of Theorem 2.1 and Corollary 2.1 imply the suÆciency of Markov policies for performance measures which only depend on the marginal distributions. Corollary 2.1 is due to Derman and Strauch [61] and the extension to Theorem 2.1 was given by Strauch and Veinott [237]. The result is further generalized to more general state and actions spaces by Hordijk [112] and Van Hee [247]. 2.1.3
Optimality criteria
Let v(i; ) be the utility function if policy is used and state i is the starting state, i 2 X. The value vector v of this utility function is de ned by v(i) := sup v(i; ); i 2 X:
(1:3)
A policy is an optimal policy if v(i; ) = v(i); i 2 X. In Markov decision theory the existence and the computation of optimal policies is studied. For this purpose a so-called optimality equation is derived, i.e. a functional equation for the value vector. Then a solution of this equation is constructed which produces both the value vector and an optimal policy. There are three standard methods to perform this: value iteration, policy iteration and linear programming. In value iteration the optimality equation is solved by successive approximation. Starting with some v0 ; vt+1 is computed from vt ; t = 0; 1; : : : . The sequence v0 ; v1 ; : : : converges to the solution of the optimality equation. In policy iteration a sequence of improving policies f0 ; f1 ; : : : is determined, i.e. v(ft+1 ) v(ft ) for all t, until an optimal policy is reached. The linear programming method can be used because the value vector is the smallest solution of a set of linear inequalities; an optimal policy can be obtained from its dual program. In this survey we consider the following utility functions: (1) total expected reward over a nite horizon; (2) total expected discounted reward over an in nite horizon; (3) average expected reward over an in nite horizon; (4) more sensitive optimality criteria for the in nite horizon. Suppose that the system has to be controlled over a nite planning horizon of T periods. As performance measure we use the total expected reward over the planning horizon, i.e. for policy we will consider for starting state i vT (i; ) :=
T X t=1
IEi; [r(Xt ; Yt )] =
T X X t=1 j;a
IPi; [Xt = j; Yt = a] r(j; a): (1:4)
A matrix P P = (pij ) is called a transition matrix if pij 0 for all (i; j ) and j pij = 1 for all i. Markov policies, and consequently also stationary policies, induce transition matrices. For the randomized Markov policy = (1 ; 2 ; : : : ) we de ne, for every t 2 N , the transition matrix P (t ) by X (1:5) [P (t )]ij := a p(j ji; a)t (i; a) for all i; j 2 X;
24
HANDBOOK OF MARKOV DECISION PROCESSES
and the reward vector r(t ) by X ri (t ) := a t (i; a)r(i; a) for all i 2 X
(1:6)
Hence the total expected reward for the Markov policy can be written in vector notation as XT vT (R) = t=1 P (1 )P (2 ) P (t 1 )r(t ): (1:7) It can be shown that an optimal Markov policy = (f1 ; f2 ; : : : ; fT ) exists, where ft is a pure decision rule 1 t T. The nonstationarity is due to the niteness of the planning horizon. Next, we consider an in nite planning horizon. In that case there is no unique optimality criterion. Di erent optimality criteria are meaningful: discounted reward, total reward, average reward or more sensitive criteria. The total expected -discounted reward, given discount factor 2 [0,1), initial state i and policy , is denoted by v (i; ) and de ned by X1 v (i; ) := t=1 IEi; [ t 1 r(Xt ; (Yt )] (1:8) X1 X = t=1 t 1 j;a IPi; [Xt = j; Yt = a]r(j; a): In section 1.3.1 it will be shown that there exists an optimal policy f 2 F and that any stationary policy satis es X1 v () = t=1 t 1 P ()t 1 r() = [I P ()] 1 r(): (1:9) When there is no discounting, i.e. the discount factor equals 1, then|for instance|we may consider the total expected reward and the average expected reward criterion. In the total expected reward criterion the utility function is P limT !1 Tt=1 IE[r(Xt ; Yt )]. Without further assumptions, this limit can be in nite or the limsup can be unequal to the liminf. When thePaverage reward T r(X ; Y ) is criterion is used, the limiting P behavior of the expectation of T1 P t t t=1 T T 1 1 considered. Since limT !1 T t=1 IE[r(Xt ; Yt )] or IE[limT !1 T t=1 r(Xt ; Yt )] does not exist, in general, and interchanging limit and expectation may not be allowed, there are four di erent evaluation measures, which can be considered for a given policy: (a) the lower limit of the average expected reward: 1 XT IE [r(X ; Y )]; i 2 X; (i; ) := lim inf (1:10) T !1
T
t=1
i;
t
t
(b) the upper limit of the average expected reward: XT (i; ) := lim sup T1 t=1 IEi; [r(Xt ; Yt )]; i 2 X; T !1
(1:11)
FINITE STATE AND ACTION MDPS
25
(c) the expectation of the lower limit of the average reward: XT r(X ; Y )]; i 2 X; (i; ) := IE [lim inf 1
(1:12)
(d) the expectation of the upper limit of the average reward: XT (i; ) := IE [lim sup 1 r(X ; Y )]; i 2 X:
(1:13)
i; T !1
i;
T !1
T
t=1
T
t=1
t
t
t
t
Lemma 2.1 (i) ( ) ( ) ( ) ( ) for every policy ; (ii) ( ) = ( ) = ( ) = ( ) for every stationary policy . Remark
In Bierth [26] it is shown that the four criteria are equivalent in the sense that the value vectors can be attained for one and the same deterministic policy. Examples can be constructed in which for some policy the inequalities of Lemma 2.1 part (i) are strict. The long-run average reward criterion has the disadvantage that it does not consider rewards earned in a nite number of periods. Hence, there may be a preference for more selective criteria. There are several ways to be more selective. One way is to consider discounting for discount factors that tend to 1. Another way is to use more subtle kinds of averaging. We will present some criteria and results. For all criteria it can be shown that optimal policies in class F exist and that these policies are (at least) average optimal. A policy is called n-discount optimal for some integer n 1; if lim inf "1 (1 ) n [v ( ) v ()] 0 for all policies . 0-discount optimality is also called bias-optimality. There is also the concept of n-average optimality. For any policy , any t 2 N and for n = 1; 0; 1; : : : , let the vector vn;t () be de ned by t v ( ) for n = 1 n;t v () := Pt vn 1;s () for n = 0; 1 : : : (1:14) s=1 is said to be n-average optimal if lim inf T !1 T1 [vn;T ( ) vn;T ()] 0 for all policies . A policy is said to be Blackwell optimal if is -discounted optimal for all discount factors 2 [ 0 ,1) for some 0 0 < 1. In a fundamental paper
Blackwell [27] presented a mathematically rigorous proof for the policy iteration method to compute an -discounted optimal policy. He also introduced the concept of bias-optimality (Blackwell called it nearly optimality) and established the existence of a discounted optimal policy for all discount factors suÆciently close to 1. In honor of Blackwell, such policy is called a Blackwell optimal policy. It can be shown that n-discount optimality is equivalent to n-average optimality, that (-1)-discount optimality is equivalent to average optimality, and
26
HANDBOOK OF MARKOV DECISION PROCESSES
that Blackwell optimality is n-discount optimality for all n N 1; where N = #X (in this chapter we will always use the notation N for the number of states). The n-discount optimality criterion and the policy iteration method for nding an n-discount optimal policy, were proposed by Veinott [257]. He also showed that Blackwell optimality is the same as n-discount optimality for n N 1: Sladky [223] has introduced the concept of n-average optimality; furthermore, he also showed the equivalence between this criterion and n-discount optimality. More details on bias optimality and Blackwell optimality can be found in Chapter 3 and Chapter 8. 2.1.4
Applications
White has published three papers on `real applications' of Markov decision theory (White [280], [281] and [284]). Many stochastic optimization problems can be formulated as MDPs. In this section we shortly introduce the following examples: routing problems, stopping and target problems, replacement problems, maintenance and repair problems, inventory problems, the optimal control of queues, stochastic scheduling and multi-armed bandit problems. In this book there are also chapters on applications in nance (Chapter 15) and in telecommunication (Chapter 16). We also mention the contribution Chapter 17 on water reservoir applications. Routing problems
In routing problems the problem is to nd an optimal route through a network. Well known is the shortest path problem. A shortest path problem in a layered network can be formulated as an MDP over a nite horizon. Another application of this kind is the maximum reliability problem. In this network the connections are unreliable: let pij be the probability of reaching node j when the arc from node i to node j is chosen. The objective is to maximize the probability of reaching a terminal node n when the process is started in some node, say node 1. Results for a stochastic version of the shortest path problem can for instance be found in Bertsekas and Tsitsiklis [23]. The maximum reliability problem is discussed in Roosta [194]. Optimal stopping problems
In an optimal stopping problem there are two actions in each state. The rst action is the stopping action and the second action corresponds to continue. If we continue in state i, a cost ci is incurred and the probability of being in state j at the next time point is pij . If the stopping action is chosen in state i, then a nal reward ri is earned and the process terminates. In an optimal stopping problem, in each state one has to determine which action is chosen with respect to the total expected reward criterion. This kind of problem often has an optimal policy that is a so-called control limit policy. The original analysis of optimal stopping problems appeared in Derman and Sacks [60], and Chow and Robbins [36]. A dynamic programming approach can be found in Breiman [28] who showed the optimality of a control limit policy.
FINITE STATE AND ACTION MDPS
27
Target problems
In a target problem one wants to reach a distinguished state (or a set of states) in some optimal way, where in this context optimal means, for instance, at minimum cost or with maximum probability. The target states are absorbing, i.e. there are no transitions to other states and the process can be assumed to terminate in the target states. These target problems can be modeled as MDPs with the total expected reward as optimality criterion. To the class of target problems we may count the so-called rst passage problem. In this problem there is one target state and the objective is to reach this state (for the rst time) at minimum cost. A second class of target problems are gambling problems (the gambler's goal is to reach a certain fortune N and the problem is to determine a policy which maximizes the probability to reach this goal). For more information about MDPs and gambling problems we refer to Chapter 13. The rst passage problem was introduced by Eaton and Zadeh [67] under the name \pursuit problem". The dynamic programming approach was introduced in Derman [56]. A standard reference on gambling is Dubins and Savage [64]. Dynamic programming approaches are given in Ross [199] and Dynkin [66]. Replacement problems
Consider an item which is in a certain state. The state of the item describes its condition. Suppose that in each period, given the state of the item, the decision has to be made whether or not to replace the item by a new one. When an item of state i is replaced by a new one, the old item is sold at price si , a new item is bought at price c, and the transition to the new state is instantaneous. In case of nonreplacement, let pij be the probability that an item of state i is at the beginning of the next period in state j , and suppose that ci is the maintenance cost|during one period|for an item of state i. This problem can be modeled as an MDP. It turns out that for the computation of an optimal policy an eÆcient algorithm, with complexity O(N 3 ), exists (see Gal [83]). Next, we mention the model of deterioration with failure. In this model the states are interpreted as `ages'. In state i there is a failure probability pi and, when failure occurs, there is an extra cost fi and the item has to be replaced by a new one. If there is no failure the next state is state i +1: It can be shown that, under natural assumptions about the failure probabilities and the costs, a control limit policy is optimal, i.e. there is an age i and the item is replaced by a new one if its age exceeds i . This property holds for the discounted reward criterion as well as for the average reward criterion. There are a lot of references on replacement models. The early survey of Sherif and Smith [222] contained already over 500 references. Results on the optimality of control limit policies for replacement problems can be found in Derman [57, 58], Kolesar [146], Ross [198] and Kao [138]. Maintenance and repair problems
In maintenance and repair problems there is a system which is subject to deterioration and failure. Usually, the state is a characterization of the condition of the system. When the state is observed, an action has to be chosen, e.g. to keep the system unchanged, to execute some maintenance or repair, or to replace one or more components by new ones. Each action has corresponding
28
HANDBOOK OF MARKOV DECISION PROCESSES
costs. The objective is to minimize the total costs, the discounted costs or the average costs. These problems can easily be modeled as an MDP. A one-component problem is described in Klein [145]. The two-component maintenance problem was introduced by Vergin and Scriabin [259]. Other contributions in this area are e.g. Oezekici [173], and Van der Duyn Schouten and Vanneste [244]. An n-component series system is discussed in Katehakis and Derman [139]. Asymptotic results for highly reliable systems can be found in Smith [226], Katehakis and Derman [140], and Frostig [81]. Inventory problems
In inventory problems an optimal balance between inventory costs and ordering costs has to be determined. We assume that the probability distribution of the demand is known. There are di erent variants of the inventory problem. They di er, for instance, in the following aspects: - stationary or nonstationary costs and demands; - a nite planning horizon or an in nite planning horizon; - backlogging or no backlogging. For all these variants di erent performance measures may be considered. In many inventory models the optimal policy is of (s; S )-type, i.e. when the inventory is smaller than or equal to s, then replenish the stock to level S . The existence of optimal (s; S )-policies in nite horizon models with xed cost K is based on the so-called K -convexity, introduced by Scarf [202]. The existence of an optimal (s; S )-policy in the in nite horizon model is shown by Iglehart [126]. Another related paper is Veinott [255]. For the relation between discounted and average costs we refer to Hordijk and Tijms [119]. For the computation of the values s and S we refer to papers like Federgruen and Zipkin [76], and Zheng and Federgruen [292]. Optimal control of queues
Consider a queueing system where customers arrive according to a Poisson process and where the service time of a customer is exponentially distributed. Suppose that the arrival and service rates can be controlled by a nite number of actions. When the system is in state i, i.e. there are i customers in the system, action a means that the arrival or the service rates are i (a) or i (a), respectively. The arrival and service processes are continuous-time processes. However, by the memoryless property of the exponential distribution, we can nd an embedded discrete-time Markov chain which is appropriate for our analysis. This technique is called uniformization (see e.g. Tijms [241]). A queue, or a network of queues, is a useful model for many applications, e.g. manufacturing, computer, telecommunication and traÆc systems. See the survey of MDPs in telecommunication, Chapter 16. Control models can optimize certain performance measures by varying the control parameters of the system. We distinguish between admission control and service rate control. In a service rate model, the service rate can be chosen from an interval [0; ]. If rate is chosen, there are service costs c() per period; we also assume that there are holding costs h(i) per period when there are i customers in the system. Under natural conditions it can be shown that a bang-bang policy is optimal, i.e. = 0 or = . For details see Weber and Stidham [268]. Surveys of optimal
FINITE STATE AND ACTION MDPS
29
control of (networks of) queues can be found in the book by Walrand [265] and the papers by Stidham [234] and Stidham and Weber [235]. Stochastic scheduling
In a scheduling problem, jobs are processed on machines. Each machine can process only one job at a time. A job has a given processing time on the machines. In stochastic scheduling, these processing times are random variables. At certain time points decisions have to be made, e.g. which job is assigned to which machine. There are two types of models: the customer assignment models, in which each arriving customer has to be assigned to one of the queues (each queue with its own server) and server assignment models, where the server has to be assigned to one of the queues (each queue has its own customers). Also in queueing models optimal policies often have a nice structure. Examples of this structure are: - c-rule : this rule assigns the server to queue k, with k the queue with k ck = maxi fi ci j queue i is nonemptyg, where ci is the cost which is charged per unit of time that the customer is in queue i and the service times in queue i are geometrically distributed with rate i ; - shortest queue policy (SQP): an arriving customer is assigned to the shortest queue; - longest expected processing time (LEPT): the jobs are allocated to the machines in decreasing order of their expected processing times; - shortest expected processing time (SEPT): the jobs are allocated to the machines in increasing order of their expected processing times. The optimality of the c-rule is established in Baras, Ma and Makowsky [9]. Ephremides, Varayia and Walrand [68] have shown the optimality of the shortest queue policy. The results for the optimality of the LEPT and SEPT policies are due to Bruno, Downey and Frederickson [30]. Related results are obtained by Weber [266] and by Chang, Hordijk, Righter and Weiss [33]. For reviews on stochastic scheduling we refer to Weiss [269], Walrand [265] (chapter 8) and Righter [193]. Multi-armed bandit problem
The multi-armed bandit problem is a model for dynamic allocation of a resource to one of n independent alternative projects. Any project may be in one of a nite number of states. At each period the decision maker has the option of working on exactly one of the projects. When a project is chosen, the immediate reward and the transition probabilities only depend on the active project and the states of the remaining projects are frozen. Applications of this model appear in machine scheduling, in the control of queueing systems and in the selection of decision trials in medicine. It can be shown that an optimal policy is the policy that selects the project which has the largest so-called Gittinsindex. Fortunately, these indices can be computed for each project separately. As a consequence, the multi-armed bandit problem can be solved by a sequence of n one-armed bandit problems. This is a decomposition result by which the dimensionality of the problem is reduced considerably. EÆcient algorithms for the computation of the Gittins indices exist. The most fundamental contribution on multi-armed bandit problems was made by Gittins (cf. Gittins and
30
HANDBOOK OF MARKOV DECISION PROCESSES
Jones [86], and Gittins [85]). In Whittle [288] an elegant proof is presented. Other proofs are given by Ross [200], Varaiya, Walrand and Buyoccoc [254], Weber [267] and Tsitsiklis [243]. Several methods are developed for the computation of the Gittins indices: Varaiya, Walrand and Buyukkoc [254], Chen and Katehakis [35], Kallenberg [135], Katehakis and Veinott [141], Ben-Israel and S.D.Flam [14], and Liu and Liu [155]. 2.2
FINITE HORIZON
Consider an MDP with a nite horizon of T periods. In fact, we can analyze with the same e ort a nonstationary MDP, i.e. with rewards and transition probabilities which may depend on the time t (1 t T ). These nonstationary rewards and transition probabilities are denotated by rt (i; a) and pt (j ji; a). By the principle of optimality, an optimal policy can be determined by backward induction as the next theorem shows. The proof can be given by induction on the length T of the horizon. The use of the principle of optimality and the technique of dynamic programming for sequential optimization was provided by Bellman [13]. Theorem 2.2 Let xTi +1 = 0; i 2 X. Determine for t = T; T decision rule f t such that
[rt (f t )]i + [P (f t )xt+1 ]i = amax frt (i; a) + 2A (i) and let xt = rt (f t ) + P t (f t )xt+1 . Then, R policy and x1 is the value vector.
X
j
1; :::; 1 a pure
pt (j ji; a) xtj+1 g; i 2 X;
= (f 1 ; f 2 ; :::; f T )
is an optimal
P If [rt (f t )]i +[P t (f t )xt+1 ]i = maxa2A(i) frt (i; a)+ j pt (j ji; a)xtj+1 g; i 2 X, then we denote rt (f t )+ P t (f t )x = maxXA frt + P t xg and f t 2 argmaxfrt + P t xg.
X A
Algorithm I ( nite horizon) 1. x := 0.
2. Determine for t = T; T 1; : : : ; 1 : f t 2 argmaxfrt + P t xg and x := rt (f t ) + P t (f t )x:
3. R := (f 1 ; f 2; : : : ; f T ) is an optimal policy and x is the value vector. X A
Remarks
1. It is also possible to include in this algorithm elimination of suboptimal . Suboptimal actions are actions that will not occur in an optimal policy. References are Hastings and Van Nunen [99] and Hubner [124]. 2. A nite horizon nonstationary MDP can be transformed in an equivalent stationary in nite horizon model. In such an in nite horizon model other options, as the treatment of side constraints, also called additional constraints, are applicable. These results can be found in Derman and Klein [59] and in Kallenberg [131], [132]. actions
3
BIAS OPTIMALITY Mark E. Lewis Martin L. Puterman
The use of the long-run average reward or the gain as an optimality criterion has received considerable attention in the literature. However, for many practical models the gain has the undesirable property of being underselective, that is, there may be several gain optimal policies. After nding the set of policies that achieve the primary objective of maximizing the long-run average reward one might search for that which maximizes the \short-run" reward. This reward, called the bias aids in distinguishing among multiple gain optimal policies. This chapter focuses on establishing the usefulness of the bias in distinguishing among multiple gain optimal policies, computing it and demonstrating the implicit discounting captured by bias on recurrent states. Abstract:
3.1
INTRODUCTION
The use of the long-run average reward or the gain as an optimality criterion has received considerable attention in the literature. However, for many practical models the gain has the undesirable property of being underselective, that is, there may be several gain optimal policies. Since gain optimality is only concerned with the long-run behavior of the system there is the possibility of many gain optimal policies. Often, this leads decision-makers to seek more sensitive optimality criteria that take into account short-term system behavior. We consider a special case of the sensitive optimality criteria which are considered in Chapter 8 of this volume. Suppose the manager of a warehouse has decided through market studies and a bit of analysis that when long-run average cost is the optimality criterion an \(s; S )" ordering policy is optimal. That is to say that past demand patterns suggest it is optimal to reorder when the inventory falls below the level s and that it should be increased to S units when orders are made. Furthermore, suppose that there are many such limits that achieve long-run average 89
90
HANDBOOK OF MARKOV DECISION PROCESSES
optimality. With this in mind, the manager has arbitrarily chosen the longrun average optimal policy (s ; S ). In fact, in this example the manager could choose any ordering policy for any ( nite) amount of time, and then start using any one of the optimal average cost policies and still achieve the optimal average cost. However, the decision-maker should be able to discern which of the optimal average cost policies is best from a management perspective and use that policy for all time. The use of the bias can assist in making such decisions. In essence, after nding the set of policies that achieve the primary objective of maximizing the long-run average reward we search for a policy which maximizes the bias. In very simple models with a single absorbing state and multiple policies to choose from on transient states the concept of bias optimality is easy to understand. In these models all policies are average optimal and the bias optimal policy is the one which maximizes the expected total reward before reaching the absorbing state. Consider the following simple example: 0
0
Let X = f1; 2g, A1 = fa; bg, and A2 = fcg. Furthermore, let p(2j1; a) = p(2j1; b) = p(2j2; c) = 1 and r(1; a) = 100, r(1; b) = 1, and r (2; c) = 1. It is easy to see that an average reward maximizing decision-maker would be indi erent which action is chosen in state 1, but any rational decisionmaker would clearly prefer action a in state 1. The analysis in this chapter will show, among other things, that using bias will resolve this limitation of the average reward criterion. Example 3.1
Unfortunately, this example gives an oversimpli ed perspective of the meaning of bias. In models in which all states are recurrent or models in which different policies have di erent recurrent classes, the meaning of bias optimality is not as transparent. It is one of our main objectives in this chapter to provide some insight on this point by developing a \transient" analysis for recurrent models based on relative value functions. We present an algorithmic and a probabilistic analysis of bias optimality and motivate the criterion with numerous examples. The reader of this chapter should keep the following questions in mind: How is bias related to average, total, and discounted rewards? How do we compute the bias? How are bias and gain computation related? In a particular problem, what intuition is available to identify bias optimal policies? Can we use sample path arguments to identify bias optimal policies? How is bias related to the timing of rewards? What does bias really mean in recurrent models?
4
SINGULAR PERTURBATIONS OF MARKOV CHAINS AND DECISION PROCESSES Konstantin E. Avrachenkov
Jerzy Filar
Moshe Haviv
In this survey we present a uni ed treatment of both singular and regular perturbations in nite Markov chains and decision processes. The treatment is based on the analysis of series expansions of various important entities such as the perturbed stationary distribution matrix, the deviation matrix, the mean-passage times matrix and others. Abstract:
4.1
BACKGROUND AND MOTIVATION
Finite state Markov Chains (MC's) are among the most widely used probabilistic models of discrete event stochastic phenomena. Named after A.A. Markov, a famous Russian mathematician, they capture the essence of the existentialist \here and now" philosophy in the so-called \Markov property" which, roughly speaking, states that probability transitions to a subsequent state depend only on the current state and time. This property is less restrictive than might appear at rst because there is a great deal of exibility in the choice of what constitutes the \current state". Because of their ubiquitous nature Markov Chains are, nowadays, taught in many undergraduate and graduate courses ranging from mathematics, through engineering to business administration and nance.
This research was supported in part by the ARC grant #A49906132.
113
114
HANDBOOK OF MARKOV DECISION PROCESSES
Whereas a MC often forms a good description of some discrete event stochastic process, it is not automatically equipped with a capability to model such a process in the situation where there may be a \controller" or a \decisionmaker" who|by a judicious choice of actions|can in uence the trajectory of the process. This innovation was not introduced until the seminal works of Howard [44] and Blackwell [16] that are generally regarded as the starting point of the modern theory of Markov Decision Processes (or MDP's for short). Since then, MDP's have evolved rapidly to the point that there is now a fairly complete existence theory, and a number of good algorithms for computing optimal policies with respect to criteria such as maximization of limiting average expected reward, or the discounted expected reward. The bulk of the, now vast, literature on both MCs and MDPs deals with the \perfect information" situations where all the model parameters|in particular probability transitions|are assumed to be known precisely. However, in most applications this assumption will be violated. For instance, a typical parameter, , would normally be replaced by an estimate ^ = + "(n)
where the error term, "(n), comes from a statistical procedure used to estimate and n is the number of observations used in that estimation. In most of the valid statistical procedures j"(n)j # 0 as n " 1, in an appropriate sense. Thus, from a perturbation analysis point of view, it is reasonable to suppress the argument n and simply concern ourselves with the e ects of " ! 0. Roughly speaking, the subject of perturbation analysis of MC's and MDP's divides naturally into the study of \regular" and \singular" perturbations. Intuitively, regular perturbations are \good" in the sense that the e ect of the perturbation dissipates harmlessly as " ! 0, whereas singular perturbations are \bad" in the sense that small changes of " (in a neighborhood of 0) can induce \large" e ects. Mathematically, it can be shown that singular perturbations are associated with a change of the rank of a suitably selected matrix. This can be easily seen from the now classical example due to Schweitzer [66] where the perturbed probability transition matrix " " "#0 1 0 1 2 2 ! P (0) = 0 1 : P (") = " 1 2" 2 but the limit matrix for " > 0 0 1 2 P (") lim P t (") = @ t!1 1 2
1 2 1 2
1 A 6! P (0) lim P t (0) = 1 0 : t!1 0 1
Indeed, the rank of P (") is 1 for all " > 0 (regardless of how close it is to zero), but it jumps to 2 at " = 0; despite the fact that P (") ! P (0). Thus, we see that singular perturbations can occur in MC's in a very natural and essential way. The latter point can be underscored by observing the behavior of (I P ) 1 as ! 1, where P is a Markov (probability transition) matrix. It is well-known (e.g., see [16] or [57]) that this inverse can be expanded as a
SINGULAR PERTURBATIONS OF MARKOV CHAINS AND DECISION PROCESSES
115
Laurent series (with a pole of order 1) in the powers of " := 1 : Indeed, much of the theory devoted to the connections between the discounted and limiting average MDP's exploits the asymptotic properties of this expansion, as " # 0: Of course, the rank of (I P ) changes when = 1(" = 0). It is not surprising, therefore, that the literature devoted to singularly perturbed MC's and MDP's has been growing steadily in recent years. In fact there have been quite a few developments since the 1995 survey by Abbad and Filar [2]. The purpose of this survey paper is to present an up to date outline of a uni ed treatment of both singular and regular perturbations in MC's and MDP's that is based on series expansions of various important entities such as the perturbed stationary distribution matrix, the deviation matrix, the meanpassage times matrix and the resolvent-like matrix (I P ) 1 . From this series expansion perspective, the regular perturbations are simply the cases where Laurent series reduce to power series. Consequently, the capability to characterize and/or compute the coeÆcients of these expansions and the order of the pole (if any, at " = 0) becomes of paramount importance. This survey covers only the results on discrete time MC's and MDP's. For a parallel development of the continuous time models we refer an interested reader to the comprehensive book of Yin and Zhang [76]. For related emerging results in game theory see Altman et al. [6]. The logical structure of the survey is as follows: in Section 2, perturbations of (uncontrolled) Markov chains are discussed from the series expansions perspective; in Section 3 the consequences of these results are discussed in the context of optimization problems arising naturally in the (controlled) MDP case; and, nally, in Section 4 applications of perturbed MDP's to the Hamiltonian Cycle Problem are outlined. This last section demonstrates that the theory of perturbed MDP's has applications outside of its own domain. 4.2 4.2.1
UNCONTROLLED PERTURBED MARKOV CHAINS Introduction and preliminaries
Let P 2 Rnn be a transition stochastic matrix representing transition probabilities in a Markov chain. Suppose that the structure of the underlying Markov chain is aperiodic. Let P = limt!1 P t which is well-known to exist for aperiodic processes. In the case when the process is also ergodic, P has identical rows, each of which is the stationary distribution of P , denoted by . Let Y be the deviation matrix of P which is de ned by Y = (I P + P ) 1 P . It is well known (e.g., see [51]) that Y exists and it is the unique matrix satisfying Y (I
P) = I
P = (I
P )Y
and P Y = 0 = Y 1
(4.1)
(where 1 is a matrix full of 1's) making it the group inverse of I P . Finally, Y = limT !1 Tt=0 (P t P ). Let Mij be the mean passage time from state-i into state-j. It is also known that when the corresponding random variable is proper, then Mij is nite. Of course, the matrix M is well-de ned if and only
5
AVERAGE REWARD OPTIMIZATION THEORY FOR DENUMERABLE STATE SPACES Linn I. Sennott
5.1
INTRODUCTION
In this chapter we deal with certain aspects of average reward optimality. It is assumed that the state space X is denumerably in nite, and that for each x 2 X, the set A (x) of available actions is nite. It is possible to extend the theory to compact action sets, but at the expense of increased mathematical complexity. Finite action sets are suÆcient for digitally implemented controls, and so we restrict our attention to this case. For initial state x, the quantity W (x) is the best possible limiting expected average reward per unit time (average reward, for short). This is an appropriate measure of the largest expected reward per unit time that can possibly be achieved far into the future, neglecting short-term behavior. Many interesting applications have the property that the average reward is independent of the initial state, i.e. W (x) is a constant. This chapter develops a theory to guarantee the existence of a stationary policy and nite constant W such that W (x) = w(x; ) W;
x 2 X:
(5.1)
Such a policy is an average reward optimal stationary policy. In this chapter a stationary policy means a nonrandomized (pure) stationary policy. Implementing such a policy requires the controller to know only the current state x of the system. Table look-up may then determine the xed action (x) appropriate in that state. 153
154
HANDBOOK OF MARKOV DECISION PROCESSES
The development takes place under the assumption that there exists a nonnegative ( nite) constant R such that r(x; a) R, for all x 2 X and a 2 A (x). Note that rewards may be unbounded below. In some applications, the actual reward is a random quantity. In these cases, r(x; a) is to be interpreted as an expected reward. In a typical reward maximization setting, it may be possible to incur costs as well as earn rewards. Costs can be built into the system as negative rewards. For example, to minimize over the set f5; 2; 8g of costs, we may calculate maxf 5; 2; 8g = 2, and then the answer is ( 2) = 2. Our framework allows rewards to be unbounded below, thereby handling the common case of costs unbounded above. For example, queueing control problems may involve holding costs that are linear in the number of customers. If the bu ers are unlimited (able to hold all arriving customers), then this would entail costs unbounded above. The theory does not allow the controller to earn arbitrarily large positive rewards. This is not a severe limitation in queueing control problems and other applications. For example, assume that the controller earns a unit reward each time a customer is admitted to the system. If the number of customers that can arrive in any slot is bounded, then the assumption will hold. If the distribution on customer batch sizes is unbounded, then we may allow the controller to earn a reward that is a function of the mean batch size. We may de ne a new reward structure by subtracting R from the rewards in the original system. By so doing, the optimal policy will not be a ected, and it will be the case that all rewards are nonpositive. Let us assume that this has already been done, so that for the rest of the chapter we make the following assumption. Assumption 5.1
We have r(x; a) 0, for all x 2 X and a 2 A (x).
Note that to recover the average reward in the original setting, it is only necessary to add R to W . To motivate our approach, let us consider the situation when X is nite. In this case, it is well-known that there exist 0 2 (0; 1) and a stationary policy that is discount optimal for 2 ( 0 ; 1). Such a policy is called Blackwell optimal, and it must also be average optimal. These claims are proved in the chapter by Hordijk and Yushkevich in this volume; also see Sennott [37, Proposition 6.2.3]. Note that in the general case, W (x) may not be constant. To motivate the assumptions to be introduced in Section 3, we give the following result. It was stated in [37, Proposition 6.4.1] for the cost minimization framework, and the proof may be recast into the reward maximization framework. Proposition 5.1
Let
X
be nite. The following are equivalent:
(i) W (x) W , for x 2 X. (ii) There exists z 2 X and a nite constant L such that jV (x; ) L, for all x 2 X and 2 (0; 1).
V (z; )j
(iii) Given y 2 X, there exists a nite constant L such that jV (x; ) V (y; )j L, for all x 2 X and 2 (0; 1).
6
TOTAL REWARD CRITERIA Eugene A. Feinberg
This chapter deals with total reward criteria. We discuss the existence and structure of optimal and nearly optimal policies and the convergence of value iteration algorithms under the so-called General Convergence Condition. This condition assumes that, for any initial state and for any policy, the expected sum of positive parts of rewards is nite. Positive, negative, and discounted dynamic programming problems are special cases when the General Convergence Condition holds. Abstract:
6.1
INTRODUCTION
This chapter deals with the total reward criterion. This criterion is natural for nite horizon problems and for in nite horizon problems in which the number of steps is not xed but it is nite along most trajectories. The examples include discounted criteria, which can be interpreted as problems with geometrically distributed horizon, as well as other problems such as sequential statistical procedures, stopping, search, and optimal selection problems. The analysis of nite horizon models is usually based on the analysis of optimality equations and optimality operators. The value function satis es the optimality equation and a Markov policy constructed by value iteration procedures is optimal. If optimality cannot be achieved, one can sequentially construct an "-optimal Markov policy for an N -horizon problem by backward induction. Such a policy is formed by actions for which the reward operators, when applied to the value function at the corresponding step, are ("=N )-close to the optimal value. For in nite horizon problems, we can distinguish two distinct approaches. The rst approach deals with the analysis of optimality (also called, dynamic programming) equations and operators. The second approach studies probability distributions on the sets of trajectories. In fact, the most interesting results have been achieved by combining these two approaches. 173
174
HANDBOOK OF MARKOV DECISION PROCESSES
As was observed in the sixties, certain properties of optimality operators, namely contracting and motonicity properties, imply the existence of optimal or nearly optimal policies within natural classes of policies. These properties also imply the convergence of algorithms. In general, dynamic programming operators may not possess these properties. However, if the one-step reward function is uniformly bounded and the nonnegative discount factor is less then one, then the contracting property holds. If rewards are nonnegative (nonpositive) then the value iteration algorithm, applied to the zero terminal value, forms a monotone and therefore convergent sequence. The mentioned three models are called discounted, positive, and negative respectively. The rst comprehensive results were obtained for these three models. Blackwell [8] Denardo [18], Strauch [61] studied discounted models, Blackwell [9], Ornstein [50], and Strauch [61] studied positive models, and Strauch [61] studied negative models. The results di er signi cantly from one model to another. To illustrate these di erences, let us consider the situation when the state space is countable and there are no additional assumptions such as compactness of action sets. For discounted and positive models, for each initial state the supremum of the expected total rewards over the class of all policies is equal to the corresponding supremum over the class of all stationary policies; Blackwell [8, 9]. However, it is not true for negative models; Example 6.5. For discounted models, for any positive constant ", there exist stationary "-optimal policies. Such policies may not exist for positive models; Blackwell [9]. However, for positive models there exist stationary "V -optimal policies also called multiplicatively "-optimal; Ornstein [50]. There are other signi cant di erences between positive, negative, and discounted programming. The di erences are so signi cant that for a long period of time it was even was not clear how to formulate uni ed results. As the result, all textbooks on dynamic programming and Markov decision processes, that deal with in nite-horizon models with total rewards, consider only positive, negative, and discounted models and deal with them separately; see e.g. Ross [53] and Puterman [52]. In addition to positive, negative, and discounted models, problems with arbitrary reward functions and without discounting have been considered in the literature for a long period of time. It turned out that the most comprehensive results can be proved when the so-called General Convergence Condition holds. This condition means that the positive part of the reward function satis es the positive programming assumptions. Positive, negative, and discounted models are particular cases of models satisfying the General Convergence Condition. Blackwell [7] and Krylov [45] proved the existence of stationary optimal policies for MDPs with nite state and action sets. Dubins and Savage [20] and Hordijk [43] described necessary and suÆcient conditions for optimality, so-called conserving and equalizing conditions. Derman and Strauch [19] and Strauch [61] proved that, for a given initial state, any policy can be substituted with an equivalent randomized Markov policy. Krylov [45] and Gikhman and Skorohod [40] showed that nonrandomized policies are as good as randomized policies. Dynkin and Yushkevich [21] proved that if someone randomizes between di erent policies, the objective function cannot be improved. Feinberg [22, 23] proved that, for a given initial state, (nonrandomized) Markov
TOTAL REWARD CRITERIA
175
policies are as good as general ones; earlier van Hee [70] showed that the supremum of the expected total rewards over the class of Markov policies is equal to the supremum over the class of all policies. Seminal papers by Blackwell [8, 9] and Strauch [61] dealt with models with Borel state and action spaces. Some of the following papers on total reward MDPs dealt just with countable state spaces. For some results, their extension from countable to uncountable models is a straightforward exercise. For other results, such extensions are either diÆcult or impossible. In addition, Blackwell and Strauch considered Borel-measurable policies and discovered that "-optimal policies may not exist but for any initial measure they exist almost everywhere. In order to establish the existence of everywhere "-optimal policies, one should expand the set of policies to universally measurable or analytically measurable policies. Such extension was introduced by Blackwell, Freedman, and Orkin [11] and it was done in a systematic and comprehensive way in the book by Bertsekas and Shreve [5]. In particular, this book expanded in a natural way almost all results on positive, negative and discounted programming in a way that "-optimality was established instead of almost sure "-optimality. The major exception was Ornstein's theorem [50]. It was proved by Ornstein [50] for a countable state space; see also Hordijk [43]. It's extension to Borel models in the sense of almost sure multiplicative nearly-optimality was formulated by Blackwell [9] as an open question. Frid [38] solved this problem (Schal and Sudderth [59] found a correctable gap in Frid's proof). The natural conjecture is that under more general measurability assumptions, in the spirit of Bertsekas and Shreve [5], almost sure nearly-optimality can be replaced with nearly optimality everywhere in Ornstein's theorem. Blackwell and Ramachandran [12] constructed a counter-example to this conjecture. For the General Convergence Condition, signi cantly deeper results are available for countable state models than for Borel state problems. First, three important particular results were discovered by van der Wal: (i) the supremum of the expected total rewards over all policies is equal to the supremum over stationary policies if the action sets are nite; [66, Theorem 2.22] (this result was generalized by Schal [56] to Borel state models with compact action sets); (ii) extension of Ornstein's theorem to models in which rewards may be nonpositive and in each state, where the value function is nonpositive, a conserving action exists; [68] (this results was generalized by van Dawen and Schal [65, 64]); (iii) existence of uniformly nearly-optimal Markov policies; [67]. The survey by van der Wal and Wessels [69] describes these and many preceding results. Feinberg and Sonin [34] generalized Ornstein's [50] theorem to models satisfying the General Convergence Condition. For the long period of time, it had not been clear even how to formulate such results. The rst clue is that in the more general formulation the value function of the class of stationary policies should be considered instead of the value function of the class of all policies. The second clue is that, in the de nition of multiplicative "-optimal policies (or in the de nition of "V -optimal policies according to another terminology), the value function V should be replaced with an excessive majorant of a value function of the class of stationary policies. The proofs in Feinberg and Sonin [34] are non-trivial and di er from Ornstein's [36] proofs. Feinberg and Sonin [36]
176
HANDBOOK OF MARKOV DECISION PROCESSES
extended these results to non-stationary policies and to more general classes of functions that approximate optimal values. Feinberg [25, 26] described the structure of uniformly nearly-optimal policies in countable state models satisfying the General Convergence Condition. As mentioned above, stationary policies can be signi cantly outperformed by nonstationary policies in negative problems; see Example 6.5. It turned out that this example demonstrates the only pathological situation when the value of the class of stationary policies is less than the value of the class of all policies. Consider the set of states at which the value function equals zero and there are no conserving actions (actions at which the optimality operator applied to the value function achieves its maximum). Then there are policies which are uniformly nearly optimal and which are stationary outside of this set; see Theorems 6.20 and 6.21. Another important feature of the papers by Feinberg and Sonin [36, 25, 26] is that they consider general classes of nonstationary policies and general methods how to deals with nonstationary policies. In particular, the information about the past plays an important role. It is possible to identify two properties of this information: (i) Non-Repeating and (ii) Transitivity Conditions. The Non-Repeating condition implies that randomized policies are as good as nonrandomized ones and there exist uniformly nearly optimal policies within any class of policies that satis es this condition. The Transitivity Condition implies that the model can be transformed into a new model in a way that the class of policies satisfying this condition in the old model becomes the class of stationary policies in the new one. This paper is a survey of results and methods for models satisfying the General Convergence Condition. We consider countable state MDPs everywhere in this paper, except Section 6.10. We present the theory for countable state MDPs, discuss Borel state MDPs, and discuss open questions, most of which deal with uncountable state spaces. In order to illustrate major concepts and counter-examples, we start our presentation with classical discounted, positive, and negative problems. 6.2
DEFINITIONS OF DISCOUNTED, POSITIVE, NEGATIVE, AND GENERAL CONVERGENT MODELS
We say that an MDP is discounted if function r is bounded and there is a constant 2 [0; 1[, called the discount factor, such that v (x; ) = IEx
X1 t=0
t r(xt ; at ):
(6.1)
An MDP is called unbounded discounted if (6.1) holds and the function r is bounded above, r(x; a) C < 1 for all x 2 X; a 2 A (x) and for some C: Discounted and unbounded discounted MDPs can be reduced to an MDP with a discount factor equal to 1. In order to do it, we add an additional state to the state space X. This state has only one action under which it is absorbent and all rewards are equal to zero in this state. This state sometimes is called a grave. The one-step transition probabilities between states in X
7
MIXED CRITERIA Eugene A. Feinberg Adam Shwartz
Mixed criteria are linear combinations of standard criteria which cannot be represented as standard criteria. Linear combinations of total discounted and average rewards as well as linear combinations of total discounted rewards with di erent discount factors are examples of mixed criteria. We discuss the structure of optimal policies and algorithms for their computation for problems with and without constraints. Abstract:
7.1
INTRODUCTION
The discounted cost criterion is widely and successfully used in various application areas. When modeling economic phenomena, the discount factor models the value of money in time and is determined by return rate (or interest rate) or, in a more general context, by the \opportunity costs" which presume that a dollar now is worth more that a dollar in a year. Being invested, current funds will bring an additional return in a year. In other areas, such as the control of communications networks, the discounted criterion may re ect the imprecise but fundamental principle that future occurrences are less important than immediate ones. In reliability, discounting models systems with geometric life time distributions. Obviously, if some part of the cost decreases (in time) at an exponential rate, then a discounted cost arises. This is the case, for example, in production processes. When a new item is manufactured, we expect some production costs to decrease as production methods are improved. Obviously there is a learning curve for all involved, along which various costs decrease. If the e ect of this learning diminishes geometrically (or exponentially), then the total cost (over the in nite time-horizon) is of the discounted type. This would also be the case if the cost of obtaining a component decreases at an exponential rate. Such an 209
210
HANDBOOK OF MARKOV DECISION PROCESSES
exponential decrease is evident in the computers industry (and one aspect goes by \Moore's law"): prices of various components decrease at an exponential rate, and since the processing speed increases at an exponential rate, the \unit cost" of processing power decreases exponentially fast. However, the rates (or discount factors) for these di erent mechanisms are clearly unrelated. When combining several such costs (such as processing speed with economic considerations), we are naturally led to deal with several different discount factors, each applicable to a component of our optimization problem. Multiple discount factors also arise in control problems for systems with multiple parallel unreliable components. For details on applications see Feinberg and Shwartz [14, 15, 17]. Similarly, in stochastic games it is natural to consider situations where each player has its own discount factor. In contrast to the discounted criterion, the average cost measures long-term behavior and, usually is insensitive to present and short-term conditions. Naturally, this criterion is appropriate for other applications, such as the long-term performance of systems. As before, if our criteria include system performance (measured through the average cost) as well as rewards (measured through a discounted cost), we are led to problems with mixed criteria. In this paper we review results concerning such criteria, and point out some open questions. We shall mention speci c, as well as general areas of potential applications. The main theoretical questions are: Existence of good (optimal, "-optimal) policies for optimization, multiobjective optimization and in particular constrained optimization problems, Structure of \good policies," and Computational schemes for the calculation of the value and good policies. We emphasize the mixed-discounted problem, where the criteria are all of the discounted type, but with several discount factors, since it possesses a rich structure and, in addition, much is already known. In Section 7.7 we shall review some other related results. We conclude this section with a brief survey of di erent mixed criterion problems. There are several treatments of discounted models where the discounting is more general than the standard one. Hinderer [27] investigates a general model where the discounting is a function of the complete history of the model. In a number of papers, see e.g. Schal [41], Chitashvili [7], Stidham [43], or Haurie and L'Ecuyer [22], the discount rate depends on the current state and action. This type of discounting arises when a discounted semi-Markov Decision process is converted into an MDP; see Puterman [37] or Feinberg [13] for details. Mixed criteria and, in particular, mixed discounted criteria are linear combinations of standard criteria. The rst two papers dealing with mixed criteria were published in 1982. Golabi, Kulkarni, and Way [21] considered a mixture of average reward and total discounted criteria to manage a statewide pavement system, see also [44]. Feinberg [10] proved for various standard criteria, by using a convex-analysis approach, that for any policy there exists a nonrandomized Markov policy with the same or better performance. In the same paper, Feinberg [10], proved that property for mixtures of various criteria.
MIXED CRITERIA
211
The 1990's saw systematic interest in criteria that mix several standard costs, which we now survey brie y. Krass, Filar and Sinha [29] studied a sum of a standard average cost and a standard discounted cost for models with nite state and action sets. They proved that for any " > 0 there exists an "-optimal randomized Markov policy which is stationary from some epoch N -onwards (a so-called ultimately deterministic or (N; 1)-stationary policy). They also provided an algorithm to compute such policies. For the special case of a communicating MDP, the resulting policy is nonrandomized. As explained in Feinberg [11], the use of results from [10] simpli es the proofs in [29] and leads to the direct proof that for any " > 0 there exists an "-optimal (nonrandomized) Markov ultimately deterministic policy. The latter result can be derived from [29] but was formulated there only for the constant gain (communicating) case. Fernandez-Gaucherand, Ghosh and Marcus [18] considered several weighted as well as overtaking cost criteria. They treat general (Borel) state and action spaces with bounded costs, and their objective is a linear combination of several discounted costs and an average cost. They prove that, for any " > 0, there exists N (") and an "-optimal deterministic policy which coincides with the average-optimal policy after time N ("). Ghosh and Marcus [20] consider a similar problem in the context of a continuous time di usion model with positive costs. There is one average component and one discounted component. It is shown that it suÆces to consider Markov (randomized) policies. Under global stability conditions they show the following. There is a minimizer in the class of stationary randomized controls and randomization is not necessary. For the more general minimization problem, there exist "-optimal policies that agree with the optimal policy for the discounted part until a xed time, and then switch to the policy which is optimal for the average part of the cost. Hernandez-Lerma and Romera [25] noticed that the \minimal pair" approach by Lasserre and Hernandez-Lerma [23, 24] can be applied to multiple criteria problems when each criterion is of the form of either average rewards per unit time or total discounted rewards where di erent criteria may have di erent discount factors. The minimal pair approach (see Chapter 12) means that a controller selects a pair (; ) where is an initial distribution and is a policy. In classical problems, the initial distribution is either xed or a controller optimizes with respect to all initial states. Hernandez-Lerma and Romera [25] considered an additional condition which essentially means that is randomized stationary and is an invariant distribution of the Markov chain with the transition probabilities de ned by : Naturally, for a one-step reward function r, in this case expected rewards at step t are equal to t 1 r, where is the discount factor. Therefore, the average rewards per unit time are equal to r and the expected total discounted rewards are equal to r=(1 ): Therefore, the change of the original formulation by using the minimal pair approach and the additional invariant condition on the initial measure reduce the problem with combined criteria (average rewards per unit time and discounted rewards) to a version of a problem with only average rewards per unit time.
212
HANDBOOK OF MARKOV DECISION PROCESSES
Filar and Vrieze [19] considered a stochastic zero-sum game with nite state and action spaces. The objective is a linear combination: either of an average cost and a discounted cost, or of two discounted costs with two di erent discount factors. The immediate costs are the same for all components. They prove that in both cases, the value of the stochastic zero-sum game exists. Moreover, when the objective combines two discounted costs, both players possess optimal Markov policies. When the objective combines a discounted cost with an average cost, there exist "-optimal (behavioral) policies. Most of the papers on mixed criteria deal either with linear combinations of total discounted rewards with di erent discount factors or with a linear combination of total discounted rewards and average rewards per unit time. It appears that linear combinations of discounted rewards are more natural and easier to deal with than the weighted combinations of discounted and average rewards. The latter ones model the situation when there are two goals: a short-term goal modeled by total discounted rewards and a long-term goal modeled by average rewards per unit time. In the case of two di erent discount factors, the weighted discounted criterion models the same situation when one of the discount factors is close to 1. When the state and action sets are nite, optimal policies exist for mixed discounted criteria, they satisfy the Optimality Equation, and can be computed. Optimal policies may not exist for mixtures of total discounted rewards and average rewards per unit time [29]. Another advantage of dealing with mixed discounting is that for this criterion there is a well-developed theory for any nite number of discount factors [14, 15], while the papers that study linear combinations of discounted and average reward criteria usually deal with linear combinations of only two criteria: discounted rewards with a xed discount factor and average rewards per unit time. Single discount problems are often interpreted as total reward problems with a geometric life time: however, as shown in Shwartz [42], mixed discount problems cannot, in general, be interpreted as problems with several time scales. In this paper we concentrate on a mixed-discounted problem for Markov decision processes. The exposition is based on the detailed study of this problem, performed by Feinberg and Shwartz [14, 15, 17]. In Section 7.2 we describe the model more precisely, and show through Example 7.1 that, although the weighted criterion seems like a small variation on a standard discounted problem, it induces quite di erent behavior on the resulting optimal policies. Then, in Section 7.3 we show that mixed-discounted problems can be reduced to standard discounted problems if we expand the state space X to X N , and that Markov policies are suÆcient for one-criterion mixed-discounted problems. Section 7.4 obtains the characterization as well as an algorithm for the computation of optimal policies for the Weighted Discount Optimization problem (WDO) with nite state and action sets. WDO problems are introduced in De nition 7.1. In Section 7.5 we treat multiplecriterion problems and in Section 7.6 we discuss nite constrained problems. In Section 7.7 we survey existing results for other relevant problems, related models of stochastic games, and discuss extensions and open problems.
8
BLACKWELL OPTIMALITY Arie Hordijk Alexander A. Yushkevich
8.1
FINITE MODELS
In this introductory section we consider Blackwell optimality in Controlled Markov Processes (CMPs) with nite state and action spaces; for brevity, we call them nite models. We introduce the basic de nitions, the Laurentexpansion technique, the lexicographical policy improvement, and the Blackwell optimality equation, which were developed at the early stage of the study of sensitive criteria in CMPs. We also mention some extensions and generalizations obtained afterwards for the case of a nite state space. In Chapter 2 the algorithmic approach to Blackwell optimality for nite models is given. We refer to that chapter for computational methods. Especially for the linear programming method, which we do not introduce. 8.1.1 De nition and existence of Blackwell optimal policies
We consider an in nite horizon CMP with a nite state space X, a nite action space A , action sets A (x) = Ax , transition probabilities pxy (a) = p(yjx; a), and reward function r(x; a) (x 2 X; a 2 Ax ; y 2 X ). Let m be the number of states in X. We refer to Chapter 1 for de nitions of various policies, of probability distributions and expectations corresponding to them, and notations. We also use the notation K = f(x; a) : a 2 A (x); x 2 Xg; (1) so that, in particular, P a f (x) =
X
y2X
pxy (a)f (y);
(x; a) 2 K :
(2) 231
232
HANDBOOK OF MARKOV DECISION PROCESSES
For every discount factor 2 (0; 1) the expected total reward v(x; ; ) = v (x; ) :=
"1 X
Ex
t=0
tr
(xt ; at )
#
(3)
converges absolutely and uniformly in the initial state x and policy , so that the value function V (x; ) = V (x) := sup v (x; ); x2X 2
is well de ned and nite. Following Blackwell [3], in this chapter we say that a policy is -optimal if v (x; ) = V (x) for all x 2 X (not to confuse with -optimal policies, for which v () V ; in this chapter we do not use them). In the case of a stationary policy ' 2 s it is convenient to write (3) in matrix notations. In that case we have an m m transition matrix P (') = P ' with entries pxy ('(x)) = p'xy , and (3) can be written in the form v (') = v ' =
1 X t=0
( P ' )t r' = (I P ' ) 1 r'
(4)
where r' is a vector with entries r(x; '(x)), x 2 X (formula (4) makes sense also for complex with j j < 1), (in the notation (2) P ' f (x) = P '(x) f (x)). For every 2 (0; 1) there exists a -optimal policy ' 2 s ; namely, one may set h i X a ' (x) = argmax r(x; a) + P v (x) ; x 2 X: a2Ax
y 2X
In the important case of undiscounted rewards, when = 1, the total expected reward in general diverges, and the simplest performance measure is the average expected reward w(x; ) = w (x) (see Chapter 1). For a stationary policy ' 1 nX1 (P ' )t r' = Q' r' = lim(1 )v' ; w' = nlim (5) !1 n "1 where
Q'
t=0
= Q(') is the stationary (or limiting) matrix
Q' = Nlim !1
N 1 X (P ' )t ; and Q'P ' = P ' Q' = Q' : N + 1 t=0
(6)
The last expression for w' in (5) follows from (3) and the fact that Cesaro summability of a divergent series implies its Abel summability to the same limit. Howard [27] proved the existence of average optimal policies in nite CMPs with the class s of admissible policies, and developed a policy improvement algorithm to nd them, related with his name. Almost at the same time Wagner [46] determined that such policies are average optimal in the class too. However, the average reward criterion is insensitive, under selective since it is entirely determined by the arbitrarily far tail of the rewards; in accordance with
BLACKWELL OPTIMALITY
233
this criterion, two policies providing rewards 100+0+0+ and 0+0+0+ are equally good (or bad). Blackwell [3] in his study of nite CMPs introduced a much more sensitive concept of optimality, that bears now his name, and proved the existence of stationary policies optimal in this new sense. De nition 8.1 A policy is said to be Blackwell optimal, if is -optimal for all values of in an interval 0 < < 1.
A stationary Blackwell optimal policy ' is average optimal. Indeed, there exists a stationary average optimal policy , and by (5) w = lim(1 )v "1
lim (1 )v ' = w' ; "1
so that w' = w . Since the last limit is the same for all Blackwell optimal policies, stationary or not (as follows from De nition 8.1), and since by Theorem 8.1 below there is a stationary Blackwell optimal policy, every Blackwell optimal policy 2 is average optimal. Theorem 8.1 In nite CMP there exists a stationary Blackwell optimal policy. Proof. Since for every positive < 1 there exists a -optimal policy ' 2 s ,
and because the set s of stationary policies is nite together with X and A , there exists a stationary policy ' which is -optimal for all = n where n " 1. We claim that ' is Blackwell optimal. Suppose the contrary. Then, because X and s are nite sets, there are a state x0 , a policy 2 s , and a sequence n " 1 such that v ' (x0 ) < v (x0 ) for = n ; n " 1: On the other hand, by the selection of ' v ' (x0 ) v (x0 ) for = n " 1: It follows that the function f ( ) = v ' (x0 ) v (x0 ) de ned for all complex with j j < 1 takes on the value 0 at an in nite sequence of di erent points zn " 1, and takes on nonzero values at the points
n " 1. By using Cramer's rule to compute the inverse matrix, we nd that each entry of (I P ') 1 is a rational function of , and the same is true with in place of '. Therefore and by (4), f ( ) is a rational function of the complex variable in the circle j j < 1 (and hence on the whole complex plane). A rational function cannot have in nitely many di erent zeros zn if it is not an identical zero. The obtained contradiction proves that ' is Blackwell optimal. The above proof is a purely existence argument, without any indication how to nd a Blackwell optimal policy '. Blackwell's original proof also did
234
HANDBOOK OF MARKOV DECISION PROCESSES
not provide a complete algorithm to obtain ', but it contained some essential elements in this direction. Blackwell used, besides the limiting matrix Q', the deviation matrix D' corresponding to ' 2 s . If the Markov chain with the transition matrix P ' is aperiodic, then D' =
1 X t=0
[(P ' )t Q'];
(7)
and the above series converges geometrically fast; in general 1
N X n X
[(P ' )t Q']: D' = lim N !1 N + 1 n=0 t=0
(70 )
An important property of this matrix is that D' is uniquely determined by the equations D' Q' = Q' D' = 0; (8) ' ' ' ' ' D (I P ) = (I P )D = I Q (9) (see, for instance, Kemeny and Snell [29]). Blackwell derived and utilized the expansion h' 1 ' v ' = (10) 1 + h0 + o(1) as " 1; where h' 1 = Q' r' ; h'0 = D'r' ; (11) and introduced the notion of a nearly optimal policy 2 . For such a policy V v = o(1) as " 1. The existence of a Blackwell optimal policy ' 2 s implies a similar expansion for the value function
h 1 (12) 1 + h0 + o(1) as " 1: It is easy to see using (12), that a policy is average optimal i v = h 1 = + o(1= ), where = 1 , and that is nearly optimal i v = h 1 = + h0 + o(1). V (x) =
8.1.2 Laurent series expansions and n-discount optimality
Average optimal and nearly optimal policies, as well as relations (10)-(12), are at the start of a chain of notions and equations developed by Miller and Veinott [33] and Veinott [42], which lead to a deeper insight into Blackwell optimal policies and to an algorithm to nd them. We present their main ideas in a slightly modi ed form. The approach is based on the Laurent series expansion of the resolvent R = (I P ) 1 = I + P + 2 P 2 + (j j < 1) (13) of a Markov chain with the transition kernel P in the neighborhood of the point = 1. This expansion is a general fact known in functional analysis (see, for
BLACKWELL OPTIMALITY
235
instance, [48]). In the particular case of an aperiodic Markov chain, it follows immediately from the geometric convergence of P t to the limiting matrix Q. Indeed, the di erence 1 2 2 R 1 Q = (I Q) + (P Q) + (P Q) + ;
in which kP n Qk C n for some < 1, is an analytic function of the complex variable in the circle j j < 1= (we use the norm in the space of (m m)matrices generated by the supremum norm in the space of m-vectors). The point = 1 is inside this circle, thus R has the same singularity at the point = 1 as 1 1 Q, i.e. has a single pole. Therefore in some ring 0 < j 1j < 0 a Laurent expansion R R = 1 + R0 + R1 + R2 2 + ; =1 (14) holds. If the Markov chain is periodic, consider the least common multiple d of the periods of all its ergodic classes. The chain with a kernel P d is then aperiodic, so that P nd converges geometrically fast to a stochastic matrix Qe as n ! 1. Similar to the preceding argument, it follows that the in nite sum Re = I + d P + 2d P 2 + is analytic in a circle j jd < 1= of a radius greater than 1, and thus has a simple pole at = 1. Then the same is true for R = (I + P + + d 1 P d 1 )Re : Instead of the Laurent series (14), one may write a similar series in powers of another small parameter equivalent to , which has the meaning of an interest rate: 1 = ; = 1 = 1 : (15) = 1 1+ Veinott [42] and most of the subsequent authors used series in . Chitashvili [6, 7, 56] and following him Yushkevich [49]{[55] used series in . We present both versions. Theorem 8.2 In a nite CMP there exists a number 0 2 (0; 1) such that for every policy ' 2 s v ' = (1 + )
1 X
1 X
0 < < 1
(16)
h' 1 = k' 1 = Q'r' = w' ; h'0 = k0' = D' r' (cf. (10) and (11)), and where for n 1 h'n = ( D' )n h'0 ; kn' = (I D' )n k0' : A similar expansion is valid for the value function 1 1 X X V = (1 + ) hn n = kn n ; 0 < < 1:
(17)
where
n=
n=
1
1
h'n n =
n=
n=
1
1
kn' n ;
(18) (19)
236
HANDBOOK OF MARKOV DECISION PROCESSES
Proof. The existence and convergence of Laurent expansions (16) follow from expansions in powers of or of R ' , respectively R ', and from the formula v ' = R 'r' equivalent to (4). To get the coeÆcients (17)-(18), observe that by (4) v ' = r' + P ' v ' , so that by (15) and (16)
(1 + )
1 X 1
h'n n = r' + P '
1 X 1
h'nn :
By the uniqueness of the coeÆcients of power series, this results in equations (to simplify writing, we temporarily skip the superscript '): h 1 = P h 1; (20) h0 + h 1 = r + P h0 ; (21) hn + hn 1 = P hn (n 1): (22) From (6) and (20) by iteration and taking a limit, we nd h 1 = Qh 1 . For the stationary matrix Q = QP = P Q, and a multiplication of (21) by Q gives Qh 1 = Qr, so that h 1 = Qr as in (17). A multiplication of (22) by Q provides Qhn = 0 (n 0). Using this, the relation h 1 = Qh 1 and (8){(9), we get after a multiplication of (21) by D = D' , that D(I P )h0 + DQh 1 = Dr, or (I Q)h0 = Dr, or nally h0 = Dr as in (17). Multiplying (22) by D, in a similar way we get D(I P )hn + Dhn 1 = 0, or hn Qhn = Dhn 1 , or hn = Dhn 1 (n 1), and this proves that hn = ( D)n h0 as in (18). Formulas (17){(18) for kn' follow absolutely similarly from equations k 1 = P k 1 , k0 + P k 1 = r + P k0 and kn + P kn 1 = P kn instead of (20){(22). Since the set s is nite, we have the expansions (16) simultaneously for all ' 2 s in some interval ( 0 ; 1). Formula (19) follows now from Theorem 8.1. Formulas of Theorem 8.2 are a generalization of (10) and (11). They stimulate a similar generalization of the average optimality and nearly optimality criteria. The following de nition is due to Veinott [43]. De nition 8.2 For n 1, a policy 2 is said to be n-discount optimal, if for every 2 (23) lim n [v ( ) v ()] 0 "1
(with in place of we have an equivalent condition).
By substituting in (23) a Blackwell optimal policy , for which v () = V and v ( ) v () 0, one may see that in nite CMPs condition (23) is equivalent to a simpler (and formally stronger) condition lim n [V v ( )] = 0: (24) "1 However, condition (23) appeared to be more suitable for an extension of sensitive criteria to denumerable and Borelian CMPs. To avoid confusion, mention that in literature 0-discount optimal policies are sometimes called bias-optimal
BLACKWELL OPTIMALITY
237
or 1-optimal; the latter name originates from Veinott [42]. Also, as seen from a comparison of (16) and (19), a stationary policy is Blackwell optimal i it is n-discount optimal for every natural n, or, brie y speaking, is 1-discount optimal. For bias optimality in models with nite state and action spaces see Chapter 3. A convenient description of n-discount optimal policies can be made in terms of sequences of coeÆcients of series (16) and (19) and a lexicographical ordering in spaces of them. De ne H ' = fh' 1 ; h'0 ; : : : g; K ' = fk' 1 ; k0' ; : : : g; (25) let Hn' and Kn' be the initial segments of H ' and K ' up to the n-th term, and let H , K , Hn and Kn have the same meaning for the series (19) (each h'n etc. is an m-vector). For those sequences and segments we introduce a natural lexicographical ordering denoted by symbols , , , . So, H ' H means that H '' 6= H , and that there exists a number N < 1 and a state x0 2 X, such that HN 1 = HN 1 (if N 0), and h'N (x0 ) < hN (x0 ) while h'N (x) hN (x) for all other x 2 X. The relation H ' H means that either H ' = H or H ' H . The relations H H ' and H H ' are equivalent to H ' H and H ' H . With this notation we have H ' H and K ' K for every ' 2 s , and the policy ' is n-discount optimal (or Blackwell optimal) i Hn' = Hn or Kn' = Kn (respectively, if H ' = H or K ' = K ). The following theorem due to Veinott [43] shows that in nite CMPs the n-th discount optimality of a stationary policy for large values of n coincides with its Blackwell optimality. Let n be the subset of s consisting of all stationary n-discount optimal policies (n 1), and let 1 be the set of all Blackwell optimal policies in s . Evidently, \ 1 0 1 ; 1 = n : Theorem 8.3 In nite CMPs with m 2 states
n
m 1 = m = : : : = 1 : Proof. It is suÆcient to show that m 1 = 1 . Consider any policy ' 2 m 1 . We have Hm' 1 = Hm 1 , or in more detail h'n = hn ; n = 1; 0; 1; : : : ; m 1: (26) Since m 2, both h0 and h1 are present in (26). We claim that m column m-vectors h0 ; h1 ; : : : ; hm 1 are linearly dependent. It is suÆcient to show that m row vectors of the corresponding square matrix are linearly dependent; these rows are fh0(x); : : : ; hm 1 (x)g = fh'0 (x); : : : ; h'm 1 (x)g, x 2 X. In fact even the in nite sequences fh'0 (x); h'1 (x); : : : ; h't(x); : : : g; x 2 X (27) are linearly dependent. Indeed, in the nite Markov chain generated by P ' there exists a stationary distribution f(x); x 2 Xg. The total discounted
238
HANDBOOK OF MARKOV DECISION PROCESSES
expected reward corresponding to the initial distribution and policy ' is equal to v ' () :=
X x2X
(x)v ' (x) =
=
1 X t=0
X t x2X
X x 2X
(x) E 'x
1 X t=0
t r(xt ; '(xt )) =
(x) E 'x r(xt ; '(xt )) =
1 X t=0
t E ' r(xt ; '(xt )): (28)
Here the P'-distribution of xt does not depend on t because is a stationary distribution, and hence the factor at t in (28) is some constant C . Thus v '
() = C
1 X t=0
t
"
1 X = 1 = C 1 + = (1 + ) C + 0 n n=0
C
#
(29)
(cf. (15)). On the other hand, by (28) and (16), v ' () = (1 + )
1 X n=
1
n
X x2X
(x)h'n (x):
A comparison P with (29) together with the uniqueness of the Laurent coeÆcients show that x (x)h'n (x) = 0 for all n 0, so that the sequences (27) are linearly dependent. Now, by (18) h'n+1 = D' h'n; hn+1 = D hn (n = 0; 1; 2; : : : ) (30) where is a Blackwell optimal policy. Let t be the maximal integer such that the vectors h0 = h'0 ; : : : ; ht = h't in (26) are linearly independent; such t 0 exists 'if only h0 6= 0, and as just proved, t < m 1. If h0 = 0, then by (26) also h0 = 0, and by (30) h'n = 0 = hn for all h' 0, so that H ' = H and ' 2 1 . If there is the required t, then ht+1 = ht+1 is a linear combination of h0 = h'0 ; : : : ; ht = h't : ht+1 =
t X i=0
Ci hi ;
h't+1 =
t X i=0
Ci h'i:
(31)
Due to (30) multiplying the rst identity by D and'the second by D' , we only increase every subscript in (31) by 1, and since hi = hi for 0 i t + 1, we get h't+2 = ht+2 . Repeating this, by induction we get h'n = hn for every n 0, so that ' 2 1 . The sets m 2 and m 1 are in general di erent. The following example, taken from [43], con rms this statement. To make it more visual, we present it for m = 5. Example 8.1 There are m = 5 states 1; 2; : : : ; 5 with mandatory transitions 2 ! 3 ! 4 ! 5, the state 5 is absorbing. In state 1 there is a choice between
BLACKWELL OPTIMALITY
239
1
1
1
2
2
-6
3
6
p11 ( )= 12 p12 ( p12 (')=1
4
-2
)= 21
5
0
Figure 8.1
two actions, which determine two stationary policies ' and (see Figure 8.1). Under ' we have a mandatory transition 1 ! 2, under transitions 1 ! 1 and 1 ! 2 are equally likely. The numbers under arrows indicate the rewards r(x; a). Mention that the rewards 2; 6; 6; 2 are the binomial coeÆcients of (A B )m 2 = (A B )3 multiplied by 2. The expected rewards v ' and v di er only at the initial state 1. For ' we have v ' (1) = 2 6 + 6 2 2 3 = 2(1 )3 = 2 3 :
= r + P v (cf. (4)) we have the equation 1 1 v = 1 + [ v (1) + v (2)]: 2 2
For , by the formula v
Thus where Hence
(2 )v (1) = 2 + v (2); v (2) = ( 6 + 6 2 2 ) = v ' (1) 2:
v ' (1) 2 3 = 2 3 2 4 + 2 5 : = 2 1+ This means that V (1) = 2 3 , that ' is Blackwell optimal, and that discount optimal, but not 4-discount optimal. Thus 3 6= 4 = 1 . v (1) =
is 3-
8.1.3 Lexicographical policy improvement and Blackwell optimality equation
Policy improvement is both a practical method to approach an optimal policy in CMPs and an important tool in their theory. Its essence is that if ' and are two stationary policies, if = '1 is a Markov policy coinciding with at the rst step of the control and coinciding with ' afterwards, and if is better than ', then is also better than '. This method, almost trivial for the discounted reward criterion with a xed < 1, was developed by Howard [27] for the average reward criterion. Howard used, besides the average reward w' (= h' 1 ), a second function, in fact equal to the term h'0 in the expansions (10) and (16) up to a constant term on each recurrence class of the Markov chain generated by '. Blackwell [3] provided a rigorous proof that a slightly
240
HANDBOOK OF MARKOV DECISION PROCESSES
di erent version of Howard's policy improvement method does converge. Miller and Veinott [33] have extended policy improvement to the case of Blackwell optimality, and Veinott [43] re ned it using the classes n . We expose this topic in a modernized form, using an operator approach developed in Dekker and Hordijk [8] in the framework of CMPs with a countable state space X. To avoid additional formulas, we do all calculations in terms of ; in terms of formulas are slightly di erent. From the structure of and (13) we have 1 1 P v' = r + P X h'nn ; v = r + P v ' = r + 1+ n= 1 while
Subtracting, we get
1 X v ' = h' 1 1 + (h'n + h'n 1 )n : n=0
1 X v v ' = (P h 1 h 1 ) 1 +(r +P h0 h0 h 1 )+ (P hn hn hn 1 )n n=1
(32) where it is understood that hn = h'n. By (18), the supremum norm kh'nk is growing no more than geometrically fast with n. It is convenient to introduce the space H of all sequences H = fhn ; n 1g of m-vectors satisfying this growth condition, and to treat the sequences of Laurent coeÆcients of the series (16), (32) etc. as elements of H. In particular H ' 2 H (see (25)), and in H we consider the same lexicographical ordering as we have introduced in connection with H '. Also, it is convenient to de ne the spaces Hn of nite collections Hn = fht; 1 t ng of m-vectors. The right side of (32) de nes an operator L in the spaces H and Hn . Since the matrix P has entries pxy (a) with a = (x), we express L through the corresponding operators La transforming functions (vectors) on X into functions of pairs (x; a) on the state-action space K de ned in (1). We have (L H )(x) = L (x)H (x); x 2 X; (33) LaH (x) = f`ha 1(x); `ha0 (x); `ha1 (x); : : : g; (x; a) 2 K ; (34) where according to (32) `ha 1 (x) = P a h 1(x) h 1 (x); `ha0 (x) = r(x; a) + P a h0 (x) h0 (x) h 1 (x); (35) a a `hn (x) = P hn (x) hn (x) hn 1 (x) (n 1): The same formulas de ne La and L , as operators on Hn . Lemma 8.1 Let '; 2 s . If (L H ' )n+1 0 for some n 1, then Hn Hn' . Moreover, if in addition (L H ' )n+1 (x0 ) 0 at some x0 2 X, then Hn (x0 ) Hn'(x0 ). The same is true with the reverse inequality signs. In particular, if L H ' = 0, then H = H '.
BLACKWELL OPTIMALITY
241
Proof. The condition (L H ' )n 0 means that v = v ' + Qn () + O(n+1 )
(36) where Qn () is a vector consisting of polynomials of degree n with lexicographically nonnegative coeÆcients, and where O(n+1 ) is uniform in x 2 X since X and A are nite sets (compare (32) with (33)-(35)). Consider policies t = t '1 , and let v(t) = v (t ), so that, in particular, v(0) = v ' , v(1) = v . We have v(t + 1) = r + P v(t); t = 0; 1; 2; : : : (37) and by (36) v(1) = v(0) + Qn + R; (38) n +1 where the remainder R is of order . From (37) and (38) by induction we get v(t) = v(0) + (I + P + ( 2 P )2 + + ( P )t 1 )(Qn + R); t 1: (we use that r + P v(0) = v(0) + Qn + R according to (37) and (38)). Since < 1, in the limit v(t) becomes v(1) = v , so that v = v ' +
1 X t=0
( P )t (Qn () + R) = v ' + R (Qn + R):
Here Qn 0 for small > 0, R is of order O(n+1 ), and the resolvent R is of order O(1 + + 2 + ) = O( 1 ). This proves that Hn 1 Hn' 1 if L Hn' 0. Other assertions are proved in a similar way. To proceed further, we need the lexicographical Bellman operator L in the spaces H and Hn : LH (x) = max La H (x); H 2 H; x 2 X; (39) a2A x where the maximum is understood in the lexicographical sense ; the same formula holds for Hn 2 Hn . This maximum always exists because the sets A x are nite. Since one may use all combinations of actions in stationary policies, formula LH = maxs L H (40) 2
de nes the same operator L. If in (32) = ' then = '1 coincides with ', and the left side of (32) is zero. Hence all the coeÆcients at the right side vanish, and this means that L'H ' = 0 for every ' 2 s . Therefore LH ' 0 for every ' 2 s . If LH ' = 0, we say that ' is unimprovable; if LHn' = 0, then ' is unimprovable of order n. The equation LH = 0 H 2H (41) is called the Blackwell optimality equation in honor of Blackwell; the similar equation LHn = 0 for Hn 2 Hn is the n-order optimality equation. Let H =
242
HANDBOOK OF MARKOV DECISION PROCESSES
fhng be the element of H corresponding to the value function V (see (19)), and let Hn be the initial segments of H . We say that a stationary policy ' is conserving (or n-order conserving) if L'H = 0 (respectively, L'Hn = 0). Theorem 8.4 A. The Blackwell optimality equation has a unique solution H = max'2s H ' . A policy ' 2 s is Blackwell optimal i H ' = H , and i ' is a conserving policy. B. For every n 1, Hn is uniquely determined by the equation LHn+1 = 0. A policy ' 2 s is n-discount optimal i Hn' = Hn , and is n-discount optimal if ' is (n + 1)-order conserving. Proof. By Theorem 8.1 there exists a Blackwell optimal policy ' 2 s . Evidently, H ' H , 2 s and ' is unimprovable, so that H ' = H := max 2s H , and LH = LH ' = 0. Since L' H ' = 0, also L' H = 0, and '
is conserving. In part A it remains to prove that the solution of (41) is unique, and that a conserving stationary policy is Blackwell optimal. If is conserving, then L H = 0, hence L H ' = 0 for a Blackwell optimal ', therefore by Lemma 8.1 (applied to every n) H = H ' = H , so that is Blackwell optimal too. Finally, suppose that He is a solution to (41). By taking for each x 2 X a lexicographical maximizer a 2 A x of La He (x), we obtain a stationary policy for which L He = He . One may check (we omit the proof) that Lemma 8.1 is true for any H 2 H in place of H ', in particular, for He . It follows that H = He , and since LH = LHe = 0, the policy is unimprovable. Hence is Blackwell optimal, so that He = H = H . The proof of part B is similar, with a reference to Lemma 8.1. Policy improvement is a basis for an algorithm to compute a Blackwell optimal policy in a nite CMP. Theoretically, one may proceed in the following way. Start with some ' 2 s and compute Hm' using formulas of Theorem 8.2 (here m is the number of states in X). Check the values of `a 1 h(x); (x; a) 2 K . For a = '(x) those values are zeros, and if `a h 1 (x ) > 0 for some pair (x ; a ), then the policy ( if x = x , (x) = a '(x) otherwise improves '. If there are no such pairs (x ; a ), repeat the same procedure with `a h0 and the shrinked sets A 0 (x) = fa 2 A 1 (x), `a h 1 (x) = 0g, K 0 = f(x; a) : a 2 A 0 (x); x 2 Xg (where A 1 (x) = A (x)). A policy as above with `a h0 (x ) > 0, (x ; a ) 2 K improves '. If there are no such pairs (x ; a ), repeat the procedure with all subscripts increased by 1, etc., until either you get a better policy , or reach the set K m . In the latter case ' is (m 1)-order discount optimal, and therefore Blackwell optimal by Theorem 8.3. Otherwise, proceed in the same way with the obtained policy . Since the set s is nite, this algorithm leads to a Blackwell optimal policy in a nite number of steps. In practice, one may improve ' simultaneously at several states x ; see Policy Iteration in chapter 10 of [34]. On the other hand, the lexicographical policy improvement approach opens a new way to prove the existence of Blackwell optimal policies via a maximiza-
BLACKWELL OPTIMALITY
243
tion of H ' over all stationary policies ' and the related Blackwell optimality equation(41). The latter idea can be used in CMPs with an in nite state space X, in which the proof of Theorem 8.1, based on the fact that the set s is nite, is inapplicable. 8.1.4 Extensions and generalizations
In [45] Veinott simpli ed and updated results of [42, 43]. In particular, he re ned Theorem 8.3 as follows: 1 = m r where r is the number of recurrent classes in X under a Blackwell optimal stationary policy. Veinott [43] introduced also the notion of n-average optimality in addition to the n-discount optimality. Let v(1) (x; ) = T
Ex
"T 1 X t=0
and de ne recursively for n 1 vT(n+1) (x; ) =
#
r(xt ; at )
T X vt(n) (x; ): t=1
Then is n-average optimal if for every policy h i lim 1 v(n+2) ( ) v(n+2) () 0: T !1
T
T
T
Veinott [43, 44] and Sladky [38] showed that in a nite CMP a policy is ndiscount optimal i it is n-average optimal. Chitashvili [6, 7, 56] extended results of Theorem 8.4 to more general models with a nite state space. In [6] he treated CMPs with arbitrary (indeed, compacti ed) action sets. He considered also what can be called (n; )-discount optimal policies; in their de nition one should replace 0 by in formula (23). In [7] he studied n-discount optimality in nite models with discount factors depending on the state x and action a: (x; a) = c1 + c2 2 + + ck k where k and ci are functions of (x; a). In this case the reward functions were of some speci c average form. In [56] Theorem 8.4 is generalized to a nite model with two reward functions r(x; a) and c(x; a). More precisely, v '
"1 X '
(x) = E x
t=0
#
t (r(xt ; at ) + (1 )c(xt ; at ))
(in [56] Chitashvili considered only stationary policies). This expected discounted reward corresponds to an undiscounted reward 1 X 1 TX1 c(x ; a ): r(x ; a ) + lim t=0
t
t
T !1
T t=0
t
t
In that case all formulas related to Theorem 8.4 remain valid, with one exception: in equations (35) de ning La , the term `h1 should be changed to `ha1 (x) = c(x; a) + P a h1 (x) h1 (x) h0 (x)
244
HANDBOOK OF MARKOV DECISION PROCESSES
(similar to the term `h0 ). As explained in the proof of Theorem 8.1, in nite CMPs the expected discounted reward v ' (x) is a rational function of . Hordijk e.a. [19] introduced a non-Archimedian ordered eld of rational functions, used a simplex method in this eld, and developed a linear programming method for the computation of -optimal policies over the entire range (0,1) of the discount factor. In particular, their method allows to compute a Blackwell optimal policy. More precisely, for some m one may nd numbers 0 := 0 < 1 < < m 1 < m := 1 and stationary policies '1 ; '2 ; : : : ; 'm such that 'j is -optimal for all 2 [ j 1 ; j ], 1 j m 1, and 'm is -optimal in the interval [ m 1 ; m ) (which means that 'm is Blackwell optimal). In section 5.4 of Chapter 2 this algorithm is studied. In Chapter 4 an asymptotic simplex method based on Laurent series expansions for the computation of a Blackwell optimal policy, is used. In CMPs with constraints the controller wants to maximize expected (discounted) rewards while keeping other expected (discounted) costs in some given bounds. For such CMPs Altman e.a. [1] gave a constructive proof for the following (weaker) version of the result obtained in [19]. There exist numbers m and j as above such that for every j = 1; : : : ; m either the constrained problem is not feasible in the open interval ( i 1 j ) or the value function is a rational function of in the closed interval [ j 1 ; j ], j m 1 and [ m 1 ; 1). Consequently, if the constrained problem is feasible in the neighborhood of = 1, then v has a Laurent series expansion at = 1. As shown in the proof of Theorem 8.1, the limits of -optimal policies, for tending to 1, are Blackwell optimal. A counterexample in Hordijk and Spieksma [22] shows that in general this is not true in unichain CMPs with a nite state space and compact action sets. This disproves a conjecture in Cavazos-Cadena and Lasserre [4]. More recently, Huang and Veinott [28] extended many results concerning Blackwell and n-discount optimality in nite models to the case when (i) the reward r(x; a; ) at the state x under the action a depends also on the interest rate , namely is an analytic function of in a neighborhood of = 0, (ii) the nonnegative transition coeÆcients pxy (a) in general do not sum to 1, only for every Markov policy the t-step transition matrices are of order O(1) as t ! 1. They proved the existence of stationary Blackwell optimal policies, extended to their case the lexicographical policy improvement, showed that if r(x; a; ) is a polynomial or a rational function of then Blackwell optimality of a stationary policy is equivalent to its n-discount optimality for some n depending on the degrees of involved polynomials. In the latter case the policy iteration algorithm provides a Blackwell optimal policy in a nite number of steps. Another constructive rule is given for nding n-discount optimal policies using a linear programming approach. 8.2
DENUMERABLE STATE MODELS
In this section we consider CMPs for which the state space X is denumerable. There are many applications of controlled Markov chains for which it is natural to take an in nite number of states. An important class of models is that of
9
THE POISSON EQUATION FOR COUNTABLE MARKOV CHAINS: PROBABILISTIC METHODS AND INTERPRETATIONS Armand M. Makowski Adam Shwartz
Abstract:
This paper considers the Poisson equation associated with time-
homogeneous Markov chains on a countable state space. The discussion emphasizes probabilistic arguments and focuses on three separate issues, namely (i) the existence and uniqueness of solutions to the Poisson equation, (ii) growth estimates and bounds on these solutions and (iii) their parametric dependence. Answers to these questions are obtained under a variety of recurrence conditions. Motivating applications can be found in the theory of Markov decision processes in both its adaptive and non-adaptive formulations, and in the theory of Stochastic Approximations. The results complement available results from Potential Theory for Markov chains, and are therefore of independent interest.
9.1
INTRODUCTION
Let P (pxy ) be the one-step transition matrix for a time-homogeneous Markov chain fXt ; t = 0; 1; : : : g taking values in some countable space X. This paper is devoted to the corresponding Poisson equation with forcing function r : X ! IR, namely ( ) + w = r(x) +
h x
X
p h(y ); y xy
x
2X
(9.1) 269
270
HANDBOOK OF MARKOV DECISION PROCESSES
for scalar w and mapping h : X ! IR. This equation arises naturally in a variety of problems associated with Markov chains as the following examples indicate. 1. As shown in Section 9.3, solving the Poisson equation provides a means to evaluate the long-run average cost w associated with the cost function r [36]: If (9.1) has a solution (h; w) and some mild growth conditions are satis ed, then Lemma 9.2 states that # " t X 1 r(Xs ) (9.2) w = limt IE t+1 s=0 where is the initial distribution and IE is the corresponding expectation operator. The function h measures the sensitivity of the cost to the initial state, and represents a second-order e ect captured through the \deviation matrix" [13]. This function h can also serve as a \Lyapunov function" in establishing ergodicity [26], and plays a key role in proving the convergence of the policy improvement algorithm [26]. Approximate solutions can be used for simulations|see Chapter 10. 2. During the last decade there has been widespread interest in stochastic approximation algorithms as a means to solve increasingly complex engineering problems [1, 5, 16, 17]. As a result, focus has shifted from the original RobbinsMonro algorithm to (projected) stochastic approximations driven by Markovian \noise" or \state" processes. Properties of solutions to an appropriate Poisson equation play an essential role when establishing the a.s. convergence of such adaptive algorithms [1, 18, 22, 24, 25, 39]. 3. In the context of Markov decision processes (MDPs), the need for adaptive policies can arise in response to both modeling uncertainties and computational limitations [40]. Several adaptive policies have been proposed as \implementations" to a Markov stationary policy, and shown to yield the same cost performance [3, 18, 19, 23, 40]. Here too, the analysis requires precise information on the solution to the Poisson equation associated with the non-adaptive policy [40]. In many of these applications, it is natural to view the forcing function r and the transition matrix P as parameterized, say by some parameter (which may be loosely interpreted as a control variable). The requisite analysis then typically exploits smoothness properties (in ) of the solution h together with various growth estimates (in x) for h. In addition, estimates on the moments of fh(Xt ); t = 0; 1; 2; : : : g are required, with the added diÆculty that the resulting process fXt ; t = 0; 1; 2; : : : g is not necessarily Markovian (say, under the given stochastic approximation scheme or adaptive policy). Our main objective is to develop methods for addressing the concerns above in a systematic fashion. Whenever possible, we emphasize a probabilistic viewpoint as we focus mostly on the following three issues: 1. Existence and uniqueness of solutions to the Poisson equation (9.1); 2. Growth estimates and bounds on these solutions; and 3. Conditions for smoothness in the parameter of these solutions when dealing with the parametric case, as would arise when establishing the a.s.
THE POISSON EQUATION FOR COUNTABLE MARKOV CHAINS
271
convergence of stochastic approximations and the self-tuning property of adaptive policies. Answers to these questions are given under a variety of recurrence conditions. As we try to keep the exposition relatively self-contained, we have included some standard material on the Poisson equation. In addition to its tutorial merit, the discussion given here provides a uni ed treatment to many of the issues associated with the Poisson equation, e.g. existence, uniqueness and representation of solutions. This is achieved by manipulating a single martingale naturally induced by the Poisson equation. Questions of existence and uniqueness of solutions to (9.1) have obvious and natural points of contact with the Potential Theory for Markov chains [15, 29]. Unfortunately many situations of interest in applications, say in the context of MDPs, are not readily covered by classical Potential Theory. Indeed, the classical theory treats the purely transient and recurrent cases separately, with drastically di erent results for each situation. This approach is thus of limited use in the above-mentioned situations, where the recurrence structure of the Markov chain is typically far more complex in that it combines both transient and recurrent states. Here, in contrast with the analytical approach of classical Potential Theory, emphasis has been put on giving an explicit representation of the solution to (9.1) with a clear probabilistic interpretation. This probabilistic approach allows for a relatively elementary treatment of questions of existence and uniqueness, under a rather general recurrence structure. We accomplish this by focusing on the discrete space case, and by keeping the assumptions as transparent as possible. The intuition developed here applies to the general state-space case, under mild conditions on the existence of petite sets|see Chapter 10 and [10, 26, 27]. Results are obtained in various degrees of completeness for both nite and countably in nite state spaces; recurrence structures include multiple positive recurrent classes, and transient classes. A representation for h is derived in detail in the case of a single positive recurrent class under integrability conditions involving the forcing function r. The derivation uses elementary methods, and provides intuition into more general situations. This representation is shown to also hold in countable case with multiple classes, and readily lends itself to establishing natural bounds on the growth rate of h (as a function of the state), and to investigating smoothness properties in the parameterized problem. Similar results are given in [10] for the ergodic case on general state spaces. In addition, when the forcing function r is positive and \increasing" (i.e. when its sub-level sets are compact), there is an elegant theory that relates geometric ergodicity to the Poisson equation; details and references can be found in Chapter 10. As evidenced by the references section, there is a very large literature on the Poisson equation; of particular note is the monograph by Nummelin [28]. In the context of MDPs, bounded solutions are discussed by Ross [34, 35], Gubenko and Shtatland [11] and Yushkevich [42]. One of the rst treatments of unbounded solutions is available in Robinson [31, 32] (with details in Chapter 5). The paper is organized as follows: The set-up is given in Section 9.2 together with the basic martingale associated with (9.1). Various uniqueness results on
272
HANDBOOK OF MARKOV DECISION PROCESSES
the the solution (w; h) are discussed in Section 9.3. We give two decomposition results in Section 9.4; the rst is based on the decomposition of the state space X into its recurrent and transient classes, while the second is an analog of the standard Green decomposition and relies on an expansion of the forcing function in terms of more \elementary" functions. To set the stage for the countably in nite case, we brie y recall an algebraic treatment of the nitestate case in Section 9.5. In Section 9.6, under a single positive recurrent class assumption, an explicit representation for the solution is developed in terms of the recurrence time to some distinguished state. An example is developed in Section 9.7 to illustrate the material of previous sections. Bounds and extensions to unbounded forcing functions and multiple recurrent classes are given in Section 9.8. Equipped with this probabilistic representation of solutions, we can now investigate the smoothness properties of solutions to the parameterized problem; methods for proving continuity and Lipschitz continuity are developed in Sections 9.9 and 9.10, respectively. To close, we note that most of the ideas which are discussed here in the context of countable Markov chains have extensions to fairly general state spaces. This is achieved by means of the so-called splitting technique [10, 26, 27, 28] which in essence guarantees the existence of an atom on an enlarged state space; details can be found in Chapter 10. 9.2
THE POISSON EQUATION AND ITS ASSOCIATED MARTINGALE
First, a few words on the notation used throughout the paper: The set of all real numbers is denoted by IR and 1[A] stands for the indicator function of a set A. Unless otherwise stated, limt , limt and limt are taken with t going to in nity. Moreover, the in mum over an empty set is taken to be 1 by convention. The Kronecker mapping Æ : X X ! IR is de ned and P by Æ (x; y ) = 1 if x = y , P Æ (x; y ) = 0 otherwise. Finally, the notation x2X is often abbreviated as x . 9.2.1
The set-up
The notion of a Markov chain we adopt in this paper is more general than the elementary one used in most applications. We do so with the view of broadening the applicability of the material developed here, especially to problems of adaptive control for Markov chains [18, 19, 22, 23, 39, 40]. The state space is a countable, and we assume the existence of a measurable space ( ; F ) large enough to carry all the probabilistic elements considered in this paper. In particular, let fFt ; t = 0; 1; : : : g denote a ltration of F , i.e. a monotone increasing sequence of - elds contained in F such that Ft Ft+1 for all t = 0; 1; : : : , and let fXt; t = 0; 1; : : : g be a sequence of X-valued rvs which are Ft -adapted, i.e. the rv Xt is Ft -measurable for all t = 0; 1; : : : . The Markovian structure of interest is de ned by postulating the existence of a family fIPx ; x 2 Xg of probability measures on F such that for all x and
10
STABILITY, PERFORMANCE
EVALUATION, AND OPTIMIZATION Sean P. Meyn
The theme of this chapter is stability and performance approximation for MDPs on an in nite state space. The main results are centered around stochastic Lyapunov functions for verifying stability and bounding performance. An operator-theoretic framework is used to reduce the analytic arguments to the level of the nite state-space case. Abstract:
10.1
10.1.1
INTRODUCTION
Models on a general state space
This chapter focuses on stability of Markov chain models. Our main interest is the various relationships between stability; the existence of Lyapunov functions; performance evaluation; and existence of optimal policies for controlled Markov chains. We also consider two classes of algorithms for constructing policies, the policy and value iteration algorithms, since they provide excellent examples of the application of Lyapunov function techniques for -irreducible Markov chains on an uncountable state space. Considering the importance of these topics, it is not surprising that considerable research has been done in each of these directions. In this chapter we do not attempt a survey of all existing literature, or present the most comprehensive results. In particular, only the average cost optimality criterion is treated, and the assumptions we impose imply that the average cost is independent of the starting point of the process. By restricting attention in this way we hope that we can make the methodology more transparent. One sees in several chapters in this volume that the generalization from nite state spaces to countable state spaces can lead to considerable technicalities. In particular, invariant distributions may not exist, and the cost functions 305
306
HANDBOOK OF MARKOV DECISION PROCESSES
of interest may not take on nite values. It would be reasonable to assume that the move from countable state spaces, to MDPs on a general state space should be at least as diÆcult. This assumption is probably valid if one desires a completely general theory. However, the MDPs that we typically come across in practice exhibit structure which simpli es analysis, sometimes bringing us to the level of diÆculty found in the countable, or even the nite state space case. For example, all of the speci c models to be considered in this chapter, and most in this volume, have some degree of spatial homogeneity. The processes found in most applications will also exhibit some level of continuity in the sense that from similar starting points, and similar control sequences, two realizations of the model will have similar statistical descriptions. We do not require strong continuity conditions such as the strong Feller property, although this assumption is sometimes useful to establish existence and uniqueness of solutions to the various static optimization problems that arise in the analysis of controlled Markov chains. An assumption of -irreducibility, to be described and developed below, allows one to lift much of the stability theory in the discrete state space setting to models on a completely general, non-countable state space. This is an exceptionally mild assumption on the model and, without this assumption, the theory of MDPs on a general state space is currently extremely weak. 10.1.2
An operator-theoretic framework
When x is -irreducible it is possible to enlarge the state space to construct an atom 2 X which is reachable from any initial condition (i.e. IPx f < 1g > 0, x 2 X). When the atom is recurrent, that is, IP f < 1g = 1, then an invariant measure (see (10.9)) is given by
fY g = IE
hX 1 i
t=1
Y (xt )
;
Y
2 F;
(10.1)
where is the rst return time to (see (10.6)), and 1Y is the indicator function of the set Y . This construction, and related results may be found in [45, 39]. In words, the quantity fY g expresses the mean number of times that the chain visits the set Y before returning to . This expression assumes that is almost surely nite. If the mean return time IE [ ] is nite then in fact the measure is nite, and it can then be normalized to give an invariant probability measure. Finiteness of the mean return time to some desirable state is the standard stability condition used for Markov chains, and for MDPs in which one is interested in the average cost optimality criterion. Unfortunately, the split chain construction is cumbersome when developing a theory for controlled Markov chains. The sample path interpretation given in (10.1) is appealing, but it will be more convenient to work within an operatortheoretic framework, following [45]. To motivate this, suppose rst that we remain in the previous setting with an uncontrolled Markov chain, and suppose that do have an state 2 X satisfying IP f < 1g = 1. Denote by s the function which is equal to one at , and zero elsewhere: That is, s = 1 . We let denote the probability measure on X given by (Y ) = p(Y j ), Y 2 F ,
STABILITY, PERFORMANCE EVALUATION, AND OPTIMIZATION
307
and de ne the `outer product' of s and by
s (x; Y ) := s(x) (Y ): For example, in the nite state space case the measure can be interpreted as a row vector, the function s as a column vector, and s is the standard (outer) product of these two vectors. Hence s is an N N matrix, where N is the number of states. In general, the kernel s may be viewed as a rank-one operator which maps L1 to itself, where L1 is the set of bounded, measurable functions on X. Several other bounded linear operators on L1 will be developed in this chapter. The most basic are the n-step transition kernels, de ned for n 1 by
P n f (x) :=
Z
X
f (y )pn (dy jx);
f
2 L1 ;
where pn ( j ) is the n-step state transition function for the chain. We set P = P 1 . We can then write, in operator-theoretic notation, IP f
n; xn 2 Y g = (P
s )n 1 1Y ;
n 1;
and hence the invariant measure given in (10.1) is expressed in this notation as
(Y ) =
X1
n=1
(P
s )n 1 1Y ;
Y
2 F:
(10.2)
It is this algebraic description of that will be generalized and exploited in this chapter. How can we mimic this algebraic structure without constructing an atom ? First, we require a function s : X ! IR+ and a probability measure on F satisfying the minorization condition, p(Y j x) s(x) (Y );
x 2 X; Y
2 F:
In operator theoretic notation this is written P s , and in the countable state space case this means that the transition matrix P dominates an outer product of two vectors with non-negative entries. Unfortunately, this `one step' minorization assumption excludes a large class of models, even the simple linear models to be considered as examples below. One can however move to the resolvent kernel de ned by
K = (1
)
X1 t=0
tP t;
(10.3)
where 2]0; 1[ is some xed constant. For a -irreducible chain the required minorization always holds for the resolvent K [39, Theorem 5.2.3]. The move to the resolvent is useful since almost any object of interest can be mapped between the resolvent chain, and the original Markov chain. In particular, the
308
HANDBOOK OF MARKOV DECISION PROCESSES
invariant measures for P and K coincide (see [39, Theorem 10.4.3], or consider the resolvent equation in (10.11) below). Much of the analysis then will involve the potential kernels, de ned via
G :=
X1 X1=0
K t:
(10.4)
t
H :=
t=0
(K
s )t :
(10.5)
In Theorem 10.1 below we demonstrate invariance of the - nite measure de ned by,
(Y ) =
Z
X
(dx)H (x; Y );
Y
2 F;
provided the chain is recurrent. The invariant measure can be written in the compact form = H . The measure will be nite, rather than just - nite, provided appropriate stability conditions are satis ed. The most natural stability assumption is equivalent to the existence of a Lyapunov function, whose form is very similar to the Poisson equation found in the average cost optimality equation. The development of these connections is one of the main themes of this chapter. 10.1.3
Overview
We conclude with an outline of the topics to follow. In the next section we review a bit of the general theory of -irreducible chains, and develop some stochastic Lyapunov theory for such chains following [39, Chapters 11-14]. Following this, in Section 10.3 we develop in some detail the computation of the average cost through the Poisson equation, and the construction of bounds on the average cost. All of these results are developed for time homogeneous chains without control. In Section 10.4 this stability theory is applied to the analysis of the average cost optimality equation (ACOE). We explore the consequences of this equation, and derive criteria for the existence of a solution. Section 10.5 concerns two recursive algorithms for generating solutions to the ACOE: value iteration and policy iteration. It is shown that (i) either algorithm generates stabilizing stationary policies; (ii) for any of these policies, the algorithms generate uniform bounds on steady state performance. However, such results hold only if the algorithms are properly initialized. Convergence is established for the policy iteration algorithm: Under suitable conditions, and when properly initialized, the algorithm converges to a solution of the ACOE. Section 10.6 illustrates the theory with a detailed application to linear models, and to network scheduling. This chapter is concluded with a discussion of some extensions and open problems.
11
CONVEX ANALYTIC METHODS IN MARKOV DECISION PROCESSES Vivek S. Borkar
Abstract:
This article describes the convex analytic approach to classical
Markov decision processes wherein they are cast as a static convex programming problem in the space of measures. Applications to multiobjective problems are described.
11.1 11.1.1
INTRODUCTION Background
Markov decision processes optimize phenomena evolving with time and thus are intrinsically dynamic optimization problems. Nevertheless, they can be cast as abstract `static' optimization problems over a closed convex set of measures. They then become convex programming (in fact, in nite dimensional linear programming) problems for which the enormous machinery of the latter elds can be invoked and used to advantage. Logically, these are extensions of the linear programming approach to nite state nite action space problems due to Manne [43]. (Further references are given in the `bibliographical note' at the end.) The attraction of this approach lies in the following: (i) It leads to elegant alternative derivations of known results, sometimes under weaker hypotheses, from a novel perspective. (ii) It brings to the fore the possibility of using convex/linear programming techniques for computing near-optimal strategies. (iii) It allows one to handle certain unconventional problems (such as control under additional constraints on secondary `costs') where traditional dynamic programming paradigm turns out to be infeasible or awkward. 347
348
HANDBOOK OF MARKOV DECISION PROCESSES
While the convex analytic formulation is available for a variety of cost criteria and fairly general state spaces, our primary focus will be on the `pathwise ergodic control' problem on a countable state space, for which the theory is the most elegant. This is done in the next section. Section 3 sketches extensions to other cost criteria and general state spaces, and the dual problem. Section 4 considers multiobjective problems, such as the problem of control under constraints. An `Appendix' discusses stability of controlled Markov chains. 11.1.2
Notation
Initially we shall work with a denumerable state space, identi ed with the set 0; 1; 2; without any loss of generality. For a Polish space X , (X ) will denote the Polish space of probability measures on X with the Prohorov topology. (See, e.g. [15], Chapter 2.) Recall the various cost criteria commonly considered: 1. Finite horizon cost : IE[PTn=0 r(xn ; an )] for some nite T > 0. n 2. Discounted cost : IE[P1 n=0 Æ r(xn ; an )] for some discount factor Æ (0; 1). 3. Total cost : Same as above for Æ = 1. 4. Cost till exit time : IE[P n=01 r(xn ; an )] with = min n 0 : xn = D for a prescribed D X. P 5. Expected ergodic cost : lim supn!1 n1 nm=01 IE[r(xm ; am)] 6. Pathwise ergodic cost : Minimize `almost surely' nX1 lim sup 1 r(x ; a ): (11.1) f
g
P
2
f
2
g
n!1 n m=0
n
n
P
7. Risk-sensitive cost : lim supn!1 n1 log IE[e nm=01 r(xn;an) ] All except the last one (which has per stage costs entering in a multiplicative, as opposed to additive, fashion) are amenable to a convex analytic formulation. Nevertheless, as already mentioned, we shall con ne ourselves to the pathwise ergodic cost for the most part as a representative case. (See [49] for a traditional dynamic programming based perspective.) Section 3.1 will brie y mention the counterparts of these results for the other costs. Note that if RS is used and x0 is in the support of an ergodic distribution (X) for the corresponding time-homogeneous Markov chain xn , then (11.1) will a.s. equal X (i)r(i; (i)) 2
2 P
f
where r : X
P
(A )
g
i
!
IR is de ned by
( ) =
r i; u
Z
( ) ( ) (i; u)
r i; a u da ;
2
( )
XP A :
This will be the starting point of our convex analytic formulation of the pathwise ergodic control problem, which we take up next.
12
THE LINEAR PROGRAMMING APPROACH Onesimo Hernandez-Lerma Jean B. Lasserre
This chapter is concerned with the Linear Programming (LP) approach to MDPs in general Borel spaces, valid for several criteria, including the nite horizon and long run expected average cost, as well as the in nite horizon expected discounted cost. Abstract:
12.1
INTRODUCTION
In this chapter we study the linear programming (LP) approach to Markov decision problems and our ultimate goal is to show how a Markov decision problem (MDP) can be approximated by nite linear programs. The LP approach to Markov decision problems dates back to the early sixties with the pioneering work of De Ghellinck [10], d'Epenoux [11] and Manne [30] for MDPs with nite state and action spaces. Among later contributions for nite or countable state and action MDPs, let us mention Altman [1], Borkar [8], [9], Denardo [12], Kallenberg [28], Hordijk and Kallenberg [25], Hordijk and Lasserre [26], Lasserre [29], and for MDPs in general Borel spaces and in discrete or continuous time, Bhatt and Borkar [7], Haneveld [13], Heilmann [14], [15], Hernandez-Lerma and Gonzalez-Hernandez [16], Hernandez-Lerma and Lasserre [19], [21], Mendiondo and Stockbridge[31], Stockbridge [39], Yamada [42]. Among the nice features of the LP approach, the most evident is that it is valid in a very general context. For instance, for the long-run expected average cost (AC) problem, one does not need to assume that the Average Cost Optimality Equation (ACOE) holds, a restrictive assumption. Under weak hypotheses, one obtains the existence of a stationary optimal policy (possibly 377
378
HANDBOOK OF MARKOV DECISION PROCESSES
on a subset S of the state space). The LP approach permits to identify this set S which is an ergodic class of minimum expected average-cost. Getting an expected AC optimal policy for all initial states (for the unichain case as well as the multichain case) requires much stronger assumptions. However, the LP approach is still possible via the introduction of additional variables. Also, it permits to handle some constrained MDPs in a very natural form. Finally, it is possible to devise simple convergent numerical approximation schemes that require to solve nite LPs for which eÆcient codes are now available. However, if convergence to the optimal value is obtained, it remains to devise a convergent approximation scheme for policies, as done in alternative methods like for instance in Hernandez-Lerma [17] or Sennott [37], [38] for control of queues. Let us brie y outline one simple way to see how the LP approach can be naturally introduced, although it was not the idea underlying the rst papers on the LP approach to MDPs. The starting point is to observe that given a policy , an initial distribution and a one-step cost function c : X A IR, the nite-horizon functional 2
J (; ; N; 1; c)
:= N 1 E R
N X1 t=0
!
c(xt ; at );
; can be written as a linear functional cd; N with N the expected (stateaction) occupation measure
; N (B )
:= N 1 E
N X1
1 (xt ; at ) B ; f
t=0
2
g
B 2 B (X A ):
Under some conditions, and with some limiting arguments as N , one may show that, for instance, minimizing the long-run expected average cost criterion (the AC problem) reduces to solving a linear program. More precisely, the AC R problem reduces to minimize the linear criterion cd over a set of probability measures on X A that satisfy some linear \invariance" constraints involving the transition kernel P . This approach for MDPs is of course related to the Birkho Individual Ergodic Theorem (for noncontrolled Markov chains) which states that given a homogeneous Markov chain Xt , t = 0; 1; : : : on X, a cost function c : X IR, and under some conditions, ! 1
!
lim N 1 E N !1
N X1 t=0
c(Xt )
=
Z
cd ;
for some invariant probability measure . However, we should note that the rst papers on the LP approach to MDPs used a di erent (in fact, dual) approach. Namely, the LP formulation was a rephrasing of the average (or discounted)-cost optimality equations. We brie y discuss this approach in Remark 12.5 that yields a dual linear program. Although the LP approach is valid for several criteria, including the N -step expected total cost, the in nite-horizon expected discounted cost, the control up to an exit time, the long-run expected average cost, the constrained
THE LINEAR PROGRAMMING APPROACH
379
discounted and average cost problems, we have chosen to illustrate the LP approach with the AC problem. With ad hoc suitable modi cations and appropriate assumptions, the reader would easily deduce the corresponding linear programs associated with the other mentioned problems. For instance, with respect to constrained MDPs, the reader is referred to Huang and Kurano [27], Altman [1], Piunovskiy [33] and Hernandez-Lerma and Gonzalez-Hernandez [18]. Similarly, for multiobjective MDPs, see for instance, Hernandez-Lerma and Romera [23]. We shall rst proceed to nd a suitable linear program associated to the Markov decision problem. Here, by a \suitable" linear program we mean a linear program (P) that together with its dual (P ) satis es that sup(P ) (MDP) inf(P);
(12.1)
where (using terminology speci ed in the following section) inf(P) := value of the primal program (P), sup(P ) := value of the dual program (P ), (MDP) := value function of the Markov decision problem: In particular, if there is no duality gap for (P), so that sup(P ) = inf(P); (12.2) then of course the values of (P) and of (P ) yield the desired value function (MDP) . However, to nd an optimal policy for the Markov decision problem, (12.1) and (12.2) are not suÆcient because they do not guarantee that (P) or (P ) are solvable . If it can be ensured that, say, the primal (P) is solvable|in which case we write its value as min (P)|and that min(P) = (MDP) ; (12.3) then an optimal solution for (P) can be used to determine an optimal policy for the Markov decision problem. Likewise, if the dual (P ) is solvable and its value|which in this case is written as max (P )|satis es max(P) = (MDP) ; (12.4) then we can use an optimal solution for (P ) to nd an optimal policy for the Markov decision problem. In fact, one of the main results in this chapter (Theorem 12.6) gives conditions under which (12.3) and (12.4) are both satis ed, so that in particular strong duality for (P) holds, that is, max(P ) = min(P): (12.5) Section 12.2 presents background material. It contains, in particular, a brief introduction to in nite LP. In Section 12.3 we introduce the program (P)
380
HANDBOOK OF MARKOV DECISION PROCESSES
associated to the AC problem, and we show that (P) is solvable and that there is no duality gap, so that (12.2) becomes sup(P) = min(P): Section 12.4 deals with approximating sequences for (P) and its dual (P). In particular, it is shown that if a suitable maximizing sequence for (P ) exists, then the strong duality condition (12.5) is satis ed. Section 12.5 presents an approximation scheme for (P) using nite-dimensional programs. The scheme consists of three main steps. In step 1 we introduce an \increasing" sequence of aggregations of (P), each one with nitely many constraints. In step 2 each aggregation is relaxed (from an equality to an inequality), and, nally, in step 3, each aggregation-relaxation is combined with an inner approximation that has a nite number of decision variables. Thus the resulting aggregation-relaxationinner approximation turns out to be a nite linear program, that is, a program with nitely many constraints and decision variables. The corresponding convergence theorems are stated without proof, and the reader is referred to [21] and [22] for proofs and further technical details. These approximation schemes can be extended to a very general class of in nite-dimensional linear programs (as in [20]), not necessarily related to MDPs. 12.2
LINEAR PROGRAMMING IN INFINITE-DIMENSIONAL SPACES
The material is divided into four subsections. The rst two subsections review some basic de nitions and facts related to dual pairs of vector spaces and linear operators whereas the last two subsections summarize the main results on in nite LP needed in later sections. 12.2.1
Let
Dual pairs of vector spaces
and be two arbitrary (real) vector spaces, and let ; be a bilinear , that is, a real-valued function on such that the map x x; y is linear on for every y , and the map y x; y is linear on for every x . Then the pair ( ; ) is called a dual pair if the bilinear form \separates points" in x and y, that is, for each x = 0 in there is some y with x; y = 0, and for each y = 0 in there is some x with x; y = 0. If ( ; ) is a dual pair, then so is ( ; ). If ( 1 ; 1 ) and ( 2 ; 2 ) are two dual pairs of vector spaces with bilinear forms ; 1 and ; 2 , respectively, then the product ( 1 2 ; 1 2 ) is endowed with the bilinear form X
form on
Y
h i
X Y
X Y
7! h
i
X
2 Y
7! h
i
Y
2 X
X
Y
6
6
X
X
X
2 Y
h
i 6
Y
2 X
h
i 6
Y
Y X
Y
h i
X
Y
h i
X
(x1 ; x2 ); (y1 ; y2 ) := x1 ; y1 1 + x2 ; y2 2 :
h
i
h
i
h
i
X
Y
Y
(12.6)
13
INVARIANT GAMBLING PROBLEMS AND MARKOV DECISION PROCESSES Lester E. Dubins Ashok P. Maitra William D. Sudderth
Markov decision problems can be viewed as gambling problems that are invariant under the action of a group or semi-group. It is shown that invariant stationary plans are almost surely adequate for a leavable, measurable, invariant gambling problem with a nonnegative utility function and a nite optimal reward function. This generalizes results about stationary plans for positive Markov decision models as well as measurable gambling problems. Abstract:
13.1
INTRODUCTION
This paper introduces the notion of a gambling problem that is invariant, in a sense to be speci ed below, under the action of a group or a semigroup of transformations. Our primary stimulus has been to understand more fully the relationship of Markov decision processes to gambling theory. It has long been known that these two theories are closely related (cf. Chapter 12 of Dubins and Savage (1965)) and perhaps each contains the other. However, it has not been possible to translate theorems about stationary plans, for example, directly from one theory to the other. It will be explained below how Markov decision processes may be viewed as invariant gambling problems. Subsequent sections will show how stationarity results in gambling theory (Dubins and Sudderth (1979)) are extended to invariant gambling problems and, in particular, to Markov decision problems. Our main interest here is in the reward structures. From a di erent 409
410
HANDBOOK OF MARKOV DECISION PROCESSES
point of view Schal (1989) has also studied the reward structures of gambling and Markov decision theory. A comparative study of the measurable structures of the two theories was made by Blackwell (1976). A secondary stimulus to us is the fact that there are a number of gambling problems which possess a natural group theoretic structure. Group invariance techniques have found many applications in statistical decision theory (cf. Eaton (1989) and the references therein) and could prove useful in Markov decision theory as well. We will begin with a review of measurable gambling theory, and then introduce invariance. 13.2
MEASURABLE GAMBLING PROBLEMS
Let F be a Borel set, that is, a Borel subset of a complete separable metric space. Let P(F ) be the set of probability measures on the Borel sigma- eld of subsets of F . Then P(F ), equipped with its customary weak topology, is again a Borel set. A gambling house on F is a subset of F P(F ) such that each section (x) of at x 2 F is nonempty. A strategy is a sequence 0 ; 1 ; : : : such that 0 2 P(F ), and , for each n 1, n is a universally measurable function from F n into P(F ). A strategy is available in at x if 0 2 (x) and n (x1 ; x2 ; : : : ; xn ) 2 (xn ) for every n 1 and x1 ; x2 ; : : : ; xn 2 F . Each strategy determines a unique probability measure, also denoted by , on the Borel subsets of the history space H = F N , where N is the set of positive integers and H is given the product topology. Let X1 ; X2 ; : : : be the coordinate process on H ; then, under the probability measure , X1 has distribution 0 and , for n 1, Xn+1 has conditional distribution n (x1 ; x2 ; : : : ; xn ) given X1 = x1 ; X2 = x2 ; : : : ; Xn = xn . A measurable gambling problem is a triple (F; ; u), where F , the fortune space, is a nonempty Borel set, is a gambling house on F which is an analytic subset of F P(F ), and u : F 7! R, the utility function, is upper analytic, which means that fx : u(x) > ag is an analytic subset of F for every real a. Such structures with and u both Borel were introduced by Strauch (1967); the extension to analytic and upper analytic u is due to Meyer and Traki (1973). In the theory of gambling (Dubins and Savage (1965)) there are two natural approaches to a gambling problem. In leavable problems, the gambler is allowed to stop playing at any time, whereas in nonleavable problems, the gambler is compelled to continue playing forever. Consider rst a leavable problem and de ne a stop rule t to be a universally measurable function from H into f0; 1; : : : g such that whenever t(h) = k and h and h agree in their rst k coordinates, then t(h ) = k . (In particular, if t(h) = 0 for some h, then t is identically 0.) A gambler with initial fortune x selects a strategy available at x and a stop rule t. The pair = (; t) is a policy available at x. The expected reward to a gambler who selects the policy is 0
0
u( ) =
Z
u(Xt ) d;
14
NEURO-DYNAMIC
PROGRAMMING: OVERVIEW AND RECENT TRENDS
Benjamin Van Roy
Neuro-dynamic programming is comprised of algorithms for solving large-scale stochastic control problems. Many ideas underlying these algorithms originated in the eld of arti cial intelligence and were motivated to some extent by descriptive models of animal behavior. This chapter provides an overview of the history and state-of-the-art in neuro-dynamic programming, as well as a review of recent results involving two classes of algorithms that have been the subject of much recent research activity: temporal-di erence learning and actor-critic methods. Abstract:
14.1
INTRODUCTION
In the study of decision-making, there is a dividing line between those who seek an understanding of how decisions are made and those who analyze how decisions ought to be made in the light of clear objectives. Among the former group are psychologists and economists who examine participants of physical systems in their full complexity. This often entails the consideration of both \rational" and \irrational" behavior. The latter group|those concerned with rational decision-making|includes engineers and management scientists who focus on the strategic behavior of sophisticated agents with de nite purposes. The intent is to devise strategies that optimize certain criteria and/or meet speci c demands. The problems here are well-de ned and the goal is to nd a \correct" way to make decisions, if one exists. The self-contained character of rational decision problems has provided a ground for the development of much mathematical theory. Results of this work|as exempli ed by previous chapters of this volume|provide an under431
432
HANDBOOK OF MARKOV DECISION PROCESSES
standing of various possible models of dynamics, uncertainties, and objectives, as well as characterizations of optimal decision strategies in these settings. In cases where optimal strategies do exist, the theory is complemented by computational methods that deliver them. In contrast to rational decision-making, there is no clear-cut mathematical theory about decisions made by participants of natural systems. Scientists are forced to propose speculative theories, and to re ne their ideas through experimentation. In this context, one approach has involved the hypothesis that behavior is in some sense rational. Ideas from the study of rational decisionmaking are then used to characterize such behavior. In nancial economics, this avenue has lead to utility and equilibrium theory. To this day, models arising from this school of economic thought|though far from perfect|are employed as mainstream interpretations of the dynamics of capital markets. The study of animal behavior presents another interesting case. Here, evolutionary theory and its popular precept|\survival of the ttest"|support the possibility that behavior to some extent concurs with that of a rational agent. There is also room for reciprocal contributions from the study of natural systems to the science of rational decision-making. The need arises primarily due to the computational complexity of decision problems and the lack of systematic approaches for dealing with it. For example, practical problems addressed by the theory of dynamic programming can rarely be solved using dynamic programming algorithms because the computational time required for the generation of optimal strategies typically grows exponentially in the number of variables involved|a phenomenon known as the curse of dimensionality. This de ciency calls for an understanding of suboptimal decision-making in the presence of computational constraints. Unfortunately, no satisfactory theory has been developed to this end. It is interesting to note that similar computational complexities arise in attempts to automate decision tasks that are naturally performed by humans or animals. The fact that biological mechanisms facilitate the eÆcient synthesis of adequate strategies motivates the possibility that understanding such mechanisms can inspire new and computationally feasible methodologies for strategic decision-making. Over the past two decades, algorithms of reinforcement learning|originally conceived as descriptive models for phenomena observed in animal behavior{ have grown out of the eld of arti cial intelligence and been applied to solving complex sequential decision problems. The success of reinforcement learning algorithms in solving large-scale problems has generated excitement and intrigue among operations researchers and control theorists, and much subsequent research has been devoted to understanding such methods and their potential. Developments have focused on a normative view, and to acknowledge the relative disconnect from descriptive models of animal behavior, some operations researchers and control theorists have come to refer to this area of research as neuro-dynamic programming, instead of reinforcement learning. In this chapter, we provide a sample of recent developments and open issues at the frontier of research in neuro-dynamic programming. Our two points of focus are temporal-di erence learning and actor-critic methods|two algo-
NEURO-DYNAMIC PROGRAMMING: OVERVIEW AND RECENT TRENDS
433
rithmic ideas that have found greatest use in applications of neuro-dynamic programming and for which there has been signi cant theoretical progress in recent years. We begin, though, with three sections providing some background and perspective on the methodology and problems that may address. 14.2
STOCHASTIC CONTROL
As a problem formulation, let us consider a discrete-time dynamic system that, at each time t, takes on a state xt and evolves according to xt+1 = f (xt ; at ; wt );
where wt is a disturbance and at is a control decision. Though more general (in nite/continuous) state spaces can be treated, to keep the exposition simple, we restrict attention to nite state, disturbance, and control spaces, denoted by X, W , and A , respectively. Each disturbance wt 2 W is independently sampled from some xed distribution. A function r : X A 7! IR associates a reward r(xt ; at ) with a decision at made at state xt . A stationary policy is a mapping : X 7! A that generates state-contingent decisions. For each stationary policy , we de ne a value function v (; ) : X 7! IR by "
v (x; ) = IE
1 X
t=0
#
t r(xt ; (xt )) x0 = x ;
where 2 [0; 1) is a discount factor and the state sequence is generated according to x0 = x and xt+1 = f (xt ; (xt ); wt ). Each v (x; ) can be interpreted as an assessment of long term rewards given that we start in state x and control the system using a stationary policy . The optimal value function V is de ned by V (x) = max v (x; ):
A standard result in dynamic programming states that any stationary policy given by h i (x) = argmax IE r(x; a) + V (f (x; a; w)) ; a2A
w
where IE[] denotes expectation with respect to the distribution of disturbances, w is optimal in the sense that V (x) = v (x; );
for every state x (see, e.g. [8]). For illustrative purposes, let us provide one example of a stochastic control problem. The video arcade game of Tetris can be viewed as an instance of stochastic control (we assume that the reader is familiar with this popular game). In particular, we can view the state xt as an encoding of the current \wall of bricks" and the shape of the current \falling piece." The decision at
Example 14.1
15
MARKOV DECISION PROCESSES IN FINANCE AND DYNAMIC OPTIONS Manfred Schal
Dedicated to Professor Dr. Dr.h.c. Karl Hinderer on the occasion of his seventieth birthday
In this paper a discrete-time Markovian model for a nancial market is chosen. The fundamental theorem of asset pricing relates the existence of a martingale measure to the no-arbitrage condition. It is explained how to prove the theorem by stochastic dynamic programming via portfolio optimization. The approach singles out certain martingale measures with additional interesting properties. Furthermore, it is shown how to use dynamic programming to study the smallest initial wealth x that allows for super-hedging a contingent claim by some dynamic portfolio. There, a joint property of the set of policies in a Markov decision model and the set of martingale measures is exploited. The approach extends to dynamic options which are introduced here and are generalizations of American options. Abstract:
15.1
INTRODUCTION AND SUMMARY
In discrete time n = 0; 1; ; N a nancial market is studied which is free of arbitrage opportunities but incomplete. In the market 1 + d assets can be traded. One of them with price process fBn , 0 n N g is called the bond or savings account and is assumed to be nonrisky. The other d assets are called stocks and are described by the d-dimensional price process fSn , 0 n N g. An investor is considered whose attitude towards risk is speci ed in terms of a utility function U. A dynamic portfolio is speci ed by a policy . The investor's objective is to maximize the expected utility of the discounted terminal wealth XN (x) when starting with an initial wealth x . 461
462
HANDBOOK OF MARKOV DECISION PROCESSES
Utility optimization is now a classical subject (Bertsekas [1], Hakansson [16]). The present paper is intended as a bridge between Markov decision processes (MDPs) and modern concepts of nance. It is assumed the reader knows some basic facts about MDPs, but knowledge about Mathematical Finance is not needed. In particular, we here are interested in using the no-arbitrage condition for the construction of a particular martingale measure by use of the optimal policy . A condition close to the no-arbitrage condition was already used by Hakansson [16]. Martingale measures are used for option pricing. The paper makes use of an approach how to base option pricing on the optimal solution to the portfolio optimization problem. This approach is also explained by Davis [4]. In a one-period model, the construction of a martingale measure by use of an optimization problem is easily explained. Let us assume d = 1. If the action (portfolio) a 2 IR is chosen then the discounted terminal wealth of the portfolio has the form x + a R where R can be interpreted as a return. If a = a is optimal then one has @E [U (x + a R)]=@a = 0 = E [U 0 (x + a R) R] for a = a . Upon de ning a new measure Q by dQ = const U 0 (x + a R)dP , one obtains R R dQ = EQ [R] = 0 which is the martingale property. Since the martingale property is a local one both in time and in space, martingale measures can be constructed by local optimization problems (see Rogers 1994) whereas we will apply global (dynamic) optimization. This has the advantage that the resulting martingale measures have interesting interpretations. The case where the utility function U is only de ned for positive values is treated in [39, 40]. A second interesting problem concerns values x for the initial wealth that allow for super-hedging discounted contingent claims X by some policy , i.e. XN (x) X . The smallest value x coincides with the maximal expectation of X under (equivalent) martingale measures. The proof can make use of dynamic programming by exploiting an analogy between the set of all policies in a stochastic dynamic programming model and the set of martingale measures. A similar problem can be considered for American options where an optimal stopping time has to be chosen. It is well-known that an optimal stopping problem can be considered as a special stochastic dynamic programming problem. Therefore it is natural from the point of view of Markov decision theory to generalize the concept of an American option. This is done in the present paper and the generalization is called a dynamic option which has interesting applications. 15.2
THE FINANCIAL MARKET
On the market an investor can observe the prices of 1+ d securities at the dates time horizon. One of the securities is a bond (or savings account) with interest rates rn , 1 n N . It is essential for the theory that the interest rates for borrowing and lending are assumed to be the same. The bond price process is de ned by
n = 0; 1; ; N where N is the
Bn := (1 + r1 ) (1 + rn ); 0 n N; where B0 = 1 :
(15.1)
Here we assume that fBn g is a deterministic process. If fBn g is given as the initial term structure, then the interest rates rn can be computed by (15.1).
16
APPLICATIONS OF MARKOV DECISION PROCESSES IN COMMUNICATION NETWORKS Eitan Altman
We present in this chapter a survey on applications of MDPs to communication networks. We survey both the di erent application areas in communication networks as well as the theoretical tools that have been developed to model and to solve the resulting control problems. Abstract:
16.1
INTRODUCTION
Various traditional communication networks have long coexisted providing disjoint speci c services: telephony, data networks and cable TV. Their operation has involved decision making that can be modeled within the stochastic control framework. Their decisions include the choice of routes (for example, if a direct route is not available then a decision has to be taken which alternative route can be taken) and call admission control; if a direct route is not available, it might be wise at some situations not to admit a call even if some alternative route exists. In contrast to these traditional networks, dedicated to a single application, today's networks are designed to integrate heterogeneous traÆc types (voice, video, data) into one single network. As a result, new challenging control problems arise, such as congestion and ow control and dynamic bandwidth allocation. Moreover, control problems that had already appeared in traditional networks reappear here with a higher complexity. For example, calls corresponding to di erent applications require typically di erent amount of network resources (e.g. bandwidth) and di erent performance bounds (delays, loss probabilities, throughputs). Admission control then becomes much more complex than it was in telephony, in which all calls required the same perfor489
490
HANDBOOK OF MARKOV DECISION PROCESSES
mance characteristics and the same type of resources (same throughput, bounds on loss rates and on delay variation). We do not aim at a complete survey of the area, since several other surveys [90, 94, 149, 119, 150, 152, 251, 248, 250, 253] on related issues already exist, see also [221]. Other references that focus on the methodology that allows one to apply MDPs to communication networks can be found in the following books [4, 54, 111, 239, 271, 283] as well as in the survey paper [192]. We have two goals in this survey. First, we wish to present to researchers who specialize in MDPs a central application area which provides a vast eld of challenging problems. We would like to familiarize these researchers with special complex features in control problems that arise in communications: complex information structure, problems with multiobjective and multiagents. A second objective is to familiarize researchers in communications with tools that have been developed for modeling and for solving control problems in networks. Problems that are described in this survey are MDPs in a general sense: they are described as Markov chains whose transition probabilities are controlled. This control can be done by a single or several controllers, having the same or having di erent objectives. It is often tools other than the standard dynamic programming that are used to solve these control problems. For completeness, we occasionally mention approaches that have been used for control of communication systems which are not based on MDPs, and then relate or compare these to MDPs. 16.2
THE CONTROL THEORETICAL FRAMEWORK
The most popular telecommunication network architectures today are the Internet and the ATM (Asynchronous Transfer Mode) networks. The Internet o ers today a \best e ort" type service, i.e. the resources in the network are shared among all users, and when the number of users increases, the quality of service (in terms of delay, throughput, losses) per user decreases. In contrast, ATM networks provide mostly guaranteed services: if a session needs a given Quality Of Services (QOS) it may establish a contract with the network that guarantees that all along the duration of the session, some required bounds would hold on given performance measures (loss probabilities, delays, delay variation, throughput), as long as the source respects the terms of the contract (in terms of the average and the maximum throughput it sends, as well as some constraints on its bursty behavior). Two guaranteed service classes are de ned in ATM: the CBR (Constant Bit Rate) and VBR (Variable Bit Rate) service classes. ATM contains also two best e ort type services: the Available Bit Rate (ABR) service and the Unspeci ed Bit Rate (UBR) service. In the ABR service, the network determines the allowed transmission rate of each source by sending to them periodically appropriate control signals. As long as a source abides to those commands, the network can guarantee some given bounds on its loss rates. In the UBR service no guarantees are given on performance measures.
17
WATER RESERVOIR APPLICATIONS OF MARKOV DECISION PROCESSES Bernard F. Lamond Abdeslem Boukhtouta
Decision problems in water resources management are usually stochastic, dynamic and multidimensional. MDP models have been used since the early fties for the planning and operation of reservoir systems because the natural water in ows can be modeled using Markovian stochastic processes and the transition equations of mass conservation for the reservoir storages are akin to those found in inventory theory. However, the \curse of dimensionality" has been a major obstacle to the numerical solution of MDP models for systems with several reservoirs. Also, the use of optimization models for the operation of multipurpose reservoir systems is not so widespread, due to the need for negotiations between di erent users, with dam operators often relying on operating rules obtained by simulation models. In this chapter, we present the basic concepts of reservoir management and we give a brief survey of stochastic in ow models based on statistical hydrology. We also present a stochastic dynamic programming model for the planning and operation of a system of hydroelectric reservoirs, and we discuss some applications and computational issues. We feel many research opportunities exist both in the enhancement of computational methods and in the modeling of reservoir applications. Abstract:
17.1
INTRODUCTION
Dams and reservoirs have long been used for storing surplus water during rainy seasons to provide irrigation and drinking water during dry periods. They prevent ooding during periods of thaw or unusually high rainfall. They also 537
538
HANDBOOK OF MARKOV DECISION PROCESSES
serve to regulate ow and depth of water in lakes and rivers for navigational purposes, and to move ships up and down locks as in the Panama canal and the Saint-Lawrence seaway. Throughout the twentieth century, hydroelectric production has become a major economic bene t of dams, reservoirs and water resources. The sequential nature of the reservoir management decisions, together with the inherent randomness of natural water in ows, explains the frequent modeling of reservoir management problems as Markov decision processes (MDPs), and their optimization by stochastic dynamic programming (SDP). The rst discussion of reservoir management in this framework is usually credited to Pierre Masse [29] in 1946. Optimization results for the hydroelectric production of a single reservoir were published a decade later, with the numerical computation of an optimal policy [25] and the analytic structure of optimal policies for hydrothermal systems [12]. These results paralleled similar developments that occurred in inventory theory at the same epoch. There is an extensive literature on models and methods for reservoir optimization. Surveys can be found in [22, 53, 54]. Nonetheless, large reservoir systems have been in operation for decades before the development of optimization models. Reservoir operators have thus relied on rule curves and other agreed upon operating rules, as well as their own judgment and experience in making reservoir release decisions [27]. While optimization models are now often used in practice for planning purposes, their use in real-time multiple-reservoir operation is not so widespread. According to [32], \The need for comprehensive negotiations and subsequent agreements on how to operate a reservoir system seems to be a main reason why most reservoir systems are still managed based on xed prede ned rules. [ : : : ] Optimization models can help de ne these prede ned rules, rules that satisfy various constraints on system operation while minimizing future spills or maximizing energy production or minimizing expected future undesired deviations from various water release, storage volume and/or energy production targets."
The \optimization models" referred to in the above citation are usually based on linear programming (LP) or nonlinear programming (NLP), with the random variables of future in ows replaced by their most recent forecasts. These (deterministic) models must be solved every period with updated forecasts and their solutions provide an open loop control . By contrast, an optimal policy of an MDP gives a closed loop control , or feedback solution , which is more in the form of traditional operating rules. On the other hand, for reservoir systems whose main purpose is hydroelectric generation, the use of solutions from optimization models is widespread. In [45, 49], for instance, MDP models are presented for the long term planning of the aggregated system, to obtain optimal policies for monthly release and storage targets. Then a hierarchy of deterministic models [13] are used for medium term (NLP) and short term scheduling (LP). See also [16]. A comparison of optimal MDP solutions with traditional rule curves solutions was made in [45] for the Brazilian system, where optimal MDP solutions were shown to have the same reliability as rule curve solutions, but with signi cantly increased pro ts.
WATER RESERVOIR APPLICATIONS OF MDP
539
Stochastic optimization models of hydroelectric production are usually needed when the planning horizon has a length of one or several years, with a time step of one month or longer. The long term scheduling of hydroelectric production is mainly concerned with the larger (annual or multiannual) reservoirs managed by a utility. Typical problems consider twenty or more such reservoirs and are therefore multidimensional. In general, an optimal decision rule for a given period would thus consist of twenty functions of at least twenty variables, each function giving a release target for one reservoir depending on the stored volumes at every reservoir in the system. Such functions are obviously impossible to tabulate numerically (curse of dimensionality). Hence research in this area has attempted to develop (1) aggregation-disaggregation methods, (2) numerical approximation and optimization methods, and (3) analytical solutions [22]. Research also addresses important modeling issues in statistical hydrology (stochastic processes of natural in ows with adequate representation of serial and spatial correlations) and energy economics (such as evaluating marginal production costs in the context of deregulated markets). The sequel is organized as follows. A brief review of the basic reservoir management concepts and traditional operating policies, are given in x17.2. A survey of several models describing the stochastic processes of natural in ows is presented in x17.3 and a dynamic programming optimization model for a multi-reservoir hydroelectric system is given in x17.4. Di erent applications of MDP models are presented in x17.5. A survey of recent research on MDP solution methods is presented in x17.6. Finally, x17.7 contains directions and open problems in water reservoir applications of MDPs, and some concluding remarks. 17.2
RESERVOIR MANAGEMENT CONCEPTS
More than 5000 years ago the Egyptians measured uctuations of the Nile river and built dams and other hydraulic structures to divert water to agricultural elds. Since then, practical water knowledge has proliferated among dam operators, farmers, and other users. But the concept of a water cycle (hydrological cycle) became rmly established in the scienti c literature only in the seventeenth century [38]. The hydrological cycle is the continuous circulation of water from the sea to the atmosphere, then to the land and back again to the sea. The water exchanges involved at the various stages of the cycle are evaporation, water-vapor transport, condensation, precipitation, and runo . Runo from land surface is the residual water of the hydrological cycle, which has not been evaporated by plants and has not in ltrated the ground surface, so it is available for use. The collection of land whose surface waters drain into a river valley forms the hydrographic basin of that river. A dam is a barrier built across a watercourse for impounding water. By erecting dams, humans can obstruct and control the ow of water in a basin. A reservoir is a (possibly arti cial) lake, usually the result of a dam, where water is collected and stored in quantity for use. Reservoirs must occupy the best available sites in the hydrographic basin because their development requires unique geological, hydrological, topographical and geographical characteristics.
Index
-perturbation, 139 -approximation, 37 -optimality, 37 c rule, 29, 508 -geometric ergodicity, 251 Abel summability, 232 absorbing set, 311 achievable region, 509 ACOE, 393, 397, 399 action additive, 419 available, 3 conserving, 181 delays, 499 elimination of suboptimal, 30 improving, 34 set, 22 space, 3 suboptimal, 30, 32 active zone, 540 actor, 446 actor-critic methods, 446 adapted, 465 adaptive control, 69 admission control, 434 aggregate
ows, 543 reservoir model, 544 system, 544 aggregation, 384, 401 algorithm, 330 dynamic programming, 184 annual
ows, 542, 543 river ows, 542 aperiodicity strong, 51 approximate policy iteration, 440 approximate value iteration, 444 arbitrage, 461 asset allocation, 434
assignment customer, 29 server, 29 asynchronous information, 496 ATM, 490 average cost, 315, 378 linear programming, 45 modi ed policy iteration, 51 optimal policy, 325 optimality equation, 42, 102, 325, 393 optimality selection equation, 102 policy iteration, 43 value iteration, 48 average evaluation equations, 96 average reward, 25, 89 average reward criterion, 40 average reward optimal stationary policy, 153 backward induction, 30 bandits, 509, 513 barycenter, 351 basis function, 436 bias as total reward, 93 constant, 99 evaluation equations, 96 optimality, 53 optimality equations, 103 optimality selection equations, 103 vector, 42 bilinear form, 380 Blackwell optimal, 91, 132, 233, 247, 255 equation, 239, 242, 255 linear programming, 55 policy, 41, 154, 255 block-pivoting, 36 bond price process, 462 Borel, 377 Borel set, 410 Borel state space, 259
559
560
HANDBOOK OF MARKOV DECISION PROCESSES
budget equation, 464 bu er management, 501 bu er zone, 540 c rule, 29, 508 c-regularity, 314 call admission control, 499{501 and routing, 505 constant rate sources, 499 integrated services, 501 variable rate sources, 500 Caratheodory functions, 262 Caroni river (Venezuela), 550 Cesaro-limit, 40 Choquet, 351 commercial navigation, 551 comparison lemma, 254 complementary slackness, 384 complete market, 466 complex information, 498 complexity policy iteration, 35 composite reservoirs, 553 congestion control, 502{504 conservation laws, 509 conservation zone, 540 conserving, 216, 242 action, 181 policy, 255 constrained control, 363 constrained MDPs, 493 monotone policies, 503 scheduling of service, 509 constrained optimization optimal, 7 weighted discount, 213 constraints additional, 30 contingent claim, 466 continuous time parameter models, 341 control admission, 28 optimal control of queues, 28 service rate, 28 controllability matrix, 320 controllable, 320, 336 controlled in ows, 540 controlled TD, 441 convex cone, 382 convex programming, 347, 352 cost function, 324 countably parameterized house, 423 criteria average reward, 5 mixed, 209 total reward, 5, 173 critic, 446, 451 curse of dimensionality, 436, 453 dam, 537, 539, 541, 550
data transformation, 49 dead storage zone, 540 deadline constraints scheduling, 510 delayed information, 495
ow control, 504 delayed sharing information, 497 deregulated market, 539, 550 deterministic policy, 4 deviation matrix, 94, 115, 234 deviation operator, 252 di usion approximations, 516 discount factor, 24 discounted occupation measure, 357 discrete in ow models, 544 random variables, 544 randomized Markov policy, 220 dominate, 219 Drazin inverse, 96 dual, 383 cone, 382 linear program, 363 pair, 363, 380, 387 duality gap, 379, 384, 393 dynamic portfolio, 464 dynamic programming algorithm, 184 electricity, 540, 545, 547, 548 eligibility vector, 438 emissions reductions, 435 energy, 538, 547, 553, 554 demand, 550 economics, 539 generation, 550 in ows, 544 production, 550 targets, 550 ergodic class, 139 ergodic occupation measure, 349 Esscher transform, 477 European call option, 466 evaporation, 539, 540, 547, 550 losses, 550 rate, 547, 550 excess reward, 92 existence of optimal policies, 326 exploration, 442 exposed set, 221 extreme point, 350 set, 221 fair hedging price, 477 feasible, 7, 213, 383 feedback law, 324 nancial market, 461 rst passage problem, 27
ood control, 540, 548, 553 zone, 540
ow control, 502{504, 540
INDEX equation, 548 forecasts, 545
uid approximations, 516 forebay elevation, 540 fortune space, 410 fuel, 545, 554 full set, 312 funnel, 216 gain, 89 gambling, 27 gambling house, 410 game theory, 491 Gauss-Seidel, 38 general convergence condition, 174, 177 geometric life time, 209 Girsanov transformation, 478 Gittins index, 29, 63, 509, 513 gradient estimation, 449 group, 412 Hamiltonian Cycle, 138 head e ect, 552, 554 High Aswan dam, 550 history, 4 history space, 410 hydraulic head, 550 hydro generations, 549 plant, 540, 548 power, 551 production, 541 hydroelectric energy, 550 generation, 538, 552, 554 production, 538, 539 system, 545 hydrographic basin, 539 hydrologic activity, 546 complexity, 551 information, 545, 546, 552 seasonality inputs, 551 variables, 545, 546 hydrological activity, 546 cycle, 539 hydropower, 550, 552 reservoirs, 550 system, 547, 553 ideal point, 369 identity, 412 implicit discounting, 108 inactive zone, 540 incomplete information, 494 indi erence value, 63 in nite linear program, 383 in ow variations, 552 in ows, 538, 542{547, 549
561
records, 551 information action delays, 499 asynchronous, 496 delayed, 495 delayed sharing, 497 incomplete, 494 nested, 498 partial, 494 quantized, 495 sampled, 496 inner approximation, 384, 403 interest rate, 55 invariant function, 413 gambling problem, 412 measure, 311 selector, 416 stationary family of strategies, 416 inventory problem, 28, 67 irreducibility ( -), 309 irrigation, 540 joining the shortest queue, 514 with fastest server, 507 largest remaining index, 65 Laurent expansion, 42 series expansion, 94, 234 leavable gambling house, 412 leavable gambling problem, 410 lexicographic, 220 Bellman operator, 241 ordering, 219, 237 policy improvement, 239, 243 lexicographically optimal policy, 219 limit Markov control problem, 134 limiting matrix, 93 linear program, 23, 134, 347, 352, 377{ 379, 385, 387, 389, 479 average rational functions, 56 average reward, multichain case, 46 average reward, unichain case, 48 Blackwell optimality, 55 discounted rewards, 36 in nite, 383 linear quadratic control delayed information, 504
ow control, 504 long-run frequency space, 141 Lyapunov equation, 320 Lyapunov function, 245, 314 m-discount optimal policy, 131 m-optimal, 325 M-randomized, 220 maintenance problem, 27 Markov arrival processes, 509
562
HANDBOOK OF MARKOV DECISION PROCESSES
Markov chain transient, 139 Markov Decision Process, 3 Markov decision processes unichained, 139 Markov games, 492 Markov policy, 324 Markov renewal problem, 68 martingale, 465 martingale measure, 465 matrix deviation, 41 fundamental, 41 stationary, 40 MDP Borel, 200 discounted, 174, 176, 184 nite-horizon, 183 general convergent, 177 negative, 174, 177, 185 non-homogeneous, 183 positive, 174, 177, 186 positive bounded, 188 unbounded discounted, 176, 185 uncountable, 200 mean return time, 306 measurable gambling problem, 410 minimum pair, 387 minorization, 307 mixed criteria, 209 weighted discount, 213 constrained, 213 models linear state space, 320, 336 network, 321, 337 modi ed policy iteration average reward, 52 discounted rewards, 40 monotone contraction mapping, 31 Moore's law, 210 multi-armed bandit, 29, 62 multi-objective MDPS, 69 multi-reservoir arborescent system, 553 hydroelectric system, 539 system, 545, 546, 551{553 multichain constrained MDPs, 47 multicriteria, 493, 494 multiobjective, 363, 369 multiplicative action, 419 n-average optimality, 243 n-discount optimal, 93, 236 n-order conserving, 242 n-order optimality equation, 242 Nash equilibrium, 227 natural in ows, 540{542, 544 water in ows, 538, 541 navigational safety, 548
near-monotone, 353 near-monotone function, 314 nearly completely decomposable, 118, 120 nearly optimal, 91, 234 nearly uncoupled, 118 nested information, 498 network scheduling, 323 neuro-dynamic programming, 431, 517 no information, 497 routing, 507 no-arbitrage condition, 467 non-repeating condition, 176, 197 nonleavable gambling problem, 410 nonrandomized policy, 324 norm-like function, 314 observable, 336 occupation measure, 378 one-armed bandit, 62 operator S , 411 optimal, 7 bias, 25 Blackwell, 25 lexicographic, 220 n-average, 25 n-discount, 25 nearly, 25 Pareto, 219 optimal bias constant, 102 optimal control of queues, 28 optimal policy Pareto, 219 optimal reward function, 411 optimal stopping, 26 linear programming, 62 monotone, 62 optimality average overtaking, 53 overtaking, 53 Pareto, 61 principle, 30 optimality equation, 6, 23, 180, 472, 483 average cost, 42 average reward, 6 discounted reward, 6 nite horizon, 6 rst, 6 second, 6 optimality operator, 184 orbit, 416 orbit selector, 416 packet admission control, 501 and routing, 506 Pareto equilibrium, 493 Pareto optimal, 219 policy, 219 Pareto point, 369 partial information, 69, 494
INDEX partial observation, 69 penstocks, 540 Perese model, 549 performance, 7 space, 219, 220 vector, 213, 219 petite set, 309 Poisson equation, 269, 273, 315 average cost, 275 continuity, 293 Green decomposition, 280 Lipschitz continuity, 296 solution, 280, 281, 284 state decomposition, 279 uniqueness, 276 policy, 4, 22, 410 (N; )-stationary, 216 (f; I )-generated, 194 bang-bang, 28 conserving, 31, 181 control limit, 26 eÆcient, 69 equalizing, 181 feasible, 7, 213 g-optimal, 186 longest expected processing time, 29 M-randomized, 220 Markov, 4 mixed, 200 monotone, 503 optimal, 23 randomized, 4 randomized Markov, 4 randomized Markov of order M, 220 randomized stationary, 4 semi-Markov, 180 shortest expected processing time, 29 shortest queue, 29 stationary, 4 stationary on a set, 198 strong (m; N ), 221 structure, 513{514 switching curve, 503, 514 threshold, 514 tracking, 194 transient, 61 uniformly nearly optimal, 199 policy improvement, 506 policy iteration, 23, 333, 399 average cost, 43 average reward, 44 average reward, unichain case, 44 complexity, 35 convergence rate, 34 modi ed, 39, 51 policy space, 4 polling, 510{511 positive cone, 382 positive operator, 254
1
563
positive recurrence, 311 potential energy, 544, 553 potential kernel, 308 power generation, 540, 548, 552 targets, 545, 550 power series algorithm, 517 primal linear program, 383, 389 principle of optimality, 30 pure policy, 4 Q-function, 443, 451 Q-learning, 443 Q-values, 451 quality of service, 226 quantized information, 495 queueing control, 105 recreational boating, 551 recurrence, 310 Harris, 313 recurrence conditions, 256 regular perturbations, 116 regular policy, 325 relative risk process, 463 relative value algorithm, 50 function, 99, 325 iteration, 51 release targets, 551 reliable power output, 551 repair problem, 27 replacement problem, 27, 67 reservoir, 539, 552 capacity, 548, 550 in ows, 542, 551 management, 538 concepts, 539 models, 549 problems, 545 operating policies, 540 operation and design, 551 operations, 554 operators, 538 release decisions, 538 policy, 545, 550 system, 540, 547, 553, 554 reservoirs, 537, 539{541, 545, 547, 549{ 553 in parallel, 553 in series, 553 resolvent, 234 restart-in k-problem, 64 return process, 463 reward, 3, 22 discounted, 5 expected total, 5 one step, 3 operator, 472
564
HANDBOOK OF MARKOV DECISION PROCESSES
terminal, 5 total expected, 23 total expected -discounted, 24 risk sensitive control, 342 river ows, 541 routing, 434, 504{508 after queueing, 508 and call admission control, 505 and packet admission control, 506 no state information, 507 routing problem, 26 rule curves, 538, 540 runo , 539 s-optimal, 325 sample path methods, 514{515 sampled information, 496 satellite communications, 511{512 random access, 512 scheduling transmission opportunities, 512 scheduling deadline constraints, 510 scheduling of service, 508{510 constrained MDPs, 509 Schur convexity, 514 seepage, 540 selector, 416 semi-Markov decision chains, 259 semi-Markov decision problem, 68 semiconductor wafer fabrication, 435 semigroup, 412 sensitive optimality criteria, 89 separable Markov decision problems, 65 separable MDPs, 67 separable problem, 66 totally, 68 serial in ow correlations, 546 side constraints, 30 signed martingale measure, 478 simplex algorithm, 36 simultaneous Doeblin condition, 256 single composite reservoir, 553 reservoir, 538, 551 singularly perturbed Markov decision processes, 130 smooth equivalence relation, 417 space action, 3 available actions, 3 policy, 4 state, 3 span, 32 special semigroup, 418 spill, 538, 546, 547 variables, 548 zone, 540 spillways, 540, 546, 552 stability analysis, 515
stabilizer subgroup, 417 stable case, 353 stable policy, 391 state space, 3, 22 state-action frequencies, 58 discounted, 36 stationary family of strategies, 416 M-randomized, 220 policy, 324 statistical hydrology, 539, 542 stochastic models in hydrology, 544 process of natural in ows, 539, 541, 542, 547, 552 reservoir systems, 554 stochastic games, 227, 492 lexicographic, 227 monotone policies, 503 perfect information, 227 stochastic gradient methods, 449 stochastic Lyapunov condition, 353, 370 stochastic scheduling, 29 stop rule, 410 stop-or-go house, 423 storage, 541, 548, 551, 553 allocation zones, 540 bounds, 553 capacities, 553 strategic measure, 178 strategy, 410 available, 410 stream ow, 544 forecasts, 545 generation, 544 process, 545 scenarios, 545 statistics, 544 strong (m; N ), 221 strong Blackwell optimal policy, 247 strong duality, 379, 384, 395, 397 strong non-repeating condition, 201 sub-model, 216 suboptimal actions, 36 successive approximation, 184 super-hedging, 468 superharmonic , 35 average, 45 supply-chain management, 435 switching curve policies, 514 switching curves, 501, 507 T-chain, 309 taboo transition matrix, 256 tailwater elevation, 540 target problem, 27 team theory, 491 temporal di erence, 438 temporal-di erence learning, 437
INDEX Tetris, 433, 437 thermal generation, 549 threshold policies, 514 tight, 394 topological group, 412 topological semigroup, 412 total cost, 357 total reward criteria, 173 trajectory, 3 transient, 33 transition densities, 259 kernel, 307 matrix, 23 probability, 3, 22 transitivity condition, 176, 195 turbine capacities, 548 eÆciencies, 540 releases, 547, 552 uncertainty on in ows, 554 uniform -geometric ergodicity, 251 uniform -geometric recurrence, 256 utility function, 22, 410 value function approximation, 436 value iteration, 23, 37, 173, 184, 330, 399, 514 average cost, 48 discounted rewards, 38
Gauss-Seidel, 38 Pre-Gauss-Seidel, 38 variability, 58, 59 vector value, 23 vector-valued MDPs, 69 w-invariant function, 413 water, 548, 550, 551, 553 head, 540 factor, 548 reservoir applications, 552 supply requirements, 550 weak duality, 383 weak topology, 381 weak-star topology, 381 weak-strong topology, 262 weighted discount optimization, 213 constrained, 213 wireless communications, 511{512 admission control, 511 control of handover, 511 mobility tracking, 512 power control, 512 random access, 512 routing, 512 ws-topology, 262 z standard policy, 163 zero-sum games, 492 zones, 540
565
There is a mistake or typo in Example 2.1, Page 132. The values of " " " 2 (u) and g2 (v ) should be the same as g1 (u) and g1 (v ) if " > 0 (since it is a unichain case). Luckily, it does not have any e ect on the conclusion. The correct expression for g" (u) and g" (v) is:
g"
8 1 > > "=0 < 0 g " (u) = 1 > > ">0 :
1
8 1 : 5 > > "=0 < 0 and g" (v) = 0:75 > > ">0 :
0:75
1
!#"%$'&)(" *,- + /.0213* -5+ 476 /.809;8:< - = 4>62?)@ A &B$'&C:< -ED GF' &H" D JIB KLF'&B"%8 $JF'M ! D N:< - O1 P6 Q ? < -0R S T(UIV K J&V5 &VK IV&W < X FTZ[&\$'&B!](IB&V"^ZL_`:< - (aFbJ&Bc&B$'(!!](IV&V D KdF' &eJ&V5 &V!fhgbM2IV i$' &W F' &H" Dkj &$'&VK IV&H" &VlK -Y Fm(a [&)(a$ D KnF' &HM !#!# A D K oe D F' (aF D K0 qpasr - 1tuM $m(!#!0vCWxw Rzy {F' & D KF'&B$J&V F|$'(aF'&Vl(a$'&h(J X &V"nFJ Z8&H}V&B$' ~KM(IF)W D KFJ D D FJ (aF D K$'&V!#(aF D K
IV&B$ F( D K !_ !#" M $F' &Y $ D o D K(!\ K " D JIV KF'&V"x (KF D F D &VBs(aF D TF' &e$'&)(J K^F'(aF D K X (K_YF' &V $'&BF D I)(!x(a8&$'UF(a$JF D K o A D F' D F D F' (JJ &V",FJ(aF pa| !#" A ! o TM]$' F' &\Z[&Vo D K K D K o \G &$'&VM $'&WF' &$'&)(" &B$ D H(!kJ M]$'&V& F' e$'&VX FJ$ D IBF|(aFJF'&VKF D KnF' eF' &HIV(J&U qpa X s &K5 X Z[&B$'% MuFJ &&V5 (aF D K ) z ((K "`) aZJ !k"Z8&$J&B!](IV&B"Z5_ z (U(K " aZ
!" #$%# & ('*),+- ./0102'3 4 5768 9 &:;<$ & <=?>:@A&BDC EGFIHJFIKMLNFIOQPSRUTSVWVYX \Z [ ]^"_%]a`b_&c^"de ]gfh^"`i_ j k&lnmo^"` pJq prspt^&urwvx^zy|{ ]apr|q ^"` pc1]a}&~cJ q ` }mo^"`&prscJ~"q ` ]gb~"]acJ~}"]c]av2~cIb2
Yp vq rs[n&`q rs]pr|~rs]~` n~"mor|q ^"`3p1]arspJsq[ ^|v(rs^ `*rs[ ]d%]aprUcI~`^&fnq a]g2p1r|~r|q ^"`~ci~`nsqq[ ^|vrs^Q ` rs[ ] d%]apr_&{ c]2pr|~r|q ^"`~c_%^&eqmogjtZ\[ ]e~rsrs]ac*_ c^"de ]gfq p[ ~cJ jZ\[ ]ac1]gu^"c]gNq `b%q ]av^&u rs[&]3mo{ cc]a`Wr pr|~rs]3^&u rs[ ]nW` ^|ve ]g"}"]zq `rs[&]~c]g~G^&u z ¡IYrs[&]ac]q pMeq rsr|e ]M[ ^"_%]Grs^i&`8~}"^W^%¢~"e }"^"cJq rs[f£rs^ mo^&f2_ {&rs] rs[ ]d%]apr_ { c1]pr|~r|q ^"`~c _%^&eqmogj Z\[ q pi[ ~cJ` ]appc]ap1{e rbu^&ee ^Jvpu¤c^&f¥rs[ ] _~_%]acd¦¨§ qe~cz~`A©cJ~ppIxª«&¬®h¯°²±a³J´¶µD±a·I´A¸"¹N º"º"» º"» l" ¼"½"½k&¾v[ ]ac1]Gq rv~pDp1[ ^|v`rs[ ~rrs[ ]¿t~"fqe rs^"`q~`ÁÀ\mae ]
c1^"de ]gfu¤^"cQ~2}"cJ~_ [vq rs[£` ^]ap ma~` d%]Dc]g{ mo]g rs^b~mo^"` prscJ~"q ` ]g ~"]acJ~}"]c]av~cJ{ `qm[ ~"q `¶z2
¶vq rs[ÂÃpr|~rs]apQ~`Q` ^f2^"c]*rs[~`¶ ~"mor|q ^"` pGq `b]g~"m[pr|~rs]gj Ä `§]gq `Wd%]ac}&Dª«&¬®¯°²±a³J´-µD±a·I´¨Å"ÆN*¼ »"Ç 1¼k Ç º"Ç"Ç"Ç q rv~pp[ ^Jv`rs[~r ` q ` }¶rs[ ]idÈ]ap1rG_ { c] pr|~r|q ^"`~c1G_È^&eqmoq `~ &q pJmo^"{ `Wrs]gn2
Áq p*~`z[~cI2_ c^"de ]gfjZ\[ q pt~"e p^3qf2_ eq ]apÉrs[ ]Qz[~cJ"` ]app c]ap{ e ru¤^"c{ `qmÊ[~"q `Ë~s"]acJ~}"]bc1]av2~cJAz2
pJj Ä `"]a]g u^"cb~¶ q pJmo^"{&`¦rs]gAz2
È^"` ]8fn~¨mo^"`&pJq]acb~` ~s"]acI~}"]c]av2~cI¢z2
vq rs[¢rs[ ]MpJ~"f2]3pr|~rs]8~`~"mor|q ^"`Ap1]arspJvq rs[¢rs[&]pJ~"f2]Mc]av~cJÁu{ ` mor|q ^"` pJ~` vq rs[nrs[&]rscJ~` pJq r|q ^"`3_ c^"d~dqeq r|q ]ap̦Í&®Î"Ï ÐIÑaÒWÉ -ÓYÌÈ®ÎÏ ÐIÑaÒW"Ôbs¼ÕÓsÖs×ÊØ Ù"vD[ ]ac]Îq prs[ ]Qq `q r|q~"e"p1r|~rs]gjZ\[ ]a` rs[ ] ~s"]acJ~}"]Gc]av~cJp*_%]acD{ ` q rr|qf2]u^"c*rs[q pG2
Ú~c]]gyÛ{~"e&rs^Mrs[ ]Qrs^"r|~"e\ q pJmo^"{&`¦rs]gc]av~cJpGu^"c*rs[ ] ^"cJq }&q `~"e z*
ÚfD{e r|q _eq ]g3dWs¼DÕ8Ósj Z\[q pGmo^"` prsc1{mor|q ^"`¶c]g{mo]ap~3mo^"` p1rscJ~"q ` ]gz q pJmo^"{ `Wrs]gz2
Árs^ ~nmo^"` p1rscJ~"q ` ]gb~"]acJ~}"]c1]av2~cJG{ `qm[ ~"q `Áz*
%j Ü\Ý"ÞnÞßà áâ «&¬®±aã-äæåÂç ß"è ± à ±é ± áà äg±a³ è
¼
