Stochastic Mechanics Random Media
Signal Processing and Image Synthesis
Applications of Mathematics Stochastic Modelling and Applied Probability
Mathematical Economics Stochastic Optimization
and Finance Stochastic Control
Edited by
Advisory Board
Springer New York Berlin Heidelberg Barcelona
Hong Kong London Milan Paris Singapore
Tokyo
43 I. Karatzas M. Yor
P. Brémaud E. Carlen W. Fleming D. Geman G. Grimmett G. Papanicolaou J. Scheinkman
Applications of Mathematics 1 Flerning/Rishel, Deterministic and Stochastic Optimal Control (1975) 2 Marchuk, Methods of Numerical Mathematics, Second Ed. (1982) 3 Balalcrishnan, Applied Functional Analysis, Second Ed. (1981) 4 Borovkov, Stochastic Processes in Queueing Theory (1976) 5 Liptser/Shiryayev, Statistics of Random Processes I: General Theory (1977) 6 Liptser/Shiryayev, Statistics of Random Processes II: Applications (1978) 7 Vorob'ev, Game Theory: Lectures for Economists and Systems Scientists
(1977) 8 Shiryayev, Optimal Stopping Rules (1978) 9 Ibragimov/Rozanov, Gaussian Random Processes (1978) 10 Wonham, Linear Multivariable Control: A Geometric Approach, Third Ed. (1985) 11 Hida, Brownian Motion (1980) 12 Hestenes, Conjugate Direction Methods in Optimization (1980) 13 ICallianpur, Stochastic Filtering Theory (1980) 14 Krylov, Controlled Diffusion Processes (1980) 15 Prabhu, Stochastic Storage Processes: Queues, Insurance Risk, Dams, and Data Communication, Second Ed. (1998) 16 Ibragimov/Has'ininskii, Statistical Estimation: Asymptotic Theory (1981) 17 Cesari, Optimization: Theory and Applications (1982)
18 19 20 21 22
Elliott, Stochastic Calculus and Applications (1982) Marchuk/Shaidourov, Difference Methods and Their Extrapolations (1983) Hijab, Stabilization of Control Systems (1986) Protter, Stochastic Integration and Differential Equations (1990) Benveniste/M6tivier/Priouret, Adaptive Algorithm and Stochastic
Approximations (1990) 23 Kloeden/Platen, Numerical Solution of Stochastic Differential Equations
(1992) 24 Kushner/Dupuis, Numerical Methods for Stochastic Control Problems in Continuous Time (1992) 25 Fleming/Soner, Controlled Markov Processes and Viscosity Solutions (1993) 26 Baccelli/Brémaud, Elements of Queueing Theory (1994) 27 Winkler, Image Analysis, Random Fields, and Dynamic Monte Carlo Methods: An Introduction to Mathematical Aspects (1994) 28 Kalpazidou, Cyde Representations of Markov Processes (1995) 29 Elliott/Aggoun/Moore, Hidden Markov Models: Esti mation and Control (1995) 30 HemAndez Lenna/Lasserre, Discrete-Time Markov Control Processes: Basic Optimality Criteria (1996) 31 Devroye/Gydrfi/Lugosi, A Probabilistic Theory of Pattern Recognition (1996) 32 Maitra/Sudderth, Discrete Gambling and Stochastic Games (1996) -
(continued after index)
Jiongmin Yong
Xun Yu Zhou
Stochastic Controls Hamiltonian Systems and HJB Equations
Springer
Jiongmin Yong Department of Mathematics Fudan University Shanghai 200433, China
Xun Yu Zhou Department of Systems Engineering and Engineering Management The Chinese University of Hong Kong Shatin, Hong Kong
Managing Editors
I. Karatzas Departments of Mathematics and Statistics Columbia University New York, NY 10027, USA M. Yor CNRS, Laboratoire de Probabilités Université Pierre et Marie Curie 4, Place Jussieu, Tour 56 F-75252 Paris Cedex 05, France
Mathematics Subject Classification (1991): 93E20, 49K45, 49L20, 49L25, 60H30, 90A09, 70H20, 35C15
Library of Congress Cataloging-in-Publication Data Yong, J. (Jiongmin), 1958– Stochastic controls : Hamiltonian systems and HJB equations / Jionginin Yong, Xun Yu Zhou. p. cm. — (Applications of mathematics ; 43) Includes bibliographical references (p. – ) and index. ISBN 0-387-98723-1 (alk. paper) 1. Stochastic control theory. 2. Mathematical optimization. 3. Hamiltonian systems. 4. Hamilton–Jacobi equations. I. Zhou, Xun Yu. 11. Title. III. Series. QA402.37.Y66 1999 629.8'312—dc21
98-55411
Printed on acid-free paper. © 1999 Springer-Verlag New York, Inc. All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York. NY 10010, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computei software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use of general descriptive names, trade names, trademarks, etc., in this publication, even if the former are not especially identified, is not to be taken as a sign that such names, as understood b) the Trade Marks and Merchandise Marks Act, may accordingly be used freely by anyone.
Production managed by Allan Abrams; manufacturing supervised by Thomas King. Photocomposed copy prepared from the authors' UTEX files. Printed and bound by Edwards Brothers, Inc., Ann Arbor, MI. Printed in the United States of America. 987654321
ISBN 0-387-98723-1 Springer-Verlag New York Berlin Heidelberg SPIN 10707573
WP-Aff1 Wctki JJ
-111:
POW ttA
To Our Parents Wenyao Yong and Xiangxia Chen Qingji Zhou and Caizhen Shen
Preface
As is well known, Pontryagin's maximum principle and Bellman's dynamic programming are the two principal and most commonly used approaches in solving stochastic optimal control problems.* An interesting phenomenon one can observe from the literature is that these two approaches have been developed separately and independently. Since both methods are used to investigate the same problems, a natural question one will ask is the following: (Q) What is the relationship between the maximum principle and dynamic programming in stochastic optimal controls? There did exist some researches (prior to the 1980s) on the relationship between these two. Nevertheless, the results usually were stated in heuristic terms and proved under rather restrictive assumptions, which were not satisfied in most cases. In the statement of a Pontryagin-type maximum principle there is an adjoint equation, which is an ordinary differential equation (ODE) in the (finite-dimensional) deterministic case and a stochastic differential equation (SDE) in the stochastic case. The system consisting of the adjoint equation, the original state equation, and the maximum condition is referred to as an (extended) Hamiltonian system. On the other hand, in Bellman's dynamic programming, there is a partial differential equation (PDE), of first order in the (finite-dimensional) deterministic case and of second order in the stochastic case. This is known as a Hamilton-Jacobi-Bellman (HJB) equation. This leads to the following question, which is essentially a rephrase of Question (Q):
(Q') What is the relationship between Hamiltonian systems and HJB equations? Or, even more generally,
(Q") What is the relationship between ODEs/SDEs and PDEs? Once the question is asked this way, one will immediately realize that similar questions have already been or are being addressed and studied in other fields. Let us briefly recall them below. Analytic Mechanics. Using Hamilton's principle and Legendre's transformation, one can describe dynamics of a system of particles by a * Here, by a stochastic optimal control problem we mean a completely observed control problem with a state equation of the Itô type and with a cost functional of the Bolza type; see Chapter 2 for details. When the diffusion coefficient is identically zero, and the controls are restricted to deterministic functions, the problem is reduced to a deterministic optimal control oroblem.
viii
Preface
family of ODEs called Hamilton's canonical system or the Hamiltonian system. On the other hand, by introducing Hamilton's principal function, one can describe the particle system by a PDE called the Hamilton-Jacobi (HJ) equation. These two ways are in fact equivalent in the sense that the solutions of the canonical system can be represented by that of the HJ equation, and vice versa. One easily sees a strong analogy between optimal control and analytic mechanics. This is not surprising, however, since the classical calculus of variations, which is the foundation of analytic mechanics, is indeed the origin of optimal control theory. Partial Differential Equations. There is a classical method of characteristics in solving PDEs. More specifically, for a first-order PDE, there is an associated family of ODEs for curves, called characteristic strips, by which the solutions to the PDE can be constructed. In the context of (deterministic) optimal controls, the Hamiltonian system involved in the maximum principle serves as the characteristics for the HJB equation involved in the dynamic programming. Stochastic Analysis. The stochastic version of the method of characteristics is the Feynman-Kac formula, which represents the solutions to a linear second-order parabolic or elliptic PDE by those to some SDEs. On the other hand, a reversed representation has been recently developed via the so-called four-step scheme, which represents the solutions to a coupled forward-backward SDE by those to a PDE. A deterministic version of this is closely related to the so-called invariant embedding, which was studied by Bellman-Kalaba-Wing. Economics. The key to understanding the economic interpretation of optimal control theory is the shadow price of a resource under consideration. The very definition of the shadow price originates from one of the relationships between the maximum principle and dynamic programming, namely, the shadow price (adjoint variable) is the rate of change of the performance measure (value function) with respect to the change of the resource (state variable). Finance. The celebrated Black-Scholes formula indeed gives nothing but a way of representing the option price (which is the solution to a backward SDE) by the solution to the Black-Scholes equation (which is a parabolic PDE). Interestingly enough, all the relationships described above can be captured by the following simple, generic mathematical formula: y(t) = 0(t,x(t)), where (x(t), y(t)) satisfies some ODE/SDE and O satisfies some PDE. For example, in the relationship between the maximum principle and dynamic programming, (x(t), y(t)) is the solution to the Hamiltonian system and [—O] is the gradient in the spatial variable of the value function (which is the solution to the HJB equation). In the Black-Scholes model, y(t) is the
Preface
ix
option price, x(t) is the underlying stock price, and 0 is the solution to the Black Scholes PDE. -
Before studying Question (Q), one has first to resolve the following two problems: (P1) What is a general stochastic maximum principle if the diffusion depends on the control and the control domain is not necessarily convex? This problem has been investigated since the 1960s. However, almost all the results prior to 1980 assume that the diffusion term does not depend on the control variable and/or the diffusion depends on the control but the control domain is convex. Under these assumptions, the statements of the maximum principle and their proofs are very much parallel to those of the deterministic case. One does not see much essential difference between stochastic and deterministic systems from those results. The stochastic maximum principle for systems with control-dependent diffusion coefficients and possibly nonconvex control domains had long been an outstanding open problem until 1988. (P2) How is one to deal with the inherent nonsmoothness when studying the relationship between the maximum principle and dynamic programming? The relationship unavoidably involves the derivatives of the value functions, which as is well known could be nonsmooth in even very simple cases. During 1987-1989, a group led by Xunjing Li at the Institute of Mathematics, Fudan University, including Ying Hu, Jin Ma, Shige Peng, and the two authors of the present book, was studying those problems and related issues in their weekly seminars. They insisted on tackling the controldependent diffusion cases, and this insistence was based on the following belief: Only when the controls/decisions could or would influence the scale of uncertainty (as is indeed the case in many practical systems, especially in the area of finance) do the stochastic problems differ from the deterministic ones. In the stimulating environment of the seminars, Problems (P1) and (P2) were solved almost at the same time in late 1988, based on the introduction of the so-called second-order adjoint equation. Specifically, Peng, then a postdoctoral fellow at Fudan, solved Problem (P1) by considering the quadratic terms in the Taylor expansion of the spike variation, via which he established a new form of maximum principle for stochastic optimal controls. On the other hand, Zhou (who was then a Ph.D. student at Fudan) found a powerful way for solving Problem (P2). By utilizing viscosity solution theory, he managed to disclose the relationship between the first-order (respectively, second-order) adjoint equations and the first-order (respectively, second-order) derivatives of the value functions. After 1989, members of the Fudan group went to different places in the world, but the research they carried out at Fudan formed the foundation of their further research. In particular, studies on nonlinear backward and forward-backward
Preface
SDEs by Pardoux-Peng, and Ma-Yong are natural extensions of those on the (linear) adjoint equations in the stochastic maximum principle. This fashionable theory soon became a notable topic among probabilists and control theorists, and found interesting applications in stochastic analysis, PDE theory, and mathematical finance. The remarkable work on the nonlinear Feynman-Kac formula (representing solutions to nonlinear PDEs by those to backward SDEs) by Peng and the four-step scheme (representing solutions to forward-backward SDEs by those to PDEs) by Ma-Protter-Yong once again remind us about their analogy in stochastic controls, namely, the relationship between stochastic Hamiltonian systems and HJB equations. On the other hand, (stochastic) verification theorems by Zhou and ZhouYong-Li by means of viscosity solutions are extensions of the relationship between the maximum principle and dynamic programming from open-loop controls to feedback controls. These verification theorems lead to optimal feedback synthesis without involving derivatives of the value functions. Finally, the recent work by Chen-Li-Zhou, Chen-Yong, and Chen-Zhou on stochastic linear quadratic (LQ) controls with indefinite control weighting matrices in costs demonstrates how fundamentally different it is when the control enters into the diffusion term. The LQ case also provides an important example where the maximum principle and dynamic programming are equivalent via the stochastic Riccati equation. The purpose of this book is to give a systematic and self-contained presentation of the work done by the Fudan group and related work done by others, with the core being the study on Question (Q) or (Q'). In other words, the theme of the book is to unify the maximum principle and dynamic programming, and to demonstrate that viscosity solution theory provides a nice framework to unify them. While the main context is in stochastic optimal controls, we try whenever possible to disclose some intrinsic relationship among ODEs, SDEs, and PDEs that may go beyond control theory. When writing the book, we paid every attention to the coherence and consistency of the materials presented, so that all the chapters are closely related to each other to support the central theme. In some sense, the idea of the whole book may be boiled down to the single formula y(t) = 0(t, x(t)), which was mentioned earlier. That said, we do not mean to trivialize things; rather we want to emphasize the common ground of seemingly different theories in different areas. In this perspective, the Black-Scholes formula, for instance, would not surprise a person who is familiar with mechanics or the Feynman-Kac formula. Let us now sketch the main contents of each chapter of the book. Chapter 1. Since the book is intended to be self-contained, some preliminary materials on stochastic calculus are presented. Specifically, this chapter collects notions and results in stochastic calculus scattered around in the literature that are related to stochastic controls. It also unifies terminology and notation (which may differ in different papers/books) that are
Preface
xi
to be used in later chapters. These materials are mainly for beginners (say, graduate students). They also serve as a quick reference for knowledgeable readers. Chapter 2. The stochastic optimal control problem is formulated and some examples of real applications are given. However, the chapter starts with the deterministic case. This practice of beginning with deterministic problems is carried out in Chapters 3-6 as well. The reasons for doing so are not only that the deterministic case itself may contain important and interesting results, but also that readers can see the essential difference between the deterministic and stochastic systems. In the formulation of stochastic control problems we introduce strong and weak formulations and emphasize the difference between the two, which is not usually spelled out explicitly in the literature. Stochastic control models other than the "standard" one studied in this book are also briefly discussed. This chapter finally provides a very extensive literature review ranging from the very origin of optimal control problems to all the models and applied examples presented in this chapter. Chapter 3. A stochastic Hamiltonian system is introduced that consists of two backward SDEs (adjoint equations) and one forward SDE (the original state equation) along with a maximum condition. The general stochastic maximum principle is then stated and proved. Cases with terminal state constraints and sufficiency of the maximum principle are discussed. Chapter 4. First, a stochastic version of Bellman's principle of optimality is proved by virtue of the weak formulation, based on which HJB equations are derived. The viscosity solution is introduced as the tool to handle the inherent nonsmoothness of the value functions. Some properties of the value functions and viscosity solutions of the HJB equations are then studied. It is emphasized that the time variable here plays a special role due to the nonanticipativeness of the underlying system. Finally, a simplified proof (compared with the existing ones) of the uniqueness of the viscosity solutions is presented. Notice that the verification technique involved in the dynamic programming is deferred to Chapter 5. Chapter 5. Classical Hamilton-Jacobi theory in mechanics is reviewed first to demonstrate the origin of the study of the relationship between the maximum principle and dynamic programming. The relationship for deterministic systems is investigated and is compared with the method of characteristics in PDE theory, the Feynman-Kac formula in probability theory, and the shadow price in economics. The relationship for stochastic systems is then studied. It starts with the case where the value function is smooth to give some insights, followed by a detailed analysis for the nonsmooth case. Finally, stochastic verification theorems workable for the nonsmooth situation are given, and the construction of optimal feedback controls is discussed. Chapter 6. This chapter investigates a special case of optimal control problems, namely, the linear quadratic optimal control problems (LQ prob-
xii
Preface
lems). They constitute an extremely important class of optimal control problems, and the solutions of LQ problems exhibit elegant properties due to their simple and nice structures. They also nicely exemplify the general theory developed in Chapters 3-5. In the chapter an LQ problem is first treated as an optimization problem in an infinite-dimensional space, and abstract results are obtained to give insights. Then linear optimal state feedback is established via the so-called stochastic Riccati equation. It is pointed out that both the maximum principle and dynamic programming can lead to the stochastic Riccati equation, by which one can see more clearly the relationship between the maximum principle and dynamic programming (actually, these two approaches are equivalent in the LQ case). We emphasize that the control weighting matrices in the cost are allowed to be indefinite in our formulation. Therefore, it is essentially different from the deterministic case. Stochastic Riccati equations are extensively studied for various cases. Finally, as an example, a mean-variance portfolio selection is solved by the LQ method developed. Chapter 7. This chapter presents the latest development on backward and forward-backward SDEs, with an emphasis on the relationship between nonlinear SDEs and nonlinear PDEs. Although the topics in this chapter go beyond the scope of stochastic controls, they originate from stochastic controls as mentioned earlier. The chapter begins with the original argument of Bismut for studying linear backward SDEs by using the martingale representation theorem. Then the existence and uniqueness of solutions to nonlinear backward SDEs are investigated for two types of time durations, finite deterministic horizon and random horizon, by virtue of two different methods. Feynman-Kac-type formulae with respect to both forward and backward SDEs are presented. Next, a kind of inverse of the Feynman-Kactype formulae, the so-called four-step scheme, which represents solutions to forward-backward SDEs by those to PDEs, is discussed. Solvability and nonsolvability of forward-backward SDEs are also analyzed. Finally, the Black-Scholes formula in option pricing is derived by the four-step scheme. The idea of writing such a book was around in late 1994 when JY was visiting XYZ in Hong Kong. While discussing the stochastic verification theorems, they realized that the series of works done by the Fudan group were rich enough for a book, and there should be a book as a systematic account of these results. The plan became firm with encouragement from Wendell Fleming (Brown), bannis Karatzas (Columbia), and Xunjing Li (Fudan). The authors are greatly indebted to Robert Elliott (Alberta), Wendell Fleming (Brown), Ulrich Haussmann (British Columbia), bannis Karatzas (Columbia), Thomas Kurtz (Wisconsin-Madison), Mete Soner (Princeton), and Michael Taksar (SUNY-Stony Brook), who substantially reviewed some or all chapters, which led to a much improved version. Michael Kohlmann (Konstanz) and Andrew Lim (CUHK) read carefully large portions of the manuscript and offered numerous helpful suggestions. During various stages in the prolonged, four-year course of
Preface
xiii
the project, many experts and friends have shown their concern and encouragement, conveyed their comments, or sent their research works for the book. Among them the following deserve special mention: Alain Bensoussan (CNES), Leonard Berkovitz (Purdue), Giuseppe Da Prato (Scuola Normale Superiore), Darrell Duffle (Stanford), Tyrone Duncan (Kansas), Fausto Gozzi (Scuola Normal Superiore), Suzanne Lenhart (Tennessee), Zhuangyi Liu (Minnesota-Duluth), John Moore (ANU), Makiko Nisio (Kobe), Bozenna Pasik-Duncan (Kansas), Thomas Seidman (MarylandBaltimore County), Hiroshi Tanaka (Keio), Wing Wong (CUHK), Jia-an Yan (Academia Sinica), George Yin (Wayne State), and Qing Zhang (Georgia). Especially, both authors would like to express appreciation to their long-time teachers and/or colleagues in the Fudan group: Shuping Chen (Zhejiang), Ying Hu (Rennes), Xunjing Li (Fudan), Jin Ma (Purdue), Shige Peng (Shandong), and Shanjian Tang (Fudan), whose elegant research work provided a rich source for this book. JY would like to acknowledge the partial support from the Natural Science Foundation of China, the Chinese Education Ministry Science Foundation, the National Outstanding Youth Foundation of China, and the Li Foundation at San Francisco, USA. In particular, with the financial support of a Research Fellowship of the Chinese University of Hong Kong (CUHK), JY visited XYZ at CUHK in the Spring of 1998 for half a year, which made it possible for the two authors to fully concentrate on finalizing the book. In the career of XYZ, he has been influenced enormously by three scholars: Xunjing Li (Fudan), Makiko Nisio (Kobe), and Hiroshi Tanaka (Keio), whom he has the privilege of having worked with for a substantial period of time, and he would like to take this opportunity to pay them his highest respect. Also, he would like to acknowledge the support from the Research Grant Council and Additional Funding for Industry Support of the Hong Kong Government, and the Mainline Research Scheme of CUHK. It has been a truly enjoyable experience to work with the staff at Springer-Verlag, especially the executive editor of statistics John Kimmel and the copyeditor David Kramer, whose helpful and professional services for the book are gratefully acknowledged. Last, but not least, both authors would like to thank their families for their long-lasting support and love.
JY, Shanghai XYZ, Hong Kong November 1998
Contents
Preface Notation Assumption Index Problem Index Chapter 1. Basic Stochastic Calculus 1. Probability 1.1. Probability spaces 1.2. Random variables 1.3. Conditional expectation 1.4. Convergence of probabilities 2. Stochastic Processes 2.1. General considerations 2.2. Brownian motions 3. Stopping Times 4. Martingales 5. Itô's Integral 5.1. Nondifferentiability of Brownian motion 5.2. Definition of Itô's integral and basic properties 5.3. Itô's formula 5.4. Martingale representation theorems 6. Stochastic Differential Equations 6.1. Strong solutions 6.2. Weak solutions 6.3. Linear SDEs 6.4. Other types of SDEs Chapter 2. Stochastic Optimal Control Problems 1. Introduction 2. Deterministic Cases Revisited 3. Examples of Stochastic Control Problems 3.1. Production planning 3.2. Investment vs. consumption 3.3. Reinsurance and dividend management 3.4. Technology diffusion 3.5. Queueing systems in heavy traffic 4. Formulations of Stochastic Optimal Control Problems 4.1. Strong formulation 4.2. Weak formulation 5. Existence of Optimal Controls 5.1. A deterministic result 5.2. Existence under strong formulation
vii xix xxi xxii 1
1 1 4 8 13 15 15 21 23 27 30 30 32 36 38 40 41 44 47 48 51
51 52 55 55 56 58 59 60 62 62 64 65 65 67
xvi
Contents
5.3. Existence under weak formulation 6. Reachable Sets of Stochastic Control Systems 6.1. Nonconvexity of the reachable sets 6.2. Noncloseness of the reachable sets 7. Other Stochastic Control Models 7.1. Random duration 7.2. Optimal stopping 7.3. Singular and impulse controls 7.4. Risk-sensitive controls 7.5. Ergodic controls 7.6. Partially observable systems 8. Historical Remarks Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems 1. Introduction 2. The Deterministic Case Revisited 3. Statement of the Stochastic Maximum Principle 3.1. Adjoint equations 3.2. The maximum principle and stochastic Hamiltonian systems 3.3. A worked-out example 4. A Proof of the Maximum Principle 4.1. A moment estimate 4.2. Taylor expansions 4.3. Duality analysis and completion of the proof 5. Sufficient Conditions of Optimality 6. Problems with State Constraints 6.1. Formulation of the problem and the maximum principle 6.2. Some preliminary lemmas 6.3. A proof of Theorem 6.1 7. Historical Remarks Chapter 4. Dynamic Programming and HJB Equations 1. Introduction 2. The Deterministic Case Revisited 3. The Stochastic Principle of Optimality and the HJB Equation 3.1. A stochastic framework for dynamic programming 3.2. Principle of optimality 3.3. The HJB equation 4. Other Properties of the Value Function 4.1. Continuous dependence on parameters
4.2. Semiconcavity 5. Viscosity Solutions 5.1. Definitions 5.2. Some properties
69 75 76 81 85 85 86 86 88 89 89 92 101
101 102 113 115 117 120 123 124 126 134 137 141 141 145 149 153 157
157 158 175 175 180 182 184 184 186 189 189 196
Contents
6. Uniqueness of Viscosity Solutions 6.1. A uniqueness theorem 6.2. Proofs of Lemmas 6.6 and 6.7 7. Historical Remarks Chapter 5. The Relationship Between the Maximum Principle and Dynamic Programming 1. Introduction 2. Classical Hamilton-Jacobi Theory 3. Relationship for Deterministic Systems 3.1. Adjoint variable and value function: Smooth case 3.2. Economic interpretation 3.3. Methods of characteristics and the Feynman-Kac formula 3.4. Adjoint variable and value function: Nonsmooth case 3.5. Verification theorems 4. Relationship for Stochastic Systems 4.1. Smooth case 4.2. Nonsmooth case: Differentials in the spatial variable 4.3. Nonsmooth case: Differentials in the time variable 5. Stochastic Verification Theorems 5.1. Smooth case 5.2. Nonsmooth case 6. Optimal Feedback Controls 7. Historical Remarks
xvii
198 198 208 212
217 217 219 227 229 231 232 235 241 247 250 255 263 268 268 269 275 278
Chapter 6. Linear Quadratic Optimal Control Problems
281
1. Introduction 2. The Deterministic LQ Problems Revisited 2.1. Formulation 2.2. A minimization problem of a quadratic functional 2.3. A linear Hamiltonian system 2.4. The Riccati equation and feedback optimal control 3. Formulation of Stochastic LQ Problems 3.1. Statement of the problems 3.2. Examples 4. Finiteness and Solvability 5. A Necessary Condition and a Hamiltonian System 6. Stochastic Riccati Equations 7. Global Solvability of Stochastic Riccati Equations 7.1. Existence: The standard case 7.2. Existence: The case C = 0, S = 0, and Q,G > 0 7.3. Existence: The one-dimensional case 8. A Mean-variance Portfolio Selection Problem 9. Historical Remarks
281 284 284 286 289 293 300 300 301 304 308 313 319 320 324 329 335 342
Contents
xviii
Chapter 7. Backward Stochastic Differential Equations
1. Introduction 2. Linear Backward Stochastic Differential Equations 3. Nonlinear Backward Stochastic Differential Equations 3.1. BSDEs in finite deterministic durations: Method of contraction mapping 3.2. BSDEs in random durations: Method of continuation 4. Feynman—Kac-Type Formulae 4.1. Representation via SDEs 4.2. Representation via BSDEs 5. Forward—Backward Stochastic Differential Equations 5.1. General formulation and nonsolvability 5.2. The four-step scheme, a heuristic derivation 5.3. Several solvable classes of FBSDEs 6. Option Pricing Problems 6.1. European call options and the Black—Scholes formula 6.2. Other options 7. Historical Remarks References
Index
345
345 347 354 354 360 372 372 377 381 382 383 387 392 392 396 398 401 433
Notation
The following notation is frequently used in the book. Rn — n-dimensional real Euclidean space. Rn x m
the set of all (n x m) real matrices.
S n — the set of all (n x n) symmetric matrices. ,57±' — the set of all (n x n) nonnegative definite matrices. §7+'
the set of all (n x n) positive definite matrices.
tr (A)
the trace of the square matrix A.
x T — the transpose of the vector (or matrix) x.
( • , • ) — inner product in some Hilbert space. Q — the set of all rational numbers. — the set of natural numbers. INI A
Lebesgue measure of the set N.
=—
Defined to be (see below).
—
A { 1, the indicator function of the set A: IA(X)= 0,
,± 2max{cp,0 ,
xEA, x
A.
ço - _ minfw,01.
}
a A bg minfa,131.
a V VA' maxla,b1,
2n — the set of all subsets of a (/) — the empty set. C([0, T]; Rn ) — the set of all continuous functions yo : [0, T]
Rn .
C( [0, co); Rn) — the set of all continuous functions ço : [0, oc)
Cb(U)
the set of all uniformly bounded, continuous functions on U.
LP (0, T; Rn ) — the set of Lebesgue measurable functions ço : [0, T] oT Iço(t)rdt
L
(0, T; Rn ) — the set of essentially bounded measurable functions
: [0,T] —■ Rn. (II, F, P)
probability space.
{Tt }t>0 — filtration. filtered probability space. (Q, 7", {F} >o, P) B(U) — the Borel cr-field generated by all the open sets in U.
a(e)= (T.) — the o--field generated by the random variable e. cr (A)— the smallest o--field containing the class A, V a Tc, a(UQ.
Ac,
IFtn suchtaf
Notations
XX
•
= Poe -1 — the probability measure induced by the random variable e. EX — the expectation of the random variable X.
ERX - EX)(Y - EY) T 1, Var X =A- Cov(X , X). E(X1G) — conditional expectation of X given Ç. • (fl; Rn ) — the set of Rn-valued G-measurable random variables X such that EIXIP < oc (p E [1, co)). L'6°(f2; — the set of bounded Rn-valued G-measurable Cov(X, Y)
random variables. LPF (0, T; Rn) — the set of all {T} t >0-adapted Rn-valued processes
X() such that E foT IX(t)rdt < oc. Lc.r° (0,T; Rn) — the set of {T} t >o-adapted Rn-valued essentially bounded processes. q(ft; C([0, T];Rn)) — the set of {Ft } t > 0-adapted Rn-valued continuous processes X(.) such that E supte [0,T] IX(t)I P < oc(/) E [ 1 1 c() )). M 2 [0, T] — the set of square-integrable martingales. M c2 [0, T] — the set of square-integrable continuous martingales. OC T] — the set of square-integrable local martingales. M2,1 A4z,10C[0, T] the set of square-integrable continuous local martingales. Wn [0, T]
C([0, T]; Rn ),
Wn c([o, co) ; Rn).
• — the set of Borel
cylinders in W'1 [0, s]. C — the set of Borel cylinders in Wn. W [O, T]
t) I ((.) E W n [O,T]l,
Bt (Wn [0 , 7])
t E [0, 7].
B(W[0,T]), t E [0,T].
ns>t 83 (Wn [0 , T]), t E [0, T). (e) E Wn }, t >0.
6' 8+(W'1 [0, T]) = g{((.
t)
Bt(W n )e-t- B(WIt'),
BtA-(Wn) 11
t > 0.
ns> , Bs (wn),
t? 0.
A'1(U) _ the set of all {B t (Wn [0 , 7])1 t> 0 -progressively measurable processes 77 : [0, x w'1[0, U. A'1 (U) — the set of all friti-(Wnit>o-progressively measurable processes 77 : [0, cc) x U. V[0, T] — the set of all measurable functions u: [0, T] -■ U. 1.1[0,T] — the set of all {.Ft } t >o-adapted processes u: [0,T] x -> U. Vad[O,T] — the set of deterministic admissible controls. /1 d [0,T] — the set of (stochastic) strong admissible controls. U[0,T] — the set of (stochastic) weak admissible controls.
Assumption Index
(B) 355 (B)' 363 (B)" 377 (B)" 380 (D1) 102, 159, 228 (D2)-(D3) 102 (D2)' 159, 228 228 (D3)' (D4) 106 (DE1)-(DE4) 66 (F) 373 (F)' 376 (FB1)-(FB3) 387 (FB2)' 390 (FB2)" 391 42 (H) (H)' 44 (H1)-(H2) 68 (L1) 301 (L2) 319 (RC) 49 114 (S0)-(S2) 114, 249 (S3) (Si)'-(S2)' 177, 248 187 (S3)' (S4) 138 143 (S5) (SE1)-(SE4) 69 (W) 50
Problem Index
Problem (D) 54, 103, 158 Problem (D sy ) 159, 228 Problem (S) 115, 176 Problem (S,v ) 177, 248 Problem (SC) 143 Problem (SL) 68 Problem (SS) 63 Problem (WS) 64 Problem (DC) 76 Problem (DT) 76 Problem (DLQ) 285 Problem (SLQ) 301
Chapter 1
Basic Stochastic Calculus
Stochastic calculus serves as a fundamental tool throughout this book. This chapter is meant to be a convenient "User's Guide" on stochastic calculus for use in the subsequent chapters. Specifically, it collects the definitions and results in stochastic calculus scattered around in the literature that are related to stochastic controls. It also unifies terminology and notation (which may differ in different papers/books) that are to be used in later chapters. Proofs of the results presented in this chapter are either given (which is the case when we think that the proof is important in understanding the subsequent material and/or when there is no immediate reference available) or else referred to standard and easily accessible books. Knowledgeable readers may skip this chapter or regard it as a quick reference. 1. Probability
In this section, some introductory probability will be briefly reviewed. The readers are assumed to have basic knowledge of real analysis. 1.1. Probability spaces Definition 1.1. Let a set It be nonempty, and let ..F C 2° (2° is the set of all subsets in SI), called a class, be nonempty. We call
(i) a 7r-system if A,B E
An B E
(ii) a )-system if {
E .7"; A,BEF, ACB B\AE.F; E .F, A,i = 1,2,... AE
(iii) a cr-field if
Il E F; A,BEF
B\AEF; 00
Ai E
= 1,2,...
U Ai E F.
If .F and g are both a-fields on n and g c .F, then g is called a subcr-field of F. It is easy to see that F is a a-field if and only if it is both a 7r-system and a A-system. In what follows, for any class A C 20 , we let u(A) be the smallest a-field containing A, called the u-field generated by A.
2
Chapter 1. Basic Stochastic Calculus
The following result is called a monotone class theorem. Lemma 1.2. Let ACFC 20 . Suppose A is a R--system and .7. is a A-system. Then a(A) C F. Proof. Let
aarl fg 2Aig is a A-system c F. Then
a is the smallest A-system containing A. Set fra {B E VA E A, AnB a} ç a. E
Clearly,
3- is also a A-system,
must have
a=
and since A is a 7r-system, A Now, let us define
{B E
VA E
a, AnB
E}
,
C
Thus, we
a. a
a
Again, .-7" is a A-system, and by = one has A C F. Hence, = F. Consequently_L for any A, B E g, regarding B EF we have AnB E Ç This is a a-field (which contains implies that g is also a r-system. Therefore, , a(A) F. A). Then
a
a,
,
.
Let {F} be a family of a-fields on ft. Define V Fa
cr(U TG ) a
It is easy to show that V and A a .FQ, are both a-fields and that they are the smallest a-field containing all F and the largest a-field contained in all respectively.
Let 1 be a nonempty set and ..F a a-field on Q. Then (Il, F) is called a measurable space. A point w E 51 is called a sample. A map P : F —> [0,1] is called a probability measure on (51,F) if P(0) =-- 0, A, E F A ,
(1.3)
I.
P(51) = 1;
nA2=.0,
1, 2, ... ,
00
00
i=i
z=i
j,
p( u Az ) = EP(Ai).
The triple (St, F, P) is called a probability space. Any A E F is called an event, and P(A) represents the probability of the event A. Two events A and B are said to be independent if P(A n B) = P(A)P(B). An event A is said to be independent of a a-field .F if A is independent of any B E F.
1. Probability
3
Two a-fields .F and g are said to be independent if any event A G .F is independent of Ç. If an event A is such that P(A) -= 1, then we may alternatively denote this by A
(1.4)
holds , P-a.s.
A set/event A E .F is called a P null set/event if P(A) = O. A probability space ( 12, F , P) is said to be complete if for any P-null set A G .7-, one has B E .F whenever B C A (thus, it is necessary that B is also a P-null set). We present a simple example below. -
Example 1.3. Let S2 = io, 1], .F the set of all Lebesgue measurable sets in [0,1], and P the Lebesgue measure on [0,1]. Then one can show that ( 12, F P) is complete. Let x (.) : —> IR be a continuous function and define E x(w) > 01. The probability of this event is the event A = ,
P(A)
dP(w).
For any given probability space (RI', P), we define
(1.5)
Jf={B c1
A E .F, P(A) = 0, B C A}
there exist A, B E .F such that P(B) and .? a TV.V. Then for any  E 0 and  \ A C B. In such a case, we define P(Â) = P(A). This extends P to F. Clearly, (f1,?, P) is a complete probability space. Any probability space can be made complete by the above augmentation procedure. In the sequel we shall assume that any probability space (S2,Y,P) is complete. Let ( (2,F) and (12', Y.') be two measurable spaces and f : —■ SI' be a map. The map f is said to be I .7.1 measurable or simply measurable if f1(y) c F. In particular, if (12', = (Rm ,B(lRin )), then f is said to be measurable (see right below for the definition of 8(11:n )). -
-
Definition 1.4. We have the following definitions.
(i)
A measurable space (12,F) is said to be Borel isomorphic to another measurable space (Sr, F') if there exists a bijection f : St —> S2' such that both f and f -1 are measurable.
(ii) If 12 is a topological space, then the smallest a-field containing all open sets of SI is called the Borel a field of SI, denoted by B( 12). -
(iii) A separable complete metric space is called a Polish space. (iv) A measurable space (S2, .F) is said to be standard if it is Borel isomorphic to one of the following: (( 1, n), 8(( 1, n))), (N, B(N)), and (M, 8(M)), where (1,n) 2,...,n}, N a{1,2,...}, M a { 0,1} 1'1 {w= ...) w 0, 1 } with the discrete topology on (1, n) and N, and the product topology on M, respectively.
a{l,
.
Chapter 1. Basic Stochastic Calculus
4
(v) A probability space (5-2,.F, P) is said to be standard if it is complete, and (Q, .7.) is a standard measurable space. (vi) A measurable space (Q, .F) is said to be countably determined (or simply, F is countably determined) if there exists a countable set To C ..F such that any two probability measures that agree on To must coincide (on .7.). Proposition 1.5. Let Q be a Polish space and .F = B(Q). Then: (i) (Q, F) is standard, and it is countably determined.
n
Then (Q', F') is (ii) For any D' E F let .F' =- Q' r) -A{D' AI A E standard. (iii) (Q, ,T) is Borel isomorphic to (R, 8 (R)) if SI is uncountable. ,
See Parthasarathy [1, pp. 133-134] and Ikeda-Watanabe [1, p. 13] for a proof and descriptions.
1.2. Random variables
Let (Q, .7') and (Q',.r) be two measurable spaces and X : Q —+ Q' an F/P-measurable map. We call X an Y/ 11 -random variable, or simply a random variable if there would be no confusion. It is called an F-random variable when (ST, = (Rm. , B(Rm )). Note that the notion of random variable is defined without probability measures. For a random variable X : Q —> Q', X'(..P) is a sub-a--field of F which is called the a-field generated by X, denoted by cr(X). This is the smallest a-field in Q under which X is measurable. Also, if (X0, 0 E el is a family of random variables ,
from D to Q', then we denote by cr(X0, 0 E 19) 0E8
the smallest sub-a-field of F under which all X9 (0 E 0) are measurable. Let X, Y : D Q' be two random variables and G a a-field on Q. Then X is said to be independent of G if a(X) is independent of G, and X is said to be independent of Y if a(X) and a(Y) are independent. Now we look at some measurability properties for random variables. The following result is closely related to Lemma 1.2. Lemma 1.6. Let A be a it-system on Q. Let 7-1 be a linear space of functions from Q to R such that 1 E 11;
(1.6)
IA E 7-i,
E71, 0 <
VA E A;
(p, (p is finite
E
{
Then contains all a(A)-measurable functions from Q to R. Proof. Set
= {A C Q IA E H}.
1. Probability
5
Then .7' is a )t-system containing A. Thus, by Lemma 1.2, a(A) C F. Now, for any a(A)-measurable function yo : R, we set z /[3 2- , <,,o(w)+<(3 +02- , ]•
c•P 3 >o
Clearly, (p, E N and 0 < cp, yo±. Hence, by our assumption, cp+ E N. Similarly, cp - E H. Thus, cp E H, proving our conclusion. 0 The following result gives a representation of one random variable in terms of another, under some measurability conditions.
Theorem 1.7. Let (SI, ..T) and (ST, 11 ) be two measurable spaces, and let (U,d) be a Polish space. Let e: -> II' and cp : SZ U be two random variables. Then cp is o(e)-measurable, i.e., (1.7)
40-1 (B(U)) g e -1 (7 ),
if and only if there exists a measurable map 77 :
cp(w) -= ii(e(w)),
(1.8)
U such that
VW E a
Proof. We need only to prove the necessity. First, we assume that U = R. For this case, set 'H afige(•))
U, measurable}.
:
Then H is a linear space, and 1 E H. Also, if A E a(e) e-1 (P), then for some B E U is , IA(S) = IB(e(•)) E N. Now, suppose 1/2 : measurable and qz (e(•)) E H such that 0 < tiz (e(•))1' ((•), which is finite. Set A = {co l E Then A E
supni (w') < oo}.
and e(12) C A. Define sup
r G A, E
\ A.
Clearly, 77 : SY -> U is measurable and ((•) = ii(e()). Thus, ((•) E N. By Lemma 1.6, H contains all o(e)-measurable random variables, in particular, E N, which leads to (1.8). This proves our conclusion for the case U = Now let (U, d) be an uncountable Polish space. By Proposition 1.5(iii) and Definition 1.4, there exists a bijection f : U -+ IR such that (B(U)) = B(R). Consider the map (7 = f : -> lit, which satisfies Ck5-1 (B(R)) = 49-1 Thus, there exists an 77 :
° .f -1 (8( 1R)) =
(MU)) g
-> 1R such that ( w) = ii(e(w)),
Vw E 9.
Chapter 1. Basic Stochastic Calculus
6
By taking 7/ =f -1 o we obtain the desired result. Finally, if (U, d) is countable or finite, we can prove the result replacing R in the above by IN or ( 1,n). 1:1 Next, let (0,T, P) be a probability space, (Sr, .r) a measurable space, and X : —> S2' a random variable. Then X induces a probability measure Px on (0', Y .') as follows:
(1.9)
Px (A')
P o X -1 (A') P(X -1 (A')) VA' E .F'.
We call Px the distribution of the random variable X. In the case where = (Rm B(R m )), Px can be uniquely determined by the following function: ('
,-r)
(1.10)
F(x) - E
E lI X i (w) < xi , 1 <j < ml.
We call F(x) the distribution function of X. Since X -=- (X1, , )(in ), we also call F(x) the joint distribution function of the (scalar-valued) random variables X1 , , X. Clearly, F(x) is nonnegative, nondecreasing in each variable x, E R, and
lim F(x) = 1. .,-0.
lim F(x) = 0,
Vi
3i
Further, if Px is absolutely continuous with respect to the Lebesgue measure, i.e., for any N E B(Rm ) with I NI = 0 (INI is the Lebesgue measure of N), one has P x(N) = 0, then by the Radon-NikodSrm theorem, there exists a (nonnegative) function f(S) E L 1 (1F1m ) such that
(1.12)
Px(A) = f f(x)dx,
VA E B(Rm).
A
In particular,
(1.13)
xi
f xm
F(x) =(f -03
-00
f Ci, • • • ,Cm)dei • • •
dem .
The function f(x) is called the density of the random variable X. special case, if 1(x) is of the form
(1.14)
f (x) = [(27)mdetC] 1 /2 e - i(x-)) T c -1 (x-x),
As a
G Rm,
where A E Ft.m , C E Rm x m, and C T = C > 0, then we say that X has a normal distribution (denoted by N(A, C)) with parameter (A, C). (Later on we will see that A is the mean and C is the covariance matrix of the normal distribution.) We also call N(A, C) the Gaussian distribution and X Gaussian. We point out that to have a particular type of random variable, the underlying probability space (n, .F) has to have certain particular properties. For example, if S/ is a finite or countable set, then (f1, F) is not rich enough to accommodate a normal distribution.
1. Probability
7
A sequence of random variables {X, } is called j. i. d. if they are independent and identically distributed (i.e., their distribution functions coincide). Now, let X: (SI, F, P) IRm be a random variable with the distribution function F(x). Suppose the following integral exists:
(1.15)
fi X (w)dP(w) f
m xdF(x)
EX.
Then we say that X has the mean EX. We also call EX the (mathematical) expectation of X. Note that a given random variable does not necessarily have a mean. We may also define the mean over a set A
(1.16)
E(X; A) g f X (w)dP(w) = f x dF(x) E(X IA)• A )C (A)
Let L.17. (SI; Rm ) L 1 (5.2, F, P; be the set of all random variables X, so that IX I has means (over the whole space SI). This is a Banach space with the norm )
(1.17)
EIXI =
L,x (u.,) ,dp (u)> =
Lmi xidF(x ) .
Next, let g : 1171 IR k be a Borel measurable function. Then, for any random variable X : Km , g(X) : S2 Rk is also a random variable. Further, if g(X) has a mean, then
(1.18)
Eg(X) = fo g(X(co))dP(co) =
km
g(x)dF(x).
The case g(x) = IxIP (p > 1) is of particular interest. If a random variable X is such that IXIP E L(1; R), then
(1.19)
EIXIP = f IX(w)IPdP(w).
Let LPF(0; 1( m ) LP (1l, F, P; 1(m) be the set of all random variables X : Rm with IXIP E L(2; R). This is again a Banach space with the norm
(1-20)
(EIX1 P
) 1/P
=
f
1/p IX(L4.7)1 0 dP(W))
.
In particular, if p =- 2, then L2F ( 1 ; 1(m ) is a Hilbert space with the inner product
(1.21)
( X, Y ) L 2,( -2, Ftm )
( X (w),Y (w)) dP(w) E(X T Y).
Also, let L7(12; Hui ) L'( 1 ,T,P;11tm ) be the set of all essentially bounded F-measurable random variables. Any X c 4.(11; 1(m ) is called an LP-random variable. From the Holder inequality, we have LPF (Si; 1(m) C
Chapter 1. Basic Stochastic Calculus
8
LI-(Q; R"') for any p > q > 1. In particular, L2F (12; IRm ) C Thus, for any X,Y E L2y.(12; R m ), EX and EY exist and the following is well-defined: (1.22)
Coy (X, Y) ERX — EX)(Y — EY)
11,
which is called the covariance of the L2-random variables X and Y. In particular,
(1.23)
Var X 'a Coy (X, X) = ERX — EX)(X — EX) 11
is called the variance of the L2-random variable X. As an example, if X is Gaussian with the distribution N(A,C), then
Var X = C.
EX = A,
Finally, for any am-valued random variable X (which is not necessarily in Ly l (12; Rm )), the following is always well-defined:
(1.24)
cpx(e) EeiC T x =
Ve E
x( w ) dP(w), o
We call wx(e) the characteristic function of X. It is known that a random variable X is Gaussian with the distribution N(.\, C) if and only if
(1.25)
e E Rm .
ÇoX(e) =
1.3. Conditional expectation Let X E L.1r (1l;Rm ) and let G be a sub-cr-field of F. Define a function G Rm as follows:
(1.26)
p(A) E(X; A) = f X(w)dP(w),
VA E G.
A
Then 1.1 is a vector-valued measure on G with a bounded total variation
hill
'al IX
(w)IdP(w) EIXI.
Moreover, tt is absolutely continuous with respect to Pig, the restriction of P on G. Thus, by the Radon-Nikod3'rm theorem, there exists a unique f E .q(ft;Rm ) E 0,11 g ;IFtm ) (called the Radon-Niko4m derivative of ti with respect to Pi g ) such that
(1.27)
P (A )
fA f(w)dPlg(w) E
f (w)dP(w),
VA E G.
Here, note that P is an extension of Pig. The function f is called the conditional expectation of X given G, denoted by E(XIG). Using this notation, we may rewrite (1.27) as follows:
(1.28)
fA
X(w)dP(w) =E(XIG)(4.0)dP(w), A
VA E G,
1. Probability
9
or VA c Ç.
E(X; A) = E(E(X1g); A),
(1.29)
E(Xog) to be the unique g-random
Indeed, we can alternatively define variable satisfying (1.29).
Let us collect some basic properties of the conditional expectation. Proposition 1.8. Let
g
be a sub-a-field of .F. Then
: 4(0; Rm. ) ■ L,(S2;Rni) is linear and bounded. (ii) E(alg) = a, P-a.s. , Va E R. (i) Map E (. 1g)
—
(iii) If X,Y E L .17 (2; It) with X > Y, then
E(Xg) >
(1.30)
P-a.s.
In particular,
(1.31)
X > 0, P-a.s. =
E(Xog) > 0, P-a.s.
(iv) Let X G L.1F (S2; Rm ), Y E L(1;Rm) and Z E 4-(51;11=e) with XZ T ,YZ T E 4-42;R"" k ). Then E(YZT 1g) = yE(zIg)T,
(1.32)
P-a.s.
In particular,
( 1.33 )
{E(Epog)z T ig) = E(xig)E(z1g)T ,
wig)
= Y,
P-a.s.
(v) A random variable X is independent of g if and only if for any Borel measurable function f such that Ef (X) exists, it holds
(1.34)
E(f(X)10
= Ef(X),
In particular, if X is independent of
P-a.s.
g, then E(X1g) = EX,
P-a.s.
(vi) Let gi c g, C F. Then (1.35)
E(E(X1g2)Ç1) = E(E(X1g1)Ç2) = E (X Ig1 ) , P-a 's*
(vii) (Jensen's inequality) Let X E L1; R71 ) and 99 : Rni --+ IR be a convex function such that cp(X) E L(1; R). Then
(1.36)
cp(E(Xg)) < E( go(x)Ig),
P-a.s.
In particular, for any p > 1, provided that El XI exists, we have
(1.37)
1E(Xig)1 19 < E(IXIPIg), P-a.s.
Proofs of the above results are straightforward by the definitions.
Chapter 1. Basic Stochastic Calculus
10
Next, we consider the situation involving two random variables. Let (S, 13) be two random variables. Then _1 (13) is a sub-a-field of T. Thus, we may define E(X (8)), the conditional expectation of X given e -1 (8). On the other hand, we may treat this in a different way. Define (compare with (1.26))
X E 4(0;R') and e : (S2, F, P)
(1.38)
VB E B.
v(B) = fX (w)dP(w), -1(B)
Then v is a vector-valued measure on (S, 13) and it is absolutely continuous with respect to the measure 1c) g P o e -1 . We denote by E(Xle = x) the Radon-NikoOm derivative of u with respect to
(1.39)
E(Xle =
x E S.
ddpi ç(x),
The above is called the conditional expectation of X given e x. Thus, E(Xle = :S 1R,"2 is a function such that (noting (1.28) and (1.38)-
(1.39))
L
-1(B)E(Xl = = f E(X1 = x)dr:(x) =
du(x) = v(B) f JB
=f
-1(B)
X (w)dP(w)
(B))(w)dP (co), VB E B.
E(X
Consequently,
(1.41)
E(Xle -1 (B))(w) = E(Xle = e(w)),
P1-- , (B) -a.s. w E a
Next we turn to the conditional probability. For a sub-a-field
g
of F,
define
(1.42)
/
P(Ag)
E(/Ç),
VA E T,
which is called the conditional probability of (the event) A given (the condition) Ç. Note that in general, we have
(1.43)
P(Ç) = o, 00
p(og) = 1, C0
114) = E P ( Ailg) ,
P( i=1
i=1
VA E F, mutually disjoint. For any given A E F, P(AIG)(w) is defined only for PI Ç -a.s.w E St. Thus for a given w E I, P(•1g)(u.,) is not necessarily a probability measure on F . However, we have the following result.
1. Probability
11
Proposition 1.9. Let (0, F, P) be a standard probability space and g be
a sub-a-field of F. Then the following hold.
[0,1], called a regular conditional (i) There exists a map p : 1 x probability given g, such that p(w, •) is a probability measure on (sz, ,A) is g-measurable for any A E .F, and for any w E (1.44) E(/A1g)(w)
P(Alg)(w) = p(w, A), Plg -a.s. Cal E fi VA ,
E F.
Moreover, the above p is unique in the following sense: If p' is another regular conditional probability given g, then there exists a P-null set N E g such that for any w N,
(1.45)
p(w, A) =
(w, A),
VA E .F.
g be a countably determined sub-a-field and let p(• ,.) be the regular conditional probability given Ç. Then there exists a P-null set N E g such that for any w N,
(ii) Let 71 C
(1.46)
p(w, A) = IA(w),
VA E
In particular, if e: (S2,1-)
(S,13) is a g/B-random variable and B is countably determined with {x} E B for all x E S, then (1.47)
p(w, , {w' : e(w') = e(w)}) = 1,
E
For the conditional expectation E(Xle = x) of X given e = x, we have the following parallel result. Let (0,.F, P) be a standard probability space and (S,13) a measurable space. Let 6 : (0, ■ (5,13) be a random variable and P P o e-1 the induced probability measure on (8, 8). R, called a regular conditional (i) Then there exists a map probability given 6 = x, such that j5(x, -) is a probability measure on (fi, F) for any x E S, 1-3(• , A) is 13-measurable for any A E ..F, and Proposition 1.10.
—
(1.48) P (A n e-i(B) ) =
fB x
Consequently, for any X E 4(0;
(1.49)
,
A )d.13 ( x ),
VA E F, B E 8.
Er), Pc a.e. x E S,
E (X le = x) = .11 X (w)f)(x , dw),
-
and
(1.50) F3(x, A) = E(/AC = x)
P(AC = x), Pc a.e. x E S, VA E F. -
(ii) Further, if (S, B) is also standard, then there exists a 13 -null set N E such that for any x N,
(1.51)
1-5(x,e -1 (B)) = 113(4
VB E B,
Chapter 1. Basic Stochastic Calculus
12 and in particular, (1.52)
1 (X))
= 1.
For proofs of Propositions 1.9 and 1.10, see Parthasarathy [1, pp. 145150]. We now give a relation between p(• ,•) and F, ( • , . Take A G .7- and (8)) and (1.48), one has B E B. Using (1.44) (with g = i(B)
p(co, A)dP(w) =E(1 - Ale -1 (B))(w)dP (w) = f
IA(w)dP(w) = P(A
n
l
(
B))
(1.53)
=
=
j5(x, A)d(P o e -1 )(x)
-1(13)
ii(e(cv),A)dP(w).
Thus,
p(w, A) = /3( (w), A),
(1.54)
Pl_ i(B) -a.s.
E f/, VA e F.
From now on in this book we will identify P( g)(.) with p(co ,•) and P( = x) with 7- 3(x, -). In other words, provided that (Q, .F, P) is standard, we always let P( g) and P( • I = x) be the corresponding regular conditional probabilities. In particular, they are well-defined for each and every sample a., E OE To conclude this subsection, we supply a result for later use. Lemma 1.11. Let e : (Q,T) (Re , B(1W)) be a random variable, and X E Lir (ft; 1an1 ). If E(g(e)X) = 0 for any g E Cb(R e ), where Cb(R t ) is the set of all bounded continuous functions from R I to R, then we must have
E(IAX) = 0,
(1.55)
VA E a(e)•
Proof. Define N
(Si, T)
(1H., 8(R)) cp is measurable, E(cpX) = 01,
and bt ), A IC - 1 (B)IB = (ai,bi) x (a2,b2) x • • • x - oo < ai < b, < +oo, i = 1,2, ... ,
A is a 7r-system and H is a linear space. Moreover, 1 E N, and if cp, E H with 0 < cp, cp, where c,o is finite, then Lemma 1.6 yields E N. On the other hand, for any A = e -1 (B) E A with B = (al,bi) x (a2, 62) x • • x (ae, be), a standard direct construction shows that there It is clear that
•
1. Probability
13
is a sequence {9,} C Cb(IR') such that g,(x) Î .41 (x), i Therefore,
oc, Vx E TR! .
E(IAX) = E(.4-1 (B) X) =f IB(C(w))X(w)dP(w) = lim f gi (e(w))X(w)dP(w) = lim E(g,(C)X) = O. —. 00 0
This shows that IA E R. Then applying Lemma 1.6 yields that 7-1 contains a(e)-measurable functions from SI to R, proving (1.55). all a (A) Proposition 1.12.
> (Rm' , B(RnN) be a sequence of
Let e, :
—
random variables (i = 1, 2, ...). Let Gay, o-(C) and X E L.2F (S-2; Rm ). Then E(XIG) = 0 if and only if E(g(C i , C)X) = O for any i and any g E Cb(R N ), where N = E; =1 Proof. The "only if" part is clear. For the "if" part, define A
A
A
00 U F
.
i=1
Then it is clear that G
= 0- ( A).
Set
1cp : (S1, .7") —> (R, B(R))Icp is measurable, E(cpX) = 01. For any A E A, we have A E .7; for some i. By the assumption and Lemma 1.11, IA E R. Moreover, if cp, E 7-{ with 0 < (pi T yo, where yo is finite, then the Lévy lemma implies that (i9 E Hence by Lemma 1.6, H contains all a(A) G-measurable functions, which completes the proof.
1.4. Convergence of probabilities To prove some existence results in stochastic analysis and stochastic controls, it inevitably involves convergence of certain probabilities. We summarize in this subsection some important results concerning the convergence of probability measures. Let (U, d) be a separable metric space and B(U) the Borel a-field. Denote by P(U) the set of all probability measures on the measurable space (U, B(U)). Definition 1.13. A sequence {P i } C P(U) is said to be weakly convergent to P E P(U) if for any f E CO), (1.56)
lim 2-.00
u
f(u)dPi (u) = f f(u)dP(u). JU
Proposition 1.14. There is a metric p on P(U) such that P, —> P weakly
is equivalent to p(P,, P)
0 as i
oc.
Definition 1.15. A set A C P(U) is said to be
Chapter 1. Basic Stochastic Calculus
14
(i) relatively compact if any sequence {13,} C A contains a weakly conver-
gent subsequence; (ii) compact if A is relatively compact and closed; (iii) tight if for any E > 0 there is a compact set K C U such that infp E A P(K) > 1 - E. Proposition 1.16.
Let A C 'P(U). Then:
(i) A is relatively compact if it is tight. (ii) If (U, d) is complete (i.e., it is a polish space), then relatively compact.
A is tight if it is
See Ikeda-Watanabe [1, pp. 6-8] for proofs of Propositions 1.14 and 1.16. Corollary 1.17. If (U,d) is compact, then any A C P(U) is tight and
relatively compact. In particular, P(U) is compact.
The proof is straightforward by Definition 1.15 and Proposition 1.16. See also Parthasarathy [1, p. 45, Theorem 6.4]. Recall that if X : F, P) -4 (U, d) is a random variable, then Px E P(U) denotes the probability induced by X (see (1.9)). We say that a family of random variables XQ : (0, .F, P) -4 (U, d) is tight if {P x„} is tight. Definition 1.18. Let X, : (122 ,.Fi , Pi ) (U,d), i = 1,2,..., and X : (0,T, P) (U, d) be random variables. We say that X, converges to X in law if Px, Px weakly as i -> oc. Definition 1.19. Let Xi , X : (12,..F,P) -> (U,d), i = 1,2,..., be random variables. We say that X, converges to X almost surely if lim d(X 2 , X) = 0, P- as.,
and that X converges to X in probability if for any E > 0, lim Piwid(Xi(u.;),X(w)) > el = 0.
It is not difficult to verify that for a sequence of random variables, (1.57)
almost surely convergence =
convergence in probability convergence in law.
The following result is a sort of reverse of the above implications. Theorem 1.20. Let (U, d) be a Polish space and {P i , i = 1, 2, .. . , PI C P(U) be such that P i converges to P weakly. Then on some probability space there are random variables X„ X: (6,fr) -> (U, 13(U)),
such that (i) P,
Px,, i = 1,2,..., and P (ii) X, -4 X as i oc, P-a.s.
Px•
This theorem is due to Skorohod. See Billingsley [1] for a proof.
2. Stochastic processes
15
2. Stochastic Processes
In this section we recall some results on stochastic processes. 2.1. General considerations Definition 2.1. Let / be a nonempty index set and (SI, .F, P) a probability space. A family {X(t), t E I} of random variables from (It, .F, P) to Iftni is called a stochastic process. For any w E St, the map t i-- X (t, w) is called a
sample path. In what follows, we let I = [0 , 7] with T > 0, or I = [0 , oo) . We shall interchangeably use {X(t), t E I}, X(.), X(t), or even X to denote a stochastic process. For any given stochastic process X(t), t E I, we can define the following:
(2.1)
1
Ft1 (x 1 )--1P{X(t 1 )
xi},
Ft1 ,t2 (xi,x2)a-P{X(ti) < xi, X(t2) < x21, ,x3 ) 12' P{X(ti) <
,X(t 3 )
Here, ti E I, r• Rm , and X(t) < x, stands for componentwise inequalities. The functions defined in (2.1) are called the finite - dimensional distributions of the process X(t). This family of functions satisfies the following conditions:
(a) Symmetry: If {i1,... , ii } is a permutation of {1, • • • ,./ } , then (2.2)
Fti i
(Xiil • •
•
'
Xi) ) =
F1
(X11 • • •
Xi)•
(b) Compatibility: For all i < j, (2.3)
(x1, • • . ,x, oo,... , oo) =
•••,
Thus, any stochastic process admits symmetric and compatible finitedimensional distributions. Conversely, given a family of functions, denoted by F, consisting of functions Ft1,...,t, , x3 ), j > 1, and satisfying the above conditions (a) and (b), it is natural to ask whether one can find a stochastic process {X(t), t E I} whose finite-dimensional distributions coincide with F. The following result of Kolmogorov gives a positive answer to this question. Let F = {Ft1 ,. (xi, , x3 ), j > 1} be a family of functions satisfying the symmetry and compatibility conditions. Then there exists a probability space ( 1 2, F , P) and a stochastic process X(t) whose finite-dimensional distributions coincide with F. Theorem 2.2.
Chapter 1. Basic Stochastic Calculus
16
For a proof, see Parthasarathy [1, pp. 143-144]. In what follows, any stochastic process will be called simply a process if no ambiguity should arise, and the probability space (0, F , P) and the time interval [0, T] will be fixed unless otherwise stated. Definition 2.3. Two processes X(t) and X(t) are said to be stochastically equivalent if (2.4)
X(t) = X(t),
P-a.s. , Vt E [0,71
In this case, one is called a modification of the other. If X(t) and X(t) are stochastically equivalent, then for any t E [0, T], there exists a P-null set Nt E F such that (2.5)
X(t, cd) = ' (t, u),
`du E li \ N.
Consequently, it follows from (2.1) that the finite-dimensional distributions of X(t) and X(t) are the same. However, the P-null set Nt depends on t. Therefore, the sample paths of X(t) and X(t) can differ significantly. Here is a simple example. Example 2.4. Let f/ = [0, 1], T > 1, P the Lebesgue measure, X(t, cd) 0, and (2.6)
X(t,cd) =
0, 1,
cd = t.
Then X(t) and X(t) are stochastically equivalent. But each sample path X(. , w) is continuous, and none of the sample paths Xe , cd) is continuous. In the present case, we actually have (2.7)
U Nt =[0,1]OE te[0,1]
Definition 2.5. The process X(t) is said to be stochastically continuous at s E [0,T] if for any e > 0, (2.8)
lim No, E f21 IX(t,cd) — X(s,cd)I > el = 0.
Moreover, X(t) is said to be continuous if there exists a P-null set N E such that for any cd E f2 \ N, the sample path X(. ,w) is continuous. Similarly, one can define the left and right (stochastic) continuity of processes. It is clear that continuity implies stochastic continuity. Next, for a given measurable space (S2, .F), we introduce a monotone family of sub-a-fields F Ç F, tE [0, T]. Here, by monotonicity we mean (2.9)
Ft, C Ft2 ,
VO < t i < t2 < T.
2. Stochastic processes
17
Such a family is called a filtration. Set Ft+
nis> t
for any t E [0, T),
for any t E (0,21. If Ft+ = .Ft (resp. _Ft _ = .Ft ), we and Ft- g Us
o is right (resp. left) continuous. We call (S1,.F, {Ft} t>o) a filtered measurable space and (fl, .F, {Ft }t>o, P) a filtered probability space. Definition 2.6. We say that (0,.F, {.Ft}t>o,P) satisfies the usual condition if (Q, F, P) is complete, .7-0 contains all the P-null sets in .F, and VtIt>o is right continuous. Note that Definition 2.1 for stochastic processes can be extended naturally to those taking values in a metric space (U, d). Definition 2.7. Let (S-2, .F, {.Ft } t>0 ) be a filtered measurable space and X(t) a process taking values in a metric space (U, d).
(i) The process X(t) is said to be measurable if the map (t, w)
X(t,w)
is (B[0, T] x .F)/B(U)-measurable. (ii) The process X(t) is said to be {Ft} t >o adapted if for all t E [0, T], the map w X(t, w) is .Ft /B(U)-measurable. -
(iii) The process X(t) is {Ft} t >o progressively measurable if for all t E [0, 71, the map (s, X(s,w) -is B[0, t] x .Ft /B(U)-measurable. -
It is clear that if X(t) is {.Ft } t >0-progressive measurable, it must be measurable and {.Ft } t >0-adapted. Conversely, on a filtered probability space, we have the following result. Proposition 2.8. Let (Q, P) be a filtered probability space, and /et X(t) be measurable and {.Ft } t> 0-adapted. Then there exists an {Ft} t >o-progressively measurable process X (t) stochastically equivalent to X (t). If in addition, X (t) is left or right continuous, then X (t) itself is IFtlt>o-progressively measurable. For a proof, see Meyer [1, p. 681. Due to the above result, we make the following convention hereafter for the sake of simplicity, nevertheless without affecting any of the results. Convention 2.9. Unless otherwise indicated, in a filtered probability space, by saying that a process X(t) is Vt l t >0-adapted, we mean that X (t) is {Ft} t>o -progressively measurable. Note that in the definition of stochastic equivalence, the probability measure P has to be involved. Thus, Proposition 2.8 and Convention 2.9 make sense only in filtered probability spaces (rather than merely filtered measurable spaces). In other words, in a filtered measurable space, we have to distinguish progressive measurability and adaptiveness. We point out that all the results in this and the next subsections carry over naturally to the case where [0, T] is replaced by [s, T] (s E [0, T)). The only modification is to replace 0 by s (in, say, {Ft}t>o, B[0,7], etc.). Let H be a Banach space with the norm H and let (Q, F, {Ft }t>o, P) be a filtered probability space satisfying the usual condition. We will denote
Chapter 1. Basic Stochastic Calculus
18
hereafter by LPF(0, T; H) the set of all H-valued, {.Ft } t >o-adapted processes X ) such that E foT IIX(t)n i dt < oc, by L7(0, T; H) the set of all Hvalued, {.Ft} t >0 adapted bounded processes, and by LPF (0; C([0, 7 ] ; H)) the set of all H-valued, {Ft } t >0-adapted continuous processes X() such that E{ suPtc[om ilX(t)II} < (.
-
Next, for T > 0, we define Wm[0, T] C([0, 1 ] ; Rni ) and define
W[0, T] (2.10)
a{C( . A t) ((-) c Wm[0,
},
Vt E [0,T
],
(B(W7 1 [0 , T])), V t E [0, T],
Bt (Wm [0 , T]) Bt+(Wm[0,71) =
fl 139 (Wm[0,7]),
Vt E [0,T).
s>t
We note that Bt (Wm[O, T]) is the a-field (in Wm[0, TD generated by B(Wit"[O,T]), thus it contains Wm[0, T]. Clearly, both the following are filtered measurable spaces:
(Wm [0, T], B(Wm [0 , TD ; {Bt (W m [0 , T])} >o) , (Wm[O,T], B(Wm [0, T]), {13t+(W m [0, T])It>o). We point out that
(2.11)
Bt+ (Wm [0,7])
Et (Wm[O,T]), Vt E [0,T];
see Karatzas—Shreve [3, p. 122], as well as some remarks in the next subsection. In what follows, for any Polish space U (in particular, for U = Rn , R etc.), we let AT(U) be the set of all {Et+ (Wm[0, T])It>oprogressively measurable processes 77: [0, T] x Wm [0, T] U. The following result is a further extension of Theorem 1.7 and will play an important role in appropriately formulating stochastic optimal control problems studied in the subsequent chapters. Theorem 2.10. Let (0,T, P) be a complete probability space and (U, d) a Polish space. Let e : [0,T] x Rm be a continuous process and = a(e(s) : 0 < s ( t). Then yo : [0, T] x U is {.F}t>0-adapted if and only if there exists an n E .4(U) such that
(2.12)
(t , co) = n(t ,
A t,w)),
P-a.s.
C4J
E
E [o, T].
In order to prove the above result, we need two lemmas. A set B C Wm[O,T] is called a Borel cylinder if there exist 0 < t i < t2 < • • • < t < T and E E 8(I1.37n ) such that
(2.13)
B = {(- E W m [0,T]
(00,02 ), . . ,((t3 )) E El.
We let Cs be the set of all Borel cylinders in WT[0, T] of the form (2.13) with , t3 E [0,4 Lemma 2.11. The a-field a(CT) generated by CT coincides with the Borel a-field B(Wm[O,T]) of Wrn[O,T].
2. Stochastic processes
19
Proof Let 0 < t1 < t2 < • • • < t3 < T be given. We define a map RP' as follows:
T: Wm[O,T]
T(() = (00,02 ), . . . , ((t3 )),
E Wm [0,
Clearly, T is continuous. Consequently, for any E E 13( R3m ), it follows that T-1 (E) E 13(Wm [0, T]) . This implies
(2.14)
CT Ç
B(w-R),71).
Next, for any Co E W m [0, T] and E > 0, we have E Wm [OIT]
(2.15)
fl
K — Colwmp,T1 5 El E W m [0, T]1 I((r) - 6)(01 5 el E u(CT),
rECIrE10,71
since I( E W m [0, 7] I((r) - (0(r)1 < el is a Borel cylinder, and Q is the set of all rational numbers (which is countable). Because the set of all sets in the form of the left-hand side of (2.15) is a basis of the open sets in Wm [0, T], we have
(2.16)
13(Wm[0, 71)
ç
cr(CT).
Combining (2.14) and (2.16), we obtain our conclusion.
0
Lemma 2.12. Let (12,.F,P) be a complete probability space and e : [0, T] x Q RI' a continuous process. Then there exists an no E .7- with such that e : Qo Wm[O,T] and for any s E [0,T], P(Oo) = 1 (2.17)
n
=
Proof Let t E [0,s] and E E B(Rm ) be fixed. Then
Bt
at( E Wm [0, 7]1 ((t) E El E C,,
and E C(Bt)
e(t,w) E E
E
Thus,
e(t,') -1 (E) Then by Lemma 2.11, we obtain (2.17). Now we are ready to prove Theorem 2.10. Proof of Theorem 2.10. We prove only the "only if" part. The “if” part is clear. For any s E [0, TI, we consider a mapping
618 (t,w)a (t A s,(. A s,w)) : [0,7] x Q -■ [0,s] x W 8772 [0, T].
Chapter 1. Basic Stochastic Calculus
20
By Lemma 2.12, we have B[0, s] x .71 = cr(0 8 ). On the other hand, (t,w) i. cp(t A s,w) is ( 13 [0, .5] x .71)/8(U)-measurable. Thus, by Theorem 1.7, there exists a measurable map ns : ([0, T] x W.:1 [0,T], B[0, s] x B s (Wm [0,71)) * U such that —
(2.18)
(p(t A s,w) =
A s,C(• A s, w)),
Vw E 11,t E [0,7 ].
Now, for any i > 1, let 0 = t < t < • • • be a partition of [0,71 (with maxj >i[t.; — t2i _ 1 ] 40asi ■ oo), and define —
-
ih; (t, ((' A q))/(t.,t=i 1(t), i>i V(t,() E [0,7] x Wm[0,
= 1/0( 0 , ((' A 0 ))/{e}(t) (2.19)
For any t E [0, 7], there exists j such that ti.7 _ 1 < t < tip Then
(2.20)
ni (t,C(• A t w)) = ri
A
,
tji , w)) = w(t, co).
Now, in the case U = IR, N, ( 1,n), we may define
(2.21)
n(tt () = lim 772 (tt ()
to get the desired result. In the case where U is a general Polish space, we need to amend the proof in the same fashion as in that of Theorem 1.7. Note that the map OM in (2.21) is only {Bt+(W m [°, T]llt>o progressively measurable, not necessarily {Bt (Wm [0, 7 ] )1 t >0 progressively measurable. -
Proposition 2.13. Let (SI,F, {Yt}t>o,P) be a filtered probability space satisfying the usual condition, and let be an {.Ft } t >0-adapted process. Then for any s E [0, 71,
(2.22)
P({w i
= C(t,w)} 1 1.8)(w) = 1 , P a.s. w E S2, Vt E [0, .9]. -
If in addition CM is continuous, then there exists a P-null set N E .F such that for any s E [0,T] and w N,
(2.23)
e(t,w i ) = e(t,w),
Vt E [0, .5],
Pe J .F5 )(w)-a.s.
E S2.
Proof. By definition, for any t E [0, s], P({w' I e(t,u/) = (t, w)} .7) (w) = = / (w) = 1, P-a.s. w E {0.,'k(t,w1=C(t,w)}
proving (2.22). By the continuity of C, we can get (2.23) easily.
0
2. Stochastic processes
21
The above result implies that under the probability P( . I Ts )(w), where w is fixed, e(t) is almost surely a deterministic constant e(t, w) for any t < s. This fact will have important applications in stochastic dynamic programming studied in later chapters. Finally, we discuss a convergence property of a sequence of continuous stochastic processes, based on the Skorohod theorem (Theorem 1.20). Theorem 2.14. Let X, = {X,(.)}, i = 1, 2, ..., be a sequence of m-dimensional continuous processes over [0,T] on a probability space (12, F, P) satisfying the following: sup EIX(0)1 -Y <+00, sup E1X, (t) {
- X, (s)1' < Kit
-
.91 1 +°, Vt,s E [0, T],
i>1
for some constants a, 0,-y > 0. Then {X, (. )} is tight as Wm[O,T]-valued random variables. As a consequence, there exists a subsequence NI, mdimensional continuous processes i,, = { je,,()} (j = 1, 2, ...), and 5e defined on a probability space (fi,.?, il) such that
P(X,, E A) = f) (5ei, E A), lim 5e, —* 5e {
J
,..
3
VA E B(W m [O,T , ])
in Wm[0,7], P-a.s.
Corollary 2.15. Let X() be an m-dimensional stochastic process over [0,T1 such that
EIX(t)- X(s)l a < Kit
-
si 1
, Vt, s E [0, 7 ] ,
for some constants a, (3 > 0. Then there exists an m-dimensional continuous process that is stochastically equivalent to X(.). See Ikeda-Watanabe Corollary 2.15.
[1, pp. 17-20] for proofs of Theorem 2.14 and
2.2. Brownian motions
Let us now introduce an extremely important example of stochastic processes, called Brownian motion.
Let (0,Y, {.Ft}e>o,P) be a filtered probability space. An {.Ft } >o-adapted Rni-valued process X(.) is called an m-dimensional {Tt}e>0 Brownian motion over [0,00) if for all 0 < s < t, X (t) X (s) is independent of F3 and is normally distributed with mean 0 and covariance s)I. Namely, for any 0 < s < t, Definition 2.16.
-
-
-
(2.24)
f E(X(t) X(s)F3 ) = 0, P a.s. , 1 E((X(t)- X(s))(X(t)- X(s)) T 1..F3 ) = (t -
-
-
s)I,
Chapter 1. Basic Stochastic Calculus
22
In addition, if P(X(0) = 0) = 1, then X() is called an m-dimensional standard Velt>o Brownian motion over [0, oc). -
Note that one can also define a Brownian motion W(.) naturally over any time interval [a, 6] or [a, 6) for any 0 < a < b < +oo. In particular, W(.) is said to be standard over [a, b] if W(a) = O. It is easy to show that if W(t), t > 0, is a standard {.Ft}t>o-Brownian motion, so are W(t + a) - W(a) (a > 0) and A -1 W(A2 t) (A 0). We now construct a Brownian motion. Define Wm = C([0, 0.) ; iRm), and let
=E
[1 w -1.13 1C([0,i);Rm)
A 1], Vw,i'vE Wm .
j>1
Clearly, ' is a metric under which Wm is a Polish space. We call a set (of the form similar to (2.13)) B=
E
Wm (00,(t2), • • • ,C(t3 )) E El,
a Borel cylinder in Wm, where 0 5 t1 < t2 < • • • < t3 < oc and E E 8(R3 m). We let C be the set of all such Borel cylinders in Wm. The following result is similar to Lemma 2.11. Lemma 2.17. The cr-field u(C) generated by C coincides with the Bord B(Wm) of Wm .
For any t
0, let E Wm 1, vvr 'am. A t) Bt(wm) 0-(B(w r )),
(2.25)
B+(W m ) =
fl Bs(w m ) ,
s>t
which is similar to (2.10). Let and let
be a probability measure on (Rm ,B(Rm ))
f(x,t) = (2t)_m/ 2 e __ ,
(2.26)
t > 0, x E Rm ,
be the density of normal distribution N(0, ti) (see (1.14)). We introduce Pp : C [0,1] as follows: For any 0 < ti < t2 < • • < t i < T and any E, E B(Rm ) (1 < i j),
w E Wm I w(ti) E a
(2.27)
= f P(dxo)f
Rm
E
E.) }
f (ti,xi — xo)dxi
f (t2 — ti,
X2 Xl)dX2
E2
• • • IE3 f(t3 -t 3 —
x3 - x 3 — i )dx 3 '
One can show that P„ is a probability measure on (C,a(C)). Then, by Lemma 2.17, we may extend P, to a probability measure on the space
3. Stopping times
23
(Wm, B(Wm)), which is called the Wiener measure with the initial distribution p. For given p, such a Wiener measure is unique. Next, on the filtered probability space (Wm, B(Wm), {B(Wm)}>0, Pt.) we define (2.28)
t > 0, w E Wm .
X(t,w) = w(t),
It can be shown that X(t) is an m-dimensional Brownian motion with the initial distribution a on (Wni, B(Wm), {Bt (Wm)} t>o , PA ). In addition, if A is a probability measure on (am, B(Rm)) concentrated at 0, the above gives an m-dimensional standard Brownian motion. We call the process X(t) defined by (2.28) the canonical realization of an m-dimensional Brownian motion. For detailed proofs and comments, see Ikeda-Watanabe [1, pp. 40-41] or Karatzas-Shreve [3, pp. 49-56]. In general, if X(.) is a Brownian motion defined on some filtered probability space (12, F, {.Ft} t>o,P), we may define (2.29)
FtX
g cr{x(s),o < s < t} c F
,
vt> O.
In particular, if X(t) is given by (2.28), then (noting Lemma 2.17) w(s) E BI 0 < s < t, B E B(R.m )}
-Ftx = (2.30) Ilw E Wm
Bt(W m ). The filtration {.F‘} t>0 is left continuous, but not necessarily right continuous due to a fact similar to that stipulated in (2.11). On the other hand, the augmentation {."7"(x- }t>o of {Fix }e>o by all the P-null sets is continuous, and X(t) is still a Brownian motion on the (augmented) filtered probability space (f2, F , {F'tx } t>o P) (see Karatzas-Shreve [3, p. 89 and p. 122] for detailed discussions). In what follows, by saying that Vtlt>11) is the natural filtration generated by the Brownian motion X, we mean that IFtli>0 is generated as (2.29) augmented by all the P-null sets in F (thus trtIt>0 is continuous). 3. Stopping Times
In this section we discuss a special class of random variables, which plays an interesting role in stochastic analysis. Let (12, 7", {.Ft }t>o, P) be a filtered probability space satisfying the usual condition (see Definition 2.6). Definition 3.1. A mapping time if (3.1)
—■
[0, co] is called an {Ft} t>0- stopping
(7 < alw E 1 r(w) < E Ft ,
For any stopping time (3.2)
T:
FT a{A
T,
Vt > 0.
define An (1- < t) Tt , Vt >0}.
Chapter 1. Basic Stochastic Calculus
24 It is clear that F is a sub-a-field of F.
Proposition 3.2. Stopping times have the following properties:
(i) A map r : Q
[0, col is a stopping time if and only if (T < t) E F,
(3.3) (ii) If T
is a stopping time, then A
E
Vt > O. if and only if
A n (7- t) E
(3.4)
Vt > O.
Proof. We first prove (ii). If A E .FT , then for any t > 0,
(3.5)
A n(r < t) = U {A n
(7-
t- -1 )} E
n>1 Conversely, if (3.4) holds, then for any t > 0, by the right continuity of Ft,
(3.6)
A n (r
t) =
n {A n
< t+ -1 )} E Ft+ Ft.
n>1
This proves (ii). Taking A = Q, we obtain (i). Now we give an example that will be useful later. Example 3.3. Let X(t) be {Tt}t>o-adapted and continuous. Let E C Rm. be an open set. Then the first hitting time of the process X(t) to E,
(3.7)
0-E(w)
x(t,w)
inflt
E El,
and the first exit time of the process X(t) from E,
(3.8)
rE(w) infft 0 X(t,w)
El,
+00.) Let us prove these are both {.Tt}t>13-stopping times. (Here, inf two facts. First of all, for any s > 0, we claim that
(3.9)
(aE < s) = U (X(r) E E) E .F8 . rEQ,r<8
To prove the equality, let w be such that cE(w) < s. Then by the continuity of X, we have some rational number r < s, such that X(r, w) E E. Hence w belongs to the right-hand side of (3.9). Conversely, let w be in the righthand side of (3.9). Then, for some rational number r < s, X(r,w) E E. This implies that crE(w) < s. Hence, the equality in (3.9) holds. Consequently, by Proposition 3.2, crE is a stopping time. Next, for any t > 0, we have
(3.10)
(TE
t) =
n (X
r
G E) E F.
rEQ,r
The proof of (3.10) is similar to that of (3.9). Hence
(3.11)
(TE
< t) =
(TE
> t) c E
25
3. Stopping times
Then, by Proposition 3.2, 7E is a stopping time. a, T, and a, be stopping times. Then are also stopping times: following (i) The
Proposition 3.4. Let
a + T, sup a-i , inf ai , lim ai , lima. (ii) The map r: (S1, FT ) -> ([0, oo], B[0, co]) is measurable and the process A
>o-progressively measurable.
Y(t)=7- A t is {Ft} t
(iii) The following holds:
(3.12)
(0"
> T),
> T),
= T) E
.FcrA r •
Moreover, for any A E A n(a-
(3.13) In particular,
(3.14)
C
on S-2
r.
In the case where (Q,T, {F} >o, P) is complete, (3.14) can be replaced
by
(3.15)
a < T,
C F.
P a.s. -
(iv) Let el =- infi>i at. Then (3.16)
CIF, =Fe,
For proofs of the above results, see Karatzas-Shreve [3, pp. 6-10]. Let (RI*, {,Ft } t >o ,P) be a filtered probability space satisfying the usual condition. Let X (t) be an {Ft } t >o-progressively measurable process and T an {.Ft } t>,3 -stopping time. Then the random variable X(r) is Tr-measurable, and the process X(r A t) is Vtlt>o-progressively measurable. Proposition 3.5.
Proof. We first prove that X(rAt) is {F } 1>o -progressively measurable. To this end, by Proposition 3.4, the process T A t is {Tt } t >o-progressively measurable. Thus, for each t > 0, the map (s,w) (7- (w) A s,w) is measurable from ([0, t] x Il B[0, t] x Ft ) into itself. On the other hand, by the progressive measurability of X(t), the map (s,w) 1-4 X(s, w) is measurable from ([0, t] x S2, B[0, t] x Ft ) to (Rm,B(IFtm)). Hence, the map (s,w) X(7 - (w) A s,w) is measurable from ([0,t] x 1,L3[0, t1 x Ft ) to (1Rni ,B(111.171 )). Hence the process X(7 A t) is Vtle>o-progressively measurable. In particular, X(7 A t) is Ft -measurable for each fixed t > 0. Now, for any B E B(IRm), ,
(3.17) (X(r) E B)
n < t) =
(X(7- A
t) E B) n < t) c .Ft , Vt > 0.
Chapter 1. Basic Stochastic Calculus
26
Thus, (X(T) E B) E .F,-, for all B E B(IRm ), proving the measurability of the map X(T) : (ft, FT ) ( 1Rni , B(Rm )). Proposition 3.6. Let T be a stopping time and C a random variable. Then
C is Fr -measurable if and only if for all t > 0, CI(,-
C3
EC/A , -4C,
where
C
as j -4 oc,
P-a.s. ,
3
7.>1
E IR and Az3 G .F,-. Hence,
(3.19)
eikr
i>1
oc, we see that C/( r < t ) is Ft -measurable. is Ft -measurable. Letting j Conversely, if for all t > 0, C/( r
a)D ,
t) = (C1 (
E Ttl
and for any a > 0,
(3.21)
(C > a)n(r
<
t) = (CI0.<0 > a) = (CI0.<0 <
E Ft .
This implies
(C < a) E FT , a < 0, (C < a) = (C > cx) c E
(3.22)
a >0.
Consequently, C is Fr -measurable. Proposition 3.7.
Let a and T be stopping times and X an integrable
random variable. Then = E(I(,),-)X1Y-7-) =- 1(o.>,-)E(X1Y -ani-), (3.23)
1( a>,-)E(XIFT ) = E(/( a>,.)X11,-) =
oAr,
E(E(Xl.FT)I.Fa) = E(X1FcrAr)•
Proof. By Proposition 3.4-(iii), we have the first equalities in the first two assertions. Now, since (3.24)
/0. >.,.)E(X1.7,-)/( oAr
E(Xl.F7)/(rr,oAr
E(XIF,-) is TT -measurable, and (a > r) E FA,- (see (3.12)), it follows that the right-hand side of (3.24) is Ft -measurable by Proposition 3.6. Hence, by Proposition 3.6 again, /(a>,.)E(XIFr ) is .F,A.,--measurable. Then, by the basic property of the conditional expectation, we have
(3.25)
E(/(,,T)X.7 1,-) = /( 0. >„)E(Xl.F.,-)= E(.1(0.>,-)E(X1.7 .7)1-ran.,-)
= I(a>..,-).E(E(X1F,-) I1 Fa A
) — 1(cr > ) E X 1.7t7 A r •
4. Martingales
27
This gives the first relation in (3.23). The second relation can be proved similarly. Finally, E(E(X1.71Y4
( 3.26)
= E(/( a>,.)E(X1.7,-)1.Fa ) + E(/( 0. ‹ .0E(Xl.F.,-)1.7-a ) = E(I(a>.,-)E(Xl-Fonr)l-Fa) EMT>a)E(X1Yanr)1.7.a) = 1.(T>T)E(Xl.F7Ar) I(T>,)E(X1Yonr) = E(XIFcrAT)•
This proves the third relation. 4. Martingales In this section we will briefly recall some results on martingales, which form a special class of stochastic processes. Let (RI", {.Ft}t>o,P) be a filtered probability space satisfying the usual condition. Definition 4.1.
A real-valued process X(t) is called a (continuous) Velt>o-martingale (resp. submartingale, superm,artingale) if it is {Ft } t>cradapted and for any t > 0, EX(t) exists with the property that (4.1)
E(X(t)I.F8 ) = X (s)
(resp. >, <), P-a.s. , VO < s < t.
It is clear that any martingale must be both a sub- and a supermartingale. As an example, any Brownian motion X(t) is a martingale by the first relation in (2.24). Proposition 4.2. fields of with gt
c
Let {Ft } t>o and {gt}t>0 be two families of sub-crYt, Vt > O. If X(t) is an {.Ft } t >0 -martingale (resp.
submartingale, supermartingale), then Y(t) E(X(t)iço is a martingale (resp. submartingale, supermartingalc). In particular, if X (t) is {Gt}t>o-adapted, then X (t) itself is a {gt}t>o-martingale (resp. submartingale, supermartingale).
Proof. The first assertion follows easily from Proposition 1.8-(vi) and the definition of a martingale, while the second one follows from the first along with Proposition 1.8-(iv). Roughly speaking, this proposition implies that a martingale with respect to a "larger" filtration is also a martingale with respect to a "smaller" filtration. Proposition 4.3. Let X (t) be a submartingale and cp : 111, 1R a nondecreasing convex function such that Ecp(X(t)) exists for all t > O. Then (P(X(t)) is a submartingale. In particular, when X (t) is a nonnegative is a subsubmartingale and EIX(t)IP exists for some p > 1, then IX (t) martingale. Proof. By the monotonicity of ço(-) and Jensen's inequality (see (1.36)), we have
(4.2)
(p(X(s)) < cp(E(X(t)
5 )) < E(y,(X(t))11-1,),
P-a.s. , s < t.
Chapter 1. Basic Stochastic Calculus
28
This proves our assertion. As we know, a stochastic process is not necessarily left or right continuous. However, for submartingales (in particular, martingales), we have the following result. Theorem 4.4. Let X (t) be an Vt I t > 13-submartingale. Then there exists N, the following limits exist: a P-null set N such that for any w E
(4.3)
lim
t
rEQ,r
X (r),
lim
reQ,r
X (r), Vt
t
E [0, 00).
If we define
(4.4)
2(0 =- lirn
rEQ,r j t
X(r),
P-a.s., Vt > 0,
2(t) as an arbitrary constant on N, then )7(0 is a submartingale that is right continuous with left limits, and for any sequence e i 10,
and define
(4.5) (4.6)
X(t
+ s3 )
X(t)
<
2 (t),
i(t),
Vt > 0, P-a.s., Vt > 0, P-a.s.
Moreover, P(X(t) = (t)) = 1 for all t > 0 if and only if t right continuous.
EX (t) is
The process i(t) in the above result is called the right continuous modification of X(t). By Theorem 4.4, if EX() is right continuous (which is the case when X(.) is a martingale), we may assume that X(.) itself is right continuous. -
Let p > 1 and X(t) be a right-continuous {.Fe}t>omartingale such that for all t > 0, EIX(t)1P 0,
Theorem 4.5.
(4.7)
P(sup IX(t)I > A) 5_ te[O,TI
EIX(T)IP
AP
and for p > 1,
(4.8)
E( sup IX(t)I) tE[0,71
)
(p
1)PEIX(T)IP•
(Optional sampling theorem) Let (11, P) be a filtered probability space satisfying the usual condition. Let X (t) be a rightcontinuous {.F } >o-martingale (resp. submartingale, supermartingale). Let {o-t } t>0 be a family of bounded Velt>o-stopping times such that Theorem 4.6.
(4.9)
P(as < at ) = 1,
s < t.
Let )7 (t) = X(o-t ) and it - =- io. t . Then {i (t)} >0 is an { it } t>o-martingale
(resp. submartingale, supermartingale).
5. Itô integral
29
See Ikeda-Watanabe [1, pp. 32-34] for proofs of Theorems 4.4-4.6. Corollary 4.7. Let (O,T, Vtlt>o, P) be the same as in Theorem 4.6. Let a < r be two bounded stopping times. Then, for any {Ft }t>o - martingale (resp. submartingale, supermartingale) X (t), (4.10)
E(X(Y)1..F a ) = X(a) (resp. >, <),
P - a.s.
Proof Let (4.11)
at = ol[o,ii(t) + T/(1,00)(t),
t > 0.
Then we have (4.9). Appealing to Theorem 4.6 by taking t> 1 and s < 1, one obtains (4.10). Corollary 4.8. Let X(t) be an Vtlt>0 martingale, and let a < r be two stopping times. Then -
(4.12)
E[X(t A r) - X(t A cr)I.F0.1 = 0,
Vt > 0.
Proof In view of Propositions 3.4 and 3.5, t A T and t A a are stopping times with tAr > tA a, P-a.s. , and X(t A 7- ) is Tt -measurable. Thus, by Corollary 4.7 and Proposition 3.7, we obtain (4.13)
X(t A cr) = E [X (t A r)1..Ftna] = EfE[X(t A T )I -Ft] 1 .7.a = E[X(t A 7)1.7].
This yields (4.12). Sometimes, we need to generalize the notion of martingale. Definition 4.9. A real-valued process X(t) is called a local {.7}>o if there exists a nondecreasing sequence of stopping times-martingle faili> i with (4.14)
P lim
= co) = 1,
so that X(t A al ) is an {.Tt } t >0-martingale for each j. We point out that every martingale is a local martingale, but it is not necessarily true the other way around (see Karatzas-Shreve [3, p. 168] for an example). We will mainly restrict ourselves to the finite interval [0, T] in this book. A local martingale X(t) restricted to [0, T] becomes a local martingale on [0, T]. We may also define a local martingale on [0, T] directly. In fact, we may simply copy Definition 4.9 with (4.14) replaced by the following: (4.15)
P( lim a.) > T) = 1.
This observation will be useful later.
30
Chapter 1. Basic Stochastic Calculus
5. Itel's Integral In this section we are going to define the integral of type
fo T f (t)dW (t),
(5.1)
where f is some stochastic process and W(t) is a Brownian motion. Such an integral will play an essential role in the rest of this book. Note that if for co E f2, the map t W(t,w) was of bounded variation, then a natural definition of (5.1) would be a Lebesgue—Sticltjes-type integral, regarding w as a parameter. Unfortunately, we will see below that the map t W(t,w) is not of bounded variation for almost all co E Q. Thus one needs to define (5.1) in a different way. A proper definition for such an integral is due to Kiyoshi Itô [1, 2]. We will introduce it after the following subsection. 5.1. Nondifferentiability of Brownian motion In this subsection we prove the almost sure nondifferentiability of Brownian motion. Actually, we will prove a little bit more than that. Again, let (ft,T, {Te } t>o,P) be a filtered probability space satisfying the usual conditions. Proposition 5.1. Let W(t) be a (one-dimensional) Brownian motion. Let and set
-y >
a{co E f2 1W (. , w) is Holder continuous with
(5.2)
exponent 7 at some t E [0, oc)}. Then (5.3)
P(Sy ) = O.
In particular, for almost all co E 12, the map t
W (t, w) is nowhere differ-
entiable. Proof. Since > there exists an integer i such that /1(7 — For any j, k > 1, define
> 1.
A ik = f w E 9 11W(t h, w) — W (t, w)l <jh, Vit E [0, 11] for some t E [0, 1]}. -
(5.4)
Also, for m= 0,1,2, ... , set
Sy [m, in I 1 ] alco E Il W(. ,w) is Wilder continuous with - -
exponent 7 at some t E [m,rn Clearly, one has (5.5)
Sy [0, 1] = U A J,k>i
1]}.
5. Itep integral
31
We will show that P(A,k) = 0 for all j, k > 1. To this end, let us fix j,k > 1. For any w E Aj k, by definition we can find a t E [0,1] such that
IW(t + h,w) - W(t,w)l < jttl' ,
(5.6)
a
Vh E [0,
Now let n > pk. Then, for the above t, there exists an i, 1 < i < n, such that n —< t < I. n Since n > pk, we have i V
(5.7)
t <
V1 ±
1
n
k'
1 < v < p, - 1.
Observe that for 1 <7/ < - 1,
1W( ÷ a v, w) W( 1,w )1 5- 1W( ÷ i v,w) W( t,w )1±1 W (' -H;L -1,w ) - W(t ,w )1 (5.8) (i + v - 1 (v + 1) 7 + v7 ) < (i + v t y t) n n This means that any w G Aj k must satisfy (5.8). Set (5-9) Cl n)
=
n
r
E 9 IW(IW) W( i+v-1 — n W)I <
i( (v+1)1r+v-r )}.
v=1
Clearly,
(5.10)
A kC
Vn > pk.
U Cln) i=1
Noting that W (! u) - W(i+vn-1 ) N(0,), we obtain
(5.11)
W('%-1 )] - N(0,1).
Since W(.) is a Brownian motion, then Zu , 1 < v < p, are independent. On the other hand, by (5.11), for any E> 0,
P (1Z, I <
(5.12)
E)
=
1
fE 1.1 2 LE edz <
2 7r
Therefore,
P(C) = P (IZ,, I
<
(5.13) n1 / 2
1)-r +
n7 - 112 ) 11
1
II (0) + 1) 7 +
a- Kn - Ph -1 / 2) ,
v=i
where K = K(j,-y, A) is a constant independent of n > pk. Hence (noting that tt(-y -) > 1),
(5.14) P
Ct(n) ) < Ep(c1n))
z-i
0,
n
Chapter 1. Basic Stochastic Calculus
32
This shows that Aj k is a subset of a P-null set. By the completeness of the probability space, we have A j k E .7. with P(Aj k) = O. Hence, P(S.[0,1]) = O. Similarly, for any m > 1, P(S.-.[m, m + I]) = 0. Then (5.3) follows. Consequently, it is clear that for almost all (4) E St, the map t W (t, w) is nowhere differentiable. 5.2. Definition of Itô's integral and basic properties In this section we give the definition of the Itô integral as well as some basic properties of such an integral. We shall describe the basic idea of defining the It6 integral without giving proofs. Those who are interested in the details should consult standard books such as Ikeda-Watanabe [I, pp. 45-51] and Karatzas-Shreve [3, pp. 129-141]. We first introduce the function space consisting of all possible integrands. Let (0, 7r, {.F} > o , P) be a fixed filtered probability space satisfying the usual condition. Let T > 0 and recall that L-( 0 , T; lit) is the set of all measurable processes f (t,w) adapted to {.Ft}t>0 such that T
(5.15)
11f113, 2'
E j f (t , w) 2 dt} < co . o
It is seen that L(0, T; 1R) is a Hilbert space. Next, we introduce the following sets, which are related to the integrals we are going to define:
M2 [0,71 = {X E L(0, T; R) I X is a right-continuous (5.16)
Vtlt>0-martingale with X(0) = 0, P-a.s. }, M![0, T] = {X E M 2 [0,7]
t
X (t) is continuous, P-a.s. 1.
We identify X, Y E M 2 [0,7] if there exists a set N E with P(N) -= 0 such that X(t, w) = Y(t,w), for all t > 0 and w N. Define
(5.17)
IXI T = (EX(T) 2 ) 1/2 ,
VX E M 2 [0,7].
We can show by the martingale property that (5.17) is a norm under which M2 [0, T1 is a Hilbert space. Moreover, M[0, T] is a closed subspace of M 2 10, T]. We should distinguish the norms • IIT (defined by (5.15)) and I • IT (defined by (5.17)). It is important to note that any Brownian motion WO is in M![0, T] with IWI = T (see (2.24)). Now we are ready to itemize the steps in defining the It6 integral for a given one-dimensional Brownian motion W(t) defined on the filtered probability space (RI-, {.Ft}t>o,P)• Step 1. Consider a subset C0[0, T] C L2T (0, T; IR.) consisting of all real processes f (t, w) of the following form (called simple processes): (5.18)
f (t,w) = fo(w)I {t=o}(t) +
E fic
(t),
t E [0,T},
5. It6 integral
33
where 0 = to < t 1 < • • •, tz < T and f(w) is ..Ft -measurable with supi suP,, LA (w )! < oc. One can show that the set Go[O,T] is dense in L(0, T; R). Step 2. Define an integral for any simple process form (5.18): For any t E [ti , t3+1 ] (j > 0), let
f E Go [0, T] of the
kn(t, w) E fi(w)[w(ti+i,w) — w(ti,w)]
(5.19)
i=o + L(w)[w(t,w) —
w(t
•, )].
Equivalently, we have the following:
(5.20)
'VW,
=
EL(w)[w(t
A t2 +1,w)
—
t G [0, 7 ].
W(t A
i>0
It is seen that lis a linear operator from .00 [0, 7 ] to M[0, T]. Moreover, ?has the property that
(5.21)
If II, Vf E Go[O,T].
Î(f)
3. For any f E L2T (0, T; R), by Step I there are fj E £0[0,T] such that Ilf o as co. From (5.21), {I(f)} is Cauchy in .A/e [0, T1. Thus, it has a unique limit in M?[0,T], denoted by r(f). It is seen from (5.21) that this limit depends only on f and is independent of the choice of the sequence h. Hence r(f) is well-defined on L.2F (0,T;IR) and is called the Itô integral, denoted by Step
(5.22)
f (s)dW (s) r(f)(t),
0 < t < T.
Further, for any f E 4(0, T; R) and any two stopping times a and 0 < a < T < T, P-a.s. , we define
(5.23)
T
with
f(s)dW(s) 4 = 1(f)(T) Î(f)( 0).
Now let us collect some fundamental properties of the Itô integral. Proposition 5.2. The Itô integral has the following properties: (0 For any f,g E 4(0,T; R) and stopping times a and T with a
(P-as. ), tAT
(5.24)
Ef
(r)dW (r)11-0.1 -= 0,
,
tAa tAT
(5.25)
tAT
(r)dW (r)][ f
E {[
g(r)dW (r)11..Fa l
tAa
tAa
tAr
=E
f (r)g(r)dri.Fa 1, tAa
P a.s. -
<
T
Chapter 1. Basic Stochastic Calculus
34
In particular, for any 0 <
(5.26)
s < t < T, P a.s. ,
f f(r)dW(r)1.71,1 = 0 , E{
(5.27)
-
f(r)dW(r)] g(r)dW (r)]1.rs } [f t [f t = E{ f f(r)g(r)drI,Fs },
(ii) For any stopping time a and f
P a.s. -
E 4(0, T; R), /et f(t,
f (t,w)i(am>0. Then
L
(5.28)
pt
tAa
f (s)dW (s) =
0
1-(s)dW (s).
See Ikeda-Watanabe [I, pp. 49-51] for a proof. We now extend the It6 integral to a bigger class of integrands than L2F (0, T; IR). To this end, we introduce
L 0c (0, T; R) = {X: [0,T] x St — R X(.) is {.Ft } t >o-adapted (5.29) and f IX(t)1 2 dt < oc, P-a.s. 1, and
(5.30)
1
m2,10c[0 , 7] = PC :
[0,7 ] x ft
RI 3 nondecreasing
stopping times o-3 with P( lim a-3 > T) = 1,
3-03 and Xe A o-3 ) E A4 2 [0, T], V j = 1,2,...}, mc2,1oc,s, [u T1 = {X E M 2 'I"[O,T] 1 t 1--, X (t) continuous, P a.s.}. -
For any f(.) E L2)'(0, T; IR), define
(5.31)
crj (w)
f If (s)I 2 ds
infft E [0,T]
j}, j = 1,2, ....
In the above, we define inf Oa T. Clearly, {o }> is a sequence of nondecreasing stopping times satisfying (4.15). Set f3 (t) f (t)Il o., >tl. Then
LT
(5.32)
a.;
Ih(S)1 26
= 0f
If (s)I 2 ds
j,
which implies f3 (.) E L2F (0, T; R). By (5.28), we have tAa,
pt
fj (s)dW(s) =f j (s)I{,, >,}dW(s) (5.33)
o
=
f (s)I { aj>8} I{, ;>,}dW (s) = 1 ‘ fi (s)dW(s), Vi < j.
f13
0
5. IV) integral
35
Hence, the following is well-defined: (5.34)
jo f (s)dW(s)
fo h(s)dw(s),
Vt E [0, a3 ],
j = 1, 2, ...
integral of f (. ) c T;111). It is easy to see that 2 1' (0 , T; R). f(t) f (s)dW (s) E M e2 'i0c[0, T]for any f E L i We point out that for f(•) E L 27 (0,T; R), (5.24)-(5.27) do not hold in general, but (5.28) remains true. This is called the Itô
Next, we briefly discuss the higher-dimensional case. Let W(t) = (W 1 (t), , W(t)) be an m-dimensional Brownian motion and f 2 (0 , T; P.m). Then, for j = 1, 2, ... , m, fot fj (s)dW 3 (s) • • • , fm) E L F
is well-defined. We define
L
(5.35)
( f (s), dW (s)) g
Lt
t > 0.
f3(s)dW3(s),
It is easy to see that the above defines an element in A/1![0, 7 ] . Similarly, for any a = (a,j ) E L .27 (0,T; R.'n ), we may define an element in 04,2[0, as follows:
(5.36)
L
a(s)dW(s) a
Er-i fot ai3(s)dwJ(s) . rn ft
,
t > 0.
E j=1 0 an3 ( S ) dWi ( 8 )
Due to (2.24), HP(t) and W 3 (t) (i the following result.
j) are independent. Thus, we have
Let f E L g2 -(0,T;Rm) and re-,6 E L3.(0,T;IR. 7" m ). Then, for all 0 <s < t < T,
Proposition 5.3.
fi (r)dWi(r) (5.37)
= 6,3 E {
j
f3 (r)dW 3 (r)i.F.,}
fi (r)f3 (r)driZ},
1
where 6z
{1, 0,
= i
j,
El ( f a(r)dW(r), f Er(r)dW(r))1.Fs }
(5.38)
t = Et
The proof is straightforward.
tr rd(r)T a(r)idr11-5 1.
Chapter 1. Basic Stochastic Calculus
36
We may define LF 23'(0, T; IRm ), M 2,1°10, Tr and A4 2, 30c[0, Tr similarly to (5.29)-(5.30). The 1tô integrals for integrands in L2it0e (0, T; Rm) can be defined accordingly. In addition, we may also define ril°c (0, T; Rm ), p> 1, naturally. The details are left to the reader. To conclude this subsection, we present an important result called the
Burkholder-Davis-Gundy inequality. Theorem 5.4. Let (SI, T, {..Ft}t>o,P) be given as before and let W(t) be an m-dimensional standard Brownian motion. Let a E L 0c (0, T; Rnxm). Then, for any r > 0, there exists a constant Kr > 0 such that for any
stopping time
T,
T 11,,7-E{
(5.39)
itr
f
Icr(s)1 2 ds} < El sup 0
0
KrEf
f0
f cr(s)dW(s) 1
2r
0
10. (8)1 2 CISY.
See Karatzas-Shreve [3, p. 1661 for a proof of Theorem 5.4. 5.3. It6's formula In this subsection we present a stochastic version of the chain rule, or change-of-variable formula, called Itô's formula/lemma/rule, which plays one of the most important roles in stochastic calculus. Let us first recall the chain rule for deterministic functions. Suppose
(5.40)
X(t) = X(0) +I b(s)ds,
t E [0,T],
for some b(.) E L 1 (0, T; Er). Then for any F E C1 ([0,7] x
F(t, X(t))
X(0))
(5.41) .1)
{Ft(s, X (s))+(Fi (s, X (s)),b(s))}ds, t G [0, T ] .
For the stochastic case, we have a similar formula, albeit significantly different from (5.41). Consider the following:
t t X(t) = X(0) + f b(s)ds + f a(s)dW(s), t E [0, T], JO o E L " (0, T; Fr) E L voc o, T; Erxm). with b(.) and a() Any process X(.) of form (5.42) is called an Itô process. (5.42)
Theorem 5.5. (Itô's formula) Let (Q,T, {Ft} t >o,P) be a filtered probability space satisfying the usual condition, W(t) an m-dimensional {Ye}t>oBrownian motion, b(.) E L i i i0c (0,T; Rn ), a E L i 2 i'(0,T; Rn x m), and let
5. Itô
37
integral
R be CI in t and C2 X(•) be given by (5.42). Let F(t,x) : [0, T] x in x with Ft , Fx , and Fxs being continuous such that
(5.43)
F(. X(')), Fx(' X(')) , b( . ) ) G L i i i" (0,T; it), tr {cr(.) T F,x (. , X())cr(.)} E L ii i"(0,T;R),
cre) T F's (. , X()) E L ) 2 0c (0,T;Rni). Then
F(t, X (t)) =F(0, X (0))+ f
IFs (s, X (s))+ ( Fr (s, X (s)), b(s))
+ -1-tr [cr(s)l- Fzx (s, X(s))o - (s)]
(5.44)
2
+1 ( Fi (s, X(s)),cr(s)dW(s)), Vt E Note that for fixed w E St and t > 0, X(s, w) is bounded on [0, t]. Thus, the first integral exists. The second integral is defined as in the previous subsection. Let us make an observation. Let (0,F, {Ft }>o P) be a filtered probability space satisfying the usual condition and let W(.) be a standard Brownian motion. Take cr(•) E 4(0, T; R) and consider
(5.45)
X(t) = f cr(s)dW(s), t E
Then X (. ) E .A/e[O,T]. By
(5.46)
It6's formula above, we have
X(t) 2 fo cr(s)2 ds + 2 fo X(s)o -(s)dW(s), t E
Note that X(•)o - (.) is in L2ii0c (0, T; lit), but not necessarily in L3:.(0,T; R). Hence, the introduction of ha's integral for integrands in L2i10`(0,T;111) is not just some routine generalization. It is really necessary even for as simple a calculus as the above. For a proof of the a more general case).
It6 formula, see Ikeda-Watanabe [1, pp. 66-73] (for
As an application of the It6 formula, we present the following result, which will be frequently used in this book. Corollary 5.6. Let Z and
(5.47)
I
2 be Rn-valued continuous processes satisfying
dZ(t) = b(t)dt + cr(t)dW(t), d2(t) =3(t)dt + (t)dW (t),
Chapter 1. Basic Stochastic Calculus
38
where 6,3, cr,d are Vtlt>o-adapted measurable processes taking values in IRn, and W(t) is a one-dimensional standard Brownian motion. Then
( Z(t), 2 (t) ) = ( Z(0), 2(0)) +i {(Z(s),i(s))+(b(s), 2 (s))+(u(s), 5.- (s))1dt
(5.48)
f
+
o
t { (a(s), 2(s)) + Z(s),&(s)) }dW (s).
This lemma can be easily proved by It6's formula with F(x, y) = ( x, y) for (x, y) E R2n . 5.4. Martingale representation theorems In view of (5.26), the Itô integral
M(t)
(5.49)
f(s)dW(s)
is a martingale (or local martingale). It is natural to ask whether a martingale (or local martingale) can be represented as an It6 integral of the form (5.49). Results on such a problem are called martingale representation theorems. They play very important roles in stochastic calculus itself as well as in stochastic control theory. There are two components on the right-hand side of (5.49), namely, the integrand f and the Brownian motion W(t). The first martingale representation theorem is concerned with how to represent a martingale (local martingale) with a fixed Brownian motion. Theorem 5.7. Let (SI, F, Vtle>o,P) be a filtered probability space satisfying the usual condition. Assume that {.Ft}t>0 is the natural filtration generated by an in-dimensional standard Brownian motion W(t). Let X E M 2 [0, T] (resp. m2,10c[07Tij) Then there exists a unique (p(•) E L(0, T; Rnt ) (resp. L2.711°c (0, T; Rm. )) such that (5.50)
X(t) =
( ço(s), dW(s) ),
Vt E [0,
P-a.s.
A proof can be found in Ikeda-Watanabe [1, pp. 80-831. Recall that Theorem 2.10 shows that if (11, F, {Tt}t>o, P) and W(.) are as in Theorem 5.7, and X(t) is a process adapted to {.Ft}t>o, then X(') can be represented as a functional I/ of W(.). Theorem 5.7 says that if in addition, X(t) is a square integrable ITtIt>o-martingale, then we have a more explicit functional of the Brownian motion W(.), namely, (5.51)
W (t A •)) = X(0) ±
f
(yo(s), dW (s)),
5. It5 integral
39
for some cp (•) E L 2.7-(0 We emphasize that Theorem 5.7 works only for the case where {Ft}e>o is generated by the (given) Brownian motion. On the other hand, it implies that if {-Tt }t>0 is generated by a standard Brownian motion, then any square-integrable {.Ft } t >0-martingale must be continuous, i.e., .A4 2 [0, Ti --=-
A42c [0 5 7] .
Let us now look at an interesting consequence of Theorem 5.7. Let Let 6 E Thus, by Theorem L 7.(0; IR). Then E(et) is an {.Fd t > 0-martingale. 5.7, there exists a z(.) E L(0, T; Rrn) such that
(n,F*,{Yt}t>o,P) and WO be the same as in the above.
(5.52)
E(el.Ft ) = E6 +
( z(s), dW (s)),
t E {0, T].
In particular, çT
6 = E6 + j ( z(s),dW(s) ) . Jo
(5.53) This shows that (5.54)
L T (;R) =R+If
z(s), dW (s)) z (. ) E L(0 ,T; ffen)}.
Next we turn to the question of how to represent a martingale (local martingale) with a fixed integrand. To do this, we first need to define several notions. Let (1, {.Fe}t>o, P) be a filtered probability space satisfying the usual condition. Further, let W(t) be an rre-dimensional standard {F}>0Brownian motion and f E L21-(0, T;111') (resp. f E (0 ,T;IFtn xin)). Define (5.55)
M(t)
a
f (s)dW (s).
By It6's formula, M(t)M(t) T - fc; f (s) f (s) T ds is an {.Ft } t>0 -martingale (resp. local martingale). Inspired by this observation, we have the following definition. Definition 5.8. Let M E A4 2 [0, Ti n (resp. mz,toc[07Tr.) An increasing process A(t) is called the quadratic variation (process) of M(t) if M(t)M(t)T - A(t) is an { 7e}e>0-martingale (resp. local martingale). We denote it by ( M)(t) A(t). The above definition makes sense, as the above A(t) is uniquely determined by Doob-Meyer's decomposition theorem. See Ikeda-Watanabe [1, p. 53] for details. According to Definition 5.8, for M defined by (5.55) we have (5.56)
Lt(M)(t=f (s) f (s) T ds.
Chapter 1. Basic Stochastic Calculus
40
Definition 5.9. Let (St, .F, Vele>o,P) and (fi,.?, {Alt>o, I- ") be two filtered probability spaces satisfying the usual condition. The latter is called an extension of the former if there exists a random variable G : (S2, .F) such that: (i) 19 -1 (.Ft ) C Vt. (ii) P = Po0 1 .= P, (see (1.9)). (iii) for any X E
k(j(CL-J)if.t)P) = E(XI.Ft)( 06)),
13-a.s.
E 6,
where we define i-e(6)) =A-- X(06)), Vw E ft Let {frt}t>o, f") be an extension of (S/,..F, {.Ft } t >o, P). If M is a square-integrable martingale with respect to (Il, F, Vtlt>o, P), then M where M(t, CD) m(t, OtD) is a square-integrable martingale with reI, spect to (fj,.7 - ff-t }t>o, i3 ). Therefore, the space M 2 [0, Tr (M2 ,loc [0 etc.) is naturally embedded into the corresponding spaces M 2 [0, Tr (m2 ,loc [0 'Ton j etc.) in the extension space. For this reason, we shall not distinguish M and M in what follows. Theorem 5.10. Let ( 11,T,Vtlt>0,P) be a filtered probability space satisfying the usual condition. Let M E M 2 [0, T]n (resp. m2 Joe [0 loc (0, T; /Fr < and a G L(0, T; Ift" m ) (resp. m )) with ca l- E T; If R') (resp. L 0c (0,T;R"n)).
(5.57)
(M)(t) =
a(s)cr(s)T ds,
then there exists an extension (f2,..F,{Ale>o,P) of (12, .7., {-Ft }e>o, P) on which there lives an m-dimensional lt>o-Brownian motion W(t) such that
(5.58)
M(t)= f a(s)dW(s).
See Ikeda-Watanabe [1, pp. 90-921 for a proof. Note that in Theorem 5.10 the filtration is not required to be generated by the Brownian motion. On the other hand, if n = m and the matrix a(s) is invertible P-a.s. , for any s, then the representation (5.58) holds on the original (rather than an extension of) (SI, .7- , {Te}e>o,P)•
6. Stochastic Differential Equations In this section we are going to study stochastic differential equations (SDEs, for short), which can be regarded as a generalization of ordinary differential equations (ODEs, for short). Since the Itô integral will be involved, the situation is much more complicated than that of ODEs, and the corresponding theory is very rich.
6. Stochastic differential equations
41
C([0,00);1R 7L) and its metric Let us first recall the space Wn defined in §2.2. Let U be a Polish space and An(U) the set of all {Bt+ (Wn)It>o-progressively measurable processes 77 Apo) x vvu. :
Lemma 6.1. Let b E An(IFV). Let (5/, {Ft }t>o, P) be given satisfying the usual condition and let X be a continuous Rn -valued Vtlt>o-adapted process. Then the process t b(t, X (• ,w)) is {.F e } t >0-adapted. Proof. For any t E [0, 00), let 4> : [0, t] x -> [0, t] x Wn be the map , co)). Then co) (s, 1-> (s,
(6.1)
b(s, X(. , w)) = (b o (I))(s, w),
(s, w) E [0, t] x
By assumption, 19 -1 (B(IV))g{b -I (B)1B E B(R n )} g B[0, t] x Moreover, (1) -1 (8[0,t] x Bt+(W n )) C 13[0, t] x Ft+ = B[0, t] x .Ft . This implies that b(s, X(. ,w)) is B[0, t] x Ft /B(Illn )-measurable, proving the desired result. Next, let b E An (Rn ) and a E An (Rn x m). equation: (6.2)
I
Consider the following
dX(t) = b(t, X)dt + a(t, X)dW(t),
In the above equation, X is the unknown. Such an equation is called a stochastic differential equation. There are different notions of solutions to (6.2) depending on different roles that the underlying filtered probability space (11,F, Velt>o, P) and the Brownian motion W(.) are playing. Let us introduce them in the following subsections.
6.1. Strong solutions Definition 6.2. Let (Q, {.Ft} t>o P) be given, W(t) be a given mdimensional standard {.Ft}t>o-Brownian motion, and e F o -measurable. An trtlt>o-adapted continuous process X(t), t > 0, is called a strong solution of (6. 2) if (6.3)
(6. 4)
X(0) = e,
P-a.s. ,
f { lb (s, X I + la(s, X )I2 Id s < oc, )
(6.5) X (t) = X(0) +
Vt > 0, P-a.s. ,
b(s, X)ds + f a(s, X)dW(s),
t > 0, P-a.s.
o If for any two strong solutions X and Y of (6.2) defined on any given (Q, It>o, P) along with any given standard {Tt } t >0-Brownian motion, we have (6.6)
P(X(t)= Y(t), 0 < t < oc) = 1,
42
Chapter 1. Basic Stochastic Calculus
then we say that the strong solution is unique or that strong uniqueness holds. In the above (6.5), the first integral on the right is a usual Lebesgue integral (regarding w E S2 as a parameter), and the second is the Itep integral defined in the previous section. If (6.4) holds, then these two integrals are ft well-defined. We refer to fot b(s, X)ds as the drift term and o- (s, X)dW(s) as the diffusion term. One should pay particular attention to the notion of strong uniqueness. It requires (6.6) to hold for any two solutions X, Y associated with every given filtered probability space (O, F, {Ft } t>o P) and m-dimensional standard {Ft } t > 0 -Brownian motion, rather than particular ones. So it may be more appropriate to talk about strong uniqueness for the pair (b, a), which are the coefficients of (6.2). See Karatzas-Shreve [3, p. 286] for a discussion on this point. Next we give conditions that ensure the existence and uniqueness of strong solutions. Let (Q, {.Ft } t>o , P) be a filtered probability space satisfying the usual condition, W(t) an m-dimensional standard {.Ft } t>0 -Brownian motion, and an Fo-measurable random variable. We make the following assumption for the coefficients of (6.2).
(H) b E An (Rn ), a E An (IR'), and there exists an L > 0 such that for all t E [0, co), x(•), y(•) E Wn, b(t, Y(*))1 < 140 ia-(t,x(•)) — a (t, Y( . ))I 5 Lx(.) — lbe ,0)1 E L 2 (0, T; it) , VT> O. lb(t,x(*)) (6.7)
I.
We say that b(t, x(•)) and cf(t,x(•)) are Lipschitz continuous in x (.) if the first two inequalities in (6.7) hold. Let (H) hold. Then, for any e E Lly0 (0;IRn) > 1), equation (6.2) admits a unique strong solution X such that for any T> 0,
Theorem 6.3.
(6.8)
/
E sup 1X(s)l e K T (1 + E'en, 05.55T E1X(t)- X(s)l e 5 Kr(1 +Elent s1 /2 ,
Vs, t E [0,7].
Moreover, if E L IF0 (0,1Ftn) is another random variable and Î(t) is the corresponding strong solution of (6.2), then for any T > 0, there exists a KT > O such that (6.9)
E sup IX(s) - Î(s) 0<s
< KrEie
Él'e •
The above theorem can be proved by a routine successive approximation argument and the Borel-Cantelli lemma; see Ikeda-Watanabe [1, pp. 166168] or Karatzas-Shreve [3, pp. 289-290]. However, below we will give
6. Stochastic differential equations
43
another proof, which is based on the contraction mapping theorem (Zeidler [1, p. 17]). Proof of Theorem 6.3. Let T > 0. We are going to prove that (6.2) admits a strong solution on [0, T]. For any 0 < T < T, set (see Section 2)
Xe[O, r] Le,(0 ;
c([o, r] ; Rn))
{x: [0,
(6.10)
T]
x
1R1' x(-) is {..Ft } t >0-adapted,
continuous, and E sup jx(t)le < col. 0
{E{ sup ix(ole] } 7 • 0
For any x(.),y(.) E
(6.11)
[0, r], define
e e { X (t) = e + f b(s , x)ds + f cr(s , x)dW (s), o o
t E [0, r].
Y (t) = + j b(s,y)ds + j cr(s,y)dW(s), Jo
Jo
By (6.7) and the Burkholder-Davis-Gundy inequality (Theorem 5.4), we have
K,
(6.12)
5 KfT 1 ix() - YOl ixdom}.
PC(*)
Here, the constant K is independent of T, e, x(), y (• ). We let r E [0,T] be given such that Kr 1 < 1, where K is in (6.12). From (6.12), it follows that for any E Liyo (SI; Rn), the map x(.) X(.) defined via (6.11) is from Xe[0, 1 to itself and is contractive. Thus, there exists a unique fixed point, which gives a strong solution X(.) to (6.2) on [0,T]. Next, repeating the procedure on [r, 2r], etc., we are able to get the unique strong solution on [0, T]. Since T > 0 is arbitrary, we obtain the strong solution on [0, co). The proof of the remaining conclusions follow easily from the Burkholder-Davis-Gundy inequality. ]
Next, we introduce a special case of SDEs. Let b: [0, oc) x IRn and a : [0, oc) x b(t,w(t)) and -> 111 m . Then the maps (t, w) (t,w) 1-4 Cr(t, WM) are progressively measurable when regarded as maps from [0, oo) x Wn to Rn and R""1 , respectively. In this case, (6.2) becomes
(6.13)
dX(t) = b(t, X (t))dt + cr(t, X (t))dW (t), X(0) = e.
Such an SDE is said to be of Markovian type. If in addition, b and a are time-invariant, then (6.13) is said to be of time homogeneous Markovian
44
Chapter 1. Basic Stochastic Calculus
type. Note that in the case a _= 0, (6.2) is reduced to a functional differential equation and (6.13) is reduced to an ordinary differential equation. Now let us present an existence and uniqueness result for (6.13). First we introduce the following assumption: xm (H)' The maps b: [0 , oo) x Rn and o: [0,00) x Rn are measurable in t E [0, oc) and there exists a constant L > 0, such that
1b(t, x) — (6.14)
+ (t, x) t E [0, 00), x,
lb( , 0)1 + 1c (. , 0)1 E L 2 (0,T; IR),
Lix E Rn ,
VT > O.
Clearly, by regarding b and a as maps from [0, oc) x NV" to Rn and R"m, respectively, we see that (H)' implies (H). Hence, the following result is clear. Corollary 6.4. Let (H)' hold. Then (6.13) admits a unique strong solution.
6.2. Weak solutions Definition 6.5. A 6-tuple (St, F , Vtlt>o, P, W, X) is called a weak solution of (6.2) if
(Q, -7-, {Ft }t>o P) is a filtered probability space satisfying the usual condition; (ii) W is an m-dimensional standard VtIt>o- Brownian motion and X is {Ft} t>cradapted and continuous; (iii) X(0) and e have the same distribution; (iv) (6.4)—(6.5) hold. The essential difference between the strong and weak solutions is the following: For the former, the filtered probability space (ft, .F, {Ft }t>o, P) and the {}t>0 -Brownian motion W on it are fixed a priori, while for the latter (II, F , {F}>0, P) and W are parts of the solution. Definition 6.6. If for any two weak solutions (S1, F, {-Ft } t >o, P, W, X) and of (6.2) with
(6.15)
P(X(0) E B) = P ( 7. (0) E B),
VB E B(Rn),
we have
(6.16)
P(X E A) =
p( -e E A),
VA E B(Wn),
then we say that the weak solution of (6.2) is unique (in the sense of probability law), or that weak uniqueness holds. Definition 6.7. If
(6.17)
P(X(t) = (t), 0 < t < oc) = 1
6. Stochastic differential equations
for any two weak solutions (St, W, .3e) of (6.2) with
45
{Ft }e>o, P, W, X) and (0, T,
p(x(o) =
(6.18)
fc(o))
P,
=1,
then we say that the weak solutions have pathwise uniqueness. Note that in the definition of pathwise uniqueness, 0,T, P, and W are the same for the two solutions under comparison. Existence of weak solutions does not imply that of strong solutions, and weak uniqueness does not imply pathwise uniqueness nor strong uniqueness. See Karatzas-Shreve [3, pp. 301-302] for an example. Relations between the strong and weak solutions are presented in the following two theorems. Theorem 6.8. Let b E An(Rn ) and a E An(R" m ). Then (6.2) admits a unique strong solution if and only if for any probability measure ji on (R n (tt )), (6.2) admits a weak solution with the initial distribution and pathwise uniqueness holds for (6.2). By and large, Theorem 6.8 tells that strong existence and uniqueness is equivalent to weak existence plus pathwise uniqueness. Theorem 6.9. Pathwise uniqueness implies weak uniqueness. See Karatzas-Shreve [3, pp. 308-311] for proofs of Theorems 6.8-6.9. The following is a general existence result of weak solutions for equations with only continuous (not necessarily Lipschitz continuous) coefficients. Theorem 6.10. Let b E An (Rn) and a E An (Rn ') be bounded and continuous. Then there exists a weak solution of (6.2). The proof can be found in Ikeda-Watanabe [1, pp. 155-158]. By the well-known Girsanov transformation, one can obtain the existence and uniqueness of weak solutions for equations with merely measurable drift coefficients. Let us now introduce this approach. (Rn xn Given a E An ) with the matrix a(t, x) invertible for any (t, x), suppose (Il, F, {Fe}t>o,P, W, X) is a weak solution of
dX(t) = a(t, X)dW(t),
(6.19)
X(0) =
Let b E An (Rn ) such that
(6.20)
Eel
ERa—ib)(3,x)12d.
vt E [0, oo).
Then the Girsanov theorem says that (6.21)
m(t)
efo'(.7
1 0(
s,x ) dw w _i Ei(cr—b)(s,x)12ds
Chapter 1. Basic Stochastic Calculus
46
is an {Ft }t>o-martingale, and for each fixed T E [0, 00),
(6.22)
W(t)
W(t) -
(o--1 b)(s, X)ds
is a standard {}t>0 -Brownian motion on (SI, probability measure PT is defined on FT as (6.23)
df"
i3T ), where the new
M(T)dP.
The family of probability measures {P T } is consistent in the sense that PT(A) = I3 T, ( A) for any A E .FT whenever 0 < T < T'. See KaratzasShreve [3, pp. 190-196] for all the proofs of the above claims. By (6.19) and X) is a weak solution of (6.22), it is easily seen that (SI, .F, {.Ft}t>o,
w,
(6.24)
{dX(t) = b(t, X)dt + o - (t, X)ca(t),
Theorem 6.11. Let a E An(R" n ) be bounded and continuous, and let
b E An (lR n ) be bounded. Moreover, assume that o- (t, x) -1 exists for all (t,x) E [0, co) x W" with lc (t , x) -1 1 uniformly bounded. Then (6.2) has a weak solution. Proof. Since a E An (Ftn ) is bounded and continuous, by Theorem 6.10, equation (6.19) has a weak solution (St, {.Ft}t>o,P,W, X). By the assumptions, it is easily verified that (6.20) is satisfied. Therefore, the weak existence of solutions to (6.2) follows from (6.21)-(6.24). 0 In Ikeda-Watanabe [1, pp. 178-180], it is proved that under the assumptions of Theorem 6.11, the weak uniqueness of (6.2) is equivalent to that of (6.19), by a sort of "reverse transformation" of (6.21). As a consequence, we have the following uniqueness result. Theorem 6.12. Under the saine assumptions of Theorem 6.11, if in addi-
tion, (7(t, x) is Lipschitz in x, uniformly in t E [0, Tb then weak uniqueness holds for solutions of (6.2). Proof. Since cr(t,x) is Lipschitz in x, uniformly in t E [0, 7 ] , (6.19) has a unique strong solution by Theorem 6.3. This implies weak uniqueness by Theorems 6.8 and 6.9. Therefore, the conclusion follows from the above observation. For Markovian-type equations, we have the following elegant weak existence result for merely measurable coefficients, due to Krylov. Theorem 6.13. Let 6: [0, oo) x
—> IFtn and a : [0, oc) x IFIn —> Sn be measurable and bounded. Moreover, assume that a is uniformly positive definite, namely, there is .5 > 0 such that (o- (t,x)A, A) > 61Al 2 , VA E Then (6.13) has a weak solution. For a proof see Krylov [2, pp. 87-91].
6.
47
Stochastic differential equations
6.3. Linear SDEs In this subsection we consider the linear SDE
dX(t) = [A(t)X(t) + b(t)frlt + [C(t)X(t) + o - (t)klW(t),
(6. 25)
where WO is a one-dimensional standard Brownian motion and f AO,
( 6.26)
E L(0,T; Rn "),
1 b(•), o- (•) E L2 (0, T; IFC).
From Theorem 6.3, we know that under (6.26), SDE (6.25) admits a unique strong solution. Our main result of this subsection is the following variation of constants formula: Theorem 6.14. For any e E L 2F0 (12;R7'), equation (6.25) admits a unique strong solution X (.), which is represented by the following:
X(t) = (I)(t)e + (DM
f (1, (s) -1 [b(s) - C(s)u(s)]ds
(6.27) + 43(t) f (13(s) - 1 cr (s)dW (s),
t E [0, 71,
where (13(•) is the unique solution of the following matrix-valued SDE:
{
(6.28) and c1(t) - 1
= A(t)(13(t)dt + C(t)(13(t)dW(t), 43(0) = I,
W(t) exists, satisfying
f
(6.29)
&NO
dT(t) = ‘11(t)[- A(t) + C(t)2 1dt -111(t)C(t)dW(t), (0) =- I.
Proof. By Theorem 6.3, we see that (6.28) admits a unique solution (DO. To show that 43(t) -1 exists, let WO be the unique strong solution of (6.29), which exists by, once again, Theorem 6.3. Applying Itô's formula to 43(t)*(t), one easily obtains that d[(1)(t)tli(t)] = O. Hence (I)(t)W(t) I, leading to 111 (t) = (13(t) -1 . Next, applying Itô's formula to xii(t)X(t), where X(t) is the unique strong solution of (6.25), we obtain
d[W(t)X(t)] = tli(t)(b(t)
-
C(t)a(t))dt + W(t)o- (t)dW(t).
This yields (6.27), where we note the fact that W(t) = (I)(t) -1 . Let us now briefly discuss the case where W(•) is multidimensional. In this case, we can write the linear equation as
(6.30)
I
trl
dX (t) = [A(t)X(t) + b(t)] dt +E [C2 (t)X (t) + j=1
X(0) = e.
Chapter 1. Basic Stochastic Calculus
48
Let 43(t) be the solution of the following:
&DM = A(t)(1)(t)dt + (6.31)
E c,(t)4(t)dw3(t), 3 =1
We can similarly prove that (DM' exists, which is the solution of Id(4)(t) -1 ) = (1)(0 -1 [- A(t)+
E C3 (t)) 2]dt 3 =1
m
(6.32)
— ») - lc, "c.13 (t)dw3 (t), ,=1 = I.
The strong solution X of (6.30) can be represented as X(t) =(13(t ) +
f (I)(s) -1 [b(s) -
E C3 CI (s)lds 3 =1
(6.33)
t
m
+ E 4)(t) f 04)(s)-1,(s)dwi(s). 3
=1
6.4. Other Types of SDEs In this subsection we discuss two types of SDEs that are encountered in stochastic control problems. One of them does not fit into the form of (6.2), while the other one does after a certain transformation. We note that in equation (6.2) the coefficients b and a do not depend on the sample co explicitly (they do implicitly through the solution X0). In the case where these coefficients are dependent on co explicitly, we have SDEs with random coefficients:
(6.34)
I
dX(t,w) = b(t, X, co)dt + a(t, X, co)dW (t, w), X (0, or) =
Since the coefficients b and a should be given a priori, equation (6.34) should be defined on a given probability space (12, .F, P), where b and a are defined. In other words, for an SDE with random coefficients, it is not sensible to have the notion of weak solutions. Definition 6.15. Let the maps b : [0,co) x Wn x f2 Ifin and RM nX a : [0, oc) x W'' x SZ — 4 be given on a given filtered probability space (9, F , {Ft } t>o, P) and an m-dimensional standard {Ft }t>o-Brownian motion W(t). Let be .F0-measurable. An {Tt } t >0-adapted continuous process X(t), t > 0, is called a solution of (6.34) if
(6.35)
X(0) =
P-a.s.,
6. Stochastic differential equations
49
fib(s, X(w),L ,J)1+ icr(s, X (w), w)I 2 }ds < co,
(6.36)
t > 0, P a.s.w E -
(6.37)
X(t,w) = e(w) + f b(s, X(w),w)ds + f o - (s, X (w), cv)dW (s, w), t > 0, P a.s. w E ft -
If
P(X(t) = Y(t), 0 < t < oc) = 1,
(6.38)
-
holds for any two solutions X and Y of (6.34) defined on the given probability space (0, F , {Tt}t>o, P) along with the given {.Ft } t>0-Brownian motion W(t), then we say that the solution is unique. Note the differences between Definitions 6.2 and 6.15. First of all, one does not need to distinguish between the strong and weak solutions for (6.34), so we simply term them as "solutions". Second, the uniqueness in Definition 6.15 is based only on the given (S1, .F, {.Ft}t>o, P) and W(t) rather than on any given (SI, .F, {Ft } t>o,P) and W(t) as in Definition 6.2. Let us now introduce the following assumption.
(1r Xrn) (RC) For any w E b(• , • , to) E An(Rn ) and o-e , • , E An and for any x E Wn, b(• ,x,.) and a (. ,x,-) are both {.Ft } t >0-adapted processes. Moreover, there exists an L > 0 such that for all t E [0, co), x(•), y(.) E Wn , and w E lb(t, x(.), w) — b(t, y(•),w)l< Lx() — (6.39)
s
la(t,x0,w)
,
cf(t,y('),w)1 5_ Llx(.) — YOlvvn, E Ly 2 (0,T;R), VT> O.
,o,.)I + la( ,o,•)1
Theorem 6.16. Let (RC) hold. Then, for any e E LtF0 (S2; Rn ) ( > 1), (6.34) admits a unique solution X such that for any T > 0,
(6.40)
E max IX(s)1 1
and
(6.41)
EIX(t) — X(s)I ( < KT(1 + Eie)It — si t/ 2 , Vs, t E [0,7 ].
Moreover, if e- E Rn ) is another random variable and Î(t) is the corresponding solution of (6.34), then for any T> 0, there exists a KT > 0 such that
(6.42)
E max IX(s)— 5e(s )I 0<s
KTEle —
The proof of this theorem is the same as that of Theorem 6.3.
Chapter 1. Basic Stochastic Calculus
50
Next, we discuss another type of equations that will play a vital role in establishing an appropriate formulation for optimal stochastic control problems. Consider the following SDE:
(6.43)
{ dX (t) = b(t, X, W)dt + cr(t, X, W)dW(t), X(0) = e,
where b and a are defined on [0, col x Wn x Wm. This equation is not W(t), readily seen to be in the form of (6.2). However, if we define Y(t) then the above equation is equivalent to
dX(t) = b(t, X, Y)dt + a(t, X ,Y)dW(t), dY(t) = dW(t), X(0) = e, Y(0) = O.
(6.44)
This is a special case of (6.2). Thus all the notions of strong and weak solutions, their uniqueness and pathwise uniqueness, apply to (6.43). Let us make the following assumption.
(W) b E An+m (iRn ), a E An+m (IR" m ), and there exists an L > 0 such that for all t E [0, co), x0, y (. ) E Wn and w(-) E Wm, x(*), w( . ))
(6.45)
—
b(t, y(*), w(.))1
/40
—
yOlvvn, x(.), w 0)1 < K(1 + lx()Iwn).
la(t,x('),w0) a(t,y0,w0)1< Lix( . )
164, x(.), w(.))1 +
Under (W), equation (6.43) has a unique strong solution. As a consequence, both pathwise uniqueness and weak existence and uniqueness hold.
Theorem 6.17.
Proof. Given any filtered probability space (S-2,Y, {.Ft}t>o, P) along with an m-dimensional {.Ft } t > o-Brownian motion W(.), Y(t) W(t) will then be fixed. Then by (W) and the same arguments employed in proving Theorem 6.3, we obtain the existence and uniqueness of X(.), which, together with Y(.), solves (6.44) or, equivalently, (6.43). The second assertion follows then from Theorems 6.8 and 6.9. We emphasize that in (W), no continuity is assumed for the coefficients in the variable w(.). Strong uniqueness still holds due to the particular role of W(.).
Chapter 2
Stochastic Optimal Control Problems
1. Introduction Uncertainty is inherent in most real-world systems. It places many disadvantages (and sometimes, surprisingly, advantages) on humankind's efforts, which are usually associated with the quest for optimal results. The systems mainly studied in this book are dynamic, namely, they evolve over time. Moreover, they are described by It6's stochastic differential equations and are sometimes called diffusion models. The basic source of uncertainty in diffusion models is white noise, which represents the joint effects of a large number of independent random forces acting on the systems. Since the systems are dynamic, the relevant decisions (controls), which are made based on the most updated information available to the decision makers (controllers), must also change over time. The decision makers must select an optimal decision among all possible ones to achieve the best expected result related to their goals. Such optimization problems are called stochastic optimal control problems. The range of stochastic optimal control problems covers a variety of physical, biological, economic, and management systems, just to mention a few. In this chapter we shall set up a rigorous mathematical framework for stochastic optimal control problems. The remainder of this chapter will be organized as follows. In Section 2, the formulation of deterministic optimal control problems will be recalled. This practice of beginning with the deterministic case will be carried out in later chapters whenever appropriate. The reason for doing so is not only that the deterministic situation itself may contain interesting results, but also that readers can see the essential differences between the deterministic and stochastic problems. In Section 3, several practical examples arising in manufacturing, finance, insurance, and industry that can be formulated as stochastic optimal control models are given. Section 4 presents strong and weak formulations of stochastic optimal control problems that will be mainly dealt with throughout this book. Section 5 is concerned with the existence of stochastic optimal controls for both strong and weak formulations. In Section 6 we demonstrate that there are fundamental differences between the stochastic and deterministic cases in terms of reachable sets, which are closely related to the existence of optimal controls as well as controllability. Section 7 introduces some stochastic optimal control models other than the one formulated in Section 3. These models, including optimal stopping, singular controls, risk-sensitive controls, and partially observed models, have been widely studied in the literature. Although they are not to be investigated in this book, the two approaches, namely Pontryagin's maximum principle and Bellman's dynamic programming, are also the principal ones to solve those problems. Finally, Section 8 gives historical remarks.
52
Chapter
2.
Stochastic Optimal Control Problems
2. The Deterministic Cases Revisited We recall deterministic optimal control problems in this section. Let us start with a production planning problem. A machine is producing one type of product. The raw materials are processed by the machine, and the finished products are stored in a buffer. Suppose at time t the production rate is u(t) and the inventory level in the buffer is x(t). If the demand rate for this product is a known function z(t) and the inventory is xo at time t = 0, then the relationship among these quantities can be described by
(2.1)
i(t) = u(t) x(0) = xo.
-
z(t),
t
> 0,
Here x(t) may take either positive or negative values. The product has a surplus when x(t) > 0, and a backlog when x(t) < O. Suppose the cost of having the inventory x and production rate u per unit time is h(x, u). A typical example of h is
(2.2)
h(x , u) = c+ x+ + c- x - + pu ,
where x+ max{x, 0}, x - max{-x, 0}, c+,c- > 0 are the marginal cost/penalty for surplus and backlog, respectively, and p is the unit cost of production. The production management wants to choose a u(.) so as to minimize the total discounted cost over the planning horizon [0, T]. Namely the following functional is to be minimized:
(2.3)
(u0) = f e - t h(x (t), u(t))dt ,
where 7 > 0 is the discount rate. Note here that the decision u (. ) is a function on [0, TI, which is called a production plan. If the machine has a maximum production rate k (called the production capacity), then any production plan must be subject to
(2.4)
0< u(t) < k, Vt E [0, 71.
On the other hand, if the buffer size is b> 0, then the inventory level x(t) must satisfy the constraint
(2.5)
x(t) 5_ b.
Any production plan that satisfies (2.4)-(2.5) (via (2.1)) is called an admissible plan. The problem is to minimize the cost (2.3) over all admissible plans. The above problem is perhaps one of the simplest examples of an optimal control problem. It is deterministic because there is no uncertainty in the system dynamics (2.1) and the constraints (2.4)-(2.5). Now we present the general formulation of (deterministic) optimal control problems. Let T, 0 < T < +co, xo c Rn , and a metric space F
2. The deterministic cases revisited
53
be given. The dynamics of a system under consideration are given by the following ordinary differential equation: ±(t) = b(t, x(t), u(t)),
(2.6) {
a.e. t E [0, 7 ],
x(0) = xo,
IRn is a given map. When T = oc, [0, T] above where b: [0, T] X Rn X r should be replaced by [0, co). A measurable map u(.) : [0, T] r is called a control, x o is called the initial state, and x (.), a solution of (2.6), is called a state trajectory corresponding to u(.). In the production planning example, the dynamics are (2.1), the control is the production rate, and the state is the inventory level. Hereafter, we assume that for any xo E Rn and any control u(.), there exists a unique solution x (.) x(• ;ti(.)) to (2.6). In such a case, we obtain an input-output relation with input u(.) and output x (.). We also call (2.6) a controlled, or control, system. A particular case of (2.6) is the linear one, where the controlled system is of the form (2.7)
I
X(t) = A(t)x(t) + B(t)u(t),
a.e. t E [0, T],
x(0) =- xo,
Ilr" being measurable maps. with A : [0, T] > Et"' and B : [0,T] Sometimes we call (2.7) a time varying linear system, since A(.) and B(.) are time-dependent. Further, the following system is called a time invariant linear system: —
-
-
X(t) = Ax(t) + Bu(t),
(2.8) {
a.e. t E [0, T1,
x(0) = xo,
where A E Rnxn and B E Rn" . As we have seen in the production planning example, there may be various constraints involved. For example, the state constraint could be given by (2.9)
x(t) E S(t),
Vt E [0, T],
and the control constraint given by (2.10)
u(t) E U(t),
a.e. t E [0, 7 ],
where S(t) : [0, T] 211n and U(t) : [0, T] 2" are given multifunctions (i.e., for each t E [0, 7], S(t) C ffi," and U(t) c r). Some other types of constraints (e.g., constraints in an integral form) are also possible. In this book we treat only the case where the control constraint U(t) is time invariant, namely, U(t) U. Remember that U itself can be regarded as a metric space. Thus, from now on, we replace r by U. Define V[0,71 afu: [0,7
]
U u(.) is measurable}.
Chapter 2. Stochastic Optimal Control Problems
54
Any u (. ) E V[0, T] is called a feasible control. Next, we are given a cost functional measuring the performance of the controls:
(2.11)
J (u(-)) = f f (t, x(t), u(t))dt + h(x(T))
R. The first and R and h Rn for given maps f : [0, TI x R x U the running cost right-hand side of (2.11) are called second terms on the respectively. and the terminal cost, Definition 2.1. A control u(-) is called an admissible control, and (x (.), li(.)) called an admissible pair, if:
(i) u(.) E V10,71; (ii) x(.) is the unique solution of equation (2.6) under u(.); (iii) the state constraint (2.9) is satisfied; (iv) t f (t, x(t), u(t)) E L 1 [0, T]. The set of all admissible controls is denoted by Vad[O, T]. Our deterministic optimal control problem can be stated as follows. Problem (D). Minimize (2.11) over Vad[O, Problem (D) is said to be finite if (2.11) has a finite lower bound, and is said to be (uniquely) solvable if there is a (unique) ii(.) E Vad[0, T] satisfying
(2.12)
J((.)) =
inf
u()EV.d[0,71
J(u()).
Any 77,(.) E Vad[O,T] satisfying (2.12) is called an optimal control, which the corresponding state trajectory TO ; U(-)) and (Y(.), T1(.)) are called an optimal state trajectory and an optimal pair, respectively. A particularly interesting and important case is that in which the controlled system is given by (2.7), the state constraint (2.9) is absent (i.e., there is no state constraint, or S(t) Rn ), the control constraint U is Rk , and the cost functional is of the form 1 T J (u) • f [ ( Q (t)x (t) , x(t)) + ( R(t)u(t) , u(t))]dt
(2.13) +
1 —
( Gx(T), x(T) ),
2
for some suitable symmetric matrix G and symmetric matrix-valued functions Q(.) and R(.). The corresponding Problem (D) is called a linear quadratic optimal control problem (LQ problem, for short). Finally, one may note that in the production planning example the terminal cost h is equal to O. In general, an optimal control problem with h = 0 is called the Lagrange problem, that with f = 0 called the Mayer problem, and that with f 0, h 4 0 called the Bolza problem. It is well known (and easy to show) that these three problems are mathematically equivalent.
3. Examples of stochastic control problem
55
3. Examples of Stochastic Control Problems This section is devoted to a presentation of various practical problems that can later be categorized into one model, namely, the stochastic optimal control model under consideration throughout this book.
3.1. Production planning Consider the optimal production planning problem with the presence of uncertainties in demand. The uncertainties are described by a filtered probability space (f2,..F, {Ft}t>o,P) satisfying the usual condition, on which a one-dimensional standard Brownian motion W(t) is defined. The demand process z(t) is no longer deterministic. Rather, it is given by the following: pt
(3.1)
z(t) = zo +
e(s)ds + j o- (s)dW (s), t E [0,7]. Jo
Here, e(t) represents the expected demand rate at time t, and the term 0-(s)dw(s) represents the fluctuation of the demand due to the uncertainty of the environment. Note that the notion of "demand" is in a rather broad sense: Any withdrawal from or addition to the inventory can be regarded as an action of the demand. Therefore, it may cover those processes such as sales return and inventory spoilage, which are in general quite random. We assume that both e(.) and a() are Vtlt>o-adapted, and the integrals in (3.1) are well-defined. To meet the demand, the factory has to adjust its production rate all the time to accommodate changing situations. Suppose the machine is reliable (i.e., it will never break down) and, as in the deterministic case, at time t the production rate is u(t) and the inventory level in the buffer is x(t). If at time t = 0 the inventory is xo , then the system can be modeled as
fo
(3.2)
dx(t) = (u(t) — z(t))dt,
x(0) = xo ,
dz(t) = e(t)dt + a(t)dW (t), z(0) = zo•
Once again, the control or the production rate u(.) is subject to the production capacity, namely,
(3.3)
0 < u(t) < k,
a.e. t E [0,71,
P-a.s.
There is another implicit constraint on the production policy u(-) due to the stochastic environment of the problem, namely, at any time the production management makes a decision based only on past information rather than any future information. Therefore, the control or decision must be nonanticipative. The precise mathematical translation of this constraint will be given in Section 4. Here, in order to avoid an infinite backlog, the maximum production capacity should be large enough overall to meet the demand. So the following minimum condition should be imposed for the problem to be meaningful: T pt
(3.4)
kT > E] z(t)dt zoT + E] o
(s)ds dt. o
Chapter 2. Stochastic Optimal Control Problems
56
On the other hand, the inventory state should not exceed the buffer size h:
V t G [0,T], P-a.s.
x(t) < b,
(3.5)
The expected total (discounted) cost is the following:
(3.6)
J(u(•)) = Et fo
f (x(t),u(t))dt +
h(x(T))},
where the first term represents the total running cost for inventory and production, the second term is a penalty of the inventory left over at the end of the production horizon (e.g., disposal cost), and -y > 0 is the discount rate. The objective of the production management is to choose a suitable production plan u(.) so that (3.2), (3.3), and (3.5) are satisfied and J(u()) is minimized.
3.2. Investment vs. consumption Suppose there is a market in which n + 1 assets (or securities) are traded continuously. One of the assets is called bond, whose price process Po (t) is subject to the following (deterministic) ordinary differential equation:
dPo(t) = r(t)Po(t)dt,
(3.7)
a.e. t E [0, T],
P0(0) = Po >0,
where r(t) > 0 is called the interest rate (of the bond). Clearly Po (t) will grow steadily for sure as time goes by, and the bond is therefore called a riskless asset. The other n assets are called stocks, whose price processes Pi (t), . ,Pm (t) satisfy the following stochastic differential equation:
(3.8)
{ dPi (t)
Pi (t){6,(t)dt + (o-t (t),dW(t))1, t E [0,T ],
P,(0) = p, > 0,
xÇ with b(t) > 0 (almost surely) is called the appreciation rate, and cr, : [0,7 ] x Rni is called the volatility or the dispersion of the stocks. All these processes are assumed to be {.T}>0adapted. Here, W(t) is an m-dimensional standard Brownian motion defined on some fixed filtered complete probability space ((2,1-, {.Ft}t>o, P). The diffusion term in (3.8) reflects the fluctuation of the stock price-s, and the stocks are therefore called risky assets. Usually, one has where b, : [0,
(3.9)
Ebt (t) > r(t) > 0,
Vt E [0, Tb
1
This is a very natural assumption, since otherwise nobody is willing to invest in the risky stocks. However, we point out that the following discussion does not depend on this assumption. We now consider an investor whose total wealth at time t > 0 is denoted by x(t). Suppose he/she decides to hold N(t) shares of the ith asset (i =
3. Examples of stochastic control problem
57
0,1, . . . , n) at time t. Then x(t) =
(3.10)
E Ni (t)P,(t),
t > O.
Suppose that the trading of shares and the payment of dividends (at the rate it(t) per unit time and per unit of money invested in the ith stock) takes place continuously. Further, we let c(t) be the rate of withdrawal for consumption. Then one has X(t + At) — x(t) =
E Ni (t)[Pi (t + At) — Pi (t)] i=o
(3.11)
+ Letting At
—■
tii(t)Nz (t)Pi(t)At — c(t)At.
0, we obtain
dx(t) = ENi (t)dPi (t) +
E iii (t)Ni (t)Pi (t)dt — c(t)dt
r(t)No (t)P0 (t) +
E [b(t) + Iii (t)J1Vi (t)Pi (t) — c(t)} dt i=1
Ni (t)Pi (t) cri (t), dW (t))
(3.12) i= 1 =
r (t)x(t) + E [b(t) i= 1
+,i(t) _ r(t)]ui (t) — c(t)} dt
Eaicoui (t), dW (t)), i=1 where
(3.13)
u(t) = Ni(t)Pi (t),
i = 0, 1, 2, ... , n,
denotes the market value of the investor's wealth in the ith bond/stock. When u•(t) < 0 (i = 1,2, ... , n), the investor is short-selling the ith stock. When uo(t) < 0, the investor is borrowing the amount luo(t)i with an interest rate r(t). It is clear that by changing u(t), the investor changes the "allocation" of his/her wealth in these n + 1 assets. We call a u(t) =(u i (t), , un (t)) a portfolio of the investor. Note that we do not include the allocation to the bond in the portfolio, as it will be uniquely determined by the allocation to the stocks, given the total wealth. As in the stochastic production planning case, any admissible portfolio is required to be nonanticipative. Now, for x(0) = xo > 0, the investor wants to choose the investment portfolio u (. ) and the consumption plan c(.) such that
(3.14)
x(t) > 0,
Vt E [0, T1,
P-a.s. ,
Chapter 2. Stochastic Optimal Control Problems
58
and such that the stream of discounted utility, (3.15)
J (u(•), c(.)) =
fo e - "it yo(c(t))dt +
h(x(T))} ,
is maximized, where 'y > 0 is the discount rate, cp (c) is the instantaneous utility from consumption c, and e --Y T h(x(T)) is the (discounted) utility that is derived from bequests. We may impose some further constraints for the above problem. For example, one constraint may be that short-selling has to be bounded. Then we will have the constraints
u(t)> -L,,
Vt e [0,7 ] ,
P-a.s. , j = 0, 1, . . . , n,
for some L, > 0. That L, = 0 means that short-selling is prohibited. 3.3. Reinsurance and dividend management We consider a model of an insurance company, which can choose a reinsurance policy to manage risks. In addition, there is a choice of the amount of dividends to be paid out to shareholders. Notwithstanding any policy decision, there is a constant payment of a corporate debt, such as bond liability or loan amortization. It is a classical approach to assess the value of a company based on the total dividend payouts. Therefore, the objective is to find the reinsurance and dividend policy that maximize the expected total discounted dividend payouts over a time period. To formulate the model, consider the company's cash reserve. There are three different components that affect it. The first two are deterministic— the payment of premiums by customers and debt repayment by the company, both happening at a constant rate. The third is payments on claims. If we denote the number of claims received up to time t by A(t) and the size of the ith claim by h, then R(t), the company's reserve at time t, is given by A(t)
(3.16)
R(t) = R(0) +pt - 6t -
E
Here p is the amount of premiums received per unit time, and 6 is the rate of debt repayment. If A(t) is a Poisson process with intensity A and all claims are i.i.d, then R(t) can be approximated by a Brownian motion with a drift p - 6 - AET/ and a diffusion coefficient a = Reinsurance allows the company to divert a proportion 1 - a of all premiums to another company, and consequently, the fraction 1 - a of each claim is paid by the other company. In this case, in equation (3.16) describing the reserve, p is replaced by ap and ij by an,. In the limiting diffusion model the quantity p - AEn in the drift term is changed into a(p - AEn), while a is changed into aa. With reinsurance one reduces the risk incurred by the company at the expense of reducing the potential
3. Examples of stochastic control problem
59
profit. This explains why reinsurance can be considered in the framework of risk management. With the above motivation we describe our model as follows. Denote the value of the liquid assets of the company at time t by x(t). Let c(t) be the dividend rate paid out to the shareholders at time t. Then the dynamics of x(t) are given by (3.17)
f
dx(t) -= [a(t)p — 6 — c(t)]dt + a(t)o-dw(t), x(0) = xo.
Here p is the difference between the premium rate and the expected paya(t) is the ments on claims per unit time (called the safety loading), 1 reinsurance fraction at time t, and xo is the initial value of the liquid assets of the company. By the nature of the problem, the following constraints must be satisfied: -
(3.18)
x(t) > 0,
0 < a(t) < 1,
Vt E
T].
The objective of the management of the company is to choose the dividend payout scheme c(.) and the risk management policy a(.), both being nonanticipative, such that (3.18) is satisfied and the following expected total dividend payouts over [0, T] is maximized: E
(3.19)
e -14 c(t)dt, o
where -y > 0 is the discount rate.
3.4. Technology diffusion There are three phases in the life cycle of a new technology: research and development (R&D), transfer and commercialization, and operation and regeneration. Technology diffusion refers to the transition of technology's economic value during the transfer and operation phases of a technology life cycle. The modeling of technology diffusion must address two aspects: regularity due to the mean depletion rate of the technology's economic value, and uncertainty due to disturbances occurring in technological evolution and innovation. Therefore, stochastic differential equations will be a good framework to model technology diffusion problems. Suppose the process of adopting a new technology starts at time 0 with an initial economic value xo, and the value depletion process x(.) of the technology satisfies the following Itô's stochastic differential equation on a filtered probability space (S2, F , {Ft} t>o, P): (3.20)
{dx(t) = b(t , x(t), u(t))dt + (t, x(t), u(t))dW
Here b is the mean depletion rate, and a- represents exogenous disturbances. The initial value x o is assumed to be deterministic and positive, which
Chapter 2. Stochastic Optimal Control Problems
60
signifies that the underlying technology must have a definite and positive value measured according to the best information available at the initial time. By the nature of depletion, the drift b must be decreasing in time (i.e., Ob(t tx,u) < 0). On the other hand, 74.) represents the company's policies in influencing the depletion rate and disturbances in the process of technology diffusion, such as technology utilization degree, learning and training, budgeting, and resource allocation. A typical drift will be
(3.21)
b(t,x,ui,u2) =
ulx
1 + u2x
where u 1 is the natural depletion rate per unit of technology utilized (which reflects the degree of technology utilization) and u2 is the compensation of depletion by the learning and training effect in terms of knowledge, skill, and new developments. As for the disturbance cr(t , x, u), it is reasonable to assume that it diminishes as the economic value of the technology goes down. Namely, we require
u(t, 0, u)
Oa t, x, u) (
0,
ax
> 0.
The exact form of a depends on different situations. If the uncertainty goes down at a roughly constant rate related to the value of the technology, then a function such as
(3.22)
,(t,x,u) =
would be a suitable one. If the uncertainty does not vary much at the initial stage and falls significantly at some later stage, then it is more reasonable to use a function such as
(3.23)
a(t,x,u) = aV1 — 71e — Aux ,
where a, À are positive constants and n is constant smaller than but close to one. The objective of the model is to choose an appropriate nonanticipative policy u(.) so as to maximize the following expected total discounted profit in a time period [0, 7]:
(3.24)
J (u0)
E
f (t, x(t), u(t))dt +
h(x(T))} ,
where -y > 0 is the discount rate, f is the profit rate, and e —rT h(x(T)) is the salvage value of the technology at the end time. 3.5. Queueing systems in heavy traffic A queueing system in heavy traffic (namely, the average customer arrival rate is close to the average server service rate) can be approximated by a (nonstandard) Brownian motion using so-called diffusion approximation. Therefore, a controlled queueing system can be approximately formulated
3. Examples of stochastic control problem
61
as a stochastic control problem, and sometimes it is advantageous from both the analytical and computational points of view. Let us consider a particular single-server queueing system, a manufacturing system with a single unreliable machine (server) producing a single type of product to meet incoming orders (customers). Let {ai } and {A}, j = 1, 2 , be two sequences of i.i.d.random variables representing successive up and down periods of the machine, and let {C} and 0, 1, i = 1, 2, ..., be two sequences of i.i.d.random variables representing successive interarrival times of demand batches and their sizes, respectively. Suppose
(3.25)
Ea, =
1 -
1 1 1 , E,3, = T) , E& = , Enz = ,
a Var a, = a-2 , Var 0, = A2 , Var e, = p2 ,
Var ii, -= 62 .
Let X(t) be the inventory of the product and U(t) the production rate at time t. Suppose that A(t) is the cumulative number of arrivals of the demand batches up to time t, and
1, if the machine is up,
G( t)= 0, if the machine is down. Then the inventory process satisfies A(t)
(3.26)
X(t) = xo +
Jo
G(s)U(s)ds
-
E z=1
A production plan U(.) must satisfy the constraint 0 < U(t) < k, where k is the maximum production capacity. The objective of this controlled queueing system problem is to select a nonanticipative production plan U(.) so as to minimize the expected total discounted costs of inventory and production:
(3.27)
e —lt h(X(t),U(t))dt,
J(U(•)) = E o
where -y > 0 is the discount rate. This actually is a very hard problem, and a closed-form solution seems impossible to obtain even for simple forms of the cost function h. Now let us approximate this problem by another one involving a diffusion process for which there are more convenient tools to apply. To this end, we first note that the expected long-run production capacity is ± a b k, and the expected long-run demand rate is 5. Therefore, the traffic intensity of the queueing system is
(3.28)
k b) p (cd ) -1 ( ab+
Usually, p is assumed to be greater than 1; otherwise, the backlog will grow to infinity in the long run. The queueing system is said to be in heavy
Chapter 2. Stochastic Optimal Control Problems
62
traffic if p is close to 1, namely,
0<E
(3.29)
bk
c
a+b
d
%■-% 0.
When the original system is in heavy traffic, its optimal control problem can be approximated by the following problem: T {
(3.30)
minimize J(u (-)) = E f e -11 h(t, x(t), u(t))dt, Jo {
dx(t) = (1 - u(t))dt + gdW (t),
subject to
x(0) = Ezo, where the diffusion coefficient is
g = k2 (a + b)-3 (a3 ba2 + ab3 A2 ) + c3d-2112 + c52 .
(3.31)
4. Formulations of Stochastic Optimal Control Problems The examples presented in the previous section have some common features: There is a diffusion system, which is described by Itô's stochastic differential equation; there are many alternative decisions that can affect the dynamics of the system; there are some constraints that the decisions and/or the state are subject to; and there is a criterion that measures the performance of the decisions. The goal is to optimize (maximize or minimize) the criterion by selecting a nonanticipative decision among the ones satisfying all the constraints. Such problems are called stochastic optimal control problems. Let us now present two mathematical formulations (strong and weak formulations) of stochastic optimal control problems in the following two subsections, respectively.
4.1. Strong formulation Given a filtered probability space (fl, {Ft} t>o P) satisfying the usual condition (see Chapter 1, Definition 2.6) on which an m-dimensional standard Brownian motion W(-) is defined, consider the following controlled stochastic differential equation:
(4.1)
f
dx(t) = b(t, x(t),u(t))dt + a (t , x(t), u(t))dW (t), x(0) = xo E
/I n x where b : [0, T] X Rn X U IFtn , a : [0 , T] X Rn X U with U being a given separable metric space, and T E (0, co) being fixed. The function 14.) is called the control representing the action, decision, or policy of the decision-makers (controllers). At any time instant the controller is knowledgeable about some information (as specified by the information field {Yt } >0 ) of what has happened up to that moment, but not able to foretell what is going to happen afterwards due to the uncertainty of the system
4. A formulation of stochastic optimal control problems
63
(as a consequence, for any t the controller cannot exercise his/her decision u(t) before the time t really comes). This nonanticipative restriction in mathematical terms can be represented as "u (. ) is {F}>0 -adapted" (recall Convention 2.9 in Chapter 1). Namely, the control u (. ) is taken from the set
Li[0,T]'a fu : [O, T]
U 1'4') is {.Ftlt>o adapted} . -
Any u(-) E U[0,7 is called a feasible control. In addition, we may have some state constraints. For any u (. ) E U[0,7], equation (4.1) is one with 21R' random coefficients (see Chapter 1, Section 6.4). Let S(t) : [0, T] be a given multifunction. The state constraint may be given by ]
x(t) E S(t),
(4.2)
P-a.s.
Vt E [0,
Note that some other types of state constraints are also possible. Next, we introduce the cost functional as follows:
(4.3)
J (u(•)) =E{ fo f (t, x(t),u(t))dt + h(x(T))}
Definition 4.1. Let (St, F, I>o, P) be given satisfying the usual conditions (see Chapter 1, Definition 2.6) and let W(t) be a given rri-dimensional standard Vilt>o-Brownian motion. A control u (. ) is called an s admissible control, and (x(.), u(.)) an s-admissible pair, if -
(i) u(•) E Li[0, 71; (ii) x (.) is the unique solution of equation (4.1) in the sense of Chapter 1, Definition 6.15; (iii) some prescribed state constraints (for example, (4.2)) are satisfied; (iv) f( . , x(.), li(.)) E L T1 (0 ,T ; R) and h(x(T)) E L, (1; IR). The set of all s-admissible controls is denoted by Lias d [0, T]. Our stochastic optimal control problem under strong formulation can be stated as follows: Problem (SS). Minimize (4.3) over ti:d [O, The goal is to find ri(.) E Uas d [O, T] (if it ever exists), such that
(4.4)
J(Ti(•)) =
inf
u(•)Eu:d [o,7]
J(u(.)).
Problem (SS) is said to be s finite if the right-hand side of (4.4) is finite, and it is said to be (uniquely) s solvable if there exists a (unique) FLO E Liasd [0, 7 such that (4.4) holds. Any Ti(.) G Ltasd [0, T] satisfying (4.4) is called an s optimal control. The corresponding state process Y (. ) and the state-control pair (Y (. ), Ti ( )) are called an s optimal state process and an s optimal pair, respectively. -
-
]
-
-
-
Chapter 2. Stochastic Optimal Control Problems
64
4.2. Weak formulation We note that in the strong formulation, the underlying filtered probability space (12,..F,Vtlt>o,P) along with the Brownian motion W(.) are all fixed. However, in certain situations it will be convenient or necessary to vary (R.F,{Ft } 1>o, P) as well as W(.) and to consider them as parts of the control. This is the case, for example, when one applies the dynamic programming approach to solve a stochastic optimal control problem originally under the strong formulation (see Chapter 4 for details). Therefore, we need to have another formulation of the problem. Definition 4.2. A 6-tuple it = T, {.Ft}t>o, P,W(•);u(•)) is called a w-admissible control, and (x(.), u(.)) a w-admissible pair, if
(i) (n,T,I.Ftle>o,P) is a filtered probability space satisfying the usual conditions; (ii) W(.) is an m-dimensional standard Brownian motion defined on
P); (0 , (iii) u(-) is an {F}>0 -adapted process on (1,F, P) taking values in U; (iv) x(.) is the unique solution (in the sense of Chapter 1, Definition 6.15) of equation (4.1) on (ft,T, Vilt>13,P) under u(-); (v) some prescribed state constraints (for example, (4.2)) are satisfied; (vi)• ,x(•),u()) e L F 1 (0,T;R) and h(x(T)) E L .k(11;111.). Here, the spaces L(0,T; R) and LT (cl; R) are defined on the given filtered probability space ( 0 ,F, {Ft}i>o, P) associated with the 6-tuple 7r. The set of all w-admissible controls is denoted by Wal[0, 71. Sometimes, we might write u(-) E U:d [O, T] instead of (f2,,F, {.Tt}t>o, P, u( . )) E T] if it is clear from the context that weak formulation is under consideration. Our stochastic optimal control problem under the weak formulation can be stated as follows: Problem (WS). Minimize (4.3) over 14[O, Namely, one seeks 77- E LeL[0, 71 (if it exists) such that
(4.5)
J(W) =
inf
[0,T]
J(7r).
As in the strong formulation, Problem (WS) is said to be w-finite if the right-hand side of (4.5) is finite. One may also define the w-solvability, the w-optimal control, the w-optimal state process, and the w-optimal pair accordingly. We emphasize here that the strong formulation is the one that stems from the practical world, whereas the weak formulation sometimes serves as an auxiliary but effective mathematical model aiming at ultimately solving problems with the strong formulation. A main reason why this might work is that the objective of a stochastic control problem is to minimize the mathematical expectation of a certain random variable that depends only
5. Existence of optimal controls
65
on the distribution of the processes involved. Therefore, if the solutions of (4.1) in different probability spaces have the same probability distribution, then one has more freedom in choosing a convenient probability space to work with. A good example is when one tries to employ the dynamic programming principle to solve the problem (see Chapter 4). It should also be noted, however, that the weak formulation fails if any of the given coefficients b, a, f, and h are also random (namely, they depend on w because in this case the probability space has to be specified and fixed a priori. As in the deterministic case, the following type of the state equation is of particular interest: { dx(t) = [A(t)x(t) + B(t)u(t)ldt 171
+
(4.6)
E [c3 (t)x(t) + Dj (t)u(t)1dW 3 (t),
3 =1
x(0) = xo, where A (. ), B(.), c3 (.), and Dj (. ) are matrix-valued functions of suitable sizes. The associated cost functional is of a quadratic form: T
[( Q (t)x(t), x(t)) + ( R(t)u(t), u(t) )1dt
J (u(•)) = E
(4.7)
1
+ —2 ( Gx(T), x(T) ) 1. Problem (SS) or (WS) with state equation (4.6) and cost functional (4.7) is called a stochastic linear quadratic optimal control problem (stochastic LQ problem, for short). We will study such problems in great detail in Chapter
6. We note that in the above formulation, T > 0 is a fixed constant. Thus, Problem (SS) or (WS) is also referred to as a fixed duration problem. Sometimes, we may face problems with nonfixed durations. We will discuss those problems as well as some others in Section 7. Finally, throughout this book when the type of formulation (strong or weak) being considered is clear from the context, we shall simply refer to a control as an "admissible control" and the set of admissible controls as Uad[O, T], etc., omitting the superscript/prefix "s" or "w".
5. Existence of Optimal Controls In this section we are going to discuss the existence of optimal controls. The essence of the theory is the following: A lower semicontinuous function defined on some compact metric space attains it minimum.
5.1. A deterministic result Let us first consider Problem (D) formulated in Section 2. We make the following assumptions.
Chapter 2. Stochastic Optimal Control Problems
66
(DE1) (U, d) is a Polish space and T > 0. (DE2) The maps b: [0, T] x x U Rn , f: [0, T] x Rtm x U R, and h : R are measurable and there exist a constant L > 0 and a modulus of continuity cw : [0, oc) —* [0,00) such that for (p(t, x, u) =
b(t, x, u), f (t, z, u), h(x), lyo(t, x, u) — yo(t,i,ît)1 _< Llx —
(5.1)
Vt E[0,T] ,x, elRtm ,u, E U, V (t, u) E [0,7] x U.
Iyo(t, 0 , u)I < L, (DE3)
+ (d(u,ii)),
For every (t, x) E [0,71 x Rn , the set
(b, f )(t, z, U) {(b i (t, z, u), f (t, z, u)) u E U, i = 1,2,...,n} is convex and closed in 1Et 7+1 .
(DE4) S(t)
Rn (i.e., there is no state constraint).
Under (DE1)-(DE4), if Problem (D) is finite, then it admits an optimal control.
Theorem 5.1.
Proof. Let (x 3 (.),u3 (.)) be a minimizing sequence (i.e., J(n3 (•)) infv [0 ,7-]./(u0) as j co). By (DE2), we see immediately that the sequences x3 (.), , x (•), u 3 0), and f , x 3 0, u3 (•)) are uniformly bounded in j. Hence, x3 (.) is equicontinuous (by the state equation). Then we may assume that (one might extract subsequences, if necessary) as j oc,
x3 (.) —> -±-(.)
in C([0, T]; IRn
be , x 3 (•), u3 (•)) — ■ b(•)
(5.2) {
),
weakly in L2 (0, T; Rn
),
weakly in L2 (0, T; R), f( . , x 3 (•), u3 (•)) —> A-) h(x 3 (T)) h(Y(T)) in R.
By the first convergence in (5.2) and Mazur's theorem (see Yosida [1, p. 120, Theorem 2]), we have some ce.,3 > 0 with Et>1 a, = 1, such that as j 4 oc, -
(5.3) {
E au b(• , Y (.), 14 +3 (•)) -- b(•) z>1
strongly in L2 (0, T; Rn ),
Eckiife,Y0,ui+J(.)) —70
strongly in L2 (0, T; R).
i>1
Hence, by assumption (DE3),
(5.4)
(b(t), f(t)) E (b, f)(t,I(t), U).
Consequently, by Filippov's lemma (see Li-Yong [1, p. 102, Corollary 2.26]), there exists a ii(.) E V[0, T] such that
(5.5)
b(t) = b(t,T(t), i(t)),
J(t) = f (t,Y(t),17,(t)), a.e. t E [0,7 ].
5. Existence of optimal controls
67
Thus, (Y(•), U(•)) is an admissible pair of Problem (D). Further, by Fatou's lemma (Yosida [1, p. 17]) , we obtain (
( )) =
f (t,Y(t),T.i(t))dt + h(Y(T))
•
0
(5.6)
< inn
Eaii { i>1
-= lim
f (t,x i+i(t),ui+i(t))dt + h(x.,+ (T)) o
E i>i
= u(.) 1i)f[cl,T1 j(u( * )) *
Thus, TIO E V{0, Ti is an optimal control. There are two essential issues in the above proof: The space 1Rn where the state x(t) takes values is locally compact (i.e., any bounded closed set is compact), which ensures that the sequence { x3 (•)} is relatively compact (i.e., it contains a convergence subsequence) under condition (DE2). The set (6, f)(t, x, U) is convex and closed (called Roxin's condition), which guarantees that Mazur's theorem and Filippov's measurable selection theorem are applicable to obtain an optimal control. We also point out that condition (DE4) can be relaxed. A certain kind of continuity for S(t) is enough. We leave the details to the interested reader. 5.2. Existence under strong formulation We now consider the stochastic case. Let us first look at the existence under strong formulation. Let (n, F , {Tt}e>o,P) be given as before and let W(.) be a given onedimensional Brownian motion. Consider the following stochastic linear controlled system: ( 5.7)
dx(t) = [Ax(t) + Bu(t)]dt + [Cx(t) + Du(t)ldW(t), t E [O, TI, x(0) = x o ,
where A, B, C, D are matrices of suitable sizes. The state x (. ) takes values in IV, and the control li(.) is in (5.8)
UL [0, 7]{u(.) E L F 2 (0,T;Ilk ) u(t) E U, a.e. t E [0,
P-a.s. },
with U Ç 1Rk . Note that we have an additional constraint that a control must be square-integrable just to ensure the existence of solutions of (5.7) under u (. ). If U is bounded, then this restriction is satisfied automatically. The cost functional is (5.9)
J(11,0) =E{Z f (x(t),u(t))dt + h(x(T))},
with f : Rn x U It and h : IFtn be stated as follows.
R. The optimal control problem can
Chapter 2. Stochastic Optimal Control Problems
68
Problem (SL). Minimize (5k) subject to (5.7) over Z490,71.
Now we introduce the following assumptions:
(M) The set U C R k is convex and closed, and the functions f and h are convex and for some (5,K > 0, f (x, u) > 61721 2
(5.10)
K, h(x) >
-
-
K, V(x, u) E IRn x U.
(H2) The set U C R k is convex and compact, and the functions f and h are convex. When either (H1) or (H2) is assumed, Problem (SL) is called a stochastic linear convex optimal control problem, since the controlled system is linear and the cost functional is convex. A special case is that in which both f and h are convex quadratic functions. Problem (SL) is then reduced to a stochastic linear quadratic problem. -
Theorem 5.2. Under either (HI) or (H2), if Problem (SL) is finite, then
it admits an optimal control.
Proof. First suppose (H1) holds. Let (x 3 (.),u3 0) be a minimizing sequence. By (5.10), we have (5.11)
E f 1u 3 (t)rdt K,
for some constant K > 0. labeled by u3 (.), such that (5.12)
Vj > 1,
Thus, there is a subsequence, which is still
1L3 (.) -> Tt(•),
weakly in L2F (0, T;R k ).
By Mazur's theorem, we have a sequence of convex combinations with aii >
0, E ai • = 1, (>1
such that (5.13)
1-13 (')
(*)
strongly in L2F (0, T;R k ).
Since the set U C R k is convex and closed, it follows that Ti(•) E U' [0, T]. On the other hand, if(.) is the state under the control ii3 (•), then we have the convergence (5.14)
Y(•),
strongly in Cy•([0, 7],1Rn ).
Clearly, (Y(.),U(.)) is admissible, and the convexity of f and h implies
J(ii(.)) = lim J( 111 (.)) -00
(5.15) =
inf
u()EuLEo,T)
i>1
J(u()).
5. Existence of optimal controls
69
Hence, ((.),u(.)) is optimal. In the case where (H2) holds, we automatically have (5.11). The above proof then applies. We have seen that the above proof seems very similar to that of Theorem 5.1. However, there is a crucial difference. In Theorem 5.1, we do not have (weak) convergence of u3 0, and we obtained the convergence of x3 (•) directly by the equicontinuity of the sequence (due to the local compactness of lEtn). On the other hand, in Theorem 5.2, the uniform convergence of x3 ) in (5.14) was shown via the convergence of u 3 (-). The linearity plays an essential role here. For the general Problem (SS), we do not have the convergence of u 3 (.), and the convergence of x 3 (.) is not necessarily valid in the given probability space due to the lack of local compactness of infinite-dimensional spaces such as L .2F (ft; 1Ftn ). Thus, the same arguments of the deterministic case do not apply for the strong formulation of general nonlinear stochastic optimal control problems. (.
5.3. Existence under weak formulation In this subsection we examine the existence of optimal controls under the weak formulation. Let us make the following assumptions.
(SE1) (U, d) is a compact metric space and T > 0. (SE2) The maps b, o, f, and h are all continuous, and there exists a constant L > 0 such that for (p(t, x, u) = b(t, x, u), (t, x, u), f (t, x , u), h(x),
(5.16)
I ko(t, x, u) — 1
Icp(t , 0 , u)1
u)1 < Lx — E [0, x V (t, u) E [0,7] x U.
E Rn
E
(SE3) For every (t, x) E [0,7 ] x 111.71 , the set
(b, au; f)(t, x U) {(1),(t, x, u), (acrT )i (t, x, u), f (t, x, u))1 u E U, i = 1,...,n, j = 1,...,m} is convex in 111n -l-rim-1-1.
(SE4) S(t)
IRn .
In proving the existence of optimal controls one typically seeks a certain compactness structure. The weak formulation enables us to find the compactness of the image measure of some stochastic processes involved on a certain functional space. However, since the control u(.) is measurable only in t and there is no convenient compactness property on the space of merely measurable functions, we need to embed it in a larger space with proper compactness.
Chapter 2. Stochastic Optimal Control Problems
70
To this end, let us denote by A the set of all (nonnegative) measures on [0, T] x U such that A([0, s] x U) = s,
(5.17)
Vs E [0,7 ] .
Since U is compact, A is tight when endowed with the weak convergence topology, which is metrizable; see Chapter 1, Proposition 1.14 and Corollary 1.17 (note that the discussions in Chapter 1, Section 1.4, for probability measures can easily be adapted to the present case). On the other hand, by (5.17), À can be represented as A(dt, du) = (t , du)dt , where )'(t,.) is a probability measure on U for almost all t and is determined uniquely except on a t-null set. Under this framework, any U-valued measurable (deterministic) function u(.) may be embedded into the space A in the sense that u ) corresponds to the Dirac measure A u (.)(dt, du) = A1. 0 (t, du)dt with the following property: For any bounded and uniformly continuous function p(t, x , u) defined on [0,7 ] x Rn x U, (.
(5.18)
p(t , x, u(t)) = f p(t, z, u)A tu( . ) (t, du) g fi(t, z, Aue)).
Now we introduce a suitable filtration on A. First of all, each À E A can be identified as a linear functional on C([0, x U) in the following way: A( f)
afo fuf (t, u)À(dt, du),
For any f E C([0,
V f E C([0, T] x U).
x U) and t E [0, 7 ] , define f t c C([0,
x U) by
f t (s, u) f (s A t, u). Since C([0, T] x U) is separable, we may let {f3 } 3 >1 be a countable dense subset (with respect to the uniform norm). Clearly, for any t c [0, Tb {ft f E C([0, x U)}. Define {f}>' is dense in the set (5.19)
Bt (A) o- {{A E AIA(fs) E B} : S E [0, t], B E B(R)}.
Then, as in Chapter 1, Lemma 2.17, we can show that 81 (A) can be generated by cylinders of the following form:
B t (A) =
{{A E MA(ff) E (a, b)} :t>sE Q,
(5.20)
j = 1, 2, ... , a, b E Q1. Next, ji is called an {.Ft } t>o -adapted A-valued random variable (A-r.v. for short) on a given filtered probability space (O, .F, {.Ft }t>o, P) if p(Bi x B2) is {.Ft } t >. 0 -measurable for any B 1 E 13([0, t]) and B2 E B(U). It is clear that if u(t,w) is an {Ft }t>o-adapted process, then its embedding A u (.) is an VtIt>o-adapted A-r.v. and vice versa.
5. Existence of optimal controls
71
We now state and prove the existence of optimal controls in the weak formulation. Theorem 5.3. Under (SE1)-(SE4), if Problem (WS) is finite, then it admits an optimal control.
Proof. Let irk (S-41-Fk7 {-rkt}t>01P k, Wk(•), minimizing sequence, namely,
(5.21)
lim J(rk) =
k—.00
inf
7r ELI:, [0,7]
%Ik ( ))
G
Ru,Jd [O,T] be a
J(r).
Let xk() be the state trajectory corresponding to irk. Define x k •) a_(x k (. ), B k • ), Ek ( ), Fk •), Wk(•)),
(5.22) where
B k(t) g
b(s, x k(s), uk(s))ds,
t (5.23)
Ek(t)
a(s7xk(s),uk(s))dWk(s),
Fk(t) g
f (s, x k(s),uk(s))ds.
By (SE2), it is routine to show that there is a constant K> 0 such that EkiXk(t)— Xk(s)1 4
5 Kit — 8 1 2 ,
Vt,S
E [0,
'c/k,
is the expectation under Pk. Noting the compactness of A, x A-random {(Xk(•), A uk (.))1 is tight as a sequence of C([0, 71;R variables (cf. Chapter 1, Theorem 2.14 and Corollary 1.17). By Skorohod's theorem (Chapter 1, Theorem 1.20), one can choose a subsequence (still labeled as {k } ) and have where
Ek
{(),
(X(.), A)
X)} {(xkO, Bk(•),
on a suitable probability space
(5.24)
Eke), FkO, Wk
law of (Xk
(r-2, .F, 13)
such that
Ak) --= law of (Xk•,Auk(.)), Vk > 1,
and P a.s. , -
(5.25)
uniformly on t E [0, 7 ],
Xk(t)— ■ X(t)
and
(5.26)
AO},
(Y(.), B(.), Ee), F(.), W(.), A),
—)
Tt
weakly on A.
72
Chapter 2. Stochastic Optimal Control Problems
Set
(5.27)
1
7F. kt
a (a{wk(s),yk(s): s
v -x, l ( Bt(A))),
< t}
.7. t I (0- {W(s),Y(s) : s < t} V A 1 (13t (A))).
By (5.20) and Chapter 1, Lemma 2.17, F kt is the a-field generated by Wk(ti),..., Wk(tt),Yk(ti),... Yk(te),Xk(fP), • • • , 7\k(f.;`), 0 5_ t1 t2 5 • •• < t e < t and j, = 1, 2, .... A similar statement can be made for Tt• Now we need to show that W k (• ) is an {F-kt}t>o- Brownian motion. To this end, note that Wk() is a a{Wk(s),xk(s); s < t} V Brownian motion in view of Chapter 1, Proposition 4.2. Thus, by Chapter 1, Proposition 1.12, for any 0 < s < t < T and any bounded continuous function g on IR,(m+n-h3)t , we have Ek{g(n)(Wk(t) — Wk(8))1 = 0,
where (5.28)
Yk
A
=IWO?), Xk(tt), Ak(f .
:0 ‘ )},
0 < < t2 < • • • < < S,
a=
In view of (5.24), Ek{g (Y k) (W k(t ) — W k ( s)) 1
= 0,
where (5.29)
k
0< t1 < t2 < • • •
a = 1, 2, ... , B.
By Chapter 1, Proposition 1.12, again, we get the first equality of (2.24) in Chapter 1. Similarly, one can prove the second equality of (2.24) in Chapter 1. This shows that Wk() is an {Ykt}t>o-Brownian motion. By (5.24), we have the following SDE on (fi, CF- kilt>o,P): k(t) = so ± 173 k(t)
k(t)
So ± f f b(8 k( 8),U)A ik(S, dU)dS 0
(5.30)
U
a(s, (s), uPt-ik (s, du)dWk(s)
+f 0 Ju A
= X0 +1 6 - (s,±- k(s),,Nk)ds +
0
5-(s,Yk(s),Xodw k (s). 0
Note that all the integrals in (5.30) are well-defined due to the fact that
5. Existence of optimal controls
W k(•) is an
73
Viet It>o-Brownian motion. Moreover,
E F k(T) =--- E
f (s, x k(s),uPt 'k(s, du)ds + h(Y k(T))}
o
(5.31)
U
gEff
(s, k(s), Ak)ds + 11 (4( 7 ))}
= J(lrk) —>
inf
J(7r), as k
rE/In [0,7
where E is the expectation under P. Letting k and noting (5.25), we get Y(t) =
(5.32)
xo + B(t) + E(t),
E F(T) =
oo,
]
oc in (5.30) and (5.31),
Vt G [0,71,
P-a.s. ,
inf J(7r). irEun [U,T]
Next let us consider the sequence ak(s)
eia-T(s, yk(s), jtk),
s E [0, T].
By (SE2), s 1114 --k7 foT lak(s)I 2 ds < +oo, and hence {ak} is weakly relatively compact in the space L2 ([0, T] x Sn). We can then find a subsequence (still labeled by {k}) and a function a E L2 ([0, T] x Sn) such that
ak —> a,
(5.33)
weakly on L 2 ([0,
x
Sn).
Denoting by a" the ijth element of a matrix a, we claim that for almost all (s, w), (5.34)
lim a (s,
<
(s,w) < lim
k—000
k—.co
(s,w),
i,j -= 1, . . . ,n.
Indeed, if (5.34) is not true and on a set A C [0, T] x — f/- of positive measure, lim aik3 (s , w) > az3 (s , w), k—.co
then we have by Fatou's lemma that lim
(s,w)dsdP(w) >
k—.00 A
(s,w)dsdr1 (w), A
which is a contradiction to (5.33). The same can be said for the lim, which proves (5.34). Moreover, by (SE2) and (5.25), for almost all (s,w), lim aku (s,w) = Hm "6 6- T (s,Y(s),A -k), (5.35)
k•—•oo
lim
k
k—000
ate (s,w) = lim &FIT (s,Y(s), Ak) • k— .oc
Then combining (5.34), (5.35), and (SE3) gives
Chapter 2. Stochastic Optimal Control Problems
74
Modify au on a null set, if necessary, so that (5.36) holds for all (s, w) E [0,T] x n. Similarly, setting { bik(s)g -6'(8,1-k(s),Ak), i = 1,...,n, (s),), k), fk(s)g f(s,Yk
one can prove that there are b2 , f E L2 ( [ 0, 1 ] x SI; R) such that (5.37)
weakly on L2 ([0, 7] x
bi, —, bi (i = 1, . . . n), fk -■ f, ,
and (5.38)
bz(s,w) E
(s,Y(s,w),U), f(s,w) E f (s,Y(s,w),U), V (s , cv) E [0,T] x K
j
,
= 1,2,...,n.
By (5.36), (5.38), (SE3), and a measurable selection theorem (see Li-Yong [1, p. 102, Corollary 2.26]), there is a U-valued, {t}t>o-adapted process Tt(-) such that (5.39)
(6, crcir, f)(s, (4.1) = (b, GraT, f )(s V(s,
(SI (V),
ri.( 8 1 w)),
E [0, T] x
Next we claim that E(t) is an { t } t >o-martingale. To see this, once again let 0 < s < t < T, and define F7k as (5.29) and
YI{W(ti),Y(ti)j(ft,)}, 0 < < t2 < • • < < s,
a = 1, 2,
(3 .
Since Ek(t) is an Y-kt -martingale, for any bounded continuous function g on 1R(m-i-12)1 , we have (5.40)
0 = E[g(Y k)(Ek(t) - Ek(s))]
E[g(Y)(E(t) -
by (5.25), (5.26), and the dominated convergence theorem. This proves that E(t) is an {:ft } t >0-martingale in view of Chapter 1, Proposition 1.12. Furthermore, from (5.30), it follows that çt
(Ek)(t) =
Jo
6-5- T (s,Yk(s),;\k)ds
j ak(s)ds,
Jo
where (k) is the quadratic variation of Ek (see Chapter 1, Definition 5.8). Hence EkET k (t) — fct, ak(s)ds is an {Ykt } t >o-martingale. Recalling ak(s) a(s) ()TI T (5,Y(5),T1(5)) weakly on L2 ([0,T] x r2), we have for any s, t E [0, T], ak(r)dr
o-cr T (r, Y(r), rt(r))dr,
weakly on L2 (0).
6. Reachable sets of stochastic control systems
75
On the other hand, by the dominated convergence theorem,
gaik)
g(F),
strongly on L2 (Q).
Thus,
E (g(Y k) f ak(r)dr)
E (g(Y) f o -o-T (r,Y(r),77,(r))dr) .
Therefore, using an argument similar to the above, we obtain that EE T (t)— fot cro-T (s,Y(s),Tt(s))ds is an {Ft } t >0-martingale. This implies
(5.41)
( E )(t) = f cruT (s, Y(s), U(s))ds.
By a martingale representation theorem (Chapter 1, Theorem 5.10), there is an extension (fi,.P, {. F—t}t>o, -13 ) of (n, .7-, 1.7-t P) on which lives an m-dimensional {.Pt}t>o-Brownian motion W(t) such that
(5.42)
E(t) -= f o-(s,Y(s),U(s))a(s).
Similarly (in fact more easily), one can show that
B (t) =f b(s,Y(s),U(s))ds, (5.43)
t F(t) = f f(s,Y(s), 711(s))ds.
Putting this into (5.32) and noting Definition 5.9 of Chapter 1, we arrive at the conclusion that Tr =(0, optimal control.
ft>o,P,W('),
E 1.1:1 [0,T] is an
The basic idea behind the above proof is the so-called relaxed control, which is needed in order to provide some compact structure. To be more precise, the set of all admissible controls can be regarded in general as a subset of a much larger set of A-valued random variables (relaxed controls), whereas A is compact under the metric induced by the weak* topology. This approach works, since under Assumption (SE3) (Roxin's condition) along with (SE1), the set of admissible controls coincides with that of relaxed controls. 6. Reachable Sets of Stochastic Control Systems Stochastic control systems are quite different from their deterministic counterparts. In this section we present several unexpected properties of stochastic control systems in terms of the reachable sets, which will help us understand that many results for deterministic control systems do not necessarily extend to stochastic ones. For the sake of simplicity, we present
Chapter 2. Stochastic Optimal Control Problems
76
the results only for linear systems. It is expected that the situation for nonlinear systems will be more complicated. 6.1. Nonconvexity of the reachable sets Consider the following deterministic controlled system:
{ ±(t) = Ax(t) + Bu(t). t E [0, T],
(6.1)
x(0) = xo,
where A E R" n and B E Rnxk . Define
(6.2)
RD(T) =- tx(T; u0)
u(.) E V[0,711 C
which is called the reachable set of the system (6.1). Clearly, the set R.D(T) changes as T varies. Investigating properties of RD (T) is very important for the study of controllability and time optimal control problems. Let us elaborate these a bit more. Let M C Rn be a nonempty set. We would like to find a control that steers the initial state xo E Ian to the target M; if it is possible, we further would like to reach M as fast as possible. To formulate the problems rigorously, we define
(6.3)
Vm
glu ) E V[0, oo) I x(T;u(•)) (.
E M, for some T > 0}
and
(6.4)
1(u(-)) = inf{T >0 x(T; u(•)) E Ml, Vii(.) E Vm,
with the convention inf
a oc. Then we pose the following problems:
Problem (DC). Find conditions under which V51 is nonempty. Problem (DT). Minimize (6.4) subject to (6.1) over VA,/ when Vm
tb.
We call Problems (DC) and (DT) the controllability problem and time optimal control problem, respectively. Let us now assume V Af
(6.5)
0,
which is equivalent to
RD(T)] n M
[
0 4' .
T>0
Under this assumption, the goal of Problem (DT) is to find the first moment when RD(T) intersects M and a control ri,(.) E V[0, oc) such that
(6.6)
x(T;r1(•)) E RD(T)nm.
When they exist, we call T, u(•), Y (• ) x(. ;Ti•)), and (Tc(•),U(.)) the minimum time, a (time) optimal control, optimal trajectory, and optimal pair, respectively.
6. Reachable sets of stochastic control systems
77
Now assume that M is convex and closed. If we also have the convexity of RD (T) for any T > 0, then it is possible to use some powerful techniques in convex analysis to study (6.6) by which the minimum time T and the time optimal pair (Y(•), U(•)) could be characterized. Hence, let us study when R.D(T) is convex. First of all, it is clear that if U C Rk is convex, then R.D(T) is convex T > O. Surprisingly, we have the following result without the any for convexity of U! Proposition 6.1. Let U C R ic be compact. Then for any T> 0, RD(T) is convex and compact in lEtn , and it coincides with RD (T), the reachable set with U replaced by U. A consequence of Proposition 6.1 is the following existence result for Problem (DT). Corollary 6.2. Let U C Ilk be compact and suppose that (6.5) holds with M closed. Then Problem (DT) admits an optimal control.
Proof. Using the same argument as in the proof of Theorem 5.1, one can show that Problem (DT) with U replaced by o U admits a time optimal control, say, ii(.) E oo) {u: [0, oo) U ILO measurable 1 with the optimal trajectory(.), and the minimum time Thus,
= q5, E "k:D(T) n M.
iiD (T) n M
l. x(T)
VT E [0,T ),
By Proposition 6.1, 'R.D(T) = it-.D (T) for all T > O. Thus, there exists a U(.) E V[0, oo) such that x(T; 77,(-)) = i(f). Hence, 77(.) is a time optimal control of Problem (DT), with the minimum time T =- T. The proof of Proposition 6.1 is based on a very deep theorem due to Lyapunov, which we now state. Theorem 6.3. the set (6.7)
(Lyapunov's theorem) Suppose f E L 1 (0, T; Ra ). Then
f f (t)dt S E 5[0,7]}
is convex, where B[0,7 ] is the Borel o--field of [0, 7 ] . We refer the reader to Hermes-LaSalle [1] for proofs of Proposition 6.1 and Theorem 6.3 (see Diestel-Uhl [1] for a more detailed discussion related to Theorem 6.3). It is seen that Proposition 6.1 and Corollary 6.2 give an elegant existence result of time optimal control for the deterministic linear system (6.1). One can actually further characterize the time optimal control and the minimum time by using the convexity of the reachable set R.D(T). A natural question is then whether we have a similar convexity property for the reachable set of a linear stochastic system.
Chapter 2. Stochastic Optimal Control Problems
78
Let (0, F , P) be a given complete probability space on which a standard Brownian motion W(.) is defined with {Ft } t>0 being the natural filtration of W(.), augmented by all the P-null sets in F. For system (5.7), we define
R.s(T) = fx(T; u(•)) u (. ) E U L [0,71} ç
L2T7, (SI; IFtn ).
We call 1Z(T) the reachable set of system (5.7) at T. Let us present a simple example below showing that Rs(T) may no longer be convex. Example 6.4. Consider the following one-dimensional control system:
f
(6.8)
dx(t) = u(t)dW(t), x(0) = 0.
The control set U is { 1, 1}. Thus, we have a linear controlled system with a nonconvex control set (which is, however, compact). By taking u(t) 1 and u(t) —1, we see that -
Vt G [0,T].
RS(t) D {W(t), — W(t)}, Thus, 0 E coRs(t),
t E [0, 71.
However, for any it() E UL [0, 71, 2
EIX(t)1 2 = E f U(S)dW(.4)1 = t. 0 Hence,
0 E co'Rs(t) \ Rs(t), Vt E (0, 71
(6.9)
showing that for any t > 0, neither Rs(t) nor 7:4W is convex. The essence behind the above example is the nonconvexity of the range of vector measures defined through the Itô integral. Let us make this more precise. Let (6.10)
S[0,7] A{S C [0, T] x 1 /s is {.7"t } t >o-adapted}.
Given f E L(0,T; Rn ), define (6.11)
p(S) f —
Theorem 6.5. Suppose
(6.12)
VS E S[0,71.
■ L F2 7, (SI; Rn ) is a vector valued measure. Define the as tt(S[0,11) g{p(S)IS E S[0,7]1. We have the following result.
Then p : S[0, 71 range of it
Is(t) f (t)dW(t),
-
f E
T; Rn ) such that
E] If 4)1 2 dt > 0.
6. Reachable sets of stochastic control systems
79
Then there exist S1, S2 E S[0, T] with
isii
(6.13)
f dt dP > 0,
Si
n S2 =
such that for any A E (0,1),
ElAkt(Si) + (1— A)p(S2)
(6.14)
A(S)1 2 b(Si, S2, A) >0, VS E
S[0,T].
In particular, the range p(S[0,7 ] ) of the vector-valued measure p as well as its closure p(S[O,T ] ) in L T ( 1Z; R) are nonconvex. Proof. For any c > 0, define
(6.15)
SE
= {( t, w) E [07 7 ] If (t,c01
E S[0,1].
Let he (t) = f
(6.16)
P(sE n
s] x 11))ds,
t E [0,T].
Clearly, he (t) is continuous, and by condition (6.12), 11,E(T) > 0 for some
E > 0. Thus, one can find to E [0, T) such that
(T).
0 < he (to) <
(6.17) Next, set
(6.18)
Si = SE
n ([ o, to ] X ft)),
S2 =
SE n
(40,71 x SI)).
From (6.17), we see that
(6.19)
f 81,52 E S[0,7], S1 if (ti co)1 ?_
el
nS2 =
cl', ISiii1S21> 0, V(t,c4i) E S U S2-
Now, for any A E (0,1), we obtain
ElAp(Si ) + (1 — A)/452) — P(S)1 2
= E f0
Ef I Als, (t) ± (1— A) I S2 (t)
= f0 =
{AI s (t) + (1— A) I S2 (0 — S (01 f (t)dW (Or
f
S (t)I 2 If (t)I2 dt
lAisi ( t, w) — Is (t, 04 1 2 1f(t, cd )1 2dtdP(w)
+f
s2
1(1 — )0/.52 (t, Ls)) — Ls(t,w)1 2 1f(t,w)1 2dtdP(w)
E2 [À2 A (1— )0 241 ,911 + 1 ,520 >0. This proves (6.14).
Chapter 2. Stochastic Optimal Control Problems
80
The above result implies that the vector measure defined through the
Itô integral has a nonconvex range, provided that (6.12) holds. The following is a simple consequence of the above result, which generalizes Example
6.4. Proposition 6.6. Suppose in (5.7) (5.8) that -
B = CD,
(6.20) and U = {uo,u1} with
D(u i
(6.21)
-
uo )
O.
Then, for any xo E IFtn and T > 0, R.s(T) is nonconvex. Proof Without loss of generality, we may assume U = {0,0 with Du L 0 and xo = 0. Let (I)(.) be an R'-valued process satisfying
f
(6.22)
d4)(t) = A4)(t)dt + C4)(t)dW(t),
Then 44(t) -1 exists, and for any u (. ) E UL [0, 7 ] , noting the condition (6.20) and x0 = 0, we have (see Chapter 1, Theorem 6.14)
(6.23)
x(t;u()) -= (NO f (1)(8) -1 Du(s)dW(s),
t E [0,7 ] .
Since U = 10,0, any u (. ) E UL[O,T] corresponds to an S E S[0,7] such that
(6.24)
u(t, (.,;) =
(t, w)fL,
(t, w) G [0,7 x f2. ]
Then (6.23) implies
4)(T) -1 x(T) = f Is (s)4)(s) -I D dW (s). Hence, applying Theorem 6.5 to the function f(t) = 4)(t) -1 Drz, we obtain the nonconvexity of (I)(T) -1 R.s(T). Our assertion follows then, since 43(T)' is an invertible linear map. Let us make a final observation. When condition (6.20) does not hold, instead of (6.23), we would have
x(t; u (. )) =4)(0 f (s)' (B - CD)u(s)ds (6.25) + 4)(t) f 4)(s) -1 Du(s)dW(s), t E [0,7 . ]
Thus, there would be some interaction between the diffusion and drift. The situation would become much more complicated and interesting. We
6. Reachable sets of stochastic control systems
81
encourage the reader to explore this situation and to obtain results that are not yet known to us. 6.2. Noncloseness of the reachable sets
Proposition 6.1 tells us that the reachable set RD(T) of the deterministic system (6.1) is convex and compact if U c IRrn is compact. For the reachable set 1Z(T) of the stochastic system (5.7), the results of the previous subsection imply that it is not convex in general even when U is compact. We now investigate whether 'R.s(T) is closed in L T ( 1l; ). Our first result is the following. Theorem 6.7. Let {
R1 = {
(6.26)
Io
T
=
h(s)ds I h(.) E L2T (0,T; IR)}, f (8)dw(s)
I f E CF([0,11; L2 (f2 ; 11))}-
Then {
(6.27)
R1 =
+R= (f2 ; Int (RI + R2) = 0. R2
In particular, both 7. 1 and R.2 are not closed. Proof. First of all, it is clear that
(6.28)
R.1 C L2FT (12;R), 7Z2 + IR C
(S-2; R.).
Thus, we need only to prove the other direction of the inclusions. For any n E L T (fl; IR), applying the martingale representation theorem (see Chapter 1, Theorem 5.7) to E(771.Ft ), we have Z(.) E L3-(0, T; Et) such that
E(nl.Ft ) = Eij +
Z(s)dW(s).
In particular,
(6.29)
71= Ei7 +
Z(s)dW(s). o
For any E
> 0, we can find a Z,(•) E C.F([0,7]; L2 (12; IR)) such that E
f0
1Z(s) — Z E (s)1 2 ds <
e.
Then
T { lle r-4- f Ze (s)dW(s) E R2 ) 0
— [E 71 + 71E11 2 < 6,
Chapter 2. Stochastic Optimal Control Problems
82
(12; R), again we have (12; R). Next, for any E E L2F (0, T; R). For any E > 0, let 1 he (s) = — E(711.7;r_e )IET_,,T1(s), s E [0,7].
proving 'K2 R = (6.29) for some Z
(6.30)
?I
Clearly, he ( • ) E L2F (0, T;
R), and T—e
A
1
3 1k = f he (s)ds = E (nl.FT_E ) = E + f
Z(s)dW (s).
Thus, T
Ellie — n i 2 = f Eiz(s)i 2ds —, o,
as
E —■
0,
'T-E
which proves T2-. 1 = L 7 2 .7, (S2; R). Next, we prove the second relation in (6.27). To this end, we construct a deterministic Lebesgue measurable function 0 satisfying the following:
/ 0(s) = ±1, Vs E [0, Tj, ifs E [T„7]1 13(s) = 1)] = Ifs E [T,, T] 1 0(s) = —1}I (6.31) T— i > 1, 2 ' for a sequence T, I T, where IAI stands for the Lebesgue measure of a measurable set A. Such a function exists by some elementary construction. We claim that
(6.32)
Jo
Suppose there exist that
(6.33)
h(.)
e3(s ) dW (s) It R1 + R.2.
E L2.7.(0,
Jo (3(s)dW(s)
T; R) and f(.) E C.F([0, T]; L2 (12; R)) such
= fo h(s)ds + fo f(s)dW (s).
Define
(6.34) Then, by
((t) = f h(s)ds + j [f (s) Jo
— 13(s)]dW
(s), t E [0,7 ].
Itô's formula, we have El((t)I 2 E f
If(s) — 0(s)1 2ds = —2E f h(s)((s)ds
T
T
= 2E f h(s)[ f h(r)dr + f (f(r) — 0(r))dW (O]ds
(6.35) = 2E fT h(s) IT h(r)drds s 2
=
f h(S)d.51 _< (T — t) f Eh(s)2ds.
6. Reachable sets of stochastic control systems
83
On the other hand, since h(.) E L27-(0, T; R) and f(.) c Cy([0, 7]; L2 (SI; R)), we have
Ej
If (7') — 0(8)1 2 ds 2E
(6.36)
f If(s) — Q(s)I 2 ds + 2E f If (T) — f(s)1 2 ds
2(T —
t)f Eh(s) 2ds + 2E f
(T) — f (s)I 2 ds
= o(T — t). By (6.31), the definition of fi(.), we have rT
(6.37)
Elf(T) — 1 1 2 ± Elf (T) + 1 1 2 )
2 ds = T'PT)—0(sI —2
T,
>T—
> 2.
Combining (6.36) and (6.37), we obtain a contradiction. Thus, (6.32) holds. Since TZ 1 + R.2 is a subspace, the second relation in (6.27) follows from the fact that for any n E R.1 + 7Z2 and any E > 0, ± 6f 0(S)dW (S)
0
R.1 + R.2.
This proves our result. Let us discuss some simple consequences of the above result. The first is related to the problem of controllability. Corollary 6.8. Let a control system be given by (5.7) with A, C E Rn "
and B,D E IR n " . Let rank D < n.
(6.38)
Then (6.39)
TZ.s(T) L T 2 , (I-2; Rn ), VT> 0.
Proof. Suppose (6.38) holds. Then we can find some 77 E IRT1 7 Inl = 1 , such that
(6.40)
771- D = 0.
Multiplying the integral version of (5.7) by tI T , we obtain
n T x (T)
17T
77 1 xo
) + Bu(t)frlt
(6.41)
f
0
n T C X(t)dW (t).
Chapter 2. Stochastic Optimal Control Problems
84
By (6.27), we see that Int [nT 'R.s(T)] Ç Int (R.i + R.2) = 45 -
(6.42)
Hence, (6.39) holds. We know that for the deterministic system (6.1), provided that Kalman's rank condition rank (B, AB,. .. , An -1 B) = n
(6.43)
holds, for any xo, x1 E Rn and T> 0, there exists a /L(.) E V[0, T] such that x(T; x o ,u(•)) = x 1 . In other words, under condition (6.40), the system (6.1) is completely controllable. However, Corollary 6.8 tells that if D has rank smaller than n, in particular if k < n, then there is some E L T ( 1 ; IV) that will not be hit by the state x(T). Another consequence of Theorem 6.7 is the noncloseness of reachable sets and the nonexistence of optimal controls for some stochastic systems. Let us look at such an example. Example 6.9. Consider the controlled system
(6.44)
f
dx(t) = u(t)dt + f3(t)dW (t), x(0) = 0, dy(t) = 2 f3(t)dW (t), y(0) = 0,
where 00 E L3(0, T; R) satisfying T
(6.45)
Jo
13(s)dW (s) E
1.27•T (11;
R) \
Such a 0(.) can be constructed as in (6.31).
We let the control set be
U[0, T] L(0, T; R) and the cost functional be (6.46)
J(u(•)) = Elx(T)
—
y(T)I 2
We now show that this (very simple) optimal control problem does not admit an optimal control. In fact, 2
(6.47)
J(u(•)) = E f u(s)ds — fo fl(s)dW (8)1 .
By (6.27), 0.7-T ( 1Z; R) = R. I . Thus, infu( )E (0 ,T;R) Au( . )) = 0. However, the infimum cannot be achieved, due to the fact (6.45). Thus, an optimal control does not exist. The above argument actually shows that there is a sequence {uk()} C U[0, T] such that the corresponding state (x (. ; uk(-)), ; uk(•))) satisfies
Elx(T;uk(')) — Y(T;uk(*))1 2 —> 0. However, no u(.) E U[0, T] exists such that
x(T; u()) = y(T; u(.)).
7.
Other stochastic control models
85
This means that the "diagonal line"
{(e e)
2 , (S2; IR,)} C 'R,s(T)\R,s(T). LF
Thus, TZ s (T) is not closed in L2FT (SI; IR2 ). This is indeed the very reason why the optimal control does not exist in this example. 7. Other Stochastic Control Models In Section 4 we have formulated optimal control problems in both strong and weak forms where the system is governed by an Itô stochastic differential equation, the system is completely observable (meaning that the controller is able to observe the system state completely), and the cost is of an integral form calculated over a fixed, deterministic time duration. This type of stochastic control problems will be the subject of study throughout this book. The reason for doing so is that we would like to capture some fundamental structure and properties of optimal stochastic control in a clean and clear way via this standard and relatively simple model, without being blurred by technicalities caused by more complicated models. Certainly, there are many other types of stochastic control problems that are of significant theoretical and practical importance. In this section we shall introduce and formulate several representative models that have been commonly encountered in practice and investigated in theory. It should be noted that a real-life problem could involve a combination of some or all of those models and therefore be very complicated. On the other hand, since this section is rather introductory, we shall omit all the proofs when we state relevant results, and the interested readers may find more details in the references surveyed in Section 8. Furthermore, only strong formulations will be discussed, and weak formulations are completely parallel. 7.1. Random duration In Problem (SS) formulated in Section 4, the time duration under consideration (called a control horizon) is a fixed, deterministic finite interval [0, 7 ] . In many real applications the control horizon may be a random duration [0, 7 where the terminal time T is a random variable at which the state of the controlled system changes critically and the control beyond T may no longer be meaningful or necessary. For example, in the insurance company model (Section 3.3) the control after the time when the company is bankrupt becomes unnecessary. Therefore, it makes better sense to consider the problem in [0, 7 where T is the bankruptcy time defined as T irlf ft > O : X(t) = O}. To formulate a general model that incorporates a random control horizon, consider the following stochastic differential equation: ]
]
dx(t) = b(t, x(t), u(t))dt + cr(t, x(t), u(t))dW (t), x(0) = x0 c 0,
(7.1) {
Chapter 2. Stochastic Optimal Control Problems
86
on a complete filtered probability space (St, F , (Ft It>o, P) on which an mis a given dimensional Brownian motion W(.) is defined. Here 0 C open set. An admissible control u (. ) is defined similarly as in Section 4.1. Define the first exit time T
Note that tional
(7.2)
T
infft > O: x(t)
01.
implicitly depends on the control taken. Given the cost func-
J (u()) = E
f (t, x(t), u(t))dt + h(x(T))} ,
the problem is to minimize (7.2) over the set of admissible controls. 1 and h 0, which corresponds An interesting case is that in which f to a stochastic time optimal control problem with the target set M =- O. Another special model has 0 = Rn , in which case T = 00. It is called an infinite horizon problem if the integral in (7.2) remains finite (as is the case when, for example, the function f is of the form e —ftg(t,x, u) with g bounded and y > 0). 7.2. Optimal stopping In the random duration model introduced in the previous subsection, the control influences the exit time only indirectly. One has a different model if the controller controls the ending time directly, which is called an optimal stopping problem. In such a situation, the stopping time itself becomes a part of the control. For instance, in the technology transfer example (Section 3.4), while one can consider the natural ending time of the technology (namely, the time when the economic value of the technology reaches zero), it is practically more important to study the early phase-out decision, that is, to decide when to terminate the use of the technology before its natural ending time. In the formulation of such models, an admissible control/stopping time is a pair (li(.), T) defined on a filtered probability space (11, F , {Ft}t>o,P) along with an m-dimensional Brownian motion W(.), where u (. ) is the control satisfying the usual conditions and T is an {.Ft } t>0 -stopping time. The optimal control/stopping problem is to minimize
(7.3)
J (u(), T)
f (t, x(t), u(t))dt + h(x(T))}
subject to (4.1) over the set of admissible controls and stopping times. 7.3. Singular and impulse controls In Problem (SS) presented in Section 4.1, the state changes continuously in time t in response to the efforts of the control. However, in many real situations, the state may be displaced drastically due to possible "singular behavior" of a control at a certain time instant. For example, in the
7. Other stochastic control models
87
insurance company model in Section 3.3, rather than paying out the dividends continuously, the company is more likely to distribute them once or twice a year. As a result, on a dividend payday the liquid assets reserve of the company changes drastically. The investment vs. consumption model presented in Section 3.2 is also just an approximation of what is really happening where changes of portfolios and withdrawals for consumption occur only at discrete time instants. A singular control can better model such situations. To begin, let us first introduce the following function space: D that are right D([0, T]; Ir) is the space of all functions Ç : [0, T] continuous with left limits (ceedlag for short). For Ç E D, we define the total variation of Ç on [0, T] by f[o,T] d(( s ) ICE I [0,71 , where I (i[o,T) = is the total variation of the ith component of Ç on [0, T] in the usual sense. We define i(it = i(i [o,t] t > 0, for simplicity. For Ç G D, we define E [0, T] A((s) 0}. Further, we define Ac(s)e ((s) ((s ) and S( —
—
BV([0,T];1Rn) =
E D IÇT <
For any Ç E BV ([0,T];Rn) we define the pure jump part of Ç by
Eo<s<, Lo), and the continuous part of Ç by (c(t) =- ((t)—( 3P(t). Since (c is still of bounded variation, it is differentiable almost everywhere, (iP (t)
and we have Ce(t) = ("(t) +("(t), t E [0, 7], where ("Me . c(s)ds. We call Cac the absolutely continuous part of (", and (se the singularly continuous part of Ç. Thus, we obtain the so called Lebesgue decomposition for -
E BV ([0,T]; IRn ):
(7.4)
«t) = ( ac (t) + ( 8 '(t) + ('(t),
t E [0, T].
It is clear that such a decomposition is unique. Now let ( 12, .F, {Ft} t>o, P) be a complete filtered probability space on which is defined an m-dimensional standard Brownian motion W(•). Let T];llr) be the set of all {Ft}t>o-adapted processes e(•) such that
f e(• ,w) E BV ([0,T]; Itn ),
1
<
For any xo E Rn and tic process
6 E BY F([0, T]; IV), we consider a controlled stochas-
(7.5) x(t) = x +
b(s , x(s))ds + j(s, x(s))dW (s) + (t), 0 5_ t < T,
j
and introduce the cost functional T =
Et
(7.6)
0 f8
o
f(t,x(t))dt +
(Older
0
f a (t)ile c (t)Ilidt
E tES[O,Tj
At, Ae(t)) + h(x(T))}.
Chapter 2. Stochastic Optimal .Control Problems
88
Here, f, fa, f8, L, and h are given functions, e(t) = Ce (0 -1- eSC (t)+VP(t) is its Lebesgue decomposition, and kar (t)II idt and Ick"(t)I are the measures generated by the total variations of ear and es`, respectively, with V denoting the L'-norm in R. Our goal is to minimize the cost functional (7.6) over BVF([0,7 ] ; Ra ). This is called an optimal singular control problem. A special case is that in which e" O. In this case e"(t) e fo ac(s)ds, and the problem reduces to a standard stochastic control problem as formulated in Section 4 if we regard 4.ar (•) as a new control variable. Another interesting case is that in which e takes a special form of a esc • pure jump process, i.e. , eac u in which case we get the so-called impulse (impulsive) control problem.
7.4. Risk-sensitive controls In the risk sensitive control model, both the system dynamics and an admissible control are defined in the same way as those of Problem (SS) in Section 4.1. The only difference is the way of defining the performance criterion. Instead of minimizing the direct cost as in (4.3), the controller aims to minimize some disutility of the cost in a risk-sensitive control problem. To be more precise, given a monotonically increasing function ep : R —0 R, the objective is to find an admissible control u(.) so as to minimize -
(7.7)
J (u0) = Ecp(foT f (t, x(t),u(0)dt + h(x(T)))
subject to the system (4.1). In economic theory the disutility function c,e) is usually taken to be either convex or concave, corresponding respectively to the risk averse or risk seeking attitude of the decision-maker. That is why the model is called a risk-sensitive one. Let us elaborate this point by an intuitive argument. Define X a- foT f (t, x(t), u(t))dt + h(x(T)) and suppose yo is differentiable at EX. Then we have the Taylor expansion -
-
-
ço(X)
(p (EX) + (,o'(EX)(X
—
EX) + ()oil (EX)(X — 2 1
—
EX) 2 .
If ço is strictly convex near EX, then ep "(EX) > 0, which introduces a penalty to the variance term (X — EX) 2 in the overall cost. This means that the controller tries to avoid large a deviation of X from its mean EX, that is, the controller is risk-averse. Conversely, if ço is strictly concave near EX, then (p" (EX) < 0, which reduces the overall cost with a large Ix EX I. This is the so-called risk-seeking case. Finally, if ço" (E X) is close or equal to 0, then E(x) (p(EX), in which case the risk-sensitive model reduces to the standard one in Section 4 (recall that (,e) is monotonically increasing). This corresponds to the risk neutral situation. A commonly used and extensively studied disutility function is the following exponential function: —
-
ipti (x) = exp(Ox),
89
7. Other stochastic control models
where 0 E 113. is a parameter representing the risk sensitivity degree of the c riterion. In particular, we is convex (respectively, concave) when 0 > 0 (respectively, 0 < 0). When 0 is sufficiently small, the problem can be well approximated by the standard stochastic control problem introduced in Section 4. Another frequently used disutility function is the so-called NARA util-
ity 1 –
if 'y
yo 7 (x) =
0,
x > 0,
ln x, if -y -=- 0, in which -y < 1, 7 > 1, and 7 = 1 correspond to risk-seeking, risk-averse, and risk-neutral situations, respectively. 7.5. Ergodic controls
Many stochastic systems, though randomly fluctuating in nature, do exhibit certain regular patterns over a sufficiently long time period. This regular pattern can be characterized by the so-called steady state distribution or invariant measure or ergodic distribution. The ergodic distribution, if it does exist, is independent of the initial state of the system and may be obtained by calculating the average of the states over a long time (called a long run average) under some conditions. Similarly, in an optimal stochastic control problem over a long time period, minimizing a long run average cost sometimes may better reflect the controller's desire to improve the performance on a long term and average basis. Such problems are called ergodic control -
-
-
-
problems. The system under this model is the same as (4.1), but now over the infinite time horizon [0, +oo). The admissible control is defined similarly. One typically seeks to minimize the expected long-run average cost
(7.8)
— 1 fT T E 0 f (t, x(t), u(t))dt, J (u0) = T li_11E10. -
or almost surely minimize the long-run average cost
(7.9)
J(u(•)) = lim
1
f f (t, x(t), u(t))dt.
1 0
Cost (7.8) is called a mean average cost, while (7.9) is called a pathwise average cost.
7.6. Partially observable systems In all the stochastic optimal control problems introduced so far, including Problem (SS), it is assumed that the controller is able to completely observe the system state. However, in many practical situations, it often happens that the state can be only partially observed via other variables, and there could be noise existing in the observation systems. For example, in the
Chapter 2. Stochastic Optimal Control Problems
90
technology diffusion example (Section 3.4) the true economic value of a technology may not be completely observable by the management; rather, they can be estimated only by other directly measurable entities such as quality level of the products or improvement degree of efficiency due to the adoption of the technology. Optimal control of such partially observed systems is called a partially observable (observed) stochastic control model. Let W and 147 be two independent standard Brownian motions on a filtered probability space (11 L - t,t>o,P), with values in IRrn and le, respectively. Consider the following controlled signal process: dx(t) = b(t, x(t), y(t), u(t))dt + cr(t, x(t), y(t), u(t))dW (t)
(7.10)
a(t, x(t), y(t), u(t))dik (t), x(0) = xo
with an observation process
f
(7.11)
dy(t) = ,3(t, x(t), y(t),u(t))dt + dW (t), Y( 0 ) == Yo.
Here a is called the correlation between the signal noise and observation noise. Let U be a given set in a metric space and set Yt
go- 0(s) :0 s
tl.
A control u(.) is called admissible if it takes values in U and is {Yc}t>oadapted. The objective of the problem is to choose an admissible control such that the following cost functional is minimized:
(7.12)
J (u(•)) =Elf f (t, x(t), y(t), u(t))dt + h(x(T), y(T))} 13
This is a substantially difficult problem compared with the completely observable case. One way of analyzing the problem is to formulate it as a completely observable stochastic control problem for a stochastic partial differential equation (SPDE for short). To do this, first we change the probability measure as follows:
di" a p-1 (T)dP , where p(t)
exp
(3(s, x(s), y(s), u(s))dy(s)
— fo
10 (s, x(s ),Y ( s ),u ( s) ) 1 2 ds}.
It can be shown that under the new probability F", W and y become independent Brownian motions. Now consider the unnorrnalized conditional
91
7. Other stochastic control models
probability p(t)(0) and its density p(t, x) determined by (7.13)
0(x)p(t, x)dx ,
(7,b(x(t))p(t)lYt)
P(t)( 1,b)
Rn
where 7,/, is any bounded Borel function on Rn (called a test function). Then p(t,-) satisfies the following SPDE (the Duncan Mortensen Z akai equation):
p(t)
-
dp(t) = A(t, y(t), u(t))p(t)dt + (7.14)
-
E M k (t , y(t),u(t))p(t)dyk(t), k=1
P( 0 )
= Po•
Here yk , k = 1, 2, ... , fiz, are the components of y, Po is the density function of xo, and the differential operators A and M k are given by
A(tb(x)
A t,j=1 n
(7.15)
aX t a
"(t X y" u) al'b(x) ) ' aX 3
a (a' (t, , y , u)0(x)) ,
Ei=1ax,
+13k (t, x, y, u)çb(x), Mk (t, y, u)0(x) g—Ectak(t,x, y, u)a°(x) ax, i=1
with ( 3 (t,x,y,u))
ai (t,x,y,u)
a- a(t, x , y,u)
a bi (t,x,y,u) -
a
(CIOT + ackT ) (t, x, y,u),
E aai (t X, , Y,U) .-i
n
j=1
hk (t,x, y,u) a fi k (t,x,y,u) -
EaCk iic (tO, ,'Xix' y ' u) , n
i=1
i,i = 1,2,...,n,
k= 1,2,...,fh.
To properly define the solution to (7.14), consider the Gelfand triple H1
H-17 where H° p(lRn),
ago
HI
is the Sobolev space
i = 1,2,...,n}
auxi with the norm
I I9V111
{Ln
1/2
( Ig4/ (X)I
2
n a V(X) 2 I E z=1 aXi
dX}
and H -1 is the dual of H 1 given that H° is identified to its dual. Denote by (• ,•) the inner product on H° and by ( • , • ) the duality pairing between
Chapter 2. Stochastic Optimal. Control Problems
92
1/ -1 and . Under this notation, the second-order differential operator A(t, y, u) given by (7.15) may be understood to be an operator from H I to 11 -1 by formally using the Green formula as follows: A(t, u)S 0 , 0)
(cti3(t,
—
y, u) aV(X) a0(X)\ aX, ) aX3
+E (az (t, x, y, u).0, 30(x)), Vcp, E HI . i=1
A {Yt } t xradapted, H I -valued process p E (0 , T; 11 1 ) defined on the filtered probability space (O, F, lYtIt>o, P) is called a solution to (7.14) if for each t/i E C 0 (Rn ) and almost all (t,(4)) E [0, x
+
(P( t),) = (Po
A(s, y(s), u(s))P(s),
ds
+f EcAlk(s,y(s),u(s))p(s),o)dyk(s). k=1 0
In terms of the new system (7.14), the original cost function (7.12) is converted to
J (u(•)) =Elf f (t, x(t), y(t), u(t)) p(t)dt + h(x(T), y(T))p(T)} (7.16) =Elf (f (t, • , y(t),u(t)), p(t))dt + (he , y(T)), p(T))} . Therefore, the original partially observable control problem can be reformulated as a problem of finding a U-valued, {Yt}t>o-adapted process u(.) such that (7.16) is minimized subject to (7.14). The advantage of this formulation is that the underlying system is completely observable and both the system dynamics and the cost functional become linear, and the disadvantage is, of course, that the problem is turned into an infinite dimensional one. -
8. Historical Remarks Although the name optimal control theory appeared in the late 1950s for the first time, the problem with essentially the same nature has a very long history; it can even be traced back to ancient times, when people discovered that the shortest path between two points is a straight line segment. The well-accepted precursor of optimal control theory is the calculus of variations, which was born in the 1600s. Let us give a brief history of it here, which also will provide certain background for some material presented in later chapters. In 1662, Pierre de Fermat (1601-1665) wrote a paper using the method of calculus to minimize the passage time of a light ray through
8. Historical remarks
93
two optical media. The result is now known as Fermat 's principle of least time. According to Goldstine [1], this probably can be regarded as the birth of the calculus of variations (although such a name was not given yet at Fermat's time). In 1685, Isaac Newton (1643-1727) studied the problem of characterizing the solid of revolution of least resistance. More interestingly, in June 1696, Johann Bernoulli (1667-1748) challenged the mathematical world at his time to solve the famous brachistochrone problem (which was first considered by Galilei Galileo (1564-1642) in 1638, but with an incorrect solution). The solution to this problem was given in the next year, 1697, by Johann Bernoulli himself, his brother Jacob (1654-1705), as well as Gottfried W. Leibniz (1646-1716), Newton, l'Hôpital (1661-1704), and Tschirnhaus (1651-1708). In 1744, Leonhard Euler (1707-1783) obtained the firstorder necessary condition of the extremals-called the Euler equation (or the Euler Lagrange equation). In 1755, Joseph L. Lagrange (1736-1813) introduced the so-called (5 calculus, which ushered in a new epoch in the area. After learning of this, Euler coined the name calculus of variations for the subject in 1756. In 1786, Adrien-Marie Legendre (1752-1833) introduced the second variation to study the sufficiency of an extremal to be either a maximum or a minimum. However, his paper contained a serious gap, which was later filled by Karl G. J. Jacobi (1804-1851) in 1838, leading to the socalled Legendre Jacobi theory. In 1833, William R. Hamilton (1805-1865) introduced Hamilton's principle of least action (which was vaguely stated by Pierre L. M. Maupertuis (1698-1759) in 1744). In 1834-35, Hamilton found a system of ordinary differential equations (which is now called the Hamiltonian canonical system) equivalent to the Euler-Lagrange equation. He also derived the Hamilton Jacobi equation, which was improved/modified by Jacobi in 1838. Afterwards, many scientists made extensive contributions to the field. Among others, we mention Karl K. Weierstrass (1815-1897), Ludwig Scheeffer, Rudolf F. A. Clebsch (1833-1872), Adolph Mayer (18391907), Oskar O. Bolza (1857-1942), Adolph Kneser (1862-1930), David Hilbert (1862-1943), Gilbert A. Bliss (1876-1951), Hans Hahn (1879-1934), and Constantin Carathéodory (1873-1950). It is well accepted that by the middle of the 20th century, the so-called classical theory of calculus of variations had been completed. For more detailed surveys and/or history of calculus of variations, see Goldstine [1], Giaquinta-Hildebrandt [1], Stampacchia [1], McShane [2], and Pesch-Bulirsch [1]. It is remarkable that among the famous 23 problems Hilbert posed at the International Congress of Mathematicians in 1900, the 23rd was on the calculus of variations, and the 19th and 20th were closely related to this subject. According to Bellman-Kalaba [1], the history of (mathematical) control theory should probably be traced back to the work of James C. Maxwell (1831-1879) in 1868 and J. Vyshnegradskii in 1876. Among many who made influential contributions in the field, we mention Adolf Hurwitz (18591919), Henri H. Poincaré (1854-1912), Aleksandr M. Lyapunov (18571918), H. Nyquist, Norbert Wiener (1894-1964), and Andrey N. Kolmogorov (1903-1987). -
-
-
-
94
Chapter 2. Stochastic Optimal Control Problems
Modern optimal control theory began its life at the end of World War II. The main starting point seems to be the (differential) games (for an obvious reason) studied both in the United States and the former Soviet Union. A group in the Rand Corporation, including R. Bellman, J. P. LaSalle, D. Blackwell, R. Isaacs, W. H. Fleming, and L. D. Berkovitz, were extensively working on differential games and related problems in the late 1940s and the 1950s. Such a study created a perfect environment for the birth of Be llman's dynamic programming method, which appeared publicly for the first time in 1952 (see Bellman [1]; Bellman claimed that it was obtained a couple of years earlier; see Bellman [5, p. 115]). On the other hand, according to L. S. Pontryagin (1908-1988) [3 ], his group, at the persistent request of the Board of Directors of the Steklov Institute of Mathematics, started the investigation of optimal processes of regulation in the fall of 1952, symbolized by the opening of a regular seminar at the Steklov Institute of Mathematics with the participants V. G. Boltyanski, R. V. Gamkrelidze, and E. F. Mishchenko. Among a large number of engineering problems, they singled out three types of problems: singular perturbation of ordinary differential equations, differential pursuit and evasion games, and optimal control theory. With their extensive study, Pontryagin's maximum principle was announced in 1956. In the late 1950s, R. E. Kalman [1] established optimal control theory for problems of linear systems with quadratic cost functional (LQ problems, for short). At the first IFAC (International Federation of Automatic Control) congress held in Moscow in 1960, Bellman's dynamic programming method, Pontryagin's maximum principle, and Kalman's LQ theory* were reported on, which formally announced the debut of modern optimal control theory. Randomness was considered in the early stages of the development of modern optimal control theory. The earliest paper mentioning "stochastic control" was probably that of Bellman [7] in 1958. However, the Ita-type stochastic differential equation was not involved. The first paper that dealt with systems with close resemblance to those of diffusion type seems to be by Florentin [1] in 1961, where Bellman's dynamic programming approach was used to derive a partial differential equation associated with a continuous-time controlled Markov process. In 1962, Kushner [1] studied the stochastic optimal control with the Itô-type SDE as the state equation. For detailed remarks concerning various aspects of this field, see historical remarks in the later chapters. In this chapter, Section 3 gives five practical examples that can be modeled as or approximated by stochastic optimal control problems. In fact, studies on these models themselves have been and remain very active. For the sake of the interested reader, here we sketch the literature of each of the 5 models. The earliest formulation of a deterministic production planning prob* These works now have been regarded as the three milestones of optimal control theory; see Fleming [7].
8. Historical remarks
95
lem was due to Modigliani-Hohn [1] in 1955 for the discrete-time case. Since then, various models including both deterministic and stochastic (diffusion) and both discrete-time and continuous-time have been studied. A systematic study and a comprehensive review of the literature in this area can be found in Bensoussan-Crouhy-Proth [1]. The stochastic production planning problem in Section 3.1 was formulated and studied by BensoussanSethi-Vickson-Derzko [1]. Other than diffusion models, production plann ing problems incorporating various discrete events (such as breakdowns of machines and sudden changes in demand) can be found in FlemingSethi-Soner [1] and Akella-Kumar [1]. The state processes involved in these works are the so-called piecewise deterministic process, the theory of which has been extensively accounted for in Davis [4]. The mathematical framework for constructing near-optimal production control policies using the singular perturbation and hierarchical control approach was set up in Lehoczky-Sethi-Soner-Talcsar [1] and further developed by Soner [2], Sethi-Zhang-Zhou [1,2], Sethi-Zhou [1], and Samaratunga-Sethi-Zhou [1]. Merton [1,2] was the first to formulate the continuous-time portfolioconsumption model (presented in Section 3.2) and apply stochastic control theory (basically the dynamic programming approach) to solve the problem in the late 1960s and early 1970s. Merton's results and technique have been further extended and developed by many authors since then. Among them we mention Richard [1], Breeden [1], Cox-Ingersoll-Ross [1], Merton [5], Fitzpatrick-Fleming [1], and Fleming-Zariphopoulou [1]. Another approach to solve the problem is the martingale method studied by Cox [1], Pliska [1], Karatzas-Lehoczky-Shreve [1], Cox-Huang [1], and ShreveSoner-Xu [1]. Reinsurance and dividend management for insurance companies is a rather new and largely unexplored area. The issue was introduced by Leland-Toft [1] in 1996 in the context of constant debt repayment, although no optimization problem was treated there. An optimal dividend payout scheme in terms of optimization of the corporate policies was studied in Asmussen-Taksar [1] and Jeanblanc-Piqué-Shiryaev [1]. More complicated models involving both control of profit/risk-related activities as well as dividend payouts can be found in Hojgaard-Talcsar [1], Radner-Shepp [1], and Taksar-Zhou [1]. The model introduced in Section 3.3 is based on that of Taksar-Zhou [1], the only difference being that one of the control variables, c(.) in (3.17), is the dividend payout rate, while a singular control problem is studied in Taksar-Zhou [1], where dividend distributions take place only at discrete time instants. Discussions on the three phases in the life cycle of a new technology can be found in Cook-Mayes [1] and Rogers [1]. Studies on the technology diffusion root in those of marketing diffusion (Bass [1], Horsky-Simon [1], Dockner-Jorgensen [1] and Klepper [1]). The stochastic diffusion models of technology transfer were formulated and studied in Reinganum [1], Roberts-Weitzman [1], Kulatilalca [1], Dixit-Pindyck [1], Chi-Liu-Chen [1], and Liu-Zhou [1]. In particular, the diffusion coefficient of the form (3.22)
96
Chapter 2. Stochastic Optimal Control Problems
was considered by Roberts-Weitzman [1], that of the form (3.23) was studied by Chi-Liu-Chen [1] (with u absent in (3.23)) and Liu-Zhou [1] (with u present in (3.23)). Diffusion approximation, typically applied to queueing systems ; is a technique to approximate some analytically difficult processes by diffusion processes, which have nice analytical structure and well-developed tools to handle. The simplest and easiest version of diffusion approximation dates back to 1951, when Donsker [1] put forward what is now known as Donsker's theorem, namely, sums of independent random variables can be approximated by Brownian motion. Prohorov [1] extended Donsker's theorem to doubly indexed families of random variables. Further, it was shown by Iglehart-Whitt [1,2] that a GIG/hoc queue (namely, a queueing system with a single server, both interarrival time and service time being of general distributions, and infinite storage capacity) in heavy traffic can be approximated by a one-dimensional reflecting Brownian motion. This forms a foundation for diffusion approximations of more complex queueing systems. A good tutorial on the relevant literature can be found in Glynn [1]. Along another line, the problem of scheduling a queueing network can be approximated by a stochastic control problem involving a diffusion process. Once the latter is solved, an optimal policy can be obtained in terms of the original queueing system. Justifications of such a procedure mainly based on intuition and simulation were provided by Harrison-Wein [1,2] and Wein [1]. A more rigorous treatment using value functions was done by Kushner-Martins [1] and Kushner-Ramachandran [1,2]. KrichaginaLou-Sethi-Taksar [1] applied diffusion approximation to study optimal production control of a single-machine system. The discussion in Section 3.5 is mainly based on Krichagina-Lou-Sethi-Taksar [1]. It should be noted, however, that a controlled queueing system is usually approximated by a singular control problem rather than the regular one proposed in (3.30). The one in (3.30) is a further approximation of the singular control problem. The main motivation for putting the diffusion approximation as one of the examples was to illustrate how diffusion processes are useful in various situations. At the very early stage of development of the stochastic control theory (e.g., Florentin [1] and Kushner [1]), the formulation of the problem was completely parallel to the deterministic case, and the strong formulation was taken for granted. However, people soon realized that one needs to consider the probability space and the Brownian motion as part of the control (i.e., the weak formulation) when they were studying the existence of optimality (e.g., Fleming-Nisio [1]). Since then, most people have employed the weak formulation in discussing stochastic control problems (see FlemingRishel [1] and Fleming-Soner [1] and the references therein), although the importance of such a formulation may not be explained or spelled out in every work. In this book we emphasize the relation and difference of the strong and weak formulations. We point out that while the strong formulation is natural from the practical point of view, the weak formulation,
8. Historical remarks
as
97
an auxiliary mathematical setup, is important and necessary not only in treating the existence of optimal controls but also in correctly applying the dynamic programming principle. Fortunately, for the problems that we study, the weak formulation does not change the probability distribution of the state process, hence nor the expected objective due to the weak uniqueness of the state equation (under certain reasonable conditions). This is the essence behind the weak formulation. The existence of deterministic optimal control (with merely measurable controls) was first studied by Roxin [1] based on the so-called Roxin condition and a measurable selection theorem. See Berkovitz [1] and Cesari [1] for extensive study of the deterministic existence problems. The existence of optimal controls for processes of diffusion type had been a quite active research area during the early stage of the development of the stochastic optimal control theory. The results obtained really depended on the different formulations of the problem, but the key was to employ and/or explore certain compactness properties. Kushner [4] considered the class of controls that are functions of the time and current state and Lipschitz continuous in state. Fleming-Nisio [1], using Prohorov's topology, studied the case where the control was separated from the state in the dynamics. Bene g [1,2] and Duncan-Varaiya [1] derived the existence for controls that depended on the whole past history of the state based on the Girsanov transformation. Almost at the same time, Davis [1] employed dynamic programmipg and the verification theorem to derive the existence of optimality. Kushner [8] made use of the martingale representation theorem to handle the problem. Along a different line, the concept of relaxed control as a compactification device, first introduced by Young [1] for the deterministic case (see also Warga [1] and Balakrishnan [1, pp. 31-36]) for a detailed exposition), was applied by Fleming [5] to handle the stochastic existence. It was followed by El Karoui-Huu Nguyen-Jeanblanc-Picqué [1], who extensively investigated the existence of optimal controls by this approach. As with the deterministic case, a relaxed control reduced to a normal control under the Roxin condition. The proof of Theorem 5.3 seems to be new (as the diffusion is allowed to depend on the control), albeit one with combined ideas from Fleming-Nisio [1], Kushner [8], and Fleming [5]. Materials presented in Section 6 are based on Yong [10]. Example 6.4 is due to S. Tang. The results in this section are used to demonstrate the difference between the stochastic and deterministic systems in terms of the reachable sets, which play an essential role in studying the problems of controllability and existence of optimality. Again, the models introduced in Section 7 have been and are still being widely studied. Let us give brief historical remarks for them for the convenience of the reader. The first to introduce the optimal stopping problem was Chernoff [1], in 1968. It was soon discovered to be related to the free boundary problem of ice melted in water, the so-called Stefan problem (see van Moorbecke [1]). Bensoussan-Lions [1] employed the variational inequality technique to solve
98
Chapter 2. Stochastic Optimal Control Problems
the problem. For a detailed account of such a treatment, see Bensoussan [2] and Friedman [1]. Another approach mainly based on the probabilistic argument, can be found in Dynkin [1], Shyriaev [1], and Krylov [2]. Bather-Chernoff [1] were the first to formulate a stochastic singular control problem in their study of spacecraft control in 1967. Beneg-SheppWitsenhaussen [1] proposed what is now known as the principle of smooth fit to explicitly solve a one-dimensional singular control problem based on the dynamic programming equation and the smoothness conjecture of the value function. Using the same idea, the one-dimensional case was solved completely by taking the form of the so-called monotone follower problem; see Karatzas [2], Karatzas-Shreve [1,2], Harrison-Taksar [1], Menaldi-Robin [1], and Chow-Menaldi-Robin [1]. For higher dimensions, the problem is much more difficult. Smoothness of the value functions was studied by Evans [1], Ishii-Koike [1], and Soner-Shreve [1,2]. When the value function is smooth, it was shown in Menaldi-Taksar [1] and Soner-Shreve [1,2] that construction of an optimal control (verification theorem) has to rely on the solution to the Skorohod problem (Skorohod [1], El Karoui-ChaleyatMaurel [1]). There has been substantial research effort devoted to the Skorohod problem; see Strook-Varadhan [1], Tanaka [1], Lions-Sznitman [1], Varadhan-Williams [1], Saisho-Tanaka [1], Kurtz [1], Dupuis-Ishii [1], and Talcsar [1]. Another interesting problem is the relationship between stochastic singular control and optimal stopping in the one-dimensional case, which states that the value function of the latter is nothing but the derivative in the state variable of the former. See Bather-Chernoff [1] and Karatzas-Shreve [1,2]. On the other hand, the existence of stochastic singular controls was investigated by Krylov [2], Menaldi-Taksar [1], and Haussmann-Suo [1]. The maximum principle was studied in CadenillasHaussmann [1], and the viscosity solution to the HJB equation was discussed in Lenhart [1], Tang-Yong [1], and Ma-Yong [2]. The (stochastic) impulse control problem was initiated by Bensoussan-Lions [2] in 1973, and a standard reference is Bensoussan-Lions [3]. For the deterministic counterpart, see Barles [1] and Yong [1,2,6,7]. A closely related problem is the so-called optimal switching problem. For this, we mention the works by Capuzzo-Dolcetta-Evans [1], Stojanovic-Yong [1,2], Lenhart-Yamada [1], and Yong [3-5]. The presentation of Section 7.3 is based on Ma-Yong [2]. Risk-sensitive control has been a very active research area recently. It was first put forth (with the exponential criterion) by Jacobson [1] in 1973 for linear systems, and then followed by Whittle [1], Speyer-DeystJacobson [1], Speyer [1], Kumar-Van Schuppen [1], and Bensoussan-Van Schuppen [1] in the early stage of this theory. Research on the related dynamic programming equation can be found in Nagai [1] and BensoussanFrehse-Nagai [1], who mainly employed some PDE methods. One of the most significant features of risk-sensitive control is that it provides a link between optimality and robustness. More specifically, when the system noise approaches zero, the risk-sensitive control problem converges to a deterministic differential game problem related to or robust control. This idea
8. Historical remarks
99
was initiated by Whittle [1,2] in an intuitive way, and was made rigorous using large deviation and viscosity solution theory by Fleming-McEneaney [1,2] and independently by James [1]. More work on the small noise limit can be found in Bensoussan-Nagai [2], Fleming-James [1], Runolfsson [1], Fleming-Hernández-Hernández [1,2], and James-Baras-Elliott [1]. Note control theory has been extensively developed in parallel; see that the Ba§ar-Bernhard [1]. Study on ergodic control dates back to 1957 when Bellman [6] dealt with the discrete-time case. The primary approach to continuous-time ergodic control is dynamic programming; see for example Lasry [1], Tarres [1,2], and Cox-Karatzas [1]. See also a survey paper by Robin [1]. It is interesting to note that the corresponding dynamic programming (HJB) equation, involving an unknown constant and an unknown function, has a close relation with the eigenvalue problem of a Schrödinger operator (an elliptic operator). See Karatzas [1], Bensoussan -Nagai [1], and Bensoussan-Frehse
[1]. Optimal control of partially observable systems consists of two components: estimation and control. The estimation part is related to the filtering problem. The linear filtering problem was elegantly solved by the ubiquitous Kalman-Bucy filter [1] in 1961. For nonlinear filtering, which is much more difficult, there have been two essentially different approaches. One is based on the so-called innovation process, which was initially developed by Kailath [1] and Kailath-Frost [I] in 1968. This theory achieved its peak with the famous paper by Fujisaki-Kallianpur-Kunita [1]. See also the books by Liptser-Shiryayev [1] and Kallianpur [1] for a systematic exposition of this approach. Another approach was introduced during the middle to late 1960s by Duncan [1], Mortensen [1], and Zakai [I] independently, who derived a stochastic partial differential equation (widely known as the Duncan-Montensen-Zakai or DMZ equation) satisfied by the unnormalized conditional density function of the state. This approach was inspired by the function space integration introduced by Kac [2] and Ray [I], and has some deep relation with the path integral formulation of Feynman-Hibbs [I] in quantum physics (see a detailed discussion on this point in Mitter [1]). In fact, it was this relation that motivated Mitter [1], along with Brockett [1] and Brockett-Clark [1], to introduce the concept of estimation algebra and use Lie-algebraic and geometric techniques to solve the nonlinear filtering problem. For details on the geometric method see an earlier survey by Marcus [1] and a recent survey by Wong-Yau [1]. Unlike the finite-dimensional Kalman-Bucy filter, nonlinear filtering in general gives rise to infinite-dimensional filters (Hazewinkel-Marcus-Sussmann [1], Chaleyat-Maurel-Michel [1]), which are computationally insurmountable. People have been working on finding examples with finite-dimensional filters, including the most notable Bene g filter [3], as well as those proposed by Ocone [1], Wong [1], Haussmann-Pardoux [1], Charalambous-Elliott [1], and Yau-Yau [1]. As for control of partially observed processes, in the case of linear systems there is the so-called separation principle or cer-
100
Chapter 2. Stochastic Optimal Control Problems
tainty equivalence principle, originally discovered by Wonham [1] in 1968. This principle allows one to solve the estimation (filtering) problem first, and then to solve a completely observable control problem driven by the observer dynamics (sufficient statistics of the filtering problem). See the book of Davis [2] for a systematic account of linear filtering and control theory. For nonlinear partially observable control problems, however, the separation principle may not hold. One may then identify an information state given by the Duncan-Mortensen-Zakai equation and control a completely observable process in terms of the information state. This approach was sketched in Section 7.6, which nevertheless leads to an infinite-dimensional control problem. There is a vast amount of research on the nonlinear partially observed problem. We mention in particular Fleming [2], Rishel [1], DavisVaraiya [1], Beneg-Karatzas [1], Bensoussan [4,6], Haussmann [5], Lions [3], Baras-Elliott-Kohlmann [1], Bensoussan-Nisio [1], Karatzas-Ocone [2,3], Zhang [1], Zhou [6,7], and Tang [1]. For study on stochastic partial differential equations including DMZ equations, see Pardoux [1], Kunita [1,3], Walsh [1], Krylov-Rozovskii [1-4], Rozovskii [1], and Zhou [8]. There are some other important stochastic control models and problems that are not mentioned in this book, including the problem of maximizing the probability of a certain event (Kulldorff [1] and Karatzas [4]) and the related Monge-Ampère equation (Pogorclov [1] and Krylov [1]), piecewise deterministic processes (Davis [4]), hybrid control problems (Branicky-Borkar-Mitter [1]), anticipative controls (Davis [3]), stochastic controls in Riemannian manifolds (Duncan [2] and Duncan-Upmeier [1]), controls of backward stochastic differential equations (Peng [5], Chen-Zhou [1], and Dokuchaev-Zhou [1]), stochastic near-optimal controls (ElliottKohlmann [1] and Zhou [12]), and numerical methods (Kushner-Dupuis [1] and Kushner-Yin [1]).
Chapter 3
Maximum Principle and Stochastic Hamiltonian Systems 1. Introduction One of the principal approaches in solving optimization problems is to derive a set of necessary conditions that must be satisfied by any optimal solution. For example, in obtaining an optimum of a finite-dimensional function, one relies on the zero-derivative condition (for the unconstrained case) or the Kuhn-Tucker condition (for the constrained case), which are necessary conditions for optimality. These necessary conditions become sufficient under certain convexity conditions on the objective/constraint functions. Optimal control problems may be regarded as optimization problems in infinite-dimensional spaces; thus they are substantially difficult to solve. The maximum principle, formulated and derived by Pontryagin and his group in the 1950s, is truly a milestone of optimal control theory. It states that any optimal control along with the optimal state trajectory must solve the so-called (extended) Hamiltonian system, which is a two-point boundary value problem (and can also be called a forward-backward differential equation, to be able to compare with the stochastic case), plus a maximum condition of a function called the Hamiltonian. The mathematical significance of the maximum principle lies in that maximizing the Hamiltonian is much easier than the original control problem that is infinite-dimensional. This leads to closed-form solutions for certain classes of optimal control problems, including the linear quadratic case. The original version of Pontryagin's maximum principle was for deterministic problems, with its key idea coming from the classical calculus of variations. In deriving the maximum principle, one first slightly perturbs an optimal control by means of the so-called spike variation, then considers the first-order term in a sort of Taylor expansion with respect to this perturbation. By sending the perturbation to zero, one obtains a kind of variational inequality. The final desired result (the maximum principle) then follows from the duality. However, one encounters an essential difficulty when trying to mimic this idea for optimal control problems of stochastic (diffusion) systems if the diffusion terms also depend on the controls. The main difficulty, roughly speaking, is that the Itô stochastic integral ftt+E crdW is only of order (rather than E as with the normal Lebesgue integral); thus the usual first-order variation method fails. To overcome this difficulty, one needs to study both the first-order and second-order terms in the Taylor expansion of the spike variation and come up with a stochastic maximum principle involving a stochastic Hamiltonian system that consists of two forward-backward stochastic differential equations and a maximum condition with an additional term quadratic in the diffusion coefficient.
102
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
The rest of this chapter is organized as follows. Section 2 reviews the deterministic case with both necessary and sufficient conditions for optimality presented. In Section 3, a statement of the stochastic maximum principle is given in which the stochastic Hamiltonian system is introduced. Examples are presented to illustrate the result. Section 4 is devoted to a detailed proof of the stochastic maximum principle, while Section 5 discusses the sufficiency of the principle. Problems with terminal state constraints are dealt with in Section 6. Finally, some historical remarks are given in Section 7. We recommend that the reader skip Sections 4 and 6 at first reading. 2. The Deterministic Case Revisited Let us first recall the formulation of the deterministic optimal control problem (as presented in Chapter 2, Section 2). We will also introduce some standard assumptions. Consider the following control system:
(2.1)
d(t) = b(t, x(t), u(t)),
a.e. t E [0, T],
x(0) = xo,
with the cost functional
(2.2)
..1(u()) = f f (t, x(t),u(t))dt + h(x(T)).
Let us assume the following: (Dl) (U, d) is a separable metric space and T> 0. (D2) The maps b: [0,1] x 1Rn x U Rn , f: [0,T] x 1Fr x U R, and h : Rn -> R are measurable, and there exist a constant L > 0 and a modulus of continuity W : [0, oc) -> [0, oc) such that for ço(t, x, u) = b(t, x, u), f (t, x,u), h(x), we have
bp(t,x,u) —
5 Lx —
(2.3)
lw(t, 0, u)l
<
L,
I+
Vt E [0, E Ftn , It, V (t, u) E [0,T] x U.
E
U,
(D3) The maps b, f, and h are C 1 in x, and there exists a modulus of continuity "cti : [0, co) -> [0, oo) such that for ço(t, x, u) = b(t, x, u), f (t,x,u),h(x), we have
(2.4)
i491(t,x,u) - cps (t,i, fi.)I < W(Ix Vt E [0, 71,
d(u, fi)), E
R'1 ,u ,
E U.
Let V[0, T] = fu (. ) : [0, T] U u ) measurable 1. It is clear that under (D1)-(D2), for any u(.) E V[0, T], equation (2.1) admits a unique solution x(-) x(- ; u()) and (2.2) is well-defined. Our deterministic optimal control problem can be stated as follows. (.
2. The
103
deterministic case revisited
Problem (D). Minimize (2.2) over V[0, 7]. Any 771,(•) E 110,T] satisfying
(2.5)
J((.))
= u( )E
j(u"
a
is called an optimal control. The corresponding state trajectory Y ( . ) .(• ; ft()) and (Y ( ), U(-)) are called an optimal state trajectory and optimal pair, respectively. Note that unlike the case in Chapter 2, Section 2, we consider here only the case where there is no state constraint, for which the feasibility and the admissibility of a control coincide. The following is the well-known Pontryagin's maximum principle, which gives a set of first order necessary conditions for optimal pairs. -
Theorem 2.1. (Deterministic Maximum Principle) Let (D1) (D3) hold. Let (I(.),17()) be an optimal pair of Problem (D). Then there exists a p(.) : [0, T] IV satisfying the following: -
( 2.6)
p(t)
(t, (t), ii(t)) T p(t) + f
(t), U(t)),
a.e. t E [0,
and
H (t, Y(t),Ti(t),p(t)) = m u Eatxj H(t,T(t),u,p(t)), a.e. t E [0, T],
(2.7) where
H(t,x,u,p) = (p, b(t, x, u)) - f (t, x, u), (t, x,u,p) E [0,7] x Rn X U
(2.8)
X Rn.
We call p ) the adjoint variable/function and (2.6) the adjoint equation, respectively (corresponding to the pair (Y (. ), fi ))), and (2.7) the maximum condition. The function H defined by (2.8) is called the Hamiltonian. The state equation (2.1), the corresponding adjoint equation (2.6), along with the maximum condition (2.7), can be written as (.
(.
(t) = Hp (t, x(t), u(t), p(t)),
go _=.
(2.9)
a.e. t E [0, 7], -111 (t, x(t), u(t), p(t)), a.e. t E [0, 7 ] ,
p(T) = x(0) = xo, H (t, x(t), u(t), p(t)) = ir,i m H (t, x(t), u, p(t)),
a.e. t E [0, T].
{
The above system is called an (extended) Hamiltonian system. It partially characterizes the optimality of the problem. In cases where certain convexity conditions are presented, the Hamiltonian system of the above form fully characterizes an optimal control. In what follows, if (x(.), u(.)) is an optimal pair and p (. ) is the corresponding adjoint function, then (x(•),u(•),p(•)) will be called an optimal triple.
104
Chapter
3.
Maximum Principle and Stochastic Hamiltonian Systems
In the rest of this section we give a proof of Theorem 2.1. The main ingredients are the Taylor expansion of the state trajectory and the cost functional with respect to the perturbation of the control variable, and the duality between the variational equation (or the linearized state equation) and the adjoint equation. In doing this, we need the so-called spike variation technique. Let us make it more precise below. Let (Y(•),U(•)) be the given optimal pair. Let E > 0 and E, C [0,71 be a measurable set whose Lebesgue measure is 1E,1 =- e. Let u( • ) E V[0, 7 ] be any given control. We define the following:
(2.10)
u E(t) =
U(t) ' u(t),
if t E [0,T] \ Ee , if t E E.
It is clear that u'(.) E V[0,7 ] . Here, we should point out that since U is just a metric space, it does not necessarily have a linear structure. Thus, in general, a perturbation like ii(t)+6u(t) is meaningless. We refer to uE(.) as a spike (or needle) variation of the control Ti(.). The following result will be very useful in proving our maximum principle. Lemma 2.2. Let (D1)-(D3) hold. Let se (•) ;ue (•)) be the solution to (2.1) under the control 14), and let yE(•) be the solution of the following:
Y E (t) = bx (t,Y (t), Tt(t))ye (t) (2.11)
+ fb(t,Y(t),u(t))-b(t,Y(t),Ti(t))1xE(t), a.e. t
E [0, T],
yE (0) = 0. Then
{ max ixe(t) - Y(t)I = 0(e), tE[0,7]
(2.12)
max
tE [O ,T]
(2.13)
IYE (01 = 0(E),
max Ix(t) - Y(t) Y e (t)1= 0 (e),
and J(it (•)) - J((.)) = ( hx (Y(T)), ye (T))
(2.14)
f
fx (t,Y(t),tit(t)),V (t))
+ {f (t, (t), u(t)) - f (t,Y(t), U(t))}xE (t) 1dt o(E).
We refer to (2.11) as the variational equation along the optimal pair (Y(.), /Te(.)), and to (2.13) and (2.14) as the Taylor expansions of the state
and the cost functional with respect to the spike variation of the control variable, respectively.
2. The deterministic case revisited
e'(t)
Proof. Let
105
xe(t) ±-(t). Then, by (2.3), we have
i e (t)i 5_ f L1C e (s)ids + K e ,
(2.15)
V t E [0, 7 ] .
Hereafter, K represents a generic constant, which may differ at different places. By Gronwall's inequality, we obtain the first relation in (2.12). The second relation in (2.12) can be proved similarly. Next, we set
n e(t)-_'axe(t)— Y(t)
(2.16)
—
yE (t) E_-- ee (t) — ye (t).
Then (noting (2.10))
(t) = b(t , xE (t), ue (t)) — b(t, Y(t), TIM) — b s (t, (t),11(t))ye (t) — {b(t,Y(t), u(t)) — b(t,Y(t),1l(t))1xE(t) =-- f b,(t,Y(t) + OCE (t),uE (MO • e' (t) — b r(t,Y(t), U(t))ye (t)
=
1 [ f
Ibr (t,Y(t) + OCE (t), ILE (t)) — b,(t,Y(t), ?LE (t))1dOle (t)
+ {b,(t,Y(t), u(t)) — b,(t, (t), 11(t))1Ce (t)xE(t) + b,(t,Y(t),ii(t))if (t). Thus, it follows that (noting (2.12) and (2.4)) (2.17)
MI
5 f LI/f(s)Ids + Kel
(lee (s)Dds K e 2
Hence, (2.13) follows from Gronwall's inequality. In the same manner, we can prove (2.14). Based on the above lemma, we now carry out a proof for Theorem 2.1. Proof of Theorem 2.1. Let p(.) be the solution of (2.6). Then applying the differential chain rule to ( p(t), y' (t)) , we have the following duality relation:
( hx(Y(T)), Y E (2")) = (P(T), Y e (T) ) — (P( 0) Y e (0) ) = (2.18)
f
( fx (t,Y(t),U(t)), ye (t))
+ ( p(t), b(t , (t), u(t)) — b(t,T(t),17,(t))) x E (01 dt The above is called the duality relation between ye() and p(.) (or that between the variational equation (2.11) and the adjoint equation (2.6)). Fix a time i E [0, T) and a u E U. Let u(t) u and Ee = R, + El with e > 0 being sufficiently small such that E, C [0, T]. Combining (2.18) with -
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
106
(2.14) and noting the optimality of i(.), we obtain 0< J(u'(•))— J((.)) = (hx (Y(T)), ye (T))
f( f x (t,Y(t),U(t)), ye (t)) + { f (t, (t), u(t)) — f (t,Y (t), 77,(t)) 1x (01 dt + 0(0 7+E =—
{H (t,Y(t), u(t), p(t)) — H (t,T(t),U(t), p(t))}dt + o(e).
This, together with the separability of the metric space U, leads to the maximum condition (2.7). We have seen that in the above proof, the starting point is the optimality of the optimal control U(.): (2.19)
J(ue (•)) — J(11(.)) > 0.
To obtain the first-order necessary conditions from this, we need some sort of Taylor expansion. Since V[0, T] does not necessarily have a linear structure, such an expansion has to be developed carefully. Once this is achieved, i.e., once (2.14) is obtained, we then have a necessary condition of the optimal pair (TO, ii(•)) in terms of ye(.) (the solution of the variational equation) that depends on the choice of u (.) e V[0,7]. However, (2.14) is a sort of implicit necessary condition that is not easily applicable, as it involves y'(•). Thanks to the duality relation (2.18), we are able to get rid of ye(.). Then our desired result follows. In summary, the maximum principle is proved via the Taylor expansion and the duality relation. We point out that Theorem 2.1 remains true if f and h are continuously differentiable in x with a polynomial growth in x; in particular, f and h are allowed to have quadratic growth in x. The reason is that the state trajectory x (. ) obtained from the state equation (2.1) is bounded as long as b has linear growth. The above maximum principle gives some necessary conditions for the optimal controls. In other words, it gives a minimum qualification for the candidates of optimal controls. One will naturally ask whether a given control satisfying the necessary conditions is indeed optimal. We now investigate sufficient conditions for optimality. To this end, let us introduce the following assumption. (D4) The control domain U is a convex body (i.e., U is convex and has a nonempty interior) in Iftk , and the maps b and f are locally Lipschitz in u. The following lemmas will be useful in the sequel and in later context.
Lemma 2.3. Let G C IV be a region and yo G —> IR be a locally Lipschitz continuous function. For any x E G, define
(2.20)
A
a(x) =
E
Rn ( C y ) <
fiTri t 0
cp(z + ty) — (z)
2. The deterministic case revisited
107
which is called Clarke 's generalized gradient of cp at x. Then:
(i) aw(x) is a nonempty, convex, and compact set in Rn .
(ii) a(-- y,)(x)
= —ap(x).
(iii) 0 E ayo(x) if yo attains a local minimum or maximum at x. (iv) Let D(cpx ) = 1z E G i cpx (z) exists 1. Then l lirn 99x (xk) 1 xk E D(cpx ), xk — x,
aço(x) =
k—.00
(2.21)
lim yax (xk) exists}.
k ,—.c o
(y) If in addition we assume that G C itn is a convex body and ça : G — ■ R is a convex function (which implies the local Lipschitz continuity), then
(2.22)
a(P(x) = acY7(x) g le E Rn 1 (e)y) 5 (p(x + y) - w(x), Vy E Rn , with x + y E G}.
Proof. (i) By assumption, for the given x E G and a small e5 > 0 with 135(X)
a z E Rn 1 1Z — xi < 6} {
iça(z) —ça(
C G, there exists a constant L > 0 such that
Vz,:i- E B6 (X).
5 Liz — îl,
Thus, for any fixed y E IRE , icp(z + ty) — yo(z)I
t
Vz e B5/2 (x),
5 LIYI,
provided that t > 0 is small enough. Hence, the following exists:
wo (x; y) A
..n 40(z
± ty)— w(z)
—,zEG to
t
It is easy to check that
(2.23)
e(x; AY) = AV ° (x; y); VY C ir; A > 0; 40° (x; Y + z) 5 90° (x; Y) + e(x; z), Vy, z E Rn .
Thus, y )— yo ° (x; y) is a convex function. Also, the second relation in (2.23) implies that
(2.24)
— ,(x; Y)
40° (x; — y), VY C IRn .
Next, we fix a z E Rn and define F : {Az 1 A E IR} — ■ R by
F(Az) = Ace(x; z),
VA E R.
Then for A > 0 (noting (2.23) and (2.24)),
{ F(Az) F(—Az)
Ace (x; z) = 99° (x; Az),
— Ace(x; z) = — 40° (x; Az) 5_ 50° (s, —Az),
108
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
which yields
F(Az) < cpc ) (x; Az), VA E R.
(2.25)
Therefore, F is a linear functional defined on the linear space spanned by z, and it is dominated by the convex function ce(x; .). By the Hahn—Banach theorem (see Yosida [1, p. 102]), there exists a C E Rn such that
{ (C,Az) = F(Az) -.=7- Ayo ° (x; z),
(2.26)
VA E 111,
40° (x; Y), VY E Rn .
(e7 y)
This implies C E ayo(x). The convexity and compactness of aço(x) follow easily from (2.20).
(ii) We note that
=
Tr—ri —(p(z -I- ty) + (p(z) t
17(7,
—yo(e) + yo(z i — ty) = (po (x; —y). t
= s' -. x
t 10
Thus, C E ô( — )(x) if and only if ( —C y ) = (C — Y) ( —W) ° (x; — y) =40° (x;Y), VY E Rn I which is equivalent to —C E O(x). (iii) Suppose ço attains a local minimum at x. Then ± tY) — (p(x) (P° (x; y) =. fiji (P(z t ITIT
> riTri f) ( x + ty) — w(x) (o
> 0 = (0,y), Vy E Rn
t
t 10
.
This implies 0 E aço(x). If cif) attains a local maximum at x, then the conclusion follows from the fact that —yo attains a local minimum at x along with (ii). (iv) Define
D*so(x) = { lim
k— ■ oo
I xk E D(CPx), Xk
(pk(xk)
—4
z,
liin Slily (Xk )
k— .oc
exists}.
For any C E D'yo(x) and y E Rn , there exists a sequence xk E D(cpx ) such that xk —, x and wi (xk) —+ C. Then
(CY) = iirll k--•oo
(Wx(Xk),Y)
,p( xk ± ty) — yo(rk)
= kLiMooli-710 This implies C E
(2.27)
aw(x).
t
< ça0
(
z y) . ;
Hence, by the convexity and closeness of a(p(x),
-
D*(,o(x)
Ç
Oyo(x).
109
2. The deterministic case revisited Next, we define
(2.28)
Ci6° (x; y) =
max
E -07, D*4p(x)
(
11m
)=
p A3z— ■ s
(401(z), y)
Note that by (2.26) (with A = 1) and (2.27), one has
° (x; y) = max ( Y ) >
(2.29)
We claim that equality holds in (2.29). In fact, since eio is (locally) Lipschitz, by Rademacher's theorem (Evans-Gariepy [1, p. 811), D((px ) is of full 0 be fixed. By the definition of measure, or ID(cp x )cl = 0. Now, let y Ce(x; y), for any E > 0 there exists a 6 > 0 such that
(2.30)
(Vx(z),Y)
Vz E B26(X)
S-043 (x; Y)+ el
n D(Çox).
Next, for any ( E B6(x), let
(2.31)
+ 0 + ty 0 E {y} 1 n B71 (0), 0 < t < q} ,
Rn
with 17 > 0 small enough so that HT, C B26(x). (z,y)= 0}. Now, by Fubini's theorem,
0 = ID(cpx )c
n R, =
f(y} 1-nB,,(0) JO
Here, {y}± 8-{z E
ID( wx y(( + 0 + ty)dt d0.
Thus, we obtain
+ 0 + ty E D(cpx ),
a.e. t E (0,77), a.e. O E {y} 1 n B(0).
Consequently, for almost all 0 E {y} ± n Bn (0) (in an (n - 1)-dimensional Euclidean space), along the line segment 1( + 0 + ty t E (0, /i )} , we have (noting (2.30) and /7-,1 ç B26(x)) (,o(( + 0 + ty) (p(( + 0) -= f (yox (C + 0 + sy),y) ds < (C Ax; y) + E)t. Since ep is continuous, the above implies that
(2.32)
cp(z + ty) - ep(z) 5_
(x; y) + E)t,
V z E B6(x), t E (0,77).
This results in ço°(x; y) < (x; y), which further leads to the equality in (2.29). Then, the convexity and closeness of the sets t6D*(p(x) and acp(x) yields that they have to be the same, proving (iv). (N) Let us first prove that (,c) is locally Lipschitz. We claim that for any
xo E G and r > 0 with B2(x0)1x E Rn
IIx — xol
G,
110
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
one has
(2.33)
I49 (x)
ça(Y)1 _5_ -2 ( sup 150 (4)1x yl, Vx,y E Br(x0)• r zEB2,-(so)
To show (2.33), we note that if 0 : [a, 13] < si < sz < .53 < 0, then
R is a convex function and
0(s2)- 0(si) < 0(s3) - 0(si)
(2.34)
S2 - 51
Indeed, noting s2 = 93 -si that
8 3 - SI
we obtain, by the convexity of 0,
+ 33t -s S3 - 82
0(32) < 0(S1) - S3 - 8 1
52 - 5 1
83 -
0(S3).
Si
Rearranging the above gives (2.34). Now we prove (2.33). The case x = y is trivial. Thus, let x clear that there exists a unique t> iy - xi such that 2Ax i Y
1Y
x E
xl
132r(x0) fx IR n
ix
-
y. It is
xoi = 21'1.
Then one must have t =12 - xi> r. Define
Clearly, 0: [0, i]
R is convex, and
0(0) = v(x), 0(iy - xi)
= v(y),
0(i) =
Now, by (2.34), we obtain
cp(y)
IY
O(iy - xi) - 0(0)
- cp(x)
xl
<
IY xl 0(t) -0(0) _
=
w(2)
12 - xl
sup
r zEB2r (r0 )
lep(z)1.
Exchanging x and y, we obtain (2.33). This proves the local Lipschitz continuity of cp(.). Next, we prove (2.22). To this end, we first note that for any y E Rn, there exists a t> 0 such that x + ty E G for all t E [ t, t]. The function -
0(t) =cp(x + ty) is convex on [-t, t]. Thus, by (2.34), the mapping t ip(x+ty)-i,o(s) is nondecreasing for t E (0, i] and bounded from below. This implies that the limit ap(x)(y) A lim
cp(x
+ ty) - yo(x)
tic)
exists. Now let 5(p(x)
E
Rn ( C y ) < ap(x)(y),
Vy E IRnI.
2. The
111
deterministic case revisited
We claim that acw(x) = 5W(x). In fact, any y E Rn and all small enough t > 0,
E
ac (p(x)
if and only if for
y ) < yo(x + ty) - w(x) which is equivalent to
(2.35)
E6 - 90(x). Thus, to prove (2.22) it suffices to prove
yo° (x; y) = Dw(x)(y),
Vy E Rn .
First of all, by definition, we have
Dcp(x)(y) yo ° (x; y),
Vy E Rn .
To prove the reverse inequality, we note that the function (t, z) so(z-l-ty)—ço(z) is continuous at (s,x) (with s > 0 and small). Hence for any E > 0, there exists a 6 > 0 such that
y9(z + ty) - (p(z) < (p(x + sy) - yo(x)
E,
with t > 0, It - si +1z - xl < V(t,Z) 6. By the monotonicity of t 1-4 w (z+tYt)- w (z) , the above inequality holds for all t > 0 small enough. Now, we first let (t, z) (0, x), and then s 0 in the above inequality to get
< yo(x + sy) - yo(x)
E
D ço (x ) (y) + e, as s
O.
Hence, (2.35) holds, and (2.22) follows. In convex analysis, for a convex function yo : G RE , the set defined on the right-hand side of (2.22) is called the subgradient of w at x (cf. Rockafellar [1]). Thus, the above (N) implies that for convex functions, Clarke's generalized gradient coincides with the subgradient. By fixing some arguments of a function, one may naturally define its partial generalized gradient. For example, if (x, u) is locally Lipschitz, by ex (p(x , u) we mean the (partial) generalized gradient of yo in x at (x, u). Let us present one more technical lemma. Lemma 2.4. Let w be a convex or concave function on Rn x U with U C 111,k being a convex body. Assume that (x, u) is differentiable in x and yox (x,u) is continuous in (x, u). Then
(2.36)
{(50 x(x s , u* ), r) r E au w(x* ,u*)} Ç oço(x*,u*), V(x * ,e) E Rn x U.
Proof. First we assume that yo is convex. For any E Rn and u E Rk , we choose a sequence {(x, h)} c Rn x R in the following way:
(x,,u*) E lEtn x U, (x„, + u* + it,u) E 1Rn x U, oo, and Ix -xI< h. hi 10, as i
112
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
Making use of the convexity of cp, we have
lim
c,o(xi + h , u * + hi u) — (x* , u* + h i u) hi (
> urn
x
(x* ,u* + hiu), x i — x* +h
)
hi
= ( epx (x*,u*),e) .
Similarly,
lim
coo(x*, u* + hi u) — w(x*, u*) hi
> ( r, u ) .
Also, lim
yo(x* , u *) — yo(x i , u*) > lim hi
(
cpx (xi, u*), x* — xi) = O. hi
Adding up the above three inequalities, we get Hm 2-00
+ he, u* + h,u) — yo(x i , u*)
cp(x i
h,
2 (40x(x * ,e),e)+ (r,u)-
Then (yo x (x*,u*),r) E ax ,„(,o(x*,u*) by the definition of the generalized gradient. In the case that w is concave, the desired result follows immediately by noting that —c,o is convex and Lemma 2.3-(ii). Theorem 2.5. (Sufficient Conditions of Optimality) Let (D1)—(D4) hold. Let (Y0,11(.)) be an admissible pair and let p (. ) be the corresponding adjoint variable. Assume that h(.) is convex and H(t, • , • ,p(t)) is concave. Then (Y(.), U(•)) is optimal if
(2.37)
H(t,Y(t),U(t), p(t)) = max H (t, Y(t), u, p(t)), a.e. t E [0, 7 ] .
uEU
Proof. By (2.37) and Lemma 2.3-(iii), we have
0 E auH(t,Y(t),ii(t),p(t)).
(2.38)
From the assumption and Lemma 2.4, we conclude that
(2.39)
(Hx (t,Y(t),U(t),p(t)),0) E a x ,„H(t,Y(t),U(t),p(t)).
Thus, by the concavity of H(t, • , • ,p(t)), we have
T (2.40)
fO
{H (t, x(t), u(t), p(t)) — H (t, (t), U(t), p(t)) dt
< f (Hx (t,Y(t),
U(t), p(t)), x(t) — Y(t)) dt,
for any admissible pair (x(),u(•)). Define e(t) = x(t) —(t), which satisfies
(2.41)
e(t) = bs (t,Y(t),U(t)g(t) + a(t),
e(0) = 0,
a.e. t E [0, Th
3. Statement of the stochastic maximum principle
113
where a(t) = —b x (t,Y(t),-Ti(t))e(t) + b(t, x(t), u(t)) — b(t,Y(t),17i(t)). By the duality relation between (2.41) and (2.6), we have = — (P(T), 6(T)) + (P(0), 0))
= - fo
T {(
f x (t, Y(t),U(t)), 6(0 ) -I- (p(t), a(t) ) } dt
T
= f Hx (t,Y(t), ri(t), p(t)), e(t)) dt - fo ( p(t), b(t , x(t), u(t)) — (
I:1
T
T
p(t))) dt ? f(3 {Mt, x(t), u(t), p(t)) — H (t, - fo (p(t), b(t, x(t), u(t)) — b(t,Y(t), ri(t))) dt = - fo f (t, x(t), u(t)) — f (t,Y(t), ri(t))}dt. (t), 11(t),
T
T
{
On the other hand, the convexity of h yields ( hs (Y(T)), e (T) )
h(x(T)) — h(Y(T)).
Combining the above two, we arrive at J(Ti (. )) < J(u(•)). Since u (. ) is arbitrary, the desired result follows.
O
3. Statement of the Stochastic Maximum Principle We first recall the strong formulation of the stochastic optimal control problem (see Chapter 2, Section 4.1), then introduce some assumptions. Let (II, . 7 ., {Ft} t>o,P) be a given filtered probability space satisfying the usual conditions (see Chapter 1, Definition 2.6), on which an mdimensional standard Brownian motion W(t) (with W(0) = 0) is given. We consider the following stochastic controlled system:
(3.1)
fdx(t) = b(t , x(t), u(t))dt + a- (t , x(t), u(t))dW (t),
x(0) = xo,
with the cost functional T
(3.2)
J (u(•)) =Eli o f (t, x(t), u(t))dt + h(x(T))) .
t E 10 , T1,
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
114
: [0,7] X Rn X U : [0,T] x IRn x U f : [0,T] X Rn X U IR, and h : Rn -4 R. We define
In the above,
RnXin ,
(b i (t, x ,u)) b(t, x,u) = bn (t, x , u) cr(t,x,u) = (u 1 (t, x, u) , am (t, x, u)), ali (t, x, u) a-2 (t, x,u) = , 1 < j < rn. u'' (t, x, u)
(3.3)
)
Let us make the following assumptions (compare with (D1)-(D3)). (SO) {Ft}t>0 is the natural filtration generated by W(t), augmented by all the P- null sets in F. (Si) (U, d) is a separable metric space and T> O. (S2) The maps b, u, f, and h are measurable, and there exist a constant L > 0 and a modulus of continuity FA.) : [ 0, co) [0, cc) such that for cp(t,x,u) = b(t, x, u), a (t, X, u), f (t, x ,u), h(x), we have
lw(t, x u) - (p(t,i , (3.4)
t. l(p(t, 0 , u)l < L,
< Lx Vt E [0, x , E Rn , u, I1 E U, V (t, u) E [0,7] x U.
(S3) The maps b, u and h are C2 in x. Moreover, there exist a constant L > 0 and a modulus of continuity (.7) : [0, co) -) [0, co) such that for yo = b, u h, we have ,
,
(3.5)
f,
f,
140s(t,x,u) - epx (t,i, 5 Llx - +0(d(u,ii)), < rvax - I + 149xx(t, x, u) - (psx (t, Vt E , Ti , x , e Rn , U, E U.
Assumption (SO) signifies that the system noise is the only source of uncertainty in the problem, and the past information about the noise is available to the controller. This assumption will be crucial below. Now we define (recall Convention 2.9 of Chapter 1)
(3.6)
U[0, T] {u : [0, T] x 11 --) U u is {.Ft } t >o- adapted}.
Given u(.) E T], equation (3.1) is an SDE with random coefficients. From Chapter 1, Section 6.4, we see that under (S1)-(S2), for any 74.) E U[0,7], the state equation (3.1) admits a unique solution x(.) x (• ; u(•)) (in the sense of Definition 6.15 of Chapter 1) and the cost functional (3.2) is well-defined. In the case that x (. ) is the solution of (3.1) corresponding to ti(.) E U[0,7], we call (x(.), u(.)) an admissible pair, and x (. ) an admissible
3. Statement of the stochastic maximum principle
115
state process (trajectory). Our optimal control problem can be stated as follows. Problem (S). Minimize (3.2) over MO, T].
Any rt (.) E
T] satisfying J(Ti(-)) =
(3.7)
inf
u(•)Eufo,x]
J(u(•))
is called an optimal control. The corresponding Y (. ) x (. ;771()) and C±-(.), U(•)) are called an optimal state process/trajectory and optimal pair, respectively. Notice that the strong formulation is adopted here for the optimal control problem (see Chapter 2, Section 4.1 for details). However, all the discussions and results in this chapter are readily adaptable to the weak formulation. The reason is that only the necessary and sufficient conditions of optimality are concerned here; hence an optimal pair (no matter whether in the strong or weak formulation) is given as a starting point, and all the results are valid for this given optimal pair on the probability space it attaches to. Also, unlike the situation in Chapter 2, Section 4, Problem (S) here has no state constraint. Hence we do not need to distinguish the notions of feasibility and admissibility of a control. The problem with state constraint(s) will be discussed in Section 6. Our next goal is to derive a set of necessary conditions for stochastic optimal controls, similar to the maximum principle for the deterministic case. To this end, we need some preparations. 3.1. Adjoint equations
In this subsection we will introduce adjoint equations involved in a stochastic maximum principle and the associated stochastic Hamiltonian system. Recall that Sn = {A E Rnxn I A T = A} and let (x(), u()) be a given optimal pair. We have seen that in the deterministic case the adjoint variable p(S) plays a central role in the maximum principle. The adjoint equation that p(.) satisfies is a backward ordinary differential equation (meaning that the terminal value is specified). It is nevertheless equivalent to a forward equation if we reverse the time. In the stochastic case, however, one cannot simply reverse the time, as it may destroy the nonanticipativeness of the solutions. Instead, we introduce the following terminal value problem for a stochastic differential equation:
(t, Y(t),•tt(t)) T g3 (t)
dp(t) = —{b x (t,Y(t),U(t)) 1- p(t) + 3 -1
(3.8)
— fr(t,Y(t),717(0)}cit + q(t)dW(t),
p(T) =
t E [0,7],
116
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
Here the unknown is a pair of {Ft }t>o-adapted processes (p0 , q(•)). We call the above a backward stochastic differential equation (BSDE, for short). The key issue here is that the equation is to be solved backwards (since the terminal value is given); however, the solution (p(•), q(•)) is required to be {.Ft } t >0-adapted. Any pair of processes (p(.), q(•)) E L 23 .(0,T; 11171 ) x (L(0, T;1Rn)) n satisfying (3.8) is called an adapted solution of (3.8). A systematic study of such equations will be carried out in Chapter 7, from which, among other results, we will have the following: Under (S0)-(53), for any (Y(.),U(.)) E L.2F (0, T;1Rn) x /1[0, 71 (3.8) admits a unique adapted solution (p(.), q(•)). Note that (SO) cannot be omitted. The adjoint variable p(•) in the deterministic case corresponds to the so-called shadow price or the marginal value of the resource represented by the state variable in economic theory. The maximum principle is nothing but the so-called duality principle: Minimizing the total cost amounts to maximizing the total contribution of the marginal value. See Chapter 5, Section 3.2, for a detailed discussion. Nevertheless, in the stochastic situation, the controller has to balance carefully the scale of control and the degree of uncertainty if a control made is going to affect the volatility of the system (i.e., if the diffusion coefficient depends on the control variable). Therefore, the marginal value alone may not be able to fully characterize the trade-off between the cost and control gain in an uncertain environment. One has to introduce another variable to reflect the uncertainty or the risk factor in the system. This is done by introducing an additional adjoint equation as follows:
dP(t) = -{bs (t,Y(t),U(t)) 1- P(t) + P(t)bx (t,Y(t),U(t))
+ E0(t, Y(t), U(t)) T P(t)o--;(t, Y(t), U(t)) j=1 rn
(3.9)
+E
Y(t), U(t)) 1- Q
(0
Q (t)crl(t, T(t), rt(t)
i= 711
+ Hsx (t,Y(t), U(t), p(t), q(t))} dt +E Q 3 (t)C1W 3 (t) =1 P(T) = -hsx (Y(T)), where the Hamiltonian H is defined by (3.10)
H(t, u, p, q) = (p, b(t, u)) +tr [qT o-(t, u)] - f (t, , u), (t,x,u,p,q) E [0,71x1Rn xU xlftn xIR"m,
and (p(.), q(.)) is the solution to (3.8). In the above (3.9), the unknown is again a pair of processes (P(.), Q ( . )) E L(0,T; Sn) x (L3-(0, T; Sn)) m Incidentally, the Hamiltonian H(t, z, u, p, q) defined by (3.10) coincides with H (t, x, u, p) defined by (2.8) when a = O.
3. Statement of the stochastic maximum principle
117
Note that equation (3.9) is also a BSDE with matrix-valued unknowns. As with (3.8), under assumptions (S0)—(S3), there exists a unique adapted solution (P(•), Q (•)) to (3.9). We refer to (3.8) (resp. (3.9)) as the first-order (resp. second-order) adjoint equations, and to /l(.) (resp. as the first-order (resp. second-order) adjoint process. In what follows, if (Y(.),Ti(-)) is an optimal (resp. admissible) pair, and (p(.),q(-)) and (P(.), Q(•)) are adapted solutions of (3.8) and (3.9), respectively, then q(•), PO, Q (•)) is called an optimal 6-tuple (resp. admissible 6-tuple).
Po)
3.2. The maximum principle and stochastic Hamiltonian systems Now we are going to state the Pontryagin-type maximum principle for optimal stochastic controls. At first glance, it might be quite natural for one to expect that a stochastic maximum principle should maximize the Hamiltonian H defined by (3.10). Unfortunately, this is not true if the diffusion coefficient a depends on the control. Here is an example. Example 3.1. Consider the following control system (n m = 1): dx(t) = u(t)dW (t),
(3.11)
t E [0,11,
x(0) = 0,
with the control domain being U = [-1, 1 ] and the cost functional being
1 (3.12)
J (y (. )) = E
1 2
[x(t) 2 — — u(t) 2 1dt + x(1)2 } .
Substituting x(t) = f(; u(s)dW(s) into the cost functional, we obtain via a simple calculation
(3.13)
3 (- — t)u(t) 2 dt.
J=E o
2
Hence, the optimal control is i(t) 0 with the optimal state trajectory I(t) 0. However, the corresponding Hamiltonian is 1 n H (t, (t), u, p(t), q(t)) = — re' + q(t)u. 2
(3.14)
This is a convex function in u, which does not attain a maximum at rt(t) = 0 for any time t. To obtain the correct form of the stochastic maximum principle, one should add the risk adjustment, which is related to the diffusion coefficient, to the Hamiltonian to reflect the controller's risk-seeking or risk-averse attitude. To do this, let us introduce the generalized Hamiltonian G(t, x, u, p, P)
(3.15)
p
= — tr (cr(t, , u) 1 Pa(t, x , u)) + (p,b(t, x , u)) — f (t, x , u), 2 (t, x , u,p, P) E [0, 7 X ]
Rn X
U
X
Rn X Sn.
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
118
Note that G(t, x, u,p, P) depends on P, but not on q. Comparing (3.10), (3.15), and (2.8), we see that
I
G(t, x, u, p, P) = H(t, x, u) + tr [a(t,x,u)T Po(t,x,u)1, H (t, x, u, p, q) = H (t, x, u) + tr [qT cr(t,z,u)].
We need to pay attention to the difference between G(t, x, u, p, P) and H (t, x, u, p, q). The function G(t, z, u, p, P) will appear in the HJB equation of stochastic optimal control problems studied in the next chapter. Next, associated with an optimal 6-tuple (Y(•),Ti(.),p(•),q(•),P(•),Q(-)), we define an 11-function H(t, z, u) H(t,x,u,p(t),q(t))— tr kr(t,Y(t),U(t)) TP(Ocr(t,Y(t),47(t))] 1 + tr [cr(t,z, u) — o- (t,Y(t),Ti(t))] T P(t) (3.16)
[a(t, z u) — cr(t, (t), ii(t))]} ,
1 — tr [cr(t, x,u) T P(t)o(t, z, u)] + (p(t), b(t, z, u)) — f (t, z, u) 2 + tr [q(t) T cr(t, x ,u)] — tr [o- (t, z u) T P(t)a(t,Y(t), U(t))] ,
G(t,x,u,p(t),P(t))+t+(t, x,u) T [q(t)— P(t)o- (t,Y(t),Ti(t))» Notice that an fl-function may be defined similarly associated with any admissible 6-tuple (x(•), u(•), p(•), q(•), P(.), Q( . )). Theorem 3.2. (Stochastic Maximum Principle) Let (S0)—(S3) hold.
Let (Y(-), U(-)) be an optimal pair of Problem (S). Then there are pairs of pro-
cesses (3.17)
(p(.),q0) E L y 2 (0 , T;
) x (L 2,(0,T;Rn)) n ,
(P(.), Q (-)) E L .2F(0, T; Sn) x (L 2y -(0,T; S n )) 7n
where (3.18)
{ 90 =
(C . ), • • • gm ( ))
Q(') = (Q (') • • • C2.0), qh) E L F 2 (0, T; R n ), Q () E L 2F (0, T; Sn ), 1 <j < m, •
satisfying the first-order and second-order adjoint equations (3.8) and (3.9), respectively, such that H (t, Y(t),U(t),p(t), q(t)) — H (t, -±(t), u, p(t), q(t)) tr (fa (t, (t), ri(t)) — cr(t, (t), u)1 T P(t) (3.19) • fcr(t,Y(t),U(t)) — cr(t, (t), u)}) > 0, Vu E U,
a.e. t E [0, 7 ] ,
P-a.s. ,
119
3. Statement of the stochastic maximum principle
or, equivalently, (3.20)
li(t, -±(t), ft(t)) = max "H(t, Y(t), u), a.e. t E [0, 7 ] , uEU
P-a.s.
The inequality (3.19) is called the variational inequality, and (3.20) is called the maximum condition. Note that the third term on the left-hand side of (3.19) reflects the risk adjustment, which must be present when a depends on u. Now let us look at Example 3.1 again. The first-order and second-order adjoint equations associated with the optimal pair ((t), ri(t)) (0,0) are
{ dp(t) = q(t)dW (t), p(1) = 0,
(3.21) and
I
(3.22)
d P (t) = 2dt + Q(t)dW (t),
P(1) = —2.
Thus, (p(t), q(t)) = (0,0) and (P (t), Q(t)) = (2t — 4,0) are the (unique) adapted solutions. The left-hand side of (3.19) now reads
1 1 1 — 771(0 2 + q(t)U(t) — — u2 — q(t)u — — N(t) — u1 2 P (t) 2 2 2 =
A u2 ? 0,
Vt E [0, 1] .
We may also calculate
(3.23)
7-l(t,Y(t), u) = 12- (P (t) + 1)u2 + q(t)u = (2t — 3)u2 .
The above function 'H is concave in u for any t E [0,1], and ri(t) maximize H. Note that for the present case (see (3.14)),
H (t,Y(t), u, p(t), q(t)) =
0 does
u2 .
Thus, one sees clearly how the second-order adjoint process (which represents the risk factor) plays a role in turning the convex function u 1-4 H (t,Y(t), u, p(t), q(t)) into the concave one u i—■ 7-i(t , Y (t), u) shown above. Let us single out two important special cases.
Case 1. The diffusion does not contain the control variable, i.e., (3.24)
o- (t, x, u) _= '7(t, x),
V(t, x, u) E [0, 7 ] x Rn x U.
In this case, the maximum condition (3.20) reduces to
H (t, -±(t), ii(t), p(t), q(t)) = ri nEar)jc 11(1, Y (t), u, p(t), q(t)), (3.25)
a.e. t E [0, T],
P-a.s. ,
120
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
which is parallel to the deterministic case (no risk adjustment is required). We note that in this case, equation (3.9) for (P(.), Q(.)) is not needed. Thus, the twice differentiability of the functions b, c • , f, and h in x is not necessary here. Case 2. The control domain U C R k is convex and all the coefficients are Cl in u. Then (3.19) implies
UM) < 0, Vu E U, a.e. t E [0, 71, P-a.s.
( Hu (t,Y (0,2740, p(t), q(t)), u
(3.26)
-
This is called a local form (in contrast to the global form (3.19) or (3.20)) of the maximum principle. Note that the local form does not involve the second-order adjoint process P(.) either. Analogous to the deterministic case, the system (3.1) along with its first-order adjoint system can be written as follows:
dx(t) = Hp (t, x(t), u(t), p(t), q(t))dt
(3 . 27)
+ Hq (t, x(t), u(t), p(t), q(t))dW (t), dp(t) = - Hx (t, x(t), u(t), p(t), q(t))dt + q(t)dW (t), t E [0, 71, p(T) = - hx(x(T)). x(0) = xo,
The combination of (3.27), (3.9), and (3.19) (or (3.20)) is called an (extended) stochastic Hamiltonian system, with its solution being a 6 tuple (x(.), u(.),p(),q(•),P(.),Q(.)). Therefore, we can rephrase Theorem 3.2 as the following. -
Let (S0)-(S3) hold. Let Problem (S) admit an optimal Then the optimal 6-tuple (Y(.),U0,p(.),q(.), 1 3 (.),Q(.)) of Problem (S) solves the stochastic Hamiltonian system (3.27), (3.9), and
Theorem 3.3. pair (Y0,2710).
(3.19) (or (3.20)). It is seen from the above result that optimal control theory can be used to solve stochastic Hamiltonian systems. System (3.27) (with u(-) given) is also called a forward backward stochastic differential equation (FBSDE, for short). In Chapter 7 we will study more about this type of equation. -
3.3. A worked-out example Let us now give a more substantial example to demonstrate how Theorem 3.2 can help in finding an optimal control. Example 3.4. Consider the following control system (n = m = 1):
(3.28)
{ dx(t) = u(t)dt + u(t)dW(t), x(0) = 1,
t E [0, 11,
with the control domain being U = R and the cost functional being
1 1 (3.29)
J(u(•)) = El - f -ru(t) 2 dt + o 2
1
3. Statement of the stochastic maximum principle
121
where r E (0, 1) satisfies (3.30)
ln r + 2 — r <0
(or 0
Suppose (Y (. ), Ti(. )) is an optimal pair (which we are going to identify). Then the corresponding adjoint equations are
dp(t) = q(t)dW (t),
(3.31)
t E [0, I],
p(1) =
and
dP(t) = Q(t)dW (t), P(1) = —1.
(3.32)
t E [0, 1],
f{
Clearly, (P(t), Q(t)) = (-1, 0) is the only adapted solution to (3.32). The corresponding fl-function is (3.33)
1
7-t(t , x, u) = — — (1 — r)u 2 + (p(t) + q(t) + 17(t))u. 2
Note that the above is a concave function of u due to the fact that r < 1. So by Theorem 3.2, a necessary condition for U(.) to be optimal is —
(1
—
r)U(t) + p(t) + q(t) + rt(t) = 0,
Or
p(t) + q(t) r
(3.34)
By plugging (3.34) into (3.28) and putting it together with (3.31), we obtain the corresponding stochastic Hamiltonian system:
I (3.35)
d(t) = p(t) + q(t) dt r dp(t) = q(t)dW (t), Y(0) = 1, p(1) =
p(t) + q(t) dW (t), r
Once we find an adapted solution (Y(•), p(.), q( • )) of the above, a candidate for the optimal control can then be given by (3.34). We now try to find an adapted solution of (3.35). Suppose ((.), p(.), q()) is an adapted solution such that the following relation holds: (3.36)
p(t) = e(t)Y(t),
Vt E [0, lb P-a.s. ,
for some deterministic function 0(•). We would like to determine the equation that 190 should satisfy. To this end, we differentiate (3.36) using Itô's formula:
dp(t) = [é(t)Y (t) — 0(0 134) ± q(t)]dt e(t)P(t) ± r r
122
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
Comparing this with (3.31), we obtain
{ 0(t)Y(t) = (3.37)
0(t)(p(t) + q(t))
r
,
0(t)(p(t) + q(t))
q(t) =
r
'
Consequently,
{q( t) =
02 (t)Y(t)
p(t)0(t)
r + 0(t) r + e(t) ' 0(t)Y(t) p(t) + q(t) _ r r + em•
(3.38)
Then, combining the first relation in (3.37) and the second in (3.38), we see that if 80 is the right choice, it should satisfy the following:
t e [0, 1],
OM =
(3(t)2 r + e(t)' 0(1) = —1.
(3.39)
Now, suppose for the time being that (3.39) admits a solution 8(.) on [0, 1]. Then it is necessary that (noting that r + 8(1) <0)
(3.40)
r + 8(t) <0,
Vt E [0, 1].
In this case, we have a candidate for the optimal control:
(3.41)
U(t) =
0(t)
r + 09(t)
Y(t),
t E [0,1], P-a.s. ,
which is in state feedback form. The corresponding state process is exactly Y(t), which is the solution to the first equation in (3.35), or to the following:
(3.42)
8(t)
{ c(t) =
r + OW
Y(t)dt
8(t) r
Y(0) = 1. Let us now show that (Y (. ), ti(.)) obtained is indeed an optimal pair. In fact, a simple calculation via Itô's formula yields that for any admissible pair (x(.), u(.)),
d(0(t)x(0 2 ) = {0(t)u(t) 2 +20(t)u(t)x(t) +
0(t)2x(t)2 r + OM
lelt
+ {. • lell V (t). Integrating from 0 to 1, taking expectations, and noting (3.39) and (3.35), it follows that 1
Ex(1) 2 = —0(0) — E] {0(t)u(t) 2 +20(t)u(t)x(t) + e(t)2x(t)2 }cit. o
r±
(No
123
4. A proof of the maximum principle
Substituting this into the right-hand side of (3.29) yields
(3.43)
J(u(.)) _= --1 Ef(r + e(t))(u(t) + 2
re±(t)(xx(to)) 2dt 12 e(0).
0
Therefore, by (3.40), the minimum value of the cost functional will be , which is exactly given by (3.41). achieved at U(t) = We now show that equation (3.39) does admit a solution 0(.) with the property (3.40). To this end, first note that (3.39) is equivalent to
r ln(-0(t)) — e(t) = t - 1+ r.
(3.44)
Define f t(A) = ln A + — t + 1 — r, A E (0, ±oo). Noting (3.30), for all t E [0,1), it follows that
f t (r) = ln r + 2 — t — r < 0, ft (1) = 1 — t > 0,
(3.45)
we conclude that there is —0(t) E (r, 1) (i.e., r + e(t) < 0) such that ft(-8(t)) = 0 for t E [0,1). Moreover,
(3.46)
d ft f t((A) À) dA
A —r Az
<0
if A < r,
>0,
if À> r,
=0,
if A = r.
Thus the 0(t) satisfying ft(—e(t)) = o and r +OM <0 is unique. Finally, f 1 (1) = 0 implies 0(1) = —1. This establishes the existence of solutions to (3.39) subject to (3.40). To conclude, an optimal feedback control is given by (3.41), and the optimal value is —0(0) (see (3.43) with e(.) solved via (3.39)).
4. A Proof of the Maximum Principle In this section we are going to give a proof of the stochastic maximum principle, Theorem 3.2. Our main idea is still the variational technique. However, due to the presence of the diffusion coefficient, which may contain the control variable, the method that works for deterministic case has to be substantially modified to fit the stochastic case. Let us look at the following simple example, from which we can see some significant differences between the stochastic and deterministic cases. Example 4.1. Consider the following control system (n = m = 1):
dx(t) = u(t)dW(t),
(4.1)
t E [0,7 ] ,
x(0) = 0, {
with the control domain being U = {0, 1} and the cost functional being
(4.2)
J(u()) = Ex(T)2.
124
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
Then the optimal pair is clearly given by (Y(t),u(t)) a- (0, 0). Now let us try to make a perturbation similar to what we have done for the deterministic case. Let Ee = [s,s + C [0,71 and u(t) 1. Set
(4.3)
if t e [0, T] \
ue(t) = rt(t)
if t E .EE .
u(t) a- 1,
Let xe(-) be the solution of
e{dx = ue(t)dW(t), e(0) = O.
(4.4)
t E [0, T],
Then, by ha's formula, for t > s + e,
(4.5)
El xe (t) — Y(t)I 2 =
Ele(t)I 2 =
jo t
(r) 1 2 dr -=
This means that the analogue of (2.12) no longer holds for the stochastic case. The basic reason is, roughly speaking, that the stochastic integral ftt+e crc/W is only of order IA., so the usual first-order Taylor expansion is inadequate. The resolution to this is to consider the second-order Taylor expansion.
4.1. A moment estimate In this subsection we prove an elementary lemma, which will be useful in the sequel. Lemma 4.2. Let Y(t) E 4(0, T; Rn ) be the solution of the following:
(4.6)
IdY (t) = {A(t)Y(t) + cx(t)Idt + E{B(t)Y(t) + 03 (01 d1473 (t), j=1 Y(0) = Yo,
where A, Bi : [0,7 ] x fl —■ illn x n and a, f3 : [0,T] x St adapted, and
(4.7)
{
IA(t)191B 3 WI < L, T
Rn are {.Ft}t>o-
a.e. t E [0, 71, P-a.s. , 1 < j
T
fo {Ela(s)1 2k }à ds+ f
o
IE1 /3i( 02k li d s < es,
1 < j < T 11,
for some k > 1. Then sup ElY(t)I 2k <1(1E11'0 1 2k +
(4.8)
tE[0,7-1
lIZ
j_1 (
{Elcr(s)1 2k }nds
)2k
fT
1433 (S)I 2k dS) k .
125
4. A proof of the maximum principle
Proof. For notational simplicity, we prove only the case m = 1 (i.e., the Brownian motion W(t) is one-dimensional), leaving the case m> 1 to the interested reader. Thus, the index j in B 3 (.) and 03 0 will be dropped. We first assume that a() and fi(.) are bounded. Let E > 0 and define (4.0)
y ) E a Viyi2
e2
VY E IRn .
( Y ), is smooth and ( Y ) e IYI as Note that for any e > 0, the map Y E ---0 0. The purpose of using such a function is to avoid some difficulties that might be encountered in differentiating functions like IYI 2k for noninteger k. Applying Itô's formula to ( Y(t) ) e , we have
E (Y (t)). k < E ( Y(0) + 2kE (4.10)
(Y (s)) 2k-1
(Y(s)), + la(s)I}ds
+ k(2k - 1)E f (Y(se -2 {1B(s)i (Y(s))e+10(s)1} 2 ds <E (Y (0))r + KoE f {(Y (s)) k +ia(s)i<1 (s)) k-1 +10(8)1 2 ( 17 (s)) 2k-2 }ds.
Here K0 = Ko(k,L) is independent of t. Applying Young's inequality, we obtain
E (Y (t) ) 2k <E ( Y(0) (4.11)
+ K E f { Y (s) o " (
)
I(s)12k + 1 0(02k} cis.
Hence, it follows from Gronwall's inequality that
(4.12)
E (Y (t))r < K {E (Y (0) )r
E f [i ck (s )12k +10(8)121 6 } t E [0, 7 ].
Here K = K(L,k,T). Note that since we assume for the time being that a(-) and 00 are bounded, the above procedure goes through (otherwise the integration on the right-hand side of (4.12) may not exist; see (4.7)). Next, we want to refine the above estimate so that (4.8) will follow. To this end, note that (4.12) implies that its left-hand side is bounded uniformly in t E [0,71. Thus, we may set (4.13)
(p(t) = { sup E (Y( s))1 112k o<s
t E [0,7 ].
We now return to (4.10), using (4.13). Define 6 = (41‘0) -1 . Then, for any
▪
126
▪
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
t E [0, 6], applying Holder's inequality and Young's inequality, we obtain cp ( t )2k < go ( 0 )2k
Ko
l w (02k t ± 99(0 2k-1 fo ( E i a(s)12k)-* ds (p(t) 2 k 2
(El/3(.5)1 211 1 dg}
0
(4.14) < (p (0) 2 k
cis] 2k
(p (t)2k
[ fo (EICI(S)12k) -11
[fo (E10(02/id si k } The constant we obtain
If1 = Ki(k, L, 6)
tp(t)2 k
in
(4.14)
is independent of t. From
1,0(0) 2k + 2K1 [ f (Ela(S)1 2k )Cid 2k 0
(4.15)
(El/3(02/i (id k
[f W(T) 2k 5_
K{(10(0) 2k
(4.16)
Vt E [0, 6]. 0
Now we can do the same thing on [6,26] and on we end up with
+
[26, 36],
and so on. Finally,
(Ela(s)12 k)* els] 2k
▪ [ f0 (E10(02k)k cis] 1 ,
with the constant K = K(L, k,T, 6). By conclude that
te
(4.13),
the definition of ço(t), we
K{ ( Y(0) )!/` + [fo (Ela(S)12
sup E (Y
(4.17)
(4.14),
[UT)
i
2k
(EI/3(02k)k cisi k
[
0 Letting e 0, we obtain (4.8). Finally, in the case that we only have (4.7) (instead of a and 0 being bounded), we can use the usual approximation.
4.2. Taylor expansions The following elementary lemma will be used below, whose proof is straightforward and is left to the reader. Lemma 4.3. Let g G C2 (IFtn ). Then, for any x,
g(x) = g(Y) + (gx (Y), x —
(4.18)
+f
E
4. A
127
proof of the maximum principle
Now, let (Y (. ), u(•)) be the given optimal pair. Then the following is satisfied:
(4.19)
fdY(t) = b(t, Y(t),c(t))dt 17 + cr(t,T(t),ii(t))dW (t), "±-(0) = xo.
Fix any u(.) E U[0, T] and
E>
ue (t) =
(4.20)
0. Define
lq(t), u(t),
t E [0,71 \Ee , t E Ee ,
where E, C [0, 7] is a measurable set with 1E, I = satisfy the following:
(4.21)
t E [0, 71,
E.
Let (xe
ue ))
{de (t) = b(t,xe (t),uE (t))dt + cr(t,xe (t),u€ (t))dW (t), t E [0, T], xe(0) = xo.
Next, for cp. = bz , o-z3 , f (1 <j < n, 1 <j <m), we define
ço(t) a ços(t, y(0,770), ,o(t)
a cpss(t, y(t),U(t)),
a „o(t, Y (t), u(t)) — cp(t,Y(t),Ft(t)), by; x (t) a ,x(t,y(t ), u(o) _ wx (t,y(t), U(t)), , um). bcpsx (t) a cpzs(t,y(t), u(o) _ (prx (t, y(t) 4(0
(4.22)
{
Let ye(.) and .ze(•) be respectively the solution of the following stochastic differential equations:
{dye (t) = b x (t)ye (t)dt + E faicoye(t) ± 6.0-3(o xEE (oldw3(t), j=1
(4.23)
t E [0, 7 ], ye(0) = 0, and
de (t) = {b(t)t(t) + bb(t)xE(t) + brx (t)ye (t) 2 1 dt
E 10-;(t)e(t)
(4.24)
bc4(t)y e (t)xE(t)
j=1
1 + — cr (t)ye (t) 2 } dW (t), t E [0,71, 2
ze (0)
= 0,
128
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
where
tr {61 1 (t)ye. (t)ye (1) T } bx1 (t)ye(t) 2 A tr {bTrix (t)yE (t)yE (t) T } (4.25) tr
I
0L(t)ye(t)2
(t)ye (t)V (t) T }
1 <j < m. tr faTs11(t)ye(t)yE(t) T I
The following result gives the Taylor expansion of the state with respect to the control perturbation. Theorem 4.4. Let (S1) (S3) hold. Then, for any k > 1, —
sup Eix e (t) — Y(t)I 2k =
(4.26)
OW),
tE[O,Tj
sup Elye(02k = o w) ,
(4.27)
tep,T1
sup Elze(02k _ 0 (62k) ,
(4.28)
tE[0,71
(4.29)
sup Eix e (t) — Y(t) _ y e (0 2k _ 0(e 2k) , tE[0,7]
(4.30)
sup Eixe(t) _ ±-( t) _ ye( t) _ zE(012k = c (e 2k) . t€[0,7]
Moreover, the following expansion holds for the cost functional: J(ue (.)) = J(()) E (hx (Y(T)), y€ (T) + (T)) 1 + — E ( hxx (Y(T))y€ (T), (T)) 2 (4.31)
E]
fx (t), yE (t) + zE (t))+ ( fsz (t)ye (t), ye (t)) + 6f (t)XE,
(01 dt + o(e).
The proof of the above theorem is rather lengthy and technical. The reader may skip it at first reading and proceed to the next subsection directly.
Proof. For simplicity of presentation, we carry out the proof only for the case n = m = 1 (thus, the indices i and j will be omitted below). The interested reader is encouraged to give a proof for the general case.
4. A proof of the maximum principle
129
I. Proof of (4.26) and (4.27). Let V(t)--.--xe(t) —(t). Then we have
cke (t) = ft;z Mee (t) + 6b(t)xE (t)}dt (4.32)
+ f(t)e E (t) + bcf(t)x „(t)}dw(t), t E [0, 71, E(Ø) = 0,
where {1 kx (t)gZ bs (t,Y(t)+19(xe(t) —Y(t)),ItE(t))de, (4.33) ur (t, -±(t)+ O(xe(t)— Y(t)), u e (t))de.
f By Lemma 4.2, we obtain
sup Er(t)I 2k Iff(f {El8b(s)xE(8)1 2k 1 A d.9) 2k te [0,7]
(4.34)
{E16CI(S)XE,(9)1 2k l i dS) k
< K { e2k
0
ek
< Ke k .
This proves (4.26). Similarly, we can prove (4.27).
2. Proof of (4.28).
Form (4.24), Lemma 4.2 and (4.27), we have sup Elze(t)1 2k tE[0,7]
<
KT( f El6b(S) X E (S)
bxx(S)Y e (8) 2 1 2k
dS) 2k
0
T {E160',(S)XE,(8)Y e ( 8 )+ Crxx(8 )Y E (S) 2 1 2k 6) k }
(
(4.35)
T
<
fo
IXE,(S) (ElY e
(S)1 4k ) A 1613) 2k
(El y e (s) 12k)i ( E i y, (s)1 4k ) i lds)k 0 -r (f14
)
KE 2k
This gives (4.28).
3. Proof of (4.29). Set
(4.36)
(t) g x€(t) — Y(t) — V(t)
ee (t) — ye (t).
130
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
we have
By (4.32) and (4.23),
(t) — bx (t)yc (t)}dt
(t) = kx (t)66 (t) + 6b(t)xE + { -61(t)ee (t) —
x (t)ye (01 dW (t)
(4.37) = {k(t)if (t) + 6b(t)xE,(t) + g (t) _ bx (t)ly' (01 dt + {-5-1 (t)iie (t) + [F4(t) — o-x (t)]ye (01 dW (t).
Thus, it follows from Lemma 4.2 that E lne(t) 12 k <
(
1.08b(s)X&(s) + rE6x (s) — b x (s)]y' (s)1 2k
0
LV
T fEl[(S) — Cf x (S)iy 6 (8)1 2k
▪
(4.38)
}
ds) 2k
CIS) !
K {(e + f0 {Ely' (s)1 4 }{Elkx (s) — b x (s)I 4k }* ds) 2k
T
fErif(S) — crx (s)1 411 -kls)
{ElY e (S)1 4k
▪
k
) 2k < K {E 2k Ek f {E(s) — bx (s)1 4k }' ds
▪ Ek f
T — 0-x (Sel l d8) k .
Note that (by (S3) and (4.26))
IT
{E(s)_b(s)I'}
= f L'
ds
T
(4.39)
—
K
[brc s,y (s)
oc xe (s) _ y( s)) ,ue ( s))
bx (s,Y(s),U(s))1d0 1
{EILIX E 0
< KIE Z
4k iE
ds
— Y(S)I+ 6b s (S)XE E (S)1 4k
(EIZ(s) —Y(.9)1 4k }ds}
Similarly, we have
(4.40)
ZT {E(s) — Cr x (S)1 4
Then (4.29) follows from (4.38).
0
d.9 5_ Ke.
dS
131
4. A proof of the maximum principle
4.
Proof of (4.30).
Set
ce (0 xe(t) —
(4.41)
(t)
—
(t)
—
CE (t) — ye(t) — .e(t)
(t) (t)
—
ze (t).
It is clear that ck e (t)
(4.42)
B(t)dt + E(t)d W(t),
CE (0) = 0 ,
where (noting (4.19) (4.24) and Lemma 4.3) —
B(t) = b(t, xE (t), (t)) — b(t,Y (t), 1 — — toss (t) ye (t) 2 2
(t)) — b s (t)[yE (t) + ze (0]
(Me (t) + "-Elesx Me (02
= bs (t, (t),
— b s (t)[y' (t) + zE (t)] — b(t)y(t)2
(4.43)
+ 15 bs (OX E, (t)e e (t)
= bs (t)( E 1 rr
+
1.0exx(t)
bss (t
,u6 (t))]ee (0 2
11 + — ass (t)x& (t) 2 + bxx(t)[e (0 2 Y e (0 2 ] 2 bs MC' (t) + cte (t), — 2
and
E(t) = a (t , xE (t), ue (t)) — a (t, -±" (t), tie (t)) — crs (t)[ye (t) + t(t)] 1 — — ass (t)V (t) 2 — Scrs(t)xE,. 2
(t)
= a (t (t),ue (t))ee (t) + -"Fi x (t)ee (t) 2 — as (t)[ye (t) + t(t)] 1 —
(4.44)
—
ass (t)y E (t) 2
—
bcrx(t)xE,(t)V(t)
2
= as WC' (t) + bcr s(t)X& (Or( (t)
+ [al.(t) — us.(t, Y(t), /LEM)] ee (0 2 + —2 barr(t)XE,(t)C E (t) 2 + —2 crxx(t)[C e (t) 2 — Y e (t) 2] —=
(t)Ce (t) + 13e (t),
with 1 { -6e (t) rx
g21 o
(4.45) Filx (t) gt
Obss (t, OY (t) + (1— 0)xe (t), uE (t))d0 , 1
2 f Ocrsx (t, BY (t) + (1 — 0)xe (t), le (t))d0 , o
132
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
and
cte (t)
(4.46)
8 6bs (t)xE Mee (t) + ['.1-)exx (t) — b sx (t,Y(t), UE (t))1 Ce (0 2 1 1 ± —2 6bx.(t)xE, (t)ee (t) 2 + —2 bri(t)[C' (t) 2 —
OE (t) 8 6c rs (t)x E (Or( (t) + rri;s (t) — a Mt, Y(t), ue (0)] e' (02 1 1 + y§crxx(t)xE(t)e e (t) 2 + -.axx(t)[C E (0 2 —
In order to use Lemma 4.2, we need to estimate cxE(.) and O'(.). To this end, recall that re appearing in (S3) is a modulus of continuity for bsx (t, • , u) (uniform in t E [0,7] and u E U). Thus for any p > 0, there exists a constant > 0 such that
K,,
OM < p+rKp ,
(4.47)
Vr > 0.
Consequently, lks (t) — b sx (t,Y(t),u€
(4.48)
(MI
< (7,(1Ce MI) < p + Kp ie' MI.
Recalling (4.36) and (4.46), as well as (4.26)—(4.29), we can estimate cte(•) as follows:
(Ela€(t)121')ndt foT T
J o
„
1
i(El a x(t)XE, We. (0 12k + (E 1
±(El (4.49)
) n
L gx (t) — bsx (t,Y(t),ue (0)] ee (t)21 2k) b.1)xx(t)xE ‘ (0 ee (t) 2 1 2 k ) 11T
+ (El . b(t) [ee (t) 2 — ye (t)11 21c ) 1.1 leit
< K I:
Ix
(t)(Ele e (t)1 2k ) à + XE (t)(Epee(t)1 4k ) à
+ (E[P + Kpie Mi ] l * ( Eiee (0 8k ) à + (E lne(t)14k) (Elee (t) + ye(t)14k) 1 dt < K{0/ 2 + e(p + liKp ) + 62 + 63/2 } . This implies T
(4.50)
jo
(Eke (t)I 2k ) dt = o(e).
4. A proof of the maximum principle
133
Similar to (4.48), we have (4.51) As
IFT".(t) - 0-„(t,t--(t),,e(t))1 5 o(icE(t)l) 5 P + Kp lee (t)1.
with (4.49), we can estimate /TO as follows. T 0p
(Elfi e (t)I 2k ) 1 dt \
T
1(.0(5 x(t)x E (One (t) rk)
f0
[-rlx(t) axx(t, (t) , ue (t))]CE (t)2
±
(E 1 6'
rz
(4.52) 2 (7'
±
(t)xE
(t)
meE( 021 2k
[eE(t)2
2k )
)
ye (0112k) }.it 1
Kf
{xE c (t) (Ellie (01 21) 1 + X E (t) (Er (01 4k ) r
+ (E[p + K + (Elif (t)I 4k )
(t)I] 4k )
(Elee (t)1 8k )
(Elee (t) + ye(t)1 4k )
dt
< Thus,
f
o
T
(El/T(01 21) 1 dt =
by Lemma 4.2, we obtain (4.30). 5. Proof of (4.31). By Lemma 4.3, we have
Then,
J (ue (.)) — J((.)) = E fh(xe (T)) — h( -±-(T))} + E f {f (t, x€ (t), ue (t)) — f (t,T(t), U(t))}cit = E ( hx (Y(T)), Cc (T)) +Ef (Oh xx (OY (T) + (1— 9)xe (T))C' (T), Ce (T)) clO
+Ef
0
+f ( 0
{6 f (t)XE(t) + f (t, Y(t),
(t)) e e ( t ) )
(t, O(t)+(1 — 0)xe (t), Ue (t)) e e (t) C E (t) ) de} dt
134
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
Now, recalling the definitions of
if, and (e, we have
J (ue (•)) — (Tie)) = E ( hx ((T)), yE (T) + ze (T)) ( hxx (Y(T))ye (T), ye (T))
+ E ( Its ((T)), (e (T)) +
1 —
E ( hxx (Y(T))if (T), ee (T) + ye (T))
2
+ E f (0 thxx (ert(T) + (1
—
0)x' (T)) (T)) do
— +E
+f
f (t)xE,(t) + ( 1 s (t) e e (t)) X E (t) f + ( fx (t), (t) + (t)) + ( fx (t), ( 6 (t)) 1 ( [fxx (t, (t) + (1— 0)x 5 (t), ue (t)) — f xx (t,Y(t),
+
1 —
(t))] (t), (t)) dB
( fx.(t)V(t), (t)) +
2
1 —
( f(t)if (t), 6e (t) + ye (t) )1 dt.
2
Then, by (4.26) (4.30), we can show that —
J (uE (.)) — J((.)) = E (hx (Y(T)), ye (T) + ze (T))
E (hxx (Y(T))ye (T), ye (T))
(4.54) +E
f(h(t), (t)+ (t))
(f xx (t)ye (t), ye (t))
+ (t)x E, (t)} dt + R(e), where R(e) is of order o(e). Hence, our conclusion follows. 4.3. Duality analysis and completion of the proof From Theorem 4.4, we conclude that a necessary condition for a given optimal pair (Y(.), U(.)) is the following:
0 (4.55)
E f(h x (Y(T)), (T) + ze (T)) +E
(hx ,(Y (T))y' (T), ye (T ))}
{( f (t), ye (t) + ze (t)) + 6 f (t)xE,(t)} cit + o(e),
( fxx (t)ye (t), ye (t))
Vu ) E U[0, TI,VE > 0, (.
135
4. A proof of the maximum principle
where ye() and zE(.) are solutions to the (approximate) variational systems (4.23) and (4.24), respectively. As in the deterministic case, we are now in a position to get rid of ye(.) and zE(•), and then pass to the limit. To this end, we need some duality relations between the variational systems (4.23)—(4.24) and the adjoint equations (3.8) and (3.9).
Lemma 4.5. Let (S0)—(S3) hold. Let V(.) and zE(•) be the solutions of (4.23) and (4.24), respectively. Let (p(.), q(.)) be the adapted solution of (3.8). Then
E (p(T), V(T)) (4.56)
=Efx f (t) Y E (t) ) +tr [q(t) T 6o- (0]xE(Oldt,
and
E ((T), ze (T)) = E] { ( f x (0, z 6 (0 ) I1.
+
(4.57)
E
1 [ ((t), bsx (t)yE (0 2 ) + 2 3=1
—
(t), o--;x (t)y e (0 2 )1
n
+ [ p(t), b(t) ) +
E( q3 , o-.; (t)ye (0)] x (01 dt. 3 =1
The proof follows immediately from It6's formula, and we leave it to the reader (see also Chapter 1, Corollary 5.6). Adding (4.56) and (4.57), and appealing to the Taylor expansions given in Theorem 4.4, we get
— E ( h x (Y(T)), (T) + ze (T)) pT
=E (4.58)
{ ( f i (t),ye + ze(0) (3
+
1 — 2
n
( p(t) , bx .(t)ye (t) 2 ) +
E
1 ( qi (t), riL(t)V (0 2 ) 2 3=1
—
+ [ ( p(t), 6b(t)) +tr (q(t) T 6c,- (t))1xE,(Oldt + o(c).
Thus, by (4.31) and the optimality of rt(.), we have
136
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
0
J(T1(•)) — J (u' (-))
1
+
( hxs (Y(T))ye (T), ye (T))
1
T
2
{ — (f xr
My' (t), ye (t)) + (p(t),bxx (t)yE (0 2 )
+ E ( q3 (t), a-L(0V (0 2 ) } dt 3=1
(4.59) —f
E f
(t) + ( p(t), 8(t)) 711
+ E (q,(0,60•3 (t)) } X E, (t)dt + o(E) 3=1
1 = — Etr {P(T)Ye (T)} 2 T
+Efo
1
[Hzz(t)Y E (t)] + 61-1(t)XE,(t)}dt + o(E),
where
/ Ye(t) = YE(t) ye(t)T, (4.60)
Hx x (t) = Hx s (t , Y (t), 771(t), p(t), q(t)), 6 H (t) = H (t , Y(t), u(t), p(t), q(t)) — H (t,Y (t), U(t), p(t), q(t)) .
We see that (4.59) no longer contains the first-order terms in y'(.) and ze(.). But, unlike the deterministic case, there are left some second-order terms in ye(.), which are written in terms of the first-order in Ye(.). Hence, we want further to get rid of YE(.). To this end, we need some duality relation between the equation satisfied by YE() and the second-order adjoint equation (3.9) (which is exactly where the second-order adjoint equation comes in). Let us now derive the SDE satisfied by YE(.). Applying It6's formula to yE(t)V(t) T and noting (4.23), one has dYe(t) = { bx (t)Ye(t)+ r(t)bx (t) T rn
+ Ea-;(t)Ye (t)al(t) T E 6a-3 (t)(5 (t)T x E j=1
(4.61)
j=1
+ E (c4(t)v(t)60-3 (t)T + 6o-2 (t)ye (t)Tc -;
XE,
3 =1 TI1
+ E (c4(t)If (t) + (t)o-(t) T ) dW (t) 3 =1 M.
+ E (60-3(t)ye 3=1
± ye (06 (t)1xE,(t)d14 (t).
dt
5. Sufficient conditions of optimality
137
To establish the duality relation between (4.61) and (3.9), we need the following lemma, whose proof follows directly from Itô's formula (which is also parallel to Chapter 1, Corollary 5.6). Lemma 4.6. Let Y P(.) E L2F (0, T; IR" n ) satisfy the following: 171
{ dY(t) = 4)(t)dt + ET,(t)dw3(t), 3 =1
(4.62)
M.
dP(t) = 0(t)dt + E Q (t)dlin (t), 3 =1
with (D(.), ill (•) , Then
(.), and C22 (.) all being elements in L2, (0 , T; Rnxn ).
E{ tr [P(t)Y(t)] — tr [P(0)Y(0)] 1 (4.63)
=El { tr [6(t)Y(t) + P(t)43(t) +
C23 (t)T 3 (t)] 1dt. 3 =1
Now we apply the above lemma to (4.61) and (3.9) to get the following (using Theorem 4.4, and noting tr [AB] = tr [BA] and Y(0) = 0):
E ftr [P(T)Ye (T)]} (4.64)
= Ef tr [6o- (t) T P(t)bo- (t)xE (t) — Hxs (t)YE (O]dt + o(e), where {
H.. (t) 6a(t) a o- (t,Y(t),u(t)) — c(t,Y(t),Tii(t)).
Hence, (4.59) can be written as
(4.65)
o(e)
E
1 1611(t) + tr [6o-(t) T P(t)450- (t)]}x&(t)dt. —
2
Then we can easily obtain the variational inequality (3.19). Easy manipulation shows that (3.19) is equivalent to (3.20). This completes the proof of Theorem 3.2. We point out that Theorem 3.2 remains true if f and h are C2 in z and allowed to have polynomial (in particular, quadratic) growth in z, provided that b and cr have linear growth. The reader is encouraged to complete a proof for this case.
5. Sufficient Conditions of Optimality In this section we will show that for the general controlled stochastic systems formulated earlier, the maximum condition in terms of the fl-function
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
138
(see (3.20)) plus some convexity conditions constitute sufficient conditions for optimality. Let us first introduce an additional assumption. (S4) The control domain U is a convex body in le. The maps 6, a , and f are locally Lipschitz in u, and their derivatives in x are continuous in (x, u). The following lemma will play an interesting role below. Lemma 5.1. Let (S0)-(S4) hold. Let (Y(•),U(.), q(-), P (. ), Q(•)) be a given admissible 6-tuple and let H be the corresponding fl-function. Then for any t E [0,7 and w E ]
(5.1)
Mgt, (t), U(t), p(t), q(t)) = Ou h(t , (t) , U(t)).
Proof. Fix a t E [0,7 and cai E Q. Define ]
{H(u) a H(t, T(t), u, p(t), q(t)), 71(u) ±±-1-- 7i(t,Y(t), u),
a (u) a cr(t,Y(t), u), 0(u) a -1 tr [a(u)T P(t)a(u)] - tr [a(u)T P(t)a(ii(t))].
2
Then, by the second equality of (3.16), we have 71(u) = H(u) + 11)(u). Note that for any r i 0, u,v E U, with u --■
zi)(u + rv) - 0(u) 1 -=-- - tr { [o- (u + rv) - o-(u)] T P(t)[o-(u + rv) + a(u) - 2c1(U(t))]} 2 = o(v). Thus,
lirn
u-7,(0,Eu 7-10
H(u + rv) - 7-1(u)
-=
— H(u + rv) - H(u)
lim
u ,-. U(00AEU ri0
Consequently, by (2.20), the desired result follows. Now we present the following sufficient condition of optimality. Theorem 5.2. (Sufficient Conditions of Optimality) Let (S0)-(S4) hold. Let (-X ( .), U(•), p(.), q(.), Q(.)) be an admissible 6-tuple. Suppose that h(.) is convex, H(t, • , • ,p(t),q(t)) is concave for all t E [0,7 almost surely, and ]
(5.2)
H(t,Y(t),Ii(t)) = mcH(t,Y(t),u),
a.e. t E [0, T], P-a.s.
139
5. Sufficient conditions of optimality
Then (I(•), Ti,(•)) is an optimal pair of Problem (S). Proof. By the maximum condition (5.2), Lemmas 5.1 and 2.3-(iii), we have
0 E au fl(t,Y(t),U(t)) =
(5.3)
(t, -±(t),
p(t), q(t)).
By Lemma 2.4, we further conclude that
(5.4)
(IIr (t, (t), rt(t), p(t), q(t)), 0) E s ,u H (t,Y(t), ri(t), p(t), q(t)). -
Thus, by the concavity of H (t, • , • , p(t), q(t)), one obtains
10
(5.5)
{H (t, x(t), u(t), p(t), q(t)) — H (t, -±(t), U(t), p(t), q(t))}dt pT (
Hz (t, (t), U(t), p(t), q(t)), x(t) — -±(t)) dt,
o
for any admissible pair (x(.), u(.)). Define the following equation:
e(t) a x(t) _
(t), which satisfies
{ dW) = {b s (t,Y(t),U(t))e(t) + a(t)}dt
+ E[0,1(t,y(t),47.(0)e(t) +03(t)idiv(t),
(5.6)
3=1
where
a —b s (t,Y(t), U(t))6(t) + b(t, x(t), u(t)) — 03 (t) a —o--;(t,Y(t),ii(t))e(t) + o 3 (t, x(t), u(t)) —
{a(t)
1 <j < m.
t E [0, 7],
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
140
By the duality relation between (5.6) and (3.8), we have
E ( h,(Y(T)), C(T)) = — E ( p(T), C(T)) +E (p(0), e(0)) = —E
0
{ ( f x (t, (t), 11(0), e(t)) + ( p(t), a(t) ) +E q3 (t), 03(0)}dt
3 =1
=El (11,(t,Y(t),U(t),p(t), q(t)), C(t)) dt 13
—E
( p(t), b(t, x(t), u(t)) — b(t,Y(t), U(t))) 771
+E< q3 (t), (t, Y(t), u(t)) —
(t, Y(t),U(t)) ) dt
3 =1
Ef
(t, x(t), u(t), p(t), q(t)) — H(t,Y(t), 717.(t), p(t), q(t))}dt
— E fo
( p(t), b(t, x(t), u(t)) — b(t,Y(t), 41(t))) 711
+E(qi (t),(73 (t,Y(t), u(t)) — cr 3 (t, (t), rt(t)) ) dt 3=1 = —E f { f (t, x(t),u(t)) — f (t,Y((t),71(t))1dt. 0
On the other hand, the convexity of h implies
E ( hx (Y(T)), ( T)) < Eh(x(T))
—
Eh(Y (T)).
Combining the above two, we arrive at
(5.7)
J(u(-)).
J ( (.))
Since ic(.) E U10, 71 is arbitrary, the desired result follows. Let us give an example. Example 5.3. Consider the following control system (n = m = 1):
dx(t) = u(t)dW (t), t E [0, 1], x(0) 0,
(5.8) {
with the control domain being U = [0,1] and the cost functional being
(5.9)
1
J (u(.)) = E — f u(t)dt + x(1) 2 }. 0
6. Problems with state constraints
141
Suppose (Y(. ), Ti(. )) is an optimal pair (which we are going to identify). Then the corresponding adjoint equations are
(5.1 0)
f
dp(t) = q(t)dW (t), t E [0, 1], p(1) = —Y(1),
and
(5.11)
f
dP(t) =- Q(t)dW (t), P(1) = —1.
t E [0, 1],
Clearly, (P(t), Q(t)) = (-1, 0) is the unique adapted solution to (5.11). The corresponding fl-function is (5.12)
1-1(t,Y(t), u) =
u2 + (1 + q(t) +ri(t))u.
The function li(t, Y(t), •) attains its maximum at u = 1 -I- q(t) +1-7(t). Hence the necessary condition of optimality as specified by the stochastic maximum principle (Theorem 3.2) would be satisfied if one could find a control U(.) such that the corresponding q(t) —1. By the first-order adjoint equation (5.10), it is clear that if we take U(t) 1 with the corresponding state Y(t) = W (t), then the unique solution of (5.10) is (p(t), q(t)) = (—W (t), —1). In order to see that l(t) 1 is indeed optimal, let us calculate
H (t, x, u, p(t), q(t)) = H (t, x, u, —W(t), —1) = u — u = 0, which is concave in (x, u). Moreover, h(x) = is convex. So the optimality follows from the sufficient condition (Theorem 5.2). One may realize now that a key fact in proving the sufficient condition (Theorem 5.2) is (5.3). Based on this, the concavity of the Hamiltonian H and the maximum condition for the fl-function, roughly speaking, imply a maximum condition for H. So one may derive the optimality in the same way as in the deterministic case (Theorem 2.5). We have seen in Examples 3.1 and 3.4 how convex Hamiltonians (in u) are turned into concave fifunctions by adding the risk adjustment involving P(t). However, when the Hamiltonian H (t, x, u, p, q) is already concave in u, the second-order adjoint process P(.) plays no role at all, as was shown in the proof of Theorem 5.2. This is because the concavity of H reflects the risk adjustment already. 6. Problems with State Constraints In many applications, the state process x(.) is subject to some constraints. This section is devoted to a study of some of these situations. 6.1. Formulation of the problem and the maximum principle We consider the state equation (3.1) with the cost functional (3.2).
In
142
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
addition, we require that the state process x(.) satisfy fT
Eh(x(T)) + Ef(t, x(t), u(t))dt
(6.1)
E r,
13
where h : IFtn Re and f : [0, T] x x U —> Re are given maps, and F is a subset in Re . We call (6.1) a state constraint. Let us recall the following notions (see Chapter 2, Section 4.1). Any u(.) E U[0, T] is called a feasible control. The corresponding solution x(.) of (3.1) and the pair (x (.), u(•)) are called a feasible state process (trajectory) and a feasible pair, respectively. If in addition, the state constraint (6.1) is satisfied and the integrals in (3.2) and (6.1) are integrable, then u(•), x(.), and (x(•),u(•)) are called an admissible control, admissible state process (trajectory), and admissible pair, respectively. The set of all admissible controls is denoted by Uad[O, 7]. We note that the situation of the previous sections can be regarded as the case where F = Re with both h and f being zero. In that case, a pair (x(.), u(.)) is admissible if and only if it is feasible. However, we do need to distinguish these two notions in this section. In what follows, we define hl (x) (6.2)
h(x) =
f l (t,x,u)) , f(t, x, u) =
he (x)
fe (t, x, u)
with hi and f being scalar functions. For notational convenience, we will also define (6.3)
h° (x) = h(x), f ° (t, x , u) = f (t, x , u),
where h and f are the functions appearing in the cost functional (3.2). Before going further, let us look at some special cases of (6.1).
Case 1. Suppose we are faced with a multiobjective optimal control problem: Minimize (Jo(u()), (u0), • • • Je(u0)), where Ji(u()) = E
f i (t, x(t), u(t))dt + (x(T))} ,
i = 0 , 1, . . . , P.
A control U(.) is called an efficient control (solution) of the problem if there does not exist any control u(.) such that (u(•)) < Ji (U(•)),
j = 0, 1,
. ,t),
with at least one strict inequality. The set of all efficient controls is called an efficient frontier. By the multiobjective optimization theory (see Zeleny
6.
143
Problems with state constraints
[1]), the efficient frontier can be obtained by solving the constrained singleobjective optimal control problem Minimize Jo (u(-)), subject to Jz (u(•)) < 7-, ,
i = 1, . . . , P,
{
and varying the values of r i E ( oc, 1 oo) for i = 1, ... , P. It is clear that the above constraints specialize (6.1). -
- -
Case 2. P = n, h(x) = x, f = 0, and
r= x {
E
(xl, • • • )xn)
xi > ai , 1 < i < n}.
In this case, (6.1) reads
Ex t (T) > ai ,
i = 1, 2, ... , n.
A typical example of this kind of constraint is the following: In a manufacturing system, say, the expected inventory of the ith product should be at least ai at the end of the time period t = T.
Case 3. = n, h = 0,
Ix E
r
f(t, x, u) = (ci (t)x i ,
(x 1, • • •
xn)
<
ai,
, cn (t)x n ) T , and 1 < i < n}.
In this case, (6.1) becomes pT
ci (t)x i (t)dt
E
i = 1, 2, . , n.
o In the manufacturing system example, this could be interpreted as meaning that the total expected running cost of inventory for the ith product cannot exceed a given value a,. More special cases may be considered. We leave the details to the reader. Now we introduce the following assumption concerning the set the functions h and f.
r and
(S5) I' is convex and closed in RE . Moreover, for each i = 1, 2, ... , t', h' and fi satisfy (3.4)-(3.5). Under (S1)-(52) and (55), for any u(.) E U[0, 7], there exists a unique solution x(.) to (3.1), i.e., (x(•), u(•)) is feasible, and both the cost functional (3.2) and the left-hand side of (6.1) are well-defined. Then one may tell whether this pair is admissible by checking (6.1). Our optimal control problem with state constraint can then be stated as follows. Problem (SC). Minimize (3.2) over Uad[0,7"].
Our goal in this section is to prove a stochastic maximum principle for the optimal controls of the above problem. We point out that the difficulty in investigating the problem is that we can only compare the values of the
144
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
cost functional among all admissible controls (rather than merely feasible controls). To state our maximum principle, let us introduce the following Hamiltonian (compare with (3.10)):
H(t,x,u,p,q,11) ° ,0)1
(6.4)
f(t,x,u)— (i,G,f(t,x,u))
(p, b(t, x, u)) +tr [qT o-(t , x, u)] , (t,r,u,p,q,11)° ,1,1)) E [0,71x1Rn xUxillnx1117' x1Rx]R e .
It is seen that if i,b° = 1 and //) =- 0, then (6.4) reduces to (3.10). Theorem 6.1. Let (S0)—(S3) and (S5) hold. Let (YO,ft (. )) be an optimal pair of Problem (SC). Then there exist a (0°,0) E R1+1 satisfying
0 ° 2 o, i o 0 12 + 1 0 2 = 1 ,
(6.5)
(6.6)
( V), z — Eh(Y(T)) + f Ef (t, Y(t), Ti(t))dt
0,
Vz E
and adapted solutions
f (p(.),q(-)) E L 2 (0,T; Rn ) X [L2 (0, T; Irtn )] m , (P (. ), Q (-)) E L 2 (0,T; Sn) x [L2 (0,T; Sn)] m of the following first- and second-order adjoint equations:
dp(t) = —H (t, (t),
(6.7)
p(t), q(t),
, 74')dt + q(t)dW (t),
p(T) =
dP(t) = —{bx (t,Y(t),T.1(t)) T P(t) + P(t)bx (t,Y(t),Ti(t)) 'Fri
+ E 0-,;(t,Y(t), rt(t)) T P(t)c4(t,Y(t),U(t)) 3 =1
+ E [cr.; (t,Y(t),U(t)) 1- Q3 (t) + C
(t)cr (t, (t), 11(t))]
3 =1
(6.8)
+ Hz , (t, (t),
+E j=1
P(T) =
i=o
(t)di V (t)
p(t), q(t),I,G ° , 0)1 dt
145
6. Problems with state constraints such that H (t, Y(t), ri(t), p(t), q(t), 1,0 , —
tr
— H (t, -±(t), u, p(t), q(t),
,
o- (t, (t), (t)) — o-(t, Y(t),u)1 1- P(t)
(6.9) • Icr(t,Y(t), UM) — cr(t,Y(t), u)))
o,
Vu E U, a.e. t E [0,2],
P-a.s.
We call (6.6) the transversality condition. In the case where both h and f are zero and F =- le, namely, when there is no state constraint, condition (6.6) implies 1/) = 0. Thus, (6.5) leads to •t/P = 1, and the Hamiltonian H defined by (6.4) coincides with (3.10). Consequently, Theorem 6.1 reduces to Theorem 3.2. In general, however, 0 0 could be zero. When //)° 0, we say that the qualification condition is satisfied. One may also give a maximum condition, equivalent to the variational inequality (6.9), in terms of an 11-function similar to (3.16). We leave the details to the interested reader. The following subsections are devoted to a proof of Theorem 6.1. The reader may skip them at first reading.
6.2. Some preliminary lemmas
In this section we present some preliminary results that are necessary for the proof of Theorem 6.1. principle.
The first one is called Ekeland's variational
Lemma 6.2. Let (V, d) be a complete metric space and let F : V —0 (—oo, +(Do] be a proper (i.e., F +oo), lower semicontinuous function bounded from below. Let vo E D(F)g-{v E V I F(v) 0 be fixed. Then there exists a f; E V such that
(6.10) (6.11)
F(V) + Ad(v,v0 ) 5 F(vo), F(V) < F(v) + Ad(v,
Vv ii
.
Proof. Without loss of generality, we may assume A .= 1, since otherwise we may change the metric to Ad(. , •). Also, by considering the function F(.)—inf vE v F(v) instead of F(.), we may assume that F(.) is nonnegative valued. Next, we define a set-valued function
(6.12)
G(v) =
E V F(w) + d(w, v) 5 F(v)}.
Since F is lower semicontinuous, for any v E V, G(v) is a closed set in V and
(6.13)
V E G(v),
Vv E V.
146
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
Next, we claim that (6.14)
w E
G(v) =
G(w) Ç G(v).
In fact, (6.14) is true if F(v) = +oo, as G(v) = V in this case. Now let F (v) < oo . Then w E G(v) implies F (w) + d(w,, v) < F(v),
(6.15)
and for any u E G(w), we have
(6.16)
F (u) + d(u , w) < F(w).
Thus, combining (6.15) and (6.16), using the triangle inequality, one has
(6.17)
F (u) + d(u, v) 5 F (u) + d(u, w) + d(w , v) < F (v).
Thus (6.14) holds. Define
(6.18)
f(v) = inf F(w), wEG (t )
Then, for any w E G (v) , we have
Vv E D (F).
1(v) < F (w) < F (v) — d(w , v). Hence,
d(w, v) < F(v) — f (v).
(6.19)
So the diameter diam G(v) of the set G(v) satisfies
(6.20)
diam G(v)
sup d(w, u) < 2(F(v) — f (v)). w ,uec(v)
Now we define a sequence in the following way: Vn +i E G(vn ), n > 0, such
that
(6.21)
1
F
By (6.14), we know that G(vn+i ) C G(vn ). Thus,
(6.22)
f (vn) f (vn+i),
n > 0.
On the other hand, f(w) < F(w), since w E G(w) (see (6.13)). Therefore, in view of (6.21) and (6.22), we obtain
(6.23)
0 5 F(vn+i) — f(v
i )
f(v)
1
By (6.20), the diameter of G(v) goes to 0 as n oc. Since G(v) is a sequence of nested closed sets in V (i.e., G(vn+i ) C G(vn ), Vn > 0) and V is complete, we must have some point ti E V such that (6.24)
nG(vn) = n>0
147
6. Problems with state constraints
In particular, V c G(vo), which gives (6.10) with A = 1. Also, since f) E G(v) for any n 0, we conclude from (6.14) that
G ( b) Ç n G(v) =
(6.25)
{O.
n>0
This implies that G(t) = 01. Hence, for any v ti , we have v
G (ti ),
which gives (6.11) (with A = 1). Corollary 6.3. Let the assumptions of Lemma 6.2 hold. Let p > 0 and vo E V be such that
F(vo)
(6.26)
licit F(v) + p.
Then there exists a vp E V such that
F(v) < F( o ),
(6.27)
VT),
d(, v0 )
and for all y E V, -jrci(v,vp ) F(v) - F(vp ).
(6.28)
Proof We take A = 05. By Lemma 6.2, there exists a vp E V such that (noting (6.26)) (6.29)
F(vp ) + jxl(v p ,v0)
F(vo)
ilEit F(v)
+ p < F (v p ) + P,
and (6.30)
F(v) < F (v) + Vïxl(v,, v p ),
Vv
p.
Then, (6.27) follows from (6.29), and (6.28) follows from (6.30). Lemma 6.4. Let d be defined by the following: (6.31)
d(u(.),
{(t , co) E [0,7 ] x 521 u(t,w)
ii(t,w)}1,
Vu (.), ;fie) G U[0, T], where IAI denotes the product measure of the Lebesgue measure and the probability P of a set A C [0,7 ] x 5-2. Then d is a metric under which U[0, T] is a complete metric space. Proof Let {un (•)} be a Cauchy sequence in U[0, T] under the metric
d-,
i.e.,
(6.32)
d(un(•), um())
-■
0, n,rn -4 oo.
Then there exists a subsequence funk (•)}, such that (6.33)
d(un k (•),un„,(•)) < 2-k ,
k > 2.
148
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
Let En, = {(t,w) E [0,7 ] x St I un (t,c,i)
(6.34)
Ak
=--- U Erip ,„,+„
uni (t,w)},
n, rn > 1,
k? 2.
p>k
Clearly, Ak D Ak+i, Vk > 1, and 00
lAki<E 2—p = 21—k ,
(6.35)
k > 2.
p=k
Consequently, I
Uk>l iqI
= T. Now we define
tE
fi(t, c4)) = u nk (t, co),
(6.36)
k > 2.
A ck,
From the definition of Ak, we see that fi(.) is well-defined and fi(.) E 11[0,T]. Moreover,
ci(un,0 0, 110)
(6.37)
lAki
_4 0.
<21_k
Therefore, d(u(.), u( - )) —■ 0, proving the completeness of U[0, T]. Next, we consider the convex and closed set
dr(z) = inf I z — z'1,
(6.38)
z'Er
We call dr the distance function (to the set Lemma 6.5. Let
r c fftt be convex
VZ
0
r in Re . Define
E Ht .
r).
and closed. Then:
(i) cir : Ri --■ IR is convex and Lipschitz continuous with Lipschitz constant 1. (ii) For any z V
r, Odr(z)
has exactly one element with its length being 1.
(iii) dr() 2 is C l . Proof. The proof of (i) is immediate.
(ii) Let z V l'. Since dr( • ) is Lipschitz continuous with Lipschitz constant 1, from (2.20), it follows that if e E adr(Z), then
dr(z' + ty) — dr(z')
(CY) 5
< IYI,
Vy e IRe .
t 10
This implies
(6.39)
Z
<
On the other hand, since z V such that
(6.40)
1,
r,
Ve
e adr (z ).
for any 0 < 6 < 1, there exists a Z6 E
dr(z) ? (1 — 6)1z — 41 > 0.
r
149
6. Problems with state constraints As dr(•) is convex, by Lemma 2.3-(v) we have
(6.41)
adr(z) =
Thus, for any
(6.42)
(e,z' — z),
E Rt dr(z') — d(z)
Vz' E
E ad(z), we have (noting dr(4) = 0) —dr(z)
((,z6 — z) .
Combining (6.40) and (6.41), we obtain
(6.43) Hence,
(6.44)
> 1 — b. By sending
0, we obtain that V E adr(z),
lel = 1 ,
provided that z r. However, adr(z) is convex by Lemma 2.3-(i). Hence, adG(z) is a (nonempty) convex and closed set on the unit sphere in IFti (with the usual Euclidean metric). Thus, adr(z) must be a singleton.
(iii) A direct computation using the definition of derivative yields (6.45)
V [dr(z) 2] =
{2dr(z)adr(z), z 0,
z E
r.
Here, due to (ii), we may denote the unique element in adr(z) by adr(z) when z r. Clearly, the right-hand side of (6.45) is continuous. Since dr(z) = 0 when z E r, it will be convenient and without any ambiguity to write (instead of (6.45))
(6.46)
V[dr(z) 2] = 2dr(z)adr(z),
Vz E
We will use this convention in a later context.
6.3. A proof of Theorem 6.1. We now present a proof of Theorem 6.1. Besides the Taylor expansion and duality analysis, we need some more technique to handle the state constraints. In what follows, for notational convenience, we denote a generic point in Ri by x = (x 1 ,... ,xi ) T . For any feasible pair, let
(6.47)
xt(t) = h'(x(t)) +
fi (s, x(s),u(s))ds, 1 < i < t, o
x(t) = (x l (t),
, x t (t)) T ,
t E [0, 71.
By adding or subtracting a constant in the cost functional, we may assume, without loss of generality, that J ( ti()) = 0. For any p> 0, define
(6.48)
Jp (u(-)) = {[(J(u(.)) + p)] 2 dr (Ex(T)) 2 1112.
150
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
We call Jp (u(•)) a penalty functional associated with our Problem (SC). It is important to point out that we will study this functional over the whole feasible control set U[0,7 ] , rather than just the set of admissible controls. In other words, we are able to get rid of the state constraints by introducing such a penalty function. d) R is continuous and It is clear that J, : (1,1[0 ,
f Jp (u(-)) > 0,
Vue) E
(6.49) l jP(U") P
u(.)114,71 4(u(*))
Recall that by Lemma 6.4, (U[0,7 ], (1) is a complete metric space. Thus, by Corollary 6.3, there exists a up () E U[0,7 ] such that
(6.50)
J(u(.))
.1p(rt('))
d(up(*), 17t0) 5-
VT),
and
(6.51)
—05d(u p (•),u(•)) < Jp (u(•)) — Jp (up (•)), Vu(.) E U[0,7 ] .
If we call Problem (SC)P the optimal control problem with the state equation (3.1) and the cost functional Jp (u(•)) + VT-4(up (•),u()) (without the state constraint), then (6.50) and (6.51) imply that up (. ) is an optimal control of Problem (SC)P, which is very close to U(•), the optimal control of Problem (SC) for which we want to derive the necessary conditions. Our idea is to derive the necessary conditions for up (. ) and then let p 0 to get proper conditions for refl. Since Problem (SC)P does not have state constraints, and it differs from Problem (S) only in the cost functional, the derivation of necessary conditions for up (.) will be very similar to what we presented in Section 4. Thus, we will omit some of the similar proofs here. Fix a p > 0 and a u (. ) E U[0,71. For any E > 0, take any Borel measurable set E, C [0, T] with 1.E, = E. Define
(6.52)
uwt) =
up(t( )t),
t t EE[0.,
\ Ee ,
Clearly,
ci(uw.),up0)< lEe X
=
E.
6. Problems with state constraints
151
Thus, by taking u(•) = ucp (.) in (6.51), we obtain — Nik
5 Jp(ucp(*)) — Jp(up(*))
R404» +P) 41 2 [P(upe» +P)1 2 Jp(uep(*))+ 4(/( . )) dr (ExEp (T)) 2
— dr (EXp (T)) 2
(u( )) + p(u p()) .
(6.53)
— J (u p (-))1 + ( 0, E [x€p (T) — x p (T)] )
E
=
p[te (xWT)) — (x p (T))
2=1
[fi ( t , X ep (t),11(t)) — f 1 ( ,
f
(t), up (t))]dt}},
where xep (. ) and x p (-) are the states under controls uep (. ) and up (.), respectively, 'cep () and xp (.) are the corresponding functions defined by (6.47), and ?hp
A [J(up()) +Pr ±
.1-19 'e
(6.54)
A {
JP(up())
‘
as E
dr (EXp (n)adr (EX p (T)) + o(1), Jp ( u p )
0,
with /PeP = P • • • OtP'e). Now, in the present case, we also have Taylor expansions (in terms of e) similar to those of Theorem 4.4 along the pair (x p (.),up (•)). Thus, we have
EE
{
[
(x p (T)), (T) z(T))
i=o (6.55)
1 + ( hxx (x p (T))ep (T), Vp (T) )1
+
1
(t)yep(t), yep(t)) +6 1" (t)x E (t)]dt} +o(e),
where yep (. ) and zpe (. ) satisfy variational equations similar to (4.23) and (4.24) with (Y(•),U(•)) replaced by (x p (•),up (.)), and f'P (t), .f;,f (t), and 6 .1 2 ' P (t) are defined in a manner similar to (4.22) with (T(•),ii()) replaced by (x p (.),up (•)). Now, as before, we let (rfp (•),qpe(•)) and (P(.), Q(.)) be the adapted solutions of the following approximate first and second order -
-
152
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
adjoint equations:
dpep (t) = — Hz (t , x p (t) , u p (t) , K(t), qep (t),i,e,' , Vrep )dt + q7,(t)dW (t), e
(6.56)
E vp-hts (xp(T)),
{ K(T) = —
=o dP;(t) = —
(t , p (t), u p (t)) T 13;
+
(06 (t , x p (t), u p (t))
Trt
+ Eo-gt,xp(t),up(0) -rp;(00--;(t,xp(t),up(o) 3
+
=1
E [avt,xp(t),up(0) -FQ(t) 3=1
+ QEp (t)C4 (t , X p(t), p(t))1
(6.57)
+ H (t , xp (t), u p (t), K(t), q7,(t) , 11),(9) '' , •K,)}cit Trt
+ 3 =1
P(T) = —
E o ihtxx (xp(T)). i =0
Then, using duality relations similar to those of Section 4, we have
1
(6.58) 05e + o(e) E
{6 .W,(t) + tr [bap(t) T Pgt)6up(t)] hEe (t)dt,
where
(t)
(6.59)
H (t , x p (t) , u p (t) , K(t),
—
,
a(t, xp(t), u(o) _ c r (t, x p (t) , u p (t)).
5o (t)
From (6.54) and Lemma 6.5, we have
(6.60)
Ott),'
0( 1 ),
6 1 2 + 1/41 2 = 1 + o(1),
1
as e
0.
Thus, there is a subsequence, still denoted by (4'e, Op, such that
!ivo''e, for some (40,Gp ) E IR1±i with
(6.61)
IA?
0,
1141 2 + 10,912 = l•
153
7. Historical remarks
Consequently, from (6.56)-(6.57) and the continuous dependence of the (adapted) solutions to the BSDEs on the parameters (see Chapter 7, The0: orems 2.2 and 3.3), we have the following limits as E
f
(KO, 0))
(Pp('), 44)),
(P;('),Q ep('))
(Pp°, QM),
strongly in the spaces L2F (0, T; Rn) x [L2F (0, T; Rn )]m and L2F (0, T; Sn) x [L2F (0, T ; Sn)]m , respectively, where (pp (.), qp ()) and (Pp°, Q p (.)) are adapted solutions to BSDEs like (6.56) and (6.57) with 04 ,', 07) ) replaced 0, by (4, Op). Then, from (6.58), by properly choosing .EE and sending E we have
E
(t) + tr [6crp (t) i- Pp (t)6Grp (t)11
E{H(t, x p (t), u(t),pp (t),qp (t), 1P, ?, 11,p ) (6.62)
+ tr ([a(t,xp (t),u(t)) - a (t, x p (t), u p (t))] T Pp (t) [a(t, x p (t), u(t)) -
a (t, x p (t), u(t))]) 1.
Now we let p --0 0. By choosing a subsequence (if necessary) and using (6.61), we may assume that E R1+1 , and (6.5) holds. Then, by the second relation in (6.50) and the continuous dependence of the solutions to BSDEs, we obtain pairs (Il(.), q()) and (PO, Q(•)) as the limits of (pp (•),qp (•)) and (Pp (.),Q p (.)), respectively, which are adapted solutions of (6.7) and (6.8), respectively. Passing to the limits in (6.62), we obtain (6.9). Finally, by the definition of 07,, we have
(6.63)
( O pe , z - ExEp (T))
-e(1),
as
E —■
0.
Sending E 0 and then p 0 in the above, we obtain the transversality condition (6.6). This completes the proof.
7 • Historical Remarks The essential idea of the maximum principle can be traced back to the 1930s, when McShane [1] studied the Lagrange problem by introducing a very influential method that inspired the Pontryagin school later (See Berkovitz-Fleming [1] for detailed comments). The form of the maximum Principle that is now familiar to us was first obtained by Hestenes [1] in 1950 as a necessary condition for the solutions to calculus of variations problems. He used the result to investigate some time optimal control problems. The maximum principle obtained by Hestenes [1] turns out to be equivalent to
154
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
the so-called Weierstrass condition and is a consequence of Carathéodory's necessary condition in the classical calculus of variations (see Carathéodory [1,2], Pesch-Bulirsch [1], and Boltyanski [2] for further discussions). Boltyanski-Gamkrelidze-Pontryagin [L] announced the maximum principle for the first time in 1956. The first rigorous proof for the linear system case was given by Gamkrelidze [1] in 1957. The general case was proved by Boltyanski [1] in 1958. A detailed presentation appeared in BoltyanskiGamkrelidze-Pontryagin [2] in 1960. These results were summarized in the classical book (Pontryagin-Boltyanski-Gamkrelidze-Mischenko [1]*). Since then the theory has been developed extensively by many authors, and different versions of maximum principle exist, which can be found literally in almost every book on optimal control theory. See for example, Lee-Markus [1], Young [1], Warga [1], and Berkovitz [1]. McShane [2] commented that Pontryagin's school "introduced one important new feature. They allow the set 11(t, y) (the control set) to be closed sets, not demanding that they be open as previous researchers had." Pesch-Bulirsch [1] pointed out that the work of Pontryagin's school "led to the cutting of the umbilical cord between the calculus of variations and optimal control theory." People have been extending the maximum principle to infinite dimensions in various aspects. For a detailed description of these, see the book by Li-Yong [2] and the references cited therein. See also Sussmann [1] for a survey of some results on manifolds and the relation to differential geometry. Theorem 2.1 is one of the easiest versions of the maximum principle (without any state constraint), whose proof nevertheless catches the essence in the original proof of the Pontryagin school, namely, the spike variation. Sufficiency of the maximum principle in the presence of some convexity conditions was shown first by Mangasarian [1] in 1966, on which Theorem 2.5 is based, and by Arrow-Kurz [1]. Lemma 2.3 was essentially extracted from Clarke [2], where a systematic treatment of the generalized gradient can be found. It should be noted that Clarke's generalized gradient can be used to derive a maximum principle by assuming only the Lipschitz continuity of the functions b, f, and h with respect to the state variable; see Clarke [1]. Attempts were made to extend the maximum principle to the stochastic (diffusion) case shortly after the work of the Pontryagin school. The earliest paper concerning the stochastic maximum principle was published in 1964 by Kushner-Schweppe [1]. See also Kushner [2,3]. In 1972, Kushner [7] employed the spike variation and Neustadt's variational principle (Neustadt [1]) to derive a stochastic maximum principle. On the other hand, Haussmann [1] extensively investigated the necessary conditions of stochastic optimal state feedback controls based on the Girsanov transformation. These works were summarized in the book by Hauss• Recently, in a widely circulated preprint, Boltyanski [2] elaborated from his point of view the process of the birth of the maximum principle.
7. Historical remarks
155
mann [4]. However, due to the limitation of the Girsanov transformation, this approach works only for systems with nondegenerate diffusion coefficients. Along another line, Bismut [1,2,3] derived the adjoint equation as what is called today the backward stochastic differential equation via the martingale representation theorem. Bensoussan [1,3] further developed this approach. However, prior to 1988 all the results on the stochastic maximum principle were obtained basically under the assumptions that the diffusion coefficients were independent of the controls and/or the systems were nondegenerate and/or the control regions were convex. The results obtained were pretty much parallel to those of the deterministic systems, and one hardly saw an essential difference between the stochastic and deterministic systems from these results. During 1987-1989, in a weekly seminar led by X. Li at the Institute of Mathematics, Fudan University, a group of people, including Y. Hu, J. Ma, S. Peng, J. Yong, and X. Y. Zhou, were combating the problems with control-dependent diffusions. Their insistence on solving the control-dependent diffusion cases was based on the following belief: Only when the controls/decisions could or would influence the scale of uncertainty (as indeed is the case in many practical systems, especially in the area of finance) do the stochastic problems differ from the deterministic ones. In terms of the necessary conditions of stochastic optimality, Peng [1] first considered the second-order term in the "Taylor expansion" of the variation and obtained a stochastic maximum principle for systems that are possibly degenerate, with control-dependent diffusions and not necessarily convex control regions. The form of his maximum principle is quite different from the deterministic ones and reflects the stochastic nature of the problem. Peng's proof was simplified by Zhou [4]. Later, CadenillasKaratzas [1] extended the result to systems with random coefficients, and Elliott-Kohlmann [2] employed stochastic flows to obtain similar results. The proof presented in Section 4 is based on Peng [1] and Zhou [4]. At the same time as Peng obtained the stochastic maximum principle, Zhou [4] established the relationship between the stochastic maximum principle and dynamic programming via (once again!) the second-order variations. See Chapter 5 for more details. It is also interesting to note that it was the study of the stochastic maximum principle that motivated Peng to formulate nonlinear backward stochastic differential equations (BSDEs, for short), because the adjoint equations are by nature BSDEs. See more comments in Chapter 7. The sufficiency of the stochastic maximum principle was investigated by Bismut [3], in which the conditions derived were rather complicated. Zhou [10] proved that Peng's maximum condition is also sufficient in the Presence of certain convexity. Section 5 is mainly based on Zhou [10]. Problems with state constraints are very difficult in general. The result in Section 6 is an extension of Peng [1], with some ideas adopted from the proof of the maximum principle for deterministic infinite-dimensional Systems as in Li-Yong [1,2]. In particular, the results in Section 6.2 are ex-
156
Chapter 3. Maximum Principle and Stochastic Hamiltonian Systems
tracted from Ekeland [1,2] and Li-Yong [1]. Here, we should mention that the constraints that we can treat are mainly of finite codimension. Otherwise, there is no guarantee of the nontriviality of the multiplier (le ,i1)). Note that in general, has to live in some infinite-dimensional spaces. See Li-Yong [1,2] for some extensive discussion on this issue, although it is not particularly for stochastic problems. There is a very extensive literature on the necessary conditions of optimality for other types of stochastic optimal control problems. Let us just mention a few: For partially observed systems, see Fleming [2], Bensoussan [4], Baras-Elliott-Kohlmann [1], Haussmann [5], Zhou [7], and Tang [1]; for systems with jumps, see Tang-Li [1]; and for stochastic impulse control, see Hu-Yong [1].
Chapter 4
Dynamic Programming and HJB Equations
1. Introduction In this chapter we turn to study another powerful approach to solving optimal control problems, namely, the method of dynamic programming. Dynamic programming, originated by R. Bellman in the early 1950s, is a mathematical technique for making a sequence of interrelated decisions, which can be applied to many optimization problems (including optimal control problems). The basic idea of this method applied to optimal controls is to consider a family of optimal control problems with different initial times and states, to establish relationships among these problems via the so-called Hamilton Jacobi Bellman equation (HJB, for short), which is a nonlinear first-order (in the deterministic case) or second-order (in the stochastic case) partial differential equation. If the HJB equation is solvable (either analytically or numerically), then one can obtain an optimal feedback control by taking the maximizer/minimizer of the Hamiltonian or generalized Hamiltonian involved in the HJB equation. This is the so-called verification technique. Note that this approach actually gives solutions to the whole family of problems (with different initial times and states), and in particular, the original problem. However, there was a major drawback in the classical dynamic programming approach: It required that the HJB equation admit classical solutions, meaning that the solutions be smooth enough (to the order of derivatives involved in the equation). Unfortunately, this is not necessarily the case even for some very simple situations. In the stochastic case where the diffusion is possibly degenerate, the HJB equation may in general have no classical solutions either. To overcome this difficulty, Crandall and Lions introduced the so-called viscosity solutions in the early 1980s. This new notion is a kind of nonsmooth solutions to partial differential equations, whose key feature is to replace the conventional derivatives by the (set-valued) super-/subdifferentials while maintaining the uniqueness of solutions under very mild conditions. These make the theory a powerful tool in tackling optimal control problems. The viscosity solutions that we are going to discuss in this book can be merely continuous (not necessarily differentiable). In a more general framework, viscosity solutions can even be discontinuous. Such a situation will not be discussed in this book (see Section 7 for more biographical remarks on this topic). The rest of this chapter is organized as follows. Section 2 reviews the deterministic case. The notion of viscosity solutions to the first-order HJB equations is introduced. Existence and uniqueness of the viscosity solutions are discussed. Section 3 proves a stochastic version of Bellman's principle of optimality, based on which the second-order HJB equation is derived. -
-
Chapter 4. Dynamic Programming and HJB Equations
158
In Section 4 some properties of the value function are discussed. Viscosity solutions to the second-order HJB equations are defined in Section 5, and some basic properties of viscosity solutions are presented. Section 6 deals with the uniqueness of the viscosity solutions. This section is very technical, and the reader is advised to skip it at first reading. Finally, Section 7 gives some historical remarks. 2. The Deterministic Case Revisited For convenience, let us recall the formulation of deterministic optimal control problems. Given x o E lftn , consider the following control system:
i(t) = b(t, x(t), u(t)),
(2.1)
a.e. t E [0, T],
where the control tt ) belongs to (.
V[0,71 = fu (. ) : [0,T] -4 U 74.) is measurable}, with U being a metric space, T> 0, and b: [0, T] x lEr x U Rn a given map. The cost functional associated with (2.1) is the following: (2.2)
J(u()) = f f (t,x(t),u(t))dt + h(x(T)),
for some given maps f and h. In what follows, proper conditions will be imposed such that for any u(.) E V[O,T], the state equation (2.1) admits a unique solution x (. ) E C([0, T]; IV) and (2.2) is well-defined. The optimal control problem is stated as follows: Problem (D). Minimize (2.2) over V[0, 7]. Such a formulation of the (deterministic) optimal control problem has been already given in Chapters 2 and 3. Note that the initial time (t = 0) and the initial state (x(0) = x 0 ) are fixed in the formulation. The basic idea of the dynamic programming method is, however, to consider a family of optimal control problems with different initial times and states, to establish relationships among these problems, and finally to solve all of them. Let us now make this precise. Let (s, y) E [0, T) x IV, and consider the following control system over [s,T] (compare with (2.1)): ±(t) = b(t,x(t),u(t)),
(2.3)
y.
a.e. t E [s, T], x(s)=
{
Here, the control li(.) is in V[s,T] fu(.) : [s,T]
U u (. ) is measurable}.
159
2. The deterministic case revisited The cost functional is the following (compare with (2.2)):
(2.4)
J (s, y; u()) = f f (t, x(t),u(t))dt + h(x(T)).
We state the corresponding optimal control problem as follows: Problem (D sy ). For given (s, y) E [0, T) x Rn , minimize (2.4) subject to (2.3) over V[s,T]. The above is a family of optimal control problems parameterized by
(s, y) E [0, T) x Rn , in which the original Problem (D) is "embedded". Actually, Problem (13 5y ) with s = 0 and y = xo coincides with Problem -
(D). Once again, it is essential that we allow the initial (s, y) to vary in [0, T) x Rn . This will bring us useful "dynamic" information of the family of problems that constitute Problem (D5y ). Before going further, let us recall first Assumption (Dl) stated in Chapter 3, Section 2: (DI) (U,d) is a separable metric space and T> 0. In addition, we introduce (D2)' The maps b : [0,7 ] x x U Rn , f : [0,Tj x 1Rn x U R, and h : Iftn R are uniformly continuous, and there exists a constant L > 0 such that for cp(t, x,u) = b(t, x, u), f (t, x, u), h(x),
(2.5)
Icp(t, x , u) — (p(t,i , u)1 < Lx — Vt E [0, T],x, E V(t,u) E [0, l(p(t,O,u)l
Rn , U
E U,
We note that unlike (D2) introduced in Chapter 3, Section 2, (D2)' requires the continuity of the given functions in (t, x, u), including t. Clearly, under (Dl) and (D2)', for any (s,y) E [0, T) x Rn and u(.) E V[s,7], (2.3) admits a unique solution x(.) x(. ; s, y, u(•)), and (2.4) is well-defined. Further, we can define the following function: V(s, y) =
inf
-
(2.6)
u(•)Ev[s,T]
V (T,y) = h(y),
J (s, y; u(•)),
(s, y) E [0, T) x
Vy E Rn ,
which is called the value function of Problem (D). The function V( , .) will play an important role in obtaining optimal controls. Thus, we would like to study V(. , .) in great detail. First of all, we present the following result, which is called Bellrnan's
Principle of optimality. Theorem 2.1. Let (Dl) and (D2)' hold. Then for any (s, y) E [0, T) X Rn ,
.§ f f (t, x(t; s, y, u ( )), u(t))dt -
V(s, y) =
(2.7)
inf
u( )Ev[s,7]
s
+ VP,x(A;
y,u(-)))}, VO < s <
Chapter 4. Dynamic Programming and HJB Equations
160
Proof. Let us denote the right-hand side of (2.7) by V(s, y). By (2.6), we have
, x(S); u(.)),
V(s, y) < J(s, y; u()) = f f (t, x(t), u(t))dt + Vu( • ) E V[s, T].
Thus, taking the infimum over u (. ) E V[5,7] we get
(2.8)
V(s,y)
V(s,y).
Conversely, for any e > 0, there exists a u, (. ) E V[S, T] such that V(s,y) + E
(2.9)
J (s,y; uE (•)) f f(t,x e (t),u,(t))dt + V(.§,x e (g))
V (s, y),
where x,() = x(. ; s, y, u,()). Combining (2.8)-(2.9), we obtain (2.7). Let us make an observation on (2.7). Suppose (Y (. ), rt( • )) is an optimal pair of Problem (D, y ) and E (s,T). Then V(s, y) = J (s , y; r1(.))
(2.10)
f
=f
f (t,Y(t),11(t))dt + J(,Y(A); Tie))
f (t,Y(t),U.(t))dt + V (h,Y(:s))
V (s, y).
The last inequality in (2.10) is due to (2.7). Thus, all the equalities in (2.10) hold, in particular,
(2.11)
V (.4 , Y(.)) = J
GO; 11 0)
f f (t,1" (t),U(t))dt + h(Y(T)).
This means that Ti(.) is optimal on [s ,T] (with the initial (s, y))
ul[g,T1(*) is optimal on [.i, 7 ] (with the initial (&,()))
.
The above may be roughly interpreted as the following "principle":
(2.13)
globally optimal =
locally optimal.
This is the essence of Bellman's principle of optimality. Also, this gives a necessary condition for 14.) to be optimal. We refer to (2.7) as the dynamic programming equation. This equation gives a relationship among the family Problem (D ay ) via the value function. Unfortunately, this equation is very difficult to handle, since the operation involved on the right-hand side of (2.7) is too complicated. Thus, we would like to explore (2.7) further, trying to get an equation for V(s, y) with a
161
2. The deterministic case revisited
s impler form. It turns out that this can be done to some extent.
The following result gives a partial differential equation that a continuously differentiable value function should satisfy. Hereafter, we let C1 ([0, TI x1Rn ) be the set of all continuously differentiable functions y : [0, TI x 1Rn —* 1R. Proposition 2.2. Let (Dl) and (D2)' hold. Suppose V E C l ([0, x Rn ). Then V is a solution to the following terminal value problem of a first-order
partial differential equation: { — vt + sup H(t, x, u, —vx ) = 0, ucu VI t=T = h(x), x E Rn ,
(2.14)
(t, x) E [0, T) x Rn ,
where H (t, x, u, p) = (p, b(t, x, u)) — f (t, x, u),
(2.15)
V(t, x, u, p) E [0, T] x Rtm x U x Rn .
We call (2.14) the Hamilton-Jacobi-Bellman (HJB, for short) equation associated with Problem (D), and the function H (t, x, u, p) defined by (2.15) the Hamiltonian, which is the same as that defined in Chapter 3, Section
2. Proof. Fix a u E U. Let x (. ) be the state trajectory corresponding to the control
u(t)
u. By (2.7) with
V(, x(.4)) — V(s, y)
0>
1
—s —
Vt(s, Y) —
s,
—s
Vx(s, Y), b(s , u)
f (t, x(t), u)dt u),
Vu E U,
which results in
(2.16)
0 > —Vt(s, y) ± sup H(s,y,u, —V x (s,y)). uEu
On the other hand, for any s > 0, 0 < s < < T with enough, there exists a u(.) = ue ,g (•) E V[s,T] such that
V(s, y) E(.4 — s) >
— s > 0 small
f (t, x(t), u(t))dt + V(, x(A)).
Chapter 4. Dynamic Programming and HJB Equations
162
Thus, it follows that (noting V G C l ([0,
V(., x(g)) - V A —s
-E <
A 1s
(
S,
y)
x WI)) 1 —
f (t, x(t),u(t))dt
— Vt (t, x(t)) — (V x (t, x(t)), b(t, x(t), u(t)) ) — f (t, x(t), u(t))}dt
(2.17) — Vt (t,x(t))+ H(t,x(t),u(t), —V x (t,x(t)))}cit 1
— Vt(t,x(t)) + sup H(t,x(t),u, —V s (t,x(t)))}dt ueu —Vt (s, y) + sup H(s, y, u, —Vz (s, y)), as ueu —
s
In proving the last limit above, we have used the fact that (2.18)
lim
sup
I yo(t, y, u) —
(s, y, u)l = 0,
yER',uEU
for cp = b, f. This is implied by the uniform continuity of functions b and f as assumed in (D2)'. Combining (2.16) and (2.17), we obtain our conclusion. Now, let us briefly discuss how the solutions to the HJB equation might help us in finding an optimal control. Suppose we have found the value function V E C 1 ([0, 7] X Rn ) via solving the HJB equation (2.14). Moreover, assume that for any (t, x) E [0, T] x Tr, the supremum in (2.14) is achieved at u = U(t, x), i.e., H (t, x, U.(t, x), —V x (t, x)) = sup H (t, x, u, —V x (t, x)), uEU
(2.19)
V (t , x) E [0,T] x In addition, suppose that for any (s, y) E [0, T) x IV, there exists a solution Y (. ) to the following:
(t) = b(t,Y
(2.20)
ti(t,Y (0)),
a.e. t E [s,
Then, by setting Tt(t) = irt(t, Y(t)), we have
d (2.21)
d
(t , (t)) = 14(t, , (t))
(Vx (t,Y(t)),b(t,Y(t),U(t)))
= — f (t, (t), 11(t)),
t E [s , T].
Integrating (2.21) from s to T, we obtain (2.22)
V(s, y) = h(Y(T)) + f f (t,Y(t),U(t))dt = J (s, y;
163
2. The deterministic case revisited
This means that (Y(.), U(•)) is an optimal pair of Problem (D 9 ). The above implies that if we could obtain the value function V by solving the HJB equation, then we could, at least formally, construct an optimal pair for each of Problem (D, y ) and, in particular, for the original Problem (D), via (2.19). This is called the verification theorem (technique). The verification technique in principle involves the following steps to solve the original Problem (D): Step 1. Solve the HJB equation (2.14) to find the value function V(t, x). Step 2. Find 11(t, x) through (2.19). Step 3. (Y(-), Ti ( . )).
Solve (2.20) (with (s,y) = (0,x0)) to get the optimal pair
In Chapter 5 we will discuss the verification theorem in a more rigorous
way. To summarize, the key points in the dynamic programming approach are as follows: (1) Introduce Problem (D sy ); (2) define the value function V(s, y); (3) derive the principle of optimality; (4) derive the HJB equation and then follow the above Steps 1-3 to obtain an optimal pair. It is clear that one of the central issues in the above approach is to determine the value function V( , -). If (i) V E C 1 ([0, T] x 111n ) and (ii) the HJB equation (2.14) admits a unique (classical) solution, then V is characterized by the solution to the HJB equation (2.14). Unfortunately, neither (i) nor (ii) is true in general. Here is an example. Example 2.3. Consider a one-dimensional control system, ±(t) = u(t)x(t),
(2.23)
a.e. t G [s,T],
x(s) = y,
with the control u(t) taking values in U being (2.24)
[ 1, 1] and the cost functional -
J(u(•)) = s(T).
One sees easily that X(t)
= yer u(r)dr ,
Vt E [8,7].
Thus, the value function is given by the following: {
(2.25)
V(s, y) =
ye T—s
ye s-T
,
y < 0, y > 0,
which is only Lipschitz and not Cl ([0,T] x IR), since Vx (s, y) has a jump at (s, 0) for all 0 < s
u, p) = sup {Puy} =IPYI. lui<1
Chapter 4. Dynamic Programming and HJB Equations
164
Thus, the associated HJB equation is (2.26)
- vt + lxvs I = 0,
I ldt=T =
(t,x) E [0,T) x R, E R.
X,
We claim that this equation does not admit any C 1 ([0, x R) solution. To see this, suppose v E C 1 ([0,71 x R) is a solution of (2.26). Then it is necessary that v s (T, x) = 1,
Vx E R.
By the continuity of vs , we have (2.27)
V(t,x)E Na{(t,x)1 cp(x) < t <
vs (t, x) >0,
,
for some continuous function coo : R [0, T), and ct9 can be chosen symmetric in x (with respect to x = 0) and nondecreasing in l xi. Then, by (2.26), v satisfies (2.28)
{ vt = xv x
,
(
t,x) E
_V+ agn{(t,x) 1 x > 0, t E [0, 7] },
x E R.
v 1t=T = x) Let (2.29)
fT = t,
z) = v(T, ze - )•
z = xe t ,
We have
vt + vs [- ze - r] = vt - xv s = O. This implies that (I, does not depend on the argument 4:0(7, z) = 4)(z). It then follows from (2.29) that (2.30)
v(t, x) = (I)(xe t ),
T.
So we may write
V(t,x) E .Af+ .
By the terminal condition in (2.28), we have (2.31)
xe -T = v(T,xe -T ) = 4)(x),
Vx E R.
Hence, combining (2.30)-(2.31), we obtain (2.32)
v(t, x) = xet-T
V(t,
X)
EN+ .
Similarly, we can show that (2.33)
v(t,x) = xeT-t7
Thus, v coincides with V in leads to a contradiction.
N.
V(t,
X)
E
\N't
Since V is not in C i (Ar), neither is v. This
From this example, we have to admit that the assumptions of Proposition 2.2 are too restrictive. To find a rigorous assertion similar in nature to
2. The deterministic case revisited
165
Proposition 2.2 but without restrictive assumptions, we need the following notion. Definition 2.4. A function y E C([0, lution of (2.14) if
x Rn) is called a viscosity subso-
y(T, x) < h(x),
(2.34)
Vx E
and for any cp E C l ([0, x R.), whenever y - yo attains a local maximum at (t,x) E [0, T) x IV, we have
(2.35)
- Vt(t,x) + sup H(t,x,u, -cp x (t,x))
A function y E C([0, T] X iRn ) is called a viscosity supersolution of (2.14) if in (2.34)-(2.35) the inequalities "<" are changed to ">" and "local maximum" is changed to "local minimum." In the case that y is both a viscosity subsolution and supersolution of (2.14), it is called a viscosity solution of (2.14). We point out that in the above definition, one may replace "local maximum (minimum)" by "local strict maximum (minimum)" or even "global strict maximum (minimum)." We encourage the reader to give a proof of such equivalences. The following gives a characterization of the value function in terms of viscosity solutions to the HJB equation. Theorem 2.5. Let (D1) and (D2)' hold. Then the value function V (- ,.) satisfies
(2.36)
Iv(s,y) —
v 3,F)I (
—
+ ( 1 + iYi V igDis V(s, y), (3,V) E [0,7] x
791),
for some K > O. Moreover, V is the only viscosity solution of (2.14) in the class C([0, T] x Rn ). Proof. We split the proof into several steps. Step 1. V (. , .) satisfies (2.36). By Gronwall's inequality, taking into account (D2)', we easily get
{ ix(t; s,Y,u(*))1 5 K( 1 + IYI), Vt E [.9 ,T], u (. ) E V[s,T], (s, y) E [0, T) x Rn,
(2.37)
x(t; s,y,u(.)) - x(t; .3, y, u(.))I
< K{ly — gl + (1+ lyl v IYI)Is — 3I , Vs, -s- c [o, Th y, Y E Rn, t c [s V 3, T], u (. ) c V[s }
A .3,
7].
Chapter 4. Dynamic Programming and HJB Equations
166
Thus, it follows that
{ Ws, Y; u( . )) — J(s,g; u('))1 Vs G [0, T), y, y E IFC, u( • ) E V[s,7],
(2.38)
K( 1 + II V IF)is — 3 1, V(s,y;u( . )) — J(3, y; u(*))1 Vs,..3 E [0,T), y E Rn U(.) E V[S A 3,7]. ,
These yield (2.36). Note that in proving (2.36) at t = T, we have used the Lipschitz continuity of h.
Step 2. V(., .) is a viscosity solution of the HJB equation (2.14). Suppose cp E C 1 ([0, x IR.) such that V — cp attains a local maximum at (s, y) G [0, T) x Iftn. Fix a u E U. Let x(.) = x(.; s,y,u) be the state trajectory under the control u(t) u. Then, by Theorem 2.1, we have (for > s with A — s > 0 small enough) 0 (2.39)
< V(s, y) — ço(s, y) — V(, x(.)) + cp(h, x(h)) —s 1 i < sf j f (t, x(t), u)dt — (s, y) + f(s, Y7u)
Set(s,
crox (s3 y) b(Sly, u) ),
as :s
s.
This leads to —
Wt(s,y) + H(s, y, u, — (Pr(s, Y)) 0,
Vu E U.
Hence,
— Vt(s,y) + sup H(s, y, u, — ws(s, Y))
(2.40)
O.
uEU
On the other hand, if V — cp attains a local minimum at (s, y) E [O, T) x then, for any E > 0 and A > s (with A — s > 0 small enough), we can find a u(.) = E V[s,T] such that (noting (2.7))
0 (2.41)
V(s, y) — yo(s, y) — V(A, x(h)) + (pCs , x(h)) —E(A — s) + f f (t, x(t), u(t))dt + cp(A, x(h)) — (p(s, y).
5
Dividing by (A — s), we get —E
1
s
f(t,x(t),u(o) — ept(t,x(t))
S—S
— (yo x (t, x(t)),b(t, x(t), u(t)))}dt
(2.42)
1
x(0) + sup H (t, x(t), u, —Mt, x(t))) dt ueu sup H(s,y,u, as S. — 50 t(s y) + (px(s,y)),
—s
{—
uE U
2. The deterministic case revisited
167
As with (2.17), in obtaining the limit above, we have used the uniform continuity of the functions f and b, which leads to (2.18). Combining (2.40) and (2.42), we conclude that V is a viscosity solution of the HJB equation (2.14).
Step 3. The HJB equation (2.14) admits at most one viscosity solution.* Let us define
(2.43)
H(t, x, p) = sup H(t,x,u,p), V(t, x,p) E [0, T] x Iftn x Rn . uEU
By (DI) and (D2)', the following inequalities hold:
(2.44)
t.
1-1 (t,x,P) — 1-1(t,x, 4)1 5_ L(1+ Ixl)lp — ql, 11/(t,x,P) — 1-1(t,Y,Y)1 al(lx1 V 1Y1, Ix — Y1( 1 + 1p1)), Vt E [0,T x,y,p,q E ],
where a, : [0, oc) x [0, Do) [0, oc) is continuous, nondecreasing in each of its arguments, and CD(r, 0) = 0 for all r > 0. Let v(s, y) and ii(s, y) be two viscosity solutions of (2.14). To prove the uniqueness it suffices to prove
(2.45)
V(s, y) E [0, T] x
v(s, y) <(s,y),
Suppose (2.45) is not true. Then we may assume that sup [v(s, y) — ii(s, y)]
(2.46)
> 0,
(8,y)EN where
(2.47)
I
N = {(t, x) E (T — T0, T) x Rn 11x1 < Lo(t L 1 . 10< To
—
T + Toil,
By (2.44) and (2.47), for all (t, x) E Ar and p,q E
(2.48)
11/(t,x,P) —
4)1 < L(1 +1x1)1P — 41 < L(1 + LoTo)1P — 41= Lo1P — 41.
Now, take E. , 6 > 0 such that
(2.49)
E
< LoTo.
Take K > 0 and
E C°°(R) satisfying
(2.50)
K>
sup
(t,x,s,y)E.Nx.AT
{v(t, x)
—
y)},
* The reader may skip this part at first reading, since it is rather technical.
168
Chapter 4. Dynamic Programming and HJB Equations
and (2.51)
((r) =
0
'
—
For any û, @,y
r<
(V) 5 0, Vr E R.
K, r > 0,
> 0, we define
4)4, x, s, y) =v(t,x)— iï(s,y) —
1 — Y1 2
1 — — it — S
(2.52)
—Lo(t — T + To))
+ (( ( y), —Lo(t— T + To)) +7(t + s) — 2-yT, V(t, r s, y) E N ,
where (•) is defined as ( z ), a(izi 2 + ,2 ) 1,2, vz E itt Since (I) is continuous and JV x N is compact, we may assume that (I) attains its maximum over x N at (to, xo, so, Yo) E . Then, from x
(1)(T,0,T,0) < (13(to , xo, so, Yo), one obtains (noting (2.49) — (2.51)) 0 = v(T ,0) —
+ 2((e — LoTo)
< v(to , xo) — (so,Yo) — t ixo — Yoi 2 (2.53)
1 — — Ito — sol 2 +(((x0)e — Lo(to — T + To))
+ (((Yo ) — Lo(so — T + To)) + 7(to + so) — 2-yT < K + (( (x0), — Lo(to — T + To)) + (( (Yo)e — Lo(so — T + To)) + (to + so) — 2-yT. Now we claim that (2.54)
{
Ixoi < Lo(to — T + To), iYoi < Lo(so — T + To).
Indeed, if (2.54) is not true, then either (note izi <
z )6)
(x0) 6 — Lo (to — T + To) > 0 or
(Y0)E
LO ( S°
T
To) > O.
Thus, by (2.51), it follows from (2.53) that 0< K
—
K + -y(to + so) — 2-yT < 0,
which is a contradiction. Hence, (2.54) holds.
169
2. The deterministic case revisited On the other hand, by the fact that x0,
to, x0)
(b (so, Yo, so, Yo) < 243(t 0 , s 0, o, ye),
we obtain
v(to ,xo ) — (to x o) + xo) E — Lo(to — T + To)) + To)) LO(S0 YO + v (so , Yo) — (so , Yo) + )6
21(to + so) — 47T < 2v(to, xo) — 2Rso, Yo)
,23 Ito — so 1 2 Yo 1 2 — 7
Ix°
— Lo(to — T + To))
±
xo
+
Yo — Lo (so — T + To)) + (to + so) — 4-yT ,
which yields
(2.55)
2 2, ,2 Ix° — Yor + — Ito — sol 2 < v(to , xo) — v(so, Yo)
—
0
a
+ i-', (to, xo) —
o, yo) 5_ 27/at° — Sol + 1 x0 - y01),
where
g (r ) = —2
{ i v ( t, x — v(s, + 07*, x) —1-74 OW
sup
)
(t,x,.,y)Exxx
Clearly, lim,—.07/(r) = 0. By the boundedness of Ar, we have A
(2.56)
7/0 = SUP n(r) <00. r>0
Then it follows from (2.55) that (2.57)
Ix° — Yol <
NAW,
Ito
—
sol _5
Combining (2.55) with (2.57), we further have (2.58)
1 1 T i t ixo — yo l 2 + Ito — so I`
5_ q(Vairo +
Zle,6 '2 1 (t7 x) C
Lo(t —
n
Set )6
T + To ) — 61.
By (2.46) and (2.51), for e,d,-y >0 small enough, we may assume 4)(t, x, t, X)
sup (t,x)EA,6
(2.59) =
sup
(t,r)eA„6
[v(t, x) — fî(t, x)
27(t — T)] > —
2
> 0.
Chapter 4. Dynamic Programming and HJB Equations
170
Next, (2.60)
sup
(1)(t,x,t,$) < sup (I)(t,x,s,y)
(t,r)EA,5
Arx = (1)(to,x0, so, Yo)
v(to, xo) —
Yo).
We claim that there exists an ro > 0 such that for any 0 < a, 3 < ro, (to,xo,so,Yo) E .AT x Ar.
(2.61)
Note that the point (to , xo , so , yo ) depends on the choice of (a, 0, e,(5,7). We now prove (2.61). If it is false, then by virtue of (2.54), there is only one possibility, i.e., for some sequence (a m , Om ) ■ (0,0), the corresponding maximum point (tin , xm , 8m , ym ) of (13 over ./V x satisfies -
tm = T,
(2.62)
or
Sm
= T,
Vm > 1.
By (2.57), we obtain
1 xm Yrni
-4 0 , tm,sm
7',
as m -■ 00.
Hence, it follows from (2.59)-(2.60) that 0< < lim [v(tni ,x m )— fi(s,,y,)] = 0, 2which is a contradiction. Thus, (2.61) holds. Now we are going to use the definition of viscosity solution. For 0 < a, f3 < ro, by the definition of (to, xo, so, Yo), the function
(t,x)
v(t,x)-
i7(so,Yo) +
Ix - Y01 2
+ It - 501 2 - (((x), - Lo(t- T + To ))
- (((y o ),
—
Lo (so
—
T +To )) - ry(t + so ) +27T}
attains a local maximum at (to, xo) E ./V. Hence, we have 2 - (so - to) - OX)L0 + (2.63)
2 +H(to,xo, 7 t t (Yo xo) + ('(X)
xo )) < 0 , ( xo )E
A
where X = (xo), - Lo(to - T + To). Similarly, the function
(s, y)
I(8, y) — {v(to,xo) —
Yi 2
1 Tjito s 2 + (((xo), — Lo(to — T + To)) + (((Y)E — Lo(s — T + To)) + 7(to + s) — 2 y71 -
2. The
171
deterministic case revisited
attains a local minimum at (so, Yo) E N. Hence, we have
2
-
to)
+ C(Y)Lo
-Y
(2.64)
+
(so, Yo, -c-2t (yo -x0)
((Y) ( : ), ))
yo ), - Lo (so - T + TO. Combining (2.63)-(2.64), we obtain
where Y
2-y < Lo [(' (X) + ('(Y)] - H (to, xo,
- xo) + ('(X) ( xxo°) E
(2.65)
+
(so Yo, a 2 (Yo xo)
((Y) yY01))e )•
Choose some sequence /3 1 0 such that (to, xo, so, ho) converges. For notational simplicity, we still denote the limit by (to , xo , so , yo ) itself. Note that for this limit, by (2.57), we must have so = to, and (2.58) becomes 1
(2.66)
— yol 2 77( cs/W0).
Then (2.65) implies (using (2.44) and (2.48))
<
LoW(X) + (( 17 )] 2 - H (to, xo, - (Yo - xo) + C(X) a H(to,ho, t (yo - xo) - ( 01
( x x o° )E
( yYoo ) e )
Lo K.' (X) + C(Y)] + Lo1C(X) ( x0°)E x + ((Y)
4 1X0I IY011 IXO
YOI [ 1 ± I t (h13
LOICI(X) ( 1 ( Y )]
L O[1(1 ( X )1
+LD(IX01 V IY0171 X0
Y01( 1
° :0),
X0) ± (1 (X )
10 17 )11
( i (X )1)
Yol 2 )•
Since C(r) < 0, we obtain
< 2-y < 6)(Ixol V Wok ixo - Yoi( 1
l((X)i)+ e ixo - Y0 1 2 ).
Sending a -■ 0 and using (2.66), we obtain a contradiction. Therefore, (2.45) holds, and our conclusion follows. 0 From the above proof we see that the uniqueness part alone does not require the linear growth of f and h on x (whereas (2.36) does). Therefore, as far as the uniqueness of viscosity solutions is concerned, the functions f and h are allowed to have more general growth in x. A typical example is the linear quadratic optimal control problem.
172
Chapter 4. Dynamic Programming and HJB Equations
Next, we introduce super- and subdifferentials. [0,71 X Rn R and (t,x) E [0,T) x Rn , define Dv(t,x) = {(q,p) E R
X
For any function y :
Rn
v(s,y)—v(t,x)—q(s—t)—(p,y—x) < is ly — xi
0}
(2.67) D tl v(t, x) -= {(g, p) E R x Rn v(s,y)—v(t,x)—q(s—t)—(p,y—x) IS tl ly — xi s—.t,sE[O,T) Hill
>01 .
Y -..
Furthermore, we define D,14v(t, x) and Dt12F7x v(t, x) by restricting sl t in (2.67). We call Dv(t, x) (resp. Dt1:;v(t, x)) the first-order super- (resp. sub-) differential of v at (t, x), and Dtil:x v(t,x) (resp. Dt14v(t,x)) the first-order right super- (resp. sub-) differential of v at (t, x). Some basic properties of the super- and subdifferentials are collected in the following proposition. Their proofs are straightforward and hence left to the reader. Proposition 2.6. The super- and subdifferentials have the following prop-
erties: (i) The sets Dti,v(t,x) and D6v(t,x) are convex and closed, satisfying
(2.68)
I
Dt1::v(t,x) C
Dv(t,x)
C
D4frx v(t,x), Dt1;7xv(t,x),
Dt1::(—v)(t,x) = —Dv(t,x),
V(t, x)
E [0, T) x Rn .
Dti2E+,x (—v)(t, x) = — D47 x v(t, x).
(ii) v(t, x) is differentiable at (t0, x0) E (0,T) x Rn if and only if
(2.69)
T4::v(to,xo)
n
r)::x v(to,xo)
0;
in this case, it is necessary that both Dv(t 0 , x0) and Dti,' v(to,xo) be singletons and (2.70)
Dv(to, so) = Dti:;- v(to, xo) = { (vt (to, xo), vx (to, x0))}.
(iii) If v(t, x) is locally Lipschitz in (t,x) at (to, x0), then (2.71)
Dt1;:v(to,
Dti :;v(t0, x 0 ) C av(to,xo),
where av(to, x0) is Clarke's generalized gradient (see Chapter 3, (2.20)). (iv) If v(t, x) is convex in (t, x), then (2.72)
M:; v(t, x) = 0v(t, x) = kv(t, x), V (t , x)
E [0,T) x Rn,
173
2. The deterministic case revisited
where acv(t,x) is the subgradient in convex analysis (see Chapter 3,
(2.22)). The following results link the super- and subdifferentials to the notion of viscosity solutions.
Lemma 2.7. Let y E C([0,71 x Rn) and (to, xo) E [0,T) x Rn be given. Then
(i) (q,p) E /:4':v(to, xo) if and only if there exists a function cp E C l (R x Rn) such that y — (p attains a strict maximum at (to, x0) and (2.73)
((p(to, x0), (p t (to, x0), (p.(to, xo)) = (v(to, x0), q, p).
(ii) (q, p) E
D4+x v(to, x0)
if and only if there exists a function yo E C l (R x
Rn) such that
(2.74) {
((,0(to xo), (Pt(to xo), W.(to, xo)) (v(to xo), P), ço(t, x) > v(t, x), V (to , x 0 ) (t, x) E [t0, x R.
Proof. We prove (ii). The proof of (i) is similar. Suppose (q,p) E Dti4itxv(to,x0). Let
(y(t, x) — y(t0, xo) — 9(t — to) — (P, x — xo ) ) V 0 t — to + Ix — xol
(t, x) =
if (to, xo)
I. 0,
(t, x) E [to,
x Rn ,
x Rn, s — t +
ly - xi < r},
otherwise,
and
sup{(D(t, x) : (t, x) E [to, if r > 0, 0,
if r < O.
Then it follows from (2.67) that E : R ing function with e(0) ---- 0. Further,
(2.75)
[0, oc) is a continuous nondecreas-
v(t,x)— v(to,xo)— q(t — to) — (P,x — xo) < (t — to + Ix — x01)E(t — to + Ix — Xo V(t,x) E [to,
x R.
Set { 12(t—tol-lx—zol)
N2
E(p)dp + (t — to + ix — x0i) , (t, x) =
° 0,
if (t, x) E [to, T ] x Rn, if (t,x) E [0, to) x Rn .
Clearly, 0 E C l (R x Rn) with 7,b(to, xo) = 0, Ot(to, xo) =- 0, O x (to, xo) = 0,
174
Chapter 4. Dynamic Programming and.HJB Equations
and (noting (2.75))
2(t-t0+1,-.00
0(t, x)
ft-to+lx-sol
E(P)dP + (t
to + Ix - x00 2
to + lx - xoDE(t - to + - xol) + (t - to + - xop 2
(t
(t - to + ix xol)e(t to + - xop v(t,x)- v(to,xo) - 9(t - to) - (P,x - xo), V(to, xo) (t, x) E [t0,7] x
>
By defining
(P(t, x) = v(to, x0) + q(t - to) + (p, z - z 0 ) +0(t, x), (t, x) G R
X Rn ,
we obtain (2.74). This proves the "only if" part. The "if" part is clear.
0 The following are the parallel results for Dti.Çv(t,x) and D4-,x v(t,x).
Lemma 2.8. Let v E C([0, 7] X Rn ) and (to , x o ) G [0, T) x Rn be given. Then (j ) (q, p) E D; v(to , xo) if and only if there exists a function w E Ci (R X Rn ) such that y - w attains a strict minimum at (to , x0) and
((gto,xo),(Pt(to,xo),(px(to,x0)) = (v(to , x 0 ), q,p).
(2.76)
(ii) (q, p) E D 1 v(to,z0) if and only if there exists a function w E C l (R x Er) such that (2.77) {
(50 (to,xo),(Pt(to,xo),(Px(to,xo)) = (v(to,xo), 4,P), ço(t, x) < y(t, x), V(to, x0) (t, x) E [to, 7 ] X Rn .
Note that Lemmas 2.7 and 2.8 remain true if in (i) the word "strict" is omitted and in (ii) the inequality ">" or "<" is replaced by ">" or respectively. Let us say a few words about the geometric meaning of Lemmas 2.7 and 2.8. Suppose (q, p) is taken from Dt1 (to ,x0 ) (resp. D tl,'; (to , x0 )). Then one can find a C 1 function ço whose graph touches that of v from above (resp. below) only at the point (to, xo) in a neighborhood of this point, and the gradient ( pt, (Px) of (p at (to, xo) coincides with (q, p). In terms of the super- and subdifferentials, we can give an equivalent definition of viscosity solution.
Theorem 2.9. A function y E C([O, 71 x Rn ) is a viscosity solution of (2.14) if and only if y(T, x) = h(x) for all x E Rn , and for all (t,x) E [0,T) x Rn , { - q + sup H (t, z, u, -p) < 0, V(q, p) E Dtilv(t, (2.78)
u€U
- q + sup H (t, x, u, -p) ? 0, ueu
X),
'
V (q , p) E Dv(t, x).
3. The stochastic principle of optimality and the HJB equation
175
Proof. Suppose y is a viscosity solution of (2.14). Then, for any (q, p) E ./)v(t,x), by Lemma 2.7 we can find a ço E C 1 ([0, x Iftn ) such that y cp attains a strict local maximum at (t, x), at which (2.73) holds. By Definition 2.4, (2.35) holds. This gives the first relation in (2.78) by noting (2.73). One can prove the second relation in (2.78) similarly. Now, suppose (2.78) holds. If w E C i ([0,1] x Rn) is such that y — attains a local maximum at (t,x) E [0, T) x IV, then one can show that (Vt(t,x), Sox(t,x)) G Dtilv(t,x). Hence, using the first relation in (2.78), we obtain (2.35), which implies that y is a viscosity subsolution of (2.14). In the same fashion, we can show that y is also a viscosity supersolution of (2.14). 0 We have introduced D i ' ± y(t x) the right super-/subdifferential. Note that Theorem 2.9 is not necessarily true if Dy(t,x) is replaced by Dti2E±x y(t, x), as the former could be strictly smaller than the latter. However, under (D1) and (D2)', we have the following stronger result for the value function V. Theorem 2.10. Let (Dl) and (D2)' hold. Then the value function V E C([0, x Rn) is the only function that satisfies the following: For all (t, x) E [0, T) x
— q + sup H(t,x,u,—p) < 0, V(q, p) E Dti2E4-,s V(t,x), uEU
(2.79)
— q + sup
H(t, x, u, —p) >0, V(q, p)
uEU
E
D47x V(t, x),
V (T, x) = h(x).
Proof. For any (q, p) E Dt121%V(t, x), take the function ço as specified by Lemma 2.7-(ii) or Lemma 2.8-(ii) (with y replaced by V). Then we can use exactly the same argument as in Step 2 of the proof of Theorem 2.5 (note that only the right limit in time was used there) to obtain (2.79). On the other hand, the uniqueness of such functions satisfying (2.79) comes from the uniqueness part of Theorem 2.5 along with Theorem 2.9, in view of the facts that Dt1::V(t,x) C Dti ll-x V(t,x) and Dti;V(t,x) C Dt14_7s V(t,x). From the above, we see that under (D1) and (D2 )', the conditions (2.78) and (2.79) are equivalent in defining the (unique) viscosity solution (which is nothing but the value function V) of the HJB equation (2.14). 3. The Stochastic Principle of Optimality and the HJB Equation We now proceed to consider the stochastic case. Though the idea of dynamic programming is similar to the deterministic case, the analysis in the stochastic case will be much more delicate. 3.1. A stochastic framework for dynamic programming Let (I2, 7-, {Ft }t>o, P) be a given filtered probability space satisfying the usual condition, on which is defined an rn-dimensional standard Brownian
Chapter 4. Dynamic Programming and HJB Equations
176
motion W(t). We consider the following stochastic controlled system:
(3.1)
{dx(t) = b(t, x(t), u(t))dt + c r(t, x(t),u(t))dW(t),
t E [0, T],
x(0) = xo,
with the cost functional
(3.2)
(u()) =Elf f (t, x(t), u(t))dt + h(x(T))}
Define (recall Convention 2.9 in Chapter 1)
(3.3)
W[0,7]
: [0, T] x
U u is measurable and
{Ft} t >o-adaptedb where the superscript "s" indicates that the strong formulation is being considered. Under certain assumptions (which will be specified below), for any u(.) E U8 [0,7] equation (3.1) admits a unique solution, and the cost (3.2) is well-defined. The optimal control problem can be stated as follows.
Problem (S). Minimize (3.2) subject to the state equation (3.1) over LP [0, 7]. As introduced in Chapter 2, Problem (S) is in a strong formulation, which is the one we would like to solve eventually. It is observed in the previous section that one needs to consider a family of optimal control problems with different initial times and states along a given state trajectory in order to apply the dynamic programming technique. However, in the stochastic case the states along a state trajectory become random variables in the original probability space. More specifically, if x(.) is a state trajectory starting from xo at time 0 in a probability space (St, • T , P) along with a filtration liTtlt>o, then for any t > 0, x(t) is a random variable in (SI, F , P) rather than a deterministic point in Nevertheless, an admissible control u(.) is Vtlt>0-adapted, meaning that at any time instant t the controller knows about all the relevant past information of the system up to time t (as specified by Ft ) and in particular about x(t). This implies that x(t) is actually not uncertain for the controller at time t. In mathematical terms, x(t) is almost surely deterministic under a different probability measure P(. I.Ti ) (see Chapter 1, Proposition 2.13). Thus the above idea requires us to vary the probability spaces as well in order to employ dynamic programming. Very naturally, we need to consider the weak formulation of the stochastic control problem as an auxiliary tool. Now we set up the framework. Let T > 0 be given and let U be a metric space. For any (s, y) E [0,T) x IFtn, consider the state equation:
(3.4)
fdx(t) = b(t, x(t), u(t))dt + (t , x(t),u(t))dW (t), t E [s ,T], x(s) = y,
3. The stochastic principle of optimality and the HJB equation
177
along with the cost functional
(3.5)
J(s, y; u(-)) =
f (t, x(t), u(t))dt + h(x(T))}
Fixing s E [0, T), we denote by Liw[s, T] the set of all 5-tuples (0, F, P, W(.), u(.)) satisfying the following:
(i) (0,F, P) is a complete probability space (see Chapter, Section 1.1). (ii) 1147(t)It>5 is an m,-dimensional standard Brownian motion defined on P) over [s, T] (with W(s) = 0 almost surely), and ..Fi5 = cr{W (r) : s s - adapted process on (1,F, P). (iii) u: [s, (iv) Under /L(.), for any y E IV equation (3.4) admits a unique solution (in the sense of Chapter 1, Definition 6.15) x(.) on (Q,-F, It>s, P)• (v) f (- , x(.), u(•)) E L F 1 (0, T; II) and h(x(T)) E LIFT (0; Ft). Here, the spaces L F 1 (0, T; R) and L T (f2; IR.) are defined on the given filtered probability space (0, F {.Ff} t > s , P) (associated with the given 5tuple). ,
Sometimes, we simply write u (. ) E Uw [s,T] instead of (1Z,T,P,W(•),u(•)) E LI' [s, 7 ] if the probability space (0,F, P) and the Brownian motion W(t) are clear from the context. Notice that in the weak formulation defined in Chapter 2, Section 4.2, the filtration {.Fil t >., is also part of the control. However, here we restrict {F}>3 to be generated by the Brownian motion. Hence the formulation in this chapter is more special than the one in Chapter 2. We emphasize that in (3.4) the initial state y is a deterministic (almost surely) variable under (0,T, P). Also note that in (3.5) the mathematical expectation E is with respect to the probability P. Our optimal control problem can be stated as follows:
Problem (S 5y ). For given (s, y) E [0, T) x IR,n, find a 5-tuple Ti,(.) (11,T, P, rt(.)) Uw [s , T1 such that (3.6)
J(s, y; U(•)) =
inf
u(.)Etoo [3,T]
J (s,y; u(.)).
We now introduce some assumptions (compare with those imposed in Chapter 3).
(Si)' (U, d) is a Polish space and T > O. f: [0,7] x Ftn x U --+ R, and h R are uniformly continuous, and there exists a constant L > 0 such that for ço(t, x, u) = b(t, x, u), cr(t, x, u), (t, x, u), h(x),
(3.7)
ko(t,x,u) — (70 (t, 2,u)i 5 Llx — 21, Vt E [0, L V (t , u) E [0, x U. 142 (t 0 , '01
x,
E R'2 ,u E U,
Chapter 4. Dynamic Programming and HJB Equations
178
Note that unlike (S1)-(S2) introduced in Chapter 3, Section 3, (51 )' )' require U to be complete and the functions involved to be continuous-(S2 in (t, x,u), including t. Under (S1 )' -(S2 )' , for any (s, y) E [0,T) x IV and /L(.) E Leu[s,T (3.4) admits a unique solution x(•) x(• ; s, y, u(•)) (see Chapter 1, Theorem 6.16), and the cost functional (3.5) is well-defined. Thus, we can define the following function: ],
V(s, y) =
(3.8)
inf
u(.)euw [3,7]
V (T, y) = h(y),
J(s, y; u(•)),
V (s , y) G [0, T) x
Vy E lEtn ,
which is called the value function of the original Problem (S). The following result gives some basic properties of the value function. Proposition 3.1. Let (Sl )' -(S2 )' hold. Then the value function V (s , y) satisfies the following:
(3.9)
(3.10)
IV(s,Y)i 5 K( 1 +
V(s,y) E [0, T] X Rn
Iv(s,Y) - V(.4,01 5_ KflY - Qj + ( 1 + y V 1 -01.9 - :51 1/2 1, E [0,
y,Q E
Comparing (2.36) with (3.10), we see that unlike the value function V(s, y) of Problem (D), which is (locally) Lipschitz continuous in the time variable s, the value function V(s, y) of Problem (S) is not necessarily
(local) Lipschitz continuous in s. This is an essential difference between the deterministic and stochastic cases. Proof. Let (s, y) E [0, T) X IFtn be fixed. For any (f2,..F,P, W(•), u()) E [s, T], by Theorem 6.16 of Chapter 1, we have
(3.11)
E sup lx(t)I < K(1 -ItE[8,71
IY1)•
Thus, by (S2)',
(3.12)
1J(s,YI u(*))1
K(1 +10,
Vu (. ) E Uw [s,
This implies (3.9). Next, we let 0 < s < < T and y, Q E IV. For any w-admissible control (f2, F , P, WO, u(.)) E Uw [s , Th let x (. ) and X (. ) be the states corresponding to (5, y, u(.)) and (.s, u(.)), respectively. Then, again by Theorem 6.16 of Chapter 1, we have
(3.13)
E sup lx(t) - ±(t)I < Kfly tE[A,T]
+ (1+ ly1 V IDIs - .§1 1 /2 1.
Hence by using (S2)', we obtain
(3.14) J (s, y; u(.)) - J(s, 9; u(•))I 5_
Kfly - Q + (1 + II v LiDis - ,i11/21.
3. The stochastic principle of optimality and the HJB equation
179
(s , 71 we get (3.10).
Taking the infimum in u(.) E
Now let us present a technical result. For any s E [0, T) and any (9,F, P, WO, u(-)) E Uw [s , 7], take g E [s,T) and an Fis-measurable random variable e. Under (S1)'-(S2)', we can solve the following SDE on (0, Y., Vnt>s, P):
(3.15)
dx(t) = b(t, x(t), u(t))dt + o- (t, x(t), u(t))dW (t), t E
Denote the solution by x(. ; g, 6, u()). We have the following lemma. and (0, .F, P, W(•), u(•)) E Uw [s , Lemma 3.2. Let (s, y) E [0,T) x Then, for any ."'s E [s, T) and .7...1 measurable random variable e, -
, 6(w); u(.)) =Eff f (t, x(t; , 6 , u(.)), u(t))dt (3.16)
+ h(x(T; , e, u()))1.Fil(w), P-a.s.w. Proof. Since u ) is {FP } t > s-adapted with Ft = cr{W (r) : s < r < t } , by Theorem 2.10 in Chapter 1 there is a function 7,b E AT(U) such that (.
u(t,w) = -0(t, W(• A t, w)),
P-a.s.w E 11, Vt E [s,7].
Therefore, (3.15) can be rewritten as
dx(t) = b(t, x(t), 0(t,W (. A t)))dt + o- (t, x(t), -0(t, We A t)))dW (t), -
(3.17)
t
E
Due to (S1)'- (S2)' and Chapter 1, Theorem 6.17, this equation has a unique strong solution, and weak uniqueness holds (i.e., unique in the distribution sense; see Chapter 1, Definition 6.6). On the other hand, by Proposition 2.13 of Chapter 1,
PILD eP) = (L4)) 17 (w) = 1, P-a.s. This means that there is an no E .7- with P( 120) = 1, so that for any fixed wo E 00, 6 becomes a deterministic constant 6(wo) under the new probability space (0, .F, P(.1.F;)(w0)). In addition, for any t? &,
u(t, w) = 71,(t, We A t, w)) --= (t, 1 ji (. A t, w) + W (g , w)), where W(t) = W(t)—W(A) is a standard Brownian motion. Note that W(.§) almost surely equals a constant W(., wo ) under the probability measure P(.I.Fn(wo ). It follows then that u(t) is adapted to (the standard Brownian motion) W(t) for t > Hence by the definition of admissible controls,
(S2, F ,P(•1.73)(wo),1717- (t) ,u(t)) E 1,1" [g , T].
Chapter 4. Dynamic Programming and HJB Equations
180
Thus if we work under the probability space (RI", P ( i-71) ( o)) and notice the weak uniqueness of (3.15) or (3.17), we obtain our result. 3.2. Principle of optimality The following is the stochastic version of Bellman's principle of optimality (compare Theorem 2.1 in the previous section). Theorem 3.3. Let (S1)'—(S2)' hold. Then for any (s, y) e [0, T) X Rn ,
V (s, y) —
inf
E{
(3.18)
f (t,x(t; s, y, u0), u(t))dt T.
Proof. Denote the right-hand side of (3.18) by V(s, y). For any e > 0, there exists an (SI, , P, WO, u(•)) E U' [8,7] such that (noting Lemma
3.2) V(s, y) + e
> J(s, y; tt (.))
-= .E1 f f (t,x(t; s, y, ti(.)), u(t))dt + h(x(T; s, y, u(.)))
=Elf
f (t,x(t; s, y,u(-)),u(t))dt
5
+ E[f f (t,x(t; s, y, u(.)),u(t))dt + h(x(T; s , y, u(-)))1.711
=
f f (t, x(t; s, y, u0), u(t))dt + E[ f f (t, x(t; x(i),u()),u(t))dt + h(x(T; x(h),u()))1.71]}
=-
f (t, x(t; s, y, u(.)), u(t))dt + J
x(i; s, y, ue)); u())}
?Elf f (t,x(t; s, y,u(-)),u(t))dt + V(, x(&; s, y,u()))}
V(s,y).
Conversely, for any E > 0, by Proposition 3.1 and its proof, there is a b = b(e) such that whenever ly — 91 < 6,
(3.19)
,Y;u0) . 1 0,9;u0)1 +
Y) — V(.,9)1 5 e, Vu (. ) E Uw[h,T].
Let {Di } i>1 be a Bord partition of Fr (meaning that Di E 13(1R11 ), Ui>1 D3 — Rn , and Di n Di = 0 if i j) with diameter diam (Di ) <6. Choose x j E D. For each j, there is (0j , Tj , Pj , Wi(.),uj(-)) E Uw P, TI such that
(3.20)
J(i,xj;uj (•)) 5 V(S,x ) +E.
3. The stochastic principle of optimality and the HJB equation
181
Hence for any x c D, combining (3.19)-(3.20), we have
(3.21) J(§, x; u3 (.)) < J (5, x 3 ; u3 (.)) +e <
x 3 ) + 2e < V(., x) +36.
By the definition of the w-admissible 5-tuple (S/ 3 , .7-3 , P3 , W3 (•), u3 (•)), there is a function 03 E AT(U) such that
u3 (t, w) = 03 (t,1,173 (• A t,w)),
u.) E 113 , Vt E [:5, 7 ] .
x(•; s, y,u()) deNow, for any (S2, P, W (. ), u()) E Uw [s, 7], let x(.) note the corresponding state trajectory for Problem (S,y ). Define a new control w)
= f u(t, w),
if t E [s, :5);
103 (t, W (. A t, Clearly, (1Z,
P, W(.),
— W(.4, w)), if t E [h, 7] and x(t, E
E D.
[s, T]. Thus,
V (s, y) < J(s, y;
= Ef
f (t, x(t; s, y,
il(t))dt + h(x(T; s, y, -11(.)))}
=Elf f (t, x(t; s, y, u(.)), u(t))dt + E [ f f (t, x(t; :5 , x(g), u(.)), ii(t))dt+ h(x(T; x(h),iii0))1.F.sill
=Elf f (t, x(t; s, y, u (. )), u(t))dt +
x(.5; s, y,u());1.74.))}
<Eff f (t, x(t; s,y,u(.)),u(t))dt + V (.5,x(g; s,y,u())) + 3e} , where the last inequality is due to (3.21). Note that in deriving the above we have employed Lemma 3.2 and the weak uniqueness of solutions. Consequently, we obtain our conclusion by taking the infimum over
u(-) E Llw[s,T]. Theorem 3.4. Let (S1)'-(S2)' hold. If (Y (.), Ti (. )) is optimal for Problem
(S. y ), then
(3.22)
V(t, (t)) =E{
f (r, -t(r), ii(r))dr + h(Y(T))1.77} , P-a.s. , Vt E [s
Proof. By the same argument as that in the proof of Theorem 3.3, we
Chapter 4. Dynamic Programming and.HJB Equations
182
have
V(s, y) = J(s, y; ri(•)) t = E{ f f (r,Y(r),U(r))dr s T
+ E[ f f (r,Y(r),U(r))dr + h(Y(T)) .77]} t
(3.23)
T
=
f (r,Y(r),Ti(r))dr + EJ(t, -±- (t);?-2-0)
E ft
? E ft
T
f (r, Y(r), ii(r))dr + EV(t,Y(t))
?- 17 (s, 0, where the last inequality is due to (3.18). Hence all the inequalities in (3.23) become equalities. In particular,
EJ(t,Y(t);/7.0) = EV(t,Y(0). However, by definition one has V(t,Y(t)) < J(t,Y(t);U(•)), P-a.s. Thus
V(t, -±(0) = J(t,Y(t);U()),
P-a.s. ,
0
which gives (3.22).
3.3. The HJB equation As in the deterministic case, we call (3.18) the dynamic programming equation. This equation is very complicated, and it seems impossible to solve such an equation directly. This subsection is devoted to a formal derivation of a partial differential equation that the value function V( ,.) should satisfy, based on equation (3.18). We let C 1,2 ([0, 7 ] X Rn ) be the set of all continuous functions y: [0,1] x Rn ■ R such that yt , yx , and yxx are all continuous in (t,x). -
Proposition 3.5. Suppose (S1)'-(S2)' hold and the value function V E C 1 ' 2 ([0, 7] X Rn ). Then V is a solution of the following terminal value problem of a (possibly degenerate) second-order partial differential equation: { (3.24)
yt + sup G(t,x,u, vs , - V ss ) =- 0, ucu x E Rn , vIt=7, = h(x), -
-
(t, x) E [0, T) x IV,
where 6, 1
(3.25)
G(t, x, u, p, P) = - tr (Pa(t, x, u)ct(t, x,u) T ) 2 + (p, b(t, s, u) ) - f (t, x, u), V(t,x,u,p, P) E [0,7] x Ill n x U X IRn X Sn.
3. The stochastic principle of optimality and the HJB equation
183
Proof. Fix (s,y) E [0, T) X Rn and u E U. Let x(.) be the state trajectory corresponding to the control u(.) E /1"[.s,T1 for Problem (Ssy) u. By (3.18) with g 1 s and Itb's formula, we obtain with u(t) O?
=
.E {V(., x(A)) — V (s, Y)} —s 1
E f
{
g 1 Ef f (t, x(t),u)dt h—s
Vt (t,x(t))
.9
+G(t, x(t),u (t), —Vx (t,x( t)),—Vxx (t,x (t) ))} dt — Vt (s, y) + G(s, y, u, —Vs (s, y), —V ss (s,y)), Vu E U. This results in
0 > —vt(s, y) + sup G(s, y, u, —17,(s y), — Vxx(s,y)).
(3.26)
uEu
On the other hand, for any e > 0, 0 < s < < Twith g — s > 0 small enough, there exists a u(.) u(.) E U w [S, 7] such that
V(s, y) + r(g
— s)
?Elf f (t, x(t), u(t))dt + V(, x(h))}.
Thus, it follows from Itô's formula that as :91 s,
E{V (., x(h)) — V (s,Y)}
—e <
1
s
s
1
g
E] f (t, x(t),u(t))dt
{ — Vt (t,x(t))
+ G(t, x(t), u(t), —V s (t,x(t)), — Vss (t,x(t)))1cit
(3.27)
{ — Vt(t, x(t))
+ sup G(t,x(t),u, —V x (t,x(t)),—V xx (t,x(t)))1dt ucu —Vt(s, y) + sup G(s,y,u, — Vx(s,Y), — Vxx(s,Y))• ucu In proving the last limit above, we have used the fact that
(3.28)
lim t
sup
bp(t,y , u) — cp(s , y, u)1 = 0,
yER',uEU
for yo = b, a, f. This is implied by the uniform continuity of the functions b, a, and f as assumed in (S2)'. Combining (3.26) and (3.27), we obtain our conclusion. We call (3.24) the Hamilton Jacobi Bellman equation (HJB equation, for short) of Problem (S). The function G(t, x, u,p, P) defined by (3.25) is called the generalized Hamiltonian, which was defined earlier in (3.15) of Chapter 3. By comparing (3.24) with (2.14), we see that (3.24) is of second -
-
184
Chapter 4. Dynamic Programming and HJB Equations
0, (3.24) reduces order, allowing degeneracy. In the case that .7(t, x, u) hand, when is uniformly to (2.14). On the other (o-o-T )(t, x, u) positive definite, along with other mild conditions, (3.24) admits a classical solution (see Fleming—Soner [1, p. 169]). 4. Other Properties of the Value Function In this section we are going to explore more properties of the value function V defined in the previous section. 4.1. Continuous dependence on parameters Suppose we have a family of parametrized optimal control problems, and we investigate whether the value function depends continuously on the parameters under proper conditions. Such dependence will be useful in approximation and/or regularization. Consider a family of problems parameterized by e E [0, 1], for which the state equations, each called (3.4),, are of the form (3.4) with b(t, x, u) and o- (t, x, u) replaced by bE (t, x,u) and cre(t, x, u), respectively, and the cost functionals, each denoted by J' (s, y; u()), are of the form (3.5) with f (t, x, u) and h(x) replaced by fe (t, x, u) and le (x), respectively. The corresponding optimal control problems are called Problem (S:y ) (and Problem (SE) if (s, y) = (0, x0)). Also, we assume that Problem (S°8y ) coincides with Problem (Ssy ) stated in Section 3.1 (i.e., 6° = b, 0M = o f° = f, and h° = h). We can define the value function, denoted by V' (s, y), of Problem (SE) in the same way as that for Problem (S), provided that (S2)' holds for 6E, o', fE, and he. We then have the following convergence result. -
,
Proposition 4.1. Suppose (S1)'—(S2)' hold for (bE , a', fE, he) with the constants in (3.7) uniformly in all e E [0, 1]. Suppose further that
(4.1)
lim içoe (t, x, u) — ço ° (t, x,u)i = 0,
uniformly in (t, u) E [0, T] x U and x in compact sets of Rn , where (p' =be , a' , fe , hE . Then
(4.2)
lim V' (s , y) = V ° (s , y)
e—P0
V (s, y),
uniformly in (s, y) in any compact set of [0, T] x Rn . Proof. By the assumption (4.1), we can find a continuous function [0, oc) that is nondecreasing in each of its arguments, with n(0, R) -= 0 for all R > 0, such that : [0,00) x [0, 00)
(4.3) lyoe(t, x, u)
—
cpc (t,x,u),<,,(E,Ix1),
v (t,x,u ) E [0, T] x Rn x U.
Let (s , y) E [0, T] x Rn be fixed and take any u(.) E Liu' [8,T]. Define x'(.) (3.4), for e E [0, 1], and x(-) = x°(.) the
x'(• ; s, y, u()) to be the solution of
-
4. Other properties of the value function
185
solution of (3.4). Then, using Itô's formula, Burkholder-Davis-Gundy's inequality, and Gronwall's inequality, we have
sup Ixe(t) - x(t)1 21 E[ s
ICTE (t , X (t) u(t)) — (t , X (t), ii(t))1 2 dt
E n(E; ix (0) 2 dt.
K
By (S2)', 2L(1
n(Ei 'XI)
'xi),
Vx E ttn ,
> 0.
Thus, for any R> 0, En(c; Ix (t)I) 2 < n(E; R) 2 + E[1{11(oi> R} 4L2 (1 + lx(t)1 2 )]
5_ 77 (E; R) 2 + KP(Ix(01 > V 12 [E(1 + lx(t)1 4 )] 1/2 2 (EIX(014)1/2 < n(E; R) + K (1 + 1y1 2 ) R2 R-F2 10) < n(E; R) 2 K (1
(4.5)
'
Combining (4.4)-(4.5), we obtain urn E[ sup Ixe(t) - x(t)I] = 0.
(4.6)
8
Consequently,
IJE (s;y; u( •))
-
As, y;
KE{ sup le(t) - x(t)1 Ih E (x(n) h(x(T))1 s
+ f if e (t, x(t), u(t)) — f (t, x(t), u(t))1dt}
(4.7)
G ICE{ sup ixE (t) - x(t)I + 77(6 ; lx(T)l) + f n(E;lx(t)Ddt} s
VR>0.
It follows that (4.8)
!inl)
(s; y; u( . )) - J(s; y; u('))1 =
o,
uniformly in (s, y) on compact sets of [0, T] x Iftn and u(.) E Leu [s , T]. This leads to (4.2).
Chapter 4. Dynamic Programming and HJB Equations
186
Let us look at a special case of the above result. Let
(4.9)
{ be (t, x, u) = b(t, x, u), fe (t, x, u) = f (t, x, u), Fi' (x) = h(x), ae (t, x, u) = (0"(t, x , U,), A,h; In)n X (rn+n).
Denote the value function of corresponding Problem (S) by 1/e(s, y). Problem (S°8y ) is basically the same as Problem (Ssy ) in view of the uniqueness of the weak solutions to the systems. Thus V ° (s , y) coincides with V(s, y). By Proposition 4.1, we have the following corollary. Corollary 4.2. Let (S1)' (S2)' hold. Then there exists a constant K > such that -
(4.10)
1VE (s,y) - V(s,y)1 < K
V(s, y) E [0, T] x Rn .
Proof. In the present case, (4.4) can be improved to
(4.11)
E [ sup Ixe(t) - x(t)I 2] < Ke, s
and (4.7) is replaced by
(4.12)
Je (s, y; u(*))
-
J( 8 7 y; u(*))1
Ks/i, Vu(S) E LI"[s,
Then (4.10) follows.
0
We know that V(s, y) is a solution of (3.24) if it is smooth. Also, if Ve(s, y) is smooth, then it is a solution of the following HJB equation: {
-
4
-
EA4 + sup G(t,x,u, -v se , -v zE x ) = 0, uEU
(4.13)
(t, x) E [0,T) x Rn , t=T
= h(x),
x E Rn .
This is a terminal value problem of a nondegenerate second-order parabolic partial differential equation. According to some standard theory of PDEs (see Krylov [2,3]), when (S1)'-(S2)' holds, (4.13) admits a unique classical solution. Thus, the above result roughly says that the classical solution V' of (4.13) approaches V, which is hopefully a "solution" (in a certain sense) of the degenerate parabolic equation (3.24). This will be made rigorous a little later. 4.2. Semiconcavity In this subsection we present another important property of the value function. Let us start with the following definition. Definition 4.3. A function ço : R is said to be semi concave if there exists a constant K > 0 such that 1,b(x) ça(x) K lxI 2 is concave, i.e., -
(4.14) 7,b(Ax + (1 - A)y) > Ab(x) + (1 - A)ii)(y), Vx, y G Rn , À c [0, 1].
4. Other p