ADVANCED TEXTS IN ECONOMETYCS General Editors
C. W. J. Granger
G. E. Mizon
STOCHASTIC LIMIT THEORY An lntroduction for Econometricians
JAMES DAVIDSON
(lxford llniversity Press 1994
Oxjord University Press, Walton Street Oxjord oxa Oxford New York Athens Auckland Banghok Bombay Calcutta Cape Tozvn Dar es Salaam Delhi Florence Hong Kong Istanbul Karachi Kuala Lumpur Madras Madrid Melbourne Mexico City Nairobi Jtzrl Singapore Ttzzc Tohyo Toronto and assoclted companies in Berlin Ibadan ,
6Dp
Published in the Unitei States by Oxford University Pyess l'nc., Nec York
(0 JamesDavidson, 1994 All rlk/lfl reserved. No part of this publication may be repvoduced, stored in a rdfrfcrtz/ system, or transmitted, in any or .',Vtz'rly means, without the yrfor permsion 2'A; zvriting t)./ Oxford Ublbcrlf.y Press. Within the UK, exceptions are allouled p respect of any fair dealing .Jorthe Azlr/t?ycoj research or private study, or crl'li'cl'lpl or reviezv, as permitted under the Co/wrr/, Desns and Patents Act, zq'l', or fr the case V veprographic reproduction in accordance zcff/l the terms V the licences issued by the Copyright Licensing Agency. A'r/tzpzrc: concerntg rcyrotucfpn outside these terms and in other countries should be sent to the Rights Depavtment, Oxjord University Press, at the address above ./brv
F/7l'.book f.ssold subject to the condition that it shall not, by zt?tz.'p oj trade or otherwe, be lent, re-sold, Itived out or otheyulise circulated without the publisher's prior consent s any formof binding (?r covey other than that l'A; which 2'J is published and xithout a similar condition i'ac/z/kg this condhion being imposed on the subsequent purchaser srzk./z Librar.y Ctzftz/twlfag in Publication Data Data available Library
of Congress Cataloging in Publication Data
Data available ISBN 0-19-877402-8 ISBN 0-19-877403-6 ('Pbk) 1 3 5 7 9 10 8 6 4 2 Printed in Great Britain on acid-jkee yl/cr by Biddles Ltd., Guildjord and King
's
Lynn
For Lynette, Julia, and Nicola.
what in me is dark Illumine, what is 1ow raise and support, That, to the height of this great argument, 1 may assert Eternal Providence, And justify the ways of God to men.'
& .
.
.
Paradise f.,t?l/, Book 1, 16-20
Conients Preface
xiii
Mathematical
Symbols and Abbreviations
AXA
Part 1: Mathematics 1. Sets 1.1 1.2 1.3 1.4 1.5 1.6
and Numbers Basic Set Theory Countable Sets The Real Continuum Sequences of Sets Classes of Subsets Sigma Fields
3 8 10 12 13 15
2. Limits and Continuity 2.1 The Topology of the Real Line 2.2 Sequences and Limits 2.3 Functions and Continuity 2.4 Vector Sequences and Functions 2.5 Sequences of Functions 2.6 Summability and Order Relations 2.7 Arrays
20 23 27 29 30 31 33
3. Measure 3.1 Measure Spaces 3.2 The Extension Theorem 3.3 Non-measurability 3.4 Ptoduct Spaces 3.5 Measurable Transfcymations 3.6 Borel Functions
36 40 46 48 50 55
4. lntegration 4.1 Construction of the Integral 4.2 Properties of the lntegral 4.3 Product Measure and Multipl 4 4 The Radon-Nikodym
57 61 64 69
lntepals
'theorem
.''' -' y'
-'
7*
(qd qd 5. Metric
Spaces and Metris Distances 1 5. ) xjj 5.2 Separability and tjmptttt t(((...'.! j y.) .. . r), 5.3 Exampls ' ..)
.....
'
.
...
...
E. .
.........y
.. y. . . . . . E)rt)yy),,:yy't,y:r(,q)'yt 2 .,
i.; ! E E. ... . . . .. ... . ... . q.
. r
. j.
. .. q .
.
...
75 78 82
Contents
5.4 Mappings on Metric Spaces 5.5 Function Spaces
84 87
6. Topology 6.1 Topological Spaces 6.2 Countability and Compactness 6.3 Separation Properties 6.4 Weak Topologies 6.5 The Topolpgy of Product Spaces 6.6 Embedding and Metrization
93 94 97 101 102 105
Part 11: Probability 7. Probability Spaces 7. 1 Probability Measures 7.2 Conditional Probability 7.3 Independence 7.4 Product Spaces
111 113 114 115
8. Random Variables 8. 1 Measures on the Line 8.2 Distribution Functions 8.3 Examples 8.4 Multivariate Distributions 8.5 Independent Random Variables
117 117 122 124 126
9. Expectations 9. 1 Averages and Integrals 9.2 Expectations of Functions of X 9.3 Theorems for the Probabilist's Toolbox 9.4 Multivariate Distributions 9.5 More Theorems for the Toolbox 9.6 Random Variables Depending on a Parameter
128 130 132 135 137 140
10. Conditioning 10.1 Conditioning in Product Measures 10.2 Conditioning on a Sigma Field 10.3 Conditional Expectations 10.4 Some Theorems on Conditional Expectations 10.5 Relationships between Subfields 10.6 Conditional Distributions
143 145 147 149 154 157
11. Characteristic Functions 11.1 The Distribution of Sums of Random Variables 11.2 Complex Numbers
161 162
Contents
11.3 The Theory of Characteristic Functions 11.4 The Inversion Theorem 11.5 The Conditional Characteristic Function
f;r
164 168 171
Part 111:Theory of Stochastic Processes 12. Stochastic Processes 12.1 Basic Ideas and Terminology 12.2 Convergence of Stochastic Sequences 12.3 The Probability Model 12.4 The Consistency Theorem 12.5 Uniform and Limiting Properties 12.6 Unifonn lntegrability
177 178 179 183 186 188
13. Dependence 13.1 Shift Transformations 13.2 Independence and Stationarity 13.3 Invariant Events 13.4 Ergodicity and Mixing 13.5 Subfields and Regularity 13.6 Strong and Uniform Mixing
191 192 195 199 203 206
14. Mixing 14.1 Mixing Sequences of Random Variables 14.2 Mixing Inequalities 14.3 Mixing in Linear Processes 14.4 Sufficient Conditions for Strong and Uniform Mixing
209 211 215 219
15. Martingales 15.1 Sequential Conditioning 15.2 Extensions of the Martingale Concept 15.3 Martingale Convergence 15.4 Convergence and the Conditional Variances 15.5 Martingale Inequalities
229 232 235 238 240
16. Mixingales 16.1 Definition and Examples 16.2 Telescoping Sum Representations 16.3 Maximal Inequalities 16.4 Uniform Square-integrability
247 249 252 257
17. Near-Epoch Dependence 17.1 Definition and zxamples 17.2 Near-Epoc Dependence and Mixingales
261 264
Contents
17.3 Near-Epoch Dependence and Transformations 17.4 Approximability
267 273
Part lV: The Law of Large Numbers 18. Stochastic Convergence 18.1 Almost Sure Convergence 18.2 Convergence in Probability 18.3 Transformations and Convergence 18.4 Convergence in Lp Norm 18.5 Examples 18.6 Laws of Large Numbers
281 284 285 287 288 289
19. Convergence in fp-Norm 19.1 Weak Laws by Mean-square Convergence 19.2 Almost Sure Convergence by the Method of Subsequences 19.3 A Martingale Weak Law 19.4 A Mixingale Weak Law 19.5 Approximable Processes
293 295 298 302 304
20. The 20. 1 20.2 20.3 20.4 20.5 20.6
306 311 313 316 318 323
Strong Law of Large Numbers Technical Tricks for Proving LLNS The Case of Independence Martingale Strong Laws Conditional Variances and Random Weighting Two Strong Laws for Mixingales Near-epoch Dependent and Mixing Processes
21. Uniform Stochastic Convergence 21.1 Stochasti Functions on a Parameter Space 21.2 Pointwise and Unifonn Stochastic Convergence 21.3 Stochastic Equicontinuity 21.4 Generic Uniform Convergence 21.5 Unifonu Laws of Large Numbers
Part V: The Central Lint
327 330 335 337 340
Theorem
22. Weak Convergence of Distributions 22.1 Basic Concepts 22.2 The Skorokhod Representation Theorem 22.3 Weak Convergence and Transformations 22.4 Convergence of Moments and Characteristic Functions 22.5 Criteria for Weak Convergence 22.6 Convergence of Random Sums
347 350 355 357 359 361
Contents
23. The 23. 1 23.2 23.3 23.4 24.
CLTS
24.1 24.2 24.3 24.4 24.5
11
Classical Central Limit Theorem The i.i.d. Case lndependent Heterogeneous Sequences Feller's Theorem and Asymptotic Negligibility The Case of Trending Variances
364 368 373 377
for Dependent Processes A General Convergence Theorem The Martingale Case Stationary Ergodic Sequences The CLT for NED Functions of Mixing Processes Proving the CLT by the Bernstein Blocking Method
380 383 385 386 391
25. Some Extensions 25.1 The CLT with Estimated Normalization 25.2 The CLT with Random Norming 25.3 The Multivariate CLT 25.4 En'or Estimation
399 403 405 407
Pa14 V1: The Functional Central Limit Theorem 26. Weak Convergence in Metric Spaces 26.1 Probability Measures on a Metric Space 26.2 Measures and Expectations 26.3 Weak Convergence 26.4 Metrizing the Space of Measures 26.5 Tightness and Convergence 26.6 Skorokhod's Representation
413 416 418 422 427 431
27. Weak Convergence in a Function Space 27.1 Measures on Function Spaces 27.2 The Space C 11.3 Measures on C 27.4 Brownian Motion 27.5 Weak Convergence on C 27.6 The Functional Central Limit Theorem 27.7 The Multivariate Case
434 437 440 442 447 449 453
28. Cadlag Functions 28.1 The Space D 28.2 Metrizing D 28.3 Billingsley's Vetric 28.4 Measures on D
456 459 461 465
Contents
Xlt
28.5 Prokhorov's Metric 28.6 Compactness and Tightness in D 29.
FCLTS
29.1 29.2 29.3 29.4 29.5
for Dependent Variables The Distribution of Continuous Functions on D Asymptotic Independence The FCLT for NED Functions of Mixing Processes Transfonned Brownian Motion The Multivariate Case
467 469 474 479 481 485 490
30. Weak Convergence to Stochastic Integrals 30.1 Weak Limit Results for Random Functionals 30.2 Stochastic Processes in Continuous Time 30.3 Stochastic Integrals 30.4 Convergence to Stochastic Integrals
496 500 503 509
Notes
517
References
519
lndex
Preface
Recent years have seen a marked increase in the mathematical sophistication of econometricresearch. While thetheory of linearparametric models which fonns the backbone of the subject makes an extensive and clever use of matrix algebra, the statistical prerequisites of this theory are comparatively simple. But now that these models are pretty thoroughly understood, research is concentrated increasingly on the less tractable questions, such as nonlinear and nonparametric estimation and nonstationary data generation processes. The standard econometlics texts are no longer an adequate guide to this new technical literature, and a sound understanding of the probabilistic foundations of the subject is becoming less and less of a luxury. The asymptotic theory traditionally taught to students of econometrics is founded on a small body of classical limit theorems, such as Khinchine's weak law of large numbers and the Lindeberg-Lvy central limit theorem, relevant to the stationary and independent data case. To deal with linear stochastic difference equations, appeal can be made to the results of Mann and Wald (1943a),but even these are rooted in the assumption of independent and identically distributed disturbances. This foundation has become increasingly inadequate to sustain the expanding editke of econometric inference techniques, and recent years have seen a systematic attempt to construct a less restrictive limit theory. Hall and Heyde' s Martingale Limit Fdt??'.y and its Application (1980)is an important landmark, as are a series of papers by econometricians including among others Halbert White, Ronald Gallant, Donald Andrews, and Herman Bierens. This work introduced to the econometrics profession pioneering research into limit theot'y under dependence, done in the preceding decades by probabilists such as J. L. Doob, 1. A. Ibragimov, Patrick Billingsley, Robert Serfling, Murray Rosenblatt, and Donald McLeish. These ltter authors devised various concepts of limited dependence for general nonstationary time series. The concept of a martingale has a long history in probability, but it was primarily Doob's Stochastic Processes (1953)that brought it to prominence as a tool of limit theory. Martingale processes behave like the wealth of a gambler who undertakes a succession of fair bets; the differences of a martingale (the net winnings at each step) are unpredictable from lagged information. Powerful limit theorems are available for martingale difference sequences involving no further restrictions on the dependence of the process. Ibragimov and Rosenblatt respectively defined strong mixing and unbrm mixing as charactermemory' or independence at long range. McLeish defined the izations of notion of a mixingale, the asymptotic counterpart of a martingale difference, becomingunpredictable m steps aheadaspz becomeslarge. This is aweakerproperty than mixing because it involves only low-order moments of the distribution, but mixingales oossess most of those attributes of mixine orocesses needed to make
,
Preface
limit theorems work. Very important from the econometrician's point of view is the property dubbed by Gallant and White (1988)near-epoch dependence from a phrase in one of McLeish's papers, although the idea itself goes back to
Billingsley (1968)andlbragimovtlg6z). Themixingproperty may notbepreserved by transformations of sequences involving an intinite number of lags, but nearepoch dependence is acondition under whichthe outputs of adynamic econometric model can be shown, given some further conditions, to be mixingales when the inputs are mixing. Applications of these results are increasingly in evidence in the econometric literature', Gallant and White's monograph provides an excellent survey of the possibilities. Limit theorems impose restzictions on the amount of dependence between sequence coordinates, and on their marginal distributions. Typically, the probability of outliers must be controlled by requiring the existence of higher-order moments, but there are almost always trade-offs between dependence and moment restrictions, allowing one to buy more of one at the price of less of the other. The fun of proving limit theorems has been to see how far out the envelope of sufficient conditions can be stretched, in one direction or another. To complicate matters, one can get results both by putting limits on the rate of approach to independence (the rate of mixing), and by limiting the type of dependence (the martingale approach), as well as by combining both types of constraint (the mixingale approach). Theresults now available areremarkably powerful,judged by the yardstick of the classical theory. Proofs of necessity are elusive and the limits to the envelope are not yet known with certainty, but they probably lie not too far beyond the currently charted points. Perhaps the major development in time-series econometrics in the 1980s has been the theory of cointegration, and dealing with the distributions of estimators when time series are generated by unit root processes also requires a new type of limit theory. The essential extra ingredient of this theory is the functionalcentral limit theorem (FCLT). The proof of these weak convergence results calls for a limit theory for the space of functions, which throws up some interesting problems which have no connterpart in ordinary probability. These ideas were pioneered by Russian probabilists in the 1950s, notably A. V. Skorokhod and Yu. V. Prokhorov. lt turns out that FCLTS hold under properties generally similar to those for the ordinary CLT (thoughwith a crucial difference), and they can be analysed with the same kind of tools, imposing limitations on dependence and outliers. The probabilistic literature wlch deals with issues of this kind has been seen as accessible to practising econometricians only with difticulty. Few concessions are made to the nonspecialist, and the concerns of probabilists, statisticians, and econometlicians are frequently different. Textbooks on stochastic processes (Cox and Miller 1965 is a distinguished example) often give prominence to topics would regard as fairly specialized (e.g. Markov chains, that econometricians processes in continuous time), while the treatment of important issues like nonstationarity gets tucked away under the heading of advanced or optional material if not omitted altogether. Probability texts are written for students of
mathemtics
and ag&ume
n fnmilinritv
wifh
fln/R
fnllrlnr-
nf
tlao
t.lxlor':
'lqne
Preface
econometricians may lack. The intellectual investmnt required is one that students and practitioners are often, quite reasonably, disinclined to make. It is with issues of this sort in mind that the present book has been written.
The t'irstobjective has been to provide a coherent and unified account of modern asymptotic theory, which can function as both a course text, and as a work of reference. The second has been to provide a grounding in the requisite mathematics and probability theory, making the treatment sufticiently self-contained that even readers with limited mathematical training might make use of it. This is not to say that the material is elementary. Even when the mathematics is mastered, the reasoning can be intricate and demand a degree of patience to absorb. Profs for nearly all the results are provided, but readers should never hesitate to pass over these when they impede progress. The book is also intended to be useful as a reference for students and researchers who only wish to know basic things, like the meaning of technical tenns, and the variety of limit results available. But, that said, it will not have succeeded in its aim unless the reader is sometimes if for no better stimulated to gain a deeper understanding of the material reason, because this is a theory abounding in mathematical elegance, and technical ingenuity which is often dazzling. -
Outline of the Work Part I covers almost a11the mathematics used subsequently. Calculus and matrix algebra are not treated, but in any case there is little of either in the book. Most readers should probably begin by reading Chapters 1 and 2, and perhaps the first sections only of Chapters 3 and 4, noting the definitions and examples but skipping a11 but the briefest proofs initially. These chapters contain some difticult material, which does not a11need to be mastered immediately. Chapters 5 and 6 are strictly required only for Chapter 21 and Part VI, and should be passed over on tirst reading. Nearly everything needed to read the probability literature is covered in these chapters, with perhaps one notable exception the theory of normed spaces. Some treatments in probability use a Hilbel't space framework, but it can be pvoided. The number of pplications exploiting this approach seemed currently too small to justify the added technical overhead, although future developments may require this judgement to be revised. Pm 11 covers what for many readers will be more familiar territory. Chapters 7, 8, and 9 contain essential background t be skimmed or studied in more depth, as appropriate. lt is the collections of inequalities in j9.3 and j9.5 that we will have the most call to refer to subsequently. The content of Chapter 10 is probably less familiar, and is very important. Most readers will want to study this chapter carefully sooner or later. Chapter 11 can be passed over initially, but is needed in conjunction with Part V. In Part 11I the main business of the work begins. Chapter 12 gives an introduction to the main concepts arising in the study of stochastic sequences. Chapters 13 and 14 continue the discussion by reviewing concepts of dependence, and Chapters 15, 16, and 17 deal with s/ecializedclasses of sequence whoseproperties -
7771
Preface
make them amenable to the application of limit theorems. Nearly all readers will want to study Chapters 12, 13, and the earlier sections of 14 and 15 before going further, whereas Chapters 16 and 17 are rather technical and should probably be avoided until the context of these results is understood. In Parts IV and V we arrive at the study of the limit theorems themselves. The aim has been to contrast alternative ways of approaching these problems, and to present a general collection of results ranging from the elementary to the very general. Chapter 18 is devoted to fundamentals, and everyone should read this before going further. Chapter 19 compares classical techniques for proving laws of large nmbers, depending on the existence of second moments, with more modern methods. Although the concept of convergence in probability is adequate in many econometric applications, proofs of strong consistency of estimators are increasingly popular in the econometrics literature, and techniques for dependent processes are considered in Chapter 20. Uniform stochastic convergence is an essential cohcept in the study of econometric estimators, although it has only recently been systematically researched. Chapter 21 contains a synthesis of results that have appeared in print in the last year or two. Part V contrasts the classical central limit theorems for independent processes with the modern results for martingale differences and general dependent processes. Chapter 22 contains the essentials of weak convergence theory for random variables. The treatment is re%onably complete, although one neglected topic, to which much space is devoted in the probability literature, is convergence to stable laws for sequences with infinite variance. This material has found few applications in econometlics to date, although its omission is anothet judgement that may need to be revised in the future. Chapter 23 describes the classic CLTS for independent processes, and Chapter 24 treats modern techniques for dependent, heterogeneous processes. Pal't V1 deals with the functional central limit theorem and related convergence results, including convergence to limits that can be identified with stochastic integrals. A number of new mathematical challenges are presented by this theory, and readers who wish to tackle it seriously will probably want to go back and apply themselves to Chapters 5 and 6 first. Chapter 26 is both the hardest going and the least essential to subsequent developments. It deals with the theory of weak convergence on metric spaces at a greater level of generality than we strictly need, apd is the one section where topological arguments seriously intrude. Almost certa in ly one shoul go first to Chapter 27 ? referring back as needed for definitions and statements of the prerequisite theorems, and pursue the rationale for these results further only as interest dictates. Chapter 28 is likewise a technical prologue to Chapers 29 and 30, and might be skipped over at first reading. The meat of this pal't of the book is in these last two chapters. Results are given on the multivariate invariance principle for heterogeneous dependent processes, paralleling the central limit theorems of Chapter 24. A number of the results in the text are, to the author's knowledge, new. These include 14.13/14, 19.11, 20.18/19, 20.21, 24.6/7/14, 29.14/29.18, and 30.13/14, although some have now appeared in print elsewhere.
Preface
A37II
Further Reading There is a huge range of texts in print covering the relevant mathematics and probability, but the following titles were, for one reason or another, the most frequently consulted in the course of writing this book. T. M. Apostol's Mathematical Analysis (2nd edition) hits just the right note for the basic bread-andbutter results. For more advanced material, Dieudonn's Foundations ofModern Analysis and Royden's Real Analysis are well-known references, the latter being the more user-friendly although the treatment is often fairly concise. Halmos's classic Measure Fdt?ry and Kingman and Taylor's Introduction to Measure and Probability are worth having access to. Willard's General Topology is a clear and well-organized text to put alongside Kelley's classic of the same name. Halmos's Naive Set Fct??'y is a slim volume whose main content falls outside our sphere of interest, but is a good read in its own right. Strongly recommended is Borowski and Borwein's Collins Reference Dictionaty ofMathematics; one can learn more about mathematics in less time by browsing in this little book, and following up the cross references, than by any other method 1 can think of. For a stimulating introduction to metric spaces see MichaelBarnsley's popularFmc/l/. Everywhere. For further reading on probability, one might begin by browsing the slim volume that stmed the whole thing off, Kolmogorov's Foundations of the F/ldtp?-yof Probability. Then, Billingsley's Probability and Measure is an inspiration, both authoritative and highly readable. Breiman's Probability has a refreshingly informal style, and just the right emphasis. Chung's A Course in Probability The value of the Fct??'y is idiosyncratic in parts, but strongly recommended. Probability and Fwr.y Love's edition) Feller's An Introdclassic texts, (4th uction to Probability Fct??-y and its Applications (3rd edition) is self-evident, although these are dense and detailed books that can take a little time and patience to get into, and are chiefly for reference. Cramr's Mathetnatical Methods of Statistics is now old-fashioned, but still useful. Two more recent titles are Shiryayev's Probability, and R. M. Dudley's tough but stimulating Real Analysis and Probqbility. Of the more specialized monographs on stochastic convergence, the following titles (in order of publiation date) are a11 important: Doob, Stochastic The fzzw- of fzzrpd Numbers', Parthasarathy, Probability Processes; Rvsz, Measures on Metric Spaces', Billingsley, Convergence of Probability Measures', losifescu and Theodorescu, Random Processes and Learning; Ibragimov and Linnik, Independent and Stationaly Sequences of Random Variables', Stout, Almost Sure Convergence; Lukacs, Stochastic Convergence; Hall and Heyde, Martingale Limit F/lct??'yand its Application; Pollard, Convetgence ofstochastic Processes; Eberlein and Taqqu (eds.),Dependence in Probability and Statistics. Doob is the founding father of the subject, and his book its O1d Testament. Of the rest, Billingsley's is the most original and influential. Ibragimov and Linnik's essential monograph is now, alas, hard to obtain. The importance of Hall and Heyde was mentioned above. Pollard's book takes up the weak convergence
Preface
story more or less where Billingsley leaves off, and much of the material complements the coverage of the present volume. The Eberlein-rraqqu collection contains up-to-date accounts of mixing theory, covering some related topics outside the range of the present work. The literature on Brownian motion and stochastic integration is extensive, butKaratzas and Shreve's BrownianMotion and Stochastic Calculus is arecent and comprehensive source forreference, and Kopp's Martingales and Stochastic Integrals was found useful at several points. These items receive an individual mention by virtue of being between hard covers. References to the journal literature will be given in context, but it is worth mentioningthatthe fourpapers by DonaldMctaeish, appearing between 1974 and 1977, form possibly the most influential single contribution to our subject. Finally, titles dealing with applications and related contributions include Serfling, Approximation Theorems ofMathentatical Statistics', White, Asymptotic Theotyfor Econometricians', Gallant, Nonlinear Statistical Methods; Gallant and White, A Un6ed Fwr.y of Estimation and Inference for Nonlinear Dynamic Models, Amemiya, Advanced Econometrics. Al1of these are highly recommended for forming a view of what stochastic limit theory is for, and why it matters.
Acknowledgements The idea for this book originated in 1987, in the course of writing a chapter of mathmatical and statistical prerequisites for a projected textbook of econometric theory. The initial, vel'y tentative draft was completed during a stay at the University of California (San Diego) Depmment of Economics in 1988, whose hospitality is gratefully acknowledged. lt has grown a great deal since then, and competing academic commitments as getting itnishedhasinvolvedas%gglewith well as the demands of family life. My family deserve special thanks for their
forbearance.
My colleaguepeterRobinson
has been agreat source of encouragement and help,
and has commented on various drafts. Other people who have read portions of the manuscript and provided invaluable feedback, not least in pointing out my errors, include Getullio Silveira, Robertde Jong, and especially Elizabeth Boardman, who took immense trouble to help me lick the chapters on mathematics into shape. l am also most grateful to Don Andrews, G raham Brightwell, Sgren Johansen, Donald
McLeish, Peter Phillips, Hal White, and a number of anonymous referees for helpful conversations, comments and advice. None of these people is responsible for the various flaws that doubtless remain. The book was written using the ChiWriter 4 technical word processor, and after conversion to Postscript format was produced as camera-ready copy on a HewlettPackard Laserlet 451 printer, direct from the original iiles.I must particularly thank Cay Horstmann, of Horstmann Software Design Corp., for his technical assistance with this task. Lrndon, June 1994
Mathematical Symbols and Abbreviations In the text, the symbol (:a is used to terminate examples and definitions, and also theorems and lemmas unless the proof follows directly. The symbol w tenninates proofs. References to numbered expressions are enclosed in parentheses. References to numbered theorems, examples etc. are given in bold face. References to chapter sections are preceded by j. In statements of theorems, roman numbers (i), (ii), (iii),...are used to indicate the parts of a multi-part result. Lower case letters (a), (b), (c),... are used to itemize the assumptions or conditions specified in a theorem, and also the components of a detinition. The page numbers below refer to fuller definitions or examples of use, as aPpropriate.
absolute value fw-norm Euclidean norm;
1,i -+ D
&
-, / S, 2 <, <(
>
i 1x(.) a.e.
AR
ARMA
a.s., a.s.gpt1
also fineness (of a q/rtition) weak convergence (of measures); also implication monotone convergence convergence almost sure convergence convergence in distribution convergence in Lp nol'm convergence in probability mapping, function composition of mappings equivalence in order of magnitude (of sequencesl; also equiyalence in distribution (of r.v.s) addition modulo 1 set difference partial ordering, inequality strict ordering, strict inequality order of magnitude inequality (sequencesl; also absolutely continuous (measures) mutually singular (of measures) indicator function almost everywhere autoregressive process autoregressive moving average process almost surely, (withresp. to p.m. g)
Symbols AC
I (A)40, Gm
dO V Bnbp) CLT
ch-f. c-d.f. C'p,1j C, Q c D
i Z (n4 dxny) ogc,jy D A :A 6
StlP E.4 E. Ix) E. 1V) eSS
H j+
J-
1
FCLT F(.)
9x(.) 9,,, iff inf
i-i-d. i-o. in pr.
LIE LII
,
lim limsup, lim liminf, 1im Ln) AP-NED MA
m.4
complement of A closure of A intedor of A strong mixing coefficient aleph-nought (cardinalityof (N) dfor every' Binomial distribution with parameters n and p Borel field central limit theorem characteristic function cumulative distribution function continuous functions on the unit interval
set containment strict containment chi-squared distribution with n degreeg of freedom distance between x and y cadlag functions on the unit interval dyadic rationals symmetric difference boundary of A set membership essential supremum expectation conditional expectation (on variable x) conditional expectation (on c-field 5')
Lebesgue measure
3
Symbols m.d. m.g.f. m.s.
martingale difference moment-generating function mean square space of measures
N(jt,c2)
Gaussian distribution with mean jt and variance
s(N()
natural numbers
rh, O
m h #!
0(.) t?(.) Op.) ov.4 p-d.f. P.m.
#(.)
#(.IA)
#(. IN)
H,
1-1
7rj(.)
(D
F.V.
/?(0,1j .f R y R R+
F F+
R& S.e. S.S.e.
SLLN suP Sxvz)
!
Jn
c(C)
c(m fdx
K QJ
(0l
intersection minimum of m and n fBig Oh' order of magnitude relation ,
Oh', strict order of magnitude relation stochastic order relation strict stochastic order relation null set probability density function Ktaittle
probability measure probability conditional probability (on event A) conditional probability (on c-field N) product of numbers', also partition of an interval product measure coordinate projection rational numbers random variable real valued functions on g0,11 relation real line non-negative reals extended real line, R t..p (-x,+x1
R+kJ
f+x)
n-dimensional Euclidean space stochastic equicontinuity strong stochastic equicontinuity strong law of large numbers suptemum e-neighbourhood, sphere sum of random variables variance of Sn c-tield generated by collection C c-field generated by r.v. X
J/Wpo fdF
Lebesgue integral Lebesgue-stieltjes integral
fdp z, E
expected value (integralwith resp. to p.m.) sum
ui
(y2
Symbols
Mlt
shift transformation union unifonn distribution on interval (tz,:4 union of c-fields maximum of m and n weak 1aw of lm'ge numbers with probability 1 sample mean of sequence (XrJZ
F
U
kJ,
Uka,bj
w, V
m V
n
MXLN
sv.p.1
Xn
x, X
Cartesian product
Z
ll l 1T,( ) .
'>O
.
.
ll ll (x1 I,:(l a,b) .
(f1,T)
(f,',p,)
(f1,F,F) (f,T;#) (S,* (X,*
c-field of product sets integers set designation', also sequence, array infinite sequences array largest integer S closed interval bounded by a,b open interval bounded by a,b measurable space measure space complete measure space probability space metric space topological space .t
Common usages A,#,C,D,... X,F,Z,... A',F.Z,...
J,#,,... :,,q
B,5'.I
2,C,O,F',.. 9,V,J't,-.. S,5-,X,... 1,,Y,-
-
d.p T
-
sets random variables random vectors functions positive constants bounding constants collections of subsets c-fields spaces measures metrics topology
1
MATHEMATICS
Sets and Numbers
1. 1 Basic Set Theory A set is any specified collection of objects. ln this book the objects in question are often numbers, but they may also be functions, or other sets, or indeed wholly arbitrary, to be determined by the context in which the theory is applied. ln any analysis there is a set which defines the universe of discourse, containing al1 the objects under consideration, and in what follows, sets denoted A, B etc., are subsets of a set X, with generic element x. q A meaning Set membership is denoted by the symbol belongs to the set writes and A #. The usual way B have the same elements, one A' To show sets A descriptive enclosed define is by in braces, so that for statement a to a set example A tx:x e #) defines membership of A in terms of membership of B, and is an alternative way of writingA #. Another way to denote set membership is by labels. If a set has n elements one can write A fxj, i 1,...,n 1, but any set of labels will do. The statement A lau, (x e C) says that A is the set of elements bearing labels a contained in another set C, called the index set for A. The labels (indices)need not be numbers, and can be any convenient objects at tends to be preferred in all. Sets whose elements are sets (theword this context) are denoted by upper-case script characters. A e C denotes that the set A is in the collection C, or using indices one could write C (Aa: a e Cl B is called a subset of A, written B c A, if al1 the elements of B are also elements of A. If B is a proper subset of A, ruling out B A, the relation is written B c A. The union of A and B is the set whose elements belong to either or both sets, written A k.p #. The union of a collection C, the set of elements belongr ing to one or more A e C, is denoted UA es,A, or, alternatively, one can write UaecAz for the union of the collection fAa: (x l C). The intersection of A and B is the set of elements belonging to both, written A rn #. The intersection of a collection C is the set of elements commop to a1l the sets in C, written f-tlsc/t or OascAa. ln pmicular, the union and intersection of (Aj ,A2, Au ) are written U7=1Ajand O7=1Aj.When the index set is implicit or unspecified we may write just U(xAa, OfAf or similar. The dterence of sets A and #, written A B or by some authors A #, is the of two sets set of elements belonging to A but not to B. The symmetric tft//rcrcc (A B) k.p (B -A). X-A is the complement of A in X, also denoted Ac is A A# when X is understood, and we have the general result that A B 4 rn Bc. The null set (or empty set) is O M, the set with no elements. Sets with no elements in common (havingempty intersection) are called disjoint. A partition of a set is a 61
'
<x
.x
,
=
.
=
=
=
=
=
dcollection'
=
.
=
...,
-
=
-
-
=
=
Mathematics
4
collection of disjoint subsets whose union is the set, such that each of its elements belongs to one and only one member of the collection. Here are the basic rules of set algebra. Unions and intersections obey commutative, associative and distributive laws: A t.p B
=
B QJ A,
A f'n B
=
B rn A,
(A t..p B4 tp C
=
A t.p (B
(A fa #) rn C
=
A f-h CBf'A C),
(1.4)
=
(A fa B4 tp (A rn C),
(1.5)
A fa
(# tp C)
?
C),
(A tp #) r'h(A kp C. There are also rules relating to complements known as de Morgan's laws: A tp
(# fa
C)
=
(A tp B4c
=
(A rhB)c
=
AC
fa
Bc
Ac t.p Bc.
(1.8)
Venn diagrams, illustrated in Fig. 1.1, are a useful device for clarifying relationships between subsets.
Fig. The distributive and de Morgan laws extend to general collections,
1.1 Theorem Let C be a collection of sets, and B a set. Then
(i) (ii)
UA
rn B
OA
tp B
=
AGC
U (A
=
AGC
f-lzt
AeC
#),
r)
(A kp B),
dgC C
(iii)
f-h
AGN
zc
(-Jic,
AEC
as follows.
Sets and Numbers C
(iv)
U
A
OAc. u
=
AeC
AeC
The Cartesian product of two sets A and #, written A x B, is the set of a1l possible ordered pairs of elements, the first taken from A and the second from #; we write A x B (@,y): e A, y G #) For a collection of n sets the Cartesian product is the set of al1 the n-tuples (orderedsets of n elements, with the th element drawn from A/), and is written .:'
=
.
n
Af
.(
=
(x1x2, ,
/=1
...,
xi e Af
Ak):
,
1-
1,...,n )
=
(1.9)
.
If one of the factor sets Af is empty, X2=1Xfis also empty. Product sets are important in a variety of different contexts in mathematics. Some of these are readily appreciated', for example, sets whose elements are s-vectors of real numbers are products of copies of the real line (see j1.3). But prodct sets are also central to the mathematical formalization of the notion of relationship between set elements. Thus: a relation R on a set A is any subset of A xA. If @,y)e #, we usually write x#y. R is said to be reflexive iff xRx, symmetric iff xR y implies yRx, antisymmetric iff xRy and y Rx implies x y, transitive iff xRy and yRz implies xR z, each in where case the indicated condition holds for every x, y, and z e A, as the and only if means case may be. (Note: relation relation equivalence that is reflexive, symmetric, and transiis An a tive. Given an equivalence relation R on A, the equivalence class of an element x e A is the set Ex (y e A: xRy If Ex and Ey are the equivalence classes of elements x and y, then either Ex fa Ey 0, or Ex Fy. The equality relation y is the obvious example of an equivalence relation, but by no means the only one. A partial ordering is any relation that is reflexive, antisymmetric, and transitive. Partial orderings are usually denoted by the symbols f or k, with the understanding that k y is the same as y f x. To every partial ordering there corresponds a strict ordering, defined by the omission of the elements @,x)for a11 x e A. Strict orderings, usually denoted b: < or >, are not reflexive or antisymmetric, but thy are transitive. A set A is said to be linearly ordered by a partial ordering f if one of the relations x < y, x > y, and x y hold forr every pair @,y)e A xA. If there exist elements a e A and b G A such that a < x for al1 x e A, or x S b for all x e A, a and b are called respectively the smallest and largest elements of A. A linearly ordered set A is called well-orderd if every subset of A contains a smallest element. It is of course in sets whose elements are numbers thai the ordering concept is most familiar. Copsider two sets X and F, which can be thought of as representing the universal sets fr a pqir of related problems. The following bundle of definitions contains =
Kiff'
=
.)
.
=
.x
=
.x
=
=
Mathematcs
b
the basic ideas about relationships between the elements of such sets. A (or transformation or function) T: X
F->
tnapping
F
is a rule that associates each element of X with a unique element of F; in other words, for each a7 e X there exists a specified element y e F, denoted F@). X is called the domain of the mapping, and F the codomain. The set Gw
(@,y):
e X, y
.z'
=
is called the graph of T. For T(4)
is called the image of
,4
:4
e /t
Jq
under F, and for B
F -1(#)
=
(1.10)
Xx F
the set
? X,
(F@): x
=
W@))
=
fx: F(x)
e
(1 11)
F
.
1', the set
c
#J c
X
(1.12)
is called the inverse rrlzwe of B under F. The set T(m is called the range of F, and if F(m 1,the mapping is said to be from X onto F, and otherwise, into K If each y is the image of one and only one x e X, so that F(x1) Tx if and only the mapping is said to be one-to-one, or 1-1. if xl =
=
=
.n,
The notions of mapping and graph are really interchangeable, and it is permissable to say that the graph is the mapping, but it is convenient to keep a distinction in mind between the rule and the subset of Xx F which it generates. The term function is usually reserved for cases when the codomain is the set of real numbers (seej1.3). The term correspondence is used for a nlle connecting -1 elements of X to elements of F where the latter are not necessarily unique. T is a con'espondence, but not a mapping unless F is one-to-one. However, the term one-toone correspondence is often used specitically, in certain contexts that will arise below, to refer to a mapping that is both 1. 1 and onto. lf partial orderings are defined on both X and F, a mapping is called order-preserving if F(x1) G Tx iff On the other hand, if X is partially ordered by f, a 1- 1 mapping induces .;q S
.
1.2 Theorem (i) For a collection (Aa c aY), F UAa (:
(ii) for a collection (#a i F), T-' (iii) for #
i', F -1(sc) .
UF(Xa);
=
'
Ufa
=
UF-1(#a);
w-1(s)c. ,
(iv) for A i X, A g F -1(z(u)). #. u (v) for B g F, F(F -1(#)) ,
Here, T -1(B)c means X- F-1(#). Using de Morgan's laws, properties
(ii) and
Sets and Numbers
(iii) are easily extended to the inverse images of intersections and differences', for example, we may show that the inverse images of disjoint sets are also F(A)C F(AC), # in general. Pal'ts (iv) and (v) are disjoint. However, F- F(A) illustrated in Fig. 1.2, where X and i' both correspond to the real line, A and B are intervals of the line, and F is a function of a real variable. =
Fig. 1.2
When F is a 1-1 con-espon dene (1-1 and onto) so is F-1 These properties then hold symmetrically, and the inclusion relations of parts (iv)and (v) also become X and B c K lf Z is a third set, and equalities for a1l A .
U1 F
F--> Z
is a further mapping, UoF: X
F--
the composite
mapping
Z
takes each x e X to the element U(F(x)) e Z. UoT operates as a simple transform-1 from X to Z, and 1.2 applies to this case. For C c Z, (&J07) c' ation -1 =
-1
F
(&
(C)).
0- x E .
j
i
.
.
.
.
.
'
.
' .
.
' .
' .
' .
.
j
F(z4c )
1
+
'
.
)Y
X1Z
Fig. 1.3
1.3 Example lf X 0- x E is a product space, having as elements the ordered pairs .x = (0,(), the mapping F: 0- x E i- E =
,
Mathematics
8
defined by F(0,() (, is called the projection tnapping onto E. The projection of 0- x E onto E (respectively,0-) is the set consisting of the second a set A (resp., first) members of each pair in A. On the other hand, for a set B e E, 0- x # It is a useful exercise to verify 1.2 for this case, and also to F -1 B4 T(AC) in general. In Fig. 1.3, 0- and E are line segments and check that F(A # F(A)C is the union of the indicated line 0- x E is a rectangle in the plane. Here, whereas T(A9 E. u segments, =
=
.
=
The number of elements contained in a set is called the cardinality or cardinal in this context is not a primitive one, number of the set. The notion of but can be reduced to fundamentals by what is called the principle. A set A is said to be equipotent with a set B if there exists a 1-1 correspondence connectingA and #. Think in terms of taking an element from each set and placing the pair in a pigeon-hole. Equipotency means that such a procedure can never exhust one set before the other. Now, think of the number 0 as being just a name for the null set, 0. Let the number 1 be the name for the set that has a single element, the number 0. Let 2 denote the set whoseelements are the numbers 0 and 1.And proceeding recursively, 1et n be the name for the set (0,...,n 1 1. Then, the statement that a set A has n elements, or has cardinal number n, can be interpreted to mean that A is equipotent with the set n. The set of natural numbers, denoted N, is the collection (n: n = 1,2,3,... 1. This collection is well ordered by the relation usually denoted <, where n S m actually means the same as n m under this definition of a number. %number'
dpigeon-hole'
-
1.2 Countable Sets Set theory is trivial when th: number of elements in the set is finite, but formalization becomes indispensable for dealing with sets having an intinite number of elements. The set of natural numbers (Nis a case in point. If n is a member so is n + 1, and this is true for every n. None the less, a cardinal number is formally assigned to iN, and is represented by the symbol Mtl When the elements of an infinite set can be put into a one-to-one correspondence with the natural numbers, the set is said to have cardinal number d(), but, more commonly, to be countable, or equivalently, denumerable. Countability of a set requires that a scheme can be devised for labelling each element with a unique element of (N.This imposes a well-ordering on the elements, such that there is a 'first' element labelled 1, and so on, although this ordering may have significance or be arbitrary, depending on the circumstances. It is the pigeon-hole principle that matters here, that each element has its own unique label. With infinite sets, everyday notions of size and quantity tend to break down. Augmenting the natural numbers by the number 0 defines the set (Ntl (0,1,2,3,... 1. The commonplace observation that (Ntlhas more' element than ENis contradicted by the fact that INand (Nnare equipotent (labeln 1 e Ntlby n e IN).Still more suprisingly, the set of even numbers, E = (2n, n e IN) also has an obvious labelling scheme demonstrating equipotency with EN.The nave idea that there are (Kaleph-nought').
=
-
,
df.xxzlr.o
eac
meanxz'
xlomontc
ln
N
re
1n IF- ;t
Axzithnllf
lfatyinnl fnllnclntlnn
Sets and Numbers
9
infinite subset A of ENhas a natural well-ordering and is equipotent with EN itself, the label of an element x e A being the cardinal number of the set (y e A:
A' V A l than EN consider the integers, I Turning to sets apparently 1,2,... ), the set containing the signed whole numbers and zero. These ( are linearly ordered although not well ordered. They can, however, be paired with scheme: the natural numbers using the .
Elarger'
=
,
...,-1,0,
Ezig-zag'
(1,0),(2,1),(3,-1), (4,2),...,(n, (n/21(-1)''),...,
where gxJdenotes the largest whole number below x. Thus, N and I are equipotent.
Then there are the rational (I)
=
f.x: .x
numbers.
=
alb, a e 1, b e 1, b # 0)
(1.13)
.
We can also show the following.
1.4 Theorem Q is a countable set. Proof We construct a 1-l correspondence between I x J and N. A 1-l correspondence between I x I and I x: is obtained by the methodjust used to show I countable, and one between I x: and ENx: is got by the same method. Then note that the number 2 3b e INis uniquely associated with each pair a,b4 e INxN ; the rule for recovering a and b from 2'3b is g et a as the number of divisions by 2 required to get an odd number, and the number so obtained is 3 ' The collection f 2 '3b : a e ENb G EN) c ENis equipotent with ENitself as shown in the preceding paragraph. The composition of a1l these mappings is the desired correspondence. w <
.
,
,
Generalizing this type of argument leads to the following fundamental result.
1.5 Theorem The union of a countable collection of countable sets is a countable set. n The concept of a sequence is fundamental to a11 the topics in this book. A sequence can be thought of as a mapping whose domain is a wellrordered countable set, the index set. Sinc there is always an order-preserving 1-1 mapping from IN to the index set, there is usually no loss of generality in considering the composite mapping and thinking of N itself as the domain. Another way to characterize a sequence is as the graph of the mapping, that is, a countable collection of pairs having the ordering conferred on it by the elements of the domain. The ranges of the sequences we consider below typically contain either sets or real numbers', the associated theory for these cases is to be found respectively in j1.4 and j2.2. The term sequence may also be applied to mappings having I or another linearly ordered set as index set. This usage broadens the notion, since while such sets can be re-indexed by N (see above) this cannot be done while preserving the original ordering.
Mathematics
10 1.3 The Real Continuum
The real-number continuum R is such a complex object that no single statement of definition can do itjustice. One can emphasize the ordinal and arithmetic properties of the reals, or their geometrical interpretation as the distances of points on a line from the origin (the point zero). But from a set-theoretic point of view, the essentials are captured by defining R as the set of countably infinite sequences of decimal digits, having a decimal point inserted at exactly one position in the sequence, and possibly preceded by a minus sign. Thus, the real number x can be written in the form X
'x
,,,(x)1(/)7-)j(.x)10-f
=
(1. 14)
i=1
where the sequence (J1(.x), J2@),...) consists of decimal digits (elementsof the set (0,1,...,9 J), px) e (Ntldenotes the position of the decimal point in the +1 if k 0, and otherwise (the string (the decimal exponent), and mlx) all of the decimal expansion sign). When dix) 0 for but a finite number terms, of x is said to terminate and the final Os are conventionally omitted from the representation. -1
.x
=
=
by (1.14) is not always unique, and there exists a 1-1 correspondence between elements of R and sequences (>,#,#1,#2,#a,... l only after eliminate arbitrary of leading zeros we must the latter are excluded. To certain 0. And since for example, 0.49999... (the stipulate that dj y: 0 unless p sequence of 9s not terminating) is the same number as 0.5, we always take the terminating representation of a number and exclude sequences having di 9 in al1 but a finite number of places. R is of course linearly ordered, and in terms of (1. 14) the ordering corresponds to the lexicographic ordering of the sequences The representation
of
.x
=
=
fmsmpnmdk,mdznmdL...l.
The choice of base 10 in the definition is of course merely conventional, and of the alternative possibilities, the most important is the binary (base2) represn-
tation,
x
=
>(x)
2d*J>7n(x)2-i f=l
,
(1.15)
where the bi are binary digits, either 0 or 1, and qx) is the binary exponent.
The integers have the representation in (1.14) with the strings terminating elements rationals of R, being those which either after px) digits. The are also terminate after a finite number of places, or else cycle repeatedly through a finite sequence of digits beyond a certain tinite point. The real numbers that are not rational are called irrational. The irrational numbers are overwhelmingly more numerous than the rationals, representing a higher order of infinity. The following is the famous argument of Cantor. ddiagonal'
1.6 Theorem The jet R is uncountable. Proof Assume a 1-1 correspondence between R and ENexists. Now construct a real
Sets and Numbers
number in the following way. Let the first digit be different from that of the real number labelled 1, the second digit be different from that of the real number labelled 2, and in general the nth digit be different from that of the real number labelled n, for every n. This number is different from every member of the labelled collection, and hence it has no label. Since this construction can be performed for any labelling scheme, the assumption is contradicted. w c, where c is the cardinal
number
of R The linear ordering on R is of interest to us chiefly for providing the basis for constnlcting the fundamental subsets of R, the intervals. The set A (.x:a < x < &) is called an open interval, since it does not contain the end points, whereas the interval # = (x: a f x S /71is said to be closed. Common notations are gJ,pj, asb), (tz,:j and Lanb) to denote closed, open and half-open intervals. A set containing just a single point a is called a singleton, written ftz) Unbounded defined by a single boundary point, are written intervals such as C = (x: a < (J,+x), (-x,:), and (J,+x), (-x,:j for the open and closed cases respectively, where the infinities +x and are the fictional (notelements of R) with > +x < for al1 R respective and An important example properties x the xe is the positive half-line (0,+x), which we will denote subsequently by R+.
We say that d ()
<
.
=
.
.x1,
Kpoints'
-=
.r
-x,
.
1.7 Theorem Every open intelwal is uncountable. Proof Let the interval in question be anb). lf a < b, there exists n k 0 such that the n + llth term of the sequence msmpntndjsmdzn...) in the expansion of (1.14) defining b exceeds that in the corresponding sequence for a, whereas the t'irst n digits of each sequence are the same. The elements of a,b4 are those reals whose expansions generate the same initial sequence, with the (n+ llth terms not exceeding that of b nor being exceeded by that of a. If a and b are distinct, n is finite. The result follows on applying the diagonal argument in 1.6 tb these expansions, beginning at position n + 2. w Other useful results concerning
R and its intervals include the follpwing.
1.8 Theorem The points of any open interval are equipotent
with R
.
Proof This might be proved by elaborating the argument of 1.7, but it is simpler just to exhibit a 1-1 mapping from R onto asb). For example, the function
y
=
a+ b
2
for x e R fulfils the requirement. 1 9 Theorem The reai plane R2 *
Proof In
+
(b
2(1
-
+
tzlx
(1.16)
IxI) .+
. =
R x R is equipotent with R
,
view of the last theorem, it suffices to show that the unit interval o Givep points x, y e g0,1J, define 1q (0, is equipotent with the unit square (0,11. the point z e (0,1j according to the decimal expansion in (1.14),by the rule
Mathematics
diz)
di-vblllx),i odd =
Jf/2@),
i
1,2,3,...
=
(1.17)
i even
In words, construct z by taking a digit from x and y alternately. Such a z exists for every pair x and y, and, given z, x and y can be uniquely recovered by setting
(1.18) This defines a 1-1 mapping from (0,112 onto gO 1j as required. w ,
,
This argument
can be extended from R 2 to Rn for any n e N.
1.10 Theorem
Every open intelwal contains a rational number.
,
Proof This is equivalent to the proposition that if x < y, there exists with x < r < y. First suppose x 0. Choose q as the smallest integer such that qy > qx + 1, and choose p as the smallest integer 1/(v qy. Then x < (,p 1)/ < y. For the case < 0 choose an integer n > < r < n +y, .x < r- n < y, where r is the rational satisfying n -x),
-x,
.x
-
.-x
above.
rational r
exceeding
exceeding and then found as
.
1.11 Corollary
Every collection
of disjoint open intervals is countable.
Proof Since each open interval contains a rational appearing in no other interval disjoint with it, a set of disjoint open intervals can be placed in 1-1 correspondence with a subset of the rationals. w The supremum of a set A c R, when it exists, is the smallest number y such that x S y for every x e A, written supA. The inpmum of A, when it exists, is the largest number y such that k y for every x e A, written inf A. These may or may a, and supgl,') not be elements of A. In particular, infgtz,hj inftc,#) suptc,:) b. Open intervals do notpossess largest or smallest elements. However, every subset of R which is bounded above (resp.below) has a supremum (resp. intimum). While unbounded sets in R lack suprema and/or infima, itis customary to called the extended real line. In V, every set has define the set V R t.p ( number or +x, and similarly, every set has an either real a tinite a supremum, infimum. The notation 2-+will also be used subsequently to denote R+ k.p (+x). .x
=
=
=
=
-x,+x),
=
l.4 Sequences of Sets will be written, variously, as (An'. n e IN), (A,,)7, Set sequences fA1,A2,A3, or just fAnl when the context is clear. with each member of A monotone sequence is one that is either non-decreasing, n), being contained in its (An the sequence successor c An+l, V or non-increasing, with each member containing its successor (An+1i An, V n4. We also speak of increasing (resp. decreasing) sequences when the inclusion is strict, with c (resp. D) replacing c (resp.Q). For a non-decreasing sequence, define the set A ...)
=
U';=1A,,and for a non-increasing
sequence the set A
=
fV=IA,;
=
(U';=1Aj)f. These
Sets and Numbers
13 '1'
sets are called the limits of the respective sequences. We may write An A and d,, A. 1 A, and also, in general, limrl-yx,As zt and A,, --->
=
1.12 Example The sequence ((0, 1/n1, n e (Nl is decreasing and has as limit the singleton f0). In fact, limn-+x(0,1/n) = (0) also, whereas 1im,,-4x(0,1/n) 0. The decreasing sequence of open intervals, ((tz 1/n, b + 1/n), n e EN), has as its (J, :1. On the other hand, the sequence ((tz+ 1/n, limit the closed interval its limit is b4. a, a b 1/nj, n e IN) is increasing, and =
-
-
is non-increasConsider an arbitrary sequence (zt,,1.The sequence Bn U,a'='=,#4,, = exists. superior is called limit of the This set the ing, so that B limn-oon (Anl, written limsupnAn, and also as limnAa. Similarly, the limit of sequence the non-decreasing sequence Cn F1e;=,,A,,,is called the inferior limit of the written liminfndn, or limnAr,. Formally: for a sequence (A,,, n e EN ), =
=
sequence,
oo
limsupz
O
=
n=1
n
liminfAn
U n=1
=
Cr
UA,s
(1.19)
m=n
OA,u1 .
m=n
/
(1.20)
The limsup is the set of De Morgan's laws imply that liminfnAn = (limsuwzvlf. elementscontained in infinitely rrltwz of the An, while the liminf is the set belongingto all but a Jl/c number of the An, that is, to every member of the sequence from some point onwards. These concepts provide us with a criterion for convergence of set sequences in genergl. LiminfnAs ? limsupnAn, and if these two sets differ, there are elements that belong to infinitely many of the An, but also do not belong to infinitely many of them. Such a sequence is not convergent. On the other hand, if liminfuA,l = limsupnAn A, the elements of A belong to infinitely many of the An and do not belong to at most a finite number of them. Then the sequence (An) is said to converge to A, and A is called the limit of the sequence. =
1.5 Classes of Subsets The set of a11 the subsets of X is called the power set of X, dnoted 2X The power set of a set with n elements has 2 elements, which accounts for its name and representation. ln the case of a countable set, the power set is thought of fprmally as having 2 20 e le ments. One of the fundamental facts of set theory is tht the number of subsets of a given set strictly exceeds the number of its elments. For finite sets this is obvious, but when extended to countable sets it M to the claim that 2 g > do. ?mntj .
n
1 13 Theorem
.'
')'. ()' td
'' -'
.
Prof '
. . .
.. .
q
E
..'
..'
.
.(
;
))t () it)(lit'(. . . . . '.. .:
'
.
c.
.
.
.
' '
' .
,'
i
(..
=
The proposition is proved if we can show that 2 is equipotent with R or, qivalently (in view of 1.8), with the unit interval (0,1j. For a set A G 2 x (.2.1liitsftqt . th sequence of binat'y digits (:1,hz,,... ) according to the rule,
E . ..
.
2B0
:
. . . ... .. .. . .
-.
.
,
Lbn
Mathematics
14
= 1 if n e 4, bn 0 otherwise'. Using formula (1.15)with m 1 ad q 0, let this sequence detine an element XA of (0,1)(the case where bn 1 for a1l n defines 1). On the other hand, for any element x e (0,1q, construct the set Ax G 2N according to the rule, n in Ax if and only if the nth digit in the binal'y expansion of is a 1'. These constructions define a 1-1 correspondence between 2 ENan d (0 11 w =
=
=
=
'include
.x
,
.
When studying the subsets of a given set, particularly their measure-theoretic properties, the power set is often too big for anything very interesting or useful to be said about it. The idea behind the following detinitions is to specify subX sets of 2 t h a t are large enough to be interesting, but whose characteristics may be more tractable. We typically do this by choosing a base collection of sets with known properties, and then specifying certain operations for creating new sets from existing ones. These operations permit an interesting diversity of class members to be generated, but important properties of the sets may be deduced from those of the base collection, as the following examples show.
1.14 Definition A ring S is a nonempty class of subsets of X satisfying (a) O q S. (b) lf A and B e R then A wB e S, A (AB G R and A B e X. n -
One generates a ring by specifying an arbitrary basic collection C, which must include 0, and then declaring that any sets that can be generated by the specified operations also belong to the class. A ring is said to be closed under the operations of union, intersection and difference. Rings lack a crucial piece of structure, for there is no requirement fclr the set X itself to be a member. lf X is included, a ring becomes aheld, or synonymously AC, this amounts to including all complements, and, in an algebra. Since X-A view of the de Morgan laws, specifying the inclusion of intersections and differences becomes redundant. =
1.15 Dennition A tield 5 is a class of subsets of X satisfying (a) X e 5. (b) lf A 6 5 then Ac 6 T. (c) If A and B G 5 then A t..p # e @. n A field is said to be closed under complementation and finite union, and hence under intersections and differences too; none of these operations can take one outside the class. These classes can be very complex, and also very trivial. The simplest case of a ring is (OJ. The smallest possible t'ield is tX,O). Scarcely less trivial is the AC, O ) where A is any subset of X. What makes any class of sets field (-Y,A, interesting, or not, is the collection C of sets it is declared to contain, which for the class. We speak of the smallest tield we can think of as the generated by C'. field containing C as Rings and fields are natural classes in the sense of being defined in terms of the simple set operations, but their structure is rather restrictive for some of '
,
tseed'
tthe
Sets and Numbers
15
the applications in probability. More inclusive definitions, carefully tailored to include some important cases, are as follows. ,/
is a non-empty class of subsets of X satisfying 1.16 Definition A semi-ring (a) O e $. then A /'n B e (b) If A, B e where (c) If A, B e and A i #, H n < cxa such that B A U7=1G', e = f''h for each j'. O j, n and .f.
,/
,/
./
=
-
.
.'
.
More succinctly, condition partition into '-sets.
that the difference of two '-sets has a finite
(c) says ,/
is a class of subsets 1.17 Dennition A semi-algebra (a) X e (b) If A, B e then A chB e (c) If A e $, 3 n < x such that Ac Uyzlcj,where for each j, j'. a
of X satisfying
.f.
.l'.
,/
=
.
e
.t/?
and C) ch Cj'
O
=
A semi-ring containing X is a semi-algebra. R, and consider the class of all the half-open intervals and Iz < together with the empty set. If h (J1,:1) for I (tz,!?J (J2,:1q, f-'h and 0. And if fl = (tza,:2),then fl h is. one of h, h, (tz1,:2), c 11 (tz2,J11, (:1,:2j, so that a) f tz1 and bj f b1, then h h is one of 3, semi-ritzg conditions defining therefore and The sat(az,atj t.p (:1,&2J, h. a are isfied, although not those detining a ring. If we now let R be a member of the class and follow 1.17, we find that the halfand (tz,+x),plus open iniervals, plus the unbounded intervals of the form In O artd R, constitute a semi-algebra.
1-18 Example
Let X
=
=
a S b
<
-n
+C,o,
=
-
-xtbl
1.6 ji gTa Fields .
. .. .
.
.
Whn we' say tht a field contains the complements ;;!,' t!)'--,$@. ,;' y' .j' r' ;'(k,-':' ..,' iip'.l!:d @'. j,' jr'r' (@.-)--81'* :)'. ):'. '''#' r', jjli'. yrd jd r'. ;d 11,;::2:r11.-*. :-'. i'-' t:'.-'-ll)ll'-' qrrrdkiy--i---.d-,jtls'ltdkdi.lhlikirrstipp.:fl.rd pFq'. yy.;'. ,;r'j-. y' yjd ()..' jt'. ijd k' j'yy.'-r)'.' 133*.* j'i'.i (,'. ;);.;-' ;'@@' r' (y'. j.' )' i@'. j..' t'' )'()' y'; g' q'd ('. (';' kt'qi'. ''Ikiii!r:-'. ((' t,,(,t'' ?' L' ..' yjjjjjyryjjj.,,d :::*-2)1-.* Ft@-'. !!)p'.(' py'i'l'g )tI-'. ytd yy..,';.-: y;j-yy;',. (';' .d 4j' ('. .LLLq'. .:!1211-----*:11... :I.EIII.-.CEII-.'. 'Iii!'.k.d ''Ikii:rr:-d q!IkE7'r!'k..'. .-'Iki:t:i;'.. 'q'IlkEjIIk,'. (.'. ,''-r.-'II::7!IIk... I'EEEII.'. ijyipl,.dtsil----'k-ljptjs..!rrl ,-jb'L)j-..L'tjLkL33,?.,.q-',i-..-j,qk.. t(' .y)'. tjd t'j,'t)' .'')'''';:E:!!.,-. ''k::E:!:.' k.-'-'''Ill::iEl1I-..t.''-.tt'$'12ii!I.-.'. rpr'y r' .-'. r'.' .-' '1I(ii''' (!11:::1,-* 4111E::::,..*..':!1L.*. it:ipd rjqj-dltdllt.-l'ttd desyrves explanation. It is clear that z4l
and finite unions, the qualifier A,, e 5 implies that U& j=3 A) .#hit lteratlon of slmple by n-fold 5 But, the constructlv: palrwlse unlon. e glven t t: (j jm. tj (jj'I Out W 1* legiti 1 k. tju further 1 mate ttlr th t ti t no e e s on q':j).: .) . ' a s P ul ati O n to r o t? 41@.;: )' .(ir.. ytt,rtlh be taken llmlt. a.,g, the By maklng n.operatlon can thls addltlonal to ))(, j).jt (..... .. . . . 'lk::Ei!lk.. @ E..t!1IIp:,I.i.. 'r '. E. !!I1L., :111:7:!1.-. ' ' ' :..-t.'E!IIk..' . .-'.EEE:!!:. ..,.jy-...'- ..:..'.. '. ' .. . :111:7:!1.-.. r,,;-. Jl'... ' ' ' '..-.. .' ' ''. . rr ... '.r.-'...(. . . .'.'i.7tr. i'.' ..luju,-..t9'.j:;:,.j)..jr, r.... f.tltz... jyy;;.jjy..... jj.lyytjr .. .. ...... .. . ;.g,..yt:jj,j.. . ..*1.: j., A .'.j'tp:('.()) r yjj''.'gt). (tsajgebra)5 ls a class Of subsets Of X satlsfylng ('. f l ...;...: uljjjtl .......,..y 't;. r;J) .j . j ?)...j!.y. -y..F..:yqqy t)?.p:. (y .kly(..yy.yq..:)...q ):;1211E11.... .!kjjj;;r;;)., .. ' ..
.' .'.' ':jls')d .'. '-' -j--!,r,!tk'. -:'. --y!j;;'-j-.' -q' -,.'tp..-.(t. -'k' -(r)'. -8)* y' :')' 24jg* -'. -' j;d ('. !111:(q*;LL.' ;..(' i.'. k' .-'. -'-' --'' -,' j'y' -:tr!)i...ti:,jp':F'--)?;t--''' E' ,-.'itt-,ttd ,'-'.k'(.E ;E-;r'.' ?' y' lrrq))d :':11,*. :L';'' ''. g')j;-',' j';'y' y't':. i--d r' s.d ,'g'' E'. )-''E -;.' ''IItEL.FX -' -,'.-y' -'. .)' .q' ''. k' .:' ,''(''jy ;'))'. ,'..(.. (t' ?.' :y'-' ;;' yyd ((' ;'. :('. F'-j. !': q' (j.,' F' C--qy;'E :'-'E' :)')' L3--'; :;' ''-.' ;,'-'L'' ,-'' )'
,-,' -' .','.:'.. -' .','.-tq'-' -,-' .)r,' ,--))'-' ,,,' 'jjjj:yr:jjj,'; .'.'. ''.-'' .!112::*. .'. ,'.'-'. 4:11::.* '''bii'''bi''-''
'.'. .''''--d .'.' .' . .'r:).. ... ''.' '..'. '. ..' . .' . .
'.
:
...
.
.
....
. E- .
(. '
... .
.
.. ' g' '
.
q
(
.'' . .' '( . . EE
...
.
. .. . . .. . . (j;) :.. . . . :
.
.
E
t
,np
E
.
:
.'
.
'!;'j .L'
'. 7 .!L(.
...;
j, ....j .; ,4k .:f .(a z ( .
.
.
. -
-.
..
....'
j.(
...:;.. .
...
...
-
.
(
.
....
'
.'
.. .
g
...
.
7
.
..
.. .
'
.
.
..
..i
..k
. .
.
.:.
.
..
. . ' .
.
.
.
..
dp
4)
..
-11::::::,11;-.
.11:2:::,...
,11::2::,1.-
.
:.. .
.
:
,
. .
y.
.
.
......'
'lk:!r:-.
,y
..
-
. .. .,k;)jjjtyy........
(lzzllp'l. ' (jj:jjj.. (j;jj..,...
.
y.
.
.
.
.
,,
.
,,
,,
7.
u.....r.
:..,. . .: . ... . ;. . ..;.,k y . (. .;..LtEL. :. . , : ry q..;..y.... jjggggyy;: . .. .-'E . .. .y E E. CE.; . . ;.... : .)Ey. -... . r :. . .E . ---i .. . ,. q .. ..E . . .q ,. jy. ..g .y.. EitttliyitztlrE . . rtjyEiljzqttigryyf ,.. . q - yy q . j . .. . . q . y . . .. j, . . . . koyyyyyyjgy . y. . . . . . yyyjj yy. . yju r . ' i EE yu y . E !' '(' :. q: i E.'rit:-. g l.E; :);y.. g yy g ().. y . .. g y y: .. jy q; ; L y. . . r Lj.,L..LjL; I.t: jL'L'q(jLL'; y. '.t;tEqq;;' gE.y))gy .. y j, . . .... .. ... , . . . . . . ....yy .q :.q( .;).(,LL.;..;.. . . . y y. r.@'ry'.'y..r?i..E.!:..... kjjjyj .;. rykt',.E..yi. ....q .. y.; . y.p; y. . . .'. .)L;jt..).q(y,jj,d!!1i-,; :.( ..E.. -y ....) ( ?.jj,.. y-1... y ' ' E i i. E - rFti... t E .j;. ;!.. E (! . !- E . jygy. . - ...;:.. E..- . E. . (E .. E ! .... . : . .;.. .kyk. . EE t'F-i''' - q (. . -. . . ...rj.. . -: -. -, . kL. . qE. i . -; -y.,.-.g.1.j..-..d..,)....y.;,..y...r.-y.-.st$y-. .. y . .;.. -. .y '. EE.. ; . t; . . . ,.f33f. .E E . . .. .- y . -. ':$l8t,E-.k;#)),pl-(i'.-.'j,b.slb . .....p.-.. .'. --..'-... -.pt -y. ,.)ki-f- --.--; #..... .. . ,,,gt;:t!jj:jjj.j .
dp
:1
.
.
.
..
.
.
q . ..
..
..y).
...
-
41
...
..:..:.
.i'.
1p
...t..
.
...;;.;,,
.
.
. .
...y;'y'y...y(iy(y)(yy:jj..yy..:y(j....
.:jy.yj...
2
.
.
:
.
.'
<.
.
.
.2.
.
...'
....
..r
:';'
'
.
.
'
.
......
:
...
.
..
.
,
-
.q.(
'rt,ywpy
..
. .
...
-
EyEi yt($.?E.r(r . .. ,E 'E. i.E k'JE C4jIIE. ..L 3.k() LLj;L.L .(:.g).-. E ;i.EE-)!'(; . ttjii;. )....y.2..2...,..J): :7.:..
E..y.(.
.
: :E..
LjL.L..kL /. ..
..
.
.
-:--
-. - . ..j .. .j(;E ;.; . (y.jjj.(q.. sl/tq E .
..
. .
'
.E.. . .. j q. ; E E/E.Lq ..L.. . . ... . . ) .. --(j ( . . --. ' . i . -.' --- . -.. . . - . E. .-.. . . -. -:--'. -. ..' ' ... g.. t):;.... ..q/.:I.jLL . .. : g (! .... j ...:k..j..g . . LLL.. ;.r .y y.. . y. gy y . .:yyy ... ' bs? qoujyyjd . ;
...:
.
,
o
.
.
E
.:
...
...
'
'
,.
.
:!)
.:
.
'
; . .j . .. . . ... . g . .. 4 . '-'112::..
iE: ':'E: . -.' ..;E.) '.'.E
...;.
.
. : f . !..p ' ... L. ' '. . L...'...2.r .' ' ;''. ' . ( . (.' ,u..km,,,,o,k,,,.,r+y,,,.,,a, .
.
-
E.'.p.:
)Eq
.
.
: .
.'
.. .
...,
..
.
. .
.
..
.
.
.
. .
.
.r.. . g ... j . ., . ...... . . ... .4: j y. .yyjaj . ts. )j; . j .. jjjj:jjj'. !!!' i.$. jys,lt. . k.. i.. . .ty jyy )- jjy . yyrtyj g '#:'l:'j':.j'.jr:!.k::: .. .yy . yyy gc y i: . . l. .r.)(k!r/.:i..tt'.j:r. .tr(g.(..). r. i)'i;r... yt .tk-. . ttr;: .;... ryki j yy y y. t. r..r) ti).;7..i-tl ... j;... t). . ryli l.t:.. ,:,.,:,, ;)).y;..)-k.t8.l9)-., gy'tt..kq.j4:yy.-.-..,.it::!:j-.,.. L,tjtLL r,.. . 442!2:r ,-L..).;;;.)-L66LLj6q?''' jj:rljjs 41222:; jjzkr:i 44:22,), .(::.-tgy..tr..;y(jtty . ...E .-..(jl. 'l.s' 4(222:, 4(22::; jjjj;, rjrr, ,...EEEEEE. 4k22::; ..j'tj'L. .1k.k,j. (jlr:jl. jljrhhhjj :jjjgCIE1j,. :jjji rjjjq- :jjj;, (jrr:lk. :.1-:.9. (jrrjl. :;:;::::;' ; '. ).. i-.ql..?-' 4(4E ,, IILLL,.:LL..).L (jj j(.@i. .-. .:y. i.-. t; t. p ;. )i. . , E. .. t. El(;. jr. .. r ; ;. tgg j)rpi j-. :18-. (..?- rjq. jyy.i . j-y . t-. ;. .-. j.. . ))-. . . '.i . r. . . .. r,r .' ty. (y;j--.;.kr. . . . ;. .. . . . .. . (. yj).r'!jz'.,..,jp'.'..-''.. . . -,y.,4tE:1'. :lir4:::: 4t::. 1. qlL y.-y1,y-, 4:::,. ....,....,.)s,j.,, 112). 4:2:: 4::2.::;7. ;p';!r 4:::). 1ki2: :5'k 11. llrlu 4(22). 1L )y.j. .l,'.k' .::2). .. L6:-. 1ty!1.. 1:EIL llrqlrrkp IIF:I::IL llrl. :5,k .. .... ..-!1-2Ik. h..dki:r (1. qlrl.IEEIk . .:E:j.. 1t) 1l: IIFI. Jlr!l. ql. Ilrl,) jll:)p .11:)'L.. tdk4(22). !rdtgr11.,1. ki-b.' u. Jlrck!q;l.. . .'... . ' L..;L.-.?-..... k,' qtthk. t.. .r)k.-...y . tl.#-''-it--.. ..----. --L(-q...L.t..-. tl),,:... . . . ytyth, t. til. u Cit-'.' . laws, of countble 1ntersecti on also. A . t.g cqqy 7. tp,y E. E ::.rgn ' .E ..r.;E-.:. ng can EF. jC.y E. . g.. .--.. t.. . ykkE (j,.t, . . y 2 ,t ...E.: j....y t( q.,y...y.. .,p... .(),. 'tj. , . .. .. ,ar. . J?'1'l ,. ;w.. tykk,yyal. @(r,t t7l7)li)'yE--jp:ytq-l'h@i'-.--.j#i.tt)r:-.)k2!l$,j);.. ;,; , , .. t..Lti-.(yy....hk-...'i'..-.-i'.---j;.rls tlot , t ,-,Ljb;LLb6.. j, ) ln the sequel )j.rq( shall t;need -)ykyy.tjj.. t:-.t.-)-t...?t-(..--.-..;-yy..,F.rconcept . -?,--y)k-ttg.-tjE '...-..-.--..:.--r-. we g ..yqE a E ;..-...y..-.:..)...;..y..--.y.y....-........... ),,r.tg2s;-...:,:.5-r):(t..---r@t()kttr;jpki-):.y.t2r).-. ,-.)j.Lb1,,.q--. (;. .y......-.. . .. .. .. . . ,,.tzk r:kjt. 1)4 ? E y,pqy),jj 33:13333,..6,31)3 qtp. 2L2t1jjk1,(1.. yo,i, . .. q l,. ;,?,. q?. y. . j,yj(. lnt:rsectlon (E. . . . .qk .. : :6. :;)..y.,@, of al1 the jyjj,, .( jy .yj ls j.j..j. ,,,. yjyg.. yyt;,y( tygyj;. contmnlng y.y,rty..y..j, ,. . : '! ;(y.tjrj. . (j(:,.tjyyy. .. . y . ty.j .. .. . . . . sj jyjjy.t.yjgy, LILLIL) jjjjjj,yyyyyjy.. yyyyy T. .r.. . .. .)g-jjjy)j.).)4jj..j)j.:.,;j.);.j,j;.jj;.)j,.j.L..,.;..j ..., y. .........j3b;,jbbb;$LLL,).jtfjj;6;3$j(. ..g ...s.sy . ... .. .. .. . . .. . .- . . :7',t.,..;r,,#.-. )),.jjj ). .. ..... . . .jj,.., ...yj.,.. . .'''.. . .j.q,jj. . . . . ' q$3t... ,;-. . . ..' r-,-iiq.-r),-iyr,!!.. j. .=-. $.-<..., . . . );,-':'!'tk. '
. '
.
-
..
.:.
-
.
-.
.
''.
-
-
.
..
.
. . .
-
.;
.
,.);.-')
-
,j,q2..
'
.. . .
tju
..
-
..
,
- ,.
,.-
- ..y;)
-
.
..;.
-,...
..
,..
..
.-.
.. ..
y
:.
.
..
. . .
.
(gjj.. -0. ' s.t.).....,-.y....ytt,..c...y.-....1. ?t..'tlt.t.,t.iy.'t?.'rtA)'.i.' .d.e.. '':.('. e..,4(i!I... ''-. . . %)' 4.) 4)d..,'',,M ttytt/p ''4kk,,.,)'#',,,..)4i yt t 14thls iis 7>1'h: j x,,soy s,: xe.)ts..T?th: ,, .
. . .
.
-.
-
.
-.
,.:r
,.
-
.
.
.-.
-
.
.
.-
.
.
.
.
.
.-
. .
.
.
.
. .-
.
. -
.
.
.j(rtrp1k.
.jry,
,,...
.
.
.
. .q;,..
-iji
.
-.
-..
.-
'
.
-.
. .
-
1k-,,,,-Jj
rjjj;;!;;q
.
.-t;,
. .. .
..
.
.
. .
. ..
.
,,,
-,,
.
.....';-.7.
..:.,2,,
'
..
.
,-,.
c-n'
',,
.;
''
,;j),
''j-.k,
-
. . .
,..
.
.
.
'.
--k...
-. .
.
-
. ..
-
.
.
--,.
..
.
be
,,,
.
.
.
..
-
.,.
,,
't(l!jt)t2),.
..
-@jjjgjjk.:
.,i(-$;ydjy-.. .....
. :
.
'
.
.
:
.
..
.
jlj..
. yyjjggk., ?jjy-.$,,;gjjjr..tyy(,.k;.y;jy;j.-
:jjjy...
.
y
.
..
y.s,.y,.y
.
.
.
, .
.
. ..
..
..
'..-,,ibn
y,.;.y
... ,
.....,.--
.
k
s..
,
...
.
.
,,
.
...)...j....j.,..
,.yyy....y,y.y
..
.
.
.,.)......),),.....,).j
.:
.
-..!it,..
....
.
,q
kf,.
,.
.
.
,,
.yy
,4j,j)),,;jjj,-j)))jj)..,;;,)j,,j...
..
-),-.t..
.
.
.
.)
-,,
,.
.
..
,,
.
....
,
.
.
.
,
..
c-flelds
--
.
-,
C
--
.
16
Mathematics
generated by C, customarily denoted c(C). The following theorem establishes a basic fact about c-fields.
calledthe c-field
1.20 Theorem If C is a finite collection c(C) is tinite, otherwise c(C) is always uncountable. Proof Define the relation R between elements of X by Ry iff x and y are elements of the same sets of C' R is an equivalence relation, and hence defines an equivalence class & of disjoint subsets. Each set of & is the intersection of a11the C-sets containing its elements and the complements of the remainder. (For example, see Fig. 1.1. For this collection of regions of R2 & is the partition defined by the complete network of set boundaries.) lf C contains n sets, & contains at most ln sets and c(C), in this case the collection of all unions of &-sets, contains at most lln sets. This proves the first pm of the theorem. Let C be infinite. If it is uncountable then so is c(C) and there is nothing more to show, so assume C is countable. ln this case every set in & is a countable intersection of C-sets or the complements of C-sets, hence & g c(C), and hence is the collection of a11the countable unions of also U4&) c(C), where flX&) t/(&) uncountable, is &-sets. If we show the same will be true of c(C). We may assume that & is countable, since otherwise there is nothing more to show. So 1et the sets of & be indexed by N. Then every union of &-sets corresponds uniquely with a subset of N, and every subset of ENcorresponds uniquely to a union of 2N, &-sets. ln other words, the elements of U4&) are equipotent with those of which are uncountable by 1.13. This completes the proof. . .
,
R, and let C ((-x,rj, r G (E1 ), the collection of closed with rational called Jltilt/'-ff/r6y endpoints. c(C) is the Borel jield of R, generally denotd B. A number of different base collections generate T. Since countable unions of open intervals can be closed intervals, and vice versa, (compare1.12), the set of open half-lines, ((-x,r), r G (EI), will also serve. Or, letting (rnJ b: a decreasing sequence of rational numbers with rn 2, x,
1.21 Example Let X
=
=
(1.21) Such a sequence exists for any G ER(see2.15), and hence the same c-field is generated by the (uncountable)collection of half-lines with real endpoints, ((-x,x), x e R ) lt easily follows that various other collections generate S, including th open intervals of R, the closed intervals, and the half-open .x
.
intervals. n
1 22 Example Let X given. It is
T
=
F, the extended
real line. The Borel
field of F is easily
# e Sl, where B is the Borel field of R. You can verify that S is a c-field, and is generated by the collection F' of 1.21 augmented by the sets (-x) and V. n =
(#, B tp (+xJ, B t.p (-x4, B QJ (+x) t-p (-x):
1.23 Example Givn an interval f of the line, the class
BI
=
LBf-h f: B e B ) is
Sets and Numbers
called the restriction of T to 1, or the Borel field on 1. ln fact, T; is the c-field generated from the collection C ((-x,r1f-h f: r e () ) n =
.
Notice how c(r) has been defined from the outside' It might be thought that c(r) the inside', in terms of a specified sequence of the operacould be defined tions of complementation and countable union applied to the elements of r. But, despite the constructive nature of the definitions, 1.20 suggests how this may be impossible. Suppose we define .Wlas the set that contains r, together with the complement of every set in T' and a11 the finite and countable unions of the sets is not c(C) because it does not contain the complements of the of r. Of course, be the set containing W1together with all the complements and unions. So let in the same finite and countable unions of the sets in W1.Defining 45, manner, itmight be thoughtthat themonotone sequence (W,,) would approach c(C) x; but in fact this is not so. ln the case of the class fm,l!, for as n it example, can be shown that Wx is strictly smaller than c(C) (see Billingsley IX This fact is 1986: 261. On the other hand, c(C) may be sm aller than for in again fm,1), j3.4. demonstrated, The union of two c-fields (the set of elements contained in either or both of them) is not generally a c-field, for the unions of the sets from one tield with those from the other are not guaranteed to belong to it. The concept of union for c-fields is therefore extended by adding in these sets. Given c-fields ; and N, the smallest c-field containing a11the elements of 5 and all the elements of Wis denoted 5 v V, called the union of 5 and V. On the other hand, 5 r'A W (A:A G . and A e V') is a c-field, although for uniformity the notation 5 ZN V may be used for such intersections. Fonnally, 5 zq N denotes the largest of the c-fields whose elements belong to both 5 and V. Both of these operations generalize to the count1,2,3,... we may define able case, so that for a sequence of c-tields 5n, n l;a. V= n n 15 n and 0* Without going prematurely into too many details, it can be said that a large part of the intellectual labour in probability and measure theory is devoted to proving that particular classes of sets are c-tields. Problems of this kind will arige throughout this book. It is usually not too hard to show that Ac e 5 whenever W e 5, but the requirement to show that a class contains the countable unions can be tough to fulfil. The following material can be helpful in this connection, A monotone class A is a class of sets such that, if tAn) is a monotone sequence withL limit 4, and An e A for a1l n, then A G A. If (Aa) is non-decreasing, then zl U7=1An.lf it is non-increasing, then A O0J=Iz4,,. The next theorem shows that, to dtenine whether or not we ar: dalinf with a c-field, it is sufficient to ,'krd r' y' jr';ty 't'ld r' yyy'.yyjjjjyjjjjjjiy.d k' t'(' y' );' jyyd lr'yyy p' )' 'y,d trytd td ttd t'. ild ctmsl'detwhdherthe limits of monotone sequences belong to it, which should often ' b j esiet to establish than the general case. .
tfrom
.d1
.4z
.44,...
--->
.
=
=
=
=
=
C' y' .'E
-'. .' -' :t' '
.'''
.' '''.'' ..'.'.'....'. ..'.' '. ' . . .'... . .. .. ..(E.
' . .. '
E
' g. ...
' . .'
r . .. : :
1);,4
.
. .lj ' .. . y. .. . . Flrqh:oykm
l q)
l g...(
.. . , . . .
l .
.
'
.
.
. .
.
.
?;; is a c-field iff it is both a field and a monotone class.
.t:)2
E
.
.
:.y
:
.
. . .. . . .r ..Eq(
,,k))
y.
t j' ' .
. . The..ftjjjjy, p tjtjjy .
''
of the
ito
is immediate. For
theorem patt the part y(t.: j. . ;) y y. j.r r ,'tlt:' t.ri.jjryry.yi.,yuit y 1:2j1. :jkq-.j(jjji. :g;pC;: ;(.):t..,.(.,)i'.:..'..y.;tt.t,jiyA......,y '.1t jkj:r qlj?ii. 4, )1:41.. .j1r:lg. 1:;(jj.. jpp:p-;,r 4:;:: ...... jlrrjs 4:2:: 422:: j,qbqf. . 4EEEE! jjr-iylr jjy-plrqjj .. jrrqls ,,krIg. :;; igjl. ji. ji. :. ,(EE L.,;.,jfL:..; ... j . I j!..I t;i:;r ni; (1q ... .g. )......)...... tjjii:;s ,b, 4:22EE jj tlij;r (yj,?)'' yj e. (. . k:j?,. .t.y(). r .., j; (jry-;;;(tj yy ). .s.u.. y... r(j-(.. (.; .i.qLL);,;j . y. . ... y... .;jtytytyt, . . . . tg -L$1t.. ttttt . . .;j )')r'--t)-tzjj i-.-tki -,.-(jkyr))-;-..---,r. y is . rj limit with =1A,, t monotone ;, );. by t sequence ) a ulyytjt U.-----.- e y-):-),::::r,:--.. --..L3LLLi..-. .,.-yt.-..,7r-...(-.......). r ;-)y)-. j/l-rjt--j,-....;.LL;L;.,.;j;;-),.,;b;qj, '''-?'''t''-' )jjt.-. (t)-.)rrtt.,..,)))-y)#)---.r),-.-. t;)--?--.--,r...r-....-.,... ...... ... . . .--. ... ;.,-(.$;.,q;-.,-LLLL1L--t;))plt:'-i yryjyti... E #,,.. .. kr. t).y-j-t,yr ....-..--...- ....-.... ttl'i?.-.-;r-..t...)k.-(--..-t..-.ti..-)-.,)---)'.-.-.t.....-..t-., .t'y4?.i@''. ... . ..Li q-?..'..I3;......;-;t.-b. .!.#,..-t,) llt))k...-.... tiy,-3$bqL-;--....').L ..::?)i..-..$kr'-'t)L' .. .)yyyiyyyi 14441111@11..... i
r
,.tE'jt.ty( : ;,
i E kE'
(
(
.
.
-.yy)j,ytj,.i...y.yE.y;;r.,...myt;y.U,
; -. . g . .
; ;
(
.
E
i..E -
-
q
,.
..
.
.,...
,,
-. .
-
-. .
- . . .
:.
-
.
-.
.E-...
,
q .
. . .
.. -.
.
.
:
E .
.
-
k.tlll.
..
-.
..
..
...-....
.....
..
,.
.
.
..
- ...-.-
.
.
. . ,
...
,,
-;::,---::--
.
-
........
-j
..-.
.,
.
.
..
...
'''--'f'':tt--
.....
define
,
-k,(
..
.....,.
x ' '--------8-'-:'''*''''% . -))
'if'
-jlqilii;,;j;r
.
....
..
-......-....
..
.
.
. .
.
.
.
.
.
..-
..
... ..... -
-p
. .
..
..
-
.
18
Mathetnatics Ur1=lAk, so the theorem follows.
assumption.Uo;=1An =
.
Another useful trick is Dynkin ' s zr- theorem.l To develop this result, we define two new classes of subsets of X.
1.25 Definition A class T is a z-system if A and B e T implies A rn B e T. A class .f is a h-system if (a) X e 1. and B (b) If A and B e c A, then A B e 1. is a non-decreasing sequence and Au A, then A e (c) If fAn e n ,f
-
'1'
,f)
.
Conditions (a)and (b)imply that a l-system is closed under complementation (put .f
m.Moreover, since (b) implies that Bn A,,+1 A,i e for each n, (c) implies that a countable union of disjoint d-sets is in 1. In fact, these implica-
X
=
=
-
tions hold in both directions, and we have the following.
is a l-system if and only if 1.26 Theorem A class (a) X e 1. then Bc e 1. () If B e is a disjoint sequence, then UnAn e (c) If (Aa e ,f
.f
,f)
.
Ia
ln particular, a c-field is a X-system, and moreover, a class that is both a is is a c-tield. This follows by 1.24, because a a-system and a class closed under unions by 1.25(c), and by de Morgan's laws is if a monotone closed under both intersections and complemntation. The following result makes these dehnitions useful. -system
-system
1.27 Dynkin's x-A theorem (Billingsley 1979, 1986: th. 3.2) If T is a a-system, ctfis a l-system, and P c 2, then c(T) c 1. containing P (theintersection of a11 Proof Let kTj denote the smallest containing P), so that in particular, (T) the c M.We show that 3.(:P)is remarks l-system. follow the above, it will then that (T) is a c-field, and By a 2, as required. hence that c(T) c &T) For a set A e 1(T), let Vztdenote the class of sets B such that A rn B e (T). We Clearly, X e Vz, so that condition 1.25(a) is shall show that gk is a satisfied. Let #1, Bz e 5k, and Bj c #2; then A rnfl e 14T) and A fa #2 e (T), and (A r''h#1) c (A fa #2), which implies that -system
-systems
-system.
(A rn #2)
(A t'''h#1)
-
A f'n
=
(#2 -
#1) e 14T).
(1.22)
But this means that Bz #1 e Vx, and condition 1.25(b) is satisfied. Lastly, suppose A f''h Bi e ,4T) for each i 1,2,... and Bi #. Then zt f-h # e (T) by 1.25(c), which means that 1.25(c) holds for Vx, and sk is a as asserted. Suppose A e P. Then B e P implies A PB e T p is a a-system) and since P i and '4T) (T), this further implies B c 5k. Hence P c 5k. Since Nx is a containing have in this smallest also (T) gk P, is the we case. So, c V',A and hence A fa B e (T). whn A e T, # e VT) implies B e We can summarize the last conclusion as: -
'1'
=
-system
-system,
-system
Sets and Numbers (A e P, B e 1(T)) Now defining Vs by analogy
(# e
(T),
yt
with
PB e
=
19
(A OB e
(T)).
(1.23)
9%, so that (T))
=
(A e Vs),
(1.24)
we see that (1.23)and (1.24)together yield P c T. Since Vs is also a by the same argument as held for Vx, and contains T, 1(T) c Ns by definition of (T). Thus, suppose B e (T) and C E (T). Then Ce Vs, which means that B f''h Cq (T). So 1(T) is a n-system as required. . -system
+.
2 Limits and Continuity
2.1 The Topology of the Real Line The purpose of this section is to treat rigorously the idea of as it applies to points of the line. The key ingredient of the theory is the distance between a pair of points, x, y e R, defined as the non-negative real number Ix y l what is formally called the Euclidean distance. ln Chapters 5 and 6 we examine the generalization of this theory to non-Euclidean spaces, and find not only that most aspects of the theory have a natural generalization, but that the concept of distance itself can be dispensed with in their development. We are really studying a special case of a very powerful general theory. This fact may be helpful in making sense of certain ideas, the definition of compactness for example, which can otherwise appear a little puzzling at first sight. An e-neighbourhood of a point e R is a set Sx, fy: Ix-yI < :1, for some s > 0. An open set is a set d c R such that for each e A, there exists for some e > 0 an e-neighbourhood which is a subset of d. The open intervals defined in j1.3 are open sets since if a < < b, : minl 1b l > 0 satisfies the la definition. R and O are also open sets on the definition. The concept of an open set is subtle and often gives beginners some diftkulty. Naive intuition strongly favours the notion that in any bounded set of points to' a point outside the set. But open sets are there ought to be one that is sets that do not have this property, and there is no shortage of them in R. For a complete understanding of the issues involved we need the additional concepts of Cauchy sequence and limit, to appear in j2.2 below. Doubters are invited to suspend their disbelief for the moment and just take the definition at face value. The collection of a1l the open sets of k is known as the topology of R. More precisely, we ought to call this the usual topology on R, since other ways of defining open sets of R can be devised, although these will not concern us. (See Chapter 6 for more information on these matters,) More generally, we can discuss subsets of (Rfrom a topological standpoint, although we would tend to use the term subspace rather than subset in this context. If A c S ? R, we say that A is open in S if for each x G 4 there exists Sxtzj, : > 0, such that .(x,E) f''AS is a subset of A. Thus, the interval (0,) is not open in R, but it is open in (0,1). These sets define the relative topology on S, that is, the topology on S relative to R. The following result is an immediate consequence of the definition. inearness',
-
,
.z7
=
.'r
-x1
.7
=
-x1
,
2.1 Theorem lf 4 is open in R, A r''hS is open in the relative topology on S.
l:n
A closure point of a set A is a point x e R such that, for every E > 0, the set
Limits and Continuity
21
is not empty. The closure points of A are not necessalily elements of A /'7 A, open sets being a case in point. The set of closure points of A is called the c losure of A, and will be denoted X-or sometimes (A)- if the set is defined by an expression. On the other hand, an accumulation point of A is a point x e R wich is a closure point of the set A f.x) An accumulation point has other points of A A, it must also arbitrarily close to it, and if x is a closure point of A and accumulation closure is point. A point that not an accumulation point (the be an of contains x former definition being satisfied because each E-neighbourhood itselfl is an isolated point of A. AC /-7 Sxnz) is A boundar.v point of a set A is a point e X such that the set points of A is denoted ?A, and X not empty for any e:> 0 The set of boundary AO A A. A closed set is one containing A t.p :z4. The interior of A is the set all its closure points, i.e. a set A such that A A. For an open interval A a,b) c R, X' (tz,1. Every point of anb) is a closure point, and a and b are also closure points, not belonging to asbj. They are the boundat'y points of both a,b) and gJ,?7). ,&(x,sl
-
.
.x
.x
.x
=
.
=
-
=
=
=
2.2 Theorem The complement of an open set in R is a closed set.
(:I
This gives an alternative definition of a closed set. According to the definitions, O (theempty set) and R are both open and closed. The half-line (-x,xq is the complement of the open set @,+x)and is hence closed. Extending this result to relative topologies, we have the following.
2.3 Theorem lf
:4
is open in S c R, then S -A is closed in S. cl
In particular, a corollary to 2.1 is that if B is closed in R then S r-hB is closed in S. But, for example, the interval l, 1) is not closed in R, although it is is open in (0,1). closed in the set (0,1),since its complement (0,r1) Some additional properties of open sets are given in the following theorems.
2.4 Theorem
(i) The
union of a collection of open sets is open.
(ii) If A and B are open, then A /-7 B is open. n
This result is be proved in a more general context below, as 5.4. Arbitrary intersections of open sets need not be open. See 1.12 for a counter-example.
2.5 Theorem Every open set A q R is the union of a countable collectin joint open intervals.
of dis-
Proof Consider a collection (&x,:x), x e A h where for each x, Ex > 0 is chosen enough 5'@,:x) Then that A. small i Uxex5'tx,:xlA, but, since necessarily A UxeA5'@,sxl UxeA.@,Exl, it follows that A. This shows that A is a union of open intervals. Now define a relation R for elements of A, such that xRy if there exists an open interval f q A with x e f and y E 1. Every x e A is contained in some interval by the preceding argument, so that xRx for all x e A. The symmetry of R is obvious. Lastly, if x,y e I A and y,z e 1' ? A, I /-7 1' is nonempty and hence I t.p 1' is also an open interval, so R is transitive. Hence R is an equivalence =
Mathematics
22
relation, and the intervals I are an equivalence class partitioning A. Thus, A is a union of disjoint open intervals. The theorem now follows from 1.11. w Recall from 1.21 that f, the Borel field of R, is the c-field of sets generated by both the open and the closed half-lines. Since every interval is the intersection of a half-line (openor closed) with the complement of another half-line, 2.2 and 2.5 yield directly the following important fact.
2.6 Theorem
B contains the open sets and the closed sets of R n .
A collection r is called a covering for a set A i R if A an open set, it is called an open covering.
UsswB. If each
B is
covering theorem If C is any collection of open subsets of R, 2.7 Lindelf's is countable subcollection (Bi e C, i e N ) such that there a X
UB U#. =
f=1
Be N
Proof Consider the collection (Sk Srkbs, rk e Q, sk e Q+) ; that is the collection of a11 neighbourhoods of rational points of R, having rational radii. xQ+ is countable by 1.5, and hence is countable; in other words. The set (E1 show exhausts by We k the set. that, for any open set B c R and point indexing e N a' e B, there is a set Sk G such that e Sk c #. Since has a E-neighbourhood inside B by definition, the desired Sk is found by setting sk to any rational from for s > 0 sufticiently small, and then choosing rk e the open interval (0,1as), 5'(a',/) as is possible by 1.10. Now for each x e UsewB choose a member of say Skp, satisfying x e Skp i B for any B e C. Letting k(x) be the smallest index which satisfies the requirement gives an unambiguous choice. The distinct members of this collection form a set that covers Ussc#, but is a subset of and hence countable. Labelling the indices of this set as k1,k2,..., choose Bi as any member of C containing Ski. and hence also for Usecs. w Clearly, U:'xlffis a countable covering for U7=15k, ,/
=
=
,/
.$
.z'
.x
.$,
,/
lt follows that, if C is a covering for a set in R, it contains a countable subcovering. This is sometimes called the Lindelf property. The concept of a covering leads on to the crucial notion of compactness. A set A is said to be compact if every open covering of A contains afinite subcovering. The words that matter in this definition are and Any open covering that has R as a member obviously contains a finite subcovering. But for a set to be compact, there must be no way to construct an inrducible, infinite open covering. Moreover, every interval has an irreducible infinite cover, consisting of the singleton sets of its individual points; but these sets are not open. Gevery'
Eopen'.
2.8 Example Consider the half-open interval (0,11.An open covering is the eountable collection (41/n,1j, n e NJ lt is easy to see that there is no finite subcollection covering (0,11 in this case, so (0,1j is not compact. EI .
A set A is bounded if zt i 5'(x,s) for some x e A and ir > 0. The idea here is that
Limits and Continuity
23
: is a possibly large but tinite number. In other words, a bounded set must be containable within a finite interval. 2.9 Theorem A set in R is compact iff it is closed and bounded. u This can be proved as a case of 5.12 below, and provides an alternative definition of compactness in R. The sufticiency part is known as the Heine-Borel theorem. A subset B of A is said to be dense in A if B c A q X. Readers may think they know what is implied here after studying the following theorem, but denseness is a slightly tricky notion. See also 2.15 and the remarks following before coming to any premature conclusions.
2.10 Theorem Let A be an interval of R, and C c A be a countable set. Then A C is dense in A. -
Proof By 1.7, each neighbourhood of apoint inA contains an uncountable number of points. Hence for each e A (whetheror not e C), the set (A C) f-h 5'(x,:) is not empty for every e > 0, so that is a closure point of A C. Thus, A C c (A C) QJ C A ? (A C) K .x
.v
-
.x
-
=
-
-
-
.
The k-fold Cartesian product of R with copies of itself generates what is called Euclidean k-space, R The points of Rk have the interpretation of k-vectors, or ordered k-tuples of real numbers, x (x1,.n,...,.q)'.All the concepts defined above for sets in R generalize directly to R The only modification required is to replace the scalars x and y by vectors x and y, and define an e-neighbourhood be the Euclidean distance betweenx andy, where 1Icll in a new way. Let Ilx 11 1/2 IEt,-1 , q is the length of the vector a (J1,...,JJ' and then define S(x,z) < el, for som s > 0. An open set A of R2 is one in which every point (y: Ilx 11 x e A can be contained in an open disk with positive radius centred on x. In R3 the open disk becomes an open sphere, and so on. .
=
.
-y
=
=
=
-y
2.2 Sequences and Limits A real sequence is a mapping from E$into R. The elements of the domain are called the indice- and those of the range variously the terms, members, or coordinates of the sequence. We will denote a sequence either by tak,n e EN), or more briefly by IakI:', or just by (xnJ when the context is clear. (xnl'kois said to converge to a limit x, if for every : > 0 there is an integer Nv for which
Ixn
-
xI
<
e for all n > Nv.
Write xn it ij often x, or = limu-jxak. When a sequence is tendinygo +x or said to diverge, but it may also be said to converge in R, to distinguish those cases when it is does not approach any fixed value, but is always wandering. increasing, non-increasing, or decreasA sequence is monotone (non-decreasing, ing) if one of the inequalities ak f xn+1, xn < xn+1, xn k ak+1, or xn > xnx.l holds for every n. To indicate that a monotone sequence is converging, one may write for emphasis either xn a7 will or xn t, x, as appropriate, although xn -.->
.x
-x
'1'
.x
--
24
Mathematics
also do in both cases. The following result does not require elaboration. 2.11 Theorem Evel'y monotone sequence in a compact set converges.
n
A sequence that does not converge may none the less visit the same point an infinite number of times, so exhibiting a kind of convergent behaviour. If fak,n e N l is real sequence, a subsequence is fxnk,k e INl where fnk, k e N ) is any increasing sequence of positive integers. If there exists a subsequence taksk e (h1 ) and a constant c such that xnk (-1)N, c, c is called a cluster point of the n 1,2,3,...) does not converge, but sequence. For example, the sequence f subsequence obtained by only taking the even values of n converges trivially. c is usually a finite constant, but +x and may be cluster points of a sequence if we If the of notion allow convergence in F. a subsequence. is convergent, then so is ) where fmk3 is an any subsequence of the subsequence, defined as (x,,u,k e EN whose members of fnkl members are also increasing sequence The concept of a subsequence is often useful in arguments concerning convergence. A typical line of reasoning employs a two-pronged attack', first one identities a convergent subsequence (a monotone sequence, perhapsl; then one uses other characteristics of the sequence to show that the cluster point is actually a limit. Especially useful in this connection is the knowledge that the members of the sequence are points in a compact set. Such sequences cannot diverge to infinity, since the set is bounded; and because the set is closed, any limit points or cluster points that exist must be in the set. Specifically, we have two uscful --
=
-x
.
results.
2.12 Theorem Every sequence in a compact set of R has at least one cluster point. Proof A monotone sequence converges in a compact set by 2.11. We show that ) hasamonotonesubsequence.Defineasubsequence (xnk) evel'y sequence (ak,ne EN follows. for 1,2,3,... let if S'et 1, and k there nl xnk..! sup,on:ak as exists a finite ?7z+1 satisfying this condition', otherwise let the subsequence terminate at nk. This subsequence is non-increasing. If it terminates, the sub?u) must contain a non-decreasing subsequence. A monotone sequence (.v, n k subsequence therefore exists in every case. w =
=
=
2.13 Theorem A sequence in a compact set either has two or more cluster points, or it converges. Proof Suppose that c is the unique cluster point of the sequence fxn) but that xn zd-yc. Then there is an infinite set of integers (nz, L e EN) such that Iau c I s for some : > 0. Define a sequence fyk)by setting yz xnk. Since (yz)is also a sequence on a compact set, it has a cluster point c' which by construction is different from c. But c' is also a cluster point of fx,,1, of which (ykl is a subsequence, which is a contradiction. Hence, xn c. w ,
-
=
-->
3 ,...,x
n 2.14 Example Consider the sequence ( 1,x,x2 or more j orma jj y ta.gl where is a real number. In the case IxI< 1, this sequence converges to n e EN()1, The condition specified zero,( I.x''lJ being monotone on the compactinterval g0,11. ,a:
.'t7
,...),
,
Limits and Continuity
25
in (2.2) is satisfied for Nc = logtel/log IxI in this case. If x = 1 it converges it to 1, trivially. lf > 1 it diverges in R, but converges in 2-to +x. If x +1 diverges, but oscillates between cluster points and neither converges nor -1
.x
=
-1.
Finally, if x < 1 the sequence diverges in R, but does not converge in Ea mately, it oscillates between the cluster points +x and -
F. Ulti-
-x.
We may discuss the asymptotic behaviour of a real sequence even when it has no limit. The superior limit of a sequence fxn) is
limsup xn n
=
inf sup xm. n
(2.3)
m kn
(Alternative notation'. limn xn.) The limsup is the eventual upper bound of a sequence. Think of (sup-znau, n 1,2,... J as the sequence of the largest values the sequence takes beyond the point n. This may be +x for every n, but in all cases it must be a non-increasing sequene having a limit, either +x or a finite real number; this limit is the limsup of the sequence. A link with the corresponding concept for set sequences is that if xn supA,, for some sequence of sets (4n c R ) then limsup xn supA, where A limsupsAn. The inferior limit is defined likewise, as the eventual lower bound: =
=
=
,
liminf xn
=
=
-
limsupt-ak)
=
sup inf xm, n
n
(2.4)
mkn
also written limnak. Necessarily, liminfaak ; limsupnak. When the limsup and liminf of a sequence are equal the sequence is convergent, and the limit is equal the sequence converges in V. to their common value. If both equal +x, or The usual application of these concepts is in arguments to establish the value of a limit. lt may not be pennissible to assume the existence of the limit, but the limsup and liminf always exist. The trick is to derive these and show them to be equal. For this purpose, it is sufticient in view of the above inequality to show liminfn xn limsupn xn. We often use this type of argument in the sequel. To determine whether a sequence converges it is not necessary to know what the limit is; 4he relationship between sequence coordinates the tail' (as tl criterion for converbecomes large) is sufficient for this purpose. The Cauchy > iff for real 0 of that Ne such ('E'i states every e: fakl converges sequence gence a that Ixn xm I < : whenever n > Nc and m > Ne. A sequence satisfying this criterion is called a Cauchy sequence. Any sequence satisfying (2.2)is a Cauchy sequence, and conversely, a real Cauchy seqqence must possess a limit in R. The two definitions are therefore equivalent (in R, at least), but the Cauchy condition may be easier to verify in practice. The limit of a Cauchy sequence whose members a1lbelong to a setA is by definition a closure point of A, though it need not itself belong to A. Conversely, for every accumulation point of a set A there must exist a Cauchy sequence in the set whose limit is x. Construct such a sequence by taking one point from each of the sequence of sets, -x,
-
.7
(A rn S(x.$In4, n
=
1,2.3,..1,
26
Mathetnatics
none of which are empty by definition. The term limit point is sometimes used synonymously with accumulation point. The following is a fundamental property of the reals.
2.15 Theorem Every real number is the limit of a Cauchy sequence of rationals. Proof For finite n 1et xn be a number whose decimal expansion consists only of zeros beyond the nth place in the sequence. lf the decimal point appears at position m, with m > n, then xn is an integer. If m f n, removing the decimal /10'J-'N point produces a tinite integer a, and xn a so ak is rational. Given any real x, a sequence of rationals (ak) is obtained by replacing with a zero every beyond the ath, for n = 1,2,... Since digit in the decimal expansion of jxa..l l < 10-/, lak) is a Cauchy sequence and xn x as n x. w =
.v
-->
.--->
-xn
The sequence exhibited is increasing, but a decreasing sequence can also be conIf x is structed, as f-yn) where (yn) is an increasing sequence tending to itself rational, this construction works by putting ak = x for every n, which trivially defines a Cauchy sequence, but certain arguments such as in 2.16 below depend on having xn # x for every n. To satisfy this requirement, choose the terminating' representation of the number', for example, instead of 1 take This does not 0.9999999..., and consider the sequence (0.9, 0.99, 0.999, work for the point 0, but then one can choose f0.1, 0.01, 0.001,...1. of a real One lnterestingcorollary of 2.15 is that, since every E-neighbourhood number must contain a rational, (E)is dense in R. We also showed in 2.10 that R (D is dense in R since Q is countable. We must be careful not to jump to the concluAnother version sion that because a set is dense, its complement must be of this proof, at least for points of the intelwal E0,1), is got by using the binary expansion of a real number. The dyadic rationals are the set -x.
<non-
...)
.
-
,
tsparse'
.
D
=
(ilzn,
i
=
1,...,2* 1, n c EN )
(2.5)
-
.
The dyadic rationals corresponding to a finite n define a covering of E0, 11 by 1/2N, which are bisected each time n is incremented. For any x intervals of width e g0,1J, a point of the set f illn' i = 1,,..,22 1 ) is contained in 5'(x,eJfor : < 2/21 so the dyadic rationals are dense in (0,11. D is a convenient analytic tool when we need to detine a sequence of partitions of an interval that is becoming dense in the limit, and will often appear in the sequel. Another set of useful applications concern set limits in R -
.
2.16 Theorem Every open interval is the limit of a sequence of closed subintervals wiih rational endpoints. Proof If a,b) is the interval, with a < &, choose Cauchy sequences of' rationals an a and bn b, with tzI < l)j (alwayspossible by 1.10). By definition, for for all n k N, and hence every x e anbj there exists N k 1 such that x gtzr,,lul > anb) i liminfnlln,lul. On the other hand, since an a and b > bn, abbjc (Jn,:,,1f for all n 2 1, so that a,bjc i liminfagtzn,hnlc. This is equivalent to limsupagtzn,:nj atb). Hence limnltzn,:nl exists and is equal to a,b). . .1e
''
Limits and Continuit.y
27
This shows that the limits of sequences of open sets need not be open, nor the limits of sequences of closed sets closed (takecomplements above). The only hard and fast rules we may 1ay down are the following corollaries of 2.4(i): the limit of a non-decreasing sequence of open sets is open, and (bycomplements) the limit of a non-increasing sequence of closed sets is closed.
2.3 Functions and Continuity hfunction of a real variable is a mapping J:S F-->T, where S c R, and 'F c R. By specifying a subset of R as the codomain, we imply without loss of generality that /S) = T', such that the mapping is onto 7. Consider the image in F, under f, of a Cauchy sequence fxnlin S converging to .x. lf the image of every such sequence converging to x e S is a Cauchy sequence the function is said to be continuous at Continuity is in 'F converging to formally defined, without invoking sequences explicitly, using the 6r approach. < # is continuous at the point x e S if for any e > 0 D > 0 such that Iy whenever < The choice here depend S. of implies lfl may on x. ye fx) I e, lf f is continuous at every point of S, it is simply said to be continuous on S. Perhaps the chief reason why continuity matters is the following result. .f@),
.x.
-
-xl
-
2.17 Theorem lf J: S F-> 'T is continuous at a1l points of S, f -1(A) j s open ju whenever A is open in 7J, and f -1(4) is closed in S whenever 4 is closed in F.
s
(:a
This important result has several generalizations, of which one, the extension to vector functions, is given in the next section. A proof will be given in a still more general context below; see 5.19. Continuity does not ensure that J(A)is open whenA is open. A mapping with this fAC) J(X)C in general, we # property is called an open mapping, although, since cannot assume that an open mapping is also a closed mapping, taking closed sets to clpsed sets. However, a homeomorphism is a function which is 1-1 onto, continuous, and has a continuous inverse. If f is a homeomorphism so is f and hence by 2.17 it is both an open mapping and a closed mapping. It therefore preserves the structum of neighbourhoods, so that, if two points are close in the domain, their images are always close in the range. Such a transformation amounts to a relabelling of axes. If fx + ) has a limit as h i 0, this is denoted f@+4.Likewise, J@-) denotes the limit of f(x ). It is not necessary to have x e S for these limits to exist, but if fx) exists, there is a weaker lkotion of cpntinuity at x. f is said at the point x e S if, for any s > 0, R 6 > 0 such that to be right-continuous and < 8 whenever 0 f x- h e S, .j
,
-
Iflx-h) fx) I < -
lt is said to be ever 0 < h <
(2.6)
E.
left-continuous at x if, for any e: > 0, 3 8 > 0 and x e S,
wch that
when-
-
If@) J@ ) I < -
-
z.
(2.7)
Mathematics
28
Right continuity at x implies J@) J@+)and left continuity at x implies fx) = the function is continuous at x. fx-). If fx) f(x+) Continuity is the property of a point x, not of the function f as a whole. Despite continuity holding pointwise on S, the property may none the less break down as certain points are approached. =
.f@-),
=
=
2.18 Example Conjider fx)
1/x, with S
=
=
T'
=
(0, x).
iff
<
For 6: > 0, 1
Iflx- ) fx) I -
=
x(x+o
and hence the choice of depends on both 0. l:l 0, but not in the limit as that
J: S
F->
sx
-
flx) is continuous
continuous if for every E
'F is unformly
Ix-yl
6: and x.
j
for all x
>
-->
.x
The function
< E
<
=
Ifx,
>
0 H
>
0 such
-.f(A') 1< e:
(2.8)
for every x,y e S. ln 2.18 the function is not uniformly continuous, for whichever is chosen, we can pick x small enough to invalidate the detinition. The problem arises because the set on which the function is defined is open and the boundary point is a discontinuity. Another class of cases that gives difticulty is the one x. where the dofnain is nbounded, and continuity at x is breaking down as x -->
However, we have the following result.
2.19 Theorem lf a function is continuous everywhere on a compact sets, then it is bounded and uniformly continuous on S. In (For proof, see 5.20 and 5.21.) Continuity is the weakest concept of smoothness of a function. So-called Lipschitz conditions provide a whole class of smoothness properties. A function f is said to satisfy a Lipschitz condition at a point x if, for any y e 5'@,) fot some 8 > 0, H M > 0 such that
If@l fx, I K -
ftlx
-yI )
(2.9)
R+ F-> R+ satisfies hld) 1 0 as d 1, 0. f is said to satisfy a uniform Lschitz condition if condition (2.9)holds, with fixed M, for a1l x,y e S. The Continuity (resp.unifonn type of smoothness imposed depends on the function uniform Lipschitz) property for any continuity) follows from the Lipschitz (resp. choice of h. Implicit in continuity is the idea that some function (.): R+ F-> R+ exists satisfying 6(E) 1 0 as 6: 1 0. This is equivalent to the Lipschitz condition holding for some (.),the case h = 8-1 B y imposig some degree of smoothness on h making it a psitive power of the argument for example we impose a degree of smoothness on the function, forbidding sharp corners' The next smoothness concept is undoubtedly well known to the reader, although differential calculus will play a fairly minor role here. Let a function J: S F-> 'F be continuous at x e S. lf
here
W
:
.
.
-
-
.
29
Limits and Continuity
fx-h) fx) h -
/:(x)
=
lim i0
(2.10)
exists, A'@)is called the left-hand derivative of f at x. The right-hand derivative, /1(x), is detined correspondingly for the case h 0. lf J+'(x) f-'(x),the common value is called the derivative of f at x, denoted f'x) or df/dx, and f is said to be dterentiable at x. lf J': S F-.> R is a continuous function, f is said to be continuously dterentiable on S. A function f is said to be non-decreasing (resp. increasing) if f@) flx) (resp. f@)> Ax))whenever y > x. lt is non-increasing (resp.decreasing) if is non-decreasing (resp.increasing). A monotone function is either non-decreasing or non-increasing. 'When the domain is an interval we have yet another smoothness condition. A function f.. (J,:1 F-+ R is of bounded variation if 3 M < x such that for every < xn b, by tinite collections of points a partition of gt7,/71 xo < x1 < '1'
=
-f
=
...
=
n
< M. lfxi) .f(.xf-.1)I 77 k=1
(2.1 1)
-
2.20 Theorem If and only if f is of bounded variation, there exist non-decreasing functions h and h such that f n .42
.f1.
=
-
(For proof see Apostol 1974: Ch. 6.) A function that satisfies the uniform Lipschitz condition on gJ,hl with ( I.x y I) Ix y I is of bounded variation on =
-
-
(J,:1.
2.4 Vector Sequences and Functions A sequence (xs) of real k-vectors is said to converge to a limit x if for every E > 0 there is an integer Nc for which
-x1I< Ilxn
E
for all H > Ne.
The sequence is called a Cauchy sequence in R iff Ne and m > Nv. A function
fl S
i--y
(2.12)
-auIl< e Ilxn
whenever
n
>
1-,
where S c Rk an d 7 R associates each point of S with a unique point of 7. 1ts graph is the subset of S x'r consisting of the k + ll-vectors (x, fx) ) for each x e S. f is continuous at x l S if for any E > 0 H > 0 such that ,
,
#
IIII <
=
I/.(x+)
-
fx) I < e
(2.13)
whenever x + b e S. The choice of may here depend on x. On the other hand, f is uniformly continuous on S if for any E > 0, 3 > 0 such that
IIII<
c:':>
sup x e S,x+: c S
l/'tx+h)
-.f@)
I<
e.
(2.14)
Mathematics
f (h,...,fm)'of functions of x is called, simply enough, a vector function.2 c on tinuity concepts apply element-wise to f in the obvious way. The A vector
=
function
f. . s
F-.
s sc ,
R -1
is said to be one-to-one if there exists a vector function f : S F-.-:hS, such that f -1(/.1)) x fOr each x e S An example of a 1-1 continuous function is the affine transformation 3 ./12) X2 + b = for constants b k x 1) and A (kx k) with IA l 0, having inverse function/-l -1 does not possess a closed form, A (y b4 ln most other cases the but there is a generalization of 2.17, as follows. =
.
=
./-1/)
:y:
-
.
2.21 Theorem Iff / S F- 1- is continuous, where S i Rk an d 1- i R'/ f-1A) is open in S when A is open in I-, and.f -1(A) is closed in S when A is closed in -T. (:1 ,
2.5 Sequences of Functions 7, 'T c R, be a function, where in tb-is case fl may be an arbitrary set, not necessarily a subset of R. Let f/'n,n e (N) be a sequence of such functions. If there exists f such that, for each e f, and E > 0, R Nvjl such that 1Jnt)l J((J)) l < e:when n > Asro,then fn is said to converge to f,pointwiseon f1. As for real sequences, we use the notations fn f, fn f, or fn 1 J, as appropwhere in the latter case the monoriate, for general or monotone convergence, tonicity must apply for eveyy (.t) e f2. This is a relatively weak notion of convergence, for it does not rule out the possibility that the convergence is breaking down at certain points of f1. The following example is related to 2.18 above. Let
fn1f
F-
-
'1'
-->
2.22 Example Let fnx) (0,x) is 1/x. But
=
nllnx
+
1),
fnlxt -
-
.x
1
e
=
x
(0,x). The
pointwise limit of
fnx) on
1 xtrl-x + 1)
,
and llxNcxx + 1)) < e: only for Nu > (1/a 1)(1/x). Thus for given E, Nvx cxo 0 and it is not possible to put an upper bound on Nu such that as x lfnx) 1/xl < s, n 2 Ncx, for every x > 0. n -->
-
--
-
To rule out cases of this type, we define the stronger notion of unifrm convergence. If there exists a function f juch that, for each : > 0, there exists N such th a t
lfnkbt J((,))I < sup eD -
fn is said to converge
E when
to J unifonnly on f1.
n
>
N,
31
Limits and Continuity
2.6 Summability and Order Relations The sum of the tenns of a real sequence faklTis called a series, written X'izzlxa (or just EAk). The terms of the real sequence (X'=lau,n e (N) are called the partial sums of the series. We say that the series converges if the partial sums converge to a tinite limit. A series is said to converge absolutely if the monotone sequence (Z'=1IxmI n e Ehl converges. ,
T x? This 2.23 Example Consider the geometric series, Zy converges to 1/(1 x) when 1x1< 1, and also converges absolutely. It oscillates between cluster points and for other values of x it diverges. u 0 and 1 for x .1
-
.
-1,
=
2.24 Theorem lf a series converges absolutely, then it converges. Proof The sequence (X;.1lxm I n e EN)is monotone, and either diverges to +x or converges to a finite limit. ln the latter case the Cauchy criterion implies that + Ixn..mI + Ixn..mI 0 as m and n tend to infinity. since Ixn I + lxnl + inequality,4 +xn.vm I by the triangle lak + convergence of (X)=1au,n E EN ) follows by the same criterion. w ,
-->
....
....
....
An alternative tenninology speaks of summability. A real sequence taklT is said to be summable if the series Xzkconverges, and absolutely summable if ( Ixnll7is summable. Any absolutely summablesequenceis summableby 2.24,and any summable sequence must b converging to zero. Convergence to zero does not imply summability (see2.27 below, fof example), but convergence of the tail sums to zero is necessary and suftkient.
2.25 Theorem 1ff (xn)7is summable, Zrlzznx- 0 as n -->
n-1
-->
x.
Proof For necessity, write IZ,,=l.zu x ! f IZ,a=1au I + I m-nxm j sincefor any : > 0 < k there exists N such that IE,7ox,ul e: for n N, it follows that lZp1=laulS ,,=,14% Then n I +: < x.n Conversely, assume summability and let A Eklxn. IE#-1 Z,n=n-Ym X Y,,=1.<,a () as n .-.y x s
sx
.
=
-1
-->
=
.
-
.
A sequence fakJTis Cesro-summable if the sequence This is weaker than ordinary convergence.
i .26 Theorem lf (akl:' cbnverges to x, its Cesro
-1
n converges. f?z Eru=laulq
sum also converges to x. n
But a sequence can be Cesro-summable in spite of not converging. The sequence f (-1)/)7 converges in Cesro sum to zero, whereas the partial sum sequence (r=()(-1)''' J7converges in Cesro sum ttl (compare2.14). Various notations are used to indicate the relationships between rates of diverfJnl!' gence or convergence of different sequences. If (xnlTis any real sequence, is a sequence of positive real numbers, and there exists a constant B < x such that IakIIan ; B for a1l n, we say that xn is (at most) of the order of magnitude of an, and write xn Oa. If takIanl converges to zero, we write xn oan), 4nd s>y that xn is of smaller order of magnitude than an. an can be incfeasing or decr:aing, so this notation can be used to express an upper bound either on the fte of eropth of a divereent seauence. or on the rate of convergence of a =
=
32
Mathematics
sequenceto zero. followsfrom the
Here are some rules for manipulation of 0(.), whose proof definition. lf xn On a) and y,j Onq), then =
.;tn +ya x 'n =
=
=
t?tzynwelxtfts; )
(2. 15)
(?(Itz+f5)
(2.16)
,
xls onsq), whenever =
a'fz is defined.
(2.17)
An alternative notation for the case xn 0 is xn f< an, which means that there is a constant, 0 < B < x, such that xn f Ban for al1 n. This may be more convenient in algebraic manipulations. The notation ak an will be used to indicate that there exist N k 0, and t'inite constants A > 0 and B k A, such that infapvak lan) 2 A and supazxtak Ian) S B. This says that l and (Jnl grow ultimately at the same rate, and is different 0. Some f'rom the relation xn Oan), since the latter does not exclude xn Ian 1. authors use xn an in the stronger sense of xn lan -
txn
.-.-)
=
.-->
-
2.27 Theorem If (ak) is a real positive sequence, and xn 1+a. n (i) if a > then Zpual.u n then Erlxlau log n; (ii) if ('x ()(sl+a) (j mzmxm < then )Lxlau x (iii) if (x < an xx
n,
-
-1
-
,
-1
=
-
-1
(x
.
.
Proot By assumption there exist N 2 1 and constants A > 0 and B k A such that An < xn K Bnz for n k #, and hence Amnzmt < mnzxm S Bxmnzm. The limit x for different values of a defines the Riemann zetahmction of 1x.l?'?ztz as n and its rates of divergence for a k for a < are standard results', see e.g. of Apostol (1974:Sects. 8. 12-8.13). Since the sum terms from 1 to N-1 is tinite, their omission cannot change the conclusions. . ..4
-1
-1,
lt is common pratice to express the rate of convergence to zero of a positive real sequence in tenns of the summability of the coordinates raised to a given power. The following device allows some f'urtherrefinement of summability condiV as v x (0) for tions. Let U(v) be a positive function of v. If &(vx)/&(v) > 0 and < p < +=, U is said to be regularly vatying at inhnity zero). lf a .x x (0), 1 for x > 0 as v positive function Lv) has the property L(%'x)lLM) frl/W/y varying said Evidently, is slowly regularly it at zero). to be any varying function can be expressed in the form &(v) = vpfXvl,where L(v) is slowly varying. While the definition allows v to be a real variable, in the cases of interest we will have v n for n e N, with U and L having the interpretation of positive sequences. --
--
-x
-->
-->
=
2.28 Example (logvltxis slowly varying at infinity, for any
G.
u
On the theol'y of regular vqriation see Feller (1971),or Love (1977).The important property is the following. 2.29 Theorem If L is slowly varying at infinity, then for any k 1 such that
>
0 there exists N
33
Limits and Continuity
v -8
Atvl
<
<
v
8
a 11v
,
N
>
.
(2.18)
(a
Hence we have the following corollary of 2.27, which shows how the notion of a convergent power series can be refined by allowing for the presence of a slowly varying function. 'ixlak
2.30Corollary
< If xn OnLnl) then Ln) which are slowly varying at infinity. u =
x
for a11(x
and al1 functions
-1
<
On the other hand, the presence of a slowly varying component can affect the summability of a sequence. The following result can be proved using the integral test for series convergence (Apostol 1974: Sect. 8.12).
2.31 Theorem If ak
X& m=
l.x,,,
l/gntlog n) 1+8 j w ith
-
>
0 then ,
log 1og n. n
-
E=nzz lak
<
If 6
x.
=
0, then
2.32 Theorem (Feller 1971: 275) lf a positive monotone function &(v) satisfies &(v.x) f7(v) -where D is dense in R+, and 0
v(.x),al1 a7 e <
v(x)<
then
x,
(2.19)
D,
v@) =
.x
P
for
<
-x
p
<
x.
To the extent that (2.19) is a fairly general property, we can conclude monotone functions are as a rule regularly varying.
u that
2.33 Theorem The derivative of a monotone regularly varying function is regularly varying at cra. Proof Given &(v) U' (v) If f7(v)
-->
=
=
vPf-(v),
write
pvP-1f-(v)
+vPfJ(v)
=
vP-1(p1,(v)
+vr(v)).
0 there is no more to show, so assume liminfvfxv)
jfzotjvxxilj
Kdv
which implies f7(vx)/f7(v) U'Mx)
U,(v)
Ly'vl
=
x
...0
-
(v)
--
L
(v) fXv)
>
0. Then
(),
(2.21)
w
(2.22)
1. Thus,
p-lpfatvx) =
f-tv-lj
jf7,(vAl
(2.20)
+
vxc'tvxl
pAtvl +vA
,
(v)
--)
p
a7
.
#
2.7 Arrays Arguments concerning stochastic convergence often involve a double-indexing of elements. An array is a mapping whose domain is the Cartesian product of countable, linearly ordered sets, such as ENx ENor r x (N, or a subst thereof. A real double array, in particular, is a double-indexed collection of numbers, or, alternatively, a sequence whose members are real sequences. We will use notation juch as (fakt,t E z J, n e (N), or just fau) when the context is clear.
Mathematics
34
-?'
A collection of tinite sequences (fau, f 1,...,k ), n e IN) where kn x as n --+ x, is called a triangular array. As an example, consider array elements of the 1,...,n) is a real sequence. The question of form xnt ytln, where tys t whether the series ( :=1.x,,,, n e N l converges is equivalent to that of the Cesro convergence of the original sequence', however, the array formulation is frequently =
,
=
=
the more convenient.
2.34 Toeplitz's lemma Suppose (y,;) is a real sequence and yn .-.-: y. lf ((.'xkf, t= such that 1,...,lk ), n e ENl is a triangula.r array (a) xnt
0 as n
--
for each fixed /,
x
--
kn
(b)
lim n-yoo
X Iak,lS
C <
x,
>1
kn lim A'-lau1, =
n-+x
>1
k then E/i,lxn,y, --- y. Proof
Iyn -y
F or y
=
(c) can be
O ,
fyn) for any s > 0 5 Ne k 1 such that for (c), and then (b) and the triangle inequality,
By assumption on I < 8C. Hence by
,
kn
n
>
Nvs
kn
lim Xxna, -y
n-yx
omitted.
X-v/t-'w
lim
=
>1
-y)
>1
n-yx
Nz
Txntyt y)
< lim
-
+
s
wj
n--kx
in view of (a). This completes the proof, since
E
(2.23)
:,
=
is arbitrary.
x
A particular case of an array fxn/)satisfying the conditions of the lemma is xnt where S = (Z''=1y,)-1y/, ty/lis a positive sequence and Z'lsxlys A leading application of this result is to prove the following theorem, a fundamental tool of limit theory. -->
=.
2.35 Kronecker's Iemma Consider sequences (tz,)';'and x, If Ll=jxtlat C < cxa as n numbers, with at -?'
tx/lTof
positive real
-->
=.
--
1
n
-Xx,
Jn
0.
--)
(2.24)
>1
Proof Detining co 0 and cn Xntcjxtlat for n G N, note that xt atct cr-1), 1,...,n. Also define tzo 1,...,n, so that an 0 and bt at fh-l for t t ntzzkbt. Now apply the identity for arbitrary sequences ao,...,an and co,...,cn, =
=
=
77(c,,.-1 Abel's known
c,-1)
as
=
=
-
-
=
n
n
(This is
=
=
=
77(tz,-1 zflcr-l -
+
ancn
-
zoco.
,-1
partial
summation formula.) We obtain
(2.25)
Limits and Continuit.v
1 n -- X xt an >1
n
1
X atlc t an
=
-
>1
= cn
35
1
-
-
ct- 1)
w-xn
--Lbtct-j an
-->
C- C
=
(2.26)
0
>1
is by the Toeplitz lemma, setting xnt
where the convergence
=
bt lan.
.
The notion of array convergence extends the familiar sequenceconcept. Consider subsequences, collection of full generality k (N for a (fxmnk,e l m e EN), an array where fnk, k e INl is an increasing sequence of positive integers. If the limit xm limz--ycxaAun: exists for each m e IN,we would say that the array is convergent; and = its limit is the infinite sequence (x,,,,m e N 1.Whether this sequence converges is a separate question from whether it exists at all. Suppose the array is bounded, in the sense that supkmlxmpulf # < x. We know by 2.12 that for each m there exists at least one cluster point, say xm, of the inner sequence (xaw,k e IN1. An important question in several contexts is this: is it valid to say that the array as a whole has a cluster point?
2.36 Theorem Corresponding to any bounded array f(xrm,k e ENJ, m e N ) there k e N J m e EN exists a sequece (x,,l, the limit of the array ( tx,ug, J as k x, subsequence n:) of f for each m. where (rl:) is the same ,
-->
,
Proof This is by construction of the required subsequence. Begin with a conver1 be a su bsequence o f (nz) such that sj gent subsequence for m 1', let (nkJ x1. Next, consider the sequence (.n,n)J. Like f-n,s:J,this is on the bounded interval (-#,#), and so contains a convergent subsequence. Let the indices of this 1 be denoted f42:) and note that latter subsequence, drawn from the members of (?1k), Proceeding in the same way for each m generxl,nj xl as well as 12,4 ates an array (fn, k e IN1, m e INJ, having the property that lxjw, k e EN) is a convergent sequence for 1 S i S m. k k e ENJ in other wotds, take the first member of Now consider the sequence fns 1 th e secon d member of (rl2k), and so on, For each m, this sequence is a subtlkl, pzth point of the sequence onwards, and hence the sequence of (n'2Jfrom the k e EN ) sequence f.x,,,4,k k PZJis cnkergent. This means that the sequence (xmmj, k = setting rl11 requirement isfies of the theorem. w is convergent, so the f fnzl sat ...-)
.n
=
,
-->
.n.
--
'
,
is called the
,
.
3 s4easure
3. 1 Measure Spaces A measure is a set function, a mapping which associates a (possiblyextended) real number with a set. Commonplace examples of measures include the lengths, areas, in and volumes of geometrical tigures,but wholly abstract sets can be an analogous way. Formally, we have the following definition. <measured'
3.1 Definition Given a class 5 of subsets of a set f, a measure
jt:
; F->
R-
is a function having the following properties: (a) g,(A) k 0, al1 A e %. 0. (b) g,(0) (c) For a countable collection (Aj e F, j e ENl with Aj DA./' and UjAj e ;, =
p. UXy
=
)'
=
O for j
j'
#
X g,(Xy). C1 j
The particular cases at issue in this book are of course the probabilities of random events in a sample space f; more of this in Chapter 7. Condition (a) is optional and set functions taking either sign may be referred to as measures (see e.g. j4.4), but non-negativity is desirable for present purposes. A measurable space is a pair (f1,T) where f is any collection of objects, and 5 is a c-field of subsets of f. When (,T) is a measurable space, the triple (,T,g) is called a measure space. More than one measure can be associated with the measurable space (f1,F), hence the distinction between measure space and measurable space is important. Condition 3.1(c) is called countable additivity. lf a set function has the property
p,tA kJ#)
=
(3.2)
p.(z4)+p,(f)
for each disjoint pair A,#, a property that extends by iteration to finite collections A1,..., An, it is said to be jnitely additive. In 3.1 5 could be a field, but the possibility of extending the properties of g, to the corresponding c-tield, by allowing additivity over countable collections, is an essential feature of a measure.
If g,(f1) < cxn the measure is said to be hnite.And if Q UjLh where lfl is a countable collection of Y-sets, and g.tfl < x for each j, g. is said to be ; G-jinite. In particular, if there is c(./) and Lh l a collection .Y'such that =
,$
=
sieasure ./
(A fa #: # e for each j, g is said to be c-finite on (ratherthan on ;). If is a measurable space and (A,?L,g) is a measure space 11 for some A q @,(A,8L) g,(AC) restriction If in of (f,T,g,) to A. 0 (equivalentto this case called the g(A) gtf1l when p.tfl < cx)l A is called a supportof the measure. When A supports 2, the sets of ?L have the same measures as the corresponding ones of 5. A point e f with the property g,(( )) > 0 is called an atom of the measure. =
=
=
3.2 Example The case closest to everyday intuition is Lebesgue measure, m, on the measurable space (R,O), where tB is the Borel field on R Generalizing the b a to an notion of length in geometry, Lebesgue measure assigns r?z((J,:1) (J,!71. plausible Additivity is an intuitively interval property if we think of measuring the total length of a collection of disjoint intervals. .
=
-
Lebesgue measure is atomless (see3.15 below), every point of the line taking ((J,:1, O(u,,j,m) denote the restriction of measure 0, but ??z(R) x. Letting (R,f,pz) to a finite interval, m is a finite measure on (t7,:1.Since IR can be partitioned into a countable collection of finite intervals, m is c-finite. n =
additional properties may be deduced from the definition:
Sole
3.3 Theorem For arbitrary T-sets A, #, and (Ay,j e N J (i) A c # = g,(A) S g,(#) (monotonicity). (ii) gtA k.p #) + g,(A ch B4 g.(A)+ g(#). (iii) p,(UyAj) f Xg,(Ay) (countablesubadditivity).
,
=
Proof To show (i) note that A and B -A are disjoint sets whose union is #, by hypothesis, and use 3.1(a) and 3.1(c). To show (ii),use A wB A QJ (B -A) and B = (A chB) k.pCB A), where again the sets in each union are disjoint. The result Al and Bn follows on application of 3.1(c). To show (iii), detine #1 UP-14 An j. Note that the sets Bn are disjoint, that Bn ? An, and that U==lfj j UT=IAy. Hence, J =
-
=
-
./
=
=
=1
X
G)
U p,/=1
=
p,
X
oo
Xp,(l U#, /=1 =
j=1
< Xp,(A?).
(3.3)
.
j=1
This proof illustrates a standard technique of measure theory, converting a sequence of sets into a disjoint sequence having the same union by taking differences. This trick will become familiar in numerous later applications. The idea behind 3.3(ii) can be extended to give an expression for the measure of any tinite union. This is the inclusion-exclusion formula'. n
UA/
p,j=
#
n
Xp,(A)
-
1
j= 1
-
77p,(+rnAzl k+j
+
p.tAycAAkchA,l 77 k:gy:y;l + jttx
f'-'h 1 r''hA:2,
...
-
...
r''hXrj),
(3.4)
where the sign of the last term is negative if n is even and positive if n is odd, and there are ln 1 tenns in the sulp in total. The proof of (3.4)is by induction from 3.3(ii), substituting for the second term on the right-hand side of -
Mathematics
38
(,U.-j+) p.(-.-)
g, repeatedly, for n
-
+
p,
-j,'-s
-U.-,'.,) p.
(,
-.-)
(
-
(3.5)
f..h
,
1. n 2,..., 1. IN be n e ) a monotone sequence of T-sets with limit A e T. A set function on g, is said to be continuous if g,(z4,,) g.(A). -
-
(AU,
Let
.--.-:
3.4 Theorem A finite measure is continuous. Proof First 1et (An) be increasing, with A,,-I cAn, and A U*a=IA,,.The sequence (Bj, j G ) where #1 Aj, and Bj Aj -Ay-I for j > 1 is disjoint by eonstrucand tion, with Bj e @, Aa U7=1#j, =
=
,
=
=
p,(An)
=
Xp,(). j=1
The real sequence (g,(A,,)) is therefore monotone, and converges since it is bounded g,(A). above by g,4Q)< x. Countable additivity implies LT) g,(U7 O';=1A,,. Consider Alternatively, 1et (An) be decreasing, with z4n-l Q An and the increasing sequence lA)) determine g,(49 by the same argument, and use finite additivity to conclude that g(A) = gtf1l g,(A9 is the limit of g(An) =l#y.)
=1g,(#j)
=
:4
=
=
,
=
-
P.(fX -
P.(XnC).
K
The finiteness of the measure is needed for the second part of the argument, but the result that g,tAul g(A) when ytu A actually holds generally, not excluding the case g,(A) x. This theorem has a partial converse: '
-->
=
3.5 Theorem A non-negative uous is countably additive.
set function g, which is finitely additive and contin-
Proof Let (#n) be a countable, disjoint sequence. If A,, U7=l, the sequence (An ) is increasing, Bn f'nAn-l 0, and so gtAnl g,(#n)+ g,(An-I) for every n, by finite additivity. Given non-negativity, it follows by induction that (g,(An)Jis ZTJ =Ip,(), whereas continuity implies that g,(A) monotone. lf A U7=tBj, g(A) LLIUTJ * lB =
=
=
=
=
=
=
.
Arguments in the theory of integution often turn on the notion of a set. In a measure space (f1,T,g), a set ofmeasure zero is (simplyenough) a set M e 5 with g.(M) 0. A condition or restriction on the elements of Q is said to occur almost everywhere (a.e.)if it holds on a set E and f E has measure zero. lf more than one measure is assigned to the same space, it may be necessary to indicate which measure the statement applies to, by writing a.e.lgl or a.e.gvq as the case may be. fnegligible'
=
-
3.6 Theorem (i) If M and N are T-sets, M has measure 0 and N i M, then N has measure 0. (ii) lf (A&) is a countable sequence with g(Mj) 0, V j, then g(UjMj4 0. (iii) If fe)) is a countable sequence with g,(r.'pl 0, V j, then g,((UjA))9 = 0. .
=
=
=
Measure Proof
39
(i) is
an application of monotonicity', (ii) is a consequence of countable additivity', and (iii) follows likewise, using the second de Morgan law. .
In j3.2 and j3.3 we will be concerned about the measurability of the sets in a given space. We show that, if the sets of a given collection are measurable, the sets of the c-field generated by that collection are also measurable (the Extension Theorem). For many purposes this fact is sufficient, but there may be sets outside the c-field which can be shown in other ways to be measurable, and it might be desirable to include these in the measure space. In particular, if g(A) g(A) wheneverd c E c #. This is g,(#) it would seem reasonable to assign jtt'l equivalent to assigning measure 0 to any subset of a set of measure 0. The measure space (f1,T,jt) is said to be complete if, for any set E e 5 with g.(F) = 0, a11 subsets of E are also in T. According to the following result, every measure space can be completed without changing any of our conclusions except in respect of these negligible sets. =
=
3.7 Theorem Given any measure space (f,;,jt), there exists a complete measure TB, (f,TB,g), called the completion of (f1,F,p), such that 5 and g(F) space c F(F) for al1 E e %. In =
Notice that the completion of a space is defined with respect to a particular has a different completion for each measure measure. The measurable space (f1,@) that can be defined on it. Proof Let
AB J;B
denote the collection of a11 subsets of i-sets of pmmeasure0, and LF
=
c
D: FA F e
AB
for some E e
.41
.
0, any set F c E satisfies the criterion of (3.7)and so is in FB as the definition requires. For F e @B, let F(#) g,(F), where E is any F-set satisfying E F e AB. To show that the choice of E is immaterial, let Ej and E1 be two such If jt(F)
=
=
sets, and note that
p,(A'lA&) Since jttFl t-p F2)
=
g,((FAF1)A (FAF2))
=
=
0.
g(F1 A F2), we must conclude
p,(F1 r'hE
+
g,tflr-hE
2 p.tfl k p,(F1 r'hF2)
(3.8) that
for i 1 and 2, or, g,(F1) p.(&). Hence, the measure is unique. When F e T, we AB, confirming that the measures agree on T. can choose E F, since FAF= O e TB is a c-tield colgaining ?F.Choosing E F in (3.7) It remains to show that AFC FB. 8/ shows If F e ;B, thenFA F e ABfor E e 5 and hence Ec for Fe 5 c TB. And finally AB where Ec if Fj e J;B foj e (N,there EF e 5, and so Fc e e exist Ej e 5 for j e EN,such that Ejh Fj e AB. Hence =
=
=
=
=
e U1 A U/) c UIF/AFJ ?
? by 3.6(ii). This means that
AB,
j
Ojb 6
TB,
and completes the proof. w
(3.10)
Mathematics
40 3.2 The Extension Theorem
You may wonderwhy, in thedefinition of ameasurable space, Fcould not simply be the set of all subsets; the power set of D. The problem is to tind a consistent method of assigning a measure to every set. This is straightforward when the space has a tinite number of elements, but not in an infinite space where there is no way, even conceptually, to assign a specitic measure to each set. It is necessary to specify a rule which generates a measure for any designated set. The problem of measurability is basically the problem of going beyond constructive methods without running into inconsistencies. We now show how this problem can be solved for c-fields. These are a sufticiently general class of sets to cope with most situations arising in probability. One must begin by assigning a measure, to be denotedgo, to the members of some basic collection C for which this can feasibly be done. For example, to construct the measure b a. Lebesgue measure we started by assigning to each interval (tz,&1 We then reason from the properties of go to extend it from this basic coltection to a11 the sets of interest. C must be rich enough to allow go to be uniquely defined by it. A collection C c 5 is called a determning class for (f,F) if, whenever jt and v are measures on 5, jt(A) v(A) for all A e C implies that g v. Given C, we must also know how to assign gmvalues to any sets derived from rby operations such as union, intersection, complementation, and difference. For disjoint sets A and B w have g)tz4QJ #) go(A) + go(#) by finite additivity, and when B c A, gotA B) go(A) 04#). We also need to be able to determine gotA fa #), which will require specitic knowledge of the relationship between the sets. When such assignments are possible for any pair of sets whose measures are themselves known, the measure is thereby extended to a wider class of sets, to be is closed and C are the same collection, but in any event denoted 'Y'.Often under various finite set operations, and must at least be a semi-ring. In the is typically either a field (algebra)or a semi-algebra. Example applications 1.18 is a god case to keep in mind. However, cannot be a c-field since at most a finite number of operations are permitted to determine go(A) for any A e .Y'.At this point we might pose the opposite question to the one we started with, and ask why might not be a rich enough collection for our needs. ln fact, events of interest frequently arise which cannot contain. 3.15 below illustrates the necessity of being able to go to the limit, and consider events that are expressible only as countably infinite unions or intersections of C-sets. Extending to the events 5 c(#) proves indispensable. We have two results, establishing existence and uniqueness respectively. -
=
=
=
-
=
-
./
-t/'
,/
'f
,/
,$
=
3.8 Extension ,/.
theorem (existence)Let $ be a semi-ring, and let go:
lf 5 = c(,$). there exists a measure jt on measure on ,YI. jL0t'l for each E e n
./
F--y
F+be
(D,?D,such that jt(F)
a =
Although the proof of the theorem is rather lengthy and some of the details are fiddly, the basic idea is simple. Take an event A c f to which we wish to assign a
Measure ./,
,/,
consider choosing a finite we have g,(A) go(A). If A measure g,(z4).If A G that is, a selection of sets Ej e or countable covering for A from members of OjEj. The object is to tind as 1,2,3,... such that A c ./, j a covering as possible, in the sense that Xgo(f;)is as small as possible. The outer measure of A is =
,/,.
teconomical'
=
P,*(X)
inf X.to(F./),
=
(3.11)
where the infimum is taken over a11tinite and countable coverings of zl by y-sets.
If no such covering exists, set g,*(A) x. Clearly, g7(A) gotAl for each A E g,* is called the outer measure because, for any eligible definition of g,(A), g,*(A) 2
Xg,(l
2
p,
j
Uf j
.9.
=
=
2 p,(A), for Ej
,Y'.
E
(3.12) ,/
The first inequality here is by the stipulation that p(Ej) goIFJfor Ej e in side majorant where exists, is infinite. The the case a covering or else the second and third follow by countable subadditivity and monotonicity respectively, because p, is a measure. We could also construct a minimal covering for Ac and, at least if the relevant outer measures are tinite, define the inner measure of 4 as jt+ (4) jt(AC) p,*(z4C). and g,*(A9 k jttAfl by (3.12), Note that since p,(z4) g,tfl g,*(f) =
=
=
-
-
p,(A) 2 g,+(A).
(3.13)
lf g,*(4) g+44), it would make sense to call this common value the measure of A, and say that A is measurable. ln fact, we employ a more stringent criterion. A set A g f is said to be measurable if, for any B c f, =
g,*(Af-'h B4
+
g,*tz4f chB)
=
(3.14)
jt*(#).
This yields g,*(A) p,+(A) as a special case on putting B f1, but remains valid even if g,tfll x. Let A denote the collection of all measurable sets, those subsets of f) satisfying (3.14).Since g,*(A) go(A) forA e and /1.:40) 0, putting 4 O in (3.14) gives the trivial equality g,*(#) g,*(#). Hence O e A, and since the definition implies that Ac E A if 4 e A, f E A too. The next steps are to detennine what properties the set function jt*: A !-+ F shares with a measure. Clearly, =
=
=
,/
=
=
=
=
jt*(A)
0 for all 4
C f1.
(3.15)
Another property which follows directly from the definition of p,*is monotonicity: X1 I X2 =
F*(X1) f >*(X2), fOr A1, 42 i
.
(3.16)
Our goal is to show that countable additivity also holds for g,* in respect of A-sets, but it proves convenient to begin by establishing countable Juhadditivity. 3.9 Lemma If fAy,j
E
EN J is any sequence of subsets of f, then
42
Mathematics
g,* UX./ f
/7 >*(X./).
(3.17)
Proof Assume g*(Ay) < x for each j. (If not, the result is trivial.) For each j, 1et fEjkq denote a countable covering of Aj by y-sets, which satisfies p.*(A>+ 2-:
< Xpo(,) k
: > 0. Such a collection always exists, by the definition of g*. Since LkkEjk, it follows by definition that U#.j ?
for any
p.*UA/ noting Z7=12S ity is strict. .
=
1.
f XpotA>l
<
j,k
j
(3.17)
Xp,*()+ e,
(3.18)
./
now follows since : is arbitrary and the last inequal-
The following is an immediate consequence of the theorem, since subadditivity supplies the reverse inequality to give (3.14).
3.10 Corollary A is measurable if, for any B c D, g,*(AC r''h#) K g*(#). n t *(A r''h ) +
(3.19)
The following lemma is central to the proof of the extension theorem. lt yields countable additivity as a corollary, but also has a wider purpose.
3.11 Lemma A is a monotone
class.
Proof Letting fAj, j e INJ be an increasing sequence of A-sets converging to 4 U#.j, we show A e A. For n > 1 and E e f1, the definition of an A-set gives =
g7(An fa E)
=
g*(4n-1 ra (An rh A=))+ g*(4k-1 f-'h (Anf''h F))
= g*(An-1 rhEI+ g7(& rn E). where Bn g,*(A()rn E)
= =
An -An-1, and the sequence 0', then by induction, ke(An chF)
=
(3.20)
(BjI is disjoint.
X pz*( rn E)
Put
ztttl
=
O so that (3.21)
y=1
holds for every n. The right-hand side of (3.21)for n e INis a monotone real x. Now, since An e A, g,*(4rh A>)as n sequence, and g,*(An(hE) .--)
-->
jt*(F)
r1*(Anrnm
+
h1*(Xrn)
k jt*(An(nE)
+
g*tAc rnfl,
=
using the monotonicity of g,*and the fact that from the foregoing argument that
AC
(3.22) Anf.Taking the limit, we have
sleasure jt*(A7 k g7(A r'A F)
+
43
g.*(AC
f'-h A'),
(3.23)
so that 4 e A by 3.10. For the case of a decreasing sequence, simply move to the complements and argue as above. . Since () is a disjoint sequence, countable additivity emerges as a by-product of the lemma, as the following corollary shows.
3.12 Corollary
If
()
is a disjoint sequence of A-sets,
g,*U j Proof Immediate on putting E
-
Xp,*().
(3.24)
j
in (3.21) and letting n
=
--> x,
noting VjBj
A. w
=
Notice how we needed 3.10 in the proof of 3.11, which is why additivity has been derived from subadditivity rather than the other way about. Proof of 3.8 We have established in (3.15)and (3.24)that g* is a measure for the elements of A. If it can be shown that 5 q A, setting g,(A) g,*(A)for al1 A e 5 will satisfy the existence criteria of the theorem. implies The t'irst step is to show that c A or, by 3.10, that 4 e =
.t/7
,/
g,*(A'r7A) + g,*(A'r-hAf)< g*(F)
(3.25)
,$1
denote a finite or countable covering of E such that Lj gotAyl < p,*(F) + E, for s > 0. If no such covering exists, g,*(A-) cxl by definis a ition and (3.25)holds trivially. Note that E r'A A c U/A; r-7A), and since /''AAC f-7WC), and by simple semi-ring the sets Aj ('''AAare in ,Y'.Similarly, E c Uyt>j algbra definition of and the set a semi-ring,
for any E i (. Let lAy e
=
,/
Aj r'Aytf
=
Aj
Aj chAl
-
=
UG:
(3.26)
k
where the Cjk are a finite collection of yksets, disjoint with each other and also with Aj r'hA. Now, applying 3.9 and the fact that g,*(#) k)(#) for B e we find .$,
=
g,*(A'faA) +
g*(Ff-$AC)
f
X jtot/j chdl X XgotGl +
j
j
k
p,*(eD =Xw)(z4y) ? <
+ :,
(3.27)
is tinitely additive, and Aj DA where the equality follows from (3.26)becau and the Cjk are mutually disjoint. Since s is arbitrary, (3.25)follows. Next, we show that A is a c-field. We have only to show that A is a field, because 3.11 implies it is also a c-field, by 1.24. We lready know that e A and A is closed under complementation, so it remains to show that unions of A-sets are in M. Suppose that Al and A2 are A-sets and E c f. Then .to
:,*(13 =
>*(A1 nA') + F*(Af rhEI
Mathematics
=
jt*(z42 c7z4jra e) +
g*(z4qrndl chE) + g,*(A2/'nAk ra A')+ jt*(A rnAf
f-h
E)
p,*(A2f-7Al ra A')
+ g*((Aq r7Al ra F)
f-h F)) rn E) k.p (Aj/-AAC
k.p (A2 chAf
+p7((42ra4j)cfaF), = jt*((A2rn41) cjE)
(3.28)
where the inequality is by subadditivity, and the rest is set algebra. By 3.10 this is sufficient for A1 f'742 e A, and hence also for A1kJA2 G A, using closure under complementation. It follows that A is a c-tield containing $, and since 5 is the smallest such c-field, we have that 5 i X, as required. . Notice that (3.28)was got by using (3.14)as the relation defining measurability. The proof does not go through using p.*(A) gx(A) as the definition. The style of this argument tells us some important things about the role of .Y'. Any set that has no covering by '-sets is assigned the measure x, so for finite measures it is a requisite that f q LbEj for a tinite or countable collection fEj o? e #') The measure of a union of y-sets must be able to approximate the measure any F-set arbitrarily well, and the bagic content of the theorem is to establish that a semi-ring has this property. To complete the demonstraiion of the extension, there remains the question of uniqueness. To get this result we need to impose c-finiteness, which was not needed for existence. =
.
3.13 Extension theorem (uniqueness)Let jt and g,' denote measures on a space (f1,T), where 5 c(.$), and is a semi-ring. If the measures are c-finite on and g,(A') jt'(F) for a1l E e .T, then g(An) g'(F) for a1l E e %. .$
,/
=
=
=
Proof We
tirst prove
the theorem for the case of finite measures, by an applitheorem. Define .J= fE e F: p.(F) = p,'(AD ) Then $z ,J by hypothBy 1.27, the proof is completed esis. If is a semi-ring, it is also a l-system. contains c(#). show hence if Fe can that .J is a l-system, and and condition 1.26(a) holds. Additivity When the measure is finite, f e implies that, for A e
cation of
the 7:-
.
./
.4
.4,
F(X3
so that 1.26(b) additivity,
=
F(f)
-
XX)
=
LUIf) -
$(A)
=
>'(A3,
(3.29)
holds. Lastly, 1et fA.j) be a disjoint sequence in co
p,
UAp
j=3
X
oo -
77p,(A> j=
-
.4.
By countable
G)
Xp,-+) -
j=1
p,' UAk
(3.30)
,
y=1
and 1.26(c) holds. It follows by 1.26 and the 7:- theorem that 5 c(./) ? W. VjBj where Bj e and pBp Now consider the c-finite case, Let ( g,'(#j) < co. % fBj rnA: A e 5 J is a c-field, so that the (Bj,5) are measurable spaces, =
./
=
=
=
Measure
45 ./
fe'h Bj. The preceding argument on which jt and $ are tinite measures agreeing on showed that, for A e 5, g(#j rnz4) bt'Bj rnA) only if g and g,' are the same =
measllre.
Consider the following recursion. By 3.3(ii) we have p.(Ara (#1t..p#2)) Letting Cn
=
f''h #2) g.(X f''h #1 f''h #2). g,(z4r'7#1) + h1(z4
=
U7=l the
(3.31)
-
same relation yields
(3.32)
rn cn) p.tz4rn Bn) + p,tA ra cn-j) p,tz4rn Bn rn cn-1). g,ta4 =
-
The terms involving Cn-l on the right-hand side can be solved backwards to yield f-'h an expression for g,(A G), as a sum of terms having the general form g,(Ara Bjj rn Bjzra Bj, ra
for some j, say j that gtD ra Bj) that in (3.32) =
=
j, in which ease D
jt'(D ra B? for all D g.(Afa Cn)
olds for any n. Since Cn
This
=
G
=
=
=
g,to ra Bj)
(3.33)
< x
A fn Bh fe'h Bh fa e F. Since we know preceding by 5 the argument, it follows ...
g,'(4 fa Cn4. f as n
--)
F(A)
)
...
-->
x,
(3.34) we obtain in the
F'(A),
limit
(3.35)
the two sides of the equality being either finite and equal, or both equal to This completes th proof, since A is arbitrary. w
+x.
3.14 Example Let A denote the subsets of R which are measurable according to (3.14) when g,* is the outer measure defined on the half-open intervals, whose measures go are taken equal to their lengths. This defines Lebesgue measure m. 'hese sets fol'm a semi-ring by 1.18, a countable collectioh of them covers R, and the extension theorem shows that, given m is a c-finite measure, A contains the Borel field on R (see 1.21), so (R,f,?n) is a measure space. It can be shown (we wonpt) that a11 the Lebesgue-measurable sets not in B are subsets of T-sets of includes a1l measure 0. For any measure g, on (R,f), the complete space @,SB,F) of the Lebzsgue-measurable sets. In The following is a basic property of Lebesgue measure. Notice the need to deal with a countable intersection of intervals to determine so simple a thing as the measure of a point. 3.15 Theorem Any countable set from R hs Lebesgue measure 0. Proof The measure
of a point
(.x) is
zero, since for x e R,
@Q
fx) ppd :
.
'
'
??z((<))q
'
: ''
' '
=
limn-jxl/n
=
=
r)
n=1
(x
-
1/n, xj e B
0. The result follows by 3.6(ii). w
(3.36)
Mathematics
46 3.3 Non-measurability
To give the ideas of the last section their true force. it needs to be shown that D X c 2 i s poss ible in other words, that ( can contain non-measurable subsets. In this section we construct such a set in the half-open unit interval (0,11,a standard counter-example from Lebesgue theory. For m y e (0,11, define the operator ,
.#4.x
A'+
1,
(3.37)
=
y+x- 1,
This is addition modulo 1. Imagine the unit interval mapped onto a circle, like a clock face with 0 at the top. y x is the point obtained by moving a hand clockwise through an angle of 2= from an initial point y on the circumference. For each set A (0,11, and x e (0,1J, detine the set .i.
A J.x
=
(y 4.x:
AJ.
l
y
(3.38)
3.16 Theorem If A is Lebesgue-measurable so is A %x, and m(A 4.a') any
=
mA), for
X.
for any real x such b +x1) b c lna,bj), unions > A-x + of intervals extends < 0 and b 1. The that a x to finite property translated by x. If A is any Lebesgue-measurable subset of (0,11,and A A-x (0,11 where A + x (y+x: y e A J, the construction of the extension similarly is measurable and mA) mA +x). implies that A and A2 A f-h (1 Now let A1 A /'7 (0, 1 11. Then pztdl +x) -(A1) and mAl + x 1) mA1), where the sets on the left-hand sides of these equalities are in each case contained in (0,11. A1 +x and A2 +x 1 are disjoint sets whose union is A %x', and hence Proof For
(J,!71
(0,11,ma
i
+x,
=
=
-
=
..x
=
-x),
-
-x,
=
=
=
=
-
mA 4.x)
rntz4l +x)
=
+
mAz
= rn(A1) + mA
+
=
x
-
1)
m(A). w
(3.39) .i.
Define a relation for points of (0,1q, letting xRy if y = x r for r e :. That is, xR y if y is separated from x by a rational distance along the cifcle. R is an equivalence relation. Defining the equivalence classes Ex
=
fy: y
.i.
=
.x
r, r e
t1),
(3.40)
(fI,
the sets of the collection x e (0,11 ) are either identical, or disjoint. distance is rational from some other point of the interval, these Since every x a sets cover (0,11.A collection formed by choosing just one of each of the identical sets, and discarding the duplicates, is therefore a partition of (0,11. (FA, Write this as x 6 C), where C denotes the residual set of indices. Another example may help the reader to visualize these sets. In the set of integers, the set of even integers is an equivalence class and can be defined as 0 the Ket of inteeers which differ from 0 an even integer. Of course F0 El = =
sfeasure
47
Eln, for any n l Z. The set of odd integers can be detined similarly as F2n+1 E 1 the set o f integers differing by an even integer from 1. F1 = F3 (A'A, for any n e Z. Discarding the redundant members of the collection e 11 F1 0 collection LE partition of the Z. d leaves just J to e tine a Now constnlct a set H by taking an element from Ex for each x e
#
=
=
...
=
,
...
= .x
,
3.17 Theorem H is not Lebesgue-measurable. Proof Consider the countable collection (H # r, r l (I)) We show that this collection is a partition of (0,11. To show disjointness, argue by contradiction. Suppose z e H rl and z e H r2, for rl # r2. This means there are points 1, hl e H, such that .
'i.
.i.
.i.
l rl
.i.
=
z
=
(3.41)
hz r2.
If rl # r1, we cannot have Jll = 2; but if Jll # hl then Jll and hz belong to different equivalence classes by construction of H, and cannot be a rational distance lrl r2 I apart; hence no z satisfying (3.41)exists. On the other hand, LjrH r), and consider any point let H e (0,11. belongs to one of the rational distance of some element of H; equivalence classes, and hence is within a but H* contains al1 the points that are a rational distance r from a point of H, for some r, and hence x e H., and it follows that (0,11c S*. mlH4 for all r. Since mH*) k Suppose mllt exists. Then by 3.16, mH% r?z((0,1J) 1, we must have m+ > 0 by 3.6(ii), but countable additivity then rnzbul'-l r) which is impossible. It follows that m+ does not gives mH.) -
.i.
.&
.x
=
=
=
=
=
=,
exist. w The definition of H involves a slightly controversial area of mathematics, since the set o f equ ivalence classes is uncountable. It is not possible to devise, even in principle, constructive rules for selecting the set C, and elements from Ex for each e C. The proposition that sets like H exist cannot be deduced from the axioms of set theory but must be asserted as an additional axiom, the so-called axiom of choice. lf one chooses to reject the axiom of choice, this counterexample fails. We have made no attempt here to treat set theol'y from the axiomatic standpoint, and the theory in Chapter 1 has been what is technically called nat've is). For us, the problem of (i.e. based on the intuitive notion of what a the axiom of choice reduces to the question: should we admit the existence of a mathematical object that cannot be constructed, even in imagination? The decision is ultimately personal, but suffice it to say that most mathematicians are willing do SO Sets like H do not belong to (0,1j = f# r'h(0,1j, B e B ). It is not difficult to show that a11the sets of (o,1j are Lebesgue-measurable', see 3.14 and restrict m to (0,1) as in 3.2. By sticking with Borel setf we shall not run into measurability difficulties on the line, but this example should serve to make us careful. ln less familiar situations (suchas will arise in Part VI) measurability cah fail in superficially plausible cases. However, if measurability is in doubt one might remember that outer measure g,*is well defined for a1l subsets of f, and coincides vk
tset'
X
to
.
Mathematics
48
with z whenever the latter is defined. Sometimes measurability problems are dealt with by working explicitly with outer measure, and forgetting about them.
3.4 Product Spaces If (f1,T) and (E,T) are two measurable fxE
((,():
=
(t) l
spaces, 1et
f,
(
G
El
(3.42)
be the Cartesian product of f and E, and define ?F@V Spg
=
((Fx G), F e @, G
G
=
c(Spg), where
V).
(3.43.)
a product space, and (f,F) and (E,W) are the coordinate factor spaces, or spaces, of the product. The elements of the collection Sys are called the measurable rectangles. The rectangles of the Euclidean The space
(f1x E, 5 * V) is called
plane R x R
=
R2
(produc ts
of intervals) are a familiar case.
3.18 Example Rather trivially, consider the two-element setsA B f(1,(21e N. The corresponding rectangle is
=
(1,(1
e @,and
=
A XB
=
1), (t,)1,t2), (*2,(1),()2,t2)l. 1(t,)1,('
The sets f(1,(1), ((*,,(2))and ((1,(2), (2,(1)) are not rectangles, unions of rectangles and so are elements of 5 @ T. n
but are
Two important pieces of terminology. If E c f x E, the set an(F) (: (,() e F) is called the projection of E onto f1. And if A D, the inverse projection of A is the set =
o-1(A)
A xE
=
=
(4),4):
)
e A,
(
EJ
e
(3.44)
.
A x E is also called a cylinder set in f x E, with base A. The latter terminology is natural if you think about the case = R2 an d E R Cylinder sets with bases in 5 and T are elements of R,e. One might tlnk that if E e 5 @ N, antA-) should be an F-set, but this is not necessarily the case. :tE'f # ln(M) in general (see is not closed 1.3) so that the collection C of projections of 5 (&N-sets onto = notice under complementation. However, that A ao(A x Z) so that 5 i C. The main task of this section is to establish a pair of results required in the construction of measures on product spaces. =
.
3.19 Theorem If C and b are semi-rings of subsets of and E, respectively, then Sws = (CxD: C e C, D e 01 is a semi-ring of
x E.
Proof There are three conditions from 1.16 to be established. First, Ses clerly contains 0. Second, consider C1, Czq C, andDlyDz G 0. Q rn Czq CandDl rhDZ e D, and as a matter of definition, (C1 xD1) rn (Q xD
=
()
e
Q, (
c Dj
) fa (
e
Q,
(
e Dal
Measure
49
= ((o e cl c2, ( e ol = (c1ch ca)x (D1 cjD1) e Rwe. fazhl
fa
Third. assume that C1 xo2)
xDl i Cz xD2, and by a similar argument,
CI
(c1 xD1)
-
(3.45)
((
=
(
e Cz,
= ((C2
-
e D2): either
cl or (
)
D1l
C1) XD1) tp (C1 X (D2 -D1))
tp ((Q
C1) X (D2 -D1)).
-
(3.46)
where the sets in the union on the right-hand side are disjoint. By hypothesis, the sets Cz C1 and D1 DI are finite disjoint unions of C-sets and O-sets respectively, say (47,...,Cn'), and (D(,...,D,,,'). The product of a finite disjoint union of sets is a disjoint union of products; for example, -
-
n
n
U
/=1
C; x D1
(tl),():(t
=
Uts (
e
,
e D1
=
UICJx D1).
(3.47)
./=1
j=1
Extending the same type of argument, we may also write m
n
(ca cllxtoa-ol) -
-
Utclxol) U(c1xoi) k-p
kp
k=1
.j=1
U(cJxo;) (3.48) .
j,k
All of the product seys in this union are disjoint (i.e., a pair ((t),))can appear in at most one of them) and al1 are in Scs. This completes the proof. . The second theorem leads to the useful result that, to extend a measure on a productspace, it suffices to assign measures to the elements of Scs, where C and D are suitable classes of the factor spaces.
3.20 Theorem If 5 = c(C) and T c(O) where C and D are semi-rings of subsets of f and E respectively, then 5 @ V = C @ 0. =
Proof It ij clear that Rwo i Sps, and hence that C (8)D converse, consider the collection of inverse projections, .t/a (J:o-1(#), F e Fj pwg.
5 (8)W.To show the
c
=
,/p
is a c-field of fl x E, and is in fact the smallIt can easily be verified that is a (;o'(C), C e C J i C (>))0. est c-field containing the colletion and since it follows (& is and hence D c-field by the T l-system, a a theorem that c(#c) ? C @)0. Exactly same conclusion holds for #s, the orresponding collection for V. Evel'y element of Sss is the intersection of an element from #v and one from #'g, and it follows that Spg c C @D. But Sys is a l-system by 3.19 and hence a further application of the 7:- theorem gives 5 (8)N ? u gs; , .1
vt/c
,t/c
=
-system,
xt/p
'
=
irlik pztjlptlof > product extends beyond pairs to triples and general n-tuples, and gd jjjajj be jnterested in the nrooerties of Euclidean a-qnnne ffRlh '
y:
,;r-.
..
.. .
. .
.
.
jjtyj)jjjujf
50
Mathetnatics
For tinite n at least, a separate theory is not needed because results can be obtained by recursion. If (Ylf,X)is a third measurable space, then trivially, fxE
x'P
((,(,v):
=
(
e f,
(,)
= f((,(),v): (,() E) xT. = (f x
e E,
ve
e DXE,
v
t1:l
e .P1
(3.49)
Either or both of (f),T) and (E,V) can be product spaces, and the last two theorems extend to product spaces of any finite dimension.
3.5 Measurable Transformations
F: D
F->
(E,N) in a different context, as domain
and
Consider measurable spaces (f,T) and codomain of a mapping E.
if F -1(#) l 5 for a11# l N. The idea is that a to be sl%-measurable measure p, defined on (,T) can be mapped into (E,V), every event B e V being assigned a measure v(#) g,(F -1(.#)) We havejust encountered one example, the projection mapping, whose inverse defined in (3.44)takes each T-set A into a measurable rectangle. Corresponding to a mesurable transformation there is always a transformed in following the sense. measure, F is said
=
.
3.21 Theorem Let jt be a measure on (f,F) and F: f ation. Then P.F -1 i s a measure on (E V) where ,
jLF -1(#)
E a measurable
F-
transfonu-
,
-1
=
(#)), each
g,(F
B e g,
(gj5o
.
.
Proof we check conditions 3.1(a)-(c). Clearly g,F -1(4 ) 1 0 a1l A e Bv. Since holds by definition, T-1(O) T -1(E) O by 1.2(iii) and so g,F-1(3) p,(T -1(O)) g,(O) 0. For countable additivity we must show ,
=
=
=
=
=
g,F-1
for a disjoint collection that the B; are disjoint becomes
U#y j
=
X/tF-1()
(3.51)
j -1
F Bj4, 1.2 shows both Bj,B1,... e E. Letting B) and that F-1(Uy#y) Equation (3.51) therefore Uy#J. =
=
p,Ur j
=
X g,(#J)
(3.52)
./
for disjoint sets Bgq,which holds because g, is a measure. The main result o
general transformations
.
is the following.
3.22 Theorem Suppose F -1(#) l 5 for each B e D where D is an arbitrary class of sets, and N c4D). Then the transfonnation F is T/v-measurable. ,
=
51
Measure
(
$.,
.
.
.
,
.
,
=
,
j'
T-3LjBj)
T-3Bj)
q @,j q s, then proof By 1.2(ii) and (iii), if l then @ @ It follows d if z'-1(#r) z'-1(#)c z'-1(,) E e ?yan
,
k
=
UjF-1()
that the class of
CtS
a is a c-field.
Since D
t,: w-1(s)s s)
.
N c 4 by definition.
.4,
c
.
This result is easily iterated. If (t1#,J't)is another measurable
is a V/x-measurable transformation, then &oF: (
F-+
space and U: E
F->
VIJ
VP
is F/s-measurable, since for C -1
R, W 1(C)
E
-1
(&0D (0
F
=
N an d hence
E
,
-1
(&
(C))
9.
1
(3.53)
E is called a measurable isomorphism if it is An important special case: F: f' 1-1 oflto, and both F and F -1 are measura ble The measurable spaces (2,1) and (E,N) are said to be isomorphic if such a mapping between them exists. The implication is that measure-theoretic discussions can be conducted equivalently in either (f,T) or (E,V). This might appear related to the homeomorphic property of realfunctions, and ahomeomorphismis indeed measurablyisomorphic. Butthereis no implication the other way. -..:
.
r0,11 F--> (0,11,defined
3.23 Example Consider g: #(A)
x+
,1
0 f x f 1 2
x
1
l <
=
by
(3.54)
.
-
2,
1)
x < 1
Note that g is discontinuous, but is 1-1 onto, of bounded variation, -j Sgo,lj/fgo,lj-measurableby 3.32 below, and g g. I::I
and hence
=
The class of measurable transformations most often encountered is where the codomain is (R,O), B being the linear Borel tield. In this case we speak of a function, and generally use the notation f instead of F. A function may also have the extended real line (RlW)as codomain. The measurability criteria are aj
follows.
3.24 Theorem (i) A function /': f
R for which (): J(() f x) e ?Ffor each x E (Dis So is a function foz which (): fk < x) e 5 for each
F-+
T/f-measurable. x e (D (ii) A function fl f.1 F-> 2- for which f(t): Q tp (+x) t.p (-x) is T/f-measurable. .
.f4))
f xl e ?F for each x e -1(#)
Proof For case (i), the sets f(t): flk f xl are of the form f B e T' where C is defined in 1.21. Since B c(C), the theorem follows by 3.22. The other cllection indicated also genertes :B, and the same argument applies. The extension to case (ii) is equally straightforward. . ,
=
52
Mathentatics
The basic properties of measurable functions follow directly.
3.25 Theorem (i) If f is measurable, so are c + f and c#, where c is any constant. (ii) If f and g are measurable, so is f + g. Proof If .X
e
jq
f
K x, then
f+
c S x + c, so
that f
c is measurable
+
by 3.24. Also, for
,
(: (:
c/)
XJ
S
/*()
((t): /)
=
< xIc ),
c
>
0
< xlclc,
c
<
0
c
=
c
=
f1, 0,
where for each of the cases on the right-hand side and each x/c in 5, proving part (i).
If and only if lt follows that
(:
J())
f+g
+.g((t))
<
there exist r e ) such that
.x,
U (2
.z')
<
=
f(l
<
r)
f-n
(:
(3.55)
0 and x k 0 0 and x < 0
f
<
r
<
R the sets are
6
x g -
:(t0) < x- r1.
(see 1.10). (3.56)
rq Q
The countable union of T-sets on the right-hand side is an T-set, and since this holds for every x, pal't (ii) also follows by 3.24(i), where in this case it is convenient to generate tB from the open half-lines. >
Combining parts (i) and (ii) shows that if h,...,fn are measurable functions so is Xhjcjfj, where the cj are constant coefficients. The measurability of suprema, infima, and limits of sequences of measurable functions is important in many applications, especially the derivation of integrals in Chapter 4. These are the main cases involving the extended line, because of the possibility that sequences in R are diverging. Such limits lying in V are called extended functions. 3.26 Theorem Let f/l be a sequence of F/s-measurable functions. Then infn/n, supn/k, liminfln, and limsuw/ are F/W-measurable. Proof For any x e
t:
F (: fnlk
f xl s 5 for each n by assumption. Hence
,
supnlnt)
<
.z')
=
Of):
n=1
fnk
so that supn/ is measurable by 3.24(ii). Since infn/'n obtain (:
infn/t)
.x)
<
(: sup,,(-Jn())> -xJ = (: supn(-Jn())S -xJf =
@,
S xl e
-supnt-ln),
=
(3.57) we also
Measure
53 C
00
O
n=1
la''. -Tn(a')K
-xl
X
=U1e):/()
<
5.
xl e
(3.58)
n=1
To extend this result from strong to weak inequalities, write X
f xl
infn/kt)
(:
O(:
=
infn/kt)
<
x+
?n=1
Similarly to
(3.59)
5,
(3.60)
may show
(3.57),we
1:
1/rlJ e 5.
supknfkt)
f
Ot:
.x1
=
.fn()
S
.x1
e
kkn
and applying (3.59)to the sequence of functions gn (:
limsupn/t)
sums/k
=
yields
K xl e F.
(3.61)
In much the same way, we can also show f x) e @.
( (t): liminfn/t)
(3.62)
condition of 3.24 is therefore satisfied in each case.
The measurability
.
We could add that limn/kt) exists and is measurable whenever limsupn/t) liminfs/kt). This equality may hold only on a subset of f1, but we say fn converges a.e. when the complement of this set has measure zero. The indicatorfunction 1s()) of a set E e 5 takes the value 1s() 1 when (.t)E characteristichmction otherwise. and 1s((l)) 0 Some authors call ls E, of F. the It may also be written as fs or as zE. We now give some useful facts about indicator functions. =
=
=
3.27 Theorem (i) 1s() is 5IB measurable 1 1s(). (ii) 1sc()) (iii)1Ujsj((l)) sup 1sf(). =
if and only if E
G
F.
-
=
i
(iv) lnjsjt)
inf 1s/)
=
i
=
I-l1sj((l)). i
Proof To show (i) note that, for each B e .S,
1F-1
(#)
'Fhse
sts
ithediyte
=
if 0 G B and 1 G B E if 1 e #, 0 # Ec if 0 e B, 1 , B 0, otherwise
(3.63)
are in 5 if and only if E e @. The other parts of the theorem fwm tlw dfinition. .
re
Mathematics
54
Indicator functions are the building blocks for more elaborate functions, constructed so as to ensure measurability. A simplefunction is a T/o-measurable function J: fl F...: R having finite range', that is, it has the fonn
ftl
af, Xaflsjtt9l =1
=
=
tt e Ei,
(3.64)
where the G1,...,(xa are constants and the collection of T-sets F1,...,Fa is a tinite partition of f1. T/f-measurability holds because, for any # 6 B,
J-1(,)
U Ei s
=
ai
@.
(3.65)
B
EE
Simple functions are ubiquitous devices in measure andprobability theory, because many problems can be solved for such functions rather easily, and then generalized to arbitrary functions by a limiting approximation argument such as the following.
l
Fig. 3.1
3.28 Theorem If f is T/o-measurable and non-negative, there exists a monotone n e EN I such thatnlt) f sequence of T/f-measurable simple functions f/'(n), for evel'y (l) e f1. '
1,...,n2'l, consider the sets Ei = f ): i 1)/2' < il2n ) Proof For i f Augment these with the set Enzn.. f : J() k nl This collection corresponds to a n2n + l-fold partition of (0,x) into S-sets, and since f is a function, each and hence belongs to one and only one Ei. The Ff maps into one and only one therefore constitute a partition of f1. Since J is measurable, Ei s 5 for each i. Define a simple function hn)on the Ei by letting i i 1)/2/, for i = 1,..., nln + 1. Then fn) f f, but A+1())k /4)) for every ); incrementing n bisects n+1)()) is equal to either i 1)/2*, each interval, and if n)()) n)(), 1)/2'+1 &+1 > follows that the sequence is 1)/2 2/ It li or ( J. (). =
-
.
=
.
.f4)),
=
=
-
=
-
-
-
sieasure
55
J(). This holds for each ) e f. To extend from monotone, and lims-yxnlt) non-negative to general functions, one takes the positive and negative parts. Define J+ maxlf,ol and ff'b f, so that both f- and J- are non-negative functions. Then if f'Injand j'-n) are the non-negative simple approximations to it is clear that f+ and f- defined in 3.28, and fnj f =
=
=
-
.f+(n)
-(n),
=
-
l/-J(n)I < lJ+-.f+(n)l+ 1.f---(n)I Fig. 3. 1 illustrates the construction for n is a function on the real line.
=
0. w
--.h
2 and the case
(3.66) =
R, so that
.f()
3.6 Borel Fqnctions lf J is a measurable function, and 5'; S g R, 'T g R is a function of a real variable, is the composite function gof measurable? The answerto this question is yes if and only if g is a Borel hmction.Let Ss (# ros: B e S), where B is the Borel field of R. fs is a c-field of subsets of S, and B DS is open (closed)in the relative topology on S whenever B is open (closed).in R (see 2.1 and 2.3). fs is called the Borel field on S. Define fv similarlywith respect to 7. Then g is called a Borel function (i.e., is Borelmeasurable)if g -1 B) e B s for all sets B l f.r.
gl S
F-+
=
3.29 Example Consider gx) Ixl g -1 ta kes e ach point of R+ into the points x is the set For any B E B + (th e res triction of B to R+) the image under and for each a7 e #, which is an element of f. E1 containing the points x and =
.
.g-1
-x.
-x
1 if is rational, 0 otherwise. Note that (D e B (see 3.30 Example Let g(x) -1 3.15), and g i s defined according to (3.63)with E Q, so g is Borel-measurable. I:a .:
=
=
non-measurable ln fact, to construct a following. but the obvious case is the
4plausible'
function is quite difticult,
3.31 Example Take a set A f; for example, let A be the set H defined in 3.17. l Now construct the indicator function 1x@): R F-> (0,1 ) Since 1x - (f 1 1) A gp Shis function is not measurable. Ia .
=
,
Necessary conditions for Borel measurability are hard to pin down, but the following suftkient conditions are convenient.
3.32 Theorem lf g'. S it is Borel-measurable. Pmof
(i) follows
F-+
'T is either
immediately j cotinuity implies that h B) in T, by 2.17. To prove (ii), consider first prperty h@l S @)when y < ..
(i) continuous
or
(ii) of
botlnded variation,
from 3.22 and the definition of a Borel field, since is open (closed)in S whenever B is open (closed) a non-decreasing function h: R F.-y R, having the x; if A (y: h@l S (x)), sup A = x and A is one =
Mathematics
of (-x,x) and (-=A, so the condition of 3.24 is satisfied. So suppose g is nondecreasing on S; applying the last result to any non-decreasing h with the property hx) = x e S, we have also shown that g is Borel-measurable because = -1 h-'B4 5') f''ASe Ss, for each Brh'T e O.r. Since a function of bounded g CBt''h variation is the difference of two non-decreasing functions by 2.20, the theorem now follows easily by 3.25. . '(x):
This result lets us add a further case to those of 3.25.
3.33 Theorem If J and g are measurable, Proof
jg
=
with 3.25(ii).
z1((J+:)2
-
fl
gl)
-
so is
fg.
and the result follows on combining 3.32(i)
.
The concept of a Borel function extends naturally to Euclidean a-spaces, and indeed, to mappings between spaces of different dimepsion. A vector function 7,' S i R 1- R'/ gl S --./
,
is Borel-measurable if g -1(#) e B s for a11B e Sv, where Bs and fv (# r-75': B e f'''J.
=
(# taS:
# e
l
=
3.34 Theorem
lf g is continuous. it is Borel-measurable.
Proof By 2.21. w Finally, note the application of 3.21 to these cases.
3.35 Theorem If jt is a measure on (R Bk) an d g : S s.y is Borel-measurable where S c Rk an d -Erc R'/ gg is a measure on (T,Sv) where 'r
,
-1
,
gg-1(B)
t-1(s))
=
(3.67)
,
for each B e f.r. n R'N
for m < k. If X is A simp le example is where g is the projection of R onto k x 1 with partition X' = (21,7+'.),where X. is m x 1 and X.. is k m) x 1, 1et gl R k F-> R*' be defined by -
r(A') -1
In this case, g,g B)
=
X..
-1
=
P.W(#))
=
jt(#
xR
(3.68) k-m
)
or p s qrrl .
4 Integration
4. 1 Construction of the lntegral The reader may be familiar with the Riemann integral of a bounded non-negative function f on a bounded interval of the line gJ,:J, usually written fdx. The objects to be studied in this chapter represent a heroic generalization of the same idea. lnstead of intervals of the line, the integral is defined on an arb-
itral'y measure space. Suppose
(f1,@,g)is :
l'1
O
a measure space and
2-6
is a T/s-measurable function into the non-negative, extended real line. The integral of f is defined to be the real valued functional
jfdb
Sup
=
tXj
fte)ljpfj
tinsfo
(4.1)
where the supremum is taken over a11finite pmitions of into sets Ei e T, and the supremum exists. lf no supremum exists, the integral is assigned the value +x. 5 T he integral of the function IAJ,where 1x() is the indicator of the set 4 e 5, is called the integral of f over A, and written fdbk. The expression in (4.1) is sometimes called the lower integral. and denoted sfdp. Likewise defining the upper integral of f,
j'ydb -
inf
E i
(supsy/j bte'i)
(4.2)
,
G
f from below and from we should like these two constnzctions, approximating .fdbk J'/W/whenever above, to agree. And indeed, ttis possible to show that f *fdbk oo if either the set > 0) has is bounded and g(f1) < x. However, (: infinite measure, or f is unbounded on sets of positive measure. Definition (4.1) is preferred because it can yield a finite value in these cases. =
.f()
=
4.1 Example A familiar case is the measure space ('R,f,-), where m is Lebesgue where f is a Borel function is the Lebesgue integral of measure. The integral Ifdm f. This is customarily written j'fdx,reflecting the fact that mx, x + #x1) tfx, even though the sets (A'j) in (4.1) need not be intervals. n =
4.2 Example Consider a measure space (R,f,g,) where g, differs from m. The where f is a Borel function, is the Lebesgue-stieltjes integral. integral Ifdbk'
Mathematics
58 The monotone function Fxt
=
p,((-x,x1)
(4.3)
F(b) F(c), and the measure of the interval has the property (x, x + #< can be written JF@). The notation IfdF means exactly the same as J/Wg.,the choice between the p. and F representations being a matter of taste. See j8.2 and j9.1 for details. n g,((J,:1)
=
-
For a contrast with these cases, consider the Riemann-stieltjes integral. For an 1et a partition into subintervals be defined by a set of points H interval Etz,!71, Akl, < < xn with a = xo < b. Another set H' is called a = (x1,..., refinement of Fl if IR c H'. Given functions f and a: R 1-+ R, let .x1
=
...
5lRvas.fl
=
77f(f)(G(A)
-
a(zf-I)),
(4.4)
i=1 bafd%,
here ti e (xf-l,xf). If there exists a number there is a partition IQ with W
(&l-I.a,.f)
'-jajd
such that for every : > 0
b < 6:
l
for all FI Q IQ and every choice of fff), this is called the Riemann-stieltjes integtal of J with respect to a. Recall in this connection the well-known formula for integration by parts, which states thqt when both integrals exist,
fbtub)
=
/J)a(J)
+
jafd ja@f' +
When (x = x and #is bounded this definition yields the ordinary Riemann integral, and when it exists, this always agrees with the Lebesgue integral of f over gtz,:). Moreover, if a is an increasing function of the form in (4.3),this integral is equal to the Lebesgue-stieltjes integral whenever it is defined. There do exist bounded, measurable functions which are not Riemann-integrable (consider3.30 for example) so that even for bounded intervals the Lebesgue integral is the more inclusive concept. However, the Riemann-stieltjes integral is defined for more general classes of integrator function. In particular, if f is continuous it exists for a of bounded not necessarily monotone. These integrals therefore fall variation on gtz,&), the class defined by (4.1), although note that when a is of bounded variaoutside tion, having a representation as the difference of two increasing functions, the Reimann-stieltjes integral is the difference between a pair of Lebesgue-stieltjes
integrals on (J,:1. The best way to understand the general integral is not to study a particular measure space, such as the line, but to restrict attention initially to particular classes of function. The simplest possible case is the indicator of a set. Then, every partition fEiI yields the same value for the sum of terms in (4.1),which is
jgdn
Integration
59
jxdp
(4.6)
BIX),
=
=
5, the integral is undefined.
for any A e 5. Note that if A
Another case of much importance is the following.
4.3 Theorem lf J
0 a.e.lgl, then
=
fdp
Proof The theorem says there exists C e C. For any partition (Fl,...,&J 1et A'; ity of g,,
77 i
inf (l) e
Ei
/.() g,(e)
=
Xi
inf (l) e
F;
=
0.
1, such that /4)) 0 for ) Ei rn C, and A': Ei F;. By additiv-
with
gtc)
=
=
=
=
/.()l g,(e':) +
Xi
inf f e e:
-
g,(rJ)
= 0,
(4.7)
the first sum of terms disappearing because /*4) 0, and the second disappearing =
by
.6(i)
since pE'i) f g,(C')
0 for each i.
=
.
A class of functions for which evaluation of the integral is simple, as their name suggests,is the non-negative simple functions.
4.4 Theorem Let (?() 5 is a partition of f.
=
where aj k 0 for X:=1%1sj()),
i
=
1,...,n, gnd EL,...,En E
Then
tFp, 77
(4. 8 )
ipEi).
=
=1
Proof Consider an arbitrary finite partition of f1, A1,...,4-, and define inftxxyst). Then, using additivity of g, Ip,(Ay) X SX 77 f=1 .j=1 j=3 =
<
i=1
xi
I
=
p,(AyraEi4
jtt/tjc
Ei4
j=L
(4.9)
ipEi),
= f=1
where the inequality uses the fact that I assumes the smallest value of Gj such that Aj ra Ei # 0, by definition. The theorem follows, given (4.1), since (4.9) holds as an equality for the case m n and Ai = Ei, i 1,...,n. w =
=
So for functions with finite range, the integral is the sum of the possible values of f, weighted by the measures of the sets on which those values hold. Look at Fig. 3.1. The Lebesgue integral of the approximating function fz) in the figure is the sum of the areas of the rectangular regions. To compute the Lebesgue-stielties inteeral with resoect to some measure u.. one renlnces the width of the qetq
62
Mathematics
= yaj
a simple function. Hence,
ja-bdb
+ tzj,
s Af ra Bj,
(t)
(4. 17)
X Xtla, +bbbviAil-bB?
-
j
i
=
zxGf7r/ttArnl
=
'7rafgtA
=
i
,xfs>7/ttAf
+
j
j
rn B?
i
bTfbbkB)
+
j
i
ajdp
+
bjydp,
(4.18)
showing that linearity applies to simple functions. Now applying 4.6,
jaf
+
bgldp
ja jygjtj jt?lj1jsyusp, (suspy +
sup
=
bz4dbk
(pf/f'
=a =
+
b
#
ajfdp
+
bjgdp.
(4. 19)
To extend the result to general functions, note that
la.f
1 If I + Ib I Ig l
bg l < a I
+
.
.
(4.20)
,
so (4.19) shows that af + bg is integrable so long f and g are integrable, and a and b are finite. The identity af
+
bg
=
afl'
afl-
-
bg4'b (bg)-
+
(4.21)
-
implies, applying (4.19), that
jaf
bntdb
+
If a k 0, then
ab+dk -
=
jaldb
afl+dbk
aft-dbk
=
J(&T)-*+ Jt/WI+dB
-
-
afl-d
Ia I f-db
=
-
Jtf/'+lt
f+db
-
=
ment applies to the terms in b and g. So
JJ-Jg,)
IJI
-fd
'-jb8t-db.
=
(4.22) ajfdbk, whereas if a < 0, alfdb. The
=
(4.16) holds as required.
same argu.
is a very useful property. The first application is to show the the integral to the behaviour of functions on sets of measure 0, of invariance extending the basic result of 4.3. Linearity
4.8 Lemma Let f and g be integrable functions. (i) If f f g a.e.lg1, then fdkt; hdp. dp. (ii) If f = g a.e.gg,l, then J/Wg,= Proof For (i), consider first the case J 0. lf g 2 0 ekerywhere, directly from (4.1). So suppose # k 0 a,e.rg,l and define =
dbk 2 0
63
Integration
Jl4)) Then h
0 a.e.gp,) but g
=
jg
0 S
fhdbk0andby
since
J('
To prove
JJJF
2 0
=,
#()
<
.
0 4.7,
jgdp jhdpjgdp, +
=
(4.23)
=
0,
(ii), let
h
J(' + hjdbk
=
:()
h k 0 everywhere, and, applying
hjdp
+
0,
4.3. Now replace g by g J in the last argument to show hence dbkk J/Wptby 4.7.
=
-pdp
+
=
-
=
dp
=
f
-
+
0 by 4.3. Then g so that h 0 a.e.gg,l, and hdp hdbk J#JF, Where the second equality is by 4.7. w =
=
=
These results permit the extension to the more commonly quoted version of the monotone convergence theorem.
4.9 Corollary lf fn k 0 and fn
'1'
limn-oxl/nv
J a.e.g.l,
=
jdp. n
T
Another implication of linearity is the following.
j
Proof
j lf 1 jf'
inequality
4.10 Modulus
lJ lJz
Jjt 2
+#-)tfp,
=
j jjj'dp jf jf'bdp .
-#p,
=
+
Jy-#>l jJ/-d: )Jy::1. .
-
-
In the form of 4.9, the monotone convergence
theorem has several other useful
corollaries. 4.11 Fatou's
lemma If
Jn k
0 a.e.Lg1,then
liminf fndp k nMx
j
liminf
Jn Jg,.
n->oo
Proof Let gn infkn/'k, so that (,n) is a non-decreasing sequence, and gn 1% g sides oo 2 Igndp. Letting n both of liminfn/n. Since Jn k gn, Jfntkyt the on inequality gives =
=
--+
liminf fndp 2 lim gndp n.-+txz
=
gdp
liminf
=
n-M
4.12 Dominated convergence theorem If fn
--
that IfnI
fn #p..
(4.24)
w
n--x
S g a.erlp1 for al1 n and
gdbk <
x,
f a.e.lgj, an there exists g such then JlnIg, .-A JJJP,.
Proof According to 4.8(i), dp < x implies JlfnI#g, < =. Let hn Ifn f I such that 0 f hn K 2g a.e.lg,j and hn 0 a.e.ggj. Applying 4.3 to liminfns, linear=
-->
jw.
.t.......:.')..:..
t.............( .) ..... .:
.......
Fatu' s lymma, attl .).( ,
.'). '
..
.(..
.
...
E .y. ... . .
..
.
.. .
j.... ..
.
-
,
64
Mathematics
ljgdp
J
=
=
liminftzg
-
j (lg jhndp
hn) Jjt f liminf
-
hnjdp
N->x
n-/co
jgdp
limsup
-
(4.25)
,
N-+=
where the last equality uses (2.4). Clearly, limsupn-yxf/lnzv k 0 the modulus inequality implies 1im n-e
=
0, and since
jfndp j/'ljt jhndp < 1im
-
=
0.
j'hndb (4.26)
.
n'M
Taking the case where the g is replaced by a tinite constant produces the following version, often more convenient:
4.13 Bounded convergence theorem If fn then limn--jxlfntfg, Ifdt < x. E1
--y
f a.e.gg1 and IA! f
B<
x
for all n,
-->
Theorem 4.7 extends by recursion from pairs to arbitrary finite sums of funcPut fn tions, and in pmicular we may assert that J(Z:=1'i)Jg, = Xl=tgiand fndbk l=tlgid, where the gi are pon-negative functions. Then, if fn -1xf Z7=l'j < x a.e., 4.9 also permits us to assert the following.
Z:=1J##g..
=
=
=
4.14 Corollary
If
f'fl is
a sequence of non-negative functions,
J Ng'
,
-
# jg,
.
(4.27)
o
i= 1
fcz1
By implication, the two sides of this equation are either both infinite, or finite and equal. This has a particular application to results involving c-finite measures. Suppose we wish to evaluate an integral gdkkusing a method that works for finite measures. To extend to the c-finite case, choose a countable partition igi, lffl of f1, such that g(fk) < x for each Letting gi loy, note that g iidp by gdp (4.27). and =
.
=
=
4.3 Product Measure and Multiple lntegrals Let (f,@,g) and (E,V,v) be measure spaces. ln general, tfx E, 5 @ N, 1) might also be a measure space, with 7t a measure o'n the sets of 5 @ V. In this case respectivlly, are measures g andv, defined by g,(#) 7:(Fx E) andvtG) a(f1 x G4 called the marginal measures conrsponding to T:. Alternatively, suppose that g and v are given, and define the set function 1: Sys F-> 2-+, =
=
where Sss denotes the measurable rectangles 7:(Fx G)
=
g,(&v(G).
of the space f x E, by
(4.28)
We will show that 1: is a measure on Syg, called the product measure. >nd has an extension to 5 * T, so that tfx E, 5 (8) V, a) is indeed a measure space. The tirst
Integration
65
step in this demonstration is to define the mapping F(o: E F-> x E by Fo(() (,(), so that, for G e V, F(o(G) f )) x G. For E e 5 @ V, let =
=
Eo
F;1(F)
=
((: (,()
=
e F)
c
E.
(4.29)
The set Eoj can be thought of as the cross-section through E at the element ), any countable collection of 5 @ V-sets (Ej, j e EN
U
((:(t,),t)
=
j
For future reference,
U)
e
Ul(:(,()
=
e
./
j
EjI
=
U().
the following.
note
e 5 (& V. If F=Fx
Proof We must show thate'to e N wheneverf G e N, it is obvious that Eo, =
c(Sps),
(4.30)
j
4.15 Lemma Toj is a W5 @ Vl-measurable mapping for each
Since 5 (&N
For
(t.
G,
E
F
e
F
(l)
e f1.
G for Fq 5 and
e N.
=
3,
(l)
(4.31)
the lemma follows by 3.22. w
The second step is to show the following.
4.16 Theorem
';:
on Sps.
is a measure
0, recalling that Fx O O x G O Proof Clearly JE is non-negative, and <0) for any F e 5 or G e V, and applying (4.28).It remains to show countable additivity. Let fE. e Sps, j e (N) be a disjoint collection, such that there exist sets Fj e 5 and Gj E V with Ej F) x Gj; and also suppose E VjEj e Spg, such that there exist sets F and G with E Fx G. Any point (,() e Fx G belongs to one and only one of the sets Fj x Gj, so that for any (t) e F, the sets of the subcollection fGjI for which ) e Fj. must constitute a partition of G. Hence, applying (4.30)and (4.31), =
=
=
=
=
=
vE.)
=
v
U j
U 3, j
U()(o j
Gj,
=v
v
=
)
e Fj =
(t)
e Fj
Xj 1,())v(%),
(4.32)
where the additivity of v can be applied since the sets Gj appearing in this dcomposition are disjoint. Since we can also write vtfo) v(G)1A)), we find =
l(F) .f....j....L..r
. ... . .
....
.
:
.
..
.
). .
....
.
=
g,tFlvtGl
=
JY(FV>())
=
J
X 1z)(t0)V(f) 1
tf/t*l
66
Mathentatics
= as required,
Xp.(F?)v(t7;)
Xz:(e;.) j
=
j
(4.33)
the penultimate equality is by 4.14. w
where
It is now straightforward
4.17 Theorem ( x E, 5
to extend the measure from R,g to 5
e N, J:)
(& V.
is a measure space. hence Svs, is a semi-ring by
5 and V are c-fields and hence semi-rings; 3.19. The theorem follows from 4.16 and 3.8. . Proof
Iterating the preceding arguments (i.e. letting (,T) and/or (E,V) be product spaces) allows the concept to be extended to products of higher order. In later chapters, product probability measures will embody the intuitive notion of statistical independence, although this is by no means the only application we shall meet. The following case has a familiar geometrical intemretation. Lebesgue measure in the plane, R2 R x R is detined for intervals
4.18Example by
=
-((Jl,:11
x
(J2,&21)
=
b1
-
J1)(:2
,
-
(4.34)
J2).
Here the measurable rectangles include the actual geometrical rectangles (products of intervals), and o the Borel sets of the plane, is generated from these as a 2 is a measure space in consequence of 3.20. By the foregoing reasoning, (R2,B which the measure of a set is given by its area. n 0A,
,m4
We now construct integrals of functions /*((t),() on the product space. The following lemma is a natural extension of 4.15, for it considers what we might think of yielding a function with as a cross-section through the mapping at a point (J) e domain E. ,
4.19 Lemma Let f:D x E F-> R be 5 @)Ws-measurable. Define Then f(0:E F- R is V/o-measurable. fixed ) e
.f(o(j)
=
A,tlfor
.
Proof We can write
f(()
=
f(eD,()
=
f(Fo(())
=
/oF(().
By 4.15 and the remarks following 3.22, the composite function measurable.
(4.35) is MIB-
fo
K
Suppose we are able to integrate f with respect io v over E. There are two quesJE/'IWV tions of interest that arise here. First, is the resulting function gl F/plmeasurable? And second, if g is now integrated over f1, what is the relationship between this integral and the integral Inxzfd:Over ( x E? The affirmative answer to the first of these questions, and the fact that the integral where these exist, are fiterated' integral is identical with the the most important results for product spaces, known jointly as the Fubini theorem. Since iterated integration is an operation we tend to take for g'ranted xxrltla mllltlnlo piemnnn nerhnns the main Doint needin: to be stressed infnyrnlq =
tdouble'
Integration
67
here is that this convenient property of product measures
(and multivariate
Lebesgue measure in particular) does not generalize to arbitrary measures on product spaces. The first step is to let f be the indicator of a set E e 5 (& V. ln this case f is the indicator of the set Efo defined in (4.29),and
J/'olv
v(F(o)
=
+(e)),
=
(4.36)
2-+is well-defined, say. ln view of 4.15, Efo e V and the function #s: fl values unless in the extended although, v is a finite measure, it may take its shown. line, half as .-.:
(& V,
4.20 Lemma Let jt and v be c-finite. For all E e 5 and Jo#FdB TCZ')'
g; is T/p-measurable
(4.37)
=
By implication, the two sides of the equality in or finite and equal.
(4.37)are
either both infinite,
Proof Assume first that the measures are finite. The theorem is proved for this theorem. Let denote the collection of sets E such that gz case using the satisfies (4.37).Spv i ,W,since if E Fx G then, by (4.31), .4
';:-
=
+() and gzdbk fl x E e 1sz- lsj,
=
(4.38)
Clearly g(#)v(G) l(F) as required. We now show is a = and since holds. lsz-sj 1.25(a) lf F1 c Ez, then, Ej,E2 e so -system.
=
.4
Js1/,()JV(()
gE,-EL4
an 5lT measurable
Js1e'1(t0,()#V(()
=
=
-
'sztt,)l
(4.39)
-%(),
function by 3.25, and so, by additivity of
'n,
.
jgjgh-zbdbk
.'
Y,
F e
.4
.4,
' r' 1*
v(G)1F(),
=
l(F2)
=
-
r(F1)
=
zE2
-
F1),
.VL.
t
(4.k0)
''
showing that .Wsatisfies 1.25(b). Finally, If 41 and A2 are disjoint so are (A1)(o + gy). and (A2)(o,and ujwxct)) u1((l)) To establish 1.25(c), let (. e W, with be IN Ej j e J a monotone sequence, F. Define the disjoint collection (Aj) = With A1 q= E$ and Aj Epj A),j > 1, so that E U7=3Ajand Aj e Wby (4.39). additlvity of v, y ountable
(
=
't
=
-
x ','(l
=
Xuytl.
yj -.t. .E )r-. )..' L.k,.j-L);(b. ..r.),).).t - ; )--'.ti't(.. (yyyE (;()t. .(.yy')-....qy . r. tf''Ett..t.q'? ;t-yjjy:jj(j .yjjjy.;gjjyjjjjjt.. .y yy. q jtyjjj yyyj,;( jqqj)q;((. .!k jyy .ryLLI. gjyj)j. jjrjjk yy. ... jj;gjr j.yy.j.jjjyrr jjygj. yjjjyj $kj)4 yryt;y.)...-i.r..t.E)). .E 7..y jj ... y..q-().r-(.t(. ...-.-!;y..:. ,.-. .. gyjj,. . r) r.q.r t. #t-.-y 1l(!(l:l.)(.. ..((r-. )-)t.. . t.-... .. .. ' ;.1 . . . . . . ... . . .. . . . ((
')'
..t.-)
-'.t.t
-..(q..)--r;:,)j);.tr-ry
.
..
..q
..
y..jt..j. . ;fj,jL)qq;... )) jtjj(j)q..... ..y,r(.jk
-.-
.li.
:
--''
-
- - .
-..)-.-.))t:..,.
-..
.
...
.
tj!jj jj.(gjrj
(4.41)
68
Mathematics x
x
Jo
X'At)
tfBtt0l
g
Xj j=1
=
j=j
v
x
o'Atttltfp,tl
X5(X/)
=
=
1(13,
(4.42)
j=j
n
.d
Since Sys is a where the first equality is by 4.14. This shows that is semi-ring it is also a a-system, and 5 @ N c(Sys) c by 1.27. This completes the proof for finite measures. To extend to the c-finite case, 1et J and (Ey) be countable partitions of f tff and E with finite g-measure and v-measure respectively', then the collection tff x Ej e Sys ) forms a countable pmition of x E having finite measures, attf x Ej) = g.tfvtWl.For a set E l 5 @V, write Eq = E r7 tf x Ej). Then by the last argument, -system.
.4
=
ghdbk
=
where gzql fk disjoint and
F-> 's((l))
nEq),
(4.43)
R+ is defined by +,y((l)) v((F(j)(J, (t) e fk. The sets Eq are vjLjjl whep (l) e ff, or =
=
gE
=
X 1f4()X+/).
(4.44)
j
i
The sum on the right need not converge, and in that case g/ F/s-measurability holds by 3.25/3.26, and
jcysdb -
-
Jo
Xjl XyS's
XXJO:S/B i
j
*'
-
'*'l
+x.
=
However,
Jp, XXn(A) i
-
(4.45)
nk,
j
using 4.14 and countable additivity. This completes the proof. Now extend from indicator functions to non-hegative
.
functions:
4.21 Tonelli's theorem Let 7: be a product measure with c-finite marginal measures g and v, and 1et fl x E F-> R+ be (@(&Vl/f-measurable. Detine functions .f:
f1
= R+ f(,(), is T/f-measurable, (i) g JsfA
E
F->
(ii)
b-V(()
jtstcafdx Jo =
and let gk
=
hfuA.
Then
tfB. E1
ln part (ii) it is again understood that the two sides of the equation are either finite and equal, or both infinite. Like the other results of this section, the and (E,N,v), and the complementary results given theorem is symmetric in (f1,@,g) roles interchanging of the marginal spaces do not require a sepapte stateby the ment. The theorem holds even for measures that are not c-tinite, but thij further complicates the proof. Proof This is on the lines of 4.6. For a partition (F1,...,Fn J of 5 @)T let f Xwlsj, and then f Xii ltsjl(sand g = Xiqis'lEil by 4.4. g is F/p-measurable
=
=
Integration
69
by 3.25, and 4.20 gives vedb gdb Xm fl i
Xaine'i)
=
=
fl
=
j
AE
j'dn,
(4.46)
so that the theorem holds for simple functions. For general non-negative f, choose a monotone sequence of simple functions converging to f as in 3.28, show measurability of g in the limit using 3.26, and apply the monotone convergence theorem. w Extending to general
f
requires the additional
assumption of integrability.
4.22 Fubini's theorem Let a be a product measure with c-finite measuresg, and v; 1et #: fl x E >-> R be (; @ Vl/f-measurable with
marginal
Jzxsl.ftt,)'tllllt,ll <
(4.47)
=;
Then and let g define f)1 E >-> R by Jsffzyfv. (i) f is N/f-measurable and integrable for (z) e A f2, with g,tfl A) (ii) g is F/f-measurable, and integrable on A; .f4,();
.406)
=
=
=
-
JsTto'cldvt'l joxsft,clflttt,'l Jo
(iii)
0',
dpt*'
=
Proof Apart from the integrability, 4.19 shows (i) and Tonelli's Theorem shows (ii) and (iii) for the functions f' = maxl,o) and f- = f'b f, where I f+ + f-. But under (4.47),I I < on a set of a-measure 1. With A defined (i), (ii) and (iii) hold for f'b and as the projection of this set onto finite of the equation in sides with both (iii). Since f f'b f (i) extends to f by 3.25, and (ii) and (iii) extend to f by 4.7. . .41
=
-
.f(),()
'x
-,
,
-,
=
-
4.4 The Radon-Nikodym Theorem Consider c-finite measures g. and v on a measurable space (f2,1).g, is said to be absolutely continuous with respect to v if v(A) 0, for E e 5, implies g(A-) 0. This relationship is written as g <( v. If g tt v and v <( go the measures are said to be equivalent. If there exists a partition (A,A ) of f1, such that g(A) 0 and v(AC) 0, then g and v are said to be mutually singular, written g, v. Mutual singularity is symmetric, such that p. v means the same as v g,. The following result defines the Lebesgue decomposition of g, with respect to v. =
=
c
=
.1.
=
.1.
.
4.23 Theorem If p, and v are c-tinite measures, there exist measures g,1and p.2 such that g = gl + ga, g,l v, and p.2
lf there is a function #: fl 1..-> F+ it follows fairly directly (chooseE such that v(F) 0) such that g(F) Js/tfv, that g, << v, and f might be thought of as the derivative of one measure with =
=
70
Mathematics
respect to the other; we could even write f dbjldv. The result that absolute continuity of g, with respect to v implies the existence of such a function is the =
theorem.
Radon-Nikodym
4.24 Radon-Nikodym theorem Let v and g,2be c-finite measures and 1et p,2t< v. There exists a T/p-measupble function fl f F-y 2-+such that g2(F) fdv for all E e 5. u =
f is called the Radon-Nikodym derivative of g, with respect to v. If g is another for all E e 5, then v(.f #: g) 0, otherwise at such function and g,2(A-) Jsgtfv and E1 (: least one of the sets Ej (): ()< gl ) must contrathe detinition. dict =
=
.j'((l))1
.f4)>
=
=
Proof of these results requires the concept of a signed measure.
4.25 Definition A signed measure on (f,F) is a set function z: 5 F-> R-
satisfying 0. (a) :(0) j) (b) z(U#. XgAj) (c) Either z < x or =
=
disjoint collection (Ay e 5
for any countable, u z>
For example, let g, and v be non-negative measures on a space least one of them finite. For a non-negative constant r, define Z(X)
for any
)
.
-x.
=
((, @), with
p,(A) rvtAl
at
(4.48)
-
A e 5. For disjoint (Ay)
,
zj+l (-s) (x/) -
j=3
p,
-
y=1
= Xtg,tll
rv
/=1
-
rvtAjll,
(4.49)
i=1
so that countable additivity holds. If A is a F-set with the property that # e 5 with B c A, A is z(#)k 0 for every complementary negative defined in the called apositive set, a set being manner. A null negative both is positive and is called careful Be that set. to distinset a guish between positive (negative,null) sets, and sets of positive measure (negative measure, measure zero). A set has measure zero if g,(A) rvtz4l in (4.48), but it is not a null set. By the definition, any subset of a positive set is positive. The following theorem defines the Hahn decomposition. :4
=
4.26 Theorem Letx be a signed measure on a measurable space (f,1), having the property x(A) < cxo for a1l 4 e 5. There exists a partition of f into a positive set
A+ and a negative
set A-
Proof Let sup x(A), where the supremum is taken over the positive sets of x. A+ Choose a sequence of positive sets (Anl such that lima-ox z(Aa) o and 1et z4+ A+. measurable consider E Undn.To show that is also a positive set, any Iaetting B, A1 and B- A- -A--1. n > 1. the seuuence $Bnl is disioint, positive ,
=
=
=
=
=
Integration
71
since Sa c A,i for each n, and UnSn = A+. Likewise, if En
Ecj Bn the sequence fEn ) is disjoint, positive since En c #n, and Unk E. Hence zlE') XnzEn) 0, and since E was arbitrary, A+ is shown to be positive. A+ -An being therefore =
=
=
positive, Z(X+)
Z(X2
=
Z(Z+ -A
+
k Z(Xu), a1l n,
(4.50)
z(A+) k 1, implying z(A+) 1. Now let A- f -A+. We show, by contradiction, that A- has no subset E with positive measure. Suppose there exists E c A- with zE4 > 0. By construction E and A+ are disjoint. Every subset of A+kJ E is the disjoint union of a subset of A+ with a subset of F, so if E is a positive set, so is A+ t? E. By definition of
and hence
=
=
X,
,
2
b-)E4 z(z4+
:+z(A),
=
(4.51)
which requires x(A>) 0, so E cannot be a positive set. If F is a subset of F, it is also a subset of A-, and if positive it must have zero measure, by the argument just applied to E. The desired contradiction is obtained by showing that if zE) > 0, E must have a subset F which is both positive and has positive measure. The technique is to successively remove subsets of negative measure from E until what is left has to be a positive set, and then to show that this remainder has positive measure. Let ?z1 be the smallest integer such that there is a subset F1 c and detine F1 E F1. Then let nl be the smallest integer E with z(F1) < In general, for k = 2,3,..., such that there exists E1 i F1 with z(F2) < 1et nk be the smallest positive integer such that Fk-j has a subset Ek satisfying Z(f) < -$Ink, and let =
-1/n1,
=
-
-1/n2.
Fk
E
=
-
U). j=
(4.52)
lf no such set exists for finite nk, let nk +x and Ek = 0. The sequence (Fk) is x. non-increasing. and so must converge to a limit F as k =
---
We may therefore write E Fb..l (U7=1A), where the sets on the right-hand are mutually disjoint, and hence, by countable additivity, =
oo
z(f)
:4#')
=
+
side
Cr
Xz(A)
<
k=1
llnk. zb7 X k=1
(4.53)
-
Since x(F) > 0 it must be the case that x(#)> 0, but since zF < oo by assumptoo as k x. < x, and hence nk This ion, it is also the case that Z7=1(1/ak) with and negative is therefore a positive measure, means that F contains no subset . having positive measure. set -->
--
For any set B e 5, define Z(#) z(A+ rn B4 and z-B) -z(A- fa #), such that x+(S) z-B). It is easy to verify that zB4 z+and x- are mutually singular, non-negative measures on (f1,F). f z- is called the Jordan decomposition of x a s ig ned measure. z+and z- are called the upper variation and lower variation of =
=
=
-
=
a,
wxvxa
+1-.zk
maooxa...x
l .,
I
l,+
-
..u
%,-
lo
-
zxollxcxzl
tlao
tntnl
a/mrizyimn
nf
q;
Tho
Tnrdnn
Mathematics
72
decomposition shows that all signed measures can be represented in the form of (4.48). Signed measures therefore introduce no new technical difticulties. We can integrate with respect to z by taking the difference of the integrals with respect V to x an (j y We are now able to prove the Radon-Nikodym theorem. It is actually most convenient to derive the Lebesgue decomposition (4.23)in such a way that the Radon-Nikodym theorem emerges as a fairly trivial corollary. It is also easiest to begin with tinite measures, and then extend the results to the c-tinite case. -
.
4.27 Theorem Finite, non-negative measures v and g have a Lebesgue decomposition g /1,1+ g2 where gl v and p,2 << v, and there exists an F-measurable R+ F-+ f such that g,2(F) function /': fdv for a11 E e 5. .1.
=
=
Proof 6 L e t G denote the class of al1 F/f-measurable functions gl f F-> R+ for which IEgS S p,('), a11 E e @. 6 is not empty since 0 is a member. Let (x sup#esf'tfv, so that (x f g,tfM< x. We show there is an element of G at which the supremum is attained. Either this element exists, and there is nothing further to show, or it is possible by detinition of a to choose an element gn of G satisfying f a, for each n G IN.Generate a monotone sequence (/k,n e IN) in G tx 1/a f Igndv follows. Put so that fn 2 max(A-1((z)), J1 :1 and define fn by fn as > %()) for n 2,3,..., and then if fn-l A-l. Define the sets Aa (: e G, =
-
,w()),
=
=
.fn-1((l))
=
Xtfv
=
fn-tdv +
=
F
Erun
Fcxq
gndv
g,(FrnAn) + p.tf f'7Ak)
=
g,(f)
(4.54)
so that fn e G. Since fn 1 f, it follows by the monotone convergence theorem that IEf* S h1,(0, and hence f e G. And since fn gn so that fndv --4
a we must conclude that
-
Lln S
J.ftfv a, =
g,2tD
=
gndv S
fndv S a, Now detine g,2 by
as was required. dv, E
C
F'
(4.55)
Evidently g,2 is a non-negative measure (for countable additivity consider the and use 4.14), and also g,2 q v. Define g,1(e')= g,(f) 41,24f), functions which is non-negative by construction of f, and also a measure. It remains to show .
that p.j
=
IV
-
,
.1.
v.
Let (A+n,A-a)be a Hahn decomposition for the measure 1,2,3,.... Then for E e T, g,(Fr7A+) n
=
p.1(Ff-74)
+
-1v(l
>n
ra x+n) +
g.l v/n, for n -
=
g2(F raA+sl
j
sck.:
y:v
=
j
srn,j
-)
.f
+
w,
(4.56)
Integration
73
and hence p.(A')
p,(Ff'3A1) +p.(FrnAa-) 2 p.(Ff7A1)
=
k
J
El'x<
f
1 tv
+
n
Note from this inequality that
(x k
Jf
+
v
+
fdv J Fzn-a
J + n-11Aj -s
1v(A+n) =
=
+
p.2(Fr-7Aa-)
fdv + JE
-)v(F
r-'hA).
(4.57)
e 6 so that ,
-1v(4+n),
a
+
(4.58)
u
implying vAhjlln 0. This holds for each n G N, so if A U';=1A, v(A) 0. Note g.1(AC) v(AC)/l An-for every n, and so for every n. Hence ; that Ac 0* N= 1Aa-i g,1(Af) 0, and-so jt1 v. w =
=
=
=
.1.
=
It remains
to extend this result to the c-finite case.
such Proof of 4.23 By c-tiniteness there exists a countable partition f l of collection with finite for finite each j. If (Ay) is any that vtf#) and p,tf#l are union and whose is letting f11 Lh -Ay-1 for A1 1 > Aj defines j measures such a partition. If different collections with tinite measures are known for v and p,, say lAjzjl and fAvyl the collection containing all the Ax rnAvk for j,k e N, is countable and of finite measure with respect to both v and g,, and after reindexing this collection can generate (f). Consider the restrictions of g, and v to the measurable spaces (Lh,%4, for j e IN,where % (F t''h Lh, E e 5 J By countable additivity, g,(F) XjttA' rn L%lwith similar equalities for g,l, g,z, and v; by implication, the two sides are in each case either finite and equal, or both +x. If v(F rn Lhl 0 implies g,2(F chLhl 0 for each j, then v(F) 0 implies g2(F) 0 for E e 9, and g,2< v. Similarly, let Aj, AJ) define partitions of the Lh such that g,1(Aj) v(A,) 0', then A U#. j, v(AC) = Qv(A,) = Ac unions, 0. p,1(A) Qg4(AJ 0, and and Ugdlare disjoint Hence g,1 v. K ,
=
,
=
,
=
=
.
=
=
=
=
=
=
=
=
=
=
.
The proof of the Radon-Nikodym theorem is now achieved by extending the other conclusion of 4.27 to the c-finite case. Proof of 4.24 In the countable partition of the last proof, 4.27 implies the such that non-negative X/f-measurable
existence of
g,(F rn Lh)
=
p,ltf
f''h
L1jl + g,2(F r-hLhl
where P,2(A't''h f#) Define
J:
F-
Jsyoo *,
=
R+ by .fttl)l
=
al1 E e
X lf/ttl'tt9l.
5.
(4.59) (4.60) -1
This is a function since the Lh are disjoint, and is g/s-measurable since f B) y-1(#) LtEj e 5 where Ej e 5j, for each # e S. Apply 4.14 to give = Uy/' =
Mathematics
74 :2(A3
=
Xp,ztFr7
Lhl
=
j
Js
1tzjfldv. - X
Xj
Js
Jsljdv
fdv
-
.
(4.6 1)
*
Consider the case where g, is absolutely continuous with respect to another gq, v(A) measure v. If the Lebesgue decomposition with respect to v is g, g,l + g,1(AC) since 0 implies 0 which in implies g,l(A) 0. But g,(A) 0 turn g,l v, too, Thus, g,1(f2) 0 and g, ga. The absolute continuity of a measure implies the existence of a Radon-Nikodym derivative f as an equivalent representation of the fOr measure, given v, in the sense that g,(F) Js/'tfv any E e @. An important application of these results is to measures on the line. =
=
=
=
=
.
=
=
=
4.28 Example Let v in the last result be Lebesgue measure, m, and 1et g, be any other measure on the line. Clearly, g,1 m requires that g,1(A'') 0 except when E is of Lebesgue measure 0. On the other hand, absolute continuity of g,2 with length', any countable collection of respect to m implies that any set of isolated points for example, must have zero measure under g2. If g, is absolutely continuous with respect to m, we may write the integral of a measurable function g as =
.
Ezero
j
+x
'(AVP'@)
=
-X
p+x
1 -
gxtfxtdx'
-X
(4.62)
so that all integrals reduce to Lebesgue integrals. Here, f is known as the density functionof the measure g, and is an equivalent representation of g,, with the relation g,(f)
=
jefxtdx
(the Lebesgue integral of f over E4 holding for each E e f. n
(4.63)
5 Metric Spaces
5. 1 Distances and Metrics Central to the properties of R studied in Chapter 2 was the concept of distance. For any real numbers x and y, the Euclidean distance between them is the number Jstx,y) Ix y I e R+. Generalizing this idea, a set (otherwisearbitrary) having a distance measure, or metric, defined for each pair of elements is called a metric space. Let S denote such a set. =
-
A metric is a mapping J: S xs F-> R+ having the properties dx,y), 0 iff if y, 2 dx, dy, (triangleinequality). metl'ic (S,#) is S A space a set paired with metric #, such that conditions (a)-(c) hold for each pair of elements of S. u
5.1 Dqfinition (a) dy.x) (b) dx,yt (c) dx,yt +
=
.'t7
=
=
0, but dx,y) 0 is possible when x # y, we If 5.1(a) and (c) hold, and dxtx) would call d a pseudo-metric. A fundamental fact is that if (A,#) is a metric space and # c A, (#,#) is also a metric space. If (E1is the set of rational numbers, Q c R and (Q,#s) is a metric space', another example is (g0,11,#s). While the Euclidean metric on R is the familiar case, and the proof that dz satisfies 5.1(a)-(c) is elementary, dz is not the only possible metric on R =
=
.
5.2 Example For x,y
E
R 1et
4@,y)
=
!
Ix +
yI
-
jx
..y
j
(5.1)
.
It is immediate that 5.1(a) :nd (b) hold. To show (c), note that Ix l doxtyjlk dx,yj). The inequality tz/(1 a) + /7/(1 b4 2 c/(1 c) simplifies c). We obtain 5.1(c) on putting a doxty), b doy,z), to a + b 2 c + abl c 4)@,z),and using the fact that 0 f do S 1. Unlike the Euclidean metric, Jo is defined for or y = ix. (V,4) is a metric spae on the definition, while Rwith the Euclidean metric is not. n -y
-
-
-
=
-
=
-
=
=
.x
lh the space R2 a 1arger va riety of metrics is found. 5.3 Example The Euclidean distance on R2 is -y1)2
defx-ytIIx-yII E(m =
d (q2r 4) p,lp ,
.
''
,
.
=
-y,)211'2
+(.n
(5.2)
'
.
is the Eglidean '
'
'
plane. An alternative is the
Etaxicab'
metric,
76
Mathematics
dvx,y)
=
Ixl
-
yl I +
1A'2
-
y2 1
(5.3)
.
dz is the shortest distance between two addresses in Manhattan as the crow flies, but dv is the shortest distance by taxi (see Fig. 5. 1). The reader will note that dv and Js are actually the cases for p 1 and p 2 of a sequence of metrics on R 2 He/she is invited to supply the definition for the case p = 3, and so for any cxo is the maximum metric, p. The limiting case as p =
=
.
-..-h
JA,g,y)
=
maxl
-y2l
1.x1yl I 1x2 -
,
1.
(5.4) R'T
Al1 these distance measures can be shown to satisfy 5.1(a)-(c). Letting R x ERx x R for any finite n, they can be generalized in the obvious fasltions (RR,#r), (R'l,#x) and so forth. n to define metric spaces (R&,Js), =
...
t/yg (-,:,)2 )
------------
t2-(-);,)7
)
Fig. 5.1 Metrics dk and dz on a space S are said to be equivalent > 0, there is a > 0 such that
dblx,yq < 8
=
dlx,yq < E
dzlxty) < 8
=
[email protected]') < E
if, for each
-v
e S and :
for each y e S. The idea here is that the two metrics confer essentially tlte same properties on the space, apart from a possible relabelling of points and axes. A metric that is a continuous, increasing function of another metric is equivalent to it; thus, if d is any metric on S, it is equivalent to the bounded metric dll + d4. Js and do of 5.2 are equivalent in R, as are are Js and du in R2 o n the other hand, consider for any S the discrete metric Jo, where for x,y e S, dE dnxby) 0 if are not y, and 1 otherwise. do is a metric, but Jo and equivalent in R ln metric space theory, the properties of R outlined in j2.1 are revealed as a special case. Many definitions are the same, word for word, although other concepts are novel. In a metric space (S,#) the concept of an open neighbourhood in R generalizes to the sphere or ball, a set Sdxn (y: y e S, J(x,y) < :) where x e S and : > 0. We write simply Sx,z) when the context makes clear which .
=
.x
=
.
=
,
Metric Spaces
77
metric is being adopted. ln (R2 d E ) S@ :) is a circle with centre at x and it is a radius E. In (R2 (rotatedsquare) centred on x with e the distance from x to the vertices. In (R2 du ) it is a regular square centred on x, with sides of 2e. For (R3 d s) we 11 think about it! An open set of (S,#) is a set A S such that, for each x e A, (; > 0 such that 5'@,) is a subset of 4. If metrics dj and dz are equivalent, a set is open in (S,#1) iff it is open in (S,J2). The theory of open sets of R generalizes straightforwardly. For example, the Borel field of S is a well-defined notion, the smallest c-field containing the open sets of (S,J). Here is the general version of 2.4. ,
,
,
tdiamond'
,#w)
,
,
,
...
5.4 Theorem (i) If C is any collection of open sets of (S,#), then
c
UA
=
Ae N
is open.
(ii) lf A and B are open in (S,#), then A ch# is open. Sx, c A and A e C, then S.z,z) c C. Since such a ball exists by a11 definition for x E A, a11 A e C, it follows that one exists for all x e C. (ii) If 5'(x,/) and S(x,k) are two spheres centred on x, then
Proof
(i) If
5'@,6k)f''h 5'@,Es)
(5.8)
5'@,E),
=
If x e A, H 5'(x,/) c A with Ex > 0, and if e #, (5S(x,ze4 E > 0. If x e A chB, 5'@,E) I A rn B, with 6: > 0. x
where e minlerx,Es) c B similarly, with
.x
=
.
The important thing to bear in mind is that openness is not preserved under
arbitrary intersections. A closure point of a set A is a point x e S (not necessarily belonging to A) The set of closure points of A, such that for all 6 > 0 ('!y e A with #(x,y) < . of Closure called the closure A. points is denoted A, are also called adherent points, sticking to' a set though nt necessarily belonging to it. If for some 8 x, so that > 0 the definition of a closure point is satisfied only for y &x,) f'-AA (xJ, x is said to be an isolated point of A. A boundary point of A is a point x e X, such that for all > 0 H z e Ac with The set of boundary points of A is denoted :A, and W A Q.J :4. The J(x,z) < interior of A is A0 A PA. A losed set is one containing al1 its closure Points, such that X A. An Qpen st dyjes not contain a1l of its closure points, since the boundary points do not belng to the set. The empty set 3 and the space S are both open and closed. A Subset pf A is said to be dense in A if B c A c X. A collection of sets T' is calld a. bvering for A if 4 Usssf. If each B is =
=
=
.
=
-
=
.
.
...:.;
.
''
.
'
.
.
;
'
.,>
covtink. A jt A is called compact if every open open, it is called an open kqtog. x is said to be relatively compact if covering of A contains a tinite S itsel? 'A,t,tk'.$,j,'..,k4) is said to be a compact space. The cott, is compact. If s is rk.,i tl compaltlktlyj, ,. ttk y))(y,y;(jty)):Vyty,yjkj4g eqqally relevant to the general case. remarks in j2. 1 about jkyyjytyyyjjy yjj yyy, jj 4::11. d!;p dr:;li 4k220 154!r 11::). tjyod jyyyyyktty.jtyz,y,g,jyjyyyyyjxyyyjyjyy,,yyyyyyytyy ii.1b;1b;Citii. 1l:4(E:1. rr.t).:y...;....,. ..,.4!k. ;yjjy .yj < x, sucj.j tjaatx s sxtrj; and also .
x
. .
.p,
,1.)
.))
'
t
.
rjg)
....r
.
.F ..--
.
Mathematics
78
totally bounded (or precompact) if for every e: > O there exists a finite collection of points x1,...,xm (calledan E-net) such that the spheres Sxz), i z4. The Sxi, can be replaced in this definition by = 1,...,- form a covering for noting that Sxi,z) is contained in Sxi, : + ) for all 4'5 their closures Sxinz), z4. > 0. The points of the E-net need not be elements of An attractive mental image is a region of R 2 covered w ith little cocktail umbrellas of radius : (Fig. 5.2). Any set that is totally bounded is also bounded. In certain cases such as ?n,dE) the converse is also true, but this is not true in general. . .k . .
.
.
... . . .
.
.
.
.
4
.
.
.
.
,
.
.
.
k...
.
.
.
.
.
.
.
.k
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
y .
.
.
.
.
'
j
.
.
.
.
.
.
.
.
.
.
.
' .
.'.
'
.. .
.
.
.
. .. '
.
.
.
.
. .
.
.
. '.
...
.
.
...
.
i
.
.
.
.
.
....
.
.
'.'k. .
.
. .
.
'.
.
.
.i; ..
.
...'
. .
.
...
.
.
4 .
.
...
. .
...'.
kj'..'
Fig. 5.2
5.5 Theorem lf a set is relatively
compact, it is totally bounded.
Proof Let A be relatively compact, and eonsider the covering of X'consisting of the e-balls Sxnz) for a1l x e X'.By the detinition this contains a finite subis an e-net for cover Sxitz), i = 1,...,-, which also covers A. Then (x1,...,au) d, and the theorem follows since e: is arbitrary. K The converse is true only when the space is complete', see 5.13.
5.2 Separability and Completeness In thinking about metric spaces, it is sometimes helpful to visualize the analogue R'J with n f 3, and use one's intuitive knowledge of problem for R, or at most for those cases. But this trick can be misleading if the space in question is too alien to geometrical intuition. A metric space is said to be separable if it contains a countable, dense subset. Separability is one of the properties that might be considered to characterize an 4R-like' space. The rational numbers (I) are countable and dense in R, so R is separable, as is R&. An aliernative definition of a separable metric space is a metric spaee for which the Lindelf property holds (see2-7). This result can be given in the following fonn.
5.6 Theorem In a metric space S the following three properties are equivalent: (a) S is separable. (b) Every open set A S has the representation
Metric Spaces
4
UBi, Bi e
=
79
'P,
izu1
where V is a countable collection of open spheres in %. Every (c) open cover of a set in S has a countable subcover.
I:a
A collection V with property (b) is called a base of S, so that separability is equated in this theorem with the existence of a countable base for the space. In topology this property is called second-countabilit.y (see j6.2). (c) is the Lindelf propely. Proof We first show that (a) implies (b). Let V be the countable collection of spheres (5'(1,r):a7e D, r e t1+), where D is a countable, dense subset of S, and Q+ is the set of positive rationals. If A is an open subset of S, then for each x e A, 3 > 0 such that 5'(x,) g A. For any such x, choose xi e D such that dxie < /2 (possiblesince D is dense) and then choose rational ri to satisfy dxip < ri < &2. Define Bi Sxitr e V, and observe that =
S(A7,8)
Bi I
(5.10)
i X.
Since V as a whole is countable, the subcollection fBiI of al1 the sets that satisfy this condition for at least one x e A is also countable, and clearly A c Uff I X, SO X = U#f. Next we show that (b) implies (c). Since V is countable we may index its elements as ( Jj, j E EN). If C is any collection of open sets covering A, cboose a subcollection ( Cp j e LNJ where Cj is a set from C which contains 5. if such by F-sets, as just exists, otherwise let Cj 0. There exists a covering of shown, and each 5. can itself be covered by other elements of V with smaller radii, so that by taking small enough spheres we may always Gnd an element of C to contain them. Thus 4 c UjG,and the Lindelf property holds. Finally, to show that (c) implies (a),consider the open cover of S by the sets (5'(x,1/n), x e S l If there exists for each n a countable subcover (Sxnktllnj, k e ENJ, for ach k there must be one br more indices k' such that dxnkpxnk')< lln. Since this must be true for every n, the countable set fak, k e (N n e EN ) must be . completes ptoof. the This in S. dense ,
:4
=
.
,
The theorem has a useful corollary, lc'tg C' k' ;''j' y' q.rljljjrjtrj'jk. k:5j';;'jll.' j(j('(jy';;)(;(jjjr-:';E' 'i'. 11(F*))7(j)1))*5i'-!tt)@'.tjjj: rjd t;(.',' y' (jd y.'r' )' :'''jt'ty'k). (y'r;. )' jy.yjy-'tyyd yj'(' (' )jjj'Lqjj.j;Lj))(L:'.. ;';)' j;'16'. illj.Fklfll'r!id j()$.tj!y' !l-i@;yy'. jj.'lq. j'td .'.'k(t(jjj.'. --' ,.' -' .'.,' ,'.r:,ggjj4;4* ..t-d '-7*
'''jd 7* ::.tky,(y)jjj4jj44.2* yjyyjj...d ..-tj).'.' ,.t,' jy'. E'. 5* k(j)jt)':. ;'-'kE''. ,':'.#y' ?'
)'jt. jyd (' jjryjjgj;rrjjj'. yy'' jjjjj..d jjj')j(jk)'. r)'k(!@'' )' h' tl'tjjg t'jkd
''. .' 57 @
Corollary A totally bounded space is separable. n .
Another important property is that gbspaces which we show as follows. y
sR
Tuoorem
If 4:.* is a sp' ..
.
.
..
of separable
spaces are separable.
ab ll,!.'pae and A c S, then A,d) is separable. l l '
.
.'
;
'.;
. .y
y.
:
5:.'
:
')'q q.ty ;..' j,.);.j.;Lj. jy yk y. ..) .. a,,#..z'inl.s.
....y.((
. ..
.'
)
;y..
:
.'
.
g
g
...
.
. d'tll(td. Constnlct the countable set E by ) qp;jt,. jytgr yy a, ( '' supposeD is countbl g.;..j y (. y .). . .... y .. ... .. . . yjtliltjt, . q F tnul iyl!q t ' 'y takingone point from each;.$.',tj,.A,, ' (qi r..i q. y ( .( .. jy- jy.j(. ..jjtk(,... rk ;,-j,bj,.,. ...;,;-j-tyjj-yyy.trr,pjrrj--t:. jq j rk.,yjj ' r...'. .(. .,.-bj',::,;;;LL?3b.,L, y ..g.. .. ;f;;,,,;j:;-f);;,-'LL?33'-.,-j;)--Ljj;::-),'b--jj.)L,,t,;b'f?;));j-.
Ivoof
.
y
E
.
:
(
E
(E $2 .'
EE .
.
E ' ly . .
jE. .
?.
.
: gy. q
.ptllko'tyq)
E. q: . .
:
.E
.
,
.
.
l'.r
-i
y..y)..
E' . .. .
.' ' ::EE. . . . . E .-' (Ej.jijjjjjyryjgggggg, .. . . g: E . y y . y, g . . yy . . . y .jjyy... ..j... . y.-.... . ,!j;pjj42)$g,-.,,y,..(y.kpjr)(.y(y(,yyjjyjjyj;jy.-..,...-y.tyjky,
E
. E .. yj' .E. !. - q. . (-. .(j. ; (E r
.
.
.
.
.
-
. -:;..,-:3626,3.-.; -...633333:,31,'.. . . . .
.
j .....-.-,...yjj..... . .-,,.... .....
#.,..- ytlyy.jitr.. )r,y,-.yyjyjjtjj)y.,,. .--..!1k.. ijijLjlijjjiiljl-. . ... .. ... ..
);.ryj-rkji-. 1,:-::,..-,k. .... lttjj?, . . .;(;;yyit..ykrj.-.yy-j-kjtjrqy-yj. .,jjt;-.. ...y'ptj,;,.-p!h!!!qqj--..,.
...-....
,
( !jr r... ..... . . .. :..
: : : . ..
,
-k,yt.yytyyy-y,......--...-
.
,....y
-.
.
;k.
.
..
.
,.
. ,.
.
),
.
..
.
,
..
,;.j,:-.,,j;;)j.yty,r.t y .(.. - .. .... -
.
.
..
.
4::). jjjjj.s ,-
tjggjjj
Mathematics
80
For any x e A and > 0, we may choose y E D such that J(x,y) < 6/2. For every such y, R z e E satisfying z e A fn S@, for r < 6/2, so that d@,z) < &2. Thus
dxa)
f dx,y)
+
J(.y,z)
<
8,
(5.12)
and since x and 8 are arbitrary it follows that E is dense in A. w This argument does not rule out the possibility that A and D are disjoint. The separability of the irrational numbers, R Q, is a case in point. On the other hand, certain conditions are incompatible with separability. A subset A of a metric space (S,J) is discrete if for each x e A, R 6 > 0 such that (5'(x,6) (xJ)DA is empty. In other words, each element is an isolated point. The integers ; are a discrete set of (R,#s), for example. If S is itself discrete, the discrete metric do is equivalent to J. -
-
5.9 Theorem If a metric space contains an uncountable discrete subset, it is not separable. Proof This is immediate from 5.6. Let A be discrete, and consider the open set where Ex is chosen small enough that the specified spheres folnn a Uxez5'@,exl, disjoint collection. This is an open cover of A, and if A is uncountable it has no countable subcover. . The separability question arises when we come to define measures on metric spaces (see Chapter 26). Unless a space is separable, we cannot be sure that a1l of its Borel sets are measurable. The space Dgtz,,jdiscussed below (5.27)is an important example of this difficulty. The concepts of sequence, limit, subsequence, and cluster point all extend from R to general metric spaces. A sequence (x,,)of points in (S,J) is said to converge to a limit x if for al1 e > 0 there exists Ne k 1 such that
dxng)
< :
for a1l n
>
(5.13)
Ne.
Theorems 2.12 and 2.13 extend in an obvious way, as follows.
5.10 Theorem Every sequence on a compact subset of S has one or more cluster points. n 5.11 Theorem If a sequence on a compact subset of S has a unique cluster point, then it converges. Ia The notion of a Cauchy sequence also remains fundamental. A sequence (xnlof points in a metric space (S,tf) is a Cauchy sequence if for a11 e, > 0, 3 Nz such that dxnbxmj < e,whenever n > Nz and m > Nv.The novelty is that Cauchy sequences in a metric space do not always possess limits. It is possible that the point on which the sequence is converging lies outside the space. Consider the space (Q,#s). The sequence fxn), where xn 1 + 1/2 + 1/6 + + 1/n! e Q, is a Cauchy 0', but of course, xn e (the base of sequence since Iak+1 l = 1/4n + 1)! the natural logarithms), an irrational number. A metric space (S,#) is said to be complete if it contains the limits of all Cauchy sequences defined on it. (?,dE) =
...
-->
-xn
--
Metric Spaces
is a complete space, while
81
Q,dE4
is not. Although compactness is aprimitive notion whichdoes notrequirethe concept of a Cauchy sequence, we can nevertheless detine it, following the idea in 2.12, in terms of the properties of sequences. This is often convenient from a practical point of view.
5.13 Theorem The following statements about a metric space (S,(f) are equivalent: (a) S is compact. (b) Every sequence in S has a cluster point in S. (c) S is totally bounded and complete. Ia Notice the distinction between completeness and compactness. In a complete space all Cauchy sequences converge, which says nothing about the behaviour of nonCauchy sequences. But in a compact space, which is also totally bounded, all sequences contain Cauchy subsequences which converge in the space. Proof We show in turn that (a) implies (b),(b) implies (c), and (c) implies (a). Suppose S is compact. Let fxn,n e IN) be a sequence in S, and detine a decreasing sequence of subsets of S by Bn fxk:k k n,J The sets Tn are closed, and the cluster points of the sequence, if any, compose the set C O';=1X,; IICIC. U';=I#,C,, If C 3, S (U= so that the open sets T'nc are a cover for S, and by assumption these contain a t'initesubcover. This means that, for some m < oo, S (O'2=1))C This leads to the contradiction Tm 0, so that C must LKLc n S U'''ol) be nonempty. Hence, (a) implies (b). Now suppose that every sequence has a cluster point in S. Considering the case of Cauchy sequences, it is clear that the space is complete; it remains to show that it is totally bounded. Suppose not: then there must exist an : > 0 for which such that no e-net exists', in other words, no tinite n and points (x1,...,AkJ x k. But letting n in this case, we have found a dxjtx S : for all j sequence with no cluster point, which is again a contradiction. Hence, (b)implies =
.
=
=
=
n=
=
=
=
=
.
:g
-->
(C).
Finally, 1et C be an arbitrary open covr of S. We assume that C contains no finite subcover of S, and obtain a contradiction. Since S is totally bounded it must possss for each n k 1 a tinite cover of the fonu B ni
sxni, 1/2'')i
=
y'
jyjjjjjyyjjjy'
1
=
...,kn.
(5.14)
,
Fixing ?z, choose an 1 for whlch Bni hs' or finite cover by C-sets (at least one such exists by hypothesis) and all thls st Dn. For n > 1, jlBni))=n 1 is also a jl tjjt X h & X n-1 jja.s Sujpcgyeyy COVCf1ng jw p n-1 ap wg ggyj gjyjpjjj Xyr,(,.)!y;yy;jy fojyjj* njj tg F yyjj . kyry,j;. :,, jyzjojjsea sequence of points (xn e D IS nonemp tyi yj.g.(..)t;(( C-sets, and accordlngly n, y .(-(j,jj(. . )y.( . y.kj. s l tontalns ak, and Ds+1 1s of radi us ) since Dn 1s a ball Of radm# .,.. yy,,.tyjjj;,,,,gy; .yr;yy, jj-j jygyrgjs jkgyy jjygj.. . jtyjjy yggjj jkgyy ;yy gjg gjygjs y....jjygjyyj jjyy;yy (j. yj.gj. ikyj. gj. ;jy yyyjy yjs gjygjyyyg jjygjs $j44,, gjs lln- gj and contalns Ak+1 dxngxq-j) .iJ (j. gyjj,, gjjgjj y 0'1: '''''j . !.E ',..j';;jjc''''j;.)',.*.,.,,;.;,)k,q..qL,.j'q,)j-,Ls))jj)q,;t)).L'j,.j !1..2,'... ......1812.....,.... 1t!12$!E;k..
w
,E
,
.
y
y
,..
..f.kj)
L;..
.
E
.
y y t .y..;.. .. y.... zt'i')>t7)Eitu . . .,2:jj,g.
,1
.
-
-
..
,-
...,,q..yyy-
..,,.j.)
.
...
.
y
..
,
....
..ksysj-sssssyyyssy,j.yr
,..syyyy
.,
.
',','t.)y;;r,))',tyyyiy),.,),y!(jjj,y)g.,,yiityy;j,t,y.,,yy,,,jyjj,
y
,
''''
..j.,
,,
..)
=
j,;.jr
..
d,
jj:rr;jj
''
-'::::g
.iltrr;,:r
..
',-
q.g...j-: .E , q..
.(...j!;r';-r:).,..-jj)...
jjggygi
.....
d.
...t.
,
:
ktrriiir'dt:l'.k;,d::r '-,.,,
'-::::::'
...,
g
..,.fj),)),)jj)j)),,
..
,
...
.
..
,
.rqrrty.r. .
.
.g .jjjj. j,.
y..ygyjjjy2jjj;;;.yyjjyygjjg ;,,.q.;u,..,..,,,,r,,..
jjjjjjya.y.
.jjjgjrj...
,4..
..,..
.,r,...r,r,,,.,....)))))jj,jj.j:)jj(j,;..(qjj.
yjyyyjyy.yyyyj;yyj.. ..,...yyyyyj;..;.
..g yt -t-;t4i!)7-.ygjjtt' -
.
yjjjgj.y,-pk@$1r....-,... . yjya
--
.'.
'' ' .' t.ylii. Ji. .;..
:..... ,-..
..
yjjjjjjjjyyjj.g. yy,,yjjjyyyj,... g.yy.. yyy ...yy.,yyyy;
....yyjgyyyyy..
..
.
--$t'-.
.
.yyjjyyjy.
jsggggggggy
,,..).:,.j
...,..y .
,y
..
yyyjy.yygyyjy,
..yjyyyy. a.aayyayay..
y.
..
y,
-;jj-t.yy.
.
-...
--jjy;;-
..
.
-zi
.
....
..
.. ....
.. yjyjy;.;yjjry,;;jy..j(.. .g...j.jgy,...;.g..g........y.y. ..... . rysg..;
jjryrrjyjtr
.
.')' yj..-T!rt)j-ji.. .. . '-ty,--i)... ,,- EEE; ;.,-. ,qjy)j)y(. . E ,;,'.-);,.. ;' E..:-...'.-..?.-.., ..k:, ;.hF(.f.
...
-..
,..y......
,
.g.yyy..
ysy. (gylrqE..jj
.
).
.
jjjjjj,jj),,,j r.:g; .,yjy,4E ..ypj.'q).k(jyj.,j:;II,..: ),j)jj,..t jpkzjlk.g))jjjjjj):...k -(...
.:;,,::7(22,-
bif'
-.;(
)F?jjjgjjj:... r.()
j .. y,y.q.yt. .E ;..... )i):y:.
j!jjjjj).
jtkkgry
jtg
,,'..
.-;p7.
...
.jrr,
jjyyg;jj jjygg;jjjggygi
.
.'r'y)'-'.
,444,.4
.j(r,
!j,
,jjy,
''
.....
.j;,k:rr
.!jjj;,.
.
''.'''''12.12
.
jl,.
-.
.E.'.:.'.y..E.E......:... ),1lpl-t''t,-;-)@--
.,
.
...
.
.tlih'tjjrt.;2).-:.lqt.1r):.
!,-
77?:?,-' .... ;,..-.. b3-,Lt;j,,1'3$;t311L;;,;(;L)3$L;-t..i;'.).j)). .'.ti;;jit)()(.. . ... ;-
1t;;$(yr';)))(;...---.j$)yyk?Ey-..,it). -. . '-'. -(jj(-,-.'. '.
.-..
yjgy;yyjygyyyyy jygyyyyy
.
.
.
..
,. -
.
.
.
. .
.
y;jjjjyy
82
Mathematics
< 9Iln ensures that Dn c Sxn. But this means Dn c A, which is a contradiction since Dn has no finite cover by C-sets, Hence C contains a finite subcover, and (c) implies (a). K In complete spaces, the set properties of relative compactness and precompactness are identical. The following is the converse of 5.5.
5.13 Corollary In a complete metric space, a totally bounded set A is relatively compact. Proof lf S is complete, every Cauchy sequence in A has a limit in S, and a1l such points are closure points of A. The subspace (A,#)is therefore a complete space. It follows from 5.12 that if A is totally bounded, X is compact. .
5.3 Examples The following cases are somewhat more remote from ordinary geometric intuition than the ones we looked at above.
5.14 Example In j12.3 and subsequently we shall encounter R=, that is, inhnitedipensional Euclidean space. lf x (x1,.n,...)e R=, and y @1,y2,...)e R= similarly, a metric for R* is given by =
=
X
d-x-y) >72-Q()(x,,y,), al
(5. 15)
=
where do is detined in (5.1).Like do, t/x is a bounded metric with #xtx,yl f 1 for all x and y. n (RX,Jx) is separable and complete.
5.15 Theorem
Proof To show separability, consider the collection
4
=
xk rational (x (x1,x2,...):
if k S m, xk
=
R=
=
0 otherwise
)
(5.16)
.
(A,u, m Am is countable, and by 1.5 the collection A and e: > 0, H x e Am such that able. For any y e =
R''O
m
=
1,2,... J is also count-
co
d-x,y)S y2-$ -al
+
-t=
5-' 2-4(0,yk)
S
m+ 1
e+ 2--.
(5.17)
Since the right-hand side can be made as small as desired by choice of e:and m, y is a closure point of z4. Hence, A is dense in R=. To show completeness, suppose (xn @1n,x2n,...), n e N ) is a Cauchy sequence in R ' Since dtxknvxkm)S 2J x @n, xm4 for any k, fxa, n e EN ) must be a Cauchy sequence in R. Since =
.
d-xx)
/
57l-kdxkakn) 1 -t=
+
2-'n
(5.18)
Metric Spaces
83
for all m, we can say that xn x @1,x2,...) xk for each k e R= iff xkn completeness of R implies that lxn) has a limit in R=. w 1,2,...., the -->
--)
=
=
5.16 Example Consider the infinite-dimensional cube', g0,1J=;the Cartesians product of an infinite collection of unit intervals. The space (g0,11=,#x) is separable by 5.8. We can also endow (0,11= with the equivalent and in this case bounded metric, X
772-11x: -ytl
Px(x,y) =
(5.19)
u
.
a1
In a metric space (S,*, where Jcan be assumed bounded without loss of generality, define the distance between a point x e S and a subset A ? S as dxl) infysxJtx,y). Then for a pair of subsets h,# of (S,#) define the function du 2 s x zs j..o R+ =
,
W here
2S
is the power set of S, by du(A,B4
max sup d@A), sup d(y,B)
=
xB
(5.20)
.
yGA
du(A,B) is called the Hausdorff distance between sets 4 and #.
5.17 Theorem Letting J'fs denote the compact, nonempty subsets of S, ls,du) is a metric space. satisfies 5.1(b) since the sets of Rs are 0, so that du is only a pseudo-metric for for any x E 4 and any z E C we have, by d is a metric,
Proof Clearly du satisfies 5.1(a). It closed, although note that duA, W) general subsets of S. To show 5.1(c), definition of dx.B) and the fact that =
sup dx,B) K supttftmz)
)
dz,B)
+
Ac#
xcA
(5.21)
.
Since C is compact, the infimum oyer C of the expression in braces on the righthand side above is attained at a point z e C. We can therefore write + dz,B)) < snp tnf(4(x,z) supltms) A ze xed
.xG
supltz,l). sup (;:,6'D z
(5.22)
.#
y', :'y' rtjy'E (;,'. :'. E' j::yt',.'Lli'y.jjy:yjj'lt-.kt4j?iyrj)j('.' y' j'yyjyy'. k'. ,', jjt!rrg;ji!r'd 111:::2;11!!'* :'i'.j,,g.' sgEy-yzgyj',d ;',rjjy't. (' !yj'. !,,' r' ;'y'g y' .L'. ;'. :@),'.,.,,;y.jjjj;g'j;g'kjrt.ilyjrg-7ldrjjklypysjlljpt'jjq'lrsr'd k' j't'yyjy,,, y'y .r' 44::7* jtd iE-':jd )' j'y;d qtd .y' jgy'1!. jydt.yy.rd j'gi i-(' litjjjltsiytj'y'. j);';' 4* r' yyd t':g'33t'. yjjy'. y;il'. y'jy,ytd yrd ytyy'.' jyy'. yrgj'.;yjyjd .'qL);)'. (''.. y' y-'( jy-'). (i'yd ygy';6!r;.'.jygjyj.yry!;(r!. jj',;L?'. .',jy',rg..jjjj. -'y -'
)k',.;jr!j'(;;'. .-' yy.j.-d .;j,' 33666666636663*, r)'... (tg,' ,,,,,,,,,,,,,'
''. .': .'.'.
sl'milarly,supyssltxz)
supzs
..??
...
-<
,-
.
.
:l::)Er!t. ax
s
(x
dr:s
ll.,l. (14:,.,, ..441k.
dEE.
.;)t:r
...1..-.
.
'..
.
..
.
'
:
.
,
.
'
,
and hence,
,
.
E ..',' .j .k.. 2..' . . . . . '...,-...,.'...g...,.....-,. ..-..:!:.......... ..,..k.., -.s. '..';
(2), drr;ir.n
---.11..
(z ) + supyss:
-<
'
.i(:).-, 'i.ii!Eil'.....,...... ..,1:11.. s..,...12.. tii''fL;.:, (;,..qI...sll.,l.jll:), .!147,.. ..(14;,.. :;.
......(2,...... . .
....
..
.
..
i
. . . ' ..
:
:
c: )y)j
'
.
'.'
. .
'
.
'
,,.-,.11..
.
.
..,..
..
:. .i. E
:
C
: E: E j) E ) ,)L.q/;j-i...L,j;...3L;.....L.. :.E.; )...E... ..'q''. . . tlLi: ..;..-..1qs? .... E..E:(:; . ifr: .q. LE . y(..!. . .: q ( q..;..sy . ;LL;.1Lj. y:.Ey.tjyy.tpyit;kr . . . .. .... ' . . E. .. . . LL..L......L. . :Lj.LL..:....j.L.. Ljf,. . .: . . E. . .. .g k!:..(#k:gk....jtt .... L).;):,.;.)..).;.:L.....j;. ....4:j.. :.. j . .: . :....:(. i. .E. .jg (.... .. ..,. .... .. j.; , :. g. . j y: .;....L.. ; . .kj. jr.). yr . j y. .. ;'. ' (yj ( jjyE. k:, y . ..j E :.' E .y . y! :. . .. .y-.L- . :. g.( q j yy.. j.:; . ..r . . r yg . .y . . . .. .. '. yy. ' y;. . . y y yy . : .,),.;.tj..$ : E i: . . . .. . . :yjytrr...,:jygj ;.,y..... y.y... ! gs.tjjtyyyjygjgytjrjry 4)f,Ljj;.L;;;j).);).f..j;q))..)):k.. L...j),)j)):)..,..,.))q,..L4)q);;q;. .. .L.(. j..-.. y y. yy . .yj r j:.j).,.;t,), yyrjyjjjjjj., y.y. y .j.. .g yy. y...
E
.1:7
-
j
.1,.!;
.
..:..jj..y!.)y.g;...::.).j
....(
..
....;)..
dr:;ir-
.
. .
t)t,.g..y t:
..
.. .
.
.
.
)5.jk; qL., ... .j.y.. (. ..tjtr .. ( y.)(.) .. j.jjyt.yjt... (j) )y(jjy,..t;.yy: y.. . . . . jj;, (yg j . .yt;jg yq . (..;.j j,(jj . t' ):b,. qi(gy-. q.. ...:::::::::( ..... yyj . j .: . . . .. ry .yt,. . . . .j.. y. . jy...yry.jj,s .. t)$:.,;jjg:. . i.. yy,y . ItIL. t.y;y. ... y kk))r.' kjl.yy ; (-. .'. jj;.,. ' ...g. Lqjjjf. rjy . yy4tjj,r.. fjjjj.. .. . ; 11k2. dtr:;.!r-'l)p--.--?-j:,,-(-,-----. ----.-b--b-,i-------;-jbv.... di:22:). ...t .-... dk:r:..'...i...---..-).)--..-.---...-..-........---...;...i..f..t--...-;:., dl::::: 4.:2::,. dliz!,.-.. 1lE5'..lI::!'... :llr7.:':1L. 4:7-.. . .-tk-. ,,- :.1. ,--L-.-----?,,--. II. :!i4;. .. t,!!i9;...qlr.... ....-.... .-........-k;!!iiIr... a.)-....-.--...-.,y.,-.;,:.-... 11:::,,. ..
ti!iiE;k ,(:!:; -,-
..
:
..
...)
. ...
.,.,..;yyj:
:(.ygjq
..
.
.
.
.-jji!!!ijiij''
-.-..11......
,j.!!.!!!!Ii;... ' .
....
.
,
.
.
.
. ..
,
. .
.
.
.
.
.
:
. .
.
....
-
.tg,
,
..
.
.
.
.;
,
.,y...
.
-
.
...
j)
..((y.- -.
- '
- . .
tjkj;j ..(.
..
.
,
.
,),))))j.
.
.
. . .
.
.
,,
(:3?'.i!EiI? dE!!!E
. -
.
(5.23)
.
. .
...
.,,,:
,
.
.r
ll.
'1r:.
.
- .
.
,,.
. .
..
.
.
.
...
.
.
.
.yjj.,.
.
-.
Mathematics
84
5.18 Example Let S = R. The compact intervals with the Hausdorff metric define a complete metric space. Thus, ((0, 1 1/n!, n l EN) is a Cauchy sequence which converges in the Hausdolf metric to (0,11. This is the closure of the set g0,1) which we usually regard as the limit of this sequence (compare2.16), but although (0.1) Cts, JVE0,1),(0,1q) 0. u -
=
Another case is where S @2 J s ) and J'fs contains the closed and bounded subsets of the Euclidean plane. To cultivate intuition about metric spaces, a useful exercise is to draw some figures on a sheet of paper and measure the Hausdorff distances between them, as in Fig. 5.3. For compact yt and B, duA,B) = 0 if and B; compare tlzis with another intuitive concept of the only if A between two sets', infxsx,yss#stmy), which is zero if the sets touch or =
,
tdistance
=
intersect.
'' . .. .
. .
. .
'
.
. .
*1'1 .
.
.
.. .
.
.. .
.. .
. .
'
'
.
.
'
' .
.
' .
. '
.
.
.
.. .
'
.:::
.
. .
.
' .
.
D
: : 727::!:::::12::!; :;:: :::7::t:::!:!J1:::: ::'' :: : : : 7t : : : : : : :: :: :: )(72:2 : 7 7 ! : : 2. :: : l : :: L : l : L: : : ' 2: : : 1 ) . t :2:::::::::r2:2E:1:::2r:2:J:;::;::::t:
. .
' .
'
.. : . . . ..
'
.
..
.'
' .
' .
'
. .
'
'
' .
. '
.
'
. '
. '
.
' .
' .
. '
.
' .
' .
.::::.:.:.:.:.:.:.:.::.:::.::l.:::.:::;;:::i::.:.:.::::.:,.. . : .
.
dr;lr
igrjr
(:) a//qik
aj!jjl ,
()h
Fig. 5.3
5.4 Mappings on metric spaces We have defined a function as a mapping which takes set elements to unique points of R, but the tenn is also used where the codomain is a general metric space. Where the domain is another metric space, the results of j2.3 arise as special cases of the theory. Sme of the following properties are generalizations of those given previously, while others are new. The terms mapping, transformation, etc., are again synonyms for function, but an extra usage ksfunctional, which refers to the case where the domain is a space whose elements are themselves functions, with (usually) R as co-domain. An example is the integral defined in j4.1. The function J: (S,#) F-> (T,p) is said to be continuous at x if for all e > 0 H > 0 such that ys
p(.f@),f(x)) < z. sup sdx,n)
(5.24)
Here, may depend on x. Another way to state the condition is that for e > 0 3 > 0 such that
f(&(x,))i
5'p(.f@),e).
(5.25)
Melrfc Spaces
85
where Sd and s'pare respectively balls in (S,J) and (7,p).Similarly, f is said to be unbrmly continuous on a set A i S if for al1 : > 0, 3 8 > 0 such that < :. sup p(.f(.y),J(x)) supy e 5'J(x,)t''4x
(5.26)
xGz4
Theorem 2.17 was a special case of the following important result. 5.19 Theorem ForA I 1-, f -1 (A) is open (closed)in S wheneverA is open in 1-, iff f is continuous at al1 points of S.
(closed)
.f-144).
Proof Assume A is open, and 1et fx) e A for x q for some E > 0. By 1.2(iv) and continuity at x,
f-'fsdx,nbLtj c
Sdxnn)
Wz have Spflxln
? A
J-1(Sp(f@),e)) i f-'(A).
(5.27)
A is closed and f -1(V A) s If A is open then by 1.2(iii), which -1 is closed if f (A) is open. This proves sufficiency. To prove necessity, suppose J -1(A) is open in S whenever A is open in 1-, and in for E > 0 is open in S. Since particu lar J-1(' p (/x),E)) e f-(S p (.j(x),:)), there is a 8 > 0 such that (5.25)holds. Use complements again for the case of closed sets. . -y-1(4)
''
-
=
-
.x
,
This property of inverse images under J provides an alternative characterization of continuity, and in topological spaces provides the primary definition of continuity. The notion of Borel measurability discussed in j3.6 extends naturally to mappings between pairs of metric spaces, and the theorem establishes that continuous transformations are Borel-measurable. The properties of functions on compact sets are of interest in a number of contexts. The essential results are as follows.
5.20 Theorem
The continuous image of a compact set is compact.
Proof We show that, if A c S is compact and Jis continuous, then /'tAlis compact. -1 Let C be an open covering of fA). Continuity of f means that the sets f @),B s C are open by 5.19, and their union covers A by 1.2(ii). Since A is compact, these sets contain a finite subcover, say, J (#1),...,J Bm4.lt follows that .A
.j
y(A)
c /'
.j
c UBj, U.f-'(#y)U.f(.f-'(>) y=1
(5.28)
-
jv1
y=1
is by 1.2(i) and ihe seond inclusion by 1.2(v). Hence, Bj,...,Bm is a finite subcover of fA) by Cij4ts. sinceT' is arbitrary, it follows .jyjyd -),'jLiii'''L. k' 11:222:::;* y'. ;'y'. jjj;jy!r;'. y' yyd !)' -i-E.(E' y'y y'ty' ry'j y;t'11I::2t!):.!q'-?q#').i. q' j-q,,,4fb.'. ..' jy'. g'g'. j';;;r2qpjjy.'. (' jj122222,,r*. y'(!jjt:r!''-, ;'(j1Ii(r:j11:qr:1I,;.. diIt!j:rjt)'. (jii).d (r:ljill;k.d .y' y' rlll:rjjd ..-,;;p9i;j:r'.-' .tjjjrrjjq:!)gry:yjj(,.'. gjjjjjjjjj'.' j';' jyj''' (.(;-'j4' :!jEi;,-' 1;:Ej5)11,,*. j'y .r.j;4'?i (' LjjkLbftjt' 11*. y-' L-t--..,,blfbqk:t''L.' rjjlg)d rljjj;;-d ..' y' yy'.'y yyy'...-y'yjj'y,(;jjry.qjjjyy.rjj(.. (yjjj.'. ty'-tj.ytjjj't. -.-,'-qi;'fqbI;;'.' (111:r::114.*. (jjlyrrr!ll.d .,' .g' ty.-y.jjy'..' t)j)-',. that fA) is compact. w )')66fLLLj. .jjqn,qjv...'k::kf'qktLLLl''
where the equality
''.'.' .'. .'rs .'.'jy,.
-(j(y'-y' -'. --((2:jp,'.' ,:pj(,jg.' -j;yjjjjj,jyjjj,,'. '-jjj::jjrjj;'. ;'F:' ..:,' ';t';--,::;;;jg!!r;y.-j)-' ---')::' -.' ,'ygjjjj)jjjj(yyrjj(y,'yyy -'.' -' ,':' :-'jj(jgj.y.' -'jyj' ;',',' E'jyyyy j'.-kk;'. )jyyjy-.'' t-,'. y'yd jjr:;rr:r;d jy'y'((r)))I''C' .i-'',,j(;)' y' t'.-jjjyjy:jjj,.' yrd .t't(;yy!ryryjjjgr;rryy.k' ('. jr:'-'.:.,,-j,jj;jjjjjjk,:.,y)--.)t;'i.'. ;,,*j3.*313413;*. ---'ibbfLi',,j,'jg'y.yyj-,-l t'yjytt'js j'yd .y' g' (y;'. 41kk2:::;* jjkykzrr:;d 11;j2r!:;'. .it')E y' :',jjjkjg,r;j'. ;'yyllllirrrjlk.'gj .-;'.jjjjjjrjjjjj.'. t.' :''jjj;:rjjjjj...' )' .y' j'yj,d
-,' .--'.'-.;k1ijj(2!t$;'y ..' .11;:2)* '',*4111::)*. .111:2k::1,,*. -' .'. ','.'i;'i---'jkg;jjpjr.. -'j);,4,j)tf,'3t',' jjryjpjiij'j.. .y'yyjjprrtjjj-;' ,'y.jy'-yj -'. ,k-'-
5.21Theorem If the set
Jis
. .
continuous on
.
,
.
.5.(' .
.)
..'
.
. '.. . .
'.
; y.' . j . '. C
.'
.
.
.
fj....
'
'
.
.
.
( (.
.
..
njpd set, it is uniformly continuous on .
.
' '. ' . . lrEi ''.
'.'
.
.
.
..
.
,.
F y. jjl kj.g::( .)()('; k...
E .r:
..j
.
....
.. .
.
.
j
.
.
ryyy,y',.'yt;9y'jr,tty:j(,.i.yyy,rr,,y; tt'tkttyt, d fr eacil x e A, continity '' a.t x S be compact. C'..h,ptt ...!)E;1il):il).l!;)7tj!: ;..r' .'ij.tE itJ))j#15! .. ... ' 4).k6.). -. 7(; (.q-( ... .. . tjq. .)j k;jy'jiI-k::IIL.. y-;:!iii;y-. 5. .. ' :,!iij;;.. y. li .;;,.:ijj,. ..f- lil-. .IIkrIT).. (,11:::,14,.. :!ij;;--:1EE5111,( ...jj...y....-....C4;CIIi:jyjj,yy.. j;j!jrr:jjy. (qjj::--. ,11I::2jII),. jjjjyrr;jj:. iljj;;: rjjygq 11(2rrjjIIl,. ' . .... . ..yy.. y...y(IIE:r:11l,.. y .. jjjjgl:::)).. . (j11(r:)l1... t!jjI(4(:jpt.. ' gg.. ... .. j . jL:jji,,'''. .. : . .. E j . y . . . . '.. i... . j.j y..... .... . Jiii .).. . .(. . . .r....yyj .. y. y jg.. j. . j. ..yy.))j,. .y . . r jjjyy:t.. ,jjjjj)f:63L... ' ' . ., rrrjj;!lL . . .d1EE!i1. .. . .q-.'.rllqq-ll:E),s jj!jjj;j,. 4422::, d;::::: . y j-. . t. j .....j . .44:)( 4(:4):2,j-:, 11E1!2, . .... (Iir11.. . . tlEsll, (Ir:'' IIEE:!'' ;,jt-.,...jr))-...jj;tj.yjy gj .yy;y-(...-.,#.,.yyj,y,k.-.-.-.yy..y?yy)......t-,,..-.yt;y-.. jky.y..,y(y.yy(jyy!..-jyy-.. . j.yy.y ..y .j.... , .) .. y yyy.yjjyt.. jyy.., j.y;jj.. yk,-..-jy.)..;(.,... ... ,yyy.jjj. . .. ... jj-.. )j.,...;,tyy; 44:)( ((;), ((;), ((;), .; ..... jy.. ;y...yjtj;.....,jy..-jj#,,-.j;L... . ,.y.... . .. . . . . . ...j..... . . .
Proof Let A
.1jlgq,
-:rjjgijr,.
,gjj!r:r
,,.
,gyy..t: ,;....
. .
y..
y
.
E
-(:!i-l'
'
--. ..!tr:())yry-
.
.
...
...
-...
. .
.
:.,..
.,-
.
-
.
... ..
.-y. -.
-.74.
;' ):: l;!. ?!-):;)!t27tE.''''. . ,b:3b... :.'.'.,.Ekikkir:....-.. .;
--k::::::
.j;1d:r7
' .
.
.
.
''.
.
.
.
-
.
..''.
..
.
'.
-
-, . -
.,
.
qE-y q. -
E $it)kyy.)jtt.i..rjjjy)y -.-y j. .- ,, . .
,
.
....
. -
-...
..
.
,-jjj,!r!,jjj(j-.
. .
.
y
.
.
.
.
.
..yy)....r.-kt..j). q. ..
....
.
.
.
,
y .:
.
.
;.;(gt;-). .-.
..
,j.
t;y
E
' '-'..
.. .
.
..
.'Erjj,?.
.
!.
.
.. .
:
.
.
, -
''
,g;j/r:'
.
.,
E'7
. .
y
.
jjjyyygyjjj. ...
.
-,yy -.,
y
.
.
y
...
tyy . ,. y..y..;jj,-, r,jj:j:r:;;. . y-jjrrjqyjjjjj-t., - ,)..t,y.-.y,.yj.t,,yt.,yy,(.t, .y . . .
,
,
.
,
..
,-,.
-
..
.
.
y.
.
j
j . j..yy,.-y.y?.-y......-E.-y-.q.-..-.(q.y . ......-j;.;.;..-,:.,j-).j.;k.q.;jr-,y,..y,.yyy.-.t.y4.
...r.
.
.
-
...(j.....
....
.
. .
.
.
.
.
-
y-. y. .y yyy.
.
-
y. .
. . . . yy. g y-. . y-. yj y. -gyy. . . . . . y.. .. . .
y. .
,
-
-.
Mathematics
86
1,...,-. compact they contain a finite subcover, say Sdlxknr, k consider of points x,y e S such that dlx,y) < minlckcmrk, and a pair < zlE, and also e Sdxk,rk) for some kn so that pfx,fyjj dxkvx)S dlxk,yt + dlx,y) f rk + 6 S lrk,
using the triangle
inequality. Hence plflxnfxl)
S
f
.
(5.29)
r1e,and
< e. S p(J(x),f (x1))+ p(Axk),J(y)) p(f@),J@))
Since, 8 independent of x and y,
Let 8 Now, y =
=
(5.30)
is unifonnly continuous on A. w
T' is 1-1 onto, and f and f -1 are cont inuous, J is called a homeomorphism, and S and are said to be homeomorphic if such a function exists. lf S is homeomorphic with a subset of T', it is said to be embedded in 'T by f. If f also preserves distances so that p(f(x4,f@)) #@,y) for each x,y e S, it is called an isometry. Metrics J1 and #2 in a space S are equivalent if and only if the identity mapping from (S,J1) to (S,J2) (themapping which takes each point of S into itselg is an homeomorphism. If
.f:
S
F..+
'r
=
5.22 Example If Jx and poo are the metrics defined in (5.15) and (5.19) respectively, the mapping gl (R=,#x) ((0,1)=,px),where g = (:1,:2,...)and --)
is an homeomorphism.
Xi
1
'(x) 2 -+
=
2(1
dxnyt<
t.tkJ is
Ixfl)
,
i
=
(5.31)
1,2,...
EI
Right and left continuity are spaces, but there is a concept of the range of the function. A semicontinuous at x if for each
If
+
not well defined notions for general metric continuity which is with respect to function f: (S,#) >-> R is said to be upper E > 0 3 5 > 0 such that, for y e S,
=
/'(J)
<
fx)
(5.32)
+:.
0, upper semicontinuity a sequence of points in S and #@n,x) implies limsupa/xn) f fx). The level sets of the form (x: f(x) < aJ are open for a1l (x e R iff f is upper semicontinuous everywhere on S. f is lower semiis upper semicontinuous, and f is continuous at x iff it is both continuous iff upper and lower semicontinuous at x. A function of a real variable is upper semicontinuous at x if it jumps at x with fx) > max(/'(x-),.f(x+)J;isolated discontinuities such as point h in Fig. 5.4 are not ruled out if this inequality is satisfied, On the other hand, upper semicontinuity fails at point #. Semicontinuity is not the same thing as right/left continuity except in the case of monotone functions', if f is increasing, right tlef't) continuity is equivalent to upper (lower)semicontinuity, and the reverse holds for decreasing functions. The concept of a Lipschitz condition generalizes to metric spaces. A function f on s,dj satisfies a Lipschitz conditionat x e S if for > 0 3 M > 0 such that, for any y e Sdx,), -->
-f
Metric Spaces P(.f(A'),.f@)) f Mhdx,yq)
(5.33)
where (.): R+ F-> ER+satisfies hd) 4, 0 as d 0. It satisfies a unrm Lipschitz conditionif condition (5.33)holds uniformly, with tixed M, for all x e S. The remarks following (2.9) apply equally here. Continuity is enforced by this conditionwith arbitrary and stronger smoothness conditions are obtained for specialcases of h. .1,
,
Fig. 5.4
5.5 Function Spaces The non-Euclidean metric spaces met in later chapters are mostly spaces of real functionson an interval of R. The elements of such spaces are graphs, subsets of R2. However, most of the relevant theory holds for functions whose domain is any metric space (S,#), and accordingly, it is this more general case that we will study.Let Cs denote the set of all bounded continuous functions S F-+ R, and .f:
de/ne
duf
g)
spl
=
J(x)
I
gx)
-
(5.34)
.
xe S
dv is a metric.
5.23 Thorem
Proof Conditions 5 1(a) and (b) re immediate. To prove the triangle inequality CN, write, given functions J, g and h .
ttd .yy.(' t'yyty'
t';'j' ;'E y' t'(' )' t''r' y;t'. j':'jyjy' j'. y' (' .);' j'g' .-j;'. -' -'.jt'
.'.'
dfbhj
=
SLIPj
...
Ies
:
'.
'
.
'
..
...
-
hlx)
.'t ' '.
''
.
.
...4(:). 11.,1.. 4:yx) <- 154, jj(:),. ...j),b:. ). ;.3:''.' .x.....j. (.'.(. ).t . 1......#..... . ).I..jI..j .........4(:4.. .......+.,.... .j.. .
j!jj,r
S
IE
<-d
'
.E .E j : ( .. . . . .'' .. .. . ... .g . j.j . .!#.,p
..
q' . . ; g.: . . ... g.. jy . , .. . '
g.. .
,g.)
.....:.(
'
. .' g.. . .
.
.
.
. bhli ..
,
(5.35)
.
;yyr.,
,, .;y:ry.,:tj..t.yjyEy.yj..-.y i.).. ;.jLE:.y. ... ... . j.y j q.. -
( .. E E riy; . E .. ;)(-. E :..
(y)j
4:yx)j
E :'
(
'
. .
. ..
rt).yjy r ; .y( y. .q.. .): .. . r:l).))-.-'jjyyjjgjj(jgj(yjj(ytk.:i'q).;'f'l. . (2yjy y.. . j.3fk.i.. .yy ..,)L-) t!y(j-(..y. .ytj) (j-EJII?). ..' ' @.q .; )jq)j,gL li.ygry j)..)j) j,yy .;yjgjjy;j(jjtj;;y yyjjt yj;( jkjgr jjyi jjjjyj. gjypj:g;yjjyj y;ry..jy.jir.,,.ggyyj.;, .;.g....y....y,j.t-(y.jy.yy. gj. .
.
.
E- .
. ..)...:t;!..k...
:
.
) ;
j
j/ri!r
--....
..
...jt
-i
. . .. - y jEjqy t..t(Ehi.( ..
..
y
r
-y,.. .. ...-
, -,:j;
..
:.r,
-
,
q:,);jj..-L.L,.j;;.,,fjfL.. . . . k; , ....
-
-,
,,
,jfjj),.
jy....(:.j)-ri..(..y,.q jkg;y jjgyr $, gjygjggjj gkyjryyt,..44:. j)yljjk.jj. . y. . gjjg;j .jj. yjjyjyyjjjyj..,.. .. .j ;q f't;jtj.. vy,, t(; , yy t-.-tjj,..--tj:.?-.,gjykjjj,y--,(11). ,1:2:11Lqf ....y)j.jyjJ;;.-,g;yyr; . dlzz!ri ,1:4::,1 (11.-y -:3)q.jL:'6t3. jl:r-.lk:. ,L...tj,j,(.L)L;));;;--):-:fi. yj jII2:bi:.I2:bi:. 66113L32:'......,. jyjy yryj,jjj-.;jjs---, llk.zll. ..--tE!i!t.r.' :i4; Jjiki11:::),. jlr,:lk.ylrrrllr.rlk.tjjgrr)l fyurjyjjy (; ou yj uou s z.lttrit/lltq $:E1,1, ,..).L......j .,,,)#t)-t!s,i(.. yjjy. j.... rlrr:)ks .. . t..-,,.. ,r,.,y.-,1,). ....,,....s....-r......-..k....y,.z..-. tt. .-,,.,,. -,t!)1y),ty,,j;-.,-..... '.y?.-.-. .. . (-v...;;,.--k.,,,. iills. dk,r jj-,......-,,,,,,,,,-,-,,.;2,... ;;,,,,-#.s---,...,.,...,..,.,...=
.-d,d!!qjj-.
--
IS Compact,
.
'1rr.
.
..,j
.k E'ktliikt..
.1rr.
r
..yyj
.E#.(. .'.j)
t
.,,
...
-
-
. .
?
.
y
4:(11. . 14k1(:2: 4(:;1!' 112. 4kj2:ir;lj!'i!r 4k22: Io i!'-l!hl!'i!. llk(1. (1. llr!l. jry;?-21. ma.j
S
(x) (x)+ glx)
.jj.x,
.
,
,
..,;,.yy.:
Cs
k!r)-;,. y.. & 11::,-.
y ut. .#.. . ..g
.
<,
,..
.
.,
. .
,
.-,..1.
.
.
.
)
.
,
,.-y-.
:(q)
s..,..l
....., ,..
.
,
.,
.
functi o ns lf tj. !lr:.L d:::r d:::r Ik,r (1k. l. dlE:,dt:EE::'t!!,; atl (:j,,. m(14:,, a t ev. 'EEE!E as . -,
.
-,k-,.
-,k-,.
,-
.
-r,,--
Mathematics
88
a uniformly continuous restriction to S, and evel'y f e Us has a continuous extension to V, say J, constructed by setting Jx) = J(x) for a' e S and limu/xu) for a sequence fx:.G S l converging to x, for each x e V-S. Note that duf,f). so that the spaces Ck and Us are for any pair fbf' G S, dvf'l but these isometric. There are functions that are continuous on S and not on cannot be uniformly continuous. The following is a basic property of Cs which holds independently of the nature of the domain S. 'x)
=
=
,
5.24 Theorem
(Cs,#g) is complete.
Proof Let (Jnl be a Cauchy sequence in C's ; in other words, for 6: > 0 3 Ns 1 such that dulfnnfm) f E for n,m > Nz. Then for each x e S, the sequences (/@) ) satisfy Ifnx) fmx)l s Jtz(/,Al; these are Cauchy sequences in R, and so have limits fx). In view of the definition of dv, we may say that jn J unifonnly in S. For any x,y 6 S, the triangle inequality gives -
-->
lfx)
-.f(A') l f
1/'(x) fnx) l + l/@) -
fnlL
-
I + Ifn
Fix e > 0. Since fn e Cs, ('i 6 > 0 such that Ifnlx) Also, by unifonn convergence (; n large enough that -
maxfIfx) fnlx)l Ifaly) f@lI l < -
-
,
so that Ifx) J(y)1< z. Hence f e -
!
-f(.y)
(5.36)
.
I< Jn(.y)
Y if dx,y)
13s,
<
.
(5.37)
C's, which establishes that Cs is complete. w
Notice hpw this property holds by virtue of the unifonn metric. It is easy to devise sequencej of continuous functions converging to discontinuous limits, but none of these are Cauchy sequences. lt is not possible for a continuous function to be arbitrarily close to a discontinuous function at every point of the domain. A number of the results to follow call for us to exhibit a continuous function which lies. uniformly close to a function in Us, but is fully specified by a finite collection of numbers. This is possible when the domain is totally bounded.
5.25 Theorem Let (S,J) be a totally bounded metric space. For any f G &s, there existj for any : > 0 a function g e Us, completely specified by points of the such that dvjqj < E. D and rational numbers J1,...,Jm, domain .x1,...,au
We specify rational numbers here, because this will allow ud to assert in applications that the set of a11possible g. is countable.
Proof n By total boundedness of S, R for 8 > 0 a finite ,1) xi, 1et Af = fx: dx,xi) k 26) and Bi (x: dxnxi) K gi: S r- 0, 1J by '
(x1,...,au)For
-net
=
gix)
dxli) =
dxAij
+ dlX,
s i ),
each and detine functions .
,
(5.38)
where dxA) = infysxltmy). dxl) is a uniformly continuous function of by constmction, and gilx) is also uniformly continuous, for the denominator is never .:
Metric Spaces
less than
).
89
Then define
'bl '4x) '7 -lp/xl7f
=
(5.39)
.
-1:/x)
Being a weighted average of the numbers (JfJ,gx) is bounded. Also, since (xflis for S, there exists for every x e S some i such that dxli) k as well a ZT=Ip(x) and < dx,Bi) dx,xi) Therefore, f k hence that gix) such as and unifonn continuity extends from the gi to g. For arbitrary f e &s, fix E > 0 and choose small enough that Ifx) fyl l < ls when #(x,y) < 26, for any x,y e G. Then tix m large enough and choose xf and ai Jfl for each i. 1,...,-, so that the Sxinn) cover S, and lfx < for i Note that if dxtx k 28 then e Aj and gix) = 0, so that in a1l cases -net
,
,
.
-
=
Y
-
-x
gixtlfx) fx I f
.ftxls.
(5.40)
-
Hence
gix)Ifxt
-
ai
I<
lfx) f
gix)
x
-
I+
Ifx
gix)
-
ai I
< gixtz for each x
6
(5.41)
S and each i. We may conclude that
duf,gl
=
sup Ifx)
-:(x)1
x6 5
S sup
ZT-1p(x)I fx)
Jfl
-
<
Z:'-1'/x)
xss
z.
(5.42)
.
The next result makes use of this approximation theorem, and is fundamental. It tells us (recallingthe earlier discussion of separability) that spaces of continuous functions are not such alien objects from an analytic point of view as they might at first appear, at least when the domain is totally bounded.
5.26 Theorem (i) If (S,#) is totally bounded then (&s,Jg) is separable. (ii) If (S,#) is compact then cn,dulis separable. Proof We need only prove part (i), since for part (ii), Cs Us by 5.21 and the same conclusion follows. Fix m and suitable points (xl,...,aulof S so as to define a countable family of functionsAm = fgmk,k e IN), whUre the g.k are defined as in 5.25, and the index k of rationals. For enumerates the countable collectibn of lrvectofs (J1,...,tu) > 0, there exists m large enppgh yhqt,for each f e &s, duf, gm < : for : each 5 ij, tits'ftibi, apd there exists gk e A such that somek. By 1. A limp-xA?nTh@'! 'd.ttlh,gtA 0. : is dense in &s. . < ,, dvy, gk) : for every > yytlyyttttt. yyty d1:::,i-....t).',t)'tt ..jj3i,1.. ,).() !j,, yj,jsy, EIl..jjojoug uyytsy mox geueraj jyy ..... ., To show that we canno :lrr-.d.pir,... j));;.,j);)q.;j...;);)... ;;)))..;....-..; .,.434... yy...;jygr-.j-y--.. .. )j-#).;-y--t..r.-,)g. y)-... yty. ...t-y.y-.)(tyy.y,yy. =
'
.
,
'.
.
=
'
..
.
.
..y,,y.
.)
'ril,.
'
.
' 'ttyljk
,
,,
'tij.'yjy,,yt.#,y,
--.i1friil-.
.1r:. .)ky,yy..yyy-. ..... r;-.)t);t@ -jyyt-r),t-.y.ytyj-,.-.
,typtl-l-y-y-)..
.
... . .
........... ..
-..y
.
-.
.
..
Mathematics
90
function space.
circumstances, we exhibit a nonseparable
5.27 Example For S (tz,:J,an interval of the real line, consider the metric space Dka,bndu) of real, bounded cadlag functions of a real variable. Cadlag is gauche) to describe droite, limites a colourful French acronym continue functions of a real variable which may have discontinuities, but are right continuous at every point, with the image of every decreasing sequence in Ltz,:)containing its limit point; in other words, there is a limit point to the left of every point. Of course, Cp,s) c Dp,x. Functions with completely arbitrary discontinuities form a larger class still, but one that for most purposes is too unstructured to permit a useful theory. =
To show that Dfa,bjsdu) is not separable, conslder the subset with elements
J/f)
0, t
<
0
=
1, t k 0
,
0
(5.43)
EJ,1.
6
This set is uncountable, containing as many elements as there are points in gtz,hJ. But duh,h 1 for 0 # 0', so it is also discrete. Hence Dta,bj,du) is not separable by 5.9. (n =
Let A denote a collection of functions f : (S,#) F-y (T,p). A is said to be equicontinuous at x G S if Y 6: > 0 3 > 0 such that
sup .feA y
sup pfyt,fxj) G
Sx,&)
,4 is also said to be unformly
supa:ssup
feA
s
(5.44)
< :.
if Y 6: > 0 3 8
equicontinuous
sup pfyl,fxl)
>
0 such that
(5.45)
< :.
yc Saxnnl
Equicontinuity is the property of a set of continuous functions (or uniformly continuous functions, as the case may be) which forbids limit points of the set to Cs (&s) but A is not be not (uniformly)continuous. ln the case when (uniformly) equicontinuous, we cannot rule out the possibility that W Cs (Us)k An important class of applications is to countable A, and if we restrict attention to the case A = ffn, n e IN) A c Cs torUs) may not be essential. lf we are willing to tolerate discontinuity in at most a finite number of the cases, the following concept is the relevant one. A sequence of functions f/, n e IN) will be said to be asymptotically equicontinuous at if Y E > 0 3 > 0 such that :4
,
.x
limsup l-yx
and asymptotically
sup
ye Sdxszj
?zzif/brzlfy
p(A(y),Jn(x)) < e,,
equicontinuous if V e, > 0 R
(5 46) >
0 such that
Metric Spaces
limsup sup n--:x
;r
e
sup
91
p(Jn@),n(x)) <
(5.47)
E.
Jx,&)
ye
If the functions fn are continuous for a11n, limsupn-yx can be replaced by supn in (5.46) and similarly for (5.47)when all the fn are uniformly continuous. ln these circumstances, the qualifier asymptotic can be dropped. The main result on equicontinuous sets is the Arzel-Ascoli theorem. This designation covers a number of closely related results, but the following version, which is the one appropriate to our subsequent needs, identifies equicontinuity as the property of a set of bounded real-valued functions on a totally bounded domain which converts boundedness into total boundedness.
5.28 Arzel-Ascoli theorem Let (S,#) be a totally bounded metric space. A set A c Cs is relatively compact under dv iff it is bounded and uniformly equicontinuous. Proof Since Cs is complete, total boundedness of A is equivalent to relative compactness by 5.13. So to prove if' we assume boundedness and equicontinuity, and construct a tinite E-net for A. It is convenient to define the modulus of continuity of f, that is, the function w: C xR+ h- R+ where ,
s
w(T,) Fix s
>
0, and choose 8
=
sup a.s s
(as is
If(.y4 J@)I
sup
.
ysdx,)
equicontinuity)
possible by unifonn
sup w(.f,)
(5.48)
-
such that
(5.49)
< :.
ysA
Boundedness of A under the unifonn metric means that there exist finite real numbers U and L such that inf
L f
/'lA.xe
be (x1,...,xm)
Let
S
#4x)f
sup DA,XC
U.
(5.50)
5
for S, and construct the finite family
-net
a Dm
fx) S
=
l'k
A, k
q
=
1,...,(v + 1)-)
(5.51)
according to the recipe of 5.25, with the constants ai taken from the finite set
where u and v are integers with v exceeding CU L4Ie and u 0,...,v. This set contains v + 1 real values between U and L which are less than 6: apal't, so that Dm has (v+ 1)= members, as indicated. Since the assumptions imply
fL
+
(&- Ljulvj
-
,
=
4 c &s, it follows by 5.25 that for every f E A there exists gk e Dm with < :. This shows that Dm is a e-net for A, and A is totally bounded. dvf,g ito suppose A is relatively compact, and hence totally bounded. To prove Trivially, total boundedness implies boundedness, and it remains to show unifonn equ icontinuity. Consider for E > 0 the set ionly
,
'
Mathematics
#k(:)
(/': w(/',1/k)
=
< :).
(5.52)
Uniform equicontinuity of A is the condition that, for any
large enough that
X'c
Bkz). lt is easily veritied
lw(.f,)
-
w(',*
6: >
0, there exists k
that
I f 2JrXJ,'),
(5.53)
R*,dE)
F--> is continuous. Bkz) is the inso that the function w(.,): (Cs,d verse image under w(.,8) of the half-line (0,:) which is open in R+, and hence
x 0 as k for each f e Bkzj is open by 5.19. By definition of Cs, w(/', 1/k) Cs In other words, w converges to 0 pointwise on Cs which implies that the Cs, and hence forW.But b.collection (#k(:),k e IN) must be an open coverinAfor covering is such of has A X hypothesis compact, every a finite subcover, and so A q Bke) for finite k, as required. . -->
--
.
,
6 Topology
6. 1 Topological Spaces Metric spaces fonn a subclass of a larger class of mathematical objects called topological spaces. These do not have a distance defined upon them, but the concepts of open set, neighbourhood, and continuous mapping are still well defined. Even though only metric spaces are encountered in the sequel (Part VI), much of the reasoning is essentially topological in character. An appreciation of the topological underpinnings is essential for getting to grips with the theory of
weak convergence. 6.1 Definition A topological space (X,I) is a set X on which is defined a topology, a class of subsets called open sets having the following properties: (a) X e T, O e 1. (b) If C i T, then U()scoe T. Oa G T, then 01 f-'h 02 q T. n (c) If 01 G 'r
'r,
These three conditions dehne an open set, so that openness becomes a primitive concept of which the notion of e-spheres around points is only one characterization. A metric induces a topology on a space because it is one way (thoughnot the only way) of detining what an open set is, and a1l metlic spaces are also topological spaces. On the other hand, some topological spaces may be made into metric spaces by defining a metric on them under which sets of are open in the sense defined in j5.1. Such spaces are called metrizable. A subset of a topological space (X,I) has a topology naturally induced on it by (A f''h 0: O G 'IJ is called the the parent space. If A c X, the collection relative topology for A. (A,n) would normally be referred to as a subspace of X. and If two topologies are defined on a space and :1 c T2, then T1 is said to be coarser, or weaker, than T2, whereas 12 is hner stronger) than I1. In particular, the power set of X is a topology, called the discrete topology, whereas (?,x J is called the trivial topology. Two metrics define the same topology on a space if and only if they are equivalent. If two points are close in one space, their images in the other space must be correspondingly close. If a set O is open, its complement Oc on X is said to be closed. The closure X' of an arbitrary set A ? X is the intersection of a1l the closed sets containing A. As for metric spaces, a set A i #, for B c X, is said to be dense in B if B c X. ':
'u
':1
=
.:2
6.2 Theorem The intersection of any collection of closed sets is closed. X and O are both open and closed. n
Mathematics
94
However, an arbitral'y union of closed sets need not be closed, just as an arbitrary intersection of open sets need not be open. For given e X, a collection Vx of open sets is called a base for the point x if for every open O containing x there is a set B e Vxsuch that x e B and B c 0. This is the generalization to topological spaces of the idea of a system of neighbourhoods or spheres in a metric space. A basefor the topology T on X is a and every x 6 0, there exists B e collection V of sets such that, for every O 6 that V such x e B c 0. The definition implies that any open set can be expressed union the of sets from the base of the topology; a topology may be defined for as specifying a base collection, and letting the open sets be defined as a space by the unions and finite intersections of the base sets. ln the case of R, for example, the open intervals form a base. .x
,r,
6.3 Theorem A collection V is a base for a topology I on X iff (a) Ulevf = X. (b) V #1,#2 e F and x e #1 chB1. 3 B5 e V such that a' e #3 c #1 /n %. Proof Necessity of these conditions follows from the definitions of base and open in terms of the base F, as follows: set. For sufficiency, define a collection 'r
O e
'r
iff, for each
.'&'
e 0, H B e F such that x
q
B c 0.
O s4tisfies the condition in (6.1), and X satisfies it given condition (a) of the theorem. lf C is a collection of T-sets, UtxwO e T since (6.1) holds in this case in respect of a base set B corresponding to any set in C which contains x. And if 01,02 G 1, and x e 01 fa O1. then, letting #1 and B1 be the base sets specified in (6.1) in respect of x and 01 and 01 respectively, condition (b) implies that x e B5 c 01 f''h O1, w ltic h shows that I is closed under finite intersections. Hence, is a topology for X. . ':
The concept of base sets allows us to generalize two further notions familiar points) of a set A in a from metric spaces. The closure points (accumulation topological space (X,'r) are the points x e X such that every set in yhe base of a7 contains a point of A (a point of A other than x). An important exercise is to show that x is a closure point of A if and only if is in the closure of z4. We have generalizations of two other familiar concepts. A sequence tak) of points in a topological space is said to converge to x if, for every open set O containing x, H N k 1 such that ak e O for all n k N. And is called a cluster point of (.:kl if, for every open O containing and every N k 1, xn e O for some n k N. ln general topological spaces the notion of a convergent sequence is inadequate for characterizing basic properties such as the continuity of mappings, and is augmented by the concepts of net and///er. Because we deal mainly with metric spaces, we do not require these extensions (seee.g. Willard 1970: Ch. 4). .x
.x
.x
6.2 Countability and Compactness The countability tutppu provide one classitkation of topological spaces according, roughly speaking, to their degree of structure and amenability to the methods
Topology
95
of analysis. A topological space is said to satisfy the first axiom of countability (to be jrst-countable) if every point of the space has a countable base. It if the space as a satisfies the second axiom of countability (is second-countable) whole has a countable base. Every metric space is first-countable in view of the existence of the countable base composed of open spheres, Sx, 1/n) for each x.
More generally, sequences in first-countable spaces tend to behave in a similar manner to those in metric spaces, as the following theorem illustrates.
6.4 Theorem ln a first-countable space, x is a cluster point of a sequence (.v, n e IN) iff there is a subsequence (xa#,k s IN) converging to x. Proof Sufficiency is immediate. For necessity, the definition of a cluster point implies that 3 n k N such that xn e 0, for every open 0 containing x and every N k 1. Let the countable base of be the collection Vx fBi, i G (11), and choose a monotone sequence of base sets (Ak, k G INl containing x (andhence nonempty) with A1 #l, and Ak c Ak-l f''h Bk for k 2,3,...; this is always possible by 6.3. Since x is a cluster point, we may construct an infinite subsequence by taking xnk as th: next member of the sequence contained in 4s for k = 1,2,... For every open set O containing x, R N k 1 such that xnk E Ak 0, for a1l k k N, and hence xnk x, as required. w --> x as k .x
=
=
=
.-..:
The point of quoting a result such as this has less to do with demonstrating a new property than with reminding us of the need for caution in assuming properties we take for granted in metric spaces. While the intuition derived from R-like situations might lead us to suppose that the existence of a cluster point and a convergent subsequence amount to the same thing, this need not be true unless we can establish first-countability. A topological space is said to be separable if it contains a countable dense subset. Second-countable spaces are separable. This fact follows directly on taking a point from each set in a countable base, and verifying that these points are dense in the space. The converse is not generally true, but it is true for metric spaces, where separability, second countability and the Lindelf property (that every open cover of X has a countable subcover) are all equivalent to one another. This is just what we showed in 5.6. More generally, we can say the
following.
6.5 Theorem A
jecond-countable
space is both separable
and Lindelf.
Proof The proof of separability is in the text above. To prove the Lindelf property, let C be an open cover of X, such that UzeCA X.For eachA e C and e A, we can find a base set Bi such thgt x 6 Sj c A. Since UT=1#fX, we may choose a countable subcollection Af, i 1,2,... such that Bi c 4f for each i, and hence U7=1Xf X. K .x
=
=
=
=
A topological space is said to be compact if every covering of the space by open countably compact if each countable sets has a finite subcover. It is said to e coveringhas a finite subcovering. And it is said to be sequentially compact if subsequpce. Somesimes, compacteach sequence on the space has a convrgnt
96
Mathematics
ness is more conveniently characterized in tenns of the complements. The complements of an open cover of X are a collection of closed sets whose intersection is empty', if and only if X is compact, every such collection must have a finite subcollection with empty intersection. An equivalent way to state this proposition is in terms of the converse implication. A collection of closed sets is said to have the Tnite intersection property if no finite subcollection has an empty intersection. Thus: 6.6 Theorem X is compact (countablycompact) if and only if no collection (countable collection) of closed sets having the finite intersection property has an empty intersection. a The following pair of theorems summarize important relationships different varieties of compactness.
between the
6.7 Theorem A first-countable space X is countably compact iff it is sequentially compact. Proof Let the space be countably compact. Let fxn,n G EN ) be a sequence in X, and detine the sets Bn fxn,ak+1,... ) n 1,2,... The collection of closed sets (Tn, n e (N) clearly possesses the finite intersection property, and hence (X'n is nonempty by 6.6, which is another way of saying that fak) has a cluster point. Since the sequence is arbitrary, sequential compactness follows by 6.4. This proves necessity. For sufficiency, 6.4 implies that under sequential compactness, al1 sequences in X have a cluster point. Let ( Ci, i e INl be a countable collection of closed sets having the finite intersection property such that A,y 0Z=IC/ # 0, for every finite n. Consider a sequence fxn) chosen such that xn e A,,, and note since (A,, ) is monotone that xn e Am for all n 2 m; or in other words, Am contains the /?z). Since (ak) has a cluster point x and Amis closed, x e Am. sequence fak,n k This is true for every m e N so that OT=1Q is nonempty, and X is countably 6.6. by compact w =
,
=
=
,
6.8 Theorem A metl'ic space (S,#) is countably Proof
compact iff it is compact.
Sufficiency is immediate. For necessity, we show first that if S is countably compact, it is separable. A metric space is first-countable, hence countable compactness implies sequential compactness (6.7),which in turn implies that every sequence in S has a cluster point (6.4).This must mean that for any 6: > 0 there exists a finite E-net (m,...,auJ such that, for a1l x e S, dx,xk) < E, for some k E ( 1,...,-) ; for otherwise, we can construct an infinite sequence fxn) with dxn,xn') k 6: for n # n', contradicting the existence of a cluster point. Thus, for each n e IN there is a finite collection of points 4,, such that, for is countable and every x e S, #@,y) < 2-n for some y e An. The set D = U*n=1An dense in S, and S is separable. Separability in a metric space is equivalent by 5.6 to the Lindelf property, that every open cover of S has a countable subcover', but countable compactness implies that this countable subcover has a finite subcover in its turn, so that
Topology
compactness is proved.
97
.
Like separability and compactness, the notion of a continuous mapping may be defined in tenns of a distance measure, but is really topological in character. In the mapping /': X F--y f is said to be contina pair of topological spaces X and uous if J -1(#) is open in X when B is open in and closed in X when B is closed in That in metric spaces this definition is equivalent to the more familiar one in terms of s- and 8-neighbourhoodsfollows from 5.19. The concepts of homeomorphism and embedding, as mappings that are respectively onto or into, and 1-1 continuous with continuous inverse, remain well defined. The following theorem gives two important properties of continuous maps. ,
,
.
6.9 Theorem Suppose there exists a continuous space X onto another space (i) If X is separable, T is separable. is compact. (ii) If X is compact,
mapping
f
from a topological
.
problem is to exhibit a countable, dense subset of Consider .f(D4 where D is dense in X. lf J(D) is the closure of J(D), the inverse image f-JfD is closed by continuity of f, and contains f -1(/'(D)), and hence also coptains D by 1.2(iv). Since X'is the smallest closed set containing D, and X c 5, /X) it follows that X c J-1C(D)). But since the mapping is onto, f ff-1(fD))4 c J(D), where the last inclusion is by 1.2(v). J(D) is therefore ' in dense as required. J(D) is countable if D is countable, and the conclusion follows directly. (ii) Let C be an open cover of Then (J -1(#): B e C) must be an open cover of X by the definition. The compactness of X means that it contains a finite subf-1B n ) such that cover, say f -1(B l ) Proof
(i) The
.
=
.
,...,
y'
=
y(x) y =
Uy-'tsp= J y-1 tjsy c /=1 Usy,
y=1
(6.2)
j=L
where the third equality uses 1.2(ii) and the inclusion, 1.2(v). Hence C contains a Gnite svbcover. . in both these results. The extension Note the importance of the stijulation and f is obvious, and can be supplied of (ii) to the case of compact subsets of X reader. the by Completeness, unlike separability, compactness, and continuity, is not a topological property. To define a Cauchy sequence it is necessary to have the concept of a distance between points. One of the advantages of defining a metric on a space is that the relatively weak notion of completeness provides some of the essential features of compactness in a wider class than the compact spaces.
6.3 Separation Properties Another classification of topological spaces is provided by the separation tztppzu, whichin one sense are more primitive than the countabilityaxioms. They are
Mathematics
98
indicators of the richness of a topology, in the sense of our ability to distinguish between different points of the space. From one point of view, they could be said to define a hierarchy of resemblances between topological spaces and metric spaces. Don't confuse separation with separability, which is a different concept altogether. A topological space X is said to be: with x # y 3 an open set containing x but not y and - a Fl-space, iff V x,y e X also an open set containing y but not Fa-) space, iff V x,y e X with x # y 3 disjoint open sets 01 and - a Hausdotffo 01 in X with e 01 and y E O2; regular iff C Hdisjoint open sets 01 and for each closed set C and space - a 02 with x e 01 and C c O2; - a normal space iff, given disjoint closed sets C1 and Cz, 3 disjoint open sets 01 and 01 such that C1 c 01 and (72 c O2. .x;
.x
.x
A regular Fl-space is called a Fg-space, and a nonnal Fl-space is called a F4-space. In a Fl-space, the singleton sets (.:) are always closed. In this case y e (xJf (xJC is the complement of a closed set, and hence open. whenever y # x, where Conversely, if the F1 property holds, evel'y y # x is contained in an open set not containing x, and the union of all these sets, alsp open by 6.1(b), is (x)f. It is easy to see that F4 implies Tz implies Fa implies T1, although the reverse implications do not hold, and without the F1 property, nonnality need not imply regularity. Metric spaces are always F4, for there is no difficulty in constructing the sets specified in the definition out of unions of E-spheres. We have the following important links btween separation, compactness, countability, and metrizability. The first two results are of general interest but will not be exploited directly in this book, so we forgo the proofs. The proof of 6.12 needs some as yet undefined concepts, and is postponed to j6.6 below.
6.10 Theorem A regular Lindelf
space is normal. a
6.11 Theorem A compact Hausdorff space is F4. 6.12 Urysohn's able. u
metrizatlon
(a
theorem A second-countable
Fg-space is metriz-
In fact, the conditions of the last theorem can be weakened, with F4 replaced by Fa in view of 6.10, since we have already shown that a second-countable space is Lindelf (6.5). The properties of functions from X to the real line play an important role in defining the separation properties of a space. The key to these results is the remarkable Urysohn's lemma.
6.13 Urysohn's Iemma A topological space X is nonnal iff for any pair A and B of disjoint closed subsets there exists a continuous function f: X (0,1j such = 0 and J(#) 1. :l that jA) .-+
=
The function
J is
called a separating
function.
Topology
99
Proof This is by construction of the required function. Recall that the dyadic rationals D are dense in (0,1j. We demonstrate the existence of a system of open sets f Ur, r e D l with the properties A c &r;
Vrfa
#
'X c
0',
=
(6.4)
Ur for r
>
(6.5)
s.
Nonnality implies the existence of an open set 171/2(say)such that 171/2contains :4 an d (&1/2)ccontains B. The same story can be told with &f/2 replacing B in the role of C1 to define 171/4,and then again with 171/2replacing A in the role of C1 to define Um. The argument extends by induction to generate sets ( Umnn, m 1,...,2* 1 ) for any n e EN,and the collection ( Ur, r e D ) is obtained on letting n --A x. lt is easy to verify conditions (6.3)-(6.5)for this collection. Fig. 6.l illustrates the construction for n 3 when A and B are regions of the plane. One of the onion' in the limiting case. must imagine countably many more =
-
=
)1
!/3/8
315ls
:/1/2
!/3/4 Llzls
:/1/4
U 1/8
B
,4
Fig. 6.1 Now define
f:
X
(0,11by
--)
inffr
e D: x e &r) x e ,
Urss Ur
x Urso Ur
(6.6)
-
1 for x e #. Because of the monotone property (6.5), where, in particular, we have f:r any a e (0,1) .f(x)
=
(x: fx)
<
al
(x:inf fr e D: x e = (x:H r < a such that =
&r, = r
<
a)
x e &r)
(6.7)
which is open. On the other hand, because D is dense in g0,1qwe can deduce that,
Mathematics
100
for any k$e
0, 1),
(x: flx)
<
I31 (x:inf tr e =
D: x e &rl <
= (x: x e Ur V r
>
jlJ
))
Ur O rn.js
=
=
r)
(6.8)
r
r>f$
which is closed. Here, the final equality must hold to reconcile the following two and second that, for al1 r > I3,there exists (sinceD facts: first that Ur i and Us Ur by (6.5).We have therefore shown that, > > dense) D with is se r s k$ c for 0 S ;$< a ; 1, ,
(x: j
<
fx)
<
a)
(.x:flx) <
=
a)
fa
fx: flx) K
;'$)C
(6.9)
is open, being the intersection of open sets. Since every open set of (0,11is a union of open intervals (see2.5), it follows that f -1(A) is open in X whenever A is open in g0,11,and accordingly f is continuous. It is immediate that f(A4 0 and f(#) 1 as required, and necessity is proved. Sufficiency is simply a matter, given the existence of f with the indicated and J-1((11j), whose images are PrOPerties of citing the two sets J-1(g0,1)) a a, and respectively, in X, disjoint, and contain B so that X is normal. w open =
=
,
,4
It is delightful the way this theorem conjures a continuous function out of thin air! It shows that the properties of real-valued functions provide a legitimate means of classifying the separation properties of the space. In metric spaces, separating functions are obtained by a simple direct construction. If 4 and B are closed and disjoint subsets of a metric space (S,#), the nonnality property implies the existence of > 0 such that infxsz,yswJtmy) k 6. The required function is
fx)
=
Jtx,Al dxv)
+ ty(x,s)
(6. 10)
where dxl) = infysxttx,y), and J(x,S) is defined similarly. The continuity of f follows since dxl) and #(x,#) are continuous in x, and the denominator in (6.10) A similar construction was used in the proof of 5.25. is bounded below by The regularity property can be strengthened by reqtliring the existence of separating functions for closed sets C and points x. A topological space X is said to be completely regular if, for a1l closed C c X and points x C, 3 a continuous 0 and flx) = 1. A completely regular Fl-space function fl X 1- E0,1)with fc' is called a Tychonoff space or Fgj-space. As the tongue-in-cheek terminology suggests, a L-space is F3j (this is immediate from Urysohn's lemma) and a F3j-space is clearly F3, although the reverse implications do not hold. Being F4, metric spaces are always F?j. .
=
Topology
101
6.4 Weak Topologies Now let's go the other way and, instead of using a topology to define a class of real functions, use a class of functions to define a topology. Let X be a space and F a class of functions f : X i-- y, where the codomains y are topological under which spaces. The weak topology induced by F on X is the weakest topology -1 is continuous. Recall, continuity F (A) is means that J open in X every f e whenever A is open in y. We can also call the weak topology the topology generated by the base of sets V consisting of the inverse images of the open sets of the y, under f e F, together with the finite intersections of these sets. The inverse images themselves are called a sub-base for the topology, meaning that the sets of the topology can be generated from them by operations of union and finite
intersection.
increase the number of sets in this base and If we enlarge F we (potentially) get a stronger topology, and if we contract F we likewise get a weaker topology. With a given F, any topology stronger than the weak topology contains a richer collection of open sets, so the elements of F must retain their continuity in this case, but weakening the topology further must by definition force some fq F to be
discontinuous.
The class of cases in which y R for each f suggests using the concept of a weak topology to investigate the structure of a space. One way to represent the richness of a given topology on X is to ask whether contains, or is contained in, the weak topology generated by a particular collection of bounded real-valued functions on X. For example, complete regularity is the minimal condition which makes the sol4 of construction in 6.13 feasible. According to the next result, this is sufficient to allow the topology to be completely characterized in terms of bounded, continuous real-valued functions on the space. =
':
'r
6.14 Theorem lf a topological space (X,':) is completely regular, the topology is the weak topology induced by the set F of the separating functions. ':
Proof Let F denote the collection of inverse images of open sets under the funcr tions of F. And 1et F denote the weak topology induced by F, such that the F-sets, together with their finite intersections, fonn a base for F. We show that F T. For any x e X, let O e be an open set containing x. Then Oc is closed, and by l and complete regularity there exists f e F taking values in (0,1j with J(I) = J(O9 J-1((, 1q) is therefore an is open in g0,1j, and B 0. The set (1z,1) with Oc containing disjoint and that B c 0. Since this holds for x so open set, every such 0, x has a base Vx consisting of inverse images of open sets under functions from F Since x is arbitrary the collection V = ( Vx,x e XJ forms a base for T. lt follows that T ? F. On the other hand, F is by definition the weakest topology under which every f E F is continuous. Since f E F is a separating function and continuous under it also follows that F c 1. . =
'r
=
=
.
'r,
Mathematics
6.5 The Topology of Product Spaces Let X and f be a pair of topological spaces, and consider the product space X x The plane R x R and subsets thereof are the natural examples for appreciating the properties of product spaces, although it is a useful exercise to think up more exotic cases. An example always given in textbooks of topology is C XC where C is the unit circle; this space has the topology of the torus (doughnut). Let the ordered pair (x,y)be the generic element of X x f. The coordinate defined by projections are the mappings 7:x: X x F-) X and z:v: X x f F-> .
,
lx(.Y,')
=
A'
(6.11)
1:(-Y,)/)
=
y.
(6.12)
are topological spaces, the coordinate projections can be used to generate a new topology on the product space. The product topology on X x is the weak topology induced by the coordinate projections. The underlying idea here is very simple. lf A i X and B q f are open sets, the 1 1(#) will be lk - (z4)an d X x B = 1:yset A x B (A x f) rn (X x #), where A x f rectangle called of X The product regarded as open in X x and is x f. an open topology on X x is the one having the open rectangles as a base. This means that and @2,y2) two points @1,y1) are close in X x provided xl is close to x2 in X, antl yl to y2 in f. Equivalently, it is the weakest topology under which the coordinate projections are continuous. If the factors are metric spaces (X,#x) and f,h), several metrics can be constructed to induce the product topology on X x including If X and
=
=
,
,
,
@2,y2)) maxllxtxl,xzl, p((x1,y1), =
Jr(A'1,y2)l
(6.13)
and p'(@1,A'1), @2,y2)) =
#x@1sx2)+#:/1,y2).
An open sphere in the space (X x p), where p is the metric in happens to be an open rectangle, for ,
5'p(@,A'),)
=
Sdsxnnt
X Sdj,vnt',
(6.14) (6.13),
also
(6.15)
but of course, this is not true for every metric. Since either X or may be a product space, the generalization of these results from two to any finite number of factors is straightforward. The generic element of the space X2=1Xfis the n-tuple (x1,...,ak: xi e Xf), and so on. But to deal with infinite collections of factor spaces, as we shall wish to do below, it is necessary to approach the product from a slightly different viewpoint. Let 4 denote an arbitrary index se and fXa, (x l A ) a collection of spaces indexed by is the collection of a1l the mappings x: 4 4. The Cartesian product X Xasxxa F-y Uaszxa such that x(a) e Xa for each (x e A. This definition contains that given in j1.1 as a special case, but is fundamentally more general in character. The coordinate projections are the mappings 1a: Xn, F-> Xa with ,t
=
Topology
103
l(x(.%) Xq),
(6.16)
=
but can also be defined as the images under x of the points a e A. Thus, a point in the product space is a mapping, the one which generates the coordinate projections when it is evaluated at points of the domain z4. In the case of a finite product, A can be the integers 1,...,l. In a countable case A EN,or a set equipotent with EN,and we should call an infinite sequence, an element of =
.x
X'='(say).A familiar uncountable example is provided by a class of real-valued functions x: R F-> R, so that A R. In this case, x associates each point a e R R with a real number x(a), and defines an element of the product R The product topology is now generalized as follows. Let (Xa, a e A ) be an arbitrary collection of topological spaces. The Tychonoff topology (product X topology) on the space X h as as base the finite-dimensional open rectangles, sets where the Oa ? Xa are open sets and Os Xa except for at of the form Xas.4otx, most a finite number of coordinates. These basic sets can be written as the interkections of tinite collections of cylinders, say =
.
=
B
-1 aajtoaj)
=
-1
f-h
chzramto oo ;
...
,
o j'y; .
for indices a1,...,a,a e A. Let be a topology under which the coordinate projections are continuous. If Oz and hence contains the Tychonoff topology. Since is open in Xa, aa-1(O(: ) e this is true for any such 1, we can characterize the Tychonoff topology as the weak topology generated by the coordinate projections. The sets C:a-1(Oa) fonn the sub-base for the topology, whose hnite intersections yield the base sets. Something to keep in mind in these intinite product spaces is that, if any of X the sets Xct are empty, X i s emp ty. Some of our results are true only for nonempty spaces, so for full rigour the stipulation that elements exist is desirable. 'r
'r
':
,
6.15 Example The space hdv) examined in j5.5 is an uncountable product space having the Tychonoff topology; the unifonn metric is the generalization of the maximum metric p of (6.13).Continuous functions are regarded as close to one another upder dv only if they are close at every point of the domain. The subsequent usefulness of this characterization of C,du) stems mainly from the fact that the coordinate projections are known to be continuous. n Thetwoessentialtheorems onproductspacesextendseparability andcompactness from the factor spaces to the product. The following theorem has a generalization to uncountable products, which we shall not pursue since this is harder to prove, and the countable case is sufticient for our purposes.
6.16 Theorem Finite or countable product spaces are separable under the product topology iff the factor spaces are separable. Proof The proof for tinite products is an easy implication of the countable case, hence consider X* XT=fXf.Let Df fdikndizn... l c Xf be a countable dense set by D each and construct a set i, detining for c =
=
X''O
104
Mathematics
Fm
=
Xojx
i=m+ 1
f=l
fl l
(6. 18)
for m = 1,2,..., and then letting D U;=IF-. Fm is equipotent with the set of m-tuples formed from the elements of the countable DI ,...,Dm, and is countable by induction from 1.4. Hence D is countable, as a countable union of countable sets. We will show that D is dense in X=. Let B = X7=IO/ be a non-empty basic set, with Oi open in niand Of niexcept for a finite number of coordinates. Choose m such that Oi Xf fOr i > m, and then =
=
=
m
B ra Fm
=
=
x X ftl X((?/rno/) f=rn+1
=1
)
y,
?.
(6.19)
l,...,m. Since recalling that the dense property implies Oi fa Di # 0, for B fa Fm B ?'7 D, it follows that B contains a point of D; and since B is an arbitrary basic set, D is dense in X7 as required. w =
One of the most powerful and important results in topology is Tychonoff's theorem, which states that arbitrary products of compact topological spaces are also compact, under the product topology. It will suftice here to prove the result for countable products of metric spaces, and this case can be dealt with using a more elementary and familiar line of argument. lt is not necessary to specify the metrics involved, for we need the spaces to be metric solely to exploit the equivalence of compactness and sequential compactness.
6.17 Theorem A finite or countable product of separable metric spaces (Xf,) is compact under the product topology iff the factor spaces are compact. Proof As before, the finite case follows easily from the countable case, so assume X= = X:=1Xj, where the Xf are separable spaces. In a metric space, which is first countable, compactness implies seprability and is equivalent to sequential compactness by 6.8 and 6.7. Since Di is sequentially compact and tirst-countable, every sequence (xfa,n e INl on Xf has a cluster point xi on the space (6.4)k Applying the diagonal argument of 2.36, there exists a single subsequence of integers, (nk,k E l ), such that xink xi, for every i. Consider the subsequence xw,...). In the product topology, xy in X*, lau, k s N ) where xng @1n:, which proves that X* is sequentlally xo...) iff for i, every xink xi .x (.x1, compact. X= can be endowed with the metric px Z7wl4/zf,which induces the product topology. X= is separable by 6.16, and sequential compactness is equivalent to compactness by 6.8 and 6.7, as above. This proves sufficiency. Necessity follows fzom 6.9(ii), by continuity of the projections as before. w -..-9
=
=
--
--
=
6.18 Example The space R= (see5.14) is endowed with the Tychonoff topology if we take as the base sets of a point the collection .x
N(x.,k,
(y: la)
-yjl
=
< :,
=
1,...,k1., k e N, er > 0.
(6.20)
A point in R* is close to a7 in this topology if many of its coorclinates are close
Topology
to those of x; another point is closer if either more coordinates are within 6: of each other, or the same coordinates are closer than e, or both. The metric Jx defined in (5.15)induces the topology of (6.20).If (xn) is a sequence in X, dxlxnvx) 0 iff V z,k 3 N 1 such that xn e N(x;k,e) for a1l n k N. We already know that R= is separable under Jx (5.15),but now we can deduce this as a purely topological property, since R* inherits separability from R by 6.16. (:1 -->
The intinite cube (0,1j=shares the topology (6.20)with R= and is a compact space by 6.17; to show this we can assign the Euclidean metric to the factor spaces (0,1j. The trick of metrizing a space to establish a topological property is frequently useful, and is one we shall exploit again below.
6.6 Embedding and Metrization Let X be a topological space, and F a class of functions F-y tion p'lzw :: X/rfyis the mapping defined by
exjf
y. The evalua-
J: X F->
xt.
=
(6.21)
The class F may be quite general, but if it were finite we would think of d(x) as the vector whose elements are the J(x),J e F (6.21)could also be written z:yoe = f where ay is the coordinate projection. A minor complication arises because f need not be onto y, and :(X) c Swy is possible. If 4 is a set of points in ly -1(4) may contain points not in e(X). We therefore y, the inverse projection need to express the inverse image of A under f, in tenns of c. as .
:-1(ay-1(4)
f -144) (xyo:)-1(4) =
=
f...j c(x))
c
y.
(6.2:)
The importance of this concept stems from the fact that, under the right conditions, the evaluation map embeds X in the product space generayed by it. It would be holeomorphic to it in the case c(X) Skyy. =
6.19 Theorem Suppose the class F sparates points of X, meaning that fx) # J@) for some f G F whenever x an d y are distinctpoints of X. If X is endowed with the weak topology induced by F, the evaluation map defines an embedding of X into ly J .
.' ,'--f,,' -' ' :.);' .'...' .'-.' .,'
k' r' ::'i;t' lkd ;'y' j' f'.f..
tjd f'
Proof lt has to be shown that e is a 1-1 mapping from X onto a subset of Xro which is continuous with continuou: inverse. Since F separates points of X, e is 1-1, since c@) # :(.y) w he never flx) #(.f@). for some f e F. To show continuity of -1 X rwheneker A is open in y under the weak e, note first that f (A) is open in topology, and sets of the form ly*(A) i open in X/ywith the product Bltt e (z:y(A)) (ayoc) (A) topology, since the projecons are y yyj ; j lh, under e of sets of the form gs J- (A), so we can conclude that. thte '-' .. ).' .T.'1.). (qgt. .y. .g j(. ..t. ),q:.., yj.).yt) ). tyi,j:lt. . .. . y'yt,jgyy,kytt,jyyyryltysy invrfs unions and intersections t; ay-1(4), 4 c :y, are open. since #.-;y.;.L.L.y ( .r (..eryt yrve y (t. (.'( ). ty t . . se sets of Xfy,which are (see 1.2) the same property extrh.d, kygtk..qjlkyrltlqplttit,tt,i ltjj tjju illwl invfj tuj , product topology, and tjo ) r))t.()),qy:,. jjy y Ljjqj.jlqjtjlllq-kjLb. f-iite intersections of these .. . ..; p cztt' yyyt( j))j;)).jjjy)(4tr.)jk;)r...'...))' yyyjyy, y y ,y...'y#)tyv-y..y,' ,.;;.bitiL.) !!F:1. L.)iLj;;Lb'-j)'3';'bb'' 1t1!4,1(E:,. ,y'').%' ,)'..; .y'' ftt. ,:... jjj7td-jjj,1!2421,:'(E:,..ki:r t::' )8,<-..,s.. ..''.y)1I)k-jyjj)'. &.-,. llr.. 154, 421,: d:::: 1ki::1l; :1Er' .kE,: l.rlijjif..' :El2 #:#*'f,ty.-')j.A.yy .rt;...'.7...,tj;-rtr 1lr1. d:E:,h 11. 11. llrl. .'. (.).,j.y,tty. gllr)p %.. ( g .. . jyyy. 1l; 1 :55;4:2::4:::,,llr11l2417:1. tjjb1'3+. )'t*..111!111k!!1-.. k(!r7qp).... $44)14. 4::2ii.'.......11#)i)... #I!2l-y.. jiii 154) ll:ll.i.1Ir71L 1ki!: 4k12: 4:::,, 11.21. .... 1l: ql::)p );.r'#..'y)yy: . ...k' 1bd;i.lq- 4:y# . p.. L3ffL'. ;EE1g (1(2..-, j:''yt,.;Ayz,t., ii. 11.,1. #.:, ()iiL' - ii. . . ''. ; 'lri
Lkwise
qtjt
,$
'tihuoj.
,
,
y
q
yk
y
,
g
,,k;
.'.
:
..
...'...'j;.ji(...
:.i:t
,, '
.
..)
=
,
tyrtj., .;i
..
=
;
,ytl,ii1.
-.)..
.:.
E...
.
.
..
.
..
.,p,y,
..
..
.'
( jy)t,,.E.j
(II'E .E!' '.j2.t(). .:E. .. . :. ,,/'::t Eyyyhtttyyjt.ttjlyyyyyyyyjyyjrjyl,y.yyyt'jtb,t,,.,) ..
..
.
(..))t
.
.
,
,))
,yttytyytyy;y
,
,y(
.:/E.:,tytlr.-)()jtEj4jjkjp.. .Ei...E -jy ;;y.yjE(j..)y.....y''E,#.
''.-.';'.7ti1ir7.
'''1.j.
..
.
4:
..'
... . ..
,
jj riEjptl-l,
'-('
,-t.k.
....
yja
.ij'
.
-
.
..
...t..
.'....TiTi.tj;)))1t-? -yy.'':;)'r.y(..y)j-.,$y.,',.y,yr.;..,-yj:..,.-p.. .
-.7
-:
...:
-,
.
.
.
E : -:-
-:
. '..
.
-
.: !.
;.
-ki. . . . . . . . .. ..
-.
,.
-,.
E .?).!-
,. ..
. .
.. -
.
-
.
.
Mathematics
106
set of the fomf -1(A) where A is open in y. Since F defines the topology on x we know this set to be open, and the finite intersections of such sets form a base for X by assumption. Since e is 1-1 and e -1 a mapp ing, it will suffice to verify that their images under e are open in e(X). Noting that B is a set of the type shown in (6.22),eB) ay-1(A) r''hd(x), but since ly-1(A) is open, eB) is open in '(x) as required. . ,
=
The following is the (forus)mostimportant case of Urysohn
's
embedding theorem.
6.20 Theorem A second-countable F4-space (X,5) can be embedded in (0,1)*.(:I The proof requires
the sufticiency
part of the following lemma.
6.21 Lemma Let x e X, and 1et O X be any open set containing x. 1ff X is a regular space, there exists an open set U with e c 0. -t7
.x
Proof Let X be regular. If O is open nd x e 0, there exist disjoint open sets U OC CC CC 0. Since & by the disand C such that e U and c C, and hence CC closed, Cc is jo intness and we have V ? OC g O and sufficiency is proved. To necessity, is a closed set not containing x, and suppose x e U and Vc 0. prove OC where U and 'P are disjoint open sets. Hence X is regular. w c .x
,
-t7'
Since the space is T4 it is F? and Proof of 6.20 Let V be a countable base for hence reglar. For any x e X and B e V containing x, we have by 6.21 a U e such that G U c #, and also by detinition of a base 3 A e V with e A c U c V, and hence x e X'c V c B. (Xis the smallest closed set containing A, note.) Since V is countable, the collection of all such pairs, say .
'1
.z7
.z'
.4
=
((A,#): A e F, B e F;
X'c BI,
(6.23)
is countable, and so we can label its elements A,B4i (Aj,Sj), i = 1,2,... Every x e X lies in Aj fr some i e IN. Since the space is normal, we have by Urysohn's lemma a separating function : X F-+ (0,IJ for each element of .W,such that fixi) 1 and hB 0. For each a' e X and closed set C such that # C, choose (Aj,#Jsuch that x e Xic Bi c Cc, 0. These separating functions form a countable and then h.@) 1 and ficj class F, a subset of U(X). Since the space is F1, C can be a point (yl so that F separates points. And since the space is F3j and hence completely regular, is the weak topology induced by F, by 6.14. It follows by 6.19 that the evaluation map for F embeds X into (0,1)*.. =
=
=
.x
=
=
':
IJ= endowed with the metric px defined in (5.19)is a compact r0, metric space. lt follows that :(X), which is homeomorphic to X under the evalRecall that
uation mapping by F, is a totally bounded metric space. lt further follows that is metrizable, since, among other possibilities, it can be endowed with the and y of X is taken to be metric under which the distance between points have metrization theorem, 6.12. We Urysohn px(c@),c@)). therefore proved the The topology induced by this metric on g0,11= is the Tychonoff topology. A base for a point p e g0,1)*in this topology is provided by sets of the (p1,p2,...) (X,I)
.x
=
Topology
107
fonn Nlp-nkte.t
=
14./e l0,11*: I#/ -
qi I
i
< E,
=
(6.24)
1,...,/c),
for some finite k, and 0 < 6: < 1, which is the same as (6.20).The topology induced on X by the embedding is accordingly generated by the base sets Nx,.kte)
=
ly e
I @) fily)I <
X:
-
:,
i
=
1,...,/c1,
(6.25)
which can be recognized as finite intersections of the inverse images, under functions from F, of e-neighbourhoods of R; this is indeed the weak topology induced by F. This further serves to remind us of the close link between product topologies and weak topologies. Since metric spaces are F4, separable metric spaces can be embedded in (0,lj= by 6.20. In this case the motivation is not metrization, but usually compactitication that is, to show that separable spaces can be topologized as totally bounded spaces. Both metrization and compactification are techniques with important applications in the theory of weak convergence, which we study in Chapter 26. Although the following theorem is a straightforward corollary of 6.20, the result is of suftkient interest to deserve its own proof; the main interest is to see how in metric spaces there always exists a ready-made collection of functions to define the weak topology. -
A separable metric space (S,#) is homeomorphic to a subset of
6.22 Theorem
(0,11*.
Proof Let Jo J/(1 + #), which satisfies 0 f Jo f 1 and is equivalent to d, so that (X,A)is homeomorphic to (X,#). By separability there exists a countable set of points (zf,i e EN) which is dense in X. Let a countable family of functions be 1,2,..., and define an evaluation map : X F-, defined by fix) dx,zi), i =
=
=
x jj y g0,11
hx)
(%(mz1),J0(x,z2),...). where px(,#) that h is an embedding in (g0,1J=,px)
(6.26)
=
We show 7=1Ihk #kl llk. lf (x,,) is a squence in X converging to x, then for each k, dolxntz 4(x,zk)k Accordingly, V k,z H N 2 1 such that xn e Nxik,e) for a11n k N, poohxnlvhxjj --> 0, and h is continuous at x. On th other hand, if xn zz...hx, there exists : > 0 such that Y N k 1, djxntx) 2 6: for some n k N. Since (z:) is dense in X, there is and J0(mzk) < doxvz I 2 so that 1doxnnz a k for which doxnnz k 1: and hence =
-
--
Y
px((x,,),(x)) 2
/,
/2
1+1
-
(6.27)
k
Since this holds for some n k N for every N k 1, it holds for infinltely many n, and hxn) zz-y (x). We have therefore shown that hxn) (x) if and only (Jxn --).x. This is the property of a 1-1 contingous function with cntinuous inverse. . --
But note too the alternative approach of transforming th distance functions into ting functions as in (6t10),@nd applying6.20.
separa
11 PROBABILITY
7 Probability Spaces
Probability Measures A random experiment is an action or observation whose outcome is uncertain in advance of its occurrence. Tosses of a coin, spins of a roulette wheel, and observations of the price of a stock are familiar examples. A probability space, the triple (D,1,#), is to be thought of as a mathematical model of a random experiment. f is the sample space, the set of all the possible outcomes of the The collection 5 experiment, called the random elements, individually denoted of random events is a c-field of subsets of f1, the event A e 5 being said to have occurred if the outcome of the experiment is an element of A. A measure # is assigned to the elements of 5, PA) being the probability of A. Fonually, we have the following. .
7.1 Definition A probability measure (p.m.)on a measurable space (f1,1)is a set function #: 5 F--> (0,1j satisfying the axioms of probability (a) #(A) k 0, for a11:4 e T. 1. (b) #(D) (c) Countable additivity: for a disjoint collection tz4j e @,j e IN) =
,
#
Uz4yTPAj).
n
=
Under the frequentist intepretation of probability, #(A) is the limiting case of theproportion of a long run of repeated experimnts in which the outcome is in 4. Alternatively, probability may be viewed as a subjective notion with #(A) said to represent n observer's degree of belief that A will occur. For present purposes, the intrpretation given to the probabilities has no relevance. The theory stands or falls by its mathematical consisteny alone, although it is then up to us to decidewhether the results acord with ur intuition and are useful in the analysis of real-world problems. Additional properties of P follow i om the axioms.
7.2 Theorem lf A, #, and (, j e Eq) am afbitrary T-sets, then (i) #(X) f 1. #(AC) 1 #(A). (ii) t E 0. (iii) PZ) ytiitity). (iv) A c # = #(A) f #() tpij lty, yy, y Ir$r7.y7,#kk. yyE ');.(.y.-- ..ty .... ... ...;. .. 4(:(y.-t,d$p1k-2. .)-.(21,q.y.. .. 44:4)..j!!E111, L-i4qii.. jit' ..ty..-. (v)(A w B) = pA) + .ji!!:::,' y )j)t yq j y,, . . . yy ,y jg tr j, ju j yy, jyky y kjy (vi) PojAj) f L)PAj) (ctllitAb. tyt,tt j j,( y y rytyrjjtyry =
-
'
=
i
,
-t.(..-...y...ty-.t
(2)jI.
..))
.jj!.::;i'
-. .
..-...k.
....
..
-'
...tE.y..-
...
.
-
.
.
,jj
.
$jjy
,tjjyyytyyyylyyy,yyqyjyj.ytlyyttyyyyyy,ly
...),
.. .
..
y...
,y
,
y,y ,
..
112
Probability
(vii) Aj
'r'
t, A
A or Aj
PAj) .-A #(A)
(continuity).cl
Most of these are properties of measures in general. The complementation property (ii) is special to #, although an analogous condition holds for any finite measure, with #(X replacing 1 in the formula. (iii) confinns P is a measure, on the definition. Proof Applying 7.1(a), P(A)
+
(b), and (c), PAc)
PA t.JAc)
=
#(f)
=
1,
=
from which follow (i) and (ii), and also (iii) on setting A by 3.3, and (vii) by 3.4. w
=
(.
(iv)-(vi)follow
To create a prbability space, probabilities are assigned to a basic class of according C, events to a hypothesis about the mechanisms underlying the random outcome. For example, in coin or die tossing experiments we have the usual hypothesis of a fair coin or die, and hence of equally likely outcomes. Then, provided C is rich enough to be a determining class for the space, (f,@,#) exists by 3.8
(extension theorem) where ?F = c(C).
7.3 Example
Let Sp,1)
=
(# fn g0,1), #
e f), where T is the Borel field on R.
Then ((0,1), fm,1J, m), where m is Lebesgue measure, is a probability space, 1. The random elements of this space are real numbers between 0 since ,7:(:0,1)) and 1, an a drawing from the distribution is called a random variable. It is said to be distributed rfnt/r/zl/y on the unit interval. The inclusion or exclusion of the endpoints is optional, remembering that /1((0, 1J) r?2440,1)) = 1. n =
=
The atoms of a p.m. are the outcomes (singletonsets of ) that have positive probability. The following is true for tinite measures generally but has special importance in the theory of distributions.
7.4 Theorem The atoms of a p.m. are at most countable. Proof Let l be an atom satisfying P((1 J) #(( J)for al1 (0 6 f1, 1et (t'/2 satisfy #(f tl)2)) k #4(1) for all e f(t)1 ), and so forth, to generate a sequence with -
#(l1l)
2 #(tt%l)
2 #(t2l)
...
(7.3)
The partial sums Z7=1#(fj)) form a monotone sequence which cannot exceed #(D) 1, and therefore converges by 2.11, implying by 2.25 that limn-hxftf.t)a 1) 0. All points with positive probability are therefore in the countable set f(, i e Nl . =
=
.
Suppose a random experimentrepresented by the space (fk9,#) is modified so as to confine the possible outcomes to a subset of the sample space, say A c f1. For example, suppose we switch from playing roulette with a wheel having a zero slot to one without. The restricted probability space is derived as follows. Let 5% denote the collection fEcj A, E e @) 5%is a c-tield (compare1.23) called 5h and #(m/#(A) for E e Fa, Ph can be is called the trace of % on A. Defining #a() (A,Fa,#A) is called the trace of (f1,F,#) on A. triple verified to be a p.m. .
=
'lhe
Probability Spaces This is similar to the restriction of a measure space to a subspace, except that the measure is renormalized so that it remains a p.m. In everyday language, we are inclined to say that events may be or certain' lf such events are none the less elements of @, and hence technically by random, we convey the idea that they will occur or not occur with here assigning them probabilities of zero or one. The usage of the tenn marks is deliberately loose, as the quotation suggest. To say an event cannot it has probability different from saying it cannot occur because is zero occur because the outcomes it contains are not elements of f1. Similarly, to say an event has probability 1 is different from saying it is the event f1. In technical discussion we therefore make the nice distinction between sure, which means the latter, and almost sure, which means the former. An event E is said to occur almostsurely fl E has prob(a.s.), or equivalently, with probability one (w.p.1) if M abilitymeasurezero.This terminologyis synonymouswithlfv/l/eveqwercta.e.) in the measure-theoretic context. When there is ambiguity about the p.m. being considered, the notation may be uded. Aimpossible'
.
tcertainty' tcertain'
=
-
4a.s.(#j'
7.2 Conditional Probability A central issue of probability is the treatment of relationships. When random experiments generate a multi-dimensioned outcome (e.g.a poker deal generates several different hands) questions always arise about relationships between the different aspects of the experiment. The natural way to pose such questions is:
0. #(. IA) satisfies the event B as #(#IA) probability axioms as long as P does and #(A) > 0. In particular, #(A IA) 1, and P(Bc IA) 1 PB lA), since B /-74 and Bc DA are disjoint, and their union is 4. The space (A,@x,fk), the trace of the set A on (D,T,#), models the random experiment from the point of view of an observer who knows that (l) e A. Events A and # are said to be dependent when PB IA) # #(#). In certain respects the conditioning concept seems a little improper. A context in which the components of the random outcome are revealed sequentially to an observer might appear relevant only to a subjective interpretation of prpbability, and lead a sceptical reader to call the neutrality of the mathematical theory ipto question. We might also protest that a random event is random, and hak no business Gthe
=
=
=
-
Probability
114
defining a probability space. ln practice, the applications of conditional probability in limit theory are usually quite remote from any considerations of subjectivity, but there is a serious point here, which is the difficulty of constructing a rigorous theory once we depart from the restricted goal of predicting random outcomes a priori. The way we can overcome improprieties of this kind, and obtain a much more powerful theol'y into the bargain, is to condition on a class of events, a c-subfield of 9. Given an event B e F, let the set function F-> (0,11 /74#1 ): '
represent the contingent probability to be assigned to B after drawing an event A from N, where V i @.We can think of V'as an infonuation set in the sense that, for each A G V, an observer knows whether or not the outcome is in A. Since the elements of the domain are random events, we must think of #(# lN) as itself a random outcome (arandom variable, in the terminology of Chapter 8) derived from the restricted probability space (f1,V,#). We may think of this space as a model of the action of an observer possessing information V, who assigns the conditional probability #(# lA) to B when be observes the occurrence of A, viewed from the standpoint of another observer who has no prior infonnation. N is a c-tield, because if we know an outcome is in A we also know it is not inAc, and if we know whether or not it is in Aj for each j 1,2,3,..., we know whether or not it is in U#.j. The more sets there are in V the larger the volume of information, all the way from the trivial set 5 (f,0) (completeignorance, with 8#1 T) #(#) a.s.) to the set 5 itself, which orresponds to almost sure knowledge of the outcome. In the latter case, P@ 11) 1 a.s. if ( e B, and 0 otherwise. lf you know whether or not (J) G A for every A e @, you eftctively know (J). 8 =
=
=
=
7.3 Independence A pair of events A, B e 5 is said to be independent if #(A ch#) equivalently, if PB jA)
=
#(#).
=
#(A)#(#), or,
(7.4)
lf, in a collection of events C, (7.4)holds for every pair of distinct sets A and B from the cotlection, C is said to be pairwise indepenttent. In addition, C is said to be totally independent if (.D-,A)
#
-
J,'tA'
(7.5)
containing two or more events. This is a stronger condition for every subset # i thanpairwise independence. Suppose Cconsists of setsA, #, and C. Knowing thatS has occurred may not influence the probability we attach to A, and similarly for C) but thejoint occunrnce of B and C may none the less imply something about A. Pairwiseindependence implies thatfA fa B) =PIAIPCBI,PCA rn C) =#(A)#(C), and PB rn C' PB)PC4, but total independence would also require P(A f'n B rn C) = =
P(A)PB)PC).
Probability Spaces
115
Here are two useful facts about independent events. In each theorem let C be a totally independent collection, satisfying ptf)xs .gA) l7ze.y#(A) for each subset =
#' I C. 7.5 Theorem The collection C' which contains independent.
yt
and Ac for each A e C is totally
Proof It is sufficient to prove that the independence of A and B implies that of Ac and B, for B can denote any arbitrary intersection of sets from the collection AC. This is certainly true, since if and (7.5) will be satisfied, for either A or #(A)#(#), then #(A f''h B) =
PCAC
r'A #)
=
PAc ra #)
= #(#)
-
+
P(A fa #)
P(A4PB4
-
#(A)#(#)
PAPB4.
=
(7.6)
w
7.6 Theorem Let () be a countable disjoint collection, and let t e collections consisting of Bj and the sets of C be totally independent for each-j. Then, if B LjBj, the collection consisting of B and C is als independent.
=
Proof Let #' be any subset of C. Using the disjointness of the sr-ts of B, and
countable additivity, P
j
rn
)
xOs ,A
-
P
sA)
zOs ) (#?
(Uy Bj ra ,A xOs XyP -
=Xy
PCB?P
7.4 Product Spaces
sA)
tj
=
Plhj
ra
,PA4.
.
and independence arise when multiple random eyperiments run in parallel, and product spaces play a natural role in the analysis of these issues. Let (f x E, 5 @ V, #) be a probability space where 5 (*)V is the
Questionsof dependence
c-fieldgenerated by the measurable rectangles of (-1x E, and PL x E) 1. The randomoutcome is a pair (,4). This is no more than a case of the general theory of j7.1 (wherethe nature of (l) is unspecified) except that it becomes possible to askquestions about the part of the outcome represented by ) or ( alone. #04/7 =
=
#(Fx E) for F e F, and #E(G) Pl x G4 for G e N, are called the marginal probabilities.(1,1,#0) and (E,T,#E) are probability spaces representing an incompletelyobserved random experiment, with ) or k, respectively, being the only things observed in an experiment generating (,(). On the other hand, suppose we observe ( and subsequently consider the experiment'of observing (t). Knowingtmeans thatforeachf x Gweknowwhetherornot (,() is in it. The conditional probabilities generated by tlzis two-stage experimentcan be wr itten by a slight abuse of notation as #(Fl V), although strictly Fx E, and the elemts of the speakingthe re 1evan t events are the cylinders uzrln enHzw+lxlv.lllv,qia 'n uzn n'zala' rmfiold nre 0./ G for G G W. Cnn ditinnine =
Probability
116
#(Fx E lfl x T). ln this context, product measure assumes a special role as the model of independence. In (f x E, 5 (& V, #), the coordinate spaces are said to be independent when PFx
G)
=
#0(#)#s(G)
for each F e 5 and G e T.
Unity of the notation is preserved since Fx G (Fx E) fa (( x G). We can also write #(Fx EIf x G) /704/-), or with a further slight abuse of notation #(FIG) #n(/'), for any pair Fq 5 and G e N. Independence means that knowing ( does not affect the probabilities assigned to sets of ;. Since the measurable rectangles are a determining class for the space, the p.m. P is entirely detennined by the =
=
marginal measures.
=
8 Random Variables
8. 1 Measures on the Line a probability space. A real random variable (r.v.) is an 5lBf'unction X: ( F.-: R 9 That is to say, aY4)) induces an inverse mapping measurable from B to 5 such that X -1(#) l 5 for every B e B where B is the linear Borel field. The term T-measurable r.v.' may be used when the role of B is understood. The symbol g. will be generally used to denote a p.m. on the line, reserving P for the p.m. on the underlying space. Random variables therefore live in the space (R,S,g,), where g. is the derived measure such that g(#) PX -1(#)) = #( X e B) The term distribution is synonymous with measure in this context. The properties of nv.s are special cases of the results in Chapter 3; in particular, the contents of j3.6 should be reviewed = in conjunction with this chapter. If g: R F-> R is-1a Borel function, then gox is also a r.v., having derived p.m. g,: according to 3.21. p(A)) If there is a set S e B having the property g(& 1, the trace of (R,S,g) on S is equivalent to the original space in the sense that the same measure is assigned to B and to B r''hS, for each # e S. Which space to work with is basically a matter of technical convenience. If X is a r.v., it may be more satisfactory to say that the Borel function X 2 is a r v. distributed on R+ than that it is distributed on R but takes values in R+ almost surely. One could substitute for (M,S,g,) the extended space (F,S,g) (see 1.22), but note that assigning a positive probability to infinity does not lead to meaningf'ul results. Random variables must be finite with probability 1. Thus (R,f,g), the trace of (F,S,g) on R, is equivalent to it for nearly a1l purposes. However, while it is always finite a.s., a r.v. is not necessarily bounded a.s.; there may exist no constant # such that 1X(l I f B for a1l (l) e C, with #(D C4 0. The essential supremum of X is
Let
(f1,1,#) be
.
,
=
.
=
,
.
=
-
ess sup X
=
inf (x: #41XI > x)
and this may be either a finite number, or
0),
=
+x.
8.2 Distribution Functions The cumulative distribution
@.b.f.)Of
f'Ifrlcft??z
X is the function F:
where
2- F-> g0,1J,
< xj, x s F. (8.2) l ( ( fty/, j'q..(( gn the values 0 and 1 to )( jj:t.:.' tk..j.,.((y. we take the domain to be R-since it' iq.(... (('..t'jr(t.t:.....t( .. .. . y)':)j)t..: .i-) ... (j).jL?, f;. ji,j.. . F(x)
=
g((-x, x1)
x
=
.
.
..
.
.
:
'
:
:.)'
.-
..'.''
....
.
..)
.... .
.
'
'
.t.
.
.:$j.i.
(Ej
.
.
....j
... y.. q .))(-.3E,l'( y j.y)y(y)jgE.t..-'(.:,.t.yy.. .t
...yr.j(,?:jyj.jy.
,,;.j..j;...(..j.
..
.
-(.y.'.,
.
. .
..
.
.
.
.
c
..
.
r)().... .. .. . y.....) ,.E.
. .
.
Probability
118
F(-x) and F(+x) respectively. No other values are possible so there is no contradiction in contining attention to just the points of R. To specify a distribution for X it is sufticient to assign a functional form for F; p, and F are equivalent representations of the distribution, each useful for different purposes. To represent g,(A) in terms of F for a set A much more complicated than an interval would be cumbersome, but on the other hand, the graph of Fis an appealing way to display the characteristics of the distribution. To see how probabilities are assigned to sets using F, start with the half-open interval (x,y) for x < y. This is the intersection of the half-lines (x,y) and (-x, (.x,+x).Let A (-=, x) and B = (-x,y), so that g(A) F@) and g(#) F(y); then -xzf
=
=
BIAC
g,(@,)/1) =
=
ra #)
= 1 (g(A) -
1 p,tA
=
+
-
1 g(#))
tJ#C)
g(#)
=
-
=
-
g(z4)
=
F@) F@), -
A and Bc being disjoint. The half-open intervals form a semi-ring (see1.18), and from the results of j3.2 the measure extends uniquely to the sets of f. a) for x c R As an example of the extension, we detennine g,((x)) PX will (compare 3.15). Putting a7 y in (8.3) not yield this result, since A cjAc = 0, not (xl We could obtain fxl as the intersection of (-x, x) and (-x, xlc, but then there is no obvious way to tind the probability for the open interval (-x, x) (-x, xl f.xl The solution to the problem is to consider the monotone sequence of half-lines (-x, x 1/n) for n e N. Since @ 1/p, (-x, aj, 1/n), according have 1/n,<) F(x) lln Fx ra g,(@ to we (-x, x = and where 1/n, x), Fx) F@-), Since T g,((x)) (8.3). (a7Je fx) 0':nltx F@-) is the left limit of F at x. Fx) exceeds Fx-j (i.e. F jumps) at the atoms of the distribution, points with g,((xJ)> 0. We can deduce by the same kind of F(y-4 Fx), g((x,y)) = F@-) F(x-), and that, reasoning that p,(@,y)) generally, measures of open intervals are the same as those of closed intervals unless the endpoints are atoms of the distribution. Certain characteristics imposed on the c.d.f. by its definition in tenns of a measure were implicit in the above conclusions. The next three theorems establish these properties. =
=
=
=
rx,+x)
.
=
-
.
.:71
-
=
-
=
=
-
-
-
-
-
-
.z:
=
-
-
1, 8.1 Theorem F is non-negative and non-decreasing, with F(-x) 0 and F(+x) and is increasing at e R iff every open neighbourhood of x has positive measure. =
=
.x
Proof These are all direct consequences of the definition. Non-negativity is from (8r2), and monotonicity from 7.2(iv). F is increasing at x if Fx + :) > F@ E) for each E > 0. To show the asserted sufficiency, we have for each such E, -
F@ + :)
-
F@ :) k F((x
+ 8)-)
-
For the necessity, suppose g,(5'(x,E))
g,(5'(x,e))= F(@ + e,)-) F@ -
-
=
E)
-
F(x
-
6,)
=
g,(5'@,y)).
(8.4)
0 and note that, by monotonicity of F, k Fx
+ :/2)
-
F@- :/2).
w
The collection of points on which F increases is known as the sunnort of
(8.5) 11
IfA
Random
Variables
119
complement in R, the largest set of zero measure, consists of points that must a11 liein open neighbourhoods of zero measure, and hencemustbe open.-rhe supportof g, is accordingly a closed set. F is right-continuous
8.2 Theorem Proof For x
l
everywhere.
R and n 2 1, additivity of the p.m. implies
g((-x,
.v
+
1/nJ) g,((-x, =
x1)
+
g,(@,x+ 1/n1).
(8.6)
g,(4-x, x+ 1/aJ)1, g.((-x, x1) by continuity of the measure, and hence limn-yxjtttx, + 1/n1) 0. lt follows that for er > 0 there exists Ne such that g,(@,x + 1/nJ) < E, and, accordingly, As n
.-.+
x,
.v
P,((-=,
=
.Y1)
f t((-=,
for n k Ne. Hence F(x+)
=
x+ 1/n1)
<
p,tt-x, x1) +
(8.7)
E,
F(x), proving the theorem since x was arbitrary.
.
If F@) had been defined as g,((-x, x)), similar arguments would show that it was left ontinuous in that case.
'W
Fig. 8.1
8.3 Theorem F has the decomposition F(x)
whereF'@) jumps, and
=
F'x)
+
F'x)
(8.8)
is a right-continuous step function with at most a countable number of r'(x) is everywhere continuous.
Proof By 7.4, the jump points of F are at most countable. Letting
denote these
points,
f-x1,x2, ...)
Probability
120 F'@)
=
X xisx
Fxi,
Fxi-j)
-
(8.9)
is a step function with jumps at the points xi, and F'x)
F(xi-)
=
Flxi) at each xi and is continuous everywhere.
Fx)
=
-
F'x)
has
*
Fig. 8. 1 illustrates the decomposition. This is not the only decomposition of F. The Lebesgue decomposition of g with respect to Lebesgue measure on R (see4.28) is g = p,1+ g.2where g,l is singular with respect to m (is positive only on a set of Lebesgue measure 0) and g2 is absolutely continuous with respect to Lebesgue measure. Recall that g2(A) = fxjdxo A e B where Tis the associated Radon-Nikodym derivative (density function). If we decompose F in the same way, such that Ff(x) w((-x,x1) for i = JQ.f(()J), implying that f@) dFz /#( I(=x.This = 1 and 2, we may write Flx) must hold for almost all x (Lebesgue measure), and we call F2 an absolutely continuous f'unction,meaning it is differentiable almost everywhere on its domain. F' S F1 since F1 may increase on a set of Lebesgue measure 0, and such sets can be uncountable, and hence larger than the set of atoms. It is customary to summarize these relations by decomposing F' into two additive components, the absolutely continuous part &, and a component F3 = F'- F2 which is continuous and also singular, constantexcept on a setof zero Lebesgue measure. This componentcan in most cases be neglected. The collection of half-lines with rational endpoints generates B (1.21) and should be a determining class for measures on @,f). The following theorem establishes the fact that a c.d.f. defined on a dense subset of R is a unique representation of g,. ,
=
=
8.4 Theorem Let z be a tinite measure on (R,f) and D a dense subset of R. The function G defined by 81)
Gx)
=
IX(-=,-Y1),
e D
.<
=
x e R D
F(x+),
-
(8.10)
is identical with F. Proof By definition, R q X'and the points of R D are a11closure points of D. For each x e R, not excluding points in R D, there is a sequence of points in D converging to x (e.g. choose a point from Sx,lln) r'AD for n g N). Since F is F(x+) for each x e R -D. w right-continuous everywhere on R, p,((-x,x)) -
-
=
Finally, we show that every F corresponds
to some g,, as well as every g to an F.
8.5 Theorem Let F: V --A E0,1) be a non-negative, non-decreasing, right=0 and F(+x) 1. There exists a uniquep.m. g on continuous function, with F(-x) (R,S) sqch that Fx) g((-x,x1) for a11a7 G R. u =
=
Right continuity, as noted above, correspopds to the convention of defining F by (8.2). If instead we defined F@) = g((-x,x)), a left-continuous non-decreasing F
Random Variables
121
would represent a p.m.
#: (0,11F- 2', defined
Proof Consider the function (4If)
=
inf (x: u < F(x) )
by
(8. 11)
.
4)can be thought of as the inverse of F; (40) (41) +x, and since F is nonIight-continuous, decreasing and # is non-decreasing and left-continuous; # is Borel-measurable by According to 3.21, we may define a meas3.32(ii). therefore ?n4 -1 (#) for each B e f, where m is Lebesgue measure on theBorel ure on (R,f) by of sets (0,11. ln particular, consider the class C of the half-open intervals (J,:1 for a11a,b e R with a < b. This is a semi-ring by 1.18, and c(C) B by 1.21. Note that t)-1((J, 4) (/g:inf (x: u K F(x)) e (tz,h1 ) (F((7),Fblj. (8.12) =
-x,
=
=
=
=
For each of these sets define the measure P,((J,#1)
=
/19
-1((J, :j))
=
Fb)
-
(8.13)
Fa).
The fact that this is a measure follows from the argument of the preceding paragraph. C is a determining class for (R,O), and the measure has an extension by 3.8. It is a p.m. since g.(R) 1, and is unique by 3.13. x =
The neat construction used in this proof has othef applications in the theory of random variables, and will reappear in more elaborate form in 922.2.The g'raph of () is found by rotating and reflecting the graph of F, sketched in Fig. 8.2; to see the fonner with the usual coordinates, turn the page on its side and view in a
mirror.
Fig. 8.2 If F has a discontinuity at x, then 4) x on the interval (F@-), F(x)1, and -1(lxl) = (F(x-), F(x)j. Thus, g,((x)) -((F(x-), F@)1) F(x) F(x-), as 41 required.On the other hand, if an interval (J,!4 has measure 0 under F, F is =
=
=
-
Probability
122
F(b) = c (say).4) constant on this interval and # has a discontinuity at Fa) -1 takes the value a at this point, by left continuity. Note that () (c) = (J,>1, so that g,((J,:J) mc) = 0, as require. =
=
8.3 Examples Most of the distributions met with in practice are either discrete or continuous. A discrete distribution assigns zero probability to a11 but a countable set of points, with F' = 0 in te decomposition of 8.3.
8.6 Example The Bemoulli (orbinal'y) nv. takes values 1 and 0 with fixed probaThink of it as a mapping from any probability space containbilities p and 1 elements. and tFailure'
isuccess'
;
1
8.7 Example The binomial distribution with parameters n and p (denotedBn,pl) is the disibution of the number of ls obtained in n independent drawings from the Bernoulli distribution, having the probability function P(X
=
x)
4
=
x
p
x
(1
n-x
-p)
x
,
=
8.8 Example The limiting case of (8.14) with p distribution, having probability function P(X
=
x)
-w
cp
1
0,...,p.
hln. as
=
(8.14)
Ia
n
-..-y
x,
is the Poisson
-h,
.
X'.
x ,
x
=
() j ,
,
z
(8.15)
,...
This is a discrete distribution with a countably infinite set of outcomes.
D
In a continuous distribution, F is absolutely continuous with FI = 0 in the Lebesgue decomposition of the c.d.f. The derivative f #F/#x exists a.e.(>1 on R and is called the probability densit.v function(p.d.f.)of th p.m. According to the Radon-Nikodym theorem, the p.d.f. has the property that for each E e f, =
g,(f)
=
jyflvdx.
,
(S' 16)
8.9 Example For the unkf'orm distribution on g0,11 (see 7.3), 0, x F@)
<
x, 0 f
=
1, x
>
0 .x
S 1.
(8.17)
1
The p.d.f is constant at 1 on the interval, but is undefined at 0 and 1.
E1
8.10 Example The standard normal or Gaussian distribution has p.d.f. f(x)
=
1
/-2a
-x2/2
e
-x
'
<
x
< +.=
,
(8.18)
Random Variables
whose graph is the well-known
123
bell-shaped curve with mode at zero. n
8.11 Example The Cauchy distribution has p.d.f.
fxq
1
=
<
-cx,
,
+x2)
zr(1
which, like the Gaussian, is symmetric
x
(8.19)
< +x,
with mode at 0. n
When it exists, the p.d.f. is the usual means of characterizing a distribution. A particularly useful trick is to be able to derive the distribution of gzk' from that of X, when g is a function of a suitable type.
8.12 Theorem Let #: S F--> 1- be a 1-1 function onto 7, where S and 1- are open subsets of R, and let h g -1 be con tinuously differentiable with dhldy # 0 for all y e 7. lf X is continuously distributed with p.d.f. fx, and F #(.X), then F is continuously distributed with p.d.f. =
=
),I
.fx(,-1(,))
s(,) -
.
(8.20)
u
The proof is an easy exercise in differential calculus. This result illustl-ates 3.21, but in most other cases it is a great deal harder than this to derive a closed form for a transformed distribution.
8.13 Example Generalize the uniform distribution (8.9)from (0,1) to an arbitrary interval gtz,:1.The transformation is linear, y so that fyy)
=
b a) -1 on
a
=
(J,:),
-
F@)
b
+
by
(.y
=
-
(8.21)
Jlx,
-
(8.20).The
allb
c.d.f. is d e jjxd
on
a).
-
gc,)j by (8.22)
Membership of the uniform family is denoted by X
-
&(c,:1. n
8.14 Exapple Linear transformations of the standard Gaussian r.v., X
=
jt + cZ,
c
>
0,
(8.23)
generate the Gaussian family of distributions, with p.d.f.s
1 e fx'' g.,0-) /--2ac
-tx-jtlz/acz
'-=
=
,
<
.7
< +=
.
(:I
(8.24)
have better-known designatiops as The location parameter ytand scaleparameter and 9.7 9.4 below. distribution; Membership of the Gaussian of see moments the 2 family is denoted by X #(g,,c ). -
A family of Cauchy distributions is generated from the standard Cauchy by linear transformations X v + Z, > 0. ne family of p.d.f.s with locationparameter v and scale parnmeter take the form
8.15 Example
=
Probability
k=N
J@;v,)
1
1
=
k''g1+
r(x -
<
-=
,
v)/) z
< +x.
.z7
(8.25)
n
8.16 Example Consider the square of a standard Gaussian nv. wit p.= 0 and c = 1. Since the transfonnation is not monotone we cannot use 8.12 to detennine the density, density, strictly speaking. But consider the Khalf-normal'
zlt&l =
2J(ff), u
k 0
0,
<
u
(8.26)
0
where f is given by (8.18).This is the p.d.f. of the absolute value of a Gaussian variable. The transformation g Iu I) = u 2 is 1-1 so the p.d.f. of Z2 is ,
1 e fz24&) W2V
-1/2
-un
u
=
applying(8.20).This is the ch-squared or z241).It is a member (withp (x =
G(?z; ',pj W
herej-+4
=
f!
() < u
,
=
and 1R?zl (n llF(n
-
(8.27)
,
-1 -ea
((xI/)P
vpe
-
.
distribution with one degree of freedom, ) of the gnmma family,
,
0
<
u
< x,
is the gammafunction, J7(P-1e-Q(
=
=
<
.
a
>
0, p
>
0,
(8.28)
having the properties F()
=
W2'V,
1). c!
8.4 Multivariate Distributions In Euclidean k-space R k the k-dimensional Borel field Tk is c(S$ where Rk denotes the measurable rectangles of Rk th e se ts of the form #1 x B1 x x Bk where Sf e B for f = 1,...,k. In a space (D,@,#), a random vector (msXi,...,.V)' = X is a measurable mapping X1 fl Rk ,
,
,
...
.-+
.
lf t is the derived measure such that g,(A) = PE) for A G T k an d E s 5 the mu ltivariate c.d.f., F: Fk (0,1), is defined for (x1,...,x' = x by ,
..-+
F@)
=
g,((-=,x11 x
...
x (-x, aJ).
(8.29)
The extension proceeds much like the scalar case.
8.17 Example Consider the random pair (X,D.Let F(x,y) = g((-x, XJx (-x, y1). The measure of the half-open rectangle (x,x+ Ax1x (z,y +Ay) is M7x, y)
=
F@ + Ax, y + Ay)
-
Fx
+
Ax,y)
-
F@, y + Ay) + Fx, y4.
(8.30)
To show this, consider the four disjoint sets of R 2 illustrated in Fig. 8.3:
Random Variables
125
:4
=
@,x + AXJx @,y + AyJ,
B
=
(-x, x) x (.y,y +Ay),
C
=
(x,x+AXJx (-x, yq,
D
=
(-x,x) x (-x,yj.
A is the set whose probability is sought. Since #(A wB tp Ck.)D) Fx + Ax,y), and #(D) Fx+ Ay + Ay), #(# QJ D) = Fxby + Ay), PCCwD) result probability is immediate from axioms. the Fx, y), the n =
= =
Fig. 8.3 Extending the approach of 8.17 inductively, the measure of the k-dimensional , xi + AQ can be shown to be rectang le Xil@f, AF(I1,...,
A'k)
=
77(:f..6/),
(8.31)
j
where the sum on the right has 2k terms, an d the Fj are the values of F at each of the verices of the k-dimensional rectangle extending from @1,....q)'with sides 1,...,k. The sign pattern depends on k', if k is odd, the Fj of length Mj, i having as arguments even numbers of upper vertices (pointsof the form xi + A'&)) take negative signs, and the others positive; wilile if k is even, the Fj with odd numbers of upper vertices as arguments are negative. Generalizing the monotonicbe ity of the univmiate c.d.f., F must satisfy the condition that AF@1,..., (R)+ non-negative for every chice of (x1,..., ' e R an d (A.x1,..., Axk)' s Applying 3.19 inductively shows that the class of half-open rectangles is a semi-ring, so that the measure defined by F extends to the sets of T k. hence (R Bk is a probability space derived from L1,5,P). If the distribution is continuous with p.d.f. flx), Fubini's iheoremgives =
.q)
.q)
.
-dimensional
,
,g)
,
Fx)
.f((1,...,(z)t:61
.dlk
=
..
(-x,xl1x,.,x(-x,xk)
=
xl
xk ...
J(k1
.,4:2)t2)1
...#(k.
,..
Theorem 8.12 has the following generalization. A dteomorphism
(8.32)
(also,coor-
Probability
126
dinate transformationj is a function g: S F- -T(S and 7 open subsets of Rk) w lli c h # 0 for all x e S, is 1-1 onto and continuously differentiable with etglx') whose element glx' Jacobian matrix is is the gllxi for i,j where (f,.j)th 1,...,1. The inverse of a diffeomorphism is also continuously differentiable. =
8.18 Theorem If F
where g is a diffeomorphism,
g
=
the p.d.f. of F is
h%l flg-1(.:))1yj
(g ag)
=
whereJ
detl:
=
-1
)j
y
'j
.
u
.
This is a standard result in the theory of multiple Lebesgue integrals Apostol 1974: 15.10-15.12).
8.19 Example Letting
denote the standard Gaussian p.d.f.
.f
k
lzi) 1-1
9(z) =
(zzo-Wexpf
=
(see8.10),
(see e.g. consider
-z1(z'z)).
(8.34)
i=1
This is a k-dimensional p.d.f., and the corresponding random vector (z1,...,Zk)', is the standard Gaussian vector. The affine transformation X
AZ+ !z,
=
Z
=
(8.35)
whereA (kx k nonsingular) and M kx 1) are constants, is 1-1 continuous with inversez A -1(X p.) having J IA-1 l 1/lA I Detine E 4A' such that (A -1) x-1 and j j4 j j jEI the positive square root being ( nderstood.Applying 8.18 produces =
=
ix,
,
=
,
-
jx I
lnl-m 14-1I =
(21)-1/2 jx j
,
1
9(A -1@ p.))
j
=
.
-1/2
-1
=
=
=
=
-
,4:4..)-1
.:4.-1
j
expf
-12
-yx
-
Jz)'(A-l)'A-1(x
-
Izlj.
.s;,x-1(.x
t
exp
-j(x .s)
j.
(8.36)
This is the multinormal p.d.f., depending on parametets p. and X. Every such distribution is generated by an affine transform applied to Z. Membership of the multinormal family is denoted X N#,X). n -
8.5 lndependent Random Variables Suppose that, out of a pair of r.v.s (.X,1')on @2T1 we are interested only in situation of interest X. In this the events are the cylinder sets in predicting R 2 having the form B xR, B e S. The marginal distribution of X is detined by (R,S,g,x) where ,g),
,
,
p,x(X)
=
t(X X R)
for W e T. The associated marginal c.d.f. is Fx
(8.37)
F@? +x). The notion of independence defined in 97.4specializes in the following way.xand =
Rattdom Variables F are called independent r.v.s iff pt(XX #)
>x(A)g.l'(S)
=
(8.38)
for al1 pairs of events A,B e f, where g.x is detined by (8.37)and g.y is analogous. Equivalently, g, is the product measure generated by g,x and yty. 8.20 Theorem X and J' are independent iff for each x,y Fxnyq
G
R
(8.39)
Fx@)Fy(X.
=
lf the distribution is continuous the p.d.f. factorizes as x,L
fxxjhb.
=
(8.40)
Proof Obviously, (8.39)is true only if g, satisties (8.38).The problem is to show that the former condition is also suftkient. Consider the half-open rectangles,
C
=
((tz,:)x (c,#1,a.c
e
R-,b,d
e R ).
lf ahd only if (8.39)holds, p,((J,/8
X (c,#1)=
= =
Fb,dl
F(:,c)
-
Fxbq
-
-
Fa,dl +Fa,c4
Fxabttdj
-
Fv)
(8.41)
g,x((J,:1)p,i'((c,tf1),
where the first equality is by 8.17 C is a determining class for (R2 S2) and g is detined by the extension of the measure satisfying (8.41)for ordinary rectangles or, equivalently, satisfying (8.39).The Extension Theorem (uniqueness partl shows that this is identical with the product measure satisfying (8.38).The extension to p.d.f.s follows directly from the detinition. . ,
.
,
With more than two variables there are alternative independence concepts k (compare j7.3). Variables m,...,Xk distributed on the space (R- pk are sa itj to be totally independent if ,g.)
,
1
XAf
p,f=1
k
=
l-lp,xJAd)
(8.42)
f=1
for all k-tuples of events A1,...,Akl 0. By contrast, pairwise independence can hold between each pair Xi&. without implying total independence of the set. Another way to think of total independence is in terms of a partitioning of a vector X = X3....,X' into subvectors .Y1 (jx 1) and Xz k x 1) for 0 < j < k. Under total independence, the measure of X is always expressible as the product measure of the two subvectors, under a11orderings and partitionings of the e lements. -j4
9 Expectations
9. 1 Averages and lntegrals or mean, of a r,v. X()l in a probability space
When it exists, the expectation, L1,5,P) is the integral EX4
Joxtovptl'
=
(9.1)
F(m measures the central tendency of the disibution of X. It is sometimes identified with the limiting value of the sample average of realized values xt drawn in n identical random experiments, xn
=
1
n
(9.2)
.t,
-
>1
as n becomes large. However, the validity of this hypothesis depends on the method of repeating the experiment. See Part IV for the details, but suffice it to say at this point that the equivalence certainly holds if F(A') exists and the random experiments are independent of one another. The connection is most evident for simple random variables. lf X = jxjjej where the ()) are a partition of f1, then by 4.4, osllj). /?(r) !F7 j cr
When the probabilities are interpreted as relative frequencies of the events Ej (: X'4(0) z)) in a large number of drawings from the disibution, (9.2) wlth large n should approximate (9.3). The values xj will appear in the sum in a proportion roughly equal to their probability of occurrence. F(m has a dual characterization, as an abstract integral on the parent probability space and as a Lebcsgue-stieltjes integral on the line, under the derived distribution. lt is equally conrct to write either (9.1) or =
=
+x
EX4
=
xdFxx).
(9.4)
Which of these representations is adopted is mainly a matter of convenience. lf 1x() is the indicator function of a set A G T, then
'(1,4m =
jgxldlk =
Jx(x).x#F@),
(9.5)
Expectations
129
where X(A) e B is the image of A under X. Here the abstract integral is obviously the more direct and simple representation, but by the same token, the Stieltjes fonn is the natural way to represent integration over a set in B.
If the distribution is discrete, X is a simple function and the formula in applies directly. Under the derived distribution,
(9.3)
'k-xj
F(m
where xj, j
=
(9.6)
p,(ta)l),
=
1,2,..., are the atoms of the distribution.
9.1 Example If X is a Bernoulli variable (8.6),EX) = 1.# + 0.(1
-#)
=
p. n
9.2 Example If X is Poisson (8.8), F(m
=
x
e-
ax x Wx
x
e-,E (x
=
vl
a
z-1
'A'
'A'
.
l ) 1.
-
vl
=
,.
u
(9.7)
For a, continuous distribution, the Lebesgue-stieltjes integral of a' coincides with the integral in ordinary Lebesgue measure of the function xfx).
9.3 Example For the uniform distribution on the interval gtz,)1(8.13), F(m
b
1
=
b a
xdx
=
1a(c +
-
a
b). n
(9.8)
9.4 Example For the Gaussian family (8.19), F(A3
1
=
-(x-jt)2/2o.2
#?
xe
Wfic
=
g.
(9.9)
This can be shown by integration by parts, but for a neater proof see 11.8.
I:a
ln a mixed continuous-discrete distzibution with atoms x1,x2,..., we can use the decomposition F F1 + F2 where F1(x) = Zouxgltlayl)and F1(x) is absolutely continuous with derivative hx). Then =
EX)
77.)
p,1(tA)l)
=
+
jxhxtdx.
(9.10)
The set of atoms has Lebesgue measure zero in R, so there is no need to exclude these from the integral on the right-hand side of (9.10). Some random variables do not have an expectation.
9.5 Example Recall the condition for integrability in (4.15),and note that for the Cauchy distribution (8.11),
1
+u
jxj
J-a(
-a
1+
x)2
dx
--
'>9
as
f'
-->
=.
El
(9.11)
130
Probability
9.2 Expectations of Functions of X lf X is a r.v. on the probability space @,f,g), and gl R F-- R is a Borel func-1), as note d in j8.1. This leads is a nv. on the space t'R,f tion, goX to the following dual characterization of the expectation of a function. =
'(m
,g.'
9.6 Theorem If g is a Borel function, F('(A'))
jglxtdbkxb /dB#-1t3')'
=
=
&n):R+
Proof Define a sequence of simple functions m i
z(a)(x)77I =
-
1
(9. 12) F-
R+ by
(9. 13)
1s,.@),
2n
1,...,-. Then, Z(n)(x) 1 and Bi = (2-&( 1), 2-'') for for x k 0, by arguments paralleling 3.28. According to 3.21, (R,S,pw -1 ) j s a g(#-1(#)) for B e S, and so by the monotone measurespace where g' -1(B)
where m
=
nln
't
+
.x
=
-
=
convergencetheorem,
-l(y) y;m
JZ(n)(A')t*'
1
-
.
f=1
zn
jydvg-t).
-1(sj)
.-.y
>'
(9. 14)
Consider first the case of non-negative g. Let 1s@) be the indicator of the set B G 0, and then if g is Borel, so is the composite function
1, glx)
(1so')@)
=
6
B
0, glx) e
B
=
1#-l(s)(x).
(9. 15)
Hence, consider the simple function (Z(,,)o#)@) =
m
77
i
1
-
2n
=1
m
(1sfo#)(x) =
By the same arguments as before, Znjog However, m
Eznjog)
=
77
j
j=1 m i
.
j
2n -
1
gn =X =1
'1'
77 f=I
i
-
2n
1
1#-1(sp@).
g, and Elznlog)
=
Eg)
=
JWg.
.j
g,(# Bi))
-
j
g,# Bi)
=
ztalllfp.'
-
1
(A),
and (9.12)follows from (9.14). To extend the result to general g, consider the non-negative g'b g separately. lt is immediate that maxf ',0J and g-
-->
(9.16)
(9. 17)
functions g'
=
Expectations E(Zn)og+4 Eznjog-) -
m
j.j
m
j-1
X =1
=
Eg-)
-
Eg),
=
(9. 18)
..j
>7 =1 2/
=
Eg')
--
of this limit separately.
so consider each component Eznlog-)
131
p,((#+) Bi)4 (9.19) jooycyg-jo),
-1
ptt# (wi ;;
2*
-..y
0
-1
-1
second equality holds because
where the g +) (Bi) g Bi) for i k g sjnce the 1 the tel'm elements of Bi are a11 positive for these cases, whereas for i disappears. Similarly, -Zn(x) 1 x for < 0, and =
=
.z'
?rj
-7r
-znjog-)
j.j
-
f=1
2/
m
i- 1
-:7
-
zn
j=1
.j
p.((#-)
Bi4)
0
j--ydv.g
-1
p,t, (#7))
--->
..j
(T),
(9.20)
1)1, and in this case the second equality holds where B; (-2-&, because (g- ) (#j) g (-Bi) j or j u z uence -2-''(f
=
-
-1
-1
=
.
Ezn,og
+) -
Ezn,og-)
j-
--
ydjw-lol,
(9.21)
and the theorem follows in view of (9.18).w The quantities F(X), for integer k 2 1, are called the moments of the distribution of X, and for k > 1 the central moments are defined by EX
-
Exlk
=
k
p
X
-
j
Eujh-kjxllk-i
(9.22)
-
A familiar case is the variance, Vartm E(X- F(m) 2 EX ) F(A') the usua j measure of dispersion about the mean. When the distribution is symmetric, with PX- F(A') e A) #(F(m X e 4) for each A e tB,the odd-order central moments are all zero. =
=
9.7 Exampl.
=
2
-
2
,
-
For the Gaussian case
(8.14),the
k'sk EX
-
g) k
=
2
kll
'
(kll)!
central
k even,
moments are
(9.23)
k odd. This formlamay be derived after some manipulation from equation (11.22) below. Vartm c 2 an d a11 the finite-order moments exist although the sequence increases monotoniclly. n =
,
Probability
132
The existence of a moment of given orderrequires the existence of the corresponding absolute moment. If FlXlP < =, for any real p > 0, X is sometimes said to belong to the set Lp (of functions Lebesgue-integrable to order p), or otherwise, to be Lp-bounded.
9.8 Example For X
N(0,c2), we have, by
-
FlXl
2(2a
=
(8.26),
y)-1/2j*.x e-xllloldx ()
(9.24)
l?2 = (2/1) c. n 'raking the corresponding root of the absolute moment is convenient forpumoses of comparison (see 9.23) and for X e Lp, the Lp-ttorm of X is defined as
(FI#I')'* 11111 p
(9.25)
=
The Gaussian disibution possesses a11finite-order moments according to (9.23), but its support is none the less the whole of R, and its p-norms are not unlrmly x, it coincides with the essential bounded. If IIXII, has a finite limit as p supremum of X, so that arandom variable belonging tofax is bounded almost surely. --
9.3 Theorems for the Probabilist's Toolbox The following inequalities for expected values are exploited in the proof of innumerable theorems in probability. The first is better known as Chebyshev inequalf@ for the special case p = 2. 's
9.9 Markov's
For er > 0 and # > 0,
inequality
#( l11 k Proof
tpp
l.Y1
Epjj
:)
=
Al
f:)
Fj aYjf
(9.26)
.
:
yvdFx) K
P
Jjzjzalxl PdFx) f E
1115.*
This inequality does not bind unless F1X1PIZP < 1, but it shows that if Fj -Yjr < x. The order of the tail probabilities converg to zero at the rate z-P as f: fv-boundedness measures the tendency of a distribution to generate outliers. The Markov inequality is a special case of (at least) two more general inequalities. =,
...-4
9.10 Corollary For any event A e F, E
Equivalently,
#((:
j
Arnt
IX()) I
Proof Obvious from 9.9.
lllkcl :1
dp fa
u
j IxI,g,. A
A) f F(1z
(9.27)
lXI#)/e'.
.
9.11 Corollary Let g: R F-> R be a function with the property that x ga) > 0, for a given constant a. Then gxj
a implies
Expectations
PX k a) <
Proof gajpx
k a4
&'(m) .
.
(9.28)
.
ga)
gajxyadFx)
=
133
jxyggxldFx)
S
K
EgT).
w
An increasing function has the requisite property for all a > 0. Let 1 ERbe any interval. A function (: I F-.: R is said to be convex on 1 if
9((1 )x +
,)?)
-
for a11 x, y e I and
.
(0,11. lf
e
(1 )(x)
f
+
-
-9
is convex
*(.y) on t # is
(9.29) on 1.
concave
9.12 Jensen's inequality lf a Borel function (1)is convex on an interval 1 containing the support of an integrable r.v. X, where ((m is also integrable,
(9.30)
9('(A')) f f@(A')). For a concave function the reverse inequality holds. n
The intuition here is easily grasped by thinking about a binary r.v. taking values with probability 1 A convex 4)is illustrated in xl with probability p and whereas F@(A')) p#(x1) + (1 This Fig. 9. 1. #(m px3 + (1 vertical chord the axis by point is mapped from F(m onto the joining and on 9, while 9(F(m) is mapped from the same point by (j itself. -p.
,:2
,-pjxz,
-p)9(x2).
=
=
-:2
.x1
Fig. 9.1 A proof of the inequality is obtaied interior of 1.
9.13 Lemma If and y e 1,
4)is convex
from the following lemma. Let 10 denote the
there exists a function A@) such that, for a11 e 10
X(-:)@
.x
-x4
f #(A') -
9(*.
(9.31)
Proof A convex function possesses right and left derivatives at all points of I0.
Probability
134
(9.29)implies
This follows because (x +
kh)
9(x)s
-
kh
(j(.7 +
for h h) h
-
>
0 that
#(x)
N s ((),jj.
'
(9.32)
+ 1/n) The sequence n e N ) is decreasing, and has a limit 9;(x). (T)(x)), In the case < 0 the inequality in (9.32)is reversed, showing the existence of 4,-/(x)as the limit of an increasing sequence. Note that ()-'@) < (J(x).Tnking the + h gives limit as 2. 4. 0 with > 0 tixed in (9.32)and y
tn(#tx
-
.z7
=
t)-'@)tJ'
S K(-:')(J'
-x4
-xt
S t)(A')
9(.1)
-
<
0 gives, for y
;(.z7)@ S #-'@)1
; th(A') t)@)
whereas the parallel argument with h -x)
-x)
F(m, y
=
X in
=
A('(m)(X-f(m) Taking expectations side has expectation
x,
(9.34)
-
lnequality (9.31)is therefore satisfied with (say)A@) Proof of 9.12 Set x
<
(9.33)
=
#+'(x)..
give
(9.31)to
#(m #(-)).
f
(9.35)
-
of both sides gives inequality
(9.30),since the
left-hand
zero. w
Next, we have an alternative approach to bounding tail probabilities which yields the Markov inequality as a corollary.
9.14 Theorem If X is a non-negative r.v. and r EX
rj-xr-zpx
r)
-
(9.36)
xvx.
>
0
Proof lntegration by parts gives, for some b joxrdFx)
0,
>
=
-
bryb) rjo
= r
-
>
0,
rjoxr-ypdx
Fpldx V--l(s(,) -
x 0
r-lp
dx.
(.x< x u
The theorem follows on letting b tend to infinity.
.
If the left-hand side of (9.37)diverges, so does the theorem is true whether or not EX) is finite.
9.15 Corollary
lf X is non-negative
J-
-xdF
-
spx k
(9.37)
light,
and in this sense the
and integrable,
o
+
j-px
> xvx.
-
(9.38)
Expectations Proof Apply 9.14 Fith r
135
1 to the r.v. 1(xk:)X. This gives
=
J- J-tlfxkslx = Jo 'dF
=
>
xvx
>
xltfx
o
$(1(xk:)X
+
j*PX
>
xldx
:
=
#(X k
E ejodx
jvP(X c
+
>
xjdx.
.
(9.39)
Not only does (9.38)give the Markov inequality on replacing non-negative X by IXI; for p > 0 and arbitral'y X, but the error in the Markov estimate of the tail probability is neatly quantified. Noting that P( IXI E) = 81.11#2 zP),
s ##( I-Yl k e)
j-
=
e
#
IXIJWF
= FIXl# where both the subtracted
j
-
j-
-
gP
P
IA-I'>
#
IXIPJF
px
-
0
I J
xldx
#(IXIP
eP
terms on the right-hand
>
xqdx,
(9.40)
side are non-negative.
9.4 Multivariate Distributions From one point of view, the integral of a function of two or more random variables presents no special problems. For example, if
gl R2
F-y
IR
is Borel-measurable, meaning in this case that g -1 B4 e Bl for every B e B, then #(#((l)),F()) is just a F/f-measurable r.v., and () =
F('(X,F))
=
Jjatldptl
(9.41)
is its expectation, which involves no new ideas apart from the particular way in which the t.v. hlk hppens to be defined. Alternatively,
the Lebesgue-stieltjes Eg-)
=
form is
jpzgx'ybdFx'L'
(9.42)
where dFxny) is to be thought of as the limiting case of &F@,yl defined in (8.30) as the rectangle tends the differential of area. When the distribution is with respect to continuous, the integral is the ordinary integral of '(x,y)J@,y) Lebesgue product measure. According to Fubini's theorem, it is equivalent to an iterated integral, and may be written +x
&'(X,l?))
=
+x
j-yqx,ytx'ybdydx. J.x
But caution must be exercised with formula
(9.43)
(9.42)because
this is not a double
Probability
136
integral in general. lt might seem more appropriate to write d lFx y) instead of JF(x,y), but except in the continuous case this would not be correct. The abstract notation of (9.41)is often preferable, because it avoids these ambiguities. In spite of these caveats, the expectation of a function of (say)X alone can in every case be constructed with respect to either the marginal distribution or te joint distribution. ,
EglxL
9.16 Theorem
jvz
#(X#F(mA')
=
jpgxjdklxj.
=
Proof Define a function +: R2 b..A (R
g
by setting g *@,y)= glx), a 11y e R :*-14#) is a cylinder in R2 with base :-1(#) e B for B e S, and g * i s Sz/s-measurable. For non-negative g, let .
m
* #(a) where m nl n+ 1 an d E i = = l-nli .j xR where Af (x: m
X
=
f=1
i
2
n
(9.44)
ls i
ln
=1
2-''(f 1) S #*(x,y) < 2-NI') e Bl Since Ei 1) K #(x) < 2-% l and tx(Af) = g,(&), -
,
-
.
=
,
j
-
j
.
(@y):
=
F(#*(,'))
j.
=
m
g,t&l
X
=
i
=1
j
-
2
n
p,x(A)
=
Elgnlj.
(9.45)
By the monotone convergence theorem the left and right-hand membrs of (9.45) *x,yldFx,y) and Egj xjdFxxj respectively. Extend converge to E(g.) non-negative general complete the from proof. * to g to =
=
The means and variances of X and F are the leading cases of this result. We also have cross moments, and in particular, the covariance of X and 1' is Cov(X,F)
=
E((X-
&m)(1'- F(F)))
=
Fubini's theorem suggests a characterization
ECXT F(mF(F).
(9.46)
-
of pairwise independence:
9.17 Theorem lf X and F are independent nv.s, Cov@(m,v(F)) of integrable Borel functions ( and
=
0 for al1 pairs
v.
Proof Fubini's theorem gives F(#(mv(F))
=
=
jp,
#@)v(y)#F@,y)
jpxtdlxjvlsbdFkl-bq
= &#(m)F(v(F)).
w
(9.47)
The condition is actually suffictent as well as necessary for independence, although this cannot be shown using the present approach; see 10-25 below.
Expectations
137
Extending from the bivariate to the general k-dimensional case adds nothing of substance to the above, and is mainly a matter of appropriate notation. lf X is a random k-vector, F(m
=
jxdFx)
(9.48)
denotes the k-vector of expectations, EXi) for 1,...,k. The variance of a scalar r.v. generalizes to the covariance matrix of a random vector. The k x k matrix, =
2 X1X2 X1
XX'
.
.
.
X3Xk
2 X2X1 X2
(9.49)
=
2
Xkxj
Xk
is called the outer product of #, and ECXX') is the k x k positive semi-detinite matl'ix whose elements are the expectations of the elements of XX'. The covariance matrix of X is Vartm
ELCX F(m(#
=
-
-
F(m)'q
=
ECXX') ExTEX4'.
(9.50)
-
is positive semi-detinite, generalizing the non-negative property of a scalar variance. lt is of full rank (notwithstanding that XX' has rank 1) unless an element of X is an exact linear function of the remainder. The following generalizes 4.7, the proof being essentially an exercise in interpreting the matl'ix formulae. Vartm
9.18 Theorem lf F= BX+c where X is an k-vector with EX4 g and Vartm = E, and B and c are respectively an m x k constant matrix and a constant v-vector, then (i) ElY) B# +r. BSB'. a (ii) Vartp =
=
=
Note that it m
>
k Vartl'l is singular, having rank k.
9.19 Example If a random vector Z (Z1,...,Zk)' is standard Gaussian (8.19),it Ik. Applying 9.18 to 0 and EZZ') is easy to verify, applying 9.17, that Ezj = the transformation in (8.35)produces F(A') g and =
=
Vartm
=
EX
-
p)(#-
g4'
=
F(AZZ'A')
=
=
AF(ZZ')A'
=
M'
=
E. n
9.5 More Theorems for the Toolbox Thefollowingcollection of theorems, together with thelensen and Markovinequalities of j9.3, constitute the basic toolbox for the proof of results in proba-
Probability
bility. The student will tind that it will suffice to have his/her thumb in these pages to be able to follow a gratifyingly large number of the arguments to be encountered in subsequent chapters. 9.20 Cauchy-schwartz
inequality
2<
Ex
with equality
attained when F
EXl)Eyl)
(9.51)
,
cX, c a constant.
=
Proof By linearity of the integral, ELaX+ F)2)
a
=
lEXl)
+ E(Yl) ZXIEIXLI
+
for any constant a. (9.51)follows on setting a equality if and only if aX+ F 0. .
=
O
-EIXYIIECXl ), an d holds as an
=
The correlation
c/elcdaf,
rxy
=
1/2 accor dingjy Cov(X,F)/(Var(mVar(D)
the interval (-1,+11. The Cauchy-schwartz inequality is a special case (forp 9.21 Hlder's p/(p
=
-
=
2) of the following.
For any p 2 1,
inequalil
11-Y11,111'11z?.
FI-YFIK where q
jies in
,
1) if p > 1, and q
Proof The proof for the case p
>
if p
x
=
1 requires
=
(9.52)
1.
a small lemma.
9.22 Lemma For any pair of non-negative numbers a,b, ab S
tzp bq
-
P
+
(9.53)
-.
q
If either a or b are zero this is trivial. lf both are oositive, let s = etlq ay = & and t = q log b. Inverting these relations gives a es y rlogtz S/PTf# d an d (9 53) follows from the fact that ex is a convex function of x, noting 3Iq 1 1/p and applying (9.29).* Proof
=
,
,
.
=
Choose a
.
,
=
-
)XI/II-Yllp,b
=
F11t?.For lr1/11
FIXFI
XIp
F
q
= Eab)
these choices, Eap)
S 1/# +
$Iq
=
=
Ebq)
1.
=
1, and
(9.54)
S Fl Xl ess sup F, which holds For the case p = 1, the inequality reduces to FI-YFI since F < ess sup i' a.s., by definition. w
The
lder inequality spawns a range of useful corollaries and special cases, includingthe following.
9.23 Liapunov's
inequality
Proof Let Z F:zlXIF, F = 1, s
(norm inequality) =
r/p. Then,
If r > p > 0, then
(9.52)gives FIZFIf
jlxllr I1-YIlp.
or IIzII,II FIIs(,-1),
Expectations E ( 1X 1#) < E 111
pslls
=
rfr
I11
E
(9.55)
.
.
This result is also obtainable as a corollary of the Jensen inequality.
9.24 Corollary
For each A e
jg IXI'IdP Proof ln
and 1/p + jIq
.
1/p
Jx
p
l71 PdP
<
1,
=
Jx
1Iq
11'IqdP
X by Xlx and i' by i'1x. x
(9.52),replace
probabilistic in character.
Alternative variants of the result are not explicitly
9.25 Corollary n
Let x1,...,x,, and y1,...,yn be any sequences n
77 izz1
1/,
77 1 izzz
lxafl S
771 IyfI
Then
for 1/p + Iq
=
1. n
fzz
be Lebesgue-integrable
functions of a real
'''-(J-
1/,+
(j-
I?f)I,z,)
'-ftkbgtkbkdt s
of numbers.
1/<
n
1x/1J'
and gl
9.26 Corollary Let fl variable. Then
J-
.
I,,(,)l-?,)''
or
lq
-
1. a
Proofs are left as an exercise. The sequences in 9.25 and the functions in 9.26 can be either real or complex-valued (see jl 1.2).
9.27 Minkowski's Proof For r
=
inequality
For r k 1, IIX+ i'llr <
i'llr. + 11 IIXIIr
1 this follows direct from the triangle inequality,
IX+ i'I < I-YI+ I1'l on taking expectations. For r
>
(9.56)
,
1, note that r-1)
FIx+ FIr F(I#+ FII-Y+ FI S E IXI+ IFI)IX+ l'I ) r-1). r-1) + E Irl I x+ I'I = E IxIIX+ l'I =
r-1
inequality to the two right-hand-side
Applying the Hlder
'
E IX+
FI
<
r)1-1/r.
+ 11 X+ l'I FIIr)(FI (II#IIr
Cancelling A'IaY+FI and rearranging r
application to the sum of m variables, generalizes directly to K r
terms yields
(9.58)
gives the result. w
By recursive
Yxi f=1
(9.57)
77Ilmllr =1
for r k 1. For an intinite series pne can write
the Minkowski inequality
(9.59)
Probability
140
77m =1
IImlIr+77xi X i=?n+1 f=1
:
r
If
0 as IIZ7=,,,+1XjIIr --
m
it is permissable to conclude that
tx.
--
(9.60) r
o/
X
i= 1
xi s
f=1
r
IIx)Ilr,
(9.6 1)
not ruling out the possibility that the right-hand side is infinite. 9.28 Love's
cr inequality
For r Fq
e' where cr
=
Txi
>
r
m
s
f=1
1 when r S 1, and cr
0,
=
r, cr XA'1 Xi I
(9.62)
i=1
m r-l wj)en r
j
.
Proof This goes by proving the inequality m
r
m
<-
ai /=1
cr f=1
Iai Ir
(9.63)
for real numbers JI,...,tza, then substituting random variables and taking expectations. Since 1Z0,zzkai lr S (E't', Iai I)r, it will suffice to 1et the ai be non-nega1, tive. For the case 0 < r S 1, 0 f zi f 1 implies z; k zi and hence if Z'?, ctajj. XTt k 1. (9.63)follows on putting zi = ai /(Z0./ For r k 1, on the other hand, convexity implies directly that .1
xlzf
=
=lz(,
j -
m
X icz:l
r
ai
K
-
1
m
XX.
*
izz,1
9.29 Theorem If X, 1', and Z are non-negative r.v.s satisfying X f tz(F+Z) a.s. for a constant a > 0, then, for any constant M > 0, FtltxyvlA')
f
zJtftlf
y>M/2c)F)+F(1(z>ArJw)Z)).
(9.64)
Proof lf we can prove the almost sure inequality 1(x>M)X S 2J(1ty>&z/2u)F+1(z>M/2u)Z),a.s.,
(9.65)
the theorem will follow on taking expectations. 1(x>M)Xis the r.v. that is equal to X if X > M and 0 otherwise. If X K M (9.65)is immediate. At least one of the inequalities F k X/M, Z k Xlla must hold, and if X > M, (9.65) is no less obviously true. w
9.6 Random Variables Depending on a Parameter Let G(,0): f x 0- F-> R, O i R, denote a random function of a real variable 0, or n other words, a family of random variables indexed on points of the real line.
141
Expectations The following results, due to Cramr nated convergence theorem.
easy consequences of the domi-
(1946),are
9.30 Theorem Suppose that for each (l) e C, with #4C) 1, G4,0) is continuous at apoint %,and IG((t),0) I < F((t))for each 0 in an open neighbourhood No of % where =
F(i')
<
Then
x.
lim EG(44
(9.66)
F(G(0o)).
=
8-+%
Proof Passage to a limit 00 through a continuum of points in 0- as indicated in (9.66), is implied by the convergence of a countable sequence in 0- Let (0v,v e NJ be such a sequence, in No, converging to 0o. Putting Gv((t)) G(,0v) defines a countable sequence of r.v.s. and limsupvGvt) and liminfvGvt) are r.v.s by 3.26. By continuity, they are equal to each other and to G(,%) for (l) e Q in other G(%) a.s. The result follows from the dominated convergence words, G(0v) ,
.
=
-->
theorem.
.
9.31 Theorem 1f, for each
)
e C with #(C)
G(),%
+
h4
1, (JG/J0)((t)) exists at a point
=
G(,%)
-
n((o)
< for 0 f
< Jl1, where F(l'1)
KdoEa)
<
j
(x,
and
is independent of
l
ttjG()..)
-
(.,
I
e.
% and
),
then
(9.67)
.
Proof The argument goes like the preceding one, by considering a real sequence (v ) tending to zero through positive values and hence the sequence of r.v.s (SVJ where Hv gG(0o+/lv) G(%)1/v, whose limit H S(0()) exists by assumption. . =
=
-
The same sort of results hold for integrals. Fubini's Theorem provides the extension to general double integrals. The following result for Riemann integrals on intervals of the line is no more than a special case of Fubini, but it is useful to note the requisite assumptions in common notation with the above.
9.32 Theorem Suppose thatforeach (l) E C, withf'to a t'initeopen interval atb), and IG(,0) l < 1'2() for
=
1, G(,0) is continuous on 0 < b, where F(F2) < x.
q <
Then
s(js(e)ts)
J-s(c(e)):e -
lf J7'IG(,0) lJ0 < 1'3(t,))for (t) E C and Ek) and b +=. of a =
-x
(9.68)
.
-
<
cx),
(9.68)holds
for either or both
=
Proof For the case of t'initea and b, consider Hlsb,tl JJG(,0)#0. This has the I lG(),f) I < F2(), for each propertiesIH% I < b J)F2(t,)), and I dHld EH) exists and by /, each 9.31, for t e a, b4 Hence, =
-
.
=
Probability
l42
d EH) E(Gt))
is continuous
=
E
dH
=
dt
EGtj).
on a,b4 by the a.s continuity
a(,) -
jtEol4d
(9.69) of G and 9.30, and hence
-s(s(?))
(9.70)
J
is differentiable on (a,b4,and dhldt = 0 at each point by (9.69).But by defi0, and hence hb) 0 for e C, so that A(J) 0 which is equivnition, H,a) alent to (9.68). =
=
=
JX
G(,0)#0 exists and is Under the stated integrability condition on G(,0), = J!xG(t0,0)#0 is well defined and has an expectation for finite on C. Hence Htt) and and/or b = x. > th argument above goes through with a = all t e R, -X
-.x
10 Conditioning
10.1 Conditioning in Product Measures It is difficult to do justice to conditioning at an elementary level. Without resort to some measure-theoretic insights, one can get only so far with the theory before nznning into problems. There are none the less some relatively simple results which apply to a resicted (albeitimportant) class of distributions. We approach, and so demonstrate the diftiintroduce the topic by way of this arise, before going culties that on to see how they can be resolved. In the bivariate context, the natural question to pose is usually: we know X what of lX,l') F?' For a random real pair is the best predictor on (,T, #) we = x, class conditional evidently define of functions for distribution j7.2) a (see can K For any A e B such that PX e A) > 0, let 'naive'
dif
F@ lX
G
A)
PX e A, F < y) PX s A)
=
( 10. 1)
.
This corresponds to the idea of working in the trace of (f1,T,#) with respect to A, once A is known to have occurred. Proceeding in this way, we can attempt to construct a theory of conditioning for random variables based on the c.d.f. We may tentatively define the conditional distribution function,F(.y Ix),when it exists, as a mapping from R2 to R w hich for tixedx q R is a non-decreasing, right-continuous function of y with F(-x lx) 0 and F(+x lx)= 1, and which for fixed y e R satisties the equation =
Px
e
,4,
J' < y)
jgFl.y
=
Ix)#Fx(x)
(10.2)
in y-space for any A e B (compareRao 1965: j2a.8). Think of the graph of F(.yIx) through the sudace of the joint distribution funcas the protile of a tion, parallel to the y-axis, at the point x. However, much care is needed in interpreting this construction. Unlike the ordinm'y c.d.f., it does not represent a probability in general. If we try to interpret it as P(Y S y IX x), we face the possibility that P(X x) 0, as in a continuous distribution. Since the integral of F@ lx) over a set in the marginal distribution of X yields a probability, as in (10.2),it might even be treated as shows that we would need a type of density function. Taking A (.Y f islice'
=
=
=
.x)
=
,(x,y)j-,'(,I()#,k(t) J-j' g,'(vI():,'x(t) -
-
ex
ex
ex
(10.3)
Probability
144
to hold. Since F(x,y) is an integral over R 2 F u bini's theorem implies that F@ 1x) is well detined only when the integrals in (10.3)are with respect to a product ,
meastlre.
If Xand Fare independent we can say unambiguously (butnot very usefully) that FLyjx) Fy%). F(? j.x)is also well defined for continuous distributions. Let Sx denote the support of X (the set on which fx > 0); the conditional p.d.f. is =
J# Ix)
fx,yj .fx@)
=
,
.x
e
sx,
(10.4)
where jxx) is the marginal density of X. We may validly write, for A e B
px
e
x, r s
jgj'
.ftvlo/'axltxo
y)
-
-
f''h
Sx,
j z-twylxlyxt.xlts (10.5) #
-x
where F(..lxl
j' Avlxlfo.
-
(10.6)
-K
The second equality of (10.5)follows by Fubini's theorem, since the function fxny) is integrated with respect to Lebesgue product measure. However, (10.6) appears to exist by a trick, rather than to have a firm relationship with our intuition. The problem is that we cannot work with the trace (A,Ta,#a) when A = (: X()) xl and #(A) 0, because then #a = #/#(A) is undefined. lt is not clear the case when (,Y = x) has occurred' when this event what it means to fails to occur almost surely. Except in special cases such as the above, the factorization dF(y,xl = dF@ Ix)#Fx@) is not legitimate, but with this very important caveat we can define the mean and other moments of the conditional distribution. The conditional expectation of a measurable function #(-Y,i'), given X = x, can be defined as =
=
Ex,n
Ix)
+*
J
=
glx-ybdFytxq,
(10.7)
-X
also written as Elgx,Y) lX = #. The simplest case is where g(X,j') is just F. f(l'Ix) is to be understood in terms of the attempt of an observer to predict F after the realization of Xhas been observed. When Xand Fare independent, &F1x) where EY4 is the ordinary expectation of F, also called the marginal or = E, unconditional expectation. ln this case, the knowledge that X x is no help in predicting K =
10.1 Example These concepts apply to the bivariate Gaussian distribution. From (8.36), the density is -1
fx,yj
1
=
2a
C11 C12
c12c22
j/2
exp
,r
-
a
611 C12
z't.x g.1,y g,21 C12 -
-
C22
/1
X-
F
-
/2
Conditioning
145
(A' g,2) -
-
1
= 2J(cl
1c22
2 cl2 g,1) (.:7 c1l (x -
-
-
c212)
exp
jzl) 2
-
-
l/2
2 c22
zcjj
c212
,
(10.8)
-
c11 the square in the exponent.
where the last equality is got by completing Evidently, fxky) fy Ixjfxx) where =
(A,
p,2) -
-
1I)
fl.y
1
=
2 WIVc2a c12 c11
exp
ja
c12 c11
2
'--fx
-
-
2 c22 -
-
c21 a
M,1)
(10.9)
c11
and fxxj
l exp 2zrc11
=
-
@- B1)1
zo.11
.
(10.10)
Thus, F(Flx)
(T12
----t.r
=
g,2 +
-
c22
p,1),
(10.1 1)
and VartFlxl If c1a
of the
=
0,
fy lx) reduces
marginals,
=
622
-
c212 .
c11
(10.12)
so that the joint density is the product and x and y are independent. n to
fvy),
10.2 Conditioning on a Sigma Field In view of the limitations of working directly with the distribution of (aY,l'), we pursue the approach inoduced in j7.2, to represent partial knowledge of the distribution of F by specifying a c-field of events N g 5 such that. for each G e N, an observer knows whether or not the realized outcome belongs to G. The idea of knowing the value of a random variable is captured by the concept of subfield measurability. A random variable .X()): fl F-> R is said to be measurable with respect to a c-field V c 5 if x-2B)
=
(:
.Y4(0)G #) e V, for al1 B e f.
(10.13)
The implication of the condition V c 5 is that the r.v. X is not a complete
Probabilit.y
representation of the random outcome ). We denote by c(m the intersection of all c-fields with respect to which X is measurable, called the c-fild generated by X.
Jf, on being confronted with the distribution of a random pair (X()),1'())), we learn that X we shall know whether or not each of the events G e ctm has by determining whether X(G) contains x. The image of each G e c(A3 occurred under the mapping .x,
=
(X()), F()))2 D
F--> R
2
is a cylinder set in R 2 an d the p.m. defined on c(A') is the marginal distribution of X. .
10.2 Example The knowledge that xl fk X ; can be represented R). R c(f(-x, x1: x < xl ), ((x, x): x > .x2
by
.x2),
=
Satisfy yourself that for every element of this c-field we know whether or not X belongs to the set; also that it contains all sets about which we possess this knowledge. The closer together a'1 and are, the more sets there are in R. Wben +x, lO,R 1. n J4 = 5 ctm, and when xl .v1 x2, R .x2
=
=
=
-x,
.:2
=
=
The relationships between transformations and subfield measurability are summarized in the next theorem, of which the first part is an easy consequence of the definitions but the second is trickier. lf two random variables are measurable with respect to the same subfield, the implication is that they contain the same information; knowledge of one is equivalent to knowledge of the other. This means tha t evety Borel set is the image of a Borel set under g- This is a stronger condition than measurability, and requires that g is an isomorphism. It suffices for g to be a homeomorphism, although this is not necessary as was shown in 3.23. .
10.3 Theorem Let X be a r.v. on the space (S,fs,p), and let i' gl S F-+ 'T is a Borel function, with S i R and 'T i R. (i) ctl') c ctm. (ii) c(1?) ctm iff g is a Borel-measurable isomorphism.
=
glX) where
=
Proof Each B e f.r has an image in fs under g -1 w hich in turn has an image in c(m under X -1 This proves (i). To prove (ii), define a class of subsets of S, C (.g-1(#): B e B v.). To every -1 A S there corresponds (sinceg is a mapping) a set B 'T such that A = g B), and making this substitution gives ,
.
=
fs
lz4:
'(z4)
e
.r 1
fp-j (#): gg
=
-1
(#)) e s v l
=
g
jj(;
,
.
j4;
where the inclusion is by measurability of g -1 an d the second equality is because -' = gg (#)) B for any B c 'T, since g is 1-1 onto. It follows from (10.14)that ,
fx -1( s -4)
j:
z4
e ls
)
g
(x-1(:-1(,)) c
f: B s f.y).
(10.15)
If F is V-measurable for some c-field V 5 (suchthat V contains the sets of the right-hand member of (10.15)), then X is also V-measurable. ln particular, ctm ctl?). Part (i) then implies c(m = c(F), proving sufficiency of the conditions. = To show the necessity, suppose first that g is not 1-1 and glxL) = y '(.n)
Conditioning
(x1) and (x2)are elements of Ss but not of C, which contains only g -1(fy)) fxl) k.p (x2) Hence : Bs C, and 3 a Os-set X for which G there is no Or-set B having the property g -1(B) A This implies that (say)for xl
y:
The sets
.n.
=
.
.:-1(4)
=
.
c(F), so that c(F) c ctm. We may therefore assume that g is 1- 1 If g is not Borel-measurable, then by C; and again, definition 34 = g -1(B) G B s suc h thatgtA) =B e tB.r,and henced Ss C, so that c(l') c ctm by the same argument. This completes the proof of
ctm but
-1
.
necessity.
.
We should briefly note the generalization of these results to the vector case. A random vector #4): f F-> Rk i s measura b1e with respect to 97 c 5 if X -1 B4
=
f
:
A)l
e #)
G
s vs ,
s pk
(1().j6)
.
If ctAr) is the c-tield generated by X, we have the following result.
10.4 Theorem Letfbe $ c R an d Bks fl =
F
k arandom vector on the probability space (S,fs,p) Sk) and consider a Borel function s: B e
f-h
,
g(m: S
=
w jaere
F--
1', T' g R2'.
(10.17)
(i) G'(1') l c(A'). (ii) If m k and g is 1-1 with Borel inverse, then c(1') =
=
ctm.
Proof This follows the proof of 10.3 almost word for word, with the substitutions of fs k an d Bkr for Ss and S.r, X and F for X and F, and so forth. .
10.3 Conditional Expectations Let i' be an integrable r.v. on (f,T,#) and N a c-tield contained in F. The term conditional expectation, and symbol F(FIN), can be used to refer to any integrable, N-measurable r.v. having the property jgFtrl
U)dP
=
F(FI joj'dp
G)#(G), all G e N.
=
(10.18)
Intuitively, F(FIV)()) represents the prediction of F() made by an observer having information V, when the outcome (.t) is realized. The second equality of (10. 18) supplies the definition of the constant S(l'l G), although this need not exist unless #4G) > 0. The two extreme cases are F(FI@) i' a.s., and F(Jrl 5) EY) a.s., where T denotes the trivial c-field with elements (f,0 ). Note that f s 5', so integrability of F is necessal'y for the existence of F(FIV). The conditional expectation is a slightly bizarre construction, not only a r.v., but evidently not even an integral. To demonstrate that an object satisfying (10.18) actually does exist, consider initially the case F 0, and define =
v(G)
=
JgF##.
10.5 Theorem v is a measure, and is absolutely
=
(10.19) continuouj with respect to #.
Probability
148
0. lt remains to show Clearly, v(G) 2 0, and #(G) = 0 implies v(G) is a disjoint sequence, then countable additivity. If ll Proof
=
vUb
v,o,Ydp
= *'
j
J
=
J
X j
o,Ydp *'
-
'
Xvt(l,
(10.20)
./
where the second equality holds under disjointness.
.
So the implication of (10.18)for non-negative F turns out to be that F(FIT) is the Radon-Nikodym derivative of v with respect to #. The extension from nonnegative to general F is easy, since we can write F F+ F- where F+ and Yare non-negative, and from (10.18),F(Fl V) F(F+IV) E(Y- IN), where both of the right-hand r.v.s are Radon-Nikodym derivatives. The Radon-Nikodym theorem therefore establishes the existence of F(FI5:); at any rate, it establishes the existence of at least one nv. satisfying (10.18).It does not guarantee that there is only one such r.v., and in the event of nonuniquenes, we speak of the different versions of F(l'IV). The possibility of multiple versions is rarely of practical concern since 10.5 assures us that they are al1 equal to one another a.s.g#j, but it does make it necessary to qualify any statement we make about conditional expectations with the tag a.s.', to indicate that there may be sets of measure zero on which our assertions do not apply. ln the bivariate context, F(Fl c(m), which we can write as FtFlm when the context is clear, is interpreted as the prediction of F made by observers who observe X. This notion is related to (10.7)by thinking of F(l'l x) as a drawing from the distribution of F(FIm. =
=
-
-
10.6 Example ln place of (10.11)we write &Flm(t,))
61 2
=
---(X() p,2 + C1 1
-
p.1),
(10.21)
which is a function of X(l, and hence a r.v. defined on the marginal distribution of X. F(FIm is Gaussian with mean g,2 and variance of c2la/cll. E! Making
F(FIx)a point
in a probability space on R circumvents the difficulty
previously with conditioning on events of probability 0, and our is valid for a11distributions. lt is possible to define F(FIG4 when
encountered construction #(G) 0. What is required is to exhibit a decreasing sequence fGn e N) with #(Gn) > 0 for every n, and Gn 1 G, such that the real sequence (F(1'1 Gn)) converges. This is why (10.7)works for continuous distributions. Take Gn gx,x + 1/p1XR s =
=
c(A'), so that G
=
Using
(x)xR.
(10.4)in (10.18),
+co f*x+1/n
f(rl Gn)
J..J
yfl,ytdldy
+x
x
=
--
+.
ujtn
uJ-?'(''.''vjvy
j-giy'xsdy =
&Fl*,
(10.22)
Conditioning x. Fubini's theorem allows us to evaluate these double integrals one as n dimension at a time, and to take the limits with respect to n inside the integrals with respect to y. --->
Conditional probability can sometimes generate paradoxical results, following case demonstrates.
as the
10.7 Example Let X be a drawing from the space (g0,11, p,IJ, m), where m is Lebesgue measure. Let N c fp,1) denote the c-field generated from the singletons fxl, x e (0,1). A1l countable unions of singletons have measure 0, while all complements have measure 1. Since either #(G) 0 or #4G) = 1 for each G e V, it is clear from (10.18)that F(XIN) F(m a.s. However, consider the following weknowwhetheror notxc Gforeach Ge V, weknow argument. 'Since fx) e N, if x. ln particular, V contains knowledge of the outcome. It ought to be the case that F(XIV) X a.s.' n =
=
=
,
=
The mathematics are unambiguous, but there is evidently some difficulty with the idea that V should always represent partial knowledge. lt must be accepted that the mathematical model may sometimes part company with intuition, and generate paradoxical results. Whether it is the model or the intuition that fails is a nice point for debate.
10.4 Some Theorems on Conditional Expectations 10.8 Law of iterated expectations
(LlE)
FgF(J'IN)j Proof lmmediate from
(10.23)
Ek.
=
(10.18),setting
G
=
D.
.
if constant' under The intuitive idea that conditioning variables can be held the conditional distribution is confirmed by the following pair of results. 'as
10.9 Theorem lf X is integrable and V-measurable, then Proof Since X is N-measurable, E
+
then
jyydp
-
=
1): X()
>
F(XIN)()1
F(.YI jg-yxf jyjxN)J#
This contradicts (10.18),so PE*) = (tt):X((t7) < #(xl V)(@) E V.
N))J#
=
=
F(#IN)
>
=
X, a.s.
e N. lf #(F+)
0.
0. By the same argument, P(E-)
>
0,
(10.24) =
0 where E-
*
10.10 Theorem lf Fis F-measurable and intepable, X is V-measurable for V g @, N) a.s. < x, ten &FaY1T) and E IA-FI -YSIFI =
Proof By definition, the theorem follows if %)dP xYFIFI
Let
xn)
=
o
'z-lajlsj
=
J s
XYdP a.s., for al1 G e T.
(10.25)
be a V-measurable simple f.v., with F1,...,Fn a pmition of
Probability
150
f and Ei e V for each i. (10.25)holds for X
X(n) since, for all G e V,
=
jgxtnwtyl cldp nwj ac, s(I,Itvp x -
i= 1
=X-
.ij
yw,
-
GIAEt'
/=1
j x-,vdp,
(10.26)
G
Ei e N when G e V and Ei e V. Let X k 0 a.s., and let (X(,,)J be a monotone sequence of simple N-measurable converging XY a.s. and IX(n)F( K lXFI functions to X as in 3.28. Then XnlY N) where F1 XFI < x by assumption. Similarly, X(n)F(FI #FIl'IV) a.s., and
noting G
fa
,--+
,
--
e-l-Ytnlftrl N) I
=
1-Y(n)1'l I9-))
F1 F(A)n)FI N) I S EE
= FlaYtall'l
e-IA'l'I<
<
(10.27)
x,
where the first inequality is the conditional modulus inequality, shown in 10.14 below, and the second equality is the LlE. It follows by the dominated convergence JcxftFl N)##, and so (10.25)holds for non%)dP theorem that Jc#(n)F(rI negative X. The extension to general N-measurable X is got by putting -->
x where X*
=
maxfmo)
eFXl 9')
=
=
X+ -X-
(10.28)
2 0 and A'- 2 0, and noting
EIYX'P
= X*
-
-
FX- IN)
X-)F(l'IN)
=
ECYX' IN) EYX- lN) -
XFIFIV) a.s.,
=
using (10.33)below and the result for non-negative X.
(10.29) .
X does not need to be integrable for this result, but the following is an important application to integrable X.
10.11 Theorem lf F is F-measurable and integrable and F(rl V) El') for N c @, then Covtmp = 0 for integrable, V-measurable X. =
Proof From 10.8 and 10.10,
ECXYI #(A'(A-FlN)) =
If
F(FIV)
=
EY) a.s.
(a constant),
=
Ft-tftrl
V)J.
= then ECXYI EkEr).
(10.30) w
Note that in general Covt#,F) is defined only for square integrable r.v.s X and K But Cov(X,Y) 0, or ECXYI= FtmFtF), is a property which an integrable pair can =
satisfy.
The following is the result that justifies the characterization of the conditional mean as the optimal predictor of J' given partial information. optimal'is sen to have the specific connotation of minirnizing the mean of the squared
prediction errors.
Conditioning 10.12 Theorem
.:
t $... .(y:,.. y... ).)... ../.
g
Let
denote any V-measurable approximation to F. Then < (1 Y- f'112. jlY- F(rI N) 112
.j;
-
...
151
(10.31)
-
. .. .. .-.
Proo
f (F-
hence
f)2 (F-F(FIN)j2+2(y-F(Fj f)2IT)
F((F-
=
Fg(F-F(Fj
a,q-
V)jgF(J'l T)
=
N))2jNj + (F(yl N)
-
a..v
Fj
-
+
gF(FIN) F)2, and
f12a.s.,
-
(10.32)
noting that the conditional expectation of the cross-product disappears by definition of F(1'1 V), and 10.10. The proof is completed by taking unconditional expectations tluough (10.32)and using the LIE. w The foregoing are results that have no counterpart in ordinary integration theory, but we can often exploit the fact that the conditional expectation behaves expectation, apart from the standard caveat we are dealing with like a nv.s so that diferent behaviour is possible on sets of measure zero. Linearity holds, for example, since %real'
hFIN)
EaX+
=
aExt T) +
:FIFIT),
(10.33)
a.s.,
is a direct consequence of the detinition in (10.18).The followipg are conditional versions of various results in Chapters 4 and 9. The first extends 4.5 and 4.12. 10.13 Lemma V) 0 a.s. (i) If X 0 a.s., then F(aYI (ii) If X f F a.s., then F(.Yl T) S F(Fl V) a.s. (iii) If X F a.s., then F(aYlN) F(FIV) a.s. =
=
=
=
Proof (i) follows diretly from (10.18).To prove (10. 18) and 4.8(i) together imply jaExk
for a11G e V. SinceA
N)##
=
(ii), note that the hypothesis,
jaxdp hk'dp S
=
JgF(Fl
G4dP
T)((t))> 6(FIT)((t))1e V, itfollows f: F(xYI
that #(A) = 0. The proof of (iii) uses 4.8(ii), and is otherwise identical to that of (ii). w
10.14 Conditional
=
IF(FIT) I f
modulus inequality
E Il'l IT) a.s.
Proof Note that IFI F+ + Y-. where F+ and Y- are defined in (10.28).These are non-negative nv.s so that F(F+l N) > 0 a.s. and EY- j N) 0 a.s. by 10-13(i) 1, and (ii). For ) e C with PC4 =
=
lF(F+
-
Y- lN)()1
=
IF(F+l T)()
-
< F(l'+IT)((o) + =
Ev+w y-
/(F-1 97)())1 EY-
I )(),
where both the equalities are by linearity.
.
lN)() (10.34)
Probability
10.15 Conditional monotone convergence theorem lf L f F and L 1- 1' a.s., then F(FnI5') 1' F(1'1 N) a.s. Proof Consider the monotone sequence Zn L K Since Zn f 0 and Zn S Zn+I, 10.13 implies that 4he sequence Ezn lN) ) is negative and non-decreasing a.s., and hence converges a.s. By Fatou's Lernma, =
-
ja limsup Ezn N) dP limsup jaE(Zn U4dP k l I n'-/oo
n'->
= limsup
Zndp
=
G
N-->=
(10.35)
0
for G e V, the tirst equality being by (10.18), and the second by regular monotone convergence. Choose G = f : limsupaAzn lT)() < 0J which is in N by 3.26, and (10.35) implies that #4G) 0. It follows that ,
=
V) lim F(ZnI
0, a.s.
=
(10.36)
.
n'-co
10-16 Conditional
Fatou's
liminf
lemma If Fn k 0 a.s. then
&FnIV) k
E liminf L IN a.s.
influnrk so that l%' is non-decreasing, and converges to F = EY' IW) by 10.15. F,, k Fr, and hence E liminfarr,. Then E'n' 5') l5>)1 E'n' lT) a.s. by 10.13(ii). The theorem follows on letting n . Proof Put
L'
(10.37)
n-'+oo
n-oo
=
--+
--)
,.
Extending the various other corollaries, such as the dominated convergence theorem, follows the pattern of the last results, and is left to the reader.
10.17 Conditional
Markov
inequality
#(1IFl
P E.l IT) S
F(l
rlpl T) EP
,
a.s.
Proof By Corollary 9.10 we have
s'jolfIrlkcll' By definition, for G e N,
jopf I1'l
f
Jgl
Y'PdP, G e @.
k El I*4dP
and jaEf
FIPIUtdp
=
=
Jclt
ll'lk.:)tf#,
Jcl
YPdP.
Substituting (10.39)and (10.40)into (10.38),it follows that
(10.38)
(10.39)
(10.40)
Conditioning JcEEf#(t
l1'I 2
El
153
I5') -F(I F1#1U4dp
(10.41)
f 0.
The contents of the square brackets in (10.41)is a V-measurable r.v. Let G e N 0. . denote the set on which it is positive, and it is clear that #(G) =
10.18 Conditional Jensen's inequality Let a Borel function () be convex on an interval 1 containing the support of a i-measurable r.v. F where F and (/(J') are integrable. Then
4'tftrlT))
< F@tl'')l
Proof The proof applies 9.13. Setting x
N), a.s.
(10.42)
F(l'IV) and
=
y
J' in
=
T)) < :(F) -#(F(1'IN)).
A(F(1'l V))(F-F(Fl
(9.31), (10.43)
V)) is a random variable. It is not certain that However, unlike A(F(D),A(F(FI the left-hand side of (10.43)is integrable, so the proof cannot proceed exactly like that of 9.12. The extra t'rick is to replace F by 1sF, where E ftl): F(FIN)() f #1 for B < x. F(Fl V) and hence also ls are V-measurable random variables, so F(1sFIT) = 1sF(l'IV) by 10.10, and =
N) e'@(1sl')I
=
= Thus, instead of
F(9(F)ls+ lsfttF)l
#(0)1rIT) V)
(1
+
-
1s)#(0).
(10.44)
(10.43),consider
A(F(ls1'IT))(1sF-
lsF(l'l V)) k #(1sF) -#(1sF(l'l
T)).
(10.45)
The majorant side of (10.45)is integrable given that (/(i') is integrable, and hence so is the minorant side. Application of 10.9 and 10.10 establishes that the conditional expectation of the latter term is zero almost surely, so with (10.44) we get
#(1sF(l'IV)) Finally, 1et #
--
x
< 1sF@(1?)l V)+
so that ls
-->
(1
-
(10.46)
1s)9(0), a.s.
1 to complete the proof.
.
The following is a simple application of the last result which will have use
subsequently.
10.19 Theorem Let X be V-measurable and fv-bounded for r k 1. If F is F-measurable, X+ F is also Lr-bounded, and F(Fl V) 0 a.s., then =
FIxY+ FIr k
FI-YIr.
(10.47)
Proof Take expectations and apply the LlE to the inequality F(1-Y+ FIr! N) 1
IEX+
irj N)Ir
=
lx1r
a.s.
.
(10.48)
Finally, we can generalize the results of j9.6. It will suffice to illustrate with the case of differentiation under the conditional expectation.
154
Probability
10.20 Theorem Let a function G(,0)
e'(-J I I
r)
dedl 9'' -
:-o
E
-
G(%)
hv
T
I
8=00
(v, v e
Proof Take a countable sequence of the conditional xpectation, G(0() + h
satisfy the conditions of 9.31. Then
F(G(%
(Nl with hv +
v)
=
lV)
(10.49)
-.s.
,
-
--
0 as v
x.
--
(G(%) IT)
By linearity
(10.50)
a.s.
hv
If Cv 6 V is the set on which the equality in (10.50)holds, with #(Cv) 1, the #(OvCv) a nd limit 1 by 3.6. The in the two sequences agree on the set Ovcv, left-hand side of (10.50)converges a.s. to the left-hand side of (10.49)by assumption, applying the conditional version of the dominated convergence theorem. Since whenever it exists the a.s. limit of the right-hand side of (10.50) is the right-hand side of (10.49)by definition, the theorem follows. . =
=
10.5 Relationships between Subfields TI
5 and Mzc 5 are independent subfields if, for every pair of events Gj e V1 and Gz s N2, #(Gj r7 G
=
#(G1)#(Gc).
(10.51)
Note that if F is measurable on Vl it is also measurable on any collection containing V1,and on ; in particular. Theorems 10.10 and lo-llcover cases where F as well as X is measurable on a subfietd.
10.21 Theorem Random variables X and F are independent iff ctm and ctF) are independent. Proof Under the inverse mapping in (10.13),G1 e c(m if and only if #1 X(G1) i T with a corresponding condition for c(F). It follows that (10.51)holds for each G1 e c(m, G1 G c(F) iff PX* Bj, i'e Bz4 PX e #1)#(Fe Bz) for eachfl X(G1), Bz F(G2). The only ifo of the theorem then follows directly from the follows, given (8.38),from the fact that every Bi e definition of ctm. The B has an inverse image in any subfield on which a r.v. is measurable. w =
=
=
=
'if'
The
only ifo in the first line of this proof is essential. Independence of the subtields always implies independence of X and F, but the converse holds only for the infimal cases, c(m and f5(1').
10.22 Theorem Let F be integrable and measurable on V1. Then 5'I a.s. for a11 V independent of
F(FIV)
=
EY)
.
Proof Define the simple Vl-measurable r.v.s l'(n) G11,...,G1s of ( where G1j e V1, each i, with l%)
On a partition = Z'lzzlyflclf 1 F as in 3.28. Then
Conditioning
jynlb
5')J#
n
X'faFtltyl/l
=
f=1
=
*'
vl,
%''
#(G)XX#(G1f)
-
155
ynyjacjf
c)
ra
=1
V,
all G e
fOr
PG4EYn)4
=
(10.52)
f=1
F(i') by the monotone convergence theorem. F(F(n) IN) is not a simple function, but EYn) IV) F(Fl V) a.s. by 10.15, and
Ejnj)
--
'1'
joEn)
IVV# --
VV#' jcfk''
(10.53)
convergence. Hence for Vl-measurable F,
by regular monotone
JgFIFIM4dP
=
for a1l G e V.
PGjEY)
(10.54)
Frop the second equality of (10.18) it follows that F(FIG) which proves the theorem. w
10.23 Corollary
If X and F are independent, the
Proof Direct from 10.21 and 10.22, putting V
=
&FI
EY4 for a11 G e V,
=
.)
=
E').
ctm and V1
=
c(F).
.
10.24 Theorem Apairof c-fields V1cF and Vac 5 areindependentiff Cov(X,D 0 for every pair of integrable r.v.s X and F such that X is measurable on V1 and F is measurable on N2. =
Proof By 10.22, independence implies the condition of 10.11 is satisfied for N tif' consider X ito To prove lcj, G1 e V1, and F 1cz for V1, proving G2 e N2. X is Nl-measurable and F is Na-measurable. For this case, =
Cov(X,D
=
Cov(X,D =
=
,
.
#(GI
fa
=
G2) #(Gl)#(G2).
(10.55)
-
0 for every such pair implies V1and V2 are independent by (10.51).w
10.25 Corollary Random variables Xand F are independent iff Cov@(m,v(F)) 0 for every pair of integrable Borel functions 4)and v.
=
Proof By 10.3(i), 9(A') is measurable with respect to c(m for al1 and v(F) is ctpimeasurable for a1l v. If and only if al1 these pairs are uncorrelated, it follows by 10.24 that c(A3. and cti') are independent subfields. The result then follows by 10.21. w ,
An alternative proof of the necessity part is given in 9.17. The next result generalizes the law of iterated expectations to subfields. We say that c-fields V1 and V2 are nested if V1 c V2.
10.26 Theorem lf N1 i Va @, then for T-measurable F, N2)1N11 F(l'l N1) a.s. (i) EEF(l'I N1)1T21 F(l'IT1) a.s. (ii)FEe-(1'l =
=
Probability
156 Proof By definition, jLELECY'
V2)IUjjdp
But, since G e T1 implies G jcyE''
J(JIFI 5'2)## for
=
6
V2)1T11##
al1 G e N1.
(10.56)
N2, (10.18)and (10.56)imply that =
jo-dp
all G e T1,
fOr
(10.57)
T2) IV1 is a version of F(Fl N1), proving (i). Part (ii) is by so that A'(F(Fl NI) is a Vc-measurable r.v. w sinceF(FI
10.9,
lf A simple application of the theorem is to a three-variable disibution. 3 (X(),l'(),Z())) is a random point in R measura ble on T, 1etc(Z) and c(F,@ be infimal c-fields on which Z and (F,@ respectively are measurable, and c(/ the c(F,@ 5. Unifying notation by writing F(l'Iz) = F(FIc(z)) and F(rlmz) = F(FIc4-Y,/), 10.26 implies that ,
F(e'(i'I-Y,z)IZ1
=
F(A'(FIz)I-X-,Z)4
=
F(1'1z).
(10.58)
Our final results derive from the conditional Jensen inequality.
10.27 Theorem Let F be a F-measurable r.v. and T1 i Uz Proof
T. lf
9(.)is convex,
F(t(F(FI5-2))) 2 F(9(e'(rI971))(). Applying 10.18 to the Tz-measurable r.v. F(FIW2)gives F@(f(FIT2))1V1) k #(F(F(l'I91)1N1)) #(F(Fl V1)) a.s.
(10.59) (10.60)
=
where the a.s. equality is by 10.26(i). The theorem follows on taking unconditional expectations and using the LlE. . The application of interest here is the comparison of absolute moments. Since lxlP 1, the absolute moments of F(FIT2)exceed thoseof F(FIT1) when is convexforr N1 i N2. ln particular,
(10.61) FgF(rIT2)21 k FgF(rIN1)2j. SinceFti'l NllandFtFl Vzlbothhavemean ofE, T2))k (10.61)impliesVar(F(FI EYl) for i 1 or 2, (the T1)). Also, F(F(FIVj)2)+F((F- F(FIVj))2) Var(F(FI =
=
expected cross-product vanishes by 10.10), so that an equivalent inequality is Fgtr-e'(J'l N2))21 J Fg(r-F(J'j T1))2j. (10.62) The intepretation is simple. V1represents a smaller infonnation set than V2, and if one predictor is based on more information than another, it exhibits more variation and the prediction error accordingly less variation. The exeme cases ;) are &FI = Fand F(FIT) = F(1'), with variances of Vartp and zero respectively, This generalizes a fundamental inequality, that a variance is non-negative, to the partial infonnation case. While (10.61)generalizes from the square to any convex function, (10.62)does
Conditioning
157
not. However, there is the following nonn inequality for prediction errors. 10.28 Theorem If F is T-measurable and N1 i N2 i ;,
IIF-FIFIN2)11,K 2I1F-F(F1Tl)IIp, p k 1. Proof Let
n
n -
F-
=
F(FIV1). Then
F(n lN2)
=
(10.63)
by 10.26(ii),
F- F(r1 N1)-&F1 T2) +&&FIV1)1T2)
= J'-s(rI va).
(10.64)
The theorem now follows, since
IIq F(q IT2)l1p< IIq11,+11F(nIT:2,)11,f 2IIq11p
(10.65)
-
by, respectively, L1E.
the (Minkowsk.i and conditional Jensen inequalities, and the
.
10.6 Conditional Distributions The conditional probability of an eventA e # can evidently be defined as #(4 IN) F(lx 1N), where 1x() is the indicator function of A. But is it therefore meaningful to speak of a conditional distribution on (D,@), which assigns probabilities PA 1N) to each A e @? nere are two ways to approach this question. First, we can observe straightforwardly that conditional probabilities satisfy the axioms of probability except on sets of probability 0 and, in this sense, satisfactorily mimic the properties of true probabilities, just as was found for the expectations. Thus, we have the following. =
10.29 Theorem (i) P(A IN) k 0, al1 A e F. (ii) Pf IN) 1 a.s. (iii) For a countable collection of disjoint sets Aj e @, =
#
UAjIT
=
X#(4jI 9') a.s.
(i),suppose 3 G e N with
Proof To prove
(10.18),
jaudp
=
PA IV)())
Jg#(4 I@4dP <
(10.66) 0 for a11(t) e G. Then, by
<
(10.67)
0,
which is a contradiction, since the left-hand member is a probability. To prove (ii), note that #(f1 lV) is N-measurable and 1et G+ e N denote the set of (l) such G+, that #(fl V)4)) > 1. Suppose #46*) > 0. Then since G+ rn =
#(&*)
=
josdp <
jo.#(QIT)##
=
jawezdp =
#(G+),
(10.68)
Probability
158
which is a contradiction. Hence, #(G+) 0. Repeating the argument for a set G- on 0. For (iii), (10.18)gives, for any which #4f1 I5')() < 1 shows that #(G-) G 6 V, =
=
J(,#(U,A'
jIU4dP
(10.69) Xyjocvbdp' jocvjpdp jvjacpdp =
=
=
since the sets G /'7 Aj are disjoint if this is tl'ue of the Aj. By definition there exists a version of #(l N) such that V G 6 5',
jocujdp Js#(X/IV) dP,
(10.70)
=
and hence
jop-v' /1 T)#/'
=
XJSPIA/IU4dP
=
./
V)) (Xy #(X/I dP.
jo
(10.71)
The left- and right-hand members of (10.71)define the snme measure on W(see 10.5)and hence #(U.?A.j lN) = Q#(AjIT) a.s. by the Radon-Nikodym theorem. w But there is also a more exacting criterion which we should consider. That is, does there exist, for fixed (z), a p.m. g,r,lon (f,F) which satisfies yu(A)
P(A l534)), each A e 5
=
(10.72)
E G where #(C) 1? lf this condition holds, the fact that conditional for all expectations and probabilities behave like regular expectations and probabilities requires no separate proof, since the properties hold for jt(o. lf a family of p.m.s lg,(o, e fJ satisfying (10.72)does exist, it is said to detine a regular conditional probability on V. However, the existence of regular conditioning is not guaranteed in every case, and counter-examples have been constructed (see e.g. Doob 19532 623-4). The = problem is this. ln (10.66),there is allowed to exist for a given collection (A e TJ an exceptional set, say Ca with PCa) 0, on which the equality fails. This in itself does not violate (10.72),but the set Ca is specific to and uncountable countable subsets number of there typically 5, we since are an Poaj would required both be for 0, be that guarantee to a p.m. cannot as g,fo and to satisfy (10.72). =
.4
=
.d,
.4
=
This is not a particularly serious problem because the existence of the family lju) has not been critical to our development of conditioning theory, but for certain purposes it is useful to know, as the next theorem shows, that p.m.s on the line do admit regular condittonal distributions. *
10.30 Theorem Given a space L1,5,P) and a subfield N c @,a random variable F has a regular conditional distribution defined by Fr/l
for
e C with #(C)
N)() =
=
j ,
lT)(), Fg(. IV)()
#((-x,y1
where
y e R, is a c.d.f. for all
(10.73) e f1.
Conditioning
159
Proof Write F(o*(y)to denote a version of #((-x,y1 I9')(). Let Mq denote the set of ) such that F' ri) > F(o*(rj)for ri, rj e :) with ri < (/ Similarly, 1et Ri denote the set of (l) on which limn-yx/trj + 1/n) # FLlril,ri e e. And tinally, 1et L denote the set of those ) for which F(l(+x) # 1 and F(o*(-x) y: 0. Then C (Uj#C is monotone and right-continuous at rn Lc is the set of on which FfLvl rn F(l(+x) with 1 and Ff)(-x) 0. For y e R let a11rational points of the line, .
-&lc
=
=
Fg'@) y e ()
=
,
Fr(. I5')()
=
F.(.y+), y e R
(,)e c,
-0
(10.74)
Otherwise,
G(X, where G is an arbitrary
c.d.f. ln view of 10.29, #(Mj) 0 for each pair i,j, PRi4 0 for each i and PL) 0. (1fneed be, work ln the completion of the space to define these probabilities.) Since this collection is countable, PC) 1, and in view of 8.4, Fy(. IV)() is a c.d.f. which satisfies (10.73),as it was required to show. . =
=
=
=
lt is straightforward, at least in principle, to generalize this argument to multivariate distributions. For # e B it is possible to write F(1s I5')(t,))
=
V)() jedkyf
(10.75)
a.s.,
and the standard lgument by way of simple functions and monotone convergence leads us full circle, to the representation
F(FI9')()
+x
J
=
-X
y#&@l
syy(y,a.,.
(10.76)
ctm, we have constructions to parallel those of j10.1. Since no restriction had to be placed on the distribution to obtain this result, we have evidently found a way around the difficulties associated with the earlier detinitions. However, Fy(. IV)() is something of a novelty, a c.d.f. that is a random element f'rom a probability space. lntuitively, we must attempt to understand this as representing the subjective distribution of F(@ in the mind of the observer who knows whether or not ) e Gfor each G e V. The particular case Fy(. IN)4) is the one of interest to the statistical modeller wheh the outcome ) is realized. Manyrandom variables may begeneratedfromtheelements of (D,1,#), notonly the but also variables outcome itself in the bivariate case the pair F(),X() and A11 have the quantiles of Fv@ these such as F(FIm(), to be thought I of as different aspects of the same random experiment. Let X and F be r.v.s, and N a subfield with V c Rx c(A') and V ? Rv c(F). We say that X 4nd F are independent conditional on V if If V
=
-
-
m().
=
Fxrtmyl 5')
=
Fx@l S'IFI-I.yI V) ms.
=
(10.77)
Probability
160
This condition implies, for example, that f(Xl'IV) g(.,(o) be the conditional measure such that g,(o(lX 6
(--,A1,
i'
(--,A'1 1)
6
=
=
F(XIV)F(FIV) a.s.
Fxvx'y
Let ju
=
1V)().
with (t) fixed this is a regular p.m. by (the bivariate generalization ofl 10.30, and gwotAfa #) g(o(A)g(o(#) for eachA e Rx and B e Xy, by 10.21. ln this sense, the subtields Rx and Ry can be called conditionally independent. =
10.31 Theorem.
lf X and F are independent conditional on V, then
f(l'1 Rx4
F(1'l N) a.s.
=
(10.78)
Proof By independence of Rx and Rv under JxftFl
XaYll/t
=
JxFtfltto
=
u we can write BtX)JFtV' X e Rx.
(10.79)
This is equivalent to
&1z&F1Jx)l 9')(t,)) F(lxl'l 5')() =
T) IT)() = F(1xF(1'I
were tlae tirst equality also with respect to #, noting f
JxF(FIRdp
=
a.s.(#1,
(10.80)
follows from 10.26(i) and 10.10. lntegrating over f e V, using 4.8(ii) and the LIE, we arrive at JxF(Fl U4dP. A e Rx.
jgk'dp =
This shows F(Fl Rx) is a version of
F(FIN), competing
(10.81)
the proof.
F(FIRx) is in principle
.
Xvmeasurable, it is in fact almost surely I#J equal to a V-measurable r.v. Needless to say, the whole argumentis symmetric in X and Rp The idea we are capturing here is that, to an observer who possesses the information in V (knowswhether (t) e G for each G e V), observing Xdoes not yield any additional information that improves his prediction of F, and vice versa. This need not be true for an observer who does not possess prior information. Equation (10.77) shows that the predictors of F based on the smaller and larger information EY4 a.s., so sets are the same a.s.(#), although this does not imply F(Fl R that X apd F are not independent in the ordinary sense. Thus, while
=
11 Charqcteristic Functions
11.1 The Distribution of Sums of Random Variables Let a pair of independent r.v.s X and F have marginal c.d.f.s Fx@) and Fy@). The c.d.f. of the sum J'P X+ F is given by the convolution of Fx and Fs the =
function
Fx*Fy(w)
+x J..Fx(w
=
(11.1)
-ytdFv(9.
11.1 Theorem If r.v.s X and F are independent, then Fx*Fy(w)
#(X+ F f w)
=
FI'*Fx(w).
=
(1 1.2)
Proof Let 1w(my) be the indicator function of the set fmy:x K w y), so that F(1w(X,F)). By independence Fxty) FxxqFvy), so this is #(X+ F < w) -
=
=
Js21w@,A')F@,A')
+x
P+x
J..J
dFyIyl
.x1w@,.')tFx(I)
=
+x
w-y J..JFx@)
J..
=
dFyvl
+x
Fxtw
=
-
y)#Fy(y),
(1 1.3)
-X
where the first equality is by Fubini's theorem. This establishes the first equality in (11.2). Reversing the roles of X and i' in (11.3) establishes the second. w For continuous distributions, the convolution
f
=
fx*fyof
p.d.f.s
fx and h
is
+x
j-.fxw -').fy(')*,
f(w)
=
such that
Jlx/'ttlx
=
(1 1.4)
F(w).
11.2 Example Let X and Fbe independent drawings from the uniform distribu1m,1j@).Applying (11.4)gives tion on 0,11, so that fxx) =
/k+r(w)
=
1
llw-l,wj#y.
()
that the graph of this function forms an isosceles triangle lt is with base (0,21and height 1. n easily veritied
Probability
162
This is the most direct result on the distribution of sums, but the formulae generated by applying the rule recursively are not easy to handle, and other approaches are preferred. The moment generatinghlnction (m.g.f.)of X, when it exists, is jedFx), Eeth Mxtf) t e R, (11.6) =
=
where e denotes the base of natural logarithms. (Integrals are taken over (-x,+x) unless otherwise indicated.) If X and i' are independent, Mx-vvt)
jjetx*'gdt-xx4dll-y,
=
=
jetxdnxjetydFyy)
= Mx(f)Mg(/).
(11.7)
This suggests a simple approach to analysing the distribution of independent sums. The difficulty is that the method is not universal, since the m.g.f. is not defined for every distribution. Considering the series expansion of e all the moments of X must evidently exist. The solution to this problem is to replace the variable f by it, where i is the imaginary number, VX. The characteristic function (ch.f.)of X is defined as ,
9x(f) =
1l
.2
Ee
itX
)
itx
(1 1.8)
e dFx).
=
Complex Numbers
A complex number is z a + ib, where a and b are real numbers and i V-X.a and b are called the real and imaginary parts of the number, denoted a = Re(z) and b lm(z). The complex conjugate of z is the number i' a ib. Complex adthmetic is mainly a matter of carrying i as an algebraic unknown, and replacing 2 j)y j /3 i 4 by 1 e tc wherever these appear in an expression. by One can represent z as a point in the plane with Cartesian coordinates a and b. The modulus or absolute value of z is its Euclidean distance from the origin, =
=
=
=
-
-
,
-,
,
.,
lz l
=
(zD 1/2
=
a
2
+
jp
1/2
)
(1j
.
.
9)
Polar coordinaies can also be used. Let the complex exponential be defined by
efo = cos 0
+ jsin
8
(11.10)
for real 0. All the usual properties of the exponential function, such as multiplication by summing exponents (according to the nzles of complex arithmetic) go through under this definition, and
Ie i l
=
+ sin2e)1/2 (coszo
=
1
(11.11)
Characteristic
Functions
163
for any 0, by a standard trigonometric identity. We may therefore write z Iz Iei where Retz) Iz Icos 0 and 1m(z) Iz Isin 0. Also note, by (11.11),that =
=
=
.Re(z)+Im(z) j Ie z 1 j =
@Re(z)
jef1m(z)j
=
=
@Re(z)
(j.j. jg;
.
.
random If X and i' are real random variables, Z X+ iY is a complex-valued variable. lts distribution is defined in the obvious way, in terms of a bivariate c.d.f., F@,y). In particular, =
EZ)
(11.13)
F(A') + iElY).
=
Whereas EZ4 is a complex variable, FIZI is of course real, and since 1zl < IXl + Il'I by the triangle inequality, integrability of X and F is suftkient for the integrability of Z. Many of the standard properties of expectations extend to the complex case in a straightforward way. One result needing proof, however, is the generalization of the modulus inequality.
11.3 Theorem
If Z is a complex random variable,
Proot Consider a complex-valued
4F(Z) I S FI I .
.
simple nv.
(a./+ lllsy, zn) 7'j=L -
where the xj and I are real non-negative constants and the Ej e 5 for j constitute a partition of f1. Write Pj F(1sy). Then
(11. 14) =
1,...,p
=
2
IEznl) 1
-
2+
77xjpj
=
2
77fjjpj
+ f5,5/,7? 77(a??+ 7777(%a,+ fbbrjn,
( 11.15)
j*k
j
whereas CEIz(n)I)2
=
=
1/2
>7(a,?+ p,?)Pj
2
./ + + jk2)1/2# j pk. X (( + f5?)#? X X (%?+ Y?)'/2(ak2 J
J
J
j*k
./
(11.16)
The modulus inequality holds for Zn) if
0S =
(E IZ(n) I)2
-
2
IF(z(n)) I
p2,)1/2(%a,+ 1%)1#./#,. 7777E(a?+p,?)''2(a,2+ -
J
(1 1.17)
The coefficients of Pjpk in this expression are the differences of pairs of nonnegative terms, and these differences are non-negative if and only if the differences of the squares are non-negative. But as required,
Probability
164
+ p2,) (%a.+ Ip.)2 (a,?+ If)(a2k -
,
a,?p2,+ a%?
=
=
zwakjf'lk
-
(ay;'J. a.I$)2z O
(11 18)
-
.
.
This result extends to any complex r.v. having non-negative real and imaginary Z X+ iY, using 3.28, and invoking the parts by letting Z(n) = Xn) + in) monotone convergence theorem. To extend to general integrable nv.s, splitXand F X* + iY* into positive and negative parts, so that Z = Z+ r, where Z+ X+ 1'+ A?'-+ iT, with X- k 0 and lr- k 0. Noting that 2 0 and k 0, and Zwith '1k
=
=
-
=
IEZ) I < IE(Z+ + r) l < e'lz++ r I FIzl
(11.19)
=
completes the proof.
.
11.3 The Theory of Characteristic Functions We are now equipped to study some of the properties of the characteristic function x(/). The fact that it is detined for any diskibution follows from the fact 1 and Eeth is finite regardless of the 1 for all x; E( Icfl'fl ) that Ie x I distribution of X. The real and imaginary parts of #x(J) are respectively Ftcos t.k and fsin tX4. =
=
k< 11.4 Theorem If E IA-I
x,
then
k
d px(/)
d
=
Eiuk
k itx
e
).
(1 1.20)
Proof
t-bht #x(J) = h
eixt'ht
+=
-
eitx
-
#(x),
h
-x
(1 1.21)
where, using (11.10), e
t-h)
h
-
e
itx
=
cOs
xt
+
h4 h
-
cOs
tx
+
j is n
.7
(/ + 4 -
siu tx .
h
The limits of the real and imaginary terms in this expression as h sin tx and i(x cos /x), so the limit if the integrand in respectively -->
-x
is
xi cos tx
-
sin fx)
=
fccos tx
+
i sin tx)
=
ie
0 are
(11.21)
it: .
since1tfxlefr-lIxl
the integral exists if A'C-Y'I This proves (11.20) for complete 1. To the case k the proof, the same argument can be applied itx inductively to the integrands (ix)k-1e j or k z a s =
,
=
.
,
,...
It follows that the integer moments of the distribution repeated differentiation with respect to t.
can be obtained by
Characteristic lf Fl#! k <
11.5 Corollary
Functions
165
th en
x,
24x(f)
k k = i EX ). n
dtkx
(1 1.22)
An alternative way to approach the last result is to construct a series expansion of the ch.f. with remainder, using Taylor's theorem. Tlzis gives rise to a very useful approximation theorem.
11-6 Theorem lf
FI.Y1k <
then
x,
k
k 77tffytil (1)x(/)j=0 j! Proof A function
,
(1 1.23)
.
differentiable k times has the expansion
which is
f
+1
l I/XI IfXl k! (k+ 1)!
f E min
-
./*(a/)
1 Jz''(0), 3 z'(0) , +
f(f) f(0) + f '(o), + =
where 0 f
; 1. The expansion
(x
of
#
e itx where yk
=
iksgntfx)
k
jwx
(e
ft)
e
=
+
...
:!
p,
)
it?
glves
#
'
itxf
y'l y!
=
+
6
z
+
Itx I yk,
(1 1.24)
k!
j; ayjtj
-
1
sgnt/x
tx k 0
,
=
.
- 1 tx
<
,
0
Applying (11 10) and (11 11), we can show that Iyk I (2 2cos a/x)1/2 < 2. However, by extending the expansion to tenn k + 1, we also have =
.
.
k
e itx
k+1
'
itx/
=
g! .
jzzo
+
ltx I
j or () < (g s j and where zk ik+1sgnt/x) e (1 1.24) and (11.25) hold, we may conclude that ,
k
d
k
'
itxf
itx y.l ./! -
c
min
(1 1.25)
k + 1)! zk,
k+1 ift.T
=
-
2 Itx I k!
jzkj
1. Gjven that both of
=
1+l
,
Itx I
(k+
1)!
The theorem now follows on replacing x with the r.v. #in ations and using the modulus inequality:
(1 1.26)
.
(11.26),taking
expect-
Probability
166 j4e'tx./l k 9x(/)77 !
t ./
-
s
,
e
k
i
-
(/m./ j--,j y!
:7
+1
k
f E min
l I/.X'1 I/-YI k! (k+ 1)! ,
.
(11.27)
w
There is no need for FIXl k+1to ex i st for this theorem to hold, and we can think of it as giving the best approximation regardless of whether I/XI is large or small. To interpret the expectation on the right-hand side of (11.27), note that, for any pair of non-negative, measurable functions g and ,
Ftminf#tm,tml)
inf F(#(m1,4
=
+
Ae 3
tmlxc),
(11.28)
the infimal set being the one containing those points x on which particular,for any s 2 0, the set A ( IXI> EJbelongs to the class the intimum in (11.28)is taken, and we get the further inequality,
< hlx). In
.(x)
=
.
E m1n
2 Itx
k'!
Ik Itxl ,
k+1
k
k+1
2 I/aYI f E 1( jxjlsj k!
k + 1)!
over which
+
2/ E A'1k1f IxI> .:) ) k! I
E
I?-YI lt jxjssj
(k+
1) !
k+1E,k+1
+
It I
k + 1) !
(1 1.29)
S
2/ I/'I F1x'1 kt F(lxI '1. ( IxI>e) ) + 1)! (.+ k! k+1
The second alternative on the light is obtained in view of the fact that k k k+11( IxI
kax+b
=
ibt E efr(cx+h) ) = e #x(J/).
The second is the counterpart variables X and F,
of
(11.7).For
a pair of independent
(1 j ao .
random
jjeitx*'tdlkxtdFy'
#x+y(/) =
=
jeitxdFxxljeivdFvlp
=$x(f)#,'(/).
(11.31)
An interesting case of the last result is F -X' where X' is an independent drawing from the distribution of X. The disibution of X- X' is the snme as that of =
Characteristic X'-X,
Functions
167
symmetric about 0. The ch.f. of X X' is real,
and hence this r.v. is
-
since
#x(f)9x(-f) I4)xtfl I2 =
(11 32) .
,
Ee-ith in v iew o f the fact that bx#aV).It can be verified from the with real ch. f., a1l the existing odd-order moments expansion in (11.23) that a be the trademark distribution. of symmetric zero, must a =
=
:=1Xj where (X1,...,aL Considering more generally a sum S 1 are a totally recursive yields collection, of application independent (11.31) =
(1),(/)f=1 11(l)x(/).
( 11.33)
=
To investigate the distzibution of S, one need only establish the fonnulae linking the ch.f.s with the relevant c.d.f.s (or where appropriate p.d.f.s) which are known for the standard sampling distributions.
11.7 Example For the Poisson distribution (8.8),
9x(f',)
Ee
=
itX
)u
)
x
(j.
77w
--
e
=
;:
X!
e
i,
e
=
)k(eyy..j j .
(1 1.34)
n
11.8 Example In the standard Gaussian case, (8.10), 9z(J)
=
Ee
itLl
)
1 =
Completing the square yields itz
z(f)
=
e-
/2
+oo
/-2l
-
z2/2
-
1
+=
27:
x
e
itz
2
/2
-z
jf)2/2
-(z
=
dz.
-
(1 1.35) /2/2
-
and hence
-(z+ff)2 -/2
dz
e
-x
=
e
.
(1 1.36)
(The integral in the middle member has the value ulV for any choice of t, note.) Accordingly, consider X = cZ+ po whose p.d.f. is given by (8.24).Using (1 1.30), we obtain
(xlt, . poc) Equation With g,
e/w-c2p/z
.
(11.22) can be used to verify the moment =
(11.37)
.
0 the ch.f. is real, reflecting
formulae given in 9.4 and 9.7.
the symmetry of the distribution. n
11.9 Example The Cauchy distribution (8.11)has no integer moments. The ch.f. turns out to be e -lfI w hich is not differentiable at t = 0, as (11.22) would lead us to expect. The ch.f. for the Cauchy family (8.15)is #x(J;
V, : )
=
e
frv-lfl .
E1
The ch.f. is also defined for multivariate disibutions. (??zx 1) the ch.f. is
(j j ag; .
For a random vector X
Probability
168
expfif'fl),
#(f)
=
(11.39)
where t is a v-vector of arguments. This case will be especially important, not least because of the ease with which, by the generalization of (11.30), the ch.f. +.d (kx 1), can be derived for an affine transformation of the vector. Let F BX where B (kx m) and d k x 1) are constants, and then we have =
g(/)
F(exp(ff'F))
=
exp(I'/'#)F(exp(//'#.Y))
=
=
expfit'dklxlB't).
(11.40)
11.10 Example Let X m x 1) be multinormal with p.d.f. as in (8.36).
ch.f. is
rrhe
#x(f;M,E)
1
=
+=
rn/z
(2a) 1x j = exp l it'p
-
+= ...
1/2
-.
expl/'.t
g)'E-1(x
@
-
-
..
-
M)l#-4r
rlfzy.z )
(1 1.41)
.
The second equality is obtained as before by completing the square: 1 g(x it 'x j/zD ,1(x g j&)I-1(x g j&) itp g,)'E-l (x p) -
where it can be shown mI1
(2,n) Ix j
1/2
=
-
-
-
-
-
-
-
-
,
of the last term integrates over R to
that the exponential
n
.
1 1.4 The lnversion Theorem Paired with (11.8) is a unique inverse transformation hom #4/)to F@), so that the ch.f. and c.d.f. are fully equivalent representations of the distribution. The clef step in the proof of this proposition is the construction of the inverse transfonnation, as follows.
11.11 Lemma If
9(f)is defined
Fb)
-
F(J)
by
1 1im 2a F->x
=
(11.8), then yn
--
e
ittl
--
e
-
itli
(dt
it
..w
for any pair a and b of continuity points of F,' with a generalization of this formula is AF(xl
,...,.q)
=
1 m
za
1im T-+x
z'
z' ...
-w
-z
k
-jx./
e
J''I
jzzj
-
e
(11.42) <
b. The multivariate
-j.%+Ax.j)
jjy
(1 1.43) xtxj...xktt,---ntktdtb---dtk. wher AF(x1,...,.q) is defined in (8.31)and the vertices of the rectangle based at the point x1,...,ak, with sides Lxj > 0, are a11continuity points of F. n --
1.,
wxza..1
xrolon:
f.n
/1 1 q0'
Tt onn
lw
xzorifiofl
Ilqin
e (11
10
thnt
Characteristic
169
-itb
-ita
d
Functions
-d
--: b a as t
0.
-->
-
it
The integrals in (11.42)and (11.43) are therefore well defined in spite of including the point t 0. Despite this, it is necessary to avoid writing (11.42) =
as '-ita
Fb)
F(J)
-
'-itb
+x e
=
e
-
(11.44)
(f)Jf,
lnit
because the Lebesgue integral on the right may not exist. Forexample, suppose the a'it.j j random variable is degenerate at the point 0; this means that (/) e and =
F
1 21
-
e
ita
,
-itb
e
-
1 r j. 7: 1 t
log F,
--dt
dt k
it
-w
=
-
-
( 11.45)
so that the criterion for I rbesgue integrability over (-x,+x) fails. However, the limits in (11.42)and (11.43) do exist, as the proof reveals. Proof of 11.11 Only the univariate case will be proved, the multivariate extension being identical in principle. After substituting for () in (11.43) we can interchange the order of integration by 9.32, whose continuity and a.s. boundedness requirements are certainly satisfied here: '-itb
-ita
'
d
r
dt
lnit
=
= Using
'-itb
-ita
F
-d
e
e
-
-F
21f/
+x
T
-x
-F
d
oo
-x
itx-a)
j/x
e dF@) dt
itx-b)
-e
t dF(x)
lxit
.
(1 1.46)
(11.10.), ' e itx-a)
-.w
-
e
itfx-b)
t
lxit
z'Sj n yx
=
F sjn j(x
.u)
t
'lt
()
-
()
zt
.y)
t,
(1 1.47)
noting that the cosine is an even function, so that the terms containing cosines x (which are also the imaginary terms) vanish in the integral. The limit as F standard fonuula of this expression is obtained from the -..j
jo'='sin
t t
dt
=
zIl,
tx
()
(:
,
-1/2, tx
>
0
.()
<
(j j.4g) 0.
Probability
170
yields the result
(11.46)
Substituting into l.x
e
fftx-a)
il(2'-8
-d
t
lxit
=
0, x
<
a or x > b
l 7:
=
c or x
<
x < b.
x
1, a
b
=
(1 1.49)
Letting F-->x in (11.46) and applying the boundedconvergence theoremnow gives -ita
lim
F-px
T -
-itb
e
-
+x
e
(tldt
lgit
w
-x
(lfc) +
= r1(F(:)
1(s) + ta,bdFx)
F(b-)
+
Fa)
-
-
F(J-)),
(11.50)
which reduces to F(b4 Fa) when a and b are continuity points of F. w -
Lemma 11.11 is the basic ingredient of the following key result, the one that primarily justifies our interest in characteristic functions.
theorem Distributions having the same ch.f. are the same.
11.12 Inversion
Proof We give the proof for the unvariate case only. By (11.42), the c.d.f.s of the two distributions are the same at every point which is a continuity point of both c.d.f.s. Since the set of jump points of each c.d.f. is countable by 8.3 their union is countable, and it follows by 2.10 that the set of continuity points is dense in R. It then follows by 8.4 that the c.d.f.s are the same everywhere. . A simple application of the inversion theorem is to provide a proof of a well known result, that affine functions of Gaussian vectors are also Gaussian.
11.13 Example Let,f Ng,X) m x 1) and F= BX d n x 1) are constants. Then by (11.42),
-d
n
-
#g(f)
=
exptff'#lFtexpt#'sfll
= expfit'Bg If ranktS'xS)
n,
=
(implyingn -(y
.f(J)
expf
=
x 1) where B n x m) and
-
Bg
+d)
-
qt'B'SBtj
(11.51)
.
< m), 11.12 implies that F has p.d.f. -
d) '(#xn')-l(y tz I#IN
(2r)n
,
1/a
I
-Bg
-#))
.
(1 1.52)
< n, (11.51) remains valid although (11.52) is not. But by the same lf rankt#xs') arguments, every linear combination c'F, where c is p x 1, is either scalar Gaussian with variance c'BSB'c, or identically zero, corresponding to the cases B'c 0 respectively. ln this case F is said to have a singular 0 and #'c Gaussian distribution, I::I :#
=
Characteristic
Functions
171
11.5 The Conditional Characteristic Function Let F be a F-measurable r.v., and 1et V c @. The conditional ch.f. of 4)ysg(f), is for each t a random variable having the property
jgtrj
g(/)##
jaeitp,
all G e V.
=
FIV,
(11.53)
The conditional ch.f. shares the properties of the regular ch.f. whenever the theory of conditional expectations parallels that of ordinary expectations according to the results of Chapter 10. Its real and imaginary pms are, respectively, the V-measurable random variables Ftcos /F1 V) and fsin fFl N). It can be expanded as in 11.6, in tenns of the existing conditional moments. lf X is V-measurable, the conditional ch.f. of X+ F is #x+gIv(/) e itxpjeityks; by 10.10. And if i' is Vl-measurable and N and V1 are independent subfields, then =
#i'l5'(J)
l'(/)
=
a.S.
The conditional ch.f. is used to prove a useful inequality due to von Bahr and Essen (1965).We start with a technical lemma which appears obscqre at first sight, but turns out to have useful applications.
11.14 Lemma Suppose
FIZIr
<
0
cx),
<
2. Then
<
r
A'(rlj..+x1 Re(#z(/)) -
Fl W here
K
=
zl
r
=
l+rfJfl-l
(J+*(1cos ?g)/1 u I -
=
-x
(1 1.54)
t
1+r
jyj
a-1r'(r+ j) sin ra/2.
n
The last equality, with F(.) denoting the gamma function, is a standard integral formula for 0 < r < 2. Proof The identity for real z,
lz lr
+oo
=
1 cos zt dt, 1+,
Ar)J.. -
(1 1.55)
jjj
is easily obtained by a change of variable in the integral on the right. The lemma Acos tZ4. . follows on applying 9.32 and noting that Re(9z(/)) =
Tl'lis equality also holds, for ) e C with PC) = 1, if F( 1Zl rl N)((t))and 9zls(J)()) r are substituted for Fl zl and #z(f).In other words, the conditional rth moment and conditional ch.f are almost surely related by the same formula. So consider fv-bounded r.v.s z and X, where Z is T-measurable, and X is N-measurable for N c 5. Suppose that #z!v(/)is a real r.v. almost surely. Then for each e f1,
1
-
Re@x+zIs(f))()
=
1 Retd -
itX
#zIs(f))())
= 1 (cosJX())(zIg(/)()) -
S
(1
-
cos /X())
+
(1
-
9zIs(J)()),
(1 1.56)
Probability
172 thedifference between whichis non-negative
the last two members being (1 cos /X()))(1 for all Hence, for 0 < r < 2, -
#z1g(/)()))
-
.
+x 1 Re@x+zIs(/))
#(r)J.. += f #(r)J.. Ar)J.. -
E IX+
zl
rl
N)
=
t
j+r
jjj
1 cos tx
+=
-
1.r
jyj
l#l
=
'
+
t
1
zl v(/)
-
+
jyj
t
j+,
Izl r IV), a-s.
E
(1 1
.57)
and taking expectations through yields
FIA-+zIr < FIXIr
f'l
+
ZIr.
(11.58)
For the case 0 < r S 1 this inequality holds by the cr inequality for general Z and X, so it is the case 1 < r < 2 that is of special interest here. Generalizing from the remarks following (11.31), the condition that (zIv(/) be real a.s. can be fulfilled by letting Z Y- Y', where F and F' are identically distributed and independent, conditional on V. Note that if R = c(F), then =
F(F'IX)
F(F'IT),
=
(11.59)
a.s.,
by 10.31. Identical conditional distributions means simply that Fv(. 1V) Fg(. IV) a.s., and equivalently that (gIs(/) #glv(/) a.s. Hence
=
=
-fry'
#r-r'1p(/)
=
ffye
Ee
ls)
= Ee itvls )eje-itvj v) IrIs(/)l 2 a.s.,
=
(j j
,
where the right-hand side is a real r.v. Now, for each identity can be verified:
2(1
Re(#y,v(/)())
-
Applying
=
1
-
la + Il
g1s(/)()
-
(t)
-
6o)
e f1, the following
9rIs(f)((.0)
2
I
(11.61)
.
and 11.14, and taking expectations, this yields the inequality
(11.60)
2A'1FIr
r,
Fl Y-
F'I
0
<
<
r
(11.62)
2,
noting that the difference between the two sides here is the non-negative g(/) 2/j J'j 1+rJ/ Of r #(r)J+=F1 1 9y.1 l These arguments lead us to the following conclusion.
function
-
,
.
..x
11.15 Theorem Suppose F(l'IV) = 0 a.s. and X is V-measurable where V i R
c(F), and both variables
FI.Y+ F1r
=i
are Lr-bounded. Then
r s FIXI
+
2FIFIr,
0
r
(11.63)
2.
Proof Let F' be independent and identical with F, conditional on V. Applying (11.59), these conditions jointly imply f(F'I R) &F'I V) F(FI9') 0. Noting =
=
=
Characteristic
Functions
173
that X+ F is A-measurable, it follows by 10.19 (in applying this result be careful to note that R plays the role of the subfield here) that E IX+
r
FI
<
FIX+ (i'- r) I
'.
(11.64)
The conclusion for 1 < r < 2 now follows on applying (11.58)for the case Z F- i'', and then (11.62). The inequality holds for 0 < r f 1 by the cr inequality, and for r 2 from elementary considerations since Elk'.,Y'I0. In these latter cases the factor 2 in (11.63) can be omitted. . =
=
=
This result can be iterated, given a sequence of r.v.s measurable on an increasing for sequence of c-tields. An easy application is to independent rov.s A'1,...,aL, 2,...,n. 0 certainly holds for t which the condition E(Xt Ic(-Y1,...,m-1)) ntzzkxt, and 11.15 c(X1,...,Xn) yields Letting Sn c(&) =
=
=
=
E IsnIr < A'l
'n-t
Ir
+
< 2XA'ImI r,
2Fl Xn Ir
0 < r < 2.
(1 1.65)
>1
If the series on the majorant side converges, this inequality remains valid as n --> x. lt may be contrasted for tightness with the cr inequality for general Xt, (9 62). In this case, 2 must be replaced by nr-1 for 1 < r S 2, which is of no use for large n. .
lll THEORY OF
STOCHASTIC PROCESSES
12 Stochastic Processes
12.1 Basic Ideas and Terminology Let (( 5 #) be a probability space, let 'T be any set, and let RV be the product space generated by taking a copy of R for each element of 'F. Then, a stochastic process is a measurable mapping x: f r--yR'T where ,
,
j
xttDl
=
l-V((,)), ':
(12.1)
e
'T is called the index Jtrf, and the r.v. Xs() is called a coordinate of the process. A stochastic process can also be characterized as a mapping from x'T to R. However, the significant feature of the definition given is the requirement of joint measurability of the coordinates. Something more is implied than having Xs(() a measurable nv. for each 1. Here, 'Ir is an arbitrary set which in principle need not even be ordered, although linear ordering characterizes the important cases. A familiar example is T' ( 1,...,k) where x is a random k-vector. Another important case of 7 is an interval of R, such that x()) is a function of a real variable and R 'T the space of random functions. And when 'T is a countable subset of R, x fXv()), G 7-) defines a stochastic sequence. Thus, a stochastic sequence is a stochastic process whose index set is countable and linearly ordered. When the Xz represent random observations equally spaced in time, no relevant information is lost by assigning and a linear ordering through ENor Z, indicated by the notations f-V()))''*-x.The definition does not rule out 1- containing information about distances between the sequence coordinates, as when the observations are irregularly spacd in time with a real number representing elapsed time from a chosen origin, but cases of this kind will not be considered explicitly. Fnmiliarly, a time series is a time-ordered sequence of observations of (say) economic variables, although the term may extend to unobserved or hypothetical variables, such as the errors in a regression model. Time-series coordinates are labelled t. If a sample is defined as a time series of finite length n (or more generally, a collection of #uch series for different variables) it is convenient observato assume that samples are embedded in infinite sequences of tions. Various mathematical functions of sample observations, statistics or estimators, will also be well known to the reader, characteristically involving a summation of terms over the coordinates. The sample moments of a time series, regression coefficients, log-likelihood functions and their derivatives, are standard examples. By letting n take the values 1,2,3,..., these functions of n observationsgeneratewhatwemay call#crfvdtfsequences.The notionof asequence =
,
':
=
(.V((t)JO
':
dpotential'
Theor.y of Stochastic Processes
in this case comes from the idea of analysing samples of progressively increasing size. The mathematical theory often does not distinguish between the types of sequence under consideration, and some of our definitions and results apply generally, but a clue to the usual application will be given by the choice of index symbol, / or n as the case may be.
A leading case which does not fall under the detinition of a sequence is where 'T is partially ordered. When there are two dimensions to the observations, as in a panel data set having both a time dimension and a dimension over agents, x may be called a random .#e/J. Such cases afe not treated explicitly here, although in many applications one dimension is regarded as tixed and the sequence notion is adequate for asymptotic analysis. However, cases where 'T is ither the product set ; x(N, or a subset thereof, are often met below in a different context. A triangular stochastic array is a doubly-indexed collection of random variables,
X1l Xz1 X51 .Yla X22 X51
11l
(12.2)
X3k1
a=l is some increasing integer compactly written as ( (Akmlmil ) n=1, w he r f/cn )'=' sequence. Array notation is called for when the points of a sample are subjected to scale transformations or the like, depending on the complete sample. A standard Z7=1Var(X,), or some where Xnt = Xtlsn, and sn example is ( lA-n,))=1)=n=j, similar function of the sample moments f'rom 1 to n. e'D
=
12.2 Convergence of Stochastic Sequences Consider the functional expression (Ak())
';
for a random sequence on the space L1,5,P). When evaluated at a point (l) e this denotes a realization of the actual collection real is of numbers generated whenthe outcome sequence, the drawn. lt is natural to consider in the spirit of ordinal'y analysis whether this e f2, we sequepce converges to a limit, say Al. Tf this is the case for every would say that Xn Xsurely (orelementwise) where, if Xnis an T/f-measurable nv. for each n, then so is X, by 3.26. But, except by direct construction, it is usually difficult to establish in terms of a given collection of distributional properties that a stochastic sequence converges surely to a limit. A much more useful notion (becausemore easily shown) is almost sure convergence. Let C ? fl be the set of outcomes such --y
Stochastic Processes
179
.Y()) as n that, for every (t) e C, Xn) x. If #(C) 1, the sequence is said almost surely, or equivalently, with probability one. The notations Xn to converge -->
-->
=
-F.&:
X a.s., and a.s.lim-,, X are al1 used to denote almost sure convergence. A similar concept, of convergence almost everywhere (a.e.),was invoked in connection with the properties of integrals in j4.2. For many purposes, almost sure convergence can be thought of as yielding the same implications as sure convergence in probabilistic arguments. However, attaching probabilities to the convergent set is not the only way in which stochastic convergence can be understood. Associated with any stochastic sequence are various non-stochastic sequences of variables and functions describing aspects of its behaviour, moments being the obvious case. Convergence of the stochastic sequence may be defined in tenns of the ordinary convergence of an associated sequence. If the sequence fExn A'')217converges to zero, th ere is a clearly a sense in which Xn X.,this is called convergence in mean square. Or suppose that for any 6: > 0, the probabilities of the events ( ): IX,,() X()) l < :1 e 5 form a real sequence converging to 1. This is another distinct convergence concept, so-called convergence in probability. In neither case is there any obvious way to attach a probability to the convergent set; this can even be zero! These issues are studied in Part lV. Another convergence concept relates to the sequence of marginal p.m.s of the coordinates, fgwl!', or equivalently the marginal c.d.f.s, (F,,l7. Here we can consider conditions for convergence of the real sequences fgwtAlITfor various sets A e or alternatively, of (F,,(x) J7 for various x e R ln the latter case, uniform or pointwise convergence on R is a possibility, but these are relatively strong notions. It is sufficient for a theory of the limiting distribution if convergence is confined just to the continuity points of the limiting function F, or equivalently (as we shall show in Chapter 22) of gwtAl, to sets A having g,tAl = 0. This condition is referred to as the wdck convergence of the distributions, and forms the subject of Part V. X, or Xn
-->
=
-
-->
-
,
.
12.3 The Probability Model Some very important ideas are implicit in the notion of a stochastic sequence.
Given the equipotency of INand 1, it will suffic to consider the random element (A((l)))T, e f1, mapping from a point of to a point in infinite-dimensional Euclidean space, R=. From a probabilistic point of view, the entire infinite sequence corresponds to a single outcome ) of the underlying abstract probability space. ln principle, a sampling exercise in this umework is the random drawing of a point in R=, called a realization or sample path of the random sequence', we tnay actually observe only a tinite segment of this sequence, but the key idea is that a random experiment consists of drawing a complete realization. Repeated sampling means observing the same tinite segment (relativeto the origin of the index set) of different realizations, not different segments of the same
realization. The reason for this characterization of the random experiment will become clear
18O
Theot'y of Stochastic Processes
in the sequel; for the moment we will just concentrate on placing this slightly outlandish notion of an intinite-dimensioned random element into perspective. To show that there is no difficulty in establishing a correspondence between a probability space of a familiar type and a random sequence, we discuss a simple example in some detail. 12.1 Example Consider a repeated game of coin tossing, generating a random sequence of heads and tails; if the game continues for ever, it will generate a sequence of infinite length. Let 1 represent a head and 0 a tail, and we have a random sequence of ls and 0s. Such a sequence corresponds to the binary (base2) representation of a real number; according to equation (1.15) there is a one-toone correspondence between infinite sequences of coin tosses and points on the unit interval. On this basis, the fundamental space (f,T) for the coin tossing experiment can be chosen as ((0,1),S(0,l)).The form of P can be deduced in an Pttails) 0.5 (i.e.the coin elementary way from the stipulation that Ptheads) is fair) and successive tosses are independent. For example, the events ftails on first toss ) and lheads on Erst toss) are the images of the sets (0,0.5)and g0.5,1)respectively, whosemeasures must accordingly be O.seach. More generally, the probability that the first n tosses in a sequence yields a given configuration 1/2N, of heads and tails out of the ln possible ones is equal in every case to so likely'. The that each sequence is (in an appropriate limiting sense) corresponding sets in (0,1)of the binary expansions with the identical pattern of =
=
tequally
Os and ls in the tirst n positions occupy intervals al1 of width precisely $l2n in the unit interval. The conclusion is that the probability measure of any interval is equal to its width. This is nothing but Lebesgue measure on the half-open interval g0,1).n This example can be elaborated from binary sequences to sequences of real variables without too much difficulty. There is an intimate connection between infinite random sequences and continuous probability distributions on the line, and understanding one class of problem is frequently an aid to understanding the other. The question often posed about the probability of some sequence predicted in advance being realized, say an infinite run of heads or a perpetual alternation of heads and tails, is precisely answered. In either the decimal or binary expansions, all the numbers whose digit sequences either tenuinate or, beyond some point, are found to cycle perpetually through a finite sequence belong to the set of rational numbers. Since the rasionals have Lebesgue measure zero in the space of the reals, we have a proof that the probability of any such sequence occurring
is zero.
Another well-known conundrum concerns the troupe of monkeys equipped with typewriters who, it is claimed, will eventually type out the complete works of Shakespeare. We can show that this event will occur with probability 1. For the sake of argument, assume that a single monkey types into a word processor, and his Ascll-encoded output takes the form of a string of bits (binarydigits). Suppose Shakespeare's encoded complete works occupy k bits, equivalent to kl5 characters allowing for a 3z-character keyboard (upper-caseonly, but including som
Stochastic Processes
181
punctuation marks). This stling is one o f the lk p ossible strings of k bits.
Assuming that each such string is equally likely to arise in k/5 random key presses, the probability that the monkey will type Shakespeare without an error from scratch is exactly 2 -k H ow ever the probability that the second string of lk bits it produces is the right one, given that the tirst one is wrong, is (1 2 '-k)2- w h en the strings are independent. In general, the probability that the monkey will type Shakespeare correctly on the m + llth independent attempt, '&2A11 these events given that the first m attempts were failures, is (1 2 -k) and summing k disjoint, probabilities a11 their 0 yields m over are 1. fmonkey types Shakespeare eventually) ,
.
-
-
.
=
In the meantime, of course, the industrious primate has produced much of the rest of world literature, not to mention a good many telephone books. It is also advisable to estimate the length of time we are likely tcj wait for the desired text to appear, which requires a further calculation. The average waiting time, l-k4m expresse d in units of the time taken to type k bits, is 2-E,1=1rrl(1 2k 1 If we scale down our ambitions and decide to be content with just $TO BE OR NOT TO BE' (5x 18 90 bits), and the monkey takes 1 minute over each attempt, we shall wait on average 2.3 x 1021years. s o the Complete Works don't really bear thinking about. What we have shown is that almost every infinite string of bits contains every t'inite string somewhere in its length', but also, that the mathematical concept of talmost surely' has no difficulty in coinciding with an everyday notion The example is frivolous, but it is useful to be indistinguishable from reminded occasionally that limit theory deals in large numbers. A sense of perspective is always desirable in evaluating the claims of the theol'y. =
-
-
.
=
'never'
.
The first technical challenge we face in the theory of stochastic processes is to handle distributions on R=. To construct the Borel field 000of events on R7 we implicitly endow R= with the Tychonoff, or product, topology. It is not essential to have absorbed the theory of j6.5 to make sense of the discussion that follows, but it may help to glance at Example 6.18 to see what this assumption implies. Given a process x f.il7 we shall write jz 7:k(x) (m,...,A): Rx F-> (jyk =
(
=
.
a)
for each k e EN,to denote the k-dimensional coordinate projection. Let C denote the collection of jnite-dimensional cylinder sets of R=, the sets C
=
(x e
In other words, elements
: ;u@) R oo
e E, E e Bk k e ,
s)
(124) .
of C have the fonn C
=
-1 (F) C:k
(1z 5) .
for some E e T k an d some finite k. Although we may wish to consider arbitrary finite dimensional cylinders, there is no loss of generality in considering the projections onto just the first k coordinates. Any tinite dimensional cylinder can ,
182
Theor.y of Stochastic Processes
be embedded in a cylinder of the form lk -1(F), E e f where k is just the largest of the restricted coordinates. The distinguishing feature of an element of C is that at most a tinite number of its coordinates are restricted. ,
C is a tield.
12.2 Theorem
Proof First, the complement in R= of a set C defined by (12.4)is a-1(Ff) C c = fx e R= : (ak(x)e Ec E e fl k =
,
,
(126) .
which is another element of C, i.e. Cc s C. Second, consider the union of sets C a-1(F') -1 k e C for E, E' G Bk C tp C' is given by (12.4) = Jck E) e C an d C' with E replaced by E tp E ' and hence Ck.l C' e C Tlzird if E e Bk and E' e Bmfor m > k, then Ex? m-ke Bm an d so the argument of the second case applies. . =
,
,
.
.
,
,
Fig. 12.1 It is not easy to imagine sets in arbitrary numbers of dimensions, but good visual intuition is provided by thinking about one-dimensional cylinders in R3 Letting @,y,z)denote the coordinate directions, the one-dimensional cylinder generated by an interval of the axis is a region of 3-space bounded by two infinite planes at right angles to the x axis (see Fig. 12. 1 for a cut-away representation). A union of x-cylinders is another x-cylinder, a collection of But the union and intersection of an x-cylinder with parallel y-cylinder are two-dimensional cylinder sets, a cross' and a column' respectively (see Fig. 12.2). These examples show that the collection of cylinder sets in R k for fixed k is in R3 is a not a field,' the intersection of three mutually orthogonal bounded not a cylinder set. The set of tinite-dimensionalcylinders is nbt closed under the operations of union and complementation (andhence intersection) except in an infinite-dimensional space. This fact is critical in considering c(C), the class obtained by adding the countable unions to C. By the lastmentioned property of unions, c(C) includes sets of the form (12.4)with k tending to infinity. Thus, we have the following theorem. .
.x
<walls'
.
twalls'
Kcube',
Stochastic Processes
12.3 Theorem c(r) ogy. I:l
=
183
f=, the Borel field of sets in R= with the Tychonoff topol-
Fig. 12.2 The condition of this result is something we can take for granted in the usual applications. Recalling that the Borel field of a space is the jmallest c-field containing the open sets, 12.3 is true by definition, since C is a sub-base for the product topology (see j6.5) and a11the open sets of R= are generated by unions and finite intersections of C-sets. To avoid explicit topological considerations, the reader may like to think of 12.3 as providing the definition of f=. One straightforward implication, since the coordinate projections are continuous mappings and hence measurable, is that, given a distribution on @=,=), finite collections of sequence coordinates can always be treated as random vectors. But, while this is obviously a condition that will need to be satistied, the real mtypractical method we have of defining distribproblem runs the other way. utions for infinite sequences is to assign probabilities to finite collections of coordinates, after the manner of j8.4. The serious question is whether this can be done in a consistent manner, so that in particular, there is one and only one p.m. on (R7S=) that corresponds to a given set of the finite-dimensional distributions. The affirmative answer to this question is the famous Kolmogorov consis'l'he
tency theorem.
12.4 The Consistency Theorem The goal is to construct a p.m. on @*,f*),and, following the approach of j3.2, the plausible first step in this direction is to assign probabilities to elements Of C Let p denote a p.m. on th space (R,f$, for k 1,2,3,.... We will say that this family of measures satisfies the consistency property if =
.
gt' )
=
g'-t'
X
R
m-k
)
(12.7)
Theory of Stochastic Processes
184
for E G Bk and al1 m > k > 0. ln other words, any k-dimensional distribution can be obtained from an v-dimensional distribution with m > k, by the usual operation of marginalization.
The consistency theorem actually generalizes to stochastic processes with uncountable index sets 1- (see 27.1) but it is sufficient for present purposes to consider the countable case. consistency theorem Suppose there exists a family of p.m.s (gkJwlich satisfy consistency condition (12.7).Then there exists a stochastic sequence (m,t e N J on a probability space @=,f=,g,)such that p is the p.m. the of coordinate functions (.L,...,Xk)'. u vector of tinite
12.4 Kolmogorov's -'r
=
The candidate
measure for
is defined for sets in C by
.x
FIC)
=
(12.8)
11,1(6-),
where C and E are related by (12.4).The problem is to show that g, is a p.m. on C. If this is the case, then, since C is a field and f= = c(C), we may appeal to the extension theorem (3.8+3.13)to establish the existence of a unique measure on (R*,%=) which agrees with g for all C e C. The theorem has a simple but important corollary. C is a determining class for (R7f=).
12.5 Corollary
E1
ln other words, if g and v are two measures on (R=,f=) and jtk finite k, then jt v.
vk for every
=
=
To prove the consistency theorem we require a technical lemma whose proof is beyond us at this stage. It is quite intuitive, however, and will be proved in a more general context as 26.23.
12.6 Lemma For every E G T an d such that pkE A') < n
0 there exists K, a compact subset of E,
>
-
.
In other words, a p.m. on the space (R Bk) has nearly all of its mass confined to a compact set; this implies in particular the proposition asserted in j8.1, that random variables are finite almost surely. ,
Proof of 12.4 We will verify that jt of (12.8)satisfies the probability axioms with respect to elements of C. When E Rk C R= so that the first two probability axioms, 7.1(a) and (b), are certainly satistied. To establish finite additivfor E e Tk E' e Tm and ity, suppose we have C-sets C gk-1(AD, an d C' zr-1(F') m m k k. If C and C' are disjoint, =
,
=
=
p,(O+ g,tC ') g( =
F) + jt m (F')
= gmE x
=
qvl-J7
=
tp E
,
pmEx R'&-$ + p,m(A'') ,
)
=
g,(c
kp
c')
,
(12.9)
where the second equality applies the consistency condition (12.7),and the third m-k and E z are disjoint if C and C z are. one uses the fact that E x R The remaining, relatively tricky, step is to extend tinite additivity to count-
Stochastic Processes
185
able additivity. This is done by proving continuity, which is an equivalent property according to 3.5. lf and only if the measure is continuous, a monotone 1 C or G' C has the property, g,(G) p,IC). sequence tCj e C ) such that CJ Since Cj Cc where g(C9 = 1 jttt7), it is sufficient to C implies consider the decreasing case. And by considering the sequence Cj C there also is no loss of generality in setting C 0, so that continuity implies g,tGl 0. To prove continuity, it is sufticient to show that if g(() 2 6: for some E > 0, for every j, then C is nonempty. lf C e C for some j k 1, then p,(G)= pkqlEj) for some set Ej G Bkct where ? /cU) ls the dimension of the cylinder Cj. By consistency, pkqlEp '1-
.
--
.1e
'l'
-
-
=
--
=
pmEj X R '''-U)
so there is no loss of generality in assuming ) for any m > 17U), kq) < < k41) < k42) We may therefore define sets F1 e LB that lU) < 1,...,./, by setting E) A) and ...
....
=
,
=
1.
=
Ei xR ko-k
is a decreasing sincefc/),t,-1
i
,
1
=
j
,...,
-
1.
(12.j())
euhsets(A1).t,-1,
sequence, so is the sequence of
for each j k 1.
Consider any tixed j. There exists, by 12-6, a compact set Kj bkkqfEj Detine the sets K1
=
E/2 y+1
&')<
-
c
Ej such that
(!g
.
.
! !)
Kix R kil-ki) e lk) by analogy with the A''1,and so define Fj
=
Bkct
(12. 12)
s)s c
( 12. 13)
O#t; e
.
'=1
Fj i E), and hence Dj ? Cj where Dj
=
lk -1
./
.
Applying 1.1(iii) and then 1.1(i), observe that Ej Fj -
=
Ej ch
./
j
K'ic
=
-
i=1
f=1
Ej K*i)
j
c
E*i -
f=1
r*f)
(12.14)
,
is decreasing. Hence where the inclusion is because the sequence tA'tl)'J.1 bkkitEj F./) f -
-
f=1
=
bklilE t, A-t,)
bkEi
-
Kit
< E/2.
(12.15)
f=1
The first inequality here is from (12.14) by finite subadditivity, which follows from finite additivity as a case of 3.3(iii). The equality applies consistency, and the secon (i inequa lity applies the summation of 2-f-1 Since Ej Fj and Fj are = disjoint and got.l g,(G)> : by assumption, it follows from (12.15) that bklh) pkqlFj) > e/2, and accordingly that Dj is nonempty. Now. we construct a Doint of C. Let Ixlij. i e (N1 denote a seauenc of ooints -
.
=
F/let??'yof Stochastic Processes
186
of R=
with
x(J) e
Dj for each j, so that
(X1U),...,Xw)U))lk(j)@U)) =
Note that for m
=
(12. 16)
Fj.
1,...,./,
1 Km, (XlU),...,Akm)U))'&(,?,)(xU))
(12.17)
=
by (12.12), where Kmis compact. Now 1et m be tixed. Our reasoning ensures that (12. 17) holds for each.j k m. A set in Rkm' is compact if and only if each of the coordinate sets in R is compact, so consider the bounded scalar scquences, fxibij, j r?z) for f 1,...,k4-). Each of these has a cluster point A'1, and we can use the diagonal method (2.36)to construct a single subsequence (Al with the = A': for each 1,...,k(-). By the compactness, property that Xijn) e Km i Em. This is true for every m e IN (X1,...,X(m)) Consider the point a7*e R= defined by 7v(,,,)(x*) for each m e N. (X1,...,X(m)) = Since x* e Cm for each m, we have x* G O'>;=ICk C, as required. . =
.-+
.
=
This theorem shows that, if a p.m. satisfying (12.7)can be assigned to the finite-dimensional distributions of a sequence m x is a random element of a We shall often wish to think of @=,f7jt) as derived probability space from an abstract probability space (f,T,#), and then we shall say that x is ble if F1(A>)e 5 for each event E e S*. This statement implies the 5lB coordinates Xt are F/f-measurable r.v.s for each /, but it also implies a great deal more than this, since it is possible to assign measures to events involving countably many sequence coordinates. ('R*,fO,g,).
''O
-measura
12.5 Uniform and Limiting Properties Much the largest part of stochastic process theory has to do with the joint distributions of sets of coordinates, under the general heading of dependence. Before getting into these topics, we shall deal in the rest of this chapter with the various issues relating exclusively to the marginal distributions of the coordinates. Of special interest are conditions that limit the random behaviour of a sequence as the index tends to infinity. The concept of a unform condition on the marginal distributions often plays a key role. Thus, a collection of rov.s tX,r, T e is said to be uniformly bounded in probability if, for any 8 > 0, there exists Bg < x such that -1)
#( IXzI sup ' It is also said to be unjrmly
)<
Lp-bounded for p
supI1ALll?,B <
<
=.
(12. 18)
:. >
0, if
(12.19)
'
For the case p = x, (12.19)reduces to the condition, supmlXzI < x a.s. ln this case we just say that the sequence is uniformly bounded a.s. For the case p = 1, we have sups f'IXz I < x and one might think it conrct to refer to this property as integrability' Unfortunately. this term is already in use for a ,
Suniform
,
Stochastic Processes
different concept (see the next section) and so must be avoided here. Speak of tuniform fal-boundedness' in this context. To interpret these conditions, recall that in mathematics a property is said to hold unifonnly if holds for all members of a class of objects, including the limits of any convergent sequences in the class. Consider the case where the /. Random collection in question is itself a sequence, with 'T = E$ and with probability 1, and for each finite t e IN,#( 1Al Be variables are finite x, > holds for for The < always point of unifonn Bet 0. : bound is some any a <s to ensure that the constants Bet are not having to get larger as t increases. p > 0, by Liapunov's inequality. Also, uniform fv-boundedness for anyp > 0 implies uniform boundedness in probability; the Markov inequality gives 'r
=
tbounded
,
-->
-x).
-->
,
#(IAI
B4
(12.20)
terminology we sometimes speak of fvboundedness in the case of (12.18). A standard shorthand (dueto Mann and Wald 1943b) for the maximum rate of (positive or negative) increase of a stochastic sequence uses the notion ot Oh' notation unifonu boundedness in probability to extend the tBig Oh' and real E lf, for exists ordinary 0, > Be < x such for there sequences (see j2.6). that the stochastic sequence fXn )T satisfies supa't 1XnI > B < E, we write Xn 0p(1). lf (L)T is another sequence, either stochastic or nonstochastic, and XnlYn = 0p(1) we say that Xn Op(L), or in words, &Xnis at most of order Fn in x, 0 as n tN(1); more we say that Xn Probability' If #( IXnI > :) t7p(1), when words, in GXn is of order lss than Xnlk'n generally Xn o (L') or r l'n in probability The main use of these notations is in manipulating small-order terms in an expression, without specifying them explicitly. Usually, Fn is a positive or negative power of n. To say that Xn tV1)is equivalent to saying that Xn converges in probability to zero, following the terminology of j12.2. Sometimes Xn Op(1) is defined by the condition that for each s > 0 there exists Be < x and an integer Ng > 1 such that #( IXnl > B < s for a1l n Nz. But Xn is finite almost surely, and there necessarily exists (this is by 12.6) a constant BL< #E') < E for 1 f n < Ne. For al1 possib1y 1arg er than Be, such that #( IXn l > practical purposes, the formulations are equivalent. Ktaittle
=
=
--)
-->
=
.
=
=
.
=
=
=,
Iheor.y of Stochastic Processes
188
12.6 Uniform lntegrability lf a r.v. X is integrable, the contributions to the integral of extreme X values must be negligible. ln other words, if FIXI f(IX1 1(IxlzM))
-->
0 as M
(12.21)
-->
x.
However, it is possible to construct unifonnly Al-bounded sequences l#nl which fail to satisfy (12.21)in the limit.
12.7 Example Detine a stochastic sequence as follows: for n 1,2,3,... let Xn = 0 with probability 1 1/rI, and Xn = n with probability 1/a. Note that E( lXn l) = nln = 1 for every n, and hence the sequence is uniformly Al-bounded. But to have =
-
limE IXnIlt lxalA/l)
=
0
(12.22)
MM=
uniformly in n requires that for each e > 0 there exists Ms such that E lXnl 1( xajkvl) < e for al1 M > Mo uniformly in a. Clearly, this condition fails, for e: < 1, in view of the cases n > Me. n Something very strange is going on in this example. Although E(Xn4= 1 for any n, x. To be precise, we may show that Xn = 0 with probability approaching 1 as n 0 (see 18.15). The intuitive concept of expectation appears to fail when Xn faced with nv.s taking values approaching intinity with probabilities approaching --+
-613...:
Zero.
The unform integrability condition rules out this type of perverse behaviour in a sequence. The collection (Xz, $ e T'J is said to be uniformly integrable if
sup 1im F( lXx11( xvl
Te
7
M)
M-yx
)
=
0.
(12.23)
ln our applications the collection in question is usually either a sequence or an array. In the latter case, uniform integrability of (Xkf) (say) is defined by taking the supremum with respect to both t and n. The following is a collection of theorems on unifonn integrability which will tind frequent application later on; 12.8 in particular provides insight into why this concept is so important, since the last example shows that the conclusion does not generally hold without uniform integrability. 12.8 Theorem Let E(Xn)
--$
E (A').
be a unifonnly l.YnlT
integrable sequence. If Xn -6J- X, then
Proof Note that E iXn I) = E IXnl 1( !xnI
s A.T+
Ftlxa
+
E lXnl 1( IxaIkM))
l1( jx,,1kM)).
(12.24)
By choosing M large enough the second tenn on the right can be made uniformly small by assgmption, and it follows that FIXnj is uniformly bounded. Fatou's
stochasticProcesses
189
lemmaimplies that FIXl < x, and F(m exists. Define l',, I.L --YI so that i',z 0 a.s. Since Fn < IaLl+ l-Yl by the triangle inequality, 9.29 gives =
f 2F( IXnl 1( lxnlxv/all
F(i'n1(yn>M))
+
2F( lXl 1( IxI>v/2)).
The second right-hand-side term goes to zero as M integrable if (A%) is. We may write f(Fn)
'(l%1(yn<M))
=
+
--
,
.--.)
so
x,
(12.25)
f Fn) is
A'(L1(yn>v)),
uniformly
(12.26)
and by the bounded convergence theorem there exists, for any E > 0, Ne such that F(l%1(ya<>)) < E/2 for n > #:, for M < oo. M can be chosen large enough that F(L1( ys>>)) < :/2 uniformly in n, so that EYn) < E for n > Nv., or, since 6: is 0. But arbitrary, EYn) --.-)
EYnq
by the
lnodulus
F(I-L--YI) k
=
IF(Xn) -F(ml
(12.27)
inequality, and the theorem follows. w
The next theorem gives an alternative form for the condition in often more convenient for establishing uniform integrability.
(12.23)which
is
12.9 Theorem A collection fXs, e 1-) of nv.s on a probability space (f1,1,#)is uniformly integrable iff it is uniformly Al-bounded and satisties the following condition: V E, > 0, 3 8 > 0 such that, for E E 5, ':
sup (F41 Xzl 1s) l < z.
P(E4 S
(12.28)
': e 'T
Proof To show sufficiency, fix : > 0 and 5 1, Markov inequality for p
E
7. By Al-boundedness and the
=
E
#(1-V1k Af) f and for M large enough, (12.28), it follows since
lxzI < M
(12.29)
x,
#(l-YzI Af) S ,1
6, for any is arbitrary that
>
0. Choosing
lA'(lX,rl 1f Ixzlwzlll < E,
Sup
to satisfy
(12.30)
TeT
and (12.23)follows since
E is arbitrary. note that, for any E e 5 and
To show necessity, F(1 Xvl 1s)
=
'r
E
1-,
1s1( Ixz1<MJ) + .E'(I.YZI 1s1( lxvI&z)) l-Y'rI
E
< MPCEI+
F(1 -Ysl1( Ixzlkvl).
(12.31)
Consider the suprema with respect to of each side of this inequality. For e > 0, (12.23) impliej there exists M < t:'o such that 'r
1F( 1AL1 1s)l
sup T
Uniform Al-oundedness
s MPCE) +
r1:.
now follows on setting F
=
f, and
(12.32) (12.28)also follows
Theor. of Stochastic Processes
190 with 8
<
zllM.
*
Another way to express condition (12.28)is to say that the measures vztp = ## must be absolutely continuous with respect to #, uniformly in :. Finally, weprove aresultwhich showswhy theuniformboundednessof moments of a given order may be important.
Jslxzl
12.10 Theorem lf 1+8
FIXzI for 0
>
0, then limv-pxlx'rl
Proof Note that
1( IxzIuv))
E IX,t11+e
(12.33)
< cxn =
0.
1+91
s (jx z j t l-vzlkxf)) S?E (IX1I1( jxvlzwzl) 2
for any 0 > 0. ne result follows on letting M (12.34) is finite by (12.33)..
..-+
x,
(12.34) since the majorant side of
Example 12.7 illustrated the fact that uniform fal-boundedness is not sufficient for unifonn integrability, but 12.10 shows that unifol'm falma-boundedness is sufficient, for any 0 > 0. Adding this result to those of j12.5, we have established the Merarchy of uniform conditions summarized in the following
theorem. 12.11 Theorem Uniform boundedness a.s.
uniform uniform = = uniform = uniform zz:
None of the reverse implications hold.
fw-boundedness, p > 1 integrability fw-boundedness, 0 < p S 1 boundedness in probability. n
13 Dependence
13.1 Shift Transformations We now consider relationships between the different members of a sequence. ln what ways, for example, might the joint distdbution of Xt and Xt-k depend upon k, and upon /? To answer questions such as these it is sometimes helpful to think aboutthe sequencein anew way. Havingintroducedthenotionof arandom sequence as a point in a probability space, there is a useful analogy between studying the relationships within a sequence and comparing different sequences, that is, different sample outcomes (l) e f. F-> copsider a 1-1 measurable mapping, F: ln a probability space (f1,@,#), (onto). This is a rule for pairing each outcome with another outcome of the space, but if each ) e f maps into an infinite sequence, Finduces a mapping from one sequence to another. F is called measure-preserving if PCTEI PE) for a11E e F. The shift transformation for a seqence (X/)JT is defined bylo =
X(F)
=
X,+1().
(13.1)
F takes each outcome ) into the outcome under which the realized value of X occurring in period t now occurs in period t- 1, for every t. In effect, each coordinate of the sequence from t 2 onwards is relabelled with the previous Xt-vk), the relationship period ' s index. More generally we can write XtlTkl between points in the sequence k periods apart becoming a characteristic of the transformation F k Since Xt is a nv. for a1l f, both the shift transformation and its inverse F -1 th e backsh;ft transformation, must be measurable. Taken together, the single r.v. X1(): f r- R and the shift transformation F, can be thoght of as generating a complete description of the sequence (X/))JT. This can be seen as follows. Given #14), apply the transformation F to ), and each ( e ( defines the mapping A%((0):f F-> R, obtain Xz = X1(F(0). Doing this and we are ready to get Xg X2(F(0). ltrating the procedure generates as many points in the sequence as we require. =
=
.
,
tor
=
be a sequence of coin tosses (with1 13.1 Example Consider 12.1, Let (Xf())JT for heads, 0 for tails) beginning 11010010001 1... (say).Somewhere on the intelwal g0,1) of real numbers (in binary representation), there is also a sequence (A()') J7 beginning 10100100011..., identical to the sequence indexed by (l) apart om the dropping of the initial digit and the backshift of the remaindr by one position. Likewise there is another sequence fXt(''4 1T,a backshifted version of ' ) 17b eg inning 0100101 1...; and so foih. If we define the transformation taVt(0
Theory of Stochastic Processes
192
)j'' F by F -1 ) = (l) r-1)' etc. the sequence (Xl(() )T can be constructed as 1-J ) )T,t h at is the sequence of first members of the sequences thesequence lm(F foundby iterating the transformation, in this case beginning 1,1,0,... I:l '
,
=
,
,
'
,
This device reveals, among other things, the complex stnlcture of the probability space we are postulating. To each point (.t) e fl there must correspond a countably infinite set of points F ) e f w hich reproduce the same sequence apart from the The intertemporal properties of a sequence can absolute date associated with then be treated as a comparison of two sequences, the original and the sequence lagged k periods. Econometricians attempt to make inferences about economic behaviour from recorded economic data. In time-series analysis, the sample available is usually a single realization of a random sequence, economic history as it actually occurred. Because we observe only one world, it is easy to make the mistake of looking on it as the whole sample space, whereas it is really only one of the many possible realizations the workings of historical chance might have generated. Indeed, in our probability model the whole economic universe is the counterpm of a single random outcome ); there is an important sense in which the time series analyst is a statistician who draws inferences from single data points ! But although a single it is realization of the sequenc can be treated as a mapping from a single linked to a countably infinite set of s corresponding to the leads and lags of the seqpence. A large part of our subsequent enquiry can be summarized as posing the question: is this set really rich enough to allow us to make inferences about # from a single realization? ,
m.
,
13.2 Independence
nnd
Stationarity
Independence and statioharity are the best-known restrictions on the behaviour of a sequence, but also the most stringent, from the point of view of describing economic time series. But while the emphasis in this book will mainly be on finding ways to relax these conditions, they remain important because of the many classic theorems in probability and limit theory which are founded on them. The degree to wlch random variations of sequnce coordinates are related to those of their neighbours in the time ordering is sometimes called the memory of a sequence; in the context of time-ordered observations, one may think in tenns of the amount of information contained in the cunrnt state of the sequence about its previous states. A sequence with no memory is a rather special kind of object, because the ordering ceases to have signiticance. lt is like the outcome of a collection of independent random experiments conducted in parallel, and indexed
arbitrarily. When a time ordering does nominally exist, we call such a sequence serially independent. Generalizing the theory of j8.6, a pair of sequences ((X,())T, l Fr() JT1e R= xR*, is independent if, for a11F1, E1 e 5*, /:()1.Y,17 e F1, (F,1: e E,4 Accordingly, a sequence
(A())T
=
#(tx,)7
e Ej4pf
r,)7 e
F2).
(13.2)
is serially independent if it is independent of
Dependence
193
fXt(T k(0) IT fOr a 11k ls 0 This is equivalent to saying that every tinite collection of sequence coordinates is totally independent. Serial independence is the simplest possible assumption about memory. Similarly, looking at the distribution of the sequence as a whole, the simplest treatment is to assume that the joint distribution of the coordinates is invariant with respect to the time index. A random sequence is called strictly stationary if Tllis implies that the sequences the shift transformation is measure-preserving. fXf 17 and fAkJ7=1have the same joint distribution, for every k > 0. Subject to the existence of particular moments, less restrictive versions of the condition are also commonly employed. Letting w E(Xt), and Xkt Covxt,xtn), consider those cases in which the sequence (g'rl7 and also the array ( fXmt are well defined. If w g,, al1 /, we say the sequence is mean J';=()l7wl, where (y,,l stationaty. If a mean stationary sequence has 17'is a sequence wt w of constants, it is called covariance stationaty, or wide sense stationary. If the marginal distriution of Xt is the same for any t, the sequence IXCJis said to be identically distributed. This concept is different from stationarity, which also restricts the joint distribution of neighbours in the sequence. However, Fhen a stochastic sequence is both serially independent and identically distributed (or i.i.d.4, this suftices for stationarity. An i.i.d. sequence is likeanarbitrarily indexedrandom sampledrawngomsomeaunderlyingpopulation. The following clutch of examples include both stationary and nonstationary .
.1
=
=
=1,
=
=
CaKS. (52 < 13.2 Example Let the sequence (6>J b e i i d with mean 0 and variance where fA)7, and let (0j17be a square-summable sequence of constants. Then ''O -x
.
.
(x,
.
*
A
=
77bet-j,
(13.3)
jcst
is a covariance stationary sequence, with Ext) t. This is the inhnite-order moving average additional details. n 13.3 Example lf
kt
=
0 and Ext)2 = c J e? ) sr every (MA(=)) process. See j14.3 for 2:.,.0
,0
is i.i.d. with mean 0, and Xt
for a constant a, then Ex
=
=
cos at
+
(13.4)
st
cos at, depending systematically on /.
I:a
13.4 Example Let (xV)be any stationary sequence with autocovariance sequence (ym, m 0J The sequence (.X)+ A%) has autocovariances given by the array .
l
'o
+
and hence it is nonstationary.
'sm
+
Yt+
'tt,vm,
m 2 0, t k 1 l
,
u
13.5 Example Let X be a r.v. which is symmetrically distributed about 0, with variance c 2 If Xt (- 1)W then (X,)Tis a stationary sequence. In particular, .
Ext)
=
=
,
0, and Cov(Xt,Xt+k4 5.2 when k is even and =
-c2
when k is odd, independent
Theory of Stochastic Processes
194 of t.
(:1
Theseexamples show that, contraryto acommon misconception, stationarity does not imply homogeneity in the appearance of a sequence, or the absence of periodic patterns. The essential feature is that any patterns in the sequence do not depend systematically on the time index. It is also important to distnguish between stationarity and limited dependence, although these notions are often closely linked. Example 13.4 is nonstationaty in view of dependence on initial conditions. The square-summability condition in 13.2 allows us to show covariance stationarity, but is actually a limitation on the long-range dependence of the process. Treatments of time series modelling which focus exclusively on models in the linear MA class often fail to distinguish between these propeties, but examples 13.3 and 13.5 demonstrate that there is no necessary connection betwen
them.
'
Stationarity is a strong assumption, particularly for the description of empirical time series, where features like seasonal patterns are commonly found. It is nonstationarity, sometling we might think of useful to distinguish between capable of elimination by local averaging of the coordinates, and global' as nonstationarity, involving features such as persistent trends in the moments. If sequences (Xr)7=1 and (a'+k)7=1have the same distribution for some (not necessarily every) k > 0, it follows that (.+x)7.1 has the same distribution as (X)+l7=1, and the same property extends to every integer multiple of k. Such a sequence accordingly has certain stationary characteristics, if we think in tenns of the distributions of successive blocks of k coordinates. This idea retains x. Consider the limit of a tinite sequence of length n, divided force even as k into (rl j blocks of length gn 1,plus any remainder, for some a between 0 and 1. here denotes the largest integer below x.) The number of blocks as well (Note, E.;r) as their extent is going to infinity, and the stationarity (or otherwise) of the sequence of blocks in the limit is clearly an issue. Important applications of these ideas arise in Parts V and VI below. It is convenient to formulate a definition embodying this concept in terms of moments. Thus, a zero-mean sequence will be said to be globally covariance stationary if the autocovariance sequences (y/uf are Cesro-summable for each )*r.1 0, where is strictly positive the k in the Cesro sum case of the variances m = m 0). The fllowing are a pair of contrasting counter-examples. ilocal'
-->
13 6 Example A sequence with
'p,
/l5
-
@
is globally nonstationary for any
j
#
0. n
13.7 Example Consider the integer sequence beginning 1,2,1,1,2,2,2,2,1,1,1,1,1,1,1,1,...*, i.e. the value changes at points t 2 k k = 1 2 3 The cesro sum of this x. It fluctuates eventually between the points sequence fails to converge as n lk k = k even. A stochastic sequence having a 5/3 at n 2 k o dd and 4/3 at n of form globally nonstationary. (:1 is variance sequence this =
,
-->
,
,
=
,
,
,
,.,.
Dependence
195
13.3 lnvariant Events The amount of dependence in a sequence is the chief factordetermining how informative a realization of given length can be about the distribution that generated it. At one extreme, the i.i.d. sequence is equivalent to a true random sample. The classical theorems of statistics can be applied to this type of distribution. At the other extreme, it ij easy to specify sequences for which a single realization can never reveal the parameters of the distribution to us, even in the limit as its length tends to intinity. This last possibility is what concerns us most, since we want to know whether averaging operations applied to sequences have useful limiting properties', whether, for example, parameters of the generation process can be consistently estimated in this way. Toclarify theseissues, imaginerepeatedsamplingof random sequences (A((t)) JT', in other words, imagine being given a function X1(.) and transfonnation F, making repeated random drawings of (.t) from fl and constructing the corresponding random sequences', 13.1 illustrates the procedure. Let the sample drawings be denoted j, 1,...,N, and imagine constructing the average of the realizations at some i ,=1 X is called an ensemble average, fixed time / e. The average kxto N-1X% o((tk) which may be contrasted with the time average of a realization of length n for n-11J=1Xj()j). Fig. 13.1 illustrates this procedure, some g iven )j e D, Xntj) showing a sample of three realizations of the sequence. The ensemble average is the average of the points falling on the vertical line labelled /c. lt is clear and the ensemble average as n and N that the limits of the time averg respectively go to infinity are not in general the same; we might expect the ensemble average to tend to the marginal expectation Ext ), but the time average 8 will not do so except in special cases. If the sequence ls nonstationary Ext depends upon /o, but even assuming stationarity, it is still possible that different realizations of the sequence depend upon random effects which are common to a1l t. =
=
,
=
O O
O
O O O
O O
O
O
Fig. 13. 1
Theory of Stochastic Processes
196
In a probability space (f1,F,#) the event E e 5 is said to be invariant under a transfonnation Tif PCTEh F) 0. The criterion for invariance is sometimes given as TE F, but allowing the two events to differ by a set of measure zero does not change anything important in the theory. The set of events in 5 that are invariant under the shift transformation is denoted #'. =
=
13.8 Theorem
,@
is a c-tield.
Proof Since F is onto,
is clearly invariant. Since F is also 1-1,
TEc A E C
CTE4cA Ec
=
(13.5)
TE A E
=
by definition. And, given 3En e #, n e IN), PCTE,tA En)
PITEn
=
&) + P(En
-
-
TEn4 = O
(13.6)
for each n, and also F
UEn
A
n
UEn UTEn A UEn,
(13.7)
=
n
l
n
using 1.2(i). By 1.1(i) and then 1.1(iii),
UTEn UEn U TEn OEL ? U(Ff n
and similarly,
f'n
=
-
n
n
-
n
&),
(13.8)
?1
n
UEn UTEn U(En n
-
n
TEn).
-
(13.9)
n
= 0 The conclusion #gT(Un&) A (Un&)1 now follows by . proof. pletingthe
(13.6)and
3.6(ii), com-
An invariant random variable is one that is J/f-measurable. An invariant nv. Z() has the property that Z(F) =Z(), and an J/f-measurable sequence (Z,()) ):' Z1()) a.s. for every t. The invariant events is trivial in the sense that Zt and associated r.v.s constitute those aspects of the probability model that do not alter with the passage of time. =
= Ff((z)) + Z(), where Xt 13.9 Example Consider the sequence (AX()))) ( F,4(0)) random and example of invariant Z((0) An being a event is E an sequence a r.v. f (.0: Z() K z, Ff() e R ). Clearly E and FF are the same event, since Z is the only thing subject to a condition. Fig. 13. 1 illustrates this case'. lf ( Ff()) ) is a zero-mean stationary sequence, the figure illustrates the cases Z((l)1) = Z((.0:) 0, and Z((l)3) > 0. Even if EZ4 out' in the limit, = 0, the influence of Z() in the time average is not as it will be from the ensemble average. I:a =
=
The behaviour of the time average as n fundamental result.
--
cxn
is summarized by the following
be 13.10 Theorem (Doob 1953: th. X.2. 1) Let a stationary sequence (.&((.t)))T defined by a measurable mapping X1()) and measure-preserving shift transform-
Dependence
ation T, such that
m()
J-1
.Y1(F
=
1imSnbln
), an d
I
Z)=1m()).If
=
F( X1j
,7)()),
ECXL
=
1et Sn ()
x,
(13.10)
(:I
a.s.
<
n-+oo
In words, the limiting case of the time average can be identitied with the mean of the distribution conditional on the c-tield of invariant events. The proof of 13.10 requires a technical lemma.
A(I3) ((t):supatkst))
13.11 Lemma Let
=
Jvca(oXl())J#()
-
nI))>
0) Then for any set M e #', .
2 F#(Mf-hA(p)).
(13.11)
Proof We establish this for the case fs 0. To generalize to any real p, consider the sequence fA p) which is stationary if (XCIis stationary. > 0), the set of outcomes Write A for A(0), and let hj f ): maxlskust) for which the partial sum is positive at least once by time j. Note, the sequence x. Also 1et A as j f A.j) is monotone and hj =
-
,
=
'1h
--
N nj
T-ih
=
-j
n
:
=
max Sklf l Kkf n
>
0
(13.12)
.
'-j
Since k
j+k
&(r?(o) XxtT) =
!F71rf(e)) sjmke)SjTb, z-
=
>1
by defining & Nnj
=
(13.13)
-
J=./m 1
0 we may also write ):
=
max j- 1f k f n
sk
-
Sjbh) > 0
,
0 f j Kn
-
1
.
(13.14)
This is the set of outcomes for which the partial sums of the coordinates f'rom j + 1 to n are positive at least once, and we have the inequality (explained below) n-1
XA+1()
lxnytl
2 0, all
)
e D.
(13.15)
Integrating this sum over the invariant set M gives n-1
n-1
r
0 X j=0 J <
u-suxjwtodrb
v
n-1
m
'0
Jn
v
R'
''nn
X y=o
j,.x.y&+1(rVt,))J#()
n
g
=X j u-g jcz0
=
.X1(9d#((9) J-
=
Jv.ayXl(t9V#(), X jc
(13.16)
1
where the tirst equa lity uses the fact that hn-j = (: T-l(kbe Nnjj and the measure-preserving property of F, and the third is by reversing the order of summation. The dominated convergence teorem applied to fm lMraayl, with I I as the
m
Theory of Stochatic
198
Processes
dominating function, yields
(13.17) Jxawxltltf#tt9l. Jsyraayxltllptl -->
limit by 2.26, so that, as required,
This limit is equal to the Cesro
-1
j'ucvs m()##()
=
1im a-,-
The inequality in
n
Xnj
m()##4)) 0. .
(13.18)
ucxxj
s,
not self-evident, but is justified as follows. The having the property expression on the left is the sum containing only those Xt realization that in the partial sums from the point t onwards are positive at least once, otherwise the tth contribution to the sum is 0. The sum includes only Xt lying in segments of the sequence over which Sk increases, so that their net contribution must be positive. It would be zero only in the case Xt S 0 for 1 f t S n. Fig. 13.2 depicts a realization. for k shows values of Sk 1,...,a, vertical shows the separations between successive so the .Yf((0)are the running sum of the terms of (13.15).The coordinates where the Xt are to be omitted from (13.15) are arrowed, the criterion being that there is no to the which exceeds the right current
(13.15)is ,
'o'
=
&o'
<+'
.
to'
'o'.
+
+
+
* *
*
0
O
*
*
Fig. 13.2 Proof of 13.10 The first step is to show that the seqpence (Snlbln ITconverges almost surely to an invariant r.v. Sl). Consider 5'() = limsupa Snlln. RF)
=
lim sup vl--hx nkm
&+l() n+ 1
n+ 1 n
X1() =
-
n
)),
(13.19)
ltl = liminfn Snln. Hence, so that Rl is invariant, the same being true for sttt)) = M(a,p) invariant. Since i s supnsnln (x < < < the event j3 V((t)ll (: A(p) M(a,p) A(p) where defined in the statement of 13.11. Hence, is V1, M(a,j) in the lemma gives putting M =
lDependence
J>(oj)X1(t0V#())2 But now replace
(
):
supnt-xsktl/n)
199
p#(M(?'p)).
(13.20)
13.11 by -AX)), and observe that M(a,p) Hence we get
in AX(t)) -a)
>
c
A(-a)
=
.
Jsy(a,j)X1(t0V#(tt9 S tx#(M(?'p))'
(13.21)
Since the left-hand sides of (13.21)and (13.20)are equal and (x < p, it follows 5'() with probability 1. that #(M(a,p)) = 0; that is, R)) 8)) This completes the first stage of the proof. It is now required to show that S = Exj I#) a.s., that is, according to equation (10.18),that =
juxltdp Sinc
=
jsjdp,
=
each M e
(13.22)
M is invariant, ''t
j
juxLlkdp =
a-
n
.,yt
X >1
vr
j.
Jyrto, ju ,
X/t0)##((t)
=
-s
Snl
:#4tl))
(13.23)
and the issue hinges on the convergence of the right-hand member of (13.24)to
E(S3u4. Since the sequence (-Y/Jis stationary and integrable, it is also uniformly integrable, and the same is true of the sequence ( F,l, where Ff Xtsu and M e #. For : > 0, it is possible by 12.9 to choose an event E e 5 with #(& < 6, such that sup E 1FJ I1e,) < e. For the same E, the triangle inequality gives =
j
E
j) j s )x-(J yn,-, >1
#,)
Ii-,I E
dp
>1
< :
(13.24)
.
By the same argument, also using stationarity and integrability of Ff,
I j
J1 n-
n
y-, dp
-,
s
Hence by 12.9 the sequence n-v I>. If n-l n n n -11 F) 12.8, by =1
=
J
complete. w
n
,.j
jl
yl
),; jap
=
s j yj j
(13.25)
tn-1Z:=1Ff)
-->
i s a 1so uniformly S a.s., it is clear that n-S
-1
Mn
Since n is arbitrary,
n
-1
:#((t))
--y
( Sudp.
2M
n
integrable, where Sku a.s., so
1M
-->
(13.26)
(13.26)and (13.23)together give (13.22),and the proof
is
13.4 Ergodicity and Mixing The property of a stationary sequence which ensures that the time average and the ensemble average have the same limit is ergodicity, which is defined in terms of
I'heory of Stochastic Processes
200
the probability of invariant events. A measure-preserving transformation F is l or #() is the c-field of 0 for a11E e Ds where ergodic if either #(F) under F. A stationary sequence (Xf() )+-7is said to be ergodic if invariant events J-1 =
x/
=
Xl (F ) for every t where F is measure-preserving and ergodic. Some authors, such ag Doob, use the tenn metrically transitive for ergodic. Events that are invariant under ergodic transformations either occur almost surely, or do not occur almost surely. ln the case of 13.9, Z must be a constant almost surely. Intuitively, stationarity and ergodicity together are seen to be sufficient conditions for time averages and ensemble averages to converge to the same limit. is the mean not just of X1 Stationarity implies that, for example, g, f(m()) but of any member of the sequence. The existence of events that are invariant under the shift transformation means that there are regions of the sample space which a particular realization of the sequence will never visit. lf PIFFA E) = yhen the event Ec occurs with probability 0 in a realization where E occurs. However, if invariant events other than the trivial ones are ruled out, we ensure that a sequence will eventually visit a1l parts of the space, with probability 1. In this case time averaging and ensemble averaging are effectively equivalent
X()
=
=
t
(
operations.
)
The following corollary is the main reason for our interest in Theorem 13.10.
13.12 Ergodic theorem Let (.X)4(0)IT be a stationary, ergodic, integrable sequence. Then 1imSnelln
=
(13.27)
F(X1), a.s.
n-4-
Proof This is immediate from 13.10, since by ergodicity, EIXj I#)
F(X1) a.s. w
=
In an ergodic sequence, conditioning on events of probability zero or one is a is trivial operation almost surely, in that the information contained in trivial. The ergodic theorem is an example of a 1aw of large numbers, the first of several such theorems to be studied in later chapters. Unlike most of the subsequent examples this pne is for stationat'y sequences. Its practical applications in econometrics are limited by the fact that the stationarity assumption is often inappropriate, but it is of much theoretical interest, because ergodicity is a very mild constraint on the dependence, as we now show. Atransformation thatis measure-preserving eventually mixes up the outcomes in a non-invariant event A with those in Ac. The measure-preserving property rules out mapping sets into proper subsets of themselves, so we can be sure that FA jAc is nonempty. Repeated iterations of the transformation generate a sequence of sets l T V ) containing different mixtures of the elements of A and Ac. A positive AC; that is, if implies a negative dependence of # on dependence of # on #(A f-h B) > P(A4PB) then P(Ac /'A Bj PB4 #(A f-'h #) < #(#) #(A)#(#) PIAc)PCB). Intuition suggests that the average dependence of B on mixtures of A and Ac should tend to zero as the mixing-up proceeds. ln fact, ergodicity can be characterized in just this kind of way, as the following theorem shows. ,/
:4
=
-
-
=
llependence
201
shift transfonnation Fis ergodic if and only 13.13 Theorem A measure-preserving for pair of if, events A, B e @, any n
1
-xpvkx
lim
rasl
(13.28)
PA)PB).
=
k=1
n-yx
Proof To show ton 1y ito let A be an invariant event and B A. Then PCTPAf'n B) = #(A) for a11 k, and hence the left-hand side of (13.27)is equal to #(A) for all 1' #(A) 2 implying 84) 0 or 17.7 k. This gives #(A) apply the ergodic theorem to the indicator function of the sets To show and ergodic, to give F V w h re T is measure-preserving =
,
=
=
,
iif'
,
,
1
n
lztxt(0l lim X &1
=
-
(13.29)
#(4), a.s.
n-yx
But for any # e 5,
jj x-1,u((o) -)
-
,(.)
k=1
j
dpv
zj I jJst) x,.-tx-
1,u((o)
#IN
-
&1
lztxt(')l
,(x)
!
dp
I
-#(A))dz'((o)
1
n
=
1 -sX#(F
k
A rhB) -PA4PB4
.
(13.30)
k=1
The sequence whose absolute value is the integrand in the left-hand member of cxl by (13.30) converges almost surely to zero as n (13.29);it is bounded uniformly + is clearly integrable. Hence, uniformly in 1 #(A) absolutely by n, so the left-hand member of (13.30)converges to zero by 12.8, and the theorem -->
follows.
.
Following from this result, ergodicity of a stationary sequence is often associated with convergence to zero in Cesro-sum of the autocovariances, and indeed, in a Gaussian sequence the conditions are equivalent.
13.14 Corollary If (Xr4(4)))Tis a stationary, ergodic, square-integrable sequence, then n 1 XCov(XIm)
*n al
--
0 as n
(13.31)
-+
x.
Proof Setting B A and defining a real sequence by the indicators of F A, Xkk equivalent to (13.31).First extend this result to sequencs = 1wu((o), (13.28)is X,'%1Aj(), so that A%((t))= X1(T) Lwlr-ujt). of simple r.v.s. Let X1(() = The main point to be established is tha.tthe differencebetween X1 and a simple r.v. can be ignored in integiation. In other words, the sets F Aj must form a =
=
.j
Theot'y of Stochastic Processes
202
partition of f1, apart possibly from sets of measure 0. Since F is measureAf)) #(UjAj) 1, using 1.2(ii), and hence #(F-1(Uf preserving, #(UF-V 1, additivity of the #(f1 UjF -1Ai) = 0. And since L#(F-V i ) = Zf#(Af) z4jl -1 is a1so disjoint apart f'rom possible measure implies that the collection (F verifying required the property. sets of measure 0, This argument extends by induction to Xk for any k e N. Hence, =
=
=
=
-
7txlxzlE 777rcslxow-u/) j =
i
= 7) 77a,uyrtA
rn T-kAjl,
,w1z'',4
7/... $J
,.-u ' .7
j
-
t.
.
(13.32)
/ (the sum being absolutely convergent by assumption), and by 13.13, n 1 TExk)
lim n -
n--yx
=
X ywm
&1
i
lim
'
'-
j
n-yx
-
n 1 X#(Af
n
-z
rn T Aj)
&1
afW#(Xf)#(Ay) = Xi X j
&XI)2,
=
(13.33)
whereEXj4 2 EX 14EXk) for any k, by stationarity. The theorem extends to generalsequences by the usual application of 3.28 and the monotone convergence theorem.> =
This result might appear to mean that ergodicity implies some fonn of asymptotic independence of the sequence, since one condition under which (13.31)certainly x. But this is not so. The following holds is where Cov(X),XJ 0 as k example illustrates nicely what ergodicity implies and does not imply. -->
-->
13.15 Example Let the probability space (f,T,#) be defined by fl = f0, 1 J, so that 5 = (fO l 0),f 1 l (0,1)) and #() = 0,5 for (.0 0 and (l) = 1. Let F be the transfonnation that sets F0 = 1 and F1 = 0. ln this setup a random sequence and generating the sequence by f Xr(fJ))J may be defined by letting .Yl4(0)= iterating F. These sequences always consist of alternating Os and 1s, but the inial value is randomly chosen with equal probabilities. Now, T is measurepreserving', the invariant events are f and 0, both trivial, so the sequence is ergodic. And it is easily verified that lima-yxn -1Z2 PITkA fa B) P(A)PB) for 0.5 for k every pair A,B e T. For instance, 1et B = f 1 ) and then #(F V f7#) even and 0 for k odd, so that the limit is indeed 0.25 as required. You can verify, equivalently, that the ergodic theorem holds, since the time average of the sequence will always converge to 0.5, which is the same as the ensemble mean (a of XI()). ,f
=
,
,
=
=j
=
,
.
In this example, Xt is perfectly predictable once we know X; for any t. This shows that ergodicity does not imply independence between different parts of the sequence, even as the time separation increases. By contrast, a mixing sequence has this property. A measure-preserving, ergodic shift transformation Wis said to be mixing if, for each A, B e F, ,
l7ependence
lim#(F 4 cjB4
203
(13.34)
PA4PB).
=
k-x
X1(Ff-')) for each T he stationary sequence f.V)7is said to be mixing if A() t where F is a mixing transformation. Compare this condition with (13.28);Cesro convergence of the sequence t#(F V /'7 #) k e EN ) has been replaced by actual convergence. To obtain a sound intuition about mixing transformations, one cannot do better than reflect on the following oft-quoted example, originally due to Halmos (1956). =
,
13.16 Example Consider a dry martini initially poured as a layer of vennouth (10% of the volume) on top of the gin (90%).Let G denote the gin, and F an arbitrary small region of the fluid, so that Fcj G is the gin contained in F. If 0.9 and #(.) denotes the volume of a set as a proportion of the whole, #(G) =
#(Ff-7 G)/#(/), the proportion of gin in F, is initially eithr 0 or 1. Let F denote the operation of stining the martini with a swizzle stick, so that #(F Vcj G)/#4p is the proportion of gin in F after k stirs. Assuming the fluid is incompressible, stirring is a measure-preserving transfonnation in that #(F #) mixes the martini we would expect the proportion = #4#) for all k. If the stirring PITkFP k hich is gin in G)/#(&, to tend to #4G), so that each region F F F, w of of the martini eventually contains 90% gin. la transfonnaThis is precisely condition (13.34).Repeated applications of a lnixing tion to an event A should eventually mix outcomes in A and Ac so thoroughly that for large enough k the composition of F V gives no clues about the original A. Mixing in a real sequence implies that events such as A ( ): A((l)) S tz) and F 1.4 = f(t): Xt-vk) S JJ are becoming independent as k increases. It is immediate, or virtually so, that for stationary mixing sequences the result of 13.14 can be x. strengthened to Cov(X1,-L) 0 as n =
--y
--y
13.5 Subfields and Regularity We now introduce an alternative approach to studying dependence which considers the collection of c-subfields of events generated by a stochastic sequence. This theory is fundamental to nearly everything we do subsequently, particularly because, unlike the ergodic theory of the preceding sections, it generalizes
(stationaty)case. Consider a doubly infinite sequence (-V,t e r) (notnecessarily stationary) and define the family of subfields l X, s f /l, where TJ = nXs,...,Xt) is the smallest c-field on which the sequence coordinates from dates s to t are measurable. The sets of Sf can be visualized as the inverse images of (/ s4dimensional cylinder sets in :8*; compare the discussion of j 12.3, recalling that IN and Z are equipotent. We can let one or other of the bounds tend to intinity, and aparticularly important sub-family is the increasing sequence (T-'x, t e Z ), which information contained in the sequence up to can be thought of as, in effect, /' which The c-field on the sequence as a whole is measurable is the date limitingcase F+-7 Vts-t.. In cases where the underlying probability model beyond the measure-preserving
-
tthe
=
Theory of Stochastic Processes
204 concerns just the sequence
(XZJwe
shall identify 1+-Zwith 5.
OfT-rx. Another interesting object is the remote c-Jcl# (or tail c-field), F-x This c-tield contains events about which we can learn something by observing any coordinate of the sequence, and it might plausibly be supposed that these events the occurred at time past when the initial conditions for the sequence were set. However, note that the set may be generated in other ways, such =
-x,
RS
t Ot 1+=
,
or
Ot 5-tt
tremote'
.
One of the ways to characterize independence in a sequence is to say that any pair of non-overlapping subtields, of the fonn Tfzland 1143where /1 k h > t5 k /4, are independent (see j10.5). One of the most famous early results in the law' for independent theory of stochastic processes is Kolmogorov's usually given for the case of a sequence (Al7, and the sequences. This theorem is is defined in this c-tield remote case as
07,.18/D,
.
13.17 Zero-one Iaw lf the sequence (A-fITis independent, every remote event is trivial, having either probability 0 or probability 1. and let T be the colProof Let A be a remote event, so that A e i= 0*2.1@=f lection of events having the property #(A rn #) #(A)#(#) if B e V. By indepenand @1f dence, are independent subfields, so that @(c V, for every f. We may therefore say that U7=1T( c V. U'7=1T(,then B e T( for some t, and B' e T(' for some t', and if t' 1 lf B,B' and accordingly U7z=lFlis a tield. t (say)ihen B e F1f'so B fa #' e ;1f' c U7 Moreover, if fBj e Nl is a monotone sequence and Bj #, then A f-'h Bj is also monotone and converges toA rn SforanyA l 1'-', adl e V'by continuity of #, so V is a monotone class. By the argument of 1.24, V therefore contains the union of and so V u c(U7=1T'() V*t any countable collection of sets from U7 2 and T.llowevervd e 5, soA e N wemay setfl=A, giving#td) =#(A) s ence p(4) = 0 or 1. . =
,
=
T'K'
=1@1r,
-->
=l97lf
.191,
=
=
.
.
The zero-one 1aw shows us that for an independent sequence there are no events, other than trivial ones, that can be relevant to a11 sequence coordinates. But clearly, not only independent sequences have the zero-one property, and from our point of view the interesting problem is to identify the wider class of sequences that possess it. A sequence fX))+-7 is said to be regular or mixing if every remote event has probability 0 or 1. Regularity is the term adopted by Ibragimov and Linnik (1971), to whom the basics of this theory are due. ln a suitably unified framework,this is essentially equivalent to the mixing concept defined in j13.4. The following theorem says that in a regular sequence, remote events must be independent of a11 events in 5. Note that trivial events are independent of themselves, on the
definition.
13.18 Theorem (lbragimovand Linnik 1971: th. 17.1.1) tall+-Zis regular if and only if, for every B e 5,
lt ( .
Dependence
sup l#(A ra #)
-
I --A
PA)PB4
205 0 as t
(13.35)
.--.) -=.
A e F!-
Proof To prove suppose 3 E E F-x with 0 regular. Then for every t, E e @-'x and so dif'
<
,
S#P I#(A
f''h
F)
I2
#(A)#(A3
-
ziez.f-
#(A)
-
P(E) < 1, so that (X;)+-=xis not PE4
2>
(ja a6)
()
,
.
which contradicts (13.35).
if assume regularity and define random variables ( lx #(A) (F-lx/f-measurable) and q (Wf-measurable), such that ls #(#) F()n). Then, by the Cauchy-schwartz inequality, #(A fa #) P(A)PB) To prove
Eonly
=
,
=
-
-
=
-
lF((n) I
=
IlF(n I@-'-)112, IuE'tsE'tnI@!-)) I < lItII2
(13.37)
where the equality is by the Iaw of iterated expectations because A e F-lx. Note which will complete the < 1. We show lIF(n l@-,-)112 0 as t that 11(112 of T-fx. roof, arbitrary since element is X an P x)()) Consider the sequence (F(ls I?F-f lt2. For any (.t) e f1, -->
-=,
--
F(1sI8/'-x)())
F(1s l@-x)(t0) as t
--
(13.38)
--> -=,
where by equation (10.18) and the zero-one property,
Jw&1sI;-x)()J#()
f-7#)
PE
=
#(#), PE4 =
1
=
#(A-) = 0
0,
,
E e Fux.
(13.39)
#(#) a.s. agrees with the definition, so It is clear that setting F(ls I@-x)() 5-t.) F(lsl P@) a.s., or, equivalently, that we may say that (F(1s IT!x) /'(#)()2 0, a.s. (13.40) Since 1s()) #(#) < 1 for al1 (1) e f, (F(ls @-fx)() /:(#)()2 is similarly =
--
--)
-
1
-
I
I
-
bounded, uniformly in f. Unifonn integrability of the sequence can therefore be assumed, and it follows from 12.8 that IlF(n l9-/-)112=
as required.
IlF(1sI@-'-)(t,))
-/'(:)112
--
0 as t
(13.41)
--+ -,x,
.
In this theorem, it is less the existence of the limit than the passage to the limit, the fact that the supremum in (13.35)can be made small by choosing large, that gives the result practical significance. When the only remote events are trivial, the dependence of Xt-vkon events in @-x, for fixed k must eventually decline as f increases. The zero-one 1aw is an instant corollary of the necessity part of the theorem, since an independent sequence would certainly satisfy -t
,
(13.35).
Theory of Stochastic Processes
206
There is an obvious connection between the properties of invariance and remoteness. lf T is a measure-preserving shift transformation we have the following simple implication.
13.19 Theorem
If FA
=
A, then A e T-x. l
l
t t t Proof lf A e T-x, then Fz4 e @-x and F A e F-x :4 followsimmediately that e (*s=ts-sxj chtO,k +
-
-
1 .
lf TA
-x@!x)
=
-
=
l
A, F A
T-x.
A and it
=
.
t
The last result of this section establishes formally the relatlonship between regularity and ergodicity which has been implicit in the foregoing discussion.
13.20 Theorem (Ibragimov and Linnik 1971: cor. 17. 1.1) If a stationary sequence x((o) (AX(o))!Zis regular, it is also ergodic. =
Proof Every set A e F+-7is contained in a set At e T-ff, with the sequence (A,) non-increasing and Ar A. Thus, A, may be constnlcted as the inverse image under .x of the (2/+ ll-dimensional cylinder set whose base is the product of the coorof x(A) e The inclusion follows by dinate f-sets for coordinates P(At4 #(A). continuity of By P, that 1.2(iv). we can assume Let A be invariant. Using the measure-preserving property of F, w find .1,
=.
-4,...,/
-->
P(At raAl
=
PAt
Since k is arbitzary, regularity Letting f
--
x
yields #(A)
=
fa
F -kA)
=
PITkA t chz4l.
(13.42)
implies by (13.34)that PAt faAl = PAt)PA). P(A) 2 so that P(A4 0 or 1, as required. . =
,
13.6 Strong and Uniform Mixing The defect of mixing (regularity)as an operational concept is that remote events are of less interest than arbitrary events which happen to be widely separated in time. The extra ingredient we need for a workable theory is the concept of dependence between pairs of c-subtields of events. There are several ways to characterize such dependence, but the following are the concepts that have been most commonly exploited in limit theory. Let L1,5,P) be a probability space, and 1et V and R be c-subfields of @;then
(x(V,X) =
sup G c F Hq
.'(
IP(G
f'n
S)
-
#(G)#(&
I
(13.43)
l
(13.44)
,
is known as the strong mixing coeftkient, #(V,A)
=
sup Gq F,f'1eJf; P(G)>0
and
G4 1/,(/./'1
-#(#)
as the uniform mixing coefficient. These are alternative measures of the dependence between the subfields V and R. If the subfields V and R are independent, then (x(V,J) = 0 and #(V,Jf) 0, and the converse is also true in the case of uniform mixing, although not for strong mixing. At first sight there may appear not much to choose between the defini=
,
Dependence
207
tions, but since they are set up in terms of suprema of the dependence measures over the sets of events in question, it is the extreme (and possibly anomalous) cases which define the characteristics of the mixing coeixcients. The strong mixing concept is weaker than the uniform concept. Since
IPG
r-'hff
-
PCG4PC* I S IPCH, G)
I f 9(5',>t)
PH)
-
(13.45)
for all G e V and H e R, it is clear that a(N,J#) f ((T,Jf). However, the following example shows how the two concepts differ more crucially. 13.21 Example Suppose that, for a sequence of subfields (V-IT,and a subfield X, a%m,R4 0 as m x. This condition is compatible with the existence of sets Gm ?. forJ xp(s) PGm) llm, andlGm fo & e Mmandffe R with thejroperties .-+
--
=alm
=
.
But (/497,s,J#)2 IPCH6Gm) #(/f) I Ia PH4 I for every tilds Mmand R are not independent in the limit. n =
-
-
,?z
k 1, showing that sub-
Evidently, the strong mixing characterization of does not rule out ppssibility dependence negligible between the sets. qf () rnixing coefficients and the only not that are can be defined, although they tx have proved the most popular in applications. Others that have appeared in the literature include tindepndence'
13(V,J#) = sup E HR
j#(SIV)
-
#(S)
j
(13.46)
and
p(V,J#)
sup
=
If((n)I
(
2
n
(13.47)
,
2
where the latter supremum is taken with respect to a1l square integrable, zero mean, V-measurable r.v.s (, and S-measurable r.v.s q. To ompare these alter#(Sl T)() #(&, so that natives, first let (() =
f5(V,X)
-
=
sup SGS
JI(1
dP
I( I js sup
dp
GC%D.H6R
sup
G e F, H R
(.LdP
*
=
a(T,3f).
(13.48)
'''
#(G) and Moreover, since for any sets G e V and H e 3f, (((t)) 1c() n() which members the of is defined, p(V,>t) and F((n) #(& set 1zz((l)) are over while l( I < 1 and lqI S 1 for these cases, it is also clear #(G f''h z.fl PGlm notwithstanding (x its designation, is the weakest of mixing, Thus, that p a. o? variants, although it is these four course stronger than ordinary trivial remote events. We also have j S 9, by an regularity characterized by immediatecorollary of the following result. =
-
=
=
-
-
istrong'
Theory of Stochastic Processes
208
13.22Theorem I#(NIV)
If
#(&
-
9(V,>t) a.s., for a1l H e R. a
The main step in the proof of 13.22 is the following lemma. !' 13.23 Lemma Let X be an talmost surely boundedy) V-measurable nv. Then
1
sup GeF,#(G)>0
jaxdp
po).
ess sup
=
x.
(13.49)
:'
r
E
Proof #(G)-1 j < ess sup X, for any set G ils the designated class. For any E > 0, consider the sets .
fccxlrj
..E
.
.
G G-
f (:
=
=
:
X() -A)
k
v
(ess sup.km 2 (ess sup
-
al
-
6:), E).
By definition of ess sup X, both these sets belong to V and at least one of them is nonempty and has positive probability. Define the set G*
G+, #(G>+)k PG-) =
G -, otherwise
and we may conclude that
1 Jc- xdp ,(c-)
(13.49) now follows on letting
E.
z (ess spp approach 0.
m -:.
(13.50)
*
of 13.22 #ut X #(SIV) #(S) in the lemma, noting that this is a Observe that, for any G e V, V-measurable r.v. lying between +1 and E'(G)-1 .' IJ crxl'l = IPCH, G) Ph'j l Hence the lemma together with (13.44) implies that, for any H e R, Proof
=
-
-1.
-
.
9(N,J#)
ess sup
k with probability 1.
.
f#(SIV)
l#(SI V)
-
#4&
l
-
,
#(S))
(13.51)
14 Mixing
14.1 Mixing Sequences of Random Variables For a sequence
T-Jx
(X,())J=-x,let
=
of...,Xt-z,Xt-j,Xt) as in
913.5,and
similarly
The sequence is said to be vmixing (or define5*t+m n(Xt+m,Xt+m+3,.%+m+1,...4. lim,u-Axa,u 0 where strong mixing) if =
=
qm
sup
=
(x(T-Jx,@7+,,,),
(14.1)
l
and a is defined in (13.43).It is limpz-jxtj,u0, where
said to be
=
Sup
=
zrl
(
b-mixingtoruniform
9(?F-f x,T7+r,,),
mixing) if
(14.2)
and (j is detined in (13.44).(-mixing implies a-mixing as noted in j 13.6, while the converse does not hold. Another difference is that g-mixing is not timex) reversible', in other words, it is not necessarily the case that supytF/ +,,I,T-f =
supytT-lx,T''; By contrast, a-mixing is time-reversible. If the sequence (.X))+-7 is a-mixing, so is the sequence ( Ffl+-7 where J'f X-t. 0 where (#,((.t)))=-xis also said to be absolutely regular if limp-xp,,, +,,).
=
=
p,,,
sup
=
t
13(8?;-'-,8?7-,+,,,)
(14.3)
and 13is defined in (13.46).According to the results in j13.6, absolute regularit,yis a condition intenuediate between svong mixing and uniform mixing. On the other hand, if (m)+= is a stationary, fmbounded sequence, and @0-x c(...,X-1,A%) and %m+* c(X,,,X,,+1,...), the sequence is said to be completely O where p is defined in (13.47).In stationary regular if p,,, p( T0 J;+oo m ) regllarity complete is equivalent to strong mixing. KolmoGaussian sequences, gorov and Rozanov (1960)show that in this case =
..x
=
-->
=
-x,
,
qm/
Lzam.
pm <
(14.4)
In a completely regular sequence, the autocovariances T Extxt-j) must tend to x. A sufficient condition for complete regularity can be expressed in 0 as j terms of the spectral density function.When it exists, the spectral density fh) is the Fourier transform of the autocovariance function, that is to say, =
-->
f z)
1 =
t=
77
c'l?
2,s jzz--sn z)
,
,
e
(-a,a1.
(14.5)
Theory of Stochastic Processes The theorem of Kolmogorov and Rozanov leads to the result proved by Ibragimov and Linnik (1971: th. 17.3.3), that a stationary Gaussian sequence is strong mixing when fk) exists and is continuous and strictly positive, everywhere on
(-a,cj.
This topic is something of a tenninological
minefield. KRegularity' is an undescriptive term, and there does not seem to be unanimity among authors with regard to usage, complete regularity and absolute regularity sometimes being used synonymously. Nor is the list of mixing concepts given here by any means exhaustive. Fortunately, we shall be able to avoid this confusion by sticking with the strong and uniform cases. While there are some applications in which absolute regularity provides just the right condition, we shall not encounter any of these. might be thought appropriate as a synonym for Incidentally, the tenn regularity, but should be avoided as there is a risk of confusion with wetz/cdependence, a term used, often somewhat imprecisely, to refer to sequences having summable covariances. Strongly dcpendent sequences may be stationary and mixing, buttheircovariances are non-summable. (dWeak' implies Icswdependencethan <strong' in this instance, not morel) Confining our attention now to the strong and uniform mixing definitions, measures of the dependence in a sequence can be based in various ways on the rate at which the mixing coefficients xmor 4u tend to zero. To avoid repetition we will discuss just strong mixing, but the following remarks apply equally to the ;-x Since the collections uniform mixing case, on substituting 4)for Gthroughout. and 5*t+m are respectively non-decreasing in t and non-increasing in t and m, the sequence f(xp,lis monotone. The rate of convergence is often quantified by a summability criterions that for some number ? > 0, xm 0 sufficiently fast that tweakmixing'
f
-->
X
7711* < r;=1
x.
(14.6)
The term size has been coined to describe the rate of convergence of the mixing numbers, although different definitions have been used by different authors, and the terminology should be used with caution. One possibility is to say that the if the mixing numbers satisfy (14.6).However, the commonsequence is of size White 1984) is to say that a sequence is G-mixing of example for est usage (see *.11 if am Om -#) for some fp > It is clear that such sequences are size when of the 1/:0, raised to summable so that this concept of size is power stronger than the summability concept. One temptation to be avoided is to define the size as < whe re f? is the largest constant such that the a) * are summable' ; for no such number may exist. Since mixing is not so much a property of the sequence (.X)) as of the sequences of c-fields generated by (A), it holds for any random variables measurable on those c-fields, such as measurable transformations of Xt. More generally, we have the following implication: -(2
-(p()
=
-(?,
14.1 Theorem Let Ff = g(Xt,Xt-3,...,Xt-x) be a measurable function, for finite of size then Ff is also. lf Xt is a-mixing (g-mixing) -(?,
T.
Mixing
211
= Proof Let V-fx= c(..., 1$-1,F, and G*twm c(I$+,,,,im+1,...). Since F, is measuri-Jx which each of Xt.%-j,...,Xt-z are measurable, Wlx i able on any c-field on and it follows that ay,,u f (u-s and M*t-bm i T7+,u-s. Let ay,,, supr (N-fx,Vc;+,,,) for m k 1. With tinite, (xm-z O(m-%) if um Om-%) and the conclusion replacing follows. The same argument follows word for word with w =
.1
=
=
d9'
ttx'
.
14.2 Mixing lnequalities Strong and uniform mixing are restrictions on the complete joint distribution of the sequence, and to make practical use of the concepts we must know what they imply about particular measures of dependence. This section establishes a set of fundamental moment inequalities for mixing processes. The main results bound the m-step-ahead predictions, Extwm I5-tx). Mixing implies that, as we try to forecast the future path of a sequence from knowledge of its histol'y to date, looking further and further forward, we will eventually be unable to improve on the predictor based solely on the distribution of the sequence as a whole, EXt+m). The r.v. Extm lT-'x) EXt+m) is tending to zero as m increases. We prove convergence of the fw-norm. -
14.2 Theorem
p k 1 and with um defined in (14.1), lI,< 2(21/: + 1)a,,,1/r-1/rl1m+,,,lI,.(14.7)
(Ibragimov 1962) For r
IlF(A+r,,IT-'-)
-
Exm)
Proof To simplify notation, substitute X for Xtmm,V for s-too, R for T7+,,,,and a for xm.It will be understood thatxis an X-measurable random variable where V, R c 5. The proof is in two stages, tirst to establish the result for IXI < Mx < cxo a.s., and then to extend it to the case where X is G-boundedfor t'inite r. Define
the V-measurable r.v. q
=
sgn(F(Xl V) F(m) -
1, F(.Yl V) k #(m, =
-1, otherwise.
(14.8)
Using 10.8 and 10.10,
9') A'IFIXI
-
F(ml
=
=
V) -F(-Y))1 FEn(F(XI FEtA'tn.YlN) -nF(m1
= Covtq,m
ICovtq,m I
=
.
(14.9)
Let F be any V-measurable r.v., such as q for example. Noting that R) -F(F)) is A-measurable, similar arguments give sgn(F(FI Icovt-Y,F)
R4 -A'(l'))) I I IF(X(&FI < F(IXII(&FIR4 -F(F))I) < Mxf'l F(y1 R) F(F) l =
-
(
=
Theory of Stochastic Processes
1Z
G A&1COv((,F)
l
(14.10)
,
where the tirst inequality is the modulus inequality. ( and n are simple random variables taking only two distinct values each, so define the sets A+ (n = 1 ), =
A-
(n
=
-1
=
together gives
), B*
FlFl.Yl T) F(m -
=
()
=
=
1 ) and B,
l K Mxl Cov(n,k)l
Mxj (#(A+
chB+) + P(A-
=
((
=
=
Mxlfttnl
fa#-)
PA+c
-
(#(A+)#(#+) +PA-)PB-)
l
-1
Putting
.
-
F(()F(n)
B-)
I
PA- rn #+)1
-
PAhP(B-j
-
(14.9)and (14.10)
PA-4PB+)j
-
< 4Mm. Since IF(-Yl N) F(m -
l < IF(#IT) 1+ lF(ml f lMx. V) F(mIIP S 2M,x(2a)1/F. llF(-YI
I
(14.11)
it follows that, for p
1,
(14.12)
-
This completes the first pal't of the proof. The next step is to let X be Lr-bounded. Choose a finite positive Mx, and define Xl 1( IxjsMxlx and .Y2 By the Minkowski inequality and (14.11), X =
-
=
m .
IlF(.YlN)
-
+ T) e'(A)11, < IIF(-LIT) -F(-Yl)lI, A'(mII, 11(721 2A.za'(2a)1/J' + zjjxallp, u -
and the problem is to bound the second right-hand-side llX2ll/?f
for r
member. But
sf-rlr IlA'IIrr/P
(14.14)
p, so we arrive at 11.E'(x1 N) F(#)ll, < -
Fillally,
(14. 13)
Choosing
Mx
1IXl)ra-
=
1Mx2%)31P I/C
zM-l--rll,-Yllr/.
(14.l5)
and simplifying yields
11F(-Y1 N) -.E'(A')11,/
which is the required result.
+
2(21/P+
lltxl/#-l/rjjxjjr,
*
There is an easy corollary bounding the autocovariances of the sequence.
14.3 Corollary For p
1 and r plp 1), (14.16) ICov(x,,.Y,+,,)l < 2(21-1* + 1)a)-1/#-1/rjjx;IIp1Ix,+mI!r. >
-
Proof
ICovxvxmb l
=
IEXtxtwmj
-
F(A)F(-'+,,,)
I
= IExtExt-vm I9,) 1lpllA'(X,+,, I@,) F(.&+,2II,/?-l) f 11.X) 2(21-1/P+ 1)jj#,ljpIlX,+mIl<)-1/P-1/r f EXt+mttj
-
l
-
(14. 17)
Vixing
213
where the second equality is by 10.8 and 10.10, noting that Xt is Fsmeasurable, the tirst inequality is the Hlder inequality, and the second inequality is by 14.2. . 1.4.4 Theorem
(Serfling 1968: th. 2.2) For r k p k 1, IlA'(Xf+??, r. I11x) EXt+m) < 29,,,1-1/r1I.&+,,,II
11p
-
where
4),,,is
defined in
(14.18)
(14.2).
Proof The result is trivial for r p 1, so assume r > 1. The strategy is to prove the result initially for a sequence of simple r.v.s. Let Xt-vm Z#,=1.xj1xj, = nxtmm,xtwm.vj, For some e ( consider the random 4/ e 17+,,,,where s-tmm T-fx)(t')), Ext-vm although for clarity of notation the dependence on ) is element l not indicated. For r > 1 and q r/tr 1), we have =
=
=
...).
=
IExt..m I@-'-)
E(Xt+m)3r
-
'>7
j
-
l@-'-)
k-xipAi
=
-
i
i
1
r
IxflI#(4fI@-'-)-#(4I
S
r
#(4f))
r '>7
= f
ImlI#( i I5t ) z1.
i
77Ixi I I#(X i I5-t X
l/rl
x) -
#(X
ij
i
S
E IXt-mIrl @-,-)+
1/t?
I #(AjI@-r-)
PAi)
-
--
#(Af) I
-
) (77
I 77I#(A I5
X?
-'
-) -
i
r)
I FIxt-vm
i
IPAi l@-'-)
-
#(Af) I
PAi)
X.
)
I
(14.19)
The second inequality here is by 9.25. The sets Aj pmition and T-rx)+#(Af'I5-t=) a.s. and PAiwAk') PAi QJA/' I@-rx) #(AfI PAi) + PAi') for # i'. Letting A) denote the union of all those Aj for which P(Ai I@-rx) PAi) k 0, and AJthe complement of 4) on f1, ,
=
=
-
77I#tAfl @-'-)-#4A/)1
I#(A) I@-'-)
=
#(A1)I+ IPCAII@-'-) #(A7)I (14.20)
-
j
-
.
By 13.22, the inequalities
lPA; I@-'-)
#(41)
IPA; I@-'-)
PA;)
-
-
l < 4u I S 9,,,
hold with probability 1. Substituting int (14.19)gives
lExmf
@-'-) Exm) -
Taking expectations
a.s. (14.21) ge'(IAu,,, Irl @-r-)+ Flx,+,,,l rjtztTl,,,lr&
Ir
and using the law of iterated expectations
t E If(xf+r,,l T-x)
-
rIq Ext,vm) Ir S lElXt+m Ir (29,,,)
and, raising both sides to the power 1/r,
then gives '
,
(14.22)
214
Theory of Stochastic Processes f 2IIA+-II41 F(A7,a)IIr -1/r.
IT-'-) Il'(kt+,,,
-
(14.23)
Inequality (14.18)follows by Liapunov's inequality. The result extends from simple to general r.v,s using the constmction of 3.28.
ke For any nv. X:+mthere exists a monotone sequence of simple r.v.s fxkywm, IxY(z)r+,,,() 0 as k N ) such that Xt-vmlt l for all e f. This converx) @-f Exkymm), k e (N) by transfers the sequences (F(aV)f+,,, l a.s. to gence 10.15. Then, assuming Xtwmis fv-bounded, the inequality in (14.22)holds as k Ir as the x by the dominated convergence theorem applied to each side, with lxfx.m dominating function, thanks to 10.13(ii). Tlis completes the proof. w -->
--)
=,
-
-
-->
The counterpart of 14.3 is obtained similarly.
14.5 Corollary
For r k 1, ICov(-Yr+,,,.x))l
where, if r
=
1, replace
s zgml/rjjxrjjrjjx/+mjlrytr-j),
by 1IX,+,,,IIx ess IlXf+r,,IIr/(r-1) =
(14.24)
sup Xt.vm.
Proof
ICov(x)+,,,,.')I I9-f-) F(-Y,+r,,)lIs(r-l) s lIA-,IlrlI'(.2t+,,, -
s 2#-17rlIx,lIrII=,+-lIr/(r-1), where the first inequality corresponds is by 14.4. .
(14.25)
to the one in (14.17),and the second one
These results tell us a good deal about the behaviour of mixing sequences. A fundamental property is mean reversion. The mean deviation sequence (A Ext) ) must change sign frequently when the rate of mixing is ligh. lf the sequence exhibits persistent behaviour with Xt Ex tending to have the same sign for a large number of successive periods, thep IE(Xt+mI5-t.) Extmm) j would likewise tend to be large for large m. If this quantity is small the sign of the mean deviation m periods hence is unpredictable, indicating that it changes frequently. Butwhile mixing implies mean reversion, mean reversion need notimply mixing. Theorems 14.2 and 14.4 isolate the properties of greatest importance, but not the only ones. A sequence having the property that Ilvar(Ap, I9-',.) Vart-mlllr > 0 is called conditionally heteroscedastic. Mixing also requires this sequence of norms to converge as m x, and similarly for other integrable functions of Xtwm. Comparison of 14.2 and 14.4 also shows that being able to assert uniform mixing can give us considerably greater flexibility in applications with respect to the existence of moments. In (14.18),the rate of convergence of the left-hand side to F-Jx) Ext+m) I zero with m does not depend upon p, and in particular, FIE(Xt+mI whenever exists 0, condition infinitesimally > for a 117,+,,,111+8 converges stronger than uniform integrability. In the corresponding inequality for am in 14.2, p < r is required for the restriction to Likewise, 14.5 for the case = p 2 yields -
-
-
-
--+
-
Kbite'
.
Mixing
215
I 2#,,,1/2jI.;t1I2IIx,+,,,Il2,
lCovxmnx
but to be useful 14.3 requires that either Xt or Xtwmbe fo-bounded, Mere existence of the variances will not suffice.
(14.26) for
>
0.
14.3 Mixing in Linear Processes fmlO-x
which arises very frequently in econometric A type of stochastic sequence modelling applications has the representation
Xt
=
q
jzt
-j
,
0
<-
(14.27)
-<
q
c'o
,
jcut where (Z,1+-7 (calledthe innovations or shoks) is an independent stochastic is a sequence of fixed coefficients. Assume without loss of sequence, and ( %lp=() generality that the Zt have zero means and that % 1. (14.27)is called a moving average process of order q (MA()). lnto this class fall the finite-order autoregressive and autoregressive-moving average (ARMA) processes commonly would clearly like to know when such model We economic series. used to time sequences are mixing, by reference to the properties of the innovations and of the sequence f 0;1. Several authors have investigated this question, including =
Ibragimov andLinniktlg7l), Chanda(1974), Gorodetskii (1977),Withers (1981a), Pham and Tran (1985),and Athreya and Pantula (1986a,1986b). Mixing is an asymptotic property, and when q < x the sequence is mixing infinitely fast. This case is called q-dependence. The difficulties arise with the cases with q x. Formally, we should think of the MA(=) as the weak limit of a sequence of MA() processes; the characteristic function of Xt has the form =
tlk/)
=
Iltllzr-/fz-t) ,
j-j
#t?f(%) ($(D(pointwisein
(1k.28)
cxp where R) as q 9r(l) is a ch.f. and 0, we may invel't the latter according to 11.12, and identify the continuous at 12 T 0Z The existence of the limit corresponding distribution as that of Xt Zy ) t-j. imposes certain conditions on the coefficient sequence f0./17,0.We clearly need oo, and for the variance of Xt to exist, it is further necesl $1 0 as j the sequence be square-summable. Note that the solutions of finitesary that order ARMA processes are characterized by the approach of l0.j1to 0 at an exponential rate, beyond a finite point in the sequence. If (4) is i.i.d. with mean 0 and variance c2, Xt is stationary and has spectral density function
and if
-->
,
--
=
=
=()
-,..:
--
.wl
co
f (1) 2:: Tjeihi jujl '''
=
2 .
(14.29)
The theorem of Ibragimov and Linnik cited in j14.1 yields thexconditionZ7=()l $1 strong-mixing in txl the for sufficient Gausjian another However, case. < as standard result (seeDoob 1953t ch. X!8, or Ibragimov and Linnik 1971: ch. 16.7)
Theoty of Stochastic Processes
states that every wide-sense stationary sequence admitting a spectral density has moving average representation with orthogonal increments and a (doubly-infinite) square summable coefficients.
But allowing more general distributions for the innovations yields surprising results. Contrary to what might be supposed, having the %tend to zero even at an exponential rate is not sufticient by itself for strong mixing. Here is a simple illustration. Recall that the first-order autoregressive process Xt pm-l + Zt, p/, j 0, 1,2,... Ip I < 1 has the MA(x) form with % =
=
,
=
14.6 Example Let (Z,)7' be an independent sequence of Bernoulli r.v.s, with P(Zt Ztl and 0) = zl Let 1) Pzt =
=
ax'c
=
Xt
=
.
=
.zXl t- j
Zt
+
=
t
F'')l-izt-p
t
1,2,3,
=
(14.30)
...
lt is not difficult to see that the term f
X
2-lZt-j
t
=
2-Xlkzk
(14.31)
belongs for each t to the set of dyadic rationals Bs (kl2t, k = 0,1,2,... 2f+1 2f+1 1 ) Each element of Ff corresponds to one of the p ossible drawings =
-
.
(Zo,...,4),
and has equal probability of 2 -fXt e Bt
whereas iff
ztl
=
=
fkllt, k
1
.
Iff Zo
0,2,4,...,242f
=
=
-
0,
1)),
1, Xt e J'F'f Bt -
=
Lkllt ,
k
=
13 5 ,
,
2f+1 I j -
,...,
.
It follows that (.Y()= 1 ) /a fX, e Btj 0, for every finite /. But it is clear that ,1 PIXO = = 0) P(Xt 6 SJ Hence for every finite m, =
=
.
)
q?n X #(
(&
=
1 ) rn (A% e Bm))
= t,
which contradicts
im
.--.h
-
#(V
=
J
1)#(Xm e Bm4
(14.32)
0. n
Since the process starts at t = 0 in this case it is not stationary, but the example is easily generalized to a wider class of processes, as follows, be an independent sequence of Bern14.7 Theorem (Andrews 1984) Let IZtIEMJx If Xt oulli r.v.s, taking values 1 and 0 tixed probabilities p and 1 (A)*-oo + mixing.' is pXf-1 Zt for p e (0,!, not strong n ,with
-#.
=
Note, the condition on p is purely to expedite the argument. The theorem surely holds for other values of p, although this cannot be proved by the present
approach.
Proof Write Xt+s = p'Xf + Xta where
Mixing
217
x-1
Xts
fZt-s-j. 'X7 j=
=
(14.33)
The support of Xzs is finite for finite p, having at most 2, distinct members. Call this set Bi, so that Wr! (0,1), Fa = (O, 1, p, 1 + p), and so on. ln general, 8$+1is obtained from Wrxby adding p' to each of its elements and forming the union of these elements with those of Bi; formally, =
WQ1= F,
QJ
tw+ p&:w
v,J,
s
s
=
(14.34)
2,3,...
For given s denote the distinct elements of B&by wj, ordered by magnitude with wl < < wJ, for J < ls. Now suppose tha t Xt e (0 p) so that ps#t e (0 p'+1) This means that Xtms for some-j. Defining events A = (aYtE assumesa va lue between wj and wy+ p&+1 r'+1) w+ (0,p)l and Bs lXf+, e Ul/=ltwj, ) we have #(#,IA) == l for any s, howeverlarge. To see that #(A) > 0, consider the case Zt Zf-l Zt-z 0 and zr-a 1 and note that ...
,
,
.
,
,
=
,
=
=
=
x
xp?
;jS j=3
for p e (0,1g.So, unless PBs)
3
x
t-j u ppz
=
j=3
j
P
<
p
.
(14.35)
p
1, strong mixing is contradicted. (Xf e (p,1)l has positive The proof is completed by showing that the set D probability, and is disjoint with #,. D occurs when Zt 0 and Zf-1 1, since =
=
=
=
then, for p e (0,,11,
X
oo
ppz p s :-l j= t-j
and hence #(D)
>
y--lp/ j P ) s
u
=
-
(14.36)
1,
/=1
0. Suppose that
min (ws1 ./k1
-
wyl k
p
t <
+
J-1
(14.37)
.
Then, if D occurs, wy+
p
J+1
pSXf B
f uj. +
p
U/ J=1 (w j,
sy
wj
p&-1s wy
(1438)
1,
.
pS+1)
0. or in other words, Bs rn D The assertion ln (14.37)is certainly true when s 1, so consider the following inductive argument. Suppose the distance between two points in J7, is at least cannot be ()'-1 Then by (14.34),the smallest distance between two points of +,+1 rlpJ-l pf-l when which pf S p' But leSS than the smaller of pf and p e (0,r1), s s impliesp'-1 p 2 p It fo llows that (14.37)holds for every s. w hence, Xtn
wj +
=
wy+
=
,
=
.
-
.
,
-
.
These results may appear surprising when one thinks of the rate at which p' but if so, this is because we are unconsciously thinking approaches 0 with about the problem of predicting gross features of the distribution of Xtmsfrom time t, things like Pxt-vs f xlA) for fixyd x, for example. The notable feature of the sets Bs is their inrlevance to such concerns, at least for large s. What we -;
Theory of Stochastic Processes
218
have shown is that from a practical viewpointthe mixingconcept has some undesi able features. The requirement of a decline of dependence is imposed over 6 events, whereas in practice it might serve our pumoses adequately to tolera certain uninteresting events, such as the Bs defined above, remaining dependent c the initial conditions even at long range. ln the next section we will derive some sufficient conditions for strong mixinl and it turns out that certain smoothness conditions on the marginal distribution of the increments will be enough to rule out this kind of counter-example. But no. consider uniform mixing. 14.8 Example 13C ons ider an AR(1) process with i.i.d. increments, Xt
pX,-l
=
+
Zt, 0
<
<
p
1,
in which the marginal distribution of Zt has unbounded support. We show that (,X)) is not uniform mixing. For 8 > 0 choose a positive constant M to satisfy m-l
plzm-j < # X 7=0
-.M
<
(14.39)
.
Then consider the events
yt
=
(L +A/)) e 10 p '-m
(&
-x
fXm < Lk e sm+* where L is large enough that #(#) k 1 8. We show #(AI> 0 for every m. Let pK = #(Zo < A'), for any constant K. Since Zo has unbounded support, either px < 1 for every K > 0 or, at worst, this holds after substituting (-Z,) for (Zf l and hence ( -XtI for (Xf). pk < 1 fOr a1l K implies, by stationarity, #(X-l < 0) = Pxo < 0) < 1. Since (.%< A-Ji (Z0 < X') QJ (fZ() 2 A') f'n IX-I < 01), independence of the (Zf l implies that B
=
-
PIXZ
So PA) that
>
PB
<
X) K pK+
(1-#r)#(X-1
<
0) < 1. PXJA'C+ ZPJ
0, since K is arbitrary. Since Xm =
I4)
=
# p'Vc +
m- 1
X pizm-j < L
by (14.39).Hence 4u k I#(#lA) means ()a, = 1 for every m. n
-
PB4
p'Ve
I> 1
-
2,
L +M
(14.40) .-)
p'Zm-j, it is clear
<
and since
(14.41) is arbitrary, this
Processes with Gaussian increments fall into the category covered by this example, and if (hmixing fails in the first-order AR case it is pretty clear that counteru examplej exist for more generalMAtx) ases too. The conditions for uniform mixing in linear processes are evidently exemely tough, perhaps too tough for this mixing condition to be very useful. ln the applications to be studied in later chapters, most of the results are found to hold in some form for strong mixing processes, but the ability to assert uniform mixing usually allows a relaxation of
Mixing
219
conditions elsewhere in the problem, so it is still desirable to develop the parallel results for the unifonn case. The strong restrictions needed to ensure processes are mixing, which these examples point to (to be explored further in the next section), threaten to limit the usefulness of the mixing concept. However, technical infringements like the ones demonstrated are often innocuous in practice. Only certain aspects of mixing, encapsulated in the concept of a mixingale, are required for many important limit results to hold. These are shared with so-called near-epoch dependenthmctions of mixing sequences, which include cases like 14.7. The theory of these dependence concepts is treated in Chapters 16 and 17. While Chapter 15 contains some necessal'y background material for those chapters, the interested reader might choose to essence, the diftkulty will be skip ahead at this point to find out how,
tn
resolved.
for Strong and Uniform Mixing
14.4 Sufficient Conitions
The problems in thecounter-examples aboveare with theformof themarginal shock distributions or unbounded, as the case may be. For strong mixing, a degree of smoothness of the distributions appears necessary in addition to summability conditions on the coefticients of linear processes. Several sufficient conditions have been derived, both for general MA(x) processes and for autoregressive and ARMA processes. The suftkiency result for strong mixing proved below is based on the theorems of Chanda (1974)and Gorodetskii (1977).These conditions are not the weakest possible in a1l circumstances, but they have the virtues of generality and comparative ease of verification. -discrete
define a random sequence taY/l=-x,where, for 14.9 Theorem Let Xt X*j=eojzt-j either 0 < r f 2 or r an even positive integer, (a) Zt is unifot-mly fv-bounded, independent, continuous with p.d.f. J'rzt, and =
Suy P
+-Ih-zt-va, j..
I
yz,tzldz u Mlzl,
-
for some whenever Ia I f =, < (y.;1&1+r) x gt (b) Zf=0 w jwre
>
,
&l' <
(14.42)
-,
0;
X
I$1 277 j=t
r,
Grtrl
r < 2,
(14.43)
=
gr-l
x
:2j y-'q
r/2 y.
g. ,
j=t
0./.'Ei
0 for a11complex numbers x with jxl Then (.&)is strong mixing with m O(Zx/z.+1 g t (r)1&1+r)) z (C)
0(1)
=
ZJ T =1
#
=
.
Before proceeding to the proof, we must discuss the implications of these three conditions in a bit more detail. Conditton 14.9(a) may be relaxed somewhat, as we
Theory of Stochastic Processes
220
show below, but we begin with this case for simplicity. The following lemma extends the condition to the joint distributions under independence. 14.10 Lemma lnequality (14.42)implies that for
k
k
Jw,
ml-ljfzrtzr''btz'l
Proof
IJrl M
=
1,...,k,
77lJ,l
.
(14.44)
>1
>1
usingFubini-s
6, t
k
dzz...dzk f I1TzJz2
-
<
theorem, k
k jpk mljfz/zr+*l
l1Tz,tzf)dzt...dzk
-
,.-1 f
jpk
+Jl) I-fzltzl
f
I
...dzk
,
, dz,...dzk l-I.fz,(o) I1.fz,(zr+zh)
+jpk +
Jzl
>2
.fzl(zl)
Ml cl
k +J) Tzltzlllllfz/z,
-
-
>2
>2
k
jvk-
mlj/ktz,
k
+
at4
-
I1Jz,(z
dzz...dzk.
>2
The lemma follows on applying the same inequality to the second term on the right, iteratively for / = 2,...,k. w Condition 14.9(b) is satisfied when I0;1 <<j-V for g. > 1 + 2/r when r K 2 and g > 3/2 + 1/r when r k 2. The double definition of Gtr) is motivated by the fact that for cases with r < 2 we use the von BalmEsseen inequality (11.15)to bound a certain sequence in the proof, whereas with r > 2 we rely on Lemma 14.11 below. Since the latter result requires r to be an even integer, the conditions in the theorem are to be applied in practice by taking r as the nearest even integer below the highest existing absolute moment. Gorodetskii (1977)achieves a further weakening of these summability conditions for r > 2 by the use of an inequality due to Nagaev and Fuk (1971). We will forgo this extension, both because proof of the Nagaev-Fuk inequalities represents a rather complicated detour, and because the present version of the theorem permits a generalization (Corollary 14.13) which would otherwise be awkward to implement. tjzt-j, = Detine W'r so that Xt = 11$+ W,and W and J''f ?Zt-j and J// Z7z of Xt, whose contribuare independent. Think of 54 as the Jcx-measurable x. tion to the sum should become negligible as t =
ZJ;0
Stail'
--
14.11 Lemma lf the sequence (Z,l is independent with zero mean, then cr
Evlm) < 22+-1 t
m
)q0?J sup Ezlm)s j=t
(14.45)
s; 0
for each positive integer m such that sup,ucz,
2-
)<
x.
Mixing
221
Proof First consider the case where the r.v.s Zt-j are symmetrically distributed, meaning that -Zt-j and Zt-j have the same distributions. In this case al1 existing odd-order integer moments about 0 are zero, and E
t+k
lm
t+k
jzt-j
t+k
=
0./j...h,nElzt-jq ...Zt-h,n)
...
jkzzzt jzm=t
j=t
t+k =
t+k
1 1 1 Qj2I .0jmE(Zt ...Zt-jm) -./1
...
..
./l=f jmzzt 1,0J:
Aq
s X 02j sup Ezlm).
(14.46)
sf 0
j=o
The second equality holds since EZt-jL...Zt-jzy) vanishes unless the factors form matching pairs, and the inequality follows slnce, for any r.v. F possessing the 0) for j,k > 0. requisite moments, EY i*k) 2 E(YqEYk) (i.e., Covtl'? F$ x. The result for symmetrically distributed Zs follows on letting k For general Zs, 1et Zs' be distributed identically as, and independent of, zx, is independent of J6, and J 7; has symmetfor each s < 0. Then J4 X; rically distributed independent increments Zt-j Z't-j. Hence ,
-->
=
ctjzl-j
-
-
m
X
Evt
lm)
< Evt
-
k';)241<
X 02j sup E(Zt.j Z't-plm -
j=t
x
V
glm- 1
y:=0
m
62 sup Elzltm-jl j
(14.47)
,
/
where the first inequality is by 10.19, the second by (14.45), and the third is the cr inequality. . Lastly, consider condition 14.9(c). This is designed to pin down the properties of the inverse transformation, taking us from the coordinates of (X,) to those of (4J 14It ensures that the function of a complex variable 0@) possesses an anajzzorykxl.for xl S 1. The particular property needed and lytic inverse I(x) l coefficient sequence (':y) is absolutely condition the by is implied the that under 14.9(c) the inverse representation is also summable. If Xt X; TJ 1 if % 1. An effect of 14.9(c) is Note that 'Io detined, as Zt example where 0@) rule for 01(x)41 x) out cases, as to with 01(.)a summable polynomial. The differencing transfonnation does zltp/yield a mixing process in general, the exception being where it reverses the previous integration of a mixing process. For a t'initenumber of terms the transformation is conveniently expressed using matrix notation. Let .
'
=
x
=()0.j4s,
=
=
tover-differenced'
=0pms.
=
=
=
-
Theory of Stochastic Processes
222 1
0
01 An
01
=
n x n),
(14.48)
0a-2 0n-1 0n-2
01
(-10jzt-j, Zy=0 z (z1,...,za)
1 n can be written x A?g where x is also lower triangulat', with elements = @1,...,ak)and z 0,...,a 1. If >' (v1,...,va)the vector &' Aa # has h replacing 0j for J t-1 f for 1 These operations elements Zyzoljvr-j, can in principle be taken x, subject to 14.9(c). to the limit as n so
that the equations xt z
-1
t
=
,.
=
..,
A,T
= .
=
.
=
-1
%
=
-
=
...,n.
=
,
-->
Proof of 14.9 Without loss of generality, the object is to show that the c-fields 0 = c(...,X-1,Xo) and Ji'x' x. The F-x m+2,..., ) are independent as m m+1 c( X-+1 result does not depend on the choice of origin for the indices. This is shown for a sequence f.f t )7+k for finite p and k, and since k and p are arbitrary, it then follows by the consistency theorem (12.4)that there exists a sequence (XfJ'lx whose finite-dimensional distributions possess the property for every k and p. This sequence is strong mixing on the definition. where 2% = Xt-p,...,Xo)' (px 1), #1 Define a p + m + k-vector X X6X1,XL)' = Xm+L,...,Xm+' k x 1), and also vectors W = (X1,...,Ak)' m x 1), and Xz (W(,W)' and F (FJ,FJ)'such that #1 IFI + F1 and Xz +2 + Fa,.(The elements and F are independent of W. of W and F are defined above 14.11.) The vectors Now, use the notation X = GXg,...,Xt) and define the following sets: ,x
=
-->
=1-p
=
=
=
=
=
.%
0 for some C e e C) e @1-p,
G
=
f(t):
#(,()
H
=
(:
A%() e
DJ
E
=
f ):
Fz()
#) e
e
for some D m e J;'7l,
G
TP ,
Bk ,
10 -x,
where # (p2:Ir21 f n) e T k lv21 denotes the vector whose elements are the is a vector of positive constants. absolute values of p2, and z1 = (q,,,+1,...,n,,,+z)' Also define D pa = (w2:.,2 +#2 e D) e T k =
,
-
.
H may be thought of as the random event that has occun-ed when, first F2 = 1/2 is realized, and then W2 e D p2. By independence, the joint c.d.f. of the variables (Wa,F2,A%) factorizes as F = FwzFh,xo (say) and we can write -
#(&
=
#(X2 e D)
JsJss.va#F(+2,V2,20) =
=
jepjekzvltdFvz,xvl'x'
(14.49)
Mixing
223
where
:(1/2)
#(W2 e D
=
-
v2)
jp-vgdFwzw.
=
( 14.50)
These definitions set the scene for the main business of the proof, which is to show that events G and H are tending to independence as m becomes large. Given 5IB P'm'k ility of X this is sufficient for the result, since C and D are arbitrary. By the same reasoning that gave (14.49),we have -measurab
,
#(G Detine k*
E4
/'7 HIA
supyzssztpz) and
=
X.PIGcjE)
jcjg(v2)#Fvz,xa(v2,&).
=
x.
infpzssztva), and
=
(14.51)
(14.51)implies (14.52)
f PIGCHIAEI < z.PGcjE4.
Hence we have the bounds
PCGchP
+#45rn PCGCDHI-?EI
=
z*#(G)
; and similarly, since
-
k k.P(G4 f
(14.53)
+#(Grnz;faFC)
k z.PGcjE)
= z.P(G)
=
PEc),
+
z. K 1,
PGcllj
Choosing G
H(nEcl
(i.e., C x.
Ec) + #(G chHcj Ec4
fa
P(Ec4.
R#) in
=
PCEC4
-
-
k.PG
( 14.54)
(14.53)and (14.54)gives
f #(& S
z*+
PCEC),
in particular
(14.55)
and combining all these inequalities yields 1#(G rn /f)
1 f x* x++lPEc).
#4G)#(&
-
-
(14.56)
Am+kZ,where Z (Z1,...,Z,,,+k)'and Amn is defined by (14.48).Since I4,?,+I 1 and the (z1,...,Z,,,+kl are independent, the change of variable formula from 8.18 yields the result that G is continuously distributed with Write IP
=
=
=
m+k
/(w) Define B '
(r:
=
y.
-z.
A'I
=
0 v2 ,
G
fztz)
=
#1
I
-z(0)
u 2 sup x(v2) vzB S 2 sup v2G#
JsI
G
11fz,(zr).
=
(14.57)
>1
Bm+k Then the following relations hold: .
j
/ctwz + #2) /c(w2) -
I
dw1
Theor.v of Stochastic Processes
224
jpm..k I /vt.'
u 2 sup, B G
&,
+
v)
l
-yv(w)
dw
m+k
m+k
flJz/z,+ J.,a+z
= 2 #sup. B'
t'%
*'
6
E
p e B'
I-lJz/tz,)
-
>1
>1
m+k
< ZM sup
r)
19,1
(14.58)
,
>-+1
where it is understood in the final inequality (whichis by 14.10) that );, l f W here 6 is defined in condition 14.9(a). The tllird equality substitutes 7 = Am-+1kv 0 if vl = 0 by lower triangularity of Amn. For v e and uses the fact that ;'I #' note that =
1
m+k
m+k
77 I I
l
t-m-
77 jszj 77 zjvt-j
v-,
1=-+1
>-+1
t-m- 1
m+k
S
77 jzujj :7
>-+1
m+k
x
XI
I&ln,-; K
':/1
/=0
A'lnr,
(14.59)
t-nm
assuming has been chosen with elements small enough that the terms in parentheses in the penultimate member do not exceed 6. Tlzis is possible by condition 14.9(c). For the final step, choose r to be the largest order of absolute moment if this is does not exceed 2, and the largest even integer moment, otherwise. Then 'rt
PEc)
lF21 >
P
=
'q)
m+k
P
=
U f I141
>-+1
>
n,l
m+k
m+k
77#(I7,I <>-+1
>
n,)K
>7FI7,I
rn-,r,
(14.60)
J=ra+1
by the Markov inequality, and E j 7, Ir < sup E Iz, IrG,(r),
(14.61)
'
where Gt is given by (14.43),applying 11.15 for r 2 (see(11.65) for the required extension) and Lemma 14.11 for r > 2. Substituting inequalities (14.58), (14.59), (14.60),and (14.61)into (14.56)yields m+k
l#(G
chffl
-
PG4PH4
I
X (n,
+
t=m+3
G(r)n;3.
(14.62)
Mixing
225
i 0 by 14.9(b), it is possible to choose m large enough that (14.59)
Since Gt
andhence (14.62)hold l P(G r'3S)
with
n,
Gtr) 1R1+8 G t (rln-r t for each f > m. We obtain
=
=
m+k
*(G)#(S)
-
l
d
t%(r)1/C1+r' 77 >-+1 X
K
1/(1+r)
77G'r) /=-+1
(14.63)
,
where the right-hand sum is finite by 14.9(b), and goes to zero as m completes the proof. w
This
-->
x.
It is worth examining this argument with care to see how violation of the conditions can lead to trouble. According to (14.56),mixing will follow from two O conditions: the obvious one is that the tail component F2, the F-x-measura bje part of A%,becomes negligible, such that is, #(F) gets close to 1 when m is large, even when Tl is allowed to approach 0. But in addition, to have x* x. disappear, 0, for any D, and whatever #(W2 e D r2) must approach a unique limit as 172 the path of convergence. When the distribution has atoms, it is easy to devise examples where this requirement fails. In 14.6, the set Bt becomes Js Bt on being translated a distance of 2-l For such a case these probabilities evidently x. do not converge, in the limiting case as t However, this is a suftkiency result, and it remains unclearjust how much more than the absence of atoms is strictly necessary. Consider an example where the distribution is continuous, having differentiable p.d.f., but condition (14.42) none the less fails. -
-->
-
-
-..-)
g
.;
14.12 Example Let fz) Coz sin (z ), z e R This is non-negative, continuous everywhere, and bounded by C()c-2 an d hence integrable. By choice of Q we can *fztdz 1, so f is a p.d.f. By the mean value theorem, have =
.
=
Ifz
+
ab
f(z) I
-
=
Ia IIf'z
wheref '(z) 8c osint/lcost/k x and hence, =
-
I
+ a(z)J)
,
2c()sin2(/)g-3
(14.64)
a(z) e (0,11, .
But note that
J+OI f'l ldz
=
-x
1
1
:*
Ifz
J
' Ia I
.-
+ J)
f U)ldz
-
'--h
'x'
as
Ia l
-->
0,
(14.65)
which contrdicts (14.42).The problem is that the density is varying too rapidly in the tails of the distribution, and Ifz + a4 fz) l does not diminish rapidly 0. enough in these regions as a The rate of divergence in (14.65)can be estimated. 15 F or fixed (small)a, sin (z+ a) 4 1 (or 0) Ifz + a) fz) I is at a local maximum at points at which tz)4 z4 4.3 + o(u2) and sin z4 0 (or 1), or in other wor ds w here (z+ :+/2. The solutions to these approximate relations can be written as z = +CI lJI -1/3for C 1 > 0 At these points we can write, again approximately (orders of magnitude are all we need here), -
--
=
-
=
-
.
=
=
Theoty of Stochastic Processes
226
Ifz
a4
+
fz) I S
-
-2 1J
13
I (-C1 It7I 21
lfz)
S ICOC 1
.
-1/3
-1/3j
C 1 jt7l by The integra l is bounded within the interval 1/3 2/3 rectangle having height 2C0C7-1 a I Outside the 4CoC -1 th e area of the I la I and h -2 is bounded by t e integral over this region is bounded by interval,J coz ,
,
.
,
+00 2C0Jcj$(,I-v3Z
Adding up the approximations
dz
for M
<
IJl
2C0Cl
=
.
yields
+x
J..I fz
1/3
.j
.p,
+ J)
fzs 1dz f
-
Ml J
l
js
(14.66)
o
x.
The rate of divergence is critical for relaxing the conditions. Suppose instead of
(14.42) that +oo
J-.I
J)
z+
zbIdz f Mtl
-
could be shown sufficient, where h ltzI) 1, 0 as Ia l 0. Since
JI ), IJ1
f y y s (;
(14.67)
,
an arbitrary increasing function with
(.) is
.
+co
J-.l fz
+ J)
ldz S
f
-
g)+x
(z)tyz m
..f
(14.68)
g
for any a, (14.67)effectively holds for any p.d.f., by the dominated convergence 16 theorem. Simple continuity of the distributions would suffice. This particular result does not seem to be available, but it is possible to relax 14.9(a) substantially, at the cost of an additional restriction on the moving average coefficients. 14.13 Corollary Modify the conditions of 14.9 as follows: for 0 that
;$S
<
1, assume
Zt is uniformly fv-bounded, independent, and continuously distributed with p.d.f. /'z,,and zz-va, (+I sup t
whenever
b-)x-
(
(C ')
tzz)
ct
1/0(x)
<
Itzl
S
(r)I5'(I5+r) <
=
T(x)
-
,
etxl
=
Then Xt is strong mixing
hjz)
:, for some
I
dz
u Mlcl', u
<
(14.69)
-.,
0,'
>
c,(r) is defined in (14.43); for 1x1S 1, and Z'r=I E''=l'r/ J (Tjl 15< J x
,
where
x.
with
m
=
t?(X*t
.,,,+1
Gt (r)lVO+8) .
Proof This follows the proof of 14.9 until (14.58),which becomes
Mixing
227
m+k
z.
-x
+
<
zu
applying the obvious extension m+k
m+k
9,1F
X I
=
using (9.63),since 0
<
y B'
f=?n+1
(14.70)
,
'
of Lemma 14.10. Note that ;$
f-m-j
X X
p=+1
p=?n+l
>'')lf;,1;'
sup
./=0
m+k
x
K
'zjvt-j
XI
X n?
'zj
j=0
(14.71)
f=/?1+1
;'J f 1. Applying assumption
14.13(c'),
m+k
I#(G r-bz,r? -#(G)#(zoI f<
and the
result
is obtained
77 (nf)+c,(r)n-,r),
(14.72)
>-+1
as before, but in this case setting
n,
=
G1,7t+* .
.
Condition 14.13(b') is satisfied when l0./1 ./-B for g > 1/p + 2/r when r < 2, and g > 1/2 + 1/r+ 1/j3 when r 2, which shows how the summability restrictions have to be strengthened when ;'$is close to 0. This is none the less a useful extension because there are important cases where 14.13(b') and 14.13(c') are easily A, both I%I and I satisfied. In pmicular, if the process is finite-order vanish beyond some finite j, and (b') and (c') either decline geometrically or both hold. Condition 14.13(a') is a strengthening of continuity since there exist functions h.) which are slowly varying at 0, that is, which approach 0 more slowly than any positive power of the argument. Look again at 14.12, and note that setting 13 la will satisfy condition 14.13(a') according to (14.65).lt is easy to generalize k Csin 2(z)z for k k 4 the earlier argument is easily the example. Putting j'z) modified to show that the integral converges at the rate Ia 1.1/(k-1)an d tus choice of f'sis appropriate. But for J(z) = Csin 2 e z)z th e integral converges F more s lowly than Ia I for al1 13> 0, and condition 14.13(a') fails. To conclude this chapter, we look at the case of uniform mixing. Manipulating inequalitie (14.52)414.55)yields 'ry1
=
-2
=
,
,
-2
IPIH' G) #4/./)1 S -
(1
z* z. PCEY + -
+
plv) ,
(14.73)
1 for all m exceeding a which shows that unifonn mixing can fail unless #(F) tinite value. Otherwise, we can always construct a sequence of events G whose probability is positive but approaching 0 no slower than PEc4. When the support of X-v,...,Xo4 is unbounded this kind of thing can occur, as illustrated by 14.8. =
The essence of this example does not depend on the AR(1) model, and similar cases could be constructed in the general MA(x) fzamework. Suftkient conditions must include a.s. boundedness of the distributions, and the summability conditions are also modified. We will adapt the extended version of the strong mixing condition in 14.13, although it is easy to deduce the relationship between these conditions and 14.9 by setting f$ 1 below. =
228
Theoty of Stochastic Processes
14.14 Theorem Modify the conditions of 14.13 as follows. Let (a') and (c') hold as before, but replace (b') by (b'') f.o I j I)l5< x, add and (d) (Z,l is uniformly bounded a.s. T hen (mJ is uniform mixing with (,,, O; =,,,+1(E7=f I0./1)f5). XO
xt
=
Proof Follow the proof of 14.9 up to (14.55),but replace (14.56)by (14.73).By condition 14.14(d), th'ere exists K < x such that suprl Zt I < K a.s., and hence I-Y,I 1 < K;=o I0.j1a.s. It further follows, recalling the definition of Fa, that PE) +k. Substituting directly into (14.73) 0./1 for t m + 1,...,< A'Z7=,1 when n, of and a nd making choice this from (14.70) (14.71), n, gives (forany G with #(G) > 0) =
=
,
I,(,.,1
c) rm I -
,
-s+(s-yojj)
(14.74)
.
l=?n+1 j=t
The result now follows by the same considerations as before.
.
x in the These summability conditions are tougher than in 14.13. Letting r OU-B) 14.13(b') for for comparability, satistied is when latter case I%I g, > 1/2 + 1/p, while the corresponding implication of 14.14(b'') is jt > 1 + 1/p. -->
=
15 Martingales
15.1 Sequential Conditioning It is trivial to observe that the arrow of time is unidirectional. Even though we can study a sample realization expost, we know that, when a random sequence is member Xt is determined in an environment in which the generated, the previous members, Xt-k for k > 0, are given and conditionally fixed, whereas the membersfollowingremaincontingent. Thepastisknown, butthefutureis unknown. The operation of conditioning sequentially on past events is therefore of central by specifyimportance intime-series modelling. Wecharacterizepmialknowledge ing a c-subtield of events from @, for which it is known whether each of the events belonging to it has occurred or not. The accumulation of infonnation by an observer as time passes is represented by an increasing sequence of c-fields, F.17 1T,1= suc h that 51 i i @-1 i J70 Tl If Xt is a random variable that is Tr-measurable for each /, ( Tfl=-xis said to The pairs (A,@,)=-x are called an adapted be adapted to the sequence ta)l=-x. /) < s < defines the minimal adapted sequence, but sequence. Setting 9t ctxx, 5t typically has the intepretation of an observer's infonnation set, and can contain more information than the history of a single variable. When Xt is integrable, the conditional expectations A'(ml Fr-1) are defined, and can be thought of as the optimal predictors of Xt from the point of view of observers looking one period ahead (compare10.12). Consider an adapted sequence l&,Tnlo-x on a probability space L't,5nP4, where ( kl is an increasing sequence. If the properties dcurrent'
...?
-x,
...
-x
=
FI,L1 < Esn
I
.n-1)
=
(15.1)
x, s%-l,
a.s.,
(15.2)
hold for every n, the sequence is called a martingale. ln old-fashioned gambling parlance, a mmingale was a policy of attempting to recoup a loss by doubling one's stake on the next bet, but the modern usage of the term in probability theory is closer to describing a gambler's worth in the course of a sequence of fair bets. In view of (10.18),an alternative version of condition (15.2)is Sndp
A
=
Sn-)dP, each A e @n-1.
A
Sometimes the sequence has a tinite lnitialindex, and may be written where 5'1 is an arbitrary integrable r.v.
(15.3)
f,L,F,,lT
F/ldt?ry of Stochastic Processes
230
(X,ITbe an i.i.d.
integrable sequence with zero mean. If Sn Z7=l-Y, and 5n c(Xn,X,,-1,...,X1), l5'q,?LlT is a mmingale, also known as a A'lil walk sequence. Note that S Z7=1A'l-Y,l < x. n random
15.1Example
Let
=
=
15.2 Example Let Z be an integrable, T/f-measurable, zero-mean r.v., ( Kl'-x an increasing sequence of c-tields with limn-yxfn = T, and Sn #tzj ;n). Then E(Sn3 9a-1) F(F(zI@n)I @n-1) F(ZIT,,-1) = .L-1, (15.4) =
=
wherethe second o martingale.
=
equality is by 10.26(i). F1 ,L1 <
FIzI <
by 10.27, so Sn is a
x
on the last definition, a martingale d#erence (m.d.) sequence on L't,5,P) satisfying the properties
Following
fm,sl'<'-x is an adapted sequence
Fl#r1 F(Al Tr-l)
(15.5)
< x, =
(15.6)
0, a.s.,
for every t. Evidently, if fSn) is a mmingale and Xt St St-j. then (.1 is a m.d. Conversely, we may define a martingale as the partial sum of a sequence of m.d.s, as in 15.1 (an independent integrable sequence is clearly a m.d.). However, if Xt has positive variance unifonnly in /, condition (15.1) holds for a1l finite n but not unifonnly in n. To define a mmingale by Sn L1=-=Xt can therefore lead to difticulties. Example 15.2 shows how a mmingale can arise without reference to summation of a difference sequence. It is important not to misunderstand the force of the integrability requirement in (15.1).After all, if we observe Sn-b, predicting Sn might seem to be just a matter of knowing something about the distribution of the increment. The problem is that we cannot eat Esn I&-1,...)as a rt7rltftprl variable without integrability of Sn. Conditioning on S-l is not the same proposition as treating it as a constant, which entails restricting the probability space entirely to the set of repeated random draWings of Xn. The latter problem has no connection with the theory of random sequences. A fundamental result is that a m.d. is uncorrelated with any measurable function of its lagged values. =
-
=
15.3 Theorem If (7r,@t)is a m.d., then Cov(X,,9(.X)-1,X,-2,...))
where
(1) is
any Borel-measurable,
Proof By 10.11
(sze also
=
0,
integrable function of the arguments.
the remarks following) noting that 9(A-1,.&-2,...), is
Tf-l-measurable. w
15.4 Corollary lf (.,121 is a m.d., then E(XtXt-k) = 0, for a1l t and all k
#
Proof Put () Xt-k in 15.3. For k < 0, redefine the subscripts, putting t and t' Il7I t, so as to make the two cases equivalent. m
t k
=
-
=
=
0. -
Martingales
231
One might think of the m.d. propel'ty as intennediate between uncorrelatedness and independence in the hierarchy of constraints on dependence. However, note the asymmetry with respect to time. Reversing the time ordering of an independent sequenceyields anotherindependent sequence, andlikewise areversed uncorrelated sequence is uncorrelated; but a reversed m.d. is not a m.d. in general. The Doob decomposition of an integrable sequence (&,?L)7' is Sn
where Ao
=
0, Mo
(15.7)
Mn +An,
=
%, and
=
Mn An
Mn-t
=
=
.4u-1
I k-1),
(15.8)
Esn lTn-1) 5'n-1.
(15.9)
+
+
Sn Esn -
-
component of Sn.Writing Anis an Fn-l-measurable sequence called Lbepredictable and A,4u hsl'n hSn Xn, we find F(Fn l;t-1), and L =
%
=
=
Xn
=
rn -
E(Ynl?h-1).
(15. 10)
Xn is known as a centred sequence, and also as the innovation sequence of Sn. It is adapted if ( l'n,iu J7'is, and since Fl Fn1 < oo by assumption, E IXn I < E Il',yI + E IEYn lJ;,,-1)I
l Fn IITn-l))
E 1Fn I + EE
=
IE
IraI <
(15.11)
x,
Minkowski' s inequality, the conditional modulus inequality, and by (respectively) 0, (.L,@n)7is a m.d. and so the LIE. Since it is evident that E(Xt I?h-1) mmingale. is lMn,Tnl7 a =
Martingales play an indispensable role in modern probability theory, because m.d.sbehave inmany importantrespects likeindependentsequences. Independence limit results, laws of is the simplifying propel'ty which permitted the large numbers and central limit theorems, to be proved. But independence is a constraint on the entire joint distribution of the sequence. The m.d. property is a much milder restriction on the memory and yet, as we shall see in later chapters, most limit theorem: which hold for independnt sequences can also be proved for m.d.s, with few if any additional restrictions on the marginal distributions. For time series applications, it makes sense to go directly to the martingale version of any result of interest, unless of course a still weaker assumption will suftke. We will rarely need a stronger one. Should we prefer to avoid the use of a definition involving c-fields on an abstract probability space, it is possible to represent a martingale difference as, for example, a sequence with the property iclassical'
E(xt IA-1,Xr-2,...)
=
0 a-s.
( 15. 12)
When a random variable appears in a conditioning set it is to be understood as representing the corresponding minimal c-subfield, in this case c(#f-1sXf-2,...).
Theory of Stochastic Processes
232 tj
This is appealing at an elementary level since it captures the notion of information available to an observer, in this case the sequence realization to date. But since, as we have seen, the conditioning information can extend more widely than the llistory of the sequence itself, this type of notation is relatively clumsy. Suppose we have a vector sequence (4XsZf)), and Xr-though not necessarily c(XsZsXr-1,Zf-1,...) in the sense of Zt is a m.d. with respect to 5t (15.6). This case is distinct from (15.12),and shws that that definition is inadequate, although (15.16)implies (15.12).More important, the representation of conditioning information is not unique, and we have seen (10.3(ii))that any measurably isomorphic transformation of the conditioning variables contains the same information as the oliginal variables. lndeed, the information need not even be represented by a variable, but is merely knowledge of the occurrence/nonoccurrenceof certain abstract events. =
-
15.2 Extensions of the Martingale Concept k Jn=1, w here An adapted triangular array ( tAk, @nf)?21 sequence of integers, for which '-'
'1Xnrl
<
(/(n )= n,z 1 is some
(15.13)
oo,
ft-vtl 9$,f-1)
(15.14)
0 a.s.
=
increasing
for each t 1,...,lk and n k 1, is called a ntartingale dterence array. In many applications we would have just kn n. The double subscripting of the subfield 5nt may be superfluous if the infonnation content of the array does not depend on n, with 5nt 5t for each n, but the additional generality given by the definition k Xnran d 5 n is harmless and could be useful. The sequence t&,Tn ITwhere Sn Xf21 5n,knis not a martingale, but the properties of mmingales can be profitably used to analyse its behaviour. Consider the case Sn = n -1/2E)=1 xt w Iwre (m,@;)is m.d. distdbution of Sn has a Such scaling by sample size the that may ensure a non-degenerate limit. Sn is not a martingale since =
=
=
=
=
ITn-1)
Esn
1/2
E(/l 1)/n1
=
-
5'n-l,
(15.15)
but each column of the m.d. array X1 2 -
-1/2
1/2
X) 3
2 -1/2X2 3
-
X1 4-1/2x 1 X1 4-1/2x 2
.
.
.
.
.
.
.
.
1/2
3 -1/2Xa
(15. 16)
-1/2
4
X5
4 -1/2.X4
is a m.d. sequence, and S,l is the sum of column n. It is a tenn in a mmingale sequence even though this is not the sequence (5'n1. An adapted sequence
of fal-bounded (&,@s)=-x
variables
satisfying
Martingales
IFa)
ESn+L
233
k Sn a.s.
(15.17)
is called a submartingale, in which case Xn Sn &-l is a submalingale difference, having the property E(Xn+3lJ;n) 2 0 a.s. In the Doob decomposition of a submartingale, the predictable sequence An is non-decreasing. Reversing the although, since -Sn is a supermmingale inequality defines a supermartingale, extension. A supermartingale might submartingale, this is minor whenever Snis a a represent a gambler's worth when a sequence of bets is unfair because of a house percentage. The generic tenn semimartingale covers al1 the possibilities. =
-
15.5 Theorem Let 94.):R i.-,b R be continuous and convex. If (&,TnJis a martinthen (9(&),Fn) is a submmingale. If () is also nongale and Fl ((Sn) l < decreasing, (Y(&),8L ) is a submartingale if (&,8Ll a submartingale. =,
Proof For the martingale case,
ITn) f(#(.%+1)
#(F(&+1 I@n))=
a.s. #(u%)
(15.18)
by the conditional Jensen inequality (10.18).For the submartingale case, in (15.18)when xl S x2 = (1)(x1) < 9(.n).x becomes
4='
E2'
If (.X),TfITis a (sublmmingaledifference, (Zr,@rl7' any adapted sequence, and
sn Xx,z,-1, ,.1
(15.19)
=
then
(&,L )1' is Esn..t
a
since (sublmartingale
I@n)
=
XX,z,-l >1
+
ZnEXn+3I?L)
=
Sn
(
(15.20)
'n).
We might think of Xt as the random return on a stake of 1 unit in a sequence of bets, and the sequence (4) as representing a betting system, a rule based on information available at time t 1 for deciding how many units to bet in the next game. The implication of (15.20)is that, if the basic game (in which the same stake is bet every time) is fair, there is no betting system (basedon no more than information about past play) that can turn it into a game favouring the player or for that matter, a game favouring the house into a fair game. ,5,P), a stopping time For an increasing sequence (1,1 of c-subfields of is a random integer having the property f : t ) e 5t. The classic example is gambling which entails withdrawing policy from the game whenever a certain a conditiop depending only on the outcomes to date (suchas one's losses exceeding some limit, or a certain number of successive wins) is realized. If is the random variable defined as the first time the said condition is met in a sequence of bets, it is a stopping time. be a stopping time of f@n Let ), and consider -
-
..:(r4))
'r()
=
'r
'r
Snxz
=
Sn, n ;
%, n
':
> T
(15.21)
Theory of Stochastic Processes
234
where n zx T stands for minf
l,'rl.
fSnoysn
l'X'n.l
is called a stopped process.
then t,Sv,T,, 15.6 Theorem If fSn,5n 1Tis a martingale (submartingale), martingale (submartingale).
JTis
a
Proof Since f 5n 1Tis increasing, f ): k T()) ) e 5n for k < rl, and hence also -1 1(t=z) + n j (a
'r()))
=
n-1
FlSnvl
f TE
151171(1....,:) l +e'lx%1(n
&1
l
n-1
+FxL l < < Xflxsj =1
n k 1.
=,
(15.22)
lf (,L,Tn JTis a martingale then for A e 5n, applying (15.3),
jghnwbjxxdp =
=
jgcsqncgljdp Jxc,jsszlx'lzd' +
Sndp +
Xfafn<ecl
S4P
drnfnll
Snszdp,
=
A
(15.23)
showing that (&+1,u,@n)T is a martingale. The submmingale case follows easily on replacing the second equality by the required inequality in (15.23).* The general conclusion is that a gambler cannot alter the basic fainzess characteristics of a game, whatever gambling policy (bettingsystem plus stopping rulel he or she selects. Al1 these concepts have a natural extension to random vectors. An adapted sequence (A),@r)=-xis detined to be a vector rnzzrffrlgtzfe dterence if and only if (A'X,F,J*-xis a scalar m.d. sequence for a11conformable tixedvectors A # 0. It has the property
f(X+1I5tj
=
(15.24)
0.
The one thing to remember is that a vector martingale difference is not the same thing as a vector of martingale differences. A simple counter-example is the twois an element vector Xt = XoXt-q)'s where Xt is a m.d.; fLX, + V2.Y;-1,@,J adapted sequence, but EhLxt
+
L4-j J?h-1)
=
l2A-j
#
0,
so it is not a m.d.. On the other hand,
f(l-V+l + klxt ITr-l) but (:14+1+ hzxt, ?&) is not adapted, since
=
ml
0, is not Tsmeasurable.
Martingales
235
15.3 Martingale Convergence Applying 15.5 to the case ()4.) l IP and taking unconditional that every martingale or submartingale has the property =
FI5'n+11: 2
.
F1 snlP, p
expectations shows
1.
(15.25)
x, either to a By 2.11 the sequence of pth absolute moments converges as n uniformly where +x. the limit In bounded, Ll-norms finite the are or to case (sublmartingales also exhibit a substantially stronger property; they converge, almost surely, to some point which is random in the sense of having a distribution over realizations, but does not change from one time period t to the next. The intuition is reasonably transparent. (Tnl is an increasing sequence of c-fields which converges to a limit Tx c ;, the c-field that contains nnfor every n. Since Esn ITn) Sn, the convergence of the sequence (Tn) implies that of a uniformly bounded sequence with the property ESn+L I1n) k Sn, so long as these expectations remain well-defined in the limit. Thus, we have the following. --
=
15.7 Theorem If (&,;n)Tis a submartingale sequence and supnfl Sn l f M < x, then Sn S a.s. where S is a i-measurable random variable with FI5'1 S M. Ia --
The proof of 15.7, due to Doob, makes use of a result called the upcrossing inequality, which is proved as a preliminary lemma. Considering the path of a submartingale tluough time, an upcrossing of an interval (a,p1 is a succession of steps starting at or below G and terminating at or above p.To complete more than one upcrossing, there must be one and only one intervening downcrossing, so downcrossings do not require separate consideration. Fig. 15.1 shows two upcrossings of ga,pq, spanning the periods marked by dots on the abscissa.
O O
O O
O O
O O
Fig. 15.1 Let the r.v. Fk be the indicator of gn upcrossing. and then, for k = 2,3,...,n,
To be precise, set
i'I
=
0,
236
Theory of Stochastic Processes
0 if either
1%
Fz-I
=
0, Sk-k
=
0, Sk-k K
a, or l'-1
>
=
1 if either Fz-j
=
1, Sk-j k
or Fk-1 = 1, Sk-j <
(x,
p, p.
(15.26)
The values Of l'k appear at the bottom of Fig. 15.1. Observe that an upcrossing begins the period after Sk falls to or below a, and ends at the tirst step thereafter where ;$is reached or exceeded. 1%is a function of Sk-k and an K-l-measurable random variable. The number of upcrossings of (a,j$)up to time n of the sequence (,s'a((l))17, to be variable. The denoted &a()), is an Fa-measurable random sequence ( 1&4)1Tis condition. satisfies but it following the monotone, by a submartin15.8 Upcrossing inequality The number of upcrossings of g(x,f$J gale (&,@aJTsatisfies Eun)
E lsn1+ Ia
f
l
j) tx
(15.27)
.
-
Proof Define Sn' maxlxa,Gl, a continuous, convex, non-decreasing function of Sn, such that fSn',snj is an adapted sequence and also a submartingale. Un is the set of upcrossings up to n for (&') as well as for (&). Write =
n
Sn 5''l -
=
NXYj X Fkyj X (1 a2
n
X (1 *=2
+
-
J%)xL/
(15.28)
,
&2
&2
where l'k is f'rom (15.26),and E
11
& =
XLis
a submmingale
difference. Then
l
-
i'1)xI
=
X jyg-oixLap k=2
=X a2 jl
ExLbsk-bldp. e,
1*k=01
(15.29)
using the definition of a conditional expectation in the second equality (recalling that i'k is g-j-measurable), and the submartingale property, to give the inequality. We have therefore shown that Esn'
,$() -
l'kAk' X &2
k E
(15.30)
.
X2=21%Xis the sum of the steps made during upcrossings, by definition of J%. Since the sum of the XLover an upcrossing equals at least ;$ G by detinition, we must have -
M
X l%-YI(p
-
a)&s,
(15.31)
&2
where Un is the number of upcrossings completed by time n. Taking the expectation of (15.31)and substituting (15.30),we obtain, as required,
Martingales
(;3 -
a)F(&n) f Eu
5') f Esn'
-
-
237
a)
-
Ifs-ozjsdp
-
E 1Sn I + Ia
< FIsn a 1 < -
(15.32)
I .
w
The upcrossing inequality contains the implication that, if the sequence is uniformly bounded in f.l, the expected number of upcrossings is finite, even as n .,..4x. This is the heart of the convergence proof, for it means that the sequence has to be settling down somewhere beyond a certain point.
f3>
Proof of 15.7 Fix (x and
a. By 15.8,
e'1snI + laI
EUn) <
j3 a
M+ l(x I
S
f3 a
<
(15.33)
x.
-
-
is a positive, non-decreasing sequence and either diverges For ) e (, (!.&((l))J:' x. +c,o Divergence for (t) e C with to or converges to a finite limit U(@ as n whichcontradicts #(C) > 0 would imply EUn) (15.33),so Un-.,-qUa.s., where --
--x,
fl&)
<
=.
and s((1)) R(t)l limsupn-oxxS
liminfu-yxs'n. lf s()) < (x < I < R)), the x, interval (a,Iqis crossed an infinite number of times as n so it must be the < < < pair is for (x a,p. Hence considef any I X) 0. This tnle case that #(j' Define
=
=
-,..:
=
lt0;srtl
<
iltllll
Uls' f
=
tx
;$f X1,
<
a,fs where the union on the right is taken over rational values of (x and #(j' < X) 0 by 3.6(ii), which is the same as j' S S a.s., where of fSn ) Finally, note that =
=
=
(15.34)
0. Evidently,
,5'
is the limit
.
FI5:1 S
liminf E Ivn
If
sup
'l-/oo
FIsnI <
(15.35)
M,
n
where the first inequality is from Fatou's lemma and the last is by assumption. This completes the proof. . Of the examples quoted eaflier, 15.1 does not satisfy the conditions of 15.7. A random walk does not converge, but wanders forever with a variance that is an increasing function of time. But in 15.2, Xt is of course converging to Z. 15.9 Corollary Let fSu,5n l=-xbe a doubly infinite mmingale. where S-x is an Al-bounded nv. as n
Then Sn
--->
S-= a.s.
-->
-x,
Proof
Let U-n denote ihe umber of upcrossings of (a,j31performed by the The argument of 15.8 shows that k j 2 f'j, -1
-nl.
sequence
EU-i,4 ;
FIsj I + Ia I pra
,
all n
1.
Arguments precisely ana 1og ous to those of 15.7 show that
(15.36)
Theoty of Stochastic Processes
238
# liminf Sn < limsup Sn
0,
=
(15.37)
n'-i-co
N-)-x
1
- is non-nega tive nonsequence (FI5k1 1-x so that the limit 5'-x exists a.s. cxo by a nd decreases < <Sdefinition of a martinby increasingas n (15.25), FI l gale. Hence FI5'-.I < x. . If a martingale does not converge, it must not be thought of as converging in F, 'rhe
,
of heading off to +x or never to return. This is an event that occurs only probability the increments having a suitably bounded distribuwith 0. Subject to tion, a nonconvergent mmingale eventually visits al1 regions of the real line, almost surely. -x,
suptlxf
15.10 Theorem Let (Xf,Ffl be a m.d. sequence with
1J=1kt.lf
C
(:
=
E
then P(E
-
C)
=
Snk
((:
=
converges
either
), and
infnxst)
>
or supnxst)
-x
I) <
<
cxa,
and 1etSn =
xl,
0.
Proof For a constant M > 0, define the stopping time Twz((o) as the smallest integer otherwise. The stopped process n such that Sn > M, if one exists, and IA(t) tx,vhn-tlxxu Tnwwl-i-rl is a martingale (15.6),and S M for al1 n. Letting fsnxzu, = S'Lzu maxf 5kZ.,OJ, =
s+nh'w
<
s+( n- 1),xw+
X'bnxzu< M+ sup
I-LI
(15.38)
.
n
= Since Ezp 0, Fj Swwl = zft5Xwl,and hence supne'l skwj < x, and Snvsxuconverges a.s., by 15.7. And since Snvsxufk Snk on the set f : supnxst) S MJ, converges a.s. on the same set. Letting M---k x, and then applying the we obtain the conclusion that Snlk converges a.s. on the same argument to that is, #(Cn E; set AD= #(A-), from which the theorem follows. . =
k%()
-%,
+x, infniat) = Since Note that Ec = ( :supnlnt) P(Cc t.p Ec), a direct consequence of the theorem is that Cc = #(m = 0, which is the claim made above. -x1
#(FC)
=
#((Cf'7F)6) ECDN where =
.
15.4 Convergence and the Conditional Variances lf (&)
is a square-integrable mmingale with diferences fAkl, E(Sln-1 + Xl + J,x S j Tu-j) k &2-j Esn 2j 5 n-1
)
=
-j
n
n n
,
1 is a submartingale.The Doob decomposition of the sequence of squares has and Su and Aa = E(X2n j5 n The the form Sn2 = Mn +A n where hhl'n X1n E(X1n j @n sequence fAn) is called the quadratic variation of f&) The following thorem reveal an intimatelinkbetween mmingaleconvergence and the summability of the conditional variances; the latter propel'ty implies the former almost surely, and AQj ?=1 E in particular, if '=' ( t 5 t-1 ) < x a.s. then Sn S a.s. -j),
=
-1).
-
.
-..4
Martingales
15.11Theorem
Let
C
then #(D
C)
=
a m.d. sequence, and Sn X:=l-V.lf xl : :7.1F(xl e @, I@,-j)() <
(X,,F,)Tbe D
-
239
=
=
0.
( f
:
=
sn
convergesl
e F,
Proof Fix M > 0, and define the stopping time ,1A#(l)) as the smallest value of n having the property Xe'(A
I@,-l)(to)
>1
(15.39)
M.
If there is no finite integer with this property then ,:A#)) = x. If DM ( : 'rA/.((l)) x), D limv-yxfk. The r.v. 1(wka)() is Fs-l-measurable, since it is known at = time n 1 whether the inequality in (15.39)is true. Detine the stopped process =
=
-
N
A7A1(zvvf)
Snxxu.
=
(15.40)
,.=1 is a snvsxu
martingale by 15.6. The increments are orthogonal, and Esnhxu)
sup n
N
=
sup E n
XV ltwkul >1
n
=
supf n
77ltwknlftl
@r-1)
<
M,
(15.41)
>1
where the final inequality holds for the expectation since it holds for each by definition of 1M((t)). By Liapunov's inequality,
supFIskwl N
S sup n
(.t) e
f
< Ml Il&.wII2
and hence Snxzu converges a.s., by 15.7. If ) e DM, xskss/)) xS4))for every n e EN,and hence Sn1 converges, except for ) tna set of zero measure. That is, PDu rn C) PDM4. The theorem follows on taking complements, and then letling =
=
M
-->
x.
.
15.12 Example To get an idea of what convergence entails, consider the case of
(Xf ) an i.i.d. sequence (compare15.1). Then Lxtlatjis a m.d. sequence for any Exlt @f-1) F(aYl) = c2, constant sequence (t7,)o f p ositive constants. Since I a which we assume finite, Sn Xlcjxlat is an a.s. convergent martingale whenever x jlat 2 < For example, at t would satisfy the requirement. a =
=
t
.1
(x,.
=
In the almost sure case of Theorem 15.11 (when#(C) #4D) 1), the summability of the conditional variances transfers to that of the ordinary varipnces, 2 Ftsupfxl) < x, the summability of the conditional variances cf2 EXt). Also w h en is almost equivalent to the summability of the Xt2 themse lves. These are consequences of the following pair of useful results. =
=
=
Theory of Stochastic Processes
240
15.13 Theorem Let lZ?) be a any non-negative stochastic sequence. (i) X*t=jEZt) < x if and only if :7 =1f(Zt1 7t-1) < cxa a.s. (ii) lf Asupzf) < x thn PD A E) 0, where =
D
=
E
=
( ): Z=f=qE(Zt197,-1)4) ( ): Z'';=lZr() < ) x
<
cxo
)
,
.
Proof (i) The first of the sums is the expected value of the second, so the if' part is immediate. Since F(ztl F,-1) is undetined unless Ezt) < x, we may assume &1=qE(Zt)< x for each finite n. These partial sums form a monotone series which either converges to a finite limit or diverges to +x. Suppose Z:=1F(Zrl Tf-1) IYf-1) 0 a.s. converges a.s, implying (bythe Cauchy criterion) that L1+=mn+3EZt &I*=mn+jEZt) Then by the monotoneconvergence theorem, so that as m A n by the same criterion k1=jEZt) k*t=jEZt) < x, as required. = (ii) Define the m.d. sequence Xt Zt F(Z,I1f-l), and 1et Sn = Z:=1m.Clearly supnskt) f k*t=1Z,((t)), and if the majorant side of this inequality is finite, Sn1l)) converges in almost every case, by 15.10. Given the definition of Xt, this 9f-1)()) < x. ln other words, PE D) 0. Now implies in turn that Z7=IF(Z/I apply the same argument to -Xt = E(Zt 111-1) Zt to show that the reverse implication holds almost surely, and PD F) = 0 also. . Konly
-->
-.+0
--)x.
--y
-
-
=
-
-
15.5 Martingale lnequalities Of the many interesting results that can be proved for mmingales, certain inequalities are essential tools of limit theory. Of particular importance are maximal inequalities, which place bounds on the extreme behaviour a sequence is capable of over a succession of steps. We prove two related results of this type. The tirst, a sophisticated cousin of the Markov inequality, was originally proved by Kolmogorov for the case where (Xr) is an independent sequence rather than a m.d., and in this fonn is known as Kolmogorov's inequality. 15.14 Theorem
Let
(&,Tn)Tbe
a martingale. For any p 2
P max lSkI > E I
E lsnj#
1Sk
Proof Define the events A1 A1 =
):
=
f
max #Sj
1u./<1
:
I>
l
kh'14)
IS
-
EP
E,
(15.42)
.
:1,
and for k
1skl) l >
e
=
2,...,n,
e 5k.
The collection A1,...,An is disjoint, and UWz =l
=
max l51
lsfn
>
e,
.
(15.43)
Martingales
c l I51
Since A1
241
el, the Markov inequality (9.10)gives
>
I IP1z:).
-pe'
PA
(15.44)
'k
<
P
By 15.5, ISn I for p 1 is a submartingale, so k f n. Since At e 5k, it follows that '%lP1Ak)
f(1
I&IP
ISnIP I1k)1xJ F( ISn
I EE
=
f
IP1zk),
lz4k where the equality applies. (10.18).Noting X2=1 required, (15.43)415.45), as
=
n
P max 1
I
.%1
>
s
for 1 <
(15.45)
1U2-lxk, we obtain from
n
X#(All
=
F(I&IPI5k4 a.s.,
<
=1
IsnIP1z:) Z-PTEC k=l
N
=
lzu PT k=1
Z-PE ISn I
1sL1#).w Z-PEC
f
(15.46)
The second result converts the probability bound of 15.14 into a moment inequal-
ity.
(&,FnJTbe
15.15 Doob's inequality Let
e'
p
Isklp s
max lffn
-
a martingale. For p p
1
s Isn I
p
lus;n
1
,%l
>
f E-1F( ISnIltmaxlsksalslo
s
(15.47)
.
(15.46)for the
Proof Consider the penultimate member of P max
>
case p
c)),
=
1, that is,
(15.48)
and apply the following ingenioud lemma involving integration by parts. 15.16 Lemma Let X and F be non-negative nv.s. If #(X > 1)1PF(F#), for p > 1. all E > 0, then F(XP) f lp
E)
S E-1F(F1(x>c)) for
-
Proof Letting Fx denote the marginal c.d.f. of X, and integrating by parts, using #(1 Fx) -#Fx and RP(1 Fx@))17 0, -
=
=
-
Exp
j,
-jhpdk
'jpdFxv
=
=
-
l
Xp(#-1(1
=
Fx(())
,
-
Fx6))X =
j*pLP-$PX o
>
j)#j.
(15.49)
> 1 when Define the function 1fx>j)@) (, and 0 otherwise. Letting Fxy denote the joint c.d.f. of X and F and substituting the assumption of the lemma into (15.49),we have =
Ex
upj-r-zstrlfxxklld
p)
0
.7
Theory of Stochastic Processes
242
(J(scsF1tx>()(I)JFx.r(I'F))X
pjo-ie-l
=
-
pj
- 11 =
jkp-zm
y
(R2)+
P
dpkvx,y)
0
jpz,-yx'--'dFkYx-ys
styxp-l;
(15.50)
.
-1
Here (R2)+ denotes the non-negative orthant of R2, or g0,x)x g0,x).The second equality is permitted by Tonelli's theorem, noting that the function Fxrt defines (R3)+. By Hlder's inequality, a c-finite product measure on ECYXP
-j ) K E ly (josl-lytxp;
Substituting into the majorant side of
(15.50)and
.
simplifying
To complete the proof of 15.15, apply the lemma to and i' putting x maxlsjss I I x .%1
,%1
=
=
gives the result. w
(15.48)to
yield
(15.47),
.
Because of the orthogonality of the differences, we have the interesting property of a mmingale l5'nlr that EzSll
=
E
XV
(15.51)
,
>1
where, with s'tl for the case p with the variance property extended
0, Xt St St-k. This lets us extend the last two inequalities 2, to link Ptmaxlsksa lSnl > :) and Ftmaxlxlsa) directly of the increments. It would be most useful if this type of to other values of p, in particular for p e (0,2). One approach to this problem is th von Bahr-Essen inequality of j11.5. Obviously, 11.15 has a direct application to martingales. =
=
-
=
15.17 Theorem
lf
is a (aYf,7f1T
m.d. sequenc
and Sn
=
1=3Xt,
E lsnIP < 2:7 E lx,IP
(15.52)
>1
for 0
<
p f 2.
Proof This is by iterating 11.15 with F Xn, V Tn-la and X = as in the m.d. sequences just as argument leading to (11.65);note that the latter holds for kv-j,
=
for lndependentsequences.
=
.
Another route to this type of result is Burkholder's
inequality (Burkholder
1973). 15.18 Theorem
Let (&,Fn
ITbe a martingale
with increments Xt = St St-j, and -
Martingales 0. For 0 p, such that So
p f 1 there exist positive constants cp and Cp, depending only on
<
=
243
n
p
g
c,l X.xt
n
K E Isn1y <
>1
p A-,) g
c?,f Xj
.
(15.53)
n
/=
side, this extends by the cr inequality to
On the majorant
n
IP
FISn I
p
XF t
< Cp E
n
<
>1
lP,
c XFI Xt 1 p
0
<
p S 1,
(15.54)
>1
which differs from (15.52) only in the specitied constant. 18 I n fact
the
,
Burkholder inequality holds for p > 1 also, although we shall not need to use this becomes result and extending the proof is fairly laborious. Concavity of applied in Readers the that be convexity, so reverse. arguments have to may like to attempt this as an exercise. lemma. The proof employs the following non-probabilistic .?
mbers with yl > 0, and of non-negative < S 1, Then, for 0 p
15.19Lemma Let fyfJ be a sequence 1etFn )7=1y?for t 1 and J'tj 0. =
=
&
F$ < y
+
F--ly, PX >2
<
(1+BP4YL,
(15.55)
where Bp k 0 is a t'inite constant depending only on p. 1 this is trivial, with Bp Proof For p in a Taylor series of tirst order to get =
1'$ where 0n
G
F$-1 +P(l%-1
=
&
=
y;
+
0. Otherwise, expand F$
px
=
ot'nf-1yn,
(J'f-1 +
(L-1 A-yn? (15 56) .
difference equation in
(0,11.Solving the YP
+
=
(15.56)yields
tytf-yt.
(15.57)
>2
Defining Kt
we obtain
-1
#-1
F,-1
=
(F,-1 +
-
%y,)P
(15.58)
,
the result by showing that n
0
< pxKtyt >2
<
BpYL.
(15.59)
r r f The left-hand inequality is immediate. For the right-hand one, note that y fbr y > x > 0 and 0 < r < 1 (see(9.63)).Hence, (15.58)implies that @ -x
-x)r,
1 -.p
Kt K
1
Ff-1
It follows that
1 -
Fl-1 +
a
tyt
=
(Ff-1(Ff-1 + 0ly)
p-1
1.v
1-p
0f yt
.
(15.60)
Theory of Stochastic Processes
244
2,-2 l-p 0 S Ktyt < F,-1 yt
(15.61)
,
and hence n F-P#XA-rzJ n
n
< pv@tlYt-tlls-ps(.y
t /l'n)''
>2
>2
(A''71''-1)X1
-p'
= PT'''/ ?
'P
yt
r-a
E Bpn),
(15.62)
where y; = ytlj'n for t 1,...,n is a collection of non-negative numbers summing to 1, F; = Yxilyr,and Bpn) denotes the supremum of the indicated sum over all such collections, given p and n. =
The terms y;/F;-1 yt /1$-1 for f k 2 are finite since yl > 0. lf at most a number of finite the yt are positive the majorant side of (15.62)is certainly finite, so assume otherwise. Without loss of generality, by reinterpreting n if terms, we can also assume yt > 0 for every t. necessary as the number of nonzero 04?T-1) Ot-1) and applying 2.27 yields the result Bp n) Then, ytlYt-t and y; = ' /41) for a1l p e (0,1). Putting Bp supnfptn) < x completes the proof. w = =
'
=
,
=
Proof of 15.18 Put An Fn
: +
=
2 an d for E Z:=1X?,
=
>
0 and 8
0, set
>
n
Sln +
(: +An)
(1 6)(: +4u) +
=
+
2Xx%-1.V,
(15.63)
>1
2 + 25'f-1 Xt for t k 2, so that in the notation of 15.19, yt = 1', F,-1 (1+ 6).Yf 2 > 0 Then by the left-hand inequality of (15.55), with yl = (1+ 8)(e+X1) =
-
.
is
S
(1 6/18 +
+
?1
X21)P
+
(1
+
t
F(:lV 2pXJ't--ljx%-j.Y/. 6)#X >2
(15.64)
+
N2
we may allow it to approach 0. Taking expectations through, using the law of iterated expectations, and the facts that -1 i s decreasing in its argument, we obtain F(.&l 7/-1) 0 and that However, : is arbitrary in
(15.64),and
.?
=
E(%l
+
6:4.)# < E (1+ S E
(1+
DjPXIP
&fXILP
+
(1+
(:2,-1+ 6x4,-j)P-1V )pX >2 N
+
P XA(--j1F t (1+ 6)45/'-1
(15.65)
.
>2
2 an d yf Xlt for t 2 the right: + 4n, with yl = e:+ But if we put now l'n hand inequality of (15.55)yields (again,as the limiting case as E 1 0)
m
=
Xlp 1
=
,
Fl
+
#XA--lV< (j + Bg)E(AL). >2
(15.66)
Martingales
?
and since (1+ (1 +
6)8P-1(1
+
)P-1
(1+
f
245
this combines with
(15.65)to
give
B )F(A$) k E(Sln+ 8z4ay
2 2:-11(.E'1 l
lp
us-
#F(4j)j,
+
(15.67) .?
where the second inequality is by the concavity of the function
for p S 1.
yields
Rearrangement
2:
FIs I
6J'g21-P(1
(15.68) )/ 1jF(4j). which is the right-hand inequality in (15.54),where Cp is given by choosing 8 to minimize the expression on the majorant side of (15.68). In a similar manner, combining the right-hand inequality of (15.55)with L : + Sn2 w ith (15.65)and (15.67),and using concavity, yields n
S
+.Bp )(1+
-
=
lp
(1+ )(1+l,)FI 5'n1
k
(1+ &)E xll' +
zE
((1
Pxip
)
+
pTn5'l.%-llxj
+
,1
U'-
(1+
Dlpxn
(./-1 +
A,-1)P-1A-l
N2
N
Esn 2 + 6.4n )p
> which rearranges
2#-1(s1
lp
snI
#A'(z4j))
+
(15.69)
as lp
FIS 1
8#(21-P(1
n
+
6)41+#p
-
11-1F(Aj),
which is the left hand inequality of (15.54),with maximize the expression on the majorant side K .
o
1, Bp 0 identically in (15.55)and reproducing the known orthogonality property. Our final result is a so-called exponential inequality. bound fof martingale processes whose increments are accordipgly related directly to the bounding constants, For the case p
=
=
(15.70)
given by choosing cl
=
C1
=
to
1 for any
,
This gives a probability a.s. bounded, which is rather than to absolute
moments.
If (aYf,?;r)';'is a m.d. sequence with 15.20 of positive constants, and Sn Z)=lA, sequence 'heorem
IAI K Bt a.s.,
where
(#fl is a
=
#( IsnI >
:)
S
2exp(-:2/2(J=1#2f) j. u
(15.71)
This is due, in a slightly different form, to Azuma (1967),although the corresponding result for independent sequences is Hoeffding's inequality, (Hoeffding 1963). The chief interest of these results is the fact that the tail probabilities decline exponentially as 6: increases. To fix ideas, consider the case Bt B for This a 11t so that the probability bound becomes #( ISn I > e) f 2exp (-elllnBlt inequalitv is trivial when n is small. since of course P( ISn I > nBj = 0 bv con=
,
.
Theory of Stochastic Processes
246
struction. However, choosing e On 1?2 ) allows us to estimate the tail probabili-1/2 with quantity the associated Sn. The fact that these are becoming ties n exponential suggests an interesting connection with the central limit results to be studied in Chapter 24. =
(-#,,#,) satisfies
Proof of 15.20 By convexity, evel'y x e etM
for any a
>
<
(Bt +
ctfr+ (Bt
.x)d
-xle-l?t
(15.72)
lBt
0. Hence by the m.d. property,
ELemt I1,-1)
qlelt+ e-Bt)
f
S exptzlazsz,
j a.s.,
(15.73)
where the second inequality can be verified using the series expansion of the exponential function. Now employ a neat recursion of 10.10: Ee'nk
@n-I)
=
=
Ee-b-xn
I@n-1)
etxk-lFtetnj
fe osn
(.1
-lexp
(15.74)
@s-1) 2s2 n
)
as .
.
Generalizing this idea yields Een)
=
@,,-1)1,,,-2)... l11))
Eeq...Eefen,
S expfz1(#u2jF(F(...F(c%-l
s Combining
(15.75)with the
e
>
)
(@u-y...( @j))
.
generalized Markov inequality 9-11 gives
2E2.1#2, f exp ( -(y,E + yx1 j 0, which for the choice a = elX1=LBlt4 becomes Psn
for
2x:-1s2,
exp fac,l
Psn
> E)
> E)
<
(15.75)
zexpt-k?!l&1=jB1t) j.
The result follows on repeating the argument of and sumrning the two inequalities. *
(15.75)q15.76)in respect
(15.76) (15.77) of -Sn
A practical application of this sort of result is to team it with a truncation or uniform integrability argument, under which the probabilities of the bound B being j exceeded can also be suitably controll: ,
16 Mixingales
16. 1 Definition and Examples Martingale differences are sequences of a rather special kind. One-step-ahead unp redictability is not a feature we can always expect to encounter in observed time series. In this chapter we generalize to a concept of asymptotic unpredict-
ability.
16.1 Dellnition On a probability space L1,5,P), the sequ e of pairs (A,Tt)=-x, where (Tf) is an increasing sequence of c-subfields of 5 and th .Xt are integrable nv.s, is called an Lp-mixingale if, for p 2 1, there exist sequences of nonx, and negative constants (cfl*-x and ((,,,)7such that (,,, 0 as m -->
-->
IleAl 1,-,211, K c'l,,,
(16.1)
F(A1 T,+,2Il, < c,(,,,+1 117,-
(16.2)
hold for a11 t, and m k 0.
In
A martingale differepce is a mixingale having (,,, 0 for all m > 0. Indeed, tmixingale differences' might appear the more logical terminology, but for the fact that the counterpm of the martingale (i.e. the cumulation of a mixingale sequence) does not play any direct role in this theory. The present terminology, due to Donald McLeish who invented the concept, is standard. Many of the results of this chapter are basically due to McLeish, although his theorems are for the 2. case p mixingales form a very general class of stochastic Unlike martingales, processes; many of the processes for which limit theorems are known to hold can be characterized as mixingles, although supplementary conditions are generally needed. Note that mixingales re not adapted sequences, in general. Xt is not assumed to be Fsmeasurable, although if it is, (16.2)holds kivially for every m k 0. The mixingale property captures the idea that the sequence (Tsl contains progressively more information about Xt as s increases; in the remote past nothing is known according to (16.1),whreas in the remote future everything will eventually be known accordig to (16.2). The constants ct are scaling fatorj to make the choice of (,uscale-independent, and multiples of 11A11pwill often fulfil this role. As for mixing processes (see if (m O(m-%) for (? > (>. j14.1), we usually say that the sequence is of size However, the discussion following (14.6)also applies to this case. =
=
-(>
=
Theory of Stochastic Processes
248
16.2 Example Consider a linear process X
x jc-,77%i-./,
(16.3)
=
where ( I7,)+-Zis a fw-bounded mmingale difference sequence, with p k 1. Also 1et 5t ct&s, s S f). Then =
X
F(Al 5t-m4
=
77 jtb-j,
(16.4)
a.s.
j=m
Xt F(.Y,l 5t+m4
=
-
77e-jUt+js a.s.
(16.5)
j=m+L
w-bounded, the Minkowski inequality shows Assuming ( Us*-ooto be unformly that (16.1)and (16.2)are satisfied with ct sup,ll U,llp for every t, and (,,,= TJ l0j1 + Ie-j!). f-Y,,F,l is therefore a fz-mixingale if X; I0.j1+ l0-./1) 0 are absolutely summable. The as m x, and hence if the coefficients (%J=-x arises sided' process in which % 0 foj < 0 more commonly in the econometric modelling context. In this case Xt is gr-measurable and Xt Ext l5t+m4= 0 a.s., which may increase with t, and does not have to but we may set ct supysfll t&11p satisfy the definition. To prove Xt integrable, given be bounded in the limit to integrbility of the &,, requires the absolute summability of the coefficients, and in this sense, integrability is effectively sufficient for a linear process to be an f-l-mixingale a =
=r,,(
.-.-9
=m
fone-
--
=
-
=
We could say that mixingales are to mixing processes as martingale differences are to independent processes', in each case, a restriction on arbitrary dependence is replaced by a restriction on a simple type of dependence, predictability of the level of the process. Just as martingale differences need not be independent, so mixingales need not be mixing. However, application of 14-2 shows that a mixing zero-mean process is an adapted Ap-mixingale for some p k 1 with respect to the subfields 5t = c(A,Xf-t,...), provided it is bounded in the relevant norm. To be precise, themeandeviations of any fv-bounded sequence whichis a-mixing for r > 1, form an p-mixingale of size 1/r) for p satisfying of size application of 14.4 1 S p < r. If the process is also (-mixingof size tightens up the mixingale size. The mean deviations of a (-mixingfv-bounded 1/r) for 1 < p S r. The is an Lxmixingale of size sequence of size reader can supply suitable definitions of ct in each case. lt is interesting that the indicated mixingale size is lower (absolutely)than the mixing size, except only in the (-mixingsequence having finite sup-nonn (fv-boundedfor all r). Although these relative sizes could be an mefact of the inequalities wlzich can be proved, rather than the sharpest available, this is not an unreasonable result. If a sequence has so many outliers that it fails to possess higher-order moments, it would not be surprising to find that it can be predicted further into the future than a sequence with the same dependence structure but more restricted -:(1//7
-(?,
-
-(?,
-:(1
-(p
-
variations.
Mixingales
249
show the type of case arising in the sequel.
The next examples
16.3 Example An Ar-bounded, zero-mean adapted sequence is an Q-mixingaleof if either r > 2 and the sequence is a-mixing of size 2), or r k 2 size 1). u and it is (-mixingof size -r/(r-
-
-r/2(r
-
16.4 Example Consider for any j k 0 the adapted zero-mean fXtXt+.i c,,?+./,5twjl
sequence
-
,
and (A) is defined as in 16.3. By 14.1 this is a-mixing where ojt-vj Extxt-, (p-mixing) of the same size as Xt for finite j, and is fv/z-bounded, since =
Il.X)-)+./ rn 11
rII.X)+./ IIAII Ilr.
:<
by the Cauchy-schwal'tz inequality. Assuming r > 2 and applying 14.2, this is an in the a-mixing case. To get this result under g-mixing fal-mixingale of size 2), by 14.4, but such a sequence is also a-mixing also requires a size of 2) so there is no separate result for the (-m' of size case. I:a -1
-r/(r-
-r/(r-
'ng
Mixingales generalize naturally
from sequences to arrays.
is an L 16.5 Definitipn The integrable array ( (Xnt, @nfJ'7=-x)';=1 J for p k 1, there exists an anuy of non-negative constants (c,:,) negative sequence ((,,,)7such that (,,, 0 as m ..-A x, and -x,
-mixingale
if, and a non-
--+
IlFtAkrl@n,,-,,,)lIp < c,,,(,?,
(16.6)
IlAk A'(Akrl5n,t+m4 11,f cn,(,u+1
(16.7)
-
hold for
all f, n, and m
0.
(:1
The other details of the definition are as in 16.1. A1l the relevant results for mixingales can be proved for either the sequence or the array case, and the proofs generally differ by no more than the inclusion or exclusion of the extra subscript. Unless the changes are more fundamental than this, we generally discuss the sequence case, and leave the details of the array case to the reader. One word of caution. This is a low-level property adapted to the easy proof of convergence theorems, but it is not a useful constnlct at the level of time-series modelling. Although examples such as 16.4 can be exhibited, the mixingale property is not generally preserved under transfonnations, in the manner of 14.1 for example. Mixingales have to little structure to permit results of that sort. The mixingale concept is mainly useful in conjunction with either mixing assumptions, or approximation results of the kind to be studied in Chapter 17. There we will find that the mxingale property holds for processes for which quite general results on transformations are available.
16.2 Telescoping Sum Representations Mixingale theory is useful mainly because of an ingenious approximation method. involving a remainder which A sum of mixingales is nearly' a martingaleprocess,
Theory of Stochastic Processes
250
canbeneglectedasymptoticallyundervriousassumptionslimiungthedependence. For the sake of brevity, 1et Es stand for Ext ITx). Then note the simple identity, for any integrable random variable Xt and any m 1, Xt
X (f
=
t+kxt
Etn-bxtj
-
+
Et-m-lxt
Xt E ?+p1X).
+
W-m
Verify that each tenn on the right-hand side of signs, except for Xt. For any k, the sequence
(16.8)
-
(16.8)appears twice with
opposite
fEtwkxt F,+1-1X,, 5t+k17=1 -
is a mmingale difference, since Et-vk-kEt.vkxt Et+k-3Xt) 0 by the LIE. When l.&,S) is a mixingale, the remainder terms can be made negligible by taking m large enough. Observe that fEt+nXt, S+,,,I';=-xis a mmingale, and since supmfl Et.vnv'tkS FI.&l< x by 10.14, it converges a.s. both as m c.a and as m 11Fr-,,j.X)IIJ, 0 and -x, by 15.7 and 15.9, respectively. In view of the fact that )1aYfEt+mxtIIp 0, the respective a.s. limits must be 0 and Xt, and hence we are able to assert that =
-
--y
--y
.-+
--
-
X
Xt
(F,+kX,- Et+k-kxtlh,a.s.
=
(16.9)
P=-x
Letting Sn
:=1Xs we similarly have the decomposition
=
m
Sn
=
l
X k=-m
Fn
+
n
XEt-m-jxt
+
>1
Xt X r=1
-
(16.10)
Et+mxt)
where
lu
=
';--Iekxt
F,+,-1A),
-
( 16. 11)
,.-1
and the processes ( Fns a+k) are martingales for each k. By taking m large enough, for fixed n, the remainders can again be made as small as desired. The avantage of this approach is that martingale properties can be exploited in studying the convergence characteristics of sequences of the type Sn. Results of this type are elaborated in j16.3 and j16.4. If the sequence (A1 is stational'y, the constants lcfl can be set to 1 with no loss of generality. ln this case, a modified form of telescoping sum actually yields a representation of a partial sum of mixingales as a single martingale process, plus a remainder whose behaviour can be suitably controlled by limiting the dependence. 16.6 Theorem (afterHall and Heyde 1980: th. 5.4) Let (Xt,S) be a stationary Al-mixingale of size There exists the decomposition -1.
Xt
where
FlZjI <
x
and
=
B
( W$,T,) is
+
Zt
-
(16.12)
Zt+I,
a stationary m.d. sequence.
(:1
Mixingales
Sn
( Fn,gnl
251
(16.13)
Fn + Z1 Zn+1
=
-
is a mmingale.
Proof Start with the identify Xt
F,,s
=
+
Zmt -
zr,,,f+1,
(16.14)
where, for m k 1, m
X
lp,,,f =
Ext..s
Et-3Xt+s4+
-
EtXt+m+L
+A-m-1
-
(16. 15)
Et-tXt-m-3
J=-m
m
Zmt
=
%-Et-$Xt+s -A-,-1 +Et-3Xt-s-t4.
(16.16)
.=o As in (16.8),every term appears twice with different sign in (16.1 Xt. Consider the limiting cases of these random variables as m designated J'Pfand Zt respectively. By stationarity, E Iff-lA+sl
=
), except -->
x,
for to be
FIEt-s-txt I
and Fl Xt-s-j
Et-jxt-s-k
-
1
E IXt Fm-Y,l ;
=
-
hence, applying the triangle inequality, X
to
E Iz,l
S TE M
IEt-s-txt I +
77E IXt r.m
-
Etnxt
I
*
< 2X(,<
(16.17)
=.
M
Writing Jpr
=
Xt Zt + Zf+1, note that -
E IJFfI S
FI-Y,I+ 2F Iz,I <
(16.18)
=,
and it remains to show that J'P'fis a m.d. sequence. Applying 10.26(i) to (16.15), Et-jb'mt
(16.19)
Et-3Xt+m+3a.s.,
=
and stationarity and (16.1) imply that
FIEt-$Xt+m+3l FIEt-m-zxt I --A 0 0 also. Anticipating a result so that FI&-1B%l
(16.20)
=
x, from the theory of as m k e IN1 stochastic convergence (18.6),this means that every subsequence tr?h, e-lW,,,w),rl subsequence IN such 0 a.s. that I contains a further j e ) fmkq), x. for such subsequence, it is possible Since Jpf Wmkqj,t to every as j conclude that A'(WfITf-1) 0 a.s. This completes the proof. w .--h
-->
-->
.-+
-->
=
Theory of Stochastic Processes
252
The technical argument in the final paragraph of this proof can be better appreciated after studying Chapter 18. It is neither possible nor necessary in this 0 a.s. approach to assert that F(G,,vl @r-l) Note how taking conditional expectations of (16.12) yields -->
F(ml 72-1) = Zt 4+1 a.s.
(16.21)
-
It follows that <, is almost surely equal to the centred r.v. Xt Ext -
IF,-l).
16.7 Example Consider the linear process from 16.2, with ( Utj a stationary integrable sequence. Then Xt is stationary, and X
FI !71I 77 I%I <
Fl.Yl l f If the coefficients 00
*.
j=-oo
satisfy a stronger summability
X
X
condition, i.e.
X
< 77 X(( %( 10-/1) 77-10-1 X?nlo-,?,l
+
+
=
za=l
rn=1 j=m
then Xt is an Al-mixingale of size decomposition of (16.12)with
(16.22)
x,
?l=1
of terms we obtain the
By a rearrangement
-1.
X
B,,
77uj &,
=
(16.23)
jcz-x
and oo
Zt
=
o
x
X j=m X 0y Ut-t,t X 0-,. Utwm-q
zrl=l
where
E'IZ,I <
cxa
(16.24)
-
,
j=m
by (16.22).u
16.3 Maximal Inequalities As with martingales, maximal inequalities are central to applications of the mixingale concept in limit theory. The basic idea of these results is to extend Doob's inequality (15.15)by exploiting the representation as a telescoping sum of martingale differences. MacLeish's idea is to 1et m go to x in (16.10),and
accordingly write
X
Sn
(16.25)
Fnz, a.s.
=
#=-x
16.8 Lemma Suppose (xaly has the representation in (16.25).Let be a = collection real with of non-negative if l'ak 0 numbers, summable ak 0 a.s., and ak > 0 otherwise. For any p > 1,
tJklX-x
=
E max Lsjs n
IsjI# u
# -
#
1
x
77a k
b=-x
p-1
>-cf-1Fl
ak>o
ynzl#.
(16.26)
Mixingales
taloJx and positive real sequence
Proof For a real sequence X7=-xlkand note that OQ
X xk
p
x =
X
A''
P=-x
253
p
xklaaklp
=
x
X tz,'-#I.xkI#,
K A'''-'
k=-x
1et K llkloo-x,
(16.27)
#=-x
where the weights aklK sum to unity, and the inequality follows by the convexity of the power transformation (Jensen's inequality). Clearly, (16.27)remains true if the terms corresponding to zero xk are omitted from the sums, and for these cases set ak 0 without loss of generality. Put xk Fn, take the max over 1 S j < n, and then take expectations, to give =
=
E max L<j
To get
Isjbl'
j
<
x
77ak
p- I
tt.---
j
ak y-')
max j yg j
p
.gy
Doob's inequality on the right-hand
(16.26),apply
.
lsjsn
akg
side. w
This lemma yields the key step in the proof of the next theorem, a maximal inequality for Q-mixingales. This may not appear a very appealing result at first sight, but of course the interesting applications arise by judicious choice of the
sequence
lJkl
.
16.9 Theorem = E7.1-Y,, and
be an fw-mixingale, 1et Sn (Macleish 1975a: th. 1.6) Let (Xf,J;2l=-x let ltul7 be any summable sequence of positive reals. Then co
E
maX
ls/fn
f
f 8
co
($2+
F7
Jk
1
(2j)J
+
t-.=l
uk.1j)
(2gu-,1 2F-4 -
=
SJ 1<)
K4
(16.29)
.
put a-k ttzkl=-x,
=
ak for k
>
0. Then,
oo
Cr
E max
N c2j >1
k=l
Proof To get a doubly intinite sequence 2, applying 16.8 for the case p
n
X ak X tz-k1F(L2J .
(16.30)
V-x Since the termsmaking up lk are mmingale differences and pairwise uncorrelated, we have F(Fn2k) Ek-vkxtEt-vk-kxt) whichit follows that NOW,
-
Also let
%)EEt+kXt -Et+k-jXt)l.
,.-1
EEk-tk-vkxtk-vk-titqj
=
E Et-vkxt
=
l=-x
=
E(Elt+kXt Et+k-lxtL$l =
ztk x,- Et-vkxpand =
-
E(Elt+k-jXt) by the LIE, from
Elt+k-;Xt).
(16.32)
it is similarly easy to verify that
EEt+kXt Et+k-LXt41 F(z,,1-1 -
(16.31)
=
=
Ezlk-j 1,
=
z,1)2
-z2,J.
(16.33)
Theory of Stochastic Processes
254
Now apply Abel's partial summation <x)
X
a-x
n
c-k1A'(F2aJX
X a-kE(Et+kXt
=
>1
(2.25),to
get
Etwk-jXtll
-
k=-x
#j
*
<>3
=X X ,-1 t.-o
6l-kE(Elt-kXt Elt-k-jxt)
+
-
n
1F(z2,,:.j z2f:) Xt7-: k=l -
<x)
)q >1
=
formula
<x)
&0-1#(F2X t
t
)
+
TECEI
X)(J-k1
t-k
-
t
tz-k-lj)
k=1 X
J1 -3Ez1 10
+
)
+
y7F(z2J(u1 j t
-
u-11)
al
(16.g4)
where the second equality follows by substituting (16.32)for the cases k f 0, and (16.33) for the cases k > 0. (16.29)now follows, noting from (16.1) that EZlk) 2 X, ) < c2(2 S c2(2 E(Et-k , t k+1 x t k and from (16.2)that .
Pptting x
K
x
-1((2 (2) 1 Tak t'af
8
=
() +
tztl
u-jy.1jl 2y-q (2:(t7-:1
+
(16.35)
-
,
k=1
this result poses the question, whether there exists a summable sequence ftzkl such that K < x. There is no loss of generality in letting the sequence ((:); be monotone. If (,u 0 for m < x, then (v,+y0 for a1lj > 0, and in this case one 2 0,...,- + 1, and K reduces to m + 1)(%2 + (1). 1, k A jtem may choose ak consider where and the case natively, (k> 0 for evel'y k. If we put at ((), then define the recursion =
=
=
=
=
(z g((1+4Jk-1) z ak ztu-j
.(j
a
=
ak is real and positive if this is true of -j
ak
=
8
x
Xtzk
=
> -2
(, so that
(1 ak 2
(1)f
(16.37)
2J(),
we have
x
(2j)+2Xtu
< 16
k=1
1=0
ln this case, for k
and the relation
-2
-c:-1
J-o1((+
(16.36)
,
tu-1
-1
l 2 is satisfied for each k. Since aj- ((c +
K
j/2
x
2
Xak
.
(16.38)
k=o
0 we find -1
=
ak
-1
-z-1)Jk
-1
-2
f ak
-2
-J1-l
(16.39)
Mixingales m
m
X (-,2G (0-2+ X k=o
-
,-z-21
=1
(16.38),we
Substituting into
ak-l
255
)
am-l.
=
(16.40)
get
( j
X*X*(
K f 16
-
2
-
' /2 2
(16.41)
This result links the maximal inequality directly with the issue of the summability of the mixingale coefficients. In particular, we have the following
corollary. 16.10 Corollary
Let
(.X),S)be
an Q-mixingale of size
E maxu/J
K
Then
#Xd,
(16.42)
>1
1f.jf n
where K
-.
< x. -1/2-
-2
Proof If (1 Ok > 0, as the theorem imposes, then Z'l=1(k ) for1(-k2)-1/2 tp(--1-8 ) an d hence is summable Over m. ( and b 2 27 Om 2+2: q ) y The theorem follows by (16.41).w =
.
=
.
=
=
However, it should be noted that the condition m
x
.j/p,
X 77(-,2 is weaker than Ok -1/21/2+ for E > 0, so that k
(k
=
(z
--
X( -2 X(k..v2)(log =
(16.43)
< c.o
). Consider tjj e x
case for evel'y g > ()
k.- 2)2+2e<
(k qk+ g)-1/2(jog =
k + 2)-1-:
ejxen
.
(??;+2)2(log
-+2)2+26
(16.44)
,
and (16.43)follows by 2.31. One may therefore prefer to define the notion of in tenns of the summability condition (16.43),rather than by orders isize of magnitude in m. However, in apractical context assigning an order of magnitud to (,,,is a convenient way to bound the dependence, and we shall find in the sequel that these summability arguments are greatly simplified when the order-ofmagnitude calculus can be routinely applied. -'
=
Theorem 16.9 has noobvious generalizationgomtheo-mixingalecasetogeneral of the Lp for p > 1, as in 15.15, because (16.31)hinges on the uncorrelatedness exist in under consideration, second not the But because moments may terms. cases < p < 2 would be valuable. This is attainable by a result for 1 comparable a slightly different approach, although at the cost of raising the mixingale size in other words, the mixingale numbers will need to be summable. from to ..ul
-1;
16.11 Theorem Let tXr,?hl'l..,obe an fv-mixingale, 1
<
2, of size
-1,
and 1et Sn
Theory of Stochastic Processes
256 E max 1sjn
p
lsjI s
tx
F
P
4p c,
P
N
y7(z 77d,
j
t-.al
(16.45)
>1
where Cp is a positive constant.
Proof Let F,,kbe defined as in (16.11), and apply Burkholder's inequality and then Love's cr inequality with r = pI2 e (, 1) to obtain
FIFazlr f CpE
n
Et-vkxt &+k-1X,)2
X >1
(15.18)
p/2
-
n
S
CAXFI(Et+kXt
&+1-1.41F.
-
(16.46)
>1
Now we have the mixingale inequalities, lIF,+zX,
f IIF,+1AIl,+ < 2c,(z .&,+z--1.Y,I1?? lI.E',+z-1-'tlI,
-
fork
0 and
<
11F,+kA -
for k
(16.47)
z'kll, A',+,-1AlI,I4z,,z-1 S 2c,(1 Iz,k.II, 11p+ < IIz,,k-1 =
-
0, where Ztk is defined above
>
(16.48)
(16.33).Hence, n
e'Irnzl' s
Ec
2,cp(f
(16.49)
>1
(put (0 1), and substitution in (16.26),with (t7z)7'a positive sequence and = ak, gives =
E
mx
lijin
p
#
IS)Ip < y+1c?,
p-j
cx>
Efz t..o k
- 1
Both ak and ak1-#(( can be summa b1efor p conclusion follows. .
>
-ak
n
x
y'-qtzj-ptj N c;.
(16.50)
,...1
t-azo
1 only in the case
(z
=
Oa,
and the
A case of special importance is the linear process of 16.2. Here we can specialize 16.11 as follows:
16.12 Corollary (i) if Xt
Z7=-oo%Ut-),then
=
E mx Lsjsn
(ii) if Xt
For 1 < p S 2,
=
IsjI# S Cp Z7
p
x
IOeI +
j
77( I01( + al
l0-zI)
p
n sup F# (7,1#., s
then
zejh-p
E max I5'./i lsjs n
P
p
s c,
P
p
!
o
p
n
I0zl 77sup FI &,IP. >'') k.'.'zo >1
st
Mixingales Proof In this case, Et-kxt
Et-k-kxt
-
and a-k
constant sequence
l57
kut-k.
=
tJkl7
Letting
be any non-negative
ak,
=
7-7JI-PFI1u1,
I0:I#7'7cf, CvTakz-p
<
ap*
JS/
supsxtll rxllpin case the results. .
wherect supxll&x11pin case (i), and c, = I0kIand substituting in (16.26)yields =
(16.51)
>1
=
ak (ii). choosing
T) Recall that the mixingale coefficients in this case are (,, I0./1 + I0-./1), so linearity yields a dramatic relaxation of the conditions for the inequalities to be satisfied. Absolute summability of the % is sufficient. This corresponds simply to (,,, 0. A mixingale size of zero suffices. Moreover, there is no 2 yields a resuli separate result for faa-bounded linear processes. Putting # that is correspondingly superior in terms of mixingale size restrictions to 16.11. =m(
=
--)
=
16.4 Uniform Square-integrability One of the most important of McLeish's mixingale theorems is a further consequence of 16.9. lt is not a maximal inequality, but belongs to the same family of results and has a related application. The question at issue is the uniform integrability of the sequence of squared partial sums.
16.13 Theorem (fromMacLeish 1975b: lemma6.5,' 1977: lemma3.5) Let taYsTfl 1m, and :,2 7=!clwhere ct is detined in be an Q-mixingaleof size 1 Sn l n l l lf the sequence (Xtlct ) ,=1 is un iformly integrable, then so is the (16.1)M16.2). =
-z,
=
.
oo
sequence(
maxlxjspl/vz,,
jooawl .
Proof A preliminary step is to decompose Xt into three components. Choose posiB 1( jx,Issc,), an d then tive numbers B and m (to be specified below), 1et 1 t =
define
Ut Ft Zt
=
(16.52)
Xt Etwmxt+ Et-mxt
=
-
B
Et-vmxt1 t
=
-
E t-m
a
lf
(1653)
t t
.
B Et-mxt 1 1 t ) Et-mxt 1 1#) t -
-
(16.54)
-
,
such that Xt Ut + Ff+ Zt. This decomposition allows us to exploit the fotlowing collection of properties. (To verify these, use various results from Chgpter 10 on conditional expectations, and consider yhe cases k 2 m and k < m separately.) =
First,
E 2Et-kut Eut
-
Et-vkut)a
for k k 0, where kv m
=
1 E Et-wmjvk S
Ext
-
maxfkml.
=
EE 2 t-k F,
=
=
EEl
t-ksm
2 2
ctLkvm,
(jy
a a Et-vkwmlxa f c/((t,v??;)+1 ,
.
55;
(jy 56; .
Second, El X/ 1#) )# , 1* , , t-m -
,
(16.57)
Theory of Stochtic
258 E
Et+kYt)2
-
Processes
F(F2/+?,, X 1*f -F2 faxkz.?n )# t 1W) , t
=
(16.58)
,
=mintk,rrll.
Theterms areboth zero if kkm and are otherwise bounded Bct. Third,
whereko.1P,m by Exlt ) ;
F#
t-k
zt
EEl
=
E(z t F,+zz,)2 -
=
1 1()
:.)m4
t-
EEl
t+
-
,aftl
1t)
-
-
-
e'lt-ttvj
lt
-
A+(k=)m(1
)), 11/)),
-
where the tenns are zero for k 2 m and bounded by f(#l(1 F(AQf(1 11 that ))/U= ElxtlctjlLf jxycgjlsl) 0 as B the assumption of uniform integrabillty.
x
-
(16.60)
1Pf))
-
-->
-.-.:
(16.59) otherwise. Note uniformly in /, by
The inequality
(x'
2
x,)
>1
s
3
s,) 2 +
z,) 2j
g(x? (x? (x' ,-,) 2
+
>1
>1
(16.61)
>1
for 1 S j S n follows from substituting Xt = Ut + F, + Zt, multiplying applying the Cauchy-schwartz inequality. For brevity. write xj
S)?/v2 n,
=
uj = (Sr=1 '
A'./=
kn
=
Ullgnl,
?= 1
zj = (Yfn Then
out, and
YttllMnl,
lZf)2/v2. s
(16.61)is
equivalent to xj f 3(w+yj +p), for each j = 1,...,n. Also 1et similarly; then clearly, maxlssaay, and define n, y'k, and
.
f 3n
+
yh +
k).
(16.62)
For any r.v. Xk 0 and constant M > 0, introduce the notation &>(A') F(1(x>>).X), x. 0 as M As a so that the object of the proof is to show that supn&ion) of 9.29, and consequence (16.62) =
-.-/
'k
f 3&v/3(
s 6eqn)
+
yh +
-->
k)
+ &wK@-n)
+En)).
(16.63)
We now show that for any er> 0, each of the expectations on the right-hand side of (16.63) can be bounded by e by choosing M laree enough. First consider &;n); =L1/2giveia (16.55)and (16.56),and assuming (,a= Om ), we can app j y j: nto this k-1-9 -1-8 = for k > m. Applying (16.29) for k S m and ak case, setting ak m ' with >1 U t substituted for S) in that expression produces .
,
E(
n
)f
8
(r?z+1)/1-1-8
= O(m ),
+
X k-1-0
&-+1
=
(2m1++2 m X (2,/ 1=>a+1
(16.64)
Mixingales
259
where the order of magnitude in m follows from 2.27(iii). Evidently we can choose m large enough that ELn) < :. Henceforth, let m be fixed at this value.
but in view of (16.59)and (16.60)we A similar argument is applied to E(L4, 2 otherwise. Write, fonnally, EEt-kzt and < choose 1, 0,.. 0 k ak ak may2 y.m, ct(2:, Etnztjl 2 an d Ezt where < t7?(k =
=
=
-
maxlu,u,,F((X,/c,)21( Ixgc,I>s)),
(2
k
<
m,
kk
m,
(16.65)
=
and then application of (16.29)leads to
ELn)f
16(-
+
1) max Elxtlctll
1( 1xgoI>s)).
1< t < n
(16.66)
x, so let B be fixed at a value large enough that This term goes to zero as B Eqnj < E. where For the remaining term, notice that l'f Z'l'=-,u+1(x -->
=
Et-vkxtf B, Et+k-3y , jl ,.
k,1 =
(j6 6y)
-
.
For each k, (efl, @f+:)is a m.d. sequence. If 16.8 is applied for the case p = 4 and ak 1 for IFcl < m, 0 otherwise, we obtain (not forgetting that for yj > 0, =
2 (maxyyy) =
..2 Eyn,
maxytyy?
-
where Fn1
=
j
1
wf
v
J)
max l sj s n
'n
77r, 1
=
F(F4
<
given l',,z
k) + n- j
4F(F3.) n
,
?
1)
w v (2-+ V =
yjup ,
4
1 4 'n
r=
Z:=1(a.Now,
EYnk 4)
4
-'
Fa-j,1+
77E
n
+ 4F(F n
nk), ),4
kzz-m
(,,k,we
6F(F2.j
+
m
(16.68)
have the recursion
2up ,
-1
,
zt3,u)+
F(j4uJ.
(16.69)
(fkare bounded
absolutely by lBct; hence consider the terms on the right-hand side of (16.69).The second one vanishes, by the m.d. property. For the third one, we have The
2 2 f f'(l%-l,t4n2 and for
2
E-n-,lBcn)
2
f
4 2
(2f) Vn-lcn,
2
the fourth one, note that by the Cauchy-schwartz
3 j k (2#)4vn FI L-l,klnk
(16.70) inequality,
-1c3u.
Making these substitutions recursivelyyields Elvkl f n
into
(16.69) and
(16.71) solving
the implied inequality
n
&
N
>1
>2
>1
+ 4Xv,-1c) + (2#)4 6Xvl-1U Xd
4 f 11(2#) 4vn.
(16.72)
Theory of Stochastic Processes
260 Plugging this bound into .Y.k 0 and a > 0, yields
(16.68),and tinally 6
an ) f &Ml6 (.y
.2
uE- (.yn ) f
applying the inequality a &u(m 4642-
(s j
+
1)411(2s)4
s,
< EX2) for
(16.73)
.
By choice of M, this quantity can be made smaller than e. < 18E for large enough M, Thus, according to (16.63)we have shown that &AQ) or, equivalently,
b-)
-->
0 as M
x.
By assumption, the foregoing argument applies uniformly
complete. w
(16.74)
--)
in n, so the proof is
The array version of this result, which is effectively identical, is quoted for the record. and let Sn be an fu-mixingale array of size 16.14Corollary Let (X ,,@nrJ 2 X1=jXnt and vn by if 2 E'Jzzlcnt, is given where cnt (16.6)V16.7), fAkfyc2n, j is is uniformly integrable. uniformlyintegrable, (maxlsyunial/vnz1-=a.! =
-,
'1
=
Proof As for 16.13, after inserting the subscript n as required.
.
.
17 Near-Epoch Dependence
17.1 Definitions and Examples As noted in j14.3, the mixing concept has a serious drawbackfrom the viewpoint of applications in time-series modelling, in that a function of a mixing sequence (even an independent sequence) that depends on an intinite number of lags and/or leads of the sequence is not generally mixing. Let X
=
#f(...,16-l,1$,'P)+1,...),
(17.1)
where Js is a vector of mixing processes. The idea to be developed in tllis chapter is that although Xt may not be mixing, if it depends almost entirely on the epoch' of f F,) it will often have properties permitting the application of limit theorems, of which the mixingale property is the most important. This idea goes back to lbragimov (1962),and had been formalized in different ways by Billingsley (1968),McLeish (1975a),Bierens (1983),Gallant and White (1988), Andrews (1988),and Ptscher and Prucha (1991a),among others. The following definitions encompass and extend most existing ones. Consider first a definition for sequences. tnear
17.1 Dennition For a stochastic sequence l Ffl+-Z,possibly vector-valued, on a is an probability space (f1,@,#),let lj+-: c(J$-,j,...,J$+,,,), such that ( 8??f+-'''''');=m lf, for of integrable 0, of > c-tields. increasing sequence r.v.s a sequence p =
lA
lt7
satisfies
IIX, -
S vp,, F(.X)I ?7-'''''')II,,
(17.2)
' where vm -- 0, and flrl+-2is a sequence of positive constants, Xt will be near-epoch dependent in fv-/ztp?'?zl(JV-NED) on (FfJ+-7.I:l
be said to
Many results in this literature are proved for the case p 2 (Gallant and White, 1988,for example) and the term near-epoch dependence, withoutqualitkation, may be used in tlzis case. As for mixingales, thre is an extension to the array case. =
A'
possibly vector-valued, on 17.2 Definition For a stochastic array f( Fnfl17-x)';=1, a probability space (f1,F,#), let snt-mt c(Fn,,-,,,,...,F,,,r+,,,).lf an integrable -,,,
=
satisfies arrayt t.krll---xl7-l, Ilxn'F(-Yn,lT,$+7-,,,)11,K dntvm, -
0, and fV-NED bn ( Fnr) u
where
Mm
-->
.
fdritt
(17.3)
is an array of positive constants, it is said to be
Theory of Stochastic Processes
262
We discuss the sequence case below with the extensions to the array case being easily supplied when needed. The size terminology wlch has been defined foi mixing processes and mixingales is also applicable here. We will say that the O(m-% for (p > (pf). when Mm sequence or array is LP-NED of size According to the Minkowski and conditional modulus inequalities, -(p0
=
117,e-'L117'''''')11,S 11*-,wll,+ Il.E'(-'t w! /7:)11, <2IIX,- 81,,11,, -
-
-
(17.4)
wherew EXt). The role of the sequence (4) in (17.2)is usually to account g,,ll, is uniformly bounded, forthe possibility of tending moments, and when 11.1, should equal finite dt constant for all f. However, a expect to set to a we drawbackwith the definition is that (l can always be chosen in such a way that =
-
11-Y,Ext -
inf t
IF,'+-:)11,
dt
=
0,
for every m, so that the near-epoch dependence property can break down in the limitwithout violating (17.2).lndeed, (17.2)might not hold except with such a choiceof constants. In many applications this would represent an undesirable weakepingof the condition, which can be avoided by imposing the requirementdrs 2jlm- gll??, or for the array case, dnt S 2II-L,- p.rllp. Under tls restriction we can set v,a < 1 with no loss of generality.
Near-epoch dependence is not an alternative to a mixing assumption', it is a propertyof the mapping from (FfJto (,&J,not of the random variables themselves. The concept acquires importance when (Ffl is a mixing process, because then (m) inherits certain useful characteristics. Note that f;txfl 1j+-:) is a finite-lag, function of amixing process and hence is also mixing, by 14.1. Aorl/s-measurable mixing in the sense of Near-epoch dependence implies that (.') is a pproximatedby weshowbelow, amixingprocess. Andas anear-epochbeingwell dependentfunction of a mixing process, subject to suitable restrictions on the moments,can be a mixingale, so that the various inequalities of j16.2 can be exploitedin this case. From the point of view of applications, near-epoch dependence captures nicely the characteristics of a stable dynamic econometric model in which a dependent variableXt depends mainly on the recent histories of a collection of explanatory variablesor shock processes Fs which might be assumed to be mixing. The symmetricdependence on past and future embodied in the definition of a G-NED functionhas no obvious relevance to this case, but it is at worst a harmless generalization. ln fact, such cases do arise in various practical contexts, such as the application of two-sided seasonal adjustment procedures, or similar smoothing filters; since most published seasonally adjusted time series are the output of a two-sided tilter, none of these variables is strictly measurable without reference to future events. tapproximately'
17.3 Example Let (Jl+-7 be a zero-meqn, As-bounded scalar sequence. and define
Near Sp/ch Dependence
xt
263
X ejvt-j.
=
(17.5)
jc-x
Then, by the Minkowski inequality,
Il, 77 111,Ext I8/',+-''',?,) j=m+3
jvt-j
=
-
e,+-,?,'''Ff-./l +
-
e-jvt-v.ie7,,,':'14+./)) P -
f dtvm,
(17.6)
Jillp, all /. Clearly, v,,z 0 if where v,u 17,.+1(I%I + I0-jI), and dt 2sup,I1 if I0./1+ l 1= the sequence (%) is absolutely summable, and v,,i is of size /4-1-:) for (? > (p. ln the one-sided case with oy 0 foj < 0, we may put #f supysfll7,11p, which may be an increasing function of t; compare 16.2. In -->
=
=
-(p0
-j
=
=
The secondexample, suggestedby GallantandWhite (1988),illustrates hownearepoch dependence generalizes to a wide class of 1ag functions subject to a dynamic stability condition, analogous to the summability condition in the linear example.
17.4 Example Let t %'t)be a Zp-bounded stochastic sequence for p sequence (m) be generated by the nonlinear difference equation Xt
hvt, Xf-1),
=
2 and 1et a
(17.7)
where ((.,.) l is a sequence of differentiable functions satisfying
:/)(r,X sup a.x FA
b
<
1.
<
(17.8)
As a function of h is called a contraction mapping. Abstracting from the of problem, write vl as the dummy first argument of ft. By the stochastic aspect repeated substitution, we have .x,
h
=
=
J,(v,,,-1(v,-1.,-2(v,-2,...))) (17.9)
',(vf,v,-1,vf-2,...),
and, by the chain rule for differentiation of composite functions,
gt
h-j
' -j
f b
vt-j
(17. 10)
.
vt-j
Detine a Fj-m/f-measurable approximation 1ag exceeding m by zeros:
to gt by replacing the arguments with
:,(14,...,v,-r,,,0,0,...). p';'(v,,...,v,-,,,)
(17.11)
=
By a Taylor expansion about 0 with respect to vt-j for j
8t # t -
cw
,:1
m,
+
=
l=m.z.l
>
vt-g.
vt-j,
(17.12)
Theory of Stochastic Processes
264
denotes evaluation of the derivatives at points in the intervals (0,vr-j1. Now define the stochastic sequence (X,) by evaluating gt at (1$,J6-1,...). Note
where
*
that IIX, -
F(X,I1',+-:)112f
#'J'(7,,...,F'-,,,)II2
IIX, -
(17.13)
by 10.12. The Minkowski inequality, (17.12),and thep (17.10) further imply that X
lIA7 -
#:'IIa 77 Gt-.ivt-j =
.j=-+1
l
<jcm- jlGt-jvt-j j)2 X
b.i- llF/-jvt-./llc
<jzm1
s
1-#
bm sup IlF,-./vf-./lia,
(17.14)
j>m *
with where Gt-j is the random variable detined by evaluating Llvt-pgtj and Ft-j bears the corresponding respect to the random point vt-j,vt-j-L with constants relationship with lvt-pft-j. Xt is therefore fU-NED of size <: supssf lIF,F,Ia, Hlder's i f particular, inequality exists. In this nonn tf exist for r > 1, and (IF,IIar/r-1) allows us to make this derivation whenever 11 7,lI2r 14112 < x and Ft is a.s. bounded. n and also if 11 ,...),
-=,
17.2 Near-Epoch Dependence and Mixingales The usefulness of the near-epoch depencence concept is due largely to the next
theorem.
17.5 Theorem Let (X,)*-x be an fv-bounded zero-mean sequence, for r > 1. lf Xt is AP-NED of size (i) faet ( Ff) be a-mixing of size on ( F1) for 1 S p fXl,T-Jx) with size fv-mixingale is of constants dtl an f (>, < r aslp 1/r)) with constants ct t< maxl 11Xf ) llr,#f of size -J. If Xt is fV-NED of size (ii) Let ( $) be (hmixing, on ( Fr) for 1 S p f r with constants (J, (X,,F-'x) is an fw-mixingale of size -.b
-a.
-min
,
-
.
-.b
.-lnint//,
J(1
-
1/r)), with constants ct <( maxf llXfIlr,4).
F,J) where %,' c(Fx,...,F). Also, for m K Proof For brevity we write Est.) = F(. l = 1, let k (-/2), the largest integer not exceeding ml2. By the Minkowski =
inequality,
Ile'-f ljp< oo-mxt
and we bound
jlJ2x-''t.Yf -
Fj+-),V) Ilp+ IIF-'x-mEtkkxtj11,,
each of the right-hand-side
jlE-t
..-'''(.4 -
Etkkxt)
K tL'-t 11p
c: '''
terms. First,
)Xt
-
P)11/P Frf+-l.& I
(17. 15)
Near Fp/c
DepeWence
265
II-Y, VtkkAllp -
< dtvk,
(17.16)
using the conditional Jensen inequality and law of iterated expectations. Second, E7PX function of F,-1,...,Ff+s and hence mixing of -.k t is a tinite-lag measurable the same size as (FfJ for finite k. Hence for part (i), we have, from 14.2, 6al/P-l/rjjxf r < 1I, 6al/P-1/rIIA:+-lA-,II
11F-'--'''(F71k-Y,) <
combining(17.16)
(17.17)into (17.15)
and
r. 11
(17.17)
yields
f maxt II-Y,IIr,4l(,,,, llF-'ol'''XIlp
where(,u
=
(17.18)
+ vk. Also applying 10.28 gives 6a11/#-1/r ,
< ldts'm < 2 lI-, Fl+-'''''W,IIp 111,F-lx+'''-Yfllp -
-
K 2(,,,.
(17. 19)
atlp 1/r) ), part (i) of the theorem holds with ct <( since (,,,is of size T he proof of pal't (ii) is identical except that in place of (17.17) maxflI#,1lr,l. we must substitute, by 14.4, E-tqn-mEftkkx < 2911 A'j+-lkxflj< 2911-l/rjjx, jj . 11 (17.20) r I1p -minth,
-
-l''rjj
,.
Let us also state the following corollary for future reference. be an fv-bounded zero-mean array, r > 1. 17.6 Corollary Let f fxnrl+f for 1 f p < r with constants (t&,)on an an-ay (i) If Xnt is AP-NED of size f Fnrl which is a-mixing of size -J, then tXpv,Fj,-xl is an fw-mixingale of alqlp 1/r) J, with respect to constants cnt (< maxt IlAk,Ilr,t&,l size i If AP-NED of size for 1 f p < r with constants (J,SJon an array Xnt (ii) which is of size -J, then (-Lr,ij,-xl is an fv-mixingale of ( FSCJ t-mixing J(1 1/r)) with respect to constants cnt < maxl llAkfIIr,&,l size '==-xl';xl
-.b
-minty,
-
.
-.b
-minlh,
-
,
.
Proof Immediate on inserting n before the t subscript wherever required in the last proof. w The replacement of Ff by F,v and ;sl by 5ntsis basically a formality here, since none of our applications will make use of it. The role of the array notation will always be to indicate a transfnnation by a function of sample size, typically the normalization of the partial sums to zero mean and unit variance, and in these @fyfor al1 a,. cases 5js Reconsider the AR process of 14.7. As a special case of 17.3, it is clear that in that example Xt is CP-NED of size on Zt, an independent process, and hence of size for is a wmixingale every p > 0. There is no need to impose smoothness assumptions on the marginal distributions to obtain these properties, which will usually be a11 we need to apply limit theoryto the process. These results allow us to tine-tuneassumptions on the rates of mixing and nearepoch dependence to ensure specitic low-levelproperties whichare needed to prove convergence theorems. Among the most important of these is summability of the sequences of autocovariances. If for example we have a sum of terms, Sn Z:=1Xr, =
-x
-x,
=
Theory of Stochastic Processes
266
we should often like to know at what rate the variance of this sum grows with n. 0 with no loss of generality, Assuming Extj =
F()
n- 1
n
XA'(Ff)+
=
2X >1
>1
n
-t
TElxtupwml
,
m=1
a sum of n 2 terms. If the sequence (zY,l is uncorrelated only the n variances appear and. assuming unifonnly bounded moments, Esn)2 = On). s or genera j dependent processes, summability of the sequences of autocovariances (EXtXt-j), j e N ) implies that on a global scale the sequence behaves like an uncorrelated 2 = o (n). sequence, in the sense that, again, E(Sn) For future reference, here is the basic result on absolute summability. To fulfil subsequent requirements, this incorporates two easy generalizations. First we consider a pair Xt,Yt), which effectively permit.s a generalization to random vectors since any element of an autocovariance matrix can be considered. To deal with the leading case discussed above we simply set Ff = Xt. Second, we frame the imposingfv-boundedness resultinsuchawayas toaccommodateendingmoments, uniform fv-boundedness. It is also noted that, like previous results, this but not extends trivially to the array case. 17.7 Theorem Let fXf,)',) be a pair of sequences, each &&r-l)-NED of size W ith respect to constants fdxtvd't) for r > 2, on either (i) an a-mixing process of -1,
size and -s(r-2),
(r,
<<
or
(ii) a g-mixingprocess
of size
-rlr-
1), where
11r,11,. vlaenthe sequences
IExtYt+ms I Xl
J't+m
r
<<
IIx,II,
(17.21)
m e (N
'
r
t,v,
of are summable for each t. Also, if arrays lAk,l%,l are similarly f,s(r-1)-NED dnYt << 11 and rnrjlr, size with respect to constants ftfr,JJllwith dnt rt IIAk/IIr the sequences -1
IE-ktl'rn,t.vmjI Xnt
yn,l+pl
r
are summable for each n and t.
(17.22)
'
r
EI
Since r > 2, the constants appearing in (17.21) and (17.22) are smaller (absolutely) than the autocorrelations, and the latter need not converge at the same rate. But notice too that r/(r- 1) < 2, so it is always sufficient for the result if the functions are JU-NED. Proof As before, let Ests)
=
E.
ity,
lE-vmx
I f 1E-vmxt
The modulus and Hlder
1TJ),and -
let k
Y+-zAll l +
inequalities give
=
(
the triangle inequal-
If(1$+,x1+-1#,)l
.
(17.23)
Near Fp/c
IEb-vmxt X where v,,, is of size 1, and b
Dependence
e+-CA)) < I
267
IlrllA V+-l-'tIIr/(,-I) 1--,+,,, 11 l+,,, Ilgzlcv-l', f 11
-
Also, applying
-1.
(17.24)
-
17.5(i) with p
rllr
=
-
1), a
=
r/tr
-
2),
=
IE
i' t+mE7kx t) I -.k
=
1$+,?,F7kXJ 1 1Ftekt+-1 IIzE)f+-.',1$+,,, r/(r-
IIA1+-',A)r
11 r/tr- 1) 111,11 r 11
11 < E-t-+kYtwm
1)
< 11 :; II-')IIrcl'+,,,(It and (; is of size Combining
F,IIr,Jl'l where cr < maxt 11 and (17.24) (17.25)in (17.23)gives
-1.
(17.25) the inequalities in
ltv( + (() Il.f/llrcr+,,, 11 F,+,,,11ryJ, 1$+,,,IIrt,,,, f
IExtYtmmlI s
maxl
(17.26)
This completes the proof of (i). The proof of where (,,, (vJ+ (() is of size using with r/tr 1) and b 17.5(ii) is similar 1. (ii) a p -1.
=
=
=
=
-
For the array generalization, simply insert the n subscript after every random variable and scaling constant. The argument is identical except that 17.6 is applied in place of 17.5. .
17.3 Near-Epoch Dependence and Transformations Suppose that (Xjt,...,Xvt4' Xt g(...,Ft-1,FsFf+1 is a v-vector of LP-NED functions, and interest focuses on the scalar sequence ((aV)J, where (t: 1- F- R, RV, is a fv/f-measurable function. We may presutne that, uner certain condi1-q tions on the function, ($(#f) will be near-epoch dependent on fJSJif the elements of Xt are. This setup subsumes the important case v 1, in which the question at issue is the effect of nonlinear transformations on the NED property. The dependence of the functional form 9f(.)on t is only occasionally needed, but is worth ,...)
=
=
=
making explicit.
The first cases we look at are the sums and products of pairs of sequences, for which specialized results exist. Let Xt and Ft be LP-NED on F, is AP-NED of size
17.8 Theorem Then Xt +
-mintfpx,tpyl
Proof Minkowski's Xt + F,) 11
-
(J/IJof respective
sizes
-(px
and
-f?y.
.
inequality gives E'7'''''(m + F,) II,<
<
+ 11 112,F7''''''A)Il, FJ
-
dhx tm
+
d% tm
''
< dvmt
-
A7''''''1',11,
(17.27)
Theory of Stochtic
268
wheredt
maxf
=
+vp,F and v,n = vxm
/vt JFlr ,
Processes
tnt,?l-'nint'x,'rll
=
.
.
A variable that is Q-NED is AP-NED for 1 S p S q, by the Liapunov inequality, so there is no loss of generality in equating the orders of norm in this result. The same consideration applies to the next theorem.
17.9 Theorem Let Xt and l be Q-NED on (FrJof respective sizes
-x
and
-tpj..
-.rninttpsfpxl
Then, XtYt is AI-NED of size
.
Proof By rearrangement, and applications of the triangle and Cauchy-schwartz
inequalities, we have E IXtn
-
F7':(mF,)
j
j
+ (me1+-,1F, (F7;m)(M+-''''''J',)) = E xtyt x,l+-rlyfl -
-
-
Fj+-,1((-Y,eX,1m)( 1', A'7': F,)) -
-
IIe'7''',''l',II2II-Y,e'7lx)llz
+ l', 17,11lI2 < 111,11211 -
F, , 11 Ilxre7,,,'''.&II + f'',v,,,Ft/'/v,l -
-
zdkivmY +
ztl%- + ir,11 11 I
11 < 11.1,
dt
=
maxt
j
-
el+-,,,'''Frll ,
(17.28)
F,1128,,dtidxtl IIx,II2t//, 11 Vm
=
X F X 1'+ Mm Mm + MmTm
and gtyyj-rninltpxrli
=
.
g
ln both of the last results we would like to be qble to set 1$ = Xtwjfor some finite j. A slight moditication of the argument is required here. 17.10 Theorem If Xt is AP-NED on ( Wl so is Xtwjfor 0 ,
<
j
<
x.
Proof lf Xt is fP-NED, then Il-Y,w Ext-q -
I@f'+Xy+-'''''') 11, 11,< 2 11.X',+./EXt+) IT',++'t.,+-:) -
S dt
using 10.28,
where
#;X
2JXf
(17.29)
vm,
we can write Xt-vj E t--Yrm./ f d'txvm' 11 l S7,1) 11p =
+j.
(17.30)
-
,
where #'
Vr?j
and
Mm' is
of size
-(p
=
if vm is of size
v0
,
Tm-j, -..
m <
j
>
J
m
.
.
Putting the last two results together gives the following corollary.
17.11 Corollary If Xt and Ff are Q-NED of size
-gx
and
-(ps
XtYtwkis AI-NED of
Near Fp/c
size
DepeWence
269
-minttps:xl
n
.
By considering Zt
the AI-NED m-gwjlk-gwj,
=
/
V?u
numbers can be given here as
Vo,
m f
+ 1 W/21
Vr,l-(1/21-1,
m
>
(W21+ 1
=
assuming wherev,a dYtvm+v,u +vruvp,, an d the constants are 4#f-wmY+k-g1a, that dxt and are not smaller than the corresponding Lz nonns. =
X
F
F
X
Al1 these results extend to the array case as before, by simply including the extra subscript throughout. Corollary 17.11 should be compared with 17.7, and care taken not to confuse the two. ln the former we have k fixed and finite, whereas x. The two theorems naturally the latter result deals with the case as m each other in applying truncation complement arguments to infinite sums of products. Applications will arise in subsequent chapters. More general classes of function can be treated under an assumption of continuity, but in this case we can deal only with cases where 9f(A) is Q-NED. Let --+
RV
4)(x):-I' F-> R, 'T i be a function of
real variables,
1/
ptxl
,.'r
2
and use the taxicab metric on V
)
IA'f1
=
2
-xi
I
(17.31)
,
izu1
RV,
2 to measure the distance between points x 1 an d x We consider a set of results that impose restrictions of differing sevelity on the types of function allowed, but offer a trade-off with the severity of the moment restrictions. To begin with, impose the uniform Lipschitz condition, .
l(1),(.1 1) 9/xl ) I < -
B , pt.tl
xll a.s.,
,
(17.32)
where Bt is a tinite constant. 17.12 Theorem Let Xitbe AZ-NED of size on f F2) for i = 1,...,v, with constants with constants dit. If (17.32)holds, ((aV)) is also AZ-NED on f F2) of size multiple maxftljll. of a finite Proof Let 4),denote a Fj+-l-measurable approximation to /ml. Then -a
-c,
II4h(A)
-
Ak
< 119,(2,) 9,112 FJ-,,''>,(.X))Il2
+
(17.33)
-
by 10.12. Since ((j+-;;1A) is an Tjlrl-measurable random for this choice of ,, and by Minkowski's inequality,
variable,
F7''''''9,(A)112< #,Ilp(.2',,.E7;7,-Y,) 112 119,(1,) -
V
u Bt
2 11-'t,e,!,,,',,-'t,II -
i=1 V
< Bt
ditvim f=1
(17.33)holds
Theoty of Stochastic Processes
270
f dtvm, and v,,,
where dt vffmaxjltitl by assumption. x =
=
(17.34)
Y the latter sequence v -1Zjazlvfm,
being of size
-.a
lf we can assume only that the Xit are TV-NED on F, for some p e (1,2),this argument fails. There is a way to get the result, however, if the functions (t are bounded almost surely. 17.13 Theorem Let Xitbe JV-NED of size dit, i = 1,...,v. Suppose that, for each t,
I9,(.1l)
-
Proof
l#1, -
For brevity, write ( i 9, I / zMmintz, 1 ). Then
=
,7(9)
on
%).
2
-4,:2
,
)) is Q-NED
x.
9r(X
I
on ( Ff ), for 1 f p f 2, with constants (A) j f M < x a.s., and also that
(#l xl)
2
#f(-Y ) l f m inl Blp
Then ((# finite multiple of maxjttf)
where Bt <
-a
,
(Ftl
of size
and let
,
2M) a.s.,
z
=
(17.35)
-apl2,
with constants a
#,p(A'l .Y2)/2M so that ,
,
Jtzs,)(9l f'(z>1,(9l <(2.2 Jtzs,lzz' Jtzoll'l -tbldr
-tg)2'
+
-
<
+
f (234)2s(z,)
= where Bjt
BitEpx,xl)pl,
(17.36)
Bq/2(gAy)1-p/2 combining (17.33)with
=
.
(17.36),we
can write
F7:,(#2Il2s #1rlIp(--F7''''''.f,) 119/2 114,,().;,) -
< #l, < B3t
v =1
v
plz
Ilx .#7j+-,,j&'.lI?) -
p11
ditpim
f=1
(17.37)
< dtvm, W
hel'e
dt
=
#1,vP/2maxj(J(/,2j and vm
assumption.
=
(v-l t=
kvimlpll ,
which is of size
-aplz
by
.
An important ekample of this case (withv = 1) is the tnmcation of Xt, although this must be defined as a continuous transformation.
17.14 Example For M
>
0 let
Near Fp/c
Dependence
.x, IxI f @)
=
M
M, x
>
M
-M, x
<
-M
(17.38)
xl x1( Ixls.v) +M(x/I
)1(I.vI>v) In this 2 l9(-Y1) #(-Y ) I < jxl x2j 1, and 17.13 can be used to show that I()IXJJis The more conventional truncation,
or, equivalently,
4,(x) =
.
case
-
-
,
so set B
G-NED.
=
Xflf IxlsM)
Q-NED if (X,) is
Xt, IXt I S M
=
(17.39)
otherwise,
0,
cannot be shown to be near-epoch dependent by this approach, because of the lack of continuity. n rther variation on 17.12 is to relax the Lipschitz condition (17.32)by letting the scale factor B be a possibly unbounded function of the random vari-
A
ables. Assume
I9,(.Y1) 9r(xl ) j -
where, for each
< B, (x1 x2)p(x1x2) a s., ,
,
(17.40)
.
f,
Btx
1xl ):
'!r
,
x'r
(17.41)
A/s-measurable function. To deal with this case requires a is a non-negative, B lemma due to Gallant and White (1988).
17.15Lemma Let B and p be non-negative < x, and Ilfpllr< x, for q k 1, and r >
Illpll2S
r.v.s and assume 2. Then
< Ilpllq 11#11#(t?-1) c'o,
2(IIpIIJ-2II#II-/1-l)Il#pII;)'/2Cr-1).
(17.42)
Proof Detine
c (IIpIlII#II/(-l)II#pII-rr)''f1-r), -
and let #1
=
Then by the Minkowski inequality,
ltapscls.
< II#1pII2+ -#1)pII2. II#pII2 11(.:
The right-hand-side
(17.43) ( 17.44)
ter'ms are bounded by the same quantity. First, 1/2
II#lpII2 =
Jspsctfplof'
jBp < c1?2
1/2
dp
/ c''2(IIpIIII#ll/f:-1))''2,
(17.45)
F/ldtpry of Stochastic Processes
272 applying the Hlder
inequality. Second,
11(/
Allplla
-
1/2
jsvwcBtldp
=
cl
<
1/2
j
-rll
(splup
Boc
Ij;/2
< t)2
'-r/zljlp
(17.46)
where the first inequality follows from r > 2 and #p/C > l Substituting for C in (17.45) and (17.46)and applying (17.44)yields the result. . .
The general result is then as follows.
17.16 Theorem Let l#f ) be a v-dimensioned random sequence, of which each element is Q-NED of size on (FCJ,and suppose thatY/m)is fvbounded. Suppose further that for 1 s q S 2, -a
< x, I1p(.X',,F7;Jl.fJllt?
< x, lI#,(-Y,,e-7:A',)IIc&c-1)
and for
>
r
2, < oo. Il#?(#/,%+--M#,)p(#f,Fj+->,)IIr
Thn
((f(A)
l
is JU-NED on
( Ff)
of size
-alr
-
2)/2(r
-
1),
Proof For ease of notation, write p for p(.X',,F7:#Jand B for BtxtnE7-zXtl. As in the previous two theorems, the basic inequality (17.33)is applied, but now we have
11(-)) e'7Jr9(A-f)l12< -
114,(.X',)#(F77X))Il2 -
llfpll2 (17.47) s 2IlpII:Cr-2)/2(r-1)Ij#jj4(;-(:21(2)(r-1)jjspjjrr/2(r-1) S
where the last step is by 17.15. For q f 2, V
Ilpllq<
< lIpIl2
V
77 f=1
dtvm7-ltrvf,,, f=1
-.E1+-,,,',,.X112 11.1/, s
=
and Mm v -1E( v im, which is of size Hence, under the stated assumptions,
where dt
=
v maxjltl)
=
.1
< div,?,tr-:D/zr-ll lIf(-'t)-.E:+-.p,''',(-Y,)lI2
7
-a
(17.48) by assumption.
(17.49)
jlrr/ztr-ll(y #J II#llJ)-(22(2)(r-1)jl#p s where =
t.
Observe the important role of 17.15 in tightening this result. Without it, the best we could do in (17.47)would be to apply Hlder's inequality directly, to obtain
Near fp/ch
273
Depeeence
< IIpIIuII#II2w(?-l), 1. lI#pII2 q
(17.50)
The minimum requirement for this inequality to be useful is that B is bounded almost surely pennitting the choice q = 1, which is merely the case covered by 17.12 with the constant scale factors set to ess sup Bt(X 1 ). The following application of this theorem may be conasted with 17.9. The moment conditions have to be strengthened by a factor of at least 2 to ensure that the product of Q-NED functions is also Q-NED, rather thanjust LI-NED. There is also a penalty in terms of the AZ-NED size which does not occur in the other case. .v2
,
Letav < x and 11 XtYt.Assume that lI#,II2r Xt,Yt)and ()tffl 1$II2r 2, and that Xt and Ff are ZZ-NED on ( Fll of size -J. Then
17.17Example <
x
for r
>
-'1F)
l
-
=
-Yl1'l K I
=
-)
I II l'l I + I K ( Ixl I + IFl I)(IFl
.11
.1)
-
-
-
IlFl l'1 + I-Y) l -
A
I)
#(A1,A)p(#l,.f),
(17.51)
=
defining B and p. For any q in the range (4/3,41,the assumptions imply K 1111 < II#(#),-Yl)Ilw(?-1) FllIw(?-1) II/(-1)+ 11 x,
11.X1 Il-Ylll < < IIl'l1I?+ Ilp(.Cl,-Yl)Il? IIl'lIl?+ Il?+
=,
(17.52) (17.53)
and 1') :K 11.11112r11 II#(-Yl,-Yl)p(-Y),-Yl)IIr Il2r+II.X1II2rIlFlll2r+II.X1lIlr lIFlII2rII FlI12r+ IIFlIllr +IIx)lI2rIlA:II2r+ IIFlII2rIIAIl2r +IIFlII2rII.X1II2r+
(17.54)
Putting -Y1
Xt and .Y1 E7-;,Xt, the conditions of 17.16 are satisfied for q in the range (4/3,21and Xtj is Q-NED of size -J(r 2)/2(r- 1). n =
=
-
17.4 Approximability ln j17.2 we showed that N-NED functions of mixing processes were mixingales, and most of the subsequent applications will exploit this fact. Another way to look at the JV-NED property is in tenns of the existence of a finite lag approximation to the process. The conditional mean E(Xt IJ;j+-/l)can be thought of as a function of the variables 6-m,...,Vt+m, and if (Ff) is a mixing sequence so is 1F(A! Fj+-'QI, by 14.1. One approach to proving limit theorems is to team a limit theorem for mixing processes with a proof that the difference between the actual sequence and its approximating sequence can be neglected. This is an alternative way to overcome the problem that 1ag funtions of mixing processes need not be mixing.
Iheory of Stochastic Processes But once this idea occurs to us, it is clear that the conditional mean might not be tlle only function to possess the desired approximability property. More generally, we might introduce a definition of the following sort. Letting 1$ be l x 1, l(2m+1) = F-y R as a gyr-yvlys-measurable function, where st*m t-m t-m we shall think of /C:R c(F,-?,,,...,F,+,,;). Dennition The sequenee fm) will be called (p 0) on the u-approximable 17.18 there seq'u of exists if for each Jncef/li'l s?i/fx'-'/'/-measurable m s s a f v,) sequence variables, and random >
II-Y,-/ClI,S dtvm,
(17.55)
c'o. (X,) will where fdtI is a non-negative constant sequence, and vm--A 0 as m approximable in probability (or mapproximable) on fFf ) if also be said to be > 0, there exist (T), (t$), and (vpr,las above such that, for every --)
#(IA
'71
-
>
dtn) f vm. u
(17.56)
The usual size tenninology can be applied here. There is also the usual extension to arrays, by inclusion of the additional subscript wherever appropriate. If a sequence is fv-approximable for p > 0, then by the Markov inequality
#( lXt 8-J'vj,;
where s'm'=
-
h'l I > dtn) < ()-PlI-Y,
hence an
-
< vm' /1'J'11p#
(17.57)
,
p-approximable
An process is also Q-approximable. although only in the casep 2 are we able to
AP-NED sequence is fp-approximable, is the best w-approximator in the sense that claim (from10.12) that Ext 11j+-'''m) the p-nonns in (17.55) are smaller than for any alternative choice of hl'. =
17.19 Example Consider the linear process of 17.3. The function
h';l X =
(17.58)
jvt-j
j=-m
unless @,'+-':) isdifferent from A'(#,I for xt since zw-approximator
112,hmt11, 77 jvt-j + jzzm1 + joy1 l l) and dt 7=,u+1( -
where vm
=
(J/',) is an independent process, but is also an -jb-vjt
=
-j
> =
< dts'm,
(17.59)
sup,ll 7,IIp. o
17.20 Example In 17.4, the functions g are fv-approximators for Xt, of intinite < x. n size, whenever sup,sf IIF,F,IIp One reason why approximability might have advantages over the AP-NED property is the ease of handling transfonnations. As we found above, transferring the JV-NED property to transformations of the original functions can present difficulties, and impose undesirable moment restrictions. With approximability, these difficulties can be largely overcome. The first step is to show that, subject to Zr-boundedness, a sequence that is approximable in probability is also fw-approximable forr < r, and moreover, that the approximator functions can be bounded for
Near Epoch Depcndence
each finite m. The following is adapted from Ptscher
and Prucha
(1991a).
17.21Theorem Suppose fXf) isfv-bounded, r> 1, andfvapproximablebyT. Then for 0 < p < r it is fv-approximable by J?;' /CI ( Ihmtj
.
Proof Since hmtis an fm-approximator of Xt, we may choose a positive sequence ,u) x. Also such that 8,u 0 and yet #( IXt h'l I > dtnm) f Mm --+ 0 as m f butlfmv,u choose a sequence of numbers (A&) having theproperties Mm -1/2 = For example, Mm vm would serve. T here is no loss of generality in assuming supm'/k-1 ; 1 B y Minkowski's inequality we are able to write -->
--
-
.-+0.
--:x,
.
IlA-/7IIpf
where
(17.60)
41,,,,II(A- 1:,)1(Ix,-pI>.,.u) 11p, =
A1t
=
:7''')1f Ix,-r;'I:.f,A<,,,) t 11(x,-
43
=
(I(xt
m
t l11
Hlder's
A)r,,+z4lm+Al,?,,
llp, 11p.
-''')1(
-
f
Ix,-'krllxtfj:m,shrylstjAfm)
inequality implies that 11A'1'11, S
> 1. II#II,qIIl'II?v&-l),
chooseq r/# and apply (17.61)to Amitfor i dtMm,again by Minkowski's inequality, IImIIr+ thefollowing inequalities. First, =
=
(17.61)
1,2,3. Noting that 112,/')'IIr< lsllfx= #(F), we obtain and that 11 -
/ Il-Y, l'7IIr#(I-Y, /'71 > dtnm) 41,,,, t4(IIA/IIrM,,,-1 + 1)MmMm. f -
-
(17.62)
Second, observe that
l IXt
'l
II';'I >
lS
dtnm.
and hence, when
slm>
&m,
Ixt- /';'I <
dtnm,
I/T1 >
P
-
dtsl)
t IXt I >
dt Mml i S #(
11,1>
dtMm
f IIXIII
rrdirum
by the
dtMm
-
-
,,,)
l
,,,)) -
nml-r
(17.63)
Markov inequality, so that Al,,, <
11.2,-:'IIr#(11,-/T1 < dtnm, I/')'I > J,M,?,)II-Y,IIrrtrfr(Az,, < (IIAlIr+ ,,,)-r
-
S
dtsl)
+ II.X',/f,II3ArJ',,,(M,,, (IIA/4Ilrr+' nm4-r. -
(17.64)
-1 b y 1 ) And lastly, (The final inequality is f'rom replacing Mm .
Atm 3 / dn, m
in view
of the fact that
7;T =
h'l on the set
(1765) .
/';I
fI I S
dtMml. We have therefore
established that Xt 11
A-''' t p S d'vm' t 11
-
(17.66)
,
where
v and
%'m'
'
M v,n+MmMm
=
0 by assumption, since r
--)
d;
=
m)-r+
-
(17.67)
1, and
>
.'
Il.Xr/ttllrr+l,IlA/t4ll rr, 1 ) ( lI.Y,/#,IIr, ztf/rnax
.
(17.68)
.
If o-approximability is satistied with dt
-approximability
17.22 Theorem Let 'J' htt...shmvt) be the fo-approximator of Xt = Xjt,...,Xvt)' RV EBtXt,h'1)E) < x for : > 0, F-> R satisfies (17.40),and If #t: of size then (j,4.Y,)is fo-approximable of size =
-(p.
-(?.
Proof Fix 6
>
0 and M
#( I#r(m) 9,('7)I > -
>
0, and define dt
Elxllff.The
=
dt) S PBt-knhllqpxphl't
+
>
#(#f(A'r,h'7)p(m,:')
F(#/(#s')')E)/3#
+
Markov inequality gives #f(m,:')
s, >
>
dtn, Btxt,h',
Plpxt,hll
>
dt&M).
M4 S M)
(17.69)
Since M is arbitrary the first term on the majorant side can be made as small as desired. The proof is completed by noting that V
Ppxt,h'l)
>
J,&M)
P
Im, X =1
KP
Ut Iff
=
-
'rri
>
dtnlM
>
ditnlM
ll
-
jzz1 %'
#( jXit
<
-
't',
t
l>
dit&lM)
V
f
Xvjm 0 as m -+
--y
x.
w
(17.70)
=l
It might seem as if teaming 17.22 with 17.21 would allow us to show that, given an and fv-bounded, Q-NED sequence, r > 2, which is accordingly Q-approximable hence Q-approximable, any transfonnation satisfying the conditions of 17.22 is and therefore also Q-NED, by 10.12. The catch with circumQ-approximable,
Near Fp/c
Dependence
277
venting the moment restrictions of 17.16 in this manner is that it is not possible to specify the Q-NED size of the transfonned sequence. In (17.67),one cannot put a bound on the rate at which thesequence (,,, ) may converge without specifying the distributions of the Xit in greater detail. However, if it is possible to do this in a given application, we have here an alternative route to dealing with transformations. Ptscher and Prucha (1991a),to whom the concepts of this section are due, define approximability in a slightly different way, in terms of the convergence of the Cesro-sums of the p-nonns or probabilities. These authors say that Xt is fv-approximable p > 0) if
limsup1 -
n
7) 111,-
'yllp
0 as m
--
11 >1
n-x
and is Q-approximable if, for every
(17.71)
.-->
x,
0,
>
n
1
limsupn-X#(lx,- :,1 > )
O as m
--->
(17.72)
--+
x.
>1
n-yx
lt is clear that we might choose to define near-epoch dependence and the mixingale property in an analogous manner, leading to a whole class of alternative convergence results. Comparing these alternatives, it turns out that neither definition dominates, eachpermitting a form of behaviourby the sequences which is ruled out by the other. If (17.55)holds, we may write n
1
limsupn Y 112,--
'/lln
'''
n-yx
''r
-
A
>1
1
n
Tdt n
< limsup
v,a
-->
0
(17.73)
>1
n--hx
so long as the limsup on the majorant side is bounded. On the other hand, if (17.71) holds we may detine n
v,,,
=
limsup
-
17) 111,hmt11,,
(17.74)
>1
n-x
will satisfy 17.18 so long as it is t'initefor and then dt = supp,t 111, Tllp/vml finite t. Evidently, (17.71)permits the existence of a set of sequence coordinates for which the p-norms fail to converge to 0 with m, so long as these are ultimately negligible, accumulating at a rate strictly less than n as n increases. On the other hand, (17.55)permits trending moments, with for example #f 0(/), -
=
,
0, which would contradict (17.71). Similarly, for ,u > 0, and vm > 0, define dtm by the relation >
#( IXt
-
h'II >
dtmm)
Mm,
=
(17.75)
0, define dt = suvmdtm. (17.56)is satisfied and then, allowing v,,i 0 and m if dt < x for each tinite t; this latter condition need not hold under (17.72).On the other hand, (17.72)could fail in cases where, for jed 8 and every m, -->
--
P IXt
-
';l
I > ) is
tending to unity as /
-->
x.
1A? THE LAW OF LARGE NUMBERS
18 Stochastic Convergence
18. 1 Almost Sure Convergence in 512.2.Sometimestheconditionis
wasdefinedformally
Almostsureconvergence stated in the form
# limsup IXn 11
> 6:
-
0, for a11 E
=
>
0.
(18.1)
n->x
1 where, for each (t) Yet another way to express the same idea is to say that #(C) > number E > finite of times as we pass and at most a any s 0, I-L() Xt)l I e C down the sequence. This is also written as =
-
#( IAk Xl -
where i.o. stands for
i.o.)
> E,
0, all E, > 0,
=
(18.2)
often'. probability in (18.2)is assigned to an attribute of the whole Note that the often' idea is sequence, not to a pmicular n. One way to grasp the words, consider the event U';=,,,l event that has occurred to IXn *-1 > erl; in 11 > for least beyond Xn el occurs whenever ( I at one n a given point m in the sequence' lf this event occurs for every m, no matter how large, ( IXn 11 > el occurs infinitely often. ln other words, tinfinitely
'infinitely
'the
-
-
-
.
Cr
f IXn 71 -
>
s, i.o.)
r
O U( IXn
=
-
#1
> El
vl=1 n=m
=
limsupt
IXn
-
11
>
el
.
(18.3)
n-x
Useful facts about this set and its complement
are contained in the following
j emma.
18.1 Lemma Let fEn
TlT be
G
an arbitrary sequence. Then X
(i) P
limsup En
=
1im P n--x
n--x
UEm
.
m=n X
(ii)#
liminf En n-'hoo
=
1im #
OEm
.
m=n
is decreasing monotonically to limsuw Fn. Part Proof The sequence (U7=,aFa1'Q1 (i) therefore follows by 3.4. Part (ii) follows in exactly the same way, since the
The frw of f-zzrg: Numbers
282
sequence f*n=mbJo;=lincreases monotonically to liminf En.
.
A fundamental tool in proofs of a.s. convergence is the Borel-cantelli /'r/;zz?.t7. This has two parts, the part and the part. The former is the most useful, since it yields a very general sufficient condition for convergence, whereas the second parq which generates a necessary condition for convergence, requires independence of the sequence. tconvergence'
Kdivergence'
18.2 Borel-cantelli lemma (i) For an arbitrary sequence of events fEn e
??7)T,
X
XpEnb
PEn, 1
0.
.0.)
<
-
=
(18.4)
n=1
(ii) For a sequence (En e 5 )7 of independent events, X
#(&)
i.o.)
Pk
=
x
=
1.
=
(18.5)
n=1
Proof By countable subadditivity, e
#
'
x
w
U
< XPEn4.
En
(18.6)
n=m
n=m
The premise in (18.4)is that the majorant side of (18.6)is tinite for m x (by 2.25), which further implies This implies X*n=mPEn) 0 as m
1.
=
-->
-->
X
1im P pl-hl
UEn
0.
=
Part (i) now follows by part (i) of 18.1. To prove (ii), note by 7.5 that the collection for any m > 0, and m' > m, /'
#
(18.7)
n=m
(AN,(
e
;)7
is independent; hence
;
;
(1 g)Ecn I-IPYn) 1-I =
h=n1
=
n=m
-
P(En44
n=m
< exp
-XPEn)
--
O as m'
( 18 8)
-+
.x,
.
n=m
by hypothesis, since e-x > 1 x. (18.8)holds for a1l m, so -
X
rtliminfEcnl
=
OYn
lim P m--koo
=
0,
(18.9)
n=m
by 18.1(ii). Hence, PEn i.o.)
=
'tlimsup
En4 = 1 Ptliminf E -
=
1.
.
(18.10)
To appreciate the role of tls result (the convergence partl in showing a.s. convergence, consider the particular case
Stochastic Convergence En
f
=
IXnlk
):
I
X())
-
283
*n=3PEn)< x, the condition PEn) > 0 can hold for at most a finite number of n. The lemma shows that P(En i.o.) has to be zero to avoid a contradiction. Yet another way to characterize a.s. convergence is suggested by the following
If
theorem. (Xn) converges a.s. to X if and only if for all 6: > 0
18.3 Theorem
lim # sup lXn
.:-1
S
-
s
nkm
pl-x
=
1.
(18.11)
e1 e @,
(18.12)
Proof Let X
A,,,(E)
=
(')1:
IS
IAk(t,))
-#()
n=m
and then (18.11) can be written in the fonn lim,,,-+x#(Am(e)) 1. The sequence U,,7=lAm(E), fA,,,(E) 17 is non-decreasing, so A,,,(E) = UXlA/:); letting A(E) 1. (18. 11) can be stated as #(A(:)) Define the set C by the property that, for each ) s C, (Xn((t))lT converges. That is, for (t) e C, =
=
=
3
lA%((J))
such that sup
??z()
-
X(l
numl)
1<
e, for all
Evidently,
e C= ) e Am(E) for some (l, so that CiA(E). tonly ito 1, proving #(A(E)) 1 for al1 E > 0. Set E To show if' assume k, and define #(A(E))
=
i) >
(18.13)
0.
HencePto
1implies
=
.
=
,
X
A*
=
f)
=
t=1
$Ik for positive integer
C
<x)
Ajlk)
=
tj
Atlkjc
(18.14)
.
&1
1. The second equality here is 1.1(iv). By 3.6(ii), #(A*) 1 #(U7=1A(1/k)9 z4* is a convergent outcome in the sense of (18.13), hence A* But every element of c C, and the conclusion follows.w =
=
-
The last theorem characterizes a.s. convergence in tenns of the unform proximity of the tail sequences ( IXnll) X()) l 1';.,,,to zero, on a setAm whose measure x. A related, but distinct, result establishes a direct link approaches 1 as m between a.s. convergence and uniform convergence on subsets of (. -
--y
-6:t-:
X, there exists for every > 0 a 18.4 Egoroff's Theorem lf and only if Xn X() uniformly on C(&). set C(8) with #(C48)) k 1 8, such that Ak() .-+
-
Proof To show if', suppose aL()) converges uniformly on sets 1 2 3 The sequence ( C(1/k) k e (N) can be chosen as non-decreasing tonicityof #, and #(U7=1C(1/k)) 1 by continuity of #. To show Eonly
1
,
;***
,
=
C(I/Q,k
=
by mono1et
tif',
The fuzw of fzzrgd Numbers
284
A,,()
O (t,):lXn(t,))-X()I
=
<
llml,
( l 8 15) .
n=klmj
-2-&'.
In view of a.s. km) being chosen to satisfy the condition #(A,,,(6)) 1 convergence and 18.3, the existence of finite klm) is assured for each m. Then if *
c(8)
nAa,tl,
=
(18.16)
rrI=1
convergence is uniform on C(l by construction; that is, for ever.y ) e C(8), IXnt)l X()) I < Im for n k km), for each m > 0. Applying 1.1(iii) and subadditivity, we tind, as required, -
#(c())
1
=
UAm()f
-#
m=1 *
1 -
E (1 -
#(z4m()))
m=1
k 1 8.
(18.17)
.
-
18,2 Convergence in Probability In spite of its conceptual simplicity, the theol'y of almost sure convergence cannot easily be appreciated without a grasp of probability fundamentals, and traditionally, an alternative convergence concept has been preferred in econometric theory. If, for any E > 0,
1im#(IXn X1 -
> f:)
=
0,
(18.18)
n'M
Xn is said to converge in probability (inpr.) to X. Here the convergent sequences are specified to be, not random elements (X,,()) 1T,but the nonstochastic sequences (#( IXn Xl > E) )T.The probability of the convergent subset of ('1 is left unspecified. However, the following relation is immediate from 18.3, since (18.11) -
implies (18.18).
18.5 Theorem If Xn
-66.9
X then Xn -E'.% X.
(a
The converse does not holdkConvergence in probability imposes a limiting condix. The tion on the marginal distribution of the pth member of the sequence as n negligible deviation of is approaches Xn X 1 probability that the from as we move down the sequence. Almost sure convergence, on the other hand, requires that byond a certain point in the sequence the probability that deviations are negligible from there on approaches 1. While it may not be intuitively obvious that a sequence can convergein pr. but not a.s., in 18.16 below we show thatconvergence in pr. is compatible with a.s. n/nconvergence. However, convergence in probability is equivalent to a.s. convergence on a ..-.h
Stochastic Convergence
285
subsequence', given a sequence that converges in pr., it is always possible, by throwing away some of the members of the sequence, to be left with an a.s. convergent sequence. 18.6 Theorem Xn -EI-> X if and only if every subsequence fAk, k e IN) contains a further subsequence fxnkcj, j e (N) which converges a.s. to X. 0 for any e > 0. This means if : suppose P( IXn XI> E) Proof To prove #( that, for any sequence of integers fns k e INl IXnk XI > e) 0. Hence for each j G LNthere exists an integer kU)such that 'only
-
--
-->
-
,
#( 1Xnk 71
>
-
1/.j) < 2-/, a11k k k(j).
(18.19)
Since this sequence of probabilities is summable over j, we conclude from the first Borel-cantelli lemma that
#( IXnkv
XI >
-
$lj i.o.)
=
0.
(18.20)
It follows, by consideration of the intinite subsequences Lj k J) 0 for every 6: > 0, and hence the that #( IXnkv XI > e i.o.) required. fxnw)J converges a.s. as : if (Ak) does not convergence in probability, there To prove > s)l subsequence (rl1)such that inflf#tlAk-rl s, for some 6: > which nlles out subsequence of in out convergence (?u), pr. on any a.s. on the same subsequence, by 18.5. w =
-
Kif'
18.3 Transformations
for J > 1/:, subsequenc: must exist a 0. This rules convergence
and Convergence
The following set tools of asymptotic even though most k-vector Xn is said
of results on convergence, a.s. and in pr., are lndamental theory. For completeness they are given for the vector case, of our own applications are to scalar sequences. A random to converge a.s. (in pr.) to a vector X if each element of Xn converges a.s. (in pr.) to the corresponding element of X. 18.7 Lemma Xn X a.s. (in pr.) if and Only if IlAk #II 0 a.s. (in pr.).19 -->
-->
-
Proof Take first the case of a.s. convergence. The relation
expressedas
k
P 1im n-yoo
for any
0. But
6: >
xni Xill X f=1
<
-
(18.21)implies
# li m 1Xni Xi I < -
:2
Il#n #ll -
-E't-y 0
may be
(18.21)
that E,
=
1,
.
..,k
=
1,
(18.22)
n'-oo
ito observe that if (18.22)holds, f'tlimp,--hxllAk #1I To prove IS 1 for any : > 0. To get the proof for convergence in pr., replace < k E) #(limn-+x...) everywhere by limn-+x#(...), and the arguments are identical. w proving
Konly
'if'.
-
,
=
,
The fww of f-zzr'd Numbers
286
There are three different approaches, established in the following theorems, to the problem of preserving convergence (a.s. or in pr.) under transformations.
18.8 Theorem Let Rk R be a Borel function, 1etCg R be the set of continuity points of g, and assume PX e G) = 1. #(m. X then g(Xn4 (i) If Xn -Z'.:o .-.E'.4 X then gxn) (ii) If Xn -->
':
-6.+
-6-+
'(m.
t
is by hypothesis a set D e @, with #(D) = 1, such that A%() #4(z)),each (t G D. Continuity and 18.7 together imply that glxnll for each (.t e X-1(Cg ) ch D. This set has probability 1 by 3.6(iii). g(#()) Toprove (ii),analogous reasoning shows that, for each r > 0, 3 > O such that Proof For case
(i), there
--
--
1t0:IlX()
-A)lI
Note that if #(#)
) c'-kr,sj
<
c (: l'(A())
1 then for any A
=
PA rn #)
1 P(Ac
=
-
LJ
G
< :).
-p(#())l
(18.23)
@,
Bc) k PA)
-
PBc)
PA4
=
(18.24)
by de Morgan's law and subadditivity of #. ln paYcular, when PX e Cg) = 1, (18.23) and monotonicity imply
#((I#n-fIl
YS
<
#(I'(Ak)
-
#(mI <
E).
(18.25)
Taking the limit of each side of the inequality, the minorant side tends to 1 by hypothesis. w We may also have cases where only the difference of two sequences is convergent.
18.9 Theorem Let t#a) and (zn) be sequences of randomk-vectors (notnecessarily converging) and g the function detined in 18.8, and let P(Xn e Cg) = P(Zn e Cg) = 1 for evel'y n. 0. (i) lf IlA% 11 O then 1:(.Yn) #(za)I -E',ly 0 then I#(.2L)#(zn)1 0. (ii)If llAk -61..>
-f..>
-zn
-
-znll
-r...y
-
OQazzjvn EZa PCEX) = Z-a1(c#), EX = ()xnmjl, and FZ = J'ut E'n = Xn-lc ), PEZ) ! and 3.6tiii). Also let D be the set on which Il#nZn 11 = 1, by assumptlon The proof is now a straightforward variant of te preceding one, with converges. X EZ r1(C the set E c g ). . playing the role of
Proof
=
.
-
The third result specifies convergence to a constant limit, but relaxes the continuity requirements. R be a Borel function, continuous at a. 18.10 Theorem Let g: R If Xn gx ga). (i) a then (ii) If X. -E(- c then g(Xn4 -E(-> ga). --,9
-6-,.-+
-6-y
Proof By hypothesis there is a set D G F, with #(D) = 1, such that Xn c, ga) for (1)e D, proving (i).Likewise, each (l) e D. Continuity implies #(A%()) --
-->
1:
llA%()
-cll
<
l c lt,):l#(Ak()) -g(c)I
and (ii) follows much as in the preceding theorems.
*
< El,
(18.26)
Stochastic Convergence
287
Theorem 18.10(ii) is commonly known as Slutsky's theorem (Slutsky 1925). These results have a vast range of applications, and represent one of the chief reasons why limit theory is useful. Having established the convergence of one set of statistics, such as the first few empirical moments of a distribution, one can then deduce the convergence of any continuous function of these. Many commonly used estimators fall into this category.
18.11 Example LetA,y be a random matrix whose elements converge a.s. (inpr.) to a limit A. Since the matrix inversion mapping is continuous everywhere, the A ) follow on app j y jyjg j.: : ejement by results a.s.lim An-1 A (plimA,y element. E! -1
-1
-1
=
=
.
The following is a useful supplementary result, for a case not covered Slutsky theorem because Fn is not required to converge in any sense.
by the
18.12 Theorem Let a sequence (Fn)Tbe bounded in probability (i.e.,Op(1) as n x); if Xn -EI-> 0, then XnYn.-CL>0. FPn
Proof For a constant # > 0, define L1 ( Iyklus). The event ( lXaF,,I 6: > 0 is expressible as a disjoint union: IXnYnI sl l IA%IIF$ l k e:1 tp f IXnIIi'n F$ l El
t
For any :
>
0,
-
.
( IXn IIF,, I k #(l-Yn lIi
Ik
c l IXn I 2 e/#l,
:1
el for
=
=
s) < #(l-Yn 1
zlB4
-->
(18.27)
and 0.
-->
(18.28)
> 0, B& < cxa such that By the 0:(1) assumption there exists, for each YB YB #( IL Fn%I > 0) < for n e ENSince ( IXnIIF,, n l sl c l Il'n n I > 01, (18.27)and additivity imply, putting B = Bs in (8.28),that -
-
-
.
lim P( lXnk
lk
e)
<
(18.29)
.
n.->x
The theorem follows since both 6: and 6 are arbitrary.
.
18.4 Convergence in L Norm #
Recall that when F(l Xn IP) < x, we have said that Xn is fw-bounded. Consider, for lf F(llxnllp < :cn all n, and limn-jxllAk --Y11p 0, p > 0, the sequence ( II-L-XIIpl7. When p = 2 we speak of Xn is said to converge in N zm?wzto x (writeXn l convergence in mean square (m.s.). Convergence in probability is sometimes called fvconvergence, terminology which can be explained by the factthatfv-convergence implies fx-convergence for 0 < q < p by Liapunov's inequality, together with the following relationship, which is immediate from the Markov inequality. =
---+
m.
-1f,+
18.1 Theorem lf Xn The converse
X for any p
>
0, then Xn .T.LA X. n
does not follow in general, but see the following theorem.
18.14 Theorem If Xn
.-#-(..>
X, and
( IL IPITis uniformly
integrable, then Xn -f:E.-)X.
288
The fzzw of zlr':
Numbers
Proof For e > 0, e-lAk
-11:
=
F(1( Ix,,-xl''>:) lXn
K e'(1(Ixn-xj''>EJ
-11#)
+
F(1f
--YlP) laY?,
Ixa-xtzkEl
+ s. l-Yn-XI#)
(18.30)
x. Uniform integrab0 as n Convergence in pr. means that #( lXn Xl > e) ility therefore implies, by 12.9, that the expectation on the majorant side of (18.30) converges to zero. The theorem follows since e: is arbitrary. K .-+
.-.+
-
We proved the a.s. counterpart of this result, in effect, as 12.8, whose conclusion can be written as: jXn X1 -E.. 0 implies A'lXn XI 0. The extension f'rom the fw1case to the Lp case is easily obtained by applying 18.8(i) to the case #(.) -,-h
-
-
lP. =I .
One of the useful features of Lp convergence is that the Lp norms of Xn X define a sequence of constants whose order of magnitude in n may be determined, providing a measure of the rate of approach to the limit. We will say for example that Xn converges to X in mean square at the rate nk if Ijxf-.Y112= On-h, but not on -k). This is useful in that the scaled random variable nkxt A') may be non-degenerate in the limit, in the sense of having positive but finite limiting variance. Determining this rate of convergence is often the tirst step in the analysis of limiting distributions, as discussed in Part V below. -
-
18.5 Exnmples Convergence in pr. is a weak mode of convergence in that without side conditions it does not imply, yet is implied by, a.s. convergence and Lp convergence. However, there is no implication from a.s. convergence to Lp convergence, or vice versa. A good way to appreciate the distinctions is to consider cases where one or other mode of convergence fails to hold. dpathological'
18.15 Example Look again at 12.7, in whichAk 0 withprobability 1 1/n, and Xn = = n with probability 1/n, for n 1,2,3,.... A convenient model for this sequence where m is Lebesgue is to let ( be a drawing from the space (r0,1),m,1j,pz) variable and random define the measure, =
Xnlj
=
n,
)
-
e (0,1/a),
(18.31)
0, otherwise.
The set f : limuAkt) # 0) consists of the point (0J, and has p.m. zero, so that Xn -E't-A 0 according to (18.1).But Fl XnlP = 0 (1 1/rl) + t'l'ln = n#-1 lt will be recalled that this sequence is not uniformly integrable. lt fails to converge in 1 for every n. The Lp for any p > 1, but for the case p = 1 we obtain Exn) limiting expectation of Xn is therefore different from its almost sure limit. n -
.
.
=
The same device can be used to define a.s. convergent sequences wltich do not converge in Lp for any p > 0. It is left to the reader to construct examples.
Stochastic Convergence
289
18.16 Example Let a sequence be generated as follows: .Y1 1 with probability 1; xztxzl are either (0,1) or (1,0) with equal probability; (X4,X5,X are chosen from (1,0,0),(0,1,0),(0,0,1) with equal probability; and so forth. For k 1,2,3,... the next k members of the sequence are randomly selected such that one of them is unity, the others zero. Hence, for n in the range (kt/c 1) + 1, jkk + 1)J,P(Xn 1) 1/k, as well as FIXnlP 1/k for p > 0. Since k .-.A x as n .-+ x, it is clear that Xn converges to zero both in pr. and in Lp norm. But since, for any n, Xnwj 1 a.s. for infinitely many j, =
=
-
=
=
=
=
#(I-L I <
i.o.)
E,
=
(18.32)
0
for 0 f 6: f 1. The sequence not only fails to converge a.s., but actually converges with probability 0. Consider also the sequence fkl/rxn J whose members are either 0 or kllr in the ,
lk(k klrx I#) PVr-1 an d b suitable n y range (r'kk 1) + 1 z + 1)j. Note that E( I choice of r we can produce a sequence that does not converge in Lv fop > r. With 1 for al1 n, but as in 18.15, the sequence is not uniformly r 1 we have Ekxn) integrable. The limiting expectation of the sequence exists, but is different from the probability limit. I:a =
-
,
,
=
=
ln these non-uniformly integrable cases in which the sequence converges in Lj but not in f.,l+e for any 0 > 0, one can see the expectation remaining formally well-defined in the limit, but breaking down in the sense of losing its intuitive intepretation as the limit of a sample average. Example 18.15 is a version of the well-known St Petersburg Paradox. Consider a game of chance in which the player announces a number n e N, and bets that a succession of coin tosses will produce n heads before tails comes up, the pay-off for a correct predidion being E2 l+1 T h e probability of winning is 2 -/-1 so th e e xp ected winnings are f 1', that is to say, game' if the stake is fixed at f 1. The sequence of random winnings it is a generated by choosing n 1,2,3,... is exactly the process specified in 18.15. ao Xn .
,
Efair
=
If n is chosen to be a vel'y large number, a moment's reflection shows that the probability limit is a much better guide to one's prospective winnings in a t'inite number of plays than the expectation. The paradox that with large n no one would be willing to bet on this apparently fair game has been explained by appeal to psychological notions such as risk aversion, but it would appear to be an adequate explanation that, for large enough n, the expectation is simply not a practical predictor of the outcome.
18.6 Laws of Large Numbers n-1Z7=1Xf.Suppose that E(Xt) = Let (m)Tbe a stochastic sequence and definex g,?and n -1Z7=1p g, w ith lp,1 < pa,' this is trivial in the mean-stationary case in which gf g, for al1 t. In this simple setting, the sequence is said to obey the weak law of large numbers (WLLN) when kn -CC..Ag,, and the strong law of large numbers (SLLN) when kn p,. These statements of the LLNS are standard and familiar, but as characterizations =
--)
=
-.62-.4
Ihe fzzw of fzzr'd Numbers
290
of a class of convergence results they are rather restrictive. We can set jt 0 with no loss of generality, by simply considering the centred sequence (A) g,flq; centring is generally a good idea, because then it is no longer necessary for the time average of the means to converge in the manner specified. We can quite easily have n -1E7=1h4 cxa at the same time that n-1E: 1(AX)) w) 0. In such cases the 1aw of large numbers requires a modified interpretation, since it ceases to make sense to speak of convergence of the sequence of sample means. =
-
--)
-
--
=
More general modes of convergence also exist. It is possible that kn does not that there exists a converge in the manner specified, even after centring, but -1 (Jn ):' such that an 1 cxa and an X1=jXt-...+0. Results sequence of positive constants will and others too, in a fully general array subsume possibilities, below these ,21 k l n=1 i s a triangular stochastic array with formulation of the problem. lf f(AICJ (1k1':.1 an increasing integer sequence, we will discuss conditions for =
Sn
kn
=
Txnt--EC->0. >1
(18.33)
A result in this form can be specialized to the familiar case with Xnt kn n, bu t there are important applications where the an-1 Xt g' and an greater generality is essential. We have already encountered two cases where the strong law of large numbers applies. According to 13.12, kn g' F(X1) when (m) is a stationary ergodic sequence and FI I < x. We can illustrate the application of this type of result by an example in which the sequence is independent, which is sufficient for =
=
-
=
-810
=
.11
ergodicity.
18.17 Example Consider a sequence of independent Bernoulli variables Xt with
1) P(Xt 0) :1.,that is, of coin tosses expressed in binary form (see 12.1). The conditions of the ergodic theorem are clearly satisfied, and we can E(Xt) j. This is called Borel's normal number conc lu de that n-1E) jA theorem, a normal number being defined as one in which Os and ls occur in its binary expansion with equal frequency, in the limit. The normal number theorem therefore states that almost every point of the unit interval is a normal number; that is, the set of normal numbers has Lebesgue measure 1. Any numberwith aterminating expansion is clearly non-normal and we know that a1l such numbers are rationals; however, rationals can be normal, as for example ,t, which has the binary expansion 0.01010101010101... This is a different result from the well-known zero measure of the rationals, and is much stronger, because the non-normal numbers include inutionals, and fonn an uncountable set. For eyample, anynumberwithabinary expansionof theformo.l 1:1 11:21 1&g11...where the bi are arbitrary digits is non-normal; yet this set can be put into 1-1 corin other words, with the points of respondence with the expansions the whole intelwal. The set of non-normal numbers is equipotent with the reals, but it none the less has Lebesgue measure 0. (a Pxt
=
=
=
=
--f..y
=
=
X
.bjbz%,...,
A useful fact to remember is that the stationary ergodic propery is preserved under measurable transformations', that is, if (A) is stationary and ergodic, so
Stochastic Convergence
291
is the sequence (#(.YJ)whenever g: R F.-+R is a measurable function. For example, 2 < x to be able to assert that n 11J=laY,2 we only need to know that E(X34 2 T he ergo dic theorem serves to establish the strong law for most stationary EX34. sequences we are likely to encounter', recall from j13.5 that ergodicity is a weaker property than regularity or mixing. The interesting problems in stochastic convergence arise when the distributions of sequence coordinates are heterogeneous, so that it is not trivial to assume that averaging of coordinates is a stable procedure in the limit. Another result we know of which yields a strong law is the mmingale conver-1 gence theorem (15.7),which has the interpretation that an Z7=1.V () wjwnever f Z7.1A)) is a submartingale with FI17=1A I < cxl uniformly in n, and an oo. This particular strong 1aw needs to be combined with additional results to give it a broad application, but this is readily done, as we shall show in j20.3. But, lest the law of large numbers appeyr an altogether trivial problem, it might also be a good idea to exhibit some cases where convergence fails to occur. --t.!..y
-
-s...,
--->
18.18 Example Let (m) denote a sequence of independent Cauchy random variables w ith characteristic function 9xf(X) e- lXI for each t (11.9).It is easy to I.I verify using formulae (11.30) and (11.33) that #:a(D= e -aI,l/,, e xxord. ing to the inversion theorem, the average of n independent Cauchy variables is also a Cauchy variable. This result holds for any n, contradicting the possibility that Xncould converge to a constant. l:a =
-
=
.
18.19 Example Consider a process l
Xt
lgsz.
=
Xt- I +
=
jthzt,
t
1,2,3,
=
(18.34)
...
J=1
with Ab 0, where IZ/J7is an independent stationary sequence with mean 0 and variance c2, and (% 17is a sequence of constant coefficients. Notice, these are indexed with the absolute date rather than the lag relative to time /, as in the linear processes considered in j14.3. For m > 0, =
COv(X/,Xtwm)
=
VartXJ c21
t
=
x=l
Fs. l
(18.35)
< x, i n this case the effect fw-bounded requires Zyzzlvs * with of the innovations declines to zero t and Xt approaches a limiting random Without the square-summability assumption, Vartl An X, variable say. example of the latter case is the random walk process, in which vs 1, all s. Since Cov(xY1,m) for every /, these processes are not mixing. Xnhas zero mean, but
For
(X,)Tto be uniformly
.
-->
=.
=
=
W1c2
n
1 7'vartx,l+
vara) nw =
>1
n
277 >2
J-l
yvartxp
(18.36)
.
j=1
c27,=1vl; otherwise Vartxa) x. In either then limn-oxvartk) fixed limit, being either stochastic fails the fYa to converge to a ) sequence case
If
17
J=
lvy?
<
-->
c,o,
=
292
The fww of zzr'd Numbers
asymptotically, or divergent. n These counter-examples illustrate the fact that, to obey the law of large numbers, a sequence must satisfy regularity conditions relating to two distinct factors: the probability of outliers (limitedby bounding absolute moments) and the degree of dependence between coordinates. ln 18.18 we have a case where the mean fails to exist, and in 18.19 an example of long-range dependence. ln neither case can Xn be thought of as a sample statistic which is estimatinj a parameter of the underlying distribution in any meaningf'ul fashion. ln Chapters 19 and 20 we devise sets of regularity conditions sufficient for weak and strong laws to operate, constraining both characteristics in different configurations. The necessity of a set of regularity conditions is usually hard to prove (the exception is when the sequences are independent), but various configurations of mixing and fw-boundedness conditions can be shown to be sufficient. These results usually exhibit a trade-off between the two dimensions of regularity; the stronger the moment restrictions are, the weaker dependence restrictions can be, and vice Versa.
One word of caution before we proceed to the theorems. In j9. 1 we sought to motivate the idea of an expectation by viewing it as the limit of the empirical average. There is a temptation to attempt to defne an expectation as such a limit', but to do so would inevitably involve us in circular reasoning, since the arguments establishing convergence are couched in the laguage of probability. The aim of the theory is to establish convergence in particular sampling schemes. lt cannot, for example, be used to validate the frequentist interpretation of probability. However, it does show that axiomatic probability yields predictions that accord with the frequentist model, and in this sense the laws of large numbers are among the most fundamental results in probability theory.
19 Convergence in Lp Norm
19.1 Weak Laws by Mean-square
Convergence
This chapter surveys a range of techniques for proving (mainly)weak laws of large numbers, ranging from classical results to recent additions to the literature. The common theme in these results is that they depend on showing convergence ih fv-norm, where in general p lies in the interval (1,21.Initially we consider the 2. The regularity conditions for these results relate directly to the case p variances and covariances of the process. While for subsequent results these moments will not need to exist, the faz case is of interest both because the conditions are familiar and intuitive, and because in certain respects the results available are more powedl. and variConsider a stochastic sequence (m)T,with sequence of means (g.r1T, 2 ances (cf )7. There is no loss of generality in setting g,f 0 by simply considering the case of faV w)7,but to focus the discussion on a familiar case, 1et us ---:: p, question, what initially assume Fn n -11J=1p< tfinite) and so consider therelation g)2 for 0:7 elementary conditions E(Xn An is are su fticient =
=
-
=
,
-->
-
Ek
g,)2
-
n
=
vartfnl+ EXn)
g)2.,
-
(19.1)
where the second term on the right-hand side converges to zero by detinition of p.. Thus the question becomes: when does Var(Xn) 0? We have -->
n
E ?z-1X(.2Vg,
Var(Xa) =
-
2
n
n
t- l
N(U 2XXcs /2
?z-2
=
+
>1
>1
(19.2)
,
ml
where (U Vartx and nts Cov(Xt,Xs). Suppose, to make life simple, we assume that the sequence is uncorrelated, with nts 0 for f # s in (19.2).Then we have the following well-known rsult. =
=
=
19.1 Theorem If
(.41Tis
uncorrelated sequence and *
77t
-2g2
,
< x
(19.3)
,
>1
-i+
then Xn
g,.
Proof
This is an application of Kronecker's 0. . implies VartYn) =
?z-21(U
lemma
(2.35), by
which
(19.3)
--
This result yields a weak 1aw of large numbers by application of 18.13, known conditton for (19.3)is that the as Chebyshev's theorem. An (amply)suftkient >nG
The zzw of fxzr'c Numbers
294
variances are uniformly bounded with, say, suprczff B < x. Wide-sense stationary 0(n-1). But since Vartxa) sequences fall into this class. ln such cases we have f1-8 /41), cl x is evidently permissable. If cl for Var(Xn)= a11we need is -lGl has O(f-1 and therefore converges by 2.27. terms of > 0, Z:=1f t is an unnecLooking at (19.2)again, it is also clear that uncorrelatedness essarily tough condition. It will suffice if the magnitude of the covariances can be suitably controlled. Imposing unifonn fu-boundebness to allow the maximum relaxation of constraints on dependence, the Cauchy-schwmz inequality tells us that lcsl S B for a11 t and s. Rearranging the formula in (19.2), =
-.->
-
-8)
,
Var(x-
n
1 n) n
+
-
n
xn x
p.2
c2, + g
<
+
n-1
+
2X
'2cn1
t,t-z
+
...
+
>3
n
x x Jc,.,-- )
u rnczl
n
>1
#
cf ,-I ,
,..1
w'
-
n
n
Xcl 2Xn
-
=
f=m+1
n-1
2
uE
n
-
(19.4)
mlBm,
-=1
where Bm supl ctz-ml and Bm ; B, all m k 1. This suggests the following variant on 19.1. =
,
< x 19.2 Theorem If (a))Tis a uniformly Q-boundedsequence, and Zm''Q1r?:-1#,,j where Bm = suprl nzt-ml then kn -V2-+g. ,
Proof Since n mjln < 1, it is suftkient by (19.4)to show the convergence of llnlkmcjBm n to zero. Tlzis fo llows immediately from the stated condition and . Kronecker's lemma. . -
-1
A sufficient condition, in view of 2.30, is Bm Ottlog m) -1- ), g s (). a very mild restriction on the autocovariances. There are two observations that we might make about these results. The first is to point to the trade-off between the dimensions of dependence and the growth of the variances. Theorems 19.1 and 19.2 are easily combined, and it is found that by tightening the rate at wlzich the covariances diminish the variances can grow faster, and vice versa. The reader can explore these possibilities using the rather simple techniques of the abote proofs, although remember that the Int,t-mI will need to be treated as growing with t as well as diminishing with m. Analogous trade-offs are derived in a different context below. which depends on these factors, can be The order of magnitude in n of Var((Xk), thought of as a measure of the rate of convergence. With no correlation and bounded variances, convergence is at the rate n-1/2 in the sense that Vartx'nl 0(n -1). but from (19.4),Bm Om-&) implies that Var(Xn) 0(n-%. lf convergence rates are thought of as indicating the number of sample observations required to get kn close to g, with high contidence, the weakest sufficient conditions evidently yield convergence only in a notional sense. lt is less easy in some of reqllltq below to link exolicitlv the rate of tlafamoro oonernl convemence with the =
,
=
,
=
=
.
Convergence in Lp Ntpn'rl
295
degree of dependence and/or nonstationarity; this is always an issue to keep in mind. Mixing sequences have the property that the covariances tend to zero, and the mixing inequalities of j14.2 gives the following corollary to 19.2. 19.3 Corollarylf
f.&l'7is either (i)uniformly Q-boundedand uniform
mixing with
X
l/2 y-lrpz-ljl
<
(19.5)
x
rrl=1
or (ii) unifonnly
for 8
fo-bounded Tm rn=1
1
W
>
/(2+)
0, and song
mixing with
(19.6)
< x,
then Xn -il.> p.. /2 Proof For part (i), 14.5 for the case r 2 yields the inequality Bm ; lB(m1 y.or /(z+y a r = 2 + 8 yields Bm S 6 IImII2+8a,,, part (ii), 14.3 for the case p xoting lIAll22+a, the conditions of 19.2 are satisfied in either case. . that B S =
.
=
.
O((1og ra)-2-f) for any 6: > 0. For A sufticient condition for 19 3(i) is (),,, -(1+V)(l+e)) Ottlog m) 19.3(ii), m sr : > () is suftkient. In the size terminology of j14.1, mixing of any size will ensure these conditiops. The most significant cost of using the strong mixing condition is that simple existence of the variances is not sufticient. This is not of course to say that no weak 1aw exists for Q-bounded strong mixing processes, but more subtle arguments, such as those of j19.4, are needed for the proof. =
=
19.2 Almost Sure Convergence by the Method of Subsequences Almost sure convergence does not follow from convergence in mean square (a counter-example is 18.16), but a clever adaptation of the above techniqueg yields a result. The proof of the following theorems makes use of the method of subsequences, exploiting the relation between convergence in pr. and convergence a.s. demonstrated in 18.6. Mainly for the sake of clarity, we first prve the result for the uncorrelated case. Notice how the condittons have to be strengthened, relative to 19.1.
19.4 Theorem lf fXf lTis uniformly fmbounded and uncorrelated, Xn -1.-y
g,. n
A natural place to start in a sufticiency proof of the strong 1aw is with the convergence pal't of the Borel-cantelli lemma. The Chebyshev inequality yields, under the stated conditions,
#(IXn FnI > -
:)
S
vart/n) s S a
z
(19.7)
for B < x, Fith the probability on the left-hand @'side going to zero with the . x. One approach to the probtem of bounding the quantity right-hand side as n -->
The fuzw of fzzr'c Numbers
296
#( IXn Fnl > E, i.o.) would be to add up the inequalities in (19.7)over n. Since the partial sums of 1/n form a divergent sequence, a direct attack on these lines 1 64 and we can add up the Juhsequence of '=' does not succeed. However, Za.1a = for 1,4,9, 16,..., probabilities in (19.7), n the as follows. -
-2
=
Proof of 19.4 By
,
.
(19.7), XP
IXn2Fn2I > e)
n2
1.64/<
f
-
(19.8)
x.
Ea
Now 18.2(i) yields the result that the subsequence (X2,n e N ) converges a.s. The proof is completed by showing that the maximum deviation of the omitted terms fkomthe nearest member of (Xn 2) also converges in mean square. For each n define
on2
lkk knzI
max
=
(19.9)
-
nl Sk<(rl+1)2
and consider the variance of vartx ) (1/n2)).1@ tends =
Of
arfangement
Dnz. Given the assumptions, the sequence of the monotonically to zero. For nl < k < (n + 1)2 rethe terms produces /
'k knz
2
n
(-,
=
-
l
j
knz+
1
-
k
Xt, X p=n2+1
(19. 10)
and when the sequence is uncorrelated the two terms on the right are also uncorrelated. Hence
vart#k Xn c)
l
=
-
n
-
2 2vart#nz)+
=B
-
-u
1
n2 B 1
-
Vartoazl cannot exceed the last tenn in
7) (rl-2-(n+
1)-2)
<
n2
-
< B
z&
nk
(k
+
k u n
-
.s.
1
n
-
nk
k
) X o'zj
v
2
< 1
2
1
>n2+1
2
)-s t,z 1
(a +
1)z
.
(19.11)
(19.11),and
F''l@-2(n+ 1)-2) -
=
1,
(19.12)
n
so the Chebyshev inequality giyes XpDnl n2
>
#
o Eu <
(19.13)
< x,
and the subsequence fDnz, n e INJ also converges a.s. Dn2k l(V-knl1for any k between n l an d (n+ 1)2 and hence, by the triangle inequality, ,
lkk g I s lknz Fn2I + Ikk knzl + lFl Fn2I -
-
-
-
.'. .:.J yyj;y''s ) qjg 5'?tj(j ?..Ey. jjjjy . . 1.
.:y '(''b
L'77;:'.. .'
.
, .
'.
x
.
Jl .'
.
. .
ggg
Convergence in Lp Notm
I+ u Iknz Fn2
(19.14)
Igk Fa2I
Dnz +
-
=
.
The sequences on the majorant side are positive and converge hence a.s. to zero, nl X But holds 8 for 1)2 does < k < (a + their sum. 1, SO (19.14) (p2,n so sr that k ranges over every integer value. We must conclude that kn l1' K --632''
We can generalize the same technique to allow autocorrelation. If (A)1' is uniformly Q-bounded, and
19.5 Corollary
B*
XBm
=
<
(19.15)
x,
m=1
where Bm
=
sup Int,t-mI then X- -E-> g.. u ,
Note how much tougher these conditions are than those of 19.2. lt will suffice -)-1-) for > 0 Instead of, in effect, having the here for Bm = Om -1(lo g merely decline to zero, we now require their summability. autocovariances .
Proof of 19.5 By (19.4),Var(Yu) < CB+ lB*)ln and hence equation themodified form, XP n2
Instead of Vartxk
(19.11) knz)
=
-
Iknl ik I >
1.64(,+zB.)
e) f
-
1 k
X j. p=n2+1
Xt
1 n
-
-
2
--
2 n Vart-Ynzl + 1 1 v = & P 2
k
1-n2-1
k
cl
Xzx.l
2
+
X X
Gt,t-m
,s=1
panaz
Nn
l-1
k
2 2 n - k 1 k -
j6)
Xnz
-
-
.
in
out and tnking expectations,
-
-
(lq
< x.
E2
we have, on multiplying
Var
(19.7)holds
X X wt-m f=nN.l
-
(19.17)
.
m=l-n2
The first term on the right-hand side is bounded by (1 n 2/k)2(#+ 2#*)/r the 2s+ 2 + 2 2 second by k n )(# + IB 4Ik and the third (absolutely)by 2(1 n lk) Adding together these latter tenns and simplifying yields -
,
-
-
.
,
Vartxk Xnz)< -
--j
1
1 1 w n+ ni Note, (1 n 2ln + 1)2)2 (19.13)we cgn write
1
-
y
-
1)-L
-
=
-
B
B
+
+
On-l)
1
2
2 ,
1
u y
+
1
1
=
Z
-
-
2 1 n -
-
n
+
1)d
+
2
B
--
1
+
n2
-
n
+
1)z
2
B
+ .
(19.18)
so the term in #* is summable. ln place of
The ww of fzzrgc Numbers
298
XpDnz
>
s) <
A-I#*
B+
<
Ez
n2
(19.19)
x,
where K3 is a finite constant. From here on the proof follows that of 19.4. w Again there is a straightforward extension to mixing sequences by direct analogy with 19.3.
19.6 Corollary If
(X/ITis eitherti)
unifonnly
Q-boundedand unifonn
mixing with
X
y')j1/2 <
(19.20)
x
tl=1
or (ii) uniformly
Qu-bounded for 8
>
0, and strong mixing with
X
/(2+) Xcu -=1
then kn
<
(19.21)
x,
-.6.'3..-h
0. n
Let it be emphasized that these results have no pretensions to being sharp ! They are given here as an illustration of technique, and also to define the limits of this approach to strong convergence. In Chapter 20 we will see how they can be
improved upon.
19.3 A Mmingale
Weak Law
We now wantto relax the requirement of tinitevariances,andprove fv-convergence for p < 2. The basic idea underlying these results is a truncation argument. Given a sequence tXf )Twhich we assume to have mean 0, define Ff 1( Ix,1ss)Xr, which equals Xt when IA l f B < c'o, and 0 otherwise. Letting Zt Xt Ff, the -EYt4 by construction, and kn Ys+ Zn. component' of Xt, notice that Ezt) Since F, is a.s. bounded and possesses al1 its moments, arguments of the type used in j19.1 might be brought to bear to show that X g,y (say).Some other An obvious technique approach must then be used to show that Zn-Z'..y g,z is to assume uniform integrability of ( IXrIPI In this case, supffl zrlr can be made as small as desired by choosing B large enough, leadin-y(vfcthe Minkowski inequality, for example) to an w-convergence result for Zn. A different approach to limiting dependence is called for here. We cannot assume that J'f is serially uncorrelated just because Xt is. The serial independence assumption would serve, but is rather strong. However, if we let Xt be a martingale difference, a mild strehgthening of uncorrelatedness, this property can also be passed on to l'r, after a centring adjustment. This is the clever idea behind the next result, based on atheorem of Y. S. Chow (1971).Subsequently (seej19.4) the m.d. assumption can be relaxed to a mixingale assumption. We will take this opportunity to switch to an array formulation. The theorems are easily specialized to the case of ordinary sample averages (see j18.6), but in subsequent chapters, array results will be indispensable. =
'tail
=
-
=
=
-i2=
-g,y.
=
.
Convergence in Lp Ntprrrl
299
19.7 Theorem Let t-L,,?L,)be a m.d. array, tcnfla positive constant array, and t/cnl an increasing integer sequence with kn x. 1f, for l S p f 2, '1'
(a) t lXntlcnt6l'l is
uniformly integrable,
kn
(b)limsup Xcn,< n-yx
kn
cn2f
(c) lim77 n-boo
0,
=
>1
--f.-
thenEylAk, t
and
x,
>1
0. o
of this result is where Xnt Xtlan, where fA,F,) with m.d. 5nt 5t and ftzr,lis a positive constant sequence. This is a sequence deserves stating as a corollary, since the formulation can be made slightly more The leading specialization
=
=
transparent. is a m.d. sequence, and fbt) lJnl, and (/cnlare 19.8 Corollary Suppose (.X',,?;,J7' constant positive sequences with an cxa and kn x, and satisfying IPl is uniformly integrable, 1 < p !-; 2, (a) ( kxtlbt kn O(Jn), and (b) ,
'1'
'1'
'/bt
=
>1
kn
b2 (c) 77 >1 ,
-1jg#g
then an
t 1J(t
=
t?(t);
.(eo g.
Proof Immediate from 19.7, defining Xnt
Xtlan and cnt
=
=
btlan. w
Be careful to distinguish the constants an and kn. Although both are equal to n in the sample-average case, more generally their roles are quite different. The case with kn different from n typically arises in arguments, where the array coordinates are generated from successive blocks of underlying sequence coorfor a e (0,1) (g.:jdenoting the largest integer dinates. We might have kn Erltxl 1-a x) where block does not exceed grl j. For an application of of length the below a this sort see j24.4. Conditions 19.8(b) and (c) together imply an 1 x, so this does not need to be separately asserted. To form a clear idea of the role of the assumptions, it is helpful to suppose that bt and an are regularly varying functions of their arguthat the conditions are observed if bt t 15 ments. It is easily verified by 2.271+F for f'J> and an log n fpr f' by choosing an n In for any ;$k a11 f, kn yields for an n particular,setting bt 1 tblocking'
=
-
-1,
-1,
-1.
-
-
=
=
IIXkII, 0. -->
=
=
(19.22)
will automatically satisfy condition (a), and condition (b) choosingan /,z1/7, willalso hold when bt Ot ). On the other hnd, a case where the conditions =
=
The Izzw of wr'd Numbers
300
l-1
t
1 and, for t > 1, bt Zy=l)y= 2 In this case condition (a) fail is where :1 2 imposes the requirement bn = Oan), so that bn2 O (Ju), contra dicting condition (b). The growth rate of bt exceeds that of t 5 for every j3> 0. =
=
.
=
Proof of 19.7 Uniform integrability implies that
supftlA%,/cn,lP1f n
Ix,,,/o,I>>))
0 as M
---)
x.
---
,1
One may therefore
tind, for
tr
SuP(1IXn,1( n
0, a constant Bs
>
<
such that
x
Ilp/cn/j f s.
(19.23)
lxa,I>s:cn,)
,f
zn, Xnt Fn,. Then since Exnt I@n,f-1) Fn,/-1)+ Znt F(Zn,I @n,,-l). F(l%,I
Define Ynt = Ak,1f Ixarlsseo/), and Xnt
=
Fn,
-
=
=
-
0,
-
By the Minkowski inequality,
177'-x-, j s jj(,--,-s(r--1v-,,-1)) j y-;k-
,
>1
+
,
,...1
Consider each of these right-hand-side
,,,,--1)) j:--7.)-j ti,,,/-str--rl
kn
'A''7 znt-fznrl
.
P
>1
(19.24)
terms. First,
t,(
))a
>--.'-j
u
9n,,-1))
(r-r-,(r,--lv,-,,-1.))
1/2
k
N
-F(I&fI@,,,,-1))2 = 77e'(l',,r ,-1 p TEYI
<
1/2
< #s
nt
1/2
k
X o2,
(19.25)
>1
>1
The tirst inequality in (19.25)is Liapunov's inequality, and the equality follows because ( Ynt ElYnt ITn,f-l)) is a m.d., and hence orthogonal. Second, -
kn
77 znt >1
kn
kn -
Ftznfl @n,,-l))
< P
+ 77Ilzsfslp 77lIF(z,,I@,,,,-1)11, >1
>1
kn
<
277 >1
kn
f leTcnt. lIz,,,1l,
(19.26)
>1
The second inequality here follows because f EE Izn,lP l@n,,-l)) F,,,,-l) 137 FIF(zn?I
=
Fl
zn,lP,
from, respectively, the conditional Jensen inequality and the law of iterated expectations. The last is by (19.3). lt follows by (c) that for ir > 0 there exists Ne k 1 such that, for n k Ns,
Convergence in Lp Ntprm
301
kn
2 < s-2:2 A-fnr f:
(19.27)
.
>1
Putting together
(19.24)with (19.25)and (19.23)shows kn
7'IXU;
,..a1 k 1 + 2,21c,,r
for n k Ne, where B this completes the proof. =
K Bz
(19.28)
p
<
x,
by con dition
(b). Since r is
arbitrary,
*
The weak law for martingale
19.9 Corollary
that
differences follows directly, on applying 18.13.
-/-% O k Under the conditions of 19.7 or 19.8, Z/21X,a u .
1 and set cnt 1/n and kn n as above, we get the result If we take the case p of faYt)is sufticient for convergence in probability of unifolnn integability that the sample mean Xn. This cannot be significantly weakened even if the martingale difference assumption is replaced by independence. If we assume identically distributed coordinates, the explicit requirement of uniform integrability can be dropped and Al-boundedness is enough; but of course, this is only because the uniform property is subsumed under the stationarity. You may have observed that (b) in 19.7 can be replaced by =
(b')
limsup-lxd,
=
=
kn <
x.
/=1
n--x
It suftices for the two tenns on the majorant side of (19.24)to converge in Lp, and the cr inequality can be used instead of the Minkowsk inequality in (19.26)
to obtain
kn
E
77(z,,,-F(z,,,I@n,,-l)
kn
p
K
c,(2k,,)P-1>7d,.
/=l
(19.29)
>1
However, the gain in generality here is notional. Condition (b') requires that limsuptr,-ox/cjc'i< x, and if this is true the same property obviously extends to /7 and an nY, flkcuf). For concreteness, put cnt = btlan as in 19.8 with bt ?7t' for x > 0, note that the where j3and y can be any real constants. With kn majorant side of (19.29)is bounded if a(1 + I$) < 0, independent of the value of p. This condition is automatically satisfied as an equality by setting an = k b t, but note how the choice of an can accommodate different choices of kn. Xt2l None the less, in some situations condition (b)is stronger than what we kmow to be sufficient. For the case p 2 it can be omitted, in addition to weakening the martingale difference assumption to uncorrelatedness, and uniform integrability to simple fu-boundedness. Here is the an'ay version of 19.1, with the conditions cast in the amework of 19.7 for comparability, although all they do is to ensure that the variance of the partial sums goes to zero. -
-
-
=
'y
-
The fww of fwr'c Numbers
302
19.10 Corollary lf (-L,) is a zero-mean stochastic an'ay with Exntxns) y: s, and (a) (Xntlcntl is uniformly Q-bounded, and
=
0 for t
kn
(b) 1im n--yx
thenEtzlxn,
7)c2a,
=
0,
>1
-1J-+
0. c!
19.4 A Mixingale Weak Law To generalize the last results from martingale differences to mixingales is not too difficult. The basic tool is the telescoping series' argument developed in j16.2. The array element Xnt can be decomposed into a tinite sum of martingale differences, to which 19.7 can be applied, and two residual components wlch can be treated as negligible. The following result, from Davidson (1993a),is an extension to the heterogeneous case of a theorem due to Andrews (1988).
19.11 Theorem Let the al'ra fAk,,V)=-xbe a Al-mixingale with respect to a constant array (cnf) lf (a) Lxntlcnttis uniformly integrable, .
kn
(b) limsup n--yoo
X cnt <
kn
(c) lim
x,
and
/=1
1 cnt
=
0,
>1
n-yx
where kn is an increasing integer-valued function of n and kn
-#.
'1'
x,
then
0. a
k xnt X/:1
There is no restriction on the size here. It suffices simply for the mixingale coefficients to tend to zero. The remarks following 19.7 apply here in just the same way. In particular, if Xt is a Al-mixingale sequence and fxtlbtj is uniformly integrable for positive constants (/7f) the theorem holds for Xnt = Xtlan and cnt btlan where an l=jbt.Theorems 14.2 and 14.4 give us the corresponding results for mixing sequences, and 17,5 and 17.6 for NED processes. It is sufficient for, say, Xnt to be fv-bounded for r > 1, and V-NED, for p 2 1, on a a-mixing process. Again, no size restrictions need to be specified. Uniform integrability of fxntlcntkwill obtain in those cases where ll-kllris finite for r > 1 and each /, and the XED constants likewise satisfy dnt >) II-L,IIr. A simple lemma is required for the proof: ,
=
=
integrable for # 2 1, so is the
19.12 Lemma lf the array fxntlcntis uniformly atray fEt-jxntlcnt J for j > 0. Proof By the necessity
part of 12.9, for any ir
>
0 3 8
>
0 such that
Convergence in Ap Norm
sup n
sup
,t
jyl.L,/o,l
dP
303
(19.30)
< :,
where the inner supremum is taken over all E e 5 satisfying #(F) < 8. Since 5n,t-j Satisfying #(F) g 5, (19.30)also holds when the supremum is taken overF e %not-j For any such F, < .
JsI
XntlcntIdP
l jyk-j
jyI
Et-jxntlcnt IJ#,
XntlcntIdp
=
(19.31)
by definition of Et-j.), and the conditional Jensen inequality (10.18).We may accordingly say that, for s > 0 H 8 > 0 such that
sup n.t
jsI
Et-jxntlcnt6dP
sup
(19.32)
< :,
taking the inner supremum over E e %n,t-jsatisfying #(F) < 8. Since Et-jxnt is Fa,f-measurable, uniform integrability holds by the sufticiency part of 12.9. . Proof of 19.11 Fix an integer j and let kn
77 k-qxnt Et-vj-kxnt). ,--1 is a martingale, for each j.
Fn./
=
The sequence
-
l Ynj, @n,n+jJ';=1
l kjxnt is uniformly
integrable by
Et-vj-txntjlcntl
-
19.12, it follows by
(a) and
-f?.t-A
k Xnf Zf21
(19.33) kn
kn
X
19.7 that
as a telescoping sum. For any M k 1,
M- 1
Ynj
(b) and (c) and
0.
vnj We now express
Since the array
X
=
j=3-M
Et+M-tXnt- X Et-uxnt,
>1
(19.34)
>1
and hence M- 1
kn
Xxn, X =
kn
Ynj +
j=1-M
>1
kn
X xnt -
Etmu-txnt)
>1
+
Xk-uxnt.
(1.35)
>1
The triangle inequality and the fal-mixingale property now give M- 1
kn
E
kn
s 77 FIFnyl + X-tn, >1
kn
XF1
xnt -
>1
j=-M
M- 1
< 77 j=-M
.#'*8N
-
az'N z'
* .
.
.
.
.
Q'
..
XFI Et-uxnt'
+
>1
kn
2(::Xcnr.
E I1k/1 +
According to the assumptions, !
A',+v-lAkll
n
- .-
the secondmemberon .3
-
(19.36)
>1
!
- - - --
.-
..
FY
the right-hand side of (19.36) ,
*
*'
%
,
'
P'
UIT-
The fww of zzr'd Numbers
304
/ for M k Mv. By choosing n large enough, the sum of 2M- 1 terms on the righthand side of (19.36)can be made smaller than z1:for any finite M, by (19.33).So, <e when n is large enough. The theorem is by choosingMk Me we have 171Vzlxkjl proved since is arbitrary. : w now A comparison with the results of j19. 1 is instructive. In an fu-bounded process, the Q-mixingaleproperty would be a stronger fonn o'f dependence restriction than specified in 19.2, just as the martingale property the limiting uncorrelatedness simple uncorrelatedness. The value of the present result is the is stronger than substantial weakening of the moment conditions.
19.5 ytpproxinAable
Processes
There remains the possibility of cases in which the mixingale property is not because of a nonlinear transformation of a AP-NED easily established process which cannotbe shown to preserve the requisite moments for application of the results in j17.3. In such cases the theory of j 17.4 may yield a result. On the assumption that the approximator sequence is mixing, so that its mean deviations converge in probability by 19.11, it will be sufficient to show that this implies the convergence of the approximable sequence. This is the object of the following -perhaps
theorem.
(Wn/l
is a stochastic array and the 19.13Theorem Suppose that, for each m e !N, centredarray fhmnt Ehmnl satisfies the conditions of 19.11. If the array l-Lrl with respect to a constant array (t&,) and limsupu-,x is Al-approximable by fJln;.'11 .-P-f-+ 0. < hen k B f Xtltdnt Ztjaa, t n t -
,
=,
Establishng the conditions of the theorem will typically be achieved using 17.21, by showing that Xnt is far-bounded for r > 1, and approximable in probability on hmntfor each m, the latter being pz-order lag functions of a mixing array of any
size.
Proof Since kn
kn
XXn,
f
>1
kn
kn
X xnt -
X
hmnt, +
>1
hmnt-F('J)
>1
by the triangle inequality, we have for
XF('lf)
(19.37)
>1
0
kn
kn
#
>
+
( j j (j j ) + (jX.j j !a) (!X.j I (a) X.jxn ,
X.jxn
uP
>
,
-
,) h:7
>
kn
kn
P
la
ht
-
F(I))
>
+
P
F(7,)
>
(19.38)
by subadditivity, since the event whose probability is on the minorant side implies at least one of those tnn the majorant. By the Markov inequality,
Convernence in fg Norm kn
P
X tXnr
-
hmntj
>
f
j'
>1
3E
g
kn
X xnt Jln >1
kn
f
3
-,
m
-
< 3 XFlkr F
305
u,I
m
-
>1
kn
X dnt v
m
(19.39)
.
>1
#( IXktzkEhmnt) I > &3) is equal to either 0 or 1, according to whether the non0 and stochasticinequality holds or does not hold, By the fact that Exnt) Al-approximability, =
IEhmntt I
=
IExnt)
-
E
(Mnf)
I < FIXnt
-
hmnt I < dmvm,
(19.40)
and hence kn
kn
kn
XF(''',,rl
<
>1
< Bvm. 77IEhmn 1,< Tdntvm
(19.41)
>1
>1
We therefore find that for each m e EN kn
limsup# n-+x
S
Txnt
>
>1
3B + limsup # 8 vm n--)x
=
3#
v,u
+
1(svs,>/a),
kn
X >1
m m Ehntjj hnt -
>
8
y
+
1(svm>&3)
(19.42)
by the assumption that hmntsatisfies the WLLN for each m e EN.The proof is x. completed by letting m w --
20 The Strong Law of Large Numbers
20.1 Technical Tricks for Proving
LLNS
In this chapter we explore the strong 1aw under a range of different assumptions, omindependentsequencestoner-epochdependentfunctionsof mixingprocesses. Many of the proofs are based on one or more of a collection of ingenious technical lemmas, and we begin by studying these results. The reader has the option of skpping ahead to j20.2, and referring back as necessary, but there is something to be said for forming an impression of the method of attack at the outset. These theorems are found in several different versions in the literature, usually in a form adapted to the pmicular problem in hand. Here we will take note of the minimal conditions needed to make each trick work. We start with the basic convergence result that shows wy maximal inequalities (for example, 15.14, 15.15, 16.9, and 16.11) are important.
20.1 Convergence Iemma Let (A)T be a stochastic sequence on a probability space (f,T,#), and 1et Sn Z:=IX, and So 0. For (t) e f, let =
=
Mll)
inf sup ISjb
=
I
Smfk
=
m j>m
lf P(M
>
e)
=
0 for all E
0, then Sn
>
-6-->
(20.1)
.
S.
Proof By the Cauchy criterion for convergence, the realization (,%())) converges u%1 ; e:for all j > m, for a11 : > 0; in other if we can find an m such that ISj words, it converges if M4(t)) < c, for al1 : > 0. . -
This result is usually applied in the following way.
20.2 Corollary Let lc,)Tbe a sequence of constants, and suppose there exists p 0 such that, for every m 0 and n > m, and every e > 0, P max lSj 5k1 -
>
e f
mmusn
wjct
where K is a t'inite constant. If X7
-
K
E
< x,
x-xn
X
c(,
(20.2)
>,1+1
then Sn
tcf) is summable
-16+
lim---jxzOf
S. =m+1c(
it follows by 2.25 that be the r.v. in (20.1). By definition, M ; supom jSj Sm1for any m
Proof Since
-
P(M
>
e)
lim m,-nt
sup ISj SmI > er -
j>m
>
0. Let M 0, and hence =
>
aw of fxzr'c Numbes
The Strong
<
r
--
307
=
Xc
lim
erpm--
pt
0
=
(20.3)
,
/=,n+1
where the final inequality is the limiting case of (20.2).20.1 completes proof. w
the
Notice how this proof does not make a direct appeal to the Borel-cantelli lemma to get a.s. convergence. The method is closer to that of 18.3. The essential trick with a maximal inequality is to put a bound on the probpbility of all occurrences of a certain type of event as we move down the sequence, by specifying a probability for the most extreme of them. Since S is tinitealmost surely, Xn 0 is an instant corollary of 20.2. Howalso used result the in a more subtle way in conjunction with can be ever, Kronecker's lemma. lf :=1 F, converges a.s., where f 1',1 (Xtlat and (th) is a 0 This sequence of positive constants wiyh an x, it follows that an-1:7 xt much of weaker L1=jXt condition itself. the of Most than is course a convergence applications feature at t, but the more general formulation also has uses. There is a standard device for extending a.s. convergence to a wider class of seqtlences, once it has been proved for a given class: the method of equivalent sequences. Sequences (A)';' and f F,):' are said to be equivalent if .-61-:
=
-j-.
,1-
*
.1
.
=
X
X Pxt
y,
J',)
<
(20.4)
x.
>1
0. By the first Borel-cantelli lemma (18.2(i)),(20.4)implies P(Xt l, i.o.) of only probability measure zero are there more than a ln other words, on a set finite number of f for which X,((J)) # F,4)).:y:
=
L1=j(Xt Fr) converges a.s.
20.3 Theorem lf Xt and Fr are equivalent,
-
Proof By definition of equivalence and 18.2(i) there exists a subset C of f1, with #(f C) 0, and with the following property: for all (J) e C, there is a finite nn((l))such that X,()) iX(9)for t > ,7(,(). Hence -
=
=
n0(tl9
n
X (m(t,))r/t,))) X (-Y/)) =
-
-
F,(t,))), V n k
/70(t,)),
>1
>1
and the sum converges, for a11 (l) e C.
.
The equivalent sequences concept is often put to use by means of the following
thorem.
20.4 Theorem Let (Al7 be a zero-mean
random
sequence satisfying
X
XE >1
Ix,IPla;
<
(20.5)
x
for some p k 1, and a sequence of positive constants the indicator function 1( Ix,Isn)()),
ttzf) Then, .
putting 17 for
lhe f-aw of fwr'd Numbers
308
X#(I#,l
>
<
a
(20.6)
x,
>1
lF(Al7ll 77 >1
lat <
(20.7)
=,
and for any r k p, X
77F(1A1
rliyaL<
>1
(20.8)
a
x.
result may be apparent. The indicator function is used to truncate a sequence, replacing a member by 0 if it exceeds a given absolute bound. The ratio of the tnlncated sequence to the bound cannot exceed 1 and possesses a1l its absolute moments, while inequality (20.6)tells us that the tnmcated sequence is equivalent to the original under condition (20.5).Proving a strong 1aw under (20.5) can therefore be accomplished by proving a strong 1aw for a truncated sequence, subject to (20.7)and (20.8). The idea behind tls
Proof of Theorem
20.4 We prove the following three inequalities:
#( IXt l
> J,)
F(1
-
11) 1/))/tz(
/ E I.Y,1#(1 < Ekxt6pllal;. -
Here the inequalities are because lXt is non-negative, respectively. F(l.Y,If71J)
IExtsll IIat
=
IF(.&tl
s
E
lx,lP(1
IP/Y>
l for
(.f) G
( IXt I >
tz,l,
and becaus
1/) Ilat
-
< F(1 Al (1
(20.9)
1/))/J,
-
-
j.;))Ia;
< E6xt3p4lal;.
(20.l0)
1/)). The -F(m(l 0, hence EXt$1) The equality in (20.10) is because Ex first inequality is the modulus inequality, and the second is because on the event lat? l 1Xt l > J,l, ( 11,1 IAIlat for p 1. Finally, by similar arguments to the above, =
=
F(lAl
for p rsLllalS F(I.)tI7'1tl)/t4 < E Ix,I#)/t7f.
The theorem follows on summing over t.
-
< r
(20.11)
.
There are a number of variations on this basic result. The tirst is a version for martingale differences in terms of thseone-step-ahead conditional moments, where
The Strong fzzw of fwtzr'd Numbers
309
the weight sequence is also allowed to be stochastic. The style of this result is appropriate to the class of martingale limit theorems we shall examine in j20.4, in which we establish almost-sure equivalence between sets on which certain conditions obtain and on which sequences converge. 20.5 Corollary Let f.X),Tf)be a m.d. sequence, let ( B$) be a sequence of positive T/-l-measurable r.v.s, and for some p k 1 let o
D
=
(t):
< XF(I -Y,IP l@,-1)((,))/M(t,))
>.
e
x
(20. 12)
>1
Also define o
D1
=
):
X#(Iml>
<,1 @,-1)())
<
e ?/
x
(20. 13)
>1 r
Dz
=
):
X IF(m11I1,-1)()
< x I/WXt,))
(20.14)
e 5
>1
D?
=
:
x XF(IA)1 >1
and 1et D' #(D')
=
=
rll
IT,-1)())/M() <
D1 r'n D1 ch D3. Then P(D
-
D')
=
e @,
x
0. ln particular, If #(D)
(20. 15) =
1 then
1.
Proof It suffices to prove the three inequalities (20.9),(20.10), and (20.11) for the case of conditional expectations. Noting that Ext ITf-l) 0 a.s. and using the fact that Wf is Ff-l-measurable, a11 of these go through unchanged, except that the conditional modulus inequality 10.14 is used to get (20.14). lt follows E D is in D'. . that almost every =
Another version of this theorem uses a different truncation, with the truncated variable chosen to be a continuous function of Xt; see 17.13 to appreciate why this variation might be useful.
20.6 Corollary Let t.Yf)Tbe a zero-mean random sequence satisfying (20.5)for p k 1. Define Xtlat,
l'f
=
Xtlz/tzf+
(.X)/I.X)I )(1 -
17)
=
IXt I f
at
1,
Xt > at
-1,
Xt
<
(20.16)
-at.
Then, fr
IEj'tb I 7-7 >1
(20.17)
The zzw of fwtzr'd Numbers X
XF1
'
1,,1
<
(20. 18)
p.
r
=,
,--1 Proof Write hat to denote atxtlk Xt I Inequalities are adapted as follows. .
(20.10)and (20.1 1)
of 20.4
f
IEtt l
lEXt1.1 + (1 llat = lExt (+tz,))(1 1t) Ilat < E IXt 1(1 Skllat + E l 1 1/ j 17))/J( + #( lX/I > < E( I.)1P(1 S 2F( IXt l#)/J(. =
-
-
12)(+J) -
-
-
at4
-
(20.19)
The second equality in (20.19) is again because Ext) = 0. The first inequality is an application of the modulus inequality and triangle inequalities in succession, and the last one uses (20.9).By similar arguments, except that here the cr inequality is used in the second line, we have F41 1'Jl
r)
< F!.Y,1t
+
(1 1tJ)(+tz 1rlaL -
< 2r-1(sj xgl/j S
r)
sj (1 1) j 2r-1(F( + P IXt I > IXt3P3L)lal; zre' jxt1pjtai. rfa;
+
-
J,))
The theorem follows on summing over t as before.
for p ; r
(20.20) .
Clearly, 20.5 could be adapted to this case if desired, but that extension will not be needed for our results. The last extension is relatively modest, but permits summability conditions for nonns to be applied.
20.7 Corollary (20.6),(20.7),(20.8),(20.17),and (20.18)a1l continue to hold if (20.5)is replaced by < 77F(lAI#)1/t#z7A
(20.21)
cxa
>1
for any q 2 1. Proof The modified
#(I#,l
>
forms of a
(20.9),and
of
(20.19)and (20.20)(say) are
l#)/Jf)1& K P IA1 > h)1& S E l-Yf
pyaqlllq lE(Yt4I S lE(Yt4I3Iq < J,1&(.s(jxtj
(20.22) ,
(20.23)
The Strong
zzw of r)l&
< (Fl yf 1
e'Ii',1r
311
zzr'c Numbebs
< zr/qE
pyalklq
Ix,1
(20.24)
1
where in each case the first inequality is because the left-hand-side member does not exceed 1. w For example, by choosing p = q the condition that the sequence summable is seen to be suftkient for 20.4 and 20.6.
( llxf/tzf lIp1is
20.2 The Case of lndependence The classic
results on strong convergence are for the case of independent sequences. The following is the three series theorem' of Kolmogorov:
20.8 Three series theorem Let (aV)be an independent sequence, and Sn = k1=3Xt. Sn S if and only if the following conditions hold for some fixed a > 0: -61-:
oo
X#(l-,I
a) <
>
(20.25)
=,
>1
77A'(1(Ix,Isc)-Y,)
<
(20.26)
x,
>1 X
XVar(1lIx,I
<
=.
(20.27)
n
,-1 5') is the same as the event (5'n+1 S'J convergence is Since the event (& invariant to shift transformations. lt is a remote event by 13.19 and hence in independent sequences occurs with probability either 0 or 1, according to 13.17. 20.8 gives the conditions under which the probability is 1, rather than 0. The theorem has the immediate corollary that Snlan -- 0, whenever an x. The basic idea of these proofs is to prove the convergence result for the truncated variables 1( lxrlsicj.X),and then use the equivalent sequences theorem to extend it to Xt itself. In view of 20.4, the condition .--)
--)
,
'1'
7
XFlx,IP
<
x,
(20.28)
1 < p < 2,
>1
is sufficient for convergence, although not necessary. Another point to notice about the proof is that the necessity part does not assign a value to a. Convergence implies that (20.25)420.27)hold for cv:ry a > 0. Proof of 20.8 Write Fr
lf Ix/lsktzl.Yf, so that the summands in (20.26)and (20.27) variances of Fr. The sequence ( Yt- F(Ff)1 is indethe means and are pendent and hence a martingale difference, so that Sn' Sm'= Z:=,,,+1(Ff EYt)4 is a martingale for tixed m k 0, and Z7=m+1Var(i'r)= Varts' Sm'4.Theorem 15.14 combined with 20.2, setting p 2 in each case and putting cf2 v ar (r)t and K 1, S' when (20.27)holds. lf (20.26)holds, together yield the result that Sn' this further implies that E7=1Fr converges. And then if (20.25) holds the =
respectively
-
-
-
=
=
-6...y
=
The fzzw of fxzr': Numbers -.F...y S, by 20.3. This proves sequences f#fl and f Ffl are equivalent, and so Sn suftkiency of the three conditions. Conversely, suppose Sn S. By 2.25 applied to xL()) for each (l) e f1, it lim--oxz/wruxf 0 a.s. This means that P(1 AI > a, i.o.) 0, for any follows that a > 0, and so (20.25)must follow by the divergence part of the Borel-cantelli lemma (18.2(ii)).20.3 then assures us that Z7.1Fr also converges a.s. Z')=I V ar( cB, ).!(Ff E(Yt)4Isn fails to x Write sn2 yt ) If sln as n converge, but is asymptotically distributed as a standard Gaussian r.v. (This is the central limit theorem see 23.6.) This fact contradicts the possibility of 1J=IF? converging, so we conclude that sn2 is bounded in the limit, which is equivalent to (20.27). Finally, consider the sequence ( I', EYt) l This has mean zero, the same variance as 1$, and #(I%- F(F,) I > 1a4 0 for a11 t. Hence, it satisfies the conditions (20.25)-420.27) (in respect of the constant 2a) and the sufficiency part of theorem implies l'/ converges, the that Z2=1(Fr F(Ff)) converges. And since Z2=1 (20.26) must hold. This completes the proof of necessity. w -'1-.y
=
=
.-+
=
--y
-
.
-
-
.
=
-
The sufficiency part of this result is subsumed under the weaker conditions of 20.10 below, and is now mainly of historical interest; it is the necessity proof that is interesting, since it has no counterpart in the LLNS for dependent sequences. In these cases we cannot use the divergence part of the Borel-cantelli lemma, and it appears difficult to rule out special cases in which convergence is achieved with arbitrary moment conditions. Incidentally, Kolmogorov originally proved the maximal inequality of 15.14, cited in the proof, for the independent case; but again, his result can now be subsumed under the case of martingale differences, and does not need to be quoted separately. Another reason why the independent case is of interest is because of the following very elegant result due to Lvy. This shows that, when we are dealing with partial sums of independent sequences, the concepts of weak and strong convergence coincide.
20.9 Theorem When fA-r)is an independent sequence and Sn = Z)=1Xt, Sn -P- S if S. and only if Sn -i1.-:
Proof Sufficiency is by 18.5. It is the necessity that is unique to the particular 1=m+jXt,and for some 6: > 0 consider the various ways in case cited. Let Smn which the event ( lSmnl > :) can occur. ln particular, consider the disjoint =
collection
max Is'xj
m V
; 2e,
1
For each k, this is the event that the sum from m onwards exceeds 2: absolutely for the hrst time at time k, and thus n
U
kmm-j
.
max #l<./K1-1
l
vmj
l<
2:,
Iskzl >
2:
=
max mjfn
1kj1 >
2:
,
(20.29)
The Strong fww of fatzr'e Numbers
313
where the sets of the union are disjoint. lt is also the case that
U
IsmjI s 2e, I5-,,,tl> 2:
max
k=pl+1 rrlKw/sk-l
ra
l Isknl K :1
i
l 1smnI> sl
,
(20.30)
where the inclusion is ensured by imposing the extra condition for each k. The events in this union are still disjoint, and by the assumption of an independent sequence they are the intersections of independent pairs of events. On applying (20.29), we can conclude from (20.30)that P
max m S./ S n
IsmjI >
rnin #(
2:
m<
I I< vkn
k fn
e) 2j#(
n
77# < k.=?n+1
IsmjI < Jz, I5-pol >
max rnf.jf-l
< #( IsmnI
IsknI <
:)
(20.31)
lf Sn .T-(-) S, there exists by definition m k 1 such that #( ISmnl
(20.32)
for all n > m. According to (20.32),the second factor on the rninorant side of (20.31) is at least as great as 1 :, so for 0 < : < 1, -
# max 1skyl> 2: < (20.33) 1 : mzjsn and then m x, the theorem now follows by 18.3. w .
-
Letting n
---
x
--
This equivalence of weak and strong results is one of the chief benefits stemming from the indepehdence assumption. Since the three-series theorem is equivalent to aweaklaw according to 20.9, we alsohave necessary conditions forconvergencein probability. As far as sufticiency results go, however, practically nothing is lost by passing from the independent to the martingale case, and since showing convergence is usually of disproportionately greater importance than showing nonconvergence, the absenc of necessary cpnditions may be regarded asmall price to pay. However, a feature of the three series theorem that is common to all the strong law results of this chapter is that it is not an array result. Being based on the convergence lemma, all these proofs depend on teaming a convergent stochastic sequence with an increasing constant sequence, such that their ratio goes to zero. Although the results cah be written down in array fonn, there is no counterpart of the weak 1aw of 19.7, more general than its specialization in 19.8.
20.3 Martingale Strong Laws Martingale limit results are remarkably powerful. So long as a sequence is a martnnd the nn ;tq aonenaencenre renllired lnaul,xalt-s-r/-nc.s- nn oarth-r roetrlr-tiranq
The fzzw of fzzr'd Numbers
moment assumptions called for are scarcely tougher than those imposed in the independent case. Moreover, while the m.d. property is stronger than the unconrlatedness assumed in j19.2, the distinction is vel'y largely technical. Given the nature of econometric time-series models, we are usually able to assert that a sequence is uncorrelated because it is a m.d., basically a sequence which is not forecastable in mean one step ahead. The case when it is uncorrelated with its own past values but not with some other function of lajged information could arise, but would be in the nature of a special case.
The results in this section and the next one are drawn or adapted chiefly f'rom Stout (1974)and Hall and Heyde (1980),althoughmany of theideasgo back toDoob (1953). We begin with a standard SLLN for f'z-bounded sequences.
o and 20.10 Theorem Let (.Yf,t.r)7' be a m.d. sequence with variance sequence (c1), 0 if (th) a positive constant sequence with at cxa. Snlan -.1%
'lh
X
77c2//2 t <
x.
t
(20.34)
u
>1
two ways to prove this result. The first is to use the martingale convergence theorem (15.7)directly, and the second is to combine the maxima1 inequality of 15.14 with the convergence lemma 20.2. ln effect, the second line of argument provides an alternative proof of martingale convergence for the square-integrable cse, providing an interesting comparison of techniques. There are
(at least)
First proof Detine Tn l=jxtlat,so that (Tn,5nIis a square-integrable martinusing the norm inequality and orthogonality of (xY)), gale. We can say, =
supFl TnI S supern) n
2 1/2
n
x
=
2
2
Xcr/l,
1/2
<
=,
(20.35)
>1
leading directly to the conclusion Tn F a.s., by 15.7. Now apply the Kronecker lemma to the sequences f L() ) for ) e f1, to show that Snlan -.Q1->0. w .-->
Second proof For m k 0, f Tn Tm, ?L) is a martingale with -
E (F
-
F
)2
=
mwjyltlalt.
(20.36) 2/ 2
Apply 15.14 fop 2, and then 20.2 with ct2 Gt at. Finally, apply the Kronecker before. w lemmaas =
=
(U
S B < oo, say, then setting at = Compare this result with 19.4. If VartX/l x 2 tzzjntlt 2 2 < #Z,=11// 1.64# < x, and the condition of the theorem t, we have Z = is satisfied, hence Xn Snln -F-.-h 0, the same conclusion as before. But the conditions on the variances are now a 1otweaker, and in effect we have converted the weak law of 19.1 into a strong law, at the small cost f substituting the m.d. assumption for orthogonality. As an example of the general formulation, suppose the sequence satisfies =
'
x
=
The Strong fzzw of fzzr.d Number X
X Elxl)l?' t
<
(20.37)
x.
>1
We cannot then rell upon knconverging to zero, but (puttingat tl) we can show that n -2E7=1X, Xnj n w jjj (jo so. The limitation of 20.10 is that it calls for square integrability. The next step is to use 20.4 to extend it to the class of cases that satisfy =
=
Co
XFI xtI#/Jf
<
(20.38)
x
>1
'1'
x. It is important to appreciate that (20.38)for for 1 < p K 2, and some (Jfl p < 2 is not a weaker condition than for p 2, and the latter does not imply the former. For contrast, consider p 1. The Kronecker lemma applied to (20.38) implies that =
=
Fl
an-1y')F1x t >1
1
(20.39)
0.
---/
For an rl, such a sequence has got to be zero or very close to it most of the time. In fact, there is a trivially direct proof of convergence. Applying the -
theorem (4.9),
monotone convergence tzn-l
E lim
Isn I
n--yx
N
< E lim c-ln
:7 I-Y,I >1
n-x
1im = n--yx
c-n1>7e'lx,l
.
>1
(20.40)
For any random variable X, E I.YI 0 if and only if X = 0 a.s.. Notlzing more is needed to show that Snlan converges, regardless of other conditions. Thus, having latitude in te value of p for which the theorem may hold is really a matter of being able to trade off the existence of absolute moments against the rateofdampingnecessrytomakethemsummable. Wemaymeetinterestingcasesin which (20.38)holds for p < 2 only rarely, but jince this extension is available at small extra cost in complexity, it makes sense to take advantage of it. =
20.11 Theorem If l.#),X)7is a m.d. sequence satisfying (20.38)for 1 f p ; 2,
Snlan
--1t.:
0.
Proof Let l'f 1( Ix,I>,)Xs and note that (aYfl and f Ff) are equivalent under by 20.4. Ff is also Tf-measurable, and hence the centred sequence (20.38), f4,;,), where Zt Ff A'(l'rl S-1), is a m.d. Now, =
=
-
Ezi)
F(e'(zlI@,-1)) F(rfl = F(F(I'lI@,-1)
=
-
1,-1)2)
The f-aw of fzzr'c Numbers
316
= s(r2,)
F(F(y,j y,-j)2).
-
(20.38)implies
According to 20.4 with r 2, Ezt) 2 < EY2), by (20.41), =
X Ezlyal t
<
,
(20.41) EYlllal /.1 t '=
that
t < x
,
and so, since
(20.42)
x.
>1
By 20.10, this is sufficient for variable. But n
n
l
'Xz,/z, 7-)
Ytlat
=
By
-.6.1..2/
Llzzjztlat
>1 15.13(i), (20.38)is
-
,..-1
>'-A'(i',j
<%t,
where 5'l is some random
5t-L)lat.
(20.43)
a.s.
(20.44)
,-z1
equivalent to
X
T,-l)/4' A7el.&lpl >1
< x,
< x, a.s. Absolute According to 20.5, (20.44)implies that Z7.11F(l'/ITr-l) may say that convergence of a series implies convergence by 2.24, so wet7n-l7=l -6YA st-Lllat -E1-.:/ '/=1F(I$l + Sz and so l'l Sz. Hence, E7=1 Ytlat 0 follows by 20.3 and equivalence of and Xt Ff the by the Kronecker lemma. lt 0. w implied by (20.38)that Snlan
tlat
-6-.3.-:
,5'1
-61.+
Notice that in this proof there are no short cuts through the martingale converis a martingale, the problem is to gence theorem. While we know that Llzubxtlat establish that it is unifonuly fal-bounded, given only information about the joint distribution of (A)), in the fonn of (20.38).We have to go by way of a result for p 2 to exploit orthogonality, wlch is where the truncation arguments come in =
handy.
20.4 Conditional Variances and Random Weighting A feature of martingale theory exploited in the last theorem is the possibility of relating convergence to the behaviour of the sequences of one-step-ahead conditional moments; we now extend this principle to the conditional variances E (Xt215 t-1 ) The elegant results of this section contain those such as 20.10 and .
20.11.
The conditional variance of a centred coordinate is the variance of the innovation, that is, of Xt Ext IA-1), and in some circumstances it may be more natural to place restrictions on the behaviour of the innovations than on the original setuence. ln regression models, for example, the innovations may coaespond to the regression disturbances. Moreover, the fact that the conditional moments are Tr-l-measurable random variables, so that any constraint upon them is probabilistic, pennits a generalization of the concept of convergence, following the results of j15.4; our confidence in the summability of the weighted conditional variances translates into a probability that the sequence converges, in the -
Fe
Strong fzzw of zzrge Numbers
317
manner of the following theorem. A nice refinement is that the constant weight can be replaced by a sequence of Tf-l-measurable random weights. sequence tt7fl zo-lzrfheoreml-et (Xf,Trlrbeam.d. sequence, (B$Janon-decreasingsequenceof 7=1Arf.Then positive, Tr-j-measurable r.v.s, and Sn =
#X
-' F(xll
@,-1)/+1 <
rntm,
x
0j
jsnlwn --
xj-
'1'
>1
=
0. u
(20.45)
The last statement is perhaps a little opaque, but roughly translated it says that 0), is not less than that the probability of convergence, of the event l5',,/Gn of the intersection of the two other events in (20.45).In particular, when one probability is 1, so is the other. --
Proof If (A) is a m.d. sequence so is (xY,/J7f), since W$is Ff-l-measurable, and 1=jXtIWtis a martingale. For ) e f, if L((l)) F()) and W%() 1 x then Tn 0 by Kronecker's lemma. Applying 15.11 completes the proof. . &()/lL() -->
=
---
See how this result contains 20.10, corresponding to the case of a fixed, divergent weight sequence and a.s. summability. As before, we now weaken the summability conditions from conditional variances to pth absolute moments for 1 f p < 2. However, to exploit 20.5 outside the almost sure case requires a moditkation to the equivalent sequences argument (20.3),as follows. of Tf-measurable r.v.s,
20.13 Theorem If (-YfJand ( Ff) are sequences XPCXt y. l'f 11,-1)
#
<
c,o
A
(A # Ff) e 5t, so that P(Xt y: F, l8h-1)
=
15.13(ii), ):
XF(1s,I877,-1)()
<
c,o
A
(t):
.1
,
=
F(1s,l 1,-1). According to
X 1s,()
<
x
=
0.
(20.47)
>1
>1
But E7 1s,((l))< t'in ite and hence
(20.46)
Cr
Cr
#
F,) converges
-
>1
>1
Proof Let Et
X Xt
cxl
Ff()) is that the numberof coordinates for which Xt x. F,())) implies < 1CX (20.47)therefore (20.46).w t () :#:
means ''o
t
-
=
Now we are able to prove the following extension of 20.11.
20.14 Theorem For 1 f p f 2, let F1 (X7=1F( lXt 1:lT/-1)/Ji$' ( J'IS XJ Under the conditions of 20.12, =
'1*
<
x
l and
Ez
=
.
#((F1
f'''h F2)
-
(&/W%
-->
0J)
=
0.
(20.48)
Proof The basic line of argument follows closely that of 20.11. As before, 1et 1$ that Zt J',- F(F,I1,-1) is a m.d. and 1( = Ix,Isu,)X,, so =
F(z2I@f-1) t
=
F(Fl 1@f-1) (F4r,1 #t-1))2 -
The fzzw of fzzr': E't 2,I
f
,-1),
Numbers
(20.49)
a-s.
Applying 20.5 and the last inequality,
#
E3
XF(z2,
-
>1
I;/.,)/J.y2t
lt follows by 15.11 and the fact that Ej P A'1
Xzr/B'r
-
< x
C
-
(Fl
5'1
--
0.
=
=
(20.50)
D)
-
QJ
(D
-
C) that
(20.51)
0,
>1
where & is some a.s. finite random variable. A second application of 20.5 gives X
P Fj
IF(F,I@r-1)l/B :-.2 >1
-
<
c.o
0,
=
(20.52)
which is equivalent (by 2.24) to
p sl
x- s(y,Iy,-,)/u,,
-
s,
--
-1
1 o, -
(20.53)
)
where S2 is another a.s. finite r.v. And a third application of 20.5 together with 20.13 gives X
P A'1
*
Xx,-X r,
-
>1
s?
-+
N1
=
0,
(20.54)
for some a.s. finite r.v. S?. Now (20.51),(20.53),(20.54),the detinition of Zt, the Kronecker lemma and some more set algebra yield, as required, *
0
=
P
=# =
e'1
-
X F,/u$ >1
(F1 r7 E1)
PEj
5'1+
-+
-
i',
.-+
>1 f'7eil
fn
E1
o
t,5k/en OJ). . -->
-
sz
(20.55)
20.5 Two Strong Laws for Mixingales The mmingale difference assumption is specialized, and the last results are not sufficient to support a general treatment of dependent processes, although they are the central prop. The key to extending them, as in the weak law case, is the mixingale concept. In this section we contrast two approaches to proving mixingale strong convergence. The tirst applies a straightforward generalization of the methods introduced by McLeish (1975a);see also Hansen (1991,1992a) for related
The Strong f-zzw of z7rg' Numbers
319
results. We have two versions of the theorem to choose from, a milder constraint on the dependence being available in return for the existence of second moments. be a fv-mixingale with respect to 20.15 Theorem Let the sequence (m,@f)=-x cfl either for constants ( with mixingale size 2, (i) p or (ii) 1 < p < 2, with mixingale size If X*t < cxo then Sn S. ,
-z1,
=
-1.
czlc'f
-6:4
Proof We have the maximal
inequality, l
E max 1 j
ISIJ'?<
KTcq', 1
(20.56)
/=
where K is a tinite constant. This is by 16.10 in the case of (i) and 16.11 in case (ii). By relabelling coordinates i't can be expressed in the fonn E max
lsj smIP
< K
-
1 j n
X cpt
(20.57)
1=rrl+1
for any choice of m and n. Moreover, P
max m
<j
I
uh
-
SmI
>
e
=
#
max m<jn
Isj smIJ' -
>
e
1 p < ,--E max ISj Sm1
(20.58)
-
m<jn
by the Markov inequality. Inequalities (20.57)and (20.58)combine to yield (20.2), and the convergence lemma 20.2 now yields the result. w We can add the usual corollary
from Kronecker's lemma.
satisfy either (i),or (ii) of 20.15 with respect 20.16 Corollary Let fxtlat,stloloo to constants fctlatt, for a positive sequence fat ) with at x. lf '1h
X
'lcnlant <
(20.59)
x,
f
>1
then Snlan
-1->
0. u
The second resuli exploits a novel and remarkably R. M. de Jong (1992).
powerful argument due to
20.17 Theorem Let (Xf,@,) be an fv-bounded, Al-mixingale with respect to constants f ctj for r 1, and let (Jfl, (#f) be positive constant sequences, and fMt J a positive integer sequence, with an x. lf ,1'
n
co '-zlanll
TM n=1
n exp
3lMlTBl n
1.=1
t
<
x,
(20.60)
The facw of f-zzrpd Numbers
320 X
:7 sl-rss x,Irlat
(20.61)
< x,
t
>1
X
< X Lutctlat ,-1
x where ((m),,,=o are the mixingale
(20.62)
x,
coefficients, then Snlan
-1'..-j
0. u
(#,J and
fA1)) are chosen freely to satisfy the conditions, given (tztJand fcrl, which suggests a considerable amount of flexibility in application. The 1, the role which was sequence (#,1 will be used to define a truncation of played by (tz,)in 20.11. The most interesting of the conditions is (.20.62),wlaich explicitly trades off the rate of decrease of the mixingale numbers with that of the sequence fctlatj. This approach is in contrast with the McLeish method of defining separate summability conditions for themomnts and mixingale numbers, as detailed in j16.3. Here
t#f
Proof Writing 1Bt for 1( 1x,lxs,), start by noting that E t+jXt = Etsj 1t,Xt + Et+j 1 11)Xt. -
Hence, we have the identity Xt
=
(Et+Mt-j
llXr
Et-M:$BtXt)+
-
(f+M,-1(1
li/lA
-
Et-Mtll
-
-
+ xt e',+,r,-1.;)+ Et-u7, -
and, by the usual
ttelescoping
1()Xf)
(t0.63)
sum' argument, Mt- 1
Bx E f+M,-l 1 t t
-
$Bx ;,-.A,z, t t
:- z,,j
=
(20.64)
./=l-Mt
where Zjt 14,I f lBt
=
is a m.d. sequence. Note that Et.t BX E ?+y-1 IW-Y t t t t, and tzys?h+yl a.s., by a double application of 10.13(ii). Summing yields
p X-Y,
-
n
=
zp X X l -Mt J=1
>1
Mt- 1
n
+
77F,+>,-1(1
n
+
=
ubn
+
ThEt-utj
-
>1
x>1
szn
-
1Bt%t
-1
jzz
-
-
1Of
)Xt
l
'+
F7(Xt
-
Et.vut-xtj
,...1
Et-uy <%n
+
s4n+ ssn.
(20.65)
The object will be to show that Sknlan -.6-.A 0 for k = 1,...,5. Stming with Sjn, the main task is to reorganize the double sum. lt can be verified by inspection that
The Strong fxzw of f-zzrge Numbers n
Mt- 1
Mn- 1
321
n
77 zp 77 Yzp 77 j=1 r=l -
j=-Mt
-Ma t-nqj
Mn- 1
Mn- 1 qj- 1
-M2
n
- X Xzy? 77 X Xzyf. '=1
N:1
j=3 -Mn
(20.66)
+
-
j=M2
-Mn
>1
1 for -M1 < j < M1, and qj t for -Mt < j f -Mt-j and Mt-j f j < Mt, / 2,...,,,. Note that, for arbitrary numbers x1,...,xk, l l I > EJ c UiI 1 ( lxi 1 > tlkk. Hence by subadditivity, and Azuma's inequality (15.20),
where qj
for
=
=
t-lxf
=
=
Mn 1 -
P
I.S3nI > an
f
n
X
XZy,
P
j=1 -Mn
anzlnMn
>
>1
Mn- 1
-M;
7) + X =Af2 '=1 -Mn
+
P
Xz,
>
anelnun
>1
,z
n
Blt '-'NX (32152
-rzlalnl
f zfaexp
>1
Mn
-Af2
:j
+
qj- 1
)j
+
'=1-Mn
f 4Mnexp
-:
1
-
exp
252/32A/2
-e
n
; #2 ,
>1
.j=Jf2
lan
n
qj- l
32M2 TBl n
t
(20.67)
.
>1 Under (20.60),these probabilities are summable over n and so Sjnlan first Borel-cantelli lemma. Now let ( l'r) be ay integrable sequence and detine
--1-4
0 by the
n
,7,1 -
X Ytlat.
(20.68)
,-1
By the Markov inequality, P max IS1 m<jxn
<%m' -
I>
1 a er S ,-E max Ifj SmI -
s !
<E -
n
7-)E II', Ilat.
rcal
XA'IYtllat >1
(20.69)
rrl<w/Kn
<
=,
(207U)
The fww of fzzr'd Numbers
322
0 by Kronecker' s then 5',l -6J-> f by an application of 20.2, and hence Sllan remaining result to each of the lemma. We apply tltis terms. For Szn. put l', --t!-
=
BJ),
an d note that
E r+M,-lX(1 1 -
n
?;
l
< A7#1,-rF1 x,$r/cr,
F7FIF,1lat < F7A'1 (1 114Xtl/J/ -
(20.71)
>1
>1
>1
B
r. S5n is dealt with in exusing the fact that IXf(1 1 t) IBt I f lXt (1 1)/#/l t s%n and %n,put successively Ff Xt Et-t-jxt and Ff actly the same way. For -
-
=
=
-
assumption,
Et-utxt, and by the mixingale
n
n
77FIF,1lat
f
77 ctlatjLut.
(20.72)
>1
>1
The proof is completed by noting that the majorant terms of are bounded in the limit by assumption. .
(20.71)and (20.72)
The conditions of 20.17 are rather difticult to apply and interpret. We will restrict them vel'y slightly, to derive a simple summability condition which can be compared directly with 20.15. with 20.18 Corollary Let (A,Tf ) be an Lr-bounded, fyl-mixingale of size respec! to constants (cf). If fcf) and ftz/l are positive, regularly varying x, and << c, and an sequences of constants with Ilmllr -(?()
'1-
n
X (c,/c,)1 <
(20.73)
,,o
Nl
where
'0 then Snlan
-6->
=
1) 2r(pc.#2(r-
(1+?.)(?c
(20.74)
1)'
+2(r-
0.
Proof Define ot (log -1- fo r > 0 This is slowly varying at infinity by 2.28, and the sequence fotltl is summable by 2.31. Apply the conditions of 20.17 with the added stipulation that fBtI and f#4l are regularly varying, increasing sequences, and so consider the conditions for summability of a series of the form Uzln) ), for q > 0. Since Lnonlnjconverges, summability follows Esitnlexpl 0. Taking logarithms, this is equivalent to f'rom (nlonllhlnlexpt U1(n4) =
.
-q
-->
-q
1ogn logtu) -
Since U(n)
nvn)
=
1og&1(n) q&a(n)
where Lnj
log n logton) -
+
+
-
(20.75)
.-.-)
-x.
is slowly varying, tls condition has the form
pllog n + 1og(fy1(a)) 1)?2Q(?7) -
(20.76)
-x,
--
where pl and pa are non-negative constants and LL(n) and Lzln) are slowly yarying. The tenns logt/n) and log(fz1(n)) can be neglected here. Put p2 0 and Lzn) Llon (logn) 1+8 an d the condition reduces to =
=
,
=
The Strong
(1 + pl
-
zzw of fzzr'd Numbers
ntlognlllogn
323
(20.77)
-x, ---
which holds for all pl for any q > 0 and 6 > 0. Condition (20.60)is therefore satisfied (recallingthat (#r) is monotone) if ,
2 2 nMnBnlan < on. 2
Similarly, conditions
(2.61)and (2.62)are B t1-rFl Xt Irja
B1
t ((
(20.78) satisfied if, respectively,
'-rclla
t
t
(20.79)
<( otlt,
and M-t%ctlat < otlt,
(20.80)
(p > (p0.
We can identify the bounding cases of Bt and Mt by replacing the second order-ofmagnitude inequality sign in (20.79),and that in (20.80),by equalities, leaving the required scaling constants implicit. Solving for Mt and Bt in this way, substituting into (20.78),and simplifying yields the condition
(c,lat /
?,/f,
'z
(20.81)
where (, (2r(p+ 2(r 1)1/(41+ rl(g + 2(r 1)j. This is sufficient for (20.60), (20.61), and (20.62)to hold. Since ct and at are specified to be regularly varying, there exist non-negative constants p3, p4, and slowly varying functions Lz and L4 such that ct tP3L5t) /P4fa is summable implies that and at 4 (/). The assumption that (lctlauj which in But (p > (?o implies ( > %, so that (p3 p4)( < (p3 p4)(o K . implies This completes the proof. 1). turn (20.8 =
-
-
=
=
-1.
-1,
-
-
Noting that 1 f Q f 2, the condition in (20.73)may be compared with (20.59).Put whereas with (pn and r 1, we get (() 2 and we obtain (e (pn 242r 1)/(3r 1) which does not exceed r in the relevant range, taling values between 1 when r 1 and 55when r 2. Square-summability of ctlat is sufficient x x. Thus, this theorem does not contain and r only in the limit as both (20 20.16. On jhe other hand, in the cases where (cfJis uniformly bounded and at f, we need only (o> 1, so that any r > 1 and (fb > 0 will serve. These dependence restrictions are on a par with those of the Lj convergence law of 19.10, and a striking improvement on 20.16. The case r 1 is not permitted for sample averftlog f) 1+ for 6 > 0. In other words, the ages, but is compatible with at theorem shows that
J
=
#,
=
=
-
=
=
-
=
=
-->
-->
=
=
=
(rltlogn) l4$-ly-qa:r
...,
t
() a.s.
(20.82)
>1
This amounts to saying that the sequence of sample means is almost surely slowly x; it could diverge, but no faster than a power of 1og n. varying as n --
20.6 Near-Epoch Dependent and Mixing Processes In
vinw
nf
fhe
lnqf
reKllltK.
there are two
Dossible
aooroaches
to the NED case. It
The frw of fxzrgd Numbers
324
turns out that neither approach dominates the other in tel'ms of pennissable conditions. We begin with the simplest of the arguments, the straightforward extension of 20.18. for z0-lgTheorem Letasequence lml=-xwith means lwl=-xbefv-NEDof size g,,llp,on a (possiblyvector-valued) sequence 1 < p S 2, with constants dt f< 11*-, which is FrlJx a-mixing (f-mixing)of size -J. lf ( -b,
-
*
11(.:)g,rl/hllti < 77 >1 for q
>
p in the a-rnixing case q
(20.83)
x
-
p in the
1) lqa lqb + lq +qlb jqxs + lq 1) ( (1 -
( t
Sxt n t= hen z-1E''
=
min
,
-
w)-fJ.-.
-
Nvhere
case) g-nzixing + +
1) gc
(20.84)
,
0.
Proof
J(1 1/)) with By 17.5, (Xt- jttl is a fal-mixingale of size respect to constants (c,), with ct
-
-
This penuits arbitrary mixing and NED sizes and arbitrary moment restrictions, so long as (20.83)holds with ( arbitrarily close to 1. By letting b one obtains result for mixing sequences, and by letting a a result for sequences that a underlying independent AP-NED Interestingly, in each of these on an process. are special cases ( ranges over the interval (1, lqlq + 1)) as the mixing/fa p -NED size is allowed to range from zero to By contrast, a result based on 20.15 would be needed if we could claim only (X, g,,l/thllpl for finite p; this rules out square-summability of the sequence ( 11 for choices of The and first of these rsults comes directly by b. (20.83) any a applying 17.5. -->
(x)
-->
(x)
-=.
-
20.20 Yheorem For real numbers b, p and r, let a squence f-vl'!x with means be JO-NED of size with constants dt << 111,wllp. f on a sequence (J$J=-x, x, For a positive constant sequence (th) let f Xt kttllatl be uniformly fv-bounded, and 1et g,,)D-.x,
-.b
-
'1'
-
X
11(-,w)/hII;< 77 >1 -
n Then an-1Xr=1( Xt ;1, b (i) p 1, 1 (ii) b (iii) b p (iv) b = 1, 1 =
=
=
,
-
=
< =
<
w)
--F--.->0
2, r > 2, p < 2, r 2, r k 2, p < 2, r
Proof By 17.5, conditions
(20.85)
=.
in each of the following cases'. 2)., (Ji) is (x-mixing of size > p, ( F,) is a-mixing of size 1),. f Ff J is (-tnixingof size 1). 2 p, f F,) is (-mixingof size
(i)-(iv)are
-r/(r
-
-#r/(r
-p);
-r/2(r
-
-r/(r-
all sufficient for
l Xt
-
P,,)/Jr,
?h1 to be
The Strong f-aw of fzzr'd Numbers
325
where 5t c(F,, s < f). The mixingale constants are of size II-Y, p.fllrl/c, 11*-, wIIr/c,.The theorem follows by 20.16. w
an fv-mixingale ctlat (t maxtlf,
=
-,
=
-
-
-z1,
As an example, 1etXt possess moments of all orders and be AZ-NED of size on x). of size close Summability of the a-mixing r,,,-.k terms to process (letting an Yaxtlat) is sufficient by (20.85).The same numbers yield ) 8,/on putting q 2 and a 1 in (20.84),which is not far from requiring summability of the Q-norms. However, this theorem requires fv-boundedness, which if r is small constrains the permitted mixing size, as well as offerig poor NED size characteristics for cases w'ith p < 2. lt can be improved upon in these situations by introducing a truncation argument. The third of our strong laws is the following. -1
=
=
=
20.21Theorem Let a sequence (A)=-x with mans lwlm-x be AP-NED of size with (( dt 1 2, < f for constants p g,II,,on a sequence ( F,)--- which is 111,-1/p
either
(i) a-mixing of size (ii) (-mixingof size
2) for r > 2 or 1) for r > 1, and r k p; and for q with p S q S r and a constant positive sequence -r/(r-
-r/2(r-
(th)
'1<
1et
x,
X
Fl 11(-4p.fl/tll'''t -
ntp,k/rl
<
(20.86)
x;
>1
-1
then an 17=1(m w)
-tl.u
g u
-
.
Note the different roles of the three constants specified in the conditions. p controls the size of the NED numbers, q is the minimum order of moment required to exist, and r controls the mixing of (Ff l The distribution of Xt does not otherwise depend on r. .
P roo f The strategy is to shbw that thete is a sequence equivalent f Xt ptllatl, and satisfying the conditions of 20.15(i). As in 20.6, let
to
-
F,
Xt
=
-
ttt3llat
:l:
(1
11),
-
(20.87)
if Xt > go otherwise. Note that denotes where 1t/ 1( Ix,I
<+'
i-'
=
-
-
-1
.
=
-
-;1
F,IIrl ( dtlapll, 11 rrllrK 2( 111,g.rll/crlWr q k p, and 11
ct << max
(20.88)
.
for any q < Here, dt < 2 IImwll for any r by the second inequality of (20.24).Condition (20.86)is therefore sufficient fr the sequence lctzJ to be summable, and ( Ft F(J)) satisfies the conditions of 20.15(i).We can conclude that Z')=1(l',EYt44 -E6-: S3, where l is some random -
-
-
-
variable.
According to 20.6, condition t/ari/xt N';
-
r'/I'
4 tla/xrefnrsx
(20.86)is
rxnnxr,xraee
sufticient for
tn n Gnlte
llmlt
Z'7.1IE)
hx, @ 94
enxr
I<
V=
x. .
r'/v
The 4
-
The zzw of fxzr'c Numbers
326 C1, and
n
X
-E->
I'J
.%
(20.89)
C1.
+
,-1
(20.22)and (20.6)further imply that F2 and Xt bktllatare equivalent sequences, and hence Inequalities
-
n
77((1, vlat >1 -
1$)
-
--6-:
(20.90)
sz,
where 51 is another random variable, by 20.3. We conclude that
X xt vlat ,-1 -
-EI-
5.1+
s, + ti'l s?,
(20.91)
=
J-1J n
say. lt follows by Kronecker ' s lemma that conclusion. w
=
-f.-..y
1(4
w)
-
0, the required
more specialized, result. The linear function of martingale differences with summable coefficients is a case of particular interest since it unifies our two approaches to the strong law. Here is a fill,
20.22Theorem sequence with
where ( U(k is a uniformly Ip-bounded m.d. Let Xt Lcej=-xjut-j 0;1 < n. Then p > 1, and lrv-xl j =
... .
1.19
lk.
((r!ji;;7''') drkr
.kigikr-r
................45))-
.
....... .11 1=1
.
s
b..i
'
-).
4:4()4 .
.. .
4..:-..)-
....
....
$tt),.'..
:..
.
t
.-y
.
.,.
....
.-).
.....-
,
u.. r
r.'t' #t-. ''.'(.
l'.
..
--?---...
.'
h.
.
k' ) '.)
. u'y.,.
.
%.
i..,..
y...y. ... ...
,. .
.
.,
with ct t< 1// and arbitrary size. It was shown Proof Ff Xtlt is a L ! in 16.12 that the maxlmal inequality (20.56)holds for this case. Application of the convergence lemma and Kronecker's lemma lead directly to the result. Alternatively, apply 20.18 to Xt with at /. w -mixingale
=
=
In these results, four features summarize the relevant characteristics of the stochastic process: the order of existing moments, the summability characteristics of themoments, andthe sizes of themixing andnear-epochdependence numbers.The way in which the currently available theorems trade off these features suggests that some unitkation should be possible. The Mct-eish-style argument is revealed by de Jong's approach to be excessively restrictive with respect to the dependence conditions it imposes, whereas the tough summability conditions the latter's theorem requires may also be an artefact of the method adopted. The repertoire of dependent strong laws is currently being extended (deJong, 1994) in work as yet too recent for incorporation in this book.
21 Uniform Stochastic Convergence
21.1 Stochastic Functions on a Parameter Space The setting for this chapter is the class of functions
2f : x 0wllere (f1,F,g.) is a measure space, and (O- is a metri space. We write to ai value which variable the real point random assumed is denote by f the a (,0), is not a random varifor fixed 0. But /(,.), alternatively written just able, but a random element of a space of functions. Ecnometric analysis is very frequently concerned with this type of object. Loglikelihoods, sums of squares, and other criterion functions for the estimation of econometric models, and also the first and second derivatives of these criterion functions, are a1l the subject of important convergence theorems on which proofs of consistency and the derivation of limiting distributions are based. Except in a restricted class of linear models, a11 of these are typically functions both of the model parameters and of random data. --+
,
.f4,0)
,p)
.f(),
To deal with convergence on a function space, it is necessary to have a criterion by which to judge when two functions are close to one another. In this chapter we examine the questions posed by stochastic convergence (almostsure or in probability) when the relevant space of functions is endowed with the uniform metric. A class of set functions that are therefore going to be central to our 2- where discussion have the fonn f*1fl --+
f*k
.f((0,0).
(21.1)
sup 6c e
=
For examjle, if g and h are two stochastic functions whose unifonn proximity is at issue, we would be interestd in the supremum of
.f(,0)
=
I#(,0)
-
(,0) I .
An important technical problem arises here which ought to be confronted at the outset. We have not so far given any results that would justify treating f* as a random variable, when (O- may be an arbitrary metric space. We can write ,p)
((,):fs
> xl
=
U f ): /'(0,)
> xl,
(21.2)
0e O
andthe results of 3.26 show that f: /*4) > xl e 5 when (: O is a subset . each0, when e is a countable set. But typically of the kind, and is uncountable. someihing .f(0,)>x)
e T for of (Rk or ,#s)
The ww of f-wr'd Numbers
328
This is one of a class of measurability problems having ramifications far beyond the uniform convergence issue, and to handle it properly requires a mathematical apparatus going beyond what is covered in Chapter 3. We shall not attempt to deal with this question in depth, and will offer no proofs in this instance. We will merely outline the main features of the theory required for its solution. The essential step is to recognize that the set on the left-hqnd side of (21.2) can be expressed as a projection. Let Ot.l denote the Borel tield of subsets of 0- that is, the smallest c-tield containing the sets of 0- that are open with respect to p. Then 1et (f1x 0, 9 @ B denote the product space endowed with the product c-tield (thec-tield generated from the measurable rectangles of 5 and fe), and suppose that f(.,.) is 5 (& t.lfp-measurable. Observe that, if ,
Ax
=
f(),0): f4,0)
>
x)
9 @ Be,
l
(21.3)
the projection of Ax into f is Ex
f
f (,0) = ((t)..f'u =
:
> >
x, 0 e x)
0-) (2 1.4)
.
In view of 3.24, measurability of f* is equivalnt to the condition that E'iI l 5 for rational x. Projections are not as a rule measurable transformations, but 5P L'j,5P,1b where is the under certain conditions it can be shown that Ex e completion of the probability space. The key notion is that of an analytic set. A standard reference on this topic is DellacherieandMeyertlg78); see alsoDudley (1989:ch.13), and Stinchcombeand White (1992).The latter authors provide the following definition. Letting (f1,T) be a measurable space, a set E c f is called F-analytic if there exists a compact metric space (O,p) such that E is the projection onto of a set A l 5 (&0e. The Also, a function J: f F-) F is collection of T-analytic sets is written for each called T-analytic if (: J()) f xl e e P Since every E e 5 is the projection of E x 0- e 5 @ e, 5 454. A measurable is not in general a c-field, set (or function) is therefore also analytic. although it can be shown to be closed under countable unions and countable intersections. Tiie conditions under which an image underprojection is known to be analytic are somewhat Weaker than the detinition might suggest, and it will actually suftice to let (O,e) be a Souslin space, that is, a space that is measurably isomorphic to an analytic subset of a compact metric space. A sufficient condition, whose proof can be extracted from the results in Stinchcombe and White (1992),is the following: ,
.4(54.
.45)
.x
.45)
,fo)
21.1 Theorem Let (f1,F) be a measurable space and (Ois in M5 (&Se), the projection of B onto n .45).
a Souslin space. If B e
define T&=() t TB where (f1,;B,F) Now , given themeasurable space (f1,@), is the completion of the probability space (f1,@,g,) (see 3.7) and Uthe intersection is taken over al1 p.m.s g defined on the space. The elements of ; are ca lled univer,
Uniform Stochastic Convergence
329
sets. The key conclusion, from Dellacherie and Mayer III.33(a)), is the following.
sally measurable
21.2 Theorem
(1978:
space (f1,T),
For a measurable
.454 c 5 & n
(21.5)
.
S ince by definition
5U
FB
for any choice of jt, it follows that the analytic sets for any choice of g. In other of 5 are measurable under the completion of (f1,@,jt) if analytic there exist is that E A, # such A c E ? # and g(A) jt(#). words, e 5 ln this sense we say that analytic sets are nearly' measurable. Al1 the standard probabilistic arguments, and in particular the values of integrals, will be and we can ignore it. We can unaffected by this technical non-measurability, legitimately treat J*((0)as a random variable, provided the conditions on 0- are observed and we can assume J(.,.) to be tnear-l5 f9 Oe/s-measurable. An analytic subset of a compact space need not be compact but must be totally bounded. It is convenient that we do not have to insist on compactness of the parameter space, since the latter is often required to be open, thanks to strict inequality constraints (thinkof variances, stable roots of polynomials and the like). In the convergence results below, we find that 0- will in any case have to be totally bounded for completely different reasons: to ensure equicontinuity; to ensure that the stochastic functions have bounded moments; and that when a stochastic criterion function is being optimized with respect to 0, the optimum is usually required to lie almost surely in the interior of a compact set. Hence, total boundedness is not an extra restriction in practice. The measurability condition on fl(.,) might be verifiable using an argument from simple functions. lt is certainly necessary by 4.19 that the cross-section functions A.,0): F-> P-and J(),.): 0- 2- be, respectively, T/plmeasurable for ,n) each 0 e 0- and s/s-measurable for each ) e f. For a finite pmition (O-1,...,0of 0- by e-sets, consider the functions c
=
-->
J(??,)(t0,0)J(,$), =
where
j
is a point of O-j. If E) Ax
=
l(,0):
.f(,,z)(tt,0)
=
0 e O-j,j
(): f,%.l
f xl
=
UFJTX
=
(21.6)
1,...,-, f a7l e 5 for each
h
then
O-je 5 (8)f(.),
(21.7)
j being a finite union of measurable rectangles. Since this is true for any fm) is 5 (8)fe/s-measurable. The question to be addressed in any particular case is whether a sequence of such partitions can be constructed such that hm) f as .;r,
--)
m
--
x.
Henceforth we shall assume without further comment that suprema of stohastic functions are random variables. The following result should be carefully noted, not least because of its deceptive similarity to the monotone convergence theprem, although this inequality goes the oppositeway.The monotoneconvergencetheorem concerns the expectation of the supremum of a class of functions (/',,4)1, whereas the present one is more precisely concerned with the envelope of a class of
The fzzw of frr'e
330 functions, the function of f.
J*()) which
assumes the value supeseltttl,ol
sup F(.f(0)) f E sup
21.3 Theorem
0e O
Numbers
0e O
J(0)
at each point
.
Proof Appealing to 3.28, it will suffice to prove this inequality for simple functions. A simple function depending on 0 has the' fonu m
((,0)
=
Xw(0)1s/(tl))w(0), ) =
(21.8)
e Ei.
f=1
Defining
G*j
=
supesowto),
sup ee e
(p4,0)
=
a*j, (t) e Ei.
(2 1.9)
Hence
supF@(0))
E sup (/0) ee O
-
e(-)
X (w(0) u14PEi)
sup
=
-
86O
< 0,
(2 1.10)
f=l
where the final inequality is by definition of a). w
21.2 Pointwise and Uniform Stochastic Convergence Consider the convergence (a.s.,in pr., in Lp, etc.) of the sequence (Q,,(0)Jto a limit function :(0), Typically this is a law-of-large-numbers-type problem, with
pn(0) X4n,(0) >1
(2 1.11)
=
(we use array notation for generality. but the case qnt qtln may usually be assumed), and :(0) 1imn-+xF(:s(0)). Alternatively, we may want to consider the 0 where case Gnj =
=
-->
G,,(0)
=
.Q
7
.(n,(0)
-
F(ts/0))).
(2 1. 12)
>1
By considering (21.12)we divide the problem into two parts, the stochastic convergence of the sum of the mean deviations to zero, and the nonstochastic onvergence assumed in the definition of Q(0).This raises the separate question of whether the latter convergence is uniform, which is a matter for the problem at hand and will not concern us here. As we have seen in previous chapters, obedience to a law of large numbers calls for both the boundedness and the dependence of the sequence to be controlled. In the case of a function on 0- the dependence question presents no extra difficulty; for example, if wr(01)is a mixing or near-epoch dependent array of a given for any 01, 02 l 0. But class, the property will generally be sharid by qnt' the existence of pmicular moments is clearly not independent of 8. If there ,
Unijrm
Stochastic Convergence
331
< exists a positive array (D,,f) such that Iqnte) I S Dnt for a1l 0 G 0- and IIDn,lIr uniformly in t and n, qnt) is said to be Lr-dominated. To ensure pointwise x, convergence on 0- we need to postulate the existence of a dominating array. There is no problem if the qnt) are bounded functions of 0. More generally it is necessary to bound 0- but since 0- will often have to be bounded for a different set of reasons, tlzis does not necessarily present an additional restriction. ,
,
,
on the dependence plus suitable domination conditions, pointwise stochastic convergence follows by considering (Gn(0)) as an ordinary stochastic sequence, for each 0 e 0- However, this line of argument does not juarantee that there is a minimum rate of convergence which applies for a1l 0, the condition of uniform convergence. If pointwise convergenceof (Gn(0)) to the limit G(0) is defined by Given restrictions
.
Gn(0)
Lp, or in pr.), each 0 e 0-
(a.s., in
0
..-0
(21.13)
,
a sequence of stochastic functions (G,,(0) ) is said to converge
lznt/rpz/y
Lp, or in pr.) on 0- if
sup lG,,(0) I 9e (-) To appreciate
0
.--y
(a.s.,in
L;,, or in pr.).
(a.s.,in (21.14)
the difference, consider the following example.
21.4 Example Let 0gnt)
ht =
define a zero-mean
g0,x),and
=
-
n
(%,(0)l
where
0 f 0 f 1/2?z
Z0, Z(3In
+
an'ay
-
0), 1/2n $In
0,
0 f $In
<
<
0
<
(21.15)
cxa
where (/ltJis a zero-mean stochastic sequence, and Z is a binary r.v. with #(Z 1 1) PZ Z:=1%(0) Hn + #n(0), where Hn ?1-1E' t 1h tn J. Then Ga(0) and =
-1)
=
=
=
=
=
Z(1 rl0), Slln
=
=
0 f 0 f 1/2s
Zn, Kn)
=
1/n
0
0 f Sln.
<
-
<
0
(21.16)
< x -61-.:
Hn for 0 > 1/n as well as for 0 0, Gn(0) We assume Hn -E1-> 0. Since Gn) 0 for each fixed 0 e 0. ln other words, Gn) converges pointwise to zero, a.s. However, supese IKn) I Izlzl = ,1 for every n 1. Because Hn converges a.s. there will exist N such that lHn I < :1 for all n k N, with probability 1. You can veri that when IHn I < :1the supremum on e of 1Hn + Kn) I is always attained at 1/2.n. Hence, Fith probability 1, the point 0 =
=
=
=
sup IGn(0) I es e
=
IHn +z1zIfor
n k N,
The fzzw of f-zzr'd Numbers
332
as n
--
It follows that the unifonn Similarly, for n k N,
=.
a.s. limit of Gn)
P sup IGn(0) I k e:
=
(4G O
(21. 17)
,.-3
is not zero.
#( IHn +rlzl
2 E).
--+#( 1z I 2 e)
=
(21.18)
1
,
so that the uniform probability limit is not zero either, although the pointwise probability limit must equal the pointwise a.s. limit. n on uniform a.s. convergence is a classic of the probability literature, the Glivenko-cantelli theorem. This is also of interest as being a case outside the class of f'unctions we shall subsequently consider. For a collection of identically distributed r.v.s (.)((l)),...,A7a((t))) on the probability space (D,F,#), the empirical distribution functionis defined as Our
tirst result
Fn(x,tn)
n
1771(-xx(,X()). n ,-1
--
=
(21.19)
ln other words, the random variable Fnlx,k is the relative frequency of the variables in the set not exceeding x. A natural question to pose is whether (and in what sense) Fn converges to F, the true marginal c.d.f. for the distribution. For fixed x, (&(x,(l)) )Tis a stochastic sequence, the sample mean of n Bernrandom variables which take the value 1 with probability F(x) oulli-distributed and 0 otherwise. lf these form a stationary ergodic sequence, for example, we know F@) a.s. for each x e R. We may say that the strong law of large that Fnlxbl numbers holds pointwise on R in such a case. Convergence is achieved at x for a1l ) e G, where #(G) 1. The problem is that to say that tbefunctions Fn converge a.s. requires that a.s. convergence is achieved at each of an uncountable set of points. We cannot appeal to 3.6(iii) to claim that pt;)xs IRCx) 1, and hence the F@) with probability 1 at a point not specsed assertion that Fnlx, beforehand cannot be proved in this manner. This is a problem for a.s. convergence additional to the possibility of convergence breaking down at certain points of the parameter space, illustrated by 21.4. However, uniform convergence is the condition that suffices to rule out either difficulty. ln this case, thanks to the special form of the c.d.f. which as we know is bounded, monotone, and right-continuous, uniform continuity can be proved by establishing a.s. convergence just at a countable collection of points of R. -->
=
=
.x
--
21.5 Glivenko-cantelli
theorem If Fnx,k
lFnlx,k sup A
-
F@) l
Proof First define, in parallel with Fn,
--
.-->
F(x) a.s. pointwise, for x e R, then 0 a.s.
I:a
(21.20)
Uniform Stochastic Convergence
333
n
F'(x,(,)) n
=
1771(--,x)(x,())),
-
(21.21)
f=1
and note that Fa'@,() integer m > 1 1et
F@-) for a1l
---2,
xjm inf (.xe R: F(x) k
)
in a set Cx', where #(Cx')
.j//n),
j
=
and also let x()r,,
and xmm
-x
=
Fxjm-)
-
1,
-
1. For an
(21.22)
so that, by construction,
+=,
=
1,...,-
=
=
Fxj-j,m) < Slm, j
=
(21.23)
1,...,-.
Lastly let M=((,))
1Fnxjm,k
maxmaxl
=
Sjnm
Then, for j F(x)
-
--
1
=
1,...,- and x
G
-
Fxjm)
I lFn'xjm.k
Ilj.
Fxjm-)
-
,
(21.24)
(xy-1,m,.xs),
Mmnk f Fxj-jvm) Mmnt -
S Fnxj-k,m.q S F%m-)
+
f Fn@.)
Mmnl
S Fn'xjm.)
f F(x)
j +
-+
Mmnlk.
(21.25)
F@) I f 3lm + Mmn That is to say, IFa@,) for every x e R By pointwise strong convergence we may say that limn-yxl/knt) 0 for finite m, (J) Cm* where hence that limn-yxsupx for al1 F(x) 1/-, Fnx, and l l; -
.
=
-
,
c*n1
=
Otcx mj
j=1
But #tlimm-+xCk')
=
fn
cx' ).
(21.26)
m
1 by 3.6(iii), and this completes the proof.
.
Another, quite separate problem calling for unifonn convergence is when a sample statistic is not merely a stochastic function of parameters, but is to be evaluated at a random point in the parameter space. Estimates of covariance matrices of estimators generally have this character, for example. One way such estimates are obtained is as the inverted negative Hessian matrix of the associated sample log-likelihood function, evaluated at estimated parameter values. The problem of provingconsistencyinvolvestwodistinctstochasticconvergencephenomena, andit does not suffice to appeal to an ordina/ 1aw of large numbers to establish convrgence to the true function evaluated at the true point. The following theorem gives suftkient conditions for the double convergence to hold.
21.6 Theorem Let (L,5,P4 be a probability space and (O- a metric space, and let Qn10- x F-+ R be T/f-measurable for each 8 e 0. If (a) 0n*-E'2 %, and (b) :s(0) -.EI.+:(0) unifonnly on an open set Bj containing %, where :(0) is a nonstochastic function continuous at %, ,p)
The zzw of
334
Qn(;. n)
then
..s.o
Numbers
-wrgc
rj()(j;.
Proof Uniform convergence in probability of Qnon Bo implies that, for any E > 0 and 8 > 0, there exists Nj k 1 large enough that, for n k Nj ,
P
Sup
0
E5
-
Bz
-,1.
r1E
I:,,(0) Q(0)I <
(2 1.27)
2 1
Aldo, since 0n*-T-r-> 0o, there exists N2 such that, for zl 2 N1,
#(0
e #0) 2 1 14.
(21.28)
-
To consider the joint occurrence of these two events, use the elementary relation P(A
#) k PA) +PB4
/'7
1.
-
(21.29)
Since
(0n*e for n
#ol
f''h
sup I:a(0) t)E5BQ
:(0()1 <
-
Y
l l:,,(0) Q(0n*)l< Y) (21.30)
c
-
,
k max(N1,N2),
#1 lQn(0n*)Q(0,1) l<
/1
-
Using continuity at
/$r3,
% and
2(1
t)
-
-/.
1
-
(21.31)
1
=
18.10(ii), there exists Nz large enough that, for n k
#( I:(0n*) Q(0())I < /) -
2 1
(21.32)
.
By the triangle inequality,
IQn(M,,*):(%*)l + l:(0,,*) Q(0c)l -
-
l:,/0,1) :()0()281 -
(21.33)
and hence
l I:,;(0n*):(0n*)l <
el
-
fa
( l:(%*) :(00) l < El < c ( I:a(0n*)p(0t))l : l -
-
Applying
(21.29)again
.
(2 l
.34)
gives, for n k max(NI,N2,Na),
#(I :n(0n*):(041 -
< E)
1
-
.
The theorem follows since 8 and e: are arbitrary.
(21.35)
.
Notice why we need unifonn convergence here. Pointwise convergence wold not allow us to assert (21.27)for a single 11 which works for a1I 0 s B). There would be the lisk of a sequence of points existing in Bo on which Nj is diverging, Suppose % = 0 and Gn) = :s(0) Q(0)in 21.4. A sequence approaching %, say ( 1/m, m e IN), has this property; we should have -
#( IQnlm) for arbitrary
6: >
0 and
45
>
-
:(1/,/7)j
<
/)
k 1 1:4 -
(21.36)
0, only for n > m. Therefore we would not be able to
Unrm
Stochastic Convergence
335
claim the existence of a finite n for which (21.31) holds, and the proof collapses. In this example, the sequence of functions (G,,(0) ) is continuous'for each n, but the continuity breaks down in the limit. This points to a link between uniform convergence alzd continuity. We had no need of continuity to prove the GlivenkoCantelli theorem, but the c.d.f. is rather a special type of function, with its behaviour at discontinuities (andelsewhere) subject to tight limitations. In the wider class of functions, not necessarily bounded and monotone, continuity is the condition that has generally been exploited to get uniform convergence results.
21.3 Stochastic Equicontinuity Example 21.4 is characterized by the breakdown of continuity in the limit of the sequence of continuous functions. We may conjecture that to impose continuity unifonnly over the sequence would suffice to eliminate failures of uniform convergence. A natural comparison to draw is with the uniform integrability property of sequepces, but we have to be careful with our terminology because, of course, uniform continuity is a well-established tenu for sometlling completely different. The concept we require is equicontinuity, or, to be more precise, asymptotic Ikplt/b/-/?/ equicontinuity', see (5.47).Our results will be based on the following of the Arzel-Ascoli theorem (5.28). version functions on a 21.7 Theorem Let (A(0),n q (N) be sequence of (nonstochastic) 0 if and only if (O,p). Then, bounded totally parameter space 1 supeee l 0 for a11 0 e 0-e, where 0-t) is a dense subset of 0, and (/ l is fno) asymptotically unifonnly equicontinuous. u .fn(0)
-->
--->
The set F t/'n,n e IN) QJ (01,endowed with the uniform metric, is a subspace of (Co,Jg), and by definition, convergence of fn to 0 in the unifonn metlic is the same thing as unifonn convergence on 0- According to 5.12, compactness of F is equivalent to the property that every sequence in F has a cluster point. ln view of the pointwise convergence, the cluster point must be unique and equal to 0, so that the conclusion of this theorem is really identical with the Arzel-Ascoli theorem, although the method of proof will be adapted to the present case. Where convenient, we shall use the notation =
.
w(/k,)
=
sup 0 (5 O
0/G
sup
510,8)
IJn(0') fn) 1
(21.37)
-
.
The function w(A,.):R+ F-+ R+ is called the modulus ofcontinuity of fn. Asympof the sequence (Jn) is the property that totic unifonn equicntinuity limsupnwt/k,8) 1 0 as 0. .1'
Proof of 21.7 To prove
iif'
:
satisfy
given
e: > 0, there exists by assumption
limsup w(/,) n--x
<
e.
:
>
0 to
(21.38)
The fww of fzzr'c Numbers
336
since 0- is totally bounded, it has a cover f5(f,&2), i 1,...,mJ.For each i, choosep e 0-n such that p(0sp) < /2 (possiblebecause 0-() is dense in 0-) and i 1,...,mJis also a cover for 0- Every 0 0--is contained note that (5'(0f,8), =
ad
=
.
N
in 5'(0/,) for some i, and for this i,
lJn(0)1f
1Jn(0')l
sup
e's
S
-s('4'j,)
Ifne'q .f(pf)I+ IJ(1) I
sup
-
.
.s'(;J,)
e'<
(2 1.39)
We can therefore write
supIfnet I <
()ce
J(pf)I+
.&(e')
max 0'e
liir?z
s w(.&-)
+
sup
-
.('j,)
max l<
max
I.f(4f)I
IJ(pf)I (21.40)
.
l Ki < m
Sufficiency follows on taking the limsup of both sides of this inequality. donly ito follows simply from the facts that unifonn convergence entails pointwise convergence, and that
w(.&,)S
2 sup 0e e
1.f(0)1.
(21.41)
.
To apply this result to the stochastic convergence problem. we must define concepts of stochastic equicontinuity. Several such definitions can be devised, of which we shall give only two: respectively, a weak convergence (in pr.) and a strong convergence (a.s.) variant. Let (O,p) be a metric space and L'j,5,P4 a probability space, and 1et (Gu(0,)), n e IN) be a sequence of stochastic functions Gn O x f F-> R, F/f-measurable for each 0 e 0. The sequence is said to be asympstochastically equicontinuous (in pr.) if for a1l e > 0 H 8 > totically I/nt/brl/y 0 such that
limsup PwGnbn)
2 :)
<
s.
(21.42)
n-x
And it is said to be strongly asymptotically uous if for all 6: > 0 R > 0 such that P limsup w(tL,) 2 :
rfnt/br/nfy stochastically equicontin-
=
0.
(21.43)
n-@x
Clearly, there is a bit of a terminology problem here! The qualifiers asymptotic' and unifonn' will be adopted in al1 the applications in this chapter, so 1et these be understood, and let us speak simply of stochastic equicontinuity and strong stochastic equicontinuity. The abbreviations s.e. and s.s.e. will sometimes
be used.
Unrm
Stochastic Convergence
337
21.4 Generic Uniform Convergence Uniform convergence results and their application in econometrics have been researched by several authors including Hoadley (1971),Bierens (1989),Andrews (1987a, 1992),Newey (lggll,andptscherandpruchatlg8g,lgg4l.Thematerialin the remainder of this chapter is drawn mainly from the work of Andrews and Ptscher and Prucha, who have pioneered alternative approaches to deriving tgeneric' uniform convergence theorems, applicable in a variety of modelling
situations.
These methods rely on establishing a stochastic equicontinuity condition. Thus, once we have 21.7, the proof of unifonn almost sure convergence is direct.
21.8 Theorem Let (Gn(0), n e EN ) be a sequence of stochastic real-valued functions metric bounded totally space (O,p). Then on a -f.-
suplG,,(0)I
0
(21.44)
()EEO
if and only if 0 for each 0 q 0-0, where 0-t) is a dense subset of 0(a) Gn) (b) (Gn) is strongly stochastically equicontinuous. --f.1..:
,
Proof Because (O- is totally bounded it is separable (5.7)and (% can be chosen to be a countable set, say 0-() (01,k G EN ) Condition (a) means that for k with is 0 for (.t) e Ck. 1 such that Gnk, Pc 1,2,... there a set Ck Condition (b)means that the sequences ( Gn J are asymptotically equicontinuous 1. By the suftkiency part of 21.7, supaso IGn(0,tt)) I for all ) e C, with PC') 0 for (l) e C* O';=lC fa C. PC*) 1 by 3.6(iii), proving if' .-
=
=
.
-->
=
=
=
c*
=
.
.
The corresponding probability' result follows very similar lines. The proof 21.7 quite so directly, but the family resemblance in the arguments cannot exploit will be noted. tin
21.9 Theorem Let tGn(0), n e ENl be asequence of stochastic real-valued functions on a totally bounded metric space (O,p). Then
IGn(e) I sup ()s O
.-J'-'.'->
0
(21.45)
if and only if (a) Gn) .-C1.+ 0 for each 0 l 0-(), where 0-() is a dense subset of 0(b) fGnl is stochastically equicontinuous. Proof To show let (5'(4f,8), 1,...,mJwith 4f G 0-() be a finite cover for ().This exists by the assumption of total boundedness and the argument used in the prpof of 21.7. Then, ,
'if'
,
=
338
P
Law of fzlrge Numbers
Fe
tsus polGnt
l
2
2E)
sup ( lG,t@') Gni) ezs.('4f,)
; P max
I + !Gn(0f) I) k
-
Lsism
< #(w(Gn,) < P(wGn,&)
E) +
P max lGnlui) I
e
1S K m
k
< #(w(Gn,)
e) + P
2:
U(iG,/VI
e.)
f=1
:) +
tA(4f)l X#41 izz,1
:),
(21.46)
where we used the fact that fx-y k 2EJ c (x k e)
l..p
(y 2
:)
(21.47)
for real numbers x and y, to get the third inequality. Taking the limsup of both sides of (21.46),(a) and (b) imply that limsup # sup IGn(0) j k 2: n-kcn
(21.48)
0e O
if', pointwise convergence follows immediately from uniform convergence, so it remains to show that s.e. holds; but this follows easily in view of the fact (see (21.41))that To prove
tonly
#(w(Gn,)
k :) S P sup 8e O
lGnl) 1 2
:/2
(21.49)
w
.
There is no loss of generality in considering the case Gn 0 in these theorems. We canjust as easily apply them to the case where Gn(0) = @n(0) :n(0) and Qnis a nonstoehastie function which may really depend on n, or just be a limit funcIn the fonner case there is no need for Qnto converge, as tion so that '-Q n long as Qn Qndoes. Applying the triangle inequality and taking complements in (21.47), we obtain -->
-
-:
=
5
-
-
lwton, ) < eJ fa (w(-p ) < 2e). (21.50) n, n, ) < EJ c fwtpu This means that (Qn -Qn ) is s.e., or s.s.e. as the case may be, provided that and f-:n) is asymptotically equicontinuous in the ordinary (:?:) is s.e., or s.s.e., senseof j5.5. This extesion of 21.8 is obvious, and in 21.9 we can insert the step 2 2E) S #(w(:,,,8) 2 E) + 1(w(7n,)E) (21.51) #(w(Qn Ua,8)
--
-
-
into (21.46),where the second tenu on the right is 0 or 1 depending on whether the indicated nonstochastic condition holds, and this tenn will vanish when n 2 N
Stochastic Convergence
Unbrm
for some N k 1, by assumption.
The s.e. and s.s.e. conditions may not be particularly easy to verify directly, and the existence of Lipschitz-type sufficient conditions could then be very convenient. Andrews (1992)suggests conditions of the following sort.
21.10 Theorem
Suppose there exists N 2 1 such that
IQn(6') Qn(0)l K #n(p(0,0')), -
(21.52)
a.s.
holds for all 0,0' e 0- and n k N, where h is nonstochastic and hx) 1 0 as x 1 0, and (Bn l is a stochastic sequence not depending on 0. Then (i) (Qn) is s.e. if Bn 0r,(1). (ii) (:,,) is s.s.e. if limsupsla < x, a.s. =
Proof The definitions imply that w(Qn,6) < Bnhln) a.s. for n 2 N. To prove > > that, for 0 and 0, 8 note any s
limsup #(w(:n,6)
k
:)
< limsup PBn k
n--)x
E//l(8)).
(i),
(21.53)
n-ex
By definition of Op(1), the right-hand side can be made arbitrarily small by choosing Elh&) large enough. ln particular, fix e > 0, and then by definition of < :. h we may take 8 small enough that limsupn--pxf'tf,l k :/48)) For (ii), we have in the same way that, for small enough 8, P limsup
w(Qn,8)2
is
f P limsup Bn k
:/(6)
< E. .
(21.54)
N'-/*
n--oo
A sufticient condition for Bn Op(1) is to have Bn unifonnly bounded in f,1 norm, i.e., supnElBn) < x (see 12.11), and it is sufticient for limsupnfn to be a.s. 0. bounded if, in addition to this, Bn E(Bn) The conditions of 21.10 offer a striking contrast in restrictiveness. Think of (21.52) as a continuity condition, which says that Qn(0')must be close to Qn(0) when 0' is close to 0. When Qnis stochastic these conditions are vel'y hard to satisfy fojxed Bn, because random changes of scale may lead the condition to be is a continuous function for a1l (l) and violated from time to time even if Q,;(6,(t)) n. The purpose of the factor Bn is to allow for such random scale variations. Under s.e., we require tat the probability of large variations declines as their magnitude increases', this is what Op(l) means. But in the s.s.e. case, the requirement that (#n) be bounded a.s. except for at most a finite number of terms implies that fQn) must satisfy the same condition. This is very restrictive. lt means for example that Q,,(0)cannot be Gaussian, nor have any other distribution with infinite support. ln such a case, no matter what (#,,) and h were chosen, the conditiop in (21.52)would be violated eventually. It does not matter that the probability of large deviations might be extremely small, because over an infinite number of sequence coordinates they will still arise with probability 1. Thus, strong uniform convergence is a phenomenon confined, as far as we are able to show, to a.s. bounded sequences. Although (21.52)is only a sufficient condition, it can be verified that this feature of s.s.e. is implicit in the =
.-61-.:
-
The frw of fzlr'e
340
Numbers
definition. This fact puts the relative merits of working with strong and weak laws of large numbers in a new light. The former are simply not available in many important cases. Fortunately, in probability' results are often sufficient for the purpose at hand, for example, detennining the limits in distribution of estimators and sample statistics', see j25. 1 for more details. Supposing (O- c ('R ,dE), suppose further that Qv(M)is differentiable a.s. at each point of 0-; to be precise, we must specify differentiability a.s. at each point of an open convex set 0-*conta i n in g 0.. (A set B c R is said to be convex if The mean value theorem x e B and y e B imply kr + (1 h)y e B for e (0,1).) * 22 yields the result that, at a pair of points 0,0 z e o,,p)
.
-
,
Qn (O)
-
Qn(O')
k Qn
77 y@y
=
f=1
(0 0r,) a.s., -
jsj;v
(21.55)
where e* e 0- is a point on the line segment joining 8 and O', which exists by *. convexity of 0- Applying the Cauchy-schwartz inequality, we get *
l Qa(0) -
k
j ljjo: jaQn ef
l K Y-% j
Qn(O')
.e.
.
.
i=1
< fnlle
.,r
j ().(j
-
e'll
.
a.s.,
j (2 1
.56)
where Bn
()
*
G
O
j' jj
jj;E) jk;pr ?:
sup
=
*
PO
.
e=e.
(2 1.57)
41jldenotes the Euclidean
length, and onlB is the gradient vector whose elements are the partials of Qn with respect to the 0j. Clearly, (21.52) is satisfied by taking h as the identity function, and Bn defined in (21.57) is a random variable for all rl. Subject to this condition, and Bn satisfying the conditions specified in 21.10, a.s. differentiability emerges as a sufficient condition for s.e.. Here
.
21.5 Uniform Laws of Large Numbers In the last section it was shown that stochastic equicontinuity (strongor in pr.) is a necessary and sufficient condition to go from pointwise to uniform convergence (strongor in pr.). The next task is to find sufficient conditions for stochastic equicontinuity when (:n(0)l is a sequence of partial sums, and hence to derive uniform laws of large numbers. There are several possible approaches to this problem, of which perhaps the simplest is to establish the Lipschitz condition of 21.10.
21.11 Theorem Let ( (tsX(.t),0)l:=1 )''J.1denote a triangular array of real stochastic satisfying, for N k 1, functions with domain (O,p),
Unform Stochastic Convergence
1qntl
?n,(0)
-
lS
#n/(p(0,0')),
341
(21.58)
a.s.,
for all 0,0' e O and n k N, where h is nonstochastic and hx) (#n,l is a stochastic array not depending on 0 with 1=kEBnt) ZVlnf(0)'then (i) Qnis s.e.; 0. (ii) Qnis s.s.e. if lzzkBnt ElBntjj
.1,
0 as 1 0, and /(1). If :n(0) .x
=
=
-61--;/
-
is only necessary by 21.10(i) and the triangle inequality to 0,(1). This follows from the stated condition by the establish that XlctBnt Markov inequality. Likewise, (ii) follows directly from 21.10(ii). w
Proof For
(i) it
=
A second class of conditions is obtained by applying a form of s.e. to the summands. For these results we need to specify Gn to be an unweighted average of n functions, since the conditions to be imposed take the form of Cesro summability of certain related sequences. It is convenient to confine attention to the case 1
Gn(,0)
=
-
n
F7(tp(X,(),0)
-
F(/X,,0))),
(21.59)
>1
where Xt e X is a randpm element drawn from the probability space (X,T,gf). Typically, though not necessarily, Xt is a vector of real r.v.s with X a subset of R''', m k 1, t' being the restriction of Bm to X. The point here is not to restrict but to specify the existence the fonn of the functional relation between qt and 6 P(Xt A) for A e T. The usual of marginal derived measures g,f,with gXA) context will have Gn the sample average of functions that are stochastic through their dependence on some kind of data set, indexed on t. The functions themselves, not just their arguments, can be different for different t. We must find conditions on both the functions qt(.,.) and the p.m.s j which yield the s.e. condition on Gn. The first stage of the argument is to establish conditions on the stochastic functions tp(0) which have to be satisfied for s.e. to hold. Andrews (1992)gives the following result. ,
=
21.12 Theorem lf (a) there exists a positive stochastic
supIqtt I S
sequence
fdt)
satisfying
(21.60)
dt, all t
: s (.) and
limsup n--yx
(b) for every
: >
n 1 TEdt n >1
-
1(>v))
0, there exists
-->
>
0 as M
-->
n
n-yx
>1
(2 1.6 1)
0 such that
> :) limsupn1 77#(w(tp,6) -
x;
<
e',
(21.62)
The fxzw of fxzrgc Numbers
342 then Gn is s.e.
l:n
Condition (21.61)is an interesting Cesro-sum variation on unifonn integrability, and actual uniform integrability of (4) is sufticient, although not necessary. Condition (a) is a domination condition, while condition (b)is called by Andrews
termwise stochastic
equicontinuity.
Proof Given : > 0, choose Msuch that limsupn-oxn -1' then 6 such that
limsup 1 --
n
n-+x
?=I F(2J t 1(u ,
>>))
< :1:2, and
-n
> 77#(w(t.p,) >1
E2) <
jM-1:2.
(21.63)
The first thing to note is that
w((t.n
'(f.n)),)
-
w(F(tp),6)
S w(,,)
+
S w(,,)
+F(w(,,)),
(2 1.64)
where the last inequality is an application of 21.3. Applying (21.64)and using
Markov's
inequality,
#(w(tA,)
>
e) S #
1 .-Q
.-wttp
-
Eqt),nj
> 6r
>1
s
,(!,y) &
2
>1
(w(o,)+s(w(.)))
>
j
n
s -Xs(w(,,)) nc >1 n
o =ns77e-(w(.p,)(1fw(t,,,)xc2s) >1 -
+ 1fs2/6<w(s8)&/')).1,
(21.65)
where the indicator functions in the last member add up to 1. Using the fact that w(t.?s8)S ldt, and hence fwttsl > AfI (24 > AJ'),and taking the limsup, we now obtain
limsup#(w(Ga,) n-+x
> E)
<
2 e2 +
-
-j-
Mlimsup
-
1
n
X# w(t?f,8)
>
p
-j-
>1
n-x
n
+
1 limsup -XF(2J,1(>,>>j) n-yco
n
/=1
(21.66) in viesv of the values chosen for 3T and
.
.
Unrm
Stochastic Convergence
21.12(a) is satisfied depends on both the distribution of Xt and functional form of qt(.4. But something relatively general 21.12(b)). Assume, following Ptscher can be said about tenuwise s.e. (condition and Prucha (1989),that Clearly,
whether
condition
p .?,(x-e)>7rk,(x),1,@,Q), =1
(21.67)
=
R, and X R for fixed 0, are T/f-measurable functions. The idea here is that we can be more liberal in the behaviour allowed to the factors rkt as functions of Xt than to the factors skt; discontinuities are permitted, for example. To be exact, we shall be content to have the rkt uniformly fvl-bounded in Cesro mean:
where rktk X
.k,4.,0):
-->
--
1
sup n -
n
n
FIrktxt) I s 77 >1
B
<
k
cs
1,...,p.
=
(21.68)
As to the factors Jkr@,0), we need these to be asymptotically equicontinuous for a sufficiently large set of x values. Assume there is a sequence of sets (Km 6 T, m 1,2,...J,such that =
n
limsup1X p,rtA1) n >1 n-+x --
and that for each m 2 1 and s
>
--+
n-koo
k
< :,
(21.69)
--.>
x,
0, there exists 8
limsup sup w(sjz(x,.),8) A G
0 as m
=
>
0 such that
1,...,p.
(2 1.70)
Km
(21.70)is a nnstochastic equicontinuity condition, but under condition (21.69)it holds (as one might say) almost surely, on average' when the r.v.
Notice that
Xt is substituted into the formula. These conditions suffice to give tennwise s.e., and hence can be used to prove s.e. of Gn by application of 21.12.
21.13 Theorem If qt(Xt,Q)is defined by (21.67),and (21.68),(21.69),and (21.70) hold, then for every : > 0 there exists 6 > 0 such that n
1 limsup XP(w(t.p,8) -
>1
n-yx
Proof Fix :
>
> E) <
e.
0, and first note that
#(wtr-l>
:) S
P
(F-Iq,l
wtsal
>
&1
,(
s
&1
fIr,-Iw(,,,,)>
j s/,I)
(21.71)
The fzzw of fxzr'c Numbers p
rkrl w(.k,,) < 77#(1 &1
>
zlp4
g,(Ir,,1w(,.,,)1,u x'ss s >
&1
J
+P
(1
)
qll w(.%,,)1rj,
>
lp
.
(21.72)
Consider any one of these p tenns. Choose m large enough that n
: 1 < limsup-Xw(#kc) l-p n-x
n
(21.73)
,
>1
and for this m choose 8 small enough that
limsupsup w(x@,.),) nKm x xs
<
2 4Bp 2
(21.74)
.
Then, by the Markov inequality,
n limsup1 X P lrkt 1w(.q,,)1Au n->x n ,= 1 -
>
K limsup s-.I'Z--+ n
s f
n
j
:
'*
1 -
qJl
n
EFI
q,l
>1
n--yx
4Bp z
,=
x
limsup
vz
1 X# 1
--
>
:
rg
:
zpp (21.75)
-,
2#
and by (21.73), limsup n-/x
-
1
n
X# >1
Irpl wt-qsllrj,
>
:
yp
f limsup
1
n
XPCXt
n-+x
<-;*
Substituting these bounds into
-a
(21.76)
#
(21.72)yields the
Km4
>1
result. w
V THE CENTRAL LIMIT THEOREM
22 Weak Convergence of Distributions
22. 1 Basic Concepts The objects we examine in this pal't of the book are not sequences of random variables, but sequences of marginal distribution functions. There will of course be associated sequences of nv.s generated from these distributions, but the concept of convergence arising here is quite distinct. Formally, if fFn JTis a sequence of c.d.f.s, we say that the sequence converges weakly to a limit F if Fn@) .-.A F@) pointwise for each x e C, where C ? R is the set of points at which F is continuous. Then, if Xn has c.d.f. Fn and X has c.d.f. F, we say that Xn converges in distribution to X. These terms are in practice used more or less interchangeably for the distributions and associated l'.v.s. Equivalent notations for weak convergence are Fn = F, and Xn .-C->X. Although the latter notation is customary, it is also slightly irregular, since to say a sequence of nv.s converges in distribution means only that the limiting r.v. has the given distribution. If both X and F have the distribution specitied by F, then Xn -P-> X and Xn -P-> F are equivalent statements. Moreover, we write things like Xn -P- N(0,1) to indicate that the limiting distribution is standard Gaussian, although #(0,1)' is shorthand for r.v. having the standard Gaussian distribution' ; it does not denote a particular r.v.. Also used by some authors is the standing for convergence in probability law', but we avoid this notation form because of possible confusion with convergence in fp-norm. Pointwise convergence of the distribution functions is all that is needed, rememberig that F is non-decreasing, bounded by 0 and 1, and that every point is either a cpntinuity point or a jump point. lt is possible that F could possess > jump at a point zb which is a continuity point of Fn for all finite n, and in these cases Fnxo) dpes not have a unique limit since any point between F(.zb-) and F(ab) is a candidate. But the jump points of F are at most countable in number, and according to 8.4 the true F can be constructed by assigning the value Fxo) at every jump point xo; hence, the above definition is adequate. If g. represents the corresponding probability measure such that F@) g,((-x, x1) for each e R, we know (seej8.2) that g. and F am equivalent representations of the same measure, and similarly for g. and Fn. Hence, the statement gn cuu) g,is equivalent to Fn F. The corresponding notion of weak convergence for the sequence of measures (jtnl is given by the following theorem. Ea
i-1-+'
=
.x
::uuh
22.1 Theorem g.
=
p, iff gn(A)
-->
p,(A) for every A e B for which j@A)
=
0. rl
The proof of this theorem is postponed to a later point in the development. Note
The Central Limit Theorem
348
meanwhile thatthe exclusion of events whose boundary points have positive probability corresponds to the exclusion of jump points of F, where the events in question have the form ((-x,x11. Just as the theol'y of the expectation is an application of the general theory of integrals, so the theory of weak convergence is a general theory for sequences of finite measures. The results below do not generally depend upon the condition gz(El = 1 for their validity, provided definitions are adjusted appropriately. However, a serious concern of the theory is whether a sequence of distribution functions has a distribution function as its limit; more specifically, should it follow because gw(R) 1 for evel'y n that g,@) = 1:7This is a question that is taken up in j22.5. Meanwhile, the reader should not be distracted by the use of the convenient notations f(.) and #(.) from appreciating the generality of the theory. ,
=
22.2 Example Consider the sequence of binomial distributions fBn,onls n 1,2,3,...), where the probability of x successes in n Bernoulli trials is given by =
Pxn
S
x)
=
(bn)x (1 Vn4n-x
=
x
-
x
,
=
0,...,n
(22.j )
(see 8.7). Here, lv is a constant parameter, so that the probability of a success for every n. falls linearly as the number of trials increases. Note that Exn) (x'')n-A x, and taking the binomial expansion of For fixed 1/x! as n e-Z whereas (1 1/rI)-* 1. We (1 bn) n sh ows that (1 hlnln as n may therefore conclude that =
--)
.x,
--
-->
-->
-
-
=,
--
!'
Pxn
x)
=
-.->
V
-k
-e
.X
x
,
-
0, 1,2,...,
=
(22.2)
.
and accordingly, Fu(c)
:j2P:ln
=
x)
=
-+
,-l
;2
06x<J
-.
x
(22.3)
1'.
0KAfJ
at all points a < x. Thus the limit (and hence the weak limit) of the sequence fBn,hln) l is the Poisson distribution with parameter E1 .
22.3 Example A sequence of discrete distributions on g0,1) is defined by Pxn
=
x)
1/?z, x
=
=
iln ,
0,
i
=
1,...,n.
otherwise
(22.4)
Tls sequence actually converges weakly to Lebesguemeasure m on (0,1), although this fact may be less than obvious; it will be demonstrated below. For any x e where denotes the largest integer (0,1), g,n((0,xq) (nxl/n x = pl(E0,.t)), There for which less than nx. are sets convergence fails, notably the set :(0,1) 1), in view of the fact that gw(Qp,1)) = 1 for every n, of all rationals in and rrl(Qp,1)) 0. But Q((),1j 0,1J and ??l(:(Q(n,1y)) 1, thus the definition of weak convergence in 22.1 is not violated. D
r/lxl
-->
=
=
r0,
=
=
Nctzk Convergence of Distributions
349
Although convergence in disibution is fundamentally different from convergcan ence a.s. and in pr., the latter imply the former. In the next result, be substituted for *-P-%% by 18.5. :-.14'
,
22.4 Theorem If Xn -T-r-->X, then Xn -P-> X. Proof For E
0, we have
>
+ < PX < x+
Plfxn < x)
-xl
f-h
( I.'L
> :))
#(1Ak 71
:) +
(22.5)
-
where the events whose probabilities appear on the right-hand side of the inequality contain (and hence are at least as probable as) the corresponding events on 0 by hypothesis, and hence the left. #( IXn -Yl > E) -.-/
-
limsup Pxn S x) S PX f x + 0.
(22.6)
N.->=
Similarly, #(X f x E) -
#( (X f x
=
:)
-
f-'h
( lXn XI S el) -
+ #((.Y f Pxn
< x)
+
< x- 6:) f-h
l 1-L--Yl
> El)
#( 4Xn -Yl > E),
(22.7)
liminf Pxn
(22.8)
-
and so PX f x
-
:) S
S x).
n-+x
f x) Since 6: is arbitrary, it follows that limn--joofax'a PX < x) at evel'y point x = = E) #(X S x). This condition is for which PX x) 0, such that limozoptx f x weak equivalent to convergence. w =
-
=
The converse of 22.4 is not true in general, but the two conditions are equivalent when the probability limit in question is a constant. A degenerate distribution has the form
F(X
0, x
<
a
(22.9)
=
1, x 2 a
If a random variable is converging to a constant, its c.d.f. converges to the step function (22.94,through a sequence of the sort illustrated in Fig. 22.1.
22.5 Theorem Xn converges in probability to a constant a iff its c.d.f. converges to a step function with jump at a. Proof For any E
0
>
#( jXn a l -
< E)
=
Pa
-
:
S Xn f a
+ E)
Central Limit Theorem
Ihe
350
= Fna
+
e) Fna -
-
e)-).
Convergence to a step function withjump
(22.10)
+ :) limpi-.-yx/kttz
Fa + :) at a implies E)-) 0 for a1l e > 0. The suffi1, and similarly limn-yxFnttc :)-) = Fa ciency pal't follows from (22.10)and the definition of convergence in probability. x, For the necessity, let the left-hand side of (22.10)have a limit of 1 as n for all er > 0. This implies =
=
-
-
=
-->
,
1im LFna
+ :)
-
Fnla
-
e)-)j
=
1.
(22.1 1)
n'M
Since 0 f F S 1, (22.11) will be satistied for all er > 0 only if Fa) F(J-) 0, which detines the function in (22.9).K
=
1 and
=
Fn n=x
1
i
I
N
l@
n = nj
E
n=nz
X
l1 <
E
!
INxs I
0
=
nl
<
<
nz
...
n,
'
-x
J
X
Fig. 22.1.
22.2 The Skorokhod Representation Theorem Notwithstanding the fact that Xn -P-> X does not imply Xn -S1..y X, whenever a sequence of distributions fFn ) converges weakly to Fone can construct a sequence of r.v.s with distributions Fn, which converges almost surely to a limit having distribution F. Shown by Skorokhod (1956)in a more general context (seej26.6), this is an immensely useful fact for proving results about weak convergence. Consider the sequence LFnl converging to F. Each of these functions is a monotone mapping from V to the interval (0,1J.The idea is to invert this mapping. Let a random variable (l) be defined on the probability space ((0,1),fp,1J,??z), where Sp,1J is the Borel field on the unit interval and m is the Lebesgue measure. Define for (t e (0,11
L()
=
inftx:
(l)
S Fn@)).
(22.12)
In words, Fa is the random variable obtained by using the inverse distribution function to map from the uniform distribution on (0,1Jonto 2-,taking care of any discontinuities in % 1() (correspon ding to intervals with zero probability mass intimum Fn) by the of under taking the eligible values. l'n is therefore a nondecreasing, lzft-continuous function. Fig. 22.2 illustrates the construction,
k
'''
W'ctzkConvergence of Distributions
.
.)r .:..
'
t
,
C
essentially the same as used in the proof of 8.5 (compareFig. 8.2). When Fn has that Fn(Fn())) iscontinuities it is only possible to assert (by light-continuity)
.
tlj.... )
351
Fn(Fn(x)) K x, by left-continuity of Fn. The tirst important feature of the Skorokhod construction constant a e R,
. j ..
.
.
:
j
)
.j
whereas
is that, for any
-
PYnu)
< a)
=
#(
S Fnaj)
=
Fna),
(22. 13)
where the last equality follows from the fact that ) is uniformly distributed on (0, 1q. Thus, Fn is the c.d.f. of L. 23 Letting F be a c. d f and F the r.v. corresponding to F according to (22.12), the second important feature of the construction is contained in the following result. .
.
1
Fig. 22.2
22.6 Theorem If Fn
=
F then Fu ---) F a.s.g/? as n
--
x.
(:1
ln working through the proof, it may be helpful to check each assertion about the functions F and i' against the example in Fig. 22.3. This represents the extreme case where F, and hence also F, is a step function; of course, if F is everywhere continuouj and increasing, the mappings are 1-1 nd the problem becomes trivial. Proof Let ) be any continuity point of 1', excluding the end points 0 and 1. For E < x < F()). any e > 0, choose x as a continuity point of F satisfying l'4l such of the countability discontinuities point will always Given the of F, a exist, and according to the definition of F, it must have the property F(x) < (t). F(x), there will be n large enough that Fn(x) < ), and hence x < lf Fnlx) L(), by definition. We therefore have -
-->
F())
-
6: < x <
L().
(22.14)
exists, since e is arbitrary (22.14) allows us Without presuming that limn-yxlv) F4(l)). to conclude that liminfn-xFnt) Next, choose y as a continuity point of F satisfying F() < y < F()) + E. The properties of F give f F(F()) f FV).For lalge enough rl we must also have <
The Central Limit Theorem
352
&@), and hence, again by definition of l%, 1%() f A' < i'()l
(22.15)
+ C.
In the same way as before, we may conclude that limsupa--jxFat) S F(). i'(). superior and inferior limits are therefore equal, and limn-oxikt))
The
=
12.* jrp/
gj(j)
F
yt
' --------.---.------.--.------.---.--.-------.----.----..--
,
(j)
jsrtx )
lj!j! 1
'j
j
..-.-------...---.-.-.-----.-...-----.-------....-----------
-.-...-----
j
I
#
I
! l
-1-:! 1'4)
-
ir
!
:
X
j j
I l i
! l:
I l
j
l i
! 1-
1. I A' F4))
1,4),
i I ! I
'
I
j
j
l,
!
24
.1'u()')
+ ir
1Xa7) Fig. 22.3 This result only holds for continuity points of K However, there is a 1-1 correspondence between the discontinuity points of F and intervals having zero probability under g, in R. A collection of disjoint intervals on the line is at most countable (1.11),and hence the discontinuities of i' (plusthe points 0 and F w.p. 1 grzlj,as 1) are countable, and have Lebesgue measure zero. Hence, L ,--.k
asserted. w
In Fig. 22.3, notice how both functions take their values at the discontinuities lnequality (22.15) holds for > at the points marked A and B. Thus, F(F((0)) ), but need not hold for (Y, a discontinuity point. A counter-example is the sequence of functions Fa obtained by vertical translations of the fixed graph f'rom below, as illustrated. In this case L()') > F(') + : for every n. '
=
.
22.7 Corollary Detine random variables Fs', so that Fn'() 1%()) at each where 0 and 0 at discontinuity points and at the function is continuous, and Fj() F/()) for every (t) e 0,1), and 1. Define Y' similarly. If Fx = F then L'() Fa' F'. and the functions of and Fn F are distribution =
=
=
-->
Proof The convergence for every (l) is immediate. The equivalence of the distributions follows from 8.4, since the discontinuity points are countable and their complement is dense in g0,1j,by 2.10. w ln the form given, 22.6 does not generalize very easily to distributions in R k fo r k > 1, although a generalization does exist. This can be deduced as a special case of 26.25, which derives the Skorokhod representation for distributions on general metric spaces of suitable type. A tinal pbitlt to observe about Skorokhod's representation is its generalization
Gdtzk Convergence of Distributions
353
to any finite measure. If Fn is a non-decreasing right-continuous function with codomain gt7,:j,(22.12) defines a function 1%() on a measure space ((tz,:1,
gu,x,m), where m is Lebesgue measure as before. With appropriate modifications, all the foregoing remarks continue to apply in this case.
The following application of the Skorokhod representation yields a different,
but equivalent, characterization of weak convergence. 22.8 Theorem Xn -P-->X iff
limElfxnll
F(J(m)
=
(22.16)
n-jx
for every bounded, continuous real function f. n The necessity
half of this result is known as the Helly-Bray theorem.
Proof To prove sufficiency, constnlct an example. For a
1,
x < a -x)/8,
fx)
a
=
a
8
-
-
G
R and
>
0, let
6
< x f
(22.17)
c
0, indicator' of the set We call this the continuous function with the properties tsmoothed
Fna
)
-
Fa
By hypothesis,
S
jfdl
S Fna),
8) <
-
JJJF
a11 z7,
(22. 18)
S Fa).
-->
-
8) S
jfdF
(22. 19)
(22.20)
S liminf Fna).
n-A= ---
(See Fig. 22.4.) It is a
fdF, and hence
fdt-n
limsup Fna Letting
(-x, t7l.
n-A*
0, combining
(22.19)and (22.20)yields
limjup Fna-)
K
(22.21)
F(J),
n-/x
F(J-)
f liminf
(22.22)
Fn(J).
These inequalities show that limnFntll exists and is equal to F(J) whenever F(a-4 = Fa), that is, Fn = F. To prove necessity, let f be a bounded function whose points of discontinuity are contained in a set Dy, where gtDyl 0, g, being the p.m. such that F@) g,((-x,x1). When Fn = F (& being the c.d.f. of Xn and F that of Ar)1%/((t) F/()) for every (l) s (0,1J,where Fn'((t))and F'((1))are the Skorolhod variables defined in f(Y'4 a.s.lg) by 18.8(i). 22.7. Since r?z(): F'((l)) 6 Dy) g.(Dy) 0, f(Yn') AU(F')), or The bounded convergence theorem then implies Efl) =
=
.-+
-->
=
=
-->
The Central Limit Theorem
354
jflgdvmLb')
jfyldbk,
(22.23)
jydbn-yb jxdbnf-'
(22.24)
--
where g.a is the p.m. corresponding to Fn. But 9.6 allows us to write
jfll'tdbtnll't =
=
=
Efxntt'
.i
with a similar equality for ElfX)l. (The trivial change of dummy argument from y to is just to emphasize the equivalence of the two fonnulations.) Hence we have E(f%44 it/ AU(m).The result certainly holds for the case Dy 0, so is proved. . .x
-->
=
Konly
' )' 1
-
.
-
1
t l l 1 ! 7 t-
tz-
x
a
Fig. 22.4 for the fundamental Notice how the proof cleverly substitutes (E0, 1),Sp,11,>?1) probability space (L1,5,P4 generating f-L), exploiting the fact that the derived distributions are the same. This result does not say that the expectations converge only for bounded continuous functions; it is simply that convergence is implied at least for al1 members of this large class of functions. The theorem also holds if we substitute any subclass of the class of bounded continuous functions which contains at least the smoothed indicator functions of half-lines, for example the bounded unifonnly continuous functions.
22.9 Example We now give the promised proof of weak convergence for 22.3. Clearly, in that example, 1
n
.fdg.n Tfiln). =
(22.25)
-
i= 1
The limit of the expression on the right of (22.25)as n c,o is by definition the Riemann integral of f on the unit interval. Since this agrees with the Lebesgue integral, we have a proof of weak convergence in this case. In -->
We shall subsequently require the generalization of Theorem 22.8 to general finite measures. This will be stated as a corollary, the modifications to the proof being left to the reader to supply; it is mainly a matter pf modifying the notation to
suit.
Gctzk Convergence of Distributions
355
22.10 Corollary Let (&) be a sequence of bounded, non-decreasing, continuous functions. Fn = F if and only if jl#Fn
right-
JJJF
(22.26)
--
for every bounded, continuous real function J.
E1
Another proof which was deferred earlier can now be given. Proof of 22.1To shtw sufficiency, considerA (-c,o, for which JA (xJ. Weak Aj p,(((-x,xj J) whenever convergence is detined by the condition gw(((-x, )) g,((xJ) 0. To show necessity, consider in the necessity pal't of 22.8 the case J(x) 1x@) for any A e :B. The discontinuity points of this function are 0, we have jtntAl g,(A) as a case of contained in the set A, and if jt(A) when Fn = F. . (22.16), .x1,
=
=
--
=
=
---)
=
22.3 Weak Convergence and Transformations The next result might be thought of as the weak convergence
counterpart of 18.8.
22.11 Continuous mapping theorem Let : R F- R be Borel-measurable with -1 0. If gz = g,, then gw/l discontinuity points confined to a set Dl. where blDR) -1 z:qhyt =
.
hY') Proof By the argument used to prove the Helly-Bray theorem, hYn') a.s.ljLl. mll h-n') (F'). Fa/() Since It follows from 22.4 that -G e A) h1n(X), -->
=
jt,/-1(4)
-1(4))
' rrlt: /1(1%()) e A)
=
zrl((l):F/4) n
e
=
(22.27)
-1
p.
for each A e B, using 3.21. Simtlarly, mlhY ) e A) jt/z *(A). According to the definition of weak convergence, hYn') -P-+ hY') is equivalent to g,,;/l-l = gW-l . -
.-
.
.
-
=
22.12 Corollary If h is the function of 22.11 and Xn -P-> X, then hxn) -P-+ 4m. Proof Imfnediate from the theorem, given that Xn
2213 @
If Xn -P- X(0,l), then
Example
-
Fn and X
-
F. w
X .-P-+x2(1).u
Our second result on transformations is from Cramr called Cramr's theorem:
(1946),and
22.14 Cramr's theorem If Xn -P-> X and L (i) (Xn + i' ) -P-> X+ J). (ii) Ynxn ; aX. (iii) XnlYn -P-> Xla, for a 0.
(constant),then
-E(-> a
is sometimes
---
:#:
Proof This is by an extension of the type of argument used in 22.4. Pxn
+
L f
.%
=
P(Xn +
L
f x,
IL
-
a
I
The Central Limit Theorem
356
S Pxn
S x
a
-
l'n <
IL
+ Pxn
+
+ :)
#( IFn a
+
.x,
-
lk
-
a
Ik
e)
E).
(22.28)
Similarly,
(22.29) and putting these inequalities together, we have
< Pxn
<
a
.x -
+ :)
#(Irn tzI
+
-
:).
(22.30)
Let Fxn and Fxn+yndenote the c.d.f.s of Xn and Xn + F,yrespectively, and let Fx be the c.d.f. of X, such that Fx limn--jxFktx) at a1l continuity points of Fx. Since limn-yxPt IL a I 2 E) 0 for a1l 6: > 0 by assumption, (22.30)implies =
=
-
Fxtx
a
-
; liminf Fxn+ynxj
:)
-
N.-e
; limsup Fxn.vvnlx)f Fxtx a -
+
(22.31)
0.
K.->co
Taking 6: arbitrarily
close to zero shows that
1im Fxn+rntx)
=
Fxtx
-
J)
(22.32)
Fxnxj
=
n.->x
point of Fx. This proves (i). first that To prove (ii), suppose a > 0. By taking 6: > 0 small enough we can ensure a e > 0, and applying the type of argument used in (i) with obvious variations, we obtain the inequalities
whenever x a is a continuity -
-
Plxna
+ E)
S x)
#(
-
IL
-
al
f P(Xn(a
E) S -
:)
Pxn'n
; x)
S a)
#( IFs a I
+
-
e).
(22.33)
Tnking limits gives Fxtx/tl
+ E))
f liminf Fxnynx) n->x
f limsup Fxnynxj f
Fx@/(J
-
E)),
(22.34)
n--hx
and thus 1im Fxnynx)
=
Fx(x/J)
=
Fa/x).
(22.35)
n-)x
0, replace i'n by -Fn and a by -J, repeat the preceding argument, apply 22.12. And if a 0, (22.33)becomes If a
<
=
Pxnz
S x)
-
#( IFnl k e) S Pxn'n
f x)
and then
W'ctz/cConvergence of Distributions S P-
f x)
+
357
#( Ii'n I
(22.36)
:).
0, this yields in the limit Fxayatx) 1, and for x < 0, Fxnvnlx) 0, which defines the degenerate distribution with the mass concentrated at 0. ln this -CC-> 0 in view of 22.5. (Alternatively, see 18.12.) case XnYn l/J if a # 0. To prove (iii) it suftkes to note by 18.10(ii) that plim 1/L Replacing L by 1/L in (ii) yields the result directly. . For x
>
=
=
=
22.4 Convergence of Moments and Characteristic Functions Paralleling the sequence of distribution functions, there may be sequences of JItF@), where it moments. If Xn -P-> X where the c.d.f. of X is F, then F(m exists, is sometimes called the asymptotic expectation of Xn. There is a temptadoes not exist tion to write A'(m limn-oxxn), but there are cases where Ex wlle F(m where for every n exists exists, and also finite E(Xn) for any n cases but F(m does not. This usage is therefore best avoided except in specific circmstances when the convergence is known to obtain. Theorem 22.8 assures us that expectations of boundedrandom variables converge under weak convergence of the corresponding measures. The following theorems indicate how far this result can be extended to more general cases. Recall that FIaYl is defined for every X, although it may take the value +x. =
=
If Xn .-P-.I X then FI.Yl < liminfn-yxflAkl
22.15 Theorem
.
Proof The function a(x) Ixl 1(Ixlua) is real and bounded. If #( IXI a) 0, it follows by 22.11 that hqxn) -P-> atm, and from 22.8 (lettingf be the identity function which is bounded in this case) that =
F(Jla(m)
=
=
lim Ehaxnl)
f liminf FlXn
distribution.
(22.37)
.
n->x
n--x
The result follows on letting a approach
I
=
+x
through continuity points of the
.
The following theorem gives a sufficient condition for F(m to exist, given that E (Xn ) exists for each n. is uniformly integrable, then F1A-1< 22.16 Theorem If Xn -P-) X and I-YnI
E (Xn)
-->
c.
and
E (A9.
--61..:
Proof Let L and J' be the Skorokhod variables of (22.12),so that Fa K Since Xn and Fn have the same distribution, unifonn integrability of (aL) implies that of ( Fn1.Hence we can invoke 12.8 to show that E(Yn) F(l'), F being integrable. F(m as required. x Reverjing the argument then gives FIXI < x and Exn) ---
.-.4
Uniform integrability is a sufticient condition, and although where it fails the existence of EX4 may not be ruled out, 12.7 showed that its interpretation is questionable in these circumstances. it):p A sequence of complex r.v.s which is always uniformlv intecrable is f e 1. for
The Central Limit Theorem
358
any sequence (-Yn). since le in l 1 Given the sequence tions f#xa(f)J,we therefore know that if Fn = F, then #xa(/) 9x(f) =
.
of characteristic func-
(22.38)
-->
(pointwise on R), where the indicated limit should be the characteristic function associated with F. In view of the inversion theorem (11.12),we could then say that Xn -P-> X only if (22.38)holds, where (),(fl is tlte ch.f. of X. However, it is rather than the that is the point of interest here. If a the of characteristic functions converges pointwise to a limit, under what sequence circumstances can we be sure that the limit is a ch.f., in the sense that inverting yields a c.d.f.? A sufficient condition for this is provided by the so-called Lvy continuity theorem: ifX
'only
Eif'
22.17 Lvy continuity theorem Suppose that fFn ) is a sequence of c.d.f.s and Fn non-negativerbounded,
whereFisany tion. If =F,
+x
9s(/)
J
=
non-decreasing,Hght-continuouslnc-
eitxdb
and (j(/) is continuous at the point t (1)is its ch.f. u
9(f),
-->
(22.39)
0, then F is a c.d.f.
=
(i.e., JJF
1) and
=
The fct that the conditions imposed on the limit F in this theorem are not unreasonable will be established by the Helly selection theorem, to be discussed in the next section. Proof Note that 4)s(0) dl c-d.f. For v > 0, =
+x
v
-1 VJgtntvt Jf =
()
1 for any n, by
=
j --
v
eitxdt
+x
(w.n
g J
=
VJ ()
..x
(22.39)and e
fvx
!
sx-
dn,
the fact that Fn is a
(22.40)
..x
the change in order of integration being permitted by 9.32. By 22.10, wlaich extends to complex-valued functions by linearity of the integral, we have, as n
'-
x,
+x e fv.x 1
+x
-
fvx
e
dFn .--A
fv.x -
1
fvx
dF
=
1
v
v
()
-
(dt,
(22.41)
where the equality is by (22.40)and the definition of #. Since () is continuous at t 0, limv--yxv-llxt/llf ((0) and since (n (h we must have ((0) 1. It follows from (22.41)that -->
=
=
=
+x
1
=
ivx 1 e JF li m Jvx
-=
+x
-
.
V-40
dF.
=
(22.42)
-=
In view of the other conditions imposed, this means F is a c.d.f. It follows by tX 22.8 that (/) = Ee ) w h ere X is a random variable having c.d.f. F. This completesthe proof. w
W'ctzkConvergence of Distributions
359
The continuity theorem provides the basic justificationfor investigating limiting distributions by evaluating the limits of sequences of ch.f.s, and then using the inversion theorem of jl 1.5. The next two chapters are devoted to developing these methods. Here we will take the opportunity to mention one useful application, a result similar to 22.4 which may also be proved as a corollal'y.
22.18 Theorem If lXn Zn I -E+ 0 and lXn J converges in distribution, then fZn) converges in distribution to the same limit. -
le in
Proof
0 by 18.9(ii). Since Ierxl = l these functions are itzn the sequence ( Ie in e I) is uniformly integrable. So by i'Zn .--I--:Z 0 However, the complex modulus inequality (11.3) e l i'Zn
e and fa-bounded,
18.14, Ie in gives
-
-
I
-'.%
-
.
FIe itxn e -
itzn
ls e ) yeitzn ) j l9xa(/)Yz.(/)I 0 as
itxn
I
(a,z43)
-
,
.
pointwise on R. Given the assumption of weak convergence, the conclusion now follows from the inversion theorem. .
so that a further consequence is
To get the alternative
-..h
-
proof of 22.4, set Zn
=
n
-->
oo,
X for each n.
22.5 Criteria for Weak Convergence Not every sequence of c.d.f.s has a c.d.f. as its limit. Counter-examples are easy to construct.
22.19 Example Consider the uniform distribution on the interval (-n,n1,such that
Fnla)
=
1a(1 + c/a),
-n
S a S n. Then Fa(c) --A
z1for all a
G
R n .
22.20 Example Consider the degenerate r.v., Xn = n w.p.1. The c.d.f. is a step 0, all a e R n function with jump at n. Fna) --)
.
Although Fn is a c.d.f. for all n, in neither of these cases is the limit F a cxa 1 (0) as a c.d.f., in the sense that Fa) (-=). Nor does intuition suggest to us that the limiting distributions are well defined. The difficulty in the first example is that the probability mass is getting smeared out evenly over al infinite support, so that the density is tending everywhere to zero. It does not make sense to detine a random variable which can take any value in R with equal probability, any more than it does to make a random variable infinite almost surely, which is the limiting case of the second example. In view of these pathological cases, it is important to establish the conditions under whichasequenceof measures canbe expectedtoconvergeweakly. The condition that ensures the limit is well-defined is called uniform tightness. A sequence lgzl of p.m.s on R is uniformly tight if there exists a tinite interval (tz,:J such that, for any : > 0, sup,,pzttl,hll > 1 :. Equivalently, if (Fn J is the sequence of c.d.f.s corresponding to lg.nl,uniform tightness is the condition that for 6: > 0 3 a,b with b a < x and -....:
--
-
-
The Central Limit Theorem
360
sup (Fb) l
Fnaj
-
)>
1
-
(22.44)
E.
It is easy to see that examples 22.19 and 22.20 both fail to satisfy the unifonn tightness condition. However, we can show that, provided a sequence of p,m.s tgzJ is uniformly tight, it does converge to a limit jt which is a p.m. This terminology property that for every derives from the designation tight for a measure with E > 0 there is a compact set A-esuch that g(A1) S e. Every p.m. on (E,f) is tight, although this is not necessarily the case in more general probability spaces. See 926.5 for details on this. An essential ingredient in this argument is a classic result in analysis, Helly's selection theorem. jthe
22.21 Helly's selection theorem If (Fn) is any sequence of c.d.f.s, there exists a subsequence fnk,k 1,2,...1 such that Fnk = F, where F is bounded, non-decreasing and right-continuous, and 0 K F K 1. =
Proof Consider the bounded array ( fFnxij, n G ENJ, i e N 1,where (a),i e EN) is an enumeration of the rationals. By 2.36, this array converges on a subsequence, so that limk-yx/ktxj) = F*x for every Note that F'xik) S F'txfa) whenever < Aa, since this property is satisfied by Fn for every n. Hence consider the nondecreasing function on R, .:/l
.
F(xl
inf F*(xj).
=
(22.45)
Xi >.T
Clearly 0 K F*xij f 1 for a11 i, since the Fnjxij have this property for every k. By definition of F, for x e R 3 xi > x such that F(x4 S F'xi) < Flxj + 8 for Fxi). Further, for any e: > 0, showing that F is right-continuous since F'xi) A7 > such that points of there exist and < F, continuity xiz x xfj x =
F(x)
-
: <
F*(xjj) f F'xiz)
<
Fx)
+
(22.46)
t.
The following inequalities hold in respect of these points for every kl F'bxiL)
lim Fnklxik)f liminf Fnkx)
=
k-yx
k-yx
f limsup Fnkx) < lim Fnklxiz) = F'xi. k-4x
Combining
(22,47)
k-x
(22.46)with (22.47), F@) s < liminf Fnkx) f limsup Fnjxj -
k-x
<
F(x)
+ :.
(22.48)
k->x
Since s is arbitrary, limk-yx/ktx)
=
F@) at all continuity points of F.
.
The only problem here is that F need not be a c.d.f., as in 22.19 and 22.20. We 1 (0) as x .--A x (-x), and tightness is the required need to ensure that F(x) property. -->
22.22 Theorem Let (Fn ) be a sequence of c.d.f.s. If
JFetzk Convergence of Distributions
361
(a) Fnk u:zh F for every convergent subsequence (rlk),and (b) the sequence is uniformly tight, then Fn
=
F, where F is a c.d.f. Condition
(b) is
also necessary.
I:n
Helly' s theorem tells us that (Fn ) has a cluster point F. Condition (a) requires that this F be the unique cluster point, regardless of the subsequence chosen, and the argument of 2.13 applied pointwise to (Fn ) implies that F is the actual limit of the sequence. Uniform tightness is necessary and sufficient for this limit F to be a c.d.f. Proof of 22.22 Let x be a continuity point of F, and suppose Fn(x) zz-y F@). Then lFn@) F(x) I : > 0 for an infinite subsequence of integers, say (rls k e (Nl Define a sequence of c.d.f.s by FJ Fnk, k = 1,2,... According to Helly's theorem, this sequence contains a convergent subsequence, lk, i G (N), say, such that FLi= F. But by (a),F' F, and we have a contradiction, given how the subsequence fkfl was constructed. Hence, Fn = F. Since Fn is a c.d.f. for every n, Fnb) Fn(J) > l for some b a < x, for F at continuity points, increase b and reduce a as necesany t: > 0. Since Fn sary to make them continuity points of F. Assuming unifonn tightness, we have by 1 and (22.44) that Fb) Fa) > 1 :, as required. lt follows that limx-jxfx) limx-o-xfx) 0. Given the monotonicity and right continuity of F established by Helly' s theorem, this means that F is a c.d.f. Fa) < 1 s On the other hand, if the sequence is not uniformly tight, Fb) and +cxn for some : > 0, and every b > a. Letting b a we have F(+=) F(-x) f 1 : < 1. Hence, either F(+x) < 1 or F(-x) > 0 or both, and F is not a c.d.f. . -
=
.
=
-
-
-
.--
-
=
-
=
-
-
-x,
--
-
--
-
The role of the continuity theorem (22.17)should now be apparent. Helly's theorem ensures that the limit F of a sequence of c.d.f.s has all the properties of a 1. Uniform tightness ensures this property, c.d.f. except possibly that of JJF and the continuity of the limiting ch.f. at the origin can now be interpreted as a sufficient condition for tightness of the sequence. lt is of interest to note what happens in the case of our counter-examples. The ch.f. corresponding to example 22.19 is =
G(V)
=
(2n)-1
n (cOs ..n
Mx+ j
siu vxlts
=
sinvrl .
Mn
We may show (usel'Hpital's rulel that (n(0) 1 for every n, whereas for a1l v # 0. ln the case of 22.20 we get =
#u(v)
=
cos s'n + i sinvn,
which fails to converge except at the point v
(22.49) n(v)
--)
0
(22.50) =
0.
22.6 Convergence of Random Sums Most of the important weak convergence results concern sequences of partial sums of a random array fxnt, t 1,...,n, n e (NJ. Let =
The Central Limit Theorem
362
Sn
=
Xnp X -1
(22.51)
and consider the distributions of the sequence (&J as n --A cxn. The array notation (double indexing) permits a normalization depending on n to be introduced. Central limit theorems, the cases in which typically Xnt n -1/2X(, an (j s n converges to the Gaussian distibution, are to be examined in detail in the following chapters, but these are not the only possibility. =
22.23 Example The Blnshln) distribution is the distribution of the sum of n inde1) hln and Plxnt = 0) pendent Bernoulli random variables Xnt, where Pxnt 1 hln. From 22.2 we know that in this case =
=
=
-
Sn
=
XXnf
---/D
Poisson with mean
,.
>1
(22.52)
n
From 11.1 we know that the distribution of a sum of independent r.v.s is given by the convolution of the distributions of the summands. The weak limits of independent sum distributions therefore have to be expressible as infinite convolutions. The class of distributions that have such a representation is necessari1y fairly limited. A distribution F is called infinitely divisible if for every n e INthere exists a distribution Fn such that F has a representation as the n-fold
convolution
F
=
Fn*Fu*...*Fn.
(22.53)
ln view of (11.33), infinite divisibility implies a corresponding multiplicative rule for the ch.f.s. 22.24 Example For the Poisson distribution, and
#x(l;D =
expl/f
9x(?;) (expf(=n)(:'-1)J)n @x(f;/n))n. =
=
-
1) from ,
(11.34), (22.54)
The sum of n independent Poisson variates having parameter hln is therefore a Poisson variate with parameter :. n In certain intinitely divisible distributions, Fn and F have a special relationship, expressed through their characteristic functions. A distribution with ch.f. #x is called stable, with index p, if for each n,
n (9x(p) .
etbn)
tx(sl/p/; () < p x z, ,
(gg.55)
where bln) is some function of n. According to (11.30),the right-hand side of (22.55) is the ch.f. of the nv. bln) + n 3IPX; that is, the sum of n independent drawings from the distribution is a drawing from the same distribution apart from a change of scale and origin. If a stable distribution is also symmetric about zero, it can be shown that the ch.f. must belong to the family of real-valued functions having the fonn
W'ctzkConvergence of Distributions
91/)
=
expl
I/1,)
-.a
363 (22.56)
a k 0.
,
0, having 22.25 Example The Cauchy distribution is stable with p 1 and bnj C(V,6) fOr exptf/v v,8) 1,...,n, 61 ch.f. #x(/; t fI), from (11.38).lf Xt then kn C(v,6). This result retlects the fact already noted (see 18.18) that Cauchy variates fail to observe the 1aw of large numbers. (:1 =
=
=
-
=
-
-
-zlt/zj
by (11.37).This 22.26Example For Gaussian X, (xff',g,,c2) expt g./ n1/2)g, with about index p 2 and bn n iSS table with2 g, 0 we have symmetry c2) and N(0 for obtain the formula in if X Thus, with t t 0, (22.56) 1 a zc 0.24. j,4 N(O, u 1 n then n-ll; =
=
=
=
-
.
=
-
.
,
=
-
It turns out that a stable distribution with index p < l possesses absolute moments of order at most r < p. Thus, the Gaussian is the only stable 1aw for which a variance exists. This last fact is most important in motivating the central limit theorem. Each of the foregoing examples illustrates the possibility of a limit 1aw for sums. A stable distribution naturally has the potential to act as an attractor for rbitrarily distributed sums. While stable convergence laws do operate in cases where the summands possess absolute moments only up to order r < 2, the key result, to be studied in detail in the following chapters, is that the Gaussian acts as the unique attractor for the distributions of scaled sums of arbitra r.v.s having zero mean and finite variances.
23 The Classical Central Linzit Theorem
23. 1 The i.i.d. Case law of error' is justly the most famous result in statistics, and to susceptible mind has an almost mystical fascination. If a sequence of random the variables (#/)Thave means of zero, and the partial sums Z:=1m,n = 1,2,3,... have variances sn2 tendin g to infinity with n although finite for each finite n, then, subject to rather mild additional conditions on the distributions and the The
tnormal
sampling process,
sn
=
-
1
Sn
n
Tx
-P-+ #(0, 1).
(23.1)
>1
This is the central limit theorem (CLT). Establishing sets of suftkient conditions is the main business of this chapter and the next, but before getting into the fofmal results it might be of interest to illustrate the operation of the CLT as an approximation theorem. Particularly if the distribution of the Xt is symmetric, the approach to the limit can be very rapid.
23.1 Example In 11.2 we derived the distribution of the sum of two independent (40, 12 drawings. Similarly, the sum of three such drawings has densty /k+I'+w)
=
f-l(.1(lltw-c-l,w-zldAdz, JoJ
(23.2)
which is plotted in Fig. 23.1. This function is actually piecewise quadratic (the tllree segments are on (0,1j, g1,21and (2,31respectively), but lies remarkably close to the density of the Gaussian nv. having the same mean and variance as
The sum of 10 or 12 independent uniform r.v.s is almost indistinguishable from a Gaussian variate; indeed, the formula S Xl=21#j6, which has mean 0 and variance 1 when Xi ULO,11 and independent, provides a simple and perfectly adequate device for simulating a standard Gaussian variate in computer modelling exercises. n
X+ F+ Z
(alsoplotted).
=
-
-
23.2 Example For a contrast in the manner of convergence consider the Bn,p) variates for fixed p e (0,1).The probadistribution, the sum of n Bernoulli) 1a and n 20 are plotted in Fig. 23.2, together with the Gaussian bilities for p with matching density mean and variance. These distributions are of course discrete for every finite n, and continuous only in the limit. The correspondence of the ordinatej is remarkably close, although remember that for p # ,1 the binomial distributiop is not symmetric and the convergence is correspondingly slower. This =
=
The Classical Central Limit Iheorem
365
example should be compared with 22.2, the non-Gaussian limit in the latter case being obtained by having p decline as a function of h. n
sumof
Nq, 14)
3 &g0,1js N
z
0.4 N
z
0
N
0
l.5
3
Fig. 23.1
0. 2
I Bin(20,
0.5)
x
x(1(),5)
?' / 0.
$
3.
1
?' ?' ?' & J'
O
x
--
O
.
10
20
Fig. 23.2 Proofs of the CLT, like proofs of stochastic convergence, depend on establishing properties for certain ncnstochastic sequences. Previously we considered sample 1, probabilities #( IxL #1 > E), and X()) I for e C with PC4 points lXnk XIP, moments FIXn as different sequences to be shown to converge to 0 to establish the convergence of Xn to X, respectively a.s., in pr., or in Lv. In the present case we consider the expectations of certain functions of the Sn; th key result is Theorem 22.8. The practical trick is to find functions that will fingerprint the limiting distribution conclusively. The characteristic function is by =
-
-
-
'he Central Limit Theorem
366
common consent the convenient choice, since we can exploit the multiplicative property for independent sums. This is not the only possible method though, and the reader can tind an alternative approach in Pollard (1984:H1.4), for example. The simplest case is where the sequence lA) is both stationary and independently drawn. 23.3 Lindeberg-Lvy variance c 2
theorem If
is an i.i.d. sequence (XTIT
having zero mean and
,
l
s
n-llTx /c
=
rl
>1
#x(1)of
Proof The ch.f.s (1 1.33),
Applying 11.6 with k
1t/x(,c whichmakes
-
n
)
-
((VVC-'X-1Y))S =
and
(23.4)
.
j
ljgyt
-
j x s mij,j
3 h-k)2 Izxt j j
(23.5)
j
( o'V 6c an w
it possible to write, for tixed 1,
pxtlc-1n
-1/2
)
=
o(,,-a/2 ).
1 yzjz,t + -
Y:=/:)(J/n)/ Applying the binomial expansion, (1+ aln? Il2 + ()(n-1/2), we j'ind x, and setting a n =
--
(11.30)
2 yields the expansion
=
1/2
(23.3)
Xt are identical for al1 /, 24 so from
p-a(X)
-1
-.P-+x(o, 1). o
t
=
(23.6) --+
;
=tg/./!
=
ea as
-,
lim
9.sa(D =
e- 2/2
(23.7)
.
n-+x
Comparing this formula with (11.36), the limiting ch.f. is revealed as that of the N(0, 1) variable. We then appeal to the inversion theorem (11.12)to establish that the limiting distribution is necessarily Gaussian. . Be careful to note how the existence of E IXl 3 is no t necessary for the expansion function whose expectation appears on the majorant in (23.6)to hold. The side of (23.5)is unquestionably of On -3/2), but also clearly j n teggable for each n on the assumption of finite variance. The Lindeberg-l-vy theorem imposes strong assumptions, but offers the benetit of a simple and transparent proof. Al1 the key features of the central limit 1/2 property are discernible. In (23.6), the expansion of the ch.f. of n - Xt cnsists either of terms common to every centred distibution with tinite variance, or of tenns that can be neglected asymptotically, a fact that ensures that the ltmiting sum distribution is invariant to the component distributions. The imaginary part of xt,c -1n ) is of smaller order than the rea j pa y.t wjch would appear to require a symmetric limit by the remarks following (11.31). The coincidence of these facts with the fact that the centred Gaussian is the only stable symmetric distribution having a second moment appears to r'ule out any Kmin'
-1/2
,
The Classical Central Limit Theorem
alternative to the central limit property under the specified conditions, that is, zero mean and finite variance. The earlier remark that symmetry of the distribution of n -1/2-Y)improves the rate of convergence to the limit can be also in (23.5)can be taken appreciated here. If we can assume Ext)3 0 the expansion -2 =
,
to third order, and the remainder in (23.6)is of On ). On the other hand, if the variance does not exist the expansion of (23.6)fails. Indeed, in the specific case we know of, in which the Akare centred Cauchy. ()(nl/2); the (n3IlX j is not tight, and n= n 1/22 sequence of distributions of n there is no weak convergence. The -1limit 1aw for the sum would itself be Cauchy under the appropriate scaling of n Thedistinction between convergenceindistribution andconvergenceinprobability, and in particular the fact that the former does not imply the latter, can be demonstrated here by means of a counter-exnmple. Consider the sequence (A)!' defined in the statement of the Lindeberg-Lvy theorem, and the corresponding Sn .
in (23.3). in probability.
23.4 Theorem Sn does not converge Proof If it was true that plims-oxs'n = Z, implying
Z, it would also be the case that plims--yxu&a,,
=
plim hn Sn) -
=
0.
(23.8)
n-4x
We will show that
false. We have
(23.8)is
ln
n
t /c + X x%u (2rl)-1/2F'').;&r
kj /c
=
-
=
n
l/2
(Sn +
(23.9)
sp,
p=l+1
>1
whereSn'
=
2n Z,=+1A7c, j.jence
Szn Sn -
=
x%41//1' -
1) +
Sn'IV.
(23.10) 1 2 2
expt and 4x() According to the Lindeberg-l-vy theorem, 4x() x,Sn'c ) contained and is in both Xt Sn sums, are independent, expf sinceno n1. Noting g1/A- 1)J2+(1//jj2 2 /j' and eachwith mean zero and variance applying the properties of ch.f.s, .-.:
-->
-r1,2c2j.
=
-r12(2 -snh)
(x2n
-->
exp
(
-
Wlcz j
.
-
(23.11)
In other words,
v%zn sn -P-> N(0, c2(2 /j')), -
-
whichis the
required
contradiction of
(23.12)
(23.8)..
with, say, (xY4))+ Ykln JT, ) G f1, whereXand F Compate the sequence (&((z))JT the latter sequence converges to a t'ixedlimit, are random variables. For given X(). On the other hand, each new contribution to Sn has equal weight Qth the others, ensured by re-scaling the sum as the sample size increases. For given (, 1/2 'nn) = n - Z)=1m())is not a convergent sequence, for as 23.4 showj Szn is ,
Ihe Central Limit Theorem
368
not neessarily close to Sn no matter how large n becomes. Weak convergence of the distribution functions does not imply convergence of the random sequence. Characteristic function-based arguments can also be used to show convergence in distribution to a degenerate limit. The following is a well-known proof of the weak law of large numbers fori.i.d. sequences, which circumvents theneed to show f,! convetgence. is an identically and independently distributed 23.5 Khinchine's theorem If (XIJT rI-17=I.Y) -ZL> g,. sequence' with tinite mean jt, then kn =
kn has the
Proof The characteristic function of
kt)
form
(xlvnqjn.
=
(23.13)
wherye by application of the argument used for 23.3, (xvnj x we find, by analogy with (23.7), + 0h ln 2). Letting n
=
1 + ihpln
-->
lim4,,:()
=
/1
eikk
(23.14)
*
n-,>oo
fB But Ee 'f) e on ly for the case where X = jt with probability 1. The distribution is degenerate, and convergence in probability follows from 22.5. . =
23.2 lndependent Heterogeneous Sequences The Lindeberg-Lvy theorem imposes conditions which are too strong for theresult to have wide practical applications in econometrics. In the remainder of this chapter we retain the assumption of independence (to be relaxed in Chapter 24), but allow the summands to have different distributions. In this theory it is convenient to work with normalized variables, such that the partial sums always have unit variance. Tlis entails a double indexing scheme. Define the triangular array
fxnt, t
=
1,...,n, n e EN ),
2 suc h that if the elements having zero mean and variances nnt, Sn
XXnC,
=
(23.15)
>1 then (underindependence) Esnh
57cjf
=
->
=
(23.16)
1;
1
Typically we would have Xnt (Fr gfl/ln where (F,Jis the sequence under 2 with study, means (w), and ss l EY f w)2.In this case c2nt 2 E (F, g. 2Isn, S0 th a t these variances sum to unity by constnlction. It is also possible to have Xnt = (F,a bknlsn, the double indexing of the mean arising in situations where the sequence depends on a parameter whose value in turn depends on n. This case arises, for example, in the study of the limiting distributions of =
Kraw'
-
=
-
-
.1
-
=
The Classical Central Limit Theorem
369
deviations from the null hypothesis, test statistics under a sequence of the device known as Pitman drift. The existence of each variance cjf is going to be a necessary baseline condition in all the theorems, just as the existence of the common variance o.2was required in the Lindeberg-Lvy theorem. However, with heterogeneity, not even unifonnly bounded variances are sufficient to get a central limit result. If the Ff are identically distributed, we do not have to worry about a small (i.e. finite) number of members of the sequence exhibiting such extreme behaviour as to influence the distribution of the sum as a whole, even in the limit. But in a heterogeneous sequence this is possible, and could interfere with convergence to the of eachindividual memberof the normal, which usually depends onthecontribution sequence being negligible. The standard result for independent, non-identically distributed sequences is the Lindeberg-Feller theorem, which establishes that a certain condition on the distributions of the summands is sufficient, and in some circumstances also necessal'y. Lindeberg is credited with the sufficiency pal't, and Feller the necessity', we look at the latter in the next section. 'local'
23.6 Lindeberg theorem Let the array Lxnt ) be independent with zero mean and variance sequence (cjr) satisfying (23.16).Then, Sn -P-+ N(0, 1) if 1im
n
X >
nOx
1
j
Xlt dp
0, for all e
=
n
l IXnt I> e.l
0.
>
(23.17)
EI
Equation (23.17)is known as the Lindeberg condition. The proof of the Lindeberg theorem requires a couple of purely mechanical
lemmas.
23.7 Lemma lf x1,...,.v and y1,...,yn are collections of complex numbers with la1 f 1 and lyrl K 1 for t 1,...,n, then =
n
l
1-l 1-1 >1 xt
y,
-
n
K
>1
Proof For n
=
77Ixt
-y,1
(23.18)
.
>1
2,
I.xl.x2yly2 l -
+ (x2 y2)y1I l(.z'1 K Im 11x2I+ 1.x2 -y2I1y1 I < 1x1 I + Ix2-y2I -z1)av
=
-
-y1
(23.19)
-y1
.
The general case follows easily by induction.
.
23.8 Lemma If z is a complex number and Iz I f ;,1 th en
I
CZ
1 -zI <
-
2.
IzI
Proof Using the triangle inequality,
Iez
-
1 zI -
=
x zj
s. j=z J.
=
2
z
x .
j=0
z +
j
2) !
S
2
Iz I
x .
j'
jz j.j +
.
a)!
(23.20)
The Central Limit Theorem
37O Since
T 2-7 Zy=.f)
2 the infinite series on the
=
4x()
Proof of 23.6 We have to show that bounded by
I
9.s-(1)
j
2/2
e-
-
n
#xa,(v) 1-1
=
-
>1
n
side cannot exceed 1.
light-hand
,
e- :2/2as n
--
2
e-'
1-1
=.
--
.
The difference is
o'l
>1
;!
.
f >1119xa,(D 11(1xzc2ntl -
-
>1
n
11
+
e '-
n
2 @
%t12
jj(j j,zgzup
-
-
,
>1
>1
gg
.g
j)
2 = 1, and the where the equality is by definition, using the fact that Z)=1cnr inequality is the triangle inequality. The proof will be complete if we can show that each of the right-hand side terms converges to zero. 2 an d The integrals in (23.17) may be expressed in the form F(1 Xnt), ( IXnt I> e:l
+ F(1( 1xng1>:)X2at) 0-2 F(1( jxa,!<:)Xn2;J Ix.I>:)xn2,) s + F(1f M f2 as n a11 1 s t < n, 2 nt
=
(23.22)
--)
x,
since the Lindeberg condition implies that the second term on the right-hand side of the inequality (whichis positive) goes to zero; since ir can be chosen arbitrarily small, this shows that max ot2
()
-->
(aa aa)
.
.
1K t f n
In the t'irst of the two terms on the majorant side of (23.21),the ch.f.s are al1 2 less than 1 in modulus, and by taking n large enough we can make l 1 g1z2cn/ j < j for any tixed value of 1. Hence by 23.7, -
n
n
I-IIVnJXI gJ(11z,2cu2p -
-
>1
>1
l
<
X jjxa/) >1
To break down the terms on the majorant side of case k 2, combined with (11.29),yields
-
(1
-
xzcznp j .
(23.24)
(23.24),note that 11.6 for
the
=
)
Ak,()
j
S FtminftkYnJz, (1 -1.2X2c2,jJ j ILALI
-
3j)
K Hence, recalling
2 Z:=1ou
=
(Vr,,(X) Yl >1 j -
nt
I>:)X2?:,) +
3:c2?g. jj j
1
,
(1 X2cl) -
( 1x 2f(1
!f
2'XA'(1( IAk,I>s)X,$)+ 161/,13: >1
(23.25)
Thc Classical Central Limit Theorem
11113: as n
0:
371 (23.26)
-0
x,
since the first majorant-side term vanishes by the Lindeberg condition. Since arbitrary, this limit can be made as small as desired. Similarly, for the second tenn of n
j'j
n
g a
-).
c#2
e
jj
.
>1
>1
(1
n
j:2c2uj) <
-
(23.21),take j
y''q je 1
j
.
j
lr/czu,
-
(23.27)
.
/=
2 12 Gnt Setting z (a rea 1 num ber actually) in 23.8 and applying of the majorant side (23.27)gives =
-;
,
X
X
22
-).
cn#2
j-j o >1
.
j'j(j
1yzo.jy;
.
u
>1
is
n large enough so that
aoV2
-
:
1414
Y
y
the result to
g4us
(23.28)
>1
But, N
n
4 (maxd/)Xc2u,max Xcn, <
=
/=l
c2u,
0 as n
--y
(23.29)
--y
x,
tzz1
by (23.23).The
proof is therefore complete.
.
The Lindeberg condition is subtle, and its implications for the behaviour of easier if we look randomsequences can for careful interpretation. Tis will be c2, variance and 17.102,. at the case xnt xtlsn,where xt has mean O and Then the Lindeberg condition becomes .2u
=
=
n
lim-i'1 X
N-x
Sn
xltdp
=
t lx t jso:)
>1
0, for a11 :
>
(23.30)
0.
One point easily verified is that, when the summands are identically distributed, Sn 'Zc7 and (23.30)reduces to limn-yx sn-lEx; 1( jxj j>o.G:)) 0. The Lindeberg condition then holds if and only if X1 has tinite variance, so that the Lindeberg theorem contains the Lindeberg-l-vy as a special case. The problematical cases the Lindeberg condition is designed to exclude are those where the behaviour of a finite subset of sequence elements dominates a1l the others, even in the limit. This can occur either by the sequence becoming excessively disorderly in the limit, or (the other side of the same coin, really) by its being not disorderly enough, beyond a certain point. Thus, the condition clearly fails if the variance sequence (c2f ) is tending to 2 2 = zero in such a way that sn )7=lcr is bounded in n. On the ot jaer jaand if n x for any fixed positive :, and the Lindeberg condition resembles a =, then sne The sum of the terms condition of ' average un iform integrability of (-Y2). t 2 f (1(lx,!>,ns)X,) must grow less fast than no matter how close E is to zero. The following is a counter-example (compare12.7). =
=
,
.2
.-+
,
--
'
,n2
,
-2 23.9 Example Let Xt F, F(Ft) where 1$ 0 with probability 1 t and t with probability f -2 Thus E(Yt) = f-1 and Xt is converging to a degenerate r.v., equ/l =
.
-
=
-
,
The Central Limit Theorem
372
1 t -2 for every t The Lindeberg to 0 with probability 1, although Vartr condition fails here. sn2 n 7.1/ an d for 0 < r f 1 we certainly have t > 1/2 Therefore, sne whenever t > n =
-
.
-2
=
-
,
.
-1
n
1 'n
n
X26IP
X
t lxrl>'nf1
>1
t
2
-2
(/ t ) t -
,GJ+l n
r-r
-
ZJ=1 n
-2
t
(23.31)
01
x. And indeed, we can show rather easily that the CLT fails here. For any as n )721J, then if no 2 1, we put Bo --
=
n
<x
o
P 77F, sup n
>
U
Bo < #
,...1
l Ii',1
f=x+l
>
0)
K
f-2, X r=na+l
(23.32)
where the majorant side can be made as small as desired by choosing no large enough. lt follows that Z:.1J'f 07,(1) and hence, since we also have L1=jEYt) .TJ.-> 0, confinning that the CLT does not operate in this Otlog n), that =
=
CaSe.
xqxtlsn
El
Uniform square-integrability is neither sufficient nor necessary for the Lindeberg condition, soparallels mustbe drawn withcaution. However, thefollowingtheorem gives a simple sufficient condition.
2) is uniformly integrable, and slln 23.10 Theorem If (.;V n (23.30) holds. Proof For any n and e n
1 -E 77e1( kn
>
0, the latter assumption
1 (x,$>&,,eJ.Y2f) < maxFtlt
>
0 for a11 n, then
implies
1x,l>,ao)Xr)j. 2
y l:g//n
>1
B
(23.33)
Hence n
1 F7F(1( 1im a--)x a an
Ix,I>os)Xl) K 1
maxtftlt
ylimsup
>1
n--yx
s
-sup
1
B
Ixjl>xuclxrlj a
1S/Kn
z lim F(1( Ix,I>.so)-Y,) t
N--co
= 0, where the last equality follows by uniform integrability.
(23.34) .
There is no assumption here that the sequence is independent. The conditions involve only the sequence of marginal distributions of the variables. None the less, the conditions are stronger than necessary, and a counter-example as well as further discussion of the conditions appears in j23.4. The following is a popular version of the CLT for independent processes.
23.11 Liapunov's
theorem A sufficient condition for (23.17)is
The Classical Central Limit Theorem N
2+8 1imXFIAk,I n-yx
Proof For 8
>
0 for some 6
=
>
(23.35)
0.
>1
0 and
0,
6r >
2+
2+
2 F( 1f Ixa,I> i;) IXntI
E IXntI
(23.36)
)
.x2
j
2 e: E
( Ixa,I>:)
nt
) .
The theorem follows since, if s olim ,,-,x k1 lF(1f Ixa,I>:)Xl) then the same holds with : rep 1ace d by 1 . =
=
0 for fixed tr > 0,
.
Condition (23.35)is called the Liapunov condition, although this tenn is also used to refer to Liapunov's original result, in which the condition was cast in terms of integer moments, i.e. n
7-)A'l xk,I3
lim
a-+x
=
().
(23.37)
>1
Although stronger than necessary,
the Liapunov condition has the advantage of
being more easily checkable, at least in principle, than the Lindeberg condition, as the following example illustrates.
23.12Theorem < cxo unifonnly
Liapunov ' s con dition holds if snhn > O uniformly in n and in /, 6 > 0.
2+8
A'IX/I
Proof Under the stated conditions, n
1
uX Sn
for all
>1
-
2+ E I.Y,I <
n, where #1 k supe'l
lim Sn2+
follows immediately.
Snc
AI2+8an d 1
n-yx
nBz
n
y7Flxfl -1
<
/1 < x
T-,
(23.38)
B2 < inf n snlln. Then 2+
=
o
(23.39)
.
Note that these conditions imply those of 23.10, by 12.10. It is sufficient to condition in which variances exist but no moments eveh avoid the fractionally higher, provided the sum of those variances is also On4. tknife-edge'
23.3 Feller's Theorem and Asymptotic Negligibility We said above that the Lindeberg condition is sufficient and sometimes also necessary. The following result specifies the side condition which implies ecessity.
23.13 Feller's theorem Let fxnt) be an indepndent sequence with zero mean and Var iance ). If Sn -P-> N, 1) and sequence (cn2f
The Central Limit Theorem
374 max #( IXarl
> E)
0 as n
-->
(23.40)
any e: > 0,
-->
x,
1f t S n
the Lindeberg condition must hold. n The proof of Feller' s theorem is rather fiddly and mechanical, but necessary and it is worth a conditions are rare and difficult to obtain in this theory, little study for that reason alone. Several of the arguments are of the type used
already in the sufticiency part. Proof Since c2nf 0 for evel'y /, the series expansion of the ch.f. suggests that 19xn,(%) 1 l converges to zero for each t. ln fact, we can show that the sum of the squares of these terms converges, and this is the first step in the proof. -->
-
Applying 11.6 for k
l9xa,() 1 I < -
0 and k
=
Ftminlz,
1 respectively,
=
I ,;L,I1) <
we can assert that
2#( 1Ak,1 > E) +
eI11
(23.41)
and
I(l)x (l)
1 I S ftmintz
-
nt
l2A$j) < hlzln'* N
IlAk l
,
(23.42)
In each case the second inequality is by (11.29), setting E = 0 for (23.42). Squaring l9xa,() 1 I adding up over t, and substituting from the inequalities 2 = 1 we o btain remembering E:=1cnf -
,
,
n
:71
j
2
I#x,,,(l).-1 l < max
1f; t < n
J=
<
n
j
jpxa,() j 77l9xa,() -
1
-
/=
2max #( lXntI
>
1S t S n
-->1:13:as
n
1
e) + e I
1I
2
,1 '
(23.43)
-->
x,
using (23.40)and (23.41).
This result is used to show that 9Ja(:) = 117=19xu,() by can be approximated 'rcose-l cz-' (X:=1#xn,(X) if d that 1 1 then I I exp x1 l Note z re an r s using (11.12). Lemma 23.7 can therefore be applied, and for some n large enough, .
n
expX(9xn,(l) >1 -
1)
ie
=
-
n
n
-
-
,
77lexplyxrtl) I-lqlxnrt'l <
-
>1
>1
1)
4x,(l)I
-
M
S
2, 1ttx /1,) 1 l 77 >1
(23.44)
-
n
where the second inequality is an application of 23.8 with z 9xa,(D 1. The condition of the lemma can be satisfied for large enough n according to (23.42) and (23.23).By hypothesis (
-->
,
---)
-
.
.
-
The Classical Central Limit Theorem
exptA-l--, (x--(l)
l)j
log
-l2,
= Xcos ,-1
-
375
hxnt 1) --
(23.45)
-
using (11.12)and (11.13)to get the equality.
Taking the real part of the expansion in (11.24) up to k 2 gives cos x 2 l 2 + k S S for for that 0 0 1, 1 1 zx cos ty-x cos so any x. Fix 6: > 0 and a choose > 2/:, so that the contents of the parentheses on the minorant side below is positive. Then we have =
/
-
2 H.so X-j-
n
?.
XI
E=' tczj J ( IAk/ I>e:)
n
X J((Ik
Xn2dp f t
:1hlxnlt -
I> c)
p.j
n
j
jzyxsj
KX /=
X E tlt,z.;Lz,1 -
+
2 dp
.j
r
Xnt > E) 1 lI l
n
S
=
.x
-
+
cos
cos hxnt)
--
tjt
dp
0,
(23.46)
>1
the last inequality holds since the integrand is positive by construction fOr evel'y Xnt, and the convergence is from (23.45)after substituting Efczn, 1. Since E is arbitrary, the Lindeberg condition must hold according to (23.46).w where
=
negligibility', under which no a condition of whole. summand The chief single may be so influential as to dominate the sum as a ()sst could unless why ---) have the Lindeberg condition, ithout we w reason ,) e (23.40) holds, is that a finite number of summands dominating all the others could happen to be individually Gaussian. The following example illustrates.
Condition
(23.40)is
Easymptotic
-:2/2
23.14 Example Let Xt NnGt) 2 w here c2t 2/ Note that sln E).12' 2?JE)-12-' 2n+1 2 and Xnn Xnlsn N, lnllln+l 2)). Clearly Sn N(0, 1) for every n, by the linearity property of the Gaussian, but condition (23.40)fails. The =
-
=
-
7
-
=
.
=
=()
=
-
-
Lindeberg condition also fails, since n
X j
( 1Xnt I> E l
Xln, dp k --A
Xl dp nn
( IXnnI> E l
F(l Izl>4Is)Z2) > 0 t
(23.47)
where Z is a standard Gaussian variate. u A condition related to P max 1f t f n
(23.40)is
I.L,I >
:
--)
0 as n
-->
x,
any
which says that the largest Xnt converges in probability to zero. 23.15 Theorem
(23.48)implies (23.40).
(23.48)
376
The Central Limit Theorem
(23.48)is
Proof
#
lA',;/IS
the same as Ptmaxlx/un max 1Aul <
:
;)f Ix,,,1;
z'
=
ls/fn
:) :)
>1
min #( j-L,l <
s
1. But
--+
E)
lslfn
=1
max #(
-
1K t < n
I#n,I>
e),
(23.49)
where the inequality is by monotonicity of #. If the tirst member of (23.49) converges to 1, so does the last. . Also, interestingly enough, we have the following.
23.16 Theorem The Lindeberg condition Proof Another way to write
integration) is
implies
(23.48). order of summation and
the (23.17)(interchanging
n
1 N-% x2 > nt .e. ((-r,,l e:)
--->
f,l
() all :
0.
>
,
-
(23 50) .
>1
Accprding to 18.13 this implies n
P
for any
E >
1t Ix X >1
nt
I>c)Xl > :2
.-#-r->
0 as n
-->
0 or equivalently, ,
,
(23.51)
x,
--
0. But notice that
7) 1( lA'af1
0:
Z7.11( jxarjosla'n,2
> ::1
>1
((t))-Y2 nt
((t))> el
=
(o:
max 1
laLf())I>
:
,
(23.52)
so (23.51)is equivalent to (23.48).w Note that the last two results hold generally, and do not impose independence on the sequence. The foregoing theorems establish a network of implications which it may be helpful to summarize symbolically. Let the Lindeberg condition; L independence of the sequence; I AG asymptotic Gaussianity @k( ) e ); , = AN asymptotic negligibility (condition(23.40));and PM max 1XntI -.E(.-y0 (condition(23.48)). Then we have the implications =
=
-2/2
=
--
=
L +1
=
AG +PM
+1
=
AG +AN +1 =
L + 1,
(23.53)
where the tirst implication is the Lindeberg theorem and 23.16, the second is by 23.15, and the third is by the Feller theorem. Under independence, conditions L, AG + PM, and AG + AN are therefore equivalent to one another.
The Classical Central Limit Theorem
377
However, this is not quite the end of the story. The following example shows the possibility of a true CLT operating under asymptotic negligibility, without the Lindeberg condition. with probabilities ,1(1 /-2) each and t and 23.17 Examp le Let Xt z1 and Jn2 ,1/-2 each, so that Ext) 0 and Var(XJ jand W ith probabilities jrl + 0(1). This case has similar characteristics to 23.9. Since l-Y,,rl tsn-1 w jtja 1 probability t -2 an d gsn o therwise, we have for any E > 0 that, whenever n is large enough that Esn > -.t
=
-
-
,
t,
=
=
=
=
-1
,
#( IX// I >
; ns-2
E)
(23.54)
where ne is the smallest integer such that nzsn-' > E. Since nz O(rl1/2) (23.40) holds in this case. However, the argument used in 23.9 shows that the Lindeberg condition is not satisfied. F(1( Ixa,I>:).Y$) sn-l for t k ng, and hence =
&
X
r= 1 l Ivn/I
xltdp z n
l
> E,
n
-
nsn-l
.-+
4 > 5
,
0.
(23.55)
However, consider the random sequence ( W'f), where Gr Xt when ImI ,1 and W'r= 0 otherwise. As t increases, Gf tends to a centred Bernoulli variate with p 1, and defining ll'n, Wtlsn, it is certainly the case that =
=
=
=
n
1#'n/-PX >1
NQ, 15).
(23.56)
is distributed like I'r in 23.9, and applying (23.32)shows that However, lX, W',I lP,1 Op(1), and hence l 7=1Xn,- Z'/=1W'n,l f E7=1IXq:-1IXnt W%,1 -P- 1(0, j), according to 22.18. n O# (n-1/e)tt follows that -
=
*
-
=
xtxnt
A CLT therefore does operate in this case. Feller's theorem is not contradicted because the limit is not the standard Gaussian. The clue to this apparent paradox is that the sequence is not unifonnly square-integrable, having a component which contributes to the variance asymptotically in spite of vanishing in probability. of 1 for every n despite the fact ln these circumstances Sn can have a 15! variance of that its limiting distribution has a tvariance'
23.4 The Case of Trending Variances The Lindeerg-Feller theorems do not imposeuniform integrability or Lr-boundedness conditions, for any r. A trending vatiance sequence, with no uniform bound, is compatible with the Lindeberg condition. lt would be sufficient if, for example, o't2 < c,o for each finite t and the unit-variance sequence tXf IGtj is uniformly square-integrable, provided that the variances do not grow so fast that the largest of them dominates the Cesro sum of the sequence. The following is an extension of the sufficient condition of 23.10. 2
2/ct ) is uniformly integrable, where (c,) is a sequence 23.18 Theorem Suppose (aYf of positive constants. Then fA) satisties the Lindeberg condition; (23.30),if
The Central Limit Theorem
378
2 nMnlsn sup n 2
where Mn
=
C
=
(23.57)
< x,
maxlsfxacr. u
One way to construct the ct might be as maxf 1,c/) The variances of the transfonned sequence are ten bounded by 1, but czf 0 is not rtlled out for some f. .
=
Proof of 23.18 The inequality of &
Ix,I>.c)x:l) 177e-(1( --i
<
u''max
max 1f t f n
of
xictb>sncctbxlcljj
czf
e'(1(
sn 1x,x,,
Sn f= l
lim sup (
(23.33)extends to
te'(
xtlctthj.
1( Ixgctlx,wctl
(23.34)then
(23.58)
gives
= .E'(1(txtlctt:t-snv-lctt.xctql) 0.
(23.59)
.
n-+txz
Notice how (23.57)restricts the growth of the variances whether this be positive 1 > 0 un jsrmjy or negative. Regardless of the choice of (cr),it requires that snln in rl. It permits the ct to grow without limit so long as they are finite for all t, so the variances can do the same; but the rate of increase must not be so rapid as to have a single coordinate dominate the whole sequence. If we 1et ct = ct maxt 1,cf) as above, (23.57)is satisfied (according to 2.27) when c,2 t for any gzf lt. G k 0 but not when ln fact, the conditions of 23.18 are stronger than necessazy in the case of decreasing variances. The variance sequence may actually decline to zero without violating the Lindeberg condition, but in this case it is not possible to state a < a < 0, we would genera1 sufficient condition on the sequence. If cl t with have to replace (23.33)by the condition' -
-
-1
-
n 177F(1( sn -i' 1
Ix,I>xa:)Xl) S
=
1 ymsyasxsFtltjxtjxsasl-vlj,a
(23.60)
egy
2 1+a 2 infnsnln ) > 0 by assumption (note,sn n under independence). cxo is not luled Convergence of the majorant side of (23.60)to zero as n out, depends distribution of the Xt. but on the The following example illustrates both possibilities.
where B
1+G
=
-
-->
23.19 Example Let (X,) be a zero-mean independent sequence with Xt Ilt-ftx,fal lG either growing with f (a > 0) or declining for some rea 1 (x such that c2t = !. with t (a < 0). However, Xt is Ax-bounded for tinite f (see 8.13). The integrals in (23.30)each take the fonn -
,
,
1 zta
tz -t t,,
1(jtI>,ac)( 2tt,
(23.61)
The Classical Central Limit Theorem 2
2a
2a
2a+1
379
1 1 Now, Zz=l'r On where sn !.Zz=1T ) for a > 1 and Otlog n) for a (2.27). Condition (23.57)is satisfied when (x 0. Note that (23.61)is zero if (Yw1T gCOWS jaster than n for all G n 2a)1n E, > ta. ( nwl 0, and hence (23.61) vanishes in the limit for every t in these cases, and the Lindeberg &(- g t1, p,n grows at tjae same ra te as condition is satisfied. But if Xt n 2231/2 the above argument does nOt apply, and the Lindeberg condition (Zw1 fails. Note how condition (23.57)is violated in this case. However, the fact that condition (23.57)is not necessal'y is evident from the ltX) fact that the variance sum diverges at a positive rate when Xt ug-ftx for any even though the variance sequence itself goes to zero. lt can be verified (x that (23.61)vanishes in the limit, and accordingly the Lindeberg conditiop holds, 2 for these cases too. On the other hand, if (x < 1 sn i s bounded in the limit and (23.17) becomes n
=
n
=
.
=
-r
-z
,r2a)1/2
,
,2
-
'
-
1
-
-,,
:3
''
lim:7 a/a -n-yx >1
tc'
1
-.t
1( ItI>4s,x)t2d,
(23.62)
(,
wheres = limn-yxzkl/ 2 < x, an db y choice of small enoughy, (23.62)canbe made arbitrarily close to 1. This is the other extreme at which the Lindeberg condition fails. n
CLTS
24 for Dependent Processes
24. 1 A General Convergence Theorem The results of this chapter are derived from the following fundamental theorem, due to Mctaeish (1974).
24.1 Theorem Let (Z,,j, i = 1,...,rn, n e IN) denote a zero-mean stochastic array, where rn is a positive, increasing integer-valued function of n, and 1et25 rn
Trn
=
11(1i
i),
+
>
(24. 1)
0.
f=1
,--OD Z%1Z,,f /40, 1) if the following conditions hold: I uniformly integrable, (a) Fra is =, 1 as n (b) E lhj
Then ' S ra
=
-->
-->
ra
ni z2 (c) X =1
.TI.A
1 as n
(d)maxlxjsrnlzsjl
-->
.-E(-> 0
x,
as n
-->
c.,.
n
There are a number of features requiring explanation here, regarding both the theorem and the way it has been expressed. This is a generic result in which the elements of the array need not be data points in the conventional way, so that theirnumberrn does notalways correspond withthenumberof sampleobservations, n. rn = n is a leading case, but see 24.6 for another possibility. lt is interesting to note that the Lindeberg condition is not imposed in 24.1, nor is anything specific assumed about the dependence of the sequence. Condition 24.1(d) is condition PM defined in (23.48),and by 23.16 it follows from the Lindeberg condition. We noted in (23.53)that under independence the condition is equivalent to the Lindeberg condition in eases whete (aswe shall prove here) the central lirnit theorem holds. But without idependence, conditions 24.1(a)-(d) need not imply the Lindeberg condition. Proof of 24.1 Consider the series expansion of the logarithmic function, defined for 1x1< 1,
lOg (1 + x )
=
x
-
/
2
3 + 1x
A
-
...
Although a complex number does not possess a unique logarithm, the arithmetic identity obtained by taking the exponential of both sides of this equation is well-detined whep x is complex. The formula yields
for Dependent
CLTS
1 + ihzni
381
,zzazj
Qjlexp (
expt
=
Processes +
where the remainder satisfies I6x) I f Ixl 3 for for i 1,...,rn yields
rihzni)
1x1<
j,
(24.2)
1. Multiplying up the terms
=
exp (ih'rn )
TrnUrn,
=
where Fra is detined in (24.1) and rn
rn
Ur
exp
=
n
1k2
-
r
z2 ni
rtjz
-
i= 1
n
(24.3)
j) .
f=1
Taking expectations produces
(srnk)
=
ETrnurn)
-:2/2
e- klnEl-rn)
=
-
so given condit ion (b) of the theorem, t1).rn (D
I
lim E Trnurn n-jx
e-khl)
-
)),
F(Frn(&rn e
+
-->
(24.4)
e-hlll if
j
(24.5)
The sequence Trnurn
-
e-
-:2/2
jhl)
=
exp fihsrn l
(24.6)
Tne
-
is unifonnly integrable in view of condition (a), the first term on the right-hand side having unit modulus. So in view of 18.14, it suffices to show that plim Fr
urn -
n
n.-/co
e
-1
2/2
)
0.
=
(24.7)
0p(1), the problem reduces, by 18.12, to showing e -klll an d for this in turn it suffices, by condition (c), if
Since Trn is clearly
plimn-pxt/o
=
that
,
rn
rihzn
plim
=
0.
(24.8)
l->:xa f=1
To show that this convergence rn
rn
77
rihzn
zz:1
obtains, we have by the triangle inequality
f
377
I l
1 izzz
3
IzrlfI s
The result now follows from conditions
rn
/
max lzniI 77zlni. l ,I $. 3
'
1f i f n
(c) and (d), and
(24.9)
f=1
18.12. w
lt is instructive to compare this proof with that of the Lindeberg theorem. A different series approximation of the ch.f. is used, and the assumption from independence, that
(b- flyzn, =
#n
is avoided. Of course, we have yet to show that conditions 24.1(a) and 24.1(b) hold under convenient and plausible assumptions about the sequence. The rest of
382
The Central Limit Theorem
this chapter is devoted to this question. M.1(b) will turn out to result from suitable restrictions on the dependence. M.1(a) can be shown to follow from a more primitive moment condition by an argument bsed on the sequences' idea. Kequivalent
24.2 Theorem For an array fZni), 1et
2i
=
znjltzkf--lznzl s 2).
Tra
=
+ iLZ-nf ) is H:1(1 ,
n
(i)
The sequence
2 E max Zni sup n j
<
(24.10)
unifonnly integrable if
(:,4
x.
j < rs
.
j j)
-.E(..y 1, then And if Etnlzzf f n G
,-2
.. (11)
Znf
pr
1;
f=1 rn
*
(iii) Sr.
N
=
.zi
.
-
has the same limiting distribution as Srn.
i'=1
Proof Let Jn
=
min
:
lzn2j E./f-
2)
>
,
0, if at all, from the point i
=
- I2 ITrn The terms
,
rn
=
1zn2j
E(I2
2
>
Jn + 1 onwards. Note that
=
Jn
j + :2z2 ) 11( szz- 1-1( j
2 ni4
+
=
ni
(24. 13)
.
f=1
f=1
2z2 nf are rea 1 and positive. The inequality 1 +x I'LeAf
that 17j(1+xij
for xi
S
(24.12)
otherwise
rn, such that Zni
if
>
< ex
for x > 0 implies
0. Hence,
Jn 1 -
2
ITrn I
=
f
(1 + :2z2ni )() + hlz j''j =1
n
)
exp (VV2-1
j (1 + klz.,h) 1Xj)
< e 22(1+
,
2Z,,Jnl, 2
(24.14)
where the last inequality is by definition of Jn. Then by (24.11), sup FI TrnI2 f e
22
(1+
2
2
sup Ezunjj
<
x.
(24.15)
Uniform boundedness of Sl /raj2 is suftkient for unifonn integrability of proving (i). Since by construction E;21 ni ,
lk,
CLTS
l
# Xi Z-ni
X Z2ni
#
for Dependent
Xi Zn2j>
#
=
i
j
by assumption,
which proves
-
Pzni y,
kni,some
--->
(ii). In
1 < i < rn)
383
2
z2aj1 < P 7) i --) 0 as n
Processes
j
> :
,
for 0 < s < 1,
(24. 16)
x,
addition,
.
Pklr Izn2: > 2)
=
--)
0 as n
--->
x,
-.E(-> 0, and by 22.18, j'n and Sn have the same limiting distribution, so Ilk Sn I proving (iii). . -
24.2 The Mmingale
Case
Although it pennits a law of large numbers, uncorrelatedness is not a strong enough assumption to yield a central limit result. But the martingale difference assumption is similar to uncorrelatedness for practical purposes, and is attractive in other ways too. The next theorem shows how 24.1 applies to this case. be a martingale difference array with finite uncondi24.3 Theorem Let tXnf,8Lrl 1. If tional variances (c$) and XT=IIU, =
,
77-Y,$
...r...y
1, and
(a)>1
(b)maxlsrxnlAkl thensn
Xk,
=
.---:D
-r..+
0,
x(0,1).
>1
Proof We use 24.1 and 24.2, setting rn n, i t, and Zni = Xnt. Conditions (a) and (b) are the same as (c) and (d) of 24.1, so it remains to show that the other conditions of 24.1 are satisfied; not actually by Xnt, but by an equivalent sequence in the sense of M.2(iii). If Tn 17:.1(1+ ihxnt), we show that limn-pxErn) 1 when (Xnf J is a m.d. multiplying out, an-ay. By repeated =
=
=
=
n
Tn
=
ihxnt) 1j41 >1 +
=
ZZ
Fn-l
Ft-
1
H'-141 ikx ?t .=1
+
) is
an
ihTn-qxnn
...
=1 =
+
+
ay-lzk-lxsf. ,.l
Tf-l-measurable r.v., so by the LIE,
(24. 17)
384
'he Central Limit lheorem
/1
Evn)
=
1+
ikTEvt-kxnt) >1 n
=
1 + fl,y7F(z',-IF(xn,l
@s,,-l))
>1
1.
=
)
(24. 18)
This is an exact result for any n, so certainly holds in the limit. If Xnt is a m.d., so is knt Xnt1(Zl=-lX2k f 2), and this satisfies 24.1(b) as above, and certainly also 24.1(d) according to condition (b)of the theorem. Since l 3EXlnt) 1, condition (24.11)holds for Xnt. Hence, knt satisfies 24.1(a) and 24.1(c) according to 24.2(1) and (ii), and so obeys the CLT. The theorem now follows by 24.2(iii). . =
=
=
This theorem holds for independent sequences as a special case of m.d. sequences, but the conditions are slightly stronger than those of the Lindeberg theorem. Underindependence,we know by (23.53)thatM.3tb) is equivalenttothe Lindeberg condition when the CLT holds. However, M.3(a) is notAconsequence of the Lindeberg condition. For the purpose of discussion, assume that Xnt = Xlsn, where sn2 2 an d cf2 EXl)t Under independence, a sufficient extra condition for X7=1cf 2/Gt2 ) be uniformly integrable. In this case, inde24.3(a) is that the sequence (-Y, pendence of (Xr1 implies in dependence of (.Y2,), (-Y2, g2,, @,j is a m.d., and 19.8 o2f)..T-(.y0. (put an = sn2 an d b t = s2) t gives sufficient conditions for This is equivalent to 24.3(a). But of course, it is not the case that (Xl czf @,) is a m.d. merely because tatsFf l is a m.d. lf 24.3(a) cannot be imposed in any other manner, we should have to require Exlt j@r-1) = o'zfa significant strengthening of the assumptions. On the other hand, the theorem does not rule out trending variances. Following the approach of j23.5, we can obtain such a case as follows. =
=
.
-
Js-2E).j(V -
-
,
,
24 4 Theorem lf (.&,?h)is a square-integrable m.d. sequence and E(XltI@,-j) = 0'2, 2 a.s., and there exists a sequence of positive constants fcf) such that fXf2ycf ) js unifonnly integrable and @
2 sup nMnlsn n 2
where Mn2
=
<
(:4
x
2 maxlxruncr, conditions 24.34a) and 24.3*)
hold for Xnt
=
.
1q)
Xtlsn.
Proof By 23.18 the sequence (.L,) satisfies the Lindeberg condition, and hence, 24.3(b) holds by 23.16. Note that neither of thes results imposes restrictions on the dependence of the sequence. To get 24.3(a), apply 19.8 to the m.d. sequence cl, and t4, c2)/c2j (Xt2 o'2 The sequence is 1, bt pu tting p t t an 2 2 assumptions, and note that kl=kbt; nMn Osn) and uniformly integrable on the t?(u4), nsfj S hat both lcsblt t as consequences of (24.19).The conditions of 19.8 are therefore satisfied, and the required result follows by 19.9. w =
-
=
=
((V -
.
=
=
CLTS
for Dependent
Processes
385
24.3 Stationary Ergodic Sequences It is easy to see that any stationary ergodic mmingale difference having finite variance satisfies the conditions of 24.3. Under stationarity, finite variance is sufficient for the Lindeberg condition, which ensures 24.3*) by 23.16, and 24.3(a) follows from the ergodicity by 13.12. The interest of this case stems from the following result, atibuted by Hall and Heyde (1980: 137) to unpublished work of M. 1. Gordin.
24.5 Theorem Let (A,@r) be a stationary ergodic Al-mixingale of size Sn Z:=IX, assume that
and if
-1,
=
limsup n -1/2FIs n j
(g4.z())
<
x
>
0 n-1IlSn .-2-.I #(0 ;1zG2) u
.
N.->oo
Then n -1/2 E lSn 1
2. 0 f 2. <
-->
,
oo.
If
,
,
,
.
Notice that the assumptions for this CLT do not include EXj)2 XBt
YBt
X'Bt
<
x.
ECXBt
.Y,1(Ix,Ixs). The centred sequence I1,-1) isJ'Pa n-llll 2#2 and hence t stationaryergodic m.d. with bounded variance cs2 < -P-->N cs2), by 24.3. Further, the m.d. orthogonality property implies Proof Let
=
=
-
.1
,
1
n
F s-1/2y-q
YB t
l =
cs2
(24.21)
.
>1
E'/=1F( l is therefore uniformly integrable (12.11),and by The sequence (n-1/21 J the continuous mapping theorem it converges in distribution to the half-Gaussian limit(see 8.16)., hence, by 9.8, n
-1/2E
B
X Ft
GB
-->
wa)1/2
(24.22)
.
>1
F/-1), corresponding to 'Bt for B x, and apply the Xt F(.&I Ff+ Zt 4+1, where F, is a stationary decomposition of 16.6 to write Xt ergodic m.d. and Zt is stationary with FIZ1 I < oo. Hence Sn Z:=1l'r + zl zn+1, and by (24.20)there exists A < x such that Now define Ff
=
=
-
=
-
=
limsup n -1/2E
X l't
=
>1
n--yx
jjmsup nIlE
j
s%
-
n
-
j zj + zu4.j
n--x
s
limsup
,,-''2(e'1
-sr,1
+
N-)oo
2FIz1I) <
4.
(24.23)
(cs2,B 1,2,...) is a monotone sequence convergNoting that cs2 J(Ixjluslxzjlr ing either to a finite limit c (say) or to 1/2+=. In the latter case, in view of > A for N n, which (24.22)there would exist N such that n - FI17=1FrI x. x < 2 conclude that king in T B we can a contradicts(24.23),so (24.22) c 1/2 0, yields, in view of the fact that n - FIz1 Zn+1I =
=
=
-
--
The Central Limit Theorem
386
FISn 1 n-1/2
=
?1-1/2F
l
y-')J' + zj zux.j
-.-y
-
,
c(2/a)1/2
(24.24)
.
,--1
1/2 Since I is now known to be a stationary ergodic m.d. with c(2/a) c2, variance and c2 > 0, 24.3 and 22.18 imply that n-1/25'n --P--yN(O,c2), finite whichcompletes the proof. w
Hence, l
=
.
This result can be thought of as the counterpart for dependent sequences of the Lindeberg-l-evy theorem, but unlike that case, we do not have to assume explicitly 2 < x. I n de p endence of (Xf) enforces the condition Zt 0 for al1 t, and that '(X1) nl. It might appear that the then the conditions of the theorem imply required for the CLT, but existence of dependence weakens the momentrestrictions this gain is more technical than real, for it is not obvious how to construct a stationary sequence such that Xl is not square-integrable but l'1 is. 'I'he most useful implication is that the independence assumption can be replaced by arbithemixingale assumption) without weakening trary localdependence (controlledby any of the conclusions of the Lindeberg-Levy theorem. =
EXI) =
24.4 The CLT for NED Functions of Strong Mixing Processes The traditional approach to the problem of general dependence is the so-called method of tBernstein sums' (Bernstein 1927). That is, break up Sn into blocks (partial sums), and consider the sequence of blocks. Each block must be so large, relative to the rate at which the memory of the sequence decays, that the degree to which the next block can be predicted from current information is negligible; but at the same time, the number of blocks must increase with n so that a CLT argument can be applied to this derived sequence. It would suffice to require the sequence of blocks to approach independence in the limit, but a result can also be obtained if it behaves asymptotically like a martingale difference. This is the approach we adopt. The theorem weprove (fromDavidson 1992, 1993b) is given in two versions. The first, 24.6, is fully general but the conditions are complicated and not very intuitive. The second, 24.7, is a special case, whose conditions are simpler but cover almost al1 the possibilities. The excptional cases for which 24.6 is essential are those in which the variances of the process tend to 0 as t increases.
24.6 Theorem Let tXnf, t 1,...,n, n 1 ) be a triangular stochastic array, let < t < x, n k 1 be a (possiblyvector-valued) stochastic array, and let ( XFt, ) 'l-m c(F,u, t m S s S t + m). Also, let Sn Xlcjxnt.Suppose the following 5nt, assumptions hold: t kf-measurable, with Exnt) 0 and ESn1) 1. (a) Xnt is Fn,-x (b) There exists a positive constant array (curl such that supngllAk/cnrllr < cxa for r > 2. (c) Xnt is Q-NED of size on (J&fl with respect to the constants lcnf)speciand + 20)r/(r 2), 0 S 0 < 1a. fied in (b), (Fpgl is a-mixing of size =
-c.,
=
=
-
,
=
=
-1
-(1
-
for Dependent
CLTS
(d) Letting bn
=
1-a
grl 1 and
rn
Processes
for Lnlbnj
=
cnrl for maxi-llbn
some
(x
1,...,rn and
=
1f
obn-
=
1/2
rn+ 1
rn+ 1
Mni
(0,11,and
defining Mni
sin,rn..j
(24.25)
),
()-.1/2
Olbn
=
(24.26)
),
izz1
=
maxrnu-crsnfcn/),
=
the following conditions hold: Mni max K
e
387
where 0 is given in (c), and rn+l 2
Mni
-1
=
Obn
f=l
Then,
sn--P-
(24.27)
).
N(0,1). n
24.7 Corollary The conclusion of 24.6 holds if assumptions (c) and (d) are replaced by of size (c') Xnt is AZ-NED of size 2). on ( J&,), which is G-mixing caf), (d') Letting Mn maxlxrxa ( -r/(r
-1
-
=
nMn2 < sup N
x.
(:,4 g8)
I:a
.
Note that (c) ' i s just 24 6(c) with 0 0. If (24.28)holds, Mni 0(n-1/2) for each i. Since rnbn n, (24.25)and (24.27)hold in this case for any (x in (0,12. While (24.26)holds only for a strictly positive choice of 0, with 0 < a S 20/(20 + 1), 24.7(c') entails satisfaction of 24.6(c) for some 0 > 0. Hence, 24.6 contains 24.7 as a special case. The adaptation assumption in 24.6(a) will be needed because of the asymptotic m.d. property the Bernstein blocks must possess', see 24.19 below for the application. This assumption says that Xntmust not depend onfuture values of the underlying mixing process, F,u for s > t. In econometric applications at least, such an assumption would typically be innocuous. The remaining parts of condition (a) specify the assumed normalization. The roles of the remaining conditions are not particularly transparent in either version of the result, and in reviewing these it will be helpful to keep in mind the leading case with Xnt (1$ plsn where n F(J=1(Ff w))2,although more general interpretations are possible, as noted in j23.2. ln this case it would often be legitimate to choose =
.
=
-
=
=
-
cnt = maxtcsl
)/Jn,
-
(24.29)
whereo't2 is the variance of 1$. The cnt have to be thought of as tending to zero or withn, although possibly growing or shrinking with t also, subject to 24.6(d) J.2 ruled is of not B is tocorrelation out, the sequence '). ecause au no 24.74d variances, partial of but the the defined is just as sum longer
The Central Limit Theorem
388
n
where okt-k
c2,+ 2:7
=X ,=1
=
f
n
-
l
X wt-k,
,-2
(24.30)
t--l
Cov(J',,l',-1).
Assumptions 24.6(c) or 24.7(c') imply, by the norlp inequality, that IIXn, Exnt6 -
for 0 < p < 2, where v,u is consequenceof 17.6.
S cnfvm, ?L'+, T-,,,)lIp
of size
(24.31)
The following lemma is an immediate
-1.
isanfw-mixingale z4.8l-zemmaunderassumptions 24.6(a), (b),and (c),(Xnr,?Lf,-x) + size 20)(r-#)/p(rwith S 1, 2)) for 1 2, Kp of constants (cnfl.n (1 -mint
-(
r/(r 1) the size is + 0), and when p 2 the size is In particular, when p apply, conclusions the assumptions 24.7 Under of the same -1. except that 0 0. There is also a more subtle implication which happeps to be very convenient for the present result. This is the following. =
=
-
=
24.9 Lemma Under 24.6(a) and (b), plus either 24.6(c) or 24.7(c'), F-'--''') cnt/ul S cntcn,t-vklm Fl E(XntXn,t+k3
(24.32)
-
and
FIX nt Xn,f+ k Exntxntnb F-'-++''')l S cntcn,t-vm..k
(24.33)
-
for each k e !N, where nnts
=
Exntxnsj, and
(??: =
Om -1-
)
:
j Or
>
() u .
These inequalities are convenient for the subsequent analysis, but in effect the lemma asserts that for each fixed k the products Xntxnvtwk,after centring, form 2 2 with constants given by maxtcnl,cn,-ll One of these fal-mixipgales of size Unt,snt,-xt, where might be written as, say, ( -1
.
Unt
=
Xn-t-fklzxn,t-vk-tuzjc,,,,-(w2j,,+k.-Ep?2j.
The mixingale coefficients here are
for m
>
(/c/21.
-
(p,' V for =
m
=
0,...,k/2), and
(,,,' (p,-w2J =
by 17.11. The Proof of 24.9 The array (Xntxnvtwklis AI-NED on ( Fnfl of size conclusion then follows by 17.6(i), noting that any constant factors generated in forming the inequalities have been absorbed into (,, in (24.32)and (24.33).. -1
Npw consider M.6(d) and 24.7(d'). These assumptions permit global nonstationarity (see j13.2). This is a fact worthy of emphasis, because in the mctional CLT (seeChapters 27 and 29 for details) global stationarity is a requirement. ln this respect the ordinary CLT is notably unrestrictive, provided we normalize by sp as we do here. The following cases illustrate what is allowed.
24.10 Example Let (F,Jbe an independent sequence with variances (U t for any 13k 0 (compare13.6). It is straightforward to verify that assumption 24.7(d') is -
for Dependent
CLTS
389
Processes
It is Z)=1(U. satisfied for Xnt i'f lsn where cnt cf lsn, and in this case however violated when (U lt, a case that is incompatible with the asymptotic negligibility of individual summands (compare23.19). lt is also violated when j3< 0 (see below). (n .,:2
=
=
=
-
24.11 Example Let ( i',) be an independent sequence with variance sequence generated by the scheme described in 13.7. Putting Xnt l', lsn, 24.7(d') is satistied with cnt 1/Jn for a11 t. u =
=
Among the cases that 24.7(d') rules out are asymptotically degenerate sequences, x, x. but 041) as n 0 as t In these cases, maxlslsacl having cl snlln certain will of 24.7/ it 0. It is usually ' ), given these be the case that assumption allow. is what going that 24.6(d) To is designed to on here, see cases regularly with it is easiest to think in terms of the array (cnf) as varying n and t, within certain limits to be determined. We have the following lemma, whose conditions are somewhat more general than we require, but are easily specialized. ..-y
-->
-->
=
--+
2 24.12 Lemma Suppose cnt
-
fG-Y-1
j3y
for
,
R Then, 24.6(d) holds iff
G
.
13f Y
and
;$ < 2g0 +y(1
13and
Notice that
(24.34) 0). ca
+
(24.35)
y can be of either sign, subject to the lndicated constraints.
Proof We establish that an a e (0, 11exists such that each of conditions (24.25)2 (ibn )V-Y-1 for j k 0, or Mnli (24.27)are satisfied. We have either Mni for I < 0, but in both cases i 1):n) fsn -..
-
-:-1
-
rn
l
Mni
-
1+pyf$
rn
1 (1+;$)1.(1-(y,)IJ-'y-
-m1
nn
n
-
.
(24.36)
izu1
Simplifying shows that condition (24.34)is necessary and sufticient for independently of the value of a, note. Next, (24.25)is equivalent to
2 max cnt
a-1
on
=
lfrsn
which (sincethe maximum is at t requirement
maxlp,ol In view of
(24.34)this
-
'/
<
<
rn
f=1
-
rl+;V2yfV2 n
n
n
t
=
1 otherwise) imposes the
(24.38) '/
<
0, and is equivalent to just
(24.39)
a.
Also, M ni
.
a.
constraint only binds if
-y
(:,4 Ly)
1)2 0, and
n for
=
),
(24.27),
-('y+1)/2
The Central Limit Theorem
390
G(1+j/2)+((l
-N
-u)k-y-
1)/2
(24.40)
,
and (24.26)reduces to
2: +y- jl
a f
1+20
(24.41)
.
l
The existence of (x satisfying (24.41)and (24.39)requires two strict inequalities and 0, but since 0 is positive the first of these, wlich to be satistied by by > (fJ is 0 $, holds (24.34).The second is -F1 + 20) < 20 + y- p, which is the same as (24.35). lt follows that (24.34)and (24.36) are equivalent to 24.6(d), as asserted. . 'y,
,
-
ln terms of the leading case with Xnj (l'f bjlsn, with sn2 given j)y (:4 ao consider first the case cl /, j k 0. To have cnt monotone in t (not essential, but analytically convenient), we could often set =
-
.
,
-
(24.42) max (cx klsn. ix st l under 24.6*) and 24.7(c'), the conditions of 17.7 are satisfied. In this case, Note however that 11 l'r wllr c, is required by 24.6(b). Since ((1)is of size we have, substituting into (24.30), cnt
=
-1
-
-
n
t 1
n
-
+ 2X//2 2 7'2fl5 7) (f
Sn <<
k)F/2(
-
>1
=
()(n1+@).
(24.43)
k=1
>1
This only provides an upper bound, and condition 24.7(c') alone does not exclude a1+Y sl for some < p.However, compliance with condition the poss ibility that n 2 2 (24.34), which follows in tur from M.7(d ' ) (note, Mn cnn in this case), This condition says that the variance of the sum must grow no more enforces X= f'1. slowly than the sum of the vmiances. Now consider the case cf2 t' with 13< 0. Here, we might be able to set 'y
-
=
-
cnt
=
suptcxl/-n.
(24.44)
st
c2,
d (24 34) we would again have n under24.7(c ') an f* < for hence Mnl a-I5-1 and 24.7(d') c2./5.2 some n .
t
=;
-
y
with ('J = y, condition (24.35)reduces to
-
t G-p-1but here Mln = ceases to hold. However, ,
f5> 1-2e +ae,
(24.45)
and it is possible for the conditions of 24.12 to be satisfied, although only with 0 > 0. As 0 is increased, limiting the dependence, a can be increased according to (24.36) and this allows smaller p, increasing the permitted rate of degeneration. As 0 approaches z1,so that the mixing size approaches 2), ;$may approach -17, with (x also approaching These conclusions are summarized in the following corollary. Pal't (i) is a case -2r/(r
-
.
CLTS
of 24.7,
(ii) a
and pal't
for Depettdent
Processes
391
case of 24.6.
rI1+ Let Xnt (l'f ptllsn where cl If either t 15and u (i) 0 ; I < x, and 24.6(b) and 24.7(c') hold with cnt detined in (24.42);or (ii) there exists 0 such that (24.45),24.6(b) and 24.6(c) hold with cnt defined by (24.44); then Sn .-P->N0, 1). (:I
24 13 Corollary
=
@
-
-
-
By an apparent artefact of the proof, t -1/2 represents a jjmit on the permitted rate of degeneration of the variances. We may conjecture that with mixing sizes but a different method of exceeding 2r/(r 2), the CLT can hold with 13f attack would be necessary to obtain this result. Also, both the above cases appear to require that (x e (0,r1),and it is not clear whether larger values of (x might be possible generally. The plausibility of both these conjectures is strengthened by the existence of the following special case. -,1,
-
24.14 Corollary If conditions 24.6(a) and (b) hold, and also (c'') (aYn t,5nt ) is a martingale difference array, and (d'') conditions (24.25)and (24.27)hold with bn 1, then Sn -P- N(0,1). n -.x
,
=
Jn2 E7=jo'2f the case Xnt (Ff (Qlsn,where by the m.d. assumption note that (24.27)is satisfied with bn 1 by constnlction, and (24.25)requires only / is permitted for any 1$ under 2.14(d''). This that sln x, so that result may be compared with 24.4, whose conditions it extends rather as 24.6 extends 24.7. The proof will be given below after the intermediate results for the proof of 24.6 have been established. As an example, 1et Fr g,rbe a m.d. with nt2 nllt where c2 is constant, so that Jn2 log n. Corollary 24.14 establishes that = ln
=
=
-
,
=
(U
'1-
-1
-
-
-
(log n4 -1/2)
(F t
.1
w)-P-+ N,
-
c2).
The limit on the permissable rate of degeneration here is set by the requirement
x. I f the variances are summable, the central limit theorem surely sn2 cxa as n fails. Here is a well-known case where the non-summability condition is violated. --y
-->
24.15 Example Consider the sequence ( Ff) of first differences, with F1 Z1 and =
Ff
=
Zt Zt- 1 t -
,
=
2,3,
...,
where (4) is azero-mean, unifonnly Lr-bounded, independentsequence, with r> 2. Here f Ff) satisfies 24.6(a)-(c), but E:=1F, Zn and sn2 V ar (Z n ) /41). n =
=
=
24.5 Proving the CLT by the Bernstein Blocling Method We shall prove only 24.6, the arguments being at worst a mild extension of those required for 24.7. We show that with a suitable Bernstein-type blocking scheme the blocks will satisfy the conditions of 24.1. In effect, we show thay the blocks behave like martingale differences. In most applications of the Bernstein approch alternating big and small blocks are defined, with the small blocks containing of
The Central Limit Theorem
392
the order of (/l15tl-t* 1summan ds for some ;'$e (0,1),small enough that their omission is negligible in the limit but increasing with n so that the big blpcks are asymptotically independent. Our martingale difference approximation method has the advantage that the small blocks can be dispensed with. Define bn and rn aj in condition 24.6(d) for some Zni
ibn Xntni
77
=
e
(0,1),and
(24.46)
1....,rn,
=
,-(f-l)!u+1
such that Sn
n =
let
rn
Xnt
Zn
=
f=1
,..1
+
+ + Xnn). xn,rnbn-vb
(24.47)
...
The final fragment has fewer than bn tenns, and is asymptotically negligible in 1. the sense that bnrnln Our method is to show the existence of (x > 0 for which an array lZnj) can be constructed according to (24.46),such that 24.1(c) is satisfied, and such that the truncated sequence (Ql defined in (24.10)satisfies the other conditions of 24.1. This will be sufficient to prove the theorem, since then E;212nf-T-r.->1 according to 24.2(ii), and Zni and 2nfare equivalent sequences when 24.1(c) holds, by 24.2(iii). according to M.8, we may apply Since lA%rl is a fmmixingale array of size 16.14 to establish that the sequences lmaxy(/-1):n+1Xn/)2*2uj, j n e EN (24.48) -->
-,1
l ib 2 At least this follows Zf=yf-lllpa+lcnl. are unifonnly integrable, where vnf 1, and it is also clear that the result generalizes to directly in the case i for, although the starting point i 1):n + 1 is increasing with n, the any of the sequence in (24.48)can be embedded in a sequence with fixed coordinate nth starting point which is uniformly integrable by 16.14. This holds uniformly in n l lvlni, 1 S i S r ,,, ra k 1 ) and f, and it follows in particular that the array (Znj is uniformly integrable. This result leads to the following theorem. =
,
=
-
,
24.16 Theorem condition.
Under assumptions M.6(a)-(d), (Znf) satisfies 2
the Lindeberg
1 S bnMni cxl by 0 as n Proof For any i, vnj (24.25)and hence, for : > 0, F(1( tzmifvnit>gvniizlni) E (1(1z,,j1>s)Z2,jj) f max max 2 dhfn2 Lsisrn lsifura vnf -->
--
--> 0 as n by unifonn integrability. The conclusion,
--)
(>a
(24.49)
for Dependent
CLTS
ftlt Iz.fI>i:)Z1)
rn
lzxI>s)Zn$) f
f(1t j=1
393
Processes
max
bnsl'ni 2
lsxra
-- 0 as n
ra
77bnulnj ./.l
(24.50)
-->
x,
now follows since the sum of rn terms on the majorant side is 0(1) from (24.27).. This theorem leads in turn to two further crucial results. First, by way of 23.16 we know that condition 24.1(d) is satistied (and if this holds for Zni it is certainly true for Zj);and second, note that for any e: > 0 rn
2 f max Zni
2
2
1/ i K rn
(24.51)
1( Izwl>i:)Znj.
E + f=1
Taking expectations of both sides of this inequality, and then the limit as n x, shpws that (24.11) holds, and therefore that %nis uniformly integrable (that is, condition 24.1(a) holds) by 24.2(i). This leaves two results yet to be proven, Ehnj to 24.1(b) for the truncated array), and 24.1(c). By -- 1 (corresponding result latter shall establish the pal'ts (ii) and (iii) of 24.2, and then in we view of 24.2(iii) the proof will be complete. We tackle 24.1(c) first. Consider -..-h
rn
ibn
rn
2 1 Zni f=1
2
Xnt
=
-
=1
1
-
=
(24.52)
Au Bn, -
r=(f-1)&+1
say, where rn
An
znli Eznlil) 77 f=l
=
-
ibn
rn = and B
=
ibn-t
2 (Xnt
-
f=1 #'
2 Gntj +
2
r=(f-l)>+1 +
(Xntxn,t.k.kGnt,t-
,
(24.53)
k=l, t
#'' where ibn
ra
B'n
=
f=1 n-1
B'n'
=
n-t
2
X t=rnbn-k
f=t-
1)!u+1 kzzzibn-t-j
c2nr+ 2
n-t
c nr, 1+k
cn,,,+1 + X &1
,
cznn.
(24.54)
(24.55)
+ c2su Here, Gnts Exntxns), and recall that ESnl4 )=-l(c2r & + zlzg-vjototy) 1. It may be helpful to visualize these expressions as made up of elements from outer product and covariance matrices divided into n blocks of dimension bn x bn, with a border corresponding to the final n rnbn terms, if any; see Fig. 24.1. =
=
-
=
Ihe
394
Central Limit Theorem
The terms in An correspond to the rn diagonal blocks, and Bn' and B'n'contain the remaining covariances, those from the off-diagonal and final blocks.
1
2
3
.
.
rn
.
b l //z (!) n
OZZ
2
z'
/?,,
Fz
3
!
:
l
:
:
(
l
l
!
y yy /,,,'''z. '-jy-'t
r
Bn :
2
:
)
I
I I
.
g y Fig. 24.1
An is stochastic and so must be shown to converge in probability, whereas the components of Bn are nonstochastic. The nonstochastic part of the problem is the more straightforward, so we start with this.
24.17 Theorem Under assumptions 24.6(a)Vd),Bn Proof Since r
>
2, r/tr
1) < 2 and conditions
-
InntsI f where ((,u) is a constant
(c) imply
by 17.7 that
(24.56)
,
-1.
Hence,
bn
77 7777
fw+1
=1
bn
24.6(b) and
cngyxljf-.sl
sequence of size
rn ra+ l
lBn'I f 277
0.
-->
n.i-jlbn-vj,q-tlbn-vk
1=l
./=1
rn rn+ 1
bn
sinisl'nl << l=f+l j=$ =1
bn
al
j j t-ijbn-vk-jI
(24.57)
.
To detennine the order of magnitude of the majorant expression, verify that the + ) and some tenns in the parentheses are Obn1- l ) ) j or j j + j order of summation and putting k l i allows us to write 5 > 0. Changing the -1-
...yyk
=
-
,
=
rn+ 1
j
-
-.k
1-
lBnz I << bn
rn
-1-
MniMn,i-vkk
al
But for every k in this sum, the Cauchy-schwartz ra+ 1
-.k
rn+ 1
77l MniMn,i..kK XMS izz1 =
;r
.
(24.58)
=1
inequality and
&(:-n'),
(24.27)give (24.59)
for Dependent
CLTS
'1 I#n
Ob-h
=
n
395
that
as required,
so that (24.58)implies,
Processes
O(rl-Y1-*
=
(24.60)
).
To complete the proof we can also show, by a similar kind of argumentbut applying (24.25), that /(1). OMl a,ra+1 b . Bn'' (24.61) a) =
=
2 Ezni)2 in To solve the stochastic part of the problem, decompose the terms Zni (24.53) into individual mixingale components, each indexed by i 1,...,rn. For a pair of positive integers j and k, 1et -
=
1PnjU,k)Xn,(-Ipa+./X,,,(f-I)!u+.j+k =
It is an immediate consequence of 24.9, specifically of for fixed j and k the triangular array
( Yij,kl,
FafZx;
=
=
Substituting from
=
=
k) is an Al-mixingale of size (?-l)u+jfor p k 1, and constants
bn
-
-
+
,
cn, (sl)?u+./c,,,
bn 1 =
with
(24.64)
i-tlbn-qn.
have the sum of bn2 suc h tnixingales,
-.j
X JLj(#,0)+ 277 j=1
(24.63) -1,
(24.62)into (24.53),we
Zlni Ezlni)
(24.32)and (24.33),that
1 S i f rn, n k N(j.kq 1,
where NU,k) minln'. rn 1, bn k j mixing coefficients tftl (0 and vp
anij,k)
(24.62)
cn,(-1)u+./,(f-1pn+./+t.
-
Wnijnkj
+
Wuj(:u,0).
(24.65)
k=1
Although this definition entails considering k of order bn as n .-A x, note that the inter-block dependence of the summands does not depend on k. The designation is convenient here, but it need not be taken more literally than of the inequalities (24.32)and (24.33)require. The crucial conclusion is that a weak 1aw of large numbers can be applied to these terms. tmixingale'
24.18 Theorem Under assumptions
24.6(a)-(d), Au -E(-> 0.
The object here is to show that the array EZlni)), Fjnx', 1 f i K rn, rn 2 1 ( )
(Xj -
(:1
(24.66)
which satisfy conditions (a),(b), and (c) is an f-j-mixingale with constants ttzajl of 19.11. The proof could be simplified in the present case by using the fact that ble (by 24.6(a)) so that the minorant side of (24.68)below Zni is 5ib n, Cx-measura actually disappears identically for p k 0. But since the result could find other applications, we establish the mixingale property formally, without appealing to this assumption.
2 an d app 1y ing 24-9 term by term, we obtain Proof By multiplying out Zni
)
E Ezlni
Ftzazjlj@-( .f
-
-*bn).
j
The Central Limit Theorem
396
bn- 1
f
j
77 jzc1
l'-tx
E ElWnii,ot
I+ + l
-*bn)
bn
-j
j
Ewniqvk) ITf-f
lTE k=1
lF.-f
E EWnibn'ot
-*bn)
j
j
''pbbnt
bn- 1
Mnli
rx
y) (1+ lbn j=3
-./))j(p-jpu+j
+
jpx
(24.67)
,
and similarly,
l
j
+*bnj E Znli Eznli lFxl' -
Mnli
bn 1
bn
-
-.j
+ Lgn 77 )(;,+1)s-y+ 2F7((p+1)u-j-k k=1
t<
,
j=1
(24.68)
where (j 0j -1-8) for 45 > 0 Write formally, tznjv; to denote the larger of the two majorant expressions in (24.67)and (24.68),such that vp* 0 and ani is fixed by setting v; 1. Evaluating (24.68)at p 1 and (24.68)at p 0 respectively gives =
,
.
-->
=
=
=
bn 1 -
1
E E (Z?lj Ezln j Txf
j
-1)9n)
-
<<Mytj
y'q(1 + j=1
-8
-y))./-1
zlbt
+
y-u1-8
lb n, <<sl'ni and also, putting j'
=
bn
and k'
-j
=
(24.69)
j' k -
bn- 1
E Zni 2 F(Z2ni j5
i bn)
-
--
<<M1 ni
-')
j'- 1
.4-1-8
+
27)/c'-1-
+
1
(24.70)
k'=.*
.j'=l 1 <<sfnibn .
Hence, ani
=
2b n for some tinite constant B. Since BMni rn
rn
XM4fn f f=1
max
MnliTMnli
lKK rn+ 1
=
olbn-lj
(24.71)
f=1
in view of (24.27)and (24.25),these constants satisfy conditions 19.11(b) and /v2ni where Z2f/w2jis unifonnly integrable, they 2Ibn Mlni < z2 ni (c). And since Zni n follows that An f' 0 and the proof is satisfy condition It 19.11(a). also complete.. ---1..+
,
This brings us to the final step in the argument, establishing the asymptotic m.d. property of the Bernstein blocks.
24.19 Theorem Under 24.6(a)-(d), limn-x/ra) Proof Applying
(24.17),
=
1..
for Dependent
CLTS
Processes
rn
'-rrn
1-I
=
=1
+ I7f-l(1 whereFf-1 sl =
(1 +
,
rn
z-n i)
l
=
+
i
T-i
z-
l ni
-
(24.72)
,
i= 1
lxn-measurable sin,--
ihk nj is an )
397
by 24.6(a), and hence
r.v.
rn
ily7Ftwf-lznj)
e'twrn)
1+
=
f=1 rn
= By the Cauchy-schwartz
1 +i,
E f=1
f=1
(24.73)
'bn) ?-l lI2IIF(2nfI@,;t,f--1 11 112,
(24.74)
-
.
inequality, rn
ra
EI
tf 1' bn )) '/f-1 E (kni 15 n,-x
'i-tEkni I@nt,':1 tbn)l /
=1
where11 h-1lI2is uniformly bounded by (24.15),which follows in turn by (24.11), the result hinges on the rate of approach to zero of IlF(znf Isni---bbn) This 112. so
cannot be less than that for Zni, so consider, more conveniently, the latter case. l
ibn
'bn)
IIF(z,,f lsy'ntf---' 2 11
E
=
1
77
1/2
'bn) Exnt I1,$'---1 '$
t-ni- 1)!u+1
il',t 'rltf--''t
Yl
f
t--zi- 1)u+1
E CE x
ibn- 1
+
Xn))2
I
1/2
ibn-t
:--) 7) IEExnt3 2f=(f-1)!u+1
Tnt--l'bnjexnt-vk3
'n)) @nt'--1
k=l
l
(24.71)
Applying 24.8, E(E(Xnt3 Fjf--l 'bnljl < c2uf((,-(j.j)s)2
(24.76)
and
!
E EExnt
*n)) 'bn4Exnt..kI@,$,f--1 IFntf,---1
j
Xn) 1 1I.E'(.kr+,, sni---'bn, 2 IIF(-Yn,I Isni--11 11 A
1
S cn?(,-(f-1)&ncn,,+1(,-(f-l):n+z,
where
(j
=
O(j-1/2-B
) for g, > 0
for the 0 defined in 24.6(c). Hence, bn
IIA'(Znf << Mni ITntf--xlxnlllz A
(24.77)
bn- 1
bn
1/2
(J?
7-7 + 277 Lj77(k
/=1
/=1
(24.78)
P=7+1
Applying (24.26), where the sum of squares is 041) and the double sum is O(:1-2pl) n 24.6(d) implies assumption .
The Central Limit Theorem
398
rn
'X l f=1
A'(/
-1
Ek
n
/1
l *n44
sni,-x
rn
I
A7A'/',,tn(max(!?,,t'-2'z)'21 1)
-
,
=1
tl-g jy-l/z = Otmaxtha n
))
,
This ensures that
71F(h-1Z--
0 as n
ol j;
=
(g4yo)
.
.
which is the desired result.
-->
x,
.
Proof of 24.6 We have established that Erilznf lbjnxnt-P-> N(0, 1). There remains the formality of extending the same conclusion to Sn, but this is easy =
since
lSn E;t1bnXnt l -
IXn'rn b
=
n
+1
+
...
+
XnnI
=
Op(rn-1/2)
(24 80)
,
.
and Sn has the same limiting distribution by 22.18. The proof of 24.6 is therefore complete. * It remains just to modify the proof for the martingale difference case, as was promised above. Proof of 24.14 It is easily seen that 24.16 holds for rn n and bn 1. ln 24.17, Bn 0 identically since (k 0 for k > 0 in (24.57).In 24.18 one may put Zni Xnt, and the conditions for 19.10 follow directly from the assumptions. Lastly, 24.19 holds since the sum in (24.79)vanishes identically under the martingale difference assumption. The proof is completed just as for 24.6. . =
=
=
=
=
25 Some Extensions
The CLT with Estimated Normalization The results of the last two chapters, applied to the case Xnt Xtlsn where EXt) E(X3Xt)l, would not be particularly useful if it were necessary to = 0 and 1 in order to apply them. know the sequences (cf2) and (cf,t-k)f or k Obviously, the relevant normalizing constants must be estimated in practice. Consider the independent case initially, and let Sn Xlcjxtlsnwhere sn2 l E)=I.Yl, 2 Also 1et n and we may write E:-1c,. =
.2n
=
,
,
=
=
=
sn
=
1
n
Sn
>1
-
xt
=
dnsn,
(25.1) -%
snln. If dn -T-+ 1, we could appeal to 22.14 to show that n #(0,1) whenever Sn -P-> N(0,1). The interesting question is whether the minimal conditions sufficient for the CLT are also sufficient for the relevant convergence W
here dn
=
in probability. If the sequence is stationary s well as independent, existence of the variance ..C..A c2 (applying23.5 c 2 is sufficient for both the CLT (23.3)and for rl-1E')=1Xl 2 to Xt4. I n the heterogeneous case, we do not have a weak law of large numbers for lXf2l b ase d solely on the Lindeberg condition. However, the various sufticient conditions for the Lindeberg condition given in Chapter 23, based on unifonn integrability, are sufficient for a WLLN. Without loss of generality, take the case of possibly trending variances. the conditions of
25.1 Theorem lf (XfJis an independent sequence satisfying 23.18, theh
(25.2) this has zero mean, is Proof Consider the sequence (Xt2 c2)/c2 t t. By assumption independent (andhence an m.d.), and unifonnly integrable. The conditions of 19.8, 2 1, bt ct,2 and an with p sn, are satisfied since, by (ga 5y) -
=
=
=
N
2S Xcr >1 where Mn
=
maxlxfxnf ctj
.
Hence
.
nM2 n
=
O(sl) n
,
,
(25.3)
The Central Limit Theorem
400
n Xl ),-1
-
,
E
c2) , =
2 sn
which is sufticient
E
En,- IF,
1
-
2
0,
--
(25.4)
sn
for convergence in probability.
.
(.V)is a martingale
When the sequence
needed for (X,2) to o bey for the mmingale CLTS precisely to the required
difference, spplementary conditions are but WLLN these turn out to be the same as are needed a of 24.3 and 24.4. In fact, condition 24.3(a) corresponds result. We have, immediately, the following theorem. ,
25.2 Theorem Let (m,@t)be a m.d., and let the conditions of 24.3 or 24.4 be satisfied; then (25.2)holds. I:l Although we have spoken of estimating the variance, there is clearly no neces2 sity for (snlnj, or any ot h er sequence ap art from fdn ), to converge. Example 24.11 is a case in point. ln those globally covariance stationary cases (see j13.2) W here snn converges variance' to a finite positive constant, say -c2,the nlln as a consistent estimator of d2. of the sequence, we conventionally refer to l But more generally, the same terminology can always be applied to n with respect to sn, 2 i n the sense of (25.2). Alternative variance estimators can sometimes be defined which exploit the particular structure of the random sequence. ln regression analysis, we typically apply the CLT to sequences of the form Xt Wtut where (Ut,5tIis assumed to be a m.d. with fixed variance c2 (the disturbance), and where Bs (a regressor) is In this case, (A),Tr) is a m.d. with variances czf c2F(Wl), 5t jr2)z).j<2j, 5k2. W for This is the usual hichsuggests the estimator :,,2 (rI-1E) t approach in regression analysis, but of course the mthod is not robust to the Z)=1
=
-l-measurable.
=
=
=
=
scedasticity-consistent. 26
Now consider the case of general dependence. The complicating factor here is 5.2 l that n contains covariances as well as variances, and n is no longer a suitable estimator. A sample analogue of sn2 must i nc lude terms of the fonn Xtxj for qjI k 0, but the problem is to know how many of these to include. 1 as well as foj If we include all of them, in other words for j 1 J,...,n t, the resulting Jn2 is converging, not and the ratio of this quantity to sum is equal to (IJ=1m)2, l 1) consistent estimation we probability 1, but in For in distribution to k ( to must make use of the knowledge that a11but a finite number of the covariances are arbitrarily close to 0, and omit the corresponding sample products from the sum. Similarly to the m.d. case, the conditions of 24.6 contain the required convergence result. Consider (24.46)and (24.47)where Xnt Xtlsn, but now write =
=
-
-
.
=
Zr:j snzni =
ibn
=
F7
Xt, i
>(f-1)!u+1
k=
1,...,r,,.
(25.5)
Some Extensions
401
In the proof of the CLT the construction of the Bernstein blocks was purely conceptual, but we might considerthe option of actually computing them. The sum Ehjzipj is consistent for in the of squares of the unweighted blocks, sense that .2n1
.%2
=
-2
,
r 'N
Jn1
2
Sn
=
2 Zni
---+
pr
1
(25.6)
,
j=1
according to 24.17 and 24.18. An important rider to this proposal is that 24.17 and 24.18 were proved using only (24.25)and (24.27),so that, as noted previously, any e (0,1) will serve to construct the blocks. In the context of the conditions of 24.12 at least, the only constraint imposed by 24.6(d) is represented by (24.39),which puts a possible lower bound on (x in the case of decreasing variances ('y> 0), but no upper bound strictly less than 1. It is sufficient for consistency if bn goes to infinity at a positive rate, and we are not bound to that satisfies the conditions of the CLT to construct the blocks in use the (25.6). But although consistent, J1j is not the obvious choice of estimator. It would be andGallantandWhite more naturaltofollow authors suchasNeweyandWesttlg87) who consider estimators alia, based all the cross-products Xtxt-k (1988), inter on for t = k + 1,...,n and k = 0,...,:,,. In terms of the array represention of Fig. 24.1, these are the elements in the diagonal band of width lbn, rather than the diagonal blocks only. (In this context, bn is referred to as the band width.) The simplest such estimator is
-2
Snl
bn
n
&1 r=kml
of 24.6, applied to Xtlsn,
Under the conditions
*
(25.7)
t +
>1
25 3 Theorem
n
Xx2 zy-l y-qxtxt-k.
=
nhlsnl
-CJ-y 1.
2 Prtmf Let Xnt Xlsn in (24.53),so that sJn deno teS the same sum constructed lz Elnhlljsnl from the Xt in place Of the Xnt. The difference between An and is the quantity =
-
ra 1 -
4* n
=
-
1 2
Sn
ibn
ibn-t- 1
2
xtxt-k okt-
f=1
/=(/-1),n+1
t=l n
77
t-nrnbn-j
bn- 1
(x2, c2,) + -
277 k=1
xtxt-k-c,,,-,)
(25.8)
The components of this sum correspond to the rn 1 triangles' which separate the 1) tenns, plus the terms diagonal blocks in Fig. 24.1, each containing closely analogous to the proof of from the lower-right corner blocks. Reasoning -iL+ 0. The sums of the corresponding covariancesconverge 24 18 shows that An' absolutely to 0 by 24.17, since they are components of Bn' in (24.54),and it folloyvs that -
qbnbn -
402
The Central Limit Theorem l
I n2
l
j
nl
-
-EL>
2
0.
Sn
The theorem therefore follows from 24.18.
.
21 Since this estimator uses sample data which are ( discarded in n there are informal reasons for preferring it in small samples. But there is a problem in 9,,22 is not a square, and nOt always that (the chosen notation notwithstanding) in limit. This difficulty the non-negative except can be overcome by inserting tixed weights in the sum. as in ,
n
2a XV =
bn
tzz1
Suppose wsk
1 as n
.--:
l l n2
l
n3
-
j
Sn2
k,z1
x
bn
2
= X(1 -
n
X Xtxt-k
wnk)
k=1
'n
X Xtxt-k.
(25.9)
t=k+ 1
for every k S K, for every fixed K
-->
'-'
n
2/7 wa:
+
-PI->
< x.
r(A3,
r=k+l
Then
(25.10)
where r(A')
2 plim
=
n-yx
bn
1
n
X
-j
Sn t=#+1
(1
wnk)
-
X Xtxt-k
(25. 11)
.
/=k+1
Since r(A') can be made as small as desired by taking K large enough, in view of l i s cons istent when the weights have this property. lt remains 25.3 and 24.18, na to see if they can also be chosen in such a way that (25.9)is a sum of squares. Following Gallant and White (1988),choose :n+ 1 real numbers Jn1,...,tzn,u+1, satisfying Zyktlty 1, and consider the n + bn variables =
F,,l
=
1'n2 =
Fnb +1 Fn bnn 'N
tzl-Y, J,,lX2 + Jn2X1,
=
anjxbn..t +
=
anjxbnn
+
l',,n antxn
+
=
anlxn
+
=
anbnxn.
=
Fn n+1 F,l,,+,n
...
...
...
...
+ +
+
an,bn-vjxj,
+
ann-vtxb
an,yn-vjxn-bn, anbnxn-bn-j,
Observe that n+&a
X /=1
F2nt
ha+1 =
X j=1
anlj
n >1
:a+l
+ 2 77 j-a
:n+1
V
+
2
X
n
mAgj
ayanj-j
j=2
>2
n
anjansi-z 77.Y,1,-2 ,.-:!
n
+
...
Xtx-bn,
+ zJa,u+llnl ,-0-+1
(25.12)
Some Extensions
403
janianj-k, k 0,...,& hich shows that any weights of the form wn1 Z/+z+l non-negativity, and give 1. also A case that fulfils the consistency impose wuo bn + 1)-1/2 alJ j which yields wuk 1 kllbn + 1). requirement is anj W
=
=
=
=
,
=
,
-
Variance estimators having the general form l
n
'X 77wt lt JIlbnbxtxs -
2...=1 .=l
as kernel estimators. The function w@) 1 Ixl for lxl < 1, 0 otherareknown 2z, which corresponds to 923, W is the Bartlett kernel. The estimator by n ise, =
-
1( Ix!u1). Other possibilities exist; contrast, uses the truncated kernel, w@) see Andrews (1991)for details. One other point to note is that much of the literature on covariance matrix estimation relies on fu-convergence results for the sums and products, and accordingly requires fv-boundedness of the variables for r k 4. The present results hold under the milder moment conditions sufficient for the CLT, by using a weak law of large numbers based on 19.11. See Hansen (1992b)for a comparable approach to the problem. =
25.2 The CLT with Random Norming Here is a problem not unconnected with the last one. We will discuss it in the context of the m.d. case for simplicity, but it clearly exists more generally. Consider a m.d. sequence (XtJ which instead of (25.2)has the property 2
n Ztxlxt .L> 2
Sn
z
n
(25. 13)
,
whereq 2i s a ran dom variable. This might arise in the following manner. Let Xt Bs&rwhere f f/f,lifltx'-xisam.d. and f W'f r.v.s which aremeasurable The S respect rmote c-field to the O=,=-x;x. implication is that W'fis with <strongly exogenous' (see Engle et al. 1983). with respect to the generation =
lO-xasequenceof
=
mechanism of &t. Then Ext
l;r-1)
=
WtEut I;r-1)
sinceR c 9,-1 for every 2t, 2hence 2 sucha way that Z:=1W'fUtlsn
=
0 a.s.,
(25.14)
Xt is a m.d. Provided the J'I are distributed in 1 the analysis can proceed exactly as in j24.2. need There is no practical to draw a distinction between nonstochastic Gf and S-measurable JK.But this need not be the case, as the following example shows. .-tr-.y
,
25.4 Example Anticipating some distributional results from Part VI, 1et W'f s2 for a11s. Also t J/x w here f&',Jis a stationary m.d. sequence with F(72)x Z,=1 2 = 2 2 2 2 c al1 /. Then, A'(W',) /T and sn assume, for simplicity, that Eut) 1 + 1)'c2 If we further assume that ( J'',) satisties the conditions of 24-3, it gnn will be shown below (see 27.14) that =
=
=
.
,
=
The Central Limit Theorem
404
n
1 T
X
2 2
n
WQf
-G
1
f=
j
1
Blrlldr,
(25.15)
0
where #(r) is a Brownian motion process. Under the distribution conditional upon R, we may treat the sequence ( F) ) as nonstochastic and apply the weak LLN (byan argument paralleling 25.2) to say that
2 2 plim 12:-l7r,
plim a-+x
2
Sn
n.->oo
WlUl , t
)
-1
=
=
Ilglnln
+
1)
2 lim u-yx
:7
+2
-1
t =
zlnl
2
n
(25.16)
,
wheren2mere2 ly denotes the limit indicated. Under the joint distribution of (UtI l is a drawing from the lirniting distribution specifie d iu (J,5 1j). o and (JK),z'q .
The application of the CLT to (XfJcan proceed under the conditional distribution (defined according to 10.30), replacing each expectation by E. lS). Let Gt2 = Eult S), defining an S-measurable random sequence, so that I n
E
X
11
X2t R
X +2c2
=
t
>1
(25. 17)
t.
>1
@'
then aPP1y a result fOr heterogeneously distributed sequences, such as and that condition 4 ltting ct = W(>. Assuming that E (Xl j 5t- 1) (24. 19) is almost surely satisfied, the conditional CLT takes the fonn
WC
C2X
24
Fl(U,
=
1
1imE exp
n
ik
2 2 1/2
X (ZJ-1W-Jc,)
Txt
S
=
e-X 2 /2 '
(25. 18)
a s. *
,-1
variance, the situation is quite But if we normalize by the J/nconditional different. Bt must be treated as stochastic and Ext 2I5 ?-1 ) EXl)f so the conditions of 24.4 are violated. However, if (25.13)holds with n an S-measurable r.v., then according to 22.14(ii) the conditional distribution has the property :y:
,
1
n
R -P-+ x(O, n2),a.s.
->7x,
(25. 19)
>1
(see j10.6 for the relevant theory). This result can also be expressed in the fonn l
ih
-77x,
limE
exp
n-px
Sn
n
R
lim
e-R
2
nu a.s.
(25.20)
>1
Hence, the limiting unconditional E exp
=
ih
distribution is given by n
-Ex, Sn
>1
Ee-k nha 2
=
).
(25.21)
This is a novel central limit result, because we have established that kl=txtlsn is not asymptotically Gaussian. The right-hand side of (25.21)is the ch.f. of a mixed Gaussian distribution. One may visualize this distribution by noting that
Some Extensions
405
drawings can be generated in the following way. First, draw 2 fr om the approprin ate distribution, for example the functional of Brownian motion defined in (25. 15).,then draw a standard Gaussian variate and multiply it by n. If X is mixed
Gaussian with respect to a marginal c.d.f. G(n) (say),and (n is the Gaussian density with mean 0 and variance n2 th e momen ts of the distribution are easily computed, and as well as F(Ar) EX 3) () we t'intj ,
=
EX
2)
=
jj* p
.
=
xzjqtxllxt/Gtql
jXnQG(q)
=
F(q2).
=
(25.22)
0
xx
However, the kurtosis is non-Gaussian, for Ef)
=
-f
j
0
where the right-hand
-
1 j.jtlnoctn) -
x4tktxldxgstnl
-
-
-x
side is in general different from
s(q4)
(25.23)
,
a
A'(q2)2
(see 9.7).
25.3 The Multivariate CLT An array (Akf) of p-vectors of random variables is said to satisfy the CLT if the joint distribution of S,i L1=3Xntconverges weakly to the multivariate Gaussian. ln its multivariate version, the central limit theorem contributes a new and powerful approximation result. Given a vector of stochastic processes exhibiting amongthemselves, wecan show thatthere arbitrary (contemporaneousldependence exist different linear combinations of the processes which are asymptotically independent of one another (uncorrelatedGaussian variables being of course independent). This is a fundamental result in the general theory of asymptotic inference in econometrics. The main step in the solution to the multivariate problem sometimes goes by the name of the dcramr-Wold device' =
.
25.5 Cramr-Wold theorem A vector random sequnce (& JT,Sn e Rk converges in distribution to a random vector S if and only if 'Sn -P-> 'S for every # 0. tixed k-vector ,
Proof For given the characteristic function of the scalar 'Sn is Eexplih'sn )) .-P-> and 'Sn if only continuity theorem if Lvy 'S (22.17), = #s,a(). By the 0. Since is arbitrary, we can put t 9u,(x(,) a(D and a is continuous at tx, and obtain -->
,
=
=
Eexvtit'sn
1)
--)
Ftexpf it's)
=
v(/),
(25.24)
(say) where by assumption the convergence is pointwise on Rk B y (1139), the left-hand side of (25.24)is the ch.f. of Sn, and the right-hand side is the ch.f of S. The continuity of at the origin is ensured by the continuity of ju at 0 v follows that Sn -P-y S. w a11 and for it .
.
.
,
Now 1et (41 be a sequence of random vectors, and let En be the variance matrix of 17=1,X).Being symmetric and positive iemidefinite by construction, this matrix Possesses the factorization '
The Central Limit Theorem
406 EN
Cnhncn'
=
=
Lnl,
(25.25)
whereLn Cnhn1/2c n an d An being respectively the eigenvector cncn' Cn'Cn= Ip) and the diagonal, non-negative matrix of =
,
matrix
(satisfying
eigenvalues. h-*C' = Let Xnt Ln-X ,, w here if E n has full rank then L-n n n so that Z-E n n Z-' n rank singular with for every n. If it is &. However, I:a need not have full 1n (,0 An =
=
=
/
=
(f%1A-In1/2'
gj ,
01 Where Cnj is the appropriate submatrix of Cn. In this case, L-nXnL-n'has either ones or zeros on the diagonal. We do however require En Iv. lf Sn to be asymptotically of f'ull rank, in the sense that LxJu-n' XzxlA%f,then for any p-vector with ' 1 we have E's n 41 1 If this such exists 'S # 0 0 the asymptotic condition fails, and there that E( n 41 distribution of Sn is said to be singular. In this case, some elements of the limiting vector are linear combinations of the remainder. Their distribution is therefore determined, and nothing is lost by dropping these variables from the let Z-' l
=
@
-->
=
---y
=
,
.
--
,
analysis. CLT it is necessary to show that the scalar sequences To obtain the multivriate If sufficient conditions hold f 'Xnt satisfy the ordinary scalar CLT, for any -P-) Nl, 1), the Cramr-Wold theorem allows us to say that Sn -P- S, and for 'Sn the ch.f. of 'S is it remains to determine the distribution of S. For any .
,
I()
=
fW))
fexpf
=
e- :2/2
(25.26)
.
be a vector of length it follows from (11.41) that (25.26) But letting t 1.) By the is the ch.f. bf a standard multi-Gaussian vector. (Recall that required therefore theorem the result S N(0, Ip. We have inversion that we get proved the following theorem. =
,
,
'
=
-
25.6 Theorem Let (.Yf) be a stochastic sequence of p-vectors and let En lf L-nxnL-n' Ip and k1=3'L-nXt -P-> N(0,1) for every F((Z:=lAQ(17=laV)'). 1, then satisfying
=
-->
'
=
n
Xz,-n.Y, -P-> x(0,&).n
(25.27)
>1
ln this result the elements of En need not have the same orders of magnitude in n. The variances can be tending to intinity for some elements of Xt, and to zero for others, within the bounds set by the Lindeberg condition. However, in the case when all of the elements of En have the same order of magnitude, say n 15f or some 6 E, a fi nite constant matrix, it is easy to manipulate > 0 such that n - En (25.27) into the form --
n
..P..y xtj)jx;. -&2Nxy
(25.28)
f= 1
Techniques for estimating
the nonnalization factors generalize naturally from
Some Extensions
4O7
the scalar case discussed in j25.1, just like the CLT itself. Consider the m.d. 1=jE(XtX't4.and assume this matrix has rankp asymptotically case in which En above. Under the assumptions of 25.2, the defined in sense =
tz'()(J-lA)-Y;)(z
nt-j'xt
=
a'E l a
'a
,
E&
-T-r-
1
(25.29)
1, where the ratio is always well defined on taking n lyrge for any with enough, by assumption. This suggests that the positive semidefinite matrix E,i ElalffAr; is the natural estimator for Xn. 1 I) can be made as small To be moreprecise: (25.29)says that #(I 'Xnl'xna # 0 by taking n large enough, since the normalization as desired for arbitrar.y unit length cancels in the ratio. This is therefore true in the Iarticular case to G* and since tu*'En) L-' 1, we are able to conclude that 'L-X L-' -E'.% 1. We can further deduce from this fact that L-nxnL-n'--CC->I To show this, note that :a 1 for every # 0, g 'Bl if a matrix B (#xp4 is nonsingular, and g4 lBal' lal', for any and the system of has the gradient vector g'j unique I solution B Ip. If Ln is the factor0 has the equations Bla' asymptotically of rank of since Ln is En, ization p it follows by 18.10(ii) that =
=
-
=
=
.
=
=
=
-
,
=
-
=
LnL-n -.C.L.: Iv, and we arrive at the desired conclusion, for comparison with
(25.27):
i--nxxt -P-> x(0,z?,).
(25.30)
>1
The extension to general dependence is a matter of estimating En by a generalX)1Zn*jZn*; ization of the consistent methods discussed in j25.1, either En1 * where Zni Xib p.f-l)u+l X t. or, letting weights (wnll represent the Bartlett kernel, =
=
kn3
=
n
a
>1
k=1
n
xt-kx'. 77x,x,,+ y7wn1 77 xtx't-k..v
(25.31)
r=k+1
The latter matrix is assuredly positive defnite, since a,'En3 by application of (25.12)with Xt Xtq.
k 0 with arbitrary
=
25.4 Error Estimation There are a number of celebrated theorems in the classical probability literature on the rates at which the deviations of distributions from their weak limits, and also stochastic sequences from their almost sure limits, decline with n. Most are for the independent case, and the extensions to general dependent sequences are not much researched to date. These results will not be treated in detail, but it is useful to know of their existence. For details and proofs the reader is referred to texts such as Chung (1974)and Love (1977). If (F,:) is a sequence of c.d.f.s and Fn = * (the Gaussian c.d.f.), the BerryEssen theorem sets limits on the largest deviation of Fn from *. The setting for
408
The Central Limit Theorem
thisresult is the integer-moment case of the Liapunov CLT; see (23.37). be a zero-mean, independent, f-a-bounded 25.7Berry-Essen theorem Let (m) Jn2 c2,), variances )7.1c2f,and let Fn be the c.d.f. of with 1et f randomsequence, There exists a constant c > 0 such that, for all n, sn l=jxtlsn. =
=
n
SUPIF n @) *@) -
2
l
S C
X E I.Y,I3Isn3.
(25.32)
'n
>1
The measure of distance between functions Fn andm appearing on the left-hand side of (25.32)is the uniform metric (seej5.5). As was noted in j23.1, convergence to the Gaussian limit can be very rapid, with favourable choice of Fn. The BerryEssen bounds represent the worst case' scenario, the slowest rate of uniform convergence of the c.d.f.s to be expected over all sampling distributions having third absolute moments. For the uniformly fo-bounded case in which sn2 = oln) 1/2 inequality (25.32)establishes convergence at the rate n Another famous set of results on rates of convergence goes under the name of the lttw of the iterated logarithm (L1L). These results yield error bounds for the strong law of large numbers although they tell us something important about the rate of weak convergence as well. The best known is the following. ,
.
25.8 Hartman-Wintner then
theorem If (X,) is i.i.d.
nl 1.Y, -
=
limsup n-/oo
jz
c(2n log 1og n4 l/2
with mean
1 a.s.
=
g, and variance c2,
(25.33)
I::I
Notice the extraordinary delicacy of this result, being an equality. lt is equivalent to the condition that, for any E > 0, both
Lnt= lA-, g, -
P
k 1 + :, i.o. 1/2 n4 log log c(2n
=
(25.34)
0,
and Z:-1.X', g, -
#
(25.35)
k
1/2 c(2n 1oglog nj
In words, infinitely many of these sequence coordinates will come arbitrarily ctose to 1, but no more than a finite number will exceed it, almost surely. By syinmetl'y, there is a similar a.s. bound of on the liminf of the sequence. n-1/2) Un der these assumptions, jtaY, g,l/c -P-y No, 1) according to the Lindeberg-l-vy theorem, and so is asymptotically supportedon the whole real line. On the other hand, n -1 7=1(Xf g,l/c 0 a.s. by the law of large numbers. It is uknifeclear there is a function of n, lying between 1 and n 1/2 represent j ng the asymptotic distributions, being edge' between the degenerate and non-degnerate the smallest scaling factor which frustrates a.s. convergence on being applied to -1
-
=
-->
-
,
Some Extensions
409
the sequence of means. The Hartman-Wintner law tells us that the knife-edge is precisely n * (l lOg log n)-1/2 .
A feel for the precision involved can be grasped by trying some numbers: (2 log l?2 3 3! A check with the tabulation of the standard Gaussian 1og 1099) will show that 3.3 is far enough into the tail that the probability probabilities it is arbitrarily close to zero. What the Ll'ta reveals is that for the exceeding of scaled partial sums this probability is zero for some n not exceeding 1099 although not for yet larger n. Be careful to note how this is true even if the Xt variables have the whole real line as their support. For nonstationary sequences there is the following version of the LIL. =
.
,
1974:th.7.5.1)Let t.YflbeindependentandAa-boundedwith zs.g-l-heoremtchung and ).joJ.Then (25.33)holds if for 6r > variancesequence fc2,), let Jn)1+6) Z7-1A'IAl 3 O(u3/(log u n =
=
0,
.
Generalizations to mmingale differences also exist; see Stout and Heyde (1980)inter alia for further details.
(1974)and
Hall
V1 THE F'UNCTIONAL CENTRAL LPWIT THEOREM
26 Weak Convergence in Metric Spaces
26. 1 Probability Measures on a Metric Space In any topological space S, for which open and closed subsets are defihed, the Borel field of S is detined as the smallest c-field containing the open sets (and hence also the closed sets) of S. In this chapter we are concerned with the properties of measurable spaces (S,.$), where S is a metric space endowed with a will always be taken to be the Borel tield of S. metric d, and lf a probability measure p, is defined on the elements of we obtain a probaand referred ((S,#),,$,p,), element is bility space to as a random element. xe S an As in the theory of random variables, it is often convenient to specify an underlying probability space (f,@,#) and 1et ((S,#),,/,/) be a derived space with the propel'ty jt(A) Px -1(4)) for each A e .T, where ./
./,
=
x: f
F->
(S,#)
is a measurable mapping. We shall often write S for (S,#) when the choice of metric is understood, but it is important to keep in mind that d matters in this is not invariant to the choice of metric; unless (/1 and dz are theory, because equivalent metrics, the open sets of (S,J1) are not the same as those of (S,J2). A property of measure spaces that is sometimes useful to assume is regularity (yet another usage of an overworked word, not to be confused with regularity of sequences etc.): (S,,/,p) is called a regular measure space (or g a regular measure with respect to (S,,/)) if for each A e #?and each E > 0 there exists an opn set ,/
OE
and a closed set Cv such that Ce
and g(O:
-
X I %
(26.1)
Q)
(26.2)
< E.
Happily, as the following theorem shows, this condition can be relied upon when S is a metric space.
26.1 Theorem
On a metric space ((S,#),,$), every measure is regular. .9'
Proof Call a set A G regular if it satisfies (26.1)and (26.2).The first step is to show that pny closed set is regular. Let A, fx: #(A,x) < 1/n1, n with of width 1/p.) 1,2,3,... denote a family of open sets. (Think of A a By continuity of WhenA is closed we may write A O''J=1An,andAn A as n 0. For any : > 0 there therefore exists N such the measure this means g.(z4u-A) :4 AN and < Choosing jttAx -A) Oe that Q = A shows that is regular. =
=
Ehalo'
-1e
-->
=.
=
-->
.
=
The Functional Central Limit Theorem
414
Since S is both open and closed, it is clearly regular. If a set is regular, 44 is closed, f'sf is open, O c Ac (7k and CcE OcE: so is its complement, since If show that the class of regular sets is also closed under we can = Ov Q. count-able unions, we will have shown that every Borel set is regular, which is the required result. Let A1, Aa,... be regular sets, and define A U';=1An. Fixing e > 0, let Onv and Cnv be open and closed sets respectively, satisfying -4
-
-
=
(26.3)
Cne i Za I Onc
and ptton:
CnE)
-
< E/2
n+1
((2,64)
.
.
Let Ov = Uxn-lone, which is open, and A c O. Also let Q = where the k Un=I = necessarily closed, but Cek latter set is not c n: where k is tinite is since k and continuity d of Cv A the c) c :, measure implies that k closed, c ; an k < s 11 For such a k, can be chosen large enough that g(Q Cs) UDaxlck,
'1-
-
.
41(0:-C k
S
p,(O:
c e) +
-
gc:
-
Ck) s
X
S Xp,(One
-
Cn
+
a=1
< e. p,(Q c'ek)
(26.5)
-
It follows that A is regular, and this completes the proof. w Often the theory of random variables has a straightforward generalization to the case of random elements. Consider the properties of mappings, for example. If (S,#) and (T,p) are metric spaces with Borel fields and 5, and fl S F--y 'T is a function, there is a natural extension of 3.32(i), as follows. ./
26.2 Theoem
If
f
is continuous, it is Borel-measurable.
Proof Direct froms 5.19 and 3.22, and the fact that of S and 1' respectively. w
./
and 5 contain the open sets
Let ((S,#),y) and ((T,p),J') be two measurable spaces, and let : S F-> -I' define a measurablemapping, such thatA e 5 implies that -1(A) e then each measure g, S has the property that g, -1 defued by ./;
on
,
g,/l-1(A)
g,(-1(A))
=
,
A e 5,
(26.6)
is a measure on ((T,p),T). This is just an application of 3.21, which does not use topological properties of the spaces and deals solely with the set mappings involved. However, tlte theory also presents some novel difticulties. A fundamental one concerns measurability. It is not always possible to assign probabilities to the Borel sts of a metric space not, at least, without violating the axiom of choice. -
26.3 Example Consider the space (Dp,1j,#u), the case of 5.27 with a 0 and b = 1. Recall that each of the random elements f: specified by (5.43)are at a =
mlltllnl
fliqtnnce
of
1 f'rom one afiother. Hence, the spheres B(fn, r1)are a1l
Convergence in Metric Spaces
WWJ:
disjoint, and any union of them is an open set (5.4).This means that the Borel field Om,1j on (Dm,1j,Jg) contains all of these sets. Suppose we attempt to construct a probability space on ((Dp,1J,Jg), Dm,1j) which assigns a uniform distribution to the A, such that g,((A:a < 0 f &1) = b a for 0 f a < b f 1. Superficially this appears to be a perfectly reasonable project. The problem is formally identical to that of constructing the unifonn distribution on g0,1j.But there is one crucial difference:here, sets of Je functions corresponding to every subset of the interval are elements of Ogo,1J. We know that there are subsets of (0, 11that are not Lebesgue-measurable unless the axiom of choice is violated; see 3.17. Hence, there is no consistent way of constructing the probability space Where ((Dg0,l1,J&),Og0,1),g,), p, assigns the unifonu measure to sets of h elements. This is merely a simple case, but any other scheme for assigning probabilities to these events would founder in a similar way. u -
There is no reason why we should not assign probabilities consistently to smaller c-fields which exclude such odd cases, and in the case of (Dm,1j,Jg) the socalled projection c-/efl will serve this purpose (see j28. 1 below for details). The point is that with spaces like this we have to move beyond the familiar intuitions of the random variable case to avoid contradictions. The space (Dg(),I1,Jg)is of course nonseparable, and nonseparability is the source of the diftkulty encountered in the last example. The characteristic of a separable metl'ic space which matters most in the present theory is the following.
26.4 Theorem In a separable metric space, there exists a countable collection V of open spheres, such that c(F) is the Borel field. where x Proof This is direct from 5.6, V being any collection of spheres Sx, ranges over a countable dense subset of S and r over the positive rationals. .
The possible failure of the extension of a p.m. to (S,,/) is avoided when there is a countable set which functionsas a detenuining class for the space. Measurability difficulties on R were avoided in Chapter 3 by sticking to the Borel sets (which are generated from countable collections of intervals, you may recall) and this dictum extends to other metlic spaces so long as they are separable. Another situation where separability is a useful property is the construction of productspaces.ln j3.4 someaspects of measures onproductspaces were discussed, but we can now extend the theory in the light of the additional structure contributed by the product topology. Let (S,,/) and (7,T) be a pair of measurable topological spaces, with and 5 the respective Borel fields. If S denotes the set of open rectangles of S x7, and (&5 = c(S), we have the following result. ,8'
,/
26.5 Theorem lf S and 1' are separable spaces, with the product topology.
,/
@ 5 is the Borel field of S x7
Proof Under the product topology, R is a base for the open sets (seej6.5). Since S x'r is separable by 6.16, any open set of S x'r can be generated as a countable union of S-sets. It follows that any c-field containing S also contains the open @ 5 contains the Borel field. Since the jets sets of S x7, nd in pmicular, ./
Central Limit Theorem
The Functional
416
of S are open, it is also true that any c-field containing the open sets of S x'T @ 5. . also contains R, and it follows likewise that the Borel field contains ,/
If either S or 'T are nonseparable, the last result does not generally hold. A counter-example is easily exhibited.
26.6 Example Consider the space (Dg(),1jx D(o,1j, pg), where pg is the max metric Let E denote the detined by (6.13)with du for each of the component union of the open balls #((ab,ye), ) over 0 G (0,11,where xe and ye are functions of the form Je in (5.43).In this metric the sets #((ab,ye), ) are mutually disjoint rectangles, of which E is the uncountable union; if R denotes the open rectangles of (Dgo,1jx Dm,1j, pg), E e c(S), even though E is in the Borel field 'metrics.
Of
D(o,1J X Dg0,l!, being an open set.
I:a
The importance of this last result is shown by the following case. Given a probability space L1,5,P4, let x and y be random elements of derived probability spaces ((S,#),./,g,x) and ((S,#),,/,g,y). Implicitly, the pair (x,y)can always be thought of as a random element of a product space of which the g.xand gy are the marginal measures. Since x and y are points in the same metric space, for given ) e f' a is a well-defined non-negative real number. The question of distance J@(),y()) whether interest is d is also a measurable function on (f1,F). This we can obvious answer as follows.
26.7 Theorem lf (S,#) is a separable space, #@,y) is a random variable. Proof The inverse image of a rectangle @,y): D F- S XS
A x B under the mapping
lies in T, being the intersection of the T-sets x -1(4) and y-1(#). The therefore 51$ @)'-measurable by 3.22. But under separability, #?(& field of GXS according to 26.5. Hence (x,y)()) (x((l)),y()))is measurable random element of S xS. If the spaces XS is endowed with topology, the function ,9'
=
d: S XS
9->
mapping is is the Borel a F/Borelthe product
R+
is continuous by construction, and this mapping is also Borel-measurable. composite mapping
The
x,ylod: f :-+ R is therefore Wf-measurable, and the theorem follows. w
26k2 Measures and Expectations As well as taking care to avoid measurability problems, we must learn to do without various analytical tools which proved fundamental in the study of random variables, in particular the c.d.f. and ch.f. as representations of the distribution. These handy constructions are available only for r.v.s. However, if Us is S 9-> R, the expectthe set of bounded, uniformly continuous real functions .f:
ations
Nctz/c Convergence in Metric Spaces Efq
=
417
f W jsfdbk'
(26.7)
e
are always well defined. (From now on, the domain of integration will be understood to be S unless otherwise specitied.) The theory makes use of this family of expectations to fingerprint a distribution uniquely, a device that works regardless of the nature of the underlying space. While there is no single all-purpose function that will do this job, like eikx in the case X e R, the expectations in (26.7)play a role in this theory analogous to that of the ch.f. in the earlier theol'y. As a preliminary, we give here a pair of lemmas which establish the unique represention of a measure on (S,.T) in terms of expectations of real functions on S. The t'irst establishes the uniqueness of the representation by integrals.
26.8 Lemma If g, and v are measures on ((S,t),,$) (,$ the Borel tield), and jfdp a11 f e Us (26.8) =
then g,
=
jfdv,
,
v.
Proof We show that Us contains an element for which (26.8)directly yields the conclusion. Let B e be closed, and detine Sn (x:dx,B) < 1/rl) Think of Bn as BnC x, B with an open halo of width 1/n. Bn 1 B as n B and are closed and mutuand disjoint, infxssj,yesltx,y) for each Let 1/n ally n. geLveG Us be a BnC and 1 for x e B (see6.13). separating function such that x,s@) 0 for x e Then ,/
=
.
-->
=
1(#) f
Jsulj,stfv
jgBL,Bd%'
jgli,Bdb =
f
=
V(#n),
(26.9)
oo, we have p,(#) < where the last inequality is because gBL,px)f 1. Letting n v(A). But jt and v can be interchanged, so g,(#) v(#). This holds for all closed sets, which form a determining class for the space, so the theorem follows. . ...-
=
Since Us i Cs, the set of all continuous functions on S, this result remains true if we substitute Cs for Us ; the point is that Us is the smallest class of general functions for which it holds, by virtue of the fact that it contains the required separating function for each closed set. The second result, although intuitively very plausible, is considerably deeper. Given a p.m. g, on a space S, detine A(.f) fdbfo f e &s. We know that A is a functional on Us with the following properties: =
fx) flx) AtJ
+
bht
=
=
0, al1 x e S
A(.f) k 0
(26.10)
1, all x e S
A(.f)
(26.11)
JA()
+
:A(.f2),
,
h
E
=
1 U, a,b e R,
(26.12)
where(26.11)holds since dbk 1, and (26.12)is the linearity property of integrals.The followinglemma states that on compact spaces the imptication also =
The Functional
418
Central Limit Theorem
runs the other way. 26.9 Lemma Let S be a compact metric space, and let A(/): Us R define a funcThere exists a unique p.m. g on (S,#) satisfytional satisfying (26.10)Y26.12). ing j'fdp A(J), each J e &s. E1 -->
=
In other words, functionals A and measures g, are uniqpely paired. At a later stage we use this result to establish the existence of a measure (the limit of a sequence) by exhibiting the corresponding A functional. We shall not attempt to give a proof of this result here; see Parthasarathy (1967: ch. 2.5) for the details. Note that because S is compact, Us and C's coincide here; see 5.21.
26.3
sveakConvergence
Consider IM,the set of a11 probability measures on ((S,t),#7). As a matter of fact, we can extend our results to cover the set of a11finite measures, and there are a couple of cases in the sequel where we shall want to apply the results of this # 1. However, the modifications required for the chapter to measures g, where Jdbk extension are trivial. lt is helpful in the proofs to have an agreed normalization, and dbt 1 is as good as any, so 1et EMbe the p.m.s, while keeping the possibility of generalization in mind. Weakconvergence concerns the properties of sequences in IX,and itis mathelnatica11y convenient to approach this problem by treating EMas a topological space. The natural means of doing this is to define a collection of real-valued functions on N, and adopt the weak topology that they induce. And in view of (26.7),a natural class to consider are the integrals of bounded, continuous real-valued functions with respect to the elements of >. For a point g, e M, define the base sets =
J'k(k,,...,.J,s)
=
v: v e N,
a
fidv hdp -
I
< E,
i
=
1,...,k
(26. 13)
,
where e f/s for each i, and 6: > 0. By ranging over a1l the possible h,...,h and % for each k e N, (26.13)defines a collection of open neighbourhoods of g,. The base collection V(k,J1,...,/k,E), g. 6 M, defines the weak topology on M. .
The idea is that two measures are close to one another when the expectations of various elements of Us are close to one another. The more functions this applies to, and the closer they are, the closer are the measures. Tlzis is not the consequence of some more fundamental notion of closeness, but is the defining propel'ty itself. This simple yet remarkable application illustrates the power of the topological ideas developed in Chapter 6. The weak topology is the basic trick which allows distributions on general metric spaces to be handled by a single theory. Given a concept of closeness, we have immediately a companion concept of convergence. A sequence of measures fgz, n e INl is said to converge in the weak topology, or converge weakly, to a limit g, written gz = g,, if, for every neighbourhood Ft, 3 N such that g,ns :% for a11n N. lf xn is a random element from a probability space (S,,/.g,n), and g,n= g,, we shall say that xn converges in distriEssenlultion to x nnd write au -P-> x. where x is a random element from (S, ,$
,g).
WretzkConvergence in Metric Spaces
419
noted in j22.1 apply in the use of this terminology. The following theorem shows that there are several ways to characterize weak convergence. tially, the same caveats
26.10 Theorem The following conditions are equivalent to one another: (a) g,n = g,. fOr (b) Il'dbk Vdbk every f e &s. -(c) limsupnpwto f g(C) for every closed set C e 'Y'. (d) liminfsgwtf) g,(#) for every open set B e 0. (:1 for which g,(:A) (e) limngwtA) g(A) for every A e ,f.
./
=
=
The equivalence of (a) and (b), and of (a) and (e), were proved for the case of measures on the line as 22.8 and 22.1 respectively', in that case weak convergence was identified with the convergence of the sequence of c.d.f.s, but this characterization has no counterpm here. A noteworthy consequence of the theorem is the fact that the sets (26.13) are not the only way to generate the topology of weak convergence. The alternative corresponding to part (e) of the theorem, for example, is the system of neighbourhoods, J'%'tk,A1
where Ai e
j
v: v e M, v(Aj) g,tAjl
.,z4k,:)
=
,..
,$,
i
=
1,...,/c and gtAjl
-
j
< E,
i
=
1,...,1
,
(26. 14)
0.
=
Proof of 26.10 This theorem is proved by showing the circular set of implications, (a) = (b) = (c) = (c),(d)= (e) (.a).The first is by definition. To show that (b) = (c), we can use the device of 26.8., let B be any closed set in and put Bm BmC are closed and infxssjywesttmy) k lm. = (.x:dx,B4 < 1/pzJ,so that B and Letting ge,e e Us be the separating function defined above (26.9),we have =:::>
,/,
o
p.(#) limsup
limsup
n-'x
n--oo
o
jgevedbkn j gBvedbk
=
=
(26.15) Jsmpsjk,stjt <
g,(#,,),
oo. where the trst equality is by (b). (c) now follows on letting m (c) = (d) is immediate since every closed set is the complement of an open set relative to S, and g,(S) 1. .--
=
To show (c)and (d) = X -AO. closed, and :A =
(e):for any A e From (c),
jtntAl limsup n->x
/ limsup
AO
,/,
W,where
IA
gwtz4-)< p,(4-)
=
AO
is open and
Wis
jt(4),
(26.16)
jt(4),
(26.17)
N--)=
and from (d), liminf gutAl
gw(AO)
k
g(4O)
=
N'->x
n'-x
hence limrlg.tdl
liminf
=
g,(A).
The one relatively tricky step is to show (e) = (what is easily verified to be) a measure, gJ,on
e Us, and define the real line (R,O) by
(a). Let f
The Functional Central Limit Theorem
420
kt1B)
=
g,((x: fx) c #)), B e S.
(26.18)
f is bounded, so there exists an interval (a,b) such that a
!7,all l S. number of atoms. Also, Recall that a distribution on @,f) has at most a countable subintervals interval be divided finite collection of disjoint into finite can a a of width not exceeding E, for any E > 0. Therefore it is possible to choose m < tm b, such that tj /j-1 < :, and points tj, with a to < fl < = 0, for each j. Use these to construct a simple nv. =
gmx) where Aj -
jdbt J
lf
f
S 2:
Since g,I:Aj)
(e),
=
-
fx)
<
.x
/((/j))
-
(26.19)
fy-l 1A/x),
=
jzc1
(a7:#-1 S flx) f 4j),and
=
jfdpn
=
...
<
8mIJptn
note that supxl j'x)
J
IT
+
8mlJ/
-
+
-
gmx)
I < e. Thus,
hmdJ8mdb -
If./-l j 1gwtz4ylg,(A./I 77 j=1
+
(26.20)
-
.
0 by the choice of j, so that limngwt/jl
limsup n--co
=
for each j by
jttyjl,
jfdvn jfdp
(26.21)
S 2:.
-
Since e:can be chosen arbitrarily small,
(a) follows
and the proof is complete.
.
,/
ft.t which satisfy class for (S,#) is a class of sets A convergence-determining ft.t the following condition: if g,ntdl g,(A) for evel'y A G with jt(:A) 0, then gw nz::> g,.This notion may be helpful for establishing weak convergence in cases where the conditions of 26.10 are difficult to show directly. The following theorem is just such an exhple. --y
=
26.11 Theorem If U is a class of sets which is closed under finite intersections, and such that evel'y open set is a finite or countable union of U-set, then t.lis convergence-determining. Proof We first show that the measures gwconverge for a finite union of U-sets A1,...,A,,. Applying the inclusion-exclusion formula (3.4), 2W-1
m
m /=1UA
X +ptn(c.),
=
(26.22)
1=1
and hence where the sets Ck consist of the Aj and a1l their mutual tntersections tt whenever sign th and indicates that the of the is given in Aj tenu are, are in accordance with (3.4). By hypothesis, therefore, <+'
UX?
tn/=1
M
g, UX7 j=1
.
(26.23)
Wetzk Convergence in Metric Spaces
421
To extend this result to a countable union B U';xlAj, note that continuity of p, =lAj) g,(#) as m cx), so for any s > 0 a finite m may be chosen impliesg,IUT gIU'?J =1A;) < enough that E. Then g,(#) large =
'1-
-->
-
liminf gz(#) k liminf p,n UAy n--yx
g, UXy
=
j=3
a--yx
yt(#)
>
-
(26.24)
E.
j=1
Since s is arbitrary and (26.24)holds for any open B e ,Yby hypothesis on U., condition (d) of 26.10 is satisfied. w A convergence-determining class must also be a determining class for the space (see j3.2). But caution is necessal'y since the converse does not hold, as the following counter-example given by Billingsley (1968)shows.
26.12 Example Consider the family of p.m.s fjt,y) on the half-open unit interval g0,1) with gw assigning unit measure to the singleton set ( 1 3In ) That is, pwtt 1 1/n1) 1. Evidently, tg.) does not have a weak limit. The collection C of half-open intervals La,b)for 0 < a < b < 1 generate the Borel tield of (0,1), and pz(gJ,:)) 0 for every fixed a > 0 and b < 1, so are a detennining class. But 0 for a1l a > and the p.m. g, for which g,tf01) 1 has the property that g,((J,:)) valid write 0. It is therefore to -
.
=
-
-..)
=
pztAl
--
=
g,(A), all A e C,
(26.25)
u,;zy even though gw g, in this case, so C is not convergence-determining.
n
'rhe last topic we need to consider in this section is the preservation of weak convergenceundermappings fromonemetric spacetoanother. Sincem = jtmeans JfJw
that
-->
for any f e Ifdbk
Jf((x))#w(x)
&s, it is clear, since foh e Us when h is continuous, J.f(/l@))Jg,(x). Writing y for hx), we have the result
--.y
J/'@)#w-'(.T)
jiybdbth-'.
-->
(26.26)
So much is direct, and relatively trivial. But what we can also show, and is often continuous have the same much more useful, is that mappings that are This is the continuous mapping theorem property. proper, the desired generalization of 22.11. talmost'
26.13 Continuous mapping theorem Let hl S F-> T'be a measurable function, and let D g S be the set of discontinuity points of h. If gz = g, and g,(D,) zt 0, then -1 bnh = g, -1
.
Proof Let C be a closed subset of F. Recalling that (A)- denotes the closure of A,
w-lto limsup
=
limsupgzt-ltc))
n--joo
s
n--/oo
< p,((-1(c))-)
S
p,(-'(c))+p,(o,)
<
limsupgwtt-ltol-) n-/oo
p,(-'(cD -
llht
g.(-1(c))
-
gz-lto,
(26.27)
The Functional Central Limit Theorem
422
noting for the third inequality that ( -1 (C)) i /j-1(C) kp p j e. a closure point of -1(&')is either in /1-1(C7 or is not a continuity point of The second inequality is by 26.10(c), and the conclusion follows similarly. w -
.
,
,
,
.
.
26.4 Metrizing the Space of Measures We can now outline the strategy for determining the wek limit of a sequence of measures tp.,,lon (S,#). The problem falls into two parts. One of these is to detennine the limits of the sequences (gw(A) l for each A e C, where C is a determining class for the space. This part of the programme is specific to the particu1ar space under consideration. The other pal't, which is quite general, is to verify conditions under which the sequence of measures as a whole has a weak limit. Without this reassurance, the convergence of measures of elements of C is not generally sufficient to ensure that the extensions to also converge. lt is this second aspect of the problem that we focus on here. lt is sufticient if every sequence of measures on the space is shown to have a cluster point. lf a subsequence converges to a limit, this must agree with the unique ordinal'y limit we have (by assumption) established for the determining class. Our goal is achieved by finding conditions under which the relevant topolgical space of measures is sequentially compact (ee j6.2). This is similar to what Billingsley (1968)calls compactness, and the required results can be derived in his framework. However, we shall follow Prokhorov (1956)and Parthasarathy (1967)in making l a metric space which will under appropriate circumstances be compact. The following theorem shows that this project is feasible; the basic idea is an application of the embedding theorem (6.20/6.22). '91
trelative'
26.14 Theorem (Parthasarathy 1967: th. H.6.2) If and only if (S,#) is separable, IX can be metrized as a separable space and embedded in g0,11*. Proof Assume (S,#) is separable. The tirst task is to show that Us is also separable. According to 6.22, S can be metrized as a totally bounded space (S,#') where J' is equivalent to d. Let V denote the completion of S under d' (includingthe limits of all Cauchy sequences on S) and then V is a compact space (5.12).The metric spaceof continuous functions t is accordingly separable undertheunifonn
(5.26(ii)).
Now, every continuous function on a compact set is also uniformly continuous (5.21), so that &s C:. Moreover, the spaces C7
-->
F(g)
=
(J'1#g,,gldbkn...j.
(26.28)
The object is to show that Fembeds EM in R=. Suppose F(g) F(v), so that for all m. Since ('s,) is dense in &s, f e Us implies that Igmdv =
Ijfdbk I J
mdp
=
-/,wg,
s I/'
-
gmIJg,
s
Js(.f,>)
<
e
(26.29)
Gctz/c Convergence in Metric Spaces
423
for some m, and every e:> 0. (The second inequality is because dbk 1, note.) The If6lM for al1 E same inequalities hold for v, and hence we may say that j'fdp f &s. It follows by 26.8 that jt v, so F is 1-1. Continuity of F follows from the equivalence of (a) and (b) in 26.10. To show that F -1 is con tinuous, let (gwlbe a sequence of measures and assume F(gn) =
=
=
--
T(g). For
f
G
i/s and any m k 1,
jfdbk J/'#w
J(J
=
-
gmtdpn
-
S lduf,gm)
+
jgm hmdbkn hmdb (26.30) jgmdp jgmdbkn +
-
Ddbk+
-
-
.
Since the second term of the majorant side converges
limsupn
jjfdvn -
JNgj
< lduf,gml
<
to zero by assumption, (26.31)
2:
0, by the right-hand inequality of (26.29). Hence fdvn J/WP,I 0, and w = p. by 26.10(b). We have therefore shown that IMis homeomorphic with the set F@) c R*, and R'='is homeomorphic to (0,11=as noted in 5.22. The distance Jx between the images of points of Munder Fdefines a metric on Mwhich induces the weak topology. The space F@) with the product topology is separable (see6.16), so applying 6.9(i) to F -1 yi e lds the result that EMis separable. This completes the suftkiency part of the proof. The necessity part requires a lemma, which will be needed again later on. Letpx 1 and e M be the degenerate p.m. with unit mass at x, that is, #x((x)) #x(S (x1) 0, and so 1et D (#x:x e S ) i lX.
for some m, and
limnl
6: >
=
-
=
-
=
=
26.15 Lemma The topological spaces S and D are homeomophic. Proof The mapping p: S F-> D taking points x e S to points px e D is clearly 1-1, implies fxnl onto. For f e Cs, fdpx fx), and xn fx) and hence pxn = px by 26.10, establishing continuity of p. Conversely, suppose xn z'r- x. There is then an ope set A containing x, such that for every N E EN,xn e S -.A for some n k N. Let f be a separating fupction such that fx) 0, J@) 1 for y e S -A, 0, so pxn * px. This establishes and 0 S f f 1. Then jfdpxn 1 and -1 of is homeomorphism, continuity as required. w an d p a p .--#
-->
-'r
=
=
=
=
Ifdh
=
,
Proof of 26.14, continued Now suppose (Mis a separable metric space. It can be embedded in a subset of 0,11=, and the subsets of M are homeomorphic to their under the embedding, which are separable sets, and hence are images in (0, themselves separable (again,by 6.16 and 6.9(i)). Since D c M,D is separable and hence S must be separable since it is homeomorphic to D by 26.15. This proves necessity. w 1JX
The last theorem showed that IMis metrizable, but did not exhibit a specific metric on EM.Note that different collections of functions (%) yield different metrics, Miven how Jx is defined. Another aooroach to the oroblem is to construct
The Functional Central Limit Theorem
424
such a metric directly, and one such was proposed by Prokhorov (1956).For a setA The define the open set A 8 f.z':dlxA) < ), that is, 0: gtz48) + k v(A), al1 A e (26.32) -halo'
,/,
=
.
=
,/
Since contains complements and g,(S) v(S) 1, it must be the case, unless g, and g,(A) < v(#) for others. The idea of v, that g(A) k v(A) for some sets A e the Prokhorov distance is to focus on the latter cases, and see how much has to be added to both the sets and their pmmeasures,to reverse all the inequalities. When the measures are close this amount should be small, but you might like to convince yourself that both the adjustments are necessary to get the desired properties. As we show below, L is a metric, nd hence is symmetric in g, and v. The properties are most easily appreciated in the case of measures on the real line, in which case the metric has the representation in terms of the c.d.f.s, =
=
=
./,
f7(F1,F2)
inf ( > 0: F2(x
=
)
-
< F1@) S F2(x+
-
for c.d.f.s F1 and F2. This is also known as Lvy
's
)+
,
e R)
tx
(26.33)
,
metric.
Fig. 26.1 Fig. 26.1 sketches this case, and F1 has been given a discontinuity, so that the form of the bounding functions Fzx.. 8) + and Fzx can be easily ) within discerned. Any c.d.f. lying wholly the region defined by these extremes, of Fz in the L* metric. such as the one shown, is within -
-
26.16 Theorem L is a metric. > 0 Proof fatg,vl f.(v,p,) is not obvious from the definition; but for any C 8 for each y e #, whereas if x e #8 consider B (A ) If x e A then dx,y) #@,y) < for some y e B; or in other words, B8 Ac If fatytv) f then =
=
,
.
,
=
g(A
c
Subtracting both sides of
)+
(5
=
g,(# ) + 8 2 v(#).
(26.34)from
g,(A) f v(# c) +
=
.
,
,
(26.34)
1 gives v(A)+
,
(26.35)
Getzk Convergence in Metric Spaces
425
andhence fvtvrg) f This means by symmetry, for which f.tv,jtl
there is no for which f.(g,,v) > f.(v,g,), nor, > k f.tg,,vl, and equality follows. is immediate that if It f.tg,,vl 0 g, v. To show the converse holds, note that and any n e N. If A is closed, A1/n if f.(p,,v) 0, g,IA1/& ) + $ln k v(A) for A e x. By continuity of p,, g,(A) limn(g,(Al/&)+ 1/n) v(A), and by 1 A as n 1imn(v(A1/N) + 1/n) k g,(A) likewise. It follows that g,(A) v(A) v(A) symmetry, for a1l closed A. Since the closed sets are a detennining class, g, v. Finally, for measures go v, and let ftg,yvl 6 and f.(v,T) n. Then for any A e $, .
=
=
,$,
=
--+
=
=
=
=
'r
gA) f v(A
)+
s
=
=
<(A)n)
+
+n
/
(26.36)
+n,
where the last inequality holds because (A )n
(x: Jtxz)
=
<
n ) c (x: #txzl
+n)J
<
A&>n
=
(26.37)
,
the inclusion being valid since d satisfies the triangle inequality. Hence f 6 + n fxg,,vl + fXv,T). w
f.(p,,T)
=
W can also show that L induces the topology of weak convergence.
26.17 Theorem lf lg,nlis a sequence of measures in N, gw = g, if and only if Z(/%,P') 0. --
,/,
limsupngwtA) Proof To show if', suppose fxgz,p,l 0. For each closed setA e 8 and B(X > + S hence, letting 0, 1 0, limsupngwtA) S g,(A) by ) 6 for every ito consider continuity. Weak convergence follows by (c)of 26.10. To show and fixed 6 the bounded function for A e --
4only
,
,/
A(a) Note that Since
A(x)
=
1 for x e
IA(x) -
=
z1,
max 0, 1 0
%@4I S
<
-
hx) f
1dxA,
dlxA) g
z18
1 for x e dly,Al
-
(26.38)
.
I
c
and
,
fA(x)
dx,y)
:
=
0 for x
e
z18.
(26.39)
,
,$)
is uniformly equicontinuous (seej5.5) independent of A, the family (A, A e and so is a subset of &s. lf pw = g,, then by 26.10(b), Aa
=
sup A .W
jfxdp jhdpn -
(26.40)
0.
--
Hence, n can be chosen large enough that Aa f 6, for any 6
>
larger,
p,n(A) f
JTAYB+An
jhdb f
f
jfdbt'b 8
or, equivplently, fxgz,g,l f 6. lt follows that f,(g.,g) lt is possible to establish the theory of convergence
f B(X$
+
0.
.
-->
8,
0. For this n or
(26.41)
on M by working explicitly
426
The Functional Central Limit Theorem
in the metric space @,A). However, we will follow the approach of Varadarajan (1958), of working in the equivalent space derived in 26.14. The treatment in this section and the following one draws principally on Parthasarathy (1967).The Prokhorov metric has an application in a different context, in j28.5.
The nexttheoremleads IX compact?
on from26.14 by answeringthecrucial
question
-
when is
./
26.18 Theorem (Parthasarathy 1967: th. H.6.4) EMis compact if and only if S is compact. Proof First, 1etS be compact, and recall that in this case Cs Us (5.21.),and Cs is separable (5.26(ii)).For simplicity of notation write just C for Cs, and write 0 for that element of C which takes the value 0 everywhere in S. Let Xcto,1) denote the closed unit sphere around 0 in C, such that supl f l ; 1 for all f e JV0,1),and let (.gpl,m e IN) be a sequence that is dense in #c(0,1). For this sequence of functions, the map Fdetined in (26.28)is a homeomorphism taking (M into F@), a subset of the compact space (-1, 14*. Tlzis follows by the argument used in 26.14. It must be shown that F@) is closed and therefore compact. Let (pw) be a sequence of measures in IMsuch that F(gw) e g-1,11*.What we have -1 to show to prove sufficiency is that y e F@). Since the mapping F onto !Mi s continuous, this would imply (6.9(ii))that (Mitself is compact. Writ hnf) J##w,and note that, since IJfll.kI f sup/l #(f)I S 1, this defines a functional =
--ky
=
An(f):
Xc(0,1)
F.->
(26.42)
(-1,11.
In this notation we have F(pQ (Aa('1), hngzj, and (%) is dense in it, we can choose for every f e e IN) converging to f. Then, as in (26.30),
Since JVO, 1) is compact V0,1)a subsequence fgmk,k
...).
=
Ihnf)
-
An'(f) I S lduf,gmkj
+
Ihnlgm
-
hn'gmk)
I
(26.43)
.
The second term of the majorant side contains a coordinate of F(j%) Ftg,ael and x by assumption. Letting k we obtain, as in converges to 0 as n and n' -
-->
=,
--
(26.31),
lim IAa(/) A,'(J)I -
'
XyN
=
(26.44)
0.
z -->X
This says that (Aa ) is a Cauchy sequence of real functionals on (-1,11, and so must have a limit A; in particular, y (A(#1), A('2),...). It is easy to verify that each An(#), and hence also hf), satisfy conditions (26.10)q26.12) for f e Xc(0,1).Since for every f e C there is a constant c > 0 where 1), we may further say, by (26.12),that hf) such that c.f e iV(0, cA*(.Fc) A*(.) is a functional on C which must also satisfy (26.10)-426.12). From 26.9, there exists a unique p, e M such that A*(f)= fdbkb f 6 C. Hence, we may write y = F(g). lt follows that F(M) contains its limit points, and being also bounded is compact; and since F -1 is a homeomorphism, EMis also compact. This completes the proof of sufficiency. =
=
Fctzk Convergence in Metric Spaces
427
>, the set shown to be homeoTo prove necessity, consider D (px:x e S) morphic to S in 26.15. lf D is compact, then so is S. D is totally bounded when IXis compact, so by 5.12 it suffices to show completeness. Every sequence in D is the image of a sequence txne S ), and can be written as (pxal so suppose pxn = q G N. If xn x G S, then q px e D by 26.15, so it suffices to show that xn i'z+ x is impossible. The possibility that (x,:) has two or more distinct cluster points in S is ruled out by the assumption pxn = q, so ak z,.-l x means that the sequence has no cluster points in S. We assume tltis, and obtain a contradiction. Let E (x1,x2,... lc S A'I be any infinite subset of E. If be the set of the sequence coordinates, and let cluster has points, point the sequence no y G E 1 is isolated, in that every F1 rn 5'(.y,e) (y) is empty for some s > 0. Otherwise, there would have to exist a sequence (y,,e FI l such that yn c S(y,1In) for every n, and y would be a cluster point of txn)contral'y to assumption. A set containing only isolated points is is closed and, by 26.10(c), closed, so =
,
---)
=
=
-
'1
qE'j) k limsup pxnEj)
=
1,
(26.45)
n.Ox
where the equality must obtain since F1 contains xn for some n k N, for every
N. Since q e EM, this has to mean qEj) 1. But clearly we can choose another subset from E. say E1s such that F1 and E1 are disjoint, and the same logic would 1. This is impossible. The contradiction is shown, concluding the give qE1)
N
G
=
=
proof. w
2.5 Tightness and Convergence In j22.5 we met the idea of a tight probability measure, as one whose mass is concentrated on a compact subset of the sample space. Formally, a measure jt on a space (S,./) is said to be tight if, for every 6: > 0, there exists a compact set Kz e such that g,(A-) f s. Letl-l JIX denote any family of measures. The family FI S :. is said to be unkformly tight if suppzsrIg,(AQ Tightness is a property of general measures, although we shall concentrate here on the case of p.m.s. ln the applications below, H typically represents the sequence of p.m.s associated with a stochastic sequence (Ak)T.If a p.m. g, is tight, then of course g,(#J > 1 e for compact Ke. In j22.5 uniform tightness of a sequence of p.m.s on the line was shown to be a necessary condition for weak convergence of the sequence, and here we shall obtain the same result for any metric space that is separable and complete. The first result needed is the ,/
-
following.
26.19 Theorem (Parthasarathy 1967: th. H.3.2) When S is separable and complete, every p.m. on the space is tight. n N Ot ice, this proves the earlier assertion that every measure on (R,f) is tight given that R is a separable, complete space. Another lemma is needed for th proof, and also subsequently.
,
The Functional Central Limit Theorem
428
26.20 Lemma Let S be a complete space, and 1et A-
x
jn
f) U'nj
=
(26.46)
,
n=1 f=1
where Sni is a sphere of radius In in S, Sniis its closure, and jn is a finite integer for each n, Then K is compact. Proof Being covered by a finite collection of the Li for each n, K is totally bounded. If (.zy, j e IN) is a Cauchy sequence in K, completeness of S implies that xj x e S. For each n, since K i UillXr,sinfinitely many of the sequence coordinates must 1ie in Kn Kcj #nkfor some k, 1 S k S jn. Since Lkhas radius 1/?z, taking n to the limit leads to the conclusion that OnA% (xJ, and hence. x e K; K is therefore complete, and the lemma follows by 5.12. w --->
=
=
Proof of 26.19 By separability, a covering of S by l/n-balls Sn Sx, 1ln4, x e S, has a countable subcover, say fsni i e N 1, for each n 1,2,.... Fix n. For any : > 0 there must exist jn large enough that jt(An) 2 1 zlln, where A,, U7J1&j; otherwise we would have g,(U7=1&j)g,(S) < 1 elln, which is a contradiction since g, is a p.m. where Wn Given :, choose 4,, in this manner for each n and let Kz O=n=1Xk Uklxnj, note. Then Ke is compact by 26.20. Further, since f =
=
=
-
=
-
.
=
=
CK1
Kec
()
3
c
z4-
=
n
tj(4-)
=
n
n=1
and noting that
g((Wn)C)
(26.47)
,
n=1
f 1 g,(z4n)S zl2n we have 1 g,(Wn)
=
-
-
X
g,(A-:) <
c
Cr
Xg,((Xk)9 s n=1
or, in other words, g,(Ak) > 1 s. -
:
771/2/
=
(26.48)
:,
n=1
.
Before moving on, note that the promised proof of 12.6 can be obtained as a corollary of 26.19.
26.21 Corollary Let (S,./,g,) be a separable complete probability space. For any E e there is for any : > 0 a compact subset K of E such that p,(F A') < s. ,/,
-
,/
Proof Let the compact set A e satisfy g,(A) > 1 e/2, as is possible by 26.19, and let (A,,L,ga)denote the trace of (S,./,g,) on A. This is a compact space, such that every set in yh is totally bounded. By regularity of the measure (26.1)there exists for any A e JQ an open set A' Q A such that g'atA' A) < e/2. Moving to AC. But the complements, A'c is a closed, and hence compact, set contained in = g,atAf -A'9g(A) < :/2. Ac A'c A' A and gtAf A'C, A)C, 1et A CEra and this argument Now for any set E e and 1et K shows that there is a compact subset K of Erh A (and hence of E4 such that g,((F r'hA) A') < J2. Since gtfr-7 A9 S g,(Y) S &2, g(# A') < E, as required. . -
-
-z4'9
-
=
-
,/
=
-
=
-
WrctzkConvergence in Metric Spaces
429
Lemma 12 6 follows from this result on noting that R is a separable complete .
space.
Theorem 26.19 tells us that on a separable complete space, every measure pwof a sequence is tight. It remains to be established whether the same propel'ty applies to the weak limit of any such sequence. Here the reader should review examples 22.19 and 22.20 to appreciate how this need not be the case. The next theorem is a partial parallel of 22.22, although the latter result goes further in giving suffiient conditions for a weak limit to exist. Here we merely establish the possibility of weak convergence, via an application of theorems 5.10 and 5.11, by showing the link between uniform tightness and compactness.
26.22 Theorem (Parthasarathy 1967: th. /.6.7) Let (S,#) be a separable complete space, and let H c IXbe a family of p.m.s on (S,#7).H is compact if and only if it is unifonnly tight. g0,11O,
by 6.22. Proof Since (S,#) is separable, it is homeomorphic to a subset of Accordingly, there exists a metric J' equivalent to d such that (S,J') is tively ompact. ln this metric, 1etS be a compact jpace containing S and 1et be but #7,the Borel tield of S, is the Borel field on S. We cannot assume that S e the trace of on S. Define a family of measures H on S such that, for p. e H, g,(A) jttA rnS), g, e 1R,for eachA e Toprove that H is compact, we show that asequence of measures fjtn, n e ENl flpm H has a cluster poiqt in H. Consider the counterpart sequence tg,n,n e (N) in H. Since S is conzpact, H is compact by 26.18, so this sequence has one or more cluster points in H. Let v be such a cluster point. The object is to show that there exists a p.m. jt e H such that v. Tightness of H means that for every integer r there is a comppct set Kr i S and p,(X)) such that g(#)) k 1 1/r, for all g, e H. Being closed in S, Kr e gatlJrf'-'h S) g(&), al1 g.e H. Since Kr is closed we have for some subsequence (nz, k e IN)
rela-
,$
,$,
./
a
A.
a
'k
a.
=
,%'.
*.
,%
zx
P,'N
=
*.
,%
,/
a
=
-
=
v(#)) k limsup gn1(A-r)2 1 1/r,
(26.49)
-
k-+x
by 26.10(c). Since Urfre 9, we have in particular that v(UrA)) 1. Now, suppos we 1et v*(S) denote the outer measureof S in terms of coverings by Xsets.Since = v(Ur#))= 1. Applying 3.10, note that S Urrrc S, we must have v*(S) k v*(Ur&) is v-measurable since the inequality in (3.19)becomes =
v*(# faS) K v*(#),
(26.50)
is the trace of 9 on S, all the sets of which holds for a1l B c i. Since are aCCOF dingly v-measurable and there exists a that H such e p.m. g ; v as of closed exists closed subset S, there such that C C D c l required. For any a % D4 f D4 and limsupkgwto S jttC'') are equivD rn S, the assertions limsuptgw/ alent, and hence, by 26.10, gw = jt. This means that ltn ) has a convergent subsequence, proving sufficiency. Notice that completeness of S is not needed for this part of the proof. ,/
,/
=
,
=
To prove necessitv. assume H is comoact. Lettine tS-;. i
G
INl he
n cnllntnhla
The Functional
430
Central Limit Theorem
coveringof S by l/n-s:heres, and fjn,n G N J any increasing subsequence of integers, define Wn Uklknf.We show tirst that the assumption, H g, e H such that, for > 0, all n, p,(:n) < l (26.51) =
-
,
is true for at least leads to a contradiction, and so has to be false. If element pf there H , is a convergent sequence tp, k e EN J in 11, with (compact) one g,l = jt, such that it holds for all g. (Even if there is only one such element, we g,, al1 k). Fix m. By 26.10, can put g ,426.51)
=
g,(Xk)K limsup g(Xn) f
1
(26.52)
-
.
k-yx
Letting n Putting
.-..-)
x =
which is a contradiction. yields p,(S) f 1 therefore zlln, we may assert -
,
n) P,(W
Letting Ke
=
follows as in
>
1 elln all n, al1 g e H.
(26.53)
-
,
this set is compact by 26.20 (S being complete) and it O';xlWn, (26.48)above that jt(Ak) > 1 :. Since g is an arbitral'y element of -
H, the family is unifonnly tight. w
We conclude this section with a useful result for measures on product spaces. See j7.4 for a discussion of the marginal measures.
26.23 Theorem A p.m' p, on the space (S x F, C) 54 with the product topology is tight iff the marginal p.m.s jtx and jty are tight. .$
.
Proof For a set K c S xF, let Kx J:x(A')denote the projection of K onto S. Since the projection is continuous (see j6.5), Kx is compact if K is compact (5.20). Since =
p,x(L) p,(&x13 =
(26.54)
p,(A3,
tightness of g, implies tightness of jtx. Repeating the argument for gy proves the necessity. For sufficiency we have to show that there exists a compact set K e ,$ (&5, having measure exceeding 1 e. Consider the setr=A xf whered e and g,x(A) > 1 e/2, and B e 5 where y(#) > 1 e/2. Note that .t/7
-
-
-
Kc
(z4x Bc) QJ (A6x #)
=
t.p (Af x Bc4,
(26.55)
where the sets of the union on the right are disjoint. Thus, ,t(A-&)<
g(4Cx#)
= gt4c x5'l = gxtztcl +
+
gtA xBc)
+
jt(S x Bc)
p.y(#C)
< k.
+
2jt(AC
xBC4
(26.56)
lf A and B are compact they are separable in the relative topologies generated from S and (5.7), and hence K is compact by 6.17. w 'r
Fdtzk Convergence in Metric Spaces
431
26.6 Skorokhod's Representation Considering a sequence of random elements, we can now give a generalization of some familiar ideas from the theory of random variables. Recall from 26.7 that separability ensures that the distance functions in the following definitions are F.V.S.
Let
fxnl be
a sequence of random elements and x a given random element of a
separable space (S,./). If
#(xn()),.x()) --- 0 for we say that
e C, with #(C)
=
1,
(26.57)
almost surely to x, and also write xn --F-*->x. Also, if
converges
.x,,
)
Pdxnr
2 E)
0, a11 ir
---
>
(26.58)
0, -F-.2:
x. A.s. we say that xn converges in probability to x, and write xn wlzich in turn is convergence is sufficient for convergence in probability, sufficient for xn -P-> x. A case subsumed in the above definition is where x a with probability 1, a being a fixed element of S. =
We now have the following result generalizing 22.18.
26.24 Theorem Given a probability space L1,5,P) let fxntlll and (yu()) be random sequences on a separable space (S,#). If xn b x and dxnty -EL 0, then --->
yn
-P- x.
,/
be a closed set, and for Proof Let A e a4: 1 A as a closed set for each :, and
1:
)7ntt,)l e
zl
l
?
0 put Ac i 0. Since
6: > 6:
lt,):x,,(t,))
Acl kp
l:
=
(x: #@,A) f
:)
./,
also
e
J(x,,(t0),A'n(t,)))k El,
we have fyn and, letting n
6
X) < Pxn
6
AJ +
Pdxn,y
k
E),
(26.59)
--+
x,
limsup Pyn e Al f limsp g'ntz4sl< g(AJ, n'-lx
(26.60)
n->x
where gwis the measure associated with xn, g the measure associated with x, and the secon d inequality of (26.60)is by hypothesi on tak)and 26.10(c). Since this inequality holds for every E > 0, we have limsup#@u e A) f g(A),
(26.61)
n-co
by continuity
of
the measure. This is sufficient for the result by 26.10.
.
of a sequence of distributions on In j22.2, we showed that the weakconvergence the line implies the a.s. convergence of a sequence of random variables. This is the Skorokhod representation of weak convergence. That result was in fact a special case of the final theorem of this chapter.
26.25 Theorem (Skorokhod 1956: 3. 1) Let fgul be a sequence of measures on the
The Functional Central Limit Theorem
432
separable, complete metric space (S,./). There exists a sequence of Og(),1J/.F measurable functions ak: (0,1)
S
.-A
,/,
where m is Lebesgue measure. such that pw(A) m(f : Ak() e A )) for eachA e If gw= g.,there exists a function x() such that g,(A) mf (0: x() e A )) for each =
=
A e
./,
and #(xn(),x())
0 a.s.g?nl as n
-->
cxk.
--)
Proof This is by construction of the functions For some k e N let (x(,0i Cfmntable G S, 6 IN) dellote a collection of points in S such that, fbr every d(X,xt ?) 1/21*1 separability. exist for every k by for some Such sequences )(11S f Or 1/2:+1 < r k < 1/2: denote a system of spheres in S having the Let SlXt jt(0'(XjO,rk)) 0 fOr every I'. An rk satisfying this condition exists, Property g(:5'(A1;O r)) Of points Since there r such that can be at most a countable number > 0 for one or more ; this fact follows from 7.4. * ) i e EN) covers S, and accordingly the sets For given k, the system (5'(x, ,v().
.<
.
,rk),
,
=
,rk
,
i 1 -
Dik
s-/l
=
,
r k)
-
sxjktrk), ,
i e
j=3
s
form a partition of S. By letting each of the k integers l dently over N, define the countable collection of sets
si
.,ik
j
o!
=
I
,..
j.
cho?I 2 ra
(26.62)
,
5
...
>
ik range indepen-
cjnbtk e y.
...
(26.63)
Each SiL,...,ik is a subset of a sphere of radius rk < 1/2, and p.(:5'jj,,,.,j:) 0. By construction, any pair Siq,...,igand 5f9,...,4 are disjoint unless ik k'.Fixing l we have
=
,...,4-1
K
's%i 1 f-k
(26 64 )
gSij.,...,ik-j ,
:''';
.
l ik.zz
and in particular, Si:
S.
=
(26.65)
'1 1 =
That is to say, for any k the collction f&j,...,j:)forms a partition of S, which gets tiner as k increases. These sets are not a11required to be non-empty. (n) For any n e ENand k e (N, defie a partition of (0,1) into intervals Aj1,...,j:, hi'st i) lies to the left of j,,,.,f: where it is understood that Aft* j iL if ij 1,...,r 1 and ir < ir' for some r, and the lengths of the segments equal for j probabilities vmSij,...,ik4. the We are now ready to define a measurable mapping from g0,11 to S. Choose an element :j1,...,j:from each non-empty SiL,...,ik, and for ) e g0,1qput =
,...,
=
-
k
.Aktt0l
=:
if
iL,...,ik
(l)
n' ,...,:. l
e
.
k+m
(26.66)
k Note that by construction dxn,xn ((t)))S 1/2 for m k 1 and taking k = 1,2,...defines a Cauchy sequence in S which is convergent since S is a complete ,
-kfnxh
lnf/-w/:x lxxr
eoc,oxxmvxtlnn
XA;v.+zx
v
rnh
-
11w%
W'TJ/C
Convergence in Metric Spaces
433
To show that ak((l)) is a random element with distribution defined by gw, it is sufficient to verify that Pw(X)
=
Pxn e X)
l?1(ltt:
=
Ak()
(26.67)
e
such that g.(?A) 0. If we 1et At) denote the union of for, at least, all 4 c 'k' the union of all Si1 all Sij,...,ik i A and A c Ac, itA'*is clear that A'kt .8'
=
-4ktc
',
A
c
,
an d that
=
z'ttk)
i so that 4 'kt 'kt Atk)) that p'utz4
(.x:dx
,
'
.
'
j
in respect of Aks and
(26.67)holds
C t1)
'
a4)s
ck' sincejtntctl)
1/2k)
.
Let
(26.68)
,
p.staz4l 0 as k .-- x, it follows 0 and hence gutz4tll p,n (A). This proves (26.67). => remains It to show that, if gw x a.s.g??1l. Since the length of p,, then xn A('0 equals g.(k%j,...,jk), I 1,, we can conclude that the sequence of intervals AiL,...,ik x. %''? has a limit Pick an interior point (.9 of as n '(21 -
-->
.-+
.
=
-->
-
,
.,-.2/
.vvik
,...,fkl
that xk meets the condition Z4) e Sit,...,ik, by definiY,O ik and tion. Then for N large enough we can be sure that, for n k N, ) e 1 1/2-1 x())) x, I etting k < hence dlxn ((,)) we conclude that dxnl S : for any : > 0 whenever n is large enough. We cannot draw this conclusion for the boundary points of the Aj(,..,,jk,but these are at most countable even as k ---) x, and have Lebesgue measure 0. This completes the proof. . Al
,
.
..
,
ik'.t
and
note
.
,
a-Nt,)ll
.)
.
,
.
.)
--
.
w
The construction of j22.2 is now revealed as a particularly elegant special case, since the mapping from g0,11to S is none other than the inverse of the cad.f. when S R In his 1956 paper, Skorokhod goes on to use this theorem to prove convergence results in spaces such as Dg0,11. We shall not use his approach directly, but this is a useful trick which has a variety of potential applications, just as in the case of R. One of these will be encountered in Chapter 30; =
.
27 Weak Convergence in a Function Space
27.1 Measures on Function Spaces This chapter is mainly about the space of continuous functions on the unit intelwal, but an important preliminary is to consider the space #r0,1j of all real functions on (0,11. We shall tend to write just R for this space, for brevity, when the context is clear. In this chapter and the following ones we also tend to use the symbols x,y etc. to denote f'unctions, and t,s etc. to denote their arguments, instead of f,g and x,y respectively as in previous chapters. This is conventional, and reflects the fact that the objects under consideration are usually to be interpreted as empirical processes in the time domain. Thus,
.x2(0,11 k-- R will be the function which assumes the value x(f) at the point t. ln What follows the element x will typically be stochastic, a measurable mapping from a probability space L1,5,P4. We may legitimately write x2 f
8.-4
#,
assigning x((t)) as the image of the element
,
but also
x: f x g0,1J8-+ R, where x(),/) denotes the value of x at ((t),8.We may also write a#) to denote the ordinate at t Fhere dependence on ) is left implicit. The potential ambiguity should be resolved by the context. Sometimes one writes xt to denote the random ordinate where xr((l)) x(),/), but we avoid this as far as possible, given our use of the subscript notation in the context of a sequence with countable domain. The notion of evaluating the function at a point is formalized as a projection mapping. The coordinate projections are the mappings 7:r: #m,1q R, where p@) =
---)
-1
a = x(/). The projections define cylinder sets in R; for example, the set C:r which functions is the collection of all of R, point through the on pass g0,1J e the plane with coordinates (a,t4.This sort of thing is familiar from j 12.3, and the union or intersection of a collection of k such cylinders with different coordinates is a k-dimensional cylinder', the difference is that the number of coordinates we have to choose fmm here is uncountable. Let (/1,...,/klbe any finite collection of points in g0,1J, and let A,l,...,r,(x)
=
(l,1(.'r),...,l,/x))e
R
k
(J),
(gg
.
j;
denote the k-vector of projections from these coordinates. The sets of the collection
Gdtzk Convergence in a Function J
=
-1 c R (0.13:B ta,1,...,,,(.:)
Space
435
e Bk t 1,..., tk e gO11 k e EN J, ,
,
(27.2)
,
are called the fnite-dimensionalsets of #m,1j. It is easy to verify that R is a t'ield. The projection c-/c/t is defined as T c(X). Fig. 27.1 shows a few of the elements of a rather simple N-set, with k 1, and -1 B an interval (tz,:j of R. The set H = afj((tz,:)) e R cons ists of all those functions that succeed in passing through a hole of width b a in a banier erected at the point /1 of the interval. Similarly, the set of all the functions passing through holes in two such barriers, at /1 and f2, is the image under -1 afj,u of a rectangle in the plane - and so forth. =
=
-
I!
&
!
0
1
!
/1 Fig. 27.1 If the domain of the function had been countable, the projection c-field P would be effectively the same collection as :8*0of 12.3. But since the domain is uncountable, T is strictly smaller than the Borel field on #. The sets of example 26.3 are Borel sets but are not in T, since their elements are restricted at uncountably many points of the interval. As that example showed, the Borel sets of R are not generally measurable', but (#,T) is a measurable space, as we now show. 1,2,3,... the family of finite-dimensional p.m.s g,fI,...,f: on Define for k (Rk Bk), indexed on the collection of all the k-vectors of indices, =
,
l(/1
tj e E0,1) j
,..-,fk)-.
,
This family will be required
=
1,
sk .
-
-
)
-
to satisfy two consistency
1,1,...,,,(A3 =
p1,...,,,,,(FX
R
m-k
properties.
The first is
(27.3)
)
for E e Bk and all m > k > 0. In other words, a k-dimensional distribution can be distribution with m > k, by the usual operation of obtained from an pz-dimensional marginalization. This is simply the generalization to arbitrary collections of coordinates of condition (12.7).The second is lif
where
n/1
...,fk
1,
=
.
h.....nfI:h i: n nermutation
1i//(1),
.,fp(k)
.
.
of
1-1
1zy 4)
'
the intejyerq
-
and
r-.: R k
The Functional Central Limit Theorem
436
transformation which reorders the elements of a k-vector denotes the (measurable) This according to the inverse permutation; that is, 9(.v(l),....vp(o) elements basically that re-ordering the vector transforms the condition means would in the if the indices instead of 1,...,k expect way we were measure =
.v1,...,-q.
/1,...,fk.
The following extends the consistency theorelp, 12.4. ?
27.1 Theorem For any family of tinite-dimensionalp.m.s (w,...,tk)satisfying conditions (27.3)and (27.4),there exists a unique p.m. g, on (#,T), such that g.'If.l,...,Jk -1 fOr eac h finite collection of indices.
gl,...,4
=
Proof Let Fdenote the set of countable sequences of real numbers from g0,1j;that is, e F if T fsj e g0,11, j e IN). Define the projections 7:v: R F-> R= by ':
=
IT@) vnT
@@l),XJ2)....).
=
(27.5)
1,2,... Then by 12.4, which applies g,1,,..,,nfor Tn vT7:-1 v'E thanks to (27.3),there exist p.m.s v on (R= S= ) suc h that n w here n n,,(-y)is the projectior of the first n coordinates of y, for y e R=. Consistency vnT vT,, if sequences I and f have their first n coordinates the requires that -1 Be s F) we may define a p.m. g, on same. Since evidently P c (7:s(#). (#,T) by setting For any
T,
write
=
=
=
,
,
=
S''Q
'r
.
,
,
g,(T:'r-1
(#))
Vs
=
(s)
zy 6) .
for each B e 0=. No extension is necessary here, since the measure is uniquely defined for each element of P. con-esponds lt remains to show that the family fwl,...,rk1 to the finitethere exists dimensional distributions of g,. For any e F such that wj,...,u RN (/1,...,fk) (.1,...,.,,1,for some n large enough. Construct a mapping v: FR k bJr first applying a pennutation p to the indices .l,...,Jn which sets xlpsijj RN for i 1,...,k, and then projecting from to R by suppressing the = xt indices svk+j),...,svn). The consistency properties imply that 'r
,
=
-1
-1
&1,...,4 g,x1,...,:nV Vr,V =
Since
vozrsoav . of =
distribution p,.
=
a,j,...,u
-1
=
Y
T
(V0ln)
projection,
-1
=
IIVOJGOJ%I .
w:,...,u
(27.7)
a finite-dimensional
If we have a scheme for assigning a joint distribution to any t'initecollection of coordinate functions (x(f1),...,x(/k) 1 with rational coordinates, this can be according the extended, theorem, to deine a unique measure on (S,T). These to p.m.s are called the tinite-dimensionaldistributions of the stochastic process x. The sets generated by considering this vector of real r.v.s are elements of R, and hence there is a corollly which exactly parallels 12.5.
27.2 Coroilary R is a determining class for (R,T).
I:n
W'ctzkConvergence in a Function Space
27.2 The Space C Visualize an element of Cg(),1j,the space of continuous real-valued functions on g0,11, as a curve drawn by the pen of a seismograph or similar instrument, as it traverses a sheet of paper of unit width, making arbitrary movements up and down, but never being lifted from the paper. Since g0,1j is a compact set, the elements of C@,lj are actually uniformly continuous. might be of interest to us, imagine To get an idea why distributions on Q(),1j realization of a stochastic sequence (5'/)lT, from a probability observing a ,5,P4, for some finite n. A natural way to study these data is to display space them on a page or a computer screen. We would typically construct a graph of Sj against the integer values of j from 1 to n on the abscissa, the discrete points plot' the kind of thing shown being joined up with ruled lines to produce a in Fig. 27.2. 4time
,
(0)
1
Fig. 27.2
We will then have done rather more thanjust drawn a picture', by connecting the points we have defined a random continuous function, a random drawing (theword here operates in both its senses !) from the space CI1,nj.lt is convenient, and there is obviously no loss of generality, if instead of plotting the points at unit intervals we plot them at intervals of 1/4rl 1)., in other words, let the width of the paper or computer screen be set at unity by choice of units of measurement. Also, relocating the origin at 0, we obtain by this means an element of C%,lj, a member of the subclass of piecewise linear functions, with fonnula -
x(f)
=
i tmjxli -
-
1)/-)
+
(1+ tm
-
ilxilm)
(27.8)
for t e (( 1)/-, ilmj, and i 1,...,-, m n 1. The points xilm) e R for i = 0,...,- are the m + 1 vertices of the f'unction. In effect, we have defined a measurable mapping between points of Rf and elements of C%,lj, and hence a family of distributions on C%,1jderived from L1,5,P), indexed on n. The specific problem to be studied is the distribution of these graphs as n tends to infinity, under particular assumptions about the sequence (5'j1.When (u%)is a sequence of scaled partial sums of independent or asymptotically independent random variables, we shall obtain a useful generalization of the central limit theorem. -
=
=
-
Central Limit Theorem
The Functional
438
C%,lj with the unifonu
As in j5.5, we metrize
l
Ix(/) -y(f)
Jg(x,yl
sup
=
t
metric
(27.9)
.
lmagine tying two pens to a rod, so that moving the rod up and down as it traverses a sheet of paper draws a band of fixed width. The unifonn distance dvfx,y) between two elements of G0.1jis the wldth of the narrowest such band that will contain both curves at a1l points. We will henceforth tend to write C for (Cm,1),#g) when the context is clear. C is a complete space by 5.24, and, since (0,1jis compact, is also separable by 5.26(ii). ln this case an approximating function for any element of C, fully determined by its values at a finite number of points of the interval (compare 5.25) is available in the form of a piecewise linear function. A set l'l,s 1 is called a partition of (0,1). < tm (/1,...,/,,) satisfying 0 tl < fl < slight This is a abuse of language, an abbreviated way of saying that the collectsuch defines ion (/f-1,h) for i a partition into subintervals, say, Aj 1,...,m 1 together with Am (/,,-1,11.The norm =
=
=
...
=
=
=
-
11H,,,11 max =
lS iS m
ln
//-1
-
l
(27.10)
is called the jineness of the partition, and a rehnement of H,,, is any partition ff-l ) of which H,u is a proper subset. We could similarly refer to minlxjxsjt/f the of FI,u. coarseness as -
The following approximation lemma specializes 5.25 with the partition H2a illn, 1,...,2* ) for n k 1 playing the role of the 6-net on the domain, with i ( in this case 6 < llln. =
=
27-3 Theorem Given x e C, letyu with
C be piecewise linear, having ln
E
j
max l.x(2-'') -ya(2-'')
1f i :( ln
There exists n large enough that dvfx,y
I
<
<
e.
1 vertices,
+
(27.11)
e.
1,...,2N. Proof Write dj g2-f( 1), 2-''1, i (lnclusionof both endpoints is lynto + innocuous herep) Applying (27.8) we find that, for t e Ai, yn /'? 2-2/ where 2-&(f lnt. t' i 1), and Noting that (1 Dyn(f'') =
=
-
=
=
-
Ix(f)
-y,,(/)
=
-
I < lal
'-xt')
I + (1
,
=
-
1) lx(/) x(/'')
I
-
-
+ 11x(/') yn(/') 1 + (1 1) lxt' -
and that for n large enough, supmrszk Ix(.) that for such n, duxty
=
max Lfifln
j
sup x(8 fexf
-
-
-
-
x(/)
ynt/)
y,,(f'') I
,
l < r1eby continuity,
j
<
.
w
(27. 12) it follows
(27.13)
W'etzkConvergence in a Function Space x, H2n -- D Note that as n important implication. --)
27.4 Theorem
If x,y e C and
(the dyadic
rationals).
y(f) whenever
.x(/)
=
439
There is the following
t e D, then
.x
=
y.
Let zn be piecewise linear with zn(f) x(/) y(/) for t e H2n. By assumption, such a zn exists for every n e 5. Fix :, and by taking n large enough that maxtt/mza), dvytznl ) < e, as is possible by 27.3, we can conclude by the triangle inequality that dux,y) < :. Since : is arbitrary it follows that Jgtx,yl = 0, and hence x y since du is a metric. w Proof
=
=
=
The continuity of certain elements of #, particularly the limits of sequences of functions, is a crucial feature of several of the limit arguments to follow. An important tool is the modulus of continuity of a function x e R. the monotone function wx: (0,11E-> R+ defined by
wx(8)
Ix(J)
sup
=
x(f)
-
Is-t I
I
(27.14)
.
<
wxhas already been encountered in the more general context of the Arzel-Ascoli theorem in j5.5. It tells us how rapidly x may change over intervals of width 8. Setting 8 1, for example, defines the range of x. But in particular, the fact the that are unifonuly continuous functions implies that, for every x e C, =
.x
wx(6) 1 0 as 8 1 0.
(27.15)
w(x,6) as a function on the domain C. Since For fixed 8, we may think of wx() w@,8) w@,6) w@,6) is continuous on C, and hence a measurable l I S ldvlx,y). of function x. The following is the version of the Arzel-Ascoli theorem relevant to C. =
-
27.5 Theorem A set A c C is relatively compact iff sup Ix(0)l
<
(27.16)
x,
AlA
lim sup wx() -+0
0.
=
(27.17)
:a
ire/t
These conditions togetherimposetotal boundedness and uniform equicontinuity z4. Consider, for some t e g0,1j and k e N,
u 1.-(0,1+
1.-.,,
x-
-(),)
--('-,',)
(27.18)
.
/=1
Equality (27.17)implies that for large enough k, supxsxwxtl/k) (27. 16) and (27.17) together imply that
sup sup t
In other words,
irf
A
on
1x(f) I<
<
x.
Therefore
(27.19)
=.
1l the elements of A must be contained in a band of finite width
nrnllnd fl Thiq thenram
iq therefnre
n
qtrniehtfnrwnrd
cnrnllnr'v
nf
G.AN
00
The Functional
Central Limit Theorem
27.3 Measures on C We now see how 27.1 specializes when we restrict the class of functions under consideration to the members of C. The open spheres of C are sets with the form Sx,r)
for x
G
(y e
=
C: dux,y)
<
(27.20)
rl
C.
Fig. 27.3 Such sets can be visualized as a bundle of continuous graphs, with radius r and the function x at the core, traversing the unit interval for example all the functions lying within the shaded band in Fig. 27.3. We shall write Oc for the Borel field on C, and since hdv) is separable each open set has a countable covering by open spheres and Sc can be thought of as the c-field generated by the open spheres of C. Each open sphere can be represented as a countable union of closed spheres, -
X
sx,r)
=
UAx,r
-
11n4,
(27.21)
n=1
and hence fc is also the c-field generated from the closed spheres.
Now consider the coordinate projeciions on C. Happily we know these to be continuous (see 6.15), and hence the image of an open (closed)finite-dimensional rectangle under the inverse projection mapping is an open (closed)elemenf of P. Letting Rc IHrh C: Se Jf) with R defined in (27.1),and so defining Pc c(>tc), we have the following important property: =
=
27.6 Theorem Oc
=
Pc.
Proof Let Hkx,
and so 1et
=
y e C: max
1
Iy(2- )
-x(
2-%
I<
(x
e Rc,
(27.22)
Convergence in a Function Space
WTJ/C
441
X
s(x,a)
=
'Hkx, 1 -k=
= y e C: sup
Iy(/) x(f) I f -
t G (D
e Pc,
a
(27.23)
where D denotes the dyadic rationals. Note that we cannot rely on the inequality in (27.22)remaining strict in the limit, but we can say by 27.4 that S(x,a)
where
k
R.x,a),
=
is the closure of S. Using
sx,r)
=
(27.24)
(27.21),we
US(x,
obtain
r- $ln).
(27.25)
n=1
It follows that the open spheres of C 1ie in Tc, and so Bc
Tc.
Fig. 27.4 To show Pcz fc consider, for (x e R and tj e (0,11,functions xn e Cdetined by the restriction to (0,1j of the functions on R,
xn(/)
=
a
+
n(n
+
1/n)(/
a
+
nn
+
Lflnlto
+
1/n to), /() 3In S t
+
-
-
1/n /),
/0 S
-
t
<
< /0
to + 1/n
(27.26)
otherwise.
a,
Every element y of the set Sxn,n) e fc has the property y(/o) > a. (This is the shaded region in Fig. 27.4.) Note that X
G(a,fO)
=
ty e
C:
7:ro('y) >
al
=
Now, G(a,/o) is an element of the collection
Rct
e Bc. U5'(ak,/l)
(27.27)
n=1
=
-J
tlr (#), B e
Rcto where for general t we define B 1.
(27.28)
lhe Functional Central Limit Theorem
442
In words, the elements of Rct are the sets of continuous functions x having x(/) e B, for each B e f. ln view of parts (ii) and (iii) of 1.2, and the fact that B can be generated by the collection of open half-lines (a,x), it is easy to see that Rct is the c-field generated from the sets of the form G(a,/) for fixed t and (x 6 R. Moreover, Rc is the c-field generated by Llct, t e (0.11) Since G(a,/) q fc Oc. * for any a and t by (27.27),it follows that Rc Q Bc and hence Pc .
of (27.26)is not an element of C, taking It will be noted that the limit the value (x at a1l points except to, and +x at /(). Of course, (xnl is not a Cauchy sequence. However, the countable union of open spheres in (27.27)is an open set (the inverse projection of the open half line) and omits this point. Pc is the projection c-field on C with respect to arbitrary points of the continuum (0,11,but consider the collection P( fHrh C: H e T'), where T' is the collection of cylinder sets of Sg(),ljhaving rational coordinates as a base. In other words, the sets of T' contain functions whose values x(f) are unrestricted except at rational /. Since elements of C which agree on the rational coordinates agree everywhere by 27.4, .zu(/)
=
PL
=
Tc.
(27.29)
This argument is just an alternative route to the conclusion (from6.22) that C ij homeomorphic to a subset of R*. However, it is not true that P T', because P is generated from the projections of every point of the continuum (0,11,and arbitrary functions can be distinct in spite of agreeing on rational f. Evidently (C,fc) is a measurable space, and according to 27.2 and 27.6, Rc is a determining class for the space. In other words, the finite-dimensional distributions of a space of continuous functions uniquely determine a p.m. on the space. Every pom. on #go,1J must satisfy the consistency conditions, but the elements of C have the special property that x4/1) and x(/a) are close together whenever fl and h are close together, and this puts a further restriction on the class of finite-dimensional distributions which can generate distributions on C. Such distributions must have the property that for any e > 0, ('i 8 > 0 such that =
I/1
-
tl 1 <
=
p,(tx:
Ix(fll -x(/2)
I<
s1)
=
1.
(27.30)
The class of p.m.s in (C,Bc), whose finite-dimensional distributions satisfy this requirement, will be denoted Nc. Note that, thanks to 26.14, we are able to treat Nc as a separable metric space. This fact will be most important below.
27.4 Brownian Motion The original and best-known example of ap.m. on C, whose theory is due to Norbert Wiener (Wiener 1923) is also the one that matters most from our point of view, since in the theory of weak convergence it plays the role of the attractor measure which the Gaussian distribution plays on the line. lt is in fact the natural generalization of that distribution to function spaces.
27.7 Definition Wiener measure
Wcelk Convergence in a Function Space
(a) lP(x(0)
=
0)
(b) <(x(f) < a4
*3
1;
=
1
=
:C2-:7
2 -.j /2/
j&e -x
d,
(; <
j
c
j
.
,
of (0,11, the increments x(/1) x(/o), for evel'y partition (/I,...,/kl x(4-I) x(/2),...,x(4) x(/3) are totally independent. u -
-
-
(a) and (b) of the
definition give the marginal distributions of the coordinate functions, while condition (c) fixes their joint distribution. Any finite collection of process coordinates (x(/j),i 1,...,k1 has the multivariate Gaussminl tj,tjit Hence, No,tj), and Extpxtj) ian distribution, with xtp which requirements x(/2) /1 with of continuity. N(0, the x(/1) l tz I), agrees This f'ull specification of the finite-dimensional disibutions suffices to define a unique measure on (C,fc). This does not amount to proving that such a measure exists, but we shall show this below; see 27.15. F may equally well be defined on the interval (0,:) for any b > 0, including b x, but the cases with b # 1 will not usually concern us here. A random element distributed according to lIJ is called a Wiener process or a Brownian motion process. The latter tenn refers to the use of this p.m. as a mathematical model of the random movements of pollen grains suspended in water resulting from thermal agitation of the water molecules, first observed by the botanistRobertBrown. 27Inprac tice thetermsWienerprocess andBrownianmotion tend to be used synonymously. The symbol F conventionally stands for the p.m., and we also follow convention in using the symbol B to denote a random element f'rom the derived probability space (C,Oc,J!/). ln terms of the underlying probability space (f1,F,#) on which we assume B1 8...y C to be a F/fc-measurable mapping, we have <(A'') #(# e A) for each set E s Sc. The continuous graph of arandorn element of Brownian motion, #4) for (t) e f1, is quite a remarkable object (see Fig. 27.5). It belongs to the lass of geometrical forms namedfractals (Mandelbrot 1983). These are curves possessing the property of selfsimilarity, meaning essentially that their appearance is invariant to scaling operations. It is straightforward to verify from the definition that if B is a Brownian motion so is #*, where t4 = k-3I1(B(.,s+kt) -B(,s)) Parts
=
=
-
-
-
.
-
=
,
=
#*(
(27.31)
for any s l (0,1)and k e (0,1 j. Varying J and k can be thought of as in' on the portion of the process from s to s + k. The key property is the one contained in part (iii) of the definition, that of independent increments. A little thought is required to see what this means. In the definition, the points /1,...,4 may be arbitrarily close together. Considering a pair of points t and t + A, the increment B,t + A) #4,/) is Gausian with variance A, and independent of #(,/). Symmetry of the Gaussian density implies that
-
-
#4:
(#(,/+
A) Bk,lB,tl -
-
#(,f
-
A))
<
0)
=
The Functional Central Limit Theorem
M4
for A f t S 1 A and evety A > 0. This is compatible with continuity, but completely rules out smoothness; in any realization of the process, almost every point of the graph is a corner, and has tangent. This property is also apparent rlo from the definition that #(). Note when we attempt to differentiate -
B,t+h)
-
Bl,tl
(27.32)
- N@, 1/).'
h
The sequence of measures defined by letting h --> 0 in (27.32)is not unifonnly tight, and fails to converge to any limit. To be precise, the probability that the difference quotients in (27.32)fall in any finite interval is zero, another way of saying that the sample path x(/,) is non-differentiable at t, almost surely. A way to think aboutBrownian motion which makes its relation to the problem of weak convergence fairly explicit is as the limit of the sequence of pmial sums of n independent standard Gaussian r.v.s, scaled by n -1n Note tjj a t .
j,.((0) s1/2(s(o,.j/n)
1)/s))
-B,(j-
=
-
x(0,1)
(27.33)
I=1 n-1/2I7 1,...,n and B,jlnq jj((,)).By taking n large enough, we can rational /, and by a.s. continuity of the in this form for any express B,tl write process we may
forj
=
=
B,t)
=
lim n -
1/2
(nlj f=1
n-yx
(f((l))a.s.
(27.34)
for any t e L0,11, where gnfldenotes the integer pal't of nt. Consider the expected sum of the absolute increments contributing to Bt). 1/2 According to 9.8, l(./1 has mean (2/a) and variance 1 2/a, and so by independenceof the increments the r.v. An(f) n-1/2:t''1I 1 j(jj has mean ntjllnlll mtvn) (say)and variance 1 2/1. Applying Chebyshev's inequality, we have that for t > 0, -
=
=
=
-
PAn
>
?n(f,n))
> #(IAn(/) '
4(l
-
-
mtnn) 2/a)
mt,n) 2
I S qmtnnl) (27.35)
@
x Since mt,n) a.s.gm for al1 t > 0. vus means that the On 1/2), An(/) random element #4(t)) is a function of unbounded variation, almost surely. Since limn-/xnt/) is the total distance supposedly travelled by a Brownian particle as it traverses the interval from 0 to t, and this turns out to be intinite for t > 0, Brownian motion cannot be taken as a literal descdption of such things as particles undergoing thenual agitation. Rather, it provides a simple limiting approximation to actual behaviour when the increments are small. anextensivefamily of StandardBrownian motionis merelytheleadingmemberof characteristics. continuous 11 having Gaussian For example, processes on a.s. if we multiply B by a constant c > 0, we obtain what is called a Brownian lpotion with variance Gl. Adding the deterministic function g,f to the process defines a -->
=
r0,
Browninn
mntion
with
drift
11..
Thllq. Yttj
=
aRttj
+
llr
renreqentq
n fnmilv
of
Wetzk Conveqence
in a Function Space
05
o-2It s I). N'(g(/ Processes having independent increments X(/) X(.) More elaborate generalizations of Brownian motion include the following. .),
-
-
-
-
27.8 Example Let X(/) Bt 1+f5 ) for < j)< x. Xis a Brownian motion which has stretching squeezing of the time domain. Like #, it is a.s. subjected and been to continuous with independent Gaussian increments. It can be thought of as the limit of a partial sum process whose increments have trending variance. Suppose (f()) Nni f$) which means the variances are tending to 0 if j) < 0, or to infinity if f'J f1+p ,=1 and i > 0. Then n -1-ps(stnt1j -1
=
-
,
')2
..0
,
n
(nl1 -(1+f5#2y-'q =1
j i ((t)) s((o,/l+p;a.s. --.j
(27.36)
u
27.9 Example Let Xt) 0(f)#(/) where 0: g0,1j F-+ R is any continuous deterministic function, and B is a Brownian motion. For s < t, =
X
-
X(J)
QBt)
=
-Bs)4
(0(/)
+
-
0@))#(J),
(27.37)
which means that the increments of this process, while Gaussian, are not index of a double pendent. It can be thought of as the almost sure limit as n partial sum process, --
(n
i- 1
n-1/2-) Qifnlqi
(0(j/a)0((j 1)/a))7)jy())
+
-
f=1
where
(j
-
-
,
(27.38)
j=1
N(0, 1). u
27.10 Example Letting B denote standard Brownian motion on (0,x), define Xt)
=
e
-f5fs(I5r)
c'y.g9)
for fixed f')> 0. This is a zero-mean Gaussian process, having dependent increments like 27.9. The remarkable feature of this process is that it is stationary, with
Xt)
-
No, 1) for al1 t
0, and
>
Exx)
e
=
This is the Onutein-uhlenbeck
jtzn1intf,yl
-t-sj
=
e
-j'J jt-s
I
.
(gy 44y)) .
process. n
27.11 Example The Brownian bridge is the process B0 s C where Bot)
=
#(/)
-
/#(1),
(0,11.
t e
(27.41)
This is a Brownian motion tied down at both ends, and has EB0t)B0s)) mint/,-) ts. A natural way to think about B0 is as the limit of the partial sums of a mean-deviation process, that is =
-
'
s'(?,(,))n--yx =
where(/(t)) -
lim
#(0,1). u
,,-1/27q
En/'l =1
(j())
-
1 -
n
n
77t./(l /=1
a.s.
(27.42)
The Functional Central Limit Theorem
06
We have asserted the existence of Wiener measure, but we have not so far offered a proof. The consistency theorem (27.1)establishes the existence of a measure on C,Bc4 whose finite-dimensional distributions satisfy conditions (a)-(c)of 27.7, so we might attempt to construct a continuous process having these properties. Conjider
(' t,lk
n-1/2
=
(lfl
+(nf y-')jj((o)
-
gnfjljsj+jt))
(27.43)
,
fzz1
where (j 140,1) and the set ((1,...,(nl are totally independent. For given fa), 1%4.,) is a piecewise linear function of the type sketched in Fig. 27.2, although 0 (the (j represent the vertical distances from one vertex to the with Fn40,) next), and is an element of C. L(/) is: Gaussian with mean 0, and -
=
L
2
Evn)
=
=
s-ltjrjfj
nt
+
/+ n-1((nf1
-
-
nt
.
ggfjlz) +
nt
-
(?1/q)2) (27.44)
t + Knnln,
-1
(say) where 0 < Kln,t4 < 2. Moreover, the Gaussian pair Fn(/) and L(/ + s + n ) -1 - l'n4/ + n ), s > 0 are independent. Extrapolating the same argument to general collections of non-overlapping increments, it becomes clear Fn4/) -P-> N(0,f), and -P-y F, then i' is a stochastic process whose finitemore generally that if l',j dimensional distributions match those of W. Fig. 27.5, which plots the pmial independent random numbers, shows sums of around 8000 (computer-generated) the typical appearance of a realization of the process approaching the limit. ,
1.3
Vhs%hx
10
if
.
N.k)'C&'*/'O'
1 %$?
05 .
0 0 Fig. 27.5 This argument does not show that the measure on (C,Sc) corresponding to F actually is <. There are attributes of the sample paths of the process which are not specified by the finite dimensional distribtions. According to the continuous mapping theorem, Fn-P-> Wwould imply that h(Yn)-P--/l(F) for any a.s. continuous function h. For example, suprl l'n(f) I is such a function, and there are no grounds ohnvlx.fnrq,Innnqine onnelaoroa G-nm tlazcx ov-cy,,mgx,atc thnt KI1n.IF-f/) l Jvtf) I .-p-ysuo/l
.
lpetzk Convergence in a Function Space
M7
However, if we are able to show that the sequence of measures corresponding to l's converges to a unique limit, this can only be W',since the finite-dimensional cylinder sets of C are a determining class for distributions on (C,Sc). This is what we were able to conclude from 27.6, in view of 27.1. This question is taken up in the next section, and the proof of existence will eventually emerge as a corollary to the main weak convergence result in 527.6.
27.5
sveakConvergence
on C
of probability measures in Mc. For example, consider the with a sequence like ( Yn, n e IN) whose elements are distributions defined in (27.43).According to 26.22, the necessal'y and sufficient condition for the family (g,a)to be compact, and hence to possess (by 5.10) a cluster point in Nc, is thay it is unifonnly tight. Theorem 27.5 provides us with the relevant compactness criteria. The message of the following theorem is that the uniform tightness of measures on C is equivalent to boundedness at the origin and continuity arising with sufticiently high probability, in the limit. Since tightness is the concentration of the mass of the distribution in a compact set, this is just a stochastic version of the Arzel-Ascoli theorem. Let
sequence tp,nlbe aassociated
,
is uniformly tight iff there exists 27.12 Theorem (Billingsley 1968: th. 8.2) tg.,,l and all > for a11n 2 N, 0 N c INsuch that, for n such that (a) there exists M < (x,
Afl) K
>n(tx:Ix(0)I> (b) for
each :
>
0, there exists
>n(lx:wx(8)
q
2 El) f
(27.45)
n;
(0,1) such that (27.46)
n. n
Condition (b) is a form of stochastic equicontinuity (comparej21.3). lt is easier to appreciate the connection with the notions of equicontinuity defined in 95.5 if we write it in the fol'm #(w(.L,8) k :) < q, where (.L) is the sequence of stochastic f'unctions on g0,lq having derived measures gw.Asymptotic equicontinuity is sufficient in this application, and the conditions need hold only over n N, for some finite N. Since C is a separable complete space, each individual member of tgwl is tight, and for unifonn tightness it suffices to show that the conditions the tail' hold
.
Proof of 27.12 To prove the necessity, 1et lgzl be unifonnly tight, and for > 0 n choose a compact set K with g,u(A')> 1 q. By 27.5, there exist M < cxo and 8 G (0, 1) such that -
K
fo- any
nk
E >
(x: Ix(0)I f
Ml
f'n
tx:wx()
< irl
0. Applying the De Morgan law,
>n(A5) 2 gntfx:
lA(0)I>
A4l kp
tx:>px()
:l)
(27.47)
Central Limit Theorem
The Functional
08
maxfg,,,tfx: 1x(0)l>
M34, p,atf.x: wa.() k :))
Hence (27.45)and (27.46)hold, for al1 n e 5. Write /t*(.) as shorthand for supnkxgwt.). To prove sufficiency, = 1,2,... the sets Ak
=
(x: wxt8k) <
)
(27.48)
.
consider for k
1//c),
(27.49)
where (6k) is a sequence chosen so that p,*( k) > 1 0/2+1 for 0 > 0. This is possible by condition (b).Also set B (x: 1x(0) l S Ml where M is ehosen so that g.*(#) > 1 0/2, which is possible by condition (a). Then define a closed set K = (O7=1Ak67 #)-, and note that conditions (27.16)and (27.17)hold for the case A = K. Hence by 27.5, K is compact. But .4
-
,
=
,
-
fr
*Kc4 S p,* UAkJlc k=1 G)
;
Xp.*(A)
+
g.*(#c)
1=1
1/2+2+ O/2
< 2eX &1
=
0.
(27.50)
This last inequality is to be read as suprlkxgwtrf) S 0, or equivalently > 1 0. Since 0 is arbitrary, and every individual gw is tight by infp,lrxjtutAM 26.19, in particular for 1 S n < N, it follows that the sequence lg,,,lis uniformly -
tight.
.
The following lemma is a companion to the last result, supplying in conjunction with it a relatively primitive sufficient condition for uniform tightness.
27.13 Lemma (adaptedfrom Billingsley 1968: th 8.3) Suppose that, for some (0,1), sup g.
0 S/K1-
Then
a7: sup l<x
Ix(.)
I
-.7(/)
<
:
e
(27.51)
q.
(27.46)holds.
1 + g1/6j, Proof Fixing 8, consider the partition (,...,fr) of (0,11,for r 2 1. Thus, for zl< 5 < 1 we have r where ti in for i = 1,...,r- 1 and fr and the pmition (, 1), for j < S r1we have r 2 and the partition (8, 28, 1), and so on. The width of these intervals is at most 6. A given interval g/,/'J with I/' /1 S must either 1ie within an interval of the partition, or at most overlap two adjoining intervals', it cannot span three or more. In the event that lx(f') x(f) I E, x must change absolutely by at Ieast r1sin at least one of the intervalts) overlapping g/,f'1,and the probability of the latter event is at least that of the former. In other words, considering all such intervals, =
=
=
=
=
-
-
Wetzk Convergence in a Function Space I(/'4-1,n1
r
sl)
>n(lx:wx()
yn
09
LJx: s,s'sup
lx(s')
f=1
x(ll
-
I k z1:
r
< 7'lg,u x.. 1 i.,..,,
<
l?n)
S
l
.x(.')
sup gfj-j
.b.'e
,fjj
-
lk
.x(J)
,1:
(27.52)
11,
where the third of these inequalities applies (27.51),and the final one follows because rb < 2. . These results provoke a technical query over measurability. In j21.1 we indicated diftkulties with standard measure theory in showing that functions sueh as supruyufu Ix@)xt) I in (27.51), and wz(8) in (27.46), are random variables. However, it is possible to show that sets such as the one in (27.51) are T-analytic, and hence nearly measurable. ln other words, complacency about this issue can be justified. The same qualification can be taken as implicit wherever such sets alise below. -
27.6 The Functional Central Limit Theorem Z7m1 1,...,n, where ( Unil Let S t) 0 and Snj Uni for j
is a zero-mean 2 applications normalized that in the previous 1 As Esnn) stochastic array, so of an'ay notation, in Part V and elsewhere, the leading example is Uni Uilsn, E(XjUi)l. Define an element L of W here ( Ui3 is a zero-mean sequence and Cg(),lj,somewhat as in (27.43)above, as follows'. =
N
=
=
=
.
=
=
F,(/)
=
+
sn,lnrj
nt
= Sngj- + nt Fn(1)
=
r-?1f1-)U,:,(,,f)+1,
-
lunj for Q'
-j
+
-
1)/n f t < g'/n, j
=
1,...,?z;
(27.53) (27.54)
Snn.
This is the type of process sketched in Fig. 27.2. The question of whether the oo is the distribution of F,ypossesses @weak limit as n one we now address. The interpolation tenns in Fa(/) are necessary to generate a continuous function, but from an algebraic point of view they are a nuisance', dropping them, we -)
obtain
Xn
=
Sn,fntj
=
Snj-j for
U
-
1)/n K t <
/n,
j
=
1,...,n,
(27.55) (27.56)
A%(1) Snn. =
lf conditions of the type discussed in Chapters 23 and 24 are imposed on f Uni),
lf for example Ui i.i.d.(0,c2), so that Uni Uilsn where snl nn this is just the Lindeberg-Levy theorem. However, the LindebergLevy theorem yields additional conclusions which are less often remarked', it is easv to verifv that. for each distinct oair /1 e r0,11.
.L(1) -P-> N(0,l 1) as =
n
--+
x.
-
,
,tz
=
The Functional Central Limit Theorem
450
Xn(f2)-Xn(/1) -P- N,
Itz
-
fl
I).
(27.57)
Since non-overlapping partial sums of independent variates are independent, we tind for example that, for any 0 < /1 < tz < f3 f 1, Xnlt .L(/1) and Xnt) Xnh) converge to a pair of independent Gaussian variates with variances h /1 and t? t1, so that their sum Xnt) Xnltj) is asymptotically Gaussian with variance t /l, as required. Under our assumptions, -
-
-
-
-
-
IFn(?)
-.%(/)1
=
nt
-
Ln/1) I&n,(n,1+1 I .T.T-->0,
(27.58)
so that L(/) and Xn(f) have the same asymptotic distribution. Since F,;(0) 0, the finite-dimensional distributions of L converge to those of a Brownian motion process as n ---) x. As noted in j27.4, this is not a sufficient condition for the convergence of the p.m.s of Fn to Wiener measure. But with the aid of 27.12 we can prove that (J%Jis uniformly tight, and hence that the sequence has at least one cluster point in Mc. =
Since a11such points must have the nite-dimensional distributions of W, and the finite-dimensional cylinders are a determining class for (C,Oc), Wrmust be the weak limit of the sequence. This convergence will be expressed either by writing gw = <, or, more commonly in what follows, by F,, -P-4 B. This type of result is called a functional central limit theorem (FCLT), although the term invariance principle is also used. The original FCLT for i.i.d. increments (the generalization of the Lindeberg-Levy theorem) is known as Donsker's theorem (Donsker 1951). Using the results of previous chapters, in particular 24.3, we shall generalize the theorem to the case of a heterogeneously distributed martingale difference, although the basic idea is the same.
27.14 Theorem Let L be defined by (27.53)and (27.54),where ( Uninsnil is a mal4 ingale difference array with varianc array lczujj,and Z')=lc2nj 1. If =
n
Uli (a) 77 f=1
(b)
-'.2:
1,
.-J'-'..y 0, max IUnif
1< i < n
1im n-cn
(n/) 2 nni
=
t, for al1 t e i)0,11,
f=1
then L -P-> B. n Conditions (a) and (b) reproduce the corresponding conditions of 24.3, and their role is to establish the finite-dimensional distributions of the process, via the conventional CLT. Condition (c)'is a global stationarity condition (see j13.2) which has no counterpart in the CLT conditions of Chapter 24. Its effect is to rule out cases such as 24.10 and 24.11. By simple subtraction, the condition is suftkient for
Wkck Convergence in a Function Space
451
gn-l
2 Gni n--jx f=(nfJ+1 lim
s
=
(27.59)
/,
-
Proof of 27.14 Conditions 24.3(a) and 24.3*) are satisfied, on writing Uni for Xnt. ln view of the last remarks, the finite-dimensional distributions of L converge to those of W', and it remains to prove that (Ll is uniformly tight (i.e., that the sequence of p.m.s of the L is unifonnly tight). b+)+1 0.2ni 2 Define, for positive integers k and m with k + m < n, snkm maximal inequality for Esn,kwm Sn l w here Sn,k-vj Snk /+J' r= +1 Uni. The martingales in 15.14 implies that, for > 0, =
.
jzu
=
-
-
,
.
P max
lsnvk-qsnkI >
El
hhkm K
-
snkIP
'n,k-vm -
(27.60)
.
tL-nt,,;)P
ju'm
In particular, set k grl/land m grl8jfor fixed 8 e (0,1)and t e g0,1 8J so Snlsnkm -C-> Z that m increases with n, and then we may say that sn,kwm satisfying N(0,1), by 24.3. For given positive numbers q and :, choose =
=
-
-
-
,
256F1 Zl 3/q:
maxf :/8,
>
and consider the case 6
ir 2/6412<
=
2
1,
(27.6 1)
1 There must exist Nt k 1 for which, with n k is sufficiently close that
No, the Gaussian approximation
FIsn,k..msnkI3
.
-
3 hsnkmt
1
'q:
< :56:2
.t 411,
=
(27.62)
.
2 Also observe from (27.59)that limn-jxskzm 8 For the choice of indicated there k k that for such and < knkm exists Nj 1 hence, combining (27.60)with p n Nj, = 3 with (27.62),such that =
.
/
# max ISn,k..j SnkI k j<j<m -
14:
< 14q6,
(27.63)
for n k maxf No,Nj J. Now,
F,l(-) Fn(/) -
for s
> /,
from
For t e
There
g0,1
nlqn
eviqtt
-
(27.53)and (27.54),where
Rnsn -
(27.64)
+ Rns, 5'n,(,,,1 s'rl,rnrj
=
=
ns
-
(l-1)t&,(n,j+1nt -
1, there
exists s'
IFn@')
Fn(/) I
n
-
lnry-
d'xnfal,
=
G
g/,t + j
tss f;r+ ala
/eox,
such that
IF,,(-)
sup ,q
>
(27.65)
(rlf))Un,g<+1.
-
Ar-'
-
1's(/) I
(27.66)
.
f'G'C.:
fnr
wanx,
c'mmz-la
+
eanz4
o'
Fw-+R
-'-'
The Functional Central Limit Theorem
452 + (n.'J S (Z/JJ rzlland
hence
I5k,En.s-)
1 :f
-$%,(n,1
-
lnax
lFn(-')
-
(27.67)
-
.
li-/s(n,l
(also invoking the triangle
It follows
lSn-kntj-q5',,,,,ryl
inequality) that for n k N1,
+ Rnls'n l I5'?,,(n.,'qsk-lnrj f lnax I5',,,(,,,q+./5k,(,,,)f + lRns'n I
1'n(/)I =
-
(27.68)
-
.
l<./<(ny
By condition (b) of the theorem, #(sups,rl#n(.,/) there exists Ns 1 1 such that, for n 2 N5,
IRnst) l k
P
)f
lk )
0 as n
-->
and hence
--->
x,
14q.
(27.69)
Inequalities (27.69)and (27.63)jointly imply that, for al1 t e N* max fNo,Nj,N1,N? 1,
g0,1 51 and -
n
=
#(lFn(.'l
1'n(/)I k .z1s)
-
S #
Isn,nv..j
,s',,-rnrp
max
-
lsvsnaq S P
I + IRns', l k
max 15'n,(nr1+./s'rl,rnrd I k 14: t.p -
lvgnl f
#
max
l
vb-n,nv-vj
5',,,(n,1 l
-
ls./lnal
l4s
+
t
A1E,
Rns'n
#( lRns'n
l4:)
I2
14:)
l
ln.
(27.70)
u7
The conclusion may be written as
sup #
0
sup lSJ
1L(J)
-
F,,(f) I k ;lE <
n6,n
2 N*.
(27.71)
Note that (27.51)is identical with (27.71)for the case gwtdl PYn q 4), and that q and e are azbitrary. Therefore, uniform tightness of the corresponding sequence of measures follows by 27.12 and 27.13. This completes the proof. . =
We conclude this section with the result promised in j27.4:
27.15 Corollary
Wiener measure exists.
Ia
The existence is actually proved in 27.14, since we derived a unique limiting distribution which satisfied the specifications of 27.7. The points which are conveniently highlighted by a separate statement are that the tightness argument developed to prove 27.14 holds independently of the existence of W'as such, and that the central limit theorem plays no role in the proof of existence.
Wktzk Convergence in a Function Space
453
Proof Consider the process i'n of (27.41).We shall show that, on putting Uni n -1/2l' COn ditions 27.14(a), (b), and (c) are satisfied. It will follow by the reasoning of 27.12 that the associated sequence of measures is uniformly tight, and possesses a limit. This limit has been shown above by direct calculation to have the finite-dimensional distributions specified by 27.7, which will conclude the proof. Condition 27.14(c) holds by construction. Condition 27.12(a)) follows from an application of the weak law of large numbers (e.g.Khinchine's theorem), recalling that since (j is Gaussian ((?,) is an independent sequence possessing a11 its moments. Finally, condition 27.14(b) holds by 23.16 if the collection ((1,...,(z) satisfy the Lindeberg condition, which is obvious given their Gaussianity and =
23.10. x
27.7 The Multivariate Case We would like to extend these results to vector-valued processes, and there is no difficulty in extending the approach of j25.3. Define the space Cgqljzn,which we write as cn' for brevity, as the space of continuous vector functions
x
=
L0,11= F-.> R''', (x1,...,xk)':
where (0,11= and R*' are the Cartesian products of m copies of g0,1qand R respectively. Cm is itself the product of m copies of C. It can be endowed with a metric such as
dx,y)
c.r
max ljm
Lduxjtyjl,
(27.72)
theproduct topology, andcoordinateprojections remain continuous. Cm is also separable by 6.16, and frt fc (8)fc@ ())Oc 11=) rectangles Borel is the of of generated by c-field the tield Cm open (the (0, pz-fold iteration of 26.5. cm,B2!)is therefore a measurable space. by Let
whichinduces
Since C is separable,
J,t2
:zi
=
(7:-,1tkB) c 1
,...,
Cm1 B e Bmk f1,...,/1 e '
(0,1q,k
...
e (NJ
,
(27.73)
denote the finite-dimensionl sets of Cm. Again thanks to the product topology, R is the field generated from the sets in the product of m copies of Rc. 27.16 Theorem R is a determining class for (Cm, fp). Proof An open sphere in B Sx,
=
(ye
is a set
Cm'. dmux,y) < a)
= y e Cm'. max sup Iyj tsjsm t The set
,
-xj
I<
a
.
(27.74)
The Functional
454
Central Limit Theorem
-k
Hk#,al
y e C m. max max Iyjl
=
i) xj -
.
z-kj)j <
(x
,
(27.75)
3% S m 1< j < lk
is an element J'fp. lt follows by the argument of 27.6 that OX
sx,
UOHkx,
=
Pmc
r- 1/n) e
=
n=1 &1
and hence, that Brt c theorem. w
J7T.Since
c(1t'J),
(27.76)
347 is a tield, the result follows by the extension
lt is also straightforward to show that B = T11,by a similar generalization f'rom 27.6, but the above is a1l that is required for the present purpose. A leading example of a measure on (C''',f;) is W'&,the p.m. of m-dimensional standard Brownian motion. A m-vector B distributed according to <'& has as its elements m mutually independent Brownian motions, such that Bt)
140, tlm),
-
(27.77)
where Im is the m x m identity matrix, and the process has independent increments with EBs)
for 0 S t
<
-#(f))(#(.)
-
#(f))')
=
(.: -
lm
(27.78)
s S 1. The following general result can now be proved.
27.17 Theorem Let funi,snij be a m-vector mmingale difference array with variance matrix array (Esj), such that Z:=1Enj Im. Then let =
Fn(f) for j
Sni-j + nt
=
1,...,H, and Fn(1)
=
=
lunj for (j
-j
+
Snn, where Sno
=
-
1)/rl K t
.j/n,
<
(27.79)
0 and
j
r
=
fazl
t/
i,
g-
=
1
,...,/1.
(27.80)
lf n '
Uni Uni
.-.--y''
r
I #l,
f=1
(b)
--->rr
max Un'iuni
0,
lifrl
(c)
lim
grllJ
n-->co
then Fn
--
D
#
7)E i n
=1
=
tlm, for al1 t e
g0,1J,
.
Proof Consider for an v-vector A of unit length the scalar prpcess A'L, having increments A'foj. By definition, fA'Uni,snil is a scalar martingale difference array with variance sequence A'EnfA. lt is easily verified that all the conditions
Gtzk
Convergence in a Function Space
455
In of 27.14 are satisfied, and so 'L -P-> #. This holds for any choice of -P-> N(0,/), with similar conclusions regarding all the finitePmicular, 'L(f) distributions of the process. dimensipnal It follows by the Cramr-Wold theorem that .
Fn(f) '-P-') X(0, tl),
(27.81)
with similar conclusions regarding al1 the tinite-dimensionaldistributions of the process', these are identical to the finite-dimensional distributions of W''''. Since R is a detennining class for Cm,Bm),any weak limit of the p.m.s of ( F,,) can only be Wr'''.It remains to show that these p.m.s are uniformly tight. But this is true provided the marginal p.m.s of the process are uniformly tight, by 26.23.
Picking A to be the ythcolumn of Im for.j 1,...,- and applying the argument of 27.14 shows that this condition holds, and completes the proof. . =
The arguments of j25.3 can be extended to covert 27.17 into provide an unusually powerful limit result. The conditions of the theorem are easily generalized, by replacing 27.17(c) by Enfl
lim XEnj
=
,
n--yx f=1
Im as in where E is an arbitrary variance matrix. Defining L- such that L-SL-' 25.6, 27.17 holds for the transformed vector process Zn L-Yn. The limit of tl:e process Yn itself can then be detennined by applying the continuous mapping theorem. This is a linear combination of independent Brownian motions, the t'initedimensional distributions of which are jointly Gaussian by 11.13. We call it an m-dimensional correlated Brownian motion, having covariance matrix E, and denoted BX). The result is written in the fonu =
=
L
-P-> #(E).
(27.82)
An invariance principle can be used in this way to convert propositions about dependencebetween stochasticprocesses convergingtoBrownianmotionintomore tractable rsults about correlation in large samples. Given an arbitrarily related set of such processes, there always exist linear combinations of the set which are asymptotically independent o f one another.z8
28 Cadlag Functions
28. 1 The Space D The proof of the FCLT in the last chapter was made more complicated by the presence of terms necessary to ensure that the random functions under consideration lay in the space C. Since these terms were shown to be asymptotically negligible, it might reasonably be asked whether they are needed. Why not, in other words, work directly with Xn of (27.55)and (27.56),instead of Fn of (27.53)and (27.54)? Fig. 28.1 shows (apart from the omission of the point aL(1), to be explained below) the graph of the process Xn corresponding to the Fn sketched in Fig. 27.2. Xn as shown does not lie in Cm,lj but it does 1ie in Dp,lj, the space of cadlag functions on the unit interval (see 5.27), of which Cgtlj is a subset. when there is no risk of confusion with Henceforth, we will write D to mean Dgt),1y
other usages.
Fig. 28.1 As shown in 26.3, D is not a separable space under the uniform metric, which means that the convergence theory of Chapter 26 will not apply to D,dv). dv is not the only metric that can be defined on D, and it is worth investigating alternatives because, once the theory can be shown to work on D in the same kind pf way that it does on C, a great simplitkation is achieved. Abandoning dv is not the only way of overcoming measurability problems. Another approach is simply to agree to exclude the pathological cases from the field of events under consideration. This can be achieved by working with the c-field To, the restriction to D of the projection c-field (see j27. 1). In contrast with the case of C, Po c fo (compare27.6) and a1lthe awkward cases such as uncountable discrete subsets are excluded from To, while all the ones likely to arise in our theory (whichexclusively concerns convergence to limit points lying
Cadlag Functions
457
in C4 are included. Studying measures on the space D,dv),Po) is an interesting line of attack, proposed originally by Dudley (1966,1967) and described in detail in the book by Pollard (1984).
While this approach represents a large potential simplification (muchof the present chapter could be dispensed with), an early decision has to be made about which line to adopt; there is little overlap between this theory and the methods pioneered by Skorokhod (1956,1957), Prokhorov (1956),and Billingsley (1968), which involves metrizing D as a separable complete space. Although the technical overheads of the latter approach are greater, it has the advantage that, once the investment is made, the probabilistic environment is familiar; at whatever remove, one is still working in an analogue of Euclidean space for which all sorts of useful topological and metric properties are known to hold. There is scope for debate on the relative merits of the two approaches, but we follow the majority of subsequent authors who take their cue from Billingsley's work. The possibility of metlizing D as a separable space depends crucially on the fact that in D the permitted departures from continuity are of a relatively limitd kind. The only ones possible are jump discontinuities (also called of the first k.ind'l: points t at which Ix(/) x(/-) I > 0. There is no possibility of isolated discontinuity points / at which both 1x(/) x(f-) 1 and Ix(/) x(/+) I are positive, because that would contradict right-continuity. There is however the possibility that x(1) is isolated; it will be necessary to x(1-). This is a little unfortunate, but since discard this point, and 1et x(1) we shall be studying convergence to a limit lying in C%,1j(e.g.,B), it will not change anything material. We adopt the following definition. Kdiscontinuities
-
-
-
=
28.1 Definition tions: (a) x(/+) (b) x(/-) (c) x(/)
Dgqlj is the space of functions satisfying the following condiexists for t e (0,1),. exists for t e (0,11., x(/+), t < 1, and x(1)
=
x(1-).
=
n
The t'irst theorem shows how, under these conditions, the maximum jumps is limited.
28.2 Theorem There exists, for all (/1,...,/r) of g0,11 with the property
e D and every E
.x
1x(/)
I<
-xs)
sup
s,t gf.j-lyffl for each i
:::::
1,
>
number of
0, a tinite partition
(28.1)
:
.,r. ..
This is by showing that tr 1 for a collection (,...,/rl satisfying supt frl, the supremum being taken (28.1), with to 0. For given x and E 1et x(/-) belongs to the exists for all t > 0, over a11 these collections. Since exists /r. that there such that is, r set; Suppose tr < 1, and consider the point /r+ 6 f 1, for some 8 > 0. By definition + 6). By of /r, Ixtr) x(/r-l) I k s. Hence consider the interval Ltr,tr Proof
=
.1
=
=
.1
=
-
Central Limit Theorem
The Functional
458
+ ) xtr) l < e. Hence choice of we can ensure by right continuity that 1.x(/r there exists an (r + ll-fold collection satisfying the conditions of the theorem. fr + 8, and the assertion that tr is contradicted. It We must have k /r+1 follows that tr 1. w -
':
':
=
=
=
This elementary but slightly startling result shows that the number of jump points at which Ix(f) I exceeds any given positive nupber are at most finite. The number of jumps such that Ix(/) I > 1In is finite for every n, and the entire set of discontinuities is a countable union of tinite sets, hence countable. Further, we see that -x(/-)
.x(f-)
-
lx(/) sup f
I<
(28.2)
x,
since for any t e g0,11,x(/) is expressible according to (28.1)as a finite sum of finite increments. The modulus of continuity wx(8) in (27.14)provides a means of discriminating between functions in Cp,1J and functions outside the space. For just the same reasons, it is helpful to have a means of discriminating between cadlag functions and those with arbitrary discontinuities. For 8 G (0,1), let H denote a partition (/1,...,/,) with r K g1/q and minjf ti ) > 8, and then define -1
-
wx'()
=
lx(/)
inf
max
r%
1< i < r s,t e (lj-1,rj)
sup
-
x(.)
I
(28.3)
.
Let's attempt to say this in English! wx'(8) is the smallest value, over all partitions of g0,1q coarser than 8, of the largest change in x within an interval of the partition. This notion differs from, and weakens, that of wx(8), in that wx'(6)can be slnall even if the points ti are jump points such that wx(8) would be large. For < 1z there is always a partition H in which ti fj-l < 26 for some i, so that for any x e D, -
wx'(8)S wx(28) for 8
< 1z. So
obviously, lim-owx't8)
1imwx'(:) -+0
(28.4)
0 for any
=
=
0
.x
e C. On the other hand,
(28.5)
is a property which holds for elements of D, but not for more general functions. 28.3 Theorem If and only if x e D, 3
such that wx'(8) < e, for any 6: > 0.
Proof Sufficiency is immediate from 28.2. Necessity follows from the fact that if x B D there is a point other than 1 at which x is not right-continuou' s; in qther words, a point t at which I.x(/)x(/+) I 6: for some 6: > 0. Choose arbitrary and consider (28.3).If t # ti for any i, then wx'(8) 2 : by definition. But even, if t ti for some i, ti e (n,n+1)and Ix(n) xtt-.) l k e, and again wx'() k -
=
E. >
-
Cadlag Functions
459
28.2 Metrizing D Recall the difficulty presented by the existence of uncountable (o,#g), such as the sets of functions
0, 0 f t
Ab(f)
<
discrete sets in
0
(28.6)
=
1 0 < t < 1, ,
the case of (5.43)with a 0 and b 1. We need a topology in which Ab and are regarded as close when I0 0/ I is small. Skorokhod (1956)devised a metric with this property. Let A denote the collection of al1 homeomorphisms 1: (0,1q F-> (0,11 with :(0) 0 and (1) 1; think of these as the set of increasing graphs connecting the opposite corners of the unit square (seeFig. 28.2). The Skorbkhod J1 metric is detined as .b'
=
=
-
=
=
dslx,y)
=
inf : ). e A
>
0: sujp I14/) /1
Ix(f)
Is
-y(l(f))
f
-
:, suyp
ej.
(28.7)
ln his 1956 paper Skorokhod proposes four metrics, denoted J1, J2, M1, and M2. We shall not be concerned with the others, and will refer to ds as is customary, tthe' Skorokhod metric. as
t Fig. 28.2 It is easy to verify that ds is a metric, if you note that supf I1,(/) t I = I where l-1 e A l = supf Ix(-1(/)) supfIf -1(/)I an d suprla/) if e A. While in the uniform metric two f'unctions are close only if their vertical separation is unifonnly small, the Skorokhod metric also takes into account the possibility that the horizontal separation is small. If is uniformly close to y except that it jumps slightly before or slightly after y, the functions would be considered close as measured by ds, if not by du. Consider ab in (28.6),and another element aw. The unifonn distance beiween these elements is 1, as noted above. To calculate the Skorokhod distance, note that the quantity in braces in (28.7)will be 1 for any for which /,40) y: 0 + Confining consideration to the subclass of A with V0) 0 + 8, choose a case -
-y(l(f))
-y(/)
-
.
,
,
.x
,
.
=
The Functional Central Limit Theorem
460
where I (/) JI < 6 (for example, the graph (t, :,4/)1, obtained by joining the three points (0,0), (0,0+ 6), and (1,1) with straight lines, will fulfil the definition) and hence, -
#XAb,Ab+)
8.
=
This distance approaches zero smoothly as
intuitive idea of
28.4 Theorem
(28.8)
8 10, whigh might conform betterto our
than the uniform meiric in these circumstances.
<proximity'
On C, ds, and du are equivalent metrics.
Proof Obviously dsxny) K duxnyjs since the latter corresponds to the case where l is the identity function in (28.7).On the other hand, for any :,
I-x(/)
dvxny,< sup
l + sup ly(l(r)) t
-y(,(/))
t
-y(?)
I
(28.9)
.
Suppose y is uniformly continuous. For every e:> 0 there must exist 6 > 0 such that, if dy(x,y) < (and hence supl (?) /1 < ), then sup Iy(,(r)) y(/) I < E. ln other words, -
-
dslx,y)
<
6
duxny)
=
<
(28.10)
+ :.
The criteria of (5.5) and (5.6) are therefore satisfied. Uniform continuity is equivalent to continuity on g0,1J, and so the stated inequalities hold for a1l .J' e
c
.
.
The following result explains our interest in the Skorokhod metric.
28.5 Theorem
(D,ds4 is separable.
Proof As usual, this is shown by exhibiting a countable dense subset. The counterpal't in D of the piecewise linear function defined for C is the piecewise constant function (as in Fig. 28. 1) defined as
y
=
yti), t e
ti, /f+1) i
=
0,...,-
-
1,
(28.11)
where the y(/j) are specified real numbers. For some n e N, define the set A,, as the countable collection of the piecewise constant functions of form (28.11),with iI2n for i = 0,...,2* 1, and y(/j) assuming rational values for each i. n Letting A denote the limit of the sequence fAn), is a set of functions taking values At rational a set of points indexed on the dyadic rationals D, and hence is countable by 1.5. According to 28.2, there exists for e D a finite partition (/1,...,& J of such each i, that, for g0,1J, =
-
:4
.v
sup I-x(.l -
J,/ 6
(n-I,rJ
.x(f)
I<
E.
Lety be a piecewise constant function constructed on the same intervals, assuming rational values y1,...,ym where yj differs by no more than 6: from a value assumed Then, dslxny) < 2E. Now, given n 1, choose z e An such that by x on gn,n+1). = Since D is dense in g0,1J,dsy x. 0 as n zj yf when jlln e g/f,/f+l). .-..h
--
46l
Cadlag Functions
Hence, dsjx,zl < Jy(x,y) + dsyvz) is tending to a value not exceeding 2s. Since by taking m large enough : can be made as small as desired, x is a closure point of a4. And since x was arbitrary, we have shown that A is dense in D. w Notice how this argument would fail under the unifonn metric in the cases where x has discontinuities at one or more of the points ti. Then, duy,z) will be small only if the two sets of intervals overlap precisely, such that ti = jlln for some j. lf ff were irrational, this would not occur for any finite m, since jlln is rational. Under these circumstances a7would fail to be a closure point of A. This shows why we need the Skorokhod topology (thatis, the topology induced by the Skorokhod metric) to ensure separability. Working with ds will none the less complicate matters somewhat. For one thing, ds does not generate the Tychonoff topology, and the coordinate projections are not in general continuous mappings. The fact that x and y are close in the Skorokhod metric does not imply that x(/) is close to y(/) for evel'y t, the examples of ab and x(j+ cited above being a case in point. We must therefore find alternative ways of showing that the projections are measurable.
Fig. 28.3 And there is another serious problem: D,ds) is not complete. This is easily seen by considering the sequence of elements lxn) where
xn(f)
1 t e ,
=
(
,
)
+ 1u
(28.12)
0, otherwise
(seeFig. 28.3). The limit of this sequence is a function having an isolated point of discontinuity at and hence is not in D. However, to calculate dsxnvxm) +) r1+ the distance is 1 for any must be chosen so that ,Ql) and ( with vertices at (0,0), (,), ( +j graph other choice. The piecewise-linear +)), and (1,1) fulfils the definition, and satisfies (28.7). lt appears that la and so fxnl is a Cauchy sequence. dsxn,xmt I ,
=
=
,
=
,
j;
,11
-
,
28.3 Billingsley's
Metric
The solution to this problem is to devise a metric that is equivalent to ds (in the sense of generating the same topology, and hence a separable space) but in
The Functional Central Limit Theorem
462
which sequences such as the one in (28.12) are not Cauchy sequences. Ingenious alternatives have been suggested by different authors. The following is due to Billingsley (1968),from which source the results of this section and the next are adapted. from (0,11to (0,11 with (0) 0 Let A be the collection of homeomorphisms and (1) 1, and satisfying =
=
l(f) sup log t.s l#J
l(x)
-
IIII
=
lIII:A F-> R+is afunctional
(28.13)
measuungthemaximumdeviationof
thegradient of from 1, so that in particular 111110 for the case (4 t. The set A is like the one defined for the Skorokhod metric with the added proviso that IIIl be and l -1 must be strictly increasing functions. Then define finite; both Here,
=
=
,
dpx,y) =
inf s
>
0:
lGA
We review the essential
IIlI <
s, su,p lxtfl y(l(f)) -
I<
ej.
(28.14)
properties of #s.
28.6 Theorem Js is a metric. Propf Jstx,yl is noted that
0 iff x
=
111-1 11 =
dpy,x) is also easy once it y is immediate. dsxny) IIII. To show the triangle inequality, note that =
=
1ltk) log -
II111 111211
k sup
+
t#
1(,))(l2(f)
(X1(/) -
k sup ,.&
1og
t
t
t#s
J'.
On setting f'
=
-
-
f
2(.)
..s
,2(5')) -
z
.) -
I(,))(l2(J')
(1(f) 1og -
for arbitrary t' and
log
+
t. x
'
k sup
l2(J)
1(.)
t
, -
2@'))
-
s
,
(28.15)
)
11(/) and s'
=
2(.),
we obtain
+ 111211 k II1ol2Il, 11 1111
wher 1,10:2(/)
,1(L4/)),and lolz z(2(l1(J))) Ix(fl I < sup supIx(fl t =
-y(1(/))
-
t
(28.16)
is clearly an element of A. Since
l+
1y(f) -z(2(/))
sup t
l
(28.17)
by the ordinal'y triangle inequality for points of R, the condition dsxa) Jy@,y) + dsynz) follows from the definition. w Next we explore the relationship between ds and #s, and verify that they are equivalent metrics. lnequalities going in both directions can be derived provided the distances are sufficientlv small. Given functions .x and v for which tlotr'vj =
Cadlag Functions
: < z1,consider e A satisfying < z. Since 1(0) particular,11111 or that Ilog(l(/)//) I < 11111 ,
=
463
the definition of Js for this pair, such that, in 0, there evidently must exist t e (0,11such
,
14/) i; < e t
e-.: < Using the series expansion of similarly, which implies that
eC,
(28. 18)
.
we find ek
-
e-C
1 S 2: for : f r1,and
-
1k
-2:
(28.19) /1 < 2e. And in view of our assumption about or, 11(/) : and hence dsxny) cannot exceed 2:. In other words,
suprlxt/)
-
dsx.yj f whenever delx,y) <
,
l
y((?))
-
ldBxnyj
(28.20)
.
Now consider a function jz e A which is piecewise-linear with vertices at the points of a pmition 1%, as defined above (28.3)for a suitable choice of to be specified. The slope of g is equal to (g(/j) jt(n-1))/(ff fj-1) on the interNotice that, if suprl p,(/) /1 S 2 vals g/j-1,fj),w'here ti /j-1 > -
-
-
-
,
.
p,(n) p,(/s1) -
ti
For
1x1< 12,
-
/j-1
- 1
f
Ij.t(/j)
-
ti I + Ig,tfj-ll 45
maxt
/2 + 1x3 3
.:)
the series expansion logt 1 +
Ilogl 1 + xl I <
n-1I u z,
-
a:
=
1x1 Ix 2Il
-
implies
...
21xI
<
.;t
-
,
(28.21)
.
(28.22)
.
Substituting for in (28.22)the quantity whose absolute value is the linorant S ide of (28.21),we must conclude that, if suprl p,(/) /1 f 82 for 0 < < 14, then .x
-
< 4. 11:11
(28.23)
2 w hich means there exists Now, suppose dsxny) e A satisfying 2 /1 < 2 an d supr Iy t) I< supfl1(/) chooseg, as the piecewise -1g, js 0,...,r. The function ht for i linear functioa with gtnl down' to the diagonal at the points of the partition; that is, it is increasing on the inten-als (/j-1,Q with -1g,(f) e (fj-1,/j)if and only if t e gn-1,/. Therefore, choosing l-l to correspond to the definition of wx'(8), we can say =
,
,
-x((/))
-
,
.
=
1x4/) -x(p,(/))
-x(,
f wx( ' ) + :2
dexty) < since for
... 1
I ; Ix(/)
Putting this together with
ttied
=
g,(/))I +
Ix(y
...
j
g,(/))
-x(p,(
j;;
j
(28.24)
.
(28.23)gives
maxt4,
,
wx'()
+
for 0 2)
<
14 the
8 f
< wx'()
+
inequality
4.
x e D we may make wx'() arbitrarily small by choice of
(28.25) ,
we have
The Functional Central Limit Iheorem
464
Z 4dvx,yl
dBxnyj
1/2
(28.26)
whenever dsx,y) is sufficiently small. We may conclude as follows. 28.7 Theorem
In D, metrics dp and ds are equivalent.
Proof Given e > 0, choose 8 S Then, for q < mint 2 ,
/)
,
t,and also smal,1 enough
dex,yt <
n= dsx.y) < n = by (28.20)and (28.25)respectively, satisfied. x
that wx'(8)+ 48 f :.
dsx,yt
< E,
(28.27)
dex,y)
<
(28.28)
e,
The criteria of
(5.5)and (5.6)are
therefore
Equivalence means that the two metrics induce the same topology on D (the Skorokhod topology). Given a sequence of elements (ak),dexnnx) 0 if and only 0, whenever if dslxnbx) e D. But it does not imply that fak) is a Cauchy whenever (D,#s) it in is a Cauchy sequence in ll,d, because the latter sequence space is incomplete and a sequence may have its limit outside the space. It is 0 only if dsxn,x) 0 and lim-oowx/t8) 0. clear in particular that dexnp For example, the sequence of functions in (28.12)is not a Cauchy sequence in --->
.--)
.x
-->
-->
=
(D,#s). To define dexnpxm) (forn k 3, m k 4) it is necessary to tind the element ,1 and + )) of A for which j + and whose gradient deviates as V1a little as possible from 1. This is obviously the same piecewise-linear function, with vertices at the points (0,0), (;j+ j, j + )) and (1,1), as defined for ds. But the maximum gradient is nlm, corresponding to the segment connecting the mint 1, Iloglnlm) I) which does not second and third vertices. dnxn,xm) approach zero for large n and m (set m ln, for example).
Q1)
=
=
,
Q1,J), =
,
=
28.8 Theorem The space (D,#s) is complete. Proof Let
(yz,k e INl be a Cauchy sequence in D,dp4 satisfying deyk,yk-vj) < k 1/2 imp 1y ing the existence of a sequence of functions lg e AJ with ,
suplyl7tf)
l<
-y1+1(g(f))
l
(28.29)
< 1/2t 11g,11
(28.30)
supf Ip+,u(f) f 1 f lllk'bm for m > 0. Define p m also an element of A for each finite m', the sequence lg,,h,,m ga-og+m-l o...ogs 1,2,... Cauchy is ) a sequence in (C,du) because =
l t f0 llows from
(28.20)that
g,,,,+1(J) sup1 t
-
g,r,,(f) 1
=
-
tp
=
sup s
Ig+,,,+I@)
,l -
< 1/2 :+r,$ .
(agaj) .
limk-oxg,p,. To show Since (C,du) is complete there exists a limit function oo. sufticient But by (28.16), that to show that lllzIl< e A, it is =
Cadlag Functions
'>7
465
m
m
1
lIg,,,,,ll Ilg.kop+lo...op,+,,,lllIg,+./Il:--( k+j <
forany m, so Note that k
<
-0
< l/2-1 IIlkl1
-1
so that
..l og,
=
supIylk-1(/))
k+1(f)) 1-1
-yk+l(
<
t
,t+1
I
=
l
j=o
ol-l
g,k
k
an d hence
Iyk(5')
sup S
,
1
(28.32)
ak-l, by
I<
-yk+l(g,k(.))
=
<
(28.29),
1/2k
(28.33)
.
So considr the sequence (ypo,k -1e D, k e EN) Accor din g to (28.33)this is actuCauchy in Dndul. But the latter space is complete; this is easily ally a sequence shown as a corollary of 5.24, whose proof shows completeness of C,dv) without using any of the properties of C, so that it applies without modification to the case of D. Hence ykok -1 h as a limit y e D. Since this means both that 112Jk1 = I --+ 0 and that 111,11 I = supflyl,tl-kltfll 11 0, and 0 has limit in D,h). dsk,y) so (ykl a y We began by assuming that (yklwas a Cauchy sequence with deyk,yk-vj) < 1/2 k But this involves no loss of generality because it suffices to show that any Cauchy sequence (ak,n e N ) contains a convergent subsequence fyk= xx, k e N ). Clearly, a Cauchy sequence cannot have a cluster point which is not a limit point. Every Cauchy sequence contains a subsequence with the required property; if -1 k daxn,xn,v) < 1/:(rl) 0 (say), choosing nk 2 g (2 ) is appropriate. This completes the proof. K .
supflyz/) -y(l,(f))
-y(/)
--
---2h
.
--
28.4 Measures on D We write BD for the Borel field on (D,#s). Henceforth, we will also write just D to denote (D,Js), and will indicate the metric specifically only if one different from s is intended. The basic property we need the measurable spae D,Bo) to possess is that measures can be fully specified by the finite-dimensional sets. An argument analogous to 27.6 is called for, although some elaboration will be needed. In particular, we have to show, without appealing (o continuity of the projections, that the tinite-dimensionaldistributions are well-detined and that there are hnite-dimensional sets which constitute a determining class for D,BD). We stal't with a lemma. Define the tield of tinite-dimensionalsets of D as RD fll'l-bD1 H l >f), where R was detined in j27.1. =
28.9 Lemma Given x Hmx,L
=
D,
G
y e D.. R
.
(x >
e A s.t.
0, and any /1,...,/,,, e
(0,11, let
I
.
(28.34)
Km
Then Hmx,
E
Rp.
Proof Since Hmxn
i
D, a1l we have to show according to
(27.2) is
that
for 'ntb,...,tmblmx,l e Bm. This is the set whose elements are @(/1),...,y(/,,;)) each y G Hmx,k. To identify these, first define the set
466
The Functional Central Limit Theorem
lal(/1)),...,x(l(?,,l)):
zArat.x,al
-
,
< al 11111 c
e A,
R'''.
(28.35)
Then it is apparent that %L,...,tmHm(x,4 z4,,,@,a) t:.iR'''. (28.36) = :1,...,:,,,: max Iai bi I < (x, (tz1,...,tz,,,) k'i<m In words, this is the set A,,,@,a) with an open a-halo, and it is an open set. lt therefore belongs to Bm. . -
To compare the present situation with that for C, it may be helpful to look at the where case k = 1. The one-dimensional projection ntllhxnl, 0: (.E.) e A s.t.
z$(x,a) fy e =
.
< a, ly(f) 11111
I<
-?(l(f))
al,
(28.37)
is in general different from &x(f),(x), that is, the interval of width 2a centred continuous If is at f the difference between these two sets can be made on arbitrarily small by taking G small enough, and at these points the projections are in fact continuous. Since the discontinuity points are at most countable, they as will be can be ignored in specifying finite-dimensional distributions for apparent in the next theorem. However, the point that matters here is that we have the material for the extension of a measure to D,Bp) from the finite-dimensional distributions. It is easily verified that Ro, like R, is a t'ield. The final link in the chain is to show that Ro is a determining class for (D,To4. .x(f).
.;t
.x,
(cf. Billingsley
Proof An opn
sphere in D,dn4 is a set of the form
5'txrul
(y e
=
=
D:
dliVnxl<
1968: 1. 14.5)
BD
28.10 Theorem
sRpl.
=
(x) < G, 11111
y e 0: R 1, e A s.t.
Iy(f)
I<
-.x(l(?))
syp
txj
(28.38)
for e D, (x > 0. Since these sets generate Bo, it will suffice to show they can be constructed by countable unions and complements (andhence also countable = O7=1Sk@,a) where Hklx, is a set intersections) of sets in Rp. Let Hxb ) k with the frm of Hm defined in (28.34),but with m = 2 1 and ti = iI2 so tjyat x. Consider the set (/1 J converges on D (the dyadic rationals) as k S(x,G). Since y e Hkx, for every k, we may choose a sequence tzl such ye each 1, k that, for .x
-
,
,...,/2k-1
-->
IllkII< max Iy(2'-k -
1
(28.39)
G,
.x(
,
z(2-h))
I<
a.
(28.40)
Making use of the fortuitous fact that lk has the properties of a c.d.f. on g0,1), Helly's theorem (22.21)may be applied to show that there is a subsequence (,1..
Cadlag Functions
467
n e INJ converging to a limit function which is non-decreasing necessarily in A, satisfying = 0 and (1) 1. is ,
=
on g0,11with
V0)
,
11111
(28.41)
according to (28.39).And in view of (28.40),and the facts that X1(/) X(/) and on g0,1), it must also satisfy either Iy(/) x((/)) I K a x is right-continuous for every t e D. Since D is dense in g0,11,this is or Iy(f) x(X(f)-) l f equivalent to --
-
-
supIy(/)-.x(l(f))I< t
The limiting inequalities
(28.42)
a.
(28.41)or (28.42)cannot
be relied on to be strict, but comparing with (28.38)we can conclude that y e JV,a).This holds for al1 such y, Put a r 1/l, and take the countable union to give so that Hx,ul q Rx,(x). =
Co
-
X
UH(x,
r
UL%x,r
1/n) i
-
1/n)
-
n=1
S(x,r).
=
(28.43)
n=1
It is also evident on comparing (28.34)with 0. Again, put a r- 1/n, and
(28.38)that 5'@,a) i
for
Hkx,
(x >
=
X
Co
Sxt
=
USx,
r
-
UHx,
1/s) I
r
-
1/n).
(28.44)
n=1
n=1
It follows that, for any x e D and r > 0, Sx, Hkx, r- 1/s) e Ro. This completes the proof.
U';=1O7=1fA@, r- 1/s)
=
where
.
The defining of measures on D,Bp4 is now possible by arguments that broadly parallel those for C. The one pitfall we may encounter when assigning measures to finite-dimensional sets is that the coordinate projections of Bp sets may have no %natural'interpretation in terms of observed increments of the random process. For example, suppose Xn e D is the process defined in (27.55)and (27.56),with respect to the underlying space (f,1,#). It is not necessarily the case that p(X,,()) is measurable with respect to in,gnfj Guni, i f g?zf)),as casual in (28.37)is the image under the intuition might suggest. A fo-set like Htx, where of D mapping Xn3( F-> t a set E e 5,' i n fact we could write E %1(zt-1(#)) B e f. But E depends on the value that x assumes at (/), and if (/) > t then E cannot be in Tn,gsrj. However, this difficulty goes away for processes lying in C almost surely. In (C,d,T in ((D,Js),fo) and ap.m. defined on the view of 28.4, we may former space can be extended to the latter, with support in C. In particular, Wiener measure is defined on (D,fo) by simply augmenting the conditions in 27.7 with the stipulation that JF'@ e C) = 1. =
,
=
,
tembed'
28.5 Prokhorov's Metric The material
of this section is not essential to the development since Billingsin D. But it is interesting
ley's metric is a1l that we need to work successfully
Central Limit Iheorem
The Functional
468
to compare it with the alternative approach due to Prokhorov (1956).We begin with an alternative approach to defining a continuity modulus for cadlag functions. Let zxtl
max
=
(mintIx(f?)x(/) I lx(/'')
sup
-
< t's t f t'' < J+
/-
l )),
-x(/) -
,
1x40)
I
-.x()
sup
0
sp
,
1-8<1<1
Ix()
-
x(1) j
.
(28.45)
Again, it may be helpful to restate this definition in English. The idea is thaq 1 8), a pair of adjacent intervals of width 8 are constructed for every t e r, around the point, and we determine the muimum change over each of these intervals; the smaller of these two values measures the at point t, and this quantity is supped over the interval. This means that the function can jump cliscontinuously without affecting fk(), so long as no two jumps are ioo close 0 must be together. The exceptions are the two points 0 and 1, which for /x() true continuity points from the right and left respectively. The following theorem parallels 28.3. -
-continuity
--+
28.11 Theorem
lf and only if x e D, 37?x(6)
1im
0.
=
(28.46)
-->0
in Proof Suppose x e D. By 28.1(c), the second and third tenns under the (28.45) definitely go to zero with :. Hence consider the tirst term. Let (fk,fl,fl denote the sequence of points at which the supremum is attained on Llk for k = 1,2,... Assume tk setting t. (If need be consider a convergent tl Then x(f+), this implies subsequence.) t and t. Since x(f) G 0, which proves sufficiency. (x(fJ Now suppose ;(8) ;x(0) > 0. Since tmax'
-->
=
-->
=
--
-x(f3
-->
--
'x(0)
=
maxt
it follows that
lx(0) x(0+) l lx(1)
l
-x(1-)
-
,
.x
,
D, proving necessity.
minf
lx(?) x(f-) I Ix(fl -
,
-
x(f+)
ll
,
.
Now define the function Wx(z)
f(e2+), z
<
VV1), z
k 0.
0,
(28.47)
=
This is non-decreasing, right-continuous, bounded below by 0 and above by ;(1). It therefore defines a finite measure on R, just as a c.d.f. defines a p.m. on R By defining a family of measures in this way (indexedon on a separable space, we can exploit the fact that a space of measures is metrizable. In fact, we can use Lvy's metl'ic L* defined in (26.33).The Prokhorov metric for D is .
.x)
dPlx,yb
=
dzl7,Fyl
+
f,*(V,x,Vy),
(28.48)
Cadlag Functions
where Fx and Fy are the graphs of
469
and y and du is the Hausdolf metric. clear. The idea here should be With the first term alone, we should obtain a of similar that the Skorokhod metric; if we write #@(f),Fy) to property .x
=
inf,Ws(x(/),y(l')), then Js(Fx,Fy)
=
max sup J@(/),Fy), sup J(Fx,y(/'))
.
(28.49)
t'
f
In words, the smallest Euclidean distances between x(/) and a point of y, and y(/) and a point of are supped over t. For comparison, the Skorokhod metric minimizes the greater of the horizontal and vertical distances separating points on Fx and Fy in the plane, subject to the constraints imposed on the choice of such as continuity. In cases such as the functions ab of (28.6),ab and xeu aze close i n CDd u ) when 8 is small. (Think in tenns of the distances the graphs would have to be moved to fit over one another.) The purpose of the second term is to ensure completeness. By 28.11, limz-j-xpxtz) = 0 if and only if x e D; otherwise this limit will be strictly positive. Unlike the case of (D,du4, it is not possible to have a Cauchy sequence in (D,J#) aproaching a point outside the space. It can be shown that dp is equivalent to ds, and hence of course also to Js, and that the space (D.dp4 is colnplete. The proofs of these propositions can be found in Pmhasarathy (1967). For practical and Js. Purposes, therefore, there is nothing to choose between .&,
,
,
X
28.6 Compactness and Tightness in D The remaining task is to characterize the compact sets of D, in parallel with the earlier application of the Arzel-Ascoli theorem for C. 1968: th. 14.3) A set W c D is relatively compact in
28.12 Theorem (Billingsly D,d.) if and only if
sup sup Ix(f)
-xG A
I
(28.50)
f
limsup wx'()
=
(28.51)
0. n
-->0XGA
This theorem obviously parallels 27.5 but there aze signiticant differences in the conditions. The modulus of continuity wx'appears in place of wx which is a weakening of the previous conditions, but, on the other hand, (28.50)replaces (27.16). where 0 denotes the element of D Instead of suprlxt/) I we could write Jst Ixl everywhere I which is identically zero on (0,11.t is no longer sufticient to bound the lements at one point of the interval to ensure that they are bounded everywhere: the whole element must be bounded. A feature of the proof that follows, which is basically similar to that of 5.28, is that we can avoid involing completeness of the space until, so to speak, the lastmoment. The sufticiency argument establishing total boundness of A is couched in terms of the more tractable Skorokhod metric, and then we can exploit the ,0),
Ixcllllvnl/ArlrtA
nf
rlo witla
sk
fxnmnlata
motrlct
qllctla
nq rln tn
(ynt
the cnmnnctnaqq
nf
W
The Functional
470
Central Limit Theorem
The argument for necessity also uses ds to prove upper semicontinuity of wx'(8), a property that, as we show, implies (28.51)when the space is compact. Proof of 28.12 Let supxsxsuprl x4/) I = M. To show suftkiency, t'ix e; > 0 and choose m as the smallest integer such that both 1Im < : and supxsxwx'tl/vl) < :16:. Such an m exists by (28.51).Construct the finite collection Em of piecewise constant functions, whose values at the discontinuity points / = jlm for j = 1), u 0, 1,...,v J where v is an 0,...,- 1, are drawn from the set fMlulv 1)R + different elements. This set is integer exceeding lMIz; hence, Em has (v shown to be an e-net for A. Given the definition of m, one can choose for x e a partition Hl/,,, above satisfy (,...,/r), defined as (28.3),to =
-
-
:4
max f
sup
l i r s, t e !;
l'j.-!
,lj)
Ix(f)
-
i
x(.)
<
=
/.
(28.52)
For i = 0,...,r 1 let ji be the integer such that jilm f ti < ( + 1)/-, noting that, since the ti are at a distance more than 1/-, there is at most one of them in any one of the intervals Ljlm, (j + 1)/-), j 0,...,- 1. Choose a piecewise with vertices linear function c A If lUf/@) tt, i = 0,...,r. Since lti < the linearity of between and l r1:, these 1/-, maxoujsrl hhlm) l -
=
-
-jilm
=
-jilm
points means that
j sup f
,(f) -
/1 G las.
(28.53)
By construction, maps points in Ljlm, U+ 1)/-) into (fj,fj+1) whenever ji S j G +1, and since x varies by at most rl: over intervals (/f,/s1),the composite + 1)/m). An example function xoiw can vary by at most : over intervals Vlmnj with m = 10 and r = 4 is sketched in Fig. 28.4*,here, jq 2, h 4 and h 6. The points fo,...,/4 must be more than a distance 1/10 apart in this instance, One can therefore choose y e Em such that ,
=
tyjlm)
-
x(lU/r?z)) l<
Y,j
=
Fig. 28.4
0,...,-
-
1.
=
=
(28.54)
Cadlag Functions Since y(/)
=
for t e
yjlm)
I<
suply(f) -x(1(/))
+ 1)/pz), we U/rn,U
max
o<jnm-
t
471
j t
(28.52)and (28.54),
I
-xhljlmj)
yjlm)
1
have by
I
sup
j
-x(l(f))
xhlyml)
f#v,+1)/?zl)
(28.55) Together,
(28.55)and (28.53)imply
dsxny) ; :, showing that Emis an s-net for as required. This proves that A is totally bounded in D,ds4. But since ds and Js are equivalent (28.7),A is also totally bounded in (D,#s); in particular, if Em is an E-net for A in lnds), then we can find n such that it is also an n-netfor # in (D,Js) according to (28.27)and (28.28),where n can be set arbitrarily small. Since (D,#s) is complete, X is therefore compact, proving sufficiency. When # is totally bounded it is bounded, proving the necessity of (28.50).To show the necessity of (28.51),we show that the functions w'@,1/rn) wx'(1/rn) are upper semicontinuous on (D,ds4 for each m. Tllis means that the sets Bm (x: wx'(1/m)< :) are open in lnds) for each e > 0. By equivalence of the metlics, they are also open in (D,Js). In this case, for any such e, the sets LBm,m e EHJ are an open covering for D by 28.3. Any compact subset of D then has a finite subcovering, or in other words, if X'is compact there is an m such that Wc Bm. By definition of Bm, this implies that (28.51)holds. To show upper semicontinuity, fix ir > 0, > 0, and x e D, and choose a partition l'I satisfying :4
=
=
1./)
sup
max
I
x(.)
-
1< i r s,t e (ff-(l,lj) :f
Also choose
n < /,
and small enough
max (n
-
)
/j-1
1< i < r
Our object is to show, aftr
lf dsxnyj
<
n there
>
+
wx'()
<
w.<'()
+ las.
(28.56)
that
(5.32),that wy'()
<
2n.
(28.57)
if y e D and dsx,y)
<
q then
+ E.
(28.58)
I
(28.59)
e A such that
is
suply(l(f)) -x(f)
t
and
I (4 t l sup t -
Letting si
=
klt,
<
(28.60)
n.
(28.57)and (28.60)and the
maxt-f si-k ) > lKffr -
max (ti
li
-
n-1J
triangle inequality imply that -
2n
>
.
(28.61)
The Functional Central Limit Theorem
472
If both s and t 1ie in
(/j-1,n) (J) follows by (28.56),(28.59),and the max
ly(J)
sup
1: i r s,t :6;
and (/) must both lie in choice of that
g-f-j,.J
G
In view of (28.61),this shows that the proof is complete. w
It
n
y(J)
-
(.f-I ,Jf).
l
wx'()
<
(28.58)holdk
,
+
(28.62)
e.
and since
e:and x are arbitrary
This result is used to characterize uniform tightness of a sequence in D. The next theorem directly parallels 27.12. We need completeness for this argument to avoid having to prove tightness of every gz, so it is necessary to specify an appropriate metric. Without loss of generality, we can cite tfs where required.
28.13 Theorem (Billingsley 1968: th. 15.2) A sequence fgw) of p.m.s on ((D,Js), fo) is uniformly tight iff there exists N G (Nsuch that, for a11 n k N, > 0 there exists M such that (a) For each
n
l> sup 1x(/) g'ntfx:
A/')) S
(b) for
each : > 0,
n>
M,n(lI:
0 there exists w28) k
el) f
(28.63)
n;
l
(O,1) such that
G
(28.64)
n.
Proof Let tjtn) be uniformly tight, and for > 0 choose a compact set K with gw(& n By 28.12 there exist M < x and 8 e (0,1) such that > 1 -
n.
K
for any sity.
: >
c
(-x:
sup Ix(/) t
0. lnequalities
l
.&f)
S
rn
(x:wx'()
< :
(28.63)and (28.64)follow
l
(28.65)
for n e :, proving neces-
The object is now to find a set satisfying the conditions of 28.12, whose clsure K satisfies supnxmtm > 1 0 for some N e N and al1 0 > 0. Because D,da) is a complete separable space, each p. is tight (26.19)and the above is suftkient for uniform tightness. As in 27.12, 1et g,* stand for supuzxp.. For 0 > 0, define -
Ak
=
fx: wx'(8)
<
1/k)
(28.66)
,
where (8z) is c hosen so that g,*(AJ> 1 llk'vb p ossible by condition (b). Also set B (x: sup lx(f)l K M) such that g,*(#) > 1 z10,possible by condition (a). Let K = (fV=1z% rn #)-, and note that K satisfies the conditions in (28.50)and (28.51), and hence is compact by 18.12.With these definitions, the argument follows that of 27.12 word for word. > -
,
=
-
The last result of this chapter concerns an issue of obvious relevance to the functional CLT; how to characterize a sequence in D which is converging to a limit in C. Since in all our applications the weak limit we desire to establish is in C, no other case has to be considered here. The modulus of continuitv w- iq tlaa
Cadlag Functions
473
natural medium for expressing this property of a sequence. Essentially, the following theorem amounts to the result that the sufficiency part of 27.12 holds in (D,#s) just as in (C,Jg). 28.14 Theorem (Billingsley 1968: th. 15.5) Let (g.) be a sequence of measures on (D,dp),Bp). lf there exists N e ENsuch that, for n 2 N, (a) for each n > 0 there is a finite M such that
>a(lx:11(0)1> Afl) (b) for each then
E >
tg,nlis unifonnly
n > 0 there is a gw(lx:wx(6) k :l)
0,
E
(28.67)
n;
8 e (0,1) such that f
(28.68)
n;
tight, and if jt is any cluster point of the sequence, p.(C)
(28.4),if (28.68)holds for a given 6 then (28.64)holds for 8/2. Let k (so that k& > 1) where 8 > 0 is specified by condition (b). Then 1)Ik4l 2 el) S according to (28.68),g,n((x: Ixtilk) xtli n for i 1,...,/c,
Proof By
= g1/81 1 +
and t
=
-
-
(0,1j. We have noted previously that
E
x(),) lx(,)Is 1x(0)1+ :-':
f (-,.1 ,)
-x
(28.69)
,
izz1
where the k intervals indicated has width less than 8. lt follows by (28.68) and (28.67)that each of
sup lx(fll > pw(l.x:
M+ lel)
<
g.ttx:
l
I.x(0) I>
Ml) ;
n,
(28.70)
so that (28.63)also holds for finite M. The conditions of 28.13 are therefore satisfied, proving uniform tightness. Let g, be a cluster point such that gw = g, for some subsequence frlk,k G EN J .
Defining
f.x:wx(8) > :J, consider the open set A0, the interior of A; for AO if wx(&2) k 2E. Then by (d) of 26.10, and (28.68), x e
,4
=
example,
p,(AO)
< liminf
p,,,:(z4O)
(28.7 1)
f q.
k-Ax
Hence g,(#) < for any set B A0. Since : and q are arbitrary here, it is posn sible to choose a decreasing sequence f6yJsuch that bkBp < Slj, where Bj (x: wxty) Sljl For each m k 1, p,(O7=,u)= 0, and so, by subadditivity, p.(#) 0 BC #C, where where B liminf Bj. But suppose x e O,,LIU''/ =mKis the set =
=
.
=
=
f.r:
< 1#, wx(Y)
m; al1 m e EN l
some j
.
is monotonic, it must be the case that lim-owxt6) Bc 1 follows. w Hence i C, and since p,(#f') 1, gtC)
Since f
./1
=
=
=
0 for this x.
29 for Dependent Variables
FCLTS
29. 1 The Distribution of Continuous Functions on D A surprising fact about Wiener measure is that redundant; if part (b) of that definition is replaced of the first two moments of a/), Gaussianity of x to a class of functional CLTS of considerably greater possible with the approach of j27,6.
definition 27.7 is actually by the specitkation merely must follow. This fact leads power and generality than is
29.1 Theorem (Billingsley 1968: th. 19.1) Letxbe a random element of Dm,lj with the following properties: (a) E (X(f)) 0, Extj 2) f, 0 < t < 1 (b) P(X e C'''I= 1. of (0,11, the increments Xt X(/1), (c) For any pmition f Xtk-L), X(tz) X(/2), independent. Xt totally are =
=
.
,...,41
-
-
-
...,
Then X
-
B. u
This is a remarkable theorem, in the apparent triviality of the conditions; if an element of D is a.s. continuous, independence of its increments is equivalent to Gaussianity! The essential insight it provides is that continuity of the sample paths is equivalent to the Lindeberg condition being satisfied by the increments. The virtuosity of Billingsley's proof is also remarkable. The two preliminary lemmas are technical, and in the second case the proof is rather lengthy', the is a random reader might prefer to take this one on trust initially. If (1,...,(,,, = and problem and define 1 St 0, the is to Sj for K j K m, we sequence, exceeding a given value. The lemmas are obviously bound the probability of l work together to this end. designed to =
xs'p,l
29.2 Lemma
ISmI f
S=1(j
2 max minf ISj I ISm -
,
sjsm
h 1J +
max j(./j .
0<jKr)
vb'm Proof Let I (0,...,m)denote the set of integers k for which I&1 < I 51 if Sm # 0 then m e 1. On the other hand, 0 e 1. It If Sm = 0 the lemma holds, follows that ther is a k e I such that k 1 e 1. For this choice of k, -
.
.and
-
lSm1f ISm K lSm
-
u%1
.%1
-
+ +
K 2 max mint fssj f m
lSkl iSk-q 1 + 1kl I l l h Il .h.
vb'm
,
-
+
max 0 sj<m
l . I%. .
(29.1)
FCLTS
for Dependent
The second lemma is a variation
Processes
on the maximal
475
inequality for partial sums.
29-3 Lemma (Billingsley 1968: th. 12.1) If Eh
Siblsk
-
k b,
Sj4l) <
-
2 ,
j
j,...,k,
=
(29.2)
/=f+1
is a collection of posifor each pair i,k with 0 < i f k f m, where f:I,...,%) a11 and a11 m, numbers, (x > > then R K 0 such that, for 0 tive 1
P max minf 0 Kj S m
where B
lsjI Ism Il 'j
K
(x
-
,
KB
(29.3)
,
4 (X
'Xjbj. ./
=
Proof For 0 f i f k S m and
z'tminfIsj siI Isk sjIl -
a)
-
,
0, we have
(x >
#(( ISj Si I
=
al
-
( lSk
fa
< #( I&. sillsk sjI k a -
1
-
k
z
x
X <(X4 /=j+1
-
5)1 k al)
)
2
bl
(29.4)
,
where Chebyshev's inequality and (29.2)give the tinal inequality. If m = 1, the minorant side of (29.3)is zero. If m = 2, (29.4)with i 0 and k 2 yields =
#tmaxto, minf I& l l
vbb
st l l l k
-
,
a) f
(&1+ :2) 4
=
2
(29.5)
,
(X
1 and hence for any K 2 1. The proof now proceeds by induction. Assuming there is a K for which (29.3) holds when m is replaced by any integer between 1 and m 1, we show it holds for m itself, with the same K. The basic idea is to split the sum into two parts, each with fewr than m terms, obtain valid inequalities for each part, and combine these. Choose h to be the largest integer such that Zlx-lljS #/2 (the sum is zero f #/2 also (the sum being zero if h if h 1); it is easy to see that L%) m4. First define
so that (29.3)holds for
zv
=
-
=hwqbj
=
=
171 =
D1
=
max minf 0 GjKh-j
lM1I ,
vh-t
minl ISh-t l lSm 5'-1 -
,
-
5)l l
(29.6)
I1.
(29.7)
Evidently,
pvt
u a) /
K
w tx
-1
Xbj j=j
1
2
<
Aw 4a
(29.8)
,
by the induction hypothesis. Also, by (29.4)with i
=
0 and k
=
m,
The Functional Central Limit Theorem
476
#(DI
a) K
k
B
2
(29.9)
w.
The object is now to show that
minl15)1 1Sm
,S)11 < 171+ Dl, 0 < j S h
-
,
If I5'.jl K &1, (29.10)holds, hence suppose IShbility according to (29.6).If D1 ISh-L I then =
mintlh I ISm h l l
ISm Sh-j l then again, mint lSj l ISm $.l l < ISm l S I =
Sj ( < &1, the only other possi-
-
Sj l + ISh-t I S 1.71+ D1
-
.
-
-
vb'j
-
,%-
1
,
Hence
(29.10)
,
(Sj I K ISh-k
-
,
And if D1
1.
-
t
(29.10)holds
-
s'./I+ lSm
-%-
-
l
lS
!71+ D1.
in al1 cases. Now, for 0 S g. f 1,
#4U1 + D1 k a) f #(( (/1 k gal k? (Dj k
f #41.71k gal <
KB2
-
(1
#(D1 k
+
(1 jt)aJ) jt)a)
-
,2
4 +
41 4yt
4
(1
(x
(29. 11)
4.
.jt;
jz)4
yields choosingjt to minimize r/4g.4+ 141 Back-substituting for and simplifying p. calculus). -
g,
+ (tx)1/5j (,1r)1/5/(1 (use
=
yields, for K k 2(1
-
(r1)1/5j-5
= 55,021,
+ 1j5 #2g(j.r)1/5 -
81.71+D1 2 a) S
G
S
4
vpz
n'u
2a4
.
(29. 12)
According to (29.10),we have bounded mint 1Sj I 1Sm 5'yl) in the range 0 f j S h 1. To do the same for the range h f j f m, define -
,
-
U1 = max minl lSj Shj lSm -
-
,
h<jsm
D2
=
minl
1
,:,,1
lsm
.,1
)
(29.13) (29.14)
).
-
,
'y1
lt can be verified by variants of the previous arguments that
minfI5'./i 1sm 5'j( l -
,
< &c+ Dz, h <
j G m,
(29.15)
and also that #(W+D2
2
a) S
KB
2a
2
4
for the same choice of K. Combining (29.16)with (29.12),we obtain
(29. 16)
for Dependent
FCLTS
Processes
; J'tmaxf .71+
P max mint ISj I ISm Sj Il k a -
,
0 %<m
ol
= P(f &1+D3
,
a)
< #4171+ D1 k a)
o2)
a)
( Uz +D2
k
Uz + k.p
+ PIUI
a))
D1 k a)
+
2
KB
<
(29.17)
k w .
Proof of 29.1 Let the characteristic function of X(/) be
#(/,) We can write, by
E e ix
=
).
(29. 18)
yu),
(29.19)
(11.25), e iu =
! + iu
1jy2
-
z
+
where Irul I S Iu I3 We shall write either Ayt or A(J,/), as is most convenient, to denote aY(.) Xt) for 0 S t S s S 1. Observe that by conditions (a) and (c) of the theorem, Etn,t)2 uence, .
,
-
=
.
Fge ij-ht) e
9(/+ h,h4 9(/,1)
=
-
= =
fhxlkr+h,t
Fle i.Y(P(f,At+y (1)(/,1)(w112/1
,
j
..
jhlhlt.v /; t
-
t
)j ,
+
r(,A,+yy,,))j
F(r(A,+,,;)j,
+
(29.20)
where the last equality is because X(/) and ht-vh,tare independent by condition 3, S 3FIAf+,,rIit follows that (c), Since Etfhhtn,)
! (/+,1) '
-
9(/,1)+ s2' t s; u
):3,1
'(/,
,
-
h
At+,,I
3
,
(29.21)
.
Now, suppose that :3 1 limi FIAr+/,,rI 0.
(29.22)
=
$0
lt will then follow that, for a110 S t
1,
<
h,h) 9(/,1) lim#(/+ h nztl -
=
-
possesses a right-hand
saj(/,u).
derivative,
(29.23)
Further, for h > 0 and h S t S 1, (29.21)holds at the point t so by considering a path to the limit tluough such points we may also conclude that -
,
9(/,) ()(/ 1im h l() -
Since
-
9(/-,1) 9(/,:) because
theorem,
=
-r
1s2j(j-,s).
is continuous in /, by condition differentiable on (0,1) and =
(1) is
,X) ()
(z9.a4) (b) of
the
Central Limit Theorem
The Functional
478
:4)
g
This differential equation
=
-J
jN2j(/,s;.
(29.25)
is well known to have the solution
(f,X)
#(0,X)d-tkllz t 2 0.
=
(29.26)
,
0 a.s., (Verify this by differentiating log (1) with respect to /.) Since X(0) N(0,/) for 1, and applying the inversion theorem we conclude that X( #(0,) () result also continuity extends of the 1. at 1, each t e (0,1). By to t Hence, the task is to prove (29.22).This requires the application of 29.2 and 1,...,-. By 29.3. For some tinite m let Y. A(f + hjlm, t + JIU 1)/-) for j = with variances the of If independent hlm. Sj Ei,=1k assumption, r.v.s (j are +jhlm, /). A(/ then =
=
-
=
=
=
-
=
E((h
-
By 29.3, setting bj P max min
ojsm
Si4l(Sk Sj)l)
-phllml
=
JI t ,(,
(j
=
-
E hl
ijlk
-
(;9.gg;
hlm. we have
r-...-zz. J s L 1j )
---,./
+
m
/)
,
l ja(, ,
,)
,, t +
+
(x
(29.28)
Hence by 29.2,
I
#( IA(/+ h,
i
IA(/+
a) / P 2 max min
--hm
ONKr?;
+
f
r*2 a*
j
,
j
j
Oij
j
-mh
t+
,
--./1.
,
t+Lh4
I Ia(,+F,,
A(/ +
max
max A(t + oxjx,a
P
+
x
/)
,
t+
j- 1 tyt
h)
j
I
m
-
1
s;
j
)
+1
Ik ,
(29.29)
4 where K + 4 K. Letting m x, the second term of the majorant member must go to zero, since X e C with probability 1 by condition (b), so we can say that -->
=
P Iht-bh'tI
we may
a)
<
2
r %
(29.30)
.
4
now use 9.1s to give
FIA,+,,I3
JC
=
1Ar+J,,,l 3dF
+
o
vwj- -#(IAf+,,,l
<
< e+r
x
2 E(
1 4/3
3 zP lA,+,fj
:) +
z
jortj z
3
A/+/,,,I> ()#(
z (1'3) *2
dL
=
:+
3# h
e
1/3
.
(29.31)
for Dependent
FCLTS
Choose er
=
3/4 3/2
(K +) h
to minimize
Processes
479
the last member
above, and we obtain
3 44s.+;3/4,3/2 E IAr+,,I u
(zoap,;
.
This condition verifies
(29.22),and
.
completes the proof.
.
Notice how (29.30)is a substantial strengthening of the Chebyshev inequality, W hich gives merely #( lAl+/,,rI a) f hlql. we have not assumed the existence of the third moment at the outset; this emerges (alongwith the Gaussianity) from the assumption of independent increments of arbitrarily small width, wlch allows us to take (29.29)to the limit.
29.2 Asymptotic lndependence denote IULJT
(D,BD). We say that Xn has asymptotia stochastic sequence in 1,...,r) cally independent increments if, for any collection of points fsi,ti,i
Let
=
such that
0 <
-1
f /1
<
n
< tl
<
...
<
sr S tr K 1,
and a11collections of linear Borel sets #1,...,#r e
,
r
Pxnti)
-xnsi)
e Bi, i
=
1,...,r)
-->
I1#(-Yn(n)
e Bi)
-xnlsil
(29.33)
f=1 x. Notice that in this definition, gaps of positive width are allowed to as n separate the increments, which will be essential to establish asymptotic independence in the partial sums of mixing sequences. The gaps can be arbitrarily small, however, and continuity allows us to ignore them as we see below. --->
Given this idea, we have the following consequence of 29.1.
29.4 Theorem Let t.Ll';xl have the following properties: (a) The increments are asymptotically independent. 2 e) f (b) For any : > 0 and n > 0, 3 8 e (0,1) s.t. limsuw-oxfwtxn,) (Xn(/)2)=s=1 11. is unifonnly integrable for each (c) t e (0, x, (d) Elxnj 0 and E(Xn(t)2) t as n eac ja t s p 1j. -->
n.
-->
--
,
Then Xn -P-> #. n
Be careful to note that w(.,) w' of (28.3).
in
(b) is the modulus
of continuity of
(27.14),not
Proof Condition (b), and the fact that FIXno) I 0 by (d), imply by 28.14 that uniformly tight. Theorem 26.22 then implies the associated sequence of p.m.s is that the latter sequence is cpmpact, and so has one or more cluster points. To complete the proof, we show that a11 such cluster points must have the characteristics of Wiener measure, and hence that the sequence has this p.m. as its unique weak limit. Consider the properties the limiting p.m. must possess. Writing Xfor the random element, 28.14 also gives PX e C) 1. Uniform integrability of Xnlt)2 and hence 0 and Ext) 2) = t, by 2 2 16 By condition (a) we of Xnt), implies that E(Xt)) -->
::
=
,
.
.
The Functional Central Limit Theorem
480
may say that the increments X(/jl X(-1),...,X(/r) X(.r) are totally independent according to (29.33).Specitkally, consider increments Xlti) Xs and -
-
-
Xtiwt)
A,fmll for the case where
-
(X(/j+I)
lim
-
Xti
+
1/&4)
ti + Slm. By a.s. continuity,
=
xf+l
Xtiwj)
=
-
(29.34)
Xti) w.p. 1,
mM* j'
.
so that asymptotic independence extends to contiguous increments. Al1 the conditions of 29.1 are therefore satisfied by X, and X B. w -
Our aim is now to get a FCLT for partial-sum processes by linking up the asymptotic independence idea with our established characterization of a dependent increment process; that is to say, as a near-epoch dependent function of a mixing process. Making this connection is perhaps the biggest difficulty we still have to argument used in the CLTS of surmount. An approach comparable to the j24.4 is needed', and in the present context we can proceed by mapping an infinite independent sequence into g0,11andidentifying the increments withasymptotically blocks of summands. This is a particularly elegant route to the result. However, an asymptotic martingale difference-type property of the type exploited in 24.6 is not going to work in our present approach to the problem. vsrhile the tenns of a mixing process (of suitable size) can be so that the blocks are definition independent less by of mixing), mixingale asymptotically (moreor theory will not serve here; near-epoch dependent functions can be dealt with only by a direct approximation argument. What we shall show is that, if the difference between two stochastic processes is ty41) and one of them exhibits asymptotic independence, so must the other, in a sense to be defined. Near-epoch dependent functions can be approximated in the required way by their near-epoch conditional expectations, where the latter are variables. This result is established in the following lemma functions of lnixing in terms of the independehce of a pair of sequences, which in the application will be adjacent increments of a partial sum process. Kblocking'
Kblocked'
29.5 Lemma (Wooldridge and White 1986: Lemma A.3) If ( 1%1 and lZ;.nl are real stochastic sequences, and 1,2; (a) Yj Z'n -EC> 0, for j -P-> (b) Yjn 4. for j = 1,2; (c) for any Al, e B, =
-
-42
#(tZ1n e All rn tZ2n e A2l) as
n
-->
--
e A1)#(Z2n e A2)
Pzn
(29.35)
=;
then #(t Fln
6
#1 l f''h l i'2,, e #21)
--
/i'1
e
BL4PCYI
for all Y/continuity sets (sets Bj c B such that Pj'j. e
e #2) Bp
=
(29.36) 0) for j
=
1,2.
Proof Considering (Z1n,4z) and (F1n,F2n) as points of R 2 w ith the Euclidean metric, (a) implies #x((z1n,Z2n),(F1n,F2n)) 0, and by an application of 26.24, --)
Jlah lmnlioe
lantla
(7.
7,. h
X
y
(Y.
Foh
onrl
l
=
17
Wrlto
for Dependent
FCLTS
#(lZ1n e #1 1f''n
fz2,,e
#21)
Ptocesses
481
p.,;tll xBz4,
=
(29.37)
where gwis the measure associated with the element (ZIn Z2u).If g, is the measure associated with (l'1,F2), define the marginal measures g,J by p?(sy) - p. e s,); 1,2 implies g(:(#1 x#2)) 0, in view of the fact that 0 for j then VBp =
=
=
Bj x #z)
Bj xR) t.p (R x B.
c
(29.38) of the joint distribu-
Applying (e)of 26.10, it follows from the weak convergence tions that, for al1 lcontinuity sets Bj, #(lZln e #1 l r'n lZ.2n e #21)
m(#1xB
=
.-+ /1,(#1xB1) = And by the weak convergence that, for these same Bj,
#((
J'I
f Fc e #2J).
e Bj l f'a
(29.39)
of both sets of marginal distributions it follows
#(Zln e F1)#(An e B1)
PY1 e Bj4PY1 e #2).
--
(29.40)
This completes the proof, since the limits of the left-hand sides of (29.40) are the same by condition (c). .
(29.39)and
29.3 The FCLT for NED Functions of Mixing Processes From 29.4 to a general invariance principle for dependent sequences is only a short step, even though some of the details in the following version of the result are quite fiddly. This is basically the one given by Wooldridge and White (1988).
29.6 Theorem Let ( Unil be a zero-mean stochastic an-ay, fcnil an an'ay of positive constants, and (#n(f),n e IN) a sequence of integer-valued, right-continuous, x incteasing functions of f, with Kn(04 0 for al1 n, and #n(f) Kns) as n -->
=
---
x
if t
(a)
X'
s. Also define Xn Euni) 0,' >
for r
< x,
size
-y,
tcnj), on an array f F,,f) limsup
sup
If
f
rG g0,1),:G
(0,1
max cni 1 i < Kn(1) K (9 Exnl
then X*' -P- #.
>
for zl; y ; 1, with respect to the constants 2)*, a-mixing of size
which is
2(f )
Vn
2-, -r/(r
-
A'n(?+*
,
<
x,
o
where vj(J,)
.--x
..
=
2
)
--.2/
n---lco =
oKn
X
o
c'Li;
f=Ak(/)+1
-f1
!5;
n
ni. 1 flou
=
=
suptsll Unilcni6kr (b) (c) Uni is AZ-NED of
(d)
-
T-1
1) ), where
t as n --A
x,
y is defined in (c);
for eac jj t s p jj ,
.
,
In 2
0, if we agree that a Right-continuity of Kn ensures that vn(f,) 0 as equal whenever the lower limit exceeds the upper. to zero sum is ..-.#
--
The Functional Central Limit Theorem
482
If is set to 1 in condition (c), condition (e) can be omitted. lt is important to emphasize that this statement of the assumptions, while technically correct, is somewhat misleading in that condition (c)is not the only constraint on the dependence. ln the leading cases discussed below, condition (9 will as a rule imply a La-NED size of Theorem 29.6 is very general, and it may help in getting to grips with it to extract a more basic and easily comprehended set of suftkient conditions. What we might think of as the case of the FCLT that of convergence of a We partial sum process to Wiener measure corresponds to the case Kn = g?7/l. writing 5Vi1l omit the K superscript to denote this The Z,tl'1& Xn(/) ni. case, full conditions of the theorem allow different modes of convergence to be defined for various kinds of heterogeneous processes, and these issues are taken up again in j29.4 below. But it might be a good plan to focus initially on the case Xn(t), mentally making the required substitutions of gn/jfor Kn in the formulae. In particular, consider the case Uni Uilsn where 'y
-1.
Estandard'
-
-
=
=
2
sn
=
11
E
(2.
Ui
n
-
-
=
f=l
n 1 n i
2
f=1
cj
+
2 f=1 rrl=l
(29.41)
cf sm,
with c?, Var(&f) and ojom Covui,uiwm). Also, require that supfll &fIIr< x, (rl/1,condition 29.6(d) r > 2. Then we may choose cni llsn, and with Kn is reducesto the requirement that sn2/n > 0 uniformly in n. ln this case, 29.6(e) ().2 < x, then E(X,,(r))2 snlln sl gy/.s2 addition If in for t satisfied y r.1 rn summarized the 29.6(9 also holds. conclusions in These following are and corollary. =
=
=
=
,
-->
.-+
=
=
29.7 Corollary Let the sequence f Ui G-mixing size and Q-NED of rI-1().j,j)2 on an
have mean zero, be uniformly fv-bounded, 2), and 1et Xnt) process of size (y2 -P-c2#. u c2, () < < x, then Xn
-,1
t
-1/2UfI=1 14./j. If
..-,
-r/(r-
=
where l limn-yx rl-1Z2=1cl + 2;=1., and l,u = Be careful to note that c2 is the variance of the limiting Brownian iwm. This and not 1imn -+xl-1E2=-Tcf motion, notwithstanding the fact that B has independent increments. The condition c2 has two parts. The first is that the limits d2 and v for m 1,2,3,... Sn2/N al1 exist, which is the condition of global wide-sense stationarity discussed in j13.2. Examples where this condition is violated is provide by 24 10 and 24.11. The second is that E,1xl3. < x, for which it is sufficient that Z':.I lokimm 1< x for each i. According to 17.7 this follows from condition 29.6(c), with the additional requirement that y 1. Thecomplications of 29.6 arechiefly to accommodate globalnonstationarity. The following is such a case. =
(51-
=
-c2
,
=
--
,
.
=
c? 7 and (justfor simplic29 8 Example Let the sequence f Ui ) have variances o ( O(n'+i5) and choosing Kntj = ity ' s sake) be serially uncorrelated. Then, Jj 1R1+O ill serve to satisfy conditions 29.6(d) and 29.6(9. n nt 1w -
,
=
,
for Dependent
FCLTS
Processes
483
It is instructive to compare the conditions of 29.6 with those of 24.6 and 24.7. N(, 1), the two theorems give alternative sets of condiSince A$(1) -P-) #(1) tions for the central limit theorem. Although they are stated in very different terms, conditions 29.64d) and (e) clearly have a role anlogous to 24.6(d). While it was pointed out above how the same condition 24.6 required a Q-NED size of is generally enforced by 29.6(9. However, 29.649 itself has no counterpart in the CLT conditions. It is not clear how tough this restriction is, given our free choice of Kn, and this is a question we attempt to shed light on in j29.4. What is clear is that the convergence of the partial stlm process Xn to B requires stronger conditions than are required just for the convergence of uL(1)to #(1), which is the CLT. -
-1,
Proof of 29.6 We will establish that the conditions of 29.4 hold for the sequence K lXf 1. C O ndition 29.4(d) holds directly, by the present condititjns (a) and (t). Conditions (a),(b), and (c) imply by 17.6(i) that ( Uni,snil is a Q-mixingale of with respect to the scaling constants lcnjl, where 5ni = Gvi-j, j k 0). size 2 2/ cni In view of the uniform fv-boundedness with r > 2, the array ( Uni j is uniformly 4'5 Knt) and m Knt + 6) Knl for integrable. If we let k e (0,1 f), it follows by 16.14 (whichholds irrespective of shifts in the coordinate index) that the set -
=
=
(Sn,k+j Sn -
2
max vn(/, 2 :) jm
,
-
-
(29.42)
n k 1
is unifonnly integrable, for any t and Further, because of condition (d)we may assume there is a positive constant M < cxo such that for any f e g0,1)and any 8 E 2 4'5)/ < M for n 1 with the property vn(/, (0, 1 /1), there exists N(t,&) .
-
Ntnn). Therefore the set
sn,k-vjSn 2 -
max
,
is also unifonnly integrable. If N* is finite.
n
Nt,nj
(29.43)
sup,A(/,8),condition (d) implies
=
that N*
Taling the case t 0 and hence k 0 and m Kn(&4in (29.43)(butthen writing t in place of for consistency of notation), we deduce uniform integrability of K lXn(f)1n=1 fOr any / e (0 11(thesummands from 1 to ,V(0,/) 1 can be included by K= condition (b)). In other words, condition 29.4/) holds for t.L )n=1. Note that hlp I#I > 1) S F(-Y21()xlu)) for any square-integrable r.v. X. Therefore, the uniform integrability of (29.43)implies that for any 8 e (0,1),any / S > 0 large enough that for n k M, and any : > 0 and q > 0, 3 1 =
=
=
'='
-
,
-
.
,
P max
Isn,k..j&11k -
2
kv
S
tusm
ME
,
8%2
(29.44)
where k amd m are defined as before. The argument now follows similar lines to the proof of 27.14. For the case 6 : lj4hl (2944) itnplies =
,
.
Central Limit Theorem
The Functional
484
P
sup < l
0
sup t
-
KJ
K Ix,,(J)
-
xKnq)
12
<J+3
< ltn6, n k N*,
:
(29.45)
which is identical to (27.71).Condition 29.4*) now follows by 27.13, as before.
The final step is to show asymptotic independence. Whereas the theorem requires us to show that (29.33)holds for any r, since the argument is based on the mixing propel'ty and the linear separation of the incremerts, it will suffice to show independence for adjacent pairs of increments i, i + 1) having ti < Jf+1 The extension to the general case is easy in principle, though tedious to write out. Hence we consider, without loss of generality, the pair of variables .
Yjn
=
K
Xnt?
-
xKns?
ratj)
g ni, J
1 and 2
.
=
izznsjj-
=
(29.46)
,
1
h < n < /2 K 1. We cannot show asymptotic independence of F1nand F2,, directly because the inerement process need not be mixing, but there is an k approximationargument direct from the NED property. Defining 5n,j ' Tfnt/ F(F1a and similarly c( F nj,..., F nk ) the r.v. r,. I ) is shlt E (F2n ls,Knszlj is Tanwntupmeasurable.By assumption (c),
where0 < J1
<
=
-measurable,
,
A'a(l)
A e 5n
,
j #(A SuP B e Fc<' x, n,Knsz)
f''h
x
x
,
#)
-
17(4)/7(#)
j
=
--
a(Kn(sz) 0
-
aS /1 -V
Ak(/1))
(29.47)
C>O
the events A include those of the form (F(l'1s I?xntlzlx ) e E ) for E e 0, and similarly events # include those of the type (F(l%n ls*n,Knszll e A') .'Fhese conditional expectations are asymptotically independent r.v.s, and it remains to show that Flrl and F2s share the same property. We show that the conditions of 29.5 are satisfied when Z1a Ejhn I?/Xnnt/ ) and E(Yzn Is*n,xnszlj. This is sufficient in view of the fact that the Zzn convergence-determining class for the sequences ( Yjn) by lcontinuity sets are a 26.10(e). The argument of the preceding paragraph has already established condition 29.5(c). To show condition 29.5(a) we have the inequalities W henever
fl <
J2, Where
=
,
CK3
=
,
Knh)
Fa e'11
-
e'tlhnITftt'
Uni k) 112s 77 11 f=A'n(.1)+l KntL
t' k) II2 Eunik @fn,
-
Q
K 2
&nf 11 -
izzKns 1)+ 1
s' (,j)) F( UniITfnnctr-l 2 11
KntL)
S 2
cniMxtkj-i f=A-n(J1)+1
FCLTS
for Dependent
Processes
485
Ak(ll)-k(.1)-1
f 2
A'
vm
cni
max <&t/l)
KnsLj<
--+ 0 as n
m'u
(29.48)
x,
--
wherewe have applied Minkowski's an d (e), and 2.27. This implies condition (d) implies that supjof
inequality, 10.28, and finally assumptions (c) that Fln E (F1nlTfa ' x ) -T-r-> 0. Note that = x, 1, (e) can 0 as n so in the case s*n,Knsvj -C!-> with. dispensed reasoning, 0 also. Ebn By be l%n the same I Since we have established that conditions 29.4(b) and 29.44d) hold, we know that the sequence of measures associated with fXnK l is uniformly tight, and so contains XK XK that nk k e EN at least one convergent subsequencesay 1, (nk, x where PX K e C) = 1 lt follows that the continuous mapping (say) as k theorem applies to the coordinate projections xtx K) XK ( and we may assert XK XKt4. Confining attention to tls subsequence, condition 29.5(b) that nk t) -P-> K XKnsp. A1l the conditions of 29.5 is satisfied for the case Ynkvj Xntp have owbeen confirmed, so these incrementsare asymptoticallyindependentinthe sense of (29.36).But since tllis is trueK for evel'y convergent subsequence (nzJ,we can conclude thatthe weaklimitof fA%l h as asymp toticallyindependentincrements K whenever it exists. A1l the conditions of 29.4 are therefore fulfilled by (Ak ), and the proof is complete. . -
'y
.,--)
--+
-
.-2-h
-such
-->
.
=
=
,
-
It is possible to relax the moment conditions of this theorem if we substitute a unifonn mixing condition for the strong mixing in condition (c). K
29.9Theorem Let f Unij tcnj)(#s(/)), and (Ak l b e defined as in 29.6' assume that conditions 29.6(a), (d),(e) and (9 hold, but replace conditions 29.6*) and (c) by the following: UnilcniIIr< x, for r 2, and ( Unlilcnli (b') suptn 11 ) is uniformly integrable; ,1 < < 1, with respect size is of Uni AZ-NED for to constants fcnjl, (c') y which Fnj) is of size 1), for r 2', array an ( (-mixing XK on -P-> B. n then n ,
,
,
-y,
-r/2(r
-
The uniforin integrability stipulation in (b') is required only for the case r 2, and the difference between this and the a-mixing case is tht tls value of r is permitted, corresponding to a (-mixingsize of =
-1.
is again an Aa-mixingale of size in this case. The before establish conditions 29.4(b),(c) and arguments that (d) hold; and, as same a(?Al) S since (440,condition (29.47)remains valid so that asymptotic indepenalso holds by the same arguments as before. . dence Proof By 17.6(ii),
( Unil
-
29.4 Transformed Brownian Motion To develop a fully general theory of weak convergence of partial sum processes, permitting global heterogeneity of the increments with possibly trending moments,
The Functional Central Limit Theorem
486
and particularly to accommodate the multivariate case, we shall need to extend the class of limit processes beyond ordinary Brownian motion. The desired generalization has already been ineoduced as example 27.8, but now we consider the theory of these processes a little more formally. A transformed (orvariance-transforme+ Brownian motion Sn will be defined as a stochastic process on g0,IJ with tinitedimensional distributions given by
fc(?)
f(n(J)).
-
t
(29.49)
E0,11.
6
where #is aBrownian motion andn is an increasing homeomorphism on (0,1jwith n40) 0. The increments of this process, #n(l) #n(J) for 0 ; t < s S 1, are therefore independent and Gaussian with mean 0 and variance n(f) n(J). Since 4(1) must be tinite, the condition n(1)= 1 can be achieved by a trivial normalization. To appreciate the relevance of these processes, consider, as was done in j27.4 the characterization of B as the limit of a partial-sum process with independent Gaussian summands. Here we let the variance of the terms change with time. SupJ2u and 1et F(IJ=1(j)2 :2.1c?,.Also suppose the variance , pose kd N(0 c?), (c?, h as JT=1 the property that, for each t e (0,11, sequence =
-
-
=
-
,
2 xlnfj 2
Sn
=
--4 q(/) as n
(29.50)
x,
--
where the limit function n: g0,1J F- (0,11is continuous and strictly increasing everywhere. ln this case, according to the definition of #n, we have
Zf=l lj f (nf Sn
o
-..-->snql,
(29.51)
for each t e (0,11.
What mode of evolution of the variances might satisfy (29.50),and give rise to this limiting result? ln what we called, in j 13.2, the globally stationary case, where the sequence (c2j)Tis Cesro-summable and the Cesro limit is strictly positive, it is fairly easy to see that n(/) t is the only possible limit for (29.50). This conclusion extends to any case where the variances are uniformly bounded and the limit exists; however, the fact that uniform boundedness of the variances is not sufscient is illustrated by 24.11. (Try evaluating the sequence in (29.50)for this case.) Alternatively, consider the example in 27.8. lt may be surprising to tind that < j) < 0) the partial sums have a well defined limit process even (for the case when the Cesro limit of the variances is 0. However, 27.8 is more general than it may at first appear. Define a continuous function on g0,x)by =
-1
#(V)
lf
u%2
satises
.
.
(29.50),g .
,.
=
2
Jvl
+ (V
-
(V1)c2gv+1j.
(29.52)
is regularly varying at intinity according to 2.32. g .-.,,.j, fjasf o(n + jj = rrv,, rajg .1 -a
.j
x
..
-
for Dependent
FCLTS
Processes
487
gln) +g'(n) for integer n, and note that by 2.33 (whichholds for right derivatives) g' is also regularly varying. The variance process of 27.8 can be generalized at most by the inclusion of a slowly varying component. This is the situation for the case of unweighted partial sums, as in (29.53), the one that probably has the greatest relevance for applications. But remember there are other ways to define the limit of a partial-sum process, using an array formulation. There need only exist a sequence (',,)T of strictly increasing functions on the integers such that gnfntjjlgnln) q(/), and the partial sums of Nnhl and the array ((,,jl, where (nf ---
-
62ni
=
(gni) gnli -
-
1))/%(?z),
(29.53)
will converge to #n. And since such a sequence can always be generated by setting .gn(E?l/))q(/)t7n, where an is any monotone positive real sequence, any desired member of the family #n can be constructed from Gaussian increments in this =
mannec.
The results obtained in j29.1 and j29.2 are now found to have generalizations from B to the class #n. For 29.1 we have the following corollary.
29.10 Corollary Let condition 29.1(a) be replaced by 0, E(X(t4h (a' ) F(X(/)) q4/), 0 t S 1. =
=
Then X
-
#n.
Proof Define X'*t) = X(q-1(/)) and apply 29.1 to X*. n-14 ) is continuous, so condition 29.1(b) continues to hold. Strict monotonicity ensures that if -1 define arbitrary non-overlapping intervals, so also do ln (f1),..., (/1 ) -1 q tmll, so 29.1(c) continues to hold. . ...,tm
Similarly, for 29.4 there is the following corollary.
29.11 Corollary Let the conditions (a),(b).and (c) of 29.4 hold, and instead of condition 29.4(d) assume 0 and Exntjl) (d' ) EX a (/)) n(/)as n x, each t E g0,1). -->
--y
-->
Then Xn -P-->#n.
Proof The argument in the proof of 29.4 shows that the conditions of 29.10 hold for X. . 29.12 Example Let ( Uill denote a sequence satisfying the conditions of 29.7, with Detine the cadlag process the extra stipulation that the Q-NED size is -1.
xn
1
=
Gn
m
Ilnll
77./f4. j=1
(29.54)
only by the multiplication of the This differs from the process a-l/zztnlll-ryc 7= values between $In and 1. The arguments summands by constant weightsy/n, taking and conditions 29.4(a), 29.6 show that of (b), (e) are satisfied for this case, and it remains to check 29.11(d'). We show that
The Functional Central Limit Theorem
488
f(Xntf) 2)
1j3
'-+
!
(z,
.
.
55) 1/2
cxa but bnln 0; bn = grl j Choose a monotone sequence (bne IN) such that bn will do. Putting rn = fntlbnjfor t e (0,1J and n large enough that rn 1, we --y
--
have
xn
=
'
ibn
rn
j
(raj
jUj 77 77 1)?u+1 f=I
Gl'l s,,
77 jUj
+
(29.56)
.
j=rnbn-k
'=(f-
The terms in this sum have the decomposition il,k
77
juj
ibnsni-bnsnsi-
=
(19.57)
s(f-l)u+1
tabtj-llu+ltz,,lyI4, where anq .j=j/-llu+lI4, and in which sni j fp ? lmply that bn Esni)a s (0,1). The assumptions, and 17.7, (ibn sr for 6 > 0. Neither each i 1,...,rr, and that bn-1IE (5-ni ni l O 1i f'1 0.1, -1F(5'*3 limsupn&-r,lEsniv ni nor limsupah,l I l exceed whereas l7-nlIE-i-i') I and -1F(5'a/ S*ni are of 0( i The same results apply to Sn ra+1 and bn l Sn*rs4.l the analogous terms corresponding to the residual sum in (29.56). Thps, consider F(Xn(f) 2). Mu lti p lying out the square of (29.56)after substituting (29.57),we have three types of summand'. those involving squares and products of the Sni ((rn+ 1)2 terms), those involving squares and products of the Sn'i ((rn + 1)2 tenns); an d those involving products of Sn'i with Sni (lrn + 1)2 terms). The tenns of the second type are each of Obnn2 ) gy-ls2) n and this block vanishes asymptotically. The terms in the third block (givenibn On)) are -2 of Olbnn ) orn ), and hence this block also vanishes. This leaves the terms of =
=
si
=
-
-jybn
-->
-1-)
vs
=
,4
=
-
-1-)
'l
,)
-
.
,
,
'
-3
=
,
=
-2
=
the tirst type, and this block has the fonn 1
rn
2
+1
5l e' 77 j::z1
ibnsni
?b?
rn
n
=
Fls
-5)
1
ra-l.l
pn
z j=1
+
pjsl ni
)
vn
1 2 rn+
f- l -
.
iqi m)
Elsivj
j.m) ,
(29.58)
-
-
n i=1
p1=1
bn
.
/, applyinj standard summation formulae and taking the limit Noting that rnbnln -P-> Sn where yields (29.55).Thus, according to 29.11, Xn n(f) jP. u -->
=
There is an intimate connection between the generalization from B to Sq and style of the result in 29.6. The latter theorem does not establish the convergence Ltllraj, either to B or to any other limit. of the partial sum sequences Xn In fact there are two distinct possibilities. ln the first, Knln converges to 11-1(?)fOr t e (0 IJ for some n as in (29.49).lf this holds, there is no loss of -1(f)), an d u nder condition 29.6(9 this has the generality in setting Kn = 'the
=
,
implication
,
r?m
FCLTS
vxn 2)
-
for Dependent
Processes
F(x&(n(f))2)
n(/).
.-+
n
..-P-A
489
(29.59)
In other words, Xn #n by 29.11. Example 29.8 is a case in point, for wlzich (XX-) 1+f5 In these cases the convergence of the process n(f) t n to Brownian motion also represented the of the partial sum process (Ak) to #q. as convergence can be exists, and the partial sums On the other hand, it is possible that no such n weak limit, as the following case demonstrates. have no =
.
29.13 Example Let a sequence (UiI have the property Ui
2 2* S i
0 a.s.,
=
<
l
2k+1
k
,
=
0, 1,2,3,...
(0,c 2), otherwise.
= Thus, 1.71 0, 174= U5 U6 Uy 0, &16 U1'7 &31= 0, and so forth. Let ,t''(1& :1 < t f 1, = &ni U.Is I n as before, and put Xn n j. Then, observe that for =
=
=
=
=
=
...
=
Xn Since this
A-n(1t)with probability
=
1 when n
=
2k l for k even,
0 when n
=
2
-
1 for k odd.
-
in the behaviour of Xn is present however large n is, Xn does not possess a limit in distribution. However, let Kn be the integer that satisfies Kn
1(2 77 izu2
2k- 1
j <
s
g2k k s y; ,
=
(29.60)
y/jy
where 1(.) is the indicator function, equal to 1 when i is in the indicated range and 0 otherwise. With this anungement, n counts the actual number of increments in the sum, while Ak(1) counts the nominal number, including the zeros; #n+1(1) A%(1)+ 1 except when #n(1) = 2 lk i n w lch case Ak1(1) 22:+1 The conditions of 29.6 are satisfied with n(f) t, and .Xfn-P- B. u
=
=
,
.
=
Incidentally, since condition 29.649 imposes Kn 1)
ExKsjl) n
one might expect that Knqjln sarily the case.
=
--->
uni y) f=1
E
2
(29.61)
1. The last example shows that this is not neces-
To get multivariate versions of 29.6 and 29.9, as we undertake in the next section, it will be necessary to restate these theorems in a slightly more general form, following the lines of 29.10 and 29.11.
29.14 Corollary Let conditions 29.6(a), (b), (c),(d), and (e) hold, and replace 29.649 by x, for each t e g0,11; as n (f' ) EX%t)l) n n(/) then XKn -P-> B11 The same modifkation in 29.9 leads to the same result. n ..-.#
--
.
The Functional
490
Central Limit Theorem
ne main practical reason why this extension is needed is because we shall wish to specify Kn in advance, rather than tailor it to a particular process', the same choice will need to work for a range of different processes to be precise, for every linear combination of a vector of processes, for each of which a compatible n will need to exist. However, the fact that partial sum processes may converge to limits different from simple Brownian motion may b,eof interest for its own sake, (rl/1)becomes the more appropriate form of the FCLT. so that 29.14 (withKn Theorem below for a case in point. 30.2 See -
=
29.5 The Multivariate Case To extend the FCLT to vector processes requires an approach similar in principle to that of j27.7. However, the results of this chapter have so far been obtained, unlike those of j27, without explicit derivation of the finite-dimensional distributions. It has not been necessary to use the results of j28.4 at any point. Because we have to rely on the Cramr-Wold device to go from univariate to multivariate limits, it is now necessary to consider the finite dimensional sets of D, and indeed to generalize the results of j28.4. This section draws on Phillips and Durlauf (1986). We detineff as the space of v-vectors of cadlag functions, which we endow with the metric
dx,y)
max fdelxj,y?
=
Lsjsm
l
(29.62)
,
where de is the Billingsley metric as before. dk induces the product topology, and the separability of D,ds) implies both separability of Dm,Gland also that B? = BD x Bn x x Bo is the Borel field of Dm,db. Also let Dm: B e Bmk /j,...,fk q (0,1J,k e (N) J'tg (';11 tkB) (29.63) ...
c
=
..'
.
.
.
:,
'
be the finite-dimensional sets of Dm, the field generated from the product of m copies of Ro. The following theorem extends 28.10in a way which closely parallels the extension of 27.6 to 27.16.
29.15 Theorem J'f;
is a determining class for (Dm, S;).
Proof An open sphere in B? is
5'(x,a) (y e =
=
Dm1d%x,y)
y e Dm: H
Define, for
,
G
<
a)
A s.t.
a, max suply//) 11111 <
Lsj<m
(/1...,/ke g0,11,k e
EN )
,
t
-xjhtl)
l
.
(29.64)
for Dependent
FCLTS
491
Processes
max maxfk Iyy(/f)xjlkt)
l<
-
l<./f m 1< f
e R?.
a
(29.65)
It follows by direct generalization of the argument of 28.10 that, for any and r > 0, X
sx,r) Hence, B? i
c(>C)as
(M)
Dm
1
UOHkx,
=
q
.x
c(>t/).
r- 1/n) e
(29.66)
n=1 k=1
required.
w
The following can be thought of as a generic multivariate convergence theorem, in that the weak limit specified need only be a.s. continuous. It is not necessar-
ily Bm.
29.16 Theoremzg Let Xn e W here PX e Cm) 1, iff =
.-2-
Dm be an rn-vector of random elements. Xn X, k'Xn A'# for every tixed with 'A 1. -2-
=
TJ Proof lf xj e D, j 1,...,-, the right, since for t e g0,1),
possesses a left limit and is continuous on
zzlya)
=
X hjxjt + :) limj=1
X hj lim xjt
=
s1o
ir-ko
j=
+ :)
=
X hjxj.
(29.67)
j=?
Hence, x @1,...,x,u)' e D. It follows that A'#n is a random e Dm implies element of D. It is clear similarly that x e Cm implies e C, and hence #('# e c'') 1 s denote the sequence of measures corresponding to To prove sufficiency, let gw A'X n, an d assume g,n = g, Fix /1,...,4 e (0,11,for t'inite k. Noting that ,,.,...,,:( BD for each B e Bk (see 28.10), the projections are '7:-1 #) chD e Rp c measurable and vn. jt,:1: rj,..,,r: j s a measure on (rjkpk) Ajtjaough ayj,...,u is X 1 implies not continuous (see the discussion in j28.2), the stipulation jt (0 vx vx = hence by points have the contin0, an d that the discontinuity n g, X uous mapping theorem (26.13).Since vn i s the p.m. of a k-vector of r.v.s, and A is arbitrary, the Cramr-Wold theorem (25.5)implies that vn = v, where vn the distribution of Xnt3),...,Xn(tk). ptnir-1 r1,.,.,?: i S the p.m. of an vk-vector, Since are arbitrary, the finite dimensional distributions of Xit 'x
=
'x
=
.
.
-1
,
=
,
.
=
-measure
=
,...,/k
Convefge.
To complete the proof of sufficiency, we must show that (g,nlis uniformly tight. Choose A ej, the vector with 1 in position j and 0 elsewhere, to show that Xnj -P-->Xj; this means the marginal p.m.s are unifonnly tight, and so tg,nlis uni-P-.h ' tight by 26.23. Then Xn X by 29.15. formly show necessity, on the other hand, simply apply the continuous mappingtheoTo A'x. . to the continuous functional hlx) =
rem
=
The Functional
492
Central Limit Theorem
Although this is a general result, note the importance of the requirement g,tC) 1. It is easy to devise a counter-example where this condition is violated, in which case convergence fails. =
29.17 Example Suppose g' is the p.m. on (D,fo) which assigns probability 1 to elements x with 0, t < 12 .'t7(/) 1 r k z1 =
.
,
Also, let g. assign probability 1 to elements with
0 t ,
x
< 1a+
=
L .
1 t k z1+ ,
lf Xjn
g, all n, and X1n
-
-
= @,x)w.p.1. gw,then clearly (Xn,X1n) -P-> (#1,2-2)
But X1n XLn is equal w.p.1 to the function in in (D,Js). n -
(28.12),which
does not converge
Now we are ready to state the main result. Let f#n(f1)) denote the family of m x 1 vector transformed-Brownian motionprocesses on g0,l 1,whose members are P defined by a vector of homeomorphisms (n' ) an d a covariance matrix Q #n(Q), the t'inite-dimensionaldistributions of X are jointly m x m). If X with independent increments, zero mean, and Gaussian '
,...,n
-
E(X(X(t)')
=
DHt4D',
nP(/) #(1) where D m xpl has rank p, DD' (2 and H(t) diaglnlt/), ) = Ip. ln other words, the yth element of X may be expressed as a linear combination Xjdjkzk where Z (Z1,...,Zp)' is a vector of independent orocesses lpzx with Zk Snk. With p < m, a singular limit is possible. Note, Z (D -,p) ,with
=
=
....,
=
-
=
-
.
29.18 Theorem Let ( Unij be an array of zero-mean stochastic m-vectors. For an XK increasing, integer-valued right-continuous funct io n Kn ( ) define n = ZY'2(0r.7' , ni, and suppose that (a) For each fixed v-vector A satisfvin/ A' = 1 there exists a scalar array lcnjl and a homeomorphism q .on (0,1) with n (0) 0 and n (1)= 1, such that the conditions of 29.14 hold for the arrays (A 'U ni l and lc'f n ), with s respect to n ' (b) Letting H be defined as above with elements denoting n for the case 1....,#, A ej Uthcolumn of the identity matrix), for j .
-
-
.
,
=
.
.q?
,
=
=
E(XKXKnt)') n
Then
XK l
-P-+
x
-
--+
DHt)D'
as n
--
x.
(29.68)
#n(f1). n
A point already indicated above is that under thes conditions Kn must be the same function for each A, and must satisfy condition 29.6(d) in each case as well as nX(1) = 1 can always be achieved by a renormalization, 29.14(f' ). The conditio
for Dependent
FCLTS
Processes
493
and simply requires that differences in scale of the vector elements be absorbed in the matrix D. Proof Consider first the case m p and D Im. Condition (a) is sfficient under X'XK B1) l where this limit is a.s. continuous, for each The 29* 14 for n K of of distribution follows Xn by 29.16. The fonn of the now convergence thejoint marginal distributions is implied by 29.14, independence of the vector elements following from condition (b). If D # Im, the theorem can be applied to the array ( D'D4-D'Unij for which the limit in (29.68)is Ht) as before. Since linear transformations preserve Gaussianity, the general conclusion follows by the continuous mapping theorem. . =
=
-%
,
.
,
Theorem 29.18 is a highly general result, and the interest lies in establishing how the conditions might come to be satistied in practice. While we permit Kn.) (?T/) to allow cases like 29.13, thse are of relatively small importance, and it will simplify the discussion if it is conducted in terms of the case Kn (rl/1. The K superscript can then be dropped and Xn becomes the vector of ordinary :#:
=
partial sum processes. Even then, the result has considerable generality thanks to the array formulation, and its interpretation requires care. We can invoke the decomposition f2
E + A + A',
=
(29.69)
where E
lim TEltlniun'il,
=
(29.70)
n--)x f=l
h
=
lim
n
i l --
Evnvi-mlzi).
(29.71)
n--yx f=2 rrluzl
But it should be observed that the conditions of 29.18 do not explicitly impose summability of the covariances. While E and A are finite by construction, without summability it would be possible to have E 0. We noted previously that conditiop 29.649 appeared to impose summability, but it remains a conjecture that the more general 29.14(f') must always do the same. This conjecture is supported by the need for a summability condition in 24.6, whose conclusion must hold whenever 29.14 holds for the partial sums, but is yet to be demonstrated. Replacing 29.14(f') with more primitive conditions on the increment processes would be a useful extension of the present results, but would probably be diftkult at the present level of generality. Note that Q, not E, is the covariance of the process, notwithstanding the fact that #a(Q) is a process with independent increments. The condition f2 1, such that the elements of Sn are independent, neither implies nor is implied by the the Uni.While uncorrelatedness at all lags is contemporaneous uncorrelatedness s?f with E I and A 0, it is important to note that when the elements sufticient, and/or with a lag) there always of Uni are related arbitrarily (contemporaneously =
=
=
=
The Functional Central Limit Theorem
494
(D 'D4 -1D' un der which the elements of the limiting one another. did for the scalar As we case, we review some of the simplest sets of sufficient Sn-$*U E(1=jUiU14. For this chice, D 1 is i where Sn conditions.Let Uni unifonnly h, (A n 1 7i/2 If (ti) is fv-bounded, choose cni imposedautomatically. /cX Then A'&nj ni is a linear combination of the Ui with weights summing to 1 and St lpnll'loj/clfllr < x holds for any A, so that conditions (a) and (b) of 29.6 are satisfied. The multivariate analogue of 29.7 is then easily obtained:
existsa linear transformation processare independent of
,
=
=
=
=
,s-
.
29.19CorollaryLet ltil beazero-mean,uniforlyAr-bounded--vectorsequence, 2).,and with each element Q-NED of size on an a-mixing process of size >l-1/2jnj1& -P.-+ x. < asstlme If Xn Xn #(Q). f2 -S then ,= it n u n -r/(r-
-
--->
=
Compare this formulation with (27.82),and as with 29.7, note the important difference from the martingale difference case, with fl taking the place of E. It is also worth reiterating how the statement of conditions is potentially misleading, given that the last one is typically hard to fulfil without a NED size of 1. Somewhat trickier is the case of trending moments, where different elements of the vector may even be subject to different trends. The discussion here will have some close parallels with j24.4. Diagonalize Sn as -
Sn
CnMncn'.
=
(29.72)
where Mn is diagonal of rank m, and Cncn' Cn'cn= Im. Assume, to fix ideas, that Cn C, which can be thought of as imposing a form of global stationarity on the 1/2 z l /2 cross-correlations. Then SnJN Cn an (j. =
--->
=
.
Elx-nxn')
Enfl Illl'l
Mn-IlC'E n
=
X UiTt./: /5 1q l-=
!:::
- 1/2%
= Mn
,,
.1/2
r!
---h
%M-nIl
11
gytj
(gp.gg;
,
where the approximation is got by setting Cn to C, and can be made as good as desired by taking n large enough. The statps of conditions 29.18(a) and (b) must be checked by evaluating the elements of Nin (29.73).An example is the best way to illustrate the possibilities. 29.20 Example Let m
=
2, and assume EUiU1-m)
EUiU1)
=
C
i;$l 0 C 0 i42
=
0 for m # 0, but 1et
,
(29.74)
fort'ixedc. Then, Mn diagtn p1+1n X+1) an d H(t) djagj/l'f'l /2+1j yol. jj 02> 1 /1+1and /2+1are increasing homeomorphisms on the unit square, and =
=
,
,
.
,
-
,
condition 29.18(b) is satisfied. It remains to check 29.18(a). Condition 29.14(f') holds for the array (A'&,,/) with respect to 1) (/)
=
;t2/p1+1 jy 7t2/42+1 1 2 ,
(29.75)
FCLTS
2
+l2 since :,1 2 which, R n (1) 1 and n(0) =
l cni
-1.
=
,
we can check conditions tind that
lt
,
)
495
Processes
1, i s an increasing homeomorphism on the unit square with 0 whenever f31, Assuming that 29.6(b) holds for 02>
=
2 IIA &,,fIl2 1,1
=
Vn
for Dependent
1
I31
=
n
j1+1
+
2 1,2
l/2
i6z
29.6(d) and 29.6(e). The latter holds for
E''e+*1
= y '=(n.1+ l
'y
=
.
We also
:
cni) z
Xl((f + 6)71+1 /71+1) -
=
--> Xl(p1+
(29.76)
#! I,z+l
1)fF1+
+
l((/+ 8)C2+1 /r2+1)
1)f'2 < %(j32+
-
x
(29.77)
is as good as desired with large enough n. satisfied, Condition 29.6(d) is and hence 29.18(a) holds. This completes the verification of the conditions. u
as
where the approximation -- 0,
30 Weak Convergence to Stochastic lntegrals
30.1 Weak Limit Results for Random Functionals The main task of this chapter is to treat an important corollary to the functional central limit theorem: convergence of a particular class of partial sums to a limit distribution which can be identified with a stochastic integral with respect to Brownian motion, or another Gaussian process. But before embarking on this topic, we tirst review another class of results involving integrals, superficially similar to what follows, but actually different and rather more straightforward. There will, in fact, turn out to be an unexpected correspondence in certain cases between the results obtained by each approach, For a probability space (f,T,#), we are familiar with the notion of a measurable
mapping
/': ( where C is Cw,lj as usual. We now want to extend measurability to functionals on C, and especially to integrals. Let F(f) fds: C i- R denote the ordinary =
Riemann integral of
f
over
g0,/q .
30.1 Theorem If f is F/oc-measurable, the composite mapping Fofl
1--.y R
is T/f-measurable for t e (0,11. Proof It is sufticient to show that F(f) is continuous on (C,Jg). This follows ds, g E C, and 0 f t < 1, since, for G(f) Jo't =
lF
-
G(/) I S
' lJ jo
,q
-
lds
S sup If s
-
gs)
I .
w
(30.1)
This shows that F(f) is a random variable for any /. Now, writing F: C F-.y C
as the mapping whose range is the set of functions assuming the values F(/) at t, it can further be shown that F is a new random function whose distribution is uniquely found by extension from the finite-dimensional distributions, just as for f The same reasoning extends to F2(f) j't ds, to X, and so on. =
Other important examples of measurable functionals under du include the extrema, suprl/f) J and inff(J(/)J As a simple example of technique, here is an ingenious argument which shows that if B is standard Brownian motion, suprttf) J has the half-nonnal distribution (see(8.26)).Consider the partial sum process Sn .
Gdtzk Convergence to Stochastic Integrals
497
where the (j are independent binary r.v.s with #((f = I7=l(f, 1a.Straightforward enumeration of the sample space shows
# nnax Si k an
2#(in
=
an4
>
Psn
+
1) that =
=
P(Li
Jn),
=
-1)
=
=
(30.2)
1S f K n
for any an k 0 (see Billingsley 1968: ch. 2. 10). Since this holds for any n, on putting an = Wa the FCLT implies that the limiting case of (30.2)applies to #, in respect of any constant (x k 0. This also defines the limit in distribution of suptxntfll for every process Xn satisfying the conditions of 29.4. This is a neat delonstration of the method of extending a limit result from a special case to a general case, using an invariance principle.
Limit results for the integrals (i.e, sample means) of partial-sum processes, or continuous functions thereof, are obtained by the straightforward method of teaming a functional central limit theorem with the continuous mapping theorem. 0 and Snj 30 2 Theorem Let S'$x() S=1Uni for j 1,...,n 1. If Xnt) #n (see29.11). For any continuous function g: R p-p R, s'n,gpwj, assume that Xn =
=
=
=
-
*',---+
n-1
' sgso
1
J
-p-->
N.
0
(30.3)
gla-jvt.
Proof Fonnally,
gsn
(/+1 )/n
ng)
=
+1)/?1
dt
jIn
n
=
(30.4)
gxntdt.
jIn
Hence, j. n- 1
n- 1
(/+1)/n D-/jjjn '(Xn(f))J/ -n77'(5k) =
1
jonxntdt.
=
/=0
j=
1 t44dt x e c, is a continuous mapping since ogx the continuous mapping theorem (26.13).. by ,
from
c to R, the result
(30.5) follows
Note how gsnn) is omitted from these sums in accordance with the convention that elements of D are right-continuous. Since the limit process is continuous almost surely, its inclusion would change nothing material. These results illustrate the importance of having 29.14 (withKnl as an alternative to 29.6 as a gz7/J) K representation of the invariance principle. The processes Xn(f) are d efined in (0, 1q, and cannot be mapped onto the integers 1,...,n by setting t jln. There is K no obvious way of detining the sample average of gxn) in the manner of (30.3), and for this purpose the partial-sum process Xn with limit Sn has no substitute. The leading cases of g.) include the identity function, and the square. For the y-1y former case, 30.2 should be compared with 29.12. Observe that Z nj ! n-IlU -1 Z2=1(n uni.If Uni y-f/c reversing the order of summation ln 29.12 n-l'k-'l5k -P-> #n(1), for the case effect, that J shows, in n(/) jf3.In other j.1). words, J(/dt N, =
=
.1
=
-
,
=
-
'f'
'f'
.
.
-
.
a
% . - .- -
!
-
-. -
- - - -.
-
-.
!
---
--
..
.-
.-
-
-
!
- - -
1
- ..
- .-
r-
-
A.1-
..
.j?.
..
..
.-.
!
....
...
...
=
The Functional Central Limit Theorem
498
limit for the case g.4 (.)2 T hese limit results do not generally yield closed formulae for the c.d.f., so there are no exact tabulations of the percentag. points such as we have for the Gaussian case. Their main practical value is il letting us know that the limits exist. Applications in statistical inferenc, usually involve estimating the percentiles of the distributions by Monte Carll simulation', in other words, tabulating random? variates generated as the average of large but finite samples of g evaluated at a Gaussian drawing, to approximab integrals of #(#n). Knowledgeof the weakconvergence assures us thatsuch approx imations can be made as close as desired by taking n large enough. Given a basic repertoire of limit results, it is not difficult to find th( distributions of other limit processes and random variables in the same manner. T( take a simple case, if ( Uij is a sequence with constant variance c2, an( .-P-h Bt) where 11.-111g% we can deduce from the continuou: (nrj/c s'knrj t;:(1&j, the partial of the sample mean deviations converge tl theorem mapping that sums the Brownian bridge', i.e., =
.
=
LX'f'1
1 1/2
Gn
here'Un
X /=1
Ui
-P-> B
l
-
-
/#(1)
=
Bo,
(30.6
?z-1'3
qUi. On the other hand, if we express the partial sum proces! rl-1Z6-15' J=0 jn we find convergenct itse% in mean deviations, Sj L where L
W
=
j=
=
-
according to
-jlss.
'
Gn
-2-
1/2
s-o
(?)
--)
0
The limit process on the right-hand side of (30.7)is the de-meaned Browni motion. One must be careful to distinguish the last two cases. The integral of th( latter over (0,11is identically zero. The mean square of the mean deviation! converges similarly, according to
j
s-1
psaXy.sj
-
n)2
1 joBlds
.-0-+
-
1 joBds
2
(30.8'
.
J
There is also an easy generalization of these results to vector processes. The following is the vector counterpart of the leading cases of 30.2, the details ol whose proof the reader can readily supply. 30 3 Corollary Let ( Unil satisfy the conditions of 29.18. lf Snj S=1I&j, ther =
n-1
1 Tsnj - /=1 1
n-1
---y
o
- /=1/snjsnj n
,
---
1
o
(30.9)
Bvdt,
1
,
BlBldt. c
n
(30.10)
W'ctzkConvergence to Stochastic Integrals
Note in particular that for #, the m-dimensional d t N( 0 ,1Im) J()#l
499
standard Brownian motion,
-
,
.
The same approach of applying the continuous mapping theorem yields an important result involving the product of the partial-sum process with its increment. The limits obtained do not appear at first sight to involve stochastic integrals, although there will turn out to be an intimate connection.
30.4 Theorem Let the assumptions of 30.2 hold, with n-1
.--AD 1
2( (z 1)
X1 SnjU
n, j+ 1
12
jzzz W
-
n(1)
=
1. Then
/2 )
(30. 11)
,
lims-p-n-llr,-lcznj.
0.1- =
here
Proof Lett ing Snj
)=1 Uni
=
2 Sn,j+3 Summing from 0 to n
2
zsnj s
Snj+
=
l
-
g2n,./+1.
(30.12)
0, yields
=
n-1
2
Sn,j+j Snjt
=
+
,,,./+1
1, setting Sn0
-
n-1
l
Snn
Snvj-j + Unj, note the identity
=
n
')
=
s
-
j=1
./=0 n- j =
n
j=1
2
Unj
-
r
(30.13)
.j=1
2 1 Snn
Snjun.j-vk
(/2nj,
na;y n,af+I +
(30.14)
.
./=1
.T-r.-> The result N@, 1) and lzzjunli Under the assumptions, snn-P-+ #n(1) followson applying the continuous mapping theorem and 22.14(i). w .1.
-
This is an unexpectedly universal result, for it actually does not depend on the FCLT at al1 for its validity. It is true so long as ( Uiit satisfies the conditions i -EU n, i-muni), the left-hand for a CLT. Since W2 1 lh where limn-yxZ2=2Z,?,=1 side of (30.11) has a mean of zero in the limit if and only if the sequence ( Unit is serially uncorrelated. There is again a generalization to the vector case, although only in a and then generalizing (30.12)we have the restricted sense. Let Snj =
-
,
=
identity
Sn
./+1u%'
:
1
summingand
j+1
=
=
S=1&nf,
Snjsn'j + Snjun'
./+1
'
Un,j-vjsn'j+ &n,./+1&n',./+1.
+
(30.15)
taking limits in the same manner as before leads to the following
result.
30.5 Theorem Let ( Unil satisfy the conditions of 29.18. Then .-2-A #c(1)#n(1)'-E + Yun,j,vksn'j Tsnjun',j-v j=3
7=l
./(4121#()1()'
Details
of the oroof
are left to the reader.
-
Th
E.
::1
(30.16)
oeculiaritv of this result is
The Functional Central Limit Theorem
500
that it does not lead to a limiting distribution for the stochastic matrix -1ZJ C'-'Snj U'n,.j+l This must be obtained by an entirely different approach, which n is explored in j30.4. -1
.
30.2 Stochastic Processes in Continuous Time To understand how stochastic integrals are cofkstructed requires some additional theory for continuous stochastic processes on (0,11.Much of this material is a natural analogue of the results for random sequences studied in Pal't 111. A yltration is a collection (@(/),t e g0,1J ) of c-subfields of events in a complete probability space (L-j,5,P) with the property
s The tiltration f 5t)
T(J)
c
when t S s.
said to be right-continuous
J is
st)
@(/+)
=
(30.17) if
OF(.).
=
(30.18)
s>t
A stochastic process X (X(/), t e (0,1)) is said to be adapted to ( 5(t4 ) if X is Tt/l-measurable for each t (comparej 15.1). Note that right-continuity of the tiltration is not the same thing as right-continuity of X, but if X e D (which will be the case in all our examples) adaptation of X(t) to 5(t) implies adaptation to T(/+) and there is typically no loss of generality in assuming (30.18). A stronger notion of measurability is needed for defining stochastic integrals of the X process. (X(/) ) is said to be progressively measurable with respect to ( T(/)J if the mappings =
X(.,.): f x
F.-: R r0,/1
are T(/) () tRgtl,fj/f-measurable,for each t e (0,1). Every progressively measurable consider the rectangles E x g0, for E e 5t)) but the process is adapted converse is not always true; with arbitrary functions, measurability problems can arise. However, we do have the following result. 'ust
30.6 Theorem
An adapted cadlag process is progressively measurable.
Proof For an adapted process X e D and any t e (0,1j, define the simple process on (0,/1: X(,,)(,-)
=
A),
2-*/c), s e
(2-''W 1), -
2-*/c), k
=
1,...,(2''/J,
(30.19)
Xnt). Xn) need not be adapted, but it is a right-continuous with Xnllttl function on f x (0,/j. lf A1 (): 2(), 2-Q) S e 5t), then =
.x)
=
Xx
t(,J):
=
= a finite union
is true for each
.x
q
A%)(),.) S
U
g2-N(k
k
-
.Yl
1), 2-lk) xf'l
t.p l/) xAn'jzafjx.l
(30.20)
of measurable rectangles, and so Ax e T(/) (&tBp,lj. This is and s, R, and hence Xn) is 5(t) (8)fm,rj/f-measurable. Fix
Weak Convergence to Stochastic Integrals
501
and note that for each n X(n)(),J)
=
X(,If),
(30.21)
X(),J) x. Since Xnu) by right-continuity, it where u > J, and u 1. s as n X(,J) everywhere on f x E0,/1and hence X is follows that X(n)(),.) st) (&fm,fj/f-measurable (apply3.26). This holds for any f, and the theorem follows. w -->
---
-.-:
Since we are dealing with time as acontinuum, we can think of the momentat which some event in the evolution of X occurs as a real random variable. For example, the first time #(fl exceeds some positive constant M in absolute value is F())
inf
=
te
g0,IJ
(f:
X(),f)
I > MI.
(30.22)
time of the filtration ( ;4/), t e (0,1) ) if ( ): F() ; /) E 5(t) (comparej15.2). It is a simple exercise to show that, if X is progresF XF(/) Xt A F). sively measurable, so is the stopped process X w here Let X e D, and let X(f) be an Ttfl-measurable r.v. for each t E g0,1J. The adapted pair (X(/),9;(f) ) is said to be a martingale in continuous time if F()
is called a stopping
=
E lX 1 sup f
IF(/))
Exs)
(30.23)
X(t) a.s.(#), 0 S t K s K 1.
=
(30.24)
lt is called a semimartingale (sub-or super-) if (30.23)plus one of the inequalities .(kj
s(x(,,)I@(,))'
x(,) a.s.zq, o < t u s s
(3o.2s)
1
hold. One way to generate a continuous-time martingale is by mapping a discretetime martingale fSp5jI1into g0,1j, rather in the manner of (27.55)and (27.56). simple function which jumps lf we let X(f) 5'gn/j+l,this is a right-continuous and the at the poilgs where En/l nt. lt is Tt/l-measurable where T(/) Fgnf!+1, collection l@(f),0 S t < 1 J is right-continuous. Properties of the martingale can often be generalized from the discrete case. =
=
=
The following result extends the maximal
30.7 Theorem Let ((X(,5(t))
(i) #
sup (E
(ii) E
1#4J)
l>
g(),f1
IP
sup 1.Ytul (E ((),/1 -:
:
t <
E
inequalities of 15.14 and 15.15.
(0,1) ) be
A'lX
zp
1P
,
p
a mmingale.
Then
1, (Kolmogorov inequality).
(,#j)PsIx(/)I,,p
>
1, (Doob inequality).
Proof These inequalities hold if they hold for the supremum oter the interval E0,t),noting in (i) that the case s t is just the Chebyshev inequtity. Given a =
502
The Functional
Central Limit Theorem
define a continuous martingale Xn) discrete mmingale fSk,5kI=twith m g2''/q, '(2n,)+I for s e on g0,/1as in the previous paragraph, by setting Xnls) inequalities hold with X(n)(f-) The 15.14 Xnj for Xn) by g0,f), and 15.15, noting that =
=
=
,%2nr).
=
IA)a)(.) sup (0,8
IF
Is'klr
max
=
lKkfrzl
JG
(30.26)
for p 1. Now, given an arbitrary continuous martingale martingale is defined by setting sknskj
(X(2-Q), F(2-&k)), k
=
=
(X(/), T(/)), a discrete
1,...,E2't/1.
(30.27)
For this case we have X(n)(,)= X(u) for u 2-''(L2'' + 1), so that u 1 s as n X(n)(J) Xsj for s E (0,/), by right continuity. . x. Hence
-->
=
-->
The class of martingale processes we shall be mainly concerned with satisfy two extra conditions: almost sure continuity (PCX e C) 1), and square integrability. A martingale X is said to be square-integrable if E(X i ) < oo for each t G g0,1q. For such processes, the inequality =
F(x(.)2I5tl)
xl
=
-x(f)j2j
F(gx(-)
+
y(f)) k
x(f)2
(g().28)
holds a.s.(#q for s 2 t in view of (30.24),and it follows that X2 i s a su bmartingale. The Doob-MayerbM) decompositionof an integrable submartingale, when it exists, is the unique decomposition X(t4
(30.29)
Mt) +A(/),
=
where M is a martingale and A an integrable increasing process. The DM decomposition has been shown to exist, with M unifonnly integrable, if the set fX(/), t e 7') is unifonnly integrable, where 5 denotes the set of stopping times of (@(f)) (see e.g. Karatzas and Shreve 1988: th. 4.10). ln particular, suppose there exists for a mmingale (X(/),T(l)) an increasing, adapted stochastic process ((A')(/), 5 l on g0,11, whose conditionally expected variations match those of Xa almost surely; that is,
f
((X('V)
l7(/))
(A')(J)
-
=
5t F(X(.)2I
-
X(/)2
a.s.g#J
(30.30)
for s k t. Rearranging (30.30)gives Exls)
-
(A')(.) l@(/))
=
Xl
-
(A')(f), a.s.(#J,
(30.31)
which shows that fX(/) 2 (A')(/), @(f)) is a martingale, and this process accordingly defines the DM decomposition of X2 An increasing adapted process -
.
((A')(f),@(/) l satisfying
quadratic variation
(30.30),which
process of X.
is unique a.s. if it exists, is called the
30.8 Example The Brownian motion process # is a square-integrable martingale with respect to the filtration 5t) = c(#@), s < t). The martingale property is an obvious consequence of the independence of the increments of B. A special feature of B is that the quadratic variation process is deterministic. Definition
Wctzk Convergence to Stochastic Integrals
503
27.7 implies that, for s k t,
f'(#(.)2IF(f))
-
B(t)l
=
E((B(s)
=x of the equality
and rearrangement (#)(/) t. Q
-
#(4j2j @(/))
-
t, a.s.j#),
(30.32)
shows that #(/) 2 t is a martingale; that is, -
=
Two additional pieces of terminology often arise in this context. process is an adapted process (X(f),@(f)) having the property Pxt
-)
+
e
yt
IT(/))
=
Pxt
+
e A lc(A/))
.)
a.s.g#l
A Markov
(30.33)
for 4 e B and t, s k 0. This means that all the information capable of predicting the future path of a Markov process is contained in its current realized value. A dlffusionprocess is a Markov process having continuous sample paths. The sample paths of a diffusion process must be describable in terms of a stochastic mechnism generating infinitesimal increments, although these need not be independent or identically distribted, nor for that matter Gaussian. A Brownian motion, however, is both a Markov process and a diffusion process. l&e shall not pursue these generalizations very far, but works such as Cox and Miller (1965)or Karatzas and Shreve (1988)might be consulted for further details.
The family Sn defined in (29.49)are diffusion processes. They are also martingales, and it is easy to verify that in this case (#n) n. However, a diffusion process need not be a martingale. An example with increments that are Gaussian but not independent is X(f) 0(f)#(/) (see 27.9). Observe that =
=
Elxt
+
.) -
X
I@(p) (0(f+ s4 =
tljB
-
(0((+,)') 1)
-
-
x(,) .o,
a.s.g#q.
(30.34)
A larger class of diffusion processes is defined by the scheme X = 0(f)#n(/), for eligible choices of 0 and n. The Ornstein-uhlenbeck process (27.10)is another example. li owever, the class B is the only one we shall be concerned with here.
n
30.3 Stochastic lntegrals ln this section we introduce a class of stochastic integrals on (0,1j. Let (M'(f),T(f)) denote a martingale having a deterministic quadratic variation process (M). For a function J e D, satisfying a prescribed set of properties to be detailed below, a stochastic process on g0,1j will be represented by Innt)
=
' jo
J(,I)#M(,I),
E0,11,
(30.35)
IJtdM. The notation corresponds, for fixed to more compactly written as I what we would use for the Riemann-stieljtes integral of over g0,/Jwith appreciate is for almost every M(%.). However, it important that, to respect to =
,
.f(,.)
The Functional Central Limit Theorem
504
this Riemann-stieljtes
integral does not exist; quite simply,
have not
we , required M((l),.) to be of bounded variation, and the example of Brownian motion Hence, a different shows that this requirement could fail for almost all interpretation of the process I is called for. .
The results we shall obtain are actually available for a larger class of integrator functions, including martingales whose quadratic variation is a stochastic process. However, it is substantially easier to prove the existence of the integral for the case indicated, and this covers the applications of interest to us. We assume the existence of a filtration (@(/),t e E0,11J, on a probability space (f1,T,#). Let
(x: (0,1J F-- R 1, with no 0 and a(1) be a positive, increasing element of D, and let G(0) restriction generality of (x to it of 11, For aj loss turns out. any t c (0, the (0,/1 induces a finite Lebesgue-stieltjes measure. That is to say, a is a c.d.f., and the function Jslat-lassigns a measure to each # e fm,/j. Accordingly we can define on the product space ( x (0,/1, 5(t4 @fgt),rj)the product measure gu, where =
=
A) yttxt
=
Jnjt f)
1x(),.)#a(.)J#())
=
E
A e @(/)(&fgtl ,j. Let La denote the class of functions
for each
J'
(30.36)
1x(,.)#G(.)
0
.f: ('1 >--> S(0,11
whichare (a) progressively measurable (andhence adapted to (@(/))),and (b) where < with respect to pu; that is to say, 11.f1) square-integrable cxa
11 11f =
1
1/2
jof
E
ld
(30.37)
It is then easy to verify that 11f is a pseudo-metric on La. nile 11.fg 1i 0 gl for every (0 e f1, it does imply that the does not guarantee that fll integrals of f and g with respect to a Will be equal almost surely g#j. In this case we call functions f and g equivalent. The chief technical result we need is to show that a class of simple functions for t e is dense in La. Let Ea c La denote the class such that J(f) ft = 0,...,1 and f(1) f41-), where f/1,...,/,,;) H,u is a partition (/s4+1), k for of (0,11 sonne sv e :. -:11
-
=
=
=
=
=
-
30.9 Lemma (afterKopp 1984) For each f e La, there exists a sequence (J(r,)e Ea, x. 0 as n n c s ) with ll/y) .f11
-->
-
--
Proof Let the domain of square-integrability,
+-
honno
f be
extended to R by setting < cxa a.s.(#1, and
J1:/(0,Y#q(/)
J-.(.f(,f )
.f(t9,f))
-
ldg
--.,
o a.s.rzr
f(f)
=
as h
0 for f
--+
0;
e g0,1).By
W-etz/cConvergence to Stochastic Integrals += lim E j..(J@
+
h4
o
flfds)
-
505
(30.38)
0
=
h.-O
by theboundedconvergence theorem. This holdsforany sequenceofpoints going to x. and t e g0,11, 0, so, given a partition H,u(n)sueh that 11H,,0)11 0 as n /, where consider the case h kn -->
--
=
kn Clearly, knl
j+-s(j' -x
-
e If, h+1), i
ff, f
=
lk(1)
0,...,v
=
1,
-
(30.39) (30.40)
/,,,-1.
=
(30.38)implies
Hence,
/.
-->
-
-fs-vkno)
gs
-
1
that
+/))2#a(f))Ja(,)
?.(,+,))ua(,)):a(,) s j' - (j+-(y(,+/o(,)) -
-1
-x
s(j+-(y(,+1-(,) 'f-1
-
-,)
-
y(s))2Ja(,)
dzo
--
-
-- 0 as n
(30.41)
--)
x,
where the first equality is an application of Fubini's theorem. Since the inner integral on the left-hand side is non-negative, (30.41)implies lim E n-x
l fs j.j
for almost all s e R. Fixing t s gives
+
J'
knb
fs
-
+
/))2Ja(f)
=
(30.42)
0
e r0,11 and making a change of variable from t to
-
1+J
js-jft-
1imE
lnt4)
n--x
where ln
=
knt
-
J)
-
fn4l
-
J(/))2Ja(/)
=
0,
(30.43)
t s4. Detine a function -
=
ft-
ln(f)),
/+
ln
e 19,11
(30.44)
otherwise,
0
noting that hnj J(/f+ for f e Eh+s, /s1 + J) f'-h g0,11 and hence hnj Jj, e E(t. Given (30.43),the proof is completed by noting that g0,1Jc g, 1, 1 + and hence .)
=
-
-.f(,))2Ja(/))
s(j-'(.f.)(f)
-
s(J'--(/(-)(,)-y(,))2#a(/)) J-1
0
s
s(j'+-(y(/+ x-1
-.f(/))2Ja(,))
&(,))
.
(30.45)
Central Limit Theorem
The Functional
508
'sls zkz2(1)1).
(30.54)
-
-
o
1 and compare this with 30.4. It is apparent (and will be proved rigorbelow) that the limit in (30.11) can be expressed as IoBdB + X ously in 30.13 1 1 d2) z( as before. n where Put t
=
,
=
-
,
A form of It's rule holds for a general class of continuous semi-martingales. The proof of the general result is lengthy (seefor example Karatzas and Shreve 1988 or McKean 1969 for details) and we will give the proof just for the case of 30.11, to avoid complications with the possible unboundedness of g''. However, there is little extra difficulty in extending from ordinary Brownian motion to the class of diffusion processes Sn.
j0
30.12 Theorem
B dBv n
r1(#q(/)2q(j)), a.s.
=
-
Proof Let IL denote the partition of g0,/1in which q. tjln for j Taylor expansions to second order to obtain the identity =
=
1,...,n. Use
n-1
#n (f)2
=
F-)(#n(j+j)2B)ltjlll -
j:,zft n-1
=
n-1
277#c(/./)(#n(/./+l)
77(#c(J./+1)
-#n(/./))
+
-
#n(J./))2.
(30.55)
y=0
j=
We show the f,z convergence of each of the sums in the right-hand member. Ln, so detine pn e En by
pn
=
#c(J./),
0,...,n j L/./,#+1),
s e
=
(30.56)
1,
-
#1) e
#n(J) for t < s S 1. This is a construction similar to that used in and pns) x. the proof of 30.9, and llpnAn 11 0 as n we may write =
.--y
-->
-
n- 1
rl- 1
t;+L '
77#n(/./)(#c(/./+l) 77 -
#n(/;))
y)
b
.j=0
-
and it follows that n-1
E
#c(#)J#n(-)
=
2
(t
#n(fy)) X#c(tJ(#n(#+1) -
-
j=0
',-(-)#s.(.),
oBdB,
J
=
E
l
:t
Jopn
-
fnl#fn
Il4,n-#n)1m,r,II20. -->
=
(30.57)
(30.58)
Considering the second sum on the right-hand side of (30.55),we have n-1
E
2
-#c(/./))2
77(#c(+l)
-
n(J)
WretzkConvergence to Stochastic Integrals n-1
2
#c(#))2
E E(#c(/sl) = 77 j= -
n-1
-
(n(/./+1) n(#))2 -
-#n(/./))2
F/E
509
-n(/y))12
= j=0 ((#c(f./+1)
-
(n(/.j+1)
n-l
= 77(n(#+1)n(f./))2 2
s <
-
2nq)
max 0 %s n-
0 as n
-->
fn(/s1) ntf./l ) l -
(30.59)
-->
x.
The second equality here is due to the fact that the cross-products disappear in expectation, thanks to the law of iterated expectations and the fact that ELB(b+j)
#f:/7,)()21
?77(p)(I n(/sl) n(/j).
-
=
(30.60)
-
The third equality applies the Gaussianity of the increments, together vvzith9.7 for the fourth moments, and the inequality uses the continuity of and the fact n that IlHnll 0. Thus,#n(/) 2can be decomposedas thesumof sequencesconverging infu-nonm to, lotBjdBj 18.6, and according L1 However, respectively, to convergence n(/). implies copvergence with probability 1 on a subsequence (nkJ.Since the choice of partitions is arbitrary so long as IlHnkII 0, the theorem follows. . -->
--->
The special step in this result is of course (30.59).ln a continuous function of bounded variation, the sum of the squared increments is dominated by the largest increment and so must vanish by continuity, just as happens with n(/)in the last line of the expression. It is because it is unbounded a.s. that the same sort of thing does not happen with #n.
30.4 Convergence to Stochastic Integrals
J=ntl(%
and l',,(f) and l JI;,) be a pir of stochastic arrays, let Xn (n(1
f Unjj
=
,
=
Gn
n-1
j
Uni
=
j=
=
f=1
1L,j+1
The Functional Central flrrlf/ Theorem
510
n-1
= j=$/xnlilnqrn''' 1)/n) FnU/f)).
(30.61)
-
This problem differs from those of j30.1 because it cannot be deduced merely from combining thefunctional CLT with the continuous mapping theorem. None the less, it is possible to show that the convergence holds unde? the conditions of 29.14. The following theorem generalizes one given by Chan and Wei (1988).See inter alia Strasser (1986),Phillips (1988),Kurtz and Protter (1991),Hansen (1992c), for alternative approaches to this type of restllt.
30.13 Theorem Let ( UnpWnjkbe a (2x 1) stochastic array satisfying the conditions of 29.18 for the case Kn g>l/).ln addition, assume that both Unj and =
Wk are Q-NED of size
(Fa)
-1
on
.
Then
axs ca-p-:jlpxdnv
(30.62)
+
0
where, with hnjk= EunjWn, Axy
lim
=
n- 1 i- 1 f=1
n-jx
h.aj-,,,,j+l
.
(30.63)
n
pncu'o
An admissable case here is Unj Bk, in which case the relevant joint distribution is singular. Setting the AZ-NED size at ensures that the covariances are summable in the sense of 17.7. This strengthening of the conditions of 29.18 is typically only nominal, in the light of the discussions that follow 29.6 and 29.18. However, be careful to see that summability is not required to ensure that lAxyl < x, whieh holds under the conditions of 29.18 merely by choice of normalization. Its role will become apparent in the course of the proof. =
-1
Proof The main ingredient of tllis proof is the' Skorokhod representation theorem, 26.25, which at crucial steps in the argument allows us to deduce weak convergence from the a.s. convergence of a random sequence, and vice versa. Let Xn,Yn4be an (D2,#s2) (see j29.5). Since element of the separable, complete metric space (Xn,l%) '-C.'> Bx,BY4
(30.64)
by 29.18, Skorokhod ' s th eorem im p lies the existence of a sequence ((X'',F&) e 172 is distributed like (.Y'',F'), and Js2((#1,Fl), (#x,#y)) n e K J such that xnnj'n) --F-&'A 0. According to Egoroff's theorem (18.4)and the equivalence of ds and Js in 1 E, D, (30.64)implies that, for a set Ce e 5 with PCe) ,
-
JJ2((r((o),I'''((,))),
sup Ce
(#x((o),#y((o))) --+
O
(30.65)
e
1 for each ir > 0. Since Bx is a.s. continuous, there exists a set Ex with #(Fx) f.tl > is if the following then > and for any property; e Fx, n 0 there a constant 8 =
n
evlt..la
that
if dctxnj-Bajjj
K 8.
Ftz/c Convergence to Stochastic Integrals 1A'n(,/) sup t
1S
Bxtb,tl
-
511
I
sup 1A'''((0,/) B/ln,ktjl -
J
IB/,tl + sup f
l
B/,klttl
-
< 8+n,
(30.66)
where 14.)) is the function from (28.7).The same result holds for F in respect of 1. It follows from (30.65)that, for (1 e C1 Cz f''h Ex f-h Ey, a set Ev with PEy) =
=
dl((A'''(t,)),F''()),
(#'(t,)),#i'(t')))) =
(30,67)
0,
--
,,
Note too that PC* P(C. For each member of an increasing integer subsequence flk, n e IN) choose an ordered subset (n1,n2,...,n) of the integers 1,...,n, with nkn n, such that x. minlsjxkaln./ nj-j ) Use these sets to define partitions of (0,1J,Fln /1,...,/u) where (j njln. Assume that fkn) is increasing slowly enough that f 2 --> 0 an d knIn kn&n 0 but note that provided kn 1- x it is always possible to will satisfy these conditions. have IlIX11 0. For example, choosing nj = Lnkjlknj The main steps to be taken are now basically two. Define
where the equality defines
=
n.
,
=
--y
=
-
=
,
-->
,
---
kn
Gn* 77.Xk(f./-1)(1&(/./) Fn(/./-1)), =
(30.68)
-
j=1
let G*nrepresent the yame expression except that the Skorokhod vaiables A''' and F'' are substituted for Xn and l%. ln view of 22.18 and the fact that G*n .-C-yLBxdBv it will suffice to and Gn*have the same distribution, to establish G1
andalso
prove that
G*n
1
jc
BxdBv -PC->0.
-
(30.69)
in view of 22.14(i), by showing
The proof will then be completed,
Axy. Gn Gn*--P-r.-4
(30.70)
-
The Cauchy-schwartz
inequality and
(30.67)give,
for each
)
e
C,
2,
kn
77(A'''(to,//-1)
-
#x(,/./-l))(l'n(,/./)
-
Ynkb,tj-tll
j=1
kn
kn
f
7(X''(,/./-1)
-
#x(?/./-1))277 (F'),/./)
-
1M(),/./-1))2
/=1
/=1
kn
< knn:2 7)(Fn((l),/j)
-
F''(,j-1))2.
(30.71)
j=3
Also the assumptions on Fn, and equivalence of the distributions, imply that
512
Central Limit Theorem
Ihe Functional
Evntp
2
n
ra(fj-1))2 E
Iyuf
=
-
t'=nj- 1+ 1
-->ni'(Q
y(tJ'-
n
-
and hence from (30.71),
l
)
<
l
k
n
E
(30.72)
cs
(A'''(f./-1)-Bxtj-jqqYnt 7-7 j=3
l'n(f./-l))1c)
-
(30.73)
Closely similar arguments give 2
k
n
BytjstBxb) F7(1'a(f./)
E
-
#x(f./-l))1cl
-
(30.74)
/=l
and also -#y(1))1cp)2
f(OX1)(FX(1)
f
(30.75)
0.
-->
We now use the method of summation by parts' ; given arbitrary real numbers 1,...,/c1 with ao = bo = co = fstl 0, we have the identity faj,bj,aj,b, j =
=
.
k
k
ajj=1
k
=
j=1
1(d7./
bj- 1)
-
xj- 1($
-
f- 1)
-
./=l
k
(tn-1 -
j-tbj
Y-1)
aklbk
+
-
-
X)
a./-1). (bj 1$)((x./
-
-
j=j
F''(,f.j), Put k kn, aj = Xnb,tp, bj j #x(,fJ, and I the left-hand side of (30.76)corresponds to G*n Pn, where =
=
=
(30.76)
-
=
Byl,tj).
Then
-
kn
Pn
=
T'-Bxftj-llByt?
#y(f./-1))
-
j=
kn q
= 7) j=1
'-
Bxjh-jldByt),
(30.77)
1
and the squares of the right-hand side terms correspond to te integrands in (30.73), (30.74),and (30.75).Since 6: is arbitrary, #4C) can be set arbitrarily close to 1, so that each of these terms vanishes in fu-norm. We may conclude that -fJl+ l G*n Pn l 0. So, to get (30.69),it suffices to show that IJ'n J(/xdBvf -il+ 0. But -
E Pn
-
-
1 joBxdBy
kn
2 =
E
X
j= 1
tj
f
*'
tj- 1
l
(B/tj-jj
-
BxljdByt)
Weck Convergence to Stochastic Integrals
kn
rfy
(n-Xf)nX(f./-l))Jn'k) 1 =77 1 jzzl -
*'
tj-
lnX(f./)
0 nX(/j-1)lY(1)
(30.78)
--->
max
-
1<.j< kn
wherethe second equality applies X(30.51)and then Fubini's theorem, and the convergenceis by continuity of n This completes the proof of (30.69). To show (30.70),we use the fact that .
l1j.--1
vntj) 1-a(/./-1) 77(I'n((f + =
-
icnj.
1)/n)
Ynilnq),
-
j
and so nj- 1
kn
Gn Gn' -
=
7-7 izmj-L 77xnilnjYni
+
1)/rl) Ynilnj) -
j=
Xnb-qYnt?
kn
nj- 1
77 xnilnb--n(f./-1))(Fn((+ =77 i=nj-L
1)/n)
-
Fn(f./-1))
Ynilnbb
-
j=
kn
nj- 1 i-nj-L
Unj-s;Nn,j+l
= 7'=1
kn
icmj-k rtl::::o ny-ny-l
-
1 nj- 1
Un j-m
= mzu.l
J'=1
(30.79)
,
i=m-nj-L
wherewe have formally set Uno 0. The final equality represents the shift from summingthe elements of a triangular array by rows to summing by diagonals, and weuse whichever of the two versions of the expression is most convenient. ln view of (30.63),Gn Gn* Axy As Bn, where (summingby diagonals) =
-
kn
Xn
=
-
rly-nsl-
-
1 nj- 1
(Un,--Vn,f+1
=
./=1
,n=0
-
Xn,-rn,f+1)
(30.80)
izzm-nj-b
and (summingby rows) Bn
kn
rj-l
/-1
=
n j-w,j+l J.=1
izznj-k
(30.8 1)
.
=f-a./-1+1
0. The problem is to show that both An .-F-r-> 0 and Bn choose a finite integer N and break up An intp N+ 1 additive components AN n,...,Anx, where by taking n large enough that N ; minlspxkafnj zn-l ) we -->
-
The Functional Central Limit Theorem
514 can write Anm
kn
ny- 1
un,i-mWn,i- Xn,f--,f+l),
=
(30.82)
-
J.=L i=m-vnj-t
for m
=
0,..'.,N
-
1, and nj nj- 1 1
kn
Anx
=
-
l
nj-
-
77 )q /=1 m=N
(&n-,?,W%,f+l
n,f-m,,.+l)
-
(30.83)
.
izzm-nj-L
For fixed finite m, the process
with respect to is, according to 17.11 and 17.6, an Ll-mixingale an-ay of size B' fcU (cW & the constant array fncnvi-mcn ), w here ni J and ni J are the constants = specified by 29.18 for A (1,0)' and (0,1)' respectively. We next show that the conditions of 19.11 are satisfied by these terms, so that -1
,+1
+ Anm
n-1 x-x
=
7 l(Jn .
-,,,#p'n,f+l
x
j-n,i-m,i+3.3.
-
. ---+
oj
(30.84)
() .
f=-+1
First, for r
(29.9),
>
2 in the a-mixing case
Uni-mWn,i-vl
supf fn
& Cn,i-mcn
,+1
,
&
Cn
in S 2 r/2sup i' n
<
#-mixingcase
Un -,nGn,f+l r/2
f 2 r/2sup
B,
2 in the
for r
r/2
,t,f-,,,,/+1
-
(29.6)or
-r,;crl
1F
,/+1
Un i-m r/2 Vn,f+l
r/2
,
U Cn,f-r?i
Wr Cn
,/+1
r
r
(30.85)
x,
where the first inequality maks use successively of Love's cr inequality and inequality, and the Jensen's inequality, the second one is by the Cachy-schwartz finiteness is because the aaays satisfy either 29.6(b) or 29.9(b') by assumption. ln the latter case, note that the assumptions include unifonn square-integrability. Therefore the array Unf-,,,Wrn,f+1 -
1
n,-r,,,f+f
U Wr Cn,i-mcn ,+l
is unifonnly integrable in either case, and condition 19.11(a) is let. U (cV Next, the arrays fcajl an d n i ) satisfy condition 29.6(d) by assumption, which by the Cauchy-schwndz inequality implies that
Wetzk Convergence to Stochastic Integrals
IG
Setting t
=
limsup 1
sup
(0,l), e
ga(?+)J-1
-
(0,1
0 and 8
n--x
<
(30.86)
x.
(30.86)gives
1 in
=
u v ca,f-vcn.sl
i=(n/1+?1+l
-?1
n-l
7limsup rl--yx
c&af-puc''z,f..l
(30.87)
f=??1+l
19.11(b), whereas setting 8
which is condition
1/n gives
=
fcui-mcwn l ojln),
maX
(30.88)
=
d+l
n
v+1
and (30.87)and (30.88)together imply n-1
2
W'
U
,sl)
cn,i-mcn /=z,,,+1 * which is condition 19.11(c). So Anm to (30.82),
--l-y
.
(30.89)
0,
--
0 is proved. But for m 2 1, according
,,r+ns1-1
A'IA';,, -A,QIS
kn
FIUn,i-mWn,i-vb z,f-,?,,f+1 I
(30.90)
-
,
j=3
fxn./-l
= oknln), --t+
L
0 a lso holds where the order of magnitude is by (30.85)and (30.88),so Anm S 2 Il,,,f-,,,,s1l Un,i-mWn,i-vj n,f-,,,,sl I 0,...,1- 1. Similarly 'for each m and applying 17.7 yields
,
.81
=
-
kn n.j-nj-l
FlAnxl
o
=
=
1
-
X 77 j=1
n.j- 1
B?
U
,+1(-+1
(30.91)
C rl,/-rzlC n
m=N
,
i=m-nj-.k
o(v)
for some > 0, where the order of magnitude follows by a combination of (30.87) according to the mixing and with the fact that the sequence ((,,1 is of size -1,
Q-NED size assumptions. Thus, limn-oxflAa ISlimn-oxFlzlnxl whichby takingN large enough can be made as close to 0 as desired. In the same manner, recalling ?n-1 l N f minlusulrj ,
-
,
IBn I
=
o
nj- 1
a
X X .j=1
i=nj-L
j- 1
of the theorem.
K
-
,
=
os-b.
-=-'n./-1+1
It follows that Gn Gn* Axi' -
cUn -,wW'n,sl(p,+1
X
--I-:Z'
0 and this completes the proof of ,
(30.92) (30.70),and
The Functional
516
Central Limit Theorem
Now let ( Unit m x 1) be a vector array satisfying the conditions of 29.18, plus the extra condition that the Q-NED size of the increments is 1 for each element. Since 30.13 holds for each element paired with each element, including itself, the argument may be generalized in the following manner. -
30.14 Theorem Let Snj
=
S=1I&f. Then
n-1
/s
.---:''
1
B,,dB, +A
y&n,j+l
mxm).
(30.93)
0
.j=1
and p, the scalar arrays ('5k) Proof For arbitrary m-vectors of unit length, satisfy conditions (p'Un,j+3 the of 30.13. Letting Gn denote the matrix on and ) the left-hand side of (30.93),and G the matrix on the right-hand side, the result h.'Gn# -P-> X'G# is therefore given. A well-known matrix formula (seee.g. Magnus and Neudecker 1988: th. 2.2) yields
h.'Gn#
=
g' @A')Vec Gn,
(30.94)
where #' (B)A' is the Kronecker product of the vectors, the row vector (g1l1,..., p.r,,,,,) (1x m 2), and vec G n ml x 1) is the vector consisting g,2l,..., p'l&u, of the columns of Gn stacked one above the other. g' (&A' is of unit length, and applying the Cramr-Wold theorem (25.5)in respect of (30.94)implies that Gn -P-> ...,
G, as asserted in
(30.93).w
This result is to be compared with 30.5. Between them they provide the intriguing incidental information that 1
1
joBdBL + LdBnBL#&)(1)#n(1)' -
-
Q.
(30.95)
(Note that the stochastic matrix on the right has rank 1.) Of the two, 30.14 is much the stronger result, since it derives from the FCLT and is specific to the pattern of the incremeht variances. Between them, 30.3 and 30.14 provide the basic theoretical tools necessary to analyse the linear regression model in variables generated as partial-sum processes (integratedprocesses). See Phillips and Durlauf (1986),and Park and Phillips (1988,1989) among many other recent references.
Notes See Billingsley (1979,1986). The detinition of a Billingsley (1979),and as 1.26 in Billingsley (1986). -system
The
is given as 1.25 in
'
symbol denotes transposition../ is a column vector, written as a row for convenience. tprime'
An aftine transformation is a linear transfonnation x F-+ 4.x followed by a translation, addition of a constant vector b. By an accepted abuse of terminology, such transfonnations tend to be referred to as
.
Ix + y I S 1x1+ IyI See j5. 1 for more details. notations JJ/V, J/g.tdl,Or simply j'f when the relevant
4. That is,
.
5. The understood, are used synonymously by different authors.
measure
is
6. 1 thank Elizabeth Boardman for supplying this proof. 7. Elizabeth Boardman also suggested this proof. 8. If there is a subset N c f such that either N or M is contained in every T-set, the elements of N cease to be distinguishable as different outcomes. An equivalent model of the random experiment is obtained by redetining f to have N itself as an element, replacing
9. Random variables
its individual members.
may also be complex-valued', see j11.2.
10. In statements of general definitions and results we usually consider the case of a one-sided sequence (X,)T.There is no difticulty in extending the concepts to the case (xYr)=-x,and this is left implicit, except when the mapping from I plays a specitk role in the argument. 11. We adhere somewhat reluctantly to the convention of defining size as a size' to mean a slow (or negative number. The use of terms such as rapid'?) rate of mixing can obviously lead to confusion, and is best avoided.
12. This is a problem of $he weak corfvergence of distributions; see j22.4 for further details. 13. This is similar to an example due to Athreya and Pantula (1986a). 14. In the theory of functions of a complex variable, an analytic function is one possessing finite derivatives everywhere in its domain. 15. 1 am grateful to Graham Brightwell for this argument. 16. For convergence to fail, the discontinuities of f (which must be Borel measurable) would have to occupy a set of positive Lebesgue measure. 17. Conventionally, and for ease of notation, the symbol 5t is used here to denote
518
Notes
what has been previously written as F-rx. No confusion need arise, since a c-subfield bearing a time subscript but no superscript will always be interpreted in this way. 18. Some quoted versions of this result (e.g.Hall and Heyde 1980) are for p > whereas the present version, adapted from Karatzps and Shreve (1988),extends to 0 < p f as well. 19.
,
(Ar'm 1/2 To avoid confusion length, of a k-vector X is 11#11 with the Q nonn of a r.v., the latter is always written with a subscript.
The norm,
Or
=
20. The original St Petersburg Paradox, enunciated by Daniel Bernoulli in 1758, considered a gnme in wlch the player wins f 2 n-1 if the first head appears on the nth toss for any n. The expected winnings in this case are infinite, but the principle involved is the same in either case. See Shafer (1988). 21. See the remarks following 3.18. It is true that in topological spaces projections are continuous, and hence measurable, under the product topology (see j6.5), but of course, the abstract space (f1,1) lacks topological structure and this reasoning does not apply. 22. Since O is here a real k-vector it is written in bold face by convention, in the notwithstanding that 0 is used to denote the generic element of (0 abstract. ,p),
23. This is the basis of a method for generating random numbers having a distribution F. Take a drawing from the unifonn distribution on (0,1J (i.e.,a random string of digits with a decimal point placed in front) and apply the transformation F -1 (or F) to gi v e a drawing from the desired distribution. 24.
is used here as the argument of the ch.f. instead of the t used in Chapter 11, to avoid confusion with the time subscript. .
25. The symbol i appearing as a factor in these expressions denotes VZ. The context istinguishes the use of the same symbol as an array index. A
26. In practice, of course, Ut usually has to be estimated by a residual Ut, depending on consistent estimates of model parameters. In this case, a result such as 21-6 is also required to show convergence. 27. More precisely, of course, Gmodels the projection of the motion of a particle in three-dimensional space onto an axis of the coordinate system. 28. A cautionary note: these combinations cannot be constlucted as residuals from least squares regressions. If E has f'ull rank, the regression of one element of L onto the rest yields coefticients which are asymptotically random. E must be estimated from the increments using the methods discussed in 925.1. 29. Compare Wooldridge and White (1988:Prop. 4.1). Wooldridge and White's result is inconrct as stated, since they omit the stipulation of almost sure o,-..s+1..k..1+.,
References Amemiya, Takeshi
(1985),Advanced
Econometrics, Basil Blackwell, Oxford.
Andrews, Donald W. K. (1984),ENon-strong mixing autoregressive processes', Journal of Applied Probability 21, 930-4.
(1987a), generic unifonu
econometric models: a Econometrica 55, 1465-71.
in nonlinear consistency
law of large numbers',
of large numbers for dependent non-identically distributed random variables' Econometric Fctp?'y 4, 458-67. ttaaws
(1988),
,
and avtocorrelation consistent
Elleteroscedasticity
(1991),
covariance matrix estimation'
,
Econometrica
59, 817-58.
(1992), EGeneric uniform convergence',
Econometric
F/lct??'y8, 24 1-57. Apostol, Tom M. Menlo Park. Ash, R.
(1974),Mathematical
Analysis
(znd edn.)
Addison-Wesley,
Analysis and Probability, Academic Press, New York.
(1972),Real
Athreya, Krishna B. and Pantula, Sastl'y G.(1986a), EMixing properties of Hanis chains and autoregressive processes', Journal of Applied Probability 23,
880-92. (1986b), EAnote on strong mixing of
Statistics and Probability
ARMA processes',
Letters 4, 187-90.
Azuma, K. (1967),Weighted sums of certain dependentrandom variables' Mathematical Journal 19, 357-67. Barnsley, Michael
(1988),Fractals
,
Tohoku
Everywhere, Academic Press, Boston.
Bates, Charles and White, Halbert (1985),KAunified theory of consistent estimation for parametic models' Econometric W/lct??'y1, 151-78. ,
Bernstein, S. (1927), sommes de quantits
Esur
l'extension du thorme du calcul des probabilits dependantes', Mathentatische Annalen 97, 1-59.
aux
Eunifonn
consistency of kernel estimators of a regression function under generalized conditions', Journal of the American Statistical Association 77, 699-707.
Bierens, Herman
(1983),
(1989), ELeast squares estimation of linear and nonlinear Working Paper, Depmment of ARMAX models under data eterogeneity', Econometrics, Free University of Amsterdam.
520
References
Billingsley, Patrick New Yprk.
(1968),Convergence
of Probability Measures, John Wiley,
(1979), Probability and Measure, John Wiley, New York. Borowski, E. J. and Borwein, J. M. (1989),The Collins Reference Dictionary of Mathematics, Collins, London and GlasgoF, Bradley, Richard C., Bryc, W. and Janson, S. (1987),
(1968),Probability,
Addison-Wesley,
Reading, Mss.
Brown, B. M. (1971),'Martingale central limit theorems' cal Statistics 42, 59-66.
,
Annals of Mathemati-
Burkholder, D. L. (1973),tDistribution function inequalities for martingales', Annals of Probability 1, 19-42.
Chan, N. H. and Wei, C. Z. (1988),dLimiting distributions of least squares estimates of unstable autoregressive processes', Annals ofstatistics, 16, 367-401. mixing properties of linear stochastic processes' Chanda, K. C. (1974), Journal of Applied Probability 11, 401-8. <strong
Chow, Y. S. (1971), On the Lp convergence Mathematical Statistics 36, 393-4
for n
&, 0
<
p
<
l
,
,
Annals of
and Teicher, H. (1978),Probability Theory: Independence, lnterchangeabilit.y and Martingales, Springer-verlag, Berlin. Chung, Kai Lai (1974),A Course in Probability Fw?-y Press, Orlando, Fla.
(2ndedn.), Academic
Cox, D. R. and Miller, H. D. (1965), The Fdt?ry of Stochastic Processes, Methuen, London.
Cramr, Harald (1946),Mathematical Methods ofstatistics, Press, Princeton, NJ. Davidson, James
Princeton University
(1992), EA central
limit theorem for globally nonstationary near-epoch dependent functions of mixing processes', Econometric Fctpry, 8,
313-29. (1993a) $An fal-convergence theorem for heterogeneous mixingale arrays with trending moments', Statistics and Probabilit.y Letters 16, 301-4 central limit theorem for globally non(1993b), near-epoch stationary dependent functions of mixing processes: the asymptotically degenerate case', Econometric T/let)r.y 9, 402-12. t'l'he
521
References
de Jong, R. M. (1992), processes',Working Paper, 4taaws
Econometric Theory, 1995).
of large numbers for dependent heterogeneous in Free University of Amsterdam (forthcoming
(1994), 4A strong law forfu-mixingale sequences', Working Paper, Department of Econometrics, University of Tilburg. North-
amd Potential,
Dellacherie, C. and Meyer, P.-A. Holland, Amsterdam.
(1978),Probabilities
Dhrymes, Phoebus J. New York.
in Advanced Econometrics, Springer-verlag,
(1989)Topics
Dieudonn, J. (1969),Foundations York and London.
of Modern Analysis, Academic Press, New
Domowitz, 1. and White, H. (1982),'Misspecified models with dependent observations', Journal of Econometrics 20, 35-58 Donkker, M. D. (1951)
Doob, J. L. (1953),Stochastic Processes, John Wiley, New York; Chapman & Hall, London. Dudley, R. M. (1966),'Weak convergence of probabilities on nonseparable metric spaces and empirical measures on Euclidean spaces', Illinois Journal ofMathematics 10, 109-26.
(1967), <Measures on non-separable
metric spaces' Illinois Journal
(1989), RealAnalysis andprobability,
Wadsworth and Brooks/cole,
,
of Mathentatics 11, 109-26. Pacific Grove, Calif. q'
Dvoretsky, A. (1972),%Asymptotic normality of sums of dependent random variables' in Proceedings ofthe Sixth Berkeley Symposium on Mathematical Statistics and Probability, ii, University of California Press, Berkeley, Calif., 513-35. ,
Eberlein, Ernst and Taqqu, Murad S. (eds.)(1986),Dependence in Probability and Statistics.. a skrvdyof Recent Results, Birkhauser, Boston. Engle, R. F., Hendry, D. F. and Richard, J.-F. metrica 51, 277-304
(1983), %Exogeneity',
Econo-
Feller, W. (1971),An Introduction to Probabilit.y F/lct??'yand its Applications, ii, John Wiley, New York. Gallant, A. Ronald
(1987),Nonlinear
St#istical Models. John Wiley, New York.
and White, Halbert (1988), Un6ed Fctp?-y of Etimation .4
Inference
for
Nonlinear Dynamic Models, Basil Blackwll,
Oxford.
nd
522
Rcferences
Gastwirth, Joseph L. and Rubin, Herman (1975), 'The asymptotic distribution theory of the empiric CDF for mixing stochastic processes' Annals of Statistics 3, 809-24. ,
Gnedenko, B. V. (1967),The F/ldor.v ofprobability ing, New York.
(4thedn.),
Chelsea Publish-
Gorodetskii, V. V. (1977),
(1956),Lectures
in Ergodic Theoty, Chelsea Publishing, New
(1960), Naive Set F/lwry, Van Nostrand Reinhold, New York. (1974), Measure F/lc/ry, Springer-verlag,
New York.
Hall, P and Heybe, C. C. (1980),Martingale Limit Fdtpr.y and its Applicatln, Academic Press, New York and London.
(1970),Multiple
Hannan, E. J. Hansen, L. P.
(1982),
4l-arge
estimators' Econometrica ,
Time Series, John Wiley, New York.
sample properties of generalized method of moments 50, 1029-54.
laws for dependent heterogeneous processes', Hansel), Bruce E. (1991), Econometric Fet??'y 7, 2 13-2 1. Estrong
(1992a), <Errata', Econometric F/letpr.y8, 421-2. (1992b), (1992c), heterogeneous processes', Herrndorf, Norbert
estimation fordependent
Econsistentcovariancematrix
heterogeneous processes'
,
,
Econometrica
60, 967-72
integrals for dependent Econometric F/ldt)ry 8, 489-500.
to stochastic convergence
(1984),'A
f'unctional central limit theorem for weakly dependent sequences of random variables', Annals of Probabilit.v 12, 141-53.
(1985),
,
Wahrscheinlichkeitstheorie
vcrw'. Gebeite
Hoadley, Bruce (1971),'Asymptotic properties of maximum likelihood estimators for the independent not identically distributed case', Annals of Mathematical Statistics 42, 1977-91. inequalities for sums of bounded random variHoeffding, W. (1963), ables' Journal of the American Statistical Association 58, 13-30. tprobability
,
lbragimov, 1. A. (1962), somelimit theorems for stationary processes', Theoty of Probability and its Applications 7, 349-82.
References
(1965),
Theory ofproba-
and Linnik, Yu. V. (1971), lndependent and Stationary Se-
quences
of Random Variables. Wolters-Noordhoff,
Groningen.
losifescu, M. and Theodorescu, R. (1969), Random Processes and fztzmn', Berlin.
Springer-verlag,
Karatzas, Ioannis and Shreve, Steven E. (1988),Brownian Motion and Stochastic Calculus, Springer-verlag, New York. Kelley, John L.
Topology, Springer-verlag,
(1955),General
New York.
Kingman, J. F. C. and Taylor, S. J. (1966), Introduction to Measure Probability, Cambridge University Press, London and New York.
and
Kolmogorov, A. N. (1950),Foundations of the Tctp?'y of Probability, Chelsea Publishing, New York (published in German as Grundbegr#e der WahrscheinSpringer-verlag, Berlin, 1933). lichkeitsrechnung,
and Rozanov, Yu. A. (1960),'On strong mixing conditions for
stationary Gaussian processes', Theot'y of Probabilit.y and its Applications 5, 204-8. Kopp, P. E. Press.
(1984),Martingales
and Stochastic Integrals, Cambridge University
Kurtz, T. G. and Protter, P. (1991), Weak limit theorems for stochastic integrals and stochastic differential equations', Annals of Probability 19, 1035-70. Love, Michel York. Lukacs, Eugene New York.
F/lctpry, i
(1977),Probability (1975),Stochastic
(4th edn.),
Convergence
Springer-verlag,
(2nd edn.),
Nw
Academic Press,
Calculus with Magnus, J. R., and Neudecker, H. (1988),Marix Dterential Applications in Statistics and Econometrics, John Wiley, Chichester. Mandelbrot, Benoit B. New York.
(1983),The Fractal
Geometry ofNature, W. H. Freeman,
Mann, H. B. and Wald, A. (1943a),
(1943b), dOn stochastic limit and order relationships' Annals of Mathematical Statistics 14, 390-402. ,
McKean, H. P., Jr.
(1969),Stochastic
Integrals, Academic Press, New York.
McLeish, D. L. (1974),tDependent central limit theorems and invariance principles' Annals of Probability 2,4, 620-8. ,
References
524
- (1975a),KAmaximal inequality and dependent strong laws'
--
Annals of Probability
,
3,5, 329-39.
(1975b), 'Invariance principles for dependent variables' Wahrscheinlichkeitstheorie
,
Z.
vcnp. Gebeite 32, 165-78.
(1977), tOn the invariance principle for nonstationary
mix-
'
ingales' Annals of Probability 5,4, 616-21. ,
inequalities for sums of Nagaev, S. V. and Fuk, A. Kh. (1971), independent random variables', Theory of Probability and its Applications tprobability
6, 643-60. Newey, W. K.
convergence in probability and stochastic equi59, 1161-8.
(1991),'Uniform
continuity', Econometrica
and West, K. (1987),'A simple positive definite heteroskedastlc-
ity and correlation consistent covariance matrix', Econometrica 55, 703-8.
inference in regressions Park, J. Y. and Phillips, P. C. B. (1988), with integrated processes, Part 19 Econometrt'c F/lwr.y 4, 468-98. istatistical
,
inference in regressions with integrated processes, Part 2', Econometric F/let?ry 5, 95-132.
(1989),
Kstatistical
Parthasarathy, K. R. (1967),Probability Measures on Metric Spaces, Academic Press, New York and London. Pham, Tuan D. and Tran, Lanh T. (1985), mixing properties of time series models', Stochastic Processes and their Applications 19, 297-303. tsome
Phillips, P. C. B. (1988),tWeak convergence to the matrix stochastic dB' Journal of Multivariate Analysis 24, 252-64. J()#l
-
integral
and Durlauf, S. N. (1986),GMultiple time series regression with integrated processes', Review of Economic Studies 53, 473-95.
Pollard, David New York.
(1984), Convergence
of Stochastic
Processes,
Springer-verlag,
Ptscher,
B. M. and Pruha, 1. R. (1989),KAuniform 1aw of large numbers for dependent and heterogeneous data processes' Econometrica 57, 675-84. ,
convergence and (1994), 'Generic equicontinuity concepts for random functions: an exploration of the basic structure', Journal of Econometrics 60, 23-63. uniform
(1991a), Basic structure of the asymptotic nonlinear econometric models, Part 1: Consistency and in dynamic theory approximation concepts', Econometric Rcvfdwx 10, 125-216.
References
(1991b), tBasic structure of the asymptotic nonlinear dynamic in econometric models, Part 1I: Asymptotic nonnaltheory ity', Econometric Reviews 10, 253-325. Prokhorov, Yu. V (1956), of random processes and limittheoremd in probability theory', Theoty of Probability and its Applications 1, 157-213. tconvergence
Rao, C. Radhakrishna (1973),Linear Statistical Inference and its Applications (2nd edn.), John Wiley, New York. Rvsz,
Pl
(1968),The
fxzwu of fzzrge Numbers, Academic Press, New York.
Rosenblatt, M. (1956),KAcentral limit theorem and a strong mixing condition' Proceedings of the National Academy of Science, USA, 42, 43-7.
,
(1972), unifonnergodicity and strong mixing', Z Wahrscheinlichkeitstheorie vcrw,. Gebeite 24, 79-84. (1978),
(1968),Real
Analysis, Macmillan, New York.
(1976),Regularly
Varying Functions, Springer-verlag,
Berlin.
Serfling, R. J. (1968), to central limit theory for dependent variables' Annals of Mathetnatical Statistics 39, 1158-75.
,
(1980), Approximation
Theorems of Mathentatical
Statistics,
John Wiley, New York. Shafer, G. (1988), tical Sciences, viii
t'l'he
Shiryayev, A. N.
St Petersburg Paradox', in Encyclopaedia of the Statis(ed. S. Kotz and N. L. Johnson), John Wiley, New York.
(1984),Probability,
Springer-verlag,
New York.
Skorokhod, A. V. (1956), tmit theorems for stochastic processes', Probability and its Applications 1, 261-90.
(1957),
ttaiinit
Theoty of
theorems for stochastic processes with indepen-
dent increments', Theory of Probability and its Applications 2, 138-71. Slutsky, E. (1925), Annalen 5, 93.
tber stochastiche
Asymptoter und Grenzwerte'
,
Math.
measurability results for exStinchcombe, M. B. and White, H. (1992), random ofEconomic Studies 59, functions of random sets', Review trema over <some
495-514.
Stone, Charles
(1963), 'Weak
convergence of stochastic processes defined on semi-infinitetimeinterkals', ProceedingsoftheAmericanMathematicalsocie# 14, 694-6.
References
526
Stout, W. F. (1974), Almost Sure Convergence, Academic Press, New York. Strasser, H. Probability
(1986), KMartingale
difference arrays and stochastic integrals' F/letpry and Related Fields 72, 83-98.
Varadarajan, V. S. (1958), Weak convergence spaces' Sankhya 19, 15-22.
,
of measures on separable metric
,
for the rth absovon Bahr, Bengt, and Essen, Carl-Gustav (1965), variables, random 1 S r K 2', Annals ofMathematical lute moment of a sum of Klnequalities
Statistics 36, 299-303.
White, Halbert (1984), Asymptotic Press, New York.
for
Fet?ry
Econometricians,
Academic
and DomoFitz, 1. (1984),KNonlinear regression with dependent observations' Econometrica 52, 143-62. ,
Wiener, Norbert (1923), dDifferential space' 2, 131-74 V/illard, Stephen
(1970),General
Withers, C. S. (1981a), Wahrscheinlichkeitstheorie
dconditions
(198 1b),
tcentral
Wahrscheinlichkeitstheorie
,
Journal of Mathematical
Physics
Topology, Addison-W-esley, Reading, Mass.
for linea.r processes to be strong-mixing' vdrw. Gebeite 57, 477-80. limit theorems for dependent variables, verw'. Gebeite 57, 509-34.
,
Z.
1' Z ,
invariance principles Wooldridge, Jeffrey M. and White, Halbert (1986), central theorems for limit dependent heterogeneous processes' University and of California (San Diego) Working Paper. dsome
,
invariance principles and central limit theorems for dependent heterogeneous processes' Econometric Fet??'y 4, 210-30.
(1988),
tsome
,
lndex Abel's partial summation fonnula 34, 254 absolute convergence 31 absolute moments 132 absolute value 162 absolutely continuous function 120
asymptotic equicontinuity 90 asymptotic expectation 357 asymptotic independence 479 asymptotic negligibility 375 asymptotic unifonn equicontinuity 90, 335 asymptotic unpredictability 247
absolutely continuous measure 69 absolutely regular sequence 2U9 absact integral 128 accumulation point 21, 94 adapted process 500 adapted sequence 229 addition modulo 1 46 adherent point 77 affine transfonnation 126
Athreya, Krishna B. 215 atom 37 of a distribution 129 of a p.m. 112 autocovariance 193 autocovariances, summability 266 autoregressive process 215 non-strong mixing 216 non-uniform mixing 218
Aleph nought 8
autoregressive-moving
algebra of sets 14 almost everywhere 38,
process 215 axiom of choice 47 axioms of probability
a-mixing, see strong mixing
113
almost sure convergence 178, 281 method of subsequences 295 uniform 331 almost surely 113 analytic function 328 analytic set 328 of C 449 Andrews, D. W. K. 216, 261, 301, 336, 338, 341 antisymmetric relation 5 Apostol, T. M. 29, 32, 33, 126 approximable in probability 274 approximable process 273 weak law of large numbers 304 approximately mixing 262 AR process, see autoregressive
process
ARMA process 215 an'ay 33 array convergence 35 Arzel-Ascoli theorem 91, 335, 439, 447, 469
average
Azuma, K. 245 Azuma's inequality 245
p-mixing 207 backshift transformation 191 ball 76 band width 401 Bartlett kernel 403, 407 base 79 for point 94 for space of measures 418 for topology 93 Bernoulli distribution 122 expectation 129 Bernoulli r.v.s 216 Bernstein sums 386, 401 Berry-Essen theorem 407 betting system 233-4 Bierens, Herman 261, 336 5ig Oh 31j 187 Bllingjley, Patrick 17, 18, 261,
Index
528 421, 422, 447, 448, 457, 466, 469, 472, 473, 474-5, 496 Billingsley metric 462 equivalent to Skorokod
metric
464 binary expansion 10 binary r.v. 122, 133 binary sequence 180 binomial distribution 122, 348, 364 bivariate Gaussian 144 blocking argument 194, 299 Borel
tield
of C 440 of D 465 intinite-dimensional 181 real line 22, 16 metric space 77, 413 topological space 413 Borel function 55, 57, 117 expectation of 130 Borel sets 47 Borel-cantelli lemma 282, 295, 307 Borel's normal number theorem 290 boundary point 21, 77 bounded convergence theorem 64 bounded function 28 bounded set 22, 77 bounded variation 29 Brown, Robert 443 Brownian bridge 445 mean deviations 498 Brownian motion 443 de-meaned 498 distribution of supremum 496 transformed 445, 486 vector 454 with drift 444 Burkholder's inequality 242 C 437
c.d.f., see cumulative distribution function cadlag function 90, 456 cadlag process 488 progressively measurable 500
Cantor, G. 10 cardinal number 8 cardinality 8 Cartesian product 5, 83, 102 Cauchy criterion 25 Cauchy distribution 123 distribution 362 as ktable characteristic function 167 no expectation 129 Cauchy family 124 Cauchy sequence 80-2, 97 real numbers 25 vectors 29 Cauchy-schwartz inequality 138 central limit theorem ergodic Ll-mixingales 385 functional 450, 480 independent sequences 368 martingale differences 383 NED functions of mixing processes 386 three series theorem 312 trending variances 379 central moments 131 central tendency 128 centred sequence 231 Cesro sum 31 ch.f., see characteristic function Chan, N. H. 510 Chanda, K. C. 215, 219 characteristic function 53, 162 derivatives 164 independent sums 166 multivariate distributions 168 series expansion 165 weak convergence 357 Chebyshev's inequality 132 Chebyshev's theorem 293 chi-squared distribution 124 chord 133 Chow, Y. S. 298 Chung, K. L. 407, 409 closed under set operations 14 closed interval 11 closed set 77
q' t'J''f '
.
Index
real line 21 closure point 20, 77, 94 cluster point 24, 80, 94 coarse topology 93 coarseness 438 codomain 6 coin tossing 180, 191 collection 3 compact 95 compact set 22, 77 compact space 77 compactificaton 107 complement 3 complete measure space 39 complete space 80 completely regular sequence 209 completely regular space 100 completeness 97 completion 39 complex conjugate 162 complex number 162 composite mapping 7 concave function 133 conditional characteristic function 171-3 distribution function 143 expectation 144, 147 linearity 151 optimal predictor 150 variance of 156 versions of 148 Fatou's lemma 152 Jensen's inequality 153 Markov inequality 152 modulus inequality 151
monotone convergence theorem 152 probability 113 variance 238, 316 conditionally heteroscedastic 214 consistency properties 183, 435 consistency theorem function spaces 436 sequences 184 contingent probability 114 continuity 112
..
: . . 7((
jz. '
t: (
529 of a measure 38 continuous distribution 122 continuous function 27, 84, 436 continuous mapping 97 continuous mapping theorem 355, 497 metric spaces 421 continuous time martingale 501 continuous time process 500 continuous truncation 271, 309 continuously differentiable 29 contraction mapping 263 converge absolutely 31 convergence 94 almost sure 178, 28 1, 331 in distribution 347, 367 in Lp norm 287, 331 in mean square 179, 287 in probability 179, 284, 349, 359, 367 in probability 1aw 347 metric space 80 on a subsequence 284 real sequence 23 space of probability naeasures 4 18 stochastic function 333 transformations 285 unifonu, 30, 331 weak 179 with probability 1 179, 331 convergence lemma 306 convergence-detenuining class 420 convex function 133, 339 convolution 161 coordinate 177 coordinate projections 102, 434 coordinate space 48 coordinate transformation 126 correlation coefficient 138 correspondence 6 countability axioms 94 countable additivity 36 countable set 8 countable subadditivity 37, 111 countably colpact 95
:
' .
E
.
'. . . . . . ..
Index
530 covariance 136 covariance matrix 137 covariance stationarity 193 covering 22, 77 Cox, D. R. 503 cr inequality 140 Cramr, Harald 141 Cramr-Wold device 405, 455, 490,
4
516 Cramr's theorem 355 cross moments 136
cumulative distribution function 117 cylinder set 48, 115 of R 434 D 456 Billingsley metric on 464 Davidson, James 301, 386 de Jong, R. M. 319, 326 de Morgan's laws 4 decimal expansion 10 decreasing function 29
decreasing sequence of real numbers 23 of sets 12 degenerate distribution 349 degree of belief 111 Dellacherie, C. 328 dense 23 dense jet 77 density function 74, 120 denumerable 8 dependent events 113 derivative 29 conditional expectation 153 expectation 141 derived sequence 177 determining class 40, 121, 127, 420 diagonal argument 10 diagonal method 35 diffeomorplzism 126 difference, set 3 differentiable 29 differential calculus 28
diffusion process 503 discontinuity, jump 457 discontinuity of first kind 457 discrete distribution 122, 129 discrete metlic 76 discrete subset 80 discrete topology 93 disjoint 3 distzibution domain 6 dominated convergence theorem 63
Donsker, M. D. 450 Donsker's theorem 450 Doob, J. L. 196, 216, 235, 314 Doob decomposition 231 Doob's inequality 241 continuous time 501 Doob-Mayer decomposition 502 double integral 66, 136 drift 444 Dudley, R. M. 328, 457 Durlauf, S. N. 490, 516 dyadic rationals 26, 99, 439 dynamic stability 263 Dynkin, E. B. 18 Dynkin's 7t-/, theorem 18
e-neighbourhood 20 see also sphere y-net 78 Egoroff s theorem 282, 510 element, set 3 embedding 86, 97, 105 empirical disibution function 332 empty set 3, 77
Engle, R. F. 403
ensemble average 195 envelope 329 equally likely 180 equicontinuity 90, 335 stochastic 336 strong stochastic 336 equipotent 6 equivalence class 5, 46 equivalence relation 5 equivalent measures 69
Index
equivalent metrics 76 equivalent sequences 307, 382 ergodic theorem 200 law of large numbers 291 ergodicity 199 asymptotic independence 202 Cesro-summability
of
autocovariances 201 Carl-Gustav 171 essential supremum 117, 132 estimator 177 Euclidean distance 20, 75 Euclidean k-space 23 Euclidean metric 75, 105 evaluation map 105 even numbers 8 exogenous 403 expectation 128 exponential function 162 exponential inequality 245 extended functions 52 extended real line 12 extended space 117 extension theorem 184 existence part 40 uniqueness part 44, 127 Essen,
of D 466 Wiener measure 446
tinite intersection property 95 finite measure 36 first countable 95 fractals 443 frequentist model 111 1aw of large numbers 292 Fubini's theorm 66, 69, 125 Fuk, A. Kh. 220
function 6 convex 339 of a real variable 27 of bounded variation 29 function space 84, 434 nonseparable 89 functional 84 functional central limit theorem 450, 480 martingale differences 440 multivariate 454, 490 NED functions of strong mixing processes 481 NED functions of unifonn mixing processes 485 Gallant, A. Ronald 261, 263, 271,
see uniform mixing Y-mixing, T-analytic sets 449 factor space 48 fair game 289 Fatou's lemma 63 conditional 152 Feller, W. 32 Feller's theorem 373 field of sets 14
filter 94 filtration 500 fine toplogy 93 tineness 438 tinite additivity 36 tinite dimensional ylinder sets 181, 435 finite dimensional distributions of C 440, 442, 446
401-2 gnmbling policy 233 gamma function 124
Gaussian distribution 123 characteristic function 167 stable distribution 363 Gaussian family 123 expectation 129 moments 131 Gaussian vector 126
generic unifonn convergence 336 geometric series 31 Glivenko-cantelli theorem 332 global stationarity 194, 388, 450, 486 Gordin, M. 385 Gorodetskii, V. 215, 219, 220 graph 6
532 Hahn decomposition 70, 72 half line 11, 21, 52, 118 half-Gaussian distribution 385 half-normal density 124 half-open interval 11, 15, 118 Hall, P. 250, 314, 385, 409 Halmos, Paul R. 202 Hansen, Bruce E. 318, 403, 510 Hartman-Wintner theorem 408 Hausdorff metric 83, 469 Hausdorff space 98 Heine-Borel theorem 23 Helly-Bray theorem 353 Helly' s selection theorem 360 Heyde, C. C. 250, 314, 385, 409 Hoadley, Bruce 336 Hoeffding's inequality 245 Hlder's inequality 138 homeomorphism 27, 51, 86, 97, 105
i.i.d. 193 Ibragimov, 1. A. 204-5, 210, 21 1, 215, 216, 261 identically distributed 193 image 6 imaginary number 162 inclusion-exclusion fonnula 37, 420 increasing function 29
increasing sequence of real numbers 23 of sets 12 independence 114 independent Brownian mtions 455 independent r.v.s 127, 154-5, 161 independent sequence 192 strong 1aw of large numbers 31 1 independent subfields 154 index set 3, 177 indicator function 53, 128 inferior limit of real sequence 25 of set sequence 13 infimum 12 infinite run of heads 180
Index
infinite-dimensional cube 83, 105 infinite-dimensional Euclidean space 83, 104 infinitely divisible distribution 362 initial conditions 194 inner measure 41 innovation sequence 215, 231 integers 9 integrability 61 integral 57 integration by parts 58, 129, 134 interior 21, 77 intersection 3 interval 11 into 6 invariance principle 450, 497 invaiant event 195 invariant nv. 196 inverse image 6 inverse projection 48 inversion theorem 168-70 irrational numbers 10, 80 isolated point 21, 77 isometry 86 isomomhic spaces 51 isomorphism 145 iterated integral 66, 135 lt integral 507 J1 metric 459 Jacobian matrix 126 Jensen's inequality 133 conditional 153 Jordan decomposition 71 jump discontinuities 457 jump points 120 Karatzas, loannis 502-3, 508 kernel estimator 403 Khinchine's theorem 368 Kolmogorov, A. N. 209-10, 311-2 Kolmogorov consistency theorem 184 Kolmogorov's inequality 240 rwnntln.xzxxn
+:*'.-*.--
rn4
lndex Kolmogorov's zero-one law 204 Kopp, P. E. 504 Kronecker product 516 Kronecker's lemma 34, 293, 307 Kurtz, T. G. 510
l-system 18 l'Hpital' s rule 361 Lv convergence 287 uniform 331 Lp norm 132 fw-approximable 274 fv-bounded 132 fv-dominated 330 largest element 5 latent variables 177 1aw bf iterated expectations 149 law of large numbers 200, 289 Cauchy r.v.s 291 definition of expectation 292 frequentist model 292 random walk 291 unifonn 340 law of the iterated logarithm 408 Lebesgue decomposition 69, 72 probability measure 120 Lebesgue integral 57 Lebesgue measure 37, 45-6, 74, 112
plane 66 product measure
135
Lebesgue-integrable nv.s 132 Lebesguecstieltjes integral 57-8,
128 left-continuity 27 left-hand derivative 29 Lvy, P. 312 Lvy continuity theorem 358 Lvy's metric 424, 468 lexicographic ordering 10 Liapunov condition 373 Liapunov's inequality 139 Liapunov's theorem 372 L 149
liminf of real sequence 25
533 of set sequence 13 limit 80 expectation of 141 of set sequence 13 limit point 26 limsup of real sequence 25 of set sequence 13 Lindeberg condition 369, 371, 380 asymptotic negligibility 376 uniform integrability 372 Lindeberg theorem 369 Lindeberg-Feller theorem, see Lindeberg theorem; Feller's
theorem Lindeberg-taevy theorem 366 FCLT 449 Lindelf property 78-79, 95 Lindelf space 98 Lindelf's covering theorem 22 linear ordering 5 linear process 193, 247, 252 strong law of large numbers 326
linearity of conditional expectation 151 of integral 62
Linnik, Yu. V. 204-5, 210, 215, 216 Lipschitz condition 28, 86, 269 stochastic 338 Little Oh 31, 187 Love, M. 32, 407 Love's cr inequality 140 log-likelihood 177 lower integral 57 lower semicontinuity 86 lower variation 71 MA process, see moving average process
Magnus, J. R. 516 Mandelbrot, Benoit 443 Mann, H. B. 187 mapping 6 marginal c.d.f. 126 marginal distributions of a
Index
534 sequence 186 marginal measures 64 marginal probability measures 115 tightness 430 Markov inequality 132, 135 conditional 152 Markov process 503 martingale 229 continuous time 501 convergence 235 mmingale difference array 232 weak law of large numbers 298 mmingale difference sequence 230 strong law of large numbers 314 maximal inequalities for linear processes 256 for martingales 240 for mixingales 252 maximum metric 76 McKean, H. P., Jr. 508 Mcl-eish, D. L. 247, 261, 318, 380 mean 128 mean reversion 214 mean stationarity 193 mean value theorem 340 mean-square convergence 293 measurability of suprema 327, 449 measurable function 117 measurable isomorphism 51 measurable rectangle 50, 48 measurable set 41 measurable space 36 measurable transformation 50 measure 36 naeasure space 36
nAeasure-preserving
transfornAation
191 rnixes outconaes 200 memory 192 method of subsequences 295 metric 75 metric space 75, 96 metrically transitive 199 metrizable space 93 metrization 107
Meyer. P.-A. 328 Miller, H. D. 503 Minkowski's inequality 139 mixed continuous-discrete distribution 129 mixed?Gaussian distribution 404 mixing 202
inequalities 211-4 MA processes 215 martini example 202 measurable functions 21 1 mixing process strong 1aw of large numbers 323 mixing sequence 204 mixing size 210 see also size mixingale 247 stationary 250 strong 1aw of large numbers 318 weak 1aw of large numbers 30l mixingale array 249 modulus 162 modulus inequality 63 complex r.v.s 163 conditional l5l modulus of continuity 91, 335, 439, 479 cadlag functions 458, 468 moment generating function 162 moments 131 Gaussian distribution 123
monkeys typing Shakespeare 180 monotone class 17 monotone convergence theorem 60, &. conditional 152 envelope theorem 329 monotone function 29 monotone sequence of real numbers 23
of sets 12 monotonicity of measure 37 of p.m. 111
Monte Carlo 498
moving average process 193 mixing 215
Index
strong law of large numbers 326 multinonnal disibution, see multivariate Gaussian multinormal p.d.f. 126 multivariate c.d.f. 125 multivariate FCLT 490 multivariate Gaussian affine transfonnations 170 characteristic function 168 mutually singular measures 69 Nagaev, S. V. 220 nave set theol'y 47 natural numbers 8 near-epoch dependence 261 mixingales 264 transfonnations 267 near-epoch dependent process strong 1aw of large numbers 323 weak 1aw of large numbers 302 nearly measurable set 329 NED, see near-epoch dependence negative set 70 neighbourhood 20 see also sphere nested subfields 155 net 94 Neudecker, H. 516 Newey, W. K. 336, 401 non-decreasing function 29
non-decreasing sequence of real numbers 23 of sets 12 non-increasing function 29 nn-increasing sequence of real numbers 23 of sets 12 non-measurable function 55 non-measurable set 46 norm inequality 139 for prediction errors 157 normal distribution 123 normal law of error 364 normal number theorel 290 nonnal space 98 null set 3, 70
535 odd-order moments 131 one-dimensional cylinder 182 one-to-one 6 onto 6, 97 open covering 22, 77 open interval 11 open mapping 27 open rectangles 102 open set 77, 93 of real line 20 order-preserving mapping 6 ordered pairs 7 orders of magnitude 31 origin 10 Ornstein-uhlenbeck
process 445 diffusion process 503 outer measure 41 outer product 137
7:-1 theorem 18, 44, 49, 67 zr-system 18 p.d.f., see probability density function p.m., see probability measure pairwise independence 114 pairwise independent nv.s 127, 136 Pantula, Sastry G. 215 parameter space 327 Park, J. Y. 516 Parthasarathy, K. R. 418, 422, 426, 427, 429, 469 partial knowledge 145 partial ordering 5 partial sums 31 partition 3 of g0,1) 438 permutation of indices 436 Pham, Tuan D. 215 Phillips, P. C. B. 490, 510, 516 piecewise linear functions 437 pigeon hole principle 8 Pitman drift 369 pointwise convergence 30 stoch as tic 331
536
Index
Poisson distribution 122, 348 characteristic function 167 expectation 129 infinitely divisible 362 polar coordinates 162 Pollard, David 457 positive semi-definite 137 positive set 70 Ptscher, B. M. 261, 274, 277, 336,
342 power set 13 precompact set 78 predictable component 231 probability density function 122 probability measure 111 weak topology 418 probability space 111, 117 product measure 64 product space 7. 48, 102, 115 prpduct topology 102 function spaces 453 product, set 5 progressively measurable 500 progressively measurable functions 504 projection 8, 48, 50 suprema 328 not measurable 328 Projection c-field 415, 435 of C 440 of D 456 Prokhorov, Yu. V. 422, 423, 457 Prokhorov metlic 424, 467 Protter, P. 510 Prucha, 1. 261, 274, 277, 336, 342 pseudo-metric 75, 504
-dependence 215 quadratic variation 238, 502 deterministic 503 p-mixing 207 R, the space 434 r.V., see random Variable
D oazxn-Mit-naxzm
aerivntive 70. 74.
120, 148 Radon-Nikodym theorem 70, 72-4, 122 random element 413, 111 random event 111 random experiment 111, 128 random tield 178 randorp pair 124 random sequence 177 memory of 192 random variable 112, 117 random vector 137 random walk 230, 291 random weighting 316 range 6
Rao, C. R. 143 rate of convergence 294 rational numbers 9 real numbers 10 real plane 11 real-valued function 27, 87, 434 realization 179 refinement 438 reflexive relation 5 regression coeftkients regular measure 413 regular sequence 204
regular space 98 regularly varying function 32 relation 5 relative compactness 422 relative equencies 128 relative topology 20, 93 relatively compact 77 remote c-field 203 repeated sampling 179 restriction of a measure space 37 Riemann integral 57 expectation 141 of a random function 496
Riemann zeta function 32 Riemann-stieltjes integral 58 and stochastic integral 503 right continuous 27 c.d.f. 119-20 filtration 500
537
Index
right-hand derivative 29 ring 14 Rozanov, Yu. A. 209-10
size mixing 210 mixingale 247
near-epoch dependence 262 Skorokhod, A. V. 350, 457, 459 Skorokhod metric 459, 469 Skorokhod representation 350, 431,
c-algebra 15 c-field 15 c-finite measure 36
510
St Petersburg paradox 289 sample 177 sample average 128 sample path 179 sample space 111 scale transformation 178 seasonal adjustment 262 second countable 79, 95 Serfling, R. J. 213 self-similarity 443 semi-algebra 15 semi-ring 15, 40, 48, 49, 118 semimartingale 233 in continuous time 501 separable set 78 separable space 95 separating function 98 separation axioms 97 sequence 9, 94 metric space 80 real 23 sequentially compact 95, 422 serial independence 192 series 31 set 3 set function 36 set of measure zero 3$, 70 shift transformation 191 shocks 215 Shreve, Steven E. 502-3, 508 signed measure 70 simple function 53 approximation by 54 integral of 59 simple random variables 128
Skorokhod topology 461 slowly varying function 32 Slutsky's theorem 287 smallest element 5 Souslin space 328 spectral density function 209 MA process 215 sphere 76 of C 440 of D 466 stable distribution 362 stationarity 193 step function 119, 349 Stinchcombe, M. B. 328 stochastic convergence, sec under
convergence stochastic equicontinuity 336 termwise 341 stochastic integral 503 stochastic process 177 continuous time 500 stochastic sequence 177 stopped process 234 stopping l'ule 234 stopping time 233 filtration 501 Stout, W. F. 314, 409 Strasser, H. 510 strict ordering 5 strict stationarity 193 strong law of large numbers 289 for independent sequence 310 for fw-bounded sequence 295, 312, 314 j. . j . . .. . C fOr martingales 3 14-7 singleton 11 ( ) 11')y9, fpr mixingales 319-23 r singular Gaussian distribu/ih y;ytyy, 111111. 11:1(714: ttj.jjtyyyjtyjyrgylrytlttytjj,yy y ,i!r:il!Ell. :::12,: (111. 1!4111', (Irr-''d:izrr 1!!1; il:.;lr. j ).';'yr'.;).. fuuctjous oj mjxjng t!!il. (IrF:II:'III. 212':11. IIL.;IE. CIEEII.III:-fjjy t,ii. .y (11. ... yy y..jy.) lrliyl .' . :!i5;kliir;r )y.t..,y..y y..j.-ty... yq.. . . .y,.j.y.yj.y jygjg..jt;)),j.j.. yj. . . . xsp y,y.-y;yj.y....t.,j j ... . .L. y. '
.
.
,(
,
y,-yqlys.. Lj.;jjj;
-
:
.2. .. .. ...t .. .. . . . .(
t,
-).
,
,
q.(.-yyytj.y;yyyy.Ey.....yy(-t,;)y).-.j..yjy,y-yy, --.j
jy.yty ,jyjy;
.4
.
;
,.j.
...
.
...
-
.yy
.
...
y.
lndex
538 processes 324-6 strong mixing autoregressive processes 216 coefticient 206 negligible events 207 smoothness of the p.d.f. 225 sufficient conditions in MA processes 219-27 see also mixing strong mixing sequence 209 law of large numbers 295, 298 strong topology 93 strongly dependent sequences 210 strongly exogenous 403 sub-base 101, 103 subcovering 22 subfield measurability 145 subjective probability 111 submartingale 233 continuous time 501 subsequence 24, 80 subset 3 subspace 20, 93 summability 31 superior limit of real sequence 25 of st sequence 13 supermartingale 233 continuous time 501 support 37, 119 supremum 12 sure convergence 178 symmetric difference 3 symmetric relation 5 Tl-space Tz-space Ta-space Tgj-space
98 98 98 100
L-space 98
tail sums 31 taxicab metric 75 'Taylor's theorem 165 telescoping sum 250 proof of weak law 301
termwise stochastic equicontinuity 341 three series theorem 31 1 tight measure 360, 427 time average 195 and ensemble average 200 limii under stationarity 196 time plot 437 time series 177 Toeplitz's lemma 34, 35 Tonelli' s theorem 68 topological space 93 topology 93 real line 20 weak convergence 418 torus 102 total independence 114 total variation 72 totally bounded 78, 79 totally independent r.v.s 127 trace 112 Tran, Lanh T. 215 transformation 6 transformed Brownian motion 486 diffusion process 503 transitive relation 5 trending moments 262 triangle inequality 31, 139 triangular array 34 stochastic 178 trivial topology 93 truncated kernel 403 truncation 298, 308 continuous 271, 309 two-dimensional cylinder 182 Tychonoff space 100 Tychonoff topology 103, 181, 461 Tychonoff s theorem 104 uncorrelated sequence, 230, 293 uncountable 10 uniform conditions 186 uniform continuity 28, 85 uniform convergence 30 unifonn distribution 112, 122, 364
Index
convolution 161 expectation 129 unifonu equicontinuity 90 uniform integrability 188 squared partial sums 257 unifonu laws of large numbers 340 uniform Lipschitz condition 28, 87 uniform metric 87 parameter space 327 unifonn mixing autoregressive process 218 coefficient 206 moving average process 227 see also mixing unifonn mixing sequence 209 strong law of large numbers 298 wak 1aw of large numbers 295 unifonn stochastic convergence 331 unifonn tightness 359, 427 Arzel-Ascoli
theorem 447 measures on C 447,448 uniformly bounded aas. 186 in L;, norm 132, 186 in probability 186 union 3 universally measurable set 328 upcrossing inequality 236 upper integral 57 upper semicontinuity 86, 470 upper variation 71 Urysohn's embedding theorem 106 Urysohn's lemma 98 Urysohn's metrization theorem 98 usual topology of the line 20 Varadjaran, V. S. 425 variable 117 variance 131
539 sample mean 293 variance-transfonned Brownian motion 486 vector 29 vector Brownian motion 498 vector martingale difference 234
Venn diagram 4 versions of conditional
expectation
148 von Bahr, Bengt 171 Wald, A. 187
weak convergence 179, 347 in metric space 418 of sums 361 weak dependence 210 weak law of large numbers 289 for Al-approximable process 304 for fal-mixingale 302 for fq-bounded sequence 293 for partial sums 312 weak topology 93, 101 space of probability measures 4 18 Wei, C. Z. 510 well-ordered 5 West, K. 401 White, Halbert 210, 261, 263, 271, 328, 401-2, 480-1 wide-sense stationarity 193 Wiener, Norbert 442 Wiener measure 442, 474 existence of 446, 452 with probability 1 113 Withers, C. S. 215 Wooldridge, Jeffrey M. 480-1
zero-one 1aw 204
