This page intentionally left blank
Probability Theory An Analytic View, Second Edition This second edition of Daniel ...
24 downloads
724 Views
5MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
This page intentionally left blank
Probability Theory An Analytic View, Second Edition This second edition of Daniel W. Stroock’s text is suitable for first-year graduate students with a good grasp of introductory undergraduate probability. It provides a reasonably thorough introduction to modern probability theory with an emphasis on the mutually beneficial relationship between probability theory and analysis. It includes more than 750 exercises and offers new material on Levy processes, large deviations theory, Gaussian measures on a Banach space, and the relationship between a Wiener measure and partial differential equations. The first part of the book deals with independent random variables, Central Limit phenomena, the general theory of weak convergence and several of its applications, as well as elements of both the Gaussian and Markovian theories of measures on function space. The introduction of conditional expectation values is postponed until the second part of the book, where it is applied to the study of martingales. This part also explores the connection between martingales and various aspects of classical analysis and the connections between a Wiener measure and classical potential theory. Dr. Daniel W. Stroock is the Simons Professor of Mathematics Emeritus at the Massachusetts Institute of Technology. He has published many articles and is the author of six books, most recently Partial Differential Equations for Probabilists (2008).
Probability Theory An Analytic View Second Edition
Daniel W. Stroock Massachusetts Institute of Technology
cambridge university press Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, S˜ a o Paulo, Delhi, Dubai, Tokyo, Mexico City Cambridge University Press 32 Avenue of the Americas, New York, NY 10013-2473, USA www.cambridge.org Information on this title: www.cambridge.org/9780521132503 c Daniel W. Stroock 1994, 2011 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First edition published 1994 First paperback edition 2000 Second edition published 2011 Printed in the United States of America A catalog record for this publication is available from the British Library. Library of Congress Cataloging in Publication data Stroock, Daniel W. Probability theory : an analytic view/ Daniel W. Stroock. – 2nd ed. p. cm. Includes bibliographical references and index. ISBN 978-0-521-76158-1 (hardback) – ISBN 978-0-521-13250-3 (pbk.) 1. Probabilities. I. Title. QA 273.S763 2010 519.2–dc22 2010027652 ISBN 978-0-521-76158-1 Hardback ISBN 978-0-521-13250-3 Paperback Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party Internet Web sites referred to in this publication and does not guarantee that any content on such Web sites is, or will remain, accurate or appropriate.
This book is dedicated to my teachers: M. Kac, H.P. McKean, Jr., and S.R.S. Varadhan
Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Table of Dependence . . . . . . . . . . . . . . . . . . . . . . xxi Chapter 1 Sums of Independent Random 1.1 Independence . . . . . . . . . . . . 1.1.1. Independent σ-Algebras . . . . . . 1.1.2. Independent Functions . . . . . . . 1.1.3. The Rademacher Functions . . . . . Exercises for § 1.1 . . . . . . . . . . . . 1.2 The Weak Law of Large Numbers . . . 1.2.1. Orthogonal Random Variables . . . 1.2.2. Independent Random Variables . . . 1.2.3. Approximate Identities . . . . . . . Exercises for § 1.2 . . . . . . . . . . . . 1.3 Cram´er’s Theory of Large Deviations . . Exercises for § 1.3 . . . . . . . . . . . . 1.4 The Strong Law of Large Numbers . . . Exercises for § 1.4 . . . . . . . . . . . . 1.5 Law of the Iterated Logarithm . . . . Exercises for § 1.5 . . . . . . . . . . . .
Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Chapter 2 The Central Limit Theorem . . . . 2.1 The Basic Central Limit Theorem . . . . . . 2.1.1. Lindeberg’s Theorem . . . . . . . . . . 2.1.2. The Central Limit Theorem . . . . . . . Exercises for § 2.1 . . . . . . . . . . . . . . . 2.2 The Berry–Esseen Theorem via Stein’s Method 2.2.1. L1 -Berry–Esseen . . . . . . . . . . . . 2.2.2. The Classical Berry–Esseen Theorem . . . Exercises for § 2.2 . . . . . . . . . . . . . . . 2.3 Some Extensions of The Central Limit Theorem 2.3.1. The Fourier Transform . . . . . . . . . . 2.3.2. Multidimensional Central Limit Theorem . . 2.3.3. Higher Moments . . . . . . . . . . . . Exercises for § 2.3 . . . . . . . . . . . . . . . vii
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . .
1 1 1 4 5 7 14 14 15 16 20 22 31 35 42 49 56 59 60 60 62 65 71 72 75 81 82 82 84 87 90
viii
Contents
2.4 An Application to Hermite Multipliers 2.4.1. Hermite Multipliers . . . . . . . 2.4.2. Beckner’s Theorem . . . . . . . 2.4.3. Applications of Beckner’s Theorem Exercises for § 2.4 . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. 96 . 96 . 101 . 105 . 110
Chapter 3 Infinitely Divisible Laws . . . . . . . . . . . . . . . 115 3.1 Convergence of Measures on RN . . . 3.1.1. Sequential Compactness in M1 (RN ) 3.1.2. L´evy’s Continuity Theorem . . . . Exercises for § 3.1 . . . . . . . . . . . 3.2 The L´evy–Khinchine Formula . . . . 3.2.1. I(RN ) Is the Closure of P(RN ) . . 3.2.2. The Formula . . . . . . . . . . Exercises for § 3.2 . . . . . . . . . . . 3.3 Stable Laws . . . . . . . . . . . . 3.3.1. General Results . . . . . . . . . 3.3.2. α-Stable Laws . . . . . . . . . Exercises for § 3.3 . . . . . . . . . . . Chapter 4 L´ evy Processes
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
115 116 117 119 122 123 126 137 139 139 141 147
. . . . . . . . . . . . . . . . . . . 151
4.1 Stochastic Processes, Some Generalities . . . 4.1.1. The Space D(RN ) . . . . . . . . . . . 4.1.2. Jump Functions . . . . . . . . . . . . Exercises for § 4.1 . . . . . . . . . . . . . . 4.2 Discontinuous L´evy Processes . . . . . . . 4.2.1. The Simple Poisson Process . . . . . . . 4.2.2. Compound Poisson Processes . . . . . . 4.2.3. Poisson Jump Processes . . . . . . . . 4.2.4. L´evy Processes with Bounded Variation . 4.2.5. General, Non-Gaussian L´evy Processes . . Exercises for § 4.2 . . . . . . . . . . . . . . 4.3 Brownian Motion, the Gaussian L´evy Process 4.3.1. Deconstructing Brownian Motion . . . . 4.3.2. L´evy’s Construction of Brownian Motion . 4.3.3. L´evy’s Construction in Context . . . . . 4.3.4. Brownian Paths Are Non-Differentiable . 4.3.5. General L´evy Processes . . . . . . . . Exercises for § 4.3 . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . .
152 153 156 159 160 161 163 168 170 171 174 177 178 180 182 183 185 187
Contents
ix
Chapter 5 Conditioning and Martingales . . . . . . . . . . . . 193 5.1 Conditioning . . . . . . . . . . . . . . . . 5.1.1. Kolmogorov’s Definition . . . . . . . . . . 5.1.2. Some Extensions . . . . . . . . . . . . . Exercises for § 5.1 . . . . . . . . . . . . . . . . 5.2 Discrete Parameter Martingales . . . . . . . . 5.2.1. Doob’s Inequality and Marcinkewitz’s Theorem 5.2.2. Doob’s Stopping Time Theorem . . . . . . . 5.2.3. Martingale Convergence Theorem . . . . . . 5.2.4. Reversed Martingales and De Finetti’s Theory . 5.2.5. An Application to a Tracking Algorithm . . . Exercises for § 5.2 . . . . . . . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
193 194 198 202 205 206 212 214 217 221 226
Chapter 6 Some Extensions and Applications of Martingale Theory . . . . . . . . . . . . . . . . . . . . . . 233 6.1 Some Extensions . . . . . . . . . . . . . . . 6.1.1. Martingale Theory for a σ-Finite Measure Space 6.1.2. Banach Space–Valued Martingales . . . . . . Exercises for § 6.1 . . . . . . . . . . . . . . . . 6.2 Elements of Ergodic Theory . . . . . . . . . . 6.2.1. The Maximal Ergodic Lemma . . . . . . . . 6.2.2. Birkhoff’s Ergodic Theorem . . . . . . . . 6.2.3. Stationary Sequences . . . . . . . . . . . 6.2.4. Continuous Parameter Ergodic Theory . . . . Exercises for § 6.2 . . . . . . . . . . . . . . . . 6.3 Burkholder’s Inequality . . . . . . . . . . . . 6.3.1. Burkholder’s Comparison Theorem . . . . . 6.3.2. Burkholder’s Inequality . . . . . . . . . . Exercises for § 6.3 . . . . . . . . . . . . . . . . Chapter 7 Continuous Parameter Martingales 7.1 Continuous Parameter Martingales . . . . . . 7.1.1. Progressively Measurable Functions . . . . 7.1.2. Martingales: Definition and Examples . . . 7.1.3. Basic Results . . . . . . . . . . . . . . 7.1.4. Stopping Times and Stopping Theorems . . 7.1.5. An Integration by Parts Formula . . . . . Exercises for § 7.1 . . . . . . . . . . . . . . . 7.2 Brownian Motion and Martingales . . . . . . 7.2.1. L´evy’s Characterization of Brownian Motion 7.2.2. Doob–Meyer Decomposition, an Easy Case . 7.2.3. Burkholder’s Inequality Again . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
233 233 239 240 244 245 248 251 253 256 257 257 262 263
. . . . . . . . . 266 . . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
266 266 267 270 272 276 280 282 282 284 289
x
Contents Exercises for § 7.2 . . . . . . . . . . . . 7.3 The Reflection Principle Revisited . . . 7.3.1. Reflecting Symmetric L´evy Processes 7.3.2. Reflected Brownian Motion . . . . . Exercises for § 7.3 . . . . . . . . . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
290 292 292 294 298
Chapter 8 Gaussian Measures on a Banach Space . 8.1 The Classical Wiener Space . . . . . . . . . . . 8.1.1. Classical Wiener Measure . . . . . . . . . . 8.1.2. The Classical Cameron–Martin Space . . . . . Exercises for § 8.1 . . . . . . . . . . . . . . . . . 8.2 A Structure Theorem for Gaussian Measures . . . 8.2.1. Fernique’s Theorem . . . . . . . . . . . . . 8.2.2. The Basic Structure Theorem . . . . . . . . . 8.2.3. The Cameron–Marin Space . . . . . . . . . . Exercises for § 8.2 . . . . . . . . . . . . . . . . . 8.3 From Hilbert to Abstract Wiener Space . . . . . 8.3.1. An Isomorphism Theorem . . . . . . . . . . 8.3.2. Wiener Series . . . . . . . . . . . . . . . . 8.3.3. Orthogonal Projections . . . . . . . . . . . . 8.3.4. Pinned Brownian Motion . . . . . . . . . . . 8.3.5. Orthogonal Invariance . . . . . . . . . . . . Exercises for § 8.3 . . . . . . . . . . . . . . . . . 8.4 A Large Deviations Result and Strassen’s Theorem 8.4.1. Large Deviations for Abstract Wiener Space . . 8.4.2. Strassen’s Law of the Iterated Logarithm . . . . Exercises for § 8.4 . . . . . . . . . . . . . . . . . 8.5 Euclidean Free Fields . . . . . . . . . . . . . 8.5.1. The Ornstein–Uhlenbeck Process . . . . . . . 8.5.2. Ornstein–Uhlenbeck as an Abstract Wiener Space 8.5.3. Higher Dimensional Free Fields . . . . . . . . Exercises for § 8.5 . . . . . . . . . . . . . . . . . 8.6 Brownian Motion on a Banach Space . . . . . . . 8.6.1. Abstract Wiener Formulation . . . . . . . . . 8.6.2. Brownian Formulation . . . . . . . . . . . . 8.6.3. Strassen’s Theorem Revisited . . . . . . . . . Exercises for § 8.6 . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
299 299 299 303 306 306 306 307 310 313 317 317 318 322 326 328 330 337 337 340 342 343 344 346 349 355 358 358 361 363 365
Chapter 9 Convergence of Measures on a Polish Space 9.1 Prohorov–Varadarajan Theory . . . . . . . . . . . 9.1.1. Some Background . . . . . . . . . . . . . . . . 9.1.2. The Weak Topology . . . . . . . . . . . . . . . 9.1.3. The L´evy Metric and Completeness of M1 (E) . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
367 367 367 370 377
Contents Exercises for § 9.1 . . . . . . . . . . . . . . . 9.2 Regular Conditional Probability Distributions . 9.2.1. Fibering a Measure . . . . . . . . . . . 9.2.2. Representing L´evy Measures via the Itˆo Map Exercises for § 9.2 . . . . . . . . . . . . . . . 9.3 Donsker’s Invariance Principle . . . . . . . . 9.3.1. Donsker’s Theorem . . . . . . . . . . . 9.3.2. Rayleigh’s Random Flights Model . . . . . Exercise for § 9.3 . . . . . . . . . . . . . . .
xi . . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
381 386 388 390 392 392 393 396 399
Chapter 10 Wiener Measure and Partial Differential Equations 10.1 Martingales and Partial Differential Equations . . . . . . . . 10.1.1. Localizing and Extending Martingale Representations . . . . 10.1.2. Minimum Principles . . . . . . . . . . . . . . . . . . 10.1.3. The Hermite Heat Equation . . . . . . . . . . . . . . . 10.1.4. The Arcsine Law . . . . . . . . . . . . . . . . . . . . 10.1.5. Recurrence and Transience of Brownian Motion . . . . . . Exercises for § 10.1 . . . . . . . . . . . . . . . . . . . . . . 10.2 The Markov Property and Potential Theory . . . . . . . . . 10.2.1. The Markov Property for Wiener Measure . . . . . . . . . 10.2.2. Recurrence in One and Two Dimensions . . . . . . . . . . 10.2.3. The Dirichlet Problem . . . . . . . . . . . . . . . . . Exercises for § 10.2 . . . . . . . . . . . . . . . . . . . . . . 10.3 Other Heat Kernels . . . . . . . . . . . . . . . . . . . . 10.3.1. A General Construction . . . . . . . . . . . . . . . . . 10.3.2. The Dirichlet Heat Kernel . . . . . . . . . . . . . . . . 10.3.3. Feynman–Kac Heat Kernels . . . . . . . . . . . . . . . 10.3.4. Ground States and Associated Measures on Pathspace . . . 10.3.5. Producing Ground States . . . . . . . . . . . . . . . . Exercises for § 10.3 . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . .
400 400 401 404 405 407 411 415 416 416 417 418 426 429 429 431 436 439 445 449
Chapter 11 Some Classical Potential Theory . . . . . . 11.1 Uniqueness Refined . . . . . . . . . . . . . . . . . 11.1.1. The Dirichlet Heat Kernel Again . . . . . . . . . . 11.1.2. Exiting Through ∂reg G . . . . . . . . . . . . . . 11.1.3. Applications to Questions of Uniqueness . . . . . . . 11.1.4. Harmonic Measure . . . . . . . . . . . . . . . . Exercises for § 11.1 . . . . . . . . . . . . . . . . . . . 11.2 The Poisson Problem and Green Functions . . . . . . . 11.2.1. Green Functions when N ≥ 3 . . . . . . . . . . . 11.2.2. Green Functions when N ∈ {1, 2} . . . . . . . . . . Exercises for § 11.2 . . . . . . . . . . . . . . . . . . . 11.3 Excessive Functions, Potentials, and Riesz Decompositions
. . . . . . . . . . . .
456 456 456 459 463 468 472 475 476 477 486 487
. . . . . . . . . . . .
. . . . . . . . .
. . . . . . . . . . . .
. . . . . . . . . . . .
xii
Contents
11.3.1. Excessive Functions . . . . . . . . . 11.3.2. Potentials and Riesz Decomposition . Exercises for § 11.3 . . . . . . . . . . . . 11.4 Capacity . . . . . . . . . . . . . . 11.4.1. The Capacitory Potential . . . . . . 11.4.2. The Capacitory Distribution . . . . . 11.4.3. Wiener’s Test . . . . . . . . . . . 11.4.4. Some Asymptotic Expressions Involving Exercises for § 11.4 . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . Capacity . . . .
. . . . . . .
. . . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
488 489 496 497 497 500 504 507 514
Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 521
Preface
From the Preface to the First Edition When writing a graduate level mathematics book during the last decade of the twentieth century, one probably ought not inquire too closely into one’s motivation. In fact, if ones own pleasure from the exercise is not sufficient to justify the effort, then one should seriously consider dropping the project. Thus, to those who (either before or shortly after opening it) ask for whom was this book written, my pale answer is me; and, for this reason, I thought that I should preface this preface with an explanation of who I am and what were the peculiar educational circumstances that eventually gave rise to this somewhat peculiar book. My own introduction to probability theory began with a private lecture from H.P. McKean, Jr. At the time, I was a (more accurately, the) graduate student of mathematics at what was then called The Rockefeller Institute for Biological Sciences. My official mentor there was M. Kac, whom I had cajoled into becoming my adviser after a year during which I had failed to insert even one micro-electrode into the optic nerves of innumerable limuli. However, as I soon came to realize, Kac had accepted his role on the condition that it would not become a burden. In particular, he had no intention of wasting much of his own time on a reject from the neurophysiology department. On the other hand, he was most generous with the time of his younger associates, and that is how I wound up in McKean’s office. Never one to bore his listeners with a lot of dull preliminaries, McKean launched right into a wonderfully lucid explanation of P. L´evy’s interpretation of the infinitely divisible laws. I have to admit that my appreciation of the lucidity of his lecture arrived nearly a decade after its delivery, and I can only hope that my reader will reserve judgment of my own presentation for an equal length of time. In spite of my perplexed state at the end of McKean’s lecture, I was sufficiently intrigued to delve into the readings that he suggested at its conclusion. Knowing that the only formal mathematics courses that I would be taking during my graduate studies would be given at N.Y.U. and guessing that those courses would be oriented toward partial differential equations, McKean directed me to material which would help me understand the connections between partial differential equations and probability theory. In particular, he suggested that I start with the, then recently translated, two articles by E.B. Dynkin which had appeared originally in the famous 1956 volume of Teoriya Veroyatnostei i ee Primeneniya. Dynkin’s articles turned out to be a godsend. They were beautifully crafted to xiii
xiv
Preface
tell the reader enough so that he could understand the ideas and not so much that he would become bored by them. In addition, they gave me an introduction to a host of ideas and techniques (e.g., stopping times and the strong Markov property), all of which Kac himself consigned to the category of overelaborated measure theory. In fact, it would be reasonable to say that my thesis was simply the application of techniques which I picked up from Dynkin to a problem that I picked up by reading some notes by Kac. Of course, along the way I profited immeasurably from continued contact with McKean, a large number of courses at N.Y.U. (particularly ones taught by M. Donsker, F. John, and L. Nirenberg), and my increasingly animated conversations with S.R.S. Varadhan. As I trust the preceding description makes clear, my graduate education was anything but deprived; I had ready access to some of the very best analysts of the day. On the other hand, I never had a proper introduction to my field, probability theory. The first time that I ever summed independent random variables was when I was summing them in front of a class at N.Y.U. Thus, although I now admire the magnificent body of mathematics created by A.N. Kolmogorov, P. L´evy, and the other twentieth-century heroes of the field, I am not a dyed-in-the-wool probabilist (i.e., what Donsker would have called a true coin-tosser). In particular, I have never been able to develop sufficient sensitivity to the distinction between a proof and a probabilistic proof. To me, a proof is clearly probabilistic only if its punch-line comes down to an argument like P (A) ≤ P (B) because A ⊆ B; and there are breathtaking examples of such arguments. However, to base an entire book on these examples would require a level of genius that I do not possess. In fact, I myself enjoy probability theory best when it is inextricably interwoven with other branches of mathematics and not when it is presented as an entity unto itself. For this reason, the reader should not be surprised to discover that he finds some of the material presented in this book does not belong here; but I hope that he will make an effort to figure out why I disagree with him. Preface to the Second Edition My favorite “preface to a second edition” is the one that G.N. Watson wrote for the second edition of his famous treatise on Bessel functions. The first edition appeared in 1922, the second came out in 1941, and Watson had originally intended to stay abreast of developments and report on them in the second edition. However, in his preface to the second edition Watson admits that his interest in the topic had “waned” during the intervening years and apologizes that, as a consequence, the new edition contains less new material than he had thought it would. My excuse for not incorporating more new material into this second edition is related to but somewhat different from Watson’s. In my case, what has waned is not my interest in probability theory but instead my ability to assimilate the transformations that the subject has undergone. When I was a student,
Preface
xv
probabilists were still working out the ramifications of Kolmogorov’s profound insights into the connections between probability and analysis, and I have spent my career investigating and exploiting those connections. However, about the time when the first edition of this book was published, probability theory began a return to its origins in combinatorics, a topic in which my abilities are woefully deficient. Thus, although I suspect that, for at least a decade, the most exciting developments in the field will have a strong combinatorial component, I have not attempted to prepare my readers for those developments. I repeat that my decision not to incorporate more combinatorics into this new edition in no way reflects my assessment of the direction in which probability is likely to go but instead reflects my assessment of my own inability to do justice to the beautiful combinatorial ideas that have been introduced in the recent past. In spite of the preceding admission, I believe that the material in this book remains valuable and that, no matter how probability theory evolves, the ideas and techniques presented here will play an important role. Furthermore, I have made some substantive changes. In particular, I have given more space to infinitely divisible laws and their associated L´evy processes, both of which are now developed in RN rather than just in R. In addition, I have added an entire chapter devoted to Gaussian measures in infinite dimensions from the perspective of the Segal–Gross school. Not only have recent developments in Malliavin calculus and conformal field theory sparked renewed interest in this topic, but it seems to me that most modern texts pay either no or too little attention to this beautiful material. Missing from the new edition is the treatment of singular integrals. I included it in the first edition in the hope that it would elucidate the similarity between cancellations that underlie martingale theory, especially Burkholder’s Inequality, and Calderon–Zygmund theory. I still believe that these similarities are worth thinking about, but I have decided that my explanation of them led me too far astray and was more of a distraction than a pedagogically valuable addition. Besides those mentioned above, minor changes have been made throughout. For one thing, I have spent a lot of time correcting old errors and, undoubtedly, inserting new ones. Secondly, I have made several organizational changes as well as others that are remedial. A summary of the contents follows. Summary 1: Chapter 1 contains a sampling of the standard, pointwise convergence theorems dealing with partial sums of independent random variables. These include the Weak and Strong Laws of Large Numbers as well as Hartman–Wintner’s Law of the Iterated Logarithm. In preparation for the Law of the Iterated Logarithm, Cram´er’s theory of large deviations from the Law of Large Numbers is developed in § 1.4. Everything here is very standard, although I feel that my passage from the bounded to the general case of the Law of the Iterated Logarithm has been
xvi
Preface
considerably smoothed by the ideas that I learned during a conversation with M. Ledoux. 2: The whole of Chapter 2 is devoted to the classical Central Limit Theorem. After an initial (and slightly flawed) derivation of the basic result via moment considerations, Lindeberg’s general version is derived in § 2.1. Although Lindeberg’s result has become a sine qua non in the writing of probability texts, the Berry–Esseen estimate has not. Indeed, until recently, the Berry–Esseen estimate required a good many somewhat tedious calculations with characteristic functions (i.e., Fourier transforms), and most recent authors seem to have decided that the rewards did not justify the effort. I was inclined to agree with them until P. Diaconis brought to my attention E. Bolthausen’s adaptation of C. Stein’s techniques (the so-called Stein’s method) to give a proof that is not only brief but also, to me, aesthetically pleasing. In any case, no use of Fourier methods is made in the derivation given in § 2.2. On the other hand, Fourier techniques are introduced in § 2.3, where it is shown that even elementary Fourier analytic tools lead to important extensions of the basic Central Limit Theorem to more than one dimension. Finally, in § 2.4, the Central Limit Theorem is applied to the study of Hermite multipliers and (following Wm. Beckner) is used to derive both E. Nelson’s hypercontraction estimate for the Mehler kernel as well as Beckner’s own estimate for the Fourier transform. I am afraid that, with this flagrant example of the sort of thing that does not belong here, I may be trying the patience of my purist colleagues. However, I hope that their indignation will be somewhat assuaged by the fact that the rest of the book is essentially independent of the material in § 2.4. 3: This chapter is devoted to the study of infinitely divisible laws. It begins in § 3.1 with a few refinements (especially The L´evy Continuity Theorem) of the Fourier techniques introduced in § 2.3. These play a role in § 3.2, where the L´evy–Khinchine formula is first derived and then applied to the analysis of stable laws. 4: In Chapter 4 I construct the L´evy processes (a.k.a. independent increment processes) corresponding to infinitely divisible laws. Secton 4.1 provides the requisite information about the pathspace D(RN ) of right-continuous paths with left limits, and § 4.2 gives the construction of L´evy processes with discontinuous paths, the ones corresponding to infinitely divisible laws having no Gaussian part. Finally, in § 4.3 I construct Brownian motion, the L´evy process with continuous paths, following the prescription given by L´evy. 5: Because they are not needed earlier, conditional expectations do not appear until Chapter 5. The advantage gained by this postponement is that, by the time I introduce them, I have an ample supply of examples to which conditioning can be applied; the disadvantage is that, with considerable justice, many probabilists feel that one is not doing probability theory until one is conditioning. Be that as it may, Kolmogorov’s definition is given in § 5.1 and is shown
Preface
xvii
to extend naturally both to σ-finite measure spaces as well as to random variables with values in a Banach space. Section 5.2 presents Doob’s basic theory of real-valued, discrete parameter martingales: Doob’s Inequality, his Stopping Time Theorem, and his Martingale Convergence Theorem. In the last part of § 5.2, I introduce reversed martingales and apply them to DeFinetti’s theory of exchangeable random variables. 6: Chapter 6 opens with extensions of martingale theory in two directions: to σ-finite measures and to random variables with values in a Banach space. The results in § 6.1 are used in § 6.2 to derive Birkhoff’s Individual Ergodic Theorem and a couple of its applications. Finally, in § 6.3 I prove Burkholder’s Inequality for martingales with values in a Hilbert space. The derivation that I give is essentially the same as Burkholder’s second proof, the one that gives optimal constants. 7: Section 7.1 provides a brief introduction to the theory of martingales with a continuous parameter. As anyone at all familiar with the topic knows, anything approaching a full account of this theory requires much more space than a book like this can give it. Thus, I deal with only its most rudimentary aspects, which, fortunately, are sufficient for the applications to Brownian motion that I have in mind. Namely, in § 7.2 I first discuss the intimate relationship between continuous martingales and Brownian motion (L´evy’s martingale characterization of Brownian motion), then derive the simplest (and perhaps most widely applied) case of the Doob–Meyer Decomposition Theory, and finally show what Burkholder’s Inequality looks like for continuous martingales. In the concluding section, § 7.3, the results in §§ 7.1–7.2 are applied to derive the Reflection Principle for Brownian motion. 8: In § 8.1 I formulate the description of Brownian motion in terms of its Gaussian, as opposed to its independent increment, properties. More precisely, following Segal and Gross, I attempt to convince the reader that Wiener measure (i.e., the distribution of Brownian motion) would like to be the standard Gauss measure on the Hilbert space H 1 (RN ) of absolutely continuous paths with a square integrable derivative, but, for technical reasons, cannot live there and has to settle for a Banach space in which H 1 (RN ) is densely embedded. Using Wiener measure as the model, in § 8.2 I show that, at an abstract level, any non-degenerate, centered Gaussian measure on an infinite dimensional, separable Banach space shares the same structure as Wiener measure in the sense that there is always a densely embedded Hilbert space, known as the Cameron– Martin space, for which it would like to be the standard Gaussian measure but on which it does not fit. In order to carry out this program, I need and prove Fernique’s Theorem for Gaussian measures on a Banach space. In § 8.3 I begin by going in the opposite direction, showing how to pass from a Hilbert space H to a Gaussian measure on a Banach space E for which H is the Cameron–Martin space. The rest of § 8.3 gives two applications: one to “pinned Brownian” motion
xviii
Preface
and the second to a very general statement of orthogonal invariance for Gaussian measures. The main goal of § 8.4 is to prove a large deviations result, known as Schilder’s Theorem, for abstract Wiener spaces; and once I have Schilder’s Theorem, I apply it to derive a version of Strassen’s Law of the Iterated Logarithm. Starting with the Ornstein–Uhlenbeck process, I construct in § 8.5 a family of Gaussian measures known in the mathematical physics literature as Euclidean free fields. In the final section, § 8.6, I first show how to construct Banach space– valued Brownian motion and then derive the original form of Strassen’s Law of the Iterated Logarithm in that context. 9: The central topic here is the abstract theory of weak convergence of probability measures on a Polish space. The basic theory is developed in § 9.1. In § 9.2 I apply the theory to prove the existence of regular conditional probability distributions, and in § 9.3 I use it to derive Donsker’s Invariance Principle (i.e., the pathspace statement of the Central Limit Theorem). 10: Chapter 10 is an introduction to the connections between probability theory and partial differential equations. At the beginning of § 10.1 I show that martingale theory provides a link between probability theory and partial differential equations. More precisely, I show how to represent in terms of Wiener integrals solutions to parabolic and elliptic partial differential equations in which the Laplacian is the principal part. In the second part of § 10.1, I use this link to calculate various Wiener integrals. In § 10.2 I introduce the Markov property of Wiener measure and show how it not only allows one to evaluate other Wiener integrals in terms of solutions to elliptic partial differential equations but also enables one to prove interesting facts about solutions to such equations as a consequence of their representation in terms of Wiener integrals. Continuing in the same spirit, I show in § 10.2 how to represent solutions to the Dirichlet problem in terms of Wiener integrals, and in § 10.3 I use Wiener measure to construct and discuss heat kernels related to the Laplacian. 11: The final chapter is an extended example of the way in which probability theory meshes with other branches of analysis, and the example that I have chosen is the marriage between Brownian motion and classical potential theory. Like an ideal marriage, this one is simultaneously intimate and mutually beneficial to both partners. Indeed, the more one knows about it, the more convinced one becomes that the properties of Brownian paths are a perfect reflection of properties of harmonic functions, and vice versa. In any case, in § 11.1 I sharpen the results in § 10.2.3 and show that, in complete generality, the solution to the Dirichlet problem is given by the Wiener integral of the boundary data evaluated at the place where Brownian paths exit from the region. Next, in § 11.2, I discuss the Green function for a region and explain how its existence reflects the recurrence and transience properties of Brownian paths. In preparation for § 11.4, § 11.3 is devoted to the Riesz Decomposition Theorem for excessive functions. Finally, in § 11.4, I discuss the capacity of regions, derive Chung’s representation of the
Preface
xix
capacitory measure in terms of the last place where a Brownian path visits a region, apply the probabilistic interpretation of capacity to give a derivation of Wiener’s test for regularity, and conclude with two asymptotic calculations in which capacity plays a crucial role. Suggestions about the Use of This Book In spite of the realistic assessment contained in the first paragraph of its preface, when I wrote the first edition of this book I harbored the na¨ıve hope that it might become the standard graduate text in probability theory. By the time that I started preparing the second edition, I was significantly older and far less na¨ıve about its prospects. Although the first edition has its admirers, it has done little to dent the sales record of its competitors. In particular, the first edition has seldom been adopted as the text for courses in probability, and I doubt that the second will be either. Nonetheless, I close this preface with a few suggestions for anyone who does choose to base a course on it. I am well aware that, except for those who find their way into the poorly stocked library of some prison camp, few copies of this book will be read from cover to cover. For this reason, I have attempted to organize it in such a way that, with the help of the table of dependence that follows, a reader can select a path which does not require his reading all the sections preceding the information he is seeking. For example, the contents of §§ 1.1–1.2, § 1.4, § 2.1, § 2.3, and § 5.1– 5.2 constitute the backbone of a one semester, graduate level introduction to probability theory. What one attaches to this backbone depends on the speed with which these sections are covered and the content of the courses for which the course is the introduction. If the goal is to prepare the students for a career as a “quant” in what is left of the financial industry, an obvious choice is § 4.3 and as much of Chapter 7 as time permits, thereby giving one’s students a reasonably solid introduction to Brownian motion. On the other hand, if one wants the students to appreciate that white noise is not the only noise that they may encounter in life, one might defer the discussion of Brownian motion and replace it with the material in Chapter 3 and §§ 4.1–4.2. Alternatively, one might use this book in a more advanced course. An introduction to stochastic processes with an emphasis on their relationship to partial differential equations can be constructed out of Chapters 6, 7, 10, and 11, and § 4.3 combined with Chapter 8 could be used to provide background for a course on Gaussian processes. Whatever route one takes through this book, it will be a great help to your students for you to suggest that they consult other texts. Indeed, it is a familiar fact that the third book one reads on a subject is always the most lucid, and so one should suggest at least two other books. Among the many excellent choices available, I mention Wm. Feller’s An Introduction to Probability Theory and Its Applications, Vol. II, and M. Lo´eve’s classic Probability Theory. In addition, for
xx
Preface
background, precision (including accuracy of attribution), and supplementary material, R. Dudley’s Real Analysis and Probability is superb.
Table of Dependence
§§11.1–11.4
§§10.1–10.3
§§7.4–7.5
§§8.1–8.5
§9.3
§§7.1–7.3
§§3.2–3.3
§§4.1–4.3
§§6.1 & 6.3
§6.2
§2.3
§§3.1 & 3.4
§§5.1 & 5.3
§1.5
§§2.1 & 2.2
§§1.1–1.4
§§9.1–9.2
xxi
Chapter 1 Sums of Independent Random Variables
In one way or another, most probabilistic analysis entails the study of large families of random variables. The key to such analysis is an understanding of the relations among the family members; and of all the possible ways in which members of a family can be related, by far the simplest is when there is no relationship at all! For this reason, I will begin by looking at families of independent random variables. § 1.1 Independence In this section I will introduce Kolmogorov’s way of describing independence and prove a few of its consequences. § 1.1.1. Independent σ-Algebras. Let (Ω, F, P) be a probability space (i.e., Ω is a nonempty set, F is a σ-algebra over Ω, and P is a non-negative measure on the measurable space (Ω, F) having total mass 1), and, for each i from the (non-empty) index set I, let Fi be a sub-σ-algebra of F. I will say that the σ-algebras Fi , i ∈ I, are mutually P-independent, or, less precisely, P-independent, if, for every finite subset {i1 , . . . , in } of distinct elements of I and every choice of Aim ∈ Fim , 1 ≤ m ≤ n, (1.1.1)
P(Ai1 ∩ · · · ∩ Ain ) = P(Ai1 ) · · · P(Ain ).
In particular, if {Ai : i ∈ I} is a family of sets from F, I will say that Ai , i ∈ I, are P-independent if the associated σ-algebras Fi = {∅, Ai , Ai {, Ω}, i ∈ I, are. To gain an appreciation for the intuition on which this definition is based, it is important to notice that independence of the pair A1 and A2 in the present sense is equivalent to P(A1 ∩ A2 ) = P(A1 )P(A2 ), the classical definition that one encounters in elementary treatments. Thus, the notion of independence just introduced is no more than a simple generalization of the classical notion of independent pairs of sets encountered in non-measure theoretic presentations, and therefore the intuition that underlies the elementary notion applies equally well to the definition given here. (See Exercise 1.1.8 for more information about the connection between the present definition and the classical one.) As will become increasing evident as we proceed, infinite families of independent objects possess surprising and beautiful properties. In particular, mutually 1
2
1 Sums of Independent Random Variables
independent σ-algebras tend to fill up space in a sense made precise by the following beautiful thought experiment designed by A.N. Kolmogorov. Let I be any index set, take F∅ = {∅, Ω}, and, for each non-empty subset Λ ⊆ I, let ! FΛ =
_
Fi ≡ σ
i∈Λ
[
Fi
i∈I
S be the σ-algebra generated by i∈Λ Fi (i.e., FΛ is the smallest σ-algebra conS taining i∈Λ Fi ). Next, define the tail σ-algebra T to be the intersection over all finite Λ ⊆ I of the σ-algebras FΛ{ . When I itself is finite, T = {∅, Ω} and is therefore P-trivial in the sense that P(A) ∈ {0, 1} for every A ∈ T . The interesting remark made by Kolmogorov is that even when I is infinite, T is P-trivial whenever the original Fi ’s are P-independent. To see this, for a given non-empty Λ ⊆ I, let CΛ denote the collection of sets of the form Ai1 ∩ · · · Ain where {i1 , . . . , in } are distinct elements of Λ and Aim ∈ Fim for each 1 ≤ m ≤ n. Clearly CΛ is closed under intersection and FΛ = σ(CΛ ). In addition, by assumption, P(A ∩ B) = P(A)P(B) for all A ∈ CΛ and B ∈ CΛ{ . Hence, by Exercise 1.1.12, FΛ is independent of FΛ{ . But this means that T is independent of FF for every finite F ⊆ I, and therefore, again by Exercise 1.1.12, T is independent of [ FI = σ {FF : F a finite subset of Λ} . Since T ⊆ FI , this implies that T is independent of itself ; that is, P(A ∩ B) = P(A)P(B) for all A, B ∈ T . Hence, for every A ∈ T , P(A) = P(A)2 , or, equivalently, P(A) ∈ {0, 1}, and so I have now proved the following famous result. Theorem 1.1.2 (Kolmogorov’s 0–1 Law). Let {Fi : i ∈ I} be a family of P-independent sub-σ-algebras of (Ω, F, P), and define the tail σ-algebra T accordingly, as above. Then, for every A ∈ T , P(A) is either 0 or 1. To develop a feeling for the kind of conclusions that can be drawn from Kolmogorov’s 0–1 Law (cf. Exercises 1.1.18 and 1.1.19 as well), let {An : n ≥ 1} be a sequence of subsets of Ω, and recall the notation
lim An ≡
n→∞
∞ [ \
An = ω : ω ∈ An for infinitely many n ∈ Z+ .
m=1 n≥m
Obviously, limn→∞ An is measurable with respect to the tail field determined by the sequence of σ-algebras {∅, An , An {, Ω}, n ∈ Z+ ; and therefore, if the An ’s are P-independent elements of F, then P lim An ∈ {0, 1}. n→∞
§ 1.1 Independence
3
In words, this conclusion can be summarized as follows: for any sequence of P-independent events An , n ∈ Z+ , either P-almost every ω ∈ Ω is in infinitely many An ’s or P-almost every ω ∈ Ω is in at most finitely many An ’s. A more quantitative statement of this same fact is contained in the second part of the following useful result. Let {An : n ∈ Z+ } ⊆ F be given.
Lemma 1.1.3 (Borel–Cantelli Lemma). Then ∞ X
(1.1.4)
P(An ) < ∞ =⇒ P
n=1
lim An = 0.
n→∞
In fact, if the An ’s are P-independent sets, then ∞ X
(1.1.5)
P(An ) = ∞ ⇐⇒ P
n=1
lim An = 1.
n→∞
(See part (iii) of Exercise 5.2.40 and Lemma 11.4.14 for generalizations.) Proof: The first assertion, which is due to E. Borel, is an easy application of countable additivity. Namely, by countable additivity, [ X An ≤ lim P(An ) = 0 P lim An = lim P n→∞
m→∞
m→∞
n≥m
n≥m
P∞
n=1 P(An ) < ∞. To complete the proof of (1.1.5) when the An ’s are independent, note that, by countable additivity, P limn→∞ An = 1 if and only if
if
\ ∞ \ [ lim An { = 0. lim P An { = P An { = P
m→∞
n→∞
m=1 n≥m
n≥m
But, by independence and another application of countable additivity, for any given m ≥ 1 we have that
P
∞ \ n=m
! An {
= lim
N →∞
N Y n=m
"
1 − P(An ) ≤ lim exp − N →∞
N X
# P An
=0
n=m
P∞ if n=1 P(An ) = ∞. (In the preceding, I have used the trivial inequality 1 − t ≤ e−t , t ∈ [0, ∞).) A second, and perhaps more transparent, way of dealing with the contents of the preceding is to introduce the non-negative random variable N (ω) ∈ Z+ ∪
4
1 Sums of Independent Random Variables
{∞}, that counts theP number of n ∈ Z+ such that ω ∈ An . Then, by Tonelli’s ∞ 1 P Theorem, E [N ] = n=1 P(An ), and so Borel’s contribution is equivalent to the EP [N ] < ∞ =⇒ P(N < ∞) = 1, which is obvious, whereas Cantelli’s contribution is that, for mutually independent An ’s, P(N < ∞) =⇒ EP [N ] < ∞, which is not obvious. § 1.1.2. Independent Functions. Having described what it means for the σalgebras to be P-independent, I will now transfer the notion to random variables on (Ω, F, P). Namely, for each i ∈ I, let Xi be a random variable (i.e., a measurable function on (Ω, F)) with values in the measurable space (Ei , Bi )). I will say that the random variables Xi , i ∈ I, are (mutually) P-independent if the σ-algebras σ(Xi ) = Xi−1 Bi ≡ Xi−1 (Bi ) : Bi ∈ Bi , i ∈ I, are P-independent. If B(E; R) = B (E, B); R denotes the space of bounded measurable R-valued functions on the measurable space (E, B), then it should be clear that P-independence of {Xi : i ∈ I} is equivalent to the statement that EP fi1 ◦ Xi1 · · · fin ◦ Xin = EP fi1 ◦ Xi1 · · · EP fin ◦ Xin for all finite subsets {i1 , . . . , in } of distinct elements of I and all choices of fi1 ∈ B Ei1 ; R , . . . , fin ∈ B Ein ; R . Finally, if 1A given by 1A (ω) ≡
1
if ω ∈ A
0
if ω ∈ /A
denotes the indicator function of the set A ⊆ Ω, notice that the family of sets {Ai : i ∈ I} ⊆ F is P-independent if and only if the random variables 1Ai , i ∈ I, are P-independent. Thus far I have discussed only the abstract notion of independence and have yet to show that the concept is not vacuous. In the modern literature, the standard way to construct lots of independent quantities is to take products of probability spaces. Namely, if E , B , µ is a probability space for each i ∈ I, i i i Q one sets Ω = i∈I Ei ; defines πi : Ω −→ Ei to be the natural projection map W for each i ∈ I; takes Fi = πi−1 (Bi ), i ∈ I, and F = i∈I Fi ; and shows that there is a unique probability measure P on (Ω, F) with the properties that P πi−1 Γi = µi Γi ) 1
for all
i ∈ I and Γi ∈ Bi
Throughout this book, I use EP [X, A] to denote the expected value under P of X over the set R A. That is, EP [X, A] = X dP. Finally, when A = Ω, I will write EP [X]. Tonelli’s Theorem A is the version of Fubini’s Theorem for non-negative functions. Its virtue is that it applies whether or not the integrand is integrable.
§ 1.1 Independence
5
and the σ-algebras Fi , i ∈ I, are P-independent. Although this procedure is extremely powerful, it is rather mechanical. For this reason, I have chosen to defer the details of the product construction to Exercises 1.1.14 and 1.1.16 and to, instead, spend the rest of this section developing a more hands-on approach to constructing independent sequences of real-valued random variables. Indeed, although the product method is more ubiquitous and has become the construction of choice, the one that I am about to present has the advantage that it shows independent random variables can arise “naturally” and even in a familiar places. § 1.1.3. The Rademacher Functions. Until further notice, take (Ω, F) = [0, 1), B[0,1) (when E is a metric space, I use BE to denote the Borel field over E) and P to be the restriction λ[0,1) of Lebesgue measure λR to [0, 1). Next define the Rademacher functions Rn , n ∈ Z+ , on Ω as follows. Take the integer part btc of t ∈ R to be the largest integer dominated by t, and consider the function R : R −→ {−1, 1} given by −1 if t − btc ∈ 0, 12 R(t) = 1 if t − btc ∈ 12 , 1 .
The function Rn is then defined on [0, 1) by Rn (ω) = R 2n−1 ω ,
n ∈ Z+ and ω ∈ [0, 1).
I will now show that the Rademacher functions are P-independent. To this end, first note that every real-valued function f on {−1, 1} is of the form α + βx, x ∈ {−1, 1}, for some pair of real numbers α and β. Thus, all that I have to show is that EP (α1 + β1 R1 ) · · · (αn + βn Rn ) = α1 · · · αn for any n ∈ Z+ and (α1 , β1 ), . . . , (αn , βn ) ∈ R2 . Since this is obvious when n = 1, I will assume that it holds for n and need only check that it must also hold for n + 1, and clearly this comes down to checking that EP F (R1 , . . . , Rn ) Rn+1 = 0 for any F : {−1, 1}n −→ R. But (R1 , . . . , Rn ) is constant on each interval m m+1 , 0 ≤ m < 2n , Im,n ≡ n , 2n 2
whereas Rn+1 integrates to 0 on each Im,n . Hence, by writing the integral over Ω as the sum of integrals over the Im,n ’s, we get the desired result. At this point I have produced a countably infinite sequence of independent Bernoulli random variables (i.e., two-valued random variables whose range is usually either {−1, 1} or {0, 1}) with mean value 0. In order to get more general
6
1 Sums of Independent Random Variables
random variables, I will combine our Bernoulli random variables together in a clever way. Recall that a random variable U is said to be uniformly distributed on the finite interval [a, b] if P(U ≤ t) =
t−a b−a
for t ∈ [a, b].
Lemma 1.1.6. Let {X` : ` ∈ Z+ } be a sequence of P-independent {0, 1}valued Bernoulli random variables with mean value 12 on some probability space (Ω, F, P), and set ∞ X X` . U= 2` `=1
Then U is uniformly distributed on [0, 1]. Proof: Because the assertion only involves properties of distributions, it will be proved in general as soon as I prove it for a particular realization of independent, mean value 12 , {0, 1}-valued Bernoulli random variables. In particular, by the preceding discussion, I need only consider the random variables
n (ω) ≡
1 + Rn (ω) , 2
n ∈ Z+ and ω ∈ [0, 1),
on [0, 1), B[0,1) , λ[0,1) . But, as is easily checked (cf. part (i) of Exercise 1.1.11), P∞ for each ω ∈ [0, 1], ω = n=1 2−n n (ω). Hence, the desired conclusion is trivial in this case. Now let (k, `) ∈ Z+ × Z+ 7−→ n(k, `) ∈ Z+ be any one-to-one mapping of + Z × Z+ onto Z+ , and set Yk,` =
1 + Rn(k,`) , 2
2 (k, `) ∈ Z+ .
Clearly, each Yk,` is a {0, 1}-valued, Bernoulli random variable with mean value 1 + 2 is P-independent. Hence, by Lemma 2 , and the family Yk,` : (k, `) ∈ Z 1.1.6, each of the random variables
Uk ≡
∞ X Yk,` `=1
2`
,
k ∈ Z+ ,
is uniformly distributed on [0, 1). In addition, the Uk ’s are obviously mutually independent. Hence, I have now produced a sequence of mutually independent random variables, each of which is uniformly distributed on [0, 1). To complete our program, I use the time-honored transformation that takes a uniform random
Exercises for § 1.1
7
variable into an arbitrary one. Namely, given a distribution function F on R (i.e., F is a right-continuous, non-decreasing function that tends to 0 at −∞ and 1 at +∞), define F −1 on [0, 1] to be the left-continuous inverse of F . That is, F −1 (t) = inf{s ∈ R : F (s) ≥ t}, t ∈ [0, 1]. (Throughout, the infemum over the empty set is taken to be +∞.) It is then an easy matter to check that when U is uniformly distributed on [0, 1) the random variable X = F −1 ◦ U has distribution function F : P(X ≤ t) = F (t),
t ∈ R.
Hence, after combining this with what we already know, I have now completed the proof of the following theorem. Theorem 1.1.7. Let Ω = [0, 1), F = B[0,1) , and P = λ[0,1) . Then, for any sequence {Fk : k ∈ Z+ } of distribution functions on R, there exists a sequence {Xk : k ∈ Z+ } ofP-independent random variables on (Ω, F, P) with the property that P Xk ≤ t = Fk (t), t ∈ R, for each k ∈ Z+ . Exercises for § 1.1 Exercise 1.1.8. As I pointed out, P A1 ∩ A2 = P A1 )P A2 if and only if the σ-algebra generated by A1 is P-independent of the one generated by A2 . Construct an example to show that the analogous statement is false when dealing with three, instead of two, sets. That is, just because P A1 ∩ A2 ∩ A3 = P A1 P A2 P A3 , show that it is not necessarily true that the three σ-algebras generated by A1 , A2 , and A3 are P-independent. Exercise 1.1.9. This exercise deals with three elementary, but important, properties of independent random variables. Throughout, (Ω, F, P) is a given probability space. (i) Let X1 and X2 be a pair of P-independent random variables with values in the measurable spaces (E1 , B1 ) and (E2 , B2 ), respectively. Given a B1 × B2 measurable function F : E1 × E2 −→ R that is bounded below, use Tonelli’s or Fubini’s Theorem to show that x2 ∈ E2 7−→ f (x2 ) ≡ EP F X1 , x2 ∈ R is B2 -measurable and that EP F X1 , X2 = EP f X2 . (ii) Suppose that X1 , . . . , Xn are P-independent, real-valued random variables. If each of the Xm ’s is P-integrable, show that X1 · · · Xn is also P-integrable and that EP X1 · · · Xn = EP X1 · · · EP Xn . (iii) Let {Xn : n ∈ Z+ } be a sequence of independent random variables taking values in some separable metric space E. If P(X n = x) = 0 for all x ∈ E and n ∈ Z+ , show that P Xm = Xn for some m 6= n = 0.
8
1 Sums of Independent Random Variables
Exercise 1.1.10. As an application of Lemma 1.1.6 and part (ii) of Exercise 1.1.9, prove the identity sin z = z
∞ Y
cos 2−n z
for all z ∈ C.
n=1
Exercise 1.1.11. Define {n (ω) : n ≥ 1} for ω ∈ [0, 1) as in the proof of Lemma 1.1.6. +
Z (i) Show that {P n (ω) : n ≥ 1} is the unique sequence {αn : n ≥ 1} ⊆ {0, 1} n such that ω − m=1 2−m αm < 2−n , and conclude that 1 (ω) = b2ωc and n+1 (ω) = b2n+1 ωc − 2b2n ωc for n ≥ 1.
(ii) Define F : [0, 1) −→ [0, 1)2 by F (ω) =
∞ X
2
−n
2n−1 (ω),
n=1
∞ X
! −n
2
2n (ω) ,
n=1
and show that λ[0,1)2 = F∗ λ[0,1) . That is, λ[0,1) {ω : F (ω) ∈ Γ} = λ2[0,1) (Γ) for all Γ ∈ B[0,1)2 . (iii) Define G : [0, ∞)2 −→ [0, 1) by ∞ X 2n (ω1 ) + n (ω2 ) , G (ω1 , ω2 ) = 4n n=1
and show that λ[0,1) = G∗ λ[0,1)2 . Parts (ii) and (iii) are special cases of a general principle that says, under very general circumstances, measures can be transformed into one another. Exercise 1.1.12. Given a non-empty set Ω, recall2 that a collection C of subsets of Ω is called a π-system if C is closed under finite intersections. At the same time, recall that a collection L is called a λ-system if Ω ∈ L, A ∪ B ∈ L whenever A and B are disjoint members S∞ of L, B \ A ∈ L whenever A and B are members of L with A ⊆ B, and 1 An ∈ L whenever {An : n ≥ 1} is a non-decreasing sequence of members of L. Finally, recall (cf. Lemma 3.1.3 in my Concise Introduction to the Theory of Integration) that if C is a π-system, then the σ-algebra σ(C) generated by C is the smallest L-system L ⊇ C. Show that if C is a π-system and F = σ(C), then two probability measures P and Q are equal on F if they are equal on C. Next use this to see that if {Ci : i ∈ I} is a family of π-systems contained in F and if (1.1.1) holds when the Ai ’s are from the Ci ’s, then the family of σ-algebras {σ(Ci ) : i ∈ I} is independent. 2
See, for example, § 3.1 in the author’s A Concise Introduction to the Theory of Integration, Third Edition, Birkh¨ auser (1998).
Exercises for § 1.1
9
Exercise 1.1.13. In this exercise I discuss two criteria for determining when random variables on the probability space (Ω, F, P) are independent. (i) Let X1 , . . . , Xn be bounded, real-valued random variables. Using Weierstrass’s Approximation Theorem, show that the Xm ’s are P-independent if and only if EP X1m1 · · · Xnmn = EP X1m1 · · · EP Xnmn for all m1 , . . . , mn ∈ N. (ii) Let X : Ω −→ Rm and Y : Ω −→ Rn be random variables. Show that X and Y are P-independent if and only if h√ i P E exp −1 α, X Rm + β, Y Rn
h√ h√ i i P = E exp −1 α, X Rm E exp −1 β, Y Rn P
for all α ∈ Rm and β ∈ Rn . Hint: The only if assertion is obvious. To prove the if assertion, first check that X and Y are independent if EP f (X) g(Y) = EP f (X) EP g(Y) for all f ∈ Cc∞ Rm ; C and g ∈ Cc∞ Rn ; C . Second, given such f and g, apply elementary Fourier analysis to write Z Z √ √ −1 (α,x)Rm ϕ(α) dα and g(y) = e −1 (β,y)Rn ψ(β) dβ, f (x) = e Rm
Rn
where ϕ and ψ are smooth functions with rapidly decreasing (i.e., tending to 0 as |x| → ∞ faster than any power of (1 + |x|)−1 ) derivatives of all orders. Finally, apply Fubini’s Theorem. Exercise 1.1.14. Given a pair of measurable spaces (E1 , B1 ) and (E 2 , B2 ), recall that their product is the measurable space E1 × E2 , B1 × B2 , where B1 × B2 is the σ-algebra over the Cartesian product space E1 × E2 generated by the sets Γ1 × Γ2 , Γi ∈ Bi . Further, recall that, for any probability measures µi on (Ei , Bi ), there is a unique probability measure µ1 × µ2 on E1 × E2 , B1 × B2 such that (µ1 × µ2 ) Γ1 × Γ2 = µ1 (Γ1 )µ2 (Γ2 ) for Γi ∈ Bi . More Q generally, for any n ≥ 2 and measurable spaces {(Ei , Bi ) : 1Q≤ i ≤ n}, one Qn n n takes 1 Bi to be the σ-algebra over 1 Ei generated by the sets 1 Γi , Γi ∈ Bi . Qn+1 Qn+1 Qn In particular, since 1 Ei and 1 Bi can be identified with ( 1 Ei ) ×
10
1 Sums of Independent Random Variables
Qn En+1 and ( 1 Bi ) × Bn+1 , respectively, one can use induction to show that, for every choice measures µi on (Ei , Bi ), there is a unique probability Qn of probability Qn Qn measure 1 µi on ( 1 Ei , 1 Bi ) such that ! n ! n n Y Y Y µi Γi = µi (Γi ), Γi ∈ Bi . 1
1
1
The purpose of this exercise is to generalize the preceding construction to infinite collections. Thus, let I be an infinite index set, and, for each i ∈ I, let (Ei , Bi ) be a measurable 6 Λ ⊆ I, use EΛ to denote the Q space. Given ∅ = Cartesian product space i∈Λ Ei and πQ Λ to denote the natural projection map taking EI onto EΛ . Further, let BI = i∈I Bi stand for the σ-algebra over EI generated by the collection C of subsets ! Y −1 πF Γi , Γi ∈ Bi , i∈F
as F varies over non-empty, finite subsets of I (abbreviated by ∅ = 6 F ⊂⊂ I). In the following steps, I outline a proof that, for every choice of Q probability measures µ on the (E , B )’s, there is a unique probability measure i i i i∈I µi on EI , BI with the property that ! !! Y Y Y −1 (1.1.15) µi πF Γi = µi Γi , Γi ∈ Bi , i∈I
i∈F
i∈F
for every ∅ = 6 F ⊂⊂ I. Not surprisingly, the probability space ! Y Y Y Ei , Bi , µi i∈I
i∈I
i∈I
is called the product over I of the spaces Ei , Bi , µi ; and when all the factors are the same space E, B, µ , it is customary to denote it by E I , B I , µI , and if, in addition, I = {1, . . . , N }, one uses E N , B N , µN . (i) After noting (cf. Exercise 1.1.12) that two probability measures that agree on a π-system agree on the σ-algebra generated bythat π-system, show that there is at most one probability measure on EI , BI that satisfies the condition in (1.1.15). Hence, the problem is purely one of existence. (ii) Let A be the algebra over EI generated by C, and show that there is a finitely additive µ : A −→ [0, 1] with the property that ! Y µ πF−1 ΓF = µi ΓF , ΓF ∈ BF , i∈F
Exercises for § 1.1
11
for all ∅ = 6 F ⊂⊂ I. Hence, all that one has to do is check that µ admits a σ-additive extension to BI , and, by a standard extension theorem, this comes down to checking that µ(An ) & 0 whenever {An : n ≥ 1} ⊆ A and An & ∅. Thus, let {An : n ≥ 1} be a non-increasing sequence from A, and T∞assume that µ(An ) ≥ for some > 0 and all n ∈ Z+ . One must show that 1 An 6= ∅. (iii) Referring to the last part of (ii), show that there is no loss in generality to assume that An = πF−1 ΓFn , where, for each n ∈ Z+ , ∅ = 6 Fn ⊂⊂ I and n ΓFn ∈ BFn . In addition, show that one may assume that F1 = {i1 } and that Fn = Fn−1 ∪ {in }, n ≥ 2, where {in : n ≥ 1} is a sequence of distinct elements of I. Now, make these assumptions, and show that it suffices to find a` ∈ Ei` , ` ∈ Z+ , with the property that, for each m ∈ Z+ , (a1 , . . . , am ) ∈ ΓFm . ( iv) Continuing (iii), for each m, n ∈ Z+ , define gm,n : EFm −→ [0, 1] so that gm,n xFm = 1ΓFn xi1 , . . . , xin
if n ≤ m
and
Z
gm,n xFm = EFn \Fm
1ΓFn xFm , yFn \Fm
n Y
! µi`
dyFn \Fm
if n > m.
`=m+1
After noting that, for each m and n, gm,n+1 ≤ gm,n and Z gm,n xFm = gm+1,n xFm , yim+1 µim+1 dyim+1 , Eim+1
set gm = limn→∞ gm,n and conclude that Z gm xFm = gm+1 xFm , yim+1 µim+1 dyim+1 . Eim+1
In addition, note that Z Z g1 xi1 µi1 dxi1 = lim Ei1
n→∞
g1,n xi1 µi1 dxi1
Ei1
= lim µ(An ) ≥ , n→∞
and proceed by induction to produce a` ∈ Ei` , ` ∈ Z+ , so that gm (a1 , . . . , am ) ≥ for all m ∈ Z+ . Finally, check that {am : m ≥ 1} is a sequence of the sort for which we were looking at the end of part (iii).
12
1 Sums of Independent Random Variables
Exercise 1.1.16. Recall that if Φ is a measurable map from one measurable space (E, B) into a second one (E 0 , B 0 ), then the distribution of Φ under a measure µ on (E, B) is the pushforward measure Φ∗ µ (also denoted by µ◦Φ−1 ) defined on (E 0 , B 0 ) by Φ∗ µ(Γ) = µ Φ−1 (Γ) for Γ ∈ B 0 . Given a non-empty index set I and, for each i ∈ I, a measurable space (Ei , Bi ) and an Ei -valued random variable Xi on the probability space (Ω, F, P), define Q X : Ω −→ i∈I Ei so that X(ω)i = Xi (ω) for each i ∈ I and ω ∈ Ω. Show that XQ i : i ∈ I is a family of P-independent random variables if and only if X∗ P = i∈I (Xi )∗ P. In particular, given probability measures µi on (Ei , Bi ), set Y Y Y Ω= Ei , F = Bi , P = µi , i∈I
i∈I
i∈I
let Xi : Ω −→ Ei be the natural projection map from Ω onto Ei , and show that {Xi : i ∈ I} is a family of mutually P-independent random variables such that, for each i ∈ I, Xi has distribution µi . Exercise 1.1.17. Although it does not entail infinite product spaces, an interesting example of the way in which the preceding type of construction can be effectively applied is provided by the following elementary version of a coupling argument. (i) Let (Ω, B, P) be a probability space and X and Y a pair of P-square integrable R-valued random variables with the property that X(ω) − X(ω 0 ) Y (ω) − Y (ω 0 ) ≥ 0 for all (ω, ω 0 ) ∈ Ω2 . Show that EP X Y ≥ EP [X] EP [Y ]. Hint: Define Xi and Yi on Ω2 for i ∈ {1, 2} so that Xi (ω) = X(ωi ) and Yi (ω) = Y (ωi ) when ω = (ω1 , ω2 ), and integrate the inequality 0 ≤ X(ω1 ) − X(ω2 ) Y (ω1 ) − Y (ω2 ) = X1 (ω) − X2 (ω) Y1 (ω) − Y2 (ω) with respect to P2 . (ii) Suppose that n ∈ Z+ and that f and g are R-valued, Borel measurable functions on Rn that are non-decreasing with respect to each coordinate (separately). Show that if X = X1 , . . . , Xn is an Rn -valued random variable on a probability space (Ω, B, P) whose coordinates are mutually P-independent, then EP f (X) g(X) ≥ EP f (X) EP g(X) so long as f (X) and g(X) are both P-square integrable.
Exercises for § 1.1
13
Hint: First check that the case when n = 1 reduces to an application of (i). Next, describe the general case in terms of a multiple integral, apply Fubini’s Theorem, and make repeated use of the case when n = 1. Exercise 1.1.18. A σ-algebra is said to be countably generated if it contains a countable collection of sets that generate it. The purpose of this exercise is to show that just because a σ-algebra is itself countably generated does not mean that all its sub-σ-algebras are. Let (Ω, F, P) be a probability space and {An : n ∈ Z+ ⊆ F a sequence of P-independent sub-subsets of F with the property that α ≤ P(An ) ≤ 1 − α for some α ∈ (0, 1). Let Fn be the sub-σ-algebra generated by An . Show that the tail σ-algebra T determined by Fn : n ∈ Z+ cannot be countably generated. Hint: Show that C ∈ T is an atom in T (i.e., B = C whenever B ∈ T \ {∅} is contained in C) only if one can write C = lim Cn ≡ n→∞
∞ \ [
Cn ,
m=1 n≥m
where, for each n ∈ Z+ , Cn equals either An or An {. Conclude that every atom in T must have P-measure 0. Now suppose that T were generated by B` : ` ∈ N . By Kolmogorov’s 0–1 Law, P B` ∈ {0, 1} for every ` ∈ N. Take ˆ` = B
B` B` {
if P B` = 1 if P B` = 0
and set
C=
\
ˆ` . B
`∈N
Note that, on the one hand, P(C) = 1, while, on the other hand, C is an atom in T and therefore has probability 0. Exercise 1.1.19. Here is an interesting application of Kolmogorov’s 0–1 Law to a property of the real numbers. (i) Referring to the discussion preceding Lemma 1.1.6 and part (i) of Exercise 1.1.11, define the transformations Tn : [0, 1) −→ [0, 1) for n ∈ Z+ so that Tn (ω) = ω −
Rn (ω) , 2n
ω ∈ [0, 1),
and notice (cf. the proof of Lemma 1.1.6) that Tn (ω) simply flips the nth coefficient in the binary expansion ω. Next, let Γ ∈ B[0,1) , and show that Γ is measurable with respect to the σ-algebra σ {Rn : n > m} generated by {Rn : n > m} if and only if Tn (Γ) = Γ for each 1 ≤ n ≤ m. In particular, conclude that λ[0,1) (Γ) ∈ {0, 1} if Tn Γ = Γ for every n ∈ Z+ .
14
1 Sums of Independent Random Variables
(ii) Let F denote the set of all finite subsets of Z+ , and for each F ∈ F, define T F : [0, 1) −→ [0, 1) so that T ∅ is the identity mapping and T F ∪{m} = T F ◦ Tm
for each F ∈ F and m ∈ Z+ \ F.
As an application of (i), show that for every Γ ∈ B[0,1) with λ[0,1) (Γ) > 0, ! [ λ[0,1) T F (Γ) = 1. F ∈F
In particular, this means that if Γ has positive measure, then almost every ω ∈ [0, 1) can be moved to Γ by flipping a finite number of the coefficients in the binary expansion of ω. § 1.2 The Weak Law of Large Numbers Starting with this section, and for the rest of this chapter, I will be studying what happens when one averages independent, real-valued random variables. The remarkable fact, which will be confirmed repeatedly, is that the limiting behavior of such averages depends hardly at all on the variables involved. Intuitively, one can explain this phenomenon by pretending that the random variables are building blocks that, in the averaging process, first get homothetically shrunk and then reassembled according to a regular pattern. Hence, by the time that one passes to the limit, the peculiarities of the original blocks get lost. Throughout the discussion, (Ω, F, P) will be a probability space on which there is a sequence {Xn : n ≥ 1} of real-valued random variables. Given n ∈ Z+ , use Sn to denote the partial sum X1 + · · · + Xn and S n to denote the average: n 1X Sn X` . = n n `=1
§ 1.2.1. Orthogonal Random Variables. My first result is a very general one; in fact, it even applies to random variables that are not necessarily independent and do not necessarily have mean 0. Lemma 1.2.1. Assume that EP Xn2 < ∞ for n ∈ Z+
and EP Xk X` = 0 if k 6= `.
Then, for each > 0, n 2 1 X P 2 E X` for n ∈ Z+ . (1.2.2) 2 P S n ≥ ≤ EP S n = 2 n `=1
In particular, if M ≡ sup EP Xn2 < ∞, n∈Z+
then
2 M , n ∈ Z+ and > 0; 2 P S n ≥ ≤ EP S n ≤ n and so S n −→ 0 in L2 (P; R) and therefore also in P-probability. (1.2.3)
§ 1.2 The Weak Law of Large Numbers
15
Proof: To prove the equality in (1.2.2), note that, by orthogonality, n X EP Sn2 = EP X`2 . `=1
The rest is just an application of Chebyshev’s inequality, the estimate that results after integrating the inequality 2 1[,∞) |Y | ≤ Y 2 1[,∞) |Y | ≤ Y 2 for any random variable Y . § 1.2.2. Independent Random Variables. Although Lemma 1.2.1 does not use independence, independent random variables provide a ready source of orthogonal functions. To wit, recall that for any P-square integrable random variable X, its variance Var(X) satisfies h 2 i 2 Var(X) ≡ EP X − EP [X] = EP X 2 − EP [X] ≤ EP X 2 . In particular, if the random variables Xn , n ∈ Z+ , are P-square integrable and P-independent, then the random variables ˆ n ≡ Xn − EP Xn , n ∈ Z+ , X are still P-square integrable, have mean value 0, and therefore are orthogonal. Hence, the following statement is an immediate consequence of Lemma 1.2.1. Theorem 1.2.4. Let Xn : n ∈ Z+ be a sequence of P-independent, P-square integrable random variables with mean value m and variance dominated by σ 2 . Then, for every n ∈ Z+ and > 0, h 2 i σ 2 . ≤ (1.2.5) 2 P S n − m ≥ ≤ EP S n − m n In particular, S n −→ m in L2 (P; R) and therefore in P-probability.
As yet I have made only minimal use of independence: all that I have done is subtract off the mean of independent random variables and thereby made them orthogonal. In order to bring the full force of independence into play, one has to exploit the fact that one can compose independent random variables with any (measurable) functions without destroying their independence; in particular, truncating independent random variables does not destroy independence. To see how such a property can be brought to bear, I will now consider the problem of extending the last part of Theorem 1.2.4 to Xn ’s that are less than P-square integrable. In order to understand the statement, recall that a family of random variables Xi : i ∈ I is said to be uniformly P-integrable if h i lim sup EP Xi , Xi ≥ R = 0. R%∞ i∈I
As the proof of the following theorem illustrates, the importance of this condition is that it allows one to simultaneously approximate the random variables Xi , i ∈ I, by bounded random variables.
16
1 Sums of Independent Random Variables
Theorem 1.2.6 (The Weak Law of Large Numbers). Let Xn : n ∈ Z+ be a uniformly P-integrable sequence of P-independent random variables. Then n
1X Xm − EP [Xm ] −→ 0 in L1 (P; R) n 1 and therefore also in P-probability. In particular, if Xn : n ∈ Z+ is a sequence of P-independent, P-integrable random variables that are identically distributed, then S n −→ EP [X1 ] in L1 (P; R) and P-probability. (Cf. Exercise 1.2.11.)
Proof: Without loss in generality, I will assume that EP [Xn ] = 0 for every n ∈ Z+ . For each R ∈ (0, ∞), define fR (t) = t 1[−R,R] (t), t ∈ R, m(R) = EP fR ◦ Xn , n
Xn(R) = fR ◦ Xn − m(R) n ,
and set (R)
Sn
n
=
1 X (R) X` n
and
(R)
Tn
n
=
Since E[Xn ] = 0 =⇒
1 X (R) Y` . n `=1
`=1
(R) mn
and Yn(R) = Xn − Xn(R) ,
= −E Xn , |Xn | > R ,
(R) (R) EP |S n | ≤ EP |S n | + EP |T n | (R) 1 ≤ EP |S n |2 2 + 2 max EP |X` |, |X` | ≥ R 1≤`≤n
R EP |X` |, |X` | ≥ R ; ≤ √ + 2 max n `∈Z+ and therefore, for each R > 0,
lim EP |S n | ≤ 2 sup EP |X` |, |X` | ≥ R .
n→∞
`∈Z+
Hence, because the X` ’s are uniformly P-integrable, we get the desired convergence in L1 (P; R) by letting R % ∞. § 1.2.3. Approximate Identities. The name of Theorem 1.2.6 comes from a somewhat invidious comparison with the result in Theorem 1.4.9. The reason why the appellation weak is not entirely fair is that, although The Weak Law is indeed less refined than the result in Theorem 1.4.9, it is every bit as useful as the one in Theorem 1.4.9 and maybe even more important when it comes to applications. What The Weak Law provides is a ubiquitous technique for constructing an approximate identity (i.e., a sequence of measures that approximate a point mass) and measuring how fast the approximation is taking
§ 1.2 The Weak Law of Large Numbers
17
place. To illustrate how clever selections of the random variables entering The Weak Law can lead to interesting applications, I will spend the rest of this section discussing S. Bernstein’s approach to Weierstrass’s Approximation Theorem. + For a given p ∈ [0, 1], let Xn : n ∈ Z be a sequence of P-independent {0, 1}-valued Bernoulli random variables with mean value p. Then P Sn = ` =
n ` p (1 − p)n−` `
for
0 ≤ ` ≤ n.
Hence, for any f ∈ C [0, 1]; R , the nth Bernstein polynomial n X n ` p` (1 − p)n−` Bn (p; f ) ≡ f n `
(1.2.7)
`=0
of f at p is equal to
EP f ◦ S n . In particular,
f (p) − Bn (p; f ) = EP f (p) − f ◦ S n ≤ EP f (p) − f ◦ S n ≤ 2kf ku P S n − p ≥ + ρ(; f ),
where kf ku is the uniform norm of f (i.e., the supremum of |f | over the domain of f ) and ρ(; f ) ≡ sup |f (t) − f (s)| : 0 ≤ s < t ≤ 1 with t − s ≤ is the modulus of continuity of f . Noting that Var Xn = p(1 − p) ≤ applying (1.2.5), we conclude that, for every > 0,
1 4
and
f (p) − Bn (p; f ) ≤ kf ku + ρ(; f ). u 2n2
In other words, for all n ∈ Z+ , (1.2.8)
f − Bn (· ; f ) u ≤ β(n; f ) ≡ inf
kf ku + ρ(; f ) : > 0 . 2n2
Obviously, (1.2.8) not only shows that, as n → ∞, Bn (· ; f ) −→ f uniformly on [0, 1], it even provides a rate of convergence in terms of the modulus of continuity of f . Thus, we have done more than simply prove Weierstrass’s theorem; we have produced a rather explicit and tractable sequence of approximating polynomials, the sequence Bn (· ; f ) : n ∈ Z+ . Although this sequence is, by no means, the
18
1 Sums of Independent Random Variables
most efficient one,1 as we are about to see, the Bernstein polynomials have a lot to recommend them. In particular, they have the feature that they provide non-negative polynomial approximates to non-negative functions. In fact, the following discussion reveals much deeper non-negativity preservation properties possessed by the Bernstein approximation scheme. In order to bring out the virtues of the Bernstein polynomials, it is important to replace (1.2.7) with an expression in which the coefficients of Bn ( · ; f ) (as polynomials) are clearly displayed. To this end, introduce the difference operator ∆h for h > 0 given by f (t + h) − f (t) . ∆h f (t) = h
A straightforward inductive argument (using Pascal’s Identity for the binomial coefficients) shows that m X m ` m (−h) ∆h f (t) = (−1) f (t + `h) ` m
for m ∈ Z+ ,
`=0
(m)
where ∆h see that
denotes the mth iterate of the operator ∆h . Taking h =
Bn (p; f ) =
n n−` X X nn − `
`
`=0 k=0 n X
k
1 n,
we now
(−1)k f (`h)p`+k
r X n n−` = p (−1)r−` f (`h) ` r − ` r=0 r
`=0
n X
X r n r = (−p) (−1)` f (`h) r ` r=0 r
`=0
=
n X r=0
n (ph)r ∆rh f (0), r
where ∆0h f ≡ f . Hence, we have proved that (1.2.9)
Bn (p; f ) =
n X `=0
n−`
n ` ∆ 1 f (0)p` n `
for
p ∈ [0, 1].
The marked resemblance between the expression on the right-hand side of (1.2.9) and a Taylor polynomial is more than coincidental. To demonstrate how 1
See G.G. Lorentz’s Bernstein Polynomials, Chelsea Publ. Co. (1986) for a lot more information.
§ 1.2 The Weak Law of Large Numbers
19
one can exploit the relationship between the Bernstein and Taylor polynomials, say that a function ϕ ∈ C ∞ (a, b); R is absolutely monotone if its mth derivative Dm ϕ is non-negative for every m ∈ N. Also, say that ϕ ∈ C ∞ [0, 1]; [0, 1]) is a probability generating function if there exists a un : n ∈ N ⊆ [0, 1] such that ∞ ∞ X X un = 1 and ϕ(t) = un tn for t ∈ [0, 1]. n=0
n=0
Obviously, every probability generating function is absolutely monotone on (0, 1). The somewhat surprising (remember that most infinitely differentiable functions do not admit power series expansions) fact which I am about to prove is that, apart from a multiplicative constant, the converse is also true. In fact, one does not need to know, a priori, that the function is smooth so long as it satisfies a discrete version of absolute monotonicity. Theorem 1.2.10. Let ϕ ∈ C [0, 1]; R with ϕ(1) = 1 be given. Then the following are equivalent: (i) ϕ is a probability generating function, (ii) the of ϕ to (0, 1) is absolutely monotone; restriction ϕ (0) ≥ 0 for every n ∈ N and 0 ≤ m ≤ n. (iii) ∆m 1 n
Proof: The implication (i) =⇒ (ii) is trivial. To see that (ii) implies (iii), first observe that if ψ is absolutely monotone on (a, b) and h ∈ (0, b − a), then ∆h ψ is absolutely monotone on (a, b − h). Indeed, because D ◦ ∆h ψ = ∆h ◦ Dψ on (a, b − h), we have that h Dm ◦ ∆h ψ (t) =
Z
t+h
Dm+1 ψ(s) ds ≥ 0,
t ∈ (a, b − h),
t
for any m ∈ N. Returning to the function ϕ, we now know that ∆m h ϕ is absolutely monotone on (0, 1 − mh) for all m ∈ N and h > 0 with mh < 1. In particular, m [∆m h ϕ](0) = lim [∆h ϕ](t) ≥ 0 t&0
and so ∆m h ϕ (0) ≥ 0 when h =
1 n
if
mh < 1,
and 0 ≤ m < n. Moreover, since
[∆n1 ϕ](0) = lim1 [∆nh ϕ](0), n
h% n
we also know that ∆nh ϕ (0) ≥ 0 when h = n1 , and this completes the proof that (ii) implies (iii). Finally, assume that (iii) holds and set ϕn = Bn ( · ; ϕ). Then, from (1.2.9) and the equality ϕn (1) = ϕ(1) = 1, we see that each ϕn is a probability generating function. Thus, in order to complete the proof that (iii) implies (i), all that
20
1 Sums of Independent Random Variables
one has to do is check that a uniform limit of probability generating functions is itself a probability generating function. To this end, write ϕn (t) =
∞ X
un,` t` ,
t ∈ [0, 1] for each n ∈ Z+ .
`=0
Because the un,` ’s are all elements of [0, 1], one can use a diagonalization procedure to choose {nk : k ∈ Z+ } so that lim unk ,` = u` ∈ [0, 1]
k→∞
exists for each ` ∈ N. But, by Lebesgue’s Dominated Convergence Theorem, this means that ϕ(t) = lim ϕnk (t) = k→∞
∞ X
u` t`
for every t ∈ [0, 1).
`=0
Finally, by the Monotone Convergence Theorem, the preceding extends immediately to t = 1, and so ϕ is a probability generating function. (Notice that the argument just given does not even use the assumed uniform convergence and shows that the pointwise limit of probability generating functions is again a probability generating function.) The preceding is only one of many examples in which The Weak Law leads to useful ways of forming an approximate identity. A second example is given in Exercises 1.2.12 and 1.2.13. My treatment of these is based on that of Wm. Feller.2 Exercises for § 1.2 Exercise 1.2.11. Although, for historical reasons, The Weak Law is usually thought of as a theorem about convergence in P-probability, the forms in which I have presented it are clearly results about convergence in either P-mean or even P-square mean. Thus, it is interesting to discover that one can replace the uniform integrability assumption made in Theorem 1.2.6 with a weak uniform integrability assumption if one is willing to settle for convergence in P-probability. Namely, let X1 , . . . , Xn , . . . be mutually P-independent random variables, assume that F (R) ≡ sup RP |Xn | ≥ R −→ 0 as R % ∞, n∈Z+ 2
Wm. Feller, An Introduction to Probability Theory and Its Applications, Vol. II, Wiley, Series in Probability and Math. Stat. (1968). Feller provides several other similar applications of The Weak Law, including the ones in the following exercises.
Exercises for § 1.2 and set
21
n
i 1 X Ph E X` , |X` | ≤ n , mn = n
n ∈ Z+ .
`=1
Show that, for each > 0, n i 1 X P h 2 X` > n E X , X ≤ n + P max P S n − mn ≥ ≤ ` ` 1≤`≤n (n)2 `=1 Z n 2 F (t) dt + F (n), ≤ 2 n 0 and conclude that S n − mn −→ 0 in P-probability. (See part (ii) of Exercises 1.4.26 and 1.4.27 for a partial converse to this statement.)
Hint: Use the formula Var(Y ) ≤ EP Y 2 = 2
Z
t P |Y | > t dt.
[0,∞)
Exercise 1.2.12. Show that, for each T ∈ [0, ∞) and t ∈ (0, ∞), X (nt)k 1 if T > t = lim e−nt n→∞ k! 0 if T < t. 0≤k≤nT
Hint: Let X1 , . . . , Xn , . . . be P-independent, N-valued Poisson random variables with mean value t. That is, the Xn ’s are P-independent and tk for k ∈ N. P Xn = k = e−t k! Show that Sn is an N-valued Poisson random variable with mean value nt, and conclude that, for each T ∈ [0, ∞) and t ∈ (0, ∞), X (nt)k = P Sn ≤ T . e−nt k! 0≤k≤nT
Exercise 1.2.13. Given a right-continuous function F : [0, ∞) −→ R of bounded variation with F (0) = 0, define its Laplace transform ϕ(λ), λ ∈ [0, ∞), by the Riemann–Stieltjes integral: Z ϕ(λ) = e−λt dF (t). [0,∞)
Using Exercise 1.2.12, show that X (−n)k Dk ϕ (n) −→ F (T ) k!
as n → ∞
k≤nT
for each T ∈ [0, ∞) at which F is continuous. Conclude, in particular, that F can be recovered from its Laplace transform. Although this is not the most practical recovery method, it is distinguished by the fact that it does not involve complex analysis.
22
1 Sums of Independent Random Variables
§ 1.3 Cram´ er’s Theory of Large Deviations From Theorem 1.2.4, we know that if Xn : n ∈ Z+ is a sequence of Pindependent, P-square integrable random variables with mean value 0, and if the averages S n , n ∈ Z+ , are defined accordingly, then, for every > 0, max 1≤m≤n Var(Xm ) , n ∈ Z+ . P S n ≥ ≤ n2 Thus, so long as Var(Xn ) −→ 0 as n → ∞, n the S n ’s are becoming more and more concentrated near 0, and the rate at which this concentration is occurring can be estimated in terms of the variances Var(Xn ). In this section, we will see that, by placing more stringent integrability requirements on the Xn ’s, one can gain more information about the rate at which the S n ’s are concentrating at 0. In all of this analysis, the trick is to see how independence can be combined with 0 mean value to produce unexpected cancellations; and, as a preliminary warm-up exercise, I begin with the following.
Theorem 1.3.1. Let {Xn : n ∈ Z+ } be a sequence of P-independent, Pintegrable random variables with mean value 0, and assume that M4 ≡ sup EP Xn4 < ∞. n∈Z+
Then, for each > 0,
4 3M4 4 P |S n | ≥ ≤ EP S n ≤ 2 , n In particular, S n −→ 0 P-almost surely. (1.3.2)
n ∈ Z+ .
Proof: Obviously, in order to prove (1.3.2), it suffices to check the second inequality, which is equivalent to EP Sn4 ≤ 3M4 n2 . But n X
E Sn4 = P
EP Xm1 · · · Xm4 ,
m1 ,...,m4 =1
and, by Schwarz’s Inequality, each of these terms is dominated by M4 . In addition, of these terms, the only ones that do not vanish have either all their factors the same or two pairs of equal factors. Thus, the number of non-vanishing terms is n + 3n(n − 1) = 3n2 − 2n. Given (1.3.2), the proof of the last part becomes an easy application of the Borel–Cantelli Lemma. Indeed, for any > 0, we know from (1.3.2) that ∞ X P S n ≥ < ∞, n=1
and therefore, by (1.1.4), that P limn→∞ S n ≥ = 0.
§ 1.3 Cram´er’s Theory of Large Deviations
23
Remark 1.3.3. The final assertion in Theorem 1.3.1 is a primitive version of The Strong Law of Large Numbers. Although The Strong Law will be taken up again, and considerably refined, in Section 1.4, the principle on which its proof here was based is an important one: namely, control more moments and you will get better estimates; get better estimates and you will reach more refined conclusions. With the preceding adage in mind, I will devote the rest of this section to examining what one can say when one has all moments at one’s disposal. In fact, from now on, I will be assuming that X1 , . . . , Xn , . . . are independent random variables with common distribution µ having the property that the moment generating function Z (1.3.4) Mµ (ξ) ≡ eξ x µ(dx) < ∞ for all ξ ∈ R. R
Obviously, (1.3.4) is more than sufficient to guarantee that the Xn ’s have moments of all orders. In fact, as an application of Lebesgue’s Dominated Convergence Theorem, one sees that ξ ∈ R 7−→ M (ξ) ∈ (0, ∞) is infinitely differentiable and that Z dn M (0) for all n ∈ N. EP X1n = xn µ(dx) = dξ n R
In the discussion that follows, I will use m and σ 2 to denote, respectively, the common mean value and variance of the Xn ’s. In order to develop some intuition for the considerations that follow, I will first consider an example, which, for many purposes, is the canonical example in probability theory. Namely, let g : R −→ (0, ∞) be the Gauss kernel |y|2 1 , y ∈ R, (1.3.5) g(y) ≡ √ exp − 2 2π
and recall that a random variable X is standard normal if Z P X∈Γ = g(y) dy, Γ ∈ BR . Γ
In spite of their somewhat insultingly bland moniker, standard normal random variables are the building blocks for the most honored family in all of probability theory. Indeed, given m ∈ R and σ ∈ [0, ∞), the random variable Y is said to be normal (or Gaussian) with mean valuem and variance σ 2 (often this is abbreviated by saying that X is an N m, σ 2 -random variable) if and only if the distribution of Y is γm,σ2 , where γm,σ2 is the distribution of the variable σX + m when X is standard normal. That is, Y is an N (m, σ 2 ) random variable if, when σ = 0, P(Y = m) = 1 and, when σ > 0, Z y−m 1 dy for Γ ∈ BR . g P Y ∈Γ = σ Γ σ
24
1 Sums of Independent Random Variables
There are two obvious reasons for the honored position held by Gaussian random variables. In the first place, they certainly have finite moment generating functions. In fact, since 2 Z ξ ξy , ξ ∈ R, e g(y) dy = exp 2 R
it is clear that
σ2 ξ2 . Mγm,σ2 (ξ) = exp ξm + 2
(1.3.6)
Secondly, they add nicely. To be precise, it is a familiar fact from elemenˆ is tary probability theory that if X is an N (m, σ 2 )-random variable and X 2 ˆ an N (m, ˆ σ ˆ )-random variable that is independent of X, then X + X is an N m + m, ˆ σ2 + σ ˆ 2 -random variable. In particular, if X1 , . . . , Xn are mutually independent, standard normal random variables, then S n is an N 0, n1 -random variable. That is, r Z n|y|2 n dy. exp − P Sn ∈ Γ = 2 2π Γ
Thus (cf. Exercise 1.3.16), for any Γ we see that (1.3.7)
h i 1 log P S n ∈ Γ = −ess inf lim n→∞ n
|y|2 : y∈Γ , 2
where the “ess” in (1.3.7) stands for essential and means that what follows is taken modulo a set of measure 0. (Hence, apart from a minus sign, the right2 hand side of (1.3.7) is the greatest number dominated by |y|2 for Lebesgue-almost every y ∈ Γ.) In fact, because
Z
∞
g(y) dy ≤ x−1 g(x) for all x ∈ (0, ∞),
x
we have the rather precise upper bound r n2 2 exp − P |S n | ≥ ≤ 2 nπ2
for > 0.
At the same time, it is clear that, for 0 < < |a|,
r P |S n − a| < ≥
n(|a| + )2 22 n . exp − 2 π
§ 1.3 Cram´er’s Theory of Large Deviations
25
More generally, if the Xn ’s are mutually independent N (m, σ 2 )-random variables, then one finds that
r P |S n − m| ≥ σ ≤
n2 2 exp − 2 nπ2
for > 0;
and, for 0 < < |a| and sufficiently large n’s,
r P |S n − (m + a)| < σ ≥
n(|a| + )2 22 n . exp − 2 π
Of course, in general one cannot hope to know such explicit expressions for the distribution of S n . Nonetheless, on the basis of the preceding, one can start to see what is going on. Namely, when the distribution µ falls off rapidly outside of compacts, averaging n independent random variables with distribution µ has the effect of building an exponentially deep well in which the mean value m lies at the bottom. More precisely, if one believes that the Gaussian random variables are normal in the sense that they are typical, then one should conjecture that, even when the random variables are not normal, the behavior of P S n − m ≥ for large n’s should resemble that of Gaussians with the same variance; and it is in the verification of this conjecture that the moment generating function Mµ plays a central role. Namely, although an expression in terms of µ for the distribution of Sn is seldom readily available, the moment generating function for Sn is easily expressed in terms of Mµ . To wit, as a trivial application of independence, we have EP eξSn = Mµ (ξ)n , ξ ∈ R.
Hence, by Markov’s Inequality applied to eξSn , we see that, for any a ∈ R,
P S n ≥ a ≤ e−nξa Mµ (ξ)n = exp −n ξa − Λµ (ξ) ,
ξ ∈ [0, ∞),
where (1.3.8)
Λµ (ξ) ≡ log Mµ (ξ)
is the logarithmic moment generating function of µ. The preceding relation is one of those lovely situations in which a single quantity is dominated by a whole family of quantities, which means that one should optimize by minimizing over the dominating quantities. Thus, we now have (1.3.9)
" P S n ≥ a ≤ exp −n
sup ξ∈[0,∞)
# ξa − Λµ (ξ) .
26
1 Sums of Independent Random Variables
Notice that (1.3.9) is really very good. For instance, when the Xn ’s are N (m, σ 2 )random variables and σ > 0, then (cf. (1.3.6)) the preceding leads quickly to the estimate n2 P S n − m ≥ ≤ exp − 2 , 2σ
which is essentially the upper bound at which we arrived before. Taking a hint from the preceding, I now introduce the Legendre transform (1.3.10) Iµ (x) ≡ sup ξx − Λµ (ξ) : ξ ∈ R , x ∈ R, of Λµ and, before proceeding further, make some elementary observations about the structure of the functions Λµ and Iµ . Lemma 1.3.11. The function Λµ is infinitely differentiable. In addition, for each ξ ∈ R, the probability measure νξ on R given by Z 1 eξx µ(dx) for Γ ∈ BR νξ (Γ) = Mµ (ξ) Γ
has moments of all orders, Z x νξ (dx) = R
Λ0µ (ξ),
Z
2
Z
x νξ (dx) −
and R
2 x νξ (dx) = Λ00µ (ξ).
R
Next, the function Iµ is a [0, ∞]-valued, lower semicontinuous, convex function that vanishes at m. Moreover, Iµ (x) = sup ξx − Λµ (ξ) : ξ ≥ 0 for x ∈ [m, ∞) and Iµ (x) = sup ξx − Λµ (ξ) : ξ ≤ 0
for x ∈ (−∞, m].
Finally, if α = inf x ∈ R : µ (−∞, x] > 0 and β = sup x ∈ R : µ [x, ∞) > 0 , then Iµ is smooth on (α, β) and identically +∞ off of [α, β]. In fact, either µ({m}) = 1 and α = m = β or m ∈ (α, β), in which case Λ0µ is a smooth, strictly increasing mapping from R onto (α, β), Iµ (x) = Ξµ (x) x − Λµ Ξµ (x) , x ∈ (α, β),
where
Ξµ = Λ0µ
−1
is the inverse of Λ0µ , µ({α}) = e−Iµ (α) if α > −∞, and µ({β}) = e−Iµ (β) if β < ∞.
§ 1.3 Cram´er’s Theory of Large Deviations
27
Proof: For notational convenience, I will drop the subscript “µ” during the proof. Further, note that the smoothness of Λ follows immediately from the positivity and smoothness of M , and the identification of Λ0 (ξ) and Λ00 (ξ) with the mean and variance of νξ is elementary calculus combined with the remark following (1.3.4). Thus, I will concentrate on the properties of the function I. As the pointwise supremum of functions that are linear, I is certainly lower semicontinuous and convex. Also, because Λ(0) = 0, it is obvious that I ≥ 0. Next, by Jensen’s Inequality, Z Λ(ξ) ≥ ξ x µ(dx) = ξ m, R
and, therefore, ξx − Λ(ξ) ≤ 0 if x ≤ m and ξ ≥ 0 or if x ≥ m and ξ ≤ 0. Hence, because I is non-negative, this proves the one-sided extremal characterizations of Iµ (x) depending on whether x ≥ m or x ≤ m. Turning to the final part, note first that there is nothing more to do in the case when µ({m}) = 1. Thus, assume that µ({m}) < 1, in which case it is clear that m ∈ (α, β) and that none of the measures νξ is degenerate (i.e., concentrate at one point). In particular, because Λ00 (ξ) is the variance of the νξ , we know that Λ00 > 0 everywhere. Hence, Λ0 is strictly increasing and therefore admits a smooth inverse Ξ on its image. Furthermore, because Λ0 (ξ) is the mean of νξ , it is clear that the image of Λ0 is contained in (α, β). At the same time, given an x ∈ (α, β), note that Z e−ξx eξy µ(dy) −→ ∞ as |ξ| → ∞, R
and therefore ξ ξx − Λ(ξ) achieves a maximum at some point ξx ∈ R. In addition, by the first derivative test, Λ0 (ξx ) = x, and so ξx = Ξ−1 (x). Finally, suppose that β < ∞. Then Z Z −ξβ ξy e e µ(dy) = e−ξ(β−y) µ(dy) & µ({β}) as ξ → ∞, R
(−∞,β]
and therefore e−I(β) = inf ξ≥0 e−ξβ M (ξ) = µ({β}). Since the same reasoning applies when α > −∞, we are done. Theorem 1.3.12 (Cram´ er’s Theorem). Let {Xn : n ≥ 1} be a sequence of P-independent random variables with common distribution µ, assume R that the associated moment generating function Mµ satisfies (1.3.4), set m = R x µ(dx), and define Iµ accordingly, as in (1.3.10). Then,
P S n ≥ a ≤ e−nIµ (a) P S n ≤ a ≤ e−nIµ (a)
for all a ∈ [m, ∞),
for all a ∈ (−∞, m].
28
1 Sums of Independent Random Variables
Moreover, for a ∈ (α, β) (cf. Lemma 1.3.11), > 0, and n ∈ Z+ , ! h i Λ00µ Ξµ (a) exp −n I (a) + |Ξ (a)| , P S n − a < ≥ 1 − µ µ n2 −1 where Λµ is the function given in (1.3.8) and Ξµ ≡ Λµ 0 .
Proof: To prove the first part, suppose that a ∈ [m, ∞), and apply the second part of Lemma 1.3.11 to see that the exponent in (1.3.9) equals −nIµ (a), and, after replacing {Xn : n ≥ 1} by {−Xn : n ≥ 1}, one also gets the desired estimate when a ≤ m. To prove the lower bound, let a ∈ [m, β) be given, and set ξ = Ξµ (a) ∈ [0, ∞). Next, recall the probability measure νξ described in Lemma 1.3.11, and remember that νξ has mean value a = Λ0µ (ξ) and variance Λ00µ (ξ). Further, if Yn : n ∈ Z+ is a sequence of independent, identically distributed random variables with common distribution νξ , then it is an easy matter to check that, for any n ∈ Z+ and every BRn -measurable F : Rn −→ [0, ∞), h h i i 1 P ξSn E e F X , . . . , X EP F Y1 , . . . , Yn = 1 n . Mµ (ξ)n In particular, if n X Tn , Tn = Y` and T n = n `=1
then, because Iµ (a) = ξa − Λµ (ξ), i h P S n − a < = M (ξ)n EP e−ξTn , T n − a < ≥ e−nξ(a+) M (ξ)n P T n − a < h i = exp −n Iµ (a) + ξ P T n − a < .
But, because the mean value and variance of the Yn ’s are, respectively, a and Λ00µ (ξ), (1.2.5) leads to Λ00 (ξ) µ . P T n − a ≥ ≤ n2
The case when a ∈ (α, m] is handled in the same way.
Results like the ones obtained in Theorem 1.3.12 are examples of a class of results known as large deviations estimates. They are large deviations because the probability of their occurrence is exponentially small. Although large deviation estimates are available in a variety of circumstances,1 in general one has to settle for the cruder sort of information contained in the following. 1
In fact, some people have written entire books on the subject. See, for example, J.-D. Deuschel and D. Stroock, Large Deviations, now available from the A.M.S. in the Chelsea Series.
§ 1.3 Cram´er’s Theory of Large Deviations
29
Corollary 1.3.13. For any Γ ∈ BR ,
h i 1 log P S n ∈ Γ n→∞ n h i 1 log P S n ∈ Γ ≤ − inf Iµ (x). ≤ lim n→∞ n x∈Γ
− inf◦ Iµ (x) ≤ lim x∈Γ
(I use Γ◦ and Γ to denote the interior and closure of a set Γ. Also, recall that I take the infemum over the empty set to be +∞.)
Proof: To prove the upper bound, let Γ be a closed set, and define Γ+ = Γ ∩ [m, ∞) and Γ− = Γ ∩ (−∞, m]. Clearly, P S n ∈ Γ ≤ 2P S n ∈ Γ+ ∨ P S n ∈ Γ− .
Moreover, if Γ+ 6= ∅ and a+ = min{x : x ∈ Γ+ }, then, by Lemma 1.3.11 and Theorem 1.3.12, Iµ (a+ ) = inf Iµ (x) : x ∈ Γ+ and P S n ∈ Γ+ ≤ e−nIµ (a+ ) .
Similarly, if Γ− 6= ∅ and a− = max{x : x ∈ Γ− }, then Iµ (a− ) = inf Iµ (x) : x ∈ Γ− and P S n ∈ Γ− ≤ e−nIµ (a− ) .
Hence, either Γ = ∅, and there is nothing to do anyhow, or P S n ∈ Γ ≤ 2 exp −n inf Iµ (x) : x ∈ Γ , n ∈ Z+ ,
which certainly implies the asserted upper bound. To prove the lower bound, assume that Γ is a non-empty open set. What I have to show is that h i 1 log P S n ∈ Γ ≥ −Iµ (a) lim n→∞ n
for every a ∈ Γ. If a ∈ Γ ∩ (α, β), choose δ > 0 so that (a − δ, a + δ) ⊆ Γ and use the second part of Theorem 1.3.12 to see that
h i 1 log P S n ∈ Γ ≥ −Iµ (a) − Ξµ (a) n→∞ n lim
for every ∈ (0, δ). If a ∈ / [α, β], then Iµ (a) = ∞, and so there is nothing to do. Finally, if a ∈ {α, β}, then µ({a}) = e−Iµ (a) and therefore P S n ∈ Γ ≥ P S n = a ≥ e−nIµ (a) .
30
1 Sums of Independent Random Variables
Remark 1.3.14. The upper bound in Theorem 1.3.12 is often called Chernoff ’s Inequality. The idea underlying its derivation is rather mundane by comparison to the subtle idea underlying the proof of the lower bound. Indeed, it may not be immediately obvious what that idea was! Thus, consider once again the second part of the proof of Theorem 1.3.12. What I had to do is estimate the probability that S n lies in a neighborhood of a. When a is the mean value m, such an estimate is provided by the Weak Law. On the other hand, when a 6= m, the Weak Law for the Xn ’s has very little to contribute. Thus, what I did is replace the original Xn ’s by random variables Yn , n ∈ Z+ , whose mean value is a. Furthermore, the transformation from the Xn ’s to the Yn ’s was sufficiently simple that it was easy to estimate Xn -probabilities in terms of Yn probabilities. Finally, the Weak Law Pn applied to the Yn ’s gave strong information about the rate of approach of n1 `=1 Y` to a.
I close this section by verifying the conjecture (cf. the discussion preceding Lemma 1.3.11) that the Gaussian case is normal. In particular, I want to check that the well around m in which the distribution of S n becomes concentrated looks Gaussian, and, in view of Theorem 1.3.12, this comes down to the following.
Theorem 1.3.15. Let everything be as in Lemma 1.3.11, and assume that the variance σ 2 > 0. There exists a δ ∈ (0, 1] and a K ∈ (0, ∞) such that [m − δ, m + δ] ⊆ (α, β) (cf. Lemma 1.3.11), Λ00µ Ξ(x) ≤ K, Ξµ (x) ≤ K|x − m|,
and
2 Iµ (x) − (x − m) ≤ K|x − m|3 2σ 2
for all x ∈ [m − δ, m + δ]. In particular, if 0 < < δ, then 2 3 − K , P |S n − m| ≥ ≤ 2 exp −n 2σ 2
and if |a − m| < δ and > 0, then |a − m|2 K 2 + K|a − m| + |a − m| . P |S n − a| < ≥ 1 − 2 exp −n 2σ 2 n
Proof: Without loss in generality (cf. Exercise 1.3.17), I will assume that m = 0 and σ 2 = 1. Since, in this case, Λµ (0) = Λ0µ (0) = 0 and Λ00µ (0) = 1, it follows that Ξµ (0) = 0 and Ξ0µ (0) = 1. Hence, we can find an M ∈ (0, ∞) and a δ ∈ (0, 1] with α < −δ < δ < β for which Ξµ (x) − x ≤ M |x|2 and Λµ (ξ) − ξ2 ≤ M |ξ|3 whenever |x| ≤ δ and |ξ| ≤ (M + 1)δ, respectively. In 2 particular, this leads immediately to Ξµ (x) ≤ (M + 1)|x| for |x| ≤ δ, and the estimate for Iµ comes easily from the preceding combined with equation Iµ (x) = Ξ(x)x − Λµ Ξµ (x) .
Exercises for § 1.3
31
Exercises for § 1.3 Exercise 1.3.16. Let E, F, µ be a measure space and f a non-negative, F-measurable function. If either µ(E) < ∞ or f is µ-integrable, show that kf kLp (µ;R) −→ kf kL∞ (µ;R)
as p → ∞.
Hint: Handle the case µ(E) < ∞ first, and treat the case when f ∈ L1 (µ; R) by considering the measure ν(dx) = f (x) µ(dx). Exercise 1.3.17. Referring to the notation used in this section, assume that µ is a non-degenerate (i.e., it is not concentrated at a single point) probability measure on R for which (1.3.4) holds. Next, let m and σ 2 be the mean and variance of µ, use ν to denote the distribution of x ∈ R 7−→
x−m ∈R σ
under µ,
and define Λν , Iν , and Ξν accordingly. Show that Λµ (ξ) = ξm + Λν (σξ), ξ ∈ R, x−m , x ∈ R, Iµ (x) = Iν σ Image Λ0µ = m + σ Image Λ0ν , 1 x−m , x ∈ Image Λ0µ . Ξµ (x) = Ξν σ σ
Exercise 1.3.18. Continue with the same notation as in the preceding. (i) Show that Iν ≤ Iµ if Mµ ≤ Mν . (ii) Show that Iµ (x) =
(x − m)2 , 2σ 2
x ∈ R,
when µ is the N m, σ 2 distribution with σ > 0, and show that Iµ (x) =
b−x b−x x−a x−a , log + log p(b − a) (1 − p)(b − a) b − a b−a
x ∈ (a, b),
when a < b, p ∈ (0, 1), and µ({a}) = 1 − µ({b}) = p. (iii) When µ is the hcentered Bernoulli distribution given by µ {±1} = 12 , show i 2 2 that Mµ (ξ) ≤ exp ξ2 , ξ ∈ R, and conclude that Iµ (x) ≥ x2 , x ∈ R. More
generally, given n ∈ Z+ , {σk : 1 ≤ k ≤ n} ⊆ R, and independent random variables X1 , . . . , Xn with this µ as their common distribution, let ν denote the
32
1 Sums of Independent Random Variables
Pn Pn x2 2 ≡ 1 σk2 . distribution of S ≡ 1 σk Xk and show that Iν (x) ≥ 2Σ 2 , where Σ In particular, conclude that a2 P |S| ≥ a ≤ 2 exp − 2 , a ∈ [0, ∞). 2Σ
Exercise 1.3.19. Although it is not exactly the direction in which I have been going, it seems appropriate to include here a derivation of Stirling’s formula. Namely, recall Euler’s Gamma function: Z (1.3.20) Γ(t) ≡ xt−1 e−x dx, t ∈ (−1, ∞). [0,∞)
The goal of this exercise is to prove that t √ t (1.3.21) Γ(t + 1) ∼ 2πt e
as
t % ∞,
where the tilde “∼” means that the two sides are asymptotic to one another in the sense that their ratio tends to 1. (See Exercise 2.1.16 for another approach.) The first step is to make the problem look like one to which Exercise 1.3.16 is applicable. Thus, make the substitution x = ty, and apply Exercise 1.3.16 to see that ! 1t 1 Z Γ(t + 1) t −→ e−1 . = y t e−ty dy tt+1 [0,∞)
This is, of course, far less than we want to know. Nonetheless, it does show that all the action is going to take place near y = 1 and that the principal factor in −t the asymptotics of Γ(t+1) tt+1 is e . In order to highlight these observations, make the substitution y = z + 1 and obtain Z Γ(t + 1) = (1 + z)t e−tz dz. tt+1 e−t (−1,∞) 2
Before taking the next step, introduce the function R(z) = log(1 + z) − z + z2
for z ∈ (−1, 1), and check that R(z) ≤ 0 if z ∈ (−1, 0] and that |R(z)| ≤ everywhere in (−1, 1). Now let δ ∈ (0, 1) be given, and show that Z −δ t tz tδ 2 −δ t 1 + z e dz ≤ (1 − δ) (1 − δ)e ≤ exp − 2 −1
and Z
∞
h it−1 Z ∞ e−tz dz ≤ 1 + δ e−δ (1 + z)e−z dz δ δ3 tδ 2 . + ≤ 2 exp 1 − 3(1 − δ) 2 1+z
δ
t
|z|3 3(1−|z|)
Exercises for § 1.3
33
tz 2
Next, write (1 + z)t e−tz = e− 2 etR(z) . Then Z Z t −tz tz 2 1+z e dz = e− 2 dz + E(t, δ), |z|≤δ
|z|≤δ
where
Z E(t, δ) =
e−
tz 2 2
etR(z) − 1 dz.
|z|≤δ
Check that Z r Z 2 tδ 2 z2 1 2 2π − tz2 e− 2 dz ≤ 1 e− 2 . dz − e = t− 2 1 |z|≤δ t t2 δ |z|≥t 2 δ
At the same time, show that Z Z 2 − tz2 +|R(z)| dz ≤ t |E(t, δ)| ≤ t |R(z)|e
|z|3 e−
|z|≤δ
|z|≤δ
tz 2 3−5δ 2 3(1−δ)
dz ≤
12(1 − δ) (3 − 5δ)2 t
p as long as δ < 35 . Finally, take δ = 2t−1 log t, and combine these to conclude that there is a C < ∞ such that C Γ(t + 1) − 1 ≤ , t ∈ [1, ∞). √ 2πt t t t e
Exercise 1.3.22. Inspired by T.H. Carne,2 here is a rather different sort of application of large deviation estimates. Namely, the goal is to show that for each n ≥ 2 and 1 ≤ m < n there exists an (m − 1)st order polynomial pm,n with the property that 2 n x − pm,n (x) ≤ 2 exp − m for x ∈ [−1, 1]. 2n
(i) Given a C-valued f on Z, define Af : Z −→ C by f (n + 1) + f (n − 1) , n ∈ Z, 2 and show that, for any n ≥ 1, An f = EP f (Sn ) , where Sn is the sum of n P-independent, {−1, 1}-valued Bernoulli random variables with mean value 0. Af (n) =
2
T.H. Carne, “A transformation formula for Markov chains,” Bull. Sc. Math., 109, pp. 399– 405 (1985). As Carne points out, what he is doing is the discrete analog of Hadamard’s representation, via the Weierstrass transform, of solutions to heat equations in terms of solutions to the wave equations.
34
1 Sums of Independent Random Variables
(ii) Show that, for each z ∈ C, there is a unique sequence {Q(m, z) : m ∈ Z} ⊆ C satisfying Q(0, z) = 1, Q(−m, z) = Q(m, z), and AQ( · , z) (m) = zQ(m, z) for all m ∈ Z. In fact, show that, for each m ∈ Z+ : Q(m, · ) is a polynomial of degree m and Q(m, cos θ) = cos(mθ),
θ ∈ C.
In particular, this means that |Q(n, x)| ≤ 1 for all x ∈ [−1, 1]. (It also means that Q(n, · ) is the nth Chebychev polynomial.) (iii) Using induction on n ∈ Z+ , show that n A Q( · , z) (m) = z n Q(m, z),
m ∈ Z and z ∈ C,
and conclude that h i z n = E Q Sn , z ,
n ∈ Z+
and z ∈ C.
In particular, if h i pm,n (z) ≡ E Q Sn , z), Sn < m = 2−n
X |2`−n|<m
n Q(2` − n, z), `
conclude that (cf. Exercise 1.3.18)
m2 sup |x − pm,n (x)| ≤ P |Sn | ≥ m ≤ 2 exp − 2n x∈[−1,1] n
for all 1 ≤ m ≤ n.
(iv) Suppose that A is a self-adjoint contraction on the real or complex Hilbert space H (i.e., (f, Ag)H = (g, Af )H and kAf kH ≤ kf kH for all f, g ∈ H). Next, assume that f, A` g H = 0 for some f, g ∈ H and each 0 ≤ ` < m. Show that
2 f, An g ≤ 2kf kH kgkH exp − m H 2n
for n ≥ m.
(See Exercise 2.3.30 for an application.) Hint: Note that f, pm,n (A)g H = 0, and use the Spectral Theorem to see that, for any polynomial p, kp(A)f kH ≤
sup |p(x)| kf kH , x∈[−1,1]
f ∈ H.
§ 1.4 The Strong Law of Large Numbers
35
§ 1.4 The Strong Law of Large Numbers In this section I will discuss a few almost sure convergence properties of averages of independent random variables. Thus, once again, {Xn : n ≥ 1} will be a sequence of independent random variables on a probability space Ω, F, P , and Sn and S n will be, respectively, the sum and average of X1 , . . . , Xn . Throughout this section, the reader should notice how much more immediately important a role independence (as opposed to orthogonality) plays than it did 1.2. in Section To get started, I point out that, for both {Sn : n ≥ 1} and S n : n ≥ 1 , the set on which convergence occurs has P-measure either 0 or 1. In fact, we have the following simple application of Kolmogorov’s 0–1 Law (Theorem 1.1.2). + Lemma 1.4.1. For any sequence a : n ∈ Z ⊆ R and any sequence n bn : n ∈ Z+ ⊆ (0, ∞) that converges to an element of (0, ∞], the set on which
lim
n→∞
Sn − an bn
exists in R
has P-measure either 0 or 1. In fact, if bn −→ ∞ as n → ∞, then both
lim
n→∞
Sn − an bn
and
Sn − an bn n→∞ lim
are P-almost surely constant. Proof: Simply observe that all of the events and functions involved can be expressed in terms of {Sm+n − Sm : n ≥ 1} for each m ∈ Z+ and are therefore tail-measurable. The following beautiful statement, which was proved originally by Kolmogorov, is the driving force behind many of the almost sure convergence results about both {Sn : n ≥ 1} and S n : n ≥ 1 .
Theorem 1.4.2. variables, and if
If the Xn ’s are independent, P-square integrable random ∞ X
(1.4.3)
Var Xn ) < ∞,
n=1
then
∞ X
Xn − EP Xn converges P-almost surely.
n=1
Note that, since (1.4.4)
! ∞ n X 1 X Var X` , sup P X` − EP X` ≥ ≤ 2 n≥N `=N
`=N
36
1 Sums of Independent Random Variables
P∞ (1.4.3) certainly implies that the series n=1 Xn − EP Xn converges in Pmeasure. Thus, all that I am attempting to do here is replace a convergence in measure statement with an almost sure one. Obviously, this replacement would be trivial if the “supn≥N ” in (1.4.3) appeared on the other side of P. The remarkable fact which we are about to prove is that, in the present situation, the “supn≥N ” can be brought inside! Theorem 1.4.5 (Kolmogorov’s Inequality). and P-square integrable, then (1.4.6)
If the Xn ’s are independent
n ! ∞ X 1 X Var Xn P sup X` − EP X` ≥ ≤ 2 n=1 n≥1 `=1
for each > 0. (See Exercise 1.4.21 for more information.) Proof: Without loss in generality, assume that each Xn has mean value 0. Given 1 ≤ n < N , note that 2 SN − Sn2 = SN − Sn
2
+ 2 SN − Sn Sn ≥ 2 SN − Sn Sn ;
and therefore, since SN −Sn has mean value 0 and is independent of the σ-algebra σ {X1 , . . . , Xn } , (*)
2 EP SN , An ≥ EP Sn2 , An for any An ∈ σ {X1 , . . . , Xn } .
In particular, if A1 = |S1 | > and n o An+1 = Sn+1 > and max S` ≤ , 1≤`≤n
n ∈ Z+ ,
then, the An ’s are mutually disjoint, BN ≡
N [ max Sn > = An ,
1≤n≤N
n=1
and so (*) implies that N N 2 X 2 X P E SN , BN = E SN , An ≥ EP Sn2 , An P
n=1
≥ 2
N X n=1
n=1
P An = 2 P B N .
§ 1.4 The Strong Law of Large Numbers
37
Thus, ∞ 2 X P sup Sn > = lim 2 P BN ≤ lim EP SN ≤ EP Xn2 , 2
n≥1
N →∞
N →∞
n=1
and so the result follows after one takes left limits with respect to > 0. Proof of Theorem 1.4.2: Again assume that the Xn ’s have mean value 0. By (1.4.6) applied to XN +n : n ∈ Z+ , we see that (1.4.3) implies ∞ 1 X EP Xn2 −→ 0 as N → ∞ P sup Sn − SN ≥ ≤ 2 n>N n=N +1
for every > 0, and this is equivalent to the P-almost sure Cauchy convergence of {Sn : n ≥ 1}. In order to convert the conclusion in Theorem 1.4.2 into a statement about S n : n ≥ 1 , I will need the following elementary summability fact about sequences of real numbers. Lemma 1.4.7 (Kronecker). Let bn : n ∈ Z+ be a non-decreasing sequence of positive numbers that tend to ∞, and set βn = bn − bn−1 , where b0 ≡ 0. If {sn : n ≥ 1} ⊆ R is a sequence that converges to s ∈ R, then n 1 X β` s` −→ s. bn `=1
In particular, if {xn : n ≥ 1} ⊆ R, then n ∞ X 1 X xn x` −→ 0 as n → ∞. converges in R =⇒ bn b n=1 n `=1
Proof: To prove the first part, assume that s = 0, and for given > 0 choose N ∈ Z+ so that |s` | < for ` ≥ N . Then, with M = supn≥1 |sn |, n Mb 1 X N + −→ as n → ∞. β` s` ≤ bn bn `=1 Pn Turning to the second part, set y` = xb`` , s0 = 0, and sn = `=1 y` . After summation by parts, n n 1 X 1 X β` s`−1 ; x` = sn − bn bn `=1
`=1
and so, since sn −→ s ∈ R as n → ∞, the first part gives the desired conclusion. After combining Theorem 1.4.2 with Lemma 1.4.7, we arrive at the following interesting statement.
38
1 Sums of Independent Random Variables
Corollary 1.4.8. Assume that {bn : n ≥ 1} ⊆ (0, ∞) increases to infinity as n → ∞, and suppose that {Xn : n ≥ 1} is a sequence of independent, P-square integrable random variables. If ∞ X Var Xn < ∞, b2n n=1
then
n 1 X X` − EP X` −→ 0 bn
P-almost surely.
`=1
As an immediate consequence of the preceding, we see that S n −→ m P-almost surely if the Xn ’s are identically distributed and P-square integrable. In fact, without very much additional effort, we can also prove the following much more significant refinement of the last part of Theorem 1.3.1. Theorem 1.4.9 (Kolmogorov’s Strong Law). Let Xn : n ∈ Z+ be a sequence of P-independent, identically distributed random variables. If X1 is P-integrable and has mean value m, then, as n → ∞, S n −→ m P-almost surely and in L1 (P; R). Conversely, if S n converges (in R) on a set of positive P-measure, then X1 is P-integrable. P Proof: Assume that X is P-integrable and that E X1 = 0. Next, set Yn = 1 Xn 1[0,n] |Xn | , and note that ∞ X
P Yn 6= Xn =
n=1
≤
∞ X
P |Xn | > n
n=1 ∞ Z n X n=1
P |X1 | > t dt = EP |X1 | < ∞.
n−1
Thus, by the first part of the Borel–Cantelli Lemma, P ∃n ∈ Z+ ∀N ≥ n YN = XN = 1. Pn In particular, if T n = n1 `=1 Y` for n ∈ Z+ , then, for P-almost every ω ∈ Ω, T n (ω) −→ 0 if and only if S n (ω) −→ 0. Finally, to see that T n −→ 0 P-almost surely, first observe that, because EP [X1 ] = 0, by the first part of Lemma 1.4.7, n
1X P E [Y` ] = lim EP X1 , |X1 | ≤ n = 0, n→∞ n→∞ n lim
`=1
and therefore, by Corollary 1.4.8, it suffices for us to check that ∞ X EP [Yn2 ] < ∞. n2 n=1
§ 1.4 The Strong Law of Large Numbers To this end, set C = sup ` `∈Z+
and note that ∞ X EP [Y 2 ] n
n=1
n2
39
∞ X 1 , n2 n=`
n ∞ X 1 X P 2 E X , ` − 1 < |X | ≤ ` = 1 1 n2 n=1 `=1
=
∞ X `=1
≤C
∞ X 1 EP X12 , ` − 1 < |X1 | ≤ ` n2 n=`
∞ X 1 `=1
`
EP X12 , ` − 1 < |X1 | ≤ ` ≤ C EP |X1 | < ∞.
Thus, the P-almost sure convergence is now established, and the L1 (P; R)-convergence result was proved already in Theorem 1.2.6. Turning to the converse assertion, first note that (by Lemma 1.4.1) if S n converges in R on a set of positive P-measure, then it converges P-almost surely to some m ∈ R. In particular,
|Xn | = lim S n − S n−1 = 0 P-almost surely; n→∞ n→∞ n and so, if An ≡ |Xn | > n , then P limn→∞ An = 0. But the An ’s are mutually independent, and by the second part of the Borel–Cantelli Lemma, we Ptherefore, ∞ now know that n=1 P An < ∞. Hence, Z ∞ ∞ X P E |X1 | = P |X1 | > t dt ≤ 1 + P |Xn | > n < ∞. lim
0
n=1
Remark 1.4.10. A reason for being interested in the converse part of Theorem 1.4.9 is that it provides a reconciliation between the measure theory vs. frequency schools of probability theory. Although Theorem 1.4.9 is the centerpiece of this section, I want to give another approach to the study of the almost sure convergence properties of {Sn : n ≥ 1}. In fact, following P. L´evy, I am going to show that {Sn : n ≥ 1} converges P-almost surely if it converges in P-measure. Hence, for example, Theorem 1.4.2 can be proved as a direct consequence of (1.4.4), without appeal to Kolmogorov’s Inequality. The key to L´evy’s analysis lies in a version of the reflection principle, whose statement requires the introduction of a new concept. Given an R-valued random variable Y , say that α ∈ R is a median of Y and write α ∈ med(Y ), if (1.4.11) P Y ≤ α ∧ P Y ≥ α ≥ 12 .
40
1 Sums of Independent Random Variables
Notice that (as distinguished from a mean value) every Y admits a median; for example, it is easy to check that
α ≡ inf t ∈ R : P Y ≤ t ≥ 12 is a median of Y . In addition, it is clear that med(−Y ) = −med(Y )
and
med (β + Y ) = β + med (Y ) for all β ∈ R.
On the other hand, the notion of median is flawed by the fact that, in general, a random variable will admit an entire non-degenerate interval of medians. In addition, it is neither easy to compute the medians of a sum in terms of the medians of the summands nor to relate the medians of an integrable random variable to its mean value. Nonetheless, at least if Y ∈ Lp (P; R) for some p ∈ [1, ∞), the following estimate provides some information. Namely, since, for α ∈ med(Y ) and β ∈ R,
|α − β|p ≤ |α − β|p P Y ≥ α ∧ P Y ≤ α ≤ EP |Y − β|p , 2 we see that, for any p ∈ [1, ∞) and Y ∈ Lp (P; R), p1 for all β ∈ R and α ∈ med (Y ). |α − β| ≤ 2EP |Y − β|p
In particular, if Y ∈ L2 (P ) and m is the mean value of Y , then (1.4.12)
|α − m| ≤
p
2Var(Y )
for all α ∈ med(Y ).
Theorem 1.4.13 (L´ evy’s Reflection Principle). Let Xn : n ∈ Z+ be a sequence of P-independent random variables, and, for k ≤ `, choose α`,k ∈ med S` − Sk . Then, for any N ∈ Z+ and > 0, (1.4.14)
max Sn + αN,n ≥
P
1≤n≤N
≤ 2P SN ≥ ,
and therefore (1.4.15)
P
max Sn + αN,n ≥
1≤n≤N
≤ 2P |SN | ≥ .
Proof: Clearly (1.4.15) follows by applying (1.4.14) to both the sequences Xn : n ≥ 1} and {−Xn : n ≥ 1} and then adding the two results.
§ 1.4 The Strong Law of Large Numbers
41
To prove (1.4.14), set A1 = S1 + αN,1 ≥ and An+1 = max S` + αN,` < and Sn+1 + αN,n+1 ≥ 1≤`≤n
for 1 ≤ n < N . Obviously, the An ’s are mutually disjoint and N [
An =
n=1
max Sn + αN,n ≥ .
1≤n≤N
In addition, {SN ≥ ⊇ An ∩ SN − Sn ≥ αN,n
for each 1 ≤ n ≤ N.
Hence, N X P SN ≥ ≥ P An ∩ SN − Sn ≥ αN,n n=1
N 1 1X P An = P max Sn + αN,n ≥ , ≥ 1≤n≤N 2 2 n=1 where, in the passage to the last line, I have used the independence of the sets An and SN − Sn ≥ αN,n . + Corollary 1.4.16. Let X : n ∈ Z be n a sequence of independent random + variables, and assume that Sn : n ∈ Z converges in P-measure to an Rvalued random variable S. Then Sn −→ S P-almost surely. (Cf. Exercise 1.4.25 as well.) Proof: What I must show is that, for each > 0, there is an M ∈ Z+ such that sup P max Sn+M − SM ≥ < . N ≥1
1≤n≤N
To this end, let 0 < < 1 be given, and choose M ∈ Z+ so that for all 1 ≤ k < n. < P Sn+M − Sk+M ≥ 2 2 Next, for a given N ∈ Z+ , choose αN,n ∈ med SM +N − SM +n for 0 ≤ n ≤ N . Then |αN,n | ≤ 2 , and so, by (1.4.15) applied to {XM +n : n ≥ 1}, P max SM +n − SM ≥ ≤ P max SM +n − SM + αN,n ≥ 1≤n≤N 1≤n≤N 2 < . ≤ 2P SM +N − SM ≥ 2
42
1 Sums of Independent Random Variables
Remark 1.4.17. The most beautiful and startling feature of L´evy’s line of reasoning is that it requires no integrability assumptions. Of course, in many applications of Corollary 1.4.16, integrability considerations enter into the proof that {Sn : n ≥ 1} converges in P-measure. Finally, a word of caution may be in order. Namely, the result in Corollary 1.4.16 applies to the quantities Sn themselves; it does not apply to associated quantities like S n . Indeed, suppose that {Xn : n ≥ 1} is a sequence of independent, identically distributed random variables that satisfy
− 12 P Xn ≤ −t = P Xn ≥ t = 1 + t2 log e4 + t2
for all t ≥ 0.
On the one hand, by Exercise 1.2.11, we know that the associated averages S n tend to 0 in probability. On the other hand, by the second part of Theorem 1.4.9, we know that the sequence S n : n ≥ 1 diverges almost surely.
Exercises for § 1.4 Exercise 1.4.18. Let X and Y be non-negative random variables, and suppose that
i 1 h P X ≥ t ≤ EP Y, X ≥ t , t
(1.4.19)
t ∈ (0, ∞).
Show that
(1.4.20)
p P p p1 , E Y p−1
p1 ≤ EP X p
p ∈ (1, ∞).
Hint: First, reduce to the case when X is bounded. Next, recall that, for any measure space E, F, µ , any non-negative, measurable f on E, F , and any α ∈ (0, ∞), Z
Z
α
α−1
f (x) µ(dx) = α E
t
Z
µ f > t dt = α
(0,∞)
tα−1 µ f ≥ t dt.
(0,∞)
Use this together with (1.4.19) to justify the relation E Xp ≤ p
Z
P
tp−2 EP Y, X ≥ t dt
(0,∞)
" Z = pE Y P
0
#
X
t
p−2
dt =
p EP X p−1 Y , p−1
and arrive at (1.4.20) after an application of H¨ older’s Inequality.
Exercises for § 1.4
43
Exercise 1.4.21. Let {Xn : n ≥ 1} be a sequence of mutually independent, integrable random variables with mean value 0, and assume that 2 P∞ P-square E X < ∞. Let S denote the random variable (guaranteed by Theorem n 1 1.4.2) to which {Sn : n ≥ 1} converges P-almost surely, and, using elementary orthogonality considerations, check that Sn −→ S in L2 (P; R) as well. Next, after examining the proof of Kolmogorov’s Inequality (cf. (1.4.6)), show that 2 2 1 P 2 P sup Sn ≥ t ≤ E S , sup Sn ≥ t , t > 0. t n∈Z+ n∈Z+
Finally, by applying (1.4.20), show that p h i 2p 2p p P EP S , (1.4.22) E sup Sn ≤ p−1
p ∈ (1, ∞),
n∈Z+
and conclude from this that, for each p ∈ (2, ∞), {Sn : n ≥ 1} converges to S in Lp (P ) if and only if S ∈ Lp (P ). Exercise 1.4.23. If X ∈ L2 (P; R), then it is easy to characterize its mean m as the c ∈ R that minimizes EP (X − c)2 . Assuming that X ∈ L1 (P; R), show that α ∈ med(X) if and only if EP |X − α| = min EP |X − c| . c∈R
Hint: Show that, for any a, b ∈ R, E |X − b| − EP |X − a| =
Z
P
b
P(X ≤ t) − P(X ≥ t) dt.
a
Exercise 1.4.24. Let {Xn : n ≥ 1} be a sequence of P-square integrable random variables that converges in probability to a random variable X, and assume that supn≥1 Var(Xn ) < ∞. Show that X is square integrable and that EP |Xn − X| −→ 0. In particular, if, in addition, Var(Xn ) −→ Var(X), show that EP |Xn − X|2 −→ 0. Hint: Let αn ∈ med(Xn ), and show that α+ = limn→∞ αn and α− = limn→∞ αn are both of med(X). Combine this with (1.4.12) to conclude that elements supn≥1 EP [Xn ] < ∞ and therefore that supn≥1 EP [X 2 ] < ∞.
Exercise 1.4.25. The following variant of Theorem 1.4.13 is sometimes useful and has the advantage that it avoids the introduction of medians. Namely, show that, for any t ∈ (0, ∞) and n ∈ Z+ , P |Sn | > t . P max |Sn | ≥ 2t ≤ 1≤m≤n 1 − max P |Sn − Sm | > t 1≤m≤n
Note that this can be used in place of (1.4.15) when proving results like the one in Corollary 1.4.16.
44
1 Sums of Independent Random Variables
Exercise 1.4.26. A random variable X is said to be symmetric if −X has the same distribution as X itself. Obviously, the most natural choice of median for a symmetric random variable is 0; and thus, because sums of independent, symmetric random variables are again symmetric, (1.4.14) and (1.4.15) are particularly useful when the Xn ’s are symmetric, since the α`,k ’s can then be taken to be 0. In this connection, consider the following interesting variation on the theme of Theorem 1.4.13. (i) Let X1 , . . . , Xn , . . . be independent, symmetric random variables, set Mn (ω) = |X` (ω)|, let τn (ω) be the smallest 1 ≤ ` ≤ n with the property that max1≤`≤n X` (ω) = Mn (ω), and define Yn (ω) = Xτn (ω) (ω)
and Sˆn = Sn − Yn .
Show that ω ∈ Ω 7−→ Sˆn (ω), Yn (ω) ∈ R2
and ω ∈ Ω 7−→ −Sˆn (ω), Yn (ω) ∈ R2
have the same distribution, and conclude first that P Yn ≥ t ≤ P Yn ≥ t & Sˆn ≥ 0 + P Yn ≥ t & Sˆn ≤ 0 = 2P Yn ≥ t & Sˆn ≥ 0 ≤ 2P Sn ≥ t , for all t ∈ R, and then that P max X` ≥ t ≤ 2P Sn ≥ t , 1≤`≤n
t ∈ [0, ∞).
(ii) Continuing in the same setting, add the assumption that the Xn ’s are identically distributed, and use part (i) to show that
lim P |S n | ≤ C = 1
n→∞
for some C ∈ (0, ∞) =⇒ lim nP |X1 | ≥ n = 0. n→∞
Hint: Note that P and that
1−(1−x)n x
max |X` | > t = 1 − P(|X1 | ≤ t)n
1≤`≤n
−→ n as x & 0.
In conjunction with Exercise 1.2.11, this proves that if {Xn : n ≥ 1} is a sequence of independent, identically distributed symmetric random variables, then S n −→ 0 in P-probability if and only if limn→∞ nP |X1 | ≥ n = 0.
Exercises for § 1.4
45
Exercise 1.4.27. Let X and X 0 be a pair of independent random variables that have the same distribution, let α be a median of X, and set Y = X − X 0 . (i) Show that Y is symmetric and that P |X − α| ≥ t ≤ 2P |Y | ≥ t
for all
t ∈ [0, ∞),
and conclude that, for any p ∈ (0, ∞),
1 1 1 2− p ∨1 EP |Y |p p ≤ 2EP |Y |p p + |α| . In particular, |X|p is integrable if and only if |Y |p is. (ii) The result in (i) leads to my final refinement of The Weak Law of Large Numbers. Namely, let {Xn : n ≥ 1} be a sequence of independent, identically distributed random variables. By combining Exercise 1.2.11, part (ii) in Exercise 1.4.26, and part (i) above, show that1
lim P S n ≤ C = 1
for some C ∈ (0, ∞) =⇒ lim nP |X1 | ≥ n = 0 n→∞ =⇒ S n − EP X1 , |X1 | ≤ n −→ 0 in P-probability. n→∞
Exercise 1.4.28. Let {Xn : n ≥ 1} be a sequence of mutually independent, identically distributed, P-integrable random variables with mean value m. As we already know, when m > 0, the partial sums Sn tend, P-almost surely, to +∞ at an asymptotic linear rate m; and, of course, when m < 0, the situation is similar at −∞. On the other hand, when m = 0, we know that, if |Sn | tends to ∞ at all, then, P-almost surely, it does so at a strictly sublinear rate. In this exercise, you are to sharpen this statement by proving that m = 0 =⇒ lim |Sn | < ∞ P-almost surely. n→∞
The beautiful argument given below is due to Y. Guivarc’h, but its full power cannot be appreciated in the present context (cf. Exercise 6.2.19). Furthermore, a classic result (cf. Exercise 5.2.43) due to K.L. Chung and W.H. Fuchs gives a much better result for the independent random variables. Their result says that limn→∞ |Sn | = 0 P-almost surely. In order to prove the assertion here, assume that limn→∞ |Sn | = ∞ with positive P-probability, use Kolmogorov’s 0–1 Law to see that |Sn | −→ ∞ Palmost surely, and proceed as follows. 1
These ideas are taken from the book by Wm. Feller cited at the end of § 1.2. They become even more elegant when combined with a theorem due to E.J.G. Pitman, which is given in Feller’s book.
46
1 Sums of Independent Random Variables
(i) Show that there must exist an > 0 with the property that P ∀` > k S` − Sk ≥ ≥ for some k ∈ Z+ and therefore that P(A) ≥ ,
where A ≡ ω : ∀` ∈ Z+ S` (ω) ≥ .
(ii) For each ω ∈ Ω and n ∈ Z+ , set o n Γn (ω) = t ∈ R : ∃1 ≤ ` ≤ n t − S` (ω) < 2
and
o n Γn0 (ω) = t ∈ R : ∃1 ≤ ` ≤ n t − S`0 (ω) < 2 ,
Pn where Sn0 ≡ `=1 X`+1 . Next, let Rn (ω) and Rn0 (ω) denote the Lebesgue measure of Γn (ω) and Γn0 (ω), respectively; and, using the translation invariance of Lebesgue measure, show that Rn+1 (ω) − Rn0 (ω) ≥ 1A0 (ω), where A0 ≡ ω : ∀` ≥ 2 S` (ω) − S1 (ω) ≥ . On the other hand, show that EP Rn0 = EP Rn
and P(A0 ) = P(A),
and conclude first that P(A) ≤ EP Rn+1 − Rn , and then that
n ∈ Z+ ,
1 P E Rn . n→∞ n
P(A) ≤ lim
(iii) In view of parts (i) and (ii), what remains to be done is show that m = 0 =⇒ lim
n→∞
1 P E Rn = 0. n
But, clearly, 0 ≤ Rn (ω) ≤ n. Thus, it is enough to show that, when m = 0, Rn n −→ 0 P-almost surely; and, to this end, first check that
Rn (ω) Sn (ω) −→ 0, −→ 0 =⇒ n n
and, finally, apply The Strong Law of Large Numbers.
Exercises for § 1.4
47
Exercise 1.4.29. As I have already said, for many applications The Weak Law of Large Numbers is just as good as and even preferable to the Strong Law. Nonetheless, here is an application in which the full strength of the Strong Law plays an essential role. Namely, I want to use the Strong Law to produce examples of continuous, strictly increasing functions F on [0, 1] with the property that their derivative F (y) − F (x) =0 y→x y−x
F 0 (x) ≡ lim
at Lebesgue-almost every x ∈ (0, 1).
By familiar facts about functions of a real variable, one knows that such functions F are in one-to-one correspondence with non-atomic, Borel probability measures µ on [0, 1] which charge every non-empty open subset but are singular to Lebesgue’s measure. Namely, F is the distribution function determined by µ: F (x) = µ (−∞, x] . +
+
(i) Set Ω = {0, 1}Z , and, for each p ∈ (0, 1), take Mp = (βp )Z , where βp on {0, 1} is the Bernoulli measure with βp ({1}) = p = 1 − βp ({0}). Next, define ω ∈ Ω 7−→ Y (ω) ≡
∞ X
2−n ωn ∈ [0, 1],
n=1
and let µp denote the Mp -distribution of Y . Given n ∈ Z+ and 0 ≤ m < 2n , show that µp m2−n , (m + 1)2−n = p`m,n (1 − p)n−`m,n , Pn n −n where ` = = m,n k=1 ωk and (ω1 , . . . , ωn ) ∈ {0, 1} is determined by m2 Pn −k ωk . Conclude, in particular, that µp is non-atomic and charges every k=1 2 non-empty open subset of [0, 1]. (iii) Given x ∈ [0, 1) and n ∈ Z+ , define n (x) =
1
if 2n−1 x − b2n−1 xc ≥
0
if 2n−1 x − b2n−1 xc <
1 2 1 2,
where denotes the integer part of s. If {n : n ≥ 1} ⊆ {0, 1} satisfies Pbsc ∞ x = 1 2−m m , show that m = m (x) for all m ≥ 1 if and only if m = 0 for infinitely many m ≥ 1. In particular, conclude first that ωn = n Y (ω) , n ∈ Z+ , for Mp -almost every ω ∈ Ω and, second, by the Strong Law, that n 1 X n (x) −→ p n m=1
Thus, µp1 ⊥ µp2 whenever p1 6= p2 .
for µp -almost every x ∈ [0, 1].
48
1 Sums of Independent Random Variables
(iv) By Lemma 1.1.6, we know that µ 12 is Lebesgue measure λ[0,1] on [0, 1]. Hence, we now know that µp ⊥ λ[0,1] when p 6= 12 . In view of the introductory remarks, this completes the proof that, for each p ∈ (0, 1) \ { 12 }, the function Fp (x) = µp (−∞, x] is a strictly increasing, continuous function on [0, 1] whose derivative vanishes at Lebesgue-almost every point. Here, one can do better. Namely, referring to part (iii), let ∆p denote the set of x ∈ [0, 1) such that
1 Σn (x) = p, n→∞ n lim
where Σn (x) ≡
n X
m (x).
m=1
We know that ∆ 12 has Lebesgue measure 1. Show that, for each x ∈ ∆ 12 and p ∈ (0, 1) \ { 12 }, Fp is differentiable with derivative 0 at x.
Hint: Given x ∈ [0, 1), define Ln (x) =
n X
2−m m (x)
and Rn (x) = Ln (x) + 2−n .
m=1
Show that
Fp Rn (x) − Fp Ln (x) = Mp
n X
! 2−m ωm = Ln (x)
= pΣn (x) (1 − p)n−Σn (x) .
m=1
When p ∈ that
(0, 1) \ { 12 }
and x ∈ ∆ 12 , use this together with 4p(1 − p) < 1 to show ! Fp Rn (x) − Fp Ln (x) < 0. lim n log n→∞ Rn (x) − Ln (x)
To complete the proof, for given x ∈ ∆ 12 and n ≥ 2 such that Σn (x) ≥ 2, let
mn (x) denote the largest m < n such that m (x) = 1, and show that mnn(x) −→ 1 as n → ∞. Hence, since 2−n−1 < h ≤ 2−n implies that Fp (x) − Fp (x − h) n−mn (x)+1 Fp Rn (x) − Fp Ln (x) , ≤2 Rn (x) − Ln (x) h
one concludes that Fp is left-differentiable at x and has left derivative equal to 0 there. To get the same conclusion about right derivatives, simply note that Fp (x) = 1 − F1−p (1 − x). (v) Again let p ∈ (0, 1) \ { 12 } be given, but this time choose x ∈ ∆p . Show that
lim
h&0
Fp (x + h) − Fp (x) = +∞. h
The argument is similar to the one used to handle part (iv). However, this time the role played by the inequality 4pq < 1 is played here by (2p)p (2q)q > 1 when q = 1 − p.
§ 1.5 Law of the Iterated Logarithm
49
§ 1.5 Law of the Iterated Logarithm Let X1 , . . . , Xn , . . . be a sequence of independent, identically distributed random variables with mean value 0 and variance 1. In this section, I will investigate exactly how large {Sn : n ∈ Z+ } can become as n → ∞. To get a feeling for what one should be expecting, first note that, by Corollary 1.4.8, for any non-decreasing {bn : n ≥ 1} ⊆ (0, ∞), Sn −→ 0 bn
P-almost surely if
∞ X 1 < ∞. 2 b n=1 n 1
Thus, for example, Sn grows more slowly than n 2 log n. On the other hand, if Sn ; the Xn ’s are N (0, 1)-random variables, then so are the random variables √ n and therefore, for every R ∈ (0, ∞),
P
Sn lim √ ≥ R n
n→∞
[ Sn S √ ≥ R ≥ lim P √N ≥ R > 0. = lim P N →∞ N →∞ n N
n≥N
Hence, at least for normal random variables, one can use Lemma 1.4.1 to see that Sn lim √ = ∞ P-almost surely; n→∞ n 1
and so Sn grows faster than n 2 . If, as we did in Section 1.3, we proceed on the assumption that Gaussian random variables are typical, we should expect the growth rate of the Sn ’s to be 1 1 something between n 2 and n 2 log n. What, in fact, turns out to be the precise growth rate is
(1.5.1)
Λn ≡
q 2n log(2) (n ∨ 3),
where log(2) x ≡ log log x (not the logarithm with base 2) for x ∈ [e, ∞). That is, one has The Law of the Iterated Logarithm: (1.5.2)
Sn = 1 P-almost surely. n→∞ Λn lim
This remarkable fact was discovered first for Bernoulli random variables by Khinchine, was extended by Kolmogorov to random variables possessing 2 + moments, and eventually achieved its final form in the work of Hartman and Wintner. The approach that I will adopt here is based on ideas (taught to me by M. Ledoux) introduced originally to handle generalizations of (1.5.2) to random
50
1 Sums of Independent Random Variables
variables with values in a Banach space.1 This approach consists of two steps. The first establishes a preliminary version of (1.5.2) that, although it is far cruder than (1.5.2) itself, will allow me to justify a reduction of the general case to the case of bounded random variables. In the second step, I deal with bounded random variables and more or less follow Khinchine’s strategy for deriving (1.5.2) once one has estimates like the ones provided by Theorem 1.3.12. In what follows, I will use the notation Λβ = Λ[β]
S[β] and S˜β = Λβ
for β ∈ [3, ∞),
where [β] is the integer part of β. Lemma 1.5.3. Let {Xn : n ≥ 1} be a sequence of independent, identically distributed random variables with mean value 0 and variance 1. Then, for any a ∈ (0, ∞) and β ∈ (1, ∞),2
lim S˜n ≤ a
n→∞
(a.s., P) if
∞ X 1 P S˜β m ≥ a β − 2 < ∞. m=1
Proof: Let β ∈ (1, ∞) be given and, for each m ∈ N and 1 ≤ n ≤ β m , let √αm,n be a median (cf. (1.4.11)) of S[β m ] −Sn . Noting that, by (1.4.12), αm,n ≤ 2β m , we know that Sn 1 ˜ ˜ 2 lim Sn = lim m−1max m Sn ≤ β lim m−1max m m→∞ β m→∞ β n→∞ ≤n≤β Λβ m ≤n≤β Sn + αm,n 1 , ≤ β 2 lim maxm m→∞ n≤β Λβ m
and therefore
P lim S˜n ≥ a ≤ P n→∞
lim max
m→∞ n≤β m
Sn + αm,n
Λβ m
! ≥ aβ
− 12
.
But, by Theorem 1.4.13,
! Sn + αm,n 1 1 ≥ aβ − 2 ≤ 2P S˜β m ≥ aβ − 2 , P maxm n≤β Λβ m
and so the desired result follows from the Borel–Cantelli Lemma. 1
See §§ 8.4.2 and 8.6.3 and, for much more information, M. Ledoux and M. Talagrand, Probability in Banach Spaces, Springer-Verlag, Ergebnisse Series 3.Folge·Band 23 (1991). 2 Here and elsewhere, I use (a.s.,P) to abbreviate “P-almost surely.”
§ 1.5 Law of the Iterated Logarithm
51
Lemma 1.5.4. For any sequence {Xn : n ≥ 1} of independent, identically distributed random variables with mean value 0 and variance σ 2 ,
lim S˜n ≤ 8σ
(1.5.5)
n→∞
(a.s., P).
Proof: Without loss in generality, I assume throughout that σ = 1; and, for the moment, I will also assume that the Xn ’s are symmetric (cf. Exercise 1.4.26). By Lemma 1.5.3, we will know that (1.5.5) holds with 8 replaced by 4 once I show that ∞ X 3 P S˜2m ≥ 2 2 < ∞.
(*)
m=0
In order to take maximal advantage of symmetry, let (Ω, F, P) be the probability space on which the Xn ’s are defined, use {Rn : n ≥ 1} to denote the sequence of Rademacher functions on [0, 1) introduced in Section 1.1, and set Q = λ[0,1) × P on [0, 1) × Ω, B[0,1) × F . It is then an easy matter to check that symmetry of the Xn ’s is equivalent to the statement that + ω ∈ Ω −→ X1 (ω), . . . , Xn (ω), . . . ∈ RZ has the same distribution under P as + (t, ω) ∈ [0, 1) × Ω 7−→ R1 (t)X1 (ω), . . . , Rn (t)Xn (ω), . . . ∈ RZ has under Q. Next, using the last part of (iii) in Exercise 1.3.18 with σk = Xk (ω), note that λ[0,1)
2m ! X t ∈ [0, 1) : Rn (t)Xn (ω) ≥ a n=1 # " a2 , a ∈ [0, ∞) and ω ∈ Ω. ≤ 2 exp − P2m 2 n=1 Xn (ω)2
Hence, if
) 2m 1 X 2 Xm (ω) ≥ 2 ω∈Ω: m 2 n=1
( Am ≡
and
2m )! X 3 , t ∈ [0, 1) : Rn (t)Xn (ω) ≥ 2 2 Λ2m
( Fm (ω) ≡ λ[0,1)
n=1
52
1 Sums of Independent Random Variables
then, by Tonelli’s Theorem,
o Z n 3 2 = Fm ω) P(dω) P ω ∈ Ω : S2m (ω) ≥ 2 Λ2m Ω
"
8Λ2m exp − P2m 2 2 n=1 Xn (ω)2 Ω
Z ≤2
#
h i P(dω) ≤ 2 exp −4 log(2) 2m + 2P Am .
P∞ Thus, (*) comes down to proving that m=0 P Am < ∞; and, in order to check this, I argue in much the same way as I did when I proved the converse statement in Kolmogorov’s Strong Law. Namely, set m
Tm =
2 X
Xn2 ,
Bm =
n=1
Tm+1 − Tm ≥2 , 2m
and T m =
Tm 2m
for m ∈ N. Clearly, P Am = P Bm . Moreover, the sets Bm , m ∈ N, are mutually independent; and therefore, by the Borel–Cantelli Lemma, I need only check that Tm+1 − Tm ≥ 2 = 0. P lim Bm = P lim m→∞ m→∞ 2m
But, by the Strong Law, we know that T m −→ 1 (a.s., P), and therefore it is clear that Tm+1 − Tm −→ 1 (a.s., P). 2m
I have now proved (1.5.5) with 4 replacing 8 for symmetric random variables. To eliminate the symmetry assumption, again let (Ω, F, P) be the probability 0 0 0 space on which the Xn ’s are defined, let Ω , F , P be a second copy of the same space, and consider the random variables (ω, ω 0 ) ∈ Ω × Ω0 7−→ Yn ω, ω 0 ≡ Xn (ω) − Xn (ω 0 ) under the measure Q ≡ P × P0 . Since the Yn ’s are obviously (cf. part (i) of Exercise 1.4.21) symmetric, the result which I have already proved says that
lim
n→∞
Sn (ω) − Sn (ω 0 )
Λn
5
≤ 22 ≤ 8
for Q-almost every (ω, ω 0 ) ∈ Ω × Ω0 .
n| Now suppose that limn→∞ |S Λn > 8 on a set of positive P-measure. Then, by Kolmogorov’s 0–1 Law, there would exist an > 0 such that
|Sn (ω)| ≥ 8 + for P-almost every ω ∈ Ω; n→∞ Λn lim
§ 1.5 Law of the Iterated Logarithm
53
and so, by Fubini’sTheorem,3 we would that, for Q-almost every (ω, ω 0 ) ∈ have 0 + + Ω × Ω , there is a nm (ω) : m ∈ Z ⊆ Z such that nm (ω) % ∞ and Sn (ω) (ω) − Sn (ω) (ω 0 ) Sn (ω) (ω) Sn (ω) (ω 0 ) m m m m ≥ . − lim ≥ lim lim m→∞ m→∞ Λnm (ω) Λnm (ω) Λnm (ω) m→∞
But, again by Fubini’s Theorem, this would mean that there exists a {nm : m ∈ Sn (ω0 ) ≥ for P0 -almost every Z+ } ⊆ Z+ such that nm % ∞ and limm→∞ Λmn m ω 0 ∈ Ω0 , and obviously this contradicts " # 2 0 1 Sn P −→ 0. = E 2 log(2) n Λn
We have now got the crude statement alluded to above. In order to get the more precise statement contained in (1.5.2), I will need the following application of the results in § 1.3. Lemma 1.5.6. Let {Xn : n ≥ 1} be a sequence of independent random variables with mean value 0, variance 1, and common distribution µ. Further, assume that (1.3.4) holds. Then, for each R ∈ (0, ∞) there is an N (R) ∈ Z+ such that ! # " r 8R log(2) n R2 log(2) n (1.5.7) P S˜n ≥ R ≤ 2 exp − 1 − K n
for n ≥ N (R). In addition, for each ∈ (0, 1], there is an N () ∈ Z+ such that, for all n ≥ N () and |a| ≤ 1 ,
(1.5.8)
h i 1 P S˜n − a < ≥ exp − a2 + 4K|a| log(2) n . 2
In both (1.5.7) and (1.5.8), the constant K ∈ (0, ∞) is the one in Theorem 1.3.15. Proof: Set Λn = λn = n
2 log(2) (n ∨ 3) n
12
.
To prove (1.5.7), simply apply the upper bound in the last part of Theorem 1.3.15 to see that, for sufficiently large n ∈ Z+ , 3 (Rλn )2 ˜ − K Rλn . P Sn ≥ R = P S n ≥ Rλn ≤ 2 exp −n 2 3
This is Fubini at his best and subtlest. Namely, I am using Fubini to switch between horizontal and vertical sets of measure 0.
54
1 Sums of Independent Random Variables
To prove (1.5.8), first note that
P S˜n − a < = P S n − an < n , where an = aλn and n = λn . Thus, by the lower bound in the last part of Theorem 1.3.15,
2 an K 2 ˜ + K|an | n + an P Sn − a < ≥ 1 − 2 exp −n 2 nn ! h i K exp − a2 + 2K|a| + a2 λn log(2) n ≥ 1− 2 2 log(2) n
for sufficiently large n’s.
Theorem 1.5.9 (Law of Iterated Logarithm). The equation (1.5.2) holds for any sequence {Xn : n ≥ 1} of independent, identically distributed random variables with mean value 0oand variance 1. In fact, P-almost surely, the set of n Sn limit points of Λn : n ≥ 1 coincides with the entire interval [−1, 1]. Equiva lently, for any f ∈ C R; R ,
lim f
(1.5.10)
n→∞
Sn Λn
=
sup f (t)
(a.s., P).
t∈[−1,1]
(Cf. Exercise 1.5.12 for a converse statement and §§ 8.4.2 and 8.6.3 for related results.) Proof: I begin with the observation that, because of (1.5.5), I may restrict my attention to the case when the Xn ’s are bounded random variables. Indeed, for any Xn ’s and any > 0, an easy truncation procedure allows us to find an ψ ∈ Cb (R; R) such that Yn ≡ ψ ◦ Xn again has mean value 0 and variance 1 while Zn ≡ Xn − Yn has variance less than 2 . Hence, if the result is known when the random variables are bounded, then, by (1.5.5) applied to the Zn ’s,
Pn m=1 Zm (ω) ˜ ≤ 1 + 8, lim Sn (ω) ≤ 1 + lim n→∞ n→∞ Λn and, for a ∈ [−1, 1],
Pn Zm (ω) lim S˜n (ω) − a ≤ lim m=1 ≤ 8 n→∞ Λn
n→∞
for P-almost every ω ∈ Ω.
§ 1.5 Law of the Iterated Logarithm
55
In view of the preceding, from now on I may and will assume that the Xn ’s are bounded. To prove that limn→∞ S˜n ≤ 1 (a.s., P), let β ∈ (1, ∞) be given, and use (1.5.7) to see that h i 1 1 P S˜β m ≥ β 2 ≤ 2 exp −β 2 log(2) β m + for all sufficiently large m ∈ Z . Hence, by Lemma 1.5.3 with a = β, we see that limn→∞ S˜n ≤ β (a.s., P) for every β ∈ (1, ∞). To complete the proof, I must still show that, for every a ∈ (−1, 1) and > 0, P lim S˜n − a < = 1.
n→∞
Because I want to get this conclusion as an application of the second part of the Borel–Cantelli Lemma, it is important that we be dealing with independent events, and for this purpose I use the result just proved to see that, for every integer k ≥ 2, lim S˜n − a ≤ inf lim S˜km − a k→∞ m→∞
n→∞
= inf
Skm − Skm−1 − a lim Λk m
P-almost surely.
k→∞ m→∞
Thus, because the events Skm − Skm−1 − a < , Ak,m ≡ Λkm
m ∈ Z+ ,
are independent for each k ≥ 2, all that I need to do is check that ∞ X P Ak,m = ∞ for sufficiently large k ≥ 2. m=1
But P Ak,m
Λkm Λkm a ˜ , < = P Skm −km−1 − Λkm −km−1 Λkm −km−1
and, because
Λkm − 1 = 0, lim max+ k→∞ m∈Z Λ m m−1 k −k
everything reduces to showing that ∞ X (*) P S˜km −km−1 − a < = ∞ m=1
for each k ≥ 2, a ∈ (−1, 1), and > 0. Finally, referring to (1.5.8), choose 0 > 0 so small that ρ ≡ a2 + 4K0 |a| < 1, and conclude that, when 0 < < 0 , h i 1 P S˜n − < ≥ exp −ρ log(2) n 2 for all sufficiently large n’s, from which (*) is easy.
56
1 Sums of Independent Random Variables
Remark 1.5.11. The reader should notice that the Law of the Iterated Logarithm provides a naturally occurring sequence of functions that converge in measure but not almost everywhere. Indeed, it is obvious that S˜n −→ 0 in 2 ˜ L (P; R), but the Law of the Iterated Logarithm says that Sn : n ≥ 1 is wildly divergent when looked at in terms of P-almost sure convergence. Exercises for § 1.5 Exercise 1.5.12. Let {Xn : n ≥ 1} be a sequence of mutually independent, identically distributed random variables for which
|Sn | < ∞ > 0. n→∞ Λn
(1.5.13)
P
lim
In this exercise I4 will outline a proof that X1 is P-square integrable, EP X1 = 0, and (1.5.14)
1 Sn Sn = EP X12 2 = − lim n→∞ Λn n→∞ Λn lim
(a.s., P).
(i) Using Lemma 1.4.1, show that there is a σ ∈ [0, ∞) such that (1.5.15)
lim
n→∞
Sn
Λn
=σ
(a.s., P).
Next, assuming that X1 is P-square integrable, use The Law of Large Strong Numbers together with Theorem 1.5.9 to show that EP X1 = 0 and 1 Sn Sn = − lim σ = EP X12 2 = lim n→∞ Λn n→∞ Λn
(a.s., P).
In other words, everything comes down to proving that (1.5.13) implies that X1 is P-square integrable. (ii) Assume that the Xn ’s are symmetric. For t ∈ (0, ∞), set ˇ nt = Xn 1[0,t] |Xn | − Xn 1(t,∞) |Xn | , X and show that ˇ t, . . . , X ˇt , . . . X 1 n 4
and
X1 , . . . , X n , . . .
I follow Wm. Feller “An extension of the law of the iterated logarithm to variables without variance,” J. Math. Mech., 18 #4, pp. 345–355 (1968), although V. Strassen was the first to prove the result.
Exercises for § 1.5
57
have the same distribution. Conclude first that, for all t ∈ [0, 1), Pn m=1 Xn 1[0,t] |Xn | ≤ σ (a.s., P), lim n→∞ Λn
where σ is the number in (1.5.15), and second that h i EP X12 = lim EP X12 , X1 ≤ t ≤ σ 2 . t%∞
Hint: Use the equation ˇ nt Xn + X , Xn 1[0,t] |Xn | = 2
and apply part (i). (iii) For general {Xn : n ≥ 1}, produce an independent copy {Xn0 : n ≥ 1} (as in the proof of Lemma 1.5.4), and set Yn = Xn − Xn0 . After checking that Pn | m=1 Ym | ≤ 2σ (a.s., P), lim n→∞ Λn conclude first that EP Y12 ≤ 4σ 2 and then (cf. part (i) of Exercise1.4.27) that EP X12 < ∞. Finally, apply (i) to arrive at EP X1 = 0 and (1.5.14).
Exercise 1.5.16. Let {˜ sn : n ≥ 1} be a sequence of real numbers which possess the properties that lim s˜n+1 − s˜n = 0. lim s˜n = 1, lim s˜n = −1, and n→∞
n→∞
n→∞
Show that the set of subsequential limit points of {˜ sn : n ≥ 1} coincides with [−1, 1]. Apply this observation to show that, in order to get the final statement in Theorem 1.5.9, I need only have proved (1.5.10) for the function f (x) = x, x ∈ R. Hint: In proving the last part, use the square integrability of X1 to see that 2 ∞ X Xn ≥ 1 < ∞, P n n=1
and apply the Borel–Cantelli Lemma to conclude that S˜n − S˜n−1 −→ 0 (a.s., P). Exercise 1.5.17. Let {Xn : n ≥ 1} be a sequence of RN -valued, identically distributed random variables on (Ω, F, P) with the property that, for each e ∈ SN −1 = {x ∈ RN : |x| = 1}, e, X1 RN has mean value 0 and variance 1. Set Pn ˜ n | = 1 P-almost surely. ˜ n = Sn , and show that limn→∞ |S Sn = m=1 Xm and S Λn Here are some steps that you might want to follow.
58
1 Sums of Independent Random Variables
(i) Let {ek : k ≥ 1} be a countable, dense subset of SN −1 for which {e1 , . . . , eN } N is orthonormal, and suppose that the sequence {˜sn : n ≥ 1} ⊆ R has the property that limn→∞ ek , ˜sn RN = 1 for each k ≥ 1. Note that |˜sn | ≤ 1 N 2 max1≤k≤N (ek , s˜n RN , and conclude that C ≡ supn≥1 |˜sn | ∈ [1, ∞). S` (ii) Continuing (i), for a given > 0, choose ` ≥ 1 so that SN −1 ⊆ k=1 B ek , C . Show that |˜sn | ≤ max e,˜sn RN + , 1≤k≤`
and conclude first that limn→∞ |˜sn | ≤1 + and then that limn→∞ |˜sn | ≤ 1. At the same time, since |˜sn | ≥ e1 , ˜sn RN , show that limn→∞ |˜sn | ≥ 1. Thus limn→∞ |˜sn | = 1.
(iii) Let {ek : k ≥ 1} be as in (i), and apply Theorem 1.5.9 to show that, for ˜ n (ω) : n ≥ 1} satisfies the condition in (i). P-almost all ω ∈ Ω, the sequence {S ˜ Thus, by (ii), limn→∞ |Sn (ω)| = 1 for P-almost every ω ∈ Ω.
Chapter 2 The Central Limit Theorem
In the preceding chapter I dealt with averages of random variables and showed that, in great generality, those averages converge almost surely or in probability to a constant. At least when all the random variables have the same distribution and moments of all orders, one way of rationalizing this phenomenon is to recognize that the mean value is conserved whereas all higher moments are driven to 0 when one averages. Of course, the reason why it is easy to conserve the first moment is that the mean of the sum is the sum of the means. Thus, if one is going to attempt to find a simple normalization procedure that conserves a quantity involving more than the mean value, one should seek a quantity that shares this additivity property. With this in mind, one is led to ask what happens if one normalizes in a way that conserves the variance. For this purpose, suppose that {Xn : n ∈ Z+ } is a sequence of mutually independent, identicallyP distributed random variables with 1 n mean value 0 and variance 1, and set Sn = 1 Xk . Then S˘n ≡ n− 2 Sn again has mean value 0 and variance 1. On the other hand, because of Theorem 1.5.9, we know that, with probability 1, limn→∞ S˘n = ∞ = − limn→∞ S˘n . Hence, from the point of view of either almost sure convergence or even convergence in probability, there is no hope that the S˘n ’s will converge. Nonetheless, the random variables {S˘n : n ≥ 1} possess remarkable stability when viewed from a distributional perspective. Indeed, if the Xn ’s are Gaussian, then so are the S˘n ’s, and therefore S˘n ∈ N (0, 1) for all n ≥ 1. More generally, even if the Xn ’s are not Gaussian, fixing their mean value and variance in this way forces all their moments to stabilize. To be precise, assume that X1 has finite moments of all orders, that its mean is 0, and that its variance is 1. Trivially, L1 ≡ limn→∞ EP [S˘n ] = 0 and L2 ≡ limn→∞ EP [S˘n2 ] = 1. Next, assume that L` ≡ limn→∞ EP [S˘n` ] exists for 1 ≤ ` ≤ m, where m ≥ 2. I will show now that Lm+1 ≡ limn→∞ EP [S˘nm+1 ] exists and is equal to mLm−1 . To this end, first note that, since EP [Xn ] = 0 and the Xn ’s are independent and identically distributed, m X m+1 m m P j+1 P m−j P E Sn = nE Xn Xn + Sn−1 =n E Xn E Sn−1 j j=0 P
59
60
2 The Central Limit Theorem m X m−1 m P j+1 P m−j = nmE Sn−1 + n E Xn E Sn−1 . j j=2 P
m+1
Thus, after dividing through by n 2 , one gets the desired conclusion when n → ∞. Starting from L1 = 0 and L2 = 1, one now can use induction to check Qm + that L2m−1 = 0 and L2m = `=1 (2` − 1) = 2(2m)! m m! for all m ∈ Z . That is,
lim EP S˘n2m−1 = 0
n→∞
and
m Y (2m)! lim EP S˘n2m = (2` − 1) = m , n→∞ 2 m! `=1
for all m ∈ Z+ . In other words, at least when the Xn ’s have moments of all orders, limn→∞ EP S˘nm exists and is independent of the particular choice of random variables. In particular, since for the Gaussian case, EP [S˘nm ] = EP [X1m ], we conclude that all moments of the S˘n ’s converge to the corresponding moments of a standard normal random variable. In this chapter we will see that the preceding stabilization result is just one manifestation of a general principle known as the Central Limit phenomenon. § 2.1 The Basic Central Limit Theorem In this section I will derive the basic Central Limit Theorem using a beautiful argument which was introduced by J. Lindeberg. Throughout, hϕ, µi denotes the integral of a function ϕ against a measure µ. § 2.1.1. Lindeberg’s Theorem. Let {Xn : n ≥ 1} be a sequence of independent,Psquare integrable random variables with mean value 0, and set 1 n S˘n = n− 2 m=1 Xm . At least when the Xn ’s are identically distributed and have moments of all orders and variance 1, we just saw that (recall that γm,σ2 is the distribution of an N (m, σ 2 )-random variable) (2.1.1) lim EP ϕ(S˘n ) = hϕ, γ0,1 i n→∞
for any polynomial ϕ : R −→ C. In this subsection, I will prove a result that shows that, under much more general conditions, (2.1.1) holds for all ϕ ∈ C 3 R; C) with bounded second and third order derivatives. In the following statement, sX p p Sn 2 , . σm and S˘n ≡ (2.1.2) σm = Var(Xm ) > 0, Σn = Var(Sn ) = Σn m=1
Notice that when the Xk ’s are identically distributed and have variance 1, the S˘n in (2.1.2) is consistent with the notation used above. Finally, set (2.1.3)
σm 1≤m≤n Σn
rn = max
and gn () =
n i 1 X P h 2 E X , X ≥ Σ m n m Σ2n m=1
§ 2.1 The Basic Central Limit Theorem
61 1
for > 0. Clearly, in the identically distributed case, rn = n− 2 and i h 1 gn () = σ1−2 EP X12 , |X1 | ≥ n 2 σ1 −→ 0 as n → ∞ for each > 0.
Theorem 2.1.4 (Lindeberg). Refer to the preceding, and let ϕ be an element of C 3 (R; R) with bounded second and third order derivatives. Then, for each > 0,
rn P
ϕ000 + gn () ϕ00 . + (2.1.5) E ϕ S˘n − hϕ, γ0,1 i ≤ u u 2 6
In particular, because rn2 ≤ 2 + gn (),
(2.1.6)
> 0,
(2.1.1) holds if gn () −→ 0 as n → ∞ for each > 0. Proof: Choose N (0, 1)-random variables Y1 , . . . , Yn which are both mutually independent and independent of the Xm ’s. (After augmenting the probability space, if necessary, this can be done as an application of either Theorem 1.1.7 or Exercise 1.1.14.) Next, set σk Yk Y˘k = Σn
and T˘n =
n X
Y˘k ,
1
and observe that T˘n is again an N (0, 1)-random variable and therefore that ∆ ≡ EP ϕ(S˘n ) − hϕ, γ0,1 i = EP ϕ(S˘n ) − EP ϕ(T˘n ) . ˘k = Further, set X
Xk Σn ,
Um =
and define X
1≤k≤m−1
Y˘k +
X
˘k X
for 1 ≤ m ≤ n,
m+1≤k≤n
where a sum over the empty set is taken to be 0. It is then clear that ∆≤
n X
∆m
˘ m − EP ϕ Um + Y˘m . where ∆m ≡ EP ϕ Um + X
1
Moreover, if Rm (ξ) ≡ ϕ Um + ξ − ϕ(Um ) − ξϕ0 (Um ) −
ξ 2 00 2 ϕ (Um ),
ξ ∈ R,
62
2 The Central Limit Theorem
˘ m and Y˘m are independent of Um and have the same first then (because both X two moments) ˘ m ) − EP Rm (Y˘m )] ≤ EP Rm (X ˘ m ) + EP Rm (Y˘m ) . ∆m = EP Rm (X In order to complete the derivation of (2.1.5), note that, by Taylor’s Theorem,
3 Rm (ξ) ≤ ϕ000 |ξ| ∧ ϕ00 |ξ|2 ; u u 6
and therefore, for each > 0, n X ˘ m )| EP |Rm (X 1
≤
n n X 2 kϕ000 ku X P ˘ 3 ˘ , |Xm | ≥ Σn E |Xm | , |Xm | ≤ Σn + kϕ00 ku EP X m 6 1 1
n 2 kϕ000 ku kϕ000 ku X σm 00 + kϕ00 ku gn (), + kϕ k g () = ≤ u n 2 6 Σ 6 n 1
while n X 1
3 n 3 X kϕ000 ku P 3 4 rn kϕ000 ku σm . ≤ E |Y1 |3 EP |Rm (Y˘n )| ≤ 6 Σ3n 6 1
Hence, (2.1.5) is now proved. Given (2.1.5), all that remains is to prove (2.1.6). However, for any 1 ≤ m ≤ n and > 0, 2 2 2 σm = EP Xm , |Xm | < Σn + EP Xm , |Xm | ≥ Σn ≤ Σ2n 2 + gn () . The condition that gn () −→ 0 for each > 0 is often called Lindeberg’s condition because it introduced by J. Lindeberg and it was he who proved that it is a sufficient condition for (2.1.1) to hold for all (cf. Theorem 2.1.8) ϕ ∈ Cb (RN ; C). Later, Feller proved that (2.1.1) for all ϕ ∈ Cb (RN ; R) plus rn → 0 imply that Lindeberg’s condition holds. Together, these two results are known as the Lindeberg–Feller Theorem. See Exercise 2.3.20 for a proof of Feller’s part. § 2.1.2. The Central Limit Theorem. If one is not concerned about rates of convergence, then the differentiability requirement on ϕ can be dropped from the last part of Theorem 2.1.4. In order to understand the reason for this, it is helpful to couch the statement of Theorem 2.1.4 entirely in terms of measures. Thus, let µn denote the distribution of S˘n . Then, under Lindeberg’s condition, Theorem 2.1.4 allows one to say that hϕ, µn i −→ hϕ, γ0,1 i for all ϕ ∈ C 3 (RN ; C) with bounded second and third order derivatives. Because we are dealing with statements about integration and integration is a very forgiving operation, this sort of result self-improves. To be precise, I prove the following lemma.
§ 2.1 The Basic Central Limit Theorem
63
Lemma 2.1.7. Suppose that {µn : n ≥ 1} is a sequence of (non-negative) locally finite1 Borel measures on RN and that µ is a locally finite Borel mea∞ N sure on RN with the property that hϕ, µn i −→ hϕ, µi for all ϕ ∈ Cc (R ; R). N Then, for any ψ ∈ C R ; [0, ∞) , hψ, µi ≤ limn→∞ hψ, µn i. Moreover, if ψ ∈ C RN ; [0, ∞) is µn -integrable for each n ∈ Z+ and if hψ, µn i −→ hψ, µi ∈ [0, ∞), then for any sequence {ϕn : n ≥ 1} ⊆ C(RN ; C) that converges uniformly on compacts to a ϕ ∈ C(RN ; C) and satisfies |ϕn | ≤ Cψ for some C < ∞ and all n ≥ 1, hϕn , µn i −→ hϕ, µi. Proof: Choose ρ ∈ Cc∞ B(0, 1); [0, ∞) with total integral 1, and set ρ (x) = −N ρ(−1 x) for > 0. Also, choose η ∈ Cc∞ B(0, 2); [0, 1] so that η = 1 on B(0, 1), and set ηR (x) = η(R−1 x) for R > 0. Begin by noting that hϕ, µn i −→ hϕ, µi for all ϕ ∈ Cc∞ (RN ; C). Next, suppose that ϕ ∈ Cc (RN ; C), and, for > 0, set ϕ = ρ ? ϕ, the convolution Z ρ (x − y)ϕ(y) dy RN
of ρ with ϕ. Then, for each > 0, ϕ ∈ Cc∞ (RN ; C) and therefore hϕ , µn i −→ hϕ , µi. In addition, there is an R > 0 such that supp(ϕ ) ⊆ B(0, R) for all ∈ (0, 1]. Hence, lim hϕ, µn i − hϕ, µi ≤ 2hηR , µikϕ − ϕku . n→∞
Since lim&0 kϕ − ϕku = 0, we have now shown that hϕ, µn i −→ hϕ, µi for all ϕ ∈ Cc (RN ; C). Now suppose that ψ ∈ C RN ; [0, ∞) , and set ψR = ηR ψ, where ηR is as above. Then, for each R > 0, hψR , µi = limn→∞ hψR , µn i ≤ limn→∞ hψ, µn i. Hence, by Fatou’s Lemma, hψ, µi ≤ limR→∞ hψR , µi ≤ limn→∞ hψ, µn i. Finally, suppose that ψ ∈ C RN ; [0, ∞) is µn -integrable for each n ∈ Z+ and that hψ, µn i −→ hψ, µi ∈ [0, ∞). Given {ϕn : n ≥ 1} ⊆ C(RN ; C) satisfying |ϕn | ≤ Cψ and converging uniformly on compacts to ϕ, one has hϕn , µn i − hϕ, µi ≤ hϕn − ϕ, µn i + hϕ, µn i − hϕ, µi .
Moreover, for each R > 0, lim hϕn − ϕ, µn i n→∞
≤ lim
sup
n→∞ x∈B(0,2R)
|ϕn (x) − ϕ(x)|hηR , µn i + lim h(1 − ηR )(ϕn − ϕ), µn i n→∞
≤ 2C lim h(1 − ηR )ψ, µn i = lim 2C hψ, µn i − hηR ψ, µn i = 2Ch(1 − ηR )ψ, µi, n→∞
1
n→∞
A Borel measure on a topological space is locally finite if it gives finite measure to compacts.
64
2 The Central Limit Theorem
and similarly lim hϕ, µn i − hϕ, µi n→∞ ≤ lim hηR ϕ, µn i − hηR ϕ, µi + C lim h(1 − ηR )ψ, µn i + Ch(1 − ηR )ψ, µi n→∞
n→∞
= 2Ch(1 − ηR )ψ, µi. Finally, because ψ is µ-integrable, h(1 − ηR )ψ, µi −→ 0 as R → ∞ by Lebesgue’s Dominated Convergence Theorem, and so we are done. By combining Theorem 2.1.4 with the preceding, we have the following version of the famous Central Limit Theorem. Theorem 2.1.8 (Central Limit Theorem). With the setting the same as it was in Theorem 2.1.4, assume that gn () −→ 0 as n → ∞ for each > 0. Then lim EP ϕn (S˘n ) = hϕ, γ0,1 i n→∞
whenever {ϕn : n ≥ 1} ⊆ C(R; C) satisfies ϕn (y) <∞ sup sup 2 n≥1 y∈R 1 + |y|
and tends to ϕ uniformly on compacts. Moreover, if −∞ ≤ a < b ≤ ∞, then 2 Z b y 1 ˘ dy. exp − (2.1.9) lim P a ≤ Sn ≤ b = γ0,1 (a, b] = √ n→∞ 2 2π a
(See Exercise 2.1.10 for more information about the identically distributed case.) Proof: Take µn to be the distribution of S˘n . By Theorem 2.1.4, we know that hϕ, µn i −→ hϕ, γ0,1 i for all ϕ ∈ Cc∞ (RN ; R). In addition, we know that hψ, µn i = 2 = hψ, γ0,1 i when ψ(y) = 1 + y 2 . Hence, the first assertion is an application of Lemma 2.1.7. Turning to the second assertion, let a < b be given. To prove (2.1.9), choose {ϕk : k ≥ 1} ⊆ Cb (R; R) and {ψk : k ≥ 1} ⊆ Cb (R; R) so that 0 ≤ ϕk % 1(a,b) and 1 ≥ ψk & 1[a,b] as k → ∞. Then, Z i P ˘ ˘ ϕk (y) γ0,1 (dy) −→ γ0,1 (a, b) lim P a < Sn < b ≥ lim E ϕk Sn = n→∞
n→∞
R
as k → ∞, and, similarly,
lim P a ≤ S˘n ≤ b ≤ lim EP ψk S˘n =
n→∞
n→∞
Z R
Finally, note that γ0,1 (a, b) = γ0,1 [a, b] .
ψk (y) γ0,1 (dy) −→ γ0,1 [a, b] .
Exercises for § 2.1
65
Exercises for § 2.1 Exercise 2.1.10. Let {Xn : n ≥ 1} be a sequence of independent, identically 1 Pn distributed random variables, set S˘n = n− 2 m=1 Xm , and assume that
lim EP S˘n2 ∧ R2 ≤ 1
n→∞
for every R ∈ [0, ∞).
In particular, by Lemma 2.1.7, this will certainly be the case whenever (2.1.1) holds for every ϕ ∈ Cc (R; R). The purpose of this exercise is to show that the Xn ’s are P-square integrable, have mean value 0, and variance no more than 1; and the method which I will use is based on the same line of reasoning as was given in Exercise 1.5.12. (i) Assuming that X1 ∈ L2 (P; R), show that EP X1 = 0 and EP X12 ≤ 1. In particular, use this together with the result in part (i) of Exercise 1.4.27 to see that it suffices to handle the case when the Xn ’s are symmetric. (ii) In this and the succeeding parts of this exercise, we will be assuming that the Xn ’s are symmetric. Following the same route as was suggested in (ii) of Exercise 1.5.12, set ˇ t = Xn 1[0,t] |Xn | − Xn 1(t,∞) |Xn | , X n
n ∈ Z+ ,
ˇ t, . . . , X ˇ nt , . . . and X1 , . . . , Xn , . . . have the same distribuand recall that X 1 tion for each t ∈ (0, ∞). Use this together with our basic assumption to see that limR→∞ sup n∈Z+ P An (t, R) = 0, where t∈(0,∞)
) ( n n X X 1 ˇt ≥ n2 R . An (t, R) ≡ Xk ∨ X k 1
1
(iii) Continuing in the setting of part (ii), set n 1 X Xk 1[0,t] |Xk | . S˘nt = 1 n2 1
After noting that the Xn 1[0,t] |Xn | ’s are symmetric, check (cf. the proof of Theorem 1.3.1) that EP |S˘nt |4 ≤ 3t4 . In particular, conclude that, for each t ∈ (0, ∞), there is an R(t) ∈ (0, ∞) such that
1 1 EP |S˘nt |2 , An t, R(t) ≤ 3 2 t2 P An t, R(t) 2 ≤ 1
for all n ∈ Z+ .
66
2 The Central Limit Theorem
(iv) Given t ∈ (0, ∞), choose R(t) ∈ (0, ∞) as in the preceding. Taking into account the identity Pn Pn ˇ t X + k t 1 1 Xk , S˘n = 1 2n 2 show that EP X12 , |X1 | ≤ t = EP |S˘nt |2 ≤ EP |S˘nt |2 , An t, R(t) { + 1 h i ≤ EP S˘n2 ∧ R(t)2 + 1
for all n ∈ Z+ and t ∈ (0, use this and our basic hypothesis ∞). In particular, to conclude first that EP X12 , |X1 | ≤ t ≤ 2 for all t ∈ (0, ∞) and then that X1 is square P-integrable. (v) Show that (2.1.1) holds if and only if X1 has mean value 0 and variance 1. Exercise 2.1.11. An interesting way in which to interpret The Central Limit Theorem is as the solution to a certain fixed point problem. Namely, let P denote the set of probability measures µ on R; BR with the properties that Z Z x2 µ(dx) = 1 and x µ(dx) = 0. R
R
Next, define T2 µ for µ ∈ P to be the probability measure on R, BR given by ZZ x+y √ µ(dx)µ(dy) for Γ ∈ BR . T2 µ(Γ) = 1Γ 2 R2
After checking that T2 maps P into itself, use The Central Limit Theorem to show that, for every µ ∈ P, Z Z n lim ϕ d T2 µ = ϕ dγ0,1 , ϕ ∈ Cb (R; C). n→∞
R
R
Conclude, in particular, that γ0,1 is the one and only element µ of P with the property that T2 µ = µ and that this fixed point is attracting. (See Exercise 2.3.21 for more information.) Exercise 2.1.12. Here is another indication of the remarkable stability of normal random variables. Namely, I will outline here a derivation2 of the L´ evy– Cram´ er Theorem which says that if X and Y are independent random variables whose sum is normal (with some mean and variance), then both X and Y are normal. 2
This derivation is based on a note by Z. Sasv´ ari, who himself borrowed some of the ideas from A. R´ enyi. I know of no derivation that does not rely on complex analysis and would be very interested in learning one.
Exercises for § 2.1
67
(i) Assume that X + Y ∈ N (a, σ 2 ), and, by subtracting a from X, reduce to the case in which X + Y ∈ N (0, σ 2 ). Next, show that there is nothing more to do when σ = 0 and that one can always reduce to the case σ = 1 when σ > 0. Thus, from now on, assume that X + Y ∈ N (0, 1). (ii) Choose r ∈ (0, ∞) so that P |X| ∨ |Y | ≥ r ≤ 12 , and conclude that
R2 , P |X| ≥ r + R ∨ P |Y | ≥ r + R ≤ 4 exp − 2
R ∈ (0, ∞).
In particular, show that the moment generating functions z ∈ C 7−→ M (z) = EP ezX ∈ C and z ∈ C 7−→ N (z) = EhP eizY ∈ C exist and are entire functions. 2
Further, note that M (z)N (z) = exp z2 , and conclude that M and N never vanish. Finally, from the fact that X + Y has mean 0, show that one can reduce to the case in which both X and Y have mean 0. Thus, from now on, we assume that M 0 (0) = 0 = N 0 (0).
(iii) Because M never vanishes and M (0) = 1, elementary complex analysis (cf. Lemma 3.2.3) guarantees that there is a unique entire function θ : C −→ C such that θ(0) = 0 and M (z) = eθ(z) for all z ∈ C. Further, from M 0 (0) = 0, note that θ0 (0) = 0. Thus, θ(z) =
∞ X
cn z n
where n!cn =
n=2
Finally, note that N (z) = exp
h
z2 2
xX dn P log E e ∈ R. dxn x=0
i − θ(z) .
(iv) As an application of H¨older’s Inequality, observe that x ∈ R 7−→ θ(x) ∈ R 2 and x ∈ R 7−→ x2 − θ(x) ∈ R are both convex. Thus, since θ0 (0) = 0, both these functions are non-increasing on (−∞, 0] and non-decreasing on [0, ∞). Use this observation to check that
θ(x) ≥ 0 ≤
x2 − θ(x) 2
for all x ∈ R.
Next, use the preceding in conjunction with the trivial remarks exp Re θ(z) = EP ezX ≤ eθ(x) and
h exp Re
z2 2
− θ(z)
i
i h 2 = EP ezY ≤ exp x2 − θ(x)
to arrive at −y 2 ≤ 2Re θ(z) ≤ x2
for z = x +
√
−1 y ∈ C.
68
2 The Central Limit Theorem
In particular, this means that |z|2 , Re θ(z) ≤ 2
z ∈ C.
(v) To complete the program, use Cauchy’s integral formula to show that, for each n ∈ Z+ and r > 0, on the one hand, Z 2π √ √ 1 θ re −1 θ e− −1 nθ dθ, r > 0, cn r n = 2π 0
while, on the other hand (since θ(z) = θ z¯) and therefore ∂z θ(z) = 0),
Z
2π
√
θ re
0=
−1 θ
e−
√
−1 nθ
dθ.
0
Hence, 1 cn r = π n
Z
2π
√ √ e− −1 nθ dθ, Re θ re −1 θ
n ∈ Z+ and r > 0.
0
Finally, in combination with the estimate obtained in (iv) and the fact that c0 = c1 = 0, this leads to the conclusion that cn = 0 for n 6= 2 and therefore that θ(z) = c2 z 2 with 0 ≤ c2 ≤ 12 .
Exercise 2.1.13. An important result that is closely related to The Central Limit Theorem is the following observation, which occupies a central position in the development of classical statistical mechanics.3 (i) For each n ∈ Z+ , let λn denote the normalized surface measure on the (n − 1)-dimensional sphere √ 1 Sn−1 n = x ∈ Rn : |x| = n 2 , (1)
and denote by λn the distribution of the coordinate x1 under λn . Check that, (1) when n ≥ 2, λn (dt) = fn (t) dt, where fn (t) = 3
ωn−2 1 2
n ωn−1
t2 1− n
n−3 2
1 1(−1,1) n− 2 t ,
Although E. Borel seems to have thought he was the first to discover this result and rhap´ sodizes about it a good deal in “Sur les principes de la cin´ etique des gaz,” Ann. l’Ecole Norm. ¨ sup., 3e t. 23, it appears already in the 1866 article “Uber die Entwicklungen einer Funktion von beliebig vielen Variabeln nach Laplaceshen Funktionen h¨ oherer Ordnung,” J. Reine u. Angewandte Math., by F. Mehler and is only a small part of what Mehler discovered there. Be that as it may, Borel deserves credit for recognizing the significance of this result for statistical mechanics.
Exercises for § 2.1
69
and ωk−1 denotes the surface area of the (k − 1)-dimensional unit sphere in Rk . Using polar coordinates to compute the right-hand side of Z |x|2 k e− 2 dx, (2π) 2 = Rk
first check that
k
ωk−1 =
2π 2 , Γ k2
where Γ(t) is Euler’s Γ-function (cf. (1.3.20)), and then apply Stirling’s formula (cf. (1.3.21)) to see that ωn−2 1 2
n ωn−1
1 −→ √ 2π
as
n → ∞.
Now, using g to denote the density for the standard Gauss distribution (i.e., the Gauss kernel in (1.3.5)), apply these computations to show that sup sup n≥3 t∈R
fn (t) < ∞ and that g(t)
fn (t) −→ 1 uniformly on compacts. g(t)
In particular, conclude that, for any ϕ ∈ L1 (γ0,1 ; R), Z Z (1) (2.1.14) ϕ dλn −→ ϕ dγ0,1 . R
R
(ii) A less computational approach to the same calculation is the following. Let {Xn : n variables, and set p≥ 1} be a sequence of independent N (0, 1) random 2 2 Rn = X1 + · · · + Xn . First note that P Rn = 0 = 0 and then that the distribution of 1 n 2 X1 , . . . , Xn θn ≡ Rn R2
is λn . Next, use The Strong Law of Large Numbers to see that nn −→ 1 (a.s., P) and conclude that, for any N ∈ Z+ , lim EP ϕ θn(N ) = EP ϕ X1 , . . . , XN , ϕ ∈ Cc RN ; R , n→∞
(N )
where, for n ≥ N , θn ∈ RN denotes the projection of θn ∈ Rn onto its first (N ) N coordinates. Conclude that if λn on RN , BRN denotes the distribution of x = (x1 , . . . , xn ) ∈ Rn 7−→ x(N ) ≡ x1 , . . . , xN ∈ RN under λn , then Z Z (N ) N lim ϕ dλn = ϕ dγ0,1 for all ϕ ∈ Cb RN ; C . n→∞
RN
RN
70
2 The Central Limit Theorem
(iii) By considering the case when N = 2, show that, for any ϕ ∈ Cb (R; R), Z (2.1.15)
lim
n→∞ √ Sn−1 ( n)
n
1X ϕ xk − n
!2
Z
k=1
ϕ dγ0,1
λn (dx) = 0.
R
Notice that the non-computational argument has the advantage that it immedi(N ) ately generalizes the earlier result to cover λn for all N ∈ Z+ , not just N = 1 (cf. Exercise 2.3.24). On the other hand, the conclusion is weaker in the sense that convergence of the densities has been replaced by convergence of integrals with bounded continuous integrands and that no estimate on the rate of convergence is provided. More work is required to restore the stronger statements when N ≥ 2. When couched in terms of statistical mechanics, this result can be interpreted as a derivation of the Maxwell distribution of velocities for an ideal gas of free particles of mass 2 and having average energy 1. Exercise 2.1.16. The most frequently encountered applications of Stirling’s formula (cf. (1.3.21)) are to cases when t ∈ Z+ . That is, one is usually interested in the formula n n √ . (2.1.17) n! ∼ 2πn e
Here is a derivation of (2.1.17) as an application of The Central Limit Theorem. Namely, take {X n : n ≥ 1} to be a sequence of independent, random variables with P Xn > x = exp −(x + 1)+ , x ∈ R for all n ∈ Z+ . For n ≥ 1, note that √
−1 Z n+n 1 1 1+4 ˘ xn e−x dx P Sn+1 ∈ 0, 4 = n! 1+n Z − 12 1 √ −1 1 n −√n y nn+ 2 e−n n + 4 1+n − 12 dy. y e 1 + n = 1 n! n− 2
By The Central Limit Theorem,
Z 14 1 x2 1 ˘ e− 2 dx. P Sn ∈ 0, 4 −→ √ 2π 0
At the same time, an elementary computation shows that 1
Z
n− 2 + 14
1 n− 2
√
1+n−1
1+n
− 12
y
n
e
√ − ny
1 4
Z dy −→ 0
e−
x2 2
dx,
§ 2.2 The Berry–Esseen Theorem via Stein’s Method
71
and clearly (2.1.17) follows from these. In fact, if one applies the Berry–Esseen estimate proved in the next section, one finds that √
2πn n!
n n e
1 = 1 + O n− 2 .
However, this last observation is not very interesting since we saw in Exercise 1.3.19 that the true correction term is of order n−1 .4 § 2.2 The Berry–Esseen Theorem via Stein’s Method As we will see in the next section, the principles underlying the passage from Theorem 2.1.4 to Theorem 2.1.8 are very general. In fact, as we will see in Chapter 9, some of these principles can be formulated in such a way that they extend to a very abstract setting. However, rather than delve into such extensions here, I will devote this section to a closer examination of the situation at hand. Specifically, in this section we are going to see how to make the final part of Theorem 2.1.8 quantitative. From (2.1.5), we get a rate of convergence in terms of the second and third derivatives of ϕ. In fact, if we assume that (2.2.1)
1 τk ≡ EP |Xk |3 3 < ∞,
1 ≤ k ≤ n,
then (cf. the proof of Theorem 2.1.4), by using the estimates 000 3 Rm (ξ) ≤ kϕ ku |ξ| 6
and σk ≤ τk ,
one sees that (2.1.5) can be replaced by (2.2.2)
Pn 3 Z 000 P E ϕ S˘n − ϕ dγ0,1 ≤ 2kϕ ku 1 τk Σ3n 3 R
when the Xk ’s have third moments. Although both (2.1.5) and (2.2.2) are interesting, neither one of them can be used to get very much information about the rate at which the distribution functions (2.2.3) 4
x ∈ R 7−→ Fn (x) ≡ P S˘n ≤ x ∈ [0, 1]
As this exercise demonstrates, Stirling’s formula is intimately connected to The Central √ Limit Theorem. In fact, apart from the constant 2π, what we now call Stirling’s formula was discovered first by DeMoivre while he was proving The Central Limit Theorem for Bernoulli random variables. For more information, see, for example, Wm. Feller’s discussion of Stirling’s formula in his Introduction to Probability Theory and Its Applications, Vol. I, Wiley, Series in Probability and Math. Stat. (1968).
72
2 The Central Limit Theorem
are tending to the error function (2.2.4)
1 G(x) ≡ γ0,1 (−∞, x] = √ 2π
Z
x
t2
e− 2 dt.
−∞
To see how (2.1.5) and (2.2.2) must be modified in order to gain such information, first observe that Z ϕ0 (x) Fn (x) − G(x) dx R Z (2.2.5) P ˘ = E ϕ(Sn ) − ϕ(y) γ0,1 (dy), ϕ ∈ Cb1 (R; R . R
(To prove (2.2.5), reduce to the case in which ϕ ∈ Cc1 (R; R) and ϕ(0) = 0; and for this case apply either Fubini’s Theorem or integration by parts over the intervals (−∞, 0] and [0, ∞) separately.) Hence, in order to get information about the distance between Fn and G, we will have to learn how to replace the right-hand sides of (2.1.5) and (2.2.2) with expressions that depend only on the first derivative of ϕ. For example, if the dependence is on kϕ0 ku , then we get information about the L1 (R; R) distance between Fn and G, whereas if the dependence is on kϕ0 kL1 (R;R) , then the information will be about the uniform distance between Fn and G. § 2.2.1. L1 -Berry–Esseen. The basic idea that I will use to get estimates in terms of ϕ0 was introduced by C. Stein and is an example of a procedure known as Stein’s method.1 In the case at hand, his method stems from the trivial observation that if µ is a Borel probability measure on R and g is the Gauss kernel in (1.3.5), then µ = γ0,1 if and only if ∂ µg = 0 in the sense of Schwartz
distribution theory. Equivalently, if A+ is the raising operator (cf. 2.4.1) given D §E µ by A+ ϕ(x) = xϕ(x) − ∂ϕ(x), then, because hA+ ϕ, µi = ϕg, ∂ g , µ = γ0,1 if and only if hA+ ϕ, µi = 0 for sufficiently many test functions ϕ. In fact, as will be shown in what follows, µ will be close to γ0,1 if, in an appropriate sense, hA+ ϕ, µi is small. To make mathematics out of the preceding, I will need the following.
Lemma 2.2.6. Let ϕ ∈ C 1 (R; R), assume that kϕ0 ku < ∞, set ϕ˜ = ϕ−hϕ, γ0,1 i, and define Z x 2 x2 − t2 2 dt ∈ R. ϕ(t)e ˜ (2.2.7) x ∈ R 7−→ f (x) ≡ e −∞
Then f ∈ Cb2 (R; R), (2.2.8) 1
kf ku ≤ 2kϕ0 ku ,
q kf 0 ku ≤ 3 π2 kϕ0 ku ,
kf 00 ku ≤ 6kϕ0 ku ,
Stein provided an introduction, by way of examples, to his own method in Approximate Computation of Expectations, I.M.S., Lec. Notes & Monograph Series # 7 (1986).
§ 2.2 The Berry–Esseen Theorem via Stein’s Method
73
and f 0 (x) − xf (x) = ϕ(x), ˜
(2.2.9)
x ∈ R.
Proof: The facts that f ∈ C 1 (R; R) and that (2.2.9) holds are elementary applications of The Fundamental Theorem of Calculus. Moreover, knowing that f ∈ C 1 (R; R) and using (2.2.9), we see that f ∈ C 2 (R; R) and, in fact, that f 00 (x) − xf 0 (x) = f (x) + ϕ0 (x),
(2.2.10)
x ∈ R.
To prove the estimates in (2.2.8), first note that, because ϕ˜ and therefore f are unchanged when ϕ is replaced by ϕ − ϕ(0), I may and will assume that ϕ(0) = 0 and therefore that |ϕ(t)| ≤ kϕ0 ku |t|. In particular, this means that Z Z q ϕ dγ0,1 ≤ kϕ0 ku |t| γ0,1 (dt) = kϕ0 ku 2 . π R
R
t2
− 2 dt = 0, an alternative expression for f ϕ(t)e ˜ Z ∞ 2 x2 − t2 dt, x ∈ R. ϕ(t)e ˜ f (x) = −e 2
Next, observe that, because is
R
R
x
Thus, by using the original expression for f (x) when x ∈ (−∞, 0) and the alternative one when x ∈ [0, ∞), we see first that Z ∞ 2 x2 ϕ˜ −t sgn(x) e− t2 dt, x ∈ R, 2 |f (x)| ≤ e |x|
and then that |f (x)| ≤ kϕ0 ku e
x2 2
Z
∞
t+
|x|
q 2 π
t2
e− 2 dt.
But, since Z ∞ Z ∞ t2 x2 t2 x2 d − t e− 2 dt − 1 = 0 e 2 dt ≤ e 2 e2 dx x x
we have that, for x ∈ R, Z Z ∞ x2 t2 x2 e− 2 dt ≤ e 2 (2.2.11) |x|e 2 |x|
∞
t2
te− 2 dt = 1 and e
for x ∈ [0, ∞),
x2 2
|x|
Z
∞
|x|
t2
e− 2 dt ≤
q
π 2;
which means that I have now proved the first estimate in (2.2.8). To prove the other two estimates there, derive from (2.2.10)
x2 d − x2 0 e 2 f (x) = e− 2 f (x) + ϕ0 (x) dx
74
2 The Central Limit Theorem
and therefore that Z Z x − t2 x2 x2 0 0 f (t)+ϕ (t) e 2 dt = −e 2 f (x) = e 2 −∞
∞
t2 f (t)+ϕ0 (t) e− 2 dt,
x ∈ R.
x
Thus, reasoning as I did above and using the first estimate in (2.2.8) and the relations in (2.2.9), (2.2.10), and (2.2.11), one arrives at the second and third estimates in (2.2.8). I now have the ingredients needed to apply Stein’s method to the following example of a Berry–Esseen type of estimate. Theorem 2.2.12 (L1 -Berry–Esseen Estimate). Continuing in the setting of Theorem 2.1.4, one has that for all > 0 (cf. (2.1.3), (2.2.3), and (2.2.4)) (2.2.13)
√
Fn − G 1 2π gn (2). ≤ 6(r + ) + 3 n L (R;R)
Moreover, if (cf. (2.2.1)) τm < ∞ for each 1 ≤ m ≤ n, then (2.2.14)
Fn − G 1 ≤ L (R;R)
6rn +
3
Pn
3 m=1 τm Σ3n
Pn 3 9 m=1 τm . ∧ Σ3n
2 In particular, if σm = 1 and τm ≤ τ < ∞ for each 1 ≤ m ≤ n, then
8τ 3 6 + 2τ 3
Fn − G 1 √ . √ ≤ ≤ L (R;R) n n
Proof: Let ϕ ∈ C 1 (R; R) having bounded first derivative be given, and define f accordingly, as in (2.2.7). Everything turns on the equality in (2.2.9). Indeed, because of that equality, we know that the right-hand side of (2.2.5) is equal to n X 2 P ˘ m f (S˘n ) , EP f 0 (S˘n ) − EP S˘n f (S˘n ) = σ ˘m E f 0 (S˘n ) − EP X m=1
where I have set σ ˘m =
σm Σn
˘m = and X
Xm Σn .
Next, define
˘m T˘n,m (t) = S˘n + (t − 1)X
for t ∈ [0, 1],
˘ m , and conclude that note that T˘n,m (0) is independent of X ˘ m f (S˘n ) = EP X
Z
1
2 0 ˘ m f T˘n,m (t) dt EP X
0
2 P =σ ˘m E f 0 T˘n,m (0) +
Z 0
1
2 0 ˘ m f (T˘n,m (t) − f 0 T˘n,m (0) dt EP X
§ 2.2 The Berry–Esseen Theorem via Stein’s Method for each 1 ≤ m ≤ n. Hence, we now see that Z n n Z X X 2 (2.2.15) EP ϕ(S˘n ) − ϕ dγ0,1 = σ ˘m Am − R
where
m=1
m=1
75
1
Bm (t) dt,
0
h i Am ≡ EP f 0 S˘n ) − f 0 T˘n,m (0)
and
h i 2 ˘m Bm (t) ≡ EP X f 0 (T˘n,m (t) − f 0 T˘n,m (0) .
Obviously, by Taylor’s Theorem and H¨ older’s Inequality, for each 1 ≤ m ≤ n, τm 00 kf 00 ku (*) |Am | ≤ σ ˘m kf ku ≤ rn ∧ Σn
while, for each t ∈ [0, 1] and > 0,
i kf 0 ku h 2 2 Bm (t) ≤ 2t˘ , |Xm | ≥ 2Σn . σm kf 00 ku + 2 2 EP Xm Σn Thus, after summing over 1 ≤ m ≤ n, integrating with respect to t ∈ [0, 1], and using (2.2.5), (2.2.15), and (*), we arrive at Z ϕ0 (x) Fn (x) − G(x) dx ≤ rn + kf 00 ku + 2gn (2)kf 0 ku , R
which, in conjunction with the estimates in (2.2.8), leads immediately to the estimate in (2.2.13). In order to get (2.2.14), simply note that Z 1 h 3 i ˘ m |3 f 00 T˘n,m (st) ds ≤ tkf 00 ku τm , Bm (t) ≤ t EP |X Σ3n 0
and again use (2.2.15), (2.2.8), and (*). § 2.2.2. The Classical Berry–Esseen Theorem. The result in Theorem 2.2.12 is already significant. However, it is not the classical Berry–Esseen Theorem, which is the analogous statement about kFn − Gku . In order to prove the classical result via Stein’s method, we must learn how to replace the kϕ000 ku in Lindeberg’s Theorem by kϕ0 kL1 (R;R) . It turns out that this replacement is far more challenging than replacing kϕ000 ku by kϕ0 ku , which was the replacement needed to prove Theorem 2.2.12. The argument that I will use is a clever induction procedure that was introduced into this context by E. Bolthausen.2 But, before I can apply Bolthausen’s argument, I will need the following variation on Lemma 2.2.6. 2
The Berry–Esseen Theorem appears as a warm-up exercise in Bolthausen’s “An estimate of the remainder term in a combinatorial central limit theorem,” Z. Wahr. Gebiete 66, pp. 379–386 (1984).
76
2 The Central Limit Theorem
Lemma 2.2.16. ϕ ∈ C 1 (R; R), and define f accordingly, as in (2.2.7). p π Let 0 Then kf ku ≤ 8 kϕ kL1 (R;R) and kf 0 ku ≤ kϕ0 kL1 (R;R) .
Proof: I will assume, throughout, that kϕ0 kL1 (R;R) = 1. Observe that, by the Fundamental Theorem of Calculus, (cf. the notation in Lemma 2.2.6) Z ϕ(x) ˜ = − ϕ˜y (x) ϕ0 (y) dy, where ϕy = 1(−∞,y] , R
and so (cf. (2.2.4)) Z f (x) = − ψy (x) ϕ0 (y) dy,
where ψy (x) =
√
2πe
x2 2
G(x∧y)−G(x)G(y) ≥ 0.
R
At the same time, these, together with (2.2.9), give Z 0 f (x) = − xψy (x) + ϕ˜y (x) ϕ0 (y) dy. R
Hence, the desired estimates come down to checking that x2 e 2 G(x ∧ y) − G(x)G(y) ≤ 14 ,
and
√ x2 2πxe 2 G(x ∧ y) − G(x)G(y) + 1(−∞,y] (x) − G(y) ≤ 1
for all (x, y) ∈ R × R. But, if x ≤ y, G(x ∧ y) − G(x)G(y) ≤ G(x) − G(x)2 =
2 1 1 − 4 G(x) − 12 4
and G(x) −
1 2 2
|x|
1 = 2π
Z
1 ≥ 8π
ZZ
2 − ξ2
e
!2 dξ
0
e−
ξ2 +η 2 2
dξdη =
x2 1 1 − e− 2 , 4
ξ 2 +η 2 ≤x2
which proves the first inequality. To get the second one, it suffices to consider each of the four cases 0 ≤ x ≤ y, x ≥ 0 & y < x, y < x < 0, and x < 0 & y ≥ x separately and take into account that, from the first part of (2.2.11), √ √ x2 x2 x ≥ 0 =⇒ 2πxe 2 1−G(x) ≤ 1 and x < 0 =⇒ 2π|x|e 2 G(x) ≤ 1.
As distinguished from Lemma 2.2.6, Lemma 2.2.16 contains no estimate on kf 00 ku . Indeed, there is no such estimate in terms of kϕ0 kL1 (R;R) . As a consequence, the proof of the following is much more involved than that of Theorem 2.2.12
§ 2.2 The Berry–Esseen Theorem via Stein’s Method
77
Theorem 2.2.17 (Classical Berry–Esseen Estimate). Let everything be as in Theorem 2.1.4, and assume that (cf. (2.2.1)) τm < ∞ for each 1 ≤ m ≤ n. Then (cf. (2.2.3) and (2.2.4)) Pn kFn − Gku ≤ 10
(2.2.18)
1
3 τm
Σ3n
.
In particular, if σm = 1 for all 1 ≤ m ≤ n, then (2.2.18) can be replaced by Pn kFn − Gku ≤ 10
(2.2.19)
1
3 τm 3
n2
3 max τm √ . ≤ 10 n 1≤m≤n
Proof: For each n ∈ Z+ , let βn denote the smallest number β with the property that Pn 3 τ kFn − Gku ≤ β 1 3 m Σn
for all choices of random variables satisfying the hypotheses under which (2.2.18) is to be proved. My strategy is to give an inductive proof that βn ≤ 10 for all n ∈ Z+ ; and, because Σ1 ≤ τ1 and therefore β1 ≤ 1, I need only be concerned with n ≥ 2. ˘m, σ Given n ≥ 2 and X1 , . . . , Xn , define X ˘m , and T˘n,m (t) for 1 ≤ m ≤ n and t ∈ [0, 1] as in the proof of Theorem 2.2.12. Next, for each 1 ≤ m ≤ n, set Σn,m =
p
2 , Σ2n − σm
τ˘m =
τm , Σn
ρn =
n X
3 τ˘m ,
and ρn,m =
1
X τ` 3 . Σn,m
1≤`≤n `6=m
Finally, set Sn,m =
X
X`
1≤`≤n `6=m
Sn,m , and S˘n,m = Σn,m
and let x ∈ R 7−→ Fn,m (x) ≡ P S˘n,m ≤ x ∈ [0, 1] denote the distribution function for S˘n,m . Notice that, by definition, kFn,m − Gku ≤ βn−1 ρn,m for each 1 ≤ m ≤ n. Furthermore, because (cf. (2.1.3)) Σ2n,m 2 =1−σ ˘m ≥ 1 − rn2 Σ2n
we know first that ρn,m ≤
and ρn,m ≤
ρn 3
(1 − rn2 ) 2
,
Σn Σn,m
1 ≤ m ≤ n,
3 ρn ,
78
2 The Central Limit Theorem
and therefore that max kFn,m − Gku ≤
(2.2.20)
1≤m≤n
ρn βn−1 3
(1 − rn2 ) 2
.
Now let ϕ ∈ Cb2 (R; R) with kϕ00 kL1 (R) < ∞ be given, define f accordingly, as in (2.2.7), and let {Am : 1 ≤ m ≤ n}
and {Bm (t) : 1 ≤ m ≤ n & t ∈ [0, 1]}
be the associated quantities appearing in (2.2.15). By (2.2.9), we have that h i h i ˘ m f (S˘n ) + EP T˘n,m (0) f (S˘n ) − f T˘n,m (0) |Am | ≤ EP X h i + EP ϕ(S˘n ) − ϕ T˘n,m (0) ˘ m | kf ku + EP |X ˘ m T˘n,m (0)| kf 0 ku ≤ EP |X Z 1 P ˘ m ϕ0 T˘n,m (ξ) dξ E X + 0
Σn,m 0 ˘ m ϕ0 T˘n,m (ξ) kf ku + max EP X kf ku + Σn ξ∈[0,1] ˘ m ϕ0 T˘n,m (ξ) . kf ku + kf 0 ku + max EP X
≤σ ˘m
≤σ ˘m
ξ∈[0,1]
˘ m from T˘m,n (0), one sees that Similarly, from (2.2.9)) and the independence of X |Bm (t)| is dominated by h i h 2 i ˘ 3 f T˘n,m (t) + EP X ˘ T˘n,m (0) f T˘n,m (t) − f T˘n,m (0) t EP X m
m
h i ˘ 2 ϕ T˘n,m (t) − ϕ T˘n,m (0) + EP X m ˘ m |3 kf ku + tEP |X ˘ m |3 EP |T˘n,m (0)| kf 0 ku ≤ tEP |X Z 1 P 3 0 ˘ m ϕ T˘n,m (tξ) dξ E X +t 0
≤
3 t˘ τm
3 0 ˘ m ϕ T˘n,m (ξ) . kf ku + kf 0 ku + t max EP X ξ∈[0,1]
In order to handle the second term in the last line of each of these calculations, introduce the function Σn,m 0 ˘ y ∈ R. (ξ, ω, y) ∈ [0, 1] × Ω × R 7−→ ψ(ξ, ω, y) ≡ ϕ ξ Xm (ω) + Σn
§ 2.2 The Berry–Esseen Theorem via Stein’s Method
79
˘ m is independent of T˘n,m (0), Then, because X h Z Z P k 0 i k ˘ m ϕ T˘n,m (ξ) − ˘ E X Xm (ω) ψ(ξ, ω, y) γ0,1 (dy) P(dω) Ω R Z Z Z k ˘ ≤ Xm (ω) ψ(ξ, ω, y) dFn,m (y) − ψ(ξ, ω, y) dG(y) P(dω) Ω
Z = Ω
≤
R
R
Z ˘ m (ω) k ψ 0 (t, ω, y) G(y) − Fn,m (y) dy P(dω) X R
h k 00 i 1 ˘ m k kϕ00 kL1 (R;R) ≤ τ˘m βn−1 kϕ kL 3(R;R) ρn , EP X (1 − rn2 ) 2 (1 − rn2 ) βn−1 ρn
3 2
k ∈ {1, 3},
where I have used ψ 0 (t, ω, y) to denote the first derivative of y ∈ R 7−→ ψ(ξ, ω, y), applied (2.2.5) and (2.2.20), and noted that kψ 0 (ξ, ω, ·)kL1 (R;R) = kϕ00 kL1 (R;R)
for all (ξ, ω) ∈ [0, 1] × Ω.
At the same time, because kψ(ξ, ω, · )kL1 (R;R) =
Σn kϕ0 kL1 (R;R) Σn,m
for all (ξ, ω) ∈ [0, 1] × Ω,
we have that, for each ξ ∈ [0, 1], Z Z k kϕ0 kL1 (R;R) τ˘m ˘ m (ω)k X ψ(ξ, ω, y) γ (dy) P(dω) ≤ 0,1 1 . Ω R 2π(1 − rn2 ) 2
Hence, by combining these estimates, we arrive at 0 1 kϕ kL (R;R) βn−1 ρn 00 |Am | ≤ τ˘m kf ku + kf 0 ku + 3 kϕ kL1 (R;R) 12 + 2 2 (1 − rn ) 2π(1 − rn2 )
and
0 1 kϕ k βn−1 ρn L (R;R) 00 3 |Bm (t)| ≤ t˘ τm kf ku + kf 0 ku + 3 kϕ kL1 (R;R) 12 + 2 2 2 (1 − rn ) 2π(1 − rn )
for all 1 ≤ m ≤ n and t ∈ [0, 1], and, after putting these together with (2.2.5) and (2.2.15), we conclude that Z ϕ0 (y) G(y) − Fn (y) dy R 3 kf ku + kf 0 ku ≤ (2.2.21) 2 βn−1 kϕ00 kL1 (R;R) ρn kϕ0 kL1 (R;R) ρn . + 3 1 + (1 − rn2 ) 2 2π(1 − r2 ) 2 n
80
2 The Central Limit Theorem
I next apply (2.2.21) to a special class of ϕ’s. Namely, set 1 h(x) = 1 − x 0
if x < 0 if x ∈ [0, 1] if x > 1,
and define h (x) =
−1
Z
η −1 y h(x − y) dy
for > 0 and x ∈ R,
R
R where η ∈ Cc∞ R; [0, ∞) satisfies R η(y) dy = 1. Finally, let a ∈ R be given, and set x ∈ R and , L > 0. ϕ,L (x) = h x−a Lρn ,
It is then an easy matter to check that kϕ0,L kL1 (R;R) = 1 while kϕ00,L kL1 (R;R) ≤ 2 Lρn . Hence, by plugging the estimates from Lemma 2.2.16 into (2.2.21) and then letting & 0, we find that, for each L > 0,
1 Z a+Lρn G(y) − Fn (y) dy sup a∈R Lρn a r 2βn−1 1 π 3 ρn . + 1+ ≤ 3 1 + 8 2 (1 − rn2 ) 2 L 2π(1 − r2 ) 2
(2.2.22)
n
But 1 Lρn
Z
a
1 Fn (y) dy ≤ Fn (a) ≤ Lρn a−Lρn
Z
a+Lρn
Fn (y) dy, a
while 0≤
1 Lρn
Z
a+Lρn
G(y) dy − G(a) = a
1 Lρn
and, similarly, 1 0 ≤ G(a) − Lρn
Z
Z a
a+Lρn
Lρn (a + Lρn − y) γ0,1 (dy) ≤ √ , 8π
a
Lρn G(y) dy ≤ √ . 8π a−Lρn
Thus, from (2.2.22), we first obtain, for each L ∈ (0, ∞), 3 kFn − Gku ≤ + 2
r
L 3βn−1 3 9π ρn , + + 1 3 1 + 32 (1 − rn2 ) 2 L (8π) 2 8π(1 − rn2 ) 2
Exercises for § 2.2
81
and then, after minimizing with respect to L ∈ (0, ∞), r r − 1 9 9π 3 1 − rn2 2 + + kFn − Gku ≤ 8π 32 2 (2.2.23) r 1 3 4 18 2 −4 2 ρn . 1 − rn β + π n−1
In order to complete the proof starting from (2.2.23), we have to consider the 1 1 . Because kFn − Gku ≤ 1, or ρn < 10 two cases determined by whether ρn ≥ 10 it is obvious that we can take βn ≤ 10 in the first case. On the other hand, if 1 and we assume that βn−1 ≤ 10, then, because ρn ≤ 10 n n n X 1 X P 2 32 1 X P 3 3 σ ˘m ≥ rn3 , E Xm = E |Xm | ≥ 3 ρn = 3 Σn 1 Σn 1 1
(2.2.23) says that kFn − Gku ≤ 10ρn . Hence, in either case, βn−1 ≤ 10 =⇒ βn ≤ 10. It is clear from the preceding derivation (in particular, the final step) that the constant 10 appearing in (2.2.18) and (2.2.19) can be replaced by the smallest β > 1 that satisfies the equation r r r 1 2 − 3 1 9 9π 3 4 18 − 23 − 2 β 2 1 − β− 3 4 . + 1−β + β= + π 8π 32 2
Numerical experimentation indicates that 10 is quite a good approximation to the actual solution of this minimization problem. However, it should be recognized that, with sufficient diligence and entirely different techniques, one can show that the 10 in (2.2.18) can be replaced by a number that is less than 1. Thus, I do not claim that Stein’s method gives the best result, only that it gives whatever it gives with relatively little pain. Exercises for § 2.2 Exercise 2.2.24. It is important to know that, at least qualitatively, one cannot do better than Berry–Esseen. To see this, consider independent, symmetric, {−1, 1}-valued Bernoulli random variables, and define Fn accordingly. Next, 1 observe that when tn = −(2n + 1)− 2 , Z 0 x2 1 √ e− 2 dx F2n+1 (tn ) − G(tn ) = 2π tn 1
and therefore that limn→∞ n 2 kFn − Gku ≥ √12π . In particular, since τm = 1 for these Bernoulli random variables, we conclude that the constant in the Berry– 1 Esseen estimate cannot be smaller than (2π)− 2 .
82
2 The Central Limit Theorem
Exercise 2.2.25. Because the derivation of Theorem 2.2.12 is so elegant and simple, one wonders whether (2.2.14) can be used as the starting point for a proof of (2.2.19). Unfortunately, the following na¨ıve idea falls considerably short of the mark. Let X1 , . . . , Xn satisfy the hypotheses of Theorem 2.2.17. Starting from (2.2.14) and proceeding as I did in the passage from (2.2.22) to (2.2.23), show that for every L > 0 Pn 3 6 1 τm L +√ , kFn − Gku ≤ 3 LΣn 8π
and conclude that kFn − Gku ≤
72 π
14 Pn 1
3 τm
Σ3n Pn −3
Obviously, this is unacceptably poor when Σn
1
12
.
3 τm is small.
§ 2.3 Some Extensions of The Central Limit Theorem In most modern treatments of The Central Limit Theorem, Fourier analysis plays a central role. Indeed, the Fourier transform makes the argument so simple that it can mask what is really happening. However, now that we know Lindeberg’s argument, it is time to introduce Fourier techniques and begin to see how they facilitate reasoning involving independent random variables. § 2.3.1. The Fourier Transform. The Fourier transform of finite, C-valued, Borel measure µ on RN is the function µ ˆ : RN −→ C given by Z h√ i (2.3.1) µ ˆ(ξ) = exp −1 ξ, x RN µ(dx) for x ∈ RN . RN
When µ is a probability measure which is the distribution of an RN -valued random variable X, probabilists usual call its Fourier transform the characteristic function of X, and when µ admits a density ϕ with respect to Lebesgue measure λRN , one uses Z h√ i (2.3.2) ϕ(ξ) ˆ = exp −1 ξ, x RN ϕ(x) dx for ξ ∈ RN RN
in place of µ ˆ to denote its Fourier transform. Obviously, µ ˆ is a continuous function that is bounded by the total variation kµkvar of µ; and only slightly less obvious1 is the fact that, for ϕ ∈ Cc∞ RN ; C , ϕˆ ∈ C ∞ RN ; C and that ϕˆ as well as all its derivatives are rapidly decreasing −1 (i.e., they tend to 0 at infinity faster than 1 + |ξ|2 to any power). √ α ϕ(ξ) = (− −1ξ)α ϕ(ξ) ˆ and concludes that One uses integration byP parts to check that ∂d α ϕk |ξ|n |ϕ(ξ)| ˆ is bounded by k∂ . 1 N L (R ) kαk=n 1
§ 2.3 Some Extensions of The Central Limit Theorem
83
Lemma 2.3.3. Let µ be a finite, measure on RN . Then, for C-valued Borel N 1 N 1 N every ϕ ∈ Cb R ; C ∩ L R ; C with ϕˆ ∈ L (R ; C), Z Z 1 ˆ(ξ) dξ. ϕ(ξ) ˆ µ (2.3.4) hϕ, µi = ϕ dµ = (2π)N RN RN
Moreover, given a sequence {µn : n ∈ Z+ } of Borel probability measures and a Borel probability measure µ on RN , µ cn −→ µ ˆ uniformly on compacts if hϕ, µn i −→ hϕ, µi for every ϕ ∈ Cc RN , R . Conversely, if µ cn (ξ) −→ µ ˆ(ξ) pointwise, then hϕn , µn i −→ hϕ, µi whenever {ϕn : n ≥ 1} is a uniformly bounded sequence in Cb (RN ; C) that tends to ϕ uniformly on compacts. (Cf. Theorem 3.1.8 for more refined information on this subject.) R Proof: Choose ρ ∈ Cc∞ RN ; [0, ∞) to be an even function that satisfies RN ρ dx = 1, and set ρ (x) = −N ρ(−1 x) for ∈ (0, ∞). Next, define ψ for ∈ (0, ∞) to be the convolution ρ ? µ of ρ with µ. That is, Z ψ (x) = ρ (x − y) µ(dy) for x ∈ RN . RN
It is then easy to check that ψ ∈ Cb RN ; C and kψ kL1 (RN ;R) ≤ kµkvar for every ˆ ∈ (0, ∞). In addition, one sees Fubini’s µ(ξ). (by Theorem) that ψ (ξ) = ρˆ( ξ)ˆ N 1 N Thus, for any ϕ ∈ Cb (R ; C ∩ L R ; C , Fubini’s Theorem followed by the classical Parseval Identity (cf. Exercise 2.3.23) yields Z Z 1 ρˆ( ξ) ϕ(ξ) ˆ µ ˆ(−ξ) dξ, hϕ , µi = ϕ(x) ψ (x) dx = (2π)N RN RN
where ϕ ≡ ρ ? ϕ is the convolution of ρ with ϕ. Since, as & 0, ϕ −→ ϕ while ρˆ( ξ) −→ 1 boundedly and pointwise, (2.3.4) now follows from Lebesgue’s Dominated Convergence Theorem. Turning to the second part of the theorem, first suppose that hϕ, µn i −→ hϕ, µi for every ϕ ∈ Cc (RN ; R), and let ξn −→ ξ in C. Then, by the last part of Lemma √ √ cn (ξn ) −→ µ ˆ(ξ). 2.1.7 applied to ϕn (x) = e −1 (ξn ,x)RN and ϕ(x) = e −1 (ξ,x)RN , µ Hence, µ cn −→ µ ˆ uniformly on compacts. Conversely, suppose that µ cn −→ µ ˆ pointwise. Again by Lemma 2.1.7, we need only check that hϕ, µ i −→ hϕ, µi n when ϕ ∈ Cc∞ RN ; C . But, for such a ϕ, ϕˆ is smooth and rapidly decreasing, and therefore the result follows immediately from the first part of the present lemma together with Lebesgue’s Dominated Convergence Theorem.
Remark 2.3.5. Although it may seem too obvious to mention, an important, and rather amazing, consequence of Lemma 2.3.3 is that a finite Borel measure on RN is completely determined by its 1-dimensional marginals. To understand this remark, recall that for a linear subspace L of RN , the marginal distribution of µ on L is the measure (ΠL )∗ µ, where ΠL denotes orthogonal projection
84
2 The Central Limit Theorem
onto L. In particular, if e ∈ SN −1 and µe is the marginal distribution of µ on the 1-dimensional subspace spanned by e, then µ ˆ(ξe) = µ ce (ξ). Hence, the Fourier transform of µ is determined by the Fourier transforms of {µe : e ∈ SN −1 }, and therefore, by Lemma 2.3.3, µ can be recovered from its 1-dimensional marginals. Of course, one should be careful when applying this observation. For instance, when applied to an RN -valued random variable X = (X1 , . . . , XN ), it says that the distribution of X can be recovered from a knowledge of the distributions of (e, X)RN for all e ∈ SN −1 , but it does not say that the distributions of the coordinates Xi , 1 ≤ i ≤ N , determine the distribution of X. § 2.3.2. Multidimensional Central Limit Theorem. The great virtue of the Fourier transform is that it behaves so well under operations built out of translation. In applications to probability theory, this virtue is of particular importance when adding independent random variables. Specifically, if X and Y are independent, then the characteristic function of X + Y is the product of the characteristic functions of X and Y. This observation, combined with Lemma 2.3.3 leads to the following easy proof of The Central Limit Theorem for independent, identically distributed, R-valued random variables {Xn : n ≥ 1} with mean value 0 and variance 1. Namely, if µn is the distribution of S˘n , then n n ξ2 ξ2 −→ e− 2 = γd + o n1 = 1− µ ˆn (ξ) = µ ˆ √ξn 0,1 (ξ) 2n
for every ξ ∈ R. Actually, as we are about to see, a slight variation on the preceding will allow us to lift the results that we already know for R-valued random variables to random variables with values in RN . However, before I can state this result, I must introduce the analogs of the mean value and variance for vector-valued random variables. Thus, given an RN -valued random variable X on the probability space (Ω, F, P) with |X| ∈ L1 (P; R), the mean value EP [X] of X is the m ∈ RN that is determined by the property that (ξ, m)RN = EP ξ, X RN for all ξ ∈ RN . Similarly, if |X| is P-square integrable, then the covariance cov(X) of X is the symmetric linear transformation C on RN determined by ξ, C η RN = EP ξ, X − EP [X] RN η, X − EP [X] RN for ξ, η ∈ RN . Notice that cov(X) is not only symmetric but is also non-negative definite, since for each ξ ∈ RN , ξ, cov(X) ξ RN is nothing but the variance of (ξ, X)RN . Finally, given m ∈ RN and a symmetric, non-negative C ∈ RN ⊗ RN , I will use γm,C to denote the Borel probability measure on RN determined by the property that Z Z N 1 (dy), ϕ ∈ Cb (RN ; R), (2.3.6) ϕ dγm,C = ϕ m + C 2 y γ0,1 RN
RN
§ 2.3 Some Extensions of The Central Limit Theorem
85
1
where C 2 is the non-negative definite, symmetric square root of C Clearly, an RN -valued random variable Y has distribution γm,C if and only if, for each ξ ∈ RN , (ξ, Y)RN is a normal random variable with mean value (ξ, m)RN and variance (ξ, C ξ)RN . For this reason, γm,C is called the normal or Gaussian distribution with mean value m and covariance C. For the same reason, a random variable with γm,C as its distribution is called a normal or Gaussian random variable with mean value m and covariance C, or, more briefly, an N (m, C)-random variable. Finally, one can use this characterization to see that h√ i 1 ξ, Cξ . −1 ξ, m) − (2.3.7) γ[ (ξ) = exp m,C 2 RN
In the following statements, I will be assuming that {Xn : n ∈ Z+ } is a sequence of mutually independent, square P-integrable, RN -valued random variables on the probability space (Ω, F, P ). Further, I will assume that, for each n ∈ Z+ , Xn has mean value 0 and strictly positive definite covariance cov(Xn ). Finally, for n ∈ Z+ , set Sn =
n X
Xm ,
Cn ≡ cov(Sn ) =
m=1
n X
cov(Xm ),
m=1
1 ˘ n = Sn . Σn = det(Cn ) 2N and S Σn
˘ n is consistent Notice that when N = 1, the above use of the notation Σn and S with that in § 2.1.1. With these preparations, I am ready to prove the following multidimensional generalization of Theorem 2.1.8. Theorem 2.3.8. Referring to the preceding, assume that the limit A ≡ lim
(2.3.9)
n→∞
Cn Σ2n
exists and that (2.3.10)
n 1 X P E |Xm |2 , |Xm | ≥ Σn = 0 for each > 0. 2 n→∞ Σn m=1
lim
Then, for every sequence {ϕn : n ≥ 1} ⊆ C(RN ; C) that satisfies (2.3.11)
sup sup n≥1
y∈RN
|ϕn (y)| <∞ 1 + |y|2
86
2 The Central Limit Theorem
and converges uniformly on compacts to ϕ, ˘ n = hϕ, γ0,A i. lim EP ϕn S
(2.3.12)
n→∞
In particular, when the Xn ’s are uniformly square P-integrable random variables with mean value 0 and common covariance C,
Sn = hϕ, γ0,C i lim EP ϕn √ n→∞ n whenever {ϕn : n ≥ 1} ⊆ C(RN ; C) satisfies (2.3.11) and converges to ϕ uniformly on compacts. Proof: Given e ∈ SN −1 , set Σn (e) =
q
e, Cn e RN
and ρn (e) =
Σn (e) . Σn
p Then, ρ(e) ≡ inf n≥1 ρn (e) ∈ (0, 1] and ρn (e) −→ (e, Ae)RN as n → ∞. In particular, if (e1 , . . . , eN ) is an orthonormal basis in RN , then N N X X ˘ n |2 = ˘ n 2N = EP |S EP ei , S ρn (ei )2 R i=1
−→
N X i=1
i=1
ei , Aei
Z
RN
=
|y|2 γ0,A (dy).
RN
Hence, by Lemmas 2.1.7 and 2.3.3 plus (2.3.7), all that we have to do is check that i h √ 1 ˘ (*) fn (ξ) ≡ EP e −1 (ξ,Sn )RN −→ e− 2 (ξ,Aξ)RN
for each ξ ∈ RN . When ξ = 0, (*) is trivial. Thus, assume that ξ 6= 0, set e = (e,Sn )RN . Because S˘n (e) =
ξ |ξ| ,
and take
Σn (e)
n X 2 1 EP e, Xm RN , e, Xm RN ≥ Σn (e) 2 Σn (e) m=1
≤
n X 2 1 EP e, Xm RN , e, Xm RN ≥ ρ(e)Σn (e) 2 2 ρ(e) Σn m=1
§ 2.3 Some Extensions of The Central Limit Theorem
87
tends to 0 for each > 0, Theorem 2.1.8 combined with Lemma 2.3.3 guarantees that, for any η ∈ R, √ 2 1 ˘ EP e −1 ηn Sn (e) −→ e− 2 |η| p for any {ηn : n ≥ 1} ⊆ R that tends to η. In particular, if η = (ξ, Aξ)RN and ηn = ρn (e)|ξ|, we find that √ 1 ˘ fn (ξ) = EP e −1 ηn Sn (e) −→ e− 2 (ξ,Aξ)RN .
When C is non-degenerate, the final part is a trivial application of the initial part. When C is degenerate but not zero, one can reduce to the non-degenerate case by projecting onto the span of its eigenvectors with strictly positive eigenvalues, and when C = 0, there is nothing to do. § 2.3.3. Higher Moments. In this subsection I will show that when the Xn ’s possess higher moments, then (2.1.1) remains true for ϕ’s that can grow faster than 1 + |y|2 . As an initial step in this direction, I give the following simple example. Lemma 2.3.13. Suppose that {Xn : n ≥ 1} is a sequence of independent, identically distributed random variables with mean value 0 and variance 1. If EP [X12` ] < ∞ for some ` ∈ Z+ , then (2.1.1) holds for any ϕ ∈ C(RN ; C) that satisfies (2.3.14)
sup y∈R
|ϕ(y)| < ∞. 1 + |y|2`
Proof: Refer to the discussion in the introduction to this chapter, and observe that the argument there shows that Z (2`)! P ˘2` y 2` γ0,1 (dy) lim E Sn = ` = n→∞ 2 `! R
whenever the 2`th moment of X1 is finite. Hence the desired conclusion is an application of the last part of Lemma 2.1.7 with ψ(y) = 1 + |y|2` . In most situations one cannot carry out the computations needed to give a direct proof that the last part of Lemma 2.1.7 applies, and for this reason the following lemma is often useful. Lemma 2.3.15. Suppose that {µn : n ≥ 1} is a sequence of finite (nonnegative) Borel measures on RN , and assume µ is a finite Borel measure with the property that hϕ, µn i −→ hϕ, µi for all ϕ ∈ Cb∞ (RN ; R). If, for some ψ ∈ C RN ; [0, ∞) and p ∈ (1, ∞), (2.3.16)
suphψ p , µn i < ∞, n≥1
then hϕn , µn i −→ hϕ, µi whenever {ϕn : n ≥ 1} ⊆ C(RN ; C) is a sequence that satisfies |ϕn | ≤ ψ for all n ∈ Z+ and converges to ϕ uniformly on compacts.
88
2 The Central Limit Theorem
Proof: By Lemma 2.1.7, all that we have to prove is that hψ, µn i −→ hψ, µi. For this purpose, note that, under our present hypotheses, Lemma 2.1.7 shows that hψ, µi ≤ limn→∞ hψ, µn i < ∞ and that hψ ∧ R, µn i −→ hψ ∧ R, µi ≤ hψ, µi for each R > 0. Thus, it suffices to observe that Z suph(ψ − ψ ∧ R), µn i = sup ψ dµn ≤ R1−p suphψ p , µn i −→ 0 n≥1
n≥1
n≥1
{ψ>R}
as R → ∞. Knowing Lemma 2.3.15, one’s problem is to find conditions under which one ˘ n )] < ∞ for an interesting class of non-negative can show that supn≥1 EP [ψ(S ψ’s. One such class is provided by the notion of a sub-Gaussian random variable. Given β ∈ [0, ∞), an RN -valued random variable X is said to be β-subGaussian if β 2 |ξ|2 EP e(ξ,X)RN ≤ e 2 ,
(2.3.17)
ξ ∈ RN .
The origin of this terminology should be clear: if X ∈ N (0, σ 2 ), then equality holds in (2.3.17) with β = σ. Lemma 2.3.18. Let X be an RN -valued random variable. If X is a β-subGaussian, then EP [X] = 0, Cov(X) ≤ β 2 I, 2 − R P |X| ≥ R ≤ 2N e 2N β2 ,
R > 0,
and, for each α ∈ [0, β −1 ),
− N α2 |X|2 ≤ 1 − (αβ)2 2 . EP e 2
α2 |X|2 < ∞ for some α ∈ (0, ∞) and EP [X] = 0, then X Conversely, if A ≡ EP e 2 √ 2(1+A) . In particular, if X is a bounded random is β-sub-Gaussian when β = α variable with mean value 0, then X is β-sub-Gaussian with β ≤ 2kXkL∞ (P;RN ) . Finally, if X1 , . . . , Xn are independent random variables, and, Pnfor each 1 ≤ m ≤ n, Xm is βm -sub-Gaussian, then for any a , . . . , a ∈ R, 1 n m=1 am Xm is pPn 2 β-sub-Gaussian when β = m=1 (am βm ) .
Proof: Since the moment generating function of the sum of independent random variables is the product of the moment generating functions of the summands, the final assertion is essentially trivial. To prove the first assertion, use Lebesgue’s Dominated Convergence Theorem to justify β 2 t2 e 2 −1 −1 P t(e,X)RN =0 ±E (e, X)RN = lim t E e − 1 ≤ lim t&0 t&0 t
P
§ 2.3 Some Extensions of The Central Limit Theorem
89
and β 2 t2 EP et(e,X)RN + EP e−t(e,X)RN − 2 e 2 −1 2 = β2 ≤ 2 lim E (e, X)RN = lim t&0 t&0 t2 t2 P
for any e ∈ SN −1 . Next, from P (e, X)RN ≥ R) ≤ e
−tR
t(e,X) N β 2 t2 R E e ≤ exp −tR + 2 P
for any ≥ 0 and e ∈ SN −1 , one gets P (e, X)RN ≥ R) ≤ e over t ≥ 0. Since
−
R2 2β 2
by minimizing
1 P |X| ≥ R ≤ 2N max P (e, X)| RN ≥ N − 2 R , e∈SN −1
α2 |X|2 , use the estimate for P(|X| ≥ R) follows. To get the estimate on EP e 2 Tonelli’s Theorem to see that Z Z − N α2 |X|2 β 2 |ξ|2 = EP e(ξ,X)RN γ0,α2 I (dξ) ≤ e 2 γ0,α2 I (dξ) = 1−(αβ)2 2 . EP e 2 R
R
α2 |X|2 < ∞ for some α ∈ (0, ∞) and that EP [X] Now assume that A = EP e 2 = 0. Then 1
|ξ|2 P 2 |ξ||X| E |X| e (1 − t)EP (ξ, X)2RN et(ξ,X)RN dt ≤ 1 + 2 0 |ξ|2 A|ξ|2 |ξ|2 |ξ|22 |ξ|2 |ξ|22 P 2 α2 |X|2 4 α2 , α α e ≤ 1+ ≤1+A 2 e e E |X| e ≤1+ α2 α 2
EP e(ξ,X)RN = 1 +
Z
from which it is clear that X is β-sub-Gaussian for the prescribed β. In par α2 |X|2 α2 K 2 ≤ e 2 for all α ≥ 0, ticular, if K = kXkL∞ (P;RN ) ∈ (0, ∞), then EP e 2 and X has mean value 0, then X is β-sub-Gaussian for β = p so, if, in addition, √ t−1 (1 + etK 2 ) for all t > 0. Taking t = K −2 , we see that β = K 1 + e ≤ 2K. When kXkL∞ (P;RN ) = 0, there is nothing to do.
By combining Lemmas 2.3.15 and 2.3.18 with Theorem 2.3.8, we get the following. Theorem 2.3.19. Working in the setting and with the notation in Theorem 2.3.8, assume that, for each n ∈ Z+ , 2 EP e(ξ,Xn )RN ≤ eβn |ξ| ,
ξ ∈ RN ,
90
2 The Central Limit Theorem
where βn ∈ (0, ∞). If pPn
m=1
β ≡ sup
2 βm
Σn
n≥1
< ∞,
then (2.3.12) holds for any ϕ ∈ C(RN ; C) satisfying α2 |y|2
|ϕ(y)| ≤ Ce 2 , y ∈ RN , for some C < ∞ and α ∈ 0, β1 . In particular, if the Xn ’s are identically 2 2 distributed with covariance C and if EP eα |X1 | < ∞ for some α ∈ (0, ∞), then, for any ϕ ∈ C(RN ; C),
1 lim |y|−2 log 1 + |ϕ(y)| = 0 =⇒ lim EP ϕ n− 2 Sn = hϕ, γ0,C i. n→∞
|y|→∞
Exercises for § 2.3 Exercise 2.3.20. Here is a proof of Feller’s part of the Lindeberg–Feller Theorem. Referring to Theorem 2.1.4 and the discussion proceeding it, assume that, as n → ∞, √ ξ2 ˘ rn −→ 0 and EP e −1ξSn −→ e− 2 for all ξ ∈ R.
(i) Show that h √ ξXm i ξ 2 σ 2 m , 1 − EP e −1 Σn ≤ 2 and conclude that, for each R > 0, there is an NR such that h √ ξXm i 1 for n ≥ NR and |ξ| ≤ R. max 1 − EP e −1 Σn ≤ 1≤m≤n 2
P∞ (1−ζ)k for ζ ∈ C (ii) Take the branch of the logarithm k given by log ζ = − k=1 with |1 − ζ| < 1, and check that (1 − ζ) + log ζ ≤ |1 − ζ|2 for |1 − ζ| ≤ 12 . Conclude first that n n X h i X h √ ξXm i R2 r2 √ ξX m n log EP e −1 Σn ≤ EP 1 − e −1 Σn + 2 m=1
m=1
for n ≥ NR and |ξ| ≤ R, and then that
n X ξXm ξ2 P −→ 0 − E 1 − cos ∆n (ξ) ≡ Σn 2 m=1 uniformly for ξ’s in compacts.
Exercises for § 2.3
91
(iii) Given > 0, show that n n X ξ2 X P 2 ξXm P E Xm , |Xm | < Σn , |Xm | < Σn ≤ E 1 − cos 2 2Σ Σ n n m=1 m=1
≤
and that
n X
ξ2 ξ2 − gn () 2 2
ξXm , |Xm | ≥ Σn ≤ −2 . E 1 − cos Σ n m=1 P
Finally, combine these and apply (ii) to get limn→∞ ξ 2 gn () ≤ −2 for all ξ ∈ R.
Exercise 2.3.21. It is of some interest to know that the second moment assumption can be removed from the hypotheses in Exercise 2.1.11 and that the result there extends to Borel probability measures on RN .R To explain what I have in mind, first use that exercise to see that if σ 2 = R x2 µ(dx) < ∞, then µ = T2 µ =⇒ µR∈ N (0, σ 2 ). What I want to do now is remove the a priori assumption that R x2 µ(dx) < ∞. That is, I want to show that, for any probability measure µ on R, µ = T2 µ ⇐⇒ µ ∈ N (0, σ 2 ) for some σ ∈ [0, ∞). Since the “⇐=” direction is obvious, R and, by the discussion above, the “ =⇒ ” direction is already covered when R x2 µ(dx) < ∞, all that remains is to show that Z (2.3.22) µ = T2 µ =⇒ x2 µ(dx) < ∞. R
See Exercise 2.4.33 for an interesting application of this result. (i) Check (2.3.22) first under the condition that µ is symmetric (i.e., µ(−Γ) = µ(Γ) for all Γ ∈ BR ). Indeed, if µ is symmetric, show that Z µ ˆ(ξ) = cos(ξx) µ(dx), ξ ∈ R. R
At the same time, show that 1 1 ˆ(ξ) 2 , µ = T2 µ =⇒ µ ˆ 2− 2 ξ = µ
ξ ∈ R.
Conclude from these two that µ ˆ > 0 everywhere and that Z −n n ˆ(ξ)2 , n ∈ N and ξ ∈ R. cos 2− 2 ξx µ(dx) = µ R
Finally, note that 1 − x ≤ − log x for x ∈ (0, 1], apply this to the preceding to get Z n n ˆ(1) < ∞, n ∈ N, 2 1 − cos 2− 2 x µ(dx) ≤ − log µ R
92 and arrive at
2 The Central Limit Theorem Z
x2 µ(dx) ≤ −2 log µ ˆ(1)
R
after an application of Fatou’s Lemma. (ii) To complete the program, let µ be any solution to µ = T2 µ, and define ν by ZZ ν(Γ) = 1Γ (x − y) µ(dx)µ(dy). R2
R Check that ν is symmetric and that ν = T2 ν. Hence, by (i), R x2 ν(dx) < ∞ (in fact, ν is centered normal). Finally, use this and part (i) of Exercise 1.4.27 R to deduce that R x2 µ(dx) < ∞. (iii) Make the obvious extension of T2 to Borel probability measures µ on RN . That is, ZZ x+y µ(dx)µ(dy) for Γ ∈ BRN . T2 µ(Γ) = 1Γ 1 22 RN ×RN
Using the result just proved when N = 1, show that µ = T2 µ if and only if µ = γ0,C for some non-negative definite, symmetric C. Exercise 2.3.23. In connection with the preceding exercise, define Tα µ for α ∈ (0, ∞) and Borel probability measures µ on RN , so that ZZ 1 Tα µ(Γ) = 1Γ 2− α (x + y) µ(dx)µ(dy), Γ ∈ BRN . RN ×RN
The problem under consideration here is that of determining for which α’s there exist nontrivial (i.e., µ 6= δ0 ) solutions to the fixed point equation µ = Tα µ. Begin by reducing the problem to the case when N = 1. Next, repeat the initial argument given in part (ii) of Exercise 2.3.21 to see that there is some solution if and only if there is one that is symmetric. Assuming that µ is a non-trivial, symmetric solution, use the reasoning in part (i) there to see that Z ∞ if α ∈ (0, 2) 2 x µ(dx) = 0 if α ∈ (2, ∞). R In particular, when α ∈ (2, ∞), there are no non-trivial solutions to µ = Tα µ. (See § 3.2.3 for more on this topic.) Exercise 2.3.24. Return to the setting of Exercise 2.1.13. After noting that, so long as e ∈ Sn−1 , the distribution of √ x ∈ Sn−1 n 7−→ (e, x)Rn ∈ R
is independent of e, use Lemma 2.3.3 to prove that the assertion in (2.1.15) follows as a consequence of the one in (2.1.14).
Exercises for § 2.3
93
Exercise 2.3.25. Begin by checking the identity (cf. (1.3.20)) Z ∞ 2 s−1 s+1 − t ts e 2β2 dt = 2 2 β s+1 Γ 2 0
for all β ∈ (0, ∞) and s ∈ (−1, ∞). Use the preceding to see that, for each p ∈ (0, ∞), r p p+1 2p P σ p if X ∈ N (0, σ 2 ). Γ (2.3.26) E |X| = 2 π
The goal of the exercise is to show that the moments of sub-Gaussian random variable display similar behavior. (i) Suppose that X is β-sub-Gaussian, and show that, for each p ∈ (0, ∞), p p p p +1 . = 2 2 +1 Γ EP |X|p ≤ Kp β p where Kp ≡ p2 2 Γ 2 2
(ii) Again suppose that X is β-sub-Gaussian, and let σ 2 be its variance. Show that 2+|p−2| + σ −(1− p 2) βp EP |X|p ≥ K4 β
for each p ∈ (0, ∞). Hint: When p ≥ 2, the inequality is trivial. To prove it when p < 2, show that, for any q ∈ (1, ∞), 1 2q−p 1 σ 2 ≤ EP |X|p q EP |X| q−1 q0 ,
where q 0 =
q q−1
is the H¨ older conjugate of q.
(iii) Suppose that X1 , . . . , Xn are independent and that, for each 1 ≤ m ≤ n, 2 Xm is βm -sub-Gaussian and has variance σm . Given {a1 , . . . , an } ⊆ R, set v v u n u n n X uX uX (am βm )2 , (am σm )2 , and B = t S= am Xm , Σ = t m=1
m=1
m=1
and show that, for each p ∈ (0, ∞), + −(1− p 2) K4
Σ B
2+|p−2|
B p ≤ EP |S|p ≤ Kp B p .
In particular, if βm = β and σm = σ for all 1 ≤ m ≤ n, then + −(1− p 2)
K4
2+|p−2| σ (βA)p ≤ EP |S|p ≤ Kp (βA)p , β
v u n uX a2m . where A = t m=1
94
2 The Central Limit Theorem
(iv) The most famous case of the situation discussed in (iii) is when the Xm ’s are symmetric Bernoulli (i.e., P(Xm = ±1) = 12 ). First use (iii) in Exercise 1.3.17 or direct computation to check that Xm is 1-sub-Gaussian, and then conclude that + −(1− p 2)
(2.3.27)
K4
n X
! p2 a2m
p # " n X P ≤E am Xm ≤ Kp
n X
m=1
m=1
! p2 a2m
m=1
for all {a1 , . . . , an } ⊆ R. This fact is known as Khinchine’s Inequality. Exercise 2.3.28. Let X1 , . . . ,P Xn be independent, symmetric (Exercise 1.4.26) n random variables, and set S = 1 Xm . Show that, for each p ∈ (0, ∞) (cf. part (ii) in Exercise 2.3.25),
+ −(1− p 2)
K4
EP
n X
! p2 ! p2 n h i X 2 2 . ≤ EP |S|p ≤ Kp EP Xm Xm 1
1
Hint: Refer to the beginning of the proof of Lemma 1.1.6, and let R1 , . . . , Rn be the Rademacher functions on [0, 1), set Q = λ[0,1) × P on [0, 1) × Ω, B[0,1) × F , and observe that n X ω ∈ Ω 7−→ S(ω) ≡ Xm (ω) 1
has the same distribution under P as (t, ω) ∈ [0, 1) × Ω 7−→ T (t, ω) ≡
n X
Rm (t)Xm (ω)
1
does under Q. Next, apply Khinchine’s inequality to see that, for each ω ∈ Ω, + −(1− p 2) K4
n X 1
! p2 Xm (ω)2
Z ≤ [0,1)
T (t, ω) p dt ≤ Kp
n X
! p2 Xm (ω)2
,
1
and complete the proof by taking the P-integral of this with respect to ω. At least when p ∈ (1, ∞), I will show later that this sort of inequality holds in much greater generality. Specifically, see Burkholder’s Inequality in Theorem 6.3.6. Exercise 2.3.29. Suppose that X is an RN -valued Gaussian random variable with mean value 0 and covariance C. (i) Show that if A : RN −→ RN is a linear transformation, then AX is an N (0, ACA> ) random variable, where A> is the adjoint transformation.
Exercises for § 2.3
95
(ii) Given a linear subspace L of RN , let FL be the σ-algebra generated by {(ξ, X)RN : ξ ∈ L}, and take L⊥C to be the subspace of η such that (η, Cξ)RN = 0 for all ξ ∈ L. Show that FL is independent of FL⊥C . Hint: Show that, because of linearity, it suffices to check that √ √ √ √ EP e −1(ξ,X)RN e −1(η,X)RN = EP e −1(ξ,X)RN EP e −1(η,X)RN
for all ξ ∈ L and η ∈ L⊥C . (iii) that N = N1 + N2 , where Ni ∈ Z+ for i ∈ {1, 2}, write RN 3 x = Suppose x(1) ∈ RN1 × RN2 , and take L = {x : x(1) = 0(1) }. Show that if Π is a x(2) linear transformation taking RN onto L that satisfies ξ − Πξ, Cη RN = 0 for all ξ ∈ RN and η ∈ L, then Π> X is independent of (I − Π> )X. (iv) Write C=
C(11) C(21)
C(12) C(22)
,
where the block structure corresponds to RN = RN1 × RN2 , and assume that C(22) is non-degenerate. Show that the one and only transformation Π of the sort in part (iii) is given by Π= and therefore that Π> =
0(11) C−1 (22) C(21)
0(11) 0(21)
0(12) I(22)
,
C(12) C−1 (22) I(22)
.
Hint: Note that Πξ = 0 if ξ(2) = 0(2) , Πξ = ξ if ξ(1) = 0(1) , and that C(I − Π) (21) = 0(21) . (v) Continuing with the assumption that C(22) is non-degenerate, show that X=
C(12) C−1 (22) Y Y
+
Z 0
,
where Y is an RN2 -valued, N (0, C(22) )-random variable, Z is an RN1 -valued N (0, B) random variable with B = C(11) − C(12) C−1 (22) C(21) , and Y is indepenN1 N2 dent of Z. Conclude −→ R that is that, for any measurable F : R × R P bounded below, E F (X(1) , X(2) ) equals Z RN2
Z RN1
F x(1) , x(2) γC(12) C−1
(22)
x(2) ,B (dx(1) ) γ0,C(22) (dx(2) ).
96
2 The Central Limit Theorem
Exercise 2.3.30. Given h ∈ L2 (RN ; C), recall that the (n + 2)-fold convolution h?(n+2) is a bounded continuous function for each n ∈ N. Next, assume that h(−x) = h(x) for almost every x ∈ RN and that h ≡ 0 off of BRN (0, 1). As an application of part (iii) in Exercise 1.3.22, show that
"
?(n+2) h (x) ≤ 2khk2 2 L
(|x| − 2)+ n khk exp − N 1 N (R ;C) L (R ;C) 2n
2 #
.
Hint: Note that h ∈ L1 (RN ; C), assume that M ≡ khkL1 (RN ;C) > 0, and define Af = M −1 h ? f for f ∈ L2 (RN ; C). Show that A is a self-adjoint contraction on L2 (RN ; C), check that h?(n+2) (x) = M n Tx h, An h L2 (RN ;C) , where Tx h ≡ h( · + x), and note that Tx h, A` h L2 (RN ;C) = 0
if ` ≤ |x| − 2.
§ 2.4 An Application to Hermite Multipliers This section does not really belong here and should probably be skipped by those readers who want to restrict their attention to purely probabilistic matters. On the other hand, for those who want to see how probability theory interacts with other branches of mathematical analysis, the present section may come as something of a revelation. § 2.4.1. Hermite Multipliers. The topic of this section will be a class of linear operators called Hermite multipliers, and what will be discussed are certain boundedness properties of these operators. The setting is as follows. For n ∈ N, define (2.4.1)
Hn (x) = (−1)n e
x2 2
dn − x2 e 2 , dxn
x ∈ R.
Clearly, Hn is an nth order, real, monic (i.e., 1 is the coefficient of the highest order term) polynomial. Moreover, if we define the raising operator A+ on C 1 (R; C) by
x2 dϕ d − x2 e 2 ϕ(x) = − (x) + xϕ(x), A+ ϕ (x) = −e 2 dx dx
then (2.4.2)
Hn+1 = A+ Hn
for all n ∈ N.
x ∈ R,
§ 2.4 An Application to Hermite Multipliers
97
At the same time, if ϕ and ψ are continuously differentiable functions whose first derivatives are tempered (i.e., have at most polynomial growth at infinity), then (2.4.3)
ϕ, A+ ψ
L2 (γ
0,1 ;C)
= A− ϕ, ψ
L2 (γ0,1 ;C)
where A− is the lowering operator given by A− ϕ = (2.4.2) with (2.4.3), we see that, for all 0 ≤ m ≤ n,
Hm , Hn
L2 (γ0,1 ;C)
= Hm , An+ H0
L2 (γ0,1 ;C)
= An− Hm , H0
,
dϕ dx .
After combining
L2 (γ0,1 ;C)
= m! δm,n ,
where, at the last step, I have used the fact that Hm is a monic mth order polynomial. Hence, the (normalized) Hermite polynomials
(−1)n x2 dn − x2 Hn (x) e 2 , = √ e2 H n (x) = √ dxn n! n!
x ∈ R,
form an orthonormal set in L2 (γ0,1 ; C). (Indeed, they are one choice of the orthogonal polynomials relative to the Gauss weight.) Lemma 2.4.4. For each λ ∈ C, set
λ2 , H(x; λ) = exp λx − 2
x ∈ R.
Then (2.4.5)
H(x; λ) =
∞ X λn Hn (x), n! n=0
x ∈ R,
where the convergence is both uniform on compact subsets of R× C and, for λ’s in compact subsets of C, uniform in L2 (γ0,1 ; C). In particular, H n : n ∈ N is an orthonormal basis in L2 (γ0,1 ; C). x2
Proof: By (2.4.1) and Taylor’s expansion for the function e− 2 , it is clear that (2.4.5) holds for each (x, λ) and that the convergence is uniform on compact subsets of R × C. Furthermore, because the Hn ’s are orthogonal, the asserted uniform convergence in L2 (γ0,1 ; C) comes down to checking that
lim
m→∞
∞ n 2 X λ Hn k2 2 L (γ0,1 ;C) = 0 |λ|≤R n=m n!
sup
for every R ∈ (0, ∞), and obviously this follows from our earlier calculation that
2
Hn 2 = n!. L (γ ;C) 0,1
98
2 The Central Limit Theorem
To prove the assertion that H n : n ∈ N forms an orthonormal basis in L2 (γ0,1 ; C), it suffices to check that any ϕ ∈ L2 (γ0,1 ; C) that is orthogonal to all of the Hn ’s must be 0. But, because of the L2 (γ0,1 ; C) convergence in (2.4.5), we would have that Z ϕ(x) eλx γ0,1 (dx) = 0, λ ∈ C, R
for such a ϕ. Hence, if ψ(x) =
e−
x2 2
√
ϕ(x) , 2π
x ∈ R,
then kψkL1 (R;C) = kϕkL1 (γ0,1 ;C) ≤ kϕkL2 (γ0,1 ;C) < ∞ and (cf. (2.3.2)) ψˆ ≡ 0, which, by the L1 (R; C) Fourier inversion formula Z √ 1 ˆ dξ α&0 −→ ψ in L1 (R; C), e−α|ξ| e− −1 xξ ψ(ξ) 2π R
means that ψ and therefore ϕ vanish Lebesgue-almost everywhere. Now that we know H n : n ∈ N is an orthonormal basis, I can uniquely determine a normal operator Hθ for each θ ∈ C by specifying that
Hθ Hn = θn Hn
for each n ∈ N.
The operator Hθ is called the Hermite multiplier with parameter θ, and clearly ) ( ∞ X 2 Dom Hθ = ϕ ∈ L2 (γ0,1 ; C) : |θ|2n ϕ, H n L2 (γ0,1 ;C) < ∞ n=1
Hθ ϕ =
∞ X
θn ϕ, H N
L2 (γ0,1 ;C)
H n,
ϕ ∈ Dom Hθ .
n=0
In particular, Hθ is a contraction if and only if θ is an element of the closed unit disk D in C, and it is unitary precisely when θ ∈ S1 ≡ ∂D. Also, the adjoint of Hθ is Hθ , and so it is self-adjoint if and only if θ ∈ R. As we are about to see, there are special choices of θ for which the corresponding Hermite multiplier has interesting alternative interpretations and unexpected additional properties. For example, consider the Mehler kernel1 " 2 2 # θx − 2θxy + θy 1 exp − M (x, y; θ) = √ 2 1 − θ2 1 − θ2 1
This kernel appears in the 1866 article by Mehler referred to in the footnote following (2.1.14). √ ∞ . It arises there as the generating function for spherical harmonics on the sphere S∞
§ 2.4 An Application to Hermite Multipliers
99
for θ ∈ (0, 1) and x, y ∈ R. By a straightforward Gaussian computation (i.e., “complete the square” in the exponential) one can easily check that Z H(y; λ) M (x, y; θ) γ0,1 (dy) = H(x; θλ) R
for all θ ∈ (0, 1) and (x, λ) ∈ R × C. In conjunction with (2.4.5), this means that Z (2.4.6)
Hθ ϕ =
M ( · , y; θ) ϕ(y) γ0,1 (dy),
θ ∈ (0, 1) and ϕ ∈ L2 (γ0,1 ; C),
R
and from here it is not very difficult to prove the following properties of Hθ for θ ∈ (0, 1). Lemma 2.4.7. For each ϕ ∈ L2 (γ0,1 ; C), (θ, x) ∈ (0, 1) × R 7−→ Hθ ϕ(x) ∈ C may be chosen to be a continuous function that is non-negative if ϕ ≥ 0 Lebesgue-almost everywhere. In addition, for each θ ∈ (0, 1) and every p ∈ [1, ∞],
Hθ ϕ p L (γ
(2.4.8)
0,1 ;C)
≤ kϕkLp (γ0,1 ;C) .
Proof: The first assertions are immediate consequences of the representation in (2.4.6). To prove the second assertion, observe that Hθ 1 = 1 and therefore, as a special case of (2.4.6), Z M (x, y; θ) γ0,1 (dy) = 1
for all θ ∈ (0, 1) and x ∈ R.
R
Hence, by (2.4.6) and Jensen’s Inequality, for any p ∈ [1, ∞), Hθ ϕ (x) p ≤
Z
At the same time, by symmetry, R, and therefore
R
M (x, y; θ) |ϕ(y)|p γ0,1 (dy).
R
Z
Hθ ϕ (x) p γ0,1 (dx) ≤
ZZ
R
M (x, y; θ) γ0,1 (dx) = 1 for all (θ, y) ∈ (0, 1)×
p
Z
M (x, y; θ) |ϕ(y)| γ0,1 (dx)γ0,1 (dy) =
R
|ϕ|p dγ0,1 .
R R×R
Hence, (2.4.8) is now proved for p ∈ [1, ∞). The case when p = ∞ is even easier and is left to the reader. The conclusions drawn in Lemma 2.4.7 from the Mehler representation in (2.4.6) are interesting but not very deep (cf. Exercise 2.4.36). A deeper fact is
100
2 The Central Limit Theorem
the relationship between Hermite multipliers and the Fourier transform. For the purposes of this analysis, it is best to define the Fourier operator F by Z √ (2.4.9) Ff (ξ) = e −1 2πξx f (x) dx, ξ ∈ R, R
1
for f ∈ L (R; C). The advantage of this choice is that, without the introduction √ of any further factors of 2π, the Parseval Identity (cf. Exercise 2.4.37) becomes the statement that F determines a unitary operator on L2 (R; C). In order to relate F to Hermite multipliers, observe that, after analytically continuing the result of another simple Gaussian computation, Z ζ2 2 eζx e−πx dx = e 4π for all ζ ∈ C, R
we see from (2.4.5) that Z √ ∞ X p 2 λn 2πp x e−πx dx e −1 2πξx Hn n! R n=0
∞ n p p 2 X λ (p − 1)λ2 √ 2πp0 ξ , θpn Hn + −1 λ 2πp ξ = e−πξ =e exp n! 2 n=0 √ 1 p is the H¨ older conjugate of p and θp ≡ −1 (p − 1) 2 . Thus, where p0 = p−1 we have now proved that, for each p ∈ (1, ∞) and n ∈ N, Z √ p p 2 2 2πp0 x e−πξ . 2πp x e−πx dx = θpn Hn (2.4.10) e −1 2πξx Hn
−πξ 2
R
In particular, when p = 2, (2.4.10) says that √ n −1 hn , (2.4.11) Fhn =
n ∈ N,
where hn is the nth (un-normalized) Hermite function given by 2 1 (2.4.12) hn (x) = Hn (4π) 2 x e−πx , n ∈ N and x ∈ R.
More generally, (2.4.10) leads to the following relationship between F and Hermite multipliers. Namely, for each p ∈ (1, ∞), define Up on Lp (γ0,1 ; C) by 2 1 1 Up ϕ (x) = p 2p ϕ (2πp) 2 x e−πx , x ∈ R.
It is then an easy matter to check that Up is an isometric surjection from Lp (γ0,1 ; C) onto Lp (R; C). In addition, (2.4.10) can now be interpreted as the statement that, for every p ∈ (1, ∞) and every polynomial ϕ, ! 12 1 pp . (2.4.13) F ◦ Up ϕ = Ap Up0 ◦ Hθp ϕ where Ap ≡ 1 (p0 ) p0
See Exercise 2.4.35, where it is shown that Ap < 1 for p ∈ (0, 1).
§ 2.4 An Application to Hermite Multipliers
101
§ 2.4.2. Beckner’s Theorem. Having completed this brief introduction to Hermite multipliers, I will now address a problem to which The Central Limit Theorem has something to contribute. The problem is that of determining the set of (θ, p, q) ∈ D × (1, ∞) × (0, ∞) with p ≤ q for which Hθ determines a contraction from Lp (γ0,1 ; C) into Lq (γ0,1 ; C). In view of the preceding discussion, when θ ∈ (0, 1), a solution to this problem has implications for the Mehler transform; and, when q = p0 , the solution tells us about the Fourier operator. The role that The Central Limit Theorem plays in this analysis is hidden in the following beautiful criterion, which was first discovered by Wm. Beckner.2 Theorem 2.4.14 (Beckner). Let θ ∈ D and 1 ≤ p ≤ q < ∞ be given. Then (2.4.15)
Hθ ϕ q L (γ
0,1 ;C)
≤ kϕkLp (γ0,1 ;C)
for all
ϕ ∈ L2 (γ0,1 ; C)
if (2.4.16)
|1 − θζ|q + |1 + θζ|q 2
q1
≤
|1 − ζ|p + |1 + ζ|p 2
p1
for every ζ ∈ C. That (2.4.16) implies (2.4.15) is trivial is quite remarkable. Indeed, it takes a problem in infinite dimensional analysis and reduces it to a calculus question about functions on the complex plane. Even though, as we will see later, this reduction leads to highly non-trivial problems in calculus, Theorem 2.4.14 has to be considered a major step toward understanding the contraction properties of Hermite multipliers.3 The first step in the proof of Theorem 2.4.14 is to interpret (2.4.16) in operator theoretic language. For this the standard Bernoulli purpose, let β denote probability measure on R, BR . That is, β {±1} = 12 . Next, use χ∅ to denote the function on R that is constantly equal to 1 and χ{1} to stand for the identity function on R (i.e., χ{1} (x) = x, x ∈ R). It is then clear that χ∅ and χ{1} constitute an orthonormal basis in L2 (β; C); in fact, they are the orthogonal polynomials there. Hence, for each θ ∈ C, we can define the Bernoulli multiplier Kθ as the unique normal operator on L2 (β; C) prescribed by
Kθ χF = 2
χ∅
if F = ∅
θχ{1}
if F = {1}.
See Beckner’s “Inequalities in Fourier analysis,” Ann. Math., # 102 #1, pp. 159–182 (1975). Later, in his article “Gaussian kernels have only Gaussian maximizers,” Invent. Math. 12, pp. 179–208 (1990), E. Lieb essentially killed this line of research. His argument, which is entirely different from the one discussed here, handles not only the Hermite multipliers but essentially every operator whose kernel can be represented as the exponential of a second order polynomial. 3
102
2 The Central Limit Theorem
Furthermore, (2.4.16) is equivalent to the statement that
Kθ ϕ q (2.4.17) ≤ kϕkLp (β,C) for all ϕ ∈ L2 (β; C). L (β;C)
Indeed, it is obvious that (2.4.16) is equivalent to (2.4.17) restricted to ϕ’s of the form x ∈ R 7−→ 1 + ζx as ζ runs over C; and from this, together with the observation that every element of L2 (β; C) can be represented in the form aχ∅ + bχ{1} as (a, b) runs over C2 , one quickly concludes that (2.4.16) implies (2.4.17) for general ϕ ∈ L2 (β; C). I next want to show that (2.4.17) can be parlayed into a seemingly more general statement. To this end, define the n-fold tensor product operator Kθ⊗n on L2 (β n ; C) as follows. For F ⊆ {1, . . . , n} set χF ≡ 1 if F = ∅ and define Y χF (x) = χ{1} (xj ) for x = x1 , . . . , xn ∈ Rn j∈F
if F 6= ∅. Note that χF : F ⊆ {1, . . . , n} is an orthonormal basis for L2 (β n ; C), and define Kθ⊗n to be the unique normal operator on L2 (β n ; C) for which Kθ⊗n χF = θ|F | χF ,
(2.4.18)
F ⊆ {1, . . . , n},
where |F | is used here to denote the number of elements in the set F . Alternatively, one can describe Kθ⊗ninductively on n ∈ Z+ by saying that Kθ⊗1 = Kθ and that, for Φ ∈ C Rn+1 ; C and (x, y) ∈ Rn × R, ⊗(n+1) Kθ Φ (x, y) = Kθ Ψ(x, · ) (y) where Ψ(x, y) = Kθ⊗n Φ( · , y) (x). It is this alternative description that makes it easiest to see the extension of (2.4.17) alluded to above. Namely, what I will now show is that, for every n ∈ Z+ ,
(2.4.19) (2.4.17) =⇒ K⊗n Φ q n ≤ kΦkLp (β n ;C) , Φ ∈ L2 (β n ; C). θ
L (β ;C)
Obviously, there is nothing to do when n = 1. Next, assume (2.4.19) for n, let Φ ∈ C Rn+1 ; C be given, and define Ψ as in the second description of ⊗(n+1) Kθ Φ. Then, by (2.4.17) applied to Ψ(x, · ) for each x ∈ Rn and by the induction hypothesis applied to Φ( · , y) for each y ∈ R, we have that Z Z
⊗(n+1) q q
K
q n+1 = Φ K Ψ(x, · ) (y) β(dy) β n (dx) θ θ L (β ;C) Rn
Z
Z ≤
pq
p
|Ψ(x, y)| β(dy) Rn
|Ψ( · , y)|p
≤ Rn
Z ≤ R
Z
β (dx) =
n
Rn
R
Z
R
q
L p (β n ;C)
pq p
Ψ( · , y) β(dy)
q
L p (β n ;C)
Z pq pq
p
= Ψ( · , y) Lq (β n ;C) β(dy) β(dy) R
pq
Φ( · , y) p p n β(dy) = kΦkqLp (β n+1 ;C) , L (β ;C)
§ 2.4 An Application to Hermite Multipliers
103
where, in the passage to the third line, I have used the continuous form of Minkowski’s Inequality (it is at this point that the only essential use of the hypothesis p ≤ q is made). I am now ready to take the main step in the proof of Theorem 2.4.14. Lemma 2.4.20. Define An : L2 (β; C) −→ L2 β n ; C) by An ϕ (x) = ϕ
Pn `=1 x` √ n
for x ∈ Rn .
Then, for every pair of tempered ϕ and ψ from C(R; C),
(2.4.21) kϕkLp (γ0,1 ;C) = lim An ϕ Lp (β n ;C) for every p ∈ [1, ∞) n→∞
and (2.4.22)
Hθ ϕ, ψ
L2 (γ0,1 ;C)
= lim Kθ⊗n ◦ An ϕ, An ψ n→∞
L2 (β n ;C)
for every θ ∈ (0, 1). Moreover, if, in addition, either ϕ or ψ is a polynomial, then (2.4.22) continues to hold for all θ ∈ C. Proof: Let ϕ and ψ be tempered elements of C(R; C), and define fn (θ) = Kθ⊗n ◦ An ϕ, An ψ
L2 (β n ;C)
and f (θ) = Hθ ϕ, ψ
L2 (γ0,1 ;C)
for n ∈ Z+ and θ ∈ C. I begin by showing that (2.4.23)
θ ∈ (0, 1).
lim fn (θ) = f (θ),
n→∞
Notice that (2.4.23) is (2.4.22) for θ ∈ (0, 1) and that In (2.4.21) follows from (2.4.22) with ϕ = 1, ψ = |ϕ|p , and any θ ∈ (0, 1). In order to prove (2.4.23), I will need to introduce other expressions for f (θ) and the fn (θ)’s. To this end, set Cθ =
1 θ
θ 1
,
and, using (2.4.6), observe (cf. (2.3.6)) that Z f (θ) = ϕ(x) ψ(y) γ0,Cθ (dx × dy). R2
Next, let, for each x ∈ R\{0}, define kθ (x, · ) to be the probability measure on R such that kθ x, {±sgnx} = 1±θ 2 , and set kθ (0, · ) = β. Then it is easy to check
104
2 The Central Limit Theorem
R R that R χ{0} (y) kθ (±1, dy) =R χ{0} (±1) and R χ{1} (y) kθ (±1, dy) = θχ{1} (±1) and therefore Kθ ϕ(±1) = R ϕ(y) kθ (±1, dy) for all ϕ. Hence, if βθ be the probability measure on R2 determined by βθ (dx × dy) = kθ (x, dy) β(dx) or, equivalently, and βθ {(±1, ∓1)} = 1−θ βθ {(±1, ±1)} = 1+θ 4 , 4
then Kθ ϕ, ψ
Z
L2 (β;C)
ϕ(x) ψ(y) βθ (dx × dy).
= R2
Proceeding by induction, it follows that Z Z ⊗n Kθ Φ, Ψ 2 = ··· Φ(x) Ψ(y) βθ (dx1 × dy1 ) · · · βθ (dxn × dyn ) L (β;C)
R2
R2
for all Φ, Ψ ∈ C(Rn ; C). Hence, if (cf. Exercise 1.1.14) Ω = R2 Z+ and Pθ = βθ , then fn (θ) = E
Pθ
Z+
, F = BΩ ,
Pn 1 Zm √ , F n
where F (z) ≡ ϕ(x) ψ(y) for z = (x, y) ∈ R2 and Zn (ω) = zn , n ∈ Z+ , when ω = (z1 , . . . , zn , . . . ) ∈ Ω. Further, under Pθ , the Zn ’s are mutually independent, identically distributed R2 -valued random variables with mean value 0 and covariance Cθ . In addition, Z1 is bounded, and therefore the last part of Theorem 2.3.19 applies and guarantees that (2.4.23) holds. To complete the proof, suppose that ϕ is a polynomial of degree k. It is then an easy matter to check that An ϕ, χF L2 (β n ;C) = 0 if |F | > k,
and therefore (cf. (2.4.18)) θ ∈ C 7−→ fn (θ) ∈ C is also a polynomial of degree no more than k. Moreover, because X |F | fn (θ) = θ An ϕ, χF L2 (β n ;C) χF , An ψ L2 (β n ;C) , F
we also know that
fn (θ) ≤ |θ| ∨ 1 k An ϕ 2 n An ψ 2 n , L (β ;C) L (β ;C)
n ∈ Z+ and θ ∈ C.
Hence, because of (2.4.21) with p = 2, {fn : n ∈ Z+ } is a family of entire functions on C that are uniformly bounded on compact subsets. At the same
§ 2.4 An Application to Hermite Multipliers
105
time, because (ϕ, Hm )L2 (γ0,1 ;C) = 0 for m > k, f is also a polynomial of degree at most k, and therefore (2.4.23) already implies that the convergence extends to the whole of C and is uniform on compacts. Finally, in the case when ψ, instead of ϕ, is a polynomial, simply note that
Kθ⊗n ◦ An ϕ, An ψ
and Hθ ϕ, ψ
L2 (γ
0,1 ;C)
L2 (β n ;C)
= Hθ¯ψ, ϕ
= Kθ⊗n ¯ ◦ An ψ, An ϕ
L2 (γ0,1 ;C)
L2 (β n ;C)
, and apply the preceding.
Proof of Theorem 2.4.14: Assume that (2.4.16) holds for a given pair 1 < p ≤ q < ∞ and θ ∈ D. We then know that (2.4.19) holds for every n ∈ Z+ . Hence, by Lemma 2.4.20, if ϕ and ψ are tempered elements of C(R; C) and at least one of them is a polynomial, then Hθ ϕ, ψ L2 (γ0,1 ;C) = lim Kθ⊗n ◦ An ϕ, An ψ 2 n n→∞
L (β ;C)
≤ lim An ϕ Lp (β n ;C) An ψ Lq0 (β n ;C) = kϕkLp (γ0,1 ;C) kψkLq0 (γ0,1 ;C) . n→∞
In other words, we now know that, for all tempered ϕ and ψ from C(R; C), (2.4.24) H ϕ, ψ θ ≤ kϕkLp (γ0,1 ;C) kψkLq0 (γ0,1 ;C) L2 (γ0,1 ;C) so long as one or the other is a polynomial. To complete the proof when p ∈ (1, 2], note that, for any fixed polynomial ϕ, (2.4.24) for every tempered ψ ∈ C(R; C) guarantees that the inequality in (2.4.15) holds for that ϕ. At the same time, because p ∈ (1, 2] and the polynomials are dense in L2 (γ0,1 ; C), (2.4.15) follows immediately from its own restriction to polynomials. Finally, assume that p ∈ [2, ∞) and therefore that q 0 ∈ (1, 2]. Then, again because the polynomials are dense in L2 (γ0,1 ; C), (2.4.24) for a fixed tempered ϕ ∈ C(R; C) and all polynomials ψ implies (2.4.15) first for all tempered continuous ϕ’s and thence for all ϕ ∈ L2 (γ0,1 ; C). § 2.4.3. Applications of Beckner’s Theorem. I will now apply Theorem 2.4.14 to two important examples. The first example involves the case when θ ∈ (0, 1) and shows that the contraction property proved in Lemma 2.4.7 can be improved to say that, for each p ∈ (1, ∞) and θ ∈ (0, 1), there is a q = q(p, θ) ∈ (p, ∞) such that Hθ is a contraction on Lp (γ0,1 ; C) into Lq (γ0,1 ; C). Such an operator is said to be hypercontractive, and the fact that Hθ is hypercontractive was first proved by E. Nelson in connection with his renowned construction of a non-trivial, two-dimensional quantum field.4 The proof that 4
Nelson’s own proof appeared in his “The free Markov field,” J. Fnal. Anal. 12, pp. 12–21 (1974).
106
2 The Central Limit Theorem
I will give is entirely different from Nelson’s and is much closer to the ideas introduced by L. Gross5 as they were developed by Beckner. Theorem 2.4.25 (Nelson). Let θ ∈ (0, 1) and p ∈ (1, ∞) be given, and set q(p, θ) = 1 +
p−1 . θ2
Then
Hθ ϕ q ≤ kϕkLp (γ0,1 ;C) , L (γ0,1 ;C)
(2.4.26)
ϕ ∈ L2 (γ0,1 ; C),
for every 1 ≤ q ≤ q(p, θ). Moreover, if q > q(p, θ), then n
sup Hθ ϕ Lq (γ
(2.4.27)
0,1 ;C)
o : kϕkLp (γ0,1 ;C) = 1 = ∞.
Proof: I will leave the proof of (2.4.27) as an exercise. (Try taking ϕ’s of 2 the form eλx .) Also, because γ0,1 is a probability measure and therefore the left-hand side of (2.4.26) is non-decreasing as a function of q, I will restrict my attention to the proof of (2.4.26) for q = q(p, θ). Hence, by Theorem 2.4.14, what I have to do is prove (2.4.16) for every 1 < p < q < ∞ and θ ∈ (0, 1) that are related by (2.4.28)
θ=
p−1 q−1
12
.
I begin with the case when 1 < p < q ≤ 2, and I will first consider ζ ∈ [0, 1). Introducing the generalized binomial coefficients r r(r − 1) · · · (r − ` + 1) ≡ `! `
one can write
for r ∈ R and ` ∈ N,
∞ X q |1 − θζ|q + |1 + θζ|q =1+ (θζ)2k 2 2k k=1
and
∞ X p |1 − ζ|p + |1 + ζ|p =1+ ζ 2k . 2 2k k=1
5
See Gross’s “Logarithmic Sobolev inequalities,” Amer. J. Math. 97 #4, pp. 1061–1083 (1975). In this paper, Gross introduced the idea of proving estimates on Hθ from the corresponding estimates for Kθ . In this connection, have a look at Exercises 2.4.39 and 2.4.41.
§ 2.4 An Application to Hermite Multipliers
107
q Noting that, because q ≤ 2, 2k ≥ 0 for every k ∈ Z+ , and using the fact that, p because pq ∈ (0, 1), (1 + x) q ≤ 1 + pq x for all x ≥ 0, we see that
|1 − θζ|q + |1 + θζ|q 2
pq
∞ pX q (θζ)2k . ≤1+ q 2k k=1
Hence, I will have completed the case under consideration once I check that
∞ ∞ X p pX q 2k (θζ) ≤ ζ 2k , q 2k 2k k=1
k=1
and clearly this will follow if I show that
p p q 2k θ ≤ q 2k 2k
for each k ∈ Z+ .
But the choice of θ in (2.4.28) makes the preceding an equality when k = 1, and, when k ≥ 2, 2k p q 2k−1 Y j−q q 2k θ ≤ 1, ≤ p j−p 2k j=2
since 1 < p < q ≤ 2. At this point, I have proved (2.4.15) for 1 < p < q ≤ 2 and θ given by (2.4.28) when ζ ∈ (0, 1). Continuing with this choice of p, q, and θ, note that (2.4.15) extends immediately to ζ ∈ [−1, 1] by continuity and symmetry. Finally, for general ζ ∈ C, set a=
|1 − ζ| + |1 + ζ| , 2
b=
|1 − ζ| − |1 + ζ| , 2
and c =
b ∈ [−1, 1]. a
Then |1 ± θζ| = 1+θ 2 (1 ± ζ) +
1−θ 2 (1
∓ ζ) ≤ a ∓ θb,
and, therefore, by the preceding applied to c, we have that
1 1 |1 − θc|q + |1 + θc|q q |1 − θζ|q + |1 + θζ|q q ≤a 2 2 1 1 1 |1 − ζ|p + |1 + ζ|p p |a − b|p + |a + b|p p |1 − c|p + |1 + c|p p . = = ≤a 2 2 2
Hence, I have now completed the case when 1 < p < q ≤ 2 and θ is given by (2.4.28).
108
2 The Central Limit Theorem
To handle the other cases, I will use the equivalence of (2.4.16) and (2.4.17). Thus, what we already know is that (2.4.17) holds for 1 < p < q ≤ 2 and the θ in (2.4.28). Next, suppose that 2 ≤ p < q < ∞. Then, since 1 < q 0 < p0 ≤ 2 and
q0 − 1 p−1 , = 0 p −1 q−1
an application to q 0 and p0 of the result that we already have yields
Kθ ϕ q = sup Kθ ϕ, ψ L2 (β;C) : ψ ∈ L2 (β; C) with kψkLq0 (β) = 1 L (β;C) = sup ϕ, Kθ ψ 2 : ψ ∈ L2 (β; C) with kψkLq0 (β) = 1 L (β;C)
≤ kϕkLp (β;C) , where the θ is the one given in (2.4.28). Thus, the only case that remains is the 1 1 one when 1 < p ≤ 2 ≤ q < ∞. But, in this case, set ξ = (p − 1) 2 , η = (q − 1)− 2 , and observe that, because the associated θ in (2.4.28) is the product of ξ with η, Kθ = Kη ◦ Kξ and therefore
Kθ ϕ q ≤ Kξ ϕ L2 (β;C) ≤ kϕkLp (β;C) . L (β;C) As my second, and final, application of Theorem 2.4.14, I present the theorem of Beckner for which he concocted Theorem 2.4.14 in the first place. The result was originally by H. Weyl, who guessed, on the basis of Fh0 = √ conjectured 0 n ( −1) h0 , that the norm kFkp→p0 of F as an operator on Lp (R; C) to Lp (R; C) should be achieved by h0 . Weyl’s conjecture was partially verified by I. Babenko, who proved it when p0 is an even integer. In particular, when combined with the Riesz–Thorin Interpolation Theorem, Babenko’s result already shows (cf. Exercise 2.4.35) that kFkp→p0 < 1 for p ∈ (0, 1).
Theorem 2.4.29 (Beckner). For each p ∈ [1, 2], (2.4.30)
kFf kLp0 (R;C) ≤ Ap kf kLp (R;C) ,
f ∈ Lp (R; C) ∩ L2 (R; C),
where F is the Fourier operator in (2.4.9), A1 = 1, and Ap is the constant in 2 (2.4.13). Moreover, if f is the Gauss kernel e−πx , then (2.4.30) is an equality. Proof: Because of (2.4.11), the second part is a straightforward computation that I leave to the reader. Also, I will only consider (2.4.30) when p ∈ (1, 2), the other cases being well known (cf. Exercise 2.4.37). Because of (2.4.13), the proof of (2.4.30) comes down to showing that
Hθp ϕ p0 (2.4.31) ≤ kϕkLp (γ0,1 ;C) , ϕ ∈ Lp (γ0,1 ; C), L (γ ;C) 0,1
§ 2.4 An Application to Hermite Multipliers where θp =
√
109
1
−1 (p − 1) 2 . Indeed, by (2.4.13), (2.4.31) implies that
kFUp ϕkLp0 (R;C) ≤ Ap kϕkLp (γ0,1 ;C)
(2.4.32)
for all polynomials ϕ. Next, if ϕ ∈ L2 (γ0,1 ; C) and {ϕn : n ≥ 1} is a sequence of polynomials which tend to ϕ in L2 (γ0,1 ; C), then, because p ∈ (1, 2), it is easy to check that ϕn −→ ϕ in Lp (γ0,1 ; C) and Up ϕn −→ Up ϕ in L2 (R; C); and therefore, since F is a bounded on L2 (R; C), Fatou’s Lemma shows that (2.4.32) continues to hold for all ϕ ∈ L2 (γ0,1 ; C). Now let f ∈ Cc (R; C), and set ϕ = Up−1 f . Then, (2.4.32) implies that (2.4.30) holds for f . Finally, if f ∈ L2 (R; C) ∩ Lp (R; C), choose {fn : n ≥ 1} ⊆ Cc (R; C) so that fn −→ f in both L2 (R; C) and Lp (R; C), and conclude that (2.4.30) continues to hold. By Theorem 2.4.14, (2.4.31) will follow as soon as I prove (2.4.16) for θp . For this purpose, write √ 1 ζ = ξ + −1 (p − 1)− 2 η, where ξ, η ∈ R.
Then, because p0 − 1 = (p − 1)−1 , proving (2.4.16) for θp becomes the problem of checking that 1 h i p20 p0 i p20 h 2 2 2 + 1 + η + (p − 1)ξ 2 1 − η + (p − 1)ξ 2
(*) h
1−ξ
2
0
+ (p − 1)η
2
i p2
≤
+
h
1+ξ
2
0
+ (p − 1)η
2
i p2 p1
2
for all ξ, η ∈ R. To prove (*), consider, α ∈ (0, ∞), the function gα : [0, ∞)2 −→ [0, ∞) 1for each 1 α defined by gα (x, y) = x α +y α . It is an easy matter to check that gα is concave or convex depending on whether α ∈ [1, ∞) or α ∈ (0, 1). In particular, since p0 2
∈ (1, ∞), when we set α =
h
1−η
2
p0 2,
p0
0
+ (p − 1)ξ
2
i p20
+
h
1+η
2
+ (p − 1)ξ
2
i p20
2
gα x− , y + gα x+ , y x− + x+ ,y ≤ gα = 2 2
=
0
0
x± = |1 ± η|p , and y = (p − 1) 2 |ξ|p , we get
|1 − η|p + |1 + η|p 2
0
! 20 p
p20 + (p − 1)ξ 2
;
110
2 The Central Limit Theorem
and similarly, because
h
1−ξ
p 2
2
∈ (0, 1), 0
+ (p − 1)η
2
i p2
+
h
1+ξ
2
0
+ (p − 1)η
2
i p2
2 " ≥
|1 − ξ|p + |1 + ξ|p 2
# p2
p2
+ (p0 − 1)η 2
.
Thus, (*) will be proved once I show that 0
(**)
|1 − η|p + |1 + η|p 2
0
! 20 p
2
+(p−1)ξ ≤
|1 − ξ|p + |1 + ξ|p 2
p2
+(p0 −1)η 2 .
But because (cf. Theorems 2.4.14 and 2.4.25) we know that (2.4.16) holds with 1 p replaced by 2, q = p0 , and θ = p − 1 2 , the left side of (**) is dominated by 1
2
(p − 1)ξ +
1 − (p0 − 1) 2 η
2
1
+ 1 + (p0 − 1) 2 η 2
2
= 1 + (p − 1)ξ 2 + (p0 − 1)η 2 .
At the same time, again by (2.4.16), only this time with p, 2, and the same choice of θ, we see that the right-hand side of (**) dominates 1
0
2
(p − 1)η +
1 − (p − 1) 2 ξ
2
1
+ 1 + (p − 1) 2 ξ 2
2
= 1 + (p − 1)ξ 2 + (p0 − 1)η 2 .
Exercises for § 2.4 2 , let π1 and Exercise 2.4.33. Define S : R2 −→ R so that S(x1 , x2 ) = x1√+x 2 π2 be the natural projection maps given by πi (x1 , x2 ) = xi for i ∈ {1, 2}, and let λR denote Lebesgue measure on R. The goal of this exercise is to prove that if f : R −→ R is a Borel measurable function with the property that
(2.4.34)
f ◦S =
f ◦ π 1 + f ◦ π2 √ 2
λ2R -almost everywhere,
then there is an α ∈ R such that f (x) = αx for λR1 -almost every x ∈ R. Here are steps which one can take to prove this result. (i) After noticing that (2.4.34) holds when λR is replaced by γ0,1 , apply Exercise 2.3.21 to see that the γ0,1 -distribution of x f (x) is γ0,α for some α ∈ [0, ∞). Conclude, in particular, that f ∈ L2 (γ0,1 ; R).
Exercises for § 2.4
111
(ii) For each n ≥ 0, let Z (n) denote span {Hn ◦ π1 Hn−m ◦ π2 : 0 ≤ m ≤ n} . S∞ 2 Show that Z (m) ⊥ Z (n) in L2 (γ0,1 ; R) when m 6= n and the span of n=0 Z (n) 2 2 is dense in L2 (γ0,1 ; R). Conclude from these that if F ∈ L2 (γ0,1 ; R), then F = P∞ (n) and the series n=0 Πn F , where Πn denotes orthogonal projection onto Z convergences in L2 (γ0,1 ; R). (iii) Using the generating (2.4.5), show that n X n −n Hm ◦ π1 Hn−m ◦ π2 , Hn ◦ S = 2 2 m m=0
and use this to conclude that for any ϕ ∈ L2 (γ0,1 ; R), ϕ, Hn L2 (γ0,1 ;R) Hn ◦ S. Πn (ϕ ◦ S) = n!
(iv) Show that if ϕ ∈ L2 (γ0,1 ; R), then Πn
ϕ ◦ π 1 + ϕ ◦ π2 √ 2
ϕ, Hn
=
L2 (γ0,1 ;R) 1 2
2 n!
Hn ◦ π1 + Hn ◦ π2 .
(v) By combining (iii) and (iv), show that f, Hn L2 (γ ;R) f, Hn L2 (γ ;R) 0,1 0,1 Hn ◦ π1 + Hn ◦ π2 . (*) Hn ◦ S = 1 n! 2 2 n! From this, show that f, Hn L2 (γ0,1 ;R) = 0 unless n = 1. When n = 0, this is obvious. When n ≥ 2, one can argue that, if f, Hn L2 (γ0,1 ;R) 6= 0, then (*) implies that Hn0 ◦ π1 = Hn0 ◦ π2 , which is possible only if Hn0 is constant. Finally, P∞ 1 f, Hn L2 (γ0,1 ;R) Hn , it follows that since f = n=0 n!
f (x) = f, H1
Z
L2 (γ0,1 ;R)
H1 (x) =
ξf (ξ) γ0,1 (dξ) x
for γ0,1 -almost every x ∈ R. Exercise 2.4.35. Because the Fourier operator F (cf. (2.4.9)) is a contraction from L1 (R; C) to L∞ (R; C) as well as from L2 (R; C) into L2 (R; C), the Riesz– Thorin Interpolation Theorem guarantees that it is a contraction from Lp (R; C) 0 into Lp (R; C) for each p ∈ (0, 1). However, this is a case in which Riesz–Thorin gives a less than optimal result. Indeed, show that t ∈ 12 , 1 7−→ log A 1t ∈ R
is a strictly convex function that tends to 0 at both end points and is therefore strictly negative. Hence, Ap < 1 for p ∈ (1, 2).
112
2 The Central Limit Theorem
Exercise 2.4.36. The inequality in (2.4.8) is an example of a general principle. Namely, if (E, B) is any measurable space, then a map (x, Γ) ∈ E × B 7−→ Π(x, Γ) ∈ [0, 1] is called a transition probability whenever x ∈ E 7−→ Π(x, Γ) is B-measurable for each Γ ∈ B and Γ ∈ B 7−→ Π(x, Γ) is a probability measure on (E, B) for each x ∈ E. Given a transition probability Π(x, · ), define the linear operator Π on B(E; C) (the space of bounded, B-measurable ϕ : E −→ C) by Z Πϕ (x) = ϕ(y) Π(x, dy), x ∈ E, for ϕ ∈ B(E; C). E
Check that Π takes B(E; C) into itself and that kΠϕku ≤ kϕku . Next, given a σ-finite measure µ on (E, B), say that µ is Π-invariant if Z µ(Γ) = Π(x, Γ) µ(dx) for all Γ ∈ B. E
Using Jensen’s Inequality, first show that, for each p ∈ [1, ∞), p Πϕ (x) ≤ Π|ϕ|p (x), x ∈ E, and then that, for any Π-invariant µ, kΠϕkLp (µ;C) ≤ kϕkLp (µ;C) ,
ϕ ∈ B(E; C).
Finally, show that µ is Π-invariant if it is Π-reversing in the sense that Z Z Π x, Γ2 µ(dx) = Π y, Γ1 µ(dy) for all Γ1 , Γ2 ∈ B. Γ1
Γ2
Exercise 2.4.37. Recall the Hermite functions hn , n ∈ N, in (2.4.12) and define the normalized Hermite functions hn , n ∈ N by 1
hn =
24 1
(n!) 2
hn ,
n ∈ N.
By noting that (cf. the discussion following (2.4.12)) hn = U2 H n , show that hn : n ∈ N constitutes an orthonormal basis in L2 (R; C), and from this together with (2.4.11), arrive at Parseval’s Identity:
kFf kL2 (R;C) = kf kL2 (R;C) ,
f ∈ L1 (R; C) ∩ L2 (R; C),
and conclude that F determines a unique unitary operator F on L2 (R; C) such that Ff = Ff for f ∈ L1 (R; C) ∩ L2 (R; C). Finally, use this to verify the L2 −1 ˜ (x) ≡ Ff (−x), x ∈ R, for ˜ where Ff = F, Fourier inversion formula F f ∈ L1 (R; C) ∩ L2 (R; C).
Exercises for § 2.4
113
Exercise 2.4.38. By the same reasoning as I used to prove Theorem 2.4.29, show√ that, for any pair 1 < p ≤ 2 ≤ q < ∞ and any complex number θ = ξ + −1 η, (2.4.16) and therefore (2.4.15) hold if both (q − 1)η 2 + ξ 2 ≤ 1 and (q − 2)(ξη)2 ≤ 1 − ξ 2 − (q − 1)η 2 (p − 1) − (q − 1)α2 − β 2 .
Exercise 2.4.39. L. Gross had a somewhat different approach to the proof of (2.4.26). As in the proof that I have given, he reduced everything to checking (2.4.17). However, he did this in a different way. Namely, given b ∈ (0, 1), he set f (x) = 1 + bx and introduced the functions −t −t ft (x) ≡ Ke−t f (x) = 1+e2 f (x) + 1−e2 f (−x), (t, x) ∈ [0, ∞) × R,
and q(t) = 1 + (p − 1)e2t , t ∈ [0, ∞), and proved that
d
ft q(t) ≤ 0. (*) L (β;C) dt Following the steps below, see if you can reproduce Gross’s calculation.
(i) Set F (t) = kft kLq(t) (β;C) , and, by somewhat tedious but completely elementary differential calculus, show that Z q(t) F (t)1−q(t) ft q(t) dF dβ − q(t) ˙ f log F (t) dt (t) = q(t)2 R Z q(t)2 q(t)−1 ft (x) ft (−x) − ft (x) β(dx) . + 2 R
Next, check that Z ft (x)q(t)−1 ft (−x) − ft (x) β(dx) R Z 1 ft (x)q(t)−1 − ft (−x)q(t)−1 ft (x) − ft (−x) β(dx), = −2 R
and, after verifying that q
ξ
q−1
−η
q−1
q
4(q − 1) ξ 2 − η 2 (ξ − η) ≥ q2
2
ξ, η ∈ (0, ∞) and q ∈ (1, ∞),
,
conclude that dF dt
(**)
Z q(t) F (t)1−q(t) ft q(t) dβ − q(t) ˙ f log (t) ≤ F (t) q(t)2 R Z q(t) 2 q(t) 2 + q(t) − 1 ft (x) 2 − ft (−x) β(dx) . R
(ii) Prove the Logarithmic Sobolev Inequality Z Z 2 2 ϕ 2 dβ ≤ 2 ϕ(x) − ϕ(−x) β(dx) (2.4.40) ϕ log kϕk 2 R
for strictly positive ϕ’s on R.
L (β;C)
R
114
2 The Central Limit Theorem
Hint: Reduce to the case when ϕ(x) = 1 + bx for some b ∈ (0, 1), and, in this case, check that (2.4.40) is the elementary calculus inequality (1 + b)2 log(1 + b) + (1 − b)2 log(1 − b) − (1 + b2 ) log(1 + b2 ) ≤ 2b2 ,
b ∈ (0, 1).
(iii) By plugging (2.4.40) into (**), arrive at (*), and conclude that (2.4.17) holds for θ ∈ (0, 1) and q = 1 + p−1 θ2 .
Exercise 2.4.41. The major difference between Gross’s and Beckner’s approaches to proving Nelson’s Theorem 2.4.25 is that Gross based his proof on the equivalence of contraction results like (2.4.17) and (2.4.15) to Logarithmic Sobolev Inequalities like (2.4.40). In Exercise 2.4.38, I outlined how one passes from a Logarithmic Sobolev Inequality to a contraction result. The object of this exercise is to go in the opposite direction. Specifically, starting from (2.4.26), show that Z (2.4.42)
2
ϕ log R
2
ϕ
kϕkL2 (γ
0,1 ;C)
Z dγ0,1 ≤ 2
|ϕ0 |2 γ0,1 (dx)
R
for non-negative, continuously differentiable ϕ ∈ L2 (γ0,1 ; C) \ {0} with ϕ0 ∈ L2 (γ0,1 ; C). See Exercise 8.4.8 for another derivation. Exercise 2.4.43. As an application of Theorem 2.4.25, show that kHn kLp (γ0,1 ;C) ≤
p
n!(p − 1) for n ∈ N and p ∈ [2, ∞).
To see that this estimate is quite good, show that kH1 kpLp (γ0,1 ;C) =
p
22 1 π2
Γ
and apply Stirling’s formula (1.3.21) to conclude that kH1 kLp (γ0,1 ;C) ∼ as p → ∞.
p+1 2
p1
, 1 p−1 2 e
Chapter 3 Infinitely Divisible Laws
The results in this chapter are an attempt to answer the following question. GivenPan RN -valued random variable Y with the property that, for each n ∈ Z+ , n Y = m=1 Xm , where X1 , . . . , Xn are independent and identically distributed, what can one say about the distribution of Y? Recall that the convolution ν1 ? ν2 of two finite Borel measures ν1 and ν2 on RN is given by ZZ ν1 ? ν2 (Γ) = 1Γ (x + y) ν1 (dx)ν2 (dy), Γ ∈ BRN , RN ×RN
and that the distribution of the sum of two independent random variables is the convolution of their distributions. Thus, the analytic statement of our problem is that of describing those probability measures µ that, for each n ≥ 1, can be of some probability measure µ n1 . written as the n-fold convolution power µ?n 1 n
I will say that such a µ is infinitely divisible and will use I(RN ) to denote the class of infinitely divisible measures on RN . Since the Fourier transform takes convolution into ordinary multiplication, the Fourier formulation of this problem is that of describing those Borel probability measures on RN whose Fourier transform µ ˆ has, for each n ∈ Z+ , an nth root which is again the Fourier transform of a Borel probability measure on RN . Not surprisingly, the Fourier formulation of the problem is, in many ways, the most amenable to analysis, and it is the formulation in terms of which I will solve it in this chapter. On the other hand, this formulation has the disadvantage that, although it yields a quite satisfactory description of µ ˆ, it leaves the problem of extracting information about µ from properties of µ ˆ. For this reason, the following chapter will be devoted to developing a probabilistic understanding of the analytic answer obtained in this chapter. § 3.1 Convergence of Measures on RN In order to carry out our program, I will need two important facts about the convergence of probability measures on RN . The first of these is a minor modification of the classical Helly–Bray Theorem, and the second is an improvement, due to L´evy, of Lemma 2.3.3. 115
116
3 Infinitely Divisible Laws
Say that the sequence {µn : n ≥ 1} ⊆ M1 (RN ) converges weakly to µ ∈ M1 (RN ) and write µn =⇒ µ when hϕ, µn i −→ hϕ, µi for all ϕ ∈ Cb (RN ; C), and apply Lemma 2.3.3 to check that µn =⇒µ if and only if µ cn (ξ) −→ µ ˆ(ξ) for every N ξ∈R . § 3.1.1. Sequential Compactness in M1 (RN ). Given a subset S of M1 (RN ), I will say that S is sequentially relatively compact if, for every sequence {µn : n ≥ 1} ⊆ S, there a subsequence {µnm : m ≥ 1} and a µ ∈ M1 (RN ) such that µnm =⇒ µ. Theorem 3.1.1. and only if
A subset S of M1 (RN ) is sequentially relatively compact if
lim sup µ B(0, R){ = 0.
(3.1.2)
R→∞ µ∈S
Proof: I begin by pointing out that there is a countable set {ϕk : k ∈ Z+ } ⊆ Cc (RN ; R) of linear independent functions whose span is dense, with respect to uniform convergence, in Cc (RN ; R). To see this, choose η ∈ Cc RN ; [0, 1] so that η = 1 on B(0, 1) and 0 off B(0, 2), and set ηR (y) = η(R−1 y) for R > 0. Next, for each ` ∈ Z+ , apply the Stone–Weierstrass Theorem to choose a countable dense subset {ψj,` : j ∈ Z+ } of C B(0, 2`); R , and set ϕj,` = η` ψj,` . Clearly {ϕj,` : (j, `) ∈ (Z+ )2 } is dense in Cc (RN ; R). Finally, using lexicographic ordering of (Z+ )2 , extract a linearly independent subset {ϕk : k ∈ Z+ } by taking ϕk = ϕjk ,`k , where (j1 , `1 ) = (1, 1) and (jk+1 , `k+1 ) is the first (j, `) such that ϕj,` is linearly independent of {ϕ1 , . . . , ϕk }. Given a sequence {µn : n ≥ 1} ⊆ S, we can use a diagonalization procedure to find a subsequence {µnm : m ≥ 1} such that ak = limm→∞ hϕk , µnm i exists for every k ∈ Z+ . Next, define the linear functional Λ on the span of {ϕk : k ∈ Z+ } PK so that Λ(ϕk ) = ak . Notice that if ϕ = k=1 αk ϕk , then
K X αk hϕk , µnm i = lim hϕ, µnm i ≤ kϕku , m→∞ m→∞
Λ(ϕ) = lim
k=1
and similarly that Λ(ϕ) = limm→∞ hϕ, µnm i ≥ 0 if ϕ ≥ 0. Hence, Λ admits a unique extension as a non-negativity preserving linear functional on Cc (RN ; R) that satisfies |Λ(ϕ)| ≤ kϕku for all ϕ ∈ Cc (RN ; R). Now assume that (3.1.2) holds. For each ` ∈ Z+ , apply the Riesz Representation Theorem to produce a non-negative Borel measure ν` supported on B(0, 2`) so that hϕ, ν` i = Λ(η` ϕ) for ϕ ∈ Cc (RN ; R). Since hϕ, ν`+1 i = Λ(ϕ) = hϕ, ν` i whenever ϕ vanishes off of B(0, `), it is clear that
ν`+1 Γ ∩ B(0, ` + 1) ≥ ν`+1 Γ ∩ B(0, `) = ν` Γ ∩ B(0, `)
for all Γ ∈ BRN .
§ 3.1 Convergence of Measures on RN
117
Hence, if ∞ X µ(Γ) ≡ lim µ` Γ ∩ B(0, `) = µ` Γ ∩ B(0, `) \ B(0, ` − 1) , `→∞
`=1
then µ is a non-negative Borel measure on RN whose restriction to B(0, `) is ν` for each ` ∈ Z+ . In particular, µ(RN ) ≤ 1 and hϕ, µi = limm→∞ hϕ, µnm i for every ϕ ∈ Cc (RN ; R). Thus, by Lemma 2.1.7, all that remains is to check that µ(RN ) = 1. But µ(RN ) ≥ hη` , µi = lim hη` , µnm i ≥ lim µnm B(0, `) m→∞ m→∞ = 1 − lim µnm B(0, `){ ,
m→∞
and, by (3.1.2), the final term tends to 0 as ` → ∞. To prove the converse assertion, suppose that S is sequentially relatively compact. If (3.1.2) failed, then we could find an θ ∈ (0, 1) and, for each n ∈ Z+ , a µn ∈ S such that µn B(0, n) ≤ θ. By sequential relative compactness, this would mean that there is a subsequence {µnm : m ≥ 1} ⊆ S and a µ ∈ M1 (RN ) such that µnm =⇒ µ and µnm B(0, nm ) ≤ θ. On the other hand, for any R > 0, µ B(0, R) ≤ hηR , µi ≤ lim µnm B(0, nm ) ≤ θ, m→∞
and so we would arrive at the contradiction 1 = limR→∞ µ B(0, R) ≤ θ. § 3.1.2. L´ evy’s Continuity Theorem. My next goal is to find a test in terms of the Fourier transform to determine when (3.1.2) holds. Lemma 3.1.3. Define 1−
s(r) = inf
θ≥r
sin θ θ
for r ∈ (0, ∞).
Then s is a strictly positive, non-decreasing, continuous function that tends to 0 as r & 0. Moreover, if µ ∈ M1 (RN ), then, for all (r, R) ∈ (0, ∞)2 , (3.1.4)
1 − µ ˆ(re) ≤ rR + 2µ {y : |(e, y)RN | ≥ R} for all e ∈ SN −1 ,
and
1 µ B(0, N 2 R){ ≤ N sup µ {y : |(e, y)RN | ≥ R} e∈SN −1
(3.1.5)
≤
N max 1 − µ ˆ(ξ) : |ξ| ≤ r . s(rR)
118
3 Infinitely Divisible Laws
In particular, for any S ⊆ M1 (RN ), (3.1.2) holds if and only if lim sup 1 − µ ˆ(ξ) = 0.
(3.1.6)
|ξ|&0 µ∈S
Proof: Given (3.1.4) and (3.1.5), the final assertion is obvious. √ (3.1.4), simply observe that 1 − e −1(re,y)RN ≤ 2 ∧ r|(e, y)RN | . Turning to (3.1.5), note that
1 − µ ˆ(ξ) ≥
Z
RN
To prove
1 − cos(ξ, y)RN µ(dy).
Thus, for each e ∈ SN −1 , 1 r
r
Z
1 − µ ˆ(te) dt ≥
Z RN \{0}
0
sin r(e, y)RN 1− r(e, y)RN
!
µ(dy)
≥ s(rR)µ {y : |(e, y)RN | ≥ R} , and therefore (3.1.7)
ˆ(ξ) ≥ s(rR)µ {y : |(e, y)RN | ≥ R} . sup 1 − µ ξ∈B(0,r)
Since the first inequality in (3.1.5) is obvious, there is nothing more to be done. I am now ready to prove L´evy’s crucial improvement to Lemma 2.3.3. Theorem 3.1.8 (L´ evy’s Continuity Theorem). Let {µn : n ≥ 1} ⊆ M1 (RN ), and assume that f (ξ) = limn→∞ µ ˆn (ξ) exists for each ξ ∈ RN . Then N there is a µ ∈ M 1 (R ) such that ˆ if and only if there is a δ > 0 for which f =µ limn→∞ sup|ξ|≤δ µ ˆn (ξ) − f (ξ) = 0, in which case µn =⇒ µ. (See part (iv) of Exercise 3.1.9 for another version.) Proof: The only assertion not already covered by Lemmas 2.1.7 and 2.3.3 is the “if” part of the equivalence. But, if µ ˆn −→ f uniformly in a neighborhood of 0, then it is easy to check that supn≥1 |1 − µ ˆn (ξ)| must tend to zero as |ξ| → 0. Hence, by the last part of Lemma 3.1.3 and Theorem 3.1.1, we know that there exists a µ and a subsequence {µnm : m ≥ 1} such that µnm =⇒ µ. Since µ ˆ must equal f , Lemma 2.3.3 says that µn =⇒ µ.
Exercises for § 3.1
119
Exercises for § 3.1 Exercise 3.1.9. One might think that to address the sort of problem posed at the beginning of this chapter, it would be helpful to know which functions f : RN −→ C are the Fourier transforms of a probability measure. Such a characterization is the content of Bochner’s Theorem, whose proof will be outlined in this exercise. Unfortunately, his characterization looks more useful than it is in practice. For instance, I will not use it to solve our problem, and it is difficult to see how its use would simplify matters. In order to state Bochner’s Theorem, say that a function f : RN −→ C is N non-negative definite if, for each n ≥ 1 and ξ1 , . . . , ξn ∈ R , the matrix f (ξi − ξj ) 1≤i,j≤n is Hermitian and non-negative definite. Equivalently,1 n X
f (ξi − ξj )ζi ζ¯j ≥ 0
for all ζ1 , . . . , ζn ∈ C.
i,j=1
Then Bochner’s Theorem is the statement that f = µ ˆ for some µ ∈ M1 (RN ) if and only if f (0) = 1 and f is a continuous, non-negative definite function. (i) It is ironic that the necessity assertion is the more useful even though it is nearly trivial. Indeed, if f = µ ˆ, then it is obvious that f (0) = 1 and that f is continuous. To see that it is also non-negative definite, write n X
e
√ −1(ξi −ξj ,x)RN
2 n X √ −1(ξ ,x) i RN ζ , ζi ζ¯j = e i i=1
i,j=1
and integrate in x with respect to µ. (ii) The first step in proving the sufficiency is to use the non-negative definiteness assumption to show that f (−x) = f (x) and f (x) ≤ f (0) for all x ∈ RN . Obviously, this proves that kf ku ≤ 1. Second, using a standard Riemann approximation procedure and the continuity of f , check that, for any rapidly decreasing, continuous ψˆ : RN −→ C, ZZ ˆ ˆ ψ(η) dx dη ≥ 0. f (x − η)ψ(x) RN ×RN
In particular, when f ∈ L1 RN ; C , set −N
Z
m(x) = (2π)
e−
√
−1 (x,ξ)RN
f (ξ) dξ,
RN 1
Recall that a non-negative definite operator on a complex Hilbert space is always Hermitian.
120
3 Infinitely Divisible Laws
and use Parseval’s Identity and Fubini’s Theorem, together with elementary manipulations, to arrive at Z ZZ ˆ ˆ ψ(η) dξ dη ≥ 0 (2π)N m(x) ψ(x)2 dx = f (ξ − η)ψ(ξ) RN
RN ×RN
for all ψ ∈ L1 (RN ; R) ∩ Cb (RN ; R) with ψˆ ∈ L1 (RN ; R). Conclude that m is non negative, and use this to complete the proof in the case when f ∈ L1 RN ; C . (iii) It remains only to pass from the case when f ∈ L1 RN ; C to the general |x|2
case. For each t ∈ (0, ∞), set ft (x) = e−t 2 f (x). Clearly, ft (0) = 1 and ft ∈ Cb (RN ; C) ∩ L1 (RN ; C). In addition, show that Z n n X X ft ξi − ξj ζi ζ¯j = f ξi − ξj ζi (x)ζ¯j (x) γ0,tI (dx) ≥ 0, RN
i,j=1
i,j=1
√
where ζi (x) ≡ ζi e −1 (ξi ,x)RN . Hence, ft is also non-negative definite, and so, by part (ii), we know thatft = µbt for some µt ∈ M1 (RN ). Finally, apply L´evy’s Continuity Theorem to see that µt =⇒µ, where µ ∈ M1 (RN ) satisfies f = µ ˆ.
(iv) Let {µn : n ≥ 1} and f be as in Theorem 3.1.8. Combining Bochner’s Theorem with Lemma 2.1.7, show that there exists a µ ∈ M1 (RN ) such that f =µ ˆ and µn =⇒ µ if and only if f is continuous. Exercise 3.1.10. Suppose that f is a non-negative definite function with f (0) = 1. As we have just seen, if f is continuous, then f = µ ˆ for some µ ∈ M1 (RN ). (i) Assuming that f = µ ˆ, show that (*)
kf ku ≤ 1
and |f (η) − f (ξ)|2 ≤ 2 1 − Re f (η − ξ) ,
ξ, η ∈ RN .
Next, show that (*) follows directly from non-negative definiteness, whether or not f is continuous. Thus, a non-negative definite function is uniformly continuous everywhere if it is continuous at the origin. Hint: Both parts of (*) follow from the fact that f (η) f (ξ) 1 1 f (ξ − η) A = f (ξ)
f (η)
f (ξ − η)
1
is non-negative definite. To get the second part, consider the quadratic form v, Av C3 with v = (v1 , 1, −1).2 2
This choice of v was suggested to me by Linan Chen.
Exercises for § 3.1
121
(ii) To understand how essential a role continuity plays in Bochner’s criterion, show that f = 1{0} is non-negative definite. Even though this f cannot be the Fourier transform of any µ ∈ M1 (RN ), it is nonetheless the “Fourier transform” of a non-negativity preserving linear functional, one for which there is no Riesz representation. To be more precise, consider the linear functional Λ on the space of functions ϕ ∈ Cb (RN ; C) for which Z 1 ϕ(x) dx exists, Λϕ ≡ lim R→∞ |B(0, R)| B(0,R) √
and show that f (ξ) = Λ(eξ ), where eξ (x) = e
−1(ξ,x)RN
.
Exercise 3.1.11. It is important to recognize the extent to which L´evy’s Continuity Theorem and, as a by-product, Bochner’s Theorem, are strictly finite dimensional results. For example, let H be an infinite dimensional, separa2 1 ble, real Hilbert space, and define f (h) = e− 2 khkH . Obviously, f is a continuous and f (0)= 1. Show that it is also non-negative definite in the sense that f (hi − hj ) 1≤i,j≤n is a non-negative definite, Hermitian matrix for each n ∈ Z+ and h1 , . . . , hn ∈ H. Now suppose that there were a Borel probability measure µ on H such that Z √ µ ˆ(h) ≡ e −1(h,x)H µ(dx) = f (h), h ∈ H. H
Show that, for any orthonormal basis {ei : i ∈ Z+ } in H, the functions Xi (h) = (ei , h)H , i ∈ Z+ , would be, under µ, a sequence of independent, N (0, 1)-random variables, and conclude from this that Z Y 2 2 e−khkH µ(dh) = Eµ e−Xi = 0. H
i∈Z+
Hence, no such µ can exist. See Chapter 8 for a much more thorough account of this topic. Hint: The non-negative definiteness of f can be seen as a consequence of the analogous result for Rn . Exercise 3.1.12. The Riemann–Lebesgue Lemma says that fˆ(ξ) −→ 0 as |ξ| → ∞ if f ∈ L1 (RN ; C). Thus µ ˆ(ξ) −→ 0 as |ξ| → ∞ if µ ∈ M1 (R) is absolutely continuous. In this exercise we will examine situations in which µ ∈ M1 (R) but µ ˆ(ξ)−→ 6 0 as |ξ| → ∞. (i) Given a symmetric µ ∈ M1 (R), show that µ ˆ is real valued, and use Bochner’s Theorem to show that µ ˆ(ξ) cannot tend to a strictly negative number as |ξ| → ∞. Hint: Let α > 0, and suppose that µ ˆ(ξ) −→ −2α as |ξ| → ∞. Choose R > 0 so that µ ˆ(ξ) ≤ −α for |ξ| ≥ R and n ∈ Z+ so that (n − 1)α > 1. Set A = µ ˆ(`R − kR) 1≤k,`≤n , and show that A cannot be non-negative definite.
122
3 Infinitely Divisible Laws
(ii) Show that µ ˆ(ξ)−→ 6 0 if µ has an atom (i.e., µ({x}) > 0 for some x ∈ R). Hint: Reduce to the case in which µ is symmetric, and therefore that µ = pδ0 + qν, where p ∈ (0, 1], q = 1 − p, and ν ∈ M1 (R) is symmetric. If p = 1, µ ˆ(ξ) = 1 for all ξ. If p ∈ (0, 1), then µ ˆ(ξ) −→ 0 as |ξ| → ∞ implies νˆ(ξ) −→ − pq < 0.
(iii) To produce an example that is non-atomic, refer to Exercise 1.4.29, take p ∈ (0, 1) \ { 12 }, and let µ = µp , where µp is the measure described in that exercise. Show that µ is a non-atomic element of M1 (R) for which µ ˆ−→ 6 0 as |ξ| → ∞. Hint: Show that µ ˆ never vanishes and that µ ˆ(2m π) is independent of m ∈ Z+ .
§ 3.2 The L´ evy–Khinchine Formula Throughout, I(R ) will be the set of µ ∈ M1 (RN ) that are infinitely divisible. My strategy for characterizing I(RN ) will be to start from an easily understood subset of I(RN ) and to get the rest by taking weak limits. The elements of I(RN ) that first come to mind are the Gaussian measures (cf. (2.3.6)) γm,C . Indeed, if m ∈ RN and C is a symmetric, non-negative definite transformation on RN , then it is clear from (2.3.7) that γm,C = γ ?n m C. n ,n Unfortunately, this is not a good starting place because it is too rigid: limits of Gaussians are again Gaussian. Indeed, suppose that γmn ,Cn =⇒ µ. Then N
√
e
−1 (ξ,mn )RN − 12 (ξ,Cn ξ)RN
−→ µ ˆ(ξ)
for all ξ ∈ RN ,
and so µ = γm,C , where m = limn→∞ mn and C = limn→∞ Cn . In other words, one cannot use weak convergence to escape the class of Gaussian measures. A more fruitful choice is to start with the Poisson measures. Recall that if ν is a probability measure on RN and α ∈ [0, ∞), then the Poisson measure with jump distribution ν and jumping rate α (see § 4.2 for an explanation of this terminology) is the measure πα,ν = e−α
∞ X αn ?n ν . n! n=0
To see that πα,ν is infinitely divisible, note that Z √ −1 (ξ,y)RN − 1 ν(dy) , πd (ξ) = exp α e α,ν
. To see why the Poisson measures provide a and therefore that πα,ν = π ?n α n ,ν more hopeful choice of starting point, let m ∈ RN and a non-negative definite, symmetric C be given, and choose (e1 , . . . , eN ) to be p an orthonormal basis of eigenvectors for C. Next, set mi = (m, ei )RN and σi = (ei , Cei )RN , and take ! N N X 1X 1 . δ σi ei + δ− σ√i ei δ mi ei + νn = n 2 i=1 √n 2N i=1 n
§ 3.2 The L´evy–Khinchine Formula
123
Then the Fourier transform of π2N n,νn is exp
N X
√
n e
−1mi (ξ,ei ) N R n
i=1
! N X σi (ξ, ei )RN −1 , −1 + n cos 1 n2 i=1
which tends to γ[ m,C (ξ) as n → ∞, and so π2N n,νn =⇒ γm,C as n → ∞. Thus, one can use weak convergence to break out to the class of Poisson measures. As I will show in the next subsection, the preceding is a special case of a result (cf. Theorem 3.2.7) that says that every infinitely divisible measure is the weak limit of Poisson measures. However, before proving that result, it will be convenient to alter our description of Poisson measures. For one thing, it should be clear that, without loss in generality, I may always assume that the jump distribution ν assigns no mass to 0. Indeed, if ν({0}) = 1, then πα,ν = δ0 = π0,ν 0 no matter how α and ν 0 are chosen. If β = ν({0}) ∈ (0, 1), then πα,ν = πα0 ,ν 0 , where α0 = α(1 − β) and ν 0 = (1 − β)−1 (ν − βδ0 ). In addition, although the segregation of the rate and jumping distribution provides probabilistic insight, there is no essential reason for doing so. Thus, nothing is lost if one replaces πα,ν by πM , where M is the finite measure αµ, in which case √
Z π d M (ξ) = exp
e
−1(ξ,y)RN
− 1 M (dy) .
With these considerations in mind, let M0 (RN ) be the space of non-negative, finite Borel measures M on RN with M ({0}) = 0, and set P(RN ) = {πM : M ∈ M0 (RN )}, the space of Poisson measures on RN . § 3.2.1. I(RN ) Is the Closure of P(RN ). Let