B
PROBABILITY AND RANDOM PROCESSES
WILEY SURVIVAL GUIDES IN ENGINEERING AND SCIENCE Emmanuel Desurvire, Editor Wiley Survival Guide in Global Telecommunications: Signaling Principles, Network Protocols, and Wireless Systems Emmanuel Desurvire Wiley Survival Guide in Global Telecommunications: Broadband Access, Optical Components and Networks, and Cryptography Emmanuel Desurvire
Fiber to the Home: The New Empowerment Paul E. Green, Jr. Probability and Random Processes Venkatarama Krishnan
B PROBABILITY AND RANDOM PROCESSES
Venkatarama Krishnan Professor Emeritus of Electrical Engineering University of Massachusetts Lowell
A John Wiley & Sons, Inc., Publication
Copyright # 2006 by John Wiley & Sons, Inc. All rights reserved Published by John Wiley & Sons, Inc., Hoboken, New Jersey Published simultaneously in Canada No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permission. Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com. Library of Congress Cataloging-in-Publication Data: Krishnan, Venkatarama, Probability and random processes / by Venkatarama Krishnan. p. cm. — (Wiley survival guides in engineering and science) Includes bibliographical references and index. ISBN-13: 978-0-471-70354-9 (acid-free paper) ISBN-10: 0-471-70354-0 (acid-free paper) 1. Probabilities. 2. Stochastic processes. 3. Engineering—Statistical methods. 4. Science—Statistical methods. I. Title. II. Series QA273.K74 2006 519.2–dc22
2005057726
Printed in the United States of America 10 9 8 7
6 5 4 3
2 1
Contents
Preface, xi
4.6 Hypergeometric Distribution, 46 4.7 Poisson Distribution, 48
CHAPTER 1 Sets, Fields, and Events, 1 1.1
Set Definitions, 1
1.2
Set Operations, 3
1.3
Set Algebras, Fields, and Events, 8
4.8 Logarithmic Distribution, 55 4.9 Summary of Discrete Distributions, 62
CHAPTER 5 Random Variables, 64 5.1 Definition of Random Variables, 64
CHAPTER 2 Probability Space and Axioms, 10 2.1
Probability Space, 10
2.2
Conditional Probability, 14
2.3
Independence, 18
2.4
Total Probability and Bayes’ Theorem, 20
5.2 Determination of Distribution and
Density Functions, 66 5.3 Properties of Distribution and Density
Functions, 73 5.4 Distribution Functions from Density
Functions, 75 CHAPTER 6
CHAPTER 3 Basic Combinatorics, 25
Continuous Random Variables and Basic Distributions, 79
3.1
Basic Counting Principles, 25
6.1 Introduction, 79
3.2
Permutations, 26
6.2 Uniform Distribution, 79
3.3
Combinations, 28
6.3 Exponential Distribution, 80
CHAPTER 4 Discrete Distributions, 37
6.4 Normal or Gaussian Distribution, 84
CHAPTER 7
4.1
Bernoulli Trials, 37
Other Continuous Distributions, 95
4.2
Binomial Distribution, 38
7.1 Introduction, 95
4.3
Multinomial Distribution, 41
7.2 Triangular Distribution, 95
4.4
Geometric Distribution, 42
7.3 Laplace Distribution, 96
4.5
Negative Binomial Distribution, 44
7.4 Erlang Distribution, 97
vii
viii
Contents
7.5 Gamma Distribution, 99
10.4
Higher-Order Moments, 153
7.6 Weibull Distribution, 101
10.5
Bivariate Gaussian, 154
7.7 Chi-Square Distribution, 102 7.8 Chi and Other Allied Distributions,
104 7.9 Student-t Density, 110 7.10 Snedecor F Distribution, 111
CHAPTER 11 Characteristic Functions and Generating Functions, 155 11.1
Characteristic Functions, 155
11.2
Examples of Characteristic Functions, 157
11.3
Generating Functions, 161
11.4
Examples of Generating Functions, 162
11.5
Moment Generating Functions, 164
11.6
Cumulant Generating Functions, 167
11.7
Table of Means and Variances, 170
7.11 Lognormal Distribution, 112 7.12 Beta Distribution, 114 7.13 Cauchy Distribution, 115 7.14 Pareto Distribution, 117 7.15 Gibbs Distribution, 118 7.16 Mixed Distributions, 118 7.17 Summary of Distributions of
Continuous Random Variables, 119
CHAPTER 12 Functions of a Single Random Variable, 173
CHAPTER 8
12.1
Random Variable g(X ), 173
12.2
Distribution of Y ¼ g(X ), 174
12.3
Direct Determination of Density fY ( y) from fX (x), 194
12.4
Inverse Problem: Finding g(x) Given fX(x) and fY ( y), 200
12.5
Moments of a Function of a Random Variable, 202
Conditional Densities and Distributions, 122 8.1 Conditional Distribution and Density
for P(A)= 0, 122 8.2 Conditional Distribution and Density
for P(A) ¼ 0, 126 8.3 Total Probability and Bayes’ Theorem
for Densities, 131 CHAPTER 9 Joint Densities and Distributions, 135
CHAPTER 13
9.1 Joint Discrete Distribution
Functions of Multiple Random Variables, 206
Functions, 135
13.1
Function of Two Random Variables, Z ¼ g(X,Y ), 206
13.2
Two Functions of Two Random Variables, Z ¼ g(X,Y ), W ¼ h(X,Y ), 222
13.3
Direct Determination of Joint Density fZW(z,w ) from fXY (x,y ), 227
13.4
Solving Z ¼ g(X,Y ) Using an Auxiliary Random Variable, 233
13.5
Multiple Functions of Random Variables, 238
9.2 Joint Continuous Distribution
Functions, 136 9.3 Bivariate Gaussian Distributions, 144
CHAPTER 10 Moments and Conditional Moments, 146 10.1 Expectations, 146 10.2 Variance, 149 10.3 Means and Variances of Some
Distributions, 150
Contents
CHAPTER 14 Inequalities, Convergences, and Limit Theorems, 241 14.1
Degenerate Random Variables, 241
14.2
Chebyshev and Allied Inequalities, 242
14.3
Markov Inequality, 246
14.4
Chernoff Bound, 248
14.5
Cauchy–Schwartz Inequality, 251
14.6
Jensen’s Inequality, 254
14.7
Convergence Concepts, 256
14.8
Limit Theorems, 259
17.4 Diagonalization of
Covariance Matrices, 330 17.5 Simultaneous Diagonalization of
Covariance Matrices, 334 17.6 Linear Estimation of Vector
Variables, 337 CHAPTER 18 Estimation Theory, 340 18.1 Criteria of Estimators, 340 18.2 Estimation of Random
Variables, 342 18.3 Estimation of Parameters (Point
Estimation), 350 CHAPTER 15 Computer Methods for Generating Random Variates, 264
18.4 Interval Estimation (Confidence
Intervals), 364 18.5 Hypothesis Testing (Binary), 373
15.1
Uniform-Distribution Random Variates, 264
15.2
Histograms, 266
15.3
Inverse Transformation Techniques, 269
Random Processes, 406
15.4
Convolution Techniques, 279
19.1 Basic Definitions, 406
15.5
Acceptance –Rejection Techniques, 280
19.2 Stationary Random Processes, 420
CHAPTER 16 Elements of Matrix Algebra, 284 16.1
Basic Theory of Matrices, 284
16.2
Eigenvalues and Eigenvectors of Matrices, 293
16.3 16.4
Vectors and Matrix Differentiations, 301 Block Matrices, 308
18.6 Bayesian Estimation, 384
CHAPTER 19
19.3 Ergodic Processes, 439 19.4 Estimation of Parameters of
Random Processes, 445 19.5 Power Spectral Density, 472
CHAPTER 20 Classification of Random Processes, 490 20.1 Specifications of Random
Processes, 490 20.2 Poisson Process, 492
CHAPTER 17 Random Vectors and Mean-Square Estimation, 311
20.3 Binomial Process, 505 20.4 Independent Increment Process, 507
17.1
Distributions and Densities, 311
20.5 Random-Walk Process, 512
17.2
Moments of Random Vectors, 319
20.6 Gaussian Process, 521
17.3
Vector Gaussian Random Variables, 323
20.7 Wiener Process (Brownian
Motion), 523
ix
x
Contents
20.8 Markov Process, 527
CHAPTER 23
20.9 Markov Chain, 536
Probabilistic Methods in Transmission Tomography, 666
20.10
Martingale Process, 551
23.1
Introduction, 666
20.11
Periodic Random Process, 557
23.2
Stochastic Model, 667
20.12
Aperiodic Random Process (Karhunen – Loeve Expansion), 563
23.3
Stochastic Estimation Algorithm, 671
23.4
Prior Distribution PfMg, 674
23.5
Computer Simulation, 676
23.6
Results and Conclusions, 678
23.7
Discussion of Results, 681
23.8
References for Chapter 23, 681
CHAPTER 21 Random Processes and Linear Systems, 574 21.1 Review of Linear Systems, 574 21.2 Random Processes through
Linear Systems, 578 21.3 Linear Filters, 592 21.4 Bandpass Stationary Random
Processes, 606
APPENDIXES A
A Fourier Transform Tables, 683
B Cumulative Gaussian Tables, 687 C Inverse Cumulative Gaussian Tables, 692 D
Inverse Chi-Square Tables, 694
Weiner and Kalman Filters, 625
E
Inverse Student-t Tables, 701
22.1 Review of Orthogonality
F Cumulative Poisson Distribution, 704
CHAPTER 22
Principle, 625
G
Cumulative Binomial Distribution, 708
22.2 Wiener Filtering, 627 22.3 Discrete Kalman Filter, 647 22.4 Continuous Kalman Filter, 660
References, 714 Index, 716
Preface
Many good textbooks exist on probability and random processes written at the undergraduate level to the research level. However, there is no one handy and ready book that explains most of the essential topics, such as random variables and most of their frequently used discrete and continuous probability distribution functions; moments, transformation, and convergences of random variables; characteristic and generating functions; estimation theory and the associated orthogonality principle; vector random variables; random processes and their autocovariance and cross-covariance functions; stationarity concepts; and random processes through linear systems and the associated Wiener and Kalman filters. Engineering practitioners and students alike have to delve through several books to get the required formulas or tables either to complete a project or to finish a homework assignment. This book may alleviate this difficulty to some extent and provide access to a compendium of most distribution functions used by communication engineers, queuing theory specialists, signal processing engineers, biomedical engineers, and physicists. Probability tables with accuracy up to nine decimal places are given in the appendixes to enhance the utility of this book. A particular feature is the presentation of commonly occurring Fourier transforms where both the time and frequency functions are drawn to scale. Most of the theory has been explained with figures drawn to scale. To understand the theory better, more than 300 examples are given with every step explained clearly. Following the adage that a figure is worth more than a thousand words, most of the examples are also illustrated with figures drawn to scale, resulting in more than 400 diagrams. This book will be of particular value to graduate and undergraduate students in electrical, computer, and civil engineering as well as students in physics and applied mathematics for solving homework assignments and projects. It will certainly be useful to communication and signal processing engineers, and computer scientists in an industrial setting. It will also serve as a good reference for research workers in biostatistics and financial market analysis. The salient features of this book are .
Functional and statistical independence of random variables are explained.
.
Ready reference to commonly occurring density and distribution functions and their means and variances is provided.
.
A section on Benford’s logarithmic law, which is used in detecting tax fraud, is included.
.
More than 300 examples, many of them solved in different ways to illustrate the theory and various applications of probability, are presented. Most examples have been substantiated with graphs drawn to scale.
.
xi
xii
Preface .
More than 400 figures have been drawn to scale to improve the clarity of understanding the theory and examples.
.
Bounds on the tails of Gaussian probability have been carefully developed. A chapter has been devoted to computer generation of random variates.
. .
.
Another chapter has been devoted to matrix algebra so that some estimation problems such as the Kalman filter can be cast in matrix framework. Estimation problems have been given considerable exposure.
.
Random processes defined and classified. A section on martingale processes with examples has been included.
.
Markov chains with interesting examples have been discussed.
.
Wiener and Kalman filtering have been discussed with a number of examples. Important properties are summarized in tables.
.
. . .
.
The final chapter is on applications of probability to tomographic imaging. The appendixes consist of probability tables with nine-place decimal accuracy for the most common probability distributions. An extremely useful feature are the Fourier transform tables with both time and frequency functions graphed carefully to scale.
After the introduction of functional independence in Section 1.2, the differences between functional and statistical independences are discussed in Section 2.3 and clarified in Example 2.3.2. The foundation for permutations and combinations are laid out in Chapter 3, followed by discrete distributions in Chapter 4 with an end-of-chapter summary of discrete distributions. Random variables are defined in Chapter 5, and most of the frequently occurring continuous distributions are explained in Chapters 6 and 7. Section 6.4 discusses the new bounds on Gaussian tails. A comprehensive table is given at the end of Chapter 7 for all the continuous distributions and densities. After discussion of conditional densities, joint densities, and moments in Chapters 8 –10, characteristic and other allied functions are explained in Chapter 11 with a presentation of an end-of-chapter table of means and variances of all discrete and continuous distributions. Functions of random variables are discussed in Chapters 12 and 13 with numerous examples. Chapter 14 discusses the various bounds on probabilities along with some commonly occurring inequalities. Computer generation of random variates is presented in Chapter 15. Elements of matrix algebra along with vector and matrix differentiations are considered in Chapter 16. Vector random variables and diagonalization of covariance matrices and simultaneous diagonalization of two covariance matrices are taken up in Chapter 17. Estimation and allied hypothesis testing are discussed in Chapter 18. After random processes are explained in Chapter 19, they are carefully classified in Chapter 20, with Section 20.10 presenting martingale processes that find wide use in financial engineering. Chapter 21 discusses the effects of passing random processes through linear systems. A number of examples illustrate the basic ideas of Wiener and Kalman filters in Chapter 22. The final chapter, Chapter 23, presents a practical application of probability to tomographic imaging. The appendixes include probability tables up to nine decimal places for Gaussian, chi-square, Student-t, Poisson, and binomial distributions, and Fourier transform tables with graphs for time and frequency functions carefully drawn to scale. This book is suitable for students and practicing engineers who need a quick reference to any probability distribution or table. It is also useful for self-study where a number of
Preface
xiii
carefully solved examples illustrate the applications of probability. Almost every topic has solved examples. It can be used as an adjunct to any textbook on probability and may also be prescribed as a textbook with homework problems drawn from other sources. During the more than four decades that I have been continuously teaching probability and random processes, many authors who have written excellent textbooks have influenced me to a considerable extent. Many of the ideas and concepts that are expanded in this book are the direct result of this influence. I owe a scientific debt of gratitude to all the authors who have contributed to this ubiquitous and exciting field of probability. Some of these authors are listed in the reference, and the list is by no means exhaustive. Most of the graphs and all the probability tables in this book were created with Mathcad software. All algebraic calculations were verified with Mathcad software. Mathcad and Mathsoft are registered trademarks of Mathsoft Engineering and Education, Inc., http://www.mathcad. com. While facing extreme personal difficulties in writing this book, the unequivocal support of my entire family have been a source of inspiration in finishing it. I acknowledge with great pleasure the support given to me by the Wiley staff. George Telecki welcomed the idea of this book; Rachel Witmer did more than her share of keeping me in good humor with her cheerful disposition in spite of the ever-expanding deadlines; and Kellsee Chu who did an excellent job in managing the production of this book. Finally, the enthusiastic support given to me by the Series Editor Emmanuel Desurvire was very refreshing. This is my second book with Wiley, and the skills of their copyeditors and staff who transform highly mathematical manuscript into a finished book continue to amaze me. I have revised this book several times and corrected errors. Nevertheless, I cannot claim that it is error-free since correcting errors in any book is a convergent process. I sincerely hope that readers will bring to my attention any errors of commission or omission that they may come across. VENKATARAMA KRISHNAN Chelmsford, Massachusetts March 2006
CHAPTER
1
Sets, Fields, and Events
B 1.1
SET DEFINITIONS
The concept of sets play an important role in probability. We will define a set in the following paragraph. Definition of Set
A set is a collection of objects called elements. The elements of a set can also be sets. Sets are usually represented by uppercase letters A, and elements are usually represented by lowercase letters a. Thus A ¼ {a1 ;a2 ; . . . , an }
(1:1:1)
will mean that the set A contains the elements a1,a2, . . . , an. Conversely, we can write that ak is an element of A as ak [ A
(1:1:2)
ak A
(1:1:3)
and ak is not an element of A as
A finite set contains a finite number of elements, for example, S ¼ f2,4,6g. Infinite sets will have either countably infinite elements such as A ¼ fx : x is all positive integersg or uncountably infinite elements such as B ¼ fx : x is real number 20g. Example 1.1.1
The set A of all positive integers less than 7 is written as A ¼ {x : x is a positive integer ,7}: finite set:
Example 1.1.2
The set N of all positive integers is written as N ¼ {x : x is all positive integers}: countably infinite set:
Example 1.1.3
The set R of all real numbers is written as R ¼ {x : x is real}: uncountably infinite set:
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
1
2
Sets, Fields, and Events
Example 1.1.4
The set R 2 of real numbers x, y is written as R2 ¼ {(x,y) : x is real, y is real}
Example 1.1.5
The set C of all real numbers x,y such that x þ y 10 is written as C ¼ {(x,y) : x þ y 10}: uncountably infinite set:
Venn Diagram
Sets can be represented graphically by means of a Venn diagram. In this case we assume tacitly that S is a universal set under consideration. In Example 1.1.5, the universal set S ¼ fx : x is all positive integersg. We shall represent the set A in Example 1.1.1 by means of a Venn diagram of Fig. 1.1.1. Empty Set
An empty is a set that contains no element. It plays an important role in set theory and is denoted by 1. The set A ¼ f0g is not an empty set since it contains the element 0. Cardinality
The number of elements in the set A is called the cardinality of set A, and is denoted by jAj. If it is an infinite set, then the cardinality is 1. Example 1.1.6 The cardinality of the set A ¼ f2,4,6g is 3, or jAj ¼ 3. The cardinality of set R ¼ fx : x is realg is 1. Example 1.1.7
The cardinality of the set A ¼ fx : x is positive integer ,7g is jAj ¼ 6.
Example 1.1.8 The cardinality of the set B ¼ fx : x is a real number ,10g is infinity since there are infinite real numbers ,10. Subset
A set B is a subset of A if every element in B is an element of A and is written as B # A. B is a proper subset of A if every element of A is not in B and is written as B , A.
FIGURE 1.1.1
1.2 Set Operations
3
FIGURE 1.1.2 Equality of Sets
Two sets A and B are equal if B # A and A # B, that is, if every element of A is contained in B and every element of B is contained in A. In other words, sets A and B contain exactly the same elements. Note that this is different from having the same cardinality, that is, containing the same number of elements. Example 1.1.9 The set B ¼ f1,3,5g is a proper subset of A ¼ f1,2,3,4,5,6g, whereas the set C ¼ fx : x is a positive even integer 6g and the set D ¼ f2,4,6g are the same since they contain the same elements. The cardinalities of B, C, and D are 3 and C ¼ D. We shall now represent the sets A and B and the sets C and D in Example 1.1.9 by means of the Venn diagram of Fig. 1.1.2 on a suitably defined universal set S. Power Set
The power set of any set A is the set of all possible subsets of A and is denoted by PS(A). Every power set of any set A must contain the set A itself and the empty set 1. If n is the cardinality of the set A, then the cardinality of the power set jPS(A)j ¼ 2n. Example 1.1.10 If the set A ¼ f1,2,3g then PS(A) ¼ f1, (1,2,3), (1,2), (2,3), (3,1), (1), (2), (3)g. The cardinality jPS(A)j ¼ 8 ¼ 23.
B 1.2
SET OPERATIONS
Union
Let A and B be sets belonging to the universal set S. The union of sets A and B is another set C whose elements are those that are in either A or B, and is denoted by A < B. Where there is no confusion, it will also be represented as A þ B: A < B ¼ A þ B ¼ {x : x [ A or x [ B}
(1:2:1)
4
Sets, Fields, and Events
FIGURE 1.2.1
Example 1.2.1 The union of sets A ¼ f1,2,3g and B ¼ f2,3,4,5g is the set C ¼ A < B ¼ f1,2,3,4,5g. Intersection
The intersection of the sets A and B is another set C whose elements are the same as those in both A and B and is denoted by A > B. Where there is no confusion, it will also be represented by AB. A > B ¼ AB ¼ {x : x [ A and x [ B}
(1:2:2)
Example 1.2.2 The intersection of the sets A and B in Example 1.2.1 is the set C ¼ f2,3g Examples 1.2.1 and 1.2.2 are shown in the Venn diagram of Fig. 1.2.1. Mutually Exclusive Sets
Two sets A and B are called mutually exclusive if their intersection is empty. Mutually exclusive sets are also called disjoint. A>B¼1
(1:2:3)
One way to determine whether two sets A and B are mutually exclusive is to check whether set B can occur when set A has already occurred and vice versa. If it cannot, then A and B are mutually exclusive. For example, if a single coin is tossed, the two sets, fheadsg and ftailsg, are mutually exclusive since ftailsg cannot occur when fheadsg has already occurred and vice versa. Independence
We will consider two types of independence. The first is known as functional independence [42]. Two sets A and B can be called functionally independent if the occurrence of B does not in any way influence the occurrence of A and vice versa. The second one is statistical independence, which is a different concept that will be defined later. As an example, the tossing of a coin is functionally independent of the tossing of a die because they do not depend on each other. However, the tossing of a coin and a die are not mutually exclusive since any one can be tossed irrespective of the other. By the same token, pressure and temperature are not functionally independent because the physics of the problem, namely, Boyle’s law, connects these quantities. They are certainly not mutually exclusive.
Numbers in brackets refer to bibliographic entries in the References section at the end of the book.
1.2 Set Operations
5
Cardinality of Unions and Intersections
We can now ascertain the cardinality of the union of two sets A and B that are not mutually exclusive. The cardinality of the union C ¼ A < B can be determined as follows. If we add the cardinality jAj to the cardinality jBj, we have added the cardinality of the intersection jA > Bj twice. Hence we have to subtract once the cardinality jA > Bj as shown in Fig. 1.2.2. Or, in other words jA < Bj ¼ jAj þ jBj jA > Bj
(1.2.4a)
In Fig. 1.2.2 the cardinality jAj ¼ 9 and the cardinality jBj ¼ 11 and the cardinality jA < Bj is 11 þ 9 2 4 ¼ 16. As a corollary, if sets A and B are mutually exclusive, then the cardinality of the union is the sum of the cardinalities; or jA < Bj ¼ jAj þ jBj
(1.2.4b)
The generalization of this result to an arbitrary union of n sets is called the inclusion– exclusion principle, given by n n n n X [ X X Ai > A j þ A i > A j > Ak jA i j Ai ¼ i¼1 i¼1 i, j¼1 i, j, k¼1 i=j
(+1)n
i=j=k
n X Ai > A j > A k > An
(1:2:5a)
i, j, k¼1 i=j=k...=n
If the sets fAig are mutually exclusive, that is, Ai > Aj ¼ 1 for i = j, then we have n n X [ jA j (1:2:5b) A i ¼ i¼1 i¼1 i This equation is illustrated in the Venn diagram for n ¼ 3 in Fig. 1.2.3, where if jA < B < Cj equals, jAj þ jBj þ jCj then we have added twice the cardinalites of jA > Bj, jB > Cj and jC > Aj. However, if we subtract once jA > Bj, jB > Cj, and jC > Aj and write jA < B < Cj ¼ jAj þ jBj þ jCj jA > Bj jB > Cj jC > Aj then we have subtracted jA > B > Cj thrice instead of twice. Hence, adding jA > B > Cj we get the final result: jA < B < Cj ¼ jAj þ jBj þ jCj jA > Bj jB > Cj jC > Aj jA > B > Cj (1:2:6)
FIGURE 1.2.2
6
Sets, Fields, and Events
FIGURE 1.2.3
Example 1.2.3 In a junior class, the number of students in electrical engineering (EE) is 100, in math (MA) 50, and in computer science (CS) 150. Among these 20 are taking both EE and MA, 25 are taking EE and CS, and 10 are taking MA and CS. Five of them are taking EE, CS and MA. Find the total number of students in the junior class. From the problem we can identify the sets A ¼ EE students, B ¼ MA students, and C ¼ CS students, and the corresponding intersections are A > B, B > C, C > A, and A > B > C. Here, jAj ¼ 100, jBj ¼ 50, and jCj ¼ 150. We are also given jA > Bj ¼ 20, jB > Cj ¼ 10, and jC > Aj ¼ 25 and finally jA > B > Cj ¼ 5. Using Eq. (1.2.6) the total number of students are 100 þ 50 þ 150 2 20 2 10 2 25 þ 5 ¼ 250. Complement
If A is a subset of the universal set S, then the complement of A denoted by A is the elements in S not contained in A; or A ¼ S A ¼ {x : x A , S and x [ S}
(1:2:7)
Example 1.2.4 If the universal set S ¼ f1,2,3,4,5,6g and A ¼ f2,4,5g, then the complement of A is given by A ¼ f1,3,6g. Difference
The complement of a subset A , S as given by Eq. (1.2.7) is the difference between the universal set S and the subset A. The difference between any two sets A and B denoted by A 2 B is the set C containing those elements that are in A but not in B and, B 2 A is the set D containing those elements in B but not in A: C ¼ A B ¼ {x : x [ A and x B} D ¼ B A ¼ {x : x [ B and x A}
(1:2:8)
A 2 B is also called the relative complement of B with respect to A and B 2 A is called the relative complement of A with respect to B. It can be verified that A 2 B = B 2 A. Example 1.2.5 If the set A is given as A ¼ f2,4,5,6g and B is given as B ¼ f5,6,7,8g, then the difference A 2 B ¼ f2,4g and the difference B 2 A ¼ f7,8g. Clearly, A 2 B = B 2 A.
1.2 Set Operations
7
FIGURE 1.2.4 Symmetric Difference
The symmetric difference between sets A and B is written as ADB and defined by ADB ¼ (A B) < (B A) ¼ {x : x [ A and x B) or {x : x [ B and x A) (1:2:9) Example 1.2.6 The symmetric difference between the set A given by A ¼ f2,4,5,6g and B given by B ¼ f5,6,7,8g in Example 1.2.5 is ADB ¼ f2,4g < f7,8g ¼ f2,4,7,8g. The difference and symmetric difference for Examples 1.2.5 and 1.2.6 are shown in Fig. 1.2.4. Cartesian Product
This is a useful concept in set theory. If A and B are two sets, the Cartesian product A B is the set of ordered pairs (x,y) such that x [ A and y [ B and is defined by A B ¼ {(x,y) : x [ A, y [ B}
(1:2:10)
When ordering is considered the cartesian product A B = B A. The cardinality of a Cartesian product is the product of the individual cardinalities, or jA Bj ¼ jAj . jBj. Example 1.2.7 If A ¼ f2,3,4g and B ¼ f5,6g, then C ¼ A B ¼ f(2,5), (2,6), (3,5), (3,6), (4,5), (4,6)g with cardinality 3 2 ¼ 6. Similarly, D ¼ B A ¼ f(5,2), (5,3), (5,4), (6,2), (6,3), (6,4)g with the same cardinality of 6. However, C and D do not contain the same ordered pairs. Partition
S A partition is a collection T of disjoint sets fAi, i ¼ 1, . . . , ng of a set S such that Ai over all i equals S and Ai Aj , with i = j empty. Partitions play a useful role in conditional probability and Bayes’ theorem. Figure 1.2.5 shows the concept of a partition.
FIGURE 1.2.5
8
Sets, Fields, and Events
FIGURE 1.2.6
TABLE 1.2.1
Table of set properties
Property
Equation
Identity
A<1¼A A>S¼A
Domination
A>1¼1 A<S¼S
Idempotent
A>A¼A A
Complementation
A ¼A
Commutative
AB¼B>A
Associative
A < (B < C ) ¼ (A < B) < C A > (B > C ) ¼ (A > B) > C
Distributive
A > (B < C ) ¼ (A > B) < (A > C ) A < (B > C ) ¼ (A < B) > (A < C )
Noncommutative
A2B = B2A
De Morgan’s law
AB A>B¼A
Example 1.2.8 If set S ¼ f1,2,3,4,5,6,7,8,9,10g, then the collections of sets fA1, A2, A3, A4g where A1 ¼ f1,3,5g, A2 ¼ f7,9g, A3 ¼ f2,4,6g, A4 ¼ f8,10g is a partition of S as shown in Fig. 1.2.6. A tabulation of set properties is shown on Table 1.2.1. Among the identities, De Morgan’s laws are quite useful in simplifying set algebra.
B 1.3
SET ALGEBRAS, FIELDS, AND EVENTS
Boolean Field
We shall now consider a universal set S and a collection F of subsets of S consisting of the sets fAi: i ¼ 1,2, . . . , n,n þ 1, . . .g. The collection F is called a Boolean field if the
1.3 Set Algebras, Fields, and Events
9
following two conditions are satisfied: 1. If Ai [ F, then Ai [ F. 2. If A1 [ F and A2 [ F, then A1 < A2 [ F. 3. If fAi , i ¼ 1, . . . , ng [ F, then < ni¼1 Ai [ F. Let us see the consequences of these conditions being satisfied. If A1 [ F and A1 F, then A1 < A1 ¼ S [ F. If S [ F, then S ¼ 1 [ F. If A1 < A2 [ F, then by condition 1 A1 < A2 [ F, and by De Morgan’s law, A1 < A2 [ F. Hence A1 > A2 [ F. Also A1 2 A2 [ F and (A1 2 A2) < (A2 2 A1) [ F. Thus the conditions listed above are sufficient for any field F to be closed under all set operations. Sigma Field
The preceding definition of a field covers only finite set operations. Condition 3 for finite additivity may not hold for infinite additivity. For example, the sum of 1 þ 2 þ 22 þ 23 þ 24 ¼ (25 2 1)/(2 2 1) ¼ 31, but the infinite sum 1 þ 2 þ 22 þ 23 þ 24 þ . . . diverges. Hence there is a need to extend finite additivity concept to infinite set operations. Thus, we have a s field if the following extra condition is imposed for infinite additivity: 4. If A1, A2, . . . An, . . . [ F then <1 i¼1 Ai [ F Many s fields may contain the subsets fAig of S, but the smallest s field containing the subsets fAig is called the Borel s field. The smallest s field for S by itself is F ¼ fS, 1g. Example 1.3.1 We shall assume that S ¼ f1,2,3,4,5,6g. Then the collection of the following subsets of S, F ¼ fS, 1, (1,2,3), (4,5,6)g is a field since it satisfies all the set operations such as (1,2,3) < (4,5,6) ¼ S, ð 1,2,3 Þ ¼ (4,5,6). However, the collection of the following subsets of S, F1 ¼ {S, 1, (1,2,3), (4,5,6), ð2Þ} will not constitute a field because (2) < (4,5,6) ¼ (2,4,5,6) is not in the field. But we can adjoin the missing sets and make F1 into a field. This is known as completion. In the example above, if we adjoin the collection of sets F2 ¼ {(2,4,5,6), (1,3)} to F1 , then the resulting collection F ¼ F1 < F2 ¼ {S, 1, (1,2,3), (4,5,6), {2}, (2,4,5,6), (1,3)} is a field. Event
Given the universal set S and the associated s field F, all subsets of S belonging to the field F are called events. All events are subsets of S and can be assigned a probability, but not all subsets of S are events unless they belong to an associated s field.
CHAPTER
2
Probability Space and Axioms
B 2.1
PROBABILITY SPACE
A probability space is defined as the triplet fS, F, Pg, where S is the sample space, F is the field of subsets of S, and P is a probability measure. We shall now define these elements. Sample Space S
Finest-grain, mutually exclusive, and collectively exhaustive listing of all possible outcomes of a mathematical experiment is called the sample space. The outcomes can be either finite, countably infinite, or uncountably infinite. If S is countable, we call it a discrete sample space; if it is uncountable, we call it continuous sample space. Examples of Discrete Sample Space
Example 2.1.1 When a coin is tossed, the outcomes are heads {H} or tails {T}. Thus the sample space S ¼ {H,T}. Example 2.1.2 When a die is tossed, the outcomes are 1,2,3,4,5,6. Thus the sample space is S ¼ {1,2,3,4,5,6}. Example 2.1.3 The set of all positive integers is countable, and the sample space is S ¼ fx : x is a positive integerg Examples of Continuous Sample Space
Example 2.1.4 The set of all real numbers between 0 and 2 is the sample space S ¼ fx : 0 x 2g. Example 2.1.5 The set of all real numbers x and y such that x is between 0 and 1 and y is between 1 and 2 is given by S ¼ fx, y : 0 x 1, 1 y 2g. The continuous sample space given in Examples 2.1.4 and 2.1.5 is shown in Fig. 2.1.1.
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
10
2.1 Probability Space
11
FIGURE 2.1.1
Field F
A field has already been defined in Section 1.3. We shall assume that a proper Borel s field can be defined for all discrete and continuous sample spaces.
Probability Measure P
A probability measure is defined on the field F of events of the sample space S. It is a mapping of the sample space to the real line between 0 and 1 such that it satisfies the following three Kolmogorov axioms for all events belonging to the field F: Axiom 1: If A belongs to F; then P(A) 0: nonnegativity:
(2:1:1)
Axiom 2: The probability of the sample space equals 1. P(S) ¼ 1: normalization:
(2:1:2)
Axiom 3a: If A and B are disjoint sets belonging to F, that is, A > B ¼ 1, then P{A < B} ¼ P(A) þ P(B): Boolen additivity:
(2:1:3)
Axiom 3b: If fA1, A2, . . . g is a sequence of sets belonging to the field 1 such that Ai > Aj ¼ 1 for all i = j, then P fA 1 < A 2 < < A i < g ¼
1 X
PfA j g: sigma additivity:
(2:1:4)
i¼1
From these axioms we can write the following corollaries: Corollary 1: P{1}
(2:1:5)
Note that the probability of the empty event is zero, but if the probability of an event A is zero, we cannot conclude that A ¼ 1. All sets with zero probability are defined as null sets. An empty set is also a null set, but null sets need not be empty. Corollary 2. Since A < A ¼ S and A > A ¼ 1, we can write P(A) ¼ 1 P(A)
(2:1:6)
12
Probability Space and Axioms
Corollary 3. For any A and B that are subsets of S, not mutually exclusive, we can write Pf A < Bg ¼ P{A < A > B} ¼ P{A} þ P{A > B} P{B} ¼ P{(A > B) < (A > B)} ¼ P{A > B} þ P{A > B} Eliminating P{A < B} between the two equations, we obtain P{A < B} ¼ P{A} þ P{B} P{A > B}
(2:1:7)
Clearly, if P{A > B} ¼ 0 implying A and B are mutually exclusive, then Eq. (2.1.7) reduces to Kolmogorov’s axiom 3a. If the probabilities P{A} and P{B} are equal, then sets A and B are equal in probability. This does not mean that A is equal to B.
The probability of the union of three sets can be calculating from the Venn diagram for the three sets shown in Fig. 1.2.3 and redrawn in Fig. 2.1.2. The results of the analysis for the cardinalities given in Eq. (1.2.6), can be extended to finding the P(A < B < C). We subtract {P(A > B) þ P(B > C) þ P(C > A)} from {P(A) þ P(B) þ P(C)} and add P(A > B > C), resulting in
P{A < B < C} ¼ P{A} þ P{B} þ P{C} P{A > B} P{B > C} P{C > A} þ P{A > B > C}
FIGURE 2.1.2
(2:1:8)
2.1 Probability Space
13
Using the inclusion– exclusion principle of Eq. (1.2.5a), we can also generalize Eq. (2.1.8) to the probabilities of the union of n sets A1, A2, . . . , An as follows: Pfni¼1 Ai g ¼
n X
P{Ai }
i¼1
n X
P{Ai > A j } þ
i, j¼1 i=j
(+1)n
n X
n X
P{Ai > A j > Ak }
i, j, k¼1 i=j=k
P{Ai > A j > Ak > An }
(2:1:9)
i, j, k¼1 i=j=k...=n
Example 2.1.6 A fair coin is tossed 3 times. Since each toss is a functionally independent trial and forms a Cartesian product, the cardinality of the three tosses is 2 2 2 ¼ 8 and the sample space consists of the following 8 events: S ¼ {½HHH, ½HHT, ½HTT, ½HTH, ½THH, ½TTH, ½THT, ½TTT} In this sample space P{2 heads} ¼ P{½HHH, ½HHT, ½HTH, ½THH} ¼
4 8
P{both heads and tails} ¼ {½HHT, ½HTT, ½HTH, ½THH, ½TTH, ½THT} ¼
6 8
Example 2.1.7 A telephone call occurs at random between 9:00 a.m. and 11:00 a.m. Hence the probability that the telephone call occurs in the interval 0 , t t0 is given by t0 =120. Thus the probability that the call occurs between 9:30 a.m. and 10:45 a.m. is given by: P{30 , t 105} ¼
105 30 75 5 ¼ ¼ 120 120 8
Example 2.1.8 Mortality tables give the probability density that a person who is born today assuming that the lifespan is 100 years as f (t) ¼ 3 109 t2 (100 t2 )u(t) and the distribution function that the person will die at the age t0 years as ð t0 F(t0 ) ¼ P{t t0 } ¼
3 109 t2 (100 t2 ) dt
0
These probabilities are shown in Fig. 2.1.3. The probability that this person will die between the ages 70 to 80 is given by P{70 , t 80} ¼ 0:1052, obtained from the following integral: ð 80 P{70 , t 80} ¼ F(80) F(70) ¼ 3 109 t2 (100 t2 ) dt ¼ 0:1052 70
The density and distribution functions are shown in Fig. 2.1.3.
14
Probability Space and Axioms
FIGURE 2.1.3
B 2.2
CONDITIONAL PROBABILITY
The conditional probability of an event B conditioned on the occurrence of another event B is defined by
P{B j A} ¼
P{B > A} P{A}
P{B > A} ¼ P{B j A}P{A}
9 > = > ;
if P{A} = 0
(2:2:1)
If P{A} ¼ 0, then the conditional probability P{B j A} is not defined. Since P{B j A} is a probability measure, it also satisfies Kolmogorov axioms. Conditional probability P{B j A} can also be interpreted as the probability of the event B in the reduced sample space S1 ¼ A. Sometimes it is simpler to use the reduced sample space to solve conditional probability problems. We shall use both these interpretations in obtaining conditional probabilities as shown in the following examples.
2.2
Conditional Probability
15
FIGURE 2.2.1
Example 2.2.1 In the tossing of a fair die, we shall calculate the probability that 3 has occurred conditioned on the toss being odd. We shall define the following events: A ¼ {odd number}: B ¼ 3 1 2
1 6
P{A} ¼ and P{B} ¼ and the event A > B ¼ {3} and hence 1 p{3} 1 ¼6¼ P{3 j odd} ¼ 1 P{odd} 3 2 Note that P{2 j odd} ¼ 0 since 2 is an impossible event given that an odd number has occurred. Also note that P{3 j 3} ¼ 1. The conditional probability of 3 occurring, given that 3 has occurred is obviously equal to 1. We can also find the solution using the concept of reduced sample space. Since we are given that an odd number has occurred, we can reduce the sample space from S ¼ {1,2,3,4,5,6} to S1 ¼ {1,3,5} as shown in Fig. 2.2.1. The probability of 3 in this reduced sample space is 13, agreeing with the previous result. Example 2.2.2 Consider a family of exactly two children. We will find the probabilities (1) that both are girls, (2) both are girls given that one of them is a girl, (3) both are girls given that the elder child is a girl, and (4) both are girls given that both are girls. The sample space S has 4 points {gg, gb, bg, bb}, where g is girl and b is boy. The event B ¼ fboth children are girlsg. The conditioning event A will depend on the following four cases: 1. The event A ¼ S. Hence P{B} ¼ 14. 2. The event A ¼ fone child is a girlg. The reduced sample space is S1 ¼ fgg, gb, bgg and the conditional probability P{B j A} ¼ 13. 3. The event A ¼ felder child is a girlg. In this case, the reduced sample space is S2 ¼ {gg, gb} and P{B j A} ¼ 12. 4. The event ¼ fboth are girlsg. The reduced sample space has only one point, namely, S3 ¼ {gg} and P{B j A} ¼ 1. As our knowledge of the experiment increases by the conditioning event, the uncertainty decreases and the probability increases from 0.25, through 0333, 0.5, and finally to 1, meaning that there is no uncertainty.
16
Probability Space and Axioms
Example 2.2.3 The game of craps as played in Las Vegas has the following rules. A player rolls two dice. He wins on the first roll if he throws a 7 or a 11. He loses if the first throw is a 2, 3, or 12. If the first throw is a 4, 5, 6, 8, 9, or 10, it is called a point and the game continues. He goes on rolling until he throws the point for a win or a 7 for a loss. We have to find the probability of the player winning. We will solve this problem both from the definition of conditional probability and the reduced sample space. Figure 2.2.2 shows the number of ways the sums of the pips on the two dice can equal 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, and their probabilities. Solution Using Definition of Conditional Probability. The probability of winning in the first throw is P{7} þ P{11} ¼
6 2 8 þ ¼ ¼ 0:22222 36 36 36
The probability of losing in the first throw is P{2} þ P{3} þ P{12} ¼
1 2 1 4 þ þ ¼ ¼ 0:11111 36 36 36 36
To calculate the probability of winning in the second throw, we shall assume that i is the point with probability p. The probability of not winning in any given throw after the first is given by r ¼ P{not i and not 7} ¼ 1 p 16. We compute the conditional probability P{win j i in first throw} ¼ p þ rp þ r 2 p þ ¼
p p ¼ 1 r p þ 16
as an infinite geometric series: P{win in second or subsequent throws} ¼
p2 p þ 16
The probability p of making the point i ¼ 4,5,6,8,9,10 is obtained from Fig. 2.2.2.
FIGURE 2.2.2
2.2
Conditional Probability
17
Thus, for i ¼ 4,5,6, we obtain
2
2
2
3 P{win after point i ¼ 4} ¼ 36
P{win after point i ¼ 5} ¼
P{win after point i ¼ 6} ¼
4 36 5 36
3 1 1 þ ¼ 36 6 36 4 1 2 þ ¼ 36 6 45 5 1 25 þ ¼ 36 6 396
with similar probabilities for 8,9,10. Thus the probability of winning in craps is P{win} ¼ P{win in roll 1} þ P{win in roll 2,3, . . .} 8 1 2 25 244 ¼ þ2 þ þ ¼ 0:4929 ¼ 36 36 45 396 495 Solution Using Reduced Sample Space. We can now use the reduced sample space to arrive at the same result in a simpler manner. After the point i has been rolled in the first throw, the reduced sample space consists of (i 2 1 þ 6) points for i ¼ 4,5,6 and (13 2 i þ 6) points for i ¼ 8,9,10 in which the game will terminate. We can see from the figure that there are 6 ways of rolling 7 for a loss, (i 2 1, i ¼ 4,5,6) ways of rolling the point for a win, and (13 2 i, i ¼ 8,9,10) ways of rolling the point for a win. Thus P{win in roll 2,3, . . . j i in roll 1} ¼ P{W j i} 8 i1 > , < ¼ i1þ6 > : 13 i , 13 i þ 6
i ¼ 4,5,6 i ¼ 8,9,10
4 5 5 4 Thus, P{W j 4} ¼ 39 , P{W j 5} ¼ 10 , P{W j 6} ¼ 11 , P{W j 8} ¼ 11 , P{W j 9} ¼ 10 , 3 P{W j 10} ¼ 9. Hence probability of a win after the first roll is P{W j i} P{i}: Hence
3 3 4 4 5 5 134 þ þ ¼ 9 36 10 36 11 36 495 3 3 4 4 5 5 8 134 244 þ ¼ 0:4929 P{win} ¼ 2 þ þ ¼ þ 9 36 10 36 11 36 36 495 495
P{win after first roll} ¼ 2
a result obtained with comparative ease. Example 2.2.4 We shall continue Example 2.1.8 and find the probability that a person will die between the ages 70 and 80 given that this individual has lived upto the age 70. We will define the events A and B as follows: A ¼ {person has lived upto 70} B ¼ {person dies between ages 70 and 80}
18
Probability Space and Axioms
The required conditional probability equals the number of persons who die between the ages 70 and 80 divided by the number of persons who have lived upto the age 70:
P{B j A} ¼
P{70 , t 80 and t 70} P{70 , t 80} ¼ P{t 70} P{t 70} Ð 80
70 ¼ Ð 100 70
3 109 t2 (100 t2 )dt 3
109 t2 (100
t2 )dt
¼
0:105 ¼ 0:644 0:163
This type of analysis is useful in setting up actuarial tables for life insurance purposes.
B 2.3
INDEPENDENCE
We have already seen the concept of functional independence of sets in Chapter 1. We will now define statistical independence of events. If A and B are two events in a field of events F, then A and B are statistically independent if and only if P{A > B} ¼ P{A}P{B}
(2:3:1)
Statistical independence neither implies nor is implied by functional independence. Unless otherwise indicated, we will call statistically independent events “independent” without any qualifiers. Three events A1 ,A2 ,A3 are independent if they satisfy, in addition to P{A1 > A2 > A3 } þ P{A1 }P{A2 }P{A3 }
(2:3:2)
the pairwise independence P{Ai > A j } ¼ P{Ai }P{A j },
i, j ¼ 1,2,3, i = j
(2:3:3)
Note that if A and B are mutually exclusive, then, from Kolmogorov’s axiom 3a, we obtain P{A < B} ¼ P{A} þ P{B}
(2:3:4)
The concepts of independence and mutual exclusivity are fundamental to probability theory. For any two sets A and B, we recall from Eq. (2.1.7) that P{A < B} ¼ P{A} þ P{B} P{A > B}
(2:1:7)
If A and B are mutually exclusive, then P{A > B} ¼ 0, but if they are independent, then P{A > B} ¼ P{A} P{B}. If this has to be zero for mutual exclusivity, then one of the probabilities must be zero. Therefore, we conclude that mutual exclusivity and independence are incompatible unless we are dealing with null events. We shall now give examples to clarify the concepts of functional independence and statistical independence. Example 2.3.1 The probability of an ace on the throw of a die is 16. The probability of a head on the throw of a coin is 12. Since the throw of a die is not dependent on the throw of a coin, we have functional independence and the combined sample space is the Cartesian product with 12 elements as shown in Table 2.3.1.
2.3 Independence
19
TABLE 2.3.1 1, H
2, H
3, H
4, H
5, H
6, H
1, T
2, T
3, T
4, T
5, T
6, T
We assume as usual that all the 12 elements are equiprobable events and the probability of 1 ace and head is 12 since there is only one occurrence of f1, Hg among the 12 points as seen in the first row and first column in Table 2.3.1. Using Eq. (2.3.1), we have 1 P{1, H} ¼ P{1} P{H} ¼ 16 12 ¼ 12 , illustrating that in this example, functional independence coincides with statistical independence.
Example 2.3.2 Two dice, one red and the other blue, are tossed. These tosses are functionally independent, and we have the Cartesian product of 6 6 ¼ 36 elementary events in the combined sample space, where each event is equiprobable. We seek the probability that an event B defined by the sum of the numbers showing on the dice equals 9. There are 4 four points {(6,3), (5,4), (4,5), (3,6)}, and hence P{B} ¼ 36 ¼ 19. We now condition the event B with an event A defined as the red die shows odd numbers. The probability of 1 the event A is P{A} ¼ 18 36 ¼ 2. We want to determine whether the events A and B are statistically independent. These events are shown in Fig. 2.3.1, where the first number is for the red die and the second number is for the blue die. 2 1 From Fig. 2.3.1 P{A > B} ¼ P{(3,6), (5,4)} ¼ 36 ¼ 18 and ¼ P{A} P{B} ¼ 12 19 ¼ 1 18 showing statistical independence. We compute P{B j A} from the point of view of reduced sample space. The conditioning event reduces the sample space from 36 points to 18 equiprobable points and the event {B j A} ¼ {(3,6), (5,4)}. Hence 2 P{B j A} ¼ 18 ¼ 19 ¼ P{B}, or, the conditioning event has no influence on B. Here, even though the events A and B are functionally dependent, they are statistically independent. However, if another set C is defined by the sum being equal to 8 as shown in 5 Fig. 2.3.2, then P{C} ¼ 36 . Here the events C and A are not statistically independent because P{C} P{A} ¼ 5 1 5 2 4 36 2 ¼ 72 = P{C > A} ¼ P{(3,5), (5,3)} ¼ 18 ¼ 72. In this example, we have the case where the events A and C are neither statistically independent nor functionally independent.
FIGURE 2.3.1
20
Probability Space and Axioms
FIGURE 2.3.2
B 2.4
TOTAL PROBABILITY AND BAYES’ THEOREM
Let {Ai , i ¼ 1, . . . , n} be a partition of the sample space and let B an arbitrary event in the sample space S as shown in Fig. 2.4.1. We will determine the total probability PfBg, given the conditional probabilities P{B j A1 }, P{B j A2 }, . . . , P{B j An }. The event B can be given in terms of the partition as B ¼ {B > S} ¼ B > {A1 < A1 < < An } ¼ {B > A1 } < {B > A2 } < < {B > An }
(2:4:1)
The events {B > A1 }, {B > A2 }, . . . , {B > An } in Eq. (2.4.1) are mutually exclusive since A1, A2, . . . , An are mutually exclusive and we have from Kolmogorov’s axiom 3a: P{B} ¼ P{B > A1 } þ P{B > A2 } þ þ P{B > An }
(2:4:2)
Using the definition of conditional probability from Eq. (2.2.1), we can rewrite Eq. (2.4.2) as P{B} ¼ P{B j A1 }P{A1 } þ P{B j A2 }P{A2 } þ þ P{B j An }P{An } ¼
n X
P{B j Ai }P{Ai }
(2:4:3)
i¼1
Equation (2.4.3) is the expression for the total probability P{B} in terms of the conditional probabilities P{B j Ai , i ¼ 1, . . . , n}. We shall give an example to illustrate this concept.
FIGURE 2.4.1
2.4 Total Probability and Bayes’ Theorem
21
Example 2.4.1 Four computer companies A, B, C, D supply microprocessors to a company. From previous experience it is known that the probability of the processor being bad if it comes from A is 4%, B is 2%, C is 5%, and D is 1%. The probabilities of picking supplier A is 20%, B is 30%, C is 10%, and D are 40%. We want to find the probability that a processor chosen at random is bad. Let us define X as the event that the processor is bad. We need to find P{X} from the conditional probabilities P{X j A} ¼ 0:04, P{X j B} ¼ 0:02, P{X j C} ¼ 0:05, and P{X j D} ¼ 0:01. We are also given that P{A} ¼ 0:2, P{B} ¼ 0:3, P{C} ¼ 0:1, and P{D} ¼ 0:4. Substituting these quantities in Eq. (2.4.3), we obtain P{X} ¼ P{X j A}P{A} þ P{X j B}P{B} þ P{X j C}P{C} þ P{X j D}P{D} ¼ 0:04 0:2 þ 0:02 0:3 þ 0:05 0:1 þ 0:01 0:4 ¼ 0:023 Thus the probability that the processor is bad is 2.3%.
Bayes’ Theorem
The next logical question to ask is, “Given that the event B has occurred, what are the probabilities that A1, A2, . . . , An are involved in it?” In other words, given the conditional probabilities P{B j A1 }, P{B j A2 }, . . . , P{B j An }, we have to find the reverse conditional probabilities P{A1 j B}, P{A2 j B}, . . . , P{An j B}. Using the total probability P{B} from Eq. (2.4.3), we can write P{Ai j B} ¼
P{Ai B} P{B j Ai }P{Ai } P{B j Ai }P{Ai } ¼ ¼ Pn P{B} P{B} j¼1 P{B j A j }P{A j }
(2:4:4)
This is the celebrated Bayes’ theorem. This theorem connects the a priori probability P{Ai } to the a posteriori probability P{Ai j B}. This useful theorem has wide applicability in communications and medical imaging, as we will see in Chapter 23. Example 2.4.2 We will now continue Example 2.4.1 and find the probability that the processor came from companies A, B, C, or D given that the processor is bad. Using the result from Example 2.4.1 that P{X} ¼ 0:023, we can write from Eq. (2.4.4) P{X j A} ¼
P{X j A}P{A} 0:04 0:2 ¼ ¼ 0:348 P{X} 0:023
P{X j B} ¼
P{X j B}P{B} 0:02 0:3 ¼ ¼ 0:261 P{X} 0:023
P{X j C} ¼
P{X j C}P{C} 0:05 0:1 ¼ ¼ 0:217 P{X} 0:023
P{X j D} ¼
P{X j D}P{D} 0:01 0:4 ¼ ¼ 0:174 P{X} 0:023
Note that the sum of the resulting probabilities, namely, 0.348 þ 0.261 þ 0.217 þ 0.174 ¼ 1, as it should be because the bad processor has to come only from any one of the four companies A, B, C, or D resulting in the probability being 1. Example 2.4.3 A communication receiver sends numbers 1,2,3. As a result of inherent noise in the channel, the conditional probabilities of receiving 1,2,3 under the condition that 1,2,3 were sent are shown in Fig. 2.4.2.
22
Probability Space and Axioms
FIGURE 2.4.2
Transition Probabilities: P{1R j 1S } ¼ 0:60
P{2R j 1S } ¼ 0:25
P{3R j 1S } ¼ 0:15
P{1R j 2S } ¼ 0:35
P{2R j 2S } ¼ 0:55
P{3R j 2S } ¼ 0:10
P{1R j 3S } ¼ 0:05
P{2R j 3S } ¼ 0:30
P{3R j 3S } ¼ 0:65
The total probability that 1 was received is obtained from Eq. (2.4.3): P{1R } ¼ P{1R j 1S }P{1S } þ P{1R j 2S }P{2S } þ P{1R j 3S }P{3S } ¼ 0:6 0:25 þ 0:35 0:3 þ 0:05 0:45 ¼ 0:2775 From Bayes’ theorem, Eq. (2.4.4), the a posteriori probabilities of 1, 2, and 3 being sent conditioned on 1 received are found as follows: P{1S j 1R } ¼
P{1R j 1S }P{1S } 0:6 0:25 ¼ 0:5405: ¼ P{1R } 0:2775
P{1S } ¼ 0:25
P{2S j 1R } ¼
P{1R j 2S }P{2S } 0:35 0:3 ¼ 0:3784: ¼ P{1R } 0:2775
P{2S } ¼ 0:3
P{3S j 1R } ¼
P{1R j 3S }P{3S } 0:05 0:45 ¼ 0:0811: ¼ P{1R } 0:2775
P{3S } ¼ 0:45
The sum of these probabilities is equal to 1, as it should be. Note that the a priori probability that 3 was sent is 0.45 and after 1 has been received, the a posteriori probability is 0.0811, a very small probability. Similar posterior probabilities of 1, 2, and 3 being sent conditioned on 2 and 3 received can be found. Example 2.4.4 (Monty Hall Problem) This classic problem is called the “Monty Hall problem” because of the game show host Monty Hall who designed this game. There are three doors A, B, and C, and behind one of them is a car and behind the other two are goats. The contestant is asked to select any door, and the host Monty Hall opens one of the other two doors and reveals a goat. He then offers the choice to the contestant of switching to the other unopened door or keep the original door. The question now is whether the
2.4 Total Probability and Bayes’ Theorem
23
probability of winning the car is improved if she switches or it is immaterial whether she switches or not. The answer is counterintuitive in the sense that one may be misled into thinking that it is immaterial whether one switches or not. We will analyze this problem in two different ways. Mathematical Analysis. Let us analyze this problem from the point of view of Bayes’ theorem. We shall first assume that the host has prior knowledge of the door behind which the car is located. Let us also assume that the host opens door B given that the contestant’s choice is door A. The a priori probabilities of a car behind the doors A, B, and C are 1 1 1 P{A} ¼ ; P{B} ¼ ; P{C} ¼ 3 3 3 We can make the following observations. If the contestant’s choice is door A and the car is behind door A, then the host will open the door B with probability of 12 (since she has a choice between B and C ). On the other hand, if the car is behind door B, then there is zero probability that she will open door B because of her prior knowledge. If the car is behind door C, then she will open door B with probability 1. Thus we can write the following conditional probabilities for the host opening door B: 1 P{B j A} ¼ ; P{B j B} ¼ 0; P{B j C} ¼ 1 2 We can now calculate the total probability P(B) of the host opening door B. P{B} ¼ P{B j A}P{A} þ P{B j B}P{B} þ P{B j C}P{C} ¼
1 1 1 1 1 þ0 þ1 ¼ 2 3 3 3 2
Using Bayes’ theorem [Eq. (2.4.4)] we can now find the a posteriori probabilities of the car behind the doors A or C (B has already been opened by the host) conditioned on B: 1 1 P{BA} P{B j A}P{A} 2 3 1 P{A j B} ¼ ¼ ¼ ¼ 1 P{B} P{B} 3 2 1 P{BC} P{B j C}P{C} 1 3 2 P{C j B} ¼ ¼ ¼ ¼ 1 P{B} P{B} 3 2 Similar analysis holds for other cases of opening the doors A or C. We can see from the result above that the contestant must switch if she wants to double her probability of winning the car. TABLE 2.4.1 First Choice
Host Opens
Second Choice
Probability
A
B
A
P{B} ¼ 12 13 ¼ 16
A
C
A
P{C} ¼ 12 13 ¼ 16
B
C
B
P{C} ¼ 1 13 ¼ 13
C
B
B
P{B} ¼ 1 13 ¼ 13
24
Probability Space and Axioms
TABLE 2.4.2 First Choice A A B C
TABLE 2.4.3 First Choice A A B C
For not switching Host Opens
Second Choice
B C C B
A A B B
Win or Loss
Probability P{B} ¼ 12 13 ¼ 16 P{C} ¼ 12 13 ¼ 16 P{C} ¼ 1 13 ¼ 13 P{B} ¼ 1 13 ¼ 13
Win Win Loss Loss
For switching Host Opens
Second Choice
B C C B
C B A A
Win or Loss
Probability P{B} ¼ 12 13 ¼ 16 P{C} ¼ 12 13 ¼ 16 P{C} ¼ 1 13 ¼ 13 P{B} ¼ 1 13 ¼ 13
Loss Loss Win Win
Heuristic Analysis. In this case we shall enumerate all the possibilities and determine which possibility gives a better chance. Assume that the car is behind door A. The probabilities of A,B,C are once again 1 P{A} ¼ ; 3
1 P{B} ¼ ; 3
P{C} ¼
1 3
Conditioned on the contestant choosing door A, the host has two equiprobable choices of opening either door B or door C. However, conditioned on the contestant choosing B or C, the host has only one choice of either opening C or B. Thus 1 P{B j A} ¼ ; 2
1 P{C j A} ¼ ; 2
P{C j B} ¼ 1;
P{B j C} ¼ 1
From these conditional probabilities we can evaluate the unconditional probabilities as shown in Table 2.4.1. We can now enumerate the various win or loss possibilities for not switching and for switching in Tables 2.4.2 and 2.4.3. From the Table 2.4.2 we can see that the probability of winning the car by not switching is 16 þ 16 ¼ 13. From Table 2.4.3 we can see that the probability of winning the car by switching is 13 þ 13 ¼ 23. Thus the probability of winning the car doubles with switching.
CHAPTER
3
Basic Combinatorics
B 3.1
BASIC COUNTING PRINCIPLES
Combinatorics is the study of arrangement and manipulation of mathematical elements in sets and essentially involves counting. We will discuss two basic counting principles. The Addition Rule
A task A consists of m subtasks A1, A2, . . . , Am that can be done in n(A1), n(A2), . . . , n(Am) ways, respectively. If no two of them can be done at the same time, then the total number of ways n(A) of doing the task A is n(A) ¼ n(A1 ) þ n(A2 ) þ þ n(Am )
(3:1:1)
This rule is the same as in set theory [Eq. (1.2.5a)] where the cardinality of the union of mutually exclusive sets is the sum of the cardinalities of the individual sets. Thus, the addition rule is a restatement of one of the properties of mutually exclusive sets. Example 3.1.1 An engineering student undertaking a capstone project T has a choice of 4 professors A,B,C,D who can give him a list of projects from which he can choose one. A has 5 projects, B has 7 projects, C has 6 projects, and D has 2 projects. Since the student can not take any other project once he has chosen a project, the number of choices he has is n(T) ¼ n(A) þ n(B) þ n(C) þ n(D) ¼ 5 þ 7 þ 6 þ 2 ¼ 20 The Multiplication Rule
A task A consists of m sequential subtasks A1, A2, . . . , Am that can be performed only after the previous one is completed. The numbers of ways n(A1), n(A2), . . . , n(Am) of performing the subtasks are functionally independent; that is, n(A2) ways are needed for the second step regardless of n(A1) of the first step. The number of ways n(A) of performing the task A is n(A) ¼ n(A1 ) n(A2 ) n(Am )
(3:1:2)
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
25
26
Basic Combinatorics
This is very similar to the Cartesian product in set theory, where the cardinality of the product is the product of the cardinalities of the individual sets. These sets are also functionally independent. They may or may not be statistically independent. Example 3.1.2 A professor assigns a capstone project T to senior engineering students consisting of four parts A,B,C,D that must be done in sequence independent of the previous ones. Part A can be done in 2 ways, part B in 3 ways, part C in 5 ways, and part D in 4 ways. Since A,B,C,D are functionally independent, the project can be done in n(T) ¼ n(A) n(B) n(C) n(D) ¼ 2 3 5 4 ¼ 120 ways
B 3.2
PERMUTATIONS
Many problems in combinatorics can be modeled as placing r balls into n cells. The two ways of considering permutations are (1) sampling with replacement and with ordering and (2) sampling without replacement and with ordering. Sampling with Replacement and with Ordering
We consider placing r balls into n cells with replacement and with ordering. The first ball can be placed in n cells in n ways. The second ball can be placed also in n ways. Since the placing of the second ball is functionally independent of that of the first ball, there are now n 2 ordered 2-permutations. Thus, for placing r balls into n cells, there will be nr ways of placing r balls into n cells with replacement and ordering: Number of distinct ordered r-tuples ¼ nr
(3:2:1)
The probability of each one of these r-tuples assuming that they are equally likely is n – r. Clearly we can place 100 balls into one cell or 1 ball in 100 cells. Hence in Eq. (3.2.1) it does not matter whether r . n, r ¼ n, or r , n. Example 3.2.1 The number of distinct ordered ways we can draw 2 balls from an urn containing 4 balls numbered from 1 to 4 is obtained as follows. The cells correspond to n ¼ 4, the total number of balls. The balls correspond to r ¼ 2. The number of ways 2 balls can be placed into 4 cells is, from Eq. (3.2.1), 42 ¼ 16 ways. These 16 ordered ways are shown in Table 3.2.1 for clarity.
Sampling without Replacement and with Ordering
We now consider placing r balls into n cells without replacement and with ordering. This situation is different from the previous case. The first ball can be placed in n cells in
TABLE 3.2.1 1,1 3,1
1,2 3,2
1,3 3,3
1,4 3,4
2,1 4,1
2,2 4,2
2,3 4,3
2,4 4,4
3.2 Permutations
27
n different ways. Since one of the cells is occupied, the second ball can be placed only in (n – 1) ways. The rth ball can be placed in (n – r þ 1) ways. Since placements of the balls are again functionally independent, we have Number of distinct r-tuples ¼ n(n 1) (n r þ 1)
(3:2:2)
These will be called r-permutations and will be defined by (n)r or nPr. Using the factorial notation, we can write (n)r ¼
n! (n r)!
(3:2:3)
We cannot place 5 balls into 2 cells without replacement because at least some cells will have more than 1 ball, violating the nonreplacement condition. Hence we define (n)r ¼ 0 whenever r . n. If r ¼ n, then (n)r ¼ (n)n ¼ n!. Example 3.2.2 We continue Example 3.2.1 and now draw 2 balls from 4 balls without replacement and with ordering. The number of ways this can be done is obtained by applying Eq. (3.2.3) with n ¼ 4 and r ¼ 2 which is equal to 4.3 ¼ 12 ways instead of the 16 ways that we obtained in Table 3.2.1. The 12 ways are shown in Table 3.2.2. In Table 3.2.2, the arrangement with ordering, (1,2), (1,3), (1,4), (2,3), (2,4), (3,4) represents an ordering different from (2,1), (3,1), (4,1), (3,2), (4,2), (4,3). We can now summarize the preceding results as follows. The number of different ways in which r balls can be placed into n cells with ordering and replacement is n r, and with ordering and without replacement is (n)r. Example 3.2.3 In the Olympic 100-m dash, there are 9 runners. In how many ways can the gold, silver, and bronze can be given to these runners? Here the 9 runners are the cells n and the 3 medals are the balls r. Hence, from Eq. (3.2.3), the number of ways that the medals can be distributed is (9)3 ¼ 9 8 7 ¼ 504 ways. Example 3.2.4 (Birthdays Problem) We want to determine the probability that in a class size of r two or more birth dates match. We shall assume that the year consists of 365 days and the year of birth is inconsequential. Here the 365 days in a year correspond to n cells and the class size, to r. If the class size is over 365, then two or more birth dates will match with probability 1. The number of ways r students can be placed into 365 birth dates with replacement is 365r. The number of ways they can be placed without replacement is (365)r, meaning that all the birth dates are different. Thus, the probability of no two birth dates match is P{no birthdays match} ¼
(365)r 365r
TABLE 3.2.2 1,2 3,1
1,3 3,2
1,4 3,4
2,1 4,1
2,3 4,2
2,4 4,3
28
Basic Combinatorics
FIGURE 3.2.1
TABLE 3.2.3 #
Probability
#
Probability
#
Probability
20 21 22 23 24 25 26
0.411438 0.443668 0.475695 0.507297 0.538344 0.5687 0.598241
30 31 32 33 34 35 36
0.706316 0.730455 0.753348 0.774972 0.795317 0.814383 0.832182
44 45 46 46 48 49 50
0.932885 0.940976 0.948253 0.954774 0.960598 0.96578 0.970374
The probability P(r) that two or more birth dates match is the complement of the preceding equation and is given by P{r} ¼ 1
(365)r 365r
P(r) is graphed for r ¼ 0– 50 in Fig. 3.2.1 and tabulated for r ¼ 20– 26, 30– 36, and 44 – 50 in Table 3.2.3. Note that for two or more birthdays to match, there is about 50% chance for 23 students, 71% chance for 30 students, and 95% chance for 46 students. In other words, we need only 46 students for 2 or more birthdays to match with 95% probability, but to get to 100% probability, we need to go to 366 students!
B 3.3
COMBINATIONS
Similar to the case of permutations we have two ways of considering combinations: 1. Sampling without replacement and without ordering 2. Sampling with replacement and without ordering
3.3 Combinations
29
Sampling without Replacement and without Ordering
We now discuss the case where we want to obtain a sample of r balls from n balls without replacement and without ordering. We have already seen that the number of ways to get r balls from n balls with ordering is (n)r. If we disregard ordering, then the number of ways is called the r-combination from the n balls. We can define this quantity as n {r-combination from n balls} ¼ (3:3:1) r n By ordering the elements in each r-combination, we can obtain from n r the rpermutations (n)r. This can be done in r! ways. Hence we can determine r from either of the following formulas: n r! ¼ (n)r r (3:3:2) n (n)r n! ¼ ¼ r!(n r)! r! r The quantity nr is also called the binomial coefficient since it occurs as the coefficient of the binomial expansion, given by n n n n1 n nr r n n n (a þ b) ¼ a þ a b þ þ a b þ þ b 0 1 r n Properties of Binomial Coefficients
n n ¼ nr r
1.
(3:3:3)
Expanding the righthand side of Eq. (3.3.3), we obtain n! n! n n ¼ ¼ ¼ r nr (n r)!½n (n r)! (n r)!r! 2. Pascal’s Identity:
nþ1 r
¼
n r1
þ
n r
(3:3:4)
Expanding the righthand side of Eq. (3.3.4), we obtain
n r1
þ
n r
n! n! þ (r 1)!(n r þ 1)! r!(n r)! n! r þ1 ¼ r!(n r)! n r þ 1 nþ1 n! nþ1 (n þ 1)! ¼ ¼ ¼ r!(n r)! n r þ 1 r!(n r þ 1)! r ¼
Pascal’s identity shows that if two adjacent binomial coefficients are added, the binomial coefficient in the next row in between these two coefficients results, as shown in Fig. 3.3.1. It is called the Pascal triangle.
30
Basic Combinatorics
FIGURE 3.3.1
3. Sum of Binomial Coefficients: n X n r¼0
r
¼ 2n
(3:3:5)
Substituting in the binomial expansion (1 þ x)n ¼ 1 þ
n 2 n n n x þ þ xþ x 2 1 n
x ¼ 1, we obtain Eq. (3.3.5). 4. Vandermonde’s Identity:
nþm r
r X n m ¼ k rk k¼0
(3:3:6)
Let us assume that there are two urns with n balls in the first and m balls in the second. nþm The total number of ways of picking r balls from both of these urns is r . We can also choose these r balls by first picking k balls from the first urn and (r 2 k) balls from the second urn. Since thechoice m is functionally independent, the number of ways of accomplishing this is nk rk . The realizations for every 0 k r are mutually exclusive, and hence these numbers add, resulting in Eq. (3.3.6). This is shown in Fig. 3.3.2.
FIGURE 3.3.2
3.3 Combinations
31
Example 3.3.1 We continue Example 3.2.2 with sampling 2 balls from 4 balls without replacement and without ordering. Thus means that there is no distinction between (12) and (21), (23) and (32), (34), and (43). Thus we have only 6 ways and this is obtained by applying Eq. (3.3.1) with n ¼ 4 and r ¼ 2, yielding (4 3)/(2 1) ¼ 6. These are shown in Table 3.3.1. TABLE 3.3.1 1,2 2,3
1,3 2,4
1,4 3,4
Example 3.3.2 1. A committee of 3 women and 4 men must be formed from a group of 6 women and 8 men. The number of ways 3 women can be chosen from 6 women is 63 from Eq. (3.2.2) and the number of ways 4 men can be chosen from 8 men is 8 4 . Since these choices are functionally independent, the total number of ways the committee can be formed is 63 84 ¼ 1400. 2. We modify the problem above by having 2 particular men be always on the committee. In this case we have a choice of only 2 men from the 6 men after taking out the 2 required men. Thus, there are 63 62 ¼ 300 ways. 3. We again modify the problem by requiring that two particular women should not be on the committee. In this case, we calculate the number of ways that the two women will be together and subtract it from the total number of ways. There are 51 84 ¼ 350 ways that the two women will be together, and hence there are 63 84 51 84 ¼ 1400 350 ¼ 1050 ways in which two given women will not be together. Example 3.3.3 (Catalan Numbers) There are n objects to be placed in computer stacks in the following way. These objects are first stored and deleted later with the result that when all the n objects are deleted, the stack is empty. Hence, the operation store (S ) has to precede the operation delete (D), and at any point along the stack the number of stores should always be more than or equal to the number of deletes. Finally, the stack should begin with store and end with delete. We need to find the number C(n) of operations required for n objects. For example, if there are n ¼ 3 objects, the number of ways of ordering the store and delete operations are 2 3 ¼ 6 given by SSSDDD, SDSDSD, SSDDSD, SDSSDD, SSDSDD, SDSDSSD. Thus, the operation starts with an S and ends in a D, and at no point along the chain can the number of D operations exceed the S operations. We will show that the number C(n) is given by C(n) ¼
1 (2n)! n þ 1 n!n!
Since there are a total of 2n operations, the total number of ways that we can obtain an n-combination is 2n n . Out of these combinations there will be many realizations in which the number of Ds will exceed the number of Ss. We will now find the number of ways in which the Ds will exceed the Ss. If we are at the point (n þ 1)
32
Basic Combinatorics
TABLE 3.3.2 #
Stacks
#
Stacks
#
Stacks
1 2 3 4 5 6 7
1 2 5 14 42 132 429
8 9 10 11 12 13 14
1430 4862 16796 58786 208012 742900 2674440
15 16 17 18 19 20
9694845 35357670 129644790 477638700 1767263190 6564120420
2n 2n along the stack, the number of ways that D will exceed S will be nþ1 ¼ n1 . Hence 2n 2n 1 2n the required number of stacks is n n1 ¼ nþ1 n , which is the desired result. The Catalan numbers are tabulated in Table 3.3.2 for n ¼ 1 to 20.
Sampling with Replacement and without Ordering
We shall now consider forming r-combinations with replacement and without ordering. In this case we pick r balls with replacement from balls in n distinct categories such as red, blue, green, and brown. We do not distinguish among balls of the same color, and the distinction is among different colors. For example, if we are picking 6 balls from the 4 color categories, then it can be arranged as shown in Figure 3.3.3. There are now 2 red balls, 1 blue ball, no green balls, and 3 brown balls. The bar symbol | separates the four categories. The arrangement cannot start with a bar or end with a bar; hence we have 4 2 1 ¼ 3 bars. We now have to choose 6 balls from 6 balls and 3 bars without replacement and without ordering. There are then 987 9 9 ¼ ¼ ¼ 84 3 6 321 ways of picking 6 balls from 4 different categories. Generalizing this result, if there are n distinct categories of balls, and if we have to pick r balls with replacement and without regard to order, then Number of r-combinations ¼
n1þr r
(3:3:7)
This is the celebrated Bose –Einstein statistics, where the elementary particles in the same energy state are not distinguishable.
FIGURE 3.3.3
3.3 Combinations
33
Example 3.3.4 An urn contains red, blue, green, yellow, white, and black balls. In how many ways can we choose 1. 10 balls? This is a direct application of Eq. (3.3.7) with n ¼ 6, r ¼ 10 and the number of ways is 6 1 þ 10 15 15 ¼ ¼ ¼ 3003 10 10 5 2. 20 balls with least 2 colors each? If we draw 2 balls of each color, then we have drawn 12 balls, leaving us to draw 20 – 12 ¼ 8 balls to be drawn without restriction. Hence, applying Eq. (3.2.7) with n ¼ 6 and r ¼ 8, we obtain the number of ways as 13 13 61þ8 ¼ 1287 ¼ ¼ 5 8 8 3. 20 balls with no more than 3 blue balls? We first calculate the number of unrestricted ways of drawing 20 balls as 6 1 þ 20 25 ¼ ¼ 53,130 20 5 We next draw at least 4 blue balls out of 20 balls. Having drawn the 4 blue balls, the number of ways to draw the remaining 16 balls is 21 21 6 1 þ 16 ¼ 20,349 ¼ ¼ 5 16 26 Thus the number of ways of drawing 20 balls with no more than 3 blue balls is 53,130 2 20,349 ¼ 32,781. 4. 20 balls with at least 4 red balls, at least 2 green balls, at least 6 white balls, and no more than 3 brown balls? We first draw the lower bounded balls, namely, 4 þ 2 þ 6 ¼ 12, before we go to the upper-bounded brown balls. We then draw the remaining 8 balls in an unrestricted manner, and the number of ways is 61þ8 13 ¼ ¼ 1287 8 5 We have to subtract from this number the number of ways 4 brown balls can be picked from the 6 colors. This number is 61þ4 9 ¼ ¼ 126 4 4 Hence the number of ways in which all the restrictions will be satisfied is 1287 2 126 ¼ 1161. Example 3.3.5 the equation
We tackle a different kind of a problem. How many solutions are there to x1 þ x2 þ x3 þ x4 þ x5 þ x6 ¼ 24
where fxk, k ¼ 1, . . . , 6g are positive integers such that
34
Basic Combinatorics
1. xk . 2 for all k? We need xk 3 for all k. This takes up 6 3 ¼ 18 of the required total of 24, leaving us to solve the equation x1 þ x2 þ x3 þ x4 þ x5 þ x6 ¼ 6 in an unrestricted manner. Here we identify, in Eq. (3.2.7), n ¼ 6 corresponding to the number of xk values and r ¼ 6 corresponding to the righthand side of the equation. Hence the number of solutions is 11 11 61þ6 ¼ 462 ¼ ¼ 5 6 6 2. x1 . 2, x2 1, x3 . 3, x4 4, x5 . 2, and x6 2? Using x1 3, x3 4, x5 3, the number of restrictions add upto 3 þ 1 þ 4 þ 4 þ 3 þ 2 ¼ 17. This leaves 24 2 17 ¼ 7 unrestricted choices. Hence the number of solutions is 61þ7 12 12 ¼ ¼ ¼ 792 7 7 5 3. x2 , 5, x3 . 10? The number of balls that can be drawn with the restriction x3 11 corresponding to n ¼ 6 and r ¼ (24 2 11) ¼ 13 is given by 18 18 6 1 þ 13 ¼ 8568 ¼ ¼ 5 13 13 The number of draws that violate the additional restriction x2 , 5 is obtained by first finding the number of draws with the additional restriction x2 6. The number of ways corresponding to n ¼ 6 and r ¼ (24 2 11 2 6) ¼ 7 is given by 61þ7 12 12 ¼ ¼ ¼ 792 7 7 5 Hence the required result is 8568 2 792 ¼ 7776 ways. Example 3.3.6 Here is an interesting example to illustrate the versatility of the formula in Eq. (3.2.7). Four dice, each with only 4 faces marked 1,2,3,4, are tossed. We have to calculate the number of ways in which the tosses will result in exactly 1 number, 2 numbers, 3 numbers, and 4 numbers showing on the faces of the dice without regard to order and with replacement. The total number of ways is tabulated in Table 3.3.3.
TABLE 3.3.3 1,1,1,1 1,1,1,2 1,1,2,2 1,2,2,2 1,1,1,3 1,1,3,3
2,2,2,2 2,2,2,3 2,2,3,3 2,3,3,3 2,2,2,4 2,2,4,4
3,3,3,3 3,3,3,4 3,3,4,4 3,4,4,4 3,3,3,1
4,4,4,4 4,4,4,1 4,4,1,1 4,1,1,1 4,4,4,2
1,1,2,3 1,1,3,4 1,1,4,2
2,2,3,4 2,2,4,1 2,2,1,3
3,3,4,1 3,3,1,2 3,3,2,4
4,4,1,2 4,4,2,3 4,4,3,1
1,2,3,4
)
4 single digits
18 double digits
12 triple digits 1 quadruple digit
3.3 Combinations
35
We can see from Table 3.3.3 that there are 4 single digits, 18 double digits, 12 triple digits, and 1 quadruple digit totaling 35 arrangements without regard to order and with replacement. The problem can be stated as follows. Four dice each with 4 faces are tossed without regard to order and with replacement. Find the number of ways in which 1. This can be achieved. With n ¼ 4 and r ¼ 4 in Eq. (3.2.7), the number of ways is
41þ4 4
¼
7 7 ¼ 35 ¼ 3 4
which agrees with the total number in the table. 2. Exactly one digit is present. With n ¼ 4 and r ¼ 1, the number of ways is
41þ1 1
¼
4 ¼4 1
which agrees with the table. 3. Exactly 2 digits are present. The number of ways 2 digits can be drawn from 4 digits is 42 . The number of ways any particular 2 digits, for example f1,2g, will occupy a 4-digit number is a restricted case as in Example 3.2.4 with both 1 and 2 occupying two slots and the rest of the two remaining unrestricted. With n ¼ 2 and r ¼ 4 2 2 ¼ 2 in Eq. 21þ2 (3.2.7), the number of ways for this particular 3 sequence f1,2g is ¼ 2 2 ¼ 3. The total number using all the combinations are 42 32 ¼ 6 3 ¼ 18 ways. This agrees with the table. 4. Exactly 3 digits are present. The number of ways 3 digits can be drawn from 4 digits is 43 ¼ 4 ways. The number of ways 3 digits can be arranged in 4 slots such that all the three digits 3 are present is again given by the restriction 4formula 3 Eq. (3.2.7) 31þ(43) ¼ ¼ 3. Hence the total number of ways is 2 1 3 1 ¼ 12 ways, agreeing with the table. 5. Exactly 4 digits are present. The number of ways 4digits will be present is obtained from the argument in item 3 above, giving 44 41þ(4þ4) ¼1 0 Generalization
The result in Example 3.3.6 can be generalized to the tossing of n n-sided dice. The total number of sample points without regard to order and with replacement is, from Eq. (3.2.7)
n1þn n
¼
2n n
(3:3:8)
Out of these, the number of ways exactly r digits will occur can be derived as follows: The number of ways r digits can be chosen from n is nr , and if any particular sequence of r digits has to fill n slots, then the number of ways this can be done is obtained by substituting r ¼ (n 2 r) in Eq. (3.2.7), yielding
r 1 þ (n r) nr
¼
n1 nr
(3:3:9)
36
Basic Combinatorics
FIGURE 3.3.4
Since all the nr sequences are functionally independent, the number of ways N(n,r) is as follows, from Eq. (3.1.2): n1 n (3:3:10) ¼ N(n,r) ¼ nr r Using Eq. (3.3.10), N(n,r) tabulated for n ¼ 1, . . . , 8 and r ¼ 1, . . . , 8 in Fig. 3.3.4. In addition, we have the following interesting identity. Combining Eqs. (3.3.8) and (3.3.9), we have X n 2n 1 n n1 ¼ (3:3:11) n r nr r¼0 Equation (3.3.11) can also be expressed by substituting m ¼ n – 1, r ¼ n, and k ¼ r in the Vandermonde identity of Eq. (3.3.6) X X r n nþm n m nþn1 n n1 ¼ ¼ ¼ : (3:3:12) r k rk n k nk k¼0 k¼0
CHAPTER
4
Discrete Distributions
B 4.1
BERNOULLI TRIALS
We will elaborate on the Cartesian product already discussed in Chapter 1. Consider the tossing of an unfair coin with success given by fhead ¼ sg with probability p and failure given by ftail ¼ f g with probability q, and p þ q ¼ 1. The sample space is S1 ¼ fs, f g with field F ¼ ½S1 , 1, {s}, { f }. If the coin is tossed twice, then the sample space S2 ¼ fss, sf, fs, ffg is an ordered set from the Cartesian product S1 S1. The cardinality of S2 is 2 2 ¼ 4. The probability of two successes is p 2 and two failures is q 2. If we toss the coin n times, then the resulting sample space is Sn ¼ S1 S1 S1 S1 and the cardinality of Sn is 2n. The probability of n successes is p n.
Bernoulli Trials
Bernoulli trials are repeated functionally independent trials with only two events s and f for each trial. The trials are also statistically independent with the two events s and f in each trial having probabilities p and (1 2 p), respectively. These are called independent identically distributed (i.i.d.) trials. Example 4.1.1 Tossing an unfair coin 4 times constitutes 4 Bernoulli trials with heads ¼ s and tails ¼ f with Pfsg ¼ p and Pf fg ¼ 1 2 p. The sample space S is the Cartesian product with ordered 4-tuples. The cardinality of S is 24 ¼ 16 points. Example 4.1.2 Tossing a die 5 times can be considered Bernoulli trials if we define getting an ace as success, s with Pfsg ¼ 16 and not getting an ace as failure, f with Pf fg ¼ 56. The resulting sample space S is the Cartesian product with ordered quintuples with number of points equaling the cardinality 25 ¼ 32.
Bernoulli Distribution
Bernoulli distribution is the simplest of a discrete distribution. A single event is Bernoulli distributed if the probability of success is p and probability of failure is q with p þ q ¼ 1.
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
37
38
Discrete Distributions
Example 4.1.3 A single die is thrown. The probability of getting an ace is p ¼ 16 and the probability of not getting an ace is q ¼ 56. Hence this experiment is Bernoulli distributed.
B 4.2
BINOMIAL DISTRIBUTION
We are given an experiment S with an event A , S with PfAg ¼ Pfsuccessg ¼ p and PfAg ¼ Pffailureg ¼ q with p þ q ¼ 1. We repeat this experiment n times, and the resulting sample space Sn is the Cartesian product Sn ¼ {S S S} n times
containing 2n elementary events. We will now determine the probability b(k; n, p) that the event A occurs exactly k times in n Bernoulli trials. We shall address this problem in two parts. In the first part we will find the probability of k successes in a particular sequence in n trials. For example, one such sequence can be {s, s, f , s, f , f , . . . , s, f } exactly k successes
For the k independent successes the probability is p k and for the (n 2 k) independent failures the probability is q n – k. Since both successes and failures are independent, we have Probability{k successes in a given sequence} ¼ pk qnk
(4:2:1)
In the second part we address the situation of k successes in any sequence in n trials. The number of ways k successes can be obtained from n trials is by sampling without replacement and without regard to order. This number from Eq. (3.3.2) is kn . These realizations are mutually exclusive since the occurrence of one realization precludes the occurrence of another. Hence the probabilities will add, giving rise to n k nk Probability{k successes in any sequence in n trials} ¼ b(k; n,p) ¼ pq (4:2:2) k This is called the binomial distribution, and b(k;n,p) is called a probability mass function. over all k. Since We will now show that this is a probability distribution by summing n n k nk q is the kth term in the binomial expansion of ( p þ q) , we can write p k n n X X n k nk p q ¼ ( p þ q)n ¼ 1 b(k; n,p) ¼ (4:2:3) k k¼0 k¼0 The probability of getting utmost r successes is given by P{number of successes r} ¼
r X k¼0
b(k; n,p) ¼
r X n k nk pq k k¼0
(4:2:4)
and the probability of getting at least r successes is given by n X n k nk P{number of successes r} ¼ pq b(k; n,p) ¼ k k¼r k¼r n X
(4:2:5)
4.2 Binomial Distribution
39
Since n! grows very rapidly with n, computational difficulties in calculating b(k;n,p) can be avoided by using the recursive formula shown below with starting value b(0;n,p). n n k p n k nk kþ1 nk1 p q b(k þ 1; n,p) ¼ ¼ pq kþ1 q k kþ1 ¼ Example 4.2.1
nk p b(k; n,p) kþ1 q
(4:2:6)
A fair die is rolled 10 times, and we will calculate
1. The probability that an ace comes up exactly 4 times. Substituting p ¼ 16 and q ¼ 56, n ¼ 10, and k ¼ 4 in Eq. (4.2.3), we obtain 10! 1 4 5 6 P{6 aces in 10 trials} ¼ ¼ 0:05427 6!4! 6 6 2. The probability that utmost 4 aces show up. In probability 1 (above) we wanted exactly 4 aces coming up. Now we are looking into the possibility of 4 or fewer aces showing up. This probability is obtained as follows: k 10k 10! 1 5 P{number of aces 4} ¼ ¼ 0:98454 k!(10 k)! 6 6 k¼0 4 X
3. The probability that the number of aces showing is between 2 and 6 inclusive. Here we have to sum b(k: 10, 16) between 2 and 6. Performing the indicated summation, we have k 10k 10! 1 5 ¼ 0:22451 P{2 number of aces 6} ¼ k!(10 k)! 6 6 k¼2 k 5 10k The graph of the probability mass function 10i ( 16 P ) ( 10 ) is i shown 10iin Fig. 4.2.1, and that of the cumulative distribution function ki¼0 10i 16 56 is shown in Fig. 4.2.2 for k ¼ 0, . . . , 10. 6 X
FIGURE 4.2.1
40
Discrete Distributions
FIGURE 4.2.2
Example 4.2.2 We shall use the recursive form [Eq. (4.2.6)] to compute the binomial coefficients in Example 4.2.1 with initial condition b(0;n,p) ¼ 0.161506. These values are substituted in Eq. (4.2.4), and the first four steps are shown: 1 b 0; 10, ¼ 0:161506 6 1 10 1 0:161506 ¼ 0:323011 ¼ b 1; 10, 6 1 5 1 9 1 b 2; 10, ¼ 0:323011 ¼ 0:29071 6 2 5 1 8 1 b 3; 10, ¼ 0:29071 ¼ 0:155045 6 3 5 The true values and the computed values are shown in Table 4.2.1, and they are the same. Example 4.2.3 Two experiments are performed. In the first experiment 6 fair dice are tossed and the person wins if she gets 1 or more aces. In the second experiment 12 dice are tossed and the person wins if she gets 2 or more aces. We want to find in which experiment the probability of winning is higher.
TABLE 4.2.1 b(k;n,p) k
Computed
Calculated
k
Computed
Calculated
0 1 2 3 4 5
0.161506 0.323012 0.290711 0.155046 0.054266 0.013024
0.161506 0.323011 0.29071 0.155045 0.054266 0.013024
6 7 8 9 10
0.002171 0.000248 0.000019 0.000001 0
0.002171 0.000248 0.000019 0.000001 0
4.3 Multinomial Distribution
41
The probability of getting one or more aces out of 6 tosses is the same as the complement of the probability of getting no aces in 6 throws. This is obtained from Eq. (4.2.5) with n ¼ 6, p ¼ 16 and q ¼ 56: 0 6 1 5 6 1 P{no aces in 6 throws} ¼ 1 ¼ 0:6651 0 6 6 The probability of getting 2 or more aces in 12 throws is the same as the complement of the sum of the probabilities of getting no aces and one ace in 12 throws. With n ¼ 12, we obtain from Eq. (4.2.5) 1 P{no aces in 12 throws} P{one ace in 12 throws} 0 12 1 11 12 12 1 5 1 5 ¼1 1 6 6 6 6 0 1 ¼ 1 0:1122 0:2692 ¼ 0:6186 Hence the probability of getting one or more aces in 6 throws is better than getting two or more aces in 12 throws. Example 4.2.4 An urn consists of 40 red balls and 60 green balls. What is the probability of getting exactly k red balls in a sample size of 10 if the sampling is done 1. With replacement? The meaning of “with replacement” is to pick a ball from the 100 in the urn note its color and replace it and pick again, and this process is continued for 10 times. Thus Pfred ballg ¼ p ¼ 25 and Pfgreen ballg ¼ q ¼ 35 and with n ¼ 10, the probability of k red balls in 10 trials is given by k 10k 2 2 3 10 b k; n, ¼ k 5 5 5 2. Without replacement? Here we sample the balls and do not replace it in the urn. 100 Thus, the number of ways of getting 10 balls from 100 balls 40 is 10 . Out of these 10 balls, the number of ways of getting kred balls is k and the number of ways of getting (10 2 k) blue balls is 1060 k . Since getting k red balls and 40 (10 602k) blue balls are functionally independent, the total number of ways are k 10 k using the product rule. Hence the probability P(k) of k red balls without replacement is given by 40 60 k 10 k P(k) ¼ 100 10
B 4.3
MULTINOMIAL DISTRIBUTION
The binomial distribution of the previous section can be generalized to n repeated independent trials where each trial has m outcomes fEi, i ¼ 1, . . . , mg with probabilities PfEig ¼ fpi, i ¼ 1, . . . , mg and p1 þ p2 þ þ pm ¼ 1
(4:3:1)
42
Discrete Distributions
Each of the outcomes fEi, i ¼ 1, . . . , mg occurs fki, i ¼ 1, . . . , mg times with k1 þ k2 þ þ km ¼ n
(4:3:2)
The probability Pfk1, k2, . . . , kmg is given by the multinomial probability distribution, n pk1 pk2 pkmm P{k1 , k2 , . . . , km } ¼ k1 , k2 , . . . , km 1 2 n! pk1 pk2 pkmm k1 !k2 !; km ! 1 2
¼
(4:3:3)
If m ¼ 2, the multinomial distribution reduces to the binomial distribution. Example 4.3.1 A pair of dice is rolled 8 times. We will find the probability that 7 or 11 occurs thrice; 2, 3, or 12 twice; any pair twice; and any other combination once. For each rolling of a pair there are four events E1, E2, E3, E4 and their probabilities are obtained from Table 2.2.1 and are shown below: E1 ¼ {7 or 11}
8 36 4 P{E2 } ¼ p2 ¼ 36 6 P{E3 } ¼ p3 ¼ 36 18 P{E4 } ¼ p4 ¼ 36 P{E1 } ¼ p1 ¼
E2 ¼ {2, 3 or 12} E3 ¼ {any pair} E4 ¼ {other combination}
In Eq. (4.3.3) we substitute k1 ¼ 3, k2 ¼ 2, k3 ¼ 2, k4¼1 with p1, p2, p3, p4 as given above, yielding 3 2 2 1 8! 8 4 6 18 P{3, 2, 2, 1} ¼ ¼ 0:003161 3!2!2!1! 36 36 36 36
B 4.4
GEOMETRIC DISTRIBUTION
Closely allied to the binomial distribution is the geometric distribution. A sequence of independent Bernoulli trials with p as the probability of success can be modeled in two different ways. The distribution of the number of successes k is binomial. The waiting time until the first success occurs is geometrically distributed. If the first success is to occur at the kth trial, then we must have (k 2 1) failures preceding the first success. Since the trials are independent, this probability is given by g(k,p) ¼ (1 p)k1 p, k ¼ 1,2, . . .
(4:4:1)
This is the geometric distribution with parameter p. Further, the waiting times between each success and the next are independent and are geometrically distributed. We can show that the sum to infinity of g(k,p) is indeed 1 as shown below: 1 X k¼1
g(k,p) ¼ p
1 X k¼1
(1 p)k1 ¼
p ¼1 1 (1 p)
4.4 Geometric Distribution
43
Example 4.4.1 In an urn there are r red balls and b blue balls. Balls are selected at random with replacement so that the first blue ball is obtained. We have to find the probability that 1. Exactly k draws are needed for the first blue ball. The probability of a blue ball is p ¼ b/(b þ r). Substituting this value p in Eq. (4.4.1), the required probability is b b k1 b g k, ¼ 1 bþr bþr bþr k1 r b br k1 ¼ ¼ bþr b þ r (b þ r)k 2. At least m draws needed for the first blue ball. The first success occurs at the mth draw after (m 2 1) failures. Since the failure probability is q ¼ [r/(b þ r)], we have from the definition of the geometric distribution this probability is [r/(b þ r)]m21. The same result can also be obtained from the cumulative summation as follows: k1 1 1 X b b X r g k, ¼ bþr b þ r k¼m b þ r k¼m # m1 " 2 b r r r þ ¼ 1þ þ bþr bþr bþr bþr 2 3 m1 m1 b r 1 r 6 7 ¼ ¼ 4 r 5 bþr bþr bþr 1 bþr This geometric distribution is shown in Fig. 4.4.1 for r ¼ 100, b ¼ 20, and k ¼ 1, . . . , 10 and the cumulative distribution in Fig. 4.4.2.
FIGURE 4.4.1
44
Discrete Distributions
FIGURE 4.4.2
B 4.5
NEGATIVE BINOMIAL DISTRIBUTION
The binomial distribution finds the probability of exactly k successes in n independent trials. The geometric distribution found the number of independent trials needed for the first success. We can generalize these two results and find the number of independent trials required for k successes. This distribution is known as the negative binomial or the Pascal distribution. Let us now define the following events to find this distribution: E ¼ n trials for exactly k successes A ¼ exactly (k 1) successes occur in (n 1) trials B ¼ nth trial results in kth success Since E ¼ A > B and A and B are independent, we have P{E} ¼ P{A} P{B}
(4:5:1)
We will now compute PfAg. In a sequence of (n 2 1) trials we have exactly (k 2 1) successes and [n 2 1 2 (k 2 1)] ¼ (n 2 k) failures. Since these trials are independent, the probability of that particular sequence is p k21q n21. There are n1 k1 mutually exclusive sequences, and hence n 1 k1 nk p q P{A} ¼ b(k 1; n 1, p) ¼ (4:5:2) k1 Since the probability PfBg ¼ p, we can substitute this and Eq. (4.5.2) in Eq. (4.5.1) and write n 1 k1 nk n 1 k nk p q nb(k; n,p) ¼ P{E} ¼ p p q , n ¼ k, k þ 1, . . . (4:5:3) ¼ k1 k1 We have to show that the sum of nb(k;n,p) from n ¼ k to 1 equals 1. Or 1 X n 1 k nk pq nb(k; n,p) ¼ ¼1 k1 n¼k n¼k
1 X
(4:5:4)
4.5
TABLE 4.5.1
Negative Binomial Distribution
45
Negative binomial
n
Probability
n
Probability
n
Probability
5 6 7 8
0.16807 0.252105 0.226894 0.158826
9 10 11 12
0.095296 0.05146 0.02573 0.01213
13 14 15
0.005458 0.002365 0.000993
By substituting (n 2 k) ¼ m, we can rewrite Eq. (4.5.4) as 1 1 1 X X kþm1 k m X kþm1 k m pq ¼ p q ¼1 nb(k; k þ m,p) ¼ k1 m m¼0 m¼0 m¼0
(4:5:5)
The Taylor series expansion of (1 2 x)2k yields (k)( k 1) (k)(k 1)(k 2) (x)2 þ (x)3 þ 2! 3! 1 X kþm1 m ¼ x m m¼0
(1 x)k ¼ 1 þ kx þ
(4:5:6)
Substituting x ¼ q in Eq. (4.5.6) and (1 2 q) ¼ p, we obtain 1 1 X X kþm1 m kþm1 k m k p ¼ q , p q , which is Eq. (4:5:5) or 1 ¼ m m m¼0 m¼0 In Eq. (4.5.3), if we substitute k ¼ 1, then we get the geometric distribution. Example 4.5.1 A computer software company wants to recruit 5 new engineers. The probability that a person interviewed is given a job offer is 0.7. We want to find the distribution of the number of people interviewed for the 5 positions. We use Eq. (4.5.3) with p ¼ 0.7 and k ¼ 5. Thus n1 P{E ¼ n} ¼ (0:7)5 (0:3)n5 , n ¼ 5, 6; . . . 51 PfE ¼ ng is shown for n ¼ 5,6, . . . ,15 in Table 4.5.1, and the probability mass function is shown in Fig. 4.5.1.
FIGURE 4.5.1
46
Discrete Distributions
From the table and the figure we see that the maximum probability of 0.2521 occurs at k ¼ 6 and it is most likely that company will interview exactly 6 persons for the 5 jobs.
B 4.6
HYPERGEOMETRIC DISTRIBUTION
The binomial distribution is obtained while sampling with replacement. Let us select a sample of n microprocessor chips from a bin of N chips. It is known that K of these chips are defective. If we sample with replacement then the probability of any defective chip is K/N. The probability that k of them will be defective in a sample size of n is given by the binomial distribution since each drawing of a chip constitutes a Bernoulli trial. On the other hand, if we sample without replacement, then the probability for the first chip is K/N, the probability for the second chip is (K 2 1)/(N 2 1), and for each succeeding sample the probability changes. Hence the drawings of the chips do not constitute Bernoulli trials and binomial distribution does not hold. Hence we have to use alternate methods to arrive at the probability of finding k defective chips in n trials. The number of sample points in this sample space is choosing n chips out of the total N chips without regard to order. This can be done in Nn ways. Out of this sample size of n, there are k defective chips and (n 2 k) good chips.The number of ways of picking k defective chips out of the total of K defective chips is Kk and the number of ways of drawing (n 2 k) good chips out of the remaining (N 2 K) good chips is NK nk . We denote the total number of ways of drawing a sample of n chips with k bad chips as the event E. Since these drawings are functionally independent the number of ways is Kk NK nk . Hence the probability of the event E is given by K NK k nk (4:6:1) P{E ¼ k} ¼ N n This is known as the hypergeometric distribution. We came across this distribution in Example 4.2.4. Since the summation over all k must be one, we must have the following in Eq. (4.6.1): n X K N K N ¼ (4:6:2) k n k n k¼0 This result follows directly from Vandermonde’s identity of Eq. (3.3.6) given by n X n m nþm ¼ (3:3:6) r k rk k¼0 with n ¼ K, m ¼ N 2 K, and r ¼ n. Example 4.6.1 (Mass Megabucks) Massachusetts megabucks lottery is played as follows. A player has to choose 6 numbers from a slate of 42 numbers. Twice a week the lottery commission draws 6 numbers. If all the 6 numbers of the player match all the 6 numbers drawn by the commission, the player wins megabucks. There are smaller prizes for matching 5, 4, or 3 numbers. We have to calculate the probability of winning megabucks or the smaller prizes.
4.6 Hypergeometric Distribution
TABLE 4.6.1
47
Mass megabucks
k
N(k)
C(k)
P(k)
0 1 2 3 4 5 6
1947792 2261952 883575 142800 9450 216 1
3 2 6 37 555 24286 5,245,786
0.371306035 0.431194105 0.168435197 0.02722185 0.001801446 0.000041176 0.000000191
Let k ¼ 0,1, . . . , 6. Substituting N ¼ 42, n ¼ 6, and K ¼ 6, we can find from Eq. (4.6.1) PfE ¼ kg: 6 42 K k 6k N(k) , k ¼ 0,1, . . . , 6 ¼ P{E ¼ k} ¼ D(k) 42 6 The results for k ¼ 0, . . . , 6 in Table 4.6.1. In this table, the second column are tabulated N(k) ¼ 6k 42K is the number of ways k digits can be formed from 6 digits. In the third 6k column C(k) ¼ D(k)/N(k) represents the chance of winning with k digits rounded to the nearest digit and in the fourth column P(k) is the probability of winning. Example 4.6.2 Hypergeometric distribution is used to estimate an unknown population from the data obtained. To estimate the population N of tigers in a wildlife sanctuary, K tigers are caught, tagged, and released. After the lapse of a few months, a new batch of n tigers is caught, and k of them are found to be tagged. It is assumed that the population of tigers does not change between the two catches. The total number of tigers N in the sanctuary is to be estimated. In Eq. (4.6.1), the known quantities are K, k, and n. The total number N has to be estimated in the following equation: K NK k nk Pk {N} ¼ N n We will find the value of N that maximizes the probability PkfNg. Such an estimate is called the maximum-likelihood estimate, which will be discussed in Chapter 18. Since the numbers are very large, we investigate the following ratio: K NK , K N1K k nk k nk Pk {N} ¼ N N1 Pk {N 1} n n N K N1 nk n Nn NK ¼ ¼ N N1K N NKnþk n nk
48
Discrete Distributions
The ratio (Pk {N})=(Pk {N 1}) is greater than 1 if (N 2 n)(N 2 K ) . N (N 2 K 2 n þ k), Nk , nK, or N , nK/k. Conclusion. If N is less than nK/k, then Pk(N) is increasing, and if N is greater than nK/k, then Pk(N ) is decreasing. Hence the maximum value of Pk(N ) occurs when N is equal to the nearest integer not exceeding nK/k. Thus, if K ¼ 100, n ¼ 50, and k ¼ 20, then the population of tigers is 100.50/20 ¼ 250. Approximation to Binomial Distribution
If N and K are very large with respect to n and k, then the hypergeometric distribution can be approximated by the binomial distribution as shown below: P{E ¼ k} ¼
K k
NK nk
N n
¼
k! (N K)! n!(N n)! k!(K k)! (n k)!(N K n þ k)! N!
¼
n! K! (N n)! (N K)! k!(n k)! (K k)! N! (N K n þ k)!
¼
n K(K 1) (K k þ 1) k N(N 1) (N k þ 1)
(N K)(N K 1) (N K n þ k þ 1) (N k)(N K 1) (N k n þ k þ 1)
(4:6:3)
Since K k and N n, Eq. (3.3.7) can be approximated as n K K K (N K)(N K) (N K) P{E ¼ k} ¼ N N N k N N N k times
(4:6:4)
Nk times
Approximating p ffi K/N and q ffi (N 2 K )/n in Eq. (3.3.8), we obtain n k nk P{E ¼ k} ffi pq k
(4:6:5)
which is a binomial distribution.
B 4.7
POISSON DISTRIBUTION
Perhaps three of the most important distributions in probability theory are (1) Gaussian, (2) Poisson, and (3) binomial. We have already discussed the binomial and other allied distributions that are characterized by discrete time (n) and discrete state, that is, the cumulative distribution function has jumps occurring only at discrete points. In the regular Poisson distribution the cumulative distribution has unit jumps at random times and not at discrete times. Thus, it is discrete in state but continuous in time.
4.7 Poisson Distribution
49
The Poisson distribution takes values 0,1, . . . k, . . . at random points with a parameter l and is given by p{k ; l} ¼ el
lk , k!
k0
(4:7:1)
We will show that this is a probability function by summing k from 0 to 1 as shown below: 1 X
p{k ; l} ¼
k¼0
1 X
el
k¼0
1 X lk lk ¼ el ¼ el el ¼ 1 k! k! k¼0
(4:7:2)
Similar to Eq. (4.2.6) we can derive a recursive form for evaluating the Poisson distribution as shown below: 1 X k¼0
p{k þ 1 ; l} ¼
1 X k¼0
el
1 lkþ1 l X lk l p{k ; l} ¼ el ¼ (k þ 1)! k þ 1 k¼0 k! k þ 1
(4:7:3)
This last equation is solved from the starting value p(0, l) ¼ e 2l. Example 4.7.1 During a certain experiment the number of a particles passing through a sensor in a 1-ms interval is 3.2. The Poisson distribution from Eq. (4.7.1) with l ¼ 3.2 can be written as p{k ; 3:2} ¼ e3:2
(3:2)k , k!
k0
This is graphed in Fig. 4.7.1. We want to find the probability that 1. Six a particles pass through the sensor in a given millisecond. Substituting l ¼ 3.2 and k ¼ 6, we obtain p{6 ; 3:2} ¼ e3:2
ð3:2Þ6 ¼ 0:06079 6!
FIGURE 4.7.1
50
Discrete Distributions
2. No more than three a particles pass through the sensor. We have to find the cumulative distribution given by 3 X
p{k ; 3:2} ¼
k¼0
3 X k¼0
e3:2
(3:2)k ¼ 0:6025 k!
Example 4.7.2 The average number of customers arriving at the service counter in a supermarket is l, and the probability of k customers arriving in the interval (0,t) minutes is given by the Poisson distribution pfk ; lg ¼ e 2l[(lt)k/k!]. In the nonoverlapping intervals (0,t1) and (t1,t), we can write the following Poisson law P{n1 customers in (0,t1 ) and n2 in (t1 ,t)} ¼ P{n1 customers in (0,t1 )} P{n2 customers in (t1 ,t)} and the two events are independent in disjoint intervals. We will find the conditional probability, Pfn1 customers in (0,t1) j (n1 þ n2) customers in (0,t)g. From the definition of conditional probability, we can write P{n1 ; (0,t1 ) j (n1 þ n2 ); (0,t)} ¼ ¼
P{½n1 ; (0,t1 )½(n1 þ n2 ); (0,t)} P{(n1 þ n2 ); (0, t)} P{½n1 ; (0,t1 )½n2 ; (t1 ,t)} P{(n1 þ n2 ); (0, t)}
elt1 (lt1 )n1 el(tt1 ) ½l(t t1 )n2 ¼ n1 ! n2 ! ¼
(lt1 )n1 ½l(t t1 )n2 (n1 þ n2 )! (lt)n1 þn2 n1 !n2 !
¼
t1n1 (t t1 )n2 (n1 þ n2 )! tn1 þn2 n1 !n2 !
,
elt (lt)n1 þn2 (n1 þ n2 )!
which yields a surprising result independent of l! Poisson distribution has been used in a number of applications such as the 1. Number of telephone calls coming into an exchange 2. Number of cars entering a highway ramp 3. Number of packets transmitted along wireless lines 4. Number of customers at the checkout counter 5. Number of misprints in a book 6. Number of photons emitted Approximation to Binomial Distribution
Poisson distribution is also used for approximating a binomial distribution with parameters (n,p) under the condition that n is large compared to k and p is small enough that the product np ¼ l is of moderate magnitude. As a rule of thumb, l may
4.7 Poisson Distribution
51
be a single digit. We shall now derive this approximation: n k nk n! pk (1 p)nk b(k;n,p) ¼ pq ¼ k!(n k)! k n(n 1) (n k þ 1) l k l nk ¼ 1 k! n n ¼
n(n 1) (n k þ 1) lk (1 l=n)n nk k! (1 l=n)k
(4:7:4)
As n becomes very large and for moderate l, the quantity (1 l=n)n can be approximated by e 2l and the quantity (1 l=n)k can be approximated as 1. Finally, as k is much smaller than n, the quantity ½n(n 1) (n k þ 1)=nk can be approximated by 1. Thus the binomial distribution b(k;n,p) can be approximated as b(k; n,p)
lk l e , k!
k0
(4:7:5)
Example 4.7.3 A computer system consists of 1000 microchips, and the probability of failure of a microchip is 0.005. We have to find the probability of k microchips failing using both the binomial distribution and its Poisson approximation for k ¼ 0,1, . . . ,15. We will also find the percentage error between the two. Using binomial distribution, k chips failing in 1000 trials given by b(k; 1000,0.005) ¼ b(k) is b(k; 1000,0:005) ¼ b(k) ¼
1000! 0:005k 0:9951000k k!(1000 k)!
The value b(k) can be solved using the recursive formula developed in Eq. (4.2.6), and the steps are 1000 k 0:005 b(k) k þ 1 0:995 1000 k 1 ¼ b(k) k þ 1 188
b(k þ 1) ¼
Using the Poisson approximation we have np ¼ l ¼ 5, and substituting in the Poisson distribution, we have k 5 5 p(k ; l) ¼ p(k ; 5) ¼ p(k) ¼ e k! Both b(k) and p(k) are shown for k ¼ 0,1, . . . , 15 in Table 4.7.1 along with the percentage error. Table 4.7.1 is also graphed in Fig. 4.7.2 for the binomial distribution and the Poisson approximation. The figure shows that the approximation is very close. The error in the approximation is also graphed in Fig. 4.7.3. From the table and the graphs we see that the approximation becomes worse as k is increased and the approximation is excellent in the vicinity of k ¼ 5 ¼ l, about 0.05 –0.25%.
52
Discrete Distributions
TABLE 4.7.1
Comparison of binomial with Poisson approximation
k
Binomial
Poisson
%Error
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0.006654 0.033437 0.083929 0.140303 0.175731 0.175908 0.14659 0.104602 0.065245 0.036138 0.017996 0.008139 0.003371 0.001287 0.000456 0.000151
0.006738 0.03369 0.084224 0.140374 0.175467 0.175467 0.146223 0.104445 0.065278 0.036266 0.018133 0.008242 0.003434 0.001321 0.000472 0.000157
21.26208 20.75577 20.352343 20.050683 0.150021 0.250272 0.250272 0.14992 20.051188 20.353762 20.758823 21.267706 21.88207 22.603907 23.435549 24.379687
FIGURE 4.7.2
FIGURE 4.7.3
4.7 Poisson Distribution
53
Example 4.7.4 (See Ref. 47) We will revisit the two or more birthdays matching problem posed in Example 3.2.4, and solve it using the Poisson approximation and compare the results. In addition, we will use the Poisson approximation to find probability of matching three or more birthdays that would have been very difficult using the exact method. Given the total numbers of persons n 365, we will find, using the Poisson approximation, the probability of two or more birthdays matching. The probability of two birth1 days matching is 365 , and this we will set equal to p. The number n of ways we can obtain a sample of two without replacement and without ordering is 2 , each of which has the 1 of matching. Hence the 2n ways constitute Bernoulli trials and same probability of 365 1 is very small and since p ¼ 365 1 n(n 1) n ¼ ¼l np ¼ 2 365 730
is less than 10, all the conditions for approximation by a Poisson distribution are satisfied. Hence
P{2 or more people share the same birthday} ¼ 1 P{no birthdays match} n(n 1) l0 ¼ 1 p(0; l) ¼ 1 exp 730 0!
and [1 2 p(0 ; l)] is shown in Table 4.7.2 for the same values of n as given in Table 3.2.3 with the corresponding percentage error shown in Table 4.7.3. The percent error curve between the true value and the Poisson approximation for values of n ¼ 1, . . . ,100 is shown in Fig. 4.7.4. An examination of the error curve reveals that the Poisson approximation underestimates the true values, and for values of n higher than 82, the error is almost zero. Even then the maximum error is only about 1.46% occurring at n ¼ 26. As mentioned before, this approximation can be extended to the case of three or more matches. Approaching, as before, the number of 3-combinations is n3 ¼ n(n1)(n2) , and 6
TABLE 4.7.2
Poisson approximation for two matches
#
Probability
#
Probability
#
Probability
20 21 22 23 24 25 26
0.0405805 0.437488 0.468938 0.500002 0.530536 0.560412 0.589513
30 31 32 33 34 35 36
0.69632 0.720282 0.743058 0.764625 0.784972 0.804097 0.82201
44 45 46 47 48 49 50
0.925113 0.933618 0.941318 0.948266 0.954517 0.960121 0.965131
54
Discrete Distributions
TABLE 4.7.3
Error in Poisson approximation
#
%Error
#
%Error
#
%Error
20 21 22 23 24 25 26
1.369162 1.397497 1.420488 1.438108 1.450358 1.457273 1.458918
30 31 32 33 34 35 36
1.415262 1.392672 1.365868 1.335125 1.300742 1.263039 1.222352
44 45 46 47 48 49 50
0.833107 0.781948 0.731364 0.681649 0.633067 0.58586 0.540238
FIGURE 4.7.4
the probability p of three matches is
1 2 365 .
Substituting in the Poisson formula we obtain
P{3 or more share the same birthday} ¼ 1 P{no 3 birthdays match} n(n 1)(n 2) l0 ¼ 1 p(0; l) ¼ 1 exp 6 3652 0! n(n 1)(n 2) ¼ 1 exp 799350
This probability is graphed in Fig. 4.7.5 for values of n ¼ 1, . . . , 150 and tabulated in Table 4.7.4 for values of n ¼ 80 – 86, 90– 96, and 104 –110. Examination of Fig. 4.7.3 and Table 4.7.4 reveals that for an even chance for at least three matches, there must be about 84 people and for a 70% chance there must be about 100 people compared to 23 and 30 for at least two matches. This approximation technique can be extended to any number of matches from 2 to 365, whereas exact computation becomes extremely cumbersome, if not impossible, for anything more than two matches.
4.8 Logarithmic Distribution (Benford’s Law)
55
FIGURE 4.7.5
TABLE 4.7.4
B 4.8 4.8.1
Poisson approximation for three matches
#
Probability
#
Probability
#
Probability
80 81 82 83 84 85 86
0.460278 0.472929 0.485593 0.498257 0.510911 0.523543 0.536141
90 91 92 93 94 95 96
0.58597 0.598231 0.610393 0.622444 0.634375 0.646176 0.657838
104 105 106 107 108 109 110
0.745102 0.755146 0.764978 0.774593 0.783987 0.793155 0.802095
LOGARITHMIC DISTRIBUTION Finite Law (Benford’s Law)
First Significant Digit
Most physical data depend on units. Examples are rainfall in inches, land area in acres, blood pressure in millimeters of mercury, stock prices in dollars, distances in miles, and volume in cubic feet. If clusters of these quantities can be described by a probability distribution P(x), then such a probability distribution must be invariant under a scale change, so that P(k x) ¼ KP(x) (4:8:1) Ð Ð where k and K are scale factors. Since P(x)dx ¼ 1, we have P(k x)dx ¼ 1=k, and hence K ¼ 1=k is the normalization factor. Thus 1 P(kx) ¼ P(x) or kP(kx) ¼ P(x) k Differentiating Eq. (4.8.2) with respect to k, we obtain P(kx) þ k
d d(kx) P(kx) ¼0 d(kx) dk
(4:8:2)
56
Discrete Distributions
Substituting k ¼ 1 in the equation above, we have d x P(x) ¼ P(x) dx The solution for Eq. (4.8.3) is P(x) ¼
1 x
(4:8:4)
Equation (4.8.4) can be considered as a probability distribution only for finite values of x since the integral for infinite values of x diverges. We shall assume that the data are defined to some arbitrary base B rather than base 10. If X1 is the random variable corresponding to the occurrence of the first digit d to the base B, then the probability of occurrence of PX1(d) is given by Ð dþ1 1 dx d PX1 (d) ¼ dÐ B ¼ Ð x ; d ¼ 1,2, . . . , B 1 B1 1 P(x)dx 1 x dx 1 ln 1 þ ln (d þ 1) ln (d) d ¼ ¼ ln (B) ln (1) ln (B) Ð dþ1
P(x)dx
(4:8:5)
We can show that indeed Eq. (4.8.5) is a probability function as follows: B1 X 1 1 1 1 ln 1 þ ¼ ln 1 þ þ ln 1 þ þ ln 1 þ d 1 2 B1 1 ¼ ln
B 1 Y 1
1 234 B 1þ ¼ ln ¼ ln (B) d 123 B 1
(4:8:6)
Thus PX1(d ) over all digits 1 to B21 sums to 1. If the base B is 10, the generic ln becomes log, and the probability of the first significant digit d from 1 to 9 is, 1 PX1 (d) ¼ log 1 þ ; d ¼ 1,2, . . . , 9 (4:8:7) d This equation is called Benford’s law, first stated in 1938 by Benford [5]. It gives the probability of occurrence of the first significant digit in a cluster of distributions. Incidentally, 0 cannot be the first significant digit. It can be a significant digit only after the first one. Example 4.8.1 In a cluster of numbers from different scale factors, we want to find the occurrence of 1,2, . . . , 9 as the first significant digit. Using Eq. (4.8.7), we have 1 PX1 (1) ¼ log 1 þ ¼ 0:30103 1 The probability of 1 as the first significant digit is 30.1% and not a uniform distribution of 11.1% as one would expect! The probability of 2 is 17.6%. The probabilities of other significant digits are tabulated in Table 4.8.1 and plotted in Fig. 4.8.1. They show that 9 as the first significant digit has the least probability of occurrence of 4.6%. Example 4.8.2 As a realistic example we have taken the values of physical constants (73 in number), in Chapter 17 of Ref. 18 and tabulated the significant digits from 1 to 9 in
4.8 Logarithmic Distribution (Benford’s Law)
TABLE 4.8.1
57
Benford distribution
d
fD(d )
1 2 3 4 5 6 7 8 9
0.30103 0.176091 0.124939 0.09691 0.079181 0.066947 0.057992 0.051153 0.045757
FIGURE 4.8.1
Table 4.8.2. The physical constants have totally different units. The probabilities of the significant digits are shown in Fig. 4.8.2 along with the graph of Benford’s law. The table and the graphs show that even for a small sample of 73, the probabilities follow the logarithmic distribution reasonably closely except for the number 3. In conclusion, in a mixed population of different scale factors, the probabilities of the digits 1 to 9 are logarithmically distributed and are not uniform—a surprising result!
TABLE 4.8.2 k
p(k)
1 2 3 4 5 6 7 8 9
0.342 0.205 0.055 0.096 0.096 0.055 0.027 0.041 0.082
58
Discrete Distributions
FIGURE 4.8.2
Second Significant Digit
We shall now compute the occurrence of 0,1, . . . , B 2 1 as the second significant digit. Note that 0 can occur as a subsequent significant digit after the first. We will only consider numbers of only first-order magnitude since the pattern will be the same for any other order. With base B and the total number of digits being B 2 1, the occurrences of 2 as the second significant digit is 1.2,2.2, . . . (B 2 1) . 2. Figure 4.8.3 shows in logarithmic scale the arrangement of numbers for base 10. Thus, the probability of 2 as the second significant digit occurring after the first digit 1 is given by an equation similar to Eq. (4.8.5): 13 ln ln (1:3) ln (1:2) 12 P{2 as second digit after first digit 1} ¼ ¼ ln (B) ln (1) ln (B)
(4:8:8)
If X2 is the random variable corresponding to the second digit, then the probability of 2 occurring after all the B 2 1 first digits is given by 13 23 (B 1)3 ln þ ln þ þ ln 12 22 (B 1)2 PX2 (2) ¼ ln (B)
(4:8:9)
Summing Eq. (4.8.10) over all the first digits d ¼ 0, . . . , B 2 1, we obtain the probability
FIGURE 4.8.3
4.8 Logarithmic Distribution (Benford’s Law)
of occurrence of the second significant digit, PX2(d ), as B1 1 X 1 PX2 (d) ¼ ln 1 þ ; d ¼ 0,1, . . . ,B 1 ln (B) k¼1 kB þ d ! B 1 Y 1 1 ln 1þ ¼ ln (B) kB þ d k¼1 It can be shown that " B1 B 1 Y X ln 1þ d¼0
k¼1
1 kB þ d
#
" ¼ ln
B 1 B 1 Y Y d¼0 k¼1
1 1þ kB þ d
59
(4:8:10)
# ¼ ln (B)
(4:8:11)
and hence, PX2(d ) is a probability function. We can extend the preceding analysis to obtain a general probability function for the (n þ 1)st significant digit after the first one digit n ¼ 0,1, . . . , B21 as shown below. The third digit after the first one can be obtained from Eq. (4.8.10). The occurrence of 3 as the third significant digit is 103, 113, . . . , 1(B 2 1)3; 203, 213, . . . , 2(B 2 1)3; . . . ; (B 2 1)03,(B 2 1)13, . . . , (B 2 1)(B 2 1)3. Thus, there are B(B 2 1) possible ways in which the digit 3 or any other digit d ¼ 0,1, . . . , B21 can occur as the third significant number. An equation analogous to Eq. (4.8.9) can be written for the probability of 3 occurring as the third significant digit: 2 1 1 BX 1 PX3 (3) ¼ ln 1 þ (4:8:12) ln (B) k¼B kB þ 3 Similar to Eq. (4.8.10), we can write the probability of any digit n occurring as the third significant digit: 2 1 1 BX 1 ln 1 þ PX3 (d) ¼ ln (B) k¼B kB þ d ! 2 BY 1 1 1 ln 1þ ¼ ; ln (B) kB þ d k¼B
d ¼ 0, . . . , B 1
(4:8:13)
We can extend Eq. (4.8.13) to all (n þ 1) significant digits after the first with n ¼ 1,2, . . . , B 2 1: Bn1 1 X 1 ln 1 þ PX(nþ1) (d) ¼ ln (B) k¼Bn1 kB þ d n ¼ 1,2, . . . , B 1;
d ¼ 0, . . . , B 1
(4:8:14)
Example 4.8.3 Using Eq. (4.8.14) with the base B ¼ 10, we will find the occurrence of the digits 0,1, . . . , 9 occurring as the first, second, . . . , tenth significant digit, noting that the probability of occurrence of 0 as the first significant digit is 0. Equation (4.8.14) becomes n 10 1 X 1 1 PX(nþ1) (d) ¼ ln 1 þ ln (10) k¼10n1 k 10 þ d n ¼ 0,1, . . . , 9;
d ¼ 0,1, . . . , 9
60
Discrete Distributions
TABLE 4.8.3
Benford distribution for 1 – 9 significant digits
d
PX1(d )
PX2(d )
PX3(d)
PX4(d )
PX5(d )
PX6(d )
PX7(d )
0 1 2 3 4 5 6 7 8 9
0 0.30103 0.176091 0.124939 0.09691 0.079181 0.066947 0.057992 0.051153 0.045757
0.119679 0.11389 0.108821 0.10433 0.100308 0.096677 0.093375 0.090352 0.08757 0.084997
0.101784 0.101376 0.100972 0.100573 0.100178 0.099788 0.099401 0.099019 0.098641 0.098267
0.100176 0.100137 0.100098 0.100059 0.100019 0.09998 0.099941 0.099902 0.099863 0.099824
0.100018 0.100014 0.10001 0.100006 0.100002 0.099998 0.099994 0.09999 0.099986 0.099982
0.100002 0.100001 0.100001 0.100001 0.1 0.1 0.099999 0.099999 0.099999 0.099998
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
FIGURE 4.8.4
The values of PX (nþ1)(d) have been tabulated in Table 4.8.3 for values of d ¼ 0, . . . , 9 and n ¼ 1, . . . , 6. It is interesting to note that the first significant digit is considerably different from uniform distribution. As the level of significance increases, the distribution tends to be more and more uniform and the significant digit is practically uniform. In fact, the third significant digit is almost a uniform distribution as shown in Fig. 4.8.4. Benford’s law is being used widely by the Internal Revenue Service for detecting fraudulent tax returns.
4.8.2
Infinite Law
We will also present two versions of a logarithmic distribution where the number of outcomes are infinite: Version 1. The first version is defined by p(k,u) ¼
uk ; k ln (1 u)
0 , u , 1, k 1
(4:8:15)
4.8 Logarithmic Distribution (Benford’s Law)
61
TABLE 4.8.4 k
P(u,k)
1 2 3 4 5 6 7 8 9 10
0.390865 0.175889 0.105534 0.071235 0.051289 0.038467 0.029675 0.023369 0.018695 0.015143
where u is the shape parameter and k are the infinite outcomes. Here u is between 0 and 1, and k is greater than or equal to 1. We will show that this is a probability distribution by summing over all k: 1 X
p(k,u) ¼
k¼1
1 X uk 1 uk ln (1 u) ¼1 ¼ ¼ ln (1 u) k ln (1 u) ln (1 u) k¼1 k k¼1
1 X
(4:8:16)
In Eq. (4.8.15), as u tends to zero, p(k,u) tends to 1 as shown below: uk kuk1 (1 u) ¼1 ¼ lim u!0 k ln (1 u) u!0 k
lim p(k,u) ¼ lim
u!0
(4:8:17)
The probability mass function (pmf) given by Eq. (4.8.15) is tabulated for k ¼ 1, . . . ,10 and u ¼ 0.9 in Table 4.8.4 and shown in Fig. 4.8.5. This logarithmic distribution is used to model the number of items purchased by a customer in a supermarket in a given amount of time.
FIGURE 4.8.5
62
Discrete Distributions
Version 2. The second version of this logarithmic distribution is obtained by substituting l ¼ (1 2 u) in Eq. (4.8.15) to obtain p(k,l) ¼
(1 l)k : 0 , l , 1, k 1 k ln (l)
(4:8:18)
Here, as l tends to 1, p(k,l) tends to 1 as shown below: (1 l)k klk1 l ¼1 ¼ lim u!0 k ln (l) u!0 k
(4:8:19)
lim p(k,l) ¼ lim
u!0
B 4.9
SUMMARY OF DISCRETE DISTRIBUTIONS
Assumptions: Pfsuccessg ¼ p; Pffailureg ¼ q; p þ q ¼ 1. Distribution Type
Definition
Bernoulli
8 < p; b(p,q) ¼ q; : 0;
Binomial
8 < n k nk pq ; k b(k; n,p) ¼ : 0;
k¼1 k¼0 otherwise
Multinomial
p(k1 ,k2 , . . . , km ) ¼ m X
ki ¼ n;
i¼1
m X
k ¼ 0,1, . . . , n otherwise
n k1 ,k2 , . . . , km
pi ¼ 1
i¼1
qk1 ; 0;
k ¼ 1, 2, . . . k0
Geometric
g(k,p) ¼
Negative binomial: Pascal
8 < n 1 k nk pq ; k1 nb( p) ¼ : 0;
Hypergeometric
pk11 pk22 pkmm
8 K NK > > > k > < n k ; p(n,k) ¼ N > > > > : n 0;
k ¼ k, k þ 1, . . . otherwise
k ¼ 0,1, . . . , n
9 > > > > =
otherwise
> > > > ;
(Continued)
4.9
Summary of Discrete Distributions
Distribution Type
Poisson
Logarithmic – Benford
63
Definition
8 <
9 k = l l p(k,l) ¼ e k! ; k ¼ 0,1, . . . : ; 0; otherwise 1 1 p(d) ¼ log 1 þ ; d ¼ 1, . . . , 10N 1 N d
Logarithmic 1—semiinfinite
p(k,u) ¼
uk ; k ln (1 u)
Logarithmic 2—semiinfinite
p(k,u) ¼
(1 u)k ; k ln (u)
0 , u , 1, k 1 0 , u , 1, k 1
CHAPTER
5
Random Variables
B 5.1
DEFINITION OF RANDOM VARIABLES
What is a random variable? An outlandish definition would be that it is neither random nor a variable! The real definition of a random variable is that it is a function X that assigns a rule of correspondence for every point j in the sample space S called the domain, a unique value X(j) on the real line R called the range. Let F be the field associated with the sample space and FX be the field associated with the real line. The random variable X induces a probability measure PX in R and hence X is a mapping of the probability space {S, F, P} to the probability space {R, FX , PX } as shown below: X: {S,F,P} ! {R, FX , PX }
(5:1:1)
Consider an event D , S [ F. The random variable X maps every point ji in the event D to points xi in the event Ix , called the image of D under X, where Ix , R [ FX . X can be a random variable only if the inverse image X 1 (Ix ) belongs to the field F of subsets of S, and hence it must be an event. The mapping and the inverse mapping are shown in Fig. 5.1.1. With this restriction we should be able to find the induced probability measure PX in terms of the probability measure P as follows: PX {Ix } ¼ P{X 1 (Ix )} ¼ P{D} ¼ P{j : X(j) [ Ix }
(5:1:2)
Since Ix belongs to the field FX , on the real line it may consist of sets of the form {1 , h x}, {x , h , 1}, {x1 , h x2 }, {h ¼ x}. Out of these sets we define Ix by Ix ¼ {1 , h x}
(5:1:3)
All the other sets {x , h , 1}, {x1 , h x2 }, {h ¼ x} can be expressed in terms of the defined Ix as follows: I x ¼ {x , h 1} Ix2 Ix1 ¼ {1 , h x2 } {1 , h x1 } ¼ {x1 , h x2 } lim {IxþDx Ix } ¼ {h ¼ x}
Dx!0þ
With the definition of Ix given by Eq. (5.1.3), we can write Eq. (5.1.2) as follows: PX {Ix } ¼ P{j : X(j) [ Ix } ¼ P{j : X(j) x}
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
64
(5:1:4)
5.1 Definition of Random Variables
65
FIGURE 5.1.1
We now define the quantity P{j : X(j) x} as the cumulative distribution function (cdf) FX (x), and rewrite Eq. (5.1.4) in abbreviated notation as P{j : X(j) [ Ix } ¼ P{X x} ¼ FX (x)
(5:1:5)
Equation (5.1.5) converts from a cumbersome set function P{X x} into a convenient point function FX(x). The value x scans the entire real line, that is, 21 , x , 1, and FX(x) must be defined for all x on the real line. Note that {X x} in Eq. (5.1.5) is meaningless unless it is defined carefully, because X is a function and a function without an argument cannot be less than a number! For example, we cannot have an exponential less than 2, unless we specify the argument of the exponential. The abbreviated notation in Eq. (5.1.5) is to be defined as the probability of the set of all points j [ S such that the number X(j) is less than or equal to the number x. The random variable X should not confused with the value x that the random variable takes. The random variable X can take any value, for example, FX (y) ¼ P{X y}. Having defined the cumulative distribution function (cdf), we now define the probability density function (pdf) as the derivative of the cdf FX(x) with respect to x: P(X x þ Dx) P(X x) Dx!1 Dx
fX ðxÞ ¼ lim
FX (x þ Dx) FX (x) d ¼ FX (x) Dx!1 Dx dx
¼ lim
(5:1:6)
The cdf FX(x) can be given in terms of the pdf fX(x) as follows: ðx FX (x) ¼
fX (j)dj
(5:1:7)
1
Whereas the distribution function FX(x) is a probability measure, the density function fX(x) is not a probability measure unless it is multiplied by the infinitesimal Dx to yield fX (x)Dx ¼ P(x , X x þ Dx)
(5:1:8)
66
Random Variables
B 5.2 DETERMINATION OF DISTRIBUTION AND DENSITY FUNCTIONS Most of the difficulty in determining distribution functions for random variables can be avoided if the following step by step procedure is followed. Step 1. Since the random variable is a mapping from the sample space S to the real line, the mapping diagram is drawn connecting every point j in the sample space to the points on the real line R. Step 2. The regions for x on the entire real line from the mapped values are determined. The regions may correspond to the different subsets of S that map to the real line R. The set of points Ix will be equal to the right closed set ð1, x or, Ix ¼ {1 , h x}. Note that whereas Ix is a right closed subset, the region of definition of x is a subset of the real line R. Step 3. On the sample space S, the probability of the set of all points that map into Ix for every region of x in step 2 is determined. Step 4. The cdf FX (x) ¼ P{X x} is graphed for all the regions of x in step 2 on the entire real line. Step 5. The derivative of FX (x) determines the density function fX (x). Example 5.2.1 A die is tossed, and the random variable X is defined by the amount won (þ) or lost (2) on the face of the die as shown in Table 5.2.1. We have to find the cdf FX(x) and the pdf fX(x). We will follow the steps to find the probability functions. Step 1. The mapping diagram (Fig. 5.2.1) is drawn with positive numbers indicating win and negative numbers indicating loss. Step 2. From the mapping diagram the regions of x are (a) x 29, (b) 29 , x 24, (c) 24 , x 5, (d) 5 , x 8, and (e) x . 8. The corresponding set Ix on the real line for all the 5 regions is Ix ¼ {1 , h x}. Step 3. We will find the all the points in the sample space that map into Ix for every region of x: (a) x 29: Since no points in S map into Ix (Fig. 5.2.1a), we have FX(x) ¼ 0. (b) 29 , x 24: In this region (Fig. 5.2.1b), only one point f3g maps into Ix. Since P{3} ¼ 16 , FX (x) ¼ 16. (c) 24 , x 5: Here the points f3,2,6g from S map into Ix (Fig. 5.2.1c). Hence FX (x) ¼ 16 þ 16 þ 16 ¼ 12. (d) 5 , x 8: In this region (Fig. 5.2.1d) the points f3,2,6,5g in S map in to Ix. Hence FX (x) ¼ 12 þ 16 ¼ 23 TABLE 5.2.1 Pips on the Die 1,4 2,6 3 5
Win or Loss $ þ8 24 29 þ5
5.2 Determination of Distribution and Density Functions
67
FIGURE 5.2.1
(e) x . 8: In this region (Fig. 5.2.1e) all six points, that is, the entire sample space S, map into Ix. Hence FX (x) ¼ 1. In terms of unit step functions we can write the cdf FX(x) as follows: 1 1 1 1 FX (x) ¼ u(x þ 9) þ u(x þ 4) þ u(x 5) þ u(x 8) 6 3 6 3 Step 4. The cdf FX(x) can be graphed as shown in Fig. 5.2.2.
68
Random Variables
FIGURE 5.2.2
Step 5. We can now find the density function fX(x) by differentiating the distribution function FX(x), bearing in mind that the differentiation of the unit step is the Dirac delta function. Performing the indicated differentiation, we obtain 1 1 1 1 fX (x) ¼ d(x þ 9) þ d(x þ 4) þ d(x 5) þ d(x 8) 6 3 6 3 Note that the sum of the strengths of the impulse function adds to 1. The pdf fX(x) is shown in Fig. 5.2.3. From Fig. 5.2.2 we can write the following probabilities: (a) P{X ¼ 4} ¼ FX (4þ) FX (4) ¼ 12 16 ¼ 13 1 1 (b) P{X 4} ¼ FX (4þ) FX (1) ¼ 2 0 ¼ 2
(c) P{5 , X 4} ¼ FX ( 4þ) FX (5) ¼ 12 16 ¼ 13 1 1 1 (d) P{4 , X 4} ¼ FX (4þ) FX (4) ¼ 2 6 ¼ 3
(e) P{5 X , 8} ¼ FX (8) FX (5þ)(Note closure on the left) ¼ 12=3 2=3 ¼ 0 The example above shows that the mapping is from a discrete sample space S to discrete points on the real line R.
FIGURE 5.2.3
5.2 Determination of Distribution and Density Functions
69
TABLE 5.2.2 Time Interval
Win or Loss ($)
6:00– 6:15 6:15– 6:45 6:45– 7:15 7:15– 8:00
þ2 24 24 þ6
Example 5.2.2 A telephone call occurs at random between 6:00 p.m. and 8:00 p.m. Depending on the interval of arrival, the win/loss table is shown in Table 5.2.2. We have to find the cdf FX(x) and the pdf fX(x). Since the phone call is uniformly distributed in 120 min, the probability of a phone call in the region (0,x] is given by x/120. We follow the steps as outlined in Example 5.2.1: Step 1. The mapping diagram is shown in Fig. 5.2.4. Note that the mapping is from the real line R to the real line R. Step 2. From the mapping diagram the regions of x are (a) x 24, (b) 24 , x 2, (c) 2 , x 6, (d) x . 6. Step 3. (a) x 24: No points map FX(x) ¼ 0. 60 ¼ 12. (b) 24 , x 2: The interval that maps is 30 min; hence FX (x) ¼ 120 75 (c) 2 , x 6: The interval that maps is 45 min; hence FX (x) ¼ 120 ¼ 58. (d) x . 6: The entire sample space of 120 min maps; hence FX (x) ¼ 1. In terms of unit step functions, the cdf FX(x) is 1 1 3 FX (x) ¼ u(x þ 4) þ u(x 2) þ u(x 6) 2 8 8 Step 4. The cdf FX(x) is graphed in Fig. 5.2.5 Step 5. The pdf fX(x) is obtained by differentiating FX(x), and the result is 1 1 3 fX (x) ¼ d(x þ 4) þ d(x 2) þ d(x 6) 2 8 8 fX(x) is graphed in Fig. 5.2.6.
FIGURE 5.2.4
70
Random Variables
FIGURE 5.2.5
FIGURE 5.2.6
In the preceding example, the sample space is continuous whereas the range space is discrete. Since the random variable takes discrete values, we have a discrete random variable. Example 5.2.3 A telephone call occurs at random between 6:00 p.m. and 8:00 p.m. We want to find the cdf FX(x) and the pdf fX(x). Here we have a continuous mapping from the real line to the real line and X is a continuous random variable. Step 1. See the mapping diagram in Fig. 5.2.7. Step 2. From the mapping diagram the regions of x are (a) x 0, (b) 0 , x 120, (c) x . 120, and Ix ¼ {1 , h x). Step 3 (a) x 0: No points in S map into Ix. Hence FX(x) ¼ 0. (b) 0 , x 120: The call is uniformly distributed in the interval (0,120], and since (0,x] is an interval mapping into Ix, we have FX(x) ¼ x/120. (c) x . 120: Here the entire sample space maps into Ix, and hence FX (x) ¼ 1. Step 4. The cdf FX (x) is graphed in Fig. 5.2.8. Step 5. The density function fX(x) is obtained by taking the derivative of FX (x), or 1 fX (x) ¼ 120 , 0 x 120. This is sketched in Fig. 5.2.9. We found the cdf and pdf for discrete random variables in Examples 5.2.1 and 5.2.2 and for continuous random variables in Example 5.2.3. We will now give an example of a mixed random variable that has both continuous and discrete components.
5.2 Determination of Distribution and Density Functions
71
FIGURE 5.2.7
FIGURE 5.2.8
FIGURE 5.2.9
Example 5.2.4 A telephone call occurs at random between 6:00 p.m. and 8:00 p.m. However, between 6:20 and to 6:40 p.m. calls accumulate and are released exactly at 6:40 p.m. with a similar situation between 7:00 and 7:20 p.m. and are released at exactly 7:20 p.m. We want to find the cdf FX(x) and the pdf fX(x). Step 1. The mapping diagram shown in Fig. 5.2.10 is similar to Fig. 5.2.7 except that there are deadspots between 6:20 and 6:40 p.m. and 7:00 and 7:20 p.m. Step 2. From the mapping diagram the regions of x are (a) x 0, (b) 0 , x 20, (c) 20 , x , 40, (d) x ¼ 40, (e) 40 , x 60, (f) 60 , x , 80, (g) x ¼ 80, (h) 80 , x 120, (i) x . 120. Step 3. (a) x 0: No points in S map into Ix; hence FX(x) ¼ 0.
72
Random Variables
FIGURE 5.2.10
(b) 0 , x 20: The call is uniformly distributed in the interval (0,120]; hence FX (x) ¼ x/120. 20 ¼ 16. (c) 20 , x , 40: There is a deadspot in this region; hence FX (x) ¼ FX (20) ¼ 120 1 (d) x ¼ 40: There is jump at this point equal to P{X ¼ 40} ¼ 6; hence FX (40) ¼ 16 þ 16 ¼ 13. (e) 40 , x 60: Here also (0,x] is an interval in Ix and we have FX (x) ¼ x=120 and FX (40) , 13. (f) 60 , x , 80: Here again we have a deadspot; and hence FX (x) ¼ FX (60) ¼ 60 1 120 ¼ 2. (g) x ¼ 80: There is again a jump at this point equal to P{X ¼ 80} ¼ 16; hence FX (80) ¼ 12 þ 16 ¼ 23. (h) 80 , x 120: Here (0,x] is an interval in Ix and we have FX (x) ¼ x=120 and FX (80) ¼ 23. (i) x . 120: The entire sample space maps into Ix; hence FX(x) ¼ 1 Step 4. The cdf FX(x) is graphed in Fig. 5.2.11.
FIGURE 5.2.11
5.3 Properties of Distribution and Density Functions
73
FIGURE 5.2.12
Step 5. The pdf fX(x) is the derivative of FX(x) and is graphed in Fig. 5.2.12. From the graph of the cdf FX(x) of Fig. 5.2.11, we can obtain the following probabilities: 1. P{X ¼ 40} ¼ P{X ¼ 80} ¼ 16. 2. P{X , 80} ¼ 12. 3. P{X 80} ¼ 23 : 4. P{40 X 80} ¼ 13 : 5. P{X ¼ 100} ¼ 0: 5 5 6. P{50 , X 100} ¼ 56 12 ¼ 12 :
B 5.3 PROPERTIES OF DISTRIBUTION AND DENSITY FUNCTIONS Cumulative Distribution Function FX(x)
1. From the three examples given in Section 5.2 the cdf of a discrete random variable is a staircase function with jumps. 2. FX(x) 0, that is, nonnegative. 3. FX(21) ¼ 0, FX(1) ¼ 1. 4. If x1 , x2, then FX(x1) FX(x2), that is, FX(x) is a nondecreasing function (monotone increasing). For a continuous random variable, if x1 , x2, then FX(x1) , FX(x2), that is, FX(x) is an increasing function (strict monotone increasing). 5. FX(x) is right-continuous: FX (x) ¼ lim FX (x þ 1) 1!0
for 1 . 0
From this definition, the value that FX (x) takes at the point of a jump, is to the right of the jump, and if there is an event A in the interval {x1 , X x2 }, then the interval is closed on the righthand side so that we can write P{x1 , X x2 } ¼ P{X x2 } P{X x1 } ¼ FX (x2 ) FX (x1 ). This ensures that the point x1 is not counted twice. Figure 5.3.1 will clarify this point. ðx FX (x) ¼ fX (j)dj ¼ P{X x} (5:3:1) 6. 1
74
Random Variables
FIGURE 5.3.1
7. At the point x ¼ a of the jump ð aþ FX (x) ¼ fX (j)dj ¼ FX (aþ) FX (a) ¼ P{X ¼ a}
(5:3:2)
a
Probability Density Function fX(x)
1. fX (x) 0, nonnegative. 2. fX (x) consists of the sum of Dirac delta functions for a discrete random variable. 3. fX (x) is not a probability measure, but fX (x)Dx is a probability measure as shown: fX (x)Dx P{x , X x þ Dx}
4.
(5:3:3)
P{x , X x þ Dx} d ¼ FX (x) Dx dx
ð5:3:4Þ
fX (j)dj FX (b) FX (a) ¼ P{a , X b}
(5:3:5)
fX (x) lim
Dx!0
ðb 5. a
Probability Mass Functions (pmf’s)
We have already come across probability mass functions in connection with discrete random variables. We will now formalize this concept. The probability of a random variable equal to a number is called the probability mass function (pmf). In Eq. (5.3.2), P{X ¼ a} is the pmf. 1. P{X ¼ a} is the probability of the jump at the point x ¼ a in the cdf FX (x). Or, P{X ¼ a} ¼ FX (a) FX (a). 2. The pmf P{X ¼ a} is nonzero for discrete random variables at x ¼ a, where they jump. 3. P{X ¼ a} ¼ 0 for continuous random variables, since at any point of continuity a, FX (a) FX (a) ¼ 0. 4. The density function fX(x) for discrete random variables involves summation of the pmf’s with Dirac delta functions multiplying the pmf at every point in the range. For example, the pdf of the binomial distribution given by the
5.4 Distribution Functions from Density Functions
n
pk qnk is n X n k nk p q d(x k) fX (x) ¼ k k¼0
probability mass function
75
k
(5:3:6)
The pdf of the Poisson distribution given by the pmf el (lk =k!) is fX (x) ¼
1 X
el
k¼0
lk d(x k) k!
(5:3:7)
B 5.4 DISTRIBUTION FUNCTIONS FROM DENSITY FUNCTIONS Given any density function (pdf), the distribution function (cdf) can be found by integrating the pdf over all x on the real line as given in Eq. (5.3.1). Care must be taken in integrating over various regions along the x axis. Example 5.4.1 expressed as
We will start with a simple example. The density function fX (x) is
fX (x) ¼
8 k > > <4,
0,x2
> > :1, 2
2,x3
and is shown in Fig. 5.4.1. We have to first find k and then the distribution function FX (x) for all x. Integrating fX (x) over the range of definition of x, we have ð3 ð3 ð2 k 1 k 1 dx þ dx ¼ þ ¼ 1, [ k ¼ 1 fX (x) ¼ 2 2 0 04 22 1 1 fX (x) ¼ u(x) þ u(x 2) 4 4
FIGURE 5.4.1
76
Random Variables
FIGURE 5.4.2
We note that there are four natural ranges for x given by (1) x 0, (2) 0 , x 2, (3) 2 , x 3, (4) x . 0. We will evaluate FX(x) in all these ranges: 1. x 0: Since fX (x) is zero, FX (x) ¼ 0. 2. 0 , x 2: Using Eq. (5.3.5), we can write ðx FX (x) ¼
1 x dj ¼ 4 04
3. 2 , x 3: Again using Eq. (5.3.5) and taking care to carry over FX (2), we have ð2 FX (x) ¼
1 dj þ 04
ðx
1 1 1 1 dj ¼ þ (x 2) ¼ (x 1) 2 2 2 2 2
4. x . 3: Since fX(x) is zero in this range, FX(x) ¼ FX(3) ¼ 1. FX(x) is graphed in Fig. 5.4.2.
FIGURE 5.4.3
5.4 Distribution Functions from Density Functions
77
Example 5.4.2 This is a more involved example that characterizes a mixed random variable. The density function is given by fX (x) ¼ K{ex ½u(x 1) u(x 5) þ ex ½u(x 1) u(x 5)g 1 þ ½d(x þ 2) þ d(x) þ d(x 2) 8 (see Fig. 5.4.3). We will find K and graph the distribution function FX(x) for all x. We integrate fX(x) over the range (25,5] to evaluate the constant K. Thus ð 5 K
ex dx þ
5
1 1 ex dx þ þ ¼ 1 8 8 1
ð5
2K½e1 e5 þ
3 ¼1 8
[K¼
5 e5 )
16(e1
The ranges for x for integrating fX(x) are given by (1) x 25, (2) 25 , x 22, (3) 22 , x 21, (4) 21 , x 0, (5) 0 , x 1, (6) 1 , x 2, (7) 2 , x 5, (8) x . 5. We shall now evaluate FX(x) in each of these ranges: 1. x 25: Since fX(x) ¼ 0, we have FX(x) ¼ 0 2. 5 , x 2:
3. 2 , x 1:
4. 1 , x 0:
5. 0 , x 1:
FX (x) ¼
FX (x) ¼
FX (x) ¼
FX (x) ¼
5 16(e1 e5 Þ
ðx
ej dj ¼
5
5(ex e5 ) 1 þ 1 5 16(e e Þ 8
5(e1 e5 ) 1 7 þ ¼ 1 5 16(e e Þ 8 16
7 1 9 þ ¼ 16 8 16
FIGURE 5.4.4
5(ex e5 ) 16(e1 e5 Þ
78
Random Variables
6. 1 , x 2:
7. 2 , x 5:
8. x . 5:
FX (x) ¼
FX (x) ¼
FX (x) ¼
9 þ 16
ðx
ej dj ¼
1
9 5(e1 ex ) þ 16 16(e1 e5 )
9 5(e1 ex ) 1 þ þ 16 16(e1 e5 ) 8
9 5(e1 e5 ) 1 þ þ ¼1 16 16(e1 e5 ) 8
The graph of FX (x) is shown in Fig. 5.4.4. In the next chapter we will discuss the various continuous distributions.
CHAPTER
6
Continuous Random Variables and Basic Distributions
B 6.1
INTRODUCTION
In the last chapter we defined a continuous random variable as the mapping from a continuous sample space S to continuous points on the real line R. We also discussed some of the properties of continuous pdf fX(x) and the continuous cdf, FX(x). In this chapter we will discuss three of the basic continuous distributions: uniform, exponential, and Gaussian. Other distributions will be discussed in the next chapter.
B 6.2
UNIFORM DISTRIBUTION
The basic continuous distributions are uniform, exponential, and Gaussian. We have already seen (in Example 5.2.3) the distribution of a telephone call occurring at random in a given interval. The pdf fX(x) of a random variable uniformly distributed in the interval (a, b], (notation X U(a, b]), is given by 8 0 xa > > > < 1 (6:2:1) fX (x) ¼ a,xb > b a > > : 0 x.b and the corresponding distribution function FX(x) is 8 0 xa > > > <xa a,xb FX (x) ¼ ba > > > : 1 x.b
(6:2:2)
Such distributions occur in transmission of sinusoids where the phase angle will be uniformly distributed in (0,2p). The density function is shown in Fig. 6.2.1, and the distribution function is shown in Fig. 6.2.2. Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
79
80
Continuous Random Variables and Basic Distributions
FIGURE 6.2.1
FIGURE 6.2.2
Example 6.2.1 A sinusoidal signal with random phase angle Q, uniformly distributed between (0,2p] is sent along a communications channel. We want to find the probability that the phase angle is between (p/4, 3p/4]. Since it is uniformly distributed, the pdf is 1/2p in (0,2p], and hence the Pfp/4, 3p/ 4g is given by p 3p 1 3p p 1 P ,u ¼ ¼ 4 4 2p 4 4
B 6.3
EXPONENTIAL DISTRIBUTION
Exponential distributions find wide applicability in queuing theory, reliability theory, and communication theory. It is the continuous analog of the geometric distribution discussed in Chapter 4. The pdf of an exponential distribution is given by 0 x0 fX (x) ¼ (6:3:1) lelx x . 0 where the parameter l is the average rate at which events occur. The corresponding cdf FX(x) is given by 0 x0 FX (x) ¼ (6:3:2) 1 elx x . 0 The pdf is shown in Fig. 6.3.1, and the cdf is shown in Fig. 6.3.2 for values of l ¼ 0.5,1,2,3,4. The exponential distribution belongs to a class of distributions possessing memoryless (Markov) properties that we will presently show. A random variable X is called memoryless if it satisfies the conditional probability P{X . x þ x0 j X . x0 } ¼ P{X . x}
(6:3:3)
6.3 Exponential Distribution
81
FIGURE 6.3.1
FIGURE 6.3.2
The condition given by Eq. (6.3.3) is equivalent to P{X . x þ x0 , X . x0 } ¼ P{X . x} P{X . x0 }
(6:3:4)
or P{X . x þ x0 } ¼ P{X . x}P{X . x0 } To demonstrate the memoryless property, let X be a random variable representing the time to failure of a computer with cdf given by Eq. (6.3.2). We are given that the computer has lasted x0 hours. We want to find the conditional probability that the computer will fail at time x beyond x0 given that it has lasted for x0 hours. We proceed as follows: P{X x þ x0 j X . xo } ¼ FX (x þ x0 j X . x0 ) ¼
P{X x þ x0 , X . x0 } P{x0 , X x þ x0 } ¼ P{X . x0 } 1 FX (x0 )
¼
FX (x þ x0 ) FX (x0 ) 1 el(xþx0 ) 1 þ elx0 ¼ 1 FX (x0 ) 1 1 þ elx0
¼
elx0 (1 elx ) ¼ 1 elx , x . x0 elx0
(6:3:5)
82
Continuous Random Variables and Basic Distributions
The conditional distribution FX(x þ x0 j X . x0) is the same as FX(x)! The computer “forgets” that it has been operating for x0 hours and has no memory of previous hours of operation. Hence the exponential distribution is called a memoryless distribution.
Poisson Arrival Process
The number k of events occurring in an interval (0,t] is Poisson-distributed with p(k,l) ¼ e 2lt[(lt)k/k!], and the waiting times for the occurrence from one event to the next are independent and are exponentially distributed as e 2lt. Example 6.3.1 A computer company leases its computers. The lifetime of one of the 1 computers is exponentially distributed with l ¼ 50;000 h. If a person leases the computer for 10,000 h, we have to find the probability that the computer company will have to replace the computer before the lease runs out. By the memoryless property derived in Eq. (6.3.5), it is immaterial how long it has been leased before, and hence we have P{computer fails in t 10,000} ¼ 1 e10000l ¼ 1 e(1=5) ¼ 0:1813 and the probability that the computer will not be replaced before the lease runs out is P{lifetime . 10,000} ¼ 1 0:1813 ¼ 0:8187 However, if the failure time FX(x) is not exponentially distributed, then the probability of replacement has to be computed, taking into account the prior use of x0 hours of the computer. This is given by the conditional probability P{lifetime . x0 þ 10,000 j lifetime . 10000} ¼
1 FX (x0 þ 1000) 1 FX (x0 )
which means that we have to know x0, the number of hours the computer was operational before the lease. Hazard Rate
Given that an item has not failed upto time t, the conditional probability that it will fail in the next small interval of time Dt is determined as follows P{t , X t þ Dt j X . t} ¼ ¼
P{t , X t þ Dt, X . t} P{X . t} P{t , X t þ Dt} fX (t)Dt ¼ ¼ b(t)Dt P{X . t} 1 FX (t)
(6:3:6)
where FX(t) ¼ PfX tg represents the failure distribution and the quantity b(t) ¼ fx(t)/ [1 2 FX(t)] is called the instantaneous hazard rate. It is not a probability function as determined by the following analysis: ðt ðt ðt fX (j) dfX (x) dj ¼ ¼ ln½1 FX (t) b(j)dj ¼ 1 F 1 FX (j) (j) X 0 0 0
6.3 Exponential Distribution
83
Ð Since t0b(j)dj ! 1 as t ! 1, we conclude that b(t) is not a probability function. However, solving for FX(t), we obtain ðt ln½1 FX (t) ¼ b(j)dj (6:3:7) 0
or ðt 1 FX (t) ¼ exp b(j)dj ¼ 0 as t ! 1 0
From Eq. (6.3.7) we can obtain the cdf FX(t), given by ðt FX (t) ¼ 1 exp b(j)dj
(6:3:8)
0
Thus, b(t) uniquely determines the failure probability distribution function FX(t). While b(t) is not a probability density function, it is to be noted that the function b(x), given by 8 < fX (x) , x . t (6:3:9) b(x) ¼ 1 FX (t) : 0, xt is the conditional failure density conditioned on x . t and b(t) is the value of b(x) at x ¼ t. Example 6.3.2 Physicians are of the opinion that the death rate of a person E exercising regularly is half that of a person S who has sedentary habits. We want to compare the probabilities of survival of E and S given that the death rate of an exercising person E is bE(t) and the rate for a sedentary person S is bS(t) with bE(t) ¼ 12bS(t). Since the rates are conditional failure rates, we will assume that both persons have lived upto the age T1, and we want to compare the probabilities of their reaching the age T2 with T2 . T1. Thus we need to find P{Xi . T2 j Xi . T1 } ¼
P{Xi . T2 } 1 FXi (T2 ) ¼ P{Xi . T1 } 1 FXi (T1 )
where Xi ¼ E or S. Substituting from Eq. (6.3.8), we have ÐT ð T2 1 FXi (T2 ) exp 0 2 bXi (j)dj ¼ Ð ¼ exp bXi (j)dj , 1 FXi (T1 ) exp T1 bX (j)dj T1 0
i ¼ E or S
i
Simplifying, we obtain ð 1 T2 P{E . T2 j E . T1 } ¼ exp bS (j)dj 2 T1 ð T2 P{S . T2 j S . T1 } ¼ exp bS (j)dj T1
Thus, the probability of an exercising person who has lived upto T1 years living to T2 years is the square root of the probability of the sedentary person. If the death rate of a sedentary person, bS(t) ¼ 0.001t, T1 ¼ 60 years and T2 ¼ 70 years, then the following probabilities
84
Continuous Random Variables and Basic Distributions
can be evaluated: ð 70 P{S . 70 j S . 60} ¼ exp 0:001j dj ¼ 0:522 60
ð pffiffiffiffiffiffiffiffiffiffiffi 1 70 P{E . 70 j E . 60} ¼ exp 0:001j dj ¼ 0:522 ¼ 0:7225 2 60 The probability of a 60 year old sedentary person reaching 70 years is 0.522, whereas the same person exercising regularly will reach 70 years with a probability of 0.7225.
B 6.4
NORMAL OR GAUSSIAN DISTRIBUTION
The most important distribution in probability theory is the Gaussian distribution. Under some general conditions, a random variable X consisting of a number of components, each with a general distribution tends to a normal distribution as the number becomes very large. The probability density fX(x) of a Gaussian distribution is given by 2 1 fX (x) ¼ pffiffiffiffiffiffiffiffiffi eð1=2Þ½(xm)=s 2ps
(6:4:1)
where m is called the mean and s2 is called the variance. These quantities will be defined later. The cumulative distribution function FX(x) is given by ðx 1 pffiffiffiffiffiffiffiffiffi e(1=2)½(jm)=s dj FX (x) ¼ (6:4:2) 2ps 1 The Gaussian density function fX(x) given by Eq. (6.4.1) is shown in Fig. 6.4.1 for m ¼ 3 and s2 ¼ 4. It also shows that the area under the Gaussian pdf between (m 2 s, m þ s] is 68.3%, between (m 2 2s, m þ 2s] is 95.5%, and between (m 2 3s, m þ 3s] is 99.7%. Thus, almost the entire pdf (99.7%) is between the m + 3s points of the Gaussian curve. The Gaussian distribution function, FX(x) is shown in Fig. 6.4.2 along with (m 2 s, m þ s], (m 2 2s, m þ 2s], and (m – 3s, m þ 3s] points.
FIGURE 6.4.1
6.4 Normal or Gaussian Distribution
85
FIGURE 6.4.2
A standard Gaussian is defined as the random variable with m ¼ 0 and s2 ¼ 1, given by 1 2 fX (x) ¼ pffiffiffiffiffiffi e(1=2)x 2p
(6:4:3)
and the corresponding cumulative distribution function is denoted by FX(x), given by ðx FX (x) ¼
2 1 pffiffiffiffiffiffi e(1=2)j dj 2p 1
(6:4:4)
There is no closed-form solution for Eq. (6.4.4). Hence, FX(x) is tabulated for values of x between 25 and 5 in Appendix B. The standard Gaussian pdf fX(x) is shown in Fig. 6.4.3 with the 68.3% points between (21,1], 95.5% points between (22,2], and the 99.7% points between (23,3].
FIGURE 6.4.3
86
Continuous Random Variables and Basic Distributions
Given a random variable X with any m and s2, the random variable Z¼
X mX sX
is a standard Gaussian. Hence, the distribution function FX(x) can be given in terms of the standard Gaussian FZ (x) as follows: FX (x0 ) ¼ P{X x0 ¼} ¼ P
X mX x0 mX x0 mX ¼ FX sX sX sX
(6:4:5)
We shall now prove that the fX(x) is a probability density function by showing that if ð1 I¼
1 2 pffiffiffiffiffiffi e(1=2)x dx 2p 1
then I ¼ 1. Direct evaluation of the integral is difficult. Hence we square I and write ð1 1 (1=2)x2 1 2 pffiffiffiffiffiffi e(1=2)y dy e dx 2p 1 2p 1 ð ð 1 1 1 ð1=2Þ(x2 þy2 ) ¼ e dx dy 2p 1 1
I2 ¼
ð1
(6:4:6)
Substituting x ¼ r cos(u) and y ¼ r sin(u) in Eq. (6.4.5) we obtain, I2 ¼
1 2p
ð 2p ð 1 0
2
re(1=2)r dr du ¼ 1
(6:4:7)
0
and the result is proven. Example 6.4.1 The number of malfunctioning computers is Gaussian-distributed with m ¼ 4 and s ¼ 3. We want to find the probability that the number of bad computers is between 3 and 5. Denoting the malfunctioning computers by the random variable X, we have to find Pf3 , X 5g. Using Eq. (6.4.5) and referring to the Gaussian tables, we have
34 X4 54 1 1 P{3 , X 5} ¼ P , ¼P ,Z 3 3 3 3 3 1 1 ¼ FX FX ¼ 0:2611 3 3 Example 6.4.2 We shall study what grading on the curve in examinations means. If the average of the class in a particular examination is m, then one method of grading is to assign A to those receiving over m þ 2s, AB to those receiving between m þ s and m þ 2s, B to those receiving between m and m þ s, C to those receiving between m 2 s and m, D to those receiving between m 2 s and m 2 2s, and F to those receiving less than m 2 2s. We want to calculate the percentage of students receiving the grades A, AB, B, C, D, and F.
6.4 Normal or Gaussian Distribution
We can calculate these probabilities from Eq. (6.4.5) as follows: Xm . 2 ¼ 1 F(2) ¼ 0:0228 ¼ 2:28% A: P{X . m þ s} ¼ P s Xm 2 ¼ F(2) F(1) AB: P{m þ s , X m þ 2s} ¼ P 1 , s ¼ 0:1359 ¼ 13:59% B:
Xm 1 ¼ F(1) F(0) P{m , X m þ s} ¼ P 0 , s ¼ 0:3413 ¼ 34:13%
C:
Xm 0 ¼ F(0) F(1) P{m s , X m} ¼ P 1 , s ¼ 0:3413 ¼ 34:13%
D:
F:
Xm 1 ¼ F(1) F(2) P{m 2s , X m s} ¼ P 2 , s ¼ 0:1359 ¼ 13:59% Xm 2 ¼ F(2) ¼ 0:0228 ¼ 2:2:8% P{X m 2s} ¼ P s
These grades are shown in Fig. 6.4.4. Properties of Standard Gaussian
Density and Distribution 1. The pdf fX(x) is an even function.
pffiffiffiffiffiffi 2. fX(x) attains its maximum value of 1= 2p at x ¼ 0. Ð 3. The cdf FX(x) ¼ x21fX(j)dj.
FIGURE 6.4.4
87
88
Continuous Random Variables and Basic Distributions
4. FX (0) ¼ 1 : 2 5. The error function erf(x) is defined as 2 erf(x) ¼ pffiffiffiffi p
ðx
2
ej dj
(6:4:8)
0
6. The complement of the error function erfc(x) is defined as ð 2 1 j2 e dj ¼ 1 erf(x) erfc(x) ¼ pffiffiffiffi p x
(6:4:9)
7. The tails of the distribution Q(x) is used in describing the probability of error in transmission of communication signals. It is defined as ð 1 1 (j2 =2) Q(x) ¼ pffiffiffiffiffiffi e dj (6:4:10) 2p x We will discuss this function in greater detail in the next section. 8. Q(x) in terms of error function complement erfc(x) is 1 x Q(x) ¼ erfc pffiffiffi 2 2
(6:4:11)
9. The general distribution function FX(x) in terms of erf(x) and erfc(x) is pffiffiffiffi ðx ð 1 1 2 ðxmÞ= 2s j2 (1=2)½(jm)=s2 FX (x) ¼ pffiffiffiffiffiffiffiffiffi e dj ¼ pffiffiffiffi e dj 2 p 1 2ps 1 1 1 xm 1 xm ¼ þ erf pffiffiffiffiffiffi ¼ 1 erfc pffiffiffiffiffiffi (6:4:12) 2 2 2 2s 2s 1 10. (6:4:13) FX (m) ¼ 2
Gaussian Tails Q(x)
As mentioned in property 7 [Eq. (6.4.10)] Q(x) is used to express the bit error probability in transmission of communication signals. Since Q(x) does not have a closed-form solution, bounds can be established. The distribution Q(x) can expanded into an asymptotic series [1] given by 2 ex 1 1 1:3 n 1:3 (2n 1) p ffiffiffiffiffiffi 1 2 þ 4 þ þ (1) Q(x) ¼ (6:4:14) x x x2n 2p x and for large x, Q(x) can be approximated by 2
ex 1 Q(x) pffiffiffiffiffiffi 2p x
(6:4:15)
For small x, Q(x) diverges, and hence this is not a very good bound. A better bound is 2
ex 1 Q(x) QA(x) ¼ pffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffi 2p 1 þ x2
(6:4:16)
6.4 Normal or Gaussian Distribution
89
Even this bound, although better than Eq. (6.4.15), is not very good for small values of x. However, a tighter bound has been established by Bo¨rjesson and Sundberg [8] by first integrating Q(x) by parts as shown below: ð pffiffiffiffiffiffi ð 1 1 (j2 =2) 1 1 (j2 =2) je Q(x) ¼ pffiffiffiffiffiffi e dj ¼ 1 2p dj 2p x x j ð1 1 1 1 (j2 =2) 2 ¼ pffiffiffiffiffiffi e(x =2) e dj 2 2p x x j and the expression for Q(x) can be manipulated to obtain Q(x) ¼
1
1 2 pffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffi e(x =2) 2 (1 a)x þ a x þ b 2p
(6:4:17)
Substituting the values a ¼ 1 and b ¼ 1 in Eq. (6.4.17), we obtain Eq. (6.4.16). Better values for a and b can be obtained by minimizing the absolute relative error function 1(x), given by
Q0 (x) Q(x)
(6:4:18) 1(x) ¼
Q0 (x) and constraining 1(0) ¼ 0. In Eq. (6.4.18) Q0(x) is the true value given by Eq. (6.4.10). As a consequence, the upper bound QU(x) is obtained for values of a ¼ 0.33919 and b ¼ 5.533425, and the lower bound QL(x) is obtained for values of a ¼ 1/p and b ¼ 2p. Thus the upper bound is QU(x) ¼
1 1 2 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffi e(x =2) 2 0:66081x þ 0:33919 x þ 5:33425 2p
(6:4:19)
and the lower bound is, QL(x) ¼
1 1 2 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffi eðx =2Þ (1 1=p) þ 1=p x2 þ 2p 2p
(6:4:20)
The functions Q(x), QU(x), QL(x), and QA(x) are shown in Fig. 6.4.5 and tabulated in Table 6.4.1 showing the upper and lower bounds are for all x . 0. The bound QA(x) is not good for low values of x, whereas QU(x) and QL(x) practically lie on Q(x).
FIGURE 6.4.5
90
Continuous Random Variables and Basic Distributions
TABLE 6.4.1 x 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Lower Bound QL(x)
True Value Q(x)
Upper Bound QU(x)
Approx. Value QA(x)
0.5 0.30496416 0.15704991 0.06633866 0.0226461 0.00619131 0.00134729 0.00023233 0.00003164 0.0000034 0.00000029
0.5 0.30853754 0.15865525 0.0668072 0.02275013 0.00620967 0.0013499 0.00023263 0.00003167 0.0000034 0.00000029
0.5 0.3071816 0.15837866 0.06684731 0.02279147 0.00622374 0.00135302 0.00023314 0.00003173 0.0000034 0.00000029
0.39894228 0.3148968 0.17109914 0.07184344 0.02414549 0.00650985 0.00140147 0.00023974 0.00003246 0.00000347 0.00000029
The absolute percentage errors for the upper QU(x) and lower bounds QL(x) are shown in Fig. 6.4.6. Table 6.4.2 shows in addition to the upper- and lower-bound errors the absolute percentage error for the approximate value QA(x). Since these errors are large, they are not shown in Fig. 6.4.6. The errors for the approximate value are high (20.2%). The upper-bound errors are better for smaller values of x, whereas the lower-bound is better for higher values of x. Even then, these bounds clearly illustrate that the approximation for all x is excellent with the maximum error of about 1.16% occurring at low values of x. This is much better than the usual approximation QA(x), where the maximum error is 20.2% for low values. For high values of x, QU(x), QL(x), and QA(x) provide reasonably good approximation.
Properties of Q Function
1. The limit as x ! 1 is zero: lim Q(x) ¼ 0
x!1
FIGURE 6.4.6
(6:4:21)
6.4 Normal or Gaussian Distribution
TABLE 6.4.2 x 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
91
Error percentages Lower Bound
Upper Bound
Approximate Value
0 1.15816594 1.01184414 0.70133395 0.45727327 0.29559943 0.19344425 0.12922798 0.08836646 0.06185728 0.04427826
0.00000176 0.43947331 0.17433835 0.06003046 0.1816945 0.226622 0.23152986 0.21848937 0.19884525 0.1779578 0.15815582
20.21154392 2.06109849 7.8433496 7.53846047 6.13342513 4.83410996 3.82069533 3.05864395 2.48602699 2.05102577 1.71571131
2. The limit as x ! 21 is 1: lim Q(x) ¼ 1
(6:4:22)
x!1
3. The value of Q(x) at x ¼ 0 is half: Q(0) ¼
1 2
(6:4:23)
4. Q(x) is even: Q(x) ¼ Q(x) 5. Q(x) can be expressed in terms of error function as 1 x 1 x Q(x) ¼ 1 erf pffiffiffi ¼ erfc pffiffiffi , 2 2 2 2
(6:4:24)
x0
(6:4:25)
6. If F(x) is a standard Gaussian, then Q(x) ¼ 1 F(x)
(6:4:26)
7. For any Gaussian distributed random variable X with mean m and variance s2 a m P{X . a} ¼ 1 FX (a) ¼ Q (6:4:27) s
Gaussian Approximation to Binomial Distribution
In Example 4.7.3 we saw that the binomial distribution can be approximated by a Poisson distribution under the condition that n is large compared to k and p is small enough that the product np ¼ l is of moderate magnitude. On the other hand, if npq is quite large, the Poisson approximation does not fare very well. In this case we use the Gaussian approximation, which is called the DeMoivre –Laplace limit theorem, which states that, if Sn is the total number of successes in n Bernoulli trials, each with probability of success p and failure q with ( p þ q ¼ 1) with pmf b(k;n,p), then, as n ! 1, the pffiffiffiffiffiffiffiffi standardized random variable Z ¼ (Sn 2 np)/ npq converges to a standard normal distribution function where m ¼ np is the mean and s2 ¼ npq is the variance. Or, if
92
Continuous Random Variables and Basic Distributions
n1 , n2, then Sn np lim Z ¼ P n1 , pffiffiffiffiffiffiffiffi n2 ! F(n2 ) F(n1 ) n!1 npq
(6:4:28)
and the probability mass function b(k;n,p) converges to the density function fZ (z). Or lim
2 n! 1 pk qnk ! pffiffiffiffiffiffiffiffiffiffiffiffiffi e(1=2)½(xnp)=npq k)! 2pnpq
n!1 k!(n
(6:4:29)
The Demoivre –Laplace limit theorem is a special case of the central limit theorem presented in a later chapter.
Example 6.4.3 (Gaussian and Poisson Approximation) We will find the Gaussian approximation for the binomial distribution given in Example 4.7.3, where the Poisson approximation has already been found. In the binomial distribution b(k;n,p) of that example, n ¼ 1000, p ¼ 0.005 with np ¼ 5, and npq ¼ 4.975. The approximate Gaussian pdf will be given by 2 1 fX (k) ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi eð1=2Þ½(k5) =4:975 b(k; 1000,0:005) 2p:4:975
For the Poisson approximation np ¼ l ¼ 5, and this is given in Example 4.7.3 as k 5 pX (k;l) ¼ pX (k;5) ¼ pX (k) ¼ e5 k! Since l ¼ 5 is a single digit and npq ¼ 4.975 is not very large, we would reasonably infer that the Poisson approximation will be better than the Gaussian. The binomial distribution with the Poisson and Gaussian approximations are shown in Fig. 6.4.7 for k ¼ 0, . . . , 15. The true value of b(5; 1000, 0.005) ¼ 0.17591, the Gaussian approximation fX(5) ¼ 0.17886 with an absolute error of 1.678% and the Poisson approximation pX(5; 5) ¼ 0.17547 with an error of 0.25%, confirming our inference that the Poisson approximation is better. Figure 6.4.7 clearly shows that the Poisson approximation follows the binomial more closely than the Gaussian.
FIGURE 6.4.7
6.4 Normal or Gaussian Distribution
93
FIGURE 6.4.8
Example 6.4.4 We shall now modify Example 6.4.3 by keeping n ¼ 1000 and changing p ¼ 0.5, yielding np ¼ l ¼ 500, and npq ¼ 250. Here both l and npq are quite large, so we can infer that the Gaussian will be a better approximation than the Poisson. The approximate Gaussian pdf is given by 2 1 fX (k) ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi e(1=2)½(k500) =250 b(k; 1000,0:5) 2p:250
and the corresponding Poisson approximation is PX (k;l) ¼ PX (k;500) ¼ PX (k) ¼ e500
500k k!
The binomial distribution with the Poisson and Gaussian approximations for this case are shown in Fig. 6.4.8. TABLE 6.4.3
Comparison of binomial with Gaussian and Poisson approximations
k
Binomial
Gaussian
Error 1BG
Error 1BP
2 3 4 5 6 7 8
0.083929 0.140303 0.175731 0.175908 0.14659 0.104602 0.035245
n ¼ 1000, p ¼ 0.005, np ¼ 5, npq ¼ 4.975 0.072391 0.084224 13.74699 0.119653 0.140374 14.718176 0.161758 0.175467 7.951344 0.17886 0.175467 1.678441 0.161758 0.146223 10.347485 0.119653 0.104445 14.38896 0.072391 0.065278 10.953113
0.352343 0.50683 0.150021 0.250272 0.250272 0.14992 0.051188
497 498 499 500 501 502 503
0.024775 0.025024 0.25175 0.25225 0.025175 0.025024 0.024775
n ¼ 1000, p ¼ 0.5, np ¼ 500, npq ¼ 250 0.024781 0.017731 0.023389 0.02503 0.017803 0.02438 0.025181 0.017838 0.024978 0.025231 0.017838 0.025177 0.025181 0.017803 0.024978 0.02503 0.017732 0.02438 0.024781 0.017626 0.023389
28.431591 28.85844 29.141872 29.283306 29.283306 29.141589 28.857017
Poisson
94
Continuous Random Variables and Basic Distributions
The true value for k ¼ 500 for binomial b(500,1000,0.5) ¼ 0.025225, the Gaussian approximation fX(k) ¼ 0.025231 with an absolute error of 0.025% and the Poisson approximation pX(k) ¼ 0.017838 with an error of 29.28%. In this case, the Gaussian approximation, as expected, fares very much better than the Poisson one. The results graphed in Fig. 6.4.8 for k ¼ 400, . . . , 600 clearly show that the Gaussian approximation follows the binomial much more closely than does the Poisson, unlike the scenario described in Example 6.4.3. The percentage absolute errors for the Gaussian and Poisson approximations to the binomial are defined by
b(k;n,p) fX (k;np)
100 Gaussian: 1BG ¼
b(k;n,p)
b(k;n,p) pX (k;l)
100 Poisson: 1BP ¼
b(k;n,p) These errors are tabulated for p ¼ 0.005 and 0.5 in Table 6.4.3 in the neighborhoods of np ¼ l ¼ 5 and 500, respectively. The absolute error percentages also show that for large n and for values of moderate np and npq, the Poisson approximation is better and for large values of np and npq, the Gaussian approximation is better.
CHAPTER
7
Other Continuous Distributions
B 7.1
INTRODUCTION
We will now discuss some of the other important continuous distributions that occur in communications, signal processing, queuing, statistical inference, and other probabilistic situations. This list is by no means exhaustive. For a more complete list of distributions, refer to Web references W1 and W2 (last two entries in References list at end of the book). Most of these distributions are derivable from the three basic distributions discussed in Chapter 6.
B 7.2
TRIANGULAR DISTRIBUTION
If two independent random variables X and Y are uniformly distributed in the interval (a, b], then Z ¼ X þ Y has a triangular density defined by 8 za > a,zb > > 2 > < (b a) (7:2:1) fZ (z) ¼ (z 2b þ a) b , z 2b a > 2 > > (b a) > : 0 otherwise The density fZ (z) is shown in Fig. 7.2.1 with the corresponding uniform densities of X and Y for a ¼ 1 and b ¼ 5. The distribution function FZ (z) is given by 8 0 za > > > > 1 z a 2 > > a,zb <2 b a (7:2:2) FZ (x) ¼ 1 z þ a 2b 2 > > >1 b , z 2b a > > 2 ba > : 1 z . 2b a The distribution function FZ (z) is shown for a ¼ 1 and b ¼ 5 in Fig. 7.2.2.
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
95
96
Other Continuous Distributions
FIGURE 7.2.1
FIGURE 7.2.2
B 7.3
LAPLACE DISTRIBUTION
The Laplace density, also called a double-exponential density, is used to model navigation and pilot errors, speech, and image processing. As the term suggests, double-exponential density is defined by fX (x) ¼
l ljxj e 2
l.0
(7:3:1)
The cdf FX(x) is given by 8 1 > < elx FX (x) ¼ 2 > : 1 1 elx 2
x0 (7:3:2) x.0
It is an even density function, as shown for l ¼ 0.5 in Fig. 7.3.1. The cdf FX(x) is shown in Fig. 7.3.2. An alternate form of Laplace density in terms of the variance s2 is defined by pffiffi 1 fX (x) ¼ pffiffiffi e 2(jxj=s) 2s
(7:3:3)
7.4 Erlang Distribution
97
FIGURE 7.3.1
FIGURE 7.3.2
with the corresponding cdf FX(x) given by 8 pffiffi 1 > < e 2(x=s) FX (x) 2 pffiffi > : 1 1 e 2(x=s) 2
x0 (7:3:4) x.0
where s2 is the variance of the Laplace distribution.
B 7.4
ERLANG DISTRIBUTION
We already saw that the waiting time between successive events of a Poisson arrival process is exponentially distributed with parameter l. The distribution of the waiting time for n successive independent events of a Poisson arrival process is given by the Erlang distribution with n degrees of freedom and is defined for positive integral values of n by fX (x) ¼
l(lx)n1 elx , (n 1)!
where n is an integer
(7:4:1)
98
Other Continuous Distributions
The Erlang distribution is obtained by the n-fold convolution of n independent exponential distributions. The cumulative distribution corresponding to Eq. (7.4.1) can be derived as follows. Let Xn denote the time at which the nth event occurs. Let N(x) be the number of events in the interval (0,x]. Hence the event fXn x) can occur only if the number of events occurring in (0,x] is at least n. Since N(x) is Poisson-distributed, the cdf FXn(x) can be given by P{Xn x} ¼ FXn (x) ¼ P{N(x) . n} ¼
1 X
el x
k¼n
n1 X (l x)k (l x)k ¼1 el x k! k! k¼0
(7:4:2)
We can obtain the pdf fXn(x) by differentiating Eq. (7.4.2) with respect to x. Performing the indicated differentiation, we can write ( ) n1 k X d d l x (l x) {FXn (x)} ¼ 1 e dx dx k! k¼0 ¼ el x
n1 X k(l x)k1 l k¼0
( ¼ lel x
k!
n1 X (l x)k
n1 X (l x)k k¼0
n1 X (l x)k1
k!
)
(k 1)! ( ) n1 n2 X (l x)k X (l x)k l x ¼ le k! k! k¼0 k¼0 k¼0
k!
þ lel x
k¼1
or d lel x (l x)n1 {FXn (x)} ¼ fXn (x) ¼ dx (n 1)!
(7:4:3)
and Eq. (7.4.3) is the Erlang distribution. This distribution is shown in Fig. 7.4.1 for n ¼ 10 and l ¼ 2,2.5,3,3.5,4,5. Another family of Erlang distributions is shown for l ¼ 4 and n ¼ 2,3,4,5,6 in Fig. 7.4.2.
FIGURE 7.4.1
7.5 Gamma Distribution
99
FIGURE 7.4.2
In both these cases the Erlang distributions reduce to an exponential distribution for n ¼ 1. In cases where the data may not be modeled as a simple exponential, the Erlang family provides greater flexibility.
B 7.5
GAMMA DISTRIBUTION
A gamma distribution is a generalization of the Erlang distribution where n ¼ a, which may not be an integer. It finds wide use in statistics. Gamma density is a two-parameter family with parameters a . 0, termed the shape parameter, and l . 0, called the scale parameter, and is defined as fX (x) ¼
l(lx)a1 elx , G(a)
a, l, x . 0
(7:5:1)
where G(a) is the gamma function defined by the integral ð1 G(a) ¼ xa1 ex dx, a . 0
(7:5:2)
0
Integrating Eq. (7.5.2) by parts with u ¼ x a21 and dv ¼ e 2x dx, we have 1 ð1 ð1 a1 x a1 x G(a) ¼ x e dx, a . 0 ¼ x e þ (a 1) xa2 ex dx 0
0
¼ (a 1)G(a 1)
0
(7:5:3)
If a is an integer n, then G(n) ¼ (n –1)G(n –1) and continued expansion yields G(n) ¼ (n– 1)! Hence the gamma function can be regarded as the generalization of the factorial function for all positive real numbers. The most useful fractional argument for the gamma function is 12 with pffiffiffiffi 1 G ¼ p (7:5:4) 2
100
Other Continuous Distributions
The incomplete gamma function Gx(a) is defined by ðx Gx (a) ¼ ja1 ej dj
(7:5:5)
0
We will use Eq. (7.5.5) later for the x2 distribution function FX(x) in Section 7.7. We shall now show that the density function given by Eq. (7.5.1) indeed integrates to 1. If we define the integral Ix by ðx l(lh)a1 elh dh (7:5:6) Ix ¼ G(a) 0 Then, substituting lh ¼ j in Eq. (7.5.6), we can rewrite ð 1 lx a1 j Ix ¼ j e dj G(a) 0
(7:5:7)
Applying Eq. (7.5.5) to Eq. (7.5.7), we have Ix ¼
Glx (a) G(a)
or
lim Ix ¼
x!1
G(a) ¼1 G(a)
thus showing the validity of Eq. (7.5.1) as a proper density function. If a is an integer, then we have the Erlang distribution. The gamma density family is shown in Fig. 7.5.1 with the parameter l ¼ 2 and a ¼ 0.5,1,1.5,2.5,3.5,4.5,5.5. The cumulative distribution function FX(x) is given by ð 1 x Gx (a) (7:5:8) FX (x) ¼ l(lj)a1 elj dj ¼ G(a) 0 G(a) This integral in Eq. (7.5.8) has no closed-form solution unless a is an integer, in which case it is the Erlang distribution given by Eq. (7.4.2). However, the integral given by Eq. (7.5.8) has been extensively tabulated. Reproductive Property of Gamma Distribution
If X1, X2, . . . , Xn are n independent gamma-distributed random variables with shape parameters a1, a2, . . . , an and having the same scale parameter l, then random variable
FIGURE 7.5.1
7.6
Weibull Distribution
101
Z ¼ X1 þ X2 þ þ Xn is also gamma-distributed with shape parameter b ¼ a1 þ a2 þ þ an. As a consequence of this property, since an exponential random variable is gamma-distributed with parameters (1,l), the sum of n exponential random variables is gamma-distributed with parameters (n,l), which is an Erlang distribution given by Eq. (7.4.1).
B 7.6
WEIBULL DISTRIBUTION
We will define the Weibull distribution as a three-parameter family of distribution functions, in which the parameter l . 0 is the scale parameter, a . 0 is the shape parameter, and m is the location parameter. It is defined by a a a l (x m)a1 e½l(xm) x . m (7:6:1) fX (x) ¼ 0 xm We will now find the cdf FX (x) of the Weibull distribution. We can write ðx a FX (x) ¼ a la (j m)a1 e½l(jm) dj
(7:6:2)
m
Substituting h ¼ ½l(j m)a and hence dh ¼ ½l(j m)a1 al dh, we can rewrite Eq. (7.6.2) as follows: ½l(xm)a ð ½l(xm) a eh dh ¼ eh , x.m (7:6:3) FX (x) ¼ 0
0
Substitution of the limits in Eq. (7.6.3) yields a 1 e½l(xm) x . m FX (x) ¼ 0 xm
and
FX (1) ¼ 1
(7:6:4)
Two families of Weibull distributions are shown for m ¼ 0 and a ¼ 2, l ¼ 0.5,1,2,3,4 (Fig. 7.6.1) and for l ¼ 2 and a ¼ 0.5,1,2,3,4,5 (Fig. 7.6.2). When m ¼ 0 and l ¼ 1 in Eqs. (7.6.1) and (7.6.4), then the resulting distribution is called the standard Weibull.
FIGURE 7.6.1
102
Other Continuous Distributions
FIGURE 7.6.2
The Weibull distribution was originally used for modeling fatigue data and is at present used extensively in engineering problems for modeling the distribution of the lifetime of an object that consists of several parts and that fails if any component part fails.
B 7.7
CHI-SQUARE DISTRIBUTION
The Laplace, Erlang, gamma, and Weibull distributions are all based on exponential distributions. The chi-square distribution straddles both the exponential and Gaussian distributions. The general gamma distribution has parameters (a, l). In the gamma distribution, if we substitute a ¼ n/2 and l ¼ 12, we obtain the standard chi-square distribution, defined by 8 (n=2)1 (x=2) x e > < , x.0 n=2 2 G(n=2) (7:7:1) fX (x) ¼ > : 0, x0 where n is the degrees of freedom for the x2 distribution. We can introduce another parameter s2 and rewrite Eq. (7.7.1) as 8 (n=2)1 (x=2s2 ) > e > <x , x.0 n=2 2 (7:7:2) fX (x) ¼ (2s ) G(n=2) > > : 0 , x0 If we substitute x ¼ x2 in Eq. (7.7.2), we obtain the familiar form of the chi-square distribution, given by 2
fX (x2 ) ¼
2
x2½(n=2)1 e(x =2s ) (2s2 )n=2 G(n=2)
(7:7:3)
where G(x) is the gamma function defined in Eq. (7.5.2). If n is even in Eq. (7.7.2), then G(n=2) ¼ (n=2 1)!, and if n is odd, then G(n=2) in Eqs. (7.7.1) pffiffiffiffi –(7.7.3) is obtained by iterating (n=2 1)G(n=2 1) with the final iterant Gð12Þ ¼ p. Equation (7.7.3) can also be obtained from the following. If X1, X2, P . . . , Xn are n i.i.d. Gaussian random variables with zero mean and variance s2, then X ¼ nk¼1 X k2 is x2 -distributed with n degrees of freedom.
7.7 Chi-Square Distribution
103
The cumulative chi-square distribution function FX(x) is given by integrating Eq. (7.7.2): 8 ð x n=21 (j=2s2 ) e < j dj x . 0 n=2 2 (7:7:4) FX (x) ¼ : 0 (2s ) G(n=2) 0 x0 This equation can be given in terms of the incomplete gamma function Gx(a) given by Eq. (7.5.5). Substituting h ¼ j/2s2 and dj ¼ 2s2 dh in Eq. (7.7.4), we obtain ðx FX (x) ¼
2
jn=21 e(j=2s ) 2s2 dj 2 n=21 G(n=2) 0 (2s )
ð x=2s2 ¼ 0
Gx=2s2 (n=2) hn=21 e(h) dh ¼ (7:7:5) G(n=2) G(n=2)
When x ! 1 then Gx=2s2 (n=2) ! G(n=2) and hence FX(x) ¼ 1. The cdf FX(x) in Eq. (7.7.5) has been extensively tabulated. The chi-square distribution is shown in Fig. (7.7.1) for the number of degrees of freedom n ¼ 2,3,4,5,6,10. The chi-square distribution is used directly or indirectly in many tests of hypotheses. The most common use of the chi-square distribution is to test the goodness of a hypothesized fit. The goodness-of-fit test is performed to determine whether an observed value of a statistic differs enough from a hypothesized value of a distribution to draw the inference whether the hypothesized quantity is the true distribution. Its utility is shown in Chapter 15. Example 7.7.1 The density function of fZ (z) of Z ¼ X 2 þ Y 2, where X and Y are independent random variables distributed as N(0, s2), can be found by substituting n ¼ 2 in Eq. (7.7.2), yielding 8 2 < e(z=2s ) z.0 FZ (z) ¼ 2 : 2s 0 z0
FIGURE 7.7.1
104
Other Continuous Distributions
which is a x2 density with 2 degrees of freedom. The corresponding cdf, FZ (z), is given by ( 2 1 e(z=2s ) z . 0 FZ (z) ¼ 0 z0
B 7.8
CHI AND OTHER ALLIED DISTRIBUTIONS
The Rayleigh and Maxwell distributions are special cases of the chi distribution. Just as the sum of the squares of n zero mean Gaussian random variables with variance s2 is chisquare-distributed with n degrees of freedom, we can consider that the sum of the square roots of the sum of the squares of n zero mean Gaussian random variables with variance s2 is chi-distributed with n degrees of freedom. Accordingly, let Zn be the random variable given by qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi (7:8:1) Zn ¼ X1 þ X2 þ þ Xn2 where fXi . i ¼ 1, . . . , ng are zero mean Gaussian random variables with variance s2. Then the x-density fZ (z) is given by 9 8 1 zn1 (1=2)(z=s)2 = < e z.0 fZ (z) ¼ 2n=21 G(n=2) sn (7:8:2) ; : 0 z0 where G(n/2) is the gamma function defined in Eq. (7.5.2). A family of chi distributions is shown in Fig. 7.8.1 for s ¼ 1.5 and n ¼ 1,2,3,5,7,9, 11,13,15. The distribution function FZ (z) is given by integrating Eq. (7.8.2): ðz 1 z n1 (1=2)(z=s)2 e dz z . 0 (7:8:3) Fz (z) ¼ n=21 G(n=2) sn 02 There is no closed-form solution for Eq. (7.8.3), but it can be expressed in terms of the incomplete gamma function GZ (z) defined in Eq. (7.5.5). By substituting y ¼ z2 =2s2 and
FIGURE 7.8.1
7.8 Chi and Other Allied Distributions
105
dz ¼ (s2 dy)=z in the in Eq. (7.8.3), we can write 2n=21 FZ (z) ¼ n=21 2 G(n=2) ¼
2n=21 2n=21 G(n=2)
ð z2 =2s2
y(n=21) ey dy
0
G(z2 =2s2 )
n G 2 2 (n=2) (z =2s ) ¼ 2 G(n=2)
(7:8:4)
where G(z2 =2s2 ) (n=2) is the incomplete gamma function defined in Eq. (7.5.5). Clearly, as z ! 1, G(z2 =2s2 ) (n=2) ! G(n=2), thus showing that FZ (1) ¼ 1. There are two very important special cases of the chi distribution when n ¼ 2 and n ¼ 3. Rayleigh Distribution (n 5 2)
When n ¼ 2 in the chi distribution of Eq. (7.8.2), we have ( z ) (1=2)(z=s)2 e z . 0 fZ (z) ¼ s2 0 z0
(7:8:5)
This is called the Rayleigh density and finds wide application in communications fading channels. If we have two independent zero mean Gaussian random variables X and Y with variance s2, and if we make the transformation X ¼ Z cos(Q) and Y ¼ Z sin(Q), pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 then Z ¼ X þ Y 2 is Rayleigh-distributed, which is a chi distribution with 2 degrees of freedom. Q is uniformly p distributed in the interval (0,2p]. The Rayleigh density is ffiffiffiffiffi shown in Fig. 7.8.2 for s ¼ 10,4,5. The cdf corresponding to the Rayleigh density can be expressed in a closed form as follows: ðz j j2 =2s2 FZ (z) ¼ e dj or, 2 0s ( 2 2 1 ez =2s z . 0 ¼ (7:8:6) 0 z0
FIGURE 7.8.2
106
Other Continuous Distributions
Maxwell Density (n 5 3)
This is one of the earliest distributions derived by Maxwell and Boltzmann from statistical mechanics considerations. They derived that the velocity vi of gas molecules in any direction i, based on the assumptions that the molecules are identical and distinguishable, and can occupy any energy state, follow the probability density function fV (vi ) ¼
e(1i =kT) Z
(7:8:7)
where 1i is the energy function given by 1i ¼ mv2i =2 with m as the mass of the particle, k is the Boltzmann constant, and T is the absolute temperature. Z is the normalization constant found from integrating Eq. (7.8.7) over all vi, or rffiffiffiffiffiffiffiffiffiffiffi ð 1 1 (mv2i =2kT) 2pkT e dvi ¼ 1 and Z ¼ Z 1 m so that the velocity density function along any direction follows a Gaussian distribution given by rffiffiffiffiffiffiffiffiffiffiffi m (mv2i =2kT) , i ¼ x,y,z (7:8:8) e fV (vi ) ¼ 2pkT From Eq. (7.8.8) the velocity vector V ¼ fvx,vy,vzg will be distributed with density function, " # m 3=2 m(v2x þ v2y þ v2z ) exp fV (vx ,vy ,vz ) ¼ (7:8:9) 2pkT 2kT Eq. (7.8.9) is a chi-square density with 3 degrees of freedom. The speed x of gas molecules will be defined by qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x ¼ v2x þ v2y þ v2z Hence x is a distributed as a chi-density with three degrees of freedom. Its probability density can be derived as h m i3=2 2 f (x) ¼ 4p x2 emx =2kT , x . 0 (7:8:10) 2pkT If we substitute kT=m ¼ s2 in the Maxwell – Boltzmann distribution of Eq. (7.8.10), we obtain rffiffiffiffi 2 x2 x2 =2s2 e , x.0 (7:8:11) fX (x) ¼ p s3 Equation (7.8.11) is a special case of the chi density of Eq. (7.8.2) with n ¼ 3 degrees of freedom. The pdf values fX(x) for the Maxwell and Rayleigh densities are compared in pffiffiffiffiffi Fig. 7.8.3 for values of s ¼ 10, 4, and 5. The cdf FX(x) corresponding to the Maxwell density fX(x) is obtained by integrating Eq. (7.8.11) and is given by rffiffiffiffi ð x 2 2 j j2 =2s2 e dj (7:8:12) FX (x) ¼ p 0 s3
7.8 Chi and Other Allied Distributions
107
FIGURE 7.8.3
There is no closed form solution for Eq. (7.8.12). However, integrating Eq. (7.8.12) by parts, it can be expressed as rffiffiffiffi pffiffiffi ð x 2 2 x x2 =2s2 j2 =2s2 e FX (x) ¼ pffiffiffiffi e dj ps s p 0 rffiffiffiffi x 2 x x2 =2s2 e ¼ erf pffiffiffi x.0 (7:8:13) ps 2s where erf(x) is as defined in Eq. (6.4.6). Rice’s Density
In the Rayleigh distribution discussed earlier, the random variables X and Y must be zero mean with the same variances s2. In many communication problems involving fading channels, X and Y will not be zero mean p and will have mean values equal to mX and mY. If ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi the random variable Z is given by Z p ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X 2 þffi Y 2 , then fZ (z) has a Rice density [49,50] with a noncentrality parameter m ¼ m2X þ m2Y , given by 2 z (z þ m2 ) mz fZ (z) ¼ 2 exp I0 2 (7:8:14) 2s2 s s where I0(z) is the zeroth-order modified Bessel function of the first kind. The general nth-order modified Bessel function is given by the series In (z) ¼
1 z n X
2
(z2 =4)k k! G(n þ k þ 1) k¼0
If n is an integer, then In(z) can also be given by ð 1 p z cos u In (z) ¼ e cos (nu) du p 0 and I0(z) the zeroth-order modified Bessel function by ð 1 X 1 p z cos u (z2 =4)k e du ¼ I0 (z) ¼ 2 p 0 k¼0 (k!) and I0(z), I1(z), I2(z), I3(z) are shown in Fig. 7.8.4.
(7:8:15)
(7:8:16)
(7:8:17)
108
Other Continuous Distributions
FIGURE 7.8.4
From Fig. 7.8.4 we see that I0(0) ¼ 1, and hence when m ¼ 0 (zero mean), then fZ (z) is Rayleigh-distributed. Rice’s density is shown in Fig. 7.8.5 for m ¼ 0,2,4,6,8,10,12,14, and m ¼ 0 corresponds to the Rayleigh density. The distribution function FZ(z) corresponding to Eq. (7.8.14) is given by integrating fZ (z): ðz j (j2 þ m2 ) mj exp I0 2 dj (7:8:18) FZ (z) ¼ 2 2 s 2s s 0 There is no closed-form solution for this integral.
Nakagami Density
The Nakagami distribution [49], like the Rayleigh density and unlike Rice’s distribution, also belongs to the class of central chi-square distributions. It is used for modeling data from multipath fading channels and has been shown to fit empirical results more generally than the Rayleigh distributions. Sometimes, the pdf of the amplitude of a mobile signal can also be described by the Nakagami m distribution.
FIGURE 7.8.5
7.8 Chi and Other Allied Distributions
109
The Nakagami distribution may be a reasonable means to characterize the back scattered echo from breast tissues to automate a scheme for separating benign and malignant breast masses. The shape parameter m has been shown to be useful in tissue characterization. Chi-square tests showed that this distribution is a better fit to the envelope than the Rayleigh distribution. Two parameters, m (effective number) and V (effective cross section), associated with the Nakagami distribution are used for the effective classification of breast masses. The Nakagami m density is defined by fX (x) ¼
2 h m im 2m1 mx2 =V x e G(m) V
(7:8:19)
where m is the shape parameter and V controls the spread of the distribution. The interesting feature of this density is that, unlike the Rice distribution, it is not dependent on the modified Bessel function. The Nakagami family is shown in Fig. 7.8.6 for V ¼ 1 and for m ¼ 3,2,1.5,1,0.75,0.5. The cdf FX(x) is given by ðx 2 h m im 2m1 mj2 =V FX (x) j e dj (7:8:20) 0 G(m) V This has no closed form solution. However, it can be given in terms of the incomplete gamma function by substituting y ¼ mj2/V and dj ¼ (V=2mj) dy in Eq. (7.8.20). This results in m 2 mj2 1 mj2 =V e dj FX (x) ¼ j V 0 G(m) m ð mx2 =V 2 2 mj2 1 emj =V ¼ dy G(m) V 2(mj2 =V) 0 ð mx2 =V 2 m y 1 y e dy ¼ G(m) 2y 0 ð mx2 =V Gmx2 =V (m) 1 m1 y y ¼ e dy ¼ G(m) G(m) 0 ðx
Since Gmx2 =V (m) ! G(m) as x ! 1 we have FX (1) ¼ 1:
FIGURE 7.8.6
(7:8:21)
110
Other Continuous Distributions
B 7.9
STUDENT-t DENSITY
The t distribution first developed by William Gosset under the pseudonym “Student” is used to estimate the confidence intervals for the unknown population mean where the variance is also unknown. In this case the statistic used for determining the confidence interval is given by T¼
m^ X mX pffiffiffi s= ^ n
(7:9:1)
and the distribution of T is governed by the Student-t distribution with density given by
nþ1 ½(nþ1)=2 G t2 2 1þ fT (t) ¼ pffiffiffiffiffiffi n n np G 2
1 , t , 1
(7:9:2)
where n is the number of degrees of freedom of T. It will be shown in Example 13.4.5 that if Z is a standard Gaussian random variable, Wn is a chi-square random variable with n degreespffiffiffiffiffiffiffiffiffiffiffi of freedom, and Z and Wn are independent, then the random variable T ¼ Z= Wn =n is t-distributed with n degrees of freedom. The Student-t distribution with n ¼ 2 is compared to a standard Gaussian in Fig. 7.9.1. After integrating the density function, we can write the distribution function FT (t) as 1 ½(nþ1)=2 ðt G n þ t2 2 1þ FT (t) ¼ dt n n 1 pffiffiffiffiffiffi np G 2 This is shown in Fig. 7.9.2.
FIGURE 7.9.1
(7:9:3)
7.10
Snedecor F Distribution
111
FIGURE 7.9.2 Limit of Student-t as n ! 1
We shall find the limit as n ! 1 in Eq. 7.9.2. We have ½(nþ1)=2 t2 2 lim 1 þ ¼ et =2 n!1 n 1 G nþ 1 2 ¼ pffiffiffiffiffiffi lim n!1 pffiffiffiffiffiffi n 2p np G 2
(7:9:4)
and hence ½vþ(1=2) G( nþ1 ) t2 1 2 lim pffiffiffiffiffiffi 2 n 1 þ ¼ pffiffiffiffiffiffi et =2 n!1 np G( ) n 2p 2
(7:9:5)
that is, as the number of degrees of freedom n goes to infinity, the Student-t tends to a standard Gaussian. If X1, X2, . . . , Xn are n independent identically distributed P Gaussian random variables with mean mX and standard deviation sX, then m^ X ¼ (1=n) ni¼1 Xi is also Gaussiandistributed with mean mX and variance s2X =n.PSince we do not know s2X , we use the sample variance given by s^ 2X ¼ 1=(n 1) ni¼1 (Xi m^ X )2 and s^ 2X is chi-squaredistributed with n – 1 degrees of freedom. Under these conditions the random variable T¼
(m^ X mX ) pffiffiffi s^ X = n
(7:9:6)
has a t distribution with n 2 1 degrees of freedom.
B 7.10
SNEDECOR F DISTRIBUTION
If X1 and X2 are two independent chi-squared-distributed random variables with n1 and n2 degrees of freedom, then the density of the random variable F ¼ (X1 =n1 )=(X2 =n2 ) is n þ n n n1 =2 1 2 1 G 2 n2 x(n1 2)=2 , x . 0 (7:10:1) fn1 ,n2 (x) ¼ n1 n2 1 þ n1 (n1 þn2 )=2 G x G 2 2 n2
112
Other Continuous Distributions
FIGURE 7.10.1
FIGURE 7.10.2
This density is shown in Fig. 7.10.1 for three cases: n1 ¼ 15, n2 ¼ 25: n1 ¼ 5, n2 ¼ 25: n1 ¼ 10, n2 ¼ 5. Note that in general fn1 ,n2 (x) is not equal to fn2 ,n1 (x). The corresponding distribution function is given by n þ n n n1 =2 1 2 1 ðx G 2 n2 (n1 2)=2 Fn2 , n1 (x) ¼ dj (7:10:2) n n 1 þ n (n1 þn2 )=2) j 0 1 2 1 G j G 2 2 n2 Equation (7.10.2) is shown in Fig. 7.10.2 for the same values of n1 and n2 as in Fig. 7.10.1. As n1 and n2 tend to infinity, then the density function becomes more sharply spiked around 1. The F distribution is used to draw inferences about the population variances in the case of two-sample situations.
B 7.11
LOGNORMAL DISTRIBUTION
A random variable X has lognormal density with parameters a, m, and s2 if the transformed random variable Y ¼ ln(X ) has a normal distribution with mean m and variance
7.11
Lognormal Distribution
113
FIGURE 7.11.1
s2. This density, defined only for positive values of x, is given by ( ) 1 1 ln (x a) m 2 exp fX (x) ¼ pffiffiffiffiffiffi , 2 s 2p s (x a)
x . a; m, s . 0
(7:11:1)
where a is the location parameter, m is the scale parameter, and s is the shape parameter. When a and m are zero, fX(x) is called the standard lognormal distribution with parameter s. The case where a ¼ 0 and m ¼ 1 is shown in Fig. 7.11.1 for s ¼ 0.4,0.5,1.0,1.5,2 for values of x . 0.1. The cumulative distribution function FX(x) is obtained by integrating Eq. (7.11.1) and is given by ðx
2 1 2 pffiffiffiffiffiffi e( ln (ja)m) =2s dj 2p s(j a) 0 ð ln (xa)m s 1 1 ln (x a) m 2 pffiffiffiffiffiffi eh =2 dh ¼ erf pffiffiffi ¼ 2 2p 2s 0
FX (x) ¼
(7:11:2)
where 1 1 x 2 pffiffiffi eh =2 dh ¼ erf pffiffiffi 2 2p 2 0
ðx
The cdf FX(x) is shown in Fig. 7.11.2 for a ¼ 0, m ¼ 1 and for the same values of s as the density function. In finance, prices for primary instruments such as stocks and bonds are often assumed to be lognormally distributed. A standard assumption for the evolution of a stock price is that in an interval Dt the fractional increase in the stock price, DS/S, is normally distributed with mean mDt and variance s2Dt. The lognormal distribution is also used to model the lifetime of units whose failures are of a fatigue– stress nature. Consequently, the lognormal distribution is an adjunct to the Weibull distribution when attempting to model these types of units.
114
Other Continuous Distributions
FIGURE 7.11.2
B 7.12
BETA DISTRIBUTION
The beta distribution is a two-parameter a and b family of density functions fX(x) for 0 x 1. It is often used to represent proportions and percentages. It has the following probability density function: fX (x) ¼
G(a þ b) a1 x (1 x)b1 , G(a)G(b)
0x1
(7:12:1)
The choices of the two parameters a and b makes it amenable to a variety of shapes. For example, for a ¼ b ¼ 1, fX(x) becomes the rectangular distribution. The triangular distribution arises when one parameter has the value 2 and the other is unity. If both parameters are less than unity and positive, the density becomes U-shaped, and if only one of the parameters is less than unity, it becomes J-shaped. The distribution is unimodal when both parameters are positive, and symmetric when they are equal. The pdf curves are shown in Fig. 7.12.1 for the cases described above. The beta distribution is related to the Student-t distribution [Eq. (7.9.2)] by the following transformation. If Z is a t-distributed
FIGURE 7.12.1
7.13
Cauchy Distribution
115
FIGURE 7.12.2
random variable with n degrees of freedom, then the transformation 1 Z X ¼ þ pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2 n þ Z2 is beta-distributed with a ¼ n/2 and b ¼ n/2. Example 7.12.1 We will give an example of how the beta distribution is used in stockmarket analysis. The analyst from his model of stock performances predicts before the start of each day that the proportion of the listed stocks has a beta distribution with parameters a ¼ 7.3 and b ¼ 5.2. The average value is a=(a þ b) ¼ 0:584. We want to find the probability that more than 70% of the stocks increase in value and this is given by ð 0:7 P{X . 0:7} ¼ 1 fX (x) dx ¼ 1 0:794 ¼ 0:206 0
This probability is 20.6%, as illustrated in Fig. 7.12.2.
B 7.13
CAUCHY DISTRIBUTION
Cauchy distribution resembles a normal distribution in appearance but has much heavier tails. However, in studying hypothesis tests that assume normality, seeing how the tests perform on data from a Cauchy distribution is a good indicator of how sensitive the tests are to heavy-tail departures from normality. The general Cauchy density is given by fX (x) ¼
b p½(x a)2 þ b2
(7:13:1)
where a is the location parameter and b is the shape parameter. It is an example of a pathological case in the sense the moments do not exist even though the first moment can be evaluated to be equal to 0 using the Cauchy principle value method. The Cauchy density is plotted for a ¼ 5 and b ¼ 2,3,4,5 in Fig. 7.13.1. Since the moments do not exist for the Cauchy distribution, it is characterized by the width at half the maximum value equal to 2b, called full width half maximum (FWHM). For b ¼ 2 the maximum is 0.159154 and the half-maximum is 0.079577. This is shown in Fig. 7.13.2. For comparison, a Gaussian distribution is superposed on the same figure for the same FWHM of 4 that is, a Gaussian with mean 5 and s ¼ 4/2.35482 ¼ 1.69864.
116
Other Continuous Distributions
FIGURE 7.13.1
FIGURE 7.13.2
FIGURE 7.13.3
7.14
Pareto Distribution
The distribution function FX(x) can be determined as follows: ðx 2 1 2 e( ln (ja)m) =2s dj FX (x) ¼ pffiffiffiffiffiffi 2p s(j a) 0 Substituting (j a)=b ¼ tan(u) in Eq. (7.13.2), we have ð tan1 {(xa)=b} sec2 (u)du FX (x) ¼ p sec2 (u) p=2 tan1 {(xa)=b} u 1 xa 1 ¼ ¼ tan1 þ p p=2 p b 2
117
(7:13:2)
(7:13:3)
The term FX(x) is shown in Fig. 7.13.3 for the same values of a and b as in Fig. 7.13.1.
B 7.14
PARETO DISTRIBUTION
Many synthetic and naturally occurring phenomena, including city sizes, incomes, word frequencies, and earthquake magnitudes, are distributed according to a power-law distribution. A power law implies that small occurrences are extremely common, whereas large instances are extremely rare. These can be modeled as a Pareto distribution. In addition, traffic measurements reveal features such as nonstationarity, long-range dependence, and significant fluctuations in different timescales (e.g., self-similarity). The essential features of the burstiness observed in measurements can be modeled as a heavy-tailed on-time and off-time distributions such as the Pareto distribution. The Pareto density is given by a b aþ1 fX (x) ¼ , x.0 (7:14:1) b xþb and the distribution function can be obtained from integrating Eq. (7.14.1): ðx a b aþ1 b a FX (x) ¼ dj ¼ 1 xþb 0b jþb
FIGURE 7.14.1
(7:14:2)
118
Other Continuous Distributions
The Pareto density and the distribution functions are plotted in Fig. 7.14.1 for a ¼ 5 and b ¼ 10,15,20. Note that the tails are long compared to other distributions. One of the uses of the Pareto distribution is to model the high end of the distribution of incomes.
B 7.15
GIBBS DISTRIBUTION
The Gibbs distribution has been widely used in image processing and pattern recognition problems. An example using this distribution in tomographic imaging is presented in Chapter 23. It is a generalization of the Maxwell – Boltzmann distribution and has origins in statistical mechanics. It is defined on a random field N as P(v) ¼
where,
1 ð1=TÞU(v) e Z
(7:15:1)
P U(v) ¼ wjk V(vj vk ) j,keN V(v)
Total energy function Potential function, where v is some vector field
wjk j, k
Weighting functions Pixels in neighboring field N
T
Absolute temperature that controls sharpness of distribution P Normalization constant given by Z ¼ v e(1=T)U(v) , also called the partition function
Z
The potential function V(v) depends on the local configuration of the field. Some of the properties of potential functions [38] are given below: 1. V(0) ¼ 0 2. V(v) 0: nonnegative 3. V(v) ¼ V(v): even function d 4. V(v) . 0 for v . 0 dv 5. lim V(v) ! 1 v!1
6.
sup 0v,1 2
7.
B 7.16
d dv V(v)
,1
d V(v) . 1 dv2
MIXED DISTRIBUTIONS
There are any number of derived and mixed distributions, depending on the application, and new distributions are also constantly being formulated. For example, the doubledouble-exponential or double-Laplace distribution, given by
p ðjxjÞ 1 p jxjb e a þ e f (x) ¼ (7:16:1) 2a 2b
7.17
Summary of Distributions of Continuous Random Variables
119
FIGURE 7.16.1
with f1 (x) ¼
p ðjxjÞ 1p e a : f2 (x) ¼ e 2a 2b
jxj b
is shown in Fig. 7.16.1 with p ¼ 0.3, a ¼ 1, b ¼ 4. This distribution has been specially constructed for modeling navigation errors over transatlantic flights consisting of both the pilot and instrument errors [51]. The pilot errors are modeled by f2(x) and the instrument errors, by f1(x). The instrument errors have a sharper peak than do pilot errors, which extend over a greater distance. Whereas the instrument are easier to eliminate, the pilot errors are more difficult. The composite error curve is f(x) shown in Fig. 7.16.1. The distributions given here are by no means exhaustive, and only some of the more important distributions are discussed. More comprehensive sets of distributions can be obtained from Web references W1 and W2 (see References at end of book). A complete list of all the densities and distributions discussed in this chapter is tabulated in Section 7.17.
B 7.17 SUMMARY OF DISTRIBUTIONS OF CONTINUOUS RANDOM VARIABLES Density Function
Name Uniform
Triangular
1 ba xa ; a,xb (b a)2 (x 2b þ a) (b a)2 b , x 2b a
Closed-Form Distribution Function xa ba 1 x a 2
a,xb 2 b a 1 x þ a 2b 2 1 2 ba b , x 2b a (continued)
120
Other Continuous Distributions
Density Function
Name Exponential Laplace
Closed-Form Distribution Function
1 lx e u(x), l . 0 l l ljxj e , l0 2
(1 elx )u(x), 1 lx e , 2
l.0
x0
1 12 elx , x . 0 Erlang
l(lx)n1 elx , (n 1)!
1
n integer
n integer; x . 0
l(lx)a1 elx , G(a)
a,l,x . 0
ala (x m)a1 e½l(xm) x.m 1 ðx2 =2Þ Standard Gaussian pffiffiffiffiffiffi e 2p
Weibull
Chi-square
k!
k¼0
x.0 Gamma
n1 X (lx)k elx
2
(x2 )(n=2)1 ex =2s n (2s2 )n=2 G 2
Gx (a) G(a) Incomplete gamma
a a
1 e½l(xm) , 1 1 x þ erf pffiffiffi 2 2 2 n Gx2 =2s2 n 2 , G 2
2
x.m
x ¼ x2
Incomplete gamma 1
Chi
2(n=2)1 G
n
xn1 (1=2)(x=s)2 e s2
2
x.0
n Gx2 =2s2 n2 G 2 Incomplete gamma
Rayleigh
z (z2 =s2 ) e , s2
Maxwell
rffiffiffiffi 2 x3 ðx2 =2s2 Þ e , p s3
Rice
x ½(x2 þm2 )=2s2 mx e I0 2 , s2 s
2
2
1 eðz =2s Þ , z . 0
z.0
x pffiffiffi s 2 erf 2s2
x.0
rffiffiffiffi 2 ðx2 =2s2 Þ e p
x.0 x.0
No closed form (continued)
7.17
Summary of Distributions of Continuous Random Variables
Density Function
Name Nakagami
Student-t(n)
Snedecor F(n1 ,n2 )
Closed-Form Distribution Function
2 m m 2m1 mx2 =V x e G(m) V
Gmx2 =V (m) G(m)
x.0
Incomplete gamma
vþ1 ½(vþ1)=2 G t2 2 1 þ pffiffiffiffiffiffi v v vp G 2 n þ n n n1 =2 1 2 1 x(n1 2)=2 2 n2 n n n1 (n1 þn2 )=2 1 2 G G 1þ x 2 2 n2 G
121
No closed form
No closed form
x.0 Lognormal
Beta
2 1 2 pffiffiffiffiffiffi e½ln (xa)m =2s 2p s(x a)
G(a þ b) a1 x (1 x)b1 G(a)G(b)
1 ln (x a) m pffiffiffi erf 2 s 2 No closed form
0x1 Cauchy
b p½(x a)2 þ b2
1 1 1 x a þ tan 2 p b
Pareto
aþ1 a b , x.0 b xþb
1
Definitions
Zero-order modified Bessel function ð 1 p x cos (u) I0 (x) ¼ e du p 0
b xþb
a ,
x.0
Error function ð 2 x j2 erf (x) ¼ pffiffiffiffi e dj p 0 Incomplete gamma function ðx Gx (a) ¼ ja1 ej dj 0
CHAPTER
8
Conditional Densities and Distributions
B 8.1 CONDITIONAL DISTRIBUTION AND DENSITY FOR P{A}= 0 In Section 2.2 we defined conditional probability for events A and B that are subsets of the sample space S as 9 P{A > B} = P{B j A} ¼ if P{A} = 0 (8:1:1) P{A} ; P{A > B} ¼ P{B j A}P{A} We will extend this to distributions FX(x) and densities fX(x). The conditional distribution FX(x j A), given that an event A has occurred, is defined as FX (x j A) ¼ P{X x j A} ¼
P{X x, A} P{A}
if
P{A} = 0
(8:1:2)
and if the event fX xg is independent of A, then PfX x j Ag ¼ PfX xg. The corresponding conditional density fX(x j A) is defined by dFX (x j A) dx P{x , X x þ Dx j A} ¼ lim Dx!0 Dx
fX (x j A) ¼
Properties of Conditional Distributions 1. FX (1 j A) ¼ 1; FX (1 j A) ¼ 0 2. FX (x j A) is right-continuous 3. FX (x2 j A) FX (x1 j A) ¼ P{x1 , X x2 j A} Ðx 4. FX (x j A) ¼ 1 fX (j j A) dj
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
122
(8:1:3)
8.1 Conditional Distribution and Density for P{A}= 0
123
Example 8.1.1 A die is tossed and, corresponding to the faces of the die, a random variable is defined by X ¼ 10k, k ¼ 1,2,3,4,5,6. The conditioning event A is given by A ¼ fthe toss resulted in an even numberg. We will first find the unconditional distribution FX(x) and then the conditional distribution FX(x j A). Using the same mapping techniques discussed in Chapter 2, the unconditional cdf is given by FX (x) ¼
6 1X u(x 10k) 6 k¼1
Since the sample space has been reduced to 3 points by the conditioning event A, the conditional cdf FX(x j A) is given by Fx (x j A) ¼ ¼
P{X 10k, X even} FX (10k), k ¼ 2,4,6 ¼ P{X even} 1=2 3 1X u(x 20k) 3 k¼1
Both these distributions are shown in Fig. 8.1.1. Conditioning Event A Depends on x. We shall now consider the case when the event A is dependent on x. Three cases arise: 1. A ¼ {X x1 }: Eq. (8.1.2) takes the form: FX (x j A) ¼
P{X x, X x1 } P{X x1 }
(8:1:4)
Two situations arise: (a) x . x1 : Since x . x1, the joint event fX x, X x1g ¼ fX x1g, and hence Eq. (8.1.4) can be rewritten as FX (x j A) ¼
P{X x1 } ¼1 P{X x1 }
(8:1:5)
(b) x x1: In this case the joint event fX x, X x1g ¼ fX xg, and hence Eq. (8.1.4) becomes FX (x j A) ¼
P{X x} FX (x) ¼ P{X x1 } FX (x1 )
FIGURE 8.1.1
(8:1:6)
124
Conditional Densities and Distributions
The corresponding density function is given by 8 < fX (x) fX (x j A) ¼ FX (x1 ) : 0
x x1
(8:1:7)
x . x1
2. A ¼ fx1 , X x2g with x1 , x2 Analogous to Eq. (8.1.4), we have the following: FX (x j A) ¼
P{X x, x1 , X x2 } P{x1 , X x2 }
(8:1:8)
Here three situations arise: (a) x . x2: FX (x j A) ¼
P{x1 , X x2 } ¼1 P{x1 , X x2 }
(8:1:9)
(b) x1 . x x2: FX (x j A) ¼
P{x1 , X x} FX (x) FX (x1 ) ¼ P{x1 , X x2 } FX (x2 ) FX (x1 )
(8:1:10)
FX (x j A) ¼
P{1} ¼0 P{x1 , X x2 }
(8:1:11)
(c) x x1:
The corresponding density function is given by 8 0 x x1 > > < fX (x) x1 , x x2 fX (x j A) ¼ > F (x ) FX (x1 ) > : X 2 0 x . x1
Example 8.1.2
(8:1:9)
A random variable has a Gaussian distribution, given by ( " #) 1 1 x4 2 fX (x) ¼ pffiffiffiffiffiffi exp 2 2 2 2p
We will find the conditional distribution FX(x j A) and the conditional density fX(x j A) given that (1) A ¼ fX 4.5g and (2) A ¼ f1.5 , X 4.5g. Using Eq. (8.1.6), we obtain 1. A ¼ fX 4.5g: FX (x j A) ¼
FX (x) 1 1 x4 ¼ 1 þ erf pffiffiffi 9 9 2 2 2 FX FX 2 2
and using Eq. (8.1.10), we obtain 2. A ¼ f 1.5 , X 4.5 g:
1 x 4) 3 pffiffiffi 1 þ erf FX FX (x) FX (x1 ) 2 2 (2 2 FX (x j A) ¼ ¼ 9 3 FX (x2 ) FX (x1 ) FX FX 2 2
8.1 Conditional Distribution and Density for P{A}= 0
125
FIGURE 8.1.2
where 2 erf(x) ¼ pffiffiffiffi p
ðx
2
ej dj
0
The corresponding density functions are as follows for (1) ( " 2 #) 1 1 x4 pffiffiffiffiffiffi exp 2 2 2 2p fX (x j A) ¼ 9 FX 2 and for (2)
( " #) 1 1 x4 2 pffiffiffiffiffiffi exp 2 2 2 2p fX (x j A) ¼ 9 3 FX FX 2 2
The conditional distribution and densities FX(x j A) and fX(x j A) along with the unconditional distributions and densities FX(x) and fX(x) are shown in Fig. 8.1.2 for (1) A ¼ fX 4.5g and in Fig. 8.1.3 for (2) A ¼ f1.5 , X 4.5g.
FIGURE 8.1.3
126
Conditional Densities and Distributions
B 8.2 CONDITIONAL DISTRIBUTION AND DENSITY FOR P{A} 5 0 In Section 8.1 we assumed that the probability of the conditioning event A is not zero. We will now determine the conditional probabilities when PfAg ¼ 0. We need to find PfB j X ¼ x1g. Since PfX ¼ x1g ¼ 0 for a continuous random variable, we have to use limiting forms to determine this probability. We will write PfB j X ¼ x1g as P{B j X ¼ x1 } ¼ lim P{B j x1 , X x1 þ Dx} Dx!0
(8:2:1)
The conditional probability PfB j x1 , x x1 þ Dxg in Eq. (8.2.1) can be written as follows: P{B j x1 , X x1 þ Dx} ¼
P{B, x1 , X x1 þ Dx} P{x1 , X x1 þ Dx}
¼
P{x1 , X x1 þ Dx j B}P{B} FX (x1 þ Dx) FX (x1 )
¼
FX {x1 þ Dx j B} FX {x1 j B}P{B} FX (x1 þ Dx) FX (x1 )
(8:2:2)
Dividing numerator and denominator of Eq. (8.2.2) by Dx and taking the limit as Dx ! 1 results in P{B j X ¼ x1 } ¼ lim
Dx!0
¼
{FX {x1 þ Dx j B} FX {x1 j B}P{B}g=Dx {FX (x1 þ Dx) FX (x1 )}=Dx
fX (x1 j B)P{B} fX (x1 )
(8:2:3)
Case of B: fY 5 yg
In an analogous manner we can find an expression for the conditional probability PfB j X ¼ xg when B is the event B:fY ¼ y). In this case we can write P{Y ¼ y j X ¼ x} ¼ lim P{y , Y y þ Dy j x , X x þ Dx} Dx!0 Dy!0
¼ lim
Dx!0 Dy!0
P{y , Y y þ Dy, x , X x þ Dx} P{x , X x þ Dx}
Dividing the numerator and denominator of Eq. (8.2.4) by Dx Dy, we have, lim P{y , Y y þ Dy j x , X x þ Dx}
Dx!0 Dy!0
½P{y , Y y þ Dy j x , X x þ Dx}=DxDy Dx!0 ½P{x , X x þ Dx=DxDy
¼ lim
Dy!0
¼
fXY (x,y) Dy fX (x)
(8:2:4)
8.2 Conditional Distribution and Density for P{A} 5 0
127
Hence
lim
Dx!0 Dy!0
P{y , Y y þ Dy j x , X x þ Dx} ¼ fYjX ( y j X ¼ x) Dy ¼ fYjx ( y j x) ¼
fXY (x,y) fX (x)
(8:2:5)
The conditional distribution FYjX( y j X xg ¼ PfY y j X xg can be obtained from the basic definition of conditional probability as follows:
FYjX ( y j X x) ¼ P{Y y j X x} ¼
P{Y y, X xg FXY (x, y) ¼ P{X x} FX (x)
(8:2:6)
Even though Eq. (8.2.5) is of exactly the same form as Eq. (8.2.6), it cannot be obtained from the definition of conditional probability but must be obtained from limiting considerations.
Point of Clarity
Since fYjX( y j x) has been defined as fYjX( y j X ¼ x), it is not the derivative with respect to y of the distribution FYjX ( y j x), which has been defined as FYjX( y j X x). Bearing this in mind, we can write the following equations for clarity: dFYjX ( y j X x) dFyjX ( y j x) ¼ ¼ fYjX ( y j X x) dy dy (8:2:7) dFYjX ( y j X ¼ x) ¼ fYjX ( y j X ¼ x) ¼ fYjX ( y j x) dy Example 8.2.1 The probability that a machine fails in x months is given by (1 2 e 2x/d) u(x). If the machine failed sometime before c months, it is desired to find the conditional distribution FX(x j X c) and the corresponding conditional density. Here we can use the definition of conditional distribution and write for FX(x j X c): FX (x j X c) ¼ P{X x j X c} ¼
¼
P{X x, X c} P{X c}
8 1 ex=d > > > < 1 ec=d
0,xc
> > > :
x.c x0
1 0
The corresponding density function for the event fX cg is dex=d fX (x j X c) ¼ 1 ec=d 0
0,xc otherwise
128
Conditional Densities and Distributions
FIGURE 8.2.1
We can now find the conditional distribution FX(x j X ¼ c) if it failed at x ¼ c months. Since PfX ¼ cg ¼ 0, this has to be solved using limiting considerations: FX (x j X ¼ c) ¼ lim P{X x j c , X c þ Dc} Dc!0
¼ lim
Dc!0
P{X x, c , X c þ Dc} P{c , X c þ Dc}
P{c , X c þ Dc j X x}P{X x}=Dc Dc!0 P{c , X c þ D c}=Dc
¼ lim
¼ fX (c j X x)
FX (x) fX (c, X x) FX (x) ¼ fX (c) FX (x) fX (c)
Hence FX (x j X ¼ c) ¼
1 x.c 0 xc
and the corresponding density function for the event fX ¼ cg is given by fX (x j X ¼ c) ¼ d(x c) These distribution and density functions are shown in Fig. 8.2.1 for c ¼ 2, d ¼ 2. Example 8.2.2
The joint density function of random variables X and Y is (1 fXY (x, y) ¼
p 0
x2 þ y2 1 otherwise
The marginal density function fX(x) is obtained by integrating fXY(x,y) over all y: ffi ð pffiffiffiffiffiffiffi 1x2
pffiffiffiffiffiffiffiffiffiffiffiffiffi dy 2 1 x2 fX (x) ¼ pffiffiffiffiffiffiffiffi ¼ p 1x2 p
8.2 Conditional Distribution and Density for P{A} 5 0
129
FIGURE 8.2.2
From Eq. (8.2.5), the conditional density fYjX (y j X ¼ x) is given by 8 1 < pffiffiffiffiffiffiffiffiffiffiffiffi fXY (x, y) 1=p ffi x2 þ y2 1 ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ 2 1 x2 fYjX ( y j X ¼ x) ¼ fX (x) 2 1 x2 =p : 0 otherwise Note that for each value of x the pdf fYjX (y j X ¼ x) is a constant and not a function of y. The corresponding conditional pffiffiffiffiffiffiffiffiffiffiffiffiffi distribution function FYjX (y j X ¼ x) is obtained by integrating the pdf from 1 x2 to y as shown below: ðy 1 y 1 FYjX (y j X ¼ x) ¼ pffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffi dh ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi þ 2 2 2 2 2 1x 1x 2 1 x In Fig. 8.2.2, the conditional pdf fYjX (y j X ¼ x) is shown as a function of x and the conditional cdf FYjX (y j X ¼ x) is shown as a function of y for x = 0.5. Example 8.2.3 This is an interesting example that again illustrates finding the conditional probability when the probability of the conditioning event is zero. Two friends A and B plan to meet between 7:00 and 8:00 p.m. However, they agree that each of them will wait only 15 min for the other. Each of them arrives independently at a random time between 7 and 8 p.m. We have to compute the probability that they will meet given that A arrives at 7:20 p.m. Let us first define these random variables. X ¼ {time of arrival of A after 7:00 p.m. in minutes} Y ¼ {time of arrival of B after 7:00 p.m. in minutes} The pdf fX(x) and fY( y) are both uniformly distributed in (0, 60] as shown in Fig. 8.2.3. Since X and Y are independent, the joint density fXY(x,y) ¼ fX(x) . fY( y) and is given by 1 1 1 fXY(x,y) ¼ 60 60 ¼ 3600 : The 60 60 Cartesian product space is shown in Fig. 8.2.4.
FIGURE 8.2.3
130
Conditional Densities and Distributions
FIGURE 8.2.4
If A comes at time x minutes after 7:00 p.m., then for the meeting to take place, we have to find the probability of the event j Y 2 X j 15 under the condition X ¼ x. Hence, we can write PfY 2 X j 15 j X ¼ xg ¼ Pf215 þ X Y 15 þ X j X ¼ xg, and we have to find the area under the shaded portion in Fig. 8.2.4. In essence, we have to integrate fY( y) in three different regions: (1) 0 , y x þ 15, 0 , x ¼ 15, (2) x 2 15 , y x þ 15, 15 , x 45, and (3) x þ 15 , y 60, 45 , x 60. Performing the indicated integration, we obtain 8 ð 15þx > dx > > 0 , x , 15 > > 60 > 0 > > < ð 15þx dx P{A and B meeting j X ¼ x} ¼ 15 , x 45 > 60 > 15þx > > ð 60 > > dx > > 45 , x 60 : 60 15þx 8 15 þ x > > 0 , x 15 > > 60 > > < 1 ¼ 15 , x 45 > 2 > > > > > : 75 x 45 , x 60 60 The probability that A and B meet if A arrives exactly x minutes after 7:00 p.m. is shown in Fig. 8.2.5. From the curve, the Pfmeet j X ¼ 20g ¼ 0.5. The total probability of A and B meeting is given by ð 60 P{meet} ¼
P{meet j x ¼ x}fX (x) 0
¼
1 60 30 15 105 7 þ þ ¼ ¼ 0:4375 ¼ 60 4 4 4 4:60 16
8.3 Total Probability and Bayes’ Theorem for Densities
131
FIGURE 8.2.5
B 8.3 TOTAL PROBABILITY AND BAYES’ THEOREM FOR DENSITIES We will use Eq. (8.2.3), shown in Eq. (8.3.1) as the starting point to derive the total probability law for density functions: P{B j X ¼ x1 } ¼
fX (x1 j B)P{B} fX (x1 )
(8:3:1)
Cross-multiplying and integrating both sides of the resulting equation, we obtain ð1
ð1 P{B j X ¼ x1 }fX (x1 )dx1 ¼
1
fX (x1 j B) P{B}dx1
(8:3:2)
1
In Eq. (8.3.2) we obtain the term ð1
ð1 fX (x1 j B)P{B}dx1 ¼ P{B}
1
fX (x1 j B)dx1 ¼ P{B} 1
and hence Eq. (8.3.2) becomes ð1 P{B j X ¼ x}fX (x)dx ¼ P{B}
(8:3:3)
1
where the subscript on x has been omitted without loss of generality. Equation (8.3.3) is the total probability law for density functions. We can now formulate Bayes’ theorem for density functions. From Eq. (8.3.1) we can write fX (x j B) ¼
P{B j X ¼ x}fX (x) P{B} P{B j X ¼ x}fX (x) 1 P{B j X ¼ x}fX (x)dx
¼ Ð1
(8:3:4)
132
Conditional Densities and Distributions
where fX(x j B) is the a posteriori pdf after observing the event B given the a priori pdf fX(x). If B is the event B ¼ {Y ¼ y}, then Eq. (8.3.4) can be written as fYjX (y j x)fX (x) 1 fYjX (y j x)fX (x)dx
fXjY (x j y) ¼ Ð 1
(8:3:5)
Equation (8.3.5) is Bayes’ theorem for connecting the a priori density fX(x) with the a posteriori density fXj Y (x j y) after observing the event fY ¼ yg.
Example 8.3.1 Let X be the random variable that a navigation system fails before x hours. Three suppliers A, B, and C have supplied the system to airlines with probabilities PfAg ¼ 0.35, PfBg ¼ 0.25, PfCg ¼ 0.4. The conditional probabilities of failure for the suppliers A, B, C are as follows: FX (x j A) ¼ (1 eax )u(x), a ¼ 1:625 104 FX (x j B) ¼ (1 ebx )u(x), b ¼ 5:108 104 FX (x j C) ¼ (1 ecx )u(x),
c ¼ 2:877 104
1. From the conditional probabilities of failure, we can find the probability density fX(x) as follows: fX (x j A) ¼ aeax u(x),
a ¼ 1:625 104 : P{A} ¼ 0:35
fX (x j B) ¼ bebx u(x),
b ¼ 5:108 104 : P{B} ¼ 0:25
fX (x j C) ¼ cebx u(x), c ¼ 2:877 104 : P{C} ¼ 0:40 and fX (x) ¼ fX (x j A)P{A} þ fX (x j B)P{B} þ fX (x j C)P{C} ¼ (0:56875eax þ 1:277ebx þ 1:1508ecx )104 u(x) 2. If the navigation system fails after x hours, we will find the probabilities PfA j X . xg, PfB j X . xg, PfC j X . xg. P{X . x} is given by P{X . x} ¼ (1 FX (x j A)ÞP{A} þ (1 FX (x j B))P{B} þ (1 FX (x j C))P{C} ¼ eax 0:35 þ ebx 0:25 þ ecx 0:40 The probability PfA j X . xg can be found from the equation
P{A j X . x} ¼
P{X . x j A}P{A} (1 FX (x j A))P{A} ¼ P{X . x} P{X . x}
8.3 Total Probability and Bayes’ Theorem for Densities
133
with similar equations for PfB j X . xg and PfC j X . xg. Hence these probabilities can be given by P{A j X . x} ¼
P{B j X . x} ¼
P{C j X . x} ¼
eax 0:35
eax 0:35 þ ebx 0:25 þ ecx 0:40
eax 0:35
ebx 0:25 þ ebx 0:25 þ ecx 0:40
ecx 0:40 eax 0:35 þ ebx 0:25 þ ecx 0:40
3. If the navigation system fails exactly at time X ¼ x, we will find PfA j X ¼ xg, P{B j X ¼ x}; and P{C j X ¼ x}. From Eq. (8.2.3) we have P{A j X ¼ x} ¼
fX (x j A)P{A} 0:56875 104 eax ¼ fX (x) fX (x)
P{B j X ¼ x} ¼
fX (x j A)P{B} 1:277 104 ebx ¼ fX (x) fX (x)
P{C j X ¼ x} ¼
fX (x j A)P{C} 1:1508 104 ecx ¼ fX (x) fX (x)
4. We will find the most likely supplier in part (3) if x ranges from 1000 to 10,000 and the total probability of a wrong decision Pe. The most likely supplier if the system failed at exactly x hours can be obtained by drawing the graphs of the probabilities PfA j X ¼ xg, PfB j X ¼ xg, and PfC j X ¼ xg as shown in Fig. 8.3.1. From the graph in Fig. 8.3.1 we can see that if the system fails between x ¼ 1000 and x ¼ 5629.17 h, the most likely supplier is C, and if it fails between x ¼ 5629.17 and
FIGURE 8.3.1
134
Conditional Densities and Distributions
10,000 h, the most likely supplier is A. The conditional probability PfB j X ¼ xg is smaller than the other two conditional probabilities in the two ranges. The probability of wrong decision Pe is given by Pe ¼ P{X 5629:17 j A}P{A} þ P{X . 5629:17 j C}P{C} þ P{B} Since supplier B does not figure in the wrong decision, PfBg appears without any multiplier. Hence P{wrong decision} ¼ Pe ¼ ½1 e5629:17a 0:35 þ ½1 (1 e5629:17c ) 0:4 þ 0:25 ¼ 0:539
CHAPTER
9
Joint Densities and Distributions
B 9.1
JOINT DISCRETE DISTRIBUTION FUNCTIONS
If the random variables fX ¼ X1, X2, . . . , Xng, are discrete, then we can define a joint probability mass function as pX1 Xn (x1 , . . . , xn ) ¼ P{X1 ¼ x1 , . . . , Xn ¼ xn }
(9:1:1)
and the total sum n X m
n X n X j
pX1 Xn (xi , . . . , xm ) ¼ 1
i
In addition, the marginal distributions are defined by n X
pX1 Xn (x1 , . . . , xk ) ¼ pX1 Xn1 (x1 , . . . , xn1 )
(9:1:2)
k
Example 9.1.1 (Discrete) An urn contains 4 red, 5 green, and 6 blue balls, and 3 balls are taken at random. Given the number of red balls represented by the random variable X and the number of green balls by the random variable Y, we want to find the joint probability pXY (k,m) of PfX ¼ k, Y ¼ mg, where k ranges from 0 to 4 and m ranges from 0 to 5. The total number of points in the sample space is n! 15! n C(n,r) ¼ ¼ ¼ ¼ 455 r r!(n r)! 3!12! The number of ways in which 3 balls can be picked from 15 balls is to choose k balls from 4 red balls, m balls from 5 green balls, and 3 2 k – m balls from 6 blue balls. Hence, the probability pXY (k,m) is given by C(6, 3 k m)C(4, k)C(5, m) k þ m 3 pXY (k,m) ¼ 0 otherwise
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
135
136
Joint Densities and Distributions
FIGURE 9.1.1
These probabilities are shown in Fig. 9.1.1 2 p(0,0) 1 6 p(1,0) 6 PXY ¼ C(15,3) 4 p(2,0) p(3,0)
for all values of k and m. 3 p(0,1) p(0,2) p(0,3) p(1,1) p(1,2) p(1,3) 7 7 p(2,1) p(2,2) p(2,3) 5 p(3,1) p(3,2) p(3,3)
As the probability mass functions PfX ¼ k) and PfY ¼ m) appear as sums in the margin, they are referred to as marginal probability mass functions. Thus, the probability of 2 red balls PfX ¼ 2g ¼ 66/455 and the probability of 3 green balls PfY ¼ 3g ¼ 10/455.
B 9.2
JOINT CONTINUOUS DISTRIBUTION FUNCTIONS
The joint distribution of n random variables fX1, X2, . . . , Xng, is the probability of the joint event fX1 x1g, . . . , Xn xn and is given by FX1 Xn (x1 , . . . , xn ) ¼ P{X1 x1 , . . . , Xn xn }
(9:2:1)
In particular, if X and Y are two random variables, the joint distribution function FXY (x,y) is given by FXY (x,y) ¼ P{X x, Y y}
(9:2:2)
The corresponding density function is given by fXY (x,y) ¼
@2 FXY (x,y) @x@y
(9:2:3)
Using the definition of the second derivative, Eq. (9.2.3) can also be given in terms of the distribution function as follows: FXY (x þ D x, y þ Dy) þ FXY (x,y) FXY (x þ D x,y) þ FXY (x,y þ Dy) lim fXY (x,y) ¼ D x!0 Dy!0 D xDy
(9:2:4)
Similarly, the distribution function FXY (x,y) can also be given in terms of the density function by ðx ðy FXY (x,y) ¼ fXY (j,h)dhdj (9:2:5) 1 1
9.2
Joint Continuous Distribution Functions
137
The marginal density functions fX(x) and fY( y) are obtained by integrating the joint density function FXY (x,y) over all y and x, respectively, giving ð1 fX (x) ¼
ð1 fXY (x,y)dy;
fY ( y) ¼
1
fXY (x,y)dx
(9:2:6)
1
Properties of Joint Densities and Distributions 1. FXY (1, 1) ¼ 0; FXY (1,1) ¼ 1 2. FXY (1,y) ¼ 0; FXY (x,1) ¼ 0 3. FXY (x,1) ¼ FX (x); FXY (1,y) ¼ FY (y) 4. P{x1 , X x2 , Y y} ¼ FXY (x2 ,y) FXY (x1 ,y) P{X x, y1 , Y y2 } ¼ FXY (x,y2 ) FXY (x,y1 ) 5. P{x1 , X x2 , y1 , Y y2 } ¼ FXY (x2 ,y2 ) FXY (x2 ,y1 ) FXY (x1 ,y2 ) þ FXY (x1 ,y1 ) 6. @FXY (x,y) ¼ @x @FXY (x,y) ¼ @y
(9:2:7)
(9:2:8)
ðy fXY (x,h)dh 1 ðx
(9:2:9) fXY (j,y)dj
1
In many problems we are given fXY (x,y) in the region {(x1,x2], ( y1,y2]} and the distribution function FXY (x,y) is desired in the entire XY plane. In these cases, we can visualize the XY plane and delineate the various regions as shown in Fig. 9.2.1 for which FXY (x,y) can be determined. Given FXY (x,y) in region I, we find that using the properties enunciated earlier: FXY (x,y) ¼ 0 in region V, FXY (x,y) ¼ FX (x) in region II, FXY (x,y) ¼ FY ( y) in region III, and FXY (x,y) ¼ 1 in region IV. We shall give three different examples to illustrate how the joint distribution function can be determined.
FIGURE 9.2.1
138
Joint Densities and Distributions
Example 9.2.1 (Regions Bounded by Constants) The joint pdf fXY (x,y) is given by K(8 x y) 1 , x 3, 1 , y 2 fXY (x,y) ¼ 0 otherwise The constant K can be found by integrating over the range, yielding ð3 ð2 1 K(8 x y)dy dx ¼ 9K ¼ 1 or K ¼ 9 1 1 We plot the regions in Fig. 9.2.2 as shown in Fig. 9.2.1.
Region V: x 1 or y 1 FXY (x,y) ¼ 0 Region IV: 2x . 3, y . 2 FXY (x,y) ¼ 1 Region I: 1 , x 3, 1 , y 2 y ðx 1 1 h2 (8 j h)dh dj ¼ 8h jh 2 1 1 19 19 ðx 1 y2 15 8y jy þ j ¼ dj 2 2 19 x 1 j2 y jy2 j2 15j 8yj þ ¼ 9 2 2 1 2 2 1 j2 y jy2 j2 15j 8yj þ ¼ 9 2 2 2 2 2 2 2 2 1 x y xy x y 15x 15y 8xy þ7 þ þ ¼ 9 2 2 2 2 2 2 ðx ðy
FIGURE 9.2.2
9.2
Joint Continuous Distribution Functions
139
Region II: 1 , x 3, y . 2 [marginal distribution FX(x)] 1 x2 13x 6 FXY (x,2) ¼ FX (x) ¼ þ 9 2 2 Region III: x . 3, 1 , y 2 [marginal distribution FY ( y)] FXY (3,y) ¼ FY ( y) ¼
1 2 y þ 12y 11 9
By differentiating the distribution function, we can write the density function for the various regions as follows: 8 0 > > > >1 > > > > 9 (8 x y) > > <1 (12 2y) fXY (x,y) ¼ 9 > > > > 1 13 > > x > > > 9 2 > : 0
x1
or y 1
1 , x 3, 1 y 2 x . 3, 1 y 2 1 , x 3, y . 2 x . 3, y . 2
We can also conclude that since fXY (x,y) = fX (x) fY ( y), the random variables X and Y are not independent. Having determined the joint probability FXY (x,y), we can find the probability for other regions such as f1.5 , X 2.5, 1 , Y 1.5g as shown in Fig. 9.2.3. ð 2:5 ð 1:5 P{1:5 , X 2:5, 1 , Y 1:5} ¼ 1:5
1
1 (8 x y)dy dx 9
¼ FXY (2:5, 1:5) FXY (1:5, 1:5) FXY (2:5, 1) þ FXY (1:5, 1) ¼ 0:4167 0:1528 0 0 ¼ 0:2639
FIGURE 9.2.3
140
Joint Densities and Distributions
We can also find the conditional probability of Y y given that X x as follows FXY (x,y) FX (x) 1 x2 y xy2 x2 y2 15x 15y 8xy þ7 þ þ 9 2 2 2 2 2 2 2 ¼ 1 x 13x 6 þ 9 2 2
FY j X (y j X x) ¼
¼
y2 þ xy x 15y þ 14 x 12
At the endpoints x ¼ 3 and y ¼ 2, we have FY j X ( y j X 3) ¼ FY ( y) and FY j X (2 j X 3) ¼ FY ( y) ¼ 1. The corresponding conditional density function fY j X( y j X x) is given by fY j X (y j X x) ¼
x þ 2y 15 , x 12
1,x3 1,y2
We will now find the conditional density fY j X ( y j X ¼ x) from Eq. (8.2.5) 1 fXY (x,y) 9 (8 x y) 2(8 x y) ¼ ¼ fY j X (y j X ¼ x) ¼ 1 13 fX (x) 13 2x x 9 2 in the region 1 , x 3, 1 , y 2. The corresponding distribution function FY j X ( y j X ¼ x) in the region 1 , x 3, 1 , y 2 is obtained by integrating the density function and is given by ðy 2(8 x h) dh FY j X (y j X ¼ x) ¼ 13 2x 1 ¼
y2 þ 2xy 2x 16y þ 15 2x 13
We see clearly from this example that FY j X ( y j X x) is different from FY j X ( y j X ¼ x) and the corresponding conditional density functions fY j X ( y j X x) and fY j X ( y j X ¼x) are also different. Note that fY j X ( y j X ¼ x) has been defined as fY j X ( y j x) in Chapter 8. Example 9.2.2 (One Region Bounded by a Function) In this example one of the bounds on the region is a functional inequality. The joint pdf fXY (x,y) of two random variables X and Y is given by (8 (1 þ 3xy) 0 , x y 1 fXY (x,y) ¼ 7 0 otherwise We have to find the joint distribution function FXY (x,y) in the entire xy plane. Integrating first along the h axis and then along the j axis (Fig. 9.2.4), we have ðx ðy 8 FXY (x,y) ¼ (1 þ 3jh)dh dj 0 j7 ¼
1 3x4 þ 6x2 y2 4x2 þ 8xy , 7
0,xy1
9.2
Joint Continuous Distribution Functions
141
FIGURE 9.2.4
Integrating along the j axis first and then along the h axis (Fig. 9.2.4), we have ðx ðh 8 (1 þ 3jh)dj dh FXY (x,y) ¼ 0 0 7 I |fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl ffl} ðy ðx 8 (1 þ 3jh)dj dh þ x 07 II |fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl} 1 1 ¼ ð3x4 þ 4x2 Þ þ ð6x4 þ 6x2 y2 8x2 þ 8xyÞ 7 7 1 ¼ ð3x4 þ 6x2 y2 4x2 þ 8xyÞ; 0 , x y 1 7 Example 9.2.3 (Both Regions Bounded by Functions) The joint probability density function of two random variables X and Y is given by 8 2 > 0,x1 > > < 6 k(1 þ 2xy) 4 1 fXY (x,y) ¼ 0 , x2 , y x þ 1 > > 2 > : 0 otherwise The two boundaries are shown in Fig. 9.2.5.
FIGURE 9.2.5
142
Joint Densities and Distributions
The constant k is found by integrating fXY (x,y) over the ranges of definition as shown below: ð 1=2 ð xþ1=2 ð1 ð1 k(1 þ 2xy)dy dx þ k(1 þ 2xy)dy dx ¼ 1 0
x2
1=2 x2
Solving the above equation we obtain, k
161 ¼1 192
or k ¼
192 161
Method 1: Marginal Distributions by Integrating fXY (x,y). We will now find the marginal density fX (x) by integrating fXY (x,y) over all y. In the region 0 , x 12, we have ð xþ1=2 192 (1 þ 2xy)dy fX (x) ¼ 161 x2 48 1 ¼ 2 þ 5x þ 4x3 4x 5 , 0 , x 161 2 In the region 12 , x 1, we have ð1
192 (1 þ 2xy)dy x2 161 192 ¼ 1 þ x x2 x5 , 161
fX (x) ¼
1 ,x1 2
By integrating fX (x) in these two regions, we can write the distribution function FX(x) as 8 8 1 > > (12x þ 15x2 þ 16x4 4x6 ) 0,x > > 161 2 > < 31 32 1 2 3 6 FX (x) ¼ þ (6x þ 3x 2x x ) ,x1 > 161 161 2 > > > x.1 > :1 0 otherwise Similarly, we integrate fXY (x,y) over all x to obtain the marginal density fY ( y) as shown below: 8 ð pyffiffi 192 1 > > > (1 þ 2xy)dx, 0,y < 161 2 fY ( y) ¼ ð 0pyffiffi > 192 1 > > (1 þ 2xy)dx, ,y1 : 161 2 y1=2 or 8 192 pffiffiffi > > y y2 > < 161 pffiffiffi fY ( y) ¼ 16 6 þ 12 y 15y þ 24y2 12y3 > > > 161 : 0
0,y
1 2
1 ,y1 2 otherwise
9.2
Joint Continuous Distribution Functions
143
Integrating fY ( y) in the above regions we obtain the distribution function FY ( y) as shown below: 8 64 3=2 > > 2y þ y3 > > 161 > < 1 16 FY ( y) ¼ þ (6y þ 8y3=2 15y2 þ 8y3 3y4 ) > 7 161 > > > > :1 0
0,y
1 2
1 ,y1 2 y.1 otherwise
fX(x), FX (x), fY ( y) and FY ( y) are shown in Fig. 9.2.6. Method 2: Marginal Distributions from FXY (x,y). We will now find the marginal distributions from the joint distribution FXY (x,y) obtained by integrating the joint density fXY (x,y) over the regions of definition. The joint distribution FXY (x,y) in the region 0 , x 12, 0 , y 12 is obtained from the following integration: ðx ðy FXY (x,y) ¼ 0
x2
192 (1 þ 2jh)dh dj 161
32 ¼ 6xy 2x3 þ 3x2 y2 x6 , 161
0,x1 0 , y 1=2
and in the region 0 , x 1, 12 , y 1, we have ð y(1=2) ð jþ(1=2) FXY (x,y) ¼ 0
ðx
j2
ðy
þ y(1=2) j
2
192 (1 þ 2jh)dh dj 161
192 (1 þ 2jh)dh dj 161
or FXY (x,y) ¼
1 8 þ 12y 15y2 þ 8y3 6y4 7 161 2 2
3
þ 24xy þ 12x y 8x 4x
FIGURE 9.2.6
6
(
0,x,1 1 ,y1 2
144
Joint Densities and Distributions
The marginal distributions FX (x) and FY ( y) can be obtained by substituting suitable boundary values in the joint distribution FXY (x,y) as shown below:
FX (x) ¼
8 1 > > F x,x þ , > XY > > 2 > > > > > < FXY (x,1), > > > FXY (1,1), > > > > > > > : FXY (0,0),
1 1 0,x , y. 2 2 1 1 , x 1, y . 2 2 1 x . 1, y . 2 1 x 0, y . 2
Similarly 8 pffiffiffi > > FXY ( y,y), > > > < pffiffiffi FY (y) ¼ FXY ( y,y), > > > > > : FXY (1,1), FXY (0,0),
1 0,y , 0,x1 2 1 , x 1, 0 , x 1 2 y . 1, x . 1 y 0, x . 1
and these are exactly the same as derived previously.
B 9.3
BIVARIATE GAUSSIAN DISTRIBUTIONS
If X and Y are jointly distributed random variables with mean value mX and mY and variances s2X and s2Y, then the joint density function fXY (x,y) is given by ( " 1 1 x mX 2 pffiffiffiffiffiffiffiffiffiffiffiffiffi exp fXY (x,y) ¼ 2(1 r2 ) sX 2psX sY 1 r2 #) 2r(x mX )(y mY ) y mY 2 þ (9:3:1) sX sY sY where r is the correlation coefficient that determines the degree of similarity between X and Y, given by sXY r¼ (9:3:2) sX sY We have to show that ðð 1 I¼
FXY (x,y)dy dx ¼ 1
(9:3:3)
1
Substituting j ¼ (x 2 mX)/sx and h ¼ ( y 2 mY)/sy in Eq. (9.3.1), we can write the integral I as follows: 1 1 2 2 pffiffiffiffiffiffiffiffiffiffiffiffiffi exp ½j 2rjh þ h dh dj 2(1 r2 ) 1 2p 1 r2
ðð 1 I¼
(9:3:4)
145
9.3 Bivariate Gaussian Distributions
Completing the squares in j in the integral (9.3.4) above, we have
ðð 1 1 1 (j rh)2 pffiffiffiffiffiffiffiffiffiffiffiffiffi exp I¼ þ h dh dj 2 1 r2 1 2p 1 r2
(9:3:5)
Substituting j rh w ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi 1 r2 and dj dw ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi 1 r2 the integral (9.3.5) can be written as follows: 2 2 ð1 ð 1 w 1 1 h pffiffiffiffiffiffi exp exp I¼ dw dh ¼ 1 2p 1 2 2 2p 1
(9:3:6)
If r ¼ 0 in Eq. (9.3.3), then X and Y are independent because fXY (x,y) ¼ fX (x) fY ( y). Thus, if Gaussian-distributed random variables X and Y are uncorrelated, they are also independent. We shall find the conditional pdf fY j X (X ¼ x) given X ¼ x. From Eq. (8.2.5) we have fXY (x,y) fX (x) 8 9 1 1 x mX > > > > pffiffiffiffiffiffiffiffiffiffiffiffiffi exp > > > > 2 > > 2(1 r2 ) sX 1 r 2ps s > > X Y > > > >
> > 2 > > > > (2r(x m )( y m ) y m X Y Y > > > > þ = < sX sY sY ¼ 2 > > 1 1 x mX ) > > > > pffiffiffiffiffiffi exp > > > > > > 2 s 2p s X > > X > > > > > > > > > > > > ; :
fY j X (y j X ¼ x) ¼
( " 2 ) 1 1 rsY ¼ pffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffi exp (x mX Þ y mY sX 2s2Y (1 r2 ) 2psY 1 r2 Thus the conditional density has a conditional mean value of rsY mY j X ¼ mY þ (x mX ) sX and a conditional variance of s2Y j X ¼ s2Y (1 r2 ):
CHAPTER
10
Moments and Conditional Moments
B 10.1
EXPECTATIONS
The expected value or the mean value of any random variable X is defined by ð1 E ½X ¼ mX ¼ x fX (x) dx (10:1:1) 1
This definition is valid regardless of whether the random variable is continuous or discrete. Since the pdf of a discrete random variable is always given by a sum of delta functions, we have ) ð 1 (X 1 E½X ¼ mX ¼ x P{X ¼ xi }d(x xi ) dx 1
¼
1 X
i¼0
xi P{X ¼ xi }
(10:1:2)
i¼0
The expected value is the first moment of the random variable X about the origin. Example 10.1.1
For a random variable uniformly distributed in the interval (a,b], we have fX (x) ¼
1 ba
and hence the mean value is given by
E½X ¼
1 ba
ðb x dx ¼ a
b 1 x2 1 b2 a2 b þ a ¼ ¼ b a 2 a 2 b a 2
Example 10.1.2 On a toss of a die, a person wins $10 if {1} or {3} results, loses $5 if {4} or {6} results, wins $5 if {2} results, and loses $10 if {5} results. From Eq. (10.1.2), Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
146
10.1 Expectations
147
the expected value of the winnings is 6 X
E{X} ¼ mX ¼
xi P{X ¼ xi }
i¼1
1 5 ¼ {10 þ 10 5 5 þ 5 10} ¼ 6 6 The expected value of any random variable is a constant. In other words, it smooths the variations of a random variable to a constant. It is a one-way operator, in the sense that once the random variable X is smoothed, there is no way of getting back the density function. Thus, the expected value of a constant a is the same constant a.
Properties of Expectations 1. It is a linear operator irrespective of the nature of random variables, in the sense that the expected value of the sum equals the sum of expected values E½X þ Y ¼ E½X þ E½Y
(10:1:3)
as shown below: ð1 ð1 E½X þ Y ¼
(x þ y)fXY (xy)dx dy 1 1 ð1 ð1
ð1 ð1 xfXY (xy)dx dy þ
¼ 1 1 ð1
ð1
xfX (x)dx þ
¼ 1
yfXY (xy)dx dy 1 1
yfY (y)dy ¼ E{X} þ E{Y} 1
2. On the contrary, the expected value of the product given by ð1 ð1 E½XY ¼ xyfXY (xy)dx dy
(10:1:4)
1 1
is not, in general, equal to the product of expected values. However, Equation (10.1.4) can be split only if X and Y are independent, in which case we have ð1 ð1 E½XY ¼ xy fXY (xy)dx dy 1 1
ð1
ð1
xfX (x)dx
¼ 1
yfY (y)dy 1
¼ E½X E½Y and the expected value of the product is equal to the product of expected values. The converse is not true. If E[XY ] ¼ E[X ] E[Y ], then X and Y are uncorrelated and need not be independent. Analogously, if E[XY ] ¼ 0, then X and Y are orthogonal. We can summarize the following definitions: Uncorrelated: E[XY ] ¼ E[X ] E[Y ] Orthogonal: E[XY ] ¼ 0 Independence: fXY (xy) ¼ fX (x) fY (y)
148
Moments and Conditional Moments
Conditional Expectation
Given that the random variable X has occurred, the expected value of Y is defined by ð1 E½Y j X ¼ yfYjX ( y j x)dy (10:1:5) 1
The conditional expectation operator does not completely smooth the random variable Y to a constant but smooths it to random variable dependent on X. Thus, if we take the expectation of the conditional expectation, we get the expected value of Y: ð 1 ð 1 E{E½Y j X} ¼ y fY j X ( y j x) dy fX (x)dx 1
1
ð1
y fXY (xy) dy dx
¼ 1 ð1
y fY ( y)dy ¼ E½Y
¼
(10:1:6)
1
If X and Y are independent, then E[Y j X ] ¼ E[Y ] since fY j X ( y j x) ¼ fY ( y).
Example 10.1.3 A random variable Y is uniformly distributed in the interval (0,5}. Hence the distribution function FY( y) is given by y/5 for 0 , y 5. We will find the conditional expectation of Y given that Y . x and 0 , x 5. We first find the conditional distribution FY ( y j Y . x): FY ( y j Y . x) ¼
P(x , Y y) FY ( y) FY (x) ¼ P(Y . x) 1 FY (x)
Differentiating this equation with respect to y, the conditional density fY( y j Y . x) is obtained as follows: fY (y) ¼
fY ( y) 1 ¼ 1 FY (x) 5 x
Hence the conditional expectation E[Y j Y . x] is given by 5 y y2 E½Y j Y . x ¼ dy ¼ 2(5 x)x x5 x ð5
¼
25 x2 5þx ¼ 2 2(5 x)
Joint Moment
We can now define the joint moment between two random variables X and Y as follows: ð1 ð1 E½XY ¼ mXY ¼ xy fXY (x,y) dx dy (10:1:7) 1
1
The joint moments determine the correlation between X and Y.
10.2
B 10.2
Variance
149
VARIANCE
The variance is the second moment about the mean value also known as the second central moment. It is defined by ð1 var½X ¼ E½X mX 2 ¼ s2X ¼ (x mX )2 fX (x) dx 1 2
¼ E½X 2mX E½X þ
m2X
¼ E½X 2 m2X
(10:2:1)
The variance expresses the variability of the random variable about the mean value. If this variability is zero, then the random variable will be no longer random but a constant. This will be discussed further in Chapter 14. The preceding definition applies for any random variable. However, for a discrete random variable, analogous to Eq. (10.1.2), we have E½X mX 2 ¼ sX2 ¼
1 X
(xi mX )2 P{X ¼ xi )
(10:2:2)
i¼0
Example 10.2.1 For a random variable, X uniformly distributed between (a,a, the mean value mX ¼ 0. Hence the variance is given by ð 1 a 2 a2 2 2 sX ¼ E½X ¼ x dx ¼ 2a a 3 Example 10.2.2 From Eq. (10.2.2), the variance of the random variable given in Example 10.1.2 is 6 X 5 21 s2X ¼ xi 6 6 i¼1 2 1 55 552 252 252 252 552 10,950 ¼ þ þ þ þ þ ¼ 6 62 216 62 62 62 62 62 Properties of Variance 1. 2.
var½aX ¼ a2 var½X ¼ a2 s2X
(10:2:3)
var½X + Y ¼ E½X + Y2 {E½X þ E½Y}2 ¼ E½X 2 {E½X}2 þ E½Y 2 {E½Y}2 + 2{E½XY E½XE½Y} ¼ sX2 þ s2Y + 2{E½XY E½XE½Y}
(10:2:4)
3. If X and Y are independent, then var½X + Y ¼ s2X þ s2Y
(10:2:5)
Conditional Variance. We can also define a conditional variance of Y given X has occurred as var½Y j X ¼ E½(Y mYjX )2 j X
(10:2:6)
150
Moments and Conditional Moments
Covariance. The covariance between two random variables X and Y is defined by cov½XY ¼ sXY ¼ E ½X mX E ½Y mY ¼ E ½XY mX mY
(10:2:7)
If X and Y are independent, then cov[XY ] ¼ 0. The converse is not true. If cov[XY ] ¼ 0, then X and Y are uncorrelated but not necessarily independent. Correlation Coefficient. The correlation coefficient is the normalized covariance and is defined by cov½XY sXY rXY ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ var½X var½Y sX sY
(10:2:8)
The correlation coefficient satisfies 1 rXY 1 and shows the degree of selfsimilarity between X and Y.
B 10.3 MEANS AND VARIANCES OF SOME DISTRIBUTIONS Discrete Random Variables Binomial Distribution
Mean: n k nk b(k; n, p) ¼ pq k E½X ¼
n X k¼0
¼
k
(10:3:1)
n! pk qnk k!(n k)!
n X
n(n 1)! p pk1 qnk (k 1)!(n k)! k¼0
¼ np
n X
(n 1)! pk1 qnk (k 1)!(n k)! k¼1
¼ np( p þ q)n1 ¼ np
(10:3:2)
Variance: var½X ¼ E½X 2 m2X E½X 2 ¼
n X k¼0
k2
n! pk qnk k!(n k)!
(10:3:3) (10:3:4)
Substituting k 2 ¼ k (k 2 1) þ k in Eq. (10.3.4), we have E½X 2 ¼
n X k¼0
k(k 1)
n X n! n! pk qnk þ pk qnk k k!(n k)! k!(n k)! k¼0
(10:3:5)
10.3
Means and Variances of Some Distributions
151
The second summation in Eq. (10.3.5) is the mean value, already calculated as np in Eq. (10.3.2). The first summation can be written as follows: n X
k(k 1)
k¼0
n X n! (n 2)! pk qnk ¼ n(n 1) p2 pk2 qnk k!(n k)! (k 2)!(n k)! k¼2
¼ n(n 1) p2 (p þ q)n2 ¼ n(n 1)p2
(10:3:6)
Thus E½X 2 ¼ n (n 1) p2 þ np
(10:3:7)
and the variance from Eq. (10.3.3) is given by var½X ¼ n (n 1) p2 þ np n2 p2 ¼ npq
(10:3:8)
Poisson Distribution
Mean: P(k; l) ¼ el E½X ¼ el
lk k! 1 X klk k¼0
k!
(10:3:9) ¼ lel
1 X lk 1 ¼ l el el ¼ l (k 1)! k¼1
(10:3:10)
Variance. We use the same technique as in the binomial distribution by substituting k 2 ¼ k(k 2 1) þ k. Hence E½X 2 ¼ el
1 X k(k 1)lk k¼0
¼ l2 el
k!
þ el
1 X klk k¼0
k!
1 X lk 2 þl (k 2)! k¼2
¼ l2 el el þ l ¼ l2 þ l
(10:3:11)
Substituting in var½X ¼ E½X 2 m2X from Eqs. (10.3.11) and (10.3.10), we have var ½X ¼ l2 þ l l ¼ l
(10:3:12)
For a Poisson distribution, the mean and the variance are the same and equal to l. Continuous Random Variables Exponential Distribution
Mean: fX (x) ¼ l elx u(x) ð1 E½X ¼ l xelx dx 0
(10:3:13) (10:3:14)
152
Moments and Conditional Moments
Integrating Eq. (10.3.14) by parts, we obtain ð1 ð1 delx ¼ E½X ¼ l x x delx l 0 0 1 ð 1 1 lx ¼ xe þ delx ¼ l 0 0 Variance. We first calculate the second moment: ð1 E½X 2 ¼ l x2 elx dx
(10:3:15)
(10:3:16)
0
Integrating by parts [Eq. (10.3.16)] twice, we get the result as ð1 l
x2 elx dx ¼
0
lx 1 e 2 2 ¼ 2 (l x þ 2lx þ 2) 2 l2 l 0
(10:3:17)
2 1 1 ¼ l2 l2 l2
(10:3:18)
and the variance is given by var½X ¼
Gaussian Distribution
Mean: 1 1 (x mX )2 pffiffiffiffiffiffi exp 2 s2X sX 2p ð1 1 1 (x mX )2 x exp dx E½X ¼ pffiffiffiffiffiffi 2 s2X sX 2p 1 ð1 1 1 (x mX )2 ¼ pffiffiffiffiffiffi (x mX ) exp dx 2 s2X sX 2p 1 ð1 m 1 (x mX )2 exp dx þ pXffiffiffiffiffiffi 2 s2X sX 2p 1
fX (x) ¼
(10:3:19)
(10:3:20)
In Eq. (10.3.20), the first integral is zero because it is an odd function and the second integral equals mX. Hence E½X ¼ mX
(10:3:21)
Variance: 1 var½X ¼ pffiffiffiffiffiffi sX 2p
1 (x mX )2 (x mX ) exp dx 2 s2X 1
ð1
2
Substituting y ¼ (x mX )=sX in Eq. (10.3.22), we have ð 1 1 2 2 (1=2)y2 sX y e dy ¼ s2X var½X ¼ pffiffiffiffiffiffi 2p 1 pffiffiffiffiffiffi Ð 1 2 where it can be shown that 1= 2p 1 y2 e(1=2)y dy ¼ 1
(10:3:22)
(10:3:23)
10.4
B 10.4
Higher-Order Moments
153
HIGHER-ORDER MOMENTS
We can now define higher-order moments. The kth-order moment is defined by E½X k ¼ mk ¼
ð1
xk fX (x) dx
(10:4:1)
1
Clearly, m0 ¼ 1 and m1 ¼ mX. Similarly, the kth-order central moments are defined by E½X
mkX
ð1
(x mkX ) fX (x) dx
¼ vk ¼
(10:4:2)
1
Here v0 ¼ 1, v1¼0, and v2 ¼ s2X. We can find relationships between the kth-order moments and the kth-order central moments. We can write these relationships as follows: v0 ¼ 1 v1 ¼ 0 v 2 ¼ s2 v3 ¼ m3 3m1 m2 þ 2m31
m0 ¼ 1 m1 ¼ m m2 ¼ v2 þ m2 m3 ¼ v3 þ 3m1 v2 þ m31
(10:4:3)
The moments of random variables are not arbitrary numbers but must satisfy certain inequalities. For example, since E½X mX 2 must be nonnegative, we have s2 ¼ m2 m21 0
and
m2k m2k 0
for all k
(10:4:4)
A random variable is completely determined if the density function is known. However, knowledge of only two moments gives only a limited information of a random variable. But under certain conditions, knowledge of moments mk for all k will uniquely determine the random variable. However, because only two of the moments of a Gaussian random variable are independent, it can be completely determined with two moments. All other moments are dependent on two moments. We will show that for a Gaussian random variable with density function 1 2 2 fX (x) ¼ pffiffiffiffiffiffi ex =2s s 2p satisfies the following moment equations: 0 k E½X ¼ 1 3 5 (n 1)sn
n ¼ 2k þ 1 for all k n ¼ 2k for all k
(10:4:5)
(10:4:6)
function. To show the result for Clearly the odd moments are 0 because fXÐ(x) is an even p ffiffiffiffiffiffiffiffi 2 1 the even moments, we take the equation 1 ebx dx ¼ p=b and differentiate it k time with respect to b, yielding rffiffiffiffiffiffiffiffiffiffiffi ð1 1 3 5 (2k 1) p 2k bx2 x e dx ¼ (10:4:7) k 2kþ1 2 b 1 and the result is obtained by substituting b ¼ 1/2s2 in Eq. (10.4.7). Thus, the Gaussian random variable is completely determined if only two moments are known.
154
Moments and Conditional Moments
Higher-Order Cross-Moments
The concept of joint moments can be extended to higher-order cases. The k, jth higher-order cross-moment is defined by, ð1 ð1 k j E½X , Y ¼ mkj ¼ xk y j fXY (x,y) dx dy (10:4:8) 1
1
and the k, jth higher order central moment is defined by ð1 ð1 k j E½(X mX ) , (Y mY ) ¼ hkj ¼ (x mX )k ( y mY ) j fXY (x,y) dx dy 1
(10:4:9)
1
and m11 ¼ E [XY ] the joint moment and h11 ¼ cov[XY ] ¼ sXY is the covariance.
B 10.5
BIVARIATE GAUSSIAN
Two random variables are jointly Gaussian-distributed if they satisfy the following density function with r as the correlation coefficient: ( " 1 1 x mX 2 pffiffiffiffiffiffiffiffiffiffiffiffiffi exp fXY (x,y) ¼ 2(1 r2 ) sX 2psX sY 1 r2 #) (x mX )(y my ) y mY 2 þ (10:5:1) 2r sY s X sY Ordinarily, uncorrelatedness does not imply independence, even though independence implies uncorrelatedness. However, in the case of Gaussian-distributed random variables if r ¼ 0, then Eq. (10.5.1) becomes ( " #) 1 1 x mX 2 y mY 2 fXY (x,y) ¼ exp þ (10:5:2) 2psX sY 2 sX sY and clearly fXY (x,y) ¼ fX (x) fY (y) ( ( ) ) 1 1 x mX 2 1 1 x mY 2 pffiffiffiffiffiffi exp ¼ pffiffiffiffiffiffi exp 2 2 sX sY 2psX 2psY and hence X and Y are independent.
(10:5:3)
CHAPTER
11
Characteristic Functions and Generating Functions
B 11.1
CHARACTERISTIC FUNCTIONS
The characteristic function (CF) of any random variable X (discrete or continuous) is defined by ð1 FX (v) ¼ E{e jvX} ¼ e jvx fX (x) dx (11:1:1) 1
From this definition it is seen that the characteristic function is the Fourier transform of the pdf fX (x) where the sign of v is positive instead of negative. However, all the properties of the Fourier transform hold good. From the definition of FX (v), we can see that FX (0) ¼ 1
(11:1:2)
This equation is used to check the correctness of the CF. Existence of CF
The CF is bounded as shown below. Applying the fact that the absolute value of an integral is less than or equal to the integral of its absolute value, we have ð1 ð 1 jvx e fX (x) dx je jvx jj fX (x)j dx 1
and since je
jvx
1
j ¼ 1 we have ð1 ð 1 jvx e fX (x) dx je jvx jj fX (x)j dx 1 1 ð1 ¼ j fX (x)jdx ¼ 1 1
Thus, the CF always exists.
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
155
156
Characteristic Functions and Generating Functions
We can also obtain the pdf fX(x) from the CF from the inversion formula for the Fourier transform given by ð 1 1 FX (v)ejvx dv (11:1:3) fX (x) ¼ 2p 1 [Comment: Since FX(v) is the Fourier transformation of the pdf fX(x), the random variable X is also completely determined if FX(v) is known. A Fourier transform of a signal characterizes the frequency content of the signal. No such characterization for the CF can be given; it is just a convenient form of representing the pdf and has other very useful properties.] Example 11.1.1
We will find the CF of an exponential distribution given by fX (x) ¼ lelx u(x)
Using Eq. (11.1.1), we have ð1 FX (v) ¼
lelx e jvx dx ¼
0
l l jv
Moment Generating Properties of CF
In Chapter 10 we found the moments of a random variable using integration techniques. Once the CF is known, the moments can be obtained by differentiation, which is a simpler process than integration. Expanding the CF as a Taylor series about the origin, we can write ð1 jvx ( jvx)2 (jvx)3 ( jvx)k þ FX (v) ¼ þ þ þ þ dv (11:1:4) fX (x) 1 þ 1! 2! 3! k! 1 Substituting ð1 mk ¼
xk fX (x) dx,
k ¼ 0,1; . . .
(11:1:5)
1
in Eq. (11.1.2), we have jvm1 ( jv)2 m2 ( jv)3 m3 ( jv)k mk FX (v) ¼ 1 þ þ þ þ þ þ 1! 2! 3! k! We now differentiate FX(v) term by term to yield d jm1 2( j)2 vm2 3( j)3 v2 m3 k( j)k vk1 mk FX (v) ¼ þ þ þ þ þ dv 1! 2! 3! k!
(11:1:6)
Substituting v ¼ 0 in Eq. (11.1.6), we get F0X (v)v¼0 ¼ jm1
and m1 ¼
F0X ð0Þ j
(11:1:7)
Differentiating FX(v) successively k times and substituting in the resulting equations v ¼ 0 we can obtain all the moments of the random variables as shown below: (k) 2 k F(2) (11:1:8) X (v) v¼0 ¼ ( j) m2 , . . . , FX (v) v¼0 ¼ ( j) mk , . . . , We will give more examples in a later section.
11.2
Examples of Characteristic Functions
157
Joint Characteristic Functions
We can also define a joint characteristic function for two random variables X and Y as FXY (v1 , v2 ) ¼ E e j(v1 Xþv2 Y) ð1 ð1 ¼ fXY (x, y)e j(v1 xþv2 y) dx dy
(11:1:9)
1 1
B 11.2
EXAMPLES OF CHARACTERISTIC FUNCTIONS
We will now give several examples of characteristic functions for both discrete and continuous random variables and determining their means and variances. Discrete Random Variables
Example 11.2.1 (Binomial Distribution) n k nk pq b(k: n,p) ¼ k
(11:2:1)
The CF is given by n X n k nk jvk pq e k¼0 k n X n ¼ pe jvk qnk ¼ ( pe jv þ q)n k k¼0
FX (v) ¼
(11:2:2)
We find that FX(0) ¼ ( p þ q) ¼ 1 Mean: F0X (v) ¼ n( pe jv þ q)n1 jpe jv F0X (0) ¼ jnp
or E{X ¼ k} ¼ m1 ¼ np
(11:2:3)
Variance. We first find EfX 2 ¼ k 2g ¼ m2: F00X (v) ¼ n(n 1)( pe jv þ q)n2 ( jpe jv )2 þ n( pe jv þ q)n1 j2 pe jv F00X (0) ¼ n(n 1)p2 np and
m2 ¼ (np)2 þ npq
Hence var½X ¼ m2 m21 ¼ npq
(11:2:4)
158
Characteristic Functions and Generating Functions
Example 11.2.2 (Poisson Distribution) p(k; l) ¼ el FX (v) ¼
1 X
lk k! el
lk jvk e k!
el
le jv jv jv ¼ el ele ¼ el(1e Þ k!
k¼0
¼
1 X
(11:2:5)
k
k¼0
(11:2:6)
and FX(0) ¼ 1. Mean: jv
F0X (v) ¼ el(1e ) l(e jv )j
(11:2:7)
F0X (0) ¼ lj or E{X ¼ k} ¼ m1 ¼ l Variance. We first find EfX 2 ¼ k2g ¼ m2: jv
jv
F00X (v) ¼ el(1e ) l(e jv )j l(e jv )j el(1e ) lj(e jv )j F00X (0) ¼ l2 l or
m2 ¼ l2 þ l
Hence var{X} ¼ m2 m21 ¼ l
(11:2:8)
Example 11.2.3 (Geometric Distribution) p{k : p} ¼ P{k failures followed by a success} ¼ qk p FX (v) ¼
1 X
pqk e jvk ¼ p
k¼0
1 X
(11:2:9)
(qe jv )k
k¼0
¼
p 1 qe jv
(11:2:10)
and FX(0) ¼ 1. Mean: F0X (v) ¼
p jpqe jv jv (jqe ) ¼ (1 qe jv )2 (1 qe jv )2
F0X (0) ¼
jpq p2
or E{x ¼ k} ¼ m1 ¼
q p
(11:2:11)
Variance. We first find EfX 2 ¼ k 2g ¼ m2. F00X (v) ¼ F00X (0)
2jpqe jv j2 pqe jv jv (jqe ) þ (1 qe jv )3 (1 qe jv )2
2pq2 pq ¼ 2 p p3
2q2 q or m2 ¼ 2 þ p p
(11:2:12)
11.2
Examples of Characteristic Functions
159
Hence variance is m2 m2X ¼
2q2 q q2 q(p þ q) q þ 2¼ ¼ 2 p p p2 p p2
(11:2:13)
Continuous Random Variables
Example 11.2.4 (Uniform Distribution) 1 ½u(x b) u(x a) ba ð b jvx e e jvb e jva FX (v) ¼ dx ¼ jv(b a) aba fX (x) ¼
(11:2:14) (11:2:15)
This is one of the more difficult CFs to obtain the mean and the variance by differentiating Eq. (11.2.15) because of the limiting processes involved. Instead, we expand Eq. (11.2.15) using Taylor series and differentiate the resulting expression. Expansion of Eq. (11.2.15) results in 1 ( jv)2 2 ( jv)3 3 FX (v) ¼ jv(b a) þ (b a2 ) þ (b a3 ) þ jv(b a) 2! 3! jv ( jv)2 2 (b þ a) þ (b þ ab þ a2 ) þ ¼ 1þ (11:2:16) 2! 3! Mean: F0X (v)
j 2j2 v 2 2 (b þ a) þ (b þ ab þ a ) þ ¼ 2! 3!
F0X (0)
j(b þ a) ¼ 2
(11:2:17)
(b þ a) or m1 ¼ 2
Variance: F00X (v) ¼ F00X (0) ¼
2j2 2 (b þ ab þ a2 ) þ 3!
(b2 þ ab þ a2 ) 3
or
m2 ¼
b2 þ ab þ a2 3
Hence var½X ¼ m2 m21 ¼
(b a)2 12
(11:2:18)
In most cases it is easier to find the mean and variance using CF. However, in this case it is easier to obtain mean and variance by integration techniques using their definitions.
Example 11.2.5 (Exponential Distribution) fX (x) ¼ lelx u(x)
(11:2:19)
160
Characteristic Functions and Generating Functions
From Example 11.1.1 the CF is ð1 FX (v) ¼
lelx e jvx dx ¼
0
l l jv
(11:2:20)
Mean: F0X (v) ¼ F0X (0)
jl (l jv)2
j ¼ l
(11:2:21)
1 m1 ¼ l
or
Variance: F00X (v) ¼
2l (l jv)3
F00X (0) ¼
2 l2
or
m2 ¼
(11:2:22)
2 l2
Hence var½X ¼ m2 m21 ¼
1 l2
(11:2:23)
Example 11.2.6 (Gaussian Distribution) 2 1 2 fX (x) ¼ pffiffiffiffiffiffi e(xm) =2s 2p s ð1 2 1 2 FX (v) ¼ pffiffiffiffiffiffi e(xm) =2s þjvx dx 2p s 1
(11:2:24) (11:2:25)
Completing the squares in the exponent of Eq. (11.2.25), we obtain the resulting expression as follows: ð1 1 (x m jvs2 )2 jvm½(s2 v2 )=2 p ffiffiffiffiffiffi FX (v) ¼ e exp dx 2s2 2p s 1 The value of the integral within braces is 1 and hence the CF of a Gaussian is 2
FX (v) ¼ e jvm½(s v
2
)=2
(11:2:26)
Mean: 2
2
F0X (v) ¼ e jvm½(s v )=2 ( jm s2 v) F0X (0) ¼ jm or m1 ¼ m
(11:2:27)
Variance: 2
F00X (v) ¼ e jvm½ðs v
2
F00X (0) ¼ m2 s2
Þ=2
( jm s2 v)2 e jvm½(s
or m2 ¼ m2 þ s2
2
v2 )=2 2
s
11.3 Generating Functions
161
Hence var½X ¼ m2 m21 ¼ s2
B 11.3
(11:2:28)
GENERATING FUNCTIONS
Certain simplifications are possible in the definition of the characteristic functions for discrete random variables taking positive integral values. This is called the generating function (GF) and is defined for a discrete random variable X taking positive discrete values as PX (z) ¼ E{zk } ¼
1 X
zk P{X ¼ k}
(11:3:1)
k¼0
with PX(1) ¼ 1. The infinite series of Eq. (11.3.1) will converge for values of jzj , 1. Equation (11.3.1) can be recognized as the z transform of discrete data with the sign reversal for z. Example 11.3.1 Six dice are tossed. Using the generating function, we want to find the probability that the sum of the faces of the dice add to 24. The GF for the toss of a single die is 1 PX (z) ¼ ½z þ z2 þ z3 þ z4 þ z5 þ z6 6 and in closed form this can be written as z 1 z6 PX (z) ¼ 6 1z Since the tosses are independent, the GF for the toss of 6 dice is [PX(z)]6 given by QX (z) ¼ ½PX (z)6 ¼
z 6 1 z6 6 6
1z
To find Pffaces add up to 24g, we have to first find the coefficient of z 24 in the power series expansion of Q(z). Alternately, we have to find the coefficient of z 18 in the expansion of RX (z) ¼
(1 z6 )6 (1 z)6
The expansion of the numerator of RX(z) is (1 z)6 ¼ 1 6z6 þ 15z12 20z18 þ 15z24 6z30 þ z36 and the expansion of the denominator of RX(z) is given by the negative exponential (1 z)6 ¼
1 X (5 þ k 1)! k¼1
5!(k 1)!
zk1
162
Characteristic Functions and Generating Functions
We now multiply the two expansions above and find the coefficient z 18 in the resulting product. The terms containing z 18 are (5 þ 7 1) (5 þ 13 1)! (5 þ 19 1)! 6 þ1 z18 20 þ 15 5!6! 5!12! 5!18! ¼ 3451z18 Hence the probability that the sum of the faces add to 24 is P{sum ¼ 24} ¼
3451 ¼ 0:073967 66
a result that would have been very difficult to obtain without generating functions.
Moment Generating Properties of GF
It is easier to use the GF in finding the moments of a discrete random variable taking positive integral values than finding it through CF. We can differentiate Eq. (11.3.1) and obtain PX (z) ¼
1 X
zk P{X ¼ k}
k¼0
P0X (z) ¼
1 X
kzk1 P{X ¼ k}
(11:3:2)
k¼0
P0X (1) ¼
1 X
kP{X ¼ k} ¼ m1
k¼0
Differentiating PX0 (z) again, we obtain P00X (z) ¼
1 X
k(k 1)P{X ¼ k}
k¼0
¼
1 X
k2 P{X ¼ k}
k¼0
1 X
kP{X ¼ k}
(11:3:3)
k¼0
P00X ð1Þ ¼ m2 m1 Hence the variance is given by
2 var½X ¼ P00X (1) þ P0X (1) P0X (1)
(11:3:4)
Unlike the CF, there is no general formula for the kth moment in the case of GF. In particular P000 X (1) ¼ m3 3m2 þ 2m1
B 11.4
(11:3:5)
EXAMPLES OF GENERATING FUNCTIONS
We will now derive generating functions for all discrete random variables discussed in Section 11.2 and determine their means and variances.
11.4 Examples of Generating Functions
163
Example 11.4.1 (Binomial Distribution) b(k: n,p) ¼
n k nk pq k
(11:4:1)
The GF is given by N X n PX (z) ¼ pzk qnk ¼ (pz þ q)N k k¼0
(11:4:2)
Clearly PX(1) ¼ 1. Mean: P0X (z) ¼ N(pz þ q)N1 p P0X (1) ¼ Np
or m1 ¼ Np
(11:4:3)
Variance: P00X (z) ¼ N(N 1)(pz þ q)N2 p2 P00X (1) ¼ N(N 1)p
or m2 ¼ N(N 1)p2 Np
Hence var½X ¼ m2 m21 ¼ Npq
(11:4:4)
This result has been obtained with comparative ease. Example 11.4.2 (Poisson Distribution) p(k; l) ¼ el
lk k!
(11:4:5)
The GF is given by PX (z) ¼
1 X
el
k¼0
lzk ¼ el(1z) k!
(11:4:6)
Note that PX(1) ¼ 1. Mean: P0X (z) ¼ el(z1) ¼ el(z1) l P0X (1) ¼ l or m1 ¼ l
(11:4:7)
a result that has been trivially obtained compared to Eq. (11.2.7). Variance: P00X (z) ¼ el(z1) l2 P00X (1) ¼ l2
or
m2 ¼ l2 l
(11:4:8)
Hence var½X ¼ m2 m21 ¼ l A result that has been obtained much more easily than using Eq. (11.2.8).
(11:4:9)
164
Characteristic Functions and Generating Functions
Example 11.4.3 (Geometric Distribution) p{k: p} ¼ qk p
(11:4:10)
The GF is PX (z) ¼
1 X
qzk p ¼
k¼0
p 1 qz
(11:4:11)
Here also PX(1) ¼ 1. Mean: pq (1 qz)2 q P0X (1) ¼ p P0X (z) ¼
(11:4:12)
Variance: P00X (z) ¼ 2 P00X (1)
pq2 (1 qz)3
pq2 2q2 ¼2 ¼ p2 (1 qz)3
(11:4:13)
or m2 ¼
2q2 q þ p p2
Hence var½X ¼ m2 m21 ¼
B 11.5
q p2
(11:4:14)
MOMENT GENERATING FUNCTIONS
The moment generating function (MGF) MX(t) if it exists is defined by ð1 MX (t) ¼ E(etX ) ¼ fX (x)etx dx
(11:5:1)
1
where t is a real variable. This is similar to the characteristic function (CF) [Eq. (11.1.1)], except that jv has been replaced by the real variable t. Unlike the CF, which always exists, the MGF may or may not exist. Expanding E(e tX) in a power series about the origin and taking expectations, if they exist, we can write t2 X 2 tn X n tX þ þ þ MX (t) ¼ E(e ) ¼ E 1 þ tX þ 2! n! ¼ 1 þ tm þ
t2 m2 t n mn þ þ þ 2! n!
(11:5:2)
11.5
Moment Generating Functions
165
If MX(t) exists, Eq. (11.5.2) can be differentiated term by term to obtain d tm2 tn1 mn MX (t) ¼ m þ þ þ þ dx 2! n! Substituting t ¼ 0 in Eq. (11.5.3), we obtain MX0 (t)t¼0 ¼ m
(11:5:3)
(11:5:4)
Differentiating Eq. (11.5.2) successively with respect to t and setting t ¼ 0 in the resulting equation, we obtain the moments as follows: MX(2) (t)t¼0 ¼ m2 , . . . , m(n) (11:5:5) X (t)jt¼0 ¼ mn , . . . These results are very similar to those in Eq. (11.1.5) except for the absence of pffiffiffiffiffiffi ffi j ¼ 1. Since the methodology of finding the moments using MGF is very similar to that of using CF, we will give only three examples. Example 11.5.1 (Negative Binomal—Pascal) pmf is
If N is the random variable, then the
nb{n; k,p} ¼ P{n trials produce k successes} ¼
n 1 k nk p q : n ¼ k, k þ 1, . . . k1
Or substituting m ¼ n 2 k, we can write the pmf with M as the random variable: nb{m; k, p} ¼ P{(m þ k) trials produce k successes} kþm1 k m p q , m ¼ 0, 1, . . . ¼ k1 Since
kþm1 k1
¼
kþm1 m
we can write the MGF for the random variable M: 1 X kþm1 MM (t) ¼ (qet )m pk m m¼0 The negative binomial expansion of (1 2 qe t)2k can be written as follows: (1 qet )k ¼
1 X kþm1 (qet )m m m¼0
Substituting this equation in the previous one, we obtain the MGF: MM (t) ¼ pk (1 qet )k Clearly MM(0) ¼ 1.
166
Characteristic Functions and Generating Functions
Mean: 0 MM ðtÞ ¼ pk qk(1 qet )k1 et 0 MM (0) ¼ E(M) ¼
kq p
and E(N) ¼
kq k þk ¼ p p
Variance. We first find EfM 2g ¼ m2: 00 MM (t) ¼ pk qk(1 qet )k2 (kqet þ 1)et 00 MM (0) ¼ E(M 2 ) ¼
qk(1 þ qk) p2
Hence the variance of M is var½M ¼
qk(1 þ qk) (qk)2 qk 2 ¼ 2 p2 p p
The variance of N can be calculated by using N ¼ M þ k in the equations above. The MGF of the Gaussian distribution is
Example 11.5.2 (Gaussian Distribution) given by 1 MX (t) ¼ pffiffiffiffiffiffi 2p s
ð1
e½(xm)
2
=2s2 þtx
dx
1
Completing the squares, we obtain 2 2
MX (t) ¼ etmþ½ðs t
Þ=2
ð1 1 (x m ts2 )2 pffiffiffiffiffiffi exp dx 2s2 2p s 1
The value of the integral within braces is 1, and hence the MGF of a Gaussian is 2 2
MX (t) ¼ etmþ½(s t
)=2
Mean: MX0 (t) ¼ etm½(s
2 2
t )=2
(m þ s2 t)
MX0 (0) ¼ m Variance: MX00 (t) ¼ etmþ½(s
2 2
t )=2
MX00 (0) ¼ m þ s2
2 2
(m þ s2 t)2 þ etmþ½(s t
and
var½X ¼ s2
)=2 2
s
11.6
Example 11.5.3 (Gamma Distribution) given by
167
We will find the MGF of a gamma density
l(lx)a1 elx G(a)
fX (x) ¼
Cumulant Generating Functions
a, l; x . 0
From Eq. (11.5.1), we have ð1
l(lx)a1 elx tx e dx, a, l . 0 G(a) 0 ð la 1 a1 (lt)x x e dx ¼ G(a) 0
MX (t) ¼
Substituting (l 2 t)x ¼ y and using the definition of a gamma function [Eq. (7.5.2)], we can write this equation as ð1 la MX (t) ¼ ya1 ey dy G(a)(l t)a 0 MX (t) ¼
la la a G(a) ¼ G(a)(l t) (l t)a
Mean: la (a)(1) ala ¼ (l t)aþ1 (l t)aþ1 a MX0 (0) ¼ l MX0 (t) ¼
Variance. We first find EfX 2g ¼ m2: MX00 (t) ¼
a(a þ 1)la (l t)aþ2
MX00 ð0Þ ¼ m2 ¼
a(a þ 1)la a2 þ a ¼ l2 laþ2
Hence the variance is m2 m2X ¼
a2 þ a a2 a 2¼ 2 l2 l l
The means and variances of most of the commonly occurring discrete (Chapter 4) and continuous (Chapters 6 and 7) distributions are shown in Section 11.7.
B 11.6
CUMULANT GENERATING FUNCTIONS
The cumulant generating function (CGF) KX(t) is defined as the logarithm of the moment generating function MX(t), or KX (t) ¼ ln½MX (t). The cumulants also called semiinvariants {ki } are the coefficients of CGF: KX (t) ¼ ln½MX (t) ¼ tk1 þ
1 i X t2 tn t k2 þ þ kn þ ¼ ki 2! n! i! i¼0
(11:6:1)
168
Characteristic Functions and Generating Functions
We will now find the cumulants in terms of the moments mk. From Eq. (11.5.2), we can expand ln[MX(t)] in terms of a Taylor series as follows: t 2 m2 t n mn þ þ þ KX (t) ¼ ln½MX (t) ¼ ln 1 þ tm1 þ 2! n! t2 m2 t n mn ¼ tm1 þ þ þ þ 2! n! 2 1 t 2 m2 tn mn tm1 þ þ þ þ 2 2! n! 3 1 t 2 m2 tn mn tm1 þ þ þ þ + þ 3 2! n!
(11:6:2)
Collecting terms in t, we have KX (t) ¼ tm1 þ
t2 (m2 m21 ) 2!
þ
t3 (m3 3m1 m2 þ 2m31 ) 3!
þ
t4 (m4 3m22 4m1 m3 þ 12m21 m2 6m41 ) 4!
þ
t5 (m5 5m4 þ 30m1 m22 þ 40m21 m3 60m31 m2 24m51 ) 5!
þ
(11:6:3)
Equating coefficients of ti in Eqs. (11.6.1) and (11.6.3), the cumulants have been expressed in terms of the moments fmig in Eq. (11.6.4), where they are also expressed in terms of the central moments vi ¼ E½X mX i : k1 ¼ m1 ¼ mX k2 ¼ m2 m21 ¼ s2X ¼ v2 k3 ¼ m3 3m1 m2 þ 2m31 ¼ v3
(11:6:4)
k4 ¼ m4 3m22 4m1 m3 þ 12m21 m2 6m41 ¼ v4 3v2 k5 ¼ m5 5m4 þ 30m1 m22 þ 40m21 m3 60m31 m2 24m51 ¼ v5 10v2 v3 þ Skewness measures the lack of symmetry of the tails of a density. A density is symmetric if it looks the same on both the left and the right of the centerpoint. Skewness is defined as v3 g1 ¼ 3=2 (11:6:5) v2 Kurtosis is defined as a normalized form of the fourth central moment of a distribution. It is a measure of whether the density is peaked or flat relative to a normal distribution. High-kurtosis density functions will have a high peak near the mean, and low-kurtosis
11.6
Cumulant Generating Functions
functions will have a flat top. Kurtosis is defined as v4 g2 ¼ 2 v2
169
(11:6:6)
For Gaussian distributions the kurtosis is 3. Example 11.6.1 We will find the cumulants of the Poisson distribution given by p(k; l) ¼ el (lk =k!). t The MGF of the Poisson distribution is MX (t) ¼ el(1e ) , and the CGF is given by t KX (t) ¼ ln el(1e ) ¼ l þ let Expanding KX (t) in Taylor series, we have t2 t3 t4 t5 l þ let ¼ l t þ þ þ þ þ 2! 3! 4! 5! Thus, all the cumulants are equal to l, or ki ¼ l, i ¼ 1,2, . . . : Example 11.6.2
We will find the cumulants of a Gaussian distribution " # 1 1 x mX 2 fX (x) ¼ pffiffiffiffiffiffi exp 2 sX 2psX 2 2
The MGF is given by MX (t) ¼ etmX þ½(t sX )=2 and the CGF, by h i t2 s2X 2 2 KX (t) ¼ ln etmX þ½(t sX )=2 ¼ tmX þ 2 and the cumulants are k1 ¼ mX , k2 ¼ s2X and ki ¼ 0 for i . 2. Thus, for the Gaussian distribution, there are only two cumulants. Joint Cumulants
By analogy to Eq. (11.6.4), we can define joint cumulants for one to four random variables, X1, X2, X3, X4, as follows: k1 ¼ E½X1 ¼ mX k12 ¼ E½X1 X2 E½X1 E½X2 ¼ sX1 X2 k123 ¼ E½X1 X2 X3 E½X1 E½X2 X3 E½X2 E½X1 X3 E½X3 E½X1 X2 þ 2E½X1 E½X2 E½X3 k1234 ¼ E½X1 X2 X3 X4 E½X1 X2 E½X3 X4 E½X1 X3 E½X2 X4 E½X1 X4 E½X2 X3 E½X1 E½X2 X3 X3 E½X2 E½X1 X3 X4 E½X3 E½X1 X2 X4 E½X4 E½X1 X2 X3 þ 3E½X1 E½X2 E½X3 X4 þ 3E½X2 E½X3 E½X1 X4 þ 3E½X3 E½X4 E½X1 X2 þ 3E½X1 E½X4 E½X2 X3 6E½X1 E½X2 E½X3 E½X4
(11:6:7)
170
Characteristic Functions and Generating Functions
If the mean values of random variables Z1, Z2, Z3, Z4 are zero, then Eq. (11.6.7) can be considerably simplified as follows: k1 ¼ E½Z1 ¼ 0 k12 ¼ E½Z1 Z2 k123 ¼ E½Z1 Z2 Z3 k1234 ¼ E½Z1 Z2 Z3 Z4 E½Z1 Z2 E½Z3 Z4
(11:6:8)
E½Z1 Z3 E½Z2 Z4 E½Z1 Z4 E½Z2 Z3 If X1, X2, Y1, Y2 are four random variables, not necessarily independent, we will express the cov[X1 X2, Y1 Y2] in terms of the joint cumulant k1234 : cov½X1 X2 , Y1 Y2 ¼ E{½X1 X2 E(X1 X2 )½Y1 Y2 E(Y1 Y2 )} ¼ E½X1 X2 Y1 Y2 E½X1 X2 E½Y1 Y2
(11:6:9)
Expressing E½X1 X2 Y1 Y2 in terms of the fourth cumulant k1234 given in Eq. (11.6.8), we can rewrite Eq. (11.6.9) as follows: cov½X1 X2 ; Y1 Y2 ¼ k1234 þ E½X1 Y1 E½X2 Y2 þ E½X1 Y2 E½X2 Y1
B 11.7
(11:6:10)
TABLE OF MEANS AND VARIANCES pþq¼1
Random Variable
Mean
Variance Discrete Distributions
Bernoulli
p
pq
Binomial
Np
Npq or Np(1 2 p)
Multinomial
npi , q p
Geometric
i ¼ 1, . . . , m
Negative binomial E(M) ¼ Hypergeometric
Kn N
Poisson
l
Logarithmic— Benford
kq p
npi (1 pi ), i ¼ 1, . . . , m q p2 kq for M ¼ N k var(M) ¼ 2 for M ¼ N k p nK(N K)(N n) N 2 (n 1) l
2 8 2 :5 log ffi 3:44 7 : 34 (d ¼ 1, . . . , 9)
2 8 2 28 : 572 2 :5 ffi 6:06 log 13 50 log 7 :3 7 : 34 (d ¼ 1, . . . , 9)
Continuous Distributions Uniform
bþa 2
(b a)2 12 (Continued )
11.7
Random Variable
Table of Means and Variances
Mean 1 l
Variance 1 l2
Gaussian
m
Triangular
b
s2 (b a)2 6 2 l2 n , n integer l2 a l2 1 aþ2 2 aþ1 G G a a l2
Exponential
Laplace—double- 0 exponential n , n integer Erlang l a Gamma l 1 aþ1 Weibull mþ G l a Chi-square
Chi
Rayleigh Maxwell
ns2
2ns4
pffiffiffi nþ1 2sG 2 n G 2 rffiffiffiffi p s 2 rffiffiffiffi 2 2s p
3 nþ1 2G 6 7 2 n 7 n s2 6 4 5 G2 2 p s2 2 2 8 2 s 3 p
mR ¼ s2
Rice
2
2
171
rffiffiffiffi 2 2 p (m2 =4s2 ) m2 m m2 m 1 þ 2 I0 I þ e 1 2 2s 4s2 2s2 4s2
s2R ¼ m2 þ 2s2 m2R ; m: noncentrality parameter
Nakagami
1 1=2 G m þ V 2 m G(m)
3 2 1 2 G mþ V6 2 7 7 V 6 4 m G(m) 5 n , n.2 v2
Student-t
0
Snedecor F
n2 , n2 2
Lognormal
e½mþ(s =2)
2v22 (n1 þ n2 2) , n1 (n2 2)2 (n2 4) 2 2 e(2mþs ) es 1
Beta
a aþb
ab (a þ b þ 1)(a þ b)2
2
n2 . 2
v2 . 4
(Continued )
172
Characteristic Functions and Generating Functions
Random Variable
Mean
Variance
Cauchy
Not defined
Not defined
Pareto
b , a1
ab2 , a.2 (a 1)2 (a 2)
a.1
CHAPTER
12
Functions of a Single Random Variable
B 12.1
RANDOM VARIABLE g(X )
In many probability problems, we have to find the pdf of a function of random variables. We have already defined (in Chapter 5) a random variable X as the mapping from the sample space S to space of real numbers Rx; that is, it is the set of all outcomes j belonging to the sample space S such that the unique number X(j) belongs to the real line Rx. Thus, the random variable X is a mapping of the probability space {S, F, P} to the probability space on the real line {Rx , FX , PX }: X: {S, F, P} ! {Rx , FX , PX }
(12:1:1)
We can now define a random variable Y as a function of another random variable X, or Y ¼ g(X ), as the mapping from the real line Rx to another real line Ry, such that it induces a probability space {Ry , FY , PY }. In other words, the random variable Y is a composite function of the random variable X such that Y(j) ¼ g(X ) ¼ g[X (j)]. The mapping is given by Y: {S, F, P} ! {Rx , FX , PX } ! {Ry , FY , PY }
(12:1:2)
and shown in Fig. 12.1.1. Conditions on Y 5 g(X ) to be a Random Variable 1. The domain of g(x) must include the range of the random variable (RV) X. 2. For each y [ Iy, fg(x) yg must form a field FY ; that is, it must consist of the unions and intersections of a countable number of intervals of the real line Ry. 3. The probability of the event fg(x) ¼ +1g is 0. In essence, the transformation Y ¼ g(X ) induces a probability measure PY on FY , and we have to find this probability measure. From Fig. 12.1.1 we can write PY {Y} ¼ PX {g1 ( Y)} ¼ PX {x : g1 ( Y) , Ix }
(12:1:3)
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
173
174
Functions of a Single Random Variable
FIGURE 12.1.1
The set Ix consists of all the countable unions and intersections of the type a , x b. Without loss in generality, we will assume that Ix ¼ {X y}. Thus, Eq. (12.1.3) becomes PY {Y} ¼ P{Y y} ¼ FY ( y) ¼ PX {x : g1 ( Y) y} ¼ PX {x : g(x) y}
(12:1:4)
In conclusion, we can say that the probability of Y y is given by the probability of the set of all values of x such that g(x) y.
B 12.2
DISTRIBUTION OF Y 5 g(X )
We will find the distribution FY (y) of the RV Y ¼ g(X) in terms of the distribution FX (x) of the RV X and the function g(x). We shall do this with an arbitrary function g(x) bounded between a and b as shown in Fig. 12.2.1. As y traverses from 21 to þ1 along the g(x) axis, we have to find the values of x along the x axis that satisfy g(x) y. In Fig. 12.2.1 there are five distinct regions for y corresponding to the intersections of y with g(x): 1. y a, no points of intersection 2. a , y y1, one point of intersection x1 3. y1 , y y2, three points of intersection x2, x20 , x002 4. y2 , y b, one point of intersection x3 5. y . b, no points of intersection
FIGURE 12.2.1
12.2 Distribution of Y 5 g(X )
175
We have to find FY ( y) ¼ probability of g(x) y in each of these five regions. 1. y a: There are no points along the x axis for which g(x) y, or IX ¼ {1}; hence FY ( y) ¼ 0. 2. a , y y1: The set of all points along the x axis for which g(x) y, or Ix ¼ (x1, 1); hence FY ( y) ¼ 1 2 FX(x1). 3. y1 , y y2: The set of all points along the x axis for which g(x) y, or Ix ¼ (x2, x20 ] < (x002, 1); hence FY ( y) ¼ FX (x20 ) 2 FX (x2) þ 1 2 FX (x002). 4. y2 , y b: The set of all points along the x axis for which g(x) y or Ix ¼ (x3, 1); hence FY ( y) ¼ 1 2 FX(x3). 5. y . b: Here the entire g(x) is below y; and hence Ix ¼ (21, 1); hence FY ( y) ¼ 1. Thus, we have used Eq. (12.1.4) to find PfY y) along the entire g(x) axis. Special Case
We now consider a situation where g(x) ¼ y1 for x1 , x x2. In this case we have P{Y ¼ y1 } ¼ P{x1 , X , x2 } ¼ FX (x2 ) FX (x1 )
(12:2:1)
Hence FY ( y) is discontinuous at y ¼ y1, and the discontinuity is equal to FX (x2 ) FX (x1 ). Summary of Steps for Finding FY( y) 5 PfY y) 1. Graph the function g(x) along the x and g(x) axes. Do not label the g(x) axis as y. 2. Draw y parallel to the x axis. 3. Find the distinct regions along the g(x) axis. The regions are determined by the number of intersections of y with g(x) as y goes from 21 to 1 along the g(x) axis. Let these be y1, y2, . . . . 4. In each region solve for all values of x for which y ¼ g(x). Let these be x, x0 , x00 , . . . . 5. For each of these regions, find Ix such that g(x) y. 6. Find FY( y) ¼ PfY yg ¼ Pfx [ Ix) ¼ Pfx : g(x) yg. We will take a number of examples of increasingly difficult transformations g(x) and solve for PfY yg using the steps listed above. We shall assume that in all the following examples FX(x) or fX(x) is given and we have to find FY( y). Example 12.2.1 Given that Y ¼ aX þ b with a . 0, we have to find FY( y). We follow the steps listed above: 1. The function g(x) ¼ ax þ b has been graphed. 2. y parallel to the x axis has been drawn in Fig. 12.2.2. 3. In this simple example there is only one region 21 , y , 1. 4. In this region, solving for y ¼ ax þ b, we have x ¼ ( y 2 b)/a. 5. The region Ix such that g(x) y is given by yb Ix ¼ 1 , x a y b yb 6. FY ( y) ¼ P{x : g(x) y} ¼ P X ¼ FX a a
176
Functions of a Single Random Variable
FIGURE 12.2.2
We can now find the density fY ( y) by differentiating FY ( y) with respect to y. The result is fY ( y) ¼
Example 12.2.2 before.
dFX (ð y bÞ=a) 1 yb ¼ fX dy a a
We repeat Example 12.2.1 with a 0. We follow the same steps as
1. The function g(x) is graphed in Fig. 12.2.3. 2. y has been drawn parallel to x. 3. In this example also, there is only one region 21 , y ,1. 4. In this region, solving for y ¼ ax þ b, we have x ¼ ( y 2 b)/a or x ¼ [( y 2 b)/jaj]. 5. The region IX such that g(x) y is given by
6.
yb ,x1 IX ¼ jaj yb FY ( y) ¼ P{x : g(x) y} ¼ P X . jaj yb yb ¼ 1 FX ¼1P X . jaj jaj
The corresponding density function fY ( y) is given by differentiating FY ( y) with respect to y, and the result is dFX fY ( y) ¼
yb 1 yb a fX ¼ jaj a dy
The result is very similar to that obtained in Example 12.2.1 except for the absolute value of a.
12.2 Distribution of Y 5 g(X )
177
FIGURE 12.2.3
Example 12.2.3 Let g(x) ¼ (x 2 a)2. This example is different from the previous two examples because there are multiple regions for y. Following similar steps, we obtain 1. The function g(x) is graphed in Fig. 12.2.4 and 2. y has been drawn parallel to x. 3. Here the two regions of y are fy 0g, no points of intersection and fy . 0g, two points of intersection. 4. In the region fy 0g, Ix ¼ 1. In the region fy.0g, we solve for y ¼ g(x) and pffiffiffi pffiffiffi obtain x1 ¼ a þ y and x2 ¼ a y. pffiffiffi pffiffiffi 5. Thus, Ix ¼ (a y, a þ y , y . 0. 6. In the first region FY ( y) ¼ 0 : y 0. In the second region pffiffiffi pffiffiffi y , X a þ y) pffiffiffi pffiffiffi ¼ FX (a þ y ) FX (a y ) : y . 0
FY ( y) ¼ P(a
FIGURE 12.2.4
178
Functions of a Single Random Variable
The corresponding density function is fY ( y) ¼ 0 : y 0 fY ( y) ¼
d pffiffiffi pffiffiffi (FX (a þ y ) FX (a y )) dy
1 pffiffiffi pffiffiffi ¼ pffiffiffi (fX (a þ y ) þ fX (a y )) : y . 0 2 y If X is a zero mean Gaussian distributed random variable given by 1 2 2 fX (x) ¼ pffiffiffiffiffiffi e(x =2s ) s 2p and if Y ¼ X 2, then fY ( y) is given by 1 1 1 2 2 2 fY ( y) ¼ pffiffiffi pffiffiffiffiffiffi e( y=2s ) þ e( y=2s ) ¼ pffiffiffiffiffiffiffiffi e( y=2s ) 2 y s 2p s 2py which is a chi-square distribution with one degree of freedom [Eq. (7.7.1)].
Example 12.2.4 This example is a complement of Example 12.2.3. Here g(x) ¼ 1=(x a)2 and we have to find FY ( y). We follow the same procedure laid out earlier. 1. The function g(x) is graphed in Fig. 12.2.5. 2. y has been drawn parallel to the x axis. 3. Here the two regions of y are fy 0g, no points of intersection and fy . 0g, two points of intersection. 4. In the region f y 0g, Ix ¼ {1} and FY( y) ¼ 0. In the region f y . 0g, we solve for pffiffiffi pffiffiffi y ¼ g(x) and obtain x1 ¼ a þ ð1= y Þ and x2 ¼ a (1= y ). 5. Thus 1 1 Ix ¼ 1, a pffiffiffi < a þ pffiffiffi , 1 y y
FIGURE 12.2.5
12.2 Distribution of Y 5 g(X )
179
6. In the region indicated in item 5 (above), we obtain 1 1 FY ( y) ¼ P X a pffiffiffi þ 1 P X a þ pffiffiffi y y 1 1 ¼ FX a pffiffiffi þ 1 FX a þ pffiffiffiffi : y . 0 y Y The corresponding density function is fY ( y) ¼ 0 : y 0 d 1 1 FX a pffiffiffi þ 1 FX a þ pffiffiffi fY ( y) ¼ dy y y 1 1 1 ¼ 3=2 fX a pffiffiffi þ fX a þ pffiffiffi : y . 0 2y y y Example 12.2.5 The previous examples illustrated continuous functions of g(x). In this example g(x) is a discrete function as shown in Fig. 12.2.6. Note that g(x) is not a true function because there are many points of y at x ¼ +a. We can still find FY ( y) given FX (x). The procedure is no different from that in the previous examples. 1. The function g(x) has already been graphed in Fig. 12.2.6. 2. y has been drawn parallel to the x axis. 3. There are four region of y, given by (1) y 2b, (2) 2b , y 0, (3) 0 , y b, (4) y . b. 4. The intersection points are (1) none, (2) x1 ¼ 2a, (3) x2 ¼ a, (4) none. 5. The corresponding four regions along the x axis are (1) Ix ¼ {1}, (2) Ix ¼ {1, a}, (3) Ix ¼ f1, a}, (4) Ix ¼ {1, 1}. 6. The distribution function FY( y) in the various regions are (1) y 2 b : FY ( y) ¼ 0, (2) 2b , y 0 : FY ( y) ¼ FX (2a), (3) 0 , y b : FY ( y) ¼ FX (a), (4) y . b : FY ( y) ¼ 1. The distribution function FY( y) is a “staircase” function, indicating that Y is a discrete random variable and the density function fY ( y)
FIGURE 12.2.6
180
Functions of a Single Random Variable
FIGURE 12.2.7
is (a) fY( y) ¼ 0, (b) fY( y) ¼ FX (2a)d(x þ b), (c) fY( y) ¼ [FX(a) 2 FX(2a)]d(x), (d) [1 2 FX(a)]d(x 2 b). The distribution and density functions are shown in Fig. 12.2.7.
Example 12.2.6 We shall now take an example of a piecewise-continuous g(x) with a specified fX(x) and FX(x). The functions g(x), fX(x) and FX(x) are shown in Fig. 12.2.8.
FIGURE 12.2.8
12.2 Distribution of Y 5 g(X )
181
We have to find FY( y) given that fX(x) is defined by Fig. 12.2.8 as 2
3 1 : 3 , x 2 5 fX (x) ¼ 4 5 0: otherwise Since fX(x) is specified, FX(x) is determined by integrating fX(x). The result is 2
0: 6x 3 FX (x) ¼ 4 þ : 5 5 1:
x 3
3
7 3 , x 2 5 x.2
and is shown in Fig. 12.2.8. In this example, the range of fX(x), x [ f23, 2g has to be factored in determining FY ( y). Other than that, we will still follow the procedure laid out earlier. 1. The function g(x) has already been graphed in Fig. 12.2.8. 2. y will scan from 21 to 1 along the g(x) axis. 3. The regions of y are given by (a) y 24, (b) 24 , y 22, (c) 22 , y21, (d) 21 , y 0, (e) 0 , y 1, (f) 1 , y 2, (g) 2 , y 4, (h) y . 4. 4. The points of intersection of y with g(x) are (a) none, (b) x ¼ 22, (c) x ¼ y, (d) x ¼ 0, (5) x ¼ 1, (6) x ¼ y, (7) x ¼ 2, (8) none. 5. The corresponding regions Ix are (a) Ix ¼ f1g, (b) Ix ¼ f21, 22g, (c) Ix ¼ f21, yg, (d) Ix ¼ f21, 0g, (e) Ix ¼ f21, 1g, (f) Ix ¼ f21, yg, (g) Ix ¼ f21, 2g, (h) Ix ¼ f21, 1g. 6. The distributions functions in the six regions are (a) y 4 : FY ( y) ¼ 0 1 (b) 4 , y 2 : FY ( y) ¼ FX (2) FX (4) ¼ 5 y 3 (c) 2 , y 1 : FY ( y) ¼ FX ( y) ¼ þ 5 5 3 (d) 1 , y 0 : FY ( y) ¼ FX (0) ¼ 5 4 (e) 0 , y 1 : FY ( y) ¼ FX (1) ¼ 5 y 3 (f) 1 , y 2 : FY ( y) ¼ FX ( y) ¼ þ 5 5 (g) 2 , y . 4 : FY ( y) ¼ FX (2) ¼ 1 since FX (x) ¼ 1 for all values of x . 2 (h) y . 4 : FY ( y) ¼ 1 By differentiating FY( y) in every region, we can express the density function fY ( y) as 1 1 1 1 fY ( y) ¼ d( y þ 3) þ ½u( y þ 1) u( y þ 2) þ d( y þ 1) þ d( y) 5 5 5 5 1 þ ½u( y 2) u( y 1) 5 where u( y) is the usual step function. FY ( y) and fY ( y) are graphed in Fig. 12.2.9.
182
Functions of a Single Random Variable
FIGURE 12.2.9
Example 12.2.7 This example is similar to the previous one with a piecewisecontinuous function g(x). The functions g(x), fX(x) and FX(x) are given by 2
3 0: x 2 "1 # 6 1: 2 , x 0 7 6 7 : 2 , x 2 7 ; g(x) ¼ 6 6 2 x: 0 , x 3 7; fX (x) ¼ 4 4 5 0: otherwise 3 2: x.3 2 3 0: x 2 6x 1 7 FX (x) ¼ 4 þ : 2 , x 2 5 4 2 1: x,2
The functions g(x), fX(x), and FX(x) are shown in Fig. 12.2.10. We have to find FY( y). Following the same procedure as before 1. The function g(x) has been graphed in Fig. 12.2.10. 2. y has been drawn, and the intersection of y with g(x) is given by x ¼ 32y. 3. The regions of y are given by (a) y 21, (b) 21 , y 0, (c) 0 , y 43, (d) 43 , y 2, (e) y . 2. Even though g(x) extends upto x ¼ 3, the function fX(x) terminates at x ¼ 2 and FX(x) ¼ 1 beyond x ¼ 2. 4. The points of intersection of y with g(x) are (a) none, (b) x ¼ 22,0, (c) x ¼ 32y, (d) x ¼ 32y, (e) none. 5. The corresponding regions Ix are (a) Ix ¼ f1g, (b) Ix ¼ (– 2, 0], (c) Ix ¼ (21, 23 x], (d) Ix ¼ (21, 23x], (e) Ix ¼ (21, 1]. 6. The distribution functions in the four regions are (a) y 21: FY ( y) ¼ 0, (b) 21 , y 0: FY( y) ¼ FX(0) 2 FX(22) ¼ 12, (c) 0 , y 43: FY( y) ¼ FXð32 yÞ ¼ 3 1 4 8 y þ 2, (d) 3 , y 2: FY ( y) ¼ 1, (e) y 2: FY( y) ¼ 1.
12.2 Distribution of Y 5 g(X )
FIGURE 12.2.10
The corresponding density function fY ( y) is given by 1 3 4 fY ( y) ¼ d( y þ 1) þ u y u( y) 2 8 3 The functions FY ( y) and fY ( y) are shown in Fig. 12.2.11.
Example 12.2.8 We shall now take a more complicated example and find FY ( y). In this example g(x) is defined by g(x) ¼
1 x2 1
for all x
FIGURE 12.2.11
183
184
Functions of a Single Random Variable
FIGURE 12.2.12
The probability density function fX(x) and the distribution function FX(x) are given by 2
2 fX (x) ¼ 4 9 0
3 if 1:5 , x 3 5 ; otherwise
2
0 1 62 FX (x) ¼ 4 x þ 9 3 1
3
if
x 1:5
if
7 1:5 x 3 5
if
x.3
The function g(x) is shown in Fig. 12.2.12 and fX(x) and FX(x) are shown in Fig. 12.2.13. 1. g(x) has been graphed in Fig. 12.2.12. 2. y scans from 21 to þ1 along the g(x) axis. 3. The regions of y are given by (a) y 21, two points of intersection in the range of fX(x), (21.5, 3]; (b) 21 , y 0, no points of intersection; (c) 0 , y 18, two points of intersection beyond the range of fX(x), (21.5, 3]; (d) 18 , y 45, one point of intersection in the range of fX(x), (21.5, 3], (e) y . 45, two points of intersection in the same range. pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 4. The p points of ffiintersection of y with g(x) areffi (a) x1 ¼pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ (1=y) and ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi x2 ¼ 1 þ (1=y), (b) none, (c) x1 ¼ 1 þ (1=y) and x2 p ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ (1=y) beyond ffi the range of (21.5, 3], (d) one point of intersection x2 ¼ 1 þ (1=y) within the
FIGURE 12.2.13
12.2 Distribution of Y 5 g(X )
185
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi rangep and anotherffi x1 ¼ 1 þ (1=y) beyond the range, (e) x1 ¼ 1 þ (1=y) and ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x2 ¼ 1 þ (1=y). 5. The corresponding regions Ix are sffiffiffiffiffiffiffiffiffiffiffi) (sffiffiffiffiffiffiffiffiffiffiffi ) ( 1 1 ðaÞ Ix ¼ 1, 1 þ < 1þ , 1 y y ðbÞ Ix ¼ (1, 1 sffiffiffiffiffiffiffiffiffiffiffi) (sffiffiffiffiffiffiffiffiffiffiffi ) 1 1 Ix ¼ 1, 1 þ < {1, 1} 1þ , 1 y y (
ðcÞ
sffiffiffiffiffiffiffiffiffiffiffi) (sffiffiffiffiffiffiffiffiffiffiffi ) 1 1 ðdÞ Ix ¼ 1, 1 þ < {1, 1} 1þ , 1 y y (
sffiffiffiffiffiffiffiffiffiffiffi) (sffiffiffiffiffiffiffiffiffiffiffi ) 1 1 Ix ¼ 1, 1 þ < {1, 1} 1þ , 1 y y (
ðeÞ
6. The distribution functions in the various regions are sffiffiffiffiffiffiffiffiffiffiffi! sffiffiffiffiffiffiffiffiffiffiffi! 1 1 1þ ðaÞ y 1 : FY ( y) ¼ FX 1 þ FX (1) þ FX (1) FX y y sffiffiffiffiffiffiffiffiffiffiffi 2 1 1 2 1 2 1 þ 1þ þ þ ¼ þ 9 y 3 9 3 9 3 sffiffiffiffiffiffiffiffiffiffiffi ! 2 1 1 1þ þ 9 y 3 sffiffiffiffiffiffiffiffiffiffiffi 4 4 1 1þ ¼ 9 9 y
ðbÞ
ðcÞ
1 , y 0 : FY ( y) ¼ FX (1) þ FX (1) ¼
2 1 2 1 4 þ þ ¼ 9 3 9 3 9
sffiffiffiffiffiffiffiffiffiffiffi! 1 1 0 , y : FY ( y) ¼ FX 1 þ FX (1) þ FX (1) FX (1) 8 y sffiffiffiffiffiffiffiffiffiffiffi! 1 þ FX (1) FX 1þ y 4 4 ¼00þ þ11¼ 9 9
186
Functions of a Single Random Variable
sffiffiffiffiffiffiffiffiffiffiffi! 1 4 1 , y : FY ( y) ¼ FX 1 þ ðdÞ FX (1) þ FX (1) FX (1) 8 5 y sffiffiffiffiffiffiffiffiffiffiffi! 1 þ FX (1) FX 1þ y sffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffi ! 4 2 1 1 10 2 1 1þ þ 1þ ¼00þ þ1 ¼ 9 9 y 3 9 9 y sffiffiffiffiffiffiffiffiffiffiffi! 4 1 ðeÞ y . : FY ( y) ¼ FX 1 þ FX (1) þ FX (1) FX (1) 5 y sffiffiffiffiffiffiffiffiffiffiffi! 1 þ FX (1) FX 1þ y sffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffi ! ! 2 1 1 4 2 1 1 1þ þ 1þ þ ¼ 0þ þ1 9 y 3 9 9 y 3 sffiffiffiffiffiffiffiffiffiffiffi 13 4 1 ¼ 1þ 9 9 y The density function fY ( y) can be obtained by differentiating FY ( y). Functions FY ( y) and fY ( y) are shown in the following equation. FY ( y) is graphed in Fig. 12.2.14a, and fY ( y) is graphed in Fig. 12.2.14b. sffiffiffiffiffiffiffiffiffiffiffi 3 2 4 4 1 1þ if y 1 7 6 7 6 9 9 y 7 6 64 17 6 if 1 , y 7 69 87 7 6 sffiffiffiffiffiffiffiffiffiffiffi 7; FY ( y) ¼ 6 6 10 2 1 1 4 7 7 6 6 9 9 1 þ y if 8 , y 5 7 7 6 7 6 sffiffiffiffiffiffiffiffiffiffiffi 7 6 5 4 13 4 1 4 1þ if y. 9 9 y 5 3 22 1 rffiffiffiffiffiffiffiffiffiffiffi if y 1 7 69 2 7 6 y 1þ1 7 6 y 7 6 6 17 60 if 1 , y 7 6 87 7 6 61 1 1 4 7 fY ( y) ¼ 6 7 6 9 rffiffiffiffiffiffiffiffiffiffiffi1 if 8 , y 5 7 7 6 2 7 6 y 1þ 7 6 y 7 6 62 1 4 7 7 6 y. 4 9 rffiffiffiffiffiffiffiffiffiffiffi1 if 5 5 2 y 1þ y
12.2 Distribution of Y 5 g(X )
187
FIGURE 12.2.14
Example 12.2.9 In Example 12.2.8 we will find the conditional probability of Y conditioned on the event A ¼ jXj . 1. Since jXj . 1, the part of the function g(x) that lies between 21 and 1 has no effect on the conditional distribution. Thus, g(x) is given by
g(x) ¼
1 x2 1
for
jxj , 1
The conditional density function fX(x j A) with A ¼ jXj . 1 is given by
fX (x j A) ¼
fX (x) fX (x) ¼ p(jXj . 1) FX (1) þ 1 FX (1) f (x) 9 X ¼ fX (x) ¼ 2 1 2 1 5 þ þ þ1 9 3 9 3
for
jxj . 1
188
Functions of a Single Random Variable
Hence, the conditional density fX(x j A) and the conditional distribution FX(x j A) can be written as 22 3 if 1:5 , x 1 65 7 6 7 fX (x j A) ¼ 6 2 7; if 1,x3 5 4 5 0 otherwise 3 2 0 if x , 1:5 7 6 2x þ 3 if 1:5 , x 1 7 6 7 6 5 7 6 1 7 6 if 1 , x 1 7 FX (x j A) ¼ 6 7 6 5 7 6 2x 1 6 if 1,x3 7 5 4 5 1 if x.3 The function g(x j A) is shown in Fig. 12.2.15. f (x j A) and FX(x j A) are shown in Fig. 12.2.16. Using exactly the same procedure as in Example 12.3.9, we can find the conditional distribution FY ( y j A) and the conditional density fY ( y j A) for the regions (a) y 18, (b) 18 , y 45, and (c) y . 45. The results are shown below: 3 2 1 if y 7 60 8 7 6 sffiffiffiffiffiffiffiffiffiffiffi! 7 6 1 1 47 6 2 7 6 3 1 þ , x if FY ( y j A) ¼ 6 5 y 8 57 7 6 sffiffiffiffiffiffiffiffiffiffiffi! 7 6 7 61 1 4 5 4 94 1þ if y . 5 y 5 2
0
6 1 6 0 6 6 C 61B 1 6 B rffiffiffiffiffiffiffiffiffiffiffiC 65@ 1A 6 y2 1 þ fY ( y j A) ¼ 6 y 6 0 1 6 6 6 B C 1 62B 6 @ rffiffiffiffiffiffiffiffiffiffiffiC 45 1A y2 1 þ y
if
if
if
y
1 8
3
7 7 7 7 1 47 ,x 7 8 57 7 7 7 7 7 7 4 7 7 y. 5 5
FY ( y j A) and fY (Y j A) are also graphed in Fig. 12.2.17.
Example 12.2.10 In the previous examples g(x) was explicitly given, and we had to find FY ( y) given fX(x). In this example g(x) is not explicitly given, but we will have to determine it from the problem. Points are uniformly distributed along the circumference of a unit circle. From each point a tangent is drawn that intersects the x axis (Fig. 12.2.18). We have to find the distribution of the points of intersection along the x axis. Since the
12.2 Distribution of Y 5 g(X )
189
FIGURE 12.2.15
points are uniformly distributed, the angle Q that the point subtends at the center of the circle is a random variable that is uniformly distributed as U(0,2p). The transformation mapping X ¼ g(Q) between Q and X is found as follows. Since the radius of the circle is 1, the cosine of the angle u subtended by any point on the unit circle at the center of the circle is given by cos(u) ¼ (1/x). Hence, the transformation between the random variable Q and the random variable X is given by X ¼ g(Q) ¼ sec(Q) The function g(u) is graphed in Fig. 12.2.19. The density function fQ(u) and the distribution function FQ(u) of the random variable Q are given by 2 3 0 if x 0 " 1 # for 0 , u 2p 6 u 7 ; FQ (u) ¼ 4 fQ (u) ¼ 2p if 0 , u 2p 5 2p 0 otherwise 1 if x . 2p They are shown in Fig. 12.2.20.
FIGURE 12.2.16
190
Functions of a Single Random Variable
FIGURE 12.2.17
The usual procedure can now be followed to determine the distribution function FX(x). 1. The function g(u) has been graphed. 2. The line x has been drawn in Fig. 12.2.19.
FIGURE 12.2.18
12.2 Distribution of Y 5 g(X )
191
FIGURE 12.2.19
3. The regions along the x axis are given by (a) x21, two points of intersection; (b) 21 , x 1, no points of intersection; (c) x . 1, two points of intersection. 4. Intersection points of x with g(u) are (a) u1 ¼ cos21 (1/x) and u10 ¼ 2p 2 cos21(1/x), (b) no points of intersection, (c) u2 ¼ cos21(1/x) and u20 ¼ 2p 2 cos21(1/x). 5. The corresponding regions Iu are p 1 S 1 3p , cos1 (a) Iu ¼ 2p cos1 , 2 x x 2 p 3p , (b) Iu ¼ 2 2 1 S p 3p S 1 , (c) Iu ¼ 0, cos1 2p cos1 , 2p x 2 2 x 6. The distribution functions in the various regions are 1 1 1 3 1 1 1 1 1 cos1 þ 1 cos1 (a) FX (x) ¼ ¼ cos1 2p x 4 4 2p x x 2 p 3 1 1 (b) FX (x) ¼ ¼ 4 4 2
FIGURE 12.2.20
192
Functions of a Single Random Variable
FIGURE 12.2.21
(c) FX (x) ¼
1 1 1 1 1 1 1 1 cos1 þ þ 1 1 cos1 ¼ cos1 þ 2p x 2 2p x x 2 p
This distribution is called the arcsine law. The density function fX(x) is obtained by differentiating the distribution function FX(x). They are both shown below, and the graphs are shown in Fig. 12.2.21: 2
3 1 1 1 cos1 if x 1 6p 7 x 2 6 7 61 7 FX (x) ¼ 6 if 1 , x 1 7; 62 7 41 5 1 1 1 cos þ if x.1 p x 2 2 3 1 p ffiffiffiffiffiffiffiffiffiffiffiffi ffi if x 1 6 px x2 1 7 6 7 6 fX (x) ¼ 6 0 if 1 , x 1 7 7 4 5 1 pffiffiffiffiffiffiffiffiffiffiffiffiffi if x.1 px x2 1 Example 12.2.11 As a final example, we will take the case when g(u) ¼ cos(u) in the region 2p , u p. In the previous example g(x) ¼ sec(u), and here g(x) ¼ cos(u). We will also assume as in the previous example that fQ(u) is uniformly distributed in
193
12.2 Distribution of Y 5 g(X )
FIGURE 12.2.22
(2p, p]. Thus, fQ(u) and FQ(u) are given by " 1 fQ (u) ¼ 2p 0
for
p , u p otherwise
2
# ;
0 1 6 u FQ (u) ¼ 4 þ 2p 2 1
if if if
x0
3
7 p , u p 5 x.p
The function g(u) ¼ cos(u) is shown in Fig. 12.2.22. The density and distribution functions are graphed in Fig. 12.2.23. Following the usual procedure 1. g(u) has been graphed. 2. The line x has been drawn (Fig. 12.2.22). 3. The regions along the x axis are given by (a) x 21, no points of intersection; (b) 21 , x 1, two points of intersection; (c) x . 1, no points of intersection. 4. Intersection points of x with g(u) are (a) no points of intersection, (b) u1 ¼ 2cos21(x) and u10 ¼ cos21(x), (c) no points of intersection. 5. The corresponding regions Iu are (a) Iu ¼ {1}, (b) Iu ¼ { p, cos1 (x)} {cos1 (x), p}, (c) Iu ¼ f.
FIGURE 12.2.23
194
Functions of a Single Random Variable
FIGURE 12.2.24
6. The distribution functions in the various regions are (a) FX (x) ¼ 0 1 1 p 1 (b) 1 cos (x) þ þ FX (x) ¼ 2p 2 2p 2 p 1 1 1 1 1 þ cos (x) þ þ ¼ 1 cos1 (x) 2p 2 2p 2 p (c) FX (x) ¼ 1 Here also the distribution is the arcsine law. The density function fX(x) is obtained by differentiating the distribution function FX(x). The distribution and density functions are given below and graphed in Fig. 12.2.24. 2
0
if
1 6 FX (x) ¼ 4 1 cos (x) if p 1 if 2 0 if 6 1 ffi if fX (x) ¼ 6 4 ppffiffiffiffiffiffiffiffiffiffiffiffi 1 x2 0 if
x 1
3
7 1 , x 1 5; x.1 x 1
3
7 1 , x 1 7 5 x.1
B 12.3 DIRECT DETERMINATION OF DENSITY fY ( y) FROM fX (x) In the previous section, for any given Y ¼ g(X ), we determined fY ( y) from fX (x) through a circuitous route by converting fX (x) to FX(x), using the transformation g(X ) to find FY ( y), and then differentiating FY ( y) to obtain fY ( y) as shown in the schematic representation in Fig. 12.3.1. Consider the function g(x) given in Fig. 12.3.2. The probability of the random variable Y lying between y and y þ dy is given by fY ( y)dy ¼ P{y , Y y þ dy}
(12:3:1)
12.3 Direct Determination of Density fY ( y) from fX (x)
195
FIGURE 12.3.1
In Fig. 12.3.2 y ¼ g(x) has three real solutions: x1, x2, x3. Hence the set of values for which {y , g(x) y þ dy} is given by the union of the set of values corresponding to the three points of intersection, {x1 , X x1 þ dx1 } {x2 þ dx2 , X x2 } {x3 , X x3 þ dx3 }, where dx1 . 0, dx3 . 0, and dx2 , 0. By the conservation of probability measure, we have fY ( y)dy ¼ P{y , Y y þ dy} ¼ P{x1 , X x1 þ dx1 } þ P{x2 þ dx2 , X x2 } þ P{x3 , X x3 þ dx3 } ¼ fX (x1 )dx1 þ fX (x2 ) j dx2 j þ fX (x3 )dx3
(12:3:2)
We can now write fY ( y) ¼ fX (x1 )
dx1 jdx2 j dx3 þ fX (x3 ) þ fX (x2 ) dy dy dy
(12:3:3)
jdx2 j 1 ¼ 0 , dy jg (x2 )j
(12:3:4)
But dx1 1 ¼ 0 , jg (x1 )j dy
FIGURE 12.3.2
dx3 1 ¼ 0 jg (x3 )j dy
196
Functions of a Single Random Variable
Substituting Eq. (12.3.4) into Eq. (12.3.3), we obtain the result for fY ( y) directly from fX(x): fY ( y) ¼
fX (x1 ) fX (x2 ) fX (x3 ) þ 0 þ 0 0 jg (x1 )j jg (x2 )j jg (x3 )j
(12:3:5)
Equation (12.3.5) can be generalized to the case when there are n real solutions to the equation y ¼ g(x), given by {x1 , x2 , . . . , xn }: fY ( y) ¼
n X fX (xi ) 0 (x )j jg i i¼1
(12:3:6)
From the derivation it is clear that certain restrictions have to be satisfied if we have to determine fY ( y) directly from fX(X ). Conditions for Direct Determination of fX(X) 1. The derivative of g(x) must exist. 2. There can be no straight-line solutions to g(x) ¼ y. We now formulate the following steps for the direct determination of fY ( y) from fX(x): Steps for Direct Determination of fY ( y) 1. Check whether g(x) satisfies restrictions 1 and 2. 2. Solve for y ¼ g(x). Let the n real roots be fx1, x2, . . . , xng. 3. Determine
dg(x)
i ¼ 1, 2, . . . , n
dx , x¼xi 4. Calculate fY ( y) ¼
n X fX (xi ) jg0 (xi )j i¼1
Thus, if the restrictions on g(x) are satisfied, it is easier to calculate fY ( y) directly from fX(x). We will take some of the examples in the previous section and solve for fX(x) directly. Example 12.3.1
We will find fY ( y) for Y ¼ aX þ b with a . 0 or, a 0.
1. g(x) satisfies the restrictions. 2. We solve for y ¼ ax þ b. The solution yields x ¼ ( y 2 b)/a.
dg(x) d(ax þ b)
¼ jaj
¼ 3.
dx dx
4. fY ( y) ¼ 1 fX y b jaj a Example 12.3.2
(12:3:7)
Y ¼ g(X ) ¼ (X 2 a)2:
1. g(x) satisfies the restrictions. pffiffiffi pffiffiffi 2. Solving for x in y ¼ (x 2 a)2, we obtain two roots: x1 ¼ a y, x2 ¼ a þ y
12.3 Direct Determination of Density fY ( y) from fX (x)
3.
dg(x) d(x a)2
pffiffiffi
¼
dx dx ¼ j2(x a)j ¼ 2 y
4.
1 pffiffiffi pffiffiffi fY ( y) ¼ pffiffiffi ½ fX (a y ) þ fX (a þ y ); 2 y
Example 12.3.3
y.0
197
(12:3:8)
Y ¼ g(X) ¼ 1=(X a)2 .
1. g(x) satisfies the restrictions.
pffiffiffi 2. Solving for x in y ¼ 1=(x a)2 , we obtain two roots x1 ¼ a (1= y), pffiffiffi x2 ¼ a þ (1= y).
1
1
3.
dg(x)
d 3=2 ¼ ¼ 2
dx dx (x a)2 (x a)3 ¼ j2y j 1 1 1 4. fY ( y) ¼ 3=2 fX a pffiffiffi þ fX a þ pffiffiffi ; y . 0 (12:3:9) 2y y y Example 12.3.4 (See Example 12.2.8) The probability density functions fX(x) and g(x) and are given by 2 3 2 1 if 1:5 , x 3 5 g(x) ¼ 2 for all x fX (x) ¼ 4 9 x 1 0 otherwise Even though g(x) is defined from 21 to 1, the range of fX(x) plays a critical role in determining fY ( y). Figure 12.2.12 is redrawn in Fig. 12.3.3 to indicate the ranges. 1. g(x) satisfies the restrictions.
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2. Solving for x in pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi y ¼ 1/(x 2 1), we obtain two roots: x1 ¼ 1 þ (1=y), x2 ¼ 1 þ (1=y).
sffiffiffiffiffiffiffiffiffiffiffi
2
3. dg(x) d 1 x 1
dx ¼ dx (x2 1) ¼ 2 (x2 1)2 ¼
2y 1 þ y
4. fY ( y) have to be calculated in the four ranges of y as shown in Fig. 12.3.3. The density function fY ( y) is calculated as follows: Region I: y 21, two points of intersection: 1 2 2 2 1 þ
rffiffiffiffiffiffiffiffiffiffiffi
rffiffiffiffiffiffiffiffiffiffiffi
fY ( y) ¼ ¼
9 2
1 9 9 1
2y2
1 þ
y 1 þ
y y 1 Region II: 1 , y , no points of intersection: 8 fY ( y) ¼ 0 1 4 Region III: , y , one point of intersection: 8 5 1 2 1 1 fY ( y) ¼
rffiffiffiffiffiffiffiffiffiffiffi
rffiffiffiffiffiffiffiffiffiffiffi
¼
9 2
1 9 1
2y2
1 þ
y 1 þ
y y
198
Functions of a Single Random Variable
FIGURE 12.3.3
4 Region IV: y . , two points of intersection: 5 1 2 2 2 1 þ fY ( y) ¼
rffiffiffiffiffiffiffiffiffiffiffi
rffiffiffiffiffiffiffiffiffiffiffi
¼
9 9
9 1 1
2y2
1 þ
y2
1 þ
y y Here we have found the result far more easily than in Example 12.2.8. Example 12.3.5 (See Example 12.2.10) by g(u) ¼ sec(u) and fQ(u) is given by " 1 fQ (u) ¼
2p 0
In Example 12.2.10, the function g(u) is given
for 0 , u 2p
#
otherwise
We will now find fX(x) directly from fQ(u). 1. g(u) satisfies the restrictions. 2. Solving for u in x ¼ sec(u), we have u1 ¼ sec21(x) and u10 ¼ 2p 2 sec21(x).
pffiffiffiffiffiffiffiffiffiffiffiffiffi
3.
dg(u)
d
¼ j sec (u) tan (u)j ¼
x x2 1
¼ sec (u)
du du 4. From Fig. 12.2.19, the regions of x are (a) x 21, (b) 2 1 , x 1, (c) x . 1. (a) x 21, two points of intersection: 1 1 1 1 fX (x) ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi þ ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi x x2 1 2p 2p px x2 1 (b) 21 , x 1, no points of intersection, fX(x) ¼ 0.
199
12.3 Direct Determination of Density fY ( y) from fX (x)
(c) x . 1, two points of intersection: 1 1 1 1 þ fX (x) ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2p 2p x x 1 px x2 1 These are the same results obtained in Example 12.2.10.
Example 12.3.6 (See Example 12.2.11) " 1 fQ (u) ¼ 2p 0
Here g(u) ¼ cos(u) and fQ(u) is given by for p , u p
#
otherwise
The three regions of x are (a) x 2 1, (b) 21 , x 1, (c) x . 1. 1. g(u) satisfies the restrictions. 2. The solutions for x ¼ cos(u) are u1 ¼ 2cos21(x) and u10 ¼ cos21(x).
pffiffiffiffiffiffiffiffiffiffiffiffiffi
3.
dg(u)
d
¼ cos (u) ¼ j sin (u)j ¼
1 x2
du du 4. fX(x) can now be found in the three regions. (a) x 21, no points of intersection: fX (x) ¼ 0: (b) 21 , x 1, two points of intersection: 1 1 1 1 þ fX (x) ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2p 2p 1x p 1 x2
(12:3:10)
(c) x . 1, no points of intersection: fX (x) ¼ 0: These are the same results obtained in Example 12.2.11.
Example 12.3.7 This example will have relevance in the next section. Here g(x) and fX(x) are given by 2
0
6 4 2 þ ln g(x) ¼ 6 4 5x 0
3
if
x1
if
7 1 , x , 57 5
if
x5
"1 fX (x) ¼
if
1,x5
#
4 0 otherwise
The functions g(x) and fX(x) are shown in Fig. 12.3.4. 1. g(x) satisfies the restrictions. 2. There are no points of intersection in the region y 2. Solving for y ¼ g(x) in the region y . 2, we obtain y ¼ 2 þ ln
4 4 ; y 2 ¼ ln 5x 5x
or 5 x ¼ 4e( y2)
200
Functions of a Single Random Variable
FIGURE 12.3.4
or x ¼ 5 4e( y2)
for
y.2
4
¼ 1 ¼ 4e( y2) ; 3. dg(x) ¼ d 2 þ ln
dx dx 5x 5x
y.2
4. Since there are no points of intersection of y with g(x) in the region y 2, it follows that fY ( y) ¼ 0. In the region y . 2, 1 fY ( y) ¼ 4e( y2) ¼ e( y2) 4
for y . 2
B 12.4 INVERSE PROBLEM: FINDING g(x) GIVEN fX(x) AND fY ( y) In the previous sections we studied finding fY ( y) given fX(x) and g(x). In this section we will find g(x) given fX(x) and fY ( y). We will use the conservation of probability measure given by an equation similar to Eq. (12.3.3): fY (h)Dh ¼ fX (j1 )Dj1 þ fX (j2 )Dj2 þ þ fX (jn )Djn
(12:4:1)
Taking the limit as Dh, Dj1, Dj2, . . . , Djn ! 0, all the jk coalesce to j, and integrating h between 21 to y and j between 21 to x, we obtain ðy ðx fY (h)dh ¼ fX (j)dj (12:4:2) 1
1
The lefthand side of Eq. (12.4.2) is a function of y, the righthand side is a function of x, and the equation can be solved for y, which yields g(x) as the transforming function. Example 12.4.1 Given fX (x) ¼ 2e2(x2) u(x 2) and fY ( y) ¼ 14 ½u( y 2) u( y 6), we have to find the transforming function g(x). The functions fX(x) and fY ( y) are shown in Fig. 12.4.1.
12.4
Inverse Problem: Finding g(x) Given fX(x) and fY ( y)
201
FIGURE 12.4.1
Integrating both fX(x) and fY ( y), we obtain the following: ðy ðy 1 y2 dh ¼ for fY (h)dh ¼ 4 4 2 2 ðx ðx fX (j)dj ¼ 2e2(j2) dj ¼ 1 e(x2) for 2
2,y6 x.2
2
Thus y2 ¼ 1 e(x2) 4
for
2 , y 6 and x . 2
or y ¼ g(x) ¼ 6 4e(x2)
for
x.2
The function g(x) is graphed in Fig. 12.4.2. Using the same techniques as in the previous section, we can show that the transformation function g(x) indeed gives a uniform distribution in (2,6], given that fX(x) is an exponentially distributed function starting at y ¼ 2. Example 12.4.2 (See Example 12.3.7) We will revisit Example 12.3.7, where fX(x) is uniformly distributed in the interval (1,5] and fY ( y) is exponentially distributed as
FIGURE 12.4.2
202
Functions of a Single Random Variable
e 2( y22) starting at y ¼ 2. We have to find the transforming functions. Proceeding as in the previous example, we have ðy ðx 1 x1 (h2) dj or 1 e( y2) ¼ , y . 2, x . 1 e dh ¼ 4 4 2 1 or e( y2) ¼
5x , 4
y . 2, x . 1
Taking logarithm on both sides, we obtain 5x 4 ( y 2) ¼ ln or y ¼ g(x) ¼ 2 þ ln , 4 5x
x.1
g(x) is exactly the same function that we used in Example 12.3.7 and is shown in Fig. 12.3.4.
B 12.5 MOMENTS OF A FUNCTION OF A RANDOM VARIABLE Expectation
We will now find the expectation of the random variable Y ¼ g(X ). We have already found the density function fY ( y). We can use the usual definition of expectations and write ð1 mY ¼ E½Y ¼ y fY ( y)dy (12:5:1) 1
However, if only the expected value of Y, and not the density function fY ( y) is required, we can find the result from the knowledge of fX(x) and g(x). Using the law of conservation of probability measure [Eq. (12.3.2)], we have ð1 ð1 E½Y ¼ y fY ( y)dy ¼ y fX (x)dx (12:5:2) 1
1
Substituting y ¼ g(x) in Eq. (12.5.2), we obtain the equation for E[Y ] in terms of the density function fX(x): ð1 E½Y ¼ g(x)fX (x)dx (12:5:3) 1
If the random variable is of the discrete type, then X X E½Y ¼ yi P( y ¼ yi ) ¼ g(xi ) P(x ¼ xi ) i
(12:5:4)
i
Variance
In a similar manner, the variance of Y ¼ g(X ) can be defined as ð1 2 2 sY ¼ E½Y mY ¼ ( y mY )2 fY ( y) dy ð1 ¼ 1
1
( y mY )2 fX (x)dx ¼
ð1 1
½g(x) mY 2 fx (x)dx
(12:5:5)
12.5
Moments of a Function of a Random Variable
203
Higher-Order Moments
The rth moment of Y ¼ g(X ) can be defined as follows: mr ¼ E½Yr ¼ ð1
ð1
( y)r fY ( y)dy
1 r
ð1
( y) fX (x)dx ¼
¼ 1
½g(x)r fX (x)dx
(12:5:6)
1
Example 12.5.1 We will take Example 12.2.7 and calculate the expected values by using the density functions fY ( y) and fX(x). Function fX(x) is uniformly distributed in (22, 2] as shown in Fig. 12.2.10. The density function fY ( y) for that example is given by 1 3 4 fY ( y) ¼ d( y þ 1) þ u y u( y) 2 8 3 and g(x), fX(x), and fY ( y) are shown in Fig. 12.5.1. Using Eq. (12.5.1) to obtain E[Y ], we have
1 3 4 d( y þ 1) þ u y u( y) dy 2 8 3 1 ð 4=3 1 3 1 3 16 1 y dy ¼ þ ¼ ¼ þ 2 8 2 8 9:2 6 0 ð1
E½Y ¼
y
FIGURE 12.5.1
204
Functions of a Single Random Variable
Using Eq. (12.5.3) to obtain E[Y ], we have ð ð2 1 0 2 x dx E½Y ¼ 1 dx þ 4 2 03 2 1 2 4 1 1 1 ¼ þ ¼ þ ¼ 4 4 3 2 2 3 6 This result could just as well have been obtained by inspection of g(x) by adding the areas of the rectangle (22) and the triangle 43 and dividing them by 14, yielding 16. All three of these methods yield exactly the same result. We will now compute the variance of Y from fY ( y) using Eq. (12.5.5): ð1 1 2 1 3 4 d( y þ 1) þ u y yþ var½Y ¼ u( y) dy 6 2 8 3 1 2 ð 4=3 1 5 3 1 2 25 91 83 yþ ¼ ¼ þ dy ¼ þ þ 2 6 6 72 216 108 0 8 Computing var[Y ] from fX(x) and using var[Y ] ¼ E[Y ]2 2 m2Y, we have ð ð 1 0 1 2 2 2 1 2 x dx var½Y ¼ (1) dx þ 4 2 4 0 3 36 ¼
2 1 32 1 83 þ ¼ 4 4 27 36 108
and the two results are the same. Example 12.5.2 We are given fX(x) and fY ( y), and we have to find g(x) and the expectation and variance of Y using fY ( y) and fX(x): 2 2 3 "1 # 3y 2,x6 ; fY ( y) ¼ 4 8 0 , y 2 5 fX (x) ¼ 4 0 otherwise 0 otherwise Integrating fX(j) from 2 to x and fY (h) from 0 to y, we have ðy 2 ðx 1 3h dj ¼ dh 4 2 0 8 1 y3 (x 2) ¼ ; 2 , x 6, 0 , y 2 4 8 Hence, y ¼ g(x) ¼ ½2(x 2)1=3 . We also note the following: Expectation with fY ( y): ð2 y
E½y ¼ 0
3y2 3 dy ¼ 2 8
Expectation with fX(x): ð6 E½ y ¼
1 21=3 3 ½2(x 2)1=3 dx ¼ 3 41=3 ¼ 4 2 4 2
12.5
Moments of a Function of a Random Variable
205
FIGURE 12.5.2
Variance with fY ( y): ð2 var½ y ¼ 0
3 y 2
2
3y2 3 dy ¼ 20 8
Variance with fX(x), using var[Y ] ¼ E[Y ]2 2 m2Y: ð6 1 9 3 9 3 var½ y ¼ ½2(x 2)2=3 dx ¼ 82=3 ¼ 4 4 5 4 20 2 Both methods give the same results. Indicator Functions
An indicator function IA(j) is a real-valued function defined on the sample space S, having a value 0 or 1 depending on whether the j point is in the event A , S. Or 1 j[A (12:5:7) IA (j) ¼ 0 jA The indicator function is a unit step for sets. It is shown in Fig. 12.5.2. The indicator function is a zero – one random variable, and if P{j [ A} ¼ p P{j A} ¼ 1 p ¼ q
(12:5:8)
then the expected value and the variance of the indicator function IA are E½IA ¼ 1 p þ 0 q ¼ p var½IA ¼ 12 p p2 ¼ pq
(12:5:9)
Properties of Indicator Functions 1. A , B () IA IB 2. IA ¼ IA2 ¼ ¼ IAn 3. Is ¼ 1: sample space 4. IAB ¼ min (IA , IB ) 6. IA ¼ 1 IA : A ¼ complement of A
(12.5.10)
CHAPTER
13
Functions of Multiple Random Variables
B 13.1 FUNCTION OF TWO RANDOM VARIABLES, Z 5 g(X,Y ) In the previous chapter we discussed a single function of a single random variable. Here we will find the probability fZ(z) of a single function Z of two random variables X and Y defined by the joint probability density function fXY(x, y). The analysis is very similar to that in the previous chapter. Following similar analysis given in Chapter 12, we have Z ¼ g(X, Y ) as a function that maps the domain space of Z given by fCxy,FXY,PXYg into the range space fRz,FZ ,PZg, which is a real line, as shown in Fig. 13.1.1 {Z ¼ g(X,Y)} : {Cxy ,FXY ,PXY } ! {Rz ,FZ ,PZ }
(13:1:1)
and the mapping of fX,Y g is from the probability space fS, F, Pg (where, S is the product space S ¼ SX SY) into the domain space of Z given by fCxy,FXY,PXYg X,Y: {S,F,P} ! {Cxy ,FXY ,PXY }
(13:1:2)
Alternately, if Dxy , Cxy and Iz , Rz, then Iz is the image of Dxy under the transformation g and Dxy is the inverse image of Iz under the transformation g – 1. Hence Iz is the set of all points (x,y) belonging to Dxy such that g(x,y) belongs to Iz. This is represented by Dxy ¼ {x,y : g(x,y) [ Iz }
(13:1:3)
We also have conditions similar to those in the last chapter for Z to be a random variable. Conditions on Z 5 g(X,Y) to Be a Random Variable 1. The domain of g(x,y) must include the range of the RV Z. 2. For each z [ Iz, fg(x,y) zg must form a field FZ; that is, it must consist of the unions and intersections of a countable number of intervals. 3. The probability of the event fg(x,y) ¼ +1g is 0.
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
206
Function of Two Random Variables, Z 5 g(X,Y )
13.1
207
FIGURE 13.1.1
In essence, the transformation Z ¼ g(X,Y ) induces a probability measure PZ on FZ , and we have to find this probability measure. From Fig. 13.1.1, we can write PZ (Z) ¼ P(Z [ Iz ) ¼ PXY (g1 (Z) [ Dxy ) ¼ PXY (X,Y [ Dxy )
(13:1:4)
The set Iz consists of all the countable unions and intersections of the type a , Z b. Without loss in generality, we will assume that Iz ¼ fZ zg, and substituting for Dxy from Eq. (13.1.3) in Eq. (13.1.4), we have PZ (Z) ¼ PXY ½(X,Y) [ Dxy ¼ PXY ½x,y:g(x,y) [ Iz ¼ PXY ½x, y:g(x,y) z ¼ FXY (z) ¼ P(Z z) ¼ FZ (z)
(13:1:5)
Equation (13.1.5) can also be written as ð
ð
PZ (Z) ¼ PXY ½(X,Y) [ Dxy ¼ FZ (z) ¼
fXY (j,h)dj dh
(13:1:6)
Dxy
where fXY (x,y) is the joint density of the random variables X and Y. In conclusion, we can say that the probability of Z z is given by the probability of the set of all values of x and y such that g(x,y) z, or by integrating the joint density fXY (x,y) over the domain Dxy. We will use both these interpretations in solving problems. Summary of Steps for Finding FZ(z) 5 P(Z z) 1. Graph the function g(x,y) in the x– y plane. 2. Equate g(x,y) ¼ z 3. Find Dxy, the distinct regions in the x– y plane such that g(x,y) z. 4. Find FZ(z) from Eq. (13.1.5) or (13.1.6). Example 13.1.1 Given Z ¼ g(X,Y ) ¼ min(X,Y ), we will find FZ(z) given the joint distribution FXY(x,y). We follow the scheme outlined above. 1. g(x,y) is shown in Fig. 13.1.2. 2. g(x,y) ¼ z is marked on the diagram. 3. The region Dxy is the set of all points in the x –y plane such that either x z or y z. It is shown by the region ABHCDF in Fig. 13.1.2.
208
Functions of Multiple Random Variables
FIGURE 13.1.2
4. The probability of Dxy is given by Pf(X z) or (Y z)g, which can be written as Pfrectangle ABEFg þ Pfrectangle GCDFg 2 Pfrectangle GHEFg. We subtract Pfrectangle GHEFg since it occurs twice in the addition of the first two terms. Hence FZ (z) ¼ FY (z) þ FX (z) FXY (z,z)
(13:1:7)
If X and Y are independent, then Eq. (13.1.7) becomes FZ (z) ¼ FY (z) þ FX (z) FX (z)FY (z)
(13:1:8)
and the density function fZ(z) is given by fZ (z) ¼ fY (z) þ fX (z) fX (z)FY (z) FX (z)fY (z)
(13:1:9)
We will define the joint density function fXY (x,y) in the region f0 , x 4g, f0 , y 2g as shown in Fig. 13.1.3 and find the distribution FZ(z).
FIGURE 13.1.3
13.1
Function of Two Random Variables, Z 5 g(X,Y )
209
The joint density fXY(x,y) and the distribution FXY(x,y) are shown below: 2 0 for x 0, y 0 61 fXY (x,y) ¼ 4 for 0 , x 4, 0 , y 2 ; 8 0 for x . 0, y . 0 2 0 for x 0, y 0 6 xy for 0 , x 4, 0 , y 2 6 68 6x FXY (x,y) ¼ 6 6 4 for 0 , x 4, y . 2 6y 6 for x . 4, 0 , y 2 4 2 1 for x . 0, y . 0 1. In region I, for fz 0g, FZ(z) ¼ 0. 2. In region II, f0 , z 2g or f0 , x 2g, f0 , y 2g we have the following formula from the equation derived earlier: FZ (z) ¼
z z z2 z þ ¼ (6 z) 4 2 8 8
for
0,z2
3. In region III, f2 , z 4g or f2 , x 4g, f2 , y 4g FZ (z) ¼ FX (z) þ FY (2) FXY (z,2) ¼
z z þ1 1¼1 4 4
for
2,z4
The same result can also be obtained from the fact that the random variable Y is always less than 2. Since the region Dxy is either X z or Y z, and z . 2, we have FZ(z) ¼ 1. 4. In region IV, fz . 4g or fx . 4g, fy . 4g, FZ(z) ¼ 1. The density function fZ(z) obtained by differentiating FZ(z) is shown below. 2 0 for z 0 63 z 6 for 0 , z 2 fZ (z) ¼ 6 64 4 for 2 , z , 4 40 0 for z . 4 Example 13.1.2 Given Z ¼ g(X,Y ) ¼ max(X,Y ), we will find FZ(z) given the joint distribution FXY(x,y). We follow the scheme outlined above. 1. g(x,y) is shown in Fig. 13.1.4. 2. g(x,y) ¼ max(x,y) ¼ z is marked on the diagram. 3. The region Dxy is the set of all points in the x –y plane such that x z and y z. This region is shown in Fig. 13.1.4. 4. The probability of Dxy is given by Pf(X,Y z)g, which can be written as FZ (z) ¼ FXY (z,z)
(13:1:10)
210
Functions of Multiple Random Variables
FIGURE 13.1.4
If X and Y are independent, then Eq. (13.1.10) becomes FZ (z) ¼ FX (z)FY (z)
(13:1:11)
and the density function fZ(z) is given by fZ (z) ¼ fX (z)FY (z) þ FX (z)fY (z)
(13:1:12)
We will use the same joint density function fXY(x,y) in the region f0 , x 4g, f0 , y 2g as shown in Fig. 13.1.3 and find the distribution FZ(z). 1. In region I, for fz 0g, FZ(z) ¼ 0. 2. In region II, f0 , z 2g or f0 , x 2g, f0 , y 2g, FZ(z) is given by FZ (z) ¼
z2 8
for
0 , x 2, 0 , y 2
3. In the region III, f2 , z 4g or f2 , x 4g, f2 , y 4g we have z 2 z FZ (z) ¼ FXY (z, 2) ¼ ¼ 4 2 4
for
2 , x 4, 2 , y 4
We can also integrate the joint density function fXY(x,y) in the ranges f0 , x zg and f0 , y 2g: ðz ð2 FZ (z) ¼ 0
1 dy dx ¼ 08
ðz
2 z dx ¼ 8 4 0
for
2 , x 4, 2 , y 4
4. In region IV, fz . 4g or fx . 4g, fy . 4g, FZ(z) ¼ 1.
13.1
Function of Two Random Variables, Z 5 g(X,Y )
211
The density function fZ(z) obtained by differentiating FZ(z) is shown below. 2 0 for z 0 6 z for 0 , z 2 6 64 fZ (z) ¼ 6 1 6 for 2 , z , 4 4 4 0 for z . 4 Example 13.1.3 edge of fXY(x,y).
Given that Z ¼ g(x,y) ¼ X þ Y, we have to find FZ(z) from the knowl-
1. The function g(x,y) is shown in Fig. 13.1.5. 2. The equation x þ y ¼ z is shown in Fig. 13.1.5. 3. Dxy is shown as the shaded region in the figure. 4. The distribution function FZ(z) is fZ (z) ¼ fX (z)FY (z) þ FX (z)fY (z)
(13:1:13)
The density function fZ(z) is obtained by differentiating FZ(z), and the result is ð 1 ð zx ð1 ð zx d d fZ (z) ¼ fXY (x,y)dy dx ¼ f (x,y)dy dx dz 1 1 1 dz 1 (13:1:14) ð1 ¼ fXY (x,z x)dx 1
If X and Y are independent in Eq. (13.1.14), then ð1 fX (x)fY (z x)dx fZ (z) ¼
(13:1:15)
1
Equation (13.1.15) is a convolution integral, and we can use characteristic functions to write FZ (v) ¼ FX (v)FY (v) fZ(z) can obtained from FZ(v).
FIGURE 13.1.5
(13:1:16)
212
Functions of Multiple Random Variables
Example 13.1.4 We are given X and Y as independent random variables with Z ¼ g(X,Y ) ¼ 2X 2 12Y. The density functions fX(x) and fY( y) are given by fX (x) ¼ 2e2x u(x);
1 fY (y) ¼ ey=2 u(y) 2
We have to find fZ(z). Substituting U ¼ 2X and V ¼ 12Y, we can obtain the corresponding densities of U and V from Eq. (12.3.7): 1 u fU (u) ¼ fX ¼ eu u(u); fV (v) ¼ 2fY (2v) ¼ ev u(v) 2 2 With this substitution, Z can be given by Z ¼ U þ (2V ). Since U and V are also independent, we can use Eq. (13.1.16) to find fZ(z). The characteristic functions of U and (2V ) are FU (v) ¼
1 1 ; FV (v) ¼ 1 jv (1 þ jv)
Using the independence property, we obtain 1 1 1 1 ¼ þ FZ (v) ¼ FU (v)FV (v) ¼ (1 jv)(1 þ jv) 2 1 jv 1 þ jv Taking the inverse transform, we obtain 1 1 fZ (z) ¼ ½ez u(z) þ ez u(z) ¼ ejzj 2 2 The functions fX(x), fY( y), and fZ(z) are shown in Fig. 13.1.6.
FIGURE 13.1.6
13.1
Example 13.1.5
Function of Two Random Variables, Z 5 g(X,Y )
213
Two independent random variables X and Y are distributed normally as
2 1 fX (x) ¼ pffiffiffiffiffiffiffiffiffiffiffiffi e(1=2)½(x2)=3 ; 2p 3
2 1 fY (y) ¼ pffiffiffiffiffiffiffiffiffiffiffiffi e(1=2)½(y3)=4 2p 4
Another random variable Z is given by Z ¼ X 2 Y. We have to find fZ(z). The characteristic functions of X and (2Y ) are given by FX (v) ¼ e jv
2
½ðv2 9Þ=2
; FY (v) ¼ ejv3½ðv
2
16Þ=2
Using the independence of X and Y, we have 2 2 3 2 2 FZ (v) ¼ e jv ½ðv 9Þ=2 e jv ½ðv 16Þ=2 ¼ ejv½ðv 25Þ=2 and fZ(z) is also normally distributed with mean 21 and variance 25, and is given by 2 1 fZ (z) ¼ pffiffiffiffiffiffiffiffiffiffiffiffi e(1=2)½(x1)=5 2p 5
Hence, the probability of a sum of independent Gaussian random variables is also a Gaussian random variable with mean equaling the sum of means, and variance equaling the sum of variances.
Example 13.1.6 This example illustrates when the two random variables X and Y are not independent but are given by a joint Gaussian density function: ( ) 1 1 2 2 fXY (x,y) ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi exp pffiffiffiffiffiffiffiffiffiffiffiffiffi ½x 2rxy þ y (13:1:17) 2p 1 r2 2 1 r2 where the correlation coefficient is r and the random variable Z ¼ X þ Y. Here we can use Eq. (13.1.14) only. Substituting fXY (x,y) into Eq. (13.1.14), we have ð1 fZ (z) ¼
fXY (x,z x)dx 1
1 2 2 ½x 2rx(z x) þ (z x) dx exp 2(1 r2 ) 1 ð1 1 1 2 2 2 ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi ½2x exp 2rxz þ 2rx þ z 2zx dx 2(1 r2 ) 2p 1 r2 1 ð1 1 1 2 2 ½2x ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi exp (1 þ r) 2xz(1 þ r) þ z dx 2(1 r2 ) 2p 1 r2 1 ð1 1 2 2 z2 =2(1r2 ) p ffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ e½(1þr)=(1r )½x xz dx e 2 2p 1 r 1 ð1 1 2 2 2 e½1=(1r)½x xz dx ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi ez =2(1r ) 2 2p 1 r 1
1 ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi 2p 1 r2
ð1
(13:1:18)
214
Functions of Multiple Random Variables
Completing squares under the integral sign in Eq. (13.1.18), we have ð1 1 1 h z i 2 z2 z2 =2(1r2 ) x fZ (z) ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi e exp dx 1r 2 4 2p 1 r2 1 ð1 1 1 h z i2 2 2 2 x exp ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi e½z =2(1r )þ½z =4(1r) dx 1r 2 2p 1 r2 1 ( ) ð1 1 z2 =4(1þr) 1 1 x (z=2) 2 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ pffiffiffiffiffiffi e exp dx 2 (1 r)=2) 2p 2pð1 r2 Þ 1 ( ) ð1 1 z2 =4(1þr) 1 1 1 x (z=2) 2 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi exp dx ¼ pffiffiffiffiffiffi e 2 (1 r)=2) 2(1 þ r) 2p 1 r 1 2p 2 |fflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl{zfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflfflffl} ¼1
1 1 2 ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffi ez =4(1þr) 2(1 þ r) 2p
(13:1:19)
Thus, we see that the sum of jointly Gaussian-distributed zero mean, unit variance random variables with correlation coefficient r is also a zero mean Gaussian with variance 2(1þ r). If r ¼0, we have independent Gaussian random variables and variance equals 2, which is the sum of unit variances. If r ¼ 1, we have essentially Z ¼ 2X and the variance is 4. Example 13.1.7 Two independent Gaussian random variables X and Y are with zero mean and variance s2. The probability density functions of X and Y are 1 1 2 2 (13:1:20) fX (x) ¼ pffiffiffiffiffiffi e(x=2s ) ; fY (x) ¼ pffiffiffiffiffiffi e(y=2s ) s 2p s 2p pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi Another random variable Z is given by Z ¼ g(X,Y) ¼ X 2 þ Y 2 . We have to find the pdf pffiffiffiffiffiffiffiffiffiffiffiffiffiffi fZ(z). The region Dxy ¼ x2 þ y2 z is shown in Fig. 13.1.7. Using Eq. (13.1.6), we have ðð
FZ (z) ¼
1 (j2 þh2 Þ=2s2 e dj dh 2 2p s pffiffiffiffiffiffiffiffiffi 2 2 x þy z
We can solve this integral by using polar coordinates using the following substitution: j ¼ r cos u; h ¼ r sin u; dj dh ¼ r dr du;
u ¼ tan1
h j
Thus ð 2p ð z
1 (r2 =2s2 ) e r dr du ¼ 2 0 0 s 2p h i 2 2 ¼ 1 e(z =2s ) u(z)
FZ (z) ¼
ðz
r (r2 =2s2 ) e dr 2 0s (13:1:21)
13.1
Function of Two Random Variables, Z 5 g(X,Y )
215
FIGURE 13.1.7
and fZ (z) ¼
z (z2 =2s2 ) e u(z) s2
(13:1:22)
where u(z) is the usual unit step function. This is a Rayleigh distribution as given in Eq. (7.8.5) or a chi-square distribution with 2 degrees of freedom. Example 13.1.8 This example is similar to the previous example except that Z ¼ g(X,Y) ¼ X 2+Y 2. The region Dxy ¼ x 2+y 2 z. We can now find the pdf fZ(z) using polar coordinates as in the previous example. Thus ð 2p ð pffiz ð pffiz 1 (r2 =2s2 ) r (r2 =2s2 ) e r dr du ¼ e dr FZ (z) ¼ 2 2 0 s 2p 0 s 0 h i 2 ¼ 1 e(z=2s ) u(z) (13:1:23) and the density function fZ(z) is given by 1 z=2s2 e fZ (z) ¼ u(z) 2s2
(13:1:24)
This is an exponential distribution or a chi-square distribution given by Eq. (7.7.3) with 2 degrees of freedom. pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi The density functions for Z ¼ X 2 þ Y 2 and Z ¼ X 2 þ Y 2 given by Eqs. (13.1.22) and (13.1.24) are shown in Fig. 13.1.8 for comparison. Example 13.1.9 In Example 13.1.7 we assumed that the independent random variables X and Y are of zero mean. In this example, we will assume that X and Y have the same variance s2 and that their mean values are mX and mY respectively. The density functions are 1 2 fX (x) ¼ pffiffiffiffiffiffi e½(xmx )=2s ; s 2p
1 2 fY (y) ¼ pffiffiffiffiffiffi e½(ymy )=2s s 2p
(13:1:25)
216
Functions of Multiple Random Variables
FIGURE 13.1.8
We will now find fZ(z) for Z ¼
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X 2 þ Y 2 . Function FZ(z) is now given by
1 (j mX )2 þ (h mY )2 exp dj dh; s2 2p 2s2 pffiffiffiffiffiffiffiffiffi 2 2 ðð
FZ (z) ¼
z.0
(13:1:26)
x þy z
Expanding Eq. (13.1.26) we have, 1 j2 þ h2 2(mX j þ mY h) þ m2X þ m2Y exp FZ (z) ¼ dj dh; s2 2p 2s2 pffiffiffiffiffiffiffiffiffi 2 2 ðð
z.0
x þy z
(13:1:27) Substituting polar coordinates in Eq. (13.1.27), we obtain the following: 2 1 r 2r(mX cos u þ mY sin u) þ m2X þ m2Y exp r dr du 2 2s2 0 0 s 2p 2 2 2 ð e(mX þmY )=s z ð1=2Þðr=sÞ2 e ¼ s2 0 ð 2p 1 r(mX cos u þ mY sin u) exp du r dr; z . 0 2p 0 s2 ð 2p ð z
FZ (z) ¼
(13:1:28)
We now define qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi m2X þ m2Y cos (c); mX ¼ c ¼ tan
1
mY mX
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi m2X þ m2Y sin (c) mY ¼ ð13:1:29Þ
Function of Two Random Variables, Z 5 g(X,Y )
13.1
and a noncentrality parameter m ¼ (13.1.29), we have
217
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi m2X þ m2Y . Substituting m in Eqs. (13.1.28) and
rm( cos u cos c þ mY sin u sin c) e exp du r dr s2 0 0 ð 2p 2 2 ð em =s z (1=2)ðr=sÞ2 1 rm( cos u c) e exp du r dr; z . 0 (13:1:30) ¼ 2p 0 s2 s2 0 2
em =s FZ (z) ¼ s2
2
ðz
(1=2)ðr=sÞ2
1 2p
ð 2p
The integral under u is the modified Bessel function of zero order with argument rm=s2 given by I0
ð 2p rm 1 rm( cos uc)=s2 e du ¼ s2 2p 0
Substituting this value in Eq. (13.1.30) and differentiating with respect to z, we obtain zm z 2 2 2 fZ (z) ¼ 2 e½(z þm )=s I0 2 ; z . 0 (13:1:31) s s This is Rice’s distribution as given in Eq. (7.8.11). With m ¼ 0, we get the Rayleigh distribution. Example 13.1.10 In this example we have Z ¼ g(X,Y ) ¼ XY. The function g(x,y) ¼ xy ¼ z is shown in Fig. 13.1.9 for z . 0. We have to find fZ(z) given the joint density fXY(x,y). The distribution function FZ(z) is obtained by integrating fXY(x,y) in the shaded region given by g(x,y) z. Since the region Dxy ¼ fxy zg is bounded by two disjoint curves, we have to integrate fXY (x,y) along the two regions f –1 , x 0g and f0 , x 1g. Hence ðð FZ (z) ¼ fXY (x,y)dy dx xyz
ð0 ð1
ð 1 ð z=x fXY (x,y)dy dx þ
¼ 1
z=x
fXY (x,y)dy dx 0
FIGURE 13.1.9
1
(13:1:32)
218
Functions of Multiple Random Variables
We differentiate Eq. (13.1.32) using the chain rule of differentiation and obtain fZ(z): ð1 z 1 z 1 fXY x, dx þ fXY x, dx x x x x 1 0 ð1 1 z fXY x, dx; z . 0 ¼ x 1 jxj ð0
fZ (z) ¼
(13:1:33)
It can be shown by similar techniques that Eq. (13.1.33) is valid for z 0. If X and Y are independent, then ð1
z 1 fX (x)fY dx x 1 jxj
fZ (z) ¼
for all z
(13:1:34)
We note that Eq. (13.1.34) is not a convolution integral even if the product of independent random variables is involved. The problem in Example 13.1.10 can also be solved using the techniques developed in Chapter 12. We will first find the conditional density fZ(z j X ¼ x), given the conditional density fY( y j X ¼ x) with Z ¼ xY. Since x is a scaling factor for the random variable Y, we can use Eq. (12.3.7) and obtain fZ (z j X ¼ x) ¼
1 z fY j X ¼ x jxj x
Hence, the unconditional density fZ(z) is given by ð1
1 z fY j X ¼ x fX (x)dx ¼ fZ (z) ¼ x 1 jxj
ð1
z 1 fXY x, dx x 1 jxj
which is the same as before. Example 13.1.11 We will now assume that X and Y are independent random variables with identical Cauchy distributions given by fX (x) ¼
2 2 ; fY (y) ¼ 2 p ð y þ 4Þ þ 4Þ
pðx2
Substituting these values in Eq. (13.1.34), we have ð1 fZ (z) ¼
1 2 2 dx 2 2 þ 4Þ jxj p x ð z 1 p 2þ4 x
Since fZ(z) is an even function, we can rewrite this equation as fZ (z) ¼
42 p2
ð1 0
x dx ðx2 þ 4Þðz2 þ 4x2 Þ
13.1
Function of Two Random Variables, Z 5 g(X,Y )
219
Substituting w ¼ x 2 and dw ¼ 2x dx and expanding into partial fractions, we have 4 p2
ð1
1 dw 2 0 ðw þ 4Þðz þ 4wÞ ð 4 1 1 1 1 ¼ 2 dw p 0 ðz2 16Þ ðw þ 4Þ ðz2 þ 4wÞ
fZ (z) ¼
1
1 4 4 wþ4 2
ln(w þ 4) ln(z þ 4w) 0 ¼ 2 2 ln 2 ¼ 2 2 p ðz 16Þ p ðz 16Þ z þ 4w 0 1
4 wþ4 4 1 4
ln 2 ln ¼ 2 2 ¼ ln 2 p ðz 16Þ z þ 4w 0 p2 ðz2 16Þ 4 z
or fZ (z) ¼
2 4 z ln p2 ðz2 16Þ 16
and fZ(z) is shown in Fig. 13.1.10. Example 13.1.12 Given Z ¼ g(X,Y ) ¼ Y/X, we have to find fZ(z) from the knowledge of the joint density function fXY(x,y). The region Dxy(x,y) ¼ fx,y: y xzg is shown in Fig. 13.1.11. In the two regions of Dxy given by (1) x 0 and (2) x . 0, FZ(z) can be written as follows: ðð FZ (z) ¼
fXY (x,y)dy dx yxz
ð0 ð1
ð 1 ð xz fXY (x,y)dy dx þ
¼ 1 xz
fXY (x,y)dy dx 0
FIGURE 13.1.10
1
(13:1:35)
220
Functions of Multiple Random Variables
FIGURE 13.1.11
Using the chain rule of differentiation in Eq. (13.1.35), we can obtain fZ(z) as ð0
ð1
fZ (z) ¼ ð1
fXY (x,xz)x dx þ
fXY (x,xz)x dx 0
1
jxjfXY (x,xz)dx
¼
(13:1:36)
1
As an illustration, we will find fZ(z) given that X and Y are jointly Gaussian random variables with means 0 and variances s2X and s2Y with correlation coefficient r, given by fXY (x,y) ¼
2 1 1 x xy y2 pffiffiffiffiffiffiffiffiffiffiffiffiffi exp 2r þ 2(1 r2 ) s2X sX sY s2Y 2psX sY 1 r2
Substituting fXY(x,y) into Eq. (13.1.36) and using the fact that fXY(x,y) is an even function we have, 2 pffiffiffiffiffiffiffiffiffiffiffiffiffi fZ (z) ¼ 2psX sY 1 r2
ð1 0
x2 1 z z2 x exp 2r þ dx sX sY s2Y 2(1 r2 ) s2x
1 s2 s2 (1 r2 ) pffiffiffiffiffiffiffiffiffiffiffiffiffi 2 X Y 2 psX sY 1 r2 sY 2rsX sY þ z2 sX pffiffiffiffiffiffiffiffiffiffiffiffiffi sX sY 1 r2 ¼ sY s2Y þ 2 ps2X z2 2rz s X sX
¼
(13:1:37)
By completing the squares in the denominator of Eq. (13.1.37), we can write pffiffiffiffiffiffiffiffiffiffiffiffiffi sX s Y 1 r2 fZ (z) ¼ 2 p s2X z r ssXY þ s2Y ð1 r2 Þ
(13:1:38)
13.1
Function of Two Random Variables, Z 5 g(X,Y )
221
FIGURE 13.1.12
This is a Cauchy distribution centered at z ¼ r(sY/sX) and is shown in Fig. 13.1.12 with s2X ¼ 4, s2Y ¼ 9, and r ¼ 35 : Integrating Eq. (13.1.38) with respect to z, we obtain the distribution function FZ(z) as " # 1 1 1 sX z rsY pffiffiffiffiffiffiffiffiffiffiffiffiffi FZ (z) ¼ þ tan 2 p sY 1 r 2
(13:1:39)
Example 13.1.13 We are given two independent random variables, X and Y, which are exponentially distributed as fX (x) ¼ 2e2x u(x);
fY (x)ey u( y)
We have to find the density function fZ(z) of the random variable given by Z ¼ (Y/X ). From Eq. (13.1.36), we have ð1 fZ (z) ¼
jxjfXY (x,xz)dx 1 ð1
x 2e
¼
2x
e
xz
ð1 dx ¼
0
¼
2xex(2þz) dx
0
2 (2 þ z)2
Integrating fZ (z), we obtain the distribution FZ (z) as FZ (z) ¼ 1 These functions are shown in Fig. 13.1.13.
2 u(z) 2þz
222
Functions of Multiple Random Variables
FIGURE 13.1.13
B 13.2 TWO FUNCTIONS OF TWO RANDOM VARIABLES, Z 5 g(X,Y ), W 5 h(X,Y ) We are given two random variables, X and Y, and their joint density function fXY(x,y). We form the functions Z ¼ g(X,Y ) and W ¼ h(X,Y ) and determine the joint density fZW(z,w) from knowledge of fXY (x,y). The analysis is very similar to the one presented in the previous section. The functions Z ¼ g(X,Y ) and W ¼ h(X,Y ) map the domain space of fZ,Wg given by fCxy, FXY, PXYg into the range space fCzw, FZW, PZWg as shown in Fig. 13.2.1. From Fig. 13.2.1, we obtain Z ¼ g(X,Y), W ¼ h(X,Y);
{Cxy , FXY ,PXY } ! {Czw , FZW , PZW }
(13:2:1)
and the mapping of fX, Y g is from the probability space fS, F, Pg (where S is the product space S ¼ SX SY) into the domain space of fZ,Wg given by fCxy, FXY, PXYg: X,Y : {S, F, P} ! {Cxy , FXY , PXY }
(13:2:2)
Alternately, if Dxy , Cxy and Dzw , Czw, then Dzw is the image of Dxy under the transformation fg,hg and Dxy is the inverse image of Dzw under the transformation {g – 1, h – 1}. Hence, Dzw is the set of all points (x,y) belonging to Dxy such that g(x,y), h(x,y) belongs
FIGURE 13.2.1
13.2
Two Functions of Two Random Variables, Z 5 g(X,Y ), W 5 h(X,Y )
223
to Dzw. This is represented by Dxy ¼ {x,y:g(x,y),h(x,y) [ Dzw }
(13:2:3)
We also have conditions similar to the last section for Z and W to be a random variables. Conditions on Z 5 g(X,Y ) and W 5 h(X,Y ) to Be Random Variables 1. The domain of fg(x,y), h(x,y)g must include the range of the RV Z and W. 2. For each z,w , Dzw, the set fg(x,y) z, h(x,y) wg must form a field FZW; that is, it must consist of the unions and intersections of a countable number of intervals. 3. The probability of the events fg(x,y) ¼ +1g and fh(x,y) ¼ +1g is 0. In essence, the transformation fZ ¼ g(X,Y ), W ¼ h(X,Y )g induces a probability measure PZW on FZW, and we have to find this probability measure. From Fig. 13.2.1 we can write PZW (Z,W) ¼ P(Z,W [ Dzw ) ¼ PXY g1 (Z,W),h1 (Z,W) [ Dxy ) ¼ PXY X,Y [ Dxy
(13:2:4)
The set Dzw consists of all the countable unions and intersections of the type fa , Z bg, fc , W dg. Without loss of generality, we will assume that Dzw ¼ fZ z,W wg. Thus, substituting for Dxy from Eq. (13.2.3) in Eq. (13.2.4), we have PZW (Z,W) ¼ PXY X,Y [ Dxy ¼ PXY ½ x,y:g(x,y),h(x,y) [ Dzw ¼ PXY ½ x,y:g(x,y) z,h(x,y) w ¼ FXY (z,w) ¼ P(Z z,W w) ¼ FZW (z,w)
(13:2:5)
Equation (13.2.5) can also be written as PZW (Z,W) ¼ PXY (X,Y) [ Dxy ¼ FZW (z,w) ¼
ðð fXY (j,h)dj dh
(13:2:6)
Dxy
where fXY(x,y) is the joint density of the random variables X and Y. In conclusion, we can say that the probability of fZ z, W wg is given by the probability of the set of all values of x and y such that fg(x,y) z, h(x,y) wg or by integrating the joint density fXY(x,y) over the domain Dxy. We will use both these interpretations in solving problems. Example 13.2.1 Here we are given that Z ¼ max(X,Y ), W ¼ min(X,Y ) and we have to find FZW(z,w) from the knowledge of the joint density FXY(x,y). Both Z and W are shown in Fig. 13.2.2. The regions min(X,Y ) w ¼ fX wg < fY zg and max(X,Y ) z ¼ fX zg > X,Y ) z ¼ fX zg > fY zg are shown in Fig. 13.2.2. There are two regions of interest: w . z and w z. In the first region w . z, Dxy equals the semiinfinite square defined by the point B ¼ (z, z). Thus FZ,W (z,w) ¼ FXY (z,z) ¼ FZ (z):
w.z
(13:2:7)
224
Functions of Multiple Random Variables
FIGURE 13.2.2
In the second region w z, Dxy equals the semiinfinite square defined by the point B ¼ (z,z) minus the square ABCD. Hence FZ,W (z,w) ¼ FXY (z,z) P(ABCD) ¼ FXY (z,z) ½FXY (z,z) FXY (z,w) FXY (w,z) þ FXY (w,w) ¼ FXY (z,w) þ FXY (w,z) FXY (w,w); w z
(13:2:8)
When w ¼ z, Eq. (13.2.8) is identical to Eq. (13.2.7). We can obtain FW(w) by letting z tend to 1 in Eq. (13.2.8), yielding FZ,W (z,w) ¼ FXY (z,w) þ FXY (w,z) FXY (w,w) ¼ FX (w) þ FY (w) FXY (w,w) ¼ FW (w) which is similar to Eq. (13.1.7) derived in the previous section. If X and Y are independent, then Eqs. (13.2.7) and (13.2.8) become w.z FX (z)FY (z): FZ,W (z,w) ¼ FX (z)FY (w) þ FX (w)FY (z) FX (w)FY (w): w z
(13:2:9)
(13:2:10)
Example 13.2.2 To continue the previous example, the density functions of two independent random variables X and Y are given by fX (x) ¼ 2e2x u(x);
fY (y) ¼ ey u(y)
Hence the joint density function fXY(x,y) is fXY (x;y) ¼ 2eð2xþyÞ u(x)u(y) Given random variables Z ¼ max(X,Y ) and W ¼ min(X,Y ), we have to find the joint distribution FZW(z,w) and the marginal distributions FZ(z) and FW(w). The joint distribution FXY(x,y) can be obtained by integrating fX(x) and fY( y): FXY (x,y) ¼ 1 e2x ð1 ey Þu(x)u(y) ¼ 1 e2x ey þ e(2xþy) u(x)u(y)
13.2
Two Functions of Two Random Variables, Z 5 g(X,Y ), W 5 h(X,Y )
Using Eq. (13.2.10), we obtain 8 1 e2z ð1 ez Þ; z . 0, > < 2z w 2w z ð1 e Þ þ 1 e ð1 e Þ 1e FZW (z,w) ¼ > : 1 e2w ð1 ew Þ; z . 0, z . 0, 1 e2z ez þ e3z ; ¼ 2z z (2zþw) (zþ2w) 3w 1e e þe þe e ; z . 0,
225
w.z wz w.z wz
as the joint distribution function FZW(z,w). The marginal distribution FZ(z) is obtained by letting w ¼ z in the second equation of FZW(z,w) since w cannot exceed z. Thus FZ (z) ¼ 1 e2z ez þ e3z ,
z.0
which is the same as the first equation. To obtain FW(w), we let z tend to 1 in the second equation of FZW(z,w) and obtain FW (w) ¼ 1 e3w ,
w.0
Example 13.2.3 We are given Z ¼ X 2 þ Y 2 and W ¼ Y/X, where X and Y are zero mean independent Gaussian random variables with joint density function fXY (x,y) ¼
1 ½(x2 þy2 )=2s2 e 2ps2
The region Dxy is the set of all values of x,y such that x 2 þ y 2z and y/x w and is shown in Fig. 13.2.3. The joint distribution FZW(z,w) is found by integrating FZW(z,w) over the shaded region of the figure. ðð FZW (z,w) ¼
1 ½(x2 þy2 )=2s2 e dx dy 2 Dxy 2ps
FIGURE 13.2.3
226
Functions of Multiple Random Variables
pffiffi pffiffi pffiffi pffiffi Using polar coordinates, x ¼ z cos (u), y ¼ z sin (u), dx dy ¼ z d z du, we can rewrite the preceding equation as ð pffiz
ðu FZW (z,u) ¼ 2
(p=2) 0
1 (z=2s2 ) pffiffi pffiffi e z d z da 2ps2
with the factor 2 appearing since there are two shaded regions. This last equation can be simplified as follows: ð 1 p z 1 (z=2s2 ) pffiffi dz uþ z pffiffi e p 2 0 s2 2 z 1 u p p 2 þ ¼ 1 eðz=2s Þ ; z . 0, , u 2 p 2 2
FZW (z,u) ¼
Since w ¼ (y/w) ¼ tan(u), we can substitute u ¼ tan – 1(w) and obtain the marginal distributions FZ(z) and FW(w) as follows: 2
FZ (z) ¼ 1 e(z=2s ) ;
z . 0;
FW (w) ¼
1 tan1 (w) þ ; 1 , w , 1 2 p
The corresponding density functions are obtained by differentiation: fZ (z) ¼
1 (z=2s2 ) e ; z . 0; 2s2
fW (w) ¼
1 ; 1 , w , 1 p(1 þ w2 )
The functions FW(w) and fW(w) are Cauchy-distributed and are shown in Figures 13.2.4 and 13.2.5. The examples given above are some of the simple problems to solve with transformations of multiple functions of multiple random variables. In general, this method of solving for distribution functions of two functions of two random variables is more difficult. We now analyze alternate methods of finding the joint density functions.
FIGURE 13.2.4
13.3
Direct Determination of Joint Density fZW(z,w ) from fXY (x,y )
227
FIGURE 13.2.5
B 13.3 DIRECT DETERMINATION OF JOINT DENSITY fZW(z,w ) FROM fXY (x,y ) It is easier to obtain fZW(z,w) directly from the joint density function fXY(x,y) exactly as we did in the case of a single function of a single random variable. We are given the transformation Z ¼ g(X,Y ) and W ¼ h(X,Y ), and the inverse images g – 1(Z,W ) and h – 1(Z,W ) exist. However, corresponding to the differential parallelogram DDzw in the Z –W P plane, there may be several differential unconnected parallelograms DDxy ¼ dxkdyk, k ¼ 1, . . . , n in the X– Y plane as shown in Fig. 13.3.1. These differential regions are connected by the relation dz dw ¼ dx1 dy1 jJ(x1 ,y1 )j þ þ dxn dyn jJ(xn ,yn )j where jJ(x,y)j is the determinant of the Jacobian matrix J(x,y), defined by 2 3 @z(x,y) @z(x,y) 6 @x @y 7 7 J(x,y) ¼ 6 4 @w(x,y) @w(x,y) 5 @x @x
FIGURE 13.3.1
(13:3:1)
(13:3:2)
228
Functions of Multiple Random Variables
The joint density fZW(z,w) can be obtained from fXY(x,y) as follows: fZW (z,w)dz dw ¼ fXY (x1 ,y1 )dx1 dy1 þ þ fXY (xn ,yn )dxn dyn
(13:3:3)
Substitution of Eq. (13.3.1) in Eq. (13.3.3) results in fZW (z,w)dz dw ¼ ¼
fXY (x1 ,y1 )dz dw fXY (xn ,yn )dz dw þ þ jJ(x1 ,y1 )j jJ(xn ,yn )j n X fXY (xk ,yk )dz dw jJ(xk ,yk )j k¼1
(13:3:4)
n X fXY (xk ,yk ) jJ(x k ,yk )j k¼1
(13:3:5)
or fZW (z,w) ¼
From this derivation it is clear that certain restrictions have to be satisfied for g(x,y) and h(x,y) if we have to determine fZW(z,w) directly from fXY(x,y): 1. All the partial derivative of g(x,y) and h(x,y) must exist. 2. There cannot be any planar solutions to g(x,y) ¼ z and h(x,y) ¼ w. We can now formulate the methodology of finding fZW(z,w) from fXY(x,y). Steps for Direct Determination of fZW(z, w) 1. Check whether the restrictions on g(x,y) and h(x,y) are satisfied. 2. Find the n real solutions to the system of equations g(x,y) ¼ z;
h(x,y) ¼ w
and express them in terms of z and w. Let the n solutions be fxk, yk, k ¼ 1, . . . , ng. 3. Find the Jacobian determinant jJj for each xk,yk, and express this also in terms of z and w. 4. Find the density function fZW(z,w) from Eq. (13.3.5): fZW (z,w) ¼ Example 13.3.1
n X fXY (xk ,yk ) jJ(x k ,yk )j k¼1
The functions Z ¼ g(X,Y ) and W ¼ h(X,Y ) are defined by g(X,Y) ¼ X þ Y; h(X,Y) ¼ X Y
where X and Y are independent positive random variables with joint density fXY (x,y) given by 6e – (3xþ2y) . u(x)u( y). We have to find fZW(z,w). 1. All partials exist for g(x,y) ¼ x þ y and h(x,y) ¼ x y. 2. In the equations x þ y ¼ z and x y ¼ w, we note that z . 0 and w can have negative values. Solving for x and y results in one real solution: 1 1 1 zþw zw z x or x ¼ ; y¼ ¼ 1 w y 2 1 2 2 Since x and y are positive, we must have z . jwj and z . w. 3. The Jacobian determinant is jJj ¼ 2.
13.3
Direct Determination of Joint Density fZW(z,w ) from fXY (x,y )
229
FIGURE 13.3.2
4. The joint density function fZW(z,w) from Eq. (13.3.5) is given by h zþw 1 z wi þ2 fZW (z,w) ¼ 6 exp 3 ¼ 3e½(5zþw)=2 ; z . 0, z , w z 2 2 2 Finding the marginal densities fZ(z) and fW(w) are a little tricky because of finding the limits of integrations with respect to z and w. ðz 3e½(5zþw)=2 dw; z . 0, z , w z fZ (z) ¼ z
¼ 6 e2z e3z u(z) Function fZ(z) is shown in Fig. 13.3.2. Since w can take positive and negative values, we have the following integrals for fW(w) for the two regions w . 0 and w 0. In the region w . 0, we have ð1 fW (w) ¼ 3e½(5zþw)=2 dz; w . 0 w
6 ¼ e3w u(z) 5 and in the region w 0, we have ð1 fW (w) ¼
3e½(5zþw)=2 dz;
w0
w
¼
6 2w e u(w) 5
The function fW(w) is shown in Fig. 13.3.3. The distribution function FW(w) is obtained by integrating fW(w) and is given by 8 3 > < (e2w ) w0 FW (w) ¼ 5 > : 3 þ 2 (1 e3w ) w . 0 5 5 It is shown in Fig. 13.3.4.
230
Functions of Multiple Random Variables
FIGURE 13.3.3
Example 13.3.2 Two independent positive random variables X and Y are jointly exponentially distributed with density function fXY(x,y) ¼ 6e 2(3x þ 2y) u(x)u( y), exactly as in the previous example. Two other random variables are Z and W, given by, Z ¼ 2X=(X þ Y) and W ¼ X þ Y. We have to find the joint density fZW(z,w) and the marginal densities fZ(z) and fW(w). 1. All partials exist for g(x,y) ¼ 2x=(x þ y) and h(x,y) ¼ x þ y. 2. Solving for 2x=(x þ y) ¼ z and x þ y ¼ w, we have x ¼ zw=2 y ¼ ½1 (z=2)w. Since x and y are positive, w . 0 and 0 , z 2. 3. The Jacobian matrix is 2
2y 4 J ¼ (x þ y)2 1
3 2x (x þ y)2 5 1
and the determinant is j Jj ¼
2 2 ¼ xþy w
FIGURE 13.3.4
and
13.3
Direct Determination of Joint Density fZW(z,w ) from fXY (x,y )
231
4. The joint density function fZW(z,w) from Eq. (13.3.5) is given by nh zw w z io fZW (z,w) ¼ 6 exp 3 þ 2(1 )w ¼ 3wew½2þ(z=2) ; w . 0, 0 , z 2 2 2 2 Integrating over all w, we obtain fZ(z) ð1 fZ (z) ¼ 3wew½2þðz=2Þ dw ¼ 0
12 ; (z þ 4)2
0,z2
and the corresponding distribution function FZ(z) is 8 0: > > z0 < 4 : 0,z2 FZ (z) ¼ 3 1 > zþ4 > : 1: z.2 The functions fZ(z) and FZ(z) are shown in Fig. 13.3.5. Integrating over all z, we obtain fW(w) ð2 fW (w) ¼ 3w ew½2þðz=2Þ dz ¼ 6(e2w e3w );
w.0
0
and this result is the same one obtained in the previous example and shown in Fig. 13.3.2. Integrating fW(w), we obtain FW(w): FW (w) ¼ 1 þ 2e3w 3e2w ;
w.0
Example 13.3.3 We will find fZW(z,w) from fXY(x,y) for Example 13.2.3, where Z ¼ 2 2 2 X 2 þ Y 2 and W ¼ Y=X and the joint density fXY (x,y) ¼ (1=2ps2 )e½(x þy )=2s . 1. All partials exist for g(x,y) ¼ x2 þ y2 and h(x,y) ¼ y=x. 2. We solve x2 þ y2 ¼ z and y=x ¼ w. Substituting y ¼ wx in x2 þ y2 ¼ z, we obtain rffiffiffiffiffiffiffiffiffiffiffiffiffiffi rffiffiffiffiffiffiffiffiffiffiffiffiffiffi z w2 z 2 2 and y ¼ + x 1 þ w ¼ z or x ¼ + 1 þ w2 1 þ w2
FIGURE 13.3.5
232
Functions of Multiple Random Variables
or
2 0rffiffiffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffi1 3 2 z w z @ A; z . 0, 1 , w , 1 7 , 6 6 7 1 þ w2 1 þ w2 (x1 , y1 ) 6 7 ¼6 7 rffiffiffiffiffiffiffiffiffiffiffiffiffiffi! rffiffiffiffiffiffiffiffiffiffiffiffiffiffi (x2 , y2 ) 6 7 2 z w z 4 5 , ; z . 0, 1 , w , 1 2 2 1þw 1þw
3. The Jacobian matrix is given by
2
2x J¼4 y 2 x
3 2y 15 x
and the Jacobian determinant
y2 jJj ¼ 2 1 þ 2 ¼ 2(1 þ w2 ) x
4. The joint density fZW(z,w) is given by 1 1 (z=2s2 ) 1 ðz=2s2 Þ e þ e fZW (z,w) ¼ 2(1 þ w2 ) 2ps2 2ps2 or
1 1 (z=2s2 ) e ; fZW (z,w) ¼ p(1 þ w2 ) 2s2
z . 0, 1 , w 1
Integrating this equation with respect to w, the marginal density fZ(z) is ð1 1 dw 2 fZ (z) ¼ 2 e(z=2s ) 2 2s 1 p(1 þ w )
1 1 1 (w)
1 2 tan 2 ¼ 2 e1(z=2s ) ; z . 0 ¼ 2 e(z=2s )
2s p 2s 1 and by a similar integration with respect to z we obtain fW(w) as 1 fW (w) ¼ p(1 þ w2 ) which is a Cauchy density. These results correspond to the results obtained in Example 13.2.3. Example 13.3.4 Two independent zero mean Gaussian random variables X and Y has joint density function fXY(x,y) given by 1 (1=2)½(x2 þy2 Þ=s2 fXY (x,y) ¼ e 2ps2 Two other random variables Z and W are defined by 8 1 Y > > < tan pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X Z ¼ X2 þ Y 2; W ¼ > Y > : p þ tan1 X
if
X.0
if
X0
13.4
Solving Z 5 g(X,Y ) Using an Auxiliary Random Variable
233
The angle W has been defined in this manner so that it covers the region ((p=2), (3p=2) since tan1 (Y=X) covers only the region ((p=2),(p=2)]. We have to find the joint density fZW(z,w) and the marginal densities fZ(z) and fW(w). We apply the usual four steps to find a solution to this problem. pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1. All partials exist for g(x,y) ¼ x2 þ y2 and h(x,y) ¼ tan1 (y=x). pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2. We solve for x2 þ y2 ¼ z and tan1 (y=x) ¼ w for x . 0 and p þ tan1 (y=x) ¼ w ffiffiffiffiffiffiffiffiffiffiffiffiffiffi for x 0. We note that tan(w) ¼ y/x for both x 0 and x . 0. Squaring p x2 þ y2 ¼ z and substituting y=x ¼ tan (w), we can write y2 2 x 1 þ 2 ¼ z2 or x2 1 þ tan2 (w) ¼ z2 or x2 ¼ z2 cos2 (w) x However, for x . 0; (p=2) , w (p=2) and cos (w) . 0 and for x 0; (p=2) , w (3p=2) and cos(w) 0. Hence, the only solutions are x ¼ z cos(w) and y ¼ z sin(w) in the range (p=2) , w (3p=2). 3. The Jacobian matrix and its determinant are given by 2 3 x y pffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 6 x2 þ y2 1 1 x2 þ y2 7 7 J¼6 ¼ 4 y y 5 and jJj ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2 z x þy x2 þ y2 x2 þ y2 4. The joint density fZW(z,w) is given by fZW (z,w) ¼
1 z (1=2)(z2 =s2 ) e ; z.0 2p s2
Integrating with respect to w in the range (p=2) , w (3p=2), we obtain z 2 2 fZ (z) ¼ 2 e(1=2)(z =s ) ; z . 0 s which is a Rayleigh distribution, and the angle w is uniformly distributed in (p=2) , w (3p=2), or fW (w) ¼
1 p 3p ; ,w 2p 2 2
and Z and W are also independent random variables.
B 13.4 SOLVING Z 5 g(X,Y ) USING AN AUXILIARY RANDOM VARIABLE The problem of finding the density function fZ(z) of a function of two random variables Z ¼ g(X,Y ) can be considerably simplified by converting to two functions of two random variables with the use of an auxiliary random variable W ¼ X or Y, provided g(x,y) satisfies the two restrictions of differentiability and no straight-line solutions. Example 13.4.1 We will now solve Example 13.1.10 using an auxiliary random variable. Here g(X,Y ) ¼ Z ¼ XY and we introduce an auxiliary random variable W ¼ h(X,Y ) ¼ X.
234
Functions of Multiple Random Variables
1. The restrictions on g(x,y) ¼ xy and h(x,y) ¼ x are satisfied. 2. Solving for xy ¼ z and x ¼ w, we obtain the real solution, x ¼ w and y ¼ z=w. 3. The Jacobian matrix and its absolute determinant are y x or kJk ¼ jxj ¼ jwj J¼ 1 0 4. The joint density fZW(z,w) is given by fZW (z,w) ¼
z 1 fXY w, jwj w
The function fZ(z) is obtained by integrating this equation with respect to w ð1 z 1 fXY w, dw FZ (z) ¼ w 1 jwj which is the same as Eq. (13.1.33). pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi Example 13.4.2 This is the same as Example 13.1.7, where Z ¼ g(X,Y) ¼ X 2 þ Y 2 . We introduce the auxiliary random variable W ¼ h(X,Y ) ¼ X and proceed to solve it using the techniques of the previous section. The joint density fXY(x,y) is given by fXY (x,y) ¼
1 (1=2)½(x2 þy2 )=s2 e 2ps2
pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1. The restrictions on g(x,y) ¼ x2 þ y2 and h(x,y) ¼ x are satisfied. pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2. Solving for x2 þ y2 ¼ z and x ¼ w,pwe obtain x2 þ y2 ¼ z2 or y2 p ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi z2 x2 , and ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2 the two solutions are x1 ¼ w, y1 ¼ z w and x2 ¼ w, y2 ¼ z2 w2 with 2z , w z for real y1 and y2. 3. The Jacobian matrix and its absolute determinant are 2 3 x y pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffiffiffiffiffiffiffi z2 w2 y 2 2 2 2 4 5 ; x þy x þy J¼ or kJk ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ z x2 þ y2 1 0 4. The joint density function fZW(z,w) is given by z 2 (1=2)(z2 =s2 ) fZW (z,w) ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi e ; z . 0, z , w z z2 w2 2ps2 Integrating with respect to the variable w, we obtain ðz z dw 2 2 1 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ; z . 0 fZ (z) ¼ 2 e(1=2)(z =s ) s p z z2 w2 Since the integrand is even, we can write this equation as ð z (1=2)(z2 =s2 ) 2 z dw pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ; z . 0 fZ (z) ¼ 2 e s p 0 z2 w2 z z 2 2 2 p 2 2 ¼ 2 e(1=2)(z =s ) ; z . 0 ¼ 2 e(1=2)(z =s ) s p2 s which is the same as the result obtained in Eq. (13.1.22).
jwj z
13.4
Solving Z 5 g(X,Y ) Using an Auxiliary Random Variable
235
Example 13.4.3 This is the same as Example 13.1.8, where Z ¼ g(X,Y) ¼ X 2 þ Y 2 . We introduce the auxiliary random variable W ¼ h(X,Y ) ¼ X and proceed to solve it using the techniques of the previous section. The joint density fXY (x,y) is given by fXY (x,y) ¼
1 (1=2)½(x2 þy2 )=s2 e 2ps2
1. The restrictions on g(x,y) ¼ x2 þ y2 and h(x,y) ¼ x are satisfied. 2. Solving for x2pþffiffiffiffiffiffiffiffiffiffiffiffiffi y2 ¼ z and x ¼ w, we obtain from Y 2 ¼ z x2 , the two solutions pffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffi pffiffi x1 ¼ w, y1 ¼ z w2 and x2 ¼ w, y2 ¼ z w2 , with z , w z for real y1 and y2. 3. The Jacobian matrix and its absolute determinant are pffiffiffiffiffiffiffiffiffiffiffiffiffi pffiffi 2x 2y or kJk ¼ 2y ¼ 2 z w2 ; z . 0, jwj z J¼ 1 0 4. The joint density function fZW(z,w) corresponding to the two solutions is given by, 1 2 (1=2)(z=s2 ) e ; fZW (z,w) ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffi 2 2 2ps 2 zw
pffiffi pffiffi z . 0, z , w z
Integrating with respect to the variable w, we obtain ð pffi 1 (1=2)(z=s2 ) z dw pffiffiffiffiffiffiffiffiffiffiffiffiffi ; z . 0 fZ (z) ¼ 2 e pffi 2 2s zp zw pffiffi pffiffi Substituting w ¼ z sin (u) and dw ¼ z cos (u)du, we obtain ð p pffiffi 2 1 z cos (u)du 2 1 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ; z . 0 fZ (z) ¼ 2 e(1=2)(z=s ) 2s p p2 z cos2 (u) ¼
1 (1=2)(z=s2 ) e ; z.0 2s2
which is the same result obtained in Eq. (13.1.24). Example 13.4.4 The random variable Z is given by Z ¼ (X þ Y )2, where X and Y are zero mean independent Gaussian random variables given by the joint density function fXY (x,y) ¼
1 (1=2)½(x2 þy2 )=s2 e 2ps2
We have to find the density function fZ(z) using auxiliary random variable W ¼ X. 1. The restrictions on g(x,y) ¼ (x þ y)2 and h(x,y) ¼ x are satisfied. pffiffi 2. Solving for (x þ y)2 ¼ z and x ¼ w, we obtain two solutions: x1 ¼ w, y1 ¼ z w pffiffi and x2 ¼ w, y2 ¼ z w. 3. The Jacobian matrix and its absolute determinant are pffiffi 2(x þ y) 2(x þ y) J¼ or kJk ¼ 2 z; 1 0
z.0
236
Functions of Multiple Random Variables
4. The joint density fZW(z,w) corresponding to the two solutions is 1 1 (1=2s2 )½w2 þ(pffizw)2 fZW (z,w) ¼ pffiffi e 2 z 2ps2 1 1 (1=2s2 )½w2 þ(pffizw)2 þ pffiffi e ; z.0 2 z 2ps2 We will now simplify this expression. Completing the squares in the first and the second terms on the righthand side of the equation, we obtain pffiffi 1 1 1 2 2 exp ½w þ ( z w) pffiffi 2 z 2ps2 2s2 ( " pffiffi2 #) 1 1 2 z z ¼ pffiffi exp 2 w þ 2 z 2ps2 2s 4 2 pffiffi 1 1 1 2 2 exp ½w þ ( z w) pffiffi 2 z 2ps2 2s2 ( " pffiffi2 #) 1 1 2 z z ¼ pffiffi exp 2 w þ þ 2 2 z 2ps2 2s 4 Hence the joint density fZW(z,w) is given by 1 1 (z=4s2 ) n (1=s2 )½w(pffiz=2)2 o fZW (z,w) ¼ pffiffi e e 2 z 2ps2 pffi (1=s2 )½wþ( z=2)2 þe ; z . 0, 1 , w , 1 We integrate this equation over all w to obtain fZ(z): ð o pffi 1 1 (z=4s2 ) 1 n (1=s2 )½w(pffiz=2)2 (1=s2 )½wþ( z=2)2 fZ (z) ¼ pffiffi e e þ e dw 2 z 2ps2 1 2
pffiffiffiffi 1 1 (z=4s2 ) pffiffiffiffi 1 e(z=4s ) pffiffiffiffi pffiffi ; z . 0 ¼ pffiffi e ½s p þ s p ¼ 2 z 2ps2 2s p z The distribution function FZ(z) is obtained by integrating fZ(z) over z: pffiffi ð z (z=4s2 ) 1 e z pffiffiffi dz ¼ erf FZ (z) ¼ pffiffiffiffi 2s 2s p 0 z Ðz 2 where erf(z) is defined by erf(z) ¼ p2ffiffipffi 0 ej dj and ð pffiffi 1 z ej erf( z) ¼ pffiffiffiffi pffiffiffi dj p 0 j Functions fZ(z) and FZ(z) are shown in Fig.13.4.1. Example 13.4.5 In this example X is a zero mean unit variance random variable given by the density function 1 2 fX (x) ¼ pffiffiffiffiffiffi e(x =2) 2p
13.4
Solving Z 5 g(X,Y ) Using an Auxiliary Random Variable
237
FIGURE 13.4.1
and Y is an independent chi-square random variable with n degrees of freedom whose density is given by Eq. (7.7.2): y(v=2)1 e(y=2) v u(y) 2v=2 G 2 pffiffiffiffiffiffiffiffi The density function of the random variable Z ¼ X Y=v is called the Student-t distribution (Section 7.9) and finds extensive application in statistical analyses. We will use an auxiliary variable W ¼ Y: fY (y) ¼
pffiffiffiffiffiffiffi 1. The restrictions on gðx;yÞ ¼ x= y=v and h(x,y) ¼ y are satisfied. pffiffiffiffiffiffiffi pffiffiffiffiffiffiffiffi 2. Solving for x= y=v ¼ z and y ¼ w, we obtain the solution x ¼ z w=v and y ¼ w. 3. The Jacobian matrix and its absolute determinant are 2
3 1 x p ffiffiffiffiffiffiffi J ¼ 4 y=v 2v(y=v3=2 ) 5 0 1
1 1 or kJk ¼ pffiffiffiffiffiffiffi ¼ pffiffiffiffiffiffiffiffi ; y=v w=v
w.0
4. The density function fZW(z,w) is given by pffiffiffiffiffiffiffiffi w(v=2)1 e(w=2) w=v 2 v u(w) fZW (z,w) ¼ pffiffiffiffiffiffi e(z w=v2) 2p 2v=2 G 2 w(v1)=2 2 e(w=2)½(z =vÞþ1 2 u(w) ¼ pffiffiffiffiffiffi v 2 pv G 2 The density function fZ(z) is found by integrating this equation over w, 1 fZ (z) ¼ pffiffiffiffiffiffi v 2 pv G 2
ð 1 (v1)=2 w 2 e(w=2)½(z =v)þ1 dw 0 2
238
Functions of Multiple Random Variables
This equation can be rewritten by substituting j ¼ ðw=2Þ½(z2 =nÞ þ 1, resulting in ð1 1 fZ (z) ¼ j(v1)=2 ej dj (vþ1)=2 pffiffiffiffiffiffi v z2 0 þ1 pv G 2 v The integral in this equation is a gamma function given by G½(v þ 1)=2. Thus vþ1 (vþ1)=2 G z2 2 fZ (z) ¼ pffiffiffiffiffiffi v 1 þ v pv G 2 which is a Student-t density [Eq. (7.9.2)].
B 13.5 MULTIPLE FUNCTIONS OF RANDOM VARIABLES We can now generalize the results of the previous section to n functions of n random variables. Let {X1 , . . . , Xn } be the n random variables defined by the joint density function fX1 , . . . , Xn (x1 , . . . , xn ). Let the n functions be Z1 ¼ g1 (X1 , . . . , Xn ), Z2 ¼ g2 (X1 , . . . , Xn ), . . . , Zn ¼ gn (X1 , . . . , Xn )
(13:5:1)
We assume that these functions satisfy the restrictions of existence of partial derivatives and no planar solutions. We now solve for g1 (x1 , . . . , xn ) ¼ z1 , g2 (x1 , . . . , xn ) ¼ z2 , . . . , gn (x1 , . . . , xn ) ¼ zn
(13:5:2)
Let the solutions be {x1 , . . . , xn } be expressed as functions of {z1 , . . . , zn }: x1 ¼ h1 (z1 , . . . , zn ), x2 ¼ h2 (z1 , . . . , zn ), . . . , xn ¼ hn (z1 , . . . , zn )
(13:5:3)
Let the Jacobian matrix be given by 2
@g1 6 @x1 J(x1 , . . . , xn ) ¼ 6 4 @gn @x1
... ...
3 @g1 @xn 7 7 @gn 5
(13:5:4)
@xn
and its determinant be expressed in terms of {z1 , . . . , zn }: kJk ¼ kJ(z1 , . . . , zn )k
(13:5:5)
The joint density function fZ1 , . . . , Zn (z1 , . . . , zn ) can now be given in terms of Eqs. (13.5.3) and (13.5.5): fZ1 , . . . , Zn (z1 , . . . , zn ) ¼
fX1 , . . . , Xn (x1 , . . . , xn ) kJ(z1 , . . . , zn )k ¼
fX1 , . . . , Xn ½h1 (z1 , . . . , zn )1 , . . . , hn (z1 , . . . , zn ) kJ(z1 , . . . , zn )k
(13:5:6)
For general functions {gk (x1 , . . . , xn , k ¼ 1, . . . , n}, {hk (z1 , . . . , zn , k ¼ 1, . . . , n} and kJ(z1 , . . . , zn )k are not that easy to solve. However, for linear functions
13.5
Multiple Functions of Random Variables
239
{gk (x1 , . . . , xn , k ¼ 1, . . . , n}, the solutions are not that difficult, as shown by the following example. Example 13.5.1 Three independent zero mean unit variance Gaussian random variables X, Y, and Z are given by the joint density function fXYZ (x,y,z) ¼
1 2 2 2 pffiffiffiffiffiffi e(1=2)(x þy þz ) 2p 2p
Three other random variables U, V, and W are given by 2 3 2 32 3 U 2 2 1 X 4 V 5 ¼ 4 5 3 3 54 Y 5 W 3 2 2 Z We have to find the joint density fUVW (u, v, w) and the marginal densities fU(u), fV(v), and fW(w). The solution to 2 3 2 32 3 u 2 2 1 x 4 v 5 ¼ 4 5 3 3 54 y 5 w 3 2 2 z is 2 3 2 x 0 4y5 ¼ 4 1 z 1
2 1 2
32 3 2 3 3 u 2v 3w 1 54 v 5 ¼ 4 u v þ w 5 4 w u 2v þ 4w
The Jacobian matrix and its determinant are given by 2 3 2 2 1 J ¼ 4 5 3 3 5; kJk ¼ 1 3 2 2 Hence, using Eq. (13.5.6), we obtain the joint density function fUVW(u, v, w) fUVW (u, v, w) ¼
2 2 2 1 1 pffiffiffiffiffiffi e(1=2)½(2v3w) þ(uvþw) þ(u2vþ4w) 1 2p 2p
1 2 2 2 pffiffiffiffiffiffi e(1=2)½(2u 9v 426w þ2uv30vw6wu) 2p 2p 8 2 319 0 u > > = < 1 1B 1 6 7C ½ u v w ½C v ¼ exp @ A 4 5 > > (2p)3=2 jCj1=2 ; : 2 w 2 3 9 19 12 where the matrix C ¼ 4 19 43 27 5and jCj ¼ 1. 12 27 17 ¼
The random variables X, Y, Z are independent, whereas the transformed random variables U, V, W are not independent. Their correlation coefficients can be calculated as follows: 19 rUV ¼ pffiffiffiffiffiffiffiffiffiffiffi ¼ 0:966, 9 43
27 rVW ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ 0:999, 43 17
12 rWU ¼ pffiffiffiffiffiffiffiffiffiffiffi ¼ 0:970 17 9
240
Functions of Multiple Random Variables
By successively integrating with respect to (v,w), (w,u) and (u,v), we obtain the marginal density functions fU(u), fV(v), and fW(w) as ð1 ð1 1 2 2 2 fU (u) ¼ pffiffiffiffiffiffi e½(1=2)(2u þ9v 426w þ2uv30vw6wu) dv dw 2p 2p 1 1 1 2 ¼ pffiffiffiffiffiffi e½(1=2)(u =9) 3 2p ð1 ð1 1 2 2 2 fV (v) ¼ pffiffiffiffiffiffi e½(1=2)(2u þ9v þ26w þ2uv30vw6wu) dw du 2p 2p 1 1 1 2 ¼ pffiffiffiffiffiffi e½(1=2)(v =43) 43 2p ð1 ð1 1 2 2 2 fW (w) ¼ pffiffiffiffiffiffi e½(1=2)(2u þ9v 426w þ2uv30vw6wu) du dv 2p 2p 1 1 1 2 ¼ pffiffiffiffiffiffi e½(1=2)(w =17) 17 2p These results can be verified from the results of Example 13.1.5. Since var(ax) ¼ a 2 var(X ), we have var(U ) ¼ 22 þ 22 þ 12 ¼ 9, var(V ) ¼ 52 þ 32 þ 32 ¼ 43, and var(W ) ¼ 32 þ 22 þ 22 ¼ 17, which checks with the results above.
CHAPTER
14
Inequalities, Convergences, and Limit Theorems
B 14.1
DEGENERATE RANDOM VARIABLES
In Section 10.2 it was stated that if the variance is zero, then the random variable will no longer be random. We can state this concept more fully as follows. If E(X 2 a)2¼0, where a is a constant, then the random variable X equals a with probability 1. We shall show this result as follows. We construct a small hole of nonzero width 21 with 1 . 0 about the value a in the region of integration of x from 21 to 1 as shown in Fig. 14.1.1. We integrate (x2a)2 fX(x) from 21 to 1 except for the hole about a and write ð1 ð E(X a)2 ¼ (x a)2 fX (x)dx (x a)2 fX (x)dx (14:1:1) jxaj.1
1
By substituting the lowest value of 1 for jx –aj in Eq. (14.1.1), we can write ð ð 2 2 2 E(X a) 1 fX (x)dx ¼ 1 fX (x)dx ¼ 12 P(jX aj . 1) jxaj.1
jxaj.1
2
Since E(X a) ¼ 0 by assumption, we must have 12 P(jX aj . 1) ¼ 0: Since 1 is a nonzero constant, we must have P(jX aj . 1 ¼ 0: As a consequence P(jX aj ¼ 0) ¼ 1
and
X ¼ a with probability 1:
(14:1:2)
2
What this result conveys is that if E(X2a) ¼ 0, then almost all realizations of X will be equal to a constant a. The random variable X is called a degenerate random variable. This method of analysis will be useful in showing some of the inequalities that will follow. Example 14.1.1 If the variance s2X ¼ 0, then the random variable X ¼ mX almost certainly. Since variance of X equals E(X mX )2 ¼ 0, then, by Eq. (14.1.2) X ¼ mX with probability 1.
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
241
242
Inequalities, Convergences, and Limit Theorems
FIGURE 14.1.1
B 14.2
CHEBYSHEV AND ALLIED INEQUALITIES
In many problems in communications we are interested in finding bounds on certain performance measures. The idea in using a bound is to substitute a simpler expression for something more complicated, thus gaining some insight into the problem at the expense of accuracy. This process usually requires a fair amount of experience with the results that are to be expected. One of the bounds that we are interested in is the tails probabilities of the type P(X . 1) or P(jXj . 1) (Fig. 14.2.1). Even if we cannot find these probabilities exactly, we should be able to find bounds on these quantities. These bounds are based on partial information about the random variable, such as mean and variance. Chebyshev Bound
The tails probability bound of the random variable X of the form P(jX –mXj , 1) is based on the knowledge of only the mean and the variance of X and is known as the Chebyshev inequality, given by P{jX mX j 1}
s2X 12
or
FIGURE 14.2.1
CB(1) ¼
s2X 12
(14:2:1)
14.2
Chebyshev and Allied Inequalities
243
where 1 is a positive constant .0 and CB(1) is the Chebyshev bound (this result can be shown using an analysis similar to that in the previous section): ð1 ð s2X ¼ (x mX )2 fX (x)dx (x mX )2 fX (x)dx (14:2:2) jxmX j.1
1
By substituting the lowest value of 1 in Eq. (14.2.2), we obtain ð s2X 12 fX (x)dx ¼ 12 P(jx mX j 1)
(14:2:3)
jxmX j.1
and hence Eq. (14.2.1). Geometric Derivation of the Chebyshev Bound We will now derive the Chebyshev bound from a different perspective. We will define a random variable Y ¼ X – mX. We will make use of the indicator functions defined in Eq. (12.5.7). Let A be a set on the real line such that A ¼ f –1, – 1g < f1, 1g. We define an indicator function IA as follows: 1 if Y [ A (14:2:4) IA ¼ 0 if Y A
The indicator function IA( y) is shown in Fig. 14.2.2. Taking the expected value of IA, we have E½IA ¼ 1 P(Y [ A) þ 0 P(Y A) ¼ P(Y [ A) ¼ P(jYj 1) ¼ P(jX mX j 1)
(14:2:5)
We will now bound IA by another random variable Y 2/12 as shown in Fig. 14.2.2. From the figure we can see that IA
Y 2 jX mX j2 ¼ 12 12
FIGURE 14.2.2
(14:2:6)
244
Inequalities, Convergences, and Limit Theorems
Taking expectation on both sides of Eq. (14.2.6), we have E½IA ¼ P(jX mX j 1)
E½Y 2 EjX mX j2 s2X ¼ ¼ 2 12 12 1
(14:2:7)
which is Chebyshev inequality. The bound is E½Y 2 =12 is called the Chebyshev bound. We will use this technique for deriving the Chernoff bound. Alternate Forms of the Chebyshev Bound By rearranging the terms in Eq. (14.2.1), we can rewrite Chebyshev’s inequality in terms of the center probability:
P{jX mX j , 1} 1
s2X 12
(14:2:8)
By substituting 1 ¼ nsX, we can write Eq. (14.2.1) as follows: P{jX mX j nsX }
1 n2
(14:2:9)
Thus, for any random variable X, the probability of a deviation from the mean mX of more than n standard deviations is 1/n 2. For example, in a standard Gaussian if n ¼ 3, 1=n2 ¼ 19 ¼ 0:1111. Chebyshev inequality tells us that the probability of deviation by more than 3 from 0 in a standard Gaussian is 0.1111. In actual cases this probability is 0.0027. We can also rewrite Eq. (14.2.5) analogous to Eq. (14.2.4) as P{jX mX j , nsX } 1
1 n2
(14:2:10)
Generalized Chebyshev Inequality
If f(x) is a convex function (Section 14.6) in the sense that its second derivative is positive for all x, then the inequality in Eq. (14.2.1) can be generalized to P{jXj . 1}
E½f(X) f(1)
(14:2:11)
This can be shown by using techniques similar to those used in demonstrating Chebyshev inequality, applying the fact that f(x) is convex: ð1 ð E½f(X) ¼ f(x)fX (x)dx f(x)fX (x)dx (14:2:12) 1
jxj.1
Using the least value for f(x), namely, f(1) in Eq. (14.2.12), we have ð E½f(X) f(1)fX (x)dx ¼ f(1)P{jXj . 1}
(14:2:9)
jxj.1
and hence Eq. (14.2.11). If f(X ) ¼ (X– mX)2 in Eq. (14.2.11), we get Chebyshev inequality. Example 14.2.1 We will apply the Chebyshev bounds to two different distributions, a Gaussian distribution and a Laplace distribution with zero means and unit variances.
14.2
Chebyshev and Allied Inequalities
245
FIGURE 14.2.3
They are given by 1 2 fN (x) ¼ pffiffiffiffiffiffi eðx =2Þ ; 2p
pffiffi 1 fL (x) ¼ pffiffiffi e 2jxj 2
and shown in Fig. 14.2.3. We will find the tails probabilities P(jXj 1) for both the distributions and the Chebyshev bound, which will be the same for both since they have the same means and variances. Applying Eq. (14.2.1), the Chebyshev bounds are P{jXj 1}
1 12
Thus the Chebyshev bound CB(1) ¼ 1/12. Since we know both the density functions, we can calculate the exact tails probabilities P (jXj 1), which are given by ð ð 1 1 x2 =2 1 1 x2 =2 1 e dx þ pffiffiffiffiffiffi e dx ¼ 1 erf pffiffiffi FN (1) ¼ pffiffiffiffiffiffi 2p 1 2p 1 2 ð 1 pffiffiffiffi ð 1 pffiffiffiffi p ffiffi 1 1 e 2x dx þ pffiffiffi e 2x dx ¼ e 2j1j FL (1) ¼ pffiffiffi 2 1 2 1 where
rffiffiffiffi ð 1 1 2 2 erf pffiffiffi ¼ ex =2 dx p 2 0
The one-sided tails probabilities FN (1) and FL(1) for the Gaussian density and the Laplace density, and the Chebyshev bound CB(1) are shown in Fig. 14.2.4 and tabulated in Table 14.2.1 for 1 ranging from 1.5 to 5. Conclusions
It is clear from Fig. 14.2.4 and Table 14.2.1 that the Chebyshev bound is a very loose one. As 1 increases, the bound becomes somewhat tighter (Fig. 14.2.4). Table 14.2.1 also shows that the Chebyshev bound is the same for both density functions, which have the same mean and variance. In fact, the Chebyshev bound is applicable to all density
246
Inequalities, Convergences, and Limit Theorems
FIGURE 14.2.4
TABLE 14.2.1 1 1.5 1.75 2 2.25 2.5 2.75 3 3.25 3.5 3.75 4 4.25 4.5 4.75 5
Normal FN(1)
Laplace FL(1)
Chebyshev CB(1) ¼ 1/12
0.133614 0.080118 0.0455 0.024449 0.012419 0.00596 0.0027 0.001154 0.000465 0.000177 0.000063 0.000021 0.000007 0.000002 0.000001
0.119873 0.084174 0.059106 0.041503 0.029143 0.020464 0.01437 0.01009 0.007085 0.004975 0.003493 0.002453 0.001723 0.00121 0.000849
0.444444 0.326531 0.25 0.197531 0.16 0.132231 0.111111 0.094675 0.081633 0.071111 0.0625 0.055363 0.049383 0.044321 0.04
functions having the same means and variances. It is useful as a theoretical tool to determine the bounds of the tails probabilities, where only means and variances are known about the density functions.
B 14.3
MARKOV INEQUALITY
There are several different versions of Markov inequality. The usual version is defined for a positive random variable X with a finite mean mX. It states that for a positive 1 . 0 m P(X 1) X (14:3:1) 1
14.3
Markov Inequality
247
This result can be shown by the usual techniques developed in the previous section ð1 ð1 ð1 mX ¼ xfX (x)dx xfX (x)dx 1 fX (x)dx ¼ 1P(X 1) (14:3:2) 0
1
1
and hence Eq. (14.3.1). In Eq. (14.3.1), the quantity mX =1 is called the Markov bound. Equation (14.3.1) can be slightly generalized to include both positive and negative random variables P(jX aj 1)
E(jX aj) 1
(14:3:3)
where a is an arbitrary constant and 1 . 0. Equation (14.3.3) can be further generalized to P(jX aj 1)
E(jX ajk ) 1k
(14:3:4)
where k is a positive integer. This inequality is called the Bienayme inequality. We will derive the value of a for which P(jX– aj 1) is a minimum in the equation P(jX aj 1)
E½jX aj2 12
(14:3:5)
for which we have to minimize E[jX – aj]2 with respect to a: E½jX aj2 ¼ E½X 2 2Xa þ a2 ¼ E½X 2 2aE½X þ a2
(14:3:6)
Differentiating Eq. (14.3.6) with respect to a and setting equal to 0, we have d {E½X 2 2aE½X þ a2 } ¼ 2a 2E½X ¼ 0 da from which we obtain a ¼ E[X ]. Thus we obtain Chebyshev inequality as follows: P(jX mX j 1)
E½jX mX j2 12
(14:2:1)
Example 14.3.1 A materials manager purchases 10,000 memory chips from a company where 3% of the chips are faulty. We have to find the probability that the number of good chips lies between 9675 and 9775. Without loss in generality, we will assume that the chips are independent. Let Xk be the zero – one random variable 1 if the kth chip is good Xk ¼ 0 if the kth chip is bad with the probability of Xk ¼ 1 is 0.97. Thus E[Xk] ¼ 1 0.97 þ 0 0.03 ¼ 0.97 and E[X2k ] ¼ 1 0.97 þ 0 0.03 ¼ 0.97. Hence var[Xk] ¼ 0.97 2 (0.97)2 ¼ 0.0291. If Y is the sum of the random variables fXk, k ¼ 1, . . . , 10,000), then the mean of Y is " # 10,000 10,000 X X Xk ¼ E½Xk ¼ 9700 E½Y ¼ mY ¼ E k¼1
k¼1
With the assumed independence of Xk, the variance of Y is " # 10,000 10,000 X X 2 Xk ¼ var½Xk ¼ 291 var½Y ¼ sY ¼ var k¼1
k¼1
248
Inequalities, Convergences, and Limit Theorems
With a ¼ ð9675 þ 9775Þ=2 ¼ 9725, 1 ¼ 50, and E½Y a2 given by E½Y a2 ¼ E½Y 2 2Ya þ a2 ¼ EY 2 2mY a þ a2 ¼ s2Y þ m2Y 2mY a þ a2 ¼ 291 þ 97002 2 9700 9725 þ 97252 ¼ 916 we can use Eq. (14.3.5) and write P(jY 9725j 50
916 ¼ 0:3664 502
What we want is the complementary event P(jY 9725j 50), and this is given by 1 2 0.3664 ¼ 0.6336. Thus, the probability that the number of good chips will lie between 9675 and 9775 is 63.36%. On the other hand, if we want the number of good chips to lie in the region 9700 + 50, namely, 9650 and 9750, then the probability using Chebyshev inequality is P(jY 9700j 50) ¼ 1 P(jY 9700j 50) ¼ 1
291 ¼ 0:8836 ¼ 88:36% 2500
Example 14.3.2 In a binomially distributed random variable b(k; n, p) ¼ nk pk qnk , we have n ¼ 50 and p ¼ 0.1. Using Markov inequality, we want to find the probability of X . 10 and compare it with the true value. The mean value is np ¼ 5 and 1 ¼ 10. Thus, from Eq. (14.3.1), we have the Markov bound P(X 10)
5 ¼ 0:5 10
and the true value is P(X 10) ¼
50 X
50! 0:1k 0:950k ¼ 0:02454 k!(50 k)! k¼10
This Markov bound is very loose! We will now find the Chebyshev bound from Eq. (14.2.7) with f(x) ¼ x 2 and compare the results. Equation (14.2.7) becomes P{jXj . 1} E½X 2 =12 and E[X 2] ¼ npq þ (np)2 ¼ 29.5. The Chebyshev bound is {X . 10} (29:5=100) ¼ 0:295, which is better than the Markov bound of 0.5, but nowhere close to the true value of 0.02454.
B 14.4
CHERNOFF BOUND
In determining the Chebyshev bound, the indicator function was bounded by a quadratic. In an effort to improve the tightness of the bound, we will use a positive exponential bound the indicator function as shown in Fig. 14.4.1. The expectation of the indicator function yields E½IA ¼ P(jYj 1) ¼ P(jX mX j 1) l(Y – 1)
(14:4:1)
, l . 0, 1 . 0, as shown in Fig. 14.4.1. From The indicator function is bounded by e the figure, IA e l(Y21) and taking expectations on both sides, we obtain E½IA ¼ P(Y 1) E½el(Y1)
14.4
Chernoff Bound
249
FIGURE 14.4.1
or P(Y 1) el1 E½elY ¼ el1 MY (l)
(14:4:2)
where jYj has been replaced with Y since we are considering only the positive bound and MY(l) is the moment generating function of Y. The bound given above will hold for any l . 0. However, we would like to choose l so that e 2l1E[e lY] is a minimum. Differentiating e 2l1E[e lY] with respect to l and setting it equal to zero, we obtain d d l(Y1) E½el(Y1) e ¼E dl dl l¼l0 l¼l0 or
E (Y 1)el0 (Y1) ¼ el0 1 E (Y 1)el0 Y ¼ 0
(14:4:3)
Rearranging terms in Eq. (14.4.3), we obtain an implicit relation for the minimum l0 in terms of the moment generating functions (MGFs):
1¼
E½Yel0 Y MY0 (l0 ) ¼ E½el0 Y MY (l0 )
(14:4:4)
The Chernoff bound is given by P(Y 1) el0 1 MY (l0 )
(14:4:5)
where l0 is given by Eq. (14.4.4). A similar result holds for the bound on the negative side. Example 14.4.1 We will find the bounds on tails probability for the standard Gaussian random variable Y using Chernoff’s formula and compare it to the Chebyshev bound found in Example 14.2.1. The MGF for any Gaussian random variable N is given by MN (t) ¼ etmþ(t MY (t) ¼ et
2
=2
2 2
s =2)
and for a standard Gaussian, Y N(0,1)
250
Inequalities, Convergences, and Limit Theorems
FIGURE 14.4.2
We will now find the optimum l from Eq. (14.4.4): 2
M 0 (l0 ) l0 e(l0 =2) ¼ 1¼ Y 2 MY (l0 ) e(l0 =2)
or l0 ¼ 1
The Chernoff bound from Eq. (14.4.5) is 2
2
P(Y 1) el0 1 el0 =2 ¼ e1 e1
2
=2
¼ e 1
2
=2
¼ ChG(1)
The Chernoff bound, ChG(1), the one-sided Ð 1 x2 =2 Chebyshev bound, CBO(1), and the one-sided normal distribution, FNo (1) ¼ p1ffiffiffiffi e dx are shown in Fig. 14.4.2 and in Table 14.4.1 2p e for 1 ¼ 1.5, . . . , 3. TABLE 14.4.1 1 1.5 1.6 1.7 1.8 1.9 2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3
Normal FNo(1)
Chernoff ChG(1)
Chebyshev CBO(1) ¼ 1/212
0.066807 0.0548 0.044566 0.035931 0.028717 0.02275 0.017864 0.013903 0.010724 0.008197 0.00621 0.004661 0.003467 0.002555 0.001866 0.00135
0.324652 0.278037 0.235746 0.197899 0.164474 0.135335 0.110251 0.088922 0.071005 0.056135 0.043937 0.034047 0.026121 0.019841 0.014921 0.011109
0.222222 0.195312 0.17301 0.154321 0.138504 0.125 0.113379 0.103306 0.094518 0.086806 0.08 0.073964 0.068587 0.063776 0.059453 0.055556
14.5
Cauchy–Schwartz Inequality
251
From the figure and the table we see that for 1 . 2 the Chernoff bound is better than the Chebyshev bound. The crossover point occurs when 1 ¼ 2.075.
B 14.5
CAUCHY–SCHWARTZ INEQUALITY
An important inequality used in communication problems is the Cauchy – Schwartz inequality. Let g(x) and h(x) be two complex functions with finite norm defined by sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð 1
g(x)g (x)dx , 1
kg(x), g(x)l ¼ kg(x)k ¼ 1
sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð
(14:5:1)
1
h(x)h (x)dx , 1
kh(x), h(x)l ¼ kh(x)k ¼ 1
The Cauchy–Schwartz inequality is given by sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð 1 ð1 ð1 2 g(x)h (x)dx jg(x)j dx jh(x)j2 dx 1
1
(14:5:2)
1
We can also state this inequality in terms of the inner product defined by ð1 kg(x), h(x)l ¼ g(x)h (x)dx ¼ kh(x), g(x)l
(14:5:3)
1
and the inequality becomes kg(x), h(x)l kg(x)k kh(x)k
(14:5:4)
If we define two real-valued functions, a(x) and b(x), as a(x) ¼ jg(x)j,
b(x) ¼ jh(x)j
then we have to show that sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð ð
ð1
1
1
1
a2 (x)dx
a(x)b(x)dx 1
b2 (x)dx
(14:5:5)
1
Let us represent a(x) and b(x) in a function space spanned by the basis vectors f1(x) and f2(x). These vectors have the following orthonormal property: ð1 ð1 ð1 2 2 f1 (x)dx ¼ f2 (x)dx ¼ 1; f1 (x)f2 (x)dx ¼ 0 (14:5:6) 1
1
1
Thus a(x) ¼ a1 f1 (x) þ a2 f2 (x);
b(x) ¼ b1 f1 (x) þ b2 f2 (x)
(14:5:7)
where a1,a2,b1,b2 are coordinates in the function space of ff1(x), f2(x)g and the vectors, a ¼ ½a1 ,a2 ; b ¼ ½b1 ,b2 as shown in Fig. 14.5.1.
252
Inequalities, Convergences, and Limit Theorems
FIGURE 14.5.1
We will find the inner product, ka(x), b(x)l ¼ can write ð1
Ð1 1
a(x)b(x)dx. From Eq. (14.5.7) we
ð1 ½a1 f1 (x) þ a2 f2 (x)½b1 f1 (x) þ b2 f2 (x)dx
a(x)b(x)dx ¼ 1
1
½using orthonormal properties of f1 (x) and f2 (x) ð1 ½a1 b1 f21 (x) þ a2 b2 f22 (x) dx ¼ a1 b1 þ a2 b2 ¼
(14:5:8)
1
From Fig. 14.5.1 we can write cos(u) ¼ cos(ua ub ) ¼ cos(ua ) cos(ub ) þ sin(ua ) sin(ub ) ¼
a1 b1 a2 b2 ab þ ¼ jaj jbj jaj jbj jajjbj
(14:5:9)
Since jcos(u)j 1, we have ja bj 1 and ja bj jajjbj (14:5:10) jajjbj ffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi Ð1 Ð1 Substituting for jaj ¼ 1 a2 (x)dx; jbj ¼ 1 b2 (x)dx in Eq. (14.5.10), we obtain Eq. (14.5.5) and hence Eq. (14.5.2). The equality is obtained when we substitute a ¼ Kb in Eq. (14.5.10) or a(x) ¼ Kb(x) and hence g(x) ¼ Kh(x). If g(X ) and h(X ) are real random variables, then we define the norms and the inner product given by Eqs. (14.5.1) and (14.5.3) as kg(X), g(X)l ¼ E½g2 (X) ¼ 2
ð1 1 ð1
kh(X), h(X)l ¼ E½h (X) ¼
g2 (x)f (x)dx , 1 (14:5:11) 2
h (x)f (x)dx , 1
1
ð1 g(x)h(x)f (x)dx ¼ kh(X), g(X)l
kg(X), h(X)l ¼ E½g(X)h(X) ¼ 1
(14:5:12)
14.5
Cauchy–Schwartz Inequality
253
FIGURE 14.5.2
and Cauchy–Schwartz inequality of Eq. (14.5.2) becomes ffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi sffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð 1 ð1 ð1 2 2 g(x)h(x)f (x)dx g (x)f (x)dx h (x)f (x)dx 1
1
(14:5:13)
1
or pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi E½g(X)h(X) E½g2 (X)E½h2 (X) Example 14.5.1 A communication system is shown in Fig. 14.5.2, where s(t) is the input signal and n(t) is a random variable representing input noise, so(t) and no(t) are the output signal and output noise, H( f ) is the transfer function. gi(t) the input signal-to-noise ratio (SNR), and go(t) the output signal-to-noise ratio are given by gi (t) ¼
s2 (t) ; E½n2 (t)
go (t) ¼
s2o (t) E½n2o (t)
The output so(t) is given by the inverse Fourier transform of S( f )H( f ), or ð1 ð1 so (t) ¼ s(t)h(t t)dt ¼ S( f )H( f )e j2pf df 1
(14:5:14)
(14:5:15)
1
and the output noise is characterized by E½n2o (t) ¼
ð1
jH( f )j2 N( f )df
(14:5:16)
1
where N( f ) is the frequency-domain characterization of the input noise n(t). With these characterizations of Eqs. (14.5.15) and (14.5.16), the output SNR go(t) can be given by Ð 1 2 j2pf df s2o (t) 1 S( f )H( f )e ¼ Ð1 go (t) ¼ 2 E½n2o (t) 1 jH( f )j N( f )df
(14:5:17)
The Cauchy–Schwartz inequality of Eq. (14.5.2) is modified to suit the frequency-domain characterization as follows 2 ð 1 ð1 ð1 2 A( f )B ( f )df jA( f )j df jB( f )j2 df 1
1
1
with the equality holding for A( f ) ¼ KB ( f ). We substitute pffiffiffiffiffiffiffiffiffiffi A( f ) ¼ H( f ) N( f )
(14:5:18)
254
Inequalities, Convergences, and Limit Theorems
and S( f )e j2pft B( f ) ¼ pffiffiffiffiffiffiffiffiffiffi N( f )
(14:5:19)
A( f )B( f ) ¼ S( f )H( f )e j2pft
(14:5:20)
Thus
Applying Eqs. (14.5.19) and (14.5.20) in Eq. (14.5.18), the Cauchy–Schwartz inequality becomes 2 ð 1 ð 1 ð1 jS( f )j2 j2pft 2 df (14:5:21) S( f )H( f )e df jH( f )j N( f )df 1 1 1 N( f ) Substituting Eq. (14.5.21) in Eq. (14.5.17), the output SNR is given by ð1 ð1 jS( f )j2 2 df jH( f )j N( f )df 2 so (t) 1 1 N( f ) Ð ¼ g (t) o 1 2 E½n2o (t) 1 jH( f )j N( f )df or ð1 go (t)
jS( f )j2 df 1 N( f )
(14:5:22)
The equality will hold for A( f ) ¼ KB ( f ), or pffiffiffiffiffiffiffiffiffiffi S ( f )ej2pfT H( f ) N( f ) ¼ K pffiffiffiffiffiffiffiffiffiffi N( f )
or Hopt ( f ) ¼ K
S ( f )ej2pfT jN( f )j
(14:5:23)
where we have substituted T the time of observation for t. The maximum output SNR is ð1 jS( f )j2 go, max df (14:5:24) 1 jN( f )j
B 14.6
JENSEN’S INEQUALITY
Convex Functions
Before we enunciate Jensen’s inequality, we want to make clear the idea of a convex function. We are interested in a convex function because convexity guarantees a unique minimum over the interval of convexity. A function h(x) is convex in the interval (a, b) if for every x1 belonging to the interval (a,b), there exists a constant m such that the following inequality is satisfied for all x: h(x) h(x1 ) þ m(x x1 )
(14:6:1)
As seen in Fig. 14.6.1, h(x1 ) þ m(x x1 ) is the equation to a straight line passing tangentially to the point h(x1) and m is the slope of the straight line at the point x1. It is clear from the figure that the inequality is satisfied geometrically. For the sake of completeness, we will give a more formal definition of convexity.
14.6
Jensen’s Inequality
255
FIGURE 14.6.1
A function h(x) is convex over an interval (a,b), if every x1 and x2 belonging to (a,b) and for 0 l 1, the following equation holds. h½lx1 þ (1 l)x2 lh(x1 ) þ (1 l)h(x2 )
(14:6:2)
If this equation is a strict inequality, then h(x) is strictly convex. Geometrically, this equation conveys that if the function h(x) lies below the line segment connecting two points in the interval (a,b), then it is convex. It can also be shown that it is necessary and sufficient that the second derivative d2 h(x)=dx2 0 for all x in the interval (a,b), for Eq. (14.6.2) to be satisfied, in which case h(x) is convex. Example 14.6.1 is nonnegative:
The following functions h(x) are convex because the second derivative
1. x2 , since h00 (x) ¼ 2, positive for all x 2. jxj, since h00 (x) ¼ 0, nonnegative for all x 3. ex , since h00 (x) ¼ ex , positive for all x 4. ln(x), for 0 , x , 1, since h00 (x) ¼ 1=x2 , positive in the range of x 5. x ln(x), for 0 , x , 1, since h00 (x) ¼ 1=x, positive in the range of x Jensen’s Inequality
Jensen’s inequality states that if h(x) is convex and if X is a random variable with EjXj , 1, then E½h(X) h E½X (14:6:3) Substituting E[X ] ¼ x1 in the convexity [Eq. (14.6.1)], we obtain h(x) h E½X þ m x E½X Taking expectations on both sides of Eq. (14.6.4) gives E½h(X) h E½X þ m E½X E½X ¼0 and hence Jensen’s inequality [Eq. (14.6.3)].
(14:6:4)
256
Inequalities, Convergences, and Limit Theorems
Example 14.6.2 Since E[X 2n]¼ E[(X n)2] is a convex function from Example 14.6.1, it follows from Jensen’s inequality that 2 E½X 2n E½X n (14:6:5) and as a consequence 2 E½X 2 E½X
(14:6:6)
which is a restatement of the fact that the variance of a random variable is nonnegative.
B 14.7
CONVERGENCE CONCEPTS
We saw in Section 14.1 that if E(X a)2 ¼ 0, then X ¼ a with probability 1 or X converges to a with probability 1. There are a few other convergence concepts, and we shall make these a little clearer. A machinist wants to make piston rings of diameter a inches to a tolerance limit of +1. Let us assume that the variability of the measurement is s2X. According to the Chebyshev inequality of Eq. (14.2.8), we have P{jX aj , 1} 1
s2X 12
(14:7:1)
If sX is very much smaller than 1, then the observed random variable X is between a 2 1 and a þ 1 is almost certain, and one measurement of X is sufficient. However, if sX is not sufficiently small compared to 1, then the estimate a will not give results of sufficient accuracy. To improve the accuracy of the estimate a, we take n measurements corresponding to n random variables fXi, i ¼ 1, . . . , ng with mean a and noise random variables Wi given by Xi ¼ a þ W i ,
i ¼ 1, . . . , n
where we will assume that the random variables Wi are independent with zero mean and variance s2. We will represent the average of these n random variables by X 1 þ X2 þ þ Xn b X¼ (14:7:2) n where the mean of b X is a and the variance is ns2X and the corresponding Chebyshev inequality is given by P{jX aj , 1} 1
s2X n12
(14:7:3)
We can see from Eq. (14.7.3) that as the number of samples n becomes very large, the probability that the jX aj is less than 1 becomes almost certain even though s2X may not be small compared to 1. Pointwise Convergence
A discrete sequence of random variables {X1 , X2 , . . . , Xn , . . .}
(14:7:4)
converges to a limiting random variable X if and only if for any 1 . 0, however small, we can find a number n0 such that jXn (j) X(j)j , 1
(14:7:5)
14.7
Convergence Concepts
257
for every n . n0 and every j. This type of convergence is also called pointwise or everywhere convergent. This may be all right for a real variable, but it is highly restrictive for a random variable. It is possible that the sequence fXng may converge for j points belonging to a subset of the sample space S. Thus, convergences using probability measures are more useful than this type of convergence. Almost Sure Convergence (a.s.)
A sequence of random variables fXng converges almost surely (a.s.), or almost everywhere (a.e.), or strongly to the random variable X, if for every j point in the sample space S satisfies the following criterion: lim jXn (j) X(j)j ! 0 with probability 1
n!1
(14:7:6)
The criterion can also be written as P(Xn ! X) ¼ 1
as
n ! 1
(14:7:7)
In Eq. (14.7.7) the quantity Xn ! X is to be interpreted as the set of all outcomes j such that the number Xn(j) tends to the number X(j). In some cases the limit X may not be known a priori. In this case the Cauchy criterion of convergence comes handy to define mutual convergence. Cauchy Convergence Criterion
A sequence of random variables fXng on the probability space {S, F, P} is convergent almost surely if the set of points j for which the real sequence {Xn (j)} is convergent with probability 1. Or, in other words P{j [ S : Xnþm (j) Xn (j) ! 0}
as
n ! 1
(14:7:8)
for every m . 0. Using this criterion, the sequence fXng converges mutually almost surely if P{Xnþm Xn } ! 0
as
n ! 1
(14:7:9)
Convergence in Probability (p)
A sequence of random variables fXng converges in probability or weakly converges to the random variable X if for every 1 . 0, however small lim P{jXn Xj 1} ¼ 0
(14:7:10)
lim P{jXn Xj 1} ¼ 1
(14:7:11)
n!1
or equivalently n!1
Using the Cauchy criterion, the sequence fXng converges mutually in probability to X if lim P{jXnþm Xn j 1} ! 0
n!1
(14:7:12)
Convergence in probability is widely used in probability theory. In particular, it is used to determine the weak law of large numbers and the consistency of estimators.
258
Inequalities, Convergences, and Limit Theorems
Convergence in Mean Square (m.s.)
A sequence of random variables fXng converges in the mean square or limit in the mean to the random variable X if lim E{jXnþm Xj2 } ! 0
n!1
(14:7:13)
This is also called quadratic mean convergence. Mutual convergence in the mean square is defined by lim E{jXnþm Xn j2 } ! 0
n!1
(14:7:14)
for all m . 0. Convergence in Distribution
The widest form of convergence is the convergence in distribution. If we denote the distribution functions of Xn and X by Fn(x) and F(x), respectively, then the sequence fXng converges in distribution to the random variable X if lim Fn (x) ! F(x)
n!1
(14:7:15)
for every continuity point x of F(x). However, the sequence fXn(j)g need not converge for any j. Properties of Convergences 1. Almost-sure convergence implies convergence in probability. The converse is not true. 2. Mean-square convergence also implies convergence in probability. Again the converse is not true. 3. If fXng converges in probability, then there exists a subsequence fXnkg of fXng that converges almost surely. 4. Convergences almost surely, probability, and mean square all imply convergence in distribution. Of course, the converse is not true. All the four convergences are shown schematically in Fig. 14.7.1.
FIGURE 14.7.1
14.8
B 14.8
Limit Theorems
259
LIMIT THEOREMS
Weak Law of Large Numbers (WLLN)
Armed with the convergence concepts, we will now explore some limit theorems. Let fXng be a sequence of independent identically distributed random variables with finite means m and finite variances s2. Let Sn be the sum of the n random variables given by Sn ¼ X 1 þ X2 þ þ Xn
(14:8:1)
Let the random variable Zn, called the sample mean, be defined by X1 þ X2 þ þ X n n
Zn ¼
Clearly the expected value of Zn is X1 þ X2 þ þ Xn nm ¼m E½Zn ¼ E ¼ n n and because of the independence of the sequence fXng, the variance of Zn is X 1 þ X2 þ þ Xn ns2 s2 var½Zn ¼ var ¼ 2 ¼ n n n
(14:8:2)
(14:8:3)
(14:8:4)
The weak law of large numbers states that for every 1 . 0 lim P½jZn mj . 1 ! 0
n!1
(14:8:5)
To show this result, we apply Chebyshev’s inequality [Eq. (14.2.1)] P½jZn mj . 1
s2 n12
(14:8:6)
and since 1 . 0, the limit of Eq. (14.8.6) as n tends to 1 is indeed 0. In other words, the probability of the sample mean Zn for a specific n of a sequence fXng of random variables deviating from the true mean m by an arbitrary 1 . 0 becomes vanishingly small as n tends to 1. Or the sample mean Zn converges in probability to the true mean. Example 14.8.1 In the design of a gyroscope for a navigation system we want the probability of the sample drift rate Zn calculated for n production gyros differing from the unknown true drift rate m by more than 0.1 be less than 2%. We want to find the number n of the gyros needed to establish this criterion expressed in terms of the unknown variance s2. Applying Chebyshev inequality given by Eq. (14.2.8), we can write 1 s2 100 . 0:98 P jZn mj , 1 10 n From 1 ð100s2 =nÞ . 0:98 we have 1 0:98 n
100s2 n 100s2 ¼ 5000s2 0:02
If the quality control is very good, s will be small, in which case the number n will be small. However, since we are using Chebyshev inequality that gives very loose bounds,
260
Inequalities, Convergences, and Limit Theorems
we expect n to be overly conservative. We will revisit this example after discussing the central limit theorem in the next section. Example 14.8.2 We will assume that there are a sequence of independent events fAig that will be denoted by the sequence of indicator functions IAi (j), given by
1 if j [ Ai IAi (j) ¼ 0 if j [ Ai Zn ¼
n 1X IA n i¼1 i
We will assume that P(Ai ) ¼ p and P(Ai ) ¼ 1 p for i ¼ 1, . . . , n. Thus E½IAi ¼ p
and
E½IA2i ¼ p
and
E½Zn ¼ p
and
var½Zn ¼
p(1 p) n
var½IAi ¼ p(1 p), i ¼ 1, . . . , n
We can now apply WLLN and write P{jZn pj , 1} 1
p(1 p) n12
Assuming p ¼ 0.4, we will estimate p within 0.1. If n ¼ 500, then this equation becomes P{jZn pj , 0:1} 1
0:4 0:6 8 117 ¼1 ¼ 500 0:01 125 125
What this equation tells us is that if we perform 500 trials, then 468 of these will result in an error of fZn 2 pg of less than 0.1. Strong Law of Large Numbers
The weak law of large numbers given by Eq. (14.8.5) states that the sample mean Zn converges in probability to the true mean m as n tends to 1. The strong law of large numbers states that this convergence is not only in probability but also with probability 1, which is almost surely (a.s.). Or n o (14:8:7) P lim (Zn m) ¼ 0 ¼ 1 (a.s.) n!1
Unlike the weak law of large numbers, which is easy to show with Chebyshev’s inequality, the strong law of large numbers is difficult to show and will not be attempted in this book. Even though Eqs. (14.8.5) and (14.8.7) look similar, they are dramatically different. Equation (14.8.7) states that the sequence of the sample means Zn for every n will tend to the true mean m and will stay close to it. The strong law gives us confidence that sample means calculated from a number of realizations will always tend to the true mean as the number increases and isolated discrepancies become unimportant. Central Limit Theorem (CLT)
Perhaps the most important theorem in statistics is the central limit theorem, which in its most general form states that if we sum a sufficient number of random variables, the resulting probability is a Gaussian. Let fXng be a sequence of n independent
14.8
Limit Theorems
261
identically distributed random variables with mean mX and variance s2X. Their sum Sn is given by Sn ¼ X 1 þ X2 þ þ Xn
(14:8:8)
The mean m and the variance s2 of Sn are E½Sn ¼ m ¼ nmX var½Sn ¼ s2 ¼ ns2X
(14:8:9)
We will now define a normalized random variable Zn with mean 0 and variance 1 as follows: Zn ¼
Sn m Sn nmX pffiffiffi ¼ s sX n
(14:8:10)
The central limit theorem states that as n tends to 1, Zn tends to a standard normal distribution: ðz 1 2 pffiffiffiffiffiffi e(x =2) dx lim P(Zn z) ¼ (14:8:11) n!1 2p 1 We can show this result heuristically by using moment generating functions (MGFs) in Zn ¼
n Sn nmX 1 X pffiffiffi ¼ pffiffiffi (Xi mX ) sX n i¼1 sX n
(14:8:12)
Taking MGF on both sides, we obtain pffiffi Pn (t=sX n) (X m ) tZn i X i¼1 MZn (t) ¼ E(e ) ¼ E e " ¼E
n Y
e
pffiffi (t=sX n)(Xi mX )
#
i¼1
( )n n Y pffiffi pffiffi ¼ E e(t=sX n)(Xi mX ) ¼ E e(t=sX n)(XmX )
(14:8:13)
i¼1
In Eq. (14.8.13) we have used the independence property of fXng and X has the same pdf as Xn. We can now expand the last exponential in the above equation and write h i pffiffi 1 t2 (t=sX n)(XmX ) 2 E e (X mX ) þ hn (X; t) ¼ E 1 þ pffiffiffi (X mX ) þ sX n 2ns2X ¼1þ0þ
t2 þ E½hn (X; t) 2n
where hn(X; t) represents the higher-order terms. Hence
n t2 MZn (t) ¼ 1 þ 0 þ þ E½hn (X; t) 2n
n t2 2 lim MZn (t) ¼ 1 þ ¼ et =2 n!1 2n
(14:8:14)
262
Inequalities, Convergences, and Limit Theorems 2
Since E[hn(X; t)] is of the order of 1/n 2, it is zero in the limit as n tends to 1. The term et =2 is the MGF of a standard Gaussian, and we have shown the central limit theorem. The number n is usually assumed to be 30 for CLT to hold, but as we shall see in the following example, it holds for n much less than 30. Example 14.8.3 Random variables Xi, i ¼ 1,2,3,4 are independent and identically distributed with a uniform distribution in the interval (0,1] with fXi(x) ¼ u(x) 2 u(x 2 1). 1 The mean values are mXi ¼ 12 and the variances are sXi2 ¼ 12 . We form the random variables Y1 ¼ X1 , Y2 ¼ X1 þ X2 , Y3 ¼ X1 þ X2 þ X3 , Y4 ¼ X1 þ X2 þ X3 þ X4 , and their density functions are fY1 (y) ¼ fX1 (y), fY2 (y) ¼ fX1 (y) fX2 (y), fY3 (y) ¼ fY2 (y) fX1 (y), fY4 (y) ¼ fY3 (y) fX1 (y). These densities are given by
fY1 ( y) ¼
1 0
if 0 , y 1 otherwise
8 if 0 , y 1 > : 0 otherwise
8 2 y > > > if 0 , y 1 > > 2 > > > 3 < 2 y þ 3y if 1 , y 2 fY3 ( y) ¼ 2 > > y2 > 9 > > if 2 , y 3 3y þ > > 2 > :2 0 otherwise 8 3 y > > if 0 , y 1 > > 6 > > > > y3 2 > > > þ 2y2 2y þ if 1 , y 2 > < 2 3 3 fY2 ( y) ¼ y 22 > if 2 , y 3 4y2 þ 10y > > 3 >2 > > > y3 32 > > > þ 2y2 8y þ if 3 , y 4 > > 6 3 : 0 otherwise These densities are compared to the corresponding normal densities given by 1 12 fNi ( y) N(mNi , s2Ni ) ¼ N i , i , i ¼ 1,2,3,4 2 12 or 2 1 fN1 ( y) ¼ rffiffiffiffi e½x(1=2) =(1=6) p 6
2 1 fN2 ( y) ¼ rffiffiffiffi e(x1) =(1=3) p 3
2 1 fN3 ( y) ¼ rffiffiffiffi e½x(3=2) =(1=2) p 2
2 1 fN4 ( y) ¼ rffiffiffiffiffiffi e(x2) =(2=3) 2p 3
These densities are shown in all four panels of Fig. 14.8.1. We see that the approximation progressively improves as n increases from 1 to 4, and it is very good when n ¼ 4 (Fig. 14.8.1d).
14.8
Limit Theorems
263
FIGURE 14.8.1
Example 14.8.4 We will revisit Example 14.8.1 in light of the CLT and determine whether we can get a better bound on the number n of gyros needed. In order to use the Gaussian tables, we have to convert the random variable Zn into the standard Gaussian form by substituting pffiffiffi n(Zn m) Zn m Wn ¼ pffiffiffiffiffiffiffiffiffiffiffiffiffiffi ¼ s var(Zn ) 1 0:98 and arrive at We can substitute Wn in P½jZn mj , 10 pffiffiffi n sjWn j 1 P pffiffiffi , 0:98 0:98 or P jWn j , 10s 10 n
Referring pffiffiffi to Gaussian tables, we have the value for a probability of 0.98 is x ¼ 2.3263. Thus, n=10s 2:3263 or n 2:32632 100s2 ¼ 541:17s2 and n 542 s2. This is a much better number than the 5000s2 that we obtained in Example 14.8.1.
CHAPTER
15
Computer Methods for Generating Random Variates
If we have an actual probabilistic system, we can obtain the pdf by analyzing the data from the system. On the other hand, if we are designing a system such as transmission of packets through a communication channel, we must get some idea as to how the system will behave before it is actually implemented. We resort to simulation of the system on a computer. The probability distributions may be known a priori, and we have to generate random data for the given probability distributions. We call such generated values random variates. It is not feasible to develop techniques for generating random variates for every possible distribution function. As it turns out, generating a uniform distribution in the interval (0,1] is fast on a computer. Starting with the uniform distribution, we can generate random variates for any given distribution function by a number of techniques that will be discussed.
B 15.1
UNIFORM-DISTRIBUTION RANDOM VARIATES
Almost every computer will have a uniform number generator built into the system using some form of the modulus function. One such generator uses the linear congruential or power residue method that produces a sequence of integers fxi, i ¼ 0, . . . , m 2 2g. It is given by the recursive relationship xiþ1 ¼ (a xi þ c)mod m
(15:1:1)
where a is a constant multiplier, c is the increment, and m is the modulus function. The initial value x0 is called the “seed.” Equation (15.1.1) can be recursively solved to yield n a 1 x n ¼ an x 0 þ c mod m (15:1:2) a1 All these parameters affect the mean, variance, and cycle length of the sequences. It is called a mixed congruential generator if c = 0 and multiplicative congruential generator if c ¼ 0. The sequence repeats after xm22 and the period of repetition is less than or equal to m 2 1 depending on the parameters.
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
264
15.1
265
Uniform-Distribution Random Variates
FIGURE 15.1.1
We will form two sequences with
Example 15.1.1
1. fxig : x0 ¼ 1, a ¼ 7, c ¼ 0, m ¼ 13 2. fyig : y0 ¼ 1, a ¼ 7, c ¼ 3, m ¼ 13 Using the recursive relationship of Eq. (15.1.1), these two sequences fxig and fyig are shown in Fig. 15.1.1 and Table 15.1.1. As we can see from Fig. 15.1.1 and Table 15.1.1, the repetition period is 13 2 1 ¼ 12. Also changing c from 0 to 3 has radically changed the sequence. We can conclude that almost all random-number generators are not truly random but pseudorandom sequences that have the appearance in the statistical sense of being random. Thus, when we refer to random numbers we really mean pseudorandom sequences where the period is very high. A random-number generator must have the following properties: 1. The numbers generated must have good statistical properties. They must be independent and identically distributed. In other words, the values may not be correlated. 2. The repetition period must be long for any simulation to be useful. 3. Random numbers must be repeatable. For each specified x0, a, c, and m, the generator must produce the same sequence of numbers. 4. Many simulations will require thousands of random numbers, and hence the generation of these numbers must be fast and thus must also have computational efficiency. 5. It must be easy to generate separate sequences of random numbers.
TABLE 15.1.1 i
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
2
1
7
10
5
9
1
10
8
7
Series 1: x0 ¼ 1, a ¼ 7, c ¼ 0, m ¼ 13 xi
1
7
10
5
9
11
12
6
3
8
4
Series 2: x0 ¼ 1, a ¼ 7, c ¼ 3, m ¼ 13 yi
1
10
8
7
0
3
11
2
4
5
12
266
Computer Methods for Generating Random Variates
The linear congruential generator may satisfy all these conditions provided a and m are carefully chosen. Good statistical properties have been obtained with a ¼ 75 ¼ 16,807 and m ¼ 231 2 1 ¼ 2,147,483,647. Under these conditions the period of the sequence is 231 2 2 ¼ 2,147,483,646. Sometimes a ¼ 742,938,285 and m ¼ 248 are also used.
B 15.2
HISTOGRAMS
Having generated random variates, we have to see how well they approximate the desired probability density. From the definition of the density function fX (x), we have fX (x)
P(X x þ Dx) P(X x) Dx
or fX (x)Dx P(x , X x þ Dx)
(15:2:1)
which gives the value of the probability mass at the point x and width Dx. Hence with the generated random variates fxig we will be able to construct a histogram that will approximate the given mass function fX (x)Dx. The purpose of the histogram is to construct the model of the desired distribution from the generated data. The usual form of the histogram is to divide the range of data points N into equal-size bins or subintervals Dx. Or the necessary number of bins K needed will be given by Number of bins K ¼
range of data N bin width Dx
(15:2:2)
Next, the number of data points fni, i ¼ 1, . . . , Kg falling into each bin is counted. The vertical axis (ordinate, y) represents the counts in each bin, and the horizontal axis (abscissa, x ) represents K . Dx, the product of the number of bins K and the bin interval Dx. If fX (x) is finite, then the product K . Dx must cover the entire range of x. If it is infinite or semiinfinite, then the product can cover a reasonable range. Finally, the histogram must be normalized so that it can be compared to the desired density function. The normalization proceeds as follows. The normalized count is the count in each bin fni, i ¼ 1, . . . , Kg, divided by the total number N of observations multiplied by the bin width Dx so that the area under the histogram equals one. This type of normalization results in a histogram that is similar to the probability density function. Even though this may be less intuitive, it is the one to be used if the histogram is to model the desired probability density function. We will now model a uniform distribution U(0,1) from the random variates generated by the linear congruential generator of a computer. For this model, N ¼ 10,000 points were generated and K ¼ 100 bins were chosen with each bin interval Dx ¼ 0.01. Note that K . Dx ¼ 1 covers the x axis of the U(0,1) distribution. The histogram subroutine is called that places the counts fni, i ¼ 1, . . . , Kg in each of the K bins. The number of points falling in the first 10 bins is shown in Table 15.2.1. The expected number in each bin is 100. The resulting histogram is divided by N . Dx ¼ 10,000 0.01 ¼ 100 to yield the model for U(0,1) distribution. The normalized histogram is shown in Fig. 15.2.1. Superimposed over the histogram is the actual uniform distribution. By “eyeballing” the two plots in Fig. 15.2.1, we see that the modeling is indeed good. More formally, we can also test the goodness of fit by a number of mathematical tests, and we will use the chisquare test to ascertain whether the histogram fits the true pdf sufficiently well. We will now enumerate the procedure for the chi-square test as applied to this case.
15.2 Histograms
267
TABLE 15.2.1 Bins Count Expected
0
1
2
3
4
5
6
7
8
9
102 100
100 100
109 100
100 100
117 100
107 100
86 100
93 100
86 100
81 100
Chi-Square Test 1. From the histogram find the number of observations ni in each of the i ¼ 1, . . . , K bins. 2. From the probability density function fX (x) calculate the occupancy mi ¼ fX(xi)NDx in each of the i ¼ 1, . . . , K bins. 3. Calculate the deviation Di from Di ¼ ni mi ; i ¼ 0; . . . ; K 1: PK1 4. Form the chi-square statistic from x ¼ i¼0 (Di =mi )2 . This will have K22 degrees of freedom. 5. Set the probability level pg or the significance level a ¼ 1 2 pg. Usually pg is set at 95% or 99% or a is set at 5% or 1%. The higher the significance level, the tighter will be the bounds. 6. From the chi-square tables find the threshold value ta for the given significance level and K22 degrees of freedom. 7. If the x value is less than ta, the hypothesis that the fit is good is accepted. Otherwise it is rejected. We can use the x2 test to determine the goodness of the fit to the uniform distribution that has been achieved. The x value is found to be x ¼ 91.39 at the 5% significance level. The corresponding ta value for 100 2 2 ¼ 98 degrees of freedom at the 5% significance level is obtained from Table 15.2.2 as ta ¼ 122.10773. The chi-square density and distribution for 98 degrees of freedom is shown in Figs. 15.2.2 and 15.2.3 with the probability and significance levels. The hypothesis that the histogram is a good fit to the true normal distribution is accepted since 91.39 , 122.10773.
FIGURE 15.2.1
268
Computer Methods for Generating Random Variates
TABLE 15.2.2 K
pg 95%
pg 97.5%
pg 99%
K
pg 95%
pg 97.5%
pg 99%
1 2 3 4 5 6 7 8 9 10 12 14 16 18 20 22 24 26 28 30
3.84146 5.99146 7.81473 9.48773 11.07050 12.59159 14.06714 15.50731 16.91898 18.30704 21.02601 23.68475 26.29620 28.86928 31.41042 33.92443 36.41502 38.88513 41.33713 43.77297
5.02389 7.37776 9.34840 11.14329 12.83250 14.44938 16.01276 17.53455 19.02277 20.48306 23.33660 26.11891 28.84532 31.52636 34.16959 36.78070 39.36407 41.92316 44.46079 46.97924
6.63490 9.21034 11.34487 13.27670 15.08627 16.81189 18.47531 20.09024 21.66599 23.20918 26.21693 29.14122 31.99992 34.80530 37.56623 40.28936 42.97982 45.64168 48.27823 50.89218
35 40 45 50 55 60 65 70 75 80 82 84 86 88 90 92 94 96 98 100
49.80185 55.75848 61.65623 67.50481 73.31149 79.08194 84.82064 90.53122 96.21667 101.87947 104.13874 106.39484 108.64789 110.89800 113.14527 115.38979 117.63165 119.87094 122.10773 124.34211
53.20335 59.34170 65.41016 71.42019 77.38046 83.29767 89.17714 95.02318 100.83934 106.62857 108.93729 111.24226 113.54360 115.84143 118.13589 120.42708 122.71511 125.00007 127.28207 129.56120
57.34207 63.69074 69.95683 76.15389 82.29212 88.37942 94.42208 100.42518 106.39292 112.32879 114.69489 117.05654 119.41390 121.76711 124.11632 126.46165 128.80325 131.14122 133.47567 135.80672
Starting from uniformly distributed random variates, we can generate other random variates given the desired probability distribution. We will discuss three widely used techniques for generating these variates: 1. Inverse transformation 2. Convolution 3. Acceptance –rejection In all the simulations that follow we have used ui U(0,1) distribution shown in Section 15.2 with N ¼ 10,001 points.
FIGURE 15.2.2
15.3
Inverse Transformation Techniques
269
FIGURE 15.2.3
B 15.3
INVERSE TRANSFORMATION TECHNIQUES
One of the more useful ways of generating random variates is through the inverse transformation technique, which is based on the following result. If U is a random variable uniformly distributed in (0,1] with distribution function FU (u) ¼ u,u 1(0,1], and X is a random variable with distribution function FX(x), then the random variates for X can be 21 generated from X ¼ F21 X (U ). We will show that FX (U ) is a random variable with distri21 bution FX(x). Assuming that X ¼ FX (U ), we have P{X x} ¼ P{FX1 (U) x}
(15:3:1)
Since FX (x) is monotonically increasing, we can write Eq. (15.3.1) as follows: P{FX1 (U) x} ¼ P{U FX (x)} ¼ FU {FX (x)}
(15:3:2)
However, U is uniformly distributed in (0,1] and hence FU(u) ¼ u, 0 , u 1. Applying this fact in Eq. (15.3.2), we have FU {FX (x)} ¼ FX (x)
(15:3:3)
This result can be used to obtain the random variates for any distribution function FX(x) only if F21 X can be solved in closed form or FX can be accurately and efficiently inverted numerically. The methodology of finding random variates fXig by the inversion technique can be enumerated as follows: 1. FX(x) is known a priori. 2. Uniform random variates fui, i ¼ 1, . . . , Ng in the interval (0,1] are generated. 3. The inverse F21 X (u) of the cdf FX(x) can be analytically or numerically determined. 4. The variates fxig are generated from xi ¼ F21 X (ui), i ¼ 1, . . . , N. 5. The histogram for fxig is constructed and tested using chi-square goodness of fit. We will use this procedure to find random variates for a number of discrete and continuous distributions starting from 10,001 points of the uniform random variates in (0,1] generated in the previous section.
270
Computer Methods for Generating Random Variates
Discrete Distributions
Since FX(x) is a monotonically increasing function, the inverse for a finitedimensional integer-valued discrete distribution functions can be obtained in a simple manner as shown in the following two examples. Example 15.3.1 given by
The distribution of packets in bytes in a communication system is 8 0, > > > > 0:2, > > < 0:4, FX (x) ¼ 0:6, > > > > 0:8, > > : 1,
0 , x 64 64 , x 128 128 , x 256 256 , x 320 320 , x 384 x . 384
and is shown in Fig. 15.3.1. It is not difficult to find the inverse function F21 X (u). From Fig. 15.3.1, for values of the uniform random variate ui given by 0.4 , ui 0.6, F21 X (u) ¼ 256 bytes. Similarly, we can obtain the complete inverse function as 8 64, 0 , ui 0:2 > > > > < 128, 0:2 , x 0:4 xi ¼ FX1 (ui ) ¼ 256, 0:4 , x 0:6 > > 320, 0:6 , x 0:8 > > : 384, 0:8 , x 1 and the sequence fxig represents the random variates for this discrete distribution. The generated uniform random variates ui are assigned values 64, 128, 256, 320, and 384 depending on the ranges in which they fall along the vertical axis. They are then sorted in ascending order and plotted with x ¼ F21 X (u) along the horizontal axis and i scaled by 1/10,000 along the vertical axis so that the range is from 0 to 1. The plot is shown in Fig. 15.3.2. As an example, the uniform random variates the falling in the shaded region in the figure between 0.4 and 0.6 represent 256 bytes.
FIGURE 15.3.1
15.3
Inverse Transformation Techniques
271
FIGURE 15.3.2
Out of the 10,001 uniform random variates generated, 2017 fall in the range 0– 0.2, 1912 fall in the range 0.2– 0.4, 1976 fall in the range 0.4– 0.6, 2017 fall in the range 0.6 –0.8, and 2079 fall in the range 0.8 –1.0 as shown in Fig. 15.3.2, which is exactly the same as Fig. 15.3.1. Example 15.3.2 In this example, we will simulate a binomial distribution FB(x) with n ¼ 5 and p ¼ 0.4 given by FB (x) ¼
x X
5! (0:4)k (0:6)nk k!(5 k)! k¼0
and shown below and graphed in Fig. 15.3.3:
FIGURE 15.3.3
272
Computer Methods for Generating Random Variates
FIGURE 15.3.4
8 0:07776, > > > > 0:33696, > > < 0:68256, FB (x) ¼ 0:91296, > > > > 0:98976, > > : 1,
0,x1 1,x2 2,x3 3,x4 4,x5 x.5
The inverse function F 21 B (U) can be found in a manner similar to that in the previous example. In Fig. 15.3.3, for values of the uniform random variate ui between 0.33696 and 0.68256, F21 B (u) ¼ 2. Similarly, we can obtain the complete inverse function as 8 0, 0 , ui 0:07776 > > > > 1, 0:07776 , ui 0:33696 > > < 2, 0:33696 , ui 0:68256 FB1 (ui ) ¼ 3, 0:68256 , ui 0:91296 > > > > 4, 0:91296 , ui 0:98976 > > : 1, 0:98976 , ui 1 which is shown in Fig. 15.3.4. 1 The vertical axis has been scaled by 10,000 for the range to correspond to (0,1]. Out of the 10,001 uniform random variates generated, 797 fall in the range 0 – 0.07776, 2555 fall in the range 0.07776 –0.33696, 3404 fall in the range 0.33696 – 0.68256, 2312 fall in the range 0.68256 –0.91296, 801 fall in the range 0.91296 – 0.98976, and 132 fall in the range 0.98976 –1. Figure 15.3.4 is the same as Fig. 15.3.3. Continuous Distributions
If the cumulative distribution function of a random variable can be expressed in a closed form, then it is not difficult to find the inverse in a closed form. We will give several examples to clarify the inverse transformation method. Example 15.3.3 (Exponential Distribution) In this example we will obtain random variates for an exponential distribution given by FX (x) ¼ 1 elx ,
x0
(15:3:4)
15.3
Inverse Transformation Techniques
273
FIGURE 15.3.5
The inverse is given by 1 elx ¼ u
or elx ¼ 1 u
and
1 x ¼ FX1 (u) ¼ ln (1 u), l
x0
The exponential random variates are generated from this equation with l ¼ 13, resulting in the following equation: xi ¼ 3 ln (1 ui ), x 0, i ¼ 0, 1, . . . , 10,000 The histogram subroutine hist(K,Dx,x) is called with K ¼ 100 bins and bin width Dx ¼ 0.1. The resulting histogram is divided by N . Dx ¼ 1000 to yield the normalized histogram fE(xk), k ¼ 1, . . . , 100. This is shown in Fig. 15.3.5. Superimposed on top of the histogram is the true density function, fX (x) ¼ 13 e(x=3) , x . 0. We can see from figure that the fit is very good. We can also apply the chi-square goodness of fit, which it easily passes. Example 15.3.4 (Weibull Distribution)
The cdf of a Weibull distribution is given by a
FX (x) ¼ 1 e½l(xm) ,
x.m
(15:3:5)
This distribution, as mentioned in Chapter 7, is a generalization of the exponential with parameters, scale l, shape a, and location m. Since this is in a closed form, the inverse is given by a a 1 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 e½l(xm) ¼ u or e½l(xm) ¼ 1 u and FX1 (u) ¼ m þ a ln(1 u) l with u , 1. The Weibull random variates are generated with l ¼ 13 , a ¼ 3 and m ¼ 2 from the following equation: pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi xi ¼ 2 þ 3 3 ln(1 mi ), i ¼ 0, . . . , 10,000 The histogram subroutine hist(K, Dx, x) is called with K ¼ 100 bins and bin width Dx ¼ 0.1. The resulting histogram is divided by N . Dx ¼1000 to yield the normalized histogram fW (xk), k ¼ 1, . . . , 100. This is shown in Fig. 15.3.6. 3Superimposed on 3top of the histogram is the true Weibull density function, fX (x) ¼ 3 13 (x 2)2 e½(x2)=3 , x . 2.
274
Computer Methods for Generating Random Variates
FIGURE 15.3.6
We can see from the figure that the fit is very good. We can also apply the chi-square goodness of fit, which it readily passes. Example 15.3.5 (Rayleigh Distribution) The random variates corresponding to a Rayleigh distribution can be generated by taking (1) the direct inverse of the distribution or (2) the square root of the sum of the squares of two independent Gaussian distributions. The cdf of a Rayleigh distribution is given by FZ (z) ¼ 1 e(z
2
=2s2 )
, z.0
(15:3:6)
Since this is in a closed form, the inverse is found as follows: 2
2
1 e(z =2s ) ¼ u
or e(z
2
=2s2 )
¼ 1 u and
z ¼ FZ1 (u) ¼
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2s2 ln(1 u)
The Rayleigh random variates are generated from this equation with s2 ¼ 1, resulting in the following equation: pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi zi ¼ 2s2 ln(1 ui ), z . 0, i ¼ 0, 1, . . . , 10,000 The histogram subroutine hist(K, Dx, x) is called with K ¼ 200 bins and bin width Dx ¼ 0.05. The resulting histogram is divided by N . Dx ¼ 500 to yield the normalized histogram fR(xk), k ¼ 1, . . . , 200. This is shown in Fig. 15.3.7. Superimposed on top of the 2 histogram is the true density function, fZ (z) ¼ ze(z =2) , z . 0. We can see from figure that the fit is very good. We can also apply the chi-square goodness of fit, which it easily passes. Special Methods
Even though some distributions may not have a closed form, random variates can be obtained by special transformations. The very important Gaussian random variates can be generated by these special transformation techniques. Two independent random variables U and V are both uniformly distributed in (0,1] and whose joint density is given by 1, 0 , u 1, 0 , v 1 fUV (u,v) ¼ (15:3:7) 0, otherwise
15.3
Inverse Transformation Techniques
275
FIGURE 15.3.7
The transformation used to generate standard Gaussian random variables is pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi X ¼ 2 ln(U) cos(2pV) pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi Y ¼ 2 ln(U) sin(2pV)
(15:3:8)
We can find the joint density fXY (x,y) from the transformation. Solving for u and v in Eq. (15.3.8), we obtain y
1 2 2 tan1 u ¼ e½(x þy )=2 : v ¼ (15:3:9) 2p x The Jacobian matrix of transformation is 2 3 2 cos (2pv) dx dx pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi u 2 ln(u) 6 du dv 7 6 6 ¼ J¼4 dy dy 5 4 sin (2pv) pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi du dv u 2 ln(u)
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2p sin(2pv) 2 ln(u)
3
7 7 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 5 2p cos(2pv) 2 ln(u)
and the absolute value of the Jacobian determinant is pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi cos (2pv) pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2p sin (2pv) 2 ln(u) 2p u 2 ln(u) 2 2 ¼ ¼ 2pe(x þy )=2 kJk ¼ sin (2pv) p ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi u pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2p cos (2pv) 2 ln(u) u 2 ln(u) Hence, from Eqs. (15.3.7) and (15.3.11) we have fUV (u,v) 1 (x2 þy2 )=2 1 (x2 =2) 1 (y2 =2) pffiffiffiffiffiffi e ¼ e ¼ pffiffiffiffiffiffi e fXY (x,y) ¼ kJk 2p 2p 2p
(15:3:10)
(15:3:11)
(15:3:12)
which yields two independent standard Gaussian random variables. This transformation is called the Box– Mueller transformation. Example 15.3.6 (Gaussian Distribution) We will now use the Box – Mueller transformation to generate two independent standard Gaussian random variates. In this example, 10,001 points of two independent uniformly distributed random variates fuig and fvig are generated. They are passed through the transformation of Eq. (15.3.8) to form the
276
Computer Methods for Generating Random Variates
Gaussian random variates fzig and fwig. Theoretically, these must be zero mean and unit variance. If they are not, they can be transformed into exactly zero mean and unit variance by another transformation wi w ; i ¼ 0; . . . , 10;000 sw qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi P P where z ¼ (1=N) Ni¼0 zi and sz ¼ (1=N) Ni¼0 (zi z)2 , N ¼ 10,000, with similar definitions for w and sw . The histogram subroutines are called with K ¼ 100 bins and bin width Dx ¼ Dy ¼ 0.08. The resulting histogram is normalized by NDx ¼ NDy ¼ 800 to yield histograms fXk and fYk, k ¼ 1, . . . , 100 for the Gaussian random variates. plots pffiffiffiffiffiffi Both 2 are shown p inffiffiffiffiffiffi Fig. 15.3.8 with the true standard Gaussian fX (x) ¼ (1= 2p)e(x =2) and 2 fY (y) ¼ ð1= 2pÞe(y =2) superimposed on them. From these figures we find that the fit is very good. These random variates are stored for later use in simulating stochastic systems. xi ¼
zi z ; sz
yi ¼
FIGURE 15.3.8
15.3
Inverse Transformation Techniques
277
Example 15.3.7 (Rayleigh Distribution with Phase) We have already found random variates for a Rayleigh distribution using a direct transformation. We will now apply the relationship to two independent standard Gaussian random variables by pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi Z ¼ X2 þ Y 2 to generate random variates for the Rayleigh distribution. The Gaussian random variates are generated by the Box – Mueller transformation as explained in the previous section. The Rayleigh random variates and the corresponding phase variates are given by pffiffiffiffiffiffiffiffiffiffiffiffiffiffi 9 zi ¼ x2i þ y2i > = , i ¼ 0, . . . , N y i > wi ¼ tan1 ; xi The sequences fzig and fwig are both passed through histogram subroutines with the number of bins K ¼ 200 and the bin width Dz ¼ Dw ¼ 0.05. Both these histograms are normalized with division by NDz ¼ NDw ¼ 500. The resulting histograms fZk and fWk 2 are shown in Fig. 15.3.9 with the true Rayleigh density fZ (z) ¼ ze(z =2) , z . 0 and uniform density U(– p/2, p/2] superimposed on them. We can see from both plots that the fit is very good. Table 15.3.1 shows the densities with their closed-form distribution functions and their inverses.
FIGURE 15.3.9
278
a b aþ1 , x.0 b xþb
b p½(x a)2 þ b2
1 (x2 þy2 )=2 e 2p
Pareto
Cauchy
Gaussian
z.0
z (z2 =s2 ) e , s2
ala (x m)a1 e½l(xm)
Rayleigh
Weibull
a
x.m
=2s2 )
,
b a , xþb
2
1 2p
1 1
ðx ðy
e(j
2
þh2 )=2
dhdj
x.0
z.0
x.m
1 1 1 x a þ tan 2 p b
1
1 e(z
a
1 e½l(xm) ,
1 lx 1 e , x 0 1 elx , 2 2
l ljxj e , 2
Laplace l0
(1 elx )u(x)
x.0
1 x a 2 a,xb 2 ba 1 x þ a 2b 2 b , x 2b a 1 ba 2
xa ba
cdf F(x)
1 lx e u(x) l
1 ba
pdf f (x)
Exponential
Triangular
Uniform (a,b)
Type
TABLE 15.3.1
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1p a ln (1 u), l
u,1
u,1
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2 ln (u1 ) cos (2pu2 ) pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi y ¼ 2 ln (u1 ) sin (2pu2 ) x¼
1 a þ b tan p u 2
1 ffiffiffiffiffiffiffiffiffiffiffi b p 1 , a 1u
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 2s2 ln (1 u), u , 1
mþ
1 1 1 ln (2u), u ln½2(1 u), l 2 l
1 ln (1 u), u , 1 l u.
pffiffiffiffiffi 1 a þ (b a) 2u, u 2 h pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffii 1 b þ (b a) 1 2(1 u) u . 2
a þ u(b a)
Inverse F 21(u)
1 2
15.4
B 15.4
Convolution Techniques
279
CONVOLUTION TECHNIQUES
In cases where it may not be possible to obtain inverses, we can use the convolution technique. From Eq. (13.1.15) we recall that the density of the sum Z of two independent random variables X and Y is the convolution of the densities of X and Y: ð1 fZ (z) ¼
fX (x)fY (z x)dx
(15:4:1)
1
We can generate random variates from independent random variates that have already been generated. We will give two examples. Example 15.4.1 (Triangular Distribution) We want to generate random variates for a triangular distribution. Two independent uniformly distributed random variates, faig and fbig, are generated. We form the sum x i ¼ ai þ bi ,
i ¼ 0, . . . , 10,000
The sequence fxig is passed through the histogram subroutine with the number of bins K ¼ 100 and the bin width Dx ¼ 0.02. The resulting histogram is normalized with NDx ¼ 10,000 0.02 ¼ 200. The histogram fXk is graphed in Fig. 15.4.1. The true triangular density given by fX (x) ¼
x, 2 x,
0,x1 1,x2
is superimposed on the histogram. The fit is very close and passes the chi-square test easily. Example 15.4.2 (Erlang Distribution) The Erlang distribution is obtained by the n-fold convolution of n independent exponential distributions as given in Eq. (7.4.3). Hence we can generate Erlang random variates with n degrees of freedom by summing n independent exponential random variates. We will now generate Erlang random variates with 2 degrees
FIGURE 15.4.1
280
Computer Methods for Generating Random Variates
FIGURE 15.4.2
of freedom. From two sets of uniform random variates faig and fbig, we generate two identically distributed exponential random variates fxig and fyig, with l ¼ 13 given by xi ¼ 3 ln (ai );
yi ¼ 3 ln (bi )
Erlang random variates fzig are formed by summing fxig and fyig. The variates fzig are then passed through the histogram subroutine with number of bins K ¼ 200 and bin width Dz ¼ 0.1. The resulting histogram is normalized by division with NDx ¼ 1000. The histogram fZk is graphed for k ¼ 0, . . . , 200 and shown in Fig. 15.4.2. The true Erlang density for 2 degrees of freedom given by 2 1 fZ (z) ¼ ze(z=3) , 3
z0
is superimposed on the histogram as shown in the figure. The fit is very good as seen in the diagram.
B 15.5
ACCEPTANCE – REJECTION TECHNIQUES
The rejection method does not require that the cumulative distribution function [indefinite integral of p(x)] be readily computable, much less the inverse of that function—which was required for the transformation method in the previous section. Usually the inverse transformation technique uses the cumulative distribution function in closed form, in which case it is the preferred method of generating random variates. However, in many cases the cdf may not be expressible in closed form or it may be very inefficient to calculate it numerically. On the other hand, the density functions may be available in closed form such as the beta or the gamma density. The rejection method is a technique for generating random deviates fxig whose density function fX(x), if known in closed form, is amenable directly to determine these variates. We will now outline the acceptance– rejection method of generating variates. Consider a function g(x) that upper-bounds the density fX(x) or g(x) fX(x) for all x. This is called the bounding or comparison function g(x) which, is not a density function
15.5 Acceptance – Rejection Techniques
281
Ð1 Ð1 since 1 g(x)dx 1 fX (x)dx ¼ 1. We define a normalization constant c ¼ Ð1 1 g(x)dx 1, and another function wX (x) ¼ g(x)=c for all x, thus rendering wX(x) a proper density function. We can now state the acceptance – rejection algorithm: 1. Generate random variates fyi, i ¼ 0, . . . , Ng having the density function wY( y). 2. Generate random variates fui, i ¼ 0, . . . , Ng uniformly distributed in (0,1] and independent of fyig. 3. If ui ½ fX ( yi )=½g( yi ) 1, accept the point yi and set fxi ¼ yig. Otherwise, reject yi. The number of accepted points fxig will be equal to N/c. 4. The sequence fxig are the desired random variates with density function fX(x). We will now show from conditional probability considerations that the generated variates fxig do indeed have the distribution FX(x). From the definition of conditional probability, we obtain P(generated X x) ¼ P(Y x j accept) ¼
P(Y x, acceptÞ P(accept)
(15:5:1)
We need the following result for any y: fX (y) P(accept j Y ¼ y) ¼ P U g(y)
(15:5:2)
Since U is uniformly distributed in (0,1], we have FU(u) ¼ u, and it is independent of Y. Hence Eq. (15.5.2) can be written as follows: fX (y) fX (y) fX (y) P U ¼ FU ¼ (15:5:3) g(y) g(y) g(y) Incorporating Eq. (15.5.3) into Eq. (15.5.2), we obtain P(accept j Y ¼ y) ¼
fX (y) g(y)
(15:5:4)
We can evaluate the denominator in Eq. (15.5.1) from Eq. (15.5.4) with substitution of wY (y) ¼ g(y)=c. ð1 P(accept) ¼ P(accept j Y ¼ y)wY (y)dy 1
ð1 ¼
1
fX (y) g(y) 1 dy ¼ g(y) c c
We will now evaluate the numerator of Eq. (15.5.1): ð1 P(Y x, accept) ¼ P(accept, Y x j Y ¼ y)wY (y)dy
(15:5:5)
(15:5:6)
1
Since Y ¼ y in Eq. (15.5.6) we can rewrite it as ðx P(Y x, accept) ¼ P(accept, y x j Y ¼ y)wY (y)dy 1
(15:5:7)
282
Computer Methods for Generating Random Variates
where the integration is terminated at x because y x. Applying Eq. (15.5.4) and substituting wY (y) ¼ g(y)=c in Eq. (15.5.7), we obtain ðx fX (y) g(y) P(Y x, accept) ¼ dy 1 g(y) c ¼
FX (x) c
(15:5:8)
Combining Eqs. (15.5.5) and (15.5.8), we obtain the result that P(generated X x) ¼
FX (x) c ¼ FX (x) c
(15:5:9)
thus showing that the acceptance –rejection method yields the desired probability distribution. Example 15.5.1 We will give an example of generating beta density variates by the method described above. The general beta density is given by fb (x) ¼
G(a þ b) a1 x (1 x)b1 , G(a)G(b)
0x1
and there is no closed-form solution for the cdf. For a and b integers, we have fb (x) ¼
(a þ b 1)! a1 x (1 x)b1 , 0 x 1 (a 1)!(b 1)!
Random variates will be generated for a beta(3,6) density (Fig. 15.5.1) given by fb (x) ¼ 168x2 (1 x)5 ,
0x1
The maximum of the density function occurs at x ¼ 27 ¼ 0.2857 and fb(27) ¼ 2.55. We can bound this density with a rectangle of width 2.55 so that g(x) ¼ 2.55, 0 ,x 1. Hence the normalizing constant c ¼ 2.55 and wX (x) ¼ g(x)=c is a U(0,1] density function. Algorithm for Acceptance – Rejection 1. Generate a sequence fyi, i ¼ 0, . . . , Ng with uniform density wY (y) ¼ g(y)=c. 2. Generate uniform random variates fui, i ¼ 0, . . . , Ng independent of fyig. 3. If ui ½168y2i (1 yi )5 =2:55; return xi ¼ yi, i ¼ 0, . . . , N. Sequences fyi, i ¼ 0, . . . , 10,000g and fui, i ¼ 0, . . . , 10,000g are generated, and if the condition 3 (above) is satisfied, xi was set equal to yi. The sequence fxig has 3936 points out of the original total of 10,001 points. This corresponds to the expected 10,001/2.55 ¼ 3922 points. The sequence fxig consisting of 3936 points was passed through a histogram subroutine with K ¼ 100 bins and bin width Dx ¼ 0.01. The histogram was normalized by division with 3936 0.01 ¼ 39.36. The normalized histogram is graphed in Fig. 15.5.1 with the true beta(3,6) density superimposed for comparison. The bounding function g(x) is also shown. From the figure it is seen that the fit is good. Discussion
We have discussed three methods for generation of random variates for a given distribution function. The inversion method is used where the cumulative distribution
15.5 Acceptance – Rejection Techniques
283
FIGURE 15.5.1
function can be expressed in a closed form or can be computed efficiently. The convolution method can be used when the density function is expressible as the resultant of a sum of independent random variates. The acceptance –rejection method is used directly on the probability density function. In generating histograms the number of bins K and the bin width Dx have to be manipulated first by “eyeballing” and later confirming that the x value is less than the ta value in the chi-square test at the significance level a. Note also that higher significance levels yield tighter bounds.
CHAPTER
16
Elements of Matrix Algebra
A number of results depend on an understanding of matrices and manipulations of matrices. We will now discuss relevant aspects of matrix algebra. We will assume familiarity with linear vectors spaces [56].
B 16.1
BASIC THEORY OF MATRICES
Definitions
A matrix X is defined as an array of m column vectors {xj , j ¼ 1, . . . , m} with each column vector x.j having n real or complex elements x.j ¼ fx1j, x2j, . . . , xi j, . . . , xn jgT. Thus the matrix X is an n m rectangular array of real or complex numbers and written as 3 2 x1 x2 xj . . . xm 2 36 x11 x12 x1j x1m 7 7 x1 6 7 6 6 x2 76 x21 x22 x2j x2m 7 7 76 (16:1:1) X¼6 7 4 xi 56 6 xi1 xi2 xij xim 7 7 xn 6 5 4 xn1 xn2 xnj xnm Notation: In the double subscript (i j ) the first subscript denotes the row, and the second subscript denotes the column. Thus, an n m array consists of n rows and m columns. Sometimes the matrix X is also written as fxijg. A square matrix is a matrix with equal number of rows and columns. It is denoted as n n square matrix or an n matrix. If the column vectors {xj ¼ ½x1j , x2j , . . . , xnj T , j ¼ 1, 2, . . . , m} are linearly independent, then the matrix X is column-independent. On the other hand, if the row vectors denoted by {xi ¼ ½xi1 , xi2 , . . . , xim , i ¼ 1, 2, . . . , n} are linearly independent, then the matrix X is row-independent. The dots in the column-vector x j and the row-vector x i represent the integers 1, . . . , n. An arbitrary rectangular matrix cannot be both column- and rowindependent unless it is a square matrix. However, not all square matrices need be row-independent and column-independent. Parts of this chapter are based, with permission, on Chapters 30, 31, and 33 from the book Linear Systems Properties—a Quick Reference, by Venkatarama Krishnan, published by CRC Press, 1988.
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
284
16.1
Basic Theory of Matrices
285
Matrix Addition
Addition is defined for two matrices X and Y of similar dimensions: X þ Y ¼ {xij } þ {yij } ¼ {xij þ yij }
(16:1:2)
The addition rule is commutative: X þ Y ¼ Y þ X. Example 16.1.1 "
#
"
1
5 2
3
4 7
3
8
6
11
10
16
,
X¼ "
3 4
8
6 9
#
" ,
Y¼
4 6
#
Z¼ 3 2
# ,
XþY¼
2
X þ Z and Y þ Z are not defined
Matrix Multiplication
Matrix multiplication is a little complex. If we denote X Y ¼ Z, then the number of columns of X must be equal to the number of rows of Y. Otherwise, multiplication is not defined. If X is ? n, then Y has to be n ? In this case the dimension of Z is ? ?. The product X Y is defined by Z¼XY¼
n X
xik ykj ¼ {zij}
(16:1:3)
k¼1
Example 16.1.2 "
3
1 2
2
# ,
X¼ 4 "
5 1
4
6 6 Y ¼ 63 4 1
1
3
7 7 27 5 5
34þ13þ21 31þ12þ25
#
XY¼
"
17 15
#
¼ 44þ53þ11 41þ52þ15 2
32 19 3
2
3
43þ14
41þ15
42þ11
16
9
9
6 6 Y X ¼ 63 3 þ 2 4 4 13þ54
31þ25
7 6 7 6 3 2 þ 2 1 7 ¼ 6 17 5 4 12þ51 23
13
7 7 18 7 5 7
11þ55
26
In general, matrix multiplication is not commutative, that is, XY = YX as shown above. As an example, consider the following symmetric matrices: e f a b , Y¼ X¼ f h b d ae þ bf af þ bh ae þ bf be þ df XY ¼ , YX ¼ be þ df bf þ dh af þ bh bf þ dh
286
Elements of Matrix Algebra
The matrices X and Y commute only if af þ bh ¼ be þ df. We usually denote in the product XY that Y is premultiplied by X or X is postmultiplied by Y. Unlike real numbers, division of one matrix by another matrix is not defined.
Transpose of Matrix
The transpose of a matrix X ¼ fxijg is obtained by interchanging the rows and columns and is denoted by XT ¼ {xij }T ¼ {x ji }
(16:1:4)
The transpose of a matrix product (XY) T is given by Y TX T whereas (X þ Y)T ¼ X T þ Y T. If X ¼ X T, then we say that the matrix X is symmetric.
Example 16.1.3
2 X¼ 1
5 3
27 (XY) ¼ 21 T
4 6
2
3 6 Y ¼ 41 4
30 ; 23
3 2 7 1 5;
2 6 XT ¼ 4 5
3
4
27 30 Y X ¼ 21 23 T
T
2
3 1 3 7 3 5 YT ¼ 2 6
1 1
4 3
Determinant of a Matrix
The nonzero determinant of a square matrix is one of the two most important invariants of a matrix. It is usually denoted by det X or jXj. It is computed as det X ¼
n X
xij Dij ¼ x1j D1j þ x2j D2j þ þ xnj Dnj
(16:1:5)
i¼1
with no implied summation over j and Dij is the cofactor of the matrix X defined by Dij ¼ (1)iþj Mij
(16:1:6)
where Mij is the minor of the matrix X obtained by taking the determinant after deleting the ith row, i ¼ 1, . . . , n, and the fixed jth column. Equation (16.1.5) is called a cofactor expansion about the jth column. The difference between the cofactor and the minor is the sign as shown in Eq. (16.1.6).
Example 16.1.4
The determinant of the following matrix is evaluated as follows: 2 3 2 5 3 2 6 1 2 4 57 7 X¼6 4 4 1 3 2 5 3 4 2 1
16.1
Basic Theory of Matrices
287
Expanding in terms of minors about the second column, the expression for the determinant jXj is 2 3 1 4 2 5 jXj ¼ 5 4 3 2 þ 2 4 3 2 3 3 2 1 2 1 2 2 3 2 3 2 5 þ 1 1 4 5 þ 4 1 4 3 4 3 2 2 1 Each determinants in the above equation can again be expanded in terms of minors about the second column as shown by the expansion of the first term: 1 4 5 1 5 4 2 5 4 3 2 ¼ 5 4 5 3 3 3 1 1 3 2 1 1 5 5 (2) ¼ 350 4 2 Evaluating the determinant 42 þ 376 ¼ 632.
by
this
expansion,
we
have
jXj ¼ 350 252 2
If only the determinant of the matrix is zero, we call that it a singular or simply degenerate matrix. In addition to the determinant, if the first- and higher-order minors are also zero, then we call the matrix multiply degenerate. Properties of Determinants (X and Y Nonsingular) 1. det X ¼ det X T: The following properties (2,3,4,5,6) are very useful in the manipulation of matrices. 2. If any row (or column) of X is multiplied by a constant a to yield a new matrix Y, then det X ¼ a . det Y. 3. If any row (or column) of X is proportional to the other one, then the rows (or columns) are linearly dependent and det X ¼ 0. 4. If any two rows (or columns) of X are interchanged an odd number of times to yield Y, then det Y ¼ 2det X. If they are interchanged an even number of times to yield Z, then det Z ¼ det X. 5. If any row (or column) of X is modified by adding to it a times, the corresponding elements of another row (or column) to yield Y, then det Y ¼ det X. (The determinant remains unchanged.) 6. The determinant of a triangular matrix X is the product of its diagonal terms det X ¼ x11.x22 . . . . .xnn. 7. If the matrices X and Y are both n n, then det ( XY) ¼ (det X)(det Y). 8. det (aX) ¼ an det X. 9. Even if X and Y are n n, det (X þ Y) = det X þ det Y in general.
288
Elements of Matrix Algebra
Trace of a Matrix
The trace of a square matrix Tr(X) is the second most important invariant of a matrix. The trace is the sum of the diagonal terms of a matrix. The trace of the matrix given in Example 16.1.4 is 2 þ 2 þ 3 þ 1 ¼ 8.
Rank of a Matrix
The column rank of a rectangular matrix is the number of linearly independent column vectors. Similarly, the row rank of a matrix is the number of linearly independent rows. If the column rank and the row rank are the same, then we call that number the rank of the matrix. Thus in a nonsingular square matrix the column rank equals the row rank and the rank is the dimension of the matrix. In general, rank of any n m matrix can be defined as the number of elements 2, 3, . . . , n or m required to form a nonzero determinant. Forming all possible combinations of determinants from any given matrix is a difficult job. Finding a rank is not always easy, but some methods are easier than others. We show one method for finding the rank and the determinant of any matrix using pivotal 2 2 determinants. Example 16.1.5 2
a11
6 6 a21 6 6 X¼6 6 a31 6 6 a41 4 a51
Find the rank and determinant of the following nonsingular matrix: a12
a13
a14
a22
a23
a24
a32
a33
a34
a42
a43
a44
a52
a53
a54
a15
2
3
1
2
3
2
7 6 6 a25 7 7 6 1 7 6 6 a35 7 7¼6 2 7 6 6 a45 7 5 4 1
1
1
2
1
2
2
2
2
1
2
1
1
2
a55
3
3
7 87 7 7 1 7 7 7 07 5
(16:1:7)
5
Step 1. Remove a11 ¼ 1 outside the matrix in Eq. (16.1.7) using property 2 as shown in the previous section. We form a 4 4 matrix X1 from 2 2 determinants of the form fa11 . ajk 2 aj1 . a1k, j ¼ 2, . . . , 5 : k ¼ 2, . . . , 5) as shown below: 2
3
6 6 3 6 X1 ¼ 1 6 6 0 4 5
4
4
8
6
1
1
5
2
11
3
7 7 7 7 7 3 7 5 1
(16:1:8)
Step 2. Normalize the first element ¼ 3 as in step 1, and the resultant matrix is shown below. 2
1
6 6 3 6 X1 ¼ 1:3 6 6 0 4 5
4 3 8
4 3 6
1
1
5
2
11 3 3 7 7 7 7 7 3 7 5 1
(16:1:9)
16.1
Basic Theory of Matrices
289
Step 3. Form a 3 3 matrix X2 in an exactly analogous fashion by forming 2 2 determinants as in step 1 and normalize the first element in X2 as shown in Eq. (16.1.10) 2 3 2 3 1 4 2 4 1 1 6 7 2 6 7 7 1 1 3 7 ¼ 1:3 4 6 (16:1:10) X2 ¼ 1:3 6 1 1 3 6 7 4 5 14 4 5 14 52 5 52 5 3 3 3 3 3 3 Step 4. Form a 2 2 matrix X3 in an exactly analogous fashion by forming 2 2 determinants as in step 1 and normalize the first element in X3 as shown in Eq. (16.1.11): 2 3 1 " # 4 1 8 1 6 2 7 23 (16:1:11) X3 ¼ 1:3 4 4 5 ¼ 1:3 4 23 19 2 19 6 6 Step 5. Continue this process of reducing the order of the matrix until we end up with either a 1 1 matrix or a matrix whose elements are all zero. Finally we form a 1 1 matrix X4 in an exactly analogous fashion by forming 2 2 determinants as in step 1: X4 ¼ 1:3 4
1 35 ¼ 70 2 3
(16:1:12)
Since we have come to a scalar, the operations stop at this point. The number of cycles is 5 and hence the rank of this matrix is 5. The determinant is given by X4 ¼ 70. Thus, we can obtain both the rank and the determinant by this method. Example 16.1.6 In this example we will determine the rank and the maximum possible determinant of a singular matrix given by 2 3 6 11 5 4 1 6 5 3 2 3 47 6 7 7 X¼6 8 12 4 2 5 6 7 4 6 7 1 0 35 3 0 3 1 2 We form the various matrices as in the previous example, and the results are shown below: 3 2 2 3 11 5 4 1 6 11 5 4 1 1 6 6 6 6 67 6 7 7 6 3 2 3 47 6 5 6 5 3 2 3 47 6 7 7 6 X¼6 2 5 7 6 8 12 4 7¼66 8 12 4 2 5 7 7 6 6 7 7 6 7 1 0 35 4 6 4 6 7 1 0 35 3 0 3 1 2 3 0 3 1 2 2
37 6 6 6 6 8 6 6 X1 ¼ 6 6 3 6 6 4 6 4 11 2
37 6 8 3 4
19 3 22 3 4
11 2
3
3 2 19 1 6 6 7 7 6 6 8 11 7 7 37 6 7 6 6 3 3 7¼6 7 6 6 6 4 2 7 7 6 5 4 11 3 2
2
1 8 3 4 11 2
38 37 22 3 4 3
3 19 37 7 7 11 7 7 7 3 7 7 27 7 35 2
290
Elements of Matrix Algebra
We now form a 3 3 matrix using 2 2 pivotal determinants. This will yield the first column to be all zeros. Interchanging two columns of the resulting matrix is also shown below: 2
3 2 170 85 170 6 7 6 37 37 7 6 6 37 37 37 4 2 7 6 6 4 16 0 16 X2 ¼ 6 7¼6 6 7 6 37 6 6 37 37 4 4 98 98 49 5 0 37 37 37 2 3 1 0 1 6 7 2 7 37 170 6 2 6 4 7 ¼6 07 6 7 6 37 6 37 37 4 98 49 5 0 37 37 0
3 85 0 7 37 7 2 7 07 7 37 5 49 0 37
We now form a 2 2 matrix " 37 170 0 X3 ¼ 6 6 37 0
0
#
"
0
0
0
0
#
¼ 170 0
After three cycles we have a premature termination with a 2 2 zero matrix. Hence 170 the rank of this matrix is 3 and the maximum determinant is 6 37 6 37 ¼ 170:
Inverse of a Matrix
Since division by a matrix is not defined, we define the inverse of a nonsingular matrix as the matrix X 21 such that XX 21 ¼ X 21X ¼ I, an identity matrix. Thus inverse matrices commute with their direct counterparts. The cofactor expansion of the determinant of the matrix X from Eq. (16.1.5) is det X ¼
n X
xij Dij ¼ x1j D1j þ x2j D2j þ þ xnj Dnj
(16:1:5)
i¼1
where the expansion is carried about the n elements in any column j. Equation (16.1.5) can be rewritten as ( n X det X, k ¼ j det X ¼ xij Dik ¼ (16:1:13) 0, k=j i¼1 In Eq. (16.1.13), when k ¼ j, we obtain det X by Eq. (16.1.5), but when k = j, then we have cofactor expansion about the n elements in the jth column but with cofactors obtained from the kth column. This is equivalent to finding the determinant by replacing the kth column with the jth column in the matrix X. Thus X will contain two identical columns and hence from property 3 in the previous section, det X ¼ 0. We divide Eq. (16.1.13) by det X and obtain ( n X 1kn Dik (16:1:14) xij ¼ dkj , det X 1jn i¼1
16.1
Basic Theory of Matrices
291
where dkj is the Kro¨necker delta, which equals 1 if k ¼ j and 0 if k = j. Equation (16.1.14) is a scalar statement of the matrix equation X21X ¼ I, and the inverse is given by 2 3 D11 D21 Dn1 7 Dik 1 6 6 D12 D22 Dn2 7 (16:1:15) X1 ¼ {aki } ¼ xij ¼ 6 .. .. 7 .. det X 4 . det X . 5 . D1n D1n Dnn Note that the cofactors are in reverse order of the sequence of elements of the matrix X. The matrix fDjig of the cofactors is sometimes called the adjoint matrix. Finding the inverse using Eq. (16.1.15) is very inefficient and is of only theoretical interest since it gives an explicit expression for the inverse. Solution of Linear Equations
We can use matrix analysis to solve linear equations of the form Ax ¼ y where x and y are n-vectors: 32 3 2 3 2 y1 a11 a12 a1n x1 7 7 7 6a 6 6 6 21 a22 a2n 7 6 x2 7 6 y2 7 7 7 6 . 6 6 (16:1:16) ¼ .. 7 6 .. 7 6 .. 7 .. 7 6 . . 54 . 5 4 . 5 . 4 . an1
an2
ann A
xn x
yn y
We will assume that A is a nonsingular n n matrix and x and y are n 1 vectors. We want to solve for x in terms of y. Since A is invertible, we can write the solution as x ¼ A1 y This equation can be expanded as 2 3 2 2 32 3 3 D j1 y1 D11 D21 Dn1 x1 n 6 D j2 7 6 x2 7 76 7 1 6 1 X 6 7 6 D12 D22 Dn2 7 6 y2 7 6 7 ¼ 6 .. 7 ¼ 6 .. 6 6 . 7yj 7 7 .. .. .. 5 det A . 4 . 5 det A 4 . 4 4 5 . . . . 5 j¼1 D1n D1n xn Dnn yn D jn x A1 y
(16:1:17)
(16:1:18)
Note again that the order of the cofactors is transposed from that of the original matrix A, that is, if aij is an element in the ith row and jth column, then Dji is in the jth row and ith column. Example 16.1.7 x2, x3, x4:
We shall now solve the following set of linear equations for x1, x1 3x3 2x4 ¼ 4 5 1 15 23 x1 þ x2 þ x3 þ 6x4 ¼ 2 2 2 2 2x1 7x3 6x4 ¼ 11 7 1 25 39 x1 þ x2 þ x3 þ 10x4 ¼ 2 2 2 2
292
Elements of Matrix Algebra
or in matrix form
2
1
0
3
6 5 6 6 2 6 6 6 2 6 4 7 2
1 2 0
15 2 7
1 2
25 2
3 2 2 3 4 72 x 3 6 23 7 7 6 1 6 7 76 7 6 2 7 76 x2 7 6 7 74 5 ¼ 6 7 x3 6 11 7 6 7 7 7 6 5 x4 5 4 39 10 2
The determinant of the matrix A is found as follows: 2 1 0 3 2 3 2 3 2 1 15 5 7 6 5 1 6 7 15 7 6 6 6 2 2 6 7 6 7 6 2 6 2 2 2 7 6 7 6 6 jAj ¼ 6 6 7 3 6 2 7 ¼ 1 6 0 7 7 6 7 6 6 2 0 7 6 7 41 5 4 7 6 25 5 4 10 7 1 25 2 2 2 10 2 2 2 2 3 5 1 15 6 2 2 2 7 6 7 6 7 þ 2 6 2 0 7 7 6 7 4 7 1 25 5 2 2 2
1 2 0 1 2
Or jAj ¼ 1 þ 3 3 ¼ 1. The inverse of matrix A is 2
A1
1
0
3
6 5 6 6 2 6 ¼6 6 2 6 4 7 2
1 2 0
15 2 7
1 2
25 2
2 31 2 1 7 7 6 7 6 16 2 7 7 ¼ 6 1 14 6 7 3 7 5 2 10
2 1 1 1 2
1 2 1 1
The solution to the original set of equations is given by 3 2 2 3 4 2 3 1 2 1 2 6 23 7 2 3 1 x1 7 6 6 2 1 2 17 7 6 7 6 x2 7 6 76 2 17 7 6 6 7 ¼ 6 1 1 1 17 7¼6 4 4 x3 5 6 76 15 7 6 4 3 1 1 56 11 7 1 x4 5 4 1 39 2 2 2 2 Properties of Inverses (X, Y, Z Same Order and Nonsingular) 1. X X21 ¼ X21X matrix commutes with its inverse 2. Xm Xn ¼ X m1n 3. (XY)21 ¼ Y21X21 4. XY ¼ XZ ) Y ¼ Z 5. (X T)21 ¼ (X21)T
3 2 17 7 17 5 1 2
3 6 7 7 7 6 7 7 5 10
16.2 Eigenvalues and Eigenvectors of Matrices
1 det X (1)iþj Mij , where Mij is the minor of X ¼ det X
293
6. det X21 ¼ 7. X1
(16.1.19)
Orthogonal Matrices
The matrix X is orthonormal if X TX ¼ I. Since X21X is also equal to the identity matrix I, we have XT X ¼ I ¼ X1 X
or XT ¼ X1
(16:1:20)
It immediately follows from Eq. (16.1.20) that the determinant of an orthogonal matrix is equal to +1. Note that the converse is not true. In any orthogonal matrix X, real or complex, individual row or column vectors x are orthonormal: 1, j ¼ i H xi x j ¼ (16:1:21) 0, j = i where H stands for conjugate transpose. An example of a 4 4 orthogonal matrix used in discrete Fourier transform analysis is shown below: 2 3 1 1 1 1 6 1 ej(2p=4) ej(4p=4) ej(6p=4) 7 7 W¼6 (16:1:22) 4 1 ej(4p=4) ej(8p=4) ej(12p=4) 5 1 ej(6p=4) ej(12p=4) ej(18p=4) It can be verified that the rows and columns of W are orthonormal and det W ¼ 1.
B 16.2 EIGENVALUES AND EIGENVECTORS OF MATRICES Eigenvalues
The concept of eigenvalues and eigenvectors plays an important role. We motivate this concept by investigating solutions to a linear system of equations given by Ax ¼ lIx ¼ lx where A is an n n matrix and x is an n 1 vector, l is a scalar constant, and I is an n n identity matrix. Simplification of the equation above yields (lI A)x ¼ 0
(16:2:1)
A trivial solution to Eq. (16.2.1) is x ¼ 0. Nontrivial solutions can exist only if det(lI A) ¼ jlI Aj ¼ 0
(16:2:2)
Equation (16.2.2) is called the characteristic equation of the matrix A. Expansion of the determinant in Eq. (16.2.2) yields a polynomial in l of degree n called the characteristic polynomial of the matrix A, given by ln þ an1 ln1 þ an2 ln2 þ þ a0 ¼ 0
(16:2:3)
294
Elements of Matrix Algebra
The characteristic equation has a fundamental significance in linear systems. The n roots of the characteristic equation are called the eigenvalues of the matrix A, and in linear systems they are also known as the “poles” of the transfer function. The eigenvalues can be real, repeated, or complex. From the well-known result for the roots of polynomials, we also have the sum of the eigenvalues as 2an21, and the product of the eigenvalues, which is also the determinant of the matrix A, is given by (21)n a0. Example 16.2.1 determined:
The eigenvalues of the following nonsingular matrix have to be 2
3
1:5625
4:5
2:0625
5:25
6 0:4375 6 6 A¼6 6 0:4375 6 4 0:0625
2:5
1:9375
6:25
2:5
1:9375
7:25
3:5
0:4375
6:25
1:25 7 7 7 1:25 7 7 7 0:25 5
1:5
3
1
0:5
0
0:75
The characteristic equation of A is det (lI 2 A)¼0 is given by 2 l þ 1:5625 4:5 6 6 0:4375 l þ 2:5 6 6 jlI Aj ¼ 6 0:4375 2:5 6 6 6 3:5 4 0:0625 0:5 0 5
4
3
2:0625
5:25
1:9375
6:25
l þ 1:9375
7:25
0:4375 1:5
l 6:25 3
3 0:75 7 1:25 7 7 7 1:25 7 7 7 0:25 7 5 l1
2
¼ l 1:25l 2:375l þ l þ 1:375l þ 0:25 ¼ 0 The solutions of the fifth-order polynomial of this equation are the eigenvalues of A, given by l1 ¼ 2, l2 ¼ 1, l3 ¼ 1, l4 ¼ 0:5, l5 ¼ 0:25 The product of the eigenvalues is the determinant given by (21)5 . (a0) ¼ 20.25 and the sum of the eigenvalues is a4 ¼ 21.25. We can also find the eigenvalues of a square matrix that is singular. In this case the number of nonzero eigenvalues represents the rank of the matrix and the number of zero eigenvalues, the degree of singularity.
Cayley– Hamilton Theorem
The Cayley– Hamilton theorem states that every square matrix satisfies its own characteristic equation. In Eq. (16.2.3), if we substitute the matrix A for l, we obtain the Cayley –Hamilton equation An þ an1 An1 þ an2 An2 þ þ a0 A0 ¼ 0 where 0 represents a zero matrix.
(16:2:4)
16.2 Eigenvalues and Eigenvectors of Matrices
Example 16.2.2
295
Given the matrix 2
0 A¼4 6 6
1 0 11
3 0 15 6
equation is l3 þ 6l2 þ 11l þ 6 ¼ 0. According to the Cayley– we should have 33 2 2 32 31 0 0 1 0 0 1 0 7 6 6 7 7 1 5 þ 64 0 0 1 5 þ 114 0 0 15 6 6 11 6 6 11 6 30 2 3 2 3 1 0 6 11 6 0 0 6 7 6 7 6 7 0 1 5 ¼ 4 36 60 25 5 þ 4 36 66 36 5 11 6 150 239 90 216 360 150 3 2 3 11 0 6 0 0 7 6 7 0 11 5 þ 4 0 6 0 5 121 66 0 0 6
the characteristic Hamilton theorem, 2 0 1 6 0 4 0 6
11 2 0 6 þ 64 0 6 2 0 6 þ4 0 66
or
2
0 ¼ 40 0
0 0 0
3 0 05 0
showing the validity of the theorem. Definiteness of Matrices
A matrix A is positive definite if all its eigenvalues are either positive or have positive real parts. For example, the matrix A in Example 16.2.2 is positive definite, whereas the matrix A in Example 16.2.1 is nondefinite. Similarly, matrices having eigenvalues that are negative or have negative real parts are known as negative definite. If some of the eigenvalues are zero, then the matrix is called positive semidefinite or negative semidefinite accordingly. Since the determinants are (21)n times the product of eigenvalues, definiteness can also be defined by the sign of the determinants of all principal minors of a matrix. Principal minors are those minors obtained by successively deleting the rows and columns of a matrix starting from the first row and first column. If all the determinants of principal minors are positive, then the matrix is positive-definite: if they are negative then they are negative definite. Example 16.2.3 eigenvalues:
We will check the definiteness of the following matrix by finding the 2
1
3
7
7
5
6
9
11
4
87 7 7 87 7 7 75
63 6 6 A¼6 63 6 4 2:5
9
11
3
8
9:5
4
0:5
0
1:5
3
1
296
Elements of Matrix Algebra
As in the previous example, setting det (lI 2 A) ¼ 0 gives 3 2 7 5 6 l 1 7 6 3 l9 11 4 8 7 6 7 6 7 2 3 2 9 l þ 11 3 8 7 jlI Aj ¼ 6 6 3 7 ¼ l (l þ 4l þ l 6) ¼ 0 6 7 4 2:5 8 9:5 l þ 4 7 5 0:5 0 1:5 3 l1 Thus the rank of the matrix A is 3 since there are two zero eigenvalues. The eigenvalues are given by l1 ¼ 23, l2 ¼ 22, l3 ¼ 1, l4 ¼ 0, l5 ¼ 0. Here the matrix A is nondefinite since it has positive, negative, and zero eigenvalues.
Eigenvectors
Eigenvectors are solutions to Eq. (16.2.1) for any given eigenvalue. We shall assume that all eigenvalues are distinct and there are no repeated eigenvalues. If we substitute any eigenvalue li in det(lI 2 A) ¼ 0, then, by the very definition, we have det(liI– A) ¼ 0 and hence solutions to Eq. (16.2.1) are possible, and these solutions are the unnormalized eigenvectors ci. However, these solutions are unique only upto a multiplicative constant. To make the solutions unique we can normalize these eigenvectors and the normalized eigenvectors are denoted by fi ¼ ci/jcij. Example 16.2.4 repeated below:
Find the eigenvectors of the nonsingular matrix of Example 16.2.1, 2
2:0625
6 0:4375 2:5 6 6 A¼6 6 0:4375 2:5 6 4 0:0625 3:5
1:9375
6:25
1:9375
7:25
0:4375
6:25
1:25 7 7 7 1:25 7 7 7 0:25 5
1:5
3
1
0:5
0
5:25 0:75
3
1:5625 4:5
Substituting l1 ¼ 2 in this equation, we can solve for the unnormalized eigenvectors: 2 32 3 c11 3:5625 4:5 2:0625 5:25 0:75 6 0:4375 4:5 6 7 1:9375 6:25 1:25 7 6 76 c12 7 6 76 7 6 7 3:9375 7:25 1:25 7 ðl1 I AÞC1 ¼ 6 6 0:4375 2:5 76 c13 7 ¼ 0 6 76 7 4 0:0625 3:5 0:4375 4:25 0:25 54 c14 5 0:5
0
1:5
3
1
c15
As mentioned earlier, the equation above can be solved for only four of the unknowns in terms of the fifth. Let us assume that c11 ¼ 1 and solve for c12,c13,c14,c15 in terms of c11. The preceding equation can be rewritten by substituting c11 ¼ 1: 3 32 2 3 2 c12 0:4375 4:5 1:9375 6:25 1:25 7 76 6 7 6 6 2:5 3:9375 7:25 1:25 76 c13 7 6 0:4375 7 7 76 6 7¼6 6 3:5 0:4375 4:25 0:25 76 c 7 6 0:0625 7 5 54 14 5 4 4 0:5 0 1:5 3 1 c15
16.2 Eigenvalues and Eigenvectors of Matrices
297
Solving for c12 ,c13 ,c14 ,c15 and adjoining c11 ¼ 1, we obtain the unnormalized eigenvector corresponding to the eigenvalue l1 ¼ 2: 3 2 3 2 c11 1 3 2 3 2 1:333333 c12 7 6 7 6 6 c12 7 6 1:333333 7 6 c 7 6 1:952381 7 6 7 6 7 7 6 13 7 6 ¼6 1:952381 7 c13 7 7 and 6 6 7¼6 6 7 6 7 4 c14 5 4 1:238095 5 7 6 7 6 4 c14 5 4 1:238095 5 1:285714 c15 1:285714 c15 Normalization of this equation yields the eigenvector f1: 3 2 3 2 2 3 0:319838 1 c11 7 6 7 6 6 7 6 1:333333 7 6 0:426451 7 6 c12 7 7 6 7 6 7 c1 6 1 6 1:952381 7 ¼ 6 0:624446 7 6 c13 7 ¼ f1 ¼ 6 7 6 7 6 7 jc1 j 6 7 7 6 7 3:126581 6 4 1:238095 5 4 0:395990 5 4 c14 5 0:411220 1:285714 c15 l1 ¼2
In a similar manner, the other eigenvectors corresponding to l2 ¼ 1, l3 ¼ 21, l4 ¼ 20.5, and l5 ¼ 20.25 are shown below: 2 2 2 3 3 3 0:275010 0:774597 0:679189 6 6 6 7 7 7 6 0:366679 7 6 0:516398 7 6 0:549819 7 6 6 6 7 7 7 6 6 7 7 7 f2 ¼ 6 6 0:641681 7 , f3 ¼ 6 0:258199 7 , f4 ¼ 6 0:032342 7 , 6 6 6 7 7 7 4 0:275010 5 4 0:258199 5 4 0:291081 5 0:550019 0 0:388108 2
l2 ¼1
3
l3 ¼1
l4 ¼0:5
0:679104 6 7 6 0:325871 7 6 7 7 f5 ¼ 6 6 0:340796 7 6 7 4 0:166667 5 0:531373 l5 ¼0:25
Modal Matrix
A matrix F consisting of the linearly independent eigenvectors of a matrix A is called the modal matrix of A. In the previous example the modal matrix of A was 2 3 0:319838 0:319838 0:774597 0:679189 0:679104 6 7 0:516398 0:549819 0:325871 7 6 0:426451 0:426451 6 7 F¼6 0:258199 0:032342 0:340796 7 6 0:624446 0:624446 7 6 7 0:258199 0:291081 0:166667 5 4 0:395990 0:395990 0:411220 0:411220 0 0:388108 0:531373 The determinant of F ¼ 20.00141.
298
Elements of Matrix Algebra
Diagonalization of Matrices
An n n square matrix A is diagonalizable if there exists another nonsingular n n matrix T such that T21AT ¼ D, where D is a diagonal matrix. The conditions of diagonalizability reduce to the existence of n linearly independent eigenvectors of A or the modal matrix F of A is invertible, in which case the jth diagonal element of D is the jth eigenvalue of A. In this case the modal matrix F is the desired matrix T. The modal matrix F is not the only matrix that will diagonalize A. There are other matrices that may also diagonalize A, but all the diagonal elements obtained from the modal matrix only are the eigenvalues of A. As a consequence of the preceding discussion, we can write the matrix equivalent of the eigenequation (16.2.1) as follows: or F1 AF ¼ L
AF ¼ FL Example 16.2.5
Diagonalize the following matrix: 2 0 1 0 6 4 0 0 A¼6 4 0 2 0 5 1 1
(16:2:5)
3 0 07 7 15 0
The eigenvalues of A are obtained by solving the determinant equation jlI 2 Aj ¼ 0, or 2 3 l 1 0 0 6 4 l 0 07 7 ¼ l4 5l2 þ 4 ¼ 0 jlI Aj ¼ 6 4 0 2 5 l 1 5 1 1 l The characteristic equation can be factored as (l 2 2)(l þ 2)(l 2 1)(l þ 1) ¼ 0, yielding the eigenvalues as l1 ¼ 2, l2 ¼ 22, l3 ¼ 1, l4 ¼ 21. We find the eigenvectors corresponding to these eigenvalues. The eigenvectors following the Example 16.2.4 are 3 3 2 2 3 3 3 3 2 2 p ffiffiffiffiffi p ffiffiffiffiffiffiffi ffi 6 74 7 6 146 7 0 0 7 7 6 6 6 6 7 6 6 7 7 7 6 6 6 pffiffiffiffiffi 7 6 pffiffiffiffiffiffiffiffi 7 6 0 7 6 0 7 7 7 6 6 7 7 6 6 6 74 7 6 146 7 6 1 7 6 1 7 f1 ¼ 6 7, f2 ¼ 6 1 7, f3 ¼ 6 pffiffiffi 7, f4 ¼ 6 pffiffiffi 7 7 6 5 7 6 6 27 6 27 6 pffiffiffiffiffi 7 6 pffiffiffiffiffiffiffiffi 7 7 7 6 6 6 74 7 6 146 7 4 1 5 4 1 5 7 7 6 6 pffiffiffi pffiffiffi 4 2 5 4 10 5 2 2 pffiffiffiffiffiffiffiffi pffiffiffiffiffi 146 74 or 2
0:348743
3
2
0:248282
3
6 0:697486 7 6 0:496564 7 6 7 6 7 f1 ¼ 6 7 , f2 ¼ 6 7, 4 0:581238 5 4 0:082761 5 0:827606
0:232495 l1 ¼2
2
l2 ¼2
3
0 6 7 0 6 7 f4 ¼ 6 7 4 0:707107 5 0:707107 l4 ¼1
2
0
3
6 7 0 6 7 f3 ¼ 6 7, 4 0:707107 5 0:707107 l3 ¼1
16.2 Eigenvalues and Eigenvectors of Matrices
299
The modal matrix F is given by 2
F ¼ ½f1
f2
f3
3 6 pffiffiffiffiffi 6 74 6 6 6 pffiffiffiffiffi 6 6 74 f4 ¼ 6 6 5 6 pffiffiffiffiffi 6 74 6 4 2 pffiffiffiffiffi 74
3 pffiffiffiffiffiffiffiffi 0 146 6 pffiffiffiffiffiffiffiffi 0 146 1 1 pffiffiffiffiffiffiffiffi pffiffiffi 146 2 10 1 pffiffiffiffiffiffiffiffi pffiffiffi 146 2
3 0 7 7 7 0 7 7 7 7 1 7 pffiffiffi 7 27 7 1 5 pffiffiffi 2
and 2
2 6 0 F1 AF ¼ 6 40 0
0 2 0 0
0 0 1 0
3 0 07 7 05 1
Symmetric Matrices
Real symmetric matrices are those matrices that satisfy the condition AT ¼ A
(16:2:6)
Complex symmetric matrices satisfy the Hermitian symmetry condition AH ¼ (A )T ¼ A
(16:2:7)
where the symbol H defined earlier stands for the conjugate transpose. Symmetric matrices play a very important role in probability; for example, the covariance matrices are all real symmetric matrices. They have some important properties that come in handy in simplifying algebraic manipulations. Some of the more important properties of symmetric matrices are 1. All eigenvalues are real. 2. The eigenvectors corresponding to distinct eigenvalues are orthogonal. Hence the eigenvectors can form the basis vectors of an n-dimensional coordinate system. 3. The modal matrix F is an orthogonal matrix, and hence F21 ¼ F T. The diagonalization of a symmetric matrix can be accomplished by the transformation F TAF ¼ L. Example 16.2.6 The eigenvectors and eigenvalues of the symmetric matrix given below are to be determined and their properties studied: 2 3 1 2 1 1 6 2 1 1 1 7 7 A¼6 4 1 1 2 2 5 1 1 2 0 The characteristic equation is det [lI 2 A] ¼ 0: l 1 2 1 1 2 l 1 1 1 jlI Aj ¼ ¼ l4 4l3 7l2 þ 22l þ 24 ¼ 0 1 1 l 2 2 1 1 2 l
300
Elements of Matrix Algebra
The eigenvalues are l1 ¼ 3, l2 ¼ 22, l3 ¼ 21, l4 ¼ 4, obtained by factoring the characteristic equation. The four normalized eigenvectors are obtained as outlined in the previous examples and they are given by 2 2 3 2 2 3 3 3 1 1 1 1 7 7 7 7 1 6 1 6 1 6 1 6 6 17 617 6 1 7 6 17 f1 ¼ pffiffiffi 6 f2 ¼ pffiffiffi 6 7 , f3 ¼ pffiffiffiffiffi 6 7, 7 , f4 ¼ pffiffiffiffiffi 6 7 3405 10 4 2 5 24 05 15 4 3 5 0
1
l1 ¼3
2
l2 ¼2
2
l3 ¼1
l4 ¼4
The modal matrix F is 2
1 1 0 1
1 1 1 1 1 6 1 6 F ¼ pffiffiffi pffiffiffi pffiffiffiffiffi pffiffiffiffiffi 4 0 2 3 10 15 0 It can be verified that
fi fj ¼
1, 0,
1 1 2 2
3 1 17 7, 35 2
jFj ¼
30 ¼1 30
j¼i j=i
showing that the modal matrix is orthogonal and jFj ¼ 1. The original matrix A can be diagonalized by the modal matrix as shown below: 2
1 6 1 6 1 FT AF ¼ 6 30 4 1
1 1
0 0
1
2
1
1
3
2
1 6 1 6 6 4 0 0
1 1 1 1 0 1
2 2
32 1 0 6 7 1 76 2 76 2 54 1
2 1
1 1
1
2
3 1 1 7 7 7 2 5
1 1 2 0 2 3 2 3 1 3 0 0 0 6 0 2 17 0 07 7 1 6 7 ¼6 7 7 3 5 30 4 0 0 1 0 5 2 0 0 0 4
Similarity Transformations
In the previous section we obtained the diagonal matrix L by transforming the matrix A by the modal matrix F. The resulting diagonal matrix L has all the fundamental properties of the original matrix A. For example these two matrices have the same determinants and the same traces. Such matrices are called similar matrices, and the transformation is known as a similarity transformation. Any nonsingular matrix A can be transformed into another similar matrix B by the following similarity transformation T1 AT ¼ B where T is another nonsingular transforming matrix. Example 16.2.7 With 2 0 1 6 1 0 T¼6 4 1 1 1 1
1 1 1 1
3 0 17 7 05 1
2
and
1 2 6 2 1 A¼6 4 1 1 2 1
3 1 2 2 17 7 1 05 1 2
16.3
Vector and Matrix Differentiation
301
we have 2
7 6 2 1 T AT ¼ B ¼ 6 4 7 2
0 7 2 2 4 6 3 1
3 5 37 7 75 2
In this example the determinants of A and B are 1 and the traces of A and B are 21.
B 16.3
VECTOR AND MATRIX DIFFERENTIATION
The definitions for differentiation with respect to a real vector and a complex vector are slightly different. In the case of differentiation of a complex vector, the question of whether the derivative is a total or partial also makes a difference. We shall first discuss differentiation with respect to real vectors.
Differentiation with Respect to Real Vectors
Let x be an n-vector variable defined by x ¼ fx1, x2, . . . , xngT. The total and partial derivative operators with respect to x are defined by 2 2 3 3 d @ 6 dx1 7 6 @x1 7 6 6 7 7 6 d 7 6 @ 7 d @ 6 7, 7 ¼6 ¼ (16:3:1) 6 7 7 dx 6 6 dx2 7 @x 6 @x2 7 4 d 5 4 @ 5 @xn dxn Just as the (d=dx)x ¼ 1, we want the derivative with respect to the vector x to be equal to the identity matrix I, taking care that the dimensional compatibility conditions are satisfied. Thus, (d=dx)x cannot be defined since the compatibility conditions are not met. We can, however, define (d=dx)xT ¼ I as shown below: 2 3 d 6 dx1 7 2 3 6 7 1 0 0 7
d T 6 d 7 4 5 x ¼6 (16:3:2) 6 dx2 7 x1 x2 xn ¼ 0 1 0 ¼ I dx 6 7 0 0 0 1 4 d 5 dxn In an analogous fashion we can define the following operation: 2 3 d 6 dx1 7 2 3 6 7 2x1 6 7
d T d 7 2 2 2 4 5 x x¼6 6 dx2 7 x1 þ x2 þ þ xn ¼ 2x2 ¼ 2x dx 6 7 2x n 4 d 5 dxn
(16:3:3)
302
Elements of Matrix Algebra
We can also write the following differentiation formulas as follows d T x Ax ¼ (A þ AT )x dx d T d y Ax ¼ xT AT y ¼ AT y dx dx
(16:3:4)
Note that in Eq. (16.3.4) x TAx is a scalar, known as the quadratic form, and y TAx is also a scalar, known as the bilinear form. A scalar is its own transpose, that is, x TAx ¼ x TA Tx and y TAx ¼ x TA Ty. Corresponding to Eq. (16.3.2), we can also define the following operations: 2 3 x1 6 x 7 T 27 d d d d 6 6 . 7¼n x¼ 7 . dx dx1 dx2 dxn 6 4 . 5 xn
d dx
T
xxT ¼
d dx1 2
d dx2 x21
x1 x2
6x x 2 6 2 1 x2 6 6 . .. 4 .. . xn x1 xn x2 T d d d xyT ¼ dx dx1 dx2 2 x1 y1 x1 y2 6x y x y 6 2 1 2 2 6 .. 6 .. 4 . . xn y1 xn y2
d dxn
3
x1 xn
x2 xn 7 7 T .. 7 7 ¼ (n þ 1)x . 5
d dxn
(16:3:5)
x2n
x1 yn
3
x2 yn 7 7 T .. 7 7 ¼ ny . 5 xn yn
Differentiate f(x) with respect to the vector x: " #" # x1
3 2 T f (x) ¼ x Ax ¼ x1 x2 ¼ 3x21 þ 6x1 x2 þ 5x22 x2 4 5
Example 16.3.1
Differentiating f(x) with respect to x1 and x2, we obtain 9 d " > (3x21 þ 6x1 x2 þ 5x22 ) ¼ 6x1 þ 6x2 > = 6 dx1 ¼ d > 6 > (3x21 þ 6x1 x2 þ 5x22 ) ¼ 6x1 þ 10x2 ; dx2 Using the formula ½d(xT Ax)=dx ¼ ½A þ AT x, we (" # " 3 2 3 d T f (x) ¼ ½A þ A x ¼ þ dx 4 5 2
obtain # )" # 4 x1 5
x2
6
#"
10
"
x1
#
x2
6
6
6
10
¼
#"
x1 x2
#
16.3
Vector and Matrix Differentiation
303
We differentiate f (x, y) with respect to the vector x in the following
Example 16.3.2 example: T
f (x, y) ¼ y Ax ¼ y1
3 y2 4
2 5
x1 x2
¼ 3x1 y1 þ 4x1 y2 þ 2x2 y1 þ 5x2 y2
Using the formula ½d(yT Ax)=dx ¼ AT y, we can write 3 4 y1 3y1 þ 4y2 ¼ AT y ¼ 2 5 y2 2y1 þ 5y2 Complex Differentiation Total Derivatives Differentiation with respect to a complex variable z ¼ x þ jy is a bit more involved. If we have a function of a complex variable
w ¼ f (z) ¼ u(x, y) þ jv(x, y)
(16:3:6)
then the total derivative of f(z) with respect to z is defined by d Dw f (z þ Dz) f (z) w ¼ lim ¼ lim Dz!0 Dz Dz!0 dz Dz
(16:3:7)
If the derivative df (z)=dz is to exist, then the limit must exist independent of the way in which Dz approaches zero. This condition imposes restrictions on the existence of the total derivative. With Dw ¼ Du þ j Dv and Dz ¼ Dx þ j Dy, we have the following from Eq. (16.3.7): lim Dz ¼ lim Dz!0
Dx!0 Dy!0
Du þ jDv Dx þ jDy
(16:3:8)
If we first let Dy ! 0 and then let Dx ! 0 in Eq. (16.3.8), we have df (z) @u @v ¼ þj dz @x @x
(16:3:9)
Similarly, if we first let Dx ! 0 and then let Dy ! 0, we obtain df (z) @v @u ¼ j dz @y @y
(16:3:10)
If the derivative df (z)=dz is to be unique then Eqs. (16.3.9) and (16.3.10) must match. Hence @u @v ¼ @x @y @v @u ¼ @x @y
(16:3:11)
These conditions are known as the Cauchy –Riemann (C– R) conditions. If C – R conditions are satisfied, then the derivative of f (z) is given by either Eq. (16.3.9) or Eq. (16.3.10).
304
Elements of Matrix Algebra
A complex function f (z) is said to be analytic in the neighborhood of z ¼ z0, if C –R conditions are satisfied in that neighborhood. In addition, if it is also analytic at z ¼ z0, then the point z0 is a regular point. However, if it is not analytic at z ¼ z0, then z0 is called a singular point. Example 16.3.3 We will find the derivative of f (z) ¼ z 2 ¼ (x þ jy)2 ¼ x 2 2 y 2 þ 2jxy. We will determine whether f (z) satisfies the Cauchy – Riemann conditions of Eq. (16.3.9). z2 ¼ (x þ jy)2 ¼ x2 y2 þ 2jxy u(x, y) ¼ x2 y2
:
v(x, y) ¼ 2xy
@u @v ¼ 2x ¼ ¼ 2x @x @y and hence
@v @u ¼ 2y ¼ ¼ 2y @x @y
The C – R conditions are satisfied and df (z)=dz is given by d 2 @u @v z ¼ þ j ¼ 2(x þ jy) ¼ 2z dz @x @x
Example 16.3.4 On the other hand, we will find the derivative of f (z) ¼ z 2 ¼ (x 2 jy)2 ¼ x 2 2 y 2 2 2jxy. We will determine whether f (z) satisfies the Cauchy – Riemann conditions of Eq. (16.3.9): z2 ¼ (x jy)2 ¼ x2 y2 2jxy u(x, y) ¼ x2 y2
: v(x, y) ¼ 2xy
@u @v ¼ 2x = ¼ 2x @x @y @v @u ¼ 2y = ¼ 2y @x @y Since the C –R conditions are not satisfied at points other than at z ¼ 0, z 2 does not have a total derivative dz2 =dz at points other than at z ¼ 0. Partial Derivatives Many functions occurring in engineering practice are not analytic, and we are interested in optimization problems involving partial derivatives with respect to a complex variable z. With z ¼ x þ jy, we shall define the partial derivative operator @=@z as
@ @ @ ¼ þj @z @x @y
(16:3:12)
16.3
Vector and Matrix Differentiation
305
From the definition given in Eq. (16.3.12), we have the following: @ @ @ z¼ þj (x þ jy) ¼ 1 1 ¼ 0 @z @x @y @ @ @ z ¼ þj (x jy) ¼ 1 þ 1 ¼ 2 @z @x @y
(16:3:13)
Note that from Example 16.3.3 the total derivative (d/dz)z does not exist since z is not analytic except at z ¼ 0. From Eq. (16.3.12) we can also write @ @ @ (z z) ¼ þj (x2 þ y2 ) @z @x @y ¼ 2(x þ jy) ¼ 2z @ 2 @ @ z ¼ þj (x2 þ 2jxy y2 ) @z @x @y ¼ 2(x þ jy) 2(x þ jy) ¼ 0 @ @ @ (z w) ¼ þj (xu þ yv þ j(xv yu)Þ @z @x @y
(16:3:14)
¼ 2(u þ jv) ¼ 2w @ @ @ (zw) ¼ þj (xu yv þ j(xv þ yu)Þ @z @x @y ¼ (u þ jv) (u þ jv) ¼ 0 Note that if we used the chain rule of differentiation in Eq. (16.3.14) and apply Eq. (16.3.13), we obtain the same results. This fact gives credibility to the definition of the partial derivative @=@z. Note also that if z and w were real, then we would get (@=@z)z2 ¼ 2z and (@=@z)zw ¼ w. These differences also manifest themselves in defining vector differentiation with respect to a complex variable.
Vector Partial Differentiation Analogous to Eq. (16.3.12), partial derivatives with respect to a vector complex variable can be defined. With z ¼ [x1 þ jy1, x2 þ jy2, . . . , xn þ jyn]T, partial derivatives @=@z and @=@z are defined as follows:
3 @ @ þj 6 @x1 @y1 7 7 6 6 @ @ 7 7 6 þj @ 6 @y2 7 ¼ 6 @x2 7; 7 @z 6 .. 7 6 . 7 6 4 @ @ 5 þj @xn @yn 2
2
x1 þ jy1
3
6 x þ jy 7 27 6 2 7; z¼6 .. 6 7 4 5 . xn þ jyn
3 @ @ j 6 @x1 @y1 7 7 6 6 @ @ 7 7 6 @ 6 @x2 j @y2 7 ¼ 7 6 7 @z 6 .. 7 6 . 7 6 4 @ @ 5 j @xn @yn 2
(16:3:15)
306
Elements of Matrix Algebra
With this definition of @=@z and @=@z , we can define the following operations: 3 @ @ þj 6 @x1 @y1 7 7 6 6 @ @ 7 7 6 þ j @ H 6 @x @y2 7 z ¼6 2 7 x1 jy1 7 6 @z .. 7 6 . 7 6 4 @ @ 5 þj @xn @yn 2 3 2 0 0 0 60 2 0 07 6 7 ¼6 7 ¼ 2In 40 0 2 05 2
0
0 0
x2 jy2
xn jyn
(16:3:16)
2
where z H is the conjugate transpose of z and In is an nth-order identity matrix. In a similar manner, we obtain 3 @ @ j 6 @x1 @y1 7 7 6 6 @ @ 7 7 6 j @ T 6 @x @y2 7 2 z ¼ 7 x1 þ jy1 6 7 6 @z .. 7 6 . 7 6 4 @ @ 5 j @xn @yn 2 3 2 0 0 0 60 2 0 07 6 7 ¼6 7 ¼ 2In 40 0 2 05 2
0
0
0
x2 þ jy2
xn þ jyn
(16:3:17)
2
In addition, we also have 3 @ @ þj 6 @x1 @y1 7 7 6 6 @ @ 7 7 6 þ j @ T 6 @x @y2 7 z ¼6 2 7 x1 þ jy1 7 6 @z .. 7 6 . 7 6 4 @ @ 5 þj @xn @yn 2
¼ 0n ¼
x2 þ jy2
xn þ jyn
(16:3:18)
@ H z @z
where 0n is an nth-order zero matrix. Equations (16.3.16) – (16.3.18) are very useful in complex vector minimization of scalar quadratic error performance criteria, as we will show in the following examples. Example 16.3.5 The terms z and w are two complex 2-vectors as shown below. It is desired to minimize the complex matrix scalar criteria 11 ¼ zH w and 12 ¼ wH z with
16.3
respect to the complex vector z. With a þ jb ; z¼ c þ jd
Vector and Matrix Differentiation
w¼
e þ jf g þ jh
307
and the corresponding error criteria
11 ¼ zH w ¼ ae þ bf þ cg þ dh þ j(af be þ ch dg) 12 ¼ wH z ¼ ae þ bf þ cg þ dh j(af be þ ch dg) we can differentiate 11 and 12 partially with respect to z, and using Eqs. (16.3.16) and (16.3.18), we obtain e þ e þ j( f þ f ) @ @ H ¼ 2w 11 ¼ z w¼ @z @z g þ g þ j(h þ h) e e þ j( f f ) @ @ @ T z w ¼ ¼0 12 ¼ w H z ¼ @z @z @z g g þ j(h h) The same result can be obtained by the chain rule of differentiation and Eqs. (16.3.16) – (16.3.18). Example 16.3.6 We take an example of minimizing a scalar quadratic error surface with respect to a complex weight vector a p called adaptive coefficients and finding the optimum value of ap. The quadratic error surface is given by H H E p ¼ s2 þ a H p R p ap þ r p ap þ a p r p
(16:3:19)
We differentiate with respect to a p and set the result to zero and solve for a po using Eqs. (16.3.16) – (16.3.18): @ @ H H Ep ¼ ½s2 þ aH p Rp ap þ rp ap þ ap rp ¼ 0 @ap @ap ¼
@ 2 @ H @ H @ H s þ a R p ap þ r ap þ a rp ¼ 0 @a p @a p p @ap p @ap p
(16:3:20)
¼ 0 þ 2Rp ap þ 2rp ¼ 0, or apo ¼ R1 p rp The solution a po ¼ R1 p rp is called the Wiener solution for the adaptive coefficients a p. Summary of Some Vector Partial Derivatives of Real and Complex Quadratic Forms @ @ @ T @ T J(x) ¼ ½xT Ax ¼ x Ax þ x AT x ¼ ½A þ AT x : x real @x @x @x @x @ @ @ H @ T J(x) ¼ ½xH Ax ¼ x Ax þ x AT x ¼ 2Ax þ 0 : x complex @x @x @x @x (16:3:21) @ @ @ T J(x, y) ¼ ½xT Ay ¼ x Ay ¼ Ay : x real @x @x @x @ @ H @ H J(x, y) ¼ ½x Ay ¼ x Ay ¼ 2Ay : x complex @x @x @x
308
Elements of Matrix Algebra
A final point to note in taking the vector partial derivatives is that if x is real, then (@=@x)xT ¼ I, and if x is complex, then (@=@x)xH ¼ 2I and (@=@x)xT ¼ 0. Differentiation with Respect to Real Matrices
In some minimization problems there is a need to differentiate a scalar function of a real matrix A with repect to the real matrix A. We will use the following result in Section 22.3. If B is any square matrix and A is any other matrix, then @ Tr½ABAT ¼ A½B þ BT @A
(16:3:22)
@ Tr½ABAT ¼ 2AB @A
(16:3:23)
If B is symmetric, then
Example 16.3.7
We are given the matrices X and B as x11 x12 b11 b12 X¼ ; B¼ x21 x22 b21 b22
We will show that Eq. (16.3.22) is true. The trace of XBX T is given by Tr½XBXT ¼ x211 b11 þ x222 b22 þ x212 b11 þ x221 b22 þ x11 x21 b12 þ x11 x21 b21 þ x22 x12 b12 þ x22 x12 b21 Taking partial derivatives with respect to x11, x22, x12, and x21, we obtain @ Tr½XBXT ¼ 2x11 b11 þ x21 (b21 þ b12 ) @x11 @ Tr½XBXT ¼ 2x22 b22 þ x12 (b21 þ b12 ) @x22 @ Tr½XBXT ¼ 2x12 b11 þ x22 (b21 þ b12 ) @x12 @ Tr½XBXT ¼ 2x21 b22 þ x11 (b21 þ b12 ) @x21 b11 x11 x12 b11 b12 þ ¼ x21 x22 b21 b22 b12
B 16.4
b21 b22
¼ X{B þ BT }
BLOCK MATRICES
In some applications we will encounter block matrices whose elements are also matrices. An (n m) (n m) matrix Z can be formulated as a block matrix consisting of
16.4
individual blocks A,B,C,D, as 2 x11 6 6 6 6 xn1 6 Z¼6 6 6 xnþ1,1 6 6 4 xnþm,1 2 A 6 6nn ¼6 6 C 4 mn
Block Matrices
309
shown below: x1n
x1, nþ1
xn,n
xn, nþ1
B
xnþ1,n
xnþ1, nþ1
A C
x1, nþm
3
7 7 7 xn, nþm 7 7 7 7 xnþ1, nþm 7 7 7 D 5 xnþm, nþm
xnþm,n xnþm, nþ1 3 B 7 nm7 7 D 7 5 mm
(16:4:1)
We can enumerate some properties of block matrices that may be useful in later chapters: 1. Multiplication. The compatibility of multiplying blocks must be satisfied as shown: 2 3 2 3 F E A B 6 7 6 6nn nm7 nm7 6 76 n n 7 4 C D 56 H 7 4 G 5 mm mn mm mn 2 3 AE þ BG AF þ BH 6 nn nm 7 6 7 6 7 (16:4:2) ¼6 CF þ DH 7 4 CE þ DG 5 mm mn 2. Determinant. If A is nonsingular and D is square, then " # A B det ¼ det A det(D CA1 B) C D
(16:4:3)
3. Inverse. a. Matrix Inversion Lemma. If A and D are nonsingular n n and m m matrices respectively, then, (A þ BD1 C)1 ¼ A1 A1 B(D þ CA1 B)1 CA1
(16:4:4)
b. If inverses for A and D exist then,
A C
0 D
A 0
B D
"
1 ¼
"
1 ¼
A1
0
D1 CA1
D1
A1 0
A1 BD1 D1
# ; #
(16:4:5)
310
Elements of Matrix Algebra
c. If inverses for A and D exist then, 1 " D1 A B A ¼ 1 C D D1 D CA
A1 BD1 D D1 D
# (16:4:6)
where D A and D D are the Schur components of the matrices A and D defined by DA ¼ A BD1 C DD ¼ D CA1 B
(16:4:7)
d. Using property 3a, the inverses of the Schur component D A and D D are given by 1 1 þ A1 BD1 D1 A ¼A D CA 1 1 D1 þ D1 CD1 D ¼D A BD
(16:4:8)
1 We can substitute for either D1 A or DD from Eq. (16.4.8) and solve Eq. (16.4.6).
CHAPTER
17
Random Vectors and Mean-Square Estimation
B 17.1
DISTRIBUTIONS AND DENSITIES
Joint Distributions and Densities
We have come across sequences of random variables in Chapter 14, and these can be represented by a random n-vectors. We can define two random vectors X and Y as 2 3 2 3 Y1 X1 6X 7 6Y 7 6 27 6 27 7 6 7 (17:1:1) X¼6 6 .. 7; Y ¼ 6 .. 7 4 . 5 4 . 5 Xn
Ym
The joint distribution of the random vector X is given by FX (X) ¼ P½X x ¼ P½X1 x1 , . . . , Xn xn
(17:1:2)
with FX(1) ¼ 1 and FX(21) ¼ 0. The corresponding joint density function fX(x) can be defined as P(x , X x þ DxÞ DX!0 Dx
fX (x) ¼ lim ¼ lim
Dx1 !0 ... Dxn !0
Pðx1 , X1 x1 þ Dx1 , . . . , xn , Xn xn þ Dxn Þ Dx1 Dxn
(17:1:3)
which is the nth partial derivative of the distribution function FX(x): fX (x) ¼
@n FX (x) @x1 @xn
(17:1:4)
Probability and Random Processes, by Venkatarama Krishnan Copyright # 2006 John Wiley & Sons, Inc.
311
312
Random Vectors and Mean-Square Estimation
Given the density function fX(x), the distribution function FX(x) can be obtained by an n-fold integration: ð xn ð x1 FX (x) ¼ fX (x0 )dx01 dx0n (17:1:5) 1
1
T Given an n-vector X ¼ X1 Xn T , the joint distribution of X and Y is given by
Joint Distribution of Two Vectors X and Y
and an m-vector Y ¼ Y1 Ym
FXY (x,y) ¼ PðX x, Y yÞ
(17:1:6)
and the joint density, by Pðx , X x þ Dx, y , Y y þ DyÞ Dx Dy
fXY (x,y) ¼ lim
Dx!0 Dy!0
x1 , X1 x1 þ Dx1 , . . . , xn , Xn xn þ Dxn
!
P ¼ lim
Dx1 !0
.. .
y1 , Y1 y1 þ Dy1 , . . . , ym , Ym ym þ Dym Dx1 Dxn Dy1 Dym
Dxn !0
.. .
Dy1 !0
.. .
Dym !0
or fXY (x,y) ¼
@n @m FXY (x,y) @x1 @xn @y1 @ym
(17:1:7)
The distribution function FXY(x,y) can also be obtained by integrating Eq. (17.1.7): ðx ðy fxy (x0 ,y0 )dy0 dx0 (17:1:8) FXY (x,y) ¼ 1
1
Since Eq. (17.1.8) is a distribution function, we infer the following relationships: FXY (x,1) ¼ FXY (1, y) ¼ 0 FXY (x,þ1) ¼ FX (x), marginal distribution of X FXY (þ1,y) ¼ FY (y), marginal distribution of Y
(17:1:9)
FXY (þ1,þ1) ¼ 1 The marginal density functions are obtained from Eq. (17.1.7) by integration as follows: ð1 fX (x) ¼
ð1 fXY (x,y)dy1 dym
1 ð1
fY (y) ¼
1 ð1
fXY (x,y)dx1 dxn
1
(17:1:10)
1
17.1
Distributions and Densities
313
T Example 17.1.1 Thejoint density T function of a random 2-vector X ¼ X1 X2 and a random 2-vector Y ¼ Y1 Y2 is given by 8 < 0 , x1 1, 0 , x2 2 1 1 , y1 2, 1 , y2 3 fXY (x,y) ¼ (x1 þ 2x2 )(4y1 þ y2 ) , : 80 0 otherwise We have to determine (1) marginal density fX(x), (2) marginal density fY(y), (3) marginal distribution FX(x), (4) marginal distribution FY(y), (5) the distribution FX(x1), and (6) the distribution FX(x2). 1. The marginal density fX(x) is obtained by integrating fXY(x,y) over all y: ð3 ð2 1 1 fX (x) ¼ ½(x1 þ 2x2 )(4y1 þ y2 )dy1 dy2 ¼ (x1 þ 2x2 ) 80 5 1 1 2. The marginal density fY(y) is obtained by integrating fXY(x,y) over all x: ð2 ð1 1 1 fY (y) ¼ ½(x1 þ 2x2 )(4y1 þ y2 )dx1 dx2 ¼ (4y1 þ y2 ) 16 0 0 80 3. The marginal distribution FX(x) is obtained by integrating fX(x) as shown below: ð x2 ð x 1 1 1 2 (j1 þ 2j2 )dj1 dj2 ¼ x1 x2 þ 2x1 x22 FX (x) ¼ 10 0 0 5 4. The marginal distribution FY(y) is obtained by integrating fY(y) as shown below: ð y2 ð y1 1 (4h1 þ h2 )dh1 dh2 FY (y) ¼ 1 1 16 1 4( y2 1Þy21 þ ( y22 1)y1 y22 4y2 þ 5 ¼ 32 5. The distribution FX(x1) is obtained by substituting the end value x2 ¼ 2 in FX(x): 1 2 1 2 x1 x2 þ 2x1 x2 FX (x1 ) ¼ ¼ x21 þ 4x1 10 5 x2 ¼2 6. The distribution FX(x2) is obtained by substituting the end value x1 ¼ 1 in FX(x): 1 2 1 2 x1 x2 þ 2x1 x22 2x2 þ x2 FX (x2 ) ¼ ¼ 10 10 x1 ¼1 The random vectors X and Y are independent since fXY(x,y) ¼ fX(x) . fY(y), but the random variables X1 and X2 are not independent since FX(x1, x2) = FX(x1) . FX(x2).
Conditional Distributions and Densities
The conditional distribution function FXjA (xjA) given that an event A has occurred is defined by FX j A (x j A) ¼ P(X x j A) ¼
P(X x, A) , P(A) = 0 P(A)
(17:1:11)
314
Random Vectors and Mean-Square Estimation
Analogous to the one-dimensional case, we can also define a total probability. If there are disjoint events fAi, i ¼ 1, . . . , ng such that n [
Ai ¼ S, and
\
Ai Aj ¼ 1
i=j
i¼0
then the total probability is given by FX (x) ¼ P(X x) ¼
n X
P(X x j Ai )P(Ai ) ¼
i¼1
n X
FX j Ai (x j Ai )P(Ai )
(17:1:12)
i¼1
The density functions corresponding to Eqs. (17.1.11) and (17.1.12) are given by @n FX j A (x j A) , P(A) = 0 @x1 @xn n X fX j Ai (x j Ai )P(Ai ) fX (x) ¼
fX j A (x j A) ¼
(17:1:13) (17:1:14)
i¼1
If there are events A and B defined by A: x , X x þ Dx and B : y , Y y þ Dy, then the probability of B occurring conditioned on A is given by P(B j A) ¼ P(y , Y y þ Dy j x , X x þ Dx) ¼ fY j X (y j X)Dy ¼
P(y , Y y þ Dy, x , X x þ Dx) P(x , X x þ Dx)
¼
fXY (x,y)Dx Dy fXY (x,y)Dy ¼ fX (x)Dx fX (x)
(17:1:15)
From Eq. (17.1.15) we can conclude that the conditional density fY j X (y j X) is given by fY j X (y j X) ¼
fXY (x,y) fX (x)
(17:1:16)
The corresponding expression for the conditional distribution FY j X (Y j X x) is FY j X (y j X x) ¼
P(Y y, X x) FYX (y,x) ¼ P(X x) FX (x)
(17:1:17)
Example 17.1.2 In Example 17.1.1 we established that the random variables X and Y are independent and the random variables X1 and X2 are not independent. We will now find the conditional density fX2 j X1 (x2 j X1 ¼ x1 ). By Eq. (17.1.16), we have fX2 j X1 (x2 j X1 ¼ x1 ) ¼
fX2 X1 (x2 , x1 ) fX1 (x1 )
(x1 þ 2x2 ) x1 þ 2x2 5 ¼ ¼ 2(x1 þ 2) 2(x1 þ 2) 5 and the corresponding conditional distribution FX2 j X1 (x2 j X1 ¼ x1 ) is obtained by integrating the conditional density and is given by ð x2 x1 þ 2j2 x2 (x1 þ x2 ) FX2 j X1 (x2 j X1 ¼ x1 ) ¼ dj2 ¼ 2(x1 þ 2) 2(x þ 2) 1 0
17.1
Distributions and Densities
315
Note that when the end value of x2 ¼ 2 is substituted in the distribution shown above, we obtain FX2 j X1 (2 j X1 ¼ x1 ) ¼ 1, as we should. We can also find the conditional distribution FX2 j X1 (x2 j X1 x1 ) as follows: FX2 j X1 (x2 j X1 x1 ) ¼
P(X2 x2 , X1 x1 ) FX2 X1 (x2 , x1 ) ¼ P(X1 x1 ) FX1 (x1 )
1 2 (x1 x2 þ 2x1 x22 ) x (x þ 2x ) 2 1 2 10 ¼ ¼ 1 2 2(x1 þ 4) (x þ 4x1 ) 5 1 Here also if the end value of x2 ¼ 2 is substituted in the preceding distribution, we obtain FX2 j X1 (2 j X1 x1 ) ¼ 1, as we should. By differentiating the preceding equation with respect to x2, we can find the corresponding conditional density function fX2 j X1 (x2 j X1 x1 ). fX2 j X1 (x2 j X1 x1 ) ¼
d x2 (x1 þ 2x2 ) x1 þ 4x2 ¼ dx2 2(x1 þ 4) 2(x1 þ 4)
The difference between these two conditional distributions is to be particularly noted. In the first case the conditioning event is fX1 ¼ x1g, and in the second case it is fX1 x1g.
Points of Clarity
In Eq. (8.2.7) we noted some points of clarity regarding conditional densities and distributions. We will elaborate on this point. The joint density function fXY(x,y) is defined in Eq. (9.2.3) but can also be defined as an infinitesimal probability fXY (x,y)Dx Dy ¼ P(x , X x þ Dx, y , Y y þ Dy)
(17:1:18)
and the corresponding distribution function FXY(x,y), by FXY (x,y) ¼ P(X x, Y y) which can also be obtained by integrating fXY(x,y) over x and y: ðx ðy FXY (x,y) ¼ fXY (j,h)dh dj
(9:2:2)
(17:1:19)
1 1
The function fXY(x,y) can also be obtained by differentiating FXY(x,y) as given in Eq. (9.2.4): fXY (x,y) ¼
@2 FXY (x,y) @x @y
(9:2:4)
There is a need for defining another density – distribution function fXY (X x,y) that can be used in formulating directly conditional density problems involving the conditioning event such as fX x): fXY (X x,y) ¼ lim
Dy!0
P(X x,y , Y y þ Dy) Dy
(17:1:20)
316
Random Vectors and Mean-Square Estimation
Equation (17.1.20) can be obtained by integrating fXY(x,y) over x as shown below: ðx fXY (j,y)dj (17:1:21) fXY (X x,y) ¼ 1
We can integrate Eq. (17.1.21) over y and obtain FXY (x,y): ðy FXY (x,y) ¼ fXY (X x,h)dh
(17:1:22)
1
The conditonal density fY j X (y j X x) can be determined as follows: fY j X (y j X x)Dy ¼ P(y , Y y þ Dy) j X x) ¼
P(y , Y y þ Dy, X x) P(X x)
¼
fYX (y, X x)Dy FX (x)
Hence fY j X (y j X x) ¼
fYX (y, X x) FX (x)
(17:1:23)
Example 17.1.3 We will continue the previous example (Example 17.1.2) and find the conditional density fX2 j X1 (x2 j X1 x1 ). From Eq. (17.1.23) we have fX2 j X1 (x2 j X1 x1 ) ¼
fX2 X1 (x2 , X1 x1 ) FX1 (x1 )
From Eq. (17.1.21) we have ð x1 j1 þ 2x2 x1 (x1 þ 4x2 ) dj1 ¼ fX2 X1 (x2 ,X1 x1 ) ¼ 10 5 0
and
FX1 (x1 ) ¼
x21 þ 4x1 5
Hence x1 (x1 þ 4x2 ) x1 þ 4x2 fX2 j X1 (x2 j X1 x1 ) ¼ 2 10 ¼ 2(x1 þ 4) x1 þ 4x1 5 By integrating fX2 j X1 (x2 j X1 x1 ) with respect to x2, we obtain the conditional distribution FX2 j X1 (x2 j X1 x1 ) ¼
x2 (x1 þ 2x2 ) 2(x1 þ 4)
Bayes’ Theorem
Similar to the one-dimensional case, we can formulate a vector form of Bayes’ theorem given that the event Ai has occurred. We want to find the a posteriori probability P(Ai j X x) given that the event fX xg has occurred. This probability is given by P(Ai j X x) ¼
P(X x, Ai ) P(X x j Ai )P(Ai ) ¼ P(X x) P(X x)
17.1
Distributions and Densities
317
Substituting for P(X x) from Eq. (17.1.12), we obtain the vector form of Bayes’ theorem: P(X x j Ai )P(Ai ) P(Ai j X x) ¼ Pn k¼1 P(X x j Ak )P(Ak )
(17:1:24)
We can also formulate Bayes’ rule for random vector densities. If fXY(x,y) represents the joint density associated with random vectors X and Y, then, from Eq. (17.1.16), we obtain fXY (x,y) ¼ fY j X (y j X ¼ x)fX (x) and Bayes’ theorem can be formulated for fX j Y(x j Y ¼ y) as follows: fX j Y (x j Y ¼ y) ¼
fY j X (y j X ¼ x)fX (x) fY (y) fY j X (y j X ¼ x)fX (x) 0 0 0 1 fY j X (y j X ¼ x )fX (x )dx
¼ Ð1
(17:1:25)
In many engineering estimation problems the random vector X cannot be observed but the random vector Y connected to X can be observed. In this connection fX(x) is called the a priori density and fX j Y(x j Y ¼ y) is called the a posteriori density after observing fY ¼ yg.
The density fX(x) of a two-dimensional random vector X is given by
Example 17.1.4
fX (x) ¼
4x1 þ 7x2 , 0 , x1 1, 0 , x2 2 18
The conditional density fY j X(y j x) of another two-dimensional random vector Y is fY j X (y j x) ¼
4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 , 5(4x1 þ 7x2 )
0 , x1 1, 0 , x2 2 1 , y1 2, 1 , y2 3
Given this information, we have to find the a posteriori density fX j Y(x j y) using Bayes’ theorem of Eq. (17.1.24). The joint density function fYX(x,y) is given by fYX (y,x) ¼ fY j X (y j x)fX (x) 4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 4x1 þ 7x2 5(4x1 þ 7x2 ) 18 0 , x1 1, 0 , x2 2 1 ¼ (4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 ), 90 1 , y1 2, 1 , y2 3
¼
The marginal density function fY(y) is obtained by integrating the preceding equation with respect to x:
1 7y1 þ 6y2 4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 dx2 dx1 ¼ 45 0 90
ð1 ð2 fY (y) ¼ 0
318
Random Vectors and Mean-Square Estimation
The a posteriori density can be obtained from Eq. (17.1.25) as fX j Y (x j y) ¼
fY j X (y j x)fX (x) fY (y)
4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 4x1 þ 7x2 5(4x1 þ 7x2 ) 18 ¼ 7y1 þ 6y2 45 0 , x1 , 1, 0 , x2 2 4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 , ¼ 2(7y1 þ 6y2 ) 1 , y1 2, 1 , y2 3 which, when integrated over all x, indeed gives the result 1. Example 17.1.5
The density fX(x) of a two-dimensional vector X is given by fX (x) ¼ a1 d(x1 m1 , x2 m2 ) þ a2 d(x1 m1 , x2 m2 )
where a1 þ a2 ¼ 1, d(x1,x2) is a two-dimensional Dirac delta function and m1,m2 are constants. The conditional density of a two-dimensional random vector Y correlated with X is 1 (y1 x1 )2 þ (y2 x2 )2 e fY j X (y j x) ¼ p From Eq. (17.1.6) the joint density function fYX(x,y) is given by fYX (y,x) ¼ fY j X (y j x)fX (x)
1 (y1 x1 )2 þ(y2 x2 )2 e a1 d(x1 m1 , x2 m2 ) þ a2 d(x1 m1 , x2 m2 ) ¼ p
1 (y1 m1 )2 þ(y2 m2 )2 ¼ e a1 d(x1 m1 , x2 m2 ) þ a2 d(x1 m1 , x2 m2 ) p The marginal density function fY(y) is obtained by integrating this equation with respect to x: ð1 ð1 a1 d(x1 m1 , x2 m2 ) 1 (y1 m1 )2 þ(y2 m2 )2 e fY (y) ¼ dx1 dx2 þa2 d(x1 m1 , x2 m2 ) 1 1 p 1 (y1 m1 )2 þ(y2 m2 )2 (y1 m1 )2 þ(y2 m2 )2 a1 e þ a2 e ¼ p The a posteriori density can be obtained from Eq. (17.1.25) as fy j x (y j x)fX (x) fY (y)
(y1 m1 )2 þ(y2 m2 )2 a1 d(x1 m1 , x2 m2 ) þ a2 d(x1 m1 , x2 m2 ) e ¼ 2 2 2 2 a1 e (y1 m1 ) þ(y2 m2 ) þ a2 e (y1 m1 ) þ(y2 m2 )
fx j y (x j y) ¼
¼ b1 (y)d(x1 m1 , x2 m2 ) þ b2 (y)d(x1 m1 , x2 m2 ) where 2
2
ai e½(y1 m1 ) þ(y2 m2 ) , bi (y) ¼ 2 2 2 2 a1 e (y1 m1 ) þ(y2 m2 ) þ a2 e (y1 m1 ) þ(y2 m2 )
i ¼ 1,2
17.2
B 17.2
Moments of Random Vectors
319
MOMENTS OF RANDOM VECTORS
Expectation Vector
Analogous to the single-dimensional case, the expected value of a random n-vector X is another n-vector mX defined by ð1 ð1 x f X (x)dx1 dxn (17:2:1) E½X ¼ mX ¼ 1
1
and ð1 E½Xi ¼ mXi ¼ ð1
ð1 ...
1
xi f X (x)dx1 dxi dxn 1
xi f Xi (xi )dxi
¼
(17:2:2)
1
since ð1
ð1 fX (x)dx1 dxi1 dxiþ1 dxn ¼ 1
1
1
Hence the mean value n-vector mX can be written as T mX ¼ mX1 mXi mXn
(17:2:3)
The term X is a 3-vector whose pdf fX(x) is given by 2 2x 3 2e 1 4 fX (x) ¼ 3e3x25, x1 ,x2 ,x3 . 0 4e4x3
Example 17.2.1
The mean value vector mX is: 2 3 1 2 3 627 E(X1 ) 6 7 617 mX ¼ 4E(X2 )5 ¼ 6 7 637 E(X3 ) 415 4 Correlation Matrix
The correlation matrix is a square matrix RX representing the set of all second moments of the components of a real random vector X of the form ð1 xi xj fXi Xj (xi , xj )dxi dxj , i = j, i, j ¼ 1, . . . , n (17:2:4) E½Xi X j ¼ 1
and is defined by 2
E½X12
E½X1 X2
6 6 E½X2 X1 E½X22 E½XX ¼ RX ¼ 6 6 .. .. 4 . . E½Xn X1 E½Xn X2 T
E½X1 Xn
3
7 E½X2 Xn 7 7 7 .. 5 . 2 E½Xn
(17:2:5)
320
Random Vectors and Mean-Square Estimation
The quantity E[XX T] is defined as the outer product of the vector X, which is equal to the P correlation matrix RX. The inner product RX is defined by E½XT X ¼ ni¼1 E½Xi2 , which is equal to the trace of RX or E[X TX] ¼ Tr[RX]. The expected value of a scalar function g(X) of a vector random variable X can be defined as follows: ð1 ð1 ð1 E½g(X) ¼ g(X)fX (x)dx ¼ g(X)fX (x)dx1 dxn (17:2:6) 1
1
1
Example 17.2.2 The correlation matrix of Example 17.2.1 can be obtained from Eq. 17.2.4 assuming that X1, X2,X3 are independent and hence the joint densities fXi Xj (xi , xj ) are given by the product of individual densities as shown below: fXi Xj (xi , xj ) ¼ fXi (xi )fXj (xj ), i = j, i, j ¼ 1,2,3 or fX1 X2 (x1 , x2 ) ¼ 6e(2x1 þ3x2 ) u(x1 )u(x2 ) fX2 X3 (x2 , x3 ) ¼ 12e(3x2 þ4x3 ) u(x2 )u(x3 ) fX1 X3 (x1 , x3 ) ¼ 8e(2x1 þ4x3 ) u(x1 )u(x3 ) Hence RX is given by 2
E½X12 6 RX ¼ 4 E½X2 X1 E½X3 X1
E½X1 X2 E½X22 E½X3 X2
2
1 6 E½X1 X3 62 7 61 E½X2 X3 5 ¼ 6 66 41 E½X32 8 3
3 1 1 6 8 7 7 2 1 7 7 9 12 7 1 1 5 12 8
The trace of the matrix Tr(Rx Þ ¼ 12 þ 29 þ 18 ¼ 61 72. We note that the correlation matrix is symmetric. Covariance Matrix
The covariance matrix CX associated with a real random value of the outer product of the vector (X 2 mX) and is given 2 2 sX1 sX1 X2 sX1 Xj 6 6 sX2 X1 s2X2 sX2 Xj 6 6 . .. .. 6 .. . . T CX ¼ E½X mX ½X mX ¼ 6 6s 6 Xi X1 sXi X2 sXi Xj 6 .. .. 6 .. 4 . . . sXn X1 sXn X2 sXn Xj
vector X is the expected by 3 sX 1 X n 7 sX 2 X n 7 7 .. 7 . 7 7 (17:2:7) sXi Xn 7 7 7 .. 7 . 5 s2Xn
where sXi Xj is the covariance between the random variables Xi and Xj given by sXi Xj ¼ E(Xi mXi )(Xj mXj )
(17:2:8)
Equation (17.2.7) can be expanded to express the covariance matrix CX in terms of the correlation matrix RX: CX ¼ E XXT E X mTX mX E XT þ mX mTX ¼ E XXT mX mTX ¼ RX mX mTX (17:2:9)
17.2
Moments of Random Vectors
321
Example 17.2.3 We will find the covariance matrix for Examples 17.2.1 and 17.2.2. The correlation matrix RX has already been found. The matrix mX mTX is given by 2 3 1 2 3 E(X1 ) 6 2 7 7 6 7 6 617 1 1 1 6 7 T mX mX ¼ 6 E(X2 ) 7 E(X1 ) E(X2 ) E(X3 ) ¼ 6 7 637 2 3 4 4 5 415 E(X3 ) 4 2 3 1 1 1 64 6 8 7 6 7 1 7 61 1 ¼6 7 6 6 9 12 7 41 1 1 5 8 12 16 From Eq. (17.2.9) the covariance matrix can be given by 2 3 2 1 1 1 1 1 62 6 6 7 8 7 64 6 6 1 7 61 1 61 2 T CX ¼ RX mX mX ¼ 6 76 6 6 9 12 7 6 6 9 41 1 1 5 41 1 8 12 8 8 12
3 2 1 1 6 7 8 7 64 1 7 6 7 ¼ 60 12 7 6 1 5 4 0 16
0 1 9 0
0
3
7 7 7 0 7 7 1 5 16
Since the cross-terms are zero, the random variables X1,X2, and X3 are pairwise uncorrelated. Joint Moments of Two Random Vectors
Given a random n-vector X and m-vector Y, with joint density fXY(x,y), the expectation of a scalar function g(X,Y) of the random vectors X,Y is defined by ð1 ð1 g(x,y)fXY (x,y)dx dy E½g(X,Y) ¼ 1
1
1
1
ð1 ð1
g(x,y)fXY (x,y)dx1 dxn dy1 dym
¼
(17:2:10)
The cross-correlation matrix RXY represents the set of all joint moments of the components of real random vectors X and Y of the form ð1 xi yj fXi Yj (xi ,yj )dxi dyj , i ¼ 1, . . . , n, j ¼ 1, . . . ,m (17:2:11) E½Xi Yj ¼ 1
and is defined by the outer product E[XY T] as follows: 2 3 E½X1 Y1 E½X1 Y2 E½X1 Ym 6 7 6 E½X2 Y1 E½X2 Y2 E½X2 Ym 7 6 7 E½XYT ¼ RXY ¼ 6 7 .. .. .. 6 7 . . . 4 5 E½Xn Y1 E½Xn Y2 E½Xn Ym
(17:2:12)
It is not a square matrix unless mP¼ n. The inner product E[X TY] is not defined unless m ¼ n, in which case E½XT Y ¼ ni¼1 E½Xi Yi .
322
Random Vectors and Mean-Square Estimation
Cross-Covariance Matrix
The cross-covariance matrix CXY associated with random n-vector X and a random m-vector Y is expected value of the outer product (X 2 mX) and (Y 2 mY) and is given by 2
CXY
sX 1 Y1 6 sX 2 Y1 6 6 . 6 .. 6 T ¼ E½X mX ½Y mY ¼ 6 6 sXi Y1 6 6 .. 4 . sX n Y1
sX 1 Y2 sX 2 Y2 .. . sX i Y2 .. . sX n Y2
sX1 Yj sX2 Yj .. . sX i Yj .. . sXn Yj
3 sX 1 Ym sX 2 Ym 7 7 .. 7 . 7 7 (17:2:13) sXi Ym 7 7 7 .. 7 . 5 sX n Ym
Again CXY is not a square matrix unless m ¼ n. If m ¼ n then Tr½CXY ¼ Sni¼1 E½Xi Yi . Similar to Eq. (17.2.9), Eq. (17.2.13) can be expanded to express the cross-covariance matrix CXY in terms of the cross-correlation matrix RXY: CXY ¼ E XYT E X mTY mX E YT þ mX mTY ¼ E XYT mX mTY ¼ RXY mX mTY
(17:2:14)
Example 17.2.4 We will now find the cross-covariance matrix of the two-dimensional vectors X and Y given the joint density
fXY (x,y) ¼
1 (4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 ), 90
0 , x1 1, 0 , x2 2 1 , y1 2, 1 , y2 3
We will first find the four densities, fX1 (x1 ), fX2 (x2 ), fY1 (y1 ), fY2 (y2 ) as follows: ð2 ð2 ð3 1
1 4x1 þ 7 (4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 )dy2 dy1 dx2 ¼ 9 1 90
1
1 7x2 þ 2 (4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 )dy2 dy1 dx1 ¼ 90 18 1
0
1 14y1 þ 24 (4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 )dy2 dx2 dx1 ¼ 90 45 1
0
1 4y1 þ 7 (4x1 y1 þ 2x1 y2 þ 5x2 y1 þ 5x2 y2 )dy1 dx2 dx1 ¼ 90 30 1
fX1 (x1 ) ¼ 0
ð1 ð2 ð3 fX2 (x2 ) ¼ 0
ð1 ð2 ð3 fY1 (y1 ) ¼ 0
ð1 ð2 ð2 fY2 (y2 ) ¼ 0
The mean vectors mX,mY and the outer product mXmTY can be obtained from the densities as follows: 2
3 29 6 54 7 7 mX ¼ 6 4 5 5, 27
2
3 134 6 45 7 7 mY ¼ 6 4 119 5, 180
2
3 1943 3451 6 1215 9720 7 7 mX mTY ¼ 6 4 134 119 5 243 972
17.3
Vector Gaussian Random Variables
323
To find the cross-correlation matrix RXY, we need the joint densities fX1 Y1 (x1 ,y1 ), fX1 Y2 (x1 ,y2 ), fX2 Y1 (x2 ,y1 ), fX2 Y2 (x2 ,y2 ). They can be determined from fXY(x,y) and are shown below: 8x1 y1 þ 8x1 þ 10y1 þ 20 45 4x1 y2 þ 12x1 þ 10y2 þ 15 fX1 Y2 (x1 ,y2 ) ¼ 90 5x2 y1 þ 10x2 þ 2y1 þ 2 fX2 Y1 (x2 ,y1 ) ¼ 45 10x2 y2 þ 15x2 þ 2y2 þ 6 fX2 Y2 (x2 ,y2 ) ¼ 180 fX1 Y1 (x1 ,y1 ) ¼
We can now find the various joint expectations as follows: 332 405 778 E½X2 Y1 ¼ 405 E½X1 Y1 ¼
454 405 , 1066 E½X2 Y2 ¼ 405
2
E½X1 Y2 ¼
hence
RXY
3 454 405 7 7 1066 5 405
332 6 405 ¼6 4 778 405
From Eq. (17.2.14) we can evaluate CXY ¼ RXY mX mTY : 2
CXY
3 2 332 454 1943 6 405 405 7 6 1215 7 6 ¼6 4 778 1066 5 4 134 243 405 405 1:516 0:766 ¼ 1:370 2:510
3 2 163,963 3451 6 108,135 9720 7 7¼6 119 5 4 1664 972 1215
3 1489 1944 7 7 12,197 5 4860
Definitions 1. Two random vectors X and Y are independent if fXY(x,y) ¼ fX(x) fY(y). 2. Two random vectors X and Y are uncorrelated if RXY ¼ mXmTY, or CXY ¼ 0. 3. Two random vectors X and Y are orthogonal if RXY ¼ 0. The definitions and properties of correlation and covariance matrices for two real random vectors X and Y are shown in Table 17.2.1.
B 17.3
VECTOR GAUSSIAN RANDOM VARIABLES
Gaussian random variables play an important role in signal processing and communication problems. For a real n-vector random variable, the multivariate Gaussian density takes the form fX (X) ¼
1 e(1=2) (x mx )T C1 X (x mx ) (2p)n=2 j CX j1=2
(17:3:1)
324
Random Vectors and Mean-Square Estimation
TABLE 17.2.1 Property
Correlation functions
Covariance functions
Autocorrelation and Autocovariance RX ¼ E[XX T] RX ¼ RTX Positive definite Greater than 0 Orthogonal Greater than 0 RX ¼ CX þ mXmTX
Definition Symmetry Definiteness Eigenvalues Eigenvectors Principal diagonals Relationship
CX ¼ E[(X 2 mX) (X 2 mX)T] CX ¼ CTX Positive definite Greater than 0 Orthogonal Greater than 0 CX ¼ RX 2 mXmTX
Cross-Correlation and Cross-Covariance RXY ¼ E[XY T] RXY ¼ RTYX Nondefinite RXY ¼ CXY þ mXmTY RXY ¼ mXmTY RXY ¼ 0 ¼ RYX RXþY ¼ RX þ RXY þ RYX þ RY RXþY ¼ RX þ RY if X, Y are orthogonal
Definition Symmetry Definiteness Relationship Uncorrelated Orthogonal Sum of X and Y
CXY ¼ E[(X 2 mX) (Y 2 mY)T] CXY ¼ CTYX Nondefinite CXY ¼ RXY 2 mXmTY CXY ¼ 0 ¼ CYX CXþY ¼ CX þ CXY þ CYX þ CY CXþY ¼ CX þ CY if X, Y are uncorrelated
The joint density of two random variables X and Y is given by a bivariate Gaussian as in Eq. (10.5.1) ( "
1 1 x mx 2 pffiffiffiffiffiffiffiffiffiffiffiffiffi exp fXY (x,y) ¼ 2(1 r2 ) sx 2psx sy 1 r2 #)
(x mx )(y my ) y my 2 2r þ sy s x sy
(10:5:1)
where r is the correlation coefficient between random variables X and Y given by r ¼ sxy =(sx sy ). This can also be expressed in the matrix form of Eq. (17.3.1) with n ¼ 2 ( 1 1 ½x mx fXY (x,y) ¼ pffiffiffiffiffiffiffiffiffi exp 2 2p jCX j
s2 y my X rsX sY
rsX sY s2Y
1
x mx y my
)!
(17:3:2) where the covariance matrix CX, its inverse CX 21, and the mean vector m are given by " CX ¼
s2X rsX sY
rsX sY s2Y
2
# ,
3 1 r 6 s2X (1 r2 ) sX sY (1 r2 ) 7 6 7, C1 X ¼4 5 r 1 sX sY (1 r2 ) s2Y (1 r2 )
mX m¼ mY
(17:3