Probability Distributions Used in Reliability Engineering

Probability Distributions Used in Reliability Engineering

Probability Distributions Used in Reliability Engineering

Andrew N. O’Connor

Center for Risk and Reliability 0151 Glenn L Martin Hall University of Maryland College Park, Maryland

Published by the Reliability Information Analysis Center (RIAC) ISBN-10: ISBN-13:

1-933904-06-2 978-1-933904-06-1

(Hardcopy) (Hardcopy)

ISBN-10: ISBN-13:

1-933904-07-0 978-1-933904-07-8

(Electronic) (Electronic)

Copyright © 2011 by the Center for Reliability Engineering University of Maryland, College Park, Maryland, USA All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from The Center for Reliability Engineering, Reliability Engineering Program.

The Center for Risk and Reliability University of Maryland College Park, Maryland 20742-7531

In memory of Willie Mae Webb

This book is dedicated to the memory of Miss Willie Webb who passed away on April 10 2007 while working at the Center for Risk and Reliability at the University of Maryland (UMD). She initiated the concept of this book, as an aid for students conducting studies in Reliability Engineering at the University of Maryland. Upon passing, Willie bequeathed her belongings to fund a scholarship providing financial support to Reliability Engineering students at UMD. The proceeds from this book will go directly towards the Miss Willie M. Webb Scholarship.

Preface Reliability Engineers are required to combine a practical understanding of material science and engineering with statistics. The reliability engineer’s understanding of statistics is focused on the practical application of a wide variety of accepted statistical methods. Most reliability texts provide only a basic introduction to probability distributions or only provide a detailed reference to few distributions. Detailed statistician texts provide theoretical detail which is outside the scope of likely reliability engineering tasks. As such the objective of this book is to provide a single reference text of closed form probability formulas and approximations used in reliability engineering. This book provides details on 22 probability distributions. Each distribution section provides a graphical visualization and formulas for distribution parameters, along with distribution formulas. Common statistics such as moments and percentile formulas are followed by likelihood functions and in many cases the derivation of maximum likelihood estimates. Bayesian non-informative and conjugate priors are provided followed by a discussion on the distribution characteristics and applications in reliability engineering. Each section is concluded with online and hardcopy references which can provide further information followed by the relationship to other distributions. The book is divided into six parts. Part 1 provides a brief coverage of the fundamentals of probability distributions within a reliability engineering context. Part 1 is limited to concise explanations aimed to familiarize readers. For further understanding the reader is referred to the references. Part 2 to Part 6 cover Common Life Distributions, Bathtub Distributions, Univariate Continuous Distributions, Univariate Discrete Distributions and Multivariate Distributions respectively. This book refers to software available at www.enre.umd.edu/tools.htm aimed at students and academic staff which allow users to: • Plot distributions and interactively vary distribution parameters to provide understanding on the effect each parameter has on the probability distribution function, cumulative distribution function and hazard rate. • Export probability distribution plots directly to the printer, picture or to a chart object in Microsoft Excel. The authors would like to thank the many students in the Reliability Engineering Program and Reuel Smith for proof reading.

Andrew N. O’Connor University of Maryland Jun 2011 [email protected]

Contents

i

Contents PREFACE .................................................................................................................................................. V CONTENTS ................................................................................................................................................I 1.

FUNDAMENTALS OF PROBABILITY DISTRIBUTIONS .................................................... 1

1.1. Probability Theory .......................................................................................................... 2 1.1.1. Theory of Probability ................................................................................................ 2 1.1.2. Interpretations of Probability Theory ....................................................................... 2 1.1.3. Laws of Probability.................................................................................................... 3 1.1.4. Law of Total Probability ............................................................................................ 4 1.1.5. Bayes’ Law ................................................................................................................ 4 1.1.6. Likelihood Functions ................................................................................................. 5 1.1.7. Fisher Information Matrix ......................................................................................... 6 1.2. Distribution Functions .................................................................................................... 9 1.2.1. Random Variables ..................................................................................................... 9 1.2.2. Statistical Distribution Parameters ........................................................................... 9 1.2.3. Probability Density Function ..................................................................................... 9 1.2.4. Cumulative Distribution Function ........................................................................... 11 1.2.5. Reliability Function ................................................................................................. 12 1.2.6. Conditional Reliability Function .............................................................................. 13 1.2.7. 100α% Percentile Function ..................................................................................... 13 1.2.8. Mean Residual Life.................................................................................................. 13 1.2.9. Hazard Rate ............................................................................................................ 13 1.2.10. Cumulative Hazard Rate ......................................................................................... 14 1.2.11. Characteristic Function ........................................................................................... 15 1.2.12. Joint Distributions ................................................................................................... 16 1.2.13. Marginal Distribution .............................................................................................. 17 1.2.14. Conditional Distribution.......................................................................................... 17 1.2.15. Bathtub Distributions.............................................................................................. 17 1.2.16. Truncated Distributions .......................................................................................... 18 1.2.17. Summary ................................................................................................................. 19 1.3. Distribution Properties ................................................................................................. 20 1.3.1. Median / Mode ....................................................................................................... 20 1.3.2. Moments of Distribution ........................................................................................ 20 1.3.3. Covariance .............................................................................................................. 21 1.4. Parameter Estimation ................................................................................................... 22 1.4.1. Probability Plotting Paper ....................................................................................... 22 1.4.2. Total Time on Test Plots ......................................................................................... 23 1.4.3. Least Mean Square Regression ............................................................................... 24 1.4.4. Method of Moments .............................................................................................. 25 1.4.5. Maximum Likelihood Estimates .............................................................................. 26 1.4.6. Bayesian Estimation ................................................................................................ 27

ii

Probability Distributions Used in Reliability Engineering

1.4.7. 1.5.

Confidence Intervals ............................................................................................... 30 Related Distributions .................................................................................................... 33

1.6. Supporting Functions .................................................................................................... 34 1.6.1. Beta Function 𝐁𝒙, 𝒚 ................................................................................................ 34 1.6.2. Incomplete Beta Function 𝑩𝒕(𝒕; 𝒙, 𝒚) .................................................................... 34 1.6.3. Regularized Incomplete Beta Function 𝑰𝒕(𝒕; 𝒙, 𝒚) ................................................. 34 1.6.4. Complete Gamma Function 𝚪(𝒌) ........................................................................... 34 1.6.5. Upper Incomplete Gamma Function 𝚪(𝒌, 𝒕) .......................................................... 35 1.6.6. Lower Incomplete Gamma Function 𝛄(𝒌, 𝒕) .......................................................... 35 1.6.7. Digamma Function 𝝍𝒙 ........................................................................................... 36 1.6.8. Trigamma Function 𝝍′𝒙 ......................................................................................... 36 9T

9T

9T

9T

9T

9T

9T

9T

1.7. Referred Distributions .................................................................................................. 37 1.7.1. Inverse Gamma Distribution 𝑰𝑮(𝜶, 𝜷) ................................................................... 37 1.7.2. Student T Distribution 𝑻(𝜶, 𝝁, 𝝈𝟐) ........................................................................ 37 1.7.3. F Distribution 𝑭(𝒏𝟏, 𝒏𝟐) ........................................................................................ 37 1.7.4. Chi-Square Distribution 𝝌𝟐(𝒗) ............................................................................... 37 1.7.5. Hypergeometric Distribution 𝑯𝒚𝒑𝒆𝒓𝑮𝒆𝒐𝒎(𝒌; 𝒏, 𝒎, 𝑵) ...................................... 38 1.7.6. Wishart Distribution 𝑾𝒊𝒔𝒉𝒂𝒓𝒕𝒅(𝒙; 𝚺, 𝒏) .............................................................. 38 9T

9T

9T

9T

9T

9T

1.8. 2.

Nomenclature and Notation ......................................................................................... 39

COMMON LIFE DISTRIBUTIONS ......................................................................................... 40

2.1.

Exponential Continuous Distribution ............................................................................ 41

2.2.

Lognormal Continuous Distribution .............................................................................. 49

2.3.

Weibull Continuous Distribution .................................................................................. 59

3.

BATHTUB LIFE DISTRIBUTIONS ........................................................................................ 68

3.1.

2-Fold Mixed Weibull Distribution ................................................................................ 69

3.2.

Exponentiated Weibull Distribution ............................................................................. 76

3.3.

Modified Weibull Distribution ...................................................................................... 81

4.

UNIVARIATE CONTINUOUS DISTRIBUTIONS ................................................................. 85

4.1.

Beta Continuous Distribution ....................................................................................... 86

4.2.

Birnbaum Saunders Continuous Distribution ............................................................... 93

4.3.

Contents iii Gamma Continuous Distribution .................................................................................. 99

4.4.

Logistic Continuous Distribution ................................................................................. 108

4.5.

Normal (Gaussian) Continuous Distribution ............................................................... 115

4.6.

Pareto Continuous Distribution .................................................................................. 125

4.7.

Triangle Continuous Distribution ................................................................................ 131

4.8.

Truncated Normal Continuous Distribution................................................................ 135

4.9.

Uniform Continuous Distribution ............................................................................... 145

5.

UNIVARIATE DISCRETE DISTRIBUTIONS ..................................................................... 151

5.1.

Bernoulli Discrete Distribution ................................................................................... 152

5.2.

Binomial Discrete Distribution .................................................................................... 157

5.3.

Poisson Discrete Distribution ...................................................................................... 165

6.

BIVARIATE AND MULTIVARIATE DISTRIBUTIONS ................................................... 172

6.1.

Bivariate Normal Continuous Distribution .................................................................. 173

6.2.

Dirichlet Continuous Distribution ............................................................................... 181

6.3.

Multivariate Normal Continuous Distribution ............................................................ 187

6.4.

Multinomial Discrete Distribution .............................................................................. 193

7.

REFERENCES .......................................................................................................................... 201

iv

Probability Distributions Used in Reliability Engineering

Introduction

1 Prob Theory

1. Fundamentals of Probability Distributions

Prob Theory

2

Probability Distributions Used in Reliability Engineering

1.1. Probability Theory 1.1.1. Theory of Probability The theory of probability formalizes the representation of probabilistic concepts through a set of rules. The most common reference to formalizing the rules of probability is through a set of axioms proposed by Kolmogorov in 1933. Where 𝐸𝑖 is an event in the event space Ω =∪𝑛𝑖=1 𝐸𝑖 with 𝑓 different events. 0 ≤ 𝑃(𝐸𝑖 ) ≤ 1

𝑃(Ω) = 1 and 𝑃(𝜙) = 0

𝑃(𝐸1 ∪ 𝐸2 ) = 𝑃(𝐸1 ) + 𝑃(𝐸2 )

When 𝐸1 and 𝐸2 are mutually exclusive.

Other representations of probabilistic concepts exist such as fuzzy logic and theory of evidence (Dempster-Shafer model) which do not follow the theory of probability. For a justification of probability theory see (Singpurwalla 2006).

1.1.2. Interpretations of Probability Theory In probability theory there exist many conceptual interpretations. The two most common interpretations are: •

Frequency Interpretation. Using the frequency interpretation of probability, the probability of an event (failure) is defined as: 𝑃(𝐾) = lim

𝑛→∞

𝑓𝑓 𝑓

Also known as the classical approach, this approach assumes there exists an exact probability of an event occurring is 𝑝. The analyst uses the observed frequency of the event to estimate the value of 𝑝. The more historic events that have occurred, the more confident the analyst is of their estimation of 𝑝. This approach does have limitations, for instance when data from events are not available (e.g. no failures occur in a test) 𝑝 cannot be estimated and this method cannot incorporate “soft data” such as expert opinion. •

Subjective Interpretation. The subjective interpretation of probability is also known as personal probability. This method defines the probability of an event as degree of belief the analyst has on an outcome. This means probability is a product of the analyst’s state of knowledge. Any evidence which would change the analyst’s degree of belief must be considered when calculating the probability (including soft evidence). The assumption is made that the probability assessment is made by a coherent person where any coherent person having the same state of knowledge would make the same assessment.

Introduction

3

1.1.3. Laws of Probability The following formulas are used to apply mathematical operations consistent with the theory of probability. Let 𝑋 = 𝐸𝑖 and 𝑌 = 𝐸𝑗 be two events within the sample space Ω where 𝑠 ≠ 𝑗.

Boolean Laws of probability are (Modarres et al. 1999, p.25): 𝑋∪𝑌 =𝑌∪𝑋 𝑋∩𝑌 =𝑌∩𝑋 𝑋 ∪ (𝑌 ∪ 𝑍) = (𝑋 ∪ 𝑌) ∪ 𝑍 𝑋 ∩ (𝑌 ∩ 𝑍) = (𝑋 ∩ 𝑌) ∩ 𝑍 𝑋 ∩ (𝑌 ∪ 𝑍) = (𝑋 ∩ 𝑌) ∪ (𝑋 ∩ 𝑍) 𝑋∪𝑋 =𝑋 𝑋∩𝑋 =𝑋 �=Ω X∪X �=Ø X∩X �=X X ���������� (𝑋 ∪ 𝑌) = 𝑋� ∩ 𝑌� ���������� (𝑋 ∩ 𝑌) = 𝑋� ∪ 𝑌� 𝑋 ∪ (𝑋� ∩ 𝑌) = 𝑋 ∪ 𝑌

Two events are mutually exclusive if: X ∩ Y = Ø,

Commutative Law Associative Law Distributive Law Idempotent Law Complementation Law

De Morgan’s Theorem

𝑃(𝑋 ∩ 𝑌) = 0

Two events are independent if one event 𝑌 occurring does not affect the probability of the second event 𝑋 occurring: 𝑃(𝑋|𝑌) = 𝑃(𝑋)

The rules for evaluating the probability of compound events are: Addition Rule:

Multiplication Rule:

𝑃(𝑋 ∪ 𝑌) = 𝑃(𝑋) + 𝑃(𝑌) − 𝑃(𝑋 ∩ 𝑌) = 𝑃(𝑋) + 𝑃(𝑌) − 𝑃(𝑋) 𝑃(𝑌|𝑋) 𝑃(𝑋 ∩ 𝑌) = 𝑃(𝑋) 𝑃(𝑌|𝑋) = 𝑃(𝑌) 𝑃(𝑋|𝑌)

When 𝑋 and 𝑌 are independent: 𝑃(𝑋 ∪ 𝑌) = 𝑃(𝑋) + 𝑃(𝑌) − 𝑃(𝑋) 𝑃(𝑌) 𝑃(𝑋 ∩ 𝑌) = 𝑃(𝑌) 𝑃(𝑌)

Prob Theory

The subjective interpretation has the flexibility of including many types of evidence to assist in estimating the probability of an event. This is important in many reliability application where the event of interest (e. g, system failure) are rare.

Prob Theory

4

Probability Distributions Used in Reliability Engineering

Generalizations of these equations: 𝑃(𝐸1 ∪ 𝐸2 ∪ … ∪ 𝐸𝑛 ) = [𝑃(𝐸1 ) + 𝑃(𝐸2 ) + ⋯ + 𝑃(𝐸𝑛 )] − [𝑃(𝐸1 ∩ 𝐸2 ) + 𝑃(𝐸1 ∩ 𝐸3 ) + ⋯ + 𝑃(𝐸𝑛−1 ∩ 𝐸𝑛 )] + [𝑃(𝐸1 ∩ 𝐸2 ∩ 𝐸3 ) + 𝑃(𝐸1 ∩ 𝐸2 ∩ 𝐸4 ) + ⋯ ] − ⋯ (−1)𝑛+1 [𝑃(𝐸1 ∩ 𝐸2 ∩ … ∩ 𝐸𝑛 )] 𝑃(𝐸1 ∩ 𝐸2 ∩ … ∩ 𝐸𝑛 ) = 𝑃(𝐸1 ) . 𝑃(𝐸2 |𝐸1 ). 𝑃(𝐸3 |𝐸1 ∩ 𝐸2 ) … 𝑃(𝐸𝑛 |𝐸1 ∩ 𝐸2 ∩ … ∩ 𝐸𝑛−1 )

1.1.4. Law of Total Probability

The probability of 𝑋 is obtained by the summation of its disjointed parts. This can be generalized by defining, 𝐴 = {𝐴1 , 𝐴2 , … , 𝐴𝑛𝐴 } where A is a subset of the sample space, 𝐴 ⊆ Ω and all the elements of A are mutually exclusive, Ai ∩ Aj = Ø, but the union of all A 𝑛𝐴 𝐴𝑖 = Ω. Then: elements cover the complete sample space, ∪𝑖=1 𝑛𝐴

𝑃(𝑋) = � 𝑃(𝐴𝑖 )𝑃(𝑋|𝐴𝑖 ) 𝑖=1

For example:

1.1.5. Bayes’ Law

𝑃(𝑋) = 𝑃(𝑋 ∩ 𝑌) + 𝑃(𝑋 ∩ 𝑌�) = 𝑃(𝑌)𝑃(𝑋|𝑌) + 𝑃(𝑌)𝑃(𝑋|𝑌�)

Bayes’ law, also known as the law of inverse probability, can be derived from the multiplication rule and the law of total probability as follows: 𝑃(𝜃) 𝑃(𝐸|𝜃) = 𝑃(𝐸) 𝑃(𝜃|𝐸) 𝑃(𝜃|𝐸) =

𝑃(𝜃|𝐸) =

𝑃(𝜃) 𝑃(𝐸|𝜃) 𝑃(𝐸)

𝑃(𝜃) 𝑃(𝐸|𝜃) 𝑛𝑌 ∑𝑖=1 𝑃(𝐸|𝜃𝑖 )𝑃(𝜃𝑖 )

If we use this formula for an observed evidence, 𝐸, to make inference about the unobserved 𝜃 the following definitions make up Bayes’ Law: 𝜃

the unknown of interest (UOI).

𝑃(𝜃)

the prior state of knowledge about 𝜃 without the evidence. Also denoted as 𝜋𝑜 (𝜃).

𝐸

𝑃(𝐸|𝜃)

the observed random variable, evidence.

the likelihood of observing the evidence given the UOI. Also denoted as 𝐿(𝐸|𝜃).

Introduction

the posterior state of knowledge about 𝜃 given the evidence. Also denoted as 𝜋(𝜃|𝐸).

𝑌 ∑𝑛𝑖=1 𝑃(𝐸|𝜃𝑖 )𝑃(𝜃) is the normalizing constant.

The continuous form of Bayes’ Law can be written as: 𝜋𝑜 (𝜃) 𝐿(𝐸|𝜃) 𝜋(𝜃|𝐸) = ∫ 𝜋𝑜 (𝜃) 𝐿(𝐸|𝜃) 𝑝𝜃

In Bayesian statistics the state of knowledge (uncertainty) of an unknown of interest (UOI) is quantified by assigning a probability distribution to its possible values. Bayes’ law provides a mathematical means by which this uncertainty can be updated given new evidence.

1.1.6. Likelihood Functions The likelihood function is the probability of observing the evidence (e.g., sample data), 𝐸, given the distribution parameters, 𝜃. The probability of observing event 1 AND event 2 AND event 𝑓 is the product of each event likelihood. The likelihood function is not dependent on the order of each event. 𝐿(𝜃|𝐸) = 𝑐∏𝐿𝑖 (𝜃|𝑡𝑖 )

𝑐 is a combinatorial constant which quantifies the number of combination which the observed evidence could have occurred. Methods which use the likelihood function do not depend on the constant and so it is omitted. The following table summarizes the likelihood functions for different types of observations: Table 1: Summary of Likelihood Functions (Klein & Moeschberger 2003, p.74) Type of Observation Exact Lifetimes Right Censored Left Censored Interval Censored Left Truncated Right Truncated Interval Truncated

Likelihood Function

Example Description

𝐿𝑖 (𝜃|𝑡𝑖 ) = 𝑝(𝑡𝑖 |𝜃)

Failure time is known

𝐿𝑖 (𝜃|𝑡𝑖 ) = 𝐹(𝑡𝑖 |𝜃)

Component failed before time 𝑡𝑖

𝐿𝑖 (𝜃|𝑡𝑖 ) = 𝑅(𝑡𝑖 |𝜃) 𝐿𝑖 (𝜃|𝑡𝑖 ) = 𝐹(𝑡𝑖𝑅𝐼 |𝜃) − 𝐹(𝑡𝑖𝐿𝐼 |𝜃) 𝐿𝑖 (𝜃|𝑡𝑖 ) =

𝐿𝑖 (𝜃|𝑡𝑖 ) =

𝐿𝑖 (𝜃|𝑡𝑖 ) =

𝑝(𝑡𝑖 |𝜃) 𝑅(𝑡𝐿 |𝜃)

𝑝(𝑡𝑖 |𝜃) 𝐹(𝑡𝑈 |𝜃)

𝑝(𝑡𝑖 |𝜃) 𝐹(𝑡𝑈 |𝜃) − 𝐹(𝑡𝐿 |𝜃)

Component survived to time 𝑡𝑖 Component failed between 𝑡𝑖𝐿𝐼 and 𝑡𝑖𝑅𝐼

Component failed at time 𝑡𝑖 where observations are truncated before 𝑡𝐿 .

Component failed at time 𝑡𝑖 where observations are truncated after 𝑡𝑈 .

Component failed at time 𝑡𝑖 where observations are truncated before 𝑡𝐿 and after 𝑡𝑈 .

Prob Theory

𝑃(𝜃|𝐸)

5

Prob Theory

6

Probability Distributions Used in Reliability Engineering

The Likelihood function is used in Bayesian inference and maximum likelihood parameter estimation techniques. In both instances any constant in front of the likelihood function becomes irrelevant. Such constants are therefore not included in the likelihood functions given in this book (nor in most references). For example, consider the case where a test is conducted on 𝑓 components with an exponential time to failure distribution. The test is terminated at 𝑡𝑠 during which 𝑠 components failed at times 𝑡1 , 𝑡2 , … , 𝑡𝑟 and 𝑠 = 𝑓 − 𝑠 components survived. Using the exponential distribution to construct the likelihood function we obtain: 𝐿(𝜆|𝐸) =

𝑛𝑆

𝑛𝐹

� 𝑝�𝜆�𝑡𝑖𝐹 � � 𝑅�𝜆�𝑡𝑖𝑆 � 𝑖=1 𝑖=1 𝑛𝐹

= � 𝜆𝑅 𝑖=1

−𝜆𝑡𝑖𝐹 𝑛𝐹

𝑛𝑆

𝑆

� 𝑅 −𝜆𝑡𝑖 𝑖=1

𝑛𝑆

𝐹

𝑆

= 𝜆𝑛𝐹 𝑅 −𝜆 ∑𝑖=1 𝑡𝑖 𝑅 −λ ∑𝑖=1 𝑡𝑖 𝐹

= 𝜆𝑛𝐹 𝑅 −𝜆�∑ 𝑡𝑖

+∑ 𝑡𝑖𝑆 �

Alternatively, because the test described is a homogeneous Poisson process 1 the likelihood function could also have been constructed using a Poisson distribution. The data can be stated as seeing r failure in time 𝑡𝑇 where 𝑡𝑇 is the total time on test 𝑛𝑆 𝑆 𝑛𝐹 𝐹 𝑡𝑖 + ∑𝑖=1 𝑡𝑖 . Therefore the likelihood function would be: 𝑡𝑇 = ∑𝑖=1 𝐿(𝜆|𝐸) = 𝑝(𝜆|𝑓𝐹 , 𝑡𝑇 ) =

(𝜆𝑡𝑇 )𝑛𝐹 −𝜆𝑡 𝑅 𝑇 𝑓𝐹 !

= 𝑐𝜆𝑛𝐹 𝑅 −𝜆𝑡𝑇

𝐹

= 𝜆𝑛𝐹 𝑅 −𝜆�∑ 𝑡𝑖

+∑ 𝑡𝑖𝑆 �

As mentioned earlier, in estimation procedures within this book, the constant 𝑐 can be ignored. As such, the two likelihood functions are equal. For more information see (Meeker & Escobar 1998, p.36) or (Rinne 2008, p.403).

1.1.7. Fisher Information Matrix The Fisher Information Matrix has many uses but in reliability applications it is most often used to create Jeffery’s non-informative priors. There are two types of Fisher information matrices, the Expected Fisher Information Matrix 𝐼(𝜃), and the Observed Fisher Information Matrix 𝐽(𝜃). 1 Homogeneous in time, where it does not matter if you have 𝑓 components on test at once (exponential test), or you have a single component on test which is replaced after failure 𝑓 times (Poisson process), the evidence produced will be the same.

Introduction

7

For a single parameter distribution: 𝐼(𝜃) = −𝐸 �

2

𝜕 2 Λ(𝜃|𝑚) 𝜕Λ(𝜃|𝑚) � = �� � � 2 𝜕𝜃 𝜕𝜃

where Λ is the log-likelihood function and 𝐸[𝑈] = ∫ 𝑈𝑝(𝑚)𝑝𝑚. For a distribution with 𝑝 parameters the Expected Fisher Information Matrix is: 𝜕 2 Λ(𝜽|𝒙) 𝜕 2 Λ(𝜽|𝒙) 𝜕 2 Λ(𝜽|𝒙) ⎡−𝐸 � �⎤ � −𝐸 � � ⋯ −𝐸 � 2 𝜕𝜃1 𝜕𝜃𝑝 ⎥ 𝜕𝜃1 𝜕𝜃2 𝜕𝜃1 ⎢ ⎢ 𝜕 2 Λ(𝜽|𝒙) 𝜕 2 Λ(𝜽|𝒙) 𝜕 2 Λ(𝜽|𝒙) ⎥ −𝐸 � � −𝐸 � � ⋯ −𝐸 � � ⎢ 2 𝐼(𝜽) = 𝜕𝜃2 𝜕𝜃1 𝜕𝜃2 𝜕𝜃𝑝 ⎥ 𝜕𝜃2 ⎢ ⎥ ⋱ ⋮ ⋮ ⋮ ⎢ ⎥ 2 2 2 ⎢−𝐸 �𝜕 Λ(𝜽|𝒙)� −𝐸 �𝜕 Λ(𝜽|𝒙)� … −𝐸 �𝜕 Λ(𝜽|𝒙)� ⎥ 𝜕𝜃𝑝 𝜕𝜃1 𝜕𝜃𝑝 𝜕𝜃2 𝜕𝜃𝑝2 ⎣ ⎦

The Observed Fisher Information Matrix is obtained from a likelihood function constructed from 𝑓 observed samples from the distribution. The expectation term is dropped. For a single parameter distribution: 𝑛

𝐽𝑛 (𝜃) = − � 𝑖=1

𝜕 2 Λ(𝜃|𝑚𝑖 ) 𝜕𝜃 2

For a distribution with 𝑝 parameters the Observed Fisher Information Matrix is: 𝜕 2 Λ(𝜽|𝒙𝒊 ) 𝜕 2 Λ(𝜽|𝒙𝒊 ) ⎡− − 2 𝜕𝜃1 𝜕𝜃2 𝜕𝜃1 ⎢ 𝑛 ⎢ 𝜕 2 Λ(𝜽|𝒙 ) 𝜕 2 Λ(𝜽|𝒙𝒊 ) 𝒊 − 𝐽𝑛 (𝜽) = � ⎢− 𝜕𝜃2 𝜕𝜃1 𝜕𝜃22 ⎢ 𝑖=1 ⋮ ⋮ ⎢ 2 2 ) 𝜕 Λ(𝜽|𝒙 𝜕 Λ(𝜽|𝒙 𝒊 𝒊) ⎢− − 𝜕𝜃𝑝 𝜕𝜃1 𝜕𝜃𝑝 𝜕𝜃2 ⎣

𝜕 2 Λ(𝜽|𝒙𝒊 ) ⎤ 𝜕𝜃1 𝜕𝜃𝑝 ⎥ 𝜕 2 Λ(𝜽|𝒙𝒊 )⎥ ⋯ − 𝜕𝜃2 𝜕𝜃𝑝 ⎥ ⎥ ⋱ ⋮ ⎥ 2 ) 𝜕 Λ(𝜽|𝒙𝒊 ⎥ … − 𝜕𝜃𝑝2 ⎦ ⋯ −

It can be seen that as 𝑓 becomes large, the average value of the random variable approaches its expected value and so the following asymptotic relationship exists between the observed and expected Fisher information matrices: plim 𝑛→∞

1 𝐽 (𝜽) = 𝐼(𝜽) 𝑓 𝑛

For large n the following approximation can be used: 𝐽𝑛 ≈ 𝑓𝐼(𝜽)

Prob Theory

The Expected Fisher Information Matrix is obtained from a log-likelihood function from a single random variable. The random variable is replaced by its expected value.

Prob Theory

8

Probability Distributions Used in Reliability Engineering

� the observed Fisher information matrix estimates the varianceWhen evaluated at 𝜽 = 𝜽 covariance matrix: � )� 𝑽 = �𝐽𝑛 (𝜽 = 𝜽

−1

𝑉𝑠𝑠(𝜃�1 ) 𝐶𝑓𝑠(𝜃�1 , 𝜃�2 ) ⋯ 𝐶𝑓𝑠(𝜃�1 , 𝜃�𝑑 ) ⎡ ⎤ � � 𝐶𝑓𝑠(𝜃 ⋯ 𝐶𝑓𝑠(𝜃�2 , 𝜃�𝑑 )⎥ , 𝜃 ) 𝑉𝑠𝑠(𝜃�2 ) 1 2 =⎢ ⋱ ⋮ ⋮ ⋮ ⎢ ⎥ … 𝑉𝑠𝑠(𝜃�𝑑 ) ⎦ ⎣𝐶𝑓𝑠(𝜃�1 , 𝜃�𝑑 ) 𝐶𝑓𝑠(𝜃�2 , 𝜃�𝑑 )

9

1.2. Distribution Functions 1.2.1. Random Variables Probability distributions are used to model random events for which the outcome is uncertain such as the time of failure for a component. Before placing a demand on that component, the time it will fail is unknown. The distribution of the probability of failure at different times is modeled by a probability distribution. In this book random variables will be denoted as capital letter such as 𝑇 for time. When the random variable assumes a value we denote this by small caps such as 𝑡 for time. For example, if we wish to find the probability that the component fails before time 𝑡1 we would find 𝑃(𝑇 ≤ 𝑡1 ). Random variables are classified as either discrete or continuous. In a discrete distribution, the random variable can take on a distinct or countable number of possible values such as number of demands to failure. In a continuous distribution the random variable is not constrained to distinct possible values such as time-to-failure distribution. This book will denote continuous random variables as 𝑋 or 𝑇, and discrete random variables as 𝐾.

1.2.2. Statistical Distribution Parameters

The parameters of a distribution are the variables which need to be specified in order to completely specify the distribution. Often parameters are classified by the effect they have on the distributions. Shape parameters define the shape of the distribution, scale parameters stretch the distribution along the random variable axis, and location parameters shift the distribution along the random variable axis. The reader is cautioned that the parameters for a distribution may change depending on the text. Therefore, before using formulas from other sources the parameterization need to be confirmed. Understanding the effect of changing a distribution’s parameter value can be a difficult task. At the beginning of each section a graph of the distribution is show with varied parameters. For a better understanding the reader is referred to the companion software Probability Distribution Plotter, available at www.enre.umd.edu/tools.htm. The reader can use sliders to change parameters and observe the change in the probability density function, cumulative density function or hazard rate.

1.2.3. Probability Density Function A probability density function (pdf), denoted as 𝑝(𝑡) is any function which is always positive and has a unit area: ∞

� 𝑝(𝑡) 𝑝𝑡 = 1, −∞

� 𝑝(𝑘) = 1 𝐸

Dist Functions

Introduction

Dist Functions

10

Probability Distributions Used in Reliability Engineering

The probability of an event occurring between limits a and b is the area under the pdf: 𝑏

𝑃(𝑠 ≤ 𝑇 ≤ 𝑏) = � 𝑝(𝑡) 𝑝𝑡 = 𝐹(𝑏) − 𝐹(𝑠) 𝑎 𝑏

𝑃(𝑠 ≤ 𝐾 ≤ 𝑏) = � 𝑝(𝑘) = 𝐹(𝑏) − 𝐹(𝑠 − 1) 𝑖=𝑎

The instantaneous value of a discrete pdf at 𝑘𝑖 can be obtained by minimizing the limits to [𝑘𝑖−1 , 𝑘𝑖 ]: 𝑃(𝐾 = 𝑘𝑖 ) = 𝑃(𝑘𝑖 < 𝐾 ≤ 𝑘𝑖 ) = 𝑝(𝑘) The instantaneous value of a continuous pdf is infinitesimal. This result can be seen when minimizing the limits to [𝑡, 𝑡 + Δ𝑡]: 𝑃(𝑇 = 𝑡) = lim 𝑃(𝑡 < 𝑇 ≤ 𝑡 + Δ𝑡) = lim 𝑝(𝑡). Δ𝑡 Δt→0

Δt→0

Therefore the reader must remember that in order to calculate the probability of an event, an interval for the random variable must be used. Furthermore, a common misunderstanding is that a pdf cannot have a value above one because the probability of an event occurring cannot be greater than one. As can be seen above this is true for discrete distributions, only because Δ𝑘 = 1. However for continuous the case the pdf is multiplied by a small interval Δ𝑡, which ensures that the probability an event occurring within the interval Δ𝑡 is less than one. 𝑓(𝑡)

𝑓(𝑘)

𝑝𝑝𝑝

𝑝𝑝𝑝

𝑃(𝑘1 < 𝐾 ≤ 𝑘2 )

𝑃(𝑡1 < 𝑇 ≤ 𝑡2 ) 𝑡1

𝑡2

𝑡

𝑘

𝑘1 𝑘2

Figure 1: Left: continuous pdf, right: discrete pdf

To derive the continuous pdf relationship to the cumulative density function (cdf), 𝐹(𝑡): lim 𝑝(𝑡). Δ𝑡 = lim 𝑃(𝑡 < 𝑇 ≤ 𝑡 + Δ𝑡) = lim 𝐹(𝑡 + Δ𝑡) − 𝐹(𝑡) = lim Δ𝐹(𝑡)

Δt→0

Δt→0

Δt→0

𝑝(𝑡) = lim

Δt→0

Δ𝐹(𝑡) 𝑝𝐹(𝑡) = Δ𝑡 𝑝𝑡

Δt→0

The shape of the pdf can be obtained by plotting a normalized histogram of an infinite number of samples from a distribution.

11

It should be noted when plotting a discrete pdf the points from each discrete value should not be joined. For ease of explanation using the area under the graph argument the step plot is intuitive but implies a non-integer random variable. Instead stem plots or column plots are often used. 𝑓(𝑘)

𝑓(𝑘)

𝑘

Figure 2: Discrete data plotting. Left stem plot. Right column plot.

𝑘

1.2.4. Cumulative Distribution Function The cumulative density function (cdf), denoted by 𝐹(𝑡) is the probability of the random event occurring before 𝑡, 𝑃(𝑇 ≤ 𝑡). For a discrete cdf the height of each step is the pdf value 𝑝(𝑘𝑖 ). 𝑡

𝐹(𝑡) = 𝑃(𝑇 ≤ 𝑡) = � 𝑝(𝑚) 𝑝𝑚 , −∞

𝐹(𝑘) = 𝑃(𝐾 ≤ 𝑘) = � 𝑝(𝑘𝑖 )

The limits of the cdf for −∞ < 𝑡 < ∞ and 0 ≤ 𝑘 ≤ ∞ are given as: lim 𝐹(𝑡) = 0 ,

𝑡→−∞

lim 𝐹(𝑡) = 1 ,

𝑡→∞

𝐸𝑖 ≤𝐸

𝐹(−1) = 0

lim 𝐹(𝑘) = 1

𝐸→∞

The cdf can be used to find the probability of the random even occurring between two limits: 𝑏

𝑃(𝑠 ≤ 𝑇 ≤ 𝑏) = � 𝑝(𝑡) 𝑝𝑡 = 𝐹(𝑏) − 𝐹(𝑠) 𝑎 𝑏

𝑃(𝑠 ≤ 𝐾 ≤ 𝑏) = � 𝑝(𝑘) = 𝐹(𝑏) − 𝐹(𝑠 − 1) 𝑖=𝑎

Dist Functions

Introduction

Dist Functions

12

Probability Distributions Used in Reliability Engineering

1

𝐹(𝑡) 𝑃(𝑇 ≤ 𝑡1 )

𝐹(𝑡1 )

𝑓(𝑡)

𝐹(𝑘)

1

𝑐𝑐𝑐

𝑃(𝐾 ≤ 𝑘1 )

𝐹(𝑘1 )

𝑡

𝑡1

𝑐𝑐𝑐

𝑘

𝑘1

𝑓(𝑘)

𝑝𝑝𝑝 𝑃(𝐾 ≤ 𝑘1 )

𝑃(𝑇 ≤ 𝑡1 ) 𝑝𝑝𝑝

𝑡

𝑡1

𝑘1

𝑘

Figure 3: Left: continuous cdf/pdf, right: discrete cdf/pdf

1.2.5. Reliability Function The reliability function, also known as the survival function, is denoted as 𝑅(𝑡). It is the probability that the random event (time of failure) occurs after 𝑡. 𝑅(𝑡) = 𝑃(𝑇 > 𝑡) = 1 − 𝐹(𝑡), ∞

𝑅(𝑡) = � 𝑝(𝑡) 𝑝𝑡 , 𝑡

𝑅(𝑘) = 𝑃(𝑇 > 𝑘) = 1 − 𝐹(𝑘) ∞

𝑅(𝑘) = � 𝑝(𝑘𝑖 ) 𝑖=𝐸+1

It should be noted that in most publications the discrete reliability function is defined as ∗ 𝑅 ∗ (𝑘) = 𝑃(𝑇 ≥ 𝑘) = ∑∞ 𝑖=𝐸 𝑝(𝑘). This definition results in 𝑅 (𝑘) ≠ 1 − 𝐹(𝑘). Despite this problem it is the most common definition and is included in all the references in this book except (Xie, Gaudoin, et al. 2002)

1

𝐹(𝑡)

1

𝑅(𝑡) 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓

𝑐𝑐𝑐

0

𝑡

13

0

𝑡

Figure 4: Left continuous cdf, right continuous survival function

1.2.6. Conditional Reliability Function The conditional reliability function, denoted as 𝑚(𝑚) is the probability of the component surviving given that it has survived to time t. 𝑅(𝑡 + x) 𝑚(𝑚) = 𝑅(𝑚|𝑡) = 𝑅(𝑡) Where: 𝑡 is the given time for which we know the component has survived. 𝑚 is new random variable defined as the time after 𝑡. 𝑚 = 0 at 𝑡.

1.2.7. 100α% Percentile Function

The 100α% percentile function is the interval [0, 𝑡𝛼 ] for which the area under the pdf is 𝛼. 𝑡𝛼 = 𝐹 −1 (𝛼)

1.2.8. Mean Residual Life

The mean residual life (MRL), denoted as 𝑠(𝑡), is the expected life given the component has survived to time, 𝑡 . ∞

1.2.9. Hazard Rate

𝑠(𝑡) = � 𝑅(𝑚|𝑡) 𝑝𝑚 = 0

∞ 1 � 𝑅(𝑚) 𝑝𝑚 𝑅(𝑡) 𝑡

The hazard function, denoted as h(t), is the conditional probability that a component fails in a small time interval, given that it has survived from time zero until the beginning of the time interval. For the continuous case the probability that an item will fail in a time interval given the item was functioning at time 𝑡 is:

Dist Functions

Introduction

Dist Functions

14

Probability Distributions Used in Reliability Engineering

𝑃(𝑡 < 𝑇 < 𝑡 + Δ𝑡|𝑇 > 𝑡) =

𝑃(𝑡 < 𝑇 < 𝑡 + Δ𝑡) 𝐹(𝑡 + Δ𝑡) − 𝐹(𝑡) Δ𝐹(𝑡) = = 𝑃(𝑇 > 𝑡) 𝑅(𝑡) 𝑅(𝑡)

By dividing the probability by Δ𝑡 and finding the limit as Δ𝑡 → 0 gives the hazard rate: ℎ(𝑡) = lim

Δ𝑡→0

𝑝(𝑡) 𝑃(𝑡 < 𝑇 < 𝑡 + Δ𝑡|𝑇 > 𝑡) Δ𝐹(𝑡) = lim = Δ𝑡→0 Δ𝑡𝑅(𝑡) Δ𝑡 𝑅(𝑡)

The discrete hazard rate is defined as: (Xie, Gaudoin, et al. 2002) ℎ(𝑘) =

𝑝(𝑘) 𝑃(𝐾 = 𝑘) = 𝑃(𝐾 ≥ 𝑘) 𝑅(𝑘 − 1)

This unintuitive result is due to a popular definition of 𝑅 ∗ (𝑘) = ∑∞ 𝑖=𝐸 𝑝(𝑘) in which case ℎ(𝑘) = 𝑝(𝑘)/𝑅 ∗ (𝑘). This definition has been avoided because it violates the formula 𝑅(𝑘) = 1 − 𝐹(𝑘). The discrete hazard rate cannot be used in the same way as a continuous hazard rate with the following differences (Xie, Gaudoin, et al. 2002): • • • •

ℎ(𝑘) is defined as a probability and so is bounded by [0,1]. ℎ(𝑘) is not additive for series systems. For the cumulative hazard rate 𝐻(𝑘) = − ln[𝑅(𝑘)] ≠ ∑𝐸𝑖=0 ℎ(𝑘) When a set of data is analyzed using a discrete counterpart of the continuous distribution the values of the hazard rate do not converge.

A function called the second failure rate has been proposed (Gupta et al. 1997): 𝑠(𝑘) = ln

𝑅(𝑘 − 1) = − ln[1 − ℎ(𝑘)] 𝑅(𝑘)

This function overcomes the previously mentioned limitations of the discrete hazard rate function and maintains the monotonicity property. For more information, the reader is referred to (Xie, Gaudoin, et al. 2002) Care should be taken not to confuse the hazard rate with the Rate of Occurrence of Failures (ROCOF). ROCOF is the probability that a failure (not necessarily the first) occurs in a small time interval. Unlike the hazard rate, the ROCOF is the absolute rate at which system failures occur and is not conditional on survival to time t. ROCOF is using in measuring the change in the rate of failures for repairable systems.

1.2.10.

Cumulative Hazard Rate

The cumulative hazard rate, denoted as 𝐻(𝑡) an in the continuous case is the area under the hazard rate function. This function is useful to calculate average failure rates. 𝑡

𝐻(𝑡) = � ℎ(𝑠)𝑝𝑠 = −ln[𝑅(𝑡)] ∞

𝐻(𝑘) = − ln[𝑅(𝑘)]

For a discussion on the discrete cumulative hazard rate see hazard rate.

1.2.11.

15

Characteristic Function

The characteristic function of a random variable completely defines its probability distribution. It can be used to derive properties of the distribution from transformations of the random variable. (Billingsley 1995) The characteristic function is defined as the expected value of the function exp(𝑠𝜔𝑚) where 𝑚 is the random variable of the distribution with a cdf 𝐹(𝑚), 𝜔 is a parameter that can have any real value and 𝑠 = √−1: 𝜑𝑋 (𝜔) = 𝐸�𝑅 𝑖𝜔𝑥 � ∞

= � 𝑅 𝑖𝜔𝑥 𝐹(𝑚) 𝑝𝑚 −∞

A useful property of the characteristic function is the sum of independent random variables is the product of the random variables characteristic function. It is often easier to use the natural log of the characteristic function when conducting this operation. 𝜑𝑋+𝑌 (𝜔) = 𝜑𝑋 (𝜔)𝜑𝑌 (𝜔)

ln[𝜑𝑋+𝑌 (𝜔)] = ln[𝜑𝑋 (𝜔)] ln[𝜑𝑌 (𝜔)]

For example, the addition of two exponentially distributed random variables with the same 𝜆 gives the gamma distribution with 𝑘 = 2: 𝑋~𝐸𝑚𝑝(𝜆), 𝑠𝜆 , 𝜑𝑋 (𝜔) = 𝜔 + 𝑠𝜆

𝑌~𝐸𝑚𝑝(𝜆) 𝜑𝑌 (𝜔) =

𝑠𝜆 𝜔 + 𝑠𝜆

𝜑𝑋+𝑌 (𝜔) = 𝜑𝑋 (𝜔)𝜑𝑌 (𝜔) −𝜆2 = (𝜔 + 𝑠𝜆)2

𝑋 + 𝑌~𝐺𝑠𝑚𝑚𝑠(𝑘 = 1, 𝜆)

This is the characteristic function of the gamma distribution with 𝑘 = 2.

The moment generating function can be calculated from the characteristic function: 𝜑𝑋 (−𝑠𝜔) = 𝑀𝑋 (𝜔)

The 𝑓𝑡ℎ raw moment can be calculated by differentiating the characteristic function 𝑓 times. For more information on moments see section 1.3.2. (𝑛) 𝐸[𝑋 𝑛 ] = 𝑠 −𝑛 𝜑𝑋 (0) 𝑝𝑛 = 𝑠 −𝑛 � 𝑛 𝜑𝑋 (𝜔)� 𝑝𝜔

Dist Functions

Introduction

Dist Functions

16

Probability Distributions Used in Reliability Engineering

1.2.12.

Joint Distributions

Joint distributions are multivariate distributions with, 𝑝 random variables (𝑝 > 1). An example of a bivariate distribution (𝑝 = 2) may be the distribution of failure for a vehicle tire which with random variables time, 𝑇, and distance travelled, 𝑋. The dependence between these two variables can be quantified in terms of correlation and covariance. See section 1.3.3 for more discussion. For more on properties of multivariate distributions see (Rencher 1997). The continuous and discrete random variables will be denoted as: 𝑚1 𝑚2 𝒙 = � ⋮ �, 𝑚𝑑

𝑘1 𝑘2 𝒌=� � ⋮ 𝑘𝑑

Joint distributions can be derived from the conditional distributions. For the bivariate case with random variables 𝑚 and 𝑦: For the more general case:

𝑝(𝑚, 𝑦) = 𝑝(𝑦|𝑚)𝑝(𝑚) = 𝑝(𝑚|𝑦)𝑝(𝑦)

𝑝(𝒙) = 𝑝(𝑚1 |𝑚2 , … , 𝑚𝑑 )𝑝(𝑚2 , … , 𝑚𝑑 ) = 𝑝(𝑚1 ) 𝑝(𝑚2 |𝑚1 ) … 𝑝(𝑚𝑛−1 |𝑚1 , … , 𝑚𝑛−2 ) 𝑝(𝑚𝑛 |𝑚1 , … , 𝑚𝑛 )

If the random variables are independent, their joint distribution is simply the product of the marginal distributions: 𝑑

𝑝(𝒙) = � 𝑝(𝑚𝑖 ) 𝑤ℎ𝑅𝑠𝑅 𝑚𝑖 ⊥ 𝑚𝑗 𝑝𝑓𝑠 𝑠 ≠ 𝑗 𝑖=1 𝑑

𝑝(𝒌) = � 𝑝(𝑘𝑖 ) 𝑤ℎ𝑅𝑠𝑅 𝑘𝑖 ⊥ 𝑘𝑗 𝑝𝑓𝑠 𝑠 ≠ 𝑗 𝑖=1

A general multivariate cumulative probability function with n random variables (𝑇1 , 𝑇2 , … , 𝑇𝑛 ) is defined as: 𝐹(𝑡1 , 𝑡2 , … , 𝑡𝑛 ) = 𝑃(𝑇1 ≤ 𝑡1 , 𝑇2 ≤ 𝑡2 , … , 𝑇𝑛 ≤ 𝑡𝑛 )

The survivor function is given as:

𝑅(𝑡1 , 𝑡2 , … , 𝑡𝑛 ) = 𝑃(𝑇1 > 𝑡1 , 𝑇2 > 𝑡2 , … , 𝑇𝑛 > 𝑡𝑛 )

Different from univariate distributions is the relationship between the CDF and the survivor function (Georges et al. 2001): 𝐹(𝑡1 , 𝑡2 , … , 𝑡𝑛 ) + 𝑅(𝑡1 , 𝑡2 , … , 𝑡𝑛 ) ≤ 1

17

If 𝐹(𝑡1 , 𝑡2 , … , 𝑡𝑛 ) is differentiable then the probability density function is given as: 𝑝(𝑡1 , 𝑡2 , … , 𝑡𝑛 ) =

𝜕 𝑛 𝐹(𝑡1 , 𝑡2 , … , 𝑡𝑛 ) 𝜕𝑡1 𝜕𝑡2 … 𝜕𝑡𝑛

For a discussion on the multivariate hazard rate functions and the construction of joint distributions from marginal distributions see (Singpurwalla 2006).

1.2.13.

Marginal Distribution

The marginal distribution of a single random variable in a joint distribution can be obtained:

𝑝(𝑚1 ) = � … � � 𝑝(𝒙) 𝑝𝑚2 𝑝𝑚3 … 𝑝𝑚𝑑 𝑥𝑑

𝑥3 𝑥2

𝑝(𝑘1 ) = � � … � 𝑝(𝒌) 𝐸2

1.2.14.

𝐸3

𝐸𝑛

Conditional Distribution

If the value is known for some random variables the conditional distribution of the remaining random variables is: 𝑝(𝑚1 |𝑚2 , … , 𝑚𝑑 ) =

1.2.15.

𝑝(𝒙) 𝑝(𝒙) = 𝑝(𝑚2 , … , 𝑚𝑑 ) ∫𝑥 𝑝(𝒙) 𝑝𝑚1

𝑝(𝑘1 |𝑘2 , … , 𝑘𝑑 ) =

Bathtub Distributions

1

𝑝(𝒌) 𝑝(𝒌) = 𝑝(𝑘2 , … , 𝑘𝑑 ) ∑𝐸1 𝑝(𝒙)

Elementary texts on reliability introduce the hazard rate of a system as a bathtub curve. The bathtub curve has three regions, infant mortality (decreasing failure rate), useful life (constant failure rate) and wear out (increasing failure rate). Bathtub distributions have not been a popular choice for modeling life distributions when compared to exponential, Weibull and lognormal distributions. This is because bathtub distributions are generally more complex without closed form moments and more difficult to estimate parameters. Sometimes more complex shapes are required than simple bathtub curves, as such generalizations and modifications to the bathtub curves has been studied. These include an increase in the failure rate followed by a bathtub curve and rollercoaster curves (decreasing followed by unimodal hazard rate). For further reading including applications see (Lai & Xie 2006).

Dist Functions

Introduction

Dist Functions

18

Probability Distributions Used in Reliability Engineering

1.2.16.

Truncated Distributions

Truncation arises when the existence of a potential observation would be unknown if it were to occur in a certain range. An example of truncation is when the existence of a defect is unknown due to the defect’s amplitude being less than the inspection threshold. The number of flaws below the inspection threshold is unknown. This is not to be confused with censoring which occurs when there is a bound for observing events. An example of right censoring is when a test is time terminated and the failures of the surviving components are not observed, however we know how many components were censored. (Meeker & Escobar 1998, p.266) A truncated distribution is the conditional distribution that results from restricting the domain of another probability distribution. The following general formulas apply to truncated distribution functions, where 𝑝0 (𝑚) and 𝐹0 (𝑚) are the pdf and cdf of the nontruncated distribution. For further reading specific to common distributions see (Cohen 1991) Probability Distribution Function: 𝑝𝑜 (𝑚) 𝑝(𝑚) = �𝐹0 (𝑏) − 𝐹0 (𝑠) 𝑝𝑓𝑠 𝑚 ∈ (𝑠, 𝑏] 0 𝑓𝑡ℎ𝑅𝑠𝑤𝑠𝑠𝑅

Cumulative Distribution Function:

𝑝𝑓𝑠 𝑚 ≤ 𝑠 ⎧ 𝑥 0 ⎪ ∫ 𝑝0 (𝑡) 𝑝𝑡 𝑎 𝐹(𝑚) = 𝑝𝑓𝑠 𝑚 ∈ (𝑠, 𝑏] ⎨𝐹0 (𝑏) − 𝐹0 (𝑠) ⎪ 1 𝑝𝑓𝑠 𝑚 > 𝑏 ⎩

1.2.17.

19

Summary Table 2: Summary of distribution functions 𝒇(𝒕)

𝑭(𝒕)

𝑹(𝒕)

𝒉(𝒕)

𝒇(𝒕) =

---

𝐹 ′ (𝑡)

−𝑅 ′ (𝑡)

ℎ(𝑡) exp �− � ℎ(𝑚)𝑝𝑚 �

𝑭(𝒕) =

� 𝑝(𝑚)𝑝𝑚

----

1 − 𝑅(𝑡)

1 − exp �− � ℎ(𝑚)𝑝𝑚 �

𝑹(𝒕) =

1 − � 𝑝(𝑚)𝑝𝑚

1 – 𝐹(𝑡)

----

exp �− � ℎ(𝑚)𝑝𝑚 �

𝒉(𝒕) =

𝑝(𝑡)

𝐹 ′ (𝑡) 1 − 𝐹(𝑡)

𝑅 ′ (𝑡) 𝑅(𝑡)

𝑡

0

𝑡

0

𝑡

1 − ∫0 𝑝(𝑚)𝑝𝑚 ∞

1 � 1 − 𝐹(𝑚)

−𝑠𝑓 � 𝑝(𝑚)𝑝𝑚

𝒖(𝒕) =

∫0 𝑚𝑝(𝑡 + 𝑚)𝑝𝑚 ∫∞[1 − 𝐹(𝑚)]𝑝𝑚 ∫∞ 𝑅(𝑚)𝑝𝑚 𝑡 𝑡 ∞ 1 − 𝐹(𝑡) 𝑅(𝑡) ∫𝑡 𝑝(𝑚)𝑝𝑚

𝑡

∞

𝑡

𝑜

𝑡

𝑜

−

𝑝{exp[−𝐻(𝑡)]} 𝑝𝑡

1 − exp{−𝐻(𝑡)}

𝑡

exp{−𝐻(𝑡)}

𝑜

𝑯(𝒕) =

ln �

𝑯(𝒕)

𝐻 ′ (𝑡)

----

𝑡

−ln{𝑅(𝑚)}

---

� ℎ(𝑚)𝑝𝑚 0

∞

𝑡

∫𝑡 exp�− ∫𝑜 ℎ(𝑚)𝑝𝑚 � 𝑝𝑚 𝑡

exp�− ∫𝑜 ℎ(𝑚)𝑝𝑚 �

∞

∫𝑡 exp{−𝐻(𝑚)} 𝑝𝑚 exp{−𝐻(𝑚)}

Dist Functions

Introduction

Dist Properties

20

Probability Distributions Used in Reliability Engineering

1.3. Distribution Properties 1.3.1. Median / Mode The median of a distribution, denoted as 𝑡0.5 is when the cdf and reliability function are equal to 0.5. 𝑡0.5 = 𝐹 −1 (0.5) = 𝑅 −1 (0.5)

The mode is the highest point of the pdf, 𝑡𝑚 . This is the point where a failure has the highest probability. Samples from this distribution would occur most often around the mode.

1.3.2. Moments of Distribution The moments of a distribution are given by: ∞

𝜇𝑛 = � (𝑚 − 𝑐)𝑛 𝑝(𝑚) 𝑝𝑚 , −∞

𝑛

𝜇𝑛 = ��𝑘𝑗 − 𝑐� 𝑝(𝑘) 𝑖

When 𝑐 = 0 the moments, 𝜇𝑛′ , are called the raw moments, described as moments about the origin. In respect to probability distributions the first two raw moments are important. 𝜇0′ always equals one, and 𝜇1′ is the distributions mean which is the expected value of the random variable for the distribution: ∞

mean = 𝐸[𝑋] = 𝜇:

𝜇′0 = � 𝑝(𝑚) 𝑝𝑚 = 1, −∞

∞

𝜇′1 = � 𝑚𝑝(𝑚) 𝑝𝑚 , −∞

𝜇′0 = � 𝑝(𝑘𝑖 ) = 1 𝑖

𝜇′1 = � 𝑘𝑖 𝑝(𝑘𝑖 ) 𝑖

Some important properties of the expected value 𝐸[𝑋] when transformations of the random variable occur are: 𝐸[𝑋 + 𝑏] = 𝜇𝑋 + 𝑏 𝐸[𝑋 + 𝑌] = 𝜇𝑋 + 𝜇𝑌 𝐸[𝑠𝑋] = 𝑠𝜇𝑋

𝐸[𝑋𝑌] = 𝜇𝑋 𝜇𝑌 + 𝐶𝑓𝑠(𝑋, 𝑌)

When 𝑐 = 𝜇 the moments, 𝜇𝑛 , are called the central moments, described as moments about the mean. In this book, the first five central moments are important. 𝜇0 is equal to 𝜇0′ = 1. 𝜇1 is the variance which quantifies the amount the random variable deviates from the mean. 𝜇2 and 𝜇3 are used to calculate the skewness and kurtosis. ∞

𝜇0 = � 𝑝(𝑚) 𝑝𝑚 = 1,

∞

−∞

𝜇1 = � (𝑚 − 𝜇)𝑝(𝑚) 𝑝𝑚 = 0, −∞

𝜇0 = � 𝑝(𝑘𝑖 ) = 1 𝑖

𝜇1 = �(𝑘𝑖 − 𝜇)𝑝(𝑘𝑖 ) = 0 𝑖

21

variance = 𝐸[(𝑋 − 𝐸[𝑋])2 ] = 𝐸[𝑋 2 ] − {𝐸[𝑋]}2 = 𝜎 2 : ∞

𝜇2 = � (𝑚 − 𝜇)2 𝑝(𝑚) 𝑝𝑚 ,

𝜇2 = �(𝑘𝑖 − 𝜇)2 𝑝(𝑘𝑖 )

−∞

𝑖

Some important properties of the variance exist when transformations of the random variable occur are: 𝑉𝑠𝑠[𝑋 + 𝑏] = 𝑉𝑠𝑠[𝑋] 𝑉𝑠𝑠[𝑋 + 𝑌] = 𝜎𝑋2 + 𝜎𝑌2 ± 2𝐶𝑓𝑠(𝑋, 𝑌) 𝑉𝑠𝑠[𝑠𝑋] = 𝑠2 𝜎𝑋2

𝑉𝑠𝑠[𝑋𝑌] = (𝑋𝑌)2 ��

𝜎𝑋 2 𝜎𝑌 2 𝐶𝑓𝑠(𝑋, 𝑌) � +� � +2 � 𝑋 𝑌 𝑋𝑌

The skewness is a measure of the asymmetry of the distribution. 𝛾1 =

𝜇3

𝜇23

⁄2

The kurtosis is a measure of the whether the data is peaked or flat.

1.3.3. Covariance

𝛾2 =

𝜇4 𝜇22

Covariance is a measure of the dependence between random variables. 𝐶𝑓𝑠(𝑋, 𝑌) = 𝐸[(𝑋 − 𝜇𝑋 )(𝑌 − 𝜇𝑌 )] = 𝐸[𝑋𝑌] − 𝜇𝑋 𝜇𝑌

A normalized measure of covariance is correlation, 𝜌. The correlation has the limits −1 ≤ 𝜌 ≤ 1. When 𝜌 = 1 the random variables have a linear dependency (i.e, an increase in X will result in the same increase in Y). When 𝜌 = −1 the random variables have a negative linear dependency (i.e, an increase in X will result in the same decrease in Y). The relationship between covariance and correlation is: 𝐶𝑓𝑠(𝑋, 𝑌) 𝜌𝑋,𝑌 = 𝐶𝑓𝑠𝑠(𝑋, 𝑌) = 𝜎𝑋 𝜎𝑌 If the two random variables are independent than the correlation is equal to zero, however the reverse is not always true. If the correlation is zero the random variables does not need to be independent. For derivations and more information the reader is referred to (Dekking et al. 2007, p.138).

Dist Properties

Introduction

Parameter Est

22

Probability Distributions Used in Reliability Engineering

1.4. Parameter Estimation 1.4.1. Probability Plotting Paper Most plotting methods transform the data available into a straight line for a specific distribution. From a line of best fit the parameters of the distribution can be estimated. Most plotting paper plots the random variable (time or demands) against the pdf, cdf or hazard rate and transform the data points to a linear relationship by adjusting the scale of each axis. Probability plotting is done using the following steps (Nelson 1982, p.108): 1. Order the data such that 𝑚1 ≤ 𝑚2 ≤ ⋯ ≤ 𝑚𝑖 ≤ ⋯ ≤ 𝑚𝑛 .

2. Assign a rank to each failure. For complete data this is simply the value i. Censored data is discussed after step 7. 3. Calculate the plotting position. The cdf may simply be calculated as 𝑠/𝑓 however this produces a biased result, instead the following non-parametric Blom estimates, are recommended as suitable for many cases by (Kimball 1960): 1 ℎ�(𝑡𝑖 ) = (𝑓 − 𝑠 + 0.625)(𝑡𝑖+1 − 𝑡𝑖 ) 𝐹� (𝑡𝑖 ) =

𝑠 − 0.375 𝑓 + 0.25

𝑝̂(𝑡𝑖 ) =

1 (𝑓 + 0.25)(𝑡𝑖+1 − 𝑡𝑖 )

𝑅�(𝑡𝑖 ) =

Other proposed estimators are:

𝑓 − 𝑠 + 0.625 (𝑓 + 0.25)

𝑠 𝑓 𝑠 − 0.3 Median (approximate): 𝐹� (𝑡𝑖 ) = 𝑓 + 0.4 Naive: 𝐹� (𝑡𝑖 ) =

Midpoint: 𝐹� (𝑡𝑖 ) = Mean ∶ 𝐹� (𝑡𝑖 ) =

Mode: 𝐹� (𝑡𝑖 ) =

𝑠 − 0.5 𝑓

𝑠 𝑓+1

𝑠−1 𝑓−1

4. Plot points on probability paper. The choice of distribution should be from experience, or multiple distributions should be used to assess the best fit. Probability paper is available from http://www.weibull.com/GPaper/.

23

5. Assess the data and chosen distributions. If the data plots in straight line then the distribution may be a reasonable fit. 6. Draw a line of best fit. This is a subjective assessment which minimizes the deviation of the points from the chosen line. 7. Obtained the desired information. This may be the distribution parameters or estimates of reliability or hazard rate trends. When multiple failure modes are observed only one failure mode should be plotted with the other failures being treated as censored. Two popular methods to treat censored data two methods are : Rank Adjustment Method. (Manzini et al. 2009, p.140) Here the adjusted rank, 𝑗𝑡𝑖 is calculated only for non-censored units (with 𝑠𝑡𝑖 still being the rank for all ordered times). This adjusted rank is used for step 2 with the remaining steps unchanged: (𝑓 + 1) − 𝑗𝑡𝑖−1 𝑗𝑡𝑖 = 𝑗𝑡𝑖−1 + 2 + 𝑓 − 𝑠𝑡𝑖 Kaplan Meier Estimator. Here the estimate for reliability is: 𝑝 � 𝑅�(𝑡𝑖 ) = � �1 − 𝑓−𝑠+1 𝑡𝑗 <𝑡𝑖

Where 𝑝 is the number of failures in rank j (for non-grouped data 𝑝 = 1). From this estimate a cdf can be given as 𝐹� (𝑡𝑖 ) = 1 − 𝑅�(𝑡𝑖 ). For a detailed derivation and properties of this estimator see (Andersen et al. 1996, p.255) Probability plots are fast and not dependent on complex numerical methods and can be used without a detailed knowledge of statistics. It provides a visual representation of the data for which qualitative statements can be made. It can be useful in estimating initial values for numerical methods. Limitation of this technique is that it is not objective and two different people making the same plot will obtain different answers. It also does not provide confidence intervals. For more detail of probability plotting the reader is referred to (Nelson 1982, p.104) and (Meeker & Escobar 1998, p.122)

1.4.2. Total Time on Test Plots Total time on Test (TTT) plots is a graph which provides a visual representation of the hazard rate trend, i.e increasing, constant or decreasing. This assists in identifying the distribution from which the data may come from. To plot TTT (Rinne 2008, p.334): 1. Order the data such that 𝑚1 ≤ 𝑚2 ≤ ⋯ ≤ 𝑚𝑖 ≤ ⋯ ≤ 𝑚𝑛 . 2. Calculate the TTT positions: 𝑖

𝑇𝑇𝑇𝑖 = �(𝑓 − 𝑗 + 1)�𝑚𝑗 − 𝑚𝑗−1 � ; 𝑠 = 1,2, … , 𝑓 𝑗=1

3. Calculate the normalized TTT positions:

Parameter Est

Introduction

Parameter Est

24

Probability Distributions Used in Reliability Engineering

𝑖

4. Plot the points �𝑛 , 𝑇𝑇𝑇𝑖∗ �.

𝑇𝑇𝑇𝑖∗ =

𝑇𝑇𝑇𝑖 ; 𝑠 = 1,2, … , 𝑓 𝑇𝑇𝑇𝑛

5. Analyze graph:

1

𝑇𝑇𝑇𝑖

0

0

𝑖� 𝑛

1

Figure 5: Time on test plot interpretation Compared to probability plotting, TTT plots are simple, scale invariant and can represent any data set even those from different distributions on the same plot. However it only provides an indication of failure rate properties and cannot be used directly to estimate parameters. For more information about TTT plots the reader is referred to (Rinne 2008, p.334).

1.4.3. Least Mean Square Regression When the relationship between two variables, 𝑚 and 𝑦 is assumed linear (𝑦 = 𝑚𝑚 + 𝑐), an estimate of the line’s parameters can be obtained from 𝑓 sample data points, (𝑚𝑖 , 𝑦𝑖 ) using least mean square (LMS) regression. The least square method minimizes the square of the residual. 𝑛

𝑆 = � 𝑠𝑖2 𝑖=1

25

The residual can be defined in many ways. Minimize y residuals 𝑠𝑖 = 𝑦𝑖 − 𝑝(𝑚𝑖 ; 𝑚, 𝑐)

𝑚 �=

𝑦𝑖 𝑓(𝑥𝑖 ; 𝑚, 𝑐)

𝑓∑𝑚𝑖 𝑦𝑖 − (∑𝑚𝑖 )(∑𝑦𝑖 ) 𝑓∑𝑚𝑖2

𝑐̂ =

𝑦

𝑟𝑖

Minimize x residuals 𝑠𝑖 = 𝑚𝑖 − 𝑝(𝑦𝑖 ; 𝑚, 𝑐)

−

2

𝑚 �=

2 �∑𝑚𝑖2 �

∑𝑦𝑖 ∑𝑚𝑖 −𝑚 � 𝑓 𝑓

𝑐̂ =

𝑦

(𝑥𝑖 , 𝑦𝑖 )

𝑓∑𝑦𝑖2 − �∑𝑦𝑖2 � 𝑓∑𝑚𝑖 𝑦𝑖 − (∑𝑚𝑖 )(∑𝑦𝑖 ) ∑𝑦𝑖 ∑𝑚𝑖 −𝑚 � 𝑓 𝑓

Regression line 𝑦 = 𝑚𝑚 + 𝑐

(𝑥𝑖 , 𝑦𝑖 )

Regression line 𝑦 = 𝑚𝑚 + 𝑐

0

𝑥

0

𝑓(𝑦𝑖 ; 𝑚, 𝑐)

𝑟𝑖

𝑦𝑖

𝑥

Figure 6: Left minimize y residual, right minimize x residual

The LMS method can be used to estimate the line of best fit when using plotting parameter estimation methods. When plotting on a regular scale in software such as Microsoft Excel, it is often easy to conduct linear least mean square (LMS) regression using in built functions. Where available this book provides the formulas to plot the sample data in a straight line in a regular scale plot. It also provides the transformation from the linear LMS regression estimates of 𝑚 � and 𝑐̂ to the distribution parameter estimates. For more on least square methods in a reliability engineering context see (Nelson 1990, p.167). MS regression can also be conducted on multivariate distributions, see (Rao & Toutenburg 1999) and can also be conducted on non-linear data directly, see (Bjõrck 1996).

1.4.4. Method of Moments To estimate the distribution parameters using the method of moments the sample moments are equated to the parameter moments and solved for the unknown parameters. The following sample moments can be used: The sample mean is given as:

𝑛

1 𝑚̅ = � 𝑚𝑖 𝑓 𝑖=1

Parameter Est

Introduction

Parameter Est

26

Probability Distributions Used in Reliability Engineering

The unbiased sample variance is given as: 𝑛 1 𝑆2 = �(𝑚𝑖 − 𝑚̅ )2 𝑓−1 𝑖=1

Method of moments is not as accurate as Bayesian or maximum likelihood estimates but is easy and fast to calculate. The method of moment estimates are often used as a starting point for numerical methods to optimize maximum likelihood and least square estimators.

1.4.5. Maximum Likelihood Estimates Maximum likelihood estimates (MLE) are a frequentist approach to parameter estimation usually obtained by maximizing the natural log of the likelihood function. Λ(θ|E) = ln{𝐿(𝜃|𝐸)}

Algebraically this is done by solving the first order partial derivatives of the log-likelihood function. This calculation has been included in this book for distributions where the result is in closed form. Otherwise the log-likelihood function can be maximized directly using numerical methods. MLE for 𝜃� is obtained by solving for 𝜃:

𝜕Λ =0 𝜕𝜃

Denote the true parameters of the distribution as 𝜽𝟎 , MLEs have the following properties (Rinne 2008, p.406): • Consistency. As the number of samples increases the difference between the estimated and actual parameter decreases: �=𝜽 plim 𝜽 •

Asymptotic normality.

𝑛→∞

lim 𝜃� ~𝑁𝑓𝑠𝑚(𝜃0 , [𝐼𝑛 (𝜃0 )]−1 )

𝑛→∞

•

•

where 𝐼𝑛 (𝜃) = 𝑓𝐼(𝜃) is the Fisher information matrix. Therefore 𝜃� is asymptotically unbiased: lim 𝐸[𝜃�] = 𝜃0 Asymptotic efficiency.

𝑛→∞

lim 𝑉𝑠𝑠[𝜃�] = [𝐼𝑛 (𝜃0 )]−1

𝑛→∞

Invariance. The MLE of 𝑝(𝜃0 ) is 𝑝(𝜃�) if 𝑝(. ) is a continuous and continuously differentiable function.

The advantages of MLE are that it is a very common technique that has been widely published and is implemented in many software packages. The MLE method can easily handle censored data. The disadvantage to MLE is the bias introduced for small sample

27

sizes and unbounded estimates may result when no failures have been observed. The numerical optimization of the log-likelihood function may be non-trivial with high sensitivity to starting values and the presence of local maximums. For more information in a reliability context see (Nelson 1990, p.284).

1.4.6. Bayesian Estimation Bayesian estimation uses a subjective interpretation of the theory of probability and for parameter point estimation and confidence intervals uses Bayes’ rule to update our state of knowledge of the unknown of interest (UIO). Recall from section 1.1.5 Bayes rule,

𝜋(𝜃|𝐸) =

𝜋𝑜 (𝜃) 𝐿(𝐸|𝜃) , ∫ 𝜋𝑜 (𝜃) 𝐿(𝐸|𝜃) 𝑝𝜃

𝑃(𝜃|𝐸) =

𝑃(𝜃) 𝑃(𝐸|𝜃)

𝑌 ∑𝑓𝑠=1 𝑃(𝐸|𝜃𝑠 )𝑃(𝜃)

respectively for continuous and discrete forms of variable of 𝜃. The Prior Distribution 𝜋𝑜 (𝜃)

The prior distribution is probability distribution of the UOI, 𝜃, which captures our state of knowledge of 𝜃 prior to the evidence being observed. It is common for this distribution to represent soft evidence or intervals about the possible values of 𝜃. If the distribution is dispersed it represents little being known about the parameter. If the distribution is concentrated in an area then it reflects a good knowledge about the likely values of 𝜃.

Prior distributions should be a proper probability distribution of 𝜃. A distribution is proper when it integrates to one and improper otherwise. The prior should also not be selected based on the form of the likelihood function. When the prior has a constant which does not affect the posterior distribution (such as improper priors) it will be omitted from the formulas within this book.

Non-informative Priors. Occasions arise when it is not possible to express a subjective prior distribution due to lack of information, time or cost. Alternatively a subjective prior distribution may introduce unwanted bias through model convenience (conjugates) or due to elicitation methods. In such cases a non-informative prior may be desirable. The following methods exist for creating a non-informative prior (Yang and Berger 1998): •

Principle of Indifference - Improper Uniform Priors. An equal probability is assigned across all the possible values of the parameter. This is done using an improper uniform distribution with a constant, usually 1, over the range of the possible values for 𝜃. When placed in Bayes formula the constant cancels out, however the denominator is integrated over all possible values of 𝜃. In most cases this prior distribution will result in a proper posterior, but not always. Improper Uniform Priors may be chosen to enable the use of conjugate priors. For example using exponential likelihood model, with an improper uniform prior, 1, over the limits [0, ∞) with evidence of 𝑓𝐹 failures in total time, 𝑡𝑇 : Prior:

𝜋0 (𝜆) = 1 ∝ 𝐺𝑠𝑚𝑚𝑠(1,0)

Parameter Est

Introduction

Parameter Est

28

Probability Distributions Used in Reliability Engineering

Likelihood:

𝐿(𝐸|𝜆) = 𝜆nF 𝑅 −𝜆𝑡𝑇

Posterior: 𝜋(𝜆|𝐸) =

1. 𝐿(𝐸|𝜆) ∞ 1. ∫0 𝐿(𝐸|𝜆) 𝑝𝜆

Using conjugate relationship (see Conjugate Priors for calculations):

•

𝜆~𝐺𝑠𝑚𝑚𝑠(𝜆; 1 + nF , tT )

Principle of Indifference - Proper Uniform Priors. An equal probability is assigned across the values of the parameter within a range defined by the uniform distribution. The uniform distribution is obtained by estimating the far 1 left and right bounds (𝑠 and 𝑏) of the parameter 𝜃 giving 𝜋𝑜 (𝜃) = = 𝑐, 𝑏−𝑎 where c is a constant. When placed in Bayes formula the constant cancels out, however the denominator is integrated over the bound [𝑠, 𝑏]. Care needs to be taken in choosing 𝑠 and 𝑏 because no matter how much evidence suggests otherwise the posterior distribution will always be zero outside these bounds. Using an exponential likelihood model, with a proper uniform prior, 𝑐, over the limits [𝑠, 𝑏] with evidence of 𝑓𝐹 failures in total time, 𝑡𝑇 : Prior:

1

𝜋0 (𝜆) = 𝑏−𝑎 = 𝑐 ∝ 𝑇𝑠𝑠𝑓𝑐𝑠𝑡𝑅𝑝 𝐺𝑠𝑚𝑚𝑠(1,0) Likelihood:

Posterior: 𝜋(𝜆|𝐸) =

𝐿(𝐸|𝜆) = 𝜆nF 𝑅 −𝜆𝑡𝑇

𝑐. 𝐿(𝐸|𝜆) 𝑏

𝑐. ∫𝑠 𝐿(𝐸|𝜆) 𝑝𝜆

for a ≤ λ ≤ b

Using conjugate relationship this results in a truncated Gamma distribution: 𝑐. 𝐺𝑠𝑚𝑚𝑠(𝜆; 1 + nF , tT ) for a ≤ λ ≤ b 0 otherwise

𝜋(𝜆) = � •

Jeffrey’s Prior. Proposed by Jeffery in 1961, this prior is defined as 𝜋0 (𝜃) = �𝑝𝑅𝑡(𝑰𝜃 ) where 𝑰𝜃 is the Fisher information matrix. This derivation is motivated by the fact that it is not dependent upon the set of parameter variables that is chosen to describe parameter space. Jeffery himself suggested the need to make ad hoc modifications to the prior to avoid problems in multidimensional distributions. Jeffories prior is normally improper. (Bernardo et al. 1992)

•

Reference Prior. Proposed by Bernardo in 1979, this prior maximizes the expected posterior information from the data, therefore reducing the effect of the prior. When there is no nuisance parameters and certain regularity conditions are satisfied the reference prior is identical to the Jeffrey’s prior. Due to the need to order or group the importance of parameters, it may occur that different posteriors will result from the same data depending on the

29

importance the user places on each parameter. This prior overcomes the problems which arise when using Jeffery’s prior in multivariate applications. •

Maximal Data Information Prior (MDIP). Developed by Zelluer in 1971 maximizes the likelihood function with relation to the prior. (Berry et al. 1995, p.182)

For further detail on the differences between each type of non-informative prior see (Berry et al. 1995, p.179) Conjugate Priors. Calculating posterior distributions can be extremely complex and in most cases requires expensive computations. A special case exists however by which the posterior distribution is of the same form as the prior distribution. The Bayesian updating mathematics can be reduced to simple calculations to update the model parameters. As an example the gamma function is a conjugate prior to a Poisson likelihood function: Prior: 𝜋𝑜 (𝜆) =

𝛽 𝛼 𝜆𝛼−1 −βλ 𝑅 Γ(𝛼)

Likelihood: 𝐿𝑖 (𝑡𝑖 |𝜆) = 𝑛𝐹

𝜆𝐸𝑖 𝑡𝑖𝐸 −𝜆𝑡 𝑅 𝑖 𝑘𝑖 !

Likelihood: 𝐿(𝐸|𝜆) = � 𝐿𝑖 (𝑡𝑖 |𝜆) = 𝑖=1

Posterior: 𝜋(𝜆|𝐸) = =

𝜋𝑓 (𝜆) 𝐿(𝐸|𝜆) ∞ 𝜋 ∫0 𝑓 (𝜆) 𝐿(𝐸|𝜆) 𝑝𝜆 ∞

� 0

∞

=

𝜆∑𝐸 ∏𝑡𝑖𝐸 −𝜆∑𝑡 𝑖 𝑅 ∏𝑘𝑖 !

𝛽 𝛼 𝜆𝛼−1 𝜆∑𝐸 ∏𝑡𝑖𝐸 −βλ −𝜆∑𝑡𝑖 𝑅 𝑅 Γ(𝛼)∏𝑘𝑖 !

𝛽 𝛼 𝜆𝛼−1 𝜆∑𝐸 ∏𝑡𝑖𝐸 −βλ −𝜆∑𝑡 𝑖 𝑝𝜆 𝑅 𝑅 Γ(𝛼)∏𝑘𝑖 !

𝜆𝛼−1+∑𝐸 𝑅 −λ(β+∑𝑡𝑖 )

∞ ∫0 𝜆𝛼−1+∑𝐸

𝑅 −λ(β+∑𝑡𝑖 ) 𝑝𝜆

Using the identity Γ(𝑧) = ∫𝑜 𝑚 𝑧−1 𝑅 −𝑥 𝑝𝑚 we can calculate the denominator using the 𝑢

𝑑𝑢

change of variable 𝑠 = 𝜆(𝛽 + ∑𝑡𝑖 ). This results in 𝜆 = 𝛽+∑𝑡 , and 𝑝𝜆 = 𝛽+∑𝑡 𝑖

limits of 𝑠 the same as 𝜆 . Substituting back into the posterior equation gives: 𝜋(𝜆|𝐸) =

𝜆𝛼−1+∑𝐸 𝑅 −λ(β+∑𝑡𝑖 ) ∞

𝛼−1+∑𝐸 1 𝑠 � � � 𝑅 −u 𝑝𝑠 𝛽 + ∑𝑡𝑖 𝛽 + ∑𝑡𝑖 0

𝑖

with the

Parameter Est

Introduction

Parameter Est

30

Probability Distributions Used in Reliability Engineering

=

Let 𝑧 = 𝛼 + ∑𝑘

𝜆𝛼−1+∑𝐸 𝑅 −λ(β+∑𝑡𝑖 )

∞ 1 ∫ 𝑠𝛼−1+∑𝐸 𝑅 −u 𝑝𝑠 (𝛽 + ∑𝑡𝑖 )𝛼+∑𝐸 0

𝜋(𝜆|𝐸) = ∞

Using Γ(𝑧) = ∫𝑜 𝑚 𝑧−1 𝑅 −𝑥 𝑝𝑚:

𝜋(𝜆|𝐸) =

Let 𝛼 ′ = 𝛼 + ∑𝑘, 𝛽 ′ = 𝛽 + ∑𝑡𝑖 :

𝜆𝛼−1+∑𝐸 𝑅 −λ(β+∑𝑡𝑖 ) ∞ 1 ∫ 𝑠 𝑧−1 𝑅 −u 𝑝𝑠 (𝛽 + ∑𝑡𝑖 )𝛼+∑𝐸 0

𝜆𝛼−1+∑𝐸 (𝛽 + ∑𝑡𝑖 )𝛼+∑𝐸 −λ(β+∑𝑡 ) 𝑖 𝑅 Γ(𝛼 + ∑𝑘) ′

𝛼′

𝜆𝛼 −1 𝛽 ′ 𝜋(𝜆|𝐸) = Γ(𝛼 ′ )

′

𝑅 −β λ

As can be seen the posterior is a gamma distribution with the parameters 𝛼 ′ = 𝛼 + ∑𝑘, 𝛽 ′ = 𝛽 + ∑𝑡𝑖 . Therefore the prior and posterior are of the same form, and Bayes’ rule does not need to be re-calculated for each update. Instead the user can simply update the parameters with the new evidence. The Likelihood Function 𝐿(𝐸|𝜃)

The reader is referred to section 1.1.6 for a discussion on the construction of the likelihood function. The Posterior Distribution 𝜋(𝜃|𝐸)

The posterior distribution is a probability distribution of the UOI, 𝜃, which captures our state of knowledge of 𝜃 including all prior information and the evidence.

Point Estimate. From the posterior distribution we may want to give a point estimate of θ. The Bayesian estimator when using a quadratic loss function is the posterior mean (Christensen & Huffman 1985):

𝜃� = 𝐸[𝜋(𝜃|𝐸)] = ∫ 𝜃𝜋(𝜃|𝐸) 𝑝𝜃 = 𝜇𝜋

For more information on utility, loss functions and estimators in a Bayesian context see (Berger 1993).

1.4.7. Confidence Intervals Assuming a random variable is distributed by a given distribution, there exists the true � , may or distribution parameters, 𝜽𝟎 , which is unknown. The parameter point estimates, 𝜽 may not be close to the true parameter values. Confidence intervals provide the range over which the true parameter values may exist with a certain level of confidence. Confidence intervals only quantify uncertainty due to sampling error arising from a limited number of samples. Uncertainty due to incorrect model selection or incorrect assumptions is not included. (Meeker & Escobar 1998, p.49)

31

Increasing the desired confidence 𝛾 results in an increased confidence interval. Increasing the sample size generally decreases the confidence interval. There are many methods to calculate confidence intervals. Some popular methods are: •

Exact Confidence Intervals. It may be mathematically shown that the parameter of a distribution itself follows a distribution. In such cases exact confidence intervals can be derived. This is only the case in very few distributions.

•

Fisher Information Matrix (Nelson 1990, p.292). For a large number of samples, the asymptotic normal property can be used to estimate confidence intervals: lim 𝜃� ~𝑁𝑓𝑠𝑚(𝜃0 , [𝑓𝐼(𝜃0 )]−1 ) 𝑛→∞

Combining this with the asymptotic property 𝜃� → 𝜃0 as 𝑓 → ∞ gives the following estimate for the distribution of 𝜃�: −1 lim 𝜃� ~𝑁𝑓𝑠𝑚 �𝜃�, �𝐽𝑛 �𝜃��� � 𝑛→∞

100𝛾% approximate confidence intervals are calculated using percentiles of the normal distribution. If the range of 𝜃 is unbounded (−∞, ∞) the approximate two sided confidence intervals are: 1+𝛾 −1 � ��𝐽𝑛 �𝜃��� 𝜃𝛾 = 𝜃� − Φ−1 � 2 1+𝛾 −1 ��� 𝜃𝛾 = 𝜃� + Φ−1 � � ��𝐽𝑛 �𝜃��� 2 If the range of 𝜃 is (0, ∞) the approximate two sided confidence intervals are: ⎡𝛷 −1 �1 + 𝛾 � ��𝐽𝑛 �𝜃���−1 ⎤ 2 � ⎥ 𝜃𝛾 = 𝜃. exp ⎢ −𝜃� ⎢ ⎥ ⎣ ⎦ −1 1 + 𝛾 −1 ⎡𝛷 � ⎤ � � 2 � �𝐽𝑛 �𝜃�� ⎥ ��� 𝜃𝛾 = 𝜃�. exp ⎢ 𝜃� ⎢ ⎥ ⎣ ⎦ If the range of 𝜃 is (0,1) the approximate two sided confidence intervals are: −1

⎡𝛷 −1 �1 + 𝛾 � ��𝐽𝑛 �𝜃���−1 ⎤⎫ 2 � � � ⎥ 𝜃𝛾 = 𝜃. 𝜃 + (1 − 𝜃 ) exp ⎢ �(1 − 𝜃�) 𝜃 ⎢ ⎥⎬ ⎨ ⎩ ⎣ ⎦⎭ −1 −1 1 + 𝛾 ⎡𝛷 −1 � ⎧ ��𝐽𝑛 �𝜃��� ⎤⎫ � 2 ���𝛾 = 𝜃�. 𝜃� + (1 − 𝜃�) exp ⎢ ⎥ 𝜃 −𝜃�(1 − 𝜃�) ⎢ ⎥⎬ ⎨ ⎩ ⎣ ⎦⎭ ⎧

The advantage of this method is it can be calculated for all distributions and is easy to calculate. The disadvantage is that the assumption of a normal distribution is asymptotic and so sufficient data is required for the confidence

Parameter Est

Introduction

Parameter Est

32

Probability Distributions Used in Reliability Engineering

interval estimate to be accurate. The number of samples needed for an accurate estimate changes from distribution to distribution. It also produces symmetrical confidence intervals which may be very inaccurate. For more information see (Nelson 1990, p.292). •

Likelihood Ratio Intervals (Nelson 1990, p.292). The test statistic for the likelihood ratio is: 𝐷 = 2[Λ�𝜃�� − Λ(𝜃)] 𝐷 is approximately Chi-Square distributed with one degree of freedom. 𝐷 = 2�Λ�𝜃�� − Λ(𝜃)� ≤ 𝜒 2 (𝛾; 1)

Where 𝛾 is the 100𝛾% confidence interval for 𝜃. The two sided confidence 𝜃𝛾 are calculated by solving: limits 𝜃𝛾 and ��� Λ(𝜃) = Λ�𝜃�� −

𝜒 2 (𝛾; 1) 2

The limits are normally solved numerically. The likelihood ratio intervals are always within the limits of the parameter and gives asymmetrical confidence limits. It is much more accurate than the Fisher information matrix method particularly for one sided limits although it is more complicated to calculate. This method must be solved numerically and so will not be discussed further in this book. •

Bayesian Confidence Intervals. In Bayesian statistics the uncertainty of a parameter, 𝜃, is quantified as a distribution 𝜋(𝜃). Therefore the two sided 100𝛾% confidence intervals are found by solving: 𝜃𝛾 1−𝛾 = � 𝜋(𝜃) 𝑝𝜃 , 2 −∞

∞ 1+𝛾 = � 𝜋(𝜃) 𝑝𝜃 2 𝜃𝛾

Other methods exist to calculate approximate confidence intervals. A summary of some techniques used in reliability engineering is included in (Lawless 2002).

Introduction

𝑋𝐵 =

𝑋𝑁 = ln(𝑋𝐿 )

1

XC = X N(k) − µ 2 �� � σ

𝑋𝐺1 𝑋𝐺1 + 𝑋𝐺2 𝛼 = 𝑘1 , 𝛽 = 𝑘2

𝑛=1

𝑋𝐵 =

Binomial 𝐵𝐵𝐵𝐵𝐵(𝑛, 𝑝)

lim 𝑛 → ∞ 𝑛𝑛 = 𝜇

V

Chi-square 𝜒 2 (𝑣)

𝑋𝐹 =

Poisson 𝑃𝑃𝑃𝑃(𝜇)

Gamma 𝐺𝐺𝐺𝐺𝐺(𝑘, 𝜆)

Student t 𝑡(𝑣)

𝑣 = 2, 𝜎

𝑋𝐸 = ln(𝑋𝑃 ⁄𝜃 ) Pareto 𝑃𝑃𝑃𝑃𝑃𝑃(𝜃, 𝛼

1� 𝛽

𝛽=2 𝜎 = √2/𝛼

Exponential 𝐸𝐸𝐸(𝜆)

𝛽=1

Rayleigh 𝑅𝑅𝑅𝑅𝑅𝑅𝑅ℎ(𝜎)

𝑋𝑅 = �𝑋𝐸

𝑋𝑊 = 𝑋𝐸

𝑣 = 2, 𝛼 = √2, 𝛽 = 2

𝑋𝐹 = 𝑋𝑇2 𝑛1 = 1, 𝑛2 = 𝑣

𝛼 = 1, 𝛽 = 1

𝑘=1

𝑋𝐺 = 𝑋𝐸1 + ⋯ + 𝑋𝐸𝐸

𝑋𝐶1 𝑛2 𝑋𝐶2 𝑛1

X χ2 = 𝑋𝜒2

𝑓𝑃 (𝑘; 𝜆𝜆) = 𝐹𝐺 (𝑡; 𝑘 + 1, 𝜆) −𝐹𝐺 (𝑡; 𝑘, 𝜆)

F 𝐹(𝑛1 , 𝑛2 )

Chi 𝜒(𝑣)

Beta 𝐵𝐵𝐵𝐵(𝛼, 𝛽)

𝑋𝑆(1) ≤ ⋯ ≤ 𝑋𝑆(𝑛) , 𝑋𝑆(𝑟) ~𝐵𝐵𝐵𝐵

𝑋𝐺 = � 𝑋𝐵𝐵 𝑛 (𝑖𝑖𝑖)

1

𝛼 = 2𝑣1 , 𝛽 = 2𝑣2

Bernoulli 𝐵𝐵𝐵𝐵𝐵𝐵𝐵𝐵𝐵(𝑝)

Normal 𝑁𝑁𝑁𝑁(𝜇, 𝜎 2 )

X C = |𝑿𝑵 |

𝑋𝐶1 , 𝑋𝐶1 + 𝑋𝐶2

Special Case

Weibull 𝑊𝑊𝑊𝑊𝑊𝑊𝑊(𝛼, 𝛽)

𝑋𝑆 = exp(−𝜆𝑋𝐸 )

Standard Uniform 𝑈𝑈𝑈𝑈(0,1)

𝑎=0 𝑏=1

Uniform 𝑈𝑈𝑈𝑈(𝑎, 𝑏)

Figure 7: Relationships between common distributions (Leemis & McQueston 2008). Many relations are not included such as central limit convergence to the normal distribution and many transforms which would have made the figure unreadable. For further details refer to individual sections and (Leemis & McQueston 2008).

Related Dist

Transformation

1.5. Related Distributions Lognormal 𝐿𝐿𝐿𝐿𝐿(𝜇𝑁 , 𝜎𝑁2 )

33

Supporting Func

34

Probability Distributions Used in Reliability Engineering

1.6. Supporting Functions 1.6.1. Beta Function 𝐁(𝒙, 𝒚)

B(𝑚, 𝑦) is the Beta function and is the Euler integral of the first kind. Where 𝑚 > 0 and 𝑦 > 0.

1

𝐵(𝑚, 𝑦) = � 𝑠 𝑥−1 (1 − 𝑠)𝑦−1 𝑝𝑠 0

Relationships:

𝐵(𝑚, 𝑦) = 𝐵(𝑦, 𝑚) Γ(𝑚)Γ(𝑦) 𝐵(𝑚, 𝑦) = Γ(𝑚 + 𝑦) ∞ 𝑓−𝑦 � � 𝑓 𝐵(𝑚, 𝑦) = � 𝑚+𝑓 𝑛=0

More formulas, definitions and special values can be found in the Digital Library of Mathematical Functions on the National Institute of Standards and Technology (NIST) website, http://dlmf.nist.gov/5/12/.

1.6.2. Incomplete Beta Function 𝑩𝒕 (𝒕; 𝒙, 𝒚) 𝐵𝑡 (𝑡; 𝑚, 𝑦) is the incomplete Beta function:

𝑡

𝐵𝑡 (𝑡; 𝑚, 𝑦) = � 𝑠 𝑥−1 (1 − 𝑠)𝑦−1 𝑝𝑠 0

1.6.3. Regularized Incomplete Beta Function 𝑰𝒕 (𝒕; 𝒙, 𝒚) 𝐼𝑡 (𝑡|𝑚, 𝑦) is the regularized incomplete Beta function: 𝐵𝑡 (t ; 𝑚, 𝑦) 𝐼𝑡 (𝑡|𝑚, 𝑦) = 𝐵(𝑚, 𝑦) 𝑎+𝑏−1

Properties:

= �

𝑗=𝑎

(𝑚 + 𝑦 − 1)! . 𝑡 𝑗 (1 − 𝑡)𝑥+𝑦−1−𝑗 (𝑚 𝑗! + 𝑦 − 1 − 𝑗)! 𝐼0 (0; 𝑚, 𝑦) = 0 𝐼1 (1; 𝑚, 𝑦) = 1

𝐼𝑡 (t; 𝑚, 𝑦) = 1 − 𝐼(1 − t; 𝑦, 𝑚)

1.6.4. Complete Gamma Function 𝚪(𝒌)

Γ(𝑘) is a generalization of the factorial function 𝑘! to include non-integer values.

For 𝑘 > 0

35

∞

Γ(𝑘) = � 𝑡 𝐸−1 𝑅 −𝑡 𝑝𝑡 0

∞

∞

= � 𝑡 𝐸−1 𝑅 −𝑡 � + (𝑘 − 1) � 𝑡 𝐸−2 𝑅 −𝑡 𝑝𝑡 0 ∞

= (𝑘 − 1) � 𝑡 𝐸−2 𝑅 −𝑡 𝑝𝑡

0

0

When 𝑘 is an integer: Special values:

= (𝑘 − 1)Γ(𝑘 − 1)

Γ(𝑘) = (𝑘 − 1)! Γ(1) = 1

Γ(2) = 1

1 Γ � � = √𝜋 2

Relation to the incomplete gamma functions: Γ(𝑘) = Γ(𝑘, 𝑡) + 𝛾(𝑘, 𝑡)

More formulas, definitions and special values can be found in the Digital Library of Mathematical Functions on the National Institute of Standards and Technology (NIST) website, http://dlmf.nist.gov/5/.

1.6.5. Upper Incomplete Gamma Function 𝚪(𝒌, 𝒕) For 𝑘 > 0

When 𝑘 is an integer:

∞

Γ(𝑘, 𝑡) = � 𝑚 𝐸−1 𝑅 −𝑥 𝑝𝑚 𝑡

Γ(𝑘, 𝑡) = (𝑘 − 1)! 𝑅

−𝑡

𝐸−1

�

𝑛=0

𝑡𝑛 𝑓!

More formulas, definitions and special values can be found on the NIST website, http://dlmf.nist.gov/8/.

1.6.6. Lower Incomplete Gamma Function 𝛄(𝒌, 𝒕) For 𝑘 > 0

When 𝑘 is an integer:

𝑡

γ(𝑘, 𝑡) = � 𝑚 𝐸−1 𝑅 −𝑥 𝑝𝑚 0

𝐸−1

γ(𝑘, 𝑡) = (𝑘 − 1)! �1 − 𝑅 −𝑡 �

𝑛=0

𝑡𝑛 � 𝑓!

Supporting Func

Introduction

Supporting Func

36

Probability Distributions Used in Reliability Engineering

More formulas, definitions and special values can be found on the NIST website, http://dlmf.nist.gov/8/.

1.6.7. Digamma Function 𝝍(𝒙)

𝜓(𝑚) is the digamma function defined as: 𝑝 Γ ′ (𝑚) 𝜓(𝑚) = ln[Γ(𝑚)] = 𝑝𝑓𝑠 𝑚 > 0 𝑝𝑚 Γ(𝑚)

1.6.8. Trigamma Function 𝝍′(𝒙)

𝜓 ′ (𝑚) is the trigamma function defined as: ∞ 𝑝2 𝜓 ′ (𝑚) = 2 𝑠𝑓Γ(𝑚) = �(𝑚 + 𝑠)−2 𝑝𝑚 𝑖=0

1.7. Referred Distributions 1.7.1. Inverse Gamma Distribution 𝑰𝑮(𝜶, 𝜷)

The pdf to the inverse gamma distribution is: −𝛽 𝛽𝛼 𝑝(𝑚; 𝛼, 𝛽) = . 𝑅 𝑥 . 𝐼𝑥 (0, ∞) Γ(𝛼)𝑚 𝛼+1 With mean: 𝛽 𝜇= for 𝛼 > 1 𝛼−1

1.7.2. Student T Distribution 𝑻(𝜶, 𝝁, 𝝈𝟐 )

The pdf to the standard student t distribution with 𝜇 = 0, 𝜎 2 = 1 is: −

𝑚2 𝑝(𝑚; 𝛼) = . �1 + � 𝛼 √𝛼𝜋Γ(𝛼⁄2) Γ[(𝛼 + 1)⁄2]

𝛼+1 2

The generalized student t distribution is:

−

(𝑚 − 𝜇)2 𝑝(𝑚; 𝛼, 𝜇, 𝜎 2 ) = . �1 + � 𝛼𝜎 2 𝜎√𝛼𝜋Γ(𝛼⁄2) Γ[(𝛼 + 1)⁄2]

With mean

1.7.3. F Distribution 𝑭(𝒏𝟏 , 𝒏𝟐 )

𝛼+1 2

𝜇=𝜇

Also known as the Variance Ratio or Fisher-Snedecor distribution the pdf is:

With cdf:

𝑝(𝑚; 𝛼) = 𝐼𝑡 �

(𝑓1 𝑚)𝑛1 . 𝑓2𝑛2 1 � . {𝑛 +𝑛 } 𝑓 𝑓 𝑚𝐵 � 21 , 22 � (𝑓1 𝑚 + 𝑓2 ) 1 2

𝑓1 𝑓2 , �, 2 2

𝑤ℎ𝑅𝑠𝑅 𝑡 =

1.7.4. Chi-Square Distribution 𝝌𝟐 (𝒗)

𝑓1 𝑚 𝑓1 𝑚 + 𝑓2

The pdf to the chi-square distribution is:

With mean:

𝑚 𝑚 (𝑣−2)⁄2 𝑅𝑚𝑝 �− � 2 𝑝(𝑚; 𝑠) = 𝑠 2𝑣⁄2 𝛤 �2� 𝜇=𝑠

37

Referred Dist

Introduction

Referred Dist

38

Probability Distributions Used in Reliability Engineering

1.7.5. Hypergeometric Distribution 𝑯𝒚𝒑𝒆𝒓𝑮𝒆𝒐𝒎(𝒌; 𝒏, 𝒎, 𝑵)

The hypergeometric distribution models probability of 𝑘 successes in 𝑓 Bernoulli trials from population 𝑁 containing 𝑚 success without replacement. 𝑝 = 𝑚/𝑁. The pdf to the hypergeometric distribution is:

With mean:

𝑝(𝑘; 𝑓, 𝑚, 𝑁) = 𝜇=

�m ��N−m � k n−k

𝑓𝑚 𝑁

�Nn�

1.7.6. Wishart Distribution 𝑾𝒊𝒔𝒉𝒂𝒓𝒕𝒅 (𝒙; 𝚺, 𝒏)

The Wishart distribution is the multivariate generalization of the gamma distribution. The pdf is given as:

With mean:

𝑝𝑑 (𝒙; 𝚺, 𝑓) =

1

|𝐱|2(n−d−1)

2𝑛𝑑⁄2 |𝚺|n⁄2 Γ𝑑

1 −𝟏 𝑓 exp �− 2 𝑡𝑠�𝒙 𝚺�� �2�

𝝁 = 𝑓𝚺

39

1.8. Nomenclature and Notation Functions are presented in the following form: 𝑝(𝑠𝑠𝑓𝑝𝑓𝑚 𝑠𝑠𝑠𝑠𝑠𝑏𝑠𝑅𝑠 ; 𝑝𝑠𝑠𝑠𝑚𝑅𝑡𝑅𝑠𝑠 |𝑅𝑠𝑠𝑅𝑓 𝑠𝑠𝑠𝑠𝑅𝑠) 𝑓

In continuous distributions the number of items under test = 𝑓𝑓 + 𝑓𝑠 + 𝑓𝐼 . In discrete distributions the total number of trials.

𝑓𝐹

The number of items which failed before the conclusion of the test.

𝑓𝐼

The number of items which have interval data

𝑡𝑖𝑆

The time at which a component survived to. The item may have been removed from the test for a reason other than failure.

𝑓𝑆

The number of items which survived to the end of the test.

𝑡𝑖𝐹 , 𝑡𝑖

The time at which a component fails.

𝑡𝑖𝑈𝐼

The upper limit of a censored interval in which an item failed

𝑡𝐿

The lower truncated limit of sample.

𝑡𝑇

Time on test = ∑𝑡𝑖 + ∑𝑡𝑠

𝐾

Discrete random variable

𝑘

A discrete random variable with a known value

𝒙

A bold symbol denotes a vector or matrix

𝜃

Upper confidence interval for UOI

𝑋~𝑁𝑓𝑠𝑚𝑑

The random variable 𝑋 is distributed as a 𝑝-variate normal distribution.

𝑡𝑖𝐿𝐼

The lower limit of a censored interval in which an item failed

𝑡𝑈

The upper truncated limit of sample.

𝑋 or 𝑇

Continuous random variable (T is normally a random time)

𝑚 or 𝑡

A continuous random variable with a known value

𝑚�

The hat denotes an estimated value

𝜃

Generalized unknown of interest (UOI)

𝜃

Lower confidence interval for UOI

Nomenclature

Introduction

Life Distribution

40

Probability Distributions Used in Reliability Engineering

2. Common Life Distributions

Exponential Continuous Distribution

41

2.1. Exponential Continuous Distribution Exponential

Probability Density Function - f(t) λ=0.5

0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00

λ=1 λ=2 λ=3

0

0.5

1

1.5

2

2.5

3

3.5

4

Cumulative Density Function - F(t) 1.00 0.80 λ=0.5

0.60

λ=1

0.40

λ=2

0.20

λ=3

0.00 0

0.5

1

1.5

2

2.5

3

3.5

4

1

1.5

2

2.5

3

3.5

4

Hazard Rate - h(t) 3.00 2.50 2.00 1.50 1.00 0.50 0.00 0

0.5

42

Common Life Distributions

Parameters & Description Parameters

𝜆

Exponential

Limits Function

Time Domain

Scale Parameter: Equal to the hazard rate. 𝑡 ≥ 0

𝑝(𝑡) = 𝜆e−λt

PDF

𝐹(𝑡) = 1 − e−λt

CDF

R(t) = e−λt

Reliability

𝑚(𝑚) = e−λx

Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡) Mean Life

𝜆>0

Laplace Domain

𝑝(𝑠) =

𝜆 , 𝜆+𝑠

𝐹(𝑠) =

𝑠 > −𝜆

𝜆 𝑠(𝜆 + 𝑠)

𝑅(𝑠) =

𝑚(𝑠) =

1 𝜆+𝑠

1 𝜆+𝑠

Where 𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡.

Residual

1 𝜆

𝑠(𝑠) =

1 𝜆𝑠

𝐻(𝑡) = 𝜆𝑡

𝐻(𝑠) =

𝜆 𝑠2

𝑠(𝑡) =

ℎ(𝑡) = 𝜆

Hazard Rate Cumulative Hazard Rate

Properties and Moments

ℎ(𝑠) =

𝜆 𝑠

𝑠𝑓(2) 𝜆

Median Mode

0

Mean - 1st Raw Moment Variance - 2nd Central Moment Skewness - 3rd Central Moment th

Excess kurtosis - 4 Central Moment Characteristic Function 100α% Percentile Function

1 𝜆

1 𝜆2 2 6 𝑠𝜆 𝑡 + 𝑠𝜆

1 𝑡𝛼 = − ln(1 − 𝛼) 𝜆

Exponential Continuous Distribution

43

Parameter Estimation Plotting Method X-Axis

Y-Axis 𝑡𝑖

Likelihood Function nF

Likelihood Functions

nS

F

failures

survivors

when there is no interval data this reduces to: 𝑤ℎ𝑅𝑠𝑅

Point Estimates

Fisher Information

100𝛾% Confidence Interval (excluding interval data)

UI

LI

nF

interval failures

𝑡𝑇 = � t Fi + � t Si = 𝑡𝑓𝑡𝑠𝑠 𝑡𝑠𝑚𝑅 𝑠𝑓 𝑡𝑅𝑠𝑡 nS

nI

LI

RI

Λ(𝐸|𝜆) = r. ln(λ) − � λt Fi − � λt s + � ln �e−λti − e−λti � ����������� ��� ��������������� i=1 i=1 i=1 failures

survivors

when there is no interval data this reduces to: Λ(𝐸|𝜆) = nF . ln(𝜆) − 𝜆𝑡𝑇

∂Λ =0 ∂λ

nI

S

𝐿(𝐸|𝜆) = λnF � e−λ.ti . � e−ti . � �e−λti − e−λti � ������� i=1���� ����� i=1 �� ��������������� i=1

𝐿(𝐸|𝜆) = 𝜆nF 𝑅 −𝜆𝑡𝑇

Log-Likelihood Functions

𝜆̂ = −𝑚

𝑠𝑓[1 − 𝐹(𝑡𝑖 )]

solve for 𝜆 to get 𝜆̂:

nS

nF

interval failures

𝑡𝑇 = � t Fi + � t Si

where nI

LI

RI

λti λti − t RI t LI nF i e i e − � t Fi − � t Si − � � �=0 𝐿𝐼 𝑅𝐼 𝜆𝑡 𝜆𝑡 λ 𝑅 𝑖 −𝑅 𝑖 ����� �� ��� ��������������� i=1 i=1 i=1 failures

survivors

interval failures

When there is only complete and right-censored data the point estimate is: nF 𝜆̂ = 𝑤ℎ𝑅𝑠𝑅 𝑡𝑇 = � t Fi + � t Si = 𝑡𝑓𝑡𝑠𝑠 𝑡𝑠𝑚𝑅 𝑠𝑓 𝑡𝑅𝑠𝑡 𝑡𝑇 λ lower 2 Sided

𝐼(𝜆) =

1 𝜆

λ upper 2 Sided

λ upper 1 Sided

Type I (Time Terminated)

𝜒 21−γ (2nF ) �

𝜒 21+γ (2nF + 2)

2 (2nF + 2) 𝜒(γ)

Type II (Failure Terminated)

𝜒 21−γ (2nF )

𝜒 21+γ (2nF )

2 (2nF ) 𝜒(γ)

� �

2

2𝑡𝑇

2

�

2𝑡𝑇

�

2 �

�

2

2𝑡𝑇 �

2𝑡𝑇

2𝑡𝑇 2𝑡𝑇

Exponential

Least Mean Square 𝑦 = 𝑚𝑚 + 𝑐

44

Common Life Distributions

Exponential

2 𝜒(𝛼) is the 𝛼 percentile of the Chi-squared distribution. (Modarres et al. 1999, pp.151-152) Note: These confidence intervals are only valid for complete and right-censored data or when approximations of interval data are used (such as the median). They are exact confidence bounds and therefore approximate methods such as use of the Fisher information matrix need not be used.

Bayesian Non-informative Priors 𝝅(𝝀) (Yang and Berger 1998, p.6)

Type

Prior

Posterior

1 𝑏−𝑠

Uniform Proper Prior with limits 𝜆 ∈ [𝑠, 𝑏] Uniform Improper Prior with limits 𝜆 ∈ [0, ∞) Jeffrey’s Prior

Truncated Gamma Distribution For a ≤ λ ≤ b 𝑐. 𝐺𝑠𝑚𝑚𝑠(𝜆; 1 + nF , t T ) Otherwise 𝜋(𝜆) = 0

1 ∝ 𝐺𝑠𝑚𝑚𝑠(1,0)

1

√𝜆

𝐺𝑠𝑚𝑚𝑠(𝜆; 1 + nF , t T )

1

∝ 𝐺𝑠𝑚𝑚𝑠(2, 0)

1 ∝ 𝐺𝑠𝑚𝑚𝑠(0,0) 𝜆

Novick and Hall

where

𝐺𝑠𝑚𝑚𝑠(𝜆; 12 + nF , t T ) when 𝜆 ∈ [0, ∞) 𝐺𝑠𝑚𝑚𝑠(𝜆; nF , t T ) when 𝜆 ∈ [0, ∞)

𝑡𝑇 = ∑ t Fi + ∑ t Si = total time in test Conjugate Priors

UOI 𝜆 from 𝐸𝑚𝑝(𝑡; λ) Example

Likelihood Model Exponential

Evidence 𝑓𝐹 failures in 𝑡𝑇 unit of time

Dist of UOI

Prior Para

Posterior Parameters

Gamma

𝑘0 , Λ 0

𝑘 = 𝑘𝑜 + 𝑓𝐹 Λ = Λ 𝑜 + 𝑡𝑇

Description , Limitations and Uses

Three vehicle tires were run on a test area for 1000km have punctures at the following distances: Tire 1: No punctures Tire 2: 400km, 900km Tire 3: 200km Punctures are a random failure with constant failure rate therefore an exponential distribution would be appropriate. Due to an exponential distribution being homogeneous in time, the renewal process of the second tire failing twice with a repair can be considered as two separate tires on test with single failures. See example in section 1.1.6. Total distance on test is 3 × 1000 = 3000km. Total number of

Exponential Continuous Distribution

45

failures is 3. Therefore using MLE the estimate of 𝜆:

With 90% confidence interval (distance terminated test): 2 2 (6) (8) 𝜒(0.95) 𝜒(0.05) = 0.272𝐸-3, = 2.584𝐸-3� � 6000 6000

A Bayesian point estimate using the Jeffery non-informative improper prior 𝐺𝑠𝑚𝑚𝑠(12, 0), with posterior 𝐺𝑠𝑚𝑚𝑠(𝜆; 3.5, 3000) has a point estimate: 3.5 𝜆̂ = E[𝐺𝑠𝑚𝑚𝑠(𝜆; 3.5, 3000)] = = 1.16̇ E − 3 3000

Characteristics

With 90% confidence interval using inverse Gamma cdf: [𝐹𝐺−1 (0.05) = 0.361𝐸-3, 𝐹𝐺−1 (0.95) = 2.344𝐸-3]

Constant Failure Rate. The exponential distribution is defined by a constant failure rate, 𝜆. This means the component is not subject to wear or accumulation of damage as time increases. 𝒇(𝟎) = 𝝀. As can be seen, 𝜆 is the initial value of the distribution. Increases in 𝜆 increase the probability density at 𝑝(0).

HPP. The exponential distribution is the time to failure distribution of a single event in the Homogeneous Poisson Process (HPP). 𝑇~𝐸𝑚𝑝(𝑡; 𝜆)

Scaling property

Minimum property

𝜆 𝑠𝑇~𝐸𝑚𝑝 �𝑡; � 𝑠

𝑛

min{𝑇1 , 𝑇2 , … , 𝑇𝑛 }~𝐸𝑚𝑝 �𝑡; � 𝜆𝑖 � 𝑖=1

Variate Generation property ln(1 − 𝑠) 𝐹 −1 (𝑠) = , 0<𝑠<1 −𝜆 Memoryless property. Pr(𝑇 > 𝑡 + 𝑚|𝑇 > 𝑡) = Pr(𝑇 > 𝑚) Properties from (Leemis & McQueston 2008). Applications

No Wearout. The exponential distribution is used to model occasions when there is no wearout or cumulative damage. It can be used to approximate the failure rate in a component’s useful life period (after burn in and before wear out). Homogeneous

Poisson

Process

(HPP).

The

exponential

Exponential

nF 3 = = 1E-3 𝑡𝑇 3000

𝜆̂ =

46

Common Life Distributions

distribution is used to model the inter arrival times in a repairable system or the arrival times in queuing models. See Poisson and Gamma distribution for more detail.

Exponential

Electronic Components. Some electronic components such as capacitors or integrated circuits have been found to follow an exponential distribution. Early efforts at collecting reliability data assumed a constant failure rate and therefore many reliability handbooks only provide a failure rate estimates for components. Random Shocks. It is common for the exponential distribution to model the occurrence of random shocks An example is the failure of a vehicle tire due to puncture from a nail (random shock). The probability of failure in the next mile is independent of how many miles the tire has travelled (memoryless). The probability of failure when the tire is new is the same as when the tire is old (constant failure rate). In general component life distributions do not have a constant failure rate, for example due to wear or early failures. Therefore the exponential distribution is often inappropriate to model most life distributions, particularly mechanical components. Online: http://www.weibull.com/LifeDataWeb/the_exponential_distribution.h tm http://mathworld.wolfram.com/ExponentialDistribution.html http://en.wikipedia.org/wiki/Exponential_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (web calc) Resources Books: Balakrishnan, N. & Basu, A.P., 1996. Exponential Distribution: Theory, Methods and Applications 1st ed., CRC. Nelson, W.B., Interscience.

1982.

Applied

Life

Data

Analysis,

Wiley-

Relationship to Other Distributions 2-Para Exp Distribution 𝐸𝑚𝑝(𝑡; 𝜇, 𝛽) Gamma Distribution 𝐺𝑠𝑚𝑚𝑠(𝑡; 𝑘, 𝜆)

Special Case:

Let Then

1 𝐸𝑚𝑝(𝑡; 𝜆) = 𝐸𝑚𝑝(𝑡; µ = 0, β = ) 𝜆

𝑇1 … 𝑇𝐸 ~𝐸𝑚𝑝(𝜆)

𝑠𝑓𝑝

𝑇𝑡 = 𝑇1 + 𝑇2 + ⋯ + 𝑇𝐸

𝑇𝑡 ~𝐺𝑠𝑚𝑚𝑠(𝑘, 𝜆) The gamma distribution is the probability density function of the sum of k exponentially distributed time random variables sharing the same constant rate of occurrence, 𝜆. This is a Homogeneous Poisson Process.

Exponential Continuous Distribution

47

Special Case: 𝐸𝑚𝑝(𝑡; 𝜆) = 𝐺𝑠𝑚𝑚𝑠(𝑡; 𝑘 = 1, 𝜆)

Let

𝑡𝑠𝑚𝑅 = 𝑇1 + 𝑇2 + ⋯ + 𝑇𝐾 + 𝑇𝐾+1 …

Then

𝐾~𝑃𝑓𝑠𝑠(k; µ = 𝜆𝑡)

Poisson Distribution 𝑃𝑓𝑠𝑠(𝑘; 𝜇)

The Poisson distribution is the probability of observing exactly k occurrences within a time interval [0, t] where the inter-arrival times of each occurrence is exponentially distributed. This is a Homogeneous Poisson Process. Special Cases: 𝑃𝑓𝑠𝑠(k = 1; µ = 𝜆𝑡) = 𝐸𝑚𝑝(𝑡; 𝜆)

Let Weibull Distribution 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽)

𝑋~𝐸𝑚𝑝(𝜆)

Then Special Case:

Let Geometric Distribution 𝐺𝑅𝑓𝑚𝑅𝑡𝑠𝑠𝑐(𝑘; 𝑝) Rayleigh Distribution 𝑅𝑠𝑦𝑠𝑅𝑠𝑅ℎ(𝑡; 𝛼)

Chi-square 𝜒 2 (𝑚; 𝑠)

Pareto Distribution 𝑃𝑠𝑠𝑅𝑡𝑓(𝑡; 𝜃, 𝛼)

Then

𝑠𝑓𝑝

−1

𝑌 = X1/β

𝑌~𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝛼 = 𝜆 𝛽 , 𝛽)

1 𝐸𝑚𝑝(𝑡; 𝜆) = 𝑊𝑅𝑠𝑏𝑠𝑠𝑠 �𝑡; 𝛼 = , 𝛽 = 1� 𝜆

𝑋~𝐸𝑚𝑝(𝜆)

𝑠𝑓𝑝

𝑌 = [𝑋],

𝑌 𝑠𝑠 𝑡ℎ𝑅 𝑠𝑓𝑡𝑅𝑅𝑅𝑠 𝑓𝑝 𝑋

𝑌~𝐺𝑅𝑓𝑚𝑅𝑡𝑠𝑠𝑐(𝛼, 𝛽)

The geometric distribution is the discrete equivalent of the continuous exponential distribution. The geometric distribution is also memoryless. Let 𝑋~𝐸𝑚𝑝(𝜆)

Then

Special Case:

Let Then

𝑠𝑓𝑝

𝑌~𝑅𝑠𝑦𝑠𝑅𝑠𝑅ℎ(𝛼 =

1

𝑌 = √X

√𝜆

)

1 𝜒 2 (𝑚; 𝑠 = 2) = 𝐸𝑚𝑝 �𝑚; λ = � 2

𝑌~𝑃𝑠𝑠𝑅𝑡𝑓(𝜃, 𝛼)

𝑠𝑓𝑝

𝑋~𝐸𝑚𝑝(𝜆 = 𝛼)

𝑋 = ln(𝑌⁄𝜃 )

Exponential

𝑇1 , 𝑇2 … ~𝐸𝑚𝑝(𝑡; 𝜆)

Given

48

Common Life Distributions

Exponential

Let Logistic Distribution 𝐿𝑓𝑅𝑠𝑠𝑡𝑠𝑐(µ , 𝑠)

Then

𝑋~𝐸𝑚𝑝(𝜆 = 1)

𝑠𝑓𝑝

𝑌 = ln �

𝑌~𝐿𝑓𝑅𝑠𝑠𝑡𝑠𝑐(0,1) (Hastings et al. 2000, p.127):

𝑅 −𝑋 � 1 + 𝑅 −𝑋

Lognormal Continuous Distribution

49

2.2. Lognormal Continuous Distribution μ'=0 σ'=1 μ'=0.5 σ'=1 μ'=1 σ'=1 μ'=1.5 σ'=1

0.90 0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00

μ'=1 σ'=1 μ'=1 σ'=1 μ'=1 σ'=1.5 μ'=1 σ'=2

0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00

0

2

4

6

0

2

4

0

2

4

0

2

4

Cumulative Density Function - F(t) 0.90

1.00

0.80 0.80

0.70 0.60

0.60

0.50 0.40

0.40

0.30

0.20

0.20

0.00

0.00

0.10 0

2

4

6

Hazard Rate - h(t) 0.70

1.80 1.60 1.40 1.20 1.00 0.80 0.60 0.40 0.20 0.00

0.60 0.50 0.40 0.30 0.20 0.10 0.00 0

2

4

6

Lognormal

Probability Density Function - f(t)

50

Common Life Distributions

Lognormal

Parameters & Description

𝜇𝑁

Scale parameter: The mean of the normally distributed ln(𝑚). This parameter only determines the scale and not the location as in a normal distribution. 𝜇2 𝜇𝑁 = ln � � �𝜎 2 + 𝜇2

−∞ < 𝜇𝑁 < ∞

Parameters

σ2N

Shape parameter: The standard deviation of the normally distributed ln(x). This parameter only determines the shape and not the scale as in a normal distribution. 𝜎 2 + 𝜇2 σ2N = ln � � 𝜇2

σ2N > 0

Limits

t>0

Distribution

Formulas 𝑝(𝑡) =

PDF

1 ln(𝑡) − 𝜇𝑁 2 exp � − � � � 2 𝜎𝑁 𝜎𝑁 𝑡√2𝜋 1

=

1 𝑠𝑓(𝑡) − 𝜇𝑁 𝜙� � 𝜎𝑁 . 𝑡 𝜎𝑁

where 𝜙 is the standard normal pdf.

1 1 ln(𝑡 ∗ ) − 𝜇𝑁 2 exp � − � � � 𝑝𝑡 ∗ 2 𝜎𝑁 𝜎𝑁 √2𝜋 0 𝜃 where 𝑡 ∗ is the time variable over which the pdf is integrated. 𝐹(𝑡) =

𝑡

1

�

=

CDF

𝑠𝑓(𝑡) − 𝜇𝑁 1 1 + erf � � 2 2 𝜎𝑁 √2 ln(𝑡) − 𝜇𝑁 = Φ� � 𝜎𝑁

where Φ is the standard normal cdf.

ln(𝑡) − 𝜇𝑁 R(t) = 1 − Φ � � 𝜎𝑁

Reliability

Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡)

Where

ln(𝑚 + t) − 𝜇𝑁 � 𝑅(𝑡 + 𝑚) 1 − Φ � 𝜎𝑁 𝑚(𝑚) = 𝑅(𝑚|𝑡) = = ln(t) − 𝜇𝑁 𝑅(𝑡) 1 − Φ� � 𝜎 𝑁

Lognormal Continuous Distribution

51

𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡. Residual

ln(𝑡) − 𝜇𝑁 � 𝜎𝑁 ℎ(𝑡) = ln(𝑡) − 𝜇𝑁 𝑡. 𝜎𝑁 �1 − Φ � �� 𝜎𝑁 𝜙�

Hazard Rate

Cumulative Hazard Rate

𝐻(𝑡) = −ln[𝑅(𝑡)]

Properties and Moments

𝑅 (𝜇𝑁 )

Median

2

𝑅 (𝜇𝑁 −𝜎𝑁 )

Mode Mean - 1st Raw Moment nd

Variance - 2

�𝑅

Central Moment

Skewness - 3rd Central Moment

2

𝑅

2 𝜎𝑁 � 2

2

− 1�. 𝑅 2𝜇𝑁 +𝜎𝑁

�𝑅 𝜎 + 2�. �𝑅 𝜎 − 1

Excess kurtosis - 4th Central Moment Characteristic Function

2 𝜎𝑁

�𝜇𝑁 +

2

2

2

2

𝑅 4𝜎𝑁 + 2𝑅 3𝜎𝑁 + 3𝑅 2𝜎𝑁 − 3

Deriving a unique characteristic equation is not trivial and complex series solutions have been proposed. (Leipnik 1991)

100α% Percentile Function

𝑡𝛼 = e(𝜇𝑁 +𝑧𝛼 .𝜎𝑁 )

where 𝑧𝛼 is the 100pthof the standard normal distribution −1 (𝛼))

Parameter Estimation

𝑡α = e(𝜇𝑁 +𝜎𝑁 Φ

Plotting Method Least Mean Square 𝑦 = 𝑚𝑚 + 𝑐

X-Axis

Y-Axis ln(𝑡𝑖 )

𝑠𝑓𝑠𝑁𝑓𝑠𝑚[𝐹(𝑡𝑖 )]

Maximum Likelihood Function

𝑐 𝜇�𝑁 = − 𝑚 1 𝜎� 𝑁 = 𝑚

Lognormal

Mean Life

∞

∫𝑡 𝑅(𝑚)𝑝𝑚 𝑅(𝑡) 𝜎𝑁2 𝑡 lim 𝑠(𝑡) ≈ [1 + 𝑓(1)] 𝑡→∞ ln(𝑡) − 𝜇𝑁 Where 𝑓(1) is Landau's notation. (Kleiber & Kotz 2003, p.114) 𝑠(𝑡) =

52

Common Life Distributions

Likelihood Functions

nF nS nI 1 𝑆 𝐹 � 𝜙(𝑧 ) . � �1 − Φ �𝑧 �� . � �Φ�𝑧𝑖𝑅𝐼 � − Φ�𝑧𝑖𝐿𝐼 �� 𝑖 𝑖 F ������������� ����������������� i=1 𝜎𝑁 . t i i=1 i=1 �������������

Lognormal

where

survivors

failures

ln(t xi ) − 𝜇𝑁 𝑧𝑖𝑥 = � � 𝜎𝑁

nF

Log-Likelihood Function

nI

i=1

survivors

interval failures

ln(t xi ) − 𝜇𝑁 𝑧𝑖𝑥 = � � 𝜎𝑁

solve for 𝜇𝑁 to get MLE 𝜇�𝑁 :

nF

nS

ϕ�ziS � ∂Λ −µN . N F 1 1 = + � ln(t Fi ) + � ∂µN σN σN σN 1 − Φ�ziS � ��������������� ����������� i=1 i=1 nI

where

failures

1 ϕ�ziRI � − ϕ�ziLI � −� � �=0 σN Φ�ziRI � − Φ�ziLI � ����������������� i=1

survivors

interval failures

solve for 𝜎𝑁 to get 𝜎�: 𝑁

ln(t xi ) − 𝜇𝑁 𝑧𝑖𝑥 = � � 𝜎𝑁

nF

nS

ziS . ϕ�ziS � ∂Λ −nF 1 1 2 = + 3 ��ln(t Fi ) − µN � + � ∂σN σN σN σN 1 − Φ�ziS � ������������������� ����������� i=1 i=1 nI

where

MLE Point Estimates

failures

+ � ln�Φ�𝑧𝑖𝑅𝐼 � − Φ�𝑧𝑖𝐿𝐼 �� ����������������� i=1

where

∂Λ =0 ∂σN

nS

1 𝑆 𝐹 Λ(µN , σN |E) = � ln � 𝐹 𝜙�𝑧𝑖 �� + � ln�1 − Φ�𝑧𝑖 �� 𝜎 𝑁 . 𝑡𝑖 ������������� ������������� i=1

∂Λ =0 ∂µN

interval failures

failures

survivors

1 ziRI . ϕ�ziRI � − ziLI ϕ�ziLI � −� � �=0 σN Φ�ziRI � − Φ�ziLI � ��������������������� i=1 interval failures

ln(t xi ) − 𝜇𝑁 𝑧𝑖𝑥 = � � 𝜎𝑁

When there is only complete failure data the point estimates can be given as: 2 ∑�ln�𝑡𝑖𝐹 � − 𝜇�𝑡 � ∑ ln(𝑡𝑖𝐹 ) σ�2N = 𝜇�𝑁 = nF nF Note: In almost all cases the MLE methods for a normal distribution

Lognormal Continuous Distribution

53

can be used by taking the 𝑠𝑓(𝑋). However Normal distribution estimation methods cannot be used with interval data. (Johnson et al. 1994, p.220)

Fisher Information

𝐼(𝜇𝑁 , 𝜎𝑁2 )

(Kleiber & Kotz 2003, p.119). 1 Sided - Lower

100𝛾% Confidence Intervals

𝝁𝑵 𝝈𝟐𝑵

(for complete data)

𝜇�𝑁 −

𝜎 � 𝑁

√𝑓𝐹

⎤ ⎥ 1 ⎥ − 4⎦ 2𝜎 0

2 Sided - Lower

𝑡γ (nF − 1)

(𝑓𝐹 − 1) 2 𝜎� 𝑁 2 𝜒𝛾 (𝑓𝐹 − 1)

1 ⎡ 2 𝜎 =⎢ 𝑁 ⎢ ⎣0

𝜇�𝑁 −

2 𝜎� 𝑁

𝜎 � 𝑁 𝑡 1−γ (nF − 1) √𝑓𝐹 � 2 �

(𝑓𝐹 − 1) 𝜒 21+γ (𝑓𝐹 − 1) �

2

Lognormal

In most cases the unbiased estimators are used: 2 ∑�ln�𝑡𝑖𝐹 � − 𝜇�𝑡 � ∑ ln(𝑡𝑖𝐹 ) 𝜇�𝑁 = σ�2N = nF nF − 1

�

2 Sided - Upper 𝜇�𝑁 +

2 𝜎� 𝑁

𝜎 � 𝑁

√𝑓𝐹

𝑡�1−γ� (nF − 1) 2

(𝑓𝐹 − 1) 𝜒 21−γ (𝑓𝐹 − 1) �

2

�

Where 𝑡γ (nF − 1) is the 100𝛾 th percentile of the t-distribution with 𝑓𝐹 − 1 degrees of freedom and 𝜒𝛾2 (𝑓𝐹 − 1) is the 100𝛾 th percentile of the 𝜒 2 -distribution with 𝑓𝐹 − 1 degrees of freedom. (Nelson 1982, pp.218-219) P

P

1 Sided - Lower

𝝁

exp �𝜇� 𝑁+

2 Sided

2 2 4 𝜎� 𝜎� 𝜎� 𝑁 𝑁 𝑁 − 𝑍1−𝛼 � + � 2 𝑓𝐹 2(𝑓𝐹 − 1)

exp �𝜇� 𝑁+

2 2 4 𝜎� 𝜎� 𝜎� 𝑁 𝑁 𝑁 ± 𝑍1−𝛼�2 � + � 2 𝑓𝐹 2(𝑓𝐹 − 1)

These formulas are the Cox approximation for the confidence intervals of the lognormal distribution mean where 𝑍𝑝 = 𝛷 −1 (𝑝), the inverse of the standard normal cdf. (Zhou & Gao 1997) Zhou & Gao recommend using the parametric bootstrap method for small sample sizes. (Angus 1994) Bayesian

Type

Non-informative Priors when 𝝈𝟐𝑵 is known, 𝝅𝟎 (𝝁𝑵 ) (Yang and Berger 1998, p.22)

Uniform Proper Prior with limits 𝜇𝑁 ∈ [𝑠, 𝑏]

Prior

Posterior

1 𝑏−𝑠

Truncated Normal Distribution For a ≤ µN ≤ b ∑𝑛𝐹 ln 𝑡𝑖𝐹 σ2N 𝑐. 𝑁𝑓𝑠𝑚 �µN ; 𝑖=1 , � 𝑓𝐹 nF Otherwise 𝜋(𝜇𝑁 ) = 0

54

Common Life Distributions

Lognormal

𝑁𝑓𝑠𝑚 �µN ;

Non-informative Priors when 𝝁𝑵 is known, 𝝅𝒐 (𝛔𝟐𝐍 ) (Yang and Berger 1998, p.23)

Type

Prior

Posterior

Uniform Proper Prior with limits 𝜎𝑁2 ∈ [𝑠, 𝑏]

1 𝑏−𝑠

Uniform Improper Prior with limits 𝜎𝑁2 ∈ (0, ∞)

1

Truncated Inverse Gamma Distribution For a ≤ σ2N ≤ b 2 (𝑓𝐹 − 2) SN 𝑐. 𝐼𝐺 �σ2N ; , � 2 2 Otherwise 𝜋(𝜎𝑁2 ) = 0 2 (𝑓𝐹 − 2) SN , � 2 2 See section 1.7.1

𝐼𝐺 �σ2N ;

1 𝜎𝑁2

Jeffery’s, Reference, MDIP Prior

Type

𝐹 ∑𝑛𝑖=1 ln 𝑡𝑖𝐹 σ2N , � 𝑓𝐹 nF when 𝜇𝑁 ∈ (∞, ∞)

1

All

2 𝑓𝐹 SN , � 2 2 with limits 𝜎𝑁2 ∈ (0, ∞) See section 1.7.1

𝐼𝐺 �σ2N ;

Non-informative Priors when 𝝁𝑵 and 𝝈𝟐𝑵 are unknown, 𝝅𝒐 (𝝁𝑵 , σ2N ) (Yang and Berger 1998, p.23) Prior

Posterior

Improper Uniform with limits: 𝜇𝑁 ∈ (∞, ∞) 𝜎𝑁2 ∈ (0, ∞)

1

Jeffery’s Prior

1 𝜎𝑁4

Reference Prior ordering {𝜙, 𝜎}

∝

𝜋𝑜 (𝜙, 𝜎𝑁2 ) 1

𝜎𝑁 �2 + 𝜙 2 where 𝜙 = 𝜇𝑁 /𝜎𝑁

2 SN � 𝑓𝐹 (𝑓𝐹 − 3) See section 1.7.2 2 (𝑓𝐹 − 3) SN 𝜋(σ2N |𝐸)~𝐼𝐺 �σ2N ; , � 2 2 See section 1.7.1

𝜋(𝜇𝑁 |𝐸)~𝑇 �µN ; 𝑓𝐹 − 3, ��� 𝑡𝑁 ,

S2 � 𝑓𝐹 (𝑓𝐹 + 1) when 𝜇𝑁 ∈ (∞, ∞) See section 1.7.2 2 (𝑓𝐹 + 1) SN 2 , � 𝜋(σN |𝐸)~𝐼𝐺 �σ2N ; 2 2 when 𝜎𝑁2 ∈ (0, ∞) See section 1.7.1

𝜋(𝜇𝑁 |𝐸)~𝑇 �µN ; 𝑁 𝐹 + 1, ��� 𝑡𝑁 ,

No closed form

Lognormal Continuous Distribution

1 𝜎𝑁

Reference where 𝜇 and 𝜎 2 are separate groups.

2 SN � nF (nF − 1) when 𝜇𝑁 ∈ (∞, ∞) See section 1.7.2 (𝑓𝐹 − 1) 𝑆𝑁2 𝜋(σ2N |𝐸)~𝐼𝐺 �σ2N ; , � 2 2 when 𝜎𝑁2 ∈ (0, ∞) See section 1.7.1

𝜋(𝜇𝑁 |𝐸)~𝑇 �µN ; 𝑁 𝐹 − 1, ��� 𝑡𝑁 ,

MDIP Prior

𝑆𝑁2 UOI

𝑛𝐹

= �(ln 𝑡𝑖 − 𝑖=1

Likelihood Model

𝜎𝑁2 from

LogNormal with known 𝜇𝑁 𝐿𝑓𝑅𝑁(𝑡; 𝜇𝑁 , 𝜎𝑁2 )

𝜇𝑁 from

LogNormal with known 𝐿𝑓𝑅𝑁(𝑡; 𝜇𝑁 , 𝜎𝑁2 ) 𝜎𝑁2

2 𝑡��� 𝑁)

and

𝑛𝐹

1 ��� 𝑡𝑁 = � ln 𝑡𝑖 nF 𝑖=1

Conjugate Priors

Evidence

𝑓𝐹 failures at times 𝑡𝑖

Dist of UOI

Gamma

𝑓𝐹 failures at times 𝑡𝑖

Normal

Prior Para

Posterior Parameters 𝑘 = 𝑘𝑜 + 𝑓𝐹 /2 𝑛𝐹

𝑘0 , 𝜆 0

1 𝜆 = 𝜆𝑜 + �(ln 𝑡𝑖 − 𝜇𝑁 )2 2 𝑖=1

𝜇𝑜 , 𝜎𝑜2

σ2 =

Description , Limitations and Uses Example

𝑛

𝐹 ln(𝑡𝑖 ) µ0 ∑𝑖=1 2+ σ0 𝜎𝑁2 𝜇= 1 𝑓𝐹 + 𝜎02 𝜎𝑁2

1 1 nF + σ20 𝜎𝑁2

5 components are put on a test with the following failure times: 98, 116, 2485, 2526, , 2920 hours Taking the natural log of these failure times allows us to use a normal distribution to approximate the parameters. ln(𝑡𝑖 ): 4.590, 4.752, 7.979, 7.818, 7.834 𝑠𝑓(hours) MLE Estimates are:

∑ ln�𝑡𝑖𝐹 � 32.974 = = 6.595 nF 5 2 ∑�ln�𝑡𝑖𝐹 � − 𝜇�𝑡 � 2 = 3.091 𝜎� 𝑁 = nF − 1

𝜇�𝑁 =

90% confidence interval for 𝜇𝑁 : 𝜎 �𝑁 �𝜇�𝑁 − 𝑡{0.95} (4), √4

𝜇 �𝑁 +

𝜎 �𝑁

√4

𝑡{0.95} (4)�

Lognormal

where

55

56

Common Life Distributions

[4.721, 8.469]

Lognormal

90% confidence interval for 𝜎𝑁2 : 4 4 2 2 �𝜎� , 𝜎� � 𝑁 2 𝑁 2 𝜒{0.95} (4) 𝜒{0.05} (4) [1.303, 17.396] A Bayesian point estimate using the Jeffery non-informative improper prior 1⁄𝜎𝑁4 with posterior for 𝜇𝑁 ~𝑇(6, 6.595, 0.412 ) and 𝜎𝑁2 ~𝐼𝐺(3, 6.182) has a point estimates: 𝜇�𝑁 = E[𝑇(6,6.595,0.412 )] = µ = 6.595

6.182 2 = 3.091 𝜎� 𝑁 = E[𝐼𝐺(3,6.182)] = 2

Characteristics

With 90% confidence intervals: 𝜇𝑁 [𝐹𝑇−1 (0.05) = 5.348, 2 𝜎𝑁 [1/𝐹𝐺−1 (0.95) = 0.982,

𝐹𝑇−1 (0.95) = 7.842]

1/𝐹𝐺−1 (0.05) = 7.560]

𝝁𝑵 Characteristics. 𝜇𝑁 determines the scale and not the location as in a normal distribution. The distribution if fixed at f(0)=0 and an increase in the scale parameter stretches the distribution across the x-axis. This has the effect of increasing the mode, mean and median of the distribution. 𝝈𝑵 Characteristics. 𝜎𝑁 determines the shape and not the scale as in a normal distribution. For values of σ N > 1 the distribution rises very sharply at the beginning and decreases with a shape similar to an Exponential or Weibull with 0 < 𝛽 < 1. As σ N → 0 the mode, mean and median converge to 𝑅 𝜇𝑁 . The distribution becomes narrower and approaches a Dirac delta function at 𝑡 = 𝑅 𝜇𝑁 .

Hazard Rate. (Kleiber & Kotz 2003, p.115)The hazard rate is unimodal with ℎ(0) = 0 and all dirivitives of ℎ’(𝑡) = 0 and a slow decrease to zero as 𝑡 → 0. The mode of the hazard rate: 𝑡𝑚 = exp(𝜇 + 𝑧𝑚 𝜎) where 𝑧𝑚 is given by: 𝜙(𝑧𝑚 ) (𝑧𝑚 + 𝜎𝑁 ) = 1 − Φ(𝑧𝑚 ) therefore −𝜎𝑁 < 𝑧𝑚 < −𝜎𝑁 + 𝜎 −1 and therefore: 2

2

𝑅 𝜇𝑁 −𝜎𝑁 < 𝑡𝑚 < 𝑅 𝜇𝑁 −𝜎𝑁 +1 2 𝜇𝑁 −𝜎𝑁 As 𝜎𝑁 → ∞, 𝑡𝑚 → 𝑅 and so for large 𝜎𝑁 : exp�𝜇𝑁 − 12𝜎𝑁2 � max ℎ(𝑡) ≈ 𝜎𝑁 √2𝜋

Lognormal Continuous Distribution

57

2

As 𝜎𝑁 → 0, 𝑡𝑚 → 𝑅 𝜇𝑁 −𝜎𝑁 +1 and so for large 𝜎𝑁 : 1 max ℎ(𝑡) ≈ 2 𝜇 −𝜎2 +1 𝑁 𝑁 𝜎𝑁 𝑅

Scale/Product Property: Let: 𝑠𝑗 𝑋𝑗 ~𝐿𝑓𝑅𝑁�𝜇𝑁𝑗 , σ2Nj � If 𝑋𝑗 and 𝑋𝑗+1 are independent:

2 � � 𝑠𝑗 𝑋𝑗 ~𝐿𝑓𝑅𝑁 ���𝜇𝑁𝑗 + ln�𝑠𝑗 ��, � 𝜎𝑁𝑗

Lognormal versus Weibull. In analyzing life data to these distributions it is often the case that both may be a good fit, especially in the middle of the distribution. The Weibull distribution has an earlier lower tail and produces a more pessimistic estimate of the component life. (Nelson 1990, p.65) Applications

General Life Distributions. The lognormal distribution has been found to accurately model many life distributions and is a popular choice for life distributions. The increasing hazard rate in early life models the weaker subpopulation (burn in) and the remaining decreasing hazard rate describes the main population. In particular this has been applied to some electronic devices and fatiguefracture data. (Meeker & Escobar 1998, p.262) Failure Modes from Multiplicative Errors. The lognormal distribution is very suitable for failure processes that are a result of multiplicative errors. Specific applications include failure of components due to fatigue cracks. (Provan 1987) Repair Times. The lognormal distribution has commonly been used to model repair times. It is natural for a repair time probability to increase quickly to a mode value. For example very few repairs have an immediate or quick fix. However, once the time of repair passes the mean it is likely that there are serious problems, and the repair will take a substantial amount of time. Parameter Variability. The lognormal distribution can be used to model parameter variability. This was done when estimating the uncertainty in the parameter 𝜆 in a Nuclear Reactor Safety Study (NUREG-75/014). Theory of Breakage. The distribution models particle sizes observed in breakage processes (Crow & Shimizu 1988)

Resources

Online:

Lognormal

Mean / Median / Mode: 𝑚𝑓𝑝𝑅(𝑋) < 𝑚𝑅𝑝𝑠𝑠𝑓(𝑋) < 𝐸[𝑋]

58

Common Life Distributions

Lognormal

http://www.weibull.com/LifeDataWeb/the_lognormal_distribution.ht m http://mathworld.wolfram.com/LogNormalDistribution.html http://en.wikipedia.org/wiki/Log-normal_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (web calc) Books: Crow, E.L. & Shimizu, K., 1988. Lognormal distributions, CRC Press. Aitchison, J.J. & Brown, J., 1957. The Lognormal Distribution, New York: Cambridge University Press. Nelson, W.B., Interscience.

1982.

Applied

Life

Data

Analysis,

Relationship to Other Distributions Normal Distribution 𝑁𝑓𝑠𝑚(𝑡; 𝜇, 𝜎 2 )

Let;

Then: Where:

𝑋~𝐿𝑓𝑅𝑁(𝜇𝑁 , σ2N ) 𝑌 = ln(𝑋) 𝑌~𝑁𝑓𝑠𝑚(𝜇, 𝜎 2 )

𝜇2 𝜇𝑁 = ln � �, �𝜎 2 + 𝜇2

𝜎 2 + 𝜇2 𝜎𝑁 = ln � � 𝜇2

Wiley-

Weibull Continuous Distribution

59

2.3. Weibull Continuous Distribution Probability Density Function - f(t) 0.90

α=1 β=2 α=1.5 β=2 α=2 β=2 α=3 β=2

0.80 0.70 0.60 0.50 0.40 0.30 0.20 0.10 0.00

0

1

2

0

2

4

Cumulative Density Function - F(t) 1.00

1.00

0.80

0.80

0.60

0.60

0.40

α=1 β=0.5 α=1 β=1 α=1 β=2 α=1 β=5

0.20 0.00 0

1

6

α=1 β=2 α=1.5 β=2 α=2 β=2 α=3 β=2

0.20 0.00

2

Hazard Rate - h(t)

0

10.00 9.00 8.00 7.00 6.00 5.00 4.00 3.00 2.00 1.00 0.00

5 4 3 2 1 0 0

0.40

1

2

2

4

α=1 β=2 α=1.5 β=2 α=2 β=2 α=3 β=2

0

2

4

Weibull

α=1 β=0.5 α=1 β=1 α=1 β=2 α=1 β=5

1.80 1.60 1.40 1.20 1.00 0.80 0.60 0.40 0.20 0.00

60

Common Life Distributions

Weibull

Parameters & Description

Parameters

𝛼 𝛽

Scale Parameter: The value of 𝛼 equals the 63.2th percentile and has a unit equal to 𝑡. Note that this is not equal to the mean.

𝛼>0

Shape Parameter: Also known as the slope (referring to a linear CDF plot) 𝛽 determines the shape of the distribution.

𝛽>0

𝑡 ≥ 0

Limits Distribution

Formulas 𝑝(𝑡) =

PDF CDF

𝛽𝑡 𝛽−1 −� 𝑡 �𝛽 𝑅 𝛼 𝛼𝛽

𝐹(𝑡) = 1 − 𝑅

Reliability

R(t) = 𝑅

Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡)

𝑚(𝑚) = 𝑅(𝑚|𝑡) =

Residual

𝑡 𝛽 −� � 𝛼

𝑡 � 𝑅(𝑡 + x) =𝑅 𝑅(𝑡)

𝛽 −(𝑡+x)𝛽

𝛼𝛽

�

Where 𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡. (Kleiber & Kotz 2003, p.176)

Mean Life

𝑡 𝛽 −� � 𝛼

𝑠(𝑡) = 𝑅

𝑡 𝛽 � � 𝛼

∞

� 𝑅 t

𝑥 𝛽 −� � 𝛼

which has the asymptotic property of: lim 𝑠(𝑡) = 𝑡 1−𝛽

𝑝𝑚

𝑡→∞

Hazard Rate Cumulative Hazard Rate

β 𝑡 𝛽−1 ℎ(𝑡) = . � � α 𝛼 𝑡 𝛽 𝐻(𝑡) = � � 𝛼

Properties and Moments Median Mode

1

𝛼�𝑠𝑓(2)�𝛽 1

𝛽−1 𝛽 𝛼� � if 𝛽 ≥ 1 𝛽 otherwise no mode exists

Weibull Continuous Distribution

61

1 𝛼Γ �1 + � 𝛽

Mean - 1st Raw Moment

3 Γ �1 + � α3 − 3µσ2 − µ3 β σ3

Skewness - 3rd Central Moment

−6Γ14 + 12Γ12 Γ2 − 3Γ22 − 4Γ1 Γ3 + Γ4 (Γ2 − Γ12 )2 where: 𝑠 Γ𝑖 = Γ �1 + � 𝛽

Excess kurtosis - 4th Central Moment

∞

Characteristic Function

�

𝑛=0

100p% Percentile Function

(𝑠𝑡)𝑛 𝛼 𝑛 𝑓 Γ �1 + � 𝑓! 𝛽 1

𝑡𝑝 = 𝛼[− ln(1 − 𝑝)]𝛽

Parameter Estimation Plotting Method Least Mean Square 𝑦 = 𝑚𝑚 + 𝑐 Likelihood Functions

X-Axis

𝛼� = 𝑅 −𝑚 𝛽̂ = 𝑚

1 ln �𝑠𝑓 � �� 1−𝐹

ln(𝑡𝑖 )

Maximum Likelihood Function tF

𝛽−1

𝛽

tS

𝛽

nS −� i � −� i � 𝛽�𝑡𝑖𝐹 � L(α, β|E) = � 𝑅 𝛼 .� 𝑅 𝛼 𝛽 𝛼 ����������������� ��������� i=1 i=1 nF nI

Log-Likelihood Function

𝑐

Y-Axis

failures 𝛽 tLI −� i � 𝛼

−�

tRI i � 𝛼

𝛽

survivors

� �𝑅 −𝑅 � i=1 ������������������� interval failures nF

𝛽

𝑡𝑖𝐹 Λ(α, β|E) = nF ln(β) − βnF ln(𝛼) + � �(𝛽 − 1) ln�𝑡𝑖𝐹 � − � � � 𝛼 ����������������������������������� i=1 nS

β t Si

nI

failures β tLI −� i � α

tRI i � α

−�

β

− � � � + � ln �e −e � α �� �� ������������������� i=1��� i=1 survivors

interval failures

Weibull

2 1 𝛼 2 �Γ �1 + � − Γ 2 �1 + �� 𝛽 𝛽

Variance - 2nd Central Moment

62

Common Life Distributions

∂Λ =0 ∂α

solve for 𝛼 to get 𝛼�:

nS

nF

∂Λ −βnF β β β β = + β+1 ��t Fi � + β+1 ��t Si � ∂α α α α ��������������� ������ i=1 i=1 ��� failures

Weibull

β tRI �i � t LI α i

∂Λ =0 ∂β

survivors β β tLI �i � t RI α i

nI � � e −� α � e ⎞ β⎛ α ⎟=0 +� ⎜ β β RI LI ⎟ α⎜ t t �i � �i � i=1 e α −e α ������������������������� ⎝ ⎠ interval failures

solve for 𝛽 to get 𝛽̂: nF

nS

β

β

t Fi t Fi t Fi t Si t Si ∂Λ nF = + � �ln � � − � � . ln � �� − � � � ln � � β α α 𝛼 α 𝛼 ∂β ����������������������� ����������� i=1 i=1 t RI i

MLE Point Estimates

β

failures

β t RI i

tLI i � α

�

β

survivors β β tRI �i � t LI α i

t LI i

nI − ln � 𝛼 � . � α � . e ⎛ln � 𝛼 � . � α � . e ⎞ ⎟=0 +�⎜ β β ⎜ ⎟ tRI tLI �i � �i � i=1 e α −e α ������������������������������������� ⎝ ⎠ interval failures

When there is only complete failure and/or right censored data the point estimates can be solved using (Rinne 2008, p.439): 1 � β � β

� β

� β

∑�t Fi � + ∑�t Si � α �=� � nF

−1

� β

∑�t Fi � ln�t Fi � + ∑�t Si � ln�t Si � 1 β� = � − � ln(t Fi )� � � β β 𝑓𝐹 S F ∑�t i � + ∑�t i �

Note: Numerical methods are needed to solve β� then substitute to find α �. Numerical methods to find Weibull MLE estimates for complete and censored data for 2 parameter and 3 parameter Weibull distribution are detailed in (Rinne 2008). Fisher Information Matrix (Rinne p.412)

2008,

𝛽2 𝛽2 Γ ′ (2) ⎡ ⎤ ⎡ 𝛼2 𝛼2 −𝛼 ⎥ ⎢ 𝐼(𝛼, 𝛽) = ⎢ ′ ′′ (2) = ⎢ (2) Γ 1 + Γ ⎢ ⎥ ⎢1 − 𝛾 𝛽2 ⎣ −𝛼 ⎦ ⎣ 𝛼 𝛽2 ⎡ 2 ≅⎢ 𝛼 ⎢0.422784 ⎣ −𝛼

1−𝛾 𝛼

⎤ ⎥ 𝜋2 2 ⎥ 6 + (1 − 𝛾 ) ⎥ ⎦ 𝛽2

0.422784 ⎤ −𝛼 ⎥ 1.823680 ⎥ 𝛽2 ⎦

Weibull Continuous Distribution

63

(complete data)

Bayesian

Bayesian analysis is applied to either one of two re-parameterizations of the Weibull Distribution: (Rinne 2008, p.517) 𝑝(𝑡; 𝜆, 𝛽) = 𝜆𝛽𝑡 𝛽−1 exp�−𝜆𝑡 𝛽 �

or

𝛽 𝛽−1 𝑡𝛽 𝑡 exp �− � 𝜃 𝜃

𝑝(𝑡; 𝜃, 𝛽) =

where 𝜃 =

1 = 𝛼𝛽 𝜆

Non-informative Priors 𝝅𝟎 (𝝀) (Rinne 2008, p.517)

Type

Prior

Prior

when

𝛽

is

Jeffrey’s Prior for unknown 𝜃 and 𝛽. where

Posterior

1 𝑏−𝑠

Uniform Proper Prior with known 𝛽 and limits 𝜆 ∈ [𝑠, 𝑏] Jeffrey’s known.

where 𝜆 = 𝛼 −𝛽

Truncated Gamma Distribution For a ≤ λ ≤ b 𝑐. 𝐺𝑠𝑚𝑚𝑠(𝜆; 1 + nF , t T,β )

1 ∝ 𝐺𝑠𝑚𝑚𝑠(0,0) 𝜆

β

1 𝜃𝛽

Otherwise 𝜋(𝜆) = 0

𝐺𝑠𝑚𝑚𝑠(𝜆; nF , t T,β ) when 𝜆 ∈ [0, ∞)

No closed form (Rinne 2008, p.527)

β

𝑡𝑇,𝛽 = ∑�t Fi � + ∑�t Si � = adjusted total time in test Conjugate Priors

It was found by Soland that no joint continuous prior distribution exists for the Weibull distribution. Soland did however propose a procedure which used a continuous distribution for α and a discrete distribution for β which will not be included here. (Martz & Waller 1982) UOI 𝜆 where 𝜆 = 𝛼 −𝛽 from 𝑊𝑏𝑠(𝑡; 𝛼, 𝛽)

Likelihood Model

Evidence

Weibull with known 𝛽

𝑓𝐹 failures at times 𝑡𝑖𝐹

Dist of UOI

Gamma

Prior Para

Posterior Parameters

𝑘0 , Λ 0

𝑘 = 𝑘𝑜 + 𝑓𝐹 Λ = Λ0 + 𝑡𝑇,𝛽 (Rinne 2008, p.520)

Weibull

The asymptotic variance-covariance matrix of (𝛼�, 𝛽̂ ) is: (Rinne 2008, pp.412-417) 𝛼� 2 1 1.1087 0.2570𝛼� −1 𝐶𝑓𝑠�𝛼�, 𝛽̂ � = �𝐽𝑛 �𝛼�, 𝛽̂ �� = � � 𝛽̂ 2 𝑓𝐹 0.2570𝛼� 0.6079𝛽̂ 2

100𝛾% Confidence Interval

Weibull

64

Common Life Distributions

𝜃 where 𝜃 = 𝛼𝛽 from 𝑊𝑏𝑠(𝑡; 𝛼, 𝛽)

Weibull with known 𝛽

Example

𝑓𝐹 failures at times 𝑡𝑖𝐹

Inverted Gamma

𝛼 = 𝛼𝑜 + 𝑓𝐹 β = β0 + 𝑡𝑇,𝛽 (Rinne 2008, p.524)

𝛼0 , β0

Description , Limitations and Uses 5 components are put on a test with the following failure times: 535, 613, 976, 1031, 1875 hours β� is found by numerically solving:

−1

� β

∑�t Fi � ln�t Fi � − 6.8118� β� = � � β ∑�t Fi �

α � is found by solving:

Covariance Matrix is:

β� = 2.275

1 � β � β F ∑�t i �

α �=�

𝛼� 2 1 1.1087 𝐶𝑓𝑠�𝛼�, 𝛽̂ � = � 𝛽̂ 2 5 0.2570𝛼�

nF

� = 1140

0.2570𝛼�

0.6079𝛽̂

90% confidence interval for 𝛼�: 𝛷 −1 (0.95)√55679 �, �𝛼�. exp � −𝛼� [811, 90% confidence interval for 𝛽: 𝛷 −1 (0.95)√0.6293 �, �𝛽̂ . exp � −𝛽̂ [1.282,

55679 58.596 � �=� 58.596 0.6293 2

𝛼�. exp �

1602]

𝛷 −1 (0.95)√55679 �� 𝛼�

𝛽̂ . exp �

4.037]

𝛷 −1 (0.95)√0.6293 �� 𝛽̂

Note that with only 5 samples the assumption that the parameter distribution is approximately normal is probably inaccurate and therefore the confidence intervals need to be used with caution. Characteristics

The Weibull distribution is also known as a “Type III asymptotic distribution for minimum values”. β Characteristics: β < 1. The hazard rate decreases with time.

Weibull Continuous Distribution

65

Note that for 𝛽 = 0.999, 𝑝(0) = ∞, but for 𝛽 = 1.001, 𝑝(0) = 0. This rapid change creates complications when maximizing likelihood functions. (Weibull.com) As 𝛽 → ∞, the 𝑚𝑓𝑝𝑅 → 𝛼.

α Characteristics. Increasing 𝛼 stretches the distribution over the time scale. With the 𝑝(0) point fixed this also has the effect of increasing the mode, mean and median. The value for 𝛼 is at the 63% Percentile. 𝐹(𝛼) = 0.632.. 𝑋~𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝛼, 𝛽)

Scaling property: (Leemis & McQueston 2008) 𝑘𝑋~𝑊𝑅𝑠𝑏𝑠𝑠𝑠�𝛼𝑘𝛽 , 𝛽�

Minimum property (Rinne 2008, p.107)

min{𝑋, 𝑋2 , … , 𝑋𝑛 }~𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝛼𝑓 When 𝛽 is fixed. Variate Generation property

−

1 𝛽 , 𝛽)

1

𝐹 −1 (𝑠) = 𝛼[− ln(1 − 𝑠)]𝛽 , 0 < 𝑠 < 1

Lognormal versus Weibull. In analyzing life data to these distributions it is often the case that both may be a good fit, especially in the middle of the distribution. The Weibull distribution has an earlier lower tail and produces a more pessimistic estimate of the component life. (Nelson 1990, p.65) Applications

The Weibull distribution is by far the most popular life distribution used in reliability engineering. This is due to its variety of shapes and generalization or approximation of many other distributions. Analysis assuming a Weibull distribution already includes the

Weibull

β = 1. The hazard rate is constant (exp distribution) β > 1. The hazard rate increases with time. 1 < β < 2. The hazard rate increases less as time increases. β = 2. The hazard rate increases with a linear relationship to time. β > 2. The hazard rate increases more as time increases. β < 3.447798. The distribution is positively skewed. (Tail to right). β ≈ 3.447798. The distribution is approximately symmetrical. β > 3.447798. The distribution is negatively skewed (Tail to left). 3 < β < 4. The distribution approximates a normal distribution. β > 10. The distribution approximates a Smallest Extreme Value Distribution.

66

Common Life Distributions

exponential life distribution as a special case.

Weibull

There are many physical interpretations of the Weibull Distribution. Due to its minimum property a physical interpretation is the weakest link, where a system such as a chain will fail when the weakest link fails. It can also be shown that the Weibull Distribution can be derived from a cumulative wear model (Rinne 2008, p.15) The following is a non-exhaustive list of applications where the Weibull distribution has been used in: • Acceptance sampling • Warranty analysis • Maintenance and renewal • Strength of material modeling • Wear modeling • Electronic failure modeling • Corrosion modeling A detailed list with references to practical examples is contained in (Rinne 2008, p.275) Resources

Online: http://www.weibull.com/LifeDataWeb/the_weibull_distribution.htm http://mathworld.wolfram.com/WeibullDistribution.html http://en.wikipedia.org/wiki/Weibull_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (interactive web calculator) http://www.qualitydigest.com/jan99/html/weibull.html (how to use conduct Weibull analysis in Excel, William W. Dorner) Books: Rinne, H., 2008. The Weibull Distribution: A Handbook 1st ed., Chapman & Hall/CRC. Murthy, D.N.P., Xie, M. & Jiang, R., 2003. Weibull Models 1st ed., Wiley-Interscience. Nelson, W.B., Interscience.

1982.

Applied

Life

Data

Analysis,

Wiley-

Relationship to Other Distributions Three Parameter Weibull Distribution 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽, 𝛾)

The three parameter model adds a locator parameter to the two parameter Weibull distribution allowing a shift along the x-axis. This creates a period of guaranteed zero failures to the beginning of the product life and is therefore only used in special cases. Special Case: 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽) = 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽, 𝛾 = 0)

Weibull Continuous Distribution

Let Exponential Distribution

Rayleigh Distribution 𝑅𝑠𝑦𝑠𝑅𝑠𝑅ℎ(𝑡; 𝛼)

χ Distribution 𝜒(𝑡|𝑠)

Special Case:

Special Case:

𝑠𝑓𝑝

𝑌~𝐸𝑚𝑝(λ = 𝛼 −𝛽 )

𝑌 = Xβ

1 𝐸𝑚𝑝(𝑡; 𝜆) = 𝑊𝑅𝑠𝑏𝑠𝑠𝑠 �𝑡; 𝛼 = , 𝛽 = 1� 𝜆

𝑅𝑠𝑦𝑠𝑅𝑠𝑅ℎ(𝑡; 𝛼) = 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽 = 2) Special Case: 𝜒(𝑡|𝑠 = 2) = 𝑊𝑅𝑠𝑏𝑠𝑠𝑠�𝑡|𝛼 = √2, 𝛽 = 2�

Weibull

𝐸𝑚𝑝(𝑡; 𝜆)

Then

𝑋~𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝛼, 𝛽)

67

68

Probability Distributions Used in Reliability Engineering

3. Bathtub Life Distributions

2-Fold Mixed Weibull Distribution

69

3.1. 2-Fold Mixed Weibull Distribution All shapes shown are variations from 𝑝 = 0.5 𝛼1 = 2 𝛽1 = 0.5 𝛼2 = 10 𝛽2 = 20 Probability Density Function - f(t) α1=2 β1=0.1 α1=2 β1=0.5 α1=2 β1=1 α1=2 β1=5

α2=10 β2=20 α2=10 β2=10 α2=10 β2=1 α2=5 β2=10

p=0 p=0.3 p=0.7 p=1

Mixed Wbl

0

5

10

0

5

10

0

5

10

5

10

0

10

10

0

10

Cumulative Density Function - F(t)

0

5

10

0

10

0

Hazard Rate - h(t)

0

5

70

Bathtub Life Distributions

Parameters & Description

Parameters

Limits

𝛼𝑖

𝛼𝑖 > 0

𝑝

0≤𝑝≤1

𝛽𝑖

Scale Parameter: This is the scale for each Weibull Distribution. Shape Parameters: The shape of each Weibull Distribution

𝛽𝑖 > 0

Mixing Parameter. This determines the weight each Weibull Distribution has on the overall density function. 𝑡 ≥ 0

Mixed Wbl

Distribution

Formulas 𝑝(𝑡) = 𝑝𝑝1 (𝑡) + (1 − 𝑝)𝑝2 (𝑡)

PDF

where 𝑝𝑖 (𝑡) =

𝛽𝑖 𝑡 𝛽𝑖 −1 −�𝛼𝑡 � 𝑅 𝑖 𝛼𝑖 𝛽𝑖

𝛽𝑖

and 𝑠 ∈ {1,2}

𝐹(𝑡) = 𝑝𝐹1 (𝑡) + (1 − 𝑝)𝐹2 (𝑡)

CDF

where 𝐹𝑖 (𝑡) = 1 − 𝑅

𝑡 𝛽𝑖 � 𝛼𝑖

and 𝑠 ∈ {1,2}

𝑅(𝑡) = 𝑝𝑅1 (𝑡) + (1 − 𝑝)𝑅2 (𝑡)

Reliability

Hazard Rate

−�

where 𝑅𝑖 (𝑡) = 𝑅

where

−�

𝑡 𝛽𝑖 � 𝛼𝑖

and 𝑠 ∈ {1,2}

ℎ(𝑡) = 𝑤1 (𝑡)ℎ1 (𝑡) + 𝑤2 (𝑡)ℎ2 (𝑡) 𝑤𝑖 (𝑡) =

𝑝𝑖 𝑅𝑖 (𝑡) ∑𝑛𝑖=1 𝑝𝑖 𝑅𝑖 (𝑡)

and 𝑠 ∈ {1,2}

Properties and Moments Median

Solved numerically

Mode

Solved numerically st

Mean - 1 Raw Moment 𝑝𝛼1 Γ �1 + Variance - 2nd Central Moment

1 1 � + (1 − p)𝛼2 Γ �1 + � 𝛽1 𝛽2

𝑝. 𝑉𝑠𝑠[𝑇1 ] + (1 − 𝑝)𝑉𝑠𝑠[𝑇2 ] +𝑝(𝐸[𝑋1 ] − 𝐸[𝑋])2 +(1 − 𝑝)(𝐸[𝑋2 ] − 𝐸[𝑋])2 2

1

𝑝. 𝛼 2 �Γ �1 + 𝛽 � − Γ 2 �1 + 𝛽 �� 1

1

2-Fold Mixed Weibull Distribution

2

71

1

+(1 − 𝑝)𝛼 2 �Γ �1 + 𝛽 � − Γ 2 �1 + 𝛽 �� 1

2

+𝑝 �𝛼1 Γ �1 + 𝛽 � − 𝐸[𝑋]� 1

2

2

1

2

+(1 − 𝑝) �𝛼2 Γ �1 + 𝛽 � − 𝐸[𝑋]� 2

100p% Percentile Function

Solved numerically Parameter Estimation

Plotting Method (Jiang & Murthy 1995) X-Axis

Y-Axis

𝑚 = 𝑠𝑓(𝑡)

1 𝑦 = 𝑠𝑓 �𝑠𝑓 � �� 1−𝐹

Using the Weibull Probability Plot the parameters can be estimated. Jiang & Murthy, 1995, provide a comprehensive coverage of this procedure and detail error in previous methods. A typical WPP for a 2-fold Mixed Weibull Distribution is: 2 1.5 1 0.5 0 -0.5 -1 -1.5 -2 -1 0 1 2 3 4 5 6 WPP for 2-Fold Weibull Mixture Model 𝑝 = 12, 𝛼1 = 5, 𝛽1 = 0.5, 𝛼2 = 10, 𝛽2 = 5

Sub Populations: The dotted lines in the WPP is the lines representing the subpopulations: 𝐿1 = 𝛽1 [𝑚 − ln(𝛼1 )] 𝐿2 = 𝛽2 [𝑚 − ln(𝛼2 )]

Mixed Wbl

Plot Points on a Weibull Probability Plot

72

Bathtub Life Distributions

Asymptotes (Jiang & Murthy 1995): As 𝑚 → −∞ (𝑡 → 0) there exists an asymptote approximated by: 𝑦 ≈ 𝛽1 [𝑚 − ln(𝛼1 )] + ln(𝑐) where 𝑝 𝑤ℎ𝑅𝑓 𝛽1 ≠ 𝛽2 𝛼1 𝛽1 𝑐=� 𝑝 + (1 − 𝑝). � � 𝑤ℎ𝑅𝑓 𝛽1 = 𝛽2 𝛼2

Mixed Wbl

As 𝑚 → ∞ (𝑡 → ∞) the asymptote straight line can be approximated by: 𝑦 ≈ 𝛽1 [𝑚 − ln(𝛼1 )] Parameter Estimation Jiang and Murthy divide the parameter estimation procedure into three cases: Well Mixed Case 𝜷𝟐 ≠ 𝜷𝟏 𝒂𝒏𝒅 𝜶𝟏 ≈ 𝜶𝟐 - Estimate the parameters of 𝛼1 and 𝛽1 from the 𝐿1 line (right asymptote). - Estimate the parameter 𝑝 from the separation distance between the left and right asymptotes. - Find the point where the curve crosses 𝐿1 (point I). The slope at point I is: 𝛽̅ = 𝑝𝛽1 + (1 − 𝑝)𝛽2 - Determine slope at point I and use to estimate 𝛽2 - Draw a line through the intersection point I with slope 𝛽2 and use the intersection point to estimate 𝛼2 . Well Separated Case 𝜷𝟐 ≠ 𝜷𝟏 𝒂𝒏𝒅 𝜶𝟏 ≫ 𝜶𝟐 𝒐𝒓 𝜶𝟏 ≪ 𝜶𝟐 - Determine visually if data is scattered along the bottom (or top) to determine if 𝛼1 ≪ 𝛼2 (or 𝛼1 ≫ 𝛼2 ). - If 𝛼1 ≪ 𝛼2 (𝛼1 ≫ 𝛼2 ) locate the inflection, 𝑦𝑎 , to the left (right) of the point I. This point 𝑦𝑎 ≅ ln[− ln(1 − 𝑝)] { or 𝑦𝑎 ≅ ln[− ln(𝑝)] }. Using this formula estimate p. - Estimate 𝛼1 and 𝛼2 : 𝑝 1−𝑝 • If 𝛼1 ≪ 𝛼2 calculate point 𝑦1 = ln �𝑠𝑓 �1 − 𝑝 + 𝑒𝑥𝑝(1)�� and 𝑦2 = ln �𝑠𝑓 �𝑒𝑥𝑝(1)��. •

Find the coordinates where 𝑦1 and 𝑦2 intersect the WPP curve. At these points estimate 𝛼1 = 𝑅 𝑥1 and 𝛼2 = 𝑅 𝑥2 . 𝑝 1−𝑝 If 𝛼1 ≫ 𝛼2 calculate point 𝑦1 = ln �− 𝑠𝑓 �𝑒𝑥𝑝(1)�� and 𝑦2 = ln �−𝑠𝑓 �𝑝 + 𝑒𝑥𝑝(1)��.

Find the coordinates where 𝑦1 and 𝑦2 intersect the WPP curve. At these points estimate 𝛼1 = 𝑅 𝑥1 and 𝛼2 = 𝑅 𝑥2 . - Estimate 𝛽1 : • If 𝛼1 ≪ 𝛼2 draw and approximate 𝐿2 ensuring it intersects 𝛼2 . Estimate 𝛽2 from the slope of 𝐿2 . • If 𝛼1 ≫ 𝛼2 draw and approximate 𝐿1 ensuring it intersects 𝛼1 . Estimate 𝛽1 from the slope of 𝐿1 . - Find the point where the curve crosses 𝐿1 (point I). The slope at point I is: 𝛽̅ = 𝑝𝛽1 + (1 − 𝑝)𝛽2 - Determine slope at point I and use to estimate 𝛽2

2-Fold Mixed Weibull Distribution

73

Common Shape Parameter 𝜷𝟐 = 𝜷𝟏 𝛼

If �𝛼2 � 1

𝛽1

≈1 then:

- Estimate the parameters of 𝛼1 and 𝛽1 from the 𝐿1 line (right asymptote). - Estimate the parameter 𝑝 from the separation distance between the left and right asymptotes. - Draw a vertical line through 𝑚 = ln(𝛼1 ). The intersection with the WPP can yield an estimate of 𝛼2 using: 𝑦1 = �

1

𝛽1

≪1 then:

1−𝑝 � 𝛼 𝛽1 exp �� 2 � � 𝛼1

- Find inflection point and estimate the y coordinate 𝑦𝑟 . Estimate p using: 𝑦𝑇 ≅ ln[− ln(𝑝)] 𝑝 1−𝑝 - If 𝛼1 ≪ 𝛼2 calculate point 𝑦1 = ln �𝑠𝑓 �1 − 𝑝 + 𝑒𝑥𝑝(1)�� and 𝑦2 = ln �𝑠𝑓 �𝑒𝑥𝑝(1)��. Find the

coordinates where 𝑦1 and 𝑦2 intersect the WPP curve. At these points estimate 𝛼1 = 𝑅 𝑥1 and 𝛼2 = 𝑅 𝑥2 . - Using the left or right asymptote estimate 𝛽1 = 𝛽2 from the slope. Maximum Likelihood Bayesian

MLE and Bayesian techniques can be used using numerical methods however estimates obtained from the graphical methods are useful for initial guesses. A literature review of MLE and Bayesian methods is covered in (Murthy et al. 2003). Description , Limitations and Uses

Characteristics

Hazard Rate Shape. The hazard rate can be approximated at its limits by (Jiang & Murthy 1995): 𝑆𝑚𝑠𝑠𝑠 𝑡: ℎ(𝑡) ≈ 𝑐ℎ1 (𝑡)

𝐿𝑠𝑠𝑅𝑅 𝑡: ℎ(𝑡) ≈ ℎ1

This result proves that the hazard rate (increasing or decreasing) of ℎ1 will dominate the limits of the mixed Weibull distribution. Therefore the hazard rate cannot be a bathtub curve shape. Instead the possible shapes of the hazard rate is: • Decreasing • Unimodal • Decreasing followed by unimodal (rollercoaster) • Bi-modal The reason this distribution has been included as a bathtub distribution is because on many occasions the hazard rate of a complex product may follow the “rollercoaster” shape instead which is given as decreasing followed by unimodal shape. The shape of the hazard rate is only determined by the two shape parameters 𝛽1 and 𝛽2 . A complete study on the characterization of

Mixed Wbl

𝛼

If �𝛼2 �

𝑝 + exp(1)

74

Bathtub Life Distributions

the 2-Fold Mixed Weibull Distribution is contained in Jiang and Murthy 1998. p Values The mixture ratio, 𝑝𝑖 , for each Weibull Distribution may be used to estimate the percentage of each subpopulation. However this is not a reliable measure and it known to be misleading (Berger & Sellke 1987) N-Fold Distribution (Murthy et al. 2003) A generalization to the 2-fold mixed Weibull distribution is the n-fold case. This distribution is defined as: 𝑛

Mixed Wbl

𝑝(𝑡) = � 𝑝𝑖 𝑝𝑖 (𝑡) 𝑖=1

𝛽

𝑛

𝛽𝑖 𝑡 𝛽𝑖 −1 −�𝛼𝑡 � 𝑖 𝑅 𝑖 𝑠𝑓𝑝 � 𝑝𝑖 = 1 where 𝑝𝑖 (𝑡) = 𝛼𝑖 𝛽𝑖

and the hazard rate is given as:

𝑛

𝑖=1

ℎ(𝑡) = � 𝑤𝑖 (𝑡)ℎ𝑖 (𝑡)

𝑤ℎ𝑅𝑠𝑅

𝑖=1

𝑤𝑖 (𝑡) =

𝑝𝑖 𝑅𝑖 (𝑡) ∑𝑛𝑖=1 𝑝𝑖 𝑅𝑖 (𝑡)

It has been found that in many instances a higher number of folds will not significantly increase the accuracy of the model but does impose a significant overhead in the number of parameters to estimate. The 3-Fold Weibull Mixture Distribution has been studied by Jiang and Murthy 1996. 2-Fold Weibull 3-Parameter Distribution A common variation to the model presented here is to have the second Weibull distribution modeled with three parameters. Resources

Books / Journals: Jiang, R. & Murthy, D., 1995. Modeling Failure-Data by Mixture of 2 Weibull Distributions : A Graphical Approach. IEEE Transactions on Reliability, 44, 477-488. Murthy, D., Xie, M. & Jiang, R., 2003. Weibull Models 1st ed., Wiley-Interscience. Rinne, H., 2008. The Weibull Distribution: A Handbook 1st ed., Chapman & Hall/CRC. Jiang, R. & Murthy, D., 1996. A mixture model involving three Weibull distributions. In Proceedings of the Second Australia– Japan Workshop on Stochastic Models in Engineering, Technology and Management. Gold Coast, Australia, pp. 260-270.

2-Fold Mixed Weibull Distribution

75

Jiang, R. & Murthy, D., 1998. Mixture of Weibull distributions parametric characterization of failure rate function. Applied Stochastic Models and Data Analysis, (14), 47-65. Balakrishnan, N. & Rao, C.R., 2001. Handbook of Statistics 20: Advances in Reliability 1st ed., Elsevier Science & Technology. Relationship to Other Distributions Weibull Distribution

Mixed Wbl

𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽)

Special Case: 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽) = 2𝐹𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼 = 𝛼1 , 𝛽 = 𝛽1 , 𝑝 = 1) 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽) = 2𝐹𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼 = 𝛼2 , 𝛽 = 𝛽2 , 𝑝 = 0)

76

Bathtub Life Distributions

3.2. Exponentiated Weibull Distribution Probability Density Function - f(t) 0.30

α=5 β=1.5 v=0.2 α=5 β=1.5 v=0.5 α=5 β=1.5 v=0.8 α=5 β=1.5 v=3

0.25

Exp-Weibull

0.20

0.30 0.25 0.20

0.15

0.15

0.10

0.10

0.05

0.05

0.00

α=5 β=1 v=0.5 α=5 β=1.5 v=0.5 α=5 β=2 v=0.5 α=5 β=3 v=0.5

0.00 0

5

10

15

0

5

0

5

0

5

Cumulative Density Function - F(t) 1.00

1.00

0.80

0.80

0.60

0.60

0.40

0.40

0.20

0.20

0.00

0.00 0

5

10

15

Hazard Rate - h(t) 0.60

0.60

0.50

0.50

0.40

0.40

0.30

0.30

0.20

0.20

0.10

0.10

0.00

0.00 0

5

10

15

Exponentiated Weibull Continuous Distribution

77

Parameters & Description

Parameters

𝛼

𝛽

Scale Parameter.

𝑠>0

Shape Parameter.

𝛽>0

𝑠

Limits

𝛼>0

Shape Parameter.

𝑡 ≥ 0

Distribution

Formulas

PDF

𝛽𝑠𝑡 𝛽−1 𝑡 𝛽 �1 − exp �− � � �� 𝛽 𝛼 𝛼

𝑣−1

= 𝑠{𝐹𝑊 (𝑡)}𝑣−1 𝑝𝑊 (𝑡)

𝑡 𝛽 exp �− � � � 𝛼

Where 𝐹𝑊 (𝑡) and 𝑝𝑊 (𝑡) are the cdf and pdf of the two parameter Weibull distribution respectively. t β 𝐹(𝑡) = �1 − exp �− � � �� α = [𝐹𝑊 (𝑡)]𝑣

CDF

t β R(t) = 1 − �1 − exp �− � � �� α = 1 − [𝐹𝑊 (𝑡)]𝑣

Reliability

Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡)

Mean Life

v

𝑚(𝑚) = 𝑅(𝑚|𝑡) =

𝑣

𝑡+𝑚 𝛽 1 − �1 − exp �− � 𝛼 � �� 𝑣

𝑡 𝛽 1 − �1 − exp �− �𝛼 � ��

Where 𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡.

Residual

Hazard Rate

𝑅(𝑡 + x) = 𝑅(𝑡)

v

𝑣

∞ 𝑡 𝛽 ∫t �1 − �1 − exp �− �𝛼 � �� � 𝑝𝑚

𝑠(𝑡) =

ℎ(𝑡) =

𝑣

𝛽−1

𝛽𝑠�𝑡�𝛼 �

𝑡 𝛽 1 − �1 − exp �− �𝛼 � ��

𝛽 �1 − exp �−�𝑡�𝛼 � ��

𝑣−1

𝛽 exp �−�𝑡�𝛼 � �

𝛽 1 − �1 − exp �−�𝑡�𝛼 � ��

𝑣

Exp-Weibull

𝑝(𝑡) =

78

Bathtub Life Distributions

For small t: (Murthy et al. 2003, p.130) 𝛽𝑠 𝑡 𝛽𝑣−1 ℎ(𝑡) ≈ � � � � 𝛼 𝛼 For large t: (Murthy et al. 2003, p.130) 𝛽 𝑡 𝛽−1 ℎ(𝑡) ≈ � � � � 𝛼 𝛼 Properties and Moments

1� −1� 𝑣 �� 𝛽

Median

𝛼 �− ln �1 − 2

For 𝛽𝑠 > 1 the mode can be approximated (Murthy et al. 2003, p.130): 𝑣 1 �𝛽(𝛽 − 8𝑠 + 2𝛽𝑠 + 9𝛽𝑠 2 ) 1 𝛼� � − 1 − �� 2 𝛽𝑠 𝑠

Exp-Weibull

Mode

Mean - 1st Raw Moment

Solved numerically see Murthy et al. 2003, p.128

Variance - 2nd Central Moment 100𝑝% Percentile Function

𝑡𝑝 = 𝛼 �− ln �1 − 𝑝

Parameter Estimation

1� 1� 𝛽 𝑣 ��

Plotting Method (Jiang & Murthy 1999) Plot Points on a Weibull Probability Plot

X-Axis

Y-Axis

𝑚 = 𝑠𝑓(𝑡)

1 𝑦 = 𝑠𝑓 �𝑠𝑓 � �� 1−𝐹

Using the Weibull Probability Plot the parameters can be estimated. (Jiang & Murthy 1999), provide a comprehensive coverage of this. A typical WPP for an exponentiated Weibull distribution is: 3 2.5 2 1.5 1 0.5 0 -0.5 -1 -1.5 -2 -1

-0.5

0

0.5

1

1.5

2

2.5

3

WPP for exponentiated Weibull distribution 𝛼 = 5, 𝛽 = 2, 𝑠 = 0.4

Asymptotes (Jiang & Murthy 1999): As 𝑚 → −∞ (𝑡 → 0) there exists an asymptote approximated by:

Exponentiated Weibull Continuous Distribution

79

𝑦 ≈ 𝛽𝑠[𝑚 − ln(𝛼)]

As 𝑚 → ∞ (𝑡 → ∞) the asymptote straight line can be approximated by: 𝑦 ≈ 𝛽[𝑚 − ln(𝛼)]

Both asymptotes intersect the x-axis at 𝑠𝑓(𝛼) however both have different slopes unless 𝑠 = 1 and the WPP is the same as a two parameter Weibull distribution.

Parameter Estimation Plot estimates of the asymptotes ensuring they cross the x-axis at the same point. Use the right asymptote to estimate 𝛼 and 𝛽. Use the left asymptote to estimate 𝑠.

Bayesian

MLE and Bayesian techniques can be used in the standard way however estimates obtained from the graphical methods are useful for initial guesses when using numerical methods to solve equations. A literature review of MLE and Bayesian methods is covered in (Murthy et al. 2003). Description , Limitations and Uses

Characteristics

PDF Shape: (Murthy et al. 2003, p.129) 𝜷𝒗 <= 1. The pdf is monotonically decreasing, 𝑝(0) = ∞. 𝜷𝒗 = 𝟏. The pdf is monotonically decreasing, 𝑝(0) = 1/𝛼. 𝜷𝒗 > 1. The pdf is unimodal. 𝑝(0) = 0.

The pdf shape is determined by 𝛽𝑠 in a similar way to the 𝛽 for a two parameter Weibull distribution.

Hazard Rate Shape: (Murthy et al. 2003, p.129) 𝜷 ≤ 𝟏 and 𝜷𝒗 ≤ 𝟏. The hazard rate is monotonically decreasing. 𝜷 ≥ 𝟏 and 𝜷𝒗 ≥ 𝟏. The hazard rate is monotonically increasing. 𝜷 < 1 and 𝜷𝒗 > 1. The hazard rate is unimodal. 𝜷 > 1 and 𝜷𝒗 < 1. The hazard rate is a bathtub curve. Weibull Distribution. The Weibull distribution is a special case of the expatiated distribution when 𝑠 = 1. When 𝑠 is an integer greater than 1, then the cdf represents a multiplicative Weibull model.

Standard Exponentiated Weibull. (Xie et al. 2004) When 𝛼 = 1 the distribution is the standard exponentiated Weibull distribution with cdf: 𝑣 𝐹(𝑡) = �1 − exp�−𝑡 𝛽 �� Minimum Failure Rate. (Xie et al. 2004) When the hazard rate is a bathtub curve (𝛽 > 1 and 𝛽𝑠 < 1) then the minimum failure rate point is: 𝑡 ′ = 𝛼[− ln(1 − 𝑦1 )]

1� 𝛽

Exp-Weibull

Maximum Likelihood

80

Bathtub Life Distributions

where 𝑦1 is the solution to: (𝛽 − 1)𝑦(1 − 𝑦 𝑣 ) + 𝛽 ln(1 − 𝑦) [1 + 𝑠𝑦 − 𝑠 − 𝑦 𝑣 ] = 0

Maximum Mean Residual Life. (Xie et al. 2004) By solving the derivative of the MRL function to zero, the maximum MRL is found by solving to t: 𝑡 ∗ = 𝛼[− ln(1 − 𝑦2 )]

Exp-Weibull

where 𝑦2 is the solution to: ∞

�

𝛽𝑠(1 − 𝑦)𝑦 𝑣−1 [− ln(1 − 𝑦)]

1 [− ln(1−𝑦)] �𝛽

Resources

1� 𝛽

𝛽

𝑣

−1�𝛽

[1 − �1 − 𝑅 −𝑥 � 𝑝𝑚 − (1 − 𝑦 𝑣 )2 = 0

Books / Journals: Mudholkar, G. & Srivastava, D., 1993. Exponentiated Weibull family for analyzing bathtub failure-rate data. Reliability, IEEE Transactions on, 42(2), 299-302. Jiang, R. & Murthy, D., 1999. The exponentiated Weibull family: a graphical approach. Reliability, IEEE Transactions on, 48(1), 6872. Xie, M., Goh, T.N. & Tang, Y., 2004. On changing points of mean residual life and failure rate function for some generalized Weibull distributions. Reliability Engineering and System Safety, 84(3), 293–299. Murthy, D., Xie, M. & Jiang, R., 2003. Weibull Models 1st ed., Wiley-Interscience. Rinne, H., 2008. The Weibull Distribution: A Handbook 1st ed., Chapman & Hall/CRC. Balakrishnan, N. & Rao, C.R., 2001. Handbook of Statistics 20: Advances in Reliability 1st ed., Elsevier Science & Technology. Relationship to Other Distributions

Weibull Distribution 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽)

Special Case: 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽) = 𝐸𝑚𝑝𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼 = 𝛼, 𝛽 = 𝛽, 𝑠 = 1)

Modified Weibull Distribution

81

3.3. Modified Weibull Distribution Probability Density Function - f(t)

b=0.5 b=0.8 b=1 b=2

0.5

1

0

0.5

λ=1 λ=5 λ=10

0

0.2

Cumulative Density Function - F(t)

b=0.5 b=0.8 b=1 b=2 0

0.5

1

a=5 a=10 a=20

0

0.5

λ=1 λ=5 λ=10

0

0.2

Hazard Rate - h(t)

0

5

0

5

0

Note: The hazard rate plots are on a different scale to the PDF and CDF

5

Modified Weibull

0

a=5 a=10 a=20

82

Bathtub Life Distributions

Parameters & Description

Parameters

Limits

𝑠

𝑠>0

𝑏

𝑏≥0

𝜆

𝜆≥0

Modified Weibull

Distribution

Shape Parameter: The shape of the distribution is completely determined by b. When 0 < 𝑏 < 1 the distribution has a bathtub shaped hazard rate. Scale Parameter. 𝑡 ≥ 0

Formulas 𝑝(𝑡) = a(b + λt) t b−1 exp(λt) exp�−at b exp(λt)�

PDF

𝐹(𝑡) = 1 − exp[−𝑠𝑡 𝑏 𝑅𝑚𝑝(𝜆𝑡)]

CDF

R(t) = exp[−𝑠𝑡 𝑏 𝑅𝑚𝑝(𝜆𝑡)]

Reliability Mean Life

Scale Parameter.

Residual

∞

𝑠(𝑡) = exp�𝑠𝑡 𝑏 𝑅 𝜆𝑡 � � exp�𝑠𝑚 𝑏 𝑅 𝜆𝑡 � 𝑝𝑚 𝑡

ℎ(𝑡) = a(b + λt)t b−1 eλt

Hazard Rate

Properties and Moments Median

Solved numerically (see 100p%)

Mode

Solved numerically st

Mean - 1 Raw Moment nd

Variance - 2

Solved numerically

Central Moment

Solved numerically Solve for 𝑡𝑝 numerically: ln(1 − 𝑝) 𝑡𝑝b exp(λt p ) = − a

100p% Percentile Function

Parameter Estimation

Plotting Method (Lai et al. 2003) Plot Points on a Weibull Probability Plot

X-Axis

Y-Axis

ln(𝑡𝑖 )

1 𝑠𝑓 �𝑠𝑓 � �� 1−𝐹

Using the Weibull Probability Plot the parameters can be estimated. (Lai et al. 2003).

Asymptotes (Lai et al. 2003):

Modified Weibull Distribution

83

As 𝑚 → −∞ (𝑡 → 0) the asymptote straight line can be approximated as: 𝑦 ≈ 𝑏𝑚 + ln(𝑠)

As 𝑚 → ∞ (𝑡 → ∞) the asymptote straight line can be approximated as (not used for parameter estimate but more for model validity): 𝑦 ≈ 𝜆 exp(𝑚) = 𝜆𝑡 Intersections (Lai et al. 2003): Y-Axis Intersection (0, 𝑚0 )

ln(𝑠) + 𝜆 = 𝑦0

Solving these gives an approximate value for each parameter which can be used as an initial guess for numerical methods solving MLE or Bayesian methods. A typical WPP for an Modified Weibull Distribution is: 10 9 8 7 6 5 4 3 2 1 0 -1 -9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

WPP for modified Weibull distribution 𝑠 = 5, 𝑏 = 0.2, 𝜆 = 10

Description , Limitations and Uses Characteristics

Parameter Characteristics:(Lai et al. 2003) 𝟎 < 𝑏 < 1 𝒂𝒏𝒅 𝝀 > 0. The hazard rate has a bathtub curve shape. ℎ(𝑡) → ∞ 𝑠𝑠 𝑡 → 0. ℎ(𝑡) → ∞ 𝑠𝑠 𝑡 → ∞. 𝒃 ≥ 𝟏 𝒂𝒏𝒅 𝝀 > 0. Has an increasing hazard rate function. ℎ(0) = 0. ℎ(𝑡) → ∞ 𝑠𝑠 𝑡 → ∞ . 𝝀 = 𝟎. The function has the same form as a Weibull Distribution. ℎ(0) = 𝑠𝑏. ℎ(𝑡) → ∞ 𝑠𝑠 𝑡 → ∞

Modified Weibull

ln(𝑠) + 𝑏𝑚0 + 𝜆𝑅 𝑥0 = 0

X-Axis Intersection (𝑦0 , 0)

84

Bathtub Life Distributions

Minimum Failure Rate. (Xie et al. 2004) When the hazard rate is a bathtub curve (0 < 𝑏 < 1 𝑠𝑓𝑝 𝜆 > 0) then the minimum failure rate point is given as: √𝑏 − 𝑏 𝑡∗ = 𝜆

Modified Weibull

Maximum Mean Residual Life. (Xie et al. 2004) By solving the derivative of the MRL function to zero, the maximum MRL is found by solving to t: ∞

𝑠(𝑏 + 𝜆𝑡)𝑡 𝑏−1 𝑅 𝜆𝑡 � exp(−𝑠𝑚 𝑏 𝑅 (𝜆𝑥)𝑑𝑥 − exp�𝑠𝑡 𝑏 𝑅 𝜆𝑡 � = 0 𝑡

Shape. The shape of the hazard rate cannot have a flat “usage period” and a strong “wear out” gradient. Resources

Books / Journals: Lai, C., Xie, M. & Murthy, D., 2003. A modified Weibull distribution. IEEE Transactions on Reliability, 52(1), 33-37. Murthy, D.N.P., Xie, M. & Jiang, R., 2003. Weibull Models 1st ed., Wiley-Interscience. Xie, M., Goh, T.N. & Tang, Y., 2004. On changing points of mean residual life and failure rate function for some generalized Weibull distributions. Reliability Engineering and System Safety, 84(3), 293–299. Rinne, H., 2008. The Weibull Distribution: A Handbook 1st ed., Chapman & Hall/CRC. Balakrishnan, N. & Rao, C.R., 2001. Handbook of Statistics 20: Advances in Reliability 1st ed., Elsevier Science & Technology. Relationship to Other Distributions

Weibull Distribution 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽)

Special Case: 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽) = 𝑀𝑓𝑝𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; a = 𝛼, 𝑏 = 𝛽, 𝜆 = 0)

Univariate Continuous Distributions

85

4. Univariate Continuous Distributions

Univariate Cont

86

Univariate Continuous Distributions

4.1. Beta Continuous Distribution Probability Density Function - f(t) α=5 β=2 α=2 β=2 α=1 β=2 α=0.5 β=2

3.00 2.50

2.50

2.00

2.00

1.50

1.50

1.00

1.00

0.50

0.50

0.00

0.00 0

α=1 β=1 α=5 β=5 α=0.5 β=0.5

0.5

1

0

0.2

0.4

0.6

0.8

1

Beta

Cumulative Density Function - F(t) 1.00

1.00

0.80

0.80

0.60

0.60

0.40

0.40

0.20

0.20

0.00

0.00 0

0.5

Hazard Rate - h(t) 10.00 9.00 8.00 7.00 6.00 5.00 4.00 3.00 2.00 1.00 0.00

1

α=5 β=2 α=2 β=2 α=1 β=2 α=0.5 β=2

α=1 β=1 α=5 β=5 α=0.5 β=0.5 0

0.2

0.4

0.6

0.8

1

4.00 3.50 3.00 2.50 2.00 1.50 1.00

α=1 β=1 α=5 β=5 α=0.5 β=0.5

0.50 0.00 0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Beta Continuous Distribution

87

Parameters & Description 𝛼

𝛽 Parameters

𝑠𝐿 𝑏𝑈

𝛼>0

Shape Parameter.

𝛽>0

Shape Parameter. Lower Bound: 𝑠𝐿 is the lower bound but has also been called a location parameter. In the standard Beta distribution 𝑠𝐿 = 0.

−∞ < 𝑠𝐿 < 𝑏𝑈

Upper Bound: 𝑏𝑈 is the upper bound. In the standard Beta distribution 𝑏𝑈 = 1. The scale parameter may also be defined as 𝑏𝑈 − 𝑠𝐿 .

𝑠𝐿 < 𝑏𝑈 < ∞

𝑠𝐿 < 𝑡 ≤ 𝑏𝑈

Limits Distribution

Formulas

𝐵(𝑚, 𝑦) is the Beta function, 𝐵𝑡 (𝑡|𝑚, 𝑦) is the incomplete Beta function, 𝐼𝑡 (𝑡|𝑚, 𝑦) is the regularized Beta function, Γ(𝑘) is the complete gamma which is discussed in section 1.6. General Form:

When 𝑠𝐿 = 0, 𝑏𝑈 = 1:

PDF

Γ(𝛼 + 𝛽) (𝑡 − 𝑠𝐿 )𝛼−1 (𝑏𝑈 − 1)𝛽−1 . (𝑏𝑈 − 𝑠𝐿 )𝛼+𝛽−1 Γ(𝛼)Γ(𝛽)

Γ(𝛼 + 𝛽) 𝛼−1 (1 − 𝑡)𝛽−1 .𝑡 Γ(𝛼)Γ(𝛽) 1 = . 𝑡 𝛼−1 (1 − 𝑡)𝛽−1 𝐵(𝛼, 𝛽)

𝑝(𝑡|𝛼, 𝛽) =

Γ(𝛼 + 𝛽) 𝑡 𝛼−1 (1 − 𝑠)𝛽−1 𝑝𝑠 �𝑠 Γ(𝛼)Γ(𝛽) 0 𝐵𝑡 (𝑡|𝛼, 𝛽) = 𝐵(𝛼, 𝛽) = 𝐼𝑡 (𝑡|𝛼, 𝛽)

F(t) =

CDF

R(t) = 1 − 𝐼𝑡 (𝑡|𝛼, 𝛽)

Reliability

Conditional Survivor Function

Mean Life

Residual

𝑚(𝑚) = 𝑅(𝑚|𝑡) =

𝑅(𝑡 + x) 1 − 𝐼𝑡 (𝑡 + 𝑚|𝛼, 𝛽) = 𝑅(𝑡) 1 − 𝐼𝑡 (𝑡|𝛼, 𝛽)

Where 𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡. ∞

∫t {𝐵(α, β) − 𝐵x (x|α, β)}dx 𝐵(α, β) − 𝐵t (t|α, β) (Gupta and Nadarajah 2004, p.44) 𝑠(𝑡) =

Beta

𝑝(𝑡; 𝛼, 𝛽, 𝑠𝐿 , 𝑏𝑈 ) =

88

Univariate Continuous Distributions

Hazard Rate

t α−1 (1 − t) 𝐵(α, β) − 𝐵t (t|α, β) (Gupta and Nadarajah 2004, p.44) ℎ(𝑡) =

Properties and Moments

Numerically solve for t: 𝑡0.5 = 𝐹 −1 (𝛼, 𝛽)

Median

𝛼−1

Mode

𝛼+𝛽−2

Mean - 1st Raw Moment

for 𝛼 > 1 𝑠𝑓𝑝 𝛽 > 1 α α+β

αβ (α + β)2 (α + β + 1)

Variance - 2nd Central Moment

2(β − α)�α + β + 1

Skewness - 3rd Central Moment 3

6[𝛼 + 𝛼

Beta

Excess kurtosis - 4th Central Moment

(α + β + 2)�αβ

2 (1

− 2𝛽) + 𝛽 2 (1 + 𝛽) − 2𝛼𝛽(2 + 𝛽)] 𝛼𝛽(𝛼 + 𝛽 + 2)(𝛼 + 𝛽 + 3) 1F1 (α; α

+ β; it)

Where 1F1 is the confluent hypergeometric function defined as: ∞ (α)k 𝑚 𝐸 (α; F β; x) = � . 1 1 (β)k 𝑘!

Characteristic Function

k=0

(Gupta and Nadarajah 2004, p.44) Numerically solve for t: 𝑡𝑝 = 𝐹 −1 (𝛼, 𝛽)

100p% Percentile Function

Parameter Estimation

Maximum Likelihood Function Likelihood Functions

Log-Likelihood Functions

∂Λ =0 ∂α

L(α, β|E) =

nF Γ(𝛼 + 𝛽)nF 𝛽−1 � 𝑡𝑖𝐹 𝛼−1 �1 − 𝑡𝑖𝐹 � . Γ(𝛼)Γ(𝛽) ������������������������� i=1 failures

Λ(α, β|E) = nF {ln[𝛤(𝛼 + 𝛽) − 𝑠𝑓[𝛤(𝛼)] − 𝑠𝑓[𝛤(𝛽)]} +(α −

𝑑

nF

1) � ln�t Fi � i=1

nF

+ (β − 1) � ln(1 − t Fi )

𝜓(α) − 𝜓(α + β) =

nF

i=1

1 � ln(t Fi ) nF i=1

where 𝜓(𝑚) = 𝑑𝑥 ln[Γ(𝑚)] is the digamma function see section 1.6.7.

Beta Continuous Distribution

89

(Johnson et al. 1995, p.223) ∂Λ =0 ∂β

Point Estimates

Fisher Information Matrix

nF

1 𝜓(β) − 𝜓(α + β) = � ln(1 − t i ) nF i=1

(Johnson et al. 1995, p.223)

Point estimates are obtained by using numerical methods to solve the simultaneous equations above. 𝐼(𝛼, 𝛽) = � 𝑑2

𝜓 ′ (𝛼) − 𝜓 ′ (𝛼 + 𝛽) −𝜓 ′ (𝛼 + 𝛽) � ′ (𝛼 ′ (𝛽) −𝜓 + 𝛽) 𝜓 − 𝜓 ′ (𝛼 + 𝛽)

−2 where 𝜓 ′ (𝑚) = 𝑑𝑥 2 𝑠𝑓Γ(𝑚) = ∑∞ is the Trigamma function. 𝑖=0(𝑚 + 𝑠) See section 1.6.8. (Yang and Berger 1998, p.5)

Confidence Intervals

For a large number of samples the Fisher information matrix can be used to estimate confidence intervals. See section 1.4.7. Bayesian Non-informative Priors √det�𝐼(𝛼, 𝛽)� where 𝐼(𝛼, 𝛽) is given above.

Beta

Jeffery’s Prior

Conjugate Priors

UOI 𝑝 from 𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(𝑘; 𝑝) 𝑝 from 𝐵𝑠𝑓𝑓𝑚(𝑘; 𝑝, 𝑓)

Example

Likelihood Model Bernoulli

Binomial

Evidence 𝑘 failures in 1 trail 𝑘 failures in 𝑓 trials

Dist of UOI

Prior Para

Posterior Parameters

Beta

𝛼0 , 𝛽0

𝛼 = 𝛼𝑜 + 𝑘 𝛽 = 𝛽𝑜 + 1 − 𝑘

Beta

𝛼𝑜 , 𝛽𝑜

Description , Limitations and Uses

𝛼 = 𝛼𝑜 + 𝑘 β = βo + n − k

For examples on the use of the beta distribution as a conjugate prior see the binomial distribution. A non-homogeneous (operate in different environments) population of 5 switches have the following probabilities of failure on demand. 0.1176, 0.1488, 0.3684, 0.8123, 0.9783 Estimate the population variability function: nF

1 � ln(t Fi ) = −1.0549 nF i=1

90

Univariate Continuous Distributions

nF

1 � ln(1 − t i ) = −1.25 nF i=1

Numerically Solving: 𝜓(α) + 1.0549 = 𝜓(β) + 1.25 Gives: 𝛼� = 0.7369 𝑏� = 0.6678 1.5924 −1.0207 � 𝐼(𝛼, 𝛽) = � −1.0207 2.0347 −1 −1 0.1851 �𝐽𝑛 �𝛼�, 𝛽̂ �� = �𝑓𝐹 𝐼�𝛼�, 𝛽̂ �� = � 0.0929

0.0929 � 0.1449

Beta

90% confidence interval for 𝛼: 𝛷 −1 (0.95)√0.1851 𝛷 −1 (0.95)√0.1851 �, 𝛼�. exp � �� �𝛼�. exp � −𝛼� 𝛼� [0.282, 1.92]

Characteristics

90% confidence interval for 𝛽: 𝛷 −1 (0.95)√0.1449 𝛷 −1 (0.95)√0.1449 �, 𝛽̂ . exp � �� �𝛽̂ . exp � ̂ −𝛽 𝛽̂ [0.262, 1.71]

The Beta distribution was originally known as a Pearson Type I distribution (and Type II distribution which is a special case of a Type I). 𝐵𝑅𝑡𝑠(𝛼, 𝛽) is the mirror distribution of 𝐵𝑅𝑡𝑠(𝛽, 𝛼). If 𝑋~𝐵𝑅𝑡𝑠(𝛼, 𝛽) and let 𝑌 = 1 − 𝑋 then 𝑌~𝐵𝑅𝑡𝑠(𝛽, 𝛼). Location / Scale Parameters (NIST Section 1.3.6.6.17) 𝑠𝐿 and 𝑏𝑈 can be transformed into a location and scale parameter: 𝑠𝑓𝑐𝑠𝑡𝑠𝑓𝑓 = 𝑠𝐿 𝑠𝑐𝑠𝑠𝑅 = 𝑏𝑈 − 𝑠𝐿

Shapes(Gupta and Nadarajah 2004, p.41): 𝟎 < 𝛼 < 1. As 𝑚 → 0, 𝑝(𝑚) → ∞. 𝟎 < 𝛽 < 1. As 𝑚 → 1, 𝑝(𝑚) → ∞. 𝜶 > 1, 𝜷 > 1. As 𝑚 → 0, 𝑝(𝑚) → 0. There is a single mode 𝛼−1 at . 𝛼+𝛽−2

𝜶 < 1, 𝜷 < 1. The distribution is a U shape. There is a 𝛼−1 single anti-mode at 𝛼+𝛽−2. 𝜶 > 0, 𝜷 > 0. There exists inflection points at:

(𝛼 − 1)(𝛽 − 1) 𝛼−1 1 ± .� 𝛼+𝛽−2 𝛼+𝛽−2 𝛼+𝛽−3

Beta Continuous Distribution

91

𝜶 = 𝜷. The distribution is symmetrical about 𝑚 = 0.5. As 𝛼 = 𝛽 becomes large, the beta distribution approaches the normal distribution. The Standard Uniform Distribution arises when 𝛼 = 𝛽 = 1. 𝜶 = 𝟏, 𝜷 = 𝟐 or 𝜶 = 𝟐, 𝜷 = 𝟏. Straight line. (𝜶– 𝟏)(𝜷– 𝟏) < 0. J Shaped.

Hazard Rate and MRL (Gupta and Nadarajah 2004, p.45): 𝜶 ≥ 𝟏, 𝜷 ≥ 𝟏. ℎ(𝑡) is increasing. 𝑠(𝑡) is decreasing. 𝜶 ≤ 𝟏, 𝜷 ≤ 𝟏. ℎ(𝑡) is decreasing. 𝑠(𝑡) is increasing. 𝜶 > 1, 0 < 𝛽 < 1. ℎ(𝑡) is bathtub shaped and 𝑠(𝑡) is an upside down bathtub shape. 𝟎 < 𝛼 < 1, 𝜷 > 1. ℎ(𝑡) is upside down bathtub shaped and 𝑠(𝑡) is bathtub shape.

Parameter Model. The Beta distribution is often used to model parameters which are constrained to take place between an interval. In particular the distribution of a probability parameter 0 ≤ 𝑝 ≤ 1 is popular with the Beta distribution.

Proportions. Used to model proportions. An example of this is the likelihood ratios for estimating uncertainty. Online: http://mathworld.wolfram.com/BetaDistribution.html http://en.wikipedia.org/wiki/Beta_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (interactive web calculator) http://www.itl.nist.gov/div898/handbook/eda/section3/eda366h.htm

Resources

Books: Gupta, A.K. & Nadarajah, S., 2004. Handbook of beta distribution and its applications, CRC Press. Johnson, N.L., Kotz, S. & Balakrishnan, N., 1995. Continuous Univariate Distributions, Vol. 2 2nd ed., Wiley-Interscience. Relationship to Other Distributions

Beta

Applications

Bayesian Analysis. The Beta distribution is often used as a conjugate prior in Bayesian analysis for the Bernoulli, Binomial and Geometric Distributions to produce closed form posteriors. The Beta(0,0) distribution is an improper prior sometimes used to represent ignorance of parameter values. The Beta(1,1) is a standard uniform distribution which may be used as a noninformative prior. When used as a conjugate prior to a Bernoulli or Binomial process the parameter 𝛼 may represent the number of successes and 𝛽 the total number of failures with the total number of trials being 𝑓 = 𝛼 + 𝛽.

92

Univariate Continuous Distributions

Let Chi-square Distribution

Xi ~𝜒 2 (𝑠𝑖 )

Then

2

𝜒 (𝑡; 𝑠)

𝑠𝑓𝑝

1

Y= 1

X1 X1 + X 2

Y~𝐵𝑅𝑡𝑠 �𝛼 = 2𝑠1 , 𝛽 = 2𝑠2 �

Let Uniform Distribution 𝑈𝑓𝑠𝑝(𝑡; 𝑠, 𝑏)

Then

Xi ~𝑈𝑓𝑠𝑝(0,1)

𝑠𝑓𝑝

X1 ≤ X 2 ≤ ⋯ ≤ X n

Xr ~𝐵𝑅𝑡𝑠(𝑠, 𝑓 − 𝑠 + 1) Where 𝑓 and 𝑠 are integers. Special Case:

𝐵𝑅𝑡𝑠(𝑡; 1,1, 𝑠, 𝑏) = 𝑈𝑓𝑠𝑝(𝑡; 𝑠, 𝑏)

For large 𝛼 and 𝛽 with fixed 𝛼/𝛽 : R

Beta

Normal Distribution 𝑁𝑓𝑠𝑚(𝑡; 𝜇, 𝜎) Gamma Distribution 𝐺𝑠𝑚𝑚𝑠(𝑡; k, λ)

Dirichlet Distribution

𝐷𝑠𝑠𝑑 (𝑚; 𝜶)

𝐵𝑅𝑡𝑠(𝛼, 𝛽) ≈ 𝑁𝑓𝑠𝑚 �µ =

α αβ ,𝜎 = � � (α + β)2 (α + β + 1) α+β

As 𝛼 and 𝛽 increase the mean remains constant and the variance is reduced. Let

Then

𝑋1 , 𝑋2 ~𝐺𝑠𝑚𝑚𝑠(k i , λi )

𝑠𝑓𝑝

Y~𝐵𝑅𝑡𝑠(𝛼 = 𝑘1 , 𝛽 = 𝑘2 )

Y=

X1 X1 + X 2

Special Case: 𝐷𝑠𝑠𝑑=1 (x; [α1 , α0 ]) = 𝐵𝑅𝑡𝑠(𝑘 = 𝑚; 𝛼 = 𝛼1 , 𝛽 = 𝛼0 )

Birnbaum Sanders Continuous Distribution

93

4.2. Birnbaum Saunders Continuous Distribution Probability Density Function - f(t) α=0.2 β=1 α=0.5 β=1 α=1 β=1 α=2.5 β=1

2.0 1.5

α=1 β=0.5 α=1 β=1 α=1 β=2 α=1 β=4

1.4 1.2 1.0 0.8

1.0

0.6 0.4

0.5

0.2 0.0

0.0 0

1

2

3

0

1

2

3

0

1

2

3

0

1

2

3

1.0

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

0.8 0.6 0.4

α=0.2 β=1 α=0.5 β=1 α=1 β=1 α=2.5 β=1

0.2 0.0 0

1

2

Birnbaum Sanders

Cumulative Density Function - F(t)

3

Hazard Rate - h(t) 2.5

1.6

2.0

1.4 1.2

1.5

1.0 0.8

1.0

0.6

0.5

0.4

0.0

0.0

0.2 0

1

2

3

94

Univariate Continuous Distributions

Parameters & Description

Parameters

β

𝛼

Limits

Scale parameter. β is the scale parameter equal to the median.

β>0

α>0

Shape parameter. 0
Distribution

Formulas

𝑝(𝑡) = =

Birnbaum Sanders

PDF

�t⁄β + �β⁄t 2𝛼𝑡√2𝜋

2

1 �t⁄β − �β⁄t exp � − � � � 2 𝛼

�t⁄β + �β⁄t ϕ(z) 2𝛼𝑡

where 𝜙(𝑧) is the standard normal pdf and: �t⁄β − �β⁄t 𝑧𝐵𝑆 = 𝛼

�t⁄β − �β⁄t 𝐹(𝑡) = Φ � � 𝛼 = Φ(𝑧𝐵𝑆 )

CDF

�β⁄t − �t⁄β R(t) = Φ � � 𝛼 = Φ(−𝑧𝐵𝑆 )

Reliability

Where Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡)

Mean Life

Residual

Hazard Rate Cumulative Hazard Rate

𝑧𝐵𝑆 =

𝑚(𝑚) = 𝑅(𝑚|𝑡) =

�t⁄β − �β⁄t , 𝛼

′ ) 𝑅(𝑡 + 𝑚) Φ(−𝑧𝐵𝑆 = 𝑅(𝑡) Φ(−𝑧𝐵𝑆 )

′ 𝑧𝐵𝑆 =

�(t + x)⁄β − �β⁄(t + x) 𝛼

𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡. ∞

∫ Φ(−𝑧𝐵𝑆 )𝑝𝑚 𝑠(𝑡) = 𝑡 Φ(−𝑧𝐵𝑆 )

ℎ(𝑡) =

�t⁄β + �β⁄t ϕ(zBS ) � � 2𝛼𝑡 Φ(−𝑧𝐵𝑆 )

𝐻(𝑡) = − ln[Φ(−𝑧𝐵𝑆 )]

Birnbaum Sanders Continuous Distribution

95

Properties and Moments β

Median

Numerically solve for 𝑡: 𝑡 3 + 𝛽(1 + 𝛼 2 )𝑡 2 + 𝛽 2 (3𝛼 2 − 1)𝑡 − 𝛽 3 = 0

Mode Mean - 1st Raw Moment

β �1 +

Variance - 2nd Central Moment

α2 � 2

α2 β2 �1 +

5α2 � 4

4α(11α2 + 6)

Skewness - 3rd Central Moment

3

(5α2 + 4)2

(Lemonte et al. 2007) Excess kurtosis - 4th Central Moment

6α2 (93α2 + 40) (5α2 + 4)2

(Lemonte et al. 2007) 2 β 𝑡𝛾 = �αΦ−1 (γ) + �4 + [αΦ−1 (γ)]2 � 4

100𝛾 % Percentile Function

Parameter Estimation

Maximum Likelihood Function Likelihood Function

Log-Likelihood Function

∂Λ =0 ∂α ∂Λ =0 ∂β MLE

Point

For complete data:

nF

2

1 �t i ⁄β − �β⁄t i �t i ⁄β + �β⁄t i exp � − � � � 𝐿(𝜃, 𝛼|𝐸) = � 2 𝛼 2𝛼𝑡𝑖 √2𝜋 i=1 ��������������������������������� failures

1 2

𝑛𝐹

3

𝑛𝐹

𝛽 𝛽 2 1 𝑡𝑖 𝛽 Λ(𝛼, 𝛽|𝐸) = −nF ln(αβ) + � ln �� � + � � � − 2 � � + − 2� 𝑡𝑖 𝑡𝑖 2𝛼 𝛽 𝑡𝑖 𝑖=1 𝑖=1 ��������������������������������������� failures 𝑛𝐹

𝑛𝐹

∂Λ nF 2 1 𝛽 1 = − �1 + 2 � + 3 � 𝑡𝑖 + 3 � = 0 ∂α α α α β 𝛼 𝑡𝑖 ������������������������� 𝑖=1 𝑖=1 𝑛𝐹

failures

𝑛𝐹

𝑛𝐹

∂Λ nF 1 1 1 1 =− +� + � 𝑡𝑖 − 2 � = 0 ∂β 2β 𝑡𝑖 + 𝛽 2𝛼 2 𝛽 2 2𝛼 𝑡𝑖 ����������������������������� 𝑖=1 𝑖=1 𝑖=1

𝛽̂ is found by solving:

failures

Birnbaum Sanders

3+

96

Univariate Continuous Distributions

Estimates where 𝑛𝐹

𝛽 2 − 𝛽[2𝑅 + 𝑅(𝛽)] + 𝑅[𝑆 + 𝑅(𝛽)] = 0 −1

1 1 𝑅(𝛽) = � � � 𝑓 𝛽 + 𝑡𝑖 𝑖=1

Point estimates for 𝛼� is: (Lemonte et al. 2007)

where Birnbaum Sanders

,

𝑖=1

𝑛𝐹

−1

1 1 𝑅=� � � 𝑓𝐹 𝑡𝑖 𝑖=1

𝑆 𝛽̂ 𝛼� = � + − 2 𝛽̂ 𝑅

2 ⎡ 2 𝛼 𝐼(𝜃, 𝛼) = ⎢ ⎢0 ⎣

Fisher Information

100𝛾% Confidence Intervals

𝑛𝐹

1 𝑆= � 𝑡𝑖 , 𝑓𝐹

⎤ ⎥ 𝛼(2𝜋)−1⁄2 𝑘(𝛼) + 1⎥ ⎦ 𝛼 2𝛽2 0

𝜋 2 2 𝑘(𝛼) = 𝛼� − 𝜋 exp � 2 � �1 − Φ � �� 2 𝛼 𝛼 (Lemonte et al. 2007)

Calculated from the Fisher information matrix. See section 1.4.7. For a literature review of proposed confidence intervals see (Lemonte et al. 2007). Description , Limitations and Uses

Example

5 components are put on a test with the following failure times: 98, 116, 2485, 2526, , 2920 hours 𝑛𝐹

1 � 𝑡𝑖 = 1629 𝑆= 𝑓𝐹

Solving:

𝑅=� 𝑛𝐹

𝑖=1 −1 𝑛𝐹

1 1 � � 𝑡𝑖 𝑓𝐹 𝑖=1

= 250.432

−1

𝑛𝐹

−1

1 1 1 1 𝛽 − 𝛽 �2𝑅 + � � � � + 𝑅 �𝑆 + � � � �=0 𝑓 𝛽 + 𝑡𝑖 𝑓 𝛽 + 𝑡𝑖 2

𝑖=1

β� = 601.949

𝑖=1

𝑆 𝛽̂ 𝛼� = � + − 2 = 1.763 𝛽̂ 𝑅

Birnbaum Sanders Continuous Distribution

90% confidence interval for 𝛼: 2 ⎡ ⎧𝛷 −1 (0.95)� 𝛼 ⎫ 2𝑓 𝐹 ⎢𝛼�. exp , −𝛼� ⎢ ⎨ ⎬ ⎣ ⎩ ⎭ [1.048,

97

2

⎧𝛷−1 (0.95)� 𝛼 ⎫⎤ 2𝑓𝐹 ⎥ 𝛼�. exp 𝛼� ⎨ ⎬⎥ ⎩ ⎭⎦ 2.966]

90% confidence interval for 𝛽: 𝜋 2 2 𝑘(𝛼�) = 𝛼�� − 𝜋 exp � 2 � �1 − Φ � �� = 1.442 2 𝛼� 𝛼� 𝛼�(2𝜋)−1⁄2 𝑘(𝛼�) + 1 = 10.335𝐸-6 𝐼𝛽𝛽 = 𝛼� 2 𝛽̂ 2

Note that this confidence interval uses the assumption of the parameters being normally distributed which is only true for large sample sizes. Therefore these confidence intervals may be inaccurate. Bayesian methods must be done numerically. Characteristics

The Birnbaum-Saunders distribution is a stochastic model of the Miner’s rule. Characteristic of 𝜶. As 𝛼 decreases the distribution becomes more symmetrical around the value of 𝛽.

Hazard Rate. The hazard rate is always unimodal. The hazard rate has the following asymptotes: (Meeker & Escobar 1998, p.107) ℎ(0) = 0 1 lim ℎ(𝑡) = 𝑡→∞ 2𝛽𝛼 2 The change point of the unimodal hazard rate for 𝛼 < 0.6 must be solved numerically, however for 𝛼 > 0.6 can be approximated using: (Kundu et al. 2008) 𝛽 𝑡𝑐 = (−0.4604 + 1.8417𝛼)2

Lognormal and Inverse Gaussian Distribution. The shape and behavior of the Birnbaum-Saunders distribution is similar to that of the lognormal and inverse Gaussian distribution. This similarity is seen primarily in the center of the distributions. (Meeker & Escobar 1998, p.107) Let: 𝑇~𝐵𝑆(𝑡; 𝛼, 𝛽)

Birnbaum Sanders

96762 96762 ⎤ ⎡ ⎧𝛷 −1 (0.95)� ⎫ ⎧𝛷−1 (0.95)� ⎫ 𝑓 𝑓𝐹 𝐹 ⎢𝛽̂ . exp ⎥ , 𝛽̂ . exp −𝛽̂ −𝛽̂ ⎢ ⎨ ⎬ ⎨ ⎬⎥ ⎣ ⎩ ⎭ ⎩ ⎭⎦ [100.4, 624.5]

98

Univariate Continuous Distributions

Scaling property (Meeker & Escobar 1998, p.107) 𝑐𝑇~𝐵𝑆(𝑡; 𝛼, 𝑐𝛽) where 𝑐 > 0

Birnbaum Sanders

Inverse property (Meeker & Escobar 1998, p.107) 1 1 ~𝐵𝑆 �𝑡; 𝛼, � 𝑇 𝛽

Applications

Fatigue-Fracture. The distribution has been designed to model crack growth to critical crack size. The model uses the Miner’s rule which allows for non-constant fatigue cycles through accumulated damage. The assumption is that the crack growth during any one cycle is independent of the growth during any other cycle. The growth for each cycle has the same distribution from cycle to cycle. This is different from the proportional degradation model used to derive the log normal distribution model, with the rate of degradation being dependent on accumulated damage. (http://www.itl.nist.gov/div898/handbook/apr/section1/apr166.htm)

Resources

Online: http://www.itl.nist.gov/div898/handbook/eda/section3/eda366a.htm http://www.itl.nist.gov/div898/handbook/apr/section1/apr166.htm http://en.wikipedia.org/wiki/Birnbaum%E2%80%93Saunders_distrib ution Books: Birnbaum, Z.W. & Saunders, S.C., 1969. A New Family of Life Distributions. Journal of Applied Probability, 6(2), 319-327. Lemonte, A.J., Cribari-Neto, F. & Vasconcellos, K.L., 2007. Improved statistical inference for the two-parameter BirnbaumSaunders distribution. Computational Statistics & Data Analysis, 51(9), 4656-4681. Johnson, N.L., Kotz, S. & Balakrishnan, N., 1995. Continuous Univariate Distributions, Vol. 2, 2nd ed., Wiley-Interscience. Rausand, M. & Høyland, A., 2004. System reliability theory, WileyIEEE.

Gamma Continuous Distribution

99

4.3. Gamma Continuous Distribution Probability Density Function - f(t) 0.30

k=0.5 λ=1 k=3 λ=1 k=6 λ=1 k=12 λ=1

0.25 0.20 0.15 0.10 0.05 0.00 0

10

0.50 0.45 0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00

20

k=3 λ=1.8 k=3 λ=1.2 k=3 λ=0.8 k=3 λ=0.4

0

2

4

6

8

0

2

4

6

8

0

2

4

6

8

Cumulative Density Function - F(t) 1.00

0.80

0.80

0.60

0.60

0.40

Gamma

1.00

0.40 k=0.5 λ=1 k=3 λ=1 k=6 λ=1 k=12 λ=1

0.20 0.00 0

10

0.20 0.00

20

Hazard Rate - h(t) 1.60

1.60

1.40

1.40

1.20

1.20

1.00

1.00

0.80

0.80

0.60

0.60

0.40

0.40

0.20

0.20

0.00

0.00 0

10

20

100

Univariate Continuous Distributions

Parameters & Description

𝜆

𝜆 >0

Parameters 𝑘

𝑘 >0

Limits Distribution

When k is an integer (Erlang distribution)

Scale Parameter: Equal to the rate (frequency) of events/shocks. Sometimes defined as 1/𝜃 where 𝜃 is the average time between events/shocks. Shape Parameter: As an integer 𝑘 can be interpreted as the number of events/shocks until failure. When not restricted to an integer, 𝑘 and be interpreted as a measure of the ability to resist shocks. 𝑡 ≥ 0

When k is continuous

Gamma

Γ(𝑘) is the complete gamma function. Γ(𝑘, 𝑡) and 𝛾(𝑘, 𝑡) are the incomplete gamma functions see section 1.6. 𝑝(𝑡) =

PDF

𝜆𝐸 𝑡 𝐸−1 −λt e (k − 1)!

𝐸−1

(𝜆𝑡)𝑛 𝐹(𝑡) = 1 − 𝑅 −𝜆𝑡 � 𝑓!

CDF

𝑛=0

𝐸−1

(𝜆𝑡)𝑛 𝑅(𝑡) = 𝑅 −𝜆𝑡 � 𝑓!

Reliability

𝑛=0

Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡) Mean Life

Residual

𝑅 −𝜆𝑥

[𝜆(𝑡 + 𝑚)]𝑛 𝑓! (𝜆𝑡)𝑛 ∑𝐸−1 𝑛=0 𝑓!

∑𝐸−1 𝑛=0

𝑝(𝑡) =

𝜆𝐸 𝑡 𝐸−1 −λt e Γ(𝑘)

with Laplace transformation: 𝜆 k 𝑝(𝑠) = � � 𝜆+𝑠 𝐹(𝑡) =

𝛾(𝑘, 𝜆𝑡) 𝛤(𝑘)

𝑅(𝑡) =

𝛤(𝑘, 𝜆𝑡) 𝛤(𝑘)

=

𝜆𝑡 1 � 𝑚 𝐸−1 𝑅 −𝑥 𝑝𝑚 𝛤(𝑘) 0

=

∞ 1 � 𝑚 𝐸−1 𝑅 −𝑥 𝑝𝑚 𝛤(𝑘) 𝜆𝑡

𝑚(𝑚) =

𝑅(𝑡 + 𝑚) 𝛤(𝑘, 𝜆𝑡 + 𝜆𝑚)) = 𝑅(𝑡) 𝛤(𝑘, 𝜆𝑡)

Where 𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡. 𝑠(𝑡) =

∞

∫𝑡 𝑅(𝑚)𝑝𝑚 𝑅(𝑡)

𝑠(𝑡) =

∞

∫𝑡 𝛤(𝑘, 𝜆𝑚)𝑝𝑚 𝛤(𝑘, 𝜆𝑡)

The mean residual life does not have a closed form but has the

Gamma Continuous Distribution

101

expansion:

𝑘 − 1 (𝑘 − 1)(𝑘 − 2) + + 𝑂(𝑡 −3 ) 𝑡 𝑡2 Where 𝑂(𝑡 −3 ) is Landau's notation. (Kleiber & Kotz 2003, p.161) 𝑠(𝑡) = 1 +

ℎ(𝑡) =

𝜆𝐸 𝑡 𝐸−1 (𝜆𝑡)𝑛 𝛤(𝑘) ∑𝐸−1 𝑛=0 𝑓!

ℎ(𝑡) =

𝜆𝐸 𝑡 𝐸−1 −𝜆𝑡 𝑅 𝛤(𝑘, 𝜆𝑡)

Series expansion of the hazard rate is: (Kleiber & Kotz 2003, p.161) −1 (𝑘 − 1)(𝑘 − 2) −3 )� + 𝑂(𝑡 ℎ(𝑡) = � 𝑡2 Limits of ℎ(𝑡) (Rausand & Høyland 2004)

Hazard Rate

lim ℎ(𝑡) = ∞ 𝑠𝑓𝑝 lim ℎ(𝑡) = 𝜆 𝑤ℎ𝑅𝑓 0 < 𝑘 < 1 𝑡→0

𝑡→∞

lim ℎ(𝑡) = 0 𝑠𝑓𝑝 lim ℎ(𝑡) = 𝜆 𝑤ℎ𝑅𝑓 𝑘 ≥ 1 𝑡→0

𝑡→∞

𝐻(𝑡) = 𝜆𝑡 − 𝑠𝑓 ��

𝑛=0

(𝜆𝑡)𝑛 � 𝑓!

𝐻(𝑡) = − 𝑠𝑓 �

Properties and Moments

𝛤(𝑘, 𝜆𝑡) � 𝛤(𝑘)

Numerically solve for t when: 𝑡0.5 = 𝐹 −1 (0.5; 𝑘, 𝜆) or γ(k, λt) = Γ(k, λt)

Median

where γ(k, λt) is the lower incomplete gamma function, see section 1.6.6. 𝑘−1 𝑝𝑓𝑠 𝑘 ≥ 1 𝜆

Mode

Mean - 1st Raw Moment Variance - 2nd Central Moment Skewness - 3rd Central Moment th

Excess kurtosis - 4 Central Moment Characteristic Function

𝑁𝑓 𝑚𝑓𝑝𝑅 𝑝𝑓𝑠 0 < 𝑘 < 1 k 𝜆

k 𝜆2

2/√𝑘 6/𝑘

it −k �1 − � λ

Gamma

𝐸−1

Cumulative Hazard Rate

102

Univariate Continuous Distributions

Numerically solve for t: 𝑡𝛼 = 𝐹 −1 (𝛼; 𝑘, 𝜆)

100α% Percentile Function

Parameter Estimation Maximum Likelihood Function

nF

𝜆𝐸𝑛𝐹 𝐿(𝑘, 𝜆|𝐸) = � 𝑡𝑖 𝐸−1 e−λti Γ(𝑘)𝑛𝐹 ������i=1 ���������

Likelihood Functions

failures

Log-Likelihood Functions

Gamma

∂Λ =0 ∂k ∂Λ =0 ∂λ

Point Estimates

nF

nF

i=1

i=1

Λ(𝑘, 𝜆|𝐸) = 𝑘𝑓𝐹 ln(𝜆) − 𝑓𝐹 ln�𝛤(𝑘)� + (k − 1) � ln(t i ) − λ � t i nF

0 = 𝑓𝐹 ln(λ) − 𝑓𝐹 𝜓(𝑘) + �{ln(t i )} i=1

𝑑

where 𝜓(𝑚) = 𝑑𝑥 ln[Γ(𝑚)] is the digamma function see section 1.6.7. 0=

nF

𝑘𝑓𝐹 − � ti λ i=1

Point estimates for 𝑘� and 𝜆̂ are obtained by using numerical methods to solve the simultaneous equations above. (Kleiber & Kotz 2003, p.165)

Fisher Information Matrix

𝜓 ′ (𝑘) 𝜆 𝐼(𝑘, 𝜆) = � � 𝜆 𝑘𝜆2

𝑑2

−2 where 𝜓 ′ (𝑚) = 𝑑𝑥 2 𝑠𝑓Γ(𝑚) = ∑∞ is the Trigamma function. 𝑖=0(𝑚 + 𝑠) (Yang and Berger 1998, p.10)

Confidence Intervals

For a large number of samples the Fisher information matrix can be used to estimate confidence intervals. Bayesian

Type Uniform Improper Prior with limits: 𝜆 ∈ (0, ∞) 𝑘 ∈ (0, ∞) Jeffrey’s Prior Reference Order: {𝑘, 𝜆}

Prior

Non-informative Priors, 𝝅(𝒌, 𝝀) (Yang and Berger 1998, p.6)

Posterior

1

No Closed Form

𝜆�𝑘. 𝜓 ′ (𝑘) − 1

No Closed Form

𝜆�𝑘. 𝜓 ′ (𝑘) −

1 𝛼

No Closed Form

Gamma Continuous Distribution

Reference Order: {𝜆, 𝑘}

𝜆�𝜓 ′ (𝑘)

103

No Closed Form

𝑑2

−2 where 𝜓 ′ (𝑚) = 𝑑𝑥 2 𝑠𝑓Γ(𝑚) = ∑∞ is the Trigamma function 𝑖=0(𝑚 + 𝑠)

Conjugate Priors

UOI Λ from 𝐸𝑚𝑝(𝑡; Λ)

Λ from 𝑃𝑓𝑠𝑠(𝑘; Λ𝑡)

Exponential

Poisson

Weibull with known 𝛽

𝜎2 Normal with from known 𝜇 𝑁𝑓𝑠𝑚(𝑚; 𝜇, 𝜎 2 ) 𝜆 from

Gamma with known 𝐺𝑠𝑚𝑚𝑠(𝑚; 𝜆, 𝑘) 𝑘 = 𝑘𝐵 𝛼 from 𝑃𝑅𝑠𝑠𝑡𝑓(𝑡; 𝜃, 𝛼)

Pareto with known 𝜃

where:

Evidence

𝑓𝐹 failures in 𝑡𝑇 𝑓𝐹 failures in 𝑡𝑇 𝑓𝐹 failures at times 𝑡𝑖 𝑓𝐹 failures at times 𝑡𝑖

𝑓𝐹 failures in 𝑡𝑇 𝑓𝐹 failures at times 𝑡𝑖

Dist of UOI

Prior Para

Posterior Parameters

Gamma

𝑘0 , 𝜆 0

𝑘 = 𝑘𝑜 + 𝑓𝐹 𝜆 = 𝜆𝑜 + 𝑡 𝑇

Gamma

𝑘0 , 𝜆 0

Gamma

Gamma

Gamma

Gamma

𝑘 = 𝑘𝑜 + 𝑓𝐹 𝜆 = 𝜆𝑜 + 𝑡 𝑇 𝑘 = 𝑘𝑜 + 𝑓𝐹 𝑛𝐹

𝑘0 , 𝜆0

𝑘0 , 𝜆 0

𝛽

𝜆 = 𝜆𝑜 + � 𝑡𝑖 𝑖=1

(Rinne 2008, p.520) 𝑘 = 𝑘𝑜 + 𝑓𝐹 /2 𝑛 1 𝜆 = 𝜆𝑜 + �(𝑡𝑖 − 𝜇)2 2

𝜂0 , Λ 0 k 0 , λ0

𝑡𝑇 = ∑ t Fi + ∑ t Si = 𝑡𝑓𝑡𝑠𝑠 𝑡𝑠𝑚𝑅 𝑠𝑓 𝑡𝑅𝑠𝑡

𝑖=1

𝜂 = 𝜂0 + 𝑓𝐹 𝑘𝐵 Λ = Λ 𝑜 + 𝑡𝑇 k = k 𝑜 + 𝑓𝐹 𝑛𝐹

𝑚𝑖 λ = λ𝑜 + � ln � � 𝜃 𝑖=1

Description , Limitations and Uses Example 1

For an example using the gamma distribution as a conjugate prior see the Poisson or Exponential distributions. A renewal process has an exponential time between failure with parameter 𝜆 = 0.01 under the homogeneous Poisson process conditions. What is the probability the forth failure will occur before 200 hours.

Example 2

𝐹(200; 4,0.01) = 0.1429

5 components are put on a test with the following failure times: 38, 42, 44, 46, 55 hours

Gamma

𝜆 where 𝜆 = 𝛼 −𝛽 from 𝑊𝑏𝑠(𝑡; 𝛼, 𝛽)

Likelihood Model

104

Univariate Continuous Distributions

Solving:

5𝑘 − 225 λ 0 = 5ln(λ) − 5𝜓(𝑘) + 18.9954 0=

Gives:

k� = 21.377

𝜆̂ = 0.4749

90% confidence interval for 𝑘:

𝐼(𝑘, 𝜆) = �

0.0479 0.4749

0.4749 � 4.8205

−1 −1 179.979 �𝐽𝑛 �𝑘� , 𝜆̂�� = �𝑓𝐹 𝐼�𝑘�, 𝜆̂�� = � −17.730

�𝑘�. exp �

𝛷 −1 (0.95)√179.979 �, −𝑘� [7.6142,

𝛷 −1 (0.95)√179.979 �� 𝑘� 60.0143]

𝛷 −1 (0.95)√1.7881 �, −𝜆̂ [0.0046,

𝛷 −1 (0.95)√1.7881 �� 𝜆̂ 48.766]

90% confidence interval for 𝜆:

Gamma

−17.730 � 1.7881

�𝜆̂. exp �

𝑘�. exp �

𝜆̂. exp �

Note that this confidence interval uses the assumption of the parameters being normally distributed which is only true for large sample sizes. Therefore these confidence intervals may be inaccurate. Bayesian methods must be done numerically. Characteristics

The gamma distribution was originally known as a Pearson Type III distribution. This distribution includes a location parameter 𝛾 which shifts the distribution along the x-axis. 𝑝(𝑡; 𝑘, 𝜆, 𝛾) =

𝜆𝐸 (𝑡 − 𝛾)𝐸−1 −λ(t−γ) e Γ(𝑘)

When k is an integer, the Gamma distribution is called an Erlang distribution. 𝒌 Characteristics: 𝒌 < 1. 𝑝(0) = ∞. There is no mode. 𝒌 = 𝟏. 𝑝(0) = 𝜆. The gamma distribution reduces to an exponential distribution with failure rate λ. Mode at 𝑡 = 0. 𝒌 > 1. 𝑝(0) = 0 Large 𝒌. The gamma distribution approaches a normal 𝐸

𝐸

distribution with 𝜇 = 𝜆 , 𝜎 = �𝜆2.

Gamma Continuous Distribution

105

Homogeneous Poisson Process (HPP). Components with an exponential time to failure which undergo instantaneous renewal with an identical item undergo a HPP. The Gamma distribution is probability distribution of the kth failed item and is derived from the convolution of 𝑘 exponentially distributed random variables, 𝑇𝑖 . (See related distributions, exponential distribution). Scaling property:

𝑇~𝐺𝑠𝑚𝑚𝑠(𝑘, 𝜆)

𝜆 𝑠𝑇~𝐺𝑠𝑚𝑚𝑠 �𝑘, � 𝑠

Convolution property: 𝑇1 + 𝑇2 + … + 𝑇𝑛 ~𝐺𝑠𝑚𝑚𝑠(∑𝑘𝑖 , 𝜆) Where 𝜆 is fixed. Properties from (Leemis & McQueston 2008)

Applications

System Failure. Can be used to model system failure with 𝑘 backup systems. Life Distribution. The gamma distribution is flexible in shape and can give good approximations to life data. Bayesian Analysis. The gamma distribution is often used as a prior in Bayesian analysis to produce closed form posteriors. Online: http://mathworld.wolfram.com/GammaDistribution.html http://en.wikipedia.org/wiki/Gamma_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (interactive web calculator) http://www.itl.nist.gov/div898/handbook/eda/section3/eda366b.htm

Resources

Books: Artin, E., 1964. The Gamma Function, New York: Holt, Rinehart & Winston. Johnson, N.L., Kotz, S. & Balakrishnan, N., 1994. Continuous Univariate Distributions, Vol. 1 2nd ed., Wiley-Interscience. Bowman, K.O. & Shenton, L.R., 1988. Properties of estimators for the gamma distribution, CRC Press.

Gamma

Renewal Theory, Homogenous Poisson Process. Used to model a renewal process where the component time to failure is exponentially distributed and the component is replaced instantaneously with a new identical component. The HPP can also be used to model ruin theory (used in risk assessments) and queuing theory.

106

Univariate Continuous Distributions

Relationship to Other Distributions

𝑝(𝑡; 𝑘, 𝜆, 𝛾, 𝜉) =

𝜉𝜆𝜉𝐸 (𝑡 − 𝛾)𝜉𝐸−1 exp{−[𝜆(𝑡 − 𝛾)]𝐸 } Γ(𝑘)

Gamma

𝜆 - Scale Parameter 𝑘 - Shape Parameter 𝛾 - Location parameter 𝜉 - Second shape parameter

The generalized gamma distribution has been derived because it is a generalization of a large amount of probability distributions. Such Generalized as: Gamma 𝐺𝑠𝑚𝑚𝑠(𝑡; 1, 𝜆, 0,1) = 𝐸𝑚𝑝(𝑡; 𝜆) Distribution 1 𝐺𝑠𝑚𝑚𝑠 �𝑡; 1, , β, 1� = 𝐸𝑚𝑝(𝑡; 𝜇, 𝛽) 𝜇 𝐺𝑠𝑚𝑚𝑠(𝑡; 𝑘, 𝜆, γ, 𝜉) 1 𝐺𝑠𝑚𝑚𝑠 �𝑡; 1, , 0, 𝛽� = 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽) 𝛼 1 𝐺𝑠𝑚𝑚𝑠 �𝑡; 1, , γ, 𝛽� = 𝑊𝑅𝑠𝑏𝑠𝑠𝑠(𝑡; 𝛼, 𝛽, 𝛾) 𝛼 𝑓 1 𝐺𝑠𝑚𝑚𝑠 �𝑡; , , 0,1� = χ2 (t; n) 2 2 𝑓 1 , 0,2� = χ(t; n) 𝐺𝑠𝑚𝑚𝑠 �𝑡; , 2 √2 1 𝐺𝑠𝑚𝑚𝑠 �𝑡; 1, , 0,2� = Rayleigh(t; σ) 𝜎 Let

Then Exponential Distribution 𝐸𝑚𝑝(𝑡; λ)

𝑇𝑡 = 𝑇1 + 𝑇2 + ⋯ + 𝑇𝐸

𝑇𝑡 ~𝐺𝑠𝑚𝑚𝑠(𝑘, 𝜆)

Special Case:

Then

𝑃𝑓𝑠𝑠(𝑘; 𝜆𝑡)

𝑠𝑓𝑝

This is gives the Gamma distribution its convolution property.

Let

Poisson Distribution

𝑇1 … 𝑇𝐸 ~𝐸𝑚𝑝(𝜆)

𝐸𝑚𝑝(𝑡; 𝜆) = 𝐺𝑠𝑚𝑚𝑠(𝑡; 𝑘 = 1, 𝜆)

𝑇1 … 𝑇𝐸 ~𝐸𝑚𝑝(𝜆)

𝑠𝑓𝑝

𝑇𝑡 = 𝑇1 + 𝑇2 + ⋯ + 𝑇𝐸

𝑇𝑡 ~𝐺𝑠𝑚𝑚𝑠(𝑘, 𝜆)

The Poisson distribution is the probability that exactly 𝑘 failures have been observed in time 𝑡. This is the probability that 𝑡 is between 𝑇𝐸 and 𝑇𝐸+1 . 𝐸+1

𝑝𝑃𝑜𝑖𝑠𝑠𝑜𝑛 (𝑘; 𝜆𝑡) = �

𝐸

𝑝𝐺𝑎𝑚𝑚𝑎 (𝑡; 𝑚, 𝜆)𝑝𝑚

= 𝐹𝐺𝑎𝑚𝑚𝑎 (𝑡; 𝑘 + 1, 𝜆) − 𝐹𝐺𝑎𝑚𝑚𝑎 (𝑡; 𝑘, 𝜆)

Gamma Continuous Distribution

Normal Distribution 𝑁𝑓𝑠𝑚(𝑡; 𝜇, 𝜎)

Chi-square Distribution

𝜒 2 (𝑡; 𝑠)

Inverse Gamma Distribution 𝐼𝐺(𝑡; 𝛼, β)

where 𝑘 is an integer.

Special Case for large k: lim 𝐺𝑠𝑚𝑚𝑠(𝑘, 𝜆) = 𝑁𝑓𝑠𝑚 �𝜇 =

𝐸→∞

Special Case: 𝜒 2 (𝑡; 𝑠) = 𝐺𝑠𝑚𝑚𝑠(𝑡; 𝑘 =

where 𝑠 is an integer Let

Then Let

Beta Distribution

Dirichlet Distribution 𝐷𝑠𝑠𝑑 (𝒙; 𝛂)

Then Let:

Then: Let:

Then:

𝑘 𝑘 , 𝜎 = � 2� 𝜆 𝜆

𝑠 1 ,𝜆 = ) 2 2

X~𝐺𝑠𝑚𝑚𝑠(𝑘, 𝜆)

𝑠𝑓𝑝

Y=

1 X

𝑋1 , 𝑋2 ~𝐺𝑠𝑚𝑚𝑠(k i , λi )

𝑠𝑓𝑝

Y=

X1 X1 + X 2

Y~I𝐺(𝛼 = 𝑘, β = λ)

Y~𝐵𝑅𝑡𝑠(𝛼 = 𝑘1 , 𝛽 = 𝑘2 )

𝑌𝑖 ~𝐺𝑠𝑚𝑚𝑠(𝜆, 𝑘𝑖 ) 𝑠. 𝑠. 𝑝 𝑠𝑓𝑝

𝑉~𝐺𝑠𝑚𝑚𝑠(𝜆, ∑𝑘𝑖 ) 𝒁=�

𝑑

𝑉 = � 𝑌𝑖 𝑖=1

𝑌𝑑 𝑌1 𝑌2 , ,…, � 𝑉 𝑉 𝑉

𝒁~𝐷𝑠𝑠𝑑 (𝛼1 , … , 𝛼𝐸 )

*i.i.d: independent and identically distributed Wishart Distribution 𝑊𝑠𝑠ℎ𝑠𝑠𝑡𝑑 (𝑓; 𝚺)

The Wishart Distribution is the multivariate generalization of the gamma distribution.

Gamma

𝐵𝑅𝑡𝑠(𝑡; α, β)

107

108

Univariate Continuous Distributions

4.4. Logistic Continuous Distribution Probability Density Function - f(t) 0.3

μ=0 μ=2 μ=4

0.2 0.2

s=0.7 s=1 s=2

0.3

0.2

0.1 0.1

0.1 0.0

0.0 -6

-1

4

9

-7

-2

3

Logistic

Cumulative Density Function - F(t) 1.0

1.0

0.8

0.8

0.6

0.6

0.4

μ=0 s=1 μ=2 s=1 μ=4 s=1

0.2 0.0

0.4

s=0.7 s=1 s=2

0.2 0.0

-6

-1

4

9

-7

-2

3

Hazard Rate - h(t) 1.0

μ=0 μ=2 μ=4

0.8 0.6

μ=0 s=0.7 μ=0 s=1 μ=0 s=2

1.4 1.2 1.0 0.8 0.6

0.4

0.4

0.2

0.2

0.0

0.0 -6

-1

4

9

-7

-2

3

Logistic Continuous Distribution

109

Parameters & Description

Parameters

µ 𝑠

Limits

Location parameter. 𝜇 is the mean, median and mode of the distribution.

−∞ < 𝜇 < ∞

Scale parameter. Proportional to the standard deviation of the distribution.

s>0

−∞ < t < ∞

Distribution

Formulas

𝑝(𝑡) = PDF where

=

ez e−z = z 2 s(1 + e ) s(1 + e−z )2 1 𝑡−𝜇 sech2 � � 4𝑠 2𝑠

CDF

=

1 ez = 1 + e−z 1 + ez

1 1 𝑡−𝜇 + tanh � � 2 2 2𝑠

R(t) =

Reliability

Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡) Mean Life

𝑡−𝜇 𝑠

Residual

Hazard Rate

Cumulative Hazard Rate

𝑚(𝑚) = 𝑅(𝑚|𝑡) =

Logistic

𝐹(𝑡) =

𝑧=

1 1 + ez

𝑡−𝜇 1 + exp � 𝑠 � 𝑅(𝑡 + 𝑚) = 𝑡+𝑚−𝜇 𝑅(𝑡) 1 + exp � � 𝑠

Where 𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡. t

𝑠(𝑡) = (1 + 𝑅 z ) �s. ln �𝑅 �s + 𝑅

𝜇� 𝑠�

1 𝐹(𝑡) = s(1 + e−z ) 𝑠 1 = 𝜇−𝑡 s + s exp � 𝑠 �

ℎ(𝑡) =

𝑡−𝜇 𝐻(𝑡) = ln �1 + exp � �� 𝑠

− t�

110

Univariate Continuous Distributions

Properties and Moments µ

Median

µ

Mode Mean - 1st Raw Moment nd

Variance - 2

µ

π2 2 s 3

Central Moment

Skewness - 3rd Central Moment

0 6 5

th

Excess kurtosis - 4 Central Moment

𝑅 𝑖𝜇𝑡 𝐵(1 − 𝑠𝑠𝑡, 1 + 𝑠𝑠𝑡) for |𝑠𝑡| < 1

Characteristic Function 100𝛾 % Percentile Function

Parameter Estimation

𝛾 𝑡𝛾 = 𝜇 + 𝑠 ln � � 1−𝛾

Logistic

Plotting Method Least Mean Square 𝑦 = 𝑚𝑚 + 𝑐

X-Axis

Y-Axis

Likelihood Function

For complete data:

ti

ln[F] − ln[1 − 𝐹]

Maximum Likelihood Function

𝑠̂ =

1 m

𝜇̂ = −𝑐𝑠̂

𝑡 −𝜇 exp � 𝑖−𝑠 � 𝐿(𝜇, 𝑠|𝐸) = � 𝑡𝑖 − 𝜇 2 i=1 s �1 + exp � ����������������� −𝑠 �� nF

failures

Log-Likelihood Function

∂Λ =0 ∂µ ∂Λ =0 ∂s

𝑛𝐹

𝑛𝐹

𝑡𝑖 − 𝜇 𝑡𝑖 − 𝜇 Λ(𝜇, 𝑠|𝐸) = −nF ln s + � � � − 2 � ln �1 + exp � �� −𝑠 −𝑠 ��������������������������������� 𝑖=1 𝑖=1 𝑛𝐹

failures

∂Λ nF 2 1 = − � 𝑡𝑖 − 𝜇 = 0 ∂µ s 𝑠 𝑖=1 �1 + exp � 𝑠 �� ������������������� failures

𝑛𝐹 𝑡𝑖 − 𝜇 ∂Λ nF 1 𝑡𝑖 − 𝜇 1 − exp � 𝑠 � = − − �� �� 𝑡 −𝜇 �=0 ∂s s 𝑠 𝑠 1 + exp � 𝑖 𝑠 � 𝑖=1 ��������������������������� failures

Logistic Continuous Distribution

MLE Point Estimates

111

The MLE estimates for 𝜇̂ and 𝑠̂ are found by solving the following equations: 𝑛𝐹

1 1 𝑡𝑖 − 𝜇 −1 − � �1 + exp � �� = 0 2 nF 𝑠 𝑖=1 𝑛𝐹 𝑡𝑖 − 𝜇 1 𝑡𝑖 − 𝜇 1 − exp � 𝑠 � 1 + �� � 𝑡 −𝜇 =0 nF 𝑠 1 + exp � 𝑖 𝑖=1 𝑠 �

These estimates are biased. (Balakrishnan 1991) provides tables derived from Monte Carlo simulation to correct the bias. Fisher Information

(Antle et al. 1970) 100𝛾% Confidence Intervals

1 3𝑠 2 𝐼(𝜇, 𝑠) = � 0

0

� 𝜋 +3 9𝑠 2 2

Confidence intervals are most often obtained from tables derived from Monte Carlo simulation. Corrections from using the Fisher Information matrix method are given in (Antle et al. 1970). Logistic

Bayesian

Type

Non-informative Priors 𝝅𝟎 (𝝁, 𝒔) Prior 1 𝑠

Jeffery Prior

Description , Limitations and Uses Example

The accuracy of a cutting machine used in manufacturing is desired to be measured. 5 cuts at the required length are made and measured as: 7.436, 10.270, 10.466, 11.039, 11.854 𝑚𝑚 Numerically solving MLE equations gives: 𝜇̂ = 10.446 𝑠̂ = 0.815

This gives a mean of 10.446 and a variance of 2.183. Compared to the same data used in the Normal distribution section it can be seen that this estimate is very similar to a normal distribution. 90% confidence interval for 𝜇: �𝜇̂ − Φ−1 (0.95)�

3𝑠̂ 2 , 𝑓𝐹

𝜇̂ + Φ−1 (0.95)�

[9.408, 11.4844]

3𝑠̂ 2 � 𝑓𝐹

112

Univariate Continuous Distributions

90% confidence interval for 𝑠:

⎡ ⎧ −1 ⎫ 9𝑠̂ 2 (0.95)� ⎢ ⎪𝛷 𝑓𝐹 (3 + 𝜋 2 )⎪ ⎢𝑠̂ . exp , −𝑠̂ ⎨ ⎬ ⎢ ⎢ ⎪ ⎪ ⎣ ⎩ ⎭

⎧ −1 ⎫⎤ 9𝑠̂ 2 (0.95)� ⎪𝛷 𝑓𝐹 (3 + 𝜋 2 )⎪⎥ ⎥ 𝑠̂ . exp 𝑠̂ ⎨ ⎬⎥ ⎪ ⎪⎥ ⎩ ⎭⎦

[0.441, 1.501]

Note that this confidence interval uses the assumption of the parameters being normally distributed which is only true for large sample sizes. Therefore these confidence intervals may be inaccurate. Bayesian methods must be calculated using numerical methods.

Logistic

Characteristics

The logistic distribution is most often used to model growth rates (and has been used extensively in biology and chemical applications). In reliability engineering it is most often used as a life distribution. Shape. There is no shape parameter and so the logistic distribution is always a bell shaped curve. Increasing 𝜇 shifts the curve to the right, increasing 𝑠 increases the spread of the curve.

Normal Distribution. The shape of the logistic distribution is very similar to that of a normal distribution with the logistic distribution having slightly ‘longer tails’. It would take a large number of samples to distinguish between the distributions. The main difference is that the hazard rate approaches 1/𝑠 for large 𝑡. The logistic function has historically been preferred over the normal distribution because of its simplified form. (Meeker & Escobar 1998, p.89) Alternative Parameterization. It is equally as popular to present the logistic distribution using the true standard deviation 𝜎 = 𝜋𝑠⁄√3. This form is used in reference book, Balakrishnan 1991, and gives the following cdf: 𝐹(𝑡) =

1 −𝜋 𝑡 − 𝜇 1 + exp � � 𝜎 �� √3

Standard Logistic Distribution. The standard logistic distribution has 𝜇 = 0, 𝑠 = 1. The standard logistic distribution random variable, 𝑍, is related to the logistic distribution: 𝑍=

𝑋−𝜇 𝑠

Logistic Continuous Distribution

113

Let: 𝑇~𝐿𝑓𝑅𝑠𝑠𝑡𝑠𝑐(𝑡; 𝜇, 𝑠)

Scaling property (Leemis & McQueston 2008) 𝑠𝑇~𝐿𝑓𝑅𝑠𝑠𝑡𝑠𝑐(𝑡; 𝜇, 𝑠𝑠)

Rate Relationships. The distribution has the following rate relationships which make it suitable for modeling growth (Hastings et al. 2000, p.127): ℎ(𝑡) =

where

𝑧 = ln �

𝐹(𝑡) � = ln[𝐹(𝑡)] − ln[1 − 𝐹(𝑡)] 𝑅(𝑡)

𝑝(𝑡) =

𝑧=

𝑡−𝜇 𝑠

𝑝𝐹(𝑡) = 𝐹(𝑡)𝑅(𝑡) 𝑝𝑡

Growth Model. The logistic distribution most common use is a growth model. Life Distribution. In reliability applications it is used as a life distribution. It is similar in shape to a normal distribution and so is often used instead of a normal distribution due to its simplified form. (Meeker & Escobar 1998, p.89) Logistic Regression. Logistic regression is a generalized linear regression model used predict binary outcomes. (Agresti 2002)

Resources

Online: http://mathworld.wolfram.com/LogisticDistribution.html http://en.wikipedia.org/wiki/Logistic_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (web calc) http://www.weibull.com/LifeDataWeb/the_logistic_distribution.htm Books: Balakrishnan, 1991. Handbook of the Logistic Distribution 1st ed., CRC. Johnson, N.L., Kotz, S. & Balakrishnan, N., 1995. Continuous Univariate Distributions, Vol. 2 2nd ed., Wiley-Interscience.

Logistic

when 𝜇 = 0 and 𝑠 = 1:

Applications

f(t) 𝐹(𝑡) = R(t) 𝑠

114

Univariate Continuous Distributions

Relationship to Other Distributions Let Exponential Distribution 𝐸𝑚𝑝(𝑡; 𝜆)

Then

𝑋~𝐸𝑚𝑝(𝜆 = 1)

𝑠𝑓𝑝

𝑌~𝐿𝑓𝑅𝑠𝑠𝑡𝑠𝑐(0,1) (Hastings et al. 2000, p.127) Let

Pareto Distribution 𝑃𝑠𝑠𝑅𝑡𝑓(𝜃, 𝛼)

𝑌 = ln �

Then

𝑋~𝑃𝑠𝑠𝑅𝑡𝑓(𝜃, 𝛼)

𝑅 −𝑋 � 1 + 𝑅 −𝑋

X α 𝑌 = − ln �� � − 1� θ

𝑠𝑓𝑝

𝑌~𝐿𝑓𝑅𝑠𝑠𝑡𝑠𝑐(0,1) (Hastings et al. 2000, p.127) Let

Logistic

Gumbel Distribution 𝐺𝑠𝑚𝑏𝑅𝑠(𝛼 , 𝛽)

Then

𝑋𝑖 ~𝐺𝑠𝑚𝑏𝑅𝑠(𝛼 , 𝛽)

𝑠𝑓𝑝

𝑌~𝐿𝑓𝑅𝑠𝑠𝑡𝑠𝑐(0, 𝛽) (Hastings et al. 2000, p.127)

𝑌 = X1 − X 2

Normal Continuous Distribution

115

4.5. Normal (Gaussian) Continuous Distribution Probability Density Function - f(t)

μ=-3 σ=1 μ=0 σ=1 μ=3 σ=1

0.5 0.4

μ=0 σ=0.6 μ=0 σ=1 μ=0 σ=2

0.6

0.3

0.4

0.2 0.2 0.1 0.0 -6

-4

0.0 0

-2

2

4

6

-6

-4

-2

0

2

4

6

0

2

4

6

Cumulative Density Function - F(t)

-4

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

-2

0

2

4

6

-6

-4

Normal

-6

1.0

-2

Hazard Rate - h(t) μ=-3 σ=1

6

μ=0 σ=1

5

μ=3 σ=1

-6

-4

-2

4

6 μ=-3 σ=0.6 μ=0 σ=1 5 μ=0 σ=2 4

3

3

2

2

1

1

0

0 0

2

4

6

-6

-4

-2

0

2

4

6

116

Univariate Continuous Distributions

Parameters & Description

Parameters

𝜇

𝜎2

Limits

Location parameter: The mean of the distribution.

−∞ < µ < ∞ σ2 > 0

Scale parameter: The standard deviation of the distribution. −∞ < t < ∞

Distribution

Formulas

1 t−µ 2 exp � − � � � 2 σ σ√2𝜋 1 𝑡−µ = 𝜙� � σ σ

𝑝(𝑡) =

PDF

1

where 𝜙 is the standard normal pdf with 𝜇 = 0 and 𝜎 2 = 1.

Normal

𝐹(𝑡) =

𝑡 1 θ−µ 2 � exp � − � � � 𝑝𝜃 2 σ σ√2𝜋 −∞

1

=

CDF

1 1 𝑡−µ + erf � � 2 2 σ√2 t−µ � = Φ� σ

where Φ is the standard normal cdf with 𝜇 = 0 and 𝜎 2 = 1. t−µ R(t) = 1 − Φ � � σ µ−t = Φ� � σ

Reliability

Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡) Mean Life

Residual

Hazard Rate

Cumulative Hazard Rate

µ−x−t � 𝑅(𝑡 + 𝑚) Φ � σ 𝑚(𝑚) = 𝑅(𝑚|𝑡) = = µ−t 𝑅(𝑡) Φ� σ �

Where 𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡. 𝑠(𝑡) =

∞

∞

∫𝑡 𝑅(𝑚)𝑝𝑚 ∫𝑡 𝑅(𝑚)𝑝𝑚 = 𝑅(𝑡) 𝑅(𝑡)

t−µ 𝜙� σ � ℎ(𝑡) = µ−t 𝜎 �Φ � σ ��

𝜇−𝑡 𝐻(𝑡) = − ln �𝛷 � �� 𝜎

Normal Continuous Distribution

117

Properties and Moments µ

Median

µ

Mode st

µ

Mean - 1 Raw Moment nd

Variance - 2

σ2

Central Moment

Skewness - 3rd Central Moment

0

th

0

Excess kurtosis - 4 Central Moment

1

exp �𝑠𝜇𝑡 − 2𝜎 2 𝑡 2 �

Characteristic Function

𝑡𝛼 = µ + σΦ−1 (α) = µ + σ√2 erf −1 (2𝛼 − 1)

100𝛼% Percentile Function

Parameter Estimation Plotting Method X-Axis

Y-Axis 𝑡𝑖

𝑠𝑓𝑠𝑁𝑓𝑠𝑚[𝐹(𝑡𝑖 )]

σ �=

Maximum Likelihood Function Likelihood Function

For complete data: 𝐿(𝜇, 𝜎|𝐸) = =

Log-Likelihood Function

µ� = −

1 �2 1 , σ = 2 𝑚 𝑚

nF 1 𝑡𝑖 − 𝜇 2 � exp �− � � � 2 𝜎 i=1 �σ√2π� �������������������������

1

nF

failures

1

𝑛𝐹

1 �(𝑡𝑖 − 𝜇)2 � nF exp �− 2𝜎 2 �σ√2π� 𝑖=1 ������������������������� failures

𝑛𝐹

1 Λ(𝜇, 𝜎|𝐸) = −nF ln�σ√2π� − 2 �(𝑡𝑖 − 𝜇)2 2𝜎 ����������������������� 𝑖=1

∂Λ =0 ∂µ

solve for 𝜇 to get MLE 𝜇̂ :

∂Λ =0 ∂σ

solve for 𝜎 to get 𝜎�:

failures 𝑛𝐹

∂Λ 𝜇𝑓𝐹 1 = 2 − 2 � 𝑡𝑖 = 0 ∂µ 𝜎 𝜎 �������� ��� 𝑖=1 failures

𝑛𝐹

∂Λ nF 1 = − + 3 �(𝑡𝑖 − 𝜇)2 = 0 ∂σ σ 𝜎 ��������������� 𝑖=1 failures

𝑐 𝑚

Normal

Least Mean Square 𝑦 = 𝑚𝑚 + 𝑐

118

Univariate Continuous Distributions

MLE Point Estimates

When there is only complete failure data the point estimates can be given as: 𝑛𝐹

1 𝜇̂ = � 𝑡𝑖 nF

𝑛𝐹

1 = �(𝑡𝑖 − 𝜇)2 nF

�2 𝜎

𝑖=1

𝑖=1

In most cases the unbiased estimators are used: 𝑛𝐹

1 𝜇̂ = � 𝑡𝑖 nF 𝑖=1

Normal

(for complete data)

𝑛𝐹

1 �(𝑡𝑖 − 𝜇)2 nF − 1 𝑖=1

1/𝜎 2 𝐼(𝜇, 𝜎 2 ) = � 0

Fisher Information 100𝛾% Confidence Intervals

�2 = 𝜎

1 Sided - Lower 𝝁 𝝈

𝜇̂ −

𝟐

𝜎�2

𝜎�

√𝑓

𝑡γ (n − 1)

(𝑓 − 1)

𝜒𝛼2 (𝑓 − 1)

0 � −1/2𝜎 4

2 Sided - Lower 𝜇̂ −

�2 𝜎

𝜎�

√𝑓

𝑡�1+γ� (n − 1) 2

(𝑓 − 1)

2 Sided - Upper 𝜇̂ +

𝜎�2

𝜒 21+γ (𝑓 − 1) �

2

�

𝜎�

√𝑓

𝑡�1+γ� (n − 1) 2

(𝑓 − 1)

𝜒 21−γ (𝑓 − 1) �

2

�

(Nelson 1982, pp.218-220) Where 𝑡γ (n − 1) is the 100𝛾 th percentile of the t-distribution with 𝑓 − 1 degrees of freedom and 𝜒𝛾2 (𝑓 − 1) is the 100𝛾 th percentile of the 𝜒 2 -distribution with 𝑓 − 1 degrees of freedom. P

P

Bayesian

Type

Non-informative Priors when 𝝈𝟐 is known, 𝝅𝟎 (𝝁) (Yang and Berger 1998, p.22) Prior

1 𝑏−𝑠

Uniform Proper Prior with limits 𝜇 ∈ [𝑠, 𝑏]

1

All

Type Uniform Proper Prior with limits 𝜎 2 ∈ [𝑠, 𝑏]

Posterior

Truncated Normal Distribution For a ≤ µ ≤ b ∑𝑛𝐹 𝑡𝑖𝐹 σ2 𝑐. 𝑁𝑓𝑠𝑚 �µ; 𝑖=1 , � 𝑓𝐹 nF Otherwise 𝜋(𝜇) = 0 𝑁𝑓𝑠𝑚 �µ;

when 𝜇 ∈ (∞, ∞)

𝐹 ∑𝑛𝑖=1 𝑡𝑖𝐹 σ2 , � 𝑓𝐹 nF

Non-informative Priors when 𝝁 is known, 𝝅𝒐 (𝝈𝟐 ) (Yang and Berger 1998, p.23) Prior

Posterior

1 𝑏−𝑠

Truncated Inverse Gamma Distribution For a ≤ σ2 ≤ b

Normal Continuous Distribution

119

(𝑓𝐹 − 2) S 2 , � 2 2 Otherwise 𝜋(𝜎 2 ) = 0 𝑐. 𝐼𝐺 �σ2 ;

1

Uniform Improper Prior with limits 𝜎 2 ∈ (0, ∞)

1 𝜎2

Jeffery’s, Reference, MDIP Prior

Type

𝑓𝐹 S 2 , � 2 2 with limits 𝜎 2 ∈ (0, ∞) See section 1.7.1 𝐼𝐺 �σ2 ;

Non-informative Priors when 𝝁 and 𝝈𝟐 are unknown, 𝝅𝒐 (𝝁, 𝝈𝟐 ) (Yang and Berger 1998, p.23) Prior

Posterior

1

Jeffery’s Prior

1 𝜎4

Reference Prior ordering {𝜙, 𝜎}

Reference where 𝜇 and 𝜎 2 are separate groups.

𝜋𝑜 (𝜙, 𝜎 2 ) 1 ∝ 𝜎�2 + 𝜙 2 where 𝜙 = 𝜇/𝜎 1 𝜎2

S2 � nF (nF − 3) See section 1.7.2 (𝑓𝐹 − 3) S 2 2 𝜋(𝜎 |𝐸)~𝐼𝐺 �σ2 ; , � 2 2 See section 1.7.1

𝜋(𝜇|𝐸)~𝑇 �µ; nF − 3, 𝑡̅,

S2 � nF (nF + 1) when 𝜇 ∈ (∞, ∞) See section 1.7.2 (𝑓𝐹 + 1) S 2 𝜋(𝜎 2 |𝐸)~𝐼𝐺 �σ2 ; , � 2 2 2 when 𝜎 ∈ (0, ∞) See section 1.7.1

𝜋(𝜇|𝐸)~𝑇 �µ; nF + 1, 𝑡̅,

No Closed Form

S2 � nF (nF − 1) when 𝜇 ∈ (∞, ∞) See section 1.7.2 (𝑓𝐹 − 1) S 2 , � 𝜋(𝜎 2 |𝐸)~𝐼𝐺 �σ2 ; 2 2 2 when 𝜎 ∈ (0, ∞) See section 1.7.1

𝜋(𝜇|𝐸)~𝑇 �µ; nF − 1, 𝑡̅,

Normal

Improper Uniform with limits: 𝜇 ∈ (∞, ∞) 𝜎 2 ∈ (0, ∞)

MDIP Prior

(𝑓𝐹 − 2) S 2 , � 2 2 See section 1.7.1

𝐼𝐺 �σ2 ;

120

Univariate Continuous Distributions

where

𝑛𝐹

𝑆 = �(𝑡𝑖 − 𝑡̅)2 2

𝑖=1

and

𝑡̅ =

𝑛𝐹

1 � 𝑡𝑖 nF 𝑖=1

Conjugate Priors UOI

Likelihood Model

Evidence

Dist of UOI

Prior Para

Posterior Parameters 𝑛

𝜇 from

Normal

𝑁𝑓𝑠𝑚(𝑡; 𝜇, 𝜎 2 )

𝜎2 from

2

𝑁𝑓𝑠𝑚(𝑡; 𝜇, 𝜎 )

𝜇𝑁 from

𝐿𝑓𝑅𝑁(𝑡; 𝜇𝑁 , 𝜎𝑁2 )

Normal with known 𝜎2

𝑓𝐹 failures at times 𝑡𝑖

Normal with known 𝜇

𝑓𝐹 failures at times 𝑡𝑖

LogNormal with known 𝜎𝑁2

𝑓𝐹 failures at times 𝑡𝑖

Normal

Gamma

u𝑜 , 𝑠0

𝑘0 , 𝜆0

𝑠=

1 1 nF 𝑠0 + 𝜎 2

𝑘 = 𝑘𝑜 + 𝑓𝐹 /2 𝑛𝐹

1 𝜆 = 𝜆𝑜 + �(𝑡𝑖 − 𝜇)2 2 𝑖=1 𝑛

Normal

𝑠𝑜 , 𝑠0

𝐹 ln(𝑡𝑖 ) u0 ∑𝑖=1 2+ σ0 𝜎𝑁2 𝑠= 1 𝑓𝐹 + 𝑠 2 𝜎𝑁2

𝑠=

Description , Limitations and Uses Example

𝐹

𝐹 𝑡𝑖 𝑠0 ∑𝑖=1 𝑠0 + 𝜎 2 𝑠= 1 𝑓𝐹 + 𝑠0 𝜎 2

1 1 𝑓𝐹 + 𝑠 2 𝜎𝑁2

The accuracy of a cutting machine used in manufacturing is desired to be measured. 5 cuts at the required length are made and measured as: 7.436, 10.270, 10.466, 11.039, 11.854 𝑚𝑚 MLE Estimates are:

∑ 𝑡𝑖𝐹 = 10.213 nF 2 ∑�𝑡𝑖𝐹 − 𝜇�𝑡 � �2 = = 2.789 𝜎 nF − 1 𝜇̂ =

90% confidence interval for 𝜇: 𝜎� 𝜎� �𝜇̂ − 𝑡{0.95} (4), 𝜇̂ + 𝑡{0.95} (4)� √5 √5 [10.163, 10.262]

Normal Continuous Distribution

121

90% confidence interval for 𝜎 2 : 4 4 �2 �𝜎 , 𝜎�2 2 � 𝜒{20.95} (4) 𝜒{0.05} (4) [1.176, 15.697]

A Bayesian point estimate using the Jeffery non-informative improper prior 1⁄𝜎 4 with posterior for 𝜇~𝑇(6, 10.213, 0.558 ) and 𝜎 2 ~𝐼𝐺(3, 5.578) has a point estimates: 𝜇̂ = E[𝑇(6,6.595,0.412 )] = µ = 10.213 �2 = E[𝐼𝐺(3,5.578)] = 5.578 = 2.789 𝜎 2

𝐹𝑇−1 (0.95) = 11.665]

1/𝐹𝐺−1 (0.05) = 6.822]

Also known as a Gaussian distribution or bell curve.

Unit Normal Distribution. Also known as the standard normal distribution is when 𝜇 = 0 and 𝜎 = 1 with pdf 𝜙(𝑧) and cdf Φ(z). If X is normally distributed with mean 𝜇 and standard deviation 𝜎 then the following transformation is used: 𝑚−𝜇 𝑧= 𝜎 Central Limit Theorem. Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 be a sequence of 𝑓 independent and identically distributed (i.i.d) random variables each having a mean of 𝜇 and a variance of 𝜎 2 . As the sample size increases, the distribution of the sample average of these random variables approaches the normal distribution with mean 𝜇 and variance 𝜎 2 /𝑓 irrespective of the shape of the original distribution. Formally: 𝑆𝑛 = 𝑋1 + ⋯ + 𝑋𝑛 If we define a new random variables: 𝑆𝑛 − 𝑓𝜇 𝑍𝑛 = , 𝑠𝑓𝑝 𝜎 √𝑓

𝑌=

𝑆𝑛 𝑓

The distribution of 𝑍𝑛 converges to the standard normal distribution. The distribution of 𝑆𝑛 converges to a normal distribution with mean 𝜇 and standard deviation of 𝜎/√𝑓.

Sigma Intervals. Often intervals of the normal distribution are expressed in terms of distance away from the mean in units of

Normal

Characteristics

With 90% confidence intervals: 𝜇 [𝐹𝑇−1 (0.05) = 8.761, 2 𝜎 [1/𝐹𝐺−1 (0.95) = 0.886,

122

Univariate Continuous Distributions

sigma. The following is approximate values for each sigma: Interval 𝜇±𝜎 𝜇 ± 2𝜎 𝜇 ± 3𝜎 𝜇 ± 4𝜎 𝜇 ± 5𝜎 𝜇 ± 6𝜎

𝚽(𝝁 + 𝒏𝝈) − 𝚽(𝝁 − 𝒏𝝈) 68.2689492137% 95.4499736104% 99.7300203937% 99.9936657516% 99.9999426697% 99.9999998027%

Truncated Normal. Often in reliability engineering a truncated normal distribution may be used due to the limitation that 𝑡 ≥ 0. See Truncated Normal Continuous Distribution. Inflection Points: Inflection points occur one standard deviation away from the mean (𝜇 ± 𝜎). Mean / Median / Mode: The mean, median and mode are always equal to 𝜇.

Normal

Hazard Rate. The hazard rate is increasing for all 𝑡. The Standard Normal Distribution’s hazard rate approaches ℎ(𝑡) = 𝑡 as 𝑡 becomes large. Let:

Convolution Property 𝑛

Scaling Property

X~𝑁𝑓𝑠𝑚(µ, σ2 )

� 𝑋𝑖 ~𝑁𝑓𝑠𝑚 �� 𝜇𝑖 , ∑𝜎𝑖2 � 𝑖=1

𝑠𝑋 + 𝑏~𝑁𝑓𝑠𝑚(𝑠𝜇 + 𝑏, 𝑠2 𝜎 2 ) Linear Combination Property: 𝑛

� 𝑠𝑖 𝑋𝑖 + 𝑏𝑖 ~𝑁𝑓𝑠𝑚 ��{𝑠𝑖 𝜇𝑖 + 𝑏𝑖 } , ∑�𝑠𝑖2 𝜎𝑖2 �� 𝑖=1

Applications

Approximations to Other Distributions. The origin of the Normal Distribution was from an approximation of the Binomial distribution. Due to the Central Limit Theory the Normal distribution can be used to approximate many distributions as detailed under ‘Related Distributions’. Strength Stress Interference. When the strength of a component follows a distribution and the stress that component is subjected to follows a distribution there exists a probability that the stress will be greater than the strength. When both distributions are a normal distribution, there is a closed for solution to the interference

Normal Continuous Distribution

123

probability. Life Distribution. When used as a life distribution a truncated Normal Distribution may be used due to the constraint 𝑡 ≥ 0. However it is often found that the difference in results is negligible. (Rausand & Høyland 2004) Time Distributions. The normal distribution may be used to model simple repair or inspection tasks that have a typical duration with variation which is symmetrical about the mean. This is typical for inspection and preventative maintenance times. Analysis of Variance (ANOVA). A test used to analyze variance and dependence of variables. A popular model used to conduct ANOVA assumes the data comes from a normal population.

Online: http://www.weibull.com/LifeDataWeb/the_normal_distribution.htm http://mathworld.wolfram.com/NormalDistribution.html http://en.wikipedia.org/wiki/Normal_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (web calc)

Books: Patel, J.K. & Read, C.B., 1996. Handbook of the Normal Distribution 2nd ed., CRC. Simon, M.K., 2006. Probability Distributions Involving Gaussian Random Variables: A Handbook for Engineers and Scientists, Springer. Relationship to Other Distributions Truncated Normal Distribution

Let:

𝑇𝑁𝑓𝑠𝑚(𝑚; 𝜇, 𝜎, 𝑠𝐿 , 𝑏𝑈 )

Then:

LogNormal Distribution

Let:

𝐿𝑓𝑅𝑁(𝑡; 𝜇𝑁 , 𝜎𝑁2 )

Then: Where:

𝑋~𝑁𝑓𝑠𝑚(𝜇, 𝜎 2 ) 𝑋 ∈ (∞, ∞)

𝑌~T𝑁𝑓𝑠𝑚(𝜇, 𝜎 2 , 𝑠𝐿 , 𝑏𝑈 ) 𝑌 ∈ [𝑠𝐿 , 𝑏𝑈 ] 𝑋~𝐿𝑓𝑅𝑁(𝜇𝑁 , σ2N ) 𝑌 = ln(𝑋) 𝑌~𝑁𝑓𝑠𝑚(𝜇, 𝜎 2 )

Normal

Resources

Six Sigma Quality Management. Six sigma is a business management strategy which aims to reduce costs in manufacturing processes by removing variance in quality (defects). Current manufacturing standards aim for an expected 3.4 defects out of one million parts: 2Φ(−6). (Six Sigma Academy 2009)

124

Univariate Continuous Distributions

Rayleigh Distribution 𝑅𝑠𝑦𝑠𝑅𝑠𝑅ℎ(𝑡; 𝜎)

𝜇2 𝜇𝑁 = ln � �, �𝜎 2 + 𝜇2

Let

Then Let

Then

𝑋𝑖 ~𝑁𝑓𝑠𝑚(µ, σ

2)

𝑠𝑓𝑝

Normal

𝐵𝑠𝑓𝑓𝑚(𝑘; 𝑓, 𝑝)

Poisson Distribution 𝑃𝑓𝑠𝑠(k; µ)

Y~𝜒 2 (𝑡; 𝑠)

Y = �X12 + X22 v

Xk − µ 2 Y = �� � σ k=1

Limiting Case for constant 𝑝 : lim 𝐵𝑠𝑓𝑓𝑚(𝑘; 𝑓, 𝑝) = 𝑁𝑓𝑠𝑚�k; µ = n𝑝, 𝜎 2 = 𝑓𝑝(1 − 𝑝)� 𝑛→∞ R

Binomial Distribution

𝑠𝑓𝑝

Y~𝑅𝑠𝑦𝑠𝑅𝑠𝑅ℎ(𝜎)

Chi-square Distribution 𝜒 2 (𝑡; 𝑠)

𝑋1 , 𝑋2 ~𝑁𝑓𝑠𝑚(0, σ)

𝜎 2 + 𝜇2 𝜎𝑁 = �ln � � 𝜇2

𝑝=𝑝

The Normal distribution can be used as an approximation of the Binomial distribution when 𝑓𝑝 ≥ 10 and 𝑓𝑝(1 − 𝑝) ≥ 10. 𝐵𝑠𝑓𝑓𝑚(𝑘; 𝑝, 𝑓) ≈ 𝑁𝑓𝑠𝑚�𝑡 = 𝑘 + 0.5; 𝜇 = 𝑓𝑝, 𝜎 2 = 𝑓𝑝(1 − 𝑝)� lim 𝐹𝑃𝑜𝑖𝑠 (𝑘; 𝜇) = 𝐹𝑁𝑜𝑟𝑚 (𝑘; 𝜇′ = 𝜇, 𝜎 = �𝜇)

𝜇→∞

This is a good approximation when 𝜇 > 1000. When 𝜇 > 10 the same approximation can be made with a correction: lim 𝐹𝑃𝑜𝑖𝑠 (𝑘; 𝜇) = 𝐹𝑁𝑜𝑟𝑚 (𝑘; 𝜇′ = 𝜇 − 0.5, 𝜎 = �𝜇)

𝜇→∞

For large 𝛼 and 𝛽 with fixed 𝛼/𝛽 : R

Beta Distribution 𝐵𝑅𝑡𝑠(𝑡; 𝛼, 𝛽) Gamma Distribution 𝐺𝑠𝑚𝑚𝑠(𝑘, 𝜆)

𝐵𝑅𝑡𝑠(𝛼, 𝛽) ≈ 𝑁𝑓𝑠𝑚 �µ =

α αβ ,𝜎 = � � 2 (α (α α+β + β) + β + 1)

As 𝛼 and 𝛽 increase the mean remains constant and the variance is reduced. Special Case for large k: lim 𝐺𝑠𝑚𝑚𝑠(𝑘, 𝜆) = 𝑁𝑓𝑠𝑚 �𝜇 =

𝐸→∞

𝑘 𝑘 , 𝜎 = � 2� 𝜆 𝜆

Pareto Continuous Distribution

125

4.6. Pareto Continuous Distribution Probability Density Function - f(t) 1.0

θ=1 σ=1 θ=2 σ=1 θ=3 σ=1

0.8 0.6 0.4 0.2 0.0 0

2

4

6

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

8

θ=1 σ=1 θ=1 σ=2 θ=1 σ=4

0

2

4

6

8

Cumulative Density Function - F(t) 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

θ=1 σ=1 θ=2 σ=1 θ=3 σ=1

0

2

4

6

8

Hazard Rate - h(t) 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 2

4

6

θ=1 σ=1 θ=1 σ=2 θ=1 σ=4 0

2

4

6

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

θ=1 σ=1 θ=2 σ=1 θ=3 σ=1

0

Pareto

1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0

8

8

θ=1 σ=1 θ=1 σ=2 θ=1 σ=4

0

2

4

6

8

126

Univariate Continuous Distributions

Parameters & Description

Parameters

Limits

θ

θ>0

𝛼

α>0

Distribution

θ≤t< ∞ 𝑝(𝑡) =

αθα t α+1

θ α 𝐹(𝑡) = 1 − � � t

CDF

θ α R(t) = � � t

Reliability Pareto

Shape parameter. Sometimes called the Pareto index.

Formulas

PDF

Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡) Mean Life

Location parameter. θ is the lower limit of t. Sometimes refered to as tminimum.

Residual

𝑚(𝑚) = 𝑅(𝑚|𝑡) =

(t)α 𝑅(𝑡 + 𝑚) = 𝑅(𝑡) (t + x)α

Where 𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡. 𝑠(𝑡) =

∞

∫𝑡 𝑅(𝑚)𝑝𝑚 𝑅(𝑡)

ℎ(𝑡) =

Hazard Rate Cumulative Hazard Rate

α t

𝑡 𝐻(𝑡) = 𝛼 ln � � 𝜃

Properties and Moments Median Mode Mean - 1st Raw Moment Variance - 2nd Central Moment Skewness - 3rd Central Moment

θ21⁄α θ

αθ , for α > 1 α−1

αθ2 , for α > 2 (α − 1)2 (α − 2)

2(1 + α) α − 2 � , for α > 3 (α − 3) α

Pareto Continuous Distribution

127

6(α3 + α2 − 6α − 2) , for α > 4 α(α − 3)(α − 4)

Excess kurtosis - 4th Central Moment

𝛼(−𝑠𝜃𝑡)𝛼 Γ(−𝛼, −𝑠𝜃𝑡)

Characteristic Function

𝑡𝛾 = θ(1 − γ)−1⁄α

100𝛾 % Percentile Function Parameter Estimation Plotting Method Least Mean Square 𝑦 = 𝑚𝑚 + 𝑐

X-Axis

Likelihood Function

For complete data:

ln(𝑡𝑖 )

ln[1 − 𝐹]

Maximum Likelihood Function nF 1 𝐿(𝜃, 𝛼|𝐸) = αnF θαnF � α+1 ������������� i=1 t i failures

Log-Likelihood Function

𝛼� = −𝑚 𝑐 � 𝜃 = exp � � 𝛼�

𝑛𝐹

Λ(𝜃, 𝛼|𝐸) = nF ln(α) + nF αln(θ) − (𝛼 + 1) � ln 𝑡𝑖 ������������������������� 𝑖=1 failures

solve for 𝛼 to get 𝛼�:

𝑛𝐹

∂Λ nF = − + nF ln θ − � ln 𝑡𝑖 = 0 ∂α α ����������������� 𝑖=1 failures

MLE Point Estimates

The likelihood function increases as 𝜃 increases. Therefore the MLE point estimate is the largest 𝜃 which satisfies θ ≤ t i < ∞: 𝜃� = min�t1 , … , t nF �

Substituting 𝜃� gives the MLE for 𝛼�: 𝛼� =

Fisher Information

100𝛾% Confidence Intervals

𝑓𝐹 𝐹 ∑𝑛𝑖=1 �ln 𝑡𝑖 − ln𝜃��

𝐼(𝜃, 𝛼) = � 1 Sided - Lower 𝛼�

if 𝜃 is

unknown

𝛼� 2 𝜒 (2n − 2) 2nF {1−γ}

−1/𝛼 2 0

0 � 1/𝜃 2

2 Sided - Lower

𝛼� 2 𝜒 1−γ (2n − 2) 2nF � 2 �

2 Sided - Upper 𝛼� 2 𝜒 1+γ (2n − 2) 2nF � 2 �

Pareto

∂Λ =0 ∂α

Y-Axis

128

Univariate Continuous Distributions

(for complete data)

𝛼� 2 𝜒 (2n) 2nF {1−γ}

𝛼�

if 𝜃 is

known

𝛼� 2 𝜒 1−γ (2n) 2nF � 2 �

𝛼� 2 𝜒 1+γ (2n − 2) 2nF � 2 �

(Johnson et al. 1994, p.583) Where 𝜒𝛾2 (𝑓) is the 100𝛾 th percentile of the 𝜒 2 -distribution with 𝑓 degrees of freedom. P

Bayesian

Non-informative Priors when 𝜽 is known, 𝝅𝟎 (𝜶) (Yang and Berger 1998, p.22)

Type

Prior

Jeffery Reference

and

Pareto

Conjugate Priors UOI

Likelihood Model

Evidence

𝑏 from

𝑈𝑓𝑠𝑝(𝑡; a, 𝑏)

Uniform with known a

𝑓𝐹 failures at times 𝑡𝑖

𝑃𝑠𝑠𝑅𝑡𝑓(𝑡; θ, 𝛼)

Pareto with known 𝛼

𝑓𝐹 failures at times 𝑡𝑖

Pareto with known 𝜃

𝑓𝐹 failures at times 𝑡𝑖

𝜃 from

𝛼 from

𝑃𝑠𝑠𝑅𝑡𝑓(𝑡; θ, 𝛼)

1 𝛼

Dist of UOI

Prior Para

Pareto

𝜃𝑜 , 𝛼0

Pareto

Gamma

a 0 , Θ0

k 0 , λ0

Description , Limitations and Uses Example

Posterior Parameters 𝜃 = max{𝑡1 , … , 𝑡𝑛𝐹 } 𝛼 = 𝛼0 + 𝑓𝐹

a = a𝑜 − 𝛼𝑓𝐹 where a0 > 𝛼𝑓𝐹 Θ = Θ0

k = k 𝑜 + 𝑓𝐹 𝑛𝐹

𝑚𝑖 λ = λ𝑜 + � ln � � 𝜃 𝑖=1

5 components are put on a test with the following failure times: 108, 125, 458, 893, 13437 hours MLE Estimates are: 𝜃� = 108

Substituting 𝜃� gives the MLE for 𝛼�: 𝛼� =

5 = 0.8029 𝐹 ∑𝑛𝑖=1 (ln 𝑡𝑖 − ln(108))

90% confidence interval for 𝛼�:

Pareto Continuous Distribution

� Characteristics

𝛼 �

10

𝜒2 {0.05} (8),

𝛼 �

10

129

𝜒2 {0.95} (8)�

[0.2194, 1.2451]

80/20 Rule. Most commonly described as the basis for the “80/20 rule” (In a quality context, for example, 80% of manufacturing defects will be a result from 20% of the causes). Conditional Distribution. The conditional probability distribution given that the event is greater than or equal to a value 𝜃1 exceeding 𝜃 is a Pareto distribution with the same index 𝛼 but with a minimum 𝜃1 instead of 𝜃.

Types. This distribution is known as a Pareto distribution of the first kind. The Pareto distribution of the second kind (not detailed here) is also known as the Lomax distribution. Pareto also proposed a third distribution now known as a Pareto distribution of the third kind.

Let:

Minimum property

Applications

Resources

For constant 𝜃.

Xi ~𝑃𝑠𝑠𝑅𝑡𝑓(θ, αi )

𝑛

min{𝑋, 𝑋2 , … , 𝑋𝑛 }~𝑃𝑠𝑠𝑅𝑡𝑓 �𝜃, � 𝛼𝑖 � 𝑖=1

Rare Events. The survival function ‘slowly’ decreases compared to most life distributions which makes it suitable for modeling rare events which have large outcomes. Examples include natural events such as the distribution of the daily rain fall, or the size of manufacturing defects. Online: http://mathworld.wolfram.com/ParetoDistribution.html http://en.wikipedia.org/wiki/Pareto_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (web calc) Books: Arnold, B., 1983. Pareto distributions, Fairland, MD: International Co-operative Pub. House. Johnson, N.L., Kotz, S. & Balakrishnan, N., 1994. Continuous Univariate Distributions, Vol. 1 2nd ed., Wiley-Interscience.

Pareto

Pareto and the Lognormal Distribution. The Lognormal distribution models similar physical phenomena as the Pareto distribution. The two distributions have different weights at the extremities.

130

Univariate Continuous Distributions

Relationship to Other Distributions Exponential Distribution 𝐸𝑚𝑝(𝑡; 𝜆) Chi-Squared Distribution χ2 (𝑚; 𝑠)

Pareto

Logistic Distribution 𝐿𝑓𝑅𝑠𝑠𝑡𝑠𝑐(µ, 𝑠)

Let Then

𝑌~𝑃𝑠𝑠𝑅𝑡𝑓(𝜃, 𝛼)

Let Then

𝑌~𝑃𝑠𝑠𝑅𝑡𝑓(𝜃, 𝛼)

𝑋~𝐸𝑚𝑝(𝜆 = 𝛼) 𝑠𝑓𝑝

𝑋~𝜒 2 (𝑠 = 2) (Johnson et al. 1994, p.526) Let

Then

𝑋~𝑃𝑠𝑠𝑅𝑡𝑓(𝜃, 𝛼)

𝑋 = ln(𝑌⁄𝜃 )

𝑠𝑓𝑝

𝑠𝑓𝑝

𝑋 = 2α ln(𝑌⁄𝜃 )

X α 𝑌 = − ln �� � − 1� θ

𝑌~𝐿𝑓𝑅𝑠𝑠𝑡𝑠𝑐(0,1) (Hastings et al. 2000, p.127)

Triangle Continuous Distribution

131

4.7. Triangle Continuous Distribution Probability Density Function - f(t) a=0 b=1 c=0.5 a=0 b=1.5 c=0.5 a=0 b=2 c=0.5

2.0 1.5 1.0 0.5 0.0 0

0.5

1

1.5

2.0 1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0

2

a=0 b=1 c=0.25 a=0 b=1 c=0.5 a=0 b=1 c=0.75

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

Cumulative Density Function - F(t) 1.0

0.8

0.8

0.6

0.6

0.4

Triangle

1.0

0.4 a=0 b=1 c=0.5 a=0 b=1.5 c=0.5 a=0 b=2 c=0.5

0.2 0.0

0.2 0.0

0

0.5

1

1.5

2

Hazard Rate - h(t) 10 9 8 7 6 5 4 3 2 1 0

10 9 8 7 6 5 4 3 2 1 0 0

0.5

1

1.5

2

a=0 b=1 c=0.25 a=0 b=1 c=0.5 a=0 b=1 c=0.75

0

0.5

1

132

Univariate Continuous Distributions

Parameters & Description

Parameters

Random Variable

𝑠

−∞ ≤ 𝑠 < 𝑏

𝑐

𝑠≤𝑐≤𝑏

𝑏

𝑠<𝑏<∞

Distribution

Minimum Value. 𝑠 is the lower bound

Maximum Value. 𝑏 is the upper bound.

Mode Value. 𝑐 is the mode of the distribution (top of the triangle).

𝑠≤𝑡≤𝑏

Formulas 2(t − a) ⎧ for a ≤ t ≤ c ⎪ (b − a)(c − a) 𝑝(𝑡) = ⎨ 2(b − t) ⎪(b − a)(b − c) for c ≤ t ≤ b ⎩

PDF

(t − a)2 ⎧ for a ≤ t ≤ c ⎪ (b − a)(c − a) 𝐹(𝑡) = 2 (b − t) ⎨ ⎪1 − (b − a)(b − c) for c ≤ t ≤ b ⎩

Triangle

CDF

Reliability

(t − a)2 ⎧1 − for a ≤ t ≤ c ⎪ (b − a)(c − a) 𝑅(𝑡) = 2 ⎨ (b − t) for c ≤ t ≤ b ⎪(b − a)(b − c) ⎩

Properties and Moments Median

1

𝑠 + �2(𝑏 − 𝑠)(𝑐 − 𝑠) 𝑝𝑓𝑠 𝑐 ≥ 1

𝑏 − �2(𝑏 − 𝑠)(𝑏 − 𝑐) 𝑝𝑓𝑠 𝑐 < Mode st

Mean - 1 Raw Moment Variance - 2nd Central Moment Skewness - 3rd Central Moment Excess kurtosis - 4th Central Moment

𝑐

𝑏−𝑠 2

𝑏−𝑠 2

𝑠+𝑏+𝑐 3

𝑠2 + 𝑏2 + 𝑐 2 − 𝑠𝑏 − 𝑠𝑐 − 𝑏𝑐 18

√2(𝑠 + 𝑏 − 2𝑐)(2𝑠 − 𝑏 − 𝑐)(𝑠 − 2𝑏 + 𝑐) 5(𝑠2 + 𝑏2 + 𝑐 2 − 𝑠𝑏 − 𝑠𝑐 − 𝑏𝑐)3/2 −3 5

Triangle Continuous Distribution

Characteristic Function

−2

100γ % Percentile Function

133

(b − c)eita − (b − a)eitc + (c − a)eitb (b − a)(c − a)(b − c)t 2

𝑡𝛾 = a + �γ(b − a)(c − a) 𝑝𝑓𝑠 𝛾 < 𝐹(𝑐)

𝑡𝛾 = b − �(1 − γ)(b − a)(b − c) 𝑝𝑓𝑠 𝛾 ≥ 𝐹(𝑐)

Parameter Estimation

Maximum Likelihood Function r nF 2(t i − a) 2(b − t i ) 𝐿(𝑠, 𝑏, 𝑐|𝐸) = � .� (b��� − a)(c −�� a) ��������������� ���i=1 ��� ���� i=r+1 (b − a)(b − c)

Likelihood Functions

=�

failers to the left of c r 2 𝑛𝐹 ti

b−a

�

Where failure times are ordered: 𝑇1 ≤ 𝑇2 ≤ ⋯ ≤ 𝑇𝑟 ≤ ⋯ ≤ 𝑇𝑛𝐹 and 𝑠 is the number of failure times less than 𝑐 and 𝑠 is the number of failure times greater than 𝑐. Therefore 𝑓𝐹 = 𝑠 + 𝑠. The MLE estimates a� , b� , and c� are obtained by numerically calculating the likelihood function for different r and selecting the maximum where c� = Xr� . where

max

𝑎≤𝑐≤𝑏

2 𝑛𝐹 � {𝑀(𝑠, 𝑏, 𝑠̂ (𝑠, 𝑏)} b−a

𝐿(𝑠, 𝑏, 𝑐|𝐸) = �

nF ti − a b − ti � i=1 (t r − a) i=r+1 (b − t r ) r−1

𝑀(𝑠, 𝑏, 𝑠) = �

𝑠(𝑠, 𝑏) = arg max 𝑀(𝑠, 𝑏, 𝑠) 𝑟∈{1,…,𝑛𝐹 }

Note that the MLE estimates for a and 𝑏 are not the same as the uniform distribution: a� ≠ min(t1F , t F2 … ) �b ≠ max(t1F , t F2 … ) (Kotz & Dorp 2004) Description , Limitations and Uses

Example

When eliciting an opinion from an expert on the possible value of a quantity, 𝑚, the expert may give : - Lowest possible value = 0 - Highest possible value = 1 - Estimate of most likely value (mode) = 0.7 The corresponding distribution for 𝑚 may be a triangle distribution with parameters: 𝑠 = 0, 𝑏 = 1, 𝑐 = 0.7

Triangle

Point Estimates

failures to the right of c

nF b − ti −a � � (b (c − c) − a) i=1 i=r+1

134

Univariate Continuous Distributions

Characteristics

Standard Triangle Distribution. The standard triangle distribution has 𝑠 = 0, 𝑏 = 1. This distribution has a mean at �𝑐 ⁄2 and median at 1 − �(1 − 𝑐)⁄2.

Symmetrical Triangle Distribution. The symmetrical triangle distribution occurs when 𝑐 = (𝑏 − 𝑠)/2. The symmetrical triangle distribution is formed from the average of two uniform random variables (see related distributions).

Applications

Subjective Representation. The triangle distribution is often used to model subjective evidence where 𝑠 and 𝑏 are the bounds of the estimation and 𝑐 is an estimation of the mode.

Triangle

Substitution to the Beta Distribution. Due to the triangle distribution having bounded support it may be used in place of the beta distribution. Monte Carlo Simulation. Used to approximate distributions of variables when the underlying distribution is unknown. A distribution of interest is obtained by conducting Monte Carlo simulation of a model using the triangle distributions as inputs. Resources

Online: http://mathworld.wolfram.com/TriangularDistribution.html http://en.wikipedia.org/wiki/Triangular_distribution Books: Kotz, S. & Dorp, J.R.V., 2004. Beyond Beta: Other Continuous Families Of Distributions With Bounded Support And Applications, World Scientific Publishing Company. Relationship to Other Distributions Let

Uniform Distribution 𝑈𝑓𝑠𝑝(𝑡; a, b) Beta Distribution 𝐵𝑅𝑡𝑠(𝑡; α, 𝛽 )

Then

Xi ~𝑈𝑓𝑠𝑝(𝑠, 𝑏)

𝑠𝑓𝑝

Y~𝑇𝑠𝑠𝑠𝑓𝑅𝑠𝑅 �a, Special Cases:

Y=

b−a , b� 2

X1 + X 2 2

𝐵𝑅𝑡𝑠(1,2 ) = 𝑇𝑠𝑠𝑠𝑓𝑅𝑠𝑅(0,0,1) 𝐵𝑅𝑡𝑠(2,1 ) = 𝑇𝑠𝑠𝑠𝑓𝑅𝑠𝑅(0,1,1)

Truncated Normal Continuous Distribution

135

4.8. Truncated Normal Continuous Distribution Probability Density Function - f(t) μ=-1 σ=1 a=0 b=2 μ=0 σ=1 a=0 b=2 μ=1 σ=1 a=0 b=2

1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0 -1

0

1

2

μ=0.5 σ=0.5 μ=0.5 σ=1 μ=0.5 σ=3

1.8 1.6 1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0

3

-1

0

1

2

3

Cumulative Density Function - F(t) 1.0

0.8

0.8

0.6

0.6

0.4

0.4 μ=-1 a=0 b=2 μ=0 a=0 b=2 μ=1 a=0 b=2

0.2 0.0 -1

0

1

2

Trunc Normal

1.0

μ=0.5 σ=0.5 a=0 μ=0.5 σ=1 b=2 μ=0.5 σ=3

0.2 0.0

3

-1

0

1

Hazard Rate - h(t) μ=-1 σ=1 a=0 b=2 μ=0 σ=1 a=0 b=2 μ=1 σ=1 a=0 b=2

1.4 1.2 1.0

2

3

μ=0.5 σ=0.5 a=0 μ=0.5 σ=1 b=2 μ=0.5 σ=3

1.2 1.0 0.8

0.8

0.6

0.6 0.4

0.4

0.2

0.2

0.0

0.0 -1

0

1

2

3

-1

0

1

2

3

136

Univariate Continuous Distributions

Parameters & Description 𝜇

−∞ < µ < ∞

𝑠𝐿

−∞ < 𝑠𝐿 < 𝑏𝑈

𝑏𝑈

𝑠𝐿 < 𝑏𝑈 < ∞

𝜎2 Parameters

σ2 > 0

Limits

Trunc Normal

Distribution

Left Truncated Normal 𝑚 ∈ [0, ∞)

for 0 ≤ 𝑚 ≤ ∞ PDF

𝜙(zx ) 𝑝(𝑚) = σΦ(−z0 ) otherwise 𝑝(𝑚) = 0

Location parameter: The mean of the distribution. Scale parameter: The standard deviation of the distribution. Lower Bound: 𝑠𝐿 is the lower bound. The standard normal transform of 𝑠𝐿 𝑎 −𝜇 is 𝑧𝑎 = 𝐿 . 𝜎

Upper Bound: 𝑏𝑈 is the upper bound. The standard normal transform of 𝑏𝑈 𝑏 −𝜇 is 𝑧𝑏 = 𝑈𝜎 .

𝑠𝐿 < 𝑚 ≤ 𝑏𝑈

General Truncated Normal 𝑚 ∈ [𝑠𝐿 , 𝑏𝑈 ]

for 𝑠𝐿 ≤ 𝑚 ≤ 𝑏𝑈

1 𝜙(zx ) σ 𝑝(𝑚) = Φ(zb ) − Φ(za ) otherwise 𝑝(𝑚) = 0

where 𝜙 is the standard normal pdf with 𝜇 = 0 and 𝜎 2 = 1 Φ is the standard normal cdf with 𝜇 = 0 and 𝜎 2 = 1 𝑖−𝜇 𝑧𝑖 = � 𝜎 � for 𝑚 < 0 CDF

𝐹(𝑚) = 0

for 0 ≤ 𝑚 < ∞ 𝐹(𝑚) =

for 𝑚 < 0 Reliability

Φ(zx ) − Φ(z0 ) Φ(−z0 )

𝑅(𝑚) = 1

for 0 ≤ 𝑚 < ∞ 𝑅(𝑚) =

Φ(z0 ) − Φ(zx ) Φ(−z0 )

for 𝑚 < aL

𝐹(𝑚) = 0

for 𝑠𝐿 ≤ 𝑚 ≤ 𝑏𝑈 Φ(zx ) − Φ(za ) 𝐹(𝑚) = Φ(zb ) − Φ(za ) for 𝑚 > bU for 𝑚 < aL

𝐹(𝑚) = 1 𝑅(𝑚) = 1

for 𝑠𝐿 ≤ 𝑚 ≤ 𝑏𝑈 Φ(zb ) − Φ(zx ) 𝑅(𝑚) = Φ(zb ) − Φ(za ) for 𝑚 > bU

𝑅(𝑚) = 0

Truncated Normal Continuous Distribution

for 𝑡 < 0 𝑚(𝑚) = 𝑅(𝑡 + 𝑚)

for 𝑡 < aL 𝑚(𝑚) = 𝑅(𝑡 + 𝑚)

for 0 ≤ 𝑡 < ∞ Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡)

137

for 𝑠𝐿 ≤ 𝑡 ≤ 𝑏𝑈

𝑅(𝑡 + 𝑚) 𝑅(𝑡) 1 − Φ(zt+x ) = 1 − Φ(zt ) µ−x−t Φ� � σ = µ−t Φ� σ �

𝑚(𝑚) = 𝑅(𝑚|𝑡) =

𝑅(𝑡 + 𝑚) 𝑅(𝑡) Φ(zb ) − Φ(zt+x ) = Φ(zb ) − Φ(zt )

𝑚(𝑚) = 𝑅(𝑚|𝑡) =

for 𝑡 > bU

𝑚(𝑚) = 0

𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡. This operation is the equivalent of t replacing the lower bound. ∞

ℎ(𝑚) = 0

for 0 ≤ 𝑚 < ∞

1 𝜙(zx )[1 − Φ(zx )] ℎ(𝑚) = σ [1 − Φ(z0 )]2

Cumulative Hazard Rate Properties and Moments Median

𝐻(𝑡) = − ln[𝑅(𝑡)]

ℎ(𝑚) = 0

for 𝑠𝐿 ≤ 𝑚 ≤ 𝑏𝑈 1

ℎ(𝑚) = σ

for 𝑚 > bU

𝜙(zx )[Φ(zb ) − Φ(zx )] [Φ(zb ) − Φ(za )]2 ℎ(𝑚) = 0

𝐻(𝑡) = − ln[𝑅(𝑡)]

Left Truncated Normal 𝑚 ∈ [0, ∞)

General Truncated Normal 𝑚 ∈ [𝑠𝐿 , 𝑏𝑈 ]

𝜇 𝑤ℎ𝑅𝑠𝑅 𝜇 ≥ 0 0 𝑤ℎ𝑅𝑠𝑅 𝜇 < 0

𝜇 𝑤ℎ𝑅𝑠𝑅 𝜇 ∈ [𝑠𝐿 , 𝑏𝑈 ] 𝑠𝐿 𝑤ℎ𝑅𝑠𝑅 𝜇 < 𝑠𝐿 𝑏𝑈 𝑤ℎ𝑅𝑠𝑅 𝜇 > 𝑏𝑈

No closed form

Mode

Mean 1st Raw Moment where

Variance 2nd Central Moment

for 𝑚 < aL

µ+

𝜎𝜙(z0 ) Φ(−z0 )

−µ 𝑧0 = σ

σ2 [1 − {−Δ0 }2 − Δ1 ] where

No closed form

where

µ+𝜎

𝑧𝑎 = where

𝜙(za ) − 𝜙(zb ) Φ(zb ) − Φ(za )

𝑠𝐿 − µ , σ

𝑧𝑏 =

𝑏𝑈 − µ σ

σ2 [1 − {−Δ0 }2 − Δ1 ]

Trunc Normal

for 𝑚 < 0 Hazard Rate

∞

∫ 𝑅(𝑚)𝑝𝑚 ∫𝑡 𝑅(𝑚)𝑝𝑚 𝑠(𝑡) = 𝑡 = 𝑅(𝑡) 𝑅(𝑡)

Mean Residual Life

138

Univariate Continuous Distributions

Δ𝐸 =

𝑧0𝐸 𝜙(𝑧0 ) Φ(𝑧0 ) − 1

−1

Skewness 3rd Central Moment

3 𝑉2

where Excess kurtosis 4th Central Moment

Δ𝐸 =

𝑧𝑏𝐸 𝜙(𝑧𝑏 ) − 𝑧𝑎𝐸 𝜙(𝑧𝑎 ) Φ(𝑧𝑏 ) − Φ(𝑧𝑎 )

[2Δ30 + (3Δ1 − 1)Δ0 + Δ2 ] 𝑉 = 1 − Δ1 − Δ20

1 [−3Δ40 − 6Δ1 Δ20 − 2Δ20 − 4Δ2 Δ0 − 3Δ1 − Δ3 + 3] 𝑉2

Characteristic Function

See (Abadir & Magdalinos 2002, pp.1276-1287)

100𝛼% Percentile Function

𝑡𝛼 = µ + σΦ−1 {α + Φ(z0 )[1 − α]}

𝑡𝛼 = µ + σΦ−1 {αΦ(zb ) + Φ(za )[1 − α]}

Trunc Normal

Parameter Estimation

Maximum Likelihood Function Likelihood Function

For limits [𝑠𝐿 , 𝑏𝑈 ]: 𝐿(𝜇, 𝜎, 𝑠𝐿 , 𝑏𝑈 ) =

1

nF 1 𝑚𝑖 − 𝜇 2 � exp �− � � � 2 𝜎 i=1 �σ√2π{Φ(zb ) − Φ(za )}� ����������������������������������� nF

failures

1

For limits [0, ∞)

𝑛𝐹

1 �(𝑚𝑖 − 𝜇)2 � = nF exp �− 2 2𝜎 �σ√2π{Φ(zb ) − Φ(za )}� 𝑖=1 ����������������������������������� failures

𝐿(𝜇, 𝜎) =

1

nF 1 𝑚𝑖 − 𝜇 2 � exp �− � � � 2 𝜎 i=1 �Φ{−z 0 }σ√2π� �����������������������������

1

nF

failures

𝑛𝐹

1 = �(𝑚𝑖 − 𝜇)2 � nF exp �− 2 2𝜎 �Φ{−z0 }σ√2π� 𝑖=1 ����������������������������� failures

Log-Likelihood Function

For limits [𝑠𝐿 , 𝑏𝑈 ]: Λ(𝜇, 𝜎, 𝑠𝐿 , 𝑏𝑈 |𝐸)

𝑛𝐹

1 = −nF ln[Φ(zb ) − Φ(za )] − nF ln�σ√2π� − 2 �(𝑚𝑖 − 𝜇)2 2𝜎 ��������������������������������������� 𝑖=1

For limits [0, ∞)

failures

Truncated Normal Continuous Distribution

139

𝑛𝐹

1 Λ(𝜇, 𝜎|𝐸) = −𝑓𝐹 ln(𝛷{−𝑧0 }) − 𝑓𝐹 ln�𝜎√2𝜋� − 2 �(𝑚𝑖 − 𝜇)2 2𝜎 ����������������������������������� 𝑖=1 failures

𝑛𝐹

∂Λ =0 ∂µ

∂Λ −nF ϕ(za ) − ϕ(zb ) 1 = � � + 2 �(𝑚𝑖 − 𝜇) = 0 ) ) ∂µ σ Φ(zb − Φ(za 𝜎 ������������������������� 𝑖=1 failures

𝑛𝐹

∂Λ =0 ∂σ

∂Λ −nF za ϕ(za ) − zb ϕ(zb ) nF 1 = 2 � � − + 3 �(𝑚𝑖 − 𝜇)2 = 0 ∂σ σ Φ(zb ) − Φ(za ) σ 𝜎 ��������������������������������� 𝑖=1

MLE Point Estimates

First Estimate the values for 𝑧𝑎 and 𝑧𝑏 by solving the simultaneous equations numerically (Cohen 1991, p.33):

failures

Where:

𝐻2 (𝑧𝑎 , 𝑧𝑏 ) = 𝑄𝑎 =

𝑄𝑎 − 𝑄𝑏 − 𝑧𝑎 𝑚̅ − 𝑠𝐿 = 𝑧𝑏 − 𝑧𝑎 𝑏𝑈 − 𝑠𝐿

1 + 𝑧𝑎 𝑄𝑎 − 𝑧𝑏 𝑄𝑏 − (𝑄𝑎 − 𝑄𝑏 )2 𝑠2 = 2 (𝑧𝑏 − 𝑧𝑎 ) (𝑏𝑈 − 𝑠𝐿 )2

𝜙(𝑧𝑎 ) 𝜙(𝑧𝑏 ) , 𝑄𝑏 = Φ(𝑧𝑏 ) − Φ(𝑧𝑎 ) Φ(𝑧𝑏 ) − Φ(𝑧𝑎 ) 𝑧𝑎 = 𝑛𝐹

𝑠𝐿 − 𝜇 , 𝜎

𝑧𝑏 =

𝑏𝑈 − 𝜇 𝜎 𝑛𝐹

1 1 �(𝑚𝑖 − 𝑚̅ )2 𝑚̅ = 𝐹 � 𝑚𝑖 , 𝑠 2 = 𝑓𝐹 − 1 𝑓 0

0

The distribution parameters can then be estimated using: 𝑏𝑈 − 𝑠𝐿 , 𝜇̂ = 𝑠𝐿 − 𝜎�𝑧�𝑎 𝜎� = 𝑧𝑏 − 𝑧�𝑎 �

(Cohen 1991, p.44) provides a graphical procedure to estimate parameters to use as the starting point for numerical solvers. For the case where the limits are [0, ∞) first numerically solve for 𝑧0 : 1 − Q 0 (Q 0 − 𝑧0 ) 𝑠 2 = (Q 0 − 𝑧0 )2 𝑚̅ where 𝜙(𝑧0 ) 𝑄0 = 1 − Φ(𝑧0 ) The distribution parameters can be estimated using:

Trunc Normal

𝐻1 (𝑧𝑎 , 𝑧𝑏 ) =

140

Univariate Continuous Distributions

𝜎� =

𝑚̅ , 𝑄0 − 𝑧�0

𝜇̂ = −𝜎�𝑧�0

When the limits 𝑠𝐿 and 𝑏𝑈 are unknown, the likelihood function is maximized when the difference, Φ(zb ) − Φ(za ), is at its minimum. This occurs when the difference between 𝑏𝑈 − 𝑠𝐿 is at its minimum. Therefore the MLE estimates for 𝑠𝐿 and 𝑏𝑈 are:

Fisher Information (Cohen 1991, p.40)

a�L = min(t1F , t F2 … ) b�U = max(t1F , t F2 … )

𝐼(𝜇, 𝜎

Trunc Normal

Where

100𝛾% Confidence Intervals

2)

1 ⎡ [1 − 𝑄𝑎′ + 𝑄𝑏′ ] ⎢ 𝜎2 =⎢ 1 2(𝑚̅ − 𝜇) ⎢ � − 𝜆𝑎 + 𝜆𝑏 � 𝜎 ⎣𝜎 2

𝑄𝑎′ = 𝑄𝑎 (𝑄𝑎 − 𝑧𝑎 ), 𝜆𝑎 = 𝑠𝐿 𝑄′𝑎 + 𝑄𝑎 , 𝜂𝑎 = 𝑠𝐿 (𝜆𝑎 + 𝑄𝑎 ),

1 2(𝑚̅ − 𝜇) ⎤ � − 𝜆𝑎 + 𝜆𝑏 � 𝜎2 𝜎 ⎥ ⎥ 1 3[𝑠 2 + (𝑚̅ − 𝜇)2 ] � − 1 − 𝜂𝑎 + 𝜂𝑏 �⎥ 𝜎2 𝜎2 ⎦

𝑄𝑏′ = −𝑄𝑏 (𝑄𝑏 + 𝑧𝑏 ) 𝜆𝑏 = 𝑏𝑈 𝑄′𝑏 + 𝑄𝑏 𝜂𝑏 = 𝑏𝑈 (𝜆𝑏 + 𝑄𝑏 )

Calculated from the Fisher information matrix. See section 1.4.7. For further detail and examples see (Cohen 1991, p.41) Bayesian No closed form solutions to priors exist. Description , Limitations and Uses

Example 1

The size of washers delivered from a manufacturer is desired to be modeled. The manufacture has already removed all washers below 15.95mm and washers above 16.05mm. The washers received have the following diameters: 15.976, 15.970, 15.955, 16.007, 15.966, 15.952, 15.955 𝑚𝑚 From data: 𝑚̅ = 15.973, 𝑠 2 = 4.3950𝐸-4

Using numerical solver MLE Estimates for 𝑧𝑎 and 𝑧𝑏 are: Therefore

𝑧�𝑎 = 0,

𝜎� =

𝑧𝑏 = 3.3351 �

𝑏𝑈 − 𝑠𝐿 = 0.029984 𝑧𝑏 − 𝑧�𝑎 �

𝜇̂ = 𝑠𝐿 − 𝜎�𝑧�𝑎 = 15.95

To calculate confidence intervals, first calculate:

Truncated Normal Continuous Distribution

𝑄𝑎′ = 0.63771, 𝜆𝑎 = 10.970, 𝜂𝑎 = 187.71,

90% confidence intervals:

𝐼(𝜇, 𝜎) = �

141

𝑄𝑏′ = −0.010246 𝜆𝑏 = −0.16138 𝜂𝑏 = −2.54087

391.57 −10699

−10699 � −209183

[𝐽𝑛 (𝜇̂ , 𝜎�)]−1 = [𝑓𝐹 𝐼(𝜇̂ , 𝜎�)]−1 = � 1.1835𝐸-4 −6.0535𝐸-6� −6.0535𝐸-6 −2.2154𝐸-7

90% confidence interval for 𝜇: 𝜇̂ + Φ−1 (0.95)√1.1835𝐸-4� �𝜇̂ − Φ−1 (0.95)√1.1835𝐸-4, [15.932, 15.968]

𝛷 −1 (0.95)√2.2154𝐸-7 �� 𝜎� 3.0769𝐸-2] 𝜎�. exp �

An estimate can be made on how many washers the manufacturer discards: The distribution of washer sizes is a Normal Distribution with estimated parameters 𝜇̂ = 15.95, 𝜎� = 0.029984. The percentage of washers wish pass quality control is: 𝐹(16.05) − 𝐹(15.95) = 49.96%

It is likely that there is too much variance in the manufacturing process for this system to be efficient. Example 2

The following example adjusts the calculations used in the Normal Distribution to account for the fact that the limit on distance is [0, ∞).

The accuracy of a cutting machine used in manufacturing is desired to be measured. 5 cuts at the required length are made and measured as: 7.436, 10.270, 10.466, 11.039, 11.854 𝑚𝑚

From data:

𝑚̅ = 10.213,

𝑠 2 = 2.789

Using numerical solver MLE Estimates for 𝑧0 is: Therefore 𝜎� =

𝑧�0 = −4.5062

𝑚̅ = 2.26643 𝑄0 − 𝑧�0

Trunc Normal

90% confidence interval for 𝜎: 𝛷 −1 (0.95)√2.2154𝐸-7 �𝜎�. exp � �, −𝜎� [2.922𝐸-2,

142

Univariate Continuous Distributions

𝜇̂ = −𝜎�𝑧�𝑎 = 10.213

To calculate confidence intervals, first calculate: 𝑄0′ = 7.0042𝐸-5, 𝜆0 = 1.5543𝐸-5, 𝜆𝑏 = −0.16138 90% confidence intervals: 𝐼(𝜇, 𝜎) = �

0.19466 −2.9453𝐸-6 � −2.9453𝐸-6 0.12237

2.4728𝐸-5 [𝐽𝑛 (𝜇̂ , 𝜎�)]−1 = [𝑓𝐹 𝐼(𝜇̂ , 𝜎�)]−1 = � 1.0274 � 2.4728𝐸-5 1.6343

90% confidence interval for 𝜇: �𝜇̂ − Φ−1 (0.95)√1.0274, 𝜇̂ + Φ−1 (0.95)√1.0274� [8.546, 11.88]

Trunc Normal

90% confidence interval for 𝜎: 𝛷 −1 (0.95)√1.6343 �𝜎�. exp � �, −𝜎� [0.8962,

𝜎�. exp � 5.732]

𝛷 −1 (0.95)√1.6343 �� 𝜎�

To compare these results to a non-truncated normal distribution: 90% Lower CI Point Est 90% Upper CI 10.163 10.213 10.262 Norm - 𝜇 Classical 1.176 2.789 15.697 Norm - 𝜎 2 Classical 8.761 10.213 11.665 Norm - 𝜇 Bayesian 0.886 2.789 6.822 Norm - 𝜎 2 Bayesian 8.546 10.213 11.88 TNorm - 𝜇 0.80317 5.1367 32.856 TNorm - 𝜎 2 *Note: The TNorm 𝜎 estimate and interval are squared.

The truncated normal produced results which had a wider confidence in the parameter estimates, however the point estimates were within each others confidence intervals. In this case the truncation correction might be ignored for ease of calculation. Characteristics

For large 𝜇/𝜎 truncation may have negligible affect. In this case the use the Normal Continuous Distribution as an approximation. Let:

X~𝑇𝑁𝑓𝑠𝑚(µ, σ2 ) where 𝑋 ∈ [𝑠, 𝑏]

Convolution Property. The sum of truncated normal distribution random variables is not a truncated normal distribution. When truncation is symmetrical about the mean the sum of truncated

Truncated Normal Continuous Distribution

143

normal distribution random variables is well approximated using: 𝑛 bi − ai 𝑌 = � 𝑋𝑖 where = µi 2 𝑖=1

𝑌 ≈ 𝑇𝑁𝑓𝑠𝑚 �� 𝜇𝑖 , � 𝑉𝑠𝑠(𝑋𝑖 )�

where 𝑌 ∈ [ ∑𝑠𝑖 , ∑𝑏𝑖 ]

Linear Transformation Property (Cozman & Krotkov 1997)

Applications

𝑌 = 𝑐𝑋 + 𝑝 𝑌~𝑇𝑁𝑓𝑠𝑚(𝑐𝜇 + 𝑝, 𝑝 2 𝜎 2 ) where 𝑌 ∈ [𝑐𝑠 + 𝑝, 𝑐𝑏 + 𝑝]

Life Distribution. When used as a life distribution a truncated Normal Distribution may be used due to the constraint t≥0. However it is often found that the difference in results is negligible. (Rausand & Høyland 2004)

Failures After Pre-test Screening. When a customer receives a product from a vendor, the product may have already been subject to burn-in testing. The customer will not know the number of failures which occurred during the burn-in, but may know the duration. As such the failure distribution is left truncated. (Meeker & Escobar 1998, p.269) Flaws under the inspection threshold. When a flaw is not detected due to the flaw’s amplitude being less than the inspection threshold the distribution is left truncated. (Meeker & Escobar 1998, p.266) Worst Case Measurements. Sometimes only the worst performers from a population are monitored and have data collected. Therefore the threshold which determined that the item be monitored is the truncation limit. (Meeker & Escobar 1998, p.267) Screening Out Units With Large Defects. In quality control processes it may be common to remove defects which exceed a limit. The remaining population of defects delivered to the customer has a right truncated distribution. (Meeker & Escobar 1998, p.270) Resources

Online: http://en.wikipedia.org/wiki/Truncated_normal_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (web calc) http://www.ntrand.com/truncated-normal-distribution/

Trunc Normal

Repair Time Distributions. The truncated normal distribution may be used to model simple repair or inspection tasks that have a typical duration with little variation using the limits [0, ∞)

144

Univariate Continuous Distributions

Books: Cohen, 1991. Truncated and Censored Samples 1st ed., CRC Press. Patel, J.K. & Read, C.B., 1996. Handbook of the Normal Distribution 2nd ed., CRC. Schneider, H., 1986. Truncated and censored samples from normal populations, M. Dekker. Relationship to Other Distributions Normal Distribution

Let:

𝑁𝑓𝑠𝑚(𝑚; 𝜇, 𝜎 2 )

Then:

𝑋~𝑁𝑓𝑠𝑚(𝜇, 𝜎 2 ) 𝑋 ∈ (∞, ∞)

𝑌~T𝑁𝑓𝑠𝑚(𝜇, 𝜎 2 , 𝑠𝐿 , 𝑏𝑈 ) 𝑌 ∈ [𝑠𝐿 , 𝑏𝑈 ]

Trunc Normal

For further relationships see Normal Continuous Distribution

Uniform Continuous Distribution

145

4.9. Uniform Continuous Distribution Probability Density Function - f(t) a=0 b=1 a=0.75 b=1.25 a=0.25 b=1.5

2.00 1.50 1.00 0.50 0.00 0

0.5

1

1.5

1

1.5

1

1.5

Uniform Cont

Cumulative Density Function - F(t) a=0 b=1 a=0.75 b=1.25 a=0.25 b=1.5

1.00 0.80 0.60 0.40 0.20 0.00 0

0.5

Hazard Rate - h(t) 10.00 9.00 8.00 7.00 6.00 5.00 4.00 3.00 2.00 1.00 0.00

a=0 b=1 a=0.75 b=1.25 a=0.25 b=1.5

0

0.5

146

Univariate Continuous Distributions

Parameters & Description

Parameters

Random Variable

𝑠 𝑏

Distribution

Minimum Value. 𝑠 is the lower bound of the uniform distribution.

0≤𝑠<𝑏

Maximum Value. 𝑏 is the upper bound of the uniform distribution.

𝑠<𝑏<∞ Time Domain

𝑠≤𝑡≤𝑏

1 𝑝(𝑡) = �b − a for a ≤ t ≤ b 0 otherwise 1 {𝑠(𝑡 − 𝑠) − 𝑠(𝑡 − 𝑏)} = b−a

PDF

Uniform Cont

Where 𝑠(𝑡 − 𝑠) is the Heaviside step function. 0 for t < 𝑠 t−a 𝐹(𝑡) = � for a ≤ t ≤ b b−a 1 for t > 𝑏 t−a {𝑠(𝑡 − 𝑠) − 𝑠(𝑡 − 𝑏)} = b−a + 𝑠(𝑡 − 𝑏)

CDF

𝑅(𝑡) = �

Reliability

For 𝑡 < 𝑠:

Conditional Survivor Function 𝑃(𝑇 > 𝑚 + 𝑡|𝑇 > 𝑡)

Mean Life

Residual

1 for t < 𝑠 b−t for a ≤ t ≤ b b−a 0 for t > 𝑏

Laplace

𝑝(𝑠) =

𝑅 −𝑎𝑠 − 𝑅 −𝑏𝑠 𝑠(𝑏 − 𝑠)

𝐹(𝑠) =

𝑅 −𝑎𝑠 − 𝑅 −𝑏𝑠 𝑠 2 (𝑏 − 𝑠)

𝑅(𝑠) =

𝑅 −𝑏𝑠 − 𝑅 −𝑎𝑠 1 + 𝑠 2 (𝑏 − 𝑠) 𝑠

1 for t + x < 𝑠 𝑅(𝑡 + 𝑚) b − (t + x) 𝑚(𝑚) = =� for a ≤ t + x ≤ b 𝑅(𝑡) b−a 0 for t > 𝑏 For 𝑠 ≤ 𝑡 ≤ 𝑏: 1 for t + x < 𝑠 𝑅(𝑡 + 𝑚) b − (t + x) 𝑚(𝑚) = =� for a ≤ t + x ≤ b 𝑅(𝑡) b−t 0 for t + x > 𝑏 For 𝑡 > 𝑏: 𝑚(𝑚) = 0 Where 𝑡 is the given time we know the component has survived to. 𝑚 is a random variable defined as the time after 𝑡. Note: 𝑚 = 0 at 𝑡. For 𝑡 < 𝑠:

For 𝑠 ≤ 𝑡 ≤ 𝑏:

1

𝑠(𝑡) = 2(𝑠 + 𝑏) − 𝑡

𝑠(𝑡) = 𝑠 − 𝑡 −

(𝑠 − 𝑏)2 2(𝑡 − 𝑏)

Uniform Continuous Distribution

For 𝑡 > 𝑏:

147

𝑠(𝑡) = 0

1 ℎ(𝑡) = �b − t for a ≤ t ≤ b 0 otherwise

Hazard Rate

0

for t < 𝑠 b−t 𝐻(𝑡) = � − ln � � for a ≤ t ≤ b b−a ∞ for t > 𝑏

Cumulative Hazard Rate

Properties and Moments Median

1 (𝑠 2

+ 𝑏)

1 (𝑠 2

+ 𝑏)

Any value between 𝑠 and 𝑏

Mode st

Mean - 1 Raw Moment

1 (𝑏 12

Skewness - 3rd Central Moment

− 𝑠)2

0

6

th

−5

Excess kurtosis - 4 Central Moment

eitb − eita it(b − a)

Characteristic Function 100α% Percentile Function Parameter Estimation

𝑡𝛼 = α(b − a) + a

Maximum Likelihood Function Likelihood Functions

LI nS nI 1 nF b − t Si t RI i − ti 𝐿(𝑠, 𝑏|𝐸) = � � .� � � . � �1 + � �� b��� − a�� ���i=1 b�− a ��������������� b−a i=1 ���� ��� failures

survivors

interval failures

This assumes that all times are within the bound a, b.

Point Estimates

When there is only complete failure data: 1 nF � 𝐿(𝑠, 𝑏|𝐸) = � b−a where 𝑠 ≤ 𝑡𝑖 ≤ 𝑏

The likelihood function is maximized when a is large, b is small with the restriction that all times are between a and b. Thus: a� = min(t1F , t F2 … ) b� = max(t1F , t F2 … )

Uniform Cont

Variance - 2nd Central Moment

148

Univariate Continuous Distributions

When 𝑠 = 0 and 𝑏 is estimated with complete data the following estimates may be used where 𝑡𝑚𝑎𝑥 = max(t1F , t F2 … t Fn ). (Johnson et al. 1995, p.286) 1. MLE. b� = 𝑡𝑚𝑎𝑥 n+2 2. Min Mean Square Error. b� = n+1 𝑡𝑚𝑎𝑥 n+1 3. Unbiased Estimator. b� = 𝑡𝑚𝑎𝑥 4.

n

1 b� = 2 �n 𝑡𝑚𝑎𝑥

Closest Estimator.

Procedures for parameter estimating when there is censored data is detailed in (Johnson et al. 1995, p.286) −1 ⎡ (𝑠 − 𝑏)2 𝐼(𝑠, 𝑏) = ⎢ 1 ⎢ ⎣(𝑠 − 𝑏)2

Fisher Information

Uniform Cont

Bayesian

1 ⎤ (𝑠 − 𝑏)2 ⎥ −1 ⎥ (𝑠 − 𝑏)2 ⎦

The Uniform distribution is widely used in Bayesian methods as a non-informative prior or to model evidence which only suggests bounds on the parameter. Non-informative Prior. The Uniform distribution can be used as a non-informative prior. As can be seen below, the only affect the uniform prior has on Bayes equation is to limit the range of the parameter for which the denominator integrates over.

𝜋(𝜃|𝐸) =

1 � 𝐿(𝐸|𝜃) 𝑏−𝑠 = 𝑏 𝑏 1 ∫𝑎 𝐿(𝐸|𝜃) �𝑏 − 𝑠 � 𝑝𝜃 ∫𝑎 𝐿(𝐸|𝜃) 𝑝𝜃 𝐿(𝐸|𝜃) �

Parameter Bounds. This type of distribution allows an easy method to mathematically model soft data where only the parameter bounds can be estimated. An example is where uniform distribution can model a person’s opinion on the value 𝜃 where they know that it could not be lower than 𝑠 or greater than 𝑏, but is unsure of any particular value 𝜃 could take. Non-informative Priors Jeffrey’s Prior

1 𝑠−𝑏

Description , Limitations and Uses Example

For an example of the uniform distribution being used in Bayesian updating as a prior, 𝐵𝑅𝑡𝑠(1,1) see the binomial distribution. Given the following data calculate the MLE parameter estimates: 240, 585, 223, 751, 255 a� = 223

Uniform Continuous Distribution

Characteristics

149

b� = 751

The Uniform distribution is a special case of the Beta distribution when 𝛼 = 𝛽 = 1.

The uniform distribution has an increasing failure rate with lim𝑡→𝑏 ℎ(𝑡) = ∞.

The Standard Uniform Distribution has parameters 𝑠 = 0 and 𝑏 = 1. This results in 𝑝(𝑡) = 1 for 𝑠 ≤ 𝑡 ≤ 𝑏 and 0 otherwise. Uniformity Property If 𝑡 > 𝑠 and 𝑡 + 𝛥 < 𝑏 then:

𝑇~𝑈𝑓𝑠𝑝(𝑠, 𝑏) 𝑡+∆

1 ∆ 𝑝𝑚 = 𝑏 − 𝑠 𝑏 − 𝑠 𝑡 The probability that a random variable falls within any interval of fixed length is independent of the location, 𝑡, and is only dependent on the interval size, 𝛥. 𝑃(𝑡 → 𝑡 + ∆) = �

Applications

Residual Property If k is a real constant where 𝑠 < 𝑘 < 𝑏 then: Pr(𝑇|𝑇 > 𝑘)~𝑈𝑓𝑠𝑝(𝑠 = 𝑘, 𝑏)

Random Number Generator. The uniform distribution is widely used as the basis for the generation of random numbers for other statistical distributions. The random uniform values are mapped to the desired distribution by solving the inverse cdf. Bayesian Inference. The uniform distribution can be used ss a non-informative prior and to model soft evidence.

Resources

Special Case of Beta Distribution. In applications like Bayesian statistics the uniform distribution is used as an uninformative prior by using a beta distribution of 𝛼 = 𝛽 = 1.

Online: http://mathworld.wolfram.com/UniformDistribution.html http://en.wikipedia.org/wiki/Uniform_distribution_(continuous) http://socr.ucla.edu/htmls/SOCR_Distributions.html (web calc)

Books: Johnson, N.L., Kotz, S. & Balakrishnan, N., 1995. Continuous Univariate Distributions, Vol. 2 2nd ed., Wiley-Interscience. Relationship to Other Distributions Beta Distribution

Let

Uniform Cont

Variate Generation Property 𝐹 −1 (𝑠) = 𝑠(𝑏 − 𝑠) + 𝑠

150

Univariate Continuous Distributions

𝐵𝑅𝑡𝑠(𝑡; α, β, a, b)

Exponential Distribution

Uniform Cont

𝐸𝑚𝑝(𝑡; 𝜆)

Then

Xi ~𝑈𝑓𝑠𝑝(0,1)

𝑠𝑓𝑝

X1 ≤ X 2 ≤ ⋯ ≤ X n

Xr ~𝐵𝑅𝑡𝑠(𝑠, 𝑓 − 𝑠 + 1) Where 𝑓 and 𝑘 are integers.

Special Case: 𝐵𝑅𝑡𝑠(𝑡; 𝑠, 𝑏|𝛼 = 1, β = 1 ) = 𝑈𝑓𝑠𝑝(𝑡; 𝑠, 𝑏) Let

Then

𝑋~𝐸𝑚𝑝(𝜆)

𝑠𝑓𝑝

Y = exp(−𝜆𝑋)

𝑌~𝑈𝑓𝑠𝑝(0,1)

Discrete Distributions

151

5. Univariate Discrete Distributions

Univar Discrete

152

Discrete Distributions

5.1. Bernoulli Discrete Distribution Probability Density Function - f(k) 0.70

0.70

0.60

0.60

p=0.3

0.50

0.50

0.40

0.40

0.30

0.30

0.20

0.20

0.10

0.10

0.00

0.00 0

1

2

p=0.55

0

1

2

Cumulative Density Function - F(k) 1.00

1.00

0.80

0.80

Bernoulli

0.60

p=0.3

0.60

0.40

0.40

0.20

0.20

0.00

0.00 0

1

2

p=0.55

0

1

2

Hazard Rate - h(k) 1.00

1.00

0.80

0.80

0.60

p=0.3

0.60

0.40

0.40

0.20

0.20

0.00

p=0.55

0.00 0

1

2

0

1

2

Bernoulli Discrete Distribution

153

Parameters & Description 𝑝

Parameters Random Variable Question Distribution

0≤𝑝≤1

Bernoulli probability Probability of success.

parameter.

𝑘 ∈ {0, 1}

The probability of getting exactly 𝑘 (0 or 1) successes in 1 trial with probability p. Formulas

𝑝(𝑘) = pk (1 − p)1−k 1 − p for k = 0 =� p for k = 1

PDF

𝐹(𝑘) = (1 − p)1−k 1 − p for k = 0 =� 1 for k = 1

CDF

𝑅(𝑘) = 1 − (1 − p)1−k p for k = 0 =� 0 for k = 1

Reliability

1−p ℎ(𝑘) = � 1

Hazard Rate

Properties and Moments

𝑘0.5 = ‖𝑝‖ 𝑤ℎ𝑅𝑓 𝑝 ≠ 0.5 𝑘0.5 = {0,1} 𝑤ℎ𝑅𝑓 𝑝 = 0.5

Mode Mean - 1st Raw Moment nd

Central Moment q−p

rd

Skewness - 3 Central Moment

�pq

Excess kurtosis - 4th Central Moment Characteristic Function Parameter Estimation

𝑝

𝑝(1 − 𝑝)

𝑤ℎ𝑅𝑠𝑅 𝑞 = (1 − 𝑝)

6p2 − 6p + 1 p(1 − p)

(1 − p) + peit

Maximum Likelihood Function Likelihood Function

𝐿(𝑝|𝐸) = 𝑝∑𝐸𝑖 (1 − 𝑝)𝑛−∑𝐸𝑖

where 𝑓 is the number of Bernoulli trials 𝑘𝑖 ∈ {0,1}, and ∑𝑘𝑖 = ∑𝑛𝑖=1 𝑘𝑖

Bernoulli

Variance - 2

for k = 0 for k = 1

154

Discrete Distributions

𝑝L =0 𝑝p

solve for 𝑝 𝑝L = ∑k. p∑(ki )−1 (1 − p)n−∑ki − (n − ∑k)p∑ki (1 − p)n−1−∑ki = 0 𝑝p ∑k. p∑(ki )−1 (1 − p)n−∑ki = (n − ∑k i )p∑ki (1 − p)n−1−∑ki ∑k i . p−1 = (n − ∑k i )(1 − p)−1 (1 − p) n − ∑k i = p ∑k i p=

Fisher Information MLE Point Estimates

∑k i n

𝐼(𝑝) =

The MLE point estimate for p:

p� =

Fisher Information Confidence Intervals

1 𝑝(1 − 𝑝)

𝐼(𝑝) =

∑k n

1 𝑝(1 − 𝑝)

See discussion in binomial distribution.

Bernoulli

Bayesian

Type

Non-informative Priors for p, 𝝅(𝒑) (Yang and Berger 1998, p.6)

Prior

Posterior

Uniform Proper Prior with limits 𝑝 ∈ [𝑠, 𝑏]

1 𝑏−𝑠

Uniform Improper Proir with limits 𝑝 ∈ [0,1]

1 = 𝐵𝑅𝑡𝑠(𝑝; 1,1)

Jeffrey’s Prior Reference Prior MDIP Novick and Hall

1

1 1 = 𝐵𝑅𝑡𝑠 �𝑝; , � 2 2 �𝑝(1 − 𝑝) 𝑝

1.6186𝑝𝑝 (1 − 𝑝)1−𝑝

−1 (1

−1

− 𝑝)

= 𝐵𝑅𝑡𝑠(0,0)

Truncated Beta Distribution For a ≤ 𝑝 ≤ b 𝑐. 𝐵𝑅𝑡𝑠(𝑝; 1 + 𝑘, 2 − 𝑘) Otherwise 𝜋(𝑝) = 0

𝐵𝑅𝑡𝑠(𝑝; 1 + 𝑘, 2 − 𝑘)

1 𝐵𝑅𝑡𝑠 �𝑝; + 𝑘, 1.5 − 𝑘� 2 when 𝑝 ∈ [0,1]

Proper - No Closed Form 𝐵𝑅𝑡𝑠(𝑝; 𝑘, 1 − 𝑘) when 𝑝 ∈ [0,1]

Bernoulli Discrete Distribution

155

Conjugate Priors UOI 𝑝 from 𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(𝑘; 𝑝) Example

Likelihood Model Bernoulli

Evidence 𝑘 failures in 1 trail

Dist of UOI

Prior Para

Posterior Parameters

Beta

𝛼0 , 𝛽0

𝛼 = 𝛼𝑜 + 𝑘 𝛽 = 𝛽𝑜 + 1 − 𝑘

Description , Limitations and Uses

When a demand is placed on a machine it undergoes a Bernoulli trial with success defined as a successful start. It is known the probability of a successful start, 𝑝, equals 0.8. Therefore the probability the machine does not start. 𝑝(0) = 0.2. For an example with multiple Bernoulli trials see the binomial distribution.

Characteristics

A Bernoulli process is a probabilistic experiment that can have one of two outcomes, success (𝑘 = 1) with the probability of success is 𝑝, and failure (𝑘 = 0) with the probability of failure is 𝑞 ≡ 1 − 𝑝.

Single Trial. It’s important to emphasis that the Bernoulli distribution is for a single trial or event. The case of multiple Bernoulli trials with replacement is the binomial distribution. The case of multiple Bernoulli trials without replacement is the hypergeometric distribution.

n

� K i ~𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(Π𝑘; 𝑝 = Π𝑝𝑖 ) i=1

Applications

Used to model a single event which have only two outcomes. In reliability engineering it is most often used to model demands or shocks to a component where the component will fail with probability p. In practice it is rare for only a single event to be considered and so a binomial distribution is most often used (with the assumption of replacement). The conditions and assumptions of a Bernoulli trial however are used as the basis for each trial in a binomial distribution. See ‘Related Distributions’ and binomial distribution for more details.

Bernoulli

𝐾~𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(𝑘|𝑝) Maximum Property max{𝐾1 , 𝐾2 , … , 𝐾𝑛 }~𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(𝑘; 𝑝 = 1 − Π{1 − 𝑝𝑖 }) Minimum property min{𝐾1 , 𝐾2 , … , 𝐾𝑛 }~𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(𝑘; 𝑝 = Π𝑝𝑖 ) Product Property

156

Discrete Distributions

Resources

Online: http://mathworld.wolfram.com/BernoulliDistribution.html http://en.wikipedia.org/wiki/Bernoulli_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (web calc) Books: Collani, E.V. & Dräger, K., 2001. Binomial distribution handbook for scientists and engineers, Birkhäuser. Johnson, N.L., Kemp, A.W. & Kotz, S., 2005. Univariate Discrete Distributions 3rd ed., Wiley-Interscience. Relationship to Other Distributions The Binomial distribution counts the number of successes in n independent observations of a Bernoulli process. Let

Binomial Distribution 𝐵𝑠𝑓𝑓𝑚(𝑘′|n, p)

Then

𝐾𝑖 ~𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(k i ; p)

𝑠𝑓𝑝

𝑛

𝑌 = � 𝐾𝑖 𝑖=1

Y~𝐵𝑠𝑓𝑓𝑚(k ′ = ∑k i |n, p) where 𝑘′ ∈ {1, 2, … , 𝑓}

Special Case:

Bernoulli

𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(k; p) = 𝐵𝑠𝑓𝑓𝑚(k; p|n = 1)

Binomial Discrete Distribution

157

5.2. Binomial Discrete Distribution Probability Density Function - f(k) 0.60 0.50

n=1 p=0.4

0.30

n=5 p=0.4

0.25

n=15 p=0.4

0.40

0.20

0.30

0.15

0.20

0.10

0.10

0.05

0.00

0.00

0

5

n=20 p=0.2 n=20 p=0.5 n=20 p=0.8

10

0

5

10

15

20

Cumulative Density Function - F(k) 1.00

1.00

0.80

0.80

0.60

0.60

0.40

0.40 n=1 p=0.4 n=5 p=0.4 n=15 p=0.4

0.20

0.00 0

5

10

0

5

10

15

20

Hazard Rate - h(k) 1.0

1.0

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0 0

5

10

0

10

20

Binomial

0.00

0.20

158

Discrete Distributions

Parameters & Description

Parameters

Random Variable Question Distribution

PDF

𝑓 𝑝

𝑓 ∈ {1, 2 … , ∞} 0≤𝑝≤1

Number of Trials. Bernoulli probability parameter. Probability of success in a single trial.

𝑘 ∈ {0, 1, 2 … , 𝑓}

The probability of getting exactly 𝑘 successes in 𝑓 trials. Formulas

n 𝑝(𝑘) = � � pk (1 − p)n−k k where k combinations from n: 𝑓! 𝑓 𝑓 � � = 𝑛𝐶𝐸 = 𝐶𝐸𝑛 = = 𝐶 𝑛−1 𝑘 𝑘! (𝑓 − 𝑘)! 𝑘 𝐸−1 k

𝐹(𝑘) = � j=0

𝑓! pj (1 − p)n−j 𝑗! (𝑓 − 𝑗)!

= 𝐼1−𝑝 (𝑓 − 𝑘, 𝑘 + 1)

where 𝐼𝑝 (𝑠, 𝑏) is the Regularized Incomplete Beta function. See section 1.6.3. CDF

When 𝑓 ≥ 20 and 𝑝 ≤ 0.05, or if 𝑓 ≥ 100 and 𝑓𝑝 ≤ 10, this can be approximated by a Poisson distribution with 𝜇 = 𝑓𝑝:

Binomial

𝐹(𝑘) ≅ e

−µ

k

� j=0

µj Γ(𝑘 + 1, 𝜇) = j! 𝑘!

≅ 𝐹𝜒2 (2𝜇, 2𝑘 + 2)

When 𝑓𝑝 ≥ 10 and 𝑓𝑝(1 − 𝑝) ≥ 10 then the cdf can be approximated using a normal distribution: 𝑘 + 0.5 − 𝑓𝑝 𝐹(𝑘) ≅ Φ � � �𝑓𝑝(1 − 𝑝) k

𝑅(𝑘) = 1 − �

Reliability

n

= �

j=0

j=k+1

𝑓! pj (1 − p)n−j 𝑗! (𝑓 − 𝑗)!

𝑓! pj (1 − p)n−j 𝑗! (𝑓 − 𝑗)!

= 𝐼𝑝 (𝑘 + 1, 𝑓 − 𝑘)

where 𝐼𝑝 (𝑠, 𝑏) is the Regularized Incomplete Beta function. See section 1.6.3.

Binomial Discrete Distribution

Hazard Rate

159

−1 n (1 + θ)n − ∑kj=0 � � θj k ℎ(𝑘) = �1 + � n � � θk k

where

𝜃=

(Gupta et al. 1997)

𝑝 1−𝑝

Properties and Moments

𝑘0.5 is either {⌊𝑓𝑝⌋, ⌈𝑓𝑝⌉}

Median

⌊(𝑓 + 1)𝑝⌋

Mode Mean - 1st Raw Moment nd

Variance - 2

𝑓𝑝

𝑓𝑝(1 − 𝑝)

Central Moment

1 − 2p

rd

Skewness - 3 Central Moment

�np(1 − p)

6p2 − 6p + 1 np(1 − p)

th

Excess kurtosis - 4 Central Moment

�1 − p + peit �

Characteristic Function 100α% Percentile Function

n

Numerically solve for 𝑘 (which is not arduous for 𝑓 ≤ 10): 𝑘𝛼 = 𝐹 −1 (𝑓, 𝑝)

k α ≅ �Φ−1 (𝛼)�𝑓𝑝(1 − 𝑝) + 𝑓𝑝 − 0.5� Parameter Estimation Maximum Likelihood Function Likelihood Function

For complete data only:

𝑛𝐵

ni 𝐿(𝑝|𝐸) = � �k � pki (1 − p)ni −ki i=1

i

= p∑ki (1 − p)∑ni −∑ki

𝑛𝐵 Where 𝑓𝐵 is the number of Binomial processes, ∑𝑘𝑖 = ∑𝑖=1 𝑘𝑖 , 𝑛𝐵 ∑𝑓𝑖 = ∑𝑖=1 𝑓𝑖 and the combinatory term is ignored (see section 1.1.6 for discussion).

Binomial

For 𝑓𝑝 ≥ 10 and 𝑓𝑝(1 − 𝑝) ≥ 10 the normal approximation may be used:

160

Discrete Distributions

𝑝L =0 𝑝p

solve for 𝑝 𝑝L = ∑k i . p∑(ki )−1 (1 − p)∑ni −∑ki − (∑ni − ∑k i )p∑ki (1 − p)∑ni −1−∑ki 𝑝p ∑k i . p∑(ki )−1 (1 − p)∑ni −∑ki = (∑ni − ∑k i )p∑ki (1 − p)−1+∑ni −∑ki ∑k i . p−1 = (∑ni − ∑k i )(1 − p)−1 (1 − p) ∑ni − ∑k i = p ∑k i

p= MLE Point Estimates

∑k i ∑ni

The MLE point estimate for p: p� =

Fisher Information

Binomial

Confidence Intervals

𝐼(𝑝) =

∑k i ∑ni

1 𝑝(1 − 𝑝)

The confidence intervals for the binomial distribution parameter p is a controversial subject which is still debated. The Wilson interval is recommended for small and large 𝑓. (Brown et al. 2001) 𝑝=

where

𝑝=

𝑓𝑝̂ + 𝜅 2 ⁄2 𝜅�𝜅 2 + 4𝑓𝑝̂ (1 − 𝑝̂ ) + 2(𝑓 + 𝜅 2 ) 𝑓 + 𝜅2 𝑓𝑝̂ + 𝜅 2 ⁄2 𝜅�𝜅 2 + 4𝑓𝑝̂ (1 − 𝑝̂ ) − 2(𝑓 + 𝜅 2 ) 𝑓 + 𝜅2 𝛾+1 𝜅 = Φ−1 � � 2

It should be noted that most textbooks use the Wald interval (normal approximation) given below, however many articles have shown these estimates to be erratic and cannot be trusted. (Brown et al. 2001) 𝑝 = 𝑝̂ + 𝜅 � 𝑝 = 𝑝̂ − 𝜅 �

𝑝̂ (1 − 𝑝̂ ) 𝑓 𝑝̂ (1 − 𝑝̂ ) 𝑓

For a comparison of binomial confidence interval estimates the reader is referred to (Brown et al. 2001). The following webpage has links to online calculators which use many different methods. http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval

Binomial Discrete Distribution

161

Bayesian Non-informative Priors for p given n, 𝝅(𝒑|𝒏) (Yang and Berger 1998, p.6)

Type

Prior

Posterior

Uniform Proper Prior with limits 𝑝 ∈ [𝑠, 𝑏]

1 𝑏−𝑠

Uniform Improper Proir with limits 𝑝 ∈ [0,1]

1 = 𝐵𝑅𝑡𝑠(𝑝; 1,1)

Jeffrey’s Prior Reference Prior MDIP Novick and Hall

UOI

Example

Otherwise 𝜋(𝑝) = 0

𝐵𝑅𝑡𝑠(𝑝; 1 + 𝑘, 1 + 𝑓 − 𝑘)

1

1 1 = 𝐵𝑅𝑡𝑠 �𝑝; , � 2 2 �𝑝(1 − 𝑝) 𝑝

1 1 𝐵𝑅𝑡𝑠 �𝑝; + 𝑘, + 𝑓 − 𝑘� 2 2 when 𝑝 ∈ [0,1]

1.6186𝑝𝑝 (1 − 𝑝)1−𝑝

−1 (1

Likelihood Model Binomial

−1

− 𝑝)

Proper - No Closed Form 𝐵𝑅𝑡𝑠(𝑝; 𝑘, 𝑓 − 𝑘) when 𝑝 ∈ [0,1]

= 𝐵𝑅𝑡𝑠(0,0)

Conjugate Priors

Evidence 𝑘 failures in 𝑓 trial

Dist of UOI

Prior Para

Beta

𝛼𝑜 , 𝛽𝑜

Posterior Parameters 𝛼 = 𝛼𝑜 + 𝑘 β = βo + n − k

Description , Limitations and Uses

Five machines are measured for performance on demand. The machines can either fail or succeed in their application. The machines are tested for 10 demands with the following data for each machine: Machine/Trail 1 2 3 4 5 𝜇𝑖

1

2 3 F=3 F=2 F=2 F=3 F=2 𝑓𝑝̂

4

5

6

7 8 S=7 S=8 S=8 S=7 S=8 𝑓(1 − 𝑝̂ )

9

10

Assuming machines are homogeneous estimate the parameter 𝑝: Using MLE:

p� =

∑k i 12 = = 0.24 ∑ni 50

Binomial

𝑝 from 𝐵𝑠𝑓𝑓𝑚(𝑘; 𝑝, 𝑓)

Truncated Beta Distribution For a ≤ 𝑝 ≤ b 𝑐. 𝐵𝑅𝑡𝑠(𝑝; 1 + 𝑘, 1 + 𝑓 − 𝑘)

162

Discrete Distributions

90% confidence intervals for 𝑝: 𝜅 = Φ−1 (0.95) = 1.64485 𝑝𝑙𝑜𝑤𝑒𝑟 =

𝑓𝑝̂ + 𝜅 2 ⁄2 𝜅�𝜅 2 + 4𝑓𝑝̂ (1 − 𝑝̂ ) − = 0.1557 2(𝑓 + 𝜅 2 ) 𝑓 + 𝜅2

𝑝𝑢𝑝𝑝𝑒𝑟 =

𝑓𝑝̂ + 𝜅 2 ⁄2 𝜅�𝜅 2 + 4𝑓𝑝̂ (1 − 𝑝̂ ) + = 0.351 2(𝑓 + 𝜅 2 ) 𝑓 + 𝜅2

A Bayesian point estimate using a uniform prior distribution 𝐵𝑅𝑡𝑠(1, 1), with posterior 𝐵𝑅𝑡𝑠(𝑝; 13, 39) has a point estimate: 𝑝̂ = E[𝐵𝑅𝑡𝑠(𝑝; 13, 39)] =

13 = 0.25 52

With 90% confidence interval using inverse Beta cdf: −1 [𝐹𝐵𝑒𝑡𝑎 (0.05) = 0.1579,

−1 𝐹𝐵𝑒𝑡𝑎 (0.95) = 0.3532]

Binomial

The probability of observing no failures in the next 10 trials with replacement is: 𝑝(0; 10,0.25) = 0.0563

Characteristics

The probability of observing less than 5 failures in the next 10 trials with replacement is: 𝑝(0; 10,0.25) = 0.9803

CDF Approximations. The Binomial distribution is one of the most widely used distributions throughout history. Although simple, the CDF function was tedious to calculate prior to the use of computers. As a result approximations using the Poisson and Normal distribution have been used. For details see ‘Related Distributions’. With Replacement. The Binomial distribution models probability of 𝑘 successes in 𝑓 Bernoulli trials. However, the 𝑘 successes can occur anywhere among the 𝑓 trials with 𝑛𝐶𝐸 different combinations. Therefore the Binomial distribution assumes replacement. The equivalent distribution which assumes without replacement is the hypergeometric distribution. Symmetrical. The distribution is symmetrical when 𝑝 = 0.5.

Compliment. 𝑝(𝑘; 𝑓, 𝑝) = 𝑝(𝑓 − 𝑘; 𝑓, 1 − 𝑝). Tables usually only provide values up to 𝑓/2 allowing the reader to calculate to 𝑓 using the compliment formula. Assumptions. The binomial distribution describes the behavior of

Binomial Discrete Distribution

163

a count variable K if the following conditions apply: 1. The number of observations n is fixed. 2. Each observation is independent. 3. Each observation represents one of two outcomes ("success" or "failure"). 4. The probability of "success" is the same for each outcome.

Convolution Property

Applications

Resources

When 𝑝 is fixed.

𝐾~𝐵𝑠𝑓𝑓𝑚(𝑓, 𝑝)

� K i ~𝐵𝑠𝑓𝑓𝑚(∑𝑓𝑖 , 𝑝)

Used to model independent repeated trials which have two outcomes. Examples used in Reliability Engineering are: • Number of independent components which fail, 𝑘, from a population, 𝑓 after receiving a shock. • Number of failures to start, 𝑘, from 𝑓 demands on a component. • Number of independent items defective, 𝑘, from a population of 𝑓 items. Online: http://mathworld.wolfram.com/BinomialDistribution.html http://en.wikipedia.org/wiki/Binomial_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (web calc)

Johnson, N.L., Kemp, A.W. & Kotz, S., 2005. Univariate Discrete Distributions 3rd ed., Wiley-Interscience. Relationship to Other Distributions The Binomial distribution counts the number of successes 𝑘 in 𝑓 independent observations of a Bernoulli process. Let Bernoulli Distribution 𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(k ′ ; p)

Then

𝐾𝑖 ~𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(k′i ; 𝑝)

𝑠𝑓𝑝

𝑛

𝑌 = � 𝐾𝑖 𝑖=1

Y~𝐵𝑠𝑓𝑓𝑚(∑k ′ i ; 𝑓, 𝑝) where 𝑘 ∈ {1, 2, … , 𝑓}

Special Case:

𝐵𝑅𝑠𝑓𝑓𝑠𝑠𝑠𝑠(𝑘; 𝑝) = 𝐵𝑠𝑓𝑓𝑚(𝑘; 𝑝|𝑓 = 1)

Binomial

Books: Collani, E.V. & Dräger, K., 2001. Binomial distribution handbook for scientists and engineers, Birkhäuser.

164

Discrete Distributions

Hypergeometric Distribution 𝐻𝑦𝑝𝑅𝑠𝐺𝑅𝑓𝑚 (𝑘; 𝑓, 𝑚, 𝑁)

The hypergeometric distribution models probability of 𝑘 successes in 𝑓 Bernoulli trials from a population 𝑁, with 𝑚 successors without replacement. 𝑝(𝑘; 𝑓, 𝑚, 𝑁) Limiting Case for 𝑓 ≫ 𝑘 and 𝑝 not near 0 or 1: 𝑚 lim 𝐵𝑠𝑓𝑓𝑚(𝑘; 𝑓, 𝑝 = ) = 𝐻𝑦𝑝𝑅𝑠𝐺𝑅𝑓𝑚(𝑘; 𝑓, 𝑚, 𝑁) 𝑛→∞ 𝑁 R

Limiting Case for constant 𝑝 : lim 𝐵𝑠𝑓𝑓𝑚(𝑘|𝑓, 𝑝) = 𝑁𝑓𝑠𝑚�k�µ = n𝑝, 𝜎 2 = 𝑓𝑝(1 − 𝑝)� 𝑛→∞ R

Normal Distribution 𝑁𝑓𝑠𝑚(𝑡; 𝜇, 𝜎 2 )

𝑝=𝑝

The Normal distribution can be used as an approximation of the Binomial distribution when 𝑓𝑝 ≥ 10 and 𝑓𝑝(1 − 𝑝) ≥ 10. 𝐵𝑠𝑓𝑓𝑚(𝑘|𝑝, 𝑓) ≈ 𝑁𝑓𝑠𝑚�𝑘 + 0.5�𝜇 = 𝑓𝑝, 𝜎 2 = 𝑓𝑝(1 − 𝑝)�

Limiting Case for constant 𝑓𝑝: lim 𝐵𝑠𝑓𝑓𝑚(𝑘; 𝑓, 𝑝) = 𝑃𝑓𝑠𝑠(k; µ = n𝑝) 𝑛→∞ n𝑝=𝜇

The Poisson distribution is the limiting case of the Binomial distribution when 𝑓 is large but the ratio of 𝑓𝑝 remains constant. Hence the Poisson distribution models rare events.

Binomial

R

Poisson Distribution 𝑃𝑓𝑠𝑠(𝑘; 𝜇)

Multinomial Distribution 𝑀𝑁𝑓𝑚𝑑 (𝐤|n, 𝐩)

The Poisson distribution can be used as an approximation to the Binomial distribution when 𝑓 ≥ 20 and 𝑝 ≤ 0.05, or if 𝑓 ≥ 100 and 𝑓𝑝 ≤ 10.

The Binomial is expressed in terms of the total number of a probability of success, 𝑝, and trials, 𝑁. Where a Poisson distribution is expressed in terms of a success rate and does not need to know the total number of trials. The derivation of the Poisson distribution from the binomial can be found at http://mathworld.wolfram.com/PoissonDistribution.html. This interpretation can also be used to understand the conditional distribution of a Poisson random variable: Let 𝐾1 , 𝐾2 ~𝑃𝑓𝑠𝑠(µ) Given 𝑓 = 𝐾1 + 𝐾2 = 𝑓𝑠𝑚𝑏𝑅𝑠 𝑓𝑝 𝑅𝑠𝑅𝑓𝑡𝑠 Then µ1 𝐾1 |n~𝐵𝑠𝑓𝑓𝑚 �k; n, p = � µ1 + µ2 Special Case:

𝑀𝑁𝑓𝑚𝑑=2 (𝐤|n, 𝐩) = 𝐵𝑠𝑓𝑓𝑚(𝑘|𝑓, 𝑝)

Poisson Discrete Distribution

165

5.3. Poisson Discrete Distribution Probability Density Function - f(k) 0.20 0.18 0.16 0.14 0.12 0.10 0.08 0.06 0.04 0.02 0.00

μ=4 μ=10 μ=20 μ=30

0

5

10

15

20

25

30

35

40

45

Cumulative Density Function - F(k) 1.00 0.80 0.60

μ=4 μ=10 μ=20 μ=30

0.40 0.20

0

5

10

15

20

25

30

35

40

45

10

15

20

25

30

35

40

45

Hazard Rate - h(k) 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0

5

Poisson

0.00

166

Discrete Distributions

Parameters & Description Shape Parameter: The value of 𝜇 is the expected number of events per time period or other physical dimensions. If the Poisson distribution is modeling failure events, then 𝜇 = 𝜆𝑡 is the average number of failures that would occur in the space 𝑡. In this case 𝑡 is fixed and 𝜆 becomes the distribution parameter. Some texts use the symbol 𝜌. R

𝜇

Parameters

𝜇>0

𝑘 is an integer,

Random Variable Distribution

Formulas

𝑝(𝑘) =

PDF

𝜇𝐸 −µ (𝜆𝑡)𝐸 −λt 𝑅 = 𝑅 k! k!

−µ

𝐹(𝑘) = e

Poisson

CDF

Reliability

𝑘 ≥ 0

k

� j=0

µj Γ(𝑘 + 1, 𝜇) = j! 𝑘!

= 𝐹𝜒2 (2𝜇, 2𝑘 + 2)

Where 𝐹𝜒2 (𝑚|𝑠) is the Chi-square CDF.

When 𝜇 > 10 the 𝐹(𝑘) can be approximated by a normal distribution: 𝑘 + 0.5 − 𝜇 𝐹(𝑘) ≅ Φ � � √𝜇 R(k) = 1 − F(k)

−1

k

Hazard Rate

k! µj ℎ(𝑘) = �1 + �eµ − 1 − � �� µ j! j=1

(Gupta et al. 1997)

Properties and Moments Median Mode Mean - 1st Raw Moment

2

See 100α% Percentile Function when 𝛼 = 0.5. ⌊𝜇⌋ where ⌊ ⌋ is the floor function 2

⌊𝜇⌋ = is the floor function (largest integer not greater than 𝜇)

𝜇

Poisson Discrete Distribution

Variance - 2nd Central Moment

167

𝜇

rd

1/�µ

Skewness - 3 Central Moment th

1/µ

Excess kurtosis - 4 Central Moment

exp{µ�eik − 1�}

Characteristic Function 100α% Percentile Function

Numerically solve for 𝑘 (which is not arduous for 𝜇 ≤ 10): 𝑘𝛼 = 𝐹 −1 (𝛼)

For 𝑘 > 10 the normal approximation may be used: k α ≅ ��µΦ−1 (𝛼) + 𝜇 − 0.5�

Parameter Estimation

Maximum Likelihood Estimates Likelihood Functions

For complete data:

𝐹

n 𝜇 𝐸𝑖 −µ 𝐿(𝜇|𝐸) = � F 𝑅 i=1 k i ! ��������� known k

Log-Likelihood Function

MLE Point Estimates

n

Λ = −nµ + �{𝑘𝑖 ln(𝜇) − ln(𝑘𝑖 !)} ��������������������� i=1 known k n

∂Λ 1 = −n + � 𝑘𝑖 = 0 ∂µ µ ������ ��� i=1

For complete data solving µ� =

∂Λ ∂µ n

known k

= 0 gives:

n

1 1 . � 𝑘𝑖 𝑓𝑠 λ� = . � 𝑘𝑖 𝑓 𝑡𝑓 i=1

i=1

Note that in this context: t = the unit of time for which the rate, 𝜆 is being measured. 𝑓 = the number of Poisson processes for which the exact number of failures, k, was known. 𝑘𝑖 = the number of failures that occurred within the ith Poisson process. When there is only one Poisson process this reduces to: 𝑘 µ� = 𝑘 𝑓𝑠 λ� = 𝑡 For censored data numerical methods are needed to maximize the

Poisson

∂Λ =0 ∂µ

where 𝑓 is the number of poisson processes.

168

Discrete Distributions

log-likelihood function. Fisher Information

𝐼(𝜆) =

𝜆 lower 2 Sided R

100𝛾% Confidence Interval (complete data only)

1 𝜆

𝜆 upper 2 Sided R

Conservative two sided confidence intervals.

𝜒 21−γ (2∑𝑘𝑖 )

𝜒 21+γ (2∑𝑘𝑖 + 2)

When 𝑘 is large (𝑘 > 10) two sided intervals

1+𝛾 𝜆̂ λ� − Φ−1 � �� 2 𝑡𝑓

1+𝛾 𝜆̂ λ� + Φ−1 � �� 2 𝑡𝑓

�

2

�

�

2𝑡𝑓

2

�

2𝑡𝑓

(Nelson 1982, p.201) Note: The first confidence intervals are conservative in that at least 100𝛾%. Exact confidence intervals cannot be easily achieved for discrete distributions. Bayesian Non-informative Priors 𝝅(𝝀) in known time interval 𝑡

Poisson

Type

Prior

Posterior

Uniform Proper Prior with limits 𝜆 ∈ [𝑠, 𝑏]

1 𝑏−𝑠

Uniform Improper Prior with limits 𝜆 ∈ [0, ∞)

1 ∝ 𝐺𝑠𝑚𝑚𝑠(1,0) 1

Jeffrey’s Prior

√𝜆

Truncated Gamma Distribution For a ≤ λ ≤ b 𝑐. 𝐺𝑠𝑚𝑚𝑠(𝜆; 1 + k, t) Otherwise 𝜋(𝜆) = 0

𝐺𝑠𝑚𝑚𝑠(𝜆; 1 + k, t)

1

∝ 𝐺𝑠𝑚𝑚𝑠(2, 0)

1 ∝ 𝐺𝑠𝑚𝑚𝑠(0,0) 𝜆

Novick and Hall

𝐺𝑠𝑚𝑚𝑠(𝜆; 12 + k, t) when 𝜆 ∈ [0, ∞) 𝐺𝑠𝑚𝑚𝑠(𝜆; k, t) when 𝜆 ∈ [0, ∞)

Conjugate Priors

UOI 𝜆 from 𝑃𝑓𝑠𝑠(𝑘; µ) Example

Likelihood Model Exponential

Evidence 𝑓𝐹 failures in 𝑡𝑇 unit of time

Dist of UOI

Prior Para

Gamma

𝑘0 , Λ 0

Description , Limitations and Uses

Posterior Parameters 𝑘 = 𝑘𝑜 + 𝑓𝐹 Λ = Λ 𝑜 + 𝑡𝑇

Three vehicle tires were run on a test area for 1000km have punctures at the following distances: Tire 1: No punctures Tire 2: 400km, 900km

Poisson Discrete Distribution

169

Tire 3: 200km Punctures can be modeled as a renewal process with perfect repair and an inter-arrival time modeled by an exponential distribution. Due to the Poisson distribution being homogeneous in time, the test from multiple tires can be combined and considered a test of one tire with multiple renewals. See example in section 1.1.6. Total time on test is 3 × 1000 = 3000km. Total number of failures is 3. Therefore using MLE the estimate of 𝜆: 𝜆̂ =

k 3 = = 1E-3 𝑡𝑇 3000

With 90% confidence interval (conservative): 2 2 (6) (8) 𝜒(0.95) 𝜒(0.05) � = 0.272𝐸-3, = 2.584𝐸-3� 6000 6000

A Bayesian point estimate using the Jeffery non-informative improper prior 𝐺𝑠𝑚𝑚𝑠(12, 0), with posterior 𝐺𝑠𝑚𝑚𝑠(𝜆; 3.5, 3000) has a point estimate: 3.5 𝜆̂ = E[𝐺𝑠𝑚𝑚𝑠(𝜆; 3.5, 3000)] = = 1.16̇ E − 3 3000

Characteristics

With 90% confidence interval using inverse Gamma cdf: [𝐹𝐺−1 (0.05) = 0.361𝐸-3, 𝐹𝐺−1 (0.95) = 2.344𝐸-3]

If the following assumptions are met than the process follows a Poisson distribution: • The chance of two simultaneous events is negligible or impossible (such as renewal of a single component); • The expected value of the random number of events in a region is proportional to the size of the region. • The random number of events in non-overlapping regions are independent. μ characteristics: • 𝜇 is the expected number of events for the unit of time being measured. • When the unit of time varies μ can be transformed into a rate and time measure, 𝜆𝑡. • For 𝜇 ≲ 10 the distribution is skewed to the right. • For 𝜇 ≳ 10 the distribution approaches a normal distribution with a 𝜇 = 𝜇 and 𝜎 = √𝜇. 𝐾~𝑃𝑓𝑠𝑠(𝜇)

Poisson

The Poisson distribution is also known as the Rare Event distribution.

170

Discrete Distributions

Applications

Convolution property 𝐾1 + 𝐾2 + … + 𝐾𝑛 ~𝑃𝑓𝑠𝑠(𝑘; ∑𝜇𝑖 )

Homogeneous Poisson Process (HPP). The Poisson distribution gives the distribution of exactly k failures occurring in a HPP. See relation to exponential and gamma distributions. Renewal Theory. Used in renewal theory as the counting function and may model non-homogeneous (aging) components by using a time dependent failure rate, 𝜆 (𝑡).

Binomial Approximation. Used to model the Binomial distribution when the number of trials is large and μ remains moderate. This can greatly simplify Binomial distribution calculations.

Rare Event. Used to model rare events when the number of trials is large compared to the rate at which events occur. Resources

Online: http://mathworld.wolfram.com/PoissonDistribution.html http://en.wikipedia.org/wiki/Poisson_distribution http://socr.ucla.edu/htmls/SOCR_Distributions.html (interactive web calculator) Books: Haight, F.A., 1967. Handbook of the Poisson distribution [by] Frank A. Haight, New York,: Wiley.

Poisson

Nelson, W.B., Interscience.

1982.

Applied

Life

Data

Analysis,

Wiley-

Johnson, N.L., Kemp, A.W. & Kotz, S., 2005. Univariate Discrete Distributions 3rd ed., Wiley-Interscience. Relationship to Other Distributions Let 𝐾~𝑃𝑓𝑠𝑠(k; µ = 𝜆𝑡)

Given Exponential Distribution 𝐸𝑚𝑝(𝑡; 𝜆)

Gamma Distribution 𝐺𝑠𝑚𝑚𝑠(𝑘|𝜆)

𝑡𝑠𝑚𝑅 = 𝑇1 + 𝑇2 + ⋯ + 𝑇𝐾 + 𝑇𝐾+1 …

Then

𝑇1 , T2 … ~𝐸𝑚𝑝(t; 𝜆)

The time between each arrival of T is exponentially distributed. Special Cases: Let Then

𝑃𝑓𝑠𝑠(k; 𝜆𝑡|𝑘 = 1) = 𝐸𝑚𝑝(𝑡; 𝜆)

𝑇1 … 𝑇𝐸 ~𝐸𝑚𝑝(𝜆)

𝑠𝑓𝑝

𝑇𝑡 = 𝑇1 + 𝑇2 + ⋯ + 𝑇𝐸

𝑇𝑡 ~𝐺𝑠𝑚𝑚𝑠(𝑘, 𝜆)

Poisson Discrete Distribution

171

The Poisson distribution is the probability that exactly 𝑘 failures have been observed in time 𝑡. This is the probability that 𝑡 is between 𝑇𝐸 and 𝑇𝐸+1 . 𝐸+1

𝑝𝑃𝑜𝑖𝑠𝑠𝑜𝑛 (𝑘; 𝜆𝑡) = �

𝐸

where 𝑘 is an integer.

𝑝𝐺𝑎𝑚𝑚𝑎 (𝑡; 𝑚, 𝜆)𝑝𝑚

= 𝐹𝐺𝑎𝑚𝑚𝑎 (𝑡; 𝑘 + 1, 𝜆) − 𝐹𝐺𝑎𝑚𝑚𝑎 (𝑡; 𝑘, 𝜆)

Limiting Case for constant 𝑓𝑝: lim 𝐵𝑠𝑓𝑓𝑚(𝑘; 𝑓, 𝑝) = 𝑃𝑓𝑠𝑠(k|µ = n𝑝) 𝑛→∞ n𝑝=𝜇

The Poisson distribution is the limiting case of the Binomial distribution when 𝑓 is large but the ratio of 𝑓𝑝 remains constant. Hence the Poisson distribution models rare events. R

Binomial Distribution 𝐵𝑠𝑓𝑓𝑚(𝑘|𝑝, 𝑁)

The Poisson distribution can be used as an approximation to the Binomial distribution when 𝑓 ≥ 20 and 𝑝 ≤ 0.05, or if 𝑓 ≥ 100 and 𝑓𝑝 ≤ 10. The Binomial is expressed in terms of the total number of a probability of success, 𝑝, and trials, 𝑁. Where a Poisson distribution is expressed in terms of a success rate and does not need to know the total number of trials. The derivation of the Poisson distribution from the binomial can be found at http://mathworld.wolfram.com/PoissonDistribution.html.

Normal Distribution 𝑁𝑓𝑠𝑚(𝑘|𝜇′, 𝜎) Chi-square Distribution 𝜒 2 (𝑡|𝑠)

lim 𝐹𝑃𝑜𝑖𝑠𝑠𝑜𝑛 (𝑘; 𝜇) = 𝐹𝑁𝑜𝑟𝑚𝑎𝑙 (𝑘; 𝜇′ = 𝜇, 𝜎 2 = 𝜇)

𝜇→∞

This is a good approximation when 𝜇 > 1000. When 𝜇 > 10 the same approximation can be made with a correction: lim 𝐹𝑃𝑜𝑖𝑠𝑠𝑜𝑛 (𝑘; 𝜇) = 𝐹𝑁𝑜𝑟𝑚𝑎𝑙 (𝑘; 𝜇′ = 𝜇 − 0.5, 𝜎 2 = 𝜇)

𝜇→∞

𝑃𝑓𝑠𝑠(𝑘|𝜇) = 𝜒 2 (𝑚 = 2𝜇, 𝑠 = 2𝑘 + 2)

Poisson

This interpretation can also be used to understand the conditional distribution of a Poisson random variable: Let 𝐾1 , 𝐾2 ~𝑃𝑓𝑠𝑠(µ) Given 𝑓 = 𝐾1 + 𝐾2 = 𝑓𝑠𝑚𝑏𝑅𝑠 𝑓𝑝 𝑅𝑠𝑅𝑓𝑡𝑠 Then µ1 𝐾1 |n~𝐵𝑠𝑓𝑓𝑚 �k; n�p = � µ1 + µ2

172

Discrete Distributions

Poisson

6. Bivariate and Multivariate Distributions

Bivariate Normal Distribution

173

6.1. Bivariate Normal Continuous Distribution Probability Density Function - f(x,y)

y x

x

y x

x

y

𝜃 = 45𝑜

𝜃 = 0𝑜

𝜌<0

x

135𝑜 < 𝜃 < 180𝑜 Adapted from (Kotz et al. 2000, p.256)

x

𝜃 = 45𝑜

x

90𝑜 < 𝜃 < 135𝑜

Bi-var Normal

x

45𝑜 < 𝜃 < 90𝑜

y

y

y

x

𝜃 = 45𝑜

𝜃 = 0𝑜

y

𝜎𝑥 < 𝜎𝑦

x

0𝑜 < 𝜃 < 45𝑜

𝜌=0

𝜎𝑥 = 𝜎𝑦

y

𝜌>0

0

y

y

𝜎𝑥 > 𝜎𝑦

174

Bivariate and Multivariate Distributions

Parameters & Description Location parameter: The mean of each random variable.

−∞ < µj < ∞ 𝑗 ∈ {𝑚, 𝑦}

𝜇𝑥 , 𝜇𝑦 𝜎𝑥 , 𝜎𝑦

𝜎𝑗 > 0 𝑗 ∈ {𝑚, 𝑦}

𝜌

−1 ≤ 𝜌 ≤ 1

Parameters

−∞ < x < ∞ 𝑠𝑓𝑝

Limits Distribution

Scale parameter: The standard deviation of each random variable. Correlation Coefficient: The correlation between the two random variables. 𝑐𝑓𝑠[𝑋𝑌] 𝜌 = 𝑐𝑓𝑠𝑠(𝑋, 𝑌) = 𝜎𝑥 𝜎𝑦 𝐸[(𝑋 − 𝜇𝑥 )(𝑌 − 𝜇𝑦 )] = 𝜎𝑥 𝜎𝑦

−∞
Formulas

1

𝑝(x, y) =

exp �

zx 2 + zy 2 − 2𝜌zx zy � −2(1 − 𝜌2 )

2𝜋σx σy �1 − 𝜌2 = 𝜙(𝑚)𝜙(𝑦|𝑚) 𝑦 − 𝜌𝑚 𝑚 − 𝜌𝑦 = 𝜙(𝑚)𝜙 � � = 𝜙(𝑦)𝜙 � � 2 �1 − 𝜌 �1 − 𝜌2

PDF

Where 𝜙 is the standard normal distribution and: x − µj zj = 𝑗 ∈ {𝑚, 𝑦} σj

Bi-var Normal

∞

Marginal PDF

Conditional PDF

CDF

∞

𝑝(𝑚) = � 𝑝(𝑚, 𝑦) 𝑝𝑦

𝑝(𝑦) = � 𝑝(𝑚, 𝑦) 𝑝𝑚

−∞

1

1 = exp � − (zx )2 � 2 𝜎𝑥 √2𝜋 = 𝑁𝑓𝑠𝑚(𝜇𝑥 , 𝜎𝑥 )

−∞

1

1 2 exp � − �zy � � 2 𝜎𝑦 √2𝜋 = 𝑁𝑓𝑠𝑚�𝜇𝑦 , 𝜎𝑦 � =

𝜎𝑥 2 𝑝(𝑚|𝑦) = 𝑁𝑓𝑠𝑚 �𝜇𝑥|𝑦 = 𝜇𝑥 + 𝜌 � � �𝑦 − 𝜇𝑦 �, 𝜎𝑥|𝑦 = 𝜎𝑥2 (1 − 𝜌2 )� 𝜎𝑦 𝜎𝑦 2 𝑝(𝑦|𝑚) = 𝑁𝑓𝑠𝑚 �𝜇𝑦|𝑥 = 𝜇𝑦 + 𝜌 � � (𝑦 − 𝜇𝑥 ), 𝜎𝑦|𝑥 = 𝜎𝑦2 (1 − 𝜌2 )� 𝜎𝑥 𝐹(x, y) =

1

2𝜋σx σy �1 − ρ2

x

y

� � exp � −∞ −∞

zu2 + zv2 − 2ρzu zv � du dv −2(1 − ρ2 )

Bivariate Normal Distribution

where zj =

Reliability

𝑅(x, y) =

where

1

2𝜋σx σy �1 − ρ2

∞

x − µj σj ∞

� � exp � x

zj =

y

x − µj σj

175

zu2 + zv2 − 2ρzu zv � du dv 2(1 − ρ2 )

Properties and Moments

𝜇𝑥 �𝜇 �

Median

𝑦

Mode st

Mean - 1 Raw Moment

𝜇𝑥 �𝜇 � 𝑦

𝜇𝑥 𝑋 𝐸 � � = �𝜇 � 𝑦 𝑌

The mean of the marginal distributions is: 𝐸[𝑋] = 𝜇𝑥 𝐸[𝑌] = 𝜇𝑦

Variance - 2nd Central Moment

The mean of the conditional distributions gives the following lines (also called the regression lines): σx E(X|Y = y) = µx + ρ. (y − µy ) σy σy E(Y|X = x) = µy + ρ. (y − µx ) σx 𝜎2 𝑋 𝐶𝑓𝑠 � � = � 1 𝑌 𝜌𝜎1 𝜎2

𝜌𝜎1 𝜎2 � 𝜎22

100α% Percentile Function

Variance of conditional distributions: 𝑉𝑠𝑠(X|Y = y) = σ2x (1 − ρ2 ) 𝑉𝑠𝑠(𝑌|𝑋 = 𝑚) = σ2y (1 − ρ2 )

An ellipse containing 100α % of the distribution is (Kotz et al. 2000, p.254): (zx 2 + zy 2 − 2ρzx zy ) = ln(1 − 𝛼) −2(1 − 𝜌2 )

Bi-var Normal

Variance of marginal distributions: 𝑉𝑠𝑠(X) = σ2x 𝑉𝑠𝑠(Y) = σ2y

176

Bivariate and Multivariate Distributions

where zj =

x − µj σj

𝑗 ∈ {𝑚, 𝑦}

For the standard bivariate normal:

x 2 + y 2 − 2ρxy = ln(1 − 𝛼) −2(1 − ρ2 )

Parameter Estimation

Maximum Likelihood Function MLE Point Estimates

When there is only complete failure data the MLE estimates can be given as (Kotz et al. 2000, p.294): �𝑥 = 𝜇

𝑛𝐹

𝑛𝐹

1 � 𝑚𝑖 nF

2 �2x = 1 �(𝑚𝑖 − 𝜇 σ �) 𝑥 nF

𝑖=1 𝑛𝐹

1 𝜇�𝑦 = � 𝑦𝑖 nF 𝑖=1

𝑖=1 𝑛𝐹

𝑛𝐹

2 �2y = 1 ��𝑦𝑖 − 𝜇�� σ 𝑦 nF 𝑖=1

1 𝜌� = �(𝑚𝑖 − 𝜇𝑥 )�𝑦𝑖 − 𝜇𝑦 � �𝜎 �n 𝜎 𝑥 𝑦 F 𝑖=1

If one or more of the variables are known, different estimators are given in (Kotz et al. 2000, pp.294-305). �2 to give the unbiased A correction factor of -1 can be introduced to the 𝜎 estimators:

Bi-var Normal

�2x = σ

𝑛𝐹

1 2 �(𝑚𝑖 − 𝜇 �) 𝑥 nF − 1 𝑖=1

�2y = σ

𝑛𝐹

1 2 ��𝑦𝑖 − 𝜇�� 𝑦 nF − 1 𝑖=1

Bayesian Non-informative Priors: A complete coverage of numerous reference prior distributions with different parameter ordering is contained in (Berger & Sun 2008). For a summary of the general Bayesian priors and conjugates see the multivariate normal distribution. Description , Limitations and Uses Example

The accuracy of a cutting machine used in manufacturing is desired to be measured. 5 cuts at the required length are made. The lengths and room temperature were measured as: 7.436, 10.270, 10.466, 11.039, 11.854 𝑚𝑚 o 19.51, 21.23, 21.41, 22.78, 26.78 C

Bivariate Normal Distribution

MLE estimates are:

177

∑ xi = 10.213 n ∑ ti 𝜇�𝑇 = = 22.342 n

𝜇 �𝑥 =

2 ∑(xi − 𝜇 �) 𝐿 = 2.7885 n−1 2 ∑(t − 𝜇 �) i 𝑇 2 𝜎� = 7.5033 𝑇 = n−1 2 𝜎� 𝑥 =

𝑛𝐹

1 𝜌� = �(𝑚𝑖 − 𝜇𝑥 )(𝑡𝑖 − 𝜇 𝑇 ) = 0.1454 �𝜎 �n 𝜎 𝑥 𝑇 F 𝑖=1

If you know the temperature is 24 oC what is the likely cutting distance distribution?

Characteristic

𝜎𝑥 2 𝑝(𝑚|𝑡 = 24) = 𝑁𝑓𝑠𝑚 �𝜇𝑥|𝑡 = 𝜇𝑥 + 𝜌 � � (𝑡 − 𝜇 𝑇 ), 𝜎𝑥|𝑡 = 𝜎𝑥2 (1 − 𝜌2 )� 𝜎𝑡 𝑝(𝑚|𝑡 = 24) = 𝑁𝑓𝑠𝑚(10.303, 2.730)

Also known as Binormal Distribution.

Let U, V and W be three independent normally distributed random variables. Then let: 𝑋 =𝑈+𝑉 𝑌 =𝑉+𝑊

Then (𝑋, 𝑌) has a bivariate normal distribution. (Balakrishnan & Lai 2009, p.483)

Correlation Coefficient 𝝆. (Yang et al. 2004, p.49) 𝝆 > 0. When X increases then Y also tends to increase. When 𝜌 = 1 X and Y have a perfect positive linear relationship such that 𝑌 = 𝑐 + 𝑚𝑋 where 𝑚 is positive. 𝝆 < 0. When X increases then Y also tends to decrease. When 𝜌 = −1 X and Y have a perfect negative linear relationship such that 𝑌 = 𝑐 + 𝑚𝑋 where 𝑚 is negative.

Bi-var Normal

Independence. If 𝑋 and 𝑌 are jointly normal random variables, then they are independent when 𝜌 = 0. This gives a contour plot of 𝑝(𝑚, 𝑦) with concentric circles around the origin. When given a value on the 𝑦 axis it does not assist in estimating the value on the 𝑚 axis and therefore are independent. When 𝑋 and 𝑌 are independent, the pdf reduces to: zx 2 + zy 2 1 𝑝(x, y) = exp �− � 2𝜋σx σy 2

178

Bivariate and Multivariate Distributions

-

𝝆 = 𝟎. Increases or decreases in X have no affect on Y. X and Y are independent.

Ellipse Axis. (Kotz et al. 2000, p.254) The slope of the main axis from the x-axis is given as: 2𝜌𝜎𝑥 𝜎𝑦 1 � 𝜃 = 𝑡𝑠𝑓−1 � 2 2 𝜎𝑥 − 𝜎𝑦2

o If 𝜎𝑥 = 𝜎𝑦 for positive 𝜌 the main axis of the ellipse is 45 from the xo axis. For negative 𝜌 the main axis of the ellipse is -45 from the x-axis.

Circular Normal Density Function. (Kotz et al. 2000, p.255) When 𝜎𝑥 = 𝜎𝑦 and 𝜌 = 0 the bivariate distribution is known as a circular normal density function. Elliptical Normal Distribution (Kotz et al. 2000, p.255). If 𝜌 = 0 and 𝜎𝑥 ≠ 𝜎𝑦 then the distribution may be known as an elliptical normal distribution. Standard Bivariate Normal Distribution. Occurs when 𝜇 = 0 and 𝜎 = 1. For positive ρ the main axis of the ellipse is 45o from the x-axis. For negative ρ the main axis of the ellipse is -45o from the x-axis. 𝑝(x, y) =

1

2𝜋�1 − ρ2

exp �−

x 2 + y 2 − 2ρxy � 2(1 − ρ2 )

Bi-var Normal

Mean / Median / Mode: As per the univariate distributions the mean, median and mode are equal. Matrix Form. The bivariate distribution may be written in matrix form as: 𝜇1 𝜎2 𝜌𝜎1 𝜎2 𝑋 � 𝑿 = � 1 � 𝝁 = �𝜇 � 𝚺 = � 1 𝑋2 2 𝜌𝜎1 𝜎2 𝜎22 when 𝑿 ∼ 𝑁𝑓𝑠𝑚2 (𝝁, 𝚺) 𝑝(𝐱) =

1

2𝜋�|𝚺|

1 exp �− (𝐱 − 𝛍)T 𝚺 −1 (𝐱 − 𝛍)� 2

Where |𝚺| is the determinant of 𝚺. This is the form used in multivariate normal distribution. The following properties are given in matrix form: Convolution Property Let 𝑿 ∼ 𝑁𝑓𝑠𝑚(𝝁𝒙 , 𝚺𝐱 ) 𝒀 ∼ 𝑁𝑓𝑠𝑚�𝝁𝒚 , 𝚺𝐲 � Where 𝑿 ⊥ 𝒀 (independent)

Bivariate Normal Distribution

Then

179

𝑿 + 𝒀~𝑁𝑓𝑠𝑚�𝝁𝒙 + 𝝁𝒚 , 𝚺𝒙 + 𝚺𝒚 �

Note if X and Y are dependent then 𝑿 + 𝒀 may not be even be normally distributed.(Novosyolov 2006) Scaling Property Let x 1 matrix Then x 2 matrix

𝒀 = 𝑨𝑿 + 𝒃 𝒀~𝑁𝑓𝑠𝑚(𝑨𝝁 + 𝒃, 𝑨𝚺𝑨𝑻 )

Y is a p b is a p x 1 matrix A is a p

Marginalize Property: 𝜇1 𝜎2 𝑋 Let � 1 � ~𝑁𝑓𝑠𝑚 ��𝜇 � , � 1 𝑋2 2 𝜌𝜎1 𝜎2

𝜌𝜎1 𝜎2 �� 𝜎22

Conditional Property: 𝜇1 𝜎2 𝑋 Let � 1 � ~𝑁𝑓𝑠𝑚 ��𝜇 � , � 1 𝑋2 2 𝜌𝜎1 𝜎2

𝜌𝜎1 𝜎2 �� 𝜎22

Then

Then

Where

𝑋1 ∼ 𝑁𝑓𝑠𝑚(𝜇1 , 𝜎1 )

𝑝(𝑚1 |𝑚2 ) = 𝑁𝑓𝑠𝑚�𝜇1|2 , 𝜎1|2 � 𝜎

𝜇1|2 = 𝜇1 + 𝜌 �𝜎1 � (𝑚2 − 𝜇2 ) 2

𝜎1|2 = 𝜎1 �1 − 𝜌2

It should be noted that the standard deviation of the marginal distribution does not depend on the given value. Applications

The bivariate distribution is used in many more applications which are common to the multivariate normal distribution. Please refer to multivariate normal distribution for a more complete coverage.

Resources

Online: http://mathworld.wolfram.com/BivariateNormalDistribution.html http://en.wikipedia.org/wiki/Multivariate_normal_distribution http://www.aiaccess.net/English/Glossaries/GlosMod/e_gm_binormal_ distri.htm (interactive visual representation) Books: Balakrishnan, N. & Lai, C., 2009. Continuous Bivariate Distributions

Bi-var Normal

Graphical Representation of Multivariate Normal. As with all bivariate distributions having only two dependent variables allows it to be easily graphed (in a three dimensional graph) and visualized. As such the bivariate normal is popular in introducing higher dimensional cases.

180

Bivariate and Multivariate Distributions

2nd ed., Springer. Yang, K. et al., 2004. Multivariate Statistical Methods in Quality Management 1st ed., McGraw-Hill Professional. Patel, J.K, Read, C.B, 1996. Handbook of the Normal Distribution, 2nd Edition, CRC

Bi-var Normal

Tong, Y.L., 1990. The Multivariate Normal Distribution, Springer.

Dirichlet Distribution

181

6.2. Dirichlet Continuous Distribution Probability Density Function - f(x)

𝐷𝑠𝑠2 ([𝑚1 , 𝑚2 ]𝑇 ; [2,2,2]𝑇 )

𝐷𝑠𝑠2 ([𝑚1 , 𝑚2 ]𝑇 ; [10,10,10]𝑇 )

𝑇

𝐷𝑠𝑠2 ([𝑚1 , 𝑚2 ]𝑇 ; [1,1,1]𝑇 )

𝑇

𝐷𝑠𝑠2 ([𝑚1 , 𝑚2 ]𝑇 ; [2,1,2]𝑇 )

𝐷𝑠𝑠2 ([𝑚1 , 𝑚2 ]𝑇 ; �12, 12, 12� )

Dirichlet

𝐷𝑠𝑠2 ([𝑚1 , 𝑚2 ]𝑇 ; �12, 1,2� )

182

Bivariate and Multivariate Distributions

Parameters & Description

𝛂 = [𝛼1 , 𝛼2 , … , 𝛼𝑑 , 𝛼0 ]𝑇

Shape Matrix. Note that the matrix 𝜶 is 𝑝+1 in length.

αi > 0

Dimensions. The number of random variables being modeled.

𝑝≥1 (integer)

𝑝 0 ≤ xi ≤ 1 𝑑

Limits

� 𝑚𝑖 ≤ 1 𝑖=1

Distribution

Formulas 𝛼0 −1

𝑑

1 𝑝(𝐱) = �1 − � 𝑚𝑖 � B(𝛂)

α −1

xi i

i=1

𝑖=1

PDF

d

�

where B(𝜶) is the multinomial beta function: ∏di=0 Γ(αi ) B(𝛂) = Γ�∑di=0 αi �

Dirichlet

The special case of the Dirichlet distribution is the beta distribution when 𝑝 = 1. Let Where

Let Marginal PDF

𝑼 ∼ 𝐷𝑠𝑠𝑠 (𝜶𝒖 ) 𝑝(𝐮) =

Γ�𝛼Σ −

𝑼 𝑿 = � � ~𝐷𝑠𝑠𝑑 ( 𝜶) 𝑽 𝑿 = [𝑋1 , … , 𝑋𝑠 , 𝑋𝑠+1 , … , 𝑋𝑑 ]𝑇 𝑼 = [𝑋1 , … , 𝑋𝑠 ]𝑇 𝑽 = [𝑋𝑠+1 , … , 𝑋𝑑 ]𝑇 𝛼Σ = ∑𝑑𝑗=0 𝛼𝑗 = sum of 𝜶 matrix elements. 𝑇

where 𝜶𝒖 = �𝛼1 , 𝛼2 , … , 𝛼𝑠 , 𝛼Σ − ∑𝑠𝑗=1 𝛼𝑗 � Γ(αΣ )

𝑠 ∑𝑗=1 𝛼𝑗 � ∏si=1 Γ(αi )

𝑠

�1 − � 𝑚𝑖 � 𝑖=1

𝛼Σ −1−∑𝑠𝑗=1 𝛼𝑗

�

s

α −1

xi i

i=1

Dirichlet Distribution

183

When marginalized to one variable: 𝑋𝑖 ~𝐵𝑅𝑡𝑠(𝛼𝑖 , 𝛼Σ − 𝛼𝑖 ) 𝑝(𝑚𝑖 ) =

Γ(αΣ )

(1 − 𝑚𝑖 )𝛼Σ −𝛼𝑖 −1 xiαi −1

Γ(𝛼Σ − 𝛼𝑠 )Γ(αi )

𝑼|𝑽 = 𝒗 ∼ 𝐷𝑠𝑠𝑑−𝑠 �𝜶𝑢|𝒗 � where 𝜶𝒖|𝒗 = [𝛼𝑆+1 , 𝛼𝑠+2 , … , 𝛼𝑚 , 𝛼0 ]𝑇 (Kotz et al. 2000, p.488) Conditional PDF

Γ�∑𝑠𝑗=0 𝛼𝑗 �

𝑝(𝐮|𝐯) =

�1 ∏si=0 Γ(αi )

𝑠

− � 𝑚𝑖 � 𝑖=1

𝐹(𝐱) = P(X1 ≤ x1 , X2 ≤ x2 , … , Xd ≤ xd ) 𝑥1

𝑥2

𝑥𝑑

= � � …�

CDF

0

0

0

𝛼0 −1

𝛼0 −1

𝑑

�1 − � 𝑚𝑖 � 𝑖=1

�

d

�

i=1

𝑠

𝛼 −1

𝑚𝑖 𝑖

𝑖=1

α −1

xi i

d𝑝, … , 𝑝𝑚2 , 𝑝𝑚1

Numerical methods have been explored to evaluate this integral, see (Kotz et al. 2000, pp.497-500)

Reliability

𝑅(𝐱) = P(X1 > x1 , X2 > x2 , … , Xd > xd ) ∞

∞

𝛼0 −1

𝑑

∞

= � � … � �1 − � 𝑚𝑖 � 𝑥1

𝑥2

𝑥𝑑

𝑖=1

d

�

i=1

α −1

xi i

d𝑝, … , 𝑝𝑚2 , 𝑝𝑚1

Properties and Moments Solve numerically using 𝐹(𝒙) = 0.5

Median

𝛼 −1

Mode st

Mean - 1 Raw Moment

𝑚𝑖 = 𝛼 𝑖 −𝑑 for 𝛼𝑖 > 0 otherwise no mode Σ

Let 𝛼Σ = ∑𝑑𝑖=0 𝛼𝑖 :

𝐸[𝑿] = 𝝁 =

𝜶 𝛼Σ

𝐸[𝑋𝑖 ] = 𝜇𝑖 =

𝛼𝑖 𝑠Σ

where

𝑠

Dirichlet

Mean of the marginal distribution: 𝜶𝑢 𝐸[𝑼] = 𝝁𝒖 = 𝛼Σ 𝑇

𝜶𝒖 = �𝛼1 , 𝛼2 , … , 𝛼𝑠 , 𝛼Σ − � 𝛼𝑗 � 𝑗=1

Mean of the conditional distribution: 𝜶𝒖|𝒗 𝐸[𝑼|𝑽 = 𝒗] = 𝝁𝑢|𝑣 = 𝛼Σ

184

Bivariate and Multivariate Distributions

where

𝜶𝒖|𝒗 = [𝛼𝑆+1 , 𝛼𝑠+2 , … , 𝛼𝑚 , 𝛼0 ]𝑇

Let 𝛼Σ = ∑𝑑𝑖=0 𝛼𝑖 :

Variance - 2nd Central Moment

𝑉𝑠𝑠[𝑋𝑖 ] =

𝛼𝑖 (𝛼Σ − 𝛼𝑖 ) 𝛼Σ2 (𝛼Σ + 1)

𝐶𝑓𝑠�𝑋𝑖 , 𝑋𝑗 � =

Parameter Estimation

−𝛼𝑖 𝛼𝑗 𝛼Σ2 (𝛼Σ + 1)

Maximum Likelihood Function MLE Point Estimates

The MLE estimates of 𝛼�𝚤 can be obtained from n observations of 𝒙𝒊 by numerically maximizing the log-likelihood function: (Kotz et al. 2000, p.505) 𝑑

𝑑

𝑛

𝑗=0

𝑗=0

𝑖=1

1 Λ(𝛂|E) = 𝑓 �𝑠𝑓Γ(𝛼Σ ) − � ln 𝛤�𝛼𝑗 �� + 𝑓 � � �𝛼𝑗 − 1� � ln�𝑚𝑖𝑗 �� 𝑓

The method of moments are used to provide initial guesses of 𝛼𝑖 for the numerical methods. Fisher Information Matrix 𝑑2

𝐼𝑖𝑗 = −𝑓𝜓 ′ (𝛼Σ ), 𝑠≠𝑗 𝐼𝑖𝑖 = 𝑓𝜓 ′ (𝛼𝑖 ) − 𝑓𝜓 ′ (𝛼Σ )

Where 𝜓 ′ (𝑚) = 𝑑𝑥 2 𝑠𝑓Γ(𝑚) is the trigamma function. See section 1.6.8. (Kotz et al. 2000, p.506) 100𝛾% Confidence Intervals

The confidence intervals can be obtained from the fisher information matrix.

Dirichlet

Bayesian Non-informative Priors Jeffery’s Prior

�det�𝐼(𝜶)�

where 𝐼(𝜶) is given above. Conjugate Priors

UOI 𝒑 from 𝑀𝑁𝑓𝑚𝑑 (𝒌; 𝑓𝑡 , 𝒑)

Likelihood Model

Multinomial d

Evidence 𝑘𝑖,𝑗 failures in 𝑓 trials with 𝑝 possible states.

Dist of UOI

Prior Para

Posterior Parameters

Dirichlet d+1

𝜶𝒐

𝜶 = 𝜶𝒐 + 𝒌

Dirichlet Distribution

185

Description , Limitations and Uses Example

Five machines are measured for performance on demand. The machines can either fail, partially fail or success in their application. The machines are tested for 10 demands with the following data for each machine: Machine/Trail 1 2 3 4 5 𝜇𝑖

1

2 3 4 5 F=3 P=2 F=2 P=2 F=2 P=3 F=3 P=3 F=2 P=3 𝑓𝑝 �𝐹 𝑓𝑝 �𝑃

6

7

8 9 S=5 S=6 S=5 S=4 S=5 𝑓𝑝 �𝐹

10

Estimate the multinomial distribution parameter 𝒑 = [𝑝𝐹 , 𝑝𝑃 , 𝑝𝑆 ]:

Using a non-informative improper prior 𝐷𝑠𝑠3 (0,0,0) after updating: 𝑝𝐹 𝒙 = �𝑝𝑃 � 𝑝𝑆

Characteristic

12 𝜶 = �13� 25

𝑝 �𝐹 = 12 50 �𝑃 = 13 𝐸[𝒙] = �𝑝 50� 𝑝 �𝑆 = 25 50

7.15𝐵−5

𝑉𝑠𝑠[𝒙] = �7.54𝐵−5� 9.80𝐵−5

Confidence intervals for the parameters 𝒑 = [𝑝𝐹 , 𝑝𝑃 , 𝑝𝑆 ] can also be calculated using the cdf of the marginal distribution 𝐹(𝑚𝑖 ). Beta Generalization. The Dirichlet distribution is a generalization of the beta distribution. The beta distribution is seen when 𝑝 = 1.

𝜶 Interpretation. The higher 𝛼𝑖 the sharper and more certain the distribution is. This follows from its use in Bayesian statistics to model the multinomial distribution parameter 𝑝. As more evidence is used, the 𝛼𝑖 values get higher which reduces uncertainty. The values of 𝛼𝑖 can also be interpreted as a count for each state of the multinomial distribution.

This formulation is popular because it is a more simple presentation where the matrix of 𝜶 and 𝒙 are the same size. However it should be noted that last term of the vector 𝒙 is dependent on {𝑚1 … 𝑚𝑚−1 } through the relationship 𝑚𝑚 = 1 − ∑𝑚−1 𝑖=1 𝑚𝑖 .

Neutrality. (Kotz et al. 2000, p.500) If 𝑋1 and 𝑋2 are non negative

Dirichlet

Alternative Formulation. The most common formulation of the Dirichlet distribution is as follows: 𝛂 = [𝛼1 , 𝛼2 , … , 𝛼𝑚 ]𝑇 where αi > 0 𝐱 = [𝑚1 , 𝑚2 , … , 𝑚𝑚 ]𝑇 where 0 ≤ xi ≤ 1, ∑𝑚 𝑖=1 𝑚𝑖 = 1 m 1 αi −1 � xi 𝑝(𝐱) = B(𝛂) i=1

186

Bivariate and Multivariate Distributions

random variables such that 𝑋1 + 𝑋2 ≤ 1 then 𝑋𝑖 is called neutral if the following are independent: 𝑋𝑗 𝑋𝑖 ⊥ (𝑠 ≠ 𝑗) 1 − 𝑋𝑖 If 𝑿 ∼ 𝐷𝑠𝑠𝑑 (𝜶) then 𝑿 is a neutral vector with each X i being neutral under all permutations of the above definition. This property is unique to the Dirichlet distribution. Applications

Bayesian Statistics. The Dirichlet distribution is often used as a conjugate prior to the multinomial likelihood function.

Resources

Online: http://en.wikipedia.org/wiki/Dirichlet_distribution http://www.cis.hut.fi/ahonkela/dippa/node95.html Books: Kotz, S., Balakrishnan, N. & Johnson, N.L., 2000. Continuous Multivariate Distributions, Volume 1, Models and Applications, 2nd Edition 2nd ed., Wiley-Interscience. Congdon, P., 2007. Bayesian Statistical Modelling 2nd ed., Wiley. MacKay, D.J. & Petoy, L.C., 1995. A hierarchical Dirichlet language model. Natural language engineering. Relationship to Other Distributions

Beta Distribution 𝐵𝑅𝑡𝑠(𝑚; 𝛼, 𝛽)

Dirichlet

Gamma Distribution

𝐺𝑠𝑚𝑚𝑠(𝑚; 𝜆, 𝑘)

Special Case: 𝐷𝑠𝑠𝑑=1 (x; [α1 , α0 ]) = 𝐵𝑅𝑡𝑠(𝑘 = 𝑚; 𝛼 = 𝛼1 , 𝛽 = 𝛼0 ) Let:

Then: Let:

Then:

𝑌𝑖 ~𝐺𝑠𝑚𝑚𝑠(𝜆, 𝑘𝑖 ) 𝑠. 𝑠. 𝑝 𝑠𝑓𝑝

𝑉~𝐺𝑠𝑚𝑚𝑠(𝜆, ∑𝑘𝑖 ) 𝒁=�

𝑌1 𝑌2 𝑌𝑑 , ,…, � 𝑉 𝑉 𝑉

𝒁~𝐷𝑠𝑠𝑑 (𝛼1 , … , 𝛼𝐸 )

*i.i.d: independent and identically distributed

𝑑

𝑉 = � 𝑌𝑖 𝑖=1

Multivariate Normal Distribution

187

6.3. Multivariate Normal Continuous Distribution *Note for a graphical representation see bivariate normal distribution Parameters & Description 𝝁 = [𝜇1 , 𝜇2 , … , 𝜇𝑑 ]𝑇

Parameters

𝜎11 ∑=� ⋮ 𝜎𝑑1

−∞ < µi < ∞

⋯ 𝜎1𝑑 ⋱ ⋮ � ⋯ 𝜎𝑑𝑑

𝜎𝑖𝑖 > 0 𝜎𝑖𝑗 ≥ 0

𝑝≥2 (integer)

𝑝

Limits

Location Vector: A ddimensional vector giving the mean of each random variable. Covariance Matrix: A 𝑝×𝑝 matrix which quantifies the random variable variance and dependence. This matrix determines the shape of the distribution. Σ is symmetric positive definite matrix. Dimensions. The number of dependent variables.

−∞ < xi < ∞

Distribution

Formulas

𝑝(𝐱) =

PDF

1

(2𝜋)𝑑/2 �|𝚺|

1 exp �− (𝐱 − 𝛍)T 𝚺 −1 (𝐱 − 𝛍)� 2

Where |𝚺| is the determinant of 𝚺. Where

𝑇

𝑼 = �𝑋1 , … , 𝑋𝑝 �

𝑇

𝑽 = �𝑋𝑝+1 , … , 𝑋𝑑 �

Marginal PDF

𝑼 ∼ 𝑁𝑓𝑠𝑚𝑝 (𝝁𝒖 , 𝚺𝒖𝒖 )

∞

𝑝(𝒖) = � 𝑝(𝒙) 𝑝𝒗 =

Conditional PDF

𝚺𝑢𝑣 �� 𝚺𝒗𝒗

−∞

1

(2𝜋)𝑝/2 �|𝚺𝒖𝒖 |

1 −1 exp �− (𝐮 − 𝝁𝒖 )T 𝚺uu (𝐮 − 𝝁𝒖 )� 2

𝑼|𝑽 = 𝒗 ∼ 𝑁𝑓𝑠𝑚𝑝 �𝝁𝑢|𝑣 , 𝚺𝑢|𝑣 �

Multivar Normal

𝝁𝑢 𝚺𝑢𝑢 𝑼 𝑿 = � � ~𝑁𝑓𝑠𝑚𝑑 ��𝝁 � , � 𝑇 𝚺𝑢𝑣 𝑽 𝒗 𝑇 𝑿 = �𝑋1 , … , 𝑋𝑝 , 𝑋𝑝+1 , … , 𝑋𝑑 �

Let

188

Bivariate and Multivariate Distributions

Where

CDF

𝐹(𝐱) =

Reliability

𝑅(𝐱) =

𝑇 −1 𝝁𝑢|𝑣 = 𝝁𝑢 + 𝚺𝑢𝑣 𝚺𝑣𝑣 (𝒗 − 𝝁𝑣 ) 𝑇 −1 𝚺𝑢|𝑣 = 𝚺𝑢𝑢 − 𝚺𝑢𝑣 𝚺𝑣𝑣 𝚺𝑢𝑣

𝐱 1 � exp �− (𝐱 − 𝛍)T 𝚺 −1 (𝐱 − 𝛍)� d𝒙 𝑑/2 2 (2𝜋) �|𝚺| −∞

1

∞ 1 � exp �− (𝐱 − 𝛍)T 𝚺 −1 (𝐱 − 𝛍)� d𝒙 2 (2𝜋)𝑑/2 �|𝚺| 𝐱

1

Properties and Moments

𝝁

Median

𝝁

Mode st

𝐸[𝑿] = 𝝁

Mean - 1 Raw Moment

Mean of the marginal distribution: 𝐸[𝑼] = 𝝁𝑢 𝐸[𝑽] = 𝝁𝑣 Variance - 2nd Central Moment

Mean of the conditional distribution: 𝑇 −1 𝝁𝑢|𝑣 = 𝝁𝑢 + 𝚺𝑢𝑣 𝚺𝑣𝑣 (𝒗 − 𝝁𝑣 ) 𝐶𝑓𝑠[𝑿] = 𝚺

Covariance of marginal distributions: 𝐶𝑓𝑠(𝐔) = 𝚺𝐮𝐮

Covariance of conditional distributions: 𝑇 −1 𝐶𝑓𝑠(𝐔|𝐕) = 𝚺𝑢𝑢 − 𝚺𝑢𝑣 𝚺𝑣𝑣 𝚺𝑢𝑣

Multivar Normal

Parameter Estimation

Maximum Likelihood Function MLE Point Estimates

When given complete data of 𝑓𝐹 samples: 𝑻

𝒙𝒕 = �𝒙1,𝑡 , 𝒙2,𝑡 , … , 𝒙𝑑,𝑡 �

where 𝑡 = (1,2, … , 𝑓𝐹 )

The following MLE estimates are given: (Kotz et al. 2000, p.161) 𝑛𝐹

1 � = � 𝒙𝑡 𝛍 nF 𝑛𝐹

𝑡=1

1 Σ�𝑖𝑗 = ��𝑚𝑖,𝑡 − 𝜇�𝚤 �(𝑚𝑗,𝑡 − 𝜇�𝚥 ) nF 𝑡=1

A review of different estimators is given in (Kotz et al. 2000). When estimates are from a low number of samples (𝑓𝐹 < 30) a correction

Multivariate Normal Distribution

189

factor of -1 can be introduced to give the unbiased estimators (Tong 1990, p.53): 𝑛𝐹

1 Σ�𝑖𝑗 = ��𝑚𝑖,𝑡 − 𝜇�𝚤 �(𝑚𝑗,𝑡 − 𝜇�𝚥 ) nF − 1 𝑡=1

Fisher Information Matrix

𝐼𝑖,𝑗 = Bayesian

Type

Non-informative Priors when 𝚺 is known, 𝜋0 (𝝁) (Yang and Berger 1998, p.22)

Prior

(𝝁𝑇 𝚺 −1 𝝁𝑇 )−(𝑑−2)

Shrinkage

Uniform Improper Prior with limits 𝚺 ∈ (0, ∞) Jeffery’s Prior

Posterior

1

Uniform Improper, Jeffrey, Reference Prior

Type

𝜕𝜇𝑇 −1 𝜕𝜇 Σ 𝜕𝜃𝑖 𝜕𝜃𝑗

MDIP

𝑡=1

No Closed Form

Prior

Posterior

1 1

𝜋�𝚺 −𝟏 �𝑬�~

𝑊𝑠𝑠ℎ𝑠𝑠𝑡𝑑 �𝚺 −𝟏 ; 𝑓𝐹 − 𝑝 − 1,

𝜋�𝚺 −𝟏 �𝑬�~

𝑺−1 � 𝑓𝐹

𝑺−1 � 𝑓𝐹 with limits 𝚺 ∈ (0, ∞)

𝑊𝑠𝑠ℎ𝑠𝑠𝑡𝑑 �𝚺 −𝟏 ; 𝑓𝐹 ,

1 |𝚺| ∏i<𝑗(λi − λj )

Proper - No Closed Form

1 |𝚺|

No Closed Form

1 |𝚺|(logλ1 − logλd )d−2 ∏i<𝑗(λi − λj )

Proper - No Closed Form

Non-informative Priors when 𝝁 and 𝚺 are unknown for bivariate normal, 𝜋𝑜 (𝝁, 𝚺). A complete coverage of numerous reference prior distributions with different parameter ordering is contained in (Berger & Sun 2008)

Multivar Normal

Reference Prior Ordered {𝜆1 , 𝜆𝑑 , 𝜆𝑖 , . . , 𝜆𝑑−1 }

when 𝝁 ∈ (∞, ∞)

Non-informative Priors when 𝝁 is known, 𝜋𝑜 (𝚺) (Yang & Berger 1994)

d+1 |𝚺| 2

Reference Prior Ordered {𝜆𝑖 , 𝜆𝑗 , . . , 𝜆𝑑 }

𝑛𝐹

1 𝚺 𝜋(𝝁|𝑬)~𝑁𝑓𝑠𝑚𝑑 �µ; � 𝒙𝑡 , � nF nF

190

Bivariate and Multivariate Distributions

Type

Prior

Posterior 1

Uniform Improper Prior

1

No Closed Form

1 |𝚺| ∏i<𝑗(λi − λj )

No Closed Form

1 |𝚺|

No Closed Form

Jeffery’s Prior

Reference Prior Ordered {𝜆𝑖 , 𝜆𝑗 , . . , 𝜆𝑑 }

Reference Prior Ordered {𝜆1 , 𝜆𝑑 , 𝜆𝑖 , . . , 𝜆𝑑−1 } MDIP

No Closed Form

d+1 |𝚺| 2

1 |𝚺|(logλ1 − logλd )d−2 ∏i<𝑗(λi − λj )

No Closed Form

where 𝜆𝑖 is the 𝑠 𝑡ℎ eigenvalue of Σ, and 𝑅� and 𝑅 are population and sample multiple correlation coefficients where: 𝑛𝐹

1 S𝑖𝑗 = ��𝑚𝑖,𝑡 − 𝜇�𝚤 �(𝑚𝑗,𝑡 − 𝜇�𝚥 ) nF − 1 𝑡=1

and

𝐱� =

𝑛𝐹

1 � 𝒙𝑡 nF 𝑡=1

Conjugate Priors UOI

Multivar Normal

𝝁 from 𝑁𝑓𝑠𝑚𝑑 (𝝁, 𝚺)

Likelihood Model Multi-variate Normal with known 𝚺

Evidenc e 𝑓𝐹 events at 𝒙 points

Dist of UOI Multivariate Normal

Prior Para

𝑼𝟎 , 𝐕𝟎

Description , Limitations and Uses

Posterior Parameters 𝑼=

𝐕0−1 𝐔𝐨 + nF 𝐕 −1 𝐱� 𝐕0−1 + nF 𝚺 −1

𝐕=

1 𝐕0−1 + nF 𝚺 −𝟏

Example

See bivariate normal distribution.

Characteristic

Standard Spherical Normal Distribution. When 𝝁 = 0, 𝚺 = 𝐼 we obtain the standard spherical normal distribution: 1 1 𝑝(𝐱) = exp �− 𝐱 T 𝐱� (2𝜋)𝑑/2 2 Covariance Matrix. (Yang et al. 2004, p.49) Diagonal Elements. The diagonal elements of Σ is the variance of each random variable. 𝜎𝑖𝑖 = 𝑉𝑠𝑠(𝑋𝑖 ) Non Diagonal Elements. Non diagonal elements give the covariance 𝜎𝑖𝑗 = 𝐶𝑓𝑠�𝑋𝑖 , 𝑋𝑗 � = 𝜎𝑗𝑖 . Hence the matrix is symmetric. Independent Variables. If 𝐶𝑓𝑠�𝑋𝑖 , 𝑋𝑗 � = 𝜎𝑖𝑗 = 0 then 𝑋𝑖 and

Multivariate Normal Distribution

-

191

𝑋𝑗 and independent. 𝝈𝒊𝒋 > 0. When 𝑋𝑖 increases then 𝑋𝑗 and tends to increase. 𝝈𝒊𝒋 < 0. When 𝑋𝑖 increases then 𝑋𝑗 and tends to decrease.

Ellipsoid Axis. The ellipsoids has axes pointing in the direction of the eigenvectors of 𝚺. The magnitude of these axes are given by the corresponding eigenvalues. Mean / Median / Mode: As per the univariate distributions the mean, median and mode are equal. Convolution Property 𝒀 ∼ 𝑁𝑓𝑠𝑚𝑑 �𝝁𝒚 , 𝚺𝐲 � Let 𝑿 ∼ 𝑁𝑓𝑠𝑚𝑑 (𝝁𝒙 , 𝚺𝐱 ) Where 𝑿 ⊥ 𝒀 (independent) Then 𝑿 + 𝒀~𝑁𝑓𝑠𝑚𝑑 �𝝁𝒙 + 𝝁𝒚 , 𝚺𝒙 + 𝚺𝒚 �

Note if X and Y are dependent then X + Y may not be normally distributed. (Novosyolov 2006) Scaling Property Let

Then

𝒀 = 𝑨𝑿 + 𝒃

𝒀~𝑁𝑓𝑠𝑚𝑑 (𝑨𝝁 + 𝒃, 𝑨𝚺𝑨𝑻 )

Y is a p x 1 matrix b is a p x 1 matrix A is a p x d matrix

Marginalize Property: Let Then

𝝁𝑢 𝚺𝑢𝑢 𝑼 𝑿 = � � ~𝑁𝑓𝑠𝑚𝑑 ��𝝁 � , � 𝑇 𝚺𝑢𝑣 𝑽 𝒗 𝑼 ∼ 𝑁𝑓𝑠𝑚𝑝 (𝝁𝒖 , 𝚺𝒖𝒖 )

Conditional Property: Let

Where

𝑼|𝑽 = 𝒗 ∼ 𝑁𝑓𝑠𝑚𝑝 �𝝁𝑢|𝑣 , 𝚺𝑢|𝑣 �

𝑼 is a p x 1 matrix 𝚺𝑢𝑣 �� 𝚺𝒗𝒗

𝑇 −1 𝝁𝑢|𝑣 = 𝝁𝑢 + 𝚺𝑢𝑣 𝚺𝑣𝑣 (𝑽 − 𝝁𝑣 ) 𝑇 −1 𝚺𝑢|𝑣 = 𝚺𝑢𝑢 − 𝚺𝑢𝑣 𝚺𝑣𝑣 𝚺𝑢𝑣

𝑼 is a p x 1 matrix

It should be noted that the standard deviation of the marginal distribution does not depend on the given values in V. Applications

Convenient Properties. (Balakrishnan & Lai 2009, p.477) Popularity of the multivariate normal distribution over other multivariate distributions is due to the convenience of the conditional and marginal distribution properties which both produce univariate normal distributions.

Multivar Normal

Then

𝝁𝑢 𝚺𝑢𝑢 𝑼 𝑿 = � � ~𝑁𝑓𝑠𝑚𝑑 ��𝝁 � , � 𝑇 𝚺𝑢𝑣 𝑽 𝑣

𝚺𝑢𝑣 �� 𝚺𝒗𝒗

192

Bivariate and Multivariate Distributions

Kalman Filter. The Kalman filter estimates the current state of a system in the presence of noisy measurements. This process uses multivariate normal distributions to model the noise. Multivariate Analysis of Variance (MANOVA). A test used to analyze variance and dependence of variables. A popular model used to conduct MANOVA assumes the data comes from a multivariate normal population. Multi-Linear Regression. Multi-linear regression attempts to model the relationship between parameters and variables by fitting a linear equation. One model to do such a task (MLE) fits a distribution to the observed variance where a multivariate normal distribution is often assumed. Gaussian Bayesian Belief Networks (BBN). BBNs graphical represent the dependence between variables in a probability distribution. When using continuous random variables BBNs quickly become tremendously complicated. However due to the multivariate normal distribution’s conditional and marginal properties this task is simplified and popular. Resources

Online: http://mathworld.wolfram.com/BivariateNormalDistribution.html http://www.aiaccess.net/English/Glossaries/GlosMod/e_gm_binormal _distri.htm (interactive visual representation) Books: Patel, J.K, Read, C.B, 1996. Handbook of the Normal Distribution, 2nd Edition, CRC

Multivar Normal

Tong, Y.L., 1990. The Multivariate Normal Distribution, Springer. Yang, K. et al., 2004. Multivariate Statistical Methods in Quality Management 1st ed., McGraw-Hill Professional. Bertsekas, D.P. & Tsitsiklis, J.N., 2008. Introduction to Probability, 2nd Edition, Athena Scientific.

Multinomial Distribution

193

6.4. Multinomial Discrete Distribution Probability Density Function - f(k)

1 1

5 𝑇

Trinomial Distribution, 𝑝([𝑘1 , 𝑘2 , 𝑘3 ]𝑇 ) where 𝑓 = 8, 𝐩 = �3 , 4 , 12� . Note 𝑘3 is not shown because it is determined using 𝑘3 = 𝑓 − 𝑘1 − 𝑘2

Multinomial

1 1 1 𝑇

Trinomial Distribution, 𝑝([𝑘1 , 𝑘2 , 𝑘3 ]𝑇 ) where 𝑓 = 20, 𝐩 = �3 , 2 , 6� . Note 𝑘3 is not shown because it is determined as 𝑘3 = 𝑓 − 𝑘1 − 𝑘2

194

Bivariate and Multivariate Distributions

Parameters & Description 𝑓 Parameters

𝐩 = [𝑝1 , 𝑝2 , … , 𝑝𝑑 ]𝑇 𝑝

n>0 (integer)

Number of Trials. This is sometimes called the index. (Johnson et al. 1997, p.31)

0 ≤ pi ≤ 1

Event Probability Matrix: The probability of event i occurring. 𝑝𝑖 is often called cell probabilities. (Johnson et al. 1997, p.31)

𝑑

� 𝑝𝑖 = 1 𝑖=1

𝑝≥2 (integer)

Dimensions. The number of mutually exclusive states of the system.

k i ∈ {0, … , 𝑓} 𝑑

Limits

� 𝑘𝑖 = 𝑓 𝑖=1

Distribution

Formulas

where

PDF

d n k 𝑝(𝐤) = �k , k , . . , k � � pi i 1 2 d i=1

n! n! Γ(n + 1) n �k , k , . . , k � = = = 1 2 n k1 ! k 2 ! … k d ! ∏di=1 k i ! ∏di=1 Γ(k i + 1)

Note that in p there is only d-1 ‘free’ variables as the last 𝑝𝑑 = 1 − ∑𝑑−1 𝑖=1 𝑝𝑖 giving the distribution: s

d−1 n k 𝑝(𝐤) = �k , k , . . , k � � pi i . �1 − � pi � 1 2 n i=1

Multinomial

i=1

n−∑d−1 i=1 ki

Now the special case of binomial distribution when 𝑝 = 2 can be seen. Let Where Marginal PDF where

𝒑𝒖 𝑼 𝑲 = � � ~𝑀𝑁𝑓𝑚𝑑 �𝑓, �𝒑 �� 𝑽 𝒗 𝑲 = [𝐾1 , … , 𝐾𝑠 , 𝐾𝑠+1 , … , 𝐾𝑑 ]𝑇 𝑼 = [𝐾1 , … , 𝐾𝑠 ]𝑇 𝑽 = [𝐾𝑠+1 , … , 𝐾𝑑 ]𝑇

𝑼 ∼ 𝑀𝑁𝑓𝑚𝑠 (𝑓, 𝒑𝒖 ) 𝑇 𝒑𝒖 = �𝑝1 , 𝑝2 , … , 𝑝𝑠−1 , �1 − ∑𝑠−1 𝑖=1 𝑝𝑖 ��

Multinomial Distribution

195

s n k 𝑝(𝒖) = �k , k , . . , k � � pi i 1 2 s i=1

When only two states 𝒑 = [𝑝, (1 − 𝑝)]𝑇 :

n k 𝑝(𝑘𝑖 ) = �k � pi i (1 − 𝑝𝑖 )𝑛−𝐸𝑖 i

where Conditional PDF

𝑼|𝑽 = 𝒗 ∼ 𝑀𝑁𝑓𝑚𝑠 �𝑓𝑢|𝑣 , 𝒑𝑢|𝒗 � 𝑑

𝑓𝑢|𝑣 = 𝑓 − 𝑓𝑣 = 𝑓 − � 𝑘𝑖

𝒑𝑢|𝑣 =

1

∑𝑠𝑖=1 𝑝𝑖

𝑖=𝑠+1

[𝑝1 , 𝑝2 , … , 𝑝𝑠 ]𝑇

𝐹(𝐤) = P(K1 ≤ k1 , K 2 ≤ k 2 , … , K d ≤ k d ) k1

kd

k2

d n j = � � … � �j , j , . . , j � � pii 1 2 d i=1

CDF

j1 =0 j2 =0

Reliability

jd =0

𝑅(𝐤) = P(K1 > k1 , K 2 > k 2 , … , K d > k d ) n n n d n j = � � … � �j , j , . . , j � � pii 1 2 d i=1 j1 =k1 +1 j2 =k2 +1

jd =kd +1

Properties and Moments 𝑀𝑅𝑝𝑠𝑠𝑓(𝑘𝑖 ) is either {⌊𝑓𝑝𝑖 ⌋, ⌈𝑓𝑝𝑖 ⌉}

Median3 Mode

st

Mean - 1 Raw Moment

𝑀𝑓𝑝𝑅(𝑘𝑖 ) = ⌊(𝑓 + 1)𝑝𝑖 ⌋ 𝐸[𝑲] = 𝝁 = 𝑓𝒑

Mean of the conditional distribution: 𝐸[𝑼|𝑽 = 𝒗] = 𝝁𝑢|𝑣 = 𝑓𝑢|𝑣 𝒑𝑢|𝒗 where 𝑑

𝑓𝑢|𝑣 = 𝑓 − 𝑓𝑣 = 𝑓 − � 𝑘𝑖

𝒑𝑢|𝑣 = 3

1

∑𝑠𝑖=1 𝑝𝑖

⌊𝑚⌋ = is the floor function (largest integer not greater than 𝑚) ⌈𝑚⌉ = is the ceiling function (smallest integer not less than 𝑚)

𝑖=𝑠+1

[𝑝1 , 𝑝2 , … , 𝑝𝑠 ]𝑇

Multinomial

Mean of the marginal distribution: 𝐸[𝑼] = 𝝁𝒖 = 𝑓𝒑𝑢 𝐸[𝐾𝑖 ] = 𝜇𝐸𝑖 = 𝑓𝑝𝑖

196

Bivariate and Multivariate Distributions

𝑉𝑠𝑠[𝐾𝑖 ] = 𝑓𝑝𝑖 (1 − 𝑝𝑖 ) 𝐶𝑓𝑠�𝐾𝑖 , 𝐾𝑗 � = −𝑓𝑝𝑖 𝑝𝑗

Variance - 2nd Central Moment

Covariance of marginal distributions: 𝑉𝑠𝑠[𝐾𝑖 ] = 𝑓𝑝𝑖 (1 − 𝑝𝑖 )

Covariance of conditional distributions: 𝑉𝑠𝑠�𝐾𝑈|𝑉,𝑖 � = 𝑓𝑢|𝑣 𝑝𝑢|𝑣,𝑖 (1 − 𝑝𝑢|𝑣,𝑖 ) 𝐶𝑓𝑠�𝐾𝑈|𝑉,𝑖 , 𝐾𝑈|𝑉,𝑗 � = −𝑓𝑢|𝑣 𝑝𝑢|𝑣,𝑖 𝑝𝑢|𝑣,𝑗 where

𝑑

𝑓𝑢|𝑣 = 𝑓 − 𝑓𝑣 = 𝑓 − � 𝑘𝑖

𝒑𝑢|𝑣 = Parameter Estimation

1

∑𝑠𝑖=1 𝑝𝑖

𝑖=𝑠+1

[𝑝1 , 𝑝2 , … , 𝑝𝑠 ]𝑇

Maximum Likelihood Function MLE Point Estimates

As with the binomial distribution the MLE estimates, given the vector k(and therefore n), is:(Johnson et al. 1997, p.51) �= 𝐩

𝐤 n

Where there are 𝑇 observations of 𝒌𝑡 each containing 𝑓𝑡 trails: �= 𝐩

Multinomial

100𝛾% Confidence Intervals (Complete Data)

1

∑nt=1 nt

𝑇

� 𝒌𝑡 𝑡=1

An approximation of the joint interval confidence limits for 100𝛾% given by Goodman in 1965 is:(Johnson et al. 1997, p.51) 𝑝𝑖 lower confidence limit:

1 4 �𝐴 + 2𝑘𝑖 − 𝐴�𝐴 + 𝑘𝑖 (𝑓 − 𝑘𝑖 )� 2(𝑓 + 𝐴) 𝑓

𝑝𝑖 upper confidence limit:

4 1 �𝐴 + 2𝑘𝑖 + 𝐴�𝐴 + 𝑘𝑖 (𝑓 − 𝑘𝑖 )� 𝑓 2(𝑓 + 𝐴)

where Φ is the standard normal CDF and: d−1+γ 𝐴 = 𝑍𝑑−1+𝛾 = Φ−1 � � d 𝑑

Multinomial Distribution

197

A complete coverage of estimation techniques and confidence intervals is contained in (Johnson et al. 1997, pp.51-65). A more accurate method which requires numerical methods is given in (Sison & Glaz 1995)

Bayesian

Type Uniform Prior Jeffreys Prior One Group Reference Prior

Prior

Non-informative Priors, 𝝅(𝒑) (Yang and Berger 1998, p.6)

Posterior

1 = 𝐷𝑠𝑠𝑑+1 (𝛼𝑖 = 1)

𝐶

𝑑 �∏𝑖=1 𝑝𝑖

𝐷𝑠𝑠𝑑+1 (𝐩|1 + 𝐤) 1

𝐷𝑠𝑠𝑑+1 �𝐩�2 + 𝐤�

1

= 𝐷𝑠𝑠𝑑+1 �𝛼𝑖 = 2�

where 𝐶 is a constant

In terms of the reference prior, this approach considers all parameters are of equal importance.(Berger & Bernardo 1992)

d-group Reference Prior

𝐶

�∏𝑑−1 𝑖=1 �𝑝𝑖 �1

−

𝑖 ∑𝑗=1 𝑝𝑖 ��

where 𝐶 is a constant

Proper. See m-group posterior when 𝑚 = 1.

This approach considers each parameter to be of different importance (group length 1) and so the parameters must be ordered by importance. (Berger & Bernardo 1992) m-group Reference Prior

𝜋𝑜 (𝒑) =

𝐶

𝑁𝑚 𝑁𝑖 𝑚−1 ��1 − ∑𝑗=1 𝑝𝑗 � ∏𝑑−1 𝑖=1 𝑝𝑖 ∏𝑖=1 �1 − ∑𝑗=1 𝑝𝑗 �

𝑛𝑖+1

𝐩𝐢 = �𝑝𝑁𝑖−1 +1 , … , 𝑝𝑁𝑖 � 𝐶 is a constant

Posterior: 𝜋(𝒑|𝒌) ∝

𝑁

𝑚 �1 − ∑𝑗=1 𝑝𝑖 �

𝑇

𝐸𝑑 −12

𝑁𝑖 𝑚−1 �∏𝑑−1 𝑖=1 𝑝𝑖 ∏𝑖=1 �1 − ∑𝑗=1 𝑝𝑗 �

𝑛𝑖+1

This approach splits the parameters into m different groups of importance. Within the group order is not important, but the groups need to be ordered by importance. It is common to have 𝑚 = 2 and split the parameters into importance and nuisance parameters. (Berger & Bernardo 1992)

Multinomial

where groups are given by: 𝑇 𝑇 𝐩𝟏 = �𝑝1 , … 𝑝𝑛1 � , 𝐩𝟐 = �𝑝𝑛1 +1 , … , 𝑝𝑛1 +𝑛2 � 𝑁𝑗 = 𝑓1 + ⋯ + 𝑓𝑗 for 𝑗 = 1, … , 𝑚

198

Bivariate and Multivariate Distributions

𝑑

MDIP Novick and Hall’s Prior (improper)

UOI

𝑖=1

𝑑

�

𝑖=1

Multinomial

𝑝𝑖−1 = 𝐷𝑠𝑠𝑑+1 (𝛼𝑖 = 0)

Conjugate Priors (Fink 1997)

Multinomial d

Evidence 𝑘𝑖,𝑗 failures in 𝑓 trials with 𝑝 possible states.

𝐷𝑠𝑠𝑑+1 (𝐩|𝐤)

Dist of UOI

Prior Para

Posterior Parameters

Dirichlet d+1

𝜶𝒐

𝜶 = 𝜶𝒐 + 𝒌

Description , Limitations and Uses

A six sided dice being thrown 60 times produces the following multinomial distribution: Face Number 1 2 3 4 5 6

Characteristic

𝐷𝑠𝑠𝑑+1 (𝐩′|𝑝𝑖 + 1 + k i )

𝑝

𝑝𝑖 𝑖 = 𝐷𝑠𝑠𝑑+1 (𝛼𝑖 = 𝑝𝑖 + 1)

Likelihood Model

𝒑 from 𝑀𝑁𝑓𝑚𝑑 (𝒌; 𝑓𝑡 , 𝒑) Example

�

Times Observed 12 7 11 10 8 12

12 ⎡6⎤ ⎢12⎥ 𝒌=⎢ ⎥ ⎢10⎥ ⎢8⎥ ⎣12⎦

0.2 ⎡ 0.1 ⎤ ⎢ 0.2 ⎥ ⎥ 𝒑=⎢ ⎢0.16̇ ⎥ ⎢0.13̇ ⎥ ⎣ 0.2 ⎦

𝑓 = 60

Binomial Generalization. The multinomial distribution is a generalization of the binomial distribution where more than two states of the system are allowed. The binomial distribution is a special case where 𝑝 = 2.

Covariance. All covariance’s are negative. This is because the increase in one parameter 𝑝𝑖 must result in the decrease of 𝑝𝑗 to satisfy Σ𝑝𝑖 = 1. With Replacement. The multinomial distribution assumes replacement. The equivalent distribution which assumes without replacement is the multivariate hypergeometric distribution. Convolution Property Let Then

𝑲𝒕 ∼ 𝑀𝑁𝑓𝑚𝑑 (𝒌; 𝑓𝑡 , 𝐩)

� 𝑲𝒕 ~𝑀𝑁𝑓𝑚𝑑 (∑𝒌𝑡 ; ∑𝑓𝑡 , 𝒑)

*This does not hold when the p parameter differs.

Multinomial Distribution

199

Applications

Partial Failures. When the states of a system under demands cannot be modeled with two states (success or failure) the multinomial distribution may be used. Examples of this include when modeling discrete states of component degradation.

Resources

Online: http://en.wikipedia.org/wiki/Multinomial_distribution http://mathworld.wolfram.com/MultinomialDistribution.html http://www.math.uah.edu/stat/bernoulli/Multinomial.xhtml Books: Johnson, N.L., Kotz, S. & Balakrishnan, N., 1997. Discrete Multivariate Distributions 1st ed., Wiley-Interscience. Relationship to Other Distributions

Binominal Distribution 𝐵𝑠𝑓𝑓𝑚(𝑘|𝑓, 𝑝)

Special Case:

𝑀𝑁𝑓𝑚𝑑=2 (𝐤|n, 𝐩) = 𝐵𝑠𝑓𝑓𝑚(𝑘|𝑓, 𝑝)

Multinomial

References

200

Probability Distributions Used in Reliability Engineering

References

201

7. References Abadir, K. & Magdalinos, T., 2002. The Characteristic Function from a Family of Truncated Normal Distributions. Econometric Theory, 18(5), p.1276-1287. Agresti, A., 2002. Categorical data analysis, John Wiley and Sons. Aitchison, J.J. & Brown, J.A.C., 1957. The Lognormal Distribution, New York: Cambridge University Press. Andersen, P.K. et al., 1996. Statistical Models Based on Counting Processes Corrected., Springer. Angus, J.E., 1994. Bootstrap one-sided confidence intervals for the lognormal mean. Journal of the Royal Statistical Society. Series D (The Statistician), 43(3), p.395–401. Anon, Six Sigma | Process Management | Strategic Process Management | Welcome to SSA & Company. Antle, C., Klimko, L. & Harkness, W., 1970. Confidence Intervals for the Parameters of the Logistic Distribution. Biometrika, 57(2), p.397-402. Aoshima, M. & Govindarajulu, Z., 2002. Fixed-width confidence interval for a lognormal mean. International Journal of Mathematics and Mathematical Sciences, 29(3), p.143–153. Arnold, B.C., 1983. Pareto distributions, Fairland, MD: International Co-operative Pub. House. Artin, E., 1964. The Gamma Function, New York: Holt, Rinehart & Winston.

Balakrishnan, N. & Basu, A.P., 1996. Exponential Distribution: Theory, Methods and Applications 1st ed., CRC.

References

Balakrishnan, 1991. Handbook of the Logistic Distribution 1st ed., CRC.

202

Probability Distributions Used in Reliability Engineering

Balakrishnan, N. & Lai, C.-D., 2009. Continuous Bivariate Distributions 2nd ed., Springer. Balakrishnan, N. & Rao, C.R., 2001. Handbook of Statistics 20: Advances in Reliability 1st ed., Elsevier Science & Technology. Berger, J.O., 1993. Statistical Decision Theory and Bayesian Analysis 2nd ed., Springer. Berger, J.O. & Bernardo, J.M., 1992. Ordered Group Reference Priors with Application to the Multinomial Problem. Biometrika, 79(1), p.25-37. Berger, J.O. & Sellke, T., 1987. Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence. Journal of the American Statistical Association, 82(397), p.112-122. Berger, J.O. & Sun, D., 2008. Objective priors for the bivariate normal model. The Annals of Statistics, 36(2), p.963-982. Bernardo, J.M. et al., 1992. On the development of reference priors. Bayesian statistics, 4, p.35–60. Berry, D.A., Chaloner, K.M. & Geweke, J.K., 1995. Bayesian Analysis in Statistics and Econometrics: Essays in Honor of Arnold Zellner 1st ed., Wiley-Interscience. Bertsekas, D.P. & Tsitsiklis, J.N., 2008. Introduction to Probability 2nd ed., Athena Scientific. Billingsley, P., 1995. Probability and Measure, 3rd Edition 3rd ed., Wiley-Interscience.

References

Birnbaum, Z.W. & Saunders, S.C., 1969. A New Family of Life Distributions. Journal of Applied Probability, 6(2), p.319-327. Bjõrck, A., 1996. Numerical Methods for Least Squares Problems 1st ed., SIAM: Society for Industrial and Applied Mathematics.

References

203

Bowman, K.O. & Shenton, L.R., 1988. Properties of estimators for the gamma distribution, CRC Press. Brown, L.D., Cai, T.T. & DasGupta, A., 2001. Interval estimation for a binomial proportion. Statistical Science, p.101–117. Christensen, R. & Huffman, M.D., 1985. Bayesian Point Estimation Using the Predictive Distribution. The American Statistician, 39(4), p.319-321. Cohen, 1991. Truncated and Censored Samples 1st ed., CRC Press. Collani, E.V. & Dräger, K., 2001. Binomial distribution handbook for scientists and engineers, Birkhäuser. Congdon, P., 2007. Bayesian Statistical Modelling 2nd ed., Wiley. Cozman, F. & Krotkov, E., 1997. Truncated Gaussians as Tolerance Sets. Crow, E.L. & Shimizu, K., 1988. Lognormal distributions, CRC Press. Dekking, F.M. et al., 2007. A Modern Introduction to Probability and Statistics: Understanding Why and How, Springer. Fink, D., 1997. A compendium of conjugate priors. See http://www. people. cornell. edu/pages/df36/CONJINTRnew% 20TEX. pdf, p.46. Georges, P. et al., 2001. Multivariate Survival Modelling: A Unified Approach with Copulas. SSRN eLibrary. Gupta and Nadarajah, 2004. Handbook of beta distribution and its applications, CRC Press. References

Gupta, P.L., Gupta, R.C. & Tripathi, R.C., 1997. On the monotonic properties of discrete failure rates. Journal of Statistical Planning and Inference, 65(2), p.255-268.

204

Probability Distributions Used in Reliability Engineering

Haight, F.A., 1967. Handbook of the Poisson distribution, New York,: Wiley. Hastings, N.A.J., Peacock, B. & Evans, M., 2000. Statistical Distributions, 3rd Edition 3rd ed., John Wiley & Sons Inc. Jiang, R. & Murthy, D.N.P., 1996. A mixture model involving three Weibull distributions. In Proceedings of the Second Australia– Japan Workshop on Stochastic Models in Engineering, Technology and Management. Gold Coast, Australia, pp. 260270. Jiang, R. & Murthy, D.N.P., 1998. Mixture of Weibull distributions parametric characterization of failure rate function. Applied Stochastic Models and Data Analysis, (14), p.47-65. Jiang, R. & Murthy, D.N.P., 1995. Modeling Failure-Data by Mixture of2 Weibull Distributions : A Graphical Approach. IEEE Transactions on Reliability, 44, p.477-488. Jiang, R. & Murthy, D.N.P., 1999. The exponentiated Weibull family: a graphical approach. Reliability, IEEE Transactions on, 48(1), p.68-72. Johnson, N.L., Kemp, A.W. & Kotz, S., 2005. Univariate Discrete Distributions 3rd ed., Wiley-Interscience. Johnson, N.L., Kotz, S. & Balakrishnan, N., 1994. Continuous Univariate Distributions, Vol. 1 2nd ed., Wiley-Interscience. Johnson, N.L., Kotz, S. & Balakrishnan, N., 1995. Continuous Univariate Distributions, Vol. 2 2nd ed., Wiley-Interscience.

References

Johnson, N.L., Kotz, S. & Balakrishnan, N., 1997. Discrete Multivariate Distributions 1st ed., Wiley-Interscience. Kimball, B.F., 1960. On the Choice of Plotting Positions on Probability Paper. Journal of the American Statistical Association, 55(291), p.546-560.

References

205

Kleiber, C. & Kotz, S., 2003. Statistical Size Distributions in Economics and Actuarial Sciences 1st ed., Wiley-Interscience. Klein, J.P. & Moeschberger, M.L., 2003. Survival analysis: techniques for censored and truncated data, Springer. Kotz, S., Balakrishnan, N. & Johnson, N.L., 2000. Continuous Multivariate Distributions, Volume 1, Models and Applications, 2nd Edition 2nd ed., Wiley-Interscience. Kotz, S. & Dorp, J.R. van, 2004. Beyond Beta: Other Continuous Families Of Distributions With Bounded Support And Applications, World Scientific Publishing Company. Kundu, D., Kannan, N. & Balakrishnan, N., 2008. On the hazard function of Birnbaum-Saunders distribution and associated inference. Comput. Stat. Data Anal., 52(5), p.2692-2702. Lai, C.D., Xie, M. & Murthy, D.N.P., 2003. A modified Weibull distribution. IEEE Transactions on Reliability, 52(1), p.33-37. Lai, C.-D. & Xie, M., 2006. Stochastic Ageing and Dependence for Reliability 1st ed., Springer. Lawless, J.F., 2002. Statistical Models and Methods for Lifetime Data 2nd ed., Wiley-Interscience. Leemis, L.M. & McQueston, J.T., 2008. Univariate distribution relationships. The American Statistician, 62(1), p.45–53. Leipnik, R.B., 1991. On Lognormal Random Variables: I-the Characteristic Function. The ANZIAM Journal, 32(03), p.327347. References

Lemonte, A.J., Cribari-Neto, F. & Vasconcellos, K.L.P., 2007. Improved statistical inference for the two-parameter Birnbaum-Saunders distribution. Computational Statistics & Data Analysis, 51(9), p.4656-4681.

206

Probability Distributions Used in Reliability Engineering

Limpert, E., Stahel, W. & Abbt, M., 2001. Log-normal Distributions across the Sciences: Keys and Clues. BioScience, 51(5), p.352, 341. MacKay, D.J.C. & Petoy, L.C.B., 1995. A hierarchical Dirichlet language model. Natural language engineering. Manzini, R. et al., 2009. Maintenance for Industrial Systems 1st ed., Springer. Martz, H.F. & Waller, R., 1982. Bayesian reliability analysis, JOHN WILEY & SONS, INC, 605 THIRD AVE, NEW YORK, NY 10158. Meeker, W.Q. & Escobar, L.A., 1998. Statistical Methods for Reliability Data 1st ed., Wiley-Interscience. Modarres, M., Kaminskiy, M. & Krivtsov, V., 1999. Reliability engineering and risk analysis, CRC Press. Murthy, D.N.P., Xie, M. & Jiang, R., 2003. Weibull Models 1st ed., Wiley-Interscience. Nelson, W.B., 1990. Accelerated Testing: Statistical Models, Test Plans, and Data Analysis, Wiley-Interscience. Nelson, W.B., 1982. Applied Life Data Analysis, Wiley-Interscience. Novosyolov, A., 2006. The sum of dependent normal variables may be not normal. http://risktheory.ru/papers/sumOfDep.pdf.

References

Patel, J.K. & Read, C.B., 1996. Handbook of the Normal Distribution 2nd ed., CRC. Pham, H., 2006. Springer Handbook of Engineering Statistics 1st ed., Springer. Provan, J.W., 1987. Probabilistic approaches to the material-related reliability of fracture-sensitive structures. Probabilistic fracture

References

207

mechanics and reliability(A 87-35286 15-38). Dordrecht, Martinus Nijhoff Publishers, 1987,, p.1–45. Rao, C.R. & Toutenburg, H., 1999. Linear Models: Least Squares and Alternatives 2nd ed., Springer. Rausand, M. & Høyland, A., 2004. System reliability theory, WileyIEEE. Rencher, A.C., 1997. Multivariate Statistical Inference and Applications, Volume 2, Methods of Multivariate Analysis Har/Dis., WileyInterscience. Rinne, H., 2008. The Weibull Distribution: A Handbook 1st ed., Chapman & Hall/CRC. Schneider, H., 1986. Truncated and censored samples from normal populations, M. Dekker. Simon, M.K., 2006. Probability Distributions Involving Gaussian Random Variables: A Handbook for Engineers and Scientists, Springer. Singpurwalla, N.D., 2006. Reliability and Risk: A Bayesian Perspective 1st ed., Wiley. Sison, C.P. & Glaz, J., 1995. Simultaneous Confidence Intervals and Sample Size Determination for Multinomial Proportions. Journal of the American Statistical Association, 90(429). Tong, Y.L., 1990. The Multivariate Normal Distribution, Springer. Xie, M., Gaudoin, O. & Bracquemond, C., 2002. Redefining Failure Rate Function for Discrete Distributions. International Journal of Reliability, Quality & Safety Engineering, 9(3), p.275. References

Xie, M., Goh, T.N. & Tang, Y., 2004. On changing points of mean residual life and failure rate function for some generalized Weibull distributions. Reliability Engineering and System Safety, 84(3), p.293–299.

208

Probability Distributions Used in Reliability Engineering

Xie, M., Tang, Y. & Goh, T.N., 2002. A modified Weibull extension with bathtub-shaped failure rate function. Reliability Engineering and System Safety, 76(3), p.279–285. Yang and Berger, 1998. A Catalog of Noninformative Priors (DRAFT). Yang, K. et al., 2004. Multivariate Statistical Methods in Quality Management 1st ed., McGraw-Hill Professional. Yang, R. & Berger, J.O., 1994. Estimation of a Covariance Matrix Using the Reference Prior. The Annals of Statistics, 22(3), p.1195-1211.

References

Zhou, X.H. & Gao, S., 1997. Confidence intervals for the log-normal mean. Statistics in medicine, 16(7), p.783–790.