Statistical Theory and Methodology in Science and Engineering

Statistical Theory and Methodology In Science and Engineering Second Edition K. A. BROWNLEE Associate Professor of Statistics The University of Chicago

John Wiley & Sons, Inc., New York. London· Sydney

© 1960, 1965

COPYRIGHT

BY JOHN WILEY

& SONS, INC.

All Rights Reserved This book or allY part thereof mllst 1I0t be reprodllced ill allY form withollt the writtell permissioll of the pllblisher.

20 19 18 17 16 15 14 13

ISBN 0 471 11355 7 LIBRARY OF CONGRESS CATALOG CARD NUMBER:

65-12717

PRINTED IN THE UNITED STATES OF AMERICA

Preface

The original intent of this book-to serve for a three-quarter sequence in statistical methods-remains unchanged. The main objective is to develop competence and self-confidence in the use of statistical methods, which require some understanding of the theoretical background and some practice in the application of statistical methods to actual data. This edition assumes that the reader has had a one-year course in elementary calculus. This change has permitted a substantial expansion in the number and scope of the exercises. Topics discussed at greater length than in the first edition include transformations of density functions (Sections 1.13 and 1.14), two X two tables (Section 5.4), Bartlett's test (Section 9.5), the confidence interval for the entire line in linear regression (Section 11.11), the effects of errors of measurement on observations from a bivariate normal population (Section 12.8), and the partial correlation coefficient (Section 13.4). Material added to this edition includes Bayes' theorem (Section 1.7), curtailed binomial sampling (Section 1.11), sampling inspection (Sections 3.15-3.17), queuing theory (Chapter 4), estimation of the parameter of an exponential distribution from truncated observations (Section 5.5), the distribution of numbers of runs of elements of two types (Section 6.3), the Friedman X~ and Cochran Q tests (Sections 7.9 and 7.10), the regression fallacy (Section 12.5), correlation coefficients between indices (Section 13.5), and the likelihood ratio test for goodness of fit in multiple regression (Section 13.8). Room for the foregoing material has been made by omitting from this edition the discussion of weighted regression and the chapter on the multiple regression approach to the analysis of variance. This text is in a form that permits wide choice in the sequence in which v

vi

PREFACE

the material is read. The sketch below shows which chapters are prerequisite to the succeeding chapters:

5

3 6 7

1-2

8-9-10{

1l-12-13 14-15-16-17

More compression is possible; Chapter 12 can be taken up after Section 11.4 and Chapter 13 can be taken up after Section 12.4. Chicago, Illinois September 1964

K. A.

BROWNLEE

Acknowledgnlents

The body of statistical theory and techniques expounded in this book is largely due to Professors R. A. Fisher and J. Neyman and their associates. I am deeply conscious of how feeble and anemic present-day statistical theory and practice would be without their work. My thanks are due R. R. Blough, W. H. Kruskal, H. V. Roberts, and several anonymous reviewers who commented on parts of an earlier draft of this textbook. I am particularly indebted to D. L. Wallace and G. W. Haggstrom, who made very many valuable comments and suggestions. The foregoing obviously have no responsibility for any inadequacies of the present form. My thanks for permission to reproduce data are due the editors of the American Jou/'llal of Public Health, Analytical Chemistry, the Astrophysical Jou/'llal, the Australian Jou/'llal of Applied Science, Chemical Engineering Progress, Food Research, Industrial and Engineering Chemistry, the Jou/'Ilal of the American Chemical Society, the Journal of the Chemical Society, the Jou/'Ilal of Hygiene, the Jou/'I1al of the Institute of Actuaries, the New York State Jou/'Ilal of Medicine, the Philosophical Transactions of the Royal Society, the Proceedings of the American Society for Testing and Materials, the Proceedings of the Berkeley Symposium on Mathematical Statistics and Probability, and Science. My thanks are also due Chapman and Hall for permission to reproduce data from Principles of Biological Assay, by C. W. Emmens, and the New York World-Telegram and Sun for data from the World Almanac and Book of Facts. I am indebted to Professor Sir Ronald A. Fisher, Cambridge; to Dr. Frank Yates, Rothamsted; and to Messrs. Oliver and Boyd, Edinburgh, for permission to reprint parts of Tables III and V from their book Statistical Tables for Biological, Agricultural and Medical Research. I am vii

viii

ACKNOWLEDGMENTS

also indebted.to Professor E. S. Pearson and the Biometrika Trustees for permission to quote extensively from some of the tables in Biometrika Tables for Statisticians, volume 1, edited by E. S. Pearson and H. O. Hartley, and to Dr. A. Hald and John Wiley and Sons for permission to quote extensively from some of the tables in Statistical Tables and For' mulas. K. A. B.

Contents

CHAPTER

1

PAGE

MATHEMATICAL IDEAS

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17

Introduction, 1 Concept of Sample Space, 2 Probability, 6 Conditional Probability, 9 Independence, 11 Exercises lA, 16 Bayes' Theorem, 18 Permutations and Combinations, 20 Exercises 1B, 23 Random Variables, Probability Functions, and Cumulative Distribution Functions, 24 The Binomial Distribution, 30 Curtailed Binomial Sampling, 32 Exercises 1 C, 35 Continuous Frequency Functions, 36 Examples of Continuous Distributions, 42 Transformations of Probability Functions and Density Functions, 45 Another Example of the Transformation of a Density Function,48 The Concept of Expectation, 51 The Expectation of a Function of a Random Variable, 55 The Concept of Variance, 57 Exercises 1D, 61 ix

X

CONTENTS

1.18 1.19 1.20 1.21 1.22 1.23 1.24 1.25 1.26 1.27

2

The Properties of the Standardized Normal Distribution, 63 Discrete Bivariate Distributions, 67 Continuous Bivariate Distributions, 70 Conditional Continuous Density Functions, 72 Independent Continuous Random Variables, 74 Expectation in the Multivariate Case, 75 Exercises IE, 76 Covariance and the Correlation Coefficient, 77 The Variance of a Mean, 80 The Addition Theorem for the Normal Distribution, 81 The X2 Distribution, 82 Exercises IF, 84

STATISTICAL IDEAS

2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18

87

Statistical Inference, 87 Some Principles of Point Estimation, 88 Maximum Likelihood Estimation, 91 A Weighted Mean Unbiased and of Minimum Variance, 95 Some Principles of Hypothesis Testing, 97 A Criterion for Choosing between Alternative Tests, 101 A One-Sided Test of an Observation from a Normal Population with Known Variance, 105 A One-Sided Test of an Observation from a Binomial Distribution, 110 The Testing of Composite Hypotheses, 111 A Two-Sided Test of an Observation from a Normal Distribution with Known Variance, 113 A Two-Sided Test of an Observation from a Normal Distribution with Unknown Variance, 118 The Comparison of Two Means, 119 The Concept of P Value, 120 Confidence Limits: The General Method, 121 Confidence Limits for the Mean of a Normal Distribution with Known Variance, 124 Confidence Limits: The Pivotal Method, 127 Confidence Limits for the Parameter (J of a Binomial Distribution, 129 The Relationship between Confidence Limits and Tests of Hypotheses, 130 Exercises, 133

CONTENTS

3

THE BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20 3.21 3.22

136

The Normal Approximation to the Binomial Distribution, 136 Testing Hypotheses about the Binomial Distribution with a Normal Approximation, 140 The Angular Transformation and Other Variance Stabilizing Transformations, 144 Testing Hypotheses about the Binomial Distribution with the Angular Transformation, 146 Confidence Limits for the Parameter of a Binomial Distribution, 148 Comparison of Two Observed Frequencies with the Normal Approximation, 150 The Correlated Two X Two Table, 154 Exercises 3A, 157 The Hypergeometric Distribution, 158 An Application of the Hypergeometric Distribution to Wild Life Population Estimation, 162 Fisher's Exact Test for Two X Two Tables, 163 The Poisson Distribution, 166 An Alternative Derivation of the Poisson Distribution, 169 Tests of Hypotheses about the Poisson Distribution, 172 Confidence Limits for a Poisson Parameter, 173 Simple Sampling Inspection, 174 Relationship between Sampling Inspection and Hypothesis Testing, 177 Rectifying Inspection, 177 Double Sampling, 180 The Addition Theorem for the Poisson Distribution, 181 The Comparison of Two Poisson-Distributed Observations, 181 The Comparison of Two Poisson-Distributed Observations with the Parameters in a Certain Hypothetical Ratio, 183 An Application to Vaccine Testing, 185 Exercises 3B, 185

4 AN INTRODUCTION TO QUEUING THEORY

4.1 4.2

xi

190

Introduction, 190 Single-Channel, Infinite, Poisson Arrival, Exponential Service Queues, 191

xii

CONTENTS

4.3 4.4 4.5 4.6

5

THE MULTINOMIAL DISTRIBUTION AND CONTINGENCY TABLES

5.1 5.2 5.3 5.4 5.5

6

6.5 6.6 6.7

7.5 7.6 7.7 7.8

221

Introduction, 221 The Mean Square Successive Difference Test, 221 Runs of Elements of Two Types, 224 An Approximation to the Distribution of the Number of Runs of Elements of Two Types, 226 Runs above and below the Median, 231 Control Charts for the Mean and Range, 232 Control Charts for Poisson-Distributed Observations, 235 Exercises, 236

SOME NONPARAMETRIC TESTS

7.1 7.2 7.3 7.4

206

The Multinomial Distribution, 206 The X2 Approximation for the Multinomial Distribution, 207 Contingency Tables, 211 The Two x Two Table, 215 Life Testing, 217 Exercises, 219

SOME TESTS OF THE HYPOTHESIS OF RANDOMNESS: CONTROL CHARTS

6.1 6.2 6.3 6.4

7

Queues with Arbitrary Service Time Distribution, 195 Single-Channel, Finite, Poisson Arrival, Exponential Service Queues, 198 Multichannel, Infinite, Poisson Arrival, Exponential Service Queues, 200 Inventory Control, 203 Exercises, 204

241

The Assumption of Normality, 241 The Sign Test, 242 The Median Test, 246 The Mean and Variance of a Sample from a Finite Population, 248 The Wilcoxon Two-Sample Rank Test, 251 The Adjustment for Ties in the Wilcoxon Two-Sample Rank Test, 253 The H Test, 256 The Wilcoxon One-Sample Test, 258

CONTENTS

xiii

7.9 The Friedman Rank Test, 260 7.10 The Cochran Q Test, 262 Exercises, 265

8

THE PARTITIONING OF SUMS OF SQUARES

8.1 8.2

271

The Distribution of Sample Estimates of Variance, 271 The Partitioning of Sums of Squares into Independent Components, 276 Exercise, 280

9

TESTS OF EQUALITY OF VARIANCES AND MEANS

9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9

282

Introduction, 282 Uses of the Sample Estimate of Variance, 282 The Variance Ratio, 285 The Interrelations of Various Distributions, 288 A Test for the Equality of Several Variances, 290 The One-Sample t Test, 295 The Two-Sample t Test, 297 The Two-Sample Test with Unequal Variances, 299 A Comparison of Simple Tests for Means and Medians, 304 Exercises, 305

10

ONE-WAY ANALYSIS OF VARIANCE

10.1 10.2 10.3 10.4 10.5 10.6 10.7 10.8 10.9

309

Introduction: Models I and II, 309 One-Way Analysis of Variance: Model 1,310 The Problem of Multiple Comparisons, 316 One-Way Analysis of Variance: Model 11,318 Interpretation of a Model II One-Way Analysis of Variance, 321 An Example of a Model II One-Way Analysis of Variance with Equal Group Sizes, 325 An Example of Model II One-Way Analysis of Variance with Unequal Group Sizes, 327 Simple Sampling Theory, 329 The Power Function of Model II One-Way Analysis of Variance, 330 Exercises, 330

xiv 11

CONTENTS

SIMPLE LINEAR REGRESSION

11.1 11.2 11.3 11.4 11.5 11.6 11.7 11.8 11.9 11.10 11.11 11.12 11.13 11.14 11.15 11.16 11.17

334

Introduction, 334 The Model, 335 An Analysis of Variance Representation, 338 An Example of Linear Regression, 342 The Use of the Regression Line in Reverse, 346 The Comparison of Two Regression Lines, 349 Parallel Line Biological Assay, 352 An Example of Parallel Line Biological Assay, 354 Regression through the Origin, 358 The Use of the Regression Line through the Origin in Reverse, 361 A Joint Confidence Region for IX, p, 362 Linear Regression with Several Observations on y at Each x, 366 An Example of Linear Regression with Several Observations on y at Each x, 371 The Comparison of Several Regression Lines: Simple Analysis of Covariance, 376 Simple Analysis of Covariance, 388 Exponential Regression, 391 Regression with Error in the Independent VariabJe Exercises, 391 Exercises, 393

12

THE BIVARIATE NORMAL DISTRIBUTION AND THE CORRELATION COEFFICIENT

12.1 12.2 12.3 12.4 12.5 12.6 12.7 12.8

397

Introduction, 397 Transformations of Bivariate Distributions, 397 The Bivariate Normal Distribution, 401 Some Properties of the Bivariate Normal Distribution, 404 The Regression "Fallacy," 409 Estimation of the Parameters of the Bivariate Normal Distribution, 410 Tests of Significance for the Correlation Coefficient, 413 The Effects of Errors of Measurement, 414 Exercises, 417

CONTENTS

13

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

xv 419

13.1 13.2

Introduction, 419 Linear Transformation of the Variables in a Bivariate Normal Distribution to Give Independent Variables, 419 13.3 Regression on Two Independent Variables, 422 13.4 The Partial Correlation Coefficient, 429 13.5 Correlation Coefficients between Indices, 431 13.6 Regression on Several Independent Variables, 433 13.7 A Matrix Representation, 439 13.8 A Test of Whether Regression on r Variables Gives a Significantly Better Fit than Regression on q Variables, 441 13.9 Polynomial Regression, 447 13.10 Further Uses for the c Matrix, 447 13.11 Biases in Multiple Regression, 452 13.12 An Example of Multiple Regression, 454 Exercises, 462 14

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

467

14.1 14.2 14.3 14.4 14.5 14.6

Introduction: The Model for Model I Analysis, 467 The Analysis of Variance, 471 Computing Forms for Two-Way Analysis of Variance, 475 Two-Way Analysis of Variance: Model II, 478 The Interpretation of a Model II Analysis, 481 Two-Way Analysis of Variance with Only One Observation per Cell, 482 14.7 Nested or Hierarchical Analysis of Variance, 482 14.8 The Two-Way Crossed Finite Population Model, 489 14.9 Discussion of the Two-Way Crossed Finite Population Model,498 14.10 Nested Classifications in the Finite Model, 499 Exercises, 501 15

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

15.1 15.2 15.3 15.4 15.5

The Model, 504 Models I and II, 508 Mixed Models, 511 Confidence Limits in Three-Way Analysis, 512 An Example of Three-Way Analysis of Variance, 514

504

xvi

CONTENTS

15.6 15.7 15.8

16

PARTIALLY HIERARCHICAL SITUATIONS

16.1 16.2 16.3 16.4

17

Orthogonal Contrasts, 517 The Partitioning ofInteractions into Orthogonal Contrasts, 520 Four-Way Analysis of Variance, 524 Exercises, 526

A Partially Hierarchical Situation and Its Model, 530 Calculation of Sums of Squares, Etc., 532 The Expectations of Mean Squares in Partially Hierarchical Models, 538 Confidence Limits in Partially Hierarchical Models, 540 Exercises, 543

SOME SIMPLE EXPERIMENTAL DESIGNS

17.1 17.2 17.3 17.4

530

547

Completely Randomized Designs, 547 Randomized Block Designs, 548 The Split-Plot Situation, 550 Relationship of Split-Plot to Partially Hierarchical Situations, 553 Exercises, 554

APPENDIX

Table I The Cumulative Standardized Normal Distribution Function, 558 Table II Fractional Points of the t Distribution, 560 Table III Fractional Points of the X2 Distribution, 562 Table IV Percentage Points of the F Distribution, 564 Table V y = 2 arcsin 570 Table VI Values of dn and Fractional Points of the Distribution of the Range, 572 Table VII up, 572 Table VIII Logarithms of n!, 573 Table IX Fractional Points of the Studentized Range, 574 Table X Fractional Points of the Largest of k Variance Ratios with One Degree of Freedom in the Numerator, 576 Table XI Random Sampling Numbers, 577

vx,

PARTIAL ANSWERS TO SELECTED EXERCISES

579

INDEX

585

Sta tis tical Theory

and Methodology In Science and Engineering

CHAPTER

I

Mathematical Ideas

1.1. Introduction This is primarily a textbook on statistics, not on probability, and we will deal with the latter only as much as is necessary. The two disciplines are, however, closely related, and are often confused. In probability, a branch c;>f mathematics, we specify the structure of a problem, construct a mathematical model to correspond, specify the values of the parameters (the numerical constants of the system), and then deduce the behavior of the system, e.g., the distribution of the relative number of times each possible outcome will occur. In statistics, we assume the structure of the system and the corresponding model, but not numerical values for the parameters, and from a set of observations on the system we attempt to infer, e.g., the values of the parameters. These characterizations will be clearer from a simple example. A random sample of size n is taken from a lot of N electric light bulbs containing a proportion e of defectives. What is the distribution of the number of defectives, X, in repeated random samples? Specifically, suppose n = 100, N = 10,000, and e = 0.1. We will not get exactly 100 X 0.1 = 10 defectives in every sample. Often we will get 10, in fact more often than any other outcome, but we will also often get 9, and 11, and 8, etc. What proportions of the time in repeated random samples will X equal ... ,6, 7, 8, ... etc.? This is a question in probability. Conversely, suppose that we take a random sample of size n and actually observe X defectives. What can we say about e? For example, what is the most likely value of e, and what is the range of values of e that is reasonably plausible, i.e., consistent in some sense with our observations? These are questions in statistics. Modern statistics is the product of many diverse influences, and some potentially important contributions were lost in the mists of indifference

2

MATHEMATICAL IDEAS

CHAP.

1

of their time. A conscientious historian would have to disinter these and give due credit even though he may be almost the first man to read them since their publication. Nevertheless, some of the main landmarks are generally agreed upon. Studies of gambling problems by the French mathematicians Pascal and Fermat in the year 1654 were the first significant investigations in probability. Over the next two centuries astronomers were interested in the theory of observational errors; in the early nineteenth century Laplace and Gauss made important contributions. For a general account up to the middle of the nineteenth century, see Todhunter [1]. By the beginning of the twentieth century a school under the leadership of Karl Pearson [2] in London was active in statistics, initially from the point of view of its application to biological measurements. An associate of this group, W. S. Gosset, published in 1908 under the pseudonym "Student" a solution to the important problem of the comparison of the mean of a small sample of normally distributed observations with a hypothetical value [3]. Modern statistics may be said to have begun with the appointment in 1919 of R. A. Fisher to the staff of the Rothamsted Experiment Station in England. Fisher's contributions [4-6] are threefold: first, a broad attack on the fundamental principles of estimation and inference, second, the solution of a large number of mathematical problems in distribution theory that were roadblocks to further progress, and third, the creation of the science of the design of experiments, involving three main principles, namely, the essentiality of replication and randomization, and the possibility of reduction of error by appropriate organization of the experiment. In the 'thirties J. Neyman, at that time in London, developed with E. S. Pearson [7] the theory of hypothesis testing and confidence intervals. In the 'forties, A. Wald and his associates of the Statistical Research Group at Columbia University created the ideas and techniques of sequential analysis [8] and statistical decision theory [9]. In recent years the volume of publication has become relatively enormous, much of it inspired, sometimes rather remotely, by the wide variety of practical problems to which statistics is now being applied. It is now difficult to be expert in more than one or two subdivisions of the field. 1.2. Concept of Sample Space The concept of sample space is a convenient method of representing the outcome of an experiment. By experiment we mean some procedure upon which we embark and at whose completion we observe certain results. For example, we may feed a vitamin supplement to a group of hogs and observe their weights after a certain number of weeks, or we may

SECT.

1.2

3

CONCEPT OF SAMPLE SPACE

toss a coin a certain number of times and observe how many times it falls with the head uppermost. The set of all possible outcomes of an experiment is represented by the sample space S. Each possible outcome is represented by a sample point. For example, if our experiment is to drop two light bulbs in sequence, with possible outcomes U = undamaged, F = filament broken, and G = glass envelope broken, then the possible outcomes are as represented in Table 1.1, where the ordering of the symbols within the parentheses ( ) corresponds to the time sequence. Here the sample space contains 9 sample points. Table 1.1 (U, U) (U, F) (U, G)

(F, U) (F,F) (F, G)

(G, U) (G,F) (G, G)

An event is the sum of sample points with some specified property. Thus for Table 1.1 the event "both undamaged" consists of the outcome (U, U). In this case the event consists of only one sample point: Such an event may be called a simple event. The event "one or more glass en:velopes broken" is made up of the sample points (U, G), (F, G), (G, U), (G, F), and (G, G); such an event, which can be decomposed further into a set of simple events, may be called a compound event. Suppose now that we consider a particular experiment. This will give rise to a fixed sample space. Consider an event E defined as a particular set of the sample points. Then all the sample points not in this set form the complementary event "not E," denoted by EO. Consider the foregoing experiment of dropping two light bulbs. Define the event E1 as "one or more undamaged." Then E1 and Ef are as in Figure 1.1. Also, events E1 , E2 may be defined such that a particular outcome may belong in more than one of them. The event E1 n E 2, called "E1 intersection E 2", is made up of those sample points belonging to both E1 and E 2 • For example, if the event E1 as before is "one or more undamaged" (U, U)

(F, U)

(G, U)

(U,F)

(F,F)

(G,F)

(U, G)

(F, G)

(G, G)

~

~

Figure 1.1

- El -+-s

4

MATHEMATICAL IDEAS

I

(U, U)

(F, U)

I

I (U,F)

I L

(F,F)

(U, G)

I

(G, U)

(F, G)

(G,F)

I

CHAP.

1

I I

(G, G)

f-- s

Figure 1.2

El nE2 (G, U)

(U, U)

(U,G)

(F,F)

(G,F)

(F,G)

(G,G)

S

Figure 1.3

(U, U)

(F, U)

(G, U)

1-+--E

,---------------~

(U, F)

(F, F)

(G, F)

(U, G)

(F, G)

(G, G)

1

-+--E3

~________________L_~-_-_~~___J----S

Figure 1.4

and the event E2 is "one or more filaments broken," then E1 and E2 are as in Figure 1.2, and E1 n E2 is made up of the points (U, F) and (F, U), as in Figure 1.3. It is possible for two events Ei and E j to be so defined that Ei n Ej is empty of sample points; in other words, the event Ei n E j is impossible. We then say that E i , E j are mutually exclusive events. For example, if we define El as before as "one or more undamaged" and E3 as "both glass envelopes broken," then E1 n E3 is empty of sample points (Figure 1.4).

SECT.

1.2

5

CONCEPT OF SAMPLE SPACE

(U, U)

I (U, F) (U, G)

I (F, U) I I

(G, U)

J

E1

(G, F)

J

E2

I (F, F)

l

(F, G)

II

___ s

(G, G)

Figure 1.5

A further piece of symbolism is useful. By E1 U E 2, read "E1 union E2," we mean the event "at least one of the two events, i.e., either E1 but not E 2, or E2 but not E1, or E1 and E2 together." For the previous definitions of E1 and E 2, E1 U E2 is shown in Figure 1.5. More generally, for events E1 and E2 defined on an arbitrary sample space S by Figures 1.6a and b, Figures 1.6c and d illustrate (E1 U E2)C and (E1 n E2)C and the relationships (E1 U E 2)" (E1 n E 2 )"

= =

E~

E~,

(2.1)

E~ U E~,

(2.2)

n

which can be proved from the axioms of elementary set theory.

r-------------.s

r-------------~s

Ef (b)

(a)

r-------------~

.----------, S

S

(E1 nE2)C= Ef UE~

(E1 UE2)C= ElnE~

(d)

(c)

Figure 1.6

6

1.3.

MATHEMATICAL IDEAS

CHAP.

1

Probab~lity

As an axiom we associate with every point AI- in the sample space a number, called the probability of Ak , denoted by Pr{A k }. These probabilities must be nonnegative and comply with the condition that (3.1)

where the summation is over the entire sample space. We further suppose that the probability of an event E, Pr{E}, is the sum of all the probabilities of the sample points Ak in E. These axioms lead to some useful rules. In (3.1), if a certain set of the points Ak corresponds to an event E, the complementary set will correspond to the complementary event EC, so E and EC will have associated with them the entire set of points, and therefore

Pr{E}

+ Pr{EC} =

1.

(3.2)

When the events E j and E j are mutually exclusive, there are no sample points belonging simultaneously to both E j and E j ; i.e.,

Pr{E; () E;} = O.

(3.3)

In this situation, Pr{E; U Ej } is the sum of the probabilities of the sample points lying in E j and those lying in Ej , there being no points lying in E; and Ej simultaneously, and (3.4)

This equation holds only when E; and E j are mutually exclusive. If events E 1, ••• ,Ek are all mutually exclusive, then there are no sample points belonging simultaneously to E j and E j for any combination of i and j, and (3.3) holds for any pair of events E; and E j • Also,

Pr{EI

U

E2

U ..• U

Ek } = Pr{E1 }

+ Pr{E2} + ... + Pr{Ek }.

(3.5)

If the events EI> ... , Ek , in addition to being mutually exclusive, are also exhaustive, so that (3.6) then (3.7)

Consider now the case where E i , Ej are not mutually exclusive, so that the event E; () E J is not empty of sample points. In Figure 1.7 we see that the region E; can be split into two parts, E j ( ) Ej and E; () E" and the region Ej can likewise be split into parts, E; () E, and Ef () EJ, the part E j ( ) Ej being common to both E; and EJ • It follows that the event

SECT.

Ei

U

1.3

7

PROBABILITY

Ei can be regarded as made up of three mutually exclusive events, so Pr{Ei

U

E j } = Pr{Ef n E j }

+ Pr{Ei n

Ei}

+ Pr{Ei n

En. (3.8)

We can simultaneously add and subtract Pr{Ei n Ei } to the right-hand side leaving the equation unchanged:

Pr{E i

U

E j } = Pr{Ef n E i }

+ Pr{Ei n

E j } + Pr{Ef n En + Pr{Ei n E i } - Pr{Ei n Ei}'

(3.9)

Now Ef n E j and Ei n E j are mutually exclusive events, so by (3.4)

+ Pr{Ei n

Pr{Ef n E j }

Ej }

=

Pr{(Ef n E j )

U

(Ei n E j )}

= Pr{E j }.

(3.10)

.---------------------~s

Figure 1.7

Likewise

Pr{Ei n En

+ Pr{Ei n

E j } = Pr{(E i n E~) = Pr{Ei}'

U

(Ei n E j )} (3.11)

Thus, substituting (3.10) and (3.11) in (3.9),

Pr{Ei

U

E j } = Pr{Ei}

+ Pr{E

j} -

Pr{Ei n E j }.

(3.12)

As an illustration, consider the experiment of drawing a single card from a well-shuffled deck. The sample space wjll consist of 52 sample points corresponding to the 52 possible cards that might be drawn. Intuitively, if the deck is well shuffled this implies that the probability of anyone card being drawn is the same as that for any other card, i.e., (3.13) But these 52 outcomes are the entire sample space, so by (3.1) Pr{A i } = 1/52 for i = 1,2, ... ,52. Now define the event E1 as the occurrence of a heart. El will contain 13 sample points, all of probability 1/52, so

Pr{E1}

=

Pr{heart}

=

1

13 x 52

= -1 . 4

(3.14)

CHAP.

MATHEMATICAL IDEAS

1

Also define the event E2 as the occurrence of an honor card (ace, king, queen, jack, or ten). There are 4 X 5 = 20 honor cards in the deck, so Pr{E2}

= -5 .

(3.15)

= Pr{heart honor} = -5 .

(3.16)

= Pr{honor} = 20

X -

1

52

13

Further, there are 5 heart honors, so Pr{E1 n E 2}

52

We can now use (3.12) to give us Pr{a heart or an honor or a heart honor} = Pr{E1 U E 2} = Pr{E1}

+ Pr{E2} -

Pr{E1 n E 2}

= 13 + 20,_ 2. = 52

52

52

28. (3.17) 52

This we can readily check as there are 13 hearts, including the 5 heart honors, plus 3 X 5 other honors, making a total of 28 cards which are either hearts or honors or heart honors. We will now develop a formula for the probability of the union of three events, analogous to (3.12). We first note that

We omit a formal proof of this, but in effect it would be a translation into mathematical language of the following argument. The left-hand side defines elements x which belong to Ei and which also belong to either E2 or E3 (or both): the right-hand side defines elements y which belong either to E1 and E2 simultaneously or to E1 and E3 simultaneously (or to both E1 and E2 and E1 and E3 simultaneously); i.e., to E1 and to either E2 or E3 (or both). Thus any x is a y and any y is an x. We will assume without proof the generalization of (3.18): E1 n (E2 u E3

U ... U

Ek)

= (E1 n E 2) U (E1 n E 3) U ... U (E1 n Ek)'

(3.19)

We now consider Pr{E1 U E2 U E3}' The event considered here can be regarded as the union of E1 with E2 U E 3, so from (3.12) Pr{E1 U E2

U

E 3} = Pr{E1} + Pr{E2 U E 3} - Pr{E1 n (E2

U

E3)}' (3.20)

For the last term on the right-hand side we use first (3.18) and then (3.12): Pr{E1 n (E2

U

E 3)}

= Pr{E1 n E 2}

=

Pr{(E1 n E2)

+ Pr{E1 n

U

(E1 n E 3)}

E 3} - Pr{(E1 n E2) n (E1 n E3)}' (3.21)

SECT.

1.4

9

CONDITIONAL PROBABILITY

The last term here can be written as

Pr{(EI

(')

Ea) (') (E1

(')

Ea)} = Pr{EI

(')

E2 (') Ea}.

(3.22)

Substituting this in (3.21), substituting (3.21) in (3.20), and using (3.12) for the term Pr{E2 U Ea} in (3.20), we get

Pr{EI

U

E2

U

+ Pr{E2} + Pr{Ea} -

Ea} = Pr{E1 }

- Pr{EI

(')

Ea} - Pr{E2

(')

Pr{EI

+ Pr{E

Ea}

I (')

(')

E2}

E2 (') Ea}. (3.23)

1.4. Conditional Probability Suppose that the events E1 , E2 are among the possible outcomes of an experiment and that we are interested in Pr{E1 }. Suppose that we are informed that Ea has occurred. What can we now say about the probability of El which we now write as Pr{E1IE2} (read as "the probability of El given E2")? We now are operating, not on the original entire sample space, but on the restricted sample space made up of the points belonging to E2 • This implies that the probabilities of the now restricted set of points At, say Pr{A t IE2}, must be adjusted so that they sum to 1 in the new sample space. This can be achieved by multiplying all the original Pr{A t} which lie in E2 by IfPr{E2}; i.e., we write 1

Pr{AiIEa} = - Pr{E a}

X

(4.1)

Pr{A i},

since if we sum the Pr{A;IE2} over the entire E2 space we get EI

!

Pr{ AilEz}

i

1

Es

Pr{ E2 }

i

= --!

Pr{ At}

1

= -Pr{ Ea}

Pr{ Ea}

X

= 1.

(4.2)

To obtain Pr{E1IE2} we sum the Pr{A i IE2} over those points which lie in E1 ; note, however, that we are already confined to E 2, so this summation is over points in El 0(') E2. Thus ElnEa

Pr{E1IE2}

= !

Pr{A t IE 2 }

t

1

ElnEa

Pr{E 2 }

i

= -- !

_ Pr{ El (') E 2 }

-

Pr{E2}

Pr{A t} (4.3)

This implies

Pr{EI

(')

Ez} = Pr{E2} Pr{E1 IEz}.

(4.4)

10

CHAP.

MATHEMATICAL IDEAS

I

Clearly an analogous argument will give

Pr{E IE } = Pr{E 2 n E 1} 2 1 Pr{E1}

(4.5)

and (4.6)

Pr{E2 n E1} = Pr{E1} Pr{E2IE1}.

For example, suppose that we have a deck of cards from which the 5 diamond honors have been removed. Let the experiment be to draw one card at random from the 52 - 5 = 47 cards in the abbreviated deck.

Let the event E1 be that the chosen card is an honor and the event E2 be that the chosen card is a heart. Then the event E1 consists of 15 sample points each with probability 1/47, so 15 Pr{honor} = Pr{E1} = - = 0.319. (4.7) 47 The event E1 n E2 is the appearance of an honor heart. The number of sample points in E1 n E2 is 5, each with probability 1/47. The number of sample points in E2 is 13, each with probability 1/47. Thus if we catch a glimpse of the card and know that it is a heart, we can then say that the probability that it is an honor is, by (4.3), Pr{honorlheart}

= Pr{E 1IE2} = Pr{E1 n E2} = 5/47 = 2. = 0.385, 13/47

Pr{ E 2 }

13

which is substantially greater than the unconditional figure of 0.319. This is a mild illustration of the bridge proverb, "one peep is worth two finesses. " Considering this situation from another viewpoint, we can calculate the probability of getting a heart honor in two ways, using either (4.4) or (4.6): Pr{honor heart}

= Pr{honor} Pr{heartlhonor} = Pr{E1} Pr{E 2IE1} = 15 X

47

2. = 2.,

(4.8)

:3 = :7 .

(4.9)

15

47

or Pr{heart honor} = Pr{heart} Pr{honorlheart}

= Pr{E2} Pr{E1IE2} = ~~

X

Equation (4.4) extends to three events: write E2 n Ea for E2 to get

Pr{E1 n E2 n Ea} = Pr{E1IE2 n Ea} Pr{E2 n Ea} = Pr{E1IE2 n Ea} Pr{E2IEa} Pr{Ea}.

(4.10)

SECT.

1.5

11

INDEPENDENCE

1.5. Independence

Suppose that the probability of the event E1 is the same whether or not the event E2 occurs, i.e., (5.1) We then say that E1 is independent of E2. Equation (5.1) constitutes a satisfactorily intuitive definition of independence, but we will now show that it implies (5.2) as this latter form is usually more convenient and is often given as the definition. From (4.3) we have

Pr{E IE} = Pr{E 1 n E 2} . 1 2 Pr{E2} We can substitute E~ for E2 to get

(5.3)

Pr{E lEe} = Pr{ E1 n EU .

(5.4)

1

2

Pr{E~}

Now if our definition of independence, (5.1), is satisfied, then the left-hand sides of (5.3) and (5.4) are equal, and hence

Pr{E1 n E2} _ Pr{E1 n E~}., Pr{E 2} Pr{Ea

(5.5)

whence

Pr{E 1 n E2} Pr{Ea = Pr{E 1 n But by (3.2), Pr{En = 1 - Pr{E2}, so

E~}

Pr{E 1 n E2}(1 - Pr{E 2}) = Pr{E1 n

Pr{E 2}.

E~}

Pr{E 2}

(5.6) (5.7)

and hence

Pr{E1 n E2} = Pr{E1 n E2} Pr{E 2} + Pr{E1 n E~} Pr{E 2} = (Pr{E 1 n E2} + Pr{E 1 n Pr{E 2} = Pr{E1 n E2 u E1 nED Pr{E 2} = Pr{E 1} Pr{E2}.

Em

(5.8)

We note also, substituting this in (5.3), that in the case of independence (5.9)

Thus we have shown that our definition of independence, (5.1), implies the usual definition (5.2). The arguments can be used in reverse to show that (5.2) implies (5.1); these two definitions are therefore equivalent.

12

CHAP.

MATHEMATICAL IDEAS

1

Presumably because the word independent and the phrase mutually exclusive have related connotations in ordinary English usage, their probabilistic definitions and implications are sometimes confused. Assume that Pr{El } > 0 and Pr{E2} > O. If the events are mutually exclusive, then the event El n E2 is impossible, and therefore Pr{El n £2} = O. But if PrfEl n E 2 } = 0, then (5.2) cannot be satisfied, and therefore the events El and E2 cannot be independent. Conversely, if the events El and E2 are independent, (5.2) is satisfied, and therefore Pr{El n E 2 } 0 and so the events El and E2 cannot be mutually exclusive. It should be noted, however, that it is possible for-two events El and £2 to be not mutually exclusive and not independent. For example, let El be the event picking a heart and E2 be the event picking a heart honor or diamond honor. Then Pr{El } = 1/4 and Pr{E2} = 10/52. Since Pr{El n E2 } = 5/52 =;6. 0, the events are not mutually exclusive, and since 5/52 =;6. 0/4) x (l0/52) equation (5.2) is not satisfied and therefore the events are not independent.

>

(AlBl)

(AlB2)

(AlB3) •••

(AlBj)

••• (AlBm)

(A2 B l)

(A2 B 2)

(A2 B 3) •••

(A2 B j)

••• (A2 B m)

(AiBl)

(Ai B 2)

(Ai B 3) •••

(AiBj)

••• (AiBm)

(AnBl)

(An B 2) (An B 3) •••

(AIIBj )

• •• (AnBm)

A\12)

B~12) J

812

Figure 1.8

We now need to consider compound experiments made up of two separate independent experiments: consider, for example, the compound experiment formed by first throwing a 6-sided die and secondly drawing a card from a 52-card deck. The outcomes of the first experiment can be represented by points Ai' i = 1, ... ,11 = 6, in the sample space Sl with probabilities Prl{A i }, and the outcomes of the second experiment can be represented by points Bj,j = I, ... ,m = 52, in the sample space S2 with probabilities Pr 2{BJ The outcomes of the joint experiment can be represented by the ordered pairs (AiBj) in the joint sample space S12 with probabilities Pr12 {(A i Bj )}; see Figure 1.8. In this sample space we

SECT.

1.5

13

INDEPENDENCE

can define the event A~12) as the union over all j of the points (AiBS): A~12)

= (AIB1)

U (AiB2) U ... U (AiBm).

(5.10)

Likewise, (5.11) The intersection of A~12) with B}12) is (AiBi)' We now assume that the separate experiments are independent, so that in S12 we have (5.12) We further assume that the probability of Ai is the same whether we consider Ai as occurring in Sl or S12: (5.13) Likewise, (5.14) These assumptions give, on substituting (5.13) an.d (5.14) in (5.12), (5.15) Now suppose that E is an event defined on Sl as the union of all points ieE

Ai conforming to the definition E; i.e.,

U Ai' and likewise F is defined i

jeF

on S2 as UBi' Represent E in the joint sample space S12 by E 12 , i.e., j

m teE E12

Likewise

=

U U (AiBi)' i

t

The occurrence of E in the first experiment and F in the second experiment, i.e., the ordered pair (EF) in the joint experiment, is equivalent in SIB to the intersection of E12 and Fn., i.e., to

14

CHAP.

MATHEMATICAL IDEAS

1

Thus the probability of observing in the joint experiment E followed by F is the product of the probability of E in the first experiment with the probability of F in the second experiment, it being assumed that the separate experiments are independent. Equation (3.12), or the lack of it, is the basis of a common probabilistic fallacy. Suppose that two missiles are fired at a target independently and that each has a probability of 0.2 of destroying the target. The popular misconception is that the probability of the destruction of the target is 0.2 + 0.2 = 0.4. The fallaciousness of this argument is evident if the probability of either missile destroying the target was 0.6; this argument would give 0.6 + 0.6 = 1.2 as the probability of the destruction of the target, an obviously impossible result. The correct answer is obtained as follows. Let E1 be the event "target destroyed by first missile" and E2 the event "target destroyed by second missile." By (3.12) Pr{target destroyed} = Pr{target destroyed by first missile, or second missile, or both}

= Pr{E1 U E2 } = Pr{E1} + Pr{E2 }

Pr{E1 (') E2 }.

-

(5.18)

Here Pr{E1 (') E2 } is the probability of the destruction of the target by both missiles. By (5.17) when the events are independent, Pr{E1 (') E2} = Pr{E1} Pr{E2 }, so Pr{target destroyed}

=

0.2

+ 0.2 -

0.2

X

0.2

= 0.36.

(5.19)

As a similar example, consider an experiment consisting of drawing one card from a deck, replacing it, and drawing another. Let E1 be the event of getting a spade on the first draw and E2 be the event of getting a spade on the second draw. Then Pr{E1} = 13/52 = 1/4 and Pr{E2 } = 13/52 = 1/4. Then Pr{both cards are spades} = Pr{E 1 (') E2 } = Pr{E 1} Pr{E 2 } 1

= "4

1

X

1

"4 = 16 '

(5.20)

since the two draws are independent. We might note in passing that Pr{at least one card is a spade} = Pr{E 1 U E 2 } = Pr{E1}

+ Pr{E 2}

1 1 1 7 =-+---=4

4

16

-

16'

Pr{E 1 ri E 2} (5.21)

SECT.

1.5

15

INDEPENDENCE

or alternatively, Pr{at least one card is a spade}

= =

1 - Pr{neither card is a spade} 1 - Pr{E~ n E~}

= 1-

~

4

X

=

1 - Pr{ED Pr{E~}

~= 2 .

4

(5.22)

16

We might also note that

u E~ n E2 } = Pr{E 1 nED + Pr{E~ n E2 } = Pr{E1} Pr{E~} + Pr{ED Pr{E 2}

Pr{exactly one card is a spade} = Pr{E1 n E~

1

3 4

3 4

1 4

6

=-x-+-x-=-. 4

16

(5.23)

To illustrate the difference dependence may make, consider the related experiment in which first one card and then another are withdrawn from the deck, this time without replacement of the first card. Here Pr{both cards are spades}

= Pr{E1 n Ea} = Pr{E 1} Pr{ EalEl} 13 = 52

12

x 51 =

1 17 '

(5.24)

since on the second draw, if a spade has already been withdrawn from the deck on the first draw then there are only 12 spades in the remaining 51 cards. The result of the second drawing is dependent on the result of the first drawing. Given three events E1, E2 , and Ea, we say that they are pairwise independent if the following equations are satisfied.

Pr{E1 n E2 } = Pr{E1} Pr{E2},

(5.25)

Pr{E1 n Ea} = Pr{E1} Pr{Es},

(5.26)

Pr{E2 n Es} = Pr{E2 } Pr{Es}.

(5.27)

We say that the events are (completely) independent if in addition to (5.25)-(5.27) the following equation is also satisfied:

Pr{El n E2 n Es} = Pr{E1 } Pr{E2} Pr{Es}.

(5.28)

It can be shown that the above definition implies a commonsense equivalent, i.e.,

Pr{E 1 IE 2 n Es} = Pr{EIIE~ n Es} = Pr{E 1 IE 2 n E~} = Pr{EIIE~ n En.

(5.29)

and two similar equations obtained by cyclically permuting the suffices.

16

MATHEMATICAL IDEAS

CHAP.

I

We will prove only the first part of (5.29). Starting with (5.28) and using (5.26), we have

Pr{El () E2 () E3}

= Pr{E 1 () E3} Pr{E2} = Pr{(E 1 () E2 () E3) U (El () E~ () E3)} Pr{E 2} = [Pr{El () E2 () E3}

+ Pr{El () E~ () E3}] Pr{E2}

= Pr{El () E2 () E 3} Pr{E2}

+ Pr{El

()

E~

() E 3} Pr{E 2}.

Pr{El () E2 () E3}(1 - Pr{E2})

=

Pr{El () E~ () E 3} Pr{E 2}. (5.31)

(5.30)

Therefore

Substituting Pr{En for (I - Pr{E2}) and dividing both sides by Pr{E2} Pr{En Pr{E3} gives

Pr{E 1 () E2 () E 3} _ Pr{El () E~ () E 3} Pr{E2} Pr{E3} Pr{En Pr{E3}

(5.32)

Pr{E 1 () (E2 () Ea)} _ Pr{E 1 () (E~ () E 3 )} Pr{E2 () Ea} Pr{E~ () Ea}

(5.33)

or

so (5.34) which was to be proved. The other parts of (5.29) can be proved similarly.

EXERCISES lA IA.I. One card is drawn at random from a standard 52-card deck. Consider the events defined as the card being E 1 , a spade; E2 , an honor (A, K, Q, J, to); Ea, the ace of spades; E4 , the ace of hearts. Answer the following questions, giving reasons for your answers: (a) Are El and E2 independent? (b) Are El and E2 mutually exclusive? (c) Are Ea and E4 independent? (d) Are Ea and E4 mutually exclusive? (e) Are E2 and Ea independent?
Pr{E1 (') E2}, Pr{E1} + Pr{E2}, Pr{E1IE2}' Pr{E1 U E2},

El and E2 are mutually exclusive, (b) under the assumption that El and E2 are independent.

(a) under the assumption that

EXERCISES

1A

17

IA.3. The event E1 is a sub-event of E 2; i.e., if E1 occurs then E2 certainly occurs, but if E2 occurs E1 mayor may not occur. You are also given that o < Pr{E1} < 1, 0 < Pr{E2} < 1. Rank in increasing order of magnitude, using less-than signs and equality signs where appropriate, the following quantities: 0, IA.4.

1,

Pr{E2}, Pr{E1 u E2}, Pr{E1 ( l E2}.

Prove that if E1 and E2 are independent, then

Pr{E1 u E 2} = 1 - Pr{E;} Pr{E:}. IA.S. If E1 and E2 are two events, not independent, defined on the same sample space, prove that Pr{E1} lies between Pr{E1JE2} and Pr{E1JE:}. IA.6. You roll two fair dice, one red and one pink, without observing the outcome. What is the probability that the sum is greater than or equal to 10, conditional on the following two situations: (a) an observer says "the red die is a five," (b) the observer is color blind and says "well, one of them is a five" (not intending by this remark to exclude the possibility that both are five)? IA.7. A deck of cards consists of just 4 cards, namely the ace of spades, the ace of hearts, the 2 of spades and the 2 of hearts. Two cards are dealt sequentially at random face down to Smith. Consider the following four circumstances: (a) He looks at both his cards and says, "I do have at least one ace." (b) He looks at both his cards and says, "I do have at least the ace of spades." (c) He looks at the first card which was dealt to him and says, "I do have an ace." (d) He looks at the first card which was dealt to him and says, "I do have the ace of spades." For each circumstance, calculate the probability, conditional on his remark, that he has both aces. IA.S. The New Republic, (October 13, 1958) discussing the perils of certain foreign policies, remarked that when " ... you are playing Russian Roulette, there is a five to one chance that the hammer will strike on an empty chamber. But there is equally a near certainty that you will be dead after pulling the trigger six times." Under the implied assumption of a one-to-five chance, i.e., a probability of 1/6, of hitting the loaded chamber, calculate the probability the author has in mind. IA.9. Jones and Smith commute jointly to work by automobile and each drives alternate days. When Jones drives the probability of being on time is 01 and when Smith drives the probability of being on time is 02' where 01 > 02' At the end of one week their boss warns them that they will be fired unless they are on time for four or more days in a row in the next five days. Jones says that since his probability of being on time is greater than the probability of Smith being on time, he (Jones) should drive Monday, Wednesday, and Friday and Smith should drive Tuesday and Thursday. Check his claim. IA.IO. An automobile dealer has eight automobiles on his showroom floor. Of the optional equipment power brakes, power steering, and automatic transmission, one automobile is without any, one has power brakes only, two have automatic transmission only, two have power steering only, one has power steering and automatic transmission, and one is "loaded" (i.e., has all three options). Let an automobile be selected at random and let E 1, E2, E3 be the

18

CHAP.

MATHEMATICAL IDEAS

1

events has power steering, has power brakes, and has automatic transmission respectively. (a) Are the events E 1 , E 2, E3 pairwise independent? (b) Are the events E 1 , E 2, E3 completely independent?

1.6. Bayes' Theorem A formula due to the Reverend Thomas Bayes [10] can be used in a particular situation, the nature of which will be clearer after some definitions. Let E10 ..• , En be n mutually exclusive and exhaustive events. By exhaustive we mean that one of the events Ei must occur: since they are mutually exclusive only one can occur. We assume that each of the events Ei has positive probability: if its probability is zero, then it is simply omitted from consideration. We assume, therefore, that

>0 Pr{Ei n E;} = 0 Pr{Ei}

for all i,

(6.1)

for i :;r!: j,

(6.2)

n

Z Pr{Ei } = 1.

(6.3)

i=l

Let F be another event on this sample space. We want to develop a formula for the probability of E j given that F has occurred. By (4.3)

Pr{E .IF} = Pr{E j n F} . , Pr{F} For the numerator we' can use (4.4), Pr{Ej

n F} = Pr{F n E;}

(6.4)

= Pr{FIEj } Pr{Ej }.

(6.5)

We obtain an expression for the denominator as follows. Equation (3.19) is valid for any set of E's, so it will be valid if we substitute F for E1 and E1, ... , En for E2, ... , Ek :

F n (E1 u E2

U ... U

En) = (F n E1)

U

(F n E2)

U ... U

(F n E .. ).

(6.6) But in the present case the E i , i = 1, ... , n, are exhaustive, so by (3.6) the left-hand side of (6.6) is just F n S = F, so we have

Pr{F} = Pr{(F n E1)

U

(F n E2)

U ... U

(F n E .. )}.

(6.7)

Here the events F n Ei and F n Ej are mutually exclusive for i:;r!: j. Therefore we can use (3.5) to write (6.7) as

Pr{F} = Pr{F n E1} + Pr{F n E2 }

+ ... + Pr{F n

En}.

(6.8)

SECT.

1.6

19

BAYES' THEOREM

We again use (4.4) for each term on the right-hand side to get

Pr{F}

= Pr{FIE 1 } Pr{El} + ... + Pr{FIE,,} Pr{E,,}

=!" Pr{FIEi} Pr{Ei}'

(6.9)

i=1

Substituting (6.5) and (6.9) in (6.4) gives Bayes' formula:

Pr{EjIF} = "Pr{FIE j} Pr{E j }. ! Pr{FIEi} Pr{Ei}

(6.10)

i=1

As an example of the use of Bayes' formula, suppose that we have three urns with contents as follows: Urn 1: 4 white balls, 3 black balls, Urn 2: 4 white balls, 5 black balls, Urn 3: 3 white balls, 7 black balls. Suppose that an urn is chosen at random, and that the probability of any particular urn being selected is 1/3. A ball is drawn at random from the urn selected. Suppose that the ball selected is black. We ask, what is the probability that the urn was urn 1 ? There are 26 balls which might be chosen so the sample space has 26 points, 7 having the property El (i.e., coming from urn 1), 9 the property E2 and 10 the property Ea. Fifteen of the points in the sample space have the property F, namely being black. Clearly Pr{FIE1 } = 3/7, Pr{E1 } = 1/3, etc., so (6.10) gives

Pr{E 1 IF} = 3

1

7

3

3 7 5

1 3 1

X -

7

1

-x-+-x-+-x9

3

10

= 0.254,

3

which differs from the a priori probability of 0.333. It is obvious that since the urns differ in content knowledge of the result of drawing a ball at random from the urn selected must affect our opinion as to which the urn was. As another example [11] of the application of Bayes' theorem, suppose that a diagnostic test for cancer has a probability of 0.95 of being positive when the subject does have cancer and a probability of 0.05 of being positive when the subject does not have cancer. In other words, if El is the event "has cancer," E2 is the event "does not have cancer," and F is the event "reacts positively," we suppose Pr{F/E1 } = 0.95, Pr{F/E2 } = 0.05. This hypothetical test has a proportion of 5 per cent "false positives,"

20

CHAP.

MATHEMATICAL IDEAS

1

i.e., of a population without cancer only 5 per cent are reported as positive, since Pr{FIE2} = 0.05. This test has a proportion of 5 per cent "false negatives"; i.e., of a population with cancer only 5 per cent are reported as negative, since Pr{£CIE1 } = 0.05. Now suppose that this test is applied to a population in which the probability of any individual having cancer is 1/200, i.e., Pr{E1 } = 0.005, Pr{E2 } = 0.995. We can now ask: What is the probability that an individual from this population, reacting positively to this test, actually has cancer:

Thus of those tested and showing positive, only 8.72 per cent (I in 11.5) actually has cancer. 1.7. Permutations and Combinations Assume that we have n distinguishable objects and a row of cells numbered 1,2, ... , r, where r ~ n. We wish to know the number of different ways in which a selection of l' of the n distinguishable objects can be placed in the l' distinguishable cells, one object to a cell. The standard symbol for this number is Prn. For filling the first cell we have a choice of n objects. For filling the second cell we have a choice of only n - 1 objects, since one object has already been allocated. The total number of ways in which we can fill the first two cells is therefore n(n - 1). Continuing down the row of cells, when we come to fill the rth cell l' - 1 objects have been used up in filling the previous l' - 1 cells, leaving only n - (1' - 1) objects available from which to make our final choice. Thus

P; =

n(/1 - 1)(/1 - 2) ... (/1 - /'

+ 1).

(7.1)

The factorial symbol ! is such that n! = n(n - 1) ... 3 X 2 X 1. We can thus write ( 1) ... ( /1-/' + 1) (/1 - /')(/1 - /' - 1) ... 2 X 1 P n =/1/1r (/1 - /')(/1 - /' - 1) ... 2 x 1 /1! =--(7.2) (/1 - /')! We note that I! = 1. Also, in general, /1! = n(n - I)!, so (n - I)! = n!/n. To have this equation hold when n = 1 we define O! = 1.

SECT.

1.7

PERMUTATIONS AND COMBINATIONS

21

In the special case where we are placing n objects in n cells, i.e., we are using all the objects, the number of possible arrangements, or permutations, is given by putting r = n in (7.1), so p~

=

n(n - 1) .. ·2

Equation (7.2), with the convention O! pn n

=

n! (n - n)!

=

=

X

1

=

n!

(7.3)

1, gives the same result:

n! O!

=

n!

(7.4)

Now consider that our n objects are not completely distinguishable, as supposed so far, but instead fall into k different groups; the different groups are distinguishable one from another, but within a group the objects cannot be distinguished from other objects of the same group. Let there be n 1 objects of the first kind, n2 objects of the second kind, etc., k

and let! n i

=

n. For example, we might have n 1 red balls, n2 yellow

i

balls, etc. If all the n objects were distinguishable, and we were to arrange all of them in a row, the total number of permutations would be p~ = n!. Now temporarily label the nl indistinguishable objects of the first kind aI' Q2, ... , a 1l1 ; i.e., temporarily make them distinguishable. A certain subset of the whole set of n! permutations will have these a's in certain cells. For example, one such permutation will be

But if the a's are actually indistinguishable, this permutation will be indistinguishable from

in which the same cells are occupied by a's, but the a's are permuted around amongst these particular cells. In both these cases the cells left blank are assumed to be occupied by exactly the same objects other than a's. The number of ways in which we can permute the a's around is p~: = nl!. Similar results hold for the other groups of objects. Thus we have the identity: Total number of permutations assuming the objects are completely distinguishable = (number of distinguishable permutations of n objects made up of k groups of objects indistinguishable within each group) X (number of permutations of objects within the first group assuming that they are distinguishable) X (same for second group) X ••. X (same for kth group). (7.5)

22

MATHEMATICAL IDEAS

CHAP.

1

The left-hand side of this identity is n!. The second, third, etc., terms on the right-hand side are n1! n2! ... nk !, since the permutation of the objects of the first group amongst themselves can be made independently of the permutation of the objects of the second group amongst themselves, etc. Thus we have Number of distinguishable permutations of n1 objects of type a, n2 objects of type b, etc. n! = ----n1! n 2!'" nk !

(7.6)

In the special case where there are only two types of objects, say n1 of one kind and n - n1 of the other, then (7.6) becomes n!

(7.7)

1I1! (n - Ill)!

We now turn to the problem of determining the number of ways of choosing x out of a set of n distinguishable objects. This is known as the number of combinations and is represented by the symbol (:). Other symbols sometimes encountered for this number are C;, nCz, etc. Imagine that the n objects are lined up in a row beside n cells. Suppose that we have x disks labeled "choose" and n - x disks labeled "do not choose." The number of different ways in which we can permute these x disks of one kind and n - x disks of the other kind is, by (7.7), n!/[x! (n - x)!]. But each different arrangement of the disks determines a different choice of the objects lined up beside the cells into which the disks are placed. Thus the number of different choices of the 11 objects x at a time is equal to the number of permutations of the disks, i.e.,

(n)x =

II!

(7.8)

x! (n - x)! .

For example, suppose that we have four different letters a, b, c and d. The possible permutations of these taken two at a time are ab

ba

ac

ca

ad da

bc

cb

bd db

cd dc.

(7.9)

P:

The number of such permutations, by (7.2), is = 4!/(4 - 2)! = 12. The possible combinations of these four letters taken two at a time are ab

ac

ad bc

bd cd.

(7.10)

The number of such combinations, by (7.8), is 4!/[2!(4 - 2)!] = 6. The permutations and combinations differ; in the former the order of the

EXERCISES

IB

23

letters is relevant, so that ab is a different permutation from ba, whereas regarded as combinations ab represents exactly the same choice as ba. Now suppose that we have two a's, two b's and one c; a, a, b, b, c. The possible permutations of all five letters are

aabbc abcab babac bcaab cbaba

aabcb abcba babca bcaba cbbaa.

aacbb acabb bacab bcbaa

ababc acbab bacba caabb

abacb acbba bbaac cabab

abbac baabc bbaca cabba

abbca baacb bbcaa cbaab

(7.11)

The number of such permutations, by (7.6), is 5 !/(2! 2! I!) = 30. Equation (7.6) will give immediately the number of different ways a deck of 52 cards can be dealt 13 cards to each of four players N, S, E, and W. Consider the 52 cards laid out in a row, and 13 disks labeled N, 13 disks labeled S, 13 disks labeled E and 13 disks labeled W being laid out in a row alongside the cards. Each permutation determines an allocation of the cards to the four players, and by (7.6) the number of permutations is 52! = 5.36 X 1028 (7.12) 13!13!13!13! ' using Table VIn of logarithms of n! in the Appendix to assist in the calculation. On the other hand, we could reason from (7.8) that there are

(in

ways of picking 13 cards for N; then 13 cards for S could be picked

from the 39 remaining cards in different deals is

(~n

ways, etc., so the total number of

52! 39! 26! 13 ! ( 52) (39) (26) (13) 13 13 13 13 = 13! 39! 13! 26! 13! 13! 13! O! 52! - 13! 13! 13! 13!'

(7.13)

as before. More generally, an alternative form for (7.6). is

nl!

II!

n2! 1l3! ... Ilk! =

(It) (11 - Ill) (11 III n2

nl 113

n2)

(Ilk)

... 11" .

(7.14)

EXERCISES 1B IB.l. Suppose that a day's production schedule calls for 9000 items. Three machines, each with a daily production capacity of 4000, are available, and the probability that an item is defective is 1, 2 and 4 per cent for machines A, Band C respectively. On a given day 4000 items are produced on machine A, 4000 on machine B, and 1000 on machine C. One item is selected at random from the

24

MATHEMATICAL IDEAS

CHAP.

1

bulked day's production and found to be defective. What is the probability that it was made on machine C? IB.2. A manufacturing process is turning out items and 20 per cent of these on the average are defective. The items are screened for shipment by an inspection operation which is not perfect: if an item is good, then the probability that the inspector reports it as good is 0.90; and if an item is defective then the probability that the inspector reports it as defective is 0.75. A purchaser picks an item at random from a batch shipped to him: What is the probability that it is defective? IB.3. From a panel of ten people we are to select a committee of three. In how many ways can this be done? (a) If we are merely to select three people, (b) if we are to name a chairman, a vice-chairman, and a secretary. IB.4. A panel of ten candidates, seven from party A and three from party B, are to sit on a rostrum facing an audience. Assuming that candidates from the same party are indistinguishable, in now many ways can the rostrum be arranged? IB.5. We ask the same question as in IB.4, but there are five from party A, three from party B and two from party C. IB.6. An investment adviser draws up a list of twelve stocks. (a) In how many ways can a client form a portfolio of five stocks from this list? (b) If the list is made up of four steels, five oils and three office equipments, in how many ways can a client pick a portfolio consisting of two steels, two oils and one office equipment? IB.7. There are ten starters in a horse race. Assuming that there are no ties, and that all horses finish the race, in how many ways can (a) the horses cross the finishing line, (b) the first three places be filled, paying no attention to the ordering 1, 2, 3, (c) the first three places be filled, paying attention to the ordering 1, 2, 3? IB.S. Twelve mice labeled 1 through 12 are to be placed in four cages labeled A, B, C and D, each cage to contain three mice. In how many ways can the mice be allocated to the cages? IB.9. On a camping trip you are badly rained on and the labels on your store of canned food are washed off. There were three cans of meat, two cans of vegetables and two cans of fruit (all cans are of the same size). You open three cans at random. What is the probability that you can get a standard meal, i.e., a meat dish, a vegetable and a fruit for desert? IB.I0. An experimenter is going to run a clinical trial comparing six treatments. He wishes to test every possible choice of three treatments together on a patient, each choice of three treatments to be on a different patient. How many patients will he need (a) if the order of testing a triplet of treatments on a patient does not matter (e.g., ABC is not to be distinguished from BAC, etc.), (b) if the order of testing a triplet on a patient is relevant (e.g., ABC is a different choice from BAC, etc.)? Reduce your answers to simple numbers.

1.S. Random Variables, Probability Functions, and Cumulative Distribution Functions A random variable is a function that takes a defined value for every point in the sample space. For example, consider the sample space for the

SECT.

1.8

RANDOM VARIABLES, PROBABILITY FUNCTIONS

25

experiment consisting of the independent tossing of two fair coins. We can define a random variable X as the total number of heads observed. The values that X can take are represented by x. The sample space consists of the four points (0,0), (0, I), (1,0) and (1,1), and at these points x takes the values 0, I, 1 and 2. Let Al be the occurrence of a head with the first coin and A2 the occurrence of a head with the second coin. The coins are assumed to be fair, and hence Pr{A i } = 1/2. The two throws are assumed to be independent, so 111 Pr{AI n A 2 } = Pr{A 1 } Pr{A 2} = - X - = -. 2 2 4 Likewise,

Pr{A~ n

A2}

= Pr{A1} Pr{A2} = (1 -!)

X

! =!,

224 etc. Thus the probability attached to each point in the sample space is 1/4. The probabilities of the events X taking the values 0, 1 and 2 are therefore 1/4, 1/4 + 1/4 = 1/2 and 1/4, respectively. This can be expressed in tabular form (Table 1.2). The probability that the random variable X takes the value x, Pr{X = x}, is known as the probability function of x and is represented by Px{x}. The sum over all possible values of x of the terms Px{x} must eq~al 1. Table 1.2

o 1/4

2 1/4

1 1/2

As another example, consider the sample space corresponding to the experiment of rolling onto a table a pair of fair 4-sided dice (i.e., regular tetrahedra with sides made up of equilateral triangles). Let the four faces of each die be numbered 1, 2, 3 and 4. If the dice are fair, the probability of a given face of one of the dice being in contact with the table is 1/4, and if the throws are independent the probability of obtaining the result (1, 1) is 1/4 X 1/4 = 1/16, and likewise for all the other possible outcomes. Thus each of the 16 points in the sample space has a probability 1/16. We can define a random variable Yas the sum of the numbers in contact with the surface of the table. The points in the sample space and the corresponding values of yare as in Table 1.3. From this we can Table 1.3

(1,1) = 2 (1,2) = 3 (1,3) = 4 (1,4) =5

(2,1) (2,2) (2,3) (2,4)

=3 =4 =5 =6

(3,1) (3,2) (3,3) (3,4)

=4 =5 =6 =7

=5 =6 =7 (4,4) = 8 (4,1) (4,2) (4,3)

26

CHAP.

MA THEMATICAL IDEAS

1

construct the probability function of the random variable Y, Table 1.4. Note that we could define another random variable Z on the same Table 1.4

y

py{y}

2 1/16

3 2/16

4 3/16

5 4/16

6 3/16

7 2/16

8 1/16

sample space; e.g., the ratio of the larger to the smaller number. The c;ample space, the same as before, and the corresponding values of z are lS in Table 1.5. Table 1.S

(1, 1) (1,2) (1,3) (1,4)

= 1 = 2 = 3

(2,1) (2,2) (2,3) (2,4)

=4

= = = =

2 1 3/2 2

(3, 1) (3,2) (3,3) (3,4)

= = = =

3 3/2 1 4/3

(4, 1) (4,2) (4,3) (4,4)

= = = =

4 2 4/3 1

From this we can construct the probability function of the random variable Z, Table 1.6. Table 1.6

z

1

pz{z}

4/16

4/3 2/16

3/2 2/16

2 4/16

3 2/16

4 2/16

The two sample spaces which we have just discussed contain only a finite number of points, actually 4 and 16 respectively. Such sample spaces are known as discrete. In addition, if the points in a sample space are infinite in number but can be arranged in a simple countable sequence, then that sample space is also regarded as discrete. We will encounter an example of this latter type in Section 3.11, the Poisson distribution. A random variable that is defined on a discrete sample space is known as a discrete random variable, and its probability function is known as a discrete probability function. It is often convenient to represent discrete probability functions graphically, by raising a line on the x axis with height equal to the corresponding probability at each value of x. Tables 1.2, 1.4, and 1.6 thus give rise to Figures 1.9, 1.10, and 1.11. Suppose that the values of x, ordered by increasing magnitude, are Xi' i = 0, 1, ... , N (it is possible, in some cases, e.g., the Poisson distribution, for N to go to infinity). Then define the cumulative distribution

1/4f-

0

JC

2 1 Figure 1.9

0

plyl 1/2r-

3/8 -

1/4 -

1/8 f-

0

I

.'3

2

4

5

6

7

I

8

Figure 1.10

plzl

3/8r-

1/4 f-

1/8 C-

O 0

2

3 4 Figure 1.11 27

z

28

CHAP.

MATHEMATICAL IDEAS

1

junction, sometimes abbreviated as cdf, as

" i } = p{xo} P{x,,} = Pr{X::;; x,,} =LP{x

+ p{X1 } + ... + p{x,,}.

i=O

(8.1)

P{X,,} is the probability that X is less than or equal to x". Tables 1.2, 1.4, 1.6 of probability functions give rise to corresponding cumulative distribution functions, Tables 1.7, 1.8, and 1.9. Table 1.7

x Px{x}

2 1

1 3/4

0 1/4

Table 1.8

y Py{y}

3 3/16

2 1/16

4 6/16

5 10/16

6 13/16

7 15/16

Table 1.9

z Pz{z}

1 4/16

4/3 6/16

3/2 8/16

2 12/16

3 14/16

I I I I I I

7/8

~

6/8 5/8

Plxl 4/8 3/8 2/8 1/8

~

I I

I I I

x

0 Figure 1.12

4 16/16

8 16/16

SECT.

1.8

RANDOM VARIABLES, PROBABILITY FUNCTIONS

29

I

,..--I I

7/8

I I

r----'

6/8

I

5/8

r--'

I I I

Plyl

I I I I

4/8

I I

3/8

r----l I I I

2/8

I

,....---J

1/8

I I I .....------' I

Figure 1.13

These cumulative distribution functions can be represented graphically, as in Figures 1.12, 1.13, and 1.14. It follows from (8.1) that P{XN} = 1, and that p{x.. } = P{x n } - P{x.._1 }. (8.2) I

I

I

7/8

~ I

I I

6/8

,----J

I

5/8

I I I

4/8

,.......J

Plzi

I

I

I

I I

3/8

r' I

2/8

I

,...J I

I

1/8

I I

I

I

0 0

1

2

3

Figure 1.14

4

z

30

MATHEMATICAL IDEAS

CHAP.

I

1.9. The Binomial Distribution Suppose that we make a series of independent trials, and that the outcome of each trial is either A, a "success," with probability Pr{A} = 0, or AC, a "failure," with probability Pr{AC} = 1 - O. We suppose that 0 is constant for all trials and that successive trials are independent. We wish to calculate the probability of exactly x successes in n trials, Pn{x}. First, consider the case where only two trials are made. The possible outcomes are AA, AAc, ACA and ACAc, and since successive trials are independent these outcomes have probabilities OZ, 0(1 - 0), (1 - 0)0 and (I - O)Z respectively. It follows that pz{O} = (1 - O)Z, pz{I} = 20(1 - 0) and pz{2} = Oz. As a check we note that Z

LPz{x} = pz{O} ",=0

= (1

-

+ pz{1} + p2{2} O)Z + 2(1 - 0)0 + OZ = [(1 -

0)

+ O]Z = 1.

(9.1)

Second, consider the case where three trials are made. The possible outcomes and their probabilities are as in Table 1.10. It follows that Table 1.10

Outcomes

Probabilities

AAA AAAc, AAcA, ACAA AAcAc, ACAAc, ACAcA ACAcAc

OZ(1 - 0) 0(1 - O)Z (1 - O)a

= (1 - o)a, Pa{I} Again as a check

Pa{O}

= 30(1

a

+ 3(1 0) + O]a =

LPa{x} = (1 - O)a ",=0

= [(1 -

- O)Z, Pa{2}

O)ZO 1.

~

= 30z(1

+ 3(1 -

- 0) and Pa{3}

O)OZ

= 03 •

+ Oa (9.2)

We will now generalize to the case where n trials are made. The outcome A ... AAC ... AC, where A occurs x times and AC occurs n - x times, and any other permutation of x A's and n - x AC's gives rise to x successes. Altogether, by (7.7), there will be n !/x! (n - x)! such permutations and each will have probability 0"'(1 - o)n-",. Thus

(9.3)

1.9

SECT.'

31

THE BINOMIAL DISTRIBUTION

The binomial theorem asserts that for any two real numbers Rand S and positive integer n (9.4) Inserting fJ for Rand 1 - fJ for S gives [fJ

+ (1 -

fJW =

o:t(:) fJO:(1 -

(9.5)

fJ),,-O:.

Thus P'n{x} is the (x + l)th term in this binomial expansion, which is why (9.3) is known as the binomial probability function. The left-hand

0.41--

0.3r-

0.21--

0.1 -

O~-7--~--7-~I~~--~--~~~~--~+x o 234 5 6 7 8 9 Figure 1.IS

side of (9.5) is automatically equal to 1, since fJ n

!

Pn{x}

= 1, as it must.

+ (1

- fJ) = 1, so

In (9.3) each choice ofn and fJ gives a different

0:=0

distribution; the constants n, fJ are known as the parameters of the distribution. The calculation of Pn{x} when fJ is a simple fraction and n and x are moderately small is comparatively easy. For example, if n = 9, fJ = 1/10, P9

(l}(l _1.\9 _ 9! lOx 99- 0387 10} - O! (9 - O)! 10

{O} _ (9) - 0 10}

9) ( 1 )1( 1 \9-1 9! P9{1} = ( 1 10 1 - 1O} = I! (9 _ I)!

9

-.

11 X98= 109

(9.6) ,

0.387, (9.7)

etc. This binomial probability function is shown in Figure 1.15: for

32

CHAP.

MATHEMATICAL IDEAS

1

x > 3, p{x} is too small numerically to be visible on the scale used. Figure 1.16 shows the binomial probability function with n = 9, () = 1/2: for = 0 and for x = 9,p{x} = (1/2)9 = 1/512 = 0.00195, which is too small to be visible on the scale used. For more awkward instances logarithms can be used: x

loglo PlI{x}

= loglo (~) + X loglo () + (II

- x) 10glo(1 - ()).

(9.8)

pIx! 0.3-

0.2 -

Figure 1.16

Table VIII of loglo II! can be used to evaluate

10glO(~)

= loglo II! -loglo x! -IOglO(1I - x)!.

(9.9)

1.10. Curtailed Binomial Sampling

Consider a contest in which II games are played by two players P and Q. It is agreed that if P wins at least k games, then he has won the contest. Ordinarily, of course, n is odd, and determined by the equation n = 2k - 1. For example, it may be that to win the contest P has to win 4 games out of7. This special relation between 12 and k is not essential to the problem, however. It could be that P is given an advantageous handicap so that he is declared the winner of the contest if he wins 3 games out of 8. In another example, if the fender on our automobile stays on so long as k out of 12 bolts do not vibrate loose, 12 and k are determined purely by the technology of the situation. Suppose that the probability of P winning any game is (), and that successive games are independent. Then the probability of P winning x games is given by (9.3), and the probability of his winning the contest, say A;;(()) is the sum of these terms for x = k, k + I, ... , 11; i.e., (10.1)

SECT.

1.10

33

CURTAILED BINOMIAL SAMPLING

Let us consider a particular case, say with n = 5, k = 3. Looking at the problem from first principles, each of the 5 games has 2 possible outcomes, so the total number of points in the sample space, which we will represent by symbols of the form (WWWWW), (WWWWL), etc., is 25 = 32. The following classes of points correspond to the definition of P winning the contest: (0) (WWWWW)with probability 05.

5

(b) (WWWWL), (WWWLW),etc., making a total of ments each with probability 04 (1 - 0)1. (c) (WWWLL), (WWLWL), etc., making a total of ments each with probability 03(1 - 0)2.

C) G)

such arrangesuch arrange-

The sum of the probabilities of these points can be found directly or from (10.1) as A:(O)

=

J3 (~)

oro(1 - 0)5-ro

= 03(10 -

150

+ 6( 2).

(10.2) (10.3)

It is obvious that once P has won k games there is no point in playing any more: in reality,. the contest will be terminated as soon as P has won k games, or as soon as Q has won n - k + 1 games. The previous analysis is thus unrealistic, although intuitively it is clear that nevertheless it must give the correct answer. A more realistic analysis involves a restricted sample space, since a point (WWWWW) will not exist in reality; if player P won the first three games he would be declared the winner of the contest and the last two games would never be played. The realistic sample space consists of the following classes of points:

(0) (WWW) with probability 03. (b) (LWWW), (WLWW), (WWLW), each with probability 03(1 - 0). (c) (LLWWW) plus other arrangements in which the first four games result in 2 W's and 2 L's, the last game always being a W. There are a

total of

G)

such arrangements, all with probability 03(1 - 0)2.

(d) (LLWWL) plus other arrangements in which the first four games result in 2 W's and 2 L's, the last game always being an L. There are a

total of

(~)

such arrangements, all with probability 02(1 - 0)3.

(e) (WLLL) plus other arrangements in which the first three games

34

CHAP.

MATHEMATICAL IDEAS

1

result in 1 Wand 2 L's, the last game always being an L. There are a total of

G)

such arrangements, all with probability 0(1 - 0)3.

(f) (LLL) with probability (1 - 0)3. The entire realistic sample space therefore consists of 1 + 3 + 6 + 6 + 3 + 1 = 20 points, and the sum of the probabilities ofthese 20 points is of course 1. Those points in classes (a), (b), and (c) conform with the definition of P winning the contest. The probability of P winning the contest, say B~(O), in this instance is B:(O)

= 03 + (~)03(1 - 0) + (~)03(1 = 03(10 - 150 + 6( 2),

- 0)2

(lOA)

(10.5)

which is equal to A:(O) in (10.3). The general form for B~(O) can be obtained by the following arguments:

~ ~~ ,-:,~ { B~(O) = Pr ~+~ ... ~W+ l!:,W'" ~W+'" (~) permutations

k

n-k

(k~ 1) permutations

k

}

+~.

(10.6)

(::=0 permutations consists of k + x

The xth term, for example, games altogether, ending in a W for player P. The preceding sequence of k - 1 + x games must include k - 1 W's, and therefore (k - 1 + x) - (k - 1) = x L's. There will be

(k - ~ + X) such possible preceding sequences. Thus

Pr{winning contest in

k + x games}

= (1 -

OY"Ok(k - ~ + x),

(10.7)

and the probability of winning the contest is

B~(O) = Ok]:(1 _ OY"(k -

! + x).

(10.8)

Since A~(O) and B;(O) give answers to two problems which must have the same answer, we must have

i (n)OIl(l Y

II=~'

0t- = Oknik(l _ O)",(k II

",=0

1 x

+ x),

(10.9)

a rather remarkable identity which can be proved algebraically. In a contest such as the Wodd Series, in which n = 7, k = 4, the number

EXERCISES

Ic

35

of games required to terminate the contest, say x, is of interest. Since either team can win, we have Pr{X

= 4} = 04 + (1

Pr{X

= 5} =

(1)[0 4(1 - 0)

Pr{X

= 6} =

G) [04(1 -

Pr{X

= 7} = (~) [0 4(1

- 0)\

+ 0(1

- 0)4],

O?

+ 02(1

- 0)4],

- 0)3

+ 03(1

-

w].

For 0 = 1/2, the probabilities are 2/16, 4/16, 5/16, and 5/16, and for = 3/4, the probabilities are 328/45, 336/45 , 225/45, and 135/45 • For an intensive examination of the World Series, see Mosteller [12].

o

EXERCISES IC te.t. A professor of psychology writes the following: "Let us now consider whether estimates of the probability of success in a given task obey rules similar to those of mathematical probability or are subject to different, psychological rules. One rule of mathematical probability convenient for such a test is the additive theorem: namely, that small, independent probabilities of a particular event add up to a larger probability. Thus if you are drawing for a lucky ticket in a pool, your chances of success will increase in proportion to the number of tickets you take. In one of our experiments we confronted our subjects with a choice between taking a single large probability or a set of smaller probabilities; e.g., they were allowed to draw either one ticket from a box of 10 or 10 tickets from 100, in the latter case putting back the ticket drawn each time before making the next draw. Mathematically, of course, the chance of drawing the prize ticket was exactly the same in both cases. But most of the subjects proved to be guided mainly by psychological rather than mathematical considerations. "If the 10 draws had to be made from 100 tickets in one box, about four-fifths of the subjects preferred to make a single draw from a box of 10." Comment. tC.Z. An aircraft has 4 engines, 2 on the left wing and 2 on the right wing. Suppose that the probability of anyone engine failing on a certain transocean flight is 0.1, and that the probability of anyone engine failing is independent of the behavior of the others. What is the probability of the crew getting wet (a) if the plane will fly on any 2 engines? (b) If the plane requires at least one (i.e., one or more) engine operating on both sides in order to fly? tC.3. The Chevalier de Mere (1654) wanted to know the following probabilities: (a) probability of seeing one or more sixes in 4 throws of a 6 sided die; (b) probability of seeing one or more double sixes in 24 throws with a pair of dice. The Chevalier thought that these two probabilities should be the same, but he

36

CHAP.

MATHEMATICAL IDEAS

1

threw dice so assiduously he convinced himself they were different. Evaluate these two probabilities. IC.4. Suppose that the probability that a light in a classroom will be burnt out is 1/4. The classroom has six lights, and is unusable if the number of lights burning is less than four. What is the probability that the classroom is unusable on a random occasion? IC.S. "The United States SAC (Strategic Air Command) is supposed to be based upon about fifty home bases. If the Soviets happened to acquire, unknown to us, about 300 missiles, then they could assign about six missiles to the destruction of each base. If the Soviet missiles had, let us say, one chance in two of completing their countdown and otherwise performing reliably, then there would be ... (about) an even chance that all the bases would be destroyed, about one chance in three that one base would survive, and a small chance that two or more bases would survive." (From Herman Kahn, "The Arms Race and Some ofIts Hazards," Daedalus, Volume 89, No.4 of the Proceedings of the American Academy of Arts and Sciences, p. 744-80, 1960.) Calculate the numerical value of the "small chance that two or more bases would survive." IC.6. Large tomatoes are packaged at random three to a container. Let (J be the probability that a tomato has some flavor. Let E1 be the event that not more than one tomato in a container has some flavor, and let E2 be the event that at least one tomato has some flavor. (a) For what values of (J, excluding 0 and 1, are E1 and E2 independent? (b) As in (a), but suppose that a container contains four tomatoes. IC.7. Consider making a sequence of independent trials with probability of success at each trial equal to (J. Let X be the number of trials necessary to observe a success. (a) Show that the probability function is

Px{x} = (1 -

x

(J)r<-1(J,

~

1.

This is known as the geometric distribution. (b) Show that the corresponding cumulative distribution function is

Px{x} = 1 - (1 -

(J)"'.

> a} and Pr{x ::;; a}? IC.S. Consider making a sequence of independent trials with probability of success at each trial equal to (J. Let Y be the number of trials necessary to achieve a successes. Show that the probability function is (c) What is the relationship between Pr{x ::;; 2alx

py{y} = ( y

-1)

a-I

(Ja(l -

(J)y-a,

y~a.

This is known as the Pascal distribution.

1.11. Continuous Frequency Functions So far we have only considered discrete probability functions, in which the probability is zero except for a discrete set of points on the line. We

SECT.

1.11

CONTINUOUS FREQUENCY FUNCTIONS

37

will now consider the case where X can vary continuously along the line. We will approach this question empirically. Consider a very large population of some item, say cucumbers, from which we take a sample of 100 and measure the lengths to the nearest 4 em, in such a way that anything between 22 and 26 is recorded as 24, etc. We can regard this as a discrete distribution, in which the cucumbers have lengths confined to nominal values 24, 28, 32, etc., and represent it by a line graph, Figure 1.17, in which the height ofthe line at each nominal

pjx! 0.4

0.3 0.2 0.1 0~--~--~~--~--~36~--~--~~--~--~x

Figure 1.17

length of cucumbers is equal to the fraction of the sample with that nominal length. For the hypothetical data in Table 1.11, since the sample size is 100, the observed numbers must be divided by 100 to give the estimated values of p{x}, the fourth line in the table. Alternatively, we can replace the line graph by a histogram, a figure in which rectangles replace the lines, with the area of each rectangle equal to the fraction of the sample with each nominal size. The vertical scale must be adjusted so that the total area is 1. Since the base of each rectangle is 4 cm, the heights of the rectangles have to be one quarter of the corresponding lines (fifth line in Table 1.11). This is given in Figure 1.18. Now suppose that we take a sample of twice the initial size, and measure the cucumbers to the nearest 2 cm, so that anything between 22 and 24 is recorded as 23 em, etc. The observed numbers might be as in the eighth row of Table 1.11. We again construct a histogram in which the area of each rectangle is equal to the fraction of the sample with each nominal size. The base of each rectangle is 2 em, so the p{x} for a line graph, ninth row, have to be divided by 2 to make each area correct and to make the total area equal to 1. This gives us Figure 1.19. Now suppose that the process is repeated indefinitely, larger and larger samples being taken and measured to finer and finer intervals. Then in the limit we will obtain a smooth curve for the histogram, as indicated in Figure 1.19. Such a curve is known as a continuous probability density distribution, and the function p{x} defining the curve is known as the probability density function.

Nominal length Actually the interval Number in sample p{x} for line graph p{x} for histogram

Nominal length Actually the interval Number in sample p{x} for line graph p{x} for histogram

28 26-30 10 0.10 0.025

32 30-34 18 0.18 0.045

36 34-38 36 0.36 0.090

40 38-42 18 0.18 0.045

44 42-46 10 0.10 0.025

48 46-50 4 0.04 0.010

> ::l (') > t""

m

~

a::

~.....

23 25 27 29 31 33 35 37 39 41 43 45 47 49 22-4 24-6 26-8 28-30 30-2 32-4 34-6 36-8 38-40 40-2 42-4 44-6 46-8 48-50 S m 20 2 6 8 12 16 36 36 20 16 12 8 6 2 > Vol O.oI 0.03 0.04 0.06 0.08 0.10 0.18 0.18 0.10 0.08 0.06 0.04 0.03 O.oI 0.005 O.oI5 0.020 0.030 0.040 0.050 0.090 0.090 0.050 0.040 0.030 0.020 0.015 0.005

24 22-26 4 0.04 0.010

Table 1.11

w

00

SECT.

1.11

39

CONTINUOUS FREQUENCY FUNCTIONS

plxl 0.10 0.09 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 O~~~~~~~L-~-L~~~~~~-L-J__~x

Figure 1.18

The probability that an observation X taken at random from a population with density function p{x} lies between Xa and Xb is given by the area under the curve between Xa and Xb (Figure 1.20). The operation of finding such an area is handled mathematically by the technique of integration, denoted by the symbol

It follows that for the function p{x} to be a probability density it must

0.09 0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0

22

26

30

34

38

Figure 1.19

42

46

50

40

CHAP.

MATHEMATICAL IDEAS

1

------------------~----~X

Xa

Xb

Figure 1.20

satisfy the condition that p{x} ~ 0 for all x and that (11.1)

L:p{x} dx = 1. Also, we see from Figure 1.20 that

Pr{xa

<X <x

b}

=

f"'bp{X} dx.

J",_

(11.2)

From the left-hand shaded area of Figure 1.21 we have Pr{X

< xa} = J::p{x} dX,

(11.3)

and from the right-hand shaded area Pr{X> xb }

= fc:tJp{x} dx.

(11.4)

J"'b

In dealing with continuous distributions, we replace the summation operation that sufficed for discrete distributions by the integration operation. Thus the cumulative distribution function in the continuous case, analogous to (8.l) in the discrete case, is

P{x} = Pr{X

< x} = J:c:tJP{f} dt.

(11.5)

Prlx<xa !

=-----_X Figure 1.21

SECT.

1.11

CONTINUOUS FREQUENCY FUNCTIONS

41

From a given probability density function p{x} we can compute P{x} for every x, and construct its graphical form. P{x} must start at zero on the left-hand end, since

f~}{x} dx =

0,

and reach 1 on the right-hand side, since

L:p{x} dx = 1 by (11.1). The upper part of Figure 1.22 gives the cumulative distribution

p{x} corresponding to the density function p{x} in the lower part of the figure. A cumulative distribution function graph has the property that

Plxl

plx+~xl~------------~

P

lxl f - L - - - - - - - E - - - I O~~~------~-+------------------------~x

plxl Prlx<x<x+~xl

01..-:=--------

x

---------~----~x x+~x

Figure 1.22

42

MATHEMATICAL IDEAS

CHAP.

1

its slope can never be negative, since the probability density function from which it is derived can never be negative. By the elementary rules of calculus, the derivative of a function f(x), denoted by the symbol df(x)/dx, is equal to the slope of the function at the point x. Furthermore, the derivative of an indefinite integral of a function is the function itself; i.e., the derivative of (11.5) with respect to x is

dP{x} - = -d

J'"

p{t} dt = p{x}, (11.6) dx dx-oo so the slope of the cumulative distribution function is equal to the probability density function at that point. To see this graphically, in the upper part of Figure 1.22, Pr{x

< X < x + ~x} =

P{x

+ ~x} -

P{x},

(11.7)

and in the lower part of Figure 1.22, Pr{x

< X < x + ~x} =

1

,"+4,"

'"

p{t} dt ~ p{x} ~x.

(11.8)

Since the left-hand sides of (11.7) and (11.8) are identical, we can equate the right-hand sides, whence

P{x

+ ~x} ~x

P{x} '" _p{} x.

(11.9)

When we take the limit of ~x tending to zero, the approximate equality sign tends to an equality sign, and the limit of the left-hand side is just the definition of the derivative of P{x}: lim P{x .6.'" .... 0

+ ~x} ~x

- P{x} = dP{x} = p{x}. dx

(11.10)

1.12. Examples of Continuous Distributions Any continuous functionf(x) which is never negative and for which

L!(X)

dx = 1

can be regarded as a probability density function. The rectangular or uniform distribution has the density function

p{x} = _1_, c-b = 0,

b

< x< c;

otherwise.

(12.1)

SECT.

1.12

43

EXAMPLES OF CONTINUOUS DISTRIBUTIONS

The parameters band c must be such that c - b case of b = -a/2, c = a/2, this becomes

p{x}

a

For the special

a

=1.,

-2:<x<2:'

=0,

otherwise,

a

> O.

(l2.2)

and is graphed in the upper part of Figure 1.23. The cumulative distribution function is P{x}= f OJ p{t}dt=-1 flU dt=-tlOJ

a

-Q()

=

~ [x -

-a/S

a

-a/S

(- ~) ] = ; + ~

(12.3)

for x satisfying the inequality -a/2 < x < a/2. Forx = -a/2,p{x} = 0; for x = 0, P{x} = 1/2; and for x = a/2, P{x} = 1. The cumulative distribution function is thus zero up to -a/2, a straight line with slope l/a from (-a/2, 0) to (a/2, I), and 1 for x > a/2, as shown in the lower part of Figure 1.23.

plxl

I/a

--------a~/2~------~O--------a~/2----------~x

Plxl

--------~------~--------~----------~x

Figure 1.23

Another simple family of continuous probability density functions is the negative exponential:

p{x}

= ()e- IJIU, = 0,

0;:5; x < 00, otherwise,

(12.4)

44

CHAP.

MATHEMATICAL IDEAS

1

where 0 is a fixed parameter greater than zero, and e is the base of natural logarithms. The cumulative distribution function is P{x}

= 0LGle- lIt

dt

= o( -

~ e-

lIt)

I: =

_e-llt

I: (12.5)

The probability density function (12.4) and the cumulative distribution function (12.5) are graphed in Figure 1.24 for the case 0 = 0.5. Each value of 0 will give a different but similar distribution. 1.0 0.9 0.8 0.7 0.6

p1xl

0.5 0.4 0.3 0.2 0.1 5

6

0

Figure 1.24

A particularly important family of continuous probability density functions is the so-called normal or gaussian:

1 e-(0:-51 S B p{ x} = --==/2a , ../217 (1

(12.6)

where ~ and (1 are fixed parameters, 17 has its traditional meaning, and e is the base of natural logarithms. The cumulative distribution function is P{x} =

fGl

1

-J217

(1

e-
(12.7)

-w

involving an integration that does not lead to a standard explicit form, and the integration has to be performed numerically. We will often write (12.6) as N(~, (12).

SECT.

1.13

45

TRANSFORMATIONS OF PROBABILITY FUNCTIONS

1.13. Transformations of Probability Functions and Density Functions We are sometimes faced with the problem of finding the distribution of a function Y = f(X) of a random variable X, already knowing the distribution of X itself. For example, suppose that the distribution of diameters D of a certain population of ball bearings is normal. The volume of a sphere is t'ITT3 = 7Td3/6 and if the density of the material is p then the weight w = p7Td3/6. We may now want to know the form of the distribution of the weights W. We assume that y = f(x) is a strictly monotonic function of x, and therefore x = gW) is a strictly monotonic function of y. In the discrete case things are simple. Let py{y} be the probability function of Y. Whenever X takes a particular value x, which it does with probability Px{x}, then Y takes the corresponding value y = f{x}, also with the same probability Px{x}, so

Px{x}

= Px{gW)} = py{y}.

(13.1)

As an example, consider the probability function of Table 1.4. This can be represented by the formula 1 16

py{y} = -(4 -Iy - 51), Consider the function v = g(v) = .J~. Then

y = 2, 3, ... ,8.

= fw) = y2,

(13.2)

which has the inverse function

'JJ

py{v}

= py{g(v)} = 116 (4 - IJii -

51),

v = 4,9, ... ,64.

(13.3)

To illustrate this, consider the case when 'JJ = 3. Then v = 9. From (13.2), 1 2 (13.4) py{y = 3} = - (4 - 13 - 51) = - , 16 16 and from (13.3), 1 2 (13.5) py{v = 9} = - (4 51) = -. 16 16

1.J9 -

The continuous case is more troublesome. Figure 1.25a s~ows Y as a function of x, y =f{x}, and x as a function ofy, x = g{y}. Figure 1.25b shows the density function of x,Px{x}. Now whenever y falls in the interval WI' Yl + 6.Yl), then x must fall in the interval (Xl' Xl + 6.xl), i.e., in the interval (g(Yl), gWl + 6.Yl))' Therefore the probabilities of these

W

(b)

U

i

,,%

1.0

~ pyly}

~

is:

~

::z:: m

0.33t

FA'

%2%2+'--

p x {%}

IVAI

.L

,

Figure 1.25

-

£ :a>

r

i

£ .~

-3

7'

~

~

L

y

-2

2

y

SECT.

1.13

47

TRANSFORMATIONS OF PROBABILITY FUNCTIONS

two events must be equal: PY{Yl} AYI = Px{X1}[g(Yl

+ AYl) -

(13.6)

g(Yl)],

whence, dropping the subscript and substituting g(y) for x: py{y} = Px{g(y)} g(y

+ Ay) -

g(y) .

(13.7)

Ay

In the limit Ay -+ 0 we obtain py{y} = Px{g(y)} dd g . Y

(13.8)

In words, the probability density function for Y equals the formula for the probability density for X, with g(y) '= x substituted for x, multiplied by the derivative of g(y) with respect to y. The foregoing discussion assumed that y was a strictly increasing function of x everywhere. The same result is readily obtained if y is a strictly decreasing function of x everywhere, the only difference being that in (13.8) in place of dg/dy the absolute value of dg/dy, namely Idg/dyl, appears: py{y} = Px{g(y)}

I~~ I·

(13.9)

As an example, consider the normal density function (12.6). As it stands, it is a different function for every pair of values of and C1, and it is obviously impossible to tabulate an infinity of normal distributions. However, all these normal distributions can be reduced to a common form, the standardized normal distribution, by a simple transformation. Define

e

y =f(x)

x-e =-= C1

x e

- - -.

C1

C1

(13.10)

The inverse function x = g(y) is obtained by solving (13.10) for x: x = g(y) = C1y

+ e.

(13.11)

The derivative of g(y) with respect to y is dg(y)/dy = C1. We now substitute in (13.8): 2 1 2 1 py(y) = --=- e- IJ /2 • C1 = -= e-'Y /2 .J27r C1 .J27r This is the standardized normal distribution, for which we reserve the symbol cp(u): (13.12)

48

CHAP.

MATHEMATICAL IDEAS

1

The cumulative form, for which we use the symbol
=J"

(13.13)

1 e- t2/ 2 dt.

-0)

.J27T

A random variable with the probability density (13.12) is often referred to as a unit or standardized normal deviate. The properties of cp(u) and of
x

= gl(Y) = -.Jy = g2(Y) = /i;

when -

00

< x < 0, < x < 00.

°

when

(14.1)

When x is in the interval gl(Y), gl(Y + /).Y), then Y is in the interval Y + /).y, and also when x is in the interval g2(Y), g2(Y + /).Y), then Y is again in the interval y, Y + /).y. The corresponding probabilities must be equal:

y,

py{y} /).Y

= PX{gl(Y)} /gl(Y + /).Y) -

gl(Y)/

+ PX{g2(Y)} /g2(Y + /).Y) whence, dividing by

/).Y

and taking the limit as

py{y} = PX{gl(Y)} 1:~ll

/).Y --+

~(1

9a

- ::), a

= 0,

(14.3)

Figure 1.27a, when

< x < a, (14.4)

= 1. The inverse functions are

= gl(Y) = -.J;'

x = g2(Y)

-2a

X2,

otherwise.

Figure 1.27 is drawn with a x

0,

+ PX{g2(Y)} 1~ I·

As an example, consider the function Y = I(x) = x has the density function (Figure 1.27b)

Px{x} =

- g2(Y)/' (14.2)

=.J;'

-2a

< x < 0, < x < a.

°

(14.5) (14.6)

SECT.

1.14

49

TRANSFORMATION OF A DENSITY FUNCTION

~------------------------~~o ~

....

+

....

:0-

~

______________________~~~~~______~~____ E J;

50

MATHEMATICAL IDEAS

.....

CHAP.

1

SECT.

1.15

THE CONCEPT OF EXPECTATION

51

Application of (14.3) gives (14.7)

=~(1 + "/Y), 9a../y

{

a

and PY(2)

<x <0 o < y < 4a 2 ,

-2a

2 ( 1 - -;; 11:2 y-~ I = 9a

"/Y)

{} Y

=

_1 (1 _"/Y) 9a../Y a'

(14.8)

(14.9)

o<x
(14.10)

0< y < a 2•

Adding these two functions gives

py{y} =

=

2 j-' 9ayy _1_(1

9a../y

+ "/Y) = _1_ + _1 a

9a../y

9a 2 '

= 0,

otherwise.

(14.11)

This density function is plotted in Figure 1.27c. 1.15. The Concept of Expectation Two of the most useful properties of a random variable are its expectation (or expected value or mean) and its variance. In this section we will discuss expectation, defined as E[X] = ~XiP{ Xi},

in the discrete case,

(15.1)

in the continuous case.

(15.2)

i

=

L:

xp{x} dx,

The usefulness of the concept of expectation is that it corresponds to our intuitive idea of average. Consider the binomial distribution of Figure 1.15 with n = 9, (J = 0.1. The random variable x could be the number of defective electronic tubes found in random samples of 9 from a population of tubes whose fraction defective is O. I. The numerical values of p{x} calculated with (9.3) are given in the first column of Table 1.12. If

52

CHAP.

MATHEMATICAL IDEAS

1

we imagine drawing 1,000,000 samples, we would expect to find adefectives in the sample 1,000,000 X 0.387,420 = 387,420 times, etc. Thus the total number of defectives found in the 1,000,000 samples would be

a x 387,420 + 1 x 387,420 + 2 x

172,187

+ ...

= 899,999,

so the average number of defectives per sample is 899,999/1,000,000 = 0.9. This calculation is exactly that ofthe expected value, (15.1). The expected value is not necessarily the most frequent value (which is known as the mode of the distribution). In fact, for a discrete distribution the expected Table 1.12

x

p{x}

1,000,000p{x}

x x 1,000,000p{x}

0 1 2 3 4 5 6 7 8 9

0.387420 0.387420 0.172187 0.044641 0.007440 0.000827 0.000061 0.000003 0.000000 0.000000

387,420 387,420 172,187 44,641 7,440 827 61 3 0 0

0 387,420 344,374 133,923 29,760 4,135 366 21 0 0

value may never occur at all. For example, the binomial distribution is confined to the integers 0, 1, 2, ... , n, and there is no reason in general why its expectation, which we shall show is nO, should be an integer. In the present example, for instance, there are actually two most-frequent values, a and 1, and the expected value 0.9 is equal to neither of them. If we visualize a distribution plotted on stiff card and cut out, and then balanced on a knife edge parallel to the p{x} axis, it will balance at the expected value of X. The expected value is equal to the center of gravity of the distribution along the x axis. A constant can be regarded as a discrete distribution entirely at one point. Thus in E[a] = ~ ap{a}, the summation is just a single term and p{a} = 1; hence E[a] = a. Thus the expectation of a constant is that constant. Also, since E[X] is a constant, E[E[X))

= E[X].

(15.3)

To consider a simple example of the expected value of a distribution, Table 1.13 shows the calculation of the expectation ofthe random variable Yequal to the sum of two 4-sided dice whose frequency distribution was given in Table 1.4.

SECT.

1.15

THE CONCEPT OF EXPECTATION

53

Table 1.13

Yi

P{Yi}

YiP{Yi}

2 3 4 5 6

1/16 2/16 3/16 4/16 3/16 2/16 1/16

2/16 6/16 12/16 20/16 18/16 14/16 8/16

7 8

E[Y] = LYiP{Yi} = 80/16 = 5 i

A more interesting example is the expectation of the binomial distribution (9.3). By the definition of expectation for a discrete random variable, (15.1), (15.4)

We note that X(II) XII! _ (II - i)! 11- (n - 1)11 . X - X! (II - x)! - (x - i)! (II - x)! - x - 1 '

so (15.4) can be written as E[X]

=

t (: := Dore-l(l -

1I0re

0t- re •

(15.5)

(15.6)

We have changed the lower limit of the summation from 0 in (15.4) to 1 in (15.6), for when x = 0 the corresponding term in the summation is zero and adds nothing to the summation, and hence can be omitted. Equation (15.6) will appear in a more satisfactory form if we write x = Y + 1, so x-I = Y, and n = m + 1, so n - 1 = m. In (15.6), x ranges over 1 to n, or 1 to m + 1, so Y ranges over 0 to m; thus E[X] = 110

i (m)OIl(l _ Y

o)m-II.

(15.7)

11=0

The summation following nO is the sum of all the terms of a binomial distribution for samples of size m, and the sum of any probability function must equal 1. Hence E[X] = nO. (15.8) To turn to continuous distributions, the rectangular distribution, (l2.2),

54

CHAP.

MATHEMATICAL IDEAS

1

has the expectation, using the definition (15.2), E[X]

=

f

a/2

-a/2

1 x . - dx a

2

x \a/2 = -1 ~ = O. a 2

(15.9)

-a/2

The expectation of the exponential distribution, (12.4), is E[X] = L':> xOe-o", dx.

(15.10)

If we use the rule for integration by parts, (15.11)

If(x)gl(X) dx =f(x)g(x) - I g(x)f'(x) dx,

and identify I(x) with x so that f'(x) = 1, and g'(x) with Oe-o", so that g(x) = -e-o"" we get E[X]

= L')xOe-O"'dx =

x(-e-O",)

I: -

L')(-e-O"') ·1· dx

(15.12) For the normal distribution, E[X]

= L:xp{x} dx =

(I2.6~,

=L:[e

+

(x - e)]p{x} dx

efoo p{x} dx +foo (x -co

-00

e)

1 e- C",-s)2/ 2.,2 .J21T C1

dx.

(15.13)

Since the integral over the whole line of any probability density function is I, the first term on the right-hand side is To evaluate the second and obtain integral, we make a change of variable to y = x -

e.

e

C1 e- lI2/2"21 CO = 1 fOO ye- lI2/2.,2 dy = _ -= -=-.J21T C1 -00 .J21T-CO Thus for the normal distribution E[X]

=

e.

o.

(15.14)

(15.15)

We have defined two of the classical so-called measures of central tendency, the expected value or the mean and the mode. The third, the median, say x"" is that value of x which has half the mass of the distribution below it and half the mass above it. Thus if we pick an X at random it is equally likely to lie above as to lie below the median X m • For symmetric distributions, e.g., the rectangular or the normal, the median coincides with the expectation, but for asymmetric distributions this will not usually

SECT.

1.16

A FUNCTION OF A RANDOM VARIABLE

55

be the case. For example, for the exponential distribution (12.4) we find the median Xm by requiring that Xm satisfy the condition that Pr{X < xm} = 1/2. But from (12.5) we have 1 (15.16) Pr{X < x".} = P{x m } = 1 - • e0 ::em I

Equating these and solving for x1/!' we obtain Xm

=!

loge 2

= 0.69315 .

()

(15.17)

()

Incidentally, we recall from (15.12) that the expectation of the exponential distribution is l/(), so the median is 0.693E[X], and Figure 1.24 shows that the mode of the exponential distribution is zero.

1.16. The Expectation of a Function of a Random Variable Suppose that Y = f(X) is a monotonic function of X and X = g(Y) is likewise monotonic. We will need to determine the expected value of Y, given the density function of X: by the definition (15.2), (16.1)

E[Y] = L:Ypy{y} dy.

To evaluate E[ Y] in this form involves finding py{y}. However, for functions with the properties specified in Section 1.13 on the transformation of distributions, so that y = f(x) is a single-valued function of x and x = g(y) is a single-valued function of y, we can substitute f(x) for y and Px{g(y)} Idgldy I for py{y}, from (13.9), in (16.1). We also need to change the variable of integration from y to x. The rule for change of variable when v = cp(u), u = 1p(v) is

J. h(v) dv = b

a

fljl(b)

h(cp(u»CP'(lI) duo

(16.2)

VI(a)

To apply this formula, replace v by y, u by x, cp(u) by f(x), 1p(v) by g(y) , and h(v) by ypy{y}. Changing the variable in ypy{y} to x gives f(x)p.-dg(y)} dxldy, corresponding to h(cp(u». Corresponding to cp'(u) = dvldu we have dyldx. Also when y = 00, x = g(oo) = 00, and when y = - 00, x = - 00. Therefore

E[f(X)]

=f

OO

-00

ypy{y} dy

= foo f(x)px{x} -00

I-dx I-dy dx dy dx

= L!(X)px{x} dx.

(16.3)

56

CHAP.

MATHEMATICAL IDEAS

1

In other words, we can obtain E[f(X)] directly from Px{x} and we do not need to determine Pflx,{f(x)}. The analogous formula holds for discrete distributions: (16.4) E[f(X)] = !f(xi)p{Xi }. i

The foregoing proof of (16.3) is of course valid only under the stated conditions of monotonicity. Equation (16.3) is, however, valid quite generally, without the restrictions of monotonicity. A proof of this can be found in Cramer [13], Sections 7.4 and 7.5. As an example, consider Y = f(X) = X2 where X has the density function (14.4) and Y the density function (14.11). We could find E[Y] literally from its definition as E[Y]

= L: yp{y} dy =

fa y . ~ dy +i4aly . (~+ ..!..)dY = Jo 9a.jy a 9a.jy 9a l

l

2

3a 2 . (16.5)

a

Alternatively, using (16.3), E[X2]

= L: x 2p{x} dx = fa

J-2a

Xl •

2(1 _~)dX = 3a 9a a 2

2

For any linear function of a random variable, say a b are constants, E[a

•

(16.6)

+ bX, where a and

+ bX] = L: (a + bx)p{x} dx = a L:p{x} dx + bL: xp{x} dx = a + bE[X]. (16.7)

If we put b = 0, then E[a] = a, so the expectation of a constant is that constant, and if we put a = 0, then E[bX] = bE[X]. Further, the expected value of the deviation of a random variable from its expectation is E[X - E[X]] = E[X] - E[E[X]] = E[X] - E[X] = 0;

(16.8)

i.e., the expected value of a random variable has the property that the expected value of the deviations of the random variable about it is zero. As another example, the standardized normal distribution (13.l2) was obtained from the general normal distribution (12.6) by the substitution u = (x - E)/q. We could obtain E[U] directly from its definition as

L:

u q,(u) duo

SECT.

1.17

57

THE CONCEPT OF VARIANCE

Alternatively, since u is a linear function of x and since E[X] = E[U]

~,

= E[X - ~J = E[X] - ~ = ~ - ~ = 0, cr

cr

(16.9)

cr

so that standardized normal distribution has expectation zero. 1.17. The Concept of Variance As a measure of the spread of a distribution the variance, defined as the expected value of the square of the deviation of the random variable X from its expectation, has many mathematical and statistical advantages: V[X] = E[(X - E[X])2].

(17.1)

The units of measurement of variance are squared units on the original scale. The square root of the variance, commonly represented by the symbol cr, is known as the standard deviation. For calculating variances the following identity is often convenient:

+ (E[X])2 - 2XE[X]] = E[X2] + E[(E[X])2] E[X2] + (E[X])2 - 2(E[X])2 = E[X2] - (E[X])2,

V[X] = E[X2

=

2E[XE[X]] (17.2)

i.e., the variance of the random variable X is equal to the expectation of the square of X minus the square of the expectation of X. An alternative identity is sometimes more convenient: V[X] = E[X2] - (E[X])2 = E[X2] - E[X] - (E[X])2

= E[X(X -

+ E[X]

1)] - E[X](E[X] - 1). (17.3)

If a and b are constants, then the variance of a linear function of X, say a + bX, is V[a

+ bX] = E[(a + bX)2] - (E[a + bX])2 = E[a2 + 2abX + b2X2] - (a + bE[X])2 = E[a 2] + 2abE[X] + b2E[X2] - a2 - 2abE[X] = b2{E[X2] -

(E[X])2} = b2 V[X].

b2(E[X])2 (17.4)

Thus adding a constant a to a random variable leaves the variance unchanged, but the variance of bX is b2 times the variance of X. To use (17.2) to calculate the variance of the random variable Y with the distribution given in Table 1.4, we need E[ y2], y2 being a function of Y. Using (16.4), E[Y2] = ~ y:p{Yt}. This summation is obtained as the t

sum of the third column of Table 1.14. In Table 1.13 we found E[Y] = 5. Substituting in (17.2) gives V[Y] = 27.5 - 52 = 2.5.

58

MATHEMATICAL IDEAS

CHAP.

1

Table 1.14

Yi

p{ytl

Y~P{Yi}

2 3 4 5 6 7 8

1/16 2/16 3/16 4/16 3/16 2/16 1/16

4/16 18/16 48/16 100/16 108/16 98/16 64/16

E[y2]

= ~y:p{ya = 440/16 = 27.5 i

To calculate the variance of the binomial distribution, (9.3), we will use (17.3). In computing E[X(X - 1)], X(X - 1) is a function of X, and by (16.4) E[X(X -

1)] =

~ xi(Xi -

i x(x -

l)p{ Xi} =

i

1)(n) 0"'(1 - o)n-",.

",=0

X

(17.5)

We note that n'. (n - 2)'. x(x - 1) ( n) = x(x - 1) . = n(n - 1) . x x!(n - x)! (x- 2)!(n - x)!

2)

= n(n - 1)' ( n . x-2 Substituting in (17.5), E[X(X -

1)]

= n(n

-

1)02~2

(17.6)

(:'= ~)01ll-2(1

- Ot-"'.

(17.7)

We have changed the lower limit of summation from 0 in (17.5) to 2 in (17.7), for clearly when x = 0 or 1 the corresponding terms in the summation are zero and add nothing to the summation, and hence can be omitted. Equation (17.7) will appear in a more satisfactory form if we write x = y + 2,sox - 2 = y,andn = m + 2,son - 2 = m. In(17.7), x ranges over 2 to n, so y ranges over 0 to m; thus E[X(X -

1)]

=

n(n - 1)0 2

I

11=0

(m)Oll(1 - o)m-II. Y

(17.8)

The summation is just the sum of all the terms of a binomial distribution, and the sum of any probability function must equal 1. Hence E[X(X - 1)]

= n(n -

1)02 •

(17.9)

SECT.

1.17

S9

THE CONCEPT OF VARIANCE

From (IS.8), E[X] = n(J. Making these substitutions in (17.3), we obtain for the variance of the binomial distribution: V[X]

= n(n

- 1)(J2 - n(J(n(J - 1)

= n(J(l

- (J).

(17.10)

If instead of considering the number of "successes" X in the binomial distribution, we consider the proportion of successes H = X/n, then by (17.4) the variance of H is V[H] = v[X] n

= 1. V[X] = n2

(J(1 - (J) . n

(17.11)

To consider some examples of the calculation of variances of continuous distributions, for the rectangular distribution, (12.2), we first obtain E[X2] by substituting for p{x} in (16.3): E[X2] =

roo x2px{x} dx

Loo

1 x31a/2

= ~

"3 -a/2 =

ra/2 x2(!)dX

=

La/2

a

a2

(17.12)

12 .

Substituting this in (17.2), and also substituting E[X] we get V[X]

= E[X2]

= 0 from

2

- (E[X])2

=~.

12 For the exponential distribution, we first obtain E[X2] as E[X2] =

(IS.9),

LOO x2(Je- 80 dx.

(17.13)

(17.14)

Using the formula for integration by parts, (IS.ll), and identifying/ex) with X2, so thatf'(x) = 2x, andg'(x) with (Je- 80, so thatg(x) = _e- 80, we have E[X2] =

x2(_e-80)1~ _

= 0

i

oo

(-e- 80)2XdX

+ 2LOO xe- 80 dx.

In (1S.12) we found E[X]

= (J

i

oo

o

xe- 80 dx

(17.1S)

1

= -, (J

(17.16)

so the integral in (17.1S) equals 1/(J2 and we get E[X2] = 22 (J

,

(17.17)

60

CHAP.

MATHEMATICAL IDEAS

1

Substituting in (17.2), we obtain, for the variance of the exponential distribution, 2 1 (17.18) V[X] = - - - = - .

(1)2

02

0

02

To find E[X2] for the normal distribution (12.6), the change of variable x = au + ~ is convenient: (17.18) (17.19)

= a2l eo

-eo

u2

• -

1 e- UI 12 du

..{br

+ 2a~leo

-eo

+ ~21eo

1 e- U I /2 du u . --=

.j21T

_1_ e- u1/2 duo

-eo..{br

(17.20)

The second integral corresponds to E[U] which from (16.9) we know to be zero. The third integral is <1>(00), (13.13), which must equal 1. To evaluate the first integral, in the formula for integration by parts, (15.11), we identify f(x) with Uj.j21T, so thatf'(x) = Ij.j21T, and g'(x) with ue- UI / 2 , so that g(x) = _r UI/ 2 :

= 0 + 1 = 1,

(17.22)

since the latter integral, when the minus sign is moved in front of the integral, is <1>(00). Substituting these results in (17.20), (17.23)

whence (17.24)

We thus see that the two parameters of the normal distribution, ~ and a 2 , are actually its mean and variance respectively. For the standard normal distribution cp(u), since u = (x - ~)ja, by (17.4), V[U]

= "21 V[X] = a

1.

"-----

(17.25)

EXERCISES 10

61

We have seen in (16.9) that E[U] = 0, so we will sometimes write NCO, 1) for cp(u). For any random variable X, the standardized form is defined as

X - E[X]

v'V[Xl

= w. '

Then

E[W]

= E[X] -

E[E[XJJ v'V[X]

(17.26)

say.

=

°

(17.27)

,

and

V[W]

= E[(W -

E[W])2]

= E[(X - E[X])2] V[X]

= E[W2] = V[X] = V[X]

1.

(17.28)

Thus any random variable when put in standardized form using (17.26) has a zero expectation and a unit variance.

EXERCISES ID 1D.1. A piggy bank contains 7 nickels, 8 dimes, 6 quarters, and 4 half dollars. You are permitted to withdraw one coin at random and then another without replacing the first. The random variable X is your gain in cents. What is (a) the distribution of X, (b) the probability that your gain will be more than 35 cents, (c) the expectation of X, (d) the variance of X? Assume that the probability of any coin being drawn is the same as for any other coin. 1D.2. In an eight-floor apartment house the elevator takes 10 seconds to get from one floor to the adjoining floor. Individuals leave and arrive with equal frequency, and the traffic arising from any floor is the same as for any other floor. The traffic density is low, so that the probability of the elevator being called by one individual while in use by another individual is to be regarded as zero. There is no traffic ~e!..ween floors, i.e., no tenant who visits any other tenant. For a tenant (a) oifthe-second floor, (b) on the eighth floor, what are (i) the distributions of waiting times? (ii) The expected values of the waiting times? (iii) The variances of the waiting times? 1D.3. A soap company distributes entry blanks to a lottery requiring nothing but the filling in of one's name and the mailing of the blank. The prize schedule is: first prize, $25,000; second prize, $10,000; third prize, $5000; next 5 prizes, SIOOO each; next 10 prizes, $500 each; next 50 prizes, S250 each; next 100 prizes, $100 each; next 1100 prizes, $25 each. Assume that 10 million entry blanks are returned. Let the random variable X be your gain from participation. (a) What is the distribution of X? (b) What is its expected value? (c) Supposing that the only cost of entry is the 5-cent stamp to mail in the entry, does it make sense to participate? 1D.4. "The word 'love' in the title of 'For Love or Money' is meant to be a synonym for prizes. The show is unusual among daytime programs in that it

62

CHAP.

MATHEMATICAL IDEAS

1

offers cash as an alternative to love. The money is hard to come by in any serious amount. A kind of electronic roulette ball ... darts from side to side of a stated number, such as 01400 and determines the payoff as it comes to rest: 80.01, 80.14, 81.40, 814, $140, or 81400. As any bookmaker's infant child could tell you, the chances of winning a respectable bundle are only two in six. The odds, therefore, are two to one against the player who goes for the cash instead of for a prize that is worth ... 8140. I'll detain you no longer with the laws of Pascal."-John Lardner in The New Yorker, July 19, 1958. Comment. [By "odds of n to 1 against" is meant a probability of 1/(1 + n). Assume that the "electronic roulette ball," i.e., the decimal point, runs back and forth between .01400 and 01400., moving instantaneously over each digit and after each move pausing for an equal interval of time. Assume that it stops after a random number of moves.] 10.5. The random variable x has the density function p{x}

=Ml -~),

0 <x
= 0, otherwise Find (a) p{x}, (b) E[x], (c) E[x 2], (d) Vex], (e) the mode, (/) the median. Sketch the graphs of the density function and the cumulative distribution function, and mark on the graph of the density function the positions of E[x], the mode, and the median. 10.6. The Pareto distribution, p{x} =

IXpax-(a+1l,

p :s: x <

00,

= 0,

otherwise, with the parameters (X and P both greater than zero, has been proposed as a representation of, e.g., data on incomes from tax returns when only those with incomes in excess of some specified figure file returns. (a) Find the expected value of X (i) when IX 1, (ii) when IX> 1. (b) Find the variance of X (i) when (X 2, (ii) when IX > 2. (c) Find the median of X. (d) Consider the variable Y defined as Y = log. X. Find the density function of Y. (e) Find the median of Y. 10.7. X is a random variable with expectation 10 and standard deviation 2. Obtain numerical answers for (a) E[K2], (b) E[E[X2]], (c) V[E[X2]], (d) V[V[X2]]. 10.S. Let X be a random variable whose logarithm Y is normally distributed with mean "I and variance (12, i.e., y = log x and 1 p{y} = ---=- e-(II-IIl s/2aS, - 00 < y < 00.

:s:

:s:

V27T(1 Then X is said to have a logarithmic normal distribution. Define log ~ = "I. Find the density function of X. Show that (a) the median of the distribution of x is ~, (b) the mode of the distribution of x is ~e-al, (c) E[K] = ~eaB/2, (d) VeX] = eal +211 (ea' - 1), (e) V[Y] = loge (1 + V[xJl(E(X])2), (/) E[Y]

= 10

( ge VI

+

E[x] ). V[x]/(E[a:])2)

~

as ell, so

/--~

SECT.

1.18

63

STANDARDIZED NORMAL DISTRIBUTION

tD.9. A box contains five electron tubes of a certain type, and you recall that two are good and three are bad, though they are unmarked. You need two good tubes, so you draw one tube at random and test it, and then draw another at random and test it, etc., until you have two good tubes, when you quit. (a) What is the probability that you have to test all five tubes? (b) What is the expected value of the number of tubes tested? tD.tO. Suppose that X is a random variable with the density function px{x}

=a'I

-3

= 0,

otherwise.

a

2a

<x


Suppose that Y is a function of x, namely Y = x 2 • (a) What is the density function ofY? (b)WhatisthemedianofY,saYYm? DoesYm = x!.? Ifso,why? Ifnot, why not? (c) What is the expected value of Y? Compute this in two ways. tD.H. For the geometric distribution, (problem IC.7), show that (a) E[X] = I/(), (b) VeX] = (1 - ())/()2. tD.t2. For the Pascal distribution, (IC.B), show that E[y] = a/(). tD.t3. What is the value of the constant c which makes E[(x - C)2] a minimum? tD.t4. Show that the expected value of Ix - cl, where c is a constant, is a minimum when c = x m , where Xm is the median. tD.tS. Suppose that Y has a density function py{y}. Define Z as the cumulative distribution function

f:<Xl py{t} dt. What is the density function of Z?

1.lS. The Properties of the Standardized Normal Distribution We saw in Section 1.13 that the substitution u = (x - ~)/(f transforms the general normal distribution, (12.6), into the standardized normal distribution cp(u), (13.12). We saw in (16.9) that the standardized normal distribution has expectation zero, and in (I7.22) that it has unit variance. We now examine in more detail the properties of 1 • cp(u) = --= e- u /2

(18.1)

.j21T

and (18.2) Firstly, since -( -U)2 fore

=

_(U)2, cp(u) is symmetric about u

cp( -u) = cp(u).

= O.

There(18.3)

64

CHAP.

MATHEMATICAL IDEAS

1

Secondly, the larger the absolute value of u, lui, the smaller is c/>(u), i.e., c/>(u) tends to zero as u tends to ± 00, and conversely c/>(u) takes its maximum at u = 0, when c/>(u) = 1/../271' = 0.3989. More formally, 1 e- 1l2/ 2

dc/>(II) =

../271'

dll

(_!). 211 2

= -lic/>(II),

(18.4)

and d 2c/>(II)

-2-

dll

=

-c/>(II)

+ (-1I)[-1Ic/>(1I)] = (11 2 -

(18.5)

1)c/>(1I),

which by the usual rules imply that c/>(u) has a maximum at u = 0 and points of inflection at u = ± 1. The function c/>(u) is graphed in the upper part of Figure 1.28.

0.4

I

I

0.3

I

0.2

I I I

I

I I

I 1

I

0.1

10 .

I

-----,f---I------I---- ----

0.9

0.8 0.7 0.6

Plu!

I

I I

0.5 04 •

0.3 0.2 0.1

o

-3

0.975 1 I I ----1----1----------1 I 1 I

I I 1 1

I I

I I

u

I

I I I I I I

I I ____:__ ..1 _____ 1 I I I I

I

I I

01

",I

I

<01

001

I

0)1

"ll

"ll

..... 1

..... 1

I

II

II

I

I

0.025' ----

I

'

-1

o

1

---+--', -2

1 ~I

@I

,,

0)1

I

..... 1

I 1

1

I

1

1

.....

I

2

3

u

Figure 1.28

Certain properties of (u) can be immediately inferred. From the symmetry about zero of c/>(u), the area in the lower tail below -u (where u is a positive number) must equal the area in the upper tail above +u, i.e.,

L~' c/>(t) dt = f~c/>(t) dt.

(18.6)

SECT.

1.18

STANDARDIZED NORMAL DISTRIBUTION

65

But

L" ,p(t) dt = 1 - L~,p(t) dt,

(18.7)

so <1>( -u) = 1 - (u).

(I8.8)

Let up be that value of u corresponding to the P point of the cumulative distribution function; i.e., if we take an observation at random from a standardized normal distribution the probability is P that the observation is less than up: (18.9) Pr{u < up} = P. But Pr{u

< up} = (up), i.e., (18.10)

From (18.8), (up) = 1 - (-up),soP = 1 - (-up), and (-up) = 1 - P. But (u1_p) = 1 - P, so (18.11)

Table I in the Appendix gives the values of P corresponding to specified values of Up. For example, if up = -2, P = 0.02275, and if up = 2, P = 0.97725. Table I can be used conversely to find up for a specified value of P, but Table VII does this directly for the more commonly used values of P. For example, if P = 0.025, Up = -1.960, and if P = 0.975, Up = 1.960. These points are plotted in Figure 1.28. These two operations, namely given up to find P and given P to find Up, can also be performed readily for any normal distribution if we know its mean ~ and its variance 0'2. In the equation for the cumulative distribution function of the normal distribution, P{x}

1 f'" =~ .J27rO'

change the variable from t to u = (t P{x}

=

e-(t-s)

•• 12fT dt,

~)/O'.

Then

(18.12)

-<Xl

1 f("'-S)/fT e-"'/2O' d ll =(x- ~). .J27rO' -00 a

(18.13)

For example, suppose that we are given that ~ = 100, a = 10, and we want the probability that an observation taken at random from this distribution is less than 120. We have

Pr{X

< 120} =

P{120} =

C 20 ~ 100)

= <1>(2) = 0.97725.

(18.14)

66

CHAP.

MATHEMATICAL IDEAS

1

Conversely, given the same distribution, what is the value of Xl' that a fraction P of all observations are less than? We solve the equation (18.15)

For example, if P is specified as 0.95, we know that the solution of UO•95 in (UO•95 ) = 0.95 is UO•95 = 1.645, so <1>(1.645) = 0.95 = <1>( X O•95a-

~).

(18.16)

Thus 1.645 = (:l;O.95 - ~)/a = (xO.05 - 100)/10, or X O•95 = 116.45. Equation (18.13) will give the probability that an observation will lie k or more times the standard deviation below the mean: Pr{X

<~-

ka}

= ((~ - ~) - ~) = (-k).

(18.17)

The probability that an observation will lie below k times the standard d~viation above the mean is (k). The probability that an observation will lie above k times the standard deviation above the mean is Pr{X > ~

+ ka} = 1 -

Pr{X

< ~ + ka} =

1 - (k)

= <1>( -k). (18.18)

We can also ask for the probability that an observation deviates from the mean by more than k times the standard deviation in either direction, i.e., that X lies outside the interval ~ ± ka. This is Pr{X

<~-

+ Pr{X > ~ + ka} = <1>( -k) + <1>( -k) = 2<1>( -k) = 2[1 - (k)]. for k = 1.96, 2<1>(-1.96) = 2 X 0.025 = 0.05, or

ka}

For example, tively 2[1 - <1>(1.96)] = 2(1 - 0.975) = 0.05. (18.17) can be written in an alternative form: Pr{X < ~

-

ka}

(18.19)

alternaThe left-hand side of

+ Pr{X > ~ + ka} = Pr{X - ~ < -ka} + Pr{X - ~ > ka} = Pr{-(X - ~) > ka} + Pr{X - ~ > ka} = Pr{IX - ~I > lw} = 2<1>( -k).

(18.20)

where k is a positive number. Complimentarily, Pr{IX - ~I

< ka} =

1 - 2<1>( -k).

(18.21)

SECT.

1.19

67

DISCRETE BIVARIATE DISTRIBUTIONS

1.19. Discrete Bivariate Distributions

Consider the experiment consisting of dropping three light bulbs in sequence with possible outcomes U = undamaged, F = filament broken, and G = glass envelope broken. The sample space contains 27 sample points. If the successive drops are independent, then for example Pr{U, U, U} = Pr{U} Pr{U} Pr{U}. Define two random variables, X = number of U's, Y = number of F's, and let Pr{U} = 1/2, Pr{F} = 1/3, Pr{G} = 1/6. For each of the 27 points Table 1.15 (U, U, U)(3, 0) (U, U, F)(2, 1) (U, U, G)(2, 0) (F, U, U)(2, 1) (F, U, F)(1, 2) (F, U, G)(1, 1) (G, U, U)(2,0) (G, U, F)(I, 1) (G, U, G)(1, 0)

27 18 9 18 12 6 9 6 3

(U, F, U)(2, 1) (U, F, F)(I, 2) (U, F, G)(I, 1) (F, F, U)(1, 2) (F, F, F)(O, 3) (F, F, G)(O, 2) (G, F, U)(1, 1) (G, F, F)(O, 2) (G, F, G)(O, 1)

18 12 6 12 8 4 6 4 2

(U, G, U)(2,0) (U, G, F)(1, 1) (U, G, G)(I, 0) (F, G, U)(I, 1) (F, G, F)(O, 2) (F, G, G)(O, 1) (G, G, U)(1, 0) (G, G, F)(O, 1) (G, G, G)(O, 0)

9 6 3 6 4 2 3 2 1

in the sample space we can tabulate the values taken by X and Yand also the probability of that point occurring; see Table 1.15. For example, for the sample point (U, U, U) we have x = 3, y = 0, and Pr{U, U, U} = (1/2)(1/2)(1/2) = 1/8 = 27/216. This sample point is therefore labeled (U, U, U) (3, 0) 27, the denominator 216 being omitted for conciseness. From Table 1.15 we can construct Table 1.16 showing the probability of obtaining any pair of values of x and y. Table 1.16

216 P{Xi, y;} (0,0) (1,0) (2,0) (3,0) (0,1) (1, 1) (2, 1) (0,2) (1,2) (0,3)

1 3+3+3= 9 9 + 9 + 9 = 27 27 2+2+2= 6 6 + 6 + 6 + 6 + 6 + 6 = 36 18 + 18 + 18 = 54 4 + 4 + 4 = 12 12 + 12 + 12 = 36 8

68

CHAP.

MATHEMATICAL IDEAS

1

Table 1.17

216 p{x;, Yi} Yi Xi

2

0

~ 216 P{Xi, Yi}

3

1 2 3

1 9 27 27

6 36 54 0

0 0

0 0 0

~ 216 P{Xi, Yi}

64

96

48

8

0

i

12 36

i

8

27 81 81 27

It is natural to express Table 1.16 in a two-way form (Table 1.17) and also to represent it graphically (Figure 1.29). In general, a discrete bivariate distribution will appear as in Table 1.18, where x and yare confined to the discrete values Xl' ••• , X" and Yh ... , Ym • The probability of any particular pair of values, Xi' Yi' is p{xj> Yi}' The marginal probabilities for X can be obtained by summing over y, and

plx,y! 0.6 0.5

y

0.4 0.3 0.2 0.1 0

x

Figure 1.29

SECT.

1.19

69

DISCRETE BIV ARIATE DISTRIBUTIONS

vice versa: m

PX{Xi}

= ~P{Xi' YI}' I

(19.1)

n

PY{Yi} = ~ P{Xi' YS}·

(19.2)

i

For example, for Table 1.17,

PY

{2}

= 12 + 36 + 0 + 0 = ~ = ~ . 216

216

9

Table 1.18

Yl

Ys

Ym

Xn

This result can be checked by regarding Y as a binomially distributed variable with n = 3, f) = 1/3, and using (9.3):

Pa{Y = 2}

=

(23) (13)~( 1 - 31)3-2= 92.

The conditional probability that X = Xi' given that Y = YS' can be found by using (4.3): Pr{X

=

xt/Y = Y;}

=

Pr{X

=

Xi and Y = Yi} Pr{Y = ys}

=

p{Xj, y,}.

(19.3)

p{ys}

It is convenient to call this the conditional probability function and use the symbol p{x/y}. Also, if X and Yare independent, using (5.9), namely Pr{E1 IE2} = Pr{E1}, gives Pr{X =

xii Y = Yi} =

Pr{X =

Xi}'

(19.4)

70

MA THEMA TICAL IDEAS

CHAP.

1

Substituting from (19.3) and writing Pr{X = Xi} = P{Xi} we find that in the case of X, y independent

p{x;, Yi} = p{x;}P{Yi};

(19.5)

i.e., the joint probability function is the product of the two marginal probability functions. In the case of the bivariate distribution of Table 1.16, for example, we have just seen that p{ Y = 2} = 2/9, whereas from the table, for example, p{ Y = 2JX = O} = (12/216)/(27/216) = 4/9, so for this distribution p{yJx} ¢ p{y} and hence X and Yare not independent. Similarly,p{X = 3}= 27/216, so p{X = 3}p{Y = 2} = (27/216)(2/9), which is not equal to p{X = 3, Y = 2}, which is O. 1.20. Continuous Bivariate Distributions Consider a function p{x, y}

~

0 for all

X,

y such that

L: L:p{x, y} dy dx = 1.

(20.1)

This function p{x, y} can be regarded as a two-dimensional, or bivariate, probability density function and is the y analog of the univariate probability density functions we have considered hitherto. Analogously to (11.5) in the univariate case, we can define a cumulative distribution function

P{Xi' Vi} = Pr{X ~ Xi' Y ~ Vi} =

f "'/ fllJ_oop{x,y}dydx. -00

(20.2)

Graphically, we can use the two horizontal dimensions to represent X, y and the vertical dimension to represent p{x, V}. Figure 1.30 Thus p{x, y} generates a surface above the plane p{x, y} = O. The total volume enclosed between this surface and this plane, by (20.1), is 1. P{x;, Vi} is the volume under the surface measured over the shaded area only (Figure 1.30). In manipulating bivariate probability density functions we will often find it convenient to reverse the order of integration, i.e., to assume

J"'bflldp{x, y} dy dx -~

=

flldJ"'b p{x, y} dx dy.

-

(20.3)

~ This is always legitimate in the situation in which we are involved. Technically, it is permissible if the integral is absolutely convergent. The

SECT.

1.20

71

CONTINUOUS BIVARIATE DISTRIBUTIONS

requirement that p{x, y} ~ 0 implies that if the integrals converge then they are absolutely convergent, and the requirement (20.1) implies that they are convergent, and hence they are absolutely convergent. The only example of a continuous bivariate distribution we will encounter in this book is the bivariate normal:

(20.4)

This will be discussed in detail in Chapter 12. We may wish to find the marginal probability density function of X alone, given p{x, y}. Intuitively, we would expect to get this by integrating p{x, y} over y. More formally, we can proceed as follows. Define Qx(r) as Qx(r) =

L:

(20.5)

PX,y{r, y} dy.

Then

Pr{-oo =

< X < x} = Pr{-oo < X < x, -00 < Y<

5:", (L"'", PX,y{l', y} dY) dr = 5:", Qx(r) dr.

oo} (20.6)

Also, considering X alone,

Pr{-oo

< X < x}

=5:", px{r} dr.

(20.7)

The left-hand sides of (20.6) and (20.7) are identical, so the right-hand sides are equal: (20.8) Intuitively, if we regard x as a variab1e we have here two functions of x which are equal for all values of x, and this implies that the functions must be identical. More formally, we can use a standard rule of calculus (see, e.g., Courant, Differential and Integral Calculus, Volume 1, p. 111), -d dx

1'"feu) du = f(x).

(20.9)

a

Differentiating both sides of (20.8) with respect to x, and substituting from (20.5) for Q, gives Px(x)

=

Qx(x)

=

L:

PX,y{x, y} dy.

(20.10)

72

CHAP.

MATHEMATICAL IDEAS

1

1.21. Conditional Continuous Density Functions We want a functionp{xIY} which will be the probability density function of X when Y is known; i.e., we want a function p{xIY} such that (21.1)

We start from our definition of conditional probability, (4.3), Pr{E1IE2} =

~------------~~*---~Y

Yo -c

Figure 1.31

Pr{E1 n E 2}/Pr{E2}, and identify E1 with the event with the event Yo - c < Y < Yo + c. Thus Pr{ xa

< X < xblyo _ -

Xa

< X < Xb and

E2

< Y < Yo + c} Pr{ < X < Yo - c < Y < Yo + c} Pr{yo - c < Y < Yo + c} c

Xa

X b,

(O:b (1I0+C

_ )0:. )1I0_/{X, y} dy dx -

(1I0+C )1I0-C py{y} dy

(21.2)

Figure 1.31 shows that the area under the curve py{y} bounded by the lines y = Yo - c, Y = Yo + c can be approximated by a rectangle with the base of length 2c and the height py{Yo'}' where Yo' is some value of y

SECT.

1.21

CONDITIONAL CONTINUOUS DENSITY FUNCTIONS

73

y

Yo

+c t-------

Yo - c t - - - - - - -

~--------~x-a---------X~b----~X

Figure 1.32

in the interval (Yo - e, Yo

f

+ e).

llO

+

In other words, we can choose Yo' so that

C

lIO-C

py{y} dy = 2epy{Yo'}'

(21.3)

The numerator of (21.2) is graphically represented in Figures 1.32 and 1.33. If we suppose that the x and y axes are in the plane of the paper and the p{x, y} axis is vertical to the paper, the shaded area in Figure 1.32 is the base of the volum~ represented by the numerator of (21.2). Figure 1.33 is a three-dimensional sketch of this volume element, resting on this base. The volume of this element could be approximated as the area of the face marked ABCD times the thickness, which is (Yo + e) (Yo - e) = 2e. A better approximation would be given by using instead of ABCD a similar area, say A'B'C'D', not actually shown, lying between the two faces, say at Yo', where Yo', not actually shown, lies in the interval (Yo - e, Yo + e). The upper edge C'D', analogous to CD but moved from y = Yo - e to y = Yo', has the functional form p{x, Yo'}' so the area A'B' C' D' will be given by integrating this from Xa to X b, to wit,

1 "'b

p{x, Yo'} dx.

"'a

p!x,y!

Figure 1.33

74

MATHEMATICAL IDEAS

CHAP.

1

Hence the volume of this element is given by multiplying this area by the thickness, 2e, giving

J"'bJ.1I0-C oX a

p{x,y}dydx = 2e f"'b p{x,Yo"} dx.

1Io-C

(21.4)

$a

We now substitute (21.4) for the numerator and (21.3) for the denominator of (21.2) and cancel out the 2e:

Pr{xa

< X < xblyo -

e

< Y < Yo + e} =

i"'bp{x, yo--} dx ::..:"'::!!.d_ _ __

PY{Yo'} (21.5) We now let e tend to 0, when both Yo' and Yo' will tend to Yo; thus in the limit,

Pr{ Xa

< X < xbl y =

i"'bp{x, Yo} dx Yo} =

"'d

(21.6)

Py{Yo}

This holds good for any Xa, X b• In particular, we can let xa = - co and Xb = x, when the left-hand side becomes the cumulative form p{xlY = Yo} of the probability density function p{xlyo}. On differentiating a cumuative form we get the density form, (11.6), so we have

p{XIY} = p{x, y} . py{y}

(21.7)

1.22. Independent Continuous Random Variables

Our definition of independence, (5.1), Pr{E1IE2} = Pr{E1IE~}, was shown to be equivalent to (5.9), Pr{E1IE2} = Pr{E1}. We now identify E1 with the event y C < Y < Yd and E2 with the event Xa < X < Xb, so when X and Yare independent,

Pr{yc

< Y < Ydlxa < X < xb} =

Pr{yc

<

Y < Yd}'

(22.1)

Using (4.3), Pr{E1IE2} = Pr{E1 (\ E 2}/Pr{E2}, the left-hand side of (22. 1) is

Pr {Yc


Pr{yc

< Y < Yd' Xa < X < x b} Pr{xa < X < x b} (22.2)

1.23

SECT.

75

EXPECTATION IN THE MULTIVARIATE CASE

The right-hand side of (22.1) is of course

fpy{Y}dY.

(22.3)

We therefore have, for, X and Y independent,

fdI.O:·p{x, y} dx dy II.

=

0:0

tid py{y} dy,

Jt

L:·px{x} dx

(22.4)

II.

whence

{lid {O:.

{O:.

Jllo Jo:op{x, y} dx dy = Jo:oPx{x} dx

flld II.

PY{Y} dy

lld

= I. O:·i py{x}py{y} dy dx. Q:"

(22.5)

'lie

This identity must hold for all xo ' x b, Yo, Yd' Intuitively, this must imply that the functions being integrated are identical, and in fact this is the case, so when x and yare independent (22.6) p{x, y} = Px{x}py{y}. 1.23. Expectation in the Multivariate Case We have seen, (16.3), that if X is a random variable with probability density function p{x}, and f{x} is a function of x, then the expectation of f(X) is (23.1) E[f(X)] = L:f(X)Px{x} dx. Now consider the case of a function h(XI , ••• , Xk ) of several random variables Xl' ... , X k with joint probability density function P{XI' ... , x k }. It is natural to define the expectation of h(XI' ... , XJ as

E[II(XI , I ••• , Xk )] =i cc •• ·i cc lI(x1> ... , xk ) _~_~

( x p{ Xl' . . . , xk } dX l ••• dxk • (23.2) We can USShiS to find the expectation of the sum of two random variables X, Y:

E[X

+ Y] ={: L:(x + y)p{x, y} dx dy = L: L: xp{x, y} dy dx

+L: L:yp{x, y} dx dy

=L:x[L:p{x, y} dY] dx

(23.3) (23.4)

+ L:y[L:p{X, y} dX] dy. (23.5)

76

MATHEMATICAL IDEAS

CHAP.

1

We now use (20.10) for the two expressions in brackets and obtain

E[X

+ Y] = L: xpx{x} dx + L:Ypy{y} dy =

E[X]

+ E[Y].

(23.6)

Thus the expectation of the sum of two random variables is equal to the sum of their expectations. Incidentally, if X and Yare separate observations, say Zl and Z2, from the same distribution with expectation " then E[ZI

+ Z2] = E[Zl] + E[Z2] =

2'.

This generalizes to the case of n observations: (23.7)

and using (16.7), (23.8)

i.e., the expected value of the mean of n observations from a distribution is the expectation of that distribution.

EXERCISES IE IE.1. The random variable X is normally distributed with a mean 100 and a standard deviation of 10. What are (a) Pr{X < 95}, (b) Pr{X > 90}, (c) Pr{80 < X < 85}, (d) Pr{IX - 1001 < 20}? What are the values of a, b, and c such that (e) Pr{X < a} = 0.95, (J) Pr{X > b} = 0.90, (g) Pr{IX - 1001 < c} = 0.90? IE.2. The random variable X is normally distributed with mean 100 and standard deviation 10. (a) What is the probability that a random observation does not deviate from 100 in either direction by more than IS? (b) What is the value of c such that Pr{IX - 1001 < c} = 0.99? (c) What is the probability that a random observation does not deviate from 80 in either direction by more than IS? IE.3. Suppose that a population of men has a mean of 150 pounds and a standard deviation of 20 pounds, and that the weights are normally distributed. (i) What are the probabilities that a man selected at random is (a) less than 100 pounds, (b) greater than 190 pounds, (c) more than 30 pounds from the population mean? (ii) (a) What is the weight that 90 per cent of the population is less than? (b) What is the weight that 80 per cent of the population is greater than? (c) Give the interval, symmetrical about the mean, in which 50 per cent of the population lie. IE.4. In a certain period of time on the average 25 per cent of make A of electric light bulbs will fail and 50 per cent of make B will fail. Three bulbs of type A and three bulbs of type B are mounted in a chandelier. What is the probability that after this specified period more of the A bulbs will have failed than the B bulbs? Assume independence.

SECT.

1.24

COVARIANCE AND THE CORRELATION COEFFICIENT

77

lE.s. Suppose that the life in hours (X) of a brand of television picture tubes has the probability density function p{x} = Oe- 6r»,

0

= 0,

<x <

00,

otherwise,

where 0 is some fixed known parameter (0) 0). Obtain the relationship between (a) the probability that the life of a tube is less than b hours, and (b) the probability that the life of a tube is less than 2b hours, given that the tube has survived b hours. lE.6. Consider the bivariate de~sity function

= abe-

p{Xl> xI}

R r»1-

Xl> XI > 0; a, b otherwise.

br»2;

=0,

> O.

(a) Evaluate p{xl } and p{xllxl }. (b) Determine whether Xl and XI are independent, giving the reasons for your conclusion. (e) Evaluate P{xl> xI}' lE.7. We have a joint density function p{x, y} = 1 for 0 < X < 1, 0 < Y < 1, and p{x, y} = 0 otherwise. Find the probabilities that (a) X > 0.5 and Y> 0.7, (b) X > 0.5, (e) X> Y, (d) X > 0.5, given that Y = 0.5, (e) X > Y, given that Y < 0.5, (I) X + Y < 1. lE.S. Over a 2-mile length OA of pipe line a blockage occurs at a point B at random in the interval OA. The blockage causes an increase in pressure and the pipe breaks at a point C at random in the interval OB. (a) What is the probability that the break C is in the interval from the origin 0 to the I-mile point? (b) What is the expected value of the length OC? lE.9. As in IE.S, except that the blockage occurs at a point B' at random in

the interval from the I-mile to the 2-mile point. lE.lO. Assuming that X and yare independent random variables, show that /

.

V[a:y] (E[x])I(E[y])1

v

where Cr» 1:= V[x]/E[x], C lI variation o{ X and y.

=

=

C2C2 III

11

v V[y]/ECy]

+

C2 r»

+

C2 11'

are the so-called coefficients of

1.24. CovaJance and the Correlation Coefficient The covariance of two random variables X and Y is defined as the expected value of the product of the deviations of X and Y from their expected values: Cov[X, Y]

= E[(X -

E[X])(Y - E[Y])].

(24.1)

For calculating a covariance, an equivalent form is often more convenient: Cov[X, Y]

= E[XY = E[XY] -

YE[X] - X£[Y] E[X]£[ Y].

+ E[X]E[Y]] (24.2)

If large positive deviations of X are associated with large positive deviations

78

MATHEMATICAL IDEAS

CHAP.

1

of Y, and likewise large negative deviations of the two variables occur together, then the covariance will be positive. Furthermore, if positive deviations of X are associated with negative deviations of Y, and vice versa, then the covariance will be negative. On the other hand, if positive and negative deviations of X occur equally frequently with positive and negative deviations of Y, then the covariance will tend to zero. Thus the covariance is a measure of a particular kind of association between two random variables. It will be noted that the variance is the special case of the covariance of a random variable with itself: Cov[X, X]

= E[(X -

E[X])(X - E[X])]

= E[(X - E[X])2] = V[X].

(24.3)

If V, Ware two random variables, and X = a + b V, Y = c + dW are two linear functions of them, a, b, c, and d being constants, then E[X] = a + bE[V] and E[Y] = c + dE[W], and so Cov[X, Y] = E[(a

+ bV -

a - bE[V])(c

+ dW -

c - dE[W])]

= bdE[(V - E[V])(W - E[W])] = bd Cov[V, W]. (24.4)

Equation (I 7.4) was a special case of this. Suppose that X, Yare two random variables with a joint distribution. The covariance of their standardized forms is known as their correlation coefficient: C [X - E[X] Y - E[Y]] (24.5) PXY = ov v'V[X] , v'V[Y] . Using the definition of covariance, (24.1), this

read~ly

Cov[X, Y]

PXY

= v' V[X]V[Y]'

reduces to (24.6)

If PXY = 0, we say that X and Yare uncorrelated, and this implies Cov[X, Y] = O.

Now consider the covariance of independent variables. We know from (22.6) that if X and Yare independent, p{x, y} = Px{x}py{y}. The definition of covariance, (24.1), involves E[XY]; using the definition of expectation in the multivariate case, (23.2), we have E[XY]

=

L: L:

xyp{x, y} dx dy,

(24.7)

SECT.

1.24

COVARIANCE AND THE CORRELATION COEFFICIENT

79

so for the case of X, Y independent,

=

E[XY]

=

L: L: L: L:

xYPx{x}py{y} dxdy

xpx{x} dx

ypy{y} dy = E[X]E[Y].

(24.8)

Substituting this in (24.2) we see that Cov[X, Y] = 0 and by (24.6) also PXy = O. Thus independent variables are uncorrelated. The converse,

y

x

Figure 1.34

however, i~Sgenerally not true. Two random variables can be uncorrelated but not in ependent. A simple example of this is to let p{x, y} = 1/4 at the pointsl (-1, 0), (0, 1), (0, -1), and (1, 0); see Figure 1.34. Then by the discrete analogy of (23.2),

E[X}]~I

Ii

i

(XtYi)P{Xt, Yi}

= (-1) x

0 x

+1x 0 x E[X]

=I

XtPx{Xt }

i

! + 0 x 1 x ! + 0 x (-1) x! 4 4 4 -1 = 0, 4

= (-1) x ! + 0 x ! + 1 x ! = 0, 4

2

4

(24.9) (24.10)

and likewise E[ Y] = O. The covariance of X and Y is thus Cov[X, Y]

= E[XY] -

E[X]E[Y]

=0-

0

x0=

O.

(24.11)

Thus X, Y have zero covariance, but are obviously not independent. For example, at x = -1, Y = 0, we have p{x, y} = 1/4, and Px{x} = 1/4.

80

CHAP.

MATHEMATICAL IDEAS

1

py{y} = 1/2, so the relationship (22.6), p{x, y} = Px{x}py{y}, which is necessary for the independence of X and Y, is not satisfied.

1.25. The Variance of a Mean We saw in (23.8) that the expectation of the mean of n observations from a distribution was the expectation of that distribution. We now want to consider the variance of a sample mean. Let us start with the sum of two random variables, Z = X + Y; its variance will be

+

V[Z] = E[Z2] - (E[Z])2 = E[(X

y)2] - (E[X + y])2

= E[X2] + 2E[XY] + E[P] - (E[X])2 = V[X] + V[Y] + 2 Cov[X, Y] = V[X] + V[ Y] + 2p J V[X] V[ Y],

2E[X]E[Y] - (E[Y])2 (25.1) (25.2)

the last line being an alternative form given by using (24.6). This result generalizes easily. Let Xl' ... ,Xn be random variables, and consider any linear combination

Z = 00

+ alXI + ... + 0nXn,

(25.3)

where the a i are fixed constants. Then E[Z]

=

ao + alE [Xl]

+ ... + anE[Xn]

(25.4)

and V[Z]

= E[(Z - E[Z))2] = E[(a o + alXI + ... + anXn -

ao - alE [Xl] - ...

- a n E[X,,))2

= E[{al(XI

-

E[X I ))

+ ... + an(Xn -

n

= .2 a~E[(Xi -

n

E[XJ?]

i

11

E[X,,]W]

11

+ .2.2 aiaiE[(Xi i ;

E[Xi))(X i - E[Xj)))

i>#i

n

n

= .2 aW[Xi ] + .2.2 aia j Cov[Xi , Xi]·

(25.5)

i i i i>#i

An alternative form for (25.5), using (24.3), is V[Z] =

" " aia i Cov[X .2.2 i

i,

i

Xi];

(25.6)

SECT .. 1.26

81

ADDITION THEOREM FOR NORMAL DISTRIBUTION

in this form the double summation includes the case i = j, which give the variance terms in (25.5). Also, the covariance term in (25.5) involves every combination of i with j, excluding i = j, and may be written as n-l

n

21 1 aia i COV[Xi' Xi]' ;=1 i=2

(25.7)

i
If in (25.3) we put 00 = sum of the Xi' and

0,0 1

n

V[Z]

=1 i

= a2 = ... = 1, then Z is just the simple n

n

i

;

V[Xi ] + 11 COV[Xi' Xi]'

(25.8)

i*i

If the variables are uncorrelated, then their co variances are zero and n

V[Z]

=

1 V[Xi]·

(25.9)

i

Often we are in the case where the variables Xi are independent, and since independence implies zero covariance, (25.9) will hold here also. To consider the ~pecial, but important, case of the mean ofn independent observations from the same distribution, the distribution having mean ~ and variance a~1 in (25.3) put ao = and 0i = lin, i = 1, ... ,n. Then

°

Xl

Z = -

"n

+ ... + -Xn n

1~ £.., Xi = X, ni

= -

i.e., with these values for the coefficients ai' Z is the mean, say X. Using (25.4) and (25.5) we have E[X] =

~E[Xd + ... + .!E[X n ] = n

n

£ + ... + ~ =~, n

n

(25.10)

and (25.11)

1.26. The Addition Theorem for the Normal Distribution In the preceding section we have proved that if Z is a linear combination of random variables Xi (25.3) then it has expectation (25.4) and variance (25.5). It is lengthy with the techniques so far at our disposal to prove further that any linear combination Z of normally distributed random variables Xi is itself normally distributed. A proof will be given at the end of Section 12.4. In particular this applies to the mean of n independent identically normally distributed variables Xi from a population N(~, ( 2 ):

82

CHAP.

MATHEMATICAL IDEAS

1

the mean will be normally distributed N(e, u 2/n), i.e., will have a density function

p{x} =

1

e-(:'t-i,",(2ul , .. ,

..}21ru/..}II

(26.1)

and cumulative distribution function

P{x}

=if

e-(t-i,",(2U",.. , dt.

1

-00

..}21ru/..}n

(26.2)

Since (x - e)/(u/..}~) is a unit normal deviate,

P{x}

=$

(:/J£)'

(26.3)

analogous to (18.13), the change being the substitution of x for x and u/..}~ for u. With the change of X for X, and u/..}; for u where necessary, (18.15) to (18.21) apply. 1.27. The X2 Distribution Suppose that we have n independent observations Xl> ••• , x .. from a normal distribution N(e, ( 2). The standardized variables Ui = (Xi - e)/u will also be independent, and the sum of their squares will have a distribution whose functional form can be determined. We define the x2(n) distribution as the distribution of the sum of the squares of n independent unit normal variables, l(n)

.

= ~ l/~.

(27.1)

i

The X2(1I) distribution is different for each value of n, and the parameter n is known as the number of degrees of freedom. Percentage points of its cumulative distribution are given in Table III of the Appendix. Figure 1.35 shows the density function of X2 for degrees of freedom n = 1, 2, 4 and 20. Since u~ can never be negative, all the X2 distributions are confined to the positive half of the X2 axis. To obtain the expectation of the X2 distribution we need E[u:J, Since U i is a unit normal deviate, it has zero expectation and unit variance, and so (27.2)

Then E[x2(n)]

=E

[. J" = ~l/~

~ E[l/~]

.

= ~(1) = n.

(27.3)

i i i

Since the u, are assumed to be independent, their covariance is zero, and

SECT. 1.27

THE Xl DISTRIBUTION

83

the variance of xll(n) is V[X Il(Il)]

We therefore need

=

V[u~]

V

[~u~J = ~ V[u~] = nV[u~].

which can be calculated in the form

V[u~]

To obtain

E[u~],

= E[(U~)2] -

J(x)g'(x) dx

= J(x)g(x) -

identifying/ex) with u3 , so thatf'(x) g'(x) = ue- ul/ll • We now have

.=1 00

=

1

C 1

U4p{lI} du~

-00

=

(E[lIm 2.

= .

00

=

u4

-00

1 -= ~2w

00

3 •

g(x)f'(x) dx,

(27.6)

3ull, and g(x) with _e- u '/I , so that

•

1 e- u•12 d.u -=

~2w

ue-II• 12 du

1 US( _e-u·/I) 100 -00

1

1_

__

roo (_e-u'/lI) . 3u s dll

fo1-oo

roo uSe-,,·/1I du = 3E[1l1l] = 3.

J2w 1-00

plx 2 1 0.7

n

f

-00

J2w

= 3'

0.6

(27.5)

we use the rule for integration by parts,

f E[1I4]

(27.4)

=1

0.5 0.4 0.3

Figure 1.35

(27.7)

84

CHAP.

MATHEMATICAL IDEAS

1

Hence, substituting (27.7) and (27.2) in (27.5), V[II~]

=

3 - 12

= 2,

(27.8)

and substituting this in (27.4), V[x2(n)] = 2n.

(27.9)

From the central limit theorem, to be referred to in Section 7.1, x2(n) is asymptotically normal for large n. As can be seen from the distribution for n = 20 in Figure 1.35, the approach to normality is quite fast. We can thus obtain an approximation to the P point of the distribution from x~>(n) - n / ~ y2n

(27.10)

lip,

whence (27.11)

A better approximation can be obtained from the fact, noted by Fisher [14], that .J2x2(n) is approximately normally distributed with mean

.J2n - 1 and variance 1. Thus the P point of x2(n) is obtained by solving (27.12)

i.e., (27.13)

If we have two independent X2-distributed random variables with degrees of freedom /1 and /2' then their sum has the X2 distribution with /1 + /2 degrees of freedom. For X2(/I) can be written as the sum of squares Of/I independent unit normal deviates: X2(fl) = 1I~

+ ... +

1I~1'

(27.14)

and X2(f2) can be likewise written as 1") = II 11+1 2 X2( J2

so

+

2 II 11+2

+ ... +

2 II 11+/2'

(27.15) (27.16)

EXERCISES IF IF.I. E[X

+

For the discrete bivariate distribution of Table 1.17, (a) calculate Y] from its definition E[X

+

Y] =

LL (xi + Yi) P{Xi' Yi}' i i

Calculate E[X] and E[Y] and check that equation (23.6) is satisfied. (c) Calculate PXY'

(b)

SECT.

1.27

EXERCISES

1F

85

IF.2. Two discrete random variables have a joint density function given by the probabilities in the table below.

y -1

o 1

-1

o

1

4/15 1/15 0/15

1/15 2/15 2/15

4/15 1/15 0/15

(a) What is the probability that X = -I? (b) What is the probability that X = -1, given that Y = O? (c) What is the probability that X = -1 and simultaneously Y = O? (d) Are the variables uncorrelated? (e) Are the variables independent? [In (d) and (e), present some explanation and/or justification for

your yes or no answer.] IF.3. X, Y, and Z are random variables each with expectation 10 and variances 1, 4, and 9 respectively. The correlation coefficients are PXY = 0, Pxz = 1/4, and PYZ = 1/2. Obtain numerical answers for the following:

(a) E[X + y), (b) V[X + Y], (c) E[X + Z), (d) V[X + Z), (e) E[X - Z), (J) V[X - Z), (g) E[X + Y - 2Z), (h) V[X + Y - 2Z), (i) E[E[X)), (j) V[E[X]], (k) E[V[X)), (I) V[V[X)).

IF.4. Suppose that among married couples in a certain population the weights of the husbands are normally distributed with a mean of 80 kilograms and a standard deviation of 9 kilograms, and the weights of the wives are normally distributed with a mean of 60 kilograms and a standard deviation of 4 kilograms. Suppose further that the correlation coefficient between the weight of a husband and the weight of a wife is 1/3. Obtain numerical answers for the following: (a) the probability that the total weight of a couple chosen at random is more than 162 kilograms; (b) the probability that a wife is heavier than her husband. IF.S. The independent random variables Xi' i = 1, 2, 3, 4, are normally distributed with means the same, namely 10, and variances the same, namely 12. We define Z as Xl + Xs + Xa - 3X4 • Obtain the numerical values of (a) the expectation of Z, (b) the variance of Z. IF.6. Axles are manufactured so that their diameters are normally distributed with a mean of 1.00 inch and a standard deviation of 0.004 inch. The bearings in which they are to run are manufactured so that their internal diameters are normally distributed with a mean of 1.01 inch and a standard deviation of 0.003 inch. Operator A assembles axles and bearings at random. Operator B is able to visually pick out axles and bearings so that the dimensions of these two items have a correlation coefficient of 2/3. The assemblies of axles and bearings are rejected if the gap between the axle and bearing lies outside the interval 0.008 to 0.013. What proportion of the assemblies of (a) operator A, (b) operator B, will be rejected? IF.7. A small plane will take four passengers in addition to the pilot. The safe payload for the four passengers is 720 lb. Assuming that the passengers are selected at random from a normal population with mean weight 150 lb and standard deviation 32 lb, how often will four passengers overload the plane? IF.S. You are packing by machine 16 cookies to a box. The label states that the box contains 1 pound of cookies. The weight of the individual cookies is normally distributed with a mean of 1 oz and a standard deviation of 0.1 oz.

86

MATHEMATICAL IDEAS

CHAP.

1

If a customer buys a box at random, what are the probabilities that he receives OZ, (c) more than 16.5 oz? (d) If you wish to change your manufacturing process so that the mean weight of the cookies is such that only one customer in a hundred will receive less than 1 pound, what should the new mean weight of the cookies be, assuming that the standard deviation remain unchanged? (e) Further, if you could also reduce the standard deviation to 0.05 oz, what mean weight would satisfy the condition specified in the previous sentence? IF.9. Suppose that you drive from New York to San Francisco, 3040 miles by a certain route, starting with new spark plugs in your eight-cylinder engine. Suppose that the satisfactory life of a plug is normally distributed with a mean of 5000 miles and a standard deviation of 1000 miles. What is the probability that you will roll over the Bay Bridge with your motor still purring (with the original plugs) on all eight cylinders? IF.IO. Pipes (for tobacco) are being packed in fancy plastic boxes. The length of the pipes is normally distributed with a mean of 5 inches and a standard deviation of 0.04 inch. The internal length of the boxes is normally distributed with a mean of 5.1 inches and a standard deviation of 0.03 inch. What proportion of the time will the box be too small for the pipe? (a) less than 1 pound, (b) less than 15

REFERENCES 1. Todhunter, I., A History of the Mathematical Theory of Probability. London: Macmillan and Co., 1865. New York: Chelsea Publishing Co., 1949. 2. Pearson, E. S., Karl Pearsoll. London: Cambridge University Press, 1938. 3. Pearson, E. S., and Wishart, J. (eds.), "Studellt's" Collected Papers. London: Biometrika Office, 1942. 4. Fisher, R. A., Statistical Methodsfor Research Workers. 1st ed. 1925, 13th ed. 1958. Edinburgh: Oliver and Boyd. 5. Fisher, R. A., The Desigll of Experimellts. 1st ed. 1935, 7th ed. 1960. Edinburgh: Oliver and Boyd. 6. Fisher, R. A., COlltributiolls to Mathematical Statistics, ed. W. A. Shewhart. New York: John Wiley and Sons, 1950. 7. Neyman, J., and Pearson, E. S., "On the Problem of the Most Efficient Tests of Statistical Hypotheses," Philosophical Trallsactiolls of the Royal Society, A, 231 (1933),289-337. 8. Wald, A., Sequelltial Allalysis. New York: John Wiley and Sons, 1947. 9. Wald, A., Statistical Decisioll FUllctiolls. New York: John Wiley and Sons, 1950. 10. Bayes, Thomas, "Essay towards Solving a Problem in the Doctrine of Chance," Philosophical Trallsactiolls of the Royal Society, 53 (1763), 370-418. Reprinted in Biometrika, 45 (1958), 293-315. 11. Wallis, W. Allen, and Roberts, Harry Y., Statistics: A New Approach. New York: The Free Press of Glencoe, 1956. 12. Mosteller, Frederick, "The World Series Competition," Joul'1lal of the Americall Statistical Associatioll, 4 (\ 952), 355-80. 13. Cramer, H., Mathematical Met/rods of Statistics. Princeton: Princeton University Press, 1946. 14: Fisher, R. A., and Yates, F., Statistical Tables for Biological, Agricultural alld Medical Research. 6th ed. Edinburgh: Oliver and Boyd, 1963.

CHAPTER

2

Statistical Ideas

2.1. Statistical Inference Statistics is frequently defined as the science of making wise decisions in the presence of uncertainty. To cite Savage's example in his review [1] of Wald's book Statistical Decision Functions [2], the decision to be made may be whether to take an umbrella on one's trip to the office. This approach requires a knowledge of the relative costs of carrying an umbrella when the day turns out to be fine and of getting wet by failing to carry an umbrella when the day turns out to be wet. The practical usefulness of this approach has been severely hindered by the rarity with which one can actually estimate with any confidence the cost functions. Whether this obstacle can be circumvented sufficiently to bring this decision theory approach into common use remains to be seen. For an elementary account, see Chernoff and Moses [3]. Classical statistical inference operates one stage further back from the decision-making process: very roughly speaking, it confines itself to establishing what can be said about the facts of the situation, and it leaves the decision as to what to actually do, if anything, to the common sense of the experimenter. Two broad categories of problems that have received particular attention are estimation and the testing of (statistical) hypotheses. The first is subdivided into point estimation and interval estimation of the parameters of a distribution. For example, given a sample of n observations from a distribution assumed normal, we may wish to obtain point estimates of the parameter (f, or the parameter (fa, or both the parameters, or conceivably some function such as ei or ~ + k(f. We may, on the other hand, wish to obtain an interval which we are confident will include the actual value of the parameter or parameters: this is the problem of interval estimation. 87

88

STATISTICAL IDEAS

CHAP.

2

The second problem, hypothesis testing, consists of making an assumption about the distribution function of a random variable, very often about the numerical values of one or more of the parameters of the distribution function, and deciding whether those values of the parameters are consistent in some sense with our sample of observations on that random variable. For example, in the case of the normal distribution, the hypothesis might be that ~ = 0, or that 0'2 = 1. Of course, if we could take infinitely large samples of observations we would be able, in an ordinary case, to determine the parameters of the distribution in question exactly, and there would be no statistical problem. The problems of statistical inference are to do the best we can with limited samples of observations. Implicit in the treatment of the problems of statistical inference is the assumption that we are able to obtain random samples of observations. A random sample can be defined as consisting of statistically independent, identically distributed random variables; i.e., random variables with the same distribution function. As an example of how we may generate a random sample, consider a sample space with all n points equiprobable. Let X be a random variable, so that associated with every point in the sample space is a characteristic Xi' Let n(xi) be the number of points with characteristic Xi' Now sample N points from the sample space, replacing each point obtained before taking the next sample. Then we will observe Xi with probability n(xi)/N. Since the drawings are with replacement, they are independent. Since all observations come from the same sample space they will have the same distribution function. Thus our observations will satisfy our definition of a random sample. Suppose we take a random sample of two observations on a random variable with distribution functionp.-dx}. The pair of observations Xl' X 2 will have a bivariate distribution function P{Xl' x2}. But if the sample was random, as supposed, the observations will be independent, and by (1.22.6)

(1.1)

Clearly this will generalize to n observations, so

P{Xl' ... , x,,}

" Px{xi }. = Px{xl } ... Px{x = II lI }

(1.2)

i

2.2. Some Principles of Point Estimation Suppose that we have a random sample of n observations from a population with distribution function of known form

p{x: 01 ,

••• ,

Ok}'

Xl' •.. , Xn

SECT.

2.2

SOME PRINCIPLES OF POINT ESTIMATION

89

involving k parameters e;, j = 1, ... ,k. For example, p{ } could be the normal distribution which has the two parameters ~ and (J. We may wish to estimate ~, or we may wish to estimate 0'2 either in the case where ~ is known or in the case where ~ is unknown. The general problem is to estimate one (or more) of the e;, the remaining e; being either known or unknown. The function of the observations we choose, 9(xl> ... , xn) is known as the estimator, and when we insert a particular sample of numerical values for the Xi we obtain a numerical value of the function which is known as an estimate. Often several alternative functions will naturally suggest themselves as estimators: for example, as estimators of the mean of a normal distribution we might use the arithmetic mean, the median, the mode, the midrange (the arithmetic mean of the largest and smallest observations in the sample), etc. What criteria should we use for choosing our estimator? Obviously, in principle the answer may depend on the particular purpose for which we want it. However, conventional statistics attempts to answer this question in general terms, in order to generate estimators of general utility, since it would be impractical to have to consider every separate problem completely from first principles. In practice, our choice is sometimes influenced by economic considerations; one estimator may be somewhat better than another, but if it requires $500.00 of computation whereas the alternative requires only $5.00, we may well decide that the margin of superiority of the former does not justify the additional expenditure and that the latter is adequate for our needs. Confining our discussion to the one-parameter case, intuitively we want our estimator 9 to yield estimates whose distribution is close in some sense to e. This broad criterion is not sufficient to resolve ambiguities, however. In Figure 2.1, which shows the distribution of three alternative estimators 9a , 9b and 9c' the first two have the advantage that their expected values are equal to e. This property, namely that E[9] =

e,

(2.1)

is known as unbiasedness. The bias of an estimator 9 is defined as (E[9] - e).

(2.2)

The estimator 9a is preferable to 9b in that medium-sized deviations occur less frequently; on the other hand, extreme deviations are more frequent with 9a than with 9b • Using the variance, V[9] = E[(9 - E[9])2], as a measure of spread, the third estimator 9c is the best ofthe three, but since its expected value is not equal to e, it is a biased estimator. Unbiased estimators have the following desirable property: if we have a series of independent unbiased estimates then their average will also be

90

CHAP.

STATISTICAL IDEAS

2

unbiased. On the other hand, the average of biased estimates will be biased, no matter how large the number being averaged.

a

c

~~~--------~~----~~~~------------~8 Figure 2.1

For comparing estimators, including biased estimators, a useful criterion is the mean square errol', defined as the expected value of the square of the deviation of the estimate from the parameter £[(9 - 0)2]. This quantity is equal to the variance plus the square of the bias, as can be shown as follows: £[(9 - 0)2]

= £[{(9 - £[9]) + (£[9] - 0)}2] = £[(9 - £[9])2] + £[(£[9] - 0)2] + 2(£[9] = v[9] + (bias)2,

0)£[9 - £[9]]

(2.3)

since £[9 - £[9]]

=

£[9] - £[£[9]]

=

£[9] - £[9]

=0

(2.4)

A further criterion, consistency, involves the behavior of an estimator as the sample size increases indefinitely Specifically, if for any small quantity E, the probability that the absolute value of the deviation of 9

SECT.

2.3

from 0 is less than

MAXIMUM LIKELIHOOD ESTIMATION E

91

tends to 1 as n tends to infinity, i.e., if

Pr{lb - 01

< E} ---+- 1

as n ---+-

00,

(2.5)

then we say that the estimator 0 is consistent. Among the class of unbiased estimators, there will be one or more with minimum variance: this (or these) is (or are) the efficient estimator(s). The efficiency of any other estimator is defined as the ratio of the variance of the efficient estimator(s) to the variance of the other estimator. 2.3. Maximum Likelihood Estimation A method of estimation which usually in practice works out satisfactorily is the method of maximum likelihood [4]; namely, to choose that value of 0 which maximizes the likelihood function defined as (3.1)

where p{x: O} is the distribution function of x, of known form but containing the unknown parameter O. In maximizing the likelihood function, the observed x's are considered as constants and the parameter 0 as a variable. Incidentally, at this point we will cease to preserve the distinction between random variables X and the values which they can take, x. Strict consistency in this convention becomes clumsy and leads to uncomfortable contradictions with long-established usage; for example, Student's t would have to be written as T. This difficulty could be avoided by the notational device of using bold-face type for random variables, but other infelicities remain, and therefore we will use lower-case letters except on those occasions in which upper-case letters are necessary to avoid confusion. Maximum likelihood estimators have the desirable properties of being consistent and asymptotically normal and asymptotically efficient for large samples under general conditions. Sections 33.2 and 33.3 of [5] and Chapter 18 of [6] contain extensive discussions of these questions. Maximum likelihood estimators are often biased, but frequently the bias is removable by a simple adjustment. As one example of maximum likelihood estimation, suppose that we take a sample of size n from a binomial population with parameter 0 and find x defectives in the sample. We wish to estimate 0, the proportion of defectives in the population. The likelihood function is L = p{ x: 11, O} =

(~) 0"'(1 -

OY-"'.

(3.2)

92

CHAP.

STATISTICAL IDEAS

2

Figure 2.2 shows plots of L against (J for the two cases, n = 10, x = 3 and n = 50, x = 15. In maximizing this function, it is more convenient to maximize its logarithm, which amounts to the same thing; those L

0.28 0.24 0.20 0.16 0.12 0.08 0.04 ~~~~~--~--~--~--~--~~--L---~9

0.4

0.5

0.6

0.7

0.8

0.9

Figure 2.2

values of the parameters which maximize log L will maximize L. The logarithm of (3.2) is log L = log (:)

+ x log (J + (11

- x) log(l -

(J).

(3.3)

The usual method for maximization with respect to a parameter is to differentiate with respect to that parameter and equate the derivative to zero: d log L d(J

= :: + (11 (J

_ x)

----=-!... 1-

(J ,

(3.4)

which yields 9 = x/n on equating to zero. The caret is conventionally used over a symbol to denote the maximum likelihood estimator. Thus 9 is the maximum likelihood estimator of (J. When the density function has several parameters, we can usually find the values of the parameters that maximize log L by setting the partial derivatives of log L with respect to the parameters equal to zero; i.e.,

SECT.

2.3

93

MAXIMUM LIKELIHOOD ESTIMATION

we set

olog L = OOI

0 log L = ... = O. 00 2

(3.5)

Frequently, but not always, the value of, say, Ol that maximizes log L depends on the value of O2 , so to estimate Ol we have also to estimate O2 , As an example, consider the estimation of the parameter ~ of a normal distribution from a sample Xl; •.• ,X n • The likelihood function is obtained by substituting the normal density function in (3.1):

(3.6) where the summation is over the index i = 1, ... ,11, as also in the remainder of this section. The logarithm of the likelihood is log. L = - -11 log ( 2m!2) - - 1 k~( Xi 2

-

/:2

~) •

(3.7)

2(12

Differentiating with respect to

~

gives

ologL = _ _ 1 I2(x. _ ~)(-1). o~

2(12

(3.8)

t

Equating to zero gives

I

~)

(Xi -

=I

Xi -

11~

=

(3.9)

0,

whence (3.10) which is the sample mean, usually denoted by x. We obtained ~ without having to estimate (12. To obtain the maximum likelihood estimator of (12 we differentiate (3.7) with respect to (12:

olog L =

_

!!.1- + _1_ I 2 (12

0(12

(Xi _

~)2.

(3.11)

2( (12)2 /'0.

If ~ is known we equate this to zero and solve for

'2

(1

(X; _ ~)2 = I'-'--"'-'----'II

(12:

(3.12)

94

CHAP.

STATISTICAL IDEAS

2

If ~ is unknown, we put the two partial derivatives (3.8) and (3.11) equal to zero, giving ~ = x as before and 0'2 = :E (Xi - ~)2/n. Substituting for ~ gives (3.13)

n

Here we had to obtain ~ in order to obtain 0'2. In the foregoing we obtained the maximum likelihood estimator of 0'2, i.e.,~. We might have chosen instead to obtain the maximum likelihood estimator of 0', i.e., a. We would then have differentiated (3.7) with respect to 0':

olog L = _ ~ + 1.. ~ (Xi _ 00'

(1

We equate to zero and solve for

a=

~)2.

(3.14)

(13

a:

.J~ (Xi - ~)2/11 = .J~ (Xi - X)2/ n.

(3.15)

We thus see that (3.16)

This illustrates a general property of maximum likelihood estimators, and one which is highly desirable, namely invariance. Generally, if fee) is a single-valued function of e, and 9 is the maximum likelihood estimator /'..

of e, then f(9) is the maximum likelihood estimator of f(e); i.e.,f(e). In the foregoing example, 0' = .J 0'2 is a single-valued function of 0'2 and

a=J~. Let us now check on the expected value of estimator. We have na2 = ~ (Xi - X)2

=

~ [(Xi -

a to see if it is a biased 2

~) - (x - ~))2

=~~-W+~-W-~-~~~-~ ~ (Xi - ~)2 - n(x - ~)2,

=

(3.17)

since 2(x - ~) ~ (Xi

-

~) = 2(x - ~)(~

= 2(x -

~)(nx

Xi -

-

n~) n~)

= 211(x -

~)2.

(3.18)

Thus Now (3.20)

and E[(x -

~)2)

= E[(x _ E[X])2]

=

V[x] = V[xi ]. 11

(3.21)

SECT.

2.4

A WEIGHTED MEAN UNBIASED AND OF MINIMUM VARIANCE

95

Hence (3.22)

In the case of the normal distribution, V[x i ] = 0'2, so 62 defined as in (3.13) is a biased estimator of 0'2. On the other hand, if we define S2 as 2

S

=

Z(X. - X)2

•

(3.23)

n-1

it will have expected value E[S2]

=

E[_n_ L(xi

/1-1

-

n

X)2J = _n_

E[62]

n-1

(3.24) (3.25)

Therefore S2 will be an unbiased estimator of the parameter 0'2 of the normal distribution; It will be noted that, though (3.13) was derived specifically as an estimator of the parameter 0'2 of a normal distribution, the arguments of (3.17)-(3.22), showing that it was a biased estimator of the variance, made no reference to the distribution being sampled being normal. Likewise (3.25), showing that S2 as defined in (3.23) is an unbiased estimator of the variance, is true for observations Xi from any distribution. The general subject of estimation is rather controversial because it is possible to give different weights to the various criteria, and to make up criteria of one's own, depending on one's view as to what the estimate is to be used for. We will not explore this topic further, except in Chapter 11 where we introduce the method of least squares, which gives results identical with those of the method of maximum likelihood when the underlying distribution is normal. 2.4. A Weighted Mean Unbiased and of Minimum Variance Suppose that we have k observations Xi' all with the same expectation ~, but with variances O'~. We wish to choose coefficients a i so that a linear combination of the Xi' defined as k

X=

L aixi ,

(4.1)

i

will be unbiased and of minimum variance. For expected value,

E[x] = E[i aixiJ = i

.

.

aiE[xi] =

x to

be unbiased, its

~ i, ai'

(4.2)

96

CHAP.

STATISTICAL IDEAS

must equal

~,

2

k

so we require that ,2 0i = 1, which we can write as i

k-l

Ok

k

+ ,2a i = ,2 ai =

(4.3)

1,

i

i

so ak =

k-l

(4.4)

1 - ,2ai . i

The variance of x, which we wish to minimize, is (4.5)

(4.6)

To find the values of 0i which make this a minimum we differentiate with respect to OJ and equate to zero: oV[x] - = 2a jO'j2 -

2

20'k

2 + 2 (k-l) ,2a i O'k

OOj

i

=

2[ajO'~ -

(1 - k~ai)O':J =0.

(4.7)

Substituting (4.4) in this, we get 2

O'k

a j = ak2 ,

j = 1, ... , k - 1.

(4.8)

0',

However, this relationship is obviously true for j fore sum over j = 1, ... , k to get

=

k also. We can there-

(4.9) k

But ,2 0i = 1, so i

(4.10) Substituting (4.10) into (4.8) gives, for any j, (4.11)

SECT.

2.5

97

SOME PRINCIPLES OF HYPOTHESIS TESTING

These a j are the values which make x =

k

!

aix j unbiased and of minimum

i

variance. If we use these coefficients, the variance of x, using (4.5), is V[x]

= !i

k

=!i

k [(

a~(1~

(1~

! -(1~1 )-lJ 2(1~ k j

(4.12) Sometimes it is convenient to define weights

a i = -1

(12 j

(k! -1 )-1 = . (12 'i

Wi

as

Wi

=

II'

From (4.11), (4.13)

_._i_ . k

!

1/(1~.

WI

i

With this notation in (4.1) and (4.5),

(4.14)

1

(4.15)

2.5. Some Principles of Hypothesis Testing The simplest situation is the test of a hypothesis against a single alternative. We suppose that under the null hypothesis Ho the density function is p{x: Oo} and under the altel'1lative hypothesis the density function is p{x: 01}' The two hypotheses in question supposedly completely define the two distributions; i.e., 00 and 01 are both exactly specified, and if there are further parameters in the density functions they are assumed known. Hypotheses of this type are known as simple hypotheses. If any parameters are not specified completely the hypothesis is called a composite hypothesis. For example, if the distribution is the normal and the parameter in which we are interested is ~, but the variance is unknown, then we would be concerned with a composite hypothesis. Again, if the alternative distribution is the normal with known variance but it is merely specified that ~1 > ~o, or that ~1 ~ ~o, then ~1 is not specified completely and so we are dealing with a composite alternative hypothesis. This and the three following sections will deal with simple hypotheses.

98

STATISTICAL IDEAS

CHAP.

2

We are going to take an observation x and we want a criterion for accepting the null hypothesis or accepting the alternative hypothesis (and rejecting the null hypothesis). We start by choosing a region on the line such that if the observation

plxl

plx:/Jol a

--!1'.!lJJzIl---_x

Region of acceptance

plxl

(a)

------------------~~--~x

Critical region (b)

Figure 2.3

falls in this region then we accept the null hypothesis; this is the region of acceptance. The complementary region, where we reject the null hypothesis, is the region of rejection or the critica/regioll. On what basis shall we choose these regions? The first criterion is that only in a small fraction (X of the time shall we commit a Type I error or an error of the .first kind, which is to reject the null hypothesis when in fact it is true. Clearly we wish to commit such an error only rarely: common

SECT.

2.5

SOME PRINCIPLES OF HYPOTHESIS TESTING

99

choices for IX are 0.05 or 0.01. The numerical value selected for IX is known as the level of significance of the test. To be more specific, suppose that under the alternative hypothesis the distribution is shifted without change of shape to the right so that the expected value of x under the alternative hypothesis, Eel [x], is greater than EeJx], the expected value under the null hypothesis. Then it is reasonable to choose as the critical region that part of the line to the right of some value xc' where Xc is chosen (see Figure 2.3a), so that if the null hypothesis is true (so that x has the distribution p{x: 0o}), the probability of a random observation falling in the critical region is IX; i.e., Xc satisfies the equation (5.1) We now consider what will happen if actually HI is true and x has the frequency function p{x: Ol}' We supposed earlier that the difference between p{x: Oo} and p{x: Ol} was that the distribution was shifted to the right, so the situation is as in Figure 2.3b. Under HI> the probability, say p, of our observation falling in the region of acceptance, x < xc, is the shaded area in Figure 2.3b; i.e., p is determined by the equation (5.2) To accept Ho when it is false is an error, known as an error of the second kind, or a Type II error. The probability of the complementary event, namely of correctly rejecting the null hypothesis, is known as the pOll'er of the test, say T1'. The power is thus equal to the probability of x falling in the region of rejection when it has the alternative distribution: (5.3) We would like to have the power large, say 0.90 or 0.99, but restrictions of time and money may mean we have to settle for substantially less. The result of a statistical test can be summarized in Table 2.1. The usual procedure is to choose IX small, say 0.05 or 0.01. For specified 00 and 01 , and a given p{x: O}, then p will be determined, and often it will be substantially larger than what we would regard as a satisfactorily small value, say 0.1. We will then be chary of actually rejecting HI; while we accept H o, we do not rule out the possibility that HI is true. Such a situation is illustrated by Figure 2.4. Here, if x falls at the point indicated, it is clearly in the region of acceptance and so we accept Ho. However, such a value of x could very easily arise in sampling from the alternative

100

CHAP.

STATISTICAL IDEAS

2

Table 2.1 Conclusion of experimenter

True situation Ho true HI false

Ho false HI true

Conclusion false

Ho true

Conclusion correct Type II error committed

HI false

Ho false

Conclusion false

HI true

Type I error committed

Conclusion correct

distribution p{x: OI}; the probability

f

"'c _ oop{x: 0I} dx

is by no means small, being about 3/16, and it would be hazardous to rule out the possibility that HI is true. Nevertheless, the customary procedure is to accept Ho until we are forced by further data to revise our opinion. This procedure is standard but clearly open to a variety of criticisms. For example, the conventional policy of choosing ex and leaving fJ to fend for itself often appears as an arbitrary asymmetry. However, it will be noted that this is analogous to a broad principle of English common law: a jury is instructed to return a verdict of not guilty (corresponding to

Critical region

Figure 2.4

SECT.

2.6

CRITERION FOR CHOOSING BETWEEN ALTERNATIVE TESTS

101

accepting the null hypothesis) unless it is satisfied beyond all reasonable doubt (corresponding to a very small IX) that the defendant is guilty. Then it rejects the null hypothesis that the defendant is not guilty and accepts the alternative hypothesis that the defendant is guilty. If this policy results in a rather large proportion of guilty men going free, society nevertheless accepts this as a price it has to pay for achieving its prime objective; namely, very rarely convicting an innocent man. The acceptance of the null hypothesis does not imply a belief that it is true, merely that the evidence for rejecting it is insufficient. The hypothesis testing procedure itself does not directly distinguish between the case in which the null hypothesis is either true or close to being true, on the one hand, and on the other hand the case in which the data is really insufficient to draw any useful conclusion. Confidence intervals, to be discussed in Section 2.14, can be used to throw light on this question. 2.6. A Criterion for Choosing between Alternative Tests We have seen that we construct a test by dividing the x axis into two regions, the region of acceptance and the critical region, and so far the only restriction we have imposed on this choice is that the probability of an error of the first kind should be

r

.R

p{x:O o} dx

= IX,

(6.1)

where R is the region of rejection. In the preceding section we stated that in the case in question a reasonable choice for the critical region was the upper tail of the distribution p{x: Oo} without discussing whether this was the best choice. We now examine this question. Consider an alternative critical region, for example that defined by Xli < X < Xc in Figure 2.5, where Xd and Xc satisfy the condition that the probability of an error of the first kind continues to be IX: (6.2)

The obvious basis for comparing the two tests, since they have the same probability of a Type I error, is the relative magnitudes of the probabilities of their Type II errors. Figure 2.6 shows the relation to the alternative distribution p{x: ell of the two alternative critical regions. If PI and P2 are the probabilities of

102

STATISTICAL IDEAS

CHAP.

p

------------~=-------------~x

Figure 2.S

pjx!

------------------~~--~x Xc

Critical region (original test) -

Xd

~Critical region (alternative test) Figure 2.6

2

SECT.

2.6

CRITERION FOR CHOOSING BETWEEN ALTERNATIVE TESTS

103

error of the second kind of the two tests, then

/31 = J::p{x:O I } /32 =

J

"'d

-00

(6.3)

dx,

p{x:Od dx

roo +)"'. p{x:OI }

dx.

(6.4)

In Figure 2.6, it is clear that /32 is much greater than /31' so the first test is much to be preferred. To proceed on a more rigorous basis, we use the Neyman-Pearson lemma [7], which states that the critical region should include those values ofx for whichp{x:O I } is as large as possible relative to p{x:Oo}. We choose those x's for which

p{x:O I } > K, p{x:Oo}

(6.5)

where K has to be adjusted so that the level of significance of the test is a preselected value IX. In other words, (6.5) will define a region of x, say R, and we must adjust K so that (6.6)

The Neyman-Pearson lemma asserts that a test constructed in accordance with the above rule will have maximum possible power (and minimum probability of error of the second kind). A proof is as follows. Suppose that our choice of tests following the Neyman-Pearson lemma, leads to a critical region R containing a small interval !5x' about a point x'. Consider whether we can do better, i.e., obtain a test with greater power, by replacing this small interval !5x' by an alternative interval !5x" about a point x" not in the original critical region R. For the alternative test to have the same level of significance as the original test the amount of probability in the new interval added to R must equal that in the old interval which has been deleted from R:

p{x": Oo}!5x" = p{x': Oo}!5x'.

(6.7)

Multiplying both sides by K gives

Kp{x":Oo}!5x"

= Kp{x':Oo}!5x'.

(6.8)

The power of the original test was, (5.3),

[ p{x: 0l} dx .

•R

With the proposed change, we remove from this integral an amount

104

STATISTICAL IDEAS

CHAP.

2

p{x': OI}~X' and substitute an amount p{x": OI}~X". The increase in power, say !:l, is thus (6.9) We know that x' lay in the original critical region R, and so from (6.5) (6.10) whence, on multiplying both numerator and denominator by ~x', we get (6.11) Also we know that x" lies in the region of acceptance of the original test, so (6.12) whence, on multiplying both numerator and denominator by ~X", we get (6.13) By (6.8), the right-hand side of (6.13) is equal to the left-hand side of(6.11), so (6.14) and hence (6.15) The left-hand side of this inequality is identical with (6.9), the increase in power of the test given by the proposed substitution. Thus we have proved that the increase in power is negative, i.e., the change produces a decrease in the power of the test. Therefore the original test must have been optimum. When we have a k-variate multivariate observation x, y, ... , then in (6.5) in place of p{x: Ol} we would use p{x, y, ... : 0l} and analogously for the denominator; i.e., the criterion becomes

p{ x, y, ... : Ol} p{ x, y, ... : eo}

::.....>.......:....::-'---~

> K.

(6.16)

The proof that this criterion gives the most powerful test follows the same lines as for the univariate case, the difference being that the critical region is now a region in Ie-dimensional Euclidean space and the integrations determining oc and 7T are now k-fold.

SECT.

2.7

A ONE-SIDED TEST FROM A NORMAL POPULATION

105

When we have a sample of n independent identically distributed random variables, the criterion becomes, using (I.2),

P{X1'· .. , X/I:01} = P{X 1:0 1}· .. p{xn:O I } p{ Xl' •.• , Xn: eo} p{ Xl: eo} ... p{ Xn : eo} n

=

II P{Xi :0

1}

i n

II p{

Xi:

> K.

(6.17)

eo}

i

It will be noted that although we used a notation implying a single parameter family of distributions, p{x: O}, at no point was this restriction to a single parameter necessary, and 0 could be replaced throughout by 01 , ••• ,Or: the only requirement is that the values of all the parameters be precisely specified under both the null and alternative hypotheses. 2.7. A One-Sided Test of an Observation from a Normal Population with Known Variance In Section 2.5 we discussed the construction of tests of a null hypothesis against a simple alternative, and in Section 2.6 we saw that for this situation the Neyman-Pearson lemma (6.5) gave a method of constructing the most powerful test at a specified level of significance ()(. When the alternative hypothesis is not a simple hypothesis, e.g., 0 = 01> but rather a composite hypothesis, e.g., 0 > 00 or 0 ~ 00 , then usually there is no test which is more powerful than any other test for all possible alternative values of 0, i.e., usually a uniformly most powerful test does not exist. One instance in which a uniformly most powerful test does exist is the problem of testing the null hypothesis that the parameter ~ of a normal population is ~o against the alternative that ~ > ~o. For a particular value ~1 > ~o for the alternative hypothesis, to apply the Neyman-Pearson lemma (6.5) we need to consider the ratio

p{ x: ~1' O'2} p{ X: ~o, O'2}

(,,/2; O')-le-
= (.J27T O')-le-
2

= e-[<"'-h)2-<"'-so)2]/2u = 2

e"'
(7.1)

The second exponential here is some known positive constant, say c, since ~l and ~o are specified and 0'2 is known. The first exponential is of the form eO'" where a = (~1 - ~0)/O'2 is a positive constant since ~1 - ~o > O. Thus p{x: ~1' O'2}/p{X: ~o, O'2} has the form ceo", with both a and c positive:

106

CHAP.

STATISTICAL IDEAS

2

therefore it increases with increasing x. Thus to select all x for which this function is greater than K, we start with x at + 00 and allow it to decrease until we have taken in an amount of the null distribution p{x: ~o, 0'2} equal to the required level of significance IX. Let the smallest x which lies in the critical region be xc. Then Xc is determined by the equation (7.2)

Using (1.18.13) Pr{x

< xcl~ = ~o} = «l>(xc ~ ~o).

Putting this equal to 1 -

IX,

(7.3)

we get (7.4)

Thus in this instance the critical region is the same for all alternatives ~l > ~o, so here we have a uniformly most powerful test. For example, if we choose a level of significance IX = 0.05, Table I in the Appendix gives UO.95 = 1.645, and Xc = ~o + 1.6450', i.e., the bottom edge of the critical region is 1.645 times the standard deviation above the expectation under the null hypothesis. The power of the foregoing test by definition is the probability that we reject the null hypothesis ~ = ~o when ~ is equal to the alternative ~l; i.e., the power is the probability that a random observation x lies in the critical region, x > xc: 7T(~1)

= Pr{ x > xcl~ = =

1 - «l>(xc

~1}

= 1-

Pr{ x

< xcl~ = ~l}

~ ~1),

(7.5)

using (1.18.13). Substituting Xc from (7.4) gives

7T(~1) = =

1-

«l>eO + O'U;_a - ~l)

«l>( -U 1- a - ~o ~ ~1)

+ ~o ~ ~l)

= 1 _ «l>(u 1_a

= «l>(u a

+ ~l ~ ~o),

(7.6)

since 1 - «l>(u) = (-u) by (1.18.8), and -u1- a = ua by (1.18.11). We have so far considered tests based on a single observation. For n observations we use (6.17). Suppose we take a sample of n independent observations Xi from a normal distribution of known variance 0'2 and unknown mean~. We wish to test the null hypothesis Ho:~ = ~o against

SECT.

2.7

107

A ONE-SIDED TEST FROM A NORMAL POPULATION

the alternative ~ in (6.17),

> ~o at the level of significance IX.

Substituting for p{x;}

p{ Xl : ~l' ()'2} ... p{ xn : ~l' ()'2} p{ Xl: ~O, ()'2} ... p{ Xn: ~O, ()'2} 1-

(.)2

2

-

(,,27T()'r le- :ten 12(1 ••• (~27T()'rle-("n-~1)

21

2(1

2

= (~27T()')-le-("1-~o)2/2(12 • •• (~27T()')-le-("n-~o)2/2(12 e-(2(12)-1E("'-~1)2

_ -

2 -1

e-(2(1)

E("i-~O)

_

ei(St-~o)/«12In)e-(~12-~o2)/(2(12In)

2 -

•

(7.7)

This is similar to (7.1) except that it is a function of x instead of X and of ()'2jn instead of ()'2. We must have the critical region of size IX; i.e.,

Pr{x so, using (1.26.3),

Pr{x

Xc

> xc\~ =

~o}

=

(7.8)

IX,

must satisfy the equation

< xc\e = ~o} = (X:;J:O) = 1 -

(7.9)

IX,

whence (7.10) The power function is

7T(~l) = Pr{x > xc\~ = ~l} = 1 = 1_

(X:;J:l)

(7.11)

(eo + ul-aC()'/Jn) - ~l)

()'/Fn = <1>(Ua+

~~/~£o) = (l/a + <5),

(7.12)

if we define <5 = (~l - eo)f«()'//;;). For example, for a test at the level of significance IX = 0.05, ua = -1.645, and when <5 = 1, 7T(~l)

=

<1>(-1.645

+ 1) =

<1>(-0.645)

=

0.259.

(7.13)

Table 2.2, prepared in this manner, is plotted in Figure 2.7. In line with Table 2.2. Power of One-Tailed Test at Level of Significance 0.05 of Mean of n Observations from a Normal Distribution ~l - ~o

-1.0

-0.5

.004

.016

0

0.5

1.5

2

2.5

3

3.5

4

uj vii 7T(~l)

.050 .126 .259 .442 .639 .804 .912 .968

.999

108

STATISTICAL IDEAS

CHAP.

2

the definitions, the power is equal to the level of significance when ~l = ~o. Also when ~l < ~o, there is some probability, but very slight, of rejecting Ho: ~ = ~o in favor of the alternative ~ = ~l > ~o. 1.0 0.9 0.8 0.7

-.

0.6

~0.5 I<

0.4 0.3 0.2 0.1 0

-1

o

1 2 (~l - ~o)/((f/-vn)

5

Figure 2.7

Figure 2.8 illustrates the basis of the power calculation for a test with (f. = 0.05 when c5 = 1; i.e., ~l - ~o = 1 X a/J-;;. The critical region, corresponding to an area (f. = 0.05 in the upper tail of the null distribution, is x > ~o + 1. 645a/J-;;. The power, calculated in (7.13) as 0.259, is the area of that part of the alternative distribution lying in the critical region. We have been discussing the test with the alternative hypothesis that ~l > ~o. When the alternative hypothesis is ~l < ~o, the critical region is x < ~o + u,.a for a single observation, and (7.14) for the mean of 11 observations. In this latter case the power function is

7T(~I) =



(U,. + ~oa/y-IiI).

(7.15)

11

Thus for a test at the level of significance (f. = 0.05, u,. = -1.645, and for (~o - ~l)/( a/J-;;) = 1, 7T(~I) = <1>(-1.645 + 1) = 0.2594. Therefore we can use Table 2.2 as it stands for the power of this other alternative hypothesis, HI: ~ = ~l < ~o, if we replace (~l - ~o)/(a/J-;;) in the table by (~o - ~l)/(a/J~).

SECT.

2.7

109

A ONE-SIDED TEST FROM A NORMAL POPULATION

We can use (7.12) to calculate the number of observations necessary to give a specified power, I - p, assuming that a is known. Putting (7.12) equal to 1 - p,
i o)

+ ;1 -

=

al.Jn

1-

p.

(7.16)

a = 0.05 o~~~----------+-------~~~------------------

~~------------~x

Figure 2.8

But, in general,


= P, and so
Solving for

11,

and writing /I

+ ;1 - / ;0 = Lll_p.

u l _«

(7.17)

alv 11

for

p, whence

-u«'

= (Lll_p + Lll_«)2(

a

y.

;1 - ;oJ

(7.18)

This equation gives surprisingly large values of n for powers that we might consider desirable. Suppose a = 0.12, and we wish to test at the level of significance CI. = 0.01, so Ll l _« = UO.99 = 2.326. Suppose that when ~l - ;0 = 0.02, i.e., one-sixth of the standard deviation, we want a probability of 0.95 of rejecting the null hypothesis, so u l _ p = LlO.95 = 1.645. These values give 11 = 568.

110

CHAP.

STATISTICAL IDEAS

2

2.S. A One-Sided Test of an Observation from a Binomial Distribution Suppose that we have a sample of size n from a binomial distribution and that the observed number of defectives is x. Our null hypothesis is that () = ()o and the alternative hypothesis is that () = ()1, ()1 > ()o. Then from (6.5), we wish to maximize ( ()I):t (~)"-:t ()o 1 - ()o

(8.1)

Since ()1/()0 > 1, and (1 - ()1)/(l - ()o) < 1, this will be maximized by selecting for the critical region those values of x as large as possible, i.e., n, n - 1, n - 2, ... , subject to the condition that the probability of x falling in the critical region under the null hypothesis is cx. For a discrete distribution there is the difficulty that it will be impossible to get cx exactly equal to the prechosen value, e.g., 0.05. For example, if n = 10 and ()o = 1/4, if the critical region is 5 :::;; x :::;; 10, then cx

10 (10) (1):t( =~ - 1:t=5

X

4

-1)1o-:t = 0.0781, 4

(8.2)

and if we reduce the size of the critical region to 6 :::;; x :::;; 10, then cx

10 (10) (1):t( = ~6 X 4: 1 - 4:1)1o-:t=

0.0197.

(8.3)

It is possible to obtain exactly a prechosen level of significance, say cx = 0.05, by the following procedure. Let cx" (= 0.0781) and CX,,_1 (= 0.0197) be the probabilities under the null hypothesis of the observation falling in the critical region with n points and the critical region with n - 1 points respectively. Then cx" - CX,,_1 = 0.0584 is the probability of the observation falling on the nth point. Define y as

y

=

cx cx" -

CX,,_1 CX,,_1

= 0.0500 -

0.0197 0.0781 - 0.0197

= 0.519.

(8.4)

Then if the observation falls in the critical region with n - 1 = 5 points we reject the null hypothesis, and if the observation falls on the nth point we reject the null hypothesis with probability y = 0.519. This procedure

SECT.

2.9

111

THE TESTING OF COMPOSITE HYPOTHESES

will give the desired probability of rejecting the null hypothesis, for Pr{reject Ho} =

IX n - 1

+ Y(IXn -

= 0.0197

+ 0.519

IX n -l)

X

=

(8.5)

IX

0.0584 = 0.050.

(8.6)

In most practical situations, however, it would be more satisfactory to quote the P value, as discussed in Section 2.13. The main use of this randomized test procedure is for comparing tests in terms of their power, since for this comparison we need to have the tests with an identical IX. Using the critical region 5 ~ x ~ 10, the power when 01 = 1/2 is

1T(0 = ~) = pr{x ~ 51 0 = ~} = ~5 e~) G)\1 - ~r-z 1

= 0.6231, and the power using the critical region 6 ~ x of the randomized test at level of significance pr{reject HolO

= =

(8.7) ~ IX

10 is 0.3770. The power

= 0.05 is

=0 =~} 1

Pr{ x ~ 6} + Pr{ x = 5} Pr{ rejecting when x 0.3770 + 0.2461 X 0.519 = 0.5047.

= 6} (8.8)

2.9. The Testing of Composite Hypotheses In Section 2.6 we saw that the criterion (6.5) defined the best critical region for a test of a simple null hypothesis against a simple alternative. For multiparameter distributions (6.5) would become

p{ x: 01a' 02a' ... , °ka} p{ x: 010 , 020 , ... , 0kO}

~--~~~--~~>

K,

(9.1)

where the O's take the values 0iO under the null hypothesis and 0ia under the alternative (i = 1, ... ,k). This test, namely, rejecting the null hypothesis when x falls in the critical region, is equivalent to using the left-hand side of (9.1) as the test statistic and rejecting the null hypothesis when it is large. We can regard the parameters 01 , ••• , Ok as being represented in a k-dimensional space, the coordinates corresponding to the parameters. For a simple hypothesis, all the coordinates are specified, and we have a single point in the parameter space. For a composite hypothesis, a certain region in the parameter space is specified. For example, with a normal distribution there are two parameters, ~ and (12 (Figure 2.9). Specifying

112

CHAP.

STATISTICAL IDEAS

2

~ = 0, a Z = I and ~ = 4, a Z = 1, gives two simple hypotheses, represented by the points (0, 1) and (4, 1). ~ < 0, a Z = 2 is the line at a Z = 2 to the left of the ~ axis and is a composite hypothesis. ~ > is the entire right-hand quadrant and is also a composite hypothesis.

°

-

~> 0

(0,1)

x (4. 1)

0~~--~~--7---~~~

o

Figure 2.9

Now suppose that, though 010 is fixed under the null hypothesis, 01a is not precisely specified under the alternative hypothesis, and the remaining parameters Oz, ... , Ok are not specified under either hypothesis, so that we are comparing two composite hypotheses. The two hypotheses will define regions in the parameter space. The symbol w is given to the region corresponding to the null hypothesis, and we will use the symbol 0' to represent the region corresponding to the alternative hypothesis. We cannot compute the left-hand side of (9.1) to use as a test statistic, as the values of 010 and Oz, ... , Ok are all unspecified. However, we can compute an analog of (9.1), in which the unspecified parameters are given their maximum likelihood values. The numerator and denominator of (9.1) are both likelihood functions, and by definition the maximum likelihood estimates of the unspecified parameters will maximize the likelihoods. If L(w) and L(O') are these two likelihood functions, then the analog of (9.1) is L(O')/L(w) and we would reject the null hypothesis for large values of this statistic. The usual likelihood ratio test procedure is a slight modification of the above. If 0 represents the region in the parameter space made up of the union of the regions of the null and alternative hypotheses, i.e., o = 0' U w, then we reject the null hypothesis for small values of the likelihood ratio A, A=L(w). L(O)

(9.2)

[Note that A is a function of the observations and hence is a random variable. Although using a Greek letter (A) for a random variable violates

SECT.

2.10

A TWO-SIDED TEST FROM A NORMAL POPULATION

113

our practice of confining the use of Greek letters to parameters, the use of A- to represent the likelihood ratio is such a widely established convention that we continue the practice.] Clearly A- is positive since it is a ratio of products of probability functions which must always be positive. Also Acannot be greater than 1 since the maximum value for varying the parameters in a region w cannot exceed the maximum value for L varying the parameters in a region 0, where w is a subset of O. Thus A- must lie in the interval 0 to 1. A small value of A- indicates that the likelihood computed using w, corresponding to the null hypothesis, is relatively unlikely, and so we should reject the null hypothesis. Conversely, a value of Aclose to 1 indicates that the null hypothesis is very plausible and should be accepted. We therefore define a critical region as

0< A- < A-c'

(9.3)

where A-c is a constant chosen so that the level of significance of the test is at the desired level <x. In a two-parameter situation the test can be pictured graphically as follows. The two horizontal coordinates represent the two parameters and the vertical coordinate the likelihood. First, confined to region w we wander round till we find as large a likelihood as possible: this is L(w). Second, confined to a larger region 0 (which includes w) we wander round till we find a likelihood as large as possible: this is L(O). If L(O) is not much larger than L(w), i.e., if A- is close to 1, then the extra liberty of selection permitted under the alternative hypothesis has yielded small improvement in the likelihood, and so the null hypothesis is plausible. On the other hand, if L(O) is much larger than L(w), then the null hypothesis appears implausible. The likelihood ratio procedure produces a plausible statistic A-, and will indicate the general nature of the critical region. The critical region cannot be determined exactly for a test at the level of significance <X unless we know the distribution of A- under the null hypothesis. Wilks [9] showed that for large sample size (and moderate restrictions on the continuity of the likelihood function) the distribution of -210g A- tends to the x2(r - s) distribution where r is the number of parameters fitted under o and s the number of parameters fitted under w. 2.10. A Two-Sided Test of an Observation from a Normal· Population with Known Variance To apply the procedure of the previous section to testing that a group of n observations came from a normal population with mean ~o and known variance (j2 against the alternative that the mean is ~1 ¥: ~o, we write down

114

CHAP.

STATISTICAL IDEAS

2

the likelihood function (3.6):

2 2 1 )11/2 L= ( - - e- U / 2u )I:C"'i-S).

(10.1)

27T(J2

In Section 2.3 we saw that, allowing ~ to vary, L was maximized by putting ~ = :E xi/n = x, (3.10). Thus (10.2) Under the null hypothesis Ho no parameters are allowed to vary, since we are assuming (J2 to be fixed and known, and ~ is fixed at ~o; thus L(w)

= (_1_)n/2e-U/2U2)I:C"'i-SO)~

(10.3)

27T(J2

Substituting these values for L(w) and L(n) in the definition of the likelihood ratio, (9.2), and canceling out common factors gives e-U/2(2) I:C"'i-so)2

=

A.

(10.4)

e-C1/2u 2 )I:C"'i-:l:) " .

But 1 (Xi - ~O)2

=1

[(Xi - x)

+ (x -

~O)]2

= 1 (Xi -

X)2

+ n(x -

~O)2,

(10.5) since 21 (Xi - x)(x - ~o)

= 2(x -

~o)

1 (Xi -

x)

= 2(x -

~o) . 0

= O. (10.6)

Thus the likelihood ratio is (10.7)

The edge of the critical region, xo' is determined by e-ClI/2U")(:fc-so)'

Xo

= A.

(10.8)

0'

or, solving for xo'

= ~o - " -2 log A.o

;n

and

~o + .J-210g A.o

;ii'

(10.9)

Since A. c is less than 1, log A. o is negative, so Xc is of the form Xc

.J

= ~o ±

k

;n

(10.10)

where k = -2 log A. c• From (10.7) we see that if x is far from ~o, in either direction, then (x - ~O)2 will be large, and hence A. small. Thus

SECT.

2.10

115

A TWO-SIDED TEST FROM A NORMAL POPULATION

large values of Ix - ~ol call for rejection of the null hypothesis. The critical region is that part of the line below ~o - ka/..J~ plus that part of the line above ~o + ka/..J;;; see Figure 2.10. The total area of the curve in the critical region has to equal oc: since the normal distribution is symmetrical about its mean this implies that the area of each tail is oc/2. The critical region will therefore be -

x

a < 5"0 + 11«/2 ..J11 I:

an

d

x

a > 5"0t. + 111-«/2 ..J11 .

a/2

(10.11)

a/2 -ku/~

Critical region _

+ku/~

~o ~-- Region of acceptance --~~ Critical region

Figure 2.10

Since

U«/2 = -U1-«/2' the left-hand term can be written x < ~o -

ul_«/2a/..J~, and (10.11) is equivalent to saying: reject if (x - ~o)/(a/..J;;) is less than

-u1 -«/2 or greater than +u1 -«/2' This is equivalent to saying:

Ixa/.Jn - I >

reject if

~o

The power of this test, when

~

=

~l>

_«/2'

(10.12)

U1

is

7T(~I) = pr{xa0~0 < -U 1-«/2} + pr{xa~o > U1-«/2} ~1

X-

~1

-

~o

= Pr { a/..Jn + a/..Jn < -U 1-«/2

}

X - ~1 ~1 - ~o } 1 + Pr {---r: a/y n + a/y /n > U -«/2 ~o} X - ~1 = Pr {-----r < U«/2 - ~1 ----"j=- + 1 a/y n a/y n

X - ~1 --r < -U«/2 a/y n

- Pr {

~1 - / ~o} . a/y n

(10.13)

116

STATISTICAL IDEAS ~

Now under the alternative hypothesis

CHAP.

=

~l'

2

the random variable

(x - ~l)/«]/.J~ is a unit normal deviate. Each of these two probability statements is therefore of the form Pr{u < k} =
(10.14)

0.7

-

0.6

£ 0.5 t-

0.4 0.3 0.2 0.1 0 -4

-3

-2

5

Figure 2.11

Then remembering that 1 -
=


(10.15) For example, for 7T(~1)

= =

IX

=

0.05,


U O•05 / 2

=

U O•025

=

-1.96, and for ()

+

=

1,

+
+ 0.1685 = 0.1700.

(10.16)

Table 2.3, prepared in this manner, is plotted in Figure 2.11. The function is symmetrical about 0, and so only the first two negative values Table 2.3. Power of Two-Tailed Test at Level of Significance 0.05 of Mean of It Observations from a Normal Distribution ~l-~O

arvl';, 1T(~1)

-1.0

-0.5

0

0.5

.170

.079

.05

.079

.170

1.5

2

2.5

3

3.5

4.0

4.5

5.0

.323

.516

.705

.851

.938

.979

.994

.999

SECT.

2.10

A TWO-SIDED TEST FROM A NORMAL POPULATION

117

are tabulated. When ~1 = ~o, the power coincides with the level of significance. The power curve for ex = 0.01 is also plotted in Figure 2.11. We gain increased protection against the risk of committing an error of the first kind at the cost of decreased protection against the risk of committing an error of the second kind. For example, when 15 = 3, a test with ex = 0.05 has power 0.851, whereas a test with ex = 0.01 has power only 0.664. p

x!

region

(-0.96) = 0.1685

Figure 2.12

Figure 2.12 illustrates the basis of the power calculation for a test with ex = 0.05 when 15 = 1, i.e., ~1 - ~o = 1 x al.J;. The critical region, corresponding to an area ex/2 = 0.025 in each of the tails of the nuIl distribution, is x < ~o - 1.96al.J; and x > ~o + 1.96al.J;;' The power, calculated in (10.16) as 0.1700, is the area of the two parts of the alternative distribution lying in the two parts of the critical region. We can use (10.13) to calculate the number of observations necessary to give a specified power 1 - p, assuming that a is known. Equation (10.13) can only be solved exactly for n by troublesome iterations. However, in a typical situation one term is much larger than the other: for example, with 15 = 1 and ex = 0.05, the term (u a/ 2 + 15) at 0.1685 was more than 100 times as large as the other at 0.0015. Even at 15 = 0.5, this

118

CHAP.

STATISTICAL IDEAS

2

term is over 10 times the other. For any calculation involving d greater than 0.5, therefore, we can neglect the other term in (10.15) and write

7r(~1) '"" (U + ~~/~:O),

(10.17)

a/ 2

and putting 7r(~1) = 1 -

fl,

we get (10.18)

whence n '"" (u1_p

+ U1_a/ 2)2(

~l

(1

-

\2.

(10.19)

~oJ

For example, suppose (1 = 0.12, and we wish to make a test at the level of significance <X = 0.01, so that U 1 - a / 2 = U O. 996 = 2.58. Suppose that if ~l - ~o = 0.02, i.e., one-sixth of the standard deviation, we wish to have a probability of 0.95 of rejecting the null hypothesis, so u1 _ P = U O. 96 = 1.645. These values give n = 641. 2.11. A Two-Sided Test of an Observation from a Normal Distribution with Unknown Variance Let us now construct a test for the null hypothesis that the mean of a normal distribution is ~o against the alternative ~l =F ~O, the variance (12 not being specified. The likelihood function (3.6) is (11.1) For the null hypothesis, the parameter ~ is specified to take the value ~O, but (12 is allowed to vary. We saw that (3.12) gave the maximum likelihood estimator for (12 for a fixed ~ as 2 = ~ (Xi - ~)2/n. Substituting this, with ~ = ~O, in (11.1) gives

a

L( ) W

= [

n

27r L (Xi _ ~O)2

JlI/2e

-11/2

(11.2)

.

For the entire parameter space n, with no restrictions on the parameters ~ and (J'2, we saw in (3.10) that ~ = ~ xi/n and in (3.13) that 2 = ~ (Xi - X)2/ n. Inserting these maximum likelihood estimators in 01.1) we get

a

(11.3)

SECT.

2.12

119

THE COMPARISON OF TWO MEANS

Hence the likelihood ratio as defined in (9.2) is _ A-

[~ (Xi - X)2 J1I 12 . ~ (Xi - ~0)2

(11.4)

Any monotone function of A can be used instead of A itself for defining the critical region: hence we may use the nth root of A2, .2/n

A.

=

~ (Xi - x)2 ~

= [1

(Xi - ~0)2

+

_1_

=

~

~ (Xi - X)2 (Xi - X)2 + lI(X - ~0)2 (x -

~0)2

n - 1 [~(Xi - x)2/(n - 1)]/11 if we define t as t=

=

[ n ( x - ~0)2J-1 1 + --'---"'-~ (Xi - X)2

J-1 = [1

+ _12_J-1,

(11.5)

11 - 1

x - ~o X - ~o =--_-, .J[~ (Xi - X)2/(1I - 1)]/../11 s/.JlI

(11.6)

with s defined as in (3.23). We recall that we reject the null hypothesis for small values of A, which is equivalent from (11.5) to large values of 111. Thus 1as defined by (11.6) can be used as a test statistic for the null hypothesis ~ = ~o against the alternative ~ =;t:. ~o, '()'2 being unknown. We obtain this same result later, in Section 9.6, by a different route. 2.12. The Comparison of Two Means

Suppose that we have independent random samples, with means Xl and and sample sizes n 1 and 11 2 , drawn from two normal distributions p{x: ~1' p{x: ~2' ()'~}, and that the variances ()'~, ()'~ are known. Define d = Xl - x2 • Then by (1.25.4), E[d] = <5 = E[X1 - x2 ] = ~1 - ~2' The means Xl and x2 will have variances ()'~/n1 and ()'~/n2' and zero covariance, so by (I .25.5) the variance of d, V[d], equals ()'~/n1 + (),~/1I2' Also by Section 1.26, d is normally distributed, so

x2 ,

()'n,

d - E[d]

d - <5

---c===-='=--;===== ../ V[d] ../ (),~/1I1 + ()':/n 2

(12.1)

is a standardized normal variable u. Just as in the two preceding sections we outlined tests for the null hypothesis that a sample mean X with variance ()'2/n came from a normal distribution with expectation ~o, so we can test the null hypothesis that d with variance ()'~/n1 + ()'~/n2 comes from a normal distribution with expectation <5 0 , For a one-sided test with <5 = <5 1 > <5 0 , (7.4) gives the critical region as (12.2)

120 and for the two-sided test 15 d

CHAP.

STATISTICAL IDEAS

=

2

15 1 =;!= 15 0 , (10.11) gives the critical region as

< 150 + ua/ 2J aUIl1 + a~/1l2

and

d

> 150 + 1I1- a/2J a~/1l1 + a~/1l2' (12.3)

The null hypothesis in which we are most frequently interested is, of course, 15 = 0, since this corresponds to ~1 = ~2' the two populations having the same expectation. 2.13. The Concept of P Value We have seen that for a one-tailed test against the alternative ~1 > ~o on an observation x from a normal distribution the critical region was defined by (7.4) as (13.1) and for a two-tailed test, adapting (10.12) to the case ofa single observation x with variance a 2 ,

I- a ~ol- > x -

(13.2)

1I 1- a/ 2•

The standard procedure is to observe x, and note whether it falls in the region of acceptance or rejection, and accept or reject the null hypothesis accordingly. There are often advantages to an inverse procedure. Instead of working to a fixed level of significance IX, and merely recording whether Ho is to be accepted or rejected at this level, we may choose to insert the observed value of the test statistic x in (13.1) or (13.2), replace the inequality sign by an equality, and solve for IX. The value of IX so found is known as the P value. The P value is thus the value of IX at which the decision regarding Ho would just be on the borderline between acceptance and rejection. For example, suppose we observe x = 125 from a normal population with variance known to be 100, and test the null hypothesis ~o = 100 at the level of significance (7; = 0.05 against two-sided alternatives. Then 1

125 - 100 I

JI00

= 2.5 > U O.975 =

1.96.

so we reject the null hypothesis at the level of significance 0.05. Alternatively we write 1

whence 1 - P/2

125 - 100 I ..)100 = 2.5 =

1I 1-P/2,

= 0.99379, and P = 0.012.

It is often more satisfactory

SECT.

2.14

CONFIDENCE LIMITS:

THE GENERAL METHOD

121

to know that the null hypothesis can be just rejected at an ex as small as 0.012 rather than rejected at ex = 0.05. 2.14. Confidence Limits: The General Method

As indicated in Section 2.2, a point estimate of a parameter may be obtained, but we know that this point estimate is a random variable distributed in some way around the true value of the parameter, and we may lead ourselves into error if we act as if the true value of the parameter were equal to our estimate. To give ourselves some guidance we want an interval which we are reasonably confident will actually include the true value of the parameter. The generally accepted method of handling this question is to construct what are known as confidence limits. The procedure which we will describe in this section will construct lower and upper 100(1 - ex) per cent confidence limits with the property that, when we say that these limits include the true value of the parameter, 100(1 - ex) per cent of all such statements will be correct (and 100ex per cent will be incorrect). Obviously we choose ex small: commonly used values are 0.05 and 0.01. We suppose that we have a sample of observations Xl> ... , Xn from a population with density function of known form p{x: O}, X being the random variable observed and 0 the unknown parameter for which we want a confidence interval. We suppose that we have an estimator T to estimate 0, that T is a function of the observed Xi' and that we know the form of the density function of T, namely p{T: O}. We discuss in the following section the construction of a confidence interval for the parameter ~ of a normal population N(~, 0'2) for which we have a sample of size n and for which the parameter 0'2 is known: in this case we know from (3.10) that X is an appropriate estimator for ~ with density function (1.26.1) and cumulative distribution function (1.26.3). Reverting to the general argument, if we assume that 0 equals some particular value, say 0', we can insert this value in the formula for p{T: O} and get the density function of T under this assumption. Under the assumption that 0 = 0', the distribution of T will have a PI point, say TI(O'), which will be determined by the equation

Pr{T< TI(O')IO

f

T1 (6')

= O'} = Loo

p{T:O'} dT= Pl'

(14.1)

Under the same assumption 0 = 0', the distribution of Twill also have a P2 point, say T 2(0'), which will be determined by the equation

Pr{T < T2(0')10 = O'} =

f Tz(6')

Loo

p{T: O'} dT = P2 •

(14.2)

122

STATISTICAL IDEAS

CHAP.

2

This equation, of course, implies that Pr{T> T2(0')10

=

O'}

= roo

p{T:O'} dT= 1 - P 2.

J

(14.3)

T s(8')

We also have from (14.1) and (14.2) that Pr{TiO')

< T < T2(O')10 =

O'} =

i

T'W)

p{T:O'} dT= P 2 - Pl' (14.4)

Tl(8')

The PI point TI(O') and the P 2 point T2(O') are indicated in Figure 2.13 for the assumption 0 = 0'. If we change the assumed value of 0 from 0' plT:6!

Figure 2.13

to some other value 0" then we change the distribution of T and so the values of TI and T2 will be changed. In other words, TI and T2 are functions of 0, say TICO) and T2(0). In principle, we can plot these two functions TICO) and TlO) against O. In idealized form these are plotted in Figure 2.14. Now assume that the true value of 0 is 00, Then TI(O) and T2(0) take the values TI(Oo) and T2(00), respectively, and Pr{T < TI(Oo)} = P lo Pr{T> TlO o)}

=

1 - P 2,

which imply (14.5) Now suppose that we have taken a sample of observations and have calculated the numerical value of the estimate, say To. In Figure 2.14 let the horizontal line parallel to the 0 axis through the point To on the T axis intercept the two curves T2(0) and TICO) at points A and B. Drop lines parallel to the T axis through the points A and B to intercept the 0

SECT.

2.14

CONFIDENCE LIMITS:

123

THE GENERAL METHOD

axis at points f} and 0. We assert that the interval (f), 0) is a P 2 confidence interval for 0 ; i.e.,

°

°

-

PI

(14.6)

The justification for this assertion is as follows. Enter 0, the true value of 0, on the axis. Erect the perpendicular at this point to cut the curves T1(O) at C and T2(O) at D. At both these points has the value 0 , so, at C, T = T1(Oo), and, at D, T = T2(Oo). Now draw the horizontal lines

°

°

°

L-------~----L-~----~------------~6

Figure 2.14

through C and D to cut the T axis at T1(Oo) and T2(Oo) respectively. As drawn, Figure 2.14 has three properties: (a) AB intersects CD,

(b) To lies in the interval (T1(Oo), T2(Oo», (c) The interval (f), 0) includes 0 •

°

Figure 2.15 is constructed by the same procedure as Figure 2.14 the difference being that in Figure 2.15 we assume that 00 is greater than 0. Figure 2.15 has three properties: (d) AB does not intercept CD,

(e) To does not lie in the interval (T1(Oo), T2(Oo»,

(f) The interval (f), 0) does not include

°

Thus the two statements (b) To lies in the interval (T1(Oo), TlO o», (c) The interval (f), 0) includes 0 ,

°

0•

124

STATISTICAL IDEAS

CHAP.

2

are always true simultaneously or false simultaneously. But by (14.5), the event (b) has probability P 2 - PI' so the event (c) must also have the same probability, namely P 2 - Pl' Hence we can write (14.7) It will be noted that ~ and 0 are functions of To, which is a function of the observations and hence is a random variable, and therefore ~ and 0

L---~--~----~-L------------------~9

Figure 2.15

are random variables. The confidence interval W, 0) is thus a random variable. On repeating the experiment of taking the sample, calculating the estimate To, and constructing the confidence interval, in general we will get different numerical values for ~,O. Nevertheless, our procedure guarantees that in a portion P 2 - PI of those experiments the intervals W, 0) will include the true value of

e.

2.15. Confidence Limits for the Mean of a Normal Distribution with Known Variance We will now illustrate the above procedure by finding confidence limits for the mean; of a normal distribution with known variance a 2 • We assume that we have a sample Xl' ... ,XII' We know that the mean of the observations, say X, is an unbiased estimator of ;, with density function (1.26.1). We also know from (1.26.3) that the cumulative

SECT.

2.15

CONFIDENCE LIMITS OF A NORMAL DISTRIBUTION

125

distribution function of X is

Pr{X < x}

=1'" p{x:n dx =
-00

(15.1)

For (14.1) we can write

- < xi~)} =f,"l(~)p{x:~} dx =

Pr{X

I

-00

Thus

xlm, corresponding to

a/

(15.2)

11

T I «() in the general case, is the solution to


PI

(15.3)

But we know that
xl~)

~+

=

lIPa

~-,

(15.7)

yll

and this is also plotted in Figure 2.16. For any value of the estimator X = x calculated from the observations, the confidence limits are given by dropping perpendiculars from the points of interception A and B to the ~ axis. Clearly £ is the solution to (15.7) when xl~) = x. Inserting this value for x2 in (15.7) and solving for ~, we get

m

t §:

Similarly, putting

=

-

x -

II Pa

a,_.

(15.8)

yll

xlm = x in (15.5), we get -

~

= x-

lII'I

a

,_.

yll

(15.9)

126

CHAP.

STATISTICAL IDEAS

2

So far we have left P 2 and P l unspecified, other than that P 2 - P l = 1 - IX. One rational basis for particularizing the choice would be to make the length on the (J axis of the interval (fl, 0) as short as possible. In general the mechanics of actually doing this are not easy, and even if

X~--------~~------------J{

Figure 2.16

they were, most people would usually still prefer confidence limits symmetric in probability, so that P l = 1 - P2 , which implies P l = 1 - P 2 = 1X/2. If the estimator has a symmetrical distribution, as it does in the present example, then the shortest interval coincides with the interval symmetric in probability. To revert to our example of the mean of a normal distribution, if we want 95 per cent confidence, so that 1 - IX = 0.95, we put

Pl

=

so we want UP2 = UO•975 = cent confidence limits are

~= -

x-

1 - P2

=

1X/2

= 0.025,

+ 1.96 and up! = UO•025 =

1.96 ..;(J_ , n

-

~

(15.10)

-1.96. The 95 per

(J

= x + 1.96 ..;_ . n

(15.11)

SECT.

2.16

CONFIDENCE LIMITS:

THE PIVOTAL METHOD

127

It is often convenient to write this as

x±

1.96 ~_.

(15.12)

"II

Frequently in practice the explicit geometric construction is omitted, and the confidence limits fl, iJ are found algebraically. Referring to Figure 2.14, we see that at the point A the function T2(() has () = fl and takes the value To, i.e., (15.13) 12(()') was defined as the solution of (14.3) for a specified () = ()' (and P2)' We can use equation (14.3) the other way round to find () as a function of T 2 • Thus fl is the solution when T 2«() = To. In other words, fl is the value of () which satisfies the equation

fOO p{T:fl} dT = 1 - P2 ,

)To

(15.14)

or equivalently (15.15) Similarly, in Figure 2.14 at the point B the function TI «() has () takes the value To, i.e.,

= iJ and (15.16)

T 1«()') was defined as the solution of (14.1) for a specified () = ()' (and PI)' Conversely, (14.1) determines () for a specified T I . In particular, for TI = To, () = iJ:

(15.17) For our example of the mean of a normal distribution, (15.15) becomes

f i

p{x:~}

dx

-00

whence ~ ~

=

-

=

x -

lip.

a

,_

(15.18)

(15.19)

"II

as before in (15.8), and from (15.17) we obtain (15.9) as before.

2.16. Confidence Limits: The Pivotal Method In some situations an easier method can be used. This method requires that we can find a function, say Y, known as a pivotal junction, of the

128

CHAP.

STATISTICAL IDEAS

2

sample observations and the parameter for which we want confidence limits with the property that its distribution does not involve the parameter in question or any other parameters. Then Y will have PI and P 2 points YI and Y2 such that (16.1) and we may be able to manipulate the inequality inside the braces to yield a confidence interval. For example, for the problem of finding confidence limits for the mean of a normal distribution with known variance we can choose

_

y-

X-~

(16.2)

,- .

a/yll

This has the desired properties of being a function of the observations since it contains X, and of the parameter in question since it contains ~, and its distribution, being a unit normal distribution, is independent of the parameter ~ and the other parameter a. We can therefore write pr{II P1

< X ~} < II P2} = P 2 a/v II

Multiplying the inequality by direction yields pr{x - IIP2 a_ .JII

-a/0t,

(16.3)

Pl.

adding X, and then reversing its

<~<X-

IIPl

~-} = P2 -

PI'

(16.4)

"1111

which is equivalent to (15.8) and (15.9). This method will also work in the problem of finding confidence limits for the mean of a normal distribution with unknown variance. We use S2, defined in (3.23), as an estimate of a 2 , and take as our function Y, X-~ y=--

s/..}Ii .

(16.5)

This is a function of the observations, since X'and s are functions of the observations, and of the parameter~. It will be shown in Section 9.6 that the distribution of Yas defined in (16.5) does not involve ~ or a 2 and is what is known as the t distribution with 11 - 1 degrees of freedom. This is well tabulated (e.g., Table II in the Appendix), so we can readily write (16.6)

SECT.

2.17

CONFIDENCE LIMITS FOR THE PARAMETER

0

129

and by manipulations analogous to those performed on (16.3) we obtain (16.7)

2.17. Confidence Limits for the Parameter 0 of a Binomial Distribution As an example for which the pivotal method cannot be made to work, consider the construction of confidence limits for a binomial parameter O. The maximum likelihood estimator of 0 is 9 = X/no We know from 0.9.3) that X has probability function

x and hence the probability function of 9 =

p{h} =

(nnil )0""(1 _

0)"-"",

= 0, 1, ... , II,

(17.1)

x/n = 11, say, is

h

=

1 2

0, - , - , ... , 1. n n

(17.2)

However, we cannot find a function of hand 0 with the property that the distribution of this function does not involve O. We therefore have to use the method of Section 2.14. For (14.1) we can write Pr{H:::;; hl(O)}

='1~) (':~JOII"(1 - oy-"" = Pl'

(17.3)

It is perhaps easier to consider the equivalent equation (17.4) For a fixed PI, each value of 0 will determine a corresponding value of XI(O). We encounter a difficulty, however. The variable x is discrete, running over the integers to 11, and in general for a particular value of 0 we will be able to find for the upper index of the summation the largest integer such that the summation is less than PI and for the next integer the summation will be greater than Pl' Thus when we attempt to construct the graphs of xl(O) against 0, solutions for xl(O) will only exist for a series of isolated points, namely those values of 0 that correspond to integral values of x 1(O) in (I 7.4). However, our use of this graph of xl(O) against 0 is to read off the value of 0 corresponding to an observed value of X, say x o, and since Xo must be an integer the fact that our graph exists only at the integers is no inconvenience.

°

130

CHAP.

STATISTICAL IDEAS

2

The analog of (IS.17) is

!p{x:O} ",=0

=:2 (11)0"'(1 ",=0

X

oy-'"

=PI'

(17.S)

and this determines 0 for a specified 11, X o, and Pl' The analog of (I S.14) is

",!'p{x:f)}

=

",!J~)f)"'(l -

f))"-'" = 1 - P2,

(17.6)

which implies (17.7) The solution of these equations from first principles would be tedious. In Section 3.S we give a rapid method that uses readily available tables.

2.18. The Relationship between Confidence Limits and Tests of Hypotheses The construction of 1 - IX confidence intervals symmetric in probability has a close connection with the testing of hypotheses at the level of significance IX against two-sided alternatives. We will consider first the specific case of an observed mean x, being an estimator of the parameter ~ of a normal distribution with known variance 0'2. For the case of IX = O.OS we have from (lS.ll) that the lower and upper confidence limits for ~ are

~ = x - 1.96 ~_, vl1

-

~

0'

= X + 1.96 ,_.

(J 8.1)

VII

These are sketched in Figure 2.17a. Now consider testing the null hypothesis that ~ = ~ against the two-sided alternative ~ ¥:~. From (10.11), with IX = O.OS,-if XCLW and XCU(~) are the lower and upper edges of the critical region, then the critical region is -

X

< xcL(f) =

f - 1.96

0'

-

X > XCU(~) =

r'

~ -

-

v l1

0' + 1.96 VII r'

(18.2)

Similarly, if X(,L(~) and XCU(~) are the lower and upper edges of the critical region for testing the null hypothesis that ~ = ~, then the critical region is -

X

_ < xCL(~) =

-

~ -

1.96

0'

r'

VII

-

X

-

0'

> XCU(~) + 1.96 r' VII

Equations (18.2) and (18.3) are sketched in Figure 2.17b and c.

( 18.3)

SECT.

2.18

131

CONFIDENCE LIMITS AND TESTS OF HYPOTHESES

It is apparent from (b) that if we tested the null hypothesis that ~ had some value less than f" then the region of acceptance would move downwards, X would fall above the region of acceptance, and the null hypothesis would be rejected. Likewise, it is apparent from (c) that if we tested the null hypothesis that ~ had some value greater than ~, then the region of Confidence interval for

~

/r------~A~_ _ _ _~\

I

(a)

I

I

(c)

=

XCLW

~ - 1.96

-l.;

~

xcu(~) = ~ + 1.96 ..;

Figure 2.17

acceptance would move upwards and Xo would fall below the region of acceptance and the null hypothesis would be rejected. Conversely, if we tested the null hypothesis that ~ had any value in the interval (~, ~), then the region of acceptance would cover X so the null hypothesis would be accepted. In other words, if ~' is any ~ in the interval (~, ~), then the null will be accepted for all values of and if is hypothesis that ~ = any ~ not in the interval (~, h then the null hypothesis that ~ = ~" will be rejected for all values of ~". In the general case, suppose that we have an observed value To of an estimator T of a parameter 0. Then f} is determined by (15.14) with 1 - P2 = (1./2:

r

r,

f

OO

To

p{T:f}} dT =

r'

(I.

-, 2

(18.4)

132

CHAP.

STATISTICAL IDEAS

and 0 is determined by (15.17) with PI

i

To

=

2

a/2:

a p{T:O} dT= -.

2

-m

(18.5)

Suppose now that we make a two-sided test at the level of significance a of the null hypothesis that 0 = fl. The critical region is defined by T < T CL(f}) and T > T c u(fl), where T CL(f}) and T c u(f}) are determined by (18.6)

f

m

'l'cu(~)

p{T:fl} dT =

a

..:.

(18.7)

2

Comparing (18.4) with (18.7), we see that To

=

(18.8)

Tcu(f}).

Thus in a test of significance of the null hypothesis that 0 = fl, .To lies exactly on the upper edge of the critical region so that we would be on the borderline of accepting or rejecting this null hypothesis. Similarly, the critical region for the test of the null hypothesis that 0 = 0 is defined by T < T CL(O) and T> T cdO) where T CL(O) and T cu(O) are determined by

f f

TCL(O)

-

(18.9)

a _ p{T:O} dT= -.

(18.10)

2

-00

oo

a

p{T:O} dT=-,

Tcu(8)

2

Comparing (18.5) with (18.9), we see that (18.11) Thus in a test of significance of the null hypothesis that 0 = 0, To lies exactly on the lower edge of the critical region, so we would be on the border line of accepting or rejecting this null hypothesis. Therefore, for a test of any null hypothesis that 0 equals some value in the interval (fl, 0), the null hypothesis would be accepted; for a test of any null hypothesis that (J is outside the interval (fl, 0), the null hypothesis would be rejected.

133

EXERCISES

EXERCISES 2.1. Suppose that ~ is an unbiased estimator of a parameter 8, and that this estimator fj has a variance V[fj] which is greater than zero. Show that ~2 is a biased estimator of 1)2 and determine the bias. 2.2. (a) Suppose that 02 is an unbiased estimator of a parameter 82 , and that this estimator has a variance V[02] which is greater than zero. Show that fj is a biased estimator of I). (b) For observations Xl> ••• ,XII from a normal distribution, S2 as defined in (2.3.23) is an unbiased estimator of a 2• It can be shown a2 /2(1I - 1). Show that, for observations from a normal distribution, V[s] that E[s] = a vI - 1/2(11 - 1).

=

2.3.

For observations from a normal distribution with variance a2 , S2

= I" (Xi

-

X)2/(1I - 1)

i II

has a variance 2(a 2)2/(1I - 1). Determine whether S2 or 62 =

I

(Xi -

x)2/11 has

• or the ratio the smaller mean square by calculating either the difference between

of the two mean square errors. 2.4. (i) For the geometric distribution (l C. 7), for a single observation X show that (a) the maximum likelihood estimator is fj = I/x, and (b),

E[~] = 8 -log 8

1- 8 .

(ii) For the Pascal di!>tribution (lC.8), for a single observation X show that the maximum likelihood estimator of 8 is a/x. This is a biased estimator. Show that (a - 1)/(x - 1) is un unbiased estimator of 8. 2.5. (a) Given a sample of size 11, namely Xl' ... , XII' from an exponential distribution,

p{x}

= 8e- 0OO ,

o~ X <

=0,

otherwise,

00,

where 8 is positive, find the maximum likelihood estimator of 8. (b) Given a sample of size 1 from an exponential distribution, find the density function of the maximum likelihood estimator of 8. Find the expected value of this estimator. (c) Suppose the exponential distribution in (a) is written in the form 1

p{x} = 1 e- oo /A• Answer the questions in (a) and (b) as applied to the maximum likelihood estimator of A. 2.6. Samples of size "i, i = 1, ... ,k, are taken from a population with fraction defective 8, and the number of defectives observed in the ith sample is

134

STATISTICAL IDEAS

CHAP.

2

Xi' (a) Form the minimum variance unbiased estimator of 0. (b) What is its variance? (c) Form the maximum likelihood estimator of 0. 2.7. Construct the best test at the level of significance rJ. = 0.05 for the null hypothesis that 1/2, -1 < x < I Po{x} = { 0, otherwise, against the alternative hypothesis that

P1{x} =

N( 0,

nn .

(a) State clearly the critical region for this test. (b) Calculate the power of this

test. 2.S. Do the same as in exercise (2.7), but interchange the null and the alternative hypotheses. 2.9. (a) What is the best critical region for a test at the level of significance rJ. = 0.05 of the null hypothesis that an observation X comes from an exponential distribution

= 0, otherwise, with 0 = 1/2 against the alternative hypothesis that 0 = 2. (b) What is the power of this test? 2.10. From long experience with a process for manufacturing an alcoholic beverage, it is known that the yield is normally distributed with a mean of 500 units and a standard deviation of 96 units. We receive a proposal for a modification which it is claimed will increase the yield (leaving the standard deviation unchanged). We propose to make a test of the null hypothesis that the yield remains unchanged at 500 units, using a level of significance 0.05. If the alternative hypothesis is ~ > 500, (a) with 50 observations, what is the critical region? (b) with 50 observations, if the yield actually is 535 units, what is the probability of rejecting the null hypothesis? (c) How many observations should we take to make the probability of rejecting the null hypothesis equal to 0.9 when the yield is actually 535 units? 2.11. Do the same as in exercise (2.10), but the proposal is now for a modification which may change the yield, in one direction or the other. If the alternative hypothesis is now ~ ~ 500, answer (a), (b) and (c) as in (2.10). 2.12. A material is obtained in nominal pound packages from two suppliers, A and B. From previous experience it is known that A and B have standard deviations 0.07 and 0.03 pound respectively. A sample of 100 packages from A is found to have a mean weight of 0.99 pound, and a sample of 400 packages from B has a mean weight of 1.01 pounds. (a) Find the P value for the null hypothesis that A and B are supplying the same weight, the alternative hypothesis being that A ~ B. (b) Give 95 per cent confidence limits for the difference B - A. 2.13. In a proposed test that the mean of a normal distribution is ~o against the alternative that it is ~l' ~l > ~o' the level of significance being rJ., a sample size n is chosen, the power of the test is calculated, and it comes out to be 0.5. This is regarded as unsatisfactory so the decision is made to change the sample size to 4n. What will be the power of the new test?

135

REFERENCES

2.14. The following is an example of a situation for which the likelihood ratio method of constructing a test proves unsatisfactory (the example is due to Charles Stein). X is a discrete random variable with probability functions under Ho and Hl as follows: X

-2 ~

Ho

2

Hl

pc

-1

0

2-~

~

2

1

~ 1- ~

(! -~) 2

1- c 1- ~

~--

-

2

~

-~

~~ (!2 -~)

1-

2 (1 - p)c

~ is a constant, 0 < ~ < 1/2, so Ho is a simple hypothesis. c is a constant satisfying the conditions ~/(2 - ~) < c < ~. P is a parameter not specified other than that 0 ~ p ~ 1, so Hl is a composite hypothesis. (a) Find the critical region given by the likelihood ratio test for a level of significance~. (b) What is the power of this test? (c) As an alternative test, consider rejecting Ho with probability ~ irrespective of the observed value of X. What is the power of his test? (d) As another alternative test, consider rejecting Ho if X = O. How does the power of this test compare with the power of the tests in (b) and (c)? 2.15. Suppose a sample of observations Xiv, f' = 1, ... , n; is taken from a normal distribution with mean ~; and variance (12, and then the sample estimator s~ of the variance (12 is calculated. Suppose that this is repeated for a total of k samples (i.e., i = 1, ... ,k). The variance of s~ is known to be (see Section 9.8) 2«(12)2/(n; - 1). What is the minimum variance unbiased estimator of (12?

REFERENCES 1. Savage, L. J., "The Theory of Statistical Decision," Journal of the American Statistical Association, 46 (1951), 55-67. 2. Wald, A., Statistical Decision Functions. New York: John Wiley and Sons, 1950. 3. Chernoff, Herman, and Moses, Lincoln E., Elementary Decision Theory. New York: John Wiley and Sons, 1959. 4. Fisher, R. A., "The Mathematical Foundations of Theoretical Statistics," Philosophical Transactions of the Royal Society, A, 222 (1922), 306-368. 5. Cramer, H., Mathematical Methods of Statistics. Princeton: Princeton University Press, 1951. 6. Kendall, Maurice G., and Stuart, Alan, The Advanced Theory of Statistics; Volume 1: Distribution Theory, 1958, Volume 2: Inference and Relationship, 1961. New York: Hafner Publishing Company. 7. Neyman, J., and Pearson, E. S., "On the Problem of the Most Efficient Type of Statistical Hypotheses," Philosophical Transactions of the Royal Society, A, 231 (1933),289-337. 8. Neyman, J., "On the Problem of Confidence Intervals," Annals of Mathematical Statistics, 6 (1935), 111-16. 9. Wilks, S. S., "The Large Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses," Annals of Mathematical Statistics, 9 (1938), 60-62.

CHAPTER 3

The Binomial, Hypergeometric, and Poisson Distributions

3.1. The Normal Approximation to the Binomial Distribution For evaluating binomial probabilities, particularly sums in either the lower or upper tails, certain extensive tabulations are of great value [1-4]. The fact that the binomial distribution can be approximated by a normal distribution with mean n(J and variance n(J(1 - (J), first demonstrated by Demoivre in 1733, is still of considerable theoretical and practical importance, however, and we will now sketch a proof. In the factorials in the binomial coefficient (:) in p{x} = (1I)(J"'(1 _ (J)"-'" = II! (J"'(l _ (J)"-"', x X!(II-X)!

(Ll)

we use Stirling's approximation, (1.2)

Thus ( II) x

II! = x!(n- x)!

'" -

II ,,+IAe-n.J"br x",+IAe-"'.J27T(1l - x)"-,,,+IA e-(n-"').J 27T

IIn+IA e- n - .J27T x"'+IA(1I - x)n-",+IA e-m-'H'" . 1

'" - - - - , - ; - - - - - - , - ; - - - -

(1.3)

The exponential terms cancel each other. We now multiply the numerator 136

SECT.

3.1 NORMAL APPROXIMATION TO THE BINOMIAL DISTRIBUTION 137

and denominator by n~; the numerator is then

(104) Thus (1.3) becomes

( 11)

~

X

(!!.)"'+~(_11 )"-O:+~

1

.j21T.j11

(1.5)

n- x

X

We can also write O "'+~ (1

o)n-o:+~

0"'(1 - 0)"-'" = - _ .j0 -'--------c.j(:=1==-::::;:::O)=-

(1.6)

Substituting (1.5) and (1.6) in (1.1), we get

1 1 Oo:+~ (1 - O)"-"'+~ p{x},-.oJ-= ~ ~ .j21T .j110(1 - 0) (x/n)"'+ [(n - x)/nY-"'+

[ 11 -

,-.oJ

1 1 [( X )O:+~J-l{ x ] .j21T.j nO(l _ 0) nO n(l - 0)

"-"'+~}-1 .

(1.7)

Taking logarithms to the base e: log p{x},-.oJ - log .j21T110(1 - 0) - (x

+~) log C~).

- (n - x+ !)2 log [ n(1n-- x0)J.

(1.8)

Now define Z=

x - nO

.jnO(l - 0)

(1.9)

.

We make the following substitutions in (1.8):

x

=

nO

+ z.jnO(l

x -=I+z

)1 - 0

nO

II -

X

n-x n(1 _ 0)

1

z

(1.11)

--,

nO

= n(1 - 0) - z.j nO(I

= -

(LlO)

- 0),

J

0 n(1 _ 0) .

- 0),

(1.12) (1.13)

Now z as defined in (1.9) is approximately a unit normal deviate, which means that large values will occur only with small probability. For example, Pr{lzl > 3},-.oJ 0.0027. Further, z.j(1 - 0)/110 will be small for

138

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

large n. It is therefore permissible to use the expansion log (1

+

y)

=

y2 y - '2

yll

+"3 - ... ,

(1.14)

which is convergent for Iyl < 1 and for which the first terms are an approximation of increasing accuracy as Iyl tends to zero, for the logarithm of (1.11) and similarly for the logarithm of (1.13). Thus log ( -x)

nO

and

log

=

log ( 1 + z

F!- R-

~

- -0) nO

~

0 - Z2 _ 1 - _0 + z -nO 2nO

[ zJn(1 0- 0)] = [n(1n-x] - 0) -z j n(1 _ z 2n(1 _

. ..

(1.15)

log 1 -

0

~

0) -

0

2

0) - ....

(1.16)

Making these substitutions in (1.8), we obtain log p{ x} '"""' -log .J27T110(1 - 0)

- [no

+ z.JnO(1

- [n(l - 0) X [-zjn(l

- 0)

+ .!][zjl 2

z.J nO(l - 0)

~ 0) -

nO

0 - z2 1 - 0 2nO

+ ... ]

+ ~]

Z2 211(/- 0) - ..

J

(1.17)

Collecting terms in powers of z, the coefficient of z is zero except for (1/2/;;)[.JOI(1 - 0) - .J(1 - 0)10], which tends to zero as n tends to infinity. The coefficient of Z2 is -1/2 plus a component in lin, which tends to zero as n tends to infinity. Higher terms in z also tend to zero; so

Z2

log p{ x} ~ -log .J27TnO(1 - 0) - - . 2 Substituting (1.9) for z, we get

(1.18)

(1.19) which is the normal probability density function with the expectation nO and variance nO(l - 0) of the binomial distribution. We have thus shown that the binomial distribution can be approximated by a normal distribution with the appropriate mean and variance.

SECT.

3.1

NORMAL APPROXIMATION TO THE BINOMIAL DISTRIBUTION

Sometimes we may prefer to deal with the observed proportion h Since h is a function of x, we use (l.13.8);

p{ h} ~

1

e-(l.-8)2/[20(1-8)/n1.

.J27T.J()(1 - ()In

139

= xln. (1.20)

The corresponding cumulative distribution functions are

<1>( x -

n()

)

.In()(l - ()

and

( h-() ) .J()(1 _ ()In

(1.21)

However, we are approximating a discrete distribution by a continuous distribution. In the discrete distribution of x the probability is concentrated at the integers, and when we approximate it by a continuous p

I

Figure 3.1

distribution the corresponding probability is spread over a figure approximately a rectangle with base stretching from -1/2 to + 112 on either side of the integer. In Figure 3.1 the integral

f:oop{x} dx will omit the shaded area from x to x + 1/2 which properly belongs to x. We therefore change the upper limit of integration to x + 1/2: Pr{X

~ x} ~ r",+1A p{x} dx = (x + 1/2 - n()).

Loo

.In()(l - ()

If on the other hand we wanted the probability that X

(1.22)

< x, the integral

140

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

would improperly include the area between x - 1/2 and x. We therefore integrate to x - 1/2: Q;-~p{x} dx =ct> 1/2 - ne) (1.23) Pr{X < x}""'" .

(x -

J

~n()(1

-00

- e)

This constant 1/2 that is added or subtracted to the limit of integration is known as the correction for continuity. Formulas (1.22) and (1.23) can be combined to give Pr{X

= x} = Pr{X ~ x} ,...... ct>(x + 1/2 ~ne(l

As an illustration, let n = 100, Pr{X = 10}""'" ct>(10

+ 1/2 -

Pr{X

ne) _ ct>(x - 1/2 - ne). - e) ~ne(1 - e)

e=

0.1, x = 10; then

100

X

(1.24)

0.1) _ ct>(10 - 1/2 - 100 X 0.1). ~100 X 0.1 X 0.9

~100 X 0.1 X 0.9 ~

< x}

I

ct>(0.1667) - ct>( -0.1667) = 0.5662 - 0.4338

(1.25) = 0.1324. The exact answer is 0.1319. The alternative forms of (1.22) and 0.23) referring to It = x/n, obtained by dividing the numerator and denominator of the argument of ct>( ) by n, are

Pr{H Pr{H

e), 1/2n - e). - e)/n

< h}""'" ct>(h + 1/2n -

(1.26)

< h} ,...... ct>(h -

(1.27)

-

~e(1

- e)/n

~e(1

For a rigorous proof of the results of this section, with a careful attention to remainder terms, etc., the reader is referred to Uspensky [5] or Feller [6]. The approximations developed in this section are usually satisfactory if ne(1 - e) > 9. Thus if e = 1/2, an n of 36 is large enough, but if = 1/10, n needs to be ~ 100.

e

3.2. Testing Hypotheses about the Binomial Distribution with a Normal Approximation. Suppose that we observe x defectives in a sample of size n and wish to test the null hypothesis that the fraction defective e equals eo against the alternative hypothesis e = e1 wpere e1 > eo. This problem was discussed in Section 2.8, in terms of lilypothesis testing at a stated level of

SECT.

3.2

TESTING HYPOTHESES ABOUT THE BINOMIAL DISTRIBUTION

141

significance. Alternatively, we can set Pr{X

~ x} = v~J:)o~(1 -

0ot- V

= P,

(2.1)

and, for example, using the tables cited in [1], [2], [3], or [4], evaluate P. This will be the P value of the null hypothesis. To use the normal approximation to the binomial distribution, we note that Pr{X < x}

+ Pr{X ~ x} =

(2.2)

1,

whence, using (1.23),

P

= Pr{ X ~} x = 1-

Pr{ X <} x f""oJ 1 - ( .j

X -

1/2 - nOo) , nOo(1 - ( 0 )

(2.3)

so the equation x - 1/2 - nOo

111_P

(2.4)

= -.j"'7-,=;t0;;=0(!;::1=-=();;=0~)

has to be solved for P. For example, if we observe 60 heads in 100 tosses of a coin, and are testing the null hypothesis that the coin is fair, i.e., Ho: 00 = 1/2, against the alternative that it is biased in favor of heads, we refer

= 60 -

II 1-P

.jl00

1/2 - 100 X 1/2 X (1/2)(1 - 1/2)

=

1.900

(2.5)

to Table I, the cumulative normal distribution, obtaining 1 - P f""oJ 0.9713, whence P f""oJ 0.0287. The exact value, obtained from tables, is 0.02844. If the alternative hypothesis is 0 = 01 < 00 , we set Pr{X

~ x} =vt(:)O~(1 -

0oyn-v =P,

(2.6)

whence, using (1.22), (2.7) and the equation (2.8) has to be solved for P. For example, if we observe 40 heads in 100 tosses of a coin, and are testing the null hypothesis that the coin is fair against the alternative that it is biased against heads, we calculate = 40

II P

Whence P

f""oJ

0.0287.

+ 1/2 -

.jl00

X

100 X 1/2 = -1.900 (1/2)(1 - 1/2) ,

(2.9)

142

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

The calculation of P values for two-sided alternatives is most easily carried out by observing whether the observed number x falls in the upper or lower tail, and calculating the appropriate single-tail P value as above, and doubling it to get the two-tailed P value. Thus if we observed 40 heads in 100 tosses of a coin, and are testing the null hypothesis that the coin is fair against the alternative that it is biased in either direction, the P value would be 2 x 0.0287 = 0.0574. Note that with the one-sided alternative the null hypothesis would be rejected at the 0.05 level of significance, whereas against the two-sided alternative the null hypothesis would be accepted at this significance level. In computing the power of these tests, since H is approximately a normally distributed variable with mean 0 and variance 0(1 - O)/n, we can use the methods of Section 2.7 for one-sided tests and Section 2.10 for two-sided tests. There is, however, a slight complication here; the variance is not a constant but instead is a function of O. The calculation of the power of a test at a specified level of significance ()( for a specified alternative 0 = 0l. Ol > 00 , proceeds as follows. We want the probability of X falling in the critical region to be ()(. Let Xc be the lower end of the critical region. Then we require (2.10)

so (2.11)

and (2.12)

Solving for

Xc,

we get (2.13)

The power by definition is the probability that X falls in the region of rejection when the alternative hypothesis is true:

7T(OI)

= Pr{X

~ xclO

~ 1 _ (noo

= 0l} =

+ 1/2 +

1 - Pr{X

1I1- a

< xclO = 0l}

JnOo(1 - ( 0) - 0l) 1

-

1/2 - n(1)

J n0 (1 ~ 1 - (~;(Oo -

I

»).

°l) + 111 a J Oo(1 - ( 0 - J 01(1 - ( 1) 01(1 - ( 1)

If we define 0'0

= ,,/00(1 -

OO)/n,

0'1

=

(2.14)

J0 1(1 - (1)/n,

(2.15)

(2.16)

SECT.

3.2

TESTING HYPOTHESES ABOUT THE BINOMIAL DISTRIBUTION

143

this can be written as

7T(01) ~ 1 _ (0 0

~ 1-

-

(11

01 + 1I1-a (10) (11

<1>[ (U1- a + 00~ 01)::J (2.17)

When the alternative hypothesis is 01

7T(01)

< 00 , similar manipulations yield

~[ (ua + 0 ~ 01)::}

(2.18)

0

For a two-sided test, the power is the probability that X falls in the critical region when 0 = 01 :

7T(01) = Pr{X ~ nOo -

! + ua/2.JnOo(1 - 00 )1 0 = 01}

+ Pr{X ~ nOo + ! + U1-a/2.J nOo(1

t:J +

~ [ ( lIa/2 + 0 ~ 01 0

- 00 ) 10

[ ( ll a/2 - 00

= 01}

~ 01)

::J

(2.19) (2.20)

by manipulations similar to those leading from (2.14) to (2.17). These formulas will give the sample size necessary for a power of 1 - P of rejecting the null hypothesis at a level of significance ex when 01 - 00 has any specified value. For the one-sided alternative 01 > 00 , putting (2.17) equal to 1 - p, we get

( Ua

- 00)(10 + 01 (10 -(11 = u1_p·

(2.21)

Substituting for (10 and (11 from (2.16) and solving for n gives

n=

(0 1

1 -

00 )

2

[U1_p.J01(1 - 01) + u1-a.J0o(1 - OoW.

(2.22)

For example, if we wish to have a power 1 - P of 0.95 of rejecting the null hypothesis 00 = 0.05 at the one-sided level of significance ex = O.ot when actually 0 is 01 = 0.10, then /I

=

1 2 (0.10 - 0.05)

. .J0.10(1 [11095 .

- 0.10)

.J0.05(1 + 11099 .

- 0.05)] 2 = 400 (2.23)

144

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

3.3. The Angular Transformation and Other Variance Stabilizing Transformations It is convenient to be able to assume that the variance of a random variable, say X, is a constant, say (12. However, situations do arise in which the variance is instead a function, say g(~), of the expected value of X,~. For example, the variance of a binomial proportion H is 0(1 - O)/n, which is a function of O. In such a situation, to permit the use of statistical techniques which assume a constant variance, we need to transform X into a function of X, say cp(X), where the function cp( ) is chosen so that the random variable so defined has a constant variance. This we can do in the following manner. Let cp(x) be a function of a real variable x. By a Taylor series, cp(x) can be approximated by a linear function near any point ~:

cp(x)

~ cpW + cp'(~)(x _ ~) + cp"(~) (x ~! ~)2

Let X be a random variable with an expected value E[X] Y = cp(X) defines a new random variable Y: Y~

cp(~) + cp'(~)(X _ ~) + cp"(~) (X

-

2!

~)2

(3.1) =~.

Then

(3.2)

If X does not vary greatly about·~, so that the variance of X is small, when we take expectations we may neglect E[(X _ ~)2]: E[Y] ~ cpW

+ cp'(~)E[X -

~] = cp(~),

(3.3)

so Y - E[Y] "" cp'(~)(X - ~),

and V[Y]

= E[(Y -

E[Y])2] "" {cp'm}2E[(X -

~ {cp'(~)}2V[X].

(3.4)

w] (3.5)

In general, V[X] will depend on various parameters. If, however, it can be written as a function of the expected value ~, say g(~), then (3.6)

If we choose cpW so that its derivative cp'm satisfies

cp'(~) = .j:(~) ,

(3.7)

SECT.

3.3

145

THE ANGULAR TRANSFORMATION

then we will have made V[ y],....., 1. Equation (3.7) is equivalent to (3.8) For example, for a binomial proportion H with expectation 0, the variance V[H] = 0(1 - O)/n. Inserting this for g(~) in (3.8), we have

tfo(O)

J

=

dO .JO(I _ 0)/11

-J

=.J1l

dO .J O(l _ 0) .

This integral can be evaluated by changing the variable to t that t 2 = 0, that 2t dt = dO, and that

J.JO(1dO- -J .Jt 2t(1 dt- t - 2J.J1 dt- t 0) -

2

2) -

= 2 arcsin t = 2 arcsin

.Je.

(3.9)

=

.Je, so

2

(3.10)

Thus, if we use the transformation tfo(H) = 2--rn arcsin .JH, the variance of tfo(H) will be I. If we omit the factor .J;; and instead use "P(H) = 2 arcsin.J H, then the variance of this transformed variable will be lIn. Table V gives this latter transformation directly from a proportion H to "P(H) = Yin radians: for example, if we enter the table with H = 0.500, it gives directly 2 arcsin .JH as Y = 1.5708. We will note here several other transformations of frequent utility. First, if the variance is proportional to the expected value, so that V[X] = k~, then (3.11)

Thus a square root transformation will give a constant variance. In particular, if k = I, as occurs with the Poisson distribution, to be discussed in Section 3.11, 2.JXwill have variance 1, and.J X will have variance 1/4. Second, if the standard deviation is proportional to the expected value, so that V[X] = k2~2, (3.12)

so that log. X will have a constant variance k 2, and loglo X a constant variance O.l89k 2 • We might note that if "P(~) = log. ~, then "P'W = Ig and "P"(~) = -1/~2. Using the Taylor series up to the second-degree

146

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

term, Y r-J

tp(~) + tp'(~)(X

_

~) + tp"(~) (X - ~)2 , 2

(3.13)

so E[Y]

r-J

tp(~) + .! tp"(~)E[(X 2

_

~)2] ~ log. ~ _ V[~]. 2~

(3.14)

This shows that when we include the second-degree term in the Taylor series the expected value of the transformed variable Y differs from (3.3), the transformation of the expected value of the original variable X, by an amount in this case - V[X]/2(£[X])2. We might also note that on substituting l/~ for tp'(e) in (3.5) we get V[Y] _ V[X] - (E[X])2'

(3.15)

Thirdly, if the standard deviation is proportional to the square of the expected value, so that V[X] = k2~4, then

cp(~)

=J.Jd~

k 2~4

=Jd~ = - ~, k~2

k~

(3.16)

so that the reciprocall/X will have constant variance k 2 • To summarize these three results, to obtain constant variance, if V[X] ex: ~ we use X, if V[X] ex: ~2 we use log X, and if V[X] ex: ~4 we use

.J

l/X.

Of course, since we have been neglecting the higher order terms in these various Taylor series, the results of this section are approximations. We have discussed transformations chosen to give constant variance in the transformed variable. If the original variable is normally distributed, then the transformed variable cannot be. Frequently, however, the lack of constancy of variance (hetel'oscedasticity) is simultaneously associated with a nonnormality, and the transformation that gives a constant variance (homoscedasticity) also simultaneously gives a distribution closer to normal. In short, this is one case where usually we can both have our cake and eat it.

3.4. Testing Hypotheses about the Binomial Distribution with the Angular Transformation If H is a binomial proportion with expectation (J based on a sample of n, then Y = 2 arcsin iii in radians is approximately normally distributed with approximate expectation 'YJ = 2 arcsin and variance lin,

.JO

SECT.

3.4 TESTING HYPOTHESES ABOUT THE BINOMIAL DISTRIBUTION 147

and Pr{H ~ h}

1

p{y} dy

(2 arcsin .J h

+ 1/2/1 -

2 arcsIn Vh+l/2n

f'"..J

-00

f'"..J

'YJ),

(4.1)

1/.JII where the 1/211 is a correction for continuity analogous to that used with the normal approximation. For a one-sided test of the null hypothesis () = ()o against the alternative () = ()l < ()o, the critical region will be determined by putting (4.1) equal to ex. and writing he for h, 'YJo = 2 arcsin -/io for 'YJ; on rearrangement we get 2 arcsin

.Jh c + 1/211 = 2 arcsin .JOo + l/'~,

(4.2)

"II

which for specified ()o, ex., and n will determine hc. If H ~ hc' we reject the null hypothesis. The other one-sided test and the two-sided test can be constructed similarly. However, the angular transformation appears to give approximations no better than the normal approximation, and these tests are not customarily used. However, since the variance of the angular transformation is a constant, the calculation of powers and necessary sample sizes can be made more expeditiously with it than with the normal approximation, in which the variance is a function of (). Ignoring the correction for continuity, the formula (2.7.12) for the power of a one-sided test applies immediately: for the alternative 'YJl > 'YJo, 7T('YJl)

= <1>(11« + 'YJ~/-:;I~o).

(4.3)

The same formula, with 'YJl and 170 interchanged, applies for the alternative 171 < 'YJo. For the two-sided test, the analog of (2.10.15) is 7T('YJl)

= <1>(11«/2 -

'YJl -:; 'YJo)

1/

+ <1>(11«/2 + 'YJl -:; 'YJo). 1/

/I

(4.4)

/I

For calculating necessary sample sizes, for the one-sided test the analog of (2.7.18) is

= (lI l_p + 111_«)2(

)2

1 , 'YJl - 'YJo and for the two-sided test, the analog of (2.10.19) is

II

/I

~ (lI l_p + IIl _«/2)2(

1 y. 'YJl - 'YJo J

(4.5)

(4.6)

148

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

For example, if we wish to have a power 1 - f3 of 0.95 of rejecting the null hypothesis 00 = 0.05 at the level of significance oc = om, making a one-sided test, when actually 0 is 01 = 0.10, we use (4.5) with u1 _ P = U O•95 = 1.645, u1 - a = UO.99 = 2.326, 170 = 2 arcsin .JO.05 = 0.4510, 171 = 2 arcsin .J0.1O = 0.6435; then

n

=

(1.645 + 2.326)2 (0.6435 - 0.4510)2

= 425 '

(4.7)

which can be compared with the result n = 400 obtained for the same problem with the normal approximation in (2.23). Both of these are approximations; there is no obvious reason to prefer one to the other. An exact calculation would be difficult and not worth the trouble. 3.5. Confidence Limits for the Parameter of a Binomial Distribution Suppose that in a sample of n light bulbs from a very large (infinite) lot we find x defective. The maximum likelihood estimator of 0 is x/no We wish to place confidence limits on 0. This problem was discussed in Section 2.17 and it remains to solve (2.17.5) for if and (2.17.6) for fl for chosen values of PI and P 2 for observed values of x and n. For moderately smalln (i.e., n < 100), with some trouble in interpolation we can use the tables [1-4]. Alternatively, we can use the relationship, proved, e.g., in Section 21.3 of [7],

i (lI)ov(1 - o)n-v = 1 _

pr{F(fl,f2)

v=o 'V

< II - x_O_}, x + 11- 0

(5.1)

where F is a random variable with the variance ratio distribution. The distribution will be discussed in Section 9.3; it is a function of two parameters known as the "degrees of freedom" (fl,f2) and is tabulated for a number of upper percentage points in Table IV. In (5.1) the variance ratio F has degrees of freedom/I = 2(x + 1),f2 = 2(n - x). In Section 9.3 we will see that we can obtain lower percentage points from the relation (5.2)

For example, the 0.05 point of F with degrees of freedom 60 and 120 is Fo 05(60, 120) .

=

1 F 1_ O•05(120,60)

= _1_ = 0.680. 1.47

(5.3)

SECT.

3.5

LIMITS FOR PARAMETER OF A BINOMIAL DISTRIBUTION

149

We can use (5.1) for (2.17.5): 1 - pr{F(2(X

+ 1),2(/1 -

x»

- X ~} x+ll-0

< /I

= PI'

(5.4)

1 - PI

(5.5)

so

pr{F(2(X

+ 1),2(/1 -

x»

But the 1 - PI point of F is F 1-

Pr{F Thus

x+ll-0

e,

o=

PI ;

< F1- p J = 1 -

/1 - x e - ' --_ =

so solving for -

X ~} x+ll-0 i.e.,

< /I

F 1_ PI (2(x

=

(5.6)

PI

+ 1),2(/1 -

x»,

(5.7)

(x + I)F 1_ PI (2(x + 1),2(/1 - x» ----'----='--"-~'"'----'--'-'---'------'-'--./1 - X + (x + I)F 1_ P P(x + 1),2(/1 - x»

(5.8)

We can also use (5.1) for (2.17.7), noting that the summation in (2.17.7) is to x-I: 1 - pr{F(2X, 2(/1 - x

+ 1» < /I

-

X + 1 ~} x 1-~

= P2•

(5.9)

By similar manipulations we obtain

o= -

/I -

xF1_ P px, 2(11 - X + 1» X + 1 + xF1_ Pz(2x,2(1I - x

(5.10)

+ 1»

However, when P 2 is, for example, 0.975, we cannot read F1-0.975 = FO•025 directly from Table IV. We need to use (5.2) to substitute l/Fp p(n - x + 1), 2x) for F 1_ P px, 2(n - x + 1»: x 0=------------(/I - X + I)F pz(2(/1 - x + 1), 2x) + x

(5.11)

-

As an example, suppose that an automobile dealer observes that in a presumably random consignment of 11 = 30 automobiles x = 8 are characterized by inexcusably shoddy workmanship. For 95 per cent confidence limits we want P 2 = 0.975, PI = 0.025, 1 - PI = 0.975. Then from Table IV

F1_ P P(x

+ 1),2(/1 -

F pzC2(/1 - x

x» = FO.975(2(8 + 1),2(30 - 8» = FO.975(18, 44) = 2.07,

+ 1), 2x) = FO.975 (2(30 - 8 + 1),2 = F O•975(46, 16) = 2.49,

(5.12)

x 8) (5.13)

150

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

giving from (5.8) and (5.11)

o=

(8 30 - 8

0=

-

+ 1) x 2.07 + (8 + 1) x 8

+ 1)

(30 - 8

= 0.459,

(5.14)

= 0.123.

(5.15)

2.07

x 2.49

+8

For large n what is known as the quadratic normal approximation can be used. For the lower confidence limit fI we use (1.23) in (2.17.7) to give ( X

-

t - nfl)

.Jnfl(l - fI)

(5.16)

= P2,

whence

! - nfl = up . .J "fI( 1 - fI) 2 This gives a quadratic equation in fI, with the solution x -

(5.17)

-;======~

O=

-

1 n

+ U~2

{ 1 U~2 x - - +2 2

For the upper limit

-

II P2

[(X - !)(n - x

11

+ t) + -1I~2]lA} . 4

(5.18)

0 we use (1.22) in (2.17.5) to give (X

+ ! - 110) = PI'

(5.19)

.J110(1 - 0) which likewise leads to 0=

1

11 + 1I~,

{ x

2

+ -1 + -IIPl 2

2

[(X

lip 1

+ t)(11 "

-

X -

t)

2

U

+ -UPl ] } . 4

(5.20) For the example of 8 poor automobiles in a consignment of 30 (11 = 30, used for purposes of illustration, is rather small to justify using this approximation), straightforward substitution and the use of P 2 = 0.975, PI = 0.025 gives 95 per cent confidence limits of (0.130,0.462), to be compared with the exact result (0.123,0.459). 3.6. Comparison of Two Observed Frequencies with the Normal Approximation

Suppose that from two infinite populations with fractions defective 01 and O2 we draw samples of sizes 111. and 11 2 . and observe 1111 and l1u defectives respectively. The data are as in Table 3.1.

SECT.

3.6

COMPARISON OF TWO OBSERVED FREQUENCIES

151

Table 3.1

Defective

Nondefective

Totals

nn n21

n12 n22

n.l

n.2

n1. n2. n

Population 1 Population 2 Totals

We suppose that the row totals (sample sizes) are fixed by the experimenter, but the column total n. 1 , being the sum of two random variables nn and n21> is a random variable, and likewise n.2. We wish to test the null hypothesis that (Jl = (J2 = (J, say. Define h1_- nn ,

h 2 -- n21

nl.

(6.1)

•

11 2.

Under the null hypothesis, (6.2)

= (J(1 To estimate V[h 1

-

-

(J)(..l 111.

+ ..l).

(6.3)

n2.

h2 ] we need an estimate of (J(1 -

(J) = (J -

(6.4)

(J2.

Under the null hypothesis that (Jl = (J2 = (J we can use the column totals as referring to a single sample of size n from a population with fraction defective (J. From (1.15.8), n.l/n is an unbiased estimator of (J, and from (1.17.9), 1)

11. 1(11. 1 -

(6.5)

1)

11(11 -

is an unbiased estimator of (J2. Thus an unbiased estimator of (J(l - (J) is 11.1

11. 1(11. 1 -

11

11(11 -

1) 1)

=

11.111.2 11(11 -

1)

,

(6.6)

152

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

whence

V[h1 - h2 ] =

=

3

(..l + ..l) 11 2.

11.111.2 1) 111.

11(11 -

=

CHAP.

11.111.2 (112. + 111.) n(n - 1) 111.112.

n. 111. 2 (11 - 1)111.112.

(6.7)

Therefore, under the null hypothesis that E[11 1 ]

E[h 2 ]

-

= 0, (6.8)

111 11 1

1111 -

=

-'-' 11

(6.9)

J

[ 111.11.1112.11.2 n 2(11 - 1)

~

is a unit deviate asymptotically normal. The form (6.9) will be discussed in Section 3.10. Empirical investigation seems to indicate that the approximation is improved by replacing the 11 - 1 in (6.8) by 11. The statistic can then be written in the form .Jh{1 - 11)(1/111.

(6.10)

+ 1/11 2,)

where h = 11. 1 /11. The approximation is also improved by introducing corrections for continuity. If the alternative hypothesis is that 01 > O2 , then the P value is derived from u 1_p =

..:......::'-;::::::=====~=;===~~:::..:.

.Jh(l -

If the alternative hypothesis is 01

up=

+ 1/211 2,) h)(I/lIl, + l/lId

(hI - 1/2111.) - (h2

(hI

(6.11)

< O2 , then the P value is derived from

+ 1/2111 ,) -

(h2 - 1/211 2,)

.Jh(l - h)(l/lIl,

+ 1/11

.

(6.12)

2 ,)

For a two-sided test, if hI > 112 we use (6.11) with U1_ P replaced by U1-P/2, and if hI < 112 we use (6.12) with Up replaced by u p / 2• A criterion for deciding whether the approximation of this section is satisfactory is to compute the quantities ll i .ll,;/n for all four combinations

SECT.

3.6

COMPARISON OF TWO OBSERVED FREQUENCIES

153

of i and j. If in every instance this quantity is greater than 5, then the approximation is good. The figure 5 is perhaps rather conservative, and something of the order of 3.5 will be usually adequate. If this criterion is not satisfied, so we cannot use this normal approximation, then this situation can be handled with Fisher's exact test, described in Section 3.10. As an example, Table 3.2 shows the number of cases with reactions observed on using two types of rubber tubing for injection of a certain substance. Ifwe assume that patients were allocated at random to the two "treatments," rubber A and rubber B, any difference between the groups will be attributable to a difference between the rubbers. (On the Table 3.2 Number of cases Type of rubber A B Totals

With reactions

Without reactions

Totals

27 5 32

13 10 23

40 15 55

other hand, if random allocation was not used, it will probably be impossible to conclude anything useful from the data. If there is a significant difference, we will not know to what it is due.) The smallest of the four quantities ni.n.;/n is 23 X 15/55 = 6.27. Since this is greater than 5, the normal approximation of this section will be satisfactory. We calculate II = 32/55 = 0.5818 and then compute, for a two-tailed test,

1) (5 1) (4027 - 2;40 - 15 + h1s = 1.981 = !.i

[0.5818(1 -

0.5818)C~ + 115) ]

!/l-PI2,

(6.13)

whence P = 0.0476. We would thus conclude at the 5 per cent level of significance that the two groups differed in their percentage reactions, presumably on account of the different rubber tubing. For calculating powers and necessary sample sizes, the angular transformation is convenient: Yl = 2 arcsin -Jill' Y2 = arcsin -J~ have variances 1/n1, 1/n 2, writing n1 = nl" n2 = n2,> so V[Y1 - Y2] = 1/n1 + 1/11 2 , Let 'YJt> 172 be equal to 2 arcsin (Jt> 2 arcsin (J2 respectively. The power

-J

-J

154

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

of a two-sided test is given by appropriate substitutions in (2.10.15): 7T(01 - ( 2)

= (U"/2

-

'YJ1 - 'YJ2 ) -./1/111 + 1/112

+ (U"/2 +

'YJ1 - 'YJ2 ), -./1/111 + 1/112

(6.14) and for a one-sided test against the alternative 01 7T(01 - ( 2) = (U"

+

> O2 , (2.7.12) gives

'YJ1 - 'YJ2 ). -./1/111 + 1/112

(6.15)

For calculating necessary sample sizes we need to make some assumption about the relative magnitudes of /1 1 and /12' It is easy to show that /1 1 + /12 for a specified power, etc., is a minimum when /11 = /1 2 = m, say, so that the standard deviation ofYl - Y2 is -./2/m. For the two-sided test, (2.10.19) gives

m = (U1_P

+ U1_,,/2)2

2 2 ('YJ1 - 'YJ2)

'

(6.16)

and for the one-sided test, (2.7.18) gives 111

=

(U1_P

+ U1_,,)2

2 2 ('YJ1 - 'YJ2)

(6.17)

For example, if we wish to have a power 1 - {3 of 0.95 of rejecting the null hypothesis 01 = O2 at the level of significance ex = 0.01, making a one-sided test, when in fact 01 = 0.10, O2 = 0.05, then m

=

2(1.645 + 2.326)2 (0.6435 - 0.4510)2

= 850.

(6.18)

Note that m = 850 implies that /1 1 = /1 2 = 850, i.e., we need 850 observations on each population, making a total of 1700 observations.

3.7. The Correlated Two x Two Table The standard way of presenting the results of two sets of independent trials is a 2 x 2 table such as Table 3.1 and the null hypothesis that the proportions of "successes" is the same in each population can be tested by the methods of Section 3.6. However, there are some experimental situations which give rise to data which may also be put in a 2 X 2 table which cannot correctly be analyzed in that way. If we were given Table 3.3 (from Mosteller [8]) and asked to test the null hypothesis that the probability of nausea was the same with either

SECT.

3.7

ISS

THE CORRELATED TWO X TWO TABLE

Drug

Table 3.3 Nausea No nausea 18 10 28

A B

Totals

Patient 1 2

3 4 5

100 100 200

82 90 172

Table 3.4 Drug A DrugB N N

N

IV IV

N

IV IV

100

drug, we might be tempted to proceed with the normal approximation for the comparison of two proportions, Section 3.6, without pausing to inquire how the data was obtained. A naIve interpretation of Table 3.3 would be that we took two samples each of 100 patients and gave one set of 100 drug A and the other set of 100 drug B. In actual fact, the data was obtained from just 100 patients, each of whom received both drugs, with results as sketched in Table 3.~, where Nand N mean nausea and no nausea. Nine patients, like patient number 1, had nausea with both drugs; 9 patients like number 2 had nausea with drug A but not with drug B; only 1 patient, number 3, had nausea with drug B but not with A; and 81 patients like patient number 4 had no nausea with either drug. The data of Table 3.4 should therefore be summarized as in Table 3.5. Table 3.6

Table 3.5

DrugB N IV Totals

Drug A N IV

Totals

9 1 9 81 18 82

10 90 100

Drug A IV N DrugB N IV Totals

Totals

'lTn

'lTI2

'lTi.

1121

'lT22

'lT2.

'IT.I

'IT.2

156

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

As can be seen from the column totals, there were 18 cases of nausea and 82 without nausea with drug A, which is how the first row of Table 3.3 was formed. Likewise the row totals, 10 and 90, are the figures for drug B in Table 3.3. The essential features of the original data in Table 3.4 have been lost in the summarization of Table 3.3 but have been retained in Table 3.5. Table 3.3 can be constructed from Table 3.5, but the reverse is not true. Table 3.6 gives the population proportions corresponding to the observed frequencies of Table 3.5. We are interested in the difference 1T1. - 1T.l' which is 1T1. -

= (1Tll + 1T12) -

1T.l

( 1Tll

+ 1T21) = 1T12 -

1T21.

(7.1)

Thus in Table 3.5, the 9 patients who had nausea with both drugs, and the 81 who had nausea with neither, tell us nothing about the difference between the drugs. All the information on this question is contained in the other diagonal, presented in Table 3.7. If there was no difference Table 3.7. Patients Who Responded Differently to the Two Drugs

Favorably to A and unfavorably to B Unfavorably to A and favorably to B Total

1 9

10

between the drugs, we would expect these 10 patients to be split on the average 50: 50 between the two categories in Table 3.7. The one-sided P value for the null hypothesis is thus given by the sum of the terms in the binomial tail: Pr{X ~ X/II, O} With 0 = 1/2,11 Pr{X

~

=

X/II, O}

10,

X

=

'" (11) = v~o 'Jl OV(1

- 0t- v•

(7.2)

1, we have

= ~0(~)OV(1 - O),,-v =

~o 1

(10) 'Jl

(!)V(1 _!2}yo-v = l..2 (~ ~) 2 0! 10! + I! 9! 10

= 0.01074,

(7.3)

or with the normal approximation (1.22), Pr{X

~ x} =
(7.4)

EXERCISES 3A

157

whence P = 0.0134. Of course, since nO(1 - 0) = 10 X (1/2) x 0/2) = 2.25, which is less than 9, the use of the normal approximation cannot be justified in this instance. For the generalization of this test to the case of k drugs, k Section 7.10.

> 2, see

EXERCISES 3A 3A.l. For the approximation (1.2), calculate the ratio of the right-hand side to the left-hand side for n = 4 and for n = 9. 3A.2. The manager of a large automobile repair shop suspects that in 6cylinder engines, the valves of the front cylinder deteriorate faster than those of the other cylinders. Records are to be kept on 115 such jobs. (a) If a test of the null hypothesis 0 = 1/6 is to be made at the level of significance c£ = 0.05 against the alternative 0 > 1/6, what is the probability of the null hypothesis being rejected if in fact 0 = 1/5? Use both the methods of Sections 3.2 and 3.4. (b) If instead of fixing the sample size at 115, the sample size is to be chosen so that the probability of rejecting the null hypothesis () = 1/6 when in fact o = 1/5 is 0.90, what should this sample size be? Use both the methods of Sections 3.2 and 3.4. Suppose that the experiment is actually carried out and in 115 jobs 27 have the front cylinder in the worst condition. (c) What is the P value of the null hypothesis that () = 1/6 (assuming that the alternative hypothsis is () > 1/6)? (d) Give 95 per cent confidence limits for the proportion O. Since the sample size is rather large, a normal approximation will be satisfactory. 3A.3. Fertilizer was stored in drums of a certain type. After a certain period, it was observed that in a sample of 63 drums 1 had split seams. Calculate 90 per cent confidence limits for the proportion of split drums in that population of drums (a) by a normal approximation (b) exactly. 3A.4. The Chevalier de Mere thought that it paid to bet evens on (i) getting one or more sixes in 4 throws of a die; but not on (ii) getting one or more double sixes in 24 throws with a pair of dice. In point of fact, the true probabilities (assuming fair dice) for these events are 0.51775 and 0.49141 respectively. Suppose that you are planning experiments to test empirically the null hypotheses (a) that the probability of (i) is 0.5, against the alternative that it is 0.51775. (b) that the probability of (i) is equal to the probability of (ii), against the alternative that they are 0.51775 and 0.49141 respectively. Assuming one-sided tests of significance with c£ = 0.05, in each case, (a) and (b), how many observations should be taken to give probability 0.9 of rejecting the null hypothesis? 3A.S. Suppose that you are planning a clinical trial for a proposed vaccine against a disease which has a low and variable incidence from year to year. Therefore it is necessary to run a control group. Suppose that the average Incidence for a season is 1 per 50,000. Suppose that you wish the trial to have a

158

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

probability of 0.99 of rejecting the null hypothesis that the vaccine is without effect if in fact the vaccine reduces the incidence by one-half. Suppose that a level of significance 0.01 is to be used, and only a one-sided alternative is to be considered. Assuming that the two groups will be made the same size, what is the total number of subjects required? (Note that for very small values of x, sin x ~ x and arcsin x ~ x.) 3A.6. In an experiment to test whether seeding clouds of a certain type causes them to produce radar echoes corresponding to the occurrence of rain (R. R. Braham, L. J. Battan, and H. R. Byers, "Artificial Nucleation of Cumulus Clouds," Report No. 24, Cloud Physics Project, University of Chicago), on each suitable day a plane flew through two clouds that met certain criteria and one of the pair, chosen at random, was seeded and the other not. For 46 flights, echoes occurred in both clouds 5 times, in the unseeded cloud only 6 times, in the seeded cloud only 17 times, and in neither cloud 18 times. Find the one-sided P value for the null hypothesis that seeding is without effect.

3.8. The Hypergeometric Distribution Suppose that from a population of N elements, of which M are defective and N - Mare nondefective, we draw a sample of size 11, without replacement. What is the probability that our sample contains exactly x defectives,

p{x}? The following equations are evident: Pr{first is defective}

= ~,

(8.1)

I

Pr{second is defective first is defective}

=

(8.2)

M - 1,

N-l

I

Pr{xth is defective first, second, ... , (x - l)th are defective}

M - (x - I)

(8.3)

=----N - (x - I) ,

+ I)th is nondefective Ifirst x were defective} = N - M, N-x Pr{(x + 2)th is nondefective Ifirst x were defective and N-M-I the (x + I)th was nondefective} = , N-x-I Pr{(x

(8.4)

(8.5)

I

PrUx + (n - x)]th is a nondefective first x were defective and (x + I)th, ... , [x + (n - x - I)]th were nondefective}

=

N - M - (II N - (x + II -

X X -

1) I)

=

N - M - II +

X

N-II+1

+ 1

(8.6)

SECT.

3.8

159

THE HYPERGEOMETRIC DISTRIBUTION

Thus the probability of drawing the sequence D··· D D··· D, in which D recurs n times and D recurs n - x times, is the product of the probabilities (8.1), ... , (8.6):

Pr{D ... D D· .. D}

:=

M(M - 1) ... (M - x + 1) X (N - M)(N - M - 1) ... (N - M - n + x N(N - 1) ... (N - x + 1) X (N - x)(N - x - 1) ... (N - n

+ 1) (8.7)

+ 1)

If we multiply numerator and denominator by (N - n)! (N - M - n

+ x)! (M -

x)!

(8.8)

we obtain

Pr{D ... D D . .. D}

=

(N - n)! M! (N - M)! (M - x)! (N - M - n + x)! N!

(8.9)

This probability was derived for the specific sequence of x D's and n - x D's, but will hold for any other sequence of the same numbers of D's and D's. By (1.7.7) the number of such sequences is nt/x! (n - x)!. Thus the probability of obtaining x D's and n - x D's, the particular sequence not being specified and being regarded as irrelevant, is n! (N - n)! M! (N - M)! p{x} = x! (/1 - x)! (M - x)! (N - M - n + x)! N! =

(~)(~ =~) (~)

,

(8.10)

where there are certain restrictions on the variable x, namely

0::;; x::;; n,

O::;;x::;;M,

0::;; n - x::;; N - M.

(8.11)

This result can be obtained more directly by the following argument. There are

(~)

different ways, all assumed equally likely, in which to

draw a sample of size n from a population of N elements. We want the number of ways which give samples with x defective elements. The x defective elements must have been drawn from the M defective elements in the population, and this can be done in

(~) ways.

Likewise the n - x

nondefective elements must have been drawn from the N - M nondefective elements in the population, and this can be done in

(Nn-x - M)

160

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

ways.

These two drawings are independent.

CHAP.

3

Therefore there are

(~) (~ =-!') ways of drawing a sample of size

n with x defectives.

Thus p{x}, being the number of ways of drawing a sample of size n with x defectives divided by the total number of ways of drawing a sample of size n, is

which is the same result as (S.IO). The hypergeometric probability function (S.lO) is tedious to handle numerically, except with the aid of a published table which goes up to N ~ 100 [9]. We might expect that in the limiting case in which the population size is large compared with the sample size it might be approximated by a binomial probability function, since in this case the distinction between sampling without replacement (hypergeometric) and with replacement (binomial) tends to zero. This we will now demonstrate. We write (S.lO) in the form P{} x

=

M! (N-M)! x! (M - x)! (11 - x)! (N - M - 11

= M(M

- 1) ... (M - x x!

+ x)!

1I!(N-1I)! N!

+ 1)

x ..:...(N_-_M----'-'.)(N_-_M_-_1-'-)_.·_·-'-(N_-_M_-_1I_+-'--x_+-----"1) (11 - x)! 11! x -------------------N(N - 1) ... (N - 11 + 1) II!

= -------x! (11 - x)!

X X

M(M - 1) ... (M - x + 1) (N - M)(N - M - 1) ... (N - M N(N - 1) ... (N - 11 + 1)

II

+ X + 1).

(S.12)

Ignoring the first part of (S.12), n !/x! (n - x)!, the numerator is made up of two sequences of terms, the first sequence, M(M - 1)· .. , being x in number, and the second sequence, (N - M)(N - M - 1) ... , being n - x in number, so the total number of these terms is x + (n - x) = n. In the denominator, the sequence N(N - 1) ... is made up of n terms. Dividing both numerator and denominator by Nn, there will be one N

SECT.

3.8

161

THE HYPERGEOMETRIC DISTRIBUTION

for every term in the numerator and denominator. Denoting the proportion of defectives in the population, M/N, as we get

e,

p{x} =

11'

x!

.

(11 -

x)!

1)... (e- 'X-I) 1) ... (1 - ee('e- N ----J:j- (1 - e) ( 1 - e- N x

1) (

(

11 -

11 - ~ X-I)

1)

II-N"'I-~

(8.13)

e held constant, and 11 and x also fixed, p{x}""" (:)e"'(1 - et-"', (8.14)

If now N tends to infinity with

which is the usual binomial probability function. A rough criterion for the validity of the approximation is that n/N < 0.1; i.e., for the sample size to be less than 10 per cent of the population size. As an illustration of the degree of approximation given in one case with I!/N = 0.1, Table 3.8 gives the values of p{x} for the hypergeometric probability function with N = 1000, M = 20, n = 100 along with the values of p{x} for the binomial probability function with = M/N = 20/1000 = 0.02, n = 100. The column for the binomial with = I! = 200, and the column headed Poisson will be referred to in Section 3.11.

e

e om,

Table 3.8

H ypergeometric

Binomial

Binomial

Poisson

x

N = 1000, M = 20, n = 100

0 1 2 3 4 5 6 7 8 9 10

0.1190 0.2701 0.2881 0.1918 0.0895 0.0311 0.0083 0.0018 0.0003 0.0000 0.0000

0.1326 0.2707 0.2734 0.1823 0.0902 0.0353 0.0114 0.0031 0.0007 0.0002 0.0000

0.1340 0.2707 0.2720 0.1813 0.0902 0.0357 0.0117 0.0033 0.0008 0.0001 0.0000

0.1353 0.2707 0.2707 0.1804 0.0902 0.0361 0.0120 0.0034 0.0009 0.0002 0.0000

100

0.0000

0.0000

0.0000

0.0000

(J

= 0.02, n = 100

(J

= 0.01, n = 200

~=2

162

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

3.9. An Application of the Hypergeometric Distribution to Wild Life Population Estimation Suppose that we have an enclosed lake containing N fish. We wish to estimate N. We take a sample of size M and mark the fish and return them to the lake. We now have a population of size N containing M marked elements. Suppose that there exists a period sufficiently long to allow adequate mixing but not so long that births and deaths will have appreciable effects on Nand M. After such a period we take a second sample of size n and find x marked fish. The frequency function of x is, by (8.10), (9.1)

We know M, n, and x. We may estimate N by the method of maximum likelihood; i.e., we find that N which maximizes L = p{x}. We regard p{x} as a function of N, say PN{X}, and consider the ratio PN{x}/PN_l{X}. Increasing N from M we find the largest value of N for which this ratio is greater than 1. Since each successive PN{X} in this sequence is larger than its immediate predecessor, the last in the sequence must give the maximum value for L = PN{x}, The ratio

=

N2

-

N2 -

is greater than one for nM

+ nM , nN + Nx

MN - nN MN -

> N.'!;,

(9.2)

so the maximum likelihood estimator

N is the integer just less than nM/x. If we take N = nM/x, we might note that maximum likelihood estimators sometimes have disconcerting properties. Thus

E[N]

nMJ

= E [-

x

=

[IJ = nM

nME x

rnin11,M

!

- . (~)(~ ==~) .

1

~=O x

Since the summation includes the term for x

= 0,

(~)

for which l/x

(9.3)

=

00,

SECT.

3.10

FISHER'S EXACT TEST FOR TWO X TWO TABLES

the expected value of N is 00. However, the expected value of better behaved, being unbiased:

E[~J = [~J = _1 E[x] = _1 . nM =1., N

nM

nM

N

nM

N

163

liN is (9.4)

using the result of exercise (3B.2a). For a detailed examination of these topics see Chapman [10]. 3.10. Fisher's Exact Test for Two

X

Two Tables

In his Design of Experiments [II] Fisher discussed an experimental investigation of a lady's claim to be able to tell by taste whether the tea was added to the milk or the milk was added to the tea. In one form of such an experiment, four cups would be prepared by each method, making a total of eight cups, and presented to the lady for tasting. She would be informed that there were four of each. Each cup could then be categorized (A), according to how it was made, and (B), according to how the lady says it was made. The data from such an experiment could be represented generally as in Table 3.9. Table 3.9 Category B

Category A Totals

B1

B2

Totals

Al

n11

n12

n1.

A2

n2l

n22

n.1

n.2

n2. n

An important feature of Table 3.9 is that the row totals nl, and n2. were fixed by the experimenter, and also the column totals n. 1 and n. 2 are fixed, since presumably the lady, knowing that in fact there are precisely nl, cups with tea added to the milk, will adjust her judgements so that the sum of her n11 and n2l , equal to n. 1, equals the known number n1' Fisher's example has, as stated above, the feature that n. 1 = nl ., and hence n. 2 = n2.> but this is special to his example. More generally, consider an urn containing n balls of which nl, are marked Al and n2. are marked A 2 • Imagine that we have a row of n cells, n.1 marked B1 and n.2 marked B 2 • Then before the sampling starts nl" n2.> n.1> and n. 2 are all fixed, and there is no requirement that nl, = n.1' We now sample the

164

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

balls one at a time, without replacement, placing them serially in the cells, till all the balls have been withdrawn. Let l1ij be the number of type Ai balls which were placed in type D j cells. Then the results of this sampling experiment can be assembled as in Table 3.9. Since all the marginal totals are fixed, knowing say 1111 implies that we know all the other l1 ij . The probability of obtaining a particular value of 1111 can be calculated immediately from the hypergeometric probability function (8.10). We can suppose that we have a finite population of size N = 11 in which M = 111. elements are defective. We take a sample of size 11 = 11. 1, and in this sample we observe x = 1111 defectives. Then N - M

= 11 -

111.

= 11 2 .,

11 -

X

=

11.1 -

1111

=

1121>

(10.1)

and (8.10) gives P{lIn}

=

(Ill) ("2) = lin

"21

(n )

ilL! 11 2.! 1I. 1! 11.2! lin! 1112! 1121! 1I22! 1I!

(10.2)

11.1

This is the probability of obtaining precisely that value for 1111' For a test of significance, we want not only this probability but also the sum of the probabilities of the possible results more extreme in the same direction; i.e., we need to sum the tail ofthe distribution. For a long tail this calculation will be tedious, but it is some help to note that the factor (10.3)

is common to all terms in the series. The data of Table 3.2 will be used to illustrate the arithmetic, though it was not collected under the condition of both sets of marginal totals fixed which we have been supposing in this section. In Table 3.2 the row totals could have been fixed, but the column totals would be random variables and would therefore not be fixed. This point will be discussed further at the end of this Section. Table 3.10 contains the arithmetic. The logarithm of C, (10.3), is I

og

40! 15! 32! 23! 55!

= 47.911645 + 12.116500 + 35.420172 + 22.412494 = 44.757130.

73.103681 (lOA)

In constructing the upper part of Table 3.10, in this instance the observed proportion of reactions with rubber D, 5/15, is less than 27/40, and so we

SECT.

3.10

165

FISHER'S EXACT TEST FOR TWO X TWO TABLES

write down all possible tables in which this proportion is smaller than 5/15, always subject to the restriction that the marginal totals are unchanged. When the entry in this cell has gone from 5 to 0, it can go no further. This gives the 2 X 2 tables across the upper part of Table 3.10. The sum of the probabilities in the last row is 0.02406. This is the P value for this tail. For a two-sided test, the usual procedure is to double it, setting 0.04812. Table 3.10 Observed table 27 and more 5 extreme tables

13 28 10 4

12 29 11 3

11 12

30 2

10 13

31 1

9 32 14 0

8 15

log I/u ! log 1/12 ! log I/u ! log 1/22!

28.036983 29.484141 9.794280 8.680337 2.079181 1.380211 6.559763 7.601156

30.946539 7.601156 0.778151 8.680337

32.423660 6.559763 0.301030 9.794280

33.915022 5.559763 0.000000 10.940408

35.420172 4.605521 0.000000 12.116500

Sum log C- sum Probability

46.470207 47.145845 48.006183 :Z.286923 3".611285 4.750957 0.01936 0.00409 0.00056

49.078733 ).678397 0.00005

50.415193 0.341937 0.00000

52.142193 K614937 0.00000

Calculation of P values by the Fisher exact test may be somewhat tedious, but tables have been prepared by Mainland [12] giving the probabilities of all possible samples for 111 = 112 ~ 20 and the probabilities of samples in the region of probabilities 0.005 and 0.025 for 111 < 112 ~ 20. Another set of tables by Finney [13] deals with all samples, equal and unequal, up to 111 = 112 = 15, and these were extended by Latsha [14] up to 111 = 112 = 20. For large samples we can develop an approximate test. By exercise (3B.2) we know that 111. 11 .1

(10.5)

E[I1 11 ] = - - , 11

V[ 1111 ] -_

111.11.1112.11.2 112(11 -

1)

(10.6)

,

so that under the null hypothesis of random sampling under the specified model we will have a unit deviate approximately normal: 1111 1I

1111 -

E[I1 11 ]

.jV[I111]

III 11 1 _._. /I

(10.7)

166

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

It appears from empirical comparisons that the approximation is improved by replacing the n - I in (10.7) by n, so we use nn -nl.n.l -n

(10.8)

with a correction for continuity of ±1/2 with the sign chosen to reduce the absolute value of the numerator. An alternative form of the statistic (10.8) is (nnn22 - n12n21).J (10.9) U= 14 (n1.n.1n2.n.2)

n

which when corrected for continuity has its numerator written as

{/nnn22 -

~}.Jn.

n21 n12/ -

(10.10)

For the data of Table 3.2, these statistics give a P value of 0.0476, to be compared with the exact value of 0.04812. It will be noted that the statistic (10.7), developed for the 2 X 2 table with all margins fixed, is identical with the statistic (6.9), developed for the 2 X 2 table with one set of margins fixed. This suggests that the Fisher exact test can be used for the latter case when the sample sizes are too small to justify the normal approximation of Section 3.6. This in fact was shown to be the case by Tocher [15]. 3.11. The Poisson Distribution

One approach to the Poisson distribution is to consider a limiting case of the binomial distribution. For the binomial, E[x] == ~ = nO; we suppose that n tends to infinity and 0 tends to zero in such a way that nO = ~ remains a nonzero, noninfinite quantity. Then p{x}

=

(n)O:l>(1 _ x

= n(n

o)n-:I> =

- 1) ... (n - x n:l>

n! (~\:I>(1 x! (n - x)! nl

+ 1) e(1 x!

_

_ ~\n-:I>

~)n-:I> 11

nl

SECT.

3.11

167

THE POISSON DISTRIBUTION

As n tends to infinity, all the terms in brackets tend to 1, and so does (1 - ~/n)-", also the limit of (I - ~/n)" is known to be e- s, so

---+,x.

~.,

p{ x}

e- s,

x

= 0, 1, ... ,

(11.2)

and this is the frequency function of the Poisson distribution. From the form of the above derivation, it is apparent that the Poisson distribution can be used as an approximation to the binomial for large n and small 0. Table 3.8 gives the values of p{x} for the binomial with 0= 0.02, n = 100 and 0 = 0.01, n = 200 and for the Poisson with ~ = nO = 100 X 0.02. It will be noted that as n increases and 0 decreases the Poisson probability function becomes a better approximation to the binomial. The expectation of the Poisson distribution is easily found: 00

E[X]

~.,

00

=!

=!

x . e- s . .,=0 x!

xp{ x}

.,=0

= e- s[ o·

~o

- + 1 . -~1 + 2 . -~2 + ...]

o!

I!

2!

= e-s[o + ~(1 + :: + ;: + ...)] = e-L ~. eS = ~

(11.3)

since ~1

~2

es =1+-+-+···. 11 2!

(11.4)

To find the variance we use (1.17.3) which requires finding ~.,

00

!

E[X(X - 1)] =

x(x - 1)e- S -

x!

.,=0

~"-2~2

00

!

=

x(x - l)e- S ------=:--='----.,=2 x(x - 1)(x - 2)!

=

~2

i e-s (x e-- 2)! = ~2 .! e-.-s ~IIy! 2

.,=2

making the substitution y

=

,

(11.5)

11=0

x - 2 in the summation. Thus

E[X(X -

1)]

=

~2,

(11.6)

whence V[X]

= E[X(X =

~2 -

~(~ -

1)] - E[X](E[X] - 1) 1)

=

~.

(11. 7)

168

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

For a random sample of observations from a Poisson distribution, the maximum likelihood estimator is found by maximizing the likelihood, Xl, ••• , X n ,

(11.8)

or equivalently, maximizing the logarithm of the likelihood, log L=

-/1~ + (~Xi) log ~ -

To maximize, we differentiate with respect to

~

(1)" Xi!)'

-/1

(11.9)

and equate to zero:

+ (ni:Xi ) . ~1 =

d log L

~=

log

(11.10)

0

whence n

;: <;

LXi

i =- = -x.

(11.11)

/1

As an example of data with a distribution closely represented by the Poisson frequency function, R. D. Clarke [16] gave the numbers of flying bombs falling into areas of 0.25 sq. km in London over a certain period. There were 576 such areas; 229 received zero bombs, 211 received one bomb, etc. (Table 3.11); i.e., X = 0 on 229 occasions, X = 1 on 211 occasions, etc. No areas received more than seven bombs. Let 11", be ro

the number of areas which received X bombs: the maximum likelihood estimator of ~ is

229 X 0

=

537 576

+

L

/1",

=

211 X 1 + 93 X 2 + 35 X 3 +7X4+0 X5+0 X 6 576

= 0.932292.

11.

From (11.11)

",=0

+

1 x 7 (11.12)

SECT.

3.12

169

DERIVATION OF THE POISSON DISTRIBUTION

Inserting £ in place of ~ in (11.2) we get the estimated probability that a random area will have a particular value of x. Multiplying p{x} by n gives the expected number of areas with particular values of x. For example,

p{O}

= (0.932292)0 e-0.932292 = 0.393650,

(11.13)

O!

and np{O} = 226.742. The agreement of n", with np{x} seems satisfactory. The last two columns of Table 3.11 are required for a method of testing that the n", are consistent with the hypothesis that their expected values are the values np{x} derived under the assumption that x is Poisson distributed, to be discussed in Section 5.2. Table 3.11 x

11",

p{x}

IIp{X}

dt

d 2Je i

0 1 2 3 4 5 6 7 8

229 211 93 35 7 0 0 1 0 576

0.393650 0.366997 0.171074 0.053164 0.012391 0.002310 0.000359 0.000048 0.000006 0.999999

226.742 211.390 98.539 30.622 7.137 1.331 0.207 0.028 0.003 575.999

2.258 -0.390 -5.539 4.378 -0.706

0.0225 0.0007 0.3114 0.6259 0.0698

0.001

1.0303

Totals

3.12. An Alternative Derivation of the Poisson Distribution While the derivation of the density function of the Poisson distribution (11.2) as a limiting case of the binomial is simple, we may obtain more insight into its meaning and range of applicability from an alternative approach. Let the probability of an event occurring in a time interval (t, t + dt) be ~ dt, where ~ is a constant, and where the length of the time interval, dt, is so small that the probability of the event occurring more than once in this time interval is of a higher order of smallness than dt and may be neglected. Let the probability that the event will occur x times in the interval (0, t) be pit). With this notation Po(t + dt) is the probability that zero events will occur in the interval (0, t + dt). To get zero events in this interval means that we must have zero events both in the interval (0, t)

170

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

and in the interval (I, I + dl). The probability of the former is poet). The probability of the latter is (1 - ~ dt). We assume that these probabilities are independent, so Po(t

+ dl) = Po(t)(l

- ~ dl)

= poet) -

~Po(/)

dl,

(12.1)

whence , -,Po,-,-(I_+-,---d-'"t)_-~P=o(-"-t) dt

= _ ~poet).

(12.2)

The limit of the left-hand side is just the derivative of poet) with respect to t, namely dpo(/)/dl. We thus have a differential equation (12.3)

whose solution is (12.4) since, as we can readily check, (12.5)

The constant of integration in (12.3) is 0, since we must have PoCO) = 1, and e-s ' 0 = 1. Now consider pil + dl) when x> 0: pil

+ dl) = Pr{x events in interval (0, I + dl)} Pr{x in interval (0, I)} Pr{O in interval (I, I + dl)} + Pr{x - 1 in interval (0, Pr{l in interval (I, I = px(/) . (I - ~ dl) + PX-l(/) . ~ dl.

=

In

+ dl)} (12.6)

Rearranging gives

The limit of the left-hand side is the derivative of pil) with respect to (, so we have the differential equation (12.8)

A solution is (12.9)

SECT.

3.12

DERIVATION OF THE POISSON DISTRIBUTION

171

as is easily checked, since dp.,(t) = ~"xt"'-l e-st dt x!

=

+ (~t)'" (_~)e-st x!

~. (~t)"'-l e-st _ ~ . (~t)'" e-st (x-1)!

= ~P"_l(t) -

x! ~p.,(t).

(12.10)

Thus (12.9) does satisfy (12.8). Therefore the distribution of the number of events in the time interval (0, t) is given by (12.9), which is a Poisson distribution with parameter ~t. There is nothing special about the origin in the interval (0, t), so (12.9) applies to any interval of length t. From the present approach we see that if the probability of each radioactive atom in a mass disintegrating is a constant, then the number of atoms disintegrating in a time period t has the distribution (12.9). Similarly, if over a given part of the day the probability of a telephone call being received by a switchboard is constant, then the distribution of the number of calls per time interval is given by (12.9). The same would apply to the number of flaws per yard of insulated wire, the number of misprints per page, the number of blood cells on individual squares on a haemocytometer, etc. The Poisson distribution, which is a discrete distribution, is closely related to the negative exponential distribution (1.12.4), p{x} = ()e- B"', < x < 00, which is a continuous distribution. In the foregoing derivation of the Poisson we assumed that the probability of an event occurring in the interval (t, t + dt) was ~ dt. The probability that the event occurs times in the interval (0, t) was then found to be Po(t) = e- st, (12.4). We can write

° °

Pr{interval between successive events

=

Pr{event occurs

Hence

> t}

°times in interval (0, tn = e-st.

Pr{interval between successive events

< t} =

(12.11)

1 - e- st . (12.12)

The left-hand side of this is in the form of a cumulative distribution function for a random variable T defined as the interval between successive events. By (1.11.6) the corresponding probability density function is obtained by differentiation with respect to t. Differentiating (12.12) we set dP{t} dt

= p{t} = ~e-st

(12.13)

which has the form of the negative exponential probability density function.

172

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

In other words, if the probability of an event in the time interval (t, t + dt) is ~ dt, then the distribution of the number of events per interval of time t is Poisson of the form (12.9) and the distribution of time between successive events is negative exponential of the form (12.13). 3.13. Test of Hypotheses about the Poisson Distribution n

To obtain sums of terms,

L p{x}, of Poisson distributions we can use

.=0

Molina's table [17]. Alternatively, we can use a relationship with the cumulative X2 distribution, tabulated in Table III. From (5.1) the cumulative sum of the terms of the binomial distribution is given exactly as Pr{X ~ x}

= P{x} =

1 - Pr {F

n-x () } <---

x+ll-0

(13.1)

where the variance ratio F has degrees offreedom 2(x + 1) and 2(n - x). We have derived the Poisson distribution as the limiting case of the binomial with 0 = ~/n and 11 ~ 00. Taking these limits, for a Poisson variable, P{x}

=

1-

pr{F < "11-~ - x. -~-} = 1 - pr{F < -~-}, x+l x+l

(13.2)

where the degrees of freedom are 11 = 2(x + 1),12 = 00. We will see in Section 9.3 that F with degrees of freedom I and 00 is distributed as another quantity, X2 withl degrees offreedom, divided byI; i.e., F(f, (0) = X2(f)/J Thus

P{ x} = 1 - pr{

2(x

X2

+ 1)

< -~-} x +1

= 1 - Pr{l

< 2~},

(13.3)

where X2 has 2(x + 1) degrees offreedom. The X2 distribution is tabulated in Table III. For example, in the case of ~ = 0.932292, Table 4.4 gives the individual terms, so we find by summation Pr{X ~ 3} = 0.9848. Using (13.3), Pr{X ~ 3} = 1 - Pr{x 2(8) < 2 X 0.9323}. (13.4) For 8 degrees of freedom, the 0,01 and 0.025 points of the X2 distribution are 1.65 and 2.18, so 1.86 must correspond to about the 0.015 point. Hence Pr{X ~ 3} ~ 1 - 0.015 = 0.985. With such a simple formula as (13.3) for the cumulative Poisson distribution, there seems small practical need for approximations. However, substituting ~ for the expectation nO and ~ for the variance nO(1 - 0) in (1.22) which is a good approximation to the cumulative binomial when

SECT.

3.14

CONFIDENCE LIMITS FOR A POISSON PARAMETER

173

nO(1 - 0) is greater than 9, which corresponds in the Poisson case to ~ > 9, gives

(13.5) Another approximation is easily obtained. In (3.11) and following we saw that if the variance of a random variable was equal to the expected value, then the square root of the variable had a variance 1/4. Hence P{x} ,-..J(Jx

+ 1/2 - J~) = [2(Jx + 1/2 - J~)]. Jl/4

(13.6)

To illustrate these formulas, suppose that we observe 4 telephone calls in an hour when we have the null hypothesis that the mean hourly rate is 8. Formulas (13.3), (13.5), and (13.6) give

Pr{X::;; 4} = 1 - Pr{x2(10) Pr{X::;; 4} ,-..Je

+ j~

Pr{X ::;; 4} ~ [2(J4

-

< 16} = 8)

+ 1/2 -

= 0.10,

(13.7)

= <1>(-1.238) = 0.109,

(13.8)

J8)]

1 - 0.90

= <1>(-1.414) = 0.079.

(13.9)

It is clear that ~ = 8 is still somewhat on the small side for the approxima-

tions to be accurate. 3.14. Confidence Limits for a Poisson Parameter Equations (2.15.14) and (2.15.17) will give confidence limits for the population parameter for a Poisson distribution from an observation. Using (13.3), the upper confidence limit ~ is defined by the equation

Pr{X::;; xle = n = PI = 1 - Pr{x 2(2(x

+ 1» < 2n,

(14.1)

so

Pr{x 2(2(x

+ 1» < 2~} =

(14.2)

1 - PI'

but in general Pr{l(2(x so

+ 1» < XL Pl(2(x + I»} = ~

1 - PI'

= txLp (2(x + 1».

(14.3)

(14.4)

1

Likewise the lower confidence limit is defined by

Pr{X::;; x - lie

= n=

P2 = 1 - Pr{x 2(2x)

< 2i},

(14.5)

so (14.6)

174

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

so

1

~

2

(14.7)

= - Xl_p.(2x).

-

2

•

For example, if on an airplane coming off an assembly line we find one missing rivet, and if we can assume that the distribution of the number of missing rivets is Poisson, i.e., with constant probability and independent, then 99 per cent confidence limits for the average number of missing rivets per plane in the population of planes are -

~ =

21

2 XO.995

«2 1 + 1»

=

14.9

"""2 =

7.45,

1 2 ( 2 X 1) = 0.010 ..I: = -X0005 - = 0.005. -

2

.

2

3.15. Simple Sampling Inspection Suppose that a producer is supplying to a consumer lots of N items. Each item can be inspected and the inspection will lead to the conclusion that the item is either good or defective. Suppose that the number of defective items in a lot is M, and define () = MIN as the fraction defective. Suppose that the two parties agree that a sample of size n is to be taken, and if the number of defectives x is less than or equal to a specified integer c, then the lot is to be accepted; if x > c, the lot is to be rejected. From the arguments of Section 3.8, we have Pr{lot is accepted} = Pr{ x ~ c} (15.1) Under the usual industrial conditions, the lot size N is large and the sampling fraction nlN is small, say <0.1" so by the arguments that led to (8.14) we can use the binomial approximation to the hypergeometric distribution. Also, usually the sample size n is quite large and the fraction defective () is quite small, () < 0.1, so we may use the Poisson approximation to the binomial to write the probability that the lot is accepted as a function of (), say A«(), as

A«() = Pr{x

~ c}~~o(:)()~(1- ()'I-:IJ

(15.2) (15.3)

SECT.

3.15

175

SIMPLE SAMPLING INSPECTION

For handling sums of Poisson distributions we can use Molina's table [17] or (13.3): A(O) = Pr{x :::;; c} = 1 - Pr{x 2

< 2nO},

f

= 2(c

+ 1).

(15.4)

The graph of A(O) against 0 is known as the operating characteristic of the sampling plan defined by a specified nand c. For a fixed nand 'c, (15.4) shows that the probability of acceptance A(O) equals 1 when 0 = 0 and tends towards zero as 0 gets large. On what basis should we choose nand c? The producer and consumer have differing requirements. The producer may demand that if the lot is relatively good, the probability of its being rejected should be low. Specifically, if 0 = 00 where 00 is called the acceptable quality level (AQL) then the probability of the lot being rejected should be a small number Qt, where Qt is called the producer's risk. Thus the producer requires that A(Oo) = 1 - Qt. The consumer may demand that if the lot is relatively bad, the probability of its being accepted should be low. Specifically, if 0 = 010 where 01 is called the lot tolerance per cent defective (LTPD), then the probability of its being accepted is a small number p, where p is called the consumer's risk. Thus the consumer demands that A(OI) = p. Using (15.4), the demands of the producer and the consumer can be written as

< 2nOo} = 1 Pr{x 2 < 2n0 1 } = p,

1 - Pr{x2

1-

where both the X2'S have degrees of freedom 2(c the two equivalent equations 2nOo = X! 2nO I =

xLp.

Qt,

(15.5) (15.6)

+ 1).

We readily obtain (15.7) (15.8)

We can solve for c by finding the smallest value of 2(c + 1), the number of degrees of freedom of the X2's, which satisfies the inequality

xLp2 < 01 • X" - 00

(15.9)

Having found c, n can then be determined from either (15.7) or (15.8). Thus the separate demands of the two parties, the producer and the consumer, can be satisfied simultaneously. The situation is shown in Figure 3.2. By a suitable choice of nand c, the operating characteristic curve can be made to pass through the points (0 0 , 1 - oc) and (010 (3), and of course it always passes through the point (0, 1), since lots with 0 per

176

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

cent defectives will be accepted with probability I. Actually, because the distribution of x is discrete, in general an exact solution will be unobtainable, and we will have to settle for an actual oc less than or equal to the desired value, and, likewise for {3.

1 - ex \ - - - - - - - \

(3 Consumer's risk

o~----------~----------~------~~----~o

o

00 Fraction defective Figure 3.2

For example, if we have

eo =

0.03, oc = 0.05, and

X~.90 =

X~.05

0.08 0.03

=

e= 1

2.67.

0.08, (3 = 0.10, then (15.10)

For 20 degrees of freedom, X~.90/X~.05 = 28.4/10.9 = 2.61, and for 18 degrees of freedom X~.90/X~.05 = 26.0/9.39 = 2.77. On account of the distribution of x being discrete, we cannot obtain an exact solution. We need to take the larger of the numbers of degrees of freedom, 20 = 2(e + I), giving e = 9. From (15.8) we obtain 11

=

X~.9o(20) = 28.4 = 177.5. 2 X 0.08

0.16

Thus if we take e = 9, n = 178 we will satisfy our requirements, in the sense that oc ~ 0.05, (3 ~ 0.10. Using (15.4), A(e) can be calculated for a number of values of and graphed as in Figure 3.3 to give the operating characteristic of this sampling plan.

e,

SECT.

3.17

RECTIFYING INSPECTION

177

1.0 0.9 0.8

§ ~

.; (,) c:

15. .,'" (,)

u

'"

'0

0.7 0.6 0.5

~

0.4

'"

0.3

:0 .Cl

£

0.2 0.1 00 Fraction defective, (J

Figure 3.3

3.16. Relationship between Sampling Inspection and Hypothesis Testing There is a one-to-one relationship between the terminologies of sampling inspection and the testing of a simple null hypothesis against a simple alternative, as summarized in Table 3.12. The parameters 00' 01 , ot, and {3 have closely analogous interpretations under the two situations. It is traditional, however, to plot the probability of accepting the lot as a function of 0 to give the operating characteristic but to plot the probability of rejecting the null hypothesis as a function 0 as the power function, so the operating characteristic and power function are complementary (for any specified 0, each is 1 minus the other). 3.17. Rectifying Inspection The foregoing sampling scheme by itself affords inadequate protection for the consumer, for if the producer is permitted to resubmit rejected lots (possibly under new serial numbers, so that the consumer does not realize that these are lots which already have been rejected) even very bad lots will eventually be accepted. Also suppose that the producer

178

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

is producing uniform lots with fraction defective (), and disposing of the rejected lots through some other channel. Then, depending on () and the operating characteristic of the sampling scheme, a fraction A«()) of the lots will be accepted and 1 - A(O) of the lots will be rejected. However, the lots rejected will be identical in quality to the lots accepted. Table 3.12

Sampling inspection

00 01

decision

c

p

Hypothesis testing

Acceptable quality level (AQL) Value of the parameter under Ho Lot tolerance per cent defective Value of the parameter under HI (LTPD) Accept lot

Accept null hypothesis that 0=00

Acceptance number

Upper bound to region of acceptance Producer's risk = Pr{lot being Level of significance = Pr{Type 1 error} = Pr{Ho:O = 00 being rejectedlO = °o} rejectedlO = 0o} Consumer's risk = Pr{lot being Pr{Type II error} = Pr{Ho:O = acceptedlO = 0l}. 00 being acceptedlO = 0l} Operating characteristic = prob- 1 - power function = 1 - probability of acceptance as funcability of rejection of Ho as function of 0 tion of 0

To guard against these possibilities, the producer may agree with the consumer that any lot rejected shall be inspected 100 per cent, item by item, and all items found defective replaced by good items. He may also agree to replace defective items found in the samples from lots which are accepted. Suppose that the producer submits K lots, each lot of N items, and that the fraction defective in these lots is (). The expected number of lots accepted is KA«()). Assuming N» n, so that we can neglect the effect of replacement of defective items found in the samples, the expected number of defective items in these accepted lots is N()KA«()). The expected number of lots rejected is K[l - A«())], but these rejected lots are "rectified" and made 100 per cent good, so that (if the inspection is really 100 per cent efficient) they will contain 0 defectives. The proportion of defectives in the K lots finally accepted, the so-called average outgoing quality (AOQ), is thus the total number of defectives

SECT.

3.17

179

RECTIFYING INSPECTION

in the lots accepted divided by the total number of items accepted: AOQ = NOKA(O) = OA(O). NK

(17.1)

From the operating characteristic curve we can readily compute the AOQ for any O. For example, for 0 = 0.04, from Figure 3.3 we see that A(0.04) = 0.82, so the AOQ = 0.04 X 0.82 = 0.0328. By allowing 0 to 0.04

AOQL

Fraction defective, 8

Figure 3.4

range over appropriate values, we can plot the AOQ as a function of 0, obtaining Figure 3.4. It will be seen that the AOQ has a maximum, close to 0.0328, known as the average outgoing quality limit (AOQL), which represents the worst average quality the consumer will find himself accepting under this rectifying scheme. Also, if 0 = 0, then there are no defective items submitted and so no defective items are accepted, and if 0 is large then almost all the lots are inspected and made 100 per cent good, so the material finally delivered has a very low proportion of defectives. Specifying nand c determines a sampling plan and its operating characteristic. For each plan we can compute the AOQ curve from the operating characteristic curve and hence determine the AOQL. In an extensive tabulation of plans, for example the Dodge-Romig tables [18], it will be found several pairs of values of(n, c) will give effectively identical AOQL's. If the AOQL is all that the consumer cares about, then he may be prepared to give the producer some leeway in the choice of sampling plan in order to minimize the total amount of sampling. For example, the Dodge-Romig tables [18] give for large lots the following combinations of nand c for an AOQL of 3.0 per cent: n c

28

65

125

215

385

690

3

6

10

17

29

ISO

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

The operating characteristic curves for these plans are, of course, widely different. Suppose that the producer is submitting lots with a process average fraction defective (). Then for the ith plan the fraction of lots accepted is A;((); for these lots a sample of size nt is inspected. A fraction 1 - A;(() of the lots is rejected and submitted to 100 per cent inspection; for these lots a "sample" of size N (i.e., the entire lot) is inspected. Thus the expected number of items inspected per lot, say ai' is (17.2)

For any specified (), at can be determined, and it will be found that for one of the plans above at is a minimum. Appendix 6 of [IS] shows that, e.g., if () < 0.0006, the plan with the smallest a i is that with n = 2S, C = 1; but if () > 0.0241, then the plan with the smallest at is that with n = 690, C = 29. 3.1S. Double Sampling It will usually be considered desirable to minimize as far as possible the amount of inspection. If the sample of size n is broken up into a first sample of size 111 with Xl defectives, and a second sample of size n2 with X 2 defectives, nl + n2 = n, if the lot is outstandingly good or outstandingly bad it may be possible to come to the decision to accept or reject solely on the basis of the first sample of size nl' For lots of intermediate quality the second sample will be necessary for forming a decision. The DodgeRomig tables [1S] are for rectifying inspection and are set up as follows:

accept the lot, and if Xl > C2 , reject the lot. C 2 , take the second sample. If Xl + X 2 :::;; C2 accept the X2 C2 , rejectthe lot (which implies 100 per cent inspection and rectification). (a) If Xl (b) If Cl

:::;; Cl ,

< Xl :::;; lot, and if Xl + >

The Dodge-Romig tables [IS] contain solutions for nl' n 2, Cl , and C2 classified by the corresponding AOQL. These double sampling plans will typically decrease the total amount of inspection by at least 10 per cent, and in certain circumstances the reduction may be near 50 per cent. However, they are more troublesome to administer than a simple single sampling plan, and the increased efficiency may not be worth while. The obvious extension of the idea of double sampling is sequential sampling (Wald [19]), in which items are taken one at a time and the decision to accept or reject the lot is made at each stage, on the basis of the cumulative record.

SECT.

3.20

181

COMPARISON OF POISSON-DISTRIBUTED OBSERVATIONS

3.19. The Addition Theorem for the Poisson Distribution Suppose that Xl' ... , Xk are independently distributed according to the poisson frequency function with parameters ~l'

•••

'~k' Then

k

X =

1

Xi

i=l

k

has the Poisson distribution with parameter 1 ~i' To prove this it will be sufficient to consider the case of k = 2. i=l The random variables Xl' X 2 take the values 0, 1,2, .... We want the probability that their sum, Xl + X 2 , is equal to r. The probability that Xl has the value sand X 2 the value l' - s is, since Xl' X 2 are independent,

e

~r-.

s!

(1' - s)!

Pr{XI = s} Pr{X2 = I' - s} = e-h ....! e-sz

.

2

(19.1)

All values of s from 0 to r will give X = Xl + X 2 = r, so the probability that X = l' is given by summing (19.1) over all permitted values of s, namely 0 to 1': r

~

•=0

/:. /:r-. l>1l>2

-h -h

£.,e

e

s! (I' - s)!

r

=e

-<Sl+SZ)~

£.,

/:str-s

l>1l>2

s=os! (1' - s)!

.

To evaluate the summation in (19.2), consider the expansion of (~l by the binomial theorem [quoted in (1.9.4)]:

(~l + ~2Y =

±(I')~:~~-s ± ~t~~-s .

s=o

s

Hence the summation in (19.2) is (~l

= I'!

s=o

s! (I' - s)!

(19.2)

+ ~2Y (19.3)

+ ~2Y/I'!, and

Pr{X = I'} = e-
+ ~2Y , I'!

(19.4)

which is a Poisson frequency function with parameter ~l + ~2' Thus the theorem is proved for the case of k = 2, and so it is obviously true for any k. 3.20. The Comparison of Two Poisson-Distributed Observations Suppose we observe that in a certain yardage carpeting A has Xl flaws and carpeting B has X 2 flaws. We wish to test the null hypothesis that the mean number of flaws per yard in the two populations of carpeting is the same, i.e., HO:~1 = ~2 = ~,say. Under the null hypothesis Xl and X 2 will both be approximately normally distributed independently with expectation ~ and variance ~, so Xl - X 2 will be approximately normally

182 BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

distributed with expectation zero and variance U. We can estimate

3 2~

as Xl + X 2 • Thus under the null hypothesis (Xl - X 2)/.J Xl + X 2 will be approximately a unit normal deviate. The approximation will be improved by introducing corrections for continuity analogous to those used in the two-sample binomial problem, (6.11). For a test against the alternative hypothesis ~l > ~2 the null hypothesis is rejected if (Xl - 1/2) - (X2 .J Xl + X 2

+ 1/2) > lI l - a •

(20.1)

An exact test can be constructed as follows. We identify EI with getting Xl equal to Xl and X 2 equal to X 2 , and E2 with getting Xl + X 2 equal to Xl + X 2 • Then, under the null hypothesis, Pr{EI n E 2 }

= Pr{XI = Xl> X 2 = X 2 and Xl + X 2 = Xl + x 2 } = Pr{XI = Xl' X 2 = X 2 } = Pr{XI = Xl} Pr{X2 = X 2 } (20.2)

and, using (19.4), Pr{ E 2 }

=

Pr{ Xl

• (U)"'l+o:2

+ X 2 = Xl + X 2 } = e-2~ --'---''-'--(Xl + x 2)!

(20.3)

Then by the formula for conditional probability, (1.4.3), ~~n~ { = Xl' X = IXl + X = Xl + } = -'----''----~ Pr{E2}

Pr Xl

X2

2

=

~"'1+"'2( Xl

+ X2)! (U)"'1+:r 2XI ! X 2 !

X2

2

=

(Xl

+ X2) (!\",1 Xl

2)

(1 _!)("'1+"'2)-"'1 2

(20.4)

which is the (Xl + l)th term in the binomial expansion for n = Xl + X 2, () = 1/2. Equation (20.4) gives the probability that Xl = Xl' given that Xl + X 2 = Xl + X 2 , and as in the Fisher exact test we also want the probabilities for all values of Xl more extreme than the observed one, Xl' Thus if Xl > X 2 , we want the probabilities given by (20.4) for Xl = Xl + I, Xl + 2, ... , Xl + X 2, the last being the maximum possible value for Xl since it corresponds with X 2 = O. The probability we want, the sum of these probabilities, is then Pr{XI ~ xllXI

+ X 2 = Xl + x 2}

"'1+"'2(XI + X2) (1).( =I - 1 - -1)"'1+"'2-.

.="'1

."

2

2

(20.5)

SECT.

3.21

183

COMPARISON OF POISSON-DISTRIBUTED OBSERVATIONS

We use (5.1) for the sum of the upper tail of a binomial with 0 = 1/2 so 0/(1 - 0) = 1 : Pr {X 1

X2 = > _ Xl Ix1 +

Xl

+ X2} =

Pr{F

< Xl + X2 Xl-

= pr{F < X 2 :

Xl

I},

+ I}

(20.6)

where the aegrees of freedom for Fare 2xl , 2(X2 + 1). Our P value for a one-sided test against the alternative ~l > ~2' is determined by

1» < X2X: I} = P

(20.7)

+ 1»

1

(20.8)

+ 1), 2xl ) = ~ . X2 + 1

(20.9)

pr{F(2Xl,2(X2 + and, since Pr{F < Fp}

= P,

F P(2Xl' 2(X2

= X2 + Xl

or

Fl_P(2(X2

To revert to our initial example, suppose that the number of flaws in carpeting A is Xl = 9 and in carpeting B is X 2 = 2. Then

Fl_P(2(2

+ 1),2

X

9)

= _9_ 2+1

or Fl _1'(6, 18) = 3. From Table IV, FO•95(6, 18) = 2.66 and FO•975(6, 18) = 3.22, so 1 - P ~ 0.965, and P,-...J 0.035. The normal approximation, (20.1), gives
I1j9-+(~ + 1/2)J = <1>(1.81) = (Ul_P)

so 1 - P = 0.965 and P = 0.035.

3.21. The Comparison of Two Poisson-Distributed Observations with the Parameters in a Certain Hypothetical Ratio Suppose that the probability of a breakdown on machine 1 in a time interval M is Al 6.t and the probability of a breakdown on machine 2 in a time interval6.t is A2 6.t. Then the number of breakdowns of machine i in a time Ti has, by (12.9), a Poisson distribution with parameter AiTi == ~i' say. If Al = A2, then ~l/Tl = ~2/T2' so ~1/~2 = T l /T2. Thus a test of the null hypothesis Al = A2 is equivalent to a test of the null hypothesis

184

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

~1/~2

=

=

~2

CHAP.

3

T l /T2 • The exact test of the previous section for the null hypothesis can be extended to this new case. As before we identify El with getting Xl equal to Xl and X 2 equal to X2, and E2 with getting Xl + X 2 equal to Xl + X 2 • Then under the null hypothesis ~l

Pr{El ('\ E 2}

= Pr{Xl = Xl' X 2 = X2

and Xl

+ X 2 = Xl + x 2}

=~~=~~=~=~~=~~~=~ = e-h

~<1)1

_1

e-S2

Xl!

~<1)2

~<1)1~<1)2

~

= e-(S1+S2)_1_2_

X2!

Xl! X2!

•

(21.1)

The sum of observations from two Poisson distributions has a Poisson distribution with parameter equal to the sum of the parameters of the two distributions, (19.4), so Xl + X 2 has a Poisson distribution with parameter ~l + ~2' and Pr{E 2}

= Pr{XI + X 2 = Xl + x 2} = e-(S1+S2) (~I + ~2y1H2 . (Xl + x 2)!

(21.2)

Then by the formula for conditional probability, (1.4.3),

= (Xl

~l

+ X2) ( ~l

Xl

~l

)<1)1(1 _

+

~2

~l

+

)<1)1+<1)2-<1)1,

(21.3)

~2

which is equal to the probability of Xl successes in Xl + X 2 binomial trials with () = ~l/(~l + ~2)' We assume that the observed value of Xl exceeds its expectation under the null hypothesis, i.e., that xl/TI > x 2/T2 • We therefore need to sum the binomial series: Pr{XI ~ xllxl = <1)1'2 (Xl ·=<1)1

+ 'II

+

X2

=

Xl

+

~l

X2) ( ~l

+

x2}

~l

)·(1 _ ~2

~l

+

)<1)1+<1)2-~

(21.4)

~2

We use (5.1) for the sum of the upper tail of a binomial, where () ~l/(~l + ~2)' so

=

(21.5)

185

EXERCISES 3B

Thus Pr {X I

' Xl + > _ Xl Ix I + X 2 = Xl + X 2} = Pr {F <

X2 -

Xl

+ 1 ~l} ~2

Xl

=

pr{F < X 2 + 1 ~}, Xl

+ 1).

where the degrees of freedom for Fare 2xl , 2(X2 probability is P. Then X2 + 1 ~ Xl

and

~2

= F p(2xl> 2(X2 +

1»

=

X2

(21.6)

~2

Suppose that this

+ 1 TI T2

(21.7)

Xl

-Xl- -~2 = FI-l' ( 2( X 2 + 1) ,2xI ) = -Xl- -T2 . ~+1~

(21.8)

~+1~

3.22. An Application to Vaccine Testing Suppose that we are testing lots of vaccine containing y live particles per liter and we test a sample of Q milliliters (m!) drawn from the wellstirr~d lot. Then on the average the sample will contain yQ/IOOO live particles: call this quantity~. Suppose that the test of the sample will find a live particle if it is present, and that if one or more live particles are found then the lot is rejected. The probability of rejection will be Pr{ X

> O}, =

1 - Pr{ X

= O} =

1-

e-~

!:

1"'=0 =

1-

e-~.

Suppose that y = 5 particles per liter and Q = 60 ml; then ~ = 0.3, and the probability of rejection is (1 - e- 0 •3) = 0.259. Thus 25.9 per cent of such lots will be rejected. Suppose that we wish to determine the size of sample that will accept only a fraction P of lots containing 5 live particles per liter; i.e., we require Pr{x = O} = p. But Pr{x = O} = e- g, so ~ = -log. p. Suppose that we require p = 0.05; then loge P= loge 5 - loge 100 = 1.60944 4.60517 = -2.99563, from a table of natural logarithms, so ~ = 2.99563. Then Q = ~/y = 2.99563/5 = 0.599 liters = 599 ml. EXERCISES 3B 3B.1. Interpret in words the conditions (8.11). 3B.2. Show that for the hypergeometric distribution (8.10) (a) E[x] = n: '

(b) Vex] = n:

(1 - ~) (1 _~ : : ~) .

186

BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

3B.3. You are dealt a hand of 13 cards from a well-shuffled bridge deck of 52 cards. What is the probability of receiving (a) exactly zero aces, (b) exactly two aces? 3B.4. Find an expression for the variance of the estimator lIN = X/11M discussed in Section 3.9. 3B.S. Fertilizer was stored in drums of two types. After a certain period it was observed that of 57 of the first type, 7 had split seams, and of 63 of the second type, I had split seams. Calculate the one-sided P value for the null hypothesis that the two types of drums do not differ in their liability to splitting (a) with the Fisher exact test, (b) with the normal approximation. 3B.6. A small car-hire company has two cars which it rents out by the day. Suppose that the number of demands for a car on each day is distributed as a Poisson distribution with mean 1.5. (a) On what proportion of days is neither car required? (b) On what proportion of days is the demand in excess of the company's capacity? (Note that e-l. S = 0.223.) 3B.7. Suppose that a system involves 11 components in series, each of which must be operative for the system to function. Suppose that the probability of the ith component failing in the time interval (I, I + dt), given that it has not already failed, is Ai dl. What is the expected value of the length of time to failure of the system?

----@---a-- ... [email protected]. (a) Suppose that a system requires a certain function to be performed at a certain stage, and that the component responsible for performing this function has probability of failure in the time interval (I, I + dl) of A dl. What is the expected value of the length of time before failure?

(b) Suppose that the component is duplicated, as in the sketch. Both components are operating, and the system functions if one or the other or both the components are functioning but fails if both components have failed. What is the expected value of the length of time before failure of the system? (c) Do the same as in (b), but with 11 components in parallel, instead of two. If I is the time to failure of the system, show that E[T] =

~ [(~)}

-

(;)

~ + ... + (_I)n-l (:)~J'

Feller [6], Volume 1, p. 63, gives the identity

! _ (11) ! + ... + (_l)n-l (11) ! (11) 1122 1111

= 1

EXERCISES

3B

187

Prove this using Feller's hint of integrating the identity n-l

!

(1 - t)v = [l - (1 - t)n]t-1 •

v=O

It follows that E[T]

= ~1 ( 1 + 21 + 31 + ... + ii1) .

38.9. (a) Do the same as in (3B.8b), but the system starts with C1 only in operation, and C1 has the property that when it fails it switches C2 on. (b) As in (a), but with a total of n components functioning on a standby and takeover basis. Show that if t is the time to failure of the system the density function of t is _

),k

PT{t} - (k _ I)! t

k-l-At

e

,

k

E[T]

38.10.

=1'

(a) Show that

i (n)x ( mx ) = (m z+ n) .

:>:=0

Z -

(b) Given that X is binomially distributed with parameters n, 9, and that Y is binomially distributed with parameters m, 9, show thatZ = X + Yis binomially distributed with parameters (n + m, 9).

38.11. A company installed two compressors at the same time. These compressors are used continuously. By the end of a year one of them has had 13 breakdowns and the other 3. Can you say that the first compressor is significantly inferior to tqe second? Make both exact and approximate tests. Discuss the assumptions underlying your test. 38.12. Suppose that the company in (3B.11) above is going to expand its plant and acquire more compressors of the second type. They would like to know within what limits the average number of breakdowns per compressor per year will lie, with 95 per cent confidence. Find these limits. 38.13. W. Allen Wallis [20] found that for the 96-year period 1837-1932 there were 59 years in which no vacancies occurred in the U.S. Supreme Court, 27 years with 1 vacancy, 9 years with 2 vacancies, 1 year with 3 vacancies, and 0 years with more than 3 vacancies. (a) Calculate the expected numbers of 0, 1, 2, 3, and more than 3, vacancies on the assumption that the number of vacancies has a Poisson distribution. (If tables of natural logarithms or the exponential function are not readily available, note that if z = eY , then y = log. z = 2.3026 loglo z). (b) What is the probability that a President will serve out his (first) four-year term without being presented with the opportunity to make any appointments? 38.14. A careful inspection shows 2 flaws in 1200 feet of Brand A magnetic tape and 10 flaws in 1800 feet of Brand B magnetic tape. Assuming that the probabilities of flaws in any length dx are ~.4 dx and ~B dx, obtain exact bounds for the P value of the null hypothesis that ~.4 = ~B' 38.15. You decide to replace the three 12AX7 tubes in the preamplifier of your hi-fi system. You purchase four such tubes from a retailer who has

188 BINOMIAL, HYPERGEOMETRIC, AND POISSON DISTRIBUTIONS

CHAP.

3

twelve of the tubes on his shelf, five of which are defective. What is the probability that you receive at least three good tubes, i.e., three or four good tubes. Reduce your answer to a simple numerical fraction. 38.16. A consumer agrees to purchase large lots of items provided that samples of size n are to be taken from each lot and the lot will be accepted if the number of defectives x ~ c. The consumer requires that if the fraction defective in a lot is 4.5 per cent then that lot has a probability of 0.90 or more of being rejected. The producer requires that if the fraction defective in a lot is 2.25 per cent then that lot has a probability of 0.10 or less of being rejected. (a) What are the smallest c and n with the desired properties ? (b) Suppose that the producer agrees to institute rectifying inspection. What is the worst possible average fraction defective that the consumer could find himself accepting? 38.17. "(The committee) came up with 12 deaths from pulmonary embolism among 1.0 million Enovid users in 1962, compared with 203 deaths among the general population of 25.6 million women of child-bearing age, or a death rate of (7.9) per million," Wall Street JOIl/'lla/, September 18, 1963. Test the null hypothesis that the death rates among Enovid users and the general population are equal.

REFERENCES 1. National Bureau of Standards, Table of the Billomial Probability Distributioll, Applied Mathematics Series 6 (1950). 2. Romig, Harry G., 50-100 Billomial Tables. New York: John Wiley and Sons, 1953. 3. Tables of the Cumulative Billomial Probabilities. Ordnance Corps Pamphlet ORDP 20-1. U.S. Government Printing Office, 1952. 4. Staff of the Computation Laboratory, Tables of the Cumulative Billomial Probability Distributioll. Cambridge: Harvard University Press, 1955. 5. Uspensky, J. V., Illtroductioll to Mathematical Probability. New York: McGrawHill Book Co., 1937. 6. Feller, William, All Introductioll to Probability Theol)1 alld Its Applicatiolls, Volume 1. 2nd ed. New York: John Wiley and Sons, 1957. 7. Hald, A., Statistical Theory with Ellgilleerillg Applicatiolls. New York: John Wiley and Sons, 1952. 8. Mosteller, Frederick, "Some Statistical Problems in Measuring the Subjective Response to Drugs," Biometrics, 8 (1952), 220-26. 9. Lieberman, Gerald J., and Donald B. Owen, Tables of the Hypergeometric Probability Distributioll. Stanford: Stanford University Press, 1961. 10. Chapman, Douglas G., "Some Properties of the Hypergeometric Distribution with Application to Zoological Sample Censuses," Ulliversity of Califomia Publicatiolls ill Statistics, 1 (1951), 131-60. 11. Fisher, R. A., The Desigll of Experimellts. 7th ed. Edinburgh: Oliver and Boyd, 1960. 12. Mainland, Donald, "Statistical Methods in Medical Research: I: Qualitative Statistics (Enumeration Data)," Calladian Joumal of Research, E, 26 (1948), 1-166. 13. Finney, D. J., "The Fisher-Yates Test of Significance in 2 x 2 Contingency Tables," Biometrika, 35 (1948), 145-56.

REFERENCES

189

14. Latscha, R., "Tests of Significance in a 2 x 2 Contingency Table: Extension of Finney's Table," Biometrika, 40 (1953), 74-86. 15. Tocher, K. D., "Extension of the Neyman-Pearson Theory of Tests to Discontinuous Variates," Biometrika, 37 (1950), 130-144. 16. Clarke, R. D., "An Application of the Poisson Distribution," Journal of the I1Istitute of Actuaries, 72 (1946) p. 481. 17. Molina, E. c., Poiss01l's Exp01le1ltial Bi1l0mial Limit. New York: Van Nostrand, 1942. 18. Dodge, Harold F., and Romig, Harry G., Sampli1lg I1Ispecti01l Tables. 2nd ed. New York: John Wiley and Sons, 1959. 19. Wald, A., Seque1ltial A1Ialysis. New York: John Wiley and Sons, 1947. 20. Wallis, W. Allen, "The Poisson Distribution and the Supreme Court," Journal of the America1l Statistical Associati01l, 31 (1936), 376-80.

CHAPTER 4

An Introduction to Queuing Theory

4.1. Introduction

An important branch of probability theory deals with the behavior of queues. Customers waiting in a barber shop, airplanes waiting their turn for a runway at an airport, looms needing the attention of an operator in a spinning mill, are all examples of queues. Queues are classified according to: (a) The time distribution of arrivals. Usually this is considered to be a Poisson process as discussed in Section 3.12. (b) The distribution of service times. The assumption that this is exponential leads to the easiest mathematics, and in some instances, e.g., duration of calls from a telephone which is not a pay phone, reasonably realistic, but in other cases other distributions, which may lead to intractable mathematics, may be appropriate. (c) The queue discipline. The simplest assumption is "first come, first served," i.e., the queue waits in line and the head of the line is the next to receive service. Sometimes, however, the service is at random. For example, the operator servicing looms can pick a loom at random from the group needing service rather than pick the one which has been waiting longest. (d) The number of service channels. An airport with only one usable runway has only one channel; a barber shop with k barbers has k channels.

Queuing theory considers primarily such questions as the distribution of the length of the queue, the distribution of waiting times, and the number of customers lost because when they arrive they see that the queue is too long so they go away. Historically, some of the earliest studies of these problems were in the context of a manual telephone exchange receiving calls from subscribers. 190

SECT.

4.2

SINGLE-CHANNEL, INFINITE, POISSON ARRIVAL

191

A. K. Erlang first published on this topic in 1909 and his most important paper appeared in 1917 (reprinted by Brockmeyer et al. [1]). The field was brought to the attention of modern statisticians by D. G. Kendall [2] in 1951; doubtless his interest was stimulated by the ubiquity of queues in all aspects of British existence in that grey and depressing era. For a monograph with many applications (and a bibliography of 910 items) see Saaty [3]. Other monographs are by Morse [4] and Takacs [5].

4.2. Single-Channel, Infinite, Poisson Arrival, Exponential Service Queues In this section we consider the simplest case, in which the arrivals are Poisson with parameter A, i.e., the probability of a new customer arriving in the interval (t, t + tJ,.t) is A tJ,.t, and there is a single server with service time having an exponential distribution with parameter ft. In other words, the probability of arriving in a time interval of length t is given by the Poisson frequency function with parameter At, (3.12.9). The expected value for a variable with a Poisson distribution is equal to the parameter, (3.11.3). Thus the average number arriving per time t equals At, and the average number arriving per unit time equals A. Also, from (1.15.12), the expected value of an exponential distribution written as p{t} = fte-I't is E[t] = l/ft. Thus the average service time is l/ft, so the average number which could be'served per unit time is ft. Hence Nft is the ratio of arrival rate to potential service rate. In order to be able to reach a state of equilibrium it is necessary that A < ft, since otherwise the queue would grow indefinitely. We assume that there are no restrictions on the possible length of the queue, and that the queue discipline is "first come, first served." A customer is regarded as being a member of the queue until his waiting for service is completed; i.e., when receiving service he is still a member of the queue. Let n be the number of customers waiting in the queue at time t and PII(t) be the probability that at time t the queue contains n customers. For n > 0, and small tJ,.t, we have

Pr{n customers in queue at t + M} = Pr{n customers in line at time t} Pr{O customers arrive in (t, t + tJ,.t)} Pr{O customers discharged in (t, t + tJ"t)}

+ Pr{n + I customers in line at time t} Pr{O customers arrive in Ct, t + M)} Pr{1 customer discharged in (t, t + tJ,.t)} + Pr{n - 1 customers in line at time t} Pr{1 customer arrives in (t, t + tJ,.t)} Pr{O customers discharged in (t, t + tJ,.t)} + Pr{n customers in line at time t} Pr{1 customer arrives in (t, t + tJ,.t)} Pr{1 customer discharged in Ct, t + M)}. (2.1)

192

AN INTRODUCTION TO QUEUING THEORY

CHAP.

4

It is assumed, of course, that the probabilities are independent, e.g., that the probability of an arrival is independent of the probability of a customer being accepted. Thus (2.1) can be written as

PII(t

+ 6.t) = f""oJ

+ P,,+1(t)(1 - A 6.t)ft 6.t + p,,(t)A 6.tft 6.t A 6.t - ft 6.t) + PII+1(t)ft I:l.t,

p,,(t)(1 - A 6.t)(1 - ft 6.t)

+ p,,_I(t)A 6.t(1 p,,_I(t)A I:l.t

- ft 6.t)

+ PnCt)(1

-

(2.2) (2.3)

terms involving (6.t)2 being neglected in (2.3). Equation (2.3) can be rearranged to give PII(t

+ 6.t) 6.t

p,,(t) f""oJ

APII_I(t) - (A

+ ft)p,,(t) + ftP,,+1(t).

(2.4)

We now take the limit as 6.t tends to zero: dplI(t) - = APn_l(t) -

(A

dt

+ ft)PII(t) + ftP"+I(t).

(2.5)

We now consider in exactly the same way Pr{O customers in queue at

t

+ 6.t}, obtaining Po(t

+ 6.t) = Po(t)(1

- A6.t)

+ Pl(t)(1

- A 6.t)ft I:l.t,

(2.6) (2.7)

(2.8) We now assume that a stationary state is reached so that p,,(t) is independent of t and can be denoted by p", i.e., we assume that dp,,/dt = 0 for n = 0, 1, .... Equations (2.8) and (2.5) then become -Apo AP,,_I - (A

+ ftPl = 0

+ ft)p" + ftPn+1 =

0

when n = 0, when n > O.

(2.9) (2.10)

From (2.9),

A

From (2.10), with n

= 1, Apo - (A

PI = - Po· ft

(2.11)

+ ft)P1 + ftP2 = 0,

(2.12)

whence, solving for P2 and substituting for PI from (2.11), (2.13)

SECT.

4.2

SINGLE-CHANNEL, INFINITE, POISSON ARRIVAL

Continuing for n

193

= 2, 3, ... , it is apparent that Pn =

AY' (;JPo,

(2.14)

Since the queue must be of some length, (2.15)

From (2.14) <Xl

LPi

=

<Xl

(A)i

(2.16)

PoL - .

i=O

t-t

i=O

If we assume that Aft-t < 1, then the summation is the sum of an infinite geometric series, and so

i

(~).

i=O

-

1

(2.17)

t-t - 1 - Aft-t .

Substituting (2.17) and (2.15) in (2.16), we have Po

=

A 1 - -,

and substituting this in (2.14) gives

(2.18)

t-t

(2.19)

Table 4.1 gives the first five terms of Pn for four values of Aft-t. The probability that a queue has more than 4 members is thus 1/1024 = 0.00098 when Aft-t = 1/4 and 525.22/1024 = 0.5129 when Aft-t = 7/8. Table 4.1.

1024Pn AI/t

n

1/4

1/2

3/4

7/8

0 1 2 3 4

768 192 48 12 3 1

512 256 128 64 32 32

256 192 144 108 81 243

128 112 98 85.75 75.03 525.22

~5

The average value of the length of the queue is the simplest aspect of the distribution, though clearly not the entire story, for if two distributions

194

AN INTRODUCTION TO QUEUING THEORY

CHAP.

4

had the same expected value, we would usually prefer the one which had fewer very large queues. The expected value of the length is

(2.20) We recall the formula for the sum of an infinite geometric series with a < 1: 1+a

+ a 2 + a 3 + ... = -1- .

(2.21)

I-a

Differentiating both sides gives

+ 2a + 3a + ... = 1 (1 - a}2 Using this in (2.20) gives, for AlP, < 1, 1

(2.22)

2

E[I1] =

(1 _p,~)~ (1 - 1Alp,}2

Afp,

=

P,

.

1 - A/p,

(2.23)

As AlP, approaches 1, the expected value of the queue length tends rapidly to infinity; for AlP, = 1/2, E[I1] = 1, but for Alp, = 15/16, E[I1] = 15. A customer will be interested not only in the length of queue as measured by the number of customers in it, but also by the time waiting for service, say fw' The average waiting for service time can be derived as follows. At any random instant the probability that the queue contains 11 customers is P1I' given by (2.19). The expected service time for each of these customers is 1/p" since the service time has an exponential distribution with parameter p,. The expected total service time for these 11 customers will be nip,. The expected waiting time E[f lV ] will be E[lIO]

= L'" (probability of n customers)

x (waiting time for n customers)

11=0

= 1p1l(~) = 1 (~)"(1 11=0

P,

11=0

P,

- ~)~ = 1111(~)"(1 - 3) . P, P,

P,

11=1

P,

(2.24)

P,

Comparing the summation with (2.20), we see that E[t lV ]

= 1 E[I1] = 1 AI~ = 3_1_ P,

p,1-A/p,

p,p,-A

(2.25)

If we add on the expected value of the service time for ourselves, 1/p" we obtain for the expected value for the sum of time waiting for service

SECT.

4.3

QUEUES WITH ARBITRARY SERVICE TIME DISTRIBUTION

195

plus time for service, say E[lq],

E[tq ]

= -A - -l+l- = -1- . fl, fl, -

A

fl,

fl, -

A

(2.26)

It will be noted that Po, (2.18), is the fraction of the time there are no customers either waiting or being serviced and the service facility is being unused. It can be made close to zero by making Affl, close to 1, but then the proportion of each customer's time that is wasted waiting in the queue rises drastically. This latter may be regarded as

E[t w ] E[t q ]

A

--=-

(2.27)

fl,

Thus, for example, if Affl, is 0.9, then 10 per cent of the server's time is wasted (resting!) but of the time each customer spends in the queue on the average 90 per cent is wasted (fuming !).

4.3. Queues with Arbitrary Service Time Distribution In Section 4.2 we derived the distribution function of the length of a queue (2.19) where the distribution of service times was exponential, and from that we obtained the expected value of the length of the queue E[n] (2.23) and the expected value of the waiting time E[t w] (2.25). In this section we relax the assumption about the form of the distribution of service time and merely suppose that it has an expected value E[ts] = T and variance a2 • We will obtain formulas for E[n] and E[tw]' The general argument follows section 2 of Kendall's paper [2]. An essential concept in the argument is that at certain time instants, called by Kendall "points of regeneration," a knowledge of the state of the process at such an instant has the property that all relevant information is contained in that knowledge alone, and further knowledge of the past history is useless for predicting the future behavior of the process. For a Poisson process, all instants have this property and are thus points of regeneration. For a queuing process with Poisson input and arbitrary service time distribution, the instant at which service is completed on a customer is a point of regeneration. Also, all instants at which the queue is empty and no one is being serviced are points of regeneration. Consider an instant at which customer Ci leaves. Let n be the size of the queue at this instant (n does not include the departing customer). Since on occasion the departing customer may leave the line empty, it is possible for n to equal zero. Let the next customer CHI to be served have service time Is (ts is a random variable with the service time distribution)

196

AN INTRODUCTION TO QUEUING THEORY

CHAP.

4

and during this time suppose that r new customers arrive. Then, conditionally on the value of 18 , r is a Poisson variable with parameter 18A and probability function analogous to (3.12.9). Using (3.11.3)

E[r] = 18A,

(3.1)

E[r(r - 1)] = E[r2] - E[r] = (t8A)2,

(3.2)

and from (3.11.6) whence it follows that (3.3)

Equations (3.1) and (3.3) are conditional on the particular value of f" and if we take expectations over time we obtain

E[r] = E[t8A] = AT

(3.4)

and (3.5) Now the variance of the service time distribution was defined as 0'2, so (3.6) whence (3.7) and substituting in (3.5),

E[r2] = A2(0'2

+ T2) + AT.

(3.8)

Now let n' be the length of the queue which customer CHI leaves behind him. We assume that statistical equilibrium will exist and therefore the marginal distributions of nand n' will be identical and in particular will have the same means and the same mean squares. We assume therefore that E[n] = E[n'], (3.9) 2 E[n ] = E[n'2]. (3.10) We now demonstrate that nand n' are related by the equation n'

=

max (n - 1,0)

+ r.

(3.11)

There are two cases to consider. First, when customer Ci leaves the queue , length is zero; i.e., n = O. Then we wait for the next customer CHI to arrive and eventually he does. He receives immediate service lasting a time Is: in this time r further customers arrive and wait in line, so that in this case (3.12) n' = r. Second, when customer Ci leaves the queue length is n > O. Then the next in line moves into service immediately, leaving n - 1 waiting in line.

SECT.

4.3

197

QUEUES WITH ARBITRARY SERVICE TIME DISTRIBUTION

Customer CHI takes time Is for service, in which time r customers arrive and add themselves to the line, so when CHI leaves the queue length is

n'

=n-

I

+ r.

(3.13)

It is easy to check that (3.12) and (3.13) can be written as (3.11). We now define a variable d as a function of n, 15(n), with the properties

d

=

15(n)

=

1

=0

if n if 11

= 0,

(3.14) (3.15)

> o.

We can then write (3.11) as (3.16) = n - 1 + d + r. It is easy to check that when n = 0, d = 1, and (3.16) gives n' = r corresponding to (3.12), and that when n > 0, d = 0, and (3.16) gives n' =

n'

n - I + r corresponding to (3.13). We note that in consequence of the definitions (3.14) and (3.15) (3.17) d 2 = d, (3.18) 11(1 - d) = n, as can be readily checked by considering the cases n = 0, n > O. We now use the assumption (3.9) that E[n] = E[n'] and take expectations of (3.16) to get E[d] = 1 - E[r] = 1 - AT, (3.19) using (3.4) for E[r]. We now square (3.16), n' = n - (1 - d) + r, to get n'2

=

n2 + 1 - 2d

+ d2 + r2 -

2n(1 - d)

+ 2nr -

2(1 - d)r, (3.20)

and substitute from (3.17) for d2 and from (3.18) for n(1 - d), to get n'2 - n2 = 2n(r - 1)

+ (r -

1)2

+ d(2r -

1).

We now take expectations and use the assumption (3.10) that E[n2] to get 2E[n(r - 1)] + E[(r - 1)2] + E[d(2r - 1)] = O.

(3.21)

= E[n'2] (3.22)

An essential part of the argument is that r is independent of nand d (d, of course, is a function of n): this permits us to write (3.22) as 2E[n]E[r - 1]

+ E[(r -

1)2]

+ E[d]E[2r -

1]

= O.

(3.23)

Solving for E[n], this gives E[n]

=

=

E[(,. - 1)2] + E[d]E[2r - 1] 2E[1 - r] E[,.2] - 2E[,.] + 1 + E[d]{2E[,.] - I}

2(1 - E[r])

(3.24)

198

AN INTRODUCTION TO QUEUING THEORY

CHAP.

4

Substituting from (3.8) for E[r 2 ], from (3.4) for E[r], and from (3.19) for E[d] gives E[n]

12 2

= AT + II T

+ 12

2

(3.25)

II (]

2(1 - AT)

This equation expresses the average length of the queue in terms of the arrival rate A and the service time mean T and variance (]2. Let f,. be the time a customer waits in the queue for service: f w does not include his service time Is. Thus tw + fs is the total time from arrival to discharge, and the average total time is E[tw

+ Is] =

E[tw]

+ T.

(3.26)

Since the arrival rate is A, the average number of arrivals in this time is A(E[tw] + T). But this must equal the average queue length immediately following his discharge, E[n], given by (3.25); thus A(E[t,.]

+ T) =

AT

+

A2T2

+ A2(]2

2(1 - AT)

,

(3.27)

so (3.28)

This shows that if we consider all service time distributions with the same average service time, that with (]2 a minimum (i.e., zero), i.e., a constant service time, gives the smallest average waiting time. 4.4. Single-Channel, Finite, Poisson Arrival, Exponential Service Queues The situation considered in this section is identical with that of Section 4.2 except for the difference that the maximum permissible queue length is N rather than infinity. This situation could arise, e.g., in a downtown

gasoline station for which the available space will only accommodate a certain number of cars and customers arriving when the station is full are not permitted to wait on the street. For n in the interval (0, N - 1) the arguments of (2.1) to (2.14) apply unchanged in the present situation. We will consider now the special case of n = N. Similar to but not identical to (2.1), we have the equation

Pr{N customers in queue at f + ~t} = Pr{N customers in line at time I} Pr{O customers discharged in (f, t + ~f)} + Pr{N - 1 customers in line at time t} Pr{l customer arrives in (f, I + M)} Pr{O customers discharged in (I, I

+ ~f)}.

SECT.

4.4

199

SINGLE-CHANNEL, FINITE, POISSON ARRIVAL

This can be written as

PN(t

+ !:!.t) = PN(t)(1

- I-' !:!.t)

+ PN-I(t) . A!:!.t . (1

- I-' !:!.t)

+ AP!I'-1(t) !:!.t,

,...., PN(t) - I-'PN(t) !:!.t

(4.2) (4.3)

which rearranged gives

PN(t

+ !:!.t) !:!.t

PN(t)

1

= JlPN_1

( )

( )

t - I-'PN t .

(4.4)

When we take the limit of this as !:!.t -- 0, the left-hand side becomes dpN(t)/dt, and if we assume that a stationary state is reached, so that dpN(t)/dt = 0, we obtain (4.5) APN-I = I-'PN' But for n

< N, (2.14) applies, so PN-I

=

( ~A)N-IPo·

(4.6)

Thus, substituting this in (4.5), we have

PN

A

A(A)N-I

= ~ PN-I = ~

~

Po

(A)N

=

~

Therefore (2.14) is valid in the present case for

(4.7)

Po·

°: ; n ::;; N: (4.8)

Analogous to (2.15), the queue must be of some length, so

1=

N (A)i LN Pi = Po L -I-' .

i=O

(4.9)

i=O

The formula for the sum of a finite geometric series is

1

+ a + a2 + ... + aN =

1 - aN+l

1-a

,

(4.10)

so

N (A)i

i~O ~ =

1 _ (A./I-')N+l 1 - A./I-' .

(4.11)

Substituting this in (4.9) gives

Po

=

1 - A./I-' 1 _ (A./I-')N+I'

(4.12)

200

AN INTRODUCTION TO QUEUING THEORY

CHAP.

4

and substituting this in (4.8) gives Pn

1 - Affl, (A)n (Affl,)N+l ; ,

o~ n ~

=1_

N.

(4.13)

Manipulations analogous to (2.20)-(2.23) give as the average length of the queue 1 - (N + l)pN + N pN+l (4.14) E[n] = p (1 _ p)(1 _ pN+l) , where p = Affl,. The proportion of the time customers are turned away because the queue is full is equal to the proportion of the time the queue is of size N, i.e., 1- P N 1 _ pN (4.15) PN = 1 N+l P = 1 - 1 N+l • -p -p 4.5. Multichannel, Infinite, Poisson Arrival, Exponential Service Queues In our discussion (Section 4.2) of a single-channel queue, with Poisson arrivals and exponential service time distribution, we had to consider the two situations n = 0, n > O. With k service channels we have to consider the three situations n = 0, 1 ~ n < k, n ~ k. The discussion for the case n = 0 leading to (2.9) remains unchanged and we continue to have

n

= O.

(5.1)

For the case 1 ~ n < k, (2.1) applies as stated. The terms involving the probabilities of 0 or I customers being discharged need consideration. For example, if there are n customers in line at time t and the probability of anyone being discharged in (t, t + 6.t) is fl, /).t, then the probability of 0 being discharged is given by the binomial probability function

(~)(fl,/).t)O(1- fl,/).t)n-o =

(1 - fl,/).t)n = 1 - C)(fl,/).t)l

+ (;)(fl, /).t)2 r-J

Similarly, if there are n discharged is

1 - nfl, /).t.

... (5.2)

+ 1 customers in line the probability of 1 being

SECT.

4.5

201

MULTICHANNEL, INFINITE, POISSON ARRIVAL

and if there are n - 1 customers in line the probability of 0 being discharged is (5.4) and if there are n customers in line the probability of 1 being discharged is

(~)(,u~t)l(l

-

,u~t)n-l~ n,uM.

(5.5)

Substituting (5.2)-(5.5) in (2.1) we get

Pn(t

+ ~t) ~ Pn(t)(1

- A- ~t)(l - n,u ~t) + PnH(t)(1 - A- ~t)(n + P,._I(t)A- ~t[l - (n - 1),u M] + p,.(t)A- ~tn,u ~t

/">oJ

Pn(t)(l - A- M - n,u ~t) +Pn_l(t)A- ~t.

+ 1),u ~t

+ PnH(t)(n + 1),u ~t (5.6)

By the same manipulations that led from (2.3) to (2.5), this gives (A-

+ n,u)Pn = A-P"_l + (n + l),up"H'

1~ n

< k.

(5.7)

Ifn > k, (2.1) applies as stated, and in place of (5.2) we have 1 - k,u ~t, in place of (5.3) we have k,u ~t, in place of (5.4) we have 1 - k,u ~t, and in place of (5.5) we have k,u M. Substituting these expressions in (2.1) we get

Pn(t

+ ~t)

/">oJ

/">oJ

Pn(t)(l - A- ~t)(l - k,u ~t) + PnH(t)(1 - A- M)k,u ~t + Pn_l(t)A- ~t(1 - k,u ~t) + Pn(t)A- M k,u M (5.8) Pn(t)(1 - A- ~t - k,u ~t) + PnH(t)k,u ~t + Pn_l(t)J.. M. (5.9)

If n = k, the changes listed above for the case n > k for (5.2), (5.3), and (5.5) apply, but in place of the change for (5.4), namely 1 - k,u ~t, we have 1 - (Ie - 1),u ~t. Making this change in (5.8) leads to the same expression as (5.9), however. By the same manipulations that lead from (2.3) to (2.5) we obtain (A-

+ k,u)Pn = Apn-l + k,uPnH'

k

~

n.

(5.10)

From (5.1) we obtain

PI Putting n

=

A-

=-

,u

Po·

(5.11)

1 in (5.7) gives (A-

+ ,u)PI = }.po + (1 + 1),uP1+1'

(5.12)

202

AN INTRODUCTION TO QUEUING THEORY

CHAP.

4

whence (5.13)

and more generally 1 (-A)" ,Po,

P1I =

f.t

1 :::;;

n.

II

< k,

(5.14)

and this equation is also true for n = O. When we put n = kin (5.10) we can derive Pk+l

=

(

A \k+l kk f.t k ) k! Po,

(5.15)

= k + 1 we can derive

and when we put n

Pk+2 ~

and in general for n

=

(

A y+2 kk f.t k )

(5.16)

k! Po,

k,

k:::;; n.

(5.17)

<Xl

1 P1l = 1 must be satisfied, so

We need to evaluate Po. The condition

11=0

k-l(A)" 1 Po 11=0 f.t 11 !

1 - -

+ 1 (A)" - -1- Po = 1

(5.18)

<Xl

"=k f.t k! k"-k

'

and 1 k-l(A)n 1 kk - 1 - - + 1A-)n Po - 11=0 f.t n! lI=k kf.t k!' <Xl

(

(5.19)

The second term on the right-hand side can be written as kk 1 A )"=k"{ A)"- 1 A )"} - 1kf.t k! lI=k kf.t k! kf.t . CI)

(

00

,,=0

(

A'-l (

(5.20)

11=0

For the right-hand side we use the formulas for the sums of infinite and finite geometric series to obtain kk 00 ( A \" kk[ 1 1 - (Ajkf.tlJ k! n~k kf.t) = k! 1 - Ajkf.t - 1 - Ajkf.t

(Ajf.t)k - Ajkf.t) .

= k! (1

(5.21)

Substituting in (5.19) we get (A/f.tl -1 - k-l(A)" 1 - -1 + --'--'-'--"---Po - 11=0 f.t 11 ! k! (1 - Aj kf.t) •

(5.22)

SECT. 4.6

INVENTORY CONTROL

203

Substituting the value for Po implied by this equation into (5.14) and (5.17) gives the solutions for all Pn' As an application of these formulas, we have that the probability of a new random arrival having to wait in line is Pr{n ~ k}

1 Po =! (A)" - ---;;=k p, k! k 00

n=k

(A/p,)k

(5.23)

where Po is given by (5.22). 4.6. Inventory, Control

In this section we will merely indicate how the methods of queuing theory are applicable to inventory control. In Section 4.4 we discussed single channel queues of finite length N, and in Section 4.5 we discussed multichannel infinite queues. Suppose that we continued with the natural ext~nsion to multichannel finite queues, and than particularized to the case where the maximum queue length N equals the number of channels k. Now consider a retailer oflarge, single, standard items, whose customers will purchase an item if he can deliver it from stock but otherwise will go away and look for it at a competitor's store. This retailer is assumed to place a reorder every time he makes a sale. This situation is the exact analogy of the queuing situation discussed in the previous paragraph. We have the relationships Number of channels = k } N _ k M' . f" Maximum length of queue = N aXlmum sIze 0 Inventory = N Idle channel Item in stock Arrival of customer and start of service Item purchased and order for replacement sent immediately End of service Delivery of ordered item Service time Time for reordered item to be delivered All channels busy No item in stock It is apparent that the solution of the queuing problem is the solution of the inventory problem. The optimum inventory will depend on the relative costs of holding an item in inventory, the profit on a sale when made, and the loss in customer good will when a customer is turned away because there are no items in stock. The reader is referred to Chapter 10 of [4] for an examination of these questions.

204

AN INTRODUCTION TO QUEUING THEORY

CHAP.

4

EXERCISES 4.1. For a single-channel infinite queue with Poisson input with parameter ,t and exponential service time with parameter It, what is the mode of the distribution of the queue length n? 4.2. For a queue of the type discussed in Section 4.2, with Poisson input with parameter ,t and exponential service time with parameter It, show that V[n]

= E[n] + (E[n])2,

where E[n] is given by (2.23). 4.3. Consider a queuing situation in which the same corporation operates both the customers and the service facility; e.g., a large cab company owning and operating both cabs and a service facility for the cabs. Suppose the cabs arrive randomly with Poisson parameter ,t and a single service channel is available for which the service times are exponential with parameter It. Suppose that the cost per hour of providing a service facility with parameter I' is MI', and that the cost in lost time, etc., for a cab in waiting and being serviced is C per hour. The constants M and C are fixed, as is the parameter A. Suppose that It can be varied, e.g., by providing a larger service staff or more equipment. What value should It be to minimize the total cost per hour to the corporation (owning both cabs and service facility)? 4.4. In a queuing situation with single server, with Poisson input with parameter,t and exponential service time with parameter /I, let nw be the number waiting in line for service (i.e., the total queue length n minus the customer being served, if any). Find the expected value of nw. 4.5. A manager operates a service facility in which customers arrive randomly with Poisson parameter ,t and queue for service. The service times are exponential with parameter It. The manager can vary It. Suppose that he wishes to choose II so that only in a fraction IX of the time, on the average, will the total queue length equal or exceed n members. Express the value of It that achieves this condition explicitly in terms of ,t, IX, and n. 4.6. For a single-channel finite queue of fixed maximum length N, prove (4.15) and show that the average length of the queue E[n] has the following limits: (a) Lim E[n] p-+O

=

p

+ p2,

(b) Lim E[n] p-+ 1

= N12,

(c) Lim E[n]

=N

- II p.

p-+ co

4.7. Show that (3.25) is consistent with (2.23), and that (3.28) is consistent with (2.25), when the appropriate values of T and (] are inserted in (3.25) and (3.28). 4.8. For a multichannel queue of the type discussed in Section 4.5, show that the average number waiting for service is

REFERENCES

205

4.9. Consider two single-channel, Poisson arrival, infinite queuing situations. In queue a, the service time is a constant, say T. In b, the service time has an exponential distribution with mean T. Obtain a simple result for the ratio of the average waiting time in queue a to the average waiting time in queue b. 4.10. Consider two single-channel, Poisson arrival, infinite queuing situations. In queue a, the service time has a rectangular distribution over the interval (0, T). In b, the service time has a constant value equal to the mean of the service time in a. Obtain a simple result for the ratio of the average waiting time in queue a to the average waiting time in queue (b). 4.11. Let PO,l be the probability of an arriving customer not having to wait for service with a single-channel infinite queue with Poisson arrivals with parameter)" and exponential service time distributed with parameter Itl. Let PO,2 be the probability of an arriving customer not having to wait for service with a two-channel infinite queue with Poisson arrivals with parameter )" and exponential service time distributed with parameter /12 = 111/2. Thus the average service time in the single-channel queue is one-half the average service time in the two channel queue. Obtain a simple expression for the ratio PO,2/PO,l in terms of Aflt2 = P2. If P2 = 1, what is P O,2/PO,l?

REFERENCES 1. Brockmeyer, E., Halmstrom, H. L., and Jensen, Arne, The Life and Works of A. K. Erlang. Transactions of the Danish Academy of Technical Sciences, Number 2. Copenhagen, 1948. 2. Kendall, David G., "Some Problems in the Theory of Queues," Journal of the Royal Statistical Society, B 13 (1951) 151-173. 3. Saaty, Thomas L., Elements of Queuing Theory. New York: McGraw-Hill Book Co., 1961. 4. Morse, Philip M., Queues, Inventories and Maintenance, New York: John Wiley and Sons, 1958. 5. Takacs, L., Introduction to the Theol}' of Queues, London: Oxford University Press, 1962.

CHAPTER 5

The Multinomial Distribution and Contingency Tables

5.1. The Multinomial Distribution So-called contingency tables are the main topic of this chapter. Their analysis requires a consideration of the multinominal distribution, which we discuss in this section. Section 5.2 treats a method of obtaining approximate percentage points in terms ofthe X2 distribution. Contingency tables themselves are discussed in Section 5.3. The multinomial distribution is a generalization of the binomial distribution in which instead of two possible outcomes A and A there are k mutually exclusive and exhaustive outcomes AI" .. ,Ak with probabilities k

01 ,

••• ,

Ok' where 10i -

=

1. We want the probability that, on n trials,

i

k

Al occurs exactly Xl times, ... , Ak occurs exactly xk times, where

The probability of getting the specific sequence ~,

:1:1

n.

i

Al ... Al ... Ak ... Ak

,

1 Xi =

(Ll)

~

times

is, since successive trials are independent, Of1 •.• O;k. The ordering of the sequence is irrelevant to our purpose, as we are merely interested in the total number of times Al occurs, etc. All sequences with a specified set of Xi have the same probability. It remains to evaluate the number of sequences with a specified set of Xi' Equation (1.7.6) gave the number of arrangements of Xl objects of type AI' x 2 objects of type A 2 , etc., as , X , ••• X k",. so n'lx . l ' 2' P r{x · b · · · , 'Xk}

n!

-

-

Xl! • , •

xk !

0"'1 . .. O"'k 1 k

is the frequency function of the multinomial distribution, 206

(1.2)

SECT.

5.2

THE

X2

207

APPROXIMATION

The multivariate distribution considered in Section 1.19 is an example ofa multinomial distribution, with ()u 1/2, ()F 1/3, ()G 1/6. In that section we calculated from first principles the entire distribution, giving the results in Table 1.17. Equation (1.2) will give the same results directly. For example,

=

Pr{2, 0, I}

=

=

=2!~: l!GJGJ(~J= ~ =~76'

(1.3)

In the case of k = 2, the multinomial distribution reduces immediately to the binomial. In general, however, it is awkward to work with, and we outline in the next section the development of an approximation to (1.2) which can then be used to obtain certain cumulative probabilities. 5.2. The X2 Approximation for the Multinomial Distribution

We use the symbol P* for the probability defined in (1.2). As in the derivation of an approximation to the binomial distribution, so here we use Stirling's approximation (3.1.2) for the factorials in (1.2). Thus

'"

(27T)-~(k-l)nll+~

This involves n to the (n n

+

k

k

i

i

II Xi("i+~) II ()fi.

(2.1)

1/2)th power. We can write

1)

k( +1 2= - 21 (k - 1) + t Xi + 2 '

(2.2)

and also k

k

i

i

II ()fi = II ()i~():"i:~).

(2.3)

Substituting in (2.1) gives p* '" (27Tn )-~(l~l) (

If k

()i

)-~

If nXi;J\"i+~. k

(

()

(2.4)

Taking logarithms, log p* ~ - 1 - (Ie - 1)1 log(27Tn)k - - L log ()i 2 2 ;

k ( + 1) -Lx; - log (~ ). 2 n()i i

(2.5)

The expected value for the ith cell is n(); = e;, say. Define the deviation

208

MULTINOMIAL DISTRIBUTION AND CONTINGENCY TABLES

CHAP.

of the observed value in the ith cell from its expected value as di = Then

- di + ei -_

1+~

Xi _

nOi and

(Xi

+ 1/2) =

log P*

f'"ooJ

-

(e i

ei

+ di + 1/2).

ei

Xi -

,

5

ei •

(2.6)

Substituting in (2.5),

11k - (k 1) log(27T11) - log 0i 2 2 i

L

- ~ (e

i

+ di +~) IOg(1 + ~J •

(2.7)

Again as in the binomial case we use the expansion 10g(1

+ y) = y -

y2 2

y3

+ - - ....

(2.8)

3

For the first terms of this series to give a good approximation we need Iyl «1. Here we are going to substitute y = dile i. We therefore need to check on the magnitude of Idileil. From (1.18.21) we have, when X has the distribution N(~, 0'2), Pr{lx - ~I

> kO'} = 2cl>(-k).

(2.9)

We can regard Xi as a binomial variable which will be approximately normally distributed with expectation nO i = ei and variance nOi(1 - Oi)' Hence Pr{lxi - eil > k..} nO i (1 - 0i)} = 2cl>( -k). (2.10) Choose for k the value ";nO;/(1 - 0;). Then we have pr{ldi l >

J

nOi ..}nOi(1 - Oi)}

1 - 0i

= 2cl>('-'

J

nOi ),

1 - 0i

(2.11)

or, since ..}nOi/(1 - 0i)..}nO i(1 - 0i) = nO i = ei, Pr{ldil

> ei} = 2cl>(-

J

nOi ).

1 - 0i

(2.12)

For moderate values and beyond of ei = nOi' say 5 or greater,..} nOil(l - 0;) will be greater than ..}5, and hence cl>( -..}nO;/(1 - 0;» will be very small. Hence Pr{1 d;le i I > I} will be very small, and we will be justified in using the expansion (2.8). We note here that d i will be of the order of its standard deviation";nO i (1 - 0;), or approximately";;;. We now examine

SECT.

5.2

THE

X2

209

APPROXIMATION

the second summation term in (2.7), using (2.8) to expand log(1

k (ei + di + -1) [do 1(d~o )2 + -1 (d....!o)3 ~ - -

~

ei

2

I

= Lk ( d; i

1 d~ 2 ei

-....!

2 ei

(d o)4 + ...]

d~ d~ + -1 -i - ... + ....! -

-

k

k

ej

j ):

-1 ....! 4 ei

3 ei

3 ei

+ dde

1 d~ -i 2 ei

1 d~ -i 4 ei

1 do + ... + -....! -

-

2 ei

+ . .. ) . (2.13)

We note that

k

k

.Li di = .L (Xi - ei ) = .L Xi - .L ei = O. i i i

terms in powers of ei , recalling that zero-order terms are

Examining the

_

d is of order .Je

we see that the

i,

i

o

(2.14) and the terms of order

1/.J;; are (2.15)

For moderately large ei , say greater than 5, all terms except those of zero order may be neglected. Hence, substituting back in (2.7) and taking antilogarithms, we get p*

,-..J

(27Tn)-l-1(k~1) (

d U 0i)-l-1 exp [1 - - ~....l k

•

k

2

2 • ei

]

.

(2.16)

Here all the terms preceding the exponential part, involving 7T, n, k, and the Oi' are constants or parameters. Thus P* is, to this approximation, k

determined by statistic:

..L dNe i ,

and historically the symbol X2 was given to this

i

(2.17) For a test of significance based on this statistic, we need to be able to calculate the probability of obtaining a value of the statistic as large as or larger than the observed value. We can write (2.17) in the form

(2.18)

210

MULTINOMIAL DISTRIBUTION AND CONTINGENCY TABLES

CHAP.

5

If we consider a particular cell, say the ith, and regard ourselves as in a binomial situation, then Xi - ei '"'-' .In(J;(1 - (Ji)

N(O, 1).

(2.19)

Hence our "X2," as defined by (2.17), is a weighted sum, the weights being 1 - (Ji' of squares of approximate unit normal deviates. These approxik

mate unit normal deviates are not quite independent, since L (Xi

-

ei )

= O.

i

Thus our "X2," because of this restriction, and on account of the weights, is not a true X2(k) as defined in (i .27.1). However, it may be shown that it is approximately a true X2(k - I). As an example, suppose that the lost-time accidents reported for a certain period for three shifts are 1, 7, and 7. We wish to test the null hypothesis that (Jl = (J2 = (J3' i.e., that (Ji = 1/3. There are a total of 11 = 15 accidents; so the expected number for each shift is e i = n(J; = 15 X 1/3 = 5, and the corresponding values of di = Xi - e i for the three shifts are 1 - 5 = -4, 7 - 5 = 2, and 7 - 5 = 2. The test statistic, with degrees of freedom k - 1 = 2, is

X2

=

i d~e = (_4)2 + 22 + 22 = 4.80. 5 5 5 i

(2.20)

i

The 0.90 and 0.95 points of X2 with 2 degrees offreedom are 4.61 and 5.99. The observed value of 4.80 being intermediate between these two points, our P value is approximately 1 - 0.91 = 0.09. As another example, consider the data of Table 3.1l. Under the null hypothesis that the observations are Poisson distributed, we have a sample of size 576 which should be multinomially distributed with (Ji' given by the column headed p{x}. The columns headed IIp{X} give the expected numbers e;; the last two columns are d; = Xi - e i and d;Je i • Since the X2 approximation for the multinomial distribution is not very satisfactory for e i < 5, the cells for X ~ 4 are combined into a single cell. Summing the entries in the last column gives X2 = 1.03. As regards the degrees of freedom for this X2, here we have five classes; so k = 5, and we have constrained :E di to be zero, and we have also estimated the parameter; of the Poisson distribution which we fitted to the data. The degrees of freedom are therefore 5 - 1 - I = 3. Table III of the appendix gives for three degrees of freedom the 0.10 and 0.25 points of the X2 distribution as 0.58 and 1.21; so 1 - P c::= 0.20 and p,....., 0.8. The data of Table 3.11 are thus very well fitted by a Poisson distribution.

5.3

SECT.

211

CONTINGENCY TABLES

5.3. Contingency Tables Suppose we have a random sample ofn objects, cross-classified according to two attributes A j , Bj" Let the probability of an object having the attributes Ai. B; be Bj ;, and the number observed in the sample be no:

k

'Ill

k

m

i

;

j

;

L L nij = n, L L B = 1. j;

Let the marginal row probabilities be

0;. and the marginal row sums be 1lj. as in Tables 5.1 and 5.2. By (1.19.1) and (1.19.2), k

B.; =

LB

(3.1)

j ;.

j

Table 5.1

Table 5.2

Aj

Ai

°.1

n. 1

0.;

n.m

11..

If the true probabilities B;; are known, the agreement between the observed and hypothetical distribution can be tested by the statistic (2.17) developed in the previous section, k '" ( B )2 2 _ '" '" ll i ; - II ji X -kk ' I1B;;

j= km -1.

j,

(3.2)

However, usually in practice the Bij have to be estimated from the data. We are usually interested in testing the hypothesis that the row and column classifications are independent, i.e., that

Pr{A B,} = Pr{A Pr{B,}, j,

i.e.,

j}

Bij

=

Bj.B. i .

(3.3) (3.4)

We now find the maximum likelihood estimators of Bj • and B.,. We first find a suitable expression for the likelihood function. Assuming

212

MULTINOMIAL DISTRIBUTION AND CONTINGENCY TABLES

CHAP.

5

that we have a multinomial distribution of the form (1.2), if we take a sample of size 1 then the xi} are all zero except for one of them, which has the value 1. Thus the X;;! equal either O! = 1 or I! = 1. Thus the frequency function is n. 0"'" .. ,O"'km = rr rr O"!I; . ,... . ,11 ,

X 11 •

10

10m

X km •

'"

•• .=1 3=1

(3.5)

.;.

Assume now that we take a sequence of n samples, each of size 1, and let nij be the sum of the X;; summed over this sequence of n observations. Thus no is equal to the number of occurrences in the ijth cell. The likelihood function (2.3.1) is n

L = p{ Xl} ... p{ Xn} =

'm

k

k

m

rr rr rr Ofj' = rr rr OFF.

(3.6)

i;

i;

Under the null hypothesis of independence of the row and column classifications, (3.4), we have

Il1](Oi. o.;ti' = (IlI}Oi~ii)(ilIlO~iI).

L=

(3.7)

Now we can write for the first parenthesis,

rr rr O?,iI = rr (O nII O!'i2 ... O?,lm) = rr 10

'"

10

t.

10

t.

t.

t.

i i i

on

O~; t.

1I1i =

i

rr O!'i. 10

t.

,

(3.8)

i

and analogously for the second parenthesis. Thus (3.9) 10

Now since

L 0i. =

1 we can write Ok. as 10-1

Ok. = 1 - LOi .. Substituting in (3.9), we get L=

(3.10)

i=l

(1 _~;;10. )1I k·rrk-1 o.ni·rr'" 0",; £., ..

•.

;=1

i=l

.; ,

(3.11)

;=1

and, taking logarithms, log L =

n.10g( 1 - :~:Oi.) + :~:ni.10g 0i. + ~t; log 0.;. k

(3.12)

To find the value of 0i. which maximizes this, we differentiate with respect to 0i. and equate to zero:

olog L

1

OOi.

1 _ £., '" 0.t.

- - = nd-1)

10-1

;=1

+ l1 i . ~1 = O. v •.

(3.13)

SECT.

5.3

213

CONTINGENCY TABLES

Using (3.10), this gives (3.14) so (3.15) k

But

1 fJi . =

1, so fJk./n k. = lin. Substituting in (3.14),

i=l

fJ.,. = n.t. (fJk.) = Ilk.

(3.16)

11;. • 11

Similar arguments must apply to the estimation of e.;; so fJ. j = njn. The maximum likelihood estimators of the eij are thus

fJ .. = Ill. n.; " n n,

(3.17)

and these can be inserted in (3.2) to give k m ( l = 11 Ilii -

i;

/

lli.Il.; 11

lI i .lI. j

)2

(3.18)

ln

However, this X2 does not have the degrees of freedom of the X2 in (3.2), namely km - I, since we have estimated a number of parameters from the data. We have estimated k parameters ei ., but, in view of the restrick

tion that 1

ei . =

1, only k - I of these are independent. Likewise m - 1

i

degrees of freedom are taken in estimation of the freedom for the X2 in (3.l8) are thus (km -

e.

j •

The degrees of

1) - (k - 1) - (m - 1) = (k - I)(m - 1).

(3.l9)

As an illustration of a contingency table, Abrahamson et al. [1] present data on the bacterial count of three types of pastry (Table 5.3). We ask whether the distribution of class of bacterial count varies according to type of pastry. We estimate the expected number for each cell as Ili . n.iln. For i = 1, j = 1, for example, this is 175 x 220/368 = 104.620. The calculations are given in detail in Table 5.4. Summing all the entries in the third part of this table, k m

11 d:ileij = i

11.389,

i

and this will be distributed as X2 with degrees offreedom (3 - 1)(3 - 1) = 4 under the null hypothesis. The 0.975 point of X2( 4) is 11.1; so P < 0.025, and therefore we reject the null hypothesis that the distribution of bacterial

214

MULTINOMIAL DISTRIBUTION AND CONTINGENCY TABLES

CHAP.

5

count is independent of the type of pastry. In other words, the pastries do differ in their distributions of bacterial counts. This is all that the X2 test as applied to a contingency table will do for us. It does not tell us in what specific way or ways the pastries differ. To form an opinion on that question we need to compare visually the table of

Type of pastry Eclairs Napoleons Coconut custard pies Totals

Table 5.3 Low Medium 92 53 75 220

37 15 19 71

High

Totals

46 19 12 77

175 87 106 368

Table 5.4 Expectations eo 104.620 52.011 63.370

33.764 16.785 20.451

36.617 18.204 22.179

Observed minus expectation do -12.620 0.989 11.630

1.5223 0.0188 2.1344

3.236 -1.785 -1.451

0.3101 0.1898 0.1029

9.383 0.796 -10.179

2.4044 0.0348 4.6716

expectations with the table of observations and see if it is obvious in what way the discrepancy between expectation and observation is arising. This has to be done. on a common-sense basis. If we attempt to formulate and test any particular hypothesis, its real significance level is distorted by the fact that we are making multiple tests suggested by the data. In the present instance, there are no obvious difficulties to the obvious interpretation of the diJ in Table 5.4: Coconut custard pies have a relatively small number of high counts and a relatively large number of low counts, napoleons are average, and eclairs have a relatively large

SECT.

5.4

THE TWO X TWO TABLE

215

number of high counts and a small number oflow counts. In other words, stay away from eclairs. For a very extensive examination of contingency tables with particular emphasis on measures of association that are relevant to particular purposes, see Goodman and Kruskal [2-4]. For a general review of the X2 test see Cochran [5, 6]. The examination of three-dimensional contingency tables proves surprisingly difficult: for a recent review see Goodman [7]. 5.4. The Two x Two Table The 2 X 2 contingency table is a special, but frequently occurring, case of the k X m contingency table. We suppose that in an infinite population all elements are categorized as Al or A2 and simultaneously as BI or B 2. For example, Al and A2 could be fair hair or otherwise and BI and B2 could be blue eyes or otherwise. We suppose that the fraction of the population classified as AiB, is ()o. We take a random sample of size n from this population and obtain the results in Table 5.5. Table 5.5 B2

Totals

nll

nl2

nl.

n21

n22 n.2

n2. n

BI Al A2 Totals

n.l

The total number of observations n is fixed, but both the row totals and the column totals are random variables. We are interested in testing the null hypothesis (3.4), for which an appropriate statistic is (3.18), which in this case will have 1 degree of freedom. For Table 5.5, (3.18) can be written after some manipulation as (4.1)

If the rows in Table 5.5 were labeled Population 1 and Population 2 and the columns were labeled Defective and Nondefective, then Table 5.5 would be identical, apart from the difference in sampling procedure, with Table 3.1. For Table 5.5 the null hypothesis is that ()ij = ()i.()., for all i andj, and in particular ()n = ()1.e.1 and ()21 = ()2.().1, so that, under

216

MULTINOMIAL DISTRIBUTION AND CONTINGENCY TABLES

CHAP.

5

the null hypothesis of independence, 0 °0 = ° = 0· 11

1.

21

.1

(4.2)

2.

From the equation for conditional probability, (1.4.5), we can write

I

Pr{defective item from population i} _ Pr{defective and from population i} Pr{item from population i}

(4.3)

Identifying the right-hand side of (4.3) with i = 1,2, with the first and last terms in (4.2), we have equivalent to the null hypothesis of independence the relationship

I

Pr{defective item from population I} = Pr{defective item from population 2}. (4.4)

I

Thus a test of the null hypothesis of independence 0ii = 0i.O.i is identical with a test that the proportion of defectives is the same in the two populations. We have encountered three situations giving rise to similar 2 X 2 tables, namely Tables 3.1, 3.9, and 5.5. The differences between the three situations was first emphasized by Barnard [8] and Pearson [9]. In Pearson's nomenclature, Problem I is what we discussed in Section 3.10, where both row totals and column totals were fixed, Problem II is what we discussed in Section 3.6, where one set of marginal totals (rows) was fixed and the other a random variable, and Problem III is what we discussed in this section, where both sets of marginal totals are random variables. The large sample statistics for testing the appropriate null hypotheses are (3.10.7) or (3.10.9) for Problem I, (3.6.8) or (3.6.10) for Problem II, and (4.1) for Problem III. We first note that (3.10.9) is algebraically identical with (3.6.10). Secondly, the distribution of the square root of (4.1) will be a unit normal deviate, since X2(1) = u2 • But the square root of (4.1) is identical with (3.10.9). In all three cases the correction for continuity improves the approximation. For (4.1) the correction takes the form of bringing each "0 closer to its expectation by 1/2. This is equivalent to replacing the factor (11111122 - 11211112)2 in the numerator by (4.5)

the other terms in (4.1) remaining unchanged. In this form the adjustment is widely known as "Yates' correction."

SECT.

5.5

217

LIFE TESTING

These three large sample approximations are, of course, only valid for large samples. For most purposes, the approximation is satisfactory if the minimum expectation, calculated as lli,I1,;/I1, regarding the 2 X 2 table as being Problem III whether it is or not, exceeds some number such as 5. There is some reason to suppose that this is rather conservative and that it would be satisfactory to use 3.5 as the minimum expectation. If the minimum expectation is too small to allow the large sample approximation to be used, if we are dealing with a Problem I we can use the Fisher exact test of Section 3.1 0 as an exact solution. If we are dealing with a Problem II or III, it was shown by Tocher [10] and Sverdrup [11] that the Fisher exact test is an exact solution to these other situations. An exposition of their arguments is beyond the level of this book. See also Lehmann [12], Sections 4.4 and 4.5. 5.5. Life Testing Suppose that 11 electron tubes are placed on life test and that we assume that their life has an exponential distribution p{t} = ()e- 6t • Tubes will fail at varying times, but there is an appreciable probability that we will have to wait a very long time for the longest surviving tube to fail. The

I I

o

n-r

I I

Figure 5.1

problem we discuss in this section is the derivation of the maximum likelihood estimator of () from the observations t l , • •. ,tr , r < 11, i.e., when the observations are terminated after the failure of the rth tube and we have no knowledge of what the lives of the remaining 11 - r tubes would be other than that they are greater than tr • We imagine the t axis divided into intervals as in Figure 5.1. The numbers above the axis are the numbers of tubes whose lives terminate in the corresponding intervals. The probability element P{tl' t 2, ... , tr } dt l dt2 ... dtr is equal to the probability given by the multinomial probability function (1.2), where the x's are 0, 1, 0, ... , 1, 11 - r, and the corresponding ()'s are

i

ll

o

p{t} dt,

ii1+dt1 11

p{t} dt, ... ,

itr+cltr

p{t} dt,

tr

i

oo

p{t} dt.

tr+dlr

218

THE MULTINOMIAL DISTRIBUTION

CHAP.

5

Thus

p{tl , t2, ... , t.} dt l dt2 ... dt.

=

II! O!l!"'l!(I1-r)!

((llp{t} dt)0(it 1+dl 1p {t} dt)l ...

Jo

I

(iOO

p{t} dt)n-. (5.1) tT+dlT

X

=

II! (11 -

r)!

ill+dllp{t} dt ... itr+dITp{t} dt(iOO p{t} dt)n-.. 11 IT IT+dIT

By the mean value theorem,

f/(X) dx = (b

iIili+dl~{t} dt ~ p{tJ dti = ()e-

8Ii

-

(5.2)

a)/W where a S ~ S

b, so

= 1, ... , r.

(5.3)

dti'

i

Also, using (1.12.5)

i

= e-8IT.

oo p{t} dt,..., fOOp{t} dt tT+dlT tT

(5.4)

Thus substituting (5.3) and (5.4) in (5.2), and canceling out the dt j , i = 1, ... , 1', we get

p{tl , ... , t.}

=

n! ()e- 811 (n - I')!

=

n! (n - I')!

()r

•••

()e-8IT(e-8IT)n-.

exp (-()

±

ti) exp [-(n - I')()t.]. (5.5)

i

This is also the likelihood function, so log L = log n! - log(n - I')!

•

+ I' log () -

() L ti - (n - I')()t.

(5.6)

i

d log L ~

= '0I' -

.t• ti -

(11 - I')t..

(5.7)

Equating this to zero and solving for (J gives (J=--_I'_--

•

Li ti + (n

(5.8)

- I')t.

For a further discussion of this topic, see Epstein and Sobel [13], who show that 21' (Jf() is distributed as x2(2r), which implies that (J is an unbiased estimator of () and that it has variance ()2fr which is independent of /l, and which permits tests to be made and confidence intervals to be constructed. Epstein and Sobel give a table which shows, for example, that

219

EXERCISES

the expected time to the tenth failure in a sample of twenty is 0.23 times the expected time to the tenth failure in a sample of ten.

EXERCISES 5.1. In a particular hour interval on four consecutive days the numbers of bugs of a certain type caught by a bug-catching device were 11, 25, 19, and 35. Test the null hypothesis that the expected numbers of bugs caught per hour were the same on the four days. 5.2. In intervals of 50, 70, 50, and 75 minutes on four consecutive days the numbers of bugs of a certain type caught by a bug catching device were 11, 25, 19, and 35. Test the null hypothesis that the expected numbers of bugs caught per hour were the same on the four days. 5.3. In a series of autopsies on 199 heavy smokers evidence of hypertension was found in 38.2 per cent of the cases. For moderate, light, and nonsmokers the corresponding numbers of cases were 288, 152, and 161 and the corresponding percentages 40.3, 45.5, and 50.3. Test the null hypothesis that the probability of hypertension is independent of the smoking category [Data from Wilens, Sigmund L., and Cassius M. Plais, "Cigarette Smoking and Arteriosclerosis," Science, 138 (1962), 975-977].

5.4. The number of window air conditioners in nine rows of row houses in an eastern city are as follows: Row Number of houses Number of houses with air conditioners

2

3

4

5

6

7

8

9

23

43

43

41

41

42

42

39

36

5

8

18

3

17

11

25

19

18

Test the null hypothesis that the probability that a house has an air conditioner is independent of which row it is in. 5.5. In two high-altitude balloon flights near the north magnetic pole, the numbers of positive and negative electrons in cosmic rays were counted, and further each particle was categorized by the energy range into which it fell: Energy interval Number of electrons MEV

Positive

Negative

50-100 100-300 300-1000

9 32 23

20 51 117

[Source: De Shong, James A. Jr., Roger H. Hildebrand, and Peter Meyer, "Ratio of Electrons to Positrons in the Primary Cosmic Radiation," Physical Review Letters 12 (1964), 3-6.]

220

THE MULTINOMIAL DISTRIBUTION

CHAP.

5

Test the null hypothesis that the relative proportions of positive and negative electrons are independent of the energy range.

REFERENCES 1. Abrahamson, Abraham E., Rubin Field, Leon Buchbinder, and Anna V. Catilli, "A Study of the Control of Sanitary Quality of Custard Filled Bakery Products in a Large City," Food Research, 17 (1952), 268-77. 2. Goodman, Leo A., and Williams H. Kruskal, "Measures of Association for Cross Classification," JOlll'llal of the American Statistical Association, 49 (1954), 723-64. 3. Goodman, Leo A., and William H. Kruskal, "Measures of Association for Cross Classifications. II: Further Discussion and References," JOlll'llal of the American Statistical Association, 54 (1959). 123-63. 4. Goodman, Leo A., and William H. Kruskal, "Measures of Association for Cross Classifications. III: Approximate Sampling Theory," JOllI'I/al of the American Statistical Association, 58 (1963), 310-64. 5. Cochran, William G., "The X2 Test of Goodness of Fit," Annals of Mathematical Statistics, 23 (1952), 315-45. 6. Cochran, William G., "Some methods for strengthening the common X2 tests," Biometrics, 10 (1954), 417-51. 7. Goodman, Leo A., "Simple Methods for Analyzing Three-Factor Interaction in Contingency Tables," JOllI'I/al of the American Statistical Association, 59 (1954), 319-52. 8. Barnard, G. A., "Significance Tests for 2 x 2 Tables," Biometrika, 34 (1947), 123-38. 9. Pearson, E. S., "The Choice of Statistical Tests Illustrated on the Interpretation of Data Classified in a 2 x 2 Table," Biometrika, 34 (1947), 139-67. 10. Tocher, K. D., "Extc;nsion of the Neyman-Pearson Theory of Tests to Discontinuous Variates," Biometrika, 37 (1950), 130-44. 11. Sverdrup, Erling, "Similarity, Unbiassedness, Minimaxibility, and Admissibility of Statistical Test Procedures," Skandinavisk Aktllarietidskrift, 36 (1953), 64-86. 12. Lehmann, E. L., Testing Statistical Hypotheses. New York: John Wiley and Sons, 1959. 13. Epstein, Benjamin, and Milton Sobel, "Life Testing," JOllmal of the American Statistical Association, 48 (1953), 486-502.

CHAPTER 6

Some Tests of the Hypothesis of Randomness: Con trol Charts

6.1. Introduction Data are frequently obtained serially in time or space. For example, a series of determinations of the velocity of light may be spread over weeks or months. The quality of insulation of a long length of wire may be determined at a number of points along its length. In calculating the variance of the mean as a2 jn, we are making the assumption that the observations are independent and are identically distributed. It is therefore desirable to have some method of checking on this assumption. In this chapter we will consider two such tests, the first appropriate for continuous observations with a normal distribution, and the second appropriate to a sequence of dissimilar elements of two types. This latter test is immediately adaptable to any continuous measurements by classifying them as above or below the median. The resulting test assumes continuity in the distribution but makes no assumptions about the form of the distribution, and is in fact an example of a nonparametric test, a class of tests to be discussed in Chapter 7. 6.2. The Mean Square Successive Difference Test The mean square successive difference test is a test of the null hypothesis that we have a sequence of independent observations Xl, ••. , X" from a population N(~, a2). We compute estimates of a2 in two ways. The first is the unbiased estimator (2.3.23), (2.1) 221

222

HYPOTHESIS OF RANDOMNESS:

CONTROL CHARTS

CHAP.

6

In computing this, it is usually convenient to use the identity n

n

n

.2 X~ - 2x.2 Xi + llX2 i i

.2 (Xi - X)2 = i

(2.2) The second estimator of (12 is d 2/2, where d 2 is defined as (2.3)

1

{_I

n~

n~

= 11 _ 1 ~;[X~l] +i~E[x~] - 2~;[Xi+l]E[Xi] = 2{E[x~] - (E[X;])2} = 2V[x] = 2(12.

}

(2.4)

It was proved [l] that, under the null hypothesis,

E[d:~2J

= 1,

and

(2.5) (2.6)

Thus the test statistic

.J(11 -

2)/(11 2 - 1)

(2.7)

is approximately distributed as a unit normal deviate under the null hypothesis. The exact distribution under the null hypothesis has been tabulated for 11 over the range 4 to 60 (but note that in [2] the sample estimate S2 was defined with 11 in the denominator instead of 11 - I). This tabulation shows that even for 11 as small as 10 the normal approximation (2.7) is good. The alternative to the null hypothesis is usually that consecutive observations tend to be correlated positively with their predecessors. The successive differences Xi+l - Xi therefore tend to be smaller than they would

SECT.

6.2

223

THE MEAN SQUARE SUCCESSIVE DIFFERENCE TEST

be under complete randomness, and so the expected value of d2j2 is less than 0'2. The numerator of (2.7) will tend to be negative, and the P value for the null hypothesis is obtained by putting (2.7) equal to Up. Table 6.1 gives the results of 23 determinations, ordered in time, of the density of the earth by Cavendish [3]. The right-hand column of the table is referred to in Section 6.5. Table 6.1

1 2 3 4 5 6 7 8 9 10 11

12

Xi

di

5.36 5.29 5.58 5.65 5.57 5.53 5.62 5.29 5.44 5.34 5.79 5.10

-0.07 +0.29 +0.07 -0.08 -0.04 +0.09 -0.33 +0.15 -0.10 +0.45 -0.69

Using (2.1) and (2.2),

B B A A A A A B B B A B

S2

13 14 15 16 17 18 19 20 21 22 23

Xi

di

5.27 5.39 5.42 5.47 5.63 5.34 5.46 5.30 5.75 5.68 5.85

+0.17 +0.12 +0.03 +0.05 +0.16 -0.29 +0.12 -0.16 +0.45 -0.07 +0.17

is computed as

[ (5.362 + ... + 5.852) _ (5.36 + ... + 5.85)2J S2 = 23 23 - 1 We compute d 2 from (2.3): d2

=

B B B A A B Median B A A A

[(-0.07)2 +2~" + (0.17)2J

= 0.036260.

= 0.061941.

From (2.6) we obtain

V[(d 2/2)J = S2

23 - 2 23 2 - 1

= 0.03977.

Thus our test statistic is u = (0.061941/2)/0.036260 - 1 = -0.14588 p -./0.03977 0.1944

= -0.732 '

whence P = 0.23. This is substantially greater than 0.05; so we would not reject the null hypothesis of randomness at the 5 per cent level of significance.

224

HYPOTHESIS OF RANDOMNESS:

CONTROL CHARTS

CHAP.

6

6.3. Runs of Elements of Two Types Suppose that we have m elements oftype A and n of type B, m + n = N, and these N elements are selected randomly one at a time without replacement and the sequence of A's and B's recorded. In this section we consider the distribution of the number of runs, say u, a run being a sequence of like elements. For example, if m = n = 5, and we observe any of the sequences

ABBBBBAAAA, AABBBBBAAA, AAABBBBBAA, AAAABBBBBA, or BAAAAABBBB, etc., (3.1) then U = 3. We want Pr{u = 3}. We will consider first two combinatorial problems. We will determine the number of ways in which r indistinguishable objects can be placed in n identifiable cells, firstly with no restriction on the number of objects that can be put in anyone cell (other than, of course, the obvious restriction that we cannot put in anyone cell more objects than we have, namely r), and secondly with the restriction that no cell can be empty. Consider a row of n cells with r objects to be placed in them. The n cells can be represented by the spaces between n + 1 bars; e.g.,

IAAAIIAAIII

°

represents a row of 5 cells with 3, 0, 2, 0, and objects A in the cells. Of the n + 1 bars, one must always be in the first position and one in the last, but the remaining n - 1 may be anywhere. Each arrangement of n - 1 objects of one type, namely bars, and r objects of a second type, namely A's, determines a different way of filling the n cells, and by (1.7.7) the number of such arrangements, and hence the number of ways of filling the n cells, is [(n - 1) + /']! (n - I)! r!

(3.2)

Now suppose that the number of objects exceeds the number of cells, i.e., r > n, and we impose the condition that no cell is to be empty. Now no interval between objects can have more than one bar, since if it did have say two, then these two bars would define a cell which would be empty. Of the r - 1 spaces between the r objects, we can choose n - 1 of them as places to put the bars (we have a total of n + 1 bars with two of them committed to the two end positions). The number of ways in

SECT.

6.3

225

RUNS OF ELEMENTS OF TWO TYPES

which we can make this choice is, by (1.7.7), ( 1'-1) n- 1

= (n

(1'-1)! - 1)! (I' - 1I)! .

(3.3)

This is the number of ways in which we can place I' objects in n cells, r ;;:: 11, with none of the cells empty. An alternative argument is as follows. Given I' objects, first place n of them in the n cells so that every cell contains precisely one object. This leaves r - 11 objects which can be placed in the n cells now without any restriction. Using (3.2) with r replaced by I' - 11, the number of ways is

[(11 - 1) + (I' - 1I)]! (I' - 1)! (3.4) (n - 1)! (I' - 1I)! (n - 1)! (I' - 1I)! ' as before in (3.3). We now turn to obtaining Pr{U = u} given m objects of one kind and /I objects of a second kind. From (1.7.7), the total number of arrangements

. (m m+ n) = (m n+ n) .

IS

We assume that all these arrangements

have equal probability, so Pr {U= u} = number of arrangements giving u runs . total number of arrangements

(3.5)

Now u is either odd or even. Let us consider the case where it is even, so that u = 2v where v is an integer. Then there must be v runs of objects of the first kind and v runs of objects of the second kind. Consider the runs of objects of the first kind and regard each run as a cell. By (3.3), the number of ways we can fill these v cells from m objects in such a way that no cell is empty is (: ~

n.

Similarly, there are (:

=:)

ways in

which the n objects of the second kind can be placed in v cells, no cell being empty. Finally, given a sequence A··· B· .. A .. " we can exchange every A group with the B group following it so that we have a sequence B ... A ... B ... , and this sequence will have the same number of runs as the first. Thus the number of arrangements giving u runs is

2(111 -

l)(nv-1 - 1).

v-1

(3.6)

Substituting this in (3.5) gives

2(11; ==

Pr{ U

D(: == D

= u = 2v} = --'--(,-n-+---'-n-)-111

(3.7)

226

HYPOTHESIS OF RANDOMNESS:

CONTROL CHARTS

A similar argument for the case of u odd, say equal to 211

CHAP.

6

+ 1, gives (3.8)

For the example of (3.1), m

Pr{V

~

3}

=

Pr{V

= n = 5, u = 3, and

= 2} + Pr{V = 3} 2 252

8 252

= - + - = 0.0397. (3.9)

This problem was discussed by Stevens [4] and Wald and Wolfowitz [5]. The cumulative probability Pr{V ~ u} has been tabulated for m ~ n ~ 20 by Swed and Eisenhart [6]. 6.4. An Approximation to the Distribution of the Number of Runs of Elements of Two Types The formulas for the exact probabilities of numbers of runs of two types of elements obtained in the previous section are too cumbersome to use in practice, particularly when we 'are beyond the range of Swed and Eisenhart's tables [6]. In this section we obtain the expected value and the variance of the number of runs, u, and then assuming that u is normally distributed we can obtain P values for observed u's. It is convenient to consider the number of transitions t rather than the number of runs u: A transition is defined as the point where one run ends and another begins. The number of transitions must be one less than the number of runs, as the last run ends at the end of the sequence, and this point is not counted as a transition. Hence t=u-l.

(4.1)

We want to find E[u] = E[t] + 1 and V[u] = V[t] under the null hypothesis. Denote the total number of gaps between elements where transitions could occur as N' = N - 1. Define ti as the number of transitions at the ith gap, i.e., if either A, A or B, B on the two sides of the gap, if either A, B or B, A on the two sides of the gap.

(4.2)

SECT.

6.4

DISTRIBUTION OF THE NUMBER OF RUNS OF ELEMENTS

227

:ATI

Then the total number of transitions t

= L ti' and E[ti ] is the probability i

that a pair of consecutive elements are dissimilar. We have Pr{first in a pair is an A} = _n_l- ,

(4.3)

m +n

I

Pr{second in a pair is a B first was an A} =

n

m +n-

1;

" IS an AB} = -mn P{ r a pair . m+nm+n-l

so

(4.4) (4.5)

Using these and similar results we can tabulate the probabilities of all possible types of pairs (Table 6.2), along with the corresponding values of ti and t~. Table 6.2 Type of pair

ti

t~

•

Probability

AA AB BA BB

0

0

1 1

1 1

0

0

m(m - 1)/(m + n)(m + n - 1) mn/(m + n)(m + II - 1) mn/(m + n)(m + n - 1) n(n - 1)/(m + 1I)(m + n - 1)

11 -

1)

It follows that E[t i ] = 1 X (111

11111 11)(111 +

+

211111 = (m + 11)(111 + 11 -

E[t~]

=

+1X 1)

(111

+

11111 11)(m + 11 - 1)

,

(4.6)

211111

(111

+ 11)(111 + 11 -

(4.7)

1)

We thus obtain

E[t]

= E[! tt] = i

since N'

= m + 11 -

! E[tt] = N' (m + 11)(m211111+ 11 i

1)

= ~, m + 11

(4.8)

1, and the expected number of runs is E[II] = 1 + 2m11 • 111

+ II

(4.9)

228

HYPOTHESIS OF RANDOMNESS:

CONTROL CHARTS

CHAP.

6

We now want the variance of u: V[u]

=

t ti N'

]

=

V[t]

=

Z V[ti ] + 2 i=l Z i=i+1 Z COV[ti' ti]'

V[

N'

N'-l

N'

(4.10)

i

The first term is readily evaluated:

z V[ti] = N'V[ti] = lV'

N'{E[t~] - (E[t i])2}.

(4.11)

i

The second term in (4.10) involves the COV[li' t 1 ] which can be calculated as since £[li] = £[t i ]. The terms tili are best considered in two groups: 1. The terms litHl which involve adjacent gaps, in which the pair of elements determining Ii has as its second element the first element of the second pair determining IH1 . 2. The terms lilk' where k > i + I, for which the two pairs of elements do not have an element in common; i.e., the ith and kth gaps are separated by one or more other gaps.

We need the numbers of terms of these two types. The total number of pairs Ii' Ii> with i < j, is the number of combinations that can be formed from N' items taken two at a time, i.e., ( N') 2

=

N'! 2! (N' - 2)!

=

N'(N' - 1) . 2

(4.13)

The number of pairs of type I is N - 2 = N' - I, since, if we consider a typical sequence of elements 1,2,3,4,5, then the only adjacent pairs ofl's that can be formed are 11 ,12 from elements 1,2,3; 12 , la from elements 2,3,4; la, 14 from elements 3,4,5. The number of pairs of type 2 can then be obtained as the total number of all pairs minus the number of pairs of type 1 : N'(N' - 1) _ (N' _ 1) 2

= (N'

- 1)(N' - 2) . 2

(4.14)

To compute the COV[li' Ii] from (4.12) we need £[I/I i ] for the two types of pairs. For type 1, the pairs t i , tiH are those pairs of transitions with one element in common. All possible examples are listed in Table 6.3. The only sequences giving nonzero values for litiH are ABA and BAB.

SECT.

6.4

DISTRIBUTION OF THE NUMBER OF RUNS OF ELEMENTS

229

Table 6.3

Sequences of elements

AAA AAB ABA ABB BAA BAB BBA BBB

000 010 1 1 1

1 0 0

1 0 0

111 010

000

These have probabilities

_111 . _ -n- -

111-1 n III n - 1 and --.----.:..:..:-m+n lI1+n-l lI1+n-2 lI1+n lI1+n-l lI1+n-2 (4.15) Thus - 1) E[ ti ti+1 ] -- 1 X - - - - - - -111/1(111 '-------'---(Ill + n)(111 + n - 1)(m + n - 2) + 1

X

(Ill

111/1(/1 - 1) n - 1)(111 + n - 2)

+ n)(111 +

111n (4.16) =------------(m + n)(111 + n - 1) Inserting this in (4.12) we get for the covariances of the pairs of the type ti' ti+1

Cov [ti , ti +1]

=

111/1 (111 + /1)(111 + n - 1)

-

2 (E[t;]).

(4) .17

There are N' - 1 = m + n - 2 such terms. To compute Cov[t i, td, k > i + 1, we need E[titk ] for the pairs of t's formed by the types of pairs of elements listed in Table 6.4. The only types giving nonzero values for titk are AB· .. AB, AB' .. BA, BA ... AB, and BA ... BA, all of which involve two A's and two B's: These types all have the same probability III /I /11-1 /I-I --' 111+/1 111+/1-1 lI1+n-2 lI1+n-3

(4.18)

Thus E[ titk ] -_

411111(111 - 1)(/1 - 1) (Ill + n)(111 + n - 1)(/11 + /I - 2)(111 + n - 3)

(4.19)

230

HYPOTHESIS OF RANDOMNESS:

CONTROL CHARTS

CHAP.

6

Inserting this in (4.12), we get of the covariances of pairs of the type f i' f k' k > i + I, 411111(111 - 1)(11 - 1) _ (E[ .)]2 Cov [f". tk ] = t, . (111 + 11)(111 + 11 - 1)(111 + 11 - 2)(111 + 11 - 3) . (4.20) By (4.14), there are (111 + 11 - 2)(m + II - 3)/2 such terms. Types of pairs

Ii

Ik

AA· ··AA AA···AB AA···BA AA···BB AB···AA AB···AB AB···BA AB···BB

0 0 0 0 1 1 1 1

0 1 1 0 0 1 1 0

Table 6.4 Types of pairs lilk BA···AA BA···AB BA···BA BA···BB BB···AA BB···AB BB···BA BB"'BB

0 0 0 0 0 1 1 0

Ii

Ik

lilk

1 1 1 1 0 0 0 0

0 1 1 0 0 1 1 0

0 1 1 0 0 0 0 0

We now substitute in (4.12) the variance term (4.11) and the two covariance terms (4.17) and (4.20), the two latter with the appropriate coefficients, namely, the numbers of terms of the two types: V[II]

= (111 + 11

- 1){E[t~] - (E[t i ])2}

+ 2(111 + 11 +2

1 (111 2

X -

+

x { (111

+

11)(111

2){

(111

11111

+ 1I)(m + 11 -

11 - 2)(111

+

1)

- (E[t;])2}

11 - 3)

411111(111 - 1)(11 - 1) - 1)(111 + 11 - 2)(111

+ II

+ 11

_ (E[t i ])2}. -

3)

-em

(4.21)

The sum of the coefficients of (E[t;])2 is easily shown to be + II - 1)2. Substituting for E[ti] from (4.6) and for E[t:] from (4.7), (4.21) reduces to - 111 - 11) V[1 ] _ 211111(211111 1 - .

(111 + 11)2(111 + 11 - 1) Thus, if u is an observed number of runs, the statistic II - [211111/(111

+ 11) + 1] + 11)2(111 + II

(4.22)

(4.23) ../211111(211111 - 111 - 11)/(111 - 1) is under the null hypothesis a standardized variable which for large m and 11 is approximately normal.

SECT.

6.5

231

RUNS ABOVE AND BELOW THE MEDIAN

The above statistic has been studied by Stevens [4] and Wald and Wolfowitz [5]. Wallis [7] suggested making a correction for continuity by bringing the observed u closer to its expectation by 1/2. For very small values of m and n the normal approximation will be unreliable, and the exact values of the distribution tabulated by Swed and Eisenhart [6] should be used. Usually the alternative hypothesis envisaged is one-sided: Usually we anticipate the runs of like elements to be greater in length and hence fewer in number than under the hypothesis of randomness. For example, if a row of tomato plants contains diseased plants we might expect these to occur in groups. The usual critical region for the statistic (4.23) is therefore
E[u] = 1 + 2 V[lI]

=

2

X

x 5

5

x

+5

5= 6 '

(4.24)

5 x 5 X (2 X 5 X 5 - 5 - 5) (5 + 5l(5 + 5 - 1)

= 2.222

(4.25) ,

and the approximate unit normal deviate is 1I

+ 1/2 -

E[u]

= 3 + 1/2 -

v'V[u]

6

= 1.677

v'2.222

(4.26) '

whence P = 0.047, to be compared with the exact figure 0.0397 in (3.9). Of course, as m and n increase the approximation improves. For example, for m = n = 20, Pr{U:::;; 15} equals 0.0388 by the approximation to be compared with the exact figure 0.0380 from [6]. The foregoing treatment of runs of two kinds of elements has been extended to the case ofthree kinds of elements: If the numbers of elements of the three types are a, b, and e,

E[u] = 2(ab + ae + be) a+b+e V[II]

= (a

[2(ab + ae + be)]2 + b + e)2(a + b + e -

1)

(a

+ 1,

(4.27)

2(ab + ae + be) + 6abe + b + e)(a + b + e - 1) (4.28)

6.5. Runs Above and Below the Median The test of the previous section can readily be adapted to test the randomness of a sequence of continuous observations, Xl' ..• ,xx. By classifying the observations as being above or below the sample median,

232

HYPOTHESIS OF RANDOMNESS:

CONTROL CHARTS

CHAP.

6

each observation will be labeled either A or B. If the number of observations is odd, we ignore the observation which falls on the median. The number of observations above the median m equals the number of observations below the median n; so m = n. Equations (4.9) and (4.22) become E[u]

= 1 + m,

(5.1)

V[u]

= m(m

(5.2)

- 1) . 2m -1

For the data of Table 6.1, the median is 5.46, and the observations are marked A or B accordingly in the last column. The number of runs u = 8, m = n = 11; E[u] = 1 + 11 = 12, and V[u]

= 11

x (11 - 1) 2 x 11 - 1

= 5.2381,

u P

= 8 + 1/2 -

.J5.2381

12

=

-1.529

(5.3) '

whence P = 0.063. This test is an example of a so-called nonparametric test; its only assumptions under the null hypothesis are that the observations are random and identically distributed from some continuous distribution whose form has not been specified. The mean square successive difference test, in contrast, assumed normality. Usually, if such a stronger assumption is justified, then the test making use of such an assumption is more powerful against a particular alternative than an analogous test not making use of the assumption, assuming that both tests are oriented toward having power against this alternative. In the present instance, the assumption of normality appears reasonable, and, if the null hypothesis had been appreciably false, we might have anticipated a smaller P value with the mean square successive difference test than the test for runs above and below the median. Actually the reverse occurred, though with both tests the null hypothesis was acceptable at the 5 per cent level of significance. Other studies have been made of the distributions of runs of various types. The distributions of runs above and below the median of specified lengths, of runs up and down by number and by length, etc., are available (see [8-11]).

6.6. Control Charts for the Mean and Range The statistical control chart, due to Shewhart [12, 13], is a device for keeping a check on the stability of a repetitive process. For example, we may be filling cans with lubricating oil and wish to maintain the contents

SECT.

6.6

CONTROL CHARTS FOR THE MEAN AND RANGE

233

at a stable level: If Xi is the weight of the contents of the ith can, if the Xi are independent observations from a population normally distributed with known mean ~ and variance 0'2, then if we take samples of n cans the mean weighs x will satisfy (1.18.19) with x written for X and O'/..;~ written for 0':

pr{x < ~ - ~~} + pr{x > ~ + ~~} = 2[1 -

(k)].

(6.1)

With k = 3.09, for example, 99.8 per cent of all sample means will lie in the interval ~ ± 3.090'/";;;. Should a sample mean lie outside this interval, since this will only happen twice in a thousand times under the stated assumptions, the natural conclusion is to suspect that the assump- , tions are incorrect in some respect. Usually the most likely way in which the system has deviated from the assumptions is for the mean to have shifted. So long as only the stated proportion of points lies outside the interval, the process is said to be in a state of statistical control, and, if more than that proportion of points is outside the interval, then the process is said to be out of control. The sample means are plotted on a chart serially as they are obtained, and lines known as control limits are drawn in at ~ ± kO'/";;;. Usually ~ is estimated by the mean of the previous 30 or more sample means. The standard deviation 0' can be estimated by (2.3.23), but almost invariably a quick but not fully efficient method is used, based on the ranges. The range w of a sample of n is defined as the difference between the largest and the smallest observations in the sample. For samples from a normal population E[w] = dna, where d n is a constant, tabulated in Table VI, dependent on n. The standard deviation is estimated by A

0'

lV

=dn

(6.2)

where IV is the mean of 30 or more ranges. For example, if ~ is estimated as 100, and lV = 10 for a large number of samples of 5, then c1 = 10/2.326. For 99 per cent control limits, we have 2[1 - (k)] = 0.01; so (k) = 0.995 and k = 2.576. The control limits are 100 ± 2.576

X

10~;26 =

100.00 ± 4.95.

The object of this procedure is to enable the operator to distinguish between random variation and variation in excess of what would reasonably occur at random. The operator has to accept the variation represented by the fluctuation of the sample means within the control limits as inevitable, given eand 0'. However, the occurrence of an out-of-control

234

HYPOTHESIS OF RANDOMNESS:

CONTROL CHARTS

CHAP.

6

point is a signal for him to suspect that something has changed, to find out what, and to restore the status quo. If a increases, points on the mean chart can go out of control without any change in~. If the standard deviation increases by a factor of 4, and we are using 99 per cent control limits for the means, so that in (6.1) k = 2.58, the limits would now be effectively at ±2.58(a/4)//;' = ±0.645a/.J;;; so the probability of a point falling outside these limits is 2[1 - <1>(0.645)] = 2(1 - 0.7405) = 0.519, a drastic change from the supposed 0.01. To guard against this ambiguity we can keep a control chart on the ranges. For samples of size n from a normal population with standard deviation a, the sampling distribution of the range II' is known,

Pr{1I'

< aWp } = P,

where Wp is tabulated in Table VI of the Appendix. Thus P 2 control limits for II' are given by

(6.3) -

PI

(6.4)

For example, if n = 10, and we want 95 per cent control limits, then we need P 2 = 0.975, PI = 0.025. From Table VI for n = 10, W O•975 = 4.79, W O•025 = 1.67, and the limits are (1.67a,4.79a). If a is estimated as wid" = lv/3.078, then the lower control limit is 0.542}v, and the upper control limit 1.5561V. A control chart for ranges is kept in an analogous manner to that for means. The sample ranges are plotted from left to right as they occur in time, and the control lines drawn in. If a point lies outside a control limit, this is evidence that the standard deviation of the population has changed. Control limits are not the same as specification limits. If a process is in a state of statistical control, the sample averages with rare exceptions will be fluctuating inside the control limits ~ ± ka/.J;; and the individual observations will be fluctuating with rare exceptions in the interval ~ ± ka: These limits, +ka and -ka, will only coincide with the process specification limits, if it has any, by accident. The limits for the individual observations are wider than the control limits for the sample means by the factor .J;;. To revert to our example of filling cans with lubricating oil, if ~ is equal to the claimed quantity, say ~O, then the individual observations x will be distributed around ~o with standard deviation a, and 50 per cent of all cans will have less than the claimed amount. If only 1 per cent of all

SECT.

6.7

235

CHARTS FOR POISSON DISTRIBUTED OBSERVATIONS

cans are to have less than the claimed amount, we require that

Pr{x

< ~o} = 0.01 = (~o ~ ~)

by (1.18.13); so (~o - ~)/(1 = UO•01 = -2.326, and so the required overfill ~o = 2.326(1. It may be economic to make quite an effort to reduce (1.

.~ -

6.7. Control Charts for Poisson Distributed Observations Suppose that we wish to keep a control chart on the number x of defects in square-yard samples of carpeting. If the process is in control, then x will be a Poisson distributed variable with parameter~. We assume that ~ n

can be estimated accurately by (3.11.11) as

x = L xi/n where 11 is large. i

The usual procedure is to use the normal approximation to the Poisson, (3.13.5). We assume that x is normally distributed with mean ~ and variance ~, where ~ is estimated by x, and the so-called three sigma limits are x ± 3Ji Continuity corrections are usually omitted. It may sometimes be justifiable to be more precise. Equating (3.13.3) with P gives (7.1) f= 2(xp + 1). Pr{x 2 < 2~} = 1 - P, But, since Pr{x2

< X~-p} = 2~

1 - P,

= X~-p,

j= 2(xp

+ 1).

(7.2)

This equation will determine Xp for a specified ~ and P: By changing x we change f and hence X~-p until it equals 2~. Since x has to change by integers, f will change by jumps of two units, and in general we will not be able to get an exact solution. As an example, suppose that ~ = 9 (there is no reason, of course, why ~ should be an integer). If we want 95 per cent control limits, we will want PI to be as close as possible to 0.025 from the lower side. Substituting in (7.2), we want 2 x 9 = X~-O.025. From the X2 table, X~.975 = 17.5, 19.0, and 20.5 for f = 8, 9, and 10, respectively. We cannot use an odd value of f, because that would make x fractional. We have to settle for f = 8, whence x = 3, and Pr{x ~ 3} = 1 - Pr{x 2(8) < 18} r-J 1 - 0.976 = 0.024, actually very close to the desired value of 0.025. If we used f = 10, to give x = 4, Pr{x ~ 4} = 1 - Pr{x2(1O) < 18} r-J 1 - 0.94 = 0.06, much larger than the desired figure. For the upper limit, we want Pr{x ~ x P2 }

=

1 - Pr{x ~

XP2 -

I}

=

Pr{x2(2x P2 )

< U),

to be less than or equal to 1 - P2 = 0.025. We try f = 30, when x

(7.3)

=

15,

236

HYPOTHESIS OF RANDOMNESS:

and

Pr{x2(30)

<2

CONTROL CHARTS

x 9} ~ 0.035,

CHAP.

6

(7.4)

which is too large; so, trying! = 32, we have

Pr{x2(32)

<2

x 9} ~ 0.02,

and so we use our upper control limit,

X O•98

1.96.Ji

(7.5)

= 16.

The crude limits x ± in this case, for which would be rounded outward to 3 and 15.

x=

9, are 3.12 and 14.88,

EXERCISES 6.1. A zoo has ten lions and three cages. Assuming that the lions are indistinguishable, in how many ways can the lions be placed in the cages (a) with no restriction on the number of lions per cage, (b) with no cage empty, (c) with no lion to be alone by himself and no cage empty? In each case give the appropriate general formula for the case of r lions and n cages before particularizing to the given numerical case. 6.2. In a street of row houses X represents the presence of an air conditioner and 0 the absence of an air conditioner: OOXOXOOOOOOOOOOOOXXXXXXOOOOOOOOOXOOOXOOXOO There are 31 O's and 11 X's. Does the observed number of runs differ significantly from the number expected under the null hypothesis of randomness? 6.3. A physician running a clinical trial on a weight-reducing product, C, included in his experiment two other substances, A and B, intended for use as placebos. The initial weights of the patients assigned to the three treatments are given below. Rank the patients by order of increasing weight. If the patients had been allocated to the three treatments at random, then the sequence of A's. B's, and C's would be a random sequence. Consider whether the observed number of runs is consistent with the hypothesis that randomization had been employed. A: 147,183.5,150,167,180.25,216.5,127.5,222.5,132,163, 221.5, 203.5 B: 180, 161.5, 157, 155, 146.5, 131.5, 136.25, 159.5, 162.5,225.5, 159.

C: 175, 168, 144.5, 125,250.5, 123.75, 124.25, 117, 118, 162.25. 6.4. The number of hurricanes on the East Coast for the years 1930 through 1954 were reported as follows: 2,2,6,10,5,5,5,2,4,2,4,4,2,5,6,4,2,4,5,8,11,8,6,5,6. Test the null hypothesis that this a random sequence (a) with the mean square successive difference test, assuming that the square root of the numbers observed is normally distributed (why?); (b) with the test for runs of two kinds of elements (Let years with 4 or fewer hurricanes be "good" and with 5 or more be "bad"); (c) with the test for runs of three kinds of elements (Let years with 4 or fewer hurricanes be "good," with 5 be "medium," and with 6 or more be "bad"). Report one-sided P values.

237

EXERCISES

6.5. Construct a control chart for the data in the preceding example, using 99 per cent control limits. 6.6. We have a machine automatically filling tubes of toothpaste. We take a large number of samples of five tubes, find the weights of the individual tubes, and take the range of weights in each sample of 5. The mean weight is 16 grams, and the mean range is 0.7 gram. (a) Suppose we want control limits that will contain 98 per cent of all sample means, on the average, when the system is under control. What should the control limits be? (b) What are 98 per cent control limits for the ranges of the samples of five ? (c) If the label claims that the tube contains 16 grams, what percentage of the customers are receiving less than they have a right to expect? (d) What percentage of the customers get less than 15.8 grams? (e) To give 99.9 per cent of the customers at least what the label claims, where should the process average be? 6.7. A mining company finds a body of ore, approximately rectangular. At equally spaced intervals along the length they bore through the body, and the cores are analyzed for a certain mineral. In this sample the minimum is 0.005, the median 0.05, and the maximum 1.32. It is apparent that this distribution is exceedingly skew, since the distance from the minimum to the median is 0.045 whereas the distance from the median to the maximum is 1.27. If we use logarithms, or more conveniently 100 times the (logarithms + 3), say x, the minimum, median, and maximum are 70, 170, and 312. It appears that the logarithm of the percentage is much more symmetrically distributed than the percentage itself. Test the null hypothesis that this sequence of observations is a random sequence with (a) the mean-square successive difference test, (b) the test for number of runs above and below the median. Bore

x

Bore

x

Bore

x

1 2 3 4 5

170 178 263 185 311

16 17 18 19 20

148 130 100 70 100

31 32 33 34 35

130 178 190 291 241

6 7 8 9 10

312 190 178 160 308

21 22 23 24 25

160 148 130 300 160

36 37 38 39 40

130 70 215 223 195

11 12 13 14 15

223 130 195 170 160

26 27 28 29 30

148 130 100 70 100

41 42 43 44 45 46

160 100 130 252 240 274

238

HYPOTHESIS OF RANDOMNESS:

CONTROL CHARTS

CHAP.

6

What practical inferences might you draw if the hypothesis of randomness was (c) acceptable? (d) Rejected at some reasonable significance level?

6.8. Construct a control chart for means and ranges for the data of the preceding exercise. Use samples of size 5 made up by grouping consecutive observations, and discard the 46th bore. Give 99 per cent control limits. 6.9. A process for purification of a fine chemical involved passing it in solution through a column containing a resin on which the chemical was adsorbed. A specified volume of solution constituted a batch, after which the flow was switched to another column. The figures in Table A give the concentration escaping from the outgoing end of the column at the conclusion of each batch operation. (a) Construct a control chart for means and ranges of samples of 4 with 99 per cent control limits. Note that a chart of totals is equivalent for all practical

1

5

9

Table A 13 17

0.5 3.0 3.8 1.1

2.9 3.0 1.7 3.1

1.0 2.5 6.7 1.9

1.9 4.6 0.8 3.7

3.5 0.7 2.1 3.0

1.2 2.0 1.4 1.6

1.7 2.2 1.1 2.7

2.8 2.9 1.9 3.2

2.3 1.7 4.0 0.8

4.2 3.0 2.2 2.1

Totals Ranges

8.4 3.3

10.7 1.4

12.1 5.7

11.0 3.8

9.3 2.8

6.2 0.8

7.7 1.6

10.8 1.3

8.8 3.2

11.5 2.1

Batches

41

45

49

53

57

61

65

69

73

77

2.3 1.4 2.2 1.5

1.8 4.0 2.6 1.2

1.3 5.2 0.9 0.8

4.2 2.4 1.0 1.9

0.5 1.8 0.6 0.8

0.8 2.7 0.9 1.4

1.2 2.9 1.0 1.0

2.3 1.5 0.3 0.3

1.2 0.7 3.4 1.2

0.7 0.5 1.3 0.3

Totals Ranges

7.4 0.9

9.6 .2.8

8.2 4.4

9.4 3.2

3.7 1.3

5.8 1.9

6.1 1.9

4.4 2.0

6.5 2.7

2.8 1.0

Batches

81

85

89

93

97

101

105

109

113

117

1.3 0.1 2.1 2.0

0.5 0.9 0.8 0.4

1.1 0.5 0.5 0.7

1.2 1.2 0.8 0.6

0.9 1.3 0.6 1.1

1.6 0.6 1.0 0.9

0.3 0.7 0.3 0.8

0.8 0.4 0.9 1.3

0.7 0.8 3.2 0.5

0.6 1.4 0.4 1.1

5.5 2.0

2.6 0.5

2.8 0.6

3.8 0.6

3.9 0.7

4.1 1.0

2.1 0.5

3.4 0.9

5.2 2.7

3.5 1.0

Batches

Totals Ranges

21

25

29

33

37

239

REFERENCES

Table B 5

9

13

17

21

25

29

33

37

0.70 1.48 1.58 1.04

1.46 1.48 1.23 1.49

1.00 1.40 1.83 1.28

1.28 1.65 0.90 1.57

1.54 0.85 1.32 1.48

1.08 1.30 1.15 1.20

1.23 1.34 1.04 1.43

1.45 1.46 1.28 1.51

1.36 1.23 1.60 0.90

1.62 1.48 1.34 1.32

4.80 0.99

5.66 0.26

5.51 0.83

5.40 0.75

5.19 0.69

4.73 0.22

5.04 0.39

5.70 0.39

5.09 0.23

5.76 0.70

41

45

49

53

57

61

65

69

73

77

1.36 1.15 1.34 1.18

1.26 1.60 1.41 1.08

1.11 1.72 0.95 0.90

1.62 1.38 1.00 1.26

0.70 1.25 0.78 0.90

0'90 1.43 0.95 1.15

1.08 1.46 1.00 1.00

1.36 1.18 0.48 0.48

1.08 0.85 1.53 1.08

0.85 0.70 1.11 0.48

5.03 0.21

5.35 0.52

4.68 0.82

5.26 0.62

3.63 0.55

4.43 0.53

4.54 0.46

3.50 0.88

4.54 0.68

3.14 0.63

81

85

89

93

97

101

105

109

113

117

1.11 0.00 1.32 1.30

0.70 0.95 0.90 0.60

1.04 0.70 0.70 0.85

1.08 1.08 0.90 0.78

0.95 1.11 0.78 1.04

1.20 0.78 1.00 0.95

0.48 0.85 0.48 0.90

0.09 0.60 0.95 1.11

0.85 0.90 1.51 0.70

0.78 1.15 0.60 1.04

3.73 1.32

3.15 0.35

3.29 0.34

3.84 0.30

3.88 0.33

3.93 0.42

2.71 0.42

3.56 0.51

3.96 0.81

3.57 0.55

Batch

Totals Ranges Batch

Totals Ranges Batch

Totals Ranges

purposes to a chart for means, with the obvious adjustment to the control limits, and saves some arithmetic. Comment on the charts. (b) Comment on a graph of the ranges against the totals for each batch. (c) Table B gives the logarithms of the data of Table A multiplied by a factor of 10. Construct charts as in (a) and comment.

REFERENCES 1. von Neumann, J., R. H. Kent, H. R. Bellinson, and B. I. Hart, "The Mean Square Successive Difference," Annals of Mathematical Statistics, 12 (1941), 153-162. 2. Hart, B. 1., "Significance Levels for the Ratio of the Mean Square Successive Difference to the Variance," Annals of Mathematical Statistics, 13 (1942),445-47.

240

HYPOTHESIS OF RANDOMNESS:

CONTROL CHARTS

CHAP.

6

3. Cavendish, H., "Experiments to Determine the Density of the Earth," Philosophical Transactions of the Royal Society, 88 (1798), 469-526. 4. Stevens, W. L., "Distribution of Groups in a Sequence of Alternatives," Annals of Eugenics, 9 (1939), 10-17. 5. Wald, A., and J. Wolfowitz, "On a Test Whether Two Samples are from the Same Population," Annals of Mathematical Statistics, 11 (1940), 147-62. 6. Swed, F. S., and C. Eisenhart, "Tables for Testing Randomness of Grouping in a Sequence of Alternatives," Annals of Mathematical Statistics, 14 (1943), 66-87. 7. Wallis, W. Allen, "Rough-and-Ready Statistical Tests," Industrial Quality Control, 8 (1952), 35-40. 8. Mosteller, Frederick, "Note on an Application of Runs to Quality Control Charts," Annals of Mathematical Statistics, 12 (1941), 228-32. 9. Olmstead, P. S., "Distribution of Sample Arrangements for Runs Up and Down," Annals of Mathematical Statistics, 17 (1946), 24-33. 10. Wolfowitz, J., "Asymptotic Distribution of Runs Up and Down," Annals of Mathematical Statistics, 15 (1944), 163-72. 11. Moore, G. H., and W. Allen Wallis, "Time Series Significance Tests Based on Signs of Differences," Jou/'llal of the American Statistical Association, 38 (1943), 153-64. 12. Shewhart, W. A., Economic Control of Quality of Manufactured Product. Princeton, N.J.: D. Van Nostrand Co., 1931. 13. Shewhart, W. A., Statistical Method from the Viewpoint of Quality Control. Washington, D.C.: Department of Agriculture, 1939.

CHAPTER 7

SaIne Nonparan1etric Tests

7.1. The Assumption of Normality A large proportion of present-day statistical techniques for dealing with continuous variables relies on the assumption that the variable has an underlying normal distribution. This assumption is not so restrictive as it may seem at first sight, for, first, we can often transform a variable, if we know the general form of its distribution, into a variable which is approximately normally distributed. Second, we can often arrange to deal with means, and the central limit theorem assures us, roughly, that if a population has a finite variance then the distribution of the sample mean approaches normality as n increases. The central limit theorem has a long history dating back to Demoivre, La Place, and Gauss; for a modern review see Cramer [1], sections 17.4 and 17.5. It is presumably on this foundation that applied statisticians have found empirically that usually there is no great need to fuss about the normality assumption. After a statistician has analyzed several quite widely differing transformations of a variable in a fair number of specific instances and found that the conclusions reached are substantially identical for all the transformations, then he ceases to worry unduly about the normality assumption in most situations. This is a convenient place to interpolate a few more remarks about the importance of the assumption of normality. The importance of the assumption depends on

1. The nature of the departure from normality 2. The statistical technique being used 3. The ultimate purpose of the analysis. For example, marked skewness can badly upset the significance level of a one-sided test but have only a small effect on a two-sided test. Again, 241

242

SOME NON PARAMETRIC TESTS

CHAP.

7

deviation from normality in the extreme tails is rather unimportant in significance testing: To think that we are rejecting a null hypothesis at the level of significance 0.0001 when in fact the true level of significance is 0.001 is rarely going to tarnish our reputation, but to have 10 vacuum tubes failing per week in a system involving 10,000 when we have predicted only 1 could be embarrassing. It is probable that deviations from normality cause fewer gross errors than two other departures from the usual assumptions: 1. Lack of constancy of variance 2. Lack of independence. Some of the effects of (1) are explored in Section 9.8 and of (2) in Section 10.5. Parenthetically, we might remark that there is some reason to believe that in some areas of research gross errors, e.g. reading the wrong gauge, reading off the wrong scale, transposing digits in writing down numbers, mistranscribing data, losing part of the sample, collecting the sample in a dirty container, making a mistake in the analytical procedure, are far commoner than ordinarily realized. In recent years techniques have been developed which make no assumption about the form of the underlying distribution except that it be continuous. These methods are known as l1ol1parametric or distributionfree methods, neither name being particularly satisfactory. The observations certainly do have a distribution, with parameters: what we are free of is assumptions about the form of that distribution. The literature of non parametric methods has grown rapidly: see I. R. Savage [2]. An advanced text is that of Fraser [3]. A general text on some ranking methods is that of Kendall [4]. 7.2. The Sign Test Consider tomato plants grown in pairs at a number of sites, one pair at each of 11 sites. Let the yields at the ith site be Xi' Yi. Then (x, y) will have a joint probability density function p{x, y}. Imagine that a supplementary fertilizer be applied to one member of each pair, and let y correspond to the yield of this member. If the fertilizer is without effect, it is reasonable to assume that the two yields Xi' Yi are identically and symmetrically distributed so that

p{a, b} = p{b, a}.

(2.1)

If the fertilizer does have an effect, we will suppose that the effect is a constant amount T. Now every Y will be greater than what it would have

SECT.

7.2

243

THE SIGN TEST

been in the absence of fertilizer by the amount T, so if we subtract T from every y we will be back in the original situation (2.1); i.e., we have

p{x, y - T} = p{y - T, x}.

(2.2)

The assumptions made imply that T is the median of y - x. To see this, we proceed as follows. Substitute IV = Y - T in (2.2), giving

p{x, IV} = p{w, x};

(2.3)

i.e., the joint density function of x, lV is symmetric about the line IV = x. The region above this line corresponds to x < lV, and hence the integral of the joint density function over this region gives Pr{x < IV}. Likewise the region below this line corresponds to IV < x, and hence the integral over this region gives Pr{w < x}. Now from the symmetry property of (2.3), these two integrals must be equal numerically, and therefore Pr{x or, substituting y -

T

< IV} =

Pr{IV

< x},

(2.4)

for w,

Pr{x


T} = Pr{y - T < x}.

(2.5)

The following two identities are always true:

Pr{y - x Pr{T


< T} = -

Pr{y - T < x},

(2.6)


(2.7)

x} = Pr{x

-

T}.

But by (2.5), the right-hand sides of (2.6) and (2.7) are equal, and hence the left-hand sides are equal, i.e.,

Pr{y - x

< T} =

Pr{T

>y -

x}.

(2.8)

In other words, T is the median of y - x = z, say. Thus a test of the null hypothesis T = is a test of the null hypothesis that the median of z is zero, and likewise, under the alternative hypothesis T > 0, the median of z will be greater than zero. To make the test, therefore, we define a variable

°

if Z; ifz;

> 0,

< 0.

(2.9)

We assume continuity in the original joint distribution p{x, y} so that the distribution of z is also continuous, and "ties," in which x; = y;, so that Zj = 0, should occur with zero probability. The Zi are independent, so we are in the binomial situation of making n independent trials in which the probability of success, namely the occurrence Z; = 1, is 1/2 on each n

trial. Thus! Z; has under the null hypothesis a binomial distribution i

244

CHAP.

SOME NONPARAMETRIC TESTS

7

with parameters n, () = 1/2. Thus the sign test, assuming that the alternative hypothesis is T > 0, when we have m positive differences in a sample of size 11 gives a P value of n Pr { ~ Zi ~ •

min, () = -I} =!n 2

"'=m

(n) (1)"' - ( 1 - -l)n-", x

2

2

1 1 n-m(n) ! (n) --! - 2 n ",=m x - 2 n ",=0 x . n

(2.10)

For moderate nand m, this can be easily computed directly, and for larger nand m we can use the normal approximation (3.2.3). The foregoing argument remains valid if the joint density function p{x, y} is a function of i; i.e., Pi{X, y} can be different for different i. As an example of the use of the sign test, Table 7.1 gives the differences A - B in potency of a series of lots of a pharmaceutical product as Table 7.1. 0.3, -1.7,

Differences between Two Analytical Methods for 16 Lots

6.3, 1.6,

3.7, -1.8,

2.8, 0.6,

5.8, 4.5,

-1.4, 1.9,

1.7, 2.4,

2.3, 6.8.

measured by two different methods A and B. The original observations were on each lot by each method: For the first lot the potencies were 89.7 and 89.4 by the methods A and B, but only the difference 0.3 is relevant. We have Zi = 1 for 13 lots and 0 for 3 lots. Using (2.10), Pr { !n Zi ~ 13ln i

1! = 16, () = 1) - = 16 2 2 =

16 13 -

(16)

",=0

X

_1_(~ + ~ + ~

65,536 O! 16!

I! 15!

2! 14!

~)

+ 3! 13!

= 0.01064,

(2.11) or, using (3.2.4),

= 13 -

u I-P

1/2 - 16 x 1/2 ../16 x (1/2)(1 - 1/2)

= 225 .,

(2.12)

whence 1 - P = 0.9878 and P = 0.0122. The sign test assumes that the underlying distributions are continuous and hence the probability of a tie occurring, giving rise to a zero difference, is zero. However, in practice all continuous measurements are only recorded to a finite number of decimal places, and ties may occur. The best procedure when this happens is to delete the ties from all consideration.

SECT.

7.2

245

THE SIGN TEST

11

The tie is ignored in computing I Zi' and the tie does not contribute to the sample size neither [5]. i Our discussion so far has proceeded under the assumption that the null hypothesis was that the median of the differences Xi - Yi = Zi was zero. If our null hypothesis is that the median of the Zi is then we can define a new variable Wi = Zi and make the foregoing sign test on the Wi' This device immediately suggests a method of constructing 1 - IX confidence intervals for ,. By trial and error we find that value for " say ~, which leads to rejection of the null hypothesis that the median of Z = ~ at the 1X/2 level, and that value of " say ~, which leads to the rejection of the null hypothesis that the median of Z = ~ at the 1X/2 level. We therefore put

'0'

'0

11

Pr {I Zi ~ i

Pr {I Zi ~ n

and

i

rln, () =

1} 11I-'(n) I

-,

(2.13)

1} 1 (n) ~ - ,

(2.14)

-

2

= --;;

2

",=0

sin, () = - = - I 2

x'

~

IX

2

8

IX

211 ",=0 x

2

and solve for rand s. Because of the discreteness of the binomial distribution we may not be able to approach very closely the desired value of 1X/2. In the present example,

and

pr{~ Zi ~ 4} = 0.0384,

(2.15)

pr{~ Zi ~ 3} = 0.0106.

(2.16)

For 95 per cent confidence limits, we would want 1X/2 = 0.025. To be conservative, we have to choose s = 3, which will give us (1 - 2 X 0.0106) X 100 = 97.88 per cent confidence. The choice of s = 4 would give us (1 - 2 X 0.0384) X 100 = 92.32 per cent confidence, which is far below our desired value of 95 per cent. From the symmetry of the binomial distribution with () = 1/2, r = 11 - S = 16 - 3 = 13. We now order the sample as in the first column of Table 7.2. We then select the maximum value of ~, ~, which gives :2: Zi = 13, and the minimum value of " ~, which gives :2: Zi = 3. Choosing' = 0.2, ~ = 4.6 gives the results in the second and third and fifth and sixth columns of Table 7.2. We could actually choose ~ as large as 0.2999 and still have only three negative signs for Z - ~, and ~ as small as 4.50001 and still only have three positive signs for zThus the 97.88 per cent confidence limits are roughly (0.29,4.51), or more accurately just under 0.3 and just over 4.5. In practice, of course, we just order the sample and count in from the bottom end to the (s + l)th observation and down from the top end likewise.

r

246

SOME NONPARAMETRIC TESTS

CHAP.

7

Table 7.2 z

z - 0.2

z - 4.6

z

z - 0.2

z - 4.6

-1.8 -1.7 -1.4 0.3 0.6 1.6 1.7 1.9

-2.0 -1.9 -1.6 0.1 0.4 1.4 1.5 1.7

-6.4 -6.3 -6.0 -4.3 -4.0 -3.0 -2.9 -2.7

2.3 2.4 2.8 3.7 4.5 5.8 6.3 6.8

2.1 2.2 2.6 3.5 4.3 5.6 6.1 6.6

-2.3 -2.2 -1.8 -0.9 -0.1 1.2 1.7 2.2

7.3. The Median Test The sign test discussed in the previous section is appropriate when we have equal numbers of observations from the two distributions and natural pairing is present. In the median test we have samples of nl observations Xi from a distribution PI{X} and nz observations Yi from a distribution pz{Y} and there is no natural pairing. We arrange the observations in a common sequence of increasing magnitude:

The null hypothesis is that the x's and y's are independent and identically distributed. The alternative hypothesis strictly is that the observations are not identically distributed. In practice, however, the (one-sided) alternative usually envisaged is that the distribution of the x's is slipped towards the right compared with the distribution of the y's, so that the x's tend to be larger than the y's. Suppose that the two samples have been taken and that each observation is categorized as lying either above or below the sample median (see the notation of Table 7.3). We can picture the situation as follows. Under Table 7.3 Below median Above median Sample 1 Sample 2 Totals

nIb

nIb

nla

n Zb

nZ a

+ nZb = nb

n 1a

+ nZa = na

Totals nl nz nl

+ nz

the null hypothesis we have a finite population of nl + nz elements of which na can be classified as "above." We take a sample of size nl. We ask for the probability that this sample contains n 1a or more "aboves."

SECT.

7.3

247

THE MEDIAN TEST

Just as in the Fisher exact test in Section 3.10, the answer is immediately given by appropriate summing of the probability function for the hypergeometric distribution. Substituting in (3.8.10) M = 11m X = ilIa' N - M = lIa + llb - Ila = Ilb' 11 - X = III - ilIa = nIb' N = III + 112,11 = 111>we have

P{IlIa}

=

( na)(llb) ilIa I1lb (Ill ~ 112)

=

Il a! . Ilb! . Ill! 112! ilIa! 112a! llIb! 1l2b! (Ill + 1l2)! '

(3.1)

similar, with the change of notation, to (3.10.2). The one-sided P value is p{nla} plus the similar probabilities for all possible integers greater than lila' For small samples the tables referred to in Section 3.10 will avoid the calculations, and for larger samples (3.10.9) can be used as an approximation. If 111 + 112 is odd, then the median of the joint sample will be actually one of the observations: Suppose that it occurs in the first sample. This observation, since it lies on the joint sample median, contributes no information on the question of whether the distribution of the first sample has its median above or below the joint sample median. It appears reasonable, therefore, to delete this observation from all consideration, excluding it also from the sample total. As an example of the application of the median test, Table 7.4 gives the lives in hundreds of hours of samples of electron tubes of two brands A and B. Incidentally, survival time data are typically skewed with a long tail to the right, and usually quite appreciably nonnormal, so a test assuming normality would be inappropriate. Table 7.4 A

B

32, 34, 35, 37, 42, 43, 47, 58, 59, 62, 69, 71, 78, 84 39,48,54,65,70,76,87,90, 111, 118, 126, 127

The median in Table 7.4 falls between 62 and 65, and so we have Table 7.5. By Fisher's exact test the two-tailed P value is 0.0472: The normal approximation, which we can consider reliable since the minimum expectation in any cell is 6, gives P = 0.0492. Table 7.5 Above median Below median Brand A Brand B Totals

10 3 13

4 9 13

Totals 14 12 26

248

SOME NONPARAMETRIC TESTS

CHAP.

7

7.4. The Mean and Variance of a Sample from a Finite Population In the section to follow this, on the Wilcoxon two-sample rank test, we need the mean and variance of a sample of size n from a finite population of N elements, the elements having attached to them numbers Xl' X 2, ••• ,XN' We wish to express them in terms of the mean and variance of the population, which we define as

(4.1)

since, for a finite population of size N, P{Xi} 0'2

=

V[x]

= E[X2]

=

lIN, and

N

- (E[X])2

= .2 X~P{Xi}

- (E[X])2

i

(4.2)

This can be written in the form

(4.3)

The set of all possible samples of size n from the population will include (Xl>

X 2, ••• , x'n-l, X.,)

(xl>

X 2 , ••• ,

X"_l' X.,+l) (4.4)

These samples are formed by picking (1.7.8) there are

(~)

II

elements at a time from N; by

ways of picking such a sample. It is axiomatic that

every possible sample is equally likely, and so the probability of any particular sample being picked is I I (~) .

7.4

SECT.

249

THE MEAN AND VARIANCE OF A SAMPLE

x;, j

Suppose that the samples in (4.4) have means The expected value of x is

= 1, 2, ... ,

(N). 11

N\-l[l = (-;; J ~ (Xl + X2 + ... + Xn) + ... + ~1 (XN-n+l + ... + XN)]

•

(4.5) Now consider the number of times any particular Xi' say Xl> will occur in (4.5). The samples containing Xl are made up by selecting 11 - 1 other elements from the available population of N - 1 elements, and this can be done in

(~= :)

ways. There will therefore be

samples containing Xl. In other words, Xl occurs The same will apply to every other Xi. Thus E[x] =

(N - I)N1 J (N)-l[l n nn-li -

Xi

(~

= -1

=:)

1N Xi =

Nt

(~= :)

times in (4.5).

~

(4.6)

by (4.1). The expected value of the sample mean is thus the population mean. The variance of x is V[x] = E[X2] - (E[X])2. (4.7) We have already found E[x]. Now consider E[X2]: (~)

E[x 2]

=

-1

(~r)

t X~p{x;} = (~) t x~.

(4.8)

But

-t(~)

x~

=

[1

~

(Xl

+ X 2 + ... + Xn)J2 + ... +

[1

~

(XN-n+1

+ ... + Xx) J2 . (4.9)

When we square each term, each

Xi

will give rise to an x~, and we have

already seen that in the similar expression (4.5) each times; so the squared part of (4.9) is -

1(N - 11) ( 2+ ... +

n2

Xl

/I -

2)

XN

•

Xi

occurs

n

(N = 11

(4.10)

250

SOME NONPARAMETRIC TESTS

CHAP.

The expression (4.9) will also give rise to product terms

Any par-

xixj•

ticular Xi and Xj will occur together in a given sequence in of the samples; so the product part of (4.9) is -2(N

2)

-

(X I X 2

+

XIXs

+ ... +

n2 n - 2 the factor 2 arising from the fact that xixj into (4.8) gives E[x2] =

= xjxi •

7

(~-=

i)

(4.11)

XN_IXN),

Substituting these back

(N)-I[l(N - 11)(X~ + ... + x~) n 2

11

11 -

+

:2(: =~)(XIX2

+ ... +

(4.12)

XN_IXN)}

We thus have E[X2] ready for substitution into (4.7) to give V[x]. Equation N

(4.7) also requires (E[X])2. In (4.6) we found E[x]) to be (lIN) L Xi; so i

(4.13) Substituting (4.12) and (4.13) in (4.7), we get

- = [(N)-l "2I(Nn -- 11) - -N1] + ... + + [( N)-l "22 (N - 2) - 22 ] + ... + /1 n-2 N (Xl2

V[x]

11

2 XN )

2

11

(X I X 2

xN_IXN)'

(4.14)

11

The coefficient of x~

(N -

+ ... + x1 can be written as

1)'

n-1

n2( : )

1

1

(N-l)!

--=----'-----''---2 2 N

n (n - I)! (N - /1)!

n!(N-1I)! N!

N-n N - l N-n = ~ = ~ 1I(N - I) =

and the coefficient of (XIX2

2(~=i) n2 ( : )

1

1

1

N

nN

N

- -2= - - -2

(1 1)

N-n N - N 2 n(N - 1) ,

+ ... + xN_IXN) is

( 4.15)

2 2 (N-2)! /1!(N-1I)! - = - - - - ' - - - ' - - - -'---~ 2 2 N n (/1 - 2)! (N - 11)! N! =

2(n - 1) 1IN(N - 1)

2 2(N - /1) N 2 = - N 2n(N - 1) .

(4.16)

SECT.

7.5

251

THE WILCOXON TWO-SAMPLE RANK TEST

Then, inserting these results in (4.14), we get - = V[x]

1 1)

2

)]

N - n [(- - 2 (x 12 +"'+XN)-2(X1X 2 2 +"'+XN-I XN n(N -1) N N N

.

(4.17) The part in brackets is identical with the expression for 0'2, (4.3), so V[x]

= 0'2 N

- n nN-1

= 0'2(1 n

(4.18)

_ n-

1).

(4.19)

N-1

Clearly, if N tends to infinity, then V[x] tends to 0'2/n, its customary form for an infinite population. 7.5. The Wilcoxon Two-Sample Rank Test The Wilcoxon two-sample rank test [6] is a test of the null hypothesis that two populations are identical, against the alternative hypothesis that they differ by a linear translation. We substitute ranks for the actual observations. As an example, Table 7.6 lists five determinations of the atomic weight of carbon from one preparation and four determinations from another preparation [8]. Ranks are allocated to the observations in order of increasing magnitude without regard to the division into two samples. Table 7.6 Preparation A Preparation B Determination

Rank

Determination

Rank

12.0072 12.0064 12.0054 12.0016 12.0077

8 7

11.9853 11.9949 11.9985 12.0061

1 2 3 6

5 4 9

Suppose that one sample is of size 11 and the other of size N - 11. The test assumes that any combination of the ranks into these two groups is equally likely. The total number of ways of grouping the ranks, given Nand

11,

is the number of ways of picking

11

elements out of N,

(~).

The test then counts how many of the possible combinations give a rank sum as extreme as or more extreme than that observed.

252

SOME NONPARAMETRIC TESTS

CHAP.

7

In Table 7.6, regarding the observations from preparation B as the sample of size 11, the rank sum is 1 + 2 + 3 + 6 = 12. The only ways we could get a rank sum as' small as or smaller than this, given 11 = 4, are 1 1

+ 2 + 3 + 4 = 10, + 2 + 3 + 6 = 12, 1 + 3 + 4 + 5 = 13

1 1

+ 2 + 3 + 5 = 11, + 2 + 4 + 5 = 12.

For example, is greater than our observed rank sum 12. The P value is equal to the ratio of the number of ways we can form a rank sum as extreme as or more extreme than that observed, namely, 4 ways, to the total possible number of ways of forming sums of 11

= 4ranks,namely, (~) =

(!) = 126ways. ThusP =

4/126

= 0.0317

for a one-sided test. The rationale of this procedure is that, if one distribution is displaced relative to the other, the low ranks will tend to fall in one sample and the high ranks in the other sample, and so the rank sums will be relatively low or high. In the case of small Nand 11, it is relatively easy to compute the P value directly as in the above example. For large samples, an approximate test is available based on the fact that the mean of the ranks of a sample is distributed around its expected value approximately normally. Let R be the rank sum and R the mean of the ranks of the sample of N

size

11.

= (lIN) L Xi

By (4.6), the expected value for R is E[R]

where

i

the Xi are the elements of the finite population, here the integers 1 to N. The sum of the first N integers is 1+ 2

+ ... +

N

=

+

N(N

1) ,

(5.1)

2 and also the sum of their squares is 12

+

22

+ ... +

N2

=

N(N

+

1)(2N 6

+

1) .

(5.2)

Using (4.1) and (5.1), E[R] =

1. . N(N + 1) =

N

2

N

+1.

The variance of R, VCR], will be given by (4.18) where stituting (5.1) and (5.2) in (4.2): (]2

= 1. . N(N + 1)(2N + 1) N

_

[1..

N(N

N

6

(5.3)

2 (]2

+ 1)J2 = 2

is given by subN2

-

1.

(5.4)

12

Thus, substituting this in (4.18),

VCR] = (N 2 - 1)/12 N 11

11

N - 1

=

(N

+ l)(N 1211

11) •

(5.5)

SECT.

7.6

253

THE ADJUSTMENT FOR TIES

We then have

R - E[R] -.J-V-[-=-R]~

R - (N

+ 1)/2

(5.6)

+ 1)

(5.7)

= ,../7.;(N:;=+:::==::l~)(N¢-=n~)/:r.=12~n 2R - n(N

= '../:=;n(;:;::;N;:::+~l)?7(N:;::::-::::::::::n);:;;
asymptotically a unit normal deviate. Kruskal and Wallis [11] have observed that the approximation is improved when the one-sided P value is found to be greater than 0.02 by introducing a correction for continuity by bringing R closer to its expectation by 1/211, or 2R closer to its expectation by 1. If the one-sided P value is less than 0.02, this correction for continuity should be omitted. In the example in Table 7.6, if we consider the results with preparation B as the sample of size 11, then 11 = 4, N = 9, R = 12, R = 12/4 = 3, and (5.6) is =

U

p

../(9

3 - (9 + 1)/2 + 1)(9 - 4)/12

X

4

= -1.96' '

(5.8)

so P = 0.025. This implies that we should use the continuity correction:

= 3 + (1/2

U

p

../(9

X

4) - (9 + 1)/2 4)/12 X 4

+ 1)(9 -

=

-1.837' '

(5.9)

so the one-sidedP = 0.033. Even though Nand 11 are small in this example, the normal approximation gives a P value close to the exact value, 0.0317, calculated earlier. A careful development of the Wilcoxon test was given by Mann and Whitney [9]. For the history of the test see Kruskal [10]. 7.6. The Adjustment for Ties in the Wilcoxon Two-Sample Rank Test If ties occur, so that two or more of the observations are identical, the best procedure is to allocate to these tied observations the mean of the ranks they would receive if they were not tied. Table 7.7 gives two series of determinations of the conversion efficiency of an ammonia oxidation plant for the production of nitric acid. If we allocate ranks to the observations in increasing order of magnitude, 90.0, 90.1, 92.1, and 93.3 receive ranks 1,2,3, and 4, respectively. We then encounter two observations at 94.7; these receive the ranks 5 and 6, so that the mean rank 5! is allocated to each of them. Later on, two observations at 95.6 both receive ranks (9 + 10)/2 = 9t. The effect of this procedure is to leave the sum of the

254

SOME NONPARAMETRIC TESTS

CHAP.

7

ranks used unchanged, and so E[R] (5.3) is unchanged. The variance is somewhat affected, however. Table 7.7

A

B

Determination

Rank

Determination

Rank

95.6 94.9 96.2 95.1 95.8 96.3

9t 7 12 8 11 13

93.3 92.1 94.7 90.1 95.6 90.0 94.7

4 3 5! 2 9! 1 5!

The variance of the ranks, see (4.2), is formed from E[X2] and (E[X])2. The latter involves the sum of the ranks, which is unchanged, and hence is unaltered. The former is the sum of the squares of the ranks, divided by N; this is affected by the substitution of mean ranks for individual ranks. Consider one group of t ties. The integers (x + 1), (x + 2), ... , (x + t) are replaced by their mean (x

+ 0 + (x + 2) + ... + (x + t)

= xt

+ t(t + 0/2 =

t

The sum of squares (x

x

t x~

+ t + 1. 2

+ ... + x~ in (4.2) is thus reduced by

t( x + t :

+ 1)2 + ... + (x + t)2 -

r

1

(6.1)

(6.2)

which, after some simplification, can be written as (t - 1)1(1

+

T

1)

12

(6.3)

= 12'

if we define T = (t - 1)1(1 + 1). The variance of the ranks is no longer given by (5.4). Now the sum of squares of the ranks is N(N

+

1)(2N

6

and this is substituted for

x~

+

1)

T

12

+ ... + x~ in the formula for 0"2 (4.2).

(6.4)

The

SECT.

7.6

255

THE ADJUSTMENT FOR TIES

sum of the ranks is unchanged at N(N Xl + ... + x",: Thus (12

+ 1)/2, and this is substituted for

= 1. [N(N + l)(2N + 1) _ N

6

= _1_

[N(N 2

_

I..] _[1..N 12

N(N

+ 1)J2 2

1) - T].

(6.5)

12N

Using (4.18), the variance of R is V[R] = N(N 2 - 1) - T. N - n . 12Nn N - 1

(6.6)

The term T = (I - l)t(t + 1) occurs for each group of ties, if there is more than one such group. In the example in Table 7.7, there are two groups of two ties; so the correction term 2 T; is ;

2; T; =

(2 - 1) . 2 . (2

+ 1) + (2 -

1) . 2 . (2

+ 1) =

12.

(6.7)

Then V[R]

= 13(132 12

x

1) - 12 . 13 - 6 = 1.3536, 13 x 6 13 - 1

(6.8)

and

R

= 9! + 7 + ... + 13 =

10.083;

6

(6.9)

so we have an approximate unit normal deviate _ 10.083 - (13 + 1)/2 _ 2 650 ..}1.3536 -.,

11 1-1' -

(6.10)

giving a one-sided P value of 0.0040. Since this is less than 0.02, we omit the correction for continuity. _We can find the level of significance by a direct enumeration as we did in the example without ties in Section 7.5. In Table 7.7 the rank sum for the smaller sample A is 60!. The various ways in which a rank sum of GO! or greater can be formed from the ranks listed in Table 7.7 are given in Table 7.8, where tied ranks are distinguished from each other by the letters a and h.

The total number of ways 6 ranks can be selected from 13 is

(13)6 =

1716. The one-sided P value is thus 6/1716 = 0.00350. The normal approximation, which gave P = 0.0040, is thus reasonably close in this instance.

256

CHAP.

SOME NONPARAMETRIC TESTS

7

Table 7.8. Ways in Which Six Ranks Can Be Selected from the Ranks of Table 7.7 Subject to Condition That Their Sum Is Greater than or Equal to 60!

Rank sum

13 12

13

13

13

13

13

12

12

12

12

12

11

11

11

11

11

11

9!a 9!b

9!a 9!b

8

7

63

62

9!a

9!b

8 7

8 7

6O!

6O!

9!a 9!b 5!b

9!a 9!b 5!a 6O!

6O!

7.7. The H Test The H test, due to Kruskal and Wallis [11], is a generalization of the two-sample Wilcoxon test to the case of k samples, k > 2. As in the Wilcoxon test, the entire set of observations, here in k groups of size II; rather than two groups as in the Wilcoxon test, are ranked, and mean ranks R; are calculated for each group. R; has an expectation (N + 1)/2 (5.3), and a variance (N + 1)(N - 11;)/1211; (5.5). The ratios {R; - E[R;]}/.) V[R;] will be standardized variables approximately normal. Kruskal and Wallis denoted by H the sum of their squares multiplied by a weighting factor 1 - 11;/N, and showed that H has approximately a X2(k - 1) distribution; i.e., H

=.± (N[R;+ -1)(N(N -+ /lj)/1211; 1)/2]2 . (1 _~) N i

r-...I

X2(k - 1).

(7.1)

approx

Two identical forms for H, more convenient for calculation, are H

= =

N - 1 N

.± [R; j

(N 2

+ 1)/2]2 1)/1211;

(7.2)

_3(N + 1).

(7.3)

(N -

12 i(R~) N(N + 1) j II;

If ties occur, H should be divided by the factor

1-

~T

N3

_

N

(7.4)

where T = (I - 1)1(1 + 1) is calculated and included in the sum for each group of ties. While H is distributed asymptotically as X2(k - 1) for large k and 11;, in small samples the approximation is not very good, and Kruskal and Wallis [11] provided tables of the exact distribution for the

SECT.

7.7

THE

H

257

TEST

case of k = 3, lli S 5. For intermediate cases they proposed approximations based on the incomplete gamma and incomplete beta distributions, for details of which see their paper. Wallace [12] has discussed several approximations, and Box [13] suggested an approximation based on the F distribution. As an example of the application of the H test, Table 7.9 gives the terminal digits of observations on six days on the mechanical equivalent of heat [14]. The first observation was 4.1849, but only the last two digits are given in Table 7.9. There are several groups of ties, the first being Table 7.9 Day 1

Day 2

Day 3

Day 4

Day 5

Day 6

Obs. Rank

Obs. Rank

Obs. Rank

Obs. Rank

Obs. Rank

Obs. Rank

49 52 43 /Ii

Rl

18 21t 15

3

42 43 42 43

12t 15 12! 15

46 52 27 51

4 54!

17 21t 5 20

4 55

11

27 71 23

1 5 23 2

38 50 41 41 4

4 63!

31

12

23(23

+ 1)

[(54 t )2 3

+ (5W + ...J- 3 4

X

(23

3 9 5 7

4 24

48

three observations at 27 which receive the mean rank (4 Substituting in (7.3), H =

24 40 27 37

8 19 10! 10!

+ 5 + 6)/3 =

+ 1) =

8.753.

5.

(7.5)

However, we need the correction for ties, there being two groups of three tied ranks and three groups of two tied ranks. For each of the triplets the correction (I - l)t(t + I) is (3 - 1) . 3 . (3 + 1) = 24, and for each of the doublets the correction is (2 - 1) . 2 . (2 + 1) = 6. The correction factor (7.4) is 1 _ 24 x 2 + 6 x 3 = 0.99457' 23 3 - 23 '

(7.6)

so the corrected H is 8.753/0.99457 = 8.801 with 6 - 1 = 5 degrees of freedom. The 0.90 point of X2(5) is 9.24, and so the null hypothesis is acceptable at the 0.10 level. We can apply the H test in the case where there are only two samples, so that k = 2. The H statistic (7.1) will then have approximately the X2 distribution with 2 - 1 = 1 degree of freedom. In Section 1.27 we saw n

that x2(n) was defined as L

u~;

so X2(1) is just the square of a single unit nor-

i

mal deviate. We shall see in Section 9.4 that specifically XL.(I)

= UL./2'

2S8

SOME NONPARAMETRIC TESTS

CHAP.

7

It is easy to show that the H statistic in the case of k = 2 is identically equal to the square of the Wilcoxon statistic (S.7). In other words, the H test for the case of k = 2 is identical with the two-sample Wilcoxon test. 7.S. The Wilcoxon One-Sample Test

Wilcoxon [7] proposed a test for the median of a single sample, in which we give ranks to the absolute magnitudes of the observations and then give to the ranks the signs of the corresponding observations. Essentially the test is of the null hypothesis that the distribution of the observations is symmetric about zero, so that any rank is equally likely to be positive or negative. Frequently in application the observations are differences between paired observations, as in Table 7.10 which reproduces the data of Table Table 7.10 Difference 0.3 6.3 3.7 2.8 5.8 -1.4 1.7 2.3 -1.7 1.6 -1.8 0.6 4.5 1.9 2.4 6.8 Rank 1 15 12 II 14 -3 5t 9 -5t 4 -7 2 13 8 10 16

7.1 on the difference in potency for 16 lots of a pharmaceutical product reported by two methods of analysis, here with ranks attached to the absolute magnitude of the differences. Table 7.10 includes the complication of two differences 1.7 and -1.7 tieing in absolute magnitude. As in the unpaired Wilcoxon test, such ties receive the mean of the ranks necessary for the group of ties, here t(S + 6) = S!. The test statistic is the sum of the ranks of either sign, 1St for the negative ranks or 120! for the' positive ranks, and the P value is the probability of getting a rank sum equal to or more extreme than the observed value. Under the null hypothesis any rank is equally likely to be positive or negative. The total number of ways rank sums can be produced is thus 2N , 216 in the case of Table 7.10. In moderate situations it is possible to enumerate all the rank sums less than or equal to the observed rank sum. Here the observed negative rank sum is 3 + S! + 7 = 1St. We proceed to write down all ways in which we can form rank sums less than or equal to IS!. Starting with 0, I, ... ,IS; continuing with I + 2, ... , I + 14, 2 + 3, ... , 2 + 13; continuing on up to 7 + 8; then counting triplets such as I + 2 + 3, quadruplets such as 1 + 2 + 3 + 4, and the quintuplets I + 2 + 3 + 4 + S!a and I + 2 + 3 + 4 + Stb; there are a total of 140 combinations of ranks with sums ~IS!. The one-sided P value is thus 140/216 = 0.00214.

SECT.

7.8

259

THE WILCOXON ONE-SAMPLE TEST

Obviously, direct enumeration can be excessively tedious and a normal approximation is useful. Consider the ranks Ri , i = 1,2, ... ,N. Construct subsidiary variables di , where di is attached to the ith rank. These variables d; take the values 0 when a rank is negative and 1 when a rank is positive. Under the null hypothesis a rank is equally likely to be positive as negative. Then E[d i ] = 1 X

E[d~] = ,

E [did;] i'/;

=

1 0 X 0 X4

12 X

+0 X

! +0 X ! 2

2

=

!,

(8.1)

2

l2 + 02 X l2 = l2'

1 1 X4

+1X

1 0 X4

(8.2)

+1X

1 1 X 4

= -1 ;

4 (8.3)

~V

The sum of the positive ranks, say S, is L diRi' Under the null hypothesis, i

(8.4)

NN

NN

i

i

= L L R~E[d~] + L L R;RjE[did;] 1 i>Fl

j

i=1

(8.5) Thus

1N V[S] = E[S2] - (E[S])2 = - L R~.

4 ;

(8.6)

If the Ri are the integers 1 to N, then using (5.1) and (5.2),

E[S]

=

N(N

+ 1) , 4

V[S] = N(N

+ 1)(2N + 1) . 24

(8.7) (8.8)

If, instead of the integers 1 to N, the ranks contain a set of t ties, as in Section 7.6 the sum of the ranks is unchanged, so E[S] continues to be

260

CHAP. I

SOME NONPARAMETRIC TESTS

given by (8.7). The sum of squares of the ranks is changed, however. In place of the ranks R j + 1, R j + 2, ... , Ri + t we have the mean rank R j + (t + 1)/2 a total of t times [see (6.1)], so the sum of squares of the ranks is increased by

+ 1)/12.

which reduces to -(t - l)t(t (8.6) and (8.8) gives V[S]

=

N(N

N

Making this change in

L R~ in ;

+ 1)(2N + 1) _

(t - l)(t 48

24

+ 1) .

(8.10)

For the data of Table 7.10, the sum of the positive ranks S = 120t, and there is one pair of ties; so E[S]

=

N(N

+ 1) =

16

X

4 V[S]

=

16 X (16

+ 1) = 68,

(16

+ 1)(2

X

16

+ 1) _

24

u_ 1 P

(2 - 1) X 2 X (2 48

= 374 - 0.125 = 373.875, = S - E[S] = 120.5 - 68 = 2.715 .JV[S]

(8.11)

4

.J373.875

+ 1) (8.12) (8.13)

'

whence P = 0.0033, to be compared with the exact figure 0.0021 previously obtained. While nothing generally appears to be known about the use of a continuity correction with this approximation, its use does not seem to be indicated. We have indicated how to handle ties in the absolute values of the ranks. However, if the observations are differences between paired samples, some samples may be tied in one or more pairs, giving rise to differences which are zero. It is not clear how these zeros should be handled. One procedure will be to ignore their existence completely, i.e., to delete them from the sample. 7.9. The Friedman Rank Test Consider the data of Table 7.11, which gives a function of the daily determination of the efficiency of a chemical plant for each of six runs, each run lasting seven days. We ask whether the null hypothesis that the median efficiencies for the different runs are the same should be rejected. At first glance we might consider using the H test of Section 7.7, but this

SECT.

7.9

261

THE FRIEDMAN RANK TEST

is not appropriate to the present problem; it assumes under the null hypothesis that all observations are identically distributed, whereas in the present problem we postulate as the null hypothesis merely that all observations on day 1 are identically distributed and that all observations on day 2 are identically distributed, etc., but we do not postulate that the day 2 distribution is the same as the day 1 distribution, etc. Table 7.11 Run number Day 2

3

4

5

6

30(2) 41(3) 40(2) 41(3) 37(3) 17(1) 12(1)

72(6) 38(2) 46(3) 47(5) 38(4) 60(6) 41(5)

35(3) 35(1) 33(1) 46(4) 47(6) 47(5) 38(4)

31

24

1 2 3 4 5 6 7

60(4) 62(5) 58(5) 52(6) 31(1) 23(2) 26(2)

64(5) 63(5) 63(6) 36(1) 34(2) 32(3) 27(3)

14(1) 46(4) 47(4) 39(2) 42(5) 43(4) 57(6)

1 Rij

25

26

26

r

15

i

In general, suppose that there are I' rows and c columns, that we wish to test for column differences, and that the rank in the ijth cell, the ranking being within rows, is Rij. For Table 7.11, for the first row the observations are 60, 64, 14, 30, 72, and 35, so these are allocated the ranks 4, 5, 1, 2, 6, and 3 given in parentheses beside the corresponding observations. For the ith row for the ranks Rii we have the integers I, ... , c, so E[Rij] = (c + 1)/2, V[Rii] = (c 2 - 1)/12 [see (5.3) and (5.4)]. Then since R. i , the mean of the ranks in the jth column, is the mean of I' R;/s, _ V[R. i ]

If we assume that

c2

1

-

= --.

(9.1)

121'

R.i is asymptotically

normal we have

R.,. - E[R .,.] "" N(O 1) .jV[R.;l "

(9.2)

and the sum of squares of c such quantities will have approximately a X2(C) distribution. However, the R.; are not independent, since c

_

c

1

r

1

r

c

1i R.; = 11 RiJ = -I '11 R;; = jl'; ; j

c( C

+ 1) 2

.

(9.3)

262

SOME NONPARAMETRIC TESTS

CHAP.

7

Along these lines Friedman [15] derived the statistic which he called X~, c

L' ' -CR.i - E[R. j ])2 1 ... 2 _ -_ _,----_ _ , Xr - -cV[R. i ] C

(9.4)

and showed that under the null hypothesis it has approximately the X2(C - 1) distribution. The statistic can be put in the form

X~ =

.i (i Rii)2 - 3r(c + 1).

12 rc(c + 1),

(9.5)

t

For Table 7.11,

X2 r

=

7

X

12 6(6

+ 1)

(25 2

+ ... + 242) -

3 X 7 X (6

= 5.612,

+ 1) (9.6)

which is substantially less than X~.9o(5) = 9.24, so the null hypothesis can be accepted. We could, of course, use the same procedure to test whether the median efficiencies for the different days are equal. An alternative, but essentially identical, form of this test was introduced by M. G. Kendall under the name coefficient of concordance [4]. 7.10. The Cochran Q Test Consider the data of Table 7.12 taken from Cochran [16]. A number of specimens, 69 in all, were tested on four different media A, B, C, and D for the presence of a certain organism. For a particular specimen on a particular medium the outcome could be growth or no growth, represented by 0 or 1. For a particular specimen, therefore, the results can be 1111, 1110, 1101, ... , 1000,0000. Table 7.12 gives the number of specimens Table 7.12

T,

A

B

C

D

Number of cases

1 1 0 0 0

1 1 1 1 0

1 1 1 1 0

4 2 3 1 59

6

10

1 0 1 0 0 7

10

with particular results: for example, there were 4 1111's, 0 1110's (not

SECT.

7.10

THE COCHRAN

Q TEST

263

shown in the table), 2 1101's, etc. We are interested in testing the null hypothesis that the probabilities of growth with the different media, averaged over all specimens, are equal. This situation is the generalization to k columns of the situation discussed in Section 3.7. There, if each patient had been tested on k drugs rather than two, then we would have the situation discussed here. Cochran proposed for a test of the null hypothesis the statistic Q defined as c

Q=

c(c - 1) L (T; - T)2 r

C

LU

; i -

i

r

(10.1)

L u~ i

where U i is the number of 1's in the ith row and T j is the sum of the entries in the jth column, allowing, of course, for the fact that in Table 7.12 each row is really repeated a total number of times equal to the corresponding number of cases. A sketch of the proof that the distribution of Q under the null hypothesis tends to X2(C - 1) as the number of rows other than those all 0 and all 1 tends to infinity is quite involved, and we can obtain this result more simply by regarding the Cochran Q test as a special case of the Friedman test with ties in the ranks. * Let xiJ be a variable with the property that Xij

=

1

=0

if ijth cell has a success, if ijth cell has a failure.

(10.2)

Then since T; is the number of successes in the jth column, r

LXi; i

and since

Ui

=

T;,

(10.3)

is the number of successes in the ith row, c

LXii = ui •

(10.4)

;

It follows that (10.5)

Consider a given row with Ui successes. The successes are allocated ranks 1, ... , U i with sum ui(u j + 1)/2 and hence mean rank (u; + 1)/2. The c - ui cells with failures are allocated ranks U j + 1, ... ,c, and the mean '" This was demonstrated to the author by Nancy D. Bailey following a surmise by William H. Kruskal at a seminar by Bailey.

264

SOME NONPARAMETRIC TESTS

of these ranks is (c

+ Ui +

1.L c

C

mean rank (c

+ Ui +

1)/2 occurring

r

Ui

C -

times and the

times. Thus

(c - IIi) ( C+

{~[lIi IIi;

-

(10.6)

Rij .

j

+ 1)/2 occurring Ui

1+ 1+

~[lIi (IIi;

V[R ii ] =

)2

c

C

we have the mean rank (u i

j

(1-.L

2

Rij -

j

c

.L Rij

7

1)/2. From (4.2), the variance of the ranks

V[Rij] = -

In

CHAP.

IIi)

(C _

+

C

1)1 + IJf

I;; + I;i

CUi - U~ = ---!..---! 4

(10.7)

Therefore V[R.i] = V [

] 1 r -.L R;i r i

1

r

4r

i

= -2.L (CUi - II~).

(10.8)

In the ith row the ranks are RiJ equal to (u i + 1)/2 or to (c depending on whether Xu = 1 or O. We can write

1 2

Rii = - [II;

+ 1 + c(l -

+ Ui +

1)/2

(10.9)

xii)]'

so l r

_

r

l

1

.L IIi + -2 (C + 2r i

R.i = - .LRij = -

r ;

1) - -

c r

.Li Xii·

2,.

(10.10)

Now R.. = (c + 1)/2, and using (10.3) and (10.5) for the first and last terms on the right-hand side we get c R.j - R .. = - 2- (Tj - T). (10.11)

,.

Therefore r

_

.Li (R.i -

_

2

R.J

c2

c

= - 2 .L (Tj 4,.

-

2

(10.12)

T) .

-

j

We now substitute (10.12) and (10.8) in (9.4) to get 2

c

~ .L(Tj

c

-

T)2

14,.2 j Xr = - - -l---'r---c - 2 .L (CII; - II~) 4,. i 2

C -

c(c - 1) .L (Tj

which is identical with Cochran's Q, (10.1).

-

j

C

r

r

i

i

.L IIi - .L II~

T)2

(10.13)

265

EXERCISES

To compute Q for the data of Table 7.12 we need a table showing the number of specimens having each possible value of Ui (Table 7.13) from which we get

..

~

IIi

=4

X

4

+3 X 5+2

X

1

+1 X 0 +0

X

=

59

33,

(10.14)

=

113. (10.15)

i

.

~ 1I~

= 42 X

+ 32 X 5 + 22 X 1 + 12 X 0 + 0 2 X

4

59

j

c

We also need ~ (T; - T)2 which can be calculated as ;

i

:i

!(i T;)2 = 62 + 102 + 7 2 + 10 2 -

T~ -

C;

33

2

4

= 12.75.

(10.16)

Thus

Q = 4(4 - 1)

X 12.75 = 8.05. 4 X 33 - 113

Since X~.95(3)

=

7.81, P

(10.17)

< 0.05. Table 7.13

II;

Number of specimens

4 4

3 5

2 1

1

o

o 59

EXERCISES 7.1. Apply the appropriate Wilcoxon rank test to the data of Table 7.4. 7.2. A group of mice are allocated to individual cages randomly. The cages are allocated in equal numbers, randomly, to two treatments, a control A and a certain drug B. All animals are infected, in a random sequence, with tuberculosis [Note: the experimenter usually wants to infect all the controls and then all the B group, but this is terrible]. The mice die on the following days following infection [one mouse got lost]. Control A: 5, 6, 7, 7, 8, 8, 8, 9, 12 Drug B: 7,8,8,8,9,9, 12, 13, 14, 17 A preliminary experiment having established that the drug is not toxic it can be assumed that the test group cannot be worse (die sooner) than the control group under any reasonable alternative hypothesis. Report the P value for the null hypothesis that the drug is without effect, using the appropriate Wilcoxon test. 7.3. Acid is concentrated continuously in a certain type of plant. Part of the plant corrodes and eventually fails. The throughput in hundreds of tons obtained between installation and failure is recorded. These parts were obtained

266

CHAP.

SOME NONPARAMETRIC TESTS

7

from three separate foundries. Test the hypothesis that the median life is the same for the three foundries. Foundry

Throughputs obtained

A B C

84, 60,40,47,.34,46 67,92,95,40,98,60,59, 108,86, 117 46,93,100,92,92

7.4. In a rain-making (?) experiment, rainfall measurements were made on 16 pairs of days. On one day in each pair the clouds were seeded and on the other day no seeding was done. The choice of which day in a pair to seed was made randomly. The total rainfall over the network of gauges for the 16 pairs of days is below. Test the null hypothesis that seeding is without effect with (a) the sign test, (b) the appropriate Wilcoxon test. Report one-sided P values. Pair no.

S

NS

Pair no.

S

NS

1 2 3 4 5 6 7 8

0 2.09 0.07 0.30 0 2.55 1.62 0

1.37 0 0 0.10 0.44 0 1.01 0.54

9 10 11 12

0 1.87 2.50 3.15 0.15 2.96 0 0

0 0.62 0 5.54 0.01 0 0 0.75

13

14 15 16

7.5. In a trial of two types of rain gauge, 69 of type A and 12 of type B were distributed at random over a certain area. In a certain period 14 storms occurred, and the average amounts of rain found in the two types of gauge were as shown in the accompanying table. Storm

Type A

Type B

Storm

Type A

TypeB

1 2 3 4 5 6 7

1.38 9.69 0.39 1.42 0.54 5.94 0.59

1.42 10.37 0.39 1.46 0.55 6.15 0.61

8 9 10 11 12

2.63 2.44 0.56 0.69 0.71 0.95 0.50

2.69 2.68 0.53 0.72 0.72 0.93 0.53

13

14

Data from E. L. Hamilton, "The Problem of Sampling Rainfall in Mountainous Areas," pp. 469-475 of Proceedings of the Berkeley Symposium 011 Mathematical Statistics and Probability, J. Neyman (ed.), University of California Press, Berkeley and Los Angeles, 1949. Obtain the one-sided P value for the hypothesis that the two types of gauge are

267

EXERCISES

giving similar results, using (a) the appropriate Wilcoxon test by exact enumeration, (b) the appropriate Wilcoxon test using a normal approximation, (c) the sign test. 7.6. On a number of days coffee is prepared by two different electric coffee makers, and the time in minutes and tenths of a minute for the coffee-making operation to be completed by each machine is recorded. The results were as below.

It is probable that the time required varies from day to day due to variation in such factors as the temperature of the water or line voltage. Use the appropriate Wilcoxon test to find the exact P value for the null hypothesis that the median times for the two machines are the same, against the one-sided alternative that machine B takes longer than machine A. Make a similar test with the sign test. 7.7. Fruit juice is stored for a number of months in four types of container and then rated for quality by eight panels of tasters. The taste scores assigned by the panels were as below. Panel Container 5 2 3 4 6 7 8 A

B C D

6.14 6.55 5.54 4.81

5.72 6.29 5.61 5.09

6.90 7.40 6.60 6.61

5.80 6.40 5.70 5.03

6.23 6.28 5.31 5.15

6.06 6.26 5.58 5.05

5.42 6.22 5.57 5.77

6.04 6.76 5.84 6.17

(a) Test the null hypothesis that there is no difference between the containers. (b) Test the null hypothesis that there is no difference between the panels. - 7.S. Samples of soil were taken at five locations on a farm and tested for the

presence of a certain fungus. The test was of the form that yields only yes or no answers, represented by 1 or 0 in the table below. The same five locations were sampled on a total of 18 occasions. Test the null hypothesis that the probability of a sample being positive is the same for all five locations. Occasion Location 1 2 3 4 5 6 7 8 9 10 11 1 2 3 4 5

0 0 1 1 0

0 0 1 1 0 1 0 1 1 0 1 1 1 1 1 I I 1 0 1 I 0 0 1 0

0 0 1 1 0

1 1 1 0 0

1 0 1 1 0

1 0 I

0 0

0 0 0 0 0

12

13

14

15

16

17 18

0 0 0 0 0

0 0 0 0 0

0 0 0 0 0

0 0 0 1 0

0 0 0 0 0

0 1 1 1 1

1 1 1 0 0

268

SOME NONPARAMETRIC TESTS

CHAP.

7

7.9. A sample of five observations, xl> ... , x5 is taken from one population and a sample of five observations Y1, ... , Y5 is taken from another population. It is found that four of the x's lie below the joint sample median. (a) Using the median test, what is the one-sided P value for the null hypothesis that the samples came from populations with the same medians? (b) Suppose that the density function of Y is py{Y} =1 =

0

1

1

-Z
and that the density function of x is

px{x} = 1 =

0

-I

1 1

< x < - 2' 2 < x < 1,

otherwise.

What is the probability of observing four or more x's below the joint sample median under these assumptions? (The relevance of this exercise is that it demonstrates that the median test does assume under the null hypothesis indentity of the two distributions.) 7.10. In the context of Section 7.4, consider a finite population of size N in which M of the elements have Xi taking the value 1 and the remaining N - M elements have Xi taking the value O. If Xi = 1 call the element defective and if Xi = 0 call the element nondefective. Take a sample of size II from this popun

lation. Then L Xi = IIX is equivalent to the number of defectives in the sample. i

Show that

E[~

Xi] =

IIM/N, and

11M (1 - M) [ " ] IV N (1 -

V ~>i =

I)

NII -_ 1 .

Relate these results to those of exercise (3B.2). 7.11. Consider the median test of Section 7.3. Suppose that the numerical values of all observations below the median are lost and all that is known is which sample they came from and that they are below the median: analogously for all observations above the median. Now analyze the data as in a two-sample Wilcoxon test with adjustment for ties. Show that the unit deviate approximately normal of the Wilcoxon test is identical with the large sample statistic for the median test in its unmodified form (3.10.7). 7.12. Suppose that we consider taking a sample of size II from a normal distribution with known variance 0"2 and testing, with the usual test at level of significance 0( assuming normality and knowledge of the variance, the null hypothesis that the mean is ;0 against the alternative that it is ;l> ;1 > ;0' and that we require that II be such that the power be some value 1 - fJ when (;1 - ;0)/0" is some small value 5. Now consider taking a sample of size 1/' from the same normal distribution and testing at level of significance 0( with the sign test the null hypothesis that the median is ;0 against the alternative that it is ;1, ;1 > ;0' and that we require /I'

REFERENCES

269

be such that the power is the same value 1 - {J when (¢1 - ¢o)/a is the same small value 0 as in the previous paragraph. Obtain a simple expression for n/Il'. This is known as the asymptotic relative efficiency of the sign test. Where appropriate use a "rectangular" approximation to the integral of a normal curve over a small interval. For the power of the sign test use a formula that assumes that the binomial distribution can be approximated by a normal distribution. Where appropriate approximate by discarding terms in 0 2 • 7.13. The statistics (5.6) was derived by picking one of the two samples and calculating its observed mean rank R, E[R], and V[R]. Show that if the other sample is selected, the absolute value of the corresponding statistic is identical with the absolute value of (5.6). 7.14. Suppose that in a two-sample Wilcoxon test one sample of observations is Xl"'" XII and the other sample is Y I , .•• , Y",. Consider all possible pairings (Xi, Yj ) and let U be the number of pairs in which Xi < Yj • Let RY j be the ranks of the Y's. Show that m

U=~RYj,=1

[Since m(m

+ 1)/2 is

a constant,

m(m + 1) 2

1'" R Yj (the sum of the ranks of the

Y's) is a

i

linear function of U. Since U is an intuitively appealing statistic, this provides a justification for the use of the rank sum as a test statistic. This is the approach taken by Mann and Whitney [9]].

REFERENCES I. Cramer, H., Mathematical Methods oJ Statistics. Princeton: Princeton University

Press, 1951. 2. Savage, I. Richard, Bibliography oJ NOllparametric Statistics. Cambridge, Mass.: Harvard University Press, 1962. 3. Fraser, D. A. S., NOllparametric Methods ill Statistics. New York: John Wiley and Sons, 1957. 4. KendaIl, M. G., Rallk Correlatioll Methods. 2nd ed.; London: Griffin and Co., 1955. 5. Putter, J., "The Treatment of Ties in Some Nonparametric Tests," Allllals of Mathe. matical Statistics, 26 (1955), 368-86. 6. Wilcoxon, F., "Individual Comparisons by Ranking Methods," Biometrics Blllletill, 1 (1947), 80-83. 7. Wilcoxon, F., "Probability Tables for Individual Comparisons by Ranking Methods," Biometrics, 3 (1947), 119-22. 8. Dean, G., "The Atomic Weight of Carbon and Silver," JOllmal oJ the Chemical Society of LOlldoll, 125 (1924),2656-72. 9. Mann, H. B., and D. R. Whitney, "On a Test of Whether One of Two Random Variables is StochasticaIly Larger than the Other," AII/lals oJMathematical Statistics, 18 (1947), 50-60. 10. Kruskal, W. H., "Historical Notes on the Wilcoxon Unpaired Two-Sample Test," JOllmal of the Americall Statistical Associatioll, 52 (1957), 356-60.

270

SOME NONPARAMETRIC TESTS

CHAP.

7

11. Kruskal, W. H., and W. Allen Wallis, "Use of Ranks in One-Criterion Analysis of Variance," Journal of tire Americall Statistical Associatioll, 47 (1952), 583-621. 12. Wallace, ~. L., "Simplified Beta~Approximations to the Kruskal-Wallis H Test." Journal of tire Americall Statistical Associatioll, 54 (1959), 225-30. 13. Box, G. E. P., and S. L. Andersen, "Permutation Theory in the Derivation of Robust Criteria and the Study of Departure from Assumption," Joumal of tire Royal Statistical Society, B (Methodological), 17 (1955),1-26. 14. Laby, T. H., and E. o. Hercus, "The Mechanical Equivalent of Heat," Philosoplrical Trallsactiolls of tire Royal Society, A, 227 (1928), 63-91. 15. Friedman, M., "The Use of Ranks to Avoid the Assumption of Normality in the Analysis of Variance," Joumal of tire Americall Statistical Associatioll, 32 (1937), 675-701. 16. Cochran, W. G., "The Comparison of Percentages in Matched Samples," Biometrika, 37 (1950), 256-66.

CHAPTER 8

The Partitioning of Sums of Squares

8.1. The Distribution of Sample Estimates of Variance This chapter, largely algebraic, is devoted to a proof of two theorems. The theorem of this section is the basis of the Student t test to be discussed in Section 9.6 and that of the following section is the basis of the analysis of variance discussed in Chapter 10. Some readers may prefer to omit this chapter temporarily. The account given here is based on an exposition by A. Hald [1]. Suppose that we have n independent observations Xl> ••• , Xn from a normal population of unknown mean and variance (J'2. In this section we will show that the usual unbiased estimator of the variance

e

~ ( Xi = -1- k.,

s2

n - 1 i=l

-)2

-

X

,....., (J'

2

x2(n

-

11 -

1) ,

1

(Ll)

where,....., is used to denote "is distributed as," and that S2 is independent of x, These two results are the basis of the Student t test to be discussed in Section 9,6, We can write 11

!

11

e)2

(Xi = i

x) + (x -

I

[(Xi i

n

m2= I

(Xi -

X)2 + n(x - e)2,

i

(1.2) Dividing by

(J'2,

(x _ 1:)2 =I_i_' n (X _ x)2 + (x _ 1:)2 I_'i_S-~, 11

i

(J'

i

(J'

271

(J'/.J 11

(1.3)

272

THE PARTITIONING OF SUMS OF SQUARES

Define ui

= (Xi

~)Ia, so that Xi

-

_ 11 = -

X

n

Xi

= a

1Ii

= aUi

(1-1 n

CHAP.

S

+~. Then

)

_

+~=

Ui

1Ii

au

+ ~,

(1.4)

so X

Xi -

a

and

= aUi + ~ - (aii + ~) = a

x-

~

---- =

alJI!

aii + ~ alfo

~

_

Ui -

ii IIJI!

(1.5)

U,

,- _

= --- = V

(1.6)

Ill/.

Thus (1.3) can be written as n

n

i

i

1 U~ = 1 (U i

ii)2

-

+ (In£()2

(1.7) (1.S)

say. n

The left-hand side

1 u~

is distributed as

x2(n).

On the right-hand side,

J~ ii = ii(lIJ~) is ~ unit normal deviate, and hence its square Q1 is distributed as x2(l). We shall now prove that Q2 is distributed as X2(11 - 1) n

and is independent of Q1' This will establish that as a2X2(11 - 1) and is independent of (x _ ~)2.

1 (Xi -

X)2 is distributed

i

11

Define Ii = ui

ii, so that Q2 = 1/~, and

-

i

iii = i Ui a.

1.

I!ii

=

i IIi t

I!

(i 11;/11)

(1.9)

= O.

t

The number of degrees of freedom for a set of variables is defined as the number of variables minus the number of independent linear relations or constraints between them. The number of degrees of freedom of the sum of squares of a set of variables is the same as the number of degrees 11

of freedom of the variables themselves. In the present instance,

1 U~ i

involves 11 variables U i with no linear relationship between them, and so it has 11 degrees of freedom; but Q2 involves 11 variables Ii with the rest ric-

" tion Iii

= 0,

and so Q2 has 11 - 1 degrees of freedom. The 11 variables

i

Ii are not independent [see exercise (S.l)], but we can construct from them /I -

1 variables n

(a) 1/~ ;=1

"

=1

(i = 2,3, ... , /I) with the properties that

v~,

i=2

(b) the

Vi

(c)

Vi

the

Vi

are independent unit normal deviates, (i = 2, ... ,11) are independent of V1 defined as

J;;ii.

SECT.

8.1

..

L

Thus the distribution of of Q2

=

273

THE DISTRIBUTION OF SAMPLE ESTIMATES OF VARIANCE

v~ is

X2(11 - 1), and therefore the distribution

l=l

L I: is also x2(n 'n

1) and is independent of Q1.

i=2

11

n

;=1

;=1

L q by using the relation L

The first step is to eliminate If from which implies that

Ii = 0, (1.10)

so I~ = I~

+ 2(12/a + 1214 + ... + 12 / + I~ + 2(1a14 + ... + la/ ..)

11 )

+ ...

+

+ I!.

(1.11)

If in Q2 = L I: will give Q2 in the form of a function i=1 1 variables Ii' i = 2, 3, ... , 11: Q2 = 2(1~ + 121a + 12/4 + ... + 12/ .. + I~ + la/4 + ... + lain 11

Substituting this for of the

11 -

+

... +

+ I~).

(1.12) This can be expressed as the sum of 11 - 1 squares, by first collecting all terms that involve 12 , the first row, and writing them as a perfect square plus whatever is left over: 2(1~

+

12/a + 12 /4

=

{v'2[12

+ ... + 12/.. )

+ ~(1a +

14

+ ... +

11l)Jf

-lI2-11 2a a4 -···-11 an

-!2 12 4

... - 11 4/1

_l12 2 n·

(1.13)

274

THE PARTITIONING OF SUMS OF SQUARES

CHAP.

8

Substituting this in (1.12), Q2

=

{J2[/2 + ~ (13 + I( + ... + In)Jf + ~ I: + 13/( + ... + 13/n 2

+ ~ I!. 2

(1.14)

The next step is to collect all the terms in 13 excluding those in the first row: They are set out in the second row of (1.14). We write them as a perfect square plus whatever is left over, just as we did for 12 , The square part is

(1.15) This procedure can be continued until we are finally left with only I!. These squares can be denoted by i = 2, ... ,11:

v:'

2= J2[/2 + ~ (13 + I( + ... + u].

V

V3 = J~[/3 + ~(/4 + 15 + ... + In)].

Vi

=

J.

Vn

=

J

i

1-1

11

Il -

We have thus expressed Q2

[Ii + ~(li+l + li+2 + ... + In)J, I

1

= !n i=l

Vi

(1.16)

In' "

l~ as! v~. It remains to show that these i=2

are unit normal deviates and are independent of each other and of V!,

8.l

SECT.

To do this we express the

Ii == Vi

= =

V;,

i> 1, in terms of the Ui

.J'1(1,1j"

1)

1 1)

.J'1(1,1-

1)

= .J'

,1 1(1 -

+ liH + ... + 1,,)

(iIi ('

",(, -

1)

,_

Hli -

IU

+ lIiH - U_+ ... + II" - U_)

(illi

+ lIiH + ... + II" -

nii)

(ill;

+ 1Ii+1 + ... + II" -

111 -

= -

by the substitution

ii:

Ui -

==

275

THE DISTRIBUTION OF SAMPLE ESTIMATES OF VARIANCE

.J'1(1,1-

+ 112 + ... + II

[111

1)

j_

1 -

112 II; -

(i -

..• lIiH -

lI i -1 ..• -

II,,)

(~,17)

1)11;].

The Vi will be normally distributed since they are linear functions of the which are normally distributed, and

E[vi ] = -

.J'1(1,1-

1)

{E[1I1]

+ ... + E[lI i - 1] -

lI;

(i - 1)E[lIiU = 0, (1.18)

V[vJ

= , ,1 1(1 -

Cov[v;, Vj]

1)

{V[lId

= E[viv j] = .J"

+ ' , , + V[lI i - 1] + (i -

E[Vi]E[Vj]

1, ,

1(1 -

1)2V[U;U = 1, (1.19)

l)J(J -

1) X

E[(1/1 +

' . , + l/i-1 -

(1/ 1 + . , , +

II ;-1 -

(i - 1)lIi)

(j - 1)1/ j)]'

(1.20)

For i < j, the expectation E[ ] will include the squared terms lI~ for i and their expectation is E[II~

+ ' , , + 1I~-1 -

(i - 1)1I~] = 0,

< j,

. (1.21)

The expectation E[ ] will also include cross products of the form kr.uru., ,. ¥= s, where k r • is some constant. Their expectation is E[kr.uru.] = kr.E[ur]E[u.] = O.

(1.22)

Thus Cov[v i , v;] = 0, It will be shown in Section 12.3 that, if V 1 , V 2 are linear functions of normally distributed variables u1 , u2 , then V 1 , V 2 are jointly distributed in the bivariate normal distribution, and that, if Cov[v 1 , v2 ] = 0, then V1 , V 2

276

THE PARTITIONING OF SUMS OF SQUARES

CHAP.

8

are independent. Thus the Vi are independent, and since they have zero means and unit variances, and are normally distributed,

Q2

= I"

i=2

V~ '"'" x2(n - 1),

(1.23)

where as before the symbol,"", is used to denote "is distributed as." It is easy to show that v1, Vi' i = 2, ... , 11, are independent, and hence Q1 is independent of Q2' To recapitulate, if we write n

n

I LI~ = (In ii)2 + I;=1(Lli ;=1

ii)2 = Q1

+ Q2'

(1.24)

then Q1 and Q2 have independent X2 distributions with degrees of freedom 1 and n - 1. Substituting Ui = (Xi - ~)/(J and multiplying both sides by (J2 gives

L" (Xi -

~)2 = /l(X -

~)2

+ I" (Xt -

i

X)2,

(1.25)

i

for which the left-hand side is distributed as (J2X 2 with n degrees of freedom and the two terms on the right-hand side are distributed as (J2X2 with degrees of freedom 1 and n - 1, and are independent. This implies that for normally distributed random variables S2

and also that

i

= _1_ (x; _ X)2,"", 11-1;

(J2

X2(/l - 1) ,

11-1

(1.26)

x is independent of S2.

8.2. The Partitioning of Sums of Squares into Independent Components The result of the preceding section will be used for the Student t test in Chapter 9. For the comparison of k means, in Chapter 10 and following, we need a stronger result, Cochran's theorem [2]. A restricted form of this theorem is as follows. We have n unit normally distributed and independent random variables U i , i = 1, ... ,n. The sum of their squares is distributed as x2(n). Suppose that Qi' j = 1, ... ,k, are sums of squares with Ii degrees of freedom. Then, if k

n

Ii

Qi

= Q1 + ... + Qk = I

II~

(2.1)

i k

and

Iii = i1 + ... + fk = i

II,

(2.2)

8.2

SECT.

277

PARTITIONING INTO INDEPENDENT COMPONENTS

then the theorem states that the Qj are independently distributed as x2(h). Actually the theorem is somewhat stronger in that the Qj can be quadratic forms,

!! ajjxixj = allx~ + a12 x l x 2 + ... + alnxlX n i

j

+ a 21 x l x 2 + a22x~ + a23x 2x 3 + .. " (2.3) where a ii = aji' We assume that each Qj is the sum of the squares of m, variables lij' i=l, ... ,mj; i.e., 2

Qj = I lj

mj

+ I~j + ... + I~,,;j = I

I~j'

(2.4)

i

We assume that, among the mj variables Iii' there are rj linear relations, so that the number of degrees of freedom for Qj is Ii = mi - r j • We are mJ

going to show that each Q,

=I

I~ can be written in the form

~

=I

Q,

i

V~"

i

in which the Vii are independent unit normal deviates. This will show that the Q, have x2(f;) distributions and are independent. If there are r i linear relations among the mi 'variables Iii' then we can use these relations to eliminate rj of the Iii and convert Qj into a quadratic form in only mj - rj = Ii variables Iii' Thus it is possible to obtain (2.5) Further, this quadratic form can be linearly transformed into another quadratic form inli variables Vli' ••• , V,;j involving only squares (2.6) in which the hi} (2.1), we have

= ± 1 (see, e.g., Section 51

of [3]). Combining (2.6) and (2.7) k

Now, if the condition (2.2) for Cochran's theorem is satisfied,

If, = 11, j

so that the number of terms on the left-hand side of (2.7) equals the number of terms on the right-hand side. The theory of quadratic forms contains the result (see, e.g., Sec;;tion 50 of [3]) that, if a real quadratic form of rank r is reduced by two nonsingular transformations to the forms

278

THE PARTITIONING OF SUMS OF SQUARES

CHAP.

8

then the number of positive c's equals the number of positive k's. In (2.7) all the coefficients of the u~ are positive, in fact being all +1, and hence aU the coefficients of the V~j must also be positive. They were already known to be ± 1; so it follows that they are all +1, and (2.6) becomes (2.9) k

We now have two sets of transformations of the quadratic form! Qj' n

k

where Qj is written as in (2.5), one leading to ! u~ and the other to ! n

i

k

Ii

i

I I·

i v~J' i

In other words, ! u~ can be linearly transformed to ! ! v~i' where k

!h =

i

11;

j

i.e., the

11 Vii'S

are linear functions of the

11

i

u/s. It will be

i

convenient at this point to change the index on V from ij, i j = 1, ... 'h' to g = 1, ... ,11. If the transformation is

= 1, ... , k,

n

Vg

=

CollI l

+

Cq2112

+ ... +

Cg"lIn

=

!

Cojll i'

g = 1, ... ,11,

(2.10)

i

then

n

We know that! g

n

v: = ! u:;

so

i

(2.12)

and the coefficients of the u's on the two sides of this equation must be equal for any i = j = j', n

! C;j = g

C~i +

c: + ... + i

C!i

= 1.

(2.13)

There are no cross products UiUj' , j :r6 j', in the left-hand side of (2. 12); so on the right-hand side the coefficient of UjUj' , j :r6 j', must be zero. CHCI }'

+

C2jC2 j'

+ ... +

C"jC"j'

= !" CojCgj' = O. g

(2.14)

SECT. 8.2

27~

PARTITIONING INTO INDEPENDENT COMPONENTS

If we multiply the first of the 11 equations (2.10) by

Cll,

the second by

C2l ,

etc., we get CnV l C2lV2

= =

C~llli

C~llli

+ Cn C1211 2 + ... + CllClnll n + C2lC22 11 2 + ... + C2 l C2n ll " (2.15)

Adding these

equations gives

11

+ C2l V2 + ... + CnlV n = IIl(C~l + C~l + ... + C~l) + 1I 2(CnC12 + C2lC22 + ... + CnlCn2) + ... + lInCcn cln + C2l C2n + ... + cnlc"n)

CUVI

(2.16) since by (2.13) the coefficient of Ul equals 1 and by (2.14) the coefficients of U2' ••• , Un are all zero. Proceeding in this manner, 'we multiply the 1/ equations (2.10) by C12, C22' etc., to get C12V l

+ C22V2 + ... + C"2V,, =

(2.17)

U2'

and so on. Thus in general, II j

=

+ C2jV2 + ... + C"jV" = !" CgjVg,

(2.18)

iii

(2.19)

cliVI

g

and so

~;U~ = j

But

±(± j

Cg;Vg)2

g

2" uJ = 2" v:; j

=

CgjCg'jVgVg'

g g'

j

=

i 1(1 g g'

CgjCg'j)VgVg,.

j

so the coefficients of the vg's on the two sides of the

g

equation

1v: = 11 (i g

are equal. Thus, for g

g

g'

(2.20)

CgjCg,,)VgVg,

,

= g',

!" cg,C = !" c!, = C!l + C!2 + ... + c!n = l. O"

,

(2.21)

j

and, for g :yl: g', n

!;

Cg,Cg'j '= CgIO'1

+ Cg2Cg'2 + ... + Cg"Cg'n =

We will now use these relations to show that the unit normal deviates.

Vg

O.

(2.22)

are independent

280

THE PARTITIONING OF SUMS OF SQUARES

CHAP.

8

From the definition of Vg in (2.10), it follows that (2.23)

c: V[UI] + ... + c: nV[u n] =

V[Vg] =

I

C!l

+ ... + c!n = 1,

(2.24)

using (2.21). The Vg are thus unit normal deviates, and it remains to show that they are independent. This we do by showing that they have zero covariance, this implying independence by the theorem of Section 12.3. The covariance is

Cov[Vg, Vg']

=

E[vgvg,] - E[vg]E[vg,]

= E[(CgIUI + ... + CgnU n)(Cg'IU I + ... + Cg'nUn)] = E[CgICg'IU~ + ... + CgnCg'nU~] + E[cross products in Ll g, Ug']' (2.25) But E[ugug,] = E[ug]E[ug,] = 0, since ug, Ug' are independent and have zero expectation. Hence

Cov[vg, Vu'] = CgICg'IE[u~]

+ ... +

CunCg'nE[u~] (2.26)

by (2.22). Since the Vg are independent unit normal deviates, each Q i = Ii

L vJi' changing the index on v back to ij from g, has the x (fj) distribution, 2

;

and is independent of the other Qi' This completes the proof of the slightly restricted form of Cochran's theorem which we stated in (2.1) and (2.2) The independence of the sample mean and variance, established at the end of Section 8.1, was required for Student's original exposition of the t test in 1908 [4]. A formal proof was given by Fisher in 1925 [5], an earlier proof by Helmert in 1876 being overlooked. Cochran's theorem was implicit in Fisher's early use of the analysis of variance [6, 7], but apparently did not receive a formal statement until 1934 [2]. Two papers by Irwin [8, 9] reviewed the mathematical basis of the analysis of variance. For a modern proof see Section 11.11 of Cramer [10].

EXERCISE 8.1. Show that Ii, defined in Section 8.1 as = 0, V[li] = (n - 1)/n, COV[li, Ii] = -1/n.

E[li]

IIi -

ii,

has the properties

REFERENCES

281

REFERENCES 1. Hald, A., Statistical Theory with Engineering Applications. New York: John Wiley and Sons, 1952. 2. Cochran, William G., "The Distribution of Quadratic Forms in a Normal System," Proceedings of the Cambridge Philosophical Society, 30 (1934), 178-91. 3. Bocher, Maxime, Introduction to Higher Algebra. New York: The Macmillan Co., 1907. 4. "Student." "The Probable Error of a Mean," Biometrika, 6 (1908), 1-25. 5. Fisher, R. A., "Applications of Student's Distribution," Metron, 5 (1925), 90-104. 6. Fisher, R. A., "The Goodness of Fit and Regression Formulae, and the Distribution of Regression Coefficients," Joumal of the Royal Statistical Society, 85 (1922), 597-612. 7. Fisher, R. A., Statistical Methods for Research Workers. 1st ed.: Edinburgh: Oliver and Boyd, 1925. 8. Irwin, J. 0., "Mathematical Theorems Involved in the Analysis of Variance," Joumal of the Royal Statistical Society, 94 (1931), 284-300. 9. Irwin, J. 0., "On the Independence of the Constituent Items in the Analysis of Variance," Supplement to the Joul"llal of the Royal Statistical Society, 1 (1934), 236-52. 10. Cramer, H., Mathematical Methods of Statistics. Princeton: Princeton University Press, 1951.

CHAPTER 9

Tests of Equality of Variances and Means

9.1. Introduction In this chapter we discuss tests as follows. Suppose that we have a sample Xu, •.• , X1n1 ' distributed normally with mean ~l and variance 0'1. In Section 9.2 we give the test for the null hypothesis O'i = O'~. In Section 9.7 we give the test for the null hypothesis that ~l = ~o. When we have a seco~d sample, X 21>" • ,X2n2 ' distributed normally with mean ~2 and variance O'~, we may wish to test the null hypothesis that O'i = O'~ (Section 9.3), or that ~l = ~2 assuming that O'i = O'~ (Section 9.7), or that ~l = ~2 without any assumption about O'i and O'~ (Section 9.8). Supposing that we have k samples, k > 2, the test for the equality of variances is given in Section 9.5 and the test for the equality of means is given in Chapter 10. Section 9.9 reviews the various tests for means in this chapter and for medians in Chapter 7.

9.2. Uses of the Sample Estimate of Variance 11

We saw in Section 8.1 that S2

= L (Xi

- x)2/(n - 1) is distributed as

i

0'2x 2(11 - 1)/(11 - 1). If we use f for the number of degrees of freedom, it follows that f S 2/0'2 is distributed as X2(J). We can obtain confidence limits for 0'2 from

(2.1)

whence on substitutingfs 2/0'2 for X2(J), taking reciprocals of the terms of the inequality, multiplying by s2f, and reversing the order of the inequality, 282

SECT.

9.2

USES OF THE SAMPLE ESTIMATE OF VARIANCE

283

we get (2.2) As discussed in Section 2.15, we might choose PI and P 2 to make the confidence interval as short as possible in some sense, but most people would prefer that the limits be symmetric in probability so that PI = 1 - P 2 = oc/2. For example, if we observe S2 = 23.394 with degrees of freedom! = 66, and want 90 per cent confidence limits, we need X~.95(66) = 86.0 and X~.05(66) = 48.3, whence the limits are 66, 23.394 X - 66 ) ( 23.394 X 86.0 48.3

=

(17.95, 31.97).

(2.3)

To derive a likelihood ratio test of the null hypothesis that 0'2 = O'~ against the alternative hypothesis that 0'2 -:;6. O'~, we need the logarithm of the likelihood function (2.3.7), 11 1 2 log L = - -log 27T0'2 - - 2 I (Xi - ~) . 2 20'

(2.4)

For ro, 0'2 is fixed as O'~, and we need the maximum likelihood estimator of~. Differentiating with respect to ~ and equating to zero gives § = x, whence L(ro) =

( 1)n/2 [1 I --2

27T0'0

exp -

-2

20'0

]

(Xi - X)2 .

(2.5)

For n, both 0'2 and ~ are allowed to vary, and we know from (2.3.13) that 8-2 = :E (Xi - X)2/11 and from (2.3.10) that § = x. Hence

The likelihood ratio A = L(ro)/L(n) can be reduced to

F )n/2 2 A = ( eF en /

(2.7)

if we define F = 8-2/0'~. The likelihood ratio A (2.7) tends to zero as F -+ 0 and as F -+ 00. Since we know the distribution of 8 2, in principle we can find the distribution of 8-2, of F, and of A. We could then make our critical

284

TESTS OF EQUALITY OF VARIANCES AND MEANS

CHAP.

9

region correspond to equal tail areas in the distribution of A.. There is no theorem that states we have to follow the likelihood ratio procedure exactly, however, and in practice we use equal tail areas in the distribution of S2. The usual critical region is thus made up of

s2


(2.8)

s2

> a2oXLa,2(f) .

(2.9)

f

o

and .'

f

For one-sided tests, rI./2 in the above formulas is replaced by For the alternative a2 > a~, the power of the test is

7T(a~) =

pr{s2

> a~X~7(f) IE[s2] = a~}

fS2 al

= Pr {- 2 =

rI..

a~ 2 2 2} >"2 XI-i!)IE[s ] = al a l

pr{l > :ixLa(f)}.

(2.10)

For example, suppose that we are making a test with rI. = 0.05, using = 19. We want the probability of rejecting the null hypothesis if aUa~ = 3/2. Substituting Xg.95(19) = 30.1,

f

7T( a~ = 3;~) =

pr{X2(19)

> ~ X30.1} =

Pr{x 2(19)

> 20.0}

f'.,J

0.39. (2.11)

Equation (2.10) can be used to calculate the number of degrees of freedom necessary to give specified power. When the alternative hypothesis is E[S2] = a~ > ag, we put (2.10) equal to 1 - (3:

7T(aD = pr{X2 > But Pr{x 2

> X~} =

:i

xLi!)}

= 1-

{3.

(2.12)

1 - {3; so (2.13)

For specified values of the ratio a~/a~, rI., and {3, this can be solved for f with the X2 table. For example, for aUa~ = 3/2, rI. = 0.05, 1 - (3 = 0.90, we find that, for f = 100, Xg.95(100) = 124.3 and X~.1O(100) = 82.4; so X~.95(100)/X~.IO = 124.3/82.4 = 1.508, very close to the required value of

SECT.

9.3

THE VARIANCE RATIO

285

1.500. Thus the solution is 1 slightly greater than 100. For large values off, we can use the approximation (1.27.13)

x~U)""" H.j2! - 1 + IIp)2.

(2.14)

substitute this in (2.13): (2.15)

and solve for I:

!,..... ! + !(1I 1-a + 1I1-P~Y. 2 2 .j«(J~/(J~) - 1 }

(2.16)

For (J. = 0.05, Ill_a = 1.645, 1 - {3 = 0.90,1I1_p = 1.282, and (JU(J~ = 3/2, we obtain/~ 102.9.

9.3. The Variance Ratio Let s~, s~ be independent sample estimates of (J2 based on/1,f2 degrees of freedom. We have seen that s~ is distributed as (J2X 2(fi)/!;' Denote by F the ratio of two such independent mean squares which are estimates of the same (J2: 2

F=~2 •

(3.1)

S2

A variance ratio will have associated with it two numbers of degrees of freedom, one for the numerator sample variance and one for the denominator sample variance. We will, when necessary, attach these in parentheses following the F: For the Fin (3.1), e.g., we write F(f1,f2)' If we substitute (J2 X2(j;)1Ji for s~ in (3.1), we obtain F = (J2lU1)/!1 = l(1)/!1

(J2lU2)/!2

X2(2)/!2 .

(3.2)

The distribution of X2 is known, and the distribution of the ratio of two X2·S. and hence of F, can be found. The cumulative form of F is tabulated in Table IV. Since X2 is never less than zero, F is distributed from 0 to 00. It can be shown that E[F] = 12/(/2 - 2), which tends to 1 as/2 tends to 00, and the mode is at 12(/1 - 2)/ftC/2 + 2). Table IV of the Appendix gives only the upper tail of the F distribution. We obtain points in the lower tail as follows. We have (3.3)

286

TESTS OF EQUALITY OF VARIANCES AND MEANS

CHAP.

9

Taking reciprocals of the inequality inside the braces will reverse the inequality sign: {S2 Pr..1 s~

>

I} = 1 -

F i - P(fi,J2)

so

pr{~ <

But

pr{~ < F p(f2,fi)} = P;

s~

1

F i - P(fi,J2)

}= P

p. ,

.

(3.4)

(3.5)

(3.6)

so, comparing the last two expressions, (3.7) For example, the 0.05 point of F for degrees of freedom 6 and 12 is found as I/Fo.95(12, 6) = 1/4.00 = 0.250. If s~, s~ are sample estimates of a~, a~ based on!1,f2 degrees of freedom, their ratio will be distributed as (3.8) We can write (3.9) Taking reciprocals and multiplying by (a~/a~)(s~/s~) gives

and hence P 2 - Pi confidence limits for a variance ratio. Here again it is customary to set Pi = 1 - P 2 = (/../2. To test the null hypothesis a~ = a~, as in the previous section we can express the likelihood ratio statistic as a function of F = s~/s~ [see exercise (9.9)]. Here again it is more convenient, and perhaps more meaningful, to use equal tail areas of the distribution of F for the critical region.

SECT.

9.3

287

THE VARIANCE RATIO.

The critical region is thus made up of (3.11) and (3.12) In practice we need never obtain the lower percentage points of the F distribution as implied by (3.11), for if s~ > s~ then we need only consider whether (3.12) is satisfied, and if s~ < s~ we can take reciprocals of (3.11) to give (3.13) and now consider whether this inequality is satisfied. For one-sided tests, (1./2 in (3.12) and (3.13) is replaced by oc. For the alternative O'~ > O'~, the power of the test for a specified value of the ratio O'~/O'~ = cp is

7T(cp) =

pr{~ > Fl_«(fl,J2)IE[s~] = O'~ = cpO'~ = cpE[S~]} (3.14) 2

2

S~//O'~ "" F(fl>f2)'

(3.15)

S2 0'2

Hence (3.14) is equivalent to

7T(cp) = pr{F(fl,J2)

> ~ F1-ifl,J2)}'

(3.16)

For example, if we are making a test of the null hypothesis O'~ = O'~ at the level of significance oc = 0.05, with /1 = 30, /2 = 10, then FO•95(30, 10) = 2.70, and the power for cp = 5 is

pr{F(30, 10) and the power for cp

> 2.70} =

=

pr{F(30, 10)

5

pr{F(10, 30)

< _5_} 2.70

rooJ

0.90,

(3.17)

1.25 is

> 2.70} = 1 1.25

rooJ

Pr{F(30, 10)

1 - 0.90

= 0.10.

< 2.16} (3.18)

288

TESTS OF EQUALITY OF VARIANCES AND MEANS

CHAP.

9

To find the number of degrees of freedom necessary to achieve a specified power, we put (3.16) equal to 1 - (3. But also

Pr{ F(fl,!2) so

cP

> F p(fl,!2)} =

= F~~(~:i» =

(3.19)

1 - (3;

F l-a(f!> 12)FI-p(f2, 11),

(3.20)

which can be solved iteratively with the F tables. For example, with ot = 0.05, 1 - (3 = 0.90, FO•95(80, 80) = 1.45 and FO. 90(80, 80) = 1.33; so their product is 1.93. The solution for cP = 2 will therefore require/I = h slightly smaller than 80, about 75. For a given sum /1 + /2' it can be shown that an equal allocation of the total number of observations to the two samples so that/I = /2 is not exactly optimum but so close that further consideration is unnecessary. 9.4. The Interrelations of Various Distributions

In (1.27.1),

x2(n)

" u~, where the was defined as L

Ui

were independent

i

unit normal deviates. When 11 = 1, X2(1) = u2 • To get corresponding probability levels for the two distributions Uand X2, we proceed as follows. Consider small areas ot in the lower and upper tails of the standardized normal distribution: These will be defined as U < ua and U > u1- a ' The sum of the areas in the two tails is 2ot, and the area between ua and u1- a is 1 - 2ot. Then (4.1) From the symmetry of the normal distribution, ua becomes

= -u1- a , and so (4.1) (4.2)

This implies Pr{1I 2

Now put 1 - 20t

= P,

so that 1 - ot Pr{1I 2

Also,

< lI~-a} =

= 1 - 2ot.

(1

+ P)/2:

< 1I~1+I')/2} =

Pr{l(1)

(4.3) Then (4.3) becomes

P.

(4.4)

< x~(1)} = P.

(4.5)

Comparing (4.4) and (4.5), and, since u2 = X2(1), x~(1)

=

1I~1+I')/2'

(4.6)

The exponents 2 in (4.6) need careful interpretation. The left-hand side, X}(l), is the P point of the distribution of X2 with 1 degree of freedom;

SECT.

9.4

THE INTERRELATIONS OF VARIOUS DISTRIBUTIONS

289

the right-hand side, U~1+P)/2' is the square of the (1 + P)/2 point of the distribution of u. For example, if P = 0.95, (1 + P)/2 = 0.975, and X~.95(1) = 3.84 = (1.96)2 =

u:.

975 •

(4.7)

X~.10(1) = 0.0158 = (0.126)2 = U~.55'

(4.8)

Or, if P = 0.10, (1

+ P)/2 =

0.55, and

In (1.27.3) we noted that E[X 2(j)] = f, and in (1.27.9) that V[X 2(j)] = 2/ It follows from the former that E[x 2(f)If] = III = 1, and from the latter that (4.9) Thus, as I tends to infinity, x2(f)1f becomes closer and closer to 1 with high probability. In (3.2) put/2 = co; the denominator X2(f2)1f2 becomes 1 and we get F(f1' co) = X2(f1) . (4.10)

11

For example, the 0.99 point of X2 with 10 degrees of freedom is 23.2: the 0.99 point of F with degrees of freedom 10 and co is 2.32, which equals

23.2110. The distribution of the ratio of a unit normal deviate to the square root of an independent X2 with I degrees of freedom, divided by f, is known as the t distribution with I degrees of freedom: (4.11) As stated earlier, x2(f)1f tends to 1 as I tends to infinity. Thus the t distribution with infinite degrees of freedom is identical with the standardized normal distribution. The ( distribution is related to the F distribution, for, if we put/1 = 1 in (3.2), the numerator becomes x2(1)/l which is just u2. Hence (4.12) Comparing (4.11) with (4.12), we see that (2 = F. Corresponding probability levels of ((j) and F(l,f) are obtained in the same way that we found the relationship for u and X2(1). The ( distribution is symmetric about

290

TESTS OF EQUALITY OF VARIANCES AND MEANS

CHAP.

9

zero; so, with t(f) replacing u and F(I,j) replacing X2(1), the argument proceeds exactly analogously to (4.1) through (4.6) and gives F p(1, f) For example, for f

=

12 and P

F o.95(1, 12)

=

t~l+P)/2(f).

(4.13)

= 0.95,

= 4.75 = (2.179)2 = 1:. 915(12).

(4.14)

9.5. A Test for the Equality of Several Variances We saw in Section 9.3 that the null hypothesis that two sample variances are estimates of a common variance can be tested with the F test. Neyman constructed a test for the k sample case by the likelihood ratio procedure. Suppose that we have observations Xij' i = I, ... , k, j = 1, ... ,l1 j , which are distributed normally with means ~i and variances O'~. We wish to test the null hypothesis (5.1)

against the alternative that the O'~ are in general different. The density function of Xi} is

(5.2) The likelihood function, defined as in (2.3.1), is L

= (27T)n/. O'~1 2

1 •••

O';k

1 ~~ k nl exp [ - - £.., £.., 2;;

(

• xii -

1= <;j

)2J

(5.3)

,

O'i

k

in which

Illi = n.

When maximizing L it is easier to deal with the

i

logarithm: log L

=-

11

-log 27T 2

111

log

0'1 -

••• -

11k

log

1

k

O'k

1

11

- - I- I(x .. 2iO'~;"

~f·

(5.4)

•

For computing L(Q), Q is the entire parameter space of both the null and the alternative hypotheses. In Q there are no restrictions on the ranges of any of the parameters other than that O'~ > O. We find the

SECT.

9.5

A TEST FOR THE EQUALITY OF SEVERAL VARIANCES

291

values of ~i and (fi which maximize L, or log L, by differentiating and equating to zero in the usual way: (5.5) (5.6) whence the maximum likelihood estimators, for

n, are (5.7)

(5.8) Substituting these in (5.3), we obtain

(5.9) For computing L(OJ), OJ is the parameter space under the null hypothesis, in which the ~i are allowed to vary but the (f~ are subject to the constraint (5.1). We obtain (5.10)

(5.11) whence the maximum likelihood estimators, for OJ, are

'" _ t, = -n1 k '" ' = x· , X,"

I.

(5.12)

j

i

(5.13)

1

k

A2 = - k'" l1 i (fi' 11 i

(5.14)

292

CHAP.

TESTS OF EQUALITY OF VARIANCES AND MEANS

9

(5.14) following on substituting (5.8) in (5.13). Substituting (5.12) and (5.13) in (5.3), we obtain L(w)

t

t LL -)2] Xi~

21 tk

= [(27ry/2 ( k I1 6Nn )nI2J-l exp [ j

111

(

k

1II

Xii -

(Xii -

i

e- 1I/2

=

k

(27Ty/2 (

t ni6~/11 )

2

xij /n

:I

(5.15)

11/2 •

Substituting (5.9) and (5.15) in (2.9.2), we obtain

(5.16)

In Q there are 2k parameters (k ~/s and k ()'~'s) and in w there are k + 1 parameters (k ~/s and ()'2). Thus from Wilks' result that - 2 log A has the x2(r - s) distribution asymptotically, where rand s are the numbers of parameters in Q and w, here k

II a?Zi,

k

-2log A = -2 log ~ = ()'

= -

k

A2

i'

62

L 11· log ()'

Li ni log 6~ + n log 62

(5.17) (5.18)

i

will have approximately a X2(k - 1) distribution. The above statistic was modified by Bartlett [1] by substituting the conventional unbiased estimators of the variances for the biased maximum likelihood estimators and substituting the degrees of freedom l1i - 1 for the l1 i • Thus Bartlett proposed using the quantity 2

k

B

s· , 1) log ~

~ = -.'(l1i -

(5.19)

S

i

where 2

S·

,

1

1Ii

_

2

= - -1~(x .. " - 'x.) .'· , ni

-

(5.20)

j

(5.21)

SECT.

9.5

A TEST FOR THE EQUALITY OF SEVERAL VARIANCES

293

A closer approximation to the X2(k - 1) distribution was obtained by using the statistic BIC, where

1 [i- i 1]. ,

C=l+

3(k - 1)

i

1- _ ni - 1

(5.22)

(ni - 1)

The final form of the test statistic is therefore

-C1 [k1 (ni -

1) log S2

i

-

1k (ni -

J

1) log s~ ,

(5.23)

i

and in this form it is commonly referred to as Bartlett's test. If the logarithms are taken to the base 10 then (5.23) is multiplied by 2.3026. Hartley [2] has shown that the approximation involved in using the X2 distribution for the distribution of BIC is poor for smalllli and has given a modification with the necessary tables. Hartley [3] has also proposed an alternative test applicable when the lli are all equal. This test uses the statistic the ratio of the largest to the smallest sample variance. A tabulation by H. A. David of the 0.95 and 0.99 points of the distribution for 12 or fewer sample variances is reproduced in the Biometrika Tables [4]. Box and Andersen [5] have concluded that this test is sensitive to departures from the assumption of normality: in fact, if the statistic is found to be lying in the critical region, it is as likely to be because the distributions are non normal as because the variances are different. As an example, the data of Table 9.1 are the final digits of determinations of the velocity of light by Michelson [6] made with six different rotating mirrors. The lower part of the table details the calculation of the s~. Table 9.2 continues the calculation of Band C. S2 is obtained from Table 9.2 as the sum of the sums of squares divided by the sum of the degrees of freedom, 22,191.317/83 = 267.365, and from a table of natural logarithms log. S2 = 5.58860. Thus B = 83 X 5.58860 - (11 X 5.54953

C= 1

+

+ ... + 15

X

4.75214) = 9.95023,

1.)

1 (0.4532 = 1.0294, 3(6 - 1) 83

and under the null hypothesis BIC = 9.95023/1.0294 = 9.666 is distributed as X2(5). For 5 degrees of freedom, the 0.90 and 0.95 points of X2 are 9.236 and 11.070; hence P f"'..J 0.08, and we cannot reject the null hypothesis at the 0.05 level of significance. It is suggestive that four of the six mirrors were made of glass and the remaining two of steel, and that there is not much variation in the s~ within each group. Forming pooled sample variances with (5.20), we get

294

TESTS OF EQUALITY OF VARIANCES AND MEANS

Table 9.1.

Table number Material Number of sides

CHAP.

Determinations of the Velocity of Light with Different Rotating Mirrors *

II Glass

III Glass

IV Glass

V Glass

VI Steel

VII Steel

8

12

16

16

12

8

47 38 29 92 41 44

9

47 62 59 44 47 41

42 36 33 0 18 57 48 15

18 45 30 27 27 66 24

3 27 48 3 27 42 69 63 30 42 60

39 66 15 7 27 37 24 15 27 42

21 33 24 39 18 63

66 27 9 6 42 12

18 12 30 30 18 48 18

9 30 27 39 27 24

30 33 12 24 57 44 30 24

21 18 33 23· 39 33 24 30

12

15

21

12

13

16

591

486

713

360

330

475

31,935

19,890

32,081

15,030

9756

15,839

8376.923

14,101.562

II; 11/

1v

Xiv

lli

Ix~

"

v

(~Xi.J/"i

29,106.750 15,746.400 24,208.048 10,800.000

11;

1 (Xi. - xi)2 v S2

2,828.250

4,143.600

7,872.952

4,230.000

1379.077

1,737.438

257.114

295.971

393.648

384.545

114.923

115.829

;

* Data from [6]. Table number

Sums of squares

II III

IV V VI VII Totals

Table 9.2 Mean Degrees of freedom squares (f;)

(s~)

log. s~

llJi

2,828.250 4,143.600 7,872.952 4,230.000 1,379.077 1,737.438

11 14 20 11 12 15

257.114 295.971 393.648 384.545 114.923 115.829

5.54953 5.69026 5.97545 5.95208 4.74426 4.75214

0.0909 0.0714 0.0500 0.0909 0.0833 0.0667

22,191.317

83

0.4532

SECT.

9.6

THE ONE-SAMPLE

t

295

TEST

estimates of 340.621 with 56 degrees of freedom for the glass mirrors and 115.426 with 27 degrees of freedom for the steel mirrors. The variance ratio 2.951 is at about the 0.9975 point of the F distribution. A two-sided test is called for, since there is no a priori reason to expect either variance to be less than the other; so the P value is about 0.005. However, since we are testing a null hypothesis to some extent suggested by the data, this P value is not legitimate, but, if we knew how to allow for this fact, probably the result would still be significant. 9.6. The One-Sample t Test Suppose that we have a sample of n independent observations Xl> ••• , X" from a normal population with mean ~ and variance 0'2. We wish to test the null hypothesis ~ = ~o against the alternative ~ ;of ~o. The parameter (12 is not known. We saw in Section 2.11 that the likelihood ratio procedure led to large absolute values of the statistic (2.11.6),

x - ~o s//ii

(6.1)

as the critical region. In Section 8.1 we saw that x was independent of S2 = :E (Xi - x)2/(n - 1), and that S2 was distributed as 0'2x 2(n - 1)/ (II - 1). In (4.11) we defined the t distribution withfdegrees of freedom as the distribution of the ratio x2(f)/f where the numerator and denominator were independent. If we write (6.1) as

u/-J

x - ~o O'//ii

(6.2)

-JS2/0'2 '

then under the null hypothesis its numerator is a unit normal deviate and independent of the denominator which has the distribution of the square root of X2(1I - 1)/(n - 1), and so (6.2) has the t(n - 1) distribution and likewise (6.1). Thus under the null hypothesis -x--, "~o , - , I ( II -

s/.jii

1) ,

and for the two-sided test against the alternative is equivalent to

I-x - r~ol > sly 11

l 1- a/ 2(1I -

(6.3) ~ ;of ~o

1).

the critical region (6.4)

296

TESTS OF EQUALITY OF VARIANCES AND MEANS

For the one-sided tests against the alternatives regions are

~

SECT.

9

> ~o, ~ < ~o, the critical (6.5)

x < ~o + fin -

1) Jii'

(6.6)

These critical regions are the analogs of those obtained when (J was assumed known. The difference is that the use of s in place of (J leads to the use of the 1(11 - 1) distribution instead of the normal distribution. Confidence limits for ~ were given in (2.16.7) as 1:

c;

-

=

-

x -

fpo

S

(6.7)

j_'

",n

As an example, we consider the data of Table 7.1 on the difference in potency of a series of lots of a pharmaceutical analyzed by two methods, A and B. These data were used in Section 7.2 to illustrate the sign test and in Section 7.8 to illustrate the Wilcoxon one-sample test, the one-sided P values for the null hypothesis that the median was zero being 0.0106 and 0.00214 by exact calculation and 0.0122 and 0.0033 by normal approximations to the exact nonparametric probabilities. To apply the t test, we need x = 2.2375 and

11-1

11-1

= (0.3)2 + ... + (6.8)2 -

(0.3 16 - 1

+ ... + 6.8)2/16 = 7.3265.

(6.8)

We thus obtain for (6.4)

- ~o I= I 2.2375 - 0 I= 2.2375 = 3 3065 Ixs/../ii ../7.3265/../16 0.67669 . , and refer this value to the 1 table with 15 degrees of freedom. The 0.995 and 0.999 levels of 1 with these degrees of freedom are 2.947 and 3.733. The two-sided P value is between 0.010 and 0.002. We can construct 95 per cent confidence limits for ~ with (6.7): /0 •975(15) = 2.131, 10 •025(15) = -2.131, s/../~ = 0.67669, and x = 2.2375; so the confidence limits are 2.2375 ± 2.131 x 0.67669 = (0.796,3.679).

SECT.

297

THE TWO-SAMPLE t TEST

9.7

9.7. The Two-Sample t Test Suppose that we have two samples, Xu, ••• ,Xln1 and X2l ' ••• , X 2112 , from normal populations with means ~l and ~2 and both with the same variance (12. We wish to test the null hypothesis that ~l = ~2: We do not assume knowledge of (12. We calCulate s~, the sample estimate of variance for the ith sample, i = 1,2. From (8.1.26), l)s~

(111 -

"1

I

=

(Xlv -

Xl )2", (12X 2(lIl -

1),

(7.1)

(X2v -

X2)2", (12l(1I2 -

1).

(7.2)

v

l)s~

(112 -

"2

I

=

v

From (1.27.16), the sum of two X 2's has a X2 distribution: 111

I

n2

(Xlv -

Xl )2

+I

v

(X 2v -

X2)2", (12l(lIl

+ 112 -

2).

(7.3)

v

If we now define

S2

2

112

II}

I

as

+I

(Xlv - Xl )2 __________

S2=~v~

(X2v - X 2)2 _______

v~

'Ili

I I ______ (X;v - X;)2 __

=~i~v

2

I

(11; -

(7.4)

1)

i

it is distributed as (1

+ 112 + 112 -

2l(lIl 111

2) 2

,

(7.5)

and it has an expected value (12. This generalizes to the case of k sample variances s~ with J; degrees of freedom:

(7.6)

Since s~ is independent of Xl and s~ is independent of x2 , S2 will be independent of Xl and x2 • A likelihood ratio argument similar to that of Section 2.11 leads to the result that

(7.7)

298

TESTS OF EQUALITY OF VARIANCES AND MEANS

CHAP.

9

where (7.8) For a two-sided test, the critical region will be large absolute values of the statistic (7.8). It is easy to show that (7.8) has the t(1I1 + 112 - 2) distribution, since it can be written as

(XI - x2) a.J1/1I 1

(~I

+

1/11 2

~2)

(7.9)

.Js2/a2

In this form the numerator is a unit normal deviate and independent of the denominator which is the square root ofaX2(1I1 + 112 - 2)/(111 + 112 - 2) variate. Therefore (7.9) has a t(111 + 112 - 2) distribution. For the null hypothesis ~I = ~2 we reject at the level of significance ex. if

XI - x I> (l-a/2(1I1 + 112 I .J1/111 + 1/112 2

(7.10)

2).

S

Confidence limits can be obtained analogously to Section 2.16 as

pr{(XI -

x2) - (P.sJ..!. + ..!. < ~I III

112

-

~2 < (XI - x2) - (PlsJ..!. + ..!.} III

= P2 -

PI'

112 (7.11)

As an illustration of the two-sample t test, we will use two series of determinations of the atomic weight of scandium by Honigschmid [7] (Table 9.3). The figures are given in units in the third decimal place; so 79 corresponds to 45.079. Certain calculations are given in Table 9.4. Table 9.3 Series A 79, 84, 108, 114, 120, 103, 122, 120, Series B 91, 103,90, 113, 108,87, 100,80,99, 54. Table 9.4 'IIi

'IIi

Series A Series B

Ie r

ni

!Xjy

Xi

!x2iv

~ !X iV n, v

8 10

850 925

106.25 92.50

92,250 88,109

90,312.50 85,562.50

~

'IIi

!(XiV - X/)2

1,937.50 2,546.50

SECT.

9.8

THE TWO-SAMPLE TEST WITH UNEQUAL VARIANCES

299

From (7.4), S2

=

1937.50 + 2546.50 8 - 1 + 10 - 1

= 280.25.

(7.12)

and for the test statistic (7.10) we have 106.25 - 92.50 I= 13.75 = 1.732 I../280.25../1/8 + 1/10 7.9408

(7.13)

which is referred to the t table with 16 degrees of freedom. The 0.90 and 0.95 points of 1(16) are 1.337 and 1.746, and so the two-sided P value is just greater than 0.10. For 95 per cent confidence limits we need 10 . 975(16) = 2.120,/0 . 025(16) = -2.120, Xl - x2 = 13.75, and S../l/l1 l + 1/11 2 = 7.9408, to get 13.75 ± 2.120 X 7.9408 = (-3.08,30.58).

9.S. The Two-Sample Test with Unequal Variances Suppose that we have two independent samples XlI>"" x ln , and x2l , ••• , x 2n , from normal distributions with means ~l and ~2 and variances C1~ and (j~. We wish to test the null hypothesis that ~l = ~2: We do not assume knowledge of (j~ or (j~, and in distinction to the previous section we do not assume that (j~ = (j~. For example, we may measure the same quantity with two different instruments and wish to test the null hypothesis that the difference between the instruments is zero: There is, in general, no reason to suppose that the measurements made by the two instruments have the same variance. In this problem both the null and the alternative hypotheses are composite. The likelihood ratio procedure for constructing a test fails in this instance. The maximum likelihood estimator for ~ in w turns out to be a solution to a cubic polynomial, and the distribution of A. involves the ratio (jU(j~, which is unknown (see Mooe! and Graybill [8]). No generally accepted solution to this problem, often called the Behrens-Fisher problem on account of a solution proposed by Behrens, exists. We will discuss a treatment by Welch [9, 10]. We saw in Section 2.12 that the means Xl> x2 are normally distributed with variances (jUlIl, (j~/112' and (Xl ../

x2) (j~/l1l

-

(~l -

+ (j~/n2

~2)

(8.1)

300

TESTS OF EQUALITY OF VARIANCES AND MEANS

is a unit normal deviate. However, we do not know the problem is to determine the distribution of

x2)

(Xl -

-

(~l

CHAP. (J'~,

and the

~2)

-

9

(8.2)

~ S~/111 + S~/112

formed by substituting sample estimates s~ with degrees of freedom 1 for the (J'~ in (8.1). We will first discuss a more general question. Suppose that s~, i = 1, ... ,k are independent mean squares with degrees of freedom J;. and expected values (J'~. Suppose that we are concerned with a linear combination of these mean squares, S2, defined as 11; -

S2

=

als~

k

+ ... + aks~ = 1; aiS~'

(8.3)

where the a; are known constants. We have E[S2]

k

]

k

k

= E [4, aiS~ = 4 , aiE[s~] = 4 , ai(J'~·

(8.4)

From (8.1.26), s~ '"" (J'~X2(J;)JJ;., and from (4.9), V[X 2({;)JJ;.] = 2JJ;., so

V[s~] = v[(J'~~(/;)J = «J'~)2V[XY)J = 2(~~)2 ,

(8.5)

and (8.6) We propose to approximate the distribution of S2 by the distribution of a mean square, say S'2, where S'2 has some number of degrees of freedom, say f', by choosing S'2 and f' so that S2 and S'2 have the same expectation and variance. Thus we put k

E[S'2] = E[S2]

= 1 ai(J'~.

(8.7)

i

From (8.5), which gives the variance of a variance estimate, V[S'2]

= (E[S'2])2}, = (~ai(J'~

J}' .

(8.8)

Equating V[S2], (8.6), and V[S'2], (8.8), and solving for f' gives

, (tk ai(J'~ )2 f =

k

(2)2·

la~.!!..i...i J;

(8.9)

SECT.

9.8

THE TWO-SAMPLE TEST WITH UNEQUAL VARIANCES

301

In practice, we do not know the values of the (j~, and so we have to substitute the sample estimates s~; so I' is estimated as

(8.10)

To revert to our particular problem, we are concerned with the linear combination of two variances s~ and s~, namely, sUn1 + s~/n2' and so our coefficients are 01 = 1/11 1,°2 = 11112. Substituting these values of the coefficients in (8.10), we get

l'

f"OooJ

[(S~/111)

+ (S~/112)]2 (1/111)2(S~)2 + (1/112)2(S~)2 111 - 1

112 - 1

(8.11) If we define (8.12) we can derive

(8.13) It is then easy to show, by differentiating 111' with respect to c and then equating to zero, that the maximum value that I' can take is /1 + /2: This occurs when s~ s~ ----''--= ------''--(8.14)

The smallest value thatl' can approach is the minimum of/1 and/2 : This occurs when either 1 - c or c approaches zero, i.e., when (s~/112)/(s~/n1) or (silI11)/(s~/112) approaches zero. As an example of this procedure we will use the data of Dean on the atomic weight of carbon, quoted in Table 7.6, for an illustration of the Wilcoxon two-sample test which gave an exact one-sided P value of 0.0317 and a normal approximation of 0.033. Here we need to compute the separate s~. Using (6.8), for the first sample A, we can subtract a constant,

302

TESTS OF EQUALITY OF VARIANCES AND MEANS

CHAP.

9

12.0000, from all observations: 2 Sl

(0.0072)2 + ...

+ (0.0077? -

(0.0072 + ... + 0.0077)2/5 5- 1

=

=

(18,381 - 16,017.80) X 10-8= 590.80 X 10-8. 4

Similarly s~ = 7460.00 X 10-8 • Also ii\ = 12.00566 and For the null hypothesis gl = g2, the test statistic (8.2) is (12.00566 - 11.99620) - 0 ..J590.80 X 10-8/5 + 7460.00 X 10-8/4

(8.15)

x2 =

= 2.124.

11.99620.

(8.16)

The approximate number of degrees of freedom of this I-like statistic is given by (8.11): First calculating

s~ = III

590.80 X 10-8 = 118.16 X 10-8, 5

S~ = 7460.00 X 112 4

(8.17)

10-8 = 1865.00 X 10-8,

(8.18)

we get

f'

r-J

(118.16 + 1865.00)2 (118.16)2/(5 - 1) + (1865.00)2/(4 - 1)

= 3.38.

(819) .

The t table (Table II of the Appendix) shows that for degrees of freedom 3 and 4 the 0.95 point is 2.353 and 2.132. For 3.38 degrees of freedom the 0.95 point will be approximately 2.27. The one-sided P value for the observed value of the statistic, 2.124, is thus greater than 0.05. If we had treated this problem as an ordinary two-sample t test we would have obtained t = 2.372 with 7 degrees of freedom corresponding to a one-sided P value of just less than 0.025, so making the assumption that the variances are from the same population makes a substantial difference in the conclusions reached. The two-sample Wilcoxon test agrees closely with the two-sample t test, but it too makes the assumption that the variances of the two populations are the same. It is interesting to note the results of falsely assuming that the two sample variances are samples of a common variance and using (7.8) instead of (8.2). The numerators of these two statistics are identical and they differ only in their denominators. If/;.= 11; - 1, and if sUs~ = F, so that s~ = Fs~, the square of the denominator of (7.8) is

11s~

+ 12S~(_1_ + _1_)

11+12

11+1

12+ 1 '

(8.20)

SECT.

9.8

THE TWO-SAMPLE TEST WITH UNEQUAL VARIANCES

303

and the square of the denominator of (8.2) is 2 _S_I_

2

+ _S_2_.

fl+ 1

f2+ 1

(8.21)

The ratio of (8.20) to (8.21) is reducible to (Ffl

+ f2)(fl + f2 + 2)

(8.22)

If we are using equal sample sizes, so that/I = /2 = f, say, then the ratio (8.22) equals (Ff + /)(2f + 2) = 1 (8.23) 2f(Ff+ F +f+ 1) no matter what the value of F. Thus, for equal sample sizes, the test statistic has the same numerical value, and the only difference would be that we would refer (7.8) to the t table with 2/ degrees offreedom whereas we would refer (8.2) to the t table with degrees offreedom given by (8.11), which in the case of/I = /2 = / becomes

f'

=/F + 1)2 F2

+1

'

(8.24)

which will always be less than 2/ except when F = 1. But, for 0.5 < F < 2, the reduction in the number of degrees of freedom does not exceed 10 per cent; for 0.333 < F < 3, not 20 per cent; so that, when the sample sizes are equal and sample variances nearly equal, there is little difference between the procedures. When the sample sizes are unequal, however, e.g., when/I» /2' (8.22) tends to F; i.e., the test statistic will be in error by a ratio approaching Since there are no theoretical bounds on F, the test statistic could be in error by any amount. We conclude, therefore, that the two-sample t test is rather sensitive to the assumption that the sample variances come from a common population, unless the sample sizes are close to equality. For markedly unequal sample sizes, and when the assumption that the sample variances are from a common population cannot be justified, a preferable procedure would be to use the methods of this section. A further conclusion is that the sample sizes should be made equal if possible. The treatment of the two-sample test with unequal variances is subject to disagreements which are still unresolved; see Fisher [11]. However important the controversy from the point of view of statistical theory, it seems that the Welch procedure described here is close enough to the truth to be used in practice without qualms. The k-sample form appears to have been first published by Satterthwaite [12].

.JF.

304

TESTS OF EQUALITY OF VARIANCES AND MEANS

CHAP.

9

9.9. A Comparison of Simple Tests for Means and Medians We have discussed seven tests for means and medians, and Table 9.5 is intended to assist in choosing the most appropriate for a particular situation. In general, we should use as powerful a test as we feel can be justified. Definitions and discussion of the power of these tests are beyond Table 9.5. Assumptions Involved in Various Tests All tests assume that the observations are independent and have continuous distributions Test

One sample: Yl>' .. , Yn

Normality and equality of variances

Normality

One sample formed as differences between paired observations: dt = xli - X2j, i = 1, ... , n

The parent populations must be normally distributed but may have different variances Symmetry about Necessary and sufficient Wilcoxon Identity of tests distributions zero of the condition is that (which implies) distribution p{xl> x 2} is symmetric, equality of so that p{a, b} = variances) p{b, a}. Median test Identity of Observations have A sufficient condition is (two samples) distributions median zero that Pj{xli' x 2 is symmetric. The Pi{ , } can be different for different i.

t

tests

Two samples: xu, •.. , x 1n1' x 2l> ... , x 2n•

a

Sign test (one sample)

Welch test

Not necessary for the d j to have identical distributions, nor that the distribution of any d j be symmetrical Normality. Inequality of variances permitted

EXERCISES

305

the scope of this book, but it is not misleading to state that when the observations are normally distributed the powers of the Wilcoxon tests are of the order of 3j7T, and the powers of the median and sign tests are of the order of 2j7T, of the corresponding t tests. An important topic in applied statistics is the question of the robustness of statistical procedures. A robust procedure is one which is affected only slightly by appreciable departures from the assumptions involved. Implicit in the concept is the feeling that conclusions in error by so much are not going to make us look foolish. For example, with a certain amount of a certain type of nonnormality the 0.05 point of the t statistic may actually be the 0.055 point, but that would not harm us appreciably. Also implicit is the feeling that in many areas of application the departures from the assumption will rarely exceed a certain degree. To discuss robustness usefully we would need to quantify these various feelings. We will not attempt this. We merely remark that, e.g., at the end of Section 9.8 we saw that in the two-sample t test when the two sample sizes are equal the test is very little affected by large departures from the assumption of equality of variances in the two populations. On the other hand, at the end of Section 9.5 we referred to Box and Andersen's conclusion [5] that Bartlett's test for the equality of k variances was very sensitive to nonnormality. These authors found, on the other hand, that the test for the comparison of k means (the analysis of variance, to be discussed in Chapter 10) was quite robust against nonnormality.

EXERCISES 9.1. Construct 95 per cent confidence limits (a) for the variance, and (b) for the mean, of Cavendish's measurements of the density of the earth given in Table 6.1. 9.2. Construct 95 per cent confidence limits for the ratio of the variances of the series A and series B measurements in Table 9.3. 9.3. Test the null hypotheses that (a) the variances are equal and (b) that the means are equal, and (c) construct 95 per cent confidence limits for the difference between the means, for the data of Table 7.4. Do this (i) on the data as it stands, (ii) on the logarithms of the observations, and (iii) on the reciprocals of the observations. In the case of (ii) and (iii), transform the confidence limits for the transformed variable back into the original scale. 9.4. Samples of very pure iron prepared by two different methods had the following melting points: A: 1493,1519,1518,1512,1512,1514,1489,1508,1508,1494 B: 1509, 1494, 1512, 1483, 1507, 1491

306

TESTS OF EQUALITY OF VARIANCES AND MEANS

CHAP.

9

(a) Test the null hypothesis that the two methods give iron with the same melting point. (b) Construct 95 per cent confidence limits for the difference A -B. Assume normality, and (i) assume the variances are the same, and (ii) do not assume that the variances are the same.

9.5. For the data of exercise (7.5), obtain the one-sided P value for the null hypothesis that the difference between the two types of gauges is zero, using (a) the data as it stands, (b) the logarithms of the data. Give an explanation for the difference between the results, and indicate which you would prefer. 9.6.1. For the data of exercise (7.2), (a) test the null hypothesis that the two samples could have come from populations with the same variance. (b) Test the null hypothesis that the means are the same, assuming normality, (i) assuming that the variances are the same and (ii) not assuming that the two variances are the same. Data for Exercise 9.7 Region A

Region B

Region C

Region D

84.0 83.5 84.0 85.0 83.1

82.4 82.4 83.4 83.3 83.1

83.2 82.8 83.4 80.2 82.7

80.2 82.9 84.6 84.2 82.8

83.5 81.7 85.4

83.3 82.4 83.3 82.6 82.0 83.2

83.0 85.0 83.0

83.0 82.9 83.4

85.0 83.7 83.6

83.1 83.5 83.6

83.1 82.5

83.3 83.8 85.1 83.1 84.2

86.7 82.6 82.4 83.4 82.7

80.6 82.3

82.9 83.7 81.5 81.9 81.7 82.5

84.1 83.0 85.8 84.0 84.2 82.2 83.6 84.9

Data from O. C. Blade, "National Motor-Gasoline Survey," Bureau of Mines Report of Investigatioll 5041.

307

EXERCISES

II. The table below gives the reciprocals of the observations of exercise (7.2). (a), (b) Perform the same tests on these data as in (a) and (b) above. (c) In

exercise (7.2), the original data were tested with the appropriate Wilcoxon test: do the same on the reciprocals. III. Compare Ia with Ha, Ib(i) with Ib(ii), Ib(i) with Hb(i), lIe with exercise (7.2). Comment on similarities and differences in the results. Control A: 0.200,0.167,0.143,0.143,0.125,0.125,0.125,0.111,0.083 Drug B: 0.143,0.125,0.125,0.125,0.111,0.111,0.083,0.077, 0.071, 0.059 9.7. The data on p. 306 are the results of octane determinations on samples of gasoline obtained in four regions of the northeastern United States in the summer of 1953. Test the null hypothesis that the variability in octane number is the same for aU four regions. Some calculations which may be useful are summarized below. Region ni

B

16

13

18

22

62.0

37.0

58.0

66.2

258.06

107.98

215.86

232.24

C

"I

LXiv

D

A

"I

LX~v v

A constant 80 has been subtracted from all observations before making these calculations. 9.S. Rosa and Dorsey [see exercise (10.3)] measured the ratio of the electromagnetic to the electrostatic unit of electricity with great precision. In a long series of observations they on occasion disassembled their apparatus, cleaned it, and reassembled it. The variances of the groups of observations, multiplied by 108 , and numbers of observations in each group, were as follows.

Group

Number of observations II

2 3 4 5

8 6 24 15

Variance 1.5636 1.1250 3.7666 4.1721 4.2666

Test the null hypothesis that these sample variances could come from a common population.

308

TESTS OF EQUALITY OF VARIANCES AND MEANS

CHAP.

9

9.9. Let s~, s~ be independent unbiased estimators of G~, G~ with nl - I, n2 - 1 degrees of freedom respectively. Show that the likelihood ratio statistic for a test of the null hypothesis G~ = G~ can be written as

A=

(nl

+ n2)
1+1l 2

1I~1/2 1I~./2

1/2[

1+

(n2 - I)S~J-nl/2[ (nl - I)S~J-"2/2 I+~-,..:.-; (n l - I)s~ (n2 - l)s~

Thus Ais sml;lll for large values of sVs~ and also for small values of sf/sf. (Hint: The derivation of A in (5.16) for comparing k variances is valid for k = 2.) 9.10. With reference to Section 9.7 on the two-sample t test, show that, in 0.

~l = Xl> ~2 = x 2, U = n__+I_[~ (Xlv n2 v

- x)2

l

and in

+ ~ (x2v v

- X2)2J '

£0,

~ = _I_(~XIV + ~X2V)' 111 + 112 v v '2 = - 1 - [~ - 2 k. (Xlv - Xl) nl + 112 v

G

~ (X2v + k. v

- - X- 2) 2J - X- 2) 2 + -nln2 - - (Xl nl + n2

,

whence Ais as determined by equations (7.7) and (7.8).

REFERENCES I. Bartlett, M. S., "Properties of Sufficiency and Statistical Tests," p,.oceedings of the Royal Society of London, A, 160 (1937), 268-82. 2. Hartley, H. 0., "Testing the Homogeneity of a Set of Variances," Biomet,.ika, 31 (1940),249-55. 3. Hartley, H. 0., "The Maximum F Ratio as a Short-Cut Test for Heterogeneity of Variance," Biomet,.ika, 37 (1950), 308-312. 4. Pearson, E. S., and H. O. Hartley (eds.), Biomet,.ika Tables fO,. Statisticians, Vol. 1. London: Cambridge University Press, 1954. 5. Box, G. E. P., and S. L. Andersen, "Permutation Theory in the Derivation of Robust Criteria and the Study of the Departures from Assumption," Journal of the Royal Statistical Society, B (Methodological), 17 (1955), 1-26. 6. Michelson, A. A., "Measurement of the Velocity of Light between Mount Wilson and Mount San Antonio," Ast,.ophysical Journal, LXV (1927), 1-14. 7. Honigschmid, 0., "Neure Atomgewichtsbestimmungen," Zeitsch,.ift fU,. Elect,.ochemie, 25 (1919), 91-96. 8. Mood, Alexander M., and Franklin A. Graybill, IIIf,.oduction to the Theo,.y of Statistics, 2nd edition. New York: McGraw-Hill Book Co., 1963. 9. Welch, B. L., "The Significance of the Difference Between Two Means When the Population Variances are Unequal," Biometrika, 29 (1937), 350--62. 10. Welch, B. L., "The Generalization of 'Student's' Problem when Several Different Population Variances are Involved," Biometrika, 34 (1947), 28-35. II. Fisher, R. A., "Comment on the Notes by Neyman, Bartlett, and Welch in This Journal," Journal of the Royal Statistical Society, B, 19 (1957), 179. 12. Satterthwaite, F. E., "An Approximate Distribution of Estimates of Variance Components," Biomet,.ics Bul/etin, 2 (1946), 1l0-14.

CHAPTER

10

One-Way Analysis of Variance

10.1. Introduction: Models I and II We have seen in Section 9.7 that two sample means can be compared with the two-sample t test. The generalization of this problem to k means, k> 2, brings us to the body of techniques known as the analysis of variance. In this chapter we shall be concerned with the case in which the data are subject to only one dimension of classification. There are two somewhat different situations. In the first, known as model I, the individuals in the sample represent the particular individuals in which we are interested. In the second, known as model II, we have no particular interest in the particular individuals that happen to be in the sample. Our concern is to estimate the variance of the infinite population from which our sample has been randomly selected. As an illustration of the difference between models I and II, consider a machine tool manufacturer producing a standard model in quantity who is interested in some variable associated with the quality of the items produced by the tool. A customer may purchase k tools, and the manufacturer may supply him with a sample taken at random from his current production. The measurements are made on a sample of the items produced by these k tools, and the data analyzed by both the customer and the manufacturer. From the point of view of the customer, he is concerned solely with these k tools sitting on his floor, and he uses a model I approach. From the point of view of the manufacturer, these k tools are merely a random sample, and he is only concerned with using these measurements to estimate the variance of the population of tools; therefore he uses a model II approach. The problem of whether any particular situation is model I or model II is often clarified by asking the question, "If we were to repeat the experiment, would we have the same individuals or not?" Clearly in the model I 309

310

ONE-WAY ANALYSIS OF VARIANCE

CHAP.

10

case we would; in the model II case, the probability of getting the identical sample, or even any single individual, a second time is zero. Model I can be regarded as the case where the sample consists of all in the population, and model II as the case where the interest is in the infinite population from which the sample came. In Section 14.8 we shall consider the intermediate case in which the population is finite; there, by letting the population size decrease to the sample size or increase to infinity, we will get models I and II as special cases.

10.2. One-Way Analysis of Variance: Model I Suppose that we have k groups of independent observations Xu. ••• , x 2n ., ••• , X kh ••• ,Xk"k from normally distributed populations with means ~l" •• , ~k' all with the same variance 0'2. Thus the model is Xiv = ~i + Ziv; i = 1, ... ,le, ')I = 1, ... , ni' (2.1) where the ~i are fixed constants and the Ziv are independent random normal deviates with zero mean and variance 0'2. It is convenient to reparameterize the model (2.1). We define ~ as the weighted average of the ~i' XI"l' X 21 , ••• ,

~ = (~nir(~ ni~i)' and then introduce new parameters 1]i

=

1]i

(2.2)

defined as

~i -

~.

(2.3)

The model (2.1) can now be written as Xiv

The weighted sum of the k

1]i

= ~ + 1]i + Ziv'

(2.4)

is zero:

k

k

!i n i1]i = ! ni(~i - ~) = ! i i

k

nJi -

~

!i

IIi

= O.

(2.5)

It is often helpful, but not always essential, when averaging over a

suffix to replace that suffix by a dot. Thus Xi.

=

"I

(l/n i )

!

Xiv'

We define

v

X"

as the weighted average of the Xi., or what comes to the same thing, "I

as the grand average of all the

Xi'"

since

nix i . =

!

Xiv:

v

(2.6)

SECT.

10.2

ONE-WAY ANALYSIS OF VARIANCE:

311

MODEL I

In consequence of these definitions, k

Z ni(x i i. -

k

k

Z nixi . - X.. Z lIi = i i

X.J =

O.

(2.7)

Xii,

(2.8)

From (8.1.26), for each group we can write fit

Z (Xiv -

lli

+ Z (XiV -

~i)2 = ni(xi . - ~i)2

v

V

where the sums of squares are distributed as 0'2X2 with degrees of freedom

nt, 1, and n; - 1 respectively, and the two terms on the right-hand side are independent. Now sum (2.8) over all k groups: k

i

k

ni

Z Z (XiV -

~i)2

= Z n;(x;. -

v

i

k

ni

+ Z Z (Xiv i v ~i)2

x;Y-

(2.9)

We saw in (1.27.16) that the sum of a number of X2(fi) is itself distributed as X2 (

*/l),

so the

~ums of :quares in (2.9~ are distributed as 0'2X2 with

Z

(L lli) - k respectively. Ifwe

Z

degrees of freedom 11;, k, (n; - 1) = define i i i

(2.10)

th;n as discussed in (9.7.6) it will be distributed as 0'2X2(fni -

(t

k) and have expected value

lli -

side of (2.9) involves Xi. Xi. - ~i

~i'

= (Xi. = (Xi.

0'2.

which we can write as

- X.. - 1]i)

When we square this and sum over i and by (2.7) and (2.5):

•

The first term on the right-hand

- X.,) - (~i - ~)

k 2(x .. - ~) ~ ll i(Xi . - X.. - 1]i)

k) /

= 2(x .. = O.

+ (X .. -

+ (X .. 11

~)

~).

(2.11)

the cross product term is zero

- ~) [k ~ Il;(x;. - x. J

- ~k Il;'Y};] • • (2.12)

Therefore k

k

k

L n;(xi • - ~i)2 = Z lI;(Xi . - X.. - 1]i)2 + (x .. - ~)2 Z 11;. i i i

(2.13)

312

ONE-WAY ANALYSIS OF VARIANCE

CHAP.

10

In (2.9) we saw that the left-hand side is distributed as a2X2(k). We have partitioned this sum of squares in two components. The first involves k variables Xi. - X .. subject to the restriction (2.7), so it will have k - 1 degrees of freedom. The second involves the single variable x .. , on which there is no restriction, so it will have one degree offreedom. By Cochran's theorem of Section 8.2 these two sums of squares will be distributed as a2x2 and be independent. If we define ~ 1 k _ _ 2 S2 = - - '" n.(x. - x - '11.) k - 1f'" .. ·It'

(2.14)

then it will be distributed as a2x2(k - l)/(k - 1), have expected value a2, and be independent of s~. Hence the ratio s~2M has the F distribution k

with degrees of freedom k - 1, ~ ". - k. Now define i

(2.15)

To find its expected value, consider the expected value of the numerator of S~2, noting that E[x •. - x.,] = ~. - ~ = 1].:

E[* lllx•. - x.. - 1]i)2J = E [ ~k ll i(Xi . •

YJ+ 4 n.1]~ - 24 ll i1]iE [xi . k

X.

k

••

X.

J

k

= E[(k - l)s~] - ~ lli1]~.

(2.16)

i

The left-hand side equals (k - 1)a2 , whence it follows that 2 2 E[S2] = a +

1

k

k - 1

i

2

- - ~ 1l.17i'

(2.17)

If the null hypothesis that ~i = ~ for all i is true, then 17i = 0 for all i, s~ = S~2, and so s~ will have the distribution a2x2(k - 1)/(k - 1), and k ~2 '"'-' F ( k - 1, ~lli .

S2 1

-

k) .

(2.18)

•

If the null hypothesis is false, so that some 17i ¥= 0, then E[s~] > a 2, and the ratio s~/s~ will not have the F distribution but instead a distribution displaced upwards in the direction of larger values of s~M. The critical region is therefore large values of s~M.

SECT.

10.2

ONE-WAY ANALYSIS OF VARIANCE:

313

MODEL I

These results are conventionally and conveniently summarized in an analysis of variance table (Table 10.1). Table 10.1 Source of variance

Sums of squares

Between groups

L ni(xi. - x]

Within groups

LL (xiv - XiY (*n i) i v

Total

LL(xiv - x.Y (*n i) i v

Degrees of Mean freedom squares

E[M. S.]

k

k -1

2 82

k "1

k

-k

82 1

"1

-1

An alternative model is sometimes preferable. In (2.2) we defined ~ as the average of the ~i weighted by the ll i • The l1i may be quite arbitrary numbers, and it may be more satisfactory to define a mean ~' as the unweighted mean 1k (2.19) ~'= - L~i·

k

Corresponding to the

17i

i

of (2.3) we define = ~i

Ti

Ti

as

r.

-

(2.10)

Thus the original model (2.1) is written as

Xiv =

~'

+ + Ziv.

(2.21)

Tj

The two models are related by the equation ~i

We have, substituting

~i

= ~ + 1]i =

~'

+ Ti·

(2.22)

from (2.22) in (2.2):

(2.23) Therefore

1]i

=

~i - ~ =

(f

+ T;)

k

L II·T·

-

(

k ) ~'+ ~

Li IIi

L lIiT i

= Ti

-7--. L IIi i

(2.24)

314

ONE-WAY ANALYSIS OF VARIANCE

Substituting this for 1Ji in (2.17), the alternative expression for

CHAP.

E[s~]

10

is

(2.25)

If the Ti are zero for all i, then the second term on the right-hand side is zero and E[s~] = (J'2 as before. Of course, if the ll i are all equal, then the two models are identical. 'We may wish to construct a confidence interval for the difference between two population means, ~i - ~i" The conventional procedure is to note that (2.26) (2.27) whence ( X. •.

x.,) - (~.• - ~.,) •. • "-'t 1)~

1 s (-+1 IIi

(Ik n.-k)

(2.28)

i"

IIi'

and 1 - oc confidence limits for

~i -

~i'

are (2.29)

This differs from the interval derived from a two-sample t test, (9.7.11), by the use of the S1 from Table 10.1, which is an estimate pooled over all k groups rather than just the i and i'th. Clearly, if there is reason to doubt the assumption of this section that (J' is constant over all k groups, then (9.7.11) should be used (and if we cannot assume that (J' is constant even over the i and i'th groups, we should use the methods of Section 9.8). As an example of one-way Model I analysis of variance, Table 10.2 gives the units in the third decimal place of determinations by Heyl [1] of the gravitational constant G; e.g., 83 corresponds to an observation of 6.683. Balls of three different materials were used. While the sums of squares in Table 10.1 can be calculated in the forms given there, it is usually more satisfactory to use identities that involve totals rather than means. We need

(~k )-1(k~ I"I IIi

•

•

v

Xiv

)2 = (1159)2 - = 83,955.062, 16

(2.30)

SECT.

10.2

ONE-WAY ANALYSIS OF VARIANCE:

315

MODEL I

Table 10.2 Gold

Platinum

Glass

83 81 76 78 79 72

61 61 67 67 64

78 71 75 72 74

6

5

5

469

320

370

Sums

k

ni

16 = ~ 11, k 'Hi

"j

~;XiV "i

~;xrv v

Ie

r

-n. ~x.tv ,

V

and then

36,735

20,516

36,660.167

k

~ (£., IIi Xi. i

1,159 = ~~XiV

X.. )2

k

+k

=

= ~k IIi

(.2

-2

v

..

Ie r

,n, "

k

2~ X .. £.,

l1 i -Xi •

(i 1 i

_v_

+ ~k IIi

IIi

i

i

Xiv)2

i

k --: ~ Xiv 84,520.167 = ~

27,380.000

~ -2 ~ -2 £., l1 i Xi . £., IIi X .. i i

-

v "I

84661 = ~~x~ ,

27,410

20,480.000

i k

XiV)

.:....-~:-'--~

£., IIi i

1 (i_,_v_

XiV)

k

~11.

£., i

k ~

•

(1_v_

XiV)

IIi

£., IIi i

k 1 (nl Xiv )2 - (k~ IIi )-1 (~k ~"I Xiv )2

= ~- ~ i l1i

=

v i i

84,520.167 - 83,955.062

By similar manipulations, ~.

"j

~ ~ (Xiv i

v

-

X;,)2

k

"j

i

v

= ~ ~ X:v -

= 565.105.

(ni Xiv )2

k 1 ~- ~

11;

i

k "I

(k )-l(k~~XiV "I )2

~~(Xiv-XJ2=~~X:v- ~lIi "j

1V

tV

t

(2.32) (2.33)

v

= 84,661 - 84,502.167 = 140.833,

k

(2.31)

v

(2.34) (2.35

,tV

= 84,661 - 83,955.062 = 705.938.

(2.36)

ONE-WAY ANALYSIS OF VA~IANCE

316

CHAP.

10

We assemble these results in Table 10.3. Table 10.3 Sums of Degrees of squares freedom

Mean squares

Materials Within materials

565.105 140.833

2 13

282.553 10.833

Total

705.938

15

Source of variance

The test of the null hypothesis that the ~i are all equal is given by the variance ratio 282.553/10.833 = 26.08 with degrees of freedom 2 and 13; this is overwhelmingly significant as the 0.999 point for these degrees of freedom is 12.31. 10.3. The Problem of Multiple Comparisons An F test as performed in the previous section may reject the null hypothesis that the ~i are all equal, but it does not tell us which ~i are significantly different from which. A single prechosen pair may be compared by an ordinary two-sample t test, either using a within-group mean square calculated only from the two groups in question, or using the within-group mean square from all the groups, and confidence limits for the difference constructed in the usual way. This is valid only for one prechosen comparison; if we start making comparisons suggested by the data, and several of them, the significance level becomes hopelessly incorrect. Several procedures have been proposed for this problem. We will discuss those offered by Scheffe [2] and Tukey [3]. Scheffe's system is more general than simple comparison between pairs of means. Suppose that we have estimates Xi of true means ~i' with variances (12/ni estimated as s2/ni' S2 being estimated with f degrees of freedom. We are interested in contrasts, defined as (3.1) k

where

L Ci = O.

The contrast () is estimated as k

H

= 1ci xi •

(3.2)

i

Assuming that the Xi are independent, V[H] =

k

2

k

2

i

IIi

i

IIi

1 C~ • ~ = (121 Ci ,

(3.3)

SECT.

10.3

317

THE PROBLEM OF MULTIPLE COMPARISONS

and the estimated variance of H is

V[H]

k

2

i

ni

= S2 Z £! .

Scheffe's result is that we can construct 1 imaginable contrasts (),

Pr{H - S.../V[H] where

(3.4) (l

confidence limits for all

< () < H + S.../V[H]) = 1 -

S2 = (k - l)Fl_aCk - 1,J).

(l,

(3.5) (3.6)

In this formula, k is the number of means which might be entering into the contrast, rather than the number of means actually entering into any particular contrast. If, for each experiment we perform, we construct confidence limits according to (3.5), then, in a fraction 1 - (l of these . experiments, all the confidence statements will be correct: In a fraction (l, one or more of the statements will be incorrect. Tukey's procedure is somewhat similar but requires that the Xi have equal variances. We construct confidence limits for each contrast as

Pr{H - Ts

< () < H + Ts} = 1 -

(l,

(3.7)

where S2 is the estimated variance of individual observations, as before, and T is defined as 1k 1 (3.8) T= - Z Icil·q· ---=-, 2 i .../11 where q is the 1 - IX point of the studentized range for a sample of k and degrees of freedom (Table IX of the Appendix). The function q is the distribution of (xmax - xmin)/s, where s is estimated with f degrees of freedom. These two procedures have somewhat different properties. In the first place, Scheffe's can be used when the lli are different; Tukey's requires that the lli be all equal to 11. Second, Tukey's gives somewhat shorter confidence intervals for simple differences, e.g., a contrast formed by coefficients ci = -1, 1, 0, ... , 0, whereas Scheffe's gives shorter confidence intervals for more complex contrasts, e.g., the difference between the sum of the first three means minus three times the mean of the fourth, which would be given by coefficients Ci = 1,1, 1, -3,0, ... ,0. To illustrate the use of these methods we will use the data of Table 10.2. So that the results of the different methods will be comparable, and since Tukey's method requires equaill i , we will omit the last observation from the gold balls, so that the three means will be 79.4, 64.0, and 74.00, respectively, but we will use the estimate of variance of a single observation unchanged at 10.833 with 13 degrees of freedom.

f

318

ONE-WAY ANALYSIS OF VARIANCE

CHAP.

10

Suppose that we wish to construct confidence limits for the differences between the metal balls and the glass ball, i.e., the contrast () = ~l

+ ~2 -

U a,

(3.9)

which will be defined by the coefficients Ci taking the values Ca = -2. Then

H= 1 V[H] S2

=

X

79.4

+1 X

10.833 [1 2 + 12 5

= (3

64.0 - 2

X

+ (_2)2] =

- 1)Fo.95(2, 13)

=

2

X

Cl

= 1, = 1, C2

74.0 = -4.6,

(3.10)

13.00,

(3.11)

3.81

= 7.62,

S../V[H] = "/7.62 "/13.00 = 9.951;

(3.12) (3.13)

so the Scheffe limits are -4.6 ± 9.95. If this were the only contrast for which we wanted confidence limits, and the choice had been made before we saw the data, it would be legitimate to use , 0 . 975(13) = 2.160 in place of S, which would give limits -46. ± 7.8. The Tukey limits involve q for three means and a variance estimated on 13 degrees of freedom. From Table IX, q = 3.73 for these values of the parameters. We need T=

1

1

k

1

-1Ic;l·q· r = - (Ill + 111 + 1-21) 2 i vn 2

X

3.73

X

1

,_

v5

= 3.336; (3.14)

so Ts = 3.336 X "/10.833 = 10.98, and the confidence limits are -4.6 ± 11.0. A third type of multiple confidence intervals was developed by Dunnett [5] for the special but important case where we have tested 1 + k treatments, of which one is a control and we wish to compare the other k with the control. These limits are easy to construct, being ±ds../l/no + 1/lIi where 110 is the number of observations on the control, IIi on the treatments, and d is a parameter analogous to t tabulated in his paper.

10.4. One-Way Analysis of Variance: Model II In model I analysis of variance we defined the model in (2.1) as Xiv = ~i + Ziv' and in (2.20) we defined Ti = ~i - ~'; so the model was (4.1) k

where 1

Ti

=

0 and z;v,...... N(O, (12). The Ti were fixed constants. A model

i

II analysis is based on the model (4.2)

SECT.

10.4

ONE-WAY ANALYSIS OF VARIANCE:

319

MODEL II

where the Yi""'" N(O, ( 2) and the Ziv""'" N(O, 0'2). Regarding the relevant populations as infinite, such a model could correspond to the situation where we take a number of sacks of wool from a large consignment and the sack means are distributed about a mean ~ with variance w 2 : We then take a sample from the sack which is distributed about the sack mean with variance 0'2. Both Yi and Ziv are random samples from infinite populations, and our only interest in the particular Yi and Ziv we happen to get in the sample is insofar as they enable us to estimate ~, w2 and 0'2. The parameters w 2 and 0'2 are known as components of variance. Since the model is so similar to model I, much of the analysis proceeds along similar lines. The total sum of squares about the grand mean is split up as before into two components: k

'111

k

k

v

i

'11/

L L (Xiv - XJ2 = Lni(Xi. - XJ2 + L L (Xiv - xil, i

i

(4.3)

v

where the two terms on the right-hand side are the numerators of s~ (2.15) and s~ (2.10), with degrees of freedom k - 1 and

k

L ni

-

k. The mean

i

square s~ as before has expected value 0'2, but the change in the model affects the expected value and the distribution of s~. To find this we proceed as follows:

To evaluate this we need E[x~.1 and E[x~]. The averages Xi. and X.. are formed by averaging the model (4.2) first over 11 to give Xi.; we then average Xi. over i to get X.. : 1 '111 ~=~+~+-L~ ~~ lIi v k

k

k

nl

L l1 iXi. L L Ziv L l1 iYi X __i _ _ _ I:+_i__ +_i_v_ •• -

k

-~

L l1i i

k

(4.6)

k'

Llli

L

l1i i i

We see that E[xd = ~ and likewise E[x..l =~. From the definition of variance, V[xd = E[xn - (E[xi.D2, we have

E[xl]

=

V[XiJ

1 + (E[XiJ? = V[Yi] + -2 V l1i

['111

L Ziv v

]

+ ~2 = w2 + -0'2 + ~2. lIi

(4.7)

320

ONE-WAY ANALYSIS OF VARIANCE

CHAP.

10

Similarly, E[x.~] = V[x.J

=

+ (E[X.J)2

[(~I1;)]-l{V[~lIiYiJ + V[~~Zi.J} + e

= [(~ 11;)]-1 (W2 ~ 11~ + (12 ~ 11;) + ~2.

(4.8)

We now substitute (4.7) and (4.8) in (4.4):

E[i •

I1;(Xi. -

x.lJ = i •

- (tk

IIi

l1i(W 2 +

(12 11;

+ ~2)

){[(kt 11;)2J-1(w tk 11 ~ + (12 t~') } 11; + ~2 2

(4.9)

so the expected value of s~ is

(4.10)

In the case when the

11; are all equal, say to 11, then (4.10) becomes (4.11)

It is apparent that model I and II one-way analyses of variance are very similar. The partitioning of the total sum of squares and the arithmetic are identical, and the only difference that emerges is the change in the expectation and distribution of s~. In the case of equall1i' the change in k

the expectation of s~ is the substitution of w 2 for ~ (~; - ~)2/(k - I). For equall1j = 11, (4.5) is i (4.12) and so Xi. has expectation E[xd V[x;.]

=

=

~

and variance 2

V[Yi]

+ V[z;.] = w 2 + !!.- . 11

(4.13)

SECT.

10.5

321

INTERPRETATION OF A MODEL II ANALYSIS

Also, since Yi and Zi are normally distributed, Xi. is normally distributed. k

Therefore

.L (Xi. -

x.Y/(k - 1) is an ordinary sample estimate of vari-

i

ance, where the variance is w2 + (12/11 , and will have the distribution (w 2 + (12/I1)X2(k - 1)/(k - 1). It follows that s~ has the distribution (0'2 + I1(2)X2(k - 1)/(k - 1).

10.5. Interpretation of a Model II One-Way Analysis of Variance

= 0, s~M will be distributed as Fwith

Under the null hypothesis that w 2 k

.L

degrees of freedom k - 1 and 11; - k; so this is the test for this null hypothesis. ; We have seen that E[x .. ] = so X.. is an obvious estimator of The expectation of x~ (4.8), in the case 11; = 11, reduces to

e;

IlW

e.

2+

2 (1

+ e2,

(5.1)

k/1

and so the variance of X.. in the case of equal

E[x..J .JV[x..J

X.. -

(5.2)

kll

=

X.. -

.J((12

e

"-' N(O, 1).

+ II(2)/kn + I1W 2 , namely, s~, based on k

Substituting the sample estimate of (12 degrees of freedom, gives X -

is

J + /1W 2

V[x..J Hence

11;

e"-' t(k -

.~

1),

(5.3)

- 1 (5.4)

.Js:/kn

e

which will give confidence limits for as in (9.6.7). The variance of the Ziv' (12, is estimated by s~ and w 2 is estimated as A2

w

s~ - s~ = ---.

(5.5)

II

Confidence limits for (12 can be obtained as in (9.2.2). Confidence limits for w2 are considerably more troublesome. We can use either a large sample approximation or a somewhat more exact approximation. The large sample treatment applies (9.8.5), V[S2] = 2((12)21f, to s~ and s~: (5.6)

322

ONE-WAY ANALYSIS OF VARIANCE

10

CHAP.

and so

Substituting the sample estimates s~ and s~ for l1C0 2

+ ()'2 and ()'2, we get (5.8)

We then assume w2 to be normally distributed with this variance, and obtain confidence limits in the usual way. This approximation is poor except for large k. Probably k should be greater than 50 before the approximation ceases to be crude. The better approximation is due to Bross [6], and the reader is referred to his article for the derivation of what strictly are known as fiducial limits. At the present level of sophistication we will regard these limits as effectively the same as confidence limits. Bross's results are that 1 - ex approximate confidence limits for w2 are !::.w 2, Lw 2, where

L=

F/Fa/2 - 1 F~/2F/Fa/2 - 1

(5.9)

where F is the observed variance ratio s~M; F1- a/2 and Fa/ 2 have degrees of freedom the same as F, namely k - 1 and k(l1 - 1); and FLa/ 2 and F:/ 2 have degrees of freedom k - 1 and 00. Frequently the observed F will be less than F 1 - a / 2 , which will make !::. as given in (5.9) negative: Since a component of variance cannot be negative, it is not easy to interpret a negative value for the confidence limit, and in these circumstances!::. may be taken as zero. Procedures for this situation are discussed in Section 7.2 of Scheffe [7]. An alternative technique is given by Williams [8]. Although we have only approximations for confidence limits for co 2 , we are able to obtain exact confidence limits for the ratio C0 2 /(),2. In the derivation of (9.3.10), s~ and s~ were sample estimates of ()'~ and ()'~ with};. andf2 degrees offreedom. Here s~ is an estimate of ()'2 with the distribution ()'2X 2(k(11 - 1»/k(l1 - 1) and s~ is an estimate of ()'2 + l1C0 2 with the distribution «()'2 + l1C( 2)X 2(k - 1)/(k - 1). The arguments o' (9.3.10) are valid here; so, substituting s~ here for the s~ in (9.3.10) and vice versa, we have 2 P r {~2 S1

1

F p.(k - 1, k(1l - 1»

< ()'2 +2nco 2 < ~22 F1-p.(k(/ I ()'

}

- 1), k - 1)

S1

(5.10)

SECT.

10.5

INTERPRETATION OF A MODEL II ANALYSIS

323

which rearranges to Pr {-1 [S~ 11

s~ F pik

1 - 1] - 1, k(11 - 1»

(J} <-

()'2

< ![~Fl_Pl(k(11 -1), k -1) -1J} s~

11

(5.11) If the observed variance ratio s~M is less than FP 2 ' then the lower confidence limit will be negative. Since both co 2 and (j2 can never be negative, their ratio can never be negative, and zero might be substituted for the negative confidence limit. Scheffe (Section 7.2 of [7]) suggests nevertheless that the unmodified results of (5.11) give a more informative picture. It is interesting to note what can happen if we are presented with kn observations, without being informed that they are really k groups of n observations, and asked to calculate confidence limits for the mean. We would start by computing as S2, k.n

S2 =

L (XiV -

(5.12)

xJ2/(kl1 - 1).

i,v

But

(5.13) so E[S2] -_ (k - 1)«(j2

+ I1C(2) + k(11 kl1 - 1

1)(j2 _ l1(k - 1) 2 + 2 co (j, k1l - 1

(5.14)

and the incorrectly estimated variance of x" would have expected value Vinc[X ] = [11(k - 1)/(kl1 - 1)]w2

..

kl1

+ (j2.

(5.15)

But we saw in (5.2) that the correct variance of x.. , say Vcor [x..1, equals

v.cor [X-;] -_ ..

+

()'2 I1W 2 -""""'----

kl1

(5.16)

The ratio of incorrect to correct variances is the ratio of(5.15) to (5.16), which can be written in the form (5.17)

324

ONE-WAY ANALYSIS OF VARIANCE

CHAP.

10

The magnitude of the error is going to depend on the values of the various parameters and particularly on (12/ W 2. If (12/ W 2 -+ 00, then the ratio (5.17) tends to I, but, if (12/ w 2 -+ 0 and II is large, then the ratio tends to (k - I)/(nk - 1), which for large k will tend to I/n. Thus a very serious error may be made by assuming that we have kn independent observations when in point of fact we do not. In this section we have so far made the simplification that the lli were all equal to n. As an illustration of the difficulties that arise when this is not the case we will compute V[x..J. From the expression (4.6) for x.. , we get

V[x..1 =

[(tllin-1tll~V[Yi] + [(t ll ;n-1*II;V[ZiV] k

L II~

=

(~

L IIi

)2

w2

+

(k )-1

til;

(5.18)

(12.

i

Whereas, in the case of equal 11;, V[x ..1 was found in (5.2) to be ((12 + n(2)/kn, where we had a mean square, s~, for estimating (12 + nw 2, here we do not have a mean square to estimate (5.18). An approximate procedure would be to insert the appropriate estimators of w 2 and (12, involving s~ and s~, and reduce the resulting expression to a linear combination of these two mean squares, and then estimate the approximate degrees of freedom of this linear combination by (9.8.10). This is clearly an awkward business of unknown accuracy. All this trouble can be avoided by the use of equal 11;. An alternative approach to the estimation of ~ is to use the results of Section 2.4 on the formation of weighted means of minimum variance. There we saw that x defined as L wiX;/L Wi had minimum variance, actually I/L W;, if the weights Wi were chosen as I/(1~. Here, from (4.5), (5.19) and so

V[x;.] = V[yJ

"I

V[z. ]

(12

v

IIi

IIi

+ L~ = w 2 + -;

so we might use weights based on the estimates of w2 and A

Wi

=

(

A2 + -8-2)-1.

W

IIi

(5.20) (12:

(5.21)

For the construction of approximate confidence limits for ~, Cochran [9] suggests using x ± V[x] where t is given (k - 1) degrees of freedom.

tJ

SECT.

10.6

325

EXAMPLE WITH EQUAL GROUP SIZES

Cochran, in the same paper [9], also discusses how to handle the stilI more awkward situation where in the model (5.22)

the Ziv are distributed normally with mean zero and variances ()'~, i.e., where the within-group component of variance varies from group to group. This situation can arise, for example, when different laboratories determine some quantity. There is no a priori reason why the variance of the measurements within a laboratory should be the same from laboratory to laboratory. The reader is referred to Cochran's paper for details. 10.6. An Example of a Model II One-Way Analysis of Variance with Equal Group Sizes Table lOA gives the measurements of a certain physical property on 88 successive sub-batches of a plastic-like material. The raw material goes through a number of stages of processing in batches, which then are Table 10.4 2

3

4

5

6

7

8

9

10

11

58 48 47 65

49 41 46 46

45 44 44 44

28 55 50 41

54 49 53 52

47 45 47 47

45 54 50 57

49 47 46 50

43 48 49 47

37 43 47 27

48 52 57 51

Totals 218

182

177

174

208

186

206

192

187

12

13

14

15

16

17

18

19

20

21

22

45 43 44 44

55 42 47 52

42 41 46 50

45 43 48 45

41 46 41 30

43 42 38 35

53 44 49 52

41 43 41 35

43 45 44 46

34 34 40 40

50 48 48 48

Totals 176

196

179

181

158

158

198

160

178

Batch

Batch

154 208

148 194

split into four sub-batches, which go through several more stages of processing. In Table lOA each column gives the measurements on the four sub-batches from one batch. The variation was in excess of that which could be tolerated. If we suppose that the individual observations Xiv are represented by the model Xiv = + Yi + Ziv' then the variance of the Yi'

e

326

ONE-WAY ANALYSIS OF VARIANCE

CHAP.

10

(02, will represent the variation due to the earlier stages and the variance of the Ziv' 0'2, the variation due to the later stages. The analysis of variance follows the same lines as in the model I example discussed in Section 10.2, except that the l1i being all equal to 11 makes the calculation slightly simpler. We need

A B

=

k

11

i

•

L Lxi. = 58 2 + 48 2 + ... + 48 2 = 186,922,

= 1.

±(i

l1i.

C

= ~(± kll

i

i

Xiv

Y = 1. (218 2 + 1822 + ... 1942) = 185,378. } 4

XiV)2

=

•

(4018)2 22 X 4

=

183,458.227.

With these figures we construct Table 10.5. Table 10.5

Source of variance Between batches Between sub-batches Total

Sums of squares

Degrees of freedom

Mean squares

[EM. S.]

B - C

= 1919.773

21

91.418

a2

A - B A - C

= 1544.000 = 3463.733

66 87

23.394

a2

+ 4(02

The succeeding analysis runs as follows. Under the null hypothesis that (02 = 0, s~M = 9l.4l8/23.394 = 3.91 '" F(21, 66). The 0.9995 point of F with these degrees of freedom is 2.94, and so we reject the null hypothesis at the level of significance ()( = 0.0005. The mean square s~ = 23.394 is an estimate of 0'2, and

w2 =

91.418 - 23.394 = 17.006. 4

(6.1)

It appears, therefore, that the variability is arising in roughly equal amounts from the two parts of the process. Application of (9.2.2) gives (17.95, 3l.97) as 90 per cent confidence limits for 0'2, as were calculated in (9.2.3). The large sample approach to confidence limits for (02 uses (5.8):

V[w 2 ] =..? [(91.418)2 + (23.296)2J = 50.78' 4 2 22 - 1

22(4 - 1)

so 90 per cent confidence limits for (02 are 17.006 (5.29, 28.7).

(6.2)

,

±

1.65

X

7.126

=

SECT.

10.7

EXAMPLE WITH UNEQUAL GROUP SIZES

327

Alternatively, and better, Bross's formulas (5.9) involve Fo.95(21, 66) = 1.72, F~.95(21, 00) = 1.56, Fo.05 (21, 66) = I/Fo.95(66, 21) = 1/1.91 = 0.5236, F~.05{21, 00) = I/Fo.95 (00,21) = 1/1.81 = 0.5525:

F = 91.418/23.394 = 3.908,

L = L =

3.908/1.72 - 1 = 0.500 (1.56 x 3.908)/1.72 - 1 ' 3.908/0.5236 - 1 (0.5525 x 3.908)/0.5236 - 1

= 2.069.

w

Since 2 = 17.006, the limits are !::.w 2 = 0.500 X 17.006 = 8.503 and £c0 2 = 2.069 x 17.006 = 35.185. Comparing these with the cruder approximate limits, we see that the crude limits fall down particularly in being, from their very method of construction, symmetric about the point . estimate w2 • Confidence limits for the ratio W 2/(J2 are obtained with (5.11). Inserting s~M = 91.418/23.394 = 3.91, k(n - 1) = 22(4 - 1) = 66, k - 1 = 22 - 1 = 21, Fo.95{21, 66) = 1.72, FO•95{66, 21) = 1.91, we get

H

3.91 X

1.~2 -

1)

< :: < ~ (3.91

X

1.91 - 1) = (0.32, 1.62).

This has the somewhat disagreeable implication that, though the point estimate of W 2/(J2 is 17.006/23.394 = 0.727, we have determined this ratio with rather poor precision, even in spite of having quite a lot of data, since the 90 per cent confidence limits range from less than half to more than twice the point estimate. Finally, if we wanted 90 per cent confidence limits for the grand mean, we would use (5.4), inserting :t .. = 4018/88 = 45.659, s~ = 91.418, kn = 88, and using f o.05(21) = -1.721 = - fo.95(21): ±f..)s~/kn = ±1.721..)91.418/88 = ±1.754, whence the 90 per cent confidence limits for ~ are (43.9, 47.4).

10.7. An Example of Model II One-Way_A!laJ)'Sis of Variance with Unequal Group Sizes To illustrate a model II one-way analysis of variance with unequal group sizes we will use the data of Table 9.1, assuming that the different groups correspond to random replications. This may not actually be the case, as the groups have mirrors with identifiably different properties. For the purpose of illustrating the arithmetical procedures, however, we will assume that the mirrors were drawn at random from a large population of mirrors. A straightforward analysis of variance proceeds by obtaining

328

CHAP.

ONE-WAY ANALYSIS OF VARIANCE

10

from the calculations in the lower part of Table 9.1 the following quantities:

1/1 X:y = 124,531,

k i

\-1 ( k

k

(

~

~-

y

i

)2 Xiv

~

IIi

•

1/1

~~

IIi)

i

1 (1/1

k

~~

Xiv

)2 = 102,339.683,

y

(2955)2

= - - = 98,112.640

89

y

These are inserted in the identities (2.31), (2.33), and (2.35) to give Table 10.6. The coefficient of w 2 in the expected mean square for between groups is, by (4.10), 1 (~ k k - 1 i

-

IIi -

i+-Il~) = - 1 ~

6- 1

klli i

( 89 - -1379) 89

= 14.701.

(7.1)

Table 10.6

Source of variance

Sums of squares

Degrees of freedom

Mean squares

Between groups Within groups

4,227.043 22,191.317

5 83

845.409 267.365

Total

26,418.360

88

E[M. S.]

a2 a2

+ 14.701002

= 0 is tested by the variance ratio

The null hypothesis that w2

845.409 267.365

= 3 16

..

Since FO•975(5, 83) = 2.72, we reject the null hypothesis and estimate w2 as (845.409 - 267.365)/14.701 = 39.320. To form a weighted mean with approximately minimum variance, we compute the weights ll'i from (5.21). For i = 1 for example,

WI

=

1

39.320

+ 267.365/12

= _1_ = 0.016234.

(7.2)

61.60

We then form l'

x=

~ lV,Xi _i- -

~

,

A

k Wi

3 459562

= .

0.103726

= 33.353.

(7.3)

SECT.

10.8

SIMPLE SAMPLING THEORY

329

This will have a variance estimated as

V[x]

= _1_ = ~

A

k Wi

1 0.103726

= 9.6408.

(7.4)

i

Making the conservative assumption that this has the usual distribution with k - 1 = 5 degrees of freedom, confidence limits for ~ are 33.353 ± 2.571y'9.6408 = 33.35 ± 7.98. If we made the incorrect assumption that we had 89 independent observations we would have calculated x.. = 2,955/89 = 33.202 and 8 Z = 26,418.360/88 = 300.209, whence V[x..1 = 300.209/89 = 3.373 and the confidence limits for ~ would be 33.202 ± 1.987y'3.373 = 33.20 ± 3.65. The length of this (incorrect) confidence interval would thus be a serious underestimate of the better estimate, being too short by a factor of more than 2.

10.S. Simple Sampling Theory In this section we will consider a simple sampling problem. Suppose that we wish to estimate the mean fat content of a large consignment of wool delivered in many bales. Suppose that it costs CI to get hold of a bale, and that once a bale is obtained it costs Cz to take each sample. Thus if we take n samples from a bale, the cost is CI + Czll, and if we take 11 samples from each of k bales the total cost is k(CI + Czll) = kCI + knc2' Suppose that we have to operate so that the total cost is some fixed constant, say C, and we wish to choose k and n, subject to this limitation, so that the variance of the estimated mean fat content is a minimum. In other words, we wish to minimize V[x ] ..

=

2 (J

+ IIW 2 kll'

(8.1)

using the notation of (5.2), subject to the condition that kC I

+ knc2 =

C.

(8.2)

Substituting for n from (8.2) in (8.1) we obtain (8.3) The value of k that makes this a minimum can be found by differentiating with respect to k and equating to zero: (8.4)

330

ONE-WAY ANALYSIS OF VARIANCE

CHAP.

10

We can substitute kl1C 2 for C - clk, from (8.2), obtaining as the optimum value of 11, (8.5) The corresponding optimum value of k is found from (8.2). Use of these formulas requires estimates of (I and (1) in addition to Cl and C2 • The formulas will give nonintegral values for 11 and k and in practice it will be adequate to use the nearest integers, though of course we cannot use an 11 of zero. For a full study of sampling techniques, see Cochran [10]. 10.9. The Power Function of Model II One-Way Analysis of Variance In (9.3.14) we found the power function 7T{cp) for a ratio of variances cp, when the null hypothesis was (1~ = (1~,

(I~/(1~ =

(9.1) Here we are dealing with mean squares s~ and s~ with expected values (12 + 11(1)2 and (12: We put cp equal to the ratio of the expected values as before:

cp =

(12

+ /1(1)2 (12

(1)2

= 1 + /1 -(12 •

(9.2)

The procedure is best illustrated by an example. Suppose we are using = 22 groups of 11 = 4 observations, with a level of significance, at = 0.05. Then/l = 21'/2 = 66. We ask what is the probability of rejecting the null hypothesis when in fact (1)2/(12 = 3/4. Then cp = 1 + 4 X 3/4 = 4, and Fo.95{21,66) = 1.72, and

k

7T(:: = ~) = pr{F(21, 66) > ~ X 1.72} = Pr{F{21, 66) > 0.43} = Pr{F{66, 21) < 2.326},

(9.3)

taking reciprocals and reversing the sign of the inequality. Since Pr{F{66, 21) < 2.173} = 0.975, the power is somewhat greater than 0.975. EXERCISES 10.1. For the data of exercise (9.7), irrespective of the outcome of that exercise: (a) Test the null hypothesis that the mean octane number is the same in the four regions.

331

EXERCISES

(b) Construct 95 per cent confidence limits for the difference in means of regions A and l! (i) assuming that it had been your original intention so to do, (ii) assuming that the idea occurred to you after looking at the data. (c) Construct 95 per cent confidence limits for the contrast defined by the difference between region A and the mean of the other three regions. Assume that this contrast was suggested by the data. 10.2. The table below gives 100 times the logarithm to the base 10 of onetenth the throughputs listed in exercise (7.3). The logarithmic transformation is selected as it seems likely to stabilize the variance as a function of the mean: The specific form mentioned gives numbers easy to work with.

Foundry

Transformed throughputs

A B

92, 78, 60, 67, 53, 66 83, 96, 98, 60, 99, 78, 77, 103, 93, 107 66,97, 100,96,96

C

(a) Test the null hypothesis that the means of the three foundries are the same. (b) Construct 95 per cent confidence limits for the difference between foundry A and the mean of foundries Band C, assuming (i) that this had been your original

intent, and (ii) that this contrast was suggested to you by the data. Quote these limits in terms of the transformed numbers and also in terms of the original scale. 10.3. The data below give some of the results of Rosa and Dorsey ["A New Determination of the Ratio of the Electromagnetic to the Electrostatic Unit of Group k

2

nl

3

62 64 62 62 65 64 65 62 62 63 64

65 64 63 62 65 63 64 63

65 64 67 62 65 62

11

8

695 43,927

62 66 64 64 63 62 64 64 66 64 66 63

Li

5

4 65 63 63 63 61 56 64 64 65 64 64 65

66 64 65 65 65 64 66 67 66 69 70 68 69 63 65

6

24

15

509

385

1,525

992

32,393

24,723

96,997

65,664

64

llj

LXiv

4,106

y llj

LX~ty

263,704

332

ONE-WAY ANALYSIS OF VARIANCE

CHAP.

10

Electricity," Bulletin of the National Bureau of Standards, 3 (1907), 433-6041 on the ratio of electromagnetic to electrostatic units of electricity, a constant which is equal to the velocity of light. The figures below have had 2.99 subtracted from them and have been multiplied by 10,000 to give numbers simple to work with. The groups correspond to successive dismantling and reassembly of the apparatus. Certain sums which may be useful are given in the lower part of the table. (a) Make a conventional analysis of variance of these data, giving the expectations of the mean squares. (b) Obtain estimates of the components of variance for within groups and between groups. Test the null hypothesis that the latter is zero. (c) Form the weighted mean which has minimum variance. (d) What is the estimated variance of this mean? (e) Supposing that all 64 observations were regarded as 64 independent observations, what would the variance of the simple mean be? 10.4. A manufacturer has been making all his product from a single large uniform batch of raw material and has achieved a reputation for uniformity of product. This reputation has brought him more business, and he now needs to consider a much larger output. The raw material is no longer obtainable in very large batches, and he must now use relatively large numbers of small batches. He considers that he will lose his reputation for uniformity if the standard deviation of his new output exceeds by 20 per cent the standard deviation of the old. In other words, if the new standard deviation is Up' and the old uT, then UT' must not exceed 1.2uT • He decides to run a trial on a random sample of five batches of raw material. He is going to make a number, say n, of parts from each batch and test for batch differences with a one-way analysis of variance using ex as his level of significance. (a) Suppose he chooses n = 9, ex = 0.01; what is the probability of his detecting a deterioration, i.e., an increase in total standard deviation of the magnitude specified above? (b) Suppose he uses ex = 0.1, what then? (c) Suppose he requires ex = 0.01 and a probability of detecting the specified deterioration in uniformity of 0.99, what should n be?

REFERENCES 1. Heyl, Paul R., "A Redetermination of the Constant of Gravitation," Jou/"llal of Research of ti,e Bureau of Standards, 5 (1930), 1243-50. 2. Schefi'6, H., "A Method for Judging All Contrasts in the Analysis of Variance," Biometrika, 40 (1953), 87-104. 3. Tukey, John W., "The Problem of Multiple Comparisons."Unpublished dittoed manuscript. 4. Pearson, E. S., and H. O. Hartley (eds.), Biometrika Tables for Statisticians, Vol. 1. London: Cambridge University Press, 1954. 5. Dunnett, Charles W., "A Multiple Comparison Procedure for Comparing Several Treatments with a Control," Jou/"llal of the American Statistical Association, 50 (1955). 1096-1121. 6. Bross, Irwin, "Fiducial Intervals for Variance Components," Biometrics, 6 (1950), 13~4.

REFERENCES

333

7. Scheffe, Henry, Allalysis of Variallce. New York: John Wiley and Sons, 1959. 8. Williams, J. S., "A Confidence Interval for Variance Components," Biometrika, 49 (1962), 278-81 9. Cochran, W. G., "The Combination of Estimates from Different Experiments," Biometrics, 10 (1954), 101-129. 10. Cochran, William G., Samplillg Techlliques, 2nd ed. New York: John Wiley and Sons, 1963

CHAPTER

II

Simple Linear Regression

11.1. Introduction When an investigator observes simultaneously two variables x and y, usually with good reason he plots the observations on a graph. If there is any sign of an association, he is usually seized with the impulse to fit a line, usually a straight line or, rather infrequently, a parabola or cubic. The purpose of this arithmetic penance of curve fitting is often not very clearly defined; one purpose, however, might be to predict x from a new observed y or vice versa. Another purpose is to use the route of testing the significance of the parameters of the line as a means of testing for association between x and y. The standard technique for fitting a line is known as "least squares," and the line so fitted is known as a regression line for curious historical reasons (see Section 12.5). In this chapter we shall consider the case in which a series of values of x have been selected by the experimenter and he observes y at those values of x. The so-called independent variable x is assumed to be measured without appreciable error. The situation in which the variables x and y vary at random outside the control of the experimenter and are only observed will be discussed in the next chapter. The present chapter is of unusual length, and some guidance to the reader is called for. Sections 11.2 and 11.3 give the usual fitting of a straight line to data by the method of least squares, and Section 11.4 is a numerical example. This is as much as is found in some elementary textbooks. Section 11.5 deals with finding confidence limits for x from an observed y. In Section 11.6 the problem of comparing two regression lines is covered. An important application of this statistical technique is to parallel-line biological assay (Section 11.7); a numerical example follows in 334

SECT.

11.2

335

THE MODEL

Section 11.8. The line discussed thus far is the two-parameter line Y = a + b(x - x). In Section 11.9 the one-parameter line through the origin y == bx is described, with its use in reverse in Section 11.10. The construction of joint confidence regions for (IX, f3) is considered in Section 11.11. Sections 11.12 and 11.13 discuss the case where we have several observations on Y at some of the x's and it becomes possible to check the line for goodness offit. Section 11.14 extends the methods of Section 11.6, for comparing two regression lines, to the case of more than two lines: This technique is also known as the analysis of covariance as it amounts to making an analysis of variance of Y adjusted for concomitant variation in x. For a thorough review of linear regression see Acton [1]. In general we assume that we have observations Yiv as follows: (1.1)

where i = 1, ... , k. If all the Il i = 1, so that there is only one observation on Y at each x, we have a special, and important case, for which the analysis, both theoretical and arithmetical, is simpler. We will therefore deal with this case first. The general case, where all the lli are not equal to 1, can always be treated as the special case by merely ignoring the fact that some of the Xi happen to be identical. For example, if III = 2, 112 = 3 and 113 = 1, we can regard the observations (Yll' Xl), (YI2' Xl), (Y2l> X2), (Y22' X2), (Y23, x 2), (Y3l> x 3)

(1.2)

(Yl> Xl), (Y2, X2), (Y3' X3), (Y4, x 4), (Y6, Xli), (Y6' x 6).

(1.3)

as The fact that in this second set of observations Xl = X 2, etc., can in general be disregarded. The only case in which it would matter is where all the x's are identical; then it is obviously meaningless to attempt to fit a line. For the present we will deal with the simpler situation and suppose that we have observations (Xi' Yi)' i = 1, ... ,k. Except where otherwise k

obvious or explicitly stated, all summation operations,

!, will

be over

i

i = I, ... , k, and so we will omit the index of the summation and write

merely:E. 11.2. The Model We assume that y is distributed normally about an expected value 'fJ with variance (12, and that all observations are independent. We further assume that 'fJ is a simple linear function of x: 'fJ = IX

+ f3(x -

x).

(2.1)

336

SIMPLE LINEAR REGRESSION

CHAP.

11

The problem is to obtain, from the data, sample estimates a, band S2 of ex, {l and ()'2 and to determine the distribution of these estimates. The estimated regression equation is Y= a

+ b(x -

x)

(2.2)

where x = ~ xi/k. We write (2.1) and (2.2) in the forms given rather than as 1] = ex' + {lx, Y = a' + bx because in this latter form a', b are dependent whereas in the form (2.2) a, b are independent, and this property is convenient when we come to consider V[ Y]. The standard method of estimation in regression is the method of least squares; this is to use those values of a, b which will minimize the sum of squares of deviations, say R, between the observed values Yi and the predictions Y i given by inserting the values of Xi in the estimated equation (2.2). Thus we minimize

R =

L (y; -

Y i )2 =

L [y; -

a - b(x; - X)]2.

(2.3)

The method of least squares appears to be largely due to Gauss. As far as estimation of the parameters is concerned, it does not require the assumption of normality, but this assumption is necessary for construction of confidence intervals for or tests of hypotheses about the parameters. With the assumption of normality, the method of maximum likelihood gives results identical with those of the method of least squares. The method of least squares has the desirable properties that the estimators it gives are unbiased and, among all unbiased linear estimators, have minimum variance. Detailed discussions have been given by David and Neyman [2] and Plackett [3]. To find the values of a and b that minimize R we differentiate (2.3) with respect to a and b and equate to zero:

oa = -2L [Yi oR = -2 L [Yi oR

a -

b(Xi - x)] = 0,

a - b(Xi - x)](xi

ob

(2.4) x) = O.

-

(2.5)

These two equations can be written as

L (Yi L (Yi -

Yi )

= 0,

(2.6)

Yi)(X i - x)

= o.

(2.7)

Rearranging (2.4) and (2.5) gives

+ b L (Xi x) + b L (Xi ka

.a

L (x; -

= L Yi' X)2 = L (Xi x)

(2.8) -

X)Yi'

(2.9)

SECT.

11.2

337

THE MODEL

Since l:: (Xi - x)

= 0, we have as estimators for ~ and p: a= b=

I

Yi = g,

(2.10)

k

I

(Xi - X)Yi '" ",(Xi - x)2

(2.11) .

The numerator of this expression for b can be written slightly differently: Since l:: (Xi - x) = 0, then g l:: (xi - x) = 0, and

I

(Xi - X)Yi

=I

(Xi - X)Yi - g

I

(Xi - x)

=I

(Xi - X)(Yi - g). (2.12)

Thus an alternative form for b is (2.13) We can readily check that b is an unbiased estimator of p. We assumed that the expected value of Yi was 1]i' given by (2.1) with X = Xi' Then

E[b]

=I

(Xi - X)E[Yi] I (Xi - X)2

= ~

I(Xi - x) I (Xi - X)2

=I

+ /-'R II

(Xi - x)[~ + P(xi - x)] I (Xi - X)2 (Xi - X)2 R = /-'. (Xi - X)2

(2.14)

The variances of a and b can be obtained directly, for inspection of (2.10) and (2.11) shows that they are linear functions of the Vi' which are assumed to be independent and have a normal distribution with variance 0'2:

I

y.]

1

V[a]

= V [---,: = k2 I

V[Yi]

= k'

V[b]

=

=I

(Xi - X)2V[Yi] X)2]2

v[I (Xi - X)Yi] (Xi - X)2

I

0'2

[I (Xi -

(2.15)

=

I

0'2

•

(Xi - X)2

(2.16) We will defer the estimation of 0'2 to the next section. A demonstration that in general the method of least squares produces estimators of smallest variance among all unbiased linear estimators is somewhat involved [2, 3], but it is often quite easy to show that a least squares estimator, when it has been obtained, is of minimum variance. We shall show that the estimator b (2.11) has the smallest variance of all unbiased linear estimators. Suppose that there exists an alternative linear estimator b' , (2.17) b' = I CiYi'

338

SIMPLE LINEAR REGRESSION

CHAP.

11

We have E[b']

=L

ciE[Yi]

=L

ci[at

+ P(xi -

=

x)]

at

L

For b' to be an unbiased estimator, so that E[b']

Ci

+ PL

= p,

(Xi - x)ci. (2.18) we require

L C = 0,

(2.19)

i

L (xi -

=

x)c i

1.

(2.20)

The variance of this alternative estimator b' is V[b']

=L

CW[Yi]

=

(12

C~ = (12 L[Ci -

L

L ~i - X-)2 Xi - X

+ LX(Xi- - X_)2J2 X (2.21)

since the cross product term is zero, for

J

L [C. Xi - x (Xi - x) • L (Xi - X)2 L (Xi - x)2

= L CtC Xi

- x) _ L (Xi - x)2 L (Xi - X)2 [L (Xi - X)2]2

=0 '

(2.22) using (2.20). Thus V[b']

=

2

(1

L [Ci

- J2 + L 2 (Xi - X) (Xi - X)

X· -""'.

k

X

-

(1

2

2'

(2.23)

in which the last term is a constant. Hence to minimize V[b'] we can only make adjustments to the first term; by putting Ci

=

Xi - x X)2

L (Xi _

(2.24)

we make the first term zero and hence make V[b'] a minimum. But, with this value of Ci , our alternative estimator (2.17) is b' = L c.y. = L (Xi - X)Yi .. L (Xi - X)2

(2.25)

which is our original least squares estimator b (2.11). Therefore b' = b, and b is the minimum variance unbiased linear estimator. Incidentally, from (2.23) we see that when Ci is given the value in (2.24) V[b] = (12/"'2:. (Xi - X)2, as found earlier in (2.16). 11.3. An Analysis of Variance Representation We will now consider regression analysis from the point of view of analysis of variance. From this approach we will obtain the variances of

SECT.

11.3

AN ANALYSIS OF VARIANCE REPRESENTATION

a and b (which we have already found directly), and in addition we will be able to show that a and b are independent of each other and of S2, our estimate of 0'2. The deviation of an observation Yi from the value predicted by the true regression equation (2.1) can be written as

Yi - '7i = (Yi - Yi) = (Yi - Yi) = (Yi - Yi )

+ (Yi -

1]i)

+ [a + b(Xi - x) - {ex + P(xi + (a - ex) + (b - P)(xi - x).

x)}] (3.1)

y

~

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _L -____________________ ~y

~X

Xi

~

Figure 11.1

This equation is represented graphically in Figure 11.1. Squaring and summing over i gives

!

(Yi - 1]i)2 = k(a - ex)2

+ (b -

P)2! (Xi - X)2

+!

(Yi - Yi)2, (3.2)

using (2.6) and (2.7) to show that two of the cross products are zero. On the left-hand side of (3.2), L (Yi - 1]i)2 is a sum of squares with k degrees of freedom and distributed as 0'2X 2(k). It is partitioned into three components. The first and second each involve one variable, a and b respectively, and will each have one degree offreedom. The third involves k variables Yi - Y i , but these are subject to the two restrictions (2.6) and (2.7), so the degrees of freedom are k - 2. Thus the sum of the three

340

SIMPLE LINEAR REGRESSION

CHAP.

11

sums of squares on the right-hand side of (3.2) equals the sum of squares on the left-hand side, and the degrees offreedom likewise, so by Cochran's theorem the sums of squares on the right-hand side are distributed as O'2X 2 with the corresponding degrees of freedom and are independent. Since ~ (y; - y;)2 is distributed as O'2X2(k - 2), S2, defined as S2

= _1_

~(y. _ y)2

k-2£'"

(3.3)

.'

will be distributed as O'2X2(k - 2)/(k - 2), will have expected value 0-2, and will be independent of a and b. Also, since k(a - ()()2 is distributed as 0-2X2(l) it has expected value 0-2, and therefore V[a]

= E[(a

2

= -0- .

_ ()()2]

(3.4)

k

By a similar argument, V[b] = O'2/~ (Xi - X)2. We can make a test ofthe null hypothesis P = 0 by substituting S2 for 0'2 in the expression for V[b]. Alternatively, since the last two terms on the right-hand side of (3.2) are independent and have distributions O'2X2(l), O'2X2(k - 2) respectively, we have that

(3.5) Thus under the null hypothesis

P = 0, (3.6)

To calculate the numerator of S2, (3.3), we usually use an identity obtained as follows. We note that Yi -

y=

[a

+ b(x; -

x)] - a

= b(Xi -

x),

(3.7)

so by (2.7)

L (Yi -

Yi)t Yi - y)

= b L (Yi

-

Yi)(X i - x)

= O.

(3.8)

Thus when we square and sum over i the identity

(3.9) the cross product is zero. Therefore

This equation is entered in the second column of Table 11.1.

11.3

SECT.

341

AN ANALYSIS OF VARIANCE REPRESENTATION

To calculate ~ (Yi we use (3.7):

! Pi -

y)2

-

y)2, known as the sum of squares due to regression,

=

b2

=

[! (Xi - X)(Yi ! (Xi - X)2

!

(Xi - X)2

= b!

(Xi - X)(Yi - y)

y)]2

(3.11) (3.12)

Table 11.1 Source of variance

Degrees of freedom

Due to regression Remainder

1 k -2

~(Yi ~(Yi

-

fj)2 y i )2

Total

k -1

~(Yi

-

fj)2

Sums of squares

Mean squares s~ S2

E[M. S.] a2 a2

+ P2~(xi

-

x)2

The other term on the right-hand side of (3.10), ~ (y; - y;)2, is the sum of squares of deviations of observed values about the estimated line, and is commonly known as the residual or remainder sum of squares. It is, of course, the numerator of S2 defined as in (3.3). Using (3.10) and (3.12), it can be calculated as

~ ( . _ y)2 = ~ (y _ -)2 _ k

Y.

•

k

i

Y

[! (X; - X)(Yi ! (x _ X)2

(3.13)

y)]2.

To find the expected value of s~ defined as in Table 11.1, we note that (b - (3)2 ~ (Xi - x)2 is distributed as 0'2X 2(1) and hence has expected value u2• Thus

0'2 = ! (Xi - x)2E[b 2] - 2{3! (Xi - x)2E[b]

+ {32! (Xi -

X)2.

(3.14)

Now from (2.14), E[b] = {3, and from (3.11),

!

(Xi - x)2E[b 2] = E[! (Yi - y)2].

(3.15)

Thus substituting in (3.14) and rearranging, E[! (li - y)2] = 0'2

+ {32 !

= 0'2(1 +

(Xi - X)2

:r:]).

(3.16) (3.17)

The latter form makes it clear that E[s~] is a function of the ratio of {32 to the variance of b.

342

SIMPLE LINEAR REGRESSION

CHAP.

11

Using (2.2) for Y, (2.15) and (2.16) for V[a] and V[b], and the fact that

a and b have zero covariance, we can write, for any fixed x, V[Y] = V[a = a2

+ b(x + (x -

[!k

x)] = V[a]

+ (x -

x)2V[b]

J.

X)2 ~:CXi - X)2

(3.18)

The expected value of Y is ex. + (l(x - x) = 1'/. Y is a linear function of a and b, and a and bare lineat:- functions of the Yi which are normally distributed. So Yis normally distributed; inserting S2 for a2 in (3.18), we have Y-1'/

(3.19)

--='""t(k - 2) . .JV[Y]

This gives confidence limits for 1'/, the true value at some specified x: Pr{Y - tp 2.JV[Y]

< 1'/ < Y -

tp 1.JV[Y]}

= P2 -

Pl'

(3.20)

A new single observation at x will be distributed about 1'/, with a variance a2 , independently of Y, so E[y - Y]

= E[y] -

V[y - Y]

=

V[y]

E[Y]

= 1'/ -

+ V[Y] =

a2

1'/

= 0,

(3.21)

[1 + !k + j,x(Xi- -x)~x)2J,

(3.22)

and .J y - Y '"" t(k - 2). V[y - Y]

(3.23)

Thus we have the conclusion Pr{Y + tP1.JV[y - Y]

< y < Y + tp)V[Y -

Y1} = P 2

-

Pl' (3.24)

This resembles a confidence interval statement, but differs, for whereas a confidence interval statement is an interval for a parameter, (3.24) is an interval for a random variable y. The interval (3.24) is known as a prediction interval and represents the limits between which we are P 2 - PI confident that a new single observation y taken at the specified x will lie. The interval (3.20) represents the limits between which we are P z - PI confident that the true value of the parameter will lie.

11.4. An Example of Linear Regression The data of Table 11.2 (personal communication from Dr. D. W. Cugel), plotted in Figure 11.2, give the results of an experiment to determine the

SECT. 11.4

AN EXAMPLE OF LINEAR REGRESSION

343

Table 11.2 xi

Yi

Xi

Yi

Xi

Yi

1190 1455 1550 1730 1745 1770

1115 1425 1515 1795 1715 1710

1900 1920 1960 2295 2335 2490

1830 1920 1970 2300 2280 2520

2720 2710 2530 2900 2760 3010

2630 2740 2390 2800 2630 2970

behavior of a method of measuring blood flow. The Yi are rates of flow estimated by this method for a series of Xi; the Xi were accurately qetermined by a direct method. We need the following: k = 18, LXi = 38,970, L Yi = 38,255,

LX: = 89,394,900,

LY~ = 86,125,825,

10- 2 X Y 34 32 30 28 26 24 22 20 18 16 14 12 10 8 6

4 2

o o

2

4

6

8

10 12 14 16 18 20 22 24 26 28 30 32

Figure 11.2

344

SIMPLE LINEAR REGRESSION

CHAP.

11

from which we derive

~ (x. ",c.,

- X)2

=

•

~ (y. ",c.,

_ y-)2

=

•

~ (Xi

",c.,

~ X~

(! Xi)2 =

-

k

",c.,.

~ y~ _

(! Yi)2 = k

",c.,.

89 394 900 - (38,970)2 " 18

86 125 825 _ (38,255)2 " 18

-) =",c.,~ XiYi - !

-)(

- X Yi - Y

=

5 024 850 '"

4 823 323.6 " ,

=

!

Xi Ie Yi

= 87,719,100 _ 38,970 X 38,255 = 4,897,025.0. 18

The sum of squares due to regression is, from (3.12),

~ (¥, _ -)2 = (4,897,025.0? = 4772 451.7. ",c.,

•

Y

5,024,850.0

(4.1)

"

We can now assemble Table 11.3, finding the residual sum of squares by difference [see (3.13)]. Table 11.3

Source of variance

Sums of squares

Degrees of freedom

Mean squares

Due to regression 4,772,451.7 Residual 50,871.9

1 16

4,772,451. 7 3,179.49

Total

17

4,823,323.6

E[M. S.] ()'2

+ fJ2'L,(Xi

_ x)2

()'2

Under the null hypothesis (3 = 0 the variance ratio 4,772,451.7/3179.49 = 1501.0 is distributed as F(I, 16). Here the result is obviously significant. An alternative test would be to use the fact that

b - (3 ""' t(k _ 2), v'V[b]

(4.2)

for which we need b, from (2.13),

b

=

!

(Xi - X)(Yi - y) ! (Xi - X)2

=

4,897,025.0 5,024,850.0

=

and V[b], which we get by replacing 0-2 in (2.16) by

V[b] =

!

0.974561,

(4.3)

S2:

S2 _ = 3179.49 = 6.32753 X 10-4 • (Xi - X)2 5,024,850.0

(4.4)

SECT.

11.4

345

AN EXAMPLE OF LINEAR REGRESSION

A test of the null hypothesis that {3

= 0 is given by

b - 0 = 0.974561 - 0 = 38.74, .JV[b] .J6.32753 X 10-4

(4.5)

which will be distributed as ((16). The null hypothesis obviously has to be rejected. Of course, (2 = (38.74)2 = 1501.0 = F of the previous test. In this particular example there is no reason to doubt the significance of the regression. A more interesting null hypothesis is that {3 = 1; for this, (4.2) gives 0.974561 - 1 = -1.012, (4.6) .J6.32753 X 10-4 which is distributed as ((16) under this null hypothesis. Clearly the null hypothesis that {3 = 1 can be accepted. We may wish to construct 95 per cent confidence limits for (3; for these, to.976(16) = 2.120 and .JV[b] = 0.02515; so t.J V[b] = 0.05333, and the 95 per cent confidence limits are 0.97456 ± 0.05333 or (0.9212, 1.028). To construct the estimated regression line we need y = 38,255/18 = 2125.278, it = 38,970/18 = 2,165.000, and b = 0.974561, whence the estimated regression line is

+ 0.974561(x -

Y = 2125.278

At x

= 0, (Y)",=o = V[(Y) _ ]

2165.000).

(4.7)

15.352. From (3.18),

= 3179.49[~ + (0 -

",-0

18

2165.000)2J 5,024,850.0

=

3142.493.

(4.8)

Under the hypothesis that (17)",=0 = 0, the ratio 15.352 - 0 = 0.274 (4.9) .J3142.493 is distributed as t( 16): Clearly the null hypothesis of a zero intercept is acceptable. Confidence limits for the intercept are 15.352 ± 2.120

X

.J3142.493 = (-103.5, 134.2).

(4.10)

Let us summarize what has been so far established. (i) The estimated regression line of y on x as given in (4.7). (ii) 95 per cent confidence limits for the regression coefficient {3 are (0.921, 1.028); so the null hypothesis {3 = 1 is acceptable. (iii) The intercept on the y axis has 95 per cent confidence limits (-103, 134); so the null hypothesis that the intercept is zero is acceptable. (iv) The estimated variance of an observation around its true value is 3179.49, corresponding to a standard deviation of 56.83.

346

SIMPLE LINEAR REGRESSION

CHAP.

II

We might ask, if we observe a new value of y, what is our estimate of :1:, and what are confidence limits on the true value of x corresponding to this observed y? This question is investigated in the following section. A further question arising from items (ii) and (iii) above is the following: While the null hypothesis that fJ = 1, the intercept unspecified, is acceptable, and also the null hypothesis that the intercept is zero, fJ being unspecified, is acceptable, is the null hypothesis that fJ = 1 and the intercept is zero simultaneously acceptable? This question is discussed from the testing point of view in Section 11.9 and from the confidence region point of view in Section 11.11.

11.5. The Use of the Regression Line in Reverse Suppose that we have a regression line Y = a + b(x - x), based on k observations (Xi' Yi)' Suppose that we now observe a new fj', the mean of m new observations known to have arisen from the same x, and we wish to predict the X corresponding to this fj', and to construct confidence limits for this prediction. Of course, m may equal 1, so that we just have a single observation y'. We can solve the estimated regression equation (2.2) for:l:, inserting fj' for Y, where fj' is the mean of the new set of observations on y, to get

x - x + fj' -ba'

(5.1)

. -.

a point estimate of the value of X corresponding to the new observed value fj'. The expected value of the new fj' is E[fj'] = 1]. Corresponding to this value of 1] is a value of x given by solving the true regression equation (2.1) for x: Denote this value of x by~. Then 1]

-Cl

~=x+-­

(5.2)

fJ

and

Cl -

= O.

(5.3)

- a - ba - x). This variable will have expected value

(5.4)

1] -

fJ(~

-

x)

We now define a new variable z as

z

E[z]

= E[fj'] -

= fj'

E[a] - (~ - x)E[b]

by (5.3). Its variance is V[z]

_ = V[y'] +

V[a]

+ (~ -

x)2V[b]

=(J'2[.!.+!+ (~_X)2 m

k

L (Xi -

X)2

J.

= 1] - Cl -

(~

-

(J'2 (J'2 = - + - + (~ m

k

x)fJ

=0

x)2 ~

(5.5)

(J'2

k(Xi -

-)2 X

(5.6)

SECT.

l1.S

347

THE USE OF THE REGRESSION LINE IN REVERSE

The random variable z is a linear function of three random normally distributed variables y', a and b, and hence will itself be normally distributed. Thus zlv'V[z] is N(O,l), and, on replacing (12 in (S.6) by its estimate S2 from (3.3), we have

y' - a - b(~ - x) "-' t(k _ 2). sv'llm + 11k + (~ - x)2/'L, (Xi - X)2

(S.7)

We can insert the above expression in place of t in the statement Pr{t1 < t < t 2 } = P 2 - Pi> to get confidence limits for~. If is the lower confidence limit, it is given by the equation

£

fj' - a - b(~ - x)

sv'llm

+ 11k + (f -

= t2 ,

(S.8)

x)2/'L, (Xi - X)2

where 12 is the P 2 point of I with the degrees of freedom of s, namely k - 2. Squaring and expanding and collecting terms in we get a quadratic equation in £:

r, £,

~2[b2 -

t~s2 _ 2J +

'L, (Xi

- X)

+ (fj' -

a

2§['L, (Xit~s2- _X) 2X -

+ bX)2 - t~S2[.!. + ! + m

k

b2x - b(fj' - a)] x2 ] = O. - X)2

'L, (Xi

(S.9)

Using the usual formula for the solution to a quadratic equation, using the negative sign for the square root term, we get ¢=

-

x+

bey' - a)

t2s

b2 - t~s2/'L, (Xi - X)2

b2 - t~s2/'L, (Xi - X)2

X

{[b 2 -

t~s2

'L, (Xi

]

- X)2

(.!.m + !)k + 'L,(fj'(Xi- - a)2X)2 }~.

(S.10)

The upper confidence limit ~ is obtained in the identical manner, using II in place of 12 , and the solution is the same as (S.10) with this change. Now let us suppose that we choose PI and P 2 so that the limits are symmetric in probability, i.e., 11 = 1«/2,12 = 11-«/2, and where there is no danger of confusion we will omit the suffix on the ('s. The behavior of (S.10) is determined by the quantity we define as g:

g= First, suppose that g

(bl.J~[bJ = < 1.

b2'L,

:::2_ X)2'

(S.11)

Then

I / I> b vV[b]

t,

(S.12)

348

SIMPLE LINEAR REGRESSION

CHAP.

II

and so b is significantly different from zero at the level of significance when tested against the two-sided alternative. Also, when g < I, 2 2

b2 _

ts (Xi - X)2

L

>0

IX

(5.13)

'

so the quantity under the square root sign in (5.10), say Q, is positive and we obtain a real solution for §. Second, suppose that g> 1. Then in (5.12) the inequality sign is reversed and b is not significantly different from zero. Also, the inequality sign in (5.13) is reversed. We will get imaginary solutions to (5.11) if Q < 0, i.e., if [

b2 _

1+ _1) + (-'Y -

] (_ t2s2 L (Xi - X)2 m

a )2

<0

L (Xi - x?

k

'

(5.14)

i.e., if

Iy' - al

< {L (Xi -

X)2[

(1. + !)}~.

t2s2 _ - b2] L (Xi - X)2 m

k

(5.15)

Thus if y' is rather close to a = y, the confidence limits are imaginary, i.e., (- 00, (0). However, if y' is rather far from a = y, then the inequality in (5.15) is reversed, and then Q > 0 and (5.10) gives real solutions. Thirdly, if g is small, say g < 0.1, then b2 _

t2s2 L (Xi - X)2

=

b2[1 _

t2s2 ] 2 b L(Xi - X)2

=

b2(1 _ g) ~ b2, (5.16)

and (5.10) and its analog for ~ become

~,....., x + y'

-

-

b

a_

t2 s[(l m

Ibl

+.!) + k

~"""'x+Y'-a_tls[(l+!)+ b

Ihl

In

k

(y' - a? J~ (Xi - X)2

(5.17)

b2

b2

{Y'-a)2 J~ L (Xi - X)2

(5.18)

L

as approximate confidence limits for~. These are usually considered valid for most purposes when g < 0.1. To illustrate these results, suppose that in the example of Section 11.4 we observe a single new observation y = 3000. To construct 95 per cent confidence limits for ~ we need t o.975 = -to.025 ' For that example S2 = 3179.49, b = 0.974561, L (Xi - X)2 = 5,024,850. Inserting these values in (5.11) we obtain g = 0.00299, which permits use of the approximations

SECT.

11.6

349

THE COMPARISON OF TWO REGRESSION LINES

(5.17) and (5.18). The 95 per cent confidence limits for ~ are 2165.000

+ 3000.000 -

2125.277 0.974561

± 2.120J3179.49[(~ 0.974561

+ -l) + (3000.000 - 2125.277)2J~

1

18

(0.974561)2

X

5,024,850

= 3062.5 ± 135.2. Thus if we observe a flow rate of 3000 by our new method, we can be 95 per cent confident that the true flow rate is in the interval (2927,3198). 11.6. The Comparison of Two Regression Lines We suppose that we have two sets of observations,

,,=l, ... ,l1i :

(XiV'

YiV )' i

= 1,2,

and We will discuss the procedure for deciding whether a single common regression line is an adequate fit, or whether separate regression lines Y1

=

Y2

=

a1 + b1(x - Xl) a2 + b2(x - x2 )

(6.1)

are necessary. We start by fitting separate lines, obtaining estimates ai' bi and s~, i = 1, 2. If the lines are identical, the s~ will be estimates of a common (12, and their ratio will be distributed as F. It is easier'to put the larger in the numerator of the variance ratio; a two-sided test will usually be appropriate. If the null hypothesis is rejected, then the lines differ in this regard, but further examination is difficult because the dissimilar variances involve the Behrens-Fisher problem. If the null hypothesis of a common residual variance is accepted, we can form a joint estimate of (12 [cf. (9.7.6)],

s:

S2

= (111 - 2)s~ + (112 111 -

2

+ 112 -

2)s:. 2

If the null hypothesis PI - P2 = 0 is true, bi distributed about 0 with a variance

-

(6.2) b2 will be normally

350

SIMPLE LINEAR REGRESSION

CHAP.

11

where (1z will be estimated by (6.2). Thus under the null hypothesis, {31 - (3z = 0, ~-~ ) -1

"1

s{[~ (Xlv -

Xl)2]

-1

"2

+ [~(X2y -

X2)2]

~ r-.J t(1I1

+ 112 -

4.

(6.4)

}

If we reject this null hypothesis, then the lines differ in slope and hence are different. If, on the other hand, we accept this null hypothesis, then We fit a joint estimate of the common slope and proceed to test whether the lines are coincident as well as parallel. We next need to estimate the common slope. We have two true equations for the two parallel lines,

'71

= (Xl + (3(x -

Xl),

(6.5)

Xl),

(6.6)

and two estimated equations, Y1

= a1 + b(x -

The sum of squares of deviations of the observations from the two parallel estimated lines is fll

R =

L (Ylv -

fl2

ylv )2

v

+ L (Y2v -

Yzv)2

v

nz

fll

= L [Ylv -

a1

b(Xlv - Xl )]2

-

v

+ L [Yzv -

a 2 - b(X2v - X2)]2.

v

(6.7)

The least squares estimates are those which make this a minimum. Differentiating with respect to aI, az and b and equating to zero gives

oR - =

oa

n1

-2 L rylv - al - b(Xlv - Xl)]

= 0,

(6.8)

v

1

(6.9)

'" [Yzv - a z - b(xzv -2 L

x2)](X2v - x2) = O.

(6.10)

v

The first two of these equations give (6.11)

SECT.

11.6

351

THE COMPARISON OF TWO REGRESSION LINES

and the third gives nl

b

=

'L Y1v(~lv -

n2

+ 'L Y2v(X2v -

Xl)

v

X2) (6.12)

v nl

'L (Xlv -

112

X1)2

+ 'L (X 2V -

v

X2 )2

v

The same arguments that were used to obtain (2.16) give (6.13)

To estimate (J'2 for the model (6.5), the sum of squares of deviations from the two parallel lines is given by (6.7). Inserting the solutions for aI, a2, and b, straightforward manipulation leads to

Since we have fitted three parameters to the data, this sum of squares has 3 degrees of freedom. We now proceed to test whether the two parallel lines are identical, i.e., lie on top of each other. If the true lines (6.5) are identical, then '71 = 'fJ2 for all X, and hence

"I + "2 -

(6.15) and

(CX l - C(2) - {l(Xl -

x2) = o.

(6.16)

x2)

(6.17)

It follows that the quantity

(al - a 2) - b(Xl -

will have expected value zero, and be distributed normally with variance

(J'2

"I

+ (J'2 + (Xl _

X2)2V[b]

= (J'2[.l + ~ +

"2

111 ,

112

2 ~~l - X2)2 ]. . '" '" ( . _ -:.)2 (6.18) ,£.,,£., X'V X, v

i

Thus, if the lines are identical,

S [1/11 1

+

(a l - a2) - b(Xl 2 1/112 + (Xl - X2)2 /

x2)

t~ 11;

~

r-.;

t

(

"I + 112 -

3)

(XiV - Xi)2] (6.19)

where s is derived from the sum ofsquates in (6.14). A numerical example will be discussed in Section 11.8.

352

SIMPLE LINEAR REGRESSION

CHAP.

11

11.7. Parallel Line Biological Assay An important application of the foregoing section is to a common form of biological assay for vitamins, hormones, etc., in which the response of the organism, usually an ~nimal, over a certain range is linearly proportional to the logarithm of the dose. For complicated substances not readily susceptible of chemical analysis this provides a method of assaying the potency of an unknown preparation in terms of a standard. Suppose that animals receive various known log doses xlv of the standard and other animals receive various known log doses x 2• of the unknown. Then the responses Yl> Y2 may be fitted by two straight lines, one for the standard and one for the unknown, with common slope b: Yl

= al + b(x -

Y2

Xl),

= a 2 + b(x - x2).

(7.1)

These lines are estimates of the corresponding true lines: "II

= otl + {3(x -

Xl),

1]2

=

ot 2

+ {3( x

-

x2)·

(7.2)

If the basic preparations of the standard and the unknown from which the various doses of each have been prepared are of the same potency, then the two lines (7.2) will be identical, but if, e.g., the unknown is p times as potent as the standard then it will take IIp units of the unknown to give the same response as is given by 1 unit of the standard. Graphically, the line for the unknown will be displaced horizontally relative to the line for the standard by an amount equal to log p = p" say. Since the lines have the same slope, they are parallel, and therefore the horizontal distance between them, p" is the same for all values of the response 1]. Let ~l be the value of Xl when 1]1 takes some convenient value, say (otl + ot z)/2, and let ~2 be the value of X 2 when 1]2 takes the same value. Then ~1 - ~2 = p" and otl

+ {3(~1 -

Xl)

= otl + ot2 = ot2 + {3(~2 - x2), 2

(7.3)

whence P,

1: = S"l -

1:

S"2

=

-

Xl -

-

X2 -

otl -

ot2

---.

(7.4)

{3

The difference ~l - ~2 = P, is the difference in the logarithms of equivalent quantities of the standard and unknown; i.e., p, is the logarithm of the potency ratio, and the antilogarithm of p" p, is the potency ratio. To obtain an estimate M of p, we replace otl' ot2 and {3 in (7.4) by the estimates aI, a2 and b. From (6.11), al = ih, a2 = Y2' and b is given by (6.12):

(7.5)

SECT.

11.7

353

PARALLEL LINE BIOLOGICAL ASSAY

The estimated potency ratio, say 1', is the antilogarithm of M. To obtain confidence limits for the logarithm of the potency ratio /-" consider the variable z defined as

z = -(th - Y2) Its expected value is

=

E[z]

substituting for

b[(~l

-

~2)

- (Xl - x2)].

-(IXI - 1X 2) - P[(~l - ~2) - (Xl -

~l -

~2

V[z] = V[fjl]

(7.6)

x2)] = 0,

(7.7)

from (7.4). The variance of z is

+ V[Y2] + [(~l -

~2) - (Xl - X2)]2 V[b]

(7.8)

(7.9)

This is similar to (5.7), with a replaced by th - Y2' ~ by ~l Xl -

x2, 11k by 1/111 + 1/112, and

~ (Xi - X)2 by

2

lIi

i

V

!! (XiV -

~2' X

-

by

Xi)2, and with

fi' and 11m omitted. Equation (7.9) can thus be handled in the same way as (5.7). If we define gas

(7.10)

analogous to (5.11), we get as the confidence limits for

(Xl - X2) _

+

~l -

~2

= /-"

-(Yl - Y2)lb

1-g Is [(1 _ b(1 - g)

g)(.!. + l) + 111

112

b2

(Yl - Y2)2 ]~ (Xiv _ Xi)2 '

±! •

(7.11)

v

where t takes the probability levels P2 and PI to give P 2 - PI confidence limits. For a comprehensive review of the statistical problems in biological assay see two books by Finney [4,5]. The first deals mainly with the awkward situation where the response at each x is not a continuous variable but instead an all-or-none affair, e.g., alive or dead. At each dose

354

SIMPLE LINEAR REGRESSION

CHAP.

11

we have a proportion hi of animals surviving. The fitting of a regression line of h on x is not straightforward since the hi are binomial variables with variances 0i(1 - 0i)/n i which are not constant but instead a function of 0. In fitting the line the points hi have to be weighted inversely as their variances. Furthermore, the variances involve the 0i which are unknown but which can be estimated from a provisional line. This will give a better line, which will give better estimates of the Oi' which give a still better line. This iterative procedure converges, but a number of theoretical problems are involved, and the calculations in practice are tedious. The second text of Finney is a comprehensive examination of all types of bioassay. See also Emmens [6]. 11.S. An Example of Parallel Line Biological Assay In an assay of an estrogenic hormone [6], three groups of rats received 0.2, 0.3 and 0.4 mg of the standard and two groups received 1 and 2.5 mg of the unknown. Table 11.4 gives a linear function of the logarithm of

the weight of the rats' uteri. The '11th (v = 1, ... , niJ) observation at the jth level (j = 1, ... , m i ) with the ith preparation (i = 1 for the standard and i = 2 for the unknown) is (x iiv , Yo.)' The lower part of Table 11.4 assembles certain sums, sums of squares and sums of products. We further need

"'1

'IIlj

j

v

k"" k""

(x liv - X1.. )2

"'1 '1111 2

= k"" k"" •x lJv j

"'1 '1111 )2 ( ~' "'11v Xliv

v

(8.1)

""

k Illi i

= 4.175858 _ (8.~!2)2 = 0.254205, (8.2)

tnt fllJ "'1 "11

"'1 'IIlJ

11 (Xliv - x1J(Yliv i

v

fh) =

11 X 1ivY1jv j

v

1nl flU

11 Xliv 11 Yliv i

v

"'1

j

v

(8.3)

""

kill} j

= 794.598 _ 8.632 X 1721 = 12.720526 19

Table 11.4*

Dose, mg log (dose x 10)

=x

0.2 0.301 1

0.3 0.477 1

0.4 0.602 1

1

2

3

1.0 1.000 2

2.5 1.398 2

1

2

ml

j

L

ni;

= Yiiy

77 93 116 78 87 86 101 104

118 85 105 76 101

6

8

5

494

742

41,912

70,100

73 69 71 91 80 110

>-i

00 m2

L

;

Response

'"trIC"l

Unknown

Standard

;

79 87 71 78 92 92

s:: '1:j

101 86 105 111 102 107 102 112

t"" tTl

0

'rl '1:j

> := > t""

t"" tTl t""

C Z

19

6

8

14

485

1,721

499

826

1,325

48,151

160,163

41,863

85,744

127,607

nij

LYiiV

tTl

><

>

v

tTl

tl:1 ....

0

t""

0

0

'nij

LY~;v

•

t""

nij

LXi;v

•

n >

6 x .301

8 x .477

6 x .301 2

8

5 x .602

8.632

6 x 1.000

8 x 1.398

4.175858

6 x 1.0002

8

17.184

> """" > ><

ni;

LX:;v

X

.4772

5

X

.6022

X

1,398 2

21.635232

'nii

LXiivY;;v

*

494 x .301

Data derived from [6].

742 x .477

485 x .602

794.598

499 x 1.000

826 x 1.398

1653.748 Vol VI VI

356

SIMPLE LINEAR REGRESSION

CHAP.

11

The similar quantities for the unknown are 0.543100, 2205.214, and 27.405143. The sum of squares due to regression for the standard is (12.720526)2 = 636.543 (8.4) 0.254205 ' and the residual sum of squares about the regression line is 4276.632 636.543 = 3640.089; so s~ = 3640.089/17 = 214.123. Likewise s~ = 68.528. The variance ratio 214.123/68.528 = 3.12 is rather close to FO•975(17, 12), 3.13, but the null hypothesis is acceptable at the two-sided 0.05 level of significance. The joint s2(6.2)is (3640.089 + 822.334)/(17 + 12) = 153.877. y

104 102 100

/

98

:t-

96 94 92 90 88 86 84

+

/

,+

82

x

0.2 Figure 11.3

The next step is to compute bi = 12.720526/0.254204 = 50.041 and b2 = 27.405143/0.543100 = 50.461. The estimated variance of bi - b2, using (6.3), is b2 ] = 153.877( 1 + 1 ) = 888.660 0.254204 0.543100 whence the statistic for testing the null hypothesis PI = P2 (6.4) is V[b i

-

(8.5)

50.041 - 50.461 = -0.014, ~888.660 which is distributed as t(29) and is obviously nonsignificant. We therefore form a joint b as given by (6.12), b

= 12.720526 + 27.405143 = 40.125669 = 0.254204 + 0.54310

0.797304

50.326687

(8.6) '

SECT.

11.8

EXAMPLE OF PARALLEL LINE BIOLOGICAL ASSAY

357

and a new residual sum of squares about the two parallel regression lines as given by (6.14): R = 4276.632

+ 2205.214 -

(40.125669)2 = 4462.454. 0.797304

(8.7)

This has 19 + 14 - 3 = 30 degrees of freedom; so the new S2 = 4462.454/30 = 148.748. The test for whether these two parallel lines can be regarded as a single coincident line is given by (6.19), requiring Xl.. = 8.632/19 = 0.454316, X2 .. = 17.184/14 = 1.227429, Yl.. = 1721/19 = 90.578947, and Y2 .. = 1325/14 = 94.642857. The statistic (6.19) has the value (90.578947 - 94.642857) - 50.326687(0.454316 - 1.227429) = 3.057

+ 1/14 + (0.454316 -

.J148.748.Jl/19

1.227429)2/0.797304

(8.8) and is distributed as t with 30 degrees of freedom. The foregoing concludes our analysis of these data according to the procedures described in Section 11.6. The null hypothesis that the lines are parallel can be accepted, but the null hypothesis that they are coincident must be rejected. We will now apply the methods of Section 11.7 to calculate the potency ratio and its confidence limits. The logarithm of the potency ratio is given by (7.5): M

= (0.454316 _

1.227429) _ 90.578947 - 94.642857 50.326687

= -0.773113 - (-0.080751) = -0.692362 = 1.3076. (8.9)

The antilogarithm of this, 0.203, is the point estimate of the potency ratio. The exact confidence limits are given by (7.11). We first calculate gas defined in (7.10) to see if approximate limits would' be acceptable. For 95 per cent confidence limits we need (0.975(30) = 2.042, and

= g

(2.042)2 X 148.748 (50.326687)2 x 0.797304

= 0.307144.

(8.10)

If g was less than 0.05 we would consider an approximation acceptable, given by putting g = 0 in (7.11), but with g = 0.307 the exact formula (7.11) must be used: -0.773113

+ -(-0.080751) ± 1 - 0.307144

X

[(1-

2.042.J148.748 50.326687 x (1 - 0.307144)

0.307144)(~ +~) 19

= -0.4374

and

14

-0.8757

+(-0.080751)2JJ.1 0.797304 = 1.5626

and

1.1243.

(8.11)

358

SIMPLE LINEAR REGRESSION

CHAP.

11

These are confidence limits for the logarithm of the potency ratio. Their antilogarithms, (0.133,0.365), are the 95 per cent confidence limits for the potency ratio p. 11.9. Regression through the Origin

There are occasions when it appears appropriate for a regression line to pass through the origin, i.e., for the true regression line to be 'YJ =

fJx.

(9.1)

In the numerical example considered in Section 11.4, where blood flow rate was measured by a new method Y and by a standard method x, it would be reasonable to expect 'YJ to equal zero when x = O. We did find in fact that the null hypothesis that the intercept was zero was acceptable. In this section we study the fitting of a line through the origin. We assume that Y is distributed normally about 'YJ as given by (9.1) with variance 0'2, and that the observations are independent. The estimated regression equation is Y= bx. (9.2) The sum of squares of deviations between the observed values Yi and the predicted values Yi is (9.3)

Differentiating with respect to b and equating to zero to make R a minimum gives dR - = -

21 (Yi -

db

(9.4)

bx;)xi = 0,

whence b

=1

XiYi 2 LXi

(9.5)

•

It is clear that b is a linear function of the Yi' and its variance is (9.6)

Analogous to (3.1), we write Yi - 'YJi = (Yi - Yi)

+ (Yi -

Squaring and summing over i gives

'YJi) = (Yi - Yi)

+ (b

-

fJ)x i •

(9.7)

SECT.

11.9

359

REGRESSION THROUGH THE ORIGIN

the terms being distributed as ()'2X 2 with k, k - 1, and 1 degrees of freedom respectively. It follows that S2 defined as S2 = ~ (y; - Yi)2/(k - 1) has expected value ()'2 and is independent of b. We have X2()'2 V[Y] = V[bx] = x 2V[b] = - . (9.9) LX~ We now write (9.10) Yi = (y; - Y i) + Y i = (Yi - Y i ) + bx; and square and sum over i:

'" Y~ = '" ( . _ y'.)2 £., Y. •

£.,.

Thus we can calculate

~

+ (L XiYi)2

(9.11)

'" Xi2

£.,

(Yi - Y i )2, which we want as the numerator of (9.12)

To find the expected value of (~XiYi)2/~X~, we note from (9.8) that E[(b - {J)2 ~ x~] = ()'2, so E[b 2 L x~] - 2{J L x~E[b] + {J2 L x~ = ()'2. (9.13) But (9.14) Thus (9.15) We can construct a table of analysis of variance {Table 11.5) corresponding to (9.11). Applying these results to the data of Table 11.2, we have as the regression coefficient, using (9.5),

b

= L Xi;i = LXi

Source of variance

87,719,100 89,394,900

=

0.981253,

Table 11.5 Sums of Degrees of squares freedom

Due to line Residual

("2:,xiYi)2f"2:,x~ "2:,(Yi - y;)2

k -1

Total about origin

"2:,Y~

k

1

(9.16)

E[M. S.] ()"2 ()"2

+ {J2"2:,x~

360

SIMPLE LINEAR REGRESSION

CHAP.

II

as the sum of squares due to the regression line,

CL Xi~i)2 = (87,719,100)2 = 86,074,714.6, .L Xi

(9.17)

89,394,900

and as the residual sum of squares, using (9.12),

.L (Yi -

Yi)2 = 86,125,825.0 - 86,074,714.6 = 51,110.4.

(9.18)

We enter these results in Table 11.6.

Source of variance

Sums of squares

Table 11.6 Degrees of freedom

Due to line Residual.

86,074,714.6 51,110.4

1 17

Total about origin

86,125,825.0

18

Mean squares 86,074,714.6 3,006.5

E[M. S.] 02 02

+ p2~X~

We can combine Tables 11.3 and 11.6 to give a test of whether a line Y = bx through the origin is an adequate fit (Table 11.7). A test of the null hypothesis that there is no improvement in fit when using the two-parameter line in place of the one-parameter line is given by the variance ratio 238.5/3179.5 = 0.075. If this null hypothesis was to

Source of variance

Table 11.7 Sums of squares

Remainder using line Y = bx Remainder using line Y = a + b(x - x)

51,110.4 50,871.9

Difference: attributable to improvement in fit through using two parameters in place of one

238.5

Degrees of freedom

Mean squares

17 16

3179.5

238.5

be rejected at the ex level of significance, this variance ratio would have to be not less than F 1- 1X(1, 16). Clearly the null hypothesis here is acceptable. This test is identical with the test that the intercept of the two-parameter line could be zero. For that test we found t = 0.2739; so [/(16)]2 = (0.2739)2 = 0.0750 = F(I, 16), as we obtained here.

SECT.

11.10

REGRESSION LINE THROUGH THE ORIGIN IN REVERSE

361

To revert to Table 11.6, there is clearly no doubt that the null hypothesis

p== 0 has to be rejected. We will be more interested in the null hypothesis p== 1. Using (9.6), V[b] = ~ = 3006.5 = 0.3363 X 10- 4 (9.19) LXi

89,394,900

and, under the null hypothesis that b- 1 .jV[b]

=

P=

1,

0.981253 - 1 .j0.3363 x 10-4

= 3.23

(9.20)

is distributed as (17). For the line constrained to pass through the origin we thus have to reject the null hypothesis that P = 1. For the unconstrained line Y = a + b(x - x) we found that the null hypothesis that P= 1 was acceptable, and that the null hypothesis that the intercept is zero was acceptable. We might have expected on this basis that, if we made the intercept zero, i.e., switched to the line constrained to pass through the origin, then we could continue to accept the null hypothesis that P= 1. However, the above result shows that this is not the case. We will gain greater insight by constructing a joint confidence region for (J. and P, and for the intercept and p. This we will proceed to do in Section 11.11.

11.10. The Use of the Regression Line through the Origin in Reverse Suppose that we observe a new fi', and we wish to predict the corresponding x and to construct confidence limits for this prediction. We can solve the estimated regression equation (9.2) to obtain a point estimate, x = fi'lb. The expected value of the new fi' is 'YJ. Corresponding to this value of'YJ is a value of x given by solving the true regression equation (9.1) forx,x = 'YJIP. Denotethisvalueofxby so that = 'YJIP,or?7 - pe = O. We now define a new variable z,

e,

z

=

e

fi' - be.

(10.1)

This variable will have expected value

E[z]

= 'YJ

- eE[b]

and variance V[z]

=

= 'YJ

- ep (j2

V[y']

= 0,

(10.2)

0'2

+ e2 V[b] = -111 + e2 ,£., '" 2 Xi

(10.3)

Substituting the residual mean square in Table 11.5 as an estimate of (j2, we have that (10.4)

362

SIMPLE LINEAR REGRESSION

CHAP.

11

The subsequent manipulation proceeds along lines similar to Section 11.5, and leads to by' b2 _ t:s 2/2,

x~ -

t2s b2 -

t~S2/2, x~

[(

2

b -

t~s2 ) 1

2, x~

as the solution for ~2' Use of 11 in place of t2 gives If we define g as

m

y,2 J!-2 + 2, x~

(10.5)

~1'

(10.6) analogous to (5.11), then, when g is less than 0.1, b is highly significant and (10.5) can be approximated by (10.7) Applying this result to the example of Section 11.4, we will suppose that we observe a single new observation, y' = 3000. For 95 per cent confidence limits, 10.975(17) = 2.11 0, and, from Table 11.6, S2 = 3006.5. We have ~x~ = 89,394,900 and b = 0.981253. Thus, using (10.6), the value of g is = (2.ll0? x 3006.5 = 0.000156 g (0.981253)2 x 89,394,900 ' which clearly permits us to use the approximation (10.7). The 95 per cent confidence limits are 3000 ± 2.110 x ~3006.5[~ + (3000)2 J!-2 = 3057.3 ± 128.4. 0.981253 0.981253 1 86,074,714 The resulting interval (2929,3186) compares closely with that obtained using the two-parameter line in Section 11.5 (2927,3198). 11.11. A Joint Confidence Region for a, {3

As indicated in Section 11.9, we may need a joint confidence region for a, {3 to show those pairs of values which are compatible with the data. By the discussion of (3.2), k(a - a)2 and (b - (3)2 ~(Xi - X)2 are distributed independently both as a2X2(1). From (1.27.16) their sum will be distributed as a2x 2(2). Also, S2 is independently distributed as a2x2(k - 2). Therefore

SECT.

11.11

A JOINT CONFIDENCE REGION FOR

In Pr{F < F1_d<}

=

1-

(l,

(l,

{3

363

we replace F by the left-hand side of (11.1):

Pr {k(a - (l)2+ (b ~/)21(Xi - X)2 < F 1_aC2, k _ 2)} = 1 _ (l. (11.2) All pairs of values of «(l, {3) which satisfy this inequality lie inside the 100(1 - (l) per cent confidence region, and the boundary of the region is given by substituting an equality for the inequality:

+ (b -

k(a - (l)2

2s

{3)21 (a:i

X)2

-

=

2

F 1-

(2 k _ 2) a

•

,

(11.3)

To determine the boundary of the confidence region, we have to find those pairs «(l, {3) which satisfy (11.3). IX

T

I

I

I

I

I

I

I

-

I-

2160 2140

2120 2100

-.;;;::::,:.

i--"

V /

I-

1-( 1-\

~

I)

r-....

............ I-

2080 0.90

I

r--

I

0.92

-

I

I

0.94

0.96

I

0.98

- l..---' I

1.00

-

-

/

I

1.02

I

1.04

Figure 11.4

For the data of the example in Section 11.4:

L (xi -

b = 0.974561 k = 18, a = fJ = 2125.277, X)2 = 5,024,850.0, S2 = 3179.49, FO•9S(2, 16) = 3.63.

Inserting these values in (11.3) gives the equation 18(2125.277 - (l)2

+ 5,024,850

X (0.974561 - {3)2

= 23,083.1.

(11.4)

When {3 = 0.974561, (l = 2089.468 or 2161.088. When (l = 2125.277, P= 0.906784 or 1.042338. Choosing values of {3 in this range, we get pairs of solutions for (l, and, plotting these and connecting them up, we get the ellipse in Figure 11.4. All pairs of values of (l, {3 inside this ellipse are jointly compatible with the data at the 0.05 level of significance.

SIMPLE LINEAR REGRESSION

364

CHAP.

11

We are, however, more interested in a joint confidence region for the intercept, say 1}0, and {3. This is easily obtained, since 1J0 is given in terms of at and {3 by inserting x = 0 in (2.1): 1Jo

=

(11.5)

{3x.

at -

Thus, for all our pairs of solutions of

at

and {3 we compute 1J0, and then

180 160 140 120 100 80 60 40 20 7/ 0

0 -20 -40

r ........

\ \

~ \

\.

\

'" ""

'\

'\ "" "\ '\. ~

"- I\.

f\.:

~ ~

"I\.

"'r--.

""

-60 -80 -100

'\

[\.

\

\ "r--.."- \ \

" '-..J

-120 -140

-160 0.90 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 1.00 1.01 1.02 1.03 1.04 1.05 fj

Figure 11.5

plot 1Jo against {3 and again come up with an ellipse (Figure 11.5). This shows that, if we want a large {3, say {3 = 1.03, then we must have 1Jo in the range -122, - 87. If we want a zero intercept 1Jo, we must have {3 in the range 0.967 to 0.9965, or, if we want {3 = 1, we must have an intercept in the range -71 to -9. The ellipse excludes the point {3 = 1, 1Jo = O. In other words, we can have a zero intercept but not a slope of 1, or we can

SECT.

11.11

A JOINT CONFIDENCE REGION FOR CY.,

{3

365

have a slope of 1 but not a zero intercept; we cannot have a slope of 1 and a zero intercept simultaneously. Having constructed the ellipse for the 95 per cent confidence region for (J. and {3, Figure 11.4, we know that any point (IX, (3) that lies outside the ellipse corresponds to a line that is inadmissible in the sense that it lies outside the 95 per cent confidence region for the line. Any point (IX, (3) that lies on the ellipse gives a line that is just admissible. If we took a whole sequence of points all round the ellipse and drew the corresponding lines we would generate two envelopes, and all lines lying inside the envelopes lie inside the 95 per cent confidence region for the line. We can construct the boundary of the confidence region for the line more easily by obtaining an explicit formula for it. Points on the boundary will be given by finding maximum and minimum values of 1]

= IX + (3(x -

(11.6)

x)

for any and all fixed values of x, with IX and {3 subject to the condition k(a - OC)2

+ (b -

(3)2 L (Xi - X)2 - 2Fl_i2, k - 2)S2

= 0,

(11.7)

derived from (11.3). The standard procedure for finding the extreme value of a functionj(x, y) subject to the subsidiary condition cp(x, y) = is Lagrange's method of undetermined multipliers. This involves solving the equations

°

al(x, y) + A acp(x, y) ax ax al(x, y)

ay

= 0,

(11.8)

+ Aacp(x, y) = 0,

(11.9)

ay

where A is a constant, the undetermined multiplier, along with the subsidiary condition (11.10) cp(x, y) = 0. Here for j(x, y) we have 1] = j(lX, (3) given by (11.6) and for cp(x, y) we have (11.7) considered as a function of IX and (3. For (11.8) and (11.9) we obtain (11.11) 1 + A(-l)' 2k(a - IX) = 0, (x - x)

+ A( -1) . 2(b -

(3) L (Xi - X)2

= 0,

(11.12)

whence

1 a-IX=-

(11.13)

2kA'

b-{3=

x-x

2A L(xj

-

. X)2

(11.14)

366

SIMPLE LINEAR REGRESSION

Inserting these expressions for a resulting equation for A gives

Ot

1 [1

CHAP.

and b - {3 in (11.7) and solving the

J~.

(x - X)2 X)2

A=±----=- -+

11

2../2Fs k L (Xi -

(11.15)

Substituting this solution for A in (11.3) and (11.4) gives Ot

(x - X)2 J-~, X)2

= a ± ../2FS[! + k

k

(11.16)

L(xi

X)"/2is[! + (x - x)~ 2J-~. L (Xi - X)2 k L (Xi - x)

{3 = b ± (x -

Finally, substituting these solutions for 'YJ =

a + b(x -

x)

±

Ot

and {3 in (11.7) gives

( ../2Fs [ -1 + LX - -)2 x _ J~ k

(11.17)

2

(Xi -

x)

•

(11.18)

For our values of a, b, s, k, x, ~ (Xi - X)2, and F1_q,(2, k - 2), this will give a pair of points for any x, and by moving x over the line we generate the hyperbolas which are identical with the envelopes to the admissible lines. Equation (11.18) is similar to (3.20), the difference being the use of J2F in place of t. The difference in interpretation is that while (3.20) gives a confidence interval for 'YJ at one, and only one, specified value of x, (11.18) gives confidence intervals for 'YJ at all values of x, and can therefore be used repeatedly. 11.12. Linear Regression with Several Observations on y at Each x In this section we assume that at Xi' i = 1, ... , k, we have ni observations Yi," 'II = 1, ... , ni • We define x as the mean of the Xi weighted in proportion to the numbers of observations at each point: k

L ll i Xi

X=_i __ . k

(12.1)

Li Ili ni

We define fit as Lyi.llli and compute the sample variance within each v

group of replicate observations: (12.2)

SECT.

11.12

LINEAR REGRESSION WITH SEVERAL OBSERVATIONS

367

In principle, we might test the null hypothesis that the S~i are drawn from a population with common parameter 0'2 with Bartlett's test. In practice, this will be frequently unrewarding since the n i are often small and the test will be of low power. Also the S~i are ordered by the magnitude of the Xi from which they were obtained, and Bartlett's test pays no attention to this ordering. If there is a departure from the null hypothesis, it is likely to be that 0'2 increases with X, instead of varying irregularly and it may be more profitable to make a rough graph of S~i against Xi and inspect this with common sense and judgment. Frequently, we must admit, no attempt is made to check this assumption, and one proceeds directly to form a pooled estimate of 0'2, according to (9.7.6): k

=

s~

k

til

I.,!.i_(ni_- _l)s~ I I _ =

(YiY - Yi)2

-"i~Y_ _ __

k

(12.3)

k

Ii

Ii

(ni - 1)

ni - k

The sum of squares of deviations between the observed values Yiv and the predicted values Yi is k

R

til

= II(YiV i

- Yi )2

k

til

i

v

= II[YiV -

v

a - b(Xi - X)]2.

(12.4)

Partially differentiating R with respect to a and to b, and equating to zero gives

oR - =

oa

k

-2 I i

oR - = ob

k

-2 I

til

I

[YiY - a - b(Xi - x)]

= 0,

(12.5)

Y

til

I

i

[YiY - a - b(Xi - X)](Xi - x)

= o.

(12.6)

Y

Rearranging, k

k

aI

k

Ili + b I lli(Xi i i

k

a Illi(Xi

-

x) + b

i

k

I

til

I I i v x) =

(12.7)

YiY,

k

III;(Xi i

ni(xi - X)2 =

i

X)Yi'

(12.8)

x)fji'

(12.9)

since k

k

til

Ii Iv (Xi -

X)YiY =

I

i

til

(Xi - x) I YiY = v

k

I II;(Xi i

The definition of x in (12.1) implies that k

III;(Xi i

k

k

IIl;Xi - XIlli = i i x) =

0;

(12.10)

368

CHAP.

SIMPLE LINEAR REGRESSION

11

so (12.7) gives k

k

111

!i

!!Ytv

a=

i

v

=

k

n/fit

-k-'-

!n i

= y,

(12.11)

!ni

i

i

and (12.8) gives k

k

b=~i

~i

! _________ lI i(Xi - x)fit ! lIlXi - x)(fii - fi) = _____________ k

!

(12.12)

k

!

ni (Xi - X)2

i

lI i(Xi - X)2

i

the second form following from the identity k

k

! lI i(Xi - X)(fii - fi) i

k

=!

ll i(X; - x)fii' - ! ni(x; - x)fi

i

i

k

= !i

(12.13)

nt( Xi - x)fii'

We will now consider an analysis of variance representation. The deviation of an observation Yiv from the value 'YJi predicted by the true regression equation (2.1) can be written as Ytv - 'YJi = (Yiv - fii) + (fii - 'YJi) = (YiV - fit) + (fit - ~) + (~ - 'YJi) = (Ytv - fit) + (fii - ~) + (a - a;) + (b - (J)(Xi - x). (12.14)

This equation is represented in Figure 11.6. Squaring and summing over i and 'JI gives k

1Ii

! ! (Yiv - 'YJi)2 i

= (a

k

- a;)2! 11;

v

+ (b -

i

+!

k

i

k

{J)2! ll i (Xi - X)2

i k

11/

ll i(fit - ~)2 + ! ! (Yiv - fii.)2. i

(12.15)

Y

k

In this equation, we have partitioned a sum of squares with! lli degrees i

of freedom into four sums of squares. The first and second components each involve only one variable, a and b, respectively, and these each have one degree of freedom. The third component involves k variables, but equating (12.5) and (12.6) to zero imposed two restrictions, and so the degrees of freedom are k - 2. The fourth component is the numerator of (12.3), which has

(f lli) -

k degrees of freedom. Thus the sum of the

sums of squares equals the sum of squares on the left-hand side, and the sum of the degrees of freedom, likewise; so, by Cochran's theorem,

SECT.

11.12 LINEAR REGRESSION WITH SEVERAL OBSERVATIONS

369

the sums of squares are distributed as ()'2X 2 and are independent. If k

(a - OC)2 L ni is distributed as ()'2X2 with one degree of freedom, analoi

gously to (3.4), (12.16)

Likewise, V[b]

= E[(b

()'2

= -k---

- E[b])2]

L "i(Xi

-

(12.17)

X)2

i

Y

(Xi, Yip) (Yi-;-yi)------------~;,Ji)

(fi:gT:___-_-_-_-_-_-___ __________ _ ~

(yt - 7I i)= (a-a)+b(xi -x) -(J(Xi -x) =(a-a)+(b-(J)(xj -x)

------------------- -------(x, a)

__

(x;~1iJ

___-~__-_-_- (i:~~_ ~:i -x)

~~== =~_-_(x,

----1--

a)

~----~~~------~L-----------~------------~x Xi

Figure 11.6

Since a and b are independent, V[Y] = V[a

+ b(x -

x)] = V[a]

+ (x -

x)2V[b] (12.18)

We now write (12.19)

370

SIMPLE LINEAR REGRESSION

CHAP.

11

and square and sum over i and 'JI: k

k

nj

i

i

v

k

nj

= ~ ~ (Yiv -

~ ~ (Yiv - fi)2

fii)2

+ ~ Il i(fii -

~)2

k

+ ~ Ili~ _

fi)2.

v i i

(12.20) The first item on the right-hand side is the numerator of s~ in (12.3), distributed as (12X 2 with ( ~ IIi) - k degrees of freedom, and the corresponding mean square has expected value (12. The second item is the third item in (12.15), and if the 'I'/i really lie on the regression line oc + (3(Xi - x), then this sum of squares is distributed as (12X2 with k - 2 degrees of freedom. It is possible, however, that the 'fit do not lie on this line. Even if the 1]i do not lie on a straight line, it will be possible to fit a least squares line to the points (Xi' 1]i), and the parameters will be, by analogy with (12.11) and (12.12), k

~ IIi Xi - X)1]i

(3=..!.~----

~ nixi -

(12.21)

X)2

i

We can write Yi) + (Yi - 1]i) Yi) + [a + b(Xi - x) - ij - (3(Xi - x)] - [1]i - ij - (3(Xi - x)] = (fii - Yi) - [1]i - ij - (3 (Xi - x)] + (a - ij) + (b - (3)(Xi - Xi)' Square and sum over i and 'JI:

fii - 1]i

= (fii = (fii -

k

~ nifii - 1]i)2 i

(12.22)

k

= ~ ni{(fii i

+ (a

~) - [1]i - ij - (3(Xi - x)W k

- ij)2 ~ IIi

k

+ (b -

(3)2 ~ nixi - X)2.

i

(12.23)

i

The left-hand side will be distributed as (12X2 with k degrees of freedom. The terms on the right-hand side will be distributed (12X2 with degrees of freedom k - 2, 1, and 1, respectively; so, considering the first term,

E[~ ni{(fii - ~) = (k

[1]i - ij - (3(Xi - X)W] -

2)(12

= E[~ ni(fii - ~)2] + E[~

l

-2E[~ nifii - ~){1]i -

i {1]i

-

1'j - (3(Xi -

xw]

ij - (3(Xi - xn} (12.24)

SECT.

11.13

371

LINEAR REGRESSION WITH SEVERAL OBSERVATIONS

The last term here can be written as k

-22 ni{r/i -

ij - {3(x; - x)}EUli - ~]

i

k

= -22 ni {17i -

ij - {3(Xi - X)}{17i - oc - (3(Xi - Xi)}

i

k

= -22 ni{17i - 17 -

(12.25)

{3(Xi - xW,

i

since oc

= ij.

Then (12.24) can be rearranged to give

E[~ nkili - ~)2]

= (k - 2)0'2

+ ~ ni{17i -

ij - {3(Xi - xW.

(12.26)

The same arguments that gave (3.16) here give

E[~ nDi - y)2]

=

0'2

+ (32 ~ ni(xi -

(12.27)

X)2.

We can now put (12.20) in a tabular form (Table 11.8). Table 11.8 Source of variance

Sums of squares

Degrees of Mean squares freedom

E[M.S.] k

k

Slope of line

~ "i( Y;

s~

- fli)2

i

Variation of true group means about line

0-2

+ {J2 ~i lI;(XI -

f)2

k

k

~ "i(fl i -

S2

k-2

y i )2

2

i

Within groups

~• ~v (YiV -

flY

(i lIi)

Totals

i -£ (Yiv -

fI)2

(i lIi) -

i

v

•

0-2

+ ~ "i(17; - ii -

{J(Xi - f»2

-,-i _ _-;-~ _ __

k-2

- k

1

i

A test of the null hypothesis that the 17i do lie on a straight line is given by the variance ratio s~/s~, and a test of the null hypothesis that {3 = 0 is given by the variance ratio s:/s~.

11.13. An Example of Linear Regression with Several Observations on y at Each x The data of Table 11.9 give the square root of the stopping distance in feet y of an automobile when traveling on a road with a certain surface at a number of speeds in miles per hour x. The results are only directly applicable to that automobile on that road; if we want to make a general

372

SIMPLE LINEAR REGRESSION

CHAP.

11

Table 11.9 xi

20.5

Yip

C'92 3.65

30.5 5.82 5.20

40.5 8.55

48.8 10.63

57.8 11.94

k

2

ni

!ni = 7

2

k ni

"I

7.57

!Yi"

11.02

8.55

10.63

11.94

!!Yi" = 49.71 i

v

v

inference about other automobiles of the same model or about other roads, we can only do it with data either from a sample of automobiles and from a sample of roads, or from a substantial body of experience that tells us that the variation between automobiles of a certain model and between roads is very slight. The reasons for choosing the square root of the stopping distance as the dependent variable are that a plot of the distance itself against speed showed obvious signs of curvature, whereas the square root appears to be linear (Figure 11.7). Since the energy of a moving object is proportional to the square of its velocity, this is reasonable. We could have chosen the stopping distance as the dependent variable, and the square of the speed as the independent variable, and this would have come to almost the same thing. However, one of the assumptions of the regression analysis is that y

12 11

10 9 8 7

6 5 4

3 2

Figure 11.7

SECT.

11.13

LINEAR REGRESSION WITH SEVERAL OBSERVATIONS

373

the variance of the dependent variable, given the independent variable, is constant, and this is much more likely to be so when we use the square root than the stopping distance itself. It is comparatively rarely that we have sufficient data to discriminate objectively between different possible models, and usually all we can do is to make plausible guesses based on previous experience in similar fields. Table 11.8 involves four sums of squares which we will now compute. The general procedure is to convert averages into sums over the appropriate index. The total sum of squares is, analogous to (10.2.35),

if (Yiv -

fj)2 = (3.92)2

+ ... + (11.94)2 _

(49.71)2

7

v

i

= 418.2643 - 353.012014 = 65.252286. The within-groups sum of squares is, analogous to (10.2.33),

~ ~ (YiV -

fji)2 = 418.2643 _

(13.1)

[(7.~7)2 + (11~02)2 + ... + (11:4)]

= 418.2643

- 418.035650 = 0.228650. (13.2) The sum of squares for the slope of the line is, analogous to (3.12), k

k

~ ni(li - fj)2

=

b2

t

Z ni(xt -

X)2

=

[i

nixi -

i

X)(fji -

jj)T (13.3)

k

Z ni ( Xi _

i

X)2

i

For this we need k

I ni(xi i

k

X)2

= Z nix~ i

(*

nixi)2

(13.4)

k

Z l1i (20.5)2 + 2 X i

= [2 X

(2 X

+ ... + 1 X (57.8)2] 20.5 + 2 X 30.5 + ... + 1 X (30.5)2

57.8)2

7 = 1199.128572

(k

and k

I i

_ _ _ k n;(xi - X)(Yi - y) = Xi

"i

Z Z i

t ~ tk "i

) (

)

Yiv l1iX; Yiv - ....:....:..~---'k'---'-''----.:..

v

(13.5)

~

k l1i i

= (20.5 X 7.57 + 30.5 X 11.02 + ...

= 277.480143,

+ 57.8 X

11.94)

49.71 X 249.1 7

374

SIMPLE LINEAR REGRESSION

CHAP.

11

whence, substituting in (13.3),

~ n.(r _ -)2 = (277.480143)2 = 64.209320.

f"

(13.6)

1199.128572

y

Finally we want the sum of squares for the variation of the true group means about the true line: k

! nkY. i

k

yt)2 =

! n.[y. -

ii - b(x. - X)]2

i

= (418.035650 -

353.012014) - 64.209320

=

(13.7) 0.814316.

We now have the numerical values of all the mean squares, and can enter them in Table 11.10. Table 11.10

Sums of squares

Degrees of freedom

Mean squares

Slope of line Variation of true group means about the line Within groups

64.209320

1

~4.209320

0.814316 0.228650

3 2

0.271439 0.114325

Total

65.252286

6

Source of variance

The test of linearity is given by 0.271439/0.114325 = 2.37 which under the null hypothesis is distributed as F with degrees of freedom 3 and 2. The 0.90 point is 9.16; so obviously the null hypothesis is acceptable. The test of the null hypothesis {3 = 0 is given by 64.209320/0.114325 = 562, which is distributed as Fwith degrees offreedom 1 and 2. The 0.995 point is 199; so obviously we reject the null hypothesis. However, many would take a point of view that either the true regression is linear or it is not. If it is, then the expected value of s~ is 0'2, and they would pool the sums of squares and degrees of freedom of s~ and s~ to get a new estimate of (J2, say

S~2,

with degrees of freedom

(* n.) -

2 = 5. On the other hand,

if the test of linearity leads to the rejection of the null hypothesis, then there is little point to a test of the null hypothesis {3 = O. Deviation from

SECT.

11.13

LINEAR REGRESSION WITH SEVERAL OBSERVATIONS

375

linearity could be of two types. The 1]; could lie erratically on either side of a line, the deviation having no systematic pattern, or 1]; could lie on some more complicated curve such as a parabola. One would plot the points and inspect them to decide what the situation was. The estimates of the parameters for the regression line are a = 49.71/7 = 7.101429 from (12.11) and b = 277.480143/1199.128572 = 0.231401 from (12.12). Also x = 249.7/7 = 35.585714 from (12.1). Thus the estimated line is

y

= 7.101429 t

O.231401(x - 35.585714)

=

-1.133141

+ 0.231401x. (13.8)

The pooled estimate S~2 of ()'2 is

S~2 = 0.814316 + 0.228650 = 0.208593. 3+2

(13.9)

= 0 equivalent to the one above can be made by substituting S~2 for ()'2 in (12.17) which gives the estimated variance ofb as V[b] = 0.208593/1199.128 = 0.0001740. Then

A test of the null hypothesis f3

b f3 "V[b] / -

f"'-.I

t

(k!

IIi -

)

(13.10)

2 .

;

Putting f3 = 0, we have t(5) = (0.2314 - 0)/../0.0001740 = 17.5. Ninety-five per cent confidence limits for f3 are 0.2314 ± 2.571 X 0.01319 or (0.1975, 0.2653). We might ask what is the predicted square root stopping distance when x = 50: (13.11) Y = -1.133141 + 0.231401 x 50 = 10.437. Using S'2

= 0.208593 with 5 degrees of freedom in place of ()'2 in (12.18), V[Y] =

0.208593[~ + (50 7

35.5857)2J = 0.065942 1199.128 '

(13.12)

whence 95 per cent confidence limits for 1} at x = 50 are ±2.571 X v'0.065942 = ±0.660 on either side of Y, i.e., (9.777, 11.097). In this case, if we are interested in not hitting something, a one-sided confidence limit might be more reasonable; for example, using to.01 (5) = -3.365 to get an upper confidence limit. 10.437 - (-3.365)

X

../0.065942

=

11.301.

(13.13)

376

SIMPLE LINEAR REGRESSION

CHAP.

11

Thus we are 99 per cent confident that 17 < 11.301. However, we are probably more interested in a prediction limit for an individual observa_ tion, for which we use the equivalent of (3.23), which gives V[y - Y]::: 0.274535. The one-sided 99 per cent prediction limit is 10,437 + 3.365.)0.274575 = 12.200. 11.14. The Comparison of Several Regression Lines: Simple Analysis of Covariance The comparison of two regression lines (Section 11.6) fell into three parts; first, testing that the variance around the separate lines could be regarded as homogeneous; second, testing whether parallel lines through the respective means could be regarded as an acceptable fit; and third, testing whether these separate parallel lines could be regarded as coincident, i.e., as a single line. This last operation is subject to a slightly different interpretation. Suppose that the two estimated parallel lines are (14.1) The observed averages of the y's of the two samples are Yl and Y2' However, the x averages of the two samples were in general different, being ii\ and x 2• We might ask what the y averages would have been if the x averages had been the same, say equal to the weighted mean for the two samples, (14.2)

Inserting this value of x in the two equations (14.1), (14.3)

so (14.4) is the difference between the y averages, adjusted for differences in the x averages. But this is precisely the same quantity as (6.17), which we developed for testing whether the lines are identical. In other words, the test that the true lines are identical is the same as a test of whether the 17 averages are the same after adjustment for differences in the x averages. A test of whether a group of population means is identical is known as the analysis of variance. Here we are making an analysis of variance

SECT. 11.14

COMPARISON OF SEVERAL REGRESSION LINES

377

with adjustment for variation in x; this is called, perhaps not very felicitously, the analysis of covariance, the variable x being called the covariate or concomitant variable (see Figure 11.8). An example suitable for covariance analysis is a study of the marketable weight of hogs fed different rations; it might be appropriate to use covariance on the weight of each hog at the start of the feeding trial. It would be most inappropriate to use as a covariate the amount of ration consumed per day, because this might be influenced by the ration itself.

12(x)t------------7f"

nr------~~ Y2r------~~~---~~

Yi(x)r-----T---~----¥

Xl X Figure 11.8

X2

In pharmacological experiments, some initial measurement on each experimental subject might be used as a covariate. In experiments on performance of tests by human subjects after taking drugs, or after receiving instruction, measurements made on the same or similar tests before the administration of the treatment might be good covariates. All that is required of a covariate is that it be independent of the treatments under study, and that it be correlated with the variable we are analyzing. If the covariate x selected is actually uncorrelated with y, then there is no significant loss in precision compared with a simple analysis of variance of y. The only consequence is the wasted effort that goes into the additional arithmetic. We assume that we have k groups of observations (XiV' Yi)' i = 1, " . ,k, ')I = 1, ... ,11;. A separate line can be fitted to each group, Y = Ii;. + bi(x - Xi.), and a sum of squares for variation about each

378

SIMPLE LINEAR REGRESSION

CHAP.

11

line s~ will be obtained. These may be tested for heterogeneity by Bartlett's test, and, if the null hypothesis is acceptable, pooled to give k

L (IIi -

2)S~1

S~=-"i _ _ __

(14.5)

k

L (IIi -

2)

i

There are four sets of lines to be considered: 1. The k individual lines,

y = fh

+ bi(x -

Xi),

(14.6)

where bi is, of course, lIi

L (Xiv - Xi)Yiv b·=..!.v-_ _ __ t

'Ill

Lv (Xiv -

xd

(14.7)

2

2. The k parallel lines with an average slope h;

Y

= fh + h(x -

Xi),

(14.8)

where, by analogy with (6.12), k

'Ill

L L (Xiv -

Xi)Yiv

h = ..:.i----'....v ----k 1Zi

L L (XiV i

(14.9)

- Xi ·)2

v

3. The least squares line for the group means (Xi., fh), A

+ b(x -

Y= Y..

X.),

(14.10)

where k

L II;(Xi. -

X")Yi.

b = ...:.\-.- - - - -

(14.11)

L nixi . - x.Y i

4. The overall regression line which assumes that all observations come from a single population,

Y

= Y.. + b(x - x.,),

b=

L L (XiV - X..)Yiv ...:.i,.......:-v_ _ _ __

where

(14.12)

11,

k

k

11,

i

v

LL(Xiv - XY

(14.13)

SECT.

11.14

379

COMPARISON OF SEVERAL REGRESSION LINES

The analysis of variance is based on the identity

- Yi.) + (Yiv - yJ, (14.14) where Y i • is the Y given by the overall regression line (14.12). When squared and summed over i and 'JI, (14.14) gives Y;v - Y..

k

'Il,.

k

!! (Yi. i

= (Yiv

'Ill

= !! (Yiv

Y.Y

i

l'

- Y;.)2

v

k

'Ill

i

v

+ !! (Y;. - yy,

(14.15)

where the last term is the sum of squares due to regression on the overall regression line (14.12), as in (3.10). The first term on the right-hand side of (14.14) represents the deviations of the individual observations from the values Y iv predicted by the overall regression line. This deviation can be written as

Yi. - Y;v = Yiv - [Y .. + b(Xi' - xJ] = {Yi. - [Yi. + b;(Xi. - xd]}

+ (b i + {Ui. - W. + h(xi . - xJ]} + [(6 - b)(Xi' - Xi,) + (h - b)(xi •

b)(XiV - Xi,)

xJ].

-

(14.16)

The result of squaring this equation and summing over i and 'JI is written as the second column in Table 11.11, with the following identity being used for the last term: k

11;

i

v

L! [(b -

b)(XiV - Xi,)

+ (b -

b)(XiV - XJ]2 (h - b)2

=~----------~~~----------~~

[~lIi(Xi. - X.fJ-l + [~~ (XiV - XifJ-1'

(14.17)

Proof of this identity is left as an exercise. Table 11.11 Source of variance Between band ;;

Sums of squares

[~",(ii. - i ..>"] + [~~ (XiV - ii.>"]

Degrees of Mean freedom squares

1

1

s:

k-2

sf

k-l

s'•

-1

Deviations of the group means about their regression line Between the individual slopes bl About the individual lines About the overall line

k

Li k

L

A

"i[O'. - {Ooo (b i -

i

k

ni

i

•

k

nj

LL

LL i

•

+ b(Xi. -

- 2 ni

b)

L

(.1:iV -

:f i .)

i oo)}]' 2

v

k

{Yiv - [Oi.

+ b i(·1:iV -

Xi.)]}"

{Yi. - [0

+ b(XiV -

i . .)]}'

00

L",-2k i k

L",-2 i

s·1

380

SIMPLE LINEAR REGRESSION

CHAP.

11

Each term in (14.16) has a useful interpretation. 1. Yiv - [Yi. + b;(XiV - Xi,)] is the deviation between Yiv and the value predicted by the ith individual regression line (14.6). The corresponding sum of squares is the numerator of s~ defined in (14.5). 2. (b i - h)(x iV - Xi,) is the deviation between the slope of the ith individual line (14.6) and the parallel lines (14.8), weighted by the deviation of Xiv from Xi.' 3. Yi. - [Y .. + h(xi. - xJ] is the deviation between the mean of the ith group and the value predicted by the regression line for group means, (14.10). 4. From the identity (14.17) we see that the sum of squares derived from the last term in (14.16) is a function of the difference between h, the regression coefficient for the group means, and h, the regression coefficient for the parallel lines. Table 11.12 gives the results of observations on three lime kilns. The variables X and Y were daily observations on the tons of lime made per day x and a measure of the quality of the lime y. The mean qualities Y}" Yv Ya. (column 4) were 70.767, 69.933, and 79.514. A simple analysis of variance could be made on Yiv to test the null hypothesis that there was no difference between the kilns as regards quality, but it was known that y was roughly linearly related to x, and the tonnage means xl.' Xv and xa. (column 6) were 74.100, 69.733, and 66.541. We therefore want to compare the Y means adjusted for variation in the x means. We need the sums of squares and products of Xiv and Yi," and these are given in columns 7, 8, "I

and 9. We next calculate

I,.

(Yi" - Yi)2, etc., in columns 10 through 15

of Table 11.12. The various sums of squares can be calculated as follows. The sum of squares for s~ is the sum of the sums of squares of deviations about the separate lines: Each of these sums of squares is

"j

.I (Yiv -

Yif -

[1 (XiV -

X;.)(YiV - y;.)

r

..:::-V_ _ " j_ _ _ _ _---==_

.I (Xiv -

(14.18)

Xi,)2

v

so the sum over i is k

111

1. _ y.,).I -.2 _ k ([~(XiV".-

.Ii .I CY,v v

i

Xi)(Yiv -

.Iv (Xiv I

_

Xi,)

2

ytJJ2) .

(14.19)

SECr.

11.14

381

COMPARISON OF SEVERAL REGRESSION LINES

This calculation is carried out in Table 11.12. The separate sums of squares for each line, (14.18), are tabulated in column 17, and their sum over i, (14.19), is given in the last row of column 17. The sum of squares for s~ is k

l

'1Zi

(b i

-

i

6)2

L (Xiv -

Xii

v

k

lli

Li Lv (Xiv -

XiJ 2

(14.20) = [16] - [IW = 13 158.245 _ (21,972.630)2 = 518.498

[13]'

38,196.689

where the numbers in square brackets are the column totals from Table 11.12. For the sum of squares for we need

s:

(

"I )2

k

t~Yiv

(14.21)

k

L /Ii i

[W =

= [10] -

604246.590 _ (8212)2 = 2131.019 [2]' 112 '

k

k'"

/I.(x. - X•• )(y-.I. - y- •• ) t I.

i

(14.22)

= [14] _ [5][3] = 572 527.370 _ 7823 X 8212 = -1066.166

[2]' k

k

'" /I.(x. _ X )2

4 i

• ,.

..

= "'" '"

(I

Xiv

v

i

112

r (i I r

'

Xiv

_

l1i

--,--,-'_v,---,-k

(14.23)

"'" "'" 11;

;

= [12] -

[W =

547370.311 _ (7823)2 = 947.731. [2]' 112

30 45 37

ni

2

70.767 69.933 79.514

2123 3147 2942

Ie "'i

iii.

""

4

I Yiv v

3

Ini IIYiv i v i = 112 = 8212

Ie

1 2 3

1

Xiv

"'i

i v = 640,642

ni

IIY~v

k

159,813 232,975 247,854

v

IY~v

7

Ie "'i

74.100 69.733 66.541

Xi.

6

I l x iv i v =7823

2223 3138 2462

"'i

1v

5

i v = 585,567

k 'ni

Ilx~v

176,675 233,698 175,194

v

'1li

Ix~v

8

Table 11.12

"

= 594,500

i

I I Yiv x iv

Ie "'i

i

r = 604,246.590

Yiv

f Cit ni

150,237.633 220,080.200 233,928.757

162,518 227,882 204,100

(~YiV) ni

"'/

10

1v YivXiv

9

ihY = 36,395.410

"

Ie "'i

11 (YiV i

ih·)2

9,575.367 12,894.800 13,925.243

1v (Yiv -

~

S z

Cf.I

ttl

~

~ ttl

0

~

ttl

Z

C

ttl

'1Ii

s::

5!l

." t"'

[7] - [10] =

11

w

tv

00

2 3

=

ni

547,370.311

4•

k Ci ~XiV

r

164,724.300 218,823.200 163,822.811

ni

~XiV ) C

12

i

-

x;)2

=

=

V

- Xi)2

38,196.689

'Ili

11,950.700 14,874.800 11,371.189

'L 'L (Xiv

k

v

'L (xiV

ni

[8] - [12]

13

ni

ni

V

ni

=

572,527.370

ni

V

ni

'L xiv 'L Yiv

'LV

k

157,314.300 219,450.800 195,762.270

V

'L Xiv 'L Viv

ni

14

ni

;

v

=

=

21,972.630

Xi)(YiV - y;)

5203.700 8431.200 8337.730

- X;)(YiV - Y;)

'L'L (XiV -

]c

'L (XiV

ni

[9] - [14]

15

Table 11.12 (Continued)

'L

k i

=

2265.850 4778.897 6113.498

- Xi)2

=

- x.)2 1. 13,158.245

IV

'L (X.

ni

=

T

T iii)

~ (XiV - X;)(YiV - Y;)

Ci

ni

'L (XiV

[ni ~ (XiV - Xi.)(Yi" -

[15]2/[13]

16

23,237.165

Sum of squares for si

7309.517 8115.903 7811.745

[11] - [16]

17

w

w

00

'"

t'"

Z I'T1

z

til

'"(5

:;.:l I'T1

a

m

:;.:l

> t'"

I'T1 :;.:l

."

'"I'T1 <:

0

Z

0

> :;.:l v.;

s:: "0

0

(")

.j::.

....

;i

rrI (")

tI>

384

SIMPLE LINEAR REGRESSION

The sum of squares for

L" lI i[il;. -

{Y ..

i

CHAP.

11

s: is

+ b(Xi. -

XJ}]2

"

[±

IIlXi .

-

YJ]

XJ(yi. -

2

'"l l i (_ _.. )2 - =..:....--:-,,------=, = k Yi. - Y

L IIlXi . -

i

(14.24)

X.Y

i

= 2131.019 _ (-1066.166)2 = 931.618. 947.731

For the sum of squares for

s! we first substitute [15] and [13] in (14.9),

h = 21,972.630 = 0.575250 38,196.689

'

and (14.22) and (14.23) in (14.11),

b=

-1066.166 947.731

=

-1.24967.

We can now calculate the sum of squares for (-1.124967 _ 0.575250)2(_1947.731

+

s! from (14.17) as

1 38,196.689

)-1= 2673.310.

For the sum of squares about the overall line we first need

(14.25)

=

[7] - [3]2 [2]

=

640642 _ (8212)2 , 112 k

~,

'/I;

L L (Yiv i

yJ( Xiv - xJ =

38526.429, '

'ni

L LXiv L L Yiv L L XivYiv - i \' ~, i \' /,

\'

k

1Z;

=

11;

(14.26)

i v ,

k i

=

[9] - [5][3] [2]

=

IIi

594 500 _ 7823 X 8212 , 112

=

20906.464, '

11.14

SECT.

and

fl,

k

fl'

k

"" (X'.tV 4"" ~

385

COMPARISON OF SEVERAL REGRESSION LINES

-

X•• )2

= 4t "" k"" X~

tv

tv

_

(i i Xiv i

tv

vk

r

(14.27)

'"

£.. lli i

=

[8] - [5]2 [2]

=

585 567 _ (7823)2 , 112

=

39 144.420. '

Then the sum of squares of deviations about the overall line is k

11,.

II i

k

(Yiv - Y;v)2

=I i

v

11,-

I

{Yi - [y ..

y

+ b(Xiv - xJW (14.28)

=

38526.429 - (20,906.464)2 = 27360.591. , 39,144.420'

The sum of squares due to regression on the overall line is k

fl'

"" "" (Y.. _ - )2 4-£.. n' Y.. t V

[± i (YiV -

yJ(xil• -

XJ]2

= -=--=--'--'-,._ ____ k

____==__

/I'

(14.29)

II(x;v - x.Y v ' _

i

= (20,906.464)2 = 11 165.838.

39,144.420

'

FinalIy, the total sum of squares about the grand mean is k

11,-

Ii Iv

(Yiv -

y.Y = 38,526.429.

We now assemble alI these sums of squares in Table 11.13. Various interpretations of Tables 11.11 and 11.13 are illustrated in Figure 11.9. In Figure 11.9a the slopes Pi of the individual lines are different and in general not equal to the average slope i3 of the separate paraIIellines. The test of the null hypothesis Pi = i3 is given by s~M = 259.249/219.219 = 1.18 which under the nulI hypothesis is distributed here as F(2, 106). Clearly here the nun hypothesis of paralIelism of the separate lines is acceptable.

386

SIMPLE LINEAR REGRESSION

Figure 11.9

CHAP.

11

SECT.

11.14

387

COMPARISON OF SEVERAL REGRESSION LINES

If we can accept the foregoing null hypothesis we then proceed to test whether the group means can be regarded as lying on a least squares line. In Figure 11.9b they do not, whereas in Figure 11.9c they do; the test is = 931.618/219.219 = 4.25, which under the null hypothesis is distributed here as F(I, 106). In this instance this null hypothesis is rejected. This establishes that the lines for the groups do differ in the way illustrated by Figure 11.9b.

s:M

Table 11.13

-

,

Degrees of freedom

Sums of squares

Source of variance Between band b Deviations of the group means about their regression line Between the individual slopes About the individual lines

931.618 518.498 23,237.165

1 2 106

About the over-all line Due to the over-all line

27,360.591 11,165.838

110 1

Total

38,526.429

111

2,673.310

Mean squares 2673.310

= s!

931.618 = si 259.249 = s~ 219.219 = si

If this null hypothesis had been accepted, then there would remain the possibility that though the individual lines are parallel with slope jj and the group means do lie on a line with slope p, yet jj ¥= p, as illustrated in Figure 11.9c. The alternative is that jj = p, as illustrated in Figure 11.9d, and this implies that a single line, the overall regression line, is an adequate fit to all groups. The test for this in Tables Il.ll and 11.13 is

2/ 2

S4 S1'

In some forms of analysis of covariance, the parallelism of the separate lines is assumed, so that s~ and s~ are pooled, and also the null hypotheses and are tested jointly, by pooling these. This will be involved in discussed in the next section. If we accept the null hypothesis that the individual lines are parallel, we can construct the adjusted means as follows. The individual lines with common slope (14.8) are

s:

s:

(14.30)

and the y mean adjusted for x equal to x .. is Yi(.r)

=

fh

+ b(x .. -

Xi}

(14.31)

388

SIMPLE LINEAR REGRESSION

For example, x..

= 7823/112 =

Y 1(:r) = 70.767

11

CHAP.

69.848, and

+ 0.57525(69.848 -

74.100) = 68.321.

The variance between two adjusted means is

+

V[Y;(£)- Y;'(X)] = V[il;,

= V[y;,]

fil. -

h(x.. - xd -

+ V[yd + (Xi.

h(x.. - Xi')]

- xi,/V[h] (14.32)

where V[h] is given by a formula analogous to (6.13). For example, the estimated variance between the adjusted means for kilns 1 and 2 is V[Y. 1(.;;)

y;

2(£)

]

= 219.219[.l + ~ + (74.100 30

69.733)2J 38,196.689

45

=

12.29

'

and the confidence limits can be immediately constructed. Table 11.14 Sums of squares

Degrees of freedom

Mean squares

Between kilns Within kilns

2,131.019 36,395.410

2 109

1065.509 333.903

Total

38,526.429

III

Source of variance

It is natural to inquire whether the analysis of covariance of these data has achieved any advantages over a simple analysis of variance of y. From the calculations given, we can readily assemble Table 11.14. The variance ratio is 3.191, which can be compared with 8.222 obtained from Table 11.1 3 by pooling s! and s:. We can also compute the variances between the unadjusted means as ~ V[Y 1

-

1 Y2 ] = 333.903 ( -:nr

+ 45) = 1

18.550

which is some 50 per cent larger than the variance between the adjusted means.

11.15. Simple Analysis of Covariance Quite frequently the analysis of the previous section is simplified. Two changes are made: 1. The sums of squares and degrees of freedom for

s! and s: in Table

SECT.

11.15

389

SIMPLE ANALYSIS OF COVARIANCE

11.11 are combined into a single line measuring differences between the adjusted y means. 2. The test for parallelism of the individual slopes involving s~ is omitted. The effect of these two changes is to simplify the arithmetic considerably. This modified analysis starts from a table of sums of squares and products (Table 11.15). Table 11.15 source of variance

Degrees of freedom

Sums of squares and products for x"

k

k

l'

~ IIlrl . -

Groups k - l

for Y'

for '''Y ~ IIMI' -

:;,.Y

.r.J(iil .

ii .. )"

-

I

= T;rz k

Error

k

~1I1-k i

k

"I

~ ~ ('''Iv v i

XI')"

Total

"i ~ ~ (,CiV i v

~lIi-1 i

= T•• k

"I

~ ~ (a'iv i v

·"I·)(Yi,· -

iii')

:r ..>"

k

"i ~ ~ ('''iv i v

=S".

"I

~ ~ (YiV i

iii')'

v

= E ••

= Ex.

l'

ii .. )"

I

= Tx.

=Exx k

~ "l(iil· -

k "i

X.. )(Yiv -

ii .. )

~ ~ (Yiv i v

ii.Y

=S..

= Sx.

To pool s! and s: in Table 11.11 involves pooling the degrees of freedom, 1 + (k - 2) = k - 1, and the corresponding sums of squares. Call the pooled mean square Then the corresponding sums of squares will be (k - l)s;. Straightforward manipulation gives for the sum of sq.uares for the adjusted y means (k - l)s; = TlIlI _ (E",y + T",y)2 + E!y, (15.1) E",,,, + T",,,, E",,,, changing from the notation of Table 11.11 to that of Table 11.15. To dispense with the test for parallelism of the individual lines amounts. to assuming that the individual lines are parallel, when s~ of Table 11.11 will have expected value ()'2. Pooling the degrees of freedom of s~ and s~ gives

s;.

(k - 1)

+

(*

2k)

11; -

=

*

11; -

k - 1.

(15.2)

Straightforward manipulation gives the error sum of squares as

( ~~' 11; ,

)

k- 1

2

S5

=

EY1l -

E!y -

E",,,,

(15.3)

changing from the notation of Table 11.11 to that of Table 11.15. Under the null hypothesis that the adjusted means are equal, s;/s: will be distributed as

F(k - 1, ~

11; -

k- 1).

390

SIMPLE LINEAR REGRESSION

CHAP.

11

To apply this simplified procedure to the data of the example of the previous section, Table 11.16 gives the numerical values corresponding to the data of Table 11.l2. The T's and the S's have been calculated earlier, the E's are most easily obtained by difference. Using (15.l), the sum of squares for adjusted y means is 2S2

= 2131.019 _

(20,906.464)2 39,144.420

6

s:

+ (21,972.630)2 = 3604.927' 38,196.689

'

so = 1802.463. It will be noted that, as a check, the sum of the sums of and si in Table 11.13 is 2673.3lO + 931.618 = 3604.928. squares for Using (15.3), the sum of squares for error is

s:

( 112 - 3 - l)s2

=

36395.410 - (21,972.630)2 = 23755.664' 38,196.689"

6,

so s~

=

s: and

219.960. It will be noted, as a check, that the sum of squares for + 23,237.165 = 23,755.663.

s~ in Table 11.13 is 518.498

Table 11.16 Source of variance

Degrees of freedom

Sums of squares and sum of products of deviations

Groups Error

2 109

947.731 Trz= E.rx = 38,196.689

Total

111

S.rx

for

for xy

xi

= 39,144.420

= -1,066.166 = 21,972.630 Sz. = 20,906.464

T•• Ex.

for y' T•• = 2,131.019

= 36,395.410 S •• = 38,526.429

E ••

Hence the test of the null hypothesis of equality of the adjusted means is

~= S6

1802.463 219.960

= 8.19,......, F(2, 108).

The arithmetic necessary for this simplified analysis of covariance is substantially less complex than in the full analysis of the previous section. Nevertheless, the full analysis would ordinarily be indicated except in routine applications where a substantial body of previous experience had established that the separate lines, if not truly parallel, at least must be very nearly so. Nonparallelism of the lines is an important feature and should not be overlooked. The interpretation of an analysis of covariance can prove remarkably tricky, and a paper by Fairfield Smith [7] should be studied in this context. For a general discussion see Cochran [8].

SECT.

11.17

391

REGRESSION WITH ERROR IN INDEPENDENT VARIABLE

11.16. Exponential Regression Suppose that our model is 'YJ =

CJ.eP"'.

(16.1)

The sum of squares of deviations between the observations Yi and the predictions Y i given by the estimated equation Y = aeb'" is (16.2) Differentiating this with respect to a and b and equating to zero gives (16.3)

oR - = ob

. + 2a 2 '" x.e2b

-2a '" x.y.e b"', £.,

• •

£.,

•

"'l

= 0

(16.4) ,

whence we have the two simultaneous equations (16.5) (16.6)

in the two unknowns a and b. An exact solution can only be approximated to by a tedious iterative procedure. An alternative approach is to take logarithms of(16.1): log 17

= log CJ. + f3x,

(16.7)

and obtain a least squares solution for log CJ. and f3 by minimizing R

=

2 (log Yi -

log Y i )2

(16.8)

in the usual way, i.e., handle the problem as if it was to regress log yon x. The use of log y in place of y means that we are minimizing the sums of squares of deviations of log y from log Y instead of y from Y; so we will obtain a different solution. Also, if V[Ylx] = (}'2, a constant, then V[log Ylx] will not be a constant, and the least-squares analysis based on log y will be incorrect. Quite often, however, V[ylx] = k2172; i.e., ·the standard deviation is proportional to the mean, and then, as discussed in Section 3.3, log y will have a constant variance. In these circumstances, then, we will be correct in regressing log y on x. 11.17. Regression with Error in the Independent Variable Throughout this chapter we have assumed that the independent variable x was observed without error. We have further supposed that y was

392

SIMPLE LINEAR REGRESSION

CHAP.

11

distributed normally about 17 with variance ()"2, and it is irrelevant whether this variation in y is in some sense genuine or represents measurement error. For example, if we were measuring the current y in a circuit for various values of the applied voltage x, then presumably variation in y is largely measurement error. On the other hand, if we were measuring the weights of hogs y fed various quantities of a ration supplement x, then the larger part of the variation in y for a fixed x would represent genuine variation in the hogs' weight. The theory of this chapter is the same whether the variation in y is measurement error or genuine variation. The assumption that x is known without error is, however, fundamental. Suppose that there is a true linear relation

v=

ex.

+ fJu

(17.1)

where ex. and fJ are constants. Suppose that we obtain the various values of x in the following manner. The controls of the plant are set to bring u into roughly the desired region, and we then measure u with error d, independent of u, so that we record the measurement as x' where x'

=

u

+ d.

(17.2)

The variable v is observed and measured with error e, independent of D, so that we record y' = V + e. (17.3) Thus v

=

y' - e, u

= x' -

d. Substituting in (17.1),

y' = ex.

+ fJx' + (e -

(17.4)

fJd).

This appears at first sight to be a standard regression model, y' being given by a linear relation ex. + fJx' plus a random error e - fJd. However, in standard regression analysis it is assumed that the random error is independent of the independent variable. In the present case this condition is not satisfied. We demonstrate this as follows: Cov [x', e - fJd] = E[(u

+ d)(e -

fJd)] - E[u

= E[ue] + E[de] - E[ud] - {E[u] + E[d]}{E[e] -

+ d]E[e -

fJd]

- fJE[d 2 ] fJE[d]}.

(17.5)

Now we are assuming that d is independent of u and of e, and e to be independent of u, and d and e to have zero expectations. It follows that Cov [x', e - fJd]

=

-fJE[d 2 ]

=

-fJV[d] =/= 0,

(17.6)

and therefore x' and e - fJd are not independent. For reviews of what can be done in this situation see Madansky [9] and Keeping [10].

393

EXERCISES

Berkson [11] pointed out that if we operate the controls to bring a gauge recording u to a selected value XI, it is true that due to random errors in the gauge, say d, we do not get the plant set at XI but instead at

u = XI - d.

(17.7)

If we substitute this value of u and v = yl - e from (17.3) in (17.1), on rearranging we get (17.8) yl = IX + (3XI + (e - (3d). This is a standard linear regression situation, since XI is a fixed, chosen variable and' the error term e - (3d is independent of XI. Hence our sample estimate of (3 will be unbiased. For a review of Berkson's model, see Lindley [12], and for a generalization, see Scheffe [13].

EXERCISES 11.1. The solubility of nitrous oxide in nitrogen dioxide was determined with results as below for temperature ranging from 263 to 283 degrees absolute. The reciprocal temperature is expressed as IOOO/T. A number of independent determinations were made at each temperature. [Data from W. Arthur Rocker, "Solubility and freezing point depression of nitrous oxide in liquid nitrogen dioxide," Analytical Chemistry, 24 (1952), 1322-1324.]

Reciprocal temperature

3.801

3.731

3.662

3.593

3.533

1.28 1.33 1.52

1.21 1.27

1.11 1.04

0.81 0.82

0.65 0.59 0.63

.Solubility, % by weight

(a) Fit a straight line of the regression of solubility on reciprocal temperature. Test this line for deviations from linearity. (c) If the hypothesis of linearity

(b)

is acceptable, form a pooled estimate of the variance of the solubility measurements. (d) What is your estimate of the true solubility fora reciprocal temperature of 3.78? (e) What are the 95 per cent confidence limits for this estimate? (f) What are the 95 per cent confidence limits for the slope of the line? (g) Suppose you took a sample and found its solubility to be 1.30. Between what limits would you have 95 per cent confidence that the reciprocal temperature lay (assuming the sample to be saturated)? Use (i) an exact method, (ii) an approximate method. 11.2. The data below are similar to those of Table 11.9 but on a sample of a different make of automobile. Xi

Yi

19.65 3.44 { 3.93

31.15 4.98 5.45

35.95 6.40

50.15 8.88

59.65 11.22

394

SIMPLE LINEAR REGRESSION

CHAP.

11

I. (a) Obtain the regression line of y on x. (b) Test the null hypothesis of linearity for this line. (c) Extrapolate the line to x = 0, and construct 95 per cent confidence limits for 1] at this value of x. II. Compare this line with that for the data of Table 11.9. (a) Compare the variances about the regression lines. (b) Compare the slopes. (c) Test whether the lines can be regarded as coincident. 11.3. In a comparison of a new method of gas analysis with a standard method, the following results were obtained on a series of samples:

Standard method (x): New method (y):

2.97 2.94

3.56 3.54

6.45 6.48

1.12 1.08

6.66 6.73

1.37 1.33

6.80 6.86

Data from Jere Mead, "A critical orifice CO 2 analysis suitable for student use," Science, 121 (1955), 103-104. Fit the line y = a + b(x - x), and test at the level of significance IX = 0.05 the null hypotheses (a) {J = 1; (b) the intercept on the y axis equals zero; (c) items (a) and (b) jointly. 11.4. In an assay of a preparation of insulin of unknown potency against a standard, three doses of the unknown and three doses of the standard were injected into rabbits, and the percentage fall in blood sugar after a certain period was observed; these are the data below. In insulin assay of this type, it is usually assumed that the plot of the above variable against log dose is a straight line. (a) Test the parallelism of the two lines. (b) Calculate the point estimate of the potency ratio. (c) Calculate 95 per cent confidence limits for the potency ratio. Preparation Unknown log dose

Standard log dose 0.36

0.56

0.76

0.36

0.56

0.76

17 21 49 54

64 48 34 63

62 72 61 91

33 37 40 16 21 18 25

41 64 34 64 48 34

56 62 57 72 73 72 81 60

11.5. A sample of 56 subjects were given a test involving mental addition, x. They were divided into four groups randomly, and each group drank one of f?ur beverages, which could not be distinguished by the subject. After a short ttme

395

EXERCISES

interval to allow the drugs in the beverages to act, the subjects were retested by a replicate of the first test, y. The scores are as below (data from H. Nash). Drug A

DrugC

DrugB

DrugD

x

y

x

y

x

y

x

Y

24 28 38 42 24

24 30 39 41 27

23 33 39 36 18

18 32 33 35 19

27 27 44 38 32

35 31 55 43 44

27 44 39 27 59

28 40 34 27 47

39 45 19 19 22

46 56 25 18 25

28 43 37 30 49

28 41 37 33 39

32 24

28 33

13

13

39 52

39 58

36 19 34 22 28

30 24 28 21 28

34 52 27 42

31 52 38 45

37 40 36 41

38 41 38 36

17 20 49 29

18 17 41 25

39 29 55 49

39 26 46 42

I. Make a simple analysis of variance on the y's. II. Make a simple analysis of variance on the differences y - x. III. (a) Make an analysis of covariance of y, including a test for the parallelism of the separate regression lines. (b) Discuss the difference between I, II, and IlIa. (c) Construct a simple (non multiple) comparison 95 per cent confidence interval for the difference between the adjusted means for drugs A and B. (d) B is a placebo and C and D are two levels of a certain drug, where the level of D is twice the level of C. The contrast B - 2C + D will measure the linearity of response. Construct a 95 per cent multiple comparison confidence interval for this contrast. Certain sums and sums of squares and products which may be useful are below

1 2 3 4

"i

fll

III

11l

ni

LYiv

LXiv

L2 ,/Ji,·

2 Lx iv

LX;vY;,'

14 14 14 14

497 468 480 460

455 490 443 507

19,367 16,332 18,842 16,040

16,249 18,008 15,787 20,265

17,623 17,066 17,018 17,941

56

1905

1895

70,581

70,309

69,648

I

"i

k

L

396

SIMPLE LINEAR REGRESSION

REFERENCES 1. Acton, Forman S., Analysis of Straight-Line Data. New York: John Wiley and Sons, 1959. 2. David, F. N., and J. Neyman, "Extension of the Markov Theorem of Least Squares," Statistical Research Memoirs, 2 (1938), 105-116. 3. Plackett, R. L., "A Historical Note on Least Squares," Biometrika, 36 (1949), 458-60. 4. Finney, D. J., Pro bit Analysis. 2nd ed.; London: Cambridge University Press, 1952. 5. Finney, D. J., Statistical Methods in Biological Assay. New York: Hafner Publishing Co., 1952. 6. Emmens, C. W., Principles of Biological Assay. London: Chapman and Hall, 1948. 7. Smith, H. Fairfield, "Interpretation of Adjusted Treatment Means and Regressions in Analysis of Covariance," Biometrics, 13 (1957), 282-308. 8. Cochran, William G., "Analysis of Covariance: Its Nature and Uses," Biometrics, 13 (1957), 261-81. 9. Madansky, Albert, "The Fitting of Straight Lines When Both Variables are Subject to Error," Joumal of the American Statistical Association, 54 (1959), 173-205. 10. Keeping, E. S., Introduction to Statistical Inferellce. Princeton, N.J.: D. Van Nostrand, 1962. 11. Berkson, J., "Are There Two Regressions?" Joumal of the American Statistical Association, 45 (1950), 164-80. 12. Lindley, D. V., "Estimation of a Functional Relationship," Biometrika, 40 (1953), 47-49. 13. Scheife, H., "Fitting Straight Lines When One Variable Is Controlled," Joumalof the Americall Statistical Associatioll, 53 (1958), 106-117.

C HAP T E R

12

The Bivariate Nornlal Distribution and the Correlation Coefficient

12.1. Introduction In the preceding chapter we supposed that we observed Y at a number of selected values of x. The values of x used were what we cared to make them, and x was not a random variable. In this chapter we discuss the case where both x and yare random variables, drawn from some hypothetical population. For example, x could be the girth of a hog and y its marketable weight; or x could be the "intelligence" of a brother and Y the "intelligence" of his sister; or x could be the grade average of members of a class and Y their income in dollars after Federal income taxes, 10 years later. In every case care should be taken to define quite precisely the population from which the. sample is taken, and to recognize that any inferences possible are strictly applicable only to that population. 12.2. Transformations of Bivariate Distributions In Section 1.14 we saw how to obtain the distribution of a function of x, given the distribution of x. In this section we do the analogous thing for a bivariate distribution. We are given a bivariate distribution with the probability density determined by the function Px(x 1 , x z). We are also given that Yl is some function of Xl> X z, namely, Yl = !1(XI, x 2 ), and likewise Yz is another function of Xl' x z, namely, Yz = !z(xI, x z). We assume that these functions have continuous first partial derivatives, and that to each point in the (xl> x z) plane there is one and only one point in the (YI, Yz) plane and vice versa, Let the inverse functions be Xl = gl(YI, Yz), Xz = gZ(YI, Yz). 397

398

THE BIVARIATE NORMAL DISTRIBUTION

CHAP.

12

Now, if Xl, X 2 are random variables, then Yl> Y2 will be random variables. Our problem is to determine

where py(Y!> Y2) is the joint probability density function of Yh Y2, given that we know Px(x I , x 2) and the functions!l and!2.

R ~-----If---Y2

-r--------------------------------~yl

Figure 12.1

Consider the area R in the (YI' Y2) plane defined by the four lines YI = YI' Yl = YI + dYI' Y2 = Y2, Y2 = Y2 + dY2 (Figure 12.1). The probability that a random point falls in the rectangle R is approximately equal to the product of the probability density PY(YI' Y2) and the area dYI dY2: (2.2) Now, for a fixed Yl> the equation YI = !1(XI, x 2) will determine a line in the (Xl' x 2) plane, say A. Similarly the equations YI + dYI = !1(X!> x2), Y2 = !2(XI , x 2), Y2 + dY2 = !2(Xl> x 2) will determine three more lines B, C, and D (Figure 12.2). These four lines will enclose an area S. Now, because of the one-to-one correspondence of points (YI, Y2) with points (xl' x 2), whenever a random point (YI , Y2) falls inside the rectangle R in Figure 12.1, the corresponding point (Xl' X 2) falls inside the figure S in Figure 12.2. The probability of (Xl' X 2) falling inside S is approximately equal to the product of the probability density Px(x I , x 2) times the area

SECT.

12.2

TRANSFORMATIONS OF BIVARIATE DISTRIBUTIONS

399

of S; hence (2.3)

We therefore need to find the area of S. S is approximately a paraJle1ogram. It is known from coordinate geometry that, if (Xl' X 2), (X~, x~), and (X~, x~) are three of the vertices of a parallelogram, then the area of the parallelogram is given by the absolute value of the determinant 1

Xl

X2

.X'1

•X'2

1 .x"1

x;

(2.4)

We therefore need the coordinates of three of the vertices of S.

-r----------------------------~Xl

Figure 12.2

To obtain the coordinates of PI, the intersection of the lines A and C, we note that along the line A we have values of (Xl> x 2) satisfying the equation Yl = /1(."/;1' x 2) and along the line C we have values of (xl> x 2) satisfying the equation Y2 = /lxl> x2). At the intersection of these two lines these two equations must be satisfied simultaneously; i.e., the coordinates of PI are the solutions for Xl' x 2 in (2.5) But we know that when Yl = YI> Y2 = Y2' (2.6)

400

THE BIVARIATE NORMAL DISTRIBUTION

CHAP.

Ii.

so these x values are the coordinates of Pl. Similarly, the point P2 is the intersection of the lines Band C, and along the line B we have values of (Xl> x 2) satisfying the equation YI + dYI = fl(x I, x 2), so the coordinates of P 2 are the solutions for Xl, x 2 in (2.7)

and we know that when YI

=

+ dYl> Y2 = Y2,

YI

(2.8)

so these X values are the coordinates of P 2 • But, by Taylor series, gl(YI

+ dYI' Y2) =

+ agl dYI + higher order terms;

(2.9)

g2(YI

+ dYI' Y2) = g2(YI, Y2) + aag 2 dYI + higher order terms;

(2.10)

gl(YI, Y2)

aYI YI

so the coordinates of P2 can be written, ignoring the higher-order terms, as approximately ( giYI' Y2)

+ agl dYI' g2(YI, Y2) + ag2 dYI). aYI

(2.11)

.aYI

Similarly, the coordinates of P a are approximately ( gl(YI' Y2)

+ agl dY2' g2(YI, Y2) + ag2 dY2). aY2

(2.12)

aY2

Substituting these three coordinates in (2.4), we get for the area of S the absolute value of 1

gl(YI, Y2) gl(Yl' Y2)

1 gl(Yl> Y2)

ag

g2(Yl> Y2) l

+ -a dYI Yl

+ aagl dY2 Y2

Expanding this determinant gives

g2(Yl> Y2) glYI, Y2)

ag2

+ -a dYl Yl

+ aag2 dY2 Y2

(2.13)

SECT.

12.3

THE BIVARIATE NORMAL DISTRIBUTION

401

substituting this for the area of Sin (2.3) gives

PY(Yl, Y2) dYl dY2

=

Px(X l , :/:2)

ag l

ag2

aYl agl

aYl dYl dY2 ag2

(2.15)

aY2 aY2 and

(2.16)

where the absolute value of the determinant is used.

12.3. The Bivariate Normal Distribution In this section we shall postulate a certain model which will lead us to the bivariate normal distribution. Let Xl' x 2 be independent normally distributed variables with zero means and unit variances. Let y~, be linear functions of Xl' x 2 defined by constants 1]i, Ii}: y~ = 1]1 + luxl + l12 x 2, (3.1) y~ = 1]2 + l2l Xl + 122x2·

Y;

It will be more convenient to deal with Yl = y~ - 1]1, Y2 = Y; - 172' since these new variables will have expected values zero. Thus Yl and Y2 are functions of Xl and X2:

+ 112 x 2, l2l Xl + 122 x 2·

Yl = fl(x l , x 2) = 1u x l Y2 = f2(x l , x 2) =

(3.2)

The variables Yl, Y2 will be normally distributed with expectations zero, and (3.3) Cov[Yl> Y2]

= E[YlY2] - E[Yl]E[Y2] = E[lu12lx~ + 112122x~ + lu122xlx2 + 112121xlX2] = lu 12l

+ 112122 ,

(3.4)

402

THE BIVARIATE NORMAL DISTRIBUTION

CHAP.

12

We can solve the equations (3.2) to give the inverse functions of x in terms of Y1, Y2: ~ _ g (y y) _ 122Y1 - 112Y2 _ 122Y1 - 112Y2 (3.5) '''1-11>2, 111122 - 112121

.,\

. (3.6) (3.7) The random variables Xl' x 2 by definition are independent and N(O, I); so

=

.l e-~(Q)la+Q)aa).

(3.8)

27T To get the distribution of (Y1> Yz) we use (2.16). We need

(3.9) so Ogl Og2 OY1

OY1

(3.10)

Ogl ogz OY2

OY2

We now use (2.16):

Define p as the correlation coefficient between Y1 and Y2: COV[Y1' Y2]

P = ";V[Y1]V[Y2]

Let V[Y1]

=

O'~, V[Y2]

0'20'2(1 _

1 2

2) P

=

111/21

+ 112/22

(3.12)

= .J(l~1 + n2)(1:1 + 1~2)'

O'~. Then

= (12 + 129)(12 + 12 )[1 11

= (111 122 -

1.

21

22

' 12 1'u)2 = .,\2,

-

+ 112122)2 ] + li2)(l:1 + 1~2)

(111 121

(1~1

(3.13)

SECT.

12.3

403

THE BIVARIATE NORMAL DISTRIBUTION

and the constant part of (3.11) outside the exponent is (3.14)

2d The exponent of (3.11) is -

2!2

{(/~2 + l~l)Y~ -

2(/11/21

+ 112/22)Y1Y2 + (l~l + 1~2)Yn

(3.15) Substituting in (3.11), 1 /

P{Y1, Y2} =

2

P

27T(h(12Y 1 -

exp { _

1 2(1 -

p2)

Y1 _ [()2 (11

2p Y1 Y2 (11 (12

+ (Y2 )2J} (12

(3.16) •

Finally, we may make the further transformation back to Yl' Y2 which is very simple since the determinant is merely 1: P{Y1'' Y2'}

=

ex p {-

1/ 2 1- p

27T(11(12Y

1 2(1 -

2

[(Y~

p)

- '(11)2 _ 2p (11

Y~

- 'Yh (11

Y~ - 'Y/2 + (Y~ (12

'Y/2)2]},

(12

(3.17) and this is the general form for the bivariate normal distribution. We can note at once an important result. If the covariance of y~, y~ is zero, then p = 0, and (3.17) becomes

= ~ exp [- !(y~ -./ 27T(11 = p{Yap{y~},

2

- 'Y/1)2] -J~ exp [- !(y~ - 'Y/2YJ 27T(12

(11

2

(12

(3.18)

so under this circumstance y~ and y~ are independent. This is the result which was assumed in the discussion in Chapter 8.

404

THE BIVARIATE NORMAL DISTRIBUTION

CHAP.

12

12.4. Some Properties of the Bivariate Normal Distribution

The general form is given in (3.17). Just as with the univariate normal distribution we found it convenient to have a standardized form, so it will be here. We define new variables (4.1) with inverse functions

so that (4.3) The probability density of (uI , u2) will be, using (2.16),

(4.4) It will be convenient to use the symbol c/>(UI' u2) for this standardized form. Of course, c/>(u l , u2) is a function of p. In studying the properties of the standardized bivariate normal distribution (4.4), it will be convenient to make a further pair of transformations. We define new variables

(4.5) (4.6)

(4.7) (4.8)

so that Ogl = 1,

au;

Ogl ou~

= 0,

Og2 _

'--2

:l,-y1-p'

UU 2

(4.9)

12.4

SECT.

PROPERTIES OF BIVARIATE NORMAL DISTRIBUTION

405

and the determinant is

Ogi Og2 Ou~

Ou~

Ogi Og2 ou~

ou~

1

p

=

(4.10)

o

-J1 _ p2

Also

u~

-

2pUIU 2

+ u~ = u1 2 - 2pul(pu~ + .J1-7l-u~) + (pu~ + -Jl - p2U~)2 = (1 - p2)(U~2 + U~2).

(4.11)

We now write down the probability density of (u~, u~), using (2.16) again:

(4.12) Thus u~, u~ are independent. We get the marginal distribution of u~ by integratingp{u~, u~} over u~ [see (1.20.10)]:

p{ul.} = f:~ p{u~,

u~} du~ (4.13)

since the integral is just the integral of a unit normal distribution which must equal 1. Hence the marginal distribution of u~ = UI is a unit normal distribution, and so the marginal distribution of y~ will be a normal distribution with mean 1h and variance O'~. The same argument shows that the marginal distribution of y~ is normal with mean 1)2 and variance O'~. The conditional distribution of U 2, given UI [see (1.21. 7)], is

(4.14)

406

THE BIVARIATE NORMAL DISTRIBUTION

CHAP.

12

which can be regarded as a normal distribution of liz with mean pill and variance 1 - p2. We now transform this by defining Xz = /2(111' liz) =:: U 20'2 + ~2' so that the inverse function is U z = gz(x l , x 2) = (X2 - ~2)/a2' and dg z/dx 2 = 1/0'2' We also define Xl = 1110'1 + ~l' Then (4.14) becomes

which can be regarded as a normal distribution of a random variable X z with mean (4.16) and variance (4.17) The system is, of course, symmetric in

Xl

and

X 2,

and so we have (4.18) (4.19)

Thus the means of the conditional distributions are linear functions of the other variable, and the variances are constants. We recall that in Section 11.2 our model for linear regression was that y was normally distributed around its expected value with a constant variance, and this expected value was a simple linear function of another variable x. Therefore the conditions for a conditional regression analysis of Xl on X 2, and for a conditional regression analysis of X 2 on Xl> are both satisfied. Here there are two true regression lines, (4.16) and (4.18), with regression coefficients (4.20) The product of the two regression coefficients is p2. To find the point of intersection of the two regression lines, we can substitute X 2 from (4.16) into (4.18) which gives Xl - ~l = p2(Xl - ~l)' which is only true if Xl = ~1'

SECT.

12.4

PROPERTIES OF BIVARIATE NORMAL DISTRIBUTION

407

and similarly we obtain X 2 = ~2' The point of intersection is therefore (~1' ~2)' We find the angle between the two regression lines as follows. In Figure 12.3,

{3",1 I'" 2

= tan A,

(3",2 I'"1

= tan C,

(4.21)

and B is the angle between the two regression lines. For three angles,

Figure 12.3

say A, B tan A

+ 90°, and C, whose sum is 180°, + tan(B + 90°) + tan C = tan A tan(B + 90°) tan C,

(4.22)

whence tan(B

+ 90°) =

p(a1/a 2) + p(a2/a 1) = _P_ a~ + a: , p(a1/a 2)p(a2/al) - 1 p2 - 1 a1a2

(4.23)

so (4.24) When p = 1, tan B = 0, and so the two regression lines are identical. When p = 0, tan B = 00, and so the lines are at right angles to each other.

408

THE BIVARIATE NORMAL DISTRIBUTION

CHAP.

12

We can write (4.17) and (4.19) as (4.25) Thus, when p = 1, the conditional variance V[x2lxl] = 0, and, when p = 0, the conditional variance equals the unconditional variance. Thus p2 can be regarded as measuring the fraction of the variance of X 2 "explained" by the regression on Xl> and vice versa. This is one of the most useful interpretations of the correlation coefficient. The nature of the surface generated by the bivariate normal distribution can be seen from the equation for its probability density (3.17). If we place this equal to a constant, this implies ( Yl 0'1

171\2 _

J

2p Yl -

171

0'1

Y2 - 'YJ2

+

(Y2 - 'YJ2 \2 = constant, (4.26)

0'2

0'2

J

dropping the primes from the y's. This equation defines ellipses, for different values of the constant, at an angle to the axes. When p = 0, this equation becomes (Yl

J

~ 'YJl + (Y2 ~ 'YJ2Y =

constant,

(4.27)

which defines ellipses whose principal axes are parallel to the (Yl> Y2) axes. We can use the results of this section to indicate the proof of an important theorem we assumed in Section 1.26, namely, that any linear combination of independent random normal variables is itself normally distributed. To prove this for 11 random variables, it will be sufficient to prove it for two random variables. The proof is along the following lines. In (3.2) we defined Yl as Inxl + 112x 2 where In and 112 are constants and Xl and X 2 are independent random variables normally distributed with means zero and unit variances. In (3.16) we found the joint density function P{Yl, Y2}, where Y2 was another linear combination 12l X 1 + 12zx2• We defined Ul as (y~ - 1h)/0'1' which is equal to Yl/O'l' We subsequently defined u1as Ul , and found in (4.13) that u1was normally distributed with mean zero and unit variance. It follows that the linear combination Inxl + 112x 2 = Yl = UlO'l will be normally distributed, with zero mean and variance O'~ = 1;1 + 1;2' The arguments are changed only in detail if, instead of being distributed N(O, 1), the Xi are distributed N(~i' O'!).

SECT.

12.5

409

THE REGRESSION "FALLACY"

12.5. The Regression "Fallacy" Equations (4.16) and (4.18) are the basis of the phenomenon of regression noted by Galton. Galton observed that on the average the sons of tall fathers are not as tall as their fathers, and similarly the sons of short fathers are not as short as their fathers; i.e., the second generation tended to regress towards the mean. But if we look at the data the other way round, we find that on the average the fathers of tall sons are not as tall as their sons and the fathers of short sons are not as short as their sons, so the first generation tends to regress towards the mean. It seems implausible that both statements can be true simultaneously, so this phenomenon has been called the regression fallacy. In (4.16) let Xl be the height of a father and X 2 be the height of his son: (5.1)

so (5.2) For convenience, assume that the population of fathers and sons is stable, one generation to the next, so that E[x 1 ] = E[x 2 ], (11 = (12' Then (5.2) becomes (5.3)

Suppose that the correlation coefficient between Xl and X 2 is positive and less than 1. Consider a particular value of Xl greater than the mean, E[x l ], so that Xl - E[x l ] > O. Then the right-hand side of (5.3) is negative, and we have (5.4)

Thus the expected value for the son's height X 2 for fathers of a particular height Xl (greater than the average of Xl) is less than that particular father's height. In other words, "on the average, sons of tall fathers are not as tall as their fathers." The foregoing arguments are symmetrical in Xl and X 2 , so for a particular x2 greater than E[x 2 ] (5.5) ,

so "on the average, fathers of tall sons are not as tall as their sons."

410

THE BIVARIATE NORMAL DISTRIBUTION

CHAP.

12

The situation is illustrated in Figure 12.4, in which a single constant probability density ellipse, (4.26), is drawn for a bivariate normal distribu_ tion with means E[x 1 ] = E[x 2 ] = 0, variance a~ = a~, and p = 0.6. It is apparent that BA < AO, which is (5.4), and DC < CO, which is (5.5). The fallacy in the regression fallacy consists in supposing that there is a fallacy.

Figure 12.4

12.6. Estimation of the Parameters of the Bivariate Normal Distribution

We use the method of maximum likelihood to obtain estimators of the bivariate normal distribution

The likelihood L =

" P{Xlv' x 2v } is II v

SECT. 12.6

411

ESTIMATION OF THE PARAMETERS

where the summation is over ')I. As is usually the case, it is easier to maximize the logarithm of the likelihood:

logL= -nlog(27T0'10'2.J1

-l)

1 [L (Xlv 2- ~1)2 - 2p L (Xlv - ~1)(X2v 2 2(1 - p ) 0'1 0'10'2

~2)

-

+

L (x 2v -2

~2)2J

0'2

.

(6.3) ~lo ~2' 0'1, 0'2,

We differentiate with respect to

and p: (6.4) (6.5)

(6.6)

Equating (6.4) and (6.5) to zero and rearranging gives -

1 ~

"".;'2

~1) A

£., (Xlv -

0'1

0'10'2

f1

(6.9)

<;;2) = 0,

0'10'2

P~.

~

P £., ~ + :-:;(X 2v -

£.,

1~.

(Xlv - ~1) - "".;'2 A

£., (X 2v -

~2)

(6.10)

- 0.

A_

0'2

Conventional manipulation leads to ~ (xlv - ~1) whence

= 0, ~ (X 2v

-

~2)

= 0,

(6.11)

412

THE BIVARIATE NORMAL DISTRIBUTION

CHAP.

12

Equating (6.6) and (6.7) to zero and rearranging gives

(6.12) (6.13) which when added give

p2 gives

Equating (6.8) to zero and multiplying by 1 -

Substituting (6.14) in (6.15) gives

,,2 (Xlv -

P

~1)(X2' A

-

A

<11<12

~2)

=

,,2

lip .

(6.16)

Substituting this in (6.12) and (6.13) gives A2 <11

and since ~l

=

1 "" ( Xl. -.£., 11

E )2, S"l

(6.17)

= Xl' ~2 = X2 (6.18)

As might have been expected, the maximum likelihood estimators of the <1~ (i = 1, 2) are biased, but of course we can obtain unbiased estimators, say s~, by substituting n - 1 for n in the denominators in (6.18). Substituting (6.18) in (6.16) and solving for p, for which the symbol/' is commonly used, gives

(6.19)

SECT.

12.7 TESTS OF SIGNIFICANCE FOR CORRELATION COEFFICIENT

413

It can be shown that E[/']

=

p _ p(l - p2) 211

+ ... ,

(6.20)

is a biased estimator of p except when p = O. The regression coefficients (3oW"1 = p(a;/ai ) can be estimated as

SO l'

"ai Si ~ (XiV - Xi)(X jv - Xi) .J'" (XiV - xi)2/(n - 1) p - = /. - = ""'i:'::=======~======== -;=£"~~==~====== ai

sJ

.J~ (XiV - Xi)2 ~ (XiV - Xi)2.J~ (XiV - xj?/(n -

~ (XiV - Xi)(X iv - Xi)

(6.21)

=~~~~~~--~

~ (X jV -

1)

Xj )2

which is identical with the ordinary estimator of a regression coefficient [c.f. (11.2.13)]. Inserting maximum likelihood estimators into (4.17) gives

(6.22)

If we replace the factor I/n by I/(n - 2) this would be the usual regression estimator, say s!;I"'J' (11.3.13). It follows that (6.22) is a biased estimator of V[xilx j ].

12.7. Tests of Significance for the Correlation Coefficient Since b is normally distributed about {3 with a variance given by (11.2.16), b"'II",; - (3"'II"';

SXjlxJ.J~ (Xiv - Xi)2

,....., t(1I _ 2).

(7.1)

From (11.3.13) we have

(7.2)

414

THE BIVARIATE NORMAL DISTRIBUTION

Substituting this in (7.1), and substituting r(Sj/si) null hypothesis (3 d"'l = 0,

I"~ Si

=

---.....:......-~~ [

1 s~(1 - 1"2)J 2 (II - l)s~

II II -

-I"JII - 2

~

CHAP.

12

= b"'II"'I' then under the

"" t(1I

- 2).

(7 3) .

The null hypothesis (3"'II"'i = 0 is identical with the null hypothesis p = o. Therefore (7.3) referred to the t table with n - 2 degrees of freedom gives a test of the null hypothesis p = o. Alternatively we can use a very good approximation due to Fisher: Z

1 1+1" = -log. -- =

2

1-1"

1+1" 1.1513 log1o - 1-1"

(7.4)

is approximately normally distributed about

,=

1 1+p -log.-2 1- p

(7.5)

1 . 11-3

(7.6)

with variance V[z]~--

zJn -

Thus, under the null hypothesis that p = 0, 3 is a unit normal deviate. Also, the null hypothesis that two independent sample correlation coefficients come from the same population p can be tested by referring

J1/(1I 1

3)

-

+ 1/(11 2 -

(7.7) 3)

to tables of the normal distribution.

12.8. The Effects of Errors of Measurement Suppose that we sample x from a normal distribution N(~, a!). Further suppose that y is normally distributed about an expected value E[Ylx]

=

1]

+ y(x -

~)

(8.1)

with variance V[ylx] = 0'=1",. We will first demonstrate that the joint distribution of x, y is a bivariate normal distribution, and then study the effects of being only able to observe y with error, and then of being only able to observe both x and y with error.

SECT.

12.8

415

THE EFFECTS OF ERRORS OF MEASUREMENT

From (1.21.7),

p{x, y}

= p{Ylx}p{x} = 1 exp{- {y -[1] + y(x -

~)]}2}

2a~J'" x _1_ ex p [- (x - ~)2]

.J27Ta1ll:o

.J27T a",

=

2a:

1 ex p{- ![(X 27Ta",all l", 2

_ 2(x - ;)(y Now define a= and

P",y

_ ~)2(1- + L) a;I'"

-1]) + (y -;, 1])2]}.

a1l 1"'/y

as

a!

(8.2)

a1l 1'"

2= alii'"2 + Y2a""2 y a! = 2 22 alii'" + y a",

(8.3)

a1l

2

2

(8.4)

P",y

We now eliminate a=l", and y from (8.2). From (8.3) (8.5) and from (8.4) we obtain another expression for y2:

y

2

2

=

2

p",yayl'" 2 2' a.,(1 - p",y)

(8.6)

Equating (8.5) and (8.6) we derive a=l",

= a=(1

and substituting this in (8.6) gives

Y

2

=

2

- p!y),

(8.7)

a:

(8.8)

P"'Y2' a",

These expressions for a=l", and y2 are substituted in (8.2), giving

1 p{x,y} = - - 1 = ==

27Ta",a1l.J(1 - p:y) exp {-

1 2(1 - p!y)

[(x a",- ~)2_ 2p",y xa",- ~y a-1] + (y a-1]Y]} J ' 1l

(8.9)

1l

which is the standard form of the density function of a bivariate normal distribution, (3.17), with parameters ~, 1}, a!, a= = a;l", + y 2 a! as defined

416

THE BIVARIATE NORMAL DISTRIBUTION

CHAP.

12

by (8.3), and PXy as defined by (8.4). We note that _

{JI/lx - PXII

ay _ ( y 2a! )~ (a!lx+ y2a!)~ _ 2 + 22 - 1', ax ayl x I' ax ax

(8.10)

so b ylx' being an unbiased estimator of (Jylx' is an unbiased estimator of Y. the regression coefficient of yon x in the model (8.1). Thus if the data we observe are generated by this model, then the parameters are estimable, I' by bylx' a! by s:, and a!lx by (7.2). Now suppose that we observe, not y, but y' = y + v, where v is a random error with the distribution N(O, a!). Then y' will be normaIIy distributed about an expected value E[yl/X] =

with a variance

+ y(x -

'YJ

~)

(8.11)

V[y/lx] = a!lx+ a!.

(8.12)

Repeating the arguments of the first part of this section, we find that the essential change is the replacement of az lx by (a:lx + a~), and the joint density function of x, y' is a bivariate normal distribution with parameters I: 2 an d corre Ia t'IOn coeffi'clent PXy', given . by 5", '7, a""2 a.y'2 = ayl2", + a"2 + I' 2ax' the modified form of (8.4) as P"'II'=

(2

y2:~

ayl x+ a"

+ I'

2 2)!--2 =

( :2a!

ax

2)~ =

a y + a"

(y2:!)!--2( 2a: 2)!--2 ay a y + a"

2 2)~'

(8.13) a+: ay a" Thus r XY ' is not the maximum likelihood estimator of PXY' On the other hand, ay, ( y 2a! )~ (a!x+ a! + y2a!)~ (8.14) f3Y'lx= PXY' - = 2 2 22 = 1', ax alll"'+ a" + I' ax ax so introducing error into the observation of y leaves bY'I", an unbiased estimator of y. Now suppose that, in addition to the error in y, we observe not x but x' = x + u, where u is a random error with the distribution N(O, 0':). We assume, of course, that the errors of measurement u, v, are independent of each other and of x and y. We have E[x' ] = ~, V[x' ] = + o'~. Since the bivariate normal distribution is symmetrical in form, if adding error into = + left unchanged, and multiplied in y changed the correlation coefficient by = PXY(

0':

a!

a:

a:, a! a!, PXy' ( =

PXY

2

a1/

a:

)!--2

2'

+ a"

(8.15)

417

EXERCISES

then adding error in x will have the analogous effects, i.e., changing a! into a!, = a! + a~ and changing the correlation coefficient into PaN given by (8.16) Thus, substituting for P,JJY' from (8.13),

Pre'v' = PreYC! ;

c:

a~r ~ a!f =

Prey{ [ ( 1 +

:D ( + :D]~rl 1

(8.17) Thus '"re'y' is not the maximum likelihood estimator of Prey. Also, (8.18) From (8.3) and (8.4), (8.19) so (8.20) Thus introducing error into the observation of x makes by""" a biased estimator of y, the regression coefficient of y on x in the original model. It is possible for the model of this section, with different values of the parameters a!, a,~, a;,,,,, a~ and y, to lead to identical bivariate normal distributions with the same values for the parameters a!" a~, and P""y'. Thus the same bivariate normal population can be produced by sampling from different regression situations.

EXERCISES 12.1. For a number of storms, the amounts of rain falling at two sites were observed. The amounts, transformed in a certain way to give better bivariate normality, were as follows: 1.05 1.40 0.69 1.41 0.51 1.49 1.38 2.00 0.96 1.31 2.07 1.02 0.89 1.51 x2 0.66 1.16 0.64 1.07 0.33 1.59 1.11 1.33 0.96 1.40 1.71 0.75 0.75 0.92

Xl

n = 14, LXI = 17.69, L x 2 = 14.38, L xf = 24.9465, L x~ = = 20.1135.

For these data, 16.7508, L x l x 2

418

THE BIVARIATE NORMAL DISTRIBUTION

CHAP.

12

(a) Obtain estimates of (i) p, (ii) V[x 21, (iii) V[x 2 ix l l. (b) Test the null hypothesis that p = O. (e) Obtain the regression equation of x 2 on xl' (d) Obtain the regression equation of Xl on x 2 • (e) If we observe Xl = 1,4, what is the predicted value of x 2 ? (f) If we observe x 2 = 1.13, what is the predicted value of xl? (g) Another set of 18 storms gave a sample correlation coefficient between another pair of sites of 0.703. Test the null hypothesis that the correlations between the two sites in each pair are equal. 12.2. For the model of Section 12.8, sup~ose that (a) a~ = 2/3, a:'3l = 55/96, /' = 3/8, a~ = 1/3, a~ = 1/3, and alternatIvely that (b) a~ = 1/2, a~IOl = 3/8, /' = 1/2, a~ = 1/2, a~ = 1/2. Compare for the two bivariate normal distributions so produced a~" a:" POl'y'. Also compare {JY'I3l' with /' for (a) and (b). 12.3. Suppose that x, yare independently normally distributed with zero means and unit variances. What is the density function of x/V?

CHAPTER 13

Regression on Several Independent Variables

13.1. Introduction Chapter 11 dealt with the regression of a dependent variable y on a single independent variable x. This chapter will deal with the regression of y on r independent variables Xi' Although it is merely a special case, we will deal with r = 2 before the general case of any r, but before that we will establish an essential result. 13.2. Linear Transformation of the Variables in a Bivariate Normal Distribution to Give Independent Variables We suppose that Xl' X 2 are distributed in a bivariate normal distribution such as (12.3.17). Now introduce new variables Yl' Y2 defined as

(2.1)

The y's correspond to a new set of coordinates centered at Xl = ;1, X 2 = ;2, and rotated with respect to the original coordinates through an angle ex. The inverse functions are (2.2)

419

420

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

We now use (12.2.16) for the probability density function of a trans~ formation:

iJg1 iJg2 iJY1 iJY1 iJg1 iJg2 iJY2 iJY2

(2.3)

Here Po: is the usual bivariate normal distribution (12.3.17) for variables x 2 • Differentiating (2.2) gives

Xl,

iJg2 = cos cx; (2.4) iJY2 so the determinant is cos cx . cos cx - (-sin cx)(sin cx)

=

1.

(2.5)

Hence

P{Y1, Y2}

=

1

/

27r(11(12Y

X

2

1- P

1 2 [(Y1 cos cx - Y2 sin CX)2 + (Yl sin cx 2(1 - p ) (11

exp {_

_ 2p (Yl cos cx - Y2 sin CX)(YI sin cx

+ Y2 cos CX)2 (12

+ Y2 cos cx]}.

(2.6)

(11(12

Denote the exponent of this function by G. Recalling that 2 sin cx cos rt. = sin 2cx, cos 2 cx - sin 2 cx = cos 2cx, G can be written as G -_

-

2 1 2 [2(COS Yl - - 2 -cx 2(1 - p ) (11

+ Y22

+ -sin- 22 -cx -

sin p -2CX) -

(12

sm cx

'2 ( --2-

'2)

+ cos cx sm cx --2- + p - 2

(11

2cx - YIY2 ( -sin -2(11

(11 (12

(12

-

sin 2cx

--2(12

(11(12

2CX)] . + 2p cos --

(2.7)

(11(12

Thus P{Y1, Y2} has the form of a bivariate normal distribution with mean (0, 0) and variances and correlation coefficient which are functions of (1~, (1~, p, and cx. If we make the coefficient of Y1Y2 in (2.7) zero, we will have Yl> Y2 with zero correlation. Putting this coefficient equal to zero requires

(2.8) or

(2.9)

SECT.

13.2

LINEAR TRANSFORMATION OF THE VARIABLES

421

{fwe use the value of IX determined by this equation, we will have PYtY, = O. Now the probability density function of YI' Y2 with zero correlation is (2.10) But we already have an equation for P{YI, Y2}, namely, (2.6). These must be identical for all values of YI' Y2' This includes YI = 0, Y2 = 0, so we can insert these values in the two equations and equate the results. This gives (2.11) Also (2.6) and (2.10) must be identical for YI

= 1

=

1, Y2

=

1, which implies

~ lC~ + ;~);

(2.12)

so, if we multiply each side by the square of the corresponding side of (2.11), we get (2.13) Adding and subtracting (2.11) gives

(O'Yt

+

O'y,)2 = O'~

(O'Yt - O'Y/

+ O'~ + 20'10'2.J1

= O'~ + O'~ -

-

l,

(2.14)

20'10'2J1 _ p2.

Thus, if we know O'I, 0'2' and P for the original distribution of (Xl> x 2), we can determine O'y! and O'y, to use in the equation for the probability density of(YI' Y2), (2.10). If we substitute back for Xl> X 2 in (2.6), we get

(2.15)

422

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

But p{Yi> Y2} is also given by (2.10). Equating these two, and using (2.11), we get

(2.16) The y's have zero means and zero covariance; so the right-hand side is the sum of squares of two independent unit normal deviat,es, which must have the X2 distribution with 2 degrees of freedom. Hence the left-hand side of (2.16) has the same distribution:

_1_2[(XI I-p

~

~1)2_ 2p Xl

-

~

~l X2- ~2 + (X2 - ~2\2J ""'l(2). ~

~

J

(2.17)

We will need this result in the following section. 13.3. Regression on Two Independent Variables We assume that y is distributed normally about an expected value 1] with variance 0'2, and that the observations are independent. We assume that 1] is a simple linear function of two independent variables Xl and x2 : (3.1) Although this is the standard terminology for Xl and X 2 , it is misleading in that there is no requirement that Xl, x 2 be independent in the statistical sense. All that is required is that they be variables whose values Xlv' x 2\. be known for each value of 'V. We assume that we have n triplets of observations (Y v' Xlv' X2v )' We wish to obtain sample estimates a, bi> b2 and S2 of the parameters ex, PI' {32 and 0'2. The estimated equation is (3.2)

and the sum of squares of deviations between the observed values Y. and the values predicted by the estimated equation is R =

Lv (Yv 11

Yv)2 =

Lv {Yv "

[a

+ bl(Xlv -

Xl)

+ b2(X 2v - x2 )W.

(3.3)

SECT.

13.3

423

REGRESSION ON TWO INDEPENDENT VARIABLES

To minimize this we differentiate with respect to a, bi> b 2 and equate to zero:

oR =

-2

ObI

i

[Yv - a - bl(Xlv - Xl) - b2(X 2v - X2)](Xlv - Xl) = 0,

v

(3.5)

oR = ab2

-2

i

[yv - a - bl(Xlv - Xl) - b2(X 2v - X2)](X2v -

x2) = 0.

v

(3.6) These three equations imply

(3.7)

" (Xlv L

Since

Xl) =

v

n

L" (yv -

Yv)(x lv - Xl)

L" (yv -

Yv)(x2v -

°= L" (X2v -

= 0,

(3.8)

x2) = 0.

(3.9)

x2), (3.4) gives us

n

n

flo

LYv - l1a - blL(Xlv - Xl) - b2 L(x 2v - X2) y

y

v

= LYv -

na

= 0,

y

(3.10) whence

1

a

n

= -n LYv' v

(3.11)

From (3.5) and (3.6) we get a pair of simultaneous equations, known as the normal equations, for the two unknown b's with coefficients involving Sums of squares and sums of products which can be calculated from the data: "

bl L (Xlv v

"

n

xl )2 + b2L (Xlv v

bl L (Xlv - XI )(X2v - X2) v

n

- XI )(X2v - X2) =

Lv Y.(Xlv -

n "

+ b2L (X 2v -

X2)2 =

L Y.(X2v v

Xl),

(3.12)

X2). (3.13)

424

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

Space can be saved by using the definitions

x: = ~ 11

~'

11

~' X:

XI )2,

(Xlv -

= ~ (X 2• -

I'

= I"

XIX2

X2)2,

•

v

xl )(a:2v - x2),

(XIV -

(3.14)

V 11

~' yXI

11

= ~ Y.(Xlv -

~' yX2

Xl),

v

= ~ Y.(X2v •

X2).

with which the normal equations (3.12) and (3.13) read

bl~' x~ + b2~' XIX2 = I' yXI' bl~' XIX2 + b2~' X: = ~'YX2' Ifwe multiply (3.15) by bl~' X~

I' x~ and (3.16) by I'

XIX2

(3.15) (3.16)

we get

I' X~ + b2~' XIX2~' X~ = ~'YXI~' X~,

bl(~' XI X2)2

+ b2~' XIX2~' X~ =

(3.17) (3.18)

~'YX2~' XI X2.

Solving for bl gives

I'

I'

b _~' yXI X~ YX2~' XIX2 I ~' XI~' X~ - (~'XIX2)2 .

(3.19)

By similar manipulations,

b2

= I' yX2 ~' X~ - I' yXI I' Xl X2 . ~' X~ ~' X~

-

(~'

(3,20)

Xl X 2 )2

The solution for bl , (3.19), can be written in the form

bl = [~' xi~' x~ - (~' XI X2)2]-1 X

=

[~' x~ ~ (Xlv -

[I' x~~' x~ - (I'

x

{~[(~' X:)(X I• -

~' XIX2 ~ (x 2• -

:i\)y. -

X2)Y.]

X I X 2)2]-1

Xl) -

(~' XIX2)(X2• -

X2)]y,,}, (3.21)

which shows that b l is a linear function of the y's. Since the y's are normally distributed, bl will be normally distributed, with variance

SECT.

13.3

425

REGRESSION ON TWO INDEPENDENT VARIABLES

If we define P"'1"'2 as the "correlation coefficient" between

Xl

and

X 2,

i.e., (3.23)

we can write (3.22) as V[b] 1

= ~. "".2 ~ Xl

[1 _ (1'

.2 "', ~ Xl ~

J- = ~

1 .

l

X l X 2)2

"',

2

"',

2

~ Xl

X2

1 - P"'1"'2 2

(3.24)

This can be compared with the expression for the variance of the regression coefficient of y on Xl alone, (N'L' x~, (11.2.16). Similarly V[b 2 ] = a 2

l' X21

"', •X21 ~ "', •X22 ~

•

1

a2

= ("" "', •x 22 1 ~ X1·X2 )2 ~

-

2 P"'1"'2

.

(3.25)

The same type of manipulation leads to

whence (3.27) If in (2.17) we substitute bl for Xl' b 2 for X2, etc., we get

We now substitute (3.22) for V[b l ], (3.25) for V[b 2 ], and use (3.27) to get (3.29)

[(bl - fJl)21' x~ + 2( bl

-

fJl)( b2 - fJ2) l' Xl X2 + (b 2- fJ2)21' x~],....., a2l(2). (3.30)

We will assume without proof the generalization of this to r independent variables Xl, ... , Xr: r

~ (b i i

r-l

-

fJi)21' x~

r

+ 21 1 i=l

(b i - fJi)(b j i=i+l

-

fJj)

l' xix

j ,.....,

a2x2(r). (3.31)

426

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

We can write the deviation between the observation Yv and the true value "Iv as

+ (Yv - "Iv) Yv) + (a - ex)

Yv - "Iv = (Yv - Yv) = (Yv -

+ (b l -

+ (b 2 -

{31)(Xlv - Xl)

{32)(X2v -

x2)·

(3.32)

Squaring and summing, and using (3.7), (3.8), and (3.9) to dispose of cross products, n

n

! (yv - "Iv)2 = ! (yv - Yv)2 v v

+ 2(b l

+ n(a -

ex)2

+ (b l

n

{31)2! (Xlv - Xt )2 v

-

n

{31)(b 2

-

-

{32)! (Xlv - XI )(X2v - X2 ) v

+ (b 2 -

n

{32)2! (X2v - X2)2.

(3.33)

v

The left-hand side of this equation is distributed as O'2 x 2(n). On the rightn

hand side! (Y v -

Yv)2 has n - 3 degrees of freedom since the Yv - Yv

v

have to satisfy the three linear restrictions (3.7), (3.8) and (3.9). The second term, involving a, has one degree of freedom. Third, we showed in (3.30) that the last three terms jointly are distributed as O'2 X2(2). Thus the conditions for Cochran's theorem are satisfied, and the three comn

ponent sums of squares, namely, ! (yv -

Y)2, n(a - ex)2 and (3.30) are

v

independently distributed as O'2 X2. If we define S2 as

(3.34) it will be distributed as O'2x2(n - 3)/(n - 3) and have expected value 0'2. We can make separate tests of the null hypotheses {31 = 0 and {32 = 0 by substituting this estimate S2 for 0'2 in (3.22) and (3.25), but in general these tests are not independent since usually Cov[bt> b2 ] :;6 O. We can make a joint test of the null hypothesis {31 = {32 = 0 as follows. Since (3.30) is distributed as O'2X2(2),

HCbl

- {31)2 ! ' x~

+ 2(b l

-

{31)(b 2

-

{32) ! '

XIX2

+ (b 2 -

{32)2 !' x~]

O'2X2(2) ,.....,-- (3.35)

2

13.3

SECT.

427

REGRESSION ON TWO INDEPENDENT VARIABLES

and has expected value (j2, and the distribution of the ratio of it to be £(2, n - 3). Hence, under the null hypothesis PI = P2 = 0,

~ (b~ I' x: + 2b l b2I' XI X2 + b: I' x:) '" £(2, Il -

2s

3).

S2

will

(3.36)

n

Both the numerator of S2, I (Yv - Y.)2, and the expression in parentheses v

in (3.36), known as the sum of squares due to regression, can be computed in the forms given. However, alternative identities are convenient. We obtain them as follows. Squaring and summing the identity Yv - y

= (Yv -

Yv)

+ (Yv -

(3.37)

y)

gives n

Iv (Yv -

n

=I

y)2

v

n

(Yv - Yv)2

+I

v

(Yv - y)2,

(3.38)

since the cross product n

2 ~ (yv - Yv)(Yv - y) v

n

= 2I

+ bl(Xlv -

(yv - Yv)[y

v n

= 2b l I v = 0,

Xl)

+ b2(X 2v -

X2) - y]

n

(yv - Yv)(X lv - Xl)

+ 2b 2I

(yv - Yv)(X2v - X2)

v

(3.39)

by (3.8) and (3.9). Rearranged, (3.38) becomes n

n

n

I(yv - Yv)2 = I(yv - y)2 - I(Yv _ y)2. (3.40) v v v The first term on the right-hand side is simply the sum of squares of deviations about the grand mean, and so we are left with finding a conn

venient form for

2 (Yv -

y)2. We note that

v

n

I

v

n

(Yv - y)2

=I

[a

+ bl(Xlv -

Xl)

+ b2(X 2v -

X2) - a]2

v

n

=I

[bl(X lv - Xl)

+ b2(X 2v -

X2)]2

v

n

= b~ I (Xlv - XI)2 v

+ b: I

n

+ 2b l b2I

(Xlv - XI )(X2v - X2)

v

n v

(X 2v - X2)2

(3.41)

428

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

We can obtain an alternative expression for the right-hand side as follows. Multiply the normal equations (3.12) and (3.13) by bl and b2 respectively: b~

n

n

2v (XlV -

XI)2

+ b l b 2 2v (Xlv -

n

bl b2

2v (XlV -

XI)(X2v -

X2)

n

+ b: 2 (X2v -

XI )(X2v - X2)

v

X2)2

n

=

bl

2v yixlv -

Xl),

(3.43)

n

=

b2

2v Yv(X 2v -

X2),

(3.44) and add: b~

n

n

2v (XlV -

+ 2bl b 2 2 (XlV -

XI)2

v

n

XI )(X2v - X2)

n

= bl 2 Yv(Xlv - Xl) v

+ b: 2 (X 2v v

X)2

n

+ b 2 2v yix2v - x2).

(3.45)

Thus the sum of squares for regression is conveniently calculated from the right-hand side of (3.45) and the remainder sum of squares is then found from (3.40). The test implicit in (3.36) is conveniently put in tabular form (Table 13.1). Table 13.1 Degrees of Mean Source of variance Sums of squares freedom squares bl~'

Due to regression

+ b2~' yX2

2

S2 2

- Yv)2

n - 3

s2

- y)2

n -1

yXI

n

2 (yv

Remainder

v n

2 (Yv

Total

v

Finally, the predicted value Y, Y

= a + bl(XI -

Xl)

+ b2(X 2 - x2)

(3.46)

+ (JI(XI -

Xl)

+ (J2(x 2 - x2).

(3.47)

has expected value 'YJ

= ex.

Since a is independent of bl and b2 , [Y]

a

=

2[1

V[a]

+ (Xl -

-+ (Xl 11

X1)2

XI )2V[b l ]

2 ,X~ -

+ (X2 + 2(XI -

X2)2V[b 2] XI )(X2 -

x2) Cov[b l , b 2]

2(XI - XI )(X2 - X2) 2' X1X2 + (X2 - X2)2I'x~J

2' X~ 2' X~ - (2' XIX2)2

.

(3.48)

13.4

SIlCT.

THE PARTIAL CORRELATION COEFFICIENT

429

13.4. The Partial Correlation Coefficient In the preceding discussion in this chapter the so-called independent variables Xl> X2 were regarded as fixed variables. They may, however, be random variables, in which case (y, Xl' X 2) will have a trivariate distribution. An important trivariate distribution is the trivariate normal: (4.1) where

OJ

is the determinant

= = since Pi;

=

1

P21

P31

P12

1

P32

P13

P23

1

(4.2)

1 - P~2 - P~3 - P~3

+ 2P12P13P23'

(4.3)

Pi;' and 4> is defined as

1(X~2

cp = -

OJn - 2 OJ 0'1

X~2 X~2 x;x; x;x; + OJ22 2" + OJ33 2" + 2OJ12 - + 2OJ13 + 2OJ 23 -X~X~) 0'2

0'3

0'10'2

0'10'3

0'20'3

(4.4) in which x~ = Xi - ~i' i element in OJ. Thus OJ n OJ12

=

-(P12 -

= 1-

=

1,2,3, and OJij is the cofactor of the ijth

P~3' OJ22

P13P23),

OJ13

=

= 1-

P~3' OJ33 = 1 - P~2'

P12P2a -

PIa,

OJ23

=

-(P2a -

P12Pla).

(4.5) The trivariate normal is the generalization of the bivariate normal to three variables. Geometrically it can be represented by concentric ellipsoids of constant density in three-dimensional space. We showed in Section 12.4 that when a bivariate normal distribution in Xl> X2 is integrated over X2 it gives a univariate normal distribution in Xl' Likewise, a trivariate normal in Xl, X 2 , Xa when integrated over Xa gives a bivariate normal distribution in Xl> X2 with parameters ~l> ~2' O'l> 0'2, P12' Now suppose that we have a trivariate normal distribution in Xl' X2, X3 and we choose to consider the correlation coefficient between Xl and X 2 with Xa "held constant," i.e., we take a thin slice through the trivariate distribution parallel to the Xl> X 2 plane and consider the distribution of Xl> X2. In other words, we consider the distribution function p{Xl> x 2!xa}. Bya simple extension of (1.21.7),

P{Xl> x21 x 3} =

p{ Xl'

X 2, X 3 }

P{Xa}

.

(4.6)

430

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

For simplicity, but with no loss of generality, we will assume that the means ~l' ~2' ~a are zero. The substitutions are straightforward. The function P{Xl' X2, xa} is given by (4.1)-(4.5). Since integrating P{Xh X2, xa} over Xl gives a bivariate normal distribution in X 2, Xa, when we further integrate over X 2 we get as the distribution of Xa a univariate normal distribution with parameters (0, O'~. After some manipulation, we can get (4.6) in the form

p{ Xl' x2 lxa}

=

{27TO'fJ 1 -

x exp {- ![1 - ( 2 _ 2

P~a0'2.J 1 - P~a [1 -

(

P12 - PlaP2a 2 YJ ~}-l .J(1 - P~a)(1 - P2a)

P12 ~ PlaP2a 2 )2J-l [[Xl - t13(O'l/~a) XS]2 .J(1 - Pls)(1 - P2S) 0'1(1 - PIa)

P12 - PlaP2a

. [Xl - PlS(0'1/O'S)Xa][x2 - P2a(0'2/O'a)Xa] .j(1 - p~s)(l - P~3) 0'1.J 1 - P~a0'2.J 1 - p~s

+ [x 2 -/2a(0'2/~a)xa]2J}.

(4.7)

0'2(1 - P2a) This has the form of a bivariate normal density function of random variables xllxa and x21xa with parameters (4.8)

E[XlIXa] = PIa 0'1 Xs = f3lSX a, O'a P2a 0'2 Xa = f32aXa, O's V[xllxa] = 0'~(1 - P~a), V[x2IXa] = 0'~(1 - P~a), E[x2IXa]

=

_ P(
P12 - PlaP2S / 2 2 Y (1 - Pls)(1 - P2a)

(4.9) (4.10) (4.11 ) '

(4.12)

using for (4.8) and (4.9) the relation Pii(O'i/O';) = {Ju from (12.4.20). The correlation coefficient (4.12) is usually written as P12.a and V[xllx a] and V[x2Ixa] are usually written as O'~.a and O':.a' Thus we see that the joint distribution of Xl and X 2 for a fixed Xa is a bivariate normal. The mean of the distribution, ({J13Xa, (J2axa), is a function of Xa, but the variances (4.10) and (4.11) are independent of Xa, as is the correlation coefficient (4.12). Thus though we derived p(xl lxa)(x2 for a fixed Xs, we obtain the same expression (4.12) no matter what the value of Xa.

!xa)

SECT.

13.5

CORRELATION COEFFICIENTS BETWEEN INDICES

431

The sample partial correlation coefficient 1'12.3 is the analogous expression with I"s in place of p's: =

1'12.3

/ V (1 -

2 1'13)(1 -

2 • 1'23)

(4.13)

Thus we can calculate 1'12.3 from the simple correlation coefficients 1'12' etc. The null hypothesis that P12.3 = 0 can be tested by the analog of (12.7.3),

1'12.3.j~ /

V1 -

(4.14)

2'

1'12.3

will be distributed under the null hypothesis as t(n - 3). We recall that there is a relationship between the simple correlation coefficient between Xl and X 2 , say P12, and the simple regression coefficient of Xl on X 2, say {312' By (12.4.20), (312 = P12(C1l /C1 2), where C1~, C1~ are the simple variances of Xl and X 2• Let {312.3 be the regression coefficient of Xl on X 2 in the multiple regression equation of Xl on X 2 and X 3 • It can be shown that, analogous to (12.4.20), {3

12.3

=

C11.3 P12.3 -

•

(4.15)

C1 2•3

Thus the partial correlation coefficient of Xl on X 2 , "holding X3 constant," is directly related to the multiple regression coefficient of Xl on X 2 in the multiple regression equation of Xl on X 2 and X 3 • 13.5. Correlation Coefficients between Indices In certain correlation studies it may be plausible to consider indices. Suppose that we have random variables Xi' i = 1, ... , 4, and form indices • 113

Xl

=-,

(5.1)

X3

It was pointed out by Karl Pearson in 1897 [I] that the interpretation of the correlation coefficient between such indices, (5.2) could present difficulties. Suppose that E[x i ] = i , V[x i ] = C1~, and the correlation coefficient between Xi and xi is PH' We can represent the observations Xi as generated by the model (5.3)

e

432

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

where E[Zi •. ] = 0, V[Zi •.] COV[ZiY'

=

E[z~,,]

=

CHAP,

13

O'~, Also

ziY] = E[ZiyziY] = E[(xiy - ~i)(xiY = Cov[xiy, xiv] = PiiO'iO'i'

~i)]

(5.4)

We first obtain an expression for E[ila]:

E[;13]

E[XlvJ

=

~1 E[(1 + Zlv)

~

~+~

~a

_ Zay

~a

~:::::,Jl(1

~a

+ Zly _ ~1

_ E[ZI.zay]

~a

~

(1 - Zay + z:; -

~1

= ~1 E[1

(1

= E[~1 + Zlv] = E[~1(1 + Zlv)es1 + zay)-IJ

=

~1~a

... )J

~a

~a

ZlvZay ~1~a

+ z:y -

... J

~:

+ E[Z!y]) = ~1(1 + O'! _ ~a

~

~a

PIa

~a

0'1 O'a). ~1 ~a

We obtain an expression for the variance of i 1a as follows:

V[i1a] =

E[ (::: - E[ila])]

={E[ilaWE[{~I( 1 + Z;:)es1( 1 + Z;:)-1 ~: X

= {E[i ]}2E[(1 13

+ O'~ _ ~:

(1 + ;i - PIa ~:;:rl -1fJ 0'10'3)-2(1 P13 ~1~3

+ 2 Zlv ~1

_ 2 Z3y

~a

+ Z~y

~~

(5.5)

SECT.

13.6 REGRESSION ON SEVERAL INDEPENDENT VARIABLES

433

Substituting (5.7) and (5.6) and its analog for E[i24] in (5.2) we get

PlS,24 =

[(O'~ + 0'= _ ~~

~=

2

O'lO'S) P13

(O'~ + O'! _

~l~S ~~

~!

(5.8)

2

0'20'4)J!--2 . P24

~2~4

This formula will give various special cases. For example, if Xs = X 4 , then i lS = X l /X 4 , i24 = X 2/X4 , PS4 = 1, O's = 0'4 and ~s = ~4' Then (5.8) becomes the correlation coefficient between Xl/XS and x 2 /X S'

PlS,2S =

[(O'~ + 0': _ ~~

~~

2

O'lO'S)

PIS

~l~S

(O'~ + 0': _ ~~

~=

(5.9)

2

0'20'S)J!--2 .

P2S

If the pairwise correlation coefficients between Xl> zero, i.e., if P12 = PIS = P2S = 0, then (5.9) becomes

PlS,2S = [( 0'12

~~

~2~S

X2

(5.10)

+ O's2) (2 0'2 + O's2)J!--2 ' ~=

~~

and Xs are all

~=

which is never zero except in the trivial case of O's = 0, which would imply that Xs was a constant. If the three coefficients of variation are equal, i.e., if (5.11) then (5.10) gives

1

(5.12)

P132S=-' , 2

13.6. Regression on Several Independent Variables The extension of regression analysis to I' independent variables is straightforward. We have (I' I)-tuples of observations, (Y., Xl;, ... , Xr,.), 'JI = 1, ... ,n. We assume that y is normally distributed with variance 0'2 about 1], where 1] is a simple linear function:

+

'7

= oc + f3l(Xl

- Xl)

+ ... + f3r(x r -

xr)'

(6.1)

Xl)

+ ... + br(xr -

x r),

(6.2)

The estimated equation is

Y

= a + b1(X l

-

434

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

and the sample estimates, a, bI> ... , br are obtained by minimizing the sum of squares of deviations between the observed and predicted valu~s, n

R =

I

n

(y. - y.)2 =

•

I

•

{Yo - [a

+ bl(xly -

Xl)

+ ... + b,.(xrv - xr)])2,

(6.3) by differentiating with respect to a, bl , ••• , br and equating to zero. This procedure gives 1n a = - I y. = fj (6.4) n • and a set of r simultaneous linear equations, the so-called normal equations:

+ b2I' Xl X2 + ... + br I' Xl Xr = I' yXl' blI' Xl X2 + b 2 I' x: + ... + br I' X2Xr = I' yX2'

bl I' x:

(6.5)

bl I' Xl Xr + b2 I' X2Xr + ... + br I' X~ = I' yxr • These equations can be solved for the b's. However, an alternative procedure which will also give us V[b i ] and Cov[b i , b;] is to be preferred. Suppose that there exist constants Cn, Cl2 , ••• , Cl r with properties to be defined later. Multiply the first of the normal equations by Cn , the second by C12 , etc., sum the resulting equations, and rearrange so as to collect all the coefficients of bl together, all the coefficients of b 2 together, etc.:

+ C12 I' Xl X2 + ... + clt'I' XlXr) + b2(c n I' Xl X2 + C12 I' x: + ... + Clr I' X2Xr )

bl(c n I' x:

+ b,.(cnI' XlXr + C12 I' X2Xr + ... + Clr I' x~) = Cn I' yXl + C12 I' yX2 + ... + Cl r I' yxr·

(6.6)

Let us require that the coefficient of bl in this equation is 1 and the coefficients of all the other b's are zero, i.e., that

+ C12 I' Xl X2 + ... + Clr I' XlXr = cnI' Xl X2 + C12 I' x~ + ... + Clr I' X2Xr =

cnI' x:

1, 0,

(6.7)

SECT.

13.6 REGRESSION ON SEVERAL INDEPENDENT VARIABLES

435

This is a set of r simultaneous linear equations for r unknowns, namely, the constants cu , j = 1, ... , r, and they can be solved to give solutions for the Cu in terms of the observed Xiv' since the various sums of squares and products 2' x~, 2' XiXi are readily calculated. Substituting the equations (6.7) in (6.6) gives bl

= Cll 2' yXI + Cl2 2' yX2 + ... + Cl r 2' yxr ;

(6.8)

so once the Cli have been found bl can be calculated simply, since the yXi are known numbers. For each bi' we can find a similar set of Cii (i = 1, ... , r) which will do the same trick for bi as (6.8) does for b l • For bi we have the set of simultaneous equations

I'

Cil

2' x: + Ci 22' XI X2 + ... + Cir 2' XlXr = 0,

Cil

2' XlXr + Ci2 2' X 2Xr + ... + Cir 2' X~ =

0,

where the right-hand sides are 1 for the ith equation and zero for all the others. The ith equation has 2' X~ as the coefficient of Cii' Analogous to (6.8), using (6.9) in (6.6), (6.10)

Recalling that 2' YXi

n

= 2 YvCXiV

-

Xi)' we can write this as

v

n

bi

= 2 Yv[Cil(Xlv -

Xl)

+ Ci2(X 2v -

X2)

+ ... + Cir(Xrv -

x r )],

(6.11)

v

which shows that b i is a linear function of the Y v ' Since the Yv are normally distributed, b i will be also. The quantity in square brackets in (6.11) is solely a function of the x's. It will be convenient to have a symbol for it, say k iv

Then

= Cil(Xlv -

Xl)

+ Ci2(X 2v -

x 2)

+ ... + cir(xrv -

xr)'

(6.12)

(6.13)

and n

V[b;]

= 2 k:vV[yv] v

=

n (J22 k:v' v

(6.14)

436

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

Thus when we have found I" k~., we will have V[b i ]. Now

I" k~. •

• = I" ki.ki• = I" ki.[Cil(Xly • • =

Cil

I"

•

ki.(xly - Xl)

-

Xl)

+ ... + Ct.(Xr• -

+ ... + Cir I"

•

ki.(xr• - xr)'

In the sequence of terms with the index ij on the c's, i fixed and j r, examine a particular one, say that for j = h: Co.

I"

(6.15)

=

1, ... ,

k i .( Xh. - Xh)

• = Cih I" •

[Cil(Xly -

" = CiI.[Cil I •

(Xly -

+ ... + Co.(Xh• - X,.) + ... + Ci.(Xr• - Xr)](Xh• " XI)(X h• - Xh) + ... + Cih I (Xh• - Xh)2 + ...

Xl)

•

CO.(Cil

Xh)

•n

+ Cir I =

Xr )]

(Xh• - X,,)( x r • - Xr)

I' XIX" + Ci2 I' x 2x" + ... + co, I' X:' + ... + Cir I' XhX,), (6.16)

We have two cases to consider; h

= i and h :F-

i. If h

= i, (6.l6) is

and the part in brackets is identically equal to the left-hand side of the ith equation in the set (6.9), for which the right-hand side is l. Thus, when h = i, (6.l6) equals Cii • For all values of h other than i, the part in brackets of(6.16) is one of the equations (6.9) which equals zero. Thus in (6.15) all the terms for which i :F- j are zero, and the one term for which i = j is equal to Cj j :

I"

k~.

=

(6.17)

Cii'

so, from (6.14), (6.18) To obtain the covariance of bi' b i , we proceed as follows: From (6.13), n

bi

+ bi = I

•

n

y.ki•

+I

•

n

y.Tc i •

=I

•

y.(ki•

+ k i .),

(6.20)

SECT.

l3.6

437

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

with variance n

V[bi

+ bi] = !

(kiv

n

+ k;v)2V[yv] =

0'2! (kiv

v

+ k;Y

v

n

= 0'2! le~v

v

n

n

v

v

+ 0'2! le~v + 20'2 !

leivle;v

n

= V[b i ]

+ V[b;] + 20'2! leivle;v.

(6.21)

Substituting in (6.19) gives

Cov[bi, b;]

=

n

(6.22)

0'2! leivle;v. v

From the definition of k iV ' (6.12), n

!v

n

leivle;v = cil ! k;.(Xlv - Xl) v

n

+ ... + cir !

k;v(xrv -

xr ).

(6.23)

In the sequence of these terms with the index if on the c's, i fixed and 1= 1, ... ,t, examine a particular one, say that for I = h:

Cih !" k;v(xhv - x,,) v

= CUI!" [C;l(Xlv -

Xl)

+ ... + c;,,(xhv -

X,,)

+ ... + c;,{xrv -

xr)](x"v - Xh)

v

(6.24) We have two cases to consider: h

=j

and h

y6

j. If II

= j, (6.24) is (6.25)

and, except that j appears in place of i, the part in brackets is identical with the ith equation in the set (6.9), for which the right-hand side is 1. For all values of II other than j, the part in brackets of (6.24) is one of the equations (6.9) for which the right-hand side is zero. Thus, in (6.23), all the terms are zero except one, which is C;j. Therefore

!"

lei,.le;v

=

(6.26)

cu ,

and so, from (6.22), (6.27) We can write the deviation between the observation Yv and the true value '7 v as Y. - 'YJv = (yv - Yv)

+

(a - CI.)

+ (b l

-

f31)(X lv

-

+ ... + (b r -

Xl)

f3r)(x rv

-

xr)' (6.28)

438

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

Squaring and summing gives n

1l

I

(y. - 1].)2

=I

v

(y. -

yy + /I(a

r

- OC)2

+I

r-l

r

+ 2I I

(b i

i=1 ;=i+l

-

(3i)2

-

I' X~

(3i)(b, - (3;) I' XiX;.

The left-hand side is distributed as ()'2x 2(n).

. I (Y. -

(b i

i

v

(6.29)

On the right-hand side

Y.)2 is distributed as ()'2x 2(n - I - r), n(a - OC)2 as ()'2X 2(1), and

•

the remaining part, as surmised in (3.31), is distributed as ()'2x2(r). Each of these components is independent of the other. If we define 1 n S2 = I (y. - y.)2, (6.30) 11-1-1'. it will be distributed as ()'2x2(n - 1 - r)/(n - 1 - 1') and have expected value ()'2. We can make separate tests of the null hypotheses (3i = 0 by substituting this estimate S2 for ()'2 in (6.18), but in general these tests are not independent since usually Cov[b i , h;] ¥= O. We can make a joint test of the null hypothesis (3i = 0 for all i as follows. Since (3.31) is distributed as ()'2x2(r),

;t 1 [r

(bi - (3i)2

r-l

r

I' X~ + 2i~ ,J_)bi -

(3i)(b; - (3;)

I' XiX,

]

()'2X2(,.) r-..J - , . -

(6.31) and has expected value ()'2, and the distribution of the ratio of it to 8 2, (6.30), will be F(r, 11 - 1 - 1'). Hence, under the null hypothesis Pi = 0 for all i,

1-(i b~ I' X~ + 1 i rs2

i=l '=i+l

i

bib, I' XiX,)

r-..J

F(r,

11 -

1 - 1'). (6.32)

We could calculate the quantity in parentheses, known as the sum of squares due to regression, in this form, but an identity is usually more convenient. We multiply the normal equations (6.5) by bl , b2 , etc. and sum: b~ I' X~ + b~ I' + . . . + b~ I' X~ + 2b 1 b2 I' Xl X2 + ...

x;

+ 2br_ I br I' Xr- 1Xr r r-l r = I b~ I' X~ + 2 I I bib, I' XiX i i=l '=i+l = bl I' yXI + ... + br I' YXr'

j

(6.33) which is the usual form for computing the sum of squares due to regression.

To calculate

i

•

(y. - Y.)2, arguments similar to those of (3.41) to (3.45)

SECT.

13.7

439

A MATRIX REPRESENTATION

will give n

!

(Y. - y)2 = b1 !'

yXI

+ ... + br !' YXr'

(6.34)

n y)2 _ !(Y. _ y)2.

(6.35)

and as in (3.40), n

n

!(Y. _ y.)2

•

= !(Y. _ •

•

Finally, the predicted value Y, (6.2), has expected value variance V[Y] = V[a]

..

+!

(Xi -

1],

(6.1), and

Xi )2V[bi ]

i=1

,,-1 r

+ 2! !

(Xi -

X;)(X; -

X;) Cov[bi , b;]

i=1 ;=£+1

13.7. A Matrix Representation The set of equations (6.9) gives a set of c;;,j = 1, ... , r, for each value of i, i = 1, ... ,r. These Cij can be written in a square matrix, say C, which is known as the c matrix:

C=

(7.1)

This section, which uses matrix notation, is not used explicitly in the remainder of this chapter and may be omitted. Its purpose is to show that C is actually the inverse of the matrix of the coefficients in the normal equations (6.5). Let the n X r matrix of values of Xi. - Xi' corresponding to the values of Xi at which observations were made on y, by X: Xu

-

Xl

X21 - X2

X"l -

X..

Xl2

-

Xl

X22 - X2

Xr2

-

Xr

X=

(7.2)

Xln - Xl

X2n - X2

Xrn - X..

440

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

Then the transpose of X, namely X', is

X' =

(7.3)

X rI -

Xr

X r2 -

xr

and, using the notation of (3.14),

X'X

k"'"

=

x

'1'X 2

k"'" X22

~'

XIX r

~'

X 2X r

(7.4) ~'

XIX,

~'

~' x~

X 2X r

is the matrix of the coefficients of the b's in the normal equations (6.5). Define Band G as -~'

yXI

~/YX2

G=

B=

(7.5) ~/yxr

br

Then the normal equations (6.5) can be written as X' XB = G. Premultiply this by (X' X)-l. Since (X' X)-I(X' X) = I, this gives B

= (X' X)-IG.

(7.6) (7.7)

Thus the regression coefficients b i can be obtained by premultiplying G, the right-hand sides of the normal equations, by the inverse of (X' X), (X' X) being the matrix of the coefficients ofthe b's in the normal equations. Suppose that C = (X'X)-l. Then (7.7) is equivalent to bl

Cll

C12

Clr

~/YXI

b2

C21

C22

C2r

~/YX2

(7.8)

=

br

Cri

Cr2

Crr

~/yxr

SECT.

13.8

A TEST COMPARING r VARIABLES AND

q

441

VARIABLES

which gives r

bk =

Ck1

1' yXI + Ck2 1' yX2 + ... + Ckr l' YXr = 1, Cki l' yx" (7.9)

which is identical to (6.10). Substituting C gives

= (X' X)-l in (X' X)-l(X' X) =1

l' X1Xr l' X 2X r

o

1

o o

l' X~

o

0

1

1 0

(7.10) Now two matrices are equal if and only if the corresponding elements are equal. If we carry out the matrix multiplication on the left-hand side of (7.10), we get a single matrix, and if we put each element in the first row equal to the corresponding element in the first row of the matrix on the right-hand side of (7.10), we get

1' X~ + c12 1' X 1X 2 + ... + Clr l' X1Xr = 1, cn1' X 1X 2 + c12 1' x: + ... + Clr l' X 2X r = 0,

C11

(7.11)

which set of equations is identical with the set (6.7), and in general equating the ith columns gives (6.9). This shows that the matrix of the cij obtained by solving the sets of equations (6.9) is in fact the inverse of the X' X matrix. 13.8. A Test of Whether Regression on r Variables Gives a Significantly Better Fit than Regression on q Variables Clearly there is no point in using a regression equation of y on r variables Xi' i = 1, ... , r, unless it gives a significantly better fit than the regression equation of y on a subset of the Xi' i = 1, ... ,q. In this section we derive a likelihood ratio test.

442

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

We first need the maximum likelihood estimators of ot and the fli' The density function of y is p{yv}

=

N(ot +

=

1

.J27T.J

*

fli(Xiv - Xi)' 0'2)

(1

[Yv - ot - *fliXiV -

exp - 0'2 2

xi)Tj

0'2

(81) '.

The likelihood function is

The logarithm is easier to handle:

J2

/n 1 n [ r log L= -n log y 27T - -log 0'2 - -2:2 Yv - ot - :2 flix;v - Xi) . 2 20' v i

(8.3) As usual, we differentiate with respect to the parameters:

a--~ log L i n [ = - -2 (-1)(2):2 Yv uot

20'

v

r

ot - :2 fli(X,v - Xi)

J= 0,

(8.4)

i

a8ii: log L i n [ = - 20'2 (-1)(2) ~ Yv -

ot -

t fli(X;v r

]

Xi) (Xiv - Xi) = 0,

(8.5)

alog L

11

~ = - 20'2 -

1

1

n [

2(-1) (0'2)2~

,

r

J = O. 2

Yv - ot - tfli(XiV - Xi)

(8.6) n

From (8.4) we get & = :2 yin, identical with the least squares solution for a in (6.4). From the set of r equations (8.5) we get a set of r simultaneous equations identical to those of (6.5) with = bi' From (8.6) we get

Pi

82 =

! iv

n

[Yv - & -

f P.cx;v i

Xi)J 2 = J

i (yv -

Yv)2.

(8.7)

11 v

Thus the maximum likelihood estimator of 0'2 is biased due to the occurrence of the divisor n in place of n - 1 - r. The maximum likelihood estimators of ot and fli are identical with the least squares estimators, but whereas the former are derived under the assumption of normality, (8.1), the derivation of the least squares estimators involved no assumption about the form of the distribution.

SECT.

13.8

A TEST COMPARING

r

VARIABLES AND

q

443

VARIABLES

We now construct a likelihood ratio test of the null hypothesis that y can be fitted by a regression equation with parameters IX, {JI> ... , {Jq against the alternative hypothesis that additional parameters {Jq+1' ..• , {Jr are necessary. be the estimate of the residual variance when q {J's are used, i.e., Let for ro, and let O'~ be the same quantity when all r {J's are used, i.e., for Q. Similarly, let P;q, Pir be the estimators of {Ji when the regression equation includes q and r (J's respectively. The likelihood ratio is

O'!

*

t

L(ro) [(2'7T(1!Y'/2r1 exp {_(2&!)-1 [yv - oc Piq(XiV - Xi)]2} A- = -------.:~--.!....----.!....----- L(Q) [(27T(1~)"/2]-1 exp {-(2&~rl ~ [yv - oc - Pir(Xiv - Xi)d .

*

(8.8)

The summations in the two exponents are identical with the summations in the expression for 8 2, (8.7), so (8.9) Any monotonic function of A will serve as our statistic. The function below on the left-hand side of (8.10) is a function of s:/s~ where s:' s~ are the usual unbiased estimators of the residual variance, and s:, s~ are, of course, functions of 8:, 8~. The statistic is usually calculated in the form given on the right-hand side, where Gr and Gq are as defined in Table 13.2: (r - q)-I[(1I - 1 - q)s: - (II - 1 - r)s~]

=

(G r

Gq )/(I' - q)

-

(8.10)

Table 13.2 Source of variance

Sums of squares

Degrees of freedom

Mean squares

A. Regression on Xl' ••• , xOJ Xq+h • . . ,

x,

G,

r

B. Regression on xb ·•• ,xO' C. Difference A - B

= difference E -

F. Total

Xq

G. r- q

D

D. Remainder using ~b ••• , xq , X O+ 1 , ••• , x, E. Remainder using Xb""

q

1I-1-r ll-1-q 11-1

G,- G. = (11- I - q)s: - (11- I - r)s~

G,- G. r-q

(11- 1 - r)s~

S2

(11- 1 - q)s:

82

,

•

444

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

While we have shown that this statistic is equivalent to the likelihood ratio statistic, we have not given any indication as to its distribution und~r the null hypothesis. In fact, under the null hypothesis it is distributed as F(1' - q, n - 1 - 1'). A proof of this for the general case is complicated (see, e.g., Section 14.3 of Anderson and Bancroft [2] and Section 5.8 of Kempthorne [3]) and we consider only the case of l' = 2, q = 1. For this special case, Table 13.2 becomes Table 13.3. Table 13.3

Sums of squares

Source of variance Regression on Xl> X 2 jointly Regression on Xl only Difference due to adding x 2 to

A B

Degrees of freedom 2 1

C=A-B

X1

Mean squares

C

Remainder using regression on xl> X 2 jointly

D =F-A

n-3

Total

F

11 -

S2

1

Our objective is to show that the difference, say C, between the sum of squares due to regression on Xl, X2 jointly, say A, and the sum of squares due to regression on Xl only, say B, is independent of the remainder sum of squares, say D, and that under the null hypothesis f32 = 0 the ratio

(8.11) [I' X1Y I' X1X I' X2Y I' X~]2 I' XUI' x~ I' x~ - (I' XlX2)2] .

2 =~--~~~=-~--~----~

(8.12)

We will derive (8.12) by a different route, which will make it clear that it is independent of S2 = D/(n - 3). Consider the regression of X 2 on Xl'

SECT.

13.8

A TEST COMPARING l' VARIABLES AND

q

445

VARIABLES

The estimated regression equation is

X2

=

_ X2

+

~'XIX2

~'X~

(Xl -

_ Xl)'

(8.13)

We define Xu as the deviation of x 2 from the value predicted by this equation, i.e., (8.14) Similarly, the estimated equation for the regression of y on

Xl

is (8.15)

and we define Y2.l as Y2.1

=

y - y- -

~'xy

£., I ( ~ Xl -

-)

(8.16)

Xl •

£., Xl

We now consider the regression of Yu on L Yu = 0 we can write

Xu'

Since

~ Xu

=0

and

(8.17) From the definitions of Xu and Yu it is straightforward to show that (8.18) (8.19) Making these substitutions in (8.17) and noting (3.20), we have b

_ (~' X~)-l[~' x~~' X2Y - ~'XIX2~' xIY] _ b (8.20) (~' XD-I[~' x~~' x~ _ (~' XIX2)2] - 2'

112.1:»2.1 -

The sum of squares due to regression of Y2.1 on Xu, say C', is, by (11.3.11 ),

(8.21)

446

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

The remainder sum of squares for the regression of Y2.1 on X2.1' say D', is

D'

=!

Y~.l

- (sum of squares due to regression of Y2.1 on xu)

(8.22) substituting for Y2.1 from (8.16) and using (8.21). It is straightforward to show that (8.22) can be written as D' =

!

(y _ y)2

_{(I' XIy)2 + (!' x~!' X2Y !' X~

!' XIX2!' XIy)2} !' x~[!' X~ !' X~ - (!' Xl X2)2] . (8.23)

Now from the identity of (8.11) and (8.12),

(!' XIy)2 + [I' xly!' XIX2 - !' x 2y!' X~]2 !' X~ !' x~[!' X~ !' X~ - (!' Xl X2)2] = sum of squares due to regression of Y on

Xl> X 2 jointly.

(8.24)

Therefore

D' =

!

(y - y)2 - (sum of squares due to regression ofy on Xl' x2 jointly)

= D.

(8.25)

By the usual arguments (see Section 11.3) for simple linear regression on one independent variable, under the null hypothesis that the regression coefficient is zero the remainder sum of squares D' is independent of the sum of squares due to regression C', and they both have the same expected value. It therefore follows under the null hypothesis that D = D' is independent of C = C', and that the ratio of the corresponding mean squares has the F distribution with the appropriate degrees of freedom. The test of Table 13.3 of whether X2 adds to the fit of the regression of Y on Xl is identical with the test of the null hypothesis that {32 in the joint regression equation is zero. For the latter test we would use (3.20) for b2 and (3.25), modified by the substitution of S2 for (12, for J7[b 2] for the statistic

1

!' x2y!' x~ - !' xly!' XIX2

= ~ (!' xDJ,2[!, x~!' x~ - (!'

XIX2)2]J,2'

(8.26)

SBCT.

13.10

FURTHER USES FOR THE

c MATRIX

447

and this will have the ten - 1 - 2) distribution under the null hypothesis. Its square is identical with the test of Table 13.3, the ratio of C as in (8.12) to S2, which has the F(l, n - 1 - 2) distribution. The two tests are therefore identical. 13.9. Polynomial Regression The conventional designation of Xl' X 2 as "independent" variables is unfortunate, for in general, except in suitably designed experiments, they are not independent, since usually their covariance is not zero. Nothing in the foregoing development is assumed about the Xi except that they are known variables measured without error. We are therefore quite at liberty to use as X 2 the square of Xl, i.e., to fit the equation

+ fJI(XI - Xl) + fJ2[(X~) = constant + fJIX I + fJ2X~

'YJ =

(l

(x~)]

The foregoing techniques can therefore be used to fit a second-degree curve if we think it necessary. The test of Table 13.3 enables us to test whether a second-degree curve gives a significantly better fit than a straight line. 13.10. Further Uses for the c Matrix One advantage of the c matrix method of handling multiple regression calculations is that once the c matrix has been obtained it is a relatively trivial matter to carry out the regression analysis of another independent variable, say z, on the same x's. We merely need the L' ZXi , i = 1, ... , r, and substitute them in (6.10) to obtain the regression coefficients of Z on the x's. In this way, e.g., we can readily compare the results of regressing on the x's various functions of y, e.g., log y or l/y, to find the functional form which gives the best fit. A statistic appropriate for comparing fits is the multiple correlation coefficient, the sample value of which is defined by

R2

=

1_

remainder sum of squares of y unconditional sum of squares of y

_ sum of squares due to regression - unconditional sum of squares of y .

(10.1) (10.2)

So far in this chapter, with the exception of Section 13.4 on the partial correlation coefficient, we have supposed that the Xi' i = 1, ... ,r, be known fixed variables. In general, however, they may be obtained in any of three different ways.

448

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

1. They may be chosen deliberately in some specified pattern. In particular, suppose that there are three variables on numerical scales, Xl taking only r values, X 2 only t values, and Xa only u values, and we obtain an observation on y for every combination of Xl with X 2 and with X 3• The total number. of observations will be rtu. In this case the simple and multiple regression coefficients of each X on each of the other x's will be zero, and by (12.4.20) and (4.12) the corresponding correlation coefficients will be zero also. 2. Certain ranges for the variables Xi may be selected, and combinations of values of Xl' X2 and X3 chosen by some randomization procedure. So far the only difference from case 1 is that the x's are chosen at random instead of by some deliberate pattern or design. However, now the regression coefficients of the x's among themselves will be distributed about expected values of zero with some variance. Furthermore, if the original choices of the Xi were made from normal distributions, then we will have a multivariate normal distribution with the sample correlation coefficients rfJJifJJj distributed around population values PfJJ/fJJ j = O. . 3. We may observe a system from the outside, without exercising any control over it. If the Xi have normal distributions, then we may have a multivariate normal distribution. Unlike case 2, in general the expected values of the regression coefficients of the x's among themselves will not now be zero. In cases 2 and 3 we may be prepared to assume the multivariate normality. As implied in Section 13.4, just as we can carry out a simple regression analysis on a bivariate normal population, so we can carry out a multiple regression analysis on a multivariate normal population. We may wish to make tests of significance and set confidence limits on the regression coefficients of xp on the other x's in the regression equation of xp on the other x's. The c matrix gives us directly these regression equations. Write out the equations (6.9) with i = 1-', omitting that with 1 on the right-hand side; at the same time move over to the right-hand side the terms involving cpp

SECT.

13.10

FURTHER USES FOR THE

Now divide throughout by

( _ C"l)

C,,"

C

MATRIX

449

-c""

~' xlxr + (_ C"2) ~' x 2x r+ ... + (_ C"r) ~' x~ = ~'x"xr' C,," C,,"

(10.4)

Comparing this set of equations. with a typical set of normal equations (6.5), it is apparent that these are the normal equations for the regression of x" on Xl> ••• , Xr (omitting x), and the regression coefficient of x" on Xl is -C"l/C"", etc. Thus, having found the C matrix (7.10), we can immediately write down the regression equation of x" on the other x's: x"

=

x"

+ (-

C"I)(XI - Xl)

e""

+ ... + (-

C"r)(Xr - xr)'

(10.5)

C,,"

Obviously in the sequence i = 1, ... , r on the right-hand side, i = I-' is omitted. Each row in the C matrix gives us the regression equation of a different x on the remaining x's. A more explicit notation for regression coefficients is sometimes desirable. Let Xi stand for the sequence Xl, X 2, ••• , Xr in which x" and X k are omitted. Then by b"'/l"'~' . we mean the regression coefficient of x" on X k in the regression equation of x" on all the x's, Xl to Xr (but naturally excluding x,,). Thus in (10.5) "'j

j

= 1, ... , r(butj =F 1,1-').

(10.6)

We can readily obtain from the C matrix the estimated variances of the regression coefficients. We first obtain the estimated residual variance of x" about the regression plane on the remaining x's, say S2. In the set of equations (10.3) we omitted the equation in (6.9) whose right-hand side was 1:

(10.7)

450

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

Dividing by (-c,,):

(- C"l) ,2'

XIX"

+ (- C"2) ,2' X 2X" + ...

C""

C"" (10.8)

Since -c"l/c"" is the regression coefficient of x" on Xl' etc., the left-hand side of this is exactly (6.33), the sum of squares due to regression. But we do not need to calculate it in that form, for (l0.8) gives it as,2' x! - I/c" . And, comparing (6.35), it is obvious that the residual sum of squares of; about the regression plane on the other x's is I/c"". This residual sum of" squares will have [11 - I - (I' - I)] = n - I' degrees of freedom; so s! = (l/c",,)/(n - 1'). We may note here another interesting relationship. * If Ri is the multiple correlation coefficient of Xi on the remaining x's, we can substitute the right-hand side of (10.8) in (10.2) to get

R~ =,2' x~ - 1/cu = 1 _ •

Solving for

Cit

~, x~1. £..,

1.._1_. Coo ~, x~, u£..,

(10.9)

and substituting this solution in (6.18) gives (10.10)

analogous to (3.24). This shows that V[b i ] increases as Xi is more highly correlated with the other x's. So far we have the regression coefficients of xI' on the other x's, and also the residual variance, but to get the variance and covariance of these regression coefficients we need the c matrix corresponding to the equations (10.4). This can be obtained from the original c matrix by the formula (lO.l1)

c;;

where is the ijth element of the c matrix obtained when X is omitted. To prove this formula, order the x's so that Xr is the variable to be eliminated. Then corresponding to (6.7), but omitting the last of those

*

Pointed out to the author by Cuthbert Daniel.

SECT.

13.10

451

FURTHER USES FOR THE C MATRIX

equations, we have

~' x~ + C12 ~' X l X 2 Cll ~' X l X 2 + C12 ~' x~

Cll

Cn ~' Xl X.._ l

+ ... + CUr-I) ~' XlXr-l + Clr ~' XIX,. = 1, + ... + CUr-I) ~' X 2Xr-l + Clr ~' X2X" = 0,

+ C12 ~' X 2Xr-l + ... + CUr-I) ~' X~l + Clr ~' X ..Xr-l =

O. (10.12)

The corresponding set of equations for the reduced

C

matrix is

I' X~ + C~2 ~' + ... + ~' XlXr-l = 1, C~l I' Xl + C~2 ~' + ... + C~(r-l) ~' = 0, C~l

Xl X2

X2

C{(r-l)

X:

X 2Xr-l

Subtracting corresponding equations in (10.12) from (10.13) gives (C~l

-

cll ) ~' x~

+ (C~2 -

c12 ) ~'

Xl X2

+ (C{(r-l) (C~l

-

Cll )

~'

XlX,._l

+ ... ~'

CUr-I»

XlXr-l -

+ (C~2 - C12) ~' X 2X r _ l + ... + (C~(r-l) - Cl(r-l» ~' X:_l

-

~'

Clr

Clr

~'

XlXr

= 0,

= 0, (10.14)

XrXr-l

whence

' + C1(,.-1)

C

Ur-l) "'" X X

k

-

I' r-l -

"'"

x x T'

£., '1'

Cl r

C ,_ C 11 11 "'" X X k ' I' r-l Clr

+ C12' -

C

12 "'" X 2'X r-l k

Clr

+ ...

I

+ CUr-I) Cl r

CUr-I) "'" X 2 k r-l

= k "'" 'X r-1'X r'

(10,15)

452

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

These equations are identical with the normal equations for the regression of Xr on Xl' ••• ,Xr-l' Thus (10.16) Substituting for

b:J),:J)j • :J)k

from (10.6) gives

(10.17) whence

(10.18) and in general, with x" rather than ,

cij

Xr

= cij -

as the eliminated variable, Ci"C"j

-- .

C,,"

(10.19)

Thus, to get the estimated variance of the regression coefficient of x" on we need

X k , S!Ckk ,

(10.20) and then

The formula (10.19) can be used also to drop from a regression equation a variable, say x,,, that appears, after construction of the regression equation, to be nonsignificant. We could, of course, start again from the beginning, but, if we have four or more independent variables, it is more expeditious to remove the /lth row and column from the C matrix and compute the new b's from (6.10). Alternatively, it can be shown that, if bi is the regression coefficient of y on Xi in the multiple regression equation of y on all the x's, and if b~ is the regression coefficient of y on Xi in the multiple regression equation of y on all the x's excluding x,,, then

(10.22) 13.11. Biases in Multiple Regression In (10.22), b~ represents the regression coefficient of y on Xi' the regression equation containing Xl> X2' ••• , Xm but excluding x"' We can write

SECT. 13.11

BIASES IN MULTIPLE REGRESSION

453

this as bY"'i' "'I' where Xj is understood to stand for the sequence of x's from Xl to Xm but excluding Xi and xI" Also, b i represents the regression coefficient of y on Xi' the regression equation containing all the x's. We can write this as bll"'i' "'/' where x; is understood to stand for the sequence of x's from Xl to X m , excluding Xi' but including xI" Also, bJJ is the regression coefficient of y on xI" the regression equation containing all the x's. We can write this as bll"'JJ' "'t, where is understood to stand for the sequence of x's from Xl to Xm but excluding xI" The remaining item in (10.22) is -ciJJ/cJJJJ • Since c iJJ = CJJi , this item is equal to -cJJ;/cJJJJ • Reference to (10.5) shows that this is the regression coefficient of xI' on Xi' the regression equation containing as independent variables all the x's except x. I' We can write this as b",I''"i ' ""I where x,' is the same as in the preceding paragraph. We thus write (10.22) as

x;

where

Xi

bY"'i''''1 = bY"'i'''';' + b"'JJ"'I·",ill"'JJ''''/ = Xl> ... , Xm but excluding Xi' XI"

x~ = Xl' ... , X;

but excluding

Xi'

= Xl' ... , Xm but excluding

xI"

X".

(11.1)

The implication of this equation may be clearer if we consider a simple specific instance, say m = 3, i = 1, f.l = 3. Then Xj = X 2' x; = X 2X s ' = X I X 2 ' and (11.1) reads

x;

(11.2) This equation shows that the regression coefficient of y on Xl, ignoring xs, is a biased estimator of the regression coefficient of yon Xl' not ignoring xs, by an amount a function of the product of the regression coefficient of y on Xs and the regression coefficient of Xs on Xl' Equation (11.2) illustrates why the application of multiple regression techniques to observational data can be so treacherous and misleading. The apparent regression of y on Xl may really be due to the fact that y is dependent on xs, and Xs is correlated with Xl' We may fail to observe xs, and attribute the regression of y on Xl to a functional dependence which may be wholly false. In most circumstances, therefore, any indications produced by a multiple regression analysis of observational data are merely a good hint to try for confirmation by a proper experiment. In a true experiment the independent variables will be properly randomized with a table of random numbers and will have low correlations differing from zero by only random fluctuation, or else in a completely balanced experiment the correlations will be exactly zero.

454

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

The justification sometimes advanced that a multiple regression analysis on observational data can be relied upon if there is an adequate theoretical background is utterly specious and disregards the unlimited capability of the human intellect for producing plausible explanations by the carload lot. For attempts to investigate these difficulties, see Tukey [4] and Simon [5]. A further reason for being suspicious of inferences from a multiple regression analysis on observational data is that there is no guarantee that the residuals are independent. 13.12. An Example of Multiple Regression The data of Table 13.4 will be used to illustrate the foregoing procedures. They were obtained from 21 days of operation of a plant for the oxidation of ammonia to nitric acid. The Xl column represents the rate of operation Table 13.4 Xl

X2

X3

Y

Xl

X2

X3

Y

80 80 75 62 62

27 27 25 24 22

89 88 90 87 87

42 37 37 28 18

58 58 58 58 50

18 17 18 19 18

89 88 82 93 89

14

62 62 62 58 58

23 24 24 23 18

87 93 93 87 80

18 19 20 15 14

50 50 50 50 56 70

18 19 19 20 20 20

86

7 8 8 9 15 15

Xl

x2 x3

y

72

79 80 82 91

13 11

12 8

= air flow = cooling water inlet temperature = acid concentration = stack loss

of the plant. The nitric oxides produced are absorbed in a countercurrent absorption tower. The third variable is the concentration of acid circulating, minus 50, times 10: i.e., 89 corresponds to 58.9 per cent acid. The X 2 column is the temperature of cooling water circulated through coils in the absorption tower. The dependent variable y is 10 times the percentage of the ingoing ammonia to the plant that escapes from the absorption

SECT.

13.12

455

AN EXAMPLE OF MULTIPLE REGRESSION

column unabsorbed, i.e., an (inverse) measure of the overall efficiency of the plant. I We will fit a linear regression equation similar to (6.2). We need all the sums, sums of squares, and sums of products (Table 13.5), and these Table 13.5

Y

Y

Xl

X2

X3

8518

23,953 78,365

8,326 27,223 9,545

32,189 109,988 38,357 156,924

368

1,269

443

1,812

xl x2 x3

Totals

are expressed as sums of squares and products of deviations from the means in Table 13.6. We now set up the equations (6.9). To give more convenient numbers we will temporarily multiply the right-hand sides by 10,000. Also, although we can obtain the hi from the c matrix from (6.10), we might as well add an additional set of right-hand sides L' YX i • We are thus solving four separate sets of three simultaneous linear equations, in which the numerical coefficients on the left-hand sides, L' x~, L' X 1 X 2' etc., are identical for the four sets, but those on the right-hand sides do differ. Table 13.6

Y Xl x2 x3

Y

Xl

X2

X3

2069.238

1715.286 1681.143

562.952 453.143 199.810

435.857 491.429 132.429 574.286

The equations (6.5) and (6.9) are repeated in rows I through 3 of Table 13.7, and the numerical values for the coefficients entered from Table 13.6 in rows 4 through 6. The problem of solving simultaneous linear equations, though simple in theory, is arduous in practice, and has a long history. For conventional desk calculation a method known as the Doolittle, though probably due to Gauss, is one of the most satisfactory. Various modifications have been proposed, but the effort in learning their details seems to outweigh the slight savings they achieve. For r greater than 5, the work becomes excessive, and recourse should be made to an electronic digital computer. Programs are available for the standard machines.

~

0\

1:;

Table 13.7 Cll

1 2 3

l:' :I:~ l:' :1:1:1:2 l:' :l:l:1:a

4 5 6

1681.143 453.143 491.429

7

-1.0

8 9

453.143 -453.143

10 11

Cia

l:' :l:l:1:a l:' :1:: l:' :l:2:1:a

l:' :l:l:1:B l:' :l:a:l:B ~' :l:aI

491.429 132.429 574.286

-0.269544590

-0.292318381

77.667756 -1.0

i= 1

Cia

453.143 199.810 132.429

199.810 -122.142244

0

Right-hand sides

Coefficients of

132.429 -132.462028 -0.033028 0.000425247

l:' Y:l:l l:' y:l:a l:' y:l:a

1715.286 562.952 435.857 -1.020309396 562.952 -462.346062 100.605938 -1.295337256

i= 2

1=3

~

~

10,000 0 0

0 10,000 0

0 0 10,000

~

10,000 0 0

0 10,000 0

0 0 10,000

t""

0

0

0 -2,695.44590

10,000 0

0 0

-2,695.44590 34.704825

10,000 -128.753559

0 0

-5.94833896

til

t'I!

rii

~

~

t'I! t'I!

-= ~t'I! z .-j ~ ~

~

@

~

-

IN

1.0

26

0.044468874 1.985070 -0.002887 -6.788121

1.0

21

-0.349133712 9.355276 -34.704837 -0.002662

-0.000064691 -0.002887760 +0.000004199 +0.009874945

-1.0

17 18 19 20

-1.0

430.632256 1

-0.000001

15 16

22 23 24 25

574.286 -143.653730 -0.000014

491.429 -491.429

132.429 -132.462028 0.033027

12 13 14

0.715644558

-1.020309396

1.295272565

-1.295337256

-65.509844 -0.152124796

435.857 -501.409626 0.042782

17.288615

-5.948339

-34.707713

-34.707724

0

128.753563

-128.753559

4.25247 0.0098749

-2,924.33004 -6.7907826

34.704825

0 0 4.25247

0 -2,923.18381 -1.14623

-6.790783

0

0.009874945

0

10,000 23.221669

10,000 0 0

-J

VI

,J:o.

!

~

~

!:; ::a

~

~

~

~

~

trl

Z

>

N

~

--

R

~

458

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

The procedure for solving the equations is as follows. 1. In line 7 divide line 4 by minus the coefficient of Cil in line 4, i.e., by -1681.143. 2. In line 8 write out line 5 again. In line 9 multiply line 4 by the coefficient of Ci2 in line 7, i.e., by -0.269544590. Add lines 8 and 9 together; the coefficient of Cil vanishes, and we get line 10. 3. In line 11 divide line 10 by minus the coefficient of Ci2 in line 10, i.e., by -77.667756. 4. In line 12 write out line 6 again. In line 13 multiply line 4 by the coefficient of Ci3 in line 7, i.e., by -0.292318381. In line 14 multiply line 10 by the coefficient of Ci3 in line 11, i.e., by 0.000425247. Add lines 12, 13, and 14 together; the coefficients of Cil and Ci2 vanish, and we get line 15. 5. In line 16 divide line 15 by the coefficient of Ci3 in line 15, i.e., by 430.632256. This gives successively, on the right-hand side, b3 = -0.152125, Cl3 = -6.790783, C23 = 0.009875, and C33 = 23.221669. Of (:ourse, the three Ci3 need multiplying by 10-4 to give the correct values. 6. Now go back to line 11. This line represents the following four equations:

-1.0b 2 + 0.000425247b3 -1.0C12 + 0.000425247c l3 -1.0C22 + 0.000425247c 23

= = = -1.0C32 + 0.000425247c33 =

-1.295337256, 34.704825, -128.753559,

o.

Substituting the solution for b3 from line 16, i.e., -0.152124796, gives

-1.0b 2 - 0.000064691

=

-1.259337256,

and this is schematically given in line 17. Solving for b 2 gives b2 = 1.295272, as is schematically given in line 21. Similarly, line 18 corresponds to substituting Cl3 = -6.7907826 in the second of the four equations above, -1.0CI2 - 0.002887760 = 34.704825, whence CI2 = -34.707713, as is given in line 21. The solutions for C22 and C32 are obtained similarly. 7. Now go back to line 7, which represents the following four equations:

= = 0.292318381c23 =

-LObI - 0.269544590b 2 - 0.292318381b3 -1.0cn - 0.269544590C 12 - 0.292318381c13

-1.0C2I - 0.269544590C22 -1.0c31 - 0.269544590C 32 - 0.292318381c 33 =

-1.020309396, -5.94833896, 0, O.

SECT.

13.12

459

AN EXAMPLE OF MULTIPLE REGRESSION

Substituting b2 = 1.295272 and b3 = -0.152124796 in the first of these gives -LObI - 0.349133712 + 0.044468874 = -1.020309396, which is represented schematically in line 22. The solution for bl is hI = 0.715644558, as is given in line 26. The three remaining equations above for Cn. C2b and C3l give lines 23, 24, and 25, respectively, and the final solutions are in the last, three columns of line 26. 8. Assemble these solutions in Table 13.8. Table 13.S. lO4cij

= 17.288615 = -34.707724 C3 1 = -6.790783 0.715645 bl =

= -34.707713 = 128.753563 0.009875 C3 2 = 1.295272 b2 =

= -6.790783 = 0.009875 c a3 = 23.221669 ba = -0.152125

Cll

Cl2

c l3

C21

C22

C23

9. Check the accuracy of these solutions by inserting them in the lefthand side of line 6 and seeing how the right-hand sides coincide with the required values. Here we get as left-hand sides 435.8569994, -0.034, -0.005, 10,000.000, which compare very well with the values in line 6. However, the need for carrying a large number of significant figures is pointed up by the discrepancy in the fifth decimal place between Cn and C12 • In the above calculations we obtained the bi directly by carrying the column L' YXi as an additional right-hand side. This was not strictly necessary, for we can use (6.10) to obtain, e.g.,

bl

=

[1715.286'

X

17.288615

+ 562.952

+ 435.857

X

(-34.707713)

X (-6.790783)] X 10-4 = 0.715633,

which differs from the direct solution inthe fifth decimai place on account of rounding errors. To make a joint test of the null hypothesis Pi = 0 for all t, we compute the sum of squares due to regression from (6.33): 0.715644

X

1715.286

+ 1.295273

X

562.952

+ (-0.152125)

X

435.857

=

1890.406.

The residual sum of squares about the regression plane is given by (6.35) as 2069.238 - 1890.406 = 178.832, and with 21 - 1 - 3 = 17 degrees of freedom this gives S2 = 10.520. These results are assembled in Table 13.9. The variance ratio, 630.135/10.520 = 60, being distributed as F(3, 17) under the null hypothesis, is overwhelmingly significant.

460

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

Source of variance

Table 13.9 Sums of squares

Due to regression About the regression plane

1890.406 178.832

Total

2069.238

CHAP.

Degrees of freedom

Mean squares

3 17

630.135 10.520

13

Separate t tests on the individual b i are made, using (6.18). In Table 13.10 tests of the individual null hypotheses Pi = 0 are performed. Under the null hypothesis, the ratios in the last column are distributed as t(17). The first two are obviously very significant, and the last nonsignificant. The individual confidence intervals for the separate b i can be constructed if desired. Table 13.10

v VTb

i 10.520 x 17.288 x 10-4 10.520 x 128.754 x 10-4 10.520 x 23.222 x 10-4

1 2 3

0.1349 0.3680 0.1563

The regression equation of Y on

Y

=

bi

i]

0.7156 1.2953 -0.1521

Xl> X 2 , X3

5.305 3.520 -0.973

is

17.524 + 0.715633(XI - 60.429) + 1.295273(x + (-0.152125)(x3 - 86.286)

2 -

= -39.919

+ 0.7156xI +

1.2953x2

-

21.095)

0.152Ix3'

To test whether X3 adds significantly to the regression on Xl and X 2, we need the latter regression. When we have only three independent variables, to drop one it is about as quick to start from scratch with the two we want. However, we will illustrate the procedure of Section 13.10. Since we are omitting X 3, in (10.20) p, = 3, and

c'

=

[17.288615 -. (-6.790783):1 x 10-4 23.221670 -J

=

[128.753603 - (0.009875):1 x 10-4 23.221670-J

11

c' 22

=

15.302766

X

= 128.753599

X

and, from (10.19),

c' = [-34.707713 - (-6.790783)(0.009875)J 1~ 23.22 1670 9

=

-34.704825 x 10-4

=

C~l'

X

10-4

10-4

'

10-",

SECT.

13.12

461

AN EXAMPLE OF MULTIPLE REGRESSION

We can calculate b~ from (6.10) as

=

b{

[1715.286 X 15.302766

+ 562.952

X

(-34.704825)] X 10--4

= 0.671159

and similarly b; b'

=

1.295338. Alternatively, we can use (10.22),

= 0.715645 _

1

(-6.790783)(-0.152125) = 0.671159 23.221669

which comes to the same thing. The sum of squares due to regression on Xl and X 2 only is, again using (6.33), (0.671159 X 1715.286

+ 1.295338 X

562.952) X lQ--4

= 1880.443.

We now construct Table 13.11 analogous to Table 13.3. The test of the null hypothesis that adding X3 to Xl' x 2 does not improve the fit is given by Table 13.11

Sums of squares

Degrees of freedom

Mean squares

Due to regression jointly on xl> x 2• X3 Due to regression jointly on Xl. x 2 Due to adding X3 to Xl> X 2 Deviations about regression plane on xl> x 2• x3

1890.406 1880.443 9.963

3 2 1

940.221 9.963

178.832

17

10.520

Total

2069.238

20

Source of variance

the variance ratio 9.963/10.520, which is clearly nonsignificant. The test of the joint hypothesis {J~ = (J; = 0 is given by the variance ratio 940.221/[(2069.238-1880.443)/(20-2)] which is clearly overwhelmingly significant. The regression equation for Xl and X 2 only is Y = 17.524 + 0.671159(x1 - 60.429) + 1.295338(X2 - 21.095) = -50.359 + 0.671159x1 + 1.295338x2.

We might ask for 95 per cent confidence limits for For these values, Y

=

-50.359

+ 0.671159

X

50

1]

at

+ 1.295338 X

Xl

18

=

50,

X2

= 18.

= 6.516.

The estimated variance of Y can be calculated from (3.48), which is applicable to the special case of two independent variables, or from (6.36), which is applicable to the general case of r independent variables. We

462

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

13

CHAP.

will use the latter form here. Since we are using only two variables, is given by pooling the last two lines in the body of Table 13.11: S2 == (9.963 + 178.832)/(1 + 17) = 10.489. Also Xl - Xl = 50 - 60.4286 == -10.4286, X2 - x2 = 18 - 21.0952 = -3.0952. Then

S2

J?'[Y]

= x

=

+ {15.302766 x (-10.4286)2 + 128.753603 (-3.0952)2 + 2( -34.704825)( -10.4286)( -3.0952)} x 10-4] 10.489[(1/21)

1.18893.

The square root of this is 1.0904, and t o.976(18) = 2.101; so confidence limits for 17 are at 6.516 ± 2.101 x 1.0904 = (4.23, 8.81). Finally, we will illustrate the use of the c matrix to give the regression equation of X3 on Xl, x 2 • Reading off the third line of Table 13.8, we have, for (10.5),

X - X • 3 -

+ (-

• 3

-6.790783)(X _ x) 23.221669' 1 • 1

= 86.286 + 0.292433(xl

-

+ (_

0.009875 )(X - x) 23.221669 2 2

60.429) - 0.000425(X2 - 21.095).

To test the significance, say, of the regression coefficient of this equation, we would use (10.21): J?'[b

] 31.2

= _1_ 21 - 3

X3

on

in

Xl,

17.288615 x 23.221669 - (-6.790783)2 = 0.03651. (23.221669)2

A test of the null hypothesis fJ31.2 1.53, which is distributed as t(18).

= 0 is given by 0.292433/~0.03651

=

EXERCISES 13.1. The following table gives data on death rate due to heart disease in males in the 55 to 59 age group, along with the proportionate number of telephones, and of fat and protein in the diet. (a) Test the significance of the regression of yon xl> in the regression of yon Xl alone. (b) Construct the multiple regression equation of yon xl> x 2 • (c) Make a joint test of tIte null hypothesis Pl = P2 = O. (d) Test whether adding X 2 to the regression equation (on Xl) has significantly improved the fit. (e) Construct the multiple regression equation of y on Xl' x 2 , and x 3 • (j) Give 95 per cent confidence limits for P3 in this equation. (g) Give 95 per cent confidence limits for 1] at Xl = 221, x 2 = 39, X3 = 7. (h) Test whether X 2 and X3 together add anything to the regression of y on Xl' (i) Construct the multiple regression equation of Xl on X 2 and X 3 • (j) Give 95 per cent confidence limits for the regression coefficient of Xl on X3'

463

EXERCISES

Australia Austria Canada Ceylon Chile Denmark Finland France Germany Ireland Israel Italy Japan Mexico Netherlands New Zealand Norway Portugal Sweden Switzerland United Kingdom United States

Xl

X2

Xa

y

124 49 181 4 22 152 75 54 43 41 17 22 16 10 63 170 125 15 221 171 97 254

33 31 38 17 20 39 30 29 35 31 23 21 8 23 37 40 38 25 39 33 38 39

8 6 8 2 4 6 7 7 6 5 4 3 3 3 6 8 6 4 7 7 6 8

81 55 80 24 78 52 88 45 50 69 66 45 24 43 38 72 41 38 52 52 66 89

xa

= 1000 (telephones per head) = fat calories as per cent of total calories = animal protein calories as per cent of

y

= 100 [log (number deaths from heart

Xl

x2

total calories disease per 100,000 for males in 55 to 59 age group) - 2] Sources: Xl from World Almanac and Book of Facts (New York: New York World Telegram, 1951) except the figures for Ireland and Ceylon, which were obtained by private communication from the countries concerned. x 2 , x a ' and x 4 from J. Yerushalmy and Herman E. Hilleboe, "Fat in the Diet and Mortality from Heart Disease: a Methodological Note," New York State JO/l/'llal of Medicine, 57 (1957), 2343-54.

464

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

Sums of Squares and Products

y xl x2 xa

Sums

y

xl

X2

Xa

78,624

123,591 288,068

39,409 68,838 21,807

1,248

1,926

667

7,504 13,226 4,042 772 124

Sums of Squares and Products of Deviations y X2 xl Xa y Xl

7828.364

14,334.273 119,455.455

x2

1,571.909 10,445.182 1,584.773

xa

469.818182 2370.363636 282.545455 73.090909

13.2. In the table below, referring to the year 1950, for 46 states, y = standardized liver cirrhosis death rate Xl = per cent of population "urban" (1950 Census definition of urban) x 2 = 100 (number of children ever born to women 4~9 years old)-l x3 = wine consumption in hundredths of U.S. gallons of absolute alcohol per capita of toal population x" = spirits consumption, likewise. Sources: y, Xa, and x" from Wolfgang Schmidt and Jean Bronetto, "Death from Liver Cirrhosis and Specific Alcohol Consumption: An Ecological Study," American Journal of Public Health, 52 (1962), 1473-82. xl and X 2 from Statistical Abstract of the United States, 1955, U.S. Department of Commerce. Two states, Oklahoma and Mississippi, are excluded on account of legal restrictions on the sale of liquor.

Alabama Idaho Iowa Maine Michigan Montana New Hampshire N. Carolina Ohio Oregon Pennsylvania

Y

Xl

X2

Xa

X"

41.2 31.7 39.4 57.5 74.8 59.8 54.3 47.9 77.2 56.6 80.9

44 43 48 52 71 44 57 34 70 54 70

33.2 33.8 40.6 39.2 45.5 37.5 44.2 31.9 45.6 45.9 43.7

5 4 3 7 11 9 6 3 12 7 14

30 41 38 48 53 65 73 32 56 57 43

465

EXERCISES

y

Utah Vermont Virginia Washington W. Virginia Wyoming Arizona Arkansas California Colorado Connecticut Delaware Florida Georgia Illinois Indiana Kansas Kentucky Louisiana Maryland Massachusetts Minnesota Missouri Nebraska Nevada New Jersey New Mexico New York N. Dakota Rhode Island S. Carolina S. Dakota Tennessee Texas Wisconsin Sums

34.3 53.1 55.4 57.8 62.8 67.3 56.7 37.6 129.9 70.3 104.2 83.6 66.0 52.3 86.9 66.6 40.1 55.7 58.1 74.3 98.1 40.7 66.7 48.0 122.5 92.1 76.0 97.5 33.8 90.5 29.7 28.0 51.6 55.7 55.5 2920.7

Xl

65 36 47 63 35 50 55 33 81 63 78 63 65 45 78 60 52 37 55 69 84 54 61 47 57 87 50 85 27 84 37 33 44 63 58 2588

X2

32.1 36.9 38.9 47.6 33.0 38.9 35.7 31.2 53.8 42.5 53.3 47.0 44.9 35.6 ·50.5 42.3 43.8 33.2 36.0 47.6 50.0 43.8 45.0 42.2 53.0 51.6 31.9 56.1 31.5 50.0 32.4 36.1 35.3 39.3 43.8 1907.9

Xa

12 10 10

14 9 7 18 6 31 13 20 19 10

4 16 9 6 6 21 15 17 7 13 8 28 23 22 23 7 16 2 6 3 8 13 533

x4

33 48 69 54 47 68 47 27 79 59 97 95 81 26 76 37 46 40 76 70 66 63 59 55 149 77 43 74 56 63 41 59 32 40 57

(a) Give 95 per cent confidence limits for the regression coefficient b1lzl' (b) Add X z to the regression equation. Test the null hypothesis PUZ"Xl = O. (c) Add Xa to the regression equation. Test the null hypothesis {J UZ3. Zl:r. = O. (d) Test the joint null hypothesis that {J"'Yr Z2Z3 = {JlI:r"zl"a = O. (e) Give the regression equation of xa on xl and x 2 • Give 95 per cent confidence limits for {J"'3:r•. J'\' (f) If you have strong views about hard liquor, make up your own exercises

involving

;1'4'

466

REGRESSION ON SEVERAL INDEPENDENT VARIABLES

CHAP.

13

Sums of Squares and Products of Deviations

y xl xB xa

y

xl

X2

Xa

24,741.3481

12,446.4783 11,158.8696

5817.9129 4209.4870 2233.0237

6167.47609 3327.95658 1403.6457 2155.152174

13.3. Suppose that xl' X B, Xa are distributed in a trivariate normal distribution. Let {J12.a be the regression coefficient of xl on X B in the regression equation of xl on x 2 and Xa, and let {J12 be the regression coefficient of Xl on X 2 in the regression equation of xl on x 2 alone. (a) Express the relationship between {J12.a and {l12 as a function involving P12, Pla.a' and P23' (b) What relationships involving P12, PIa, and P2a will make {J12.3 = {J12 ? 13.4. Assume that xl and X B are random variables. (a) Obtain an expression for the correlation coefficient between x I /x2 and x 2 , involving the parameters 0''''1' 0''''2' gl> g2' and P"'1"'2' (b) Suppose that the coefficients of variation of Xl and x 2 are equal, i.e., that 0'' '1/ g1 = 0''''2/g2' Give a simple form for the correlation coefficient between x I /x2 and x 2 • (c) Suppose further that P"'1"'2 = O. Obtain a numerical value for the correlation coefficient between x I /x 2 and x B• 13.5. Starting from equations of the form (11.2.13) for bY"'l and b"'1"2' and from equations of the form (3.19) of bY"'l'Y2 and bY"'2'Y1' show that

bY"'l

= bY"'1''''2 + b"'1"'2bY"'2''''1' REFERENCES

1. Pearson, Karl, "Mathematical Contributions to the Theory of Evolution. On a Form of Spurious Correlation Which May Arise When Indices are Used in the Measurement of Organs," Proceedillgs of the Royal Society, 60 (1897), 489-98. 2. Anderson, R. L., and T. A. Bancroft, Statistical Theory ill Research. New York: McGraw-Hili Book Co., 1952. 3. Kempthorne, 0., The Desigll alld Allalysis of Experimellts. New York: John Wiley and Sons, 1952. 4. Tukey, John W., "Causation, Regression, and Path Analysis," Chapter 3, pp. 35-66, in Statistics alld Mathematics ill Biology, Oscar Kempthorne et al. (eds.). Ames, Iowa: Iowa State College Press, 1954. 5. Simon, Herbert A., "Spurious Correlation: a Causal Interpretation," 10l//'llal of lite Americall Statistical Associatioll, 49 (1954),467-79.

CH APT E R 14

Two-Way and Nested Analysis of Variance

14.1. Introduction: The Model for Model I Analysis In Chapter 10 we considered the analysis of variance where the data were classified in one way. In this chapter we will discuss the analysis of two-way tables, first when both classifications are model I, then when both are model II, and finally the mixed case where one classification is model I and the other model II. We suppose that the data are in the form of Table 14.1. For example, rows could correspond to varieties of corn, and columns to quantity of some fertilizer, and we have n independent estimates xijv for each row X column combination. We assume that the 1'1 sets of 11 observations are random samples from I't separate populations, each normally distributed about means ~ij but all with the same variance 0'2. The model is Xi}.

i

=

=

~H

+ ziJ.: =

1, ... ,I': j

zii. "" N(O,

1, ... ,t:

'JI

=

(1.1)

0'2),

1, ... ,11. n

The means of the sample observations in each cell are xu. The mean of the sample means in the ith row is

1

X.

...

tIt

= (1/n) 1 xi}v, •

n

= -1 x.. = -11 xii' I; III i . •

(1.2)

l}.

and the mean of the sample means in the jth column is 1

1

r

r

n

I'll i

•

x. i . = -1 Xii. = -11 XiJ.· l' i

467

(1.3)

468

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

Table 14.1

j

Averages

2

2

r Averages

X.t.

X.I.

Table 14.2 gives the population means Ei; for each cell, the row and t

l'

= (l/t) 2 EI ;, L = (1/r) 2 Ei;, and the grand

column means, defined as ~i. mean defined as 11'_ ~ = -r 2~' I"

=

; l1't -rt 2i 2; E11..

i

1'_

= -t2; "~ '.

(1.4)

The deviations of each row mean from the grand mean are denoted by 1}1' 'fJi

=

~I.

~,

-

(1.5)

and the deviations of each column mean from the grand mean by';: (1.6)

Clearly, both these sets of deviations sum to zero: t

l'

2'fJI i

= 0 = 2';· i

(1.7)

Table 14.2 j

2 1 2

~11 ~21

r Means Deviations

~r.

E.• L -~=t.

~12 ~22

~r2

E.a E.a -~ =tz

~1I ~at

~rt

L

~.t-~=tt

Means

Deviations

E•• Ea.

~ •. -~=1J1

~r.

~r. -~=1J,

~

~a. -~=1Ja

14.1

saCC.

INTRODUCTION:

469

THE MODEL FOR MODEL I ANALYSIS

We define quantities Oil as the difference between the true mean for the ijth cell, ~ii' and what we would expect on the basis of an additive model,

+ row effect + column effect = ~ + 'lJi + ~i;

Grand mean

(1.8)

i.e. (1.9) Thus, if the true cell mean is equal to the prediction (1.8), then the additive JDodel holds and Oij = O. In other words,

= ~ + 'lJi + ~; + 0ii'

~ij

(1.10)

the Oij measuring the departure from the additive model (1.8). The Oil are known as the interaction constants. When we say that we have zero interaction, we mean that we have an additive model as in (1.8). The Oil sum to zero over each suffix, for each value of the other suffix; for the sum over i, r

r

But I 'lJi

r

r

Ii Oij = I ~ii i

r~ - I'lJi - r~j.

( 1.11)

i

= 0, from (1.7), and, from (1.6),

i

r{j

=

r~., 0

r~

-

r

= k~ E.o; 1

r~;

(1.12)

+ rE = o.

(1.13)

= 0,

(1.14)

i

so r

r

=I

IOij

i

r

~ij

I i

~ij

rE - 0 -

-

i

Specifically, we have t relations r

I

r

Oil

=I

i

j

and r relations

r

0i2

= ... = I

Oil

i

(1.15) However, effectively there are only r + t - 1 independent restrictions on the Oil. This is because we have, say, the t restrictions (1.14), which

(r ) = O.

when summed determine the relation ~t ~ Oii r

r

But then the ~ Oil

(t ) t ~ 0ii' must also sum to zero.

(l.l5), whose sum is ~ ~ Oil =

t

r

In

other words, only r - 1 of the r relations (1.15) will be independent. Thus the total number of independent restrictions on the Oij is r + t - 1. Since the number of the ii is rl, the number of independent constants in this group is (1.16) rt - (r + t - 1) = (r - 1)(1 - I).

°

470

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

We now rewrite the original model (1.1) as Xiiv

= ~ + 17i + ~i + ()ii + Ziiv'

(1.17)

The original model involved rt independent cell means ~ii; this model involves the grand mean ~, r - 1 independent row constants 17i' t - 1 independent column constants ~i' and (r - 1)(t - 1) independent interaction constants ()ii' The total number of independent parameters in the new model (1.17) is

1

+ (r -

1)

+ (t -

1)

+ (r -

1)(t - 1) = rt,

(1.18)

the same as the original model (1.1). If the full number of parameters are needed, very little will be gained by the change in the model. However, if the null hypothesis of additivity can be accepted, we can regard the interaction constants ()ij as zero, and then we have only to consider the grand mean and row and column parameters, a very substantial reduction in the number of parameters. Furthermore, it may happen that either the row or column parameters, or both, can be regarded as zero, which would allow further reduction in the number of parameters necessary to describe the situation. In fact, if both were zero, the only parameter left is the grand mean ~. Corresponding to the population parameters ~, 17i' ~i and ()ii' we will have sample analogs. The sample analog of ~ (1.4) is

_

1

X ...

r

t

n

= -111 xiiv, I'tn i i v

that of 17i (1.5) is (Xi .. - X.. ,), that of ~i (1.6) is (x.]. - x.. ,), and that of ()ij (1.9), is Xii. -

[x ...

+ (Xi .. -

x.. ,)

+ (x. i . -

x.. ,)]

= Xii.

- Xi .. -

x. i .

+ x.... (1.19)

Using these expressions we can write an identity analogous to (1.10) Xii.

= x... + (Xi ..

- X. ..>

+ (x. i . -

X.. ,)

+ (Xii.

- Xi .. -

x.i.

+ X...>.

(1.20)

lfwe subtract this from (1.10), we get an expression for the deviation of the sample mean for the ijth cell from the true value for the mean of that cell: Xii. - ~ii

= (Xii.

- Xi .. -

+ (Xi .. -

X... -

+ X... 17i) + (x. i . -

x. i .

()ii) X... -

~j)

This identity will be used in the following section.

+ (X ... -

~). (1.21)

14.2

SECT.

J

471

THE ANALYSIS OF VARIANCE

14.2. The Analysis of Variance

We start with the identity X·· x ••• = (x..lJ. - x.Z.. - x • i • 111' -

+ X ) + (x.

t..

•••

+ (X. i . - X.J + (XiiV -

- x . .. )

(2.1)

xii)'

and square and sum over i,j and 'V: r

t

n

t

r

L~i ~v (xiiv - x.,Y = ~ ~ (Xii. i i

Xi .. - x. i .

II

i

r

t

+ lit ~i (Xi .. -

+ 111' ~; (X. i . -

X.. /

r

+ x.J 2

t

n

+ ~i ~; ~ (XiiV -

x .. l

Xii/.

(2.2)

l'

This equation is entered in the second column of Table 14.3. We wish to Table 14.3 Source of variance

Degrees of freedom

Sums of squares

Mean squares

EIM.S.] r

1111: 112

r

Rows

111

1: (Xi.. -

r -

x.Y

s~

1

0'2+_'_

1

r-l

I

,,, 1:"

I

Columns

IIr 1: (x.}. - X... )2

t -

1

S2 3

1

0'2+ _ 1 _

1

t r

r

Interaction

111: 1: (xu. I

r

Within cells

Total

X.I. +

1)(t -

0'2 +

1)

. (r -

i

}

v

r

I

n

1: 1: 1: (xiiv 1

i

I

1)(t -

1)

X.. .)2

n

1: 1: 1: (xuv -

{

(r -

Xi ..

I

i

I

111: 1: 0:1

I

,.

XU.)2

rt(1I -

X.. ,)2

rtll -

1)

S2 1

0'2

1

obtain the distributions of the various mean squares s~, etc., under the various null hypotheses 17 i = 0 for all i, etc. Consider the identity XiJv - ~ii = (xiiV - xii)

+ (Xii.

- ~i:l)'

(2.3)

We square this and then sum over 'V: n

~ (xiiv - ~iJ? y

n

= ~ (XiiV y

XiiY

+ n(xi:l. -

~i:l)2.

(2.4)

472

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

The left-hand side is distributed as O'2x 2(n), and the two terms on the right_ hand side are distributed as O'2x 2(n - 1) and O'2X 2(1). Now sum over i and j.

By the additivity of X2's, the left-hand side will be distributed as O'2x2(nn), and the two terms on the right-hand side as O'2x2(l't(n - 1» and O'2x2(rt). If we define

(2.6) it will have expected value 0'2 and be independent of the other term on the right-hand side of (2.5). Now consider this other term. Ifwe square the identity (1.21) and sum over i,j and 'I' (summing over 'I' amounts to multiplication by n), we get r

11

t

Li L; (xii.

- ~ii)2

r

t

= 11 Li L; (Xii.

r

- Xi .. - X. i .

+ X... -

( 0 )2

t

+ Ilt Li (Xi .. -

X... - 17i)2

+ nl' Li (X. i . -

X... - 'i)2

+ I1tl'(X... - W. (2.7)

The left-hand side is distributed as O'2X2(l't). The degrees of freedom for the first term on the right-hand side follow from the fact that the random variables Xii. - Xi .. - x. i . + X... are subject to the conditions r .. k'" (xtJ.

X·t.. ~ xi ..

+ X.0. ) = 0

for each j,

(2.8)

Xi .. - iX. i .

+ X...) = 0

for each i,

(2.9)

i

t

Li (Xii . -

and, similar to the argument for the analogous Oii' these form only /' + t - 1 independent relations. The degrees of freedom for this term are therefore (2.10) 1'1 - (I' + t - 1) = (I' - 1)(1 - 1). For the second and third terms we have the conditions r

'" "'" i

t

(xi.. -

X ) -....

0 -- "'" '" (x•. i.

-

X... )

(2.11)

j

and so the degrees of freedom are I' - 1 and 1 - 1, respectively. The last term involving the single random variable X... has 1 degree offreedom. Hence, by Cochran's theorem, these sums of squares will be distributed as O'2X2 and be independent.

SBCT.

14.2

473

THE ANALYSIS OF VARIANCE

If we define

S;2 as r

11

t

L L:i (xii. -

Xi .. - x. i .

i

+ X... -

eii)2

(2.12)

(I' - 1)(t - 1) then

12

S2 ,...., (]

2l«I' - 1)(t - 1)) , (I' - 1)(t - 1)

(2.13)

and S;2 will have expected value (]2 and be independent of s~. Hence the ratio S;2/S~ has the F distribution with degrees of freedom (r - 1)(t - 1), rt(n - 1). Now define s~ as r

S~=

t

n L L__________________ (Xii. - Xi .. - X. i. + X..Y __ __ t~'~i

(2.14)

(I' - 1)(t - 1)

To find its expected value, consider the expected value of the numerator of S;2;

E[n

**

(xii. - Xi .. - x. i.

= E -

+

X... - eii

)2]

[n ii ±(Xii. - Xii .. - X. i. + x.yJ + i ±e:i i 211

**

11

E [eii(Xii . - Xi .. - X. i .

= E[(I' - 1)(t -

l)s~J

-

n

**e:

+

i

x.J] (2.15)

i.

The left-hand ,side equals (r - 1)(t - 1)(]2, whence it follows that

(2.16) If the null hypothesis e ii S2

~,...., S2

= 0 for all i, j

is true, then s~

F«I' - 1)(t - 1), I't(n - 1)).

= S;2,

and so (2.17)

1

If the null hypothesis is false, so that some eii =;t:. 0, then E[s~] > (]2 and the ratio s~/s~ will not have the F distribution but instead a distribution displaced upwards in the direction of larger values of s~/s~. The critical region is therefore large values of s~/s~. The other two mean squares in Table 14.3 can be treated similarly.

474

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

If the null hypothesis ()iJ = 0 for all i, j is rejected, then the true cel\ means are not given by an additive model, and we might as well deal with the individual cell means. On the other hand, if the null hypothesis is acceptable, then we may assume that the ()iJ are zero, and proceed to test the row and column effects. If the 17i = 0 for all i, then s! is an estimate of 0'2 independent of s~, and the ratio s!M will be distributed as F«I' - 1), I't(n - 1»). We may wish to construct confidence limits for ~i. - ~i'., the difference between two row means. Since, using (1.5),

~i.

-

~i" = (~i.

~)

-

-

(~i"

~) =

-

1]i -

1]i"

(2.18)

confidence limits for ~i. - ~i" will be identical with confidence limits for 1];" Averaging the model (1.17) over j and v gives

1]i -

Xi ..

since ~. Hence

= ~ + 1]; + ~. + 0i. + Zi .. = ~ + 1]i + Zu t

= (l/t) 1

'f =

(2.19)

t

0 by (1.7) and

0i.

= (l/t) 1 ()If = 0

by (1.15).

f

Xi .. -

Xi' ..

=

(17i -

1]i')

+ (Zi .. -

(2.20)

Zi'.J,

with expectation (2.21) and variance V[Xi .. -

x i •..1

=

V[Zi..1

+ V[Zi'..1 = 20'2 .

(2.22)

tn

We can therefore derive confidence limits for

1]i -

1k from

(2.23) Usually, there is little or no point in constructing confidence limits for ~i. - ~i'. unless ~ii - ~i'i is constant over j, i.e., unless the difference between the two rows is the same for all columns. From (1.10), (2.24) Thus (2.23) will also give confidence limits for ~if - ~i'j if we can assume that the interaction constants ()if are zero. Ifwe cannot make this assumption, we can obtain confidence limits for ~i:l - ~i':I as follows. Since Xii. -

Xi':/.

= 17i -

1k

+ ()ii -

()i':I

+ zii. -

zi':/.

(2.25)

has expectation E[xi:I. -

xi':/.]

= 1Ji -

1]i'

+ ()if -

()i':I

=

~i:l

-

~i':I'

(2.26)

SECT.

14.3

FORMS FOR TWO-WAY ANALYSIS OF VARIANCE

475

from (2.24), and variance V[Xii .

-

Xn] = V[z;;J

+ V[zi'j.] =

20'2 ,

(2.27)

11

/

confidence limits for

(~iJ

- ~i'i) can be obtained from

(X;;. - xi'i,) - (~;; - ~i'i) ,...,., t(rt(1l _

.J2s~/1l

1».

(2.28)

Comparison of the denominator of (2.28) with the denominator of (2.23) shows the advantage gained if the assumption of zero interaction is permissible. 14.3. Computing Forms for Two-Way Analysis of Variance Table 14.4 gives the per cent reduction in blood sugar a certain time after injection of insulin into rabbits. A group of 24 rabbits was divided at random into six groups of four rabbits each, and each rabbit received an injection of insulin. Two factors were involved, the dose at three levels and the preparation of insulin, A and B, at two levels. Table 14.4 Dose Preparation

2.29

3.63

A

17 21 49 54

64 48 34 63

62

141

209

286

33 37 40 16

41 64 34 64

56 62 57

126

203

247

576

267

412

533

1212

B

72

61 91 636

72

476

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

While the sums of squares in Table 14.3 can be calculated in the forms given there, it is usually more satisfactory to use identities that involve totals rather than means. We need the following sums of squares: r

t

n

i

;

v

LLI X~iv = 172 + 212 + ... + 722 = 69,358, ! n

±±(i XiiV)2 = i

i

±(± i XiiV)2 nr l

nt

i

i

i

= 267 2

+ ... + 533 2 =

v

(3.2)

65,640.25,

(3.3)

4 X2

v

±(± i XiiV)2

+ 2472 = 65,863.00,

4

v

~

i

1412 + ...

(3.1)

+

= 636 2 576 2 = 61,356.00, 4 X 3

(3.4)

1 (r t n )2 -nrt I Ii Iv Xiiv = 4 X12122 = 61,206.00. 2 X 3

(3.5)

i

By manipulations similar to those of (10.2.31) we get for the total sum of squares

(3.6)

= 69,358.00 -

= 8152.00.

61,206.00

The within-cells of squares is

(3.7)

= 69,358.00 -

65,863.00

= 3495.00.

The rows sums of squares is nt

-2 lr(tn Ir_ (Xi .. - X.J = - I I I tn i i i

Xijv

)2

-

v

l(rtn I LI

-

rtn

)2

(3.8)

(rI It In )2

(3.9)

i

i

Xiiv

v

= 61,356.00 - 61,206.00 = 150.00

Similarly, the columns sum of squares is I1r

t (X.i. - X..l = -1 It(rI In )2 rl1

Ii

Xiiv

i

i

v

= 65,640.25 -

1 -

-

rtl1

61,206.00

Xiiv

i

i

v

= 4434.25

14.3

SECT.

477

FORMS FOR TWO-WAY ANALYSIS OF VARIANCE

A convenient computing form for the interaction sum of squares is (see ~xercise

14.5)

r t

n k.L., ~ '" (x.. ",. i

X.t.. - X..1.

;

=1 n

i i(i i

i

+X

. ..

)2

J... (i J'tn i

XiiV)2 -

v

r

- nt 2 (Xi .. - X.. Y i

-

ii i

XiiV)2

v

t

nJ'

2i (X. i . - x..Y

(3.10)

= 65,863.00 - 61,206.00 - 150.00 - 4434.25 = 72.75.

It will sometimes be worth while to calculate directly the quantities (1.19), because they are estimates of the ()ii and a pattern in them may tell us something about the nature of the interaction. The cell means xii. are given in Table 14.5a. For i = 1, j = 1, for example, we have xu. = 141/4 = 35.25. Also Xl .. = 636/4 x 3 = 53.00, x.I. = 267/4 x 2 = 33.75, and X... = 1212/4 x 2 x 3 = 50.50, so Xu. - Xl .. -

x.I. -

X...

=

35.25 - 53.00 - 33.75

+ 50.50 =

-0.625. (3.11)

The quantities Xii. - Xi .. - x. i . + X... are tabulated in Table 14.5b. They have the property of summing to zero in each row and in each column [see (2.8) and (2.9)]. Their sum of squares, multiplied by n, is of course the interaction sum of squares, and this provides an alternative to the identity (3.10): r

t

i

i

n 22( x·Ii. - x·I.. - x .i.

+ -'")2 X

= 4[(-0.625)2

+ ... + (-2.375)2] =

72.75.

(3.12)

Table 14.5 (a) Xii.

(b) Xii. - Xi .. -

j

1 2

35.25 31.50

x.i. +

X...

j

2

3

52.25 50.75

71.50 61.75

-0.625 +0.625

2

3

-1.750 +1.750

+2.375 -2.375

These results are assembled in Table 14.6. It is apparent that the null hypothesis that the interaction is zero is acceptable, and hence there is unlikely to be anything of interest in Table 14.5b. The null hypothesis

478

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

that the difference between preparation is zero is acceptable since we have F(I, 18) < I, but the dose effect, with a variance ratio 2217.125/194.167 :::: 11.4 being distributed under the null hypothesis as F(2, 18), is highly significant.

Source of variance Rows = preparations

Table 14.6 Sums of Degrees of freedom squares

Mean squares

150.00

150.000

E[M. S.]

12

r

O'2+T~11~ t

Columns = doses

4434.25

2

2217.125

8 t

0'2

+2~'~ 1

Interaction

72.75

2

36.375

Within cells

3495.00

18

194.167

Total

8152.00

23

4 r t O'2 +-!!O2 2 j i /1 0'2

14.4. Two-Way Analysis of Variance: Model II In model II two-way analysis of variance we assume that both rows and columns are random effects, sampled from infinite populations. For example, in a large factory with very many identical machines, machines could correspond to rows, columns to a random sample of batches of raw material, and the replicates in the cells are several items made from each batch on each machine. The model is (4.1)

where gj, e;, Yii and Zjiv are independently sampled from normal populations with zero means and variances V'~, V'~, (02 and (j2, respectively. The usual objective of the analysis is to estimate and construct confidence limits for the parameters of the model, namely, the grand mean and the four components of variance. The analysis of variance involves the same equation (2.2) as in the model I analysis. The computing forms, the degrees of freedom, and the mean squares are all the same as in Table 14.3, and the only differences between the two models lie in the expectations and distributions of the mean squares. The mean square s~ continues to have the same expected value a2 and the same distribution as before.

SECT.

14.4

TWO-WAY ANALYSIS OF VARIANCE:

479

MODEL II

To find the expected value of the interaction mean square s~, we proceed as follows. We form various averages of the model (4.1), /

+ gi + e} + Yu + zo" Xi" = ~ + gi + e, + fh + Zi,,' = ~ + g, + e} + y,} + X,,, = ~ + g, + e, + y" + Z,,'; Xi}, = ~

X,},

(4.2) (4.3) (4.4)

Z,}"

(4.5)

so we have the identity

Xii. - Xi" - X,}, + X"' = (Yii - fh - y,)

+ Y.,) + (zu.

+ zoo,),

- Zi" - z,i,

(4.6)

Now

!E[CI' - 1)(t -

n

= E[

*t

1)s~] = E[ii(x ii , • ,

Xi .. - x,i.

+ Y.fJ + E

(Yii - Yi. - Y.i

[* t

+ x,y]

(Zii, - Zi" - Z.J,

+ z,J J 2

(4.7) the expectation of the cross product being zero since our model assumes independence of the y's and z's. Consider the two parts of (4.7) separately. It is straightforward to show that

+ z..

E[**CZiJ' - Zi .. - z,J,

yJ

= E[~ ~(ztJ... £., £., i

Z.••

;

)2J - tE[~(Z' £., i

t..

- Z. ..

- I'Elt(Z,J, Now (I't -

r

t

i

J

1)-112 (Zi),

-

z"Y

)2J

Z,J 2] .

(4.8)

is the ordinary sample estimate of the

variance of the ziJ,' which is (12/11, so

E[i ±(ziJ, - z..fJ = I't -n J

Analogously, (I' -

1 (12.

(4,9)

i

i

1)-11 (Zi"

-

z"Y is

the ordinary sample estimate of

i

the variance of the Zi,,' which is (12/111, so

E[~(Z" - Z )2J = I' -til 1 (12. f....,

(4,10)

480

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

Likewise, E

[!

t (- )2J Z ·-Z .1....

(- 1 2 .

E[

ft( Z.. r

t

I).

14

(4.11)

=--(1

I'll

j

Thus

CHAP.

J

Z. - Z . + Z.J2 d.

l..

I't - 1 I' - 1 = (- - t- -

t* tn

11

1) (12 = -1 (I'

t -

I' - I'll

(Yo - fh -

The consideration of E[

1)(t - 1)(12. (4.12)

-

11

ii. + ii.,)2J, j

the other part of

(4.7), will follow the analogous course, the only difference being that terms of the type V[Yij] = w 2, V[fi,j] = w 2jr, etc., involve w 2 and not (12, and omit the factor Ijl1. We can thus assert

E[**(Yii - iii. - ii.j + ii./J =

w 2(1' - 1)(t - 1).

(4.13)

+ (I' -

(4.14)

Substituting (4.12) and (4.13) in (4.7) gives

E[*

*

(Xii. - Xi .. - x. j. + x..

=!

n

/J

(r - 1)«( - 1)(12

1)(t - 1)w 2,

whence, multiplying by Ilj(r - 1)(t - 1), E[s~] = (12

+ nw 2•

(4.15)

Now consider E[s!]. We have from (4.3) and (4.5) that

Xi .. - X... = (gi so

E[*(X; .. - x..

/J =

g,)

E[*(gi -

+ Wi. - ii.,) + (Zi .. - z.. ,),

(4.16)

g/J

+ E[*(ii;. - ii.fJ + E[*(Zi .. - z.. f}

(4.17)

the expectations of the cross products being zero since our model assumes independence of the g's, y's and z's. In (4.10) we have evaluated the last term, and the first two are analogously

E[*(gi - gyJ E[

f Uk r

J

= (r -

Y.Y = (r -

1)1p~, 2

t .

1) w

(4.18)

(4.19)

SECT.

14.5

481

THE INTERPRETATION OF A MODEL II ANALYSIS

substituting in (4.17) and multiplying by tn/(r - 1) gives (4.20) Similarly, (4.21) These results are assembled in Table 14.7.

Table 14.7 Source of variance

Degrees of freedom

Sums of squares

Mean squares

EIM.S.]

r

RoWS

1/1

~ (Xi .. - X.. ,)2

I' -

1

s~

(12

+ IlW 2 + tl/lp~

t -

1

S23

(12

+ IlW· + rmpi

s:

(12

+ IlW 2

S21

(12

I

t

Columns

Ill'

~ (x.l. - x .. .)2 I

t

r

Interaction

II ~ ~ (xii: - XI .. i

I

t

r

Within cells

I

t

r

I

1)

+ x .. .>" rt(1l -

1)

v

1/

~ ~ :E (xilv - x .. .>" I

1)(t -

1/

~ ~ ~ ('''Uv - x u J2 i

Total

- x.l.

(I' -

I'tll-

1

v

14.5. The Interpretation of a Model II Analysis The tests of significance for the null hypotheses co 2 = 0, 1J'~ = 0, 1J'~ = 0 are obvious from the column of expected mean squares in Table 14.7. Whereas, in the model I analysis (Table 14.3), the main effects for rows and columns were tested against the within-cells mean square, here they are tested against the interaction mean square. The parameters of the model are estimated in the obvious ways: ~

-2 s~ - s~ -2 s~ - s~ co = - - , 1J'1=--- ,

= x... , 62 = s~,

til

II

2

s~ - s~

1J'2= - - - . I'll

(5.1) Exact confidence limits for ()'2 and for C0 2/()'2, and approximate confidence limits for w2 , 1J'~ and 1J'~ can be obtained as in Section to.5. For confidence limits for ~, from (4.5) E[x .. .1 = ~, and _ V[x ...l

1J'~

1J'~

co 2

(12

,.

t

I't

I'tll

= - +- +- +- =

()'2

+ IlW 2 + Ilt1J'~ + llI"IP~ rIll

.

(5.2)

482

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

To obtain a mean square with expected value equal to the numerator in (5.2) we use the linear combination s! + s: - s:, since it has expected value (5.3)

Thus

x... - ~

J(s! + s~ -

s~)lrtn

t"'.J

f(r),

(5.4)

approx

where f' is given by application of (9.8.10). Confidence limits for ~ can be derived in the usual way, but these will be approximate on account of the approximation in (5.4). 14.6. Two-Way Analysis of Variance with Only One Observation per Cell Sometimes we have data in a two-way classification with only one observation per cell. The analysis is similar to that with n observations per cell, but, with n = 1 the model becomes, in the model I case, Xii

= ~ + 17i +

'1 + Oij +

zij

(6.1)

+ gi + ei + Yij + Zij'

(6.2)

and in the model II case Xii

=

~

In the table of analysis of variance there is no item "within cells," and the analysis is as in Table 14.8. The expectations of the mean squares for the two models are given in Table 14.9. In both models there is no test for interaction. In the model II analysis, the tests for the main effects of rows and columns are unchanged. In the model I analysis, the test for the row and column main effects may, if there is appreciable interaction so that

r

t

i

i

L L 0~1 =;t:. 0, be inefficient, since the denominator mean square will

be inflated by the extra component. On the other hand, if either variance ratio is significant, then it may be taken that the corresponding effect is real. 14.7. Nested or Hierarchical Analysis of Variance In our discussion of one-way model II analysis of variance in Section 10.4 we postulated a sampling situation in which sacks of wool were taken from a large consignment, and then samples were taken from each sack at random. If we now suppose that several analyses are performed on each sample, an appropriate model would be (7.1)

SECT.

14.7

483

NESTED OR HIERARCHICAL ANALYSIS OF VARIANCE

Table 14.8 /

Source of variance

Sums of squares

Rows

1

r

I

(XI' -

1

i

X/j

i

Mean squares

r-l

s'«

1

s'3

x.. )'

C J rllCI I

= -1 Ir I

Degrees of freedom

- -

t X/j )'

i

i

t

rI

Columns

(X'i -

x.. )'

t -

i

= -1rIit C Ii Xli Remainder

r

t

i

i

II

Total

J lCI I J -

~

t Xii

It

i

i

(by difference)

(,. - 1)(1 - 1)

x.. )'

,.1 - 1

(X/j -

rt = IIx~i i

i

lC

~

t IIx/j

It

i

i

,

s'

J

where ~ is the grand mean, Vi corresponds to the sack effect, Yo to the sample within sack effect, and zo. to analyses within samples. The random variables Vi' Yo, and zo. are independently normally distributed with zero means and variances 1jJ2, w 2 , and 0"2. Suppose there are r sacks, t samples per sack, and 11 analyses per sample. Squaring and summing the identity Xii. -

x... = (Xi .. -

+ (Xii.

X..,)

-

+ (Xii. -

Xi.,)

(7.2)

Xii)

Table 14.9 Source of variance

E[M. S.]

Mean squares

Model I t

Rows

112

+ "-1~ - - '" 1]~ • ,.

Columns

S2

3

112

1-

112

112

+ w + tlf'~

112

+w

2

t

+--I~~ 1

i

1

Remainder

Model II

+ (,. _ 1)(1 _

., r

t

1) ~ ~

(j~i

2

484

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

gives r

t

r

11

t

r

:L:L:L (Xii. - X.J 2 = nt :L (xi .. - X.. Y + II :L:L (Xii. i • i i i

Xi.l

i

r

t

11

i

j

y

+ :L:L:L (Xii' -

Xii?'

(7.3)

These sums of squares are entered in Table 14.10, along with their degrees of freedom. We now determine the expected value of these mean squares, as given in the last column. Table 14.10 Source of variance

Sums of squares

Degrees of freedom

Mean squares

1

S23

(12

+ IIW 2 + II/'P'

s~

(1'

+ IIW'

S21

(12

EIM.S.]

r

Between sacks

111 l: (Xi •• -

r -

X •• .)2

I

t

r

Between samples within sacks

Ill: l: (xu. i

r

Between analyses within samples

l: l: l: (xii" -

Total

l: l: l:,. (,vii" - x.Y

i

i

"

r

t

n

i

r(t -

Xi..>"

1)

i t n

rt(lI -

XU.)2

1)

rtll- 1

i

Considering s~, averaging the model (7.1) over xii.

Y

gives

= ~ + Vi + Yu + Zii.;

(7.4)

so

(7.5) Squaring and summing over

Y

gives

11

:L (Xii. -

11

Xii)2

= :L (zu. -

ZiiY

(7.6)

Divided by 11 - 1, this is a sample estimate of the variance of the zUv' namely, a 2 • Pooling the sums of squares and degrees of freedom through summation over i andj leaves this unchanged; so we obtain

E[s~] = E

[±i i

(Xii' -

• , \' I't(n -

Xii)2]

= a2 •

1)

(7.7)

Now consider s~: Averaging (7.4) over j gives Xi ..

= ~ + Vi + fh + Zi .. ;

X.t..

= (y .. -

so X', j .

-

l,)

y-.) t..

+ (z'j

t •

- z·t •. ).

(7.8)

(7.9)

SECT.

14.7 NESTED OR HIERARCHICAL ANALYSIS OF VARIANCE

485

Squaring and summing over j and then taking expectations gives

E[~(Xii. - xi.l]

[*

E

=

(Y/i -

[*

Yil] + E (Zii. - Zi.lJ. (7.10)

the expectation of the cross product vanishing by reason of the indet

pendence of the Yii and

.2 (Yo - Yi,)2/(t -

It is apparent that

Ziiv'

1)

i

is a sample estimate of the variance of the Yii' namely, w 2 • Similarly t

.2 (zii. ;

the

ziiv

-

Zi.y/(t -

1) is a sample estimate of the variance of the Zii .. Since

have variance

E

0'2,

the

Zii. have variance O'2/n, and so

[±(xii. - xi.ll = j

t- 1

2

w2

+!!.....

(7.11)

11

Pooling the sums of squares and degrees of freedom by summation over i, and multiplying by n, gives

E[s~]

Now consider

s::

= E[

11

±±(Xii. - x.y] = i

i

0'2

l'(t - 1)

+ I1W2•

(7.12)

Averaging (7.8) over i gives X...

= ~ + D. + Y.. + Z... ;

(7.13)

so, subtracting this from (7.8) gives Xi .. - X...

=

(Vi - V,)

+ Wi.

-

Y.,) + (Zi .. - z.. ,).

(7.14)

Squaring and summing over i and then taking expectations gives

E[*(Xi .. - x.. l] = E[*(Vi -

D.?] + E[*Wi. - YY] + E[*(Zi.. - z..YJ.

(7.15)

the expectations of the cross products vanishing since our model assumes r

independence of the v's, y's, and z's. It is apparent that

.2 (Zi .. - z.. Y/ i

(,. - I) is a sample estimate of the variance of the

z... , namely O'2/tn.

486

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

Table 14.11 x' = quantity

Batch

2

3

4

5

6

7

8

9

10

11

12

13

14

Sample 1 (1) 2 (1) 3 (1) 4 (1) 1 (2) 2 (2) 3 (2) 4 (2) 1 (3) 2 (3) 3 (3) 4 (3) 1 (4) 2 (4) 3 (4) 4 (4) 1 (5) 2 (5) 3 (5) 4 (5) 1 (6) 2 (6) 3 (6) 4 (6) 1 (7) 2 (7) 3 (7) 4 (7) 1 (8) 2 (8) 3 (8) 4 (8) 1 (9) 2 (9) 3 (9) 4 (9) 1(10) 2(10) 3(10) 4(10) 1(11) 2(11) 3(11) 4(11) 1(12) 2(12) 3(12) 4(12) 1(13) 2(13) 3(13) 4(13) 1(14) 2(14) 3(14) 4(14)

Subsamples 76 69 72 75 110 119 120 111 130 143 141 129 62 50 71 66 62 48 80 87 91 87 78 87 101 89 78 76 136 108 128 96 140 92 107 84 81 86 103 85 108 102 102 109 106 100 99 102 93 85 78 89 116 104 118 112

85 82 78 84 109 106 121 119 140 121 147 140 67 61 74 67 64 50 86 91 97 90 74 83 97 96 96 87 123 131 119 82 136 80 114 113 99 83 94 87 98 102 103 III

107 104 98 91 89 89 80 87 117 116 119 109

Sample totals 161 151 150 159 219 225 241 230 270 264 288 269 129 111 145 133 126 98 166 178 188 177 152 170 198 185 174 163 259 239 247 178 276 172 221 197 180 169 197 172 206 204 205 220 213 204 197 193 182 174 158 176 233 220 237 221

x = 100(log x' - 1)

Batch totals

621

915

1091

518

568

687

720

923

866

718

835

807

690

911

Subsamples 88 84 86 88 104 108 108 105 111 116 115 111 79 70 85 82 79 68 90 94 96 94 89 94 100 95 89 88 113 103 111 98 115 96 103 92

91 93 101 93 103 101 101 104 103 100 100 101 97 93 89 95 106 102 107 105

93 91 89 92 104 103 108 108 115 108 117 115 83 79 87 83 81 70 93 96 99 95 87 92 99 98 98 94 109 112 108 91 113 90 106 105 100 92 97 94 99 101 101 105 103 102 99 96 95 95 90 94 107 106 108 104

Sample totals 181 175 175 180 208 211 216 213 226 224 232 226 162 149 172 165 160 138 183 190 195 189 176 186 199 193 187 182 222 215 219 189 228 186 209 197 191 185 198 187 202 202 202 209 206 202 199 197 192 188 179 189 213 208 215 209

Batch totals

711

848

908

648

671

746

761

845

820

761

815

804

748

845

SECT.

14.7 NESTED OR HIERARCHICAL ANALYSIS OF VARIANCE

487

r

Likewise I (fk - 'ii.Y/(r - 1) is a sample estimate of the variance of the r

i

'iii., namely

I

and

0)2/t,

(Vi - v.)2/(r - 1) is a sample estimate of the

i

variance of the v's, namely we get E[s~] = E

[

1p2.

Thus multiplying (7.15) by tn/(r - 1),

I OiL - x.. f ]

lit r

=

i

1'-1

(i

+ 110)2 + Ilt1p2.

(7.16)

Equations (7.7), (7.12), and (7.16) are entered in the last column of Table 14.10. The data of Table 14.11 give the results of taking four samples from each of 14 batches ofa slurry. The quantity of matter in suspension in each sample x' was determined in duplicate by dividing each sample into two subsamples. The model for the analysis will be (7.1). The range 11';; between the duplicates x;;v and x;;v' can be used as an estimate of the standard deviation (J, since c1;; = 1V;;/d2 ; see (6.6.2). If we tabulate the ranges according as the batch total is in the intervals (500 - 599), (600 - 699), etc., we get mean ranges of 4.25, 5.50, 7.75, 6.83,8.25 and 12.25. It appears that the standard deviation (J is increasing approximately linearly with the mean. The model (7.1) makes the assumption that the z;;v are distributed normally with a constant variance (J2. However, we saw in Section 3.3 that, when the variance is proportional to the square of the mean, the logarithm of the variable will have a constant variance. The analysis will therefore be performed on x = 100[(log x') - 1], this transformation producing numbers easy to handle. Computing forms for the sums of squares in Table 14.10 are easily obtained. Defining

A

B

r

t

"

i

i

v

=I I I

xf;v

= -1 Ir It ("I IIi;

v

Xiiv

c = -1 Ir (tI In til

D

i

= ~(i l'tll

i

)2

±i i

)2

X iiv

v

;

= 88 2 +

v

=

93 2 +

181 2 +

... +

... + 2

104 2 = 1,078,281,

209 2

=

2 = 7112 + ... + 845 =

XiiV)2

4 x 2

=

(10,931)2 14 x 4 X 2

1,077,756.5,

1,075,573.375,

= 1,066,846.080,

(7.17)

(7.18)

(7.19)

(7.20)

488

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

straightforward manipulation yields

..

nt 1 (Xi .. - X.. /

=c-

=

8727.295,

(7.21)

n 11 (Xii. - Xi.Y = B - C = 2183.125,

(7.22)

i

l'

..

t

i

;

t

n

111 (Xii. i ; v .. t

Xii)2

=

D

A - B

=

524.500,

(7.23)

= 11,434.920.

(7.24)

n

111 (Xii. - X.Y = A i ; •

D

The analysis of variance is in Table 14.12. The test of the null hypothesis 0)2 = 0 is given by the variance ratio 51.979/9.366 = 5.55: FO.9996(30, 50) = 2.86; so clearly this null hypothesis is rejected. The test of the null hvpothesis 1p2 = 0 is given by the variance ratio 671.330/51.979 = 12.9: Fo.9995(1O,40) = 4.21, so this null hypothesis is rejected. The point

Source of variance Between batches Between samples within batches Between subsamples within samples Total

Table 14.12 Sums of Degrees of squares freedom

Mean squares

8,727.295

13

671.330

0'2

+ 2002 + 8'1'2

2,183.125

42

51.979

0'2

+ 200 2

524.500

56

9.366

0'2

11,434.920

111

E[M. S.]

estimates of the components of variance are

0-2 = 9.366,

A2 _

0)

-

51.979-9.366 2

= 21 3 1/12 = .,

671.330-51.979 = 77.4. 4 X 2 (7.25)

It is apparent that there is considerable variation between samples within a batch. The largest component of variance, however, is between batches. For estimating the mean of a batch, the estimated variance of one subsample from one sample would be 0-2 + 2 = 9.4 + 21.3 = 30.7. The estimated variance of two subs am pies from one sample would be 0-2/2 2 = 9.4/2 + 21.3 = 26.0; so there is very little gain in the precision of the estimation of a batch mean through analyzing two subsamples instead of one. It might, nevertheless, be justifiable to analyze two subsamples as a check against gross errors and mistakes.

w

w

+

SECT.

14.8

THE TWO-WAY CROSSED FINITE POPULATION MODEL

489

Confidence limits for the overall mean of the process, averaged over batches, can be constructed as follows. The variance of (7.13) is V[X .. .1

=

V[v.1

+

V[y..1

+

V[Z .. .1

2 2 2 2+ 2+ = '!L + ~ + !!...-. = /leo (1

I'

rt

rt/l

2

IIt1p,

rtn

(7.26) and

V[x ] = ...

s:

=

rtll

671.330 14 x 4 x 2

= 5.994.

(7.27)

Since

x... -

¢

' " t(r -

1),

(7.28)

v'V[x...1 we find 95 per cent confidence limits for ¢ to be (92.3, 102.9). These are on our transformed scale. Transforming back, the 95 per cent confidence limits for the process average on the original scale are approximately (83.3, 106.9). 14.8. The Two-Way Crossed Finite Population Model The model II analysis of Section 14.4 postulated random sampling from infinite populations. Random sampling from finite populations was first considered by Tukey [1], Cornfield and Tukey [2], and Bennett and Franklin [3]. This finite population model is of interest in itself, for sometimes the assumption that a population, e.g., of machines, is infinite is too gross. This model is also of interest because, if we let the population sizes go to infinity, with the additional assumption of normality, then we get model II, and, if we decrease the population size until it equals the sample size, so that the sample contains the entire population, then we get model!. Our main motivation for considering the finite population model, however, is that it will give us a procedure for handling the mixed model, in which one category, say rows, is model I and the other, columns, is model II. Such mixed models occur frequently in practice. The arguments to follow only involve the ideas of expectation and of combinations but they are somewhat lengthy and involved, and some readers may be content to read only through (8.6) and then proceed to Section 14.9. For a two-way crossed classification with replication in the cells the model is Xii. = ¢ + 1]i + ~i + Oij + Zii.' (8.1) similar to (1.17), with i = 1, ... , r; j = 1, ... , t; 'JI = 1, ... , n. However, here the 1]i and ~i' referring to rows and columns, respectively,

490

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

14 are random samples from populations of sizes Rand T and satisfy the conditions CHAP.

(8.2)

Selecting a particular i and a particular j determines the row and column and hence the cell that forms their intersection, and with this cell is associated the interaction constant OiJ. The interaction constants satisfy the conditions R

!

i

0ii

T

=0

!i

for each j,

Oij = 0 for each i.

(8.3)

We make the following definitions: 2

(1" = R

2

1

~

2

1

~

,2

_ 1 f 'Y/i'

(1'=T_17

(1: = (R

(8.4) (8.5)

i'

1 _ l)(T _ 1)

t t O:i' R T

(8.6)

These definitions are not consistent with (7.4.2) which would give, e.g., R

(1: = (I/R) ! 17;, but they are more convenient in the present instance. i

The conventional partitioning of the total sum of squares is identical with that of Table 14.3. We will now evaluate the expectations of the corresponding mean squares under this different model. We will first need the expected values of the squares of certain sums of the Zii' Oji' ~i' and '7i' The Zjiv have zero expectation and are independent and hence have zero covariances, and so Therefore (8.7)

Also

But the variance of the sum of variance (12 is 11(12, and so

11

independent observations each with r t

= ! ! /1(12 = i

i

I'tl1(12.

(8.8)

SECT.

14.8

491

THE TWO-WAY CROSSED FINITE POPULATION MODEL

Similarly,

E[*(~~ZiJ.n = E[*(*~Zii.n = E[(**~zii.)l = rtna (8.9) We next find E[ 1]~l The total number of ways a sample of r can 2

*

be taken from R is

E

(~), and these are all assumed equally likely.

•

Thus

[* 1]~J = (~rl[(1]~ + ... + 1]~) + ... + (1]~-r+1 + ... + 1]~»). (~

Any particular 1]i will occur in

(8.10)

=:)

samples and hence appear in the

square brackets this number of times as 1]:. Therefore, using (8.4),

E[

t r

We next find

(R)-l(Rr _- 11) t

2J =,.

1]i

E[ (*

1]i)].

R

1)

2= r (1 - Ii a".2

1]i

(8.11)

By similar arguments,

(8.12) Now (

R

)2

=

R

+

RR

(8.13)

21]i 21]: 221]i1]; i i i ; i~;

and by (8.2) the left hand is zero. Hence, substituting in (8.12),

t r

(R)-l[(Rr -- 11) - (R,. _- 22)JRt

)~ 'J = r

E [ ( 1]£

2

1]i

= r( 1 -

r) a". R 2

(8.14) Expressions for {, similar to (8.11) and (8.14) are obtained by the same arguments. We now find

E[ (* ~ OiiJJ.

Specifying a particular row, say the pth,

and a particular column, say the qth, determines a particular cell, and associated with this cell is the constant Opq. The total number of ways we can choose,. rows out of a possible R, and t columns out of a possible T,

492 is

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

(~) (~).

CHAP.

14

However, if we specify that the sample of I' rows is to include

the pth row, and the sample of t columns is to include the qth column, the number of ways in which we can select the remaining I' - 1 rows and

. (R - 1)1 (Tt _- 1)l '

I

t - 1 co umns possible ()ii is

t t

r

~

(

IS

\2

()ij J '

I' _

h

.

Thus, III . t e summatIon of all

the coefficient of the direct square of each element

(R- 11) (Tt -- 11) . I' -

r

t

i

:I

The cross products in ( L L

()ii

)2 are of three types:

1. Those with both ()'s in the same row but different columns,

2()'J)q()'J)u,

q =F u. 2. Those with both ()'s in the same column but different rows, 2()'J)iw p =F s. 3. Those with the two ()'s differing both in row and in column number, 2()'J)q().lt'

p =F s, q =F u.

In how many of the possible

(i ±

()i:l)2

i

does a particular cross product

:I

of the first type, 2()'J)q()'J) II , occur? A sample of the ()i:l will arise from a choice of I' rows and t columns. Suppose that it contains a specified row, say the pth, and two specified columns, say the qth and the uth. Then the remaining I'

-

1 rows can be selected from the R - 1 available in

(~

:= !)

ways, and the remaining t - 2 columns from the T - 2 available in

specifie~ 2t()pi'J); occurs (~ := when we consider all possible (L L()i:I) . (~:= ~)

ways. Thus a

D(~:= D

times

From (8.3) we can writei:l T

o = L ()J') = :I

so T

-()!q = ()'J)q

L ()'J):I =

:I*q

()pi()Pl

T

()pq

+ L ()j,j;

(8.15)

}*q

+ ... + ()P(<<-l) + ()'J)(q+1) + ... + ()'J)T)' (8.16)

We now write down the 2()'J)q()'J)Il' q =F u, for a fixed p: ()Pl«()p2

+ ... + ()pT) + ()P2«()j,l + ()p3 + ... + ()pT) + ... + ... + ()'J)T) + ... + ()pq«()pl + ... + ()p(q-l) + (8.17) + ()pT«()'J)l + ... + ()p(T-l»' ()j,(q+1)

SECT.

14.8

493

THE TWO-WAY CROSSED FINITE POPULATION MODEL

=

From (8.16) each term in (8.17) is equal to - O!i' j

1, ... , T, and so the

T

sum of all terms in (8.17) is sible

20 11

ifJl<'

Therefore the sum of all pos-

i

q¢

above that each

t

L O!r

u,

20 11q0 l1U

occurs

20 11q 0 l1U

i

1

over all possible

(R- l)(Ttt t Oij' 2)R T

-

T

1

I' -

But we saw

(R =- 11) (Tt =- 22) times in(rall possible )2

(tr t )2,and so the sum of the r °u

R

L L O~r

summed over all p, is -

tt

2

t

0u

is

(8.18)

2

For cross products of the second type, formed by O's in the same column, the corresponding sum is

- 2)(T1)11 0:1' t-l

_(~/-2

(8.19)

ii

With cross products of the third type, with O's differing in both row and

isu will occur in (~ =- ;) (~=- ;) possible

column number, a specified 20 11

samples, since with two rows fixed the remaining r - 2 can be selected from the R - 2 available in

(~ =- ;) ways, analogously for columns.

The sum of all possible cross products of the third type containing 011'1 is R 011qL

T LOst<

S,,"l1 ",,"'I

=

+ ... + 01('1-1) + 01(q+1) + ... + 01T + 0 21 + ... + O2('1-1) + O2('1+1) + ... + 02T

0 11'1(0 11

+ 0(11-1)1 + ... + 0(1'_1)('1_1) + 0(1'-1)('1+1) + ... + O(1)-I)T + 0(11+1)1 + ... + 0(1)+1)('1-1) + 0(1)+1)('1+1) + ... + 0(1l+1)T + OR! + ... + 0R(q-l) + OR(q+1) + ... + 0RT) = 0 11'1 ( 1,,"'1 f 0li + ... +i,,"qf 0(1'-1)1 +1,,"'1f 0(1'+1)1 + ... +1,,"'1f Om) = Opq( - 01'1 -

••• -

0(1'_1)'1 -

= (1)q(-.f 0;(1) = 0:'1' ',,"I'

0(1'+1)'1 -

••• -

0Rq)

(8.20)

494

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

This was for a specified pq. Therefore the sum of all possible cross products of the third type is (8.21) and the sum of all possible cross' products of the third type in all possible

(**Oii) is

(~ ~

n(;~ ~)~ ~

Hence the sum of all possible

(

0:;.

(8.22)

f t 0ii)2 is given by summing the pure r

t

squared terms and the three types of cross product:

[(~ ~ n(;~

n-

(~~ ;)(;~~)

-(~~ ;)(;~ n (~~ ~)(;~ ~)J~~o~; +

=

(R - 11) l' -

1! ii

l)(R -

(T r)(T - t) 02 • t - 1 (R - l)(T _ 1); ;

Dividing by the number of possible samples

0': from (8.6), we get

(~) (~),

(8.23)

and substituting

When we put t = 1 and /' = 1 separately and then jointly, we get

E[(*OiiJ] = 1'(1- ~)(1- ~)O'=,

(8.25)

E[(*OiiJ] =t(l-i)(l-~)O':. E[O:;] = (1 - i) (1 - ~) 0':.

(8.26) (8.27)

We can now evaluate the expectations of the various mean squares. From the model (8.1), It It I tn Xi .. = ~ + 1]; + - L ,; + - L 0H + - L L zHv, (8.28) t ; t ; til ; v l r It Irt Irtn X... = ~ + - L 1]i + - L ,; + - L L 0;; + - L L L zHv; (8.29) l' i

t

j

rt

i

j

rtll

i

;

v

14.8

SECT.

so

495

THE TWO-WAY CROSSED FINITE POPULATION MODEL

_ _ = ( 1.L

Xi .. - X...

'lJi - r

r

'lJi

t + (1-.L ()ij t :I

)

i

1

t

1.L .L ();;

-

rt

r

t

i

:I 1

n

+ ( -.L .L zi:lv tn :I

) t

r

n

.L.L.L zijv rtn i :I

-

v

)

(8.30)

•

v

When we square (8.30) and sum over i, the expectations of the sums of cross products are zero. For example, E[

1 r ( -.L 'lJi ti

1 r ) 'lJi ri

.L

±1

= E [!t i

{'lJi

:I

1 r t )] .L.L ()i:l ri:l

.L:I ()i:1 -

( t

()o -

-

!r 'lJi

±1 i:l

(± (1 ()

!r i

()i:l -

'lJi)

:I

:Ii)

+ ~(*'lJi)

=

=

!t E

(* ~ ()i:I)}] [± 1 ! (± (± 1 -! (± (± 1 'lJi ()i:1 i:l

ri

'lJi)

i:l

()i:l)

ri

!t E[±i :'lJi l ()o] -.!.rt E[(±'lJi) (±i:l

1

()i:l)

(* ** i:l

'lJi)(

()o) ]

()i:l)]'

(8.31)

t

r

For the first term, the products

+;

1

i

'lJi)

.L 'lJi .L ()ij

can be formed by picking

:I this can be done in rows out of R, and t columns outi of T; ways. Thus l'

(R) (T)t I'

()i:I] = [(~)(~) r1[171«()n + ... + ()u) + ... + 'lJl«()l(T-t+ll + ... + ()IT) + 'lJ2«()21 + ... + ()2t) + ... + 'lJ2«()2(T-t+ll + ... + ()2T) + .. . + 'lJR«()Rl + ... + ()Rt) + ... + 'lJR«()R(T-t+1l + ... + ()RT)]' (8.32)

E[*'lJi*

In any row in the square brackets, the

()j:l

have a fixed i, say p, and in this

row any particular value of j, say q, occurs E[

t1 r

t

]

};t()ji

= ~l(R)(T)]-l[ r t 171

(T-l) t-l

R

T

(~=

Dtimes.

Thus

(T-l)T (T-l)T ] t-l. t()lj+"'+'lJR t-l

= (~)(;) t~'t 9;; = 0,

t()Ri

(8.33)

496

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

using (8.3). For the second term in (8.31) we have

* r

E[ (*17)(* Oli)] i

= [(~) (~) 1 [(1h + ... + 17r){(On + '" + 0lt) + ... + (Orl + ... + 0rt) + ... + (Ol(T-t+1) + ... + 0IT) + ... + (OrCT-t+1) + ... + 0rT)} + .. . + (1JR-r+1 + ... + 17R){(OCR-r+1)1 + ... + 0CR-r+1)t) + ... + (OR! + ... + 0Rt) + ... + (OCR-r+1)(T-t+1) + ... + 0CR-r+1)T) + (ORCT-t+1) + ... + 0RT)}]

~ 1Ji(~ Oli)

= (constant) X

= 0,

(8.34)

using (8.3). Hence (8.31) is zero, and so from (8.30) we have

E[l1t

*

=

x.J 2]

(Xi .. -

I1tE[i (17i - ! i 1JiY J] +!!.t E[i (ii Oij - ! i I'i

i

+ .!.. I1t E

[i (i i i

i

i Oii\2JJ ! iii \2J . J i

I'i

i

v

Ziiv -

Ziiv

I' i

i

(8.35)

v

The first term is

=

t. /I

I' • t (

1-

1)(1 - Tt)911 1')(1 - Tt)2 R 0'8 - H. I't (1 - R (Jo

(8.37)

SECT.

14.8

497

THE TWO-WAY CROSSED FINITE POPULATION MODEL

using (8.24) and (8.26). The third term is 1 [r(tll l rtll -E !!ziiV - - !!!ZjiV nt

! i

v

i

=

I" i

1. 11 t

i

E[±(i i i

v

;

)2J

v

ZiiV\2J )

1... E[(± ii .i I"tll i

Zii.YJ = (I" }

l)a2,

(8.38) using (8.9). Inserting (8.36), (8.37), and (8.38) in (8.35) and dividing by r - 1, we obtain for the expectation of the row mean square (8.39) The expectation of the column mean square is similarly E [ t _111" 1

t (x.;. t

x.. l

J

= nl"a~

+ 11 (r) 1 - R a~ + a 2•

(8.40)

For the interaction sum of squares, after averaging the model (8.1) over the appropriate indices we get

1r

! I" i

= (Jii - -

(Jij -

It

! t

-

(Ji;

j

1 rt

+ -I"t ! !i

(Jij

i

1t/l

1"

+ -11 !v

Ziiv -

-

1 rn

! !v zijv

1"11 i

1 rtn

- -tll !i !v zijv + -!!! Zi;v. I"tn i i v

(8.41)

When we square and sum and take expectations, the result is a set of terms involving (J and a set of terms involving z. The former reduces to

=

(1 - i)(1- ~) - ; . (1 - ~)(1- ~) -;. r(1- ~) (1 - ~) + ~. rt(1-~)(1- ~) Ja~

11 [I"t

= n(r - l)(t - l)a:.

t

(8.42)

Similarly the expectation of the term involving the z's reduces to

498

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

(r - 1)(t - l)a 2 • Hence the expected value of the interaction mean square is E[

n

±±(xii. -

X; .. -

.Y] =

x.;. + x.

i;

(/' - 1)(t - 1)

a2 + na:.

(8.43)

These results, (8.39), (8.40), and (8.43) are assembled in Table 14.13. Table 14.13 Source of variance Rows

Mean squares

Sums of squares

•

III ~ (XI" -

X.. .)2

I

I

Columns

lIT ~ (X.I. - X.. .)2 i

Interaction

•

I

i

i n

II ~ ~ (XII. -

•

t

Within cells

~ ~ ~ (xilv i I v

Total

~ ~ ~ (Xliv i i v

•

I

XI .. X/I.)2

X.I.

+ X•• .)2

EIM.S.]

f.)

+ lIIa~

s:

a2

+ 11(1 -

S2

3

a2

+ 11(1 - i)a~ + IIrGe

S2

2

a2

+ lIa:

S2

a2

1

a:

n

X•• .)2

14.9. Discussion of the Two-Way Crossed Finite Population Model In Table 14.13, if both the populations are infinite in extent, so that 1 - tfTand 1 - rfR both tend to 1, then we have exactly the expectations of the mean squares given in Table 14.7 for the model II case. If we further assume that the 17;, ,;, and 0;; are normally distributed with variances a:, a~, and a:, then we have exactly model II. On the other hand, if the samples constitute the entire population, so that t = T and r = R, then 1 - tfT = 1 - rfR = 0 and the expectations of the mean squares become those given in Table 14.3 for model I, and the finite model becomes exactly model!. If the rows are a sample from an infinite population and the sample of columns is the entire population, then we have the mixed model with the expectations of the mean squares as given in Table 14.14. The appropriate Source of variance Rows Columns Interactions Within cells

Table 14.14 Type of effect Mean squares II I Mixed II

E[M. S.]

+ 1I1a'12 + na: + I/I'a,. a 2 + I/a:

a2

a2

a2

SECT.

14.10

499

NESTED CLASSIFICATIONS IN THE FINITE MODEL

denominator for the F test, the error term, for the random row effect is the within-cell mean square, whereas the error term for the fixed column effect is the interaction mean square. While O'~ continues to be a sum of squares of a set of fixed constants, with an appropriate divisor, 0'2, 0':, and O'! are true components of variance and can be estimated in the usual way. The fixed column effects can be estimated from the column averages x,i,' We obtain confidence limits for the difference between two column means as follows. Averaging the model (8.1) over i and 'JI gives (9.1) where

_

_

1"

'YJ. =- !'YJi' r

1

i

_

r

! r

(),i =. -

()u,

Z .3. .

i

1

r

n

= -rn '"i '"v Z'i ~~

tV.

Then

If we now assume that ()ij is distributed normally with zero mean and variance 0': for each value of j, then ii,i is distributed normally with zero mean and variance O':/r for each value ofj. Thus E[ii,J = 0, and (9.3) V[X,i. _

x,rJ = 20': + 20'2 l'

I'n

=2(0'2

+ 110':) .

(9.4)

I'll

Confidence limits can be obtained from the relation (9.5) With only one observation per cell, there is no within-cell mean square. Reference to Table 14.14, with n = 1, shows that the validity of the test for the model I effect, columns, is not affected by the change, the interaction being a satisfactory error term, but the test for the model II effect, rows, will only be satisfactory if the interaction component is small compared with 0'2.

0':

14.10. Nested Classifications in the Finite Model Similar to (7.1), we have the model (10.1)

500

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

where i = 1, ... ,r; j = 1, ... ,t;

'V

CHAP.

14

= 1, ... ,n. The 1]; are random R

L 1]; =

samples drawn from a population of size R, and

O. Associated

i

with each i is a population of Aj(i): For each of these populations, of T

size T, we have the condition that LAw) = O. The complete array of j

is indicated in Table 14.15. For the Aj(i), the sampling procedure consists in selecting at random r of the columns (i) and then selecting at random t values of j and taking the corresponding Aj(i). As shown by the last line of the table, for each value of i the entire set of T Aj(i)'s sum to zero, but of course in general the Aj(i) do not sum to zero for the sample of t unless t = T. The Aj(i) do not in general sum to zero within a row; Aj(i)

R

i.e., in general

L

Aj(i)

=F O.

i

Table 14.15 i

= 1,

i

= 2,

, .. ,

i

... ,

= i,

i=R

j=1

A1(1)

A1(2)

AW)

Al(R)

j=2

A2(1)

A2(2)

A2(i)

A2(R)

AT(l)

AT(2)

AT(;)

AT(R)

j =j

j=T T

T L

LAjlll = 0, j

j

Aj(2)

= 0,

T L

.

1

R

,,- R - 1

t"/ i'

= 0,

j

The ziiv are distributed independently N(O, (12- _ _ ~'YI2

Aj(i)

(12).

1

(1~ = R(T _ 1)

T LAj(R) j

=

°

We define

f t A~(i). R

T

(10.2)

The analysis of variance is based on the partitioning of the sum of squares in (7.3). The derivation of the expectations of the mean squares follows the same general lines of the crossed situation in Section 14.8, and leads

501

EXERCISES

to the following results for the mean squares listed in Table 14.10:

E[s:]

=

()'2

E[s:]

=

()'2

E[s:] =

+ 11(1 - ~)()'~ + I1t()'~, + 11(),~,

(10.3) (10.4) (10.5)

()'2.

If the population size of samples within sacks can be regarded as infinite, using the terminology of Table 14.10, T = become identical with those of Table 14.10.

00

and these expectations

EXERCISES 14.1. The accompanying table gives the total acidities of samples of three types of brown coal determined in duplicate, using ethanolic NaOH of three concentrations. (a) Make a conventional analysis of variance of these data. Regard both factors as model I effects. (b) Can the null hypothesis that there is no interaction between the three methods of analysis and the three types of coal be accepted? (c) Can the null hypothesis that there is no difference between the three methods of analysis be accepted? (d) Construct a simple 95 per cent confidence interval for the difference between 00404N and 0.786N ethanolic NaOH, assuming that the factor type of coal is (i) model I, (ii) model II. Ethanolic NaOH 00404N 0.626N 0.786N

Morwell 8.27, 8.03,

8.17 8.21

8.60,

8.20

Yallourn 8.66, 8042, 8.61,

8.61 8.58 8.76

Maddingley 8.14,

8.02,

7.96 7.89

8.13,

8.07

Data from B. Sternhall, "Chemistry of Brown Coals, VI," Australian Journal of Applied Science, 9 (1958), 375-79.

14.2. The accompanying data give the compressive strength of concrete cylinders. A number of cylinders were made from each of five batches of concrete. From each batch, nine cylinders were selected at random. A capping treatment C1 was applied to three of them, another capping treatment C2 applied to another three, and a third capping treatment Ca applied to the other three. The compressive strengths were then measured. The treatments are a model I effect and the batches a model II effect. (a) Give bounds for thePvalues (e.g., 0.10 > P > 0.05) for the null hypotheses that there are (i) no interaction between batches and treatments, (ii) no differences between batches, and (iii) no differences between treatments. (b) Give estimates of the components of variance (i) between cylinders in the same batch given the same treatment, (ii) for interaction between batches and treatments, and (iii)

502

TWO-WAY AND NESTED ANALYSIS OF VARIANCE

CHAP.

14

between batches. (c) Give 95 per cent confidence limits for the difference between treatments C2 and Ca. Batch

2

3

4

5

613 631 603

656 637 649

648 638 649

637 637 617

602 585 608

591 591 597

618 613 619

575 608 612

614 591 580

545 534 547

583 609 614

641 617 634

641 634 614

625 639 627

597 566 593

The sum of squares of all observations is 16,815,853. [Source: George Werner, "The Effect of Type of Capping Material on the Compressive Strength of Concrete Cylinders," Proceedillgs of the American Society for Testillg alld Materials, 58 (1958), 1166-86.] 14.3. For the data of exercise (7.7), (a) test the null hypothesis that there is no difference between containers, (b) test the null hypothesis that there is no difference between panels, (c) construct 95 per cent confidence limits for the difference between containers A and B (i) assuming that that was your original intention and (ii) assuming that this comparison was suggested by the data. (d) What is the estimated variance of the difference between observations on two containers by a single pa~el, i.e., if Xij is the observation on the ith container by the jth panel, what is V[xij - xi'j]? (e) What is the estimated variance of the difference between an observation on one container by one panel and an observation on another container by another panel, i.e., what is V[xij - xi';']? 14.4. Concrete beams were made with different "cement factors," namely 5, 6 and 7 sacks of cement per cubic yard. For each cement factor five replicate batches of concrete were prepared, and from each batch two replicate beams were cast. The modulus of rupture was determined on each beam. Thus in the table below, 671 and 595 are the results for the two beams from the first batch, and 648 and 618 are the results for the two beams from the second batch, etc., made with cement factor 5. The cement factor is a model I factor. Regard the five batches of concrete prepared for each level of the cement factor as a random sample from an infinite population, and the two beams cast from each batch as a random sample from an infinite population. (a) Give bounds for thePvalues (e.g., 0.10 > P > 0.05) for the null hypotheses that there are no differences between (i) batches of concrete made with the same cement factor, and (ii) levels of the cement factor. (b) Give estimates of the components of variance (i) between duplicate beams from the same batch, and

503

REFERENCES

(ii) between batches made with the same cement factor. (c) Give 95 per cent

confidence limits for the difference between cement factors 6 and 7. Moduli of rupture of concrete beams, in psi Five batches made at each level of cement factor Cement factor Two beams cast from each batch

5

671 595

648 618

548 559

604 640

519 596

6

714 618

684 688

618 628

644 657

629 624

7

708 617

683 696

633 665

725 672

634 608

The sum of squares of all observations is 12,283,103. [Source: Stanton Walker and Delmar L. Bloem, "Studies of Flexural Strength of Concrete. Part 3: Effects of Variations in Testing Procedure," Proceedings of the American Society for Testing and Materials, 57 (1957), 1127-1139.] 14.5. Prove the identity (3.10). 14.6. Prove the identity (4.8).

REFERENCES 1. Tukey, John W., "Interaction in a Row-by-Column Design," Memorandum Report 18, Statistical Research Group, Princeton University. 2. Cornfield, Jerome, and John W. Tukey, "Average Values of Mean Squares in Factorials," Annals of Mathematical Statistics, 27 (1956), 907-949. 3. Bennett, Carl A., and Norman L. Franklin, Statistical Analysis in Chemistry and the Chemical Industry. New York: John Wiley and Sons, 1954.

C HAP T E R IS

Three-Way and Four-Way Analysis of Variance

15.1. The Model The methods of analysis of two-way classification of data in the previous chapter generalize to the three-way case, in which observations can be classified according to three independent criteria. Imagine a threedimensional lattice in which the index i refers to rows which might correspond to varieties of corn, j to columns which might correspond to quantity of fertilizer, and k to arrays, say date of harvesting. Suppose that in general in the population there are R rows, T columns, and U arrays, so that there will be RTU cells in the lattice. Suppose that in the sample there are I'tu cells formed by the intersection of I' rows, t columns, and u arrays. Let 'JI be the index of the observation in each cell, going to 11 in the sample. Let ~iik be the true mean for the ijkth cell. Then the model is

(1.1) where the 'Ziikv are normally distributed with zero mean and variance a2• To represent averaging of the ~iik over any suffix, we will use a bar over the ~ and replace the suffix averaged over by an x. The overall average of the ~ijk' i.e., ~""'''''' we will however represent simply by~. The deviation of the ith row mean from the over-all mean we represent by 17i .. : 17i ..

=

~iXX

-

~.

(1.2)

Similarly the column and array effects are 17.i,

= ~Xix

-

~,

(1.3)

17 .. k

= ~XXk

-

~.

(1.4)

504

SECT.

15.1

505

THE MODEL

The deviations of the cell means in the row and column table, formed by averaging over arrays, from the values expected on the assumption that they would be the grand mean ~ plus the row effect plus the column effect are denoted by 'H.: (1.5) The 'ii. are the constants for the row X column interaction. The row X array interaction will be represented by 'i.k and the column X array interaction by '.ik' The deviations of the cell means ~iik from the values expected on the assumption that they would be the grand mean plus the row, column, and array effects plus the three two-way interactions are represented by iik:

°

'ii.

~iik - (~ + 17i .. + 1].i. + 1] .. k + + 'i.k + Ck)' (1.6) The 0iik are the constants for the three-way interaction among rows, columns, and arrays. We can now rewrite the model (1.1) in the form 0iik

=

'ii.

Xijkv = ~ + 1]i .. + 1].i. + 17 .. k + + 'i.k + '.ik + 0iik + Zijkv' (1.7) To represent averaging any of the constants we will use the same convention as for the ~iik' Thus L", is the average over arrays for the jth column of the column X array interaction constants. We will consider the finite population model which will give the other models as special cases. The algebra of the partitioning of the sums of squares and the arithmetic of the calculation of the sums of squares are identical for all models. In the finite population model we suppose that the 1]i .. are a sample of size /' from a population of size R, and the 1].i. and the 17 .. k are samples of t and ufrom populations of size Tand U, respectively. In the population, the various parameters sum to zero over each index: R

!

T

i

= !i

R

T

1]i ..

U

1].i.

= !k

'ii.

RUT

!i 'ii. = ! =! i i R

! i

T

0iik

= 0,

1] .. k

'i.k

= !k

'i.k

= !i

U Ck

= !k

Ck

= 0,

(1.8)

U

=!

Ojik

i

=!

Ojik

= O.

k

We make the definitions ()"2.

-""

()"2' B

,,>

= (R =

()"~Ba =

1

R

'" 17~

- 1) ft .. ,

1

(R -

etc., RT

'" '" 1)(T-1)77 1

(R _ l)(T _ l)(U -

'i

2)"

.

etc.,

(1.9)

R T U

l)'t t f °i

2 ik'

506

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

CHAP.

15

We can write the deviation of Xi;kY from the grand mean x.... as an identity in which the successive terms are sample estimates of the parameters:

x.. J + (x. i .. - x.. J + (x .. k. - x.. J x;. .. + X....) + (X.•. k. - X..... - X..k. + X. ... )

xiikY - x.... = (Xi ... -

+ (X.d..

- X..... -

+ (X. ik. + (Xiik. + (xiiky -

x. i ..

x.. k. + x.. J

-

Xii .. - Xi.k. - X. ik.

+ Xi ... + x. i .. + x.. k. - x.. J (1.10)

Xiik ).

+ X.... in the form [x .... + (Xi ... - x... J + (x. i .. - x... J],

If we write Xii .. - Xi ... - X.; .. Xii .. -

(1.11)

it is more obviously a sample estimate of 'ii.; see (1.5). Similarly, the penultimate term in (1.10) is more obviously a sample estimate of ()iik, see (1.6), if written as

+ (Xi ... - x... J + (X.; .. - x... J + (x .. k. - x... J + (x.f,.. - X..... - x;. .. + X....) + (X ..k. . . - Xf... - X..k.. + X.... ) + (X. ik . - X. i .. - X.. k. + x... J]. (1.12)

Xi;k. - [x ....

Squaring and summing (1.10) over all indices gives an equation which is entered in column 2 of Table 15.1: the sums of all the cross products are zero. The last column gives the expectations of the mean squares, which can be derived by an extension of the methods of Section 14.8 [1]. The closed forms of sums of squares in Table 15.1 are inconvenient for calculation, and open forms involving totals instead of means are more satisfactory. For example, the sum of squares for the main effect for A is

r(_

_)2 = -L 1 r(t "n )2 LLL XiikY -

tun LXi... - X.... i

tun

i

i

k

Y

-

(rLLLL t "n Xi;kY )2.

1

rtun

i

;

k

Y

(1.13)

For the AB interaction r

t

nil LL( x· 'i.. - x·•... - X .J.. i

;

=

1i un i

+ -X ••.•)2

±(f f ;

k

r

Xiiky)2 - _1

Y

- ntu L (Xi ... - X..Y i

rtun

-

(i ±f f i

J

k

t

nru L (x. i .. i

Xiiky)2

y

-

x... Y (1.14)

Total

Within cells

ABC

BC

AC

AB

C

B

A

Source of variance

I

I

..

I

"

I

,

..

11

. .

k

i

I

k

v

~ ~ ~ ~ (xijkv -

;

i

~ ~ ~ ~ (xilkV -

,

Xii •• -

x ••• Y

Xiik.)·

:i ••. .).

rtun -

1

rtu(n·- 1)

(r - 1)(t - 1){u -

I)(u - I)

(t -

+ X ••• .)2

x.lk·

I)(u - I)

(r -

(r - I)(t -

1)

E[M. S.]

j)a"u

a'

a2 + nahc

a2 + n( 1 - j)ah c + nraic

2

1- f)( 1- :i)akc + nt(1- j)a~c + nr(1- f)a i c + nrta~ a + n(1- ~)ahc + nuak a' + n(1 - f)ah c + nta~c

+ nr(1 - -o)aic + nruai

n(1 - j)(1 - ~)akc + nU(1 -

+ nt(1 - ~)a~c + ntual

n(1 - f)(1 - -o)a"uc + nU(1 - f)ak

a2 + n(

a2 +

1

t-

u-I

a2 +

1

r-

I)

Degrees of freedom

+ X ••• Y

+ X ••• .)2

Xi.k. -

X •• k.

X •• k.

X.I ••

+ Xi ••• + x.I •• + X ••k. -

n ~i ~; ~k (xilk. -

,

x.f •• -

..

nr ~I ~k (x.lk. -

I

Xi ••• -

Xi .•• -

X .•. .)2

X ••• .)·

X ••• .)·

nt ~i ~k (Xi.k. -

.

,

I

nu ~i ~ (Xii •• -

r

k

Ilrt ~ (X •• k. -

.

I

nru ~ (:i. I •• -

I

i

ntu ~ (Xi ••• -

,

Sums of squares

Table 15.1

>-!

VI

-.I

0

t"'

0 t::I m

a::

!f\

:I:

Y'

~

--

'"!f\()

508

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

CHAP.

15

where the last two terms are the sums of squares for the A and B main effects. The total sum of squares is "

t

It

n

1 ("

2

t

It

n

)2

!!!!XiikV - -rtun !!!!Xiikv , i i k v i i k v

(1.15)

and the within-cells sum of squares is

(1.16) The sum of squares for the ABC interaction is obtained by difference.

15.2. Models I and II We will get the expected mean squares for a model I three-way analysis of variance by putting I' = R, t = T, and u = U in the coefficients of the components of variance in the last column in Table 15.1. All the factors in parentheses will be zero, giving the second column of Table 15.2, in

Table 15.2 E[M. S.]

Source of variance A B

Model I 0'2 0'2

C

0'2

AB AC BC ABC Within cells

0'2 0'2 0'2 0'2 0'2

+ ntll0'1 + nrllO'~ + nrtO'~

+ 1Il10'1B + ntO'~c + nrO'~c + nO'~BC

Model II 0'2 0'2 0'2 0'2 0'2 0'2 0'2 0'2

+ nO'!be + nllO'!b + ntO'!e + ntllO'! + nO'!be + 1I1100!b + nrO'~e + nrllO'~ + nO'!be + ntO'!e + nrO'~e + n,.tO'~ + nO'!be + nllO'!b + nO'!be + ntO'!e + nO'! be + n,.O'~c + 1I00!be

which all the components of variance are really sums of squares of constants, e.g., O'~

R

=!

'YJ~.. /(R -

1), and are not true variances. All the

i

effects are clearly to be tested against the within-cell mean square. If there is only one observation per cell, the analysis will be unsatisfactory unless O'~BC is small compared with 0'2. We will get the expected mean squares for a model II analysis by putting R = T = U = 00 in the coefficients of the components of variance in the

SECT.

15.2

MODELS I AND II

509

last column of Table 15.1. All the factors in parentheses will be equal to 1, giving the last column of Table 15.2. Here the components of variance are true variances, and we will adopt the convention of using lower-case letters in the subscript to denote this fact. The usual testing procedure is to test the three-way interaction against the within-cells mean square. The next step depends on our point of view. A strict "nonpooler" proceeds to test the two-way interactions against the three-way interaction, irrespective of the outcome of this first test. Then he would test the main effects against appropriate linear combinations. For example, to test the A main effect he would use the mean squares for AB + AC - ABC, since the expected value for this combination is

(2.1) This linear combination of three mean squares would have its approximate degrees of freedom estimated by (9.8.10). Alternatively, some people would indulge in some judicious pooling. If at the first test the mean square for ABC was neither statistically significant at the chosen level of significance nor had a variance ratio exceeding 2, the "sometimes-pooler" would pool the sums of squares and degrees of freedom of the ABC and within-cells term, and use this as an estimate of 0'2. The assumption is being made that O'!bC = 0, and so O'!bC is stricken out of all the expected mean squares. Similarly, if one of the interactions, say AB, was statistically significant at the chosen level of significance, or if the variance ratio exceeded 2, the sometimes-pooler would leave this mean square untouched, but otherwise he would assume that O'!b was zero, pool its sum of squares and degrees of freedom with those for ABC + within cells to get a better estimate of 0'2, and also strike O'!b out of the expectations of the mean squares wherever it occurred. With this procedure, if none of the two-way interactions was significant nor had variance ratios exceeding 2, they would all end up by being pooled with the within-cells mean square, and the three main effects would be tested against this pooled error term. The "never-pooler" can be confident that his errors of the first kind have the stated probability. The sometimes-pooler may be somewhat uncomfortable about this, but he will claim that his errors of the second kind have smaller probability than those of the never-pooler. If the sample sizes are large, so that degrees of freedom are plentiful, the motivation, or temptation, to be a sometimes-pooler are less. The rule about the factor 2 comes from Paull [2]. If, in a model II analysis, there is only one observation per cell, of course there is no within-cell mean square, but putting n = 1 in the expectations of mean squares in Table 15.2 does not affect the testing procedure.

510

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

CHAP.

15

A further complication in the testing of complicated analyses of variance is the multiplicity of tests being performed. For example, in a five-factor analysis there will be 5 main effects, 10 two-way interactions, and 10 three-way interactions. Thus, ignoring the four-way and the five-way interactions, we may be testing 5 + 10 + 10 = 25 effects. Suppose that Xl, •.• , xn are identically distributed independent observations. Then one of the sample of n will be the largest: Call this X mnX . The condition that the largest is less than some value, say x, is the same as the condition that they are all less than x, i.e., Pr{xi

< X, X 2 < X, ••• }

= Pr{x1Il8X

< x}.

(2.2)

Also, since the observations are !ndependent, Pr{xi

< x, X 2 < X, ••• }

=

so Pr{xlllftX

< x} Pr{x2 < x} ... (Pr{x/ < x})";

= Pr{xi

< x} =

(Pr{xi

< x})".

(2.3)

(2.4)

Now there will be a P point of the cumulative distribution of X lll8X such that (2.5) Also, substituting xmnx,P for x in (2.4), Pr{xmftx

< xm8x ,p} = (Pr{xi < xmftx,p})".

(2.6)

Comparing (2.5) and (2.6), we see that

(2.7) or

(2.8) In other words, the P" point of the distribution of xmax is equal to the P point of the distribution of x. Thus, if we are testing 10 independent F ratios, all with the same degrees of freedom, so that they have the same distribution under the various null hypotheses, the 0.99 point of the distribution for the largest of the 10 is actually the ~0.99 ~ 0.999 point of the ordinary F distribution. Of course, in the usual analysis-of-variance situation the Fratios are not independent since they use a common denominator mean square. Finney [3] showed that for moderately large, say greater than 10 or preferably 20, denominator degrees of freedom the F ratios could be assumed to be independent without serious error. For the special case where the numerator degrees of freedom are 1, Nair [4] tabulated the 0.95 and 0.99 points of the largest variance ratio for denominator degrees of freedom starting at

SECT.

15.3

MIXED MODELS

511

10, reproduced as Table X of the Appendix. Unfortunately his table only / goes to the largest of 10 variance ratios. Daniel [5] and Birnbaum [6] have developed a scheme for the significance testing of a large number of mean squares with single degrees of freedom which has promise. However, a completely satisfactory procedure must give weight to the relevant a priori probabilities. It is found by experience that main effects are more frequently significant than two-way interactions, and two-way interactions are more frequently significant than three-way interactions, and so on. Thus, if we find two effects, one a main effect and the other a four-way interaction, both significant at the 0.025 level, we would have little hesitation in accepting the former as real and dismissing the latter as an instance of random fluctuation. Also, the pattern of significance conveys relative information. If we find the main effects A, B, and C and their interactions AB, AC, and BC significant, we would not be surprised to find ABC significant, whereas, if ABC was significant without any of the other effects mentioned being significant, we would be tempted to regard this as an accident of random fluctuation. It seems clear that an efficient analysis of a multi-factor experiment is at present somewhat subjective. One could certainly lay down certain rules, but they would lead to a higher proportion of errors, of the first and second kinds, than an intelligent and experienced practitioner would commit. 15.3. Mixed Models In three-way analysis of variance there are two mixed models, one in which one factor is random and two factors are fixed, and vice versa. In the first case, if a is random and Band C are fixed, we put R = 00, t = T, and u = U in the expectations of mean squares in Table 15.1, and obtain the left-hand side of Table 15.3. On a nonpooling basis, the tests of significance are clear. For example, aB and aC are tested against within cells whereas BC is tested against aBC, and a is tested against within cells whereas Band C are tested against the aB and aC interactions. If there is only one observation per cell, then satisfactory tests for a, aB, and aC exist only if (J'~BC is small, but the tests for B, C, and BC remain valid. When a and b are random and C fixed we put R = T = 00, U = U in Table 15.1, and this gives the right-hand side of Table 15.3. The tests are straightforward except that a linear combination of mean squares aC + bC - abC is necessary to provide a satisfactory error term for C. If there is only one observation per cell, so that there is no within-cell mean square, the tests are unchanged except for ab, which requires (J'~bC to be small for a satisfactory test. -

512

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

CHAP.

15

Table 15.3 a random, Band C fixed

a and b random, C fixed

E[M. S.l

Effect

E[M. S.l

Effect

C

+ IIfIlU! + IIIIU!B + IIrllu~ U· + IIfU!o + IIrtU~

C

+ IIIIU!b + IIfIlU! + IIIIU!b + IIrllu: U· + IIU!bO + IIfU!o + IIru:o + IIrta~

aB aC BC aBC

+ IIIIU!B U· + IIfU!o u· + IIU!BO + IIru~o u· + IIU!BO

ab aC bC abC

+ IIIIU!b + IIU!bO + IIfU!o u· + IIU!bO + IIru:o a' + IIU!bO

Within cells

u·

Within cells

u·

a B

u· u·

u·

a b

u·

u·

u·

u·

15.4. Confidence Limits in Three-Way Analysis We will discuss the situation where there is only one observation per cell and it is assumed that the three-way interaction is zero; i.e., the model (1.7) with ')I = 1 and (Jiik = 0 becomes X iik

=

e + 1]i .. + 1].i. + 1] .. k + 'ii. + 'i.k + '.ik + Ziik'

(4.1)

We will discuss the construction of confidence limits for the difference between two row means. Using (1.2) 1]i .. -

1k ..

= af.,., -

e) - (~i'"''

-

e)

= ~f"" - ~i'='

(4.2)

Thus confidence limits for 1]i .. - 1]i' .. are identical with confidence limits for ~f"" - ~i'''''' In the model I case, if we want confidence limits for the difference between two row means, then we will be making the assumption that the interaction involving rows, namely, the 'ii. and 'i.k, are zero; so the model becomes (4.3) X iik = + 1]i.. + 1].i. + 17 .. k + '.ik + Ziik •

e

Averaging over j and k, we get Xi ..

since

=

e+ 17i .. + 17..,. + 17..., + L., + Zi .. = e + 1]i .. + Zi .. , 1 tIT i ii ., ..,. --"''11 k ·'.i. --"'17 k .i. -0 , tit

since t = T, and similarly X.'t-.. -

17..., =

r.,., =

x.,1 •• = (17'i... -

(4.4)

(4.5)

j

O. Thus

'11., 'I t •• )

+ (z.

~..

- z·, ) t ••

,

(4.6)

SECT.

15.4

513

CONFIDENCE LIMITS IN THREE-WAY ANALYSIS

with expectation and variance E[xi ..

-

Xi,.J = 17/ .. -

V[X i ..

-

Xi,.J = V[zd

(4.7)

1];, .. ,

+ V[Zi'.J =

(4.8)

20'2.

til

Confidence limits for 1k. - 17r .. can be obtained by inserting in place of 0'2 the mean square for ABC, which is assumed to estimate 0'2. In the mixed model of the type (a, B, C), if we are going to construct confidence limits for the difference between two column means, 17.i. - 17.1'.' we will be making the assumption that the BC interaction '.ik = 0, and so the model (4.1) becomes xiik

= ~ + 171 .. + 17.i. + 17 .. k +

'ii. +

+ Ziik'

(4.9)

+ 17",.. + 17.i. + 17..", + ~"'i. + ~""'" + i.i.,

(4.10)

'i.k

Averaging over i and k, we get

x. i. = ~ whence

x. i . The

x.i'. = (17.i. - 17.1'')

-

+ (~"'i.

- ~"'i',)

+ (i.i.

- i.1'')'

(4.11)

'ii. is a mixed interaction subject to the conditions 1 '.i. = 1 'ii. = 0, R

T

i

J

t

00

but, since R

'ii.

=

00

and T

=

t, these conditions become

1i 'ii. = 1 j

'ii. =

O.

Thus is distributed normally with zero mean and variance O'!B for each level of j, and hence ~"'i. is distributed normally with zero mean and variance O';n!r for each level of j. Hence E[x. i .

-

x.1'.l

=

(4.12)

(17.i. - 17.1''), 20'2

20'2

2

I'll

I'll

V[x. i . - x.d = ---.!!!!. + - = r

A

V[x.i. - x. i -.]

2

=-

(0'2

+ 1I0'!n),

(M. S. for aB),

(1. 13) (4.14)

I'll

and confidence limits follow in the usual way. In the mixed model of the type (a, b, C), we have

x.. k -

x.. k = ~ + 17", .. + 17.",. + 17 .. k + ~"''''. + ~",.k + Lk + i .. k, x.. k' = (17 .. k - 17 .. d + (~",.k - ~",.k') + (r"'k - Lk') + (i .. k -

(4.15)

i .. k,)·

(4.16)

The

'i.k and '.n. are mixed interactions distributed normally with zero

514

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

means and variances E[X"k -

_

V[X"k -

X"k']

_

X"k']

J?'[X"k - X"k']

O'!e, 0':0 for each level of k.

(4.17) (4.18) (4.19)

abC],

I't

15

Hence

= '/}ook - '/}"k" 20'~0 20':0 20'2 2 2 ) =- + -- + - = -2 ( 0'2 + to'aO + 1'00w , I' t I't I't = ~ [M. So for aC + bC -

CHAP.

and the linear combination of mean squares will have its approximate degrees of freedom given by (9.8.10). It will be noticed from the foregoing results that it is unnecessary in practice to go through these calculations to obtain the estimated variance of the difference between two means, since the answer always comes out to be twice the appropriate error mean square for testing the corresponding effect divided by the number of original observations in each mean being compared. 15.5. An Example of Three-Way Analysis of Variance The data of Table 15.4 (taken from [7]) were obtained in a research program aimed at developing procedures for the bacteriological testing of milk. Milk samples were tested in an apparatus which involved two major components, a bottle and a tube. The result of a single test is simply Table 15.4 Bottle (k)

II

Tube {j) Milk sample (i)

1 2 3 4 5 6 7 8 9 10 11 12

A

B

C

A

B

C

1 3 3 2 2 1 5 1 0 3 0 0

1 4 2 4 1 1 5 1 1 4 0 1

1 2 4 1 3 2 5 1 2 5 4 2

1 2 3 1 2 0 3 0 2 1 0 0

3 1 3 0 4 2 5 2 2 1 2 3

2 3 6 0 6 1 5 0 2 3 1 1

recorded as growth or failure to grow. All six combinations of two types of bottle and three types of tube were tested on each sam pIe, and 10 tests were run with each sample X bottle X tube combination. Table 15.4 gives the number of positive tubes in each set of 10. As discussed in Section 3.3, this variable should be binomially distributed, and, to obtain a variable with a stable variance, we should use the inverse sine transformation. However, our main purpose in presenting this example is as an illustration of the calculations for a three-way analysis of variance, and this will be

15.5

SECT.

I

515

AN EXAMPLE OF THREE-WAY ANALYSIS OF VARIANCE

achieved better by using the simple integers of Table 15.4. It is, however, instructive to also carry out the analysis of variance on the transformed variable and see how little the conclusions are affected. Let i refer to samples, j to tubes, and k to bottles. Regarding Table 15.4 as a three-way classification with one observation per cell, the operations of summing over cells and calculating a within-cells sum of squares do not arise. The first step in the analysis is to form sums over every index and every combination of indices; e.g., we sum over samples (i) to obtain a tube r

Table 15.5. Sums over Samples (i),

2

Tube (j)

XiJk

,

,

~~xiJk

Bottle (k)

A

B

C

I II

21 15

25 28

32

30

78 73

36

53

62

151

i

,

u

~~xilk I

I

k

r

(j)

bottle (k) table containing

X

2

Xi;k

(Table 15.5). This table is then

i

summed over bottles (k) to give the tube totals r

t

i

j

r

it

i

k

r

t

22

X iJk

and over tubes

(j) to give the bottle totals 2 2 X iJk • The sum of the tube totals, equal to the sum of the bottle totals, is the grand total and 15.7 are obtained similarly. Table 15.6. t Sums over Tubes (j),

2

Bottle (k) II

3

2

3

9 9

4

7 6

Sample (i)

A

B

C

9

1 2 3

2 5 6 3

4

3 5 10 1

4

5 3 10 3 3 5 2

4

3

4

~~"'lIk

78

73

151

I

,

I

15 21 8 18 7 28 5

4

5 6 7 8

9

9

1 8 1 2

10

4

11

0 0

12 ,

4

9

3 10 1 4

4

8 5 3

53

62

k

9

15 21 8 18 7 28 5 9 17

7 7

u

~~xilk I

5 5

u

~~Xilk I

k

11

,

XiJk

,

~~x{lk

12

15 3 3 12

tI

2 k

u

17 7 7

9

10

4

Table 15.7. Sums over Bottles (k),

Xi;k

6 6 12 1 12 3 13 2 6 5 3

5 6 7 8

Tables 15.6

Xiik"

k

j

Tube (j)

I

1

i

J ,

Sample (i)

/I

222

k

36

151

516

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

CHAP.

15

With these preliminary summations the subsequent calculations are straightforward. The sum of squares for samples is given by (1.13) with the modification that 11 = 1 and summation over p is not involved. S.S. for samples

2 X 3

S.S. for tubes

=

S.S. for bottles

=

For the sample 22

2 + 15 2 + ... + 7 2 = 9:...-....:......---'------'-----

+

52

+

= 93.486 '

36 2 + 53 2 + 62 2 (151)2 = 14.527 12 X 2 2 X 3 X 12 ' 78 2 + 73 2 (151? = 0.347. 12 X 3 2 X 3 X 12

tube interaction, the sums of squares is given by (1.14):

X

62 +

(151)2 2 X 3 X 12

... +

2

32

(151? _ 93.486 - 14.527 = 22.806. 2 X 3 X 12

The other two interactions are calculated similarly. The total sum of squares is given by (1.15): (12

+

32 +

... +

12) _

(151)2 = 184.319. 2 X 3 X 12

All these items are entered in Table 15.8 and the three-way interaction is calculated by difference. Table 15.8 Sums of squares

Degrees of freedom

Mean squares

Samples

93.486

11

8.499

Tubes

14.527

2

7.263

Bottles

0.347

Source of variance

EIM.S.]

0.347

a2 + 3 X 2a! a2 + 2a!T + 2 X 12a~ a2 + 3a:B + 3 X 12a1

Samples X tubes

22.806

22

1.037

a2

Samples X bottles

27.153

11

2.468

1.695

2

0.847

24.305

22

1.105

a a2 a2

184.319

71

Tubes X bottles Samples X tubes X bottles Total

2

+ 2a:T + 3a:B + 12a~B

The tube and bottle are fixed effects and the sample is a random effect. If s, T, and B refer to samples, tubes, and bottles, application of the left half of Table 15.3, with 11 = 1 and a~TB = 0, gives the expectations of mean squares in the last column of Table 15.8. Interpreting this table as a nonpooler, it is apparent that bottles, samples X tubes, and tubes X bottles are nonsignificant. Samples X bottles has a

15.6

SECT.

517

ORTHOGONAL CONTRASTS

variance ratio 2.23, and FO•95(11, 22) = 2.26; so it does not quite reach the 0.05 level of significance. The tube main effect, tested against the sample X tube interaction, has a variance ratio of 7.00, and FO•975(2, 22) = 6.81; so there is no doubt as to the significance of the tube effect. The sample effect is also highly significant, and the component of variance (J'! is estimated as (8.499 - 1.105)/6 = 1.232. Confidence limits can be constructed with (4.14). For example, for 95 per cent confidence limits between the two bottles, we want (0.975(11) = 2.201, and (X .. I - x.. u) = (78 - 73)/12 X 3 = 0.139: Also A

V[x .. I

-

x.. u] =

2

12 X 3 [M.S. for samples X bottles]

= 0.1371.

The confidence limits are 0.139 ± 2.201y'0.1371, or (-0.676,0.954). Confidence limits for the difference between any two tube averages can be constructed similarly, although if we are interested in more than one comparison it would be advisable to use Tukey's multiple-comparison technique (cf. Section 10.3). 15.6. Orthogonal Contrasts It is possible to split the r - 1 degrees of freedom of a model I factor up into r - 1 separate single degrees of freedom, each corresponding to a specific contrast, in such a way that the r - 1 contrasts are independent and the sum of the sums of squares corresponding to each contrast adds up to the original unpartitioned sum of squares. Suppose that we have r means Xi distributed normally with means ~i r

and variances

(J'2/11i'

We define a contrast as in (10.3.1) as ()

= 2 Ci~j, i

r

where

2

r

Cj

= 0. As in (10.3.2) the contrast is estimated as H =

i

2 cix

j'

i

and H is distributed normally with expectation () and variance V[H] = r

2'" 2/

(J' £., C i ll i •

Therefore H - () r (

(J'2

2 C~/Ili

)v. ,. . ., N(O, 1),

(6.1)

i

and (6.2)

518

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

= 0 the quantity M

Thus under the null hypothesis ()

M

(i

CHAP.

15

defined as

CiXi\2

= ~ir _-','--

(6.3)

~ c~/ni i

will be distributed as

and the ratio of (6.3) to an independent estimate 8 2 of 0'2 with! degrees of freedom will be distributed as F(I,f). More generally, taking expectations of (6.2), n

0'2

2

~ E.!

0'2X2(1),

= E[H2] + ()2 _

i ni

2()E[H]

= E[H2]

_

()2,

(6.4)

r

whence, substituting H = ~ CiXi and rearranging, i

(6.5)

If the Xi are based on totals can write M, (6.3), as

Ti

M

of n i observations, so that

xt =

(i Ci Ii/IIi\2

= -!...!..i_ _-,''-r

Ti/lli'

we

(6.6)

~ C~/Ili i

and, when the ni all equal n, this becomes M =

(i CiIi)2

....:....:...i_--,-r

11

(6.7)

~ c~ i

Suppose now that we have two contrasts, (6.8) and we require these to be independent. We saw in Section 12.3 that for independence it is sufficient, when HI, H2 are jointly normal, to show that they have zero covariance. Now V[HI

so

+ H 2] =

V[HI ]

+ V[H2] + 2 COV[HI' H 2];

(6.9)

(6.10)

SECT. 15.6

But

*

519

ORTHOGONAL CONTRASTS

V[H I + H2] = V[* Clixi

+

C2i Xi]

*(C

= V[*(C li + C2i)Xi] =

li

+ C2i)2V[Xi ] (6.11)

so

2 COV[HI' H 2] =

20'2

i c nc

(6.12)

li 2i •

i

i

Thus independence between HI and H2 implies (6.13)

or, when the ni are all equal to n, r

!

CliC2i

(6.14)

= 0,

i

For l' - 1 degrees of freedom, it is possible to construct infinitely many sets of l' - 1 orthogonal contrasts, but we are only interested in contrasts that are reasonable. In the milk testing example in Section 15.5, the three tubes were such that tubes Band C were quite similar in size but A was much smaller. It is therefore reasonable to define contrasts with coefficients C li

= 0, -1, 1,

C 2j

= -2, 1, 1. t

These coefficients do define contrasts since! cmj

(6.15)

=

°for

X

1

;

111

=

1 and 2.

Also they are orthogonal as they satisfy (6.14): t

!;

CI ;C2j

°

= x (- 2) + (-1) x 1 + 1

= 0.

(6.16)

The first contrast will measure the difference between Band C, and the second will measure twice the difference between A and the mean of B and C. By (6.7) the sum of squares for the contrast H", will be (

M III =

t

)2

ru

~ Cmj~!

Xijk

'k

1

t I'll

!

j

C~,,;

.

(6.17)

520

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

For m

=

15

1, this is [0 X 36 + (-1) X 53 + 1 X 62]2 12 X 2[0 2 + (_1)2 + 12]

and, for

CHAP.

111

=

1.687

(6.18)

,

= 2, it is

[(-2) X 36 + 1 X 53 + 1 X 62]2 12 X 2[(-2)2 + 12 + 12]

= 12.840.

(6.19)

Each of these sums of squares has one degree of freedom, and the sum 1.687 + 12.840 = 14.527 is the unpartitioned sum of squares for tubes with two degrees of freedom. The appropriate error term for the partitioned mean square is the same as for the unpartitioned mean square, the sample X tube interaction in this case. For the two components the F's are 1.687/1.037 = 1.63 and 12.840/1.037 = 12.4. The first contrast, between the similar tubes, is completely nonsignificant, but the second contrast, between the small tube and the mean of the larger tubes, is highly significant. The set of orthogonal contrasts in Table 15.9 is applicable to the case where the factor is a numerically valued variable for which the gaps between successive levels are equal. The coefficients ~i, ~~, etc., will give the sums of squares for linear, quadratic, cubic, etc., terms in a polynomial of the dependent variable y on the factor x. Obviously, for k levels only k - 1 terms can be fitted. To construct the polynomial regression equation is not difficult, but we will not deal with it here: See [8], Chapter 16. Coefficients up to the fifth degree for numbers of levels k up to 52 are given in [9]. Table 15.9 k=3

"=5

k=4

e~

e;

e~

e;

e;

e~

e;

e;

e~

-1

1 -2 1

-3 -1 1 3

1 -1 -1 1

-1 3 -3 1

-2 -1

2 -1 -2 -1 2

-1 2

1 -4 +6 -4 1

0

1

0

1 2

0

-2 1

15.7. The Partitioning of Interactions into Orthogonal Contrasts A t X u two-way table will have (t - 1)(u - 1) degrees of freedom for interaction, and, if both the classifications are fixed effects which have been partitioned into orthogonal contrasts, it is possible to compute sums of squares with single degrees of freedom corresponding to the pairwise

SECT.

15.7

521

THE PARTITIONING OF INTERACTIONS

/

interaction of single degrees of freedom of the two sets of orthogonal contrasts. An example will make this much clearer. Table 15.10 (a)

dl d2

= -1 =1

(b)

Cn

Cl2

0

-1

1

0 0

1

-1

-1 1

C21

Cl3

dl d2

= =

-1 1

C22

C23

-2

1

1

2 -2

-1

-1 1

1

For the milk testing example in Section 15.5, we partitioned the main effect of tubes into two contrasts in Section 15.6. The other fixed e.ffect, bottles, has only two levels, and so only one degree of freedom, but we can regard this as a single contrast defined by coefficients dk : dl = -1, d2 = 1. We then set up Table 15.10. We form the products dkc m; in the body of the table. The coefficients so formed are contrasts, since within each table they sum to zero, and they are orthogonal, since the sum of products of corresponding coefficients is zero. (If the factor bottles had four levels, and we had partitioned it into three contrasts, there would have been three sets of d's, namely, d1k , d2k , and d3k , and we would have gotten 6 two-way tables of coefficients corresponding to die 6 individual degrees of freedom of the interaction.) To get the sums of squares corresponding to these two degrees of freedom, we use (6.7) again, and get the sum of the products of the coefficients in Tables 15. lOa and b with the totals in Table 15.5. From Table 15. lOa, [0 X 21

+ 1 X 25 + (-1) X 32 + 0 X 15 + (-1) X 28 + 1 X 30]2 12[02 + 12 + (_1)2 + 0 2 + (_1)2 + 12] (_5)2

= - - = 0.521.

12 X 4

(7.1)

Table 15.lOb gives a sum of squares 1.173, and the sum of these two components is 1.694 with 2 degrees of freedom, which is equal to the unpartitioned sum of squares for the bottle X tube interaction. In the example of Section 15.5, two of the two-way interactions are mixed model interactions. For example, the tube X sample interaction has tube as a fixed effect, which is partitionable into orthogonal contrasts, and samples as a random effect, which cannot be partitioned into individual degrees of freedom. However, the (r - 1)«( - 1) degrees of freedom can be partitioned into ( - 1 components, each consisting of r - 1 degrees :>f freedom, representing the interaction of each component of the tube

522

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

CHAP.

15

effect with samples. We calculate the value of the contrast for each sample, and then find the variance of these. For example, consider the contrast defined by the coefficients cli = 0, -1, 1. Let Hli be the value of this contrast for the ith sample. (7.2)

Then E[Hli]

t

t

i

i

= ~ cljE[xuJ = ~ Clj!ij", = 0li> t

V[H li ] = ~ c~;V[Xid ;

(7.3)

(12 t

~ c~;.

=-

II

(7.4)

i

Suppose that the null hypothesis that all the 0li

= iiI. = 01 is true.

Then

r

~ (Hli -

Hl.)2/(r - 1) is the usual sample estimate of the variance of the

i

Hli and

1

~ (H l'

H-)2 r-..J

- - £. I' -

1

(12

~

2

X2(1' - 1)

£. Cli "'--0._----" u; (I' - 1)

1 i t .

,

(7.5)

so if we define N as r

II

~ (Hli -

Hd 2

N= - i - - -t- -

(7.6)

(I' - 1) ~ c~; ;

it is distributed as (12X2(I' - 1)/(1' - 1). Thus under the null hypothesis 0li = 01> the ratio N/S2, where S2 is an independent estimate of (12 with / degrees of freedom, has the F(r - 1,/) distribution. Substituting (7.2) for H li , N can be calculated from the relationship

We will now

~nd

E[N]. We have the identity

Hli - 0li

= (Hli -

HI.) - (Oli - iiI.)

+ (HI.

- iiI),

(7.8)

SECT.

15.7

523

THE PARTITIONING OF INTERACTIONS

whence l'

Li (Hli -

l'

()li)2 =

Li [(H1i -

ii1J -

01J]2 + r(ii1. - 01J2, (7.9)

«()li -

since the cross product is zero: l'

2 L (iiI. - 01J[(Hli

H1', )--

-

«()li -

01J]

i

t

The left-hand side of (7.9) is distributed as «(]2/U) L c~ix2(r). Thus taking expected values, we have i

Thus

E[

t (Hli r

_

Hd

2J

(]2

= (r - 1) -;;

t

2 Cli

t

+ t «()li l'

2

-

()lJ ,

(7.12)

whence, referring to (7.6), r

E[N]

=

II

(]2

L «()li - 01i

+ ----!i--~t--

(7.13)

(r - I)Lc~i i

" x iik are The sums .2

obtained by summing over bottles and were given

k

u

It

in Table 15.7. The sums

L Xm , .2 X12k'

1t

L Xl3k

are given by the k k k entries in the first row, i = 1, in that table: 2,4, and 3, respectively. Then for i = 1, 2, 3, ... , r, we have UHli taking the values t

"

i

k

t

"

i

k

t

"

i

k

IIHll=Lc1iLxlik= [0

and

X

2+(-1)

X

4+ 1

X

3]= -1,

[0

X

5 + (-1)

X

5+ 1

X

5]

UH1r=Lc1iLxrik= [0

X

0+(-1)

X

4+ 1

X

3]= -1.

uH12

=

Lcli.2X2ik

=

=

0,

(7.14) (7.15)

(7.16)

524

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

CHAP.

15

Then, since u = 2, the first term in (7.7) is

i (± c

li

•

't

~

r

X iik

k

=

(_1)2 + 02 + 52 + ... + (_1)2 75 2x[02+(-1)2+12] =2X2=18.750,

~ , C~i (7.17) whence the sum of squares for the interaction of this contrast with samples is 18.750 - 1.687 = 17.063. A similar calculation for the second contrast gives for the interaction sum of squares L/

32 + 02 + 3 2 + ... + 7 2 - 12.840 = 18.583 - 12.840 = 5.743. (7.18) 2[( - 2)2 + 12 + 1 ]2 The sum of these two components, 17.063 + 5.743 = 22.806, equals the unpartitioned sum of squares for the sample X tube interaction. 15.8. Four-Way Analysis of Variance

The analysis of a four-way classification is the obvious extension of three-way analysis, and we will review it only briefly. The model is Xiikmv

where the

~iikm

=

~iikm

+ Ziik",v

(8.1)

have the structure

~iikm = ~ + 'Jt ... +

'i.

+ 1] .. k. + '7 ... 111 + 111 + '.ik. + C.", + ' .. kllt . + e (8.2) IIt i k", + e. ik ", + W ii"", and the Ziik""" 'JI = I, ... ,11, are normally distributed with zero mean and variance (12. The sample sizes for i,j, k, mare r, t, u, v, and the population sizes are R, T, U, V. The ,];. .. are a sample of size r from a population of size R and sum to zero in the population '7.i ..

+ 'ii .. + 'i.k. + eiik . + eil .

R

L'h .. = 0; i

(8.3)

similarly for the other main effects. The two-way terms sum to zero in the population over each index, e.g., R

T

Li 'ii .. = L 'ii .. =

O.

(8.4)

j

The three-way terms and the four-way terms also sum to zero in the population over each index, e.g.,

(8.5) and

(8.6)

SECT.

15.8

525

FOUR-WAY ANALYSIS OF VARIANCE

We define O'~, O'~B' etc., analogously to the definitions (1.9). The usual identity xOk>llv - x ..... = etc. has on its right-hand side the following groups of terms: 1. Sample estimates of the four main effects, e.g., Xi .... - X..... , which is an estimate of 'YJi ... ' 2. Sample estimates of the six two-way interactions, e.g., xii ... - Xi .... x. i ... + X..... , which is an estimate of ~ij .. ' 3. Sample estimates of the four three-way interactions, e.g., Xiik .. xii ... - Xi . k.. - X. ik .. + Xi .... + X. i ... + X.. k.. - X..... , which is an estimate of ()jjk .. 4. The sample estimate of the single four-way interaction, which will have the form xUk>ll. - [x ..... + main effects + two-way interactions + three-way interactions]. 5. The deviations of the individual observations from the cell means, e.g.,

xii 1..·m.-

xi;knu' -

Squaring and summing over all indices gives the partitioning of the total sum of squares. Typical sums of squares and computing identities are r

A

= IIVUt L (Xi .... -

X... .f

i

= -1t

i (iiii

U VII i

i

=

'"

2 -

-1-(iiii~>iiklllv)2, i

I" til VII

i

k

.".

V

(8.7)

L L (xii ... -

IIVU

Xiik>llV)

v

t

r

AB

k

i

Xi .... -

X. i ...

j

r t (" " )2 = - L L LLLxiikmv 1

+ x... J 2

11

UVII i

i

k

v

III

-

1

I"tllVII

(rLLLLLxiik>llV til" " )2 i

i

k

m

v

- sum of squares for A - sum of squares for B, r

ABC =

IIV

t

u

L Li Lk (xiik .. -

xii ... - Xi .k.. - X. ik ..

i

=~

iii (i i

VII i

i

k

Xiik1llV)2 -

+ Xi .... + x. i ... + x..k •• - x... .f

_1 (i iii i

I"tuVII

v

>II

i

i

k

III

Xiik>llv)2

V

- (sums of squares for A, B, C, AB, AC, BC),

ABC D

=!

iii i (i

II i

i

k

III

Xiikmv)2 -

_1_ I"tu VII

v

(i iii i i

j

~.

>II

r

til"

n

= L L L L L X:j~''''V -

Xjjk>ll,.)2

iikmv

r t i l " (n )2 L L L L L Xijkmv . l1iikm v 1

-

(8.9)

V

( sums of squares for all main effects) two-way and three-way interactions' Within cells

(8.8)

(8.10) (8.11)

526

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

CHAP.

15

The degrees of freedom for the foregoing are r - 1, (r - 1)(1 - 1), (r - 1)(1 - 1)(u - 1), (r - 1)(1 - 1)(u - 1)(v - 1), and rluv(n - 1), respectively. The expectations of the mean squares are as follows: E[M. S. for A]

~) (1 - ~) (1 - ~)(J~BaD

= (J2 + 11(1 -

1- ;)(1 - ~)(J~Ba + 11 1- ~)(1- ~)(J~BD + 11/(1 - ~)(1 - ~)(J~aD + nuv(l - ~)(J~B + I1tV(1- ~)(J~a + ntu (1 - ~ )(J~D + ntuv(J~, (8.12) + IlV (

E[M. S. for AB]

U(

= (J2 + 11(1 -

~)(1 - ~)(J~BaD + lIV(l - ~)(J~Ba + 1111

(1 - ~ )(J~BD + nllv(J~B'

~ )(J~BaD + nV(J~Ba, E[M. S. for ABCD] = (J2 + lI(J~BOD'

E[M. S. for ABC]

= (J2 + 11 (1

-

(8.13) (8.14)

(8.15) The various models can be obtained from this finite population model.

EXERCISES 15.1. Four strains of a microorganism producing an antibiotic in a fermentation process were to be compared. The comparison was made on four batches of the main ingredient of the broth. A further factor was introduced into the experiment, namely, concentration of the broth, at three levels, equally spaced. The yields were as follows:

Concentration

Concentration Batch

Strain A

B C D

2

A

B C D

40 52 78 59 47 64 73 77

2 69 71 100 76

3 70 91 110 108

Batch

76

91 99 143 127

4

72

122 106

3

55 61 71 78

2 79 83 106 103

3 102 94 106 127

44 69 87 76

77 75 106 107

85 116 131 125

Strain A

B C D A

B C D

527

EXERCISES

The strains are a model I effect, and in fact the main objective of the experiment is to select the best strain. The batches are a model II effect, and the concentration is a model I effect, partitionable into linear and quadratic components. (a) Give the conventional analysis of variance with the expectations of the mean squares. Make tests of significance of the various effects. State succinctly your interpretation of the effects of the various factors. (b) Give 95 per cent confidence limits for the difference between strain C and strain D, averaged over batches and concentrations. 15.2. The table below gives the results of an experiment on the amounts of niacin found in peas after various treatments. The factors were: (a) A comparison between blanched peas Po and processed peas Pl' (b) Two temperatures of cooking: Co = 175°F, C l = 200°F. (c) Two times of cooking: To = 2t min, Tl = 8 min. (d) Three d1fferent sieve sizes of peas: Sl' S2' S3'

Po

PI

C

S

To

Tl

To

Tl

0

1 2' 3

91 92 112

72 68 73

86 85 101

68 72 73

1 2 3

84 94 98

78 78 73

83 90 94

76 71 76

Source: R. Wagner, F. M. Strong, and C. A. Elvehyem, "Effects of Blenching on the Retention of Ascorbic Acid, Thiamine, and Niacin in Vegetables," Industrial and Engineering Chemistry, 39(1947),990-93. Make an analysis of variance of these data. Suppose that the levels of the factor sieve size are spaced equally, and partition the main effect of sieve size into components corresponding to linear and quadratic terms: likewise for the first-order interactions. Include in the analysis the second-order interactions, but do not bother with any partitioning of these. The highest-order interaction mean square will have too few degrees of freedom to be a satisfactory estimate of error (since there is no replication, there is no explicit estimate available). Therefore, for an estimate of error pool the second-order interactions with the highest-order interaction (i.e., assume that the second- and third-order interactions are all zero and hence the corresponding mean squares are all estimates of a 2 ). What effects do you consider to be statistically significant at the 0.05 level? For uniformity, (a) be a never-pooler (apart from that recommended in the previous paragraph); (b) in testing the first-order interactions, make due allowance for the fact that there are quite a large number of them; (c) on the other hand, be more generous to the main effects, and test them individually. Summarize in appropriate tables the effects you find statistically significant, and list the effects you consider nonsignificant.

528

THREE-WAY AND FOUR-WAY ANALYSIS OF VARIANCE

CHAP.

15

15.3. In a comparison of three different compositions of mortar, three lots of cement were supplied to each of five laboratories. Each laboratory using each lot of cement made up batches of mortar according to the three compositions. Two-inch cubes of mortar were cast and the compressive strength determined. The factor composition is a model I effect, and the factors cements and laboratories are model II effects. (a) Make a conventional analysis of variance of this data, presenting the results in standard tabular form, including the expectations of the mean squares. (b) Test the null hypotheses that the various main effects and interactions are zero. _(c) Give estimates of the components of variance corresponding to the various random terms in your model. (d) Give 95 per cent confidence limits for the difference between compositions 1 and 2. The compositions were such that it would be interesting to consider the following contrasts: (i) between compositions 1 and 2. (ii) between the mean of compositions 1 and 2 and composition 3. (e) Partition the main effect for composition into single degrees of freedom corresponding to these two contrasts. Test the null hypotheses that each contrast is zero. (f) Partition the laboratory x composition interaction into two components and test the null hypotheses that they are zero.

2

Cement Composition Lab 1

2 3 4 5

812 746 797 850 829

2

3

839 744 802 896 829

723 689 731 757 735

870 799 771 864 887

3

2

3

838 797 737 877 903

798 759 724 779 765

859 787 781 843 863

2

3

863 765 821 854 940

761 709 753 759 777

The sum of squares of all observations is 29,068,527. Source: Working Committee on Plastic Mortar Tests, "Report on Investigation of Mortars by Seven Laboratories," Proceedillgs of the Americall Society for Testillg alld Materia/s, 40 (1940), 210-25.

REFERENCES 1. Cornfield, Jerome, and John W. Tukey, "Average Values of Mean Squares in Factorials," Annals of Mathematical Statistics, 27 (1956), 907-949. 2. Paull, A. E., "On a Preliminary Test for Pooling Mean Squares in the Analysis of Variance," Annals of Mathematical Statistics, 21 (1950),539-56. 3. Finney, D. J., "The Joint Distribution of Variance Ratios Based on a Common Error Mean Square," Annals of Ewrellics, 11 (1941), 136--40. 4. Nair, K. R., "The Studentized Form of the Extreme Mean Square Test in the Analysis of Variance," Biometrika, 35 (1948), 16-31. 5. Daniel, C., "Fractional Replication in Industrial Research," Third Berkeley Symposium on Mathematical Statistics and Probability, vol. 5, 87-98. J. Neyman (ed.). Berkeley: University of California Press, 1956.

REFERENCES

529

6. Birnbaum, A., "On the Analysis of Factorial Experiments without Replication." Contributed paper at Annual Meeting of the Institute of Mathematical Statistics, Cambridge, Massachusetts, August 25-28,1958. 7. Barkworth, H., and J. o. Irwin, "Comparative Detection of Coliform Organisms in Milk and Water by the Presumptive Coliform Test," Joufllal of Hygielle, 43 (1943), 129-35. 8. Anderson, R. L., and T. A. Bancroft, Statistical Theory ill Research. New York: McGraw-Hill Book Co., 1952. 9. Fisher, R. A., and F. Yates, Statistical Tables for Biological, Agricultural, alld Medical Research. 3rd ed.; Edinburgh: Oliver and Boyd, 1948.

C HAP T E,R r6

Partially Hierarchical Situations

16.1. A Partially Hierarchical Situation and Its Model In an investigation of the can-making properties of tin plate, two methods of annealing were studied. Three coils were selected at random out of a supposedly infinite population of coils made by each of these two methods. From each coil, samples were taken from two particular and reproducible locations, namely, the head and tail of each coil. From each sample, two sets of cans were made up independently, and from each set an estimate of the can life was obtained: These are the data in Table 16.1, taken from Table 16.1 Annealing method i 1 Location

Duplicates

k

11

1(1)

2(1)

1 2 1 2

288 295 278 272

355 369 336 342

2

2

Coils within anneals j(i) 2(2) 3(1) 1(2)

329 343 320 315

310 282 288 287

303 321 302 297

3(2) 299 328 289 284

[1]. For a definitive study, one would require a substantially larger sample of coils, but the data of Table 16.1 will suffice to demonstrate the principles involved in the analysis. The structure of this experiment is rather different from what we have previously encountered. If the coils were crossed across annealing method, so that the first coil with annealing method 1 corresponded in some way with the first coil with annealing method 2, and the second coils likewise, 530

SECT.

16.1

HIERARCHICAL SITUATION AND ITS MODEL

531

etc., then we would have a three-way analysis with replication in the cells. However, this is not the case: The coils are not so crossed, but instead are nested within the annealing methods. Alternatively, if the locations were random samples from each coil, so that they were nested within the coils, with no crossing of location 1 across coils or annealing methods, then we would have a purely nested or hierarchical situation, with four classifications, namely, annealing methods, coils within annealing methods, locations within coils, and replications within locations. However, this is not the case: The locations are a model I effect crossed across coils and annealing methods. What we have is a partially hierarchical or crossed nested situation, which can be represented by the model x jjkv = ~

+ OCi + {c(oc)}j(j) + Ak + (AOC)ik + {AC(OC)}k;W + ZOkv'

(1.1)

in which ~ is the grand mean, OC i is the annealing method effect, {c(oc)}j(i) represents the random coil effect within annealing method, Ak represents the location effect, (AOC)ik represents the interaction of locations with annealing methods, {AC( oc) h-j(i) represents the interaction of locations with coils within anneals, and ZOkv is a random error distributed normally with zero mean and variance a 2 • In (1.1), single symbols could be used in place of {c(oc)}, (AOC), {AC(OC)}, but the use of these multiple symbols is helpful in identifying immediately the meaning of each. It is also helpful to use Greek letters to denote fixed, model I effects, e.g., oc and A, and Roman letters for random, model II effects, e.g., C and z, and this has been done in (1.1), somewhat prematurely, as we will first consider the finite population model and then move to this particular case. Thus for the finite population model, i goes to r in the sample and R in the population, j to t in the sample and T in the population, k to u in the sample and U in the population, and 11, which is from an infinite population, to n in the sample. There are various side restrictions on the model (1.1). First, R

LOCi

U

R

U

k

i

k

= 0, L Ak = 0, L (AOC);k = L (AOC)ik = O.

i

(1.2)

These conditions are similar to those for a two-way model, (14.8.2) and (14.8.3). Second, the coils-within-annealing-methods term {C(OC)}j(i) is a standard nested term similar to the Aj(i) of Section 14.10 and Table 14.15, i.e.; T

L; {c(oc)}J(;) = 0

for each i.

(1.3)

The complete array of {AC(OC)h-j(i) is indicated in Table 16.2. For each

532

PARTIALLY HIERARCHICAL SITUATIONS

CHAP.

16

value of i we have a two-way table of interaction constants with the usual property of two-way interaction constants of summing to zero in each row and in each column [see (14.8.3)], i.e., T

L:I {AC(cx)h:i(i) = 0

for each (k, i),

(1.4)

for each j(i).

(1.5)

U

L {AC(CX)h:i(i) = 0 k

However, for a fixed k, i.e., in Table 16.2 for a fixed row, the {AC(CX)}kj(i) do not sum to zero over i for a fixed j. In this respect they behave similarly to the A;/(i) of Table 14.15.

16.2. Calculation of Sums of Squares, Etc. The identity corresponding to the model (1.1) is X·tJft!V .•. -

x = (x.t... - x....) + (x't i .0 ••••

+ (x ..n;.•. - x.... ) + (x,.. - x" .0. - x.0";.•. + x....) + (x,.,... - x,':1 - x,," + XI -

X.t... )

Ai,

... 1\0.

••

,Ai.

...

)

(2.1) The terms on the right-hand side are the sample estimates of the terms on the right-hand side of the model (1.1), excluding the grand mean. The first and third terms need no comment. The second term is similar to an ordinary nested term such as the second term in (14.7.2). The fourth term is an ordinary two-way interaction resembling (15.1.11). The fifth term can be obtained by regarding it as the difference between xiik . and what would be predicted as the grand sum X.... plus the annealing method effect Xi ... - x.... plus the coils-within-annealing-method effect Xii .. - X.... plus the location effect x.. k • - X.... plus the location X anneal two-way interaction Xi .k . - Xi ... - x.. k . + x.... ; i.e., Xiik . -

[x ....

+ (Xi ...

- :1:... .>

+ (Xii .. -

+ (Xi.k. -

+ (x .. k. - x.... ) Xi ... - x.. k. + x.. J] = Xiik . - Xii .. - xu. + Xi.... Xi .. )

(2.2)

An alternative way of constructing partially hierarchical models is to consider them as degenerate cases of fully crossed models. For example, we suppose momentarily that the coil effect is fully crossed, so that there will be a coil main effect x. i .. - x.... and an anneal X coil interaction

gj

~ ..... 0\ N Table 16.2

i

=1

j(i) = 1(1) . .. j(i) = T(1)

k

=1

~

i =R

T

{AC(tX)}ll(1) ... {Ac(tX)hT(l)

()

j(i) = l(R) .. -j(i) = T(R)

I

c::

T

{Ac(tX)h;(1)

=0

{AC(tX)}U(R) ... {Ac(tX)hT(R)

I

{Ac(tX)h;(1)

=0

{AC( tX )}kl(R)

.I; {Ac(tX)h;(R) = 0

{AC(tX)}li(R)

=0

i

T

k=k

{AC( tX) hl(1)

{AC( tX )}kT(l)

I; T

k=U

{Ac(tX)}Ul!l)

{Ac(tX)}UT(l)

I

{AC(tX)}U;(1)

=0

T

{AC( tX )}kT(R)

{AC( tX) }Ul!R)

{Ac(tX)}UT(R)

U

U

I

{Ac(tX)hl!1)

k

=0

Ik {Ac(tX)hT(1)

I

~ ~ CI.I

c:: E::

CI.I

T

i

§

{AC(tX)}U;(R)

=0

~ CI.I

I:>

U

u

Ik {Ac(tX)hl!R) I

{Ac(tX)hT(R)

k

~

ttl

=0

=0

=0

~

Ul W

W

534

PARTIALLY HIERARCHICAL SITUATIONS

CHAP.

16

+ X..... Now we admit that the coil effect is not a main effect and pool it with its interaction with anneals:

Xii .. - Xi ... - X. i ..

(X.i.. - X.... )

+ (x.ti..

-

X.t... - X.i..

+ X.... ) = x.ti..

-

X.t ... '

(2.3)

which is the second term on the right-hand side of (2.1). Similarly, if coils was a crossed effect, then it would have an interaction with locations, and its interaction with anneals would also have an interaction with locations. But, since coils is not a crossed effect, these two interactions are pooled together: (X. ik . -

x. i ..

-

+ (Xiik. -

x.. k. + x... ,) Xii .. - Xi . k. - X. ik.

+ Xi ... + x. i .. + x..k • -

X... ,)

= xiik. - Xii .. - Xu. + Xi ... , (2.4) which is identical with (2.2). This viewpoint also applies to the degrees of freedom. For coils within anneals, each anneal contributes I - 1 degrees of freedom and there are /' anneals; so the degrees of freedom are /'(t - 1). But, taking the viewpoint of (2.3), the degrees of freedom should be (t - 1)

+ (/' -

1)(t -

1)

= /'(1 -

1),

(2.5)

which is the same result. For the interaction of locations x coils within anneals, since locations have u - 1 degrees of freedom and coils within anneals have /'(t - 1) degrees of freedom, their interaction will have /'(t - 1)(u - 1) degrees of freedom. From the viewpoint of (2.4), the degrees of freedom will be (I - 1)(u - 1)

+ (/' -

1)(u - 1)(1 - 1)

= /'(t

- 1)(u - 1),

(2.6)

which is the same result. Squaring and summing over all indices (2.1) gives an equation entered in the second column of Table 16.3. To calculate these sums of squares we form Table 16.4 by summing over p. We next sum over k to get the coils-within-annealing-method table. We also sum over j to get the locations X annealing method table. Summing this over i gives the location totals and over k gives the annealing method totals. The sum of squares for annealing methods is an ordinary main effect [see (15.1.13)]: Illit

Lir t(-;.x·. . -

2 2 -;)2 x -_ 3842 + 3590 .... 2 X 3X 2 2

X

(7432)2 _ 2646000 . . 3X 2 X 2 (2.7)

til

Source of variance

Annealing methods Coils within annealing methods Locations

Sums of squares

'1'

nut

1 (Xi ... -

1

t

'1'

r(t -

null(x"" ".. -x" I ••• )2 i

nrt

- x ... J2

u

11 (x" k

t. •

i

'1'

1)

k

t

- x"t... - X.. k.

+ X. ... )2

-

:-'l

0\

E[M. S.]

a2

a2

j

1" (x .. k. r

nt

+

(1 -~)(1 -~)na~c(a)

+

(

1- U

u -I

a2

+ (I

(r - I)(u - 1)

a2

+ n(1

u) na;'c(a) 2

(I -

N +t(1

-~)na~a

~) una~(a) + tuna!

2 + unaC(a)

-~)na~c(a) + (I -i)tn~a +rtna;. -

~)a~c(a) + tn~1X

(')

> r< (')

C

r<

~ 0

'"r1 til

c a::til 0

'"r1 til

t:J

C

> ::c m

U

nlll(x""k " . -x"" ",.. -X"k t. . +x"t ... )2 ; k i r

Within cells

r -

x ... )2

(')

+

k

Locations x annealing methods Locations x coils within annealing methods

m

Table 16.3 Degrees of freedom

r(t - l)(u - 1)

a2

+ na~c(a)

Y'

m o-j

$1

tun

1111 (Xi;kV i ; k

- Xi;k)2

rtu(n - 1)

a2

v

r

Total

tun

1111 (Xi;kV i ; k v

x ... )2

rtun -

1

VI

v.>

VI

n

v

LLxiikV k

It

1133

1402

678

550

k =2

Location, k

724

2(1)

583

1(1)

=1

Coils within annealing methodj(i)

Annealing method i

635

672

3(1)

t

X

v

n

1863

1979

v

n

LLxlikV i

t

1167

575

592

1(2)

1223

599

624

2(2)

v

j

t

v

Lk L L x2ikv = 3590

"

n

1747

573

v

1843

1200

r

t

It

v

n

3610

3822

j

= 7432

v

LLLLxiikV ik

r

i

t n

Sum

LLx2 i i1CV LLLxmcv

t n

627

3(2)

i=2

P,

xUkv of Table 16.1 Summed over L xi;kV

Lk Li L likV = 3842

It

n

=1

1307

i

Table 16.4. The

0\

.-

:c

~

'"

z

g

>

'" ::; c:::

t""

n >

:x::

(')

> ::d

m ::=

:x::

~

t""

;;

> ~

"C

0\

VI W

SECT.

16.2

537

CALCULATION OF SUMS OF SQUARES, ETC.

The sum of squares for coils within anneals is an ordinary nested effect: t

T

IlU

~ ~ (Xii" - xi .. l :I

i

1

t ("~~Xi:lk')2

T

11

= -UII ~~ i:l

~.

v

1133 2 + 1402 2 + ... 2x2

= = 9701.333.

1

T

tl/II

i

(t

u"

i

k v

-~ ~~~XijkV

-

+ 12002

)2

38422 + 35902 3x2x2 (2.8)

The sum of squares for locations is an ordinary main effect [(see (15.1.13)]: IIrt

~ (x . _

X )2

f ..k.

= 38222 + 36102 _ 2 x 3 x 2

.•..

= 1872.667.

(7432)2 2 x 3 x 2 X 2

(2.9) The sum of squares for locations x annealing methods is an ordinary twoway interaction [see (15.1.14)]: T

II

lit'" '" (x•.. k.. - x.•..• - x ..k.. kk k

i

=

1979 2

+ X•.•• )2

+ ... + 17472

_~(7:....:.4=-=32:L)2_ _ 2646.000 _ 1872.667 2x3x2x2

2 x 3

= 16.667.

(2.10)

The sum of squares for locations x coils within annealing methods can be calculated from the identity ,. II

t

II

~ ~ ~ (Xi:l k • i

:I

xi:l .. -

-

Xi .k •

k

r

=

II

-

u

r

~ ~ ~ (xiik . Ill/

x.. J 2 -

-

k

i

i

-

t

+ Xi.J 2

r

t

i

:I

~ ~ (Xii .. - xi .. l r

"

j

k

lIut

~ (Xi ... - X... l i

-

" IIrt ~ (X .. k. -

X...

Y

k

x.•... - x..k.. + X••.• )2

lit "''''(X' k k k •..

(2.11)

where

j

:I

k

-)2 x ••.. -_

t

"

("

~~~ ~ T

,. t " '" (-Xi:lk. n'" k'" kk

i

:I

k

•

11

Xi:l k •

)2

(

r

t

"n

~~~~Xi:lk' • , k •

)2

rtl/l1

(2.12)

538

PARTIALLY HIERARCHICAL SITUATIONS

CHAP.

16

Numerically this is (583 2

+ ... + 573 2) 2 2x3x2x2 - 2646.000 - 9701.333 - 1872.667 - 16.667 = 211.667. (2.13)

The sum of squares within cells is (288 2

+ ... + 2842)

_

583 2

+ ... + 573 2 = 1269.000.

(2.14)

2

These sums of squares, and the corresponding degrees of freedom, are entered in Table 16.5. In the next section we shall discuss obtaining the expected values of the mean squares. Table 16.5 Source of variance Annealing methods Coils within annealing methods Locations Locations X annealing methods Locations X coils within annealing methods Within cells Total

Sums of squares

Degrees of freedom

2,646.000 9,701.333 1,872.667

Mean squares 2646.000

4

16.667

2425.333 1872.667 16.667

211.667

4

52.917

1,269.000

12

105.750

15,717.333

23

EIM.S.)

+ 1II1a:(", + fIIna~ aD + 1II1a:(", a + na!.(", + rlllal a + nal.(", + lIIa1" a + lIa1.(", aD

9

2

2

a2

16.3. The Expectations of Mean Squares in Partially Hierarchical Models Bennett and Franklin [2] have given a procedure for obtaining the expected values of mean squares in partially hierarchical situations. Of course, wholly hierarchical and wholly crossed models are special cases. The procedure is to construct a two-way table, columns corresponding to the indices used in the model, i, j, k, 11 in the present case, rows corresponding to the terms in the model. This has been done for the model (1.1) in Table 16.6. The numbers of elements in the sample and in the population are entered in parentheses. The orders of the rows and the columns are not important other than that a systematic order of some kind helps to avoid mistakes. We now consider any row. For those columns whose indices are not in the suffix for the term defining that row, enter the number of elements in

SECT.

16.3

539

THE EXPECTATIONS OF MEAN SQUARES

the sample. For example, the j, k, and 11 indices do not appear in the suffix to ex i ; hence we enter t, U, and n in the corresponding columns opposite ex i • Table 16.6

zi:lk.

{.k(ex)h:lCf) (Aex)ik Ak

{c( ex)}:lCi) exi

i

j

k

(r, R)

(I, T)

(II, U)

1 1

1 1 - tIT

1 1 - II/U 1 - II/U 1 - u/U

1 - r/R r

1

1

1 - tIT

1 - r/R

u u

v n

n n n n n

Next, if any term is an unrestricted random variable, enter 1 in the cells left vacant after the previous stage. Here Ziik. is the only restricted random variable, and 1's are entered in all columns. Next, if any term contains a suffix inside parentheses, enter 1 in the column corresponding to the index inside the parentheses. Here {c(ex)h(i) and {Ac(ex)h:lCi) each have i inside parentheses; so 1 goes in the i column opposite these terms. Finally, wherever a cell is still empty after the preceding operations, enter 1 - clC, where c and C are the number of elements in the sample and in the population. This last step will fill up all the cells in the table. The expectation of a mean square will include all components of variance corresponding to those terms in the model which include in their suffices the index or indices which are in the suffix of the term in question. The expectation of the mean square for locations, e.g., will, since the location term is Ak, include the components of variance corresponding to Ak, (Aex)ik' {AC(ex)}kiCi)' and Ziik, namely (1~, (1~a' (1~c(al' and (12. Each component of variance has as coefficient the product of all the numbers in the row for the term corresponding to that component of variance excluding the number or numbers in the column or columns corresponding to the index or indices of the mean square in question. For example, for the expected value of the mean square for locations, (1~ will have the coefficient rln; (1~a the coefficient (1 - 1'1 R)ln; (1~cca) the coefficient (1 - tIT)n; and (12 the coefficient 1. The expected value for the location mean square is thus

540

PARTIALLY HIERARCHICAL SITUATIONS

CHAP.

16

This is entered in the last column of Table 16.3, and the other entries in that column are obtained similarly. In the specific example we have been considering, the annealing methods and the locations were fixed effects; so the sample contained the entire population, and hence 1 - I'/R = 1 - u/U = O. On the other hand, the coils were samples from infinite populations; so 1 - tiT = 1. Inserting these values in the expectations of the mean squares in Table 16.3, we get the particular results in Table 16.5. The appropriate error term for testing each mean square is now obvious. 16.4. Confidence Limits in Partially Hierarchical Models Calculation of confidence limits in a partially hierarchical model requires some care. They are derived from the model (Ll). xi;kv = ~

+ rxi + {c(rx)}iW + Ak + (Arx)ik + {Ac(rx)hiW + ziikv'

(4.1)

Two of the terms need particular attention. The term {c(rx)};w is a standard nested term and by (1.3) sums to zero in the population T. However, T = 00, so we make the further assumption of normality: (4.2)

Thus {c(rx)}.w

I

t

t

J

= - 'L {c(rx)}i!i) "'" N(O, (J~(a.)/t).

(4.3)

The term {Ac(rx)hJ(i) is more troublesome. As stated in (1.4) and (1.5), U

T

'Lk {Ac(rx) hili) = 'Li {AC(rx)}kJ(i) =

0;

(4.4)

i.e., it sums to zero in the population over j and k. However, the sample of locations u includes the entire population U; i.e., u = U; so -{AC(rx)}.J(i}

= -I 'L {Ac(rx)hJ(i} = -I 'L {Ac(rx)hiW = O. U

U

U k

(4.5)

U k

Since this is zero, averaging over another suffix will leave it zero: {AC(rx)}..(i)

= O.

However, the sample of coils t is from an infinite population, T = {AC( rx) h.w "'" N

(0, (J~;(a.»),

(4.6) 00;

so

(4.7)

SECT.

16.4

541

LIMITS IN PARTIALLY HIERARCHICAL MODELS

and, if we further average over i, {Ac(ex)h.(.),....,

N(0, O'~c:a»).

(4.8)

We will now compute the variance of the difference between two annealing methods averaged over locations and of course over coils. We would only be looking at the annealing main effect if we assume that its interaction with the other fixed effect, locations, (Aex)ik' is zero. From the model (4.1)

xi ... = ~ + exi + {C(ex)}.(i) + X. + {AC(ex)}.. (i) + Zi....

(4.9)

However, by (4.6), {AC(ex)}..(i) = 0; so Xi ... - Xi' ...

= (exi

-

+ {c(ex)}.(i) -

exi ,)

{C(ex)}.(i')

+ Zi ... -

Zi' ... ,

(4.10) (4.11) (4.12)

V[X.

....

_

x., ] = 2. (M. S. for coils within) • ...

tun

annealing methods

.

(4.13)

We next compute the variance of the difference between two locations, averaged over annealing methods and coils, and again assuming (Aex)ik = 0:

+ iX. + {c(ex)L.) + Ak + {AC(ex)h,(.) + Z.. k., x.. k'. = (Ak - Ak,) + {Ac(ex)h.(.) - {Ac(ex)h,.(.) + Z.. k. X.. k. = ~

x.. k. -

(4.14)

Z.. k'., (4.15)

E[x ..k. - x ..k,J

-

= Ak -

Ak"

_ = 2 (O'~c(a) 0'2 ) 2 -- + = -2 (0'2 + nO';'c(a»)' rt rtn rtn

V[X .. k. - X..k,J

V['

x .. k •

_ -

x .. k '.

] _ 2 (M. S. for locations x COilS) -

rt1l within annealing methods .

(4.16) (4.17)

(4.18)

If we are unwilling to assume that the location x annealing method interaction is zero, i.e., we admit the possibility that (Aex)ik ¥: 0 in general, then it makes little sense to compare annealing methods averaged over locations or to compare locations averaged over annealing methods. We would rather compare annealing methods for a particular location and locations for a particular annealing method. To compute the variance of

542

PARTIALLY HIERARCHICAL SITUATIONS

CHAP.

16

the difference between two locations for the same annealing method we start with so

+ {AC(OC)h,(i)

+ Zi.k. -

- {Ac(oc)h,.(i)

Zi.k'.. (4.20)

Thus _

_

V[Xi.k. - Xi.k,J

O'2) 2 = 2 (O'~c(<<) -- + - = -

t

tn

-] - ~

"V[X i .k • -

Xi .k '.

-

nt

(O'

2 + nO').c(<<»'

2

nt

(M.within S. forannealing locations COilS) methods' X

(4.21) (422) .

Now we compute the variance of the difference between two annealing methods for the same location:

= (oci

Xi .k. - Xi'.k.

oci')

-

+ {c(oc)}.(;) -

+ {AC(OC)}k.(i) _

_

V[Xi .k. - Xi'.kJ

{Ac(OC)h.(i')

O'~c(<<) = 2 (O'~(<<) + -+ -O'2) = -2 t

t

+ (AOC)ik - (AOC)i'k + Zi.k. - Zi'.k., (4.23)

{C(OC)}.(i')

tn

nt

[O'

2

2 2 + l1O').c(<<) + nO'c(<<)]'

(4.24) The analysis of variance table does not provide any mean square with the appropriate expected value. If, as is often the case, the term {AC(OC)} is omitted from the model, we have to construct a linear combination of mean squares with expected value O'2 + nO'~(<<). This is given by

~ [(u -

u

1)(M. S. for remainder)

+ M. S. for coils within annealing methods],

(4.25)

which has the expected value

=!

[(1I -

I)O'2

+ O'2 + tlnO'~(<<)] = O'2 + IlO'~(<<).

(4.26)

1/

The omission of {AC(OC)} from the model has the end result of pooling the sums of squares and degrees of freedom of this term with those for within cells to form the "remainder" sum of squares. If we are unwilling to make the foregoing simplifying assumption, we can obtain a linear combination

543

EXERCISES

with the desired expected value included in the brackets in (4.24) by using

I - [-(M. S. for within cells)

u

+ u(M. S. for locations X coils within annealing methods) + (M. S. for coils within anneals)], (4.27)

which has the expected value 2 + u2 -I( -u 2+ uu 2+ unul.c(,,)

u

2 + + unuc2) (,,) = u 2+ nul.c(,,)

2 nuc(,,). (4.28)

Finally, the variance of the difference between the combination of the ith annealing method with the kth location and the combination of the i'th annealing method with the k'th location is the same as that just discussed, the difference between two annealing methods for the same location.

EXERCISES 16.1. Five laboratories cooperated in measuring the brightness of six lamps of each of two types. The lamps were sent from laboratory to laboratory for measurement. See p'. 544. (a) Make an appropriate analysis of variance of these data, including giving the E[M.S.J. We are particularly interested in testing the main effect of laboratories and the laboratory x type-of-Iamp interaction. (b) Construct 95 per cent confidence limits for the difference between types (for purposes of this question assume that laboratory x type-of-Iamp interaction is zero). (c) Construct 95 per cent confidence limits for the difference between laboratory A and laboratory E (again assuming laboratory x type-of-Iamp interaction to be zero). (d) Construct 95 per cent confidence limits for the difference between type 1 in laboratory A and type 2 in laboratory B. (e) Estimate the component of variance for lamps within types. Actually the lamps initially were measured in laboratory A (the Bureau of Standards), were then sent to laboratory B (National Physical Laboratory) who sent them on to laboratory C (Laboratoire Centrale), who returned them to the National Physical Laboratory (denoted this time by D), who returned them to the Bureau of Standards (denoted this time by E). Let the Bureau of Standards and the National Physical Laboratory on the second time around, E and D, be denoted by A' and B'. Then the following comparisons seem to be of particular interest: (i) Early measurements versus late measurements: i.e., (A + B) - (A" + B'). (ii) Change with time in difference between A and B, i.e. (A - B) - (A' - B'). (iii) Difference between (B + B') and C: i.e., (B + B') - 2G.

544

PARTIALLY HIERARCHICAL SITUATIONS

CHAP.

16

(iv) Difference between (mean of B + B' and C) and (mean of A and A'), i.e.,2(B + B' + C) - 3(A + A'). (f) Construct contrasts to provide the above comparisons. (g) Are these contrasts orthogonal? (h) Partition the sums of squares for main effect for laboratories and for the interaction laboratory x type into components corresponding to these contrasts. Values of Candlepower Obtained at Different Laboratories All figures have been multiplied by 100 and then 1000 subtracted from them Laboratory Type

Lamp

A

B

C

D

E

Totals

I

1 2 3 4 5 6

741 731 731 759 738 770

768 763 763 779 758 795

770 755 757 775 750 800

772 742 760 774 750 800

738 724 728 752 730 768

3,789 3,715 3,739 3,839 3,726 3,933

Totals

4470

4626

4607

4598

4440

22,741

1 2 3 4 5 6

625 590 602 578 578 625

650 611 630 607 604 673

655 605 640 640 605 670

651 625 630 608 608 664

615 588 605 581 573 631

3,196 3,019 3,107 3,014 2,968 3,263

Totals

3598

3775

3815

3786

3593

18,567

Totals

8068

8401

8422

8384

8033

41,308

II

Source: Edward P. Hyde, "A Comparison of the Unit of Luminous Intensity of the United States with Those of Germany, England, and France," Bulletin of the Bureau of Standards, 3(1907), 68-80. 16.2. Five laboratories cooperated in a comparison of their testing procedures for impact strength of a type of fiberboard. Panels from two batches of board were sent to the five laboratories. Each laboratory tested each batch in duplicate on three days. There is no correspondence between the three days in one laboratory and the three days in any other laboratory. For notational consistency let i = 1, ... , r (= 5) refer to laboratories J. (model I), j = 1, ... , t (= 2) refer to batches {3 (model I), k = 1, ... , II (= 3) refer to days d (modellI), 111 = 1, ... , v (= 2) refer to duplications of a lot within a day (model II).

545

EXERCISES

The impact strengths reported were: Laboratory Batch

Day

1 2

2 2 1 3 2

A

B

C

D

E

1483 1496 1504 1505

1449 1400 1465 1423

1499 1472 1506 1537

1428 1401 1407 1416

1509 1439 1480 1429

1441 1416 1477 1457

1477 1471 1418 1445

1483 1509 1578 1486

1404 1419 1455 1435

1416 1441 1364 1441

1450 1478 1435 1478

1446 1398 1424 1426

1489 1435 1499 1491

1414 1446 1423 1442

1419 1444 1437 1438

(a) Write down an appropriate linear model, and make an analysis of variance of these data, including the expected values of the mean squares. Make the appropriate tests of significance. (b) Give simple 95 per cent confidence limits for the difference between laboratories Band C. (c) Estimate the within-day component of variance averaged over laboratories. (d) Estimate the between-day component of variance averaged over laboratories. (e) A batch measured repeatedly by laboratory A had an average impact strength of 1420. If a single test is made on this batch in laboratory D, estimate the probability of this batch being rejected by this laboratory, if the specification is to reject if the observed impact strength is less than 1400. If we subtract 1000 from every figure in the table, two sums of squares which may be useful are: r

t

It

~~~ i j

k

V

L X~jkm 111

= 12,414,257,

rtlt(v

I~~ ~Xijkm I·

J

k

)2 = 24,786,137.

'"

16.3. Section 16.1 discusses an investigation of the can-making properties of tin plate. In that discussion u particular locations from each coil were used, so location was a model I effect. The same locations were used for all coils. Consider the following modifications. (a) Instead of the two replicate measurements being obtained by randomization over the rtu samples of tin plate, suppose that one replicate was made immediately and the other 6 months later. Thus in place of duplicates we have a model I factor, age, say y, at II levels.

546

PARTIALLY HIERARCHICAL SITUATIONS

CHAP.

16

(b) Instead of II particular locations (model I), a random sample of II locations, say 11> ••• , lu, would be selected, so that locations would be a model II effect. (c) Instead of the single random sample oflocations 11> ••• , lu being taken for all coils for all annealing methods, as in (b), one random sample of locations is taken for the first annealing method, a second random sample of locations is taken for the second annealing method, and so on. All coils for a particular annealing method are sampled at the same locations. (d) Instead of a single random sample of locations serving for all coils for a particular annealing method, as in (c), a separate random sample of locations is taken for every coil. For each of these situations, using as far as possible the sample terminology and notation, but where necessary making appropriate changes, list the sources of variance for an analysis of variance table with the corresponding degrees of freedom and the expected values of the mean squares. State the error terms appropriate for testing the null hypotheses that there are no differences between annealing methods and between locations.

REFERENCES 1. Vaurio, V. W., and C. Daniel, "Evaluation of Several Sets of Constants and Several

Sources of Variability," Chemical Engineering Progress, 50(1954), 81-86. 2. Bennett, Carl A., and Norman L. Franklin, Statistical Analysis ill Chemistry and the Chemical Industl)l. New York: John Wiley and Sons, 1954.

CHAPTE R I7

Some Simple Experimental Designs

17.1. Completely Randomized Designs We have in previous chapters developed techniques suitable for analyzing the simpler experimental designs. The simplest experiment perhaps is to compare I' levels of a single experimental factor: We may be comparing three brands of gasoline, or four thicknesses of shoe leather, or five quantities of supplement to a hog ration. If we make IIi independent.observations on each level of the factor, in a completely random order, the model for the experiment will be

and the appropriate analysis will be model lone-way analysis of variance as discussed in Section 10.2. Usually it will be preferable to have the IIi all equal, since then the variances of the differences between any pair will be the same, and also since Tukey's method of multiple comparisons, which requires equal IIi and is more efficient than Scheffe's method for simple comparisons, can then be applied. However, if our objective is to compare all the other levels of the factor with one particular level, the "control," as can be done effectively with Dunnett's technique, it is advantageous to have the IIi for the control equal to .J"k times the 11; for the other levels, where k is the number of other levels. The next simplest experiment is to investigate two factors simultaneously in all combinations with 11 independent replicates per combination. For example, one factor could be brand of gasoline at I' levels, and the other mean speed of automobile at t levels. Or one factor could be quantity of supplement to the hog ration and the other factor the breed of hog. The total number of observations required is rIll, and these should be obtained in a completely random order. The appropriate analysis will be two-way 547

548

SOME SIMPLE EXPERIMENTAL DESIGNS

CHAP.

17

analysis of variance as discussed in Chapter 14. The model may be model I, or II, or mixed, depending on the nature of the factors. For example, in the hog example just given, the first factor, the quantity of supplement, is a fixed or model I factor, and the second factor, breed of hog, would be a model I factor if we were interested only in these particular breeds, but a model II factor if we were interested in generalizing to a larger population of breeds. Table 17.1 Treatments Random numbers Ordering

11

12

13

14

15 77 01 64 3 13 2 10

15

21

22 23

24 25

69(2) 11

69(8) 12

58 40 81 8 7 14

31

32 33

34 35

16 60 20 00 84 22 4 9 5 1 15 6

In discussing the principles of experimental design, it is convenient to have a general word for the basic experimental unit that gives rise to the measurement which we analyze. In the preceding paragraph, a given trip with a particular automobile using a particular gasoline will give rise to a single determination of gasoline consumption and is the basic experimental unit. In the hog feeding example, the individual hog is the experimental unit. The great bulk of the theory of experimental design was developed in agronomy, in which the basic experimental unit is a plot of ground on which is grown a crop and to which is applied fertilizers, etc. It is convenient to use the word plot for basic experimental unit in general. Randomization of treatments on to plots is best performed with a table of random numbers (Table Xl). Suppose that we have an experiment involving five replicates of three treatments. We can denote the jth replicate of the ith treatment as ij. We write out in any order, systematic if convenient, these 3 X 5 = 15 treatments (Table 17.1, first row). Under each treatment we enter a two-digit random number from Table XI. We then order in increasing magnitude these random numbers. The particular set we chose happens to have a tie, two 69's. This tie is resolved by picking two further random digits and attaching one to each of the ties. The next two-digit random number is 28, and so the first 69 is regarded as 69.2 and the second as 69.8. The third row gives the ordering of the random numbers. Then plot 1 receives treatment (33), plot 2 receives treatment (13), etc. 17.2. Randomized-Block Designs If we are studying a single factor at t levels, with,. replicates on each level, it may be possible to arrange the observations in ,. blocks of t observations. In agricultural experimentation each observation of, say,

SECT.

17.2

549

RANDOMIZED-BLOCK DESIGNS

yield, comes from a plot of ground, and we may group t adjacent plots to form a block. In executing the experiment, we randomly allocate the t levels to the t plots in the first block, repeat the randomization for the second block, and so on. This is quite different from the completely 'lndomized experiment where there was a single randomization of rn ;reatments, actually /' treatments repeated n times, on to rn plots. Usually the blocks will be regarded as a random or model II factor, and the factor under study will be a model I factor. The appropriate analysis will be a mixed model with one observation per cell. A suitable modification of the model (14.8.1) to this circumstance would be (2.1)

and Table 14.14 with no within-cell term and n = 1 becomes Table 17.2. The remainder mean square is an appropriate error term for testing the effect of the factor and for constructing confidence limits for differences. The test of the block effect is only satisfactory if it can be assumed that is small compared to 0'2. This limitation is usually of small import as our main objective is to study the factor.

0':

Source of variance Blocks Factor Remainder

Table 17.2 Mean square Type of effect S2 4 S2 3 S2

II I

2

E[M.S.] a 2 + ta~ a2+a:+ra~ a 2 + a:

The objective of using a randomized-block design instead of a completely randomized arrangement is to reduce experimental error. Adjacent plots clustered together in a block should be more alike in their response to the same treatment than plots at opposite ends of the field. Let O'~b and O'~r be the error terms in the two forms. We define the more efficient design, the randomized blocks, as the standard, and the efficiency of the other is defined as 1=

2 O'Ob.

O'~r

(2.2)

This is a reasonable definition of efficiency, for to get the same accuracy on the comparison of treatment means we would have to use 1// times as many replicates with the less efficient design; e.g., if the efficiency is 0.50, we need 1/0.5 = 2 times as many observations with the inefficient design compared with the efficient design to get the same accuracy.

550

SOME SIMPLE EXPERIMENTAL DESIGNS

CHAP.

17

Suppose that we have t treatments and l' blocks, or replicates. Let E be the error mean square and B be the block mean square, and let F = BIE. Then it can be shown. that the estimated efficiency of the completely randomized design relative to the randomized-block design is

1=

A2 aOb

a~r

E

= [(I' - l)B

+

I'(t - l)E]/(1'1 - 1)

_

I' t - 1 F(I' - 1) + r(t - 1) (2.3)

As we would expect, the larger the F, the lower is the estimated relative efficiency of the completely randomized design. If we have two factors, say Band C, at t and u levels, and /' blocks containing tu plots, we can randomly allocate the tu treatment combinations to the plots in the first block, repeat the process for the second block, etc. If the blocks are regarded as a random effect and the two factors are fixed effects, the appropriate analysis is similar to that discussed in Section 15.3, and the expected values of the mean squares are like those given in the left-hand side of Table 15.3, where a! corresponds to blocks, with the modification that n = I and there is no within-cells sum of squares. The appropriate error term for the factor B is its interaction with blocks, and analogously for C, and the two-way interaction B X C is to be tested against the remainder mean square. Actually, many practitioners of the art pool the sums of squares and degrees of freedom for all the interactions with blocks, namely aB, aC, and aBC, and use this pooled mean square as an error term. This procedure has implicit in it the assumption that a!B = a!o = a!BO = O. It is not clear in what fields of experimentation such a set of assumptions is or is not valid. 17.3. The Split-Plot Situation

Suppose that we have two factors, say B corresponding to varieties of potato and C to quantity of fertilizer. We might plan the experiment as in the previous section, in l' randomized blocks, each block containing tu plots. Alternatively, suppose we group these tu plots into t groups of u plots, and now change the terminology so that the ultimate unit is a subplot and the groups of u su bplots are whole plots, the block containing t such whole plots. We now design the experiment as follows. We randomly allocate the t levels of factor B (variety of potato) to the t whole plots in the first block, repeat this procedure for the second block, etc. We now randomly allocate the u levels of factor C (quantity of fertilizer) to the u subplots contained

SECT.

17.3

551

THE SPLIT-PLOT SITUATION

in the first whole plot in the first block, repeat this procedure for the second whole plot, etc. If there are three varieties and four fertilizers, the first two blocks might have the arrangement shown in Table 17.3. Table 17.3 va!4

Va!2

VI/I

VI/a

V2/2

V2!1

vaft

va!a

VI/2

VI/4

V2!4

vda

VI/I

I VI/a

Vall

Va!4

V2!2

V2!1

VI/4 1 VI/2

va/2

va!a

V2!a

l'2!4

Block I

Block II

Such an arrangement is known as a split-plot design, here in randomized blocks. We would be motivated to use it in the present example if, on the one hand, it was technically inconvenient to plant the varieties in small subplots, switching from variety to variety as we cover the four subplots in a whole plot, whereas on the other hand it was not troublesome to switch from fertilizer to fertilizer going from one subplot to another in the same whole plot. The essential feature of a split-plot design is that instead of the rtu ultimate experimental units being obtained after randomization over the whole number of rlu units, as in a completely randomized design, or being obtained after r separate randomizations over tu units, as in a simple randomized block design, they are obtained by first randomizing the treatments C on to the u subplots, this randomization being performed rt times, and then randomizing the treatments B on to the t whole plots, this randomization being performed r times, once for each of the r blocks. An appropriate model for the experiment of Table 17.3 is x iik

= ~ + bi + "Pi + em) + CPk + (CP"P)jk + (bcp)ik + Zijk'

(3.1)

In this model, bi is the random-block effect, "Pi is the fixed variety effect, CPk is the fixed fertilizer effect, (CP"P)ik is their interaction, and (bCP)ik is the interaction of blocks with fertilizer. The motivation for writing ej(i) in this form is that if the experiment had been run with every whole plot receiving identical varieties and every subplot receiving identical fertilizers, then we would have a simple nested situation. The'random variation of whole plots within the blocks is represented by the unrestricted error term ei(i)' and the random variation of the subplots within the whole plots

552

SOME SIMPLE EXPERIMENTAL DESIGNS

CHAP.

17

is represented by the unrestricted error term Zi;k' Let e and Z have variances w 2 and 0'2, respectively. To obtain the expected values of the mean squares in the analysis of variance we follow the procedure of Section 16.3 and set up Table 17.4. Table 17.4 j

Z;;k (b,P)ik (4)''');k 4>k

(r, R)

(t, T)

1 1 - r/R

I

r r

1 - I/T

e;(i)

1

'tp;

r

1 1 - I/T

bi

1 - r/R

I

k (II,

V)

1 1 - II/V 1 - II/V 1 - II/V II II II

In general, in split-plot experiments the whole-plot treatments may be either model I or model II, and likewise the subplot treatments, though it is more usual for them both to be model I, as in the present case. Thus here R = 00, so 1 - I'/R = 1, and t = T and u = V, so 1 - tiT = 1 - u/V = O. We can now write down the expectations of the mean squares in Table 17.5. It is a matter of opinion in any particular instance Table 17.5

Source of variance

Degrees of freedom

Blocks Varieties Whole-plot error Fertilizers Varieties x fertilizers Blocks x fertilizers Subplot error

r - 1 1-1 (r - 1)(1 /I-I (I - 1)(u (I' - 1)(11 (r. - 1)(1

Total

I'lu - 1

- 1) - 1) - 1) - 1)(u - 1)

E[M.S.] a2 a2 a2 a2 a2 a2 a2

+ IIW2 + tlla~ + uw2 + rlla4>2 + IIW 2 + ta2b4> + rta4>2 + ra:Ip + ta blp2

whether the term (br/»ik should or should not be included in the model. It is perhaps more usual to exclude it. In that case its sum of squares and degrees of freedom are pooled with subplot error to form a new estimate of 0'2 based on t(1' - 1)(k - 1) degrees of freedom, and of course to'~ gets stricken out of the expectation of the fertilizer mean square.

SECT.

17.4

553

RELATIONSHIP OF SPLIT-PLOT TO HIERARCHICAL SITUATIONS

17.4. Relationship of Split-Plot to Partially Hierarchical Situations

In our discussion of the split-plot experiment of the preceding section, there was nothing essential to the split-plot concept that the experiment be run in randomized blocks. As far as the whole-plot part of the experiment is concerned, it might just as well be run as r completely randomized replicates of the t whole-plot treatments. If we omit the block effects from the model (3.1), we get (4.1) If we now refer back to the model (16.1.1) for the partially hierarchical situation discussed in Section 16.1, and modify it to conform to the situation where there is only one replicate per cell by omitting the suffix " and the final error term Zijk. and by substituting ZUk for {Ac(oc)hi(i)' we get XUk = ~ + OCi + {C(oc)}iW + Ak + (AOC)ik + Ziik> (4.2) which is essentially identical with (4.1). We therefore see that the split-plot situation is an agricultural example of a partially hierarchical classification, and the two can be considered together. We can use the results of Section 16.4 on variances of various types of differences, with the afore-mentioned differences. Thus the analog of (16.4.13) becomes V[X. i .

-

x. i ',] = ~ (M.S. for whole-plot error).

(4.3)

1'U

Since, with" becomes V[X ..k

-

=

1, the {AC(OC)}ki{i) has been replaced by ziik, (16.4.18)

x.. k '] = ~

(M.S. for subplot error),

(4.4)

= ~ (M.S. for subplot error).

(4.5)

1't

and (16.4.22) becomes V[X. ik - X.ik']

r

From (16.4.24) and (16.4.25), V[X. ik - X.J'k] =

l. [(u ru

- l)(M.S. for subplot error)

+ (M.S. for whole-plot error)].

(4.6)

554

SOME SIMPLE EXPERIMENTAL DESIGNS

CHAP.

17

EXERCISES 17.1. Suppose that in a single experiment k treatments are to be compared only with a control treatment, and are not to be compared with each other. Suppose that no replications are to be run on the control and n t replicates are to be run on each of the k other treatments. The total number of replicates N = no + kn t is fixed. Show that the choice of no and n t such that no = vkn t makes V[xo' - x/.], averaged over i, a minimum. Assume no and nt to be large enough to be regarded as continuous variables. 17.1. An experiment was run to compare three similar magnesium salts, A, B, C, in the production of an antibiotic by fermentation. In the first replication, three fermentations were started, one containing salt A, another salt B, and the third salt C. After five days, samples for analysis were withdrawn from each fermentation, and likewise after six days. The whole operation was repeated a total of four times. The replications should be regarded as blocks. (a) Make an appropriate analysis of variance for these data, and report the F values, with the corresponding degrees of freedom, for the main effects of salt, age (five days versus six), and their interaction. (b) Give 95 per cent confidence limits for (i) salt A - salt B, (ii) six days - five days, (iii) salt A - salt B, both at six days. Magnesium salt A

C

B

Replication

5 days

6 days

5 days

6 days

5 days

6 days

1 2 3 4

69 82 67 69

84 78 74 77

91 75 78 85

98 82 92 92

81 72 66 73

86 77

79 81

17.3. A comparison was made of the effects of two levels of temper, half and quarter, on the shearing modulus of elasticity of a certain stainless steel. From each of five random lots of this type of steel three tubes were chosen at random. Each tube was cut in half, and one portion given the quarter-temper treatment and the other portion the half-temper treatment. 1(1)

Sample, j(i)

3(i)

2(i)

Temper, k

k=1

k=2

k=1

k=2

k=1

k=2

Lot, i = 1 i= 2 i= 3 i= 4

1073 1022 942 1011 1044

1024 962 900 1000 1010

1026 1009 927 1058 1027

990 1005 887 1006 1010

996 1150 942 1038 1035

972 985 885 1022 1004

i= 5

Sum of squares of all observations = 30,013,864. Source: C. W. Muhlenbruch, V. N. Krivobok, and C. R. Mayne, "Mechanical Properties in Torsion and Poisson's Ratio for Certain Stainless Steel Alloys," Proceedillgs oftile Americall Society for Testillg alld Materials, 51 (1951), 837-52.

EXERCISES

555

(a) Write down an appropriate linear model and compute the corresponding analysis of variance. Give the expectations of the various mean squares. (b) Test the null hypothesis that the difference between tempering treatments is constant over lots. (c) Give a 95 per cent confidence limit of the difference between the two tempering treatments. (d) ESjimate the various components of variance. 17.4. An experiment is run to compare r treatments T;, i = 1, ... ,r in t randomized blocks b;, j = 1, ... ,t. The treatments are randomized on to the plots in each block. Observations are made at the end of 1 year. The same treatments are used on the same plots for a total of II years Ykt k = 1, ... , II, and observations made each year. Thus no further randomization beyond the initial one is used. It is to be assumed, however, that the sequence of II years is in some sense a random sample of all years. The treatments are to be regarded as a model I effect, and the blocks as a model II effect. (a) Write down a suitable linear model for this experiment. (b) Present a table listing the names of the mean squares you would compute, with the corresponding numbers of degrees of freedom and the expectations of the mean squares. (c) Indicate how you would test the main effect for treatments. 17.5. Suppose that the experiment in exercise (17.4) was replicated each year at several places. Let the index on the place symbol be m, m = 1, ... ,v. Suppose that the places are (i) a random sample from an infinite population, and (ii) the only places of interest. (a) Write down suitable linear models for these two situations, and (b) present a table listing the names of the mean squares you would compute, with the corresponding degrees of freedom and the expectations of the mean squares for the two situations. (c) Under supposition (i), how would you test the main effect for treatments? (d) Under supposition (ii), how would you test the treatment x place interaction, and assuming that this was zero, the treatment main effect?

Appendix

558

APPENDIX

Table I. The Cumulative Standardized Normal Distribution Function* 1

4i(u) =

u

~~

zt

.-'az

V2n --

FOR

-4 099 ~ u ~ 0 0000

"

°00

°01

°oz

003

004

005

°06

°°7

°08

°°9

°O °1 °2 °3 °4

°5000 °4602 °4 207 °3821 °3446

°4960 °4562 °4 168 °3783 °3409

°49 20 °45 22 °4 129 °3745 °337 2

°4880 °44 83 °40go °37°7 °3336

°4 840 °4443 °4052 °3669 °3300

°4801 °4404 °401 3 °3632 °3 264

°4761 °4364 °3974 °3594 °3228

°4721 °4325 °3936 °3557 °3 19 2

°4681 °4286 °3897 °3520 °3156

°4 64 L °4 247 °3 859 °34 83 °3121

°5 06 0708 °9

°3085 02743 02420 °21I9 0184 1

°3050 :27°9 °2389 °20go 0181 4

°3015 °2676 023511 02061 01 788

02 981 02643 °23 27 02033 01 762

°2946 °261I °22 97 02005 °1736

°2912 °2578 02266 °1977 °17 1I

02877 °2546 °2236 01 949 01685

02843 °2514 02206 °19 22 01660

02810 °24113 °2177 011194 01635

°2776 0245 1 °2148 01867 °161I

01562 °1335 °1I31 °°9510 °°7927

°1539 °1314 °11I2 009342 007780

°15 15 °1292 °log3 °°9 17 6 °07636

01 492 °1271 01075 °Ogo12 007493

°1469 °1251 01056 008851 °°7353

°1446 °1230 01038 008691 °°7 21 5

°1423

-1°2 -1°3 - 1°4

°1587 °1357 °1I51 00g680 °08°76

008534 °07078

01 401 °1I90 °1003 °08379 006944

°1379 °1I7° °og853 008226 0068u

- 1°5 -1 06 - 1°7 -1 08 - 1°9

006681 °05480 004457 °03593 002872

006552 °0537° 004363 003515 002807

°06426 005262 0°4272 003438 002 743

006301 °05 155 004182 003362 002680

006178 °05050 004093 003288 002619

006057 004947 004006 003216 °02559

005938 °04846 °03920 °03144 °02500

005821 °04746 003836 003074 002 44 2

°057°5 °04648 003754 °03005 °02385

005592 004551 °03673 002938 °02330

-2°O

0022 75 °017116 °01 39° °01072 00'11198

'02222

002169 001 700 001321 00101 7 °°'7760

°021I8 001659 001287 °O'9go3 °0'7549

002068 001618 °012 55 00'9642 00'7344

002018 °01 578 00'9387 00'7 143

001 9 23 °01500 °01I60 00'8894 00'6756

°01876 °01463 °01I3° 00' 8656 00'6569

001831 001 426

'01222

001 97° °01 539 °OU91 0°'9137 00'6947

00'5868 00'4396 00'3264

00'5543 00'4 145 00'3072 °°'2256 °0' 1641

00'5386 00'4025 °0'2980 °0'2186 °°'1589

00'5234 °O'3go7 °o' 2890 °O'21I8 °0'1538

00'5085 00'3793 °0'28°3 °0'1489

00'4940 00'3611 1 °0'27 18 °°'1988 °0'144 1

00'4799 00'3573 °0' 2635 °°'1926 °0'1395

'011223

°O' 1I07 0°'7888 °O' 5571 °°'3 897 °°'27° 1

°0' 1°7° 0°'7622 °°'5377 °°'375 8 °0'2602

°0' 1°35 °°'736 4 °°'5 19° °°'362 4 °0'25°7

°0'7u 4 00'5009 00'3495 °O' 2415

°°' 1854

°O' 1785 °°' 121 3 00'8162 °°'544 2 °°'3594

°O' 1718 °0'1I66 °°'7 84 1 °0'5223 °°'3446

°0'7532 °0'5012 °0'3304

°°'2252 °°'1458 °0'9345 °°'5934 °°'373 2

°°' 21 57 °0'1395 00'8934 00'5668 °O' 3561

°°'2325 °O' 1434 00'8765 006 53 04 °0'3 179

°0'2216 °°'1366 00'8339 0065042 °0'3° 19

-

-1'0 -1'1

-2'1

-2'2

-2°3 - 2°4 -2°5 -2 06 - 2°7 -2 08 - 2°9 -3°O -3°1 -3°2 -3°3

-3"4 -3°5 -3°6

-3"7 -3°11 -3°9 -4°O -4° 1 -4°2 -4°3

-4"4 -4°5 -4°6 -4°7 -4°8 -4°9

'0 1 6210

001 743 °01 355 001044 00'7976

'1210 '1020

00'466 1 00'3467 °0'2555 °0'1866

00' 6037 00'4527 00'3364 °0'2477 °0' 18°7

°0'175°

00'5703 °O'426g 00'3167 °0'2327 °0' 1695

°0'135° °°'96 76 00' 6871 °°'4 834 °°'336 9

°0'13°6 00'9354 00'6637 00'4665 °°'3 248

°0' 1264 °0'go43 00'6410 °0'4501 °0'313 1

00'8740 °0'61go 00'4342 °O' 3018

°O'1I83 00'8447 °°'5976 °°'4 189 °0'29°9

°O' 1I44 00' 8164 °0'5770 °0'4041 °°' 28°3

°0'2326 °°'1591 °°'1°7 8 00'7235 °0'4810

°O' 2241 °O' 1531 °°' 1036 00'6948 00'4615

°°' 21 58 °0'1473 00'9961 00'6673 00'4427

°°' 2°7 8 °°'14 17 00'9574 00'6407 00'4247

'0)2001

°°'1363 00'9201 00' 6152 00'4074

°O' 1926 °0' 13Il 00'8842 °°'5906 °O'3go8

°°'3 167 °0'2066 °0'1335 00'8540 °O' 5413

°°'3°36 °°'1978 °°' 1277 00'8163 °°'5 169

°°'2910 °°' 1894 °°'7 801 °°'4935

°°'27 89 °°' 181 4 °O'1I68 °0'7455 °0'47 12

°°' 2673 °0'1737 °O'Il18 00'7124 00'4498

°0'2561 °0'1662 °°' 1069 00'6807 °°'4 294

00'6503 °°'4°98

°0'2351 °°'15 23 0°'9774 00'6212 °0'39Il

°°'3°9 2 °°'19 19 °o' Il79 00'7 178 °06 4327

°0'2949 °0'1828 °0' 1I23 00' 6827 °064IlI

°0'2813 °°'174 2 °°' 1069 00'6492 °O'3go6

°0'2682 °0'1660 00' 1017 0066173 °06 37"

°°'2558 °O' 1581 00'9680 00'58 6g 006 3525

°0'2439 °0'15°6 00'92Il 00'5580 °06 3348

°°'3398

°0'3241

'0'2112

'052013

°0'13°1 00'7933 °06 4792

°0' 1239 00'7547 00'4554

'012 4°1

'0 4 1222

'0 1 1261

00'8496 00'5669 00'3747 °0'2454 °0'1591 '04 1022

Example: 4i (-3 057) = 00'1785 = 0 000017850

'0 1

2°52

'01101

00'8424 00'6387

'0 1 1001

°0'1653 '0 1 1121

559

APPENDIX

Table I.

The Cumulative Standardized Normal Distribution Function (collfinued) 1

!Il(u)

u

..

= ~~ e-'dz

V2n

FOR 0,00 ~ U

~

4,99,

-«>

u

'0O

'01

'02

'03

'04

'05

'06

'07

'08

'09

'0

'5°4° '5438 '5832 ,6217 ,6591

'5°8o '5478 '5871 ,6255 '6628

'512O '55 17 '59 1o '62 93 ,6664

'5160 '5557 '5948 '633[ ,67°°

'5 199 '5596 '5987 ,6368 ,6736

'5 239 '5636 ,6026 ,64°6 ,6772

'5 279 '5 675 ,6064 ,6443 ,6808

'53 19 '57 14 '6103 ,6480 ,6844

'5359 '5753 '614[ ,6517 ,6879

,695° '7 291 '7611 '79 1O '8186

'6g85 '7324 '7642 '7939 '8212

'701 9 '7357 '7 673 '7967 '8238

'7°54 '7389 '77°3 '7995 ,8264

'7088 '7422 '7734 '8023 ,8289

'7 123 '7454 '7764 '8051 ,8315

'7 157 '7486 '7794 '8078 ,834°

'7 1go '75 17 '782 3 ,8106 ,8365

'7224 '7549 '7852 ,8133 ,8389

hr

'5000 '5398 '5793 '6179 '6554 ,6915 '7 257 '758o '7 881 '81 59 ,8413 ,8643 ,8849 'go32O '91924

,8438 ,8665 ,8869 'go4go '92°73

'846[ ,8686 ,8888 'go658 '92220

,8485 ,87°8 '8go7 'go824 '92364

,85°8 ,8729 ,8925 'go988 '925°7

,8531 ,8749 ,8944 '9 11 49 '9 2647

,8554 ,877° ,8962 ' '9 1309 '9 2785

,8577 '87go '8g80 '91466 '9 2922

,8599 ,8810 ,8997 '91621 '93056

,8621 ,883° 'go147 '9 1774 '93 189

['5 [,6 1'7 1,8 1'9

'933 19 '94520 '95543 '96407 '97 128

'9344 8 '94630 '95637 '96485 '97[93

'93574 '94738 '957 28 '96562 '97 257

'93699 '94845 '958[8 '96638 '973 20

'93822 '94950 '95go7 '967[2 '9738[

'93943 '95053 '95994 '96784 '97441

'94062 '95[54 '96080 '96856 '97500

'94179 '95 254 '96164 '96g26 '97558

'94295 '95352 '962 46 '96g95 '9761 5

'94408 '95449 '96327 '97062 '97670

2'0 2'1 2'2 2'3 2'4

'977 25 '9821 4 '98610 '9 8928 '9'1802

'97778 '982 57 '98645 '98956 '9' 202 4

'97831 '98300 '98679 '98983 '9'2240

'97882 '9 834 1 '98713 '9'0097 '9'2451

'9793 2 '98382 '98745 '9'0358 '9' 2656

'97982 '98030 '980 77 '9812 4 '98169 '98422 '98461 '98500 '98537 '98574 '98778 '98809 '98840 'g887° '98899 '9' 0613 '9'0863 '9'1106 '9'1344 '9'1576 '9' 2857 '9'3053 '9'3 244 '9'343[ '9'361 3

2'5 2,6 2'7 2,8 2'9

'9'37go '9'5339 '9'6533 '9'7445 '9' 8134

'9'3963 '9'5473 '9'6636 '9'75 23 '9' 81 93

'9'4 132 '9'5604 '9'6736 '9'7599 '9' 82 50

'9'4 297 '9'5731 '9' 6833 '9'7 673 '9'8305

'9'4457 '9'5855 '9'6g28 '9'7744 '9'8359

'9'461 4 '9'5975 '9'7° 20 '9'781 4 '9'8411

'9'4766 '9' 6093 '9'7 110 '9'7 882 '9'8462

'9'49 15 '9' 6207 '9'7 197 '9'7948 '9'8511

'9'5060 '9'6319 '9'7 282 '9'8012 '9'8559

'9'5 201 '9'6427 '9'73 65 '9!8074 '9' 8605

3'0 3'1 3'2

3"4

'9' 8650 '9'0324 '9' 3 129 '9' 5166 '9'6631

'9' 8694 '9'°646 '9'3363 '9'5335 '9'6752

'9'8736 '9'0957 '9' 35go '9'5499 '9' 6869

'9'8777 '9' 881 7 '9'1260 '9'1553 '9'3 810 '9'402 4 '9'5658 '9' 5811 '9'6982 '9'709 1

'9'8856 '9,' [836 '9'4230 '9'5959 '9'7 197

'9'8893 '9'2112 '9'44 29 '9' 6103 '9'7 299

'9'8g30 '9'8965 '9'2378 '9' 2636 '9'4623 '9'481b '9' 62 42 '9'6376 '9'7398 '9'7493

'9'8999 '9'2886 '9'499 1 '9'6505 '9'75 85

3'5 3,6 3"7 3,8 3'9

'9'7674 '9'8409 '9'8922 '9'2765 '9'5 1go

'9'7759 '9'8469 '9'8964 '9'3052 '9'5385

'9'7842 '9'7922 '9'8527 '9'8583 '9'0039 '9'0426 '9'33~7 '9'3593 '9'5573 '9'5753

'9'7999 '9' 8637 '9'0799 '9'3848 '9'5926

'9' 8074 '9' 8689 '9'1158 '9'4094 '9' 6092

'9'8146 '9'8739 '9'1504 '9'433 1 '9' 6253

'9' 821 5 '9'8787 '9' 1838 '9'4558 '9'6406

'9'8282 '9'8834 '9' 21 59 '9'4777 '9'6554

'9'8347 '9'8879 '9'2468 '9'4988 '9'6696

4'0 4'1 4'2 4'3 4"4

'9' 6833 '9'7934 '9'8665 '9'1460 '9'4587

'9'6964 '9'8022 '9'872,3 '9' 1837 '9'483 1

'9'7090 '9'7 211 '9'73 27 '9'8106 '9'8186 '9' 8263 '9'8778 '9'8832 '9'8882 '9' 21 99 '9'2545 '9' 2876 '9'5065 '9'5 288 '9'5502

'9'7439 '9'8338 '9'8931 '9' 3 193 '9'5706

'9'7546 '9'84°9 '9'8978 '9'3497 '9'5go2

'9'7 649 '9'7748 '9'8477 '9'8542 '9'°226 '9'°655 '9'37 88 '9'4066 '9' 6089 '9'6268

'9'7843 '9' 8605 '9' 1066 '9'4332 '9'6439

4'5 4,6 4'7 4,8 4'9

'9'6602 '9'7888 '9' 8699 '962067 '965208

'9'6759 '9'7987 '9'8761 '962 453 '965446

'9'6go8 '9'8081 '9'882[ '962822 '965673

'9'7 187 '9' 8258 '9'893[ '963508 '966094

'9'73 18 '9'8340 '9'8983 '96382 7 '966289

'9'7442 '9'8419 '960320 '964[3[ '966475

:9'7561 '9'8494 '960789 '9644 20 '966652

'9'7675 '9'8566 '96 [235 '964696 '96682[

'9'77 84 '9' 8634 '96 [66[ '964958 '96698[

'I

'2 '3 '4 '5 ,6 '7 ,8 '9 ['0 1'[ 1'2 1'3

3"3

'9'7°5 1 '9' 81 72 '9'8877 '963[73 '96588 9

Example: !Il(3'57)

= '9'8215 = 0,9998215,

• Abridged from Table II of Statistical Tables and Formulas by A. Hald, John Wiley & Sons, New York, 1952.

560

APPENDIX

Table II. Fractional Points of the t Distribution*

X

0.750

0.900

0.950

0.975

0.990

0.995

I 2 3 4 5

1.000 0.816 0.765 0.741 0.727

3.078 1.886 1.638 1.533 1.476

6.314 2.920 2.353 2.132 2.015

12.706 4.303 3.182 2.776 2.571

31.821 6.965 4.541 3.747 3.365

63.657 9.925 5.841 4.604 4.032

318 22.3 10.2 7.173 5.893

6 7 8 9 10

0.718 0.711 0.706 0.703 0.700

1.440 1.415 1.397 1.383 1.372

1.943 1.895 1.860 1.833 1.812

2.447 2.365 2.306 2.262 2.228

3.143 2.998 2.896 2.821 2.764

3.707 3.499 3.355 3.250 3.169

5.208 4.785 4.501 4.297 4.144

II 12 13 14 15

0.697 0.695 0.694 0.692 0.691

1.363 1.356 1.350 1.345 1.341

1.796 1.782 1.771 1.761 1.753

2.201 2.179 2.160 2.145 2.131

2.718 2.681 2.650 2.624 2.602

3.106 3.055 3.012 2.977 2.947

4.025 3.930 3.852 3.787 3.733

16 17 18 19 20

0.690 0.689 0.688 0.688 0.687

1.337 1.333 1.330 1.328 1.325

1.746 1.740 1.734 1.729 1.725

2.120 2.110 2.101 2.093 2.086

2.583 2.567 2.552 2.539 2.528

2.921 2.898 2.878 2.861 2.845

3.686 3.646 3.610 3.579 3.552

0.999

561

APPENDIX

Table II• Fractional Points of the t Distribution (continued)

.X

0.750

0.900

0.950

0.975

0.990

0.995

0.999

21 22 23 24 25

0.686 0.686 0.685 0.685 0.684

1.323 1.321 1.319 1.318 1.316

1.721 1.717 1.714 1.711 1.708

2.080 2.074 2.069 2.064 2.060

2.518 2.508 2.500 2.492 2.485

2.831 2.819 2.807 2.797 2.787

3.527

26 27 28 29 30

0.684 0.684 0.683 0.683 0.683

1.315 1.314 1.313 1.311 1.310

1.706 1.703 1.701 1.699 1.697

2.056 2.052 2.048 2.045 2.042

2.479 2.473 2.467 2.462 2.457

2.779 2.771 2.763 2.756 2.750

3.435 3.421 3.408 3.396 3.385

40 60 120

0.681 0.679 0.677 0.674

1.303 1.296 1.289 1.282

1.684 1.671 1.658 1.645

2.021 2.000 1.980 1.960

2.423 2.390 2.358 2.326

2.704 2.660 2.617 2.576

3.307 3.232 3.160 3.090

co

3.5(l~

J.485 3.467 3.450

*Abridged from Table 12 of Biometrika Tables for Statisticians, vol. I, edited by E. S. Pearson and H. O. Hartley, Cambridge University Press, Cambridge (1954), and Table III of Statistical Tables for Biological, Agricultural, and Medical Research, R. A. Fisher and F. Yates, Oliver & Boyd, Edinburgh, 1953.

562

APPENDIX

Table lli. Fractional p

f

0.005

0.010

0.025

1 2 3 4 5

0.0'393 0.0100 0.0717 0.207 0.412

0.0 3 157 0.0201 0.ll5 0.297 0.554

0.0 3982 0.0506 0.216 0.484 0.831

6 7 8 9 10

0.676 0.989 1.34 1.73 2.16

0.872 1.24 1.65 2.09 2.56

II 12 13 14 15

2.60 3.07 3.57 4.07 4.60

16 17 18 19 20

0.05

0.10

0.20

0.30

0.40

0.0 2393 0.103 0.352 0.7ll l.l5

0.0158 0.211 0.584 1.06 1.61

0.0642 0.446 1.00 1.65 2.34

0.148 0.713 1.42 2.19 3.00

0.275 1.02 1.87 2.75 3.66

1.24 1.69 2.18 2.70 3.25

1.64 2.17 2.73 3.33 3.94

2.20 2.83 3.49 4.17 4.87

3.07 3.82 4.59 5.38 6.18

3.83 4.67 5.53 6.39 7.27

4.57 5.49 6.42 7.36 8.30

3.05 3.57 4.ll 4.66 5.23

3.82 4.40 5.01 5.63 6.26

4.57 5.23 5.89 6.57 7.26

5.58 6.30 7.04 7.79 8.55

6.99 7.81 8.63 9.47 10.3

8.15 9.03 9.93 10.8 11.7

9.24 10.2 ILl 12.1 13.0

5.14 5.70 6.26 6.84 7.43

5.81 6.41 7.01 7.63 8.26

6.91 7.56 8.23 8.91 9.59

7.96 8.67 9.39 10.1 10.9

9.31 10.1 10.9 11.7 12.4

11.2 12.0 12.9 13.7 14.6

12.6 13.5 14.4 15.4 16.3

14.0 14.9 15.9 16.9 17.8

21 22 23 24 25

8.03 8.64 9.26 9.89 10.5

8.90 9.54 10.2 10.9 ll.5

10.3 11.0 11.7 12.4 13.1

11.6 12.3 13.1 13.8 14.6

13.2 14.0 14.8 15.7 16.5

15.4 16.3 17.2 18.1 18.9

17.2 18.1 19.0 19.9 20.9

18.8 19.7 20.7 21.7 22.6

26 27 28 29 30

11.2 ll.8 12.5 13.1 13.8

12.2 12.9 13.6 14.3 15.0

13.8 14.6 15.3 16.0 16.8

15.4 16.2 16.9 17.7 18.5

17.3 18.1 18.9 19.8 20.6

19.8 20.7 21.6 22.5 23.4

21.8 22.7 23.6 24.6 25.5

23.6 24.5 25.5 26.5 27.4

35 40 45 50 75 100

17.2 20.7 24.3 28.0 47.2 67.3

18.5 22.2 25.9 29.7 49.5 70.1

20.6 24.4 28.4 32.4 52.9 74.2

22.5 26.5 30.6 34.8 56.1 77.9

24.8 29.1 33.4 37.7 59.8 82.4

27.8 32.3 36.9 41.4 64.5 87.9

30.2 34.9 39.6 44.3 68.1 92.1

32.3 37.1 42.0 46.9 71.3 95.8

563

APPENDIX

Points of the X2 Distribution * 0.50

0.60

0.455 1.39 2.37 3.36 4.35

0.708 1.83 2.95 4.04 5.13

0.70

0.80

0.90

0.975

0.990

3.84 5.99 7.81 9.49 11.1

5.02 7.38 9.35 11.1 12.8

6.63 9.21 11.3 13.3 15.1

7.88 10.6 12.8 14.9 16.7

10.8 13.8 16.3 18.5 20.5

0.95

0.995

0.999

1.07 2.41 3.67 4.88 6.06

1.64 3.22 4.64 5.99 7.29

6.21 7.28 8.35 9.41 10.5

7.23 8.38 9.52 10.7 11.8

8.56 9.80 11.0 12.2 13.4

10.6 12.0 13.4 14.7 16.0

12.6 14.1 15.5 16.9 18.3

14.4 16.0 17.5 19.0 20.5

16.8 18.5 20.1 21.7 23.2

18.5 20.3 22.0 23.6 25.2

22.5 24.3 26.1 27.9 29.6

10.3 11.3 12.3 13.3 14.3

11.5 12.6 13.6 14.7 15.7

12.9 14.0 15.1 16.2 17.3

14.6 15.8 17.0 18.2 19.3

17.3 18.5 19.8 21.1 22.3

19.7 21.0 22.4 23.7 25.0

21.9 23.3 24.7 26.1 27.5

24.7 26.2 27.7 29.1 30.6

26.8 28.3 29.8 31.3 32.8

31.3 32.9 34.5 36.1 37.7

15.3 16.3 17.3 18.3 19.3

16.8 17.8 18.9 19.9 21.0

18.4 19.5 20.6 21.7 22.8

20.5 21.6 22.8 23.9 25.0

23.5 24.8 26.0 27.2 28.4

26.3 27.6 28.9 30.1 31.4

28.8 30.2 31.5 32.9 34.2

32.0 33.4 34.8 36.2 37.6

34.3 35.7 37.2 38.6 40.0

39.3 40.8 42.3 43.8 45.3

20.3 21.3 22.3 23.3 24.3

22.0 23.0 24.1 25.1 26.1

23.9 24.9 26.0 27.1 28.2

26.9 27.3 28.4 29.6 30.7

29.6 30.8 32.0 33.2 34.4

32.7 33.9 35.2 36.4 37.7

35.5 36.8 38.1 39.4 40.6

38.9 40.3 41.6 43.0 44.3

41.4 42.8 44.2 45.6 46.9

46.8 48.3 49.7 51.2 52.6

25.3 26.3 27.3 28.3 29.3

27.2 28.2 29.2 30.3 31.3

29.2 30.3 31.4 32.5 33.5

31.8 32.9 34.0 35.1 36.3

35.6 36.7 37.9 39.1 40.3

38.9 40.1 41.3 42.6 43.8

41.9 43.2 44.5 45.7 47.0

45.6 47.0 48.3 49.6 50.9

48.3 49.6 51.0 52.3 53.7

54.1 55.5 56.9 58.3 59.7

34.3 39.3 44.3 49.3 74.3 99.3

36.5 41.6 46.8 51.9 77.5 102.9

38.9 44.2 49.5 54.7 80.9 106.9

41.8 47.3 52.7 58.2 85.1 111.7

46.1 51.8 57.5 63.2 91.1 118.5

49.8 55.8 61.7 67.5 96.2 124.3

53.2 59.3 65.4 71.4 100.8 129.6

57.3 63.7 70.0 76.2 106.4 135.6

60.3 66.8 73.2 79.5 110.3 140.2

66.6 73.4 80.1 86.7 118.6 149.4

5.35 6.35 7.34 8.34 9.34

2.71 4.61 6.25 7.78 9.24

* Abridged from Table V of Statistical Tables alld Formulas by A. Hald, John Wiley and Sons, New York, 1952.

'"

1.72 1.66 1.60 1.55 1.49

1.54 1.47 1.40 1.32 1.24

1.57 1.51 1.44 1.37 1.30

1.61 1.54 1.48 1.41 1.34 1.64 1.57 1.51 1.45 1.38

1.67 1.61 1.54 1.48 1.42 1.77 1.71 1.66 1.60 1.55

1.82 1.76 1.71 1.65 1.60

1.85 1.79 1.74 1.68 1.63

1.88 1.83 1.77 1.72 1.67

1.93 1.87 1.82 1.77 1.72

1.98 1.93 1.87 1.82 1.77

2.05 2.00 1.95 1.90 1.85

2.14 2.09 2.04 1.99 1.94

2.28 2.23 2.18 2.13 2.08

2.49 2.44 2.39 2.35 2.30

2.88 2.84 2.79 2.75 2.71

30 40 60 120

1.59 1.58 1.57 1.56 1.55

1.63 1.61 1.60 1.59 1.58

1.66 1.65 1.64 1.63 1.62

1.69 1.68 1.67 1.66 1.65 1.72 1.11 1.70 1.69 1.68

1.77 1.76 1.75 1.74 1.73 1.82 1.81 1.80 1.79 1.78

1.87 1.86 1.85 1.84 1.83

1.89 1.88 1.87 1.87 1.86

1.93 1.92 1.91 1.90 1.89

1.97 1.96 1.95 1.94 1.93

2.02 2.01 2.00 2.00 1.99

2.09 2.08 2.07 2.06 2.06

2.18 2.17 2.17 2.16 2.15

2.32 2.31 2.30 2.29 2.28

2.53 2.52 2.51 2.50 2.50

2.92 2.91 2.90 2.89 2.89

1.72 1.70

25 26 27 28 29

1.74 1.72 1.70 1.69 1.67 1.13

1.77 1.75 1.79 1.78 1.76 1.74 1.73

1.84 1.83 1.81 1.80 1.78

1.89 1.87 1.86 1.84 1.83

1.94 1.92 1.90 1.89 1.88

1.96 1.95 1.93 1.92 1.91

2.00 1.98 1.97 1.95 1.94

2.04 2.02 2.01 1.99 1.98

2.09 2.08 2.06 2.05 2.04

2.16 2.14 2.13 2.11 2.10

2.25 2.23 2.22 2.21 2.19

2.38 2.36 2.35 2.34 2.33

2.59 2.57 2.56 2.55 2.54

2.97 2.96 2.95 2.94 2.93

20 21 22 23 24

1.15

1.71 1.69 1.67 1.66 1.64

1.73

1.68 1.66 1.64 1.62 1.61

1.75 1.72 1.70

1.18

1.82

1.85 1.81 1.78

1.87 1.84 1.81 1.78 1.76

2.11 2.03 1.96 1.90 1.86

3.14 2.76 2.51 2.34 2.21

1.50 1.42 \.35 1.26 1.17

1.56 1.54 1.53 1.52 1.51

1.64 1.62 1.60 1.59 1.57

i.79 1.75 1.72 1.69 1.67

2.08 2.00 1.93 1.88 1.83

3.12 2.74 2.49 2.32 2.18

63.06 9.48 5.14 3.78

62.79 9.47 5.15 3.79

1.90 1.87 1.84 1.81 1.79

1.92 1.89 1.86 1.84 1.81

1.97 1.94 1.91 1.89 1.86

2.02 1.99 1.96 1.93 1.91

2.06 2.03 2.00 1.98 1.96

2.09 2.06 2.03 2.00 1.98

2.12 2.09 2.06 2.04 2.02

2.16 2.13 2.10 2.08 2.06

2.21 2.18 2.15 2.13 2.11

2.27 2.24 2.22 2.20 2.18

2.36 2.33 2.31 2.29 2.27

2.49 2.46 2.44 2.42 2.40

2.70 2.67 2.64 2.62 2.61

3.07 3.05 3.03 3.01 2.99

3.16 2.78 2.54 2.36 2.23

3.17 2.80 2.56 2.38 2.25

120

60

2.13 2.05 1.99 1.93 1.89

62.53 9.47 5.16 3.80

40

62.26 9.46 5.17 3.82

30

2.16 2.08 2.01 1.96 1.91

15 16 17 18 19

2.18 2.10 2.04 1.98 1.94

2.20 2.12 2.06 2.01 1.96

2.24 2.17 2.10 2.05 2.01

2.28 2.21 2.15 2.10 2.05

2.32 2.25 2.19 2.14 2.\0

2.35 2.27 2.21 2.16 2.12

2.38 2.30 2.24 2.20 2.15

2.41 2.34 2.28 2.23 2.19

2.46 2.39 2.33 2.28 2.24

2.52 2.45 2.39 2.35 2.31

2.61 2.54 2.48 2.43 2.39

2.73 2.66 2.61 2.56 2.52

2.92 2.86 2.81 2.76 2.73

3.29 3.23 3.18 3.14 3.10

10 II 12 13 14

3.19 2.82 2.58 2.40 2.28

3.21 2.84 2.59 2.42 2.30

3.24 2.87 2.63 2.46 2.34

3.27 2.90 2.67 2.50 2.38

3.30 2.94 2.70 2.54 2.42

3.32 2.96 2.72 2.56 2.44

3.34 2.98 2.75 2.59 2.47

3.37 3.01 2.78 2.62 2.51

3.40 3.05 2.83 2.67 2.55

3.45 3.11 2.88 2.73 2.61

3.52 3.18 2.96 2.81 2.69

3.62 3.29 3.07 2.92 2.81

3.78 3.46 3.26 3.11 3.01

4.06 3.78 3.59 3.46 3.36

5 6 7 8 9

62.00 9.45 5.18 3.83

61.74 9.44 5.18 3.84

61.22 9.42 5.20 3.87

60.71 9.41 5.22 3.90

60.19 9.39 5.23 3.92

59.86 9.38 5.24 3.94

59.44 9.37 5.25 3.95

58.91 9.35 5.27 3.98

58.20 9.33 5.28 4.01

57.24 9.29 5.31 4.05

55.83 9.24 5.34 4.11

53.59 9.16 5.39 4.19

49.50 9.00 5.46 4.32

I

24

20

15

12

10

9

8

7

6

39.86 8.53 5.54 4.54

4

3

2

Percentage Points of the F Distribution * 90 per cent pOints

2 3 4

~I

Table IV.

1.46 1.38 1.29 1.19 1.00

1.52 1.50 1.49 1.48 1.47

1.61 1.59 1.57 1.55 1.53

1.76 1.72 1.69 1.66 1.63

1.97 1.90 1.85 1.80

~.06

3.10 2.72 2.47 2.29 2.16

63.33 9.49 5.13 3.76

'"

VI

><

0

Z

'"C tTl

> '"C

~

0\

25 26 27 28 29

30 40 60 120

00

2.40 2.39 2.37 2.36 2.35

2.49 2.47 2.46 2.45 2.43

2.60 2.59 2.57 2.56 2.55

2.53 2.45 2.37 2.29 2.21

2.76 2.74 2.73 2.71 2.70

2.69 2.61 2.53 2.45 2.37

2.99 2.98 2.96 2.95 2.93

2.92 2.84 2.76 2.68 2.60

3.39 3.37 3.35 3.34 3.33

3.32 3.23 3.15 3.07 3.00

4.24 4.23 4.21 4.20 4.18

4.17 4.08 4.00 3.92 3.84

2.42 2.34 2.25 2.17 2.10

2.51 2.49 2.46 2.44 2.42

2.60 2.57 2.55 2.53 2.51

2.71 2.68 2.66 2.64 2.62

2.87 2.84 2.82 2.80 2.78

3.10 3.07 3.05 3.03 3.01

3.49 3.47 3.44 3.42 3.40

2.33 2.25 2.17 2.09 2.01

2.27 2.18 2.10 2.02 1.94

2.34 2.32 2.31 2.29 2.28 2.21 2.12 2.04 1.96 1.88

2.28 2.27 2.25 ·2.24 2.22

2.39 2.37 2.34 2.32 2.30

4.35 4.32 4.30 4.28 4.26

20 21 22 23 24

2.71 2.66 2.61 2.58 2.54

2.79 2.74 2.70 2.66 2.63

2.90 2.85 2.81 2.77 2.74

3.06 3.01 2.96 2.93 2.90

3.29 3.24 3.20 3.16 3.\3 2.45 2.42 2.40 2.37 2.36

2.59 2.54 2.49 2.46 2.42

2.64 2.59 2.55 2.51 2.48

3.68 3.63 3.59 3.55 3.52

4.54 4.49 4.45 4.41 4.38

3.22 3.09 3.00 2.92 2.85

3.33 3.20 3.11 3.03 2.96

3.48 3.36 3.26 3.18 3.11

15 16 17 18 19

3.02 2.90 2.80 2.71 2.65

3.07 2.95 2.85 2.77 2.70

3.14 3.01 2.91 2.83 2.76

3.71 3.59 3.49 3.41 3.34

4.10 3.98 3.89 3.81 3.74

4.96 4.84 4.75 4.67 4.60

10 11 12 13 14

4.77 4.10 3.68 3.39 3.18

4.82 4.15 3.73 3.44 3.23

4.95 4.28 3.87 3.58 3.37

5.05 4.39 3.97 3.69 3.48

5.79 5.14 4.74 4.46 4.26

6.61 5.99 5.59 5.32 5.12

5 6 7 8 9

4.88 4.21 3.79 3.50 3.29

2.16 2.08 1.99 1.91 1.83

2.24 2.22 2.20 2.19 2.18

2.01 1.92 1.84 1.75 1.67

2.09 2.00 1.92 1.83 1.75

1.93 1.84 1.75 1.66 1.57

2.01 1.99 1.97 1.96 1.94

2.09 2.07 2.06 2.04 2.03

2.16 2.15 2.13 2.12 2.10

2.33 2.28 2.23 2.19 2.16

2.77 2.65 2.54 2.46 2.39

4.56 3.87 3.44 3.15 2.94

248.0 19.45 8.66 5.80

20

2.12 2.10 2.07 2.05 2.03

2.28 2.25 2.23 2.20 2.18

2.35 2.32 2.30 2.27 2.25

2.40 2.35 2.31 2.27 2.23

2.85 2.72 2.62 2.53 2.46

4.62 3.94 3.51 3.22 3.01

245.9 19.43 8.70 5.86

15

2.20 2.18 2.15 2.\3 2.11

2.48 2.42 2.38 2.34 2.31

2.91 2.79 2.69 2.60 2.53

4.68 4.00 3.57 3.28 3.07

243.9 19.41 8.74 5.91

12

2.54 2.49 2.45 2.41 2.38

2.98 2.85 2.75 2.67 2.60

4.74 4.06 3.64 3.35 3.14

241.9 19.40 8.79 5.96

5.19 4.53 4.12 3.84 3.63

215.7 19.16 9.28 6.59

199.5 19.00 9.55 6.94

161.4 18.51 10.13 7.71

1 2 3 4

5.41 4.76 4.35 4.07 3.86

10

9

95 per cent points 8 240.5 19.38 8.81 6.00

5

7

230.2 234.0 236.8 238.9 19.37 19.30 19.33 19.35 9.01 8.94 8.89 8.85 6.16 6.04 6.26 6.09

4

6

224.6 19.25 9.12 6.39

2

1.89 1.79 1.70 1.61 1.52

1.96 1.95 1.93 1.91 1.90

2.08 2.05 2.03 2.01 1.98

2.29 2.24 2.19 2.15 2.11

2.74 2.61 2.51 2:42 2.35

4.53 3.84 3.41 3.12 2.90

249.1 19.45 8.64 5.77

24

1.84 1.74 1.65 1.55 1.46

1.92 1.90 1.88 1.87 1.85

2.04 2.01 1.98 1.96 1.94

2.25 2.19 2.15 2.11 2.07

2.70 2.57 2.47 2.38 2.31

4.50 3.81 3.38 3.08 2.86

250.1 19.46 8.62 5.75

30

60

1.79 1.69 1.59 1.50 1.39

1.87 1.85 1.84 1.82 1.81

1.99 1.96 1.94 1.91 1.89

2.20 2.15 2.10 2.06 2.03

2.66 2.53 2.43 2.34 2.27

4.46 3.77 3.34 3.04 2.83

1.74 1.64 1.53 1.43 1.32

1.82 1.80 1.79 1.77 1.75

1.95 1.92 1.89 1.86 1.84

2.16 2.11 2.06 2.02 1.98

2.62 2.49 2.38 2.30 2.22

4.43 3.74 3.30 3.01 2.79

251.1 252.2 19.47 19.48 8.59 8.57 5.72 5.69

40

1.68 1.58 1.47 1.35 1.22

1.77 1.75 1.73 1.71 1.70

1.90 1.87 1.84 1.81 1.79

2.11 2.06 2.01 1.97 1.93

2.58 2.45 2.34 2.25 2.18

4.40 3.70 3.27 2.97 2.75

253.3 19.49 8.55 5.66

120

1.62 1.51 1.39 1.25 1.00

1.71 1.69 1.67 1.65 1.64

1.84 1.81 1.78 1.76 1.73

2.07 2.01 1.96 1.92 1.88

2.54 2.40 2.30 2.21 2.13

4.36 3.67 3.23 2.93 2.71

254.3 19.50 8.53 5.63

00

"C

VI 0\ VI

~

m Z t1

> "C

4.77 4.69 4.62 4.56 4.51

6.20 6.12 6.04 5.98 5.92

5.87 5.83 5.79 5.75 5.72

5.69 5.66 5.63 5.61 5.59

5.57 5.42 5.29 5.15 5.02

15 16 17 18 19

20 21 22 23 24

25 26 27 28 29

30 40 60 120

co

5.46 5.26 5.10 4.97 4.86

6.94 6.72 6.55 6.41 6.30

10 II 12 13 14

3.86 3.82 3.78 3.75 3.72

3.69 3.67 3.65 3.63 3.61

3.59 3.46 3.34 3.23 3.12

4.29 4.27 4.24 4.22 4.20

4.18 4.05 3.93 3.80 3.69

4.15 4.08 4.01 3.95 3.90

4.83 4.63 4.47 4.35 4.24

7.76 6.60 5.89 5.42 5.08

4.46 4.42 4.38 4.35 4.32

8.43 7.26 6.54 6.06 5.71

10.01 8.81 8.07 7.57 7.21

5 6 7 8 9

3

4

3.25 3.13 3.01 2.89 2.79

3.35 3.33 3.31 3.29 3.27

3.51 3.48 3.44 3.41 3.38

3.80 3.73 3.66 3.61 3.56

4.47 4.28 4.12 4.00 3.89

7.39 6.23 5.52 5.05 4.72

799.5 864.2 899.6 39.00 39.17 39.25 16.04 15.44 15.10 10.65 9.98 9.60

647.8 38.51 17.44 12.22

2

I 2 3 4

~I

3.03 2.90 2.79 2.67 2.57

3.13 3.10 3.08 3.06 3.04

3.29 3.25 3.22 3.18 3.15

3.58 3.50 3.44 3.38 3.33

4.24 4.04 3.89 3.77 3.66

7:15 5.99 5.29 4.82 4.48

921.8 39.30 14.88 9.36 6.52 5.37 4.6"' 4.20 3.87

6.62 5.46 4.76 4.30 3.96 3.72 3.53 3.37 3.25 3.15

6.68 5.52 4.82 4.36 4.03 3.78 3.59 3.44 3.31 3.21 3.12 3.05 2.98 2.93 2.88

3.20 3.12 3.06 3.01 2.96

2.87 2.74 2.63 2.52 2.41

2.97 2.94 2.92 2.90 2.88

3.13 3.09 3.05 3.02 2.99

2.91 2.87 2.84 2.81 2.78

2.75 2.73 2.71 2.69 2.67

3.01 2.97 2.93 2.90 2.87

2.85 2.82 2.80 2.78 2.76

2.75 2.62 2.51 239 2.29

3.29 3.22 3.16 3.10 3.05

3.95 3.76 3.61 3.48 3.38

4.07 3.88 3.73 3.60 3.50

3.41 3.34 3.28 3.22 3.17

6.85 5.70 4.99 4.53 4.20

2.65 2.53 2.41 2.30 2.19

3.85 3.66 3.51 3.39 3.29

1.7!i 1.64 1.48 1.31 1.00 1.87 1.72 1.58 1.43 1.27

1.94 1.80 1.67 1.53 1.39 2.01 1.88 1.74 1.61 1.48 2.07 1.94 1.82 1.69 1.57 2.14 2.01 1.88 1.76 1.64 2.20 2.07 1.94 1.82 1.71 2.31 2.18 2.06 1.94 1.83 2.41 2.29 2.17 2.05 1.94 2.51 2.39 2.27 2.16 2.05

2.57 2.45 2.33 2.22 2.11

1.91 1.88 1.85 1.83 1.81 1.98 1.95 1.93 1.91 1.89 2.05 2.03 2.00 1.98 1.96 2.12 2.09 2.07 2.05 2.03 2.18 2.16 2.13 2.11 2.09 2.24 2.22 2.19 2.17 2.15 2.30 2.28 2.25 2.23 2.21 2.41 2.39 1.36 2.34 2.32 2.51 2.49 2.47 2.45 2.43

2.61 2.59 2.57 2.55 2.53

2.68 2.65 2.63 2.61 2.59

2.09 2.04 2.00 1.97 1.94 2.16 2.11 2.08 2.04 2.01 2.22 2.18 2.14 2.11 2.08 2.29 2.25 2.21 2.18 2.15 2.35 2.31 2.27 2.24 2.21 2.41 2.37 2.33 2.30 2.27 2.46 2.42 2.39 2.36 2.33

2.57 2.53 2.50 2.47 2.44

2.68 2.64 2.60 2.57 2.54

2.77 2.73 2.70 2.67 2.64

2.84 2.80 2.76 2.73 2.70

2.40 2.32 2.25 2.19 2.13 2.46 2.38 2.32 2.26 2.20

2.52 2.45 2.38 2.32 2.27 2.59 2.51 2.44 2.38 2.33 2.64 2.57 2.50 2.44 2.39 2.70 2.63 2.56 2.50 2.45

2.76 2.68 2.62 2.56 2.51

2.86 2.79 2.72 2.67 2.62

2.96 2.89 2.82 2.77 2.72

3.06 2.99 2.92 2.87 2.82

3.08 2.88 2.72 2.60 2.49 3.14 2.94 2.79 H6 2.55 3.20 300 2.85 2.72 2.61 3.26 3.06 2.91 2.78 2.67 3.31 3.12 2.96 2.84 2.73

3.37 3.17 3.02 2.89 2.79

3.42 3.23 3.07 2.95 2.84

6.02 4.85 4.14 3.67 3.33 6.07 4.90 4.20 3.73 3.39 6.12 4.96 4.25 3.78 3.45

6.18 5.01 4.31 3.84 3.51

3.52 3.33 3.18 3.05 2.95

co

3.62 3.43 3.28 3.15 3.05

120

6.23 5.07 4.36 3.89 3.56

60

6.28 5.12 4.42 3.95 3.61

40

6.33 5.17 4.47 4.00 3.67

30

6.43 5.27 4.57 4.10 3.77

24

lOIN 1014 1010 1006 997.2 1001 39.50 39.49 39.48 39.47 39.46 39.46 13.90 13.95 13.99 14.04 14.08 14.12 8.26 8.31 8.36 8.41 8.46 8.51

20 993.1 39.45 14.17 8.56

6.76 5.60 4.90 4.43 4.10

984.9 39.43 14.25 8.66

976.7 39.41 14.34 8.75

968.6 39.40 14.42 8.84

963.3 39.39 14.47 8.90

956.7 39.37 14.54 8.98

948.2 39.36 14.62 9.07

15

12

10

9

7

6.98 5.82 5.12 4.65 4.32

937.1 39.33 14.73 9.20

6

Table IV. Percentage Points of the F Distribution (continued) 97.5 per cent points

VI

><

9

Z

'tl 1TI

> 'tl

0\ 0\

4

5

6

7

8

9

10

12

15

20

24

30

40

60

120

co

5.85 5.78 5.72 5.66 5.61

5.57 5.53 5.49

5.42

5.39 5.18 4.98 4.79 4.61

8.68 8.53 8.40 8.29 8.18

8.10 8.02 7.95 7.88 7.82

7.77 7.72 7.68 7.64 7.60

7.56 7.31 7.08 6.85 6.63

15 16 17 18 19

20 21 22 23 24

25 26 27 28 29

30 40 60 120 co

5.4S

6.36 6.23 6.11 6.01 5.93

10.04 9.65 9.33 9.07 8.86

7.56 7.21 693 6.70 6.51

2.01 1.80 1.60 1.38 1.00 2.11 1.92 1.73 1.53 1.32 2.21 2.02 1.84 1.66 1.47 2.30 2.11 1.94 1.76 1.59 2.39 2.20 2.03 1.86 1.70 2.47 2.29 2.12 1.95 1.79 2.55 2.37 2.20 2.03 1.88 2.70 2.52 2.35 2.19 2.04 2.84 2.66 2.50 2.34 2.18 2.98 2.80 2.63 2.47 2.32

3.07 2.89 2.72 2.56 2.41

3.17 2.99 2.82 2.66 2.51

3.30 3.12 2.95 2.79 2.64

3.47 3.29 3.12 2.96 2.80

3.70 3.51 3.34 3.17 3.02

4.02 3.83 3.65 3.48 3.32

4.51 4.31 4.13 3.95 3.78

2.17 2.13 2.10 2.06 2.03 2.27 2.23 2.20 2.17 2.14 2.36 2.33 2.29 2.26 2.23 2.45 2.42 2.38 2.35 2.33 2.54 2.50 2.47 2.44 2.41 2.62 2.58 2.55 2.52 2.49 2.70 2.66 2.63 2.60 2.57 2.85 2.81 2.78 2.75 2.73

2.99 2.96 2.93 2.90 2.87

3.13 3.09 3.06 3.03 3.00

3.22 3.18 3.15 3.12 3.09

3.32 3.29 3.26 3.23 3.20

3.46 3.42 3.39 3.36 3.33

3.63 3.59 3.56 3.53 3.50

3.85 3.82 3.78 3.75 3.73

4.18 4.14 4.11 4.07 4.04

4.68 4.64 4.60 4.57 4.54

2.94 2.88 2.83 2.78 2.74

3.09 3.03 2.98 2.93 2.89

3.23 3.17 3.12 3.07 3.03

3.37 3.31 3.26 3.21 3.17

3.46 3.40 3.35 3.30 3.26

3.56 3.51 3.45 3.41 3.36

3.70 3.64 3.59 3.54 3.50

3.87 3.81 3.76 3.71 3.67

4.10 4.04 3.99 3.94 3.90

4.43 4.37 4.31 4.26 4.22

2.42 2.36 2.31 2.26 2.21 2.52 2.46 2.40 2.35 2.31 2.61 2.55 2.50 2.45 2.40 2.69 2.64 2.58 2.54 2.49 2.78 2.72 2.67 2.62 2.58 2.86 2.80 2.75 2.70 2.66

4.94 4.87 4.82 4.76 4.72

2.87 2.75 2.65 2.57 2.49 2.96 2.84 2.75 2.66 2.58 3.05 2.93 2.83 2.75 2.67 3.13 3.02 2.92 2.84 2.76 3.21 3.10 3.00 2.92 2.84 3.29 3.18 3.08 3.00 2.92

3.37 3.26 3.16 3.08 3.00

3.52 3.41 3.31 3.23 3.15

3.67 3.55 3.46 3.37 3.30

3.80 3.69 3.59 3.51 3.43

3.89 3.78 3.68 3.60 3.52

4.00 3.89 3.79 3.71 3.63

4.14 4.03 3.93 3.84 3.77

4.32 4.20 4.10 4.01 3.94

4.56 4.44 4.34 4.25 4.17

4.89 4.77 4.67 4.58 4.50

5.42 5.29 5.18 5.09 5.01

3.91 3.60 3.36 3.17 3.00 4.00 3.69 3.45 3.25 3.09 4.08 4.78 3.54 3.34 3.18 3.86 3.62 3.43 3.27

~.17

4.25 3.94 3.70 3.51 3.35

4.33 4.02 3.78 3.59 3.43

4.41 4.10 3.86 3.66 3.51

4.56 4.25 4.01 3.82 3.66

4.71 4.40 4.16 3.96 3.80

4.85 4.54 4.30 3.10 3.94

4.94 4.63 4.39 4.19 4.03

5.06 4.74 4.50 4.30 4.14

5.39 5.07 4.82 4.62 4.46

5.64 5.32 5.06 4.86 4.69

5.99 5.67 5.41 5.21 5.04

6.55 6.22 5.95 5.74 5.56

5.20 4.89 4.64 4.44 4.28

6366 6339 6313 6287 6261 6235 6209 6157 6106 6022 6056 5982 5928 5859 5625 5764 4n52 ..,,99.' 54U3 99.50 99.49 99.48 99.47 99.47 99.46 99.45 99.43 99.42 99.40 99.39 99.37 99.36 99.30 99.33 99.25 99.00 99.17 98.50 26.13 26.22 26.32 26.41 26.50 26.60 26.69 26.87 27.05 27.23 27.35 27.67 27.49 27.91 28.24 29.46 28.71 34.12 30.82 13.46 13.56 13.65 13.75 13.84 13.93 14.02 14.20 14.37 1455 14.66 14.80 15.21 14.98 15.52 15.98 21.20 18.00 16.69 9.11 9.02 9.20 9.29 9.38 9.47 9.72 9.55 9.89 10.16 10.05 10.29 10.46 10.97 10.67 11.39 12.06 13.27 16.26 6.88 6.97 7.06 7.14 7.23 7.31 7.40 7.56 7.72 7.87 7.98 8.10 8.47 8.26 8.75 9.15 10.92 9.78 13.75 5.65 5.74 5.82 5.91 6.07 5.99 6.16 6.31 6.62 6.47 6.72 6.84 7.19 6.99 7.46 7.85 8.45 12.25 9.55 4.86 4.95 5.12 5.03 5.20 5.28 5.36 5.67 5.52 5.81 5.91 6.03 6.37 6.18 7.01 6.63 7.59 11.26 8.65 4.40 4.31 4.48 4.57 4.65 4.73 4.81 4.96 5.11 5.26 5.47 5.35 5.61 6.06 5.80 6.42 8.02 6.99 10.56

2

10 II 12 13 14

5 6 7 8 9

2 3 4

I

,Xli

99 per cent points

CI\ -.J

VI

sa:><

m Z

."

> ."

4

5

6

7 9

3.58 3.35 3.13 2.93 2.74

3.74 3.51 3.29 3.09 2.90

3.95 3.71 3.49 3.28 3.09

4.23 3.99 3.76 3.55 3.35

4.62 4.37 4.14 3.92 3.72

5.24 4.98 4.73 4.50 4.28

6.35 6.07 5.79 5.54 5.30

9.18 8.83 8.49 8.18 7.88

30 40 60 120

3.78 3.73 3.69 3.65 3.61

3.94 3.89 3.85 3.81 3.77

4.15 4.10 4.06 4.02 3.98

4.43 4.38 4.34 4.30 4.26

4.84 4.79 4.74 4.70 4.66

5.46 5.41 5.36 5.32 5.28

6.60 6.54 6.49 6.44 6.40

25 26 27 28 29

9.48 9.41 9.34 9.28 9.23 3.45 3.22 3.01 2.81 2.62

3.64 3.60 3.56 3.52 3.48

3.96 3.88 3.81 3.75 3.69

4.09 4.01 3.94 3.88 3.83

4.26 4.18 4.11 4.05 3.99

4.47 4.39 4.32 4.26 4.20

4.76 4.68 4.61 4.54 4.49

5.17 5.09 5.02 4.95 4.89

5.82 5.73 5.65 5.58 5.52

9.94 9.83 9.73 9.63 9.55

20 21 22 23 24

6.99 6.89 6.81 6.73 6.66

4.54 4.38 4.25 4.14 4.04

4.67 4.52 4.39 4.28 4.18

4.85 4.69 4.56 4.44 4.34

5.07 4.91 4.78 4.66 4.56

5.37 5.21 5.07 4.96 4.85

5.80 5.64 5.50 5.37 5.27

6.48 6.30 6.16 6.03 5.92

10.80 10.58 10.38 10.22 10.07

15 16 17 18 19

7.70 7.51 7.35 7.21 7.09

5.97 5.54 5.20 4.94 4.72

6.12 5.68 5.35 5.08 4.86

6.30 5.86 5.52 5.25 5.03

6.54 6.10 5.76 5.48 5.26

6.87 6.42 6.07 5.79 5.56

7.34 6.88 6.52 6.23 6.00

12.83 12.23 11.75 11.37 11.06

10 11 12 13 14

8.08 7.60 7.23 6.93 6.68

13.77 10.39 8.51 7.34 6.54

13.96 10.57 8.68 7.50· 6.69

9.43 8.91 8.51 8.19 7.92

14.20 10.79 8.89 7.69 6.88

14.51 11.07 9.16 7.95 7.13

14.94 11.46 9.52 8.30 7.47

15.56 12.03 10.05 8.81 7.96

18.31 14.54 12.40 11.04 10.11

16.53 12.92 10.88 9.60 8.72

22.78 18.63 16.24 14.69 13.61

ao

2

10

12 15

20

i!4 30

40 60

120

co

3.34 3.12 2.90 2.71 2.52

3.54 3.49 3.45 3.41 3.38

3.85 3.77 3.70 3.64 3.59

4.42 4.27 4.14 4.03 3.93

5.85 5.42 5.09 4.82 4.60

13.62 10.25 8.38 7.21 6.42

3.01 2.97 2.93 2.89 2.86 2.82 2.60 2.39 2.19 2.00

3.20 3.15 3.11 3.07 3.04 3.01 2.78 2.57 2.37 2.19 3.18 2.95 2.74 2.54 2.36

3.37 3.33 3.28 3.25 3.21

3.32 3.24 3.18 3.12 3.06

3.50 3.43 3.36 3.30 3.25

3.68 3.60 3.54 3.47 3.42

3.88 3.73 3.61 3.50 3.40

5.27 4.86 4.53 4.27 4.06

12.90 9.59 7.75 6.61 5.83

4.07 3.92 3.79 3.68 3.59

5.47 5.05 4.72 4.46 4.25

13.15 9.81 7.97 6.81 6.03

4.25 4.10 3.97 3.86 3.76

5.66 5.24 4.91 4.64 4.43

13.38 10.03 8.18 7.01 6.23

2.73 2.50 2.29 2.09 1.90

2.92 2.87 2.83 2.79 2.76

3.22 3.15 3.08 3.02 2.97

3.79 3.64 3.51 3.40 3.31

5.17 4.76 4.43 4.17 3.96

12.78 9.47 7.65 6.50 5.73

2.63 2.40 2.19 1.98 1.79

2.82 2.77 2.73 2.69 2.66

3.12 3.05 2.98 2.92 2.87

3.69 3.54 3.41 3.30 3.21

5.07 4.65 4.33 4.07 3.86

12.66 9.36 7.53 6.40 5.62

2.52 2.30 2.08 1.87 1.67

2.72 2.67 2.63 2.59 2.56

3.02 2.95 2.88 2.82 2.77

3.58 3.44 3.31 3.20 3.11

4.97 4.55 4.23 3.97 3.76

12.53 9.24 7.42 6.29 5.52

2.42 2.18 1.96 1.75 1.53

2.61 2.56 2.52 2.48 2.45

2.92 2.84 2.77 2.71 2.66

3.48 3.33 3.21 3.10 3.00

4.86 4.44 4.12 3.87 3.66

12.40 9.12 7.31 6.18 5.41

1.36

2.30 2.06 1.83 1.61

2.50 2.45 2.41 2.37 2.33

2.81 2.73 2.66 2.60 2.55

3.37 3.22 3.10 2.99 2.89

4.75 4.34 4.01 3.76 3.55

12.27 9.00 7.19 6.06 5.30

2.18 1.93 1.69 1.43 1.00

2.38 2.33 2.29 2.25 2.21

2.69 2.61 2.55 2.48 2.43

3.26 3.11 2.98 2.87 2.78

4.64 4.23 3.90 3.65 3.44

12.14 8.88 7.08 5.95 5.19

24224 24426 24630 24836 24940 25044 25148 25253 25359 25465 16211 20000 21615 22500 23056 23437 23715 23925 24091 199.3 199.4 199.4 199.4 199.4 199.4 199.4 199.4 199.5 199.5 199.5 199.2 199.3 199.5 199.5 199.5 199.2 199.0 198.5 43.88 43.69 43.39 43.08 42.78 42.62 44.84 44.43 44.13 42.47 42.31 42.15 41.99 41.83 46.19 45.39 47.47 55.55 49.80 20.70 20.44 20.17 19.47 21.97 21.62 21.14 20.97 19.32 22.46 21.35 20.03 19.89 19.75 19.61 23.15 26.28 24.26 31.33

5 6 7 8 9

1 2 3 4

~I

Table IV. Percentage Points of the F Distribution (continued) 99.5 per cent points VI

><

m Z 0

> '"d '"d

00

0\

3

4

8.25 7.94 7.68 7.46 7.26

9.34 9.00 8.73 8.49 8.28

8.10 7.94 7.80 7.67 7.55

7.45 7.36 7.27 7.19 7.12

7.05 6.60 6.17 5.79 5.42

11.34 10.97 10.66 10.39 10.16

9.95 9.77 9.61 9.47 9.34

9.22 9.12 9.02 8.93 8.85

8.77 8.25 7.76 7.32 6.91

16.59 16.12 15.72 15.3"8 15.08

14.82 14.59 14.38 14.19 14.03

13.88 13.74 13.61 13.50 13.39

13.29 12.61 11.97 11.38 10.83

15 16 17 18 19

20 21 22 23 24

25 26 27 28 29

30 40 60 120

6

7 9

10

12

99.9 per cent points 8 IS

20

24

30

40

60

120 <Xl

5.53 5.13 4.76 4.42 4.10

5.88 5.80 5.73 5.66 5.59

6.46 6.32 6.19 6.08 5.98

7.57 7.27 7.02 6.81 6.62

10.48 9.58 8.89 8.35 7.92

29.75 20.81 16.21 13.49 11.71

5.12 4.73 4.37 4.04 3.74

5.46 5.38 5.31 5.24 5.18

6.02 5.88 5.76 5.65 5.55

7.09 6.81 7.56 6.35 6.18

9.92 9.05 8.38 7.86 7.43

28.84 20.03 15.52 12.86 11.13

4.82 4.44 4.09 3.77 3.47

5.15 5.07 5.00 4.93 4.87

5.69 5.56 5.44 5.33 5.23

6.74 6.46 6.22 6.02 5.85

9.52 8.66 8.00 7.49 7.08

28.16 19.46 15.02 12.40 10.70

4.58 4.21 3.87 3.55 3.27

4.91 4.83 4.76 4.69 4.64

5.44 5.31 5.19 5.09 4.99

6.47 6.19 5.96 5.76 5.59

9.20 8.35 7.71 7.21 6.80

27.64 19.03 14.63 12.04 10.37

4.39 4.02 3.69 3.38 3.10

4 ..71 4.64 4.57 4.50 4.45

5.24 5.11 4.99 4.89 4.80

6.26 5.98 5.75 5.56 5.39

8.96 8.12 7.48 6.98 6.58

27.24 18.69 14.33 11.77 10.11

4.24 3.87 3.54 3.24 2.96

4.56 4.48 4.41 4.35 4.29

5.08 4.95 4.83 4.73 4.64

6.08 5.81 5.58 5.39 5.22

8.75 7.92 7.29 6.80 6.40

26.92 18.41 14.08 11.54 9.89

4.00 3.64 3.31 3.02 2.74

4.31 4.24 4.17 4.11 4.05

4.82 4.70 4.58 4.48 4.39

5.81 5.55 5.32 5.13 4.97

8.45 7.63 7.00 6.52 6.13

26.42 17.99 13.71 11.19 9.57

3.75 3.40 3.08 2.78 2.51

4.06 3.99 3.92 3.86 3.80

4.56 4.44 4.33 4.23 4.14

5.S4 5.27 5.05 4.87 4.70

8.13 7.32 6.71 6.23 5.85

25.91 17.56 13.32 10.84 9.24

3.49 3.15 2.83 2.53 2.27

3.79 3.72 3.66 3.60 3.54

4.29 4.17 4.06 3.96 3.87

5.25 4.99 4.78 4.59 4.43

7.80 7.01 6.40 5.93 5.56

25.39 17.12 12.93 10.48 8.90

3.36 3.01 2.69 2.40 2.13

3.66 3.59 3.52 3.46 3.41

4.15 4.03 3.92 3.82 3.74

5.10 4.85 4.63 4.45 4.29

7.64 6.85 6.25 5.78 5.41

25.14 16.89 12.73 10.30 8.72

3.22 2.87 2.55 2.26 1.99

3.52 3.44 3.38 3.32 3.27

4.00 3.88 3.78 3.68 3.59

4.95 4.70 4.48 4.30 4.14

7.47 6.68 6.09 5.63 5.25

24.87 16.67 12.53 10.11 8.55

3.07 2.73 2.41 2.11 1.84

3.37 3.30 3.23 3.18 3.12

3.86 3.74 3.63 3.53 3.45

4.80 4.54 4.33 415 3.99

7.30 6.52 5.93 5.47 5.10

24.60 16.44 12.33 9.92 8.37

2.92 2:57 2.25 1.95 1.66

3.22 3.15 3.08 3.02 2.97

3.70 3.58 3.48 3.38 3.29

4.64 4.39 4.18 4.00 3.84

7.12 6.35 5.76 5.30 4.94

24.33 16.21 12.12 9.73 8.19

2.76 2.41 2.08 1.76 1.45

3.06 2.99 2.92 2.86 2.81

3.54 3.42 3.32 3.22 3.14

4.47 4.23 4.02 3.84 3.68

6.94 6.17 5.59 5.14 4.77

24.06 15.99 11.91 9.53 8.00

2.59 2.23 1.89 1.54 1.00

2.89 2.82 2.75 2.69 2.64

3.38 3.26 3.15 3.05 2.97

4.31 4.06 3.85 3.67 3.51

6.76 6.00 5.42 4.97 4.60

23.79 15.75 11.70 9.33 7.81

"Abridged from Table 18 of Biometrika Tables for Statisticians. vol. I. edited by E. S. Pearson and H. O. Hartley, Cambridge University Press, Cambridge, 1954, and Table V of Statistical Tablesfor Biological, Agricultural, and Medical Research, R. A. Fisher and F. Yates, Oliver &. Boyd, Edinburgh, 1953. tMultiply these entries by 100.

<Xl

6.12 5.70 5.31 4.95 4.62

6.49 6.41 6.33 6.25 6.19

7.10 6.95 6.81 6.69 6.59

11.28 10.35 9.63 9.07 8.62

12.55 11.56 10.80 10.21 9.73

14.91 13.81 12.97 12.31 11.78

21.04 19.69 18.64 17.81 17.14

10

12 13 14

31.09 21.92 17.19 14.39 12.56

33.20 23.70 18.77 15.83 13.90

37.12 27.00 21.69 18.49 16.39

47.18 35.51 29.25 25.42 22.86

II

5

4053t 5000t 5404t 5625t 5764t 5859t 5929t 5981t 6023t 6056t 6107t 6158t 6209t 6235t 626 It 6287t 6313t 6340t 6366t 999.2 999.2 999.0 999.3 999.3 999.4 999.4 998.5 999.4 999.4 999.4 999.4 999.4 999.5 999.5 999.5 999.5 999.5 999.5 167.0 148.5 141.1 137.1 134.6 132.8 130.6 129.9 129.2 131.6 128.3 127.4 126.4 125.9 125.4 125.0 124.5 124.0 123.5 74.14 61.25 56.18 53.44 51.71 50.53 49.66 49.00 48.47 48.05 47.41 46.76 46.10 45.77 45.43 45.09 44.75 44.40 44.05

2

5 6 7 8 9

2 3 4

I

1;X,

1

VI

1,0

0'1

~

~

~

570

APPENDIX

Table V. y = 2 arcsin

v;*

'000

'001

'002

'003

'°°4

'005

'006

'007

'008

'009

0'1°96 0'2285 0'3°45 0'3653 °'4 178

0'1266 0'2372 0'3111 0'37°9 0'4227

0'1415 0'2456 °'3 176 0'3764 0'4275

0'1551 0'2537 0'3239 0'3818 0'4323

0'1675 0'2615 0'33°1 0'3871 0'4371

0'1791 °' 269 1 °'3362 0'3924 °'44 18

0'1900 0'2766 °'3423 °'3976 °'4464

'00 '01 '02 '°3 '°4

0'0000

0'0633

0'2°°3 0'2838 0'3482 0'4°27

0'29°9 0'354° °'4°78

0'0895 0'2195 0'2977 0'3597 0'4128

'°5 '06 '°7 '08 '°9

0'4510 0'4949 0'5355 0'5735 0.6094

0'4556 0'499 1 0'5394 0'5772 0.6129

°'4601 0'5°33 0'5433 0'5808 0.6163

°'4646 0'5°74 0'5472 0'5845 0.6198

0'469° 0'5115 0'5510 0'5881 0.6232

0'4734 °'5 156 °'5548 0'5917 0·6266

0'4778 0'5196 0'5586 0'5953 0·63°0

0'4822 0'5236 0'5624 0'5988 0·6334

°'4 864 °'5 276 0'5661 0.6024 0.6368

0'49°7 0'5316 °'5698 0'6°59 0·64°2

'10 '11 '12 '13 '14

0,6435 0.6761 0'7°75 0'7377 0'7 6 7°

0.6468 0·6793 °'7 106 0'74°7 0'7699

0·65°1 0·6825 0'7136 0'7437 0'7727

0·6534 0.6857 0'7167 °'7466 °'775 6

0·6600 0.6920 °'7 22 7 0'7525 °'781 3

0·6632 0.6951 0'7258 0'7554 °'784 1

0.6729 0'7°44 0'7347 °'7641 °'79 26

0'7954 0·823° 0.8500 0.8763 0'9021

0'7982 0.8258 0·8526 0.8789 °'9046

0·8010 0.8285 0·8553 0.8815 0'907 1

0.8°38 0·8312 0·8579 0.8841 0'9097

0·8°93 0.8366 0.8632 0.8892 0'9147

0·8121 0·8393 0.8658 0.8918 0'9 173

0·6665 0.6982 °'7 288 °'75 83 °'7 87° 0.8148 0.8420 0·8685 0·8944 0'9198

0·6697 0'7°13 0'7318 °'7 612 °'7898

'15 '16 '17 '18 '19

0.6567 0·6888 °'7 197 0'7495 °'7785 0.8065 0·8339 0·8606 0'8867 0'9122

0.8176 0.8446 0'8711 0·89°9 0'9223

0·82°3 0·8473 0·8737 0·8995 °'9 248

'20 '21 '22 '23 '24

0'9273 0'9521 °'97 64 1'0°°4 1'0239

°'9 298 0'9545 °'9788 1'0027 1'0263

0'9323 °'957° 0'9812 1'0051 1'0286

°'9348 0'9594 °'9836 1'0075 1'°310

0'9373 0'961 9 0'9860 1'°°98 1'°333

0'9397 °'9 643 °'9 884 1'°356

0'9422 °'9 66 7 0'99°8 1'°146 1'0,379

0'9447 0'9692 0'9932 1'010g 1'°4°3

0'9471 °'97 16 °'995 6 1'0193 1'°426

°'949 6 0'974° °'9980 1'0216 1'°449

'25 '26 '27 '28 '29

1'°47 2 1'°701 1'°928 1'1152 1'1373

1'°495 1'°724 1'°95 1 1'1174 1'1396

1'°5 18 l'oi47 1'°973 1'I196 1'1418

1'°54 1 1'°77° 1'°995 1'1219 1'144°

1'°56 4 1'°792 1'1018 1'1241 1'1461

1'°587 1'0815 1'1°4° 1'1263 1'1483

1'0610 1'0838 1'1063 1'1285 1'15°5

1'0633 1'0860 1'1085 1'13°7 1'15>!7

1'065° 1'0883 1'11°7 1'1329 1'1549

1'0679 1'°905 1'113° 1'1351 1'1571

'3° '31 '32 '33 '34

1'1593 1'1810 1'2025 1'2239 1'2451

1'1615 1'1832 1'2°47 1'2260 1'2472

1'1636 1'1853 1'2068 1'2281 1'2493

1'1658 1'1875 1'2°9° 1'23°3 1'2514

1'1680 1'1896 1'2324 1'2535

1'17°2 1'1918 1'2132 1'2345 1'2556

1'1723 1'1939 1'2154 1'2366 1'2577

1'1745 1'1961 1'21 75 1'2387 1'2598

1'1767 1'1982 1'2196 1'24°8 1'26'9

1'1788 1'2°°4 1'221 7 1'243° 1'264°

'35 '36 '37 '38 '39

1'2661 1' 287° 1'3°78 1'3284 1'349°

1'2682 1'2891 1'3°98 1'33°5 1'3510

1'27°3 1'2912 1'3119 1'3325 1'3531

1'2724 1'2932 1'3'40 1'3346 1'3551

1'2745 1'2953 1'3161 1'3367 1'3572

1'2766 1'2974 1'3181 1'3387 "359 2

1'2787 1'2995 1'3202 1'34°8 1'36 '3

1'28°7 1'30,6 1'3222 "34 28 "3633

1'2828 1'3°36 1'3243 1'3449 1'3654

1'2849 "3°57 1'3264 1'3469 1'3674

'4° '4 1 '4 2 '43 '44

1'3694 1'3898 1'4101 1'43°3 1'45°5

1'3715 1'3918 1'4121 1'4324 1"45 25

1'3735 1'3939 1'4142 1'4344 1'4545

1'3756 1'3959 1'4162 1'4364 1'4565

1'3776 1'3979 1'4182 1'4384 1'4586

1'3796 1'4°°0 1'4202 1'44°4 1'4606

1'3817 "4020 1'4222 1'4424 1'4626

1'3837 1'4°4° "4243 1'4445 1'4646

1'3857 1'4°6, 1'4263 1'4465 1'4666

1'3878 "4°81 1'4283 "44 85 "4686

'45 '4 6 '47 '4 8 '49

1'47°6 1'4907 1'5108 1'53°8 1'55°8

1'4726 1'4927 I'SI28 1'5328 1'5528

1'4746 1'4947 1'5148 1'5348 1'5548

1'476 7 1'496 7 1'5168 1'5368 1'5568

1'4787 1'4987 1'5188 1'5388 1'5588

1'48°7 1'5007 1'5208 1'54°8 1'5608

1'4827 1'5°27 1'5228 1'5428 1'5628

1'4847 1'5°48 1'5248 "5448 1'5648

1'4867 1'5°68 1'5268 1'5468 1'5668

1'4887 "5°88 1'5288 "5488 1'5688

0'2101

1'2111

1'0122

Example: 2 arcsin 1/0,296 = "'505.

571

APPENDIX

Table V. y = 2 arcsin

V; (continued)

'000

'001

'002

'003

'004

'005

'006

'007

'008

'009

1'5708 1'5908 1,6108 1,63°8 1'65°9 1,6710 1'69II 1'7 II3 1'7315 1'7518

1'5728 1'5928 1,6128 1,6328 1,6529

1'5748 1'5948 1'6148 1,6348 1,6549

1'5768 1'5968 1,6168 1,6368 1,6569

1'5788 1'5988 1,6188 1,6388 1,6589

1'5808 1,6008 1,6208 1,64°9 1'6609

1'5828 1,6028 1,6228 1,6429 1,6629

1'5848 1'6048 1,6248 1,6449 1,6649

1'5868 1,6068 1,6268 1,6469 1,6669

1'5888 1,6088 1,6288 1,6489 1,6690

1,673° 1,6931 1'7133 1'7335 1'7538

1,675° 1,6951 1'7153 1'7355 1'7559

1,677° 1'6971 1'7173 1'7376 1'7579

1,6790 1,6992 1'7193 1'7396 1'7599

1,6810 1'7° 12 1'7214 1'7416 1'7620

1,683° 1'7°32 1'7234 1'7437 1'7640

1,685° 1'7°52 1'7254 1'7457 1'7660

1,6871 1'7°72 1'7274 1-7477 1'7681

1,6891 1'7°92 1'7295 1'7498 1'77°1

1'7722 1'7926 1,8132 1,8338 1,8546

1'7742 1'7947 1,8152 1,8359 1,8567

1'7762 1'7967 1'8173 1,8380 1,8588

1'7783 1'7988 1'8193 1,84°0 1,8608

1'7803 1,8008 1,8214 1,8421 1,8629

1'7824 1'8029 1,8235 1,8442 1,865°

1'7844 1,8°<\9 1,8255 1,8463 1,8671

1'7865 1'8070 1'8276 1,8483 1,8692

1'7885 1,80go 1'8297 1,85°4 1,8713

1'7906 1,81II 1,8317 1,8525 1,8734

,65 ,66 ,67 ,68 ,69

1,8755 1,8965 1'9177 1'9391 1'9606

1,8776 1,8986 1'9198 1'9412 1'9628

1,8797 l'go08 1'9220 1'9434 1'9649

1,8818 l'go29 1'9241 1'9455 1'9671

1,8839 l'go50 1'9262 1'9477 1'9693

1,8860 l'go71 1'9284 1'9498 1'97 14

1,8881 1'9°92 1'9305 1'9520 1'9736

1,89°2 1'9II3 1'9326 1'9541 1'9758

1,8923 1'9135 1'9348 1'9563 1'9780

1,8944 1'9156 1'9369 1'9584 1'9801

'7° '7 1 '7 2 '73 '74

1'9823 2'0042 2'0264 2'0488 2'0714

1'9845 2'0064 2'0286 2'0510 2'0737

1'9867 2'0087 2'0309 2'0533 2'0760

1'9889 2'0109 2'0331 2'0556 2'0783

1'99II 2'0131 2-0353 2'0578 2'0806

1'9932 2'0153 2'0376 2'0601 2'0829

1'9954 2'0175 2'0398 2'0624 2'0852

1'9976 2'0197 2'0420 2'0646 2'0875

1'9998 2'0219 2'0443 2,0669 2'0898

2'0020

2'0242 2'0465 2'°692 2'0921

'75 '76 '77 '78 '79 ,8o ,81 ,82 ,83 ,84

2'0944 2'II76 2'1412 2'1652 2'1895

2'0967 2'I436 2'1676 2'1920

2'0990 2'1223 2'1460 2'1700 2'W44

2'1013 2' 1247 2'1484 2'1724 2'1969

2'1037 2' 1270 2'1508 2'1749 2'1994

2'1060 2'1294 2'1532 2'1773 2'2019

2'1083 2'1318 2'1556 2'1797 2'2043

2'II06 2'1341 2'1580 2'1822 2'2068

2'II30 2'1365 2'1604 2'1846 2'2093

2'1153 2'I389 2'1628 2' 1871 2'2II8

2'2143 2'2395 2'2653 2'2916 2'3186

2'2168 2'2421 2'2679 2'2943 2'3213

2'2193 2'2446 2'2705 2'2970 2'3 24°

2'2218 2'2472 2'2731 2'2996 2'3268

2'2243 2'2498 2'2758 2'3023 2'3295

2'2269 2'2784 2'3050 2'3323

2'2294 2'2549 2'2810 2'3°77 2'3351

2'2319 2'2575 2'2837 2'31°4 2'3378

2'2345 2'2601 2'2863 2'3131 2'3406

2'2370 2'2627 2' 28 9° 2'3158 2'3434

,85 ,86 ,87 ,88 ,89

2'3462 2'3746 2'4039 2'4341 2'4655

'2'3490 2'3775 2'4068 2'4372 2'4687

2'3518 2'3804 2'4098 2'4403 2'4719

2'3546 2'3833 2'4128 2'4434 2'4751

2'3575 2'3862 2'4158 2'4465 2'4783

2'3603 2'3891 2'4189 2'4496 2'4816

2'3631 2'3921 2'4219 2'4528 2'4849

2-3660 2'3950 2'4249 2'4559 2'4882

2'3689 2'3979 2'4280 2'4591 2'4915

2'3717 2'4009 2'4310 2'4623 2'4948

'9° '91 '92 '93 '94

2'4981 2'5322 2'5681 2,6061 2,6467

2'5014 2'5357 2'5718 2,6100 2,65°9

2'5048 2'5392 2'5755 2'6140 2,6551

2'5082 2'5428 2'5792 2' 61 79 2,6594

2'5 115 2'5463 2'5830 2,6220 2,6638

2'5149 2'5499 2'5868 2,6260 2,6681

2'5184 2'5535 2'5go 6 2,63°1 2,6725

2'5218 2'5571 2'5944 2,6342 2,677°

2'5253 2'5607 2'5983 2,6383 2,6815

2'5287 2'5644 2,6022 2,6425 2,6860

'95 '96 '97 '98 '99 1'00

2,69°6 2'7389 2'7934 2,8578 2'9413 3'1416

2,6952 2'7440 2'7993 2,865° 2'9516

2,6998 2'7492 2,8053 2,8725 2'9625

2'7045 2'7545 2,8115 2,8801 2'9741

2'7°93 2'7598 2'8177 2,8879 2'9865

2'7141 2'7652 2'8240 2,8960 3'0001

2'7189 2'77°7 2,83°5 2'go44 3'0150

2'7238 2'7762 2,8371 2'9131 3'0320

2'7288 2'7819 2,8438 2'9221 3'0521

2'7338 2'7876 2,85°7 2'9314 3'0783

'50 '51 '52 '53 '54 '55 '56 '57 '58 '59 ,60 '61 '62 ,63 ,64

2'1200

2'~523

Example: 2 arcsin 110.724 = 2'0353,

*Reproduced from Table XII of Statistical Tables 'lIld Formllias by A, Hald, John Wiley & Sons, 'New York, 1952,

572

APPENDIX

Table VI.

Values of dn and Fractional Points of the Distribution of the Range*

n

d"

0.001

0.005

0.025

0.050

0.95

0.975

0.995

0.999

2 3 4 5 6 7 8 9 IO II 12

1.128 1.693 2.059 2.326 2.534 2.704 2.847 2.970 3.078 3.173 3.258

0.00 0.06 0.20 0.37 0.54 0.69 0.83 0.96 1.08 1.20 1.30

0.01 0.13 0.34 0.55 0.75 0.92 1.08 1.21 1.33 1.45 1.55

0.04 0.30 0.59 0.85 1.06 1.25 1.41 1.55 1.67 1.78 1.88

0.09 0.43 0.76 1.03 1.25 1.44 1.60 1.74 1.86 1.97 2.07

2.77 3.31 3.63 3.86 4.03 4.17 4.29 4.39 4.47 4.55 4.62

3.17 3.68 3.98 4.20 4.36 4.49 4.61 4-70 4.:79 4,;86

3.97 4.42 4.69 4.89 5.03 5.15 5.26 5.34 5.42 5.49 5.54

4.65 5.06 5.31 5.48 5.62 5.73 5.82 5.90 5.97 6.04 6.09

4192

*Abridged from Tables 20 and 22 of Biometrika Tables lor Statisticians, vol. I, edited by E. S. Pearson and H. O. Hartley. Cambridge University Press, Cambridge (I 954).

Table VII.

lip

P

lip

P

lip

0.20 0.10 0.05 0.025 0.01 0.005 0.001 0.0005

-0.8416 -1.2816 -1.6449 -1.9600 -2.3263 -2.5758 -3.0902 -3.2905

0.80 0.90 0.95 0.975 0.99 0.995 0.999 0.9995

0.8416 1.2816 1.6449 1.9600 2.3263 2.5758 3.0902 3.2905

573

APPENDIX

Table VIII. Logarithms of n !* n I

2 3 4 5 6 7 8 9 10

I

I

I

I

I

log nl

n

logn!

n

I og n!

n

0'0000 0'3010 0'7782 1'3802 2'0792

51' 52 53 54 55

66'lgo6 67'go66 6g,6309 71'3633 73' 1037

101 102 103 104 105

159'9743 161'9829 163'995 8 166'0128 168'°34°

151 152 153 154 155

264'9359 267'l 1 77 269'302 4 2 71 '4 899 273' 6803

201 202 203 204 205

377'2001 379'5°54 381 '8129 384'1226 386 '4343

2,8573 3'7°24 4,6055 5'5598

74'8519 76'6077 78'3712 80'1420 81'9202

106 107 108 109 IIO

170'0593 172'0887 174'1221 176'1595 178'2009

156 157 158 159 160

275,8734 278'0693 280'2679 282'4693 284'6735

206 20 7 208 209 210

388'7482 391'0642 393'3822 395'7° 24 398'0246

lognl

n

log nl

6'55gB

56 57 58 59 60

n 12 13 14 15

7,6012 8,6803 9'7943 10'9404 12'n65

61 62 63 64 65

83'7055 85'4979 87'2972 89'1034 go'9163

III II2 II3 II4 II5

180'2462 182'2955' 184'3485 186'4°54 188'4661

161 162 163 164 165

286,8803 -2 89'0898 291'3020 293'5168 295'7343

2II 212 21 3 214 215

400 '3489 402 '6752 405'0036 4°7'334° 4°9'6664

16 17 18 19 20

13'3206 14'55II 15,8063 17'°851 18'3861

66 68 6g 7°

92 '7359 94'561 9 96 '3945 98'2333 100'0784

n6 II7 II8 II9 120

19o'5306 192'5988 194'6707 196'7462 198,8254

166 167 168 169 17°

297'9544 300'1771 302'4024 304'6303 306,8608

216 21 7 218 21 9 220

412'0009 414'3373 416'6758 419'0162 421 '358 7

21 22 23 24 25

19'7083 21'0508 22'4125 23'7927 25'Igo6

71 72 73 74 75

101'9297 103'7 870 105,6503 107'5 196 109'3946

121 122 123 124 125

200,go82 202'9945 205'0844 2°7'1779 2°9'2748

171 172 173 174 175

309'0938 31 1'3 2 93 31 3'5674 31 5'80 79 31 8'0509

221 222 223 224 225

423'7°3 1 426'0494 428 '3977 430 '74 80 433'1002

26 27 28 29 30

26,6056 28'°37° 29'4841 3°'9465 3 2 '4237

76 77 78 79 80

III'2754 II3'I619 II5'054° n6'9516 n8'8547

126 127 128 129 130

2II'3751 213'47go 215'5862 21 7'6967 219,8107

176 177 178 179 180

320'2965 32 2'5444 3 24'7948 3 27'0477 3 29'3030

226 227 228 229 230

435'4543 437'8103 440'1682 442'5281 444'8898

31 32 33 34 35

33'9 150 35'4202 36 '9387 38 '4702 40'0142

81 82 83 84 85

120'7632 122,67.70 124'5961 126'5204 128'4498

131 132 133 134 135

221'9280 224'0485 226'1,24 228'2995 230'4298

181 182 183 184 185

33 1 '5606 333' 820 7 336'0832 338 '3480 340 ' 61 52

231 232 233 234 235

447'2534 449' 6189 45 I 'gB62 454'3555 456 '7 265

36 37 38 39 40

41'57°5 43'1387 44'7 185 46 '3096 47'9 II6

B6 87 88 89 go

130'3843 132'3238 134'2683 136'2177 138'1719

136 137 138 139 140

232'5634 234'7001 236'8400 238'9830 241'1291

186 187 188 189 19o

34 2,8847 345'1565 3 47'4307 34 9'7°7 1 35 1'9859

236 237 238 239 240

459'0994 461 '474 2 463'8508 466'2292 468,6094

41 42 43 44 45

49'5244 51'1477 52 '78II 54'4246 56'0778

91 92 93 94 95

140'1310 142'0948 144'0632 146'0364 148'0141

141 142 143 144 145

243'2783 245'4306 247'5860 249'7443 25 1'9°57

191 192 193 194 195

35 4'2669 356'5502 358,8358 36l'I236 363'4136

241 242 243 244 245

47°'9914 473'375 2 475'7608 478'1482 480 '5374

46 47 48 49 50

57'7406 59'4 127 61'0939 62'7841 64'4831

96 97 98 99 100

149'9964 151'9831 153'9744 155'9700 157'97°0

146 147 148 149 150

254'0700 256'2374 258 '4076 260'5808 262'7569

196 197 198 199 200

365'7059 368'0003 370'2970 372'5959 374,8969

246 247 248 249 250

482 '9283 485'3 210 487'7 154 4go'III6 49 2 '50 96

67

*Abridged from Table XIII of Statistical Tables and Formulas by A, Hald, John Wiley & Sons, New York, 1952,

3.64 3.46 3.34 3.26 3.20

3.15 3.11 3.08 3.06 3.03

3.01 3.00 2.98 2.97 2.96

2.95 2.92 2.89 2.86

2.83 2.80 2.77

5 6 7 8 9

10 II 12 \3 14

15 16 17 18 19

20 24 30 40

60 120

<Xl

18.0 6.09 4.50 3.93

---

2

I 2 3 4

~ f~

3.40 3.36 3.31

3.58 3.53 3.49 3.44

3.67 3.65 3.63 3.61 3.59

3.74 3.69 3.63

3.96 3.90 3.84 3.79

4.08 4.05 4.02 4.00 3.98

4.33 4.26 4.20 4.15 4.11

5.22 4.90 4.68 4.53 4.42

4.60 4.34 4.16 4.04 3.95

3.88 3.82 3.77 3.73 3.70

32.8 9.8 6.82 5.76

4

27.0 8.3 5.91 5.04

3

3.98 3.92 3.86

4.23 4.17 4.10 4.04

4.37 4.33 4.30 4.28 4.25

4.65 4.57 4.51 4.45 4.41

5.67 5.31 5.06 4.89 4.76

37.1 10.9 7.50 6.29

5

4.16 4.10 4.03

4.45 4.37 4.30 4.23

4.60 4.56 4.52 4.49 4.47

4.91 4.82 4.75 4.69 4.64

6.03 5.63 5.36 5.17 5.02

40.4 11.7 8.04 6.71

6

4.31 4.24 4.17

4.62 4.54 4.46 4.39

4.78 4.74 4.71 4.67 4.65

5.12 5.03 4.95 4.88 4.83

6.33 5.89 5.61 5.40 5.24

43.1 12.4 8.48 7.05

• 7

P 8

0.95

4.55 4.48 4.39

4.90 4.81 4.72 4.63

4.77 4.68 4.60 4.52 4.44 4.36 4.29

5.08 5.03 4.99 4.96 4.92

5.46 5.35 5.27 5.19 5.13

6.80 6.32 6.00 5.77 5.60

47.4 \3.5 9.18 7.60

9

4.94 4.90 4.86 4.82 4.79

5.30 5.20 5.12 5.05 4.99

6.58 6.12 5.82 5.60 5.43

45.4 13.0 8.85 7.35

=

4.65 4.56 4.47

5.01 4.92 4.83 4.74

5.20 5.15 5.11 5.07 5.04

5.60 5.49 5.40 5.32 5.25

6.99 6.49 6.16 5.92 5.74

49.1 14.0 9.46 7.83

10

Table IX. Fractional Points of the Studentized Range,

4.73 4.64 4.55

5.11 5.01 4.92 4.82

5.31 5.26 5.21 5.17 5.14

5.72 5.61 5.51 5.43 5.36

7.17 6.65 6.30 6.05 5.87

50.6 14.4 9.72 8.03

11

q = (Xmax

12

4.81 4.72 4.62

5.20 5.10 5.00 4.91

5.40 5.35 5.31 5.27 5.23

5.83 5.71 5.62 5.53 5.46

7.32 6.79 6.43 6.18 5.98

4.88 4.78 4.68

5.28 5.18 5.08 4.89

5.49 5.44 5.39 5.35 5.32

5.93 5.81 5.71 5.63 5.55

7.47 6.92 6.55 6.29 6.09

53.2 15.1 10.1 8.37

13

Xmin)/S*

52.0 14.7 9.95 8.21

-

4.94 4.84 4.74

5.36 5.25 5.15 5.05

5.58 5.52 5.47 5.43 5.39

6.03 5.90 5.80 5.71 5.64

7.60 7.03 6.66 6.39 6.19

54.3 15.4 10.3 8.52

14

5.00 4.90 4.80

5.43 5.32 5.21 5.11

5.65 5.59 5.55 5.50 5.46

6.11 5.99 5.88 5.79 5.72

7.72 7.14 6.76 6.48 6.28

55.4 15.7 10.5 8.66

15

VI

><

Z tl

ttl

'1;j

> '1;j

~

-..l

4.48 4.39 4.32 4.26 4.21

4.17 4.13 4.10 4.07 4.05

4.02 3.96 3.89 3.82

3.76 3.70 3.64

10 11 12 13 14

15 16 17 18 19

20 24 30 40

60 120

4.28 4.20 4.12

4.64 4.54 4.45 4.37

4.83 4.78 4.74 4.70 4.67

4.60 4.50 4.40

5.02 4.91 4.80 4.70

5.25 5.19 5.14 5.09 5.05

5.77 5.62 5;50 540 5.32

7.80 7.03 6.54 6.20 5.96

164 22.3 12.2 9.17

4

4.82 4.71 4.60

5.29 5.17 5.05 4.93

5.56 5.49 5.43 5.38 5.33

6.14 5.97 5.84 5.73 5.63

8.42 7.56 7.01 6.63 6.35

186 24.7 13.3 9.96

5

5.79 5.61 5.45 5.73 5.56 5.40 5.67 5.51 5.35 5.60 5.44 5.29 5.53 5.38 5.23

5.45 5.30 5.16

4.99 4.87 4.76

5.13 5.01 4.88

5.36 5.21 5.08

6.52 6.33 6.14 5.96 6.45 6.26 6.08 5.90 6.37 6.19 6.01 5.84 6.29 6.11 5.93 5.77 6.19 6.02 5.85 5.69 6.09 5.92 5.76 5.60 5.97 5.81 5.65 5.50

5.84 5.69 5.54 5.39

5.69 5.54 5.40 5.27

5.51 5.37 5.24 5.11 5.25 5.12 4.99

6.93 6.82 6.73 6.65 6.58 6.84 6.74 6.66 6.58 6.51 6.76 6.66 6.57 6.50 6.43 6.66 6.56 6.48 6.41 6.34

6.55 6.46 6.38 6.31 6.2.5

6.44 6.35 6.27 6.20 6.14

6.31 6.22 6.15 6.08 6.02

6.16 6.08 6.01 5.94 5.89

5.99 5.92 5.85 5.79 5.73

5.80 5.72 5.66 5.60 5.55

7.81 7.56 7.36 7.19 7.05

7.71 7.46 7.26 7.10 6.96 7.60 7.36 7.17 7.01 6.87 7.48 7.25 7.06 6.90 6.77

7.36 7.13 6.94 6.79 6.66

11.2 9.95 9.12 8.55 8.13 11.1 9.81 9.00 8.44 8.03

10.9 9.65 8.86 8.31 7.91

10.7 9.49 8.71 8.18 7.78

7.21 6.99 6.81 6.67 6.54

9.32 8.32 7.68 7.24 6.91 7.05 6.84 6.67 6.53 6.41

6.43 6.25 6.10 5.98 5.88

8.91 7.97 7.37 6.96 6.66 6.87 6.67 6.51 6.37 6.26

277 35.4 18.5 13.5

272 34.8 18.2 13.3

266 34.1 17.9 13.1

260 33.4 17.5 12.8

253 32.6 17.1 12.6

246 31.7 16.7 12.3

6.67 6.48 6.32 6.19 6.08

15

14

13

12

11

10

10.5 9.30 8.55 8.03 7.65

237 30.7 16.2 11.9

9

10.2 9.10 8.37 7.87 7.49

227 29.5 15.6 11.5

8

9.97 8.87 8.17 7.68 7.32

216 28.2 15.0 11.1

7

9.67 8.61 7.94 7.47 7.13

202 26.6 14.2 10.6

6

·Abridged from Table 29 of Biometrika Tables for Statisticians. vol. I. edited by E. S. Pearson and H. O. Hartley. Cambridge University Press. Cambridge. 1954.

co

6.97 6.33 5.92 5.63 5.43

5.70 5.24 4.95 4.74 4.60

5 6 7 8 9

5.27 5.14 5.04 4.96 4.89

135 19.0 10.6 8.12

90 14.0 8.26 6.51

2

1 2 3 4

Ixl

P = 0.99

VI

-....)

VI

saX

Z

tTl

'"C)

> '"C)

576

APPENDIX

Fractional Points of the Largest of k Variance Ratios with One Degree of Freedom in the Numerator*

Table X.

P

X 10 12 15 20 30 60 00

0.95

=

I

2

3

4

5

6

7

8

9

10

4.96 4.75 4.54 4.35 4.17 4.00 3.84

6.79 6.44 6.12 5.81 5.52 5.25 5.00

8.00 7.53 7.11 6.72 6.36 6.02 5.70

8.96 8.37 7.86 7.40 6.97 6.58 6.21

9.78 9.06 8.47 7.94 7.46 7.02 6.60

10.52 9.68 8.98 8.39 7.87 7.38 6.92

1l.l8 10.20 9.43 8.79 8.21 7.68 7.20

11.79 10.68 9.82 9.13 8.51 7.96 7.44

12.36 11.12 10.19 9.44 8.79 8.20 7.65

12.87 11.53 10.5 2 9.71 9.03 8.41 7.84

P = 0.99

X

I

2

3

4

5

6

7

8

9

10

10 12 15 20 30 60

10.04 9.33 8.68 8.10 7.56 7.08 6.63

13.17 11.88 10.82 9.93 9.16 8.49 7.88

15.08 13.52 12.18 11.08 10.14 9.34 8.61

16.43 14.73 13.21 11.93 10.86 9.95 9.15

17.43 15.69 14.03 12.61 11.43 10.43 9.54

18.25 16.47 14.72 13.19 11.90 10.82 9.87

18.91 17.12 15.30 13.67 12.31 11.15 10.16

19.48 17.68 15.81 14.09 12.66 11.45 10.41

19.97 18.16 16.26 14.49 12.97 11.72 10.62

20.41 18.60 16.66 14.83 13.26 11.95 10.82

00

* Reproduced from Table 19 of Biometrika Tables for Statisticians, vol. I, edited by E. S. Pearson and H. O. Hartley. Cambridge University Press, Cambridge (1954).

577

APPENDIX

Table XI.

Random Sampling Numbers*

15 77 01 64 69 85 40 51 40 10 47 69 35 90 95 13 26 87 40 20 10 55 33 20 47

69 15 16 40 54

58 33 17 81 16

40 94 45 46 86

16 65 29 09 16

60 57 16 74 59

20 62 70 99 34

00 94 48 16 71

84 1>4 02 92 55

9B

81 II 86 08 II

22 99 99 84

28 05 59 85 03

26 57 33 19 48

46 22 93 01 17

66 71 28 23 60

36 77 58 II 13

86 99 34 74 38

34 II 34 41 91

49 14 07 69 83

00

66 68 32 00 71

17 12 24 79 23

05 65 59 31 91

06 50 68 31 59

67 89 53 05 46

26 18 31 36 44

77 74 55 48 45

14 42 73 75 49

85 07 47 16 25

40 52 68 50 15 69 16 49 79 00 21 II 36 12 07

60 86 69 42 25

41 97 80 44 90

94 40 76 84 89

25 16 46 55

18 88 60 84 25

62 14 58 83 83

20 17 53 20 47

94 73 07 49 17

03 92 04 17 23

71 07 53 12 93

60 26 45 17 93 II 93 45 66 94 94 18 2~ 93 34 61 99 56 14 39

92 15 13 16 16

63 89 70 14 92

59 72 51 15 46

73 47 21 99 90

21 46 03 60 39

67 94 18 44 99

80 78 50 62 64

00 25 58 25 56 10 65 97 21 99 49 73 72 38 18 36 08 00 97 27

72 84 06 63 54

06 79 99 92 96

12 42 19 61 63

86 31 24 55 40

74 54 49 94 96 39 93 77 54 . 34

79 15 43 66 70

70 31 10 82 27

85 13 14 10 48

88 09 12 91 18

71 45 94 81 68

58 43 08 51 59

21 03 55 67 91

9~

82 54 01 83

48 81 70 47 32

81 87 74 94 69

23 54 73 55 21

17 42 84 14 94

13 46 98 00 26

01 56 13 97 20

37 28 II 32 73

57 89 48 51 90

92 02 25 92 70

16 06 33 47 92

34 98 39 03 76

15 59 27 92 49

80 90 36 33 14

90 74 08 73 60

25 13 99 20 34

64 38 57 21 43

67 98 60 29 90

77 66 42 77 51

29 23 88 37 72

95 20 68 06 II

84 2j 25 98 07

80 90 22 64 75

84 55 89 63 94

84 31 67 34 19

87 83 83 31 49

22 48 16 43 40

82 25 82 83 73

36 06 37 71 13

36 22 97 07 79

89 30 60 22 15

29 87 92 15 12

87 87 76 17 18

70 44 39 55 34

08 48 17 56 22

71 90 84 82 24

98 91 34 62 75

49 38 67 88 56

00

53 65 83 47

89 10 52 86 45

89 60 89 38 22

99 29 90 14 81

29 40 62 63 30

08 07 97 89 82

02 58 04 39 38

72 97 33 81 34

32 84 81 90 52

68 09 91 25 57

16 04 27 62 48

29 33 56 58 30

82 56 46 68 34

19 72 35 87 17

91 33 56 88 87

28 47 66 40 63

00

55 25 52 88

57 62 32 02 23

30 57 38 29 62

92 08 64 82 51

12 21 70 69 07

38 77 26 34 69

95 31 27 50 59

21 05 67 21 02

15 64 77 74 89

70 74 40 00 49

78 04 04 91 14

50 93 34 27 98

88 42 63 52 53

01 20 98 98 41

07 19 99 72 92

90 09 89 03 36

72 71 31 45 07

77 46 16 65 76

99 37 12 30 85

53 32 90 89 37

04 69 50 71 84

34 73 69 89 28 96 45 91 37 47

32 44 94 13 78

25 61 44 24 27

21 88 08 40 84

15 23 67 09 05

08 13 79 00 99

82 01 41 65 85

34 59 61 46 75

57 47 41 38 67

57 64 15 61 80

35 04 60 12 05

22 99 II 90 57

03 59 88 62 05

33 96 83 41 71

48 20 24 II 70

84 30 82 59 21

37 87 24 85 31

37 31 07 18 99

29 33 78 42 99

38 69 61 61 06

37 45 89 29 96

89 58 42 88 53

76 48 58 76 99

25 00 88 04 25

09 83 22 21 13

69 48 16 80 63

42 04 82 38 91

39 52 85 48 19

30 43 77 84 05

02 96 30 88 68

34 38 16 24 22

99 13 69 55 58

46 83 32 46 04

68 80 46 48 63

45 72 46 60 21

15 34 30 06 16

19 20 84 90 23

74 84 20 08 38

15 56 68 83 25

50 19 72 83 43

17 49 98 98 32

44 59 94 40 98

80 14 62 90 94

13 85 63 88 65

86 42 59 25 35

38 99 44 26 35

00

40 45 82 71 16 34 89 06 85 74 55 16 91 07

13 33 15 80 12

44 79 87 85 43

54 65 49 03 56

81 43 98 98 04

87 75 71 68 21

21 12 31 89 34

31 91 80 39 92

40 20 59 7I 89

46 36 57 87 81

17 25 32 32 52

62 57 43 '4 15

63 92 07 99 12

99 33 85 42 84

71 65 06 10 II

14 95 6-1 25 12

12 48 75 37 66

64 75 27 30 87

51 68 00 06 29 17 08 27 47 21

50 65 06 75 06

60 25 II 43 86

78 90 30 97 08

22 16 68 54 35

69 29 70 20 39

51 34 97 69 52

98 14 8.7 93 28

37 43 21 50 09

48 23 43 40 66

09 36 95 97 10 96 97 55 45 05 08 50 97 10 69

36 57 98 79 02

20 74 35 89 25

82 07 69 58 36

53 95 45 '9 43

32 26 96 86 71

89 44 80 48 76

92 93 46 27

68 08 26 98 67

50 43 39 99 56

88 30 96 24 12

17 41 33 08 69

37 86 60 9-1 07

02 74 73 15 55

23 33 30 81 1>3

43 78 79 29 31

63 8-1 17 82 50

24 33 19 '4 72

69 38 03 35 20

80 76 47 88 33

91 73 28 03 36

00

92 45 20 19 89

578

APPENDIX

Table XI. Random Sampling Numbers (continued) 15 77 18 08 16

62 81 87 58 07

38 15 05 53 79

72 14 09 63 57

92 67

54 13 05 46 95 27 23 17 22 39 44 74 69 95 21 30 75 75 63 97

17 39

96

66 61

80 II

12

03 55 45 13 42

76 24 14 07 19

09 22 72 04 68

80 93 67 07 21

04 67 46 46 59

24 54 69 37 72 02 96 40 12 07

67 72 59 20 04

60 22 06 86

I2

77 36 12 39 60

57 93 90 53 52

46 75 41 41 82

51 94 19 40 72

14 77 48 04 47

39 09 46 16 72

17 23 72 78 14

21 48 51 67 37

39 89 07 75 91 69 I2 97 39 29 83 41 72 69 75

9B

30 20 41 48 15

75 55 46 71

05 51 24 53 80 24 44 48 25 95 28 63 9B 81 38 00 II 57 05 86

10 43 17 79 99 88

79 26 46 32 49 I2 II 81 22 39

21 56 73 74 75

60 03 411 14 15 75 25 I2 lI8 23 52 48 II 15 23 17 16 69 13 84

35 03 83 18 48

47 55 35 30 72

87 51 83 90 21

44 09 23 44 52

36 74 17 37 51

62 47 29 64 81

08 04 97 53 26

74 88 35 09 36

79 45 74 93 68

30 80 9B 60 05 75 28 29 48 09

70 II 66 79 25 90 92 74 77 87 42 13 49 48 38 80 19 68 30 45 37 69 26 22 80

88 40 74 94 23

01 18 19 49 34

94 65 06 49 10

52 87 42 71 45

31 37 60 21 70

38 08 20 93 83

57 68 79 93 51

62 90 71 07

71 39 81 30 37

62 52 77 34 44

12 84 18 52 62

56 74 51 65 96

61 90 71 83 74

01 68 27 40 42

54 18 27 13 64

49 03 21 56 72

16 51 88 41 39

57 79 87 73 19

15 78 28 33 70

56 75 23 59 01

28 98 80 98 33

II

47 53 24 04

39 85 89 24 84

69 07 07 87 63

55 26 87 06 27

38 02 93 75 65

53 61 30 99 84

06 28 17 52 39

97 01 84 09 45

20 22 17 88 55

42 16 74 05 31

09 14 16 86 95

14 90 I2 15 53 31 25 43 88 93

43 67 39 50 90

48 22 01 94 37

97 18 53 60 09

28 87 40 49 16

25 81 49 02 72 08 II 75 45 03 41 56 12 75 04

71 69 22 04 51 74 52 16 03 82 13 56 85 31 37 78 33 77 28 92 39 69 95 00 48

56 20 09 21 26

46 19 17 90 85

56 15 66 23 71 96 10 62 28 73

10 62 79 01 08

69 37 39 97 66

59 51 50 06 92

99 04 79 45 10

50 89 27 01 66

29 31 62 19 75

33 32 71 95 62

50 19 14 12 61

93 85 53 18 82

09 57 03 52 57

64 31 80 24 59

20 28 04 60 16

19 07 28 22

57 07 28 57 61 28 88 63 74 II 44 69 78 70 80

51 60 41 76 09

34 29 26 90 65

54 85 92 81 32

9B 70 42 17 68

50 79 33 85 26

58 20 53 97 72

79 74 48 41 17

63 23 44 16 04

15

22 73 03 59 01

94 75 03 50 22

16 59 95 24 27

II

87 58 47 51 26

54 77 76 19 29

88 03 35 34 18

26 73 67 54 08 97 78

54 76 78 24 44

24 09 28 73 43

06 99 IO 44 09 39 86 72 58 92

54 18 90 16 55

29 34 46 93 71

98 22 56 94 70

27 73 19 65 92

40 57 50 70 04

51 40 58 95 22

92 67 33 95 53

07 17 84 83 19

13 28 53 20 29

58 63 14 91 67

41 59 56 57 74 36 74 17 40 42 57 95 29 13 56

94 18 73 63 70

16 32 51 42 65 55 25 50 86 II 04 02 00 86 29 02 45 73 45 05

54 68 04 53 04

77 35 02 02 32

37 90 28 27 43

13 85 03 49 62 86 70 30 93

19 38 36 95 41

99 48 52 97 36

19 21 37 50 05

72 49 68 52 09

58 72 15 53 18

35 97 53 52

09 19 98 78 01

26 64 30 21 63

00 74 81 82 16 31 68 69 17 60

26 78 83 57

II

49 79 22 26 71

II

42 92 24 79 65

02 51 29 89 07

83 28 24 55 44

31 79 85 81 86

85 13 44 75 19

65 20 25 24 58

66 82 50 52 92

31 34 75 51 23

97 81 62 32 71

67 39 83 79 32

52 46 95 97 96

15 86 41 05 19

20 13 94 13 02

79 07 26 55

70 89 82 88 44 24

09 72 37 38 97

30 08 43 43 71

81 14 00 37 34 23 75 37 97 93

53 80 93 75 14 94 00 14 50 43 83 85 12 70 89

71 83 96 53 42

94 10 18 14 85 06 72 66 85 41 17 71 74 54 62 99 52 33 24 91

83 07 69 68 05

69 47 20 93 87

76 30 15 74 53

53 25 17 II 9B 82 43 95 15 77

27 16 79 06 49

36 65 65 02 63 97 69 68 50 26 79 78 92 83 97

05 30 31 87 80

34 13 18 91 76

90 67 75 25 24

96 06 55 52 00

63 34 82 57 14

54 98 66 15 92

22 04 34 21 14

36 80 27 40 12

85 01 67 67 77

36 18 65 51 46

49 92 16 84 48

24 10 43 72 50

72 47 83 74 36

10 04 18 32 30

95 54 I2 17 93

14 45 48 70 08

64 90 22 24 37

9B

84 20 77 54 29

I2

38 12 71 05 17

99 54 79 50 73

94 51 87 42 19

25 54 85 66 44

52 96 69 40 43

03 20 92 45 24

27 76 68 69 30

50 65 74 30 24

70 88 02 86 48 80 29 19 98 92

06 80 06 33 84 57 47 35 16 84 65 73 90 50 46 00

18 82 68 40 01

26 42 87 90 39

*Abridged from Table XIX of Statistical Tables and Formulas by A. Hald, John Wiley & Sons, New York, 1952.

Partial Answers to Selected Exercises

lA.l. (a) Yes. (b) No. (c) No. (d) Yes. (e) No. (f) No. lA.2. (a) 0 = Pr{El ( l E z} = Pr{E1IEz} < Pr{E1} < Pr{El U E z} = Pr{E1} + Pr{Ez}. (b) 0 < Pr{El ( l E z} < Pr{E1IEz} = Pr{E1} < Pr{El U E z} < Pr{E1} + Pr{Ez}. lA.3.0 < Pr{El ( l E z} = Pr{E1} < Pr{Ez} = Pr{El U E z} < Pr{EzIE1} = 1. lA.6. (a) 1/3. (b) 3/11. lA.S. 1 - (5/6)6 = 0.665. lA.9. Probability of not being fired when Jones drives Monday, etc., is 0~0~(2 - ( 1), Smith should drive Monday, etc. lA.lO. (a) Yes. (b) No. IB.l. 1/4. . IB.3. (a)

(~O)

= 120. (b) Plo = 720.

lB.S. 10!/5! 3! 2! IB.6. (a)

en

= 2520.

= 792.

(b)

(i) G) (n =

180.

lB.S. 12!/3! 3! 3! 3! = 369,600. lB.IO. (a)

(~) = 20.

(b) Pg

= 120.

le.2. (a) 0.0037. (b) 0.0199. lC.3. (a) 1 - (5/6)4 = 0.51775. (b) 1 - (35/36)24 = 0.49140. lC.4. 694/4096 = 0.169. lC.6. (a) 0 = 0.5. (b) 0 = 0.59. ID.l. (b) 0.35. (c) 37.2. (d) 465.8. lD.2. (a ii) 2. (b ii) 5. (a iii) 3. (b iii) 6. ID.4. Expected value of cash reward is $171.109. 579

580

PARTIAL ANSWERS TO SELECTED EXERCISES

1D.S. (a) 2x/b - x 2/b 2. (b) b/3. (c) b2/6. (d) b2/18. (e)

o.

([HI -

1/v'2)b

=

0.2929b.

m.6. (a i) co. (a ii) af3/(a - 1). (b i) co. (b ii) af32/(a - 1)2(a - 2). 21/ af3. (d) py{y} = af3ae-a y = ae-a(y-Iog!l). (e) (I/a) log 2 + log 13.

(c)

m.7. (a) 104. (b) 104. (c) O. (d) O. 1D.13. E[x]. 1D.1S. Pz{z} = 1,0 < z < 1, Pz{z} = 0 otherwise. lE.1. (a) 0.3085. (b) 0.8413. (c) 0.0441. (d) 0.9545. (e) 116.45. (J) 87.18. (g) 16.45. 1E.3. (i a) 0.0062. (i b) 0.02275. (i c) 0.1336. (ii a) 175.64. (ii b) 133.17. (ii c) 136.61, 163.49. 1E.4. 35/256. lE.S. Both (a) and (b) equal 1 - e- b8 • lE.7. (a) 0.15. (b) 0.5. (c) 0.5. (d) 0.5. (e) 0.75. (f) 0.5. lE.8. (1 + log 2)/2. 1F.1. (a) 2.5. (c) -1/v''2. 1F.3. (a) 20. (b) 5. (c) 20. (d) 11.5. (e) O. (f) 8.5. (g) o. (II) 26. (i) 10. (j) O. (k) 1. (I) O. 1F.4. (a) 0.02275. (b) 0.0096. 1F.8. (a) 0.5. (b) 0.0062. (c) 0.1056. (d) 1.058. (e) 1.029. 1F.9. 0.8167. 1F.10. 0.02275. 2.1. E[02] = 02 + V[O].

2.3. (MSE s2)/(MSE 62) = 1/[1 - (311 - 1)/2112]. 2.5. (a) 0 = I/x. (b) E[l/x] = co. (c) i = x. (d) E[x] = J.. 2.7. (a) - co < x < -1, -0.05 < x < 0.05,1 < x < co. (b) 0.1251. 2.8. (a) -1 < x < -0.8335,0.8335 < x < 1. (b) 0.167. 2.9. (a) 0 < x < 0.1024. (b) 0.1852. 2.10. (a) x > 542.3. (b) 0.8245. (c) 64.4. 2.11. (a) x < 473.4, x > 522.6. (b) 0.732. (c) 79.1. 2.12. (a~ 0.0052. (b) (0.006, 0.034). 2.13.1 :- a. 2.14. (a) x = -2, 2. (b) c. (c) a. (d) Power is a(1 - c)/(l - a). The powers of the three tests have the relation c < a < a(1 - c)/(1 - a). 3A.2. (a) 0.261, 0.235. (b) 1140, 1155. (c) 0.033. (d) (0.163,0.326). 3A.3. (0.0011, 0.0800). (b) (0.00081, 0.0730). 3A.4. (a) 6800. (b) 29,000. 3A.S. 12,600,000. 3A.6.0.D18. 3B.2. (a) 0.3038. (b) 0.2135.

PARTIAL ANSWERS TO SELECTED EXERCISES

581

3B.5. (a) 0.0218. (b) 0.0238. 3B.6. (a) 0.223. (b) 0.192. 3B.7. ItLA;. 3B.S. (a) 1/ A. (b) 3/2A. 3B.9. (a) 2/ A. 3B.ll. (a) 0.0106. (b) 0.012. 3B.12. (0.62, 8.75). 3B.13. (b) 0.1353. 3B.14. F 1_ P (6, 20) = 2.22; 0.10 > P > 0.05. 3B.15. 14/33. 3B.16. (a) c = 14, 11 = 448. (b) AOQL = 0.021.

= O. 4.3. It = A + vJ: V C/M. 4.5. It = A (1.-1/". 4.1.11

5.1. X2(3) = 13.64.

5.2. ;(2(3) 5.4. ;(2(8)

= 6.25. = 41.38.

6.1. (a) [(11 - 1) + ,.]! (II-I)!,.!

= 66.

(,. - I)! (b) (11 _ I)! (,. _ II)!

= 36.

(,.-11-1)! =15 (11 - 1)! (,. - 211)! . 6.2. P = 0.064. 6.3. P = 0.012. 6.4. P = 0.0053. (b) 0.0023. (c) 0.0028. 6.7. (a) P = 0.0009. (c) P = 0.0009.

(c)

7.1. P ~ 0.0044.

7.2. P ~0.0186. 7.3. H = 6.452 ,...., ;(2(2). 7.4. (a) P = 0.212 (exact), 0.211 (approx.). (b) P = 0.150. 7.5. (a) 0.0043. (b) MQ58~. (c) 0.0112 (exact), 0.013 (approx.). 7.12

2/1T.

9.1. (a) (0.0217, 0.0725). (b) (5.401, 5.566).

9.2. (0.233, 4.715). 9.4. (a i) 1(14) = 1.283. (b i) (-4.95, 19.68). (a ii) 1(10.0) = 1.258. (b ii) ( -5.68,20.41). 9.6. I. (a) F(9,8) = 2.775, P ~0.10 (two-sided). (b i) 1(17) = 2.142. (b ii) 1(14.95) = 2.199. 9.7. ;(2(3) = 12.04.

582

PARTIAL ANSWERS TO SELECTED EXERCISES

9.8. 7.2(4) = 6.08. 9.10. (a) 117.4. (b) 128.0.

10.1. (a) F(3, 65) = 2.53, P =:: 0.05. (b i) (0.19, 1.87). (b ii) (-0.1 S, 2.24). (e) (-0.08,1.78). 10.2. (a) F(2, 18) = 4.50. (b i) (:....35.6, -6.2). (b ii) (-39.5, -2.2). 10.3. (a) F(4, 59) = 5.99. (b) 3.357, 1.394. (e) 64.14. (d) 0.3423. (e) 0.0691. 11.1. (a) Y = -9.402 + 2.8845x. (b) F(3, 7) = 0.86. (e) 8 2 = 0.005235. (d) 1.350. (e) (1.28,1.42). (f) (2.39,3.30). (g i) (3.703,3.827). (g ii) (3.701, 3.823). 11.3. Estimated line is Y = 4.1371 + 1.01835(x - 4.1329). (a) 1(5) = 7.15. (b) 1(5) = 5.90. 11.5.

Source of variance

S.S.

D.F.

M.S.

Between band b Deviations of group means about their regression line Between individual slopes About individual lines About overall line Due to overall line

322.207 18.256

1 2

322.207 9.132

152.058

3

50.686

938.161 1430.691 4346.291

48 54 1

19.545

5776.982

55

Total

12.1. (a) r = 0.8574, V[x 2 ] = 1523.4, V[x2lxl] = 437.1. (b) 1(12) = 5.771. (e) X 2 = 0.0805 + 0.7492xl' (d) Xl = 0.2557 + 0.9813x 2• (e) 1.13. (J) 1.37. (g) II = 1.033. 12.3. 1/17[1 + (x/y)2]. Hint: Define II = X, V = x/y, then obtain joint density function of (II, v), then integrate over II to get density function of v. Remember that (2.16) uses the absolute value of the determinant. 13.1. (a) 1(20) = 2.373. (b) Y = 56.727 + 0.07852(XI - 87.545) - 0.47438(x2 - 30.318). (e) F(2, 19) = 2.98. (d) F(1, 19) = 0.48. (e) Y = 56.727 - 0.006784(x l - 87.545) - 0.47824(x2 - 30.318) 8.49662(x3 - 5.636). (f) (0.421,16.572). (g) (46.0,80.5). (II) F(2, 18) = 2.73. (i) Xl = 87.545 2.6031(x2 - 30.318) + 22.367(x3 - 5.636). (j) (2.41, 42.33).

13.3. (a) {J12.3 P12P23'

= {J 12 1 - 1 P13P23/P12 _ 2'

(b) {J

12.3

. = {J12 I'f eIther P23 = 0

P23

13.4. (b)P"'l!"".",. = -VI -P"'l"'/V2.

(e)p"'l!""'''''

= -1/ v2.

or

P13

+

+ =

583

PARTIAL ANSWERS TO SELECTED EXERCISES

14.1. (b) F(4,9)

= 0.21.

(b) F(2,9)

= 3.66.

(d i) (-0.263, 0.077). (d ii)

(-0.190,0.003). 14.2. (a i) F(8,30) = 1.38. (a ii) F(4,30) = 28.5. (a iii) F(2,8) (b i) 157.2. (b ii) 19.9. (b iii) 480.6. (c)( -39.0, -14.2).

= 26.7.

15.1.

Source of variance

D.F.

S.S.

M.S.

Concentration: linear quadratic Batches Strains Batch x concentration (linear) (quadratic) Strain x concentration (linear) (quadratic) Batch x strain Remainder

1 1 3 3 3 3 3 3 9 18

15,051.125 104.167 1,712.500 8,701.667 168.625 56.250 140.125 369.583 1,001.833 882.792

15,051.125 104.167 570.833 2,900.556 56.208 18.750 46.708 123.194 111.315 49.044

Total

47

28,188.667

Confidence limits for C - D, using batch x strain interaction as error term, are (-4.41, 15.08). A "pooler" who pooled all interactions to form an error term would obtain (-1.43,12.10). 16.1. (a) Model might be

Xijk

=~

Source of variance

+ Ti + {/(T)h(i) + Ak + (TA)tk + zUk' S.S.

D.F.

M.S.

Types of lamp Lamps within type Laboratories Types x laboratories Remainder

290,371.2 20,441.3 12,490.1 345.6 1,652.7

1 10 4 4 40

290,371.2 2,044.1 3,122.5 84.6 41.3

Total

325,300.9

59

(b) (113,165). (c) (-2.39,8.22). (d) (89.3, 142.4). (e) 0f(T)

E[M.S.]

a2 +'5of(T) + 30~ 0 2 + 50l(T) 02 02

+

02

= 400.6.

of squares for contrasts between laboratories: Contrast , (i) (ii) (iii)

(iv)

12a~

+ 6a~A

Main effect of contrast

Interaction of contrast with type

56.33 6.75 48.35 12,378.67

85.33 4.08 86.68 169.46

(II) Sums

584

PARTIAL ANSWERS TO SELECTED EXERCISES

17.3. (a) Xijl,; = ~

+ Ii + {s(l)}j(i) +

TI,;

+ (/T)il,; + Z;jl,;

Source of variance

S.S.

D.F.

M.S.

Lots Samples within lots Temper Lots x temper Remainder

54,395.9 11,795.0 13,568.2 2,533.1 7,505.7

4 10 1 4 10

13,599.0 11,795.5 13,568.2 633.3 750.6

Total

89,797.9

29

(b) F(4, 10) = 0.84. (c) (17.0,68.0). (d) 2070.

a2

= 751,

afT

E[M.S.] a2 a2 a2 a2 a2

+ 2a~(I) + 6a~ + 2a~(I) + 3a~ + 15a2 + 3a2TIT T

= 0, a~(!) = 214,

af

=

Index

Abrahmson, Abraham E., 213, 220 Acceptable quality-level, 175 Acceptance, region of, 98 Acton, Forman S., 335, 396 Additive model, 469 Analysis of covariance, 376 Analysis of variance: crossed nested, 531 four-way, 524 hierarchical, 482 nested, 482 one-way, model I, 310 one-way, model II, 318 partially hierarchical, 531 three-way, 504 two-way, model I, 467 two-way, model II, 478 two-way, mixed model, 498 Andersen, S. L., 270, 293, 308 Anderson, R L., 444, 466, 529 Angular transformation, 144 Arcsin transformation, 144 Average outgoing quality, 178 limit, 179 Bailey, Nancy D., 263 Bancroft, T. A., 444, 466, 529 Barkworth, H., 529 Barnard, G. A., 216, 220 Bartlett, M. S., 292, 308 Bartlett's test, 292 Battan, L. J., 158

Bayes, Thomas, 86 Bayes' theorem, 18 Behrens-Fisher problem, 299 BeIlinson, H. R., 239 Bennett, Carl A., 489, 503, 538, 546 Berkson, J., 393, 396 Bias, 89 in multiple regression, 452 Binomial distribution, 30, 136 and angular transformation, 144 and hypergeometric distribution, 160 and Poisson distribution, 166 expectation of, 53 normal approximation to, 136 variance of, 59 Biological assay, 352 Birnbaum, A., 511, 529 Bivariate distribution: continuous, 70 discrete, 67 normal, 401 Blade, O. C., 306 Blocks, randomized, 548 Bloem, Delmar L., 503 Bocher, Maxime, 281 Box, G. E. P., 257, 270, 293, 308 Braham, R. R., 158 Brockmeyer, E., 191,205 Bronetto, Jean, 464 Bross, Irwin, 322, 332 Byers, H. R, 158

585

586

INDEX

Cavendish, H., 223, 240 Confidence limits, for variance ratio, 286 Chapman, Douglas G., 163, 188 mUltiple, 316 Chernoff, Herman, 87, 135 Consistency, 90 Chevalier de Mere, 35, 137 x 2 distribution, 82, 288 Consumer's risk, 175 Contingency tables, 211 test, 207 Clarke, R. D., 168, 189 Continuity, correction for, 140 c matrix, 439, 447 Contrast, 316 Cochran, William G., 215, 220, 262, Control charts: 270, 281, 324, 325, 330, 333, 390, for means, 232 396 for Poisson-distributed observations, Cochran Q test, 262 235 Cochran's theorem, 276 for ranges, 234 Combinations, 22 Cornfield, Jerome, 489, 503 Comparison of several observed: Correlated two X two table, 154 means, 310 Correlation coefficient, 78, 408, 413 medians, 256 estimator of, 410 regression lines, 376 mUltiple, 447 variances, 290 partial, 429 Comparison of two observed: Covariance, 77 correlation coefficients, 414 analysis of, 376 frequencies, ISO, 216 Cramer, H., 56, 86,241, 269, 280, 281 means, 297, 299 Critical region, 98 medians, 242, 246 Cugel, D. W., 342 Poisson-distributed observations, 181, Cumulative distribution function: 183 continuous, 26 regression lines, 349 discrete, 40 variances, 286 Curtailed binomial sampling, 32 Complementary event, 3 Completely randomized design, 547 Daniel, Cuthbert, 450, 511, 528, 546 Components of variance, 319 David, F. N., 336, 396 Compound event, 3 . David, H. A., 293, 308 Compound experiment, 12 Dean, G., 269 Concomitant variable, 377 Degrees of freedom, 272 Conditional density function, 72 Demoivre, 136 Conditional probability, 9 Density function, 37 Conditional probability function, 69 De Shong, James A., Jr., 219 Confidence limits, 121 Distribution: for a and (3 jointly, 362 bivariate, 67, 70 for binomial parameter, 129, 148 conditional, 69, 72 for components of variance, 322 continuous, 36 for differences, in means, 314, 474, cumulative, 28, 40 512, 540, 553 discrete, 24 in medians, 245 marginal, 68, 71 for mean, 126, 128, 321 Distribution-free methods, 241 for Poisson parameter, 173 Dodge, Harold F., 180, 189 for predicted x, 361 Doolittle method, 455 for predicted '1, 342 Dorsey, N. E., 307, 331 for variance, 282

INDEX

Double sampling, 180 Dunnett, Charles W., 218, 332 Efficiency: of estimator, 91 of randomized block design, 548 Eisenhart, C., 226, 231, 240 Elvehyem, C. A., 527 Emmens, C. W., 354, 396 Epstein, Benjamin, 218, 220 Erlang, A. K., 191 Error: mean square, 90 type I, 98 type II, 99 Errors of measurement, 414 Estimator, 89 maximum likelihood, 91 Events, 3 exhaustive, 6, 12 independent, 11, 15 mutually exclusive, 3 Expectation, 51 of a function, 55 of a multivariate function, 75 of binomial distribution, 53 of x 2 distribution, 82 of exponential distribution, 54 of geometric distribution, 63 of hypergeometric distribution, 185 of logarithmic normal distribution, 62 of normal distribution, 54 of Pareto distribution, 62 of Pascal distribution, 63 of rectangular distribution, 53 Exponential distribution, 43 F distribution, 285, 289 Feller, William, 140, 186, 188 Fermat, 2 Finite population, 248 Finite population model, 489, 499 Finney, D. J., 165, 188, 353, 396, 510, 528 Fisher, R. A., 2, 86, 135, 163, 188,280, 281, 303, 308, 529 Fisher exact test, 163

587

Fit: in multiple regrcssiQn, 441 in simple linear regression, 366 Franklin, Norman L., 489, 503, 538, 546 Fraser, D. A. S., 242, 269 Friedman, M., 270 Friedman test, 260 Geometric distribution, 36, 63 Goodman, L. A., 215, 220 Gossett, W. S., see Student Graybill, Franklin A., 299, 308 Hald, A., 188, 271, 281 Halmstrom, H. L., 205 Hamilton, E. L., 266 Hart, B. I., 239 Hartley, H. 0., 293, 308, 332 Hercus, E. 0., 270 Heteroscedasticity, 146 Heyl, Paul R., 314, 332 Hildebrand, Roger H., 219 Hilleboe, Herman E., 463 Histogram, 37 Homoscedasticity, 146 Honigschmid, 0., 298, 308 H test, 256 Hyde, Edward P., 544 Hypergeometric distribution, 158 Hypothesis: alternative, 97 composite, 97, 111 null, 97 simple, 97 Independence, 10, 12 complete, 15 pairwise, 15 Independent variable, 422 Indices, 431 Interaction: in three-way analysis, 505 in two-way analysis, 469 Intersection, 3 Invariance, 94 Inventory control, 203 Irwin, J. 0., 280, 281, 529

588

INDEX

Jensen, Arne, 205 Joint confidence region, 362 Keeping, E. S., 392, 396 Kempthorne, 0., 444, 466 Kendall, D. G., 195, 205 Kendall, Maurice G., 135, 242, 262, 269 Kent, R. H., 239 Krivobok, V. N., 554 Kruskal, W. H., 215, 220, 253, 263, 269 Laby, T. H., 270 Latscha, R., 165, 188 Least squares, 95, 336 Lehman, E. L., 217, 220 Level of significance, 99 Lieberman, Gerald J., 188 Life testing, 217 Likelihood function, 91 Likelihood ratio, 112 Lindley, D. V., 393, 396 Linear combination, 80 Logarithmic normal distribution, 62 Lot tolerance per cent defective, 175 Madansky, Albert, 392, 396 Mainland, Donald, 165, 188 Mann, H. B., 253, 269 Maximum likelihood, 91 Mayne, C. R., 554 Mead, Jere, 394 Mean square error, 90 Mean square successive difference, 221 Median, 54 Median test, 246, 304 Meyer, Peter, 219 Michelson, A. A., 293, 308 Mixed models, 467, 498, 511 Mode, 52 Model, additive, 409 Model, finite population, 489, 499 Model I and model II, 309 Molina, E. C., 175, 189 Mood, Alexander M., 299, 308 Moore, G. H., 240 Morse, Philip M., 191, 205 Moses, Lincoln E., 87, 135

Mosteller, Frederick, 35, 86, 231 Muhlenbruch, C. W., 554 Multinomial distribution, 206 Multiple comparisons, 316 Multiple correlation coefficient, 447 Mutually exclusive events, 3 Nair, K. R., 510, 528 National Bureau of Standards, 188 Negative exponential distribution, 43 Neyman, J., 2, 86, 135, 290, 336, 396 Neyman-Pearson lemma, 103 Nonparametric tests, 241 Normal distribution, 44 addition theorem for, 81, 408 approximation to binomial, 136 bivariate, 401 properties of standardized, 63 standardized, 47 trivariate, 429 Normal equations, 423, 434 Normality, assumption of, 241 Olmstead, P. S., 240 Operating characteristic, 175 Ordnance Corps, 188 Orthogonal contrasts, 517 Owen, Donald B., 188 Pairwise independence, 15 , Pareto distribution, 62 Partial correlation coefficient, 429 Partially hierarchical situations, 530 Pascal,2 Pascal distribution, 36, 63 Paull, A. E., 509, 528 Pearson, E. S., 2, 86, 135, 216, 220, 332 Pearson, Karl, 2, 43, 466 Permutations, 20 Pivotal function, 127 Placket!, R. L., 336, 396 Plais, Cassius M., 219 Plot, 548 Poisson distribution, 166,.I 69 addition theorem for, 181 Pooling, 509 Power, 99 Prediction interval, 342

INDEX

Probability density function, 37 Probability function, 25 Producer's risk, 175 Putter, J., 269 P value, 120 Queues, 190 finite, 198 infinite, 191 multichannel, 200 single-channel, 191 Randomization, 548 Randomized blocks, 540 Randomized test, 11 0 Randomness, tests for, 221 Random numbers, 548 Random variable, 24 Range, 233 Rectangular distribution, 42 Rectifying inspection, 177 Region of acceptance, 98 Region of rejection, 98 Regression, 334, 409 comparison of several lines, 376 comparison of two lines, 349 exponential, 391 fallacy, 409 in bivariate normal distribution 406 . ' In reverse, 346, 361 multiple, 419 on several independent variables, 433 on two independent variables, 422 polynomial, 447 simple linear, 334 through origin, 358 Rejection, region of, 98 Roberts, H. V., 86 Robustness, 305 Rocker, W. Arthur, 393 Romig, Harry G., 180, 188, 189 Rosa, E. B., 307, 331 Runs: above and below median, 231 of three types of elements, 231 of two types of elements, 224, 226 of various types, 232 Saaty, Thomas L., 191, 205 Sample space, 2

589

Sampling, 329 Sampling inspection, 174 and hypothesis testing, 177 Satterthwaite, F. E., 303, 308 Savage, I. R., 242, 269 Savage, L. J., 87, 135 Scheffe, Henry, 316, 322, 323, 333, 393, 396 Schmidt, Wolfgang, 464 Sequential analysis, 2, 180 Service time distribution: arbitrary, 195 exponential, 191 Shewhart, W. A., 232, 246 Significance, level of, 99 Sign test, 242, 304 Simon, Herbert A., 454, 466 Smith, H. Fairfield, 390, 396 Sobel, Milton, 218, 220 Space: parameter, 112 sample, 2 Split-plot design, 550 Standard deviation, 57 Standardized normal distribution, 47, 63 Standardized variable, 61 Statistical control, 233 Statistical inference, 87 Stein, Charles, 135 Stevens, W. L., 226, 231, 240 Strong, F. M., 527 Stuart, Alan, 135 Student, 2, 86, 280, 281 Subplot, 550 Sverdrup, Erling, 217, 220 Swed, F. S., 226, 231, 240 Takacs, L., 191, 205 t distribution, 289 Tests: likelihood ratio, 112 of hypotheses, 97 uniformly most powerful, 105 Ties: in H test, 256 in sign test, 244 in Wilcoxon one-sample test, 259 in Wilcoxon two-sample test, 253

590

INDEX

Tocher, K. D., 166, 185 Todhunter, I., 2, 86 Transformations: angular, 144 of bivariate distributions, 3~7 of density functions, 45 of probability functions, 45 variance stabilizing, 144 z,414

Trivariate normal distribution, 429 t test: one-sample, 295, 304 two-sample, 297, 304 Tukey, J. W., 316, 332, 454, 466, 489, 503 Two-sample test with unequal variances, 299, 304 Two X two table, 215 Type I error, 98 Type II error, 99 Unbiasedness, 89 Uniform distribution, 42 Uniformly most powerful test, 105 Union,S Unit normal deviate, 48 Uspensky, J. V., 140, 188 Variable: independent, 422 random, 24 Variance, 57 of binomial distribution, 58 of x!! distribution, 84 of exponential distribution, 59 of geometric distribution, 63 of hypergeometric distribution, 185 of linear combination, 80 of logarithmic distribution, 62 of mean, 81

Variance, of mean of sample from finite population, 248 of multiple regression coefficient, 424, 436, 450, 452 of normal distribution, 60 of Pareto distribution, 62 of Poisson distribution, 167 of rectangular distribution, 59 of regression coefficient, 337, 340 of variance, 300 Variance ratio distribution, 285 Variance stabilizing transformations, 144 Vaurio, V. W., 546 Von Neuman, J., 239 Wagner, R., 527 Wald, A., 2, 86, 87, 135, 180, 189, 226, 23 I, 240 Walker, Stanton, 503 Wallace, D. L., 257, 269 Wallis, W. Allen, 86, 187, 189,231,240 Weighted mean, 95 Welch, B. L., 299, 308 Whitney, D. R., 253, 269 Wilcoxon, F., 269 Wilcoxon: one-sample test, 258, 304 two-sample test, 251, 304 Wild life estimation, 162 Wilens, Sigmund L., 219 Wilks, S. S., 113, 135 Williams, J. S., 322, 333 Wolfowitz, J., 226, 231, 240 Yates, F., 529 Yates' correction, 216 Yerushalmy, 1., 463 Z

transformation, 414