Measurement Error Models

Measurement Error Models

Measurement Error Models WAYNE A. FULLER Iowa State University Ames, Iowa

JOHN WILEY & SONS

New York

Chichester

Brisbane

Toronto

Singapore

A NOTE TO THE IEADER

This book has been electronically reproduced from digital itiforniation stored at Jolui Wiley I% Sons, hic. We are pleased that the use of this new technology will enable 11s to keep works of enduring scholarly value in print as long as there is a reasonable demand for them. The content of this book is identical to previous printings.

Copyright @ 1987 by John Wiley & Sons, Inc. All rights reserved. Published simultaneously in Canada.

Reproduction or translation of any part of this work beyond that permitted by Section 107 or 108 of the 1976 United States Copyright Act without the permission of the copyright owner is unlawful. Requests for permission or further information should be addressed to the Permissions Department, John Wiley & Sons, Inc. Library of Congress Cataloging in Publication Data: Fuller, Wayne A. Measurement error models. (Wiley series in probability and mathematical statistics. Applied probability and statistics, ISSN 0271-6356) Bibliography: p. Includes index. 1. Error analysis (Mathematics) 2. Regression analysis. I. Title. 11. Series. QA275.FS5 1987 ISBN 0-471-86187- 1

Printed and bound in the United States of America by Braun-Brumfield, lnc. 109 8 7 6 5 4

To Doug and Bret

Preface

The study of regression models wherein the independent variables are measured with error predates the twentieth century. There has been a continuing interest in the problem among statisticians and there is considerable literature on the subject. Also, for over 80 years, studies have documented the presence of sizable measurement error in data collected from human respondents. Despite these two lines of research, only a fraction of the statistical studies appearing in the literature use procedures designed for explanatory variables measured with error. This book is an outgrowth of research on the measurement error, also called response error, in data collected from human respondents. The book was written with the objective of increasing the use of statistical techniques explicitly recognizing the presence of measurement error. To this end, a number of real examples have been included in the text. An attempt has been made to choose examples from a variety of areas of application, but the reader will understand if many of the examples have an agricultural aspect. The book may be used as a text for a graduate course concentrating on statistical analyses in the presence of measurement error. It is hoped that it will also find use as an auxiliary text in courses on statistical methodology that heretofore have ignored, or given cursory treatment to, the problems associated with measurement error. Chapter 1 was developed to provide an introduction to techniques for a range of simple models. While the models of Chapter 1 are special cases of models discussed in later chapters, it is felt that the concepts are better communicated with the small models. There is some flexibility in the order in which the material can be covered. One can move from a section in Chapter 1 to the corresponding section in Chapter 2 or Chapter 4. To facilitate flexible use, Sections 1.2, 1.3, and 1.4 are largely self-supporting. As a result, there is some duplication in the treatment of topics such as prediction. Some repetition seems advantageous because the vii

viii

PREFACE

models of this book differ from those typically encountered by students in courses on regression estimation. The proofs of most of the theorems require a background of statistical theory. One will be comfortable with the proofs only if one has an understanding of large sample theory. Also, the treatment assumes a background in ordinary linear regression methods. In attempting to make the book useful to those interested in the methods, as well as to those interested in an introduction to the theory, derivations are concentrated in the proofs of theorems. It is hoped that the text material, the statements of the theorems, and the examples will serve the person interested in applications. Computer programs are required for any extensive application of the methods of this book. Perhaps the most general program for normal distribution linear models is LISREL@ VI by Joreskog and Sorbom. LISREL VI is available in SPSSXTM and can be used for a wide range of models of the factor type. A program with similar capabilities, which can also perform some least squares fitting of the type discussed in Section 4.2, is EQS developed by Bentler. EQS is available from BMDP@Statistical Software, Inc. Dan Schnell has placed the procedures of Chapter 2 and Section 3.1 in a program for the IBM@ Personal Computer AT. This program, called EV CARP, is available from the Statistical Laboratory, Iowa State University. The packages S A P and BMDP contain algorithms for simple factor analysis. A program, ISU Factor, written with Proc MATRIX of SAS by Sastry Pantula, Department of Statistics, North Carolina State University, can be used to estimate the factor model, to estimate multivariate models with known error variances, and to estimate the covariance matrix of the factor estimates. A program for nonlinear models, written with Proc MATRIX of SAS by Dan Schnell, is available from Iowa State University. I have been fortunate to work with a number of graduate students on topics related to those of this text. Each has contributed to my understanding of the field, but none is to be held responsible for remaining shortcomings. I express my sincere thanks to each of them. In chronological order they are James S. DeGracie, Angel Martinez-Garza, George E. Battese, A. Ronald Gallant, Gordon D. Booth, Kirk M. Wolter, Michael A. Hidiroglou, Randy Lee Carter, P. Fred Dahm, Fu-hua Yu, Ronald Mowers, Yasuo Amemiya, Sastry Pantula, Tin-Chiu Chua, Hsien-Ming Hung, Daniel Schnell, Stephen Miller, Nancy Hasabelnaby, Edina Miazaki, Neerchal Nagaraj, and John Eltinge. I owe a particular debt to Yasuo Amemiya for proofs of many theorems and for reading and repair of much of the manuLISREL is a registered trademark of Scientific Software, Inc. SPSS' is a trademark of SPSS, Inc. BMDP is a registered trademark of BMDP Statistical Software, Inc. SAS is a registered trademark of SAS Institute, Inc. IBM AT is a registered trademark of International Business Machines, Inc.

PREFACE

ix

script. I thank Sharon Loubert, Clifford Spiegelman, and Leonard Stefanski for useful comments. I also express my appreciation to the United Kingdom Science and Engineering Research Council and the U.S. Army European Research Ofice for supporting the “Workshop on Functional and Structural Relationships and Factor Analysis” held at Dundee, Scotland, August 24 through September 9, 1983. Material presented at that stimulating conference had an influence on several sections of this book. I am grateful to Jane Stowe, Jo Ann Hershey, and Christine Olson for repeated typings of the manuscript. A part of the research for this book was supported by joint statistical agreements with the United States Bureau of the Census and by cooperative research agreements with the Statistical Reporting Service of the United States Department of Agriculture.

WAYNEA. FULLER Ames, Iowa

February 1987

Contents

List of Examples

xv

List of Principal Results

xix

List of Figures 1. A Single Explanatory Variable

xxiii 1

1.1. Introduction, 1 1.1.1. Ordinary Least Squares and Measurement Error, 1 1.1.2. Estimation with Known Reliability Ratio, 5 1.1.3. Identification, 9 1.2. Measurement Variance Known, 13 1.2.1. Introduction and Estimators, 13 1.2.2. Sampling Properties of the Estimators, 15 1.2.3. Estimation of True x Values, 20 1.2.4. Model Checks, 25 1.3. Ratio of Measurement Variances Known, 30 1.3.1. Introduction, 30 1.3.2. Method of Moments Estimators, 30 1.3.3. Least Squares Estimation, 36 1.3.4. Tests of Hypotheses for the Slope, 44 1.4. Instrumental Variable Estimation, 50 1.5. Factor Analysis, 59 1.6. Other Methods and Models, 72 1.6.1. Distributional Knowledge, 72

xi

xii

CONTENTS

1.6.2. The Method of Grouping, 73 1.6.3. Measurement Error and Prediction, 74 1.6.4. Fixed Observed X,79 Appendix 1.A. Large Sample Approximations, 85 Appendix l.B. Moments of the Normal Distribution, 88 Appendix l.C. Central Limit Theorems for Sample Moments, 89 Appendix l.D. Notes on Notation, 95

2. Vector Explanatory Variables

100

2.1. Bounds for Coefficients, 100 2.2. The Model with an Error in the Equation, 103 2.2.1. Estimation of Slope Parameters, 103 2.2.2. Estimation of True Values, 113 2.2.3. Higher-Order Approximations for Residuals and True Values, 118 2.3. The Model with No Error in the Equation, 124 2.3.1. The Functional Model, 124 2.3.2. The Structural Model, 139 2.3.3. Higher-Order Approximations for Residuals and True Values, 140 2.4. Instrumental Variable Estimation, 148 2.5. Modifications to Improve Moment Properties, 163 2.5.1. An Error in the Equation, 164 2.5.2. No Error in the Equation, 173 2.5.3. Calibration, 177 Appendix 2.A. Language Evaluation Data, 181

3. Extensions of the Single Relation Model 3.1. Nonnormal Errors and Unequal Error Variances, 185 3.1.1. Introduction and Estimators, 186 3.1.2. Models with an Error in the Equation, 193 3.1.3. Reliability Ratios Known, 199 3.1.4. Error Variance Functionally Related to Observations, 202 3.1.5. The Quadratic Model, 212 3.1.6. Maximum Likelihood Estimation for Known Error Covariance Matrices. 217

185

CONTENTS

xiii

3.2. Nonlinear Models with N o Error in the Equation, 225 3.2.1. Introduction, 225 3.2.2. Models Linear in x, 226 3.2.3. Models Nonlinear in x, 229 3.2.4. Modifications of the Maximum Likelihood Estimator, 247 3.3. The Nonlinear Model with an Error in the Equation, 261 3.3.1. The Structural Model, 261 3.3.2. General Explanatory Variables, 263 3.4. Measurement Error Correlated with True Value, 271 3.4.1. Introduction and Estimators, 271 3.4.2. Measurement Error Models for Multinomial Random Variables, 272 Appendix 3.A. Data for Examples, 281

4. Multivariate Models

292

4.1. The Classical Multivariate Model, 292 4.1.1. Maximum Likelihood Estimation, 292 4.1.2. Properties of Estimators, 303 4.2. Least Squares Estimation of the Parameters of a Covariance Matrix, 321 4.2.1. Least Squares Estimation, 321 4.2.2. Relationships between Least Squares and Maximum Likelihood, 333 4.2.3. Least Squares Estimation for the Multivariate Functional Model, 338 4.3. Factor Analysis, 350 4.3.1. Introduction and Model, 350 4.3.2. Maximum Likelihood Estimation, 353 4.3.3. Limiting Distribution of Factor Estimators, 360 Appendix 4.A. Matrix-Vector Operations, 382 Appendix 4.B. Properties of Least Squares and Maximum Likelihood Estimators, 396 Appendix 4.C. Maximum Likelihood Estimation for Singular Measurement Covariance, 404

Bibliography

409

Author Index

433

Subject Index

435

List of Examples

Number

Topic

1.2.1

Corn-nitrogen. Error variance of explanatory variable known. Estimates, 18

1.2.2

Corn-nitrogen. Estimated true values, 23

1.2.3 1.3.1

Corn-nitrogen. Residual plot, 26 Pheasants. Ratio of error variances known, 34 Rat spleens. Both error variances known, 40

1.3.2 1.3.3 1.4.1 1.5.1

1.5.2

1.6.1 1.6.2 2.2.1 2.2.2. 2.2.3 2.3.1 2.3.2 2.3.3

Rat spleens. Tests and confidence intervals, 48 Earthquake magnitudes. Instrumental variable, 56 Corn hectares. Factor model, 63 Corn hectares. Standardized factors, 69 Corn-nitrogen. Prediction for random model, 75 Earthquakes. Prediction in another population, 77 Coop managers. Error variances estimated, 110 Coop managers. Estimated true values, 114 Corn-nitrogen. Variances of estimated true values and residuals, 121 Apple trees. Estimated error covariance, 130 Corn-moisture experiment. Estimated error covariance, 134 Coop managers. Test for equation variance, 138 xv

xvi 2.3.4 2.4.1 2.4.2 2.5.1

LIST OF EXAMPLES

Rat spleens. Variances of estimated true values and residuals, 142 Language evaluation. Instrumental variables, 154 Firm value. Instrumental variables, 158

3.1.1

Corn-nitrogen. Calibration, 179 Corn-nitrogen. Duplicate determinations used to estimate error variance, 197

3.1.2 3.1.3

Farm size. Reliability ratios known, 201 Textiles. Different slopes in different groups, 204

3.1.4

Pig farrowings. Unequal error variances, 207 Tonga earthquakes. Quadratic model, 214

3.1.5 3.1.6 3.1.7 3.2.1 3.2.2 3.2.3

Quadratic. Both error variances known, 21 5 Supernova. Unequal error variances, 221 Created data. Linear in true values, 226 Berea sandstone. Nonlinear, 230 Berea sandstone. Nonlinear multivariate, 234

3.2.4

Hip prosthesis. Implicit nonlinear, 244

3.2.5 3.2.6

Quadratic, maximum likelihood. Large errors, 247 Pheasants. Alternative estimators of variance of estimated slope, 255

3.2.7

Pheasants. Alternative form for estimated variance of slope, 257 Quadratic. Large errors. Modified estimators, 257 Quadratic. Error in the equation. Weighted, 266

3.2.8. 3.3.1 3.3.2 3.4.1 4.1.1 4.1.2 4.2.1

Moisture response model. Nonlinear, 268 Unemployment. Binomial, 275 Mixing fractions. Known error covariance matrix, 308 Cattle genetics. Error covariance matrix estimated, 3 13 Two earthquake samples. Least squares estimation, 325

4.2.2

Corn hectares. Estimation of linear model, 330

4.2.3 4.2.4

Corn hectares. Distribution-free variance estimation, 332 Earthquakes. Least squares iterated to maximum likelihood, 337

LIST OF EXAMPLES

4.2.5 4.3.1 4.3.2 4.3.3

xvii

Corn hectares. Least squares estimation fixed and random models, 344 Bekk smoothness. One factor, 364 Language evaluation. Two factors, 369 Language evaluation. Not identified, 374

List of Principal Results Theorem

1.2.1

1.3.1 1.4.1 1.6.1 1 .A. 1 1 .c.1

1.c.2 1.C.3 2.2.1

Topic

Approximate distribution of estimators for simple model with error variance of explanatory variable known, 15 Approximate distribution of estimators for simple model with ratio of error variances known, 32 Approximate distribution of instrumental variable estimators for simple model, 53 Distribution of estimators when the observed explanatory variable is controlled, 8 1 Large sample distribution of a function of sample means, 85 Large sample distribution of first two sample moments, 89 Limiting distribution of sample second moments containing fixed components, IID observations, 92 Limiting distribution of sample second moments containing fixed components, independent observations, 94 Limiting distribution of estimators for vector model with error covariance matrix of explanatory variables known, 108

2.3.1

Maximum likelihood estimators for vector model with no error in the equation, 124

2.3.2

Limiting distribution of estimators for vector model with no error in the equation. Limit for small error variances and (or) large sample size, 127 Limiting distribution of instrumental variable estimator, 15 1

2.4.1

xix

xx

2.5.1 2.5.2

LIST OF PRINCIPAL RESULTS

Moment properties of modified estimator for simple model with known error variance and an error in the equation, 164 Vector version of Theorem 2.5.1, 171

2.5.3

Moment properties of modified estimator for model with no error in the equation, 173

3.1.1

Limiting distribution of weighted estimator for unequal error variances and an error in the equation, 187

3.1.2

Limiting distribution of weighted estimator for unequal error variances with no error in the equation, 190

3.1.3

Limiting distribution of maximum likelihood estimator for model with unequal known error covariance matrices, 218 Limiting distribution of estimators for nonlinear model with no error in the equation and known covariance matrix, 240 Limiting distribution of estimator of variance for nonlinear model, 243 Multivariate maximum likelihood estimator, error covariance matrix known up to a constant, 293 Multivariate maximum likelihood estimator, error covariance matrix estimated, 296

3.2.I 3.2.2 4.1.1 4.1.2 4.1.3 4.1.4 4.1.5 4.2.1

4.2.2 4.2.3 4.2.4 4.3.1 4.3.2 4.B.1

Multivariate likelihood ratio statistic, 301 Strong consistency of multivariate maximum likelihood estimators, 303 Limiting distribution of multivariate maximum likelihood estimators, 305 Limiting distribution of least squares estimator for structural model, 323 Least squares estimation of a diagonal covariance matrix, 329 Limiting distribution of least squares for normal functional model, 339 Equivalence of least squares estimators for the normal functional and structural models, 342 Limiting distribution of estimators for factor model, 360 Condition under which the factor model is not identified, 372 Almost sure convergence of the estimator maximizing a continuous function, 398

LIST OF PRINCIPAL RESULTS

xxi

4.B.2

Limiting distribution of estimator maximizing the normal distribution likelihood-structural model, 398

4.B.3

Limiting distribution of estimator maximizing normal distribution likelihood-functional model, 401

4.B.4

Limiting distribution of least squares estimator computed from sample covariances, 402

4.c. 1

Maximum likelihood estimator for model with known singular error covariance matrix, 404

List of Figures

Figure

1.2.1 1.3.1

Title

2.2.2 2.5.1

Residual plot for the corn-nitrogen data, 26 Pheasant data and estimated structural line, 36 Estimated minimum distance line and estimated true values for two types of cells, 42 Plot of residuals against estimated true x values, 67 Prediction in a second population, 80 Plot of residual against estimated true value for value orientation, 116 Normal probability plot for residuals, 117 Histogram for 2000 maximum likelihood estimates, 168

2.5.2

Histogram for 2000 modified estimates, 168

1.3.2 1.5.1 1.6.1 2.2.1

3.1.1 3.1.2 3.2.1 3.2.2

Plot of mean of two Y observations against mean of two

X observations for pig farrowing data, 209

Plot of weighted residuals against estimated true values, 212 Estimated function and observed value for wave velocity, 233 Estimated ellipse and observed data for image of hip prosthesis, 247

3.2.3

True function and maximum likelihood estimated function for 120 observations generated by quadratic model, 249

4.3.1

Deviation Z , , - it,plotted against ftl,368

xxiii

Measurement Error Models Edited by WAYNE A. FULLER Copyright 0 1987 by John Wiley & Sons, Inc

CHAPTER 1

A Single Explanatory Variable In Section 1 of this chapter we introduce the linear model containing measurement error and investigate the effects of measurement error on the ordinary least squares estimators. Estimation for a simple model is considered. In Sections 2 and 3 we study the two-variable measurement error model for two types of information about the measurement error variances. Because the objective is to introduce the reader to models and estimation procedures, we concentrate o n the normal bivariate model. Sections 4 and 5 treat situations in which three variables are observed. The estimation methods employed in Sections 2-5 are closely related to maximum likelihood estimation. Because of the simple nature of the models, most estimators can be obtained by the method of moments. Some specialized models and methods are considered in Section 6. Later chapters will be devoted to models of higher dimension and to an expanded treatment of models with fixed independent variables measured with error. In particular, Sections 2.1 -2.4 of Chapter 2 extend the results of Sections 1.1 -1.4 of this chapter to vector explanatory variables.

1.1. INTRODUCTION 1.1.1.

Ordinary Least Squares and Measurement Error

The classical linear regression model with one independent variable is defined by ~=ljlo+ljllx,+e,,

t =

1,2, . . . , n,

(1.1.1)

where (x,, x2, . . . ,x,) is fixed in repeated sampling and the e, are independent N(0, oee)random variables. We shall use the convention of identifying the 1

2

A SINGLE EXPLANATORY VARIABLE

mean and variance of a random variable within parentheses and shall use N to identify the normal distribution. It is well known that the least squares estimator

is unbiased for PI and has the smallest variance of unbiased linear estimators. In a second form of the regression model, the x, are assumed to be independent drawings from a N(pL,,oxx)distribution. In the second model it is also assumed that the vector (el, e 2 , . . . ,en) is independent of the vector (xl,x2, . . . , x,,), The estimator is the maximum likelihood estimator for Pi and is unbiased for under both models. We shall study models of the regression type where one is unable to observe x, directly. Instead of observing x,, one observes the sum

b1

X I = XI

+ u,,

(1.1.2)

where u, is a (0, uuu)random variable. The observed variable X I is sometimes called the manifest variable or the indicator variable. The unobserved variable x, is called a latent variable in certain areas of application. Models with fixed x, are calledfunctional models, while models with random x, are called structural models. To aid in remembering the meaning of functional and structural, note that “F” is the first letter of the words fixed and functional, while “S” is the first letter of the words stochastic and structural. As an example of a situation where x, cannot be observed, consider the relationship between the yield of corn and available nitrogen in the soil. Assume that (1.1.1) is an adequate approximation to the relationship between yield and nitrogen. The coefficient p , is the amount that yield is increased when soil nitrogen increases one unit. To estimate the available soil nitrogen, it is necessary to sample the soil of the experimental plot and to perform a laboratory analysis on the selected sample. As a result of the sampling and of the laboratory analysis, we do not observe x, but observe an estimate of x,. Therefore, we represent the observed nitrogen by X I , where X,satisfies (1.1.2) and u, is the measurement error introduced by sampling and laboratory analysis. The description of the collection of the soil nitrogen data permits two interpretations of the true values x,. First, assume that the fields are a set of experimental fields managed by the experiment station in ways that produce different levels of soil nitrogen in the different fields. For example, the application of varying rates of fertilizer and the growing of different fractions of legumes in the rotations would produce different levels of soil nitrogen. In such a situation, one would treat the true, but unknown, nitrogen levels in the different fields as fixed. On the other hand, if the fields were a random

1.1.

3

INTRODUCTION

sample of farmers’ fields in the state of Iowa, the true values of soil nitrogen could be treated as random variables. Let us investigate the effect of measurement error on the least squares coefficient in the simple model (1.1.1) and (1.1.2), under the assumption that the x, are random variables with ox, > 0. We assume (xt,

e,, ~,)l * NI[@.x, 0, OY, diag(oxx, gee,

ouu)],

(1.1.3)

where * NI is an abbreviation for “distributed normally and independently,” and diag(o,,, bee,ouu)is a diagonal matrix with the given elements on the diagonal. It follows from the structural model (1.1.3) that the vector ( y , X,)’, where Y; is defined by (1.1.1) and X, is defined in (1.1.2),is distributed as a bivariate normal vector with mean vector

ENY, XI> = (PY,P x ) = (Po + P I P , ,

PA

and covariance matrix (1.1.4) Let

be the regression coefficient computed from the observed variables. By the properties of the bivariate normal distribution,

+ ~uu)-lou = YI

m c ) = G;~*jly=Pl(~u

(1.1.6)

We conclude that, for the bivariate model with independent measurement error in X, the least squares regression coefficient is biased toward zero. It is important to remember that Equation (1.1.6) was derived under the assumption that the measurement error in X, is independent of the true values, x,, and of the errors, e,. One way to describe the effect of measurement error displayed in (1.1.6) is to say that the regression coefficient has been attenuated by the measurement error. There are a number of names for the ratio K,, = o;~o,, that defines the degree of attenuation. The ratio is called the reliability of X, in the social science literature, but because reliability is a heavily used word in statistics, we call K,, the reliability ratio. The reliability ratio is called heritability in genetics. An observed characteristic of a plant or animal, the X value, is called the phenotype and the unobserved true genetic makeup of the individual, the x value, is called the

4

A SINGLE EXPLANATORY VARIABLE

genotype. The phenotype is the sum of the genotype and the environment effect, where the environment effect is the measurement error. Because the bias in $Ilc as an estimator of fll is multiplicative, the test of the hypothesis that p1 = 0 remains valid in the presence of independent measurement error. That is, if model (1.1.1)-(1.1.5) holds with p1 = 0 and geu= 0, the population correlation between X,and Y, is zero for all values of cr,,. It follows that the usual regression “f test” of the hypothesis that pl = 0 has Student’s f distribution when fll = 0, whether or not the explanatory variable is measured with error. The use of the t distribution for hypotheses other than H,:p1 = 0 leads to biased tests in the presence of measurement error. Also, the presence of measurement error will reduce the power of the test of p1 = 0. See Exercise 1.1. For the classical regression model with fixed x,, the estimator

is called a linear estimator. The adjective “linear” is a modifier for the random variables I:and means that the estimator is linear in Y,. The term linear can also be interpreted to mean that the error in the estimator is a linear function of the random variables e,. If there are errors of measurement in the explanatory variable, the error in the least squares estimator is not linear in the full set of random errors. When X,= x, + u,, where u, is measurement error, the error in the least squares estimator is

[I:l

c {(x, - m,

- E)

1

+ (u, - W O , - io>

?

where u, = e, - utpl.The measurement errors u, enter as squares and as products with x, in the denominator, and as squares and as products with e, in the numerator of yIlc - pl. The covariance between u, and u, produces the bias characterized in (1.1.6). It is possible for the measurement errors u, to be correlated with the true values x, and with e,. In such cases the terms (x, - Z)(u, - ii) and (u, - @(el - i?) will also make a contribution to the bias. The population squared correlation between x, and is

I:=

I:=

RfY = ( ~ , , ~ Y Y ) - l ~ f Y = G v : % x 7

while the population squared correlation between X,and yl is R i y = (oxxuyy)-lgiy = ~ , , R r y .

1.1.

5

INTRODUCTION

Thus, the introduction of independent measurement error leads to a reduction in the squared correlation, where the factor by which the correlation is reduced is the factor by which the regression coefficient is biased toward zero. As with the regression coefficient, it is said that the correlation has been attenuated by the presence of measurement error. 1.1.2.

Estimation with Known Reliability Ratio

We have shown that the expected value of the least squares estimator of PI is the true PI multiplied by K,,. Therefore, if we know the ratio t7;jox,, it is possible to construct an unbiased estimator of PI. There are a number of situations, particularly in psychology, sociology, and survey sampling, where information about K,, is available. Assume we are interested in a population of individuals possessing a trait x. Examples of such traits are intelligence, community loyalty, social consciousness, willingness to adopt new practices, managerial ability, and the probability that the individual will vote in a particular election. It is impossible to observe directly the value of the trait for a particular individual. One can obtain only an estimate for the individual. This estimate, or response, for an individual is obtained frequently as a score constructed from the answers to a number of questions. The battery of questions is sometimes called an instrument. The score of an I Q test is, perhaps, the best known example. The difference between the individual's value for the trait and the response is the measurement error. Often the instrument has been studied and the ratio K,, is so well estimated that it may be treated as known. The reliability ratio K,, is published as a part of the supporting material for a number of standard instruments, such as IQ tests. Suppose that we draw a simple random sample of individuals from a population for which the ratio K,, is known. An unbiased estimator of the strucrelating a characteristic Y to the true value of tural regression coefficient /I1 a trait x of model (1.1.1) is given by /j - - I * (1.1.7) 1

- K,,

Yld,

where f l C is the least squares coefficient defined in (1.1.5). The coefficient (1.1.7) is sometimes called the regression coefficient corrected for attenuation. Because XI) is distributed as a bivariate normal, the conditional distribution of yIIC is normal and the conditional mean and variance of ylld, given X = (XI, X s , . . . ,XJ,are

(x,

E{PldIXJ = YI,

( 1.1.8)

6

A SINGLE EXPLANATORY VARIABLE

An unbiased estimator of (1.1.9)(for n > 2) is W l G J X }= where

"=

1

(X,- X ) 2

1-

st,

(1.1.10)

n

1 [k; - 7 - yIIG(xt - XI]'. r=

s t = ( n - 21-l

1

The unconditional variance of yIld is the variance of the conditional expectation plus the expected value of the conditional variance. Because the estimator yIIG is conditionally unbiased for yl, the unconditional variance is the ZL (X,- R)2 expected value of the conditional variance. The quantity is distributed as a chi-square random variable with n - 1 degrees of freedom, and the unconditional variance of yIlc is obtained by evaluating the expectation of (1.1.9). For n > 3,

(1.1.1 1) ~ { y I , , > = [(n - 3bxxI-l(flYY - Ylax,). The estimated conditional variance (1.1.10) is an unbiased estimator of the unconditional variance because f l G is conditionally unbiased. Therefore, an of (1.1.7) is unbiased estimator of the conditional variance of

P{j,lX}=

B1

[:;.

1=1

(X,- x ) ' ] - l s t .

(1.1.12)

8,

The conditional and unconditional distributions of are closely related and the estimated variance of the conditional distribution is also an estimator of the variance of the unconditional distribution. This is not true for

(1.1.13) po = F - B,x because the conditional expected value of Bo is a function of X. The limiting

distribution of the estimator of yhere V = n-'

c:=,

Po is established by noting that

Bo = P o

+

-

(B,-

Pl)Z

(1.1.14)

u, and u, = e, - urP1. It follows that n1i2(fl, - Po,

PI - PI) is normally distributed in the limit and that a consistent estimator of the covariance matrix of

(Po, P1)' is

where S,"

and

= (n

- 2)-l

2 (I:- so

t= 1

?{I, IX} is defined in (1.1.12).

- /jlX,)2

I. I.

7

INTRODUCTION

Also, by standard regression results,

(1.1.16)

is distributed as Student's t with n - 2 degrees of freedom. Any linear hypothesis about p1 can be transformed into a hypothesis about y 1 by using the reliability ratio. Therefore, in the bivariate situation, knowledge of the ratio c;~c,, permits one to construct an unbiased estimator of the parameter p1and to apply the usual normal theory for hypothesis testing and confidence interval construction. Unfortunately, these simple results do not extend to the vector-x case. See Section 3.1.3. To construct the estimator (1.1.7), we must assume that the X values are from a distribution where the reliability ratio is known. Generally, this means that the X values of the current study must be a random sample from the population that generated the random sample used to estimate the ratio. For example, assume that the population of graduate students scores higher on IQ tests than does the general population. Then the ratio c i i o x x of an IQ test computed for the general population is not applicable for a sample of graduate students. Recalling that R:, = (cxxdYY)Y:.'

= K,,R&,

an estimator of the squared correlation between x and Y is (1.1.17)

where

& = (mxxmyy)- 'mi,and (my,,mxy, mxx)is the sample estimator of

(cyy,ex,,exx).The

estimator (1.1.17) is said to be the squared correlation corrected for attenuation. It is possible for k:, defined by (1.1.17) to exceed one. In such cases, the maximum likelihood estimator of R& is one and the maximum likelihood estimator of /3, is 8, = (sgn m X Y ) [ m ; ~ ~ ~ ~ m , , 1 1 / 2 . We have studied the effect of measurement error on ordinary regression statistics and have demonstrated one method of correcting for that effect. We now give an indication of the order of magnitude of such effects. Table 1.1.1 contains estimates of the effect of measurement error on the regression coefficient for a number of socioeconomic variables. The attenuation coefficients, denoted by xXx, of the table are the correlations between two determinations on the same characteristic. For continuous variables, such as income, the attenuation coefficient is c;~e,,, For zero-one variables,

8

A SINGLE EXPLANATORY VARIABLE

TABLE 1.1.1. Attenuation coefficients for selected variables Sample Size, R i Y = 0.5 Variable

K,,

Sex" Age"

0.98 0.99 0.92 0.88 0.85 0.17 0.58

Age" (45-49)(0-1)

Educationb Incomeb

Unemployed' (0-1) Poverty statusd(0-1)

so/so 2157 6976 132 58

35 14 5

MEM Superior 97

171

27 19 15 11

7

All persons. Persons 14 and over. Persons 16 and over. All families. Unemployed attenuation coefficient is from Fuller and Chua (1984). All other coefficients calculated from data in U.S. Department of Commerce (1975). a

such as sex, the correlation between two independent, identically distributed, determinations defines the multiplicative effect of measurement error on the regression coefficient of the latent class model. See Section 3.4. The estimates of the table were constructed from repeated interview studies conducted by the United States Bureau of the Census. In such studies, the same data are collected in two different interviews. Most of the estimates of Table 1.1.1 come from a comparison of responses in the 1970 Census with the same data collected in the Current Population Survey. In survey sampling the measurement error in data collected from human respondents is usually called response error. The increase in response error as one moves down the table is associated with a corresponding increase in the complexity of the concept being measured. Sex and age are relatively well-defined characteristics, while income, poverty, and unemployment are all relatively complex. In fact, whether or not a family is above the poverty level, or an individual is unemployed, is determined by responses to a number of questions. Age illustrates how subdividing the population can increase the effect of measurement error. For the population as a whole about 1% of the observed variation in age is due to measurement error. However, few studies investigate the effect of age for the entire population. If one is interested in a single 5-year category, such as ages 45-49, the effect of measurement error on the estimated regression coefficient increases to about 8%. Measurement error variance is about 15% of total variation for income. This level of variation is typical of many related variables, such as occupational status and socioeconomic status. As with age, the percent of variation

1.1.

9

INTRODUCTION

due to measurement error increases if we restrict the population. For example, K,, = 0.82 for persons with some income. The 50/50 sample size given in the table is the sample size for which 50% of the mean square error of an ordinary least squares regression coefficient is squared measurement error bias, given that R i Y for the observed variables is 0.5. If the sample size is larger than that of the table, more than half of the mean square error is due to squared bias. The last column of Table 1.1.1 contains the sample size for which the correction-for-attenuation estimator (1.1.7) has the same mean square error as the ordinary least squares estimator in a population with R i Y = 0.5. For any larger sample size, the corrected coefficient of (1.1.7) has smaller mean square error. The sample size is determined by solving the equation (n - 3)-'(1 - Ri,,)

+ (1 - K , , ) ~ I c ; ~=R(n~-~ 3)-'~;:(1

-

Riy)

(1.1.18)

for n, where the left side of the equality is the mean square error of the ordinary least squares coefficient and the right side of the equality is the variance of the estimator (1.1.7), both in standardized units. This equation was derived under the assumption of bivariate normality, but it is a useful approximation for a wide range of distributions. References containing discussions of measurement error and its effects include Morgenstern (1963), Cochran (1968), Hunter (1980), and Pierce (1981). Dalenius (1977a-c) has given a bibliography for response errors in surveys. 1.1.3. Identification The model (1.1.1)-(1.1.3) can be used to illustrate the idea of identification. Identification is a concept closely related to the ability to estimate the parameters of a model from a sample generated by the model. For the purposes of our discussion we consider a model to be a specification of: (a) Variables and parameters of interest. (b) Relationships among the variables. (c) Assumptions about the stochastic properties of the random variables. It is understood that the parameters of interest are specified by a vector

8 E 0,where 0 is the space of possible parameter values and the dimension

of 8 is the minimum required to fully define the model. The observable random vectors have a sampling distribution defined on a probability space. Let Z be the vector of observable random variables DeJinirion Z.I.I. and let F,(a: 8 ) be the distribution function of Z for parameter 8 evaluated at Z = a. The parameter 8 is identified if, for any 8, E 0 and 8, E 0,8, # 8,

10

A SINGLE EXPLANATORY VARIABLE

implies that

Fz(a: 0,) # Fz(a: 0,) for some a. If the vector 8 is identified, we also say that the model is identified. Definition 1.1.2. The parameter Bi,where Bi is the ith element of 8, is identified if no two values of 0 E 0,for which Bi differ, lead to the same sampling distribution of the observable random variables. The model is identified if and only if every element of 0 is identified.

For the model of Section 1.1.1, Equations (1.1.1) and (1.1.2) give the algebraic relationship among the variables x,, e,, u,,X,, and where only I; and X I can be observed. For the normal structural model, the random vector (x,, u,, el) is distributed as a multivariate normal with G,, = ox,, = G,, = 0 and c,, > 0. This exhausts our prior information about the distribution of the variables. The vector of unknown parameters for the model is 0 = (pxlox,, flee, ouu,PO, P1). Under the assumptions, the distribution of the X I ) is bivariate normal. Hence, the distribution of (q,X,) observations is characterized completely by the elements of its mean vector and its covariance matrix, a total of five parameters. Because the model contains six parameters, there are many different parametric configurations (different 0) that lead to the same distribution of the observations. Therefore, the model is not identified. For example, the parameter sets

x,

(x,

81 = ( ~ x g, x x ,

gee, auu,

Po,

PI)

= (1,

1, 1, 1, 1, 1)

and 02

= (Px,

a x x , Gee, g u u ,

PO, 81) = (1, 2, 1.5,0, 1.5,0.5)

are both such that

While the normal structural model is not identified, one of the parameters is identified. The mean of x is equal to the mean of X. Thus, the parameter p x is identified because, given the sample distribution, the parameter vector 8 is restricted to a subspace where p, is equal to the mean of X. For a model that is not identified, it is possible for the sample distribution to contain some information about the parameters. We have seen that one of the parameters, the mean of x, is identified even though the other parameters are not identified. In fact, the sample distribution associated with model (1.1.1)-( 1.1.3) contains additional information about the parameters. We note that we are able to detect the situation P, = 0 from the distribution of

1.1.

11

INTRODUCTION

x).

That is, given o,, = 0, we have PI = 0 if and only if oxy = 0. However, we are still unable to determine o,, and IT,, when PI = 0. In general, given CT,, = 0, the distribution of ( X t , I;) permits one to establish bounds for ouu.Because o,, 2 0, we need only establish an upper bound for ouu.From the equations following (1.1.4), we can obtain an equation for o,, in terms of ouuand an equation for o,, in terms of gee.These-equations are

(Xt,

o'ee = o y y

nu,

=

- o;Y(~~xx- ~ u u ) - ' ,

o x x - ~ : ~ ( I T Y Y - gee)-'*

A maximum value for o,, is obtained by setting gee= 0 and a maximum value for oeeis obtained by setting o,, = 0. These bounds lead to the bounds for the parameters displayed in Table 1.1.2. When PI # 0, the bounds for PI are the population regression of Y on X and the inverse of the population regression of X on Y . The reader should remember that the assumptions vXx> 0 and oeu= 0 were used in this development. It is not posssible to set bounds for the parameters if ce, is also unknown. To summarize, the sampling distribution of ( X , , I;) tells us something about the measurement error model (1.1.1)-(1.1.3). However, the model is not identified because it is not possible to find a unique relationship between the parameter vector of the distribution of (X,, and the parameter vector 8. If we are to construct a consistent estimator for the vector 8, our model must contain a specification of additional information. In Section 1.1.2, knowledge of o&o,, enabled us to estimate the remaining parameters. In the following sections we use the method of moments and the method of maximum likelihood, methods that are often equivalent under normality, to construct estimators of the unknown parameter vector 8. The properties of the estimators depend on the type of information that is used to identify the model.

x)

TABLE 1.1.2. Bounds for the parameters of model (1.1.1)-(1.1.3) obtained from the distribution" of (X,,

x)

The covariance u,, is zero and u,, > 0.

12

A SINGLE EXPLANATORY VARIABLE

REFERENCES Allen (1939), Cochran (1968), Fisher (1966), Moran (197 l), Spiegelman (1982), Department of Commerce (1975).

US.

EXERCISES 1. (Section 1.1.1)Assume that model (1.1.1)-(1.1.3) holds with u,, = 1, uee= 0.1,and n = 100. Compute the power of the regression t test ( 0.05 level) of the hypothesis 8, = Oagainst H A :8 , # 0 at P , = 0.063for u,, = 0 and for u,, = 0.5. You may approximate the noncentral t distribution ' 2 variance 1. by the normal distribution with mean E { ~ l l } [ n - ' E { s ~ } ] - 1and 2. (Section 1.1.1)Let

Y = Po + PIX, + el, X I = x, + u,, k.el, u3' NI[(P,, 40)'.diag(a,,, ueet%AIy a, = U, - E { I;} - yl(X, - E { X , } ) , and 7, = U,:u,y.

-

(a) Show that V { a , } = u,, + y:uuu + (8, - Y~)~U,,. (b) Let fir = rn;irnxr. Give the unconditional variance of fir. Is.it possible for the unconditional variance of .ilc to be smaller than that of m;;rnxy? 3. (Section 1.1.3)In discussing the identification of models we often compared the number of unknown parameters of the model with the dimension of the minimal sufficient statistic. If a model is to be identified, it is necessary that the number of parameters be no greater than the dimension of the minimal sufficient statistic. Construct an example to demonstrate that not all parameters need be identified if the number of unknown parameters is less than or equal to the dimension of the minimal sufficient statistic. 4. (Section 1.1.3)Assume the model y, = Po t xJ,, with

+

-

(x,, e,, d' NI[(P,. 0,W,block diag(u,,,

&,)I

and (I;,X,) = (y,, x,) (e,, u,). Discuss carefully the identification status for cases (a), (b), and (c), giving the parameter space in each case. In all cases all parameters except those specified are unknown. (a) The parameters Po, u,,, and uuuare known. (b) The parameters ueeand u,, are known. (c) The parameters Po, u,,, and u,, are known. 5. (Section 1.1.2)Assume that the twelve (I;,X,)pairs (3.6,3.9), (2.5,2.9), (3.9,4.4), (5.0,5.9), (4.9,5.4),(4.5,4.2), (2.9,2.3), (5.2,4.5), (2.7,3.9, (5.8,6.0),(4.1,3.3),and (5.L4.1)satisfy model (1.1.1)-(1.1.3) with K,, = 0.85. Estimate (Po, PI). Estimate the covariance matrix of your estimators. 6. (Section 1.1.3)Let the following model hold:

2

I; = Po + j - 1

+

(e,,u,, xfY

XflP,

+ e,,

- NI[(O, 0,cxY,block diaidu,,,

o,,,

L)],

where XfI= xI1 u,. Obtain an expression for the expected value of the coefficient for X I , in the ordinary least squares regression of U, on X f I ,x,,, xf3,. . . ,x l p . Establish bounds for 8 , in XI). terms of the population covariance matrix of

(x,

1.2.

13

MEASUREMENT V A R I A N C E KNOWN

7. (Section 1.1.2, Appendix

l.B)Let

-

[(Y,, x,), (e,, 141’ NI[(I, 0, 0,OY, block diag(Z:,,, 0.21)],

(x,

+

where u,, = uyy= 1, uXy= p , and X , ) = ( y , , x,) (el, u,). (a) What is the reliability ratio of X,? Of X,+ Y,? (b) What is the reliability ratio of Y:? Of X:? Of X,Y,? (c) What is the reliability ratio of Y:? Of X:? Of X:Y,? 8. (Section 1.1.2) Verify the equation used to construct the last column ofTable 1.1.1 by showing that the left side of Equation (1.1.18) is the standardized mean sguared error of PIC and that the right side of (1.1.18) is the standardized mean squared error of 0,. both as estimators of P I . 9. (Section 1.1.3) Assume the model

-

+ e,, X,= x, + u,, NI[(P,, 0, OY, block diag(u,,, Zt,,)],

V, = Po + Blx,

(x,. e,, 4’

with ox,, u,,, ururand a,, unknown and u,, > 0. Show that, given any positive definite covariance matrix for (Y,,X,),the interval (- 00, 0 0 ) is the set of possible values for PI. 10. (Section 1.1.2, Appendix l.A) Show that PI of (1.1.7) satisfies

j , -B,

= (n - 1)-1u;~

1 ( x ,- X)(rl - F) + oP(n-l),

I= I

where r, = Y, - x, - y l X , and y,, = E( Y , yI X,]. Hence. i, and b, - PI. where V is dctined in (1.1.14). are independent in the limit. I I. (Section 1.1.2, Appendix l.A) In Section 1.1.2 it was assumed that kXx= u i i u x x was known. Prove the following theorem. ~

Theorem. Let model (1.1.1)-(1.1.3) hold and let ,?,I dependent of ( y qXI), r = 1 , 2 , . . . ,and

fl”*(Z,,

Let

- Kxx)

be an estimator of K,,, where I?,, is in-

N(0, owJ.

j , = Z;xl~lC. Then flliz(81

- 81) 5 “0,

Ki.{Uii(urr

- YIUXY) + K;:Y2y:U,,]].

1.2. MEASUREMENT VARIANCE KNOWN 1.2.1. Introduction and Estimators In this section, we retain the normal distribution model introduced in Section 1.1.1. That is, we assume X , = x, + u,, Y, = Po + P , x , + e,, (xu e,, u,Y NI[(P,, 0, 01’9 diag(g,,, g e e . %”)I.

(1.2.1)

The first equation of (1.2.1) is a classical regression specification, but the true explanatory variable x, is not observed directly. The observed measure of x,, denoted by X,,may be obtained by asking people questions, by reading an imperfect instrument, or by performing a laboratory analysis. It is assumed

14

A SINGLE EXPLANATORY VARIABLE

that the variance of the measurement error, u,,, has been determined, perhaps by making a large number of independent repeated measurements. Estimators of the remaining parameters will be derived under the assumption that a,, is known. X I )is bivariate normal, the sample mean 2 = ( y, P) and Because Z, = sample covariances (myy, m x y , m,,), where, for example,

(x,

C (x,- @(I; - P),

mxy = ( n - I ) - '

1=1

form a set of sufficient statistics for estimation of the parameters. If the parameter vector is identified, the maximum likelihood estimator will be a function of these statistics. See Kendall and Stuart (1979 Vol. 2, Chaps. 17 and 23). If there are no parametric restrictions on the covariance matrix of Z,, then n- ' ( n - l)rnzz is the maximum likelihood estimator of the covariance matrix of Z,, where

m,,

= (n - 1)-

n

* 1 (2,- Z)'(Z, - 2). r= 1

We shall use the unbiased estimator m,, in our discussion. We call mZzthe maximum likelihood estimator adjusted for degrees of freedom. X,)satisfy Recall that, under model (1.2.1), the population moments of (uYY,

oxy,oxx)= ( f l i o x x

+ Gee, f l l u x x , o x x + ow)

(x,

(1.2.2) See (1.1.4). We create estimators of the unknown parameters by replacing the unknown population moments on the left side of (1.2.2) with their sample estimators to obtain a system of equations in the unknown parameters. Solving, we have (1.2.3) a 1 = (mxx - %)-Imxy,

v,

(fix,

= (mxx - uUu, m y y - Blmxy), so, = (8, - s , q .

&ee)

v

The knowledge of a,, has enabled us to construct a one-to-one mapping from the minimal sufficient statistic to the vector (fix,

8x.x- Bo, B 1 , dee)*

For the quantities defined in (1.2.3) to be proper estimators (i.e., to be in the parameter space), 8,. and d,, must be nonnegative. The estimators of ox, and u,, in (1.2.3) will be positive if and only if m,y(mxx

-

UUU)

- m:x > 0.

(1.2.4)

1.2.

15

MEASUREMENT VARIANCE K N O W N

Estimators for samples in which (1.2.4) is violated are Be, = 0,

b, = m,/myy,

with

and 8,, = m -y1y m2x y ,

(fix,8,) defined by the last equation of (1.2.3).

1.2.2. Sampling Properties of the Estimators The sampling behavior of the estimator 8, defined in (1.2.3) is not obtained easily. While the properties of the normal distribution can be used to derive the distribution of the estimator conditional on the observed X values, the expressions are not particularly useful because the conditional mean is a function of the X,and the X,are functions of the measurement errors. (See Exercise 1.13.) Therefore, it seems necessary to u;e large sample theory to develop an approximation to the distribution of P1. The sample moments (m,,, m x y ,m x x ) are unbiased for the population moments and all have variances that are decreasing at the rate n-’. These properties enable us to obtain the limiting distribution of the estimators. Theorem 1.2.1. Let model (1.2.1)hold with a,, know?, CJ,, > 0, and a,, > 0. Then the vector n”’[(/?, - Po), - PI)], where (Po, is defined in (1.2.3), converges in distribution to a normal vector random variable with zero mean and covariance matrix

(8,

8,)

Po - X,Pl = e, - u,B1 and ax”= B,,,, = -Plaulr. Furtherwhere 5 more, nV{(Po, PI)’} converges in probability to r, where

B,, = m x x - a,,,,,and 8,,

=

-Plauu.

Proof. Under the normality assumption, mxy is unbiased for b X yand V m x y } = (n - l ) - l ( ~ x x ~ Y+Y4 Y ) .

(See Appendix 1.B for the moments of the normal distribution.) Because the sample moments are converging in probability to the population moments,

16

A SINGLE EXPLANATORY VARIABLE

we can expand 81

8, in a Taylor series about the population values to obtain = ( o x x - ~u,)-loxY

+ (oxx - cur#)-

-(ox, - ~ u u ) - 2 ~ X Y ( m X X- Cxx)

YmXY

- GXY)

+0pW1).

(See Appendix 1.A.) After algebraic simplification, this expression can be written nlQ,

- 0') = n1/20;:(mxv - ox,)

c:=

x)u,.

+ ~ , ( n -'"),

where m,, =*(n - l)-' (X, It follows that the limiting distribution of n1'2(p1- 8') is the same as the limiting distribution of n1/20;:(mxu - ouu)= n1/20;:(mx,

+ mu,- ou,,).

In a similar manner

fi0 = k - f i J = P o + plx + z- j l ( X + 6 ) = P o - (A - P l ) P x + + O,(n- '1,

where ij = i? - PIU, and the limiting distribution of n'12(fio - Po) is the same as that of n'12F - (/il

- Pl)Pxl.

By the properties of the normal distribution, the covariance between m,, and 0 is zero and, by Corollary 1.C.1 of Appendix 1.C, the limiting distribution of n"2(rnx, - ouv,ii) is bivariate normal. The limiting distribution of nllZ[(/io, fiJ - (Po, /I1)'] follows because, to the order required, the error in the estimators is a linear function of (mxu- o,,,,V), By the n y m a l moment properties, 6,, = oxx+ O,(n-'/z), and we have shown that - p1 = 0,(n-'12). Therefore, ,=1

= o,,

It follows that

nO{jl]

= oo :;,

+ O,(n-

112).

+ o,;t(~,,o,, + p:o;,,) + 0,(n-''2).

Similar results hold for the remaining two entries of (1.2.6).

0

The random variable ur, introduced in Theorem 1.2.1, will be central in our study of measurement error models. Its role is analogous to that of the deviation from the population regression line in ordinary fixed-x regression models. This analogy can be made more apparent if we substitute x, = X, w, into the first equation of (1.2.1) to obtain

1.2.

17

MEASUREMENT VARIANCE KNOWN

The u, differs from the error in the ordinary fixed-x regression model because X , and u, are correlated. In Theorem 1.2.1 it is assumed that the true values x, are normally distributed. The theorem holds if (e,, u,) is independent of x, and the x, satisfy mild regularity conditions. Normality of the error vector (e,,u,) permits us to give an explicit expression for the covariance matrix of the estimators in terms of second moments of the distribution of the original variables. The approximations based on normality remain useful for error distributions displaying modest departures from normality, provided the error variances are small relative to the variance of x. Also see Theorem 2.2.: and Section 3.1. The variance of the limiting distribution of n’/’(P1 - bl) is considerably larger than the corresponding variance for the ordinary least squares regression coefficient constructed with X,.The difference is due to several sources. The divisor, (mxx - ouu),has a smaller expectation than the divisor of the ordinary least squares estimator and the variance of u, is larger than the variance about the ordinary least squares line. The quantity nl/’ was used to standardize the estimators in obtaining the limiting distributions. We choose to use the divisor (n - 1) in the estimated variance (1.2.7) because the variance estimator so defined reduces to the ordinary least squares variance estimator when o,, = 0. For the same reason we used the divisor (n - 2) in the definition of suu. The use of the divisor ( n - 2) in the estimator of ouuleads to an internal inconsistency with the estimator of o,, define! in (1.2.3) because ouu= o,, b$~,,. The estimator n?{(Po, is an estimator of the covariance matrix of the limiting distribution of nilz($, - Po, - bl). Because the expectation of jl is not defined, it is notJechnically correct to speak of (1.2.7) as an estimator of the variance of PI. It is an estimator of the variance of the approximating distribution. Because n9{(bo,b1)’}is a consistent estimator of (1.2.5), it follows that

+

8,)’)

bl

(1.2.8) is approximately distributed as a N ( 0 , 1) random variable. In practice it seems reasonable to approximate the distribution of (1.2.8) with the distribution of Student’s t with n - 2 degrees of freedom. Care is required when using this approximation. Throughout, we have assumed oxx> 0. If nXxis small relative to the variance of mxx - o,,, the approximations of Theorem 1.2.1 will not perform well. A test of the hypothesis that a,, = 0 is given by the statistic 22

=a , ’

2 (X,-

,=1

X)2.

(1.2.9)

If ox, = 0, the distribution of x’ is that of a chi-square random variable with n - 1 degrees of freedom. For the approximations of Theorem 1.2.1 and Equation (1-2.8) to perform well, the population analogue of (1.2.9) should

18

A SINGLE EXPLANATORY VARIABLE

be large. That is, (n - ~ ) C T ~ ~ should ' C J ~ ~be large. As a rule of thumb, one might judge Student's t distribution with n - 2 degrees of freedom an adequate approximation to the distribution of (1.2.8) if (n -

1)-l(mXx

- C T ~ ~ ) - ~
(1.2.10)

This means that the approximation will perform well if n is large and (or) oUuis small. A partial theoretical justification for the rule of thumb is developed in Exercise 1.59. If the x2 value of (1.2.9) is small compared to the tabular value of the chi-square distribution with n - 1 degrees of freedom, other testing and confidence interval methods for should be considered. See Section 2.5.1.

Example 1.2.1. To illustrate the computation of estimates when cuUis known, we use a small data set adapted from Voss (1969) and presented by DeGracie and Fuller (1972).The data given in Table 1.2.1 are yields of corn and determinations of available soil nitrogen collected at 11 sites on Marshall soil in Iowa. The estimates of soil nitrogen contain measurement error arising from two sources. First, only a small sample of soil is selected from each plot and, as a result, there is the sampling error associated with the use of the sample to represent the whole. Second, there is a measurement error associated with the chemical analysis used to determine the level of nitrogen in the soil sample. The variance arising from these two sources has been estimated to be oUu= 57. For the purposes of our example, we assume nu,, is known and that model (1.2.1) holds. The statistics associated with the data of Table 1.2.1 are

(Y,X)= (97.4545,70.6364), mxy

mxx

1

87.6727 104.8818 104.8818 304.8545

'

(1.2.11)

TABLE 1.2.1. Yields of corn on Marshall soil in Iowa

Site 1 2 3 4 5

6

Soil

Yield (Y)

Nitrogen (XI

86 115 90 86 110 91

70 97 53 64 95 64

Site

I

8 9 10 11

Yield ( Y)

99 96 99 104 96

Soil

Nitrogen (X)

50 70 94 69 51

19

1.2. MEASUREMENT VARIANCE KNOWN

To investigate the magnitude of the variance of the unobserved x,, we com pu te au;’mxx = (57)-’304.8545 = 5.35. (1.2.12) If ox, = 0, this ratio is distributed as Snedecor’s F with 10 and infinity degrees of freedom. Because of the large value of the ratio relative to the tabular value, we retain our assumption that ox, > 0. The determinant 87.6727 104.8818 = 10729.88 104.8818 247.8545

and the estimators (1.2.3) are in the parameter space. We have

- oUu = 304.8545 - 57 = 247.8545, pl = [ m x x - o , , ] - l m X y = (247.8545)-’104.8818 = 0.4232, 6,, = m y y - &S,, = 87.6727 - (0.4232)2(247.8545)= 43.2910, Do = F - DIX = 97.4545 - (0.4232)(70.6364) = 67.5613.

d,,

= mxx

The slope of the measurement error line is 8, and the line passes through the sample mean (8,X). The estimated covariance matrix (1.2.6) is 157.3 153 -2.1506

- 2.1506

1

0.0304 ’

( 1.2.13)

where s,, = 59.4440. Because the sample is small and m;io,, is of moderate size, the distribution of the statistic in (1.2.8) may deviate considerably from that of Student’s t . Nevertheless, we use Theorem 1.2.1 and (1.2.8) to establish (0.0288,0.8176) as an approximate 95% confidence interval for PI. By analogy to ordinary least squares theory, we used t = 2.262 from the table of Student’s t with nine degrees of freedom to construct the interval. Because our distribution theory is approximate, no simple relationship exists between the test of the hypothesis Ha: p1 = 0 and the approximate standard error of the estimate of pl. Recall that the ordinary least squares coefficient can be used to test the hypothesis that p, = 0. For these data the least squares test statistic for the hypothesis p1 = 0 is tl = (0.0188)-

1’20.3440 = 2.51,

(1.2.14)

where 0.3440 is the ordinary least squares refiression coefficient. If we use and the standard error computed from v{(pO,/Il)’}, we have t , = (0.0304)- ‘120.4232 = 2.43.

b1

( 1.2.15 )

The two “t statistics” are similar but not identical. The least squares statistic is the preferred method for testing the hypothesis that p1 = 0 but is appropriate for that hypothesis only. The statistic (1.2.8) or the likelihood ratio

20

A SINGLE EXPLANATORY VARIABLE

statistic discussed in Section 2.5 should be used to set approximate confidence intervals for p1 or to test H,:PI = fly, when /$’ # 0. The computed statistics enable one to estimate the correlation between yield I: and true soil nitrogen x, by

R;, 1.2.3.

‘ m i y = 0.5062.

= (c?xxmyy)-

on

Estimation of True x Values

In this section we construct an estimator of the x value that generated the X,).The prediction of given an observation on X,, is disvector cussed in Section 1.6. With knowledge of the parameters of the structural relationship and an observation X,), it is possible to construct an estimator of x, superior to X,. In constructing an estimator of x,, the unknown x, can be treated as fixed or random, and the two assumptions lead to different procedures. We begin by treating the x, as fixed unknown constants. Let the error assumptions of model (1.2.1) hold and let Do, PI, gee,and c,, be known. Then we can write

(x,

x, (x,

(1.2.16)

Equation (1.2.16) is given in the form of the classical regression model, where

x, is the unknown parameter to be estimated. Therefore, if we treat x, as a fixed unknown constant, the best linear unbiased estimator of x, is given by

the generalized least squares estimator 2, = [(PI, 1)

‘(Fly

I)’]-

‘(Pi,

where C,, is the covariance matrix of e, expression for x,reduces to =

[P:cuu

1)K‘(I:- P o , X J ’ ,

= (e,, u,).

(1-2.17)

If Z,, = diag(a,,, o,,), the

+ ~ e e l - ~ [ g u u ( X- P 0 ) P I + ceextl.

The variance of estimator (1.2.17) is ( 1.2.18)

which, for diagonal S:,,,reduces to 2 -1 q x , - x,} = [P1Ue,

+ .3-’

A representation for 2, in terms of the variable v, will be very useful. The quantity x, is a linear combination of I; and XI, say

% = a , ( x - Do) + aJ,,

(1.2.19)

21

1.2. MEASUREMENT VARIANCE KNOWN

with the properties that E { x , l x , } = x,, for all x,,

(1.2.20)

and V{X,lx,} is a minimum for estimators of the form (1.2.19). Now u, is also a linear combination of r, and X,,so that it must be possible to write x, = C , X ,

+ c2v, + c j .

(1.2.21)

To satisfy (1.2.20) for all x,, we must have c 1 = 1 and c j = 0 because E { u, 1 x,} = 0. The variance of (1.2.21) is

V {x,+ C2Ur I x,} = V {u, + c2u,},

(1.2.22)

which is minimized for c2 = - o;'~,,,,, where - c2 is the regression coefficient in the population regression of u, on u,. Therefore, we can write

x, = x,- o,'a,,v,.

(1.2.23)

Using this regression form of the estimator, we have

V{X,- x,} In most situations, estimator

Po, P1, and

(1.2.24)

= a,, - 6", - 1 a,, 2. aeewill

be estimated. Let 2, denote the

(1.2.25) "a,, lY]-V,,1)%3r, - XIY, where e,, = diag(B,,, a,,) and (a,, bl, &), is defined in (1.2.3). The error in the estimator (b,, al, tee) is 0,(n-'12), and

2,L = 3% ) l

2, - x, = [ ( P I ,

ao,

1)X'(al, I)']-'(a,,I)X Ye,, u,)' + 0,(n-'12).

It follows that (1.2.18) can be used to approximate the variance of the estimation error in 2,, and the estimator

q 2 , - x,}

=

[(A,1 ) % 3 a l , l Y r l

(1.2.26)

is a consistent estimator of the variance of it.Using (1.2.23), the estimator 2, can be written 2, = x,- & ; u % u u ~ , ,

where G, =

I; - 8, - X,j1,B,,

(1.2.27)

= a,, - /llauu,and

In (1.2.27) we defined 2, using &, the estimator of a,,"constructed with a divisor of (n - l), instead of suu, which has a divisor of (n - 2). The forms (1.2.25) and (1.2.27) are identical with B,, as the divisor. An alternative form

22

A SINGLE EXPLANATORY VARIABLE

for the estimator of variance that is equivalent to the one in (1.2.26) is

?{it - x,} = 8,,

*-1 A 2 - 8”” OU”.

(1.2.28)

It follows from (1.2.23) that i tand u, are uncorrelated. This property is mirrored in the estimators in that t= 1

@, = 0.

(1.2.29)

The estimation procedure transforms the original vector (k;, X,)into two uncorrelated parts 3, and 8,. This is analogous to the result that the explanatory variables and the deviations from fit are uncorrelated in ordinary least squares regression. In the presence of measurement error it is necessary to estimate x, and it is the estimated x, that is uncorrelated with the residual. In the preceding discussion of the estimation of x,, the quantity x, was treated as fixed. One also can predict x, under the assumption that the x, values are a random sample chosen from a population of x values. Under the assumption of random x,, the population covariance matrix of (k;, X,, x,) is

[

Pi8x.x Plflxx

+ oee

P1oxx

+ ge,

PlCxx

gxx

+

oeu

Plbxx

+ cuu

cxx

(1.2.30)

% X I .

gxx

Therefore, under the assumption of normality, the expected value of x, given 2, is

W,l~,I = P x + r * ( X - P,) + Y 2 W t - P J = Yo

+ YlX f

Y2X,,

where 2, = ( K X,), Y o = (1 - Y 2 ) P x - YIPy, P y = Po (Yl, Y2Y = W

P l % X ,

Using the fact that Zzz= (P1, l)’crxX(jl,1) can be written J+,

where Zzu = ZEu= (c,,,

+ 81Px9 and

ox,)’.

+ ZEe,the expected value (1.2.31)

I2,) = x,- (2,- c z ) ~ i Z l ~ ’ z u , r~,,,,)’

(1.2.31)

(1.2.32)

and e, = (e,, 14,). The conditional variance of x, is

Wt(Z,J = ox, - (71, Y2)(Plaxx,

oxx)’.

It is interesting to compare the estimator (1.2.23) that treats x, as fixed with the predictor (1.2.32) constructed by treating x, as random. In both situations the estimator (predictor) is obtained by subtracting a predictor of u, from X,.If x, is fixed, the predictor of u, is constructed from the random variable u, that contains no fixed part. If x, is random, the best predictor of u, is constructed using the random vector (Y, X I ) .

1.2.

23

MEASUREMENT VARIANCE KNOWN

Treating x, as random and replacing parameters with estimators, the X,), is predictor of x,, given

(x,

+ Fly, +

1,= yo X,

=

where yo = ( 1 -

-

jj2)x- jjl r,

(?l,yJ'

y2xt

(1.2.33)

(Z, - z)m.ii(geu>b u u Y ,

- (Cew

= m,-,l[mz,

guu)'ly

n

1 (Z, - Z)'(Z, - Z , X , - X). r=

(m,,, mzx)= (n - I)-'

1

As before, the use of estimators in constructing the predictor introduces an additional error that is O,(n - ' I 2 ) . If one ignores this additional component, an estimator of the variance of the prediction error is

f{gr - xr 1 Zr> = g u u - ( g e u ,

ouu)mi;(geu,

ouu)'.

(1.2.34)

Example 1.2.2. We use the estimates from Example 1.2.1 to construct an improved estimate of the soil nitroge? at each site. From Example 1.2.1, uuu= 57, be, = 43.29, Po = 67.56, and PI = 0.4232. Treating x, as a fixed unknown constant, the estimator (1.2.25')is A

2,

= 0.4510(x -

67.5613) + 0.8092X,

and the estimated variance (1.2.26) is f(2, - x,}

=

[bft?,' + 0A1]-' = 46.12.

Using the information available in reduces the variance of our estimate of soil nitrogen by about 20%, to a variance of 46 from a variance of 57. Table 1.2.2 contains the 2, and 8,. The reader may verify that (n - l ) - '

1 (x - Y)6, = 43.29 = d,,, n

1=

(n - 1)-'

1

n

1 ( X , - X)C, = -24.13

= -blgUu,

I =1

(n -

1)-1

1 (a, - X)G, = 0.

1-

1

We now construct the predictor of x, under the assumption that the x, are a random sample from a normal distribution. The estimated mean vector is (&

6,) = (Y,X) = (97.4545, 70.6364)

24

A SlNGLE EXPLANATORY VARIABLE

TABLE 1.2.2. Predicted soil nitrogen, estimated soil nitrogen, and deviations from fit

Observed Site 1 2 3 4 5 6

I

8 9 10 11

Observed Nitrogen

Yield YI

x,

4

86 115 90 86 110 91 99 96 99 104 96

70 97 53 64 95 64 50 70 94 69 51

-11.18 6.39 0.01 - 8.65 2.24 - 3.65 10.28 - 1.18 - 8.34 7.24 6.86

Estimated (Fixed x)

Predicted (Random x)

64.96 99.88 53.00 60.10 96.01 62.36 54.64 69.47 90.24 72.26 54.09

65.85 95.29 55.77 61.75 92.03 63.66 51.14 69.65 87.16 72.01 56.69

4

f,

and by (1.2.33),

I:] [ =

87.6727 104.8818]-'[ 104.8818 304.8545

104,88181 = [0,3801] 247.8545 0.6822

*

It follows that the estimated prediction equation for x,, given (Y;, X,)and treating x, as random, is

I, = - 14.5942 + 0.3801 Y;

+ 0.6822X,.

By (1.2.34), the estimated variance of the prediction error is

P(2,- x,((Y;,X,)} = 38.90. If we are willing to treat the x, as a random sample from a normal distribution and to condition on the observed Z,, the variance of the prediction error is about 84% of that of the estimate constructed treating x, as fixed. The values of 2,are compared to the values i f ,where the 2, were computed treating x, as fixed, in Table 1.2.2. Note that the IIvalues are all slightly closer to the mean of X than are the 2, values. See Exercise 1.18. on In Example 1.2.2 we constructed estimators of the x, values under the two assumptions, fixed x, and random x,. In fact, when we considered the x, to be random, we assumed them to be normally distributed. In the fixed-x case, the x values are the same for every possible sample that we are considering. Therefore, if x = (xl, x 2 , . . . , x,) is fixed and one is constructing

1.2.

MEASUREMENT VARIANCE K N O W N

25

unbiased estimators of x, one is asking that the average of the estimator be x, where the average is over all samples in the population of samples of

interest. In the soil nitrogen example the population of fixed-x samples are those samples always selected from the same set of 11 fields (sites). It is possible to think of a sampling scheme wherein the fields of Example 1.2.2 are selected at random and then the soil sampled in each selected field. If we consider the total sampling scheme consisting of random selection of fields combined with random selection of soil within fields, we have a population of samples in which the x values are random. However, it is also legitimate to consider the subset of all possible samples that contain the 11 fields actually selected. That is, we may restrict our attention to the conditional distribution of sample outcomes conditional on the particular set of 11 fields that was selected. When we restrict attention to the subpopulation of samples of soil selected from the 11 fields, we are treating the x values for those 11 fields as fixed. When the data are collected in an operation where fields are randomly selected, it is legitimate, in one context, to treat the x values as randomfor example, to estimate the variance of x-and in another context to treat the x values as fixed-for example, to estimate the individual x values for the 11 fields. In the first context the population of interest is the overall population of fields from which the sample was randomly selected. In the second context the population of interest is the set of 11 fields. 1.2.4.

Model Checks

In ordinary regression analysis it is good practice to plot the residuals from the fitted regression equation. For examples of such plots, see Draper and Smith (1981, Chap. 3). The construction of plots remains good practice when fitting measurement error models. The form of the plots for ordinary least squares and for measurement error models will differ somewhat because of the different nature of the measurement error problem. For an ordinary least squares problem, a common plot is that of the residuals against the independent variables. This plot often will give an indication of nonlinearity in the regression, of lack of homogeneity of the error variances, of nonnormality of the errors, or of outlier observations. For a fitted measurement error model, the quantites that correspond most closely to the residual and to the independent variable of ordinary least squares are G, and 2,, respectively. A measurement error model such as that given by (1.2.1) postulates constant variance for e, and u,. Therefore, u, = e, - plu, will also have constant variance. Because (e,, u,) is independent of x,, the expected value of u, given x, is zero. The best estimator of x, is 2,

26

A SINGLE EXPLANATORY VARIABLE

defined in (1.2.17) and (1.2.23). Under normality, u, and 2, are independent. Therefore, the mean of u, is zero for all 2, and the variance of v, is G,, for all xt. We are able to observe only 6,and i t ,not u, and i, but , the properties of the estimators should be similar to the properties of the true variables. It is suggested that the plot of Ct against P, be used in the same manner as the analogous ordinary least squares plot of deviations from fit against the explanatory variable. That is, the plot can be used to check for outliers, for nonlinearity, and for lack of variance homogeneity. Also, for samples of reasonable size, tests of normality can be applied to the G,. Estimators for models with heterogeneous error variances, for models with nonnormal errors, and for nonlinear models are discussed in Sections 3.1 and 3.2. As in ordinary least squares, the variances of the 6, are not all the same. Because Po and are estimated, the variances of the 6,will differ by a quantity of order n-’. An estimator of the approximate covariance matrix of (G,, 02, . . . , 6,) is given in Section 2.2.3. Using the estimated variances of that section, one could plot the standardized quantities [f{6,}]-1’28r against i t . However, for most purposes, the plot of 8, against 2, is adequate. Example 1.2.3. Figure 1.2.1 is a plot of the G values against the 2 values for the corn yield-soil nitrogen data, where the plotted values are taken from Table 1.2.2. This plot is for a small sample, but the plot contains no obvious anomalies. The mean and the range of the G values are similar for large, medium, and small P values. 00

10

-

8-

. 0

4-

l-

.

a

I 0 I

>

-4 -

-8 -

*

-10 I

0

I

I

I

I

I

1.2.

MEASUREMENT VARIANCE KNOWN

REFERENCES Birch (1964), DeGracie and Fuller (1972), Fuller (1980), Kendall a n d Stuart (1979), Lord (1960), Madansky (1959), Neyman (1951), Neyman and Scott (1948).

EXERCISES 12. (Sections 1.1.3, 1.2) (a) Assume model (1.2.1) holds with P , # 0 and no information availableAon o,, and (rep. Extend the results of Section 1.1.3 by constructing a pair of estimators and POL sych that the parameter Po is contained in the closed interval formed by plim pol and plim POL. (b) Assume that the data below were generated by model (1.2.1).

tot

t

1

2 3 4 5 6

1 8 9 10

XI

3.58 2.52 3.85 5.02 4.94 4.49 2.57 5.05 1.84 5.82

X I 3.94 2.44 3.95 5.93 5.31 4.55 2.60 3.81 1.75 6.04

t

11 12 13 14 15 16 17

18

19 20

XI

3.88 5.93 5.84 4.50 5.81 3.67 5.94 3.54 4.34 5.25

r, 3.32 5.70 5.25 4.24 4.6 I 3.63 4.20 4.03 4.22 4.33

Using these data construct an interval in p, space such that the probability of 8, being contained in a region so constructed is greater than or equal to 0.95. (c) Assume that the data listed below were generated by model (1.2.1). Construct a region in PI space such that the probability of P , being contained in a region so constructed is greater than or equal to 0.95.

t

1 2 3 4 5 6 7

a

9 10

x,

Y

t

x,

Y,

3.62 I .64 3.12 3.28 4.24 2.61 4.73 2.76 4.16 2.50

1.67 0.45 4.43 1.39 1.08 2.41 1.19 2.94 I .92 2.12

11 12 13 14 15 16 17 18 19 20

2.46 0.85 3.59 2.76 4.51 3.18 4.17 4.06 1.53 2.38

0.93 1.05 3.30 1.71 4.74 2.15 1.55 2.6 1 2.42 0.70

A SINGLE EXPLANATORY VARIABLE

28

13. (Section 1.2.2) (a) Let model (1.2.1) hold and let b, be defined by (1.2.3). Fpr samples with a,, and a,, positive, show that the conditional mean and variance of PI, given the sample values of XI, are

where X = ( X , , X,,. . . , XJ. (b) Show that (W

- c)-'dF(w),

where F(w)is the distribution function of a chi;square random variable and c is a positive number, is not defined. Hence, show that E { j , } is not defined. 14. (Section 1.2.2,Section 1.2.3) Reilly and Patino-Leal (1981) give the following data: Replicate ( j )

Condition (i)

v,

xi i

3.44 3.78 3.33 1.11 1.78 2.19

16.61 19.73 17.23 9.19 11.89 10.37 10.43 20.55 17.33

1.85

3

6.86 5.00

Assume the data satisfy the model

Xi = Po + P I X i j + eij, ( x i j ,eij, uijY

-

X,j = xij + uij,

NI[(pL,,0, O)', diag(u,,, u,,,

WI.

(a) Using the nine (Tj,Xi,) observations, estimate (Po, PI, ox,, uee).UsingATheorem 1.2.1, estimate the covariance matrix of the approximate distribution of (Po, PI). Estimate the x values treating them as fixed. Give the estimated variance of the estimated x values. Plot Bij against P,j. (b) Assume that uev= -0.4 and u,, = 0.8. Transform the data so that the covariance of the measurement errors in the transformed variables is zero. Estimate the transformed parameters using equations (1.2.3). Estimate the covariance matrix of the transformed parameters. Then, using the inverse transformation, estimate the original parameters. 15. (Section 1.2.1) Let XI = ( X I o ,Xll) = (1, XI,), and let (Po, /I1) of model (1.2.1) be estimated by (So,, S I M Y = (Mxx - %"I- ' M x , ,

I:=

Xi(x, XI) and Zuy= diag(0, uuu).How do these estimators differ where (M,,, M x x ) = C 1 from those defined in Equations (1.2.3)? 16. (Section 1.2.1) Show that a,, of (1.2.3) satisfies $,

= my" - (bl

- Pi).xr

t O,(n-') = m,, - /l:uUut oP(n-').

1.2.

29

MEASUREMENT VARIANCE KNOWN

17. (Section 1.2) Suppose that the relationship between phosphorus content in the leaves of corn ( Y ) and the concentration of inorganic phosphorus (x) in the soil is defined by the linear model Y, = Po + blx, + e,, X,= x, + u,, (x,, e,, 4’ NI[(P,, 0, O)’, diag(rrx,,,gee,0.25)],

-

where X, is the observed estimate of inorganic phosphorus in the soil. The chemical determinations for 18 soils are as follows: r

u,

x,

t

u,

XI

1

64 60 71 61 54 77

1.18 1.18 2.02 1.26 2.39 1.64 3.22 3.33 3.55

10 I1 12 13 14 15 16 17 18

51 76 96 77 93 95 54 68 99

3.69 3.45 4.9 I 4.91 4.75 4.91 1.70 5.27 5.56

2 3 4 5 6 7 8 9

81

93 93

(a) Calculate estimates of the parameters of the model. (b) Estimate the covariance matrix of the approximate distribution of (Po, P1). (c) Estimate the true x, values treating the x, as fixed. Estimate the variance of these estimates. (d) Plot 8, against L,. 18. (Section 1.2.3) (a) Using the I?, values of Table 1.2.2, regress X, on 0, using an ordinary regression program. Do not include an intercept in the regression. Verify that the deviations from fit of this regression are the 2, values of Table 1.2.2. How would you interpret the "Y-hat" values obtained from a regression of X I on u,? (b) Verify that the f, values of Example 1.2.2 satisfy A

2, -

x = [B""B,, + Beer7..]

~

.

%""BXX(L,- S),

where L, is defined in (1.2.25) and f, is defined in (1.2.33). Show that i l=

x + &?,

-

X),

where y* = [3{q,)]-'C{,<,, x,}, q, = X , - u , u ~ ' u u , ,and . P{q,} is an estimator of the variance of 9,. (c) Use u, = (e,, u,)(l, -PI)' and the equality -

x, =

[(PI, I)% '(PI, I)']- '(PI, 1)K'(el, 4'

to show that E{(x, - x,)u,} = 0. (d) Compute I 2: for Example 1.2.1. Express this quantity as a function of the estimated parameters PI, Bxx, 8,, and 9,. 19. (Section 1.2.3) (a) Verify Equation (1.2.32). (b) Show that the vector form of Equation (1.2.34) is

1;;

where i, = ( j , ,2,).

~ { Z ; J Z= ; )zCe -~ c c m ~ ~ ~ r c r

30

A SINGLE EXPLANATORY VARIABLE

1.3. RATIO OF MEASUREMENT VARIANCES KNOWN

1.3.1. Introduction

In this section we shall study parameter estimation for the measurement error model, given information on the relative magnitude of the error variances associated with the two variables in the model. We write the model as Yl

= Po

( K X,)=(Y,,

+ P1-G x,)

(1.3.1)

+ (e,, urh

where (y, X,) is observed, y , is the true value of the dependent variable, x, is the true value of the independent variable, and (e,, u,) is the vector of measurement errors. We have retained the terms “independent variable” and “dependent variable” although the model is symmetric in x and y for p1 # 0. The model (1.3.1) with oeu= 0 and the ratio

s = au;lo,, known, can be called the classical errors-in-variables model. R. J. Adcock (1877, 1878)considered estimation for the model under the (implicit) assumption that 6 = 1 and K. Pearson (1901) suggested the same estimator obtained by Adcock. C. H. Kummell (1879) considered the problem for a general 6, though he did not formulate the model in precisely our manner. The model was treated in considerable detail by Koopmans (1937). Others discussing the model include Tintner (1945), Lindley (1947), Madansky (1959). Anderson (1951b), Barnett (1967), Moran (1971), Kendall and Stuart (1979, Chap. 29), and Fuller (1980). We devote considerable space to this model for two reasons. First, it is an important model in its own right, furnishing an approximation to real world situations. Second, a careful examination of this simple model helps one to understand the theoretical underpinnings of methods for other models, including those of Sections 1.4 and 1.5 and Chapters 3 and 4. 1.3.2.

Method of Moments Estimators

To introduce estimation for the model we assume that ( x t , e,, u,)are normally and independently distributed, (% e,, l4,)’

-

NI[(P,,

0, OY, diagb,,,

oee,

ouu)],

where o,, = 60,~.Then, under the specification (1.3.1) and (1.3.2),

(1.3.2)

I .3.

31

RATIO O F MEASUREMENT VARIANCES KNOWN

We assume that a sample of n observations on (k;, X , ) is available. To obtain estimators of Po and PI, we recall that the sample mean and covariance matrix are minimal sufficient statistics for the normal distribution estimation problem. Let the sample mean be ( y, and let the sample covariance matrix be

x)

where Z, = ( y , X , ) and the elements of m, are m Y Y ,m x y , and m x x . We define estimators by equating the sample moments to their expectations. Under the normality assumption the estimators so obtained are the maximum likelihood estimators adjusted for degrees of freedom. That is, the estimators of crxx and oUudiffer from the maximum likelihood estimators by a factor of n(n - l)-’. We have myy

+

= &?,, 68,,,

(mxu, mxx) = (

fix = 2,&,

=

B I L ,

r - j , x ,and 8,, = 68,,.

from (1.3.3) that

+ a*uux

(1.3.3)

For samples with m,, # 0, it follows

myy - Sm,, = ):S,, Bl8xx

8,,

- 68,,,

( 1.3.4)

(1.3.5)

= mxy,

and j:m,,

- fll(myy - am,,)

- ~m,, = 0.

(1.3.6)

There are two solutions to (1.3.6) *and th? two roots are real, one positive and one negative. Because m x y = p16,,, p1 will have the same sign as m x y . Therefore, we choose the root 8 1 =

myy - dm,,

+ [(my, -

+ 46m:,]”*

2mXY

(1.3.7)

as our estimator of pl. Given bl, the estimators of the other parameters are given by (1.3.3) and (1.3.5). To obtain an explicit exprfssion for 8,,, we can solve (1.3.5) for b1 and substitute the expression for p1 into (1.3.4) to obtain

SS:,

+ ( m y y- 6mxx)8,, - m i y = 0.

The quadratic has a positive and a negative root. The estimator of ox, is the positive root 8,, = (26)-’{[(mYy - d n ~ , , ) ~+ 46m:y]”2

- ( m y y- Sm,,)}.

(1.3.8)

32

A SINGLE EXPLANATORY VARIABLE

By (1.3.7) and (1.3.8),

/j 1

*-I

- ox,

(1.3.9)

mxy.

We canAconstructan explicit expression for a, by substituting the expressions for /llBXx and 8,, from the mxx and m x y equations of (1.3.3) into the myy equation to obtain

+

68:, - (mYy 6mxx)8,, It follows that

+ mYYmXX- mir = 0. +

+

8,, = (26)-l{mYy Sm,, - [(my,, - 6mxx)2 46m:y]1'2},

(1.3.10)

where we choose the smallest root of the quadratic equation so that 8,, < mxx*

A

If mxy = 0 and m y y < Gm,,, it is natural to take p1 = 0 and d,, = m x x 6-'myY. If m x y = 0 and myy > hxx, the p u m p t i o n that ox, is positive is suspect and it is reasonable to consider j1to be undefined. For random variables with a density, the probability that mxy = 0 is zero. Because the estimators are continuous differentiable functions of the sample moments, the methods of Appendix l.A can be used to establish the limiting distribution of the estimators.

Theorem 1.3.1. Let the model defined by (1.3.1) and (1.3.2) hold with

o,, > 0 and ox, > 0, where 6 > 0 is known. Let 6 = (Do, pl,o,,,,)' and let h =

(bo,/jl, 8,J

be defined by (1.3.3), (1.3.7), and (1.3.10). Then, as n -+

00,

where Tee =

+

o,, = gee Pto,,, a,, = -plouu,and the symbol 5 is used to denote convergence in distribution, also called convergence in law.

x,

Proof. The estimators are functions of the vector (F, m y y ,m X y ,m x x ) = b. This vector differs from the sample mean of the vectors [ X,, (T - p#, (X,- pX)(yt - py), ( X , - p J 2 ] by a quantity that is O,(n- '). In fact, for normal (yt, X,), it is known that (F, X)is independent of (my,, m x y , mxx)and that (myy, inxy,mxx) can be expressed as the mean of (n - 1) independent random vectors. By the assumptions that cxx> 0 and cr,, > 0, the first and

x,

1.3.

RATIO O F MEASUREMENT VARIANCES KNOWN

33

second derivatives of fl are continuous at the point ( p y ,p x , cyy,cxy,axx)= b = E { & } .Therefore, Corollary l.A.l of Appendix l.A can *be used to obtain the limiting distribution of n112((e*- 0). Letting (e* = g(b), we have, for example,

where g2(b) = p1 and the partial derivatives are evaluated at the parameter vector b. Using the derivatives, we have

6- ($1 - Dl)PU, + OP(fi-9, = G 3 - m x u - ~.vvl~uv~uul + O P h -l ) , = (B? + 6)- l h U "- 0 " U ) + O @ l ) ,

B o - Bo = 81

- B,

4"- nu,

(1.3.11)

where mUu= (n - I ) - ' (u, - iT)2, c,, = (6 + f f f ) ~ and ~ ~ ~ ex" , = oUv= -file,,,. Equations (1.3.11) correspond to Equation (1.A.3) of Appendix l.A. The result follows by applying Corollary l.A.l and evaluating the covariance matrix of the leading terms of (1.3.11). 0

I:=,

The assumption of normality was used in the proof of Theorem 1.3.1 only to obtain expressions for the variances of the sample moments. The estifi,) that define the line are often of the most interest and the mators covariance matrix for these estimators is that of Theorem 1.3.1 for normal (e,, u,) under a wide range of assumptions for x,. Comparing the distributions of Theorem 1.2.1 and of Theorem 1.3.1, we see that the variance of the limiting distribution of b1 computed with 6 = ~ u ~ l cknown, ,, is smaller than the variance of the limiting distribution of computed with cuuknown. The difference between the two variances is the quantity

(bo,

b1

uo;2a;

= 2c,;2p:e;".

A? estimator of the covariance matrix of the approximate distribution of 1.3.1 is

(j0,jl)of Theorem

34

A SINGLE EXPLANATORY VARIABLE

where

V{&} S",

= (n - 1)-~ax;2[axxs,, = ( n - 2)-1

+ B,,S,"

- 63,

f: [y1- r - j 1 ( X ,- X ) ] Z

I=1

= (n - 2 ) - ' ( n -

1)(6

+

a,

= -PldUu, a?d 8, is defined in (1.3.8). Because nl"(PI - PI) is approximately normally distributed and n3{j1} is a consistent estimator of the variance of the limiting distribution, the quantity

t =

[WJI- "'(/il

( 1.3.1 3)

- PI)

is approximately distributed as a N(0, 1) random variable. It is suggested that, in small samples, the distribution of the statistic defined in (1.3.13) be approximated by Student's t distribution with n - 2 degrees of freedom. We also suggest that the distribution of the statistic a,;'(n - l)Pyube approximated by that of a chi-square random variable with n - 2 degrees of freedom. Example 1.3.1. To illustrate the estimation of the parameters of model ( 1 . 3 4 we use some data collected by the Iowa Conservation Commission. Table 1.3.1 contains indexes of the number of hen pheasants in Iowa at two

times during the year. These indexes are based on the average number of

TABLE 1.3.1. Measures of the number of hen pheasants in Iowa

Year I976 1975 I974 1973 1972 1971 I970 1969

August Hens

Spring

Hens

August

Y,

x,

Year

8.0 6.0 9.8 10.8 9.7 9.3 9.2 6.9

9.0 6.6 12.3 11.9 11.9

1968 1967 1966 1965 1964 1963 1962

12.0 9.6 7.5

Hens Y, 8.1 8.7 8.7 7.4 10.1

10.0 7.3

Spring Hens

X, 10.9 10.4 10.2 7.4 11.0 11.8

8.2

Source: Data from Iowa Conservation Commission. Data for 1962-1968 have been adjusted.

I .J.

35

RATIO OF MEASUREMENT VARIANCES KNOWN

birds sighted by trained observers traveling a number of specific routes in late April and early May, and again in August. Both measures are subject to error for two reasons. First, the routes are a sample of all possible routes in Iowa. Second, observers cannot be expected to sight all pheasants along the route. In August, it is relatively easy to sight hen pheasants because most hens are accompanied by a brood of chicks at that time. O n the basis of other analyses, it has been estimated that the error variance for the spring count is about six times that for August. For the purposes of the present example, we treat this ratio as known. We do not believe that the number of hen pheasants in August is a perfect linear function of the number in the spring, but such a specification is considered a reasonable approximation. Although we proceed with this model, we must remember that our estimated error variances will contain components due to variations in survival rates as well as that due to observational and sampling error. The working model is Y , = Po

-

+ BlX,?

( X , X,)= ( Y t , x,) (x,,

+ k,

4 1 9

e,, 4' N I [ ( p x ,OY, diag(a,,, 6-'q,,,

a,,)].

The sample means are (F, d )= (8.6667, 10.0467)and the sample covariances are

(my,,,m y x ,mxx) = (1.84952,2.35167, 3.62124). With S = 6-', Equation (1.3.6) becomes 2.35167); - 1.24598)' - 0.39195 = 0 and the estimated parameters of the line are (J0,

1,)= (1.1158, 0.7516).

The data and the estimated structural line are plotted in Figure 1.3.1. If, at first glance, the line does not appear to be drawn correctly, recall that the ratio of a,, to cr, is 6-I. This means that horizontal deviations are six times more important in determining the line than are vertical distances. TCe estimated covariance matrix of the approximate distribution of t B O 9 P I ) is 0.95929 - 0.09291 B1)') = -0.09291 0.00925 ' w

o

3

[

where 8,, = 3.1290, 8,, = 0.4923, 6," = 0.3700, and s,, = (n - 2)-'(n - 1)(S

1

+ &,,= 0.38783.

36

A SINGLE EXPI.ANATORY VARIABLE

For illustrative purposes we have chosen a rather small data set. However, even in this small example, different conclusions can be drawn if the measurement error is ignored. The ordinary least squares regression of Yton X, is = 2.142 + 0.649Xl, (0.845) (0.083)

where the numbers in parentheses are the ordinary least squares estimated standard errors. The estimated line constructed under the assumption 6aee= auuhas an intercept of 1.116 with a standard error of 0.979. Therefore, the ordinary least squares line has a significant positive intercept while one can easily accept the hypothesis of a zero intercept for the structural line.

n o

1.3.3. Least Squares Estimation

We now derive an estimator of PI for model (1.3.1) by the method of least squares. In the classical least squares problem the estimator of (Po, fll) is the (Po,b,)for which the sum of squared vertical deviations of the points x,) from the line y , = Po + pix, is minimized. The vertical distance is the proper statistical distance to consider because the x values are fixed in repeated sampling; the observation vector deviates from the true line by the distance el in the vertical direction.

(x,

I .?.

37

RATIO OF MEASUREMENT VARIANCES KNOWN

In the errors-in-variables situation an observation can deviate from the true line in both the horizontal and vertical direction. That is, the X,) = point on the line is (Po + Pix,, x,) and the observed point is (Po + Pix, + e,, x, + u,). The squared Euclidean distance from the observation to the point on the line that generated the point is

(x,

[x

-

(Po + B1x,)]2 + (X, - x,)'

= e:

+ u:.

(1.3.14)

If e, and u, are independent random variables with unit variance, the Euclidean distance is the proper statistical distance to consider. If e, and u, are independent, but with different variances, the squared statistical distance is [statistical distance]'

=e:'a ;,

+ a,i'u,".

( 1.3.15)

If the covariance matrix of (e,, u,) is Zet,where XEEis nonsingular, then the squared statistical distance from X , ) to the point on the line associated with x, is

(x,

( x - Bo - BlX,, x,- x , P i 1 K - P o - SIX,, x,-

XJ'.

(1.3.16)

Using the least squares:riterion we shall choose as our estimators of Po and D1 those values (&,PI) for which the sum of squared statistical distances of the observed points from the estimated line is a minimum. That x*,, i',. . . ,a,) to minimize the sum of (1.3-16)over the n is, we find (&, observations. Given (Po, PI), the x value for the point on the line that is the smallest statistical distance from (Y, X , ) was obtained in (1.2.17) of Section 1.2.3. That x value is

bl,

2, = [(Pl, 1)%YP17 =

x,- a,laU"u,,

1)'1-'fBI, 1KYx - Po. X,Y

+

where cr," = gee - 2C31a,, /?:o,,, a," = cr, - /?lau,,, and u, = PIX,. The corresponding y value is

j;, = Po

+

( 1.3.17)

x - Po -

(1.3.18)

PIX,.

If we substitute (1.3.17) and (1.3.18) into (1.3.16), we find that the minimum squared statistical distance from (X,, y) to the line is (Gee

- 2Plaeu + P:auu>- '(Y1 - Po - PIX,)'.

( 1.3.19)

Thus, the least squares (minimum statistical distance) estimator of (Po, P1) for model (l.3.1), with general error covariance matrix, is the (bo,8,) which minimizes (gee

- 2P1aeu

+ P:auu)-l

n

C ( Y - Po r=1

-

P1XJ2.

(1.3.20)

38

A SINGLE EXPLANATORY VARIABLE

The partial derivative of (1.3.20) with respect to

Po is zero for

j0= Y - p l x .

(1.3.21)

Substituting (1.3.21) into (1.3.20) and dividing by (n - l), we obtain (gee

- 2Plaeu

+ P:cuJ1(mYy - 2PImxy + Pfmxx).

(1.3.22)

If we set c,, = 0 and cilcee= 6 and equate the derivative of (1.3.22) with respect to p1 to zero, we obtain (1.3.6) as the equation defining the estimator of pl. Therefore, the method of moments estimator and the least squares estimator of PI are the same. We consider (1.3.22)further to obtain an alternative form for the estimator. If we set j1= --a2aL1, we can write (1.3.22) as (a:c,,

+ 2a1a2c,, + a ~ e u u ) - l ( a ~ m+y2a1a2mx, y + aimxx).

(1.3.23)

Minimizing the ratio (1.3.23) is formally equivalent to minimizing the Lagrangian a:myy 2a1a2mxy a:mxx - A(atcee 2a1a2ceu a~c,,- A) (1.3.24)

+

+

+

+

with respect to aIand a2,where A is any positive constant. Setting the partial derivatives with respect to a1 and u2 equal to zero, we obtain 2oi1my, 2oi,mx,

+ 2oi,mxy - 23Loiluee- 23Loi,ee, = 0, + 2oi2mxx - 2iloi1ae, - 2iloi2c,, = 0.

(1.3.25)

This system of equations has a nontrivial solution only if (1.3.26) We choose the smallest root of the determinantal equation (1.3.26) to minimize the original ratio (1.3.23).Then the estimator of P1 = - a 2 a ; 1 is given by the solution to (mxx - l e u u ) 8 1 = ~ x -y i c e u * (1.3.27) While not obvious, this estimator of PI is identical to that of (1.3.7) when a,,, = 0 and cee= 60,,.~ Given an estimator (Bo, one can replace (Po, B1) with (bo,bl) in (1.3.17) and (1.3.18) to obtain an estimator of (x,, yJ,

bl),

2, = [(B,, l ) G 1 ( B l , lYJ-l(iL 1 ) X 1 ( X - B o , x,y ,.-I = x,- c,, ~ , " V * t ,

(1.3.28) (1.3.29)

A

9, = B* + /%a,,

+

where 8,, = a,, - 21j1ueu &,, 6," = be,,- B1cuu, and 8, = X - Po - BIX,. It is interesting that the estimator of xt, treating x, as fixed, can be constructed *

A

I .3.

39

RATIO O F M E A S U R E M E N T V A R I A N C E S K N O W N

by subtracting the best predictor of u,, treating u, as random, from X,.That is, expression (1.3.28) is the sample estimator of jl

-x,-u -x,-VP,

1 -

1 -

g,ur

where the best predictor of u,, given u,, is

ii, = u,o~u~c7uu, and o;ulou, is the population regression coefficient for the regression of u, on v,. Also see Equation (1.2.23) of Section 1.2.3. Because the error made in estimating the parameters is small relative to the errors (e,, u,), the variance expression V{x,- x,} is generally adequate for 2,. For an illustration of the exception, see Example 2.3.4 of Section 2.3.4. An estimator of the variance of 2, - x, is

q.4 - X J

= [(iil, W i 1 ( i i 1 , =

U'I-'

* - I 1.2 o u u - 0"" ouu.

(1.3.30)

The second expressions of (1.3.28) and (1.3.30) are valid for singular E,,. In the derivation of this section we treated the unknown x, values as fixed. Just as the estimator of PI was the same as that obtained in Section 1.3.2 under the random-x assumption, the covariance matrix of the approximate distribution given in Theorem 1.3.1 is appropriate with the understanding that ox, stands for the mean squaresof the true x values. If the entire error covariance matrix is known, instead of being known up to a multiple, the estimated variance of PI takes a different form. The estimated variance is

P{&} = (n - 1)-1A;~[Ax,riuu + o,,8,,

where a,, = o,,

-

bUu, ri,, = (1,

-bl)E,,(l,

fix,= fi;(mzz - ~ fi2

= [cot 1)' - (1,

-6 3

(1.3.31)

-bJ,

f i ~ ,

-BJ o,, ouul. ,1.-1

L.

The estimator 8," is superior to s,, because it contains no variation due to variation in the u:, only variation due to estimating PI. The estimator A,, is the maximum likelihood estimator for ox, of the structural model with known C,,, provided the largest root of (1.3.26) is greater than one. It is the mean square of the 2, reducedAbyan estimator of the variance of 2, and equals the estimator (1.3.8) for 1 = 1 . See Exercise 1.24. The estimator A ,, is derived in Section 4.1. In Section 1.3.2 the model specified o,, = 0 and oee= 6o,,, where 6 is known, but oepand o,, are unknown. This is equivalent to the specification

C,, = Y,&02= ouudiag(6, l),

(1.3.32)

40

A SINGLE EXPLANATORY VARIABLE

where o2 = o,, is unknown. Then the smallest root of the determinantal equation (1.3.26), with diag(8, 1) replacing Eee,is

1= (26)-’{rn,,

+ am,,

-

[(rn,, - 6m,,)2

+ 48m:y]”2}.

(1.3.33)

x

Thus, 1= B,,, where B,, is defined in (1.3.10) and the smallest root is an estimator ofAcr2.If n = 2, the estimated line passes through the two observed points and A = 0. Therefore, it is reasonable to define the estimator of o2 for n > 2 by

6 2 = (n - 2 ) - ’ ( n

- 1)i.

(1.3.34)

This definition of B2 is also consistent with the approximate chi-square distribution suggested for c?”,,. For unknown o2 there are three sample covariances (myy, mxy, mxx) available for the estimation of three parameters (oxx,PI, 0 2 ) .For known o2 (known I&&) there are three sample covariances available for the estimation of two parameters (oxx,pl). Therefore, knowledge of the full covariance matrix permits us to check for model validity. The smallest root of the determinantal equation (1.3.26) satisfies

1= (nee- 21jlae, + &,,)-l(myy

- 2/?,mxy

+ b&).

(1.3.35)

The numerator of this statistic is a multiple of the estimator of o,, constructed with deviations from the fitted line. The denominator is an estimator of c,, constructed with the known &. If were replaced by pl, the ratio would be distributed as Snedecor’s F with n - 1 and infinity degrees of freedom. In the next chapter it is demonstrated that (n - 2)- ‘ ( n - 1)x is approximately distributed as Snedecor’s F with n - 2 and infinity degrees of freedom, Therefore, when nee,oeu,and ouuare known, the st!tistic ( n - 2)-’(n - l)A can be used as a test of the model. If ( n - 2 ) - ’ ( n - l ) A is large relative to the tabular value of Snedecor’s F with n - 2 and infinity degrees of freedom, the model is suspect. Our derivation of Equation (1.3.27) demonstrates that the knowledge of o2 does not change the estimator of PI for the functional model. But knowledge of the error variance permits us to construct a test of model adequacy. This can be compared to the case of ordinary fixed-x regression. Knowledge of the error variance does not change the form of the fixed-x regression estimator of p,, but knowledge of the error variance permits construction of a test for lack of fit.

8,

Example 1.3.2. We analyze some data of Cohen, D’Eustachio, and Edelman (1977). See also, Cohen and D’Eustachio (1978) and Fuller (1978). The basic data, displayed in Table 1.3.2, are the numbers of two types of

1.2.

41

RATIO O F MEASUREMENT VARIANCES KNOWN

TABLE 1.3.2. Numbers of two types of cells in a fraction of the spleens of fetal mice

Individual

Number of Cells Forming Rosettes

Number of Nucleated Cells

Y

X

52 6 14 5 5

337 141 177 116 88

7.211 2.449 3.742 2.236 2.236

18.358 11.874 13.304 10.770 9.381

Source: Cohen and D’Eustachio (1978)

cells in a specified fraction (aliquot) of the spleens of fetal mice. Cohen and D’Eustachio (1978) argued, on the basis of sampling, that it is reasonable to assume the original counts to be Poisson random variables. Therefore, the square roots of the counts, given in the last two columns of the table, will have, approximately, constant variance equal to one-fourth. The postulated model is Y, = Po

+ BlX,,

where (y, X,) = (y,, x,) + (e,, u,), is the square root of the number of cells forming rosettes for the tth individual, and X , is the square root of the number of nucleated cells for the tth individual. On the basis of the sampling, the (e,,u,) have a covariance matrix that is, approximately, ZEE= diag(0.25,0.25). The square roots of the counts cannot be exactly normally distributed, but we assume the distribution is close enough to normal to permit the use of the formulas based on normality. Thus, our operating assumption is

(e,, u,)’

-

NI(O,0.251)

and we assume (e,, u,) is independent of xj for all t and j . The statistics associated with the data of Table 1.3.2 are (3.5748, 12.7374) and

(F, x)=

(my,, mxy, m x x ) = (4.5255, 7.1580, 11.9484).

The smallest root of

lmzz - A(0.25)II = 0 is i = 0.69605. Using (1.3.21) and (1.3.27), the estimated parameters of the line are (&, bl) = (-4.1686,0.6079).

12

A SINGLE EXPLANATORY VARIABLE

TABLE 1.3.3. Estimated numbers of two types of cells Observed t

I:

Observed

x,

E,

9,

4

1 2 3 4 5

7.211 2.449 3.742 2.236 2.236

18.358 11.874 13.304 10.770 9.381

7.0508 2.8878 3.8714 2.3403 1.7237

18.4563 11.6078 13.2260 10.7071 9.6929

0.2194 - 0.6009 -0.1772 - 0.1428 0.7015

The estimates of the x, and y, are, by (1.3.28) and (1.3.29),

+

+

it= 1.8503 0.4439Y; O.7302Xt, 9, = - 3.0438 0.2698Y; O.4439Xt.

+

+

The estimated true values are given in Table 1.3.3 and are plotted in Figure 1.3.2 as the dots on the estimated line. The original observations are plotted as crosses. In this example the covariance matrix of the measurement errors is proportional to the identity matrix. Therefore, the statistical distance is proportional to the Euclidean distance and the lines formed by joining (I;,X,) and (j$, 2,) are perpendicular to the estimated functional line. The statistical distance from (Y;, X,) to ( j , , a,) is, by (1.3.16) and (1.3.19), statistical distance =

(0.25)(1

+ &)

Y 6-

4-

2-

+

FIGURE 1.3.2. Estimated minimum distance line and estimated true values for two types of

cells.

I .?.

43

RATIO 01;M E A S U R E M E N T V A R I A N C E S K N O W N

Therefore, the statistical distance is proportional to the absolute value of 6,. The 6,are given in Table 1.3.3. As in Example 1.2.2,the sample correlation between 2, and 6,is zero. The estimation procedure has transformed the X,)into a new vector (C,, it), where the sample covariance observed vector between 0, and 2, is zero. By (1.3.30),the approximate variance of 2, is

(x,

q(2, - x,> = C(0.6079, 1)41(0.6079,1)’I-l = 0.1825. If the model is true, ( n - 2)-’(n - 1 ) x is approximately distributed as Snedecor’s F with three and infinity degrees of freedom. Because F = (3-’)4(0.69605)= 0.9281 is approximately equal to one, the data and the model are compatible. Figure 1.3.2might lead one to conclude that there is a nonlinear relationship between y, and x,. This interpretation should be tempered by two facts. First there are only five observations in the plot. Second, as the F statistic indicates, the deviations are not large relative to the inGependent estimate of the standard deviation of u,, where 6fi’ = O S ( 1 P?)”* = 0.59. Also see Exercise 3.10. We can check the assumption that c(x, - X)2 > 0 by computing

+

aui’mxx = 4(11.9484) = 47.79.

If mxx = ( n - l) - lx(x, - 2)’ were zero, the ratio would be distributed as Snedecor’s F with four and infinity degrees of freedom. It seems clear that mxx is positive and that the model is identified. By (1.3.31), w o ,

where 8,, = -bla,,

SJ>=

= -0.1520,

[

1

1.2720 - 0.0945 -0.0945 0.0074 ’

6,, = 0.3424,

q(2, - x,} = 11.7192, H z = (1 + /$)- ‘(jl, 1) = (0.4439,0.7302),

A,,

= H;m,,H2

-

A hypothesis of interest in the original study is the hypothesis that zero. On the basis of the approximate distribution of Theorem 1.3.1, t = (1.2608)-”’(-4.1686) = -3.71

Do is

is approximately distributed as a N ( 0 , 1) random variable when Po is zero. Thus, the hypothesis that the intercept is zero would be rejected at the 0.001 level. An alternative method of testing the hypothesis that Po = 0 is to estimate p1 using the uncorrected sums of squares and products. Under the null hypothesis that Po = 0, an estimator of PI is W X X

-

hJ ‘h?

44

A SINGLE EXPLANATORY VARIABLE

where (M,,,M,,) = n-'

X , ( x , X,)and

is the smallest root of

JMZz- I(0.25)I) = 0. For these data the smallest root is = 4.0573. If the model is true, then (n - l ) - ' d is approximately distributed as Snedecor's F with four and infinitity degrees of freedom. Because F = 5.07, the hypothesis that Po = 0 is rejected at the 0.001 level. The t test is a test of the hypothesis that y, = Pix, given th!t the alternative model for y, is y, = Po + Pix,. One conjectures that the I test will have smaller po_wer than the t test against the alternative of the affine model, because the 1 test has power against a wide range of 00 alternatives. One cannot hope to specify parametric configurations completely such that the approximate distribution of Theorem 1.3.1 will be appropriate in practice. However, some guidance is possible. One rule of thumb is to consider the approximations adequate if (n - 1)-1ii;:6:u

(1.3.36)

< 0.001.

This is roughly equivalent to requiring the coefficient of variation of (m,, - ouu)-lmXX as an estimator of m;:(mxx ouu) to be less than 5%. This suggestion is supported by Monte Carlo studies such as that of Martinez-Garza ( 1970) and Miller (1984), by higher-order expansions such as those described in Cochran (1977, p. 162), DeGracie and Fuller (1972), and Anderson (1974, 1976, 1977), and by the exact distributional theory of Sawa (1969) and Mariano and Sawa (1972). Also see Exercises 1.59 and 1.60. For the pheasant data of Example 1.3.1,

+

(n - l)-'rii;:&,

= (14)-'[(3.12895)-'(0.49229)]* = 0.0018.

For the cell data of Example 1.3.2,

(n - l)-'A;;a&

= (4)-'[(11.7744)-'0.25]*

= 0.0oO1.

Thus, although the sample size, n, for the pheasant example is larger, we expect the approximations to perform better for the cell example because A;:auu is smaller for the cell example. This is confirmed when we compare exact and approximate confidence intervals for the two examples. See Exercise 1.31 and Example 1.3.3. 1.3.4. Tests of Hypotheses for the Slope

pl,

While we are unable to establish the exact distribution of we can construct some exact tests for the model. Let model (1.3.1)-(1.3.2) hold with 6 =

I .3.

RATIO OF MEASUREMENT VARIANCES KNOWN

known and oe, = 0. We consider the null hypothesis that

fly, then

45

a, = by. If j1=

y, - B - p:(x,- X)= u, - 0, where u, = e, - P~u,, and the variance of u, is ow = b e e

+ @?)2cuu

=

[a + ( P ~ ) ' l ~ u u *

Because we know 6 and because (e,, u,) is normally distributed, we can construct a second random variable, say h,, that will be independent of u, if P1 = fly. We define

+ a 2 X ,= al(Po+ pix,) + a2x, + ale, + a,u,, where a , and a2 (a: + a: # 0) are chosen so that, under the null, the coh, = a,Y,

variance between h, and u, is zero. Because

C{h,, 4) = (a16 - ~zB:)b,Y, h, defined with a, = S - ' P y and a2 = 1 will be uncorrelated with u, under the null. If # 0, the definitions of h, and u, are symmetric. That is, the leads to a pair that is a multiple of the pair hypothesis that jl = -d(j:)-' (u,, A,), but with the identification interchanged. This symmetry means that we must consider two types of alternative hypothesis. First, assume that we know the sign of pl. Then the hypotheses are Ho:

= fly,

H A : p ,

20,

and

PI # P?,

(1.3.37)

where we have chosen nonnegative values as the parameter space for PI, with no loss of generality. Under alternative hypothesis (1.3.37),the null hypothesis p1 = by corresponds to the hypothesis that the correlation between u, and h, is zero. A t test or an F test can be used to test for zero correlation. The null hypothesis that PI = fly is accepted if the hypothesis of zero correlation is accepted, Furthermore, the set of PI with P, E [0, GO) for which the hypothesis of zero correlation is accepted at level a constitutes a 1 - CI confidence set for jl. We now consider the hypotheses, H,:

Pi

H A : p1

= f

p?*

p? # 0

With this alternative hypothesis the symmetry in the definition of u, and h, must be recognized. The acceptance of the hypothesis of zero correlation is equivalent to accepting the hypothesis that PI = or that PI = -6(fl:)-'.

46

A S l N G L E E X P L A N A T O R Y VARIABLE

However, if

= p: and oXx> 0, then

+

V { h , } = [S-'(p:)' l]'Oxx V {u,} = d,, (p:)20,, = [1

+

Therefore, the hypothesis that pothesis

pl =

H,: C{h,,v,} = 0 and

+ [s-'(pY + + (p:)26 - '160,".

1]Ouu7

corresponds to the composite hyV { h , } 2 6 - VCv,}.

Under the null hypothesis the three mean squares

1=

1

are mutually independent. This is because ms(,, is the residual mean square computed from the regression of u, on h, and mq2, is the mean square due to regression. Under the null, h, and u, are independent and, hence, mhhis independent of Xu;. Because the regression residual mean square ms,,, is independent of m h h , it follows that ms(2,is independent of m h h . Furthermore, E{ms(l,} = Ebs,,,} = 0"". Also, the sum

(n - l)muu= ( n - 2)ms(,,

+ ms,,,

is independent of the ratio [ms(,,]-'ms,,,. Therefore, the F test of the hypothesis that C{h,, u,} = 0, given by Ff-' = [ m ~ ~ ~ , ] - ' m s ( ~ , ,

is independent of the F test of the hypothesis that

F::

= 6-1m~'mu,.

O),h 2

(1.3.38)

6-'a,,,

given by (1.3.39)

These two tests may be combined in a number of different ways to obtain a test of the composite hypothesis. One procedure is composed of two steps.

Step 1. Test at level a1 the hypothesis that the correlation between h, and u, is zero using the F test Ff-, = [ms,,,]-'msol.

I .3.

47

RATIO OF MEASUREMENT VARIANCES K N O W N

If the hypothesis of zero correlation is rejected, reject the hypothesis that P, = Py. If the hypothesis of zero correlation is accepted, proceed to Step 2. Step 2. Test at level a2 the hypothesis that 2 h-la,, against the alternative that d-la,, > ah,, using the F test Fn n - 1l - d - l m ~ l m u v . If the hypothesis that h-'a,, ,< is accepted, accept the null hypothesis that 8, = fly. Otherwise, reject the hypothesis that p1 = fly. The probability of rejecting the hypothesis P{reject pylp,}

= a1

PI = fly

+ ( I - a,)P{reject d - ' ~ , ,

when it is true is ahhlPl

= fly}.

The probability of rejecting the hypothesis that S-lo,, d ahhis a function of the true gXx.This probability achieves its maximum of a2 for a,, = 0. Therefore, a1

< P{reject pyIpl = py} < a, + (1 - a1)az.

Given a test, it is possible to construct the associated confidence set. Therefore, a confidence set for 0 , with coverage probability greater than or equal to a1 + ( 1 - al)crZ is the set of fll for which and The confidence sets constructed by the methods of this section will not always be a single interval and can be the whole real line. For composite on the real line is consistent with the data when the quahypotheses any /I1 dratic in PI defined by (1.3.39) has no real root. This is roughly equivalent to the data failing to reject the hypothesis that axxis zero. Also, it is possible that the hypothesis of zero covariance will be accepted for all PI. In our treatment we have assumed that oeu= 0. This simplifies the discussion and represents no loss of generality. One can always transform the problem to one with equal-variance uncorrelated errors. For example, if Yae is known, we define Y: = (Yee- reur&lYue)1'2( I: - X,YL1Yue),

x: = y-l/zx uu

1'

Then the error covariance matrix for (YF, X : ) is a multiple of the identity matrix. The transformed model is

Y: = 0:

+ prx, + e,,

(1.3.40)

48

A S I NGLE EX P LA N A TOR Y VARIABLE

where the hypothesized value for

rl(u1/2rue.

fl:

is (Tee- TeuTi,1Tue)1/2T,!/2P~ -

Example 1.3.3. We illustrate the methods of this section using the data of Example 1.3.2. We construct a test of the hypothesis

H,:

fll

= 0.791

against the alternative, H A : /3, 2 0 and fll # 0.791. Under the null hypothesis I: - 0.791X, has variance

[1 + (0.791)’]0.25 = 0.4064

+

and is uncorrelated with h, = 0.791Y, X,.Because we assume we know that fl, 2 0, the null hypothesis that /3, = fly is equivalent to the hypothesis that y1 = 0 in the regression equation where u, obtain

-

- O.791Xr = y o

+

ylh,

+ u,,

NI(O,O.4064).Computing the regression of I; - O.791Xron h,, we

Thus, if fll = 0.791, the probability of getting a 9, this small or smaller is 0.025. Because we know crvv under the null hypothesis, the distribution of the “t statistic” is that of a N(0, 1) random variable. We chose the value 0.791 to yield a value of -1.96 for the test statistic. It can be verified that the hypothesis H,: PI = 0.450 will yield a test statistic of 1.96. Therefore, an exact 95% confidence interval for fll is the interval (0.450,0.791). That is, all values in the interval are accepted by the test. If we use the approximate distribution theory and the estimated variance P{I,} = 0.007418 calculated in Example 1.3.3, we obtain the approximate 95% confidence interval whose end points are 0.6079 f 1.96(0.0861). This interval is (0.439,0.777), which is very close to the exact interval (0.450,0.791). In this example the approximate theory works well because o,, is small relative to mxx. on REFERENCES Adcock (1877, 1878),Amemiya (1980), Anderson (1951b, 1976, 1977, 1984),Birch (1964), Fuller (1980),Kendall(l951, 1952),Kendall and Stuart (1979),Koopmans (1937),Kummell(1879),Pearson (1901),Sprent (1966),Tintner (1945),Villegas (1961). Section 1.3.4. Creasy (1956),Fieller (1954),Gleser and Hwang (1983, Williams (1955). Sections 1.3.1-1.3.3.

I. 3 .

49

RATIO OF MEASUREMENT VARIANCES KNOWN

EXERCISES 20. (Sections 1.3.2,1.3.3)The data below are 10 pairs of observations on hectares of corn for 10 area segments as determined by aerial photography ( y) and by personal interview (Xi).

Y = (97.1,89.8,84.2,88.2,87.0,93.1,99.6,94.7, 83.4,78.5), = (96.3, 87.4,88.6,88.6,88.6,93.5,92.9,99.0,77.7,76.1).

X

Assume that the data satisfy model (l.3.l), (1.3.2)with u,, = ow”. (a) Estimate (Po, BI, uuu)and the covariance matrix of the estimators. (b) Treating x , as fixed, compute the estimates of (x,, y,). (c) Plot the data, Ihe line, and (a,,j,). Plot G, against P,. (d) Compute an estimate of the variance of 4,. 21. (Sections 1.3.2,1.3.3) (a) Verify, for o,, = 0, that the estimators (1.3.27)and (1.3.7)are algebraically equivalent. (b) Verify Equation (1.3.19). 22. (Section 1.3.2) Estimate the parameters of the model studied in Example 1.3.1 under the assumption that 6 = 3. Estimate the covariance matrix of your estimators. 23. (Section 1.3.2) Let yi = Po + Plxrr ( x f ,e,, ulY

-

NI[(P,, 0,OY, diag(u,,,

u,,)l,

gee,

and (U,,Xi) = (y,, x,) + (e,,u,). Let yyy = ueeu;yl and y x x = uuuu,l, where uyy= P ~ u x xThe . ratio y x x = uuuu;xl sometimes is called the noise-to-signal ratio of X . Give the variance expression of Theorem 1.3.1 in terms of yyy and yxx. 24. (Sections 1.3.2,1.3.3)Let PI, 0,. and x^, be defined by (1.3.27),(1.3.28),and (1.3.29). (a) Show that I i?$, = 0. (b) Show that

x;=

(c) Let A,,, d ,,”, and d,, be defined by (1.3.31).Show that

x

where is the smallest root of (1.3.27). 25. (Sections 1.3.2,1.3.3)Estimate the true x and y values for Example 1.3.1,assuming the x values to be a random sample from a normal population. Give the approximate covariance matrix of your estimators. 26. (Section 1.3.4)Construct a 95% confidence interval for the PI of Example 1.3.1 given that the parameter space is the entire real line. 27. (Section 1.3.2)Show the equivalence of the two expressions for s,, defined following (1.3.12). 28. (Sections 1.3.2, 1.3.3)Let

(F, X , ) = (Y,, x,) + (e,, u,), for 1 = I, 2,. . . ,n, where E{(e,, u,)) = (0,O).Let u, = 0 so that y, = Px,,

%)I = o,,

E{(ei, @lY(ec,

diagtI, 0).

Use (1.3.27)to obtain an estimator of /I. Show that this estimator is the ordinary least squares regression coefficient obtained by regressing U, on x,. Use formula (1.3.12)to construct an estimator of the variance of the estimator.

50

A SINGLE EXPLANATORY VARIABLE

so

29. (Section 1.3.3) Treating and as known parameters, compute the estimated variance of 3, - y , for Example 1.3.2. Why is the estimated variance of 3, - y, smaller than the estimated variance of i,- x,? 30. (Section 1.3.3) Let X, = (1, X,,),

1 ( X , X,Y(U,, XJ, (1

M

= n-l

I= 1

F, x,2 - X 2 H K -

m=(n-

where X, = n-

'

X t 2 . Let iand f be the smallest roots of IM - LEI= 0 and Im - yI,I = 0,

respectively, where E diag(1, 0, 1) and I2 is the identity matrix of dimension two. What is the relationship between I and f? 31. (Section 1.3.4) Using the methods of Section 1.3.4, construct an exact 95% confidence interval for /l, for the data of Example 1.3.1. You may assume that it is known that PI > 0. Compare the exact and approximate intervals. 32. (Section 1.3.4) Using the data of Example 1.3.1, construct a test of the hypothesis H,: (/lo, PI) = (0.10,0.95) against the alternative, HA:(/lo, PI) # (0.10,0.95), PI > 0. 33. (Section 1.3.3) Let I, > I, be the two roots of lmzz -

Arezl= 0,

where (K, X,) satisfies model (1.3.2), with Zcc= Tapz,and u2 is unknown. Show that

a,,

=

[(PI, w;l&, i ~ ~ -- i2), ~ ( i ~

where a,, is defined in (1.3.8). 34. (Sections 1.3.2, 1.3.3) Assume that the data of Exercise 1.5 satisfy model (1.3.1)with aA= 0.5. Estimate (Po, P1, uuu).Estimate the covariance matrix of the approximate distribution of (Po, P1). Plot 8, against 2,.

1.4. INSTRUMENTAL VARIABLE ESTIMATION

In Sections 1.1.2, 1.2, and 1.3, we constructed estimators of the parameters of structural models that were identified by knowledge about the error variances. In this section we consider the use of a different type of auxiliary information. Assume that the model of interest specifies

y; = P o + P I X I +

el,

XI

= x,

+

(1.4.1)

UI,

for t = 1,2,. . . , n, where the e, are independent (0, gee)random variables. The vector XI)is observed, where u, is the measurement error in X,.In addition, we have available a third variable, denoted by known to be correlated with x,. For example, we might conduct an agronomic experiment where X, is the observed nitrogen in the leaves of the plant and is the dry weight of the plant. We would expect the true nitrogen in the leaves, x,,

(x,

w,

x

I .4.

51

INSTRUMENTAL VARIABLE ESTIMATION

w,

to be correlated with nitrogen fertilizer, applied to the experimental plot. Furthermore, it is reasonable to assume that both u, and el are independent of K. We give the definition of a variable such as K. Definitiun 1.4.1. Let model (1.4.1) hold and let a variable

tt: satisfy (1.4.2) (1.4.3)

where W = n - ' model (1.4.1).

I:= w.Then

is called an instrumental variable for x, of

It is convenient to have a parametric expression for the fact that x, and are related, and we use the parameters of the population regression of x, on to quantify the relationship. Let

n I 2= E ( X

-

(1.4.5)

nz,w}.

Condition (ii) of Definition 1.4.1 is equivalent to specifying n z z # 0. The II coefficients are defined with double subscripts so that they will be consistent with the notation of higher-order models. Using the n coefficients, we can write (1.4.6) t = 1,2,. . . , n, x, = n,, + n22M: + r,,

w.

By the where r, is the failure of x, to be perfectly linearly related to regression method of construction, r, has zero correlation with q.Equation (1.4.6) and the definition of X, yield

+

XI

= n,,

E{z:=,

+ %zw + a,,,

(1.4.7)

where afz = r, u, and Pi@,) = 0. In model (1.4.1) and Definition 1.4.1, x, can be fixed, random, or a sum of fixed and random components. Likewise, the variable Pi( can be fixed, random, or a sum of random and fixed components. If M: and x, contain fixed components, nZ2and n12could properly be subscripted with n because the expected value will be a function of the fixed components of the n observaare fixed, condition (ii) of Definition 1.4.1 reduces to tions. If both x, and

c (lqn

,-I

1= 1

W)X,

# 0.

(1.4.8)

52

A SINGLE EXPLANATORY VARIABLE

One possible choice for W; is a measurement of x, obtained by an independent method. If x, is fixed and if W; is a measure of x, containing error, then & is a sum of fixed and random components. We defined an instrumental variable in terms of model (1.4.1). If it is known that Po = 0, then the conditions (i) and (ii) become

Therefore, if the mean of x, is not zero, the variable that is identically one becomes a possible instrumental variable for the model with structural line passing through the origin. The idea behind instrumental variable estimation is seen easily when all variables are normally distributed. Therefore, to introduce the estimators, we assume (1.4.9)

where px = nI2 + n2,pw and we assume oXw= 7 1 2 2 ~ w w # 0. The model specifies two zero values in the mean vector and two zero covariances in the covariance matrix of (1.4.9), but the remaining parameters are unknown. In particular, we have not assumed oeuto be known or rsXuto be zero. Under the assumptions, the observed vector X,,F)is normally distributed with mean vector (PI,,Px3 Pw) = (Po + PI7112 + P17122Pw3 7112 + 7122PWI PW) and covariance matrix

(x,

+ 2 P 1 = x e + Gee + Ox, + P 1 o x u + oeu

fl:=xx

PI&

PlDxx

+ =xe + PI=,, + 6,” + 2=xu + =”u

=xx

P17122=ww

P17122=ww 71226ww

r22uww

=ww

1 ’

(1.4.10) The model (1.4.1), (1.4.9) contains 12 independent unknown parameters. One set of parameters is

b w - n1297122, P o , P I , =ww, =xxr

=xe, =xu, =uu, Gee, ueu).

The set of minimal sufficient statistics for a sample of n observations is the mean vector and the sample covariances of (Y,, X I &).There are nine statis-

I .4.

53

INSTRUMENTAL VARIABLE ESTIMATION

tics in this set. Therefore, we cannot hope to estimate all 12 of the parameters of the model. However, we note that the ratio of covariances ~i&JYW

=(

~ 2 2 ~ W W ) - 1 8 1 ~ 2 2= ~ w81. W

P1 and Po by j 1-1 - mXWmYWI

(1.4.11)

It follows that we can estimate

(1.4.12) (1.4.13)

p, = r - j,s,

where (7, X) = n-'

c:=(x,X , ) and

(my,+,,mxw> = (n - I ) - '

C (x- Y,X,- XI(& - TI).' n

,= I

Because the sample moments are consistent estimators of the population moments, the estimators of p1 and Po will be consistent under the model assumptions. The assumption that cxW# 0 is critical. If-cr,, = 0, the denominator of pl defined in (1.4.12) is estimating zero and 8, is not a consistent estimator of PI. The limiting properties of the estimators are given in Theorem 1.4.1. The assumptions of the theorem are less restrictive than the assumption of trivariate normality used in introducing the estimator.

w)'

Theorem 1.4.1. Let model (1.4.1) hold. Let the vectors (x,, e,, u,, be independently and identically distributed with mean (px,0, 0, p,)' and finite are zero and that fourth moments. Assume that the covariances oweand cWu oxW# 0. Also assume E { ( W - P W ) 2 u : ) = owwcvu, E { $ ( & - PW)) = 0,

I"[(.

(jo, j , ) be defined by (1.4.13) and (1.4.12). Then - Po] L + + P Y 2 2 -Pxv22]) >

where Y, = e, - Plu,. Let nl,2[Bo

(1.4.14)

81 - 81

0 '

["..

-PxVzz

v,2

where V2, = cr.&WWa,u.

Proof. The error in the sample moments is 0,(n-"2) because they are, approximately, the mean of n independent, identically distributed random variables with finite variance. Using the method of statistical differentials of Appendix 1.A, we have

54

A SINGLE EXPLANATORY VARIABLE

+

c:= (w

+

+

where we have used I: = Po PIX, u, and myw = Plmxw mwu. The variance of n - l - pw)u, is n-lowwo,,by assumption. The quantity n - 1 / 2 M W converges u in distribution to a normal random variable because the random variables (4- pW)v,are independently and identically distributed with zero mean and finite variance. The limiting distribution of n'/2(j1 - pl) then follows. Writing iio

- P o = v - PAiil = n-

c

- 01)

n

,=1

[Dl

+ O,(n- l )

- P x G k ( 4- Pw)01l

+ O,(n- l )

and using E{u?(W - pW)} = 0, we obtain the result for

j0.

0

Theorem 1.4.1 covers a broad range of possibilities. In particular, it is possible for (el, u,) to be correlated with x,, and e, may be correlated with u,. The critical assumptions owe = ow" = 0 and oXw# 0 permit the estimation of PI in the presence of measurement error that is correlated with the true values. The theorem was given for random (x,, but the result holds for fixed x, and for x, that are the sums of fixed and random components, under mild conditions. Because the sample moments are consistent estimators of the population moments, a consjsteFt estimator of the covariance matrix of the approximate is distribution of (/lo,/I1)

w),

s,,

= ( n - 21-1

I =1

-

F - jl(x,- X)]'.

(1.4.16)

In most practical applications of the method of instrumental variables, one will wish to check the hypothesis that oWx# 0. A test that owx = 0, under = 0, can be constructed by testing the hypothesis the assumption that oUw that the regression coefficient, in the regression of X , on i4( is zero. While the distribution of Theorem 1.4.1 is only approximate, one can construct an exact test of the hypothesis against the alternative, H A : P1 # P?, under the stronger assumptions that (e,,u,) is normally distributed and independent of i4(. The random variable

4 = r; - PYX, = Po + (P1 - P 3 x , - P%, + e,

(1.4.17)

I .4.

55

INSTRUMENTAL VARIABLE ESTIMATION

is then independent of W , if and only if p , = py. It follows that a test of the hypothesis that the population regression coefficient of h, on r/c: is zero is a test of the hypothesis that P1 = fly. Also, the set of values of for which the hypothesis is accepted at the q level constitutes a 1 - q confidence set for pl. The confidence set is the set of py for which ( n - 2)-'tf or, equivalently, the set of

(mhhmWW- miw)-lrn&,, for which

where t, is the q point of Student's t distribution with n - 2 degrees of freedom. The boundaries of the set are the solutions to the quadratic equation Imxxmww - [ 1 - [I

+ t i 2 ( n - 2)Im:wlP:

- 2{mwwmx,

+ t i 2 ( n - 2)]mywmxw}Pl+ mYYmWW- [I + a,

-

2)]miw = 0, (1.4.19)

provided the solutions are real. The values of satisfying (1.4.18)are generally those in the closed interval with end points given by the solutions to (1.4.19). However, the set is sometimes the real line with the open interval deleted and, if the solutions to (1.4.19) are imaginary, the set is the entire real line. The condition (1.4.18) must be satisfied for the confidence set. Given that one has an instrumental variable available, one might ask if the ordinary least squares estimator obtained by regressing k; on X , is unbiased for p,. The ordinary least squares estimator is unbiased when oxy= Pigxx, and under this condition, it follows from (1.4.10)that the population regression of I: on ( X , , W,) is

From (1.4.20)a test of the hypothesis that the ordinary least squares estimator of jl1is unbiased is equivalent to a test of the hypothesis that the coefficient of in the multiple regression of I; on X , and is zero. That is, we test the hypothesis that y = 0 in the equation

w

w

I: = P o + PlX, + YK +

ot*

(1.4.21)

The analysis associated with (1.4.21) also explains why variables whose theoretical coefficients are zero are sometimes significant in an ordinary least squares regression. If theory specifies Y to be a function of x only, x is measured imperfectly by X , and W is correlated with x, then the coefficient for W in the multiple regression of Y on X and W is not zero.

56

A SINGLE EXPLANATORY VARIABLE

Example 1.4.1. To illustrate the use of an instrumental variable we study reported magnitudes of Alaskan earthquakes for the period from 1969 to 1978. The data are from the National Oceanic and Atmospheric Administration’s Hypocenter Data File (Meyers and von Hake, 1976). These data have been studied by Ganse, Amemiya, and Fuller (1983). Three measures of earthquake magnitude are the logarithm of the seismogram amplitude of 20 second surface waves, denoted by Y,, the logarithm of the seismogram amplitude of longitudinal body waves, denoted by XI,and the logarithm of maximum seismogram trace amplitude at short distance, denoted by 4. Table 1.4.1 contains the three reported magnitudes for 62 Alaskan earthquakes. These magnitudes are designed to be measures of earthquake “strength.” Strength is a function of such things as rupture length and stress drop at the fault, both of which increase with strength. A model could be formulated to specify average rupture length and stress drop for a given strength. In addition to variations in fault length and stress drop from averages given by the strength model, there is a measurement error associated with the observations. The measurement error includes errors made in determining the amplitude of ground motion arising from such things as the orientation of a limited number of observation stations to the fault plane of the earthquake. In this example the relationship between the amplitude of surface waves, I;, and the true value of body waves, x,, is of interest. The proposed model is

I; = Po + P l x r + el,

X,= x, + u,,

where x, is the true earthquake strength in terms of body waves and (el, u,) is the vector of measurement errors, it being understood that “measurement error” also includes the failure of the basic model to hold exactly for each The earthquake. We assume (e,, u,) is uncorrelated with the measurement vectors (Y,, XI,W;) are assumed to satisfy the conditions of Theorem 1.4.1. The sample mean vector for the data of Table 1.4.1 is

w.

(Y,X,

W )= (5.0823,5.2145,5.2435)

and the sample covariance matrix is

[::: mwu

myx muw mwx mxx

1

0.6198 0.2673 0.4060 0.2673 0.2121 0.2261

.

mww m x w ] = [ 0.4060 0.2261 0.4051

The test of the hypothesis that g X w = 0 is constructed by regressing XIon and testing the hypothesis that the coefficient of W; is zero. The regression coefficient is 0.5581 and the t statistic is 9.39. Therefore, we are comfortable using as an instrumental variable. By (1.4.12)and (1.4.13),the instrumental

1.4.

57

INSTRUMENTAL VARIABLE ESTIMATION

TABLE 1.4.1. Three measures of strength for 62 Alaskan earthquakes Observation t 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Surface Wave

K

Body Wave Xl

Trace

Observation

5.5 5.7 6.0 5.3 5.2 4.7 4.2 5.2 5.3 5.1 5.6 4.8 5.4 4.3 4.4 4.8 3.6 4.6 4.5 4.2 4.4 3.6 3.9 4.0 5.5 5.6 5.1 5.5 4.4 7.0 6.6

5.1 5.5 6.0 5.2 5.5 5.0 5.0 5.7 4.9 5.0 5.5 4.6 5.6 5.2 5.1 5.5 4.7 5.0 5.1 4.9 4.7 4.7 4.5 4.8 5.7 5.7 4.7 4.9 4.8 6.2 6.0

5.6 6.0 6.4 5.2 5.7 5.1 5.0 5.5 5.0 5.2 5.8 4.9 5.9 4.7 4.9 4.6 4.3 4.8 4.5 4.6 4.6 4.3 4.6 4.6 4.9 5.5 4.7 4.1 4.9 6.5 6.3

32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62

u:

Surface Wave

Body Wave Xl

Trace

5.4 5.3 5.7 4.8 6.4 4.2 5.8 4.6 4.7 5.8 5.6 4.8 6.2 6.8 6.0 4.6 4.1 4.4 4.0 5.0 5.9 5.7 5.0 5.3 5.7 4.7 4.6 4.2 6.5 4.9 4.4

5.7 5.7 5.9 5.8 5.8 4.9 5.7 4.8 5.0 5.7 5.0 5.1 5.2 5.5 5.8 4.9 4.7 4.9 5.3 5.0 5.6 5.5 4.1 5.5 5.5 4.8 4.8 4.5 6.0 4.8 5.0

5.1 5.7 5.8 5.7 5.7 4.9 5.9 4.3 5.2 5.9 5.4 5.5 5.8 6.2 5.8 4.7 4.5 4.6 5.2 5.6 5.9 5.9 5.3 5.5 5.4 4.7 4.6 4.6 7.1 4.6 5.3

I:

t

w

Source: Meyers and von Hake (1976).

variable estimator of (Po, P1) is

(bo,bl)= (-4.2829,

1.7960).

By (1.4.16) and (1.4.19, s,, = 0.3495 and the estimated covariance matrix is 1.2404

-0.2368

- 0.2368

1

0.0454

'

58

A SINGLE EXPLANATORY VARIABLE

Using P{bl},an approximate 95% confidence interval for p1is the interval (1.370,2.222). To illustrate the computation of an exact test for pi, we test the hypothesis that p1 = 1.3. We assume (el, u,)’to be normally distributed and regress the created variable

h, = Y, - 1.3X, on W;.Student’s t statistic for the hypothesis that the coefficient of is zero is t = 2.72. If we calculate the t values for several h, created with values of /.I1 near 1.37 and for several h, created with values of p1 near 2.22, we find that we obtain a t value of -2.00 for pi = 2.300 and a t value of 2.00 for p1 = 1.418.Note that the two-sided 5% point of Student’s t distribution with 60 degrees of freedom is 2.00. Therefore, an exact 95% confidence interval for p1 is (1.418, 2.300). Alternatively, one can construct the interval using (1.4.19)and (1.4.18). To test the hypothesis that the ordinary least squares regression of I: on X, would provide an unbiased estimator of PI, we compute the multiple regression of Y, on XIand K. The estimated regression line is

f

=

- 1.258 + O.474Xt+ 0.738%, (0.653) (0.195)

(0.141)

where the numbers in parentheses are the estimated standard errors. The test of the hypothesis that the coefficient of is zero is t = 5.23. Therefore, we reject the hypothesis that the ordinary least squares estimator of /.I1 is unbiased. See Exercise 1.42 for further analyses of these data. no

REFERENCES Anderson (1976), Anderson and Rubin (1949, 1950), Basmann (1960), Durbin (1954), Fuller (1977), Halperin (1961),Johnston (1972),Sargan(1958),Sargan and Mikhail (1971).

EXERCISES 35. (Section 1.4) (a) Let model (1.4.1) hold, let variable. Show that

‘f

1=1

j, be = 1=1

defined by (1.4.12),and let

w[x - F - B , ( X , - XI] = 0.

be the instrumental

I ,5.

59

FACTOR ANALYSIS

-

(b) Let model (1.4.1) hold with .Y, fixed and let a fixed instrumental variable W( be available. Let m,, = 0. If (e,, u,) NI(0, I d ) , what is the distribution of PI? 36. (Section 1.4) In Example 1.3.1 the estimated intercept was nonsignificant. Assuming the intercept to be zero, use the method of instrumental variables, with the constant function as the instrumental variable, to estimate PI of the model

Y, = Dlx, + el,

XI = x, + u,.

Estimate the variance of the approximate distribution of the instrumental variable estimator of 81. 37. (Section 1.4) Assume the model

x = Po + Pix, + el,

-

( x 1 3el, 4’

X,= xc+ u,,

NIC(P,, OY, diag(a,,,

%)I.

gee,

Let two independent identically distributed X determinations and a Y determination be made on a sample of n individuals. Consider the estimator

cr=

where XI, and XI2are the two determinations on XI, and X , i = n - ’ XIi.Give the variance of the limiting distribution of the estimator in terms of the parameters of the model. 38. (Section 1.4) Assume that only the value W,,= 6.5 is available for a new earthquake of the type studied in Example 1.4.1. Predict x63. If the vector (Y,,, x,,,w63)= (6.5, 6.1, 6.5) is available, estimate x , ~treating x6, as fixed. Compare the estimated variances of the two procedures under the added assumption that up”= 0. In computing the variances, treat all estimates as if they were parameters. (Hint: Under the assumption u,, = 0, one can estimate useand L.) 39. (Section 1.4) Estimate the parameters of the model of Exercise 1.14 using condition, where condition is the value of i, as an instrumental variable. Estimate the parameters using (i - 2)’ as an instrumental variable. Compare the estimated variance of Dl for the esimates constructed with the two different instrumental variables. 40. (Sections 1.1.2, 1.4) Let model (1.4.1) hold and let u: = x, + b,, where (e,, u,. b,) are independent identically distributed zero mean vectors with diagonal covariance matrix, finite fourth satismoments and uuu= ubb. Let (e,, u,, 6,) be independent of xj for all t and j . Show that fies the conditions for an instrumental variable. For x, satisfying the conditions of Theorem 1.4.1 show that the correlation between XIand I.t; estimates K ~ ~ .

1.5.

FACTOR ANALYSIS

The model of Section 1.4 contained a third variable and the assumption that the errors in the original ( Y , X)pair were uncorrelated with the third variable. These assumptions enabled us to construct estimators of the parameters of the equation involving x, but were not sufficient to permit estimation of all parameters. In this section we consider the model of Section 1.4 under added

60

A SINGLE EXPLANATORY VARIABLE

assumptions. Let

+ P l l X , + ell, x 2 = P o 2 + P 1 2 X r + er2, X I = Po1

x,= x, + UI,

(1.5.1)

where (XI, X2,X,)can be observed. Assume that ex, > 0, Pll# 0, and (Xp

eft, etz, d’ NI[(P~,O,O,O)’, diag(oxx, Q e e l l r ‘V

PI2 # 0,

eee22, cuu)I*

(175.2)

It is the assumption that the covariance matrix is diagonal that will enable us to estimate all parameters of the model. The model is the simplest form of the factor model used heavily in psychology and sociology. In the language of factor analysis, xris the commonfactor or latent factor in the three observed variables. The variables e,,, er2,and u, are sometimes called uniquefuctors. Note that the first subscript of Pi, identifies the x variable (factor) and that the second subscript identifies the Y variable. The two Y variables enter the model in a symmetric manner, and we choose to identify them with the same letter but different subscripts. This emphasizes the special assumptions of the factor model relative to the instrumental variable model. In fact, the three variables XI, X2,and X,enter the model in a completely symmetric way, as we illustrate in Example 1.5.2. Henceforth, we will often use the four subscript notation for population and sample moments. For example, the covariance matrix of e, = (e,,, er2)is denoted by

and the covariance between the first element of the vector Y,= (TI, the second element of e, is denoted by cYel2= C{ XI, e t 2 } . Under assumptions (151) and (1.5.2), P:lcxx

+ Dee11

PlIP12nxx

x2)and

P1lcx.x

(1.5.3) where PYl = P o 1 + P 1 1 P x ,and P Y Z = P o 2 + PltPX. The ratio of the variance associated with the common factor to the total variance of an observed variable is called the communality of the observed variable in factor analysis. Thus, the communality of k;, is ~ 4 = 1 CP:lcxx

+ ceellI-lP?lcxx

= 1 - ci*11geeii*

1.5.

61

FACTOR ANALYSIS

was called the reliability ratio in Section 1.1. The ratio uniqueness of variable E;, . Given a sample of n observations, the matrix of sample moments about the mean is Note that

nyyI - 1 I neel is called the

[

mYYll

mzz=

where Z, = ( X I ,

,I

mYXll

mYY12

myyzi

m ~ ~ 2 mrx21 2

mXYll

mXYlZ

3

mXXll

x2.X , ) , and, for example,

Since there is only one X variable, we can denote the sample variance of by m x x or by mxx11. The model contains nine independent unknown parameters. One set of pa~ , n,,, d.Using (1.53, we can rameters is (Pol, PI1, PO2,P12, n e e l l ,o , , ~ n,,,,, equate the sample moments to their expectations to obtain the estimators;

x

(ix, 801, 8 0 2 ) = (X, TI A

A

(0119 PI21

-

a,,x,yz

= (mx:12mYYI2,

(1.5.4)

-8 1 2 n

m,:I,m,Y2,),

for i = 172,

n*eel1. . = myyii - 8 ; i a x x nu,,

= mxx -

6xx

= miy112mxYllmxY12.

nxx,

The estimator of PI is the instrumental variable estimator for the relationship between and x, using as the instrumental variable. In the same way, the estimator of PI2 is the instrumental variable estimator of the relationship between q2 and x, using TI as the instrumental variable. For the # 0. The condimodel to be identified we must have n x y l l # 0 and cXyl2 tions on the two covariances correspond to the conditions required of instrumental variables. The estimators of the error variances are symmetric in the moments. For example, m~~,,m~,,,m,,,, is an estimator of that portion of the variance of XI that is associated with the common factor. The expression for 6,, is the analogous expression for X I . Alternative expressions for the estimators of the error variances are

x2

x,

GPeii= iiuuii - r(i:i6u,,,

(1.5.5)

i = 1,2,

6uu = ~ ~ 1 1 8 1 2 ~ - 1 ~ u u 1 2 ~ where iiuvij = (n - l ) - ' 6,i6,j, and 6,i = - 8 - (X, - X ) P , , . Because

c:=l

xi

-

E{4lU,2} = P 1 1 P 1 z ~ u u ,

+

E{u:} = neeii P:inuu,

i = 1, 2,

A

62

A S I NGLE EX P LA N A TOR Y VARIABLE

where uti = eti - urPli for i = 1,2, we see that the estimators of the error variances given in (k.5.5) are obtained by replacing (ql, ot2) with (8tl, 42), (Pll, P12) with pI2),and equating the resulting estimated sample moments of uIi to the expectation of the true sample moments. Under the normal distribution assumption, the estimators defined by (1.5.4) are maximum likelihood estimators adjusted for degrees of freedom, provided the solutions are in the parameter space. It is possible for some of the estimated variances of (1.5.4) to be negative. Maximum likelihood estimation for these types of samples is discussed in Section 4.3. The estimators (1.5.4) are continuous differentiable functions of the sample moments and we can use the method of statistical differentials to express the estimators as approximate linear functions of moments. By the arguments of Theorem 1.4.1 we have

(Ill,

811

/$i

- Pli = 0;:jlmYuji + ~ p ( n - ~ ) ,

- poi = vi - B(p*li- p l i )

+ OP(rP),

(1.5.6)

for j # i and i = 1,2. The vector of partial derivatives of 8,, with respect to the vector (mYXllrmyYl2,mxxI mxy12) is ( - m 2 1 2 m x y 1 2 , m;A

2mXY 1 191,

- m;A

2mxy 1 1).

If we evaluate these derivatives at the expected values of the moments we have

4"- 0," = ~ a 1 1 P 1 2 ) - 1 C ~ u u 1 2 ouu121 + beeii

qJ(n-1)9

(1.5.7)

- ceeii = muuii - PliP,lmvvij + Op(n-

for i # j and i = 1,2. The expressions for the estimated error variances can also be obtained directly from (1.5.5). The approximate expressions for the estimators enable us to construct the covariance matrix of the limiting normal distribution of the estimators. Construction of the estimated covariance matrix is illustrated in Example 1.5.1. .The approximate expressions for the estimators are informative for several reasons. First, in the limit, the estimators of the error variances, crceii, are functions only of the moments of u1 and u2. That is, the nature of the distribution of the true x vaiues has no influence on the distribution of the estimators of error variances. Second, given the independence otx, and the (et1,et2, uI), the covariance matrix of the limiting distribution of (poi,Sli)can be given an explicit expression that depends only on the second moments of the original distribution. Finally, only the distribution of 6,, depends on the higher moments of x,. This means that expressions for the limiting distributions of the estimated coefficients and for the estimated error variances can be obtained under quite mild assumptions on the x,.

1.5. FACTOR

63

ANALYSIS

An estimator of the uniqueness of variable 1 - Rii

TI is

-1

(1.5.8)

= mrriiceeii,

where R,, = m;;, ,(&,8J is the estimated reliability ratio (communality) for An estimator of the variance of the approximate distribution of R1 is

xl.

c{Rll}= m ; ~ l l ~ { r 3 e e l , }+ 2(n

-

l)-’(l - R1,)2(2R,1 - I),

(1.5.9)

where 3{6,e11 } is constructed in Example 1.5.1. It is sometimes of interest to estimate the true x, value that generated the vector X,). We can write model (1.5.1)as

(xIyx2,

[““,PO’] x 2 -Po2

=

I”:’]

4

PI2

+

[%] el2

(1.5.10)

.

Therefore, if (pol,/Io2, PI,, p12)and the covariance matrix of (e,,, er2, u,) are known, the generalized least squares estimator of x,, treating x, as fixed, is

See Exercise 1.43 and Section 1.2.3. The estimated value for x, obtained by replacing the parameters with estimators is

An alternative form for i,is i r

where mu”and

(fit,,

= Xt - ( f i t 1 7

A

-1

(1.5.13)

A

fii2)muv muu,

Gr2) are defined in (1.5.5) and h ” U

= (-bll&u9

-ij128uu)l.

See Exercise 1.43 and Chapter 4. An estimator of the variance of 2, - x, is

qXI,

-

x,) = (j:lLi&;

1

+

j:,s,:2

+

&;I)-

(1.5.14)

where the effect of estimating the parameters is ignored. The predictor of x,, treating x, as random, is given in Exercise 1.46 and is discussed in Section 4.3.

Example 1.5.1. In 1978 the U.S. Department of Agriculture conducted an experiment in which the area under specific crops was determined by three different methods. The three methods were digitized aerial photography, satellite imagery, and personal interview with the farm operator. We

64

A SINGLE EXPLANATORY VARIABLE

denote the hectares of corn determined for an area segment by the three and X,,respectively. An area segment is an area of the methods by &, earth's surface of approximately 250 hectares. Observations for a sample of 37 area segments in north-central Iowa are given in Table 1.5.1. We begin by assuming the data satisfy model (lS.l), (1.5.2). The sample mean vector is

x2,

(Fl, F2,X)= (123.28, 133.83, 120.32) and the sample covariance matrix for

m,,

=

[

(xl, x2,X I )is

1

1196.61 908.43 1108.74 908.43 1002.07 849.87 . 1108.74 849.87 1058.62

By the formulas (1.5.4) we have

[ ~ o l , ~ l l , ~ 0 2=, ~[-5.340, ,2] 1.069 35.246,0.819], (4.646,0.038, 11.738, 0.094) d,,, 8,,] = [11.47,305.74, 21.36 1037.261. [deel 8ee22, (23.10, 73.31, 20.70 250.27) The numbers in parentheses below the estimates are the estimated standard errors which will be obtained below. The estimated covariances of (utl, ut2) are

1 2 , ~ ~=~(35.8776, ) 18.7056, 320.0740), (fiuul fiuv21, where muu= (n - l)-' X).We note that

c;=l(Ctl, Cf2)'(Ctlr C,J

and Cjj =

xi - Fl - bli(X, -

8,, = 21.3557 = ( / ? ~ l / ? 1 2 ) - 1 ~ v u l ~ .

The (fiIl, Ct2) values are given in columns five and six of Table 1.5.1. The estimated values for myv21 and mYvl2are zero by the construction of the estimators. Also see Exercise 1.35. Under the normality assumption, the estimated covariance matrix of (myv21, mYvl2I muu119 ' n v v 1 2 7 muu22, mxx) is 998.663 569.442 569.442 10638.993 0 11.921 304.701 101.989 317.726 0 - 1077.782 - 1077.740

0 11.921 71.511 37.284 19.439 29.077

304.701 3 17.726 - 1077.782 101.989 0 - 1077.740 37.284 19.439 29.077 328.705 332.621 22.242 332.621 5691.520 17.014 22.242 17.014 62259.794

TABLE 1.5.1. Hectares of corn determined by three methods

Photograph Segment

XI

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37

167.14 159.04 161.06 163.49 97.12 123.02 11 1.29 132.33 116.95 89.84 84.17 88.22 161.87 106.03 87.01 159.85 209.63 122.62 93.08 120.19 I 15.74 125.45 99.96 99.55 163.09 60.30 101.98 138.40 94.70 129.50 132.74 133.55 83.37 78.51 205.98 110.07 134.36

Satellite

L

168.30 162.45 129.60 166.05 94.05 140.85 110.70 158.85 121.95 106.65 99.00 153.00 159.75 1 17.45 84.15 157.50 194.40 165.15 99.45 166.05 154.35 153.90 132.30 92.70 142.20 65.25 113.40 131.85 92.70 135.90 159.75 132.75 100.35 113.85 206.55 130.50 138.15

Interview

xt

41

165.76 162.08 152.04 161.75 96.32 114.12 100.60 127.88 116.90 87.41 88.59 88.59 165.35 r04.00 88.63 153.70 185.35 116.43 93.48 121.00 109.91 122.66 104.21 92.88 149.94 64.75 99.96 140.43 98.95 131.04 127.07 133.55 77.70 76.08 206.39 108.33 118.17

- 4.702 - 8.869

3.883

- 4.066 - 0.497

6.376 9.098 0.978 - 2.665 1.747 -5.185 - 1.135 -9.534 0.204 -2.387 0.899 16.848 3.507 - 1.502 - 3.808 3.596 - 0.322 - 6.09 1 5.610 8.158 - 3.572 0.472 - 6.367 - 5.729 - 5.230 2.254 - 3.863 5.656 2.527 - 9.292 -0.385 13.387

4

2

- 2.760

- 5.594

-30.218 - 1.724 - 20.1 15 12.101 -6.972 18.827 -9.077 -0.215 -8.831 45.169 - 10.974 -3.007 -23.714 - 3.678 7.290 34.508 - 12.388 31.664 29.050 18.154 11.671 - 18.646 - 15.898 -23.048 - 3.747 - 18.456 -23.620 -6.712 20.391 - 11.919 1.441 16.269 2.201 6.495 6.083

3, 162.76 156.42 153.93 159.17 95.65 118.34 06.18 28.83 15.07 88.50 85.18 88.69 159.18 104.07 86.71 154.20 196.05 119.25 92.32 119.18 112.69 122.79 100.60 96.06 154.77 62.10 100.19 136.11 94.93 127.64 128.85 130.91 81.27 77.96 200.60 108.21 126.67

65

66

A SINGLE EXPLANATORY VARIABLE

This matrix is calculated using the expressions of Appendix l.B and the estimated moments, including mu". It follows from expressions (1.5.6) and (1.5.7) thtt the e$imated covariance matrix of the approximate distribution of (lOPll, ~ O P I Z Z, e e i i , Zee22, a^,, O.lO~,,) is -

0.138 0.060 -4.677 0.990 4.094 1.050 - 1.093 -0.705 0.060 0.865 -4.677 - 1.093 533.680 - 114.372 -447.073 92.102 0.990 -0.705 - 114.372 5374.730 428.550 4.094 1.050 -447.073 92.102 44.713 -40.315 - 1.678 - 1.077 -9.214

- 1.678-1.077 44.713 -9.214 -40.3 15 626.376

For example, the variance of the approximate distribution of Beell is estimated with where P{m,,,,} = 71.511 and P{mUul2}= 3?8.705. We omit the portion of the covariance matrix associated with &,,/Io2) because it is a simple function of the given covaTiance matrix and of the covariance matrix of (El, E J . See the equation for (poi - poi)in (1.5.6). The estimated communality and uniqueness for the three variables are given in Table 1.5.2. For example, the estimated uniqueness of variable is

x1

1 - G l l = m;$llBeell = 0.009586. By (1.5.9), the estimated variance of the approximate distribution of rZI1 is

P{tl1}= m&lP{Beell} = 0.0003777.

+ 2(n - 1)-'(1

-R

, ~ ) ~ ( ~ G-, 1) ,

The estimated values for x,, treating x, as fixed, are 2, = 2.7124

+ o.6270K1 + o.0180Y2 + 0.3151X,,

TABLE 1.5.2. Estimated communality and uniqueness Variable x1

x2

x,

Communality

Uniqueness

Standard Error

0.99041 0.69489 0.97983

0.00959 0.30511 0.02017

0.01944 0.08584 0.02008

I .5.

67

FACTOR A N A L Y S I S

where the coefficients are defined in (13.12). The estimated variance of the error in these estimators is given in (1.5.14) and is

P{it- x,}

=

6.7283.

The estimated true values are given in the last column of Table 1.5.1. Figure 1.5.1 contains a plot of i?,, against 2,. This plot is analogous to the plot of ordinary regression residuals against Y-hat. The variability of the seems to increase with ft.To test the hypothesis of homogeneous variance, we regress i?fl on 2,. The t statistic for itin the regression containing an intercept is t = 3.64. Therefore, we are led to reject the original model that postulated identically distributed errors.

I12.5

5

,

O

~

10.0

7.5

0,I

. .

5.0

..

2.5

0

.

*

-

-2.5

- 5.0 -7.5 -10.0 50

70

90

110

130

150

170

190

ESTIMATED TRUE VALUE FIGURE 1.5.1. Plot of residuals against estimated true x values.

21

-

68

A SINGLE EXPLANATORY VARIABLE

The estimated variances of the 8-coefficients do not require normality (see Theorem 1.4.1), but the estimated variances of the error variances (heell, hee22,6"") rest heavily on the normality assumption. The assumption of homoskedastic errors is required for all variance calculations. See Exercise 3.7 for alternative variance calculations for the instrumental variable estimator, Exercise 4.20 for alternative variance calculations for the vector of estimates of this example, and Exercise 1.44 for an alternative model. 0 0 In the form (1.5.1) the systematic part of the variable X, is called the factor. Assuming that Pll # 0 and P 1 2 # 0, we could as easily have designated y,, or yt2 as the factor. In factor analysis it is common to code the common factor by fixing its variance at one and its mean at zero. That is, the model is often written Zti

=noi

+ n l i f , + Eli,

(1.5.15)

NUO, I),

f,

for i = 1, 2, 3, where Z,= (Zll, ZI2,Z I 3 )= (XI, X2,X,), and e, = (ell, er2, t4J. The coefficients nil,n12,and n l 3 are calledfuctor loadings. Under the model ( 1.5.15), Zi

N

+ &,)

NI(nb,

(1.5.1 6)

where C,, = diag(ueell,c ~a ,,,,) ~ and~ nj = ~ (njl, ~ nj2, , nj3)for j = 0, 1. It follows that the parameters of model (1515) can be estimated by g e&ii . . = mZ Z i i fi:i fioi

-I

- mZZjkmZZjimZZki,

(1.5.17)

= mi.ZljkmZZjimZZki, -

= zi,

for i = 1,2, 3 and i # j # k. The estimators of the error variances are identical to those of (1.5.4). There is a sign ambiguity in the definition of xi, associated with model (1.5.15). If we choose the sign of n 1 3to be positive, then the sign of n l l is the sign of mzZl3 and the sign of xI2is the sign of mzz23. With this choice of standardization we see that ( 2 1 1 , A12, f i 1 3 )

-1/2

= Ox,

-

A

(fill, 8129

l),

(1.5.18)

and a,, are defined in (1.5.4). Because the estimators fiij are where jll,jl2, continuous differentiable functions of the moments, their sampling properties follow by the method of statistical differentials. The covariance matrix of A i 2 , A:3) is estimated by (1.5.19)

I .5,

FACTOR

69

ANALYSIS

where

Aij = m~~jljkmZZki for i # j # k, and ?{(rnZZ12, m Z z 1 3 , mZZ23)’} is the estimated covariance matrix of the vector of three moments. The covariance matrix of A , is constructed by using E { A l i , ;tlj}

&

(4;t1iA1j)-1E{R:i,A:j}.

Sometimes the observed variables are standardized and the factor parameters estimated from the sample correlation matrix. In this case the R f j are the estimated communalities of the original variables and the estimated “error variance” for a standardized variable is the estimated uniqueness of that variable. If we know the parameters of the model, the least squares estimator off, constructed on the basis of (1.5.15) is

I.=

xllgeiil(xl

+ x 1 2 ° i i 2 ( x 2 - ‘ 0 2 ) + x 1 3 g i 1 ( x t - n03) C l + n:zG?:z + .:3G1

- nOI) &

(1.5.20)

The estimator is the generalized least squares estimator treating f, as fixed. In practice, parameters on the right side of (1.5.20)can be replaced by sample estimators. The estimator obtained in this manner is a linear function of the estimator defined in (1.5.12).

1

Example 1.52. Using the data of Example 1.5.1, the estimated coefficients of the standardized factor are

(A,,,

$12, R 1 3 ) = (34.426,26.387,32.207)-

where we have chosen A 1 3 to be positive. Using (1.5.19),we find that the estimated covariance matrix of the approximate distribution of (R, 1, A,,, A,3), calculated under the assumption of trivariate normality, is

?(al) =

[

i

16.8900 12.5856 15.2983 12.5856 18.2291 11.8308 , 15.2983 1 1.8308 15.0968

where the matrix G,, is constructed using

(Al12, iI3, i2J = (1.3046, 1.0689,0.8193)

70

A SINGLE EXPLANATORY VARIABLE

and the covariance matrix of (rnZzlz, m Z Z 1 3 , ~ z z z 3 is ) constructed using the results of Appendix l.B and the sample moments given in Example 1.5.1. Q O While the parameterization (1.5.15) is more common in factor analysis than the parameterization (1.5.1), we prefer the parameterization (1.5.1). The p parameters of (1.5.1) are analogous to regression coefficients and the R parameters of (1.5.15) are analogous to correlation coefficients. Generally, regression coefficients have proved to be much more portable from one situation to another. This is because regression coefficients are largely independent of the parameters of the distribution of x. The mean of a regression coefficient does not depend on the distribution of x and the variance of the coefficient depends only on the second moments of the x distribution. The same is true of the jl,of the factor model. The distribution of a correlation coefficient is much more dependent on the form of the x distribution. For the factor model the mean of Al is a function of the variance of x and the distribution of 12, depends on the distribution of x. Any screening of the data that changes the variance of x changes the nl.If two different samples are to be compared, comparison of the R’S typically requires the distribution of x to be the same in the two populations. On the other hand, a specification of equality for the p’s of two samples can be made with the distribution of x in the two populations left unspecified. In the models discussed to this point, the number of sample covariances has never exceeded the number of parameters to be estimated. In applications of factor analysis, it is often the case that the number of covariances is much larger than the number of parameters. Consider a model with p > 3 observed variables that are a function of a single latent factor

+

i = 1 , Z . . . ,p . Z,, = nOi+ nljf, eti, There will be p means, $ p ( p + 1) sample covariances, and 3p parameters to be estimated from them. Inspection of a covariance matrix such as (1.5.3) will lead to several possible estimators for the parameters. In such a situation the parameters are said to be oueridentijied. The application of the method of maximum likelihood will lead to unique estimators of the parameters that are not equivalent to instrumental variable estimators. Estimation for models of higher dimension is discussed in Section 4.3.

REFERENCES

Barnett (1969), Harman (1976), Joreskog (1978), Lawley (1940, 1941, 1943), Lawley and Maxwell (1971).

I .5.

71

F A C T O R ANALYSIS

EXERCISES 41. (Section 1.5) As an example of a simple factor analysis model, consider some data studied by Grubbs (1948). The data given in the table are the time of burning for fuses on projectiles as recorded by three different observers. Assume the data satisfy model (1.5.1). The choice of observer to be identified with X I is arbitrary because we have no reason to believe that one observer is superior to the others. (a) Fit model (1.5.1) to these data. Using the covariance matrix of the approximate distribution, test the hypothesis that the variance of the measurement error is the same for the three observers. (b) Write the solution in the parametric form

&=s,,+n,,f;+e,,,

xi

i=l,2,3,

where u,, = I , is the observation on the tth fuse made by the ith observer, and the measurement error. Give the covariance matrix of (sol, so,,A,,, A , , , A,2, A13).

el, is

Data for Exercise 41: Observed fuse burning times Index

Observer B

Observer A

I

XI

YZ

I 2 3 4 5 6 7 8 9

10.07 9.90 9.85 9.71 9.65 9.83 9.75 9.56 9.68 9.89 9.61 10.23 9.83 9.58 9.60 9.73 10.32 9.86 9.64 9.49 9.56 9.53 9.89 9.52 9.52 9.43 9.67 9.76 9.84

10.10 9.98 9.89 9.79 9.67 9.89 9.82 9.59 9.76 9.93 9.62 10.24 9.84 9.62 9.60 9.74 10.32 9.86 9.65 9.50 9.56 9.54 9.89 9.53 9.52 9.44 9.67 9.77 9.86

10

II 12 13 14 15 16 17 18 19 20 21 22 23 24

25

26 27 28 29 Source: Grubbs (1948).

Observer C

x,

10.07 9.90 9.86 9.70 9.65 9.83 9.79 9.59 9.72 9.92 9.64 10.24 9.86 9.63 9.65 9.74 10.34 9.86 9.65 9.50 9.55 9.54 9.88 9.51 9.53 9.45 9.67 9.78 9.86

72

A SINGLE EXPLANATORY VARIABLE

42. (Section 1.5) Assume that the data of Table 1.4.1 of Example 1.4.1 satisfy the factor model (1.5.1) with surface wave equal to yI,, trace amplitude equal to and body wave equal to XI.

xz,

(a) Estimate the parameters of the model. (b) Calculate fill, Br2, and 2,. Plot &against 2,. Plot fiIz against 2,. (c) Estimate the parameters of the model with observation number 54 deleted. Plot fill and &2 against 2, for the reduced data set. (d) Estimate the covariance matrix of the approximate distribution of the estimates obtained in part (c). 43. (Section 1.5) (a) Using the expressions given in (1.5.6) and (1.5.7), derive expression (1.5.9). (b) Using the method associated with (1.2.16), derive expression (1.5.11) for the estimated true value x,, treating x, as fixed. (c) Show that f, of(1.5.11)can also be written XI

= X I - ii,,

where oIi = eli - u,Pli and 6, = (o,,, u l z ~ ~ ~ l ~ ~..zll'. uuull,

44. (Section 1.5) Fit the one-factor model to the logarithms of the data of Example 1.5.1.

Compute the covariance matrix of your estimates. Plot O,, and OI2 against the estimated true values, all for the model in logarithms. Regress ir?,,l and (&(on 2,. What do you conclude? 45. (Section 1.5) Compute the ordinary least squares regression of X I on Crl and fill for the data of Table 1.5.1. Compare the calculated residuals for this regression with 2, of Table 1.5.1. 46. (Section 1.5) Assume that the parameters of the factor model (1.5.1) are known. Let an observation (TI, yIz, X I )be given. Treating x, as random, obtain the best predictor of x,. Show that this predictor is x', = A + [ ~ { f , ~-lPJ, -'~xx(f, where i , is defined in (1.5.1 i), y:, = X;vlZor, and

v4 =

W

l

- P X l Z }= uxx

- Y."Z"..

Show that V{f, - x,} = u, - [V{xl}]-lu~x.,, 47. (Section 1.5) Find a and b such that 1;= (I bf,, where is defined in (1.5.20) and f, is defined in (1.5.11). 48. (Section 1.5) Prove that, at most, one of the variance estimates defined by Equations (1.5.4) can be negative.

+

1.6. OTHER METHODS AND MODELS In the first five sections of this chapter we have discussed methods that have found considerable use in application. In this section we mention some other procedures that appear in the literature associated with errors in variables. We also describe two special situations in which ordinary least squares remains an appropriate procedure in the presence of measurement error.

1.6.1. Distributional Knowledge In Sections 1.1-1.5 we often assumed both the errors and the true values to be normally distributed. Geary (1942, 1943) has demonstrated that the parameters of a model containing normal measurement error can be esti-

I .6.

73

OTHER METHODS AND MODELS

mated if it is known that the distribution of x, is not normal. Let

I; = Po + Pix, + e,, 6- N W ,&J,

X,= x, + u,,

(1.6.1)

where E, = (e,, u,) is independent of x j for all t andj. Also, assume, for example, that it is known that E b , - P A 3 } z 0.

(1.6.2)

This knowledge can be interpreted as the availability of an instrumental then variable. That is, if we set l4( = (X, -

x)2,

E { ( X , - P d w - P d I = (1 - n- 1)2E{(X, - P J 3 } f 0,

and, by the properties of the normal distribution, E { Wu,}= E { we,} = 0.

Therefore, the methods of Section 1.4 can be used to construct the estimator Pl

-1

= mxWmYW,

(1.6.3)

where

In this development, the assumption that the conditional mean of

I; given

x, is linear is used in a very critical way. It is a part of the identifying

information. The distributional theory of Section 1.4 can be extended to cover estimator (1.6.3), but the estimated variance expression (1.4.15) is not appropriate for estimator (1.6.3) because the U: of this section does not satisfy assumptions (1.4.14). See Exercise 3.7 for a variance estimator. Under model (1.6.1), (1.6.2), ( y - Y ) 2and (X, - z)(y- 7) are also possible instrumental variables. See Section 2.4 for the use of multiple instrumental variables. Reiersol (1950) showed that P , of (1.6.1) is identified for any model with nonnormal x. Estimation methods for P1 using only the fact that the distribution of x is not normal are possible, but the procedures are complex and have been little used in practice. See Spiegelman (1979) and Bickel and Ritov (1985) for discussions of such estimation. 1.6.2.

The Method of Grouping

Wald (1940) suggested an estimator of P1 for model (1.6.1) constructed by dividing the observations into two groups. Let the observations ( Yl, XI), ( Y2,X2),. . . , (K, X,) constitute group one and the remaining observations

’

74

A SINGLE EX P LA N A TOR Y VARIABLE

group two. Then Wald’s estimator is (1.6.4)

Wald showed that the estimator is consistent for PI of model (1.6.1) if the grouping is independent of the errors and if lim

n+ m

infix,,, -

> 0,

(1.6.5)

where Z(o is the mean of the true x values for the ith group. If the grouping is independent of the errors, this implies that there exists a variable,

1 if element is assigned to group one otherwise,

w={0

(1.6.6)

that is independent of the errors. Therefore, with grouping independent of the errors, Wald’s method reduces to the method of instrumental variables. Wald’s method has often been interpreted incorrectly. For example, it has been suggested that the method can be applied by randomly assigning elements to two equal-sized groups. The random method of assigning elements is independent of the errors of measurement, but (1.6.5) is not satisfied because both sample means are converging to the true mean. It has also been suggested that the groups be formed by splitting the sample on the basis of the size of the observed X values. Splitting the sample on the basis of X will satisfy (1.6.5) but the group an element falls into will generally be a function of u,. Therefore, the first condition of Wald’s theorem will generally be violated when the observed X values are used to form the groups. See Exercises 1.50 and 2.19. A method related to the method of grouping involves the use of ranks. It may be that the x values are so spaced, and the distribution of the errors is such that XI < X, implies x, < x,. Given this situation, the rank of the observed X values is independent of the measurement error and highly correlated with x,. Therefore, in the presence of these strong assumptions, the rank of the X can be used as an instrumental variable. Dorff and Gurland (1961a, b) and Ware (1972) have investigated this model. 1.6.3. Measurement Error and Prediction

One often hears the statement, “If the objective is prediction, it is not necessary to adjust for measurement error.” As with all broad statements, this statement requires a considerable number of conditions to be correct. Let

k; = Po + Plx, (x,, 8,Y

-

+ e,,

+

X , = x, u,, NI[(,u,, O)‘, block diag(a,,, U ] ,

(1.6.7)

1.6.

75

OTHER METHODS AND MODELS

where (e,, u,) = e,. Let a sample of n vectors (k;, X , ) be available and let X , , be observed. Because X,)' is distributed as a bivariate normal random variable, the best linear unbiased predictor of Y,,, conditional on (Xl, X 2 , .. . , Xn+Ais

(x,

I

Y,+l

= 'yoc + YIlGXn+l,

(1.6.8)

where r n yle =

1 ( X , - ""J

1-1

1 = 1

c n

I=

1

( X ,- X ) ( K -

Y),

I:=

yIoG = 7 - yllcX, and X = n - l X,. The assumption of zero covariances between (e,, u,) and x, is not required for the optimality of the predictor. Thus, if one chooses a random element from the same distribution, the simple regression of observed k; on observed XI gives the optimal prediction of Y,+ if(& X , ) is distributed as a bivariate normal random vector. If the joint distribution is not normal, the predictor is best in the class of linear predictors. The introduction of normal measurement error will destroy the linearity of the relationship between X and Y when x is not normally distributed. If

,

Y; = B o

+ B l X , + e,,

-

PI

f

0,

and XI = x, + u,, where (e,, u,)' NI(0, diag{c,,, G,,"}) and oUu> 0, then the expected value of Y given X is a linear function of X if and only if x is normally distributed. See Lindley (1947) and Kendall and Stuart (1979, p.

438).

It is worth emphasizing that the use of least squares for prediction requires the assumption that X , , be a random selection from the same distribution that generated the X , of the estimation sample. For example, it would be improper to use the simple regression of Yon X of Example 1.2.1 to predict yield for a field in which the nitrogen was determined from twice as many locations within the field as the number used to obtain the data of Table 1.2.1.

Example 1.6.1. Assume that the fields in Example 1.2.1 were selected at random from the fields on Marshall soil. Assume further that a 12th field is selected at random from the population of fields. Assume that twice as many soil samples are selected in the field and twice as many laboratory determinations are made. Under these assumptions, the variance of the measurement error in X12 for the 12th field is inuu= 28.5. Given that the observed soil nitrogen is 90, we wish to predict yield. We treat the estimates of Example

76

A SINGLE EXPLANATORY VARIABLE

1.2.1 as if they were known parameters. Because the variance of the measurement error in XI,is &,, the covariance matrix for (Y12, XI2)is Pl0.u [ P 12 ; o : x o e e

oxx

] [

+ +aUU

=

1

87.6727 104.8818 104.8818 276.3545

It follows that, conditioning on the observed value of X12,the predicted yield for the new randomly selected field is ?12(X12)

= 97.4546

+ 0.3795(90- 70.6364) = 104.80.

Under the assumption of known covariance structure, the expected value of the error in the predictions for fields with an observed nitrogen of 90 is zero. That is, the expected value of the prediction error conditional on X I 2is zero. Note that the average of the predictions is for a fixed observed value, not for a fixed true value. The unobserved true value is a random variable. The average squared prediction error for those samples with the observed X12equal to 90 is E([Y12 - F(X12)]21X12= 90) = oyy- 0.37950,~= 47.87.

Because of the bivariate normality, the variance of the prediction error is the same for all observed values of X. Let the true nitrogen value for the field be x I 2 and consider all possible predictions of yield that can be constructed for the field under consideration (or for all fields with true nitrogen equal to xlZ).The conditionally unbiased predictor of yield, holding xI2 fixed, is FI2(xl2fixed) = 97.4546

+ 0.4232(90- 70.6364)= 105.65.

Thus, if the true value of nitrogen is held fixed, the conditionally unbiased predictor of yield is obtained by using the structural equation as the prediction equation. The average squared prediction error for those fields with x = x I 2 and the same number of soil determinations as used to obtain XI,is

E{[Y12 -

f((xI2 fixed)I2lx = xI2}= oee+ (0.4232)*(~0,,) = 53.50.

The variance of the prediction error is a function of oeeand CT,, only. The variance does not depend on the true, but unknown, x value. Given that the variance of the prediction error for the prediction conditional on X is less than the variance conditional on x, why would one choose a prediction conditional on x? The model that permits one to use the prediction conditional on X assumes that x (the field) is selected at random from the population of fields. If this is not a reasonable assumption, the prediction constructed for fixed x should be used. As an example, assume that a farmer asks that his field be tested and corn yield be predicted for that field.

1.6. OTHER METHODS AND MODELS

77

Is it reasonable to treat the field as randomly selected? Assume further that the farmer is known to sell a soil additive product “Magic Grow Sand” that is 100% silica. Is it reasonable to treat the field as randomly selected? To investigate the random and fixed specification further, consider the problem of making statements about the true nitrogen x, on the basis of the observed nitrogen X,. If the sampled field is randomly chosen from the population of fields, it is reasonable to assume that the observed nitrogen and the true nitrogen are distributed as a bivariate normal. If we treat the estimates of Example 1.2.1 as the parameters, we have

I>

304.8545 247.8545 247.8545 247.8545

’

Therefore, the best predictor of true nitrogen conditioning on the observed nitrogen value is ?,(XI) = 70.6364 + O.81303(Xr- 70.6364).

Under bivariate normality, this predictor is conditionally unbiased, E{Z,(X,) - x , I X , = t} = o for all

5.

However, when we look at the conditional distribution of Z,(X,) holding

x fixed, we have

E{ x’,(X,)- X, IX,

= W ) = - 0.18697(0 - 70.6364).

Also,

E{[&(X,) - x , ] ~ ) x ,= W}= 37.6780

+ [0.18697(0 - 70.6364)]’.

It follows that the mean square error of x’,(X,)as an estimator for x, is greater than the mean square error of X, for Ix, - 70.63641 > 23.5101. Because the distribution of x, has a standard deviation of 15.7434, the mean square error of Z,(X,) as an estimator of x, is less than the mean square error of XIfor those fields whose true x is less than 1.493 standard deviations from the mean. See Exercise 1.5 1. no

In Example 1.6.1 we illustrated the construction of a predictor in a situation where the nature of the population of observables was changed by a change in the measuring procedure. In practice one may also find that the distribution of x in the prediction population differs from that of the estimation sample.

Example 1.6.2. To illustrate prediction for a second population, consider the earthquake data introduced in Example 1.4.1. In the National Oceanic and Atmospheric Administration’s Hypocenter Data File, values for surface

78

A SINGLE EXPLANATORY VARIABLE

waves (Y) are generally available for large earthquakes but not for small ones, while values for longitudinal body waves (X) are generally available for small earthquakes but not for large ones. The group of earthquakes for which the XI, y ) is available is a relatively small subset of the total. In the triplet data file there are 5078 Alaskan earthquakes with only X values reported. It is of interest to construct Y values for these earthquakes. The sample mean and variance for the group of earthquakes with only X values is

(x,

(X(2),a*,,(,))

= (4.5347, 0.2224),

while the corresponding vector for the data of Example 1.4.1 is (x(l), dxx(l))= (5.2145,0.2121).

Clearly, the mean of X differs in the two populations from which the two samples were chosen. Let us assume that the model Xi

= Po

+ BlXri + eti,

Xti = xri

+ Uri,

(1.6.9)

Wi = Y O + YlXti + Cri, > diag(axx(i),see, c u u , ~ c c ) ] , (xrij &ti)' N l [ ( ~ x ( i ) 01'9 holds for i = 1,2, where eri= (eri,uri, cri),i = 1 denotes the population from which the triplets were selected, and i = 2 denotes the population for which only X values are available. While the observations suggest that the X distribution is not exactly normal, it seems that normality remains a good working approximation. Model (1.6.9) is the factor model in one factor and contains stronger assumptions about the error covariance matrix than the instrumental variable model used in Example 1.4.1. Under the factor model, we can estimate the covariance matrix of (er,u,, cl). By Equations (1.5.6), the estimates are

(a*,,

8,,,

= (0.1398,0.0632,0.0617).

The estimates of the variances of x in the two populations are dxx(l)= dxx(1)- a*,, = 0.1489, dXx(2)= dxx(2) - 8," = 0.1592.

From Example 1.4.1, the estimated structural equation is =

-4.2829

+ 1.7960~~.

It follows that estimates of the remaining parameters of the (Y, X) distribution for population two are fiy(2)

= -4.2829

+ 1.796ox(2)= 3.8614,

dYy(,,= (1.7960)2(0.1592)+ 0.1398 = 0.6533, a*,,(,) = (1.7960)(0.1592) = 0.2859.

I .6.

OTHER METHODS AND MODELS

79

Let us assume that we are asked to predict the Y value for an earthquake with an X value of 3.9 selected from population two. On the basis of our model E { J;2) IX(2)) = PY(2) + 4 ( 2 ) 0 X Y ( 2 ) ( X ( 2 )- P X ( 2 ) )

( 1.6.10)

and

(1.6.11) Our estimators of these quantities are

B{ v2)1 X ( 2 ) }= 3.8614 + 1.2855(x,,, - 4.5347), V { y2,1 X ( 2 ) }= 0.2858.

Therefore, the predictor of Y for an observation in the second population with X = 3.9 is

P = 3.8614 + 1.2855(-0.6347) = 3.0455,

and the estimated standard deviation of the prediction error is 0.5346. One could modify the estimated prediction standard deviation by recognizing the contribution to the error of estimating the parameters. See Ganse, Amemiya, and Fuller (1983). Figure 1.6.1 illustrates the nature of prediction for the second population. The two clusters of points represent the two populations. The solid line represents the structural line common to the two populations. The two dashed lines are the least squares lines for the two populations. For our example the two clusters of points would overlap, but we have separated them in the figure for illustrative purposes. The point X, represents the value of X for which we wish to predict Y in the second population. The prediction is denoted by f,. The predictor computed by evaluating the least squares line for the first population at X, is denoted by Also see Exercise 1.52. on

t.

1.6.4.

Fixed Observed X

We have discussed two forms of the model

where e, = (e,, ut). In the first form, the x, are treated as random variables, often assumed to be normally distributed. In the second form of the model, the x, are treated as fixed constants. There is a third experimental situation that leads to a model that appears very similar to (1.6.12),but for which the stochastic behavior is markedly different. Assume that we are conducting an experiment on the quality of cement being created in a continuous mixing operation. Assume that quality is a function of the amount of water used in

80

A SINGLE EXPLANATORY VARIABLE

I

xp

I

1

PX(2)

PX(I)

X

FIGURE 1.6.1. Prediction in a second population.

the mixture. We can set the reading on the dial of a water valve controlling water entering the mixture. However, because of random fluctuations in water pressure, the amount of water actually delivered per unit of time is not that set on the dial. The true amount of water x, is equal to the amount set on the water dial XIplus a random error u,. If the dial has been calibrated properly, the average of the u, is zero. Thus, x, = XI- u,,

(1.6.13)

where the u, are (0, cUu)random variables. We used the negative sign on the right side of (1.6.13) so that the form of (1.6.12) is retained. It is important that in this experiment the observed XI,the reading on the dial, is Jixed. The observed value is controlled by the experimenter. Also, if it is assumed that variations in pressure are independent of the valve setting, the u, are In the model of earlier sections, x, and u, are independent, independent of XI. while X, and u, are correlated. In the cement experiment, x, and u, are

1.6. OTHER METHODS A N D MODELS

81

correlated, while X, and u, are independent. Berkson (1950) observed that when X,is controlled, ordinary least squares can be used to estimate the parameters of the line.

Theorem 1.6.1.

Let

x, = X , - u,, k; = Po + Pix, + e,, where (e,, ul), t = 1,2,, . . , n, are independent vectors with zero mean and

covariance matrix

E{(e,,uJ‘(e,, ut)} = diadoee, o,,), and X’= (Xl, X,, . . . , Xn)is a vector of fixed constants. Let -1

so,

n

(1.6.14)

,= 1

=

B - j,,x,

be the ordinary least squares estimators of Po and /I1. Then

N P o c , i j l d = (Po, P I )

and

where Axx = (n - l)mxx, u, = e, - u,Pl, and o,, = c,,

+ B?oUU.

Proof. Substituting the definition of x, into the equation for k;, we have

I: = P o + PIX, + 4,

(1.6.16)

where u, = e, - u,Pl. By the assumptions, the vector (e,, u,) is independent of X,. Therefore, u, is independent of X,. Substituting (1.6.16) into (1.6.14), we obtain BOl

- Po =

-

RSl, - P l ) ,

The mean and variance results then follow because the X, are fixed. Because (1.6.16) has the form of the classical fixed-X regression model, all the usual estimators of variance are appropriate. The residual mean square S”,

= (n - 2 ) 4

1= 1

[I:- F - ( X , - X ) f i 1 f ] 2

82

A SINGLE EXPLANATORY VARIABLE

is an unbiased estimator of ouXa n t replacing o,, of (1.6.15) with s,, produces an unbiased estimator of V{(Po,, P,,)'}. Theorem 1.6.1 and the discussion to this point suggest that errors of measurement associated with the controlled variable in an experiment can be ignored. This is not so. It is only estimation for the simplest model that remains unaffected. Berkson (1950)pointed out that tests in replicated experiments will be biased if the same error of measurement holds for several replicates. For example, assume that a fertilizer experiment is being conducted and consider the following two experimental procedures: A. Fertilizer is applied with a spreader. A rate is set on the spreader and fertilizer is applied to every plot randomly selected to receive that rate. The spreader is then set for the next experimental rate and the process is repeated. B. The fertilizer amounts are weighted separately for each plot. The amount shown by the scale is the same for all plots receiving the same treatment. The scale is calibrated with a standardized weight between each weighing. The quantity assigned to a particular plot is scattered evenly over the plot.

With procedure A, an error made in setting the spreader will be the same for all plots treated for that setting. We can write x,i =

x,- u, - a,,,

where u, is the error in the rate applied that is common for all plots treated with a single setting of the spreader and afi is the additional error arising from the fact that the material does not always feed through the spreader at the same rate. If the response Y is a linear function of the true application rate x,, we have Xi

+ eri = P o + P I X , + e,i - P l ~ r Plari= PO + P l x r

Because a part of the measurement error, u,, is common to the tth treatment, the treatment mean of the response variable is

z*

= Po

+ P I X , + e;. - P I % - P I G .

and the expected value of the within-treatment mean square is

where o,, is the variance of a,,, we assume e,,, u,, and a,, to be independent,

I .6.

83

OTHER METHODS AND MODELS

and we assume each of n treatements is observed on r plots in a completely randomized design. The expected value of the residual mean square obtained in the regression of treatment means on X, is bod

-

Therefore, with procedure A the estimator of PI is unbiased, but the usual within-treatment estimator of variance is a biased estimator of the true error variance. With procedure B, the setting for a plot (the weighing operation) is repeated for each plot, If the calibration produces unbiased readings, we can write

x,i = x, + Uti,

where one can reasonably assume the uti to be uncorrelated. Then

Ti = P o + PIX, + eti -

utiP1,

and the within-treatment mean square will be an unbiased estimator of 6””= o e e

+ a:guu.

Presence of measurement error in the experimental levels of a controlIed experiment produces complications if the response is not linear. To investigate the nature of this problem, assume that

Y; = Bo x, = (et,

+ P I X , + PZXT + e,,

x,- u,,

( 1.6.17)

4’ NI[O, diag(cee, ~ ~ u u ) ] , ‘V

where (e,, u,) is independent of X i for all t and j . Then, using x, = X, - u,, we obtain E { Y ; I X J = (Bo + P2fJ”U)

+ PIX, + Pa:.

(1.6.18)

If one estimates the quadratic function by ordinary least squares, one obtains unbiased estimators for p1 and pz, but the estimator of the intercept is an Also, the conditional variance of Y; given unbiased estimator of Po P2cUu. X,is the variance of

+

e, - P l U , - 2PZXtUt + P2(uT - %u), which is a function of X,. Therefore, ordinary least squares will not be the most efficient estimation procedure and the ordiyry p i p a t e s of the variance of the ordinary least squares estimators of (Po, PI, B2) will be biased. The effects of measurement errors in the controlled variable of experiments with nonlinear response have been discussed by Box (1961).

84

A SINGLE EXPLANATORY VARIABLE

REFERENCES Bickel and Ritov (1985), Geary (1942,1943), Kendall and Stuart (1979), Madansky (1959), Malinvaud (1970), Neyman and Scott (1951), Reiersol(1950), Spiegelman (1979).

Section 1.6.1. Section 1.6.2. (1972). Section 1.6.3. Section 1.6.4.

DortT and Gurland (1961a, 1961b), Pakes (1982), Wald (1940), Ware Ganse, Amemiya, and Fuller (1983), Lindley (1947). Berkson (1950), Box (1961), Draper and Beggs (1971).

EXERCISES 49. (Section 1.6.1) Assume that model (1.6.1) holds and that x, is distributed as an exponential random variable. That is, the density of xt is e-x, x 2 0.Give the variance of the approximate distribution of the estimator defined in (1.6.3). 50. (Section 1.6.2) Let the normal distribution model (1.2.1) of Section 1.2.1 hold. Let n be even and let r = 4 2 . Find the probability limit of Wald’s estimator (1.6.4) if group one is composed of those observations with the r smallest X values. 51. (Section 1.6.3) Assume that a 13th field is randomly selected from the population of fields of Example 1.2.1. Assume that only one-half as many soil sites and chemical determinations are made for the field. It follows that the measurement error associated with the observed soil nitrogen for the 13th field is 20: = 114.Assume that observed soil nitrogen is X I , = 60. Treating the estimates of Example 1.2.1 as parameters: (a) Estimate true nitrogen treating x I 3as fixed. Give the variance of your estimation error. (b) Estimate true nitrogen conditioning on X13and treating x I 3as random. Give the variance of your prediction error. (c) Predict observed yield conditioning on X I 3and treating x I 3as random. Give the variance of your prediction error. 52. (Section 1.6.3) Assume that the true parameters of a population of soil nitrogen values and yields such as that of Example 1.2.1 are IT,, = 200, px = 80, /lo = 50, PI = 0.5, and u,, = 60. Assume that a population of samples of X values with measurement error uuu= 100 is created by repeated sampling from a given field with true nitrogen equal to ~ 1 3 . (a) Find the mean and variance of the prediction error in the predicted value of true nitrogen constructed as f W 1 3)

= P*

+ ui;uxx(xl3 - A).

Compare the mean square error of this predictor to that of the unbiased estimator X I S . For what values of x , does ~ the predictor have smaller mean square error? (b) Assuming that ( Y , , , X,,)is observed, find the best predictor of x I 3conditional on (YI3, XIJ. Find the best estimator of x 1 3treating x I 3as fixed. For fixed x 1 3find the mean and variance of the error in the predictor of xIJthat conditions on (YI3, X13). 53. (Section 1.6.3) Let (X, x) be distributed as a bivariate normal random vector with mean (p,, /I,), u x x= uzx+ a,,, and uxr = uxx.Show that .?(X), defined in Example 1.6.1, has a smaller mean square error for x than that of X for Ix - /I# < a2axx,where a’ =

- u:,).

APPENDIX 1.A.

85

LARGE SAMPLE APPROXIMATIONS

APPENDIX LA. LARGE SAMPLE APPROXIMATIONS In this appendix we state a theorem that is used in numerous places throughout the text.

Theorem 1.A.1. Let g(a) be a real valued continuous function of the kdimensional vector a for a an element of k-dimensional Euclidean space. Let g(a) have continuous first derivatives at the point p = (pl, p z , . . , ,pk)'. Let the vector random variables X,= ( X , , ,X2,, . . . , X,J, t = 1,2,. . . , be inde... pendently and identically distributed- with mean p and covariance matrix Z x x . Let

x = ,-I

c x,. n

1=

1

(1.A-1)

converges in distribution to a normal random variable with mean zero and variance [g'l'(P), 9'2'(co, * . * 9 g'k'(P)l&x[g'l'(B), g'z'(P), '

' '

9

g'k'(P)l'9 (1.A.2)

where &)(p) is the partial derivative of g(a) with respect to aievaluated at a = p.

Proof. Because the derivatives are continuous at p, there exists a closed ball B with p as an interior point such that the derivatives are continuous on B. The sample mean vector X is converging to p in probability. Therefore, given E > 0, there is an N such that

P{X E B } > 1 - +&

(1.A.3)

for n > N . For X E B, by Taylor's theorem, = goC)

+ CI g"'(x*)(xi - pi), k

i=

where g'i'(x*) is the derivative of g(a) with respect to ai evaluated at x* and x* is on the line segment joining X and p. Because X is converging in probability to p and because g")(a) is continuous on B, g"'(x*) is converging in probability to g"'(p). It follows that, for % E B, i=I

[g(')(x*)

- g ( i ) ( p ) ] ( Xi pi)

86

A SINGLE EXPLANATORY VARIABLE

and that

i

plim n112 g(X) - g(p) -

k

C g“)(p)(Xi i= 1

-

I

p i ) = 0.

Therefore, the limiting distribution of nl’’[g(X) - g(p)] is the limiting distribution of

where = Cf= g(i)(p)(Xti - pi). See, for example, Fuller (1976, p. 193). The 4 are independently and identically distributed with mean zero and variance given in (1.A.2). Therefore, the conclusion follows by the Lindeberg central limit theorem. 0 The theorem extends immediately to vector valued functions. Corollary 1.A.1.

Let g(a) = [g,(a), gz(a), . *

* 9

g,(a>l’

be a vector valued function, where gi(a) are real valued functions satisfying the assumptions of Theorem l.A.l. Let X be as defined in Theorem l.A.l. Then n”2[g(x) - g(p)]

5 “0,

G%m,

where the 0th element of G is the derivative of gi(a) with respect to uj evaluated at a = I(.

Proof. Omitted.

0

If the function has continuous second derivatives, it is possible to evaluate the order of the remainder in the approximation. Corollary 1.A.2. Let the assumptions of Theorem 1.A.1 hold. In addition, assume that g(a) has continuous second derivatives at the point p. Then, given E > 0, there is an N and an M, such that

for all n > N.

APPENDIX I .A.

LARGE SAMPLE APPROXIMATIONS

87

Proof. Let B be the compact bail defined in Theorem 1.A.1. Then for of g(R) about the point p, with remainder, is

X E B, the first-order Taylor expansion

(1.A.4)

where g(ij)(x*) is the second partial derivative of g(a) with respect to a, and aj evaluated at a = x*, and x* is on the line segment joining p and X. By the continuity of the derivatives on B, there is an M , such that )g‘’j)(a)lc MI for a E B. Furthermore, there exists an M , , such that

P { ( ( X ,- pi)(Xj- pj)I > n-‘M2,}< +& for all n. Let M , ( 1 .AS).

=

(1.A.5)

M , M 2 , . The conclusion follows from (l.A.3),(l.A.4), and 0

The application of Theorem l.A.l is sometimes called the method of statistical differentials or the delta method. These terms are appropriate because the mean and variance expressions are obtained by expanding the function g(x) in a first-order Taylor series. As an example of the use of Theorem 1.A.1, we consider the function g(a) = u - l

and let R be the mean of n normal independent (p,a’) random variables, where p # 0. One method of expressing the fact that the distribution of is centered about the true value with a standard error that is proportional to n - ” , is to write

x

x -p =

op(n-l’2).

Formally, a sequence of random variables { Wn}is O,(a,) if, for every E > 0, there exists a positive real number M E such that P{JW,I> Mean} 6 for all n. See Fuller (1976, chap. 5). If we expand g(2) in a first-order Taylor series about the point p, we obtain g ( X ) = g ( p ) - p - 2 ( X - p)

+ OP(K1).

Ignoring the remainder, we have g(XE;)= g(p) - P- 2(f - PI.

88

A S I N G L E E X P L A N A T O R Y VARIABLE

The mean and variance of the right side of this expression are g(p) and n - 1p-4u2,respectively, which agrees with the theorem. It is not formally correct to say that p-402is the limiting variance of n”2Cg(8) - g(p)]. One should say that p-402 is the variance of the limiting distribution of n1/2[g(X) - g(p)]. Note that the theorem is not applicable if p = 0 because the function a - l is not continuous at a = 0. Very few restrictions are placed on the vector random variable X of Theorem l.A.l. For example, X, might be a function of X,.Consider the function g(a1, a,) = a;

1’2Ul.

Let XIbe a normal (0,a’) random variable and let X, = X:. Then we have

(

g(X1,X2)= n - I

,:1, x:, x )-I/,

= o-lxl

1

- X;”2X

1

+qfl-’),

because g(pl, p 2 ) = g(,I(pl, p,) = 0. We conclude that n’/’g(X,, 8,)is approximately distributed as a N(0, 1) random variable. For a function with g(‘)(p)= 0 for i = 1,2,. . . , k, the differencen’12[g(X) g(p)] converges in distribution to the constant zero. As an example, consider the function g(a) = a2

and let

X

be distributed as a normal (0, n - ’ ) random variable. Then n1’2(P

- 0)

converges in probability and, hence, in distribution to the constant zero.

APPENDIX l.B. MOMENTS OF THE NORMAL DISTRIBUTION

Let Y,= (XI, X,, . . . , Xk)be distributed as a multivariate normal with mean zero and covariance matrix X, where X has typical element oij. Given a random sample of vectors Y,, t = 1 , 2 , . . . , n, let

Y=n-’

c Y,, n

t=1

sij = (n - l)-’

11 ( y ,- E)( q, - T), n

I=

i = 1,2, . . . , k; j = 1,2, . . . , k.

APPENDIX l.C.

CENTRAL LIMIT THEOREMS FOR SAMPLE MOMENTS

89

It is well known that P is independent of all sij and that all odd moments of are zero. See, for example, Anderson (1958). Also,

for all i, j , k, m. If we let X, = (Xtl,X t z , . . . , X r k )be distributed as a multivariate normal with mean = (pi,p2,. . . , pk) and covariance matrix E,then C(Xtixtj,X

d , r )

= bircjq

+ (Jiqbjr + P i P q c j r + P i P r u j q + P j P q c i r + P j W i q .

APPENDIX l.C. CENTRAL LIMIT THEOREMS FOR SAMPLE MOMENTS In this appendix we give the limiting distribution for the properly normalized sample mean and covariance matrix constructed from a random sample of n observations. Two situations are considered. In the first, the vector of observations is a random vector. In the second, the vector of observations is a sum of a fixed vector and a random vector. See Appendix 4.A for the definition of vech used in Theorem 1.C.1.

Theorem 1.C.1. Let {Z,} be a sequence of independent identically distributed p-dimensional random vectors with mean p,, covariance matrix Czz, and finite fourth moments. Let be the sample mean, m , , the sample covariance matrix,

z

vech mzz = (mzzll, mzz21,. . . , mzzpl, mzz22,.. . m z z P 2 , .. . mzzPpY, vech x z z = (azz11,azz21,. ' ' rJZZp19 a22229 * * * azzpz, . f l Z Z P P ) l ' al = (2, - pZ,[ v e c W , - az)'(z,- aZ)- L41')7 9

7

'

' 3

and 0 = E{a,a;}.Then

nl"[l(Z - p,), (vech m,, - vech Z,,)']'

L

-+

N(0, a).

90

A SINGLE EXPLANATORY VARIABLE

Proof. We have mzz = (n - 1 I - I

[,IlC

1

(Z, - pzY(Zl - az) + n(Z - pZY@ - pz) , W . 1 )

where n‘I2(Z - pz)’(Z - pz) is converging to zero in probability. Therefore, the limiting distribution of n’/2(vech mzz - vech Ezz) is the same as that of n’I2(n - 1)-’

1=1

vech{(Z, - pz)’(Z,- pZ) - Zzz}.

For any arbitrary fixed row vector d, where dd’ # 0, the sequence {da,} is a sequence of independent identically distributed random variables with mean zero and variance dad’. The conclusion follows by the Lindeberg central limit theorem.

Corollary l.C.1. Let the Z,of Theorem 1.C.1 be normally distributed. Then fi = block diag(Xzz, fi22), where the element of fi22associated with rnzzij and mZZklis aZZikaZZjl

+ aZZiluZZjk*

Proof. For normal random variables Z and m, are independent. See, for example, Kendall and Stuart (1977, Vol. 1, p. 384). The covariances of the sample moments from a normal distribution are given in Appendix l.B. In Theorem 1.C.2 we give the limiting distribution of the vector composed of the means, mean squares, and mean products of variables that are the sum of fixed and random components. As a preliminary, we present two central limit theorems for weighted sums of random variables. Lemma l.C.l. Let e, be a sequence of independently and identically distributed (0, Zee)p-dimensional random row vectors, where Eeeis nonsingular. Let {c,} be a sequence of fixed p-dimensional row vectors with clc; # 0 and

lim

n-tm

Then

n

n-l

C c,CeecI= A > 0.

1=1

APPENDIX I.C.

91

CENTRAL LIMIT THEOREMS FOR SAMPLE MOMENTS

Proof. The random variables g r = cfei are independent with zero means and variances, E{g:)

=GeeC:.

We have

where F(e) is the distribution function of the vector e,

R , , = {e:(c,e')2> t 2 V n ) ,

and Ic,12 = c,ci. The ratio V,' assumption,

c:=l

Ic,J2 is

bounded and, by the limit

Because the random variables let[ are independently and identically distributed with finite second moment,

Therefore, the array { V - 1 / 2 g 1 }satisfies the conditions of the Lindeberg n central limit theorem. O

Lemma 1.C.2. Let e, be a sequence of independently distributed (0, Z,) p-dimensional random row vectors with uniformly bounded 2 6 (6 > 0) moments. Let { c f }be a sequence of fixed p-dimensional row vectors with c,c', # 0. Let n - ' X , where

+

be bounded above and below by positive real numbers for all n, assume lim n-' sup ]c,l'= 0 l s f s n

n-m

and assume

c n

n-'

lCf12

92

A SINGLE EXPLANATORY VARIABLE

Proof. We have

1c,I2, R , , = {e: (c,e’)2> <’V,}, R,, = {e: [el2> where dnS= <-2V,1 dn;I}, F,(e) is the distribution function of el, and K is the uniform bound for E { 1ef12+’}. By the assumptions

and the conditions of the Lindeberg central limit theorem are satisfied.

0

The limiting distribution for sample means, and mean products of random vectors with fixed components and identically distributed measurement errors, is given in Theorem 1.C.2.

+

Theorem 1.C.2. Let Z, = z, e,, where the el are independent identically distributed p-dimensional random row vectors with zero mean vector, positive definite covariance matrix C,,, and finite fourth moments. Let {z,} be a fixed sequence satisfying lim Z = pz,

n-tm

lim m,, = lim (n -

n-m

n

t)-l

C (z, - z)’(z, - 2 ) = mzz.

,=1

n-tm

+

Let b = [Z, (vech mZ,,’]’ and 8, = [Z, (vech mzz vech E,,)’]’,where vech is defined in Definition 4.A.2 of Appendix 4.A. Then

~,,?(d

-

en)5 N(O, I),

where the elements of G , are the covariances of the elements of

8,

APPENDIX I.C.

CENTRAL LIMIT THEOREMS FOR SAMPLE MOMENTS

93

Proof, We first derive the elements of G , associated with the higher moments. We have

where

=

n-'rijk.

Also C { m z z i j ,mZZk/} is given by

= 0,

94

A SINGLE EXPLANATORY VARIABLE

To establish the limiting normality of the vector of estimators, consider the linear combination n1/2fit((j

- 0,) = n- 1 / 2

r= 1

gr

+ op(n-

'19

where g1 = ([Z,

- Z], [vech((Z, - Z)'(Z, - Z) - M,,

- ZEe)]')6

and 6 is an arbitrary vector such that 6'6 # 0. Using

- Zi)(Z,j - zj, - rF&Eij + (Zrj - Z;.))Efi + E,jE,i - cerij,

(Z,i - Fi)(Ztj- Tj) - (z,: = (2,i

-

we can write

where bri is a linear function of function of 8. If we let C, = (b,,, b r 2 9

er =

ErZ,

.

* *

* 7 7

(2,

- Tr), r = 1, 2, . . . ,p , and d,, is a linear

brp, d t l l , d r 1 2 r 2

Erp,

*

- oeell, ~

.

* 9

dtpp),

-~

r 1 ~ 1 2

.

E E ~ Z ,*

2 Erp

- ~Ecpp),

the c, and e, satisfy the conditions of Lemma l.C.l. The limiting normality is established because 6 is arbitrary. 0 Corollary 1.C.2. Let the assumptions of Theorem 1.C.1 hold and, in addition, assume the e, to be normally distributed. Then G, = block diag(Ze,, G22n), where the elements of G22nare

c{%Zij,

m.Uk!}

l ) - '(mlziku&&jd + mrziGcwjk + mzzjkce&il + mazj!g&&ik + ue&ik(JEEj! + beeiduex]k)*

= (n -

Proof. The result follows from the moment properties of the normal 0 distribution.

The limiting normal distribution for a vector of sample moments in which the measurement errors are not identically distributed, but possess finite 4 + S (6 > 0) moments, is given in Theorem 1.C.3. Theorem 1.C.3. Let Z, = z, + e,, where the e, are independent p-dimensional random row vectors with zero means, positive definite covariance

APPENDIX I.D.

95

NOTES ON NOTATION

matrices ?&,, and bounded 4 and let

+ 6 (6 > 0) moments. Let z, be a fixed sequence

lim Z = p,,

n+

Iim n-'

n-+m

where

zE,is

1.C.2. Then

lim m,,

n-t w

w

n

f=1

positive definite. Let

G,

1z 6

=

= m,,,

E,,,

and 0" be as defined in Theorem

"'(6 - 0,) A N(0, I),

where Gn= V{6}.

Proof. The proof parallels that of Theorem 1.C.2, with Lemma 1.C.2 used to establish normality. 0 APPENDIX 1.D. NOTES ON NOTATION In this section we summarize, for reference, some of the notation that is used throughout the book. Because the treatment of measurement errors is a topic in many areas of statistics and in many areas of application, there is no standard notation at the present time. Therefore, any notation chosen for book length treatment will, at some point, be in conflict with the reader's previous experience. We reserve capital I: and X,to denote observable random variables. If these letters are boldface, Y, and X,, they denote row vectors. Often Y,and X, are combined into a single row vector Z, = (Y,, X,). Lowercase letters y,, x,, y,, x,, and z, are reserved for the true values of q, X , , Y,, X,, and Z,, respectively. The variables denoted by lowercase letters may be either fixed or random, the nature of the variables being specified by the model. We desired a notation in which the observed and true values could be matched easily, and chose lowercase and capital letters as the simplest and most direct of the alternatives. The observed values, true values, and measurement errors are defined by

K = y , + e,, X,= x, + u,, z, = ( X , XI) = z, + e,, where e, = (e,, u,) is the row vector of measurement errors. Measurement error models are related to the regression model and we chose a notation with close ties to regression. To permit us to write models

96

A SINGLE E X P L A N A T O R Y VARIABLE

in the regression form

(1.D.I) I; = X t B + e,, to be a row vector and B to be a column vector. This causes

we define x, our notation to differ from that often used for multivariate models. When the regression model with an error in the equation is being considered, the error in the equation is denoted by q, and the pure measurement error in Y; by w,. In such models e, = w, + q, and

+ 4,.

Yr = xrB

To be consistent with the linear model, we define the vector of parameters to be a column vector identified with a Greek symbol. For example,

B=

(Po9

PI)’.

For models of the type (l.D.1) we reserve the letter u, for the deviation u, =

I; - X,B = e, - u,B.

If Y, is a row vector, there is a row vector of deviations v, = Y,

- X,fl = e, - u,B.

We shall follow the common statistical practice and use the same notation for a random variable and for a realization of that random variable, with a few exceptions. We use py to denote the mean of the random variable and C, to denote : , a,,, and V{Y;} the mean of the random vector Y,. The three symbols a will be used to denote the variance of the random variable Y. The bold V{X’} or Z x x will denote the covariance matrix of the column X’. The ijth element of X x x is a x x i jThe . covariance of the scalar random variables X and Y is denoted by ax, or by C ( X , Y}. The matrix Ex, = E { ( X - px)’(Y - p,)} is the covariance matrix of the column X and the row Y. The covariance between the ith element of X, and the jth element of Y, is c x y i j . Lowercase letter rn, appropriately subscripted, is used for the sample covariance matrix. For example,

Z,, and the ijth element of m,, where Z, = (Ztl,Z,,, . . . , Z,J, Z = n-’ is mZzij. Capital letter M, appropriately subscripted, is used for the matrix of raw mean squares and products. Thus,

M,,

= n-’

r=1

XIY,.

APPENDIX I.D.

97

NOTES ON NOTATION

We use the letter S, appropriately subscripted, for an estimator of the covariance matrix of the error vector. Thus, S,, is an estimator of the covariance matrix of E,. We shall generally use column vectors when discussing the distribution of a vector random variable. Thus, we write

z;

Iv

N W , &z),

where -NI(O, X z z ) signifies that the random column vectors Z;, t = 1,2, . . . , are distributed normally and independently with zero mean vector and covariance matrix Zzz. Special notation for mapping a matrix into a vector is defined in Appendix 4.A.

REFERENCES Appendix l.A. Cramer (1946), Fuller (1976), M a n n and Wald (1943). Appendix l.B. Anderson (1958). Appendix l.C. Kendall and Stuart (1977), Fuller (1976, 1980).

EXERCISES 54. (Appendix l . A ) Using the method of statistical differentials, find the mean and variance of the limiting distribution of nl”(Bx - ax), where

and the X, are normal independent (p, ):u random variables. 55. (Appendix l.A) Using the method of statistical differentials, obtain the mean and variance of the limiting distribution of R = Tx-’, where ( y , X,) are normal independent vectors with p, # 0 and pup; I = R. 56. (Appendix l.A) Let X, ( p x ,uXX),where px > 0. Let C x x= p;’u,, be the squared coefficient of variation of X . (a) Show that the coefficient of variation of the approximate distribution of X - ’ is equal to the coefficient of variation of 2. (b) Show that, for pi # 0 and p, # 0, the squared coefficient of variation of the approximate distribution of R obtained in Exercise 55 can be written

-

c,-9 = n-yc,,

- Zt,,

+ C,),

where C x y= p; ‘ p i I u X y . 57. (Appendixes 1.A, 1.C. Section 1.1) In Section 1.1 it was shown that E { f , , } = B , u i ~ u x x under the assumption that the (x,, e,, u,) are normally and independently distributed with diagonal covariance matrix.

98

A SINGLE EXPLANATORY VARIABLE

(a) Show that, for n

3,

WlA = ( n - 31- l ~ i : ( u X X ~ +p eP:%,%") under the assumption that the (x,, e,, u,) are normally and independently distributed with diagonal covariance matrix. (b) Assume model (l.l.l),(1.1.2) and assume {x,} is a fixed sequence with

Assume (e,, u,)'

-

I=!

Il-m

NI[O, diag(u,,, uuu)]and show that n'"(fll

+

where y l n = (mxx u J l ~ l m x l

v = bXX + U""F

- Y,,)5 N(O, v),

and lo,,

+ bXX + ~"")-4P:~xx~""(~:, + 4").

(c) Show that, under the assumptions of part (a),

~ ~ ~ C 9 1 ~ l ~=, ,2nl l' P h , , + ~ u u ) ~ 4 ( ~ ~ x ~ u u ) 2 . where the approximation is understood to mean that only the expectation of the leading term of the Taylor series is evaluated. 58. (Appendix LA, Section 1.2) Using the method of statistical differentials, find the limiting distribution of n112(s,, - u,,), where s,, is defined in (1.2.7). 59. (Appendix I.A, Section 1.2) Let X I NI(pc,,a,,) and let m x x = ( n - l ) - ' ( X I - XI2. (a) Obtain the exact mean and variance of m :; for n > 5. Compare this expression with the mean and variance of the approximate distribution derived by the methods of Appendix l.A. (b) Obtain the coefficient of variation of the approximate distribution of D = (1 - m;iusu). (c) Let PI = f(1 - m~:u,,)-', where 9 = m;:mx, and ( x , X,)are bivariate normal and satisfy model (1.2.1).Assume PI z-0. Show that the squared coefficient of variation of the approximate distribution of PI is C,, = C,, + C,,, where C,, and C,, are the squared coefficients of variation of the approximate distributions of 9 and D,respectively. (d) On the basis of higher-order approximations, Cochran (1977, p. 162) suggests that the approximate squared coefficient of variation of a ratio of uncorrelated approximately normally distributed random variables will be within 96% of the true squared coefficient of variation if the squared coefficient of variation of the denominator is less than 6%. For what values of u & ~and ~ n~ will the coefficient of variation of the approximate distribution of the denominator D be less than S%? 60. (Appendix 1.A. Section 1.2) Let (x,}? be a fixed sequence and let X I = x, u,. (a) Show that for u, NI(0, u,,),

-

I;=

+

-

V{m,, - mXx}= 4(n - I)-'

1 (x, - x)*u,, + Z(n - l)-'uiu.

I= I

(b) Using result (a), show that the coefficient of variation of A,, 0.2 if

m;iuuu < [0.02(n - 1) + 11"' - 1. Show that the coefficient of variation of A,, is less than 0.2 if

+

[(n - ~)m~,]-'uuu(uuum x x )< 0.01.

=

m,,

- uuuis less than

APPENDIX 1.D.

99

NOTES ON NOTATION

(c) Using Exercise 56, show that the squared coefficient of variation of the approximate distribution of (mxx - uuJ1mxx is

< 2(n - I)-lm;:u:u, where t = m,'u,,. 61. (Appendix 1.A, Section 13) Let b,, the estimator of p, of model (1.3.1), be defined by (1.3.7). (a) Let 0 = arctan fll and 0 = arctan PI. Show that, for u,, = u,,,

4 tan 26 = [ m x x - m,,l-l~ixy. Obtain the limiting distribution of 6.

+

(b) Show that the variance of the limiting distribution of n1'2(s,, - u",,) is 20;" u:ul-22, is defined in Theorem 1.3.1. where s,, is defined in (1.3.12) and 62. (Appendix I.A, Section 1.5)The estimates ofueelI and u,, of Example 1.5.1 are quite similar. Using the estimates and the estimated covariance matrix of Example 1.5.1, construct estimates of the parameters subject to the restriction ueell = uuu.That is, use generalized least squares to construct restricted estimates under the assumption that the estimators of Example 1.5.1 are normally distributed with the estimated covariance matrix constructed in Example 1.5.1. Do not use the original data in your calculations. Estimate the covariance matrix of your estimators. 63. (Appendix l.A) Let X I ) be distributed as a bivariate normal random vector. Using the methods of Appendix LA, obtain the approximate distribution of y i ; , where f x y = m;;mxy. 64. (Appendix l.B) Let 1: Nl(py,by,). Prove that

(x,

-

f{myy} = 2(n + ~ ) - ' m $ ,

is an unbiased estimator of V{m,,} = 2(n - l)-'u;,. Extend this result to the estimation of C{m,,, m x y l . 65. (Appendix l.B) Prove the following lemma (Joreskog, 1973).

Lemma. Let Z;

-

NI(0, E),t = 1,2,. . . , and let

w = Z-'(m and 2 = n -

?=

- Z)E-',

m

Z,. Then

= (n

- I)-'

CI (z,- Z)'(Z,- 21,

I=

+

C{wi,, w,,} = (n - 1)- '(uiruj' u"o*), where 0'' is the ijth element of E-'and wii is the ijth element of W.

-

66. (Appendix 1.B) Let Z; = (Zl,, ZJ NI(0, Z), where Z is a 2 x 2 matrix with 0th element equal to uij. Prove that the conditional variance of Z1,given Z2,is (.'I)- ', where ui' is the ijth element of Z- I , Generalize this result to higher dimensions.

Measurement Error Models Edited by WAYNE A. FULLER Copyright 0 1987 by John Wiley & Sons, Inc

CHAPTER 2

Vector Explanatory Variables

In this chapter we study models with more than one x variable. Maximum likelihood estimation is investigated and certain problems associated with the likelihood function are identified. Estimators of parameters that are extensions of the estimators of Chapter 1 are presented. The first four sections of this chapter parallel the first four sections of Chapter 1. Section 2.5 is devoted to the small sample properties of estimators and to modifications designed to improve the small sample behavior. The calibration problem is also considered in Section 2.5.

2.1. BOUNDS FOR COEFFICIENTS

In Section 1.1.3 we demonstrated that it is possible to establish bounds for the population regression coefficient in a simple regression with the independent variable subject to measurement error, provided the measurement error is independent of the error in Given a diagonal error covariance matrix, such bounds can be constructed for the model with a vector of explanatory variables. Let

x.

-

Zr = zr + er, N I [ ( p x ,O)', block diag(L,, L)],

Y, = Po (Xr, 8J'

+ XrB,

(2.1.1)

where Z, = (y, Xr), xr is a k-dimensional row vector, fl is a k-dimensional column vector, and the covariance matrix of Z,, denoted by Zzz,is positive definite. Under this model, the unknown parameter vector must satisfy (Zzz 100

w 1,

-

FY = 0.

2.1.

101

BOUNDS FOR COEFFICIENTS

Thus, if E,, is any positive semidefinite matrix, the vector

that satisfies

(ZZZ- XE,,)( 1, - j?)’ = 0,

where

2 is the smallest root

of

pzz- lee,(= 0,

is an acceptable parameter vector. We now assume that X,, is a positive semidefinite diagonal covariance matrix. This permits us to establish upper bounds for the error variances. If we solve the equation IX;zz - LiDiil = 0

for At, where Dii is a diagonal matrix with the ith diagonal element equal to one and all other entries equal to zero, we obtain the upper bound for the error variance of the ith component of Z,. The coefficient liis the population regression residual mean square obtained in the population regression of the ith element of Z, on the remaining k elements of Z,. Setting bounds for the elements of /Iis more difficult. We illustrate by considering a model with two explanatory variables. Let

I; = P o + P l X t l + P 2 x t 2 + ef and retain the assumptions of model (2.1.1). Then we can write the model as w o = Po* + P 2 W 2

+ e,,

wo

where wf2 is the portion of xt2 that is orthogonal to xtl, is the portion of I: that is orthogonal to x f I ,and (e0, = (w,,,, w , ~ ) (el, ut2).If we fix a value for ouull. say iiuull,in the permissible range for buul,,then one can define the new variables

w2)

( W O , Wf2) =

where

+

(I;- COlxtl, XI2 - C21X,I),

.. - 1 (CO,, C21) = ~ . x x l l ( ~ x Y 1 1 . ~ x x 1 2 ) ~

and iixxl1= oxxII - iiUull. It follows that the bounds of Section 1.1.3 apply to the reduced model and, for cuul= iiuul and 0 w w 0 2> 0,

d ~W~02~wwoo. If 0 w w 0 2has the same sign for all iiuullin the permissible range, then the derivatives of the bounds with respect to ijuull also never change sign. It follows that the unconditional bounds for will be in the set of conditional bounds associated with the minimum and maximum values of GUul If iiuul = 0, the lower bound for P2 is the regression coefficient of X 2 obtained ~ W ~ 2 2 ~ w w o 8 22

102

VECTOR EXPLANATORY VARIABLES

in the regression of Y on X, and X,, and the upper bound for p, is the inverse of the regression coefficient for Y in the regression of X, on Y and X,. At the maximum value for iiuull,the conditional upper and lower bounds for p, coincide. The value of 8, is the negative of the ratio of the coefficient for X, to the coefficient for Y in the regression of X , on Y and X,. If the conditional covariance between KOand W2 has a different sign for two values of ifuul in the permissible range, then there is a ifuul for which owWo2 = 0 and the range of possible values for the p’s is infinite. If the conditional covariances do not change sign, then the range of possible values is finite. For the general problem, consider the set of k + 1 values of /l constructed by using each of the k + 1 variables in Z, as the dependent variable in an ordinary least squares regression. In all cases the least squares solution is standardized so that the equation is written in the form Y = X/l. The bounds for /l will be finite if each coefficient vector in the set of k + 1 vectors obtained from the k + 1 regressions has the same sign structure. If the k + 1 vectors are all in the same orthant of the k-dimensional space, the set of possible values for /l is composed of the convex linear combinations of the k 1 coefficients obtained from the k + 1 regressions. If the set of k + 1 population vectors are not all in the same orthant, the set of possible values for /l is unbounded. See Klepper and Leamer (1984) and Patefield (1981). For the three-variable case, the possible set of values for (pl, 8,) is a triangle in (pl, p,) space with vertices corresponding to the values obtained from the three ordinary least squares regressions using each of the three variables as the dependent variable, provided all three solutions are in the same quadrant.

+

REFERENCES

Bekker, Wansbeek, and Kapteyn (1985), Klepper and Leamer (1984), Koopmans (1937), Patefield (1981). EXERCISES 1. (Section 2.1) Let model (2.1.1) hold with ZUe= 0. For one explanatory variable and simple uncorrelated measurement error it was shown in Section 1.1.1 that the expected value of the ordinary least squares estimatofi of the slope was closer to zero than the true value. (a) Show that

where y = ZXiZXu.(Hint: See (4.A.20) of Appendix 4.A.)

2.2.

103

THE MODEL WITH AN ERROR IN THE EQUATION

(b) Let the two squared multiple correlations be defined by K ? , = u,&’E,,,E~~ZXyand

R:,

=

u~~EyxE~xlZxy.

Show that R i X < R ; x . Show that the inequality is strict if at least one element of Zuuis positive. 2. (Section 2.1) Let model (2.1.1) hold and let XI = (xll, XJ be observed, where j’= (j’, B;), , x,, is measured without error, and X,2 = x , ~ + u12.Show that, typically, the least squares coefficients obtained in the regression of V, on x, are biased for both PI and f12. Show that the least squares estimator of 8 , is unbiased [or j1if E { x ; , x l 2 } = 0. See Carroll, Gallo, and Gleser ( I 985).

2.2.

THE MODEL WITH AN ERROR IN THE EQUATION 2.2.1.

Estimation of Slope Parameters

The generalization of model (1.2.1) of Section 1.2 to a model containing a vector of x variables is

r, = x,B + e,,

X,

=

x,

+ u,,

(2.2.1)

for t = 1,2, . . . , n, where x, is a k-dimensional row vector, /?isa k-dimensional column vector, and the (k 1)-dimensional vectors E: = (et, u,)’ are independent normal (0,C8J random vectors. It is assumed that the vector of covariances between e, and u,, Ze,, and the covariance matrix of u,, C,,, are known. The variance of e,, cr,,, is unknown. Instead of assuming the x variables to be random variables, as we did in Section 1.2, we initially assume that {x,} is a sequence of fixed k-dimensional row vectors. We examine the likelihood function assuming that XEEis nonsingular and, with no loss of generality, that

+

Xu”= I and EUe= 0.

(2.2.2)

With these assumptions, the density of (e,, u,)’ is (271)- W + 1)D0,1/2

exp(

ei2

+ uru3>-

(2.2.3)

Because x, is fixed, the Jacobian of the transformation of (e,, u,) into ( y, X,) is one, and the logarithm of the likelihood for a sample of n observations is log L = -+n[(k

+ 1) log 2.n + log

Oee]

To obtain the maximum likelihood estimators, the likelihood is to be maximized with respect to p, gee,and x,, t = 1,2,. . . , n. Differentiating (2.2.4),

104

VECTOR EXPLANATORY VARIABLES

we obtain

(2.2.5)

Equating the partial derivative with respect to x, to zero and using the hat

(*) to denote estimators, we obtain

i; = [8i1jpt I]-1[8,11;fi Noting that

flp + 11-

*-IAA

[o,,

we obtain

= I - [l

+ Xi],

t =

1,2,. . . , n.

(2.2.6)

+ 8,18'jyjjv;,f,

r, - 2,s = (8,,+ j$)-'(y, - X,j)S,,, x, - 2, = (8, + bj)-'(Y;- X,j)b.

(2.2.7) (2.2.8) If we substitute (2.2.7)into the second equation of (2.2.5)and set the derivative equal to zero, we obtain

8,'(8,,

+ b'j) = (8ee+ Pj)-'[n-l

I=1

(Y, - X&].

(2.2.9)

Equation (2.2.9)signals a problem. We know that the variance of

+

Y, - X,fl = e, - u,fl

is o,, fl'fl. Therefore, the quantity on the right side of Equation (2.2.9) should be estimating one, but the quantity on the left side of the equation is clearly not estimating one. We conclude that the method of maximum likelihood has failed to yield consistent estimators for all parameters of the model. This is a common occurrence when the likelihood method is applied to a model in which the number of parameters increases with n. In our model the parameters are (o,,, fl, x2,. . . ,xn). We should not be surprised that we cannot obtain consistent estimators of n + 2 parameters from n observations. Estimation in the presence of an increasing number of parameters has been discussed by Neyman and Scott (1948), Kiefer and Wolfowitz (1956), Kalbfleisch and Sprott (1970), and Morton (1981a). Anderson and Rubin (1956) showed that the likelihood function (2.2.4)is unbounded.

xl,

2.2.

105

THE MODEL WITH AN ERROR IN THE EQUATION

Let us now examine maximum likelihood estimation for the model with random x values. We shall see that the application of likelihood methods to the normal structural model produces estimators of the unknown parameters that are consistent and asymptotically normal under a wide range of assumptions. Let

X = Bo + x,Sl + e,,

+ u,,

X, = x,

(2.2.10)

where

Assume that C,, and Zueare known. To be conformable with the dimensions of (2.2.1), we let X, be a (k - 1)-dimensional vector. Let a sample of n vectors Z, = (X,X,) be available. Then twice the logarithm of the likelihood adjusted for degrees of freedom is 2 log L,(B) = - kn log 27c - ( n - 1) loglC,,I

-

(n - 1) tr{mzZ&i}

(2.2.11)

- n(Z - C Z ) G A Z - P Z Y ,

where 8’ = [ A , P o , B;

9

bee,

(vech LJ’]

and Czz = (A,I Y ~ X X ( P 1 ,1) + Z E E .

(See Appendix 4.A for the definition of vech Zxx.)The number of unknown parameters in B is equal to the number of elements in the vector of statistics [Z, (vech mZJ]. It is well known that 4, = and kzz = m,, maximize the likelihood (2.2.11) with respect to pz and Zzz when there are no restrictions on pz and Czz. Therefore, due to the functional invariance property of the method of maximum likelihood, the maximum likelihood estimators adjusted for degrees of freedom are

z

(A, Po, = (X, Sl

-

Xh,

= (mxx - ~

8 e e = mYY

’-

u u ) -I ( m X Y

(2.2.12) - Cue),

2mY,B,+ blm,.J,

ex,

+ 2xeuB1 - B;Ls~,

and Ex, = m,, - E,,, provided is positive definite and Be, 2 Ee,?&Zue, where Xiu is the Moore-Penrose generalized inverse of Xu,.If either of these conditions is violated, the estimators fall on the boundary of the parameter space. Let < i;t2< * * * < ;IF!l < 1; be the positive values of I - ’ that

106

VECTOR EXPLANATORY VARIABLES

satisfy (2.2.13)

where iiaall = Ze,,EiuEueand (2.2.14)

If A, c 1 and

A,-

ex,= mxX- ik&,u,

> 1, the maximum likelihood estimators are j 1

= (mxx

- lkZuu)-

l(mXl'

- RkZue),

(2.2.15)

and bee= ZeuZiUZue. If I,- < 1, the maximum likelihood estimator of Zxxis singular and the estimator of /I1is indeterminate. Estimator (2.2.15) is derived in the proof of Theorem 4.C.1. The estimator of given in (2.2.12) is the vector analogue of the estimator given in (1.2.3). On the basis of degrees-of-freedom arguments, we suggest that the estimator of oeein (2.2.12) be replaced by = svv - B;cuJl + 2EeuB1,

x;=l bee

(2.2.16)

where s,, = (n - k)-' [I; - 7 - (X, - X)$,]'. In many practical situations the matrices &, and Eueare not known but are estimated. For example, it is common for laboratories to make repeated determinations on the same material to establish the magnitude of the measurement error. If the covariance matrices for measurement error are estimated, a more detailed specification of the model than that given in (2.2.1) is required. This is because the random variable e, entering Equation (2.2.1) may be composed of two parts. The true values y, and x, will not be perfectly related if factors other than x, are responsible for variation in y,. Thus, one might specify Y f = x,P

+ 41.

(2.2.17)

where the q, are independent (0, oQq) random variables, and 4, is independent of x j for all t and j. The random variable 4, is called the error in the equation. We observe

(X,XI) = (Y,, XI) + (w,,u,),

(2.2.18)

where (w,,u,) = a, is a vector of measurement errors, a;

NI(& Zoo),

(2.2.19)

and a, is independent of (qj,x j ) for all t and j. In terms of the original model (2.2.1), e, = w, q, is the sum of an error in the equation and an error made in measuring y,. Typically, the variance of q, is unknown, but it is possible to conduct experiments to estimate the covariance matrix of a, = (w,,u,). Be-

+

2.2.

THE MODEL WITH AN ERROR IN' THE EQUATION

107

cause w,and q, are assumed to be independent, the covariance between u, and w, is equal to the covariance between u, and e,. Let S,, denote an unbiased estimator of Zoo.Then the estimator of fi analogous to (2.2.12) is

S = (Mxx - StdU)-'(MxY- Suw).

(2.2.20)

We use matrices of uncorrected sums of squares and products to define the estimator so that the estimator is conformable with model (2.2.17). The definition of the estimators permits the matrix S,, to be singular. For example, if the model contains an intercept term, one of the elements of X,is always one and the corresponding row and column of S,, are zero vectors. Given the estimated error covariance matrix S,,, a consistent estimator of aqqis (2.2.21) gqq= s,, - (SWW- 2 B U W + P"'S,,B),

c:=,

where s,, = (n - k)(I:- X,B)'. In Theorem 2.2.1 we give the limiting distribution of the standardized estimators of fi and oqqunder less restrictive conditions than those used to obtain the maximum likelihood estimator. The x, are assumed to be distributed with a mean p,, and covariance matrix Xxx, where the covariance matrix C,, can be singular, This specification contains both the functional and structural models as special cases. Under the fixed model, x, = pxr are fixed vectors. Under the random model I(,, 3 (1, p,), where p, is fixed over t, and the lower right (k - 1) x (k - 1) portion of X,, is positive definite. Dolby (1976) suggested that the model with random true values whose means are a function of t be called the ultrastructural model. To obtain a limiting normal distribution, some conditions must be imposed on the sequence of means of the true values. We assume that (2.2.22)

+

and let M,, = M,, Zxx,where M,, is positive definite. Commonly, the error covariance matrix will be estimated from a source independent of the M,, matrix. To assure that the error made in estimating the error covariance matrix enters the covariance matrix of the limiting distribution, we assume that the error in the estimator of C,, is of the same order as the error in Mzz. This is accomplished by assuming that the degrees of freedom for S,, is nearly proportional to n. Thus, we assume that lim d;'n = v,

n-m

(2.2.23)

where d , is the degrees of freedom for S,, and v is a fixed number. If the covariance matrix of the measurement error is known, then v = 0.

108

VECTOR EXPLANATORY VARIABLES

Theorem 2.2.1. Let model (2.2.17)-(2.2.19) hold and let (x, - p,,, 4,) be independently and identically distributed with zero mean vector and covariance matrix, E((xr - pxrr qrY(xr - ~ x r 41)) , = block diag(Cxx, nqqh

where { p x r }is a sequence of fixed k-dimensional vectors, q, has finite fourth moments, and q, is independent of x,. Let S,, be an unbiased estimator of ZOathat is distributed as a multiple of a Wishart matrix with d, degrees of freedom, independent of (3, XI) for all t . Let (2.2.22) and (2.2.23) hold. Let 0 = (p,oqq)'and let 8 = (p,dJ, where is defined in (2.2.20) and dqq is defined in (2.2.21).Then

n'''(8 where the submatrices of

- e)

A N(O, r),

r are

rsa= M,'o,,

+ M;:[z,,~~~ + C,,~Z~~]M;:

+ vM.L'[Lorr + LZ:vu]Mxxl, rq4= v{u:}+ 2 ~ 4 , = 2MixlGinuu

r p q

and or, = ow, - 2Z,,j?

+ B'Z,,j?.

+ vow),

Proof. Using the fact that X,= x,

+ u,, we may write

y; = X,B + 4, where u, = e,

- u,B.

It follows that

B = (Mxx - S,J=B

W x x S + Mxu - s,, + S U U B - S,,B) + (M,x - Suu)-'C(Mxu - Cur) - (Sur - Zur)],

c:=

where Mxu = n-' X,u,, Zuw- Z,,~ = Zuu.NOW M,, = n -

I,

= w,

- ur/?, Sur= S,, - S,,p, and Cur=

(xix,

+ xIu, + uix, + u)~,)

n

r= 1

and, by the weak law of large numbers,

2.2.

109

THE MODEL WITH AN ERROR IN THE EQUATION

Consider the linear combination n1'*S'(MxU - Xu,,) = n-'12

C C S,[x,,u, + uriu, - ouuil] 1=1 i = l n

k

(2.2.24) where ouuilis the covariance between tiir and u,, S = (dl, S,, . . . ,dk)' is an arbitrary vector such that 6'6 # 0, and the random variables, k

gt =

i=1

+ utiur - ousill,

Si[xtiut

are linear functions of independent and identically distributed random vectors. The mean of gr is zero and the variance is

Eb:}

= a"(P:IPx*

+L+U

o u u

+ &"Lula*

Therefore, by Lemma 1.C.1, n-1/2

f gr 5 N { O ,s ' M ~ +~ a'(~,,,,o,,~ s ~ ~ ~+ ~

r=1

~ ~ ~ ~ ~ ) t j } .

Because the nonzero S was arbitrary,

+

n1'2(Mxu- XU")5 "0, MxPua ~

+

u u ~ u u~

" U ~ " , ) .

conThe limiting distribution for n l i 2 ( B - p) follows because nl/'(S,,, - Zu,) verges in distribution to a multivariate normal random vector with mean zero and covariance matrix

+L

~ F u u o r r

r L )

and because S,, is independent of Mxv. We have shown that - j = O,(n-'/'), and it follows that

d,, = ( n - k)-'

I=I

u:

- (1, -b')Soo(l, -/?')'

Because S,, is independent of M,,,and

incl,,the

+ O,(n-').

conclusion follows.

0

110

VECTOR EXPLANATORY VARIABLES

can be estimated by

The variance of the approximate distribution of

V{/Q

-

= n-'[M,'s,,

+ M~J(s,,,,~,, + ~,,,~,,,,~M~~]

+ d j lM;JIS,,s,, + s,,&]M;J,

where M,, = M,, - S,,,

s,, = S,,

(2.2.25)

-

- S,,fl,

s, = (1, -P)SaaU,

-PI',

and s,, is defined in (2.2.21).If Z,,, and Zueare known, the estimated covariance matrix is given by (2.2.25) with d j ' = 0. The normality of u, and the assumption that S,,, is a Wishart matrix were used in Theorem 2.2.1 to obtain an explicit expression for the covariance matrix of the limiting distribution. If M;JS,, is not large, the variance estimator (2.2.25) will be satisfactory for modest departures from normality. Occasionally, some of the variances and (or) covariances in the matrix Xa,are known to be zero, while others must be estimated. A common model in sociology and psychology postulates a diagonal covariance matrix for the measurement error. In most such cases the estimators of the individual error variances will be independent. An estimator of fl for the model with diagonal covariance matrix is

s' = Wxx - Suu)-'Mx~,

(2.2.26)

and the elements of (Suull,Suu22,. . . , where S,, = diag(S,,, Suu22,. . . , Suukk) Suukk)are independent unbiased estimators of (uuull, o , , , ~ .~, ,. , uUukk) distributed as multiples of chi-square random variables with dr!, d,,, . . .,d f k degrees of freedom, independent of (yt, X,) for all t . The estimator of the covariance matrix of the approximate distribution of 8 is

where M,,

a{@}= M;~[n-'(MxxsuU .. .. - + s,,6,,)+ 2&,]fi;J,

= M,,

- S,,, S,, I

= -S,,fl,

k,= diag{d;?~%,ll, d;21&S:uz2,. . .

(2.2.27)

dYk1Bk - 2 s 2uu~d,

and s,, is defined in (2.2.21). Because the knowledge that some covariances are zero is used in constructing the estimator, the variance estimated by (2.2.27) is smaller than the variance estimated by (2.2.25), for comparable degrees of freedom. Example 2.2.1. In this example we consider some data studied by Warren, White, and Fuller (1974). In the original study the responses of 98 managers of Iowa farmer cooperatives were analyzed. We use a subsample of the original data containing 55 observations. The data are given in Table

TABLE 2.2.1. Data from role performance study Observation

Knowledge

Value Orientation

Role Satisfaction

Past Training

Role Performance

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

1.193 1.654 1.193 1.077 f ,539 1.385 1.462 1.385 1.539 1.654 1.462 1.154 1.424 1.116 1.270 1.347 1.116 1.077 1.423 0.923 1.385 1.270 1.116 1.346 0.846 1.077 1.231 0.962 1.500 1.577 1.885 1.231 1.808 1.039 1.385 1.846 1.731 1.500 1.231 1.346 1.347

2.656 3.300 2.489 2.478 2.822 3.000 3.111 2.545 2.556 2.945 2.778 2.545 3.61 1 2.956 2.856 2.956 2.545 3.356 3.21 1 2.5f 6 2.589 2.900 2.167 2.922 1.711 2.556 3.567 2.689 2.978 2.945 3.256 2.956 2.81 1 2.733 2.400 2.944 3.200 2.91 1 3.167 3.322 2.833

2.333 2.320 2.737 2.203 2.840 2.373 2.497 2.61 7 2.997 2.150 2.227 2.017 2.303 2.5 17 1.770 2.430 2.043 2.410 2.150 2.180 2.490 1.920 2.663 2.520 3,150 2.297 2.307 2.830 2.731 3.1 17 2.647 2.217 2.321 2.447 2.347 2.410 2.277 2.577 2.507 2.653 2.587

2.000 2.000 2.000 2.000 2.000 2.000 2.667 2.167 2.000 2.167 1.833 2.167 2.333 2.333 2.333 2.000 2.000 2.000 2.000 2.000 2.000 2.333 1.333 2.000 1.500 2.000 2.167 1.333 2.000 2.167 2.667 1.667 2.000 2.000 2.167 2.000 2.000 2.000 2.333 2.500 2.667

- 0.054

19

20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

0.376 0.072 -0.150 0.171 0.042 0.188 - 0.052 0.3 10 0.147 0.005 - 0.088 0.044 - 0.073 0.224 0.103 -0.108 -0.019 - 0.062 -0.239 -0.159 0.069 -0.1 18 0.083 -0.255 -0.159 0.014 0.102 0.109 0.006 0.334 - 0.076 - 0.043 -0.126 - 0.056 0.203 0.023 0.047 0.01 1 0.153 0.100

111

112

VECTOR EXPLANATORY VARIABLES

TABLE 2.2.1.

(Continued)

Observation

Knowledge

Value Orientation

Satisfaction

Role

Past Training

42 43 44 45 46 47 48 49 50 51 52 53 54 55

1.154 0.923 1.731 1A08 1.193 1.308 1.424 1.385 1.385 1.347 1.539 1.385 1.654 1.308

2.967 2.700 3.033 2.91 1 3.311 2.245 2.422 2.144 2.956 2.933 3.41 1 1.856 3.089 2.967

3.140 2.557 2.423 2.793 2.283 2.210 2.350 2.330 2.130 2.837 2.600 2.790 2.500 2.8 13

2.000 1.833 2.000 2.000 2.333 2.000 2.000 2.000 2.000 2.167 2.167 2.000 2.000 2.667

Role

Performance - 0.089

0.007 0.089 0.182 0.259 0.007 - 0.015 - 0.023 -0.150 0.152 0.377 0.043 0.184 0.127

2.2.1. The postulated model is 4

Y, = P o

+ iC X t i P i + qr, = 1

(2.2.28)

where y, is the role performance of the tth manager, x1 is knowledge of the economic phases of management, x 2 is value orientation, x 3 is role satisfaction, and x4 is past training. The random variable q, is assumed to be a normal (0,oq4)random variable. Value orientation is the tendency to rationally evaluate means to an economic end, role satisfaction is the gratification obtained from the managerial role, and past training is the amount of formal education. The amount of past training is the total years of formal schooling divided by six and is assumed to be measured without error. Role performance, knowledge, value orientation, and role satisfaction were measured on the basis of responses to several questions for each item. Using replicated determinations on the same individuals, measurement error variances were estimated to be 0.0037, 0.0203, 0.0438, and 0.0180 for role performance, knowledge, value orientation, and role satisfaction, respectively. Each error variance estimate is based on 97 degrees of freedom and it is assumed that Z,,is,,diagonal. This is the model for which estimator (2.2.26)is appropriate. The matrix of mean squares and products corrected for the mean is

i

0.0598 0.0345 mxx = 0.0026 0.0188

0.0345 0.0026 0.1414 -0.0186 -0.0186 0.0887 0.0474 - 0.0099

1

0.0188 0.0474 - 0.0099 0.0728

2.2. THE

MODEL WITH AN ERROR IN THE EQUATION

113

and

m i u = (0.0210,0.0271, 0.0063,0.0155). using the program SUPER CARP. For reasons We compute the estimate of /l that will be explained in Section 2.5, the estimator is computed by subtracting a multiple, slightly less than one, of Suufrom the moment matrix. The estimator of expressed in terms of uncorrected moments is

b = [Mxx - n - ' ( n

- 6)Suu]-'MXi =[-1.24, 0.36, 0.149, 0.117, 0.040]', (0.27) (0.13) (0.094) (0.069) (0.075)

(2.2.29)

where the first coefficient is the intercept and the numbers in parentheses are the estimated standard errors of the coefficients.The sample size is n = 55 and 5 is the number of parameters estimated. The standard errors are obtained from the covariance matrix of expression (2.2.27). In this example the increase in the variance associated with the estimation of the error variances is small. For example, the estimated variance of the coefficient for knowledge, assuming the error variance to be known, is 0.0157. The estimated variance of the knowledge coefficient, under the assumption that the estimated error variance is based on 97 degrees of freedom, is 0.0173. By Equation (2.2.21), an estimator of oqqis Cqq= 0.0129 - 0.0075 = 0.0054,

x:=,

where s,, = 0.0129 and S,, + in the proof of Theorem 2.2.1,

Suuiiflf = 0.0075. By the arguments used

(2.2.30) and an estimator of the variance of the approximate distribution of Gqq is

= (0.00266)2.

(2.2.31)

The estimated standard error for Cqq is about one-half of the estimate. An alternative test of the hypothesis that nqq= 0 is developed in Section 2.4 and illustrated in Example 2.4.2. 00

2.2.2. Estimation of True Values In Section 1.2.3 we obtained, for fixed x,, the best estimators of the x, using the model information. We now extend those estimators to the model of this section. We assume model (2.2.17)-(2.2.19).and, as usual, we let u, = e, - u,fl

114

VECTOR EXPLANATORY VARIABLES

and a, = (w,,u,). In the model (2.2.17) the true y, differs from x,p by 4,. Hence, it is of interest to estimate both y, and x,. To estimate (yt, x,), we predict (e,, u,) and subtract the predictor of (el, u,) from (X, XI).Under normality, the best predictor of (e,, u,), given u,, is

(2.2.32)

(e,, U,) = u,6’,

where 6’= r ~ ~ ; ~ & , ~It. follows that the best estimator of (y,, x,), treating x, as fixed, is

(2.2.33)

( j , ,XI) = (K, XI) - U,d’.

For the estimator of z, constructed with estimated model parameters, we let

-

2, = ( j , ,a,) = (u,, X,)- OI,&

(2.2.34)

-p)’.

where s* = o,, Xu,, &, = (1, -p)Se,, and 8,, = (1, -F)Mzz(l, An estimator of the covariance matrix of the error in the approximate distribution of 2, is A-1-

cr{i;lx,} = s,, - uuu&&. ^.-I-

(2.2.35)

The estimator (2.2.35) ignores the contribution of estimation error in band 8,, to the variance. An estimator of the variance of ( j , ,n,) that contains order n-’ terms associated with parameter estimation is given in (2.2.41) of Section 2.2.3.

Example 2.2.2. We construct estimates of the individual y and x values for Example 2.2.1, treating x, as fixed. Let where xI is knowledge, x2 is value orientation, x j is role satisfaction, and x4 is past training. Then the estimated covariance matrix of the measurement error is

The matrix I

-

0.712 -0.356 0.103 0.043 0.034

0.0 0.564 0.507 0.164 0.627 0.204 1.0 0.698 0.0 0.799 -0.181 -0.059 0.925 -0.024 0.0 -0.084 0.981 0.0 -0.066 -0.059 - 0.011 0.0 -0.022 -0.020 -0.007

0.0 0.0 0.0 0.0 0.0 1.0

2.2.

115

THE MODEL WITH AN ERROR IN THE EQUATION

TABLE 2.2.2. Estimated true values for role performance data

i Observation

1 2 3 4 5 6

7

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

B

Knowledge

Value Orientation

Role Satisfaction

Role Performance

0.010 0.182 0.1 14 - 0.003 0.028 -0.018 0.043 - 0.080 0.188 0.0 19 - 0.026 0.037 -0.126 -0.061 0.284 0.056 0.034 - 0.027 -0.141 - 0.046 -0.172 0.105 0.034 0.03 1 - 0.003 - 0.034 - 0.074 0.212 - 0.032 - 0.208

1.198 1.756 1.257 1.075 1.554 1.374 1.486 1.340 1.644 1.664 1.447 1.175 1.353 1.081 1.430 1.378 1.135 1.062 1.344 0.897 1.288 1.329 1.135 1.363 0.845 1.058 1.189 1.081 1.482 1.460

2.661 3.392 2.546 2.476 2.836 2.991 3.133 2.504 2.65 I 2.954 2.765 2.563 3.547 2.925 2.999 2.984 2.561 3.342 3.140 2.533 2.502 2.953 2.184 2.938 1.710 2.538 3.529 2.796 2.962 2.839

2.335 2.350 2.755 2.203 2.845 2.370 2.504 2.604 3.028 2.153 2.222 2.023 2.283 2.507 1.817 2.439 2.049 2.406 2.127 2.173 2.462 1.937 2.669 2.525 3.150 2.291 2.295 2.865 2.732 3.082

- 0.057

0.324 0.039 -0.149 0.163 0.047 0.176 - 0.029 0.256 0.142 0.012 - 0.099 0.080 - 0.056 0.142 0.087 -0.118 -0.011 - 0.021 -0.226 -0.110 0.039 -0.128 0.074 - 0.254 -0.149 0.035 0.04 1 0.1 I8 0.066

where s,,, = 0.0129, (1,

(d,,,

-p)= (1, 1.24, -0.36,

-0.149, -0.117, -0.040),

2””)= (0.0037,0.0, -0.00731,

-0.00653, -0.00211, 0.0).

The estimated y and x values for the first 30 observations are given in Table 2.2.2. Because xo = X , = 1 and because past training is assumed to be measured without error, estimates for xo and x4 are not given in Table 2.2.2. The estimated covariance matrix of the estimator of z, defined

116

VECTOR EXPLANATORY VARIABLES

in (2.2.35)is

-

0.0026 0.0 0.0

0.0

0.0021 0.0019 0.0006 0.0

0.0 0.0 0.0 0.0

0.0021 0.0019 0.0 0.0 0.0162 -0.0037 -0.0012 0.0 -0.0037 0.0405 -0.0012 -0.0011 0.0177 0.0 0.0 0.0 0.0 0.0

The quantity 17, = pixfiis also given in Table 2.2.2. Under the model, u, and 1, of (2.2.33)are uncorrelated. Therefore, as with the simple

0.25

0.20 0.15

.

0.10

. .

0.05

5I 3

0

1

. *

-0 . 0 5 -0.10

*

.

. :

. .' 5

.

.

.*

.. .'

-0.15

--0.25 1.5

1.7

o1.9 2.1

2.3 ,2.5

2.7

2.9

2 3.1

3.3 3.5 0

EST. TRUE VALUE FOR VALUE ORIENTATION FIGURE 2.2.1. Plot oF residual against estimated true value for value orientation.

~

2.2.

117

THE MODEL WITH AN ERROR IN THE EQUATION

model of Section 1.2, plots of 6,against the elements of fit should be constructed as model checks. Figure 2.2.1 contains a plot of 6,against the estimated true value for value orientation, ?,. This plot gives no reason to reject the assumptions of linearity and normality. The plots of 6,against the other estimated true values also gave no reason to reject the original model. Figure 2.2.2 contains a plot of the ordered G, against the expected value of the normal order statistics for a sample of size 55. The KolmogorovSmirnov statistic for the test of normality has a significance level of 0.084 when computed under the assumption that the 6, are a simple random sample of size 55. Given that the 6,are based on estimated parameters it

0.25

0.20 0.15

...

0.10

.*

0.05

...*'.

I-

2I 0 >

....

..'

-0.05

*.*-

-0.10 -0.15

.'

-0.20 -0.25. *

I

-2.0

x

1

1

I

-1.0

1

1

I

I 0

I

I

I

I

1

1.0

1

1

I

I

1

I

2.0

118

VECTOR EXPLANATORY VARIABLES

seems that one would be willing to use calculations based on the normal no model.

2.2.3. Higher-Order Approximations for Residuals and True Values The estimator of the variance for the estimated x, given in (2.2.35) ignored the increase in variance of the estimator that comes from replacing parameters with estimators. In this section, we develop an estimator for the variance of the approximate distribution of 2, that contains an estimator of the contribution to the variance arising from the estimation of the parameters. We retain model (2.2.17)-(2.2.19) with the assumption of normal qr and the assumption that the covariance matrix Z,, is known. Assume that we have completed estimation for a sample of size n so that we haveTstimators of /?,oq4,and o,, and an estimator of the covariance matrix of fl. We wish to estimate the vector of true values (y,, x,) and the covariance matrix of the estimation error, treating x, as fixed. Let

-b') 8' = &;leva

a^' = (1,

and a' = (1, -/?'), and 6' = (T;'Z,,, and u, = Z,a = eta,

i?, = Z,d

' c:=

utl, 8, = (w, + q,, u,) = (el, u,),

where Z, = (X,X,),a, = (w,, a*'&,,, a, = n i?;, and The estimator of z; is

2; = z; - 60, = Z; - 6u, - (,$ - S)U, where

S^ - s = a,['(~,

- 26E,,)(oi

Therefore,

&a

= a'z,,,

e,, =

fl is defined by (2.2.20) with Z,, replacing So,.

+

- SZ,(&- a)

+ oP(n-'),

- a) - ~(m,,, - o,,)]

(2.2.36) (2.2.37)

+ op(n-').

2; - z; = Z; - - [Sz, utou;'(Z,, - 6Z,,)](& - a) v,o,'6(mu, - o,,) O,(n- 1) = 2; - 2; [SZ, u,(J;'(E,, - SZ,,)][O, (M,X - Zuu)M,']' (2.2.38) v , ~ , ' ~ ( m ,, o,,) Op(n-

+

+

+

+

+

+

'x

where z, = Z, - ~ ~ 6 ' . We drop the term of O,(n-') in (2.2.38)and evaluate the covariance matrix of the leading terms. The random vector 2, - 2, = E, - U,s' (2.2.39)

is independent of v, and it follows that the covariance between e, - 0,s' and the remaining 0,(n-''2) terms of (2.2.38)is zero. The only cross product term

119

Therefore, the covariance matrix of the approximate distribution of 2, - z, is Xaa -

+ 6d'E{~[v,,z;}+ c;uyz,,

- SzU,)v,,(Xa0 - dXuO) - SC,,)(O, ZuuMi:)'Xu,

+ 2(n - k)-'66'0,, + 2n-'0;~'[(&, + 2=uuM,')(C,, - G,)l,

(2.2.40)

%io9

where V,, is the covariance matrix of the approximate distribution of ai and M,, is the matrix of mean squares and products of the true x,. Because the first row and column of V,, contain only zeros, zv z = g v 21 t

aa I

f

pp

17

b.

where V,, is the covariance matrix of the approximate distribution of All terms in expression (2.2.40)have sample analogues. Therefore, an estimator of the variance of the approximate distribution of 2, - z, is

-

V(2, - z,} = c,, - 6Zu,

+ d,yC,,

+ 2n- l[(X,,

va,

+ 8S11[%,Vp,%;+ 2(n - k)- 'auu]

- &,)vUa(C,, - &),

- s^2,,)(0,ZuuM;;y&

+ &o, 2uuM;:)(L,

v,,)

e,,

-

&,)I,

(2.2.41)

where = block diag(0, and is >he estimator of the covariance matrix of the approximate distribution of p. The last three terms of (2.2.41) are products of three matrices, where two of the matrices are matrices of error variances and the third is a function of the inverse of an estimator of Mxx.Therefore, the last three terms of (2.2.41) will be small relative to the other terms when the error variance is small relative to the variation in x,. An approximation to the first three terms of (2.2.41) is ${il - zl} = c,, - n-'(n - k - 2)SC,,. * A

(2.2.42)

To develop higher-order approximations for the predictor constructed , x, = (1, x , ~ and ) xllis under the structural model, we let ztl = (y,, x , ~ ) where the (k - 1)-dimensional vector that is the random portion of x,. Then a predictor of z , ~is Z,, = Z1 + (Zt1- %)(I - C m i A I L l l ) ,

(2.2.43)

120

VECTOR EXPLANATORY VARIABLES

where c = (n - l)-'(n - k - 2), inzz1 is the sample moment matrix of ZI1, CIIoI1is the covariance matrix of aI1,and a,, is the error associated with ztl. The predictor is an estimator of the conditional expectation of z,, given ZI1. Under the normal distribution assumption, an estimator of the variance of Z,, - zI1is W I l

- Z I l > = CPclll

- C2~oa11~i;lI~.P11.

(2.2.44)

The multiplier c appearing in expressions (2.2.43) and (2.2.44) is based on the work of Fuller and Harter ( 1987). One can also use Taylor series to derive the covariance matrix of the approximate distribution of the residuals fi,. Let 9 denote the n-dimensional column vector $ = (i?,, 8,, . . . , 8J = Y - xg, where Y = (Yl, Y 2 , .. . , Y")', and X' = (Xl, Xi,. . . , Xk). By our usual expansions,

X(/ - /I) = v - XM,'(M,, -q+ o&+).

0 =v

-

(2.2.45)

Now, for fixed x,, tr{[~;~tcvu +( where aj, is Kronecker's delta. It follows that

- %J} =

E{ujXW,'(M,u

E { [ v - n-'XM,'(M,,

- Cuu)][V

= InCcuu

- n-'XM,'(M,,

- 2n-I t r { ( L e u u

L c u u

+ ~:uFuuPjr]Mxxl},

- ZuU)]']

+ L&w)ML'} + tr{LVpp}]

- ~ ~ - ' x M , L x ' o ,+, n-'xVpp~' + O(n-2).

The quantity of most interest for model checking is the vector of standardized residuals s~;"~$. Using - 112 = A - 1 / 2 - LA - 3/2 uu

SUU

= A,"2[ 1

2 uu [- 2 ~ u u ~ , 1 ~ u u-( Bn1 + O,(n- e,'C,,(j - /?)I + O,(n- I),

we have [v - X(B - /91[1 - G 1 & w ( B - n1 + O p ( n - 7 Aiu1'2[I,- n-'SM,'f']v Op(K1), (2.2.46)

- 1/29 = A-

suu

uv

=

1/2

+

where x = X - vc;,,lCUu, and A,, = ( n - k ) - ' ~ ' ( 1 ~n- 1X'ML:X')2~.

In ordinary least squares estimation, with x observed, it is common practice to estimate the covariance matrix of the residuals with [I, - x(x'x)-'x']s2,

2.2. THE MODEL

121

WITH AN ERROR IN THE EQUATION

where I, is the n-dimensional identity matrix and s2 is the residual mean square. By (2.2.46),the analogous estimator for the measurement error problem is

[I,

+ B ( ~ ; ’ Q ~ -~ 2n-’M,’)i‘]s,,,

(2.2.47)

where 9 is the n x k matrix of estimated true values and nMxx is the estimator 1 x;x,. of Expression (2.2.47) for the approximate covariance matrix of the 0, contains terms that are not present in the covariance matrix of ordinary least squares residuals. Nonetheless, the covariance matrix

I:=

[I,

- qB’2)-12’]S”,

furnishes a useful approximation in many applications. For example, if one has available a program for diagnostic checking for ordinary least squares, the 6, and 9, can be read into the program as the “dependent variable” and the vector of “independent variables,” respectively. The resulting diagnostics can be treated, approximately, in the same manner as one would treat the statistics for ordinary least squares.

Example 2.2.3. We use the data and model of Example 1.2.1 to compute estimates of the variances of 2,and 9,. From Examples 1.2.1 and 1.2.2, we have n = 1 1 , b, = 0.4232, uuu= 57,8,, = 53.4996,8, = 247.8545, and 8,, = - 24.1224. We assume that the measurement error made in determining yield for a sample of plots within the field is uww= 25 and we also assume that uuw= 0. It follows that A-18’ = oliu Xua = 8 u i 1 ( ~ u0,w8,J , = (0.4673,0, -0.4509),

1.

where the zero element in the vector is for the intercept. The first-order approximation to the covariance matrix of (5, - y,, I, Zr - x,) is

c,, -sc,, *-

=

[

13.3177 0 11.2722 0 0 0 11.2722 0 46.1235

For a model with an intercept, the computations are somewhat simplified and are less subject to rounding error if all independent variables, except the variable that is identically equal to one, are coded as deviations from the mean. If we let

Z, = ( x , 1, X , - 70.6364),

then d’ = ( 1 , -97.4545, -0.4232), nMxx= diag( 1 1 , 2478.545), and

quu= diag(0, 4.8636,0.0304).

122

VECTOR EXPLANATORY VARIABLES

TABLE 2.2.3. Order approximations to the variance of ( i f$,, $) for corn yield and soil nitrogen

Site

i ,- R

1 2 3 4 5 6 7 8 9

- 5.68

10

11

29.24 - 17.64 - 10.54 25.37 - 8.28 - 16.00 - 1.17 19.60 1.62 - 16.55

Wf - x,)

W f- Y J

P{0,)

51.5 56.6 53.2 52.0 55.3 51.7

16.8 22.2 18.6 17.3 20.9 17.0 18.3 16.6 19.1 16.6 18.4

54.2 43.7 50.7 53.2 46.4 53.8 51.4 54.6 49.7 55.2 51.7

52.9

51.3 53.7 51.3 53.0

[P{O f } ] - "20,

- 1.52

0.97 0.00 - 1.19 0.33 - 0.50 1.43 -0.16 - 1.18 0.97 0.95

From (2.2.41), the sum of the terms in the covariance matrix of the approximate distribution of if- zf that are constant over t is 15.5213 0

I

8.9926

(2.2.48)

[ O8.9926 0O 50.3172 0

The estimator (2.2.41) is the matrix (2.2.48) plus &i,?,,,ii.Because the row and column of V { i , - z f } associated with the intercept will always be zero, one can easily restrict the computations to 2 x 2 matrices. Of course, i,q,,ii must include the terms for the intercept. We have carried out the computations for the full 3 x 3 matrix for illustrative purposes. The estimated variances of 22, - x, and j , - y, are given in Table 2.2.3. The average increase in the estimated variance of 2, - x, due to estimating the parameters is 6.8, an increase of about 15%. The average increase in the estimated variance of 9,- y, due to estimating the parameters is 5.0, an increase of about 38%. The largest percentage increase in the estimated variance of 9, - yf is about 67% and occurs for the second observation, the observation with the largest absolute value of 2f - s. One should remember that this is a small sample and, hence, the estimation effect is relatively large. Expression (2.2.47)for the estimated variance of 9 can be written as a function of the original X values or as a function of deviations from the mean. Using deviations from the mean, we have

q { 9 } = (59.4440)11 + i(e,, - 2n- 'M;~6,,)9

where the tth row of 2 is (1, if- 70.6364) and s,, = 59.4440. The 2, -2 and the diagonal elements of q{$}are given in Table 2.2.3. The last column

2.2.

THE MODEL WITH AN ERROR IN THE EQUATION

123

of Table 2.2.3 contains 8, divided by the estimated standard error of u,. These quantities can be used in place of C, for model checking.

on

REFERENCES Anderson and Rubin (1956), Fuller (1975, 1980), Kalbfleisch and Sprott (1970), Kiefer and Wolfowitz (1956), Morton (1981a).

EXERCISES 3. (Sections 2.2.1, 2.2.2) Using the data of Table 2.2.1 and the associated error covariance matrix, estimate the parameters of the model

Estimate x, = ( x I Ixf2,xI3) , for t = 1, 2, . . . , 20 treating xI1 as fixed. Give the estimated covariance matrix for your estimates of x f I . Predict the true values treating x,, as random. Give the estimated covariance matrix of the prediction error defined in (2.2.44). 4. (Section 2.2.1) Show that the likelihood in (2.2.4) increases without bound as uee-+ 0. (Hint: Fix ueeand maximize (2.2.4).) 5. (Section 2.2.3) Using expression (2.2.45), show that the covariance matrix of the approximate distribution of (I?,, uIj) is Using this expression for cov{u*,, ulj} and expression (2.2.46), construct an alternative estimator for x,. 6. (Section 2.2.1) A random sample of 40 of the plots used in the soil moisture experiment described in Example 2.3.2 gives the sample moments

(my,,mXY,m x x ) = (0.5087,0.1838,0.6006). Assume that these data satisfy the model

Estimate ooe,uxx.and PI. Give the estimated covariance matrix for the approximate distribution of the estimators. 7. (Section 2.2.1) (a) Prove (2.2.7) from formulas (2.2.5) and (2.2.6). (b) Verify the formula for E { g : } that follows (2.2.24). (Hint: First show that x, and u, are uncorrelated under the conditions of, Theprem 2.2.1.) 8. (Section 2.2.3) Verify the formula for 6 - S that follows (2.2.37). (Hint: First show that o,, = a'Z,,a = a'T,,a under the assumptions of the model.) 9. (Section 2.2.1) The estimator of@given in (2.2.12) was obtained by maximizing the likelihood for normal x, and is a method of moments estimator for any random x,. Let Zeu= 0 for model (2.2.I).

124

VECTOR EXPLANATORY VARIABLES

(a) Show that minimizes

of (2.2.12)(or $ of (2.2.20) with S,, = EuM and S,, = 0) is the value of /I that

Q(B) =

n

1 (E: - X,/I)’ - n S ’ L b .

1=1

Does the function have a minimum for all values of X, and Zuu?Explain. (b) Show that the minimum over 6 of E {,=, [(

- X,6)’

- 6’Luu6]]

occurs with 6 = /Io,where /?’ is the population parameter, and that the minimum of the expectation is nu,,.

2.3. THE MODEL WITH NO ERROR IN THE EQUATION In this section we consider maximum likelihood estimation of the errors-invariables model when the entire error covariance structure, including u,,, is known or is known up to a scalar multiple. The model is the extension of the model of Section 1.3 to a model with a vector of explanatory variables. 2.3.1. The Functional Model We first derive the maximum likelihood estimator treating the vector of true values x, as fixed in repeated sampling. The model is

(2.3.1) for t = 1,2, . . . , n, where {x,} is a sequence of fixed k-dimensional row vectors and 8, = (e,, u,)’ is the vector of measurement errors. We can also write the defining equations (2.3.1) as Z, = z,

z,a = 0,

where z, = (y,, x,), Z, = (y, XI), and a’ = (al, a*, . .

+ a,,

(2.3.2)

. ,a k + l )= (1, -p).

-

Theorem 2.3.1. Let model (2.3.1) and the associated assumptions hold. Let 8, NI(0, C,,), where ZRE= Y,,02 and Ye, is known. Then the maximum likelihood estimators of fl and u2 are

S = ( ~ x -x IYJ- -

6 :

= (k

+1)4,

IYue),

2.3. THE MODEL WITH NO ERROR

IN THE EQUATION

where M,, = n-' E;= ZiZ, and

2 is the smallest root of

125 (2.3.3)

IM,, - AY,,J = 0.

The maximum likelihood estimator of z,, t = 1, 2,. . . , n, is

2, = Zr

-

(Y, - X,B)[(l, -&Y,,(l,

-/?)Ye,.

-)')']-'(l,

(2.3.4)

Proof. We derive the estimator under the assumption that Y E ,is nonsingular. The density for e, is

127~r,,0~1 -

exp{ - ( 2 0 ~ ) '(&,re; I&;)}.

(2.3.5)

Because the z, are fixed, the logarithm of the likelihood for a sample of n observations is log L = - 2-

1,

iogj2nr,,021 - (2a2)- 1

c ( z , - z,)rE; yzt- z,y.

r= 1

(2.3.6)

We first maximize (2.3.5) with respect to z,, t = 1,2,. . . , n, for a given / I This reduces to the least squares problem of minimizing

( X - XrS, Xr - xr)yL1(T - xrB x, - Xr)' 9

(2.3.7)

with respect to x,. It follows that the least squares estimator of z; is 2;

Ik)yi '(b?Id'] = Z; - Y,,a(a'r,,a)- 'a'z;. = (B, I k ) ' [ ( B ?

'(fly

lz;

(2.3.8) (2.3.9)

Substituting expression (2.3.8) into (2.3.5), we have

1% L = - T ' n log)2nr,,aZI - (2a2)-'

n

1 Z,a(a'Y,,a)-'a'ZI.

I =1

(2.3.10)

Therefore, the a that maximizes (2.3.6) is the vector that minimizes

n-

' ,C= I (a'Y,,a)-'a'ZiZra= (a'Y,,a)n

'a'M,,a,

(2.3.11)

'

where we have introduced the factor n - for notational convenience. By Corollary 4.A.10 of Appendix 4.A, the a minimizing (2.3.1 1) is given by

( M -~Ire,)$ ~ = 0,

(2.3.12)

where 2 is the smallest root of (2.3.3). The smallest root 2 is also the minimum value for the ratio (2.3.1 1)

I = (oilr,,a*)loil~,,a*.

(2.3.13)

126

VECTOR EXPLANATORY VARIABLES

With probability one, the rank of M,, - ITe& is k and a* is determined up to a multiple. Using a‘ = (1, -/3’), the estimator of /? is

b = W x x- ~ r u u ) - l ( M -x yATue),

(2.3.14)

4, = Z,[I - k($rEeOi)idjlreEl.

(2.3.15)

where Tee, re,,and Y,, are the submatrices of re,,and M,, - AT,, is nonsingular with probability one. See the discussion in the text below for singular Y E & . By (2.3.9), the maximum likelihood estimator of z, is Differentiating (2.3.5) with respect to a2, we obtain

and the maximum likelihood estimator of

= (k

+1 ) 4 ,

.

where 2 is defined in (2.3.13) and (2.3.3).

0’

is

(2.3.16)

0

Observe that the maximum likelihood estimators of a and z, given in Theorem 2.3.1 would be the same if u2 were known. That is, the estimators of a and z, are the same whether ZeEor Teeis known. In deriving the maximum likelihood estimators we assumed ZeEto ke nonsingular. This was an assumption of convenience. The definition of /I given by Equation (2.3.14) does not require Teeto be nonsingular, provided m,, is nonsingular. Likewise, the estimator of z, defined in (2.3.15) is the maximum likelihood estimator for singular TEE. If Oi is a consistent estimator for a, then the maximum likelihood estimator of u2 will be a consistexit estimator for (k + 1)-’a2. As in Section 2.2, maximum likelihood fails to produce consistent estimators of all parameters for the functional model. The problem is essentially a “degrees of freedom” problem. We estimate (nk k ) parameters, but there is no adjustment in the maximum likelihood estimator for this fact. The estimator (2.3.14) of /3 has considerable appeal and we shall demonstrate that it possesses desirable large sample properties. In line with our earlier results, a reasonable consistent estimator of a’ is 8’ = ( n - k)-’nI. (2.3.17)

+

2.3.

127

THE MODEL WITH NO ERROR I N THE EQUATION

In Theorem 2.3.2 we demonstrate that the estimator of /l is approximately normally distributed if the error variances are small relative to the variation in x, or (and) if the sample size is large. To accommodate the two types of limits, the sequence of estimators is indexed by v , where v = n,T,, n, is the sample size, and the error covariance matrix for a particular v is a fixed matrix multiplied by T,-'. It is assumed that the sequences {n,} and {T,} are nondecreasing in v . Henceforth, we shall suppress the subscript v on n and T. In practice, we often have an estimator of XEe,rather than knowing the matrix YE,.We give the limiting distribution of the estimators for such a situation, and the estimation of the error covariance matrix contributes a term to the covariance matrix of the approximate distribution of

b.

Theorem 2.3.2.

Let

+

k; = x,B + e,, (e,, u,)'

-

X, = x, u,, NI(0, T - 'a),

for t = 1, 2, , . . , n, where R is a fixed positive semidefinite matrix and { x t } is a fixed sequence of k-dimensional vectors. Let n 2 k 1, T 2 1, v = nT, and lM,,l > 0 and let lim Z = p,, lim M,, = M,,,

+

v+ w

v+ m

where M,, is a positive definite symmetric matrix. Let S,, be an unbiased estimator of R, where S,, is the lower right k x k portion of S,,. Let S,, be distributed as a multiple of a Wishart matrix with d, degrees of freedom, independent of XI) for all t. Let d, 2 n t , , where tl is a fixed positive number. Let ) = (Mxx - AT-'SJ1(MXy - jT1 Sue),

(x,

where 1is the smallest root of Then, for n > k

+ 1,

IM,, n(n - k)-

(2.3.18)

- I.T-'S,J = 0.

'1= F + O,(n-'l2v-

'I2

1 3

(2.3.19)

where F is a random variable distributed as Snedecor's F with n - k and d, degrees of freedom. Furthermore, as v

+

co,where

r; 1 / 2 ( p "

- p) 5 N(O, I)

and, for example, a",,= T - '( 1, - /?')a( 1, - /?')'.

128

VECTOR EXPLANATORY VARIABLES

+

+

Proof. Now, for example, M x x i j= MXxij + Mxuij Mxuji MUuij, and V{MXUij} = n-~MxxiiOuujj = (nT)-’Mxxiimuuij = O(v-’), V{MUuij}= n - ’ ~ - 2 ( m u u i i q , u j j miuij) = O(v-’T-’),

+

where is the ijth element of R,, and R,, is the lower right k x k portion of R. I t follows that Mzz = M,, ZEt+ O,(V- ‘I2),

+

where Zee= T-’R. Because 1is the minimum root of (2.3.18), we have

X = T[(1, -&te(l,

-F)’]-’(l, -F)MZz(l,

6 TM”,[U, -P)See(l,

-P)’]-’.

-b)’

(2.3.20)

The ratio on the right side of the inequality in (2.3.20) is distributed as an F random variable with n and d, degrees of freedom. Hence, X is bounde: in probability, The root is a continuous function of the elements of 8 = [(MZz,See)- (Mzz ZfC,R)] with continuous first derivatives in a region about B = 0. Because B = OP(n-’l2),it follows that

x

+

1- 1 = O,(n-’j2), (2.3.21) where the result holds for fixed n because 1is bounded in probability. From

the definition of

S,

S - /I= (Mxx - XT-’S,,)-’[Mxu - -IT-’(S,, - S,,p)] = M,’[M,, + Mu, - T-’(Se, - S,,/?) - (X - l)Zuu]+ O,(V-’) (2.3.22)

= O,(v - ‘/2),

where we have used

(X - I ) T - ’ S ~=~O , ( n - l / Z T - l Mxx - XT-’S,, = M,, + O , ( V - ” ~ ) , Mx, - (1- l)T-’(Sue- S,,P) = M,, - (X - l)& + OP(v-’). By the definitions of x and S, 19

I - 1 = TW,, - 2 m x r + B’MXXS) - 1 see- 2B’s,,

+ B’s,,B”

T[M,, - s;, - 2(S - B)’(Mx, + (S - B)’(Mxx - S,,)(S T[%”- 2(S - B)’SUU+ (S - nrLcS - n1 = S;;’[M,,- SVu- 2(8 - /I)‘(Mx, - S,,,,) (S - /I)’(Mxx - 8,,)(8 - j3)] O , ( V - ” ~ ~ - ” ~ ) ,

kJ)

-

where

s,,

+

= T-’S,,,

+

s,, = T-’(S,, - SJ),

and

.?, = T-’(l, -fl’)SEe(l,-/?’)’,

n1

2.3.

129

THE MODEL WITH NO ERROR IN THE EQUATION

By the second expression of (2.3.22)and the properties of the sample moments,

j- p = M,'M,, + o,(v-1 / 2 ~ M,, - S,, = M,, O,(V-"~T-'/~), (Mxx - I$,,)-'(Mxx - S,,,) = I + O,(V-"~T-~~~).

-

19

+

It follows that and

(B - /?)'(M,, - s,,,)($- B) = M,,M~,'M,, + O,(V-'T-"~ 1 1= CL1(A4,,,- M,,M,'M,,)

+ O,(V-'/*~-'/~ 1.

(2.3.23)

By assumption, the numerator and denominator of (2.3.23) are independent, and d p i ' & , is distributed as a chi-square random variable with d , degrees of freedom. The limiting distribution of n(n - k)-'1 follows because (n - l)cru~l(Muu - M,,M~'M,,) is distributed as a chi-square random variable with n - k degrees of freedom. If T increases without bound, then

) - fl = M,'Mx,

+ o,,(v-~/').

Because x, is fixed, [V{Mxu)]-'~2Mx,,is a N ( 0 , I) random vector for all v. Now VIM,,} = T-'n-'MxX(l, -/3')f2(1, -p)'

and r, = V{M~,'Mx,,}+ O,,(v-'T-'). Therefore, the limiting distribution of r; ' l 2 ( f i - 8) is established for increasing T. If n - l = O(v-'),we write n'/2(j -

fl) = n-'/2M