Institute of Mathematical Statistics LECTURE NOTES-MONOGRAPH SERIES
Selected Proceedings of the Symposium on Estimating...
21 downloads
983 Views
39MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
Institute of Mathematical Statistics LECTURE NOTES-MONOGRAPH SERIES
Selected Proceedings of the Symposium on Estimating Functions Ishwar V. Basawa, V.P. Godambe and Robert L. Taylor, Editors
Volume 32
Institute of Mathematical Statistics LECTURE NOTES-MONOGRAPH SERIES Volume 32
Selected Proceedings of the Symposium on Estimating Functions Ishwar V. Basawa, V.P. Godambe and Robert L. Taylor, Editors
Institute of Mathematical Statistics Hayward, California
Institute of Mathematical Statistics Lecture Notes-Monograph Series Editorial Board Andrew A. Barbour, Joseph Newton, and David Ruppert (Editor)
The production of the IMS Lecture Notes-Monograph Series is managed by the IMS Business Office: Miriam Gasko Donoho, IMS Treasurer, and James H. Sanders, IMS Business Manager.
Library of Congress Catalog Card Number: 97-077203 International Standard Book Number 0-940600-44-7 Copyright © 1997 Institute of Mathematical Statistics All rights reserved Printed in the United States of America
TABLE OF CONTENTS An Overview of the Symposium I.V. Basawa, V.P. Godambe and R.L. Taylor Estimating Functions: A Synthesis of Least Squares and Maximum Likelihood Methods V.P. Godambe
1
5
S E C T I O N 1: L I K E L I H O O D A N D R E L A T E D T O P I C S . . . 17 Partial Likelihood and Estimating Equations P. Greenwood and W. Wefelmeyer
19
Avoiding the Likelihood C.C. Heyde
35
Likelihood and Pseudo-likelihood Estimation Based on Response-Biased Observation J.F. Lawless
43
Likelihood From Estimating Functions P.A. Mykland
57
SECTION 2: GENERAL THEORY
63
Estimating Functions in Semiparametric Statistical Models . . . .
65
S. Amari and M. Kawanabe Estimating Functions, Partial Sufficiency and Q-Sufficiency in the Presence of Nuisance Parameters V. P. Bhapkar
83
Estimating Functions and Higher Order Significance D.A.S. Fraser, N. Reid and J. Wu
105
On Consistency of Generalized Estimating Equations B. Li
115
SECTION 3: QUASILIKELIHOOD
137
Extended Quasilikelihood and Estimating Equations J.A. Nelder and Y. Lee
139
Quasilikelihood Regression Models for Markov Chains W. Wefelmeyer
149
SECTION 4: APPLICATIONS TO LINEAR MODELS AND ECONOMETRICS
175
Optimal Instrumental Variable Estimation for Linear Models With Stochastic Regressors Using Estimating Functions A.C. Singh and R. P. Rao
177
On Estimating Function Approach in the Generalized Linear Mixed Model B.C. SutradharandV.P. Godambe
193
Using Godambe-Durbin Estimating Functions in Econometrics . 215 H.D. Vinod Estimating Functions and Over-Identified Models T. Wirjanto SECTION 5: APPLICATIONS TO TIME SERIES, BIOSTATISTICS AND STOCHASTIC PROCESSES
239
257
On the Prediction for Some Nonlinear Time Series Models Using Estimating Functions B. Abraham, A. Thavaneswaran and S. Veins
259
Estimating Function Methods of Inference for Queueing Parameters I.V. Basawa, R. Lund and U.N. Bhat
269
Optimal Estimating Equations for State Vectors in Non-Gaussian and Nonlinear State Space Time Series Models 285
/. Durbin
Estimating Functions in Failure Time Data Analysis /?.L. Prentice and L. Hsu
293
Estimating Functions for Discretely Observed Diffusions: A Review M. Sorensen
305
Fitting Diffusion Models in Finance
327
D.L. McLeish and A. W. Kolkiewicz SECTION 6: APPLICATIONS TO SPATIAL STATISTICS 351 Prediction Functions and Geostatistics 353 A.F. Desmond Efficiency of the Pseudo-Likelihood Estimate in a One Dimensional Lattice Gas 369 J.L. Jensen Estimating Functions for Semivariogram Estimation 5. Lele
381
SECTION 7: NONPARAMETRICS, ROBUST INFERENCE AND BOOTSTRAP 397 Estimating Covariance Matrices Using Estimating Functions in Nonparametric and Semiparametric Regression RJ. Carroll, SJ. Iturria and R.G. Gutierrez
399
Estimating Equations and the Bootstrap F. Hu and J.D. Kalbfleisch
405
Estimating Functions: Nonparametrics and Robustness P.K. Sen
417
SECTION 8: FURTHER TOPICS
437
Inference From Stable Distributions H. El Barmi and P. I. Nelson
439
Separate Optimum Estimating Function for the Ruled Exponential Family Γ. Yanagimoto and Y. Hiejima
457
Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES
AN OVERVIEW OF THE SYMPOSIUM ON ESTIMATING FUNCTIONS I. V. Basawa University of Georgia V. P. Godambe University of Waterloo R. L. Taylor University of Georgia The Symposium on Estimating Functions was held at the University of Georgia from March 21, 1996 to March 23, 1996. The Symposium was cosponsored by the Institute of Mathematical Statistics and the Statistical Society of Canada and represented continuing efforts by the two professional societies to focus special attention on some of the more prominent directions in probability and statistics. Partial funding by the University of Georgia's "State-of-the-Art" Conference Program and a National Security Agency Grant contributed to the success of the Symposium and is gratefully acknowledged. The Symposium attracted 119 registered participants from several countries including Australia, Canada, Denmark, England, Germany, Hong Kong, India, Japan, Kuwait, Sweden, and the United States of America. The program consisted of 13 sessions with 35 invited speakers and 14 contributed talks. The main theme of the Symposium can be summarized as "Statistics at a Juncture of a Synthesis." The 'likelihood function' has provided a basic methodology for the parametric inference for decades. On the other hand semi-parametric inference has been primarily based on the 'least-squares' methodology for a longer time. The unification and extension of the two methodolgies, achieved during recent years through estimating functions was the main theme of the opening talk of the Symposium by V. P. Godambe. The discussions and presentations which followed were energetic and covered a wide variety of topics: C. C. Heyde suggested avoiding the likelihood; J. A. Nelder presented extensions to quasilihood; N. Reid discussed higher order significance; J. Durbin discussed applications to nonlinear state space time series; J. D. Kalbfleish presented bootstrap using estimating functions. There appeared to be a consensus at the Symposium that Statistics was at
2
BASAWA, GODAMBE AND TAYLOR
a juncture of a synthesis of the two of its main methodologies, namely the likelihood and least squares, brought about by estimating functions. Many of the major results which were presented at the Symposium are summarized in these selected proceedings. Specifically, the papers are organized into eight sections. A general historical paper by V. P. Godambe precedes the following sections: 1. 'Likelihood' with papers by P.Greenwood & W. Welfelmeyer, C. C. Heyde, J. F. Lawless, and P. A. Mykland. Topics which are central to this section include likelihood, partial likelihood, pseudo-likelihood and other alternatives to the likelihood. 2. 'General Theory' with papers by A. Amari & M. Kawanabe, V. P. Bhapkar, D.A.S. Eraser, N. Reid & J. Wu, and B. Li. Papers in this section deal with problems on estimating functions in semiparametric models, nuisance parameters, higher order significance and consistency. 3. 'Quasilikelihood' with papers by J.A. Nelder & Y. Lee, and W. Wefelmeyer. Extended quasilikelihood and regression models for Markov chains are discussed in this section. 4. 'Applications to Linear Models and Econometrics' with papers by A. C. Singh & R. P. Rao, B.C. Sutradhar & V. P. Godambe, H. D. Vinod, and T. Wirjanto. Papers in this section address problems in instrumental variable estimation, generalized linear mixed models, Godambe-Durbin estimating functions in econometrics and over- identified models. 5. 'Applications to Time Series, Biostatistics and Stochastic Processes' with papers by B. Abraham, A. Thavaneswaran & S. Peiris, I. V. Basawa, R. B. Lund & U. N. Bhat, J. Durbin, R. L. Prentice & L. Hsu, M. Sorensen, and D. L. McLeish & A. W. Kolkiewicz. Applications in this section are in nonlinear state space models, prediction, failure time data analysis, queueing parameter estimation, diffusion processes and models in finance. 6. 'Applications to Spatial Statistics' with papers by A. F. Desmond, J. L. Jensen, and S. Lele. Prediction in geostatistics, pseudo-likelihood estimation for lattice models and semivariogram estimation are covered in papers in this section. 7. 'Nonparametrics, Robust Inference and Bootstrap' with papers by R. J. Carroll, S. J. Iturria & R. G. Gutierrez, F. Hu & J. D. Kalbfleisch, and P. K. Sen. These papers contain results for estimating covariance matrices, nonparametrics and robustness and bootstrap techniques. 8. 'Futher Topics' with papers by H. El Barmi & P. I. Nelson, and T. Yanagimoto & Hiejima. The two papers in this final section are concerned with inference from stable distributions and inference for the ruled exponential family. The editors of these selected proceedings of 29 papers are very grateful to numerous referees who very carefully and critically reviewed all papers which
SYMPOSIUM OVERVIEW
3
were submitted for publication in the proceedings. Also, the willingness of individual authors to limit the number of pages of their articles helped produce this volume in the IMS Lecture Notes Series. Special thanks go to Connie Durden for the preparation of this Volume. Ms. Durden worked tirelessly and patiently with the various authors in securing software files of their papers, providing uniformity of margins, spacing and similar editorial changes which greatly enhanced the general appearance of this volume.
Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES
ESTIMATING FUNCTIONS: A SYNTHESIS OF LEAST SQUARES AND MAXIMUM LIKELIHOOD METHODS V.P. Godambe University of Waterloo ABSTRACT The development of the modern theory of estimating functions is traced from its inception. It is shown that this development has brought about a synthesis of the two historically important methodologies of estimation namely, the 'least squares' and the 'maximum likelihood'. Key Words: Estimating functions; likelihood; score function.
1
Introduction
In common with most of the historical investigations, it is difficult to trace the origin of the subject of this conference: 'Estimating Functions'. However, in the last two centuries clearly there are three important precursors of the modern theory of estimating functions (EF): In the year 1805, Legendre introduced the least squares (LS) method. At the turn of the last century Pearson proposed the method of moments and in 1925 Fisher put forward the maximum likelihood (ML) equations. Of these three, the method of moments faded out in time because of its lack of any sound theoretical justification. However the other two methods namely the LS and the ML even at present play an important role in the statistical methodology. These two methods would also concern us in the following. The LS method was justified by what today is called the Gauss-Markoff (GM) theorem: The estimates obtained from LS equations are 'optimal' in the sense that they have minimum variance in the class of linear unbiased estimates. This was a finite sample justification. At about the same time Laplace provided a different 'asymptotic justification' for the method. Fisher justified the ML estimation, for it produced estimates which are asymptotically unbiased with smallest variance. This left open the question, is there a finite sample justification for the ML estimation corresponding to the GM theorem justification for the LS estimation? The modern EF theory provided such a justification. According to the Όptimality criterion' of the EF theory, the score function (SF) is 'optimal'.
6
2
GODAMBE
SF Optimality
To state the just mentioned result formally we introduce briefly some notation. Let X = {x} be the sample (observations) space and a class of possible distributions (densities) on X be given by {/( |0),0 G Ω}, Ω being the parameter space, which we assume here to be the real line. If the function / is completely specified up to the (unknown) parameter 0, f{ \0) is called a parametric model. For this model the (score function) SF = d\ogf( \θ)/dθ. Any real function of x and 0 say g(x,0) is called an estimating function, (EF). It is said to be unbiased if its mean value for 0 E Ω, is zero; £g = 0. Further, for reasons which would be clear later, corresponding to every EF g we define a standardized version g/{£{τ^)}uu
Now in a class Q = {g} of
unbiased estimating functions, g* is said to be 'optimal' if the variance of the standardized EF g, is minimized for g — g*: ε
^)2/{£(%)}2
< ε(g)2/{εφ}2,
θen,9eg.
(2.1)
SF Theorem (Godambe, 1960). For a parametric model /( |0), granting some regularity conditions, in the class of all unbiased EFs, the optimal estimating function is given by the SF i.e. g* = dlogf{.\θ)/dθ. The optimality of the SF given by the above Theorem should be distinguished from the optimality of the LS estimates based on the GM theorem. The SF optimality (though with some additional assumptions implies asymptotic optimality of the ML estimate) is essentially optimality of the 'estimating function' while the LS optimality is optimality of the 'estimate'. The concept underlying optimality criterion of the EF theory became more vivid and compelling in relation to the problem of nuisance parameters.
3
Conditional SF Optimality
Now let the parameter 0 consist of two components 0\ and #2, θ = (#i,#2) and the parametric model be /( |0i,#2) where θ\ is real and 02 is a vector; θ £ Ω, θ\ G Ωi, 02 £ Ω2 and Ω = Ωi x Ω2. Further suppose we want to estimate only 0\ (the interesting parameter) ignoring 02 (the nuisance parameter). How to proceed? To this question the ML estimation provides no satisfactory answer. If 0\ and §2 are jointly ML estimates for θ\ and 02, as is well known, the estimate 0\ can be inconsistent (unacceptable) in case the dimensionality of the parameter 02 goes on increasing with the number of observations (cf. Neyman-Scott, 1948). The EF theory, for the present
ESTIMATING FUNCTIONS situation implies, restricting to that part of likelihood function which is governed by the interesting parameter θ\ only. Formally for the parametric model /( |0i,02), let Q\ be the class of all unbiased EFs g(x,θ\), that is functions of x and 0χ only: Gi = {9
9 = s ( M i ) , S(g) = 0, Θ e Ω}.
Further let t be a complete sufficient statistic for the parameter 02, f° r every fixed 0χ. Assuming the statistic t is independent of the parameter θ\ we have Conditional SF Theorem (Godambe, 1976). Granting some regularity conditions, in the class of EFs Q\, the 'optimal' EF g* is given by the conditional SF i.e. g* = dlog/( |t;0i)/30i. Note in the above theorem the definition of optimality is obtained from (2.1) just by replacing in it Q by Gi and consequently S(dg/dθ) by ε(dg/dθι). That is the criterion of optimality is unconditional. In the case of the Neyman-Scott example, unlike the ML estimate 0, the equation 'conditional SF = 0' provides a consistent estimate of θ\. Further the EF optimality criterion suggests a definition of 'conditional SF' in case the statistic t depends on the parameter θ\. If ί(0io) is the value oΐt at θ\ = 0χo then we define the conditional SF by g* where g* = {dlogf(.\t(θ10),θ1,θ2)/dθι}θl0=Θl.
(3.1)
This definition is motivated as follows. The EF g* in (3.1) £ Q\ though it depends on 02- It further is 'optimal' in Q\ though only locally at 02 (Lindsay, 1982). Unlike the previous situation, when the sufficient statistic t, was independent of 0χ, now no universally optimal g* (i.e. for all 02 £ Ω2) exists in Q\. Further though the EF g* in (3.1) depends on 02, it is orthogonal to the marginal SF of the sufficient statistic ί, hence the substitution of an estimate 02 derived from the latter, in the former would still leave the former nearly optimal for large samples, (Lindsay, 1982; Godambe, 1991; Small and McLeish, 1994; Liang and Zeger, 1995). The equation
would provide a (nearly optimal) consistent estimate of θ\. Note in the forgoing discussion, conditioning is used just as a 'technique' to obtain (unconditionally) 'optimum' EFs; but it is not used as a principle of inference. In fact, without invoking any conditioning at all, Godambe and Thompson (1974) established, in case of the normal distribution iV(0χ,02), the optimality of the EF (s 2 - 02), for the interest parameter 02, ignoring the nuisance parameter θ\. How this (unconditional) optimality leads to a very 'flexible conditioning' will be discussed later.
GODAMBE For a general perspective on the topic of conditioning and optimality we refer to Small and McLeish (1988), Lindsay and Waterman (1991) and Lindsay and Li (1995). Lloyd (1987) and Bhapkar (1991) have given results concerning optimality of 'marginal SF' under 'conditional completeness'. From the above discussion it is clear that the EF theory has corrected a major deficiency of the ML estimation in case of the nuisance parameters. Some earlier references in respect of the nuisance parameters are Bartlett (1936), Cox (1958), Barnard (1963), Kalbfleisch and Sprott (1970), BarndorffNielsen (1973) and others. Some of these authors tried to obtain conditions under which the marginal distribution of t does not contain any information about θ\ the parameter of interest. As we have seen the optimality criterion of the EF theory yields such a condition in terms of 'completeness of the statistic t\ Though not universally applicable (as none can be, I suppose) it by now has been commonly used for its mathematical manageability. It also carries with it greater conviction for it is derived from an optimality criterion which has proved to be fruitful very generally. In the following we would show that the EF theory, just as it corrected ML estimation, also corrects some major inadequacies of the LS estimation and the GM theorem.
4
Quasi-Score Function
We now replace the abstract (observation) sample in the discussion by n real variates X{ : i = 1, ...,n which are assumed to be independently distributed with means μi(θ) and variances Vi(θ) (μι and v% being some specified functions of θ) i = 1, ...,n. For simplicity let θ be a scalar parameter. Initially we consider the special case where μι are linear functions of θ and V{ are independent of θ. Here the LS equation is given by Σι{xi — μ%){-^)hi — 0 uu The solution of the equation, as said before, according to GM theorem, has smallest variance in the class of all linear unbiased estimates of 0; hence is 'optimal'. The estimating function Σ{χi ~ tJLi){-^E')lvi ιs a l s o 'optimal' uθ according to criterion (2.1), in the class of all EFs of the form n
9 = Σ(xi
~ μ%)(H
(4.1)
where aι can be any arbitrary functions of θ. (Actually here we minimize 8(g2) subject to holding ε(dg/dθ) = const.. This will explain the standardization of EF mentioned earlier.) Note this EF optimality implies more than the GM optimality, for the solutions corresponding to all the equations g = 0 include not only all linear unbiased estimates of θ but many more.
ESTIMATING FUNCTIONS Now let the means μ* and variances V{ be arbitrarily specified functions of θ. Here the LS equation is given by 3 + B = 0 where g
=
, n ,„. 1
..jdμi/dθ) υ%
and
(4.2)
Clearly in (4.2), S{g) = 0 and £(B) = ΣΐdlogVi/dθ. Note for large n, (/n) ~ 0 while ( £ / n ) could still be very large. Hence because of the bias term (B) the LS equation Ίj + B = 0 would generally lead to an inconsistent estimate. On the other hand according to the optimality criterion (2.1) of the EF theory, in the class of EFs given by (4.1) for different functions di(θ), i — 1,..., n, g given by (4.2) is Optimal'. Generally the equation ~g = 0 would lead to a consistent solution. (Here GM theorem cannot be of any avail for the solution of g = 0 would generally lead to a biased estimate.) For reasons to be explained soon, we would call the estimating function ~g a quasi-score function, (Quasi-SF). Interestingly the EF optimality of the quasi-SF ~g was first established in a wider setting of discrete stochastic processes with a martingale structure. Quasi-SF Theorem (Godambe 1985). If μι and V{ denote the means and variances of X{ conditional on the past observations #i_i,...,α;o i.e. μι = μi(0, #0, •••, Xi-i) and Vi — Vi(θ, x0,..., x;_i) for i = 1,..., n then _ λdμi/dθ 9 = 2^{x% - μ%) •—
(4-3)
Here g is the optimal EF in the class of the EFs given by (4.1) where aι are functions of a?<-i, •••, ^o in addition to θ. Among precursors to the EF g in (4.3) above are included the following: Durbin (1960) gave a GM theorem analogue for linear time series model. Klimko and Nelson (1978) obtained conditional LS equations. Kalbfleisch and Lawless (1983) suggested a special case of g for Markov models. For further generalizations of the EF optimality results, relating to g in (4.3) we refer to Godambe (1985), Godambe and Heyde (1987), Godambe and Thompson (1989). Returning back for simplicity to the case where the variates x^ i = 1,.., n are independently distributed, we summarize important properties of the EF g given by (4.2). The 'optimality' of g is for the semi-parametric model defined by means μ%{θ) and variances Vi(θ). As a special case when μ; is linear in θ and V{ is independent of θ the EF optimality of ~g implies the GM optimality of the LS estimates. In this special case, if the underlying distribution
10
GODAMBE
is normal, the LS estimates coincide with the ML estimates. Generally, for the exponential family distributions the SF coincides with the optimal EF g given by (4.2). Even outside the exponential family of distributions, ~g 2 2 satisfies a very general property of the SF; ί[SF) = -S(dSF/dθ) simi2 2 larly we have E(g) = —S(dg/d9) . Further if SF denotes a generic 'score function' for the class of distributions consistent with the semi-parametric 2 2 model mentioned above then S(g - SF) < E(g - SF) , (the expectation being taken w.r.t. the distribution that corresponds to the SF), for all the EFs g given by (4.1). These are the properties which justify the previously introduced term ' quasi-SF' for ~g. (Even before the EF optimality of g was discovered the term quasi-likelihood was commonly used in the literature on generalized linear models, McCullagh and Nelder 1983, 1989). As we have seen previously the EF theory corrected a major deficiency in ML estimation relating to nuisance parameters. The above discussion points to yet another accomplishment of the EF theory. It brought about, via quasiscore function g, a kind of synthesis of two historically distinct methods of estimation: LS for semi-parametric models and ML for parametric models. The same criterion of the EF optimality namely (2.1) is satisfied in case of the latter by the SF and in case of the former by the quasi-SF ~g in (4.2). Only the classes of competing EFs are different. They are taken appropriate to the model (see Godambe and Thompson, 1989 Appendix). Of course, the forgoing discussion also shows that the quasi-SF g does provide, not only a unification (of the two methods LS and ML) but much more. It provides a generalization to deal with problems outside the scope of both LS and ML methods. As a further contribution of the EF theory to statistics, below we briefly outline a very 'flexible conditioning' that the theory permits and the consequent incorporation of the Bayesian factor within its methodology.
5
A Generalization
It was mentioned earlier that within the framework of martingales and corresponding filtering, the EF theory suggested use of weighted conditional least square estimation, on grounds of its optimality property. But to deal with general spatial processes one needs more flexible conditioning than used before; this was provided by Godambe and Thompson (1989): Let as before X = {x} be an abstract sample space and T = {F} be a class of distributions on X. Further let θ be a real parameter, defined on T\ {Θ(F),F € T) = Ω. Now suppose hj is a real function on X x Ω and Xj a specified partition (or a σ-field generated by a partition) of X such that Xj) = o,
j = ι,...,k.
(5.i)
11
ESTIMATING FUNCTIONS
The functions /ij, j = l,...,fc are called the elementary EFs; they are not exhaustive. Their choice is determined by the problem at hand. Now suppose the elementary EFs, h\,...,hk are mutually orthogonal (Def. Godambe and Thompson 1989) and the class of underlying distributions T satisfy certain conditions. Then in the class of all EFs g of the form (5.2) 3=1
where qj are some real functions o n ί x ί ί which are measurable on Xj, j = 1,..., k the 'optimal' one is given by
3=1
where q*j = {ε(dhj/dθ\Xj)}/{ε(hj\Xj)}. Here the criterion of optimality as always is unconditional, given by (2.1) for a real parameter θ (or its appropriate version if θ is a vector); the expectation is taken with respect to Fef. Up to the above results the EF theory was 'restricted' to the classical setup where distributions on the sample space X for some fixed values of the parameters are considered. But the formalism of the EF optimality criterion is flexible enough and the just mentioned 'restriction' can be set aside if we know something about the prior distribution of 0; for instance its mean (0Q) and variance (VQ). Under such Bayesian setup the only changes that are required to be done are as follows: (i) In (5.1) now Xj is not necessarily a partition of just the sample space X, but it can be a partition of X x Ω, Ω as before being the parameter space, (ii) Some elementary EFs hj, j = l,...,fc can now be functions exclusively of the parameter θ. (in) All expectations in the optimality criterion (2.1) are now with respect to the joint distributions of (x,θ) (and not as before with respect to distributions of x given θ). Following is an illustration. Let the partitions of the sample space Xj and the elementary estimating functions hj, j = l,...,fc be the same as in (5.1). Further, as suggested before, let the mean value (0o) and the variance (v$) of the prior distribution of θ be known. Now, to the set of elementary estimating functions h\,...,hk we add one more, namely hk+ι = θ — ΘQ. In this case the optimal EF is given by g* + (θ — ΘQ)/VQ, where g* is the same as in (5.3). Similarly now the quasi-SF ~g in (4.2), which was obtained under the assumption '0 is fixed' will now have to be replaced by ~g — (θ — ΘQ)/VQ (Godambe, 1994). The 'optimality' of the EF given by the derivative of the logarithm of the posterior density was established in a 'parametric setup' by Ferreira (1982) and Ghosh (1993). Naik-Nimbalkar and Rajarshi (1995) have established some optimality results in 'semi-parametric Bayesian setup'.
12
6
GODAMBE
Other Topics
Now, following are a few remarks (possibly only tangential) about the likelihoods: empirical, partial, profile, quasi and the like. Basically when the likelihood function is precisely known, with no nuisance parameters, likelihood ratio test is 'optimal' in the conventional sense of the term. Also the SF satisfies the EF criterion of Όptimality'. Now the various likelihoods, empirical partial, quasi just mentioned, try to 'approximate' the underlying (true, precise) likelihood in situations of nuisance parameters and/or of semiparametric models. In similar situations EF theory tries to 'approximate' the (true) underlying SF. However, unlike the former, the latter 'approximation' can be assessed with a plausible finite sample criterion. Suppose g(x, θ) is a real function of the sample x and the parameter of interest θ such that the expectation £ (g) = 0 for all possible underlying distributions F i.e. for F G T. Let further SF be a score function corresponding to F in T. Then the finite sample criterion of assessing the approximation g for SF is given by £{g - SF)2, for all F £ T. This criterion as said before leads to the Όptimality' criterion (2.1) of the EF theory. As I have previously shown optimal or approximately optimum EFs are found in many practical problems and in fact by now they are in common use. Now while optimum EFs and approximations thereof can provide a handy instrument for constructing confidence intervals and related tests cf. Rao's test (Rao 1947, Basawa 1991), for some other problems some kind of 'approximate likelihood' would be more handy. I think, to be safer, construction of such approximate likelihoods should be tied to the optimum EFs, whenever possible. It is good to note already a strong trend in that direction (Qin and Lawless, 1994). An often asked question (cf. Liang and Zeger 1995) is how does the EF optimality relate to the properties of the corresponding estimate? How good is the estimate? Usually the answer is given in terms of the 'error' of the estimate. Now this 'error' is somewhat of an involved concept. Certainly, error is not just a square root of an arbitrary (unbiased or nearly so) estimate of variance. However for a parametric model the concept is clear. The error is derived from the conditional (or the natural estimate of) variance of the SF. Thus error is the inverse of the square root of observed Fisher information (Efron and Hinkley, 1978). This methodology is formalized and extended by the EF theory. Consider the confidence intervals, θ ± const (error), where the estimate θ is obtained from the unbiased estimating equation g(θ) = 0. Here a more direct way of obtaining the confidence intervals is by inverting the distribution of the standardized version (cf. Godambe 1991, eq. 40) of the EF g around θ. These intervals, compared to former ones, are easier to compute. Also if g is the optimal EF, the corresponding intervals are shortest compared to that of any other unbiased EF, (Godambe and Heyde 1987).
13
ESTIMATING FUNCTIONS
The standardizing factor of the EF g directly leads to the computation of the 'error' for the estimate θ (Godambe, 1995). For important previous review articles on the subject we refer to Heyde (1989) and Godambe and Kale (1991). The present review highlights some more recent developments and presents older results with different emphasis and interpretations. A further reference along this line is Desmond (1997).
References Barnard, G.A. (1963). Some logical aspects of the fiducial argument. J.R. Statist Soc. B, 25, 111-114. Barndorff-Nielsen, O.E. (1973). On M-ancillarity. Biometrika 60, 447-455. Bartlett, M.S. (1936). The information available in small samples. Proc. Camb. Phil. Soc, 34, 33-40. Basawa, I.V. (1991). Generalized score tests for composite hypotheses. Estimating functions, (ed. V.P. Godambe), Oxford Univ. Press, Oxford. 121-131. Bhapkar, V.P. (1991). Sufficiency, ancillarity and information in estimating functions. Estimating Functions. (Ed. V.P. Godambe), Oxford Univ. Press, Oxford. 240-254. Cox, D.R. (1958). Some problems connected with statistical inference. Ann. Math. Statist. 29, 357-372. Desmond, A.F. (1997). Optimal estimating functions, quasi-likelihood and statistical modelling (with discussion). J. Stat. Plan. Inf. 60, 77-121. Durbin, J. (1960). Estimation of parameters in time series regression models. J. Roy. Statist. Soc. B, 22, 139-153. Ferreira, P.E. (1982). Multiparametric estimating equations. Ann. Stat. Math. 34, 423-431. Fisher, R.A. (1925). Theory of statistical estimation. Proc. Cambridge Phil. Soc. 22, 700-706. Ghosh, M. (1990). On a Bayesian analog of the theory of estimating functions. C.G. Khatri Memorial Volume of Gujard Statistical Review, 17A, 47-52. Godambe, V.P. (1960). An optimum property of regular maximum likelihood estimation. Ann. Math. Statist. 31, 1208-1212. Godambe, V.P. (1976). Conditional likelihood and unconditional optimum estimating equations. Biometrika, 63, 277-284. Godambe, V.P. (1985). The foundations of finite sample estimation in stochastic processes. Biometrika 72, 419-428.
14
GODAMBE
Godambe, V.P. (1991). Orthogonality of estimating functions and nuisance parameters. Biometrika 78, 143-151. Godambe, V.P. (1994). Linear Bayes and optimal estimation. Tech. Report STAT-94-11, University of Waterloo. Godambe, V.P. (1995). Discussion of the paper, 'Inference Based on estimating functions in the presence of nuisance parameters' by Liang, K.Y. and Zeger, S.L. Statistical Science 10, 173-174. Godambe, V.P. and Heyde, C.C. (1987). Quasi-likelihood and optimal estimation. Int. Stat. Rev. 55, 231-244. Godambe, V.P. and Kale, B.K. (1991). Estimating functions: an overview. Estimating Functions. (Ed. V.P. Godambe), Oxford University Press, Oxford. 1-20. Godambe, V.P. and Thompson, M.E. (1974). Estimating equations in presence of nuisance parameters. Ann. Stat. 2,568-571. Godambe, V.P. and Thompson, M.E. (1989). An extension of quasi-likelihood estimation (With Discussion). J. Stat. Plan. Inf. 22, 137-172. Heyde, C.C. (1989). Quasi-likelihood and optimality of estimating functions: some current unifying themes. Bull. Int. Stat. Inst. Book 1, 19-29. Kalbfleisch, J.D. and Sprott, D.A. (1970). Applications of likelihood methods to models involving large number of parameters. (With Discussion). J.R. Statist. B, 32, 175-208. Kalbfleisch, J.D., Lawless, J.F. and Vollmer, W.M. (1983). Estimation in Markov models from aggregate data. Biometrics 39, 907-919. Klimko, L.A. and Nelson, P.I. (1978). On conditional least squares estimation for stochastic processes. Ann. Statist. 6, 629-642. Legendre, A.M. (1805). Nouvelles methodes pour la determination des orbites des cometes. Paris: Courcier. Liang, K.Y. and Zeger, S.L. (1995). Inference based on estimating functions in the presence of nuisance parameters. (With Discussion). Statistical Science 10, 158-195. Lindsay, B. (1982). Conditional score functions: some optimality results. Biometrika 69, 503-512. Lindsay, B. and Waterman, R.P. (1991). Extending Godambe's method in nuisance parameter problems. Proceedings of a Symposium in honour of Prof. V.P. Godambe. University of Waterloo, 1-43. Lindsay, B.G. and Li, B. (1995). Discussion of the paper, 'Inference based on estimating functions in the presence of nuisance parameters' by Liang, K.Y. and Zeger, S.L. Statistical Science 10, 175-177. Lloyd, C.J. (1987). Optimality of marginal likelihood estimating equations. Comm. Stat Theory and Meth. 16, 1733-1741. McCullagh, P. and Nelder, J.A. (1983, 1989). Generalized linear models (1st and 2nd editions). Chapman and Hall, London. Naik-Nimbalkar, U.V. and Rajarshi, M.B. (1995). Filtering and smoothing via estimating functions. J. Amer. Statist. Asso. 90, 301-306.
ESTIMATING
FUNCTIONS
15
Neyman, J. and Scott, E.L. (1948). Consistent estimates based on partially consistent observations. Econometrika, 16, 1-32. Qin, J. and Lawless, J.F. (1994). Empirical likelihood and general estimating equations. Annals of Statistics 22, 300-325. Rao, C.R. (1947). Large sample tests for statistical hypotheses concerning several parameters with applications to problems of estimation. Proc. Camb. Phil. Soc. 44, 50-57. Small, C. and McLeish, D.L. (1988). The theory and applications of statistical inference functions. Lecture Notes in statistics No. 44, Springer Verlag. Heidelberg, New York, London. Small, C. and McLeish, D.L. (1994). Hilbert space methods in probability and statistical inference. John Wiley and Sons, Inc. New York.
19
Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES
PARTIAL LIKELIHOOD AND ESTIMATING EQUATIONS Priscilla E. Greenwood University of British Columbia and Wolfgang Wefelmeyer University of Siegen Abstract Consider a regression model for discrete-time stochastic processes, with a (partially specified) model for the conditional distribution of the response given the covariate and the past observations. Suppose we also have some knowledge about how the parameter of interest affects the conditional distribution of the covariate given the past. We assume that these two model assumptions give rise to two martingale estimating functions, and determine an optimal combination. We indicate for the case of jump processes how our result carries over to continuous time. The resulting estimators are efficient.
1 Introduction Suppose we know something about how the parameter of interest in a regression model appears both in the conditional distribution of the response given the covariate, and in the distribution of the covariate. How can we exploit this knowledge? Let us illustrate our approach in the case of independent and identically distributed observations (Xj, Y;), with Xι the covariate and Yi the response. In a regression model one usually specifies the conditional distribution of Y given X, either fully, by a parametric model, or partially. An example of a partial specification is a model for the conditional mean of Y given X, say E(Y|X) = ΰX. More generally, we specify a function ## (X, Y) such that E{gϋ{X,Y)\X) = 0. In the example, gΰ(X,Y) = Y -ϋX. We assume a similar partial specification of the distribution of the covariate X, say Έg*$(X) — 0. The two functions g# and g*$ give rise to two estimating equations
fχYi)=0,
£>*(*<) =0. t=l
supported by NSERC, Canada.
20
GREENWOOD AND WEFELMEYER
By the usual Taylor expansion argument, their solutions ϋ = ϋn are asymptotically normal with variances 7 " 1 and I~x, respectively, where
Since g${Xi, Y%) and g*$(Xi) are uncorrelated, the combined estimating equation i)) = 0 leads to an estimator with asymptotic variance
Applying the Schwarz inequality to the denominator, one sees that this variance is minimized for E
E
opt _ opt _ %
opt _ flti9 W
Έy
-
The minimal asymptotic variance is (7 + I*)~ι. The weights wopt and w*pt depend on ϋ and, in general, also on other features of the distribution of (X, Y). In the estimating function, they must be replaced by estimators ώ£ pt and ώ*n\ say by using empirical estimators for the distributions involved. This does not change the asymptotic variance (/ + I*)" 1 . Can we do better than using the combined estimating equation? Note that we can multiply 'g^(X^Y) by a function w(X) of X and still have conditional expectation zero, E(w(X)gϋ(X,Y)\X)=0. This leads to new estimating equations iM) + w*g*ϋ(Xi)) - 0
(1.1)
1
with asymptotic variance (Iw + A ) " , where (E(w(X)E(g'ΰ(X,Y)\X)))2 Applying again the Schwarz inequality, one sees that l^ is maximized by
The weight again depends on ϋ and, in general, also on other features of the distribution of (X, Y) and must be estimated, say by using appropriate
21
PARTIAL LIKELIHOOD
nonparametric estimators for the conditional expectations involved. With p # ° (X) denoting such an estimator, we arrive at the estimating equation = 0.
(1.2)
The asymptotic variance of the estimator corresponding to this equation is 1 (7 + 1 * ) - with 2 (E(g'ϋ(X,Y)\X)) 7 _ 2
E(gύ(X,Y) \X) ' By the Schwarz inequality (2.10) below, / is strictly larger than / unless both conditional expectations do not depend on X. For the example given above, g#(X, Y) = Y-ΰX, we have Έ(g'ΰ(X, Y)\X) = -X and E{gϋ{X,Y)2\X) = E((F - ϋX)2\X), the conditional variance of Y given X. The estimating equation (1.2) is not only optimal among estimating equations (1.1) but even efficient among all (regular) estimators as long as we do not impose additional restrictions on the distribution of (X, Y) which involve i?. Let us give a sketch of the argument, referring to Bickel, Klaassen, Ritov and Wellner (1993) for an account of the concepts involved. The model is described by all distributions p(dx, dy) = p*(dx)p(x, dy) of (X, Y) such that Jp(x,dy)gΰ(x,y)=E(gΰ{x,Y)\x) p*{dx)g*i){x) =Eg*#
/•
= 0 for all x,
(1.3)
= 0
(1.4)
if ΰ is true. Introduce a local model by perturbing p and p* as p(x, dy)(l + n~ι/2uh{x,y)) and p*(dx)(l + n~ιί2uh*(x)) such that the two conditions (1.3) and (1.4) hold (approximately) with ϋ replaced by ϋ + n~ιl2w.
I p{x, dy) (l +n~
ι/2
J
gϋ+n-i/2u(x, y) = 0 for all x,
uh(x,y))
p*(dx) (l + n-^uh^x))
g^ΰ+n-i,2u(x)
= 0.
Then h and /ι* must fulfill
j p{x,dy)(h{x,y)g {x,y)+g' {x,y)) ΰ
ϋ
j p*{dx){K(x)g*ϋ(x)+g'^{x))
= 0 for all x,
= 0.
The perturbed p is approximately p{dx,dy) ( l + n~1/2u (h(x,y) + M
(1.5)
(1.6)
22
GREENWOOD AND WEFELMEYER
This means that the tangent space in the sense of Bickel et al. (1993, p. 50, Definition 2) consists of the functions u(h + h*) with h and /&* fulfilling (1.5) and (1.6). We view the parameter ϋ as a function of p and determine its canonical gradient in the sense of Bickel et al. (1993, p. 58). This is a function ύ in the tangent space such that nι'2{ϋ
+ n-ι'2u
-ϋ) = u = uEύ(h + h*)
for all h, K fulfilling (1.5), (1.6).
According to Bickel et al. (1993, p. 63, Theorem 2B, and p. 65, Theorem 1A), an estimator ϋn is regular and efficient if and only if ι/2
n 0n
- 0) = n" 1 / 2 Σ *(**, γi) + OP(1).
(1.7)
In particular, a lower bound for the asymptotic variance of regular estimators of ΰisEi/2.
Since the tangent space is generated by the aίfine space of functions h + h* with h and h* fulfilling (1.5) and (1.6), we can write the canonical gradient as v = (Έ(s + s*)2)~ (s + 5*), where 5 + s* is the optimal score function, minimizing E(h+h*)2 over all h and /ι* fulfilling (1.5) and (1.6). In particular, the lower bound for the asymptotic variance of regular estimators can be written 1/E(s + s*)2. The function s + s* is characterized by E(s + θ*)(Λ + K) = E(s + s*)2
for all Λ, K fulfilling (1.5), (1.6).
Since h and h* are orthogonal, this is equivalent to ESh = E5 2
for all h fulfilling (1.5),
Es*/ι* = Es 2 for all K fulfilling (1.6).
One easily checks that the solution is
By the usual Taylor series argument, the solution ϋ = ΰn of the optimal estimating equation (1.2) is seen to fulfill
i=l
By the characterization (1.7), this means that this estimator is efficient. In Sections 2 and 3 we show how the calculation of the optimal estimating function carries over to ergodic discrete-time stochastic processes and jump processes, respectively. Efficiency also carries over, but we will not give the details. All results extend immediately to vectors ϋ and vector-valued functions g# and g*$. We do not give precise regularity conditions for our results.
23
PARTIAL LIKELIHOOD
2
Discrete-time stochastic processes
Suppose we observe a stochastic process (Xi, Yi) at times i = 1,..., n. The law of the process is determined by the conditional distributions pi(dx,dy) of (Xi,Y{) given the past observations. Here and in the following, we suppress the dependence of pi and similar objects on the past, (AΊ, Yi),..., (X -i, Yi-ι). As in the i.i.d. case considered in Section 1, we describe a regression model by (partial) specifications of (1) the conditional distribution of the response given the present value of the covariate and now also the past observations, and of (2) the conditional distribution of the covariate, now also given the past. We factor pi into marginal and conditional, Pi(dx, dy) = p*i(dx)pi(x, dy),
(2.1)
and specify two functions g^{x^y) and g*iΰ(x), possibly depending on the past, such that EX9iu = I Pi{xidy)giΰ(x,y)
= 0 for all x,
E*ig*iu = / P*i(dx)g*iΰ(x) = 0. They give rise to estimating equations
How can we combine them in an optimal way? Our result holds for geometrically ergodic processes and under appropriate smoothness and moment conditions which can be seen from the sketch of the proof. Result 1. From estimating equations of the form Σ (Wi(Xi)giϋ{XuY^
+ w.ig*n>(Xi)) = 0,
(2.2)
i=l
an estimator with minimal asymptotic variance is obtained using weights which are consistent estimators of Wi(Xi) w«
= Έ^/Έf^,
(2.3)
= KisUϋβ*i9Ϊiΰ
(2 4)
The estimator is asymptotically normal. Its asymptotic variance is the limit
24
GREENWOOD AND WEFELMEYER
Sketch of proof. To simplify the notation, we introduce w = (w,w*) and gφ = (g#,(7*tf), and write the estimating equation (2.2) as w
*i9*iϋ = °
Let # be a solution. Under appropriate differentiability conditions, a Taylor expansion gives 0
= Σ wi9a * Σ «w» + (*-*) Σ «w«
Then /A
Conditionally on the past, the martingale increments w^Xij'g^Xi^Yi) i) are orthogonal:
— /
ί-
-
I
I
J
and
Introduce an inner product (v,w) = \ J Έ/*iϋiWi + / ^ v * with corresponding norm ||w|| 2 = (w,w). Interpret products υw of vectors componentwise,
Consider first the numerator in (2.7). With (2.8), the predictable quadratic variation of Σwigi$ is (w2,υ) = Huw1/2!!2, where ϋi(x) = Ts Consider now the denominator in (2.7). The compensator of ΣtUip^ (tϋ,m), where rΠi(x) = Έ*g'iΰ, m*i = E^gliϋ.
ιs
Since Σwιg'iϋ — (w,m) is a martingale, ^ (ΣVt0<0 ~ {W >m)) ιs asymptotically negligible, and we may replace ^ Σwi9iΌ by ^(u;,m). If the process is ergodic, ^(w,m) is asymptotically constant. Hence the predictable quadratic variation of n 1 / 2 ^—ϋ) is approximately n||ii;?;1/2||2/(ii;, m) 2 . By the Schwarz inequality, /
/
l
(2.9)
25
PARTIAL LIKELIHOOD In other words, w,m)z
Urn?;"1/2!!2
Hence \\wvι/2\\2 / (w,m)2 is minimized by w = mv~1, and the minimum is -1
By an appropriate martingale central limit theorem, nι/2(ϋ — #) is asymptotically normal with variance equal to the limit of nUmv" 1 / 2 !!" 2 , and the assertion follows. D Efficiency of the estimator based on the optimal estimating equation can be proved by an approach similar to that outlined in Section 1 for the i.i.d. case. If we use predictors to estimate Wi(x) and w*i, i.e. estimators involving only the past observations (Xi, YΊ),..., (Xi_i, Yi_i), then the optimal estimating function is a martingale. We may allow weights wι, w*i to depend on ϋ. Then the derivative of ΣwiQiΰ in the expansion (2.6) has a second term Σwi9iΰ I* ι s asymptotically negligible since the g^ are martingale increments. Remark 1. Usually one takes predictable weights to combine two martingale estimating functions; e.g. Heyde (1987). For the estimating functions Σgi#{Xi,Yi) and Σί7*ztf(Xΐ) this would mean using weights Wi rather than i). Then the best weights would be
and w*i as above, and the minimal asymptotic variance would be the limit of an expression of the form (2.5) with E*» (jβ?%ΰ)2/Ίϊfig'ϊ^) replaced by 2 2 the simpler (Eig'ild) /Eig ΰ. The resulting variance is, in general, larger than our minimum variance (2.5) because Eg 2
^ l
Exg2
This inequality follows from the Schwarz inequality:
(E/)
2
=
(2,0,
26
GREENWOOD AND WEFELMEYER
Remark 2. The weight W{ depends only on p{ and giϋ, and w^ depends only onp*i and 5*^. This is due to orthogonality (2.8). Indeed, the weights (2.3) are optimal for estimating functions of the form Σ Wi(Xi)<7^(Xi, Yi)i a n d weights (2.4) are optimal for estimating functions of the form ^ w Remark 3. Suppose we have a parametric model piϋ(Xi, dy) for the conditional distribution of the response Y{ given the present covariate X{ and the past observations. Differentiating under the integral, we obtain 0 = (Exiΰgiΰ)
= Έ%g'iϋ
where
Hence, by the Schwarz inequality, [E^'g'^j /Έ^βΊjfβ is maximal for "g^ = 2iϋ, and the optimal weight (2.3) for giΰ = liΰ is ϊϋi(Xi) = —1. In particular, the estimating function YJ-i${Xi<>Yi) is optimal among estimating functions Y^Wi(X{)'gil&(Xi1Yi). The optimal estimating function is the partial score function, i.e., the derivative of the partial likelihood ratio of Cox (1975), <Φ(Xύ)
Π
Hence the optimal estimating function gives the maximum partial likelihood estimator. If the observations (XΪ, Y{) are independent, the partial likelihood ratio is the conditional likelihood ratio for Yi,...,Y n given the covariates A i, . . . , Xn. Similarly, if there is a parametric model p*i$ for the distribution of the covariate Xi given the past, the optimal g*i$ is g { χ ) =d ^ i x )
=:
£ { χ )
and the optimal weight (2.4) is —1. Moreover, if there is a fully specified parametric model Piϋ(dx, dy) = p*iϋ(dx)piϋ(x, dy), then the likelihood ratio can be written dp*i dP(χi')
Π
27
PARTIAL LIKELIHOOD
and the optimal estimating function is the score function Σί'iϋ(Xi, Y%) with 4?(*>y) := dT^j^{x,y)
=ίiϋ{x,y) + ί!Hΰ{x).
Hence the optimal estimator is the maximum likelihood estimator, and its asymptotic variance is the limit of
Remark 4. For discrete-time processes, it is common to model the conditional distribution of the response given the past and the present value of the covariate. In the continuous-time setting of Section 3 one usually models the conditional distribution of the response given only the past. This is just a convention: We may consider X%-i rather than X{ the 'present' covariate of the response.
3
Jump processes
Suppose we observe a jump process (X, Y) = (JΓ5,Ys)s>o on a finite time interval [0, ί]. The corresponding multivariate point process is given by the jump measure μ(ds,dx,dy) = The law of the process is determined by the compensator of the jump measure. Assume, for simplicity, that the compensator has the form Ks(dx, dy)ds, so that there are no time points at which the process has a positive probability of jumping. We can write Ks(dx,dy) = asps(dx,dy) withp s a probability measure, the jump size distribution at time 5 given the past, and as the jump intensity. For the theory of continuous-time processes and limit theorems we refer to Jacod and Shiryaev (1987). The multivariate point process corresponding to the response process Y is μγ(ds,dy)= ]Γ ε(sAγs)(ds,dy). s .AYs^O
A regression model is given by a (partial) specification of the compensator of μ y , say Ks(dy)ds. As noted in Remark 4, this is not exactly analogous to the discrete-time case. We specify a predictable function ~gs${y) such that Ksgsΰ = j Ks(dy)gsΰ(y)
=0
28
GREENWOOD AND WEFELMEYER
and obtain a martingale estimating equation
We want to assume a similar partial specification of the distribution of the covariate process X. It will be based on a factorization of Ks(dx,dy) analogous to the factorization (2.1) of the distribution pi(dx,dy). We must take into account the possibility that X jumps while Y does not. Following Arjas and Haara (1984) and Greenwood and Wefelmeyer (1996), we write Ks(dx,dy)
= K-Os(dx,dy)
+ K*s(dx)εo(dy),
where K-QS(dx,dy) does not charge the subspace described by y = 0. Then K*s governs those jumps of X that do not occur simultaneously with jumps of Y. As in (2.1), but with the roles of X and Y interchanged, we factor K-Os(dx,dy)
(3.1)
= Kβ{dy)K-.a(dx,y).
We note that K-*s(dx,y) is a probability measure, the conditional distribution of the jump size of the covariate given a jump of size y of the response, and given the past. Additional specifications of the model may now be given by predictable functions 5*s1?(x) and <7-*stf(#,y) such that K*s9*sϋ = / K*s(dx)g*sϋ(x) Ki+8g-*84 = I K-*s(dx,y)g-*8*fay)
= 0, = 0
for
all y.
They give rise to additional martingale estimating equations =
0,
= 0. How can we combine the three estimating functions in an optimal way? Again, our result holds for geometrically ergodic processes under appropriate smoothness and moment conditions which can be seen from the sketch of the proof. Result 2. From estimating equations of the form (3.2)
wsgso(AYs)
= 0,
29
PARTIAL LIKELIHOOD
an estimator with minimal asymptotic variance is obtained using weights which are consistent estimators of
= KdsϋIKsg2sϋ, ' l
ws
(3.3) (3.4) (3.5)
^ .
The estimator is asymptotically normal. Its asymptotic variance is the limit
of
t
(f ψfU ψf + / ψU£*+ JK^ψί^Δ V
A
#
J
J X
J
9
Λ9
. ,3.6) )
Sketch of proof. To simplify the notation, we introduce w = (w, w*,w-*) and g$ = (<7tf,ί7*τ?, <7-*#), and write the estimating equation (3.2) as -*sg-*sϋ = 0. Let ϋ be a solution. Under appropriate differentiability conditions, a Taylor expansion gives
O = Σ ws9s#« Σ Ws^ + (*-*) Σ wds*
(37)
Then
The martingale £tu* 5 g* s tf is orthogonal to the martingales ΣΰJ s ί? s l ? a n ( i 2 w-*sg-*sΰ because it lives on time points 5 with AYS = 0 while the two other martingales do not jump at these time points. Because K-os(dx,dy) does not charge y = 0, we may and will assume that g-.*s$(x,0) = 0. Then w-*sg-*sϋ
= ws JΈs(dy)gsΰ{y)w-*s(y) Hence ΣΰJsΊJsΰ product
a n d
Σw-*s9-*sΰ
(υ,w) = / vsws+ Jo
(3.9)
J K-*s(dx,y)g-*s#(x,y) = 0.
are also orthogonal. Introduce an inner
/ υ* 5 w* 5 + / Jo Jo J 2
Έs{dy)υ-*s{y)w-*s(y)ds
with corresponding norm ||w|| = (w,w). Consider first the numerator in (3.8). With (3.9) and orthogonality of Σw*s9*sϋ and ΣwsVsϋ, t h e P r e " w 2 2 2 dictable quadratic variation of Σ s9sΰ is {w ,v) = | | W / | | , where vs =
30
GREENWOOD AND WEFELMEYER
Consider now the denominator in (3.8). The compensator of Σws9sϋ (w,ra), where rns = ~Ksg'sϋ, m*5 = K^g1^,
is
m-*s{y) = #-* 5 5-* s tf
Since Σws9ta<β — (w,m) is a martingale, \ (Σws9fs>β — (w,m)) is asymptotically negligible, and we may replace \ Σ ws9's o by \(w->m) ^ the P r o c e s s is ergodic, \(w, m) is asymptotically constant. Hence the predictable quadratic variation of tιί2(ϋ — ϋ) is approximately t\\wvιl2\\2/(w,m)2. By the inequal1 ity (2.9), this is minimized by w = m υ " , and the minimum is
\Jo vs
JQ
V*S
JO
By an appropriate martingale central limit theorem, tιl2{ϋ — ϋ) is asymptotically normal with variance equal to the limit of ίUmi;" 1 / 2 !!" 2 , and the assertion follows. D As in Remark 2, the weight (3.3) is optimal for estimating equations wsgsΰ(AYs)=0, and the weights (3.4) and (3.5) have analogous optimality properties on their own. Remark 5. Suppose we have a parametric model Ksΰ{dy) for the compensator of the jump measure μγ of the response. Write
When the intensity UTS#(R) of the response depends on #, then Ks$ist& will not be zero in general. This differs from the discrete-time case. Differentiating under the integral, we obtain 0 = (KsΰgsΰY
= Ks$g[sύ + KsϋIsϋgsϋ
- Ks^gιsϋ + Ks#(ϊsϋ
-
Ks#lsΰ)gsϋ.
Using the Schwarz inequality, we see that (Ksΰg'sΰ)2
_
is maximal for gs# = l's# — Ks$£rs#, and then the optimal weight is ^tJs = —1.
31
PARTIAL LIKELIHOOD In particular,
is optimal among estimating functions of the form
and the asymptotic variance of the optimal estimator is the limit of
As in Remark 3, the optimal estimating function turns out to be the partial score function in the following sense. A partial likelihood ratio for jump processes was introduced by Arjas and Haara (1984) as - l)ds) . J
See also Andersen, Borgan, Gill and Keiding (1993) and, for general semimartingales, Jacod (1987) and (1990). The partial score function is the derivative of the partial likelihod ratio at τ — ϋ. Using Vs$# = 1 we see that the derivative equals (3.10). Remark 6. Suppose we have a parametric model K*s$(dx)ds for the compensator of the jump measure of those jumps of the covariate X that do not occur simultaneously with jumps of the response Y. Write V*sΰτ = -J77
j
Ksΰ = dτ=
As in Remark 5, the best g*s# is t*sΰ~κ*sϋKsϋ' (3.4) is w*s = — 1. In particular,
an
d then the optimal weight
(3.H) is optimal among estimating functions of the form
32
GREENWOOD AND
WEFELMEYER
Remark 7. Suppose we have a parametric model for the compensator K-*s$(dx,y) of the conditional jump size distribution at time s of X given a jump Y = y and the past. Write
Since K-*s$(dx,y) is a probability measure, we have Ky_^sι&ί'_^sΰ = 0. As in Remark 5, the best g~*sΰ is ^/_*stf, and then the optimal weight (3.5) is w-*s = —1. In particular, (3.12) is optimal among estimating functions of the form ΔYS).
Remark 8. Suppose we have a fully parametric model K8$, K*s$, K_*5ί?. According to Result 2 and Remarks 5 to 7, the best estimating equation is
To show that this gives the maximum likelihood estimator, we recall a representation of Greenwood and Wefelmeyer (1996) of the likelihood ratio, - l)ds)
(3.13)
J
exp (- f K*sΰ(V*sΰτ - l)ds)
For a heuristic derivation in terms of product integrals, see Andersen et al. (1993, p. 107). We have already noted in Remark 5 that the derivative of the first factor, the partial likelihood ratio, equals the partial score function
33
PARTIAL LIKELIHOOD
(3.10). The derivative of the second factor is obtained similarly. Finally, dτ=φV-*s#τ = ^'_*5tf by definition. The representation (3.13) of the likelihood can be used in the partially specified model of Result 2 to prove that the optimal estimating function obtained there is efficient as long as no additional restrictions involving ΰ are imposed on the model. The arguments are similar to those outlined in Section 1 for the i.i.d. case. In Greenwood and Wefelmeyer (1996) a representation analogous to (3.13) is given for general semimartingales and can be used to generalize the results obtained here to partially specified semimartingale regression models. References Andersen, P. K., Borgan, 0., Gill, R. D. and Keiding, N. (1993). Statistical Models Based on Counting Processes. Springer-Verlag, New York. Arjas, E. and Haara, P. (1984). A marked point process approach to censored failure data with complicated covariates. Scand. J. Statist. 11,193-209. Bickel, P.J., Klaassen, C.A.J., Ritov, Y. and Wellner, J.A. (1993). Efficient and Adaptive Estimation for Semiparametric Models. Johns Hopkins University Press, Baltimore. Cox, D. R. (1975). Partial likelihood. Biometrika 62, 269-276. Greenwood, P. E. and Wefelmeyer, W. (1996). Cox's factoring of regression model likelihoods for continuous time processes. To appear in Bernoulli. Heyde, C.C. (1987). On combining quasi-likelihood estimating functions. Stochastic Process. Appl. 25 281-287. Jacod, J. (1987). Partial likelihood process and asymptotic normality. Stochastic Process. Appl. 26, 47-71. Jacod, J. (1990). Sur le processus de vraisemblance partielle. Ann. Inst. Henri Poincare 26, 299-329. Jacod, J. and Shiryaev, A. N. (1987). Limit Theorems for Stochastic Processes. Grundlehren der mathematischen Wissenschaften 288, SpringerVerlag, Berlin.
35
Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES
AVOIDING THE LIKELIHOOD C.C.Heyde Columbia University and Australian National University ABSTRACT For the estimation of a finite dimensional parameter in a stochastic model it has become increasingly clear that it is usually possible to replace likelihood based techniques by quasi-likelihood alternatives in which only assumptions about means and covariances are made in order to obtain estimators. If it is available, the likelihood does provide a basis for benchmarking of alternative approaches but not more than that. The challenge is to see whether everything that can be done via likelihoods has a corresponding quasi-likelihood approach from which the likelihood based results can be recovered, if they are available. It is conjectured that this is the case. In this paper, various illustrations are sketched of avoiding the likelihood in contexts where alternative approaches have not been obvious. Key Words: Quasi-likelihood; E-M algorithm; constrained estimation; nuisance parameters; diffusions; REML estimation.
1
Introduction
This paper is concerned with promoting the thesis that: For parameter inference (1) it is advantageous to make minimalist assumptions on models (initially concerning only means and covariance structure), and (2) there is a sensible quasi-likelihood (QL) alternative/ generalization of any likelihood based methodology, at least to the first order of asymptotics. We have also come to rely on the full distribution theory as a basis for a wide range of statistical procedures. Indeed, questions of appropriateness of the model are often supressed in order to make use of easy analytical methods (as with the Black-Scholes model in Finance). However, many ostensibly likelihood based methods do not actually require full distributional assumptions. They can readily be extended to the estimating functions context when there is a conservative quasi-score. That is, an estimating function which is the gradient of a scalar objective function which plays the role of the
36
HEYDE
likelihood if and when it exists. However, it is argued that a scalar objective function for which the quasi-score is the gradient is inessential. These pronouncements may be regarded as controversial. However, they are motivated by a wish to promote serious consideration and debate and not out of intrinsic dogmatism. We shall give a smorgasbord of examples to elucidate the point of view described above. After discussion the general QL framework (Section 2), we shall describe some Projection-Solution (P-S) methods, namely in the contexts of constrained parameter estimation, nuisance parameters and the E-M algorithm (Section 3). Then we shall discuss bypassing the likelihood through examples for diffusion processes and REML estimation (Section 4). There are, of course, many other areas in which there is substantial progress towards the use of estimating functions without direct recourse to likelihood ideas. These include the areas of multiple roots (e.g. Heyde and Morton (1996b)), likelihood ratio tests (e.g Li (1993)) and Bayesian analysis (e.g. Godambe (1994)). A much broader perspective will soon be available in Heyde (1997).
2
General QL Principles
Suppose we have a sample {Xt, t G T} of vectors of dimension r, T possibly being discrete, continuous or lattice. The possible probability measures {Pθ} for {Xt} are the union of families of models and the θ = (0χ,..., ΘPY to be estimated is a vector of dimension p. The approach is via the set of p dimensional vector estimating functions Q = {Gτ(θ) = Gτ({Xut GT},0)} which are functions of the data and θ for which EGT = 0 for each PQ and the matrices EGT{Θ) =
dθ
{EdGτ,i{Θ)l j)
and EGτ{θ)Gτ{θ)f are nonsingular, the prime denoting transpose. The QL theory focuses on suitably chosen subsets of Q and involves choice of an estimating function GT to maximize, in the partial order of non-negative definite (nnd) matrices, the information criterion:
S{Gτ) = {EOTyiEGrG^iEOT) which is a natural generalization of Fisher information. We have the following definition. Definition. Suppose that G*τeUcG.
If
37
AVOIDING THE LIKELIHOOD
is nnd for all GT G Ή we say that Gΐp is a quasi-score estimating function (QSEF) within U. The choice of the family Ή is completely open and should be tailored to the particular application. The estimator θ^ obtained from Gγ{θ^) = 0 which is termed a quasilikelihood estimator has, under broad conditions, certain minimum size asymptotic confidence zone properties for 0, at least within Ή. Indeed, the basic properties are those of the maximum likelihood estimator, but restricted to the class Ή. The theory does not require a parametric setting, let alone the existence of a likelihood score function Uτ{θ). However, if UT G Ή, as can ordinarily be arranged in exponential family problems, then UT is the QSEF within 7ί and can easily be calculated without using likelihoods. It is not usually practicable to find a quasi-score estimating function directly from the definition. However, the criterion given in the following proposition is easy to use in practice. Proposition 1. Let Ή. G Q. Then Gj* G H is a quasi-score estimating function within H if ) EGχGj> = CT (2-1) for all GT G H, where CT is a fixed matrix. Conversely, if H is convex and Gγ is a quasi-score estimating function then (2.1) holds.
3
Projection Based Methods
Many problems for which the use of likelihood based ideas is standard, and for which there is not an obvious quasi-likelihood analogue in the absence of a conservative quasi-score, can be dealt with via projection based methods. We give three illustrations in this section.
3.1 Constrained Parameter Estimation Here we wish to estimate θ subject to the constraint F'θ = cί, F being a q x p matrix which does not depend on the data or θ. Suppose we have an unconstrained quasi-score Q(θ) G Ή. and, using a minus to denote generalized inverse, define the projection matrix P = 1
FiF'V^FyF'V'1
for V = EQQ . In the case where a likelihood L(θ) is available, the usual procedure is to use the method of Lagrange multipliers and maximize L(θ) + \'(Ffθ — d) where λ is determined by the constraint. Thus, we differentiate with respect
38
HEYDE
to θ and solve the equations
for λ and 0, U being the score function. The striking thing is that this procedure works in general for quasilikelihood, even in the non-conservative case. We solve the equations
for λ and 0, that is, ( I - P ) Q ( 0 ) = O , F ' 0 = d. Optimality is preserved. For details see Heyde and Morton (1993). 3.2 Nuisance Parameters Here we have θ' = (φ', φ1) where φ is the parameter of interest and ψ is a nuisance parameter. Then, supposing that we have a quasi-score Q(θ) E Ή, we use the partitioned forms
and write Φ
φ
\ VΨΦ ( VΦΦ \ Vφφ
The projection identifies the information about φ for φ given and the estimating equation (I-PΨ)Q = O is optimal for the estimation of φ in the presence of the nuisance parameter ψ. The sensitive dependence of Q on φ has been removed in the sense that
This is a first order approach. In the language of McLeish and Small (1988), (/ - Pψ)Q is locally E-ancillary for ψ and PψQ is locally E-sufficient for ψ.
39
AVOIDING THE LIKELIHOOD
3.3 E-M Algorithm Generalization The E-M method is used for parameter estimation where there is missing data. In the first (E) step, one takes the conditional expectation of the complete data likelihood with respect to the available data. In the second (M) step, one maximizes over possible distributions. However, it is possible to avoid the likelihood completely by introducing a project-solve (P-S) method. Suppose that the full data is denoted by z, the observed data by y and θ is the parameter of interest. We seek to adapt a quasi-score Q(θ; x) G Hx to obtain a quasi-score Q{θ;y) G Ήy. In fact E(Q* - Q)(Q* - QY = inf E(G - Q)(G - Q)' Grfc Hy
and Q* is the element of Ή,y with minimum dispersion distance from
Q G nx.
If the likelihood score U G Ή x , then Q = E(U\y) provided this belongs to Ήy as in the E-M case. However, Q* is given in general just as a least squares predictor and mostly
Q*(θ y)φEθ(Q(θ,x)\y). A detailed discussion of the method can be found in Heyde and Morton (1996a). There is an algorithm for solving Q*(θ; y) = 0 along the lines of the E-M algorithm. This usually gives a first order rate of convergence. However, the equation can often be solved directly with a second order rate of convergence, for example using Fisher's method of scoring.
4
Bypassing the Likelihood
In this section we give examples of the derivations of score functions without having first to find a likelihood to differentiate. In such cases there is the added advantage of being useful under distinctly broader distributional conditions than are imposed by the need to prescribe a likelihood.
4.1 Parameters in Diffusion Type Models Here we have a model described by the stochastic differential equation dXt = a(t, Xu θ)dt + 62 (t, Xt)dWt where α, b are known functions and Wt is standard Brownian motion. The usual approach to estimation of θ is to obtain an appropriate RadonNikodym derivative. This is tedious from first principles. Differentiation with respect to θ then gives the likelihood score.
40
HEYDE
Alternatively, one may consider the family of martingale estimating functions rT
n ={ kt(θ)(dXt - a(t, Xu θ)dt), ktpredictable}. Jo The quasi-score estimating function from this family can be written down almost immediately using Proposition 1 as jf (a(t,Xuθ))'(b(t,Xt))-ι(dXt
-a{t,Xuθ)dt),
and this is equivalent to the likelihood score. The explanation is straightforward. Note that the elements of H are (martingale) stochastic integrals with respect to Wt Also, a likelihood score is a martingale under modest regularity conditions. Furthermore, all square integrable martingales living on the same probability space as this process can be described as stochastic integrals with respect to the Brownian motion. Thus Ή, contains the likelihood score and the QSEF will pick it out. Now the quasi-likelihood method goes much further. One does not have to perturb a diffusion type model much to destroy the likelihood. For example, in the Cox-Ingersoll-Ross model used for interest rates in financial modelling, 1
dXt = Oί^β — Xfjdt + (JXI dWt, the Radon-Nikodym derivative will not exist if the volatility σ is rate dependent on α. However, the QSEF is unaffected. For more details see Heyde (1994a). 4.2 Restricted (or Residual) Maximum Likelihood Here the problem is of estimating dispersion in a linear model; the nxr vector y has the multivariate normal distribution MVN(Xβ, V(θ)) with mean Xβ, covariance V(θ) and θ is to be estimated. Take the rank of X as r, the dimension of β and let A be any matrix with n rows and rank n-r satisfying A'X = 0. Then A'y has the MVN(0, A'VA) distribution. The striking thing here is that the likelihood function does not depend on A. Indeed, for all A, A(A'VA)-A' = V~ιQ where Q — I-P, P being the projector onto the subspace R(X) (the range space of X) with respect to the inner product a'Vb. The likelihood function of θ based on A'y is, omitting a constant multiplier,
(f{li)2eχp{-\y'V-ιQy)
41
AVOIDING THE LIKELIHOOD ι
where the k are the non-zero eigenvalues of V~ Q. This is not a straightforward calculation, nor is the differentiation with respect to θ required to obtain the REML estimating equations ι
^
^- Q)y,i
= 1,2, ...p,
tr denoting trace. For the quasi-likelihood approach we no longer require multivariate normality but instead that y has mean vector Xβ, covariance matrix V(θ), and that each yι has kurtosis 3. The crucial step is taken by noting that we expect to use quadratic functions of the data to estimate covariances. For fixed A, let z = A!y and take θ as a scalar for clarity. Now introduce the family of estimating functions U = {G(S) = z'Sz - Ez'Sz, Ssymmetric}. Write W = A'VA. Then EG{S)G{S*) = 2tr(WSWS*) Ez'Sz = tr{WS) EG(S) = - t r ί ^ S ) , and we see via Proposition 1 that S* for the QSEF is given by
The REML estimating equations then follow since AW~A! = V~ιQ. For more details see Heyde(1994b).
42
HEYDE References
Godambe, V.P. (1994). Linear Bayes and optimal estimation. Technical Report STAT-94-11, University of Waterloo, Canada. Heyde, C.C. (1994a). A quasi-likelihood approach to estimating parameters in diffusion-type processes. Studies in Applied Probability. J. Applied Prob., 31A, 283-290. Heyde, C.C. (1994b). A quasi-likelihood approach to the REML estimating equations. Statistics and Probability Letters, 21, 381-384. Heyde, C.C. (1997). Quasi-Likelihood Theory and its Application. Springer, New York. Heyde, C.C. and Morton, R. (1993). On constrained quasi-likelihood estimation. Biometrika, 80, 755-761. Heyde, C.C. and Morton, R. (1996a). Quasi- likelihood and generalizing the E-M algorithm. J. R. Statist. Soc. Ser. B, 58, 317-327. Heyde, C.C. and Morton, R. (1996b). Multiple roots and dimension reduction issues for general estimating equations. Unpublished manuscript. Li, B. (1993). A deviance function for the quasi-likelihood method. Biometrika, 80, 741-753. McLeish, D.L. and Small, C.G. (1988). The Theory and Applications of Statistical Inference Functions, Lecture Notes in Statistics Vol.44, Springer, New York.
43 Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES
LIKELIHOOD AND PSEUDO LIKELIHOOD ESTIMATION BASED ON RESPONSE-BIASED OBSERVATION J.F. Lawless University of Waterloo
ABSTRACT Response-biased observation refers to situations where the probability a unit is observed depends on the value of a response associated with that unit. We discuss the construction of estimating equations for parametric regression models through likelihood and pseudo likelihoods, for situations in which responses are stratified and sampling is stratum-specific. Properties of the resulting estimators are reviewed and an illustration involving field reliability data is presented.
1
Introduction
In many observational studies the probability that a specific individual or unit is observed or selected in a sample depends upon responses or covariates associated with that unit. That is, if units in some population have associated response variables y and covariates #, then the probability unit i is selected depends upon the values (yi,Xi) for that unit. When the probability of selection depends upon yι, we call the observation scheme response-selective, or response-biased. For simplicity of exposition I will focus mainly on situations where the probability of selection depends solely on yι. However, as described at the end of Section 2 and in Section 4, situations where the probability of selection depends on both yι and X{ may also be handled using the methods considered here. Examples of response-selective observation are abundant. In socio-economic studies based on samples drawn from administrative records, selection is often response-related (e.g. Hausman and Wise 1981). Similarly, in a study of factors affecting low birth weight of humans, one might select newborns over a period of time and measure covariates so as to over-sample babies with
44
LAWLESS
low birth weights. Extreme forms of response-selection are embodied in casecontrol or choice-based sampling used in epidemiology and economics (e.g. Breslow and Cain 1988; Hsieh, Manski and McFadden 1985); in this case the response variable is categorical and covariates are observed for samples of individuals selected from each response category. Examples of selection bias in more complicated settings are given by Hoem (1985) and Kalbfleisch and Lawless (1988a), who discuss the observation of life history events for human populations. When the probability of selection p(y) for a unit is a known function of y, methods that weight log likelihood or estimating function components ι according to p{y)~ are often used. Several other approaches can also be used in various contexts. This article discusses four methods of estimation for a rather broad class of situations. The approaches described are not new, but questions remain about their properties. Our purpose is to review the four methods and recent investigations into their properties, and to indicate connections with other areas. It is assumed that (y, x) values for individuals or units in some "population" from which units will be sampled are generated from a probability distribution with density or mass function f(y\x;θ)g(x)
yey,xeX
(1)
and that our objective is to estimate the p-dimensional parameter θ. We wish to avoid strong parametric assumptions about g(x) and the corresponding distribution function G(x), as is common in regression modelling. Sampling is response-selective in the following sense: the range of y is partitioned into strata 5i,...,5fc and if yι G Sj then unit i is sampled (selected) with probability pj. In other words, p(yi) = γjj=ιpjl(yi G Sj), where I (A) is the indicator function which equals 1 if event A is true and 0 otherwise. More specifically, we assume that a finite population of N units has values (yi,Xi), i = l,...,iV generated as independent realizations from (1). In survey sampling terminology, we have a stratified population and wish to estimate parameters in the superpopulation model (1). Samples may be selected in various ways. We consider two, termed basic stratified sampling (BSS) and basic variable probability sampling (BVPS). In BSS a simple random sample of specified size Πj is selected from units in the j ' t h stratum (i.e. with yι G Sj). In BVPS each unit in the population is considered for selection independent of every other unit. The j ' t h stratum size (number of units with yι G Sj) is denoted by Nj (j = 1,..., k). In the case of BSS the fixed sample size from stratum j is nj — NjPj, whereas in the case of BVPS the size of the sample from the j ' t h stratum is random. It is an important feature of our framework that the stratum sizes iVi,...,iV^ are known, or observable. This latter feature is not present in some applications, e.g. for
45
RESPONSE-BIASED ESTIMATION
many instances of case-control or choice-based sampling. The remainder of the paper is as follows. Section 2 makes the observational framework precise and then describes methods of estimation based both on likelihood and on pseudo likelihood functions. Section 3 discusses asymptotic properties and variance estimation, and Section 4 illustrates the methodology. Section 5 concludes with remarks on extensions and relationships with missing data methods.
2
Likelihood and Pseudo Likelihood Estimating Functions
Suppose that individual pairs (y^Zi), ί — l,...,iV are generated independently from (1), and that Nj units have yι E Sj (j = 1, ...,&). Units are selected by either BSS or BVPS as described in Section 1 and (yi,Xi) observed. Let Ri = /(unit i is selected), and Dj = {i : Ri = l,yj € Sj} denote the units selected from stratum j , where \Dj\ = nj. As is customary, Y and X are used to represent the random variables of which yι and X{ are realizations. For simplicity I consider only situations where the population size N and stratum sizes Nj are fixed at the time units are selected. It is also assumed that the iV^'s are known and further, that for units not selected all that is known is which stratum they are in. In some contexts such as the birth weight study, the values of N and the JVj's may be unknown until the end of the sampling period and the j/i's (but not the x^s) may be known for units not selected. Such features may also be dealt with via the methods discussed here; Lawless, Wild and Kalbfleisch (1997) consider a variety of response-selective sampling schemes. Under BSS or BVPS in the framework described, the data include the stratum sizes iVΊ,..., JV^, the pairs (j/i,x<), i € Dj (j = l,...,fc) for the selected units, and for BVPS, the sample sizes ni,...,rifc. In either case the probability density function for the observed data defines a likelihood function for the unknown parameters θ and G (the distribution function of X) which is proportional to LF(Θ,G)
= l[{Qj(θ,G)Ni-ni
Π f{Vi\*ϊJ)M!(*i)h
(2)
j=\
where Qj(θ,G) = Pr(Y e Sr,θ,G) =
ίPr(Y
e Sj\x\θ)dG(x)
(3)
46
LAWLESS
It is recognized in the notation that (2) depends on both θ and G. Prom our point of view G is a nuisance parameter but because of (3) it is necessary to estimate it in order to estimate θ by maximizing (2). One approach is to maximize the semiparametric likelihood (2) jointly with respect to θ and G. This is feasible when Y is categorical (Wild 1991, Scott and Wild 1997) or when G is discrete with relatively few points of support (Hsieh et al. 1985), and recent work suggests it is feasible quite generally. A second line of attack is to maximize LF(Θ,G), where G is a simple nonparametric estimate of G\ this is an extension of the parametric pseudo likelihood idea of Gong and Samaniego (1981) to a semiparametric setting. Noting that
G(x) = ΣPr{Xi
< x\Yi e Sj)Pr(Yi e 5, ),
3=1
we propose to use the estimate (4) where Gj (x) is the empirical cumulative distribution function (cdf) based on the Xi's for units i G Dj sampled from the j ' t h stratum. Inserting (4) into (3) and taking d\ogL(θ,G)/dθ, we obtain the pseudo score function
sp(θ) =
^y,
Σ
diogfiyilxi θ) dθ
' ι=i
(5)
ί=l
where pi =
and
= Pr(YeSj\x;θ).
(6)
We estimate θ by solving the equation SP(Θ) = 0. This idea has been used in other contexts by Pepe and Fleming (1991), and Hu and Lawless (1997). A third possibility is to weight score contributions for sampled units to give the weighted pseudo score (7) 3=1
RESPONSE-BIASED ESTIMATION
47
It is apparent that S\γ(θ) is unbiased (has mean 0) under BSS or BVPS, where expectation is with respect to (1) and the selection scheme. Weighting is common in survey sampling (eg. Holt, Smith and Winter 1980, Binder 1983, Binder and Patak 1994) and has been considered for maximum likelihood methods by Hsieh et al. (1985), Scott and Wild (1986), Kalbfleisch and Lawless (1988ab) and others. Robins et al. (1994, 1995) consider weighted pseudo score functions when the selection probabilities p(yi) may depend on unknown parameters. The final method considered is based on the observation that in the case of BVPS the distribution of the observed responses, conditional on the values of Rι and X{ (i = 1, ...,iV), yields the conditional, or selection-biased likelihood N
Lc(θ) =
= ΠΠ j=l
(8)
ieDj
U=i
The corresponding score function is
Sc(θ) =
aiog/(y z | ^ ;(
(9)
dθ ί=l
Straightforward calculation shows that Sc{θ) is unbiased under BSS as well as under BVPS. lΐ pi = = pk then Sw(θ) = 0 and Sc(θ) = 0 yield the same estimate, but otherwise the estimators obtained from the estimating equations Sp(θ) = 0, Sw(θ) = 0 and Sc{θ) = 0 appear to be distinct. General results concerning the relative efficiencies of the estimators θp,θw and θc obtained from these equations, and θp obtained by maximization of Lp(θ,G) with respect to θ and G, are not available, though results of Robins et al. (1994, 1995) show that the pseudo-likelihood methods are asymptotically inefficient. Another important point is that in the case of BVPS it is preferable to use pj = nj/Nj rather than prior selection probabilities. In Section 3 we discuss asymptotic properties of the estimates and in Sections 4 and 5 some limited simulation results. We conclude this section by noting that analogous estimating functions may be given for cases in which the probability of selection for a unit depends
48
LAWLESS
on both y and x. In particular, suppose that k strata are defined according to whether {yi.Xi) E Sj, where SΊ, ...,£& partitions y x X. The likelihood function based on the observed data is now given by (2), with Qj(θ,G) redefined as Qj(θ,G) =
=
Pr{(Y,X)eSjiθ,G}
ίPr{{Y,x) e Sj\x',θ}dG{x).
(10)
A pseudo score Sp(θ) corresponding to (5) is obtained by estimating (10) via (4), where G(x) is as before the empirical cdf based on the x^s for units i E Dj. A weighted pseudo score function is given by (7) once again, and a conditional score is given by (9) with Q}(XΪ,Θ) replaced by QUx;θ) = Pr{{Y,x)eSι\x ,θ}.
3
(11)
Asymptotic Properties, Variance Estimates and Confidence Limits
By taking limits as N -> oo and with pj = rij/Nj (j = 1, ...,&) fixed positive values, we may show that under mild conditions the estimators θ obtained by solving Sw{θ) = 0 or Sc(θ) = 0 are consistent and asymptotically normal. Special cases have been considered by Kalbfleisch and Lawless (1988b), Wild (1991) and Scott and Wild (1997). Asymptotics under BVPS may also be obtained. A rigorous development of asymptotics for the case of full maximum likelihood based on (2) or for the estimating equation Sp(θ) = 0 is more difficult. For the former Wild (1991) and Scott and Wild (1997) deal with the case of categorical responses, and for the latter Hu and Lawless (1997) deal with the special problem described in Section 4. We outline the asymptotic normal results for Sw(θ) and Sc(θ) given by (7) and (9), respectively; Lawless et al. (1997) give a fuller treatment. Both (7) and (9) may be written in the form N
;Xi;θ).
(12)
For Sw{θ), for example, k
U(Yϊ;Xi',θ) = Σpfm 3=1
6 Sj)dV>zfWXi\θ)ISΘ.
(13)
49
RESPONSE-BIASED ESTIMATION Assume p\im(Nj/N) = 7Γ, > 0 as N ->• 00 and define
A(θ) =plimAN(θ)
B{θ) = UmBN{θ).
Under mild regularity conditions on the model (1), we have yjNφ -Θ0)ΛN
(θ,A{θo)-ιB{θo)A{θo)-1) ,
where ΘQ represents the true value of θ. The asymptotic variance of θ may be estimated as Ά~ιBA~ι, where A and B are consistent estimates of A(ΘQ) and JB(0o) The matrix A = AN(Θ) may be used to estimate A{ΘQ). For Sc(θ) in the case of BVPS the estimating function is a likelihood score function, so A(ΘQ) = B(ΘQ). For the other cases we require a consistent estimator B, which is not hard to obtain. For Sw{θ), for example, extending the approach of Kalbfleisch and Lawless (1988b) and defining Vi{θ) = dlogf(yi\Xi;θ)/dθ YΆτ{Sw(θ)}
and w
= VΆτγ]xEmx{Sw(θ)} = E(^)
+
C(θ),
(θ) = Σi-.y.eSj Vi(θ)/Nj, we get +
Eγ]xVΆτmx{Sw(θ)} (14)
where for BSS we have
C(θ) = EYlx J2 n ^
P
i \ Σ foM- ^ 0 ) (θ)}[vi(θ)- vU) (*)]'. (15)
j^iPjW-Vines,
χ
Since N- E{-dSw/dθ) = A(θ), equations (14) - (15) indicate that B(θ) may be estimated by B = A + C, where
= h Σ ^r^i-^)
Σte(β)- ^ωW]NW - tJ^ίβ)]', (16)
where i Confidence intervals for parameters may be obtained by treating y/N(θ — θ) as approximately normal with a suitably estimated covariance matrix. An alternative would be to use some form of bootstrap. Investigation of specific problems by simulation is needed to gain insight into the adequacy of confidence interval procedures for different sample and population sizes. As noted in Section 5, relatively little is known about the efficiency of Sw (0), Sc(θ) and Sp(θ) in general situations; this too deserves investigation.
50
4
LAWLESS
An Example
Kalbfleisch and Lawless (1988b), Hu and Lawless (1996) and others have considered problems in epidemiology and reliability in which a response time yι > 0 and covariates z% for an individual or unit in some population are always observed if yι does not exceed an associated censoring time rι > 0. The number of units for which the response time is censored (i.e. yι> Ti) is known but the values of T{ and Z{ are not, so a fraction p2 of the censored units are sampled, and their T{ and Z{ values are obtained. The objective is to estimate the distribution f{yi\zi\θ), where it is assumed that Y{ and T{ are independent, given Z{. This problem involves response-selective sampling of the type described at the end of Section 2. In particular, let xι = {r^Zi) be an extended covariate vector representing the censoring time τι and covariates zι for unit i. The data for N units i — 1,..., JV are assumed to come from (1), where f{yi\xi',θ) = f{yi\zi',θ)- Consider two strata for (y,τ,z) defined by S\ = {(y,τ,z) - y < r} and S2 = {{y,τ,z) : y > r}. Units with (y^Xi) G Si are selected with probability pi = 1 and those with (yi, X{) G S2 a r e selected with probability p2 < 1. We extend the four estimation procedures of Section 2 slightly to deal with the stratification on both y and x, as described at the end of Section 2, and to reflect the fact that for units i G D<ι we know only that y%> τι, and not yi's exact value. We obtain the likelihood function corresponding to (2) as
LF(Θ,G)=
Π
Π
ieDi
i£D2
(17) where F(τ\x; θ) = /r°° f{y\x; θ)dy and
Q2(θ,G) = JF(τ\x;θ)dG(x). The pseudo score Sp(θ) corresponding to (5) is then
=
^
d\ogf{yi\xi ,θ)
L
QΘ dF(τi\xi ,θ) , 1 dθ VI ^ — n\ — Π2)
dθ
51
RESPONSE-BIASED ESTIMATION The weighted pseudo score corresponding to (7) is ^ Sw(θ) = ^
d]ogf(yi\xi ,θ) 1 ^ + -
Finally, the conditional (pseudo) score corresponding to (9) is
5
*
F(φi
&
dθ
^ ; g)/gfl 1 θ)+p2F(φi θ) }
where F ( φ ; 0) = 1 - F ( φ ; 0). Hu and Lawless (1996, 1997) illustrate the use of the four estimation methods on problems involving automobile warranty data, and compare the methods in a simulation study. In their context N is large (N = 4000 in the simulation) and the proportion of the population falling into stratum 1 is .25 or smaller. They found that with selection probabilities p2 in the range .05 - .20, the four estimation methods were all close to unbiased and gave estimators with roughly the same variance. In addition, normal approximations for 0 were adequate for the range of population and sample sizes considered. The estimators based on Sw{θ) and Sc{θ) are easier to deal with in terms of variance estimation, and SW(0) has the added convenience of being computable with standard censored lifetime data software that allows variable case weights.
5
Additional Remarks
This article reviews several approaches to estimation of parameters when sampling is response-selective with known selection probabilities. Discussion was restricted to two common selection procedures (BSS and BVPS), but extensions to other schemes are possible. For example, in some applications, such as the birth weight study mentioned in Section 1, quota sampling may be used so that the total size N of the population assumed to be generated by (1) is random. Information about the relative efficiencies of the different estimation procedures is at present quite limited. For the scenario described in Section 4 Hu and Lawless (1997) found all four methods to be comparable. However, in the case of binary responses Wild (1991) and others have found that estimators based on the weighted pseudo score Sw{θ) can be considerably less efficient than those based on Sc{θ) or on LF(Θ,G). Wild also found that Sc{θ) gave estimates very nearly as efficient as those obtained
52
LAWLESS
from the full likelihood Lp(θ,G). A limited simulation study by Robins et al. (1994) in a binary response problem, however, revealed situations where all of Sw(θ)iSc(θ) and Sp(θ) were rather inefficient relative to LF(Θ^G). Further investigation is desirable. It should be mentioned that another feature of weighted pseudo scores is their applicability in more complex probability sampling situations (e.g. Binder 1983, Binder and Patak 1994) and in situations where only moments of Y given X are modelled, rather than f(y\x\ θ) (OΉara Hines 1997). However, other more efficient pseudo likelihood methods can also be developed (Robins et al. 1994). There is a close connection between the methods discussed here and methods for dealing with missing data. Indeed, the present framework can be viewed as one in which covariate values are missing for units that are not selected. The approaches to estimation used here may also be applied with more general missing data problems. Robins et al. (1994, 1995) and Carroll et al. (1995, Chapter 9) provide wide-ranging discussions. Hu and Lawless (1997) also provide general discussion, and some simulation results. Robins et al. deal with very general problems in which the probability an observation is incomplete (has data missing) may depend upon unknown parameter values. They obtain asymptotically optimal estimators of θ within semiparametric models but their methods are generally difficult to implement. As remarked earlier, it is of considerable interest to compare the various approaches in more detail. Lawless et al. (1997) give some results. Finally, we note another method of estimation that is suggested by the use of the EM algorithm to maximize Lp{θ,G). By considering the "complete" data log likelihood based on knowledge of all x^s (i = 1,..., iV), N 2=1
+
(1 - Λi){logQ£(z i; 0) +
we obtain the following E-M algorithm, which leads to a stationary point of £F{Θ,G): let x\,...,x*m denote the distinct x^s observed, and denote gr = dG(x*). Then we have E-step: Given current estimates 0, G = (pi, ...,fifm)5 compute w
rj
=
M-step: Obtain the updated estimate of θ by solving
r=l
53
RESPONSE-BIASED ESTIMATION Obtain the updated estimate of G from
_
N ~ where dr = ΣΪLi I{Ri = l,Xi = x*). If instead of (18) we use the empirical estimates 9r
ώ
r, = Σ
in (19), we obtain an estimating equation SM{Θ) = 0, where
j=ι
n
J
A similar idea has been used in a different context by Reilly and Pepe (1995), and it would be of interest to see how it performs in the current framework. It is easily seen that for the example in Section 4, SM(Θ) is identical with Sw (θ) and it is also identical when y is categorical with strata corresponding to categories. More generally, however, it is different.
Acknowledgements This work was partially supported by grants from the Natural Sciences and Engineering Research Council of Canada.
References Binder, D.A. (1983). On the variance of asymptotically normal estimators from complex surveys. Int. Statist. Rev. 51, 279-292. Binder, D.A. and Patak, Z. (1994). Use of estimating functions for estimation from complex surveys. J. Amer. Statist. Assoc. 89, 1035-1043. Breslow, N.E. and Cain, K.C. (1988). Logistic regression for two-stage casecontrol data. Biometrika 75, 11-20. Carroll, R.J., Ruppert, D. and Stefanski, L.A. (1995). Measurement Error in Nonlinear Models. London: Chapman and Hall. Gong, G. and Samaniego, F.J. (1981). Pseudo maximum likelihood estimation: theory and applications. Ann. Statist. 9, 861-869.
54
LAWLESS
Hausman, J.A. and Wise, D.A. (1981). Stratification on endogenous variables and estimation: The Gary Income Maintenance Experiment. In Structural Analysis of Discrete Data with Econometric Applications, eds. C.F. Manski and D. McFadden, Cambridge, MA: MIT Press, pp. 364-391. Hoem, J.M. (1985). Weighting, misclassification and other issues in the analysis of survey samples of life histories. Chapter 5 in Longitudinal Analysis of Labor Market Data, eds. J.J. Heckman and B. Singer. Cambridge, UK: Cambridge University Press. Holt, D., Smith, T.M.F. and Winter, P.D. (1980). Regression analysis of data from complex surveys. J. Roy. Statist. Soc. A 143, 474-487. Hsieh, D.A., Manski, C.F. and McFadden, D. (1985). Estimation of response probabilities from augmented retrospective observations. J. Amer. Statist. Assoc. 80, 651-662. Hu, X.J. and Lawless, J.F. (1996). Estimation from truncated lifetime data with supplementary information on covariates and censoring times. Biometrika 83, 747-761. Hu, X.J. and Lawless, J.F. (1997). Pseudo likelihood estimation in a class of problems with response-related missing covariates. To appear in Canad. J. Statistics. Kalbfleisch, J.D. and Lawless, J.F. (1988a). Likelihood analysis of multistate models for disease incidence and mortality. Statist. Med. 7, 149-160. Kalbfleisch, J.D. and Lawless, J.F. (1988b). Estimation of reliability from field performance studies, (with discussion), Technometrics 30, 365388. Lawless, J.F., Wild, C.J. and Kalbfleisch, J.D. (1997). Likelihood and pseudo likelihood estimation for response-stratified data. U. of Waterloo Technical Report Stat-97-07. O'Hara Hines, R.J. (1997). Fitting generalized linear models to retrospectively sampled clusters with categorical responses. To appear in Canad. J. Statistics. Pepe, M.S. and Fleming, T.R. (1991). A nonparametric method for dealing with mismeasured covariate data. J. Amer. Statist. Assoc. 86, 108113.
RESPONSE-BIASED ESTIMATION
55
Reilly, M. and Pepe, M.S. (1995). A mean score method for missing and auxiliary covariate data in regression models. Biometrika 82, 299-314. Robins, J.M., Rotnitzky, A., and Zhao, L.P.(1994). Estimation of regression coefficients when some regressors are not always observed. J. Amer. Statist Assoc. 89, 846-866. Robins, J.M., Hsieh, F. and Newey, W. (1995). Semiparametric efficient estimation of a conditional density with missing or mismeasured covariates. J. Roy. Statist. Soc. B 57, 409-424. Scott, A.J. and Wild, C.J. (1986). Fitting logistic regression models under case-control or choice-based sampling. J. Roy. Statist. Soc. B 48, 170-182. Scott, A.J. and Wild, C.J. (1997). Fitting regression models to case-control data by maximum likelihood. Biometrika 8^ 57-71. Wild, C.J. (1991). Fitting prospective regression models to case-control data. Biometrika 78, 705-717.
57
Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES
LIKELIHOOD FROM ESTIMATING FUNCTIONS P. A. Mykland University of Chicago ABSTRACT Likelihoods based on estimating equations have typically been designed to retain some, but not all, of the desirable properties of true likelihood. We discuss how these differences in intention affect a number of commonly used procedures, and, in particular, how various likelihoods live up to the ideal: true likelihood. Key Words: Accurary, adaptive estimation, conditionality, dual likelihood, efficiency, empirical likelihood, projective likelihood, quasi-likelihood.
1
Introduction
Likelihood inference has the nice property that it solves several different problems at the same time. Such inference is first order efficient. It has nice accuracy properties, in particular when using the likelihood ratio statistic, its signed square root R, and the associated R* (Barndorff-Nielsen (1986)) statistic. The accuracy is both unconditional and conditional (McCullagh (1984), Jensen (1992, 1997)). One gets a likelihood surface. And likelihood incorporates notions of inferential correctness from both the frequentist and Bayesian viewpoints. This concatenation of desirable features would seem hard to replicate in a less than full parametric setting. Most approaches attempt to solve one of the above problems rather than all of them at the same time. Quasi- and projective likelihood (Godambe (1960), Wedderburn (1974), Godambe and Heyde (1987), McLeish and Small (1992)) and adaptive inference (going back to Beran (1974), Sacks (1975) and Stone (1975); see Bickel, Klaassen, Ritov and Wellner (1993) for a quite comprehensive account) are, essentially, solutions to the efficiency problem. Dual likelihood and empirical likelihood for estimating equations (Kolaczyk (1994), Qin and Lawless (1994), Mykland (1995)) are solutions to the unconditional accuracy problem. The original empirical likelihood (Owen (1988, 1990), see also DiCiccio and Romano (1989) and DiCiccio, Hall and Romano (1991)) appears to have been motivated by both considerations.
58
MYKLAND
To what exent do these methods cope with the problems which they were not designed to solve?
2
Unintended Properties
As far as the author is aware, this question has not been heavily studied. In the following, we summarize what appears to be known about the properties mentioned above, (i) Optimality This is the high ground of quasi-likelihood and adaptive inference. If an empirical or dual likelihood is based on a quasi-score, it will have the same asymptotic efficiency as the score itself (Kolaczyk (1994), Mykland (1995)). One can also base such likelihoods on other scores, but this would be unnatural if one knows what the second moment structure is like. We are not aware of any work concerning any possible connection between adaptive inference and empirical or dual likelihood, (ii) Unconditional Accuracy This is what empirical and dual likelihood are good at, though things start breaking down in the presence of nuisance parameters (Lazar (1996), Lazar and Mykland (1996), Mykland (1996)). The quasi-log likelihood does not typically satisfy Bartlett identities of order higher than 2, so the accuracy properties of the R statistic and its cousins do not hold (cf. Mykland (1996)). If one wishes these properties, one can instead use projective likelihood (McLeish and Small (1992)), which is based on the same inferential ideas as quasi-likelihood. By virtue of the projective likelihood being a true Radon-Nikodym derivative, accuracy will be as for likelihood. There is no free lunch, however, as we shall see next. Little is known about the accuracy of adaptive inference beyond first order, (iii) Likelihood Surface This exists for quasi- and empirical likelihood, and can presumably also be defined in the context of adaptive inference. For the other two approaches which we are discussing, the issue is more problematic. Projective likelihood needs a reference parameter value; the log is on the form IΘ (Θ), and, typically, lθo(θι) + lθι(θ2) is not the same as /0O(#2) One can locate the reference value at the MPLE, but this is not a completely satisfying solution. For dual likelihood, this question is not fully explored. If it is a dual criterion function to a nonparametric likelihood (empirical or point process, cf. Section 6 in Mykland (1995)), one can presumably use the surface from the nonparametric quantity. For a 'pure' dual likelihood (based on an estimating equation only, with no nonparametric counterpart, such as Aalen's (1980, 1989) linear regression), we do not know whether a likelihood surface exists. O
59
LIKELIHOOD INFERENCE
(iv) Conditional Properties We are not aware of any work in this direction, so here is a first stab at this. Let us suppose that the quasi-score and its derivative is correctly specified, in the sense that they coincide with the first two derivatives in the 'true' (unknown) log likelihood. In this case, the quasi and projective R statistics are, obviously, second order locally sufficient in the sense of McCullagh (1984). This, however, is not the case for empirical or dual R. In regular cases, the argument goes as follows: by the Hajek-LeCam convolution theorem (see, e.g., Hajek (1969)), A = [/,/](#) + 1{Θ) is first order ancillary. In view of McCullagh (1984), there is a J5, B = O p (l), so that A = A + B is second order ancillary. Hence, by McCullagh (1984), cov(A, ϊ(θ)) + cum{Aj(θ)J{θ))
= o(n), and so
cov(Λ, -[/, ϊ](θ)) + cum(A, Z(0), i(θ)) = -var(iί) + o{n). This expression is o(n) only if var(A) is o(n), which, under standard assumptions, translates into A = O p (l). Since A = [lj](θ) + ϊ(θ) + d(θ) + O p (l), this means in turn that Idiβ + δ)-
ld(θ) = ld(θ + μ)-
ld(θ) +
Op{μ%
where Id and / are the dual and true likelihood, respectively, and where μ = δ — δ2/2. Hence, by McCullagh (1984), the dual (and hence the empirical) R statistic is second order locally sufficient only if the dual likelihood coincides with the true one (and hence with the quasi-likelihood) to second order locally at θ. This leads, inter alia, to the conclusion that the dual R is unconditionally more accurate than the quasi-i?, but conditionally less so! What if the second order structure is not known? One may then have a choice between overdispersed quasi- and empirical/dual likelihood. In this case, things are less clear. The issue is pursued in Lazar (1996). There are a number of other questions here. To mention a few: What about adaptive estimation? And the above only tackles second order local sufficiency. What about the large deviation properties documented in Skovgaard (1990, 1996), Jensen (1992, 1997) and Barndorff-Nielsen and Wood (1995)?
3
Conclusion
It has hopefully been illustrated in the above that there are a substantial number of unresolved issues in this area. Even more fundamentally, there are also more questions which need to be asked. If likelihood is the gold
60
MYKLAND
standard, then what are the properties of likelihood anyway? New ones keep being discovered, as the rich recent literature on the subject can testify. And are there criterion functions yet to be discovered which come closer to the gold standard than the ones we have discussed?
Acknowledgements This research was supported in part by National Science Foundation grants DMS 93-05601 and DMS 96-26266, and Army Research Office grant DAAH04-95-1-0105. The manuscript was prepared using computer facilities supported in part by the National Science Foundation grants DMS 89-05292, and DMS 87-03942 awarded to the Department of Statistics at The University of Chicago, and by The University of Chicago Block Fund.
References Aalen, O. (1980). A model for nonparametric regression analysis of counting processes, Led. N. Statist. 2 1-25 (Springer, New York). Aalen, O. (1989). A linear regression model for the analysis of life times, Statist, in Medicine 8 907-925. Barndorίf-Nielsen, O.E. (1986). Inference on full or partial parameters based on the standardized signed log likelihood ratio, to appear in Biometrika 73 307-322. Barndorff-Nielsen, O. E., and Wood, A. T. A. (to appear). On large deviations and choice of ancillary for p* and the modified directed likelihood. Bernoulli. Beran, ίt. (1974). Asymptotically efficient adaptive rank estimates in location models, Ann. Statist. 2 63-74. Bickel, P.J., Klaassen, C.A.J., Ritov, Y., and Wellner, J.A. (1993). Efficient and Adaptive Estimation for Semiparametric Models (Johns Hopkins). DiCiccio, T.J., Hall, P., and Romano, J.P. (1991). Empirical likelihood is Bartlett-correctable, Ann. Statist. 19 1053-1061. DiCiccio, T.J., and Romano, J.P. (1989). On adjustments based on the signed root of the empirical likelihood ratio statistic, Biometrika 76 447456. Godambe, V.P. (1960). An optimum property of regular maximum likelihood estimation, Ann. Math. Statist. 31 1208-1212. Godambe, V.P., and Heyde, C.C. (1987). Quasi-likelihood and optimal estimation, Int. Statist Rev. 55 231-244. Hajek, J. (1969). A characterization of limiting distributions of regular estimates, Z. Wahrsch. verw. Gebiete 14 323-330. Jensen, J. L. (1992). The modified signed likelihood statistic and saddlepoint approximations. Biometrika 79 693-703.
LIKELIHOOD INFERENCE
61
Jensen, J. L. (1997). A simple derivation of r* for cuved exponential families. Scand. J. Statist. 24 33-46. Kolaczyk, E.D. (1994). Empirical likelihood for generalized linear models, Statistica Sinica 4 199-218. Lazar, N. (1996). Some Inferential Aspects of Empirical Likelihood. Ph.D. dissertation, University of Chicago. Lazar, N., and Mykland, P. A. (1996). Empirical likelihood in the presence of nuisance parameters, Technical report no. 400, Dept. of Statistics, University of Chicago. McCullagh, P. (1984). Local sufficiency. Biometrika 71 233-244. McLeish, D.L., and Small, C.G. (1992). A projected likelihood function for semiparametric models, Biometrika 79 93-102. Mykland, P.A. (1995). Dual likelihood. Ann. Statist. 23 396-421. Mykland, P.A. (1996). The accuracy of likelihood, Technical report no. 420, Department of Statistics, University of Chicago. Owen, A.B. (1988). Empirical likelihood ratio confidence intervals for a single functional, Biometrika 75 237-249. Owen, A.B. (1990). Empirical likelihood ratio confidence regions, Ann. Statist. 18 90-120. Qin, J., and Lawless, J. (1994). Empirical likelihood and general estimating equations, Ann. Statist. 22 300-325. Sacks, J. (1975). An asymptotically efficient sequence of estimators of a location parameter, Ann. Statist. 3 285-298. Skovgaard, Ib (1990). On the density of minimum contrast estimators. Ann. Statist. 18 779-789. Skovgaard, Ib M. (1996). An explicit large-deviation approximation to oneparameter tests. Bernoulli 2 145-165. Stone, C.J. (1975). Adaptive maximum likelihood estimators of a location parameter, Ann. Statist. 3 267-284. Wedderburn, R.W.M. (1974). Quasi-likelihood functions, generalized linear models, and the Gauss-Newton method, Biometrika 61 439-447.
65
Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES
ESTIMATING FUNCTIONS IN SEMIPARAMETRIC STATISTICAL MODELS S. Amari The Institute of Physical and Chemical Research Saitama, Japan M. Kawanabe University of Tokyo ABSTRACT The geometrical structure of estimating functions is elucidated by information geometry in the framework of semiparametric statistical models. A condition which guarantees the existence of an estimating function is given. Moreover, the set of all the estimating functions is obtained explicitly when it is not null. The optimal estimating function is derived, and the maximum Godambe information is explicitly given. A geometrical condition is given which guarantees that the Godambe information is the maximal available information. Key Words: Semiparametric models; estimating functions; dual parallel transport; Godambe information; ra-curvature free; efficient score function.
1
Introduction
Godambe (1960, 1976) proposed the estimating function method as a generalization of the maximum likelihood method for parameter estimation. An estimating function gives a ^/n-consistent estimator by a simple and tractable procedure under certain regularity conditions. Moreover, the method is applicable even to semiparametric models. However, the class of estimators derived from estimating functions might not necessarily include the Fisher efficient estimator. Therefore, it is important to study efficiency of estimators derived from the estimating function method. It is another important problem to know how we can obtain the optimal estimating function. Recently researches on estimating functions have been developed and been applied to semiparametric models. It has been naturally understood that the
66
AMARI AND KAWANABE
optimal estimating function is given by projecting the score function to the linear space consisting of all the estimating functions (Small and McLeish (1989), Waterman and Lindsay (1996), Durairajan (1996), Chan and Ghosh (1996), Li (1996)). However, there still remain many important problems to be studied further. They are, for example, as follows : 1. To obtain a condition which guarantees the existence of estimating functions. 2. To obtain the linear spaces of all the estimating functions explicitly. 3. To obtain the amount of information, called the Godambe information, included in the optimal estimating function. 4. To obtain a condition which guarantees that the Godambe information is full, that is, equal to the maximal information in the general sense. The present paper studies these problems by using the Hubert bundle formalism of information geometry in the semiparametric context (Amari (1985), Amari and Kumon (1988)). This is a simplified version of the paper (Amari and Kawanabe (1996)), but this includes some new developments and a new example.
2
Estimating Functions in Semiparametric Models
Let p(x, 0, φ) be a probability density function of a random variable x with respect to a common dominating measure μ(dx), specified by two kinds of parameters θ = (0 1 , ,0 m ) and φ, where θ E Θ is a finite-dimensional vector, Θ is an open set of Rm and φ G Φ is a finite or an infinite dimensional parameter, typically living in a space of functions. The set of distributions S = {p(x, 0, ψ)} is called a statistical model with a nuisance parameter and is called in particular a semiparametric statistical model when Φ is infinitedimensional. Here θ is called the parameter of interest and φ is called the nuisance parameter. Let y(x, θ) = [yi{x, 0)], i = 1, , m, be a vector-valued smooth function of 0, not depending on φ, of the same dimension as 0. Such a function is called an estimating function (Godambe (1976, 1991)), when it satisfies the following conditions, E*>(M)]
= 0,
(2.1)
det\EθJdθy(x,θ)]\
φ 0,
(2.2)
Vθ,φ[\\y(x'θ)W2]
< °°>
Eθφ[\dθy(x,θ)\2]
< oo,
(2.3)
for all θ and φ, where Eβ denotes the expectation with respect to the distribution p(x,θ,φ), dgy is the gradient of y with respect to 0, i.e., the
67
SEMIPARAMETRIC MODELS
matrix whose elements are (dyi/dθi) in the component form, det | | denotes 2 the determinant of a matrix, and || y || is the squared norm of the vector 2 2 V > \\y\\ = ΣXί/ΐ) We further need that Jypdμ is differentiate with respect to θ and that integration and differentiation are interchangeable. The condition (2.1) is called the unbiasedness condition. When the above conditions (2.1) — (2.3) hold in a neighborhood N(φo) of φo, such y(x,θ) is called a local estimating function at ψQ. When an estimating function y(x, θ) exists, by replacing the expectation in (2.1) by the empirical sum, we have an estimator θ of θ by solving the estimating equation -,0)=O,
(2.4)
where a?i, ,a;n are n independently and identically distributed observations. This is called the estimating equation and such an estimator is called an M-estimator. It might be thought that the additive form (2.4) is too restrictive for obtaining good estimators. However, we shall prove that the Fisher efficient estimator is included in this class in the m-curvature free models. The asymptotic behavior of an M-estimator θ is known by the following theorem (Godambe (1976), McLeish and Small (1988) for example). Theorem 1 Under the ordinary regularity conditions, the estimator θ obtained from an estimating function y(x,θ) is consistent and is asymptotically normally distributed, with the asymptotic covariance matrix AV[Θ; y] = A-ιEθφ[yyΎ}{Aτ)-\
(2.5)
where the asymptotic covariance matrix is defined by AV[θ;y] = lir^nEgJiθ
- θ)(θ - Θ)Ύ],
(2.6)
A is the matrix defined by A =
Eθtφ[dθy(x,θ)]t
and the superfix T denotes the transposition of a vector or a matrix. Let T(θ) be a non-singular m x m matrix smoothly depending on θ. It should be noted that y*(x,θ) — T(θ)y(x,θ) gives an estimating function equivalent to y in the sense of yielding the same estimator. The present paper aims at obtaining the minimum value of AV[0;y]. Before that, we give two examples of semiparametric statistical models.
68
AMARI AND KAWANABE
Neyman-Scott problem and mixture models : Let {q(x,θ,ζ)} be a regular statistical model, where both the parameter of interest θ and the nuisance parameter £ are of finite dimensions. Let X{, i = 1,2, ,n, be n independent observations from q(xi >θ->£>%)-> where θ is common but ^{ takes a different value at each observation. Then, estimating θ from observations x = (xι, " >χn) is called the Neyman-Scott problem, where the underlying probability distribution
includes the nuisance parameters ξ 1 ? , ξ n as large as the number of observations. This problem can be treated by the following semiparametric model. Let us assume that the unknown ^ are independently generated subject to a common but unknown probability distribution having a density function φ(ζ) Then, the X{ are regarded as independent observations from the semiparametric model (2.7) where φ(£) is the nuisance parameter of function-degrees of freedom. This model is called the mixture model. This type of problems was studied by Neyman and Scott (1948) and has attracted many researchers (Andersen (1970), Lindsay (1982), Kumon and Amari (1984), Amari and Kumon (1988), Pfanzagl (1990) etc.). There are a lot of interesting and important examples in this class. A typical example is the following class of distributions of the form, q(x, θ, ξ) = exp{ξ s(x, θ) + r(s, θ) - ψ(θ, £)},
(2.8)
where s(x, θ) is a vector not depending on £ and is the inner product. Here, the distribution is of exponential type for £ when θ is fixed. 2. Blind separation of mixture signals : Let 5 α , α = 1,2, ,r, be r signal sources which produce r time serieses s α (t), t = 1,2, We assume that each sa{t) is an ergodic time series having the probability density qa{sa) at any ί. Moreover, si, , sr are assumed to be independent. Then, their joint probability is written as Ψ(S) = Π Qa(Sa)
(2.9)
α=l
at any t where s = (si, , sr). We assume that we cannot directly observe the r signals sa(t) but
69
SEMIPARAMETRIC MODELS we can observe their mixtures, r
Xi{t) = ΣMta8a(t),
» = 1, ••-,*•,
(2.10)
α=l
where M = (Mιa) is an r x r non-singular matrix consisting of fixed mixing coefficients Mιa. Then, the joint probability density function of xΎ = (#1, , xr) is given by
p(x) = where (2.12) If we know M or W, the original source signals s(t) are recovered from the observed x(t) by β(ί) = Ifa (t).
(2.13)
When we do not know M or W, we should estimate W from the observed ίc(t), ί = 1,2, , where the density functions ςi(si), , qr{sr) are usually unknown. Such a problem often occurs in medical or communication signal processing, and is called the blind separation of sources. See Amari et al. (1996). This gives a typical semiparametric statistical model, p(x, W, φ) = \W\φ(Wx),
(2.14)
where W is the parameter of interest and r s
ψ( ) = Π ^s^ a=l
is the nuisance functions.
3
Hubert Tangent Spaces and Score Functions
Given a probability density function p(x), let us consider a one-parameter statistical model p ( M ) = p ( * ) { l + tα(αO}, where t (0 < t < ε) is the parameter. The constraint Ep[a{x)} = 0
ΐ3-1)
70
AMARI AND KAWANABE
holds where Ep is the expectation with respect to p(#), because of fp(x,t)dμ(x) = ίp(x){l + ta(x)}dμ(x) = 1.
(3.2)
When t is small, p(x, t) is a small deviation in the direction of a(x) from p(x). The model (3.1) is a curve parameterized by t in the set of all the probability density functions. Let us consider the linear space of functions a(x) which satisfy
Ep[α(z)] = 0,
2
E p [{a(x)} ] < oo.
(3.3)
The set of all such a(x) is a Hubert space Hp with the inner product of a(x) and b(x) defined by
(3-4)
)Kx)]
The Hubert space Hp consists of all the deviations a(x) of probability distribution from p{x). The random variable (3.5)
a χ
( ) = T:
t=o
is the tangent vector of the curve (3.1) at p(x). This is the score function for the one-dimensional statistical model (3.1) parameterized by t. Given a semiparametric model S = {p(x, 0, <£>)}, we construct the Hubert space HQ denoting the set of all the deviations iτomp(x) = p(x, θ, φ). Since we have interest in estimating functions, we define it by 2
Hθψ = {a(x) \EθJa(x)} = 0, E ^ , [{a(x)} } < oo for all ψ'} restricting the space such that it consists of functions a(x) which are square f integrable at all p(x, 0, φ ) even when it is defined at (0, φ). The tangent directions along the parameter of interest are the score functions (x, θ,φ) = -^i logp{x, θ, φ).
(3.6)
Eβjμt] = 0
(3.7)
Ui
Obviously, and we further assume that U{ is square-integrable at any {θ,φ'). Then it belongs to HQ . We call the subspace spanned by these u^s the tangent subspace TQ along the parameter of interest. The vector score function is U = (lil,-
,ϋm).
We next define the tangent directions along the nuisance parameter. Let us consider a curve c(t) connecting functions ψ and φ' such that c(0) = ψ and
71
SEMIPARAMETRIC MODELS
c(ίo) = φr. We then have the one-dimensional statistical model p{x, 0, c(t)} parameterized by t. Its score function is given by
(3.8)
v(x,θ,φ,c) = — t=o
which we assume to belong to the HQ . This v is the tangent vector along c(t) of the nuisance parameter. There are infinitely many curves c(t) and the corresponding υ's, when φ is a function. Let τ£ be the smallest closed subspace including all such v's. We call it the nuisance tangent space. This is a closed subspace of HQ . Now, let us project the score function uι to the subspace orthogonal to T ^ , that is to (τ$
)
which is the orthogonal complement of T$ . The
result is the function uf = U{ — v that minimizes EQ [\u{ — v\2],
v e TQ .
E
The vector function u = (uf) is called the efficient score function and the uf are called the components of the efficient score function (see Begun et al. (1983), Amari and Kumon (1988), Small and McLeish (1989)). Let T$ be the subspace of HQ spanned by the components uf of the efficient score function. Let Tit be the orthogonal complement of TQ θ Tp . It is called the ancillary subspace and spans directions orthogonal to any changes in the parameter of interest and the nuisance parameter. We thus have the orthogonal decomposition of the Hubert space (see Amari (1987), Amari and Kumon (1988), see also Small and McLeish (1988)), (3 9)
H
θ«=τiφ®τL®τL
E
The matrix G
= (gfj) defined by using the efficient score function
(3.10)
g?j(θ,φ)=EθJufuf]
is called the efficient Fisher information matrix. Begun et al. (1983) proved that GE gives the Cramer-Rao bound of the asymptotic covariance of estimators 0, Ύ
lim nE \(θ - θ){θ - Θ) ] >
(GE)~1
(3.11)
for any asymptotically normally distributed unbiased estimators in a semiparametric model. There is, however, no guarantee that this bound is asymptotically attainable by choosing an estimating function even when φ is finitedimensional. (This bound is attainable when φ is finite dimensional by taking the joint m.l.e. (θ, ψ) of θ and ψ.) So we need to search for a new bound, called the Godambe information bound, attainable by an estimating function explicitly.
72
4
AMARI AND KAWANABE
Global Decomposition of Hubert Spaces
Let us temporarily fix a φo When φo is the true nuisance parameter,
gives a good estimator. However, uE{xi, θ, φo) is not an estimating function in general, because it does not satisfy the condition
An estimating function y{x,θ) should satisfy the unbiasedness condition (2.1) for all φ. Such a global structure is elucidated by introducing two parallel transports of the Hubert spaces along the nuisance space. Let a(x) be a random variable belonging to HQ . Let us fix 0, and consider the subset SQ = {p{x,θ,φ)\φ G Φ}. We define two parallel transports of a vector a(x) from HQ to HQ , (Amari (1987)). The following (e)
Uia(x)
= a(x)-Έθφt[a(x)],
(4.1)
are called the e-parallel transport and the m-parallel transport of a(x) from (#,<£>) to (0,<^/), respectively. The parallel transports are generalizations of the dual geometrical structures derived from the underlying e- and m-connections or e- and m-covariant derivatives (Amari (1985), see also Amari and Kumon (1988)), but we do not go into mathematical details of differential geometry. The following lemma shows an important property connecting the two parallel transports. The proof is immediate and hence is omitted. Lemma 1 The two parallel transports are dual in the sense that, for any two a(x),b(x) € HQ , the inner product is kept invariant when one is etransported and the other is m-transported to HQ ,,
Π^'Π^)
>
(4 3)
where the suffix {θ,φ) denotes that the inner product or the expectation is taken with respect to p(x, 0, φ).
SEMIPARAMETRIC
73
MODELS
It is remarked that an estimating function is e-invariant, (β)
(e)
because of (2.1) where J J operates componentwise. Now we rewrite the unbiased condition by using the parallel transport. Let us consider a curve ψ = φ{t), φo = <£(0), in the nuisance space. By differentiating (2.1) with respect to t along the curve φ = φ(t), we have
— Jp{x,θ,φ(t)}y{x,θ)dμ{x)
_
= J v{x, θ, φo}p{x, θ, φo}y{x, θ) dμ{x)
where
v= —\ogp{x,θ,φ{t)} t=o
is the nuisance tangent direction at φo along the curve φ(t). This holds for any ψo so that any estimating function y(x,θ) is orthogonal to v(θ,φ) at any point (θ,φ). However, from /(m)
(e)
ι(m)
v
( J » ) ^»)
v
/ θ,φ
'
(44)
/θ,φ 1
where v is a nuisance tangent direction at φ , the orthogonality condition at φ1 is transferred to that at φ by the m-parallel transports of v &t φ1. This shows that an estimating function y is orthogonal, not only to the nuisance tangent direction at any φ, but to the m-parallel transports from ψ1 to φ of the nuisance tangent directions at any φ'. To incorporate with this global structure, we define the enlarged nuisance tangent space Fg by
that is, FQ is the subspace of HQ spanned by the m-parallel transports of TQ , from φ1 to φ for all ψ1. It might occur that the m-parallel transport
74
AMARI AND KAWANABE
of a(x) e Tβ , does not belong to HQ because of
= OO.
E
θ,φ
In this case, just ignore it. We next project TQφ or Tξ
to the subspace orthogonal to F$ . The
resultant subspace is called the information subspace and is denoted by Fk . The subspace orthogonal to Fβ
and Ffo is called the shrinked ancillary
space FQ . We then have the following orthogonal decomposition of HQ
:
Obviously,
T' W
5
r- I?N
rpA
—v τ?A
Estimating Functions and Godambe Information
The decomposition (4.5) of the Hubert space HQ makes it possible to characterize the set of all the estimating functions. We first show an important lemma (see Amari and Kawanabe (1996) for the details of the proof). Lemma 2 A necessary and sufficient condition for a function w(x, θ) to be e-inυariant is that it belongs to Fk Θ FQ for some φ. Proof
When w(x,θ) is e-invariant, we have Eθ%φ[w(x,θ)]=0.
We have already shown that w belongs to FQ 0 FQ
for any φ in this case.
On the other hand, let w(x, θ) be a function belonging to FQ
θ FQ .
In order to show that it is e-invariant, we consider a path φ = φ(t) in the nuisance space and put f(t)=EθMt)[w(x,θ)]. Obviously, /(0) = 0 where φ(0) = <po> By differentiating this with respect to ί, we can prove that
for any t (see Amari and Kawanabe (1996)), showing that f(t) = 0 for any t. It is also proved that, when w belongs to FQ ®FQ for a <ρ, it automatically
belongs to Ffo θ Ffi at all φ'.
Π
75
SEMIPARAMETRIC MODELS
Lemma 3 Let y{x,θ) be an estimating function and let y((x^θ) be the projection of the i-th component ofy(x, θ) to Fk . Then y/(#, 0), i — 1, , ra, span Fk
at any φ.
Proof
By differentiating (2.1) with respect to 0, we have []
=0
at all φ, where (u,y) is a matrix whose elements are {u^yj). Prom this, we have where the components u{ of the information score u1 are the projections of the components of the score u to Fk . When y is an estimating function, this is non-degenerate from (2.2), proving the lemma. D Combining the above lemmas, we have the following fundamental theorem, which gives the set of all estimating functions. Theorem 2 Any estimating function y{x,θ) posed at any φ as a sum
= {yi(x>θ)} can be decom-
y(x, θ) = T{θ, φ)ur(x, θ, φ) + a(x, θ, φ),
(5.1)
where the component ai(x,θ,φ) of a belongs to Fn and T(θ,φ) is a nonsingular matrix. Conversely, any function y{x,θ) defined in the form of (5.1) at a fixed φo gives an estimating function provided the projections of the components yι{x,θ) to Fk , span Fk , at every ψ1. It is possible to choose a basis for the information scores such that T(0, ψ) becomes the identity at a φ. The theorem also shows a condition for the existence of an estimating function. Theorem 3 A local estimating function at φo exists when and only when Fk is non-degenerate, that is, m-dimensional. A necessary condition for the existence of a global estimating function is that Fk
is non-degenerate
at all φ. Proof The necessary condition follows immediately from (2.2) and (5.1). When Fk is non-degenerate, u\x^ 0, φo) is a local estimating function in a neighborhood of φoD Remark
When we treat local estimating functions, the definition of FQ
and hence the decomposition (4.5) should be defined locally.
76
AMARI AND KAWANABE
We now derive the optimal estimating function and the amount of information (Godambe information) derived thereby. Let us define G\θ, ψ) = EQJU^X, θ, φHu'ix, θ, φ)}τ].
(5.2)
Given y, we also define GA(θ,φ;a) = EθJaaT],
(5.3)
where any estimating function can be decomposed as y(x, θ) = u*(x, θ, φ) + a(x, θ, φ) where α G Ffi . It is immediate to show
Hence, we have the following result. Theorem 4 The asymptotic covarίance matrix derived from an estimating function is AV[£; y] = ( G 7 ) - 1 + (GI)~lGA{GIy1. (5.4) The estimating function u^x^θ^φo) where φo is fixed is the optimal estimating function at φo and the Godambe information is given by G1.
6 Curvature-freeness The information score u1 is different from the efficient score uE in general, and GE > Gι. (6.1) The quantity
GE - G1 = EθJ(uE - u1)^ - uψ]
(6.2)
is the loss of information caused by using estimating functions. However, in many cases, uE — u1 and GE = G1 for any θ and ψ. In this case, the estimating function method is fully efficient, if the optimal y is chosen. When does this happen? To answer this question, we consider the statistical submodel SQ = {p(x, θ, φ)} by fixing θ where φ G Φ is the only free parameter. The tangent vectors of SQ compose the nuisance tangent space Γ ^ . Let us consider the m-parallel transports of TQ , from (0, φ') to (0, φ) and see how it is
77
SEMIPARAMETRIC MODELS
different from TQ . A manifold in general is said to be flat or curvature-free when its tangent directions are the same at all the points. In the present case, we can compare two tangent spaces TQ and TQ , by the m-parallel transport of one to the other. We give formal definitions of m-flatness and m-information-curvature-freeness. Definition 1 A semiparametric statistical model S is said to be m-flat or m-convex, when the TQ , are invariant under the m-parallel transports, that is, (m)
63
Λ^ί
( >
for any φ, φ' and θ. When the m-parallel transports °fTn
, from (θ,φ') to
(θ, φ) does not include the TQ components for any ψ, ψ' and θ, that is, (m)
X ^ ί
)
(6.4)
the model S is said to be m-curυature free in the information directions, or shortly, m-information-curvature free. It is easy to see that, when S is m-flat, it is m-information-curvature free. When Sβ is not m-flat, SQ is curved in general, because its tangent directions change as ψ changes. Theorem 5 When S is rn-information-curvature free, GE = G1 for any θ and φ. Moreover, y(x,θ) — u^x^θ^φo) = uE(x,θ,φo) is the optimal estimating function at φo and is efficient at φo. It should be noted that most semiparametric models so far treated by many researchers are m-flat. The important role of the m-flatness in the estimation function method is noted by Amari and Kumon (1988), Amari (1987), and also by Bickel et al. (1993) under the name of convexity. The present result shows that the m-information-curvature freeness is essential, establishing a necessary and sufficient condition that the estimating function method is fully efficient. However, the optimal estimating function depends on the true φ so that there is still a serious problem of choosing a good estimate φo from observed data to derive a good estimating function. It is a merit of estimating functions that, even if we misspecify the true φ and choose a wrong <£>o, the estimator is still -y/n-consistent. A practical method of choosing a good estimating function is given by Amari and Kawanabe (1996).
78
7
AMARI AND KAWANABE
Examples
Example 1. Mixture model A mixture model is m-flat and is hence ra-information-curvature free. In particular, when a model is given by (2.7) and (2.8), the 0-score, the information score u 7 , and the nuisance score in direction α(ξ) can be calculated explicitly. The 0-score u is given by 1 r u = ——-—- / (ΘQS ξ + dnr — dβψ)φ(ξ) exp{£ s + r — ψ}dξ. p\x,u,φ) J
(7.1)
Noting that the conditional distribution p(ξ\s) of £ conditioned on s is written as
Pttk) =
™'~™
ψ
> ,
(7.2)
J φ(ξ)exp{ξ s-ψ}dξ the 0-score may be written as u = dθs EK|θ] + dθr - E[dθψ\s], (7.3) where E[ \s] is the conditional expectation. Similarly, the nuisance score in the direction of α(ξ) is given by (7.4) Therefore, v[a] depends on x only through s so that the nuisance subspace Tβ is generated by the random variable s(x,0). It is known that the projection of a random variable t to the space generated by Si is given by the conditional expectation E[φj] and the projection to the orthogonal complement is t — Έ[t\si\. Hence, the efficient score, which is the same as the information score in this case, is given by u1 = uE
=
u-
E[u|β]
s)l
(7.5)
where the vector notation should be understood appropriately. This gives the efficient estimating function. Example 2. Blind separation of mixture signals In order to assure the identifiability, we put further restrictions E[8a] = 0 2
E[(sa) } = 1
(7.6) (7.7)
SEMIPARAMETRIC
79
MODELS
for the source distributions (α = 1, are
, r). Then, the score functions (matrix)
U =
where
In the component form, the score functions at the distribution (W,φ) are uιa(x, W, φ) = lfa(sa)xi
-f MQI
(7-8)
where Mιa are the components of the mixing matrix M. We next search for the nuisance scores. Let us consider a small change in the form of ψ. We can write it as qa(s,τ)=qa(s){l + τaa(s)},
(7.9)
where r is the parameter to denote the changes of functions qa in the direction of α α (θ). The score function in the direction of the nuisance parameter φ in the direction of α = (aa) is given by the score function υ{a)
=
— ,
\ogp{x,W,φ{s,τ)} , r=o
(7.10)
The linear space spanned by the functions υ(a) is called the nuisance tangent space, T # ι V = span{v(α)}. (7.11) We then have the important observation that this model is m-informationcurvature free. So the information score is given by the efficient score. The information score is given by 2
F^φ = span{ζ(s α )s 6 , (s α ) - casa - 1},
(7.12)
where ca = E[(s α ) 3 ]. The above consideration gives a very effective learning algorithm to this problem of blind signal separation. See Amari and Cardoso (1997) for details.
80
AMARI AND KAWANABE
References [1] Amari, S. (1985). Differential-Geometrical Method in Statistics. Lecture Notes in Statistics Vol.28, Springer, New York. [2] Amari, S. (1987). Dual connections on the Hubert bundles of statistical models, in Dodson, C.T.J. (ed.) Geometrization of Statistical Theory. 123 - 152. ULDM, Lancaster. [3] Amari, S. and Cardoso, J. -P. (1997). Blind source separation - Semiparametric statistical approach. IEEE Trans, on Signal Processing, accepted. [4] Amari, S., Cichocki, A., and Yang, M. (1996). A new learning algorithm for blind signal separation, in NIPS'95 Vol.8, MIT Press, 757-763. [5] Amari, S. and Kawanabe, M. (1996). Information geometry of estimating functions in semiparametric statistical models. Bernoulli, 3, 29-54. [6] Amari, S. and Kumon, M. (1988). Estimation in the presence of infinitely many nuisance parameters — geometry of estimating functions. Ann.Statist. 16, 1044-1068. [7] Andersen, E. B. (1970). Asymptotic properties of conditional maximum likelihood estimators. J. R. Statist. Soc. B 32, 283 - 301. [8] Begun, J. M., Hall, W. J., Huang, W. M., and Wellner, J. A. (1983). Information and asymptotic efficiency in parametric-nonparametric models. Ann. Statist. 11, 432 - 452. [9] Bickel, P. J., Klaassen, C. A. J., Ritov, Y , and Wellner, J. A. (1993). Efficient and Adaptive Estimation for Semiparametric Models. Johns Hopkins University Press, Baltimore. [10] Chan, S. and Ghosh, M. (1996). Estimating functions: a linear space approach. Symposium on Estimating Functions, Univ. of Georgia. [11] Durairajan, T.M. (1996). Optimal estimating function for estimation and predictions in semiparametric models. Symposium on Estimating Functions, Univ. of Georgia. [12] Godambe, V. P. (1960). An optimum property of regular maximum likelihood estimation. Ann. Math. Statist. 31, 1208 - 1212. [13] Godambe, V. P. (1976). Conditional likelihood and unconditional optimum estimating equations. Biometrika 63, 277 - 284.
SEMIPARAMETRIC
MODELS
81
[14] Godambe, V. P. (ed.) (1991). Estimating Functions. Oxford University Press, New York. [15] Kumon, M. and Amari, S. (1984). Estimation of a structural parameter in the presence of a large number of nuisance parameters. Biometrika 71, 445 - 459. [16] Li, B. (1996). A minimax approach to consistency and efficiency for estimating equations. Symposium on Estimating Functions, Univ. of Georgia. [17] Lindsay, B. G. (1982). Conditional score functions : Some optimality results. Biometrika 69, 503 - 512. [18] Lindsay, B. G. (1985). Using empirical partially Bayes inference for increased efficiency. Ann. Statist. 13, 914 - 931. [19] McLeish, D. L. and Small, C. G. (1988). The theory and applications of statistical inference functions, Lecture Notes in Statistics 44. Springer, New York. [20] Nagaoka, H. and Amari, S. (1982). Differential geometry of smooth families of probability distributions. Technical Report 82 - 7, University of Tokyo. [21] Neyman, J. and Scott, E. L. (1948). Consistent estimates based on partially consistent observations. Econometrica 32, 1 - 32. [22] Pfanzagl, J. (1990). Estimation in Semiparametric Models: Some Recent Developments. Lecture Notes in Statistics Vol.63, Springer, New York. [23] Small, C. G. and McLeish, D. L. (1988). Generalization of ancillarity, completeness and sufficiency in an inference function space. Ann. Statist 16, 534 - 551. [24] Small, C. G. and McLeish, D. L. (1989). Projection as a method for increasing sensitivity and eliminating nuisance parameters. Biometrika 73, 693 - 703. [25] Waterman, R. P. and Lindsay, B. G. (1996). Projection score methods for approximating conditional scores. Biometrika 83, 1 - 13.
83
Institute of Mathematical Statistics
LECTURE NOTES — MONOGRAPH SERIES
Estimating Functions, Partial Sufficiency and Q-Sufficiency in the Presence of Nuisance Parameters Vasant P. Bhapkar University of Kentucky Abstract When there exists a statistic which has its distribution free of nuisance parameters, the optimality of the marginal score function can be investigated in the context of generalized Fisher information for parameters of interest. In the case of a partially sufficient statistic, i.e. a statistic sufficient for parameters of interest, the marginal score function is the optimal estimating function. With the new concept of q-sufficiency for parameters of interest, the marginal score function is operationally equivalent to the optimal estimating function. AMS Subject Classification: 62A20, 62F99 Key Words: Generalized Fisher information matrix, completeness of a family, partial sufficiency.
1
INTRODUCTION
The optimality of the conditional score function as an estimating function for the parameter of interest, in the presence of unknown nuisance parameters, was established by Godambe (1976) in the situation where there exists a statistic which is ancillary for the parameter of interest and which is also complete for nuisance parameters. Such a statistic has been termed a complete p-ancillary statistic in an earlier paper (Bhapkar, 1989). Refer to Liang and Zeger (1995) for a review of estimating functions theory, some discussion of optimality of the conditional score function under the conditions assumed by Godambe, and references to further work (for example, Lindsay 1982) to find approximately optimal estimating functions in more general situations. The analogous question concerning optimality of the marginal score function in the complete case, has not yet been satisfactorily resolved. Lloyd (1987) considered this question; however, his assertion of optimality of the
84
BHAPKAR
marginal score function has been shown recently by Bhapkar (1995) to be invalid. In a special case where the two appropriate components of the sufficient statistic happen to be independent, the marginal score function does turn out to be optimal i.e., the property (6.2) holds. In such a special case, the appropriate component possesses the property of p-sufficiency, symbolized by the relation (6.1); this p-sufficiency then implies the optimality of the marginal score. However, in the general case where the two components of the sufficient statistic are not necessarily independent, the property of p-sufficiency does not necessarily hold (as shown by the counter-example by Bhapkar, 1995). This paper now establishes a certain weaker property, referred in this paper as q-sufficiency; this property is symbolized by the relation (7.7). For example, in the case of a random sample X = (X\,..., Xn) from a normal population with mean μ and variance σ2, T = X is a complete pancillary statistic for σ 2 and 5 = Σi(Xi ~^)2 i s p-sufficient for σ 2 . Here S satisfies the property (6.1). However, in examples (4.1) and (5.1) S satisfies only the weaker relation (7.6), but not (6.1). Thus, here the marginal score function of 5 satisfies only the weaker optimality relation (7.7), but not (6.2), which is satisfied in the first example. In both these examples, the family of distributions of Γ, given S = s and 0, is complete for the nuisance parameter
φ. The optimality of marginal score function, in its weaker or stronger forms, is shown to be related to certain generalized Fisher information functions. Section 2 introduces the basic terminology, Section 3 considers information in a statistic (or its marginal distribution) as well as the information in the conditional distribution given the statistic. Section 4 discusses the special case where the marginal distribution of a statistic is free of the nuisance parameters, while Section 5 deals with the optimality of estimating functions. The case where the statistic happens to be sufficient for parameter of interest (i.e. p-sufficient) is described in Section 6 while the last section establishes a somewhat weaker property (viz. q-sufficiency) that holds in general in the complete case, and in another specific situation.
2
INFORMATION IN ESTIMATING FUNCTION
Suppose that the random variable X has probability distribution Pω and the probability density function (pdf) p(x; ω) with respect to a α-finite measure μ over the measurable space (χ,A). The parameter wGίl and assume that ω = (0, φ), where θ is the parameter of interest, φ is the nuisance parameter and 0, φ are variation independent in the sense that Ω = Θ x Φ, where Ω is an open interval in the d-dimensional space Rd, and d\ is the dimensionality
85
PARTIAL SUFFICIENCY oΐθ.
Let l'ω{X) = [dlogp(x]ω)/dω], be the row-vector of ω-scores, and its components [lβ(X),l'φ(X)] are referred to as 0-scores and φ-scores of X, respectively. We assume the standard Cramer-Rao type regularity conditions: R:
i. J p(x;ω)dμ(x) = 1 can be differentiated twice under the integral sign with respect to elements of ω; ii. The Fisher information matrix I(ω) — Eω[lω(X)l'ω(X)] definite (pd).
is positive
If θ is real-valued, g — g(x\ θ) is said to be a regular unbiased estimating function (RUEF) for θ if it satisfies the regularity conditions: RG:
i. Eωg{X;θ)=0,Eωg2{X;θ)
< oo;
ii. f g(x;θ)p(x;ω)dμ(x) = 0 can be differentiated under the integral sign with respect to elements of ω. Operationally, g(x; θ) = 0 is to be solved to produce an estimate θ = θ(x). In the d\-dimensional case, we shall still use the term estimating function for a real-valued function g = g(x; θ) if it satisfies conditions RQ', however to solve for θ we need an estimating equation g{x',θ) = 0, all elements of which satisfy conditions RQ. Furthermore, in order to ensure that the equation leads to a solution θ = θ(x), we also need the conditions: (a) the covariance matrix σg(ω) = Eω[ggf] is pd. (b) G(ω) = Eω[dg{X;θ)/dθ]
is nonsingular.
In any case, we denote by G the space of real-valued functions satisfying the regularity conditions RQ The generalized Fisher information for 0, when ω = {θ^φ) is the full parameter in the distribution Pω of X is defined (Godambe, 1984) in the one-dimensional case as 2
IG{θ;ω) = minEω [lθ(X) - u(X ω)} .
(2.1)
uζU
Here U is the space of real-valued functions u = u(x\ ω) such that (i) Eωu(X\ω) (ii)
= 0, Eωu\X\ώ)
< oo
(2.2)
Eω[u{X ω)g(X; θ)] = 0, for all g E G.
On the other hand, the information concerning θ in the RUEF g = g(x\ θ) is defined as
86
BHAPKAR
gι is then said to be more efficient than g if Igi(θ\ω) > Ig(θ\ω) for all ω with strict inequality for at least one ω. We have, then, the generalized information inequality (Godambe, 1984) (2.4)
Ig(θ;ω)
for all g EG. The multi-dimensional analogs of (2.1), (2.3) and (2.4) are (see, Bhapkar, 1989): IG(θ;ω)
= mmEω[lθ(X)-u][lθ(X)-u}'
Ig(θ;ω) Ig(θ;ω)
= G'(ω)σg (ω)G(ω), < IG(Θ]ω).
1
,
(2.5)
(2.6) (2.7)
Here, of course, A < B for non-negative-definite (nnd) matrices means B — A is nnd, min denotes the minimal matrix M* = M(u*) in the class of nnd matrices {M(u)} such that M{u) > M* for all tx, and u denotes a vector with all elements u G ZY, which satisfy (2.2). It has been proved (Bhapkar and Srinivasan, 1994) that IQ exists; in fact IG(θ',ω) = Eω[g*g*'},
(2.8)
where g* is the projection of IΘ{X) onto the space spanned by G. This G-form indeed generalizes the usual Fisher information matrix in the sense that ; θ) = I(θ) = Eθ[lθ(X)l'θ(X)]. (2.9) In order to prove (2.8) and the results in this paper we find the Hubert space technique quite useful. Another motivation for this is the need to tackle the identifiability problem posed by the distinction between a function involving parameter 0; e.g. g = g{x\θ) and a function involving full parameter ω, e.g. u = n(x α ), for the given ω. To treat ω as a variable, along with x, Bhapkar and Srinivasan (1994) introduced an arbitrary probability distribution π over the measurable space (Ω,/3) for "random variable" ω, and consider the joint distribution Pω x π of (X,ω) over
(Xxίϊ).
Thus in the Appendix the space C of real-valued functions c = c(x; ω) is defined such that 1. Ec(X]ω) = /c(x',ω)p(x;ω)dμ(x)dπ(ω) 2. Ec2(X]ω)
=0
87
PARTIAL SUFFICIENCY
Here we write E for joint expectation, while we write Eω for expectation, given ω. The technical discussion in this context, along with proofs where necessary, is given in the Appendix, in order to provide validity to statements in sections 4, 5 and 7 of the text of the paper, which specialize the general results for arbitrary π to the one-point distribution π at ω. Thus Cω denotes the space of real-valued functions c = c(x; ω) such that 1. Eωc(X\ώ) ΞΞ f c(x\ω)p{x\ώ)dμ{x) = 0 2. Eωc2(X;ω)
Similarly, Q denotes the closure G of the subspace G of functions g = g(x\ 0), which depend on ω only through θ. Then Qω denotes the space with π a one-point distribution at ω. The orthogonal complement of Qω in Cω is then
3
INFORMATION IN A STATISTIC
Suppose now (5, T) is a minimal sufficient statistic for the family {Pω ) : ω G Ω} of probability distributions Pύ ' of X. Without loss of generality, we replace hereafter X by (S,T) in view of the anticipated result stated here as a lemma (Bhapkar, 1991): Lemma 3.1 Under regularity conditions R for the distribution of X, and conditions RQ, I{GX\θ;ω)
= I{GS'T)(θ;ω).
(3.1)
Now we want to investigate the information contained in S alone, in relation to (5,T); similarly we want to find out the maximum information, concerning 0, for estimating function based on S alone, in relation to such maximum information in the functions based on both S and T. Let then /(s; ω) denote the marginal pdf of 5 with respect to measure ι/, and h(t\ ω\s) the pdf of conditional distribution of Γ given 5, the value of 5, with respect to measure τ?s, when (5, T) has pdf p(s, t; ω) with respect to μ. In view of regularity conditions R for p, we assume the following conditions for / and h: β*:
1. f f(sm,ω)dv(s) = 1 can be differentiated, twice under the integral sign, with respect to elements of ω; 2. f h(t;ω\s)dηs(t) = 1 can be differentiated twice with respect to elements of ω under the integral sign for almost all (y)s\
88
BHAPKAR 3. The Fisher information matrix J ( 5 ) (α;) = Eω[lω(S)l'ω(S)] exists where l'ω(S) = [d\ogf{s\ω)/dω\ is the marginal ω-score of 5; 4. For almost all (v)s, the information matrix of T, given s, viz I™{ω) = Eω[lω(T\s)l'ω(T\s)] exists where lω(T\s) = [dlogh{T;ω\s)/dω] is the conditional ω-score of T.
Let Cω denote the Hubert space of real functions c = c(s,t\ω) such that Eωc = 0 and Eωc2 < oo (see Lemma A.2), and Qω = Gω the closed linear subspace, where Gω is the set of functions g = g(s, t; θ) in Cω, which satisfy regularity conditions as in RQ. Hereafter we drop the subscript ω from Cω<>GωMω >- for simplicity of notation. Thus, we have (3.2)
C = G®U
where U is the orthogonal complement of Q in C; the elements u = u(s^ t\ ω) are orthogonal to g G Q i.e. Eω[gu] = 0 for all g G GNow we denote by C(S) the subset of functions c = c(s ω) in C, which depend on (s,t) only through 5. Similarly G{S) = G(S) denotes the closure of G(S), the subset of G depending on (s,t) only through s. Thus, G{S) = GnC{S). Note that C(S) is itself a Hubert space, which is a subspace of C, and the inner-products (or norms) for both spaces coincide for cχ,C2 G ^(5), in view of the fact that
G(S) is a closed linear subspace of C(S) and we denote the orthogonal complement of G{S) by V(5); thus
C(S)=£(S)ΘV(S).
(3.3)
Observe that V(S) contains both functions u — u(s\ω) in ZY, not depending on ί, and functions v = u(s;ω) = Eω[u(s,T]ω\s] for the remaining u in ZY, since Eω[gu] — 0 for all g eG(S) for any uGW. We can then define the generalized Fisher information matrix for 5, with respect to parameter of interest θ when ω is the full parameter, as = mmEω[lθ(S) in analogy with (2.5).
- u][lβ(S) - «/]',
(3.4)
89
PARTIAL SUFFICIENCY
Similarly, for each value s of 5, we consider the conditional distribution of T, given s, and define the Hubert space Cs of real functions cs = cs(t;ω) satisfying Eω(cs\s) = 0, Eω(c2s\s) < oo, (3.5) in view of Proposition A.2. The closed linear subspace £ s = (5S, and its orthogonal complement ys in Cs, are defined like Cs above; thus we have (3.6)
Cs = Gs®ys,
for each value 5 of S. The generalized information in the conditional distribution Pω \ given s, is defined now as
- ys] [lθ(T\s)
- ys]'\s} ,
(3.7)
in analogy with (2.5). Finally we define the subspace C(T\S) in C of functions c(s,£;u;) = cs(£;ω), G(T\S) as the subspace in G of functions (g(s,t]θ) — #s(£;0), G{T\S) = G i r l S ) , and y{T\S) as the subspace in C(T\S) of functions y(s,t\ω) — ys(t;ω). Thus these functions satisfy (3.5) a.e. (Pi ; ) and, thus, Q (T |S) C C{T \S) and {y(T \S) C C{T \S)). We also note that {G{T \S)
= gnc(τ\s)). We now have the following lemma from Bhapkar and Srinivasan (1994), which is the special version of Lemma A.3. Lemma 3.2 C = C{S)®C{T\S)
.
(3.8)
PROOF It is easy to see that if c E C(T \S), then c _L C(S). We want to prove the converse that if c _L C(5), then c G C(T \S). Let then c ± C(5), and Eω ( φ , T ; ω ) \s) = c*(β;ω). Since c* 6 C(5), and c ± C(5), we have Eω(cc*) = 0. But Eω{cc*) = ^ [c*(5;ω)β(l,(c|Sf)] = J5ω[c*2(5;ω)]. Hence Eω{c*2) = 0, which implies c*(5;ω) = 0, a.e. ( P ^ ) . Thus, c G C(T 15) and the lemma is established. Note that the decomposition of the space C into C(S) and C(T \S) corresponds to the fact that, for any ceC, Eω{c\s) E C(S), while c - Eω(c\s) E C{T\S). In view of Lemma A.4 we have C{T\S) = Q{T\S) Θ y{T\S). The generalized information for θ in the conditional distribution of T, given 5, is defined as (0; ω) = ^ e m m | s ) £ ω [i,(T |5) - y] [Iβ(T \S) - y]' .
(3.9)
90
BHAPKAR
Lemma 3.3 In view of (3.7) and (3.9), {
lS)
(3.10)
I G (θ;ω)>Eω[IG(θ;ω\S)] PROOF. The proof is immediately obtained by noting
min Eω \[as - ys] [as with a(s, t\ ω) = lθ{T \S) and as = lθ(T \s). We are now in a position to prove the following theorem; Theorem 3.1 Assume the regularity conditions R* for the marginal distribution of S and the conditional distribution of T, given S. Then
> i£V;") + -rgΊ5)(0;ω);
(3.u)
ω) > I(g)(θ;ω) + Eω[IG(θ )ω\S)}.
(3.12)
(i) ifτ\θ-,ω) and
(ii) I^Hθ
PROOF. Let n(u) = lo{S,T) - ix, m(i/) = lθ{S) - v and r(w) = IQ{T \S) — iϋ, where 1/(5; ω) = Eω[u \s] and w = u — v for any u eU (i.e. every element of u belongs to U). Then v 6 V(5), since u _L Q{S) implies E^ftep] = Eω\gu] = 0 for all g e G(S). On the other hand, if g = 0 is the only g in ^(5), then 1/ 6 V{S) in view of (3.3). Also, for all g e G{T |5), Eω[gw] = £?ω[flf(tι - v)] = ^[ptt] —JSα b^] = 0? a n d then tϋ E [V(T |5) in view of Lemma A.4. It is also true that w e y(T\S) if g = 0 is the only g in G(T\S). Now n(ϊz) = m(i/) + r(w) and we have Eω[n{u)n'(uj\ = Eω[m(u)m'{u)\ +
Eω[r(w)r'(w)].
Since we have 1/ G V(*S) and iϋ E y{T \S) for every tx € W, it follows that mmEω[n(u)ri(u)]
> min. Eω[m(u)mf(u))
+
min
i.e. inequality (i) holds. The inequality (ii) follows immediately from Lemma (3.3). Thus the theorem is established. Remarks, (a) The relation (i) would be a strict equality if there exists u — u* for which Eω[u* \s] = v* minimizes Eω[m(υ)m'(υ)] for v € V(S) and simultaneously y* = u* — υ* minimizes Eω[r(y)rf(y)] for y E y(T \S). The relation (ii) would be a strict equality if, furthermore, y* = y*s(t;w) where y* minimizes Eω[r(ys)rf(ys) \s] for ys E ys a.e. (Pώ )•
PARTIAL SUFFICIENCY
91
(b) The inequality (3.12) was earlier given as Theorem 4.2 in Bhapkar (1991), subject to the condition that Ic{θ\ω) exists. Since existence of such a generalized information matrix has been established by Bhapkar and Srinivasan (1994), such an existence qualification is no longer needed. Furthermore, the present proof is mathematically more rigorous. (c) For the special case discussed in next section, when S has distribution depending on ω only through 0, VG' (0; ω) reduces to 1^ (θ) in the inequality (3.12). Such an inequality was earlier established as Theorem 3.1 in Bhapkar (1991) subject to existence qualification. The previous comments (a) and (b) apply to this case as well. (d) The fact that we need a possible inequality in (3.11), rather than a strict equality, is seen from the following example: Suppose X = (X\1X2)1 where X\,X2 are independent N(φ, θ) variables. Then IG{Θ; ω) = 1/202, but forS = Xi,T = X 2 , wehave4 5 ) (0;ω) = 0, and J™(0;u;) = I{P{θ;ω) = 0 as seen from Example 4.2 Bhapkar (1991).
4
SPECIAL CASE
Now we deal with the special case where the marginal distribution of S depends on ω only through θ. Then every element of the marginal score function of 5, viz IQ(S) is a RUEF in view of condition R*{ϊ). We can now characterize all the RUEF in G in view of Lemma A.6 in the Appendix. Theorem 4.1 Assume that the joint distribution o/(5, T), given ω, satisfies the regularity conditions R* in Section 3, the marginal distribution of S depends only on θ, and the estimating functions g = g(s, t; θ), for parameter of interest θ, satisfy the regularity assumptions RQ Then any RUEF in G can be expressed in the orthogonal decomposition: g(s, t; θ) = b'(s, t; θ)lθ(s) + «>(*; θ)
(4.1)
where Eω[b(s,T]θ) \s] = α(ω), and g0 e G0(S), the set of RUEF in g(S) uncorrelated with IΘ{S). Remark (a) Although we are assuming here that ω is fixed (i.e. π is a one-point distribution at the given α;), as in most of the text, the strict validity of the statements with ω = (θ, φ) as variables follows from the proofs given in Appendix (e.g. Lemma A.6 for Theorem 4.1) where (5,T;ω) are variables. (b) Observe from Lemma A.6 that Q = Q{S,T) Θ Go(S), where Q(S,T) is defined by (A.10), and QX(S) C G{S,T). In particular, if G(T\S) is empty, G is not necessarily equal to G(S). See Example 4.1 below in this regard.
92
BHAPKAR
Example 4.1. Let X = (Xχ,...,X n ) be independent pairs Xi — (Yi,Zi), where Y{,Z{ are independent exponentially distributed variables with means φ and φθ respectively. Letting Y = £ ^ Y{ and Z = Σ% Z% ? ( ^ ^0 is minimally sufficient for ω = {θ,φ). Equivalently, (S,T) is a minimal sufficient statistic with S = Z/Y and T = Z. The marginal distribution of S is
free of >, and the conditional distribution of T, given s, has 2n-l
t
-t/δ(s)
e
where 5(s) = sθ φ/(s + θ). Here G(T \S) is empty (in view of completeness of Γ, given s, for (/> with fixed θ. We have lθ(s) = n(s - θ)/θ{s + θ). But there exists g = g{s,t]θ) =
t(s-θ)/θ2s
5
= b(s, t; θ)lθ(s) where 6(5, t; 0) = t{s+θ)/nθs so that E^ftls = 20.
OPTIMALITY OF ESTIMATING FUNCTIONS
A RUEF g* = #*(s, t; θ) in G is said to be optimal (Godambe and Thompson, 1974) for estimating the one-dimensional parameter θ, in the presence of unknown 0, if Ig*(θ;ω) > Ig(θ\ω) for all g G G, where Ig is the information concerning 0, defined by (2.3). The matrix analog of (2.3) is given by (2.6) for the case of multi-dimensional θ. Now we wish to consider the marginal score function of 5, when the distribution of S is free of the nuisance parameter φ. The general form of j G G has been shown to be (4.1) in this case. For simplicity of proofs here we consider the case of one-dimensional θ. We now show that go{s;θ) is the non-informative part of g in (4.1) in the sense that the information Igo(θ;ω) is zero; also if g(s, t\ θ) = g*(s, t; θ) + go(s-, θ) in (4.1), then Ig(θ; ω) < Ig* (0; ω) with strict inequality unless go(s; θ) ΞΞO.
In view of Theorem 4.1, g0 is a RUEF belonging to G(5), and it is uncorrelated with IΘ(S). Differentiating /go(s; θ) f (s; θ)dιy(s) = 0 with respect to θ under the integral sign we have
Hence Igo(θ;ω) = 0 in view of (2.3).
93
PARTIAL SUFFICIENCY
More generally, for g = g* + g0, where Eω[g*g0] = 0, we have Eω[g2] = Eω[g*2] + Eω[g2]. Also Eω[dg/dθ] = Eω[dg*/dθ]
+
Eω[dg0/dθ] = Eω[dg*/dθ].
Hence >
Λ
(Θ
IAΘM
2
~
Thus we have proved
Eω[g* ]
* K^ΪTEM)=l9iθ;ω)
τ ( a
λ
Proposition 5.1 If g(s,t\θ) = g*(s,t',θ)+go{s 1θ), where go(s;θ) is uncorrelated with IΘ{S), and also with g*, then Ig(θ;ω) < Ig+(θ;ω), with strict inequality unless go{s]θ) = 0. Now we consider the case where G(T\S) is empty. First suppose that S and T are independent. We plan then to show that Ig{θ;ω) < Iιθ{s){θ\ω) for any g G G. If G(T\S) is empty, then we may assume g = 6(s, ί; Θ)IQ{S) where £?α;[6|5] = a(ω) φ 0, in view of (4.1) and Proposition 5.1. But b(s,t]θ) = b*(t;θ) for some b* in view of independence of S and T. Thus, g — b*(T',θ)lβ(S), so that Eω{g2) = Eω(b*2)Eθ(l2θ{S)) = Eω{b*2)l(s\θ), where I^{θ) is the usual Fisher information in S. Also, dg/dθ = {db*/dθ)lθ(s) + b*{dlθ/dθ) so that
£
=
Eω[b*(T;θ)}Eθ
d2 log f(S-,θ) θθ2
Hence, from (2.3),
Since jEg(6*) < ^ ( ^ * 2 ) , we have Proposition 5.2 7/5 α n d T are independent, the distribution of S depends only on θ and G(T\S) is empty, then Ig(θ;ω) < The matrix analogs of Propositions 5.1 and 5.2 can be developed for the general case of vector θ in terms of the information matrix Ig(θ;ω), given by (2.6). Thus the marginal score function of 5, viz. IΘ{S), is the optimal RUEF for θ (uniquely except for multiplication by functions of θ alone), when G(T\S) is empty, if S and T are independent.
94
BHAPKAR
If the family of conditional distributions {P^ \φ G Φ} of T, given s foτφeΦ is complete for every fixed 0, a.e. (Pω ), then G(T \S) is empty. Is then IQ(S) an optimal RUEF if the distribution of S depends only on 0? Proposition 5.2 requires the additional assumption of independence of S and T to prove optimality of lβ(s). Lloyd (1987) asserted optimality of IΘ(S), without requiring independence, in the case of one-dimensional 0; a similar assertion was made by Bhapkar and Srinivasan (1993) for the general case. Both these assertions are now seen to be invalid in view of the following counter-example (Bhapkar, 1995). Example 5.1. (continuation of Example 4.1) In Example 4.1, for given 0 and 5, T is complete for φ and hence G(T\S) is empty. However /^(5)(0; ω) = n 2 /0 2 (2n + 1), while Ig(θ;ω) = n/20 2 > I^s\θ) = Ilθ{s)(θ;ω), for g = g(s,t θ) =t{s-θ)/θ2s. Note that although, here, both le(s) and g produce the same estimate, viz. θ = S,IQ and g are distinct functions, i.e., g is not of the type k(θ)lo(s), and these distinct functions lead to different information functions.
6
PARTIAL SUFFICIENCY OF S FOR θ
If the marginal distribution of S depends on ω only through 0, S has been termed partially sufficient (p-sufficient) for 0 if I{GX\θ',ω) = lW(θ).
(6.1)
Thus, it has been shown by Bhapkar (Theorem 3.3, 1991) that 5 is psufficient for 0 if the following condition holds: (i) The conditional pdf of T, given s, depends on ω = (0, φ) only through a parametric function δ = δ(ω), which is differentiate and is such that ω is a one-to-one function of η = (0,5), for almost all (PQ )S. Earlier the term partial sufficiency had been used by some authors to describe the property of statistic Γ when δ(ω) in (i) is φ itself. The property of partial sufficiency of Γ was, then, also referred to as S-sufficiency by Basu (1977), among others. There is another situation where the property (6.1) holds; in this situation, condition (i) is replaced by the following condition: (ii) The family of conditional probability distributions {Pω : φ E Φ} of T, given s, for φ G Φ is complete for each 0 a.e. {PQ ) , and (5,T) are independent. The proof that the equality (6.1) holds, under the regularity conditions R*,RG and the additional condition (ii), when S has pdf depending only
PARTIAL SUFFICIENCY
95
on 0, follows essentially from Proposition 5.2, in view of inequality (2.4). A direct proof has been given by Bhapkar (1990) under a condition (ii'), which is equivalent to (ii). In condition (ii7) T has been termed a complete p-ancillary statistic for θ. The independence of (5, T) in the completeness condition (ii) is crucial, as shown by Example 5.1, and thus the assertions in Lloyd (1987) and Bhapkar and Srinivasan (1993) appear to be invalid. If T is complete for 0, given s and 0, then the second terms on the right hand sides of both (3.11) and (3.12) are seen to vanish in view of (3.7) and (3.9). However, it does not follow that Ic{θ\ω) = IQ\0',UJ), in view of comment (d) after Theorem (3.1). Thus the equality (6.1) is not necessarily true when S-distribution depends on ω only through 0, and Γ is complete for >, given s and 0. When the relation (6.1) holds, i.e. when S is p-sufficient for 0, then the marginal score function of 5, viz. Zfl(s), is the optimal RUEF for 0, in the sense that Ig(θ;ω)
7
Q-SUFFICIENCY OF S FOR θ
If T is complete for >, given s and 0, then G(T\S) is empty. Lemma A.7 shows that all RUEF's for 0 can be represented in the form g(sjt θ) = bf{s,t;θ)lθ(s) +go{s), where Eω[b(s,T;θ)\s] = a(ω). Operationally, a dλvector g = g(s, ί; θ) can be considered an estimating function for d\-dimensional θ if the equation g{s,t\θ) = 0 can be solved to produce an estimate θ = θ(s,t). Thus, if Γ is complete for φ, given s and 0. g(8, t; θ) = B(8, t; θ)lθ(s) + go(s),
(7.1)
where all the elements of gr0 belong to Qo{S) and Eω[B(SjT;θ\s] — A(ω). In order to find the optimal g, which maximizes the information matrix Ig(θ\ ω)
96
BHAPKAR
given by (2.6), one may assume without any loss of generality that g0 — 0, in view of Proposition 5.2. Now solving the estimating equation B(s,t',θ)lθ(s)
=0
(7.2)
for producing an estimator θ = θ(s,t) is operationally equivalent, in fact, to solving the regular unbiased estimating equation IΘ{S) = 0. For instance, in the case d\ = l,b(s,t\θ) is not an unbiased estimating function since Eω[b(s,T',θ)\s] = a{ω) implies Eωb(S,T\θ) = a{ω) and a{ω) Φ 0 in view of the assumption that G(T\S) is empty. One can then define a weaker concept (or property) of sufficiency for the parameters of interest, viz, q-sufficiency, if there exists statistic S with distribution depending on ω only through θ such that the inequality Ig{θ;ω)
(7.3)
for all RUEF g holds for some g* of the form (7.2), which are operationally equivalent to IΘ{S), in the sense that the estimator θ = θ(s,t) is produced by the Z#(s) part. Note that in (7.2) the part 6(s,t;0), in the case d\ = 1, is not a RUEF itself in the sense that Eωb Φ 0. In the d\-dimensional case no row of B($, t\ θ) in (7.2) can be a RUEF itself. Thus, if T is complete for φ, given s and 0, then S is at least q-sufficient. If S and T happen to be independent, then S enjoys the stronger property of p-sufficiency. If S happens to be q-sufficient, one could then define the q-information matrix IQ{θ;ω) = mmEω[lθ(S,T) - u][lθ(S,T) - u]f], (7.4) where now u = u(S,T;ω) is permitted to have elements in U* which is the space of functions in C orthogonal to 5*, which is obtained from Q by excluding g(s,t;θ) = b'(s,t\θ) lθ{s) such that Eω[b(s,T;θ)\s] = a(ω), which is not of the form a*(θ). Thus Q* is the sub-space of functions g* = a'{θ)lθ{s) +go{s-,θ), where g0 e Go{S); hence Q* = G{S). Since Q* C Q, we have W D U, so that IQ(θ]ω)
(7.6)
we then have Ig(θ;ω)
< / | # ( f ) (0;ω) = I ( S ) (θ) - IQ(θ;ω)
(7.7)
97
PARTIAL SUFFICIENCY
for all g with elements in G* Another situation where S happens to be q-sufficient, but not necessarily p-sufficient, is somewhat similar to the one covered by condition (i) in Section 6 with the difference that the parametric function δ = δ(ω) does depend on s as well. Specifically, for the statistic (5,T), which is sufficient for ω, we have S-distribution depending on ω only through 0, and (iii) the conditional pdf of T, given 5, depends o n ω = {θ,φ) only through δs = δ(ω; s), which is differentiate such that ω is a one-to-one function of (0, £ s ), given s, for almost all s. Under condition (iii), the conditional pdf h(t;ω\s) = h*(t;δs\s). Then for the case d\ — 1 we have =
dθ
ί
dφ
Jd ( ω ; S)'
(7 8)
'
for a suitable function d( ; s), arguing as in Theorem 2.2 of Bhapkar (1991). In view of regularity assumption R* (ii) every element of dlogh/dφ belongs to the space ys, given s. The same is true oΐdlogh/dθ, i.e. IQ(T\S), because of the relation (7.8). Since Gs®ys is the orthogonal decomposition of C5, for every gs{t\ 0) E Gs we have Eθ[lθ(T\s)gs(T',θ)\s} = 0. (7.9) We have, thus, Eω[dgs(T',θ)/dθ\s} = 01
(7.10)
again in view of R* (ii); thus Eω[dg(a,T',θ)/dθ\8]=0 for every g eG(T\S). It then follows that Eω[dg(S,T;θ)/dθ]=0
(7.11)
for every g eG{T\S). Therefore, under assumption (iii), Ig(θ;ω) = 0 for g e G{T\S). Hence g is optimal only if g has the form 3(5, t; 0) = 6(s, t; Θ)IQ(S), in view of Proposition 5.1, where Eωb{S,T;θ) = α(ω) φ 0. Thus, arguing as in the complete case, S happens to be q-sufficient for 0 in the sense that the optimal g*(s,t;θ) is a RUEF operationally equivalent to lθ{s), and the property (7.7) holds, when assumption (iii) is satisfied.
98
BHAPKAR References
Basu, D. (1977). On the elimination of nuisance parameters. Journal of the American Statistical Association, 72, 355-366. Bhapkar, V.P. (1989). Conditioning on ancillary statistics and loss of information in the presence of nuisance parameters. Journal of Statistical Planning and Inference, 21, 139-160. Bhapkar, V.P. (1990). Conditioning, marginalization and Fisher information functions. Proceedings of the R.C. Bose Symposium, R.R. Bahadur (editor), Wiley Eastern Ltd., 123-136. Bhapkar, V.P. (1991). Loss of information in the presence of nuisance parameters and partial sufficiency. Journal of Statistical Planning and Inference, 28, 185-203. Bhapkar, V.P. (1995). Completeness and optimality of marginal likelihood estimating equations. Communications in Statistics, Theory and Methods, 24, 945-952. Bhapkar, V.P. and Srinivasan, C. (1993). Estimating functions: Fisher information and optimality. Probability and Statistics, Editors: S.K. Basu and B.K. Sinha, Narosa Publishing House, New Delhi, India, 165-172. Bhapkar, V.P. and Srinivasan, C. (1994). On Fisher Information inequalities in the presence of nuisance parameters. Annals of the Institute of Statistical Mathematics, 46, 593-604. Godambe, V.P. (1976). Conditional likelihood and unconditional optimum estimating equations. Biometrika, 63, 277-284. Godambe, V.P. (1984). On ancillarity and Fisher information in the presence of a nuisance parameter. Biometrika, 71, 626-629. Godambe, V.P. and Thompson, M.E. (1974). Estimating equations in the presence of a nuisance parameter. Annals of Statistics, 2, 568-571. Liang, K.Y. and Zeger, S.L. (1995). Inference based on estimating functions in the presence of nuisance parameters. Statistical Science, 10, 158-173. Lindsay, B. (1982). Conditional score functions: some optimality results. Biometrika, 69, 503-512. Lloyd, C.J. (1987). Optimality of marginal likelihood estimating equations. Communications in Statistics, Theory and Methods, 16, 1733-1741. Rudin, Walter (1974). Real and Complex Analysis, McGraw-Hill Book Company, New York.
Appendix Proposition A.I Let (X,A,ψ) be a σ-finite measure space and L2{X the set of all real-valued functions f for which j f2(x)dφ(x) < oo; then is a Hilbert space.
99
PARTIAL SUFFICIENCY
The inner product < /i,/2 > is then given by / fιf2dψ and the norm is defined by | | / | | 2 = J72ώ/> The proof is given in mathematical texts (see, e.g., p. 81, Rudin, 1974). Such a proof can be modified, as appropriate, to establish Lemma A.I Let L(X,ψ) be the set of all real-valued functions f for which f f(x)dψ(x) = 0 and f f2(x)dψ(x) < oo. Then L(X,φ) is a Hilbert space with the inner product and norm defined as before. Lemma A.2 Let C = C(X x Ω,μ x π) be the space of real-valued functions 2 c = c(x\ω) satisfying E(c) = /c(x;ω)p(x;ω)dμ(x)dπ(ω) = 0 and E(c ) = 2 f c pdμdπ < oo, where p(x;ω) is the conditional pdf of X, given ω £ Ω, with respect to a σ-finite measure μ, and π an arbitrary probability measure over Ω. Then C is a Hilbert space with the inner product < ci,C2 > defined as E(c\C2). Proof. The set C forms a linear space over the field R of real numbers. It remains to prove that C is complete, i.e. to show that every Cauchy sequence {cn} in C converges to an element c in C with respect to the norm ||c|| defined by ||c|| 2 = E(c2) = ίc2pdμdπ
= ί d2{χ-,ω)d<ψ{χ ω),
(A.I)
where d(x\ω) = c{x',ω)\p{x\ω)]ιl2 and ψ = μ x π. If {en} is Cauchy, there exists a subsequence {cnk},rtι < n 2 < . . . such that -cnk\\<2-\k
= \χ...
1
Let oo '
a
=
In view of the triangle inequality, and (A.2), we have k
Applying Fatou's lemma to {α^}, // hminf liminf
/
k
\\
{1
liminf liminf
r
I k -> oo a{ 1 pdφ oo / a{pdφ < 1,
i.e. ||α|| < 1. Thus a{x\ω) < oo a.e. (Ψ), and the series oo
ii+i(α;ω) - c n i ( ^ ; ω ) }
(A.2)
100
BHAPKAR
converges absolutely, say to c{x\ω) a.e^). Thus for cnk(x;ω) = cnι(x\ω) + lim cnk (x ω) = c(x; ω), α.e.(^).
(A.3)
We now show that c£C, and \\cn — c\\ -> 0 as n -> oo. For any e > 0 there exists N such that ||c n — cm \\ < e for m,n greater than N. Applying Fatou's lemma, for m > N we have r
/
(c - Cm) pdφ
=
liminf
2
/ k -> oo (c n i b - c m ) pdt/; liminf /• 2
2
< fc -> oo y (c n , - c m ) p # < e .
(A.4)
It follows that E(c2) = / c 2 p # < oo. Also - Cm + Cm\ = \E{c1 / 2- Cm)\ = 1\ J{c / 2
ΓΓ ΓΓ
1
- Cm)pd<ψ\
[J
in view of Cauchy-Schwartz inequality. Since fpdψ = f pdμdπ = f dπ = 1, we have |£7(c)| < e, in view of (A.4). Since e is arbitrary, this proves that E(c) = 0. Thus c eC, and ||c m — c\\ -» 0 as m -> oo in view of (A.4). Thus the lemma is established. Assume the regularity conditions #*, in Section 2, for the joint distribution of (5,T), given ω. Consider now the probability distribution π = πi x π2 of CJ = {β, φ) over θ x Φ. C is the Hubert space of real-valued functions c = c(s,£;ω), which satisfy E(c) = E(c 2 )
/ c(s, ί; ω)p(5, ί; ω)dμ(s, t)dπ(ω) = 0 / c2pdμdπ < oo .
=
See Lemma A.2 for the proof of assertion that C is a Hubert space. The subspaces (and their closures in C, as needed) considered later on are defined below: g is differentiable with respect to elements of θ } g U C(S) G{S) G(S) V(S)
= = = = = =
G
g1- so that C = G®U {c:c = φ ; ω) and ceC} {g:g G(S) ±
101
PARTIAL SUFFICIENCY For a given value s of 5, define Cs
= {cs : cs = c5(£;ω) ,-Eα,(cs |s) =
/ csh(t; ω \s )dηs (ί) = 0, a.e. (π) E(<ξ \s) = ίc2shdηsdπ{ω)
< oo} .
(A.5)
The proof that Cs is itself a Hubert space, given 5, is very similar to that of Lemma A.2; first we have Proposition A.2 Every Cauchy sequence {cs,nk},nι < n^ < . . . in Cs converges to an element in Cs; hence Cs is a Hilbert space. Proof. Arguing as in the proof of Lemma A.2, we have lim cSiTlk (t; ω) = cs(t; ω), a.e. (ηs x π)
(A.6)
>oo
in view of (A.3). It remains to verify that Eω[cs(T;ω) \s] = 0, a.e.(π); t h e remainder of the proof goes along the lines of Lemma A.2 proof. Now
Eω[cs \s] = J(cs - c s , m + ca,m)dPW8)
= J(cs - cs,
J(
(cs,nk
-
by applying Fatou's lemma to {cs,njc} as k -> oo. Hence JSω[cθ|s] < 0, in view of (A.6), noting that (A.6) holds a.e.(π). A similar argument applied to — cs gives the inequality — J E ^ C ^ S ] < 0. Thus, E^jcsls] = 0, a.e.(π). The proposition is established by arguing as in the proof of Lemma A.2. In C5, we define the subspaces Qs and ys as given below. = {9s : 9s = 9s (*; θ), gs G Cs and gs is differentiate with respect to θ} Gs = Gs ys = GsLmCs,i.e.cs = gs®ys
Gs
Finally we define in C C(T\S) G(T\S)
= {c:c = φ , ί ; w ) , c 6 C and c(s,t;ω) = cs(t;ω),cs e Cs a.e.(z/ x π)} = {g: g = g(s,t;θ),geG and g{s,t;θ) =gs{t',θ),gs e Gs a.e.(i/ x π)}
y(s,P,ω) = ys(t;ω),ys Lemma A.3 C = C{S) Θ C(T\S).
6 ^ 5 a.e.(ι/ x π)
102
BHAPKAR
PROOF Let c* = c*(s;ω) G C(S) and c = c(s,t;ω) G C(T\S). Then
E{cc*) = Jc*(s;ω) [|' c(a,t;u>)dP™] dP^dπ(ω) = 0 and thus C(S) ± C{T\S). Now we show that C(S)1 is C(T\S). Suppose now c J_ C(S), and let c*{s;ω) = Eω(c\s). Since c* G C(5), E{cc*) = 0. But
Hence c*(s;ω) = 0, a.e.(z/ x π). Thus c G C(T|5), and the lemma is proved. Lemma A.4 C(T\S) = C?(T|5) ®y(T\S). PROOF. If y G y(T\S) and5 G ί?(Γ|5), then
E(gy) = J gydP^dPWdπiω) = J [ | 5 ί y s dPf Is)] dP^WM = 0 Thus £(T|S) j . y(T|5). Suppose now c G CCΠS) and c ± ^(Γ|5). Since c G C(T\S), c = c(s,t;ω) = cs(t;ω), where cs G Cs, a.e.(^ x π). Consider the orthogonal decomposition of cs in Cs, viz
cs(t;ω)=g*s(t;θ) + y*s(t;ω).
Consider now g*(s,t;θ) = gl(t;θ); then 3* G G(T\S) and, hence, c ± g*, i.e. £?(<#*) = 0. But
E(cg*) = J'cg*dP^dP^d-κ{ω)
= J [J'cs(t;ω)g*s(t;θ)dP
Hence g*(s,t;θ) = 0, a.e.(μ x TΓI). Then gζ(t]θ) = 0 a.e.(^ x TΓI), which implies cs = y* a.e.(v x π) so that c(s,t;ω) = y*(s,t;ω), where y*(s,t;ω) = y*(t;ω). Thus c G ^( Γ| 5 ) , and the lemma is proved. Lemma A.5 Suppose that the distribution of S, given ω, depends on ω only through θ, Gι(S) = {gι:gi=gi(s;θ) = α'(θ)lθ(s)eg(S) for some α(θ)},
(A.7)
and £o(#) is the sub-space in £(S) orthogonal to Gi(S). Then G*(S,T),
(A.8)
103
PARTIAL SUFFICIENCY where G*(S,T) =
{g*:g*=g*(s,t;θ)eg<md
k*(s;θ) = Jg*(s,t;θ)dP™dπ2(φ)eGι(S),
a.e. (1/x πi)}.(A.9)
PROOF. When the distribution of 5, given ω, depends only on 0, then each component of IQ(S) belongs to G(S) in view of regularity condition R* (i). Since Q(S) is complete, it is a Hubert space with decomposition We also note that Q is a complete subspace of C and, hence, Q is a Hubert space. G*(S,T) is seen to be orthogonal to Go(S). It remains to show that G*(S,T) is the orthogonal complement of Go(S) in G> Let then g € G and suppose g ± Go{S). Then for all go G Go(S)
0 = E[gog] = J =
E\gok*].
Thus k* JL ^0(5). Since k* <Ξ G(S), it follows that k* G £i(S). Thus the lemma is established. Lemma A.6 Under the assumptions of Lemma A.5, let g{S,T)
that JBw[6(β,T;0)|s]
= {g:g = g{s,t;θ) = b'{sΛθ)lθ{s)
= α(ω), and g e G} .
such
(A.10)
(i) If there is a 5* e G*(S,T) orthogonal to £(S,T), thenfc*(5,6>)= 0, a.e. ( ) , and (ii) for every j G ? ( 5 , T ) , we have the orthogonal decomposition g = a*'(θ)lθ(s) + [b'(8,t;θ) - a*'(θ)]lθ(s),
(A.ll)
where Eω[b{s ,T;θ\s)] = α(ω), and /α(ω)cίπ2(<£) = a*{θ). Furthermore, (iii) if π is a one-point distribution at ω, then ^ has the orthogonal decomposition G = Go(S)®G(S,T). PROOF. Note first that G(S,T) C 0*(S,T). Also observe that ^i(5) C G{S,T). Hence if there is a g* G ζ?*(5,T) orthogonal to 5(5,T), then we have for all ^1 € ( ) 0 = E\gιg*] =
Jg1(s,θ)k^s]θ)dP{θS)dπι(θ)
in view of (A.9). However fc* G £1(5) in view of lemma A.5 and (i) follows. To prove (ii), note that the two components of g in (A.ll) are orthogonal. We need to show that the subspace spanned by the second component is
104
BHAPKAR
the orthogonal complement of Gi{S) in G(S,T). Let then g G G(S,T) and suppose g ± Gι(S). Then for all gx G Gι(S) 0 = E\gιg]
Jgi(s;θ)af(ω)lθ(s)dP{ΘS)dπ(ω)
= = =
J
a[(θ)lθ(s)ϊθ(s)a(ω)dP{θS)dπ(ω) ίa[(θ)I^(θ)a{ω)dπ{ω),
in view of (A.7) and (A. 10). Since this is true for all αi(0), we have Ja(ω)dπ2{φ) = 0. Thus (ii) follows. Finally, if π is a one-point distribution at ω, then ^[ff*^] = 0 and, thus, g* G G(T\S), which is a subset of G{S,T). Since 3* is orthogonal to £(S,T), it follows that g* — 0. Thus we have assertion (iii). Lemma A.7. Under the assumptions of Lemma A.5 and A.6, suppose π is a one-point distribution at ω. If now G{T\S) is empty, every g G G has a representation where 50 e (zoOS) and £„[&($,T;0)|s] = α(ω) 7^ 0. Remark. Although (A. 12) gives the general representation of Q for the case where G{T\S) is empty and π is a one-point distribution at ω, the decomposition (A.ll) shows that g in (S,T) belongs to Gi(S) only if Eω[b{s,Γ; θ)\s] = o*(0) for some α*, i.e. 6(s,ί;θ) = α*(θ) in view of assumption that (?(T|S) is empty. Then g is outside Q(S) only if J5w[6(s,T;0)|3] = a*(ω) for some function α* which depends also on φ in a non-trivial manner. The orthogonal decomposition of Q(S,T) into Gi{S) and its complement in G(S,T) as given in (A.ll) is now possible only as g(s, t; 0) = a'(ω)lθ(s) + [b'(s, t; 0) - α'(ω)]I,(*) where α(α ), for the given ω, depends non-trivially on co-ordinates φ of given ω.
105
Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES
ESTIMATING FUNCTIONS AND HIGHER ORDER SIGNIFICANCE D.A.S. Eraser, N. Reid, and Jianrong Wu University of Toronto ABSTRACT Estimating functions provide inference methods based on a model for specified functions of the response variable, such as the mean and variance. The inference methods can use the limiting distribution of the estimating function, or the derived limiting distribution of the estimating function roots, or the derived limiting distribution of the quasi-likelihood function. In fully specified parametric models more accurate inference can be obtained by using recently developed higher order approximations based on likelihood asymptotics. We consider the recent third order methods in the estimating function context using the quasi-likelihood function. We focus on inference for a scalar parameter of interest: profile quasi-likelihood for the scalar parameter is defined and we describe a method for using this to approximate significance probabilities. Key Words: Ancillary directions; asymptotic inference; estimating functions; profile likelihood; quasi-likelihood; tail area approximations.
1
Introduction
In a fully specified parametric model, quantities for testing a parameter or parameter component can be constructed from the score function, the maximum likelihood estimator, or the likelihood ratio statistic. These quantities are equivalent in first order asymptotic theory, although examples tend to indicate that the likelihood ratio statistic provides the most reliable assessment, and the score statistic the least reliable assessment of a parameter of interest. It is also possible by considering higher order asymptotics to derive a modification of the likelihood ratio statistic with much better inferential properties. In the estimating equations context, optimally weighted estimating functions serve as an equivalent to, or extension of, the score function, and the quasi-likelihood function serves as an equivalent to, or extension of, the likelihood function. Our goal in this paper is to explore the extension of
106
FRASER, REID AND WU
likelihood asymptotics to the estimating equation context. We assume that the parameter of interest is a scalar, as this is central to the methods of likelihood asymptotics. In Section 2 we summarize the main results for third order inference based on the recent likelihood based asymptotics, with emphasis on the approach developed in Eraser and Reid (1995). In Section 3 we discuss the application of these methods to the estimating equation context. Some examples are discussed in Section 4 and limitations of the current work and possible extensions are outlined in Section 5.
2
Likelihood and significance
We assume in this section that we have a log-likelihood function ί(θ) = t(θ\y), based on a continuous variable y with n coordinates, and that the parameter is partitioned as θ = (λ, φ), with φ a scalar parameter of interest and λ a vector of p — 1 nuisance parameters. We will denote the observed information function —i"(θ) by j(θ) and the nuisance parameter submatrix by jχχ{θ). The profile log-likelihood function is ίp{φ) = i(\φ,φ), where λ^ is the maximum likelihood estimate of λ with φ fixed. We will often write θφ for {\φ,φ)> and i(θφ) for the profile log-likelihood function. In fairly wide generality, an approximate p-value for testing the hypothesis HQ : φ = φo can be computed from either of the following formulas: Φ1(r,q) = Φ(r) + φ(r)(r-1-q-1)
, Φ2(r,g) = Φ{r - r " 1 log(r/ς)}
(1)
where φ and Φ are the standard normal density and distribution functions. These are approximations to the distribution function of r, typically with relative error O(n~ 3 / 2 ). The first formula, an extension of the Lugannani and Rice (1980) approximation, often gives better accuracy with exponential models, while the second formula, due to Barndorff-Nielsen (1986, 1991) avoids anomalous values outside [0,1]. In (1) the quantity r — r(φo) is usually the signed square root of the profile log-likelihood ratio statistic: r = r(φ) = sgn(φ - φ) • [2{£(0;y) - tφtfy)}]*
(2)
and q is a complementary first order quantity, the explicit form of which is determined by the problem. For example, in the case of a canonical exponential family model with no nuisance parameters, the first version of (1) is the Lugannani and Rice (1980) approximation, with q taken to be the standardized maximum likelihood departure for the canonical parameter (φ — φ){j{φ)}λ^2 In the general one parameter setting q can be taken to be
HIGHER ORDER SIGNIFICANCE
107
where it is assumed that there is a one-to-one transformation from y to (?/>, α), with a exactly or approximately ancillary, and ty(ψ',y) = dί(ψ]y)/dy and t"ψ]y(Ψ',y) = dly(φ;y)/dφ, with α held fixed. In the presence of nuisance parameters a general formula for q was established in Barndorff-Nielsen (1986, 1991) under the assumption that there is available an explicit exact or approximate ancillary statistic a such that the conditioned variable given a is a one to one function of θ. In the special case τ of a canonical exponential family /(y; θ) = exp{ψs4-λ £—c(λ, ψ)—d(y)}, the minimal sufficient statistic is a one-to-one function of θ and the expression for q can be simplified to
where ρ2{ψ,ψ) = |jλλ( Equation (1) with (2) also gives an approximation to the Bayes posterior marginal cumulative distribution function with the choice q= An alternative expression for q was derived in Eraser and Reid (1995) which provides a more easily implemented calculation for the case with effective variable of the same dimension as the parameter as covered by BarndorffNielsen (1986, 1991) and also handles the general case where the dimension of the effective variable is larger than the dimension of θ. The dimension reduction from y to θ is effected by constructing a new parametrization ?, which is obtained as the gradient of the log likelihood function at the observed data taken in p directions V = {v\... υp).
V
=
dyτ dy τ
dθ
(2,0,00)
(4)
The latter differentiation is for fixed values of appropriate pivotal quantities, as discussed in Eraser and Reid (1995). The quantity q can be viewed as a standardized maximum likelihood departure q = q(φ) = sgn(V> - φ) • \χ(θ) - χ(θφ)\{\j{θθ)(θ)\/\j{Xλ)(θφ)\}ϊ
(5)
where 3{βθ)Φ) ιs the observed full information matrix and j(χ\){θψ) is the nuisance information matrix, and both are recalibrated in terms of the new
108
FRASER, REID AND WU
parameterization φ(θ): see eq. (17) below. The scalar parameter χ(0) replacing φ(θ) is linear in φ(θ)
3
Quasi-likelihood and significance
The quasi-score estimating function (Wedderburn, 1974) is denoted here by UQ to suggest its nominal role as a derivative with respect to 0, uθ(θ;y)=μJ(θ)Σ-1(θ){y-μ(θ)}.
(7)
This is based on n observations y with mean μ(θ), variance matrix Σ(0), and location gradient μe(β) = (d/dθτ)μ(θ). The more general optimally weighted estimating function (McCullagh & Nelder, 1989) uθ(θ;y) = μJ(θ)Σ-\θ)d(y;θ)
(8)
is based on a vector d(y; 0) recording some version of departure of y from 0 with mean E{d(y;θ)\θ} = 0, variance matrix Σ(0), and location gradient μβ(θ) = E{(d/dθ) d(y;0);0}. A further extension to handle conditional means and variances given a conditioning quantity A(θ) is discussed in Hanfelt and Liang (1995). Under reasonable conditions a root θ of the quasi-score estimating equation uβ(θ;y) = 0 is asymptotically normal with mean θ and variance I^{θ) where = μJ(θ)Σ-1(θ)μθ(θ) (9) is the variance of the estimating function. A quasi-likelihood ratio for 02 versus 0χ is obtained (Wedderburn, 1974) as a line integral (10) this will in general be path dependent if the quasi-score does not form an integrable vector field. We examine quasi-likelihood as constructed from estimating equations (7) and (8). With 0 expressed as (λ,^), the equations have coordinates corresponding to λ and φ:
109
HIGHER ORDER SIGNIFICANCE
The roots θ and θψ of the estimating equations will typically be obtained by iterative solution of UQ = 0 and u\ = 0, for example as
λj
+1)
)
1
)
)
= λ!ί + / λ - λ (^ K(λ!ί ,V;?/)
(12)
Now consider the interest parameter ψ. As a definition of the corresponding profile log quasi-likelihood we take £PQ(Ψ) to be Hλ^ o ,^o)
= L .
nψ(λψ ? ^;y)#
(13)
where the effective integration curve C — {θψ} = {(λψ,^)} is along the path of constrained solutions θψ as suggested in Barndorff-Nielsen (1995). It would typically be calculated iteratively from the overall solution θ. The signed likelihood root is then given uniquely by r{φ) = s g n ^ - <ψ)[2{ίPQ{ψ) - ePQ(φ)}]ϊ
(14)
The nominal reparameterization φ(θ) is needed only along the profile curve C and is obtained by an integral paralleling (13); it uses a n n x p matrix of ancillary directions V = (υ\... vp) which will be discussed later in this section. Again integrating along C, we define VτΣ-1(θ)μθ{θ)dθ
in the estimating equation case (7), and Vτd^(y;θ)Σ-1(θ)μθ(θ)dθ
(15)
in the more general case (8) with the notation dy(y;θ) = (d/dyτ)d(y;θ) .
This involves an integration for each of p coordinates but most of the calculations are common and related to (13). The Jacobian of the parameter change from θ to φ. φθτ(θ) = VT(ξ(y]θ)Σ-1(θ)μθ(θ)
,
(16)
is needed only at θ and θψ and is a by product of the calculation for (15). The inverse ofφΘτ(θψ) gives the coefficients ψφτ(θψ) for the linear parameter χ{θ) defined by (6).
110
FRASER, REID AND WU
The nominal information matrices JΘΘΦ) and j\\{θψ) are calculated from gradients of the quasi-scores jθθ(θ)
= - M Θ )
(Θ)
The recalibrated information matrices for use in (7) are then obtained using (16): \W)0)\
=
\Jθθ(θ)\\φθτ(θ)\-\ 1
\j(χχ)Φψ)\ = \3χχ(θ^)\\ψlΦΦ)φχτ(θΦ)\-
(17)
We now have all the ingredients for using the third order formula (1) with (2), (5), and (6) except the n x p matrix of ancillary directions V = (υi,..., υp) at the observed data point. In most inference contexts the number of variables n will exceed the number of parameters p and some procedure is needed to effectively reduce the number of variables to p. Higher order asymptotics indicates that the appropriate procedure is to condition on an ancillary of dimension n — p, thus giving p free variables. The higher order approximation described in Section 2 needs only the value of the likelihood function at the observed data point, and the gradient of the likelihood in p directions tangent to the ancillary, which give the new parameter φ in (4). The only information needed concerning the ancillary is thus the array V of tangent directions. For third order inference the vectors V need to be tangent to just a second order ancillary (Fraser and Reid, 1995; Skovgaard, 1986) and for second order inference the vectors V need to be tangent to just a first order ancillary. In the estimating equations context where the model is specifed as Eyi = μ^, vaπ/^ = Σ(μi), and the components are independent, a first order ancillary can be derived using results from curved exponential family theory (Amari, 1985). The resulting directions in this case are given by V = μθτ(θ) = ^μ(θ)\§.
(18)
In the case of the more general estimating equation (8), a separate argument is needed to establish V.
4
Examples
As a first example we consider a mean and variance function corresponding to exponential regression: the coordinates yι have mean μι and variance μf
111
HIGHER ORDER SIGNIFICANCE
with μι = exp{α + β{x% — x)}. We assume interest centers on the regression parameter β. The corresponding estimating equations for a and β are ua
=
uβ
= Σ(xi - x)μ~l(yi - μi)
Σμ^iyi-μi) (19)
These integrate on a path-free basis to give the quasi-likelihood
£{a,β) = -Σexp{-α - β(xi - x)}y{ - na which in fact coincides with the actual likelihood for the exponential regression model. The corresponding profile quasi-likelihood for β thus coincides with the ordinary profile ίP{β)
= Σyiμ'1
+nά- ϋyφjβ
- nάβ
where μι = exp{ά + β(xi — x)} and fiiβ = exp{άβ + β{x{ — x)}. For this example the zth row of the matrix V of ancillary directions (18) for second order inference is
The ancillary directions for third order inference are also available (Eraser et a l , 1994): the ith row of V is
The implied statistical model has in fact location model structure with an exact ancillary so exact p-values are available for comparison. We consider a random sample of size 5 from the Fiegl and Zelen leukemia data given as Set U in Cox and Snell (1981). Table 1 shows the exact and quasi-likelihood based p-values for selected values of /?, the regression coefficient. The full sample size is n = 17, and for the full sample there is almost no difference among the first order, quasi-likelihood and third order methods. The first order method here refers to using the normal approximation for the profile log-likelihood root. Table 1. Exact and approximate p-υalues: exponential regression.
β -4.7 -4.3 -3.8 -3.4 -2.9 1.5 1.0 2.0
first order 0.9954 0.9899 0.9747 0.9505 0.8956 0.0177 0.0370 0.0096
using (18) 0.9952 0.9896 0.9747 0.9514 0.8994 0.0240 0.0466 0.0134
exact 0.9960 0.9911 0.9776 0.9564 0.8953 0.0249 0.0513 0.0109
112
FRASER, REID AND WU Our second example is a binomial regression model: we assume that
Eyi = μι, vary; = rriiμi(l — μ;), with rrii known and μι = a + βx{.
We
are using a non-canonical parametrization, as there is no exact conditional method for inference about β in this setting. For illustration we fit this model to data from Example 1.5 of Cox and Snell (1989). The data values are I
Xi
1 2 3 4 5
1.0 1.7 2.2 2.8 4.0
y% 4 4 2 1 1
TH
110 105 62 65 45
As y is discrete the arguments of Section 2 do not apply directly, but the ancillary directions given by (18) are easily computable. Table 2 compares the significance functions for the first order and higher order approximations Table 2. First order and second order p-values: binomial example.
β -0.0245 -0.0225 -0.02 -0.018 -0.0155 -0.0115 -0.0065 0.000 0.0055 0.009 0.0125 0.017 0.0195
first order 0.9953 0.9894 0.9735 0.9487 0.8948 0.7454 0.4956 0.2413 0.0956 0.0512 0.0259 0.0010 0.0056
using(lί 0.9939 0.9867 0.9678 0.9398 0.8826 0.7375 0.5061 0.2587 0.1074 0.0591 0.0307 0.0122 0.0071
In both these examples the quasi-likelihood obtained from integrating the score equation is identical to the log-likelihood. It is necessary to introduce some dependence in the score equations, along the lines of Liang and Zeger's generalized estimating equations, or to use an optimally weighted estimating function as at (8), to derive a quasi-likelihood that is not also a log-likelihood. Unfortunately, we have not as yet determined a way to compute the ancillary directions V for these more general problems. Further comment on this point is given in the next section.
HIGHER ORDER SIGNIFICANCE
5
113
Discussion
In the application of formula (1) the choice of q seems to be crucial, and seems to need an argument based on approximate ancillarity in the nominal model leading to the estimating equation. For example, Hanfelt and Liang (1995) give an example using a moment type estimating equation for the shape parameter of a gamma distribution, and compare the first order approximations based on the quasi-likelihood ratio statistic and the standardized root of the estimating equation. We tried combining the two statistics, essentially using the standardized root as the q in (1), but the approximation was worse than either of the first order approximations. The derivation of V above uses the fact that μι is a location parameter, and we do not at the moment have an argument that applies to non-location parameters, such as over-dispersion parameters or additional dependence parameters in Σ. It may be possible to incorporate over-dispersion parameters using Nelder and Pregibon's (1987) extended quasi-likelihood, which essentially gives a REML-type marginal likelihood for the scale parameters. Alternatively it might be possible to substitute a consistent estimate of the scale parameter into the profile log quasi-likelihood, and still have an improved approximation, but we have not investigated this. The recent third order asymptotic methods address various model features not handled by typical first order methods: (i) Third and fourth order moments of the distributions of the component variables are implicitly involved. (ii) The coordinates means μi{θ) can measure θ and thus ψ on differing measurement scales. (iii) Non linearity in the parameter of interest as a function of θ or μ(θ) is allowed for. While the first of these was the initial stimulus for recent asymptotics, (ii) and (iii) are perhaps of greater importance. In the estimating equation context, (ii) and (iii) are of particular interest while information for (i) is generally not available. Accordingly we have adapted the asymptotic methods to the estimating equation context primarily to handle the complications (ii) and (iii). However concerning (i) we note that the application of the asymptotic methods starts with the quasi likelihood and we feel it is appropriate to make the most accurate extraction of significance probabilities from that likelihood.
114
FRASER, REID AND WU Acknowledgements
This work was partially supported by the Natural Sciences and Engineering Research Council of Canada. References Amari, S.-I. (1985) Differential Geometric Methods in Statistics. New York: Springer-Verlag. Barndorff-Nielsen, O.E. (1986). Inference on full or partial parameters based on the standardized signed log likelihood ratio. Biometrika 73, 307-322. Barndorff-Nielsen, O.E. (1991). Modified signed log likelihood ratio. Biometrika 78, 557—64. Barndorff-Nielsen, O.E. (1995). Pseudo profile and directed likelihoods from estimating equations. Ann. Inst. Statist. Math. 47, 461-464. Cox, D.R. and Snell, E.J. (1981). Applied Statistics. London: Chapman and Hall. Cox, D.R. and Snell, E.J. (1989). The Analysis of Binary Data. London: Chapman and Hall. Eraser, D.A.S. and Reid, N. (1995). Ancillaries and third order significance. Utilitas Mathematica 47, 33-53. Eraser, D.A.S., Monette, C , Ng, K.W., and Wong, A. (1994). Higher order approximations with generalized linear models. Multivariate Analysis and its Applications, eds. T.W. Anderson, K.T. Fang and I. Olkin. IMS Lecture Notes Monograph Series, 24, 253-262. Hanfelt, J.J. and K.-Y. Liang (1995). Approximate likelihood ratios for general estimating functions. Biometrika 82, 461-477. Lugannani, R. and Rice, S.O. (1980). Saddlepoint approximation for the distribution of the sums of independent random variables. Adv. Appl. Prob. 12, 475-490. McCullagh, P. and Nelder, J.A. (1989). Generalized Linear Models. 2nd ed. London: Chapman and Hall. Nelder, J.A. and Pregibon, D. (1987). An extended quasi-likelihood function. Biometrika 74, 221-232. Skovgaard, I.M. (1986). Successive improvements of the order of ancillarity. Biometrika 74, 516-519. Wedderburn, R.W.M. (1974). Quasi-likelihood functions, generalized linear models, and the Gauss-Newton method. Biometrika 82, 439-447.
115
Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES
ON THE CONSISTENCY OF GENERALIZED ESTIMATING EQUATIONS BY BING LI
Pennsylvania State University
ABSTRACT We study the consistency of generalized estimating equations. Our consistency result differs from the known results in two respects. First, it identifies a specific sequence of consistent solutions to be the minimax point of a deviance function; this is stronger than the known consistency results, which assert only the asymptotic existence of a consistent sequence. Second, the minimax procedure applies and gives consistent estimate even when the generalized estimating equation itself is not defined, as would be the case if the mean function is not differentiate, or if the support of the random observations depend on the parameters. We also provide two practical criteria based on which we can decide whether a solution is consistent by fairly simple computations.
Key words and phrases: Quasi likelihood estimation; generalized estimating equations; deviance; projected likelihood ratio; Doob-Wald approach to consistency.
116
LI
1. INTRODUCTION Many data sets that arise from scientific research consist of repeated, or clustered, measurements, which results in dependence among the observations. It is therefore necessary to incorporate the dependence into the estimation of the parameters and the assessment of errors. The generalized estimating equations and related techniques are very effective for such purposes. See Liang & Zeger (1986), and Zeger &; Liang (1986). In the present paper we will establish the consistency of generalized estimating equations, and provide verifiable conditions under which consistency holds. Applying Crowder's general theory (Crowder 1986; Small & McLeish 1994, page 96), it is not difficult to demonstrate that, with probability tending to one, a generalized estimating equation has a consistent solution. However, here we aim at a more specific statement of consistency; that is, a specific sequence of solutions, which can be identified in practice, is consistent. This latter statement is important because, in many applications, a generalized estimating equation may either have multiple solutions or have none at all; See McCullagh (1990), Hanfelt k Liang (1995), and Li (1996). In the classical theory of maximum likelihood estimation, the more specific statement of consistency is known as the Doob-Wald statement, demonstrated by Doob (1934), Wald (1949), and Wolfowitz (1949). It is established via a property of likelihood functions, which states that the maximizer of expected log likelihood is the true parameter value. This implies that, under certain regularity assumptions, the maximizer of the averaged log likelihood converges in probability to the true parameter value. For estimating equations, however, there is often no such likelihood functions to which the Doob-Wald argument can be directly applied. This is because, unlike a likelihood score function which is by definition the gradient of the log likelihood, an estimating equation need not be the gradient of any potential function, and hence we have no function to maximize. See McCullagh (1990), McCullagh & Nelder (1989), Firth & Harris (1991), and Li & McCullagh (1994). However, Li (1996) pointed out that (i) the likelihood property used in Doob-Wald argument can be restated as: the minimax of the expectation of the likelihood ratio is the true parameter value, (ii) for many estimating equations it is possible to construct a function which behaves similarly as the log likelihood ratio, so that the minimax of the expectation of this function is always the true parameter value. Based on these observations Li (1996)
CONSISTENCY OF GEE
117
demonstrated that every minimax point of this function is necessarily a consistent solution. This leaves no ambiguity when the estimating equation has multiple solutions or has no solution at all. Moreover, the consistency of the minimax holds even when the estimating equation itself is not defined, as would be the case for quasi likelihood equation if the mean function is not diίferentiable or if the support of the random observations depend on the parameter. In this note we will extend the minimax approach of Li (1996) to demonstrate the consistency of generalized estimating equations. In addition, we will provide two practical criteria by which one can judge, via fairly simple computation, whether a solution is consistent. In §3 we demonstrate the consistency in the special cases in which the estimating equations do integrate to potential functions. In §4, we describe the minimax approach to consistency, and introduce a deviance function that will facilitate this approach. §5 contains all the technical preparations towards the proof of consistency: assumptions, lemmas, and examples. The consistency in general cases will then be demonstrated in §6. In §7, we provide two practical criteria for consistency. Finally, in §8, a numerical example is carried out to illustrate the use and effectiveness of these criteria.
2. GENERALIZED ESTIMATING EQUATIONS AND EANCILLARITY The data sets with which we shall be concerned are of the form {Yit : t = l,...,7iΐ,z = l,...,ϋf}. The observations are independent for different z, but dependent for different t within the same i. For example, {Yit : t = 1,..., n^} may be the measurements from the same subject at different times. Associated with each Yit are a p-dimensional explanatory variable Xu and a p-dimensional regression parameter /?, which together determine the expectation of Yit. We will denote the whole vector {Yit} by Y, and the sample space of Y by y. Let μit(β) and φ~ιVit(β) be the mean and variance of the observation Yit. We shall assume that μit{β) = μ{Xjtβ) and Vit(β) = V{X%β) for some known functions μ( ) and V( ). The dispersion parameter φ is always taken to be positive. The inverse of μ is often called the link function, and Xftβ the linear predictor. Typically, we take μ""1( ) to be the natural link function from a linear exponential family and V(-) to depend on μ according to that family, but in principal they can be other functions as well. See McCullagh & Nelder (1989). The dependence within each i is modeled by correlation matrices R(a) — {Rw(a) : t,£' = 1, ...,τij}, where a is an s dimensional parameter and Rtt'{ )
118
LI
known functions. Across i, Rte(ά) is assumed to remain constant, in so far as both observations Yit and Yit» are present in the cluster i. In other words, R(a) is the same for all i except for its dimension. In practice, the matrices R(a) need not be correctly assumed, and is called the working correlation matrices. Whether or not R(a) is correctly assumed, we can formally calculate the covariance matrices of Yi = {Yit : t = 1,..., Πi} based on i?(α), as
These are called the working covariance matrices. If the working correlation assumption were correct, then for each fixed φ and α, the optimal linear combination of {Yn — μu(β)} which yields the highest information about /?, in the sense of Godambe (1960), is q{β, φ, a) = φ~ι Σ{μi{β)}T{Wi{β,
φ, α ) } " 1 ^ " Mi(/*)} = 0,
(1)
where μι(β) is the rij x 1 vector {μu{β) : t — l,...n»}, /x»(/?) is the Πi x p dimensional gradient matrix of μi(β) Equation (1), considered as the estimating equation for β given φ and α, combined with any y/Kconsistent estimate φ(β) of φ, and V^-consistent estimate ά{β, φ(β)} of α, is the generalized estimating equation for β. In other words, we estimate β by solving the equation
If R(a) is the true correlation matrix, and if a is known, equation (2) reduces to the quasi-likelihood equation as defined by Wedderburn (1974) and McCullagh (1983). In this case, it is known that any solution to (2), if it is consistent, is asymptotically normally distributed, and is efficient among the solutions to linear estimating equations. See also Jarrett (1984), McLeish (1984), Godambe & Heyde (1987). In the sequel, we will denote the nuisance parameter (φ,aτ)τ by τ τ 7, and the combined parameter (β,j ) by θ. The parameter space for β will be written as B. Notice that if the working assumption is incorrect, then the nuisance parameter 7 may have nothing to do with the underlying distribution P. Therefore we will use Eβ to represent the expectation under the distribution P for which β = β(P)- We denote the true value of the regression parameter as /?0, and will often abbreviate EβQ by E.
119
CONSISTENCY OF GEE
What is remarkable is the way in which the nuisance parameters 7 enter into the equation (2). Specifically, Eβ{q{β,>y)} = Q for all 7.
(3)
Property (3) is called the E-ancillarity of the estimating equation q relative to the parameter 7; See Small k McLeish (1988, 1989). This is a fundamental ingredient of a generalized estimating equation, out of which arise most of the desirable properties of the method. In particular, as implicit in Liang & Zeger (1986), if R(-) is correct, substituting \fK consistent estimates of φ and a into (1) does not impair the efficiency of the estimate of/?; even if i?( ) is incorrect, in which case ά may not estimate anything, substituting φ and ά into (1) does not impair the consistency and asymptotic normality of /3, as long as y/K(ά — a) is bounded in probability for some a. Property (3) also plays a vital role in our demonstration of consistency.
3. DOOB-WALD CONSISTENCY IN SPECIAL CASES If an estimating equation does integrate to a potential function, then, under certain conditions, the Doob-Wald argument can be extended to verify that the global maximum of the potential function is consistent. In this section we sketch the proof of the consistency of quasi likelihood estimation in these simple cases. The argument can be extended to generalized estimating equation (2), provided that, with respect to the regression parameter /?, it integrates to a potential function for each fixed 7. This extension is fairly straightforward and will be omitted. For further studies of the potential functions of estimating equations, see Li & McCullagh (1994). Let Y = {Yi : i — 1, ...,n} be independent observations with mean μi(β) and variance Vχ(β) where, similarly as in §2, μι{β) = μ{Xiβ) and Vi(β) = V(Xiβ) for some known functions μ and V. Let s(β, Y) be the quasi score function
s{β,Y) = ΣμiWVΓHβM
- μi(β))
(4)
i=l
PROPOSITION 1. Suppose that the quasi score integrates to some potential function /(/?, Y); in other words (dl/dβ)(β, Y) = s(β, Y), and that there is a measure v with respect to which the integral of 1^^ does not depend on β; that is
ί
(5)
120
LI
Then, under certain regularity conditions, the global maximum ofl(β, Y) is a consistent estimate of βo. PROOF. The argument is similar in spirit to that used in Gourieroux, Monfort, and Trognon (1984). For each β, define a probability measure dQβ(y) = eι^^du(y)/c. Differentiating equation (5) with respect /?, we find that μ(β) = f ydQβ(y). Since /(/?, y) is a linear function of y, it follows that El(β, Y) = f /(/?, y)dQβo{y) for all β. Hence
E{l{β,Y) - l(βo,Y)} = J{l(β,y) < log J
l(βoMdQβo(y)
eι^~ι^dQβo{y)
= log J{dQβ(y)/dQβ0(y)}dQβ0(y) = 0. That is, the expectation El(β,Y) is maximized at β0. The rest of the argument follows Wald (1949), with appropriate regularity conditions imposed to ensure that n~ιl(β, Y) converges in probability to n~ιEl{β, Y) uniformly in β. D
4. A DEVIANCE FUNCTION AND A MINIMAX APPROACH The argument of the last section depends on the existence of the potential function /(/?, Y), and is inapplicable if no such function exists, as is the case for many applications. Thus the questions arise: What is the crucial element in Doob-Wald argument? Must one have something like a likelihood function to maximize in order to apply this argument? At first sight, the existence of a likelihood function seems essential: if there is a function /(/?, y ) , whose expectation is uniquely maximized at /?o, and if f(β,Y) converges in probability to Ef(β,Y) uniformly in /?, then the maximizer of f(β,Y) will be a consistent estimator of the maximizer of Ef(β,Y). But a more careful look at the DoobWald argument reveals that the maximization of a likelihood is not indispensable: if we can uniquely identify the true parameter value βo by examining the function Ef(β,Y), may it be the maximum, the minimum, the turning point, or the minimax, then the β that can be identified in the same way by empirical version of Ef(β,Y), namely /(/?, y ) , should be a consistent estimator of βo. To illustrate this idea, let {Yί,..., Yn} be independent and identically distributed observations with a common density Pβ(yi). Since βo is the maximizer of Eβo logp^(Yi), it is also the minimax point of the function
121
CONSISTENCY OF GEE lo
γ
γ
Eβo &{Pβ'{ \)l Pβ( ι)}
I n
sup Eβo log{pβ>(Yι)/pβo(Y1)} 'eB
β
symbols, = inf sup Eβo \og{pβ>(Y1)/pβ(Y1)}. βεBβ'B ι
This suggests that the minimax of n~ Σ\og{pβι(Yi)/pβ(Yi)}, namely the β defined by the relation ^ _
_
, .„.,#)} = irfsup^iogW(^)/^(^)},
should be a consistent estimator of β0. This is indeed the case: it can be easily verified that β defined above is identical to the maximum likelihood estimate in this case. See Li (1996). The passage from the maximum of a likelihood to the minimax of a likelihood ratio is important because, unlike the likelihood, the likelihood ratio can be generalized to many estimating equations, so that the minimax argument applies very generally. Li (1993a) introduced such a generalization to quasi likelihood equations. This is then further extended in Li (1993b) to generalized estimating equations, which is now recorded below. DEFINITION 1. Let, in the notation of 12, θλ = (βuφuaj)τ and θ2 = (β2,φ2,a2)τ be two points in the parameter space θ. Let ι D : θ x θ x y H> R be the function D(θuθ2) =
The deviance function of the generalized estimating equation (2) is a mapping R : B x B x y *-ϊ Rι defined by
R(βu β2) = D{βx, φifa), ά(βu φiβ,))- fa φ(β2), ά(ft, φih))}The centering function of R(βι,β2) is the mapping J : B x B H-> j?1 defined by J(βuβ2)
=
ED{βι,Eφ(β1),Eά(β1,Eφ(β1)); β2,Eφ(β2),Eά(β2,Eφ(β2))}.
122
LI
Notice that the dependence of D and R on the random observations is suppressed from the notation. The next proposition provides some elementary properties of J, which can be verified along the lines of Li (1993a). P R O P O S I T I O N 2. The centering function J(βuβ2) lowing properties:
(i) Jtfufo) (ii) J(βo,β)
has the fol-
= "(&, βι) for all β1, β2 in B is the negative quadratic form
, Eφ(β), Eά(β, Eφ(β))}{μi(β)
-
(βo)}.
μi
If the matrices W's in (ii) are all positive definite, and if β is identifiable by the assumption of the mean; that is, different βs correspond to different sets of means {μit(β)}, then J(βo,β) < 0 for all β in B. This, combined with (i), implies that supJ(/%,/?)=inf Thus, intuitively, if R(β,β') converges to J(β,β'), satisfies the minimax relation
then any β that
sup #(/?,/?) = inf sup R(β,β') should be a consistent estimator of /?0 This will be proved rigorously in the next two sections.
5. ASSUMPTIONS AND LEMMAS We shall frequently use the condition of stochastic equicontinuity, which can be found in Pollard (1984, page 139). The following definition is a combination of the condition Cl and Lemma 2.1 of Crowder (1986); it is to assume that a sequence of random functions and the corresponding sequence of centered random functions are both stochastically equicontinuous. Let T be a compact set in a Euclidean space, let {/ n (ί;Xχ,...,X n ) : n = 1,2,...}, abbreviated as {/ n (ί,X)}, be a sequence of random functions defined on T, and let {fn{t) : n = 1,2,...} be a sequence of (deterministic) functions of ί, considered as the "centering" functions of {/n}.
123
CONSISTENCY OF GEE
DEFINITION 2. Let T be a compact set in a Euclidean space. The sequence of random functions, {fn(t,X)}, and the associated centering sequence, {/n(*)}, o,re said to obey condition Cl if the following are satisfied: (i) {fn(t,X)} is stochastically equicontinuous on T; that is, for each e > 0, η > 0 there is a positive number δ > 0 such that, limsupPί sup
\fn(t,X)-fn{s,X)\>e}<η
,
(ti) {fn{t)} is equicontinuous on T; (in) There is t0 G T such that {/n(ίo) n = 1, 2,...} is bounded. By the Arzela-Ascoli theorem (Conway 1990, page 175) and the compactness of T, (ii) and (iii) imply that {fn(t)} is totally bounded. Therefore, if {φ(β)} and {Eφ(β)J obey Cl, then the set Fo = {Eφ(β) : β e B,n = 1,2,...} is contained in a compact set, F say. And, without loss of generality we assume that F is such that inf{\φ — φ'\ : φ e F o , φ1 G Fc} > CQ for some e0 > 0. Suppose that, in addition to Cl, we have the convergence φ(β) - Eφ(β) 4 0.
(6)
By Lemma 3.2 of Crowder (1986), the pointwise convergence (6), together with assumptions (i) and (ii), implies the the uniform convergence snpβeB \φ(β) — Eφ(β)\ A 0. It follows that, with probability tending to one, the set {φ(β) : β G B} lies in F , because P{φ(β) e F for all β} > P{\φ(β) - Eφ{β)\ < e0 for all β} =
P{8xιp\φ(β)-Eφ{β)\<eo}-+l.
This proves the following result. LEMMA 1. Suppose {φ(-),Eφ( ) : n = 1,2,...} obeys Cl, and suppose (6) holds. Then, with probability tending to one, {φ{β) : β G B} is contained in a compact set F. With this construction, the condition Cl is easily passed from simple functions to composite functions, a passage of importance since we are to study the limit behaviour of the substituted estimates such as ά(β, φ(β)) as a random function of β. LEMMA 2. Suppose {<£(•), Eφ{ )} obeys Cl on B, and {ά( , •), Eά{; •)} obeys Cl on B x F. Then {ά( ,φ(-)),Eά(ΊEφ( ))} obeys Cl on B.
124
LI
PROOF. By Billingsley (1968, page 221), conditions (ii) and (iii), as applied to {φ( ),Eφ( )} and {ά( , ),Eά( , •)}, imply that the sets {Eφ(β) : β E B,n = 1,2,...} and {Eά(β,φ) : (β,φ) <E B x F,n = 1,2,...} are bounded. So for any β £ B, sup\Eά(β,Eφ{β))\ < sxxpsup\Eά(β,φ)\ < supsupsup|£ά:(/3,(/>)| < oo. «
n φeF
n βς-B φeF
Hence (iii) holds for {Eά( ,Eφ(-))}. To prove (ii), let e > 0, and let δo > 0 be such that limsupsup{|£ά(/?, φ) - Ea(β', φ')\ : \\β - β'\\ < δ0, \\φ - φ'\\ < δ0} < e. n—ϊoo
This is possible by stochastic equicontinuity of {ά( , •)} and compactness of B x F. Let δι > 0 be such that limsupn_^oosup{|£l(^(i5) — Eφ(β')\ : ||/3 - β'\\ < δx} < δ0. It follows that - Eά{β',Eφ{β'))\ : \\β - β'\\
limsnpsnip{\Eά(β,Eφ(β)) n—> o o
To prove (iii), let e > 0, η > 0. Since lim P{φ(β) G F, for all β G B} = 1,
71—> OO
we can, and do, assume φ(β) G F without altering our limit argument. Let δ0 > 0 be such that limsupP{sup \ά(β, φ) - a(β', φ')\ > e} < η/2, n—>oo
supremum being over the set {(/?, φ, βf, φ1) : \\β - β'\\ < δ0, \\φ - φ'\\ < δo} Let δι > 0 be such that - φ{β')\ > δ0} < η/2,
n—>oo
supremum being over the set {(β,β') : \\β - β'\\ < ίi} min{<5o, δι}. It follows that limsupP{ sup n->oo l^ΊK
ά(β,φ(β))-ά(β',φ{β'))
<»
\\β-β'\\<S
ά(β,φ(β))-ά(β',φ(β'))
Put δ =
125
CONSISTENCY OF GEE
sup
φ(β) - φ(β')
S-β'\\<δ
< limsupP {sup ά(β, φ) - ά(β', φ') > e} + % < η, n—y<x>
where the supremum on the last line is over the set {\\β — β'\\ < δo,\\φ —
Φ'\\ < δ0}.
•
L E M M A 3. Suppose (a) for each β G B and φ G F, φ(β) Eφ(β) A 0 and ά(β,φ) - Eά(β,φ) 4 0, and (b) {φ(β),Eφ(β)} obeys Cl on B, and {ά(β,φ),Eά(β,φ)} obeys Cl on B x F. Then the sequence of random functions {ά(;φ(-))-Eά(;Eφ(-)):n
= l,2,...}
(7)
converges weakly in C(B) to a random function degenerated at constant 0; where C(B) is the class of continuous functions defined on the compact set B. P R O O F . First, we show that for each β e B, ά(β, φ(β))-Eά(β, Eφ(β)) converges in probability to 0. In other words, the finite dimensional distributions of the sequence of random functions (7) converge to those of the random function degenerated at constant 0. Let e > 0, and let δ > 0 be such that limsupPJ
sup
\ά(β,φ) - ά{β,φ')\ > e/2J - 0.
(8)
From the discussion preceding Lemma 1, assumptions (a) and (b) imply that the sequence {ά( , ) : n = l,2,...} converges weakly to constant 0 in C(B x F). It follows that limsupP {\ά{β, Eφ(β)) - Eά{β, Eφ(β))\ > e/2} n—xx)
\ά(β,φ) - Eά{β,φ)\ > e/2} = 0. (9)
βeB,φ£F
Therefore,
hmsnpP{\ά(β,φ(β)) - Eά(β,Eφ(β))\ > e} = \imsupP{\ά(β,φ(β)) - Ea(β,Eφ(β))\ > e, \φ(β) - Eφ(β)\ < δ}
126
LI
[by assumption (a)] < limsupP{\ά(β,φ(β)) - Eά(β,Eφ(β))\ > e, \φ(β) - Eφ(β)\ < δ, n—>oo
\ά(β, Eφ(β)) - Eά(β,Eφ(β))\
< e/2} [by (9)]
-ά{β,Eφ{β))\>e/2, n-ϊoo
\φ{β) - Eφ(β)\ < δ] = 0,
(10)
where the right most limit equals 0 because it is no more than the left hand side of (8). By Lemma 2 we know that the sequence (7) obeys Cl. This, together with (10), implies that the sequence (7) is tight in C(B). This proves the asserted result. • At this stage it is helpful to see through an example just what assumption Cl means, and how it can be verified for a specific moment structure. We do so by look into the moment estimate of φ suggested by Liang & Zeger (1986). The assumptions about the estimates of a suggested may be investigated following a similar procedure. E X A M P L E 1. For simplicity, we assume that πi = n for all i. Let B be a compact set in RP. Liang & Zeger (1986) suggested estimating φ by averaging over the residues based on the marginal moment assumptions, as follows
Actually, Liang & Zeger used (K — p)n as the denominator; we can ignore the finite number p without affecting the limit argument. Let β be an arbitrary but fixed parameter value, and βf a point in an open ball Oβ centered at /?, whose radius is yet to be determined. Let e and η be two positive numbers. By direct calculation,
\μu(β') +μu(β)\} * \μ*(P) ~ μ*(β)\
We now verify that {φ(β)} is stochastically equicontinuous if the following conditions are satisfied
127
CONSISTENCY OF GEE (i) For each ί, the sequences {μui') ' i — 1,2,...} and {Vit( ) : i — 1,2,...} are equicontinuous and the second sequence is bounded away from 0. (ii) There are parameter values βa and βb for which {μit{βa) '• i = 1,2,...} and {Vit(βb) : i = 1,2,...} are bounded.
By (i) and (ii), for each ί, \μit(β)\ is bounded for i = 1,2,..., and β e B. Therefore, since n is finite, \μa(β)\ < Mx. Similarly, l/Vit{β) < M2. Hence (11) implies that 1.
sup \φ(β') - φ(β)\ < - Λ Σ {\Yit\ x sup \μit(β') - μit(β)\}
β'eo0
Kn
K n
β>eO
i t
Σ SUP iMit(^) - μu(β)\ Σ SUP
Σ { l y « - ^(β)\2 x ^up |vit(/?') - vit{β)\). Assumptions (i) and (ii) also imply that the sequence
+ /4(A>)} : K = 1, 2,...} is bounded, and so both {{Kn)~ι Σiit \Yit\ : i — 1,2,...} and {(Kn)~ιΣiyt \Yit - μ%t{β)\2 i = 1,2,...} are bounded in probability. In other words, there is a positive number M3 for which or
Σ\Yit-μit{β)\2>Mz}<e-
By assumption (i) we can find δ > 0 as the radius of Oβ such that
max { sup \μit(βf) - μit(β)l
sup \Vit(β') - Vit(β)\\
for all ί = l,2,...,n;i = 1,2,.... Then, l i m s u p ^ ^ P ί l ^ ' ) - φ{β)\ > η) < e and, hence, {>(•)} is stochastic equicontinuous. Evidently assumptions (i) and (ii) also imply that {Eφ(-)} is a equicontinuous and bounded sequence. Therefore {φ(β),Eφ(β)} satisfies C l . •
128
LI
From Lemma 3 and the argument preceding Lemma 1, it is easy to see that, with probability tending to one, the random set {ά(β, φ{β)) : s β e B} is contained in a compact set in R . We write the set as A, and write the compact set B x F x A in RP+1+S as θ . It now takes only a small step further to obtain the limit behaviour of the deviance function introduced in the last section. L E M M A 4. Suppose that, in addition to the assumptions of Lemma ι ι 3, the sequence of random functions {rΓ D{θι,θ2) — n~ ED(θι,θ2) : 2 2 n = 1,2,...} obeys Cl in θ , and that for each (θuθ2) G θ , 1
n- D{θuθ2)
1
- n- ED(θuθ2)
4 0.
Then the sequences {n~1i?( , ),n~ 1 J( , •) : n = 1,2,...} obey Cl in B2, and n~1{i?( , •) — J ( , •)} converges weakly in C(B2) to a constant function 0. The proof is similar to that of lemma 3, and will be omitted. 6. CONSISTENCY OF GENERALIZED ESTIMATING EQUATIONS We are now ready to prove that any minimax point of the function R{β\ >β2) is consistent. As we shall see in the next section, under mild conditions, any minimax points are solutions to generalized estimating equation (2). In other words, we can identify the consistent solutions of (2) by verifying that they are minimax points. Two numerically more convenient criteria will be presented in the next section. The function J(β, β0) plays the same role as the Kullback-Leibler information number in the Wald's proof of consistency of maximum likelihood estimate, and R(βuβ2) plays the role of log likelihood ratio. However, in our case, neither R(βι,β2) nor J(β\,β2) has the form f(β2) — f{β\)' This is why a minimax deviance procedure must be used in place of the maximum likelihood procedure used in the Wald's proof. THEOREM 1. Suppose (a) The sequence {φ{β), Eφ(β)} obeys Cl in B, {ά(β, >), Eά(β, φ)} obeys Cl inBxF, and{n-ιD(θuθ2),n-ιED{θuθ2)} obeys Cl in θ 2 ; (b) For each fleθ, φ(β) - Eφ(β) A 0, ά(β, φ) - Eά(β, φ) 4 0, and rΓ1D(θuθ2) - rΓιED(θuθ2) 4 0;
129
CONSISTENCY OF GEE (c) For each β in B, β φ βo, the sequence of quadratic forms 1 {n~ J(β,βo)} satisfies
>0. Then any parameter value β that satisfy the relation sup R(β, β) = inf sup R(β, β')
(12)
is a consistent estimate o Notice that the estimates 0, ά need not be consistent; in fact, they need not converge in probability at all. Furthermore, the theorem asserts that all minimax points (if there are more than one) are consistent. The proof of the theorem is along the lines of Theorem 1 of Li (1996); here we only describe the idea briefly and highlight the difference. PROOF. Let Oβ0 be an arbitrary but fixed open ball centered at the true parameter value βo. Let β φ βo and Oβ be an open ball whose closure does not contain βo. By assumptions (a), (b), and Corollary 1, the sequence n~ι{R(β, βo) — J(β, βo)} converges weakly in C(B) to the random function degenerate at constant 0. This, together with assumption (c), implies that with probability tending to 1, inf^o^ n~ιR(β\ β0) > δ for some positive δ that may be dependent on β. Now the class of such open balls {Oβ : β G B \ Oβ0} form an open cover of B \ Oβ0. By compactness of B \ Oβ0 there is a finite subcover {Oi : ί = 1, ...,&}. Now on each O^ there is a ^ > 0 such that, with probability tending to one, inϊβ'zOi n~1R(βf, βo) > δι. It follows that, with probability tending to 1, mΐβt^Oβ0r^~1R(β^ βo) > δ for some positive δ. Since n-ιR(βu β2) > udβ>ίo0o n~lR(P, βo), we see that lim P { inf sup rΓιR{β, β') > δ\ = 1.
(13)
Meanwhile, by a similar argument, one can show that for every positive δ>0 lim P {inf sup rΓιR{β, β1) < δ] = 1. n->oo
^βEBβ,eB
(14)
>
However, (13) and (14) imply that, with probability tending to one, any minimax point β of R(β, β1) is in Oβo. In other words, β converges in probability to βo Π
LI
130
7. TWO PRACTICAL CRITERIA FOR CONSISTENCY Theoretically, Theorem 1 solves the consistency problem. Practically, the search for the global minimax of R(β, β') may be numerically difficult, and so we need simpler criteria for consistency. In this section we will introduce two such criteria. To achieve these, we need to assume that the function R(β, β') do not behave too irregularly along the straight line β2 = β\, specifically, that, with probability tending to one, the following condition is satisfied inf sup R(β, β') = sup inf R(β, β'). B
β£
(15)
βεB
β'eB
Roughly, the condition requires that, as β moves pass the true parameter value βo, the mode of the function R(β, •) moves continuously. For further discussion of this point, see Li (1996). In practice, such continuity is almost always satisfied; It is a challenge to find a counter example. Nevertheless, condition (15) is not implied by the antisymmetry of the function of i?, nor assumption (iii) of Theorem 1. The latter two conditions only guarantee that Pin"1] 1
inf sup R{β,β') - sup inf R(β,β')\ β€B β'€B
P€BP€B
< e) -> 1 for each e > 0, }
as can be seen from the proof of Theorem 1. Condition (15) is not crucial from a theoretical point of view because, as we have seen, the minimax β defined in (12) is consistent whether or not (15) holds. And, without requiring condition (15), we can show that β is efficient using a method similar to that used in the proof of Theorem 2 of Li (1996). However, the condition does simplify the computation and discussion, because it guarantees that the minimax point of R is necessarily a solution to the generalized estimating equation. Let B be the set of all solutions to the generalized estimating equation (2). C O R O L L A R Y 1. Suppose (a) the condition (15) is satisfied, (b) the minimax of R is in the interior of B, (c) i?( , •) is a differentiate everywhere in B2. Then, under the assumptions of Theorem 1, any solution β of equation (2) that satisfies )=0
(16)
βeB
is consistent. PROOF. Let β be a (any) minimax of R. Under the assumptions (a), (b), and (c), it is easy to show that β is in B. The argument is similar to
131
CONSISTENCY OF GEE
the proof of Theorem 3(b) of Li (1996), and the detail is omitted. Now let β be a solution to (2) that satisfies (16). Since β is the minimax, and since β € B, we have
0 < sup R(β, β) < sup R(β, β) = 0; so sup R(β, β) = 0. βB
Since βeB, the above implies R(β,β) < 0. But, since β € J3, (16) implies R(β, β) < 0. Hence, by the antisymmetry of R, R(β, β) = 0. Now let βo be the true value of the parameter /?, let e > 0 and p > 0 be arbitrary but fixed, and let Oβ0 be the open ball of radius p centered at βo. By an argument similar to that used in the proof of Theorem 1, there is a positive number η for which P { sup n~ιR(βOj β) < -η\ -> 1, as n -> oo. ι
(17)
}
βίoβ
Now by Lemma 4, the sequence {n~1i2( , •) : n = 1,2,...} is stochastically uniformly equicontinuous, therefore there is a δ > 0 for which \imsvLpP{sup\n-ιR(βuβ2)-n-ιR(β[,β2)\
> η/2} < e,
where the supremum is taken over the set {(βuβ[,β2) β[\\ <δ}. By Theorem 1,
(18)
G B3 : \\βι —
C δ) = 1.
(19)
Combining (18) and (19), it follows that ),β)-R(βo,β)\>η/2}<e.
(20)
Hence, by (17) and (20), n-too
< lim sup P{/? i Oβo, rΓι sup R(βo,β) < -η, n-*oo 1
β$Oβ0
n- suv\R(β,β)-R(β0,β)\<η/2}
+e
< lim sup P {n-ιR{βQ, β) < -η, n" 1 \R(β, β) - R(β0, β)\ < η/2} + e n—>oo
< lim sup P {nΓιR{β, β) < -η/2} + e = e. 71—>OO
132
LI
Since e is arbitrary we conclude that P(β £ Oβ0) -> 0.
•
If we do not want to find all the solutions of (2), we can use the following immediate consequence of Corollary 1. C O R O L L A R Y 2. Under the conditions of Corollary 1, any solution β of (2) that satisfies ()
(21)
=0
βeB
is consistent. Thus, in order to determine whether a solution β belongs to a consistent sequence, it suffices to check whether #(/?, β) < 0 for all β. We now apply the minimax-deviance approach to a numerical example. 8. A NUMERICAL EXAMPLE EXAMPLE 2. Let {Yit : t = 1,2; i = 1,..., 30} be thirty bivariate observations. For each z, (lα, l y follows a bivariate normal distribution with expectation located at an unknown point of a cardioid and correlation matrix completely unspecified. That is ( μ\ \
( 2cos/?-cos2/3 \
V μf J = [ 2 8m/? - sin2/3 J
(x/
v
λ
^rθ(YιUYi2)
./ 1
=φ^
a
a\ χ
J,
where θ = (/?, 0, a) are unknown and β is the parameter of interest. Thirty observations are generated with β = τr/4, φ = 0.3 and a = 0.3, and are presented in Figure 1. There are four solutions to the generalized estimating equation. The solution β = 0 will be ignored since it has nothing to do with data. The data and the solutions are presented in Figure 1. The three non-trivial solutions are βQ = 0.28τr, βι = 0.91π, and β2 = 1.92π. In this simple example, the likelihood function is available, and it takes a global maximum at /30, a local maximum at β2, and a global minimum at β\. Plotted in Figure 2 is the curve l(β) — sup^/ 6 B R(β, /?'); so the minimum point of l(β) is the minimax of R(β,βr). Thus by Theorem 1, β0 is a consistent solution. Figure 2 also indicates that lφ0) — inf/^β l(β) = 0; in other words sup^ G β R(βo> β) = 0. So Corollary 2 also tells us that βo is consistent. ACKNOWLEDGEMENT I wish to thank a referee for his (or her) very useful comments. The research is supported by the National Science Foundation Grant DMS-9306738.
CONSISTENCY OF GEE
133
Figure 1: Multiple solutions for the cardioid model. bO, bl, b2 are βo, βu $2- The three +'s mark the positions that correspond to the three solutions.
LI
134
8 -
saddle point = consistent solution
200
300
beta
Figure 2: Saddle point of the deviance function. The curve is supβ,eBR(β,β') as a function of β. The minimum point corresponds to the saddle point of R.
CONSISTENCY OF GEE
135 REFERENCES
[I]
P. (1968). Convergence of Probability Measures. John Wiley & Sons, Inc.
[2]
CONWAY, J.B. (1990). A Course in Functional Analysis. 2nd edition. New York: Springer.
[3]
DOOB,
[4]
CROWDER, M. (1986). On consistency and inconsistency of estimating equations. Econometric Theory 3, 305-30.
[5]
FIRTH, D. & HARRIS, I.R. (1991). Quasi-likelihood for multiplicative random effects. Biometrika 78, 545-555.
[6]
GODAMBE,
[7]
GODAMBE,
[8]
C., MONFORT, A., & TROGNON, A. (1984). Pseudo maximum likelihood methods: Theory. Econometrica 52, 681700.
[9]
R.G. (1984). Bounds and expansions for Fisher information when moments are known. Biometrika 74, 233-245.
BILLINGSLEY,
J.S. (1934). Probability and Statistics. Math. Soc. 36, 759-775.
Trans.
Amer.
V.P. (1960). An optimum property of regular maximum likelihood estimation. Ann. Math. Statist. 31, 1208-1211. V.P. and HEYDE, C.C. (1987) Quasi-likelihood and optimal estimation. Int. Statist. Rev. 55, 231-244. GOURIEROUX,
JARRETT,
[10] Li, B. (1993a). A deviance function for the quasi likelihood method. Biometrika 80, 741-753. [II] Li, B. (1993b). Deviance functions for generalized estimating equations, unpublished manuscript. [12] Li, B. & MCCULLAGH, P. (1994). Potential functions and conservative estimating functions. Ann. Statist. 22, 340-356. [13] Li, B. (1996). A minimax approach to consistency and efficiency for estimating equations. To appear in Ann. Statist. 24. [14]
LIANG,
K.Y. & ZEGER, S.L. (1986). Longitudinal data analysis using generalized linear models. Biometrika 73, 13-22.
[15]
MCCULLAGH,
11, 59-67.
P. (1983). Quasi-likelihood functions. Ann.
Statist.
136
LI
[16]
MCCULLAGH, P. (1990). Quasi-likelihood and estimating functions. In Statistical Theory and Modelling: in honour of Sir David Cox. Ed by D.V.Hinkley, N.Reid, and E.J.Snell. London: Chapman & Hall.
[17]
MCCULLAGH,
[18]
D.L. (1984). Estimation for aggregate models: the aggregate Markov chain. Can. J. Statist. 12, 265-282.
[19]
D. (1984). Convergence of Stochastic Processes. New York: Springer.
[20]
C.G. k MCLEISH, D.L. (1988). Generalization of ancillarity, completeness and sufficiency in an inference function space, Ann. Statist 16, 534-551.
[21]
C.G. & MCLEISH, D.L. (1989). Projection as a method for increasing sensitivity and eliminating nuisance parameters, Biometrika 76, 693-703.
[22]
SMALL,
[23]
WALD,
[24]
WEDDERBURN,
[25]
WOLFOWITZ,
P. & NELDER, J.A. (1989). Generalized Linear Models. 2nd edition. London: Chapman & Hall.
MCLEISH,
POLLARD,
SMALL,
SMALL,
C.G. & MCLEISH, D.L. (1994). Hubert'Space Methods in Probability and Statistical Inference. New York: John Wiley. A. (1949). Note on the consistency of maximum likelihood estimate, Ann. Math. Statist. 20, 595-601. R.W.M. (1974). Quasi-likelihood, generalized linear models, and the Gauss-Newton method. Biometrika 61, 439447. J. (1949) On Wald's proof of the consistency of the maximum likelihood estimate. Ann. Math. Statist. 20, 601-602.
139
Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH SERIES
EXTENDED QUASI-LIKELIHOOD AND ESTIMATING EQUATIONS J. A. Nelder Imperial College, London Y. Lee Seoul National University, Korea
1
INTRODUCTION
This paper compares the estimation of mean and dispersion parameters using (i) extended quasi-likelihood (EQL) and (ii) optimum estimating equations (OEE). 1.1
Extended quasi-likelihood (EQL)
Quasi-likelihood (QL) was introduced by Wedderburn (1974) as a way of weakening the distributional assumptions of GLMs (McCullagh and Nelder, 1989) by specifying only the form of the linear parameters β and the variance function V(μ), which expressed the variance as a function of μ. The linear score function dl/dμ = (y-μ)/V(μ) of GLMs was preserved in the wider class of models, and estimates obtained from maximizing the QL have many properties analogous to ML estimators. The importance of QLs is that they allow us to extend the ideas of GLMs where no exponential family exists to supply an error structure of the original form. EQL was introduced by Nelder and Pregibon (1987) to allow comparison of different variance functions on the same data. It is the key to methods for the joint modelling of mean and dispersion based on QL ideas. A QL model defines a deviance component d{ for observation i with mean μi given by
140
NELDER AND LEE
The deviance D = Σdi. EQL is most simply stated in terms of its extended deviance ^
)
.
(2)
Here the dispersion parameter φ is allowed to vary over the observations. For distributions of the GLM family, the EQL can be derived as a saddlepoint approximation in which all the factorials are replaced by their Stirling approximations. It is thus exact for the Normal and inverse gamma distributions, which have no factorials, and for the gamma differs only in the normalizing factor. Note that the EQL corresponds to an exact distribution of the kind Jorgensen (1996) calls a regular proper dispersion model whenever the normalizing factor is exactly one. The quantity D+ can be used as a criterion to fit models when both the mean and dispersion are assumed functions of explanatory variables.
1.2
Optimum estimating equations (OEE)
If a function g(Y θ) of a random variable Y, having a distribution depending on 0, has zero mean for all 0, then g(Y θ) is an estimating function. If θ is a scalar then the optimum estimating equation (OEE) is based on
h = g.E(dg/dθ)/E(g2) and for a sample of n independent Ys the OEE is given by
More generally if θ is a vector, V is the conditional covariance matrix of 3, given a model matrix A and E(g\A) = 0, the optimum estimating function is given by where
If U is linear (quadratic) in y we have an optimum linear (quadratic) estimating function (Godambe, 1991). It may happen that we require more than one estimating function, and this occurs if we require to estimate both mean and dispersion parameters in a model. Godambe and Thompson (1989) derive such joint estimating equations, and use components (y — μ) and (y — μ) 2 , orthogonalizing the second with respect to the first. We discuss this case further in Section 2.2 below.
141
EXTENDED QUASI-LIKELIHOOD
2 2.1
JOINT ESTIMATION OF MEAN AND DISPERSION The EQL criterion
For φi given, the estimates of β obtained from (2) are just those given by the Wedderburn QL equations
Σ
y
μjdμ dβ ΦiVi dβ
while for given μ, D+ is the same as would be obtained from a QL model with response variable d and variance function φ2, i.e. that of the gamma distribution. That this may be a good approximation even when y has a nonnormal distribution may be seen from the result of Pierce and Schafer (1986) that the deviance function is nearly an optimal normalizing transform. Figure 1 shows for the Poisson distribution the expectation and variance of d, which should be 1 and 2 respectively, and also the correlation of (y — μ) and d, which should be zero. For μ > 3 the approximations are acceptable, but, not surprisingly, break down for small μ. We are thus led to form two interlinked GLMs, one for the mean and one for the dispersion. That for the mean has response y, prior weight φ~ι, a linear predictor η = ΈβjXj, and some suitable link and variance functions. For the dispersion we have response d, a linear predictor ξ — Σjk^k with explanatory variables ui,^2,..., a, variance function φ2 and a suitable link, usually chosen as log. An obvious algorithm for fitting the joint model is a 'see-saw' one, in which the model for the mean is fitted first with current estimates of φι, then the model for the dispersion is fitted given current estimates of μι and hence of d{. Three cycles, starting with 0 = 1, are often sufficient tofitthe joint model. Standard GLM techniques may be used to check both models for internal consistency. 2.2
The OEE criterion
Godambe and Thompson (1989) derive what they call an extended quasiscore function which gives estimating equations of the form ^
0,
(3)
for the mean parameters β where, hi = {Vi - μi)2 - φV(μi) - Ίu{φV{μi))ll2{yi
- μi)
(4)
142
NELDER AND LEE
and Σhi = 0, for the dispersion parameter φ. In these equations ju and 72; are the standardized third and fourth cumulants and 1/2
2
α* = (7ii - Ίu)/{Φ V(μif (Ί2i
2
+ 2 - 7ii) },
where
is the exponential skewness. For simplicity these equations are given assuming a constant dispersion. If φ is structured they can be extended in a standard way. The form in which α; is defined shows immediately that for distributions of the GLM family aι is identically zero, because 71 = 7J, so that (3) reduces to the QL equations for β. Thus for GLM models, the optimum linear, the optimum quadratic and the quasi-likelihood estimating equations for β are all the same. In (4) the last term is identically zero for normal errors, or for gamma errors with a log link, and is usually much smaller than the first; thus, approximately, (4) is equivalent to equating the Pearson X2 to its expectation, uncorrected for d.f. lost in fitting β.
3
COMPARISON OF EQL AND OEE CRITERIA
There are two basic (but connected) differences in the equations for the joint estimation of mean and dispersion produced by the use of EQL and OEE. The first concerns the response variable for the dispersion; the use of the function (y — μ)2 in OEE equations leads to the Pearson X 2 , whereas the EQL equations use the deviance component. The second difference, which follows from the first, is that the OEE equations, in general, require knowledge of 71 and 72, or equivalently the third and fourth cumulants, whereas the EQL equations do not. For normal models 71 = 72 = 0, and the EQL and OEE equations are identical. For GLM non-normal models the equations for the mean parameters are the same, but those for the dispersion differ, while for nonGLM models both sets of equations are different. What reasons are there for preferring one method to the other?
3.1
Pseudo-likelihood v quasi-likelihood
Nelder and Lee (1992) compared estimates from the EQL having D+ = Σdi/φi
+
Σlog(2πφiV(yi)),
EXTENDED QUASI-LIKELIHOOD
143
with pseudo-likelihood (PL) estimates obtained from minimizing Dp = ΣXf/φi + Σ\og(2πφiV(μι)) for three non-GLM models. Model (1) involved the NBα distribution, a form of negative binomial distribution obtained mixing the Poisson with a gamma where the shape parameter v varies with μ, instead of the scale parameter a. The response y has υar(y) = μ(l + α), i.e. looks like an overdispersed Poisson. We did a 5-factor simulation in a 2 5 " 1 fractional factorial, the factors being aspects of the configuration of the means we thought might be important. Model (2) was a mixture of two Poisson samples with a ratio of means of 4:1; the variance function was assumed to have the form >μα, and estimates of a were of interest. Model (3) was a Poisson-Inverse Gaussian mixture with the inverse Gaussian distribution parameterized so that the variance function had the form μ+aμ2 (the IG-2 distribution). The experimental factors were sample size and value of α. For detailed results see our paper. The main conclusion was that though the bias of the maximum EQL (MEQL) estimator was usually larger than that of the maximum PL (MPL) estimator, this was more than offset in moderate sample sizes by the larger variance of the latter; the result was that in terms of MSE, the MEQL estimator was never appreciably inferior to the MPL estimator, and was often much better. An interesting result was that in finite samples the MEQL estimate was frequently better than the ML estimate. 3.2
The value of knowing 71 and 72
The OEE for mean and dispersion require knowledge of 71 and 72 for nonGLM models, and knowledge of these can improve estimates. For example, Lee and Nelder (unpublished) consider a model with log link and NBα errors, and two groups having means μ and 2μ. Table 1 shows the asymptotic variance ratios for estimates of the group difference for various μ and φ. The ML estimates are equivalent to those derived by knowing all the cumulants, the estimates based on quadratic estimating functions (QEF) to knowing the third and fourth cumulants, and the MEQL estimates to assuming an exponential-family pattern for the cumulants. For large overdispersion (φ = 5) there is considerable loss of efficiency of MEQL relative to the ML estimates, with the QEF results showing that a considerable part of this loss can be recovered if 71 and 72 are known. Similar results were found with another example using the IG-2 distribution for errors. While these results are interesting, they assume that the distribution is known at least up to the fourth cumulant. In practice this is almost never so.
144
NELDER AND LEE
Attempts to estimate 71 and 72 from moderate amounts of data may lead to estimates with considerable errors; for example, for a sample of 10 from a Poisson distribution with μ = 5, 100 simulated samples gave 33% of estimates of 71 with negative values, i.e. of the wrong sign. We need to know, therefore, what loss of information arises with various estimators when we assume values p\ and p2 for the unknown 71 and 72. We shall be mainly concerned with the optimum quadratic estimating function, the ML estimator and the MEQL estimator. For simplicity we restrict the argument to independent Y{. The QL equations assume a GLM pattern of cumulants. The optimum QEF equations improve efficiency by using information from the third and fourth cumulants and the ML equations from the all the cumulants. So the ML estimator will be most informative if the true model is known. But it is often not consistent if a model is wrongly chosen. Consider the trace or determinant of normalized asymptotic variance rΓ l coυφ) as the risk. For a given V(μ) with μ fixed, the MEQL estimator is a mini-max estimator among QEF estimators, since its risk remains constant, i.e. does not depend upon the true values of 71 and 72; it attains the minimum risk under the GLM skewness. The MEQL estimator is also the ML estimator if a GLM family exists. If so it is again a minimax estimator among ML estimators for the class of distributions with a given V(μ). Therefore, the MEQL estimator would be most conservative (in the sense that the possible maximum risk is minimal) against a possible misspecification of either the likelihood or cumulants of the model. λ 2
Godambe and Thompson's joint QEFs with p\ — ρ2 = 0 lead to the Normal ML estimator for β. For Normal heteroscedastic linear models, assuming only the first moment is correctly specified, Carroll and Ruppert (1982) showed that the MEQL estimator is robust against a small variancefunction mis-specification compared with the Normal ML estimator. Under mild regularity conditions, the consistency of the MEQL estimator depends only upon the correct specification of the regression while that of the optimum QEF estimator requires also the correct specification of V(μ)\ see Crowder (1987). The QEF equations (2) become the QL equations when a{ = 0. When V(μ) is mis-specified, the QEF estimator is no longer consistent unless aι = 0; see (2). So the MEQL estimator is most robust among QEF estimators against a mis-specification of V(μ). We illustrate the nature of the minimax property of the MQL estimator by two of the examples from Section 3.1 (Lee and Nelder, unpublished). In the first the unknown true distribution is the NBα distribution. For 1/2 1 2 this 71 = 0i/μ and 72 = 02/μ, where θλ = (1 + 2α)/(l + a) ' and 2 0 2 = (1 + 6α + 6α )/(l + a). Suppose we assume values p\ = λi/μ 1 / 2 and p2 = \2Jμ with λi and λ2 being values of Θ2 and Θ2 for which a = 0.2, 1 and
EXTENDED QUASI-LIKELIHOOD
145
5, corresponding to small, moderate and large amounts of overdispersion. Figure 2 shows the lower bound of the variance ratio for the QEF estimator with respect to the MEQL estimator over the θ\ scale, when μ = 1, an unfavourable value where the MEQL estimator is known from simulation to have low efficiency. It is clear that having the assumed value of θ\ too high can lead to much greater loss of efficiency than the corresponding gain in efficiency when the correct value is chosen. For larger μ this effect would be even more marked. In the second example the unknown true distribution is the IG-2 distri1 2 2 bution. Here 71 = 3/v / and 72 = 15/^, where var(Y) = μ + μ /v. Let assumed values p\ and P2 be 71 and 72 values at v = 0.5, 1, and 2, so that the corresponding values of pi are 4.243, 3, and 2.121 respectively. Figure 3 shows similar curves to Figure 2 over 71, the true skewness scale. Here the losses from using the MEQL estimator when p\ and p2 are nearly right are much less than the gains when they are too large.
4
CONCLUSION
We can derive the EQL equations for mean and dispersion from the OEEs by replacing the hi of equation (4) by d{ — φi, where d is the deviance component and φ the dispersion parameter, which may be structured. If we make three approximations, namely that E(d) = >, υar(d) = 2φ2 and corr(y — μ d) = 0, the resulting estimating equations are given by
and these are just those obtained from the use of EQL. Note that the three approximations become exact as μ -> 00; however, their joint effects for small μ need further investigation. The properties of the MEQL estimator give it a special place among estimators in the joint estimation of mean and dispersion.
References
CARROLL, R. J. and RUPPERT, D. (1982). A comparison between maximum likelihood and generalized least squares in a heteroscedastic linear model. J. Am Statist Assoc, 77, 878-882.
146
NELDER AND LEE
COX, D. R. (1993). Some remarks on overdispersion. Biometrika, 70, 279274. CROWDER, M. (1986). On consistency and inconsistency of estimating equations. Econometrics Theory, 3, 305-330. CROWDER, M. (1987). On linear and quadratic estimating functions. Biometrika, 74, 591-597. FIRTH, D. (1987). On the efficiency of quasi-likelihood estimation. Biometrika, 74, 2333-245. FIRTH, D. (1988). Multiplicative errors: lognormal or gamma? J. R. Statist Soc. B, 50, 266-268. GODAMBE, V. P. and THOMPSON, M. E. (1989). An extension of quasilikelihood estimation. J. Statist. Plann. Inference, 22, 137-152. GODAMBE, V. P. (Ed.) (1991). Estimating Functions. Oxford: Clarendon Press. JORGENSEN, B. (1996). Proper dispersion models. Braz. J. Probab. Statist, (to appear). McCULLAGH, P. (1983). Quasi-likelihood functions. Ann. Statist, 11, 59-67. McCULLACH, P. and NELDER, J. A. (1989). Generalized Linear Models, 2nd edn. London: Chapman and Hall. NELDER, J. A. (1989). Discussion of the paper by Godambe and Thompson. J. Statist. Plann. Inference, 22, 158-160. NELDER, J. A. and LEE, Y. (1992). Likelihood, quasi-likelihood and pseudo-likelihood: some comparisons. J. R. Statist. Soc. B, 54, 273284. NELDER, J. A. and PREGIBON, D. (1987). An extended quasi-likelihood function. Biometrika, 74, 221-231. PIERCE, D. A. and SCHAFER, D. W. (186). Residuals in generalized linear models. J. Am. Statist Assoc, 81, 977-986. WEDDERBURN, R. W. M. (1974). Quasi-likelihood functions, generalized linear models and the Gauss-Newton method. Biometrika, 61, 439-447.
EXTENDED QUASI-LIKELIHOOD
147
Table 1: Asymptotic variance ratios of the ML, MQL and optimum QEF estimators of group difference for NBa distribution
ML/MQL QEF/MQL
1 .875 .900
φ=2 5 .963 .966
10 .980 .981
50 .996 .996
1 .608 .768
φ=5 5 .805 .865
10 .885 .911
50 .974 .976
2.5-
var(d)
2.0-!
1.5-
Eld)
1.0-;
cov(y-y; d)
0.0-1 0.0
5.0
75
10.0 12.5 15.0 17.5
20.0
Figure 1: Properties of the Poisson deviance as a function of μ.
148
NELDER AND LEE
Figure 2: Asymptotic variance ratio of QEF/MEQL estimators as a function of θ example with NBα distribution. Curves with increasing λ; correspond to increasing amounts of actual overdispersion. Abscissa is assumed amount of overdispersion.
Figure 3: Asymptotic variance ratio of QEF/MEQL estimators as a function of 7i example with IG-2 distribution. Curves with increasing p - 1 correspond to increasing amounts of actual overdispersion. Abscissa is assumed amount of overdispersion.
149
Institute of Mathematical Statistics LECTURE NOTES — MONOGRAPH
SERIES
QUASI-LIKELIHOOD REGRESSION MODELS FOR MARKOV CHAINS Wolfgang Wefelmeyer University of Siegen Abstract We consider regression models in which covariates and responses jointly form a higher order Markov chain. A quasi-likelihood model specifies parametric models for the conditional means and variances of the responses given the past observations. A simple estimator for the parameter is the maximum quasi-likelihood estimator. We show that it does not use the information in the model for the conditional variances, and construct an efficient estimating function which involves estimators for the third and fourth centered conditional moments of the responses. In many applications one assumes that the innovations are not arbitrary martingale increments but independently and identically distributed. We determine how much additional information about the parameter such an assumption contains. To make the exposition more readable, we first treat the case in which only the conditional mean is specified. AMS 1991 subject classifications. Primary 62G20; secondary 62M05. Key words and Phrases. Efficient estimator, quasi-likelihood, Markov chain, weighted nonlinear least squares, autoregression.
1
Introduction
Suppose we observe covariates Xι and responses Yi which jointly form a homogeneous p-order Markov chain Z{ = (Xi,Yi). We write Q(Zi_i, . . . , Zi-p,dz) for its transition distribution, and for the conditional mean and variance of the response we write =
/ / Q(zp-ι,...,zo,dx,dy)y,
=
J J Q{zp-u
, *o,dx, dy){y - m{zp-U . . . , z0))2
150
WEFELMEYER
If we have a parametric model m = m^ for the conditional mean of the response, we can introduce a large class of martingale estimating functions ...,
U
Zi-P)),
(1.1)
i=p
with w#(zp-ι, ...,ZQ) an arbitrary weight function. The corresponding estimators for ϋ are defined as solutions of M$n — 0. We indicate in Section 3 that an efficient estimator is obtained with the choice wϋ(Zi-U
. . . , Zi-p) = ΰi-iίZi-i,..., Z i - p J - ^ ί Z i - i , . . . , Zi_p),
(1.2)
with ϋi-ι an estimator for υ based on the observations up to time i — 1. By efficiency we mean asymptotic optimality among all regular estimators in the sense of an appropriate version of Hajek's (1970) convolution theorem, not just optimality within some class of estimating functions. For the case of no covariates, a first-order chain and a one-dimensional parameter, a rigorous proof is given in Wefelmeyer (1996a). The estimating function is an adaptive version of the quasi-score function. The model is described by all transition distributions Q which fulfill m = πiϋ for some ΰ. It could be interpreted as a semiparametric model by writing Q(zp_i,..., ZQ, dx, dy) = M(zp-u
••-,*()> dx, dy - m#(zp-ι,...,
* 0 ))
with / / M ( z p - ι , . . . ,zo,dx,dy)y = 0, and considering M as nuisance parameter. In many applications one uses more specific models, Yi = mϋ(Zi-ι,...,
Zi-p) + si,
where the ε» are i.i.d. with mean zero and known or unknown distribution, rather than arbitrary martingale increments. We call such models regressionautoregression models. Again, m — πiβ. We show that the specific structure contains additional information about ΰ, except when the ει are normal. The efficient estimators that have been constructed for specific such models are, however, not based on estimating functions. If we have, in addition to m = ra^, a parametric model v = v# for the conditional variance of the response, with the same parameter ΰ, then the model is called a quasi-likelihood model. The best weight in (3.1) is l then w$ = Vfi rhu, giving the maximum quasi-likelihood estimator. It is as good as the estimator corresponding to the estimated weights (1.2). This implies that the maximum quasi-likelihood estimator does not use any of the information in the model assumption υ = υ#. We also note that if the
151
QUASI-LIKELIHOOD REGRESSION
model v = υ# is misspecified, then the weights (1.2) lead to a strictly better estimator. In a quasi-likelihood model one can introduce further martingale estimating functions 2
i - mϋ{Zi-U ..., Zi.p))
-
We show in Section 4 that an appropriate combination with (3.1), with weights involving estimators for the conditional centered third and fourth moments of the response, gives an efficient estimator. For the case of no covariates, a first-order chain and a one-dimensional parameter, a rigorous proof is given in Wefelmeyer (1996b). The estimating function is an adaptive version of the extended quasi-score function. Recent reviews of quasi-likelihood methods are McCullagh (1991) and Firth (1993). Again, in many applications one uses more specific models Yi = mt{Zi-u
, Zi-p) +
p
where the e% are i.i.d. with mean zero and variance one. We call such models heteroscedastic regression-autoregression models and show again that the specific structure contains additional information about ϋ. We do not give precise regularity conditions for our results. They can be obtained by fairly straightforward, if tedious, modifications of Wefelmeyer (1996a, 1996b).
2
Notation
We observe fc-dimensional covariates X{ and real-valued responses Yi. We suppose that Z{ — (X^, Yi) form a homogeneous and ergodic p-order Markov chain. For the p values of the chain preceding Zi we write Zi-\ = (Zi-ι,... ,Zi-p)'. The chain starts with an initial value Z p _i = (Zp-ι,... ,Zo)f. For the transition distribution of Zi given Z;_i = t we write Q(t,dz). Here and in the following, we will always write z = (x,y) for the variables corresponding to the random variables Zi = (Xi,Yi), and t = (r,s) corresponding to Z»_i = pfj_i, Y<_i). Boldface letters denote corresponding p-dimensional vectors, with components numbered backwards. The conditional distribution of the response Yi given the past depends only on the value Zi_χ = t and is given by the marginal of the transition distribution of Z^
152
WEFELMEYER
For the conditional mean and variance of the response given the past observations we write m (t) = j Qr{t,dy)y,
v(t) =
jQr(t,dy)(y-m(t))2.
Let π(dz) denote the stationary law of Z[. For the expectation of a function /(Zi) under π we write
πf = j π(dz)f(z). Similarly, for the expectation of a function /(Zj_i, Y{) under the stationary law π ® Qr we write
π®Qr/ = 11 π(Λ)QΓ(t,dy)/(t,y). 3
Modeling the conditional mean of the response
3.1. Estimating functions. Suppose we have a parametric model m = m$ for the conditional mean of the response, where ϋ is a ς-dimensional parameter. Recall that a large class of martingale estimating functions can be constructed as follows. Note that Yi-m^(Zi-i) are martingale increments with respect to the filtration generated by the Z{. Choose a g-dimensional vector w#(t) of weight functions. Then w$(Zi-i) is predictable, and the components of the vector w$(Zi-\)(Yi — m^{Zi-\)) are again martingale increments, so that the estimating functions Mϋn ^ΣwviZi^Yi-m^Zi^))
(3.1)
i=p
form a martingale. An estimator is obtained as solution ϋ = ϋn of the estimating equation M$n = 0. We do not give conditions for existence and uniqueness here. Call an estimator Tn for ϋ asymptotically linear with influence function /(t,y)if
nιl2{Tn - ϋ) = n" 1 / 2 JΓ f(Zi^Yi)
+ oP(l)
i=p
and /Qr(t^y)/(t,y) = 0 for all t. Then the components of the vector f(Zi-ι,Yi) are martingale increments. If the components of / are π ® Qrsquare integrable, a martingale central limit theorem holds, and Tn is asymptotically normal with covariance matrix π ® Qrf f- See, e.g., Billingsley (1968, p. 206).
153
QUASI-LIKELIHOOD REGRESSION
Let us recall how one shows that the solution ΰ = ϋn of the estimating equation M# n = 0 is asymptotically linear. We use a dot on top of a vector of functions to denote the matrix of partial derivatives with respect to ΰ. A Taylor expansion gives 0 = MKn
= Mΰn + Mΰn{ΰn
- 0) + •
with matrix of partial derivatives
i=p
i=p
Note that m$ is a row vector. Since the entries of the matrix — m#(Zi-ι)) are mean zero martingales, Yi — m$(Zi-ι)) is negligible if the entries of the matrix are π ® Qr-square integrable. Furthermore,
n .
The above arguments show that i9n has influence function /(t,y) = ( π ^ m τ ? ) - 1 ^ ( t ) ( y - m ? ? ( t ) )
(3.2)
and asymptotic covariance matrix 1
nvw$w# (πrh^w1^)'1.
(3.3)
For dependent observations, weak conditions for asymptotic linearity of estimators may be found in Hosoya (1989), Andrews and Pollard (1994) and Andrews (1994). Remark 1. We have restricted attention to weights w^(Zi-ι) which depend only on the p previous observations Zi_i,..., Z{-v of the p-order Markov chain. Let us show that there is no point in using weights w$ which depend on observations preceding Zi_ p . Note first that with such weights we would also get a covariance matrix of the form (3.3), with π now denoting the stationary law of more than p successive observations. Let WQ denote the weight function in which the additional arguments appearing in w$ have been integrated out. Then nw^m^ equals πm#m#. By Jensen's ineqality, φ — πvw$Wβ is positive semi-definite. Hence
is positive semi-definite.
•
154
WEFELMEYER
3.2. Known conditional variance. Suppose, for the moment, that the conditional variance v of the response is known. Then we can determine a weight function which is optimal in the sense that it minimizes the asymptotic covariance matrix (3.3). Recall (e.g., Horn and Johnson, 1985, p. 472) that if a block matrix I ι
, - I is symmetric and positive definite, so is ι
ι
1
C - B'A~ B and hence also B'~ CB~ - A' . Applying this result to the l 2 l 2 covariance matrix of {v~ / rn!d^v / w^y under π, we see that 4
ι
1
(7nu#rhtf)~ πvwφWφ { nm'^w'^)~ — (πv^m^m^)'
is positive semi-definite.
This means that the covariance matrix (3.3) is minimized for wΰ = v~ιrn!ϋ,
(3.4)
and that the minimal covariance matrix is ι
ι
(3.5)
.
This result is well known in the context of quasi-likelihood models; see Subsection 4.1. The influence function (3.2) of the estimator corresponding to the optimal weight w$ = v"ιm'^ is /(t, y) = (nv^rh'trht)-1 v{t)'ln^(tY{y
- m*(t)).
(3.6)
3.3. An adaptive estimating function. If υ is not known, we can construct an estimating function which is adaptive in the sense that for each v it is asymptotically as good as the best estimating function (3.1) for known υ, with weight w# = υ~ιrn!ϋ. It suffices to replace the conditional variance v by an estimator; compare Wefelmeyer (1996a). Specifically, let θj_i(t) be estimators for υ(t) based only on the observations Zo,..., Z%-\ up to time i — 1. For the construction of such estimators see, e.g., Collomb (1984) and Truong and Stone (1992). We obtain the adaptive estimating function
Σϋi-ΛZi-iΓ^iZi^yiYi^mviZi^)).
(3.7)
i=p
This estimating function is an adaptive version of the quasi-score function discussed in Subsection 4.1. Since the weight is predictable, the estimating function is again a martingale. If the Ό^—i are strongly consistent, a Taylor expansion as in Subsection 3.1 shows that the corresponding estimator is asymptotically normal, its asymptotic covariance matrix is again (3.5), and its influence function is again (3.6). For the case of a one-dimensional
155
QUASI-LIKELIHOOD REGRESSION
parameter, and when there are no covariates, a rigorous proof is given in Wefelmeyer (1996a). 3.4. Efficiency of the adaptive estimating function. Does the adaptive estimating function (3.7) lead to an efficient estimator? In other words, is this estimator optimal not only among estimators based on estimating functions of the form (3.1), but also in the larger class of regular estimators? To answer this question, we must indicate that the model given by m = m# is locally asymptotically normal in an appropriate sense, and determine a bound for the asymptotic covariance matrices of regular estimators of ϋ in the sense of a convolution theorem. The basic reference for this theory in the i.i.d. case is Bickel et al. (1993). For the case of a one-dimensional parameter, and when there are no covariates, a rigorous proof of the efficiency of the adaptive estimating function is in Wefelmeyer (1996a). To accomodate covariates, we recall that by Cox (1972) the likelihood factors into two terms. The first is the partial likelihood and depends only on the conditional law Qr{t,dy) of the responses. The second depends only on the conditional law of the covariates given the past observations and the present responses. Our model m = ra$ is a condition on Qr only. Hence the second factor of the likelihood varies independently of ϋ. This means that the bound for the asymptotic covariance matrices can be determined from the partial likelihood. Fix Qr{t,dy). The model is described by a parametric family of side conditions m = m$ . To introduce a local model, we perturb Qr(t,cfy) such that the perturbed transition distribution is still in the model. This means that a perturbed condition m = ra#, with ϋ replaced by ϋ + n~ιl2u, say, holds. Su