Applications of differential geometry to econometrics

Applications of Differential Geometry to Econometrics

Although geometry has always aided intuition in econometrics, it is only recently that differential geometry has become a standard tool in the analysis of statistical models, offering a deeper appreciation of existing methodologies and highlighting the essential issues which can be hidden in an algebraic development of a problem. This volume is the ®rst collection of papers applying these techniques to econometrics. An introductory chapter provides a brief tutorial for those unfamiliar with the tools of Differential Geometry. The topics covered in the following chapters demonstrate the power of the geometric method to provide practical solutions and insight into problems of econometric inference. p a u l m a r r i o t t is an Associate Professor in the Department of Statistics and Applied Probability at the National University of Singapore. He was previously based at the University of Surrey. m a r k s a l m o n is Deutsche Morgan Grenfell Professor of Financial Markets at City University Business School in London. He was previously Professor of Economics at the European University Institute in Florence.

Applications of differential geometry to econometrics

Edited by

PAUL MARRIOTT AND MARK SALMON

PUBLISHED BY THE PRESS SYNDICATE OF THE UNIVERSITY OF CAMBRIDGE

The Pitt Building, Trumpington Street, Cambridge, United Kingdom CAMBRIDGE UNIVERSITY PRESS

The Edinburgh Building, Cambridge, CB2 2RU, UK www.cup.cam.ac.uk 40 West 20th Street, New York, NY 10011-4211, USA www.cup.org 10 Stamford Road, Oakleigh, Melbourne 3166, Australia Ruiz de AlarcoÂn 13, 28014 Madrid, Spain # Cambridge University Press 2000 This book is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2000 Printed in the United Kingdom at the University Press, Cambridge Typeface 10/12pt Times

System Advent 3B2

[KW]

A catalogue record for this book is available from the British Library

ISBN 0 521 65116 6 hardback

Contents

List of ®gures and tables List of contributors

page vii ix

Editors' introduction

1

1

An introduction to differential geometry in econometrics p a u l m a r r i o t t and m a r k s a l m o n

7

2

Nested models, orthogonal projection and encompassing m a o z u l u and g r a y h a m e. m i z o n

64

3

Exact properties of the maximum likelihood estimator in exponential regression models: a differential geometric approach g r a n t h i l l i e r and r a y o ' b r i e n

85

4

Empirical likelihood estimation and inference richard j. smith

119

5

Ef®ciency and robustness in a geometrical perspective russell davidson

151

6

Measuring earnings differentials with frontier functions and Rao distances uwe jensen

7

First-order optimal predictive densities j . m . c o r c u e r a and f . g i u m m o l e©

8

An alternative comparison of classical tests: assessing the effects of curvature kees jan van garderen

184 214

230 v

vi

9 10

Contents

Testing for unit roots in AR and MA models thomas j. rothenberg

281

An elementary account of Amari's expected geometry 294 f r a n k c r i t c h l e y , p a u l m a r r i o t t and m a r k s a l m o n Index

316

Figures and tables

Figures page 72 2.1 …0 † 6ˆ …ÿ…0 †† 3.1 Densities for n ˆ 2, 4, 8 and 16 equispaced points z 0z ˆ 1 112 3.2 Marginal densities for t ÿ , nˆ3 114 3.3 Joint density for t ÿ , a ÿ , nˆ3 115 3.4 Marginal densities for a ÿ , nˆ3 115 5.1 Ef®ciency gain from added information 170 6.1 Isocircles, normal distribution, d ˆ 0:25, 0.52, 1 193 6.2 Isocircle, normal distribution, d ˆ 3:862 193 6.3 Estimated normal distributions of log income 207 6.4 Estimated inverse Gaussian distributions of income 208 6.5 Estimated truncated normal distributions of conditional inef®ciency 210 8.1 Best critical regions 238 8.2 Point optimal tests based on 240 the points  and  8.3 The one-sided LM, LMP and LB test 244

8.4 Critical region of the one-sided Wald test 249 8.5 Disconnected critical region (CRI [ CRII ) for non-monotonic Wald statistic 250 8.6 The geodesic test 252 8.7 The intersection of constant MLE and constant LR lines 256 8.8 The boundary of the LR critical region 257 8.9 The effect of decreasing the signi®cance level from 1 to 258 2 8.10 The boundary of the LR test for two different signi®cance levels 1 > 2 259 8.11 The distance of  ˆ  … † from 0 to the constant LR hyperplane 260 8.12 The angle between the gradient and the difference vector 261 8.13 Canonical and expectation manifolds for the AR(1) model 267 8.14 Testing for serial correlation 270 8.15 Testing for a unit root 271 vii

viii

List of figures and tables

8.16 Testing for a unit root against explosive alternatives 272 8.17 The Wald test for three values of b (2.5, 5 and 7). The test with b ˆ 5 and critical value 1.01 rejects if ^ ˆ 0:79 275 8.18 Disconnected critical region of the Wald test for a unit root based on equation (53) with b ˆ 5. ˆ1 is the expectation of (s; d) under H0 . T ˆ 25, y0 ˆ 0 275 9.1 Asymptotic power functions for unit AR root (5% level; normal quantile scale) 286 9.2 Asymptotic power functions for unit MA root (stationary model; 5% level; normal quantile scale) 289 9.3 Asymptotic power functions for unit MA root (nonstationary model; 5% level; normal quantile scale) 291 10.1 One-dimensional Poisson example 302 10.2 One-dimensional normal example 302 10.3 One-dimensional exponential example 303 10.4 One-dimensional Bernoulli example 303

10.5 The distributions of the natural and expected parameter estimates 310 10.6 The distributions of the 13 af®ne parameter estimates: the exponential case 312 10.7 The effect of sample size on the relationship between the score vector and the MLE: the exponential case 313

Tables 3.1 Means, variances and skewness for ®gure 3.1 112 3.2 Properties of the marginal densities: n ˆ 3 114 3.3 Moments of the marginal density of T 116 3.4 Marginal density of T. Tail areas: n ˆ 5 and n ˆ 10 116 3.5 Marginal density of T. Tail area (uniformly distributed z's); combined tails 117 6.1 Estimation results 202 10.1 One-dimensional examples: Poisson, normal, exponential and Bernoulli 301

Contributors

J. M. Corcuera Universitat de Barcelona Frank Critchley Open University Russell Davidson GREQEM, Marseille, and Queens University Kees Jan van Garderen CORE, Louvain-la-Neuve F. Giummole` Universita degli Studi di Padova Grant Hillier University of Southampton Uwe Jensen Institu¨t fu¨r Statistik und O¨konometrie, Kiel Maozu Lu University of Southampton Paul Marriott National University of Singapore Grayham Mizon European University Institute, Florence Ray O'Brien University of Southampton Thomas J. Rothenberg Berkeley University Mark Salmon City University Business School, London Richard J. Smith University of Bristol

Editors' introduction

Geometry has always aided intuition in econometrics but has perhaps only very occasionally aided analytic development. However, advances in statistical theory have recently been greatly enhanced through a deeper understanding of differential geometry and the geometrical basis of statistical inference; see, for instance, Kass and Vos (1997), BarndorffNielsen and Cox (1994) and McCullagh (1987), as well as a rapidly expanding array of journal articles. What value do these advances hold for econometrics? How can practical econometric methods be improved by a deeper understanding of the geometrical basis of the problems faced by econometricians? This volume attempts to stimulate the appetite of econometricians to delve further into the sometimes complex but always enlightening world of differential geometry and its application to econometrics. The reward we have found is not only a deeper appreciation of existing methodologies but the recognition that further advance in a host of different directions can readily be made. It is this potential gained through studying what may be standard problems from an alternative perspective that makes the application of differential geometry to econometrics so exciting. Geometry is mathematically fundamental, and frequently a geometric understanding of a problem highlights the essential issues which can be hidden in an algebraic development of the same issue. Take for instance the basic notion of orthogonal projection that underlies much of econometric and statistical method and its extension to curved statistical spaces and the question of invariance to different parameterisations, the interpretation of higher-order asymptotic expansions and statistical curvature and the role of ancillarity and exogeneity. How can we formalise and hence understand more fundamentally the notions of encompassing in inference and the relationship between likelihood and method of moment based inference? These are some of the questions raised in the papers collected in this volume, which is in part based on the papers 1

2

Applications of differential geometry to econometrics

presented at a conference held at the European University Institute on the Applications of Differential Geometry to Econometrics supported by a Human Capital and Mobility Grant (ERB CHRX-CT940514) from the European Union. We hope that you enjoy the experience of reading the papers gathered here and perhaps at the end appreciate the power of the geometric method. We would like to thank the authors for their patience as this volume was put together ± some more than others (!) and ®nally Tom and Tara for undertaking the burden of proofreading the ®rst draft. The ®rst chapter in this volume, by Paul Marriott and Mark Salmon, surveys the mathematics needed to understand the methods of differential geometry and their application to econometric inference. A complete development of the tools required fully to understand differential geometry is clearly beyond the scope of this chapter, but we hope that this introduction provides the basis for understanding the issues and facilitates the reading of the later chapters in the volume. This is not to say that the papers collected here cannot be read without a deep understanding of differential geometry; indeed it is clearly the case that the intuition provided by the geometric framework can be grasped quite easily without the need for a full mastery of the techniques involved. Maozu Lu and Grayham Mizon, in the second chapter, consider several aspects of the important issues of encompassing and nesting in model selection. Encompassing has a natural geometric intuition and feel but it is not necessarily straightforward to develop the associated geometry mathematically, partly because the concept of encompassing needs to be carefully de®ned. They investigate and develop the potentially surprising observation made by GourieÂroux and Monfort (1995) that it is not always the case that a more general model encompasses models that are simpler than itself. This has important implications for the inferential process of reduction from general-to-simple since in this case the sequence of parsimoniously encompassing models will not form a partial order and hence critical information might be lost if the intermediate models were discarded in the process. Lu and Mizon explore the conditions under which a general model encompasses simpler models independently of its relationship to the data-generation process (DGP). They explore the notion of nesting and demonstrate that nesting implies more than one model being an algebraic simpli®cation of another. The importance of dimension-reducing nesting constraints is explained by showing that the nesting model will encompass models nested within it independently of the nature of the data-generation process when there is an orthogonal parameterisation. They de®ne this property as automatic encompassing and show that it is not possible amongst non-nested models. They also

Editors' introduction

3

discuss how the concept of parsimonious encompassing can be used to establish encompassing relationships between non-nested models given a competing model that nests each of the non-nested models. Encompassing hypotheses are considered as mappings in Hilbert space and it is shown that a suf®cient condition for a model to encompass models nested within it is that these mappings are orthogonal projections in the relevant space. Grant Hillier and Ray O'Brien apply a recently developed and geometrically motivated technique (Hillier and Armstrong (1996) for obtaining exact distribution results for implicitly de®ned estimators to study exact properties of the maximum likelihood estimator (MLE) in the exponential regression model. The method provides an integral formula for the exact density of the MLE, expressed as a surface integral over the manifold in the sample space upon which the MLE is de®ned. The method does not require that the MLE be a known function of data; all that is required is that the manifold or the level set of the MLE be known. This enables the formula to be used to get the exact density of the MLE even though the estimator may be only implicitly de®ned in terms of the data. Hillier and O'Brien provide a detailed discussion of the application of the approach to the exponential regression model. They show that the surface integral formula is tractable in general and that from particular numerical results for the exponential regression case the exact densities are well behaved and well approximated by the asymptotic density. The generalised method of moments (GMM) approach to estimation and inference relaxes the need to have full knowledge of the likelihood function and is essentially a semi-parametric method. In consequence there is generally a trade-off between robustness and ef®ciency. Richard Smith considers instead empirical likelihood methods of inference, an approach that embeds the moment conditions used in GMM estimation within a non-parametric likelihood function through the use of additional parameters associated with these moment conditions. This enables equivalents of the test statistics based on classical likelihood to be developed and, in particular, tests of misspeci®cation on the over-identifying moment conditions. He also considers the possibility of applying the empirical likelihood to testing non-nested hypotheses. Russell Davidson develops a general geometric basis for the analysis of ef®ciency and robustness for estimators de®ned by estimating equations. The general class of such estimators was considered originally by Godambe (1960) and is essentially equivalent to the generalised method of moments. Davidson shows that, when a parameterised model is considered as a Hilbert manifold in an underlying space of DGPs, the tangent space at any particular DGP can be expressed as the direct sum of

4

Applications of differential geometry to econometrics

three mutually orthogonal subspaces. Robustness of an estimator is de®ned as its root-n consistency at all points in the model and ef®ciency by the asymptotic variance which is natural in the Hilbert spaces context. Consistent estimators of the model's parameters all have the same component in one of these subspaces which contains the asymptotically ef®cient estimator. Inef®cient estimators have a non-vanishing component in the orthogonal complement of the tangent space to the model and they can then be seen to lose ef®ciency by including random variation in directions excluded by the model's speci®cation. Information about parameters is represented geometrically by tangents that lie in the tangent space of the model. Ef®cient estimating equations for model parameters can be obtained by orthogonally projecting arbitrary root-n consistent estimators onto the ef®cient subspace. This projection can often be implemented by arti®cial regression. Uwe Jensen provides a discussion of Rao distances and associated tests in certain two-parameter models. These tests are sometimes also called geodesic tests (see Critchley, Marriott and Salmon (1996)) and potentially offer an alternative approach to inference beyond the standard likelihood based trinity of the Likelihood Ratio, Lagrange multiplier and Wald tests. Jensen then develops an interesting earnings frontier model for German data and applies the Rao distance approach to examine various hypotheses on earnings differentials. Jose Manuel Corcuera and Federica GiummoleÁ consider the highly relevant issue of determining an optimal predictive density. They show how this predictive density may lie outside the class of the original parametric model describing the variable of interest and how geometrically a global solution can be constructed by considering formally all possible directions in which the original model may be extended. Their theory is based on minimising the leading term in the asymptotic expansion of risk based on general loss functions that re¯ect the discrepancy between distributions. Speci®cally they employ  divergences as loss functions to prove the existence of an optimal solution and to get explicit solutions for a number of examples in relevant econometric models, including the standard regression model and the AR(1) model. Kees Jan van Garderen provides a detailed and powerful analysis of the effects of statistical curvature on the classical tests used in econometrics: the LR, LM, Wald and point optimal tests, as well as the geodesic test put forward by Critchley, Marriott and Salmon (1996). Most practical econometric models are statistically curved and yet the effects of statistical curvature are largely unrecognised. When a model is statistically curved, no uniformly powerful test will exist and any test will necessarily compromise power against different alternatives. Van

Editors' introduction

5

Garderen employs a geometric approach avoiding a complex mathematical development to graphically analyse the partitioning of the sample space into the critical and acceptance regions for the particular tests. The analysis is valid for all sample sizes and is global in that it considers the whole of the parameter and sample spaces. The standard asymptotic equivalence result often seems to suggest that the choice of which test to employ is simply a matter of computational convenience, but the approximate nature of this ®rst-order asymptotic result can be highly misleading if the model is seriously curved, as van Garderen shows since the different tests employ very different critical regions. The point optimal and Lagrange multiplier tests give a straight boundary to the critical region and cannot guarantee global consistency. The Wald test gives a curved boundary but the numerical value of the statistic depends on the algebraic formulation of the restrictions, and the geodesic test begs the fundamental question of which geodesic to calculate. The likelihood ratio test uses the best critical region in the maximum likelihood direction and, since it implicitly and automatically takes into account the curvature of the model, it should be favoured over the other tests when the model is signi®cantly curved. Van Garderen provides a detailed analysis of testing for serial correlation and unit roots. Tom Rothenberg considers the question of testing for unit roots in autoregressive and moving-average models. These models are statistically curved and inference will often be critically affected. Moreover the effects of curvature get substantially larger the closer we get to the unit root (see van Garderen (1997)), indicating one aspect of the non-standard nature of inference in the unit root case. Although standard asymptotic theory does not apply, Rothenberg ®nds somewhat surprisingly that the large sample power curves and envelopes mimic those predicted by standard second-order asymptotic theory. This result requires further explanation. Finally, Critchley, Marriott and Salmon consider the very general issue of the geometry of transformations and the role of parameterisation in econometric models. In particular, they provide an elementary and visual account of Amari's (1990) expected geometry, focusing on the full exponential family case. Formal de®nitions of af®ne connections are not exploited; instead they exploit the ®rst two moments of the score function under the true distribution. Amari's fundamental non-metric af®ne connection appears as the natural measure of the non- constancy of the true covariance of the score across parameter space. This covariance is constant in the natural parameters. Non-linearity of the graph of the mean score in the natural parameter is then seen to re¯ect a curvature present in nearly all parametric families. They introduce the notion of ÿ-duality, which is a natural duality between the score function in one parameter-

6

Applications of differential geometry to econometrics

isation and the maximum likelihood estimator in another. It is seen to correspond to, and therefore provide a statistical interpretation of, the notion of duality in Amari's expected geometry. A number of generalised linear model examples are used throughout. References Amari, S. (1990), Differential-Geometrical Methods in Statistics, 2nd edn, Lecture Notes in Statistics No. 28, Berlin: Springer Verlag. Barndorff-Nielsen, O.E. and D.R. Cox (1994), Inference and Asymptotics, Monographs on Statistics and Applied Probability No. 52, London: Chapman & Hall. Critchley, F., P. Marriott and M. Salmon (1996), `On the Differential Geometry of the Wald Test with Nonlinear Restrictions', Econometrica, 64: 1213±1222. Garderen, K.J. van (1997), `Exact Geometry of Explosive Autoregressive Models', University of Bristol Working Paper. Godambe, V. P. (1960), `An Optimum Property of Regular Maximum Likelihood Estimation', Annals of Mathematical Statistics, 31: 1208±1212. GourieÂroux, C. and A. Monfort (1995), `Testing, Encompassing and Simulating Dynamic Econometric Models', Econometric Theory, 11: 195±228. Hillier, G.H. and M. Armstrong (1996), `On the Density of the Maximum Likelihood Estimator', University of Southampton Discussion Papers in Economics and Econometrics No. 9645; also in Econometrica, 67(6),1999: 1459±1470. Kass, R.E. and P.W. Vos (1997), Geometrical Foundations of Statistical Inference, Chichester: John Wiley. McCullagh, P. (1987), Tensor Methods in Statistics, London: Chapman & Hall.

1

An introduction to differential geometry in econometrics Paul Marriott and Mark Salmon

1

Introduction

In this introductory chapter we seek to cover suf®cient differential geometry in order to understand its application to econometrics. It is not intended to be a comprehensive review either of differential geometric theory, or of all the applications that geometry has found in statistics. Rather it is aimed as a rapid tutorial covering the material needed in the rest of this volume and the general literature. The full abstract power of a modern geometric treatment is not always necessary and such a development can often hide in its abstract constructions as much as it illuminates. In section 2 we show how econometric models can take the form of geometrical objects known as manifolds, in particular concentrating on classes of models that are full or curved exponential families. This development of the underlying mathematical structure leads into section 3, where the tangent space is introduced. It is very helpful to be able to view the tangent space in a number of different but mathematically equivalent ways, and we exploit this throughout the chapter. Section 4 introduces the idea of a metric and more general tensors illustrated with statistically based examples. Section 5 considers the most important tool that a differential geometric approach offers: the af®ne connection. We look at applications of this idea to asymptotic analysis, the relationship between geometry and information theory and the problem of the choice of parameterisation. Section 6 introduces key mathematical theorems involving statistical manifolds, duality, projection and ®nally the statistical application of the classic geometric theorem of Pythagoras. The last two sections look at direct applications of this geometric framework, in particular at the problem of inference in curved families and at the issue of information loss and recovery. Note that, although this chapter aims to give a reasonably precise mathematical development of the required theory, an alternative and 7

8

Paul Marriott and Mark Salmon

perhaps more intuitive approach can be found in the chapter by Critchley, Marriott and Salmon in this volume. For a more exhaustive and detailed review of current geometrical statistical theory see Kass and Vos (1997) or, from a more purely mathematical background, see Murray and Rice (1993).

2

Parametric families and geometry

In this section we look at the most basic relationship between parametric families of distribution functions and geometry. We begin by ®rst introducing the statistical examples to which the geometric theory most naturally applies: the class of full and curved exponential families. Examples are given to show how these families include a broad range of econometric models. Families outside this class are considered in section 2.3. Section 2.4 then provides the necessary geometrical theory that de®nes a manifold and shows how one manifold can be de®ned as a curved subfamily of another. It is shown how this construction gives a very natural framework in which we can describe clearly the geometrical relationship between full and curved exponential families. It further gives the foundations on which a fully geometrical theory of statistical inference can be built. It is important at the outset to make clear one notational issue: we shall follow throughout the standard geometric practice of denoting components of a set of parameters by an upper index in contrast to standard econometric notation. In other words, if  2 Rr is an r-dimensional parameter vector, then we write it in component terms as ÿ
0  ˆ 1 ; 2 ; . . . ; r : This allows us to use the Einstein summation convention where a repeated index in both superscript and subscript is implicitly summed over. For example if x ˆ …x1 ; . . . ; xr † 0 then the convention states that  i xi ˆ

2.1

r X iˆ1

 i xi :

Exponential families

We start with the formal de®nition. Let  2   Rr be a parameter vector, X a random variable, continuous or discrete, and s…X† ˆ …s1 …X†; . . . ; sr …X†† 0 an r-dimensional statistic. Consider a family of

Introduction to differential geometry

9

continuous or discrete probability densities, for this random variable, of the form p…xj† ˆ exp fi si ÿ ý…†gm…x†:

…1†

Remember we are using the Einstein summation convention in this de®nition. The densities are de®ned with respect to some ®xed dominating measure, . The function m…x† is non-negative and independent of the parameter vector . We shall further assume that the components of s are not linearly dependent. We call  the natural parameter space and we shall assume it contains all  such that … exp fi si gm…x† d < 1: A parametric set of densities of this form is called a full exponential family. If  is open in Rr then the family is said to be regular, and the statistics …s1 ; . . . ; sr † 0 are called the canonical statistics. The function ý…† will play an important role in the development of the theory below. It is de®ned by the property that the integral of the density is one, hence …  i ý…† ˆ log exp f si gm…x†d : It can also be interpreted in terms of the moment generating function of the canonical statistic S. This is given by M…S; t; † where M…S; t; † ˆ exp fý… ‡ t† ÿ ý…†g;

…2†

see for example Barndorff-Nielsen and Cox (1994, p. 4). The geometric properties of full exponential families will be explored later. However, it may be helpful to remark that in section 5 it is shown that they have a natural geometrical characterisation as the af®ne subspaces in the space of all density functions. They therefore play the role that lines and planes do in three-dimensional Euclidean geometry. 2.1.1

Examples Consider what are perhaps the simplest examples of full exponential families in econometrics: the standard regression model and the linear simultaneous equation model. Most of the standard building blocks of univariate statistical theory are in fact full exponential families including the Poisson, normal, exponential, gamma, Bernoulli, binomial and multinomial families. These are studied in more detail in Critchley et al. in chapter 10 in this volume.

10

Paul Marriott and Mark Salmon

Example 1. The standard linear model of the form

Consider a linear model

Y ˆ Xb ‡ r; where Y is an n  1 vector of the single endogenous variable, X is an n  …k ‡ 1† matrix of the k weakly exogenous variables and the intercept term and r is the n  1 matrix of disturbance terms which we assume satis®es the Gauss±Markov conditions. In particular, for all i in 1; . . . ; n i  N…0;  2 †: The density function of Y conditionally on the values of the exogenous variables can then be written as     0 ÿ 0
þ ÿ 0
1 exp YY XY ‡ 2 ÿ2 2  0 0  þ X Xþ 2 ‡ …n=2† log …2 † : ÿ 2 2 This is in precisely the form for a full exponential family with the parameter vector  0  þ 1 0  ˆ ; 2 ÿ2 2 and canonical statistics ÿ …s…Y†† 0 ˆ Y 0 X


Y 0Y :

Example 2. The simultaneous equation model simultaneous linear equations

Consider the set of

BYt ‡ !Xt ˆ Ut ; where Y are endogenous variables, X weakly exogenous, U the random component and t indexes the observations. Moving to the reduced form, we have Yt ˆ ÿBÿ1 !Xt ‡ Bÿ1 Ut ; which gives a full exponential family in a similar way to Example 1. However, an important point to notice is that the natural parameters  in the standard full exponential form are now highly non-linear functions of the parameters in the structural equations. We shall see how the geometric analysis allows us to understand the effect of such non-linear reparameterisations below.

Introduction to differential geometry

11

Example 3. Poisson regression Moving away from linear models, consider the following Poisson regression model. Let i denote the expected value for independent Poisson variables Yi , i ˆ 1; . . . ; n. We shall initially assume that the i parameters are unrestricted. The density for …y1 ; . . . ; yn † can be written as, ( ) n n n X Y X 1 : yi log …i † ÿ i exp y! iˆ1 iˆ1 iˆ1 i Again this is in full exponential family form, with the natural parameters and canonical statistics being i ˆ log …i †; si …y1 ; . . . ; yn † ˆ yi ; respectively. For a true Poisson regression model, the i parameters will be predicted using covariates. This imposes a set of restrictions on the full exponential family which we consider in section 2.2. 2.1.2

Parameterisations There is a very strong relationship between geometry and parameterisation. In particular, it is important in a geometrically based theory to distinguish between those properties of the model that are dependent on a particular choice of parameterisation and those that are independent of this choice. Indeed, one can de®ne the geometry of a space to be those properties that are invariant to changes in parameterisation (see Dodson and Poston (1991)). In Example 2 we noted that the parameters in the structural equations need not be simply related to the natural parameters, . Structural parameters will often have a direct econometric interpretation, which will be context dependent. However, there are also sets of parameters for full exponential families which always play an important role. The natural parameters, , are one such set. A second form are the expected parameters . These are de®ned by i …† ˆ Ep…x;† …si …x††: From equation (2) it follows that these parameters can be expressed as i …† ˆ

@ý …†: @i

…3†

In a regular full exponential family the change of parameters from  to  is a diffeomorphism. This follows since the Jacobian of this transformation is given from equation (3) as

12

Paul Marriott and Mark Salmon

@i @2 ý …†: j ˆ @ @i @j This will be non-zero since for a regular family ý is a strictly convex function (see Kass and Vos (1997), p. 16, Theorem 2.2.1). 2.1.3

Repeated sampling and suf®cient statistics One important aspect of full exponential families concerns the properties of their suf®cient statistics. Let us assume that we have a random sample …x1 ; . . . ; xn † where each observation is drawn from a density p…x; j † ˆ exp fi si …x† ÿ ý…†gm…x†: The log-likelihood function for the full sample will be `…; …x1 ; . . . ; xn †† ˆ i

n X jˆ1

si …xj † ÿ ný…†:

Thus if the parameter space is r-dimensional then there is always an rdimensional suf®cient statistic, namely ý ! n n X X s1 …xj †; . . . ; sr …xj † : jˆ1

jˆ1

Note that the dimension of this suf®cient statistic will be independent of the sample size n. This is an important property which we shall see in section 2.3 has important implications for the geometric theory. 2.2

Curved exponential families

In the previous section we mentioned that full exponential families will be shown to play the role of af®ne subspaces in the space of all density functions. Intuitively they can be thought of as lines, planes and higher-dimensional Euclidean spaces. We can then ask what would be the properties of curved subfamilies of full exponential families? In general there are two distinct ways in which subfamilies can be de®ned: ®rstly by imposing restrictions on the parameters of the full family, and secondly as parametric families in their own right. We use this second approach as the basis of a formal de®nition. Let  be the r-dimensional natural parameter space for the full exponential family given by p…x j † ˆ exp f i si ÿ ý…†gm…x†:

Introduction to differential geometry

13

Assume that there is a mapping from , an open subset of Rp to , A:!  } …†; which obeys the following conditions: 1. the dimension of  is less than that of , 2. the mapping is one-to-one and smooth and its derivative has full rank everywhere, 3. if the sequence of points fi ; i ˆ 1; . . . ; rg  A…† converges to 0 2 A…†, then Aÿ1 …i † converges to Aÿ1 …0 † in . Under these conditions the parametric family de®ned by p…x j † ˆ exp fi …†si ÿ ý……††gm…x† is called a curved exponential family. In particular noting the dimensions of the relevant spaces, it is an …r; p†-curved exponential family. 2.2.1

Examples We now look at a set of examples to see how this class of curved exponential families is relevant to econometrics. For further examples see Kass and Vos (1997) or Barndorff-Nielsen and Cox (1994), where many forms of generalised linear models, including logistic, binomial and exponential regressions, non-linear regression models, time-series models and stochastic processes, are treated. Another important source of curved exponential families is the imposition and testing of parametric restrictions (see Example 5). Finally we mention some general approximation results which state that any parametric family can be approximated using a curved exponential family (see, for example, Barndorff-Nielsen and Jupp (1989)). Example 3. Poisson regression (continued) Let us now assume that the parameters in the Poisson regression model treated above are assumed to be determined by a set of covariates. As a simple example we could assume the means follow the equation log …i † ˆ  ‡ þXi ; where X is an exogenous variable. Hence, in terms of the natural parameters we have i ˆ  ‡ þXi : Thus the map de®ning the curved exponential family is

14

Paul Marriott and Mark Salmon

ÿ
…; þ† ! 1 …; þ†; . . . ; n …; þ† ; and we have a …n; 2†-curved exponential family. Example 4. AR…1†-model

Consider the simple AR…1† model

xt ˆ xtÿ1 ‡ t ; where the disturbance terms are independent N…0;  2 † variables, and we assume x0 ˆ 0. The density function will then be of the form (  ý !  X n n n ÿ1 X  ÿ2 X 2 exp xi ‡ 2 xt xtÿ1 ‡ x2tÿ1 2 2 2 iˆ1  iˆ1 2 iˆ1 ) n 2 ÿ log …2 † : 2 This is a curved exponential family since the parameters can be written in the form 1 …; † ˆ

ÿ1 ; 2 2

2 …; † ˆ

 ; 2

3 …; † ˆ

ÿ2 : 2 2

The geometry of this and more general ARMA-families has been studied in Ravishanker (1994). Example 5. COMFAC model Curved exponential families can also be de®ned by imposing restrictions on the parameters of a larger, full or curved, exponential family. As we will see, if these restrictions are nonlinear in the natural parameters the restricted model will, in general, be a curved exponential family. As an example consider the COMFAC model, yt ˆ ÿxt ‡ ut ; where x is weakly exogenous and the disturbance terms follow a normal AR…1† process ut ˆ utÿ1 ‡ t : Combining these gives a model yt ˆ ytÿ1 ‡ ÿxt ÿ ÿxtÿ1 ‡ t which we can think of as a restricted model in an unrestricted autoregressive model yt ˆ 0 ytÿ1 ‡ 1 xt ‡ 2 xtÿ1 ‡ !t :

Introduction to differential geometry

15

We have already seen that the autoregressive model gives rise to a curved exponential structure. The COMFAC restriction in this simple case is given by a polynomial in the parameters 2 ‡ 0 1 ˆ 0: The family de®ned by this non-linear restriction will also be a curved exponential family. Its curvature is de®ned by a non-linear restriction in a family that is itself curved. Thus the COMFAC model is curved exponential and testing the validity of the model is equivalent to testing the validity of one curved exponential family in another. We shall see later how the geometry of the embedding of a curved exponential family affects the properties of such tests, as discussed by van Garderen in this volume and by Critchley, Marriott and Salmon (1996), among many others. 2.3

Non-exponential families

Of course not all parametric families are full or curved exponential and we therefore need to consider families that lie outside this class and how this affects the geometric theory. We have space only to highlight the issues here but it is clear that families that have been excluded include the Weibull, generalised extreme value and Pareto distributions, and these are of practical relevance in a number of areas of econometric application. An important feature of these families is that the dimension of their suf®cient statistics grows with the sample size. Although this does not make an exact geometrical theory impossible, it does considerably complicate matters. Another property that the non-exponential families can exhibit is that the support of the densities can be parameter dependent. Thus members of the same family need not be mutually absolutely continuous. Again, although this need not exclude a geometrical theory, it does make the development more detailed and we will not consider this case. In general the development below covers families that satisfy standard regularity conditions found, for instance, in Amari (1990, p. 16). In detail these conditions for a parametric family p…x j † are: 1. all members of the family have common support, 2. let `… ; x† ˆ log Lik… ; x†, then the set of functions   @` … ; x† j i ˆ 1; . . . ; n @i are linearly independent, 3. moments of @`=@i … ; x† exist up to suf®ciently high order,

16

Paul Marriott and Mark Salmon

4. for all relevant functions integration and taking partial derivatives with respect to  are commutative. These conditions exclude a number of interesting models but will not, in general, be relevant for many standard econometric applications. All full exponential families satisfy these conditions, as do a large number of other classes of families. 2.4

Geometry

We now look at the general relationship between parametric statistical families and geometric objects known as manifolds. These can be thought of intuitively as multi-dimensional generalisations of surfaces. The theory of manifolds is fundamental to the development of differential geometry, although we do not need the full abstract theory (which would be found in any modern treatment such as Spivak (1979) or Dodson and Poston (1991)). We develop a simpli®ed theory suitable to explain the geometry of standard econometric models. Fundamental to this approach is the idea of an embedded manifold. Here the manifold is de®ned as a subset of a much simpler geometrical object called an af®ne space. This af®ne space construction avoids complications created by the need fully to specify and manipulate the in®nite-dimensional space of all proper density functions. Nothing is lost by just considering this af®ne space approach when the af®ne space we consider is essentially de®ned as the space of all loglikelihood functions. An advantage is that with this construction we can trust our standard Euclidean intuition based on surfaces inside threedimensional spaces regarding the geometry of the econometric models we want to consider. The most familiar geometry is three-dimensional Euclidean space, consisting of points, lines and planes. This is closely related to the geometry of a real vector space except for the issue of the choice of origin. In Euclidean space, unlike a vector space, there is no natural choice of origin. It is an example of an af®ne geometry, for which we have the following abstract de®nition. An af®ne space …X; V† consists of a set X, a vector space V, together with a translation operation ‡. This is de®ned for each v 2 V, as a function X !X x }x ‡ v which satis®es …x ‡ v1 † ‡ v2 ˆ x ‡ …v1 ‡ v2 †

Introduction to differential geometry

17

and is such that, for any pair of points in X, there is a unique translation between them. Most intuitive notions about Euclidean space will carry over to general af®ne spaces, although care has to be taken in in®nite-dimensional examples. We shall therefore begin our de®nition of a manifold by ®rst considering curved subspaces of Euclidean space. 2.4.1

Embedded manifolds As with our curved exponential family examples, curved subspaces can be de®ned either using parametric functions or as solutions to a set of restrictions. The following simple but abstract example can most easily get the ideas across. Example 6. The sphere model Consider in R3 , with some ®xed origin and axes, the set of points which are the solutions of x2 ‡ y2 ‡ z2 ˆ 1: This is of course the unit sphere, centred at the origin. It is an example of an embedded manifold in R3 and is a curved two-dimensional surface. At least part of the sphere can also be de®ned more directly, using parameters, as the set of points ÿ
þ cos…1 † sin…2 †; sin…1 † sin…2 †; cos…2 † j  1 2 …ÿ; †; 2 2 …0; † : Note that both the north and south poles have been excluded in this de®nition, as well as the curve …ÿ sin…2 †; 0; cos…2 ††: The poles are omitted in order to ensure that the map from the parameter space to R3 is invertible. The line is omitted to give a geometric regularity known as an immersion. Essentially we want to keep the topology of the parameter space consistent with that of its image in R3 . The key idea here is we want the parameter space, which is an open set in Euclidean space, to represent the model as faithfully as possible. Thus it should have the same topology and the same smoothness structure. We shall now give a formal de®nition of a manifold that will be suf®cient for our purposes. Our manifolds will always be subsets of some ®xed af®ne space, so more properly we are de®ning a submanifold. Consider a smooth map from , an open subset of Rr , to the af®ne space …X; V† de®ned by i :  ! X:

18

Paul Marriott and Mark Salmon

The set i…† will be an embedded manifold if the following conditions apply: (A) the derivative of i has full rank r for all points in , (B) i is a proper map, that is the inverse image of any compact set is itself compact (see BroÈcker and JaÈnich (1982), p. 71). In the sphere example it is Condition (A) that makes us exclude the poles and Condition (B) that excludes the line. This is necessary for the map to be a diffeomorphism and this in turn is required to ensure the parameters represent unique points in the manifold and hence the econometric model is well de®ned and identi®ed. Another way of de®ning a (sub)manifold of an af®ne space it to use a set of restriction functions. Here the formal de®nition is: Consider a smooth map  from an n-dimensional af®ne space …X; V† to Rr . Consider the set of solutions of the restriction  þ x j …x† ˆ 0 ; and suppose that for all points in this set the Jacobian of  has rank r, then the set will be an …n ÿ r†-dimensional manifold. There are two points to notice in this alternative de®nition. Firstly, we have applied it only to restrictions of ®nite-dimensional af®ne spaces. The generalisation to the in®nite-dimensional case is somewhat more technical. Secondly, the two alternatives will be locally equivalent due to the inverse function theorem (see Rudin (1976)). We note again that many standard differential geometric textbooks do not assume that a manifold need be a subset of an af®ne space, and therefore they require a good deal more machinery in their de®nitions. Loosely, the general de®nition states that a manifold is locally diffeomorphic to an open subset of Euclidean space. At each point of the manifold there will be a small local region in which the manifold looks like a curved piece of Euclidean space. The structure is arranged such that these local subsets can be combined in a smooth way. A number of technical issues are required to make such an approach rigorous in the current setting. Again we emphasise that we will always have an embedded structure for econometric applications, thus we can sidestep a lengthy theoretical development. Also it is common, in standard geometric works, to regard parameters not as labels to distinguish points but rather as functions of these points. Thus if M is an r-dimensional manifold then a set of parameters …1 ; . . . ; r † is a set of smooth functions i : M ! R:

Introduction to differential geometry

19

In fact this is very much in line with an econometric view of parameters in which the structural parameters of the model are functions of the probability structure of the model. For example, we could parameterise a family of distributions using a ®nite set of moments. Moments are clearly most naturally thought of as functions of the points, when points of the manifolds are actually distribution functions. 2.4.2

Statistical manifolds In this section we show how parametric families of densities can be seen as manifolds. First we need to de®ne the af®ne space that embeds all our families, and we follow the approach of Murray and Rice (1993) in this development. Rather than working with densities directly we work with log-likelihoods, since this enables the natural af®ne structure to become more apparent. However, because of the nature of the likelihood function some care is needed with this de®nition. Consider the set of all (smooth) positive densities with a ®xed common support S, each of which is de®ned relative to some ®xed measure . Let this family be denoted by P. Further let us denote by M the set of all positive measures that are absolutely continuous with respect to . It will be convenient to consider this set up to scaling by a positive constant. That is, we will consider two such measures equivalent if and only if they differ by multiplication of a constant. We denote this space by M . De®ne X by X ˆ flog …m† j m 2 M g: Because m 2 M is de®ned only up to a scaling constant, we must have the identi®cation in X that log …m† ˆ log …Cm† ˆ log …m† ‡ log …C†;

8 C 2 R‡ :

Note that the space of log-likelihood functions is a natural subset of X. A log-likelihood is de®ned only up to the addition of a constant (Cox and Hinkley (1974)). Thus any log-likelihood log …p…x†† will be equivalent to log …p…x†† ‡ log …C† for all C 2 R‡ . Finally de®ne the vector space V by V ˆ f f …x† j f 2 C 1 …S; R†g: The pair …X; V† is given an af®ne space structure by de®ning translations as log …m† } log …m† ‡ f …x† ˆ log …exp … f …x†m††: Since exp… f …x†m† is a positive measure, the image of this map does lie in X. It is then immediate that …log …m† ‡ f1 † ‡ f2 ˆ log …m† ‡ … f1 ‡ f2 † and the translation from log …m1 † to log …m2 † is uniquely de®ned by log …m2 † ÿ log …m1 † 2 C1 …S; R†, hence the conditions for an af®ne space apply.

20

Paul Marriott and Mark Salmon

Using this natural af®ne structure, consider a parametric family of densities that satis®es the regularity conditions from section 2.3. Condition 1 implies that the set of log-likelihoods de®ned by this family will lie in X. From Condition 2 it follows that Condition (A) holds immediately. Condition (B) will hold for almost all econometric models; in particular it will always hold if the parameter space is compact and in practice this will not be a serious restriction. Hence the family will be an (embedded) manifold. We note further that the set P is de®ned by a simple restriction function as a subset of M. This is because all elements of P must integrate to one. There is some intuitive value in therefore thinking of P as a submanifold of M. However, as pointed out in section 2.4.1, the de®nition of a manifold by a restriction function works most simply when the embedding space is ®nite-dimensional. There are technical issues involved in formalising the above intuition, which we do not discuss here. However, this intuitive idea is useful for understanding the geometric nature of full exponential families. Their log-likelihood representation will be i si …x† ÿ ý…†: This can be viewed in two parts. Firstly, an af®ne function of the parameters  ®ts naturally into the af®ne structure of X. Secondly, there is a normalising term ý…† which ensures that the integral of the density is constrained to be one. Very loosely think of M as an af®ne space in which P is a curved subspace; the role of the function ý is to project an af®ne function of the natural parameters back into P. Example 4 AR…1†-model (continued) We illustrate the previous theory with an explicit calculation for the AR…1† model. We can consider this family as a subfamily of the n-dimensional multivariate normal model, where n is the sample size. This is the model that determines the innovation process. Thus it is a submodel of an n-dimensional full exponential family. In fact it lies in a three-dimensional subfamily of this full exponential family. This is the smallest full family that contains the AR…1† family and its dimension is determined by the dimension of the minimum suf®cient statistic. The dimension of the family itself is determined by its parameter space, given in our case by  and . It is a …3; 2† curved exponential family. Its log-likelihood representation is

Introduction to differential geometry

21



 n  X n ÿ1 X  2 `…;  : x† ˆ xi ‡ 2 xt xtÿ1 2 2 iˆ1  iˆ1 ý

‡

! n ÿ2 X n x2tÿ1 ÿ log …2 2 †: 2 2 2 iˆ1

2.4.3

Repeated samples The previous section has demonstrated that a parametric family p…x j † has the structure of a geometric manifold. However, in statistical application we need to deal with repeated samples ± independent or dependent. So we need to consider a set of related manifolds that are indexed by the sample size n. The exponential family has a particularly simple structure to this sequence of manifolds. One reason for the simple structure is the fact that the dimension of the suf®cient statistic does not depend on the sample size. If X has density function given by (1), then an i.i.d. sample …x1 ; . . . ; xn † has density ( ) n n X Y i p……x1 ; . . . ; xn † j † ˆ exp  si …xj † ÿ ný…† m…xj †: jˆ1

jˆ1

This is also therefore a full exponential family, hence an embedded manifold. Much of the application of geometric theory is concerned with asymptotic results. Hence we would be interested in the limiting form of this sequence of manifolds as n ! 1. The simple relationship between the geometry and the sample size in full exponential families is then used to our advantage. In the case of linear models or dependent data, the story will of course be more complex. There will still be a sequence of embedded manifolds but care needs to be taken with, for example, the limit distribution of exogenous variables. As long as the likelihood function for a given model can be de®ned, the geometric construction we have set up will apply. In general econometric models with dependent data and issues of exogeneity, the correct conditional distributions have to be used to de®ne the appropriate likelihood for our geometric analysis, as was implicitly done in the AR…1† example above with the prediction error decomposition. 2.4.4

Bibliographical remarks The term curved exponential family is due to Efron (1975, 1978 and 1982) in a series of seminal papers that revived interest in the geometric aspects of statistical theory. This work was followed by a series of papers by Amari et al., most of the material from which can be found in

22

Paul Marriott and Mark Salmon

Amari (1990). The particular geometry treatment in this section owes a lot to Murray and Rice's (1993) more mathematical based approach, as well as to the excellent reference work by Kass and Vos (1997). Since the exponential family class includes all the standard building blocks of statistical theory, the relevant references go back to the beginnings of probability and statistics. Good general references are, however, Brown (1986) and Barndorff-Nielsen (1978, 1988). 3

The tangent space

We have seen that parametric families of density functions can take the mathematical form of manifolds. However, this in itself has not de®ned the geometric structure of the family. It is only the foundation stone on which a geometric development stands. In this section we concentrate on the key idea of a differential geometric approach. This is the notion of a tangent space. We ®rst look at this idea from a statistical point of view, de®ning familiar statistical objects such as the score vector. We then show that these are precisely what the differential geometric development requires. Again we shall depart from the form of the development that a standard abstract geometric text might follow, as we can exploit the embedding structure that was carefully set up in section 2.4.2. This structure provides the simplest accurate description of the geometric relationship between the score vectors, the maximum likelihood estimator and likelihood-based inference more generally. 3.1

Statistical representations

We have used the log-likelihood representation above as an important geometric tool. Closely related is the score vector, de®ned as   @` @` 0 ; . . . ; : @r @1 One of the fundamental properties of the score comes from the following familiar argument. Since … p…x j †d ˆ 1; it follows that … … @ @ p…x j †d ˆ 0 p …x j †d ˆ @i @i using regularity condition 4 in section 2.3, then

Introduction to differential geometry

 Ep…x;†

@` @i



… ˆ

1 @ p…x j †p…x j †d ˆ 0: p…x j † @i

23

…4†

We present this argument in detail because it has important implications for the development of the geometric theory. Equation (4) is the basis of many standard asymptotic results when combined with a Taylor expansion around the maximum likelihood esti^ mate (MLE), ;   ^i ÿ i ˆ I ij @` ‡ O 1 …5† n @j where ý

!ÿ1 @2 ` ^ ˆ I ij ÿ i j …† @ @

(see Cox and Hinkley (1974)). This shows that, in an asymptotically shrinking neighbourhood of the data-generation process, the score statistic will be directly related to the MLE. The geometric signi®cance of this local approximation will be shown in section 3.2. The ef®ciency of the maximum likelihood estimates is usually measured by the covariance of the score vector or the expected Fisher information matrix: ý !   @2 ` @` @` ; : Iij ˆ Ep…x;† ÿ i j ˆ Covp…x;† @ @ @i @j Efron and Hinkley (1978), however, argue that a more relevant and hence accurate measure of this precision is given by the observed Fisher information I ij ˆ ÿ

@2 ` ^ …†; @i @j

since this is the appropriate measure to be used after conditioning on suitable ancillary statistics. The ®nal property of the score vector we need is its behaviour under conditioning by an ancillary statistic. Suppose that the statistic a is exactly ancillary for , and we wish to undertake inference conditionally on a. We then should look at the conditional log-likelihood function `… j a† ˆ log …p…x j; a††: However, when a is exactly ancillary,

24

Paul Marriott and Mark Salmon

@` @` … : ja† ˆ i …†; i @ @ in other words, the conditional score will equal the unconditional. Thus the score is unaffected by conditioning on any exact ancillary. Because of this the statistical manifold we need to work with is the same whether we work conditionally or unconditionally because the af®ne space differs only by a translation that is invariant. 3.2

Geometrical theory

Having reviewed the properties of the score vector we now look at the abstract notion of a tangent space to a manifold. It turns out that the space of score vectors de®ned above will be a statistical representation of this general and important geometrical construction. We shall look at two different, but mathematically equivalent, characterisations of a tangent space. Firstly, we note again that we study only manifolds that are embedded in af®ne spaces. These manifolds will in general be non-linear, or curved, objects. It is natural to try and understand a non-linear object by linearising. Therefore we could study a curved manifold by ®nding the best af®ne approximation at a point. The properties of the curved manifold, in a small neighbourhood of this point, will be approximated by those in the vector space. The second approach to the tangent space at a point is to view it as the set of all directions in the manifold at that point. If the manifold were rdimensional then we would expect this space to have the same dimension, in fact to be an r-dimensional af®ne space. 3.2.1

Local af®ne approximation We ®rst consider the local approximation approach. Let M be an r-dimensional manifold, embedded in an af®ne space N, and let p be a point in M. We ®rst de®ne a tangent vector to a curve in a manifold M. A curve is de®ned to be a smooth map ÿ : …ÿ; †  R ! M t } ÿ…t†; such that ÿ…0† ˆ p. The tangent vector at p will be de®ned by ÿ 0 …0† ˆ lim

h!0

ÿ…h† ÿ ÿ…0† : h

Introduction to differential geometry

25

We note that, since we are embedded in N, ÿ 0 …0† will be an element of this af®ne space (see Dodson and Poston (1991)). It will be a vector whose origin is p. The tangent vector will be the best linear approximation to the curve ÿ, at p. It is the unique line that approximates the curve to ®rst order (see Willmore (1959), p. 8). We can then de®ne TMp , the tangent space at p, to be the set of all tangent vectors to curves through p. Let us put a parameterisation  of an open neighbourhood which includes p on M. We de®ne this as a map   : … Rr † ! N; where …† is an open subset of M that contains p, and the derivative is assumed to have full rank for all points in . Any curve can then be written as a composite function in terms of the parameterisation,   ÿ : …ÿ; † !  ! N: Thus any tangent vector will be of the form @ di : @i dt Hence TMp will be spanned by the vectors of N given by   @ ; i ˆ 1; . . . ; r : @i Thus TMp will be a p-dimensional af®ne subspace of N. For completeness we need to show that the construction of TMp is in fact independent of the choice of the parameterisation. We shall see this later, but for details see Willmore (1959). 3.2.2

Space of directions The second approach to de®ning the tangent space is to think of a tangent vector as de®ning a direction in the manifold M. We de®ne a direction in terms of a directional derivative. Thus a tangent vector will be viewed as a differential operator that corresponds to the directional derivative in a given direction. The following notation is used for a tangent vector, which makes clear its role as a directional derivative @ ˆ @i : @i It is convenient in this viewpoint to use an axiomatic approach. Suppose M is a smooth manifold. A tangent vector at p 2 M is a mapping Xp : C 1 …M† ! R

26

Paul Marriott and Mark Salmon

such that for all f, g 2 C 1 …M†, and a 2 R: 1. Xp …a:f ‡ g† ˆ aXp … f † ‡ Xp …g†, 2. Xp … f :g† ˆ g:Xp … f † ‡ f :Xp …g†. It can be shown that the set of such tangent vectors will form an rdimensional vector space, spanned by the set  þ @i ; i ˆ 1; . . . ; r ; further that this vector space will be isomorphic to that de®ned in section 3.2.1. For details see Dodson and Poston (1991). It is useful to have both viewpoints of the nature of a tangent vector. The clearest intuition follows from the development in section 3.2.1, whereas for mathematical tractability the axiomatic view, in this section, is superior. 3.2.3

The dual space We have seen that the tangent space TMp is a vector space whose origin is at p. We can think of it as a subspace of the af®ne embedding space. Since it is a vector space it is natural to consider its dual space TMp . This is de®ned as the space of all linear maps TMp ! R:

Given a parameterisation, we have seen we have a basis for TMp given by f@1 ; . . . ; @r g: The standard theory for dual spaces (see Dodson and Poston (1991)) shows that we can de®ne a basis for TMp to be  1 þ d ; . . . ; dr ; where each di is de®ned by the relationship ( 1 if i ˆ j j @i …d † ˆ 0 if i 6ˆ j: We can interpret di , called a 1-form or differential, as a real valued function de®ned on the manifold M which is constant in all tangent directions apart from the @i direction. The level set of the 1-forms de®nes the coordinate grid on the manifold given by the parameterisation . 3.2.4

The change of parameterisation formulae So far we have de®ned the tangent space explicitly by using a set of basis vectors in the embedding space. This set was chosen through the

Introduction to differential geometry

27

choice of parameterisation. Since this parameterisation is not intrinsic to the geometry of the manifold we must be able to convert from the basis de®ned in terms of one parameterisation to that de®ned in terms of another. Indeed we must show that the tangent space and its dual exist independently of the choice of parameters used. The change of parameterisation formulae in fact turn out to be repeated applications of the standard chain rule from multivariate calculus. Suppose we have two parameterisations  and , for an open subset of the manifold M. Owing to the full rank of the derivative of parameterisations there is a diffeomorphism connecting the parameters, i.e.  ˆ …† is a smooth invertible map, whose inverse is smooth. This follows from the inverse function theorem and the fact that we are only working locally in an open neighbourhood of some point in the manifold (see for example Kass and Vos (1997), p. 300). We write @a i to denote the i'th partial derivative of  with respect to the a'th component of . So we have @a i ˆ

@i …†: @a

Application of the chain rule then allows us to connect the basis in the parameters, which we denote by f@a j a ˆ 1; . . . ; rg, indexed by a, with the basis relative to the -parameterisation, which is denoted by f@i j i ˆ 1; . . . ; rg. Thus, recalling from section 3.2.2 that the basis elements are de®ned as differential operators, we can apply the chain rule. This gives the formula connecting the two bases as being @a ˆ …@a i †@i : Note here, as throughout, the use of the Einstein summation notation. Since this is just an invertible linear transformation, this result immediately implies that the tangent space spanned by the two basis sets will be the same. Hence we have shown that the previous de®nitions are indeed well de®ned. Further, to complete the change of basis formulae, suppose we have a tangent vector X. We could write it in component terms relative to the parameterisation as X ˆ X a @a : Thus, changing basis gives ÿ
X ˆ X a @a i @i ÿ
ˆ X a @a i @i :

28

Paul Marriott and Mark Salmon

Hence the components will have the following change of basis formula …X 1 ; . . . ; X r † } …X a @a 1 ; . . . ; X a @a 1 †:

…6†

The change of basis formulae for the dual space can be derived in a similar way. The relationship connecting the basis relative to the parameterisation, fda j : a ˆ 1; . . . ; rg, with that of the -parameterisation, fdi j : i ˆ 1; . . . ; rg, is given by da ˆ @i a di ; where @i a ˆ @i =@a . Note that there is also the important relationship between the two changes of basis matrices, which states ( 1 if i ˆ j @i a @a j ˆ …7† 0 if i 6ˆ j: That is, viewed as matrices, @i a and @a j are mutual inverses. 3.2.5

Statistical interpretation We can now relate this previous theory to our statistical manifolds. Recall that we are representing the manifolds by their log-likelihood representations. Let us now consider the representation of the tangent space in terms of the local af®ne approximation to the manifold. Let the parametric family be given by p…x j †, then its log-likelihood representation is given by log p… j x† ˆ `…†. A path in this manifold, through the point represented by 0 , is given by ÿ : …ÿ; † !  ! X t } `……t††: Thus the tangent space will be spanned by the set of random variables   @` @` ; . . . ; : @r @1 Hence the set of score vectors spans the tangent space. It will be convenient to swop freely between the two interpretations of tangent vectors ± ®rstly random variables, which are elements of the above vector space, and secondly directional derivatives. We shall use the following notation throughout, @i `…† ˆ

@` @i

when we wish to emphasise the random variable nature of the tangent vector.

Introduction to differential geometry

29

3.2.6

Taylor's theorem Suppose we have a real valued function h de®ned on our statistical manifold M. Most asymptotic theory is de®ned using a form of expansion of such functions. Using the machinery we have already developed, we can look at a ®rst-order Taylor expansion that is relevant for curved manifolds. Suppose we have a parameterisation  on the manifold, then Taylor's expansion will be h…† ˆ h…0 † ‡ @i h…0 †di ‡ higher-order terms: We note however, owing to the change of basis formulae in section 3.2.4, in particular equation (7), that this expansion will be invariant. That is to say, the expansion will be the same for all parameterisations. The question of invariance to reparameterisation is an important one in statistics, and has motivated much of our geometric development. We shall see that the extension of this formula, which is needed for higher-order asymptotics, requires more knowledge of the underlying geometry of the manifold, and is considered in detail in Blsild (1987). 3.2.7

Further structures Having de®ned a single tangent space for a manifold M at the point p 2 M, we can de®ne the tangent bundle, denoted by TM. This is the collection of all tangent spaces, i.e. TM ˆ fTMp j p 2 Mg:

The tangent bundle itself will have the structure of a manifold; in fact, if M is r-dimensional then TM will be a 2r-dimensional manifold (see Dodson and Poston (1991)). We will also use the notion of a tangent ®eld on the manifold. Intuitively we can think of this associating a tangent vector at each point p that lies in the tangent space TMp . Formally, a tangent ®eld X is a (smooth) map between manifolds X : M ! TM p } X…p† 2 TMp : We denote the set of all tangent ®elds by …M†. In a similar way we can de®ne the cotangent bundle T  M by T  M ˆ fT  Mp j p 2 Mg; and a cotangent ®eld X  by

30

Paul Marriott and Mark Salmon

X  : M ! T M p } X  …p† 2 T  Mp : We denote the set of all cotangent ®elds by  …M†. 4

Metric tensors

Having carefully de®ned the tangent and cotangent spaces we are now able to de®ne part of the underlying geometrical structure of a statistical manifold. Given that the tangent space can be viewed as a vector space, it is natural to ask if we have some way of measuring lengths of tangent vectors and angles between tangent vectors. We will see how this will enable us to de®ne, amongst other things, lengths of paths. The tool that we use is called a metric tensor. Since each tangent space is a vector space, the simplest way to measure lengths and angles is to prescribe a quadratic form for each tangent space. However, we would require that this quadratic form varies smoothly across the manifold. This gives the de®nition: a metric tensor, denoted by h ; i, is a smooth function h ; i : …M†  …M† ! C 1 …M† …X; Y† }hX; Yi: It satis®es the properties that for all X; Y; Z 2 …M†, and for all f 2 C1 …M†: 1. hX; Yi ˆ hY; Xi (symmetry), 2. h fX; Yi ˆ f hX; Yi and hX ‡ Y; Zi ˆ hX; Zi ‡ hY; Zi (bilinearity), 3. hX; Xi > 0 (positive de®niteness). This de®nition is best understood when you consider what happens at the tangent space at a particular point p. On this vector space, Tp M, the metric h ; ip is simply a quadratic form. The key to the above de®nition lies in ensuring that this quadratic form varies smoothly across the manifold with p. Suppose now that we introduce a parameterisation, . Since the metric is a bilinear form it can be written as an element of the tensor product of 1-forms, T  M þ T  M. We can express it in terms of the basis for T  M given by the parameterisation, i.e. relative to fd1 ; . . . ; dr g. Thus we have the expression h ; i ˆ gab da db ;

where gab ˆ h@a ; @b i:

Just as we write any tangent vector in terms of its components with respect to the basis f@1 ; . . . ; @r g, the metric is often simply denoted by

Introduction to differential geometry

31

its components gab ˆ gab …† relative to the basis fd1 ; . . . ; dr g. It is important then to understand how this `component form' representation of the metric changes with a change of parameters. Suppose that  is another parameterisation, then we have h ; i ˆ gab @i a @j b di dj ˆ g~ ij di dj ; where g~ij is the coordinate form for the -parameterisation. Hence the change of basis formula for the coordinate version of the metric is g~ ij ˆ @i a @j b gab :

…8†

Remember that we are using the Einstein convention. From the above de®nition we can see that, as long as the components of the metric obey the transformation rule in equation (8), then lengths and angles de®ned by the metric will be invariant to changes in parameterisation. The following example shows how the metric can be induced by the geometry on the embedding space. Example 6. The sphere example (continued) embedded in R3 , which is given parametrically by

Consider the sphere

f…cos…1 † sin…2 †; sin…1 † sin…2 †; cos…2 †† j 1 2 …ÿ; †; 2 2 …0; †g: The tangent space at the point …1 ; 2 † is spanned by the vectors in R3 ÿ
@1 ˆ ÿ sin…1 † sin…2 †; cos…1 † sin…2 †; 0 ÿ
@2 ˆ cos…1 † cos…2 †; sin…1 † cos…2 †; ÿ sin…2 † : This surface is embedded in R3 . Let us suppose we impose on R3 the standard Euclidean inner product, h ; iR3 . Since @1 ; @2 2 R3 we can measure their lengths and angles with this inner product. Thus we can de®ne gij ˆ h@i ; @j iR3 ; and we have the metric 0 1 sin2 …2 † 0 @ A: 0 1 In general, if the embedding space N is not just an af®ne space, but also has a ®xed inner product h ; iN , then this will induce the embedding metric. In component form, relative to a parameterisation , this will be gij ˆ h@i ; @j iN :

32

Paul Marriott and Mark Salmon

We shall return to this notion in section 4.2. Having a metric tensor on the manifold not only de®nes the length of tangent vectors; it also can be used to de®ne the length of curves. Suppose that the curve ÿ : …0; 1†… R† ! M t } ÿ…t† is a smooth map such that ÿ…0† ˆ p1 and ÿ…1† ˆ p2 . For any t 2 …0; 1† the corresponding tangent vector is dÿ dt …t†, with length s    dÿ dÿ …t†; …t† dt dt ÿ…t† and the length of the curve from p1 to p2 is de®ned as    … 1 s dÿ dÿ …t†; …t† dt: dt dt 0 ÿ…t† Owing to the invariance properties of the metric, this will be an invariant measure of arc-length. Given two such points on a manifold we can ask the natural geometric question: what is the shortest path connecting them? Although this is a natural question, in general the complete answer is rather complex. There are examples where there will not be a shortest path in a manifold, or there may be many. There is a simpler answer, however, to the question: is it a path of locally minimum length, in a variational sense? That is, its length is a local minimum in the class of all paths joining p1 and p2 . A path that satis®es this condition is called a geodesic. Given a metric on a manifold, its geodesics can be characterised by a set of differential equations. We will see this representation in section 5. 4.1

Tensors

Tangent vectors, cotangent vectors and metrics are all examples of the general geometric notion of a tensor. All of these examples show how the careful de®nition of transformation rules allows de®nitions of invariant quantities to be made. This invariance to the choice of parameterisation is very important not only geometrically but also for statistical purposes. In general we would not want the tools of inference to depend on arbitrary choices in the model speci®cation. Using a geometric approach will, almost by de®nition, avoid these problems; for an example of this see Critchley, Marriott and Salmon (1996). For a good reference to where

Introduction to differential geometry

33

tensors have had a direct application to statistical theory, see McCullagh (1987). A general tensor is made up of multivariate linear functions and functionals of the tangent ®elds. They come in two types. The ®rst are covariant tensors, which on each tangent space are high-dimensional generalisations of matrices. For example, the metric is simply a bilinear form that varies smoothly with the point of evaluation. The second type are called contravariant tensors, which are simply products of tangent vectors when evaluated at each point. We have the following formal de®nitions: A covariant tensor of degree r on Tp M is an r-linear real valued function on the r-fold product Tp M  . . .  Tp M. A contravariant tensor ®eld of degree r is an r-linear real valued function on the r-fold product Tp M  . . .  Tp M. A tensor ®eld is a set of tensors on Tp M that varies smoothly with p 2 M. 4.1.1

Components of tensors To work with these tensors in computations it is necessary to write them out in component form with respect to the bases which span TMp and TMp . Given a parameter system , these bases will be f@1 ; . . . ; @r g and fd1 ; . . . ; dr g: Thus a k-covariant tensor can be written as Ai1 i2 ...ik di1 di2 . . . dik : This is, by de®nition, a k-linear real valued function on the k-fold product Tp M  . . .  Tp M, which acts on tangent vectors in the following way. If we have a set of k tangent vectors v1 ; . . . ; vk , which we write with respect to the basis as vi ˆ vij @j ; then the tensor acts on the set of vectors as Ai1 i2 ...ik di1 di2 . . . dik …v1 ; . . . ; vk † ˆ Ai1 i2 ...ik vi11 . . . vikk ; recalling again the Einstein convention. To see how the components of this tensor will transform under a change of parameters, we apply the same methodology as we did for the metric. Let  be the original parameterisation and let the components of the tensor with respect to this parameterisation be Aa1 a2 ...ak . Let  be

34

Paul Marriott and Mark Salmon

the new parameterisation, with components A~ i1 i2 ...ik . This will give a transformation rule A~ i1 i2 ...ik ˆ @i1 a1 @i2 a2 . . . @ik ak Aa1 a2 ...ak :

…9†

For a contravariant tensor the component form will be Ai1 i2 ...ik @i1 . . . @ik ; which acts on k cotangent vectors v1 ; . . . ; vk , which we write as vi ˆ vij dj : The action is then Ai1 i2 ...ik @i1 ...@ik …v1 ; . . . ; vk † ˆ Ai1 i2 ...ik v1i1 . . . vkik ; and the transformation rule will then be A~ i1 i2 ...ik ˆ @a1 i1 @a2 i2 . . . @ak ik Aa1 a2 ...ak :

…10†

4.1.2

Raising and lowering indices The general theory of dual spaces tells us that if V is a vector space and V  its dual space, there will be a natural isomorphism between V and V  , the dual of the dual space, de®ned by V ! V  v } v ;

where v : V  ! R is de®ned by v …A† ˆ A…v† for all A 2 V  . However, in general there will not be a natural isomorphism between V and V  unless there is an inner product h ; i on V. In that case the isomorphism is de®ned by V ! V v }hv; :i; where we interpret hv; :i as an element of V  since hv; :i : V ! R w }hv; wi: For a manifold with a metric, of course, we do have an inner product on each of the vector spaces de®ned by the tangent space. This will, by the above theory, allow us to de®ne a natural isomorphism between tangent space TM and dual space T  M. More generally it will enable

Introduction to differential geometry

35

us to convert between covariant and contravariant tensors. Thus, if in component terms we have a 1-covariant tensor, given by Ai , then it can be converted to a contravariant tensor by using the metric. In detail we get Aj ˆ gij Ai , where gij is the inverse to the metric gij : 4.2

Statistical metrics

Having seen the concept of a metric and a tensor in generality, we turn now to statistical applications. In this section we shall see that these tools are not new to statistical theory and that they are essential to understanding the nature of invariance. We look ®rst at three examples of metrics in statistics and their application. 4.2.1

Expected Fisher information We have already seen the expected Fisher information matrix in section 3.1: ý !   @2 ` @` @` ; Iij ˆ Ep…x;† ÿ i j ˆ Covp…x;† : @ @ @i @j These are the components of a covariant 2-tensor which can be easily checked by seeing how it transforms under a change of parameters using equation (9). This is a simple exercise in the chain rule. Further, it will, under regularity, be positive de®nite and a smooth function of the parameter. Hence it has the properties of the components of a metric tensor. In general we will abuse notation and refer to Iij as being a metric, dropping reference to components and explicit parameter dependence. For a full exponential family, in the natural parameterisation, the expected Fisher information matrix will be given by Iij …† ˆ

@2 ý …†: @i @j

Applying equations (3) and (8) shows immediately that the components in the expected -parameters will be ý !ÿ1 @2 ý ……†† : @i  j Using the convexity of ý we see that the positive de®niteness is assured, hence in the full exponential family case the expected Fisher information matrix is a metric tensor.

36

Paul Marriott and Mark Salmon

Although this has the formal properties of being a metric, we have to ask in what way does this act on the tangent space? In particular, what statistical interpretation can be given? This will become clear if we use the interpretation of the tangent vectors as being random variables, elements of the space spanned by   @` @` : ; . . . ; @r @1 The expected Fisher information is the covariance for this space of random variables. Due to the additive properties of the log-likelihood, the central limit theorem tells us that asymptotically linear combinations of score vectors will be normally distributed. Equation (4) tells us that the mean will be zero, thus the metric completely determines the stochastic behaviour of the space, at least to ®rst order. This interpretation, although being completely standard, does need careful re®nement in our geometric viewpoint. We will return to this issue below. Having seen the expected Fisher information metric in the case of full exponential families, it is natural to ask what form it takes in a curved exponential family. Given an …r; t†-curved exponential family p…x j † ˆ expfi …†si ÿ ý……††gm…x†; the tangent space to this family will be spanned by the set   @` @` ; ; . . . ; @t @1 where   @` @j @ý @j ˆ i sj ÿ j ˆ i @`j : i @ @ @ @ The construction of an embedding metric, as seen in Example 6, can be used here. The metric on the curved family is completely determined by that on the embedding full exponential family. In component form this is given by gij , where gij ˆ 4.2.2

@k @l I : @ i @j kl

…11†

Applications We now give two examples where the metric properties of the expected Fisher information have found application in statistics. The ®rst

Introduction to differential geometry

37

concerns the question of an invariant de®nition of length, and the second an invariant de®nition of angle. The ®rst example concerns the score test. Suppose we have an …r; 1†curved exponential family p…x j † ˆ expfi …†si …x† ÿ ý……††g; and we wish to test the null hypothesis  ˆ 0 using the score test. The components of the score vector will be   i @ý d ……0 ††: S……0 †† ˆ si ÿ i ……0 †† d @ The variance of the score is the expected Fisher information matrix, which can be calculated using equation (11) as Vˆ

dk dl I : d d kl

Hence the score statistic is S 0 V ÿ1 S: This is an invariant quantity for any reparameterisation of . It has the geometric interpretation of the length of a cotangent vector which is a tensor. For an example of how this invariance is important in econometrics, see Critchley, Marriott and Salmon (1996). The second example is where an angle is measured invariantly. Parameter orthogonality is a useful property when estimation is being made in the presence of nuisance parameters. For an example of this, see Barndorff-Nielsen and Cox (1994, p. 49). 4.2.3

Preferred point metrics The expected Fisher information metric is not the only possible statistically based metric. The concept of a preferred point geometry and the related preferred point metric was introduced in Marriott (1989). This is a very natural geometric and statistical construction for the embedding given in section 2.4.2. The embedding space can be viewed as a space of random variables. These random variables are simply functions of the data. It seems natural then that the data-generation process (DGP) plays some role in de®ning the geometry. Let us suppose that we have a parametric family p…x j † and the data-generation process is given by a member of this family, namely p…x j †. The point , although unknown, plays a distinctive role in the family p…x j †. It is called the preferred point. The tangent space at a general point  is spanned by the random variables   @` @` …†; . . . ; …† : @r @1

38

Paul Marriott and Mark Salmon

We can de®ne the covariance matrix of this space relative to the DGP, i.e.   @` @`  Iij …† ˆ Covp…x j † …†; j …† : …12† @ @i The important difference between this and the expected Fisher information matrix is that the covariance here is evaluated with respect to some ®xed distribution for all tangent spaces. In the Fisher information case the evaluating distribution varies with tangent space. By the same argument as for the Fisher information, equation (12) de®nes the components of a metric tensor, at least for all  in a neighbourhood of . The properties of such preferred point metrics are explored in Critchley, Marriott and Salmon (1993, 1994, and in this volume). We note further that we must reconsider equation (4) in the preferred point context. The mean value of the score vector will be zero only in the tangent space of the preferred point. In general we de®ne   @`  …† : …13† i …† ˆ Ep…x j † @i This de®nes a covariant 1-tensor on the manifold. For a full exponential family it is easy to check that the preferred point metric, in the natural parameters, is given by Iij …† ˆ

@2 ý …†: @i @j

Note that this is independent of  and thus is constant across all tangent spaces. We return to this property later. 4.2.4

Observed information metrics As mentioned in section 3.1, Efron and Hinkley (1978) argue that in general the observed Fisher information I ij ˆ ÿ

@2 ` ^ …†; @ i @j

^ since it is a better measure of the covariance of the score vector at , re¯ects the conditional distribution of the score. This will also give a statistically based metric. Here, though, the change of parameterisation rule requires some care. By applying the chain rule to @2 ` …† @i @j twice, we have the following formula:

Introduction to differential geometry

39

@2 ` @a @b @2 ` @2 a @` ……†† ˆ …† ‡ …†: @i @j @a @b @i @j @a @i @j The second term on the right-hand side of this expression disappears ^ giving the correct covariant 2-tensor transformation when evaluated at , rule. At all other values of , however, this will not transform as a tensor. For a full exponential family the metric has components 2 ^ ˆ @ ý …†; ^ I ij …† @i @j

…14†

the same as the expected Fisher metric. However, for a curved exponential family the metrics will differ. The observed information metric will be a stochastic function, whereas the expected form will be deterministic. The use of the observed information metric is best explained in the context of conditional inference. Let us suppose that we can write the minimum suf®cient statistic, …s1 …x†; . . . ; sr …x††, for the curved exponential ^ family p…x j † in the form ……x†; a…x††, where a is any ancillary statistic for  and ^ is the maximum likelihood estimate. We have seen in section 3.1 that the tangent space is invariant under conditioning on an exact ancillary statistic. Thus, as an asymptotic approximation the observed metric will be the conditional covariance of the tangent vector and this will be the observed information matrix. 4.2.5

Bibliographical notes The fact that the expected Fisher information has the properties of a metric tensor was ®rst observed by Rao (1945, 1987). The full theory of so-called expected geometry was developed in a series of papers by Amari (see Amari (1987, 1990)). Preferred point metrics were introduced by Marriott (1989). The relationship between these two forms of geometry is explored in the companion chapter in this volume by Critchley, Marriott and Salmon. The observed geometry was developed by Barndorff-Nielsen (1987) and developed to a more general theory of yokes, by Blsild (1987, 1991). 5

Af®ne connections

In section 2 it was stated that full exponential families can be thought of as af®ne subspaces, and that curved exponential families are curved subfamilies of these af®ne subspaces. In this section we shall see the formalisation of these statements. We have already studied the construction of the manifold, tangent bundle and metric structure of a parametric family. One ®nal structure is needed before a satisfactory geometric the-

40

Paul Marriott and Mark Salmon

ory for statistics can be developed. This is the idea of curvature and the absence of curvature, straightness and ¯atness. The tool used in differential geometry to describe these features is called an af®ne connection. Consider for motivation a one-dimensional subfamily of a parametric family p…x j †. We have already seen two ways in which we might consider this to be `straight'. Firstly, we have the af®ne structure of the embedding space. Af®ne subspaces are often considered to be straight. Secondly, we have the concept of a geodesic with respect to a metric tensor. Loosely, a geodesic is a curve of minimum length, and this is often intuitively thought of as being equivalent to straightness. However, detailed investigation shows that for parametric families these two notions are quite different. A curve that is `straight' according to this ®rst criterion will not be `straight' according to the second. Amari (1990) realised that the way out of this seeming paradox lay in the careful de®nition of the correct connection structure for statistics. In this section we shall ®rst examine the general geometric theory of connections and then de®ne the relevant statistical examples. 5.1

Geometric af®ne connections

We take as our fundamental notion of straightness for a curve that its tangent vectors are always parallel to some ®xed direction. However, to formalise this we must have a way of measuring the rate of change of a tangent ®eld. An af®ne connection gives us that tool. It can be thought of as a way to calculate the rate of change of one tangent ®eld with respect to another. We would want the result of the differentiation itself to be a tangent ®eld. Formally we de®ne a symmetric af®ne connection r to be a function r : …M†  …M† ! …M† …X; Y† }rX Y; which satis®es the following properties: for all X; X1 ; X2 ; Y; Y1 ; Y2 2 …M† and f 2 C 1 …M†, we have 1. rX1 ‡X2 Y ˆ rX1 Y ‡ rX2 Y, 2. rX …Y1 ‡ Y2 † ˆ rX Y1 ‡ rX Y2 , 3. rfX Y ˆ f rX Y, 4. rX …fY† ˆ f rX Y ‡ X…f †Y, 5. rX Y ÿ rY X ˆ XY ÿ YX. Note that here we are explicitly using the derivative version of the tangent vector given in section 3.2.2. Thus X… f † is the directional derivative of the function f in the direction stipulated by the tangent vector X. Conditions

Introduction to differential geometry

41

1, 2 and 3 de®ne the linearity properties of the connection. Condition 4 states that the connection is a derivative and hence satis®es the product rule. Condition 5 is the condition that makes the connection symmetric. We shall consider only symmetric connections. Just as in the case of a metric, we can write a connection in component form. Let us suppose that the vector ®elds X and Y are given with respect to the basis de®ned by the parameterisation , i.e. X ˆ X i @i Y ˆ Y i @i : By de®nition rX Y is also a tangent ®eld. If we de®ne r@i @j ˆ ÿkij @k ; then by the above Conditions 1±5 it must follow that rX Y ˆ …X i @i Y k ‡ X i Y j ÿkij †@k :

…15†

Thus the connection is completely determined by the three-way array ÿkij . These are called the Christoffel symbols for the connection. Note that Condition 5 implies that, for all i; j and k, ÿkij ˆ ÿkji ; hence the connection is called symmetric. It is important to note that the Christoffel symbols do not transform as tensors. If ÿcab are the Christoffel symbols relative to the -parameters and ÿ~ kij relative to the -parameters, then the two sets of components are connected according to the rule   ÿcab ˆ @k c @2ab k ‡ @a i @b j ÿ~ kij : …16† One consequence of this not being a tensor rule is that Christoffel symbols can be identically zero with respect to one parameterisation, but non-zero in another. Having de®ned a connection, we can see how the symbols are able to de®ne a straight line, or, as it is called, a geodesic. Let us de®ne a path in terms of a parameterisation, ÿ : ‰0; 1Š… R† ! M t } …t†: Note that this is a slight abuse of notation, identifying …1 …t†; . . . ;  r …t†† with the point in M with those parameters. The path will be a geodesic if it satis®es the equation

42

Paul Marriott and Mark Salmon

d 2 k di dj k ÿ ˆ 0: ‡ dt dt ij dt2

…17†

If the tangent vector to this path is X…t†, this equation is the component form of the equation rX X ˆ 0:

…18†

5.1.1

Connections and projections In order to understand the nature of a connection operator it is helpful to return to the sphere example and see how the most natural connection there operates on tangent ®elds. Example 6. Sphere example (continued) embedded in R3 with parameterisation

For the sphere

f…cos…1 † sin…2 †; sin…1 † sin…2 †; cos…2 †† j 1 2 …ÿ; †; 2 2 …0; †g; consider a path …1 ; 2 † ˆ …t; =2†: In R3 it is the path …cos…t†; sin…t†; 1†; the equator of the sphere. Suppose we differentiate with respect to t, giving a vector in R3 X…t† ˆ …ÿ sin…t†; cos…t†; 0†: This is the tangent vector to the curve. A connection allows us to differentiate tangent ®elds. The result of this differentiation should itself be a tangent vector, since the tangent space gives the set of all directions in the manifold. However, differentiating X…t† directly gives X_ ˆ …ÿ cos…t†; ÿ sin…t†; 0†: This does not lie in the tangent space, which is spanned by @1 ˆ …ÿ sin…t†; cos…t†; 0†

and

@2 ˆ …0; 0; ÿ1†:

If we want the derivative of the tangent ®eld, to be a tangent ®eld we must _ project X…t† back into TMp . Using the standard inner product on R3 , _ is in fact orthogonal to the tangent space. Hence h ; iR3 , we ®nd that X…t† the rate of change of X, relative to the sphere, will be zero. In other words, in the sphere the curve is straight. This is completely consistent

Introduction to differential geometry

43

with intuition. The equator is a great circle that is well known to provide the shortest distance between two points on a sphere. The intuition from the above example gives an interpretation of equation (15). It can be viewed as a standard derivative operation combined with a projection onto the tangent space. The exact form of the projection will depend on the form of the Christoffel symbols. For details of this construction, see Dodson and Poston (1991). We return to this view of the connection in section 5.2.1. 5.1.2

Metric connections Having seen how a connection de®nes a geodesic, we return to the second de®nition of straightness. In section 4 a geodesic was de®ned as a path of (locally) minimum length. It can be shown, that given a metric, there is a unique af®ne connection r0 , called the Levi±Civita connection, for which the two de®nitions of geodesic agree. That is to say, de®ning a metric automatically de®nes a particular connection, such that if a path satis®es equation (18) it will be a path of minimum length. We quote here two important ways of characterising the Levi±Civita connection. (i)

Theorem For a manifold M with a metric h ; ip , the Levi±Civita connection r0 is characterised as the only symmetric connection that satis®es XhY; Zip ˆ hrX0 Y; Zip ‡ hY; rX0 Zip ;

…19†

for all tangent ®elds X; Y; Z. (ii) If we introduce a parameter system , the Christoffel symbols of the Levi±Civita connection are given by   @gjl @gij 0k kl @gil ‡ i ÿ l ; ÿ ij ˆ g …20† @j @ @ where gij are the components of the metric. The proof of this theorem can be found in Dodson and Poston (1991). 5.1.3

Non-metric connections Connections do not, however, have to be constructed in this way. Any connection that is not the Levi±Civita connection of some underlying metric will be called a non-metric connection. Formally, all that is required is the de®nition of a set of Christoffel symbols that transform under a change of parameters according to equation (16). We shall see in

44

Paul Marriott and Mark Salmon

the following section that for statistical purposes non-metric connections play a more important role than metric ones. 5.2

Statistical connections

As we have previously noted, we have two distinct notions of straightness or ¯atness for our statistical manifolds: ®rstly from the af®ne structure of the embedding space, and secondly from the Levi±Civita connection of a statistically based metric. Initially we consider the case with the expected Fisher information as the metric tensor on the manifold which represents the parametric family. 5.2.1

The ‡1-connection We wish to de®ne a connection that is consistent with the af®ne structure of the embedding space. Relative to this embedding we have the tangent vector @i ˆ

@` ; @i

and hence @i @ j ˆ

@2 ` …; x†; @i @j

a random variable. This, as we have seen in section 5.1.1, will not necessarily lie in the tangent space. The embedding structure gives a possible way of projecting into TM . It is natural to use h f ; gi ˆ Covp…x j † … f ; g† for all elements f ; g of X. Using this we project @i @j onto the tangent space, which is spanned by the set   @` @` …†; . . . ; …† : @r @1 This will then de®ne a connection whose Christoffel symbols are ý ! @2 ` @` ‡1 k kl …† l …† : ÿ ij …† ˆ I …†Ep…x j † @ i @j @

…21†

This connection will be denoted by r‡1 . Calculating the Christoffel symbols for a full exponential family in its natural parameterisation and exploiting equation (4), gives that, for all i; j; k 2 f1; . . . ; rg,

Introduction to differential geometry

45

k

ÿ‡1 ij …† ˆ 0: We will return to this result later. 5.2.2

The 0-connection In section 5.1.2 it was shown that the metric tensor gives rise to the Levi±Civita connection. The Christoffel symbols are de®ned by equation (20). If we take I, the expected Fisher information matrix, as our metric, then the associated Levi±Civita connection will be given by ý ! @2 ` @` ‡1 k kl ÿ ij ˆ I …†Ep…x j † …† l …† @i @j @   1 kl @` @` @` …† j …† l …† : ‡ I …†Ep…x j † 2 @ @i @ For the full exponential family in the natural parameterisation, the Christoffel symbols are given by 1 @3 ý k ÿ0 ij ˆ I kl i j l : 2 @ @ @ This connection will be denoted by r0 . 5.2.3

The ÿ1-connection Again considering connections as the combination of a derivative operation followed by a projection to the tangent space gives us yet another plausible connection. As in section 5.2.1, we wish to project the random variable @2 ` …† @i @j into the tangent space. From equation (4) it is clear that any member of the tangent space must have zero expectation. Hence this motivates the projection ý ! @2 ` @2 ` @2 ` …† } i j …† ÿ Ep…x j † …† : @i @j @ @ @i @j

This operation is enough to de®ne a connection, rÿ1 , and its Christoffel symbols will be ý( ) ! @2 ` @` @` @` ÿ1 k kl ÿ ij ˆ I Ep…x; j † …† ‡ i …† j …† …† : @ @i @j @ @l

46

Paul Marriott and Mark Salmon

In the full exponential family, again in the natural coordinates, this has the form k

ÿÿ1 ij …† ˆ I kl …†

@3 ý …†: @i @j @l

Notice the similarity to the r0 case. 5.2.4

The -connection Amari (1987) pointed out that these different connections are all special cases of a large family of connections that have many statistical uses. They are called the -connections, and denoted by r . They can be thought of as a linear combination of any two of the above types. The general de®nition is that the Christoffel symbols are given by ý( ) ! @2 ` 1 ÿ  @` @` @` k kl …† ‡ …† j …† …† : ÿ ij ˆ I Ep…x; j † 2 @i @ @i @j @l

In the full exponential family this simpli®es to be ÿkij …† ˆ

1 ÿ  kl @3 ý …†: I 2 @i @j @l

in the natural parameterisation. 5.2.5

Statistical manifolds The development in the previous section concentrated on expected geometry. As we have seen, this is not the only sensible or important geometrical structure for a parametric family of distributions. However, the structure that was developed does seem to have a general applicability across most of the possible geometric structures. This was recognised by Lauritzen (1987). He de®ned a general structure that encompassed most forms of statistical geometry. A statistical manifold is de®ned as …M; g; T†, where M is a manifold of distribution functions, g is a metric tensor and T is a covariant 3-tensor that is symmetric in its components and called the skewness tensor. For this structure there will always be a set of af®ne connections that parallels the structure in the previous section. Theorem For any statistical manifold …M; g; T† there is a onedimensional family of af®ne connections de®ned by  ÿ ijk ˆ ÿ0 ijk ÿ Tijk ; 2 where r0 is the Levi±Civita connection for the metric g.

Introduction to differential geometry

47

In the case of expected geometry we have that   @` @` @` Tijk ˆ Ep…x j † : @i @j @k For the observed geometry we assume that the data can be written in ^ a†, where a is ancillary. The likethe conditional resolution form x ˆ …; lihood can then be written as ^ a†: `… ; x† ˆ `… ; ; We use the following notation `i ˆ

@ ^ a†; `… ; ; @i

and

`;i ˆ

@ ^ a†: `… ; ; @^i

…22†

In this notation it can be shown that gij ˆ ÿ`ij ˆ `i; j is the observed Fisher information metric, and Tijk ˆ `ijk is the skewness tensor.

6

Three key results

In this section we look at the key mathematical theorems of statistical manifolds; the applications to statistics are shown in sections 7 and 8. Throughout we shall assume that we have the structure of a statistical manifold …M; g; T† as de®ned in section 5.2.5. 6.1

Duality

The concept of duality for a statistical manifold is tied up with that of ¯atness. A connection de®nes the geometry of a manifold; in particular, it de®nes the curvature. There are many ways in which we can measure the curvature of a manifold. One issue of great interest is when there is no curvature, which is measured using the Riemann curvature tensor. This is a covariant 2-tensor de®ned in component terms using a connection as m m m Rijkl ˆ glm …@i ÿm jk ÿ @j ÿik † ‡ …ÿiml ÿjk ÿ ÿjml ÿik †:

…23†

If this tensor is identically zero for all  and all i; j; k; l 2 f1; . . . ; rg, then the manifold is said to be ¯at. It is an important theorem of differential geometry that if a manifold is ¯at then there exists a parameterisation  that has the property that the components of the Christoffel symbol in this parameterisation will be identically zero (see section 7). If the connection that de®nes the Riemann curvature is the Levi±Civita connection of the metric g, then this theorem extends. In this case there exists a parameterisation that has the property that the components gij …† will

48

Paul Marriott and Mark Salmon

be independent of , so that the metric will be a constant. In either of these cases the parameterisation is called an af®ne parameterisation. The duality theorem for statistical manifolds then states Theorem If …M; g; T† is a statistical manifold such that M is ¯at with respect to the connection r , then it will be ¯at with respect to r ÿ . We say that the ÿ-connection is dual to the -connection. The 0-connection, which is the Levi±Civita connection derived from the metric, will be self dual. One important application of this theorem is to the full exponential family. In section 5.2.1 it is shown that in the natural parameterisation the Christoffel symbols of the r‡1 -connection are identically zero. Thus for this connection a full exponential family is ¯at and the natural parameters are af®ne. The theorem states that a full exponential family will also be ¯at with respect to the ÿ1-connection. There is also therefore a set of parameters that is af®ne for this connection. These parameters are the expectation parameters de®ned in section 2.1.2. The relationship between a dual pair of connections is illuminated by the following result, which should be compared with the de®ning equation for the Levi±Civita connection, equation (19). Theorem If the metric is denoted by h ; i, then for a statistical manifold …M; g; T† XhY; Zi ˆ hrX Y; Zi ‡ hY; rXÿ Zi:

6.2

…24†

Projection theorem

In the following two sections we concentrate on the statistical manifold structure de®ned by Amari (1990). The geodesic distance in a manifold is one way to measure distances between points of the manifold. However, for probability distributions there are other more statistically natural ways of de®ning a form of distance. Consider the following de®nition: An -divergence between p…x† and q…x†, two density functions, is de®ned as ( Eq …log …q† ÿ log …p†† ˆ1 …25† D …p; q† ˆ Ep …log …p† ÿ log …q††  ˆ ÿ1:

Introduction to differential geometry

49

Note that these divergences are the Kullback±Leibler information from q to p when  ˆ 1 and from q to p when  ˆ ÿ1. These are not formally distance functions as they do not obey the standard axioms for a distance, such as symmetry or the triangle inequality. For a clari®cation of their geometric role, see Critchley, Marriott and Salmon (1994). Amari's projection theorem connects these divergence functions with the geodesic structure. We give a simpli®ed version of his theorem here, for exponential families. For more details see Amari (1990, p. 89). Theorem Let p…x j † be a full exponential family, and p…x j † a curved subfamily. Let 1 represent a distribution in the embedding family. Let  be an extremal point for the -divergence, that is, in ,  is an extremal point of the function D …p…x j 1 †; p…x j ††: If  is an  extremal point, then the  geodesic connecting p…x j 1 † and p…x j  † cuts the family p…x j † orthogonally. One natural application here is in the case of a misspeci®ed model. Suppose that the model is incorrectly speci®ed as being the curved exponential family p…x j †, while the data-generation process lies in the embedding family, and is denoted by p…x j 0 †. It is well known that the MLE on the misspeci®ed model will converge to the value that is closest to p…x j 0 † in terms of the Kullback±Leibler divergence. Hence it will be a ÿ1-extremal point and the DGP and the pseudo-true estimate are connected by a ÿ1-projection. 6.3

Pythagoras theorem

The ®nal result presented in this section brings to the geometry of statistical families the classical geometric theorem of Pythagoras. We state the theorem for completeness. Theorem Let 1 ; 2 and 3 be three points in the parameter space of a full exponential family. Suppose that the -geodesic joining 1 and 2 is orthogonal to the ÿ-geodesic joining 2 and 3 . For this triangle we have the Pythagorean relationship D …1 ; 2 † ‡ D …2 ; 3 † ˆ D …1 ; 3 †:

…26†

A more complete version of this theorem can be found in Amari (1990, p. 86).

50

Paul Marriott and Mark Salmon

7

Inference in curved families

In the ®nal two sections of this chapter we look at how the machinery previously set up enables us to understand both the problem of inference in curved families and how to assess the information content of a general statistic. This issue is taken up in several chapters in this volume, in particular by Kees Jan van Garderen and Tom Rothenberg. 7.1

Curvature due to parameterisations

We have seen how the connection structure of a manifold enables us to de®ne curvature, both of a curve and of a set of parameters. We look ®rst at the issue of parameter-dependent curvature. Example 7. Non-linear regression models Bates and Watts (1988, p. 241) distinguish between two types of parameterisation that are important in the analysis of non-linear regression models. These curvatures are called the parameter effects and intrinsic curvatures. The ®rst of these will be dependent on the exact form of the parameterisation and can be removed in principle by reparameterisation. The second will be independent of the parameterisation chosen. This distinction carries over from the non-linear regression case to a general statistical manifold. One example where parameter effects are important is the bias of the maximum likelihood estimate. In the case of a full exponential family, the ®rst-order bias of the maximum likelihood estimate is given by bi …† ˆ

ÿ1 ij kl ÿ1 g g ÿjkl : 2n

…27†

Consider this formula in the context of linear models. Example 1. The linear model (continued) There is a natural geometric interpretation for ordinary least squares (OLS) estimation, which connects Euclidean geometry and regression (see, for example, Bates and Watts (1988)). For presentational simplicity we consider the case where the variance parameter is ®xed and known. We have the model Y ˆ Xb ‡ r: There is an n-dimensional space of possible response vectors Y and a …k ‡ 1†-dimensional submanifold of this af®ne space is de®ned by the expectation surface b ! Xb  Rn :

Introduction to differential geometry

51

This is a …k ‡ 1†-dimensional af®ne subspace. When  is known, the geometry of the model is simply that of Euclidean geometry. In this model the natural parameters are given by þ and the expectation parameters by X 0 Xb : 2 Thus there is a ®xed linear relationship between the ‡1-af®ne and ÿ1af®ne parameters. Equation (16) implies that the Christoffel symbols for the r‡1 and rÿ1 connections will be simultaneously zero in both the natural and expectation parameterisations. Equation (27) then implies that estimating b using maximum likelihood estimation will have zero bias in both those parameterisations. However, in any non-linear reparameterisation of the model the Christoffel symbols ÿÿ1 need not be zero. A simple example of this is given by a simultaneous equation model, when we parameterise by the parameters in the structural equations. In general, choosing a parameterisation can improve statistical properties. This was recognised by Hougaard (1982), and similar results are found in Amari (1990). In section 6.1 we have seen that the Riemann curvature tensor can characterise the existence of a set of af®ne parameters. The following theorem formalises this idea. Theorem For a manifold M with metric g and connection r we have the following: (a) Assume that the Riemann curvature tensor vanishes identically for all points in a suf®ciently small open neighbourhood U. Then there exists an af®ne parameterisation  covering U such that the Christoffel symbols for r vanish identically. Further all r-geodesics in U are characterised by being af®ne functions of the -parameters. (b) In the general case, the Riemann curvature tensor will be non-zero. For any point p0 2 M there will be a parameterisation  around p0 such that the Christoffel symbols for r vanish at p. Further, the set of geodesics through p0 , in U, are all af®ne functions of the -parameters. The parameterisation in case (b) is often called the geodesic normal parameterisation. For details of this and the related exponential map, see Dodson and Poston (1991). When a manifold has an af®ne parameterisation, the geometry becomes that of an af®ne space. All geodesics are the one-dimensional af®ne subspaces. This is the simplest possible geometry. When the Riemann curvature is non-zero, the fundamental geometry of the manifold creates an obstruction to the existence of

52

Paul Marriott and Mark Salmon

such an af®ne parameterisation. In this case the geodesic normal parameterisation is, in a sense, closest to being truly af®ne. In the case where the connections come from the expected geometric statistical manifold, then af®ne parameterisations, when they exist, and geodesic normal parameters, when they do not exist, have useful statistical properties. The following list summarises the properties of some of the more important of these. Each of these is global property if the parameterisation is af®ne, otherwise they hold locally in the neighbourhood of a ®xed point. *

*

*

*

*

‡1-connection: The af®ne parameters for these are the natural parameters for the full exponential family. 1=3-connection: This is the quadratic log-likelihood parameterisation. In this parameterisation, the expectation of the third derivative of the log-likelihood ÿ
Ep…x j † @i @j @k ` vanishes. Thus the likelihood closely approximates that from a normal family. This is the expected form of the directed likelihood parameterisation, which has been shown to be a very effective tool in classical and Bayesian analysis (see Sweeting (1996) and references therein). 0-connection: This is the covariance stabilising parameterisation in which the Fisher information will be (locally) constant. ÿ1=3-connection: This is called the zero asymptotic skewness parameterisation, because any ®rst-order ef®cient estimator will have zero asymptotic skewness in this parameterisation. ÿ1-connection: We have already seen that in this parameterisation we have asymptotically minimum bias.

7.2

Intrinsic curvature

The distinction raised by Bates and Watts between parameter effects and intrinsic curvature is an important one in geometry. In this section we look at how intrinsic curvature is particularly important when undertaking inference on curved exponential families. Suppose that N is the manifold representing the …r; t†-curved exponential family p…x j †. N is embedded in the r-dimensional full exponential family, p…x j †, which is denoted by M. In terms of manifolds, N is an embedded submanifold of M (see section 2.4.1). In the natural parameters, M has many of the properties of an af®ne space, and intuitively it is useful to think of N as a curved subspace of an af®ne space. It is this curvature that will de®ne the intrinsic curvature of the model.

Introduction to differential geometry

53

We ®rst de®ne a set of parameters that is suited to describe the embedding of N in M. Theorem For each 0 2 N there exists an open neighbourhood U of M, such that p…x j 0 † 2 U and U can be parameterised by   X… Rt  Rrÿt † ! U  M …; † } p…x j …; †† where U \ N ˆ fp…x j …; 0†† j  2 g: The parameters for the full family are split such that the ®rst t components refer to the submanifold, while the remaining …r ÿ t† components ®ll out the rest of the space, in such a way that they are zero on the submanifold. For notational convenience we will denote the ®rst t components of such a system by indices i; j; k; . . ., while the remaining …r ÿ t† indices will be denoted by a; b; c; . . .. Hence, if we consider the tangent space to M at a point in N, denoted by p0 , in this space there will be directions that also lie in tangent space to N. In the above parameterisation these are spanned by the set f@i ˆ @@ i j i ˆ 1; . . . ; tg: There will remain those directions that are not tangent to N. The set of tangent vectors f@a ˆ @@ a j a ˆ 1; . . . ; …r ÿ t†g all have this property. We think of TNp0 as being a subspace of TMp0 . In fact a parameterisation …; † can be re®ned to have the property that   @ @ ; ˆ0 h@i ; @a iM ˆ @i @a M for all i ˆ 1; . . . ; t and for all a 2 1; . . . ; …r ÿ t†. Hence the splitting of the tangent space TMp0 can be made into TNp0 and its orthogonal component. It will be convenient to use such a parameterisation throughout this section. Suppose now that there is a metric and a connection on M. These will induce a metric and a connection on the submanifold N in the following way. Let gij be the components of the metric on M relative to the …; †parameterisation. By de®nition, the parameterisation has the property that gia ˆ 0 for all i ˆ 1; . . . ; t and for all a 2 1; . . . ; …r ÿ t†. Relative to the -parameterisation, de®ne the metric on N, g~ to have components

54

Paul Marriott and Mark Salmon

g~ij ˆ h@i ; @j iM :

…28†

Let X; Y be tangent ®elds on N. They can then also be thought of, slightly loosely, as tangent ®elds on M. Hence we can de®ne the rate of change of Y relative to X using the connection on M. This gives rX Y a tangent ®eld in M. However, a connection on N gives this rate of change as a tangent ®eld on the submanifold. Hence we need to project rX Y from TM to TN. We do this using the metric on M and an orthogonal ~ on N is de®ned by projection. The connection r r~ X Y ˆ N …rX Y †

…29†

where N is the orthogonal projection of TM to TN. The embedding curvature of N in M is de®ned as the difference between the result of using the connection relative to M and relative to N. Formally it is a 2-tensor H de®ned by ~ X Y: H…X; Y† ˆ rX Y ÿ r

…30†

H…X; Y† will lie in the orthogonal complement to TN. It can be written with respect to the parameterisation de®ned above as H…@i ; @j † ˆ ÿ~ aij @a ˆ Hija @a :

…31†

7.2.1

Auxiliary spaces Suppose that we now wish to conduct inference in a curved exponential family N. An estimator is a function that maps the data x to a point in N. It is convenient to use the fact that in the full exponential ^ family M, which embeds N, there is a bijection between x and …x†, the maximum likelihood estimate in M. We can therefore think of an estimator, geometrically, as a function T :M!N ^ } T……x††: ^ …x† We then study the properties of the estimator in terms of the geometry of this function. T is said to be regular if it is a smooth map with derivative of full rank and the restriction of T to the subspace, TjN , is the identity. De®ne the auxiliary space of the estimator to be the points in M given by the set of parameters A…† ˆ f j T…† ˆ g:

…32†

Introduction to differential geometry

55

We shall assume that for each point 0 2 N there exists an open subset U of M such that A…0 † \ M is an …r ÿ t†-dimensional submanifold of M. This assumption will be satis®ed by all of the commonest estimators used in econometrics and in practice it will not be restrictive. We shall use the geometry of these submanifolds together with that of N to explore the behaviour of estimators. A simple way of measuring the ef®ciency of an estimator is in terms of mean square error. We de®ne an estimator to be k'th-order ef®cient if the asymptotic expansion of its mean square error is minimal among all …k ÿ 1†'th ef®cient estimators. The following theorems, due to Amari, characterise this asymptotic ef®ciency. Theorem A consistent estimator is ®rst-order ef®cient if the associated auxiliary space A…† cuts N orthogonally with respect to the expected Fisher information metric.

to be

Theorem

ba …† ˆ

The ®rst-order bias of a consistent estimator is de®ned

o ÿ1 ij n kl ÿ1 g g ÿijk ‡ gab hÿ1 abj ; 2n

…33†

where h is the embedding curvature of the submanifold A…† with respect to the ÿ1-connection and is de®ned as ÿ1 hÿ1 abj ˆ hH …@a ; @b †; @j i:

Thus for a curved exponential family the bias term comes in two parts. The ®rst, as we have seen before, is parameterisation dependent; the second, however, depends only on the intrinsic geometry of the auxiliary space. This depends on the exact form of the estimator but not on the parameterisation use. Thus we have a decomposition into parameter effects and intrinsic curvature. Using this geometric characterisation, the third-order most ef®cient estimator for a curved exponential family can be found. Theorem The biased correct maximum likelihood estimator is third-order ef®cient for a curved exponential family. All the results shown in this section are proved in Amari (1990); for a good treatment, see Kass and Vos (1997, p. 227).

56

Paul Marriott and Mark Salmon

8

Curvature and information loss

Once we move from the full exponential family case we lose the property that the maximum likelihood estimate (MLE) will be a suf®cient statistic. Fisher (1925) was the ®rst to suggest using the observed information as the best means of recovering information lost by reducing the data to just the maximum likelihood estimate. There is an elegant geometric extension to higher-order recovery which provides a complete decomposition of the information in a minimal suf®cient statistic. The geometry provides a natural framework for expressing the higher-order calculations but also an explanation of why these terms successfully recover additional information. Having analysed the decomposition of the information in a statistic in geometric terms, there remains the question of how to use this information for inference. 8.1

Information loss

Since the MLE is a suf®cient statistic for a full exponential family, standard inferential procedures will automatically make use of all available information in the data. We look at geometrical properties that make a family exponential, and hence ensure that the full information content in the data is exploited. 8.1.1

One-dimensional case First consider a one-dimensional family as in Efron (1975). If we have an r-dimensional full exponential family de®ned by p…x j † ˆ expfi si …x† ÿ ý…†gm…x†; then any one-dimensional af®ne map }i …† ˆ i  ‡ þi will de®ne a one-dimensional full exponential family. In general we have the following result. Theorem

A one-dimensional curved exponential family

p…x j † ˆ expf…†si …x† ÿ ý……††gm…x†

…34† 1

is a full exponential family if and only if there exist constants … ; . . . ; r † and …þ1 ; . . . ; þr † such that …† ˆ i f …† ‡ þi for some smooth monotone function f .

Introduction to differential geometry

57

Hence we see that being a full exponential family depends on the family being de®ned as an af®ne subspace of the natural parameters of the embedding family. In section 5.2.1 we saw that the r‡1 connection is the appropriate tool for measuring curvature relative to the natural parameters. Efron (1975) approached this question more directly. He de®ned the statistical curvature of equation (34) to be ÿ ˆ ÿ…† ˆ

h 00 …†N ;  00 …†N i1=2 h 0 …†;  0 …†i

…35†

where  0 …† ˆ d=d and  00 …†N denotes the component of the second derivative which is normal to the tangent direction  0 …†. The inner product is the expected Fisher information metric. The motivation for this comes directly from the de®nition of the curvature of a one-dimensional path in Rn . The important result concerning statistical curvature is given by the following theorem. Theorem The curved exponential family given by equation (34) is a one-parameter full exponential family if and only if its statistical curvature is zero. One important property of the full exponential family is that the observed information equals the expected Fisher information. They ^ in the natural parameterisation. The ranboth equal ‰…@2 ý†=…@i @j †Š…† domness of the observed information is purely a function of the MLE. In fact Murray and Rice (1993) characterise a full exponential family by the property that the second derivative of the log-likelihood lies in the span of the ®rst derivatives. In a curved family, the expected and observed information matrices will differ. The statistical curvature, de®ned above, gives a useful measure of the amount of variability in the observed information, given the MLE. This comes from the following result (see Efron and Hinkley (1978)). ^ is the Theorem In a regular curved exponential family, if I …† ^ observed Fisher information and I…† the expected information for one observation, then ^ ÿ nI…†† ^ …I …† p ^ ^ ! N…0; 1† nI…†ÿ…† where the convergence is in law as n ! 1.

…36†

58

Paul Marriott and Mark Salmon

8.1.2

The general case Suppose now that we are working in an …r; t†-curved exponential family. Again we wish to understand the amount of information lost if we ^ To do this we need to de®ne the information use only the statistic . contained in a general statistic T…x†. Using an asymptotic measure of information, the information in a statistic T is de®ned as being the expected Fisher information for the likelihood for  based on the statistic T…x†. That is ý T

I …† ˆ Ep…t j †

! @2 ` ÿ i j …t ; † : @ @

…37†

This measures, by the Cramer±Rao theorem, the variance of the best possible estimator based only on the statistic T…x†. Of course, with this notation we would have I X …† ˆ I…†, the standard Fisher information. We can generalise the previous results on information loss to multidimensional families. Theorem

In a curved exponential family p…x j † then

^

nIij ÿ Iij ˆ gkl hH ‡1 …@i ; @k †; H ‡1 …@j ; @l †i;

…38†

where H ‡1 is the embedding curvature of the family relative to the ‡1connection, and the inner product is relative to the expected Fisher metric. For a proof of this result, see Kass and Vos (1997, p. 222). Thus the ‡1embedding curvature plays the role of Efron's statistical curvature in higher dimensions.

8.2

Information recovery

Having seen that the geometric structure of a family allows the measurement of loss in information, we now ask in what way this information ^ will not in can be recovered and used in inference. Since the MLE, , general be suf®cient, we need to add a further statistic that will recover (approximate) suf®ciency. It is convenient to construct the conditional ^ a†, resolution. This is a decomposition of the suf®cient statistic into …; where a is (approximately) ancillary, combined with an (approximate) expression for p…^ j ; a†. For more details of this construction, see Barndorff-Nielsen and Cox (1994).

Introduction to differential geometry

59

8.2.1

Choosing ancillaries One dif®culty with using a conditional resolution structure is the problem of the non-uniqueness of the ancillary statistic, raised by Basu (1964). Solutions to this problem were considered by Barnard and Sprott (1971) and Cox (1971). We look at a geometrical construction de®ned in Kass and Vos (1997, p. 222), which gives a way of ensuring that the ancillary constructed captures information to progressively higher asymptotic order. First we use geometry to construct approximately suf®cient statistics for a curved exponential family. These are then transformed into approximately ancillary statistics. Consider the …r; t†-curved exponential family N given by expfi …†si …x† ÿ ý……††gm…x†. Clearly the statistic …s1 …x†; . . . ; sr …x†† is suf®cient, but this can be very high dimensional. In fact, as Examples 3, 4 and 5 show, in econometric examples this suf®cient statistic can be of order n. It is therefore natural to see if we can reduce the dimension of the suf®cient statistic. Using the de®nition of tangent vector and the af®ne embedding given in section 3.2.1, the best local af®ne approximation to N at 0 will be given by the t-dimensional full exponential family, M‰1Š , p…x j …1 ; . . . ; t †† ˆ expf…i …0 † ‡ j @j i …0 ††si …x† ÿ ý……0 † ‡ j @j …0 ††gm…x† ˆ expfj …@j i …0 †si …x†† ÿ ý…†gm1 …x† ˆ expfj s~j ÿ ý…†gm1 …x†; say, for a measure m1 …x†. The natural parameters are given by …1 ; . . . ; t † 0 . It follows from the theorems in section 8.1.1 that this will ^ be a full exponential family. Its suf®cient statistic will be equivalent to , the MLE for the curved family. This is de®ned by the equations   @i ^i @ý @i ^i @ý ^ ~ ˆ s …  † s ÿ ÿ … † i …† i i @j @ j @i @ ˆ 0: Owing to the regularity conditions of section 2.3, this equation will be invertible. Hence the information in ^ is precisely equivalent to …~s1 ; . . . ; s~t † 0 . In general we can de®ne a sequence of full exponential families M‰1Š ; M‰2Š ; . . . ; M‰kŠ where M‰kŠ is the family

60

Paul Marriott and Mark Salmon

p…x j i ; ij ; . . . ; i1 ...ik † ˆ expf…i …0 † ‡ j @j i …0 † ‡ j1 j2 @2j1 j2 i …0 † ‡


‡ j1


jk @kj1


jk i …0 ††si …x† ÿ ý……††gm…x†: The suf®cient statistic for this full exponential family is given by ^ @i i `…†; ^ . . . ; @ki ...i `…††: ^ …; 1 2 1 k We therefore have constructed a sequence of full exponential families, which give progressively better approximations to the curved family N. These in turn give a sequence of statistics, which become progressively better approximations to the suf®cient statistic for N. It is necessary to check that this construction is geometrically well de®ned, so that the suf®cient statistics generated will be independent of any choice of parameterisation. This follows since the linear spaces given by T‰kŠ ˆ spanf@i ; @2i1 i2 ; . . . ; @ki1


ik g will be parameterisation independent (see Murray and Rice (1993)). ^ @i i `…†; ^ . . . ; @ki ...i `…†† ^ can then The sequence of suf®cient statistics …; 1 2 1 k be transformed to be approximately in the conditional resolution form ^ a†. De®ne …; k‡1 hi1


ik‡1 ˆ P? ‰kŠ @i1


ik‡1 ;

where P? ‰kŠ is the orthogonal projection into T‰kŠ . This orthogonalisation ensures that the terms are uncorrelated and asymptotically independent; further, the expected Fisher information based on hi1


;ik‡1 will be of asymptotic order nÿk‡1 . To achieve approximate ancillarity these statistics are adjusted to give zero mean and unit variance to the correct asymptotic order. For further details, see Kass and Vos (1997). Note that the terms hi1


ik are simply generalisations of the embedding curvature in section 7.2. The p -formula Having used geometry in the construction of an approximately ^ a†; the second part of a conditional suf®cient statistic of the form …; resolution is to approximate the distribution p…^ j; a†. BarndorffNielsen, in a series of papers, proposes a very good higher-order approximation, based on the saddlepoint method given by the so-called p -formula. For a derivation of results in this section, see Barndorff-Nielsen and Cox (1994, p. 238). The p -approximation is given by 8.2.2

Introduction to differential geometry 

p …^ j ; a† ˆ cj |^ j1=2 e` ;

61

…39†

where |^ ˆ

@2 ` ^ …† @i @j

and ^ a† ˆ `… j ; ^ a† ÿ `…^ j ; ^ a†:  j ; `… The constant c is de®ned to make the above density integrate to one. When this is not known, the approximation  py …^ j; a† ˆ …2†ÿt=2 j |^j1=2 e`

…40†

can be used. This version is accurate to order O…nÿ1 †, whereas the p formula is accurate to order O…nÿ3=2 †: For a further discussion of this issue of the distribution of the maximum likelihood estimator, see chapter 3 by Grant Hillier and Ray O'Brien in this volume.

References Amari, S.I. (1987), `Differential Geometric Theory of Statistics', in S.I. Amari, O.E. Barndorff-Nielsen, R.E. Kass, S.L. Lauritzen and C.R. Rao (eds.), Differential Geometry in Statistical Inference, Hayward, Calif.: Institute of Mathematical Statistics. (1990), Differential-Geometrical Methods in Statistics, 2nd edn, Lecture Notes in Statistics No. 28, Berlin: Springer. Barnard, G.A. and D.A. Sprott (1971), `A Note on Basu's Examples of Anomalous Ancillary Statistics (with Discussion)', in V.P. Godambe and D.A. Sprott (eds.), Foundations of Statistical Inference, Toronto: Holt, Rinehart & Winston, pp. 163±176. Barndorff-Nielsen, O.E. (1978), Information and Exponential Families in Statistical Theory, London: Wiley. (1987), `Differential Geometry and Statistics: Some Mathematical Aspects', Indian Journal of Mathematics, 29, Ramanujan Centenary Volume. (1988), Parametric Statistical Families and Likelihood, New York: Springer. Barndorff-Nielsen O.E. and D.R. Cox (1994), Inference and Asymptotics, Monographs on Statistics and Applied Probability 52, London: Chapman & Hall. Barndorff-Nielsen, O.E. and P. Jupp (1989), `Approximating Exponential Models', Annals of Statistical Mathematics, 41: 247±267. Basu, D. (1964), `Recovery of Ancillary Information', Sankya A, 26: 3±16.

62

Paul Marriott and Mark Salmon

Bates, M. and D.G. Watts (1988), Nonlinear Regression Analysis and Its Applications, London: Wiley. Blñsild, P. (1987), `Yokes: Elemental Properties with Statistical Applications', in C.T.J. Dodson (ed.), Geometrization of Statistical Theory, Proceedings for the GST Workshop, University of Lancaster, ULDM Publications, University of Lancaster, pp. 193±198. (1991), `Yokes and Tensors Derived from Yokes', Annals of the Institute of Statistical Mathematics, 43: 95±113. BroÈcker, T.H. and JaÈnich, K. (1982), Introduction to Differential Geometry, Cambridge: Cambridge University Press. Brown, L.D. (1986), Fundamentals of Statistical Exponential Families with Applications in Statistical Decision Theory, Haywood, Calif.: Institute of Mathematical Statistics. Cox, D.R. (1971), `The Choice between Alternative Ancillary Statistics', Journal of the Royal Statistical Society C, 33: 251±255. Cox, D.R., and D.V. Hinkley (1974), Theoretical Statistics, London: Chapman & Hall. Critchley, F., P.K. Marriott and M. Salmon (1993), `Preferred Point Geometry and Statistical Manifolds', Annals of Statistics, 21: 1197±1224. (1994), `On the Local Differential Geometry of the Kullback±Leibler Divergence', Annals of Statistics, 22: 1587±1602. (1996), `On the Differential Geometry of the Wald Test with Nonlinear Restrictions', Econometrica, 64: 1213±1222. Dodson, C.T. and T. Poston (1991), Tensor Geometry: The Geometric Viewpoint and Its Uses, 2nd edn, New York: Springer Verlag. Efron, B. (1975), `De®ning the Curvature of a Statistical Problem (with Applications to Second Order Ef®ciency)', Annals of Statistics, 3: 1189±1217. (1978), `The Geometry of Exponential Familes', Annals of Statistics, 6: 362±376. (1982), `Maximum Likelihood and Decision Theory', Annals of Statistics, 10: 340±356. Efron, B. and D.V. Hinkley (1978), `Assessing the Accuracy of the Maximum Likelihood Estimator: Observed versus Expected Fisher Information', Biometrika, 65: 457±481. Fisher, R.A. (1925), `Theory of Statistical Estimation', Proceedings of Cambridge Philosophical Society, 22: 700±725. Hougaard, P. (1982), `Parametrisations of Nonlinear Models', Journal of the Royal Statistical Society, Series B Methodological, 44: 244±252. Kass, R.E. and P.W. Vos (1997), Geometrical Foundations of Asymptotic Inference, Probability and Statistics series, New York: Wiley. Lauritzen, S.L. (1987), `Statistical Manifolds', in S.I. Amari, O.E. BarndorffNielsen, R.E. Kass, S.L. Lauritzen and C.R. Rao (eds.), Differential Geometry in Statistical Inference, Hayward, Calif.: Institute of Mathematical Statistics, pp. 163±216.

Introduction to differential geometry

63

McCullagh, P. (1987), Tensor Methods in Statistics, Monographs on Statistics and Applied Probability No. 29, London: Chapman & Hall Marriott, P.K. (1989), `Applications of Differential Geometry to Statistics', Ph.D. dissertation, University of Warwick. Murray, M.K. and J.W. Rice (1993), Differential Geometry and Statistics, Monographs on Statistics and Applied Probability No. 48, London: Chapman & Hall. Rao, C.R. (1945), `Asymptotic Ef®ciency and Limiting Information', Proceedings of the Fourth Berkeley Symp. Math. Statist. Probab., 1: 531±545. (1987), `Differential Metrics in Probability Spaces', in S.I. Amari, O.E. Barndorff-Nielsen, R.E. Kass, S.L. Lauritzen and C.R. Rao (eds.), Differential Geometry in Statistical Inference, Hayward, Calif.: Institute of Mathematical Statistics, pp. 217±240. Ravishanker, N. (1994), `Relative Curvature Measure of Nonlinearity for TimeSeries Models', Communications in Statistics±Simulation and Computation, 23(2): 415±430. Rudin, W. (1976), Principles of Mathematical Analysis, 3rd edn, New York: McGraw-Hill. Spivak, M. (1979), A Comprehensive Introduction to Differential Geometry, 2nd edn, Boston: Publish and Perish. Sweeting, T. (1996), `Approximate Bayesian Computation Based on Signed Roots of Log-density Ratios (with Discussion)', in Bayesian Statistics 5 (ed. J.M. Bernardo, J.O. Berger, A.P. Dawid and A.F.M. Smith), Oxford: Oxford University Press, pp. 427±444. Willmore, T.J. (1959), An Introduction to Differential Geometry, Oxford: Clarendon.

2

Nested models, orthogonal projection and encompassing Maozu Lu and Grayham E. Mizon

1

Introduction

An important objective in econometric modelling is to obtain congruent and encompassing models, which are thus consistent with data evidence and economic theory, and are capable of explaining the behaviour of rival models for the same economic phenomena. Hendry (1995) and Mizon (1995b) inter alia present this view of modelling, as well as recording that a general-to-simple modelling strategy is an ef®cient way to isolate congruent models. Indeed such a modelling strategy requires the models under consideration to be thoroughly tested and progressively evaluated within a sequence of nested hypotheses, so that each model not rejected in the process is a valid simpli®cation of (i.e. parsimoniously encompasses) all models more general than it in the sequence. Models arrived at as the limit of the reduction process therefore contain all relevant information available in the initial models, thus rendering the latter inferentially redundant. Testing the ability of models to parsimoniously encompass more general ones, thus ensuring that no information is lost in the reduction process, creates a partially ordered sequence of models. Further, Lu and Mizon (1997) point out that the intermediate models considered while testing the parsimonious encompassing ability of models in a nested sequence are mutually encompassing models and hence observationally equivalent to each other. Note, though, that there is an important distinction between observational equivalence in the population and in a sample, which may contain weak evidence. In the theory of reduction, attention is focused on parsimonious encompassing, and not on the ability of nesting models to encompass Financial support from the UK Economic and Social Research Council, under grant R000233447, and from the EUI Research Council grant Econometric Modelling of Economic Time Series, is gratefully acknowledged. We also thank Sùren Johansen for helpful discussions on the topic.

64

Orthogonal projection and encompassing

65

simpli®cations of themselves. Though the latter might be thought to be automatic, GourieÂroux and Monfort (1995) give a counter-example, thus showing that it is not always the case that a general model encompasses models simpler than itself. Clearly, if this happens in a process of reduction, the sequence of parsimonious encompassing models will not form a partial order, and some vital information might be lost if the intermediate models were discarded. It is therefore important to investigate the conditions under which a general model encompasses simpler models independently of its relationship to the data-generation process (DGP). This is the topic of the present chapter. Bontemps and Mizon (1997), on the other hand, establish that congruence of a general model is a suf®cient condition for it to encompass models that it nests. The chapter is organised as follows. In section 2 nesting is de®ned, it being emphasised that nesting is more than one model being an algebraic simpli®cation of another model. A distinction is drawn between exact and inequality constraints on parameters, the former being dimension reducing and the latter not. The importance of dimension-reducing nesting constraints is illustrated by showing that the nesting model in cases where there is an orthogonal parameterisation will encompass models nested within it independently of the nature of the process generating the data. This property is de®ned as automatic encompassing. Section 2.3 records that automatic encompassing is not possible amongst non-nested models. Section 3 indicates how the concept of parsimonious encompassing can be used to establish encompassing relationships between non-nested models within the framework of a completing model that nests each of the non-nested models. In section 4 encompassing hypotheses are considered as mappings in Hilbert space, and it is shown that a suf®cient condition for a model to encompass models nested within it is that these mappings are orthogonal projections. Section 5 presents concluding remarks.

2

Nested and restricted models

A parametric probability model for the n random variables …x1;t ; x2;t ; . . . ; xn;t † ˆ xt0 de®ned on the probability space …S; =; P…:†† consists of a family of sequential densities indexed by a parameter vector h: M1 ˆ f f …xt j Xtÿ1 ; h†;

h 2 ?  Rp g …X0 ; X1tÿ1 †,

…1†

with X0 being initial conwhen Xtÿ1 ˆ …X0 ; x1 ; . . . ; xtÿ1 † ˆ ditions. Denoting this probability model by M1 and considering the alternative probability model M2 :

66

Maozu Lu and Grayham E. Mizon

M2 ˆ fg…xt j Xtÿ1 ; w†;

w 2 )  Rq g

…2†

enables nesting to be de®ned. De®nition 1

M2 is nested within M1 if and only if M2  M1 :

Hence nesting is a property of the joint density of all the variables involved in M1 and M2 ; and heuristically requires M2 to be a restricted version of M1 . However, there are important differences between nested and restricted models. In particular, note that if ? were restricted (e.g. the model was restricted to have only seasonal dynamics) but ) were not so restricted, then M2 cannot be nested in M1 since M2 could not be a subset of M1 : Further, the nature of restrictions applied to models can affect the nesting and encompassing relationships between models. This is illustrated in section 2.1 for the important cases of equality and inequality restrictions. There exist alternative de®nitions of nested models in the literature, one of which, based on an information criterion, was proposed by Pesaran (1987): De®nition 2 For the models M1 and M2 de®ned in (1) and (2) M2 is nested within M1 if and only if for 8 w 2 ), 9h 2 ?, such that …

  g…w † log g…w † dy ˆ 0 f …h †

…3†

where f …h † ˆ f …xt j Xtÿ1 ; h † and g…w † ˆ g…xt j Xtÿ1 ; w †: Despite their apparent differences, these de®nitions are essentially the same. In particular, it is shown in A.1 of the Appendix that, for any given w 2 ); h is the pseudo true value h…w † that solves the minimisation problem:     … g…w † g…w †  g…w † dy: g…w † dy ˆ log min log h2? f … h† f …h…w †† …

It is further shown in A.2 that, if M2 is nested in M1 in the sense of (3), the parameter space ) is a subset of ? with probability one. It then follows that, although the concept of nesting de®ned by (3) employs a measure of distance between densities, it is closely concerned with parameter restrictions leading to )  ?:

Orthogonal projection and encompassing

2.1

67

Nested models with exact restrictions

A special case of De®nition 1, which is commonly used in the discussion of nesting, arises when f …xt j Xtÿ1 ; h† and g…xt j Xtÿ1 ; w† belong to the same family of densities but )  ? with q < p; so that M2  M1 solely as a result of restrictions on ? de®ning the restricted parameter space ): In this latter case, when ? is an unrestricted parameter space and the constraints on ? that lead to ) are exact or equalities, a reparameterisation of M1 is possible that makes explicit the relationship between M1 and M2 : Proposition 1 In the context of (1) and (2) let /…h† ˆ 0 be the r ˆ p ÿ q > 0 exact, continuous, and
differentiable constraint equations on ÿ h which de®ne w; with rank @/=@h 0 N ˆ r when Nh0 is a neighbourhood of h0 the pseudo-true value h0 of the maximum likelihood (ML) estimator h^ T of there
exists ÿh. Then ÿ an isomorphic
0 reparameterisation k ofÿh : k ˆ
k…h† ˆ 0 k1 …h† 0 ; k2 …h† 0 ˆ k1 …h† 0 ; /…h† 0 8 h 2 Nh0 such that rank @=@h 0 N ˆ p h0 and …k1 ; k2 † 2 ,1  ,2 . (See Sargan (1988, p. 143) for discussion of such a reparameterisation.) Note that the r constraint equations /…h† ˆ 0 are dimension reducing, in that when applied they reduce the dimension of parameter space from dim ? ˆ p to dim ) ˆ q ˆ p ÿ r: Further note that k2 …h† is equal to the parameters of the constraint equations, and although k1 …h† 6ˆ w in general, when the constraints are satis®ed k1 …h† ˆ w 8 h 2 ?0 ˆ fh j h 2 ? such that /…h† ˆ 0g. Hence ) ˆ fw j w ˆ k1 …h† such that /…h† ˆ 0 8 h 2 ?0 g: Another important property of the reparameterisation is the fact that k1 …h† and k2 …h† are variation free. These properties of the reparameterisation do not ensure that M1 E M2 : A special class of model that does ensure that M1 E M2 is that in which the parameters k1 and k2 are orthogonal. An important feature of this special case is that the expected information matrix for the parameters k1 …h† and k2 …h† (and hence /…h†) is block diagonal, so that the score vectors with respect to k1 and k2 are asymptotically uncorrelated. Thus, if /…h† were set to zero, the ML estimator of k1 will be unaffected asymptotically. In particular, when there are T observations on xt available, the log-likelihood function for M1 is: L1 …h† ˆ

ˆ

T X tˆ1 T X tˆ1

log f …xt j Xtÿ1 ; h†

log f …xt j Xtÿ1 ; k1 ; /† ˆ L1 …k1 ; /†;

…4†

68

Maozu Lu and Grayham E. Mizon

and when M2  M1 as a result of the exact restrictions /…h† ˆ 0 the loglikelihood function for M2 is: L2 …w† ˆ

T X tˆ1

log f …xt j Xtÿ1 ; w; 0†:

The parameters k1 …h† and /…h† are globally orthogonal when: " # @2 L1 …k1 ; /† EDGP ˆ0 8h2? @k1 @/ 0

…5†

…6†

and locally orthogonal at h0 when (6) holds for h ˆ h0 ± see inter alia Cox and Reid (1987) and Barndorff-Nielsen and Cox (1994), who also point out that parameter orthogonality is achievable in the regular exponential family of densities. In this special case Proposition 1 has the following corollary. Corollary 2 Under the conditions of Proposition 1 the condition that k1 …h† and k2 …h† are locally orthogonal at h0 is suf®cient for M1 E M2 …w† independently of the data-generation process (DGP), i.e. w0 ˆ k1 …h0 † ˆ w…h0 † 8 DGPs when w0 is the pseudo-true value of the maximum likelihood (ML) estimator w^ T of w. As an example, consider the ÿ
following two nested regression models …M2  M1 † when xt0 ˆ yt ; zt0 with n ˆ 3; and the validity of conditioning on zt is assumed: ÿ
t j zt  N 0;  2 M1 : yt ˆ z1;t þ1 ‡ z2;t þ2 ‡ t …7† ÿ
M2 : yt ˆ z1;t  ‡ vt vt j zt  N 0;  2 : Note that model M2 may be obtained by setting þ2 ˆ 0 in model M1 ; so that  ˆ þ1 and  2 ˆ  2 when the restriction þ2 ˆ 0 is valid. However, it is also possible to reparameterise M1 as: ÿ
…8† t j zt  N 0;  2 M1 : yt ˆ z1;t  ‡ z2;t þ2 ‡ t PT 2 PT  when z2;t ˆ …z2;t ÿ z1;t sˆ1 z1;s z2;s = sˆ1 z1;s † is the projection of z2;t onto the space orthogonal to z1;t so that the relationship between M1 and M2 is explicit, in that they both involve ; independently of the restriction þ2 ˆ 0: This reparameterisation leaves  and þ2 variation free, and it is valid independently of the DGP. Hence M1 encompasses M2 with respect to  …M1 E M2 …†† independently of the DGP (see Mizon and Richard (1986)). Heuristically this means that anything that can be learned about  from M2 ; can be obtained equally from M1 . In particular, note

Orthogonal projection and encompassing

69

P P that the ML estimator of  in M2 is ^T ˆ Ttˆ1 z1;t yt = Ttˆ1 z21;t , which has a pseudo-true value of 0 under the DGP. Further, using (8) yields: " #," # T T X X  2 ^T ˆ z1;t …z1;t  ‡ z2;t þ2 ‡ t † z1;t tˆ1

" ˆ‡

tˆ1

T X tˆ1

ˆ‡

T X tˆ1

z1;t z2;t þ2

z1;t t =

T X tˆ1

‡

T X tˆ1

z1;t t

#," T X tˆ1

# z21;t

z21;t

so that ^T has a probability limit under M1 given by: p lim ^T M1 ˆ ˆ 0 : T!1

The latter equality holds since the ML estimator of  in M1 is identical to that for  in M2 as a result of z1;t and z2;t being orthogonal. This example and Corollary 2 serve to indicate the value in econometric and statistical modelling of adopting orthogonal parameterisations, a point made by Barndorff-Nielsen and Cox (1994, section 3.6) and Hendry (1995, p. 552). Section 4 further illustrates the importance of orthogonality by interpreting encompassing as involving mappings in Hilbert space and proving that a suf®cient condition for automatic encompassing (M1 E M2 …w†) is that the mappings be orthogonal projections. Although the nesting of M2 in M1 in the above example (7) was determined by parameter restrictions, the form of the densities f …xt j Xtÿ1 ; h† and g…xt j Xtÿ1 ; w† can also affect the nestedness of models. For example, the models in (7) are still nested when the normality of t and vt is replaced by the assumption that they are i.i.d. with zero means and ÿ
®nite variances. However, they are non-nested when t j zt  N 0;  2 and vt j zt  U…ÿh; ‡h† since then (2 is no longer a subset of (1 . On the other hand, the following conditional models: ÿ
M3 : yt ˆ þzt ‡ t t j zt ; Xtÿ1  i.i.d. 0;  2 ÿ
M4 : yt ˆ ÿytÿ1 ‡ t t j zt ; Xtÿ1  i.i.d. 0;  2 are non-nested when t and t are i.i.d., but if t is serially dependent the models might be nested. For example, if t were speci®ed as being generated by the stationary AR…1† process t ˆ tÿ1 ‡ t (j  j< 1), then M3 can be rearranged as: M3 :

yt ˆ þzt ‡ ytÿ1 ÿ þztÿ1 ‡ t ;

70

Maozu Lu and Grayham E. Mizon

so that the restriction þ ˆ 0 applied to M3 yields M4 with  ˆ ÿ and  2 ˆ  2 when the common factor restriction involved in the AR…1† error process is valid, in which case these models are nested (see Hendry and Mizon (1978), Mizon (1995a) and Sargan (1980)). However, M3 has a restricted parameter space in that though it has 3 regression coef®cients they involve only 2 parameters þ and ; and as a result M3 does not nest M4 in general. Indeed, only if the DGP belonged to M3 would the latter nest M4 : Note that in this example, and that developed in Mizon (1995a), a model that appears to be an algebraic simpli®cation of another (e.g., the restriction þ ˆ 0 renders M3 algebraically the same as M4 † is neither nested nor encompassed by the more general model as a result of an auxiliary restriction on the parameter space of the general model (the common factor restriction) not being a feature of the parameter space of the simple model. Thus the requirement that ? be unrestricted, or if restricted the same restrictions apply to ); is critical for M1 to nest M2 ; and hence for an apparently more general model to encompass its algebraic simpli®cation.

2.2

Nested models with inequality restrictions

In addition to the requirement that ? be unrestricted in order for the reparameterisation presented in Proposition 1 to hold, it is also necessary that the constraint equations be exact (and hence dimension reducing), and not inequality restrictions as the following example illustrates. Let M1 be an inequality restricted version of M1 from (7): M1 : yt ˆ z1;t þ1 ‡ z2;t þ2 ‡ ut s:t:

0  þ2  þ1 ;

if þ1  0

0  þ2  jþ1 j

if þ1  0

…9†

then M1  M1 ; but since the constraints are inequalities they are not dimension reducing and Proposition 1 does not apply. Indeed, for the generic nested models M1 and M2 ; whenever M2  M1 as a result of restrictions on the parameter space, but the constraints de®ning the nested model are inequalities (e.g. /…h† > 0 8 h 2 ?†; neither Proposition 1 nor its Corollary 2 apply. As a result, the nesting model M1 will not in general encompass the nested model M2 , though if the DGP lies in M1 it will. Hence the nature of the restrictions relating the nested model M2 to the nesting model M1 can determine whether M1 E M2 independently of the DGP.

Orthogonal projection and encompassing

2.3

71

Non-nested models

Now consider the relationship of M1 in (9) to M2 in (7). Even though M2 appears as an algebraic simpli®cation of M1 it is not nested in M1 since it is not subject to the inequality restrictions in (9): Further, although M1 and M2 are each nested in M1 of (7), and model M1 in (7) encompasses M2 independently of the DGP, M1 does not encompass M2 unless the DGP lies in M1 of (9) or the inequality restrictions are not binding. Indeed, if yt were generated by a process not contained in M1 ; it can be shown that, as long as the projection of yt on the space spanned by z1t and z2t is in the interior of the area of 0  þ2  jþ1 j, M1 will encompass M2 irrespective of the actual form of the DGP. A more revealing case arises when the minimum distance from yt to M1 , generated by a DGP h…xt j Xtÿ1 ; 0 † other than M1 or M2 ; lies in the parameter subspace in which þ1 ˆ þ2 . Given this relationship between M1 and the unknown data-generation process, the apparently more general model M1 may not encompass the simple model M2 : Within the parameter subspace the restriction þ1 ˆ þ2 holds, so model M1 in (7) becomes: M1y : yt ˆ þ1 …z1t ‡ z2t † ‡ wt

…10†

ˆ þ1 zyt ‡ wt where zyt ˆ z1t ‡ z2t : Model M1y encompasses M2 in (7) if and only if: …0 † ÿ …b…0 †† ˆ 0; that is: ü plim

T X

T!1

tˆ1

!ÿ1 z21t

plim

T X

T!1 tˆ1

8 > > > > <

9 T X y > > z t yt > > = tˆ1

ˆ0 z1t yt ÿ zyt T > X y2 > > > > > > > zt ; :

…11†

tˆ1

(see Mizon and Richard (1986)). The condition in (11) requires that the residuals T X

yt ÿ zyt

zys ys

sˆ1 T X

zy2 s

sˆ1

;

t ˆ 1; 2; . . . ; T

…12†

72

Maozu Lu and Grayham E. Mizon

Figure 2.1 …0 † 6ˆ …þ…0 ††

be orthogonal to z1t , and it is shown in A.3 of the Appendix that this is equivalent to: ( ) T T T X X X y y y2 …13† plim z2t yt ÿ zt z s ys zs ˆ 0: T!1 tˆ1

sˆ1

sˆ1

Hence, the condition (11) requires that the residuals (12) be orthogonal to the space S …þ† spanned by z1t and z2t . However, M1 will encompass only M2 when the orthogonal projection of yt onto S …þ† lies in the subspace satisfying the triangular restriction in (9). This is illustrated by the graph in ®gure 2.1 (see also ®gure 1 in GourieÂroux and Monfort (1995)). The analysis of section 2 can be summarised in the following proposition. Proposition 3 In the context of the generic models M1 and M2 of (1) and (2) respectively and De®nition 2: (i) if M2  M1 with /…h† ˆ 0 being the constraints on h de®ning the lower dimensional parameters w then M1 E M2 …w† for all DGPs whenever w and / are orthogonal; (ii) if M2  M1 with /…h† < 0 being the constraints on h de®ning the nested model M2 then M1 E M2 …w† if the DGP belongs to M1 ; (iii) if M1 and M2 are non-nested then M1 E M2 …w† if the DGP belongs to M1 and conversely M2 E M1 …w† if the DGP belongs to M2 : Hence nesting and the nature of the restrictions de®ning the nested relationship, as well as the class of density used, are critical in determin-

Orthogonal projection and encompassing

73

ing whether a general model will automatically encompass a simple model. An implication of this proposition is that it is important to establish carefully that models thought to be nested are indeed nested. The fact that a restriction placed on the parameters of an apparently more general model yields a simple model does not imply that the general model nests the simple model. Further, since the presence of auxiliary restrictions on the parameter space of a general model will generally result in that model not nesting an algebraic simpli®cation of it, it is important to test the validity of such auxiliary restrictions, and to proceed to a comparison of the general and simple models only if they are not rejected. Thus the congruence of general models is important whenever a general-to-simple modelling strategy is adopted.

3

Encompassing and parsimonious encompassing

In this section the role of nested models in encompassing is discussed further, and it is shown that when a completing model exists for non-nested models it is possible to use the concept of parsimonious encompassing to establish encompassing relationships amongst the non-nested models. Although general conditions for encompassing amongst non-nested models are dif®cult to establish, and will usually involve the unknown DGP, the concept of parsimonious encompassing can be used to yield encompassing relationships amongst non-nested models that are nested within the completing model. For example, M1 in (9), M1y in (10) and M2 in (7) are all non-nested, but they are each nested within M1 in (7): Hence if any of the three non-nested models is a valid reduction of M1 (i.e. parsimoniously encompasses M1 † it contains as much relevant information as M1 ; and hence at least as much as each of the other non-nested models. In other words, when any of M1 , M1y and M2 parsimoniously encompasses M1 it will also encompass the other two non-nested models. This result can now be stated more generally for generic non-nested models M1 and M2 as follows. De®nition 3 M2 parsimoniously encompasses Mc (denoted M2 E p Mc ) when M2 E Mc and M2  Mc with Mc ˆ M2 [ M1? being the completing model which contains M2 and all aspects of M1 that are not already in M2 : (See Hendry and Richard (1989) and Hendry (1995).) This de®nition and the following theorem provide a framework for establishing encompassing relationships between non-nested models.

74

Maozu Lu and Grayham E. Mizon

Theorem 4 If M1  Mc and M2  Mc then M2 E M1 if and only if M2 E p Mc : (See Hendry and Richard (1989) and Hendry (1995, pp. 511±512).) Hence it is possible to compare alternative non-nested models via their encompassing properties within the framework of an appropriate choice of completing model. For example, since each of M1 in (9), M1y in (10) and M2 in (7) is nested within M1 , if any of them parsimoniously encompasses M1 in the population, that model will encompass the other two models. If more than one of these models parsimoniously encompasses M1 then they are mutually encompassing and hence observationally equivalent ± see Lu and Mizon (1997) for more details.

4

Encompassing and orthogonal projection

In this section a generalisation of the results in section 2 is presented which interprets encompassing as orthogonal projections in Hilbert space. In particular, a more general suf®cient condition for a model to encompass models nested within it, when the DGP need not be parametric, is given. 4.1

Mapping and orthogonal projection

Let a Hilbert space RT , of which the random vectors y1 and y2 are elements, be de®ned as a T-dimensional complete linear space equipped with the scalar product: ü ! T X ý  y1 ; y2 ˆ E 8 y 1 ; y2 2 RT y1;j y2;j jˆ1

and the norm of any element y 2 RT is given by: ü !1=2 T X ÿý 
1=2 2 ˆ E yj ; k y k ˆ y; y jˆ1

where `E' is expectation taken under the speci®cation of y. Two T  1 random vectors y1 and y2 2 RT are equal if and only if: k y1 ÿ y2 k ˆ 0; which requires y1 and y2 be equivalent with probability one. With these de®nitions and notation it is possible to discuss encompassing in a Hilbert space generated by T  1 random vectors y ˆ

Orthogonal projection and encompassing

75

…y1 ; . . . ; yT †. In order to do so, it is convenient to translate the encompassing relationship, which is often de®ned in parameter space, to the mapping relationship between the elements y in RT . This not only facilitates the discussion in RT , but also has the advantage of accommodating those random vectors y that are generated by a non-parametric DGP (cf. GourieÂroux and Monfort (1995)). ÿ
Let xt be partitioned as xt0 ˆ yt ; zt0 and consider the conditional models M1 and M2 : M1 :

ff …yt j zt ; Xtÿ1 ; h†;

h 2 ?  Rp g

M2 :

fg…yt j zt ; Xtÿ1 ; w†;

w 2 )  Rq g

…14†

which are such that the conditional densities f …yt j zt ; Xtÿ1 ; h† and g…yt j zt ; Xtÿ1 ; w† are suf®ciently well-behaved functions to ensure the global uniqueness of the pseudo-true values h0 and w0 . It is assumed that conditioning on zt is valid (see Engle, Hendry and Richard (1983)), and that yt does not Granger cause zt (see Granger (1969)). It is also assumed that the random variable of interest, yt , can be uniquely expressed as: ÿ
yt ˆ yt …h† ˆ rt zt ; Xtÿ1 ; 1;t ; h and

ÿ
yt ˆ yt …w† ˆ kt zt ; Xtÿ1 ; 2;t ; w

from models M1 and M2 , respectively. Further, the functions ÿ
ÿ
and kt zt ; Xtÿ1 ; 2;t ; w rt zt ; Xtÿ1 ; 1;t ; h are assumed to have derivatives for all arguments, and to be contraction mappings (with probability one) with respect to Xtÿ1 when it includes lagged values of yt . Let yt …w^ T † ˆ kt …zt ; Xtÿ1 ; 0; w^ T † and yt …w1 …h^ T †† ˆ kt …zt ; Xtÿ1 ; 0; w1 …h^ T †† when w1 …h† is the M1 pseudo-true value of w^ T ; or binding function from ? to ) (see GourieÂroux and Monfort (1995)). Under these conditions, the encompassing contrast w0 ÿ w1 …h0 † for the hypothesis M1 E M2 …w† takes the form: w0 ÿ w1 …h0 † ˆ plim fw^ T ÿ w1 …h^ T †g ˆ 0: T!1

…15†

76

Maozu Lu and Grayham E. Mizon

The contrast in (15) can be translated to the following relationship in RT : fy…w^ T † ÿ y…w1 …h^ T ††g ˆ 0 where y 0 ˆ …y1 ; . . . ; yT †. y…w^ T † and y…w1 …h^ T †† 2 RT are two T  1 random vectors constrained by parameter sets ? and ). Therefore, since w^ T and h^ T are functions of y 0 ˆ …y1 ; y2 ; . . . ; yT †; y ! y…w^ T † and y ! y…w1 …h^ T †† de®ne two mappings on RT ! RT . These mappings will form the basis of the following discussion of encompassing. In addition, de®ne the following mapping for w^ T 2 ): MTý : RT 7 !S T …)†  RT such that for 8 y 2 RT , ^ 2 ); MTý …y† ˆ y…w^ T † 2 S T …)†  RT ; w T

…16†

and the corresponding mapping for h^ T 2 ?: MT : RT 7 !S T …?†  RT such that for 8 y 2 RT , MT …y† ˆ y…h^ T † 2 S T …?†  RT ;

^ hT 2 ?:

…17†

When the only restrictions on the parameter spaces ) and ? are of the form '…w† ˆ 0 and /…h† ˆ 0 so that they are dimension reducing, S T …)† and S T …?† are subspaces of RT : By the construction of MTý and MT , y…w1 …h^ T †† can be expressed as: y…w1 …h^ T †† ˆ MTý
MT …y†; ÿ
where MTý
MT denotes the product mapping. Hence the condition for M1 E M2 at y 2 RT is equivalent to: k MTý …y† ÿ MTý
MT …y† k ˆ 0:

…18†

The nature of the mappings MTý , MT and MTý
MT not only depends on the functional form of f …h† and g…w† and the relation between S T …)† and S T …?†, but also depends on the nature of the DGP. This makes the derivation of general conditions for (18) to hold impossible ± in general the encompassing relation between f …h† and g…w† requires separate investigation for each particular case. However, there is a degenerate case of (18), in which MTý …y† ˆ MTý
MT …y†

…19†

Orthogonal projection and encompassing

77

holds independently of the nature of the DGP, and this is referred to as automatic encompassing as it does not provide any relevant information about the DGP. Corollary 2 in section 2.1 gives suf®cient conditions for a nesting model M1 automatically to encompass a nested model M2 : An important component of those suf®cient conditions is the requirement that nesting model M1 has orthogonal parameters. A similar, but more general, condition can be derived for (19) when the mappings MTý and MT are orthogonal projections de®ned on S T …)† and S T …?†, respectively. A continuous linear mapping MTý de®ned on subspace S T …)†  RT is T orthogonal, if it is self-adjoint ± MTý ˆ MT ý , where Mý is the adjoint T T T projection ± and idem-potent so that Mý
Mý ˆ Mý : This condition is often satis®ed, especially when both densities f …h† and g…w† belong to the linear exponential family. In the sequel it is assumed that the mappings MTý and MT are orthogonal projections de®ned on S T …)† and S T …?†, respectively. Since MTý is an orthogonal projection, for (19) to hold it is necessary that the product projection MTý
MT …
† also be orthogonal. However, the fact that MTý and MT are orthogonal projections is not suf®cient for the orthogonality of the product projection MTý
MT . A necessary and suf®cient condition for MTý
MT to be an orthogonal projection is provided by the following proposition. Proposition 4 Let MTý and MT be orthogonal projections de®ned on S T …)† and S T …?†, respectively. A necessary and suf®cient condition for the product projection MTý
MT to be an orthogonal projection is that MTý and MT commute, i.e. MTý
MT ˆ MT
MTý :

…20†

In this case MTý
MT is the orthogonal projection on S T …)† \ S T …?†: (For a proof see e.g. Bachman and Narici (1966).) An implication of Proposition 4 is that the product projection MTý
MT cannot de®ne the same orthogonal projection as MTý if the two subspaces are disjoint, i.e. S T …)† \ S T …?† ˆ fg. Since having disjoint parameter sets is a de®ning characteristic of non-nested models, it follows that automatic encompassing can never occur between non-nested models. As an illustration of the result in (20) consider the following two conditional regression models:

78

Maozu Lu and Grayham E. Mizon

M1 :

y ˆ Z1 ÿ 1 ‡ u1

M2 :

y ˆ Z2 ÿ 2 ‡ u2

…21†

when y is T  1; Zi is T  ki ; ÿ 1 is ki  1; and ui  N…0; 1† for i ˆ 1; 2: Let Z ˆ …Z1 ; Z2 † have rank k ˆ k1 ‡ k2 so that the two parameter subspaces ) and ? are disjoint, when h ˆ ÿ 1 and w ˆ ÿ 2 . The two subspaces S T …)† and S T …?† are given by: ÿ
ÿ
ÿ1 S T …)† ˆ fy ÿ^ 2 ˆ MTý …y† ˆ Z2 Z20 Z2 Z20 y ˆ P2 y; ÿ^2 2 )g ÿ
ÿ
ÿ1 S T …?† ˆ fy ÿ^ 1 ˆ MT …y† ˆ Z1 Z10 Z1 Z10 y ˆ P1 y; ÿ^1 2 ?g ÿ
ÿ1 when ÿ^ i ˆ Zi0 Zi Zi0 y for i ˆ 1; 2. It is easy to see that the product projection MTý
MT is not orthogonal because in general MTý and MT do not commute: P1 P2 6ˆ P2 P1 :

…22†

The next result shows that, when the two models are nested, the orthogonality of MTý and MT is suf®cient for the product projection to be orthogonal. Proposition 5 Let MTý and MT be orthogonal projections de®ned on S T …)† and S T …?†, respectively. If M2 is nested in M1 with )  ? so that S T …)†  S T …?†, then the product projection MTý
MT …
† is an orthogonal projection de®ned on S T …)†: (Proof see A.4 in the Appendix.) Note that the nesting of M2 in M1 alone is not suf®cient for this result. The requirement that MTý and MT are orthogonal projections is critical. Also note that when MTý and MTý
MT are orthogonal projections de®ned on S T …)† it follows that MTý ˆ MTý
MT on S T …)†, which implies the following proposition. Proposition 6 If MTý and MT are orthogonal projections on S …)† and S T …?†, respectively, and S T …)† is a subspace of S T …?†, then T

MTý ˆ MTý
MT and hence M1 E M2 …w† automatically. Note that Proposition 3(i) is the special case of 6 that arises when both ) and ? are Euclidean subspaces with /…h† ˆ 0 being the dimensionreducing constraints that de®ne ) as a subspace of ?.

Orthogonal projection and encompassing

79

As an illustration, consider the previous example but further assume that Z2  Z1 so that M2  M1 , and Z2 ˆ Z1 &. Hence substituting for Z2 into (22) yields: P2 P1 ˆ P1 P2 ˆ P2 and thus, from Proposition 6, M1 E M2 …w† automatically irrespective of the nature of the DGP. 4.2

Limit of orthogonal projection MTý

Finally, it is shown that the above analysis remains valid when the dimension T of the Hilbert space RT tends to in®nity. This is important since much statistical inference using econometric models, including encompassing analyses, relies on limiting distribution theory. Noting that S T …)† is the subspace on which the orthogonal projection MTý is de®ned, it is necessary to show that as T ! 1 the limit of MTý exists and de®nes an orthogonal projection in the in®nite-dimensional Hilbert space H ˆ limT!1 RT : Let fS T …)†; T ˆ l; l ‡ 1; . . .g be a series of T-dimensional subspaces of the Hilbert space RT , fMTý ; T ˆ l; l ‡ 1; . . .g be a series of orthogonal projections de®ned on RT } S T …)† ; and S …)† ˆ limT!1 [Tl S T …)†. It is shown in A.5 of the Appendix that Mý is the orthogonal projection de®ned on H } S …)†, where MTý ! Mý : The limits of MT and MTý
MT also exist for the same reasons, thus yielding: Proposition 7 If MTý and MT are orthogonal projections on S …)† and S T …?†, respectively, and S T …)† is a subspace of S T …?†, then T

MTý ˆ MTý
MT so that M1 E M2 …w†. This encompassing relation remains valid when T ! 1, since Mý ˆ lim MTý ˆ lim MTý
MT ˆ Mý
M de®nes an orthogonal projection on H. 5

Conclusion

The ability of a general model to encompass a simpli®cation of itself has been questioned rarely. However, GourieÂroux and Monfort (1995) presented an example in which a general model was unable to encompass a

80

Maozu Lu and Grayham E. Mizon

simple model. This paper has presented conditions under which a model encompasses models nested within it independently of the nature of the DGP, and thus automatically encompasses the nested models. Equally importantly it was pointed out that nesting is more than one model being an algebraic simpli®cation of another. By interpreting encompassing as the existence of orthogonal projections in Hilbert space, it was shown that, when the model M2 is nested within M1 , a suf®cient condition for M1 to encompass M2 is that the individual mappings associated with the models are orthogonal projections. As a by-product, it was also pointed out that automatic encompassing cannot occur between non-nested models. One of the important methodological implications of these ®ndings is that care should be taken to establish the presence and nature of nesting amongst models prior to implementing hypothesis tests for model reduction, as for example is done in general-to-simple modelling.

Appendix A.1 If w and h satisfy (3), then:   … g…w † g…w † dy 0 ˆ log f …h  †   … g…w † g…w † dy  min log h2? f … h†   … g…w † ˆ log g…w † dy f …h…w ††  0: Therefore, h ˆ h…w † is the M2 pseudo-true value of h 2 ? 8 w 2 ). GourieÂroux and Monfort (1995) de®ne h ˆ h…w † as the binding function from ) to ?. A.2 Here it is shown that if g…w† is nested in f …h† in the sense of (3) then )  ?; or ) \ ? ˆ ) with probability one. g…w† is nested in f …h† in the sense of (3) if 8 w 2 ); there exists h 2 ?  ?, such that   … g…w† log g…w† dy ˆ 0: …23† f …h †

Orthogonal projection and encompassing

81

First, if for each w 2 ), there exists h 2 ?  ?; then it is necessary and suf®cient that g…w† and f …h † are equivalent with  probabilityþ one. Therefore, there is a one-to-one mapping between   þ ÿ
ÿ
g…w† j w 2 ) and  . Otherwise, if both g w and g w2 are mappings to f …h † j h 2 ? 1 ÿ
ÿ
f …h †, then g w1 and g w2 are identical with probability one, and  hence w1 ˆ w2 : Further, the equivalence between g…w†  and f …h †þ holds under absolutely continuous measure, so g…w† j w 2 ) and   any þ f …h † j h 2 ? are isomorphic. Let :0 de®ne the set of zero measure on which g…w† 6ˆ f …h †; g…w† can then be equally written as  f …h † on ? ÿ :0 g…w† ˆ g…w† on :0 g…w† is a density function de®ned on ? ; and )  ? with probability one. On the other hand, if for each w 2 ); g…w† can be obtained as a special case of f …h†; then there exists h 2 ? so that (23) holds. A.3 Here, it is shown that: 9 8 T X > y > > > > zs ys > > > T = < X y sˆ1 plim z1t yt ÿ zt T ˆ0 > > X T!1 tˆ1 > y2 > > > > > zs ; : implies:

…24†

sˆ1

9 8 T X > y > > > > zs ys > > > = < T X y sˆ1 ˆ0 plim z2t yt ÿ zt T > X > T!1 tˆ1 > y2 > > > > zs > ; :

…25†

sˆ1

and vice versa. The following are equivalent to the orthogonal condition in (24): ( ) T T T T X X X X y2 y plim zt z1t yt ÿ z1t zt …z1t ‡ z2t †yt T!1

tˆ1

ˆ plim

tˆ1

( T X

T!1

ˆ plim

tˆ1

( T X

T!1

tˆ1

tˆ1

z1t yt z1t yt

T  X tˆ1 T X tˆ1

zyt

tˆ1

ÿ

z2t zyt



z1t zyt

ÿ

T X tˆ1

ÿ

T X tˆ1

z1t zyt

z1t zyt

T X tˆ1

T X tˆ1

) z2t yt

)

z2t yt

ˆ 0:

…26†

82

Maozu Lu and Grayham E. Mizon

On the other hand, condition (25) can be rewritten as: ( ) T T T T X X X X y y zt z2t yt ÿ z2t zt …z1t ‡ z2t †yt plim T!1

tˆ1

ˆ plim

T!1

ˆ plim

T!1

tˆ1

( T X tˆ1

( T X tˆ1

tˆ1

z2t yt z2t yt

T  X tˆ1 T X tˆ1

tˆ1

zyt

ÿ

z1t zyt



z2t zyt

ÿ

T X tˆ1

ÿ

T X tˆ1

z2t zyt

z2t zyt

T X tˆ1

)

T X

z1t yt

tˆ1

)

z1t yt

ˆ 0:

…27†

Therefore, the conditions in (26) and (27) are equivalent. A.4 Proof. Consider any y 2 RT that can be decomposed in the following form: y ˆ z ‡ z? ;

z 2 S T …)†;

z? 2 S T …)†? :

Therefore, MTý …y† ˆ z 2 S T …)†  S T …?†; which implies, MT
MTý …y† ˆ z and it follows that MT
MTý …
† ˆ MTý …
†: Taking adjoints in (28), we have, ÿ T T
T T ˆ MT MT ý ˆ M
Mý ý
M ˆ MTý
MT ˆ MTý :

…28†

…29†

The last equality sign holds because of the orthogonality MT and MTý . Combining (28) and (29) yields MTý
MT …
† ˆ MT
MTý …
†; i.e. MTý …
† and MT …
† commute. Therefore, MTý
MT …
† is an orthogonal projection de®ned on S T …)† \ S T …?† ˆ S T …)†. &

Orthogonal projection and encompassing

83

A.5 Proof. First, the orthogonal projection MTý on S T …)† is continuous and bounded, and for 8 y 2 RT k MTý …y† k  k y k < 1: On the other hand, the subspaces fS T …)†; T ˆ l; l ‡ 1; . . . ; g form an increasing sequence: S l …)†  S l‡1 …)† 


therefore: MTý
MT‡1 ˆ MTý ý and 2 k MTý …y† k2 ˆ k MTý
MT‡1 ý … y† k 2  k MTý k2
k MT‡1 ý … y† k 2 2  k MT‡1 ý … y† k  k y k :

Hence, 2 2 k MTý …y† k2  k MT‡1 ý … y† k 


 k y k

implies MTý …
† strongly converges in the norm k
k to the projection Mý …
† which is de®ned on S …)†  H. & References Bachman, G. and L. Narici (1966), Functional Analysis, New York: Academic Press. Barndorff-Nielsen, O.E. and D.R. Cox (1994), Inference and Asymptotics, Monographs on Statistics and Applied Probability No. 52, London: Chapman & Hall. Bontemps, C. and G.E. Mizon (1997), `Congruence and Encompassing', European University Institute, mimeograph. Cox, D.R. and N. Reid (1987), `Parameter Orthogonality and Approximate Conditional Inference', Journal of the Royal Statistical Society, Series B, 49: 1±49. Engle, R.F., D.F. Hendry and J.-F. Richard (1983), `Exogeneity', Econometrica, 51: 277±304. GourieÂroux, C. and A. Monfort (1995), `Testing, Encompassing and Simulating Dynamic Econometric Models', Econometric Theory, 11: 195±228. Granger, C.W.J. (1969), `Investigating Causal Relations by Econometric Models and Cross-Spectral Methods', Econometrica, 37: 424±438.

84

Maozu Lu and Grayham E. Mizon

Hendry, D.F. (1995), Dynamic Econometrics, Oxford: Oxford University Press. Hendry, D.F. and G.E. Mizon (1978), `Serial Correlation as a Convenient Simpli®cation, Not a Nuisance: A Comment on a Study of the Demand for Money by the Bank of England', Economic Journal, 88: 549±563. Hendry, D.F. and J.-F. Richard (1989), `Recent Developments in the Theory of Encompassing', in B. Cornet and H. Tulkens (eds.), Contributions to Operations Research and Economics. The XXth Anniversary of CORE, Cambridge, Mass.: MIT Press, pp. 393±440. Lu, M. and G.E. Mizon (1997), `Mutual Encompassing and Model Equivalence', EUI Economics Department Working Paper No. 97/17. Mizon, G.E. (1995a), `A Simple Message for Autocorrelation Correctors: Don't', Journal of Econometrics, 69: 267±288. (1995b), `Progressive Modelling of Macroeconomic Time Series ± The LSE Methodology', in K.D. Hoover (ed.), Macroeconometrics: Developments, Tensions, and Prospects, Dordrecht: Kluwer Academ Press, pp. 107±170. Mizon, G. E. and J.-F. Richard (1986), `The Encompassing Principle and Its Application to Non-nested Hypothesis Tests', Econometrica, 54: 657±678. Pesaran, M.H. (1987), `Global and Partial Non-nested Hypotheses and Asymptotic Local Power', Econometric Theory, 3: 69±97. Sargan, J.D. (1980), `Some Tests of Dynamic Speci®cation for a Single Equation', Econometrica, 48: 879±897. (1988), Lectures on Advanced Econometric Theory, Oxford: Basil Blackwell.

3

Exact properties of the maximum likelihood estimator in exponential regression models: a differential geometric approach Grant Hillier and Ray O'Brien

1

Introduction

In a recent paper Hillier and Armstrong (1996) have given an integral formula for the (exact) density of the maximum likelihood estimator (MLE). The formula,1 which expresses the density at a point t, does not require that the estimator be a known function of the data, but does require that the manifold on which the MLE is ®xed (i.e. the level set of the MLE) be known. One situation, but by no means the only case, in which this occurs is when the MLE is uniquely de®ned by the vanishing of the score vector. The importance of this result lies in the fact that the formula can be used to obtain the exact density even when the estimator is only implicitly de®ned in terms of the data. The exponential regression model is well known to be of this type, and in this chapter we apply the Hillier and Armstrong result to the MLE for this model. The observations x1 ; . . . ; xn are assumed to be independent realisations of exponential random variables with means i ˆ expf 0 wi g;

i ˆ 1; . . . ; n;

…1†

where  is a k  1 vector of parameters, and wi is a k  1 vector of covariates, assumed non-random. The joint density of the data is thus: ( 0

 exp ÿ pdf …x1 ; . . . ; xn ; † ˆ expfÿn wg

n X iˆ1

) 0

xi exp‰ÿ wi Š ;

…2†

1 A surface integral over the manifold in the sample space upon which the MLE has the value t.

85

86

Grant Hillier and Ray O'Brien

for xi > 0; i ˆ 1; . . . ; n, where w is the vector of sample means of the wi . Let x ˆ …x1 ; . . . ; xn † 0 , and abbreviate the condition xi > 0 for i ˆ 1; . . . ; n to simply x > 0: The log-likelihood, score vector and observed information matrix are: `…x; † ˆ ÿn 0 w ÿ

n X iˆ1

xi expfÿ 0 wi g;

…3†

u…x; † ˆ u…x; ; W† ˆ @`…x; †=@ ˆ ÿnw ‡

n X iˆ1

xi wi expfÿ 0 wi g;

…4†

j…x; † ˆ j…x; ; W† ˆ ÿ@2 `…x; †=@@ 0 ˆ

n X iˆ1

xi wi wi0 expfÿ 0 wi g;

…5†

respectively. Provided the matrix W…n  k† with rows wi0 , i ˆ 1; . . . ; n, has rank k, it is well known that the MLE for  is the unique solution to the equations u…x; † ˆ 0; but the MLE cannot be expressed directly in terms of x1 ; . . . ; xn . This has hitherto prevented an analysis of the smallsample properties of the MLE in this model, but the Hillier/Armstrong formula makes such results accessible, at least for small values of k, as we shall see. To the best of our knowledge the only other analytic study of the exact properties of the MLE in this model is Knight and Satchell (1996). This used an approach suggested by Huber (1964; see also Shephard (1993)) and characteristic function inversion techniques to deduce some properties of the density for the cases k ˆ 1 and k ˆ 2, but this approach does not generalise easily. In fact, our differential geometric formula can be regarded as a generalisation of the Huber approach to the multi-parameter case, avoiding the need for characteristic function inversion. We denote the MLE for  by T ˆ T…x† ˆ T…x; W†, and a particular value of T by t. The density of T (with respect to Lebesgue measure, dt) at T ˆ t will be denoted by pdfT …t; †, or, if the dependence on W is important, by pdfT …t; ; W†. From Hillier and Armstrong (1996, equation (26)), we have the following expression for the density of T:

Exact properties of the MLE

87

ÿ ÿÿ1=2 n ÿX ÿ ÿ ÿ 0 0  ÿ pdfT …t; † ˆ expfÿn wg wi wi expfÿ2t wi gÿ ÿ iˆ1 ÿ 0

ÿ ÿ n ÿX ÿ ÿ ÿ 0 0  xi wi wi expfÿt wi gÿ ÿ ÿ S…t† ÿ iˆ1 …

"

 exp ÿ

n X iˆ1

#

0

xi expfÿ wi g …dS…t††;

…6†

P  and …dS…t†† denotes where S…t† ˆ fx; x > 0; niˆ1 xi wi expfÿt 0 wi g ˆ nwg, the (canonical) volume element on the manifold S…t† (see Hillier and Armstrong (1996), Appendix A, for de®nitions and technical details). That is, S…t† is the intersection of an …n ÿ k†-dimensional hyperplane with the non-negative orthant. The key problem, therefore, is the evaluation of the surface integral in (6). Because the surface S…t† is, in this case, ¯at, S…t† admits a global coordinate chart, so that this surface integral can be reduced to an ordinary integral over a region in Rnÿk . Nevertheless, the evaluation of this integral presents considerable dif®culties: the region of interest consists of a polyhedron in Rnÿk bounded by the coordinate axes and k intersecting hyperplanes. In the present chapter we give details of the evaluation of this integral for the cases k ˆ 1 and k ˆ 2: The completely general case can no doubt be dealt with similarly; see Schechter (1998) for one possible algorithm.

2

Some properties of the density in the general case

Before considering the evaluation of (6) in detail we make some general observations on the density of T that follow almost trivially from equations (4)±(6). Consider ®rst a transformation of the wi ; wi ! A 0 wi ; i ˆ 1; . . . ; n, where A is a k  k non-singular matrix, so that W ! WA. From (4) we see that u…x; ; WA† ˆ A 0 u…x; A; W†. Hence, T…x; W† ˆ AT…x; WA†, so that the transformation W ! WA induces the transformation T ! Aÿ1 T on the MLE. It follows from this observation that pdfT …t; ; W† ˆ kAkÿ1 pdfT  …t ; ; WA†, where T  ˆ T…x; WA† ˆ Aÿ1 T…x; W†, and t ˆ At . That is, the density of T ˆ T…x; W† is trivially obtainable from the density of T  ˆ T…x; WA†, the MLE when W is replaced by WA, so that there is no loss of generality in standardising the wi0 's so that, for instance, W 0 W ˆ Ik . Note that W 0 W is the Fisher information matrix for .

88

Grant Hillier and Ray O'Brien

Next, for a ®xed value of t one can transform variables in equation (6) from xi to x~ i ˆ expfÿt 0 wi gxi ; i ˆ 1; . . . ; n. This maps the manifold S…t† ~ into a new manifold S…t†, say, and using Hillier and Armstrong (1996, ~ are related by: equation (A4)), the volume elements on S…t† and S…t† ÿ ÿÿ1=2 n ÿX ÿ ÿ ÿ ~  ÿ xi wi wi0 expfÿ2t 0 wi gÿ …d S…t†† ˆ expfÿnt 0 wg ÿ iˆ1 ÿ  j W 0 W j1=2 …dS…t††: Rewriting (6) in terms of the transformed xi we obtain:  j W 0 W jÿ1=2 pdfT …t; † ˆ expfn…t ÿ † 0 wg ÿ " # … ÿÿX n n ÿ X ÿ 0ÿ 0  ÿ xi wi wi ÿ exp ÿ xi expf…t ÿ † wi g …dS†; ÿ S ÿ iˆ1 iˆ1 …7† P  does not depend on t. It follows at where S ˆ fx; x > 0; niˆ1 xi wi ˆ nwg once from (7) that the density of T depends on …t; † only through their difference …t ÿ †. When regarded as a function of d ˆ …t ÿ †, it is easy to see that, at each point on the surface S, the integrand (with the term  attached) is maximised at d ˆ 0. Hence, the mode of the expfnd 0 wg (joint) density is at the point t ˆ . Write the density of T ˆ T…x; W† in (7) as f …d; W†. It is clear from (7) that f …d; W† is invariant under permutations of the rows of W, and also that the density of T  ˆ T…x; WA† ˆ Aÿ1 T…x; W† is f …d; WA† ˆ kAkf …Ad; W† for any non-singular k  k matrix A. In particular, f …d; ÿW† ˆ f …ÿd; W† (on choosing A ˆ ÿIk †, and thus, if ÿW ˆ PW for some permutation matrix P, the density of …t ÿ † is symmetric about the origin. If the model contains an intercept, and the remaining variables are symmetric about their means, the density of the estimates of the coef®cients of those variables will be symmetric about the corresponding true values, and hence will be unbiased if their means exist.

3

The one-parameter case

We consider ®rst the case k ˆ 1; and work from expression (7) for the density, which, becomes (on replacing wi by zi †:

Exact properties of the MLE

 pdfT …t; † ˆ expfn…t ÿ †zg "  exp ÿ

n X iˆ1

89

" n X iˆ1

z2i

#ÿ1=2 … " n X S

iˆ1

# xi z2i

# xi expf…t ÿ †zi g …dS†;

…8†

P where S is the intersection of the hyperplane niˆ1 xi zi ˆ nz with the nonnegative orthant. To simplify matters we assume that the zi are all of the same sign, and there is no loss of generality in taking this to be positive. Results for the case where the zi0 's are of mixed signs require only minor modi®cations in what follows. We also assume for convenience that the zi0 's are distinct. The manifold S admits a global coordinate chart, and we can use x2 ; . . . ; xn as coordinates, setting þ x1 ˆ

zÿ1 1

nz ÿ

n X iˆ2

! xi z i :

…9†

We then have (from Hillier and Armstrong (1996), equation (A3)): " …dS† ˆ

zÿ1 1

n X iˆ1

#1=2 z2i

dx2 dx3 . . . dxn ;

and, in view of (9), the region of integration becomes: ( R ˆ xi > 0; i ˆ 2; . . . ; n;

n X iˆ2

) xi zi < nz :

To P further simplify the integration it will be helpful to `lift' the term ‰ niˆ1 xi z2i Š in the integrand in …8) into the exponential. This can be done by writing "

n X iˆ1

# xi z2i

( exp ÿ

n X iˆ1

)

"

(

xi ri ˆ …@=@w† exp ÿ

n X iˆ1

)# xi …ri ÿ

wz2i †

; wˆ0

where we have put ri ˆ expf…t ÿ †zi g; i ˆ 1; . . . ; n. Substituting for x1 from (9), and noting that the differentiation (with respect to w† commutes with the integration, the density becomes:

90

Grant Hillier and Ray O'Brien

pdfT …t; † ˆ zÿ1 1 expfn…t ÿ †zg…@=@w†‰expfÿ…nza1 =z1 †g ( # … n X  exp ÿ xi ‰ai z1 ÿ a1 zi Š=z1 dx2 . . . dxn R

iˆ2

wˆ0

…10† where we have now set ai ˆ ri ÿ wz2i ; i ˆ 1; . . . ; n. The essential problem, therefore, is the evaluation of the integral in (10). De®ne dsi ˆ …zs ai ÿ zi as †=zs ;

i ˆ 2; . . . ; n;

s < i:

…11†

The x2 -integral in (10) is (with the term expfÿnza1 =z1 g attached): ( exp ÿnza1 =z1 ÿ where u2 ˆ zÿ1 2 ‰nz ÿ

iˆ3

exp ÿnza1 =z1 ÿ ( ˆ

iˆ3

Pn

(

ÿ1 d12

n X

xi d1i

u2

0

) expfÿx2 d12 gdx2 ;

…12†

xi zi Š, giving, on integrating out x2 ;

n X

(

)( …

iˆ3

) ÿ1 xi d1i d12 f1 ÿ exp…ÿd12 u2 †g

exp ÿnza1 =z1 ÿ (

ÿ exp ÿnza2 =z2 ÿ

n X iˆ3

n X iˆ3

) xi d1i ))

xi d2i

;

…13†

since a1 =z1 ‡ d12 =z2 ˆ a2 =z2 and d1i ÿ zi d12 =z2 ˆ d2i . The integral is interpreted as zero if u2  0: Integrating now with respect to x3 we have: ( ÿ1 d12

( exp ÿnza1 =z1 ÿ (

n X iˆ4

ÿ exp ÿnza2 =z2 ÿ

) ÿ1 xi d1i d13 ‰1 ÿ expfÿd13 u3 gŠ

n X iˆ4

)

) xi d2i

ÿ1 ‰1 d23

ÿ expfÿd23 u3 gŠ ;

…14†

Pn where u3 ˆ zÿ1 3 ‰nz ÿ iˆ4 xi zi Š. There appear to be four distinct terms here, but identities similar to those below (13) yield:

Exact properties of the MLE

( …d12 d13 †

ÿ1

exp ÿnza1 =z1 ÿ

(  exp ÿnza2 =z2 ÿ (  exp ÿnza3 =z3 ÿ

n X iˆ4 n X iˆ4

91 n X iˆ4

) xi d1i ÿ …d12 d23 †ÿ1

)

xi d2i ‰…d12 d13 †ÿ1 ÿ …d12 d23 †ÿ1 Š ) xi d3i :

…15†

The iterative relation is now clear: the p'th step in the integration yields a linear combination of terms ( ) n X xi dsi ; s ˆ 1; . . . ; p; exp ÿnzas =zs ÿ iˆp‡1

in which the coef®cients are simply the coef®cients at the previous step ÿ1 ÿ1 ÿ1 multiplied by d1p ; d2p ; . . . ; dpÿ1;p , respectively, except for the last term …s ˆ p†, whose coef®cient is minus the sum of the coef®cients of all lower terms. Thus, if we denote P by cp the p  1 vector of coef®cients of the terms expfÿnzas =zs ÿ niˆp‡1 xi dsi g after integrating out xp , we may write: cp ˆ Lp cpÿ1 ; where Lp is the p  …p ÿ 1† matrix: 2 ÿ1 3 0 :: 0 d1p 6 7 6 7 ÿ1 d2p :: : 60 7 6 7 6 7 Lp ˆ 6 : : :: : 7: 6 7 6 7 ÿ1 0 :: dpÿ1;p 60 7 4 5 ÿ1 ÿ1 ÿ1 ÿd1p ÿd2p :: ÿdpÿ1;p

…16†

After integrating out x2 ; . . . ; xn , therefore, we are left with a linear combination of the terms gn … j† ˆ expfÿnzaj =zj g;

j ˆ 1; . . . ; n;

…17†

with vector of coef®cients, cn , given by the recursive relation: cn ˆ Ln Lnÿ1 . . . L2 ˆ starting with

nÿ1 Y iˆ1

Lnÿi‡1 ;

…18†

92

Grant Hillier and Ray O'Brien

0 L2 ˆ @

ÿ1 d12 ÿ1 ÿd12

1 A:

We therefore have a very simple expression for the density:    ÿ †g…@=@w† cn0 gn wˆ0 pdfT …t; † ˆ zÿ1 1 expfnz…t

   ÿ †g …@cn =@w† 0 gn ‡ cn0 …@gn =@w† wˆ0 ; ˆ zÿ1 1 expfnz…t …19†

where gn is the n  1 vector with elements gn … j† given in (17). It remains to evaluate the derivatives in (19), and then set w ˆ 0: It is easy to see that @gn … j†=@w jwˆ0 ˆ nzzj expfÿnzrj =zj g ˆ nzzj g~ n … j†; say:

…20†

De®ning L~ p as Lp has been de®ned above, but with the dsp replaced by d~sp ˆ ‰zs rp ÿ zp rs Š=zs (since ai ˆ ri when w ˆ 0†, we de®ne c~n  cn jwˆ0 ˆ

nÿ1 Y iˆ1

L~ nÿi‡1 :

…21†

This deals with the second term in the ‰
Š in (19). Now, from the de®nition of cn in terms of the Lp in (18), we have that: @cn =@w ˆ

nÿ1  X  Ln Lnÿ1 . . . Lnÿi‡2 …@Lnÿi‡1 =@w†Lnÿi Lnÿiÿ1 . . . L2 : iˆ1

The matrices @Lp =@w jwˆ0 that occur here have the same structure as the L~ p except that the elements ÿ1 ÿ2 d~sp are replaced by zp …zp ÿ zs †d~sp . Denote these matrices by L~ p , p ˆ 2; . . . ; n: Then clearly @cn =@w jwˆ0 ˆ

nÿ1 X ‰L~ n . . . L~ nÿi‡2 L~ nÿi‡1 L~ nÿi . . . L~ 2 Š ˆ c~n ; say: iˆ1

Hence we ®nally have an expression for the density in the form:  ÿ †g pdfT …t; † ˆ zÿ1 1 expfnz…t

n X jˆ1

g~n … j†fc~n … j† ‡ nzzj c~n … j†g:

…22†

Unfortunately, because the vectors c~n and c~n are both de®ned only by recursive formulae, it is dif®cult to study the properties of the density (22)

Exact properties of the MLE

93

analytically. In section 5 below we brie¯y summarise some results obtained by direct numerical evaluation of (22).

4

Inclusion of a constant term

Suppose now that E…xi † ˆ expf ‡ zi g, so that wi ˆ …1; zi † 0 in the notation used in the Introduction. We denote the (®xed values of) the MLEs for …; † by …a; t†. From equation (7) we have: ÿ þ !þ ! 0 ÿÿ1=2 n ÿX 1 1 ÿÿ ÿ n ÿ pdfA;T …a; t; ; † ˆ 0 expfn…t ÿ †zg ÿ ÿ iˆ1 z z ÿ i

i

þ !þ ! 0 ÿ … ÿÿX n 1 1 ÿ ÿ ÿ  ÿ xi ÿ ÿ S iˆ1 z z ÿ i

(

 exp ÿ0

n X iˆ1

i

)

xi ri …dS†

…23†

where 0 ˆ exp…a ÿ †, and we have again put ri ˆ expf…t ÿ †zi g; i ˆ 1; . . . ; n. The integral is now over the surface S de®ned by: n X iˆ1

xi ˆ n;

and

n X iˆ1

 xi zi ˆ nz;

and xi > 0; i ˆ 1; . . . ;n. In what follows we assume that the zi0 's are distinct, and are ordered so that z1 < z2 <


< zn . It is clear from (23) that the density is invariant to the order of the zi0 's, so the assumption that the zi0 's are ordered is not restrictive. The assumption that the zi0 's are distinct is restrictive, but unlikely to be important in practice. We ®rst choose n ÿ 2 coordinates for the surface S, and for this purpose it will be convenient to use x2 ; . . . ; xnÿ1 . Writing x1 and xn in terms of x2 ; . . . ; xnÿ1 we have:  x1 ˆ n…zn ÿ z†=…z n ÿ z1 † ÿ

xn ˆ n…z ÿ z1 †=…zn ÿ z1 † ÿ

nÿ1 X iˆ2 nÿ1 X iˆ2

xi ‰…zn ÿ zi †=…zn ÿ z1 †Š

…24†

xi ‰…zi ÿ z1 †=…zn ÿ z1 †Š:

…25†

Note that the constants, and the coef®cients of the xi , in both of these expressions are all positive, because of our ordering of the zi0 's.

94

Grant Hillier and Ray O'Brien

With these coordinates the integral (23) becomes an ordinary integral over x2 ; . . . ; xnÿ1 , and the volume element becomes: ÿ þ !þ ! 0 ÿ1=2 n ÿX 1 1 ÿÿ ÿ1 ÿ …26† …dS† ˆ ‰…zn ÿ z1 †Š ÿ ÿ …dx2 dx3 . . . dxnÿ1 †: ÿ iˆ1 z z ÿ i

i

Because x1 and xn in (24) and (25) must be positive, the region of integration becomes that part of the non-negative orthant (for x2 ; . . . ; xnÿ1 † within which: nÿ1 X iˆ2

 and xi …zn ÿ zi † < n…zn ÿ z†

nÿ1 X iˆ2

xi …zi ÿ z1 † < n…z ÿ z1 †: …27†

That is, the region of integration becomes the subset, R say, of the …n ÿ 2†-dimensional non-negative orthant below both of the hyperplanes de®ned by replacing the inequalities in (27) by equalities. It is straightforward to check that the two hyperplanes involved must intersect, so neither lies entirely below the other. This obviously complicates the integration problem to be dealt with. We ®rst set out a notation that will be helpful in the evaluation of  i ˆ 1; . . . ; n, and the integral in (23). First, we de®ne bi ˆ zi ÿ z; bjk ˆ zj ÿ zk ; j; k ˆ 1; . . . ; n; j 6ˆ k. Note that bjk will be positive for j > k because of the ordering of the zi , and that the bi will necessarily be negative for i less than some integer p, say, …1  p < n†, and positive thereafter. This property of the bi0 's will be important in what follows. The following identities, easily derived from the de®nitions of the bi and bjk , will be used repeatedly in what follows to combine products of terms: br bis ‡ bs bri ÿ bi brs ˆ 0

…28†

bir bjs ÿ bij brs ÿ bis bjr ˆ 0:

…29†

Next we de®ne `rs ˆ nbs ‡

nÿ1 X iˆr

xi bis

…r ˆ 2; . . . ; n ÿ 2; s < r†;

…30†

and `r ˆ nbn ÿ

nÿ1 X iˆr

xi bni :

…31†

Substituting for x1 and xn from (24) and (25) into the determinantal factor in the integrand of (23) we ®nd that:

Exact properties of the MLE

95

ÿ þ !þ ! 0 ÿ n ÿX 1 1 ÿÿ ÿ xi ÿ ÿ ˆ n2 f…bn z21 ÿ b1 z2n †=bn1 ÿ z2 g ÿ iˆ1 z z ÿ i

i

‡n

nÿ1 X iˆ2

xi fbn1 z2i ÿ bni z21 ÿ bi1 z2n g=bn1 :

…32†

Note that this is linear in x2 ; . . . ; xnÿ1 , not quadratic. As before, it will be helpful to `lift' the determinantal factor (32) into the exponential term in the integrand. We thus write the integrand in the form: " 2 nÿ1 0 ‰@=@wŠ expfÿn0 wz g

(  exp ÿn0 …‰bn a1 ÿ b1 an Š=bn1 † ÿ 0

nÿ1 X iˆ2

)# xi dn1i

; wˆ0

where we have de®ned dijk ˆ ‰ak bij ÿ ai bkj ÿ aj bik Š=bij ;

…33†

gij ˆ ‰bi aj ÿ bj ai Š=bij ;

…34†

ai ˆ …ri ÿ wz2i †;

…35†

and with i ˆ 1; . . . ; n:

Assuming that differentiation with respect to w commutes with the integration, we therefore have: pdfA;T …a; t; ; † ˆ nnÿ1 expfn…t ÿ †zg‰bn1 Šÿ1 …@=@w† 0 " expfÿn0 wz2 g expfÿn0 gn1 g

 … 

R

( exp ÿn0

nÿ1 X iˆ2

)

#

xi dn1i dx2 . . . dxnÿ1

: wˆ0

…36† Our ®rst task is therefore to evaluate the integral in the last line of (36), though in what follows we shall include the term expfÿn0 gn1 g in the derivation of the results, because this will facilitate their simpli®cation as we proceed.

96

Grant Hillier and Ray O'Brien

Consider now the integral with respect to x2 . From (27), the range of x2 is restricted by " # nÿ1 X xi bni =bn2 ˆ `3 =bn2 ; x2 < nbn ÿ iˆ3

and

" x2 < ÿnb1 ÿ

nÿ1 X iˆ3

# xi bi1 =b21 ˆ ÿ`31 =b21 :

Now, the difference between these two upper bounds is " # nÿ1 X `3 =bn2 ‡ `31 =b21 ˆ …bn1 =…bn2 b21 †† nb2 ‡ xi bi2 iˆ3

ˆ …bn1 =…bn2 b21 ††`32 :

…37†

For x3 ; . . . ; xnÿ1 such that `32 > 0 and `31 < 0; the x2 -integral is over the interval …0; ÿ`31 =b21 †, while for x3 ; . . . ; xnÿ1 such that `32 < 0 and `3 > 0; it is over the interval …0; `3 =bn2 †. Notice that, because all coef®cients other than b2 in `32 are positive, if b2 ˆ z2 ÿ z > 0 the only possibility is `32 > 0: However, assume for the moment that b2 < 0; so that `32 can be either positive or negative. Since the regions of subsequent integration with respect to x3 ; . . . ; xnÿ1 are disjoint, the integral with respect to x2 may be expressed as a sum of two terms, each to be subsequently integrated over different regions for x3 ; . . . ; xnÿ1 . The result of integrating out x2 is thus the sum of two terms: ( " #) nÿ1 X ÿ1 xi dn1i ‰1 ÿ exp…0 dn12 `31 =b31 †Š; ‰0 dn12 Š exp ÿ0 ngn1 ‡ iˆ3

to be integrated over the region f`32 > 0; `3 < 0g, and ( " #) nÿ1 X ÿ1 xi dn1i ‰1 ÿ exp…ÿ0 dn12 `3 =bn2 †Š; ‰0 dn12 Š exp ÿ0 ngn1 ‡ iˆ3

to be integrated over the region f`32 < 0; `3 > 0g. After some tedious algebra the two results above for the x2 -integral become (apart from the factor ‰0 dn12 Šÿ1 †: " ( " #) ( " #)# nÿ1 nÿ1 X X ÿ exp ÿ0 ng21 ‡ ; exp ÿ0 ngn1 ‡ xi dn1i xi d21i iˆ3

iˆ3

…38†

Exact properties of the MLE

97

to be integrated over f`32 > 0; `31 < 0g, plus " ( " #) ( " #) # nÿ1 nÿ1 X X ÿ exp ÿ0 ngn2 ‡ ; exp ÿ0 ngn1 ‡ xi dn1i xi dn2i iˆ3

iˆ3

…39† to be integrated over f`32 < 0; `3 > 0g. Now, the ®rst terms in (38) and (39) are the same, but are to be integrated over two disjoint regions. When added, therefore, the integral of this term will be over the union of the regions f`32 > 0; `31 > 0g and f`32 < 0; `3 > 0g, i.e. over the region f`31 > 0; `3 > 0g. The result of the x2 -integration is therefore a sum of three terms, each to be integrated over a different region of …x3 ; . . . ; xnÿ1 †-space. These are (again apart from the term ‰0 dn12 Šÿ1 †, together with their respective regions of subsequent integration: ( " #) nÿ1 X …`31 < 0; `3 > 0†; ‡ exp ÿ0 ngn1 ‡ xi dn1i ; …40† (

"

ÿ exp ÿ0 ngn2 ‡ (

"

ÿ exp ÿ0 ng21 ‡

iˆ3

nÿ1 X iˆ3 nÿ1 X iˆ3

#) xi dn2i

;

…`32 < 0;

`3 > 0†;

…41†

;

…`32 > 0;

`31 < 0†:

…42†

#) xi d21i

 the second term here is Note that if `32 cannot be negative (i.e. if z2 > z†, missing. The ®nal form of the result we seek is certainly not yet apparent, so we need to proceed to integrate out x3 in the same way. To do so we need to deal with the three terms in (40) to (42) separately, since each has a different region of integration for …x3 ; . . . ; xnÿ1 †. Proceeding as above for the x2 -integration, after isolating x3 each region gives rise to a sum of two x3 -integrals, and each of these yields two distinct terms. At this stage we have: from (40) ‡ ‰0 dn13 Šÿ1 multiplied by ( " #) nÿ1 X xi dn1i ‰1 ÿ exp…0 dn13 `41 =b31 †Š exp ÿ0 ngn1 ‡ iˆ4

…`43 > 0;

`41 < 0†

98

Grant Hillier and Ray O'Brien

plus (

"

exp ÿ0 ngn1 ‡

nÿ1 X iˆ4

#) ‰1 ÿ exp…ÿ0 dn13 `4 =bn3 †Š

xi dn1i

…`43 < 0;

`4 > 0†

from (41) ÿ‰0 dn23 Šÿ1 multiplied by ( " #) nÿ1 X xi dn2i ‰1 ÿ exp…0 dn23 `42 =b32 †Š exp ÿ0 ngn2 ‡ iˆ4

…`43 > 0;

`42 < 0†

plus (

"

exp ÿ0 ngn2 ‡

nÿ1 X iˆ4

#) ‰1 ÿ exp…ÿ0 dn23 `4 =bn3 †Š

xi dn2i

…`43 < 0;

`4 > 0†

from (42) ÿ‰0 d213 Šÿ1 multiplied by ( " #) nÿ1 X xi d21i ‰exp…0 d213 `42 =b32 † exp ÿ0 ng21 ‡ iˆ4

ÿ exp…0 d213 `41 =b31 †Š

…`43 > 0;

`42 < 0†

plus (

"

exp ÿ0 ng21 ‡

nÿ1 X iˆ4

#) xi d21i

‰1 ÿ exp…0 d213 `41 =b31 †Š …`41 < 0;

`42 > 0†

From the ®rst of these sets of results we get three terms (ignoring the factor ‰0 Šÿ1 which occurs in all terms):

Exact properties of the MLE ÿ1 dn13 multiplied by: ( "

‡ exp ÿ0 ngn1 ‡ (

"

ÿ exp ÿ0 ng31 ‡ (

"

ÿ exp ÿ0 ngn3 ‡

nÿ1 X iˆ4 nÿ1 X iˆ4 nÿ1 X iˆ4

99

#) xi dn1i

…`41 < 0;

`4 > 0†

…43†

…`43 > 0;

`41 < 0†

…44†

…`43 < 0;

`4 > 0†:

…45†

…`42 < 0;

`4 > 0†

…46†

…`43 > 0;

`42 < 0†

…47†

…`43 < 0;

`4 > 0†:

…48†

…`41 < 0;

`42 > 0†

…49†

…`43 > 0;

`42 < 0†

…50†

…`43 > 0;

`41 < 0†: …51†

#) xi d31i #) xi dn3i

From the second group we get: ÿ1 multiplied by: ÿdn23 ( "

‡ exp ÿ0 ngn2 ‡ (

"

ÿ exp ÿ0 ng32 ‡ (

"

ÿ exp ÿ0 ngn3 ‡

nÿ1 X iˆ4 nÿ1 X iˆ4 nÿ1 X iˆ4

#) xi dn2i #) xi d32i #) xi dn3i

Finally, from the third group we get: ÿ1 multiplied by: ÿd213 ( "

‡ exp ÿ0 ng21 ‡ (

"

‡ exp ÿ0 ng32 ‡ (

"

ÿ exp ÿ0 ng31 ‡

nÿ1 X iˆ4 nÿ1 X iˆ4 nÿ1 X iˆ4

#) xi d21i #) xi d32i #) xi d31i

To simplify the summary of these results, write ( " #) nÿ1 X …k† xi dnsi ; k ˆ 3; . . . ; n ÿ 1; s < k; f ns ˆ exp ÿ0 ngns ‡ iˆk

…52† and, for j > s,

100

Grant Hillier and Ray O'Brien

( fjs…k†

"

ˆ exp ÿ0 ngjs ‡

nÿ1 X

#) xi djsi

;

k ˆ 3; . . . ; n ÿ 1; s < k:

iˆk

…53† From (40)±(42), the result after integrating out x2 may, with this notation, be expressed as: …3† …3† …3† ‰0 dn12 Šÿ1 fÿf21 ‡ fn1 ÿ fn2 g;

…54†

…3† , f`31 < 0; `3 > 0g, for with respective regions of integration: for fn1 …3† …3† fn2 ; f`32 < 0; `3 > 0g, and for f21 ; f`32 > 0; `31 < 0g. Likewise, from (43)±(51), the result after integrating out x3 may be expressed in the form: …4† ÿ1 …4† ÿ1 …4† ÿ1 ÿ1 ÿ1 …4† fn1 ÿ dn23 fn2 ÿ ‰dn13 ÿ dn23 Š fn3 ÿ d213 f21 ‰20 dn12 Šÿ1 fdn13 …4† …4† ÿ1 ÿ1 ÿ1 ÿ1 ÿ‰dn13 ÿ d213 Š f31 ÿ ‰d213 ÿ dn23 Š f32 g;

…55†

with subsequent regions of integration: for fns…4† ;

f`4s < 0; `4 > 0g;

s ˆ 1; 2; 3;

fjs…4† ;

f`4j > 0; `4s < 0g;

j ˆ 2; 3;

for

s < j:

Evidently, so long as …k ÿ 1† < p (recall that p is the ®rst value of i for which bi ˆ zi ÿ z > 0), the result of integrating out x2 ; . . . ; xkÿ1 will be a linear combination of k…k ÿ 1†=2 terms, the …k ÿ 1† terms fns…k† ; s ˆ 1; . . . ; k ÿ 1; together with the …k ÿ 1†…k ÿ 2†=2 terms fjs…k† ; j ˆ 2; . . . ; …k ÿ 1†; s ˆ 1; . . . ; … j ÿ 1†, each term to be subsequently integrated over a different region for …xk ; . . . ; xnÿ1 †. As in the one-parameter case, the coef®cients in this linear combination can be generated recursively. To deduce the transition rules for the recursion, write the result of integrating out x2 ; . . . ; xkÿ1 (assuming k ÿ 1 < p) in the form: kÿ1 X sˆ1

…k† a…k† ns fns ‡

kÿ1 X rÿ1 X rˆ2 sˆ1

…k† …k† ars frs ;

…56†

with regions of subsequent integration: for fns…k† : …`ks < 0;

`k > 0†

for frs…k† : …`kr > 0;

`ks < 0†:

We proceed now to integrate out xk . There are two cases to consider: (i) the case k < p, and (ii) the case k ˆ p.

Exact properties of the MLE

101

Case (i): k < p We begin with the ®rst sum in (56). Isolating xk in the inequalities …`ks < 0; `k > 0†, we ®nd that xk must satisfy both of the inequalities: xk < ÿ`k‡1;s =bks

and

xk < `k‡1 =bnk :

Using the identities (28) and (29), the difference between these upper bounds is `k‡1 =bnk ‡ `k‡1;s =bks ˆ …bns =…bnk bks ††`k‡1;k : The xk -integral of an fns…k† term in (56) therefore splits into a sum of two terms (each multiplied by fns…k‡1† †: … ÿ`k‡1;s =bks   exp…ÿ0 xk dnsk †dxk ˆ ‰0 dnsk Šÿ1 1 ÿ exp…0 dnsk `k‡1;s =bks † 0

…if `k‡1;k > 0;

`k‡1;s < 0†;

…57†

plus … `k‡1 =bnk 0

  exp…ÿ0 xk dnsk †dxk ˆ ‰0 dnsk Šÿ1 1 ÿ exp…0 dnsk `k‡1 =bnk † …if `k‡1;k < 0;

`k‡1 > 0†: …58†

As before, the sum of the two equal terms fns…k‡1† to be integrated over the disjoint regions …`k‡1;k > 0; `k‡1;s < 0† and …`k‡1;k < 0; `k‡1 > 0† is simply the integral over the union of those regions, i.e. over the region …`k‡1;s < 0; `k‡1 > 0†. Using the identities (28) and (29) again, we see that …k‡1† ; ks

f

…k‡1† ns

exp…0 `k‡1;s dnsk =bks † ˆ f

f

…k‡1† ns

exp…ÿ0 `k‡1 dnsk =bnk † ˆ f

and …k‡1† : nk

Hence, integration of the ®rst sum in (56) yields the sum: ÿ1 0

kÿ1 X ‰ansk =dnsk Šf fns…k‡1† ÿ f sˆ1

…k‡1† ks

ÿf

…k‡1† g; nk

with regions of subsequent integration: for fns…k‡1† : f`k‡1;s < 0;

`k‡1 > 0g;

for fks…k‡1† : f`k‡1;k > 0;

`k‡1;s < 0g;

for

…k‡1† fnk

: f`k‡1;k < 0;

`k‡1 > 0g:

…59†

102

Grant Hillier and Ray O'Brien

Consider now the second sum in (56). Isolating xk in the inequalities …`kr > 0; `ks < 0† gives: xk > ÿ`k‡1;r =bkr

and xk < ÿ`k‡1;s =bks :

The difference between the upper and lower limits is `k‡1;r =bkr ÿ `k‡1;s =bks ˆ …brs =…bkr bks ††`k‡1;k : The integral vanishes, of course, if this is non-positive, i.e. if `k‡1;k  0: We again get a sum of two terms (each to be multiplied by frs…k‡1† †: … ÿ`k‡1;s =bks exp…ÿ0 xk drsk †dxk ÿ`k‡1;r =bkr

ˆ ‰0 drsk Šÿ1 ‰exp…0 `k‡1;r drsk =bkr † ÿ exp…0 `k‡1;s drsk =bks †Š (if `k‡1;r < 0; plus

… ÿ`k‡1;s =bks 0

`k‡1;k > 0†

…60†

`k‡1;s < 0†:

…61†

exp…ÿ0 xk drsk †dxk

  ˆ ‰0 drsk Šÿ1 1 ÿ exp…0 `k‡1;s drsk =bks † (if `k‡1;r < 0;

Again using the identities (28) and (29) we ®nd that: frs…k‡1† exp…0 `k‡1;r drsk =bkr † ˆ fkr…k‡1† ; and frs…k‡1† exp…0 `k‡1;s drsk =bks † ˆ fks…k‡1† : The two equal terms in (60) and (61) combine as usual to give the term fks…k‡1† , to be integrated over f`k‡1;k > 0; `k‡1;s < 0g. Hence the second sum in (56) becomes, after integrating out xk , ÿ1 0

kÿ1 X rÿ1 X …k† ‰ars =drsk Šf frs…k‡1† ‡ fkr…k‡1† ÿ fks…k‡1† g rˆ2 sˆ1

with regions of subsequent integration: for frs…k‡1† : f`k‡1;r > 0;

`k‡1;s < 0g;

for

fkr…k‡1†

: f`k‡1;r < 0;

`k‡1;k > 0g;

for

fks…k‡1†

: f`k‡1;s < 0;

`k‡1;k > 0g:

…62†

Exact properties of the MLE

103

Case (ii): k ˆ p In the case k ˆ p, `p‡1;p cannot be negative, so only (57) occurs when xp is integrated out in the ®rst term of (56). We therefore get, in place of (59), ÿ1 0

pÿ1 X …p‡1† ‰a…p† ÿ fps…p‡1† g; ns =dnsp Šf fns

…63†

sˆ1

with both terms to be integrated over the region (for xp‡1 ; . . . ; xnÿ1 † determined by the single condition `p‡1;s < 0: Notice that the term fnp…p‡1† does not appear in (63). Turning to the second term in (56), equations (60) and (61) apply with k ˆ p, but the condition `p‡1;p > 0 in (60) is automatically satis®ed. Hence the only change needed for this case is that the regions of subsequent integration of the terms fpr…p‡1† and fps…p‡1† in (62) are determined by the single inequalities `p‡1;r < 0 and `p‡1;s < 0; respectively. Combining these results, the result of integrating out xk in (56) is, apart from the factor ‰0 Šÿ1 , for case (i): k < p kÿ1 X sˆ1

…k‡1† ‰a…k† ÿ fks…k‡1† ÿ fnk…k‡1† g ns =dnsk Šf fns kÿ1 X rÿ1 X …k‡1† ‰a…k† ‡ fkr…k‡1† ÿ fks…k‡1† g rs =drsk Šf frs

‡

…64†

rˆ2 sˆ1

for case (ii): k ˆ p pÿ1 X …p‡1† ‰a…p† ÿ fps…p‡1† g ns =dnsp Š ffns sˆ1

‡

pÿ1 X rÿ1 X …p‡1† ‰a…p† ‡ fpr…p‡1† ÿ fps…p‡1† g: …65† rs =drsp Šf frs rˆ2 sˆ1

Identifying (64) with the analogue of (56): k X sˆ1

a…k‡1† fns…k‡1† ‡ ns

k X rÿ1 X rˆ2 sˆ1

a…k‡1† frs…k‡1† rs

…66†

…k‡1† …k‡1† …k‡1† the `new' terms are fnk …in the ®rst sum) and fk1 ; . . . ; fk;kÿ1 (in the …k‡1† occurs second), a total of k new terms. Now, for s ˆ 1; . . . ; k ÿ 1; fns only in the ®rst line of (64). Hence, if k < p,

a…k‡1† ˆ a…k† ns ns =dnsk ;

s ˆ 1; . . . ; …k ÿ 1†:

…67†

104

Grant Hillier and Ray O'Brien

…k‡1† Also, the term fnk occurs only in the ®rst line of (64), so that

a…k‡1† ˆÿ nk

kÿ1 X sˆ1

‰a…k† ns =dnsk Š:

…68†

In the second line of (64), the terms frs…k‡1† with r < k occur with coef®cients a…k† rs =drsk , so that ˆ a…k† a…k‡1† rs rs =drsk ; The new terms fkj…k‡1† ; coef®cients ÿa…k† ns =dnsk , term with coef®cients ‡

r ˆ 2; . . . ; …k ÿ 1†;

s ˆ 1; . . . ; …r ÿ 1†:

…69†

j ˆ 1; . . . ; k ÿ 1; occur in the ®rst line of (64) with and twice in the second line of (64), in the second

jÿ1 h i X a…k† js =djsk ;

… j > 1†

sˆ1

and in the third term with coef®cients ÿ

kÿ1 h X rˆj‡1

i arj…k† =drjk ;

… j < k ÿ 1†:

Hence the coef®cients of the terms fkj…k‡1† in (64) are: …k‡1†

ak1

…k†

ˆ ÿan1 =dn1k ÿ

ˆ ÿa…k† a…k‡1† nj =dnjk ‡ kj

kÿ1 X …k† ‰ar1 =dr1k Š; rˆ2

jÿ1 kÿ1 X X ‰a…k† ‰a…k† js =djsk Š ÿ rj =drjk Š; rˆj‡1

sˆ1

j ˆ 2; . . . ; k ÿ 2; …k† a…k‡1† k;kÿ1 ˆ ÿan;kÿ1 =dn;kÿ1;k ‡

kÿ2 X sˆ1

…k† ‰akÿ1;s =dkÿ1;s;k Š:

…70†

Equations (67)±(70) specify the recursive relations between the coef®cients in equation (56) up to the integration with respect to xpÿ1 . For the next step, integration with respect to xp , we need to identify (65) with (66) (with k replaced by p†. This gives: …p‡1† ans ˆ a…p† ns =dnsp ;

s ˆ 1; . . . ; p ÿ 1;

…71†

…p‡1† ˆ 0; anp …p‡1† ˆ a…p† ars rs =drsp ;

…72† r ˆ 2; . . . ; p ÿ 1;

s ˆ 1; . . . ; r ÿ 1;

…73†

Exact properties of the MLE …p‡1† ap1 ˆ ÿa…p† n1 =dn1p ÿ

…p‡1† …p† apj ˆ ÿanj =dnjp ‡

pÿ1 X rˆ2

105

‰a…p† r1 =dr1p Š

pÿ1 jÿ1 X X …p† …p† ‰ajs =djsp Š ÿ ‰arj =drjp Š; rˆj‡1

sˆ1

j ˆ 2; . . . ; p ÿ 2 …p‡1† ˆ ÿa…p† ap;pÿ1 n;pÿ1 =dn;pÿ1;p ‡

pÿ2 X ‰a…p† pÿ1;s =dpÿ1;s;p Š:

…74†

sˆ1

Thus, after integrating out xp , we shall have an expression: pÿ1 X sˆ1

…p‡1† …p‡1† ans fns ‡

pÿ1 X rÿ1 X rˆ2 sˆ1

…p‡1† …p‡1† ars frs ‡

pÿ1 X sˆ1

…p‡1† …p‡1† aps fps

…75†

with regions of subsequent integration: for fns…p‡1† : `p‡1;s < 0; for frs…p‡1† : f`p‡1;r > 0; for fps…p‡1† : `p‡1;s < 0;

`p‡1;s < 0g

…r  p ÿ 1†;

s ˆ 1; . . . ; p ÿ 1:

In general, after integrating out xkÿ1 , with p < k ÿ 1 < n ÿ 1; we shall have an expression of the form: pÿ1 X sˆ1

…k† a…k† ns fns ‡

pÿ1 X rÿ1 X rˆ2 sˆ1

…k† …k† ars frs ‡

pÿ1 kÿ1 X X rˆp sˆ1

…k† a…k† rs frs

…76†

with regions of subsequent integration: for fns…k† : `ks < 0;

s ˆ 1; . . . ; p ÿ 1;

for frs…k† : `kr > 0;

`ks < 0;

for frs…k† : `ks < 0;

r ˆ p; . . . ; n ÿ 2;

r ˆ 2; . . . ; p ÿ 1;

s ˆ 1; . . . ; r ÿ 1;

s ˆ 1; . . . ; p ÿ 1:

Integration of (76) with respect to xk then yields (apart from the factor ÿ1 0 †:

106

Grant Hillier and Ray O'Brien pÿ1 X …k‡1† ‰a…k† ÿ fks…k‡1† g ns =dnsk Šf fns sˆ1

‡

pÿ1 X rÿ1 X …k‡1† ‰a…k† ‡ fkr…k‡1† ÿ fks…k‡1† g rs =drsk Šf frs rˆ2 sˆ1

‡

pÿ1 kÿ1 X X …k‡1† ‰a…k† ÿ fks…k‡1† g: rs =drsk Šf frs

…77†

rˆp sˆ1

Identifying this with the analogue of (76) with k ÿ 1 replaced by k: pÿ1 X sˆ1

a…k‡1† fns…k‡1† ‡ ns

pÿ1 X rÿ1 X rˆ2 sˆ1

a…k‡1† frs…k‡1† ‡ rs

pÿ1 k X X rˆp sˆ1

a…k‡1† frs…k‡1† ; rs

…78†

the only additional terms are the p ÿ 1 terms fks…k‡1† ; s ˆ 1; . . . ; p ÿ 1: Comparison of (77) with (78) yields the recursive relations for the coef®cients for terms beyond the p'th, but with k < n ÿ 1: a…k‡1† ˆ a…k† ns ns =dnsk ;

s ˆ 1; . . . ; p ÿ 1;

ˆ a…k† a…k‡1† rs rs =drsk ;

r ˆ 2; . . . ; p ÿ 1;

s ˆ 1; . . . ; r ÿ 1;

…80†

ˆ a…k† a…k‡1† rs rs =drsk ;

r ˆ p; . . . ; k ÿ 1;

s ˆ 1; . . . ; p ÿ 1;

…81†

a…k‡1† ˆ ÿa…k† k1 k1 =dn1k ÿ a…k‡1† ˆ ÿa…k† kj kj =dnjk ÿ ‡

jÿ1 X sˆ1

pÿ1 X ‰a…k† r1 =dr1k Š; rˆ2

pÿ1 X rˆj‡1

‰a…k† js =djsk Š;

‰a…k† rj =drjk Š j ˆ 2; . . . ; p ÿ 2

…k† a…k‡1† k;pÿ1 ˆ ÿak;pÿ1 =dn;pÿ1;k ‡

a…k‡1† ks

ˆ 0;

…79†

s > p ÿ 1:

pÿ2 X ‰a…k† pÿ1;s =dpÿ1;s;k Š; sˆ1

…82†

Note that each integration with respect to an xk , with k > p ÿ 1; adds only p ÿ 1 terms to the sum, not k. Consider now the integral with respect to the last variable, xnÿ1 . We assume that p < n ÿ 2; slight modi®cations of what follows are needed in

Exact properties of the MLE

107

the cases p ˆ n ÿ 1 or p ˆ n, but since these cases are unlikely in practice we omit those details. We need to integrate (78) (with k replaced by n ÿ 2† with respect to xnÿ1 . For the terms fns…nÿ1† the range of integration is `nÿ1;s < 0; or nbs ‡ xnÿ1 bnÿ1;s < 0; so that xnÿ1 < ÿnbs =bnÿ1;s ;

s ˆ 1; . . . ; p ÿ 1:

Hence, integration of these terms with respect to xnÿ1 yields the sum: ÿ1 0

pÿ1 X …nÿ1† ‰ans =dns;nÿ1 Šffns ÿ fnÿ1;s g;

…83†

sˆ1

where here and below we set frs ˆ expfÿn0 grs g. For the second sum in (78), the inequalities `nÿ1;r > 0 and `nÿ1;s > 0 give xnÿ1 < ÿnbr =bnÿ1;r and xnÿ1 > ÿnbs =bnÿ1;s . The difference between the upper and lower limits here is a positive multiple of bnÿ1 , and is thus positive unless p ˆ n, which we rule out. Since the lower limit is certainly positive (because bs < 0 when s < p ÿ 1†, integration with respect to xnÿ1 yields: ÿ1 0

pÿ1 X rÿ1 X …nÿ1† ‰ars =drs;nÿ1 Šf fnÿ1;r ÿ fnÿ1;s g:

…84†

rˆ2 sˆ1

Note particularly that (84) yields no terms frs with r < p. Finally, the third sum in (78) yields ÿ1 0

pÿ1 nÿ2 X X …nÿ1† ‰ars =drs;nÿ1 Šf frs ÿ fnÿ1;s g:

…85†

rˆp sˆ1

Hence, after integrating out the ®nal variable xnÿ1 , we shall have a linear combination of …n ÿ p ‡ 1†…p ÿ 1† terms: frs ˆ expfÿn0 grs g;

r ˆ p; . . . ; n;

s ˆ 1; . . . ; p ÿ 1:

…86†

If we write this linear combination in the form: pÿ1 n X X rˆp sˆ1

ars frs ;

…87†

…nÿ1† : equations (79)±(85) yield the relations between the ars and the ars …nÿ1† =drs;nÿ1 ; ars ˆ ars

r ˆ p; . . . ; n ÿ 2;

…nÿ1† anÿ1;1 ˆ ÿan1 =dn1;nÿ1 ÿ

s ˆ 1; . . . ; p ÿ 1;

nÿ2 X …nÿ1† ‰ar1 =dr1;nÿ1 Š; rˆ2

…88† …89†

108

Grant Hillier and Ray O'Brien …nÿ1† anÿ1;s ˆ ÿans =dns;nÿ1 ‡

ÿ

nÿ2 X

sÿ1 X ‰asj…nÿ1† =dsj;nÿ1 Š jˆ1

…nÿ1† ‰ars =drs;nÿ1 Š;

s ˆ 2; . . . ; p ÿ 1;

…90†

rˆs‡1

…nÿ1† ans ˆ ans =dns;nÿ1 ;

s ˆ 1; . . . ; p ÿ 1:

…91†

As in the single-parameter case, these recursive relations can be expressed in terms of a product of matrices of increasing dimension. To do so, ®rst assume that k < p, and let ck be the ‰k…k ‡ 1†=2Š  1 vector of coef®cients of the frs…k‡1† after integrating out xk , with an analogous de®nition of ckÿ1 (which, of course, is ‰k…k ÿ 1†=2Š  1†. Assume that the elements of ck are arranged in lexicographic order, i.e. in the order (for k < p†: ……21†; …31†; …32†; . . . ; …k1†; …k2†; . . . ; …k; k ÿ 1†; …n1†; . . . ; …nk††: In the case k  p, only pairs (rs) with s  p ÿ 1 occur. In the transition from ckÿ1 to ck …k < p† the terms ……k1†; . . . ; …k; k ÿ 1†† and …nk† are added. Let Lk denote the ‰k…k ‡ 1†=2Š  ‰k…k ÿ 1†=2Š matrix that takes ckÿ1 to ck : ck ˆ Lk ckÿ1 . From the results in (67)±(70), the structure of the matrix Lk , for k < p, is as follows: 2 3 0 Lk11 6 7 6L 7 6 k21 Lk22 7 7; …92† Lk ˆ 6 6 7 6 0 Lk32 7 4 5 0

Lk42

where ÿ1 ÿ1 ÿ1 ÿ1 ÿ1 Lk11 ˆ diagfd21k ; d31k ; d32k ; . . . ; dkÿ1;1k ; . . . ; dkÿ1;kÿ2;k g

…93†

is a ‰…k ÿ 1†…k ÿ 2†=2Š  ‰…k ÿ 1†…k ÿ 2†=2Š diagonal matrix, ÿ1 ÿ1 ; . . . ; dn;kÿ1;k g Lk32 ˆ diagfdn1k

…94†

is a …k ÿ 1†  …k ÿ 1† diagonal matrix, Lk22 ˆ ÿLk32 ; ÿ1 ÿ1 Lk42 ˆ …ÿdn1k ; . . . ; ÿdn;kÿ1;k †

…95†

is a 1  …k ÿ 1† vector, and Lk21 is a …k ÿ 1†  ‰…k ÿ 1†…k ÿ 2†=2Š matrix with the following structure:

Exact properties of the MLE

109

for j ˆ 1; . . . ; …k ÿ 1†, the non-zero elements in row j are, in their lexÿ1 ÿ1 for r > j, and the terms djsk for icographic positions, the terms ÿdrjk s ˆ 1; . . . ; j ÿ 1: Row 3, for instance, is: ÿ1 ÿ1 ÿ1 ÿ1 ÿ1 f0; d31k ; d32k ; 0; 0; ÿd43k ; 0; 0; ÿd53k ; . . . ; ÿdkÿ1;3;k ; 0; . . . ; 0g:

For p  k < n ÿ 1 the following modi®cations to Lk must be made: (1) ÿ1 ÿ1 ; . . . ; dn;pÿ1;k g is …p ÿ 1†  the last row is absent; (2) Lk32 = diagfdn1k …p ÿ 1†, and hence so is Lk22 ˆ ÿLk32 ; (3) in Lk11 , the diagonal terms ÿ1 drsk for r > p appear only for s ˆ 1; . . . ; …p ÿ 1†, so that Lk11 is square of dimension ‰p…p ÿ 1†=2 ‡ …k ÿ p†…p ÿ 1†Š, and, correspondingly, (4) Lk21 is now …p ÿ 1†  ‰p…p ÿ 1†=2 ‡ …k ÿ p†…p ÿ 1†Š, with the same structure as ÿ1 above except that the drsk that occur are for r ˆ 1; . . . ; …p ÿ 1† only. Hence, for p  k < n ÿ 1; Lk is ‰p…p ÿ 1†=2 ‡ …k ÿ p ‡ 1†…p ÿ 1†Š  ‰p…p ÿ 1†=2 ‡ …k ÿ p ‡ 2†…p ÿ 1†Š: Finally, the matrix Lnÿ1 is ‰…n ÿ p ‡ 1†…p ÿ 1†Š  ‰…n ÿ p†…p ÿ 1† ‡ p…p ÿ 1†=2Š with the following structure: 2 3 Lnÿ1;11 0 6 7 7 Ln ˆ 6 4 Lnÿ1;21 Lnÿ1;22 5 0 Lnÿ1;32 ÿ1 ; s ˆ 1; . . . ; p ÿ 1g is …p ÿ 1†  …p ÿ 1†; where Lnÿ1;32 ˆ diagfdns;nÿ1 Lnÿ1;22 ˆ ÿLnÿ1;32 ; Lnÿ1;21 is …p ÿ 1†  ‰…n ÿ p ÿ 1†…p ÿ 1† ‡ p…p ÿ 1†=2Š with the same structure as in the case p  k < n ÿ 1 above, and Lnÿ1;11 is …n ÿ p ÿ 1†…p ÿ 1†  ‰…n ÿ p ÿ 1†…p ÿ 1† ‡ p…p ÿ 1†=2Š with the form:   ÿ1 Lnÿ1;11 ˆ 0; diagfdrs;nÿ1 ; r ˆ p; . . . ; n ÿ 2; s ˆ 1; . . . ; p ÿ 1g ;

where the initial block of zeros is …n ÿ p ÿ 1†…p ÿ 1†  ‰…p ÿ 1†…p ÿ 2†=2Š. The ®nal vector cnÿ1 , of dimension ‰…n ÿ p ‡ 1†…p ÿ 1†Š  1; is then given by the recursive formula: cnÿ1 ˆ Lnÿ1 Lnÿ2 . . . L2 ; starting with

0

ÿ1

…96†

1

C B ÿ1 B C L2 ˆ dn12 @ ‡1 A ÿ1 (see (54) above). Letting fnÿ1 denote the vector of functions

…97†

110

Grant Hillier and Ray O'Brien

frs ˆ expfÿn0 ‰br as ÿ bs ar Š=brs g; ordered lexicographically as above, we have:  ÿ †g…@=@w† pdfA;T …a; t; ; † ˆ …n0 bÿ1 n1 † expfnz…t 0  ‰expfÿn0 wz2 gcnÿ1 fnÿ1 Šwˆ0 :

It remains now to evaluate the differential operator, and set w ˆ 0: The results are exactly analogous to those for the single-parameter case given earlier. First we de®ne c~nÿ1 ˆ cnÿ1 jwˆ0 ˆ L~ nÿ1 L~ nÿ2 . . . L~ 2 ;

…98†

where the L~ p are de®ned exactly as the Lp are de®ned above, but with dijk replaced by d~ijk ˆ ‰bij rk ÿ bkj ri ÿ bik rj Š=bij :

…99†

Before proceeding we note that, at the point t ˆ ; ri ˆ 1 for all i, and d~ijk ˆ 0; so that the L~ r are not de®ned at t ˆ . The results that follow therefore hold everywhere except at t ˆ . At the point t ˆ  we have, directly from (23), pdfA;T …a; t ˆ ; ; † ˆ n0 expfÿn0 g  cn …z†;

…100†

where cn …z† is a constant. (100) follows from (23) when t ˆ  because, in the integrand of (23), the exponential term becomes ( ) n X exp ÿ0 xi ˆ expfÿn0 g iˆ1

P on S (since, on S; niˆ1 xi ˆ n). The integral is then a function only of z ˆ …z1 ; . . . ; zn † 0 and n. Since (100) is proportional to the conditional density of A given that T ˆ , which must integrate to one, we also obtain an expression for the density of T at t ˆ : pdfT …t ˆ ; † ˆ nÿn ÿ…n†cn …z†:

…101†

The constant cn …z† in (100) and (101) can be evaluated by methods like those above, but we omit these details. Next, let f~nÿ1 be de®ned as fnÿ1 , which has been de®ned above, but with the frs replaced by f~rs ˆ expfÿn0 ‰br rs ÿ bs rr Š=brs g;

…102†

 to be the vector with elements and de®ne f~nÿ1

…@=@w†‰expfÿn0 wz2 g frs Š jwˆ0 ˆ ÿn0 br bs f~rs :

…103†

Exact properties of the MLE

111

As before, @cnÿ1 =@w jwˆ0 ˆ

nÿ2 X iˆ1

‰L~ nÿ1 . . . L~ nÿi‡1 . . . L~ 2 Š ˆ c~nÿ1 ; say;

…104†

ÿ1 where L~ p is again de®ned as Lp is de®ned above but with dijk replaced by 2 2 ‰bij z2k ÿ bkj z2i ÿ bik z2j Š=‰bij d~ijk Š ˆ bki bkj =d~ijk :

…105†

Combining these results, we have a relatively simple expression for the density of …A; T†: 0 ~ 0 ~  ÿ †g…n0 bÿ1 pdfA;T …a; t; ; † ˆ expfnz…t n1 †‰c nÿ1 fnÿ1 ‡ c~nÿ1 fnÿ1 Š: …106†

4.1

The marginal density of T

Remarkably, it is straightforward to integrate out a in (106) to obtain the marginal density of T, because the two terms in the ‰
Š in (106) are linear combinations of terms expfÿn0 grs g and …ÿnbr bs 0 † expfÿn0 grs g, respectively, with coef®cients that do not depend on a. Transforming from …a ÿ † to 0 ˆ expf…a ÿ †g > 0; and integrating out 0 we obtain: 0 0   ÿ †gbÿ1 pdfT …t; † ˆ expfnz…t n1 ‰c nÿ1 hnÿ1 ‡ c~nÿ1 hnÿ1 Š;

gÿ1 rs ,

hnÿ1

…107†

ÿbr bs gÿ2 rs .

and has elements Again, the where hnÿ1 has elements density is de®ned by (107) at all points other than t ˆ . At t ˆ  expression (101) must be used. 5

Properties of the exact densities

Because the coef®cients in the exact expressions (22), (106) and (107) are generated recursively it is dif®cult to study the properties of the densities analytically. However, given a choice for the vector z, it is straightforward to analyse the densities numerically, although in the two-parameter case we did have some dif®culty with the numerical stability of the calculations near t ˆ  (see below). 5.1

Properties in the case k ˆ 1

From the remarks in section 2, the density depends only on d ˆ …t ÿ †, has its mode at t ˆ , and the density at d when z > 0 is the density at ÿd with z < 0, so that the density with negative z's is simply the density with positive z's re¯ected about the origin.

112

Grant Hillier and Ray O'Brien

Table 3.1. Means, variances and skewness for ®gure 3.1 n

Mean

Variance

Skewness

2 4 8 16

70.429 70.315 70.227 70.161

1.386 1.216 1.114 1.058

70.841 70.628 70.454 70.323

Figure 3.1 Densities for n ˆ 2; 4; 8 and 16 equispaced points z 0 z ˆ 1

It also follows from the remarks in section 2 that there is no loss of generality in scaling the zi 's so that z 0 z, the (expected) Fisher information for , is unity, corresponding to an asymptotic variance of one. Figure 3.1 shows the density, calculated from equation (22), for the case of equispaced positive z's, scaled so that z 0 z ˆ 1; for the cases n ˆ 2; 4; 8 and 16. Different patterns of z's produce little change in the graphs. In table 3.1 we give the means, variances and skewness for the cases that appear in ®gure 3.1 (calculated by numerical integration). Both ®gure 3.1 and table 3.1 suggest that the approach to the asymptotic distribution of the MLE is quite rapid, that (with positive z's) the estimator is slightly negatively biased, and that the asymptotic variance (ˆ 1 in this example) slightly understates the true variance.

Exact properties of the MLE

5.2

113

Properties for the model with a constant term

In view of the remarks in section 2, the joint density has its mode at the point (a; t† ˆ …; †: Further properties must be derived from the formulae above. Consider ®rst the case n ˆ 3: In this case no recursion is needed, and we have directly from (54): pdfA;T …a; t; ; † ˆ expf3…t ÿ †z ÿ 30 g~ 31 g‰30 bÿ1 31 Š  ‰ÿb21 b32 2 …d~312 † ÿ 0 b1 b3 1 …d~312 †Š

…108†

where 1 …y† ˆ ‰1 ÿ eÿþy Š=y;

…109†

2 …y† ˆ ‰1 ÿ …1 ‡ þy†eÿþy Š=y2 ;

…110†

and with þ ˆ n0 b3 =b32 if p ˆ 3 and þ ˆ ÿn0 b1 =b21 if p ˆ 2: Note that (108) is well de®ned for all …a; t†, including t ˆ , since both 1 and 2 in fact do not involve negative powers of d~312 . The marginal density of T is readily obtained from (108) by direct integration, giving:  ÿ1 pdfT …t; † ˆ expf3…t ÿ †zgb 31  ‰ÿÿ2 b21 b32 g~31 ÿ b1 b3 ÿ…6g~31 ‡ ÿd~312 †Š=‰g~231 …3g~ 31 ‡ ÿd~312 †Š;

…111†

where now ÿ ˆ þ=0 . The marginal density of A does not seem to be obtainable analytically from (108), but is easily obtained from the joint density by numerical integration. Figure 3.2 presents three cases of the marginal density of t in (111) 0:9; 1Š corresponding to vectors z 0 ˆ ‰ÿ1; ÿ0:9; 1Š; ‰ÿ1; 0; 1Š, and P ‰ÿ1; that were subsequently standardised to phave z ˆ 0 and z2i ˆ 1 (so that the asymptotic covariance matrix of ‰ n…a ÿ †; …t ÿ †Š is an identity matrix). The ®rst and third cases, of course, are identical except for re¯ection about the origin. Even for such a small sample size, ®gure 3.2 reveals that the density is quite concentrated around zero, showing slight skewness (depending on the pattern of the z's). Table 3.2 presents some properties of the marginal densities for the case n ˆ 3; and the two (unstandardised) z-vectors (a) ‰ÿ1; ÿ0:9; 1Š; (b) ‰1; 0; 1Š. Figure 3.3 shows the joint density pdfA;T …a; t; ; ; z† for case (b). The marginal density of A, pdfA …a; ; z†, can be obtained by numerical

114

Grant Hillier and Ray O'Brien

Table 3.2. Properties of the marginal densities: n ˆ 3 pdfT …t; †

pdfA …a; †

Case

(a)

(b)

(a)

(b)

Mean Variance Skewness Kurtosis Correlation

70.25 1.53 70.56 4.22 70.03

0.0 1.65 0.0 4.15 0.03

70.64 1.39 70.61 3.64 70.03

70.62 1.38 70.62 3.68 0.03

(a) z 0 ˆ ‰ÿ1; ÿ0:9; 1Š; (b) z 0 ˆ ‰1; 0; 1Š.

Figure 3.2 Marginal densities for t ÿ ; n ˆ 3

integration, and is shown for all three cases in ®gure 3.4 (although cases (a) and (c) are exactly superimposed). It does not exhibit the mirror symmetry of pdfT …t; ; z†, being always negatively skewed. The density for (b) lies slightly to the right. The density of A has variance and kurtosis closer to the asymptotic values than those for T, but the mean and skewness are further from their asymptotic values. The correlation between A and T is negative for case (a), and positive for cases (b) and (c), which is the mirror image of (a). Given the small sample size, the results look well behaved. For n ˆ 4, the marginal density of T is symmetric if the data are symmetric about its mean (e.g. z 0 ˆ ‰0:1; 0:2; 0:3; 0:4Š), and positively skewed z's (e.g. z 0 ˆ ‰0:1; 0:26; 0:33; 0:4Š) give the mirror image of the

Exact properties of the MLE

115

Figure 3.3 Joint density for t ÿ , a ÿ , n ˆ 3

Figure 3.4 Marginal densities for a ÿ , n ˆ 3

density with negatively skewed z's (e.g. z 0 ˆ ‰0:1; 0:17; 0:24; 0:4Š). Accordingly we can illustrate almost the full range of behaviour by using the symmetric z above, and the negatively skewed case z 0 ˆ ‰0:1; 0:11; 0:12; 0:4Š. Table 3.3 shows the resulting moments in its ®rst two columns. For n ˆ 5, the cases z 0 ˆ ‰0:1; 0:2; 0:29; 0:4; 0:5Š, which gives the mirror image of the results with z3 changed to 0:31, and z 0 ˆ ‰0:1; 0:11; 0:12; 0:13; 0:5Š, are illustrated in columns 3 and 4 of table 3.3. The z-vectors given for table 3.3 (and elsewhere) are in their unstandardised form. Progress towards limiting normality is masked by the possibility of greater skewness for n ˆ 5 observations than for n ˆ 4.

116

Grant Hillier and Ray O'Brien

Table 3.3. Moments of the marginal density of T nˆ4

Mean Variance Skewness Kurtosis

nˆ5

(a)

(b)

(a)

(b)

0.00 1.43 0.00 3.48

70.329 1.45 70.56 3.58

70.016 1.40 70.05 3.71

70.393 1.55 70.77 4.22

For n ˆ 4: (a) z 0 ˆ ‰0:1; 0:2; 0:3; 0:4Š, (b) z 0 ˆ ‰0:1; 0:11; 0:12; 0:4Š For n ˆ 5: (a) z 0 ˆ ‰0:1; 0:2; 0:29; 0:4; 0:5Š, (b) z 0 ˆ ‰0:1; 0:11; 0:12; 0:13; 0:5Š

Table 3.4. Marginal density of T. Tail areas: n ˆ 5 and n ˆ 10 Nominal tail area

lower%

upper%

sum%

n ˆ 10, sum%

10 5 1

14.4 10.4 7.0

2.3 0.9 0.3

16.7 11.3 7.3

17.2 12.0 8.7

n ˆ 5: z 0 ˆ ‰0:1; 0:11; 0:12; 0:13; 0:5Š n ˆ 10: z 0 ˆ ‰0:1; 0:11; 0:12; 0:13; 0:14; 0:15; 0:16; 0:17; 0:18; 1:0Š

Because of the numerical instability of the calculations near t ˆ , and because the cases above for small n suggest reasonably rapid convergence to the asymptotic distribution, we now concentrate on the tail area of the densities. Table 3.4 gives the exact tail area in the marginal density of T (obtained by numerical integration) for n ˆ 5 and n ˆ 10, and for three nominal (i.e. asymptotic) levels. If one chooses z in this way, so that, for example, when n ˆ 10, all the observations except the largest are crowded together at the lower end of the range, the skewness increases, and this acts against the effect of increasing n, to leave the tail areas more or less unchanged over the range n ˆ 5; . . . ; 10, as the last column of table 3.4 illustrates. However, for uniformly distributed z's, for example z 0 ˆ ‰0:1; 0:2; 0:3; 0:4; 0:5; 0:6; 0:7; 0:8; 0:9; 1:0Š; the distribution is symmetric, the tail probabilities are equal, and are given in table 3.5. There is evidently steady progress towards the nominal values as n increases, slightly slower the further one is into the tails. For larger values of n the computational dif®culties mentioned above have so

Exact properties of the MLE

117

Table 3.5. Marginal density of T: Tail area (uniformly distributed z's); combined tails Nominal %

1

5

10

nˆ6 nˆ8 n ˆ 10 n ˆ 12

4.7 4.0 3.6 3.3

8.7 7.8 7.3 6.9

14.3 13.3 12.7 12.3

far prevented us from carrying out a more detailed analysis of the densities. 6

Conclusion

We have shown that the surface integral formula for the exact density of the MLE given by Hillier and Armstrong (1996) provides a tractable expression for the exact density in the case of an exponential regression model with a k-covariate exponential mean function, at least for small values of k. It seems clear that an algorithm could, in principle, be written to provide similar results for arbitrary k. The discussion in section 2 also shows that, even for arbitrary k, the general formula can, by itself, provide considerable information about the properties of the exact density. It is also worth noting that the general approach used here extends easily to more general speci®cations for the mean function (i.e. non-exponential functions of the wi ), provided only that the level set of the MLE is known. It will remain true for more general models that the surface integral that needs to be evaluated is over an …n ÿ k†-dimensional hyperplane. Finally, as far as the results for the speci®c model under consideration are concerned, our main conclusion is that the exact densities are well behaved and well approximated by the asymptotic densities, even for quite small sample sizes. The sample behaviour of the covariates certainly has an impact on the properties of the estimator, as one would expect, but this effect is not dramatic. References Hillier, G.H. and M. Armstrong (1996), `On the Density of the Maximum Likelihood Estimator', University of Southampton Discussion Papers in Economics and Econometrics No. 9645; also in Econometrica, 67 (6), 1999: 1459±1470.

118

Grant Hillier and Ray O'Brien

Huber, P.J. (1964), `Robust Estimation of a Location Parameter', Annals of Mathematical Statistics, 35: 73±101. Knight, J.L. and S.E. Satchell (1996), `The Exact Distribution of the Maximum Likelihood Estimator for the Linear-regression Negative-exponential Model', Journal of Statistical Planning and Inference, 1: 91±102. Schechter, M. (1998), `Integration over a Polyhedron: An Application of the Fourier±Motzkin Elimination Method', American Mathematical Monthly, 105: 246±251. Shephard, N. (1993), `Distribution of the ML Estimator of the MA(1) and a Local Level Model', Econometric Theory, 9: 377±401.

4

Empirical likelihood estimation and inference Richard J. Smith

1

Introduction

Since Hansen's (1982) seminal paper, the generalised method of moments (GMM) has become an increasingly important method of estimation in econometrics. Given assumed population moment conditions, the GMM estimation method minimises a quadratic form in the sample counterparts of these moment conditions. The quadratic form is constructed using a positive de®nite metric. If this metric is chosen as the inverse of a positive semi-de®nite consistent estimator for the asymptotic variance matrix of the sample moments, then the resultant GMM estimator is asymptotically ef®cient. Estimation using GMM is semi-parametric and, therefore, a particular advantage of GMM is that it imposes less stringent assumptions than, for example, the method of maximum likelihood (ML). Although consequently more robust, GMM is of course generally less ef®cient than ML. There is some Monte Carlo evidence indicating that GMM estimation may be biased in small samples where the bias seems to arise mainly from the metric used. For example, Altonji and Segal (1996) suggest that GMM estimators using an identity matrix metric may perform better in ®nite samples than an ef®cient GMM estimator obtained using the inverse of the estimated asymptotic variance matrix of the sample moments.1 When the variables of interest in the data-generation process (DGP) are independent and identically distributed, an important recent paper (Qin and Lawless (1994)) shows that it is possible to embed the components of the sample moments used in GMM estimation in a non-parametric likelihood function; namely, an empirical likelihood (EL) 1 See also the special issue of Journal of Business and Economic Statistics (1996), vol. 14, no. 3, for further evidence on the ®nite sample performance of GMM estimators and test statistics.

119

120

Richard J. Smith

framework. The resultant maximum EL estimator (MELE) shares the same ®rst-order properties as Hansen's (1982) ef®cient GMM estimator. Hence, the EL framework provides the possibility of using classical-type methods to examine hypotheses for the parameters of interest. See DiCiccio, Hall and Romano (1989) and Owen (1988, 1990, 1991) for a discussion of the method of EL. Qin and Lawless (1994) also discuss some simulation evidence that indicates that the MELE may represent an improvement over the ef®cient GMM estimator. In the light of Qin and Lawless's (1994) and Altonji and Segal's (1996) Monte Carlo evidence, the EL method offers a potentially useful alternative and asymptotically ®rst-order equivalent approach to GMM estimation. Empirical likelihood appears to have been largely ignored by econometricians. A relevant related paper is that of Back and Brown (1993), which, however, does not make explicit reference to the literature on EL. More recently, Imbens (1997) proposes the use of an estimator, which is essentially the MELE of Qin and Lawless (1994) and associated test statistics. Imbens, Spady and Johnson (1998) use an informationtheoretic approach also for the independently and identically distributed case and suggest an estimator and test statistics which are also asymptotically ®rst-order equivalent to those based on EL (Qin and Lawless (1994)) and GMM (Hansen (1982)). However, the application of EL and related information-theoretic approaches to time-series DGPs is relatively limited. A notable exception is Kitamura and Stutzer (1997), which uses an approach based on an empirical Kullback±Leibler information criterion that yields estimators ®rst-order equivalent to ef®cient GMM estimation. Kitamura and Stutzer's (1997) information-theoretic estimation method specialises to that of Imbens, Spady and Johnson (1998) for independent and identically distributed DGPs. As will be seen below, for time-series DGPs, Kitamura and Stutzer's (1997) method and ef®cient GMM estimation are asymptotically equivalent to estimation based on a suitably smoothed EL function. Smith (1997), by adapting the approach of Chesher and Smith (1997) from the parametric context, introduces a class of semiparametric likelihood criteria which includes the criteria of EL, Kitamura and Stutzer (1997) and Imbens, Spady and Johnson (1998) as special cases. The emphasis in the statistical literature, however, has tended to be on inference for the parameters of interest rather than issues of misspeci®cation. A major concern of this chapter is to discuss misspeci®cation test statistics based on EL criteria for moment conditions. These tests mimic the classical tests in an obvious way. A typical approach in the econometric literature to the construction of misspeci®cation tests is based on

Empirical likelihood

121

using a quadratic form in estimated sample analogues of assumed or implicit population moment conditions, which is examined to ascertain whether or not the sample moments differ signi®cantly from the corresponding theoretical moment conditions (see Newey (1985a, 1985b)). We show that a suitable test, which shares the asymptotic properties of its GMM counterpart, may be based on an empirical likelihood ratio (ELR) statistic. Section 2 brie¯y reviews GMM estimation and introduces EL estimation in the context of time-series data. A modi®ed EL estimation procedure is introduced that results in consistent and asymptotically ef®cient estimators. We also present an ef®cient estimator for the stationary distribution of the time-series DGP, which might prove useful in examining parametric distributional hypotheses. ELR and other classical-type tests for over-identifying moment conditions which have the requisite central chi-squared distribution are obtained in section 3, together with tests for additional moment and parametric restrictions. Section 4 discusses the application of the EL method to testing non-nested hypotheses. Various proofs are given in the Appendix. 2

Empirical likelihood estimation

2.1

Generalised method of moments estimation

Consider the strictly stationary process fxt g1 tˆÿ1 . Suppose there exists a unique value 0 of the p-vector of parameters  which lies in a compact parameter space   Rp such that the following r  p moment conditions hold: E fg…xt ; 0 †g ˆ 0; t ˆ 0; 1; . . . ;

…1†

where E f:g denotes expectation taken with respect to the true but unknown distribution or probability measure . For economy of notation but without loss of generality, we will deliberately suppress any dependence of the moment vector g…xt ; 0 † on lagged values xtÿs , s ˆ 1; 2; . . . ; and current and lagged values of exogenous variables. Given observations fxt gTtˆ1 , the optimal GMM estimator (Hansen (1982)) ^T is de®ned by ^T  arg min T ÿ1 2

T X tˆ1

ÿ1

^T g…xt ; † 0 V

T X tˆ1

g…xt ; †;

…2†

^ T is a positive semi-de®nite consistent estimator for the asympwhere V P totic variance matrix V of the random vector T ÿ1=2 Ttˆ1 g…xt ; 0 † which is assumed to be positive de®nite; namely

122

Richard J. Smith

( V  lim Var T T!1

ÿ1=2

T X tˆ1

) g…xt ; 0 † :

…3†

See inter alia Andrews (1991) and Newey and West (1987) for positive semi-de®nite consistent estimation of asymptotic variance matrices. Under suitable conditions and (1), it may be shown that T 1=2 …^T ÿ 0 † ) N‰0; …G 0 Vÿ1 G†ÿ1 Š; where `)' indicates weak convergence, G  E fr0 g…x; 0 †g, which is assumed to have full column rank, and r  @=@; see Hansen (1982).

2.2

Empirical likelihood estimation

De®ne the space of probability measures P…† ˆ fP : EP fg…x; †g ˆ 0g; where EP f:g denotes expectation taken with respect to the probability measure P, which is assumed absolutely continuous with respect to the measure . Consider obtaining the maximum likelihood estimator for dP…†=d; that is, maximise E fln‰dP…†=dŠg subject to EP f1g ˆ 1 (namely, that P is a probability measure) and EP fg…x; †g ˆ 0.2 It is straightforward to show that the solution is given by dP…†=d ˆ ‰1 ‡  0 g…x; †Šÿ1 , where  is a vector of Lagrange multipliers associated with the constraint EP fg…x; †g ˆ 0. Consequently, it may be seen that dP…†=d ˆ 1, that is, P…† ˆ , if and only if  ˆ 0. Moreover, given the uniqueness of 0 for E fg…x; 0 †g ˆ 0, we have that P…0 † ˆ . Note that dropping the constraint EP fg…x; †g ˆ 0 results in dP…†=d ˆ 1. Given observations fxt gTtˆ1 , the appropriate empirical counterparts to dP…†=d and d are, respectively, dPt …†=dt ˆ ‰1 ‡  0 g…xt ; †Šÿ1 and dt ˆ T ÿ1 , t ˆ 1; . . . ; T. Therefore, dPt …† ˆ T ÿ1 ‰1 ‡  0 g…xt ; †Šÿ1 , t ˆ 1; . . . ; T. For Pt …†, t ˆ 1; . . . ; T, to de®ne a probability PT measure we Palso require dPt …†  0, t ˆ 1; . . . ; T, and tˆ1 dPt …† ˆ T ÿ1 Ttˆ1 ‰1 ‡  0 g…xt ; †Šÿ1 ˆ 1. The empirical measures fdPt …†gTtˆ1 may then be used to construct a semi-parametric likelihood function for the observations fxt gTtˆ1 . Hence, the log empirical likelihood (EL) criterion is given by 2

In this exposition, we ignore the requirement that dP…†=d  0.

Empirical likelihood

ÿT ln T ÿ

T X tˆ1

123

ln‰1 ‡  0 g…xt ; †Š:

This and the above discussion suggest basing an estimation procedure for 0 by ®nding the saddle-point of the criterion ÿT ln Tÿ P T 0 tˆ1 ln‰1 ‡  g…xt ; †Š or, equivalently, the log EL criterion QT …; † 

T X tˆ1

ln‰1 ‡  0 g…xt ; †Š:

…4†

Consequently, the estimatedP vector of Lagrange multipliers T …† associated with the constraint T ÿ1 Ttˆ1 ‰1 ‡  0 g…xt ; †Šÿ1 g…xt ; † ˆ 0 for given  is given by T …† ˆ arg max QT …; †; 

and the corresponding maximum EL estimator (MELE) T by T ˆ arg min max QT …; †: 2



…5†

Denote T  T …T †. Although the consistency of the saddle-point estimator T and the MELE T for 0 and 0 respectively may be shown by a similar method to the Proof of Lemma 1 given below, the MELE T of (5) based on QT …; † of (4) is in general inef®cient relative to the GMM estimator ^T of (2); cf. the Proof of Theorem 1 below. See also Kitamura and Stutzer (1997). In particular, the MELE T and Lagrange multiplier estimator T are asymptotically correlated if the moment indicators fg…xt ; 0 †g1 tˆÿ1 are serially correlated. Consequently, there is additional information from the moment conditions, E fg…xt ; 0 †g ˆ 0, t ˆ 0; 1; . . . ; which is currently ignored and which may usefully be exploited. However, when, for example, the DGP for fxt g1 tˆÿ1 is independently and identically distributed or fg…xt ; 0 †g1 tˆÿ1 is a martingale difference process, then T is asymptotically relatively ef®cient; cf. inter alia Qin and Lawless (1994). In this case, T and T are asymptotically uncorrelated. 2.3

Ef®cient maximum empirical likelihood estimation

The inef®ciency of the MELE T of (5) relative to the GMM estimator ^T of (2) is caused by the log EL criterion function QT …; † of (4) implicitly imposing an inef®cient metric. In effect, QT …; † of (4) ignores any serial correlation among the vectors fg…xt ; †g1 tˆÿ1 . In order to remedy this dif®culty, consider the smoothed moment indicators

124

Richard J. Smith

g!tT …† 

m…T† X

!…s; m…T††g…xtÿs ; †;

sˆÿm…T†

with the weights f!…s; m…T††g acting to take into account any serial correlation among the vectors fg…xt ; †g1 tˆÿ1 ; cf. section 2.2. The weights f!…s; P m…T††g are chosen so that !…s; m…T††  0, s ˆ ÿm…T†; . . . ; m…T†, m…T† and sˆÿm…T† !…s; m…T†† ˆ 1. We de®ne the lead and lag truncation parameter m…T† such that m…T† ˆ o…T 1=2 †; cf. Andrews (1991). Moreover, we choose the weights f!…s; m…T††g such that E fg!tT …†g ˆ 0 , E fg…xt ; †g ˆ 0; t ˆ 0; 1; . . .. Hence, we may reformulate the moment conditions (1) equivalently as E fg!tT …0 †g ˆ 0; t ˆ 0; 1; . . .

…6†

Kitamura and Stutzer (1997, equation (8), p. 865) propose the weights !…s; m…T†† ˆ 1=…2m…T† ‡ 1†, s ˆ ÿm…T†; . . . ; m…T†, which give rise to the Bartlett kernel in the estimation of the variance matrix V of (3); see below (10). The smoothed log EL criterion function corresponding to the rede®ned moment conditions (6) is given by RT …; † 

T X tˆ1

ln‰1 ‡  0 g!tT …†Š:

…7†

The criterion RT …; † of (7) may be thought of as being constructed by rede®ning the empirical measures fdPt …†gTtˆ1 as t …; †  T ÿ1 ‰1 ‡  0 g!tT …†Šÿ1 ; t ˆ 1; . . . ; T;

…8†

cf. section 2.2.3 P Note that it is necessary to ensure t …; †  0, t ˆ 1; . . . ; T, and Ttˆ1 t …; † ˆ 1.4 Smoothing the moment indicators 3

Consider the following optimisation P problem: maximise with respectPto dPt …†, t ˆ 1; . . . ; T, the empirical likelihood Ttˆ1 lnt …† subject to the restrictions Ttˆ1 dPt …† ˆ 1, dPt …†  0, t ˆ 1; . . . ; T, and the empirical moment condition T X dPt …†g!tT …† ˆ 0: tˆ1

Ignoring for presentational purposes the inequality constraints dPt …†  0, t ˆ 1; . . . ; T, the Lagrangean is þ ! T T T X X X ln dPt …† ‡  0 dPt …†g!tT …† ‡ ' 1 ÿ dPt …† : tˆ1

tˆ1

tˆ1

It is straightforwardly shown that the solution is given by dPt …† ˆ T ÿ1 ‰1 ‡  0 g!tT …†Š; t ˆ 1; . . . ; T: P Alternatively, the rede®nition t …; †= Ttˆ1 t …; †, t ˆ 1; . . . ; T, guarantees summation to unity. 4

Empirical likelihood

125

fg…xt ; †gTtˆ1 using the weights f!…s; m…T††g to take account of their potential serial correlation renders the implicit metric imposed by the smoothed log EL criterion of (7) equal to the required metric V of (3). In a likewise manner to that of section 2.2, the Lagrange multiplier estimator ^T …† for given  is de®ned by ^T …†  arg max RT …; †; 

and the corresponding ef®cient maximum empirical likelihood estimator (EMELE) ^T by ^T  arg min max RT …; †; 2



…9†

write ^T  ^ T …^T †.5 We have deliberately used the same notation ^T in (9) for the EMELE as that for the optimal GMM estimator of (2) because, as will be seen in Theorem 1 below, the EMELE (9), although not numerically identical, shares the same ®rst-order asymptotic properties as the ef®cient GMM estimator of (2). We may prove Lemma 1 (Consistency of the EMELE ^T and Lagrange multiplier estimator ^T .) Under (1), the EMELE ^T of (9) is a consistent estimator for 0 and ^T !P 0. Pm…T† !…s; m…T††2 , we may prove the following De®ning …T†  sˆÿm…T† theorem, which demonstrates the ®rst-order equivalence of the EMELE ^T with the optimal GMM estimator of (2). Theorem 1 (Limit distribution of the EMELE ^T and Lagrange multiplier estimator ^ T .) Under (1), 0 T 1=2 @

…T†^T

1

A ^T ÿ 0 0þ ! 0 11 0 Vÿ1 ÿ Vÿ1 G…G 0 Vÿ1 G†ÿ1 G 0 Vÿ1 00 AA: ) N@ ;@ 0 0 …G 0 Vÿ1 G†ÿ1

The empirical measures t …^T ; ^T †, t ˆ 1; . . . ; T, may be regarded as implied probabilities for the stationary distribution of fxt g1 tˆÿ1 (see Back and Brown (1993)). See section 2.4.

5

126

Richard J. Smith

From Theorem 1, we see that the Lagrange multiplier estimator ^T is asymptotically uncorrelated with the EMELE ^T , which ensures that the moment conditions (1) are used ef®ciently in constructing the EMELE; see section 2.2. In the course of the proof of Theorem 1, we require that VT 

2m…T† X

! …s; m…T††C…s† !P V;

sˆÿ2m…T†

P where the (infeasible) sample autocovariances C…s†  T ÿ1 Ttˆs‡1 g…xt ; 0 † g…xtÿs ; 0 † 0 , s  0, and C…ÿs† ˆ C…s† 0 , s ˆ ÿ2m…T†; . . . ; 2m…T†. The weights f! …s; m…T††g obey the relations ! …s; m…T††  …1=…T††

m…T† X

!…r ‡ s; m…T††!…r; m…T††; s  0;

rˆÿm…T†ÿs

 …1=…T††

m…T†ÿs X

!…r ‡ s; m…T††!…r; m…T††; s  0:

rˆÿm…T†

…10† 





Note that ! …s; m…T††  0, ! …0; m…T†† ˆ 1 and ! …s; m…T†† ˆ ! …ÿs; m…T††, s ˆ ÿ2m…T†; . . . ; 2m…T†. Moreover, we require that the boundedness condition of Andrews (1991, (2.6), p. 821) P2m…T† limT!1 sˆÿ2m…T† ! …s; m…T††2 < 1 is satis®ed. Hence, the weights f!…s; m…T††g give rise to the (infeasible) positive semi-de®nite consistent estimator VT for V. A particular example of the sequence of weights f!…s; m…T††g is the ¯at kernel de®ned by !…s; m…T†† ˆ 1=…2m…T† ‡ 1†, s ˆ ÿm…T†; . . . ; m…T†, which results in the Bartlett-type kernel ! …s; m…T†† ˆ 1 ÿ jsj=…2m…T† ‡ 1†, s ˆ ÿ2m…T†; . . . ; 2m…T†, with …T† ˆ 1=…2m…T† ‡ 1†; see Newey and West (1987) and Kitamura and Stutzer (1997, equation (8), p. 865). For other examples, see Andrews (1991). A feasible positive semi-de®nite consistent estimator for V of (3) may therefore be de®ned as ^T  V

2m…T† X sˆÿ2m…T†

^ ! …s; m…T††C…s†;

…11†

^ ^ 0, ^ ˆ T ÿ1 PT ^ ^ 0 s  0, and C…ÿs† ˆ C…s† C…s† tˆs‡1 g…xt ; T †g…xtÿs ; T † ,  s ˆ ÿ2m…T†; . . . ; 2m…T†, and the weights f! …s; m…T††g are de®ned in (10) above. Note that an alternative positive semi-de®nite consistent esti^ T is output as minus …T…T††ÿ1 times the …; † component of mator to V the Hessian of RT …; † of (7) evaluated at …^T ; ^T †.

Empirical likelihood

2.4

127

Ef®cient cumulative distribution function estimation

The information contained in the moment conditions E fg…xt ; 0 †g ˆ 0, t ˆ 0; 1; . . . ; of (1) may be exploited to provide a more ef®cient estimator of the stationary distribution  of the P process fxt g1 tˆÿ1 than the T ÿ1 empirical distribution function T …x†  T 1…x  x†, where 1…:† t tˆ1 denotes an indicator function, that is, 1…xt  x† ˆ 1 if xt  x, 0 otherwise, t ˆ 1; . . . ; T. Consider the empirical likelihood cumulative distribution function (ELCDF) estimator based on the estimated probabilities t …^T ; ^T †, t ˆ 1; . . . ; T, from (8): þ ! m…T† T X X !…r; m…T††1…xtÿr  x† t …^T ; ^T †: …12† ^ T …x†  tˆ1

rˆÿm…T†

Theorem 2 (Limit distribution of the ELCDF estimator ^ T …x†.) Under (1), the ELCDF estimator ^ T …x† of (12) has limiting distribution given by T 1=2 …^ T …x† ÿ …x†† ) N…0; !2 †; where

ÿ
!2   2 ÿ B 0 Vÿ1 ÿ Vÿ1 G…G 0 Vÿ1 G†ÿ1 G 0 Vÿ1 B; 1 X

B and

sˆÿ1

2 

E f1…xt  x†g…xtÿs ; 0 †g

1 ÿ X
E f1…xt  x†1…xtÿs  x†g ÿ …x†2 :

sˆÿ1

The empirical distribution function T …x† has a limiting distribution described by T 1=2 …T …x† ÿ …x†† ) N…0;  2 †: Hence, the ELCDF estimator ^ T …x† is more ef®cient. Moreover, it may be straightforwardly seen that it dominates other estimators that incorporate the information in the moment conditions E fg…xt ; 0 †g ˆ 0, t ˆ 0; 1; . . . ; of (1). The various components in the variance !2 may be consistently estimated by ^ 2 

2m…T† X sˆÿ2m…T†

! …s; m…T††^ 2 …s†;

128

Richard J. Smith

ÿ
P where ^ 2 …s†  T ÿ1 Ttˆs‡1 1…xt  x†1…xtÿs  x† ÿ T …x†2 , s  0, ^ 2 …ÿs† ˆ ^ 2 …s†, s ˆ ÿ2m…T†; . . . ; 2m…T†, ^ B

2m…T† X

^ ! …s; m…T††B…s†;

sˆÿ2m…T†

where 2m…T†,

^  T ÿ1 Pmin…T;T‡s† 1…xt  x†g…xtÿs ; ^T †, B…s† tˆmax…1;s‡1† ^ T is given in (11) and V ^ T  T ÿ1 G

T X tˆ1

s ˆ ÿ2m…T†; . . . ;

r0 g…xt ; ^T †:

…13†

Alternatively, G may be estimated consistently by T ÿ1 3

Empirical likelihood inference

3.1

Tests for over-identifying moment conditions

PT

tˆ1

r0 g!tT …^T †.

In order to gauge the validity of the moment conditions E fg…xt ; 0 †g ˆ 0, t ˆ 0; 1; . . . ; of (1), Hansen (1982) suggested the statistic GT  T ÿ1

T X tˆ1

^ ÿ1 g…xt ; ^T † 0 V T

T X tˆ1

g…xt ; ^T †;

…14†

which may be shown under the validity of (1) to possess a limiting chisquared distribution with …r ÿ p† degrees of freedom. In the context of the smoothed log EL criterion (7), we might think of the validity of the moment conditions (1) as corresponding to the parametric restrictions  ˆ 0. As will soon become evident, this viewpoint allows us to de®ne straightforwardly classical-type tests for the validity of the moment conditions (1) based directly on consideration of the hypothesis  ˆ 0 in the smoothed log EL criterion RT …; † of (7). However, it should be emphasised that the parametric hypothesis  ˆ 0 is equivalent to that of the validity of the moment conditions (1). Firstly, we consider a likelihood ratio test based on the smoothed log EL criterion RT …; † of (7). Under  ˆ 0, RT …; 0† ˆ 0. When  6ˆ 0, the estimated smoothed log EL criterion is evaluated at …^T ; ^T †, namely RT …^T ; ^T †. From the classical viewpoint then, the difference between estimated smoothed log EL criteria may be used to de®ne an empirical likelihood ratio (ELR) statistic for testing  ˆ 0 or, rather, the overidentifying moment conditions E fg…xt ; 0 †g ˆ 0, t ˆ 0; 1; . . . ; of (1); namely ELRT  2…T†RT …^T ; ^T †:

…15†

Empirical likelihood

129

Theorem 3 (Limit distribution of the ELR statistic for overidentifying moment conditions.) Under (1), the ELR statistic ELRT of (15) has a limiting chi-squared distribution with …r ÿ p† degrees of freedom. Secondly, by analogy with the classical Wald test for  ˆ 0, an EL Wald statistic for the over-identifying moment conditions (1) is de®ned in terms of the Lagrange multiplier estimator ^T ; namely ^ T ^T ; …16† EW   T…T†2 ^T0 V T

^ T is de®ned in (11). where V

Proposition 1 (Limit distribution of the EL Wald statistic for over-identifying moment conditions.) Under (1), the EL Wald statistic EW T of (16) has a limiting chi-squared distribution with …r ÿ p† degrees of freedom. To de®ne a classical score test for  ˆ 0 based on the smoothed log EL criterion RT …; † raises a particular dif®culty; namely, when  is set equal to 0 the parameter vector  is no longer identi®ed. This class of problem has been considered by Davies (1977, 1987) in the classical and other contexts. For a recent treatment of this problem, see Andrews and Ploberger (1994). To circumvent this dif®culty, consider a score statistic based on the ®rst-order derivatives of the smoothed log EL criterion RT …; † evaluated at ^T and 0, the former of which could be regarded as using the least favourable choice of estimator for 0 . The score vector associated with the smoothed log EL criterion RT …; † is de®ned by   T X g!tT …† t …; † ST  ; r g!tT …† 0  tˆ1

where t …; †, t ˆ 1; . . . ; T, are given in (8). Now, ST evaluated at ^T and ^ T , has second block equal to a vector of zeroes. Also, t …^T ; 0† ˆ T ÿ1 , 0, S t ˆ 1; . . . ; T. Hence, the EL score statistic is given by þ ! ^ T ÿ1 ^T G V 0  ^T ^ S ES T  T ST 0 ^ GT 0 T  ÿ1  X ^ ^ 0 ^ ÿ1 ^ ÿ1 ^ ^ ÿ1 ^T ÿ V ^ ÿ1 ˆ T ÿ1 g!tT …^T † 0 V T GT …GT VT GT † GT VT tˆ1



T X tˆ1

g!tT …^T †;

^ T and V ^ T are de®ned in (13) and (11) respectively. where G

…17†

130

Richard J. Smith

Proposition 2 (Limit distribution of the EL score statistic for over-identifying moment conditions.) Under (1), the EL score statistic ES T of (17) has a limiting chi-squared distribution with …r ÿ p† degrees of freedom. Because the EMELE ^T of (9) is ®rst-order equivalent to the optimal GMM estimator of (2), the EMELE asymptotically obeys the optimal GMM estimator's ®rst-order conditions; see Proof of Proposition 2. Hence, an equivalent score-type statistic may be based on T ÿ1

T X tˆ1

^ ÿ1 g!tT …^T † 0 V T

T X tˆ1

g!tT …^T †

ˆ T ÿ1

T X tˆ1

ÿ1

^T g…xt ; ^T † 0 V

T X tˆ1

g…xt ; ^T † ‡ oP …1†;

the latter statistic being the GMM statistic GT of (14) using ^T of (9). Moreover, the EL Wald statistic and the P GMM statistic GT are ®rst-order equivalent as T 1=2 …T†^T ˆ Vÿ1 T ÿ1=2 Ttˆ1 g…xt ; ^T † ‡ oP …1†; that is, EW T ˆ GT ‡ oP …1†. Moreover, from the proofs of Theorem 1 and Propositions 1 and 2, the ELR, EL Wald and EL score statistics are all ®rst-order equivalent. Therefore, the classical-type EL statistics above offer a potentially useful alternative class of tests for the over-identifying moment conditions (1). Although not discussed here, other ®rst-order equivalent tests based on the C…† principle may also be de®ned in a parallel fashion; see inter alia Neyman (1959) and Smith (1987). 3.2

Tests for additional moment conditions

Typically, it may be of interest to examine whether an additional s-vector of moments also has zero mean and, thus, might be usefully incorporated to improve inferences on the vector 0 ; namely E fh…xt ; 0 †g ˆ 0; t ˆ 0; 1; . . . :

…18†

The approach due to Newey (1985a, 1985b) would set up a conditional moment P test based on a quadratic form in the estimated sample moments T ÿ1=2 Ttˆ1 h…xt ; ^T † using P as metric the inverse of the asymptotic variance matrix of T ÿ1=2 Ttˆ1 h…xt ; ^T †. However, similarly to the ELR test for over-identifying moment conditions given in (15), we may de®ne the log EL criterion appropriate for the incorporation of the additional moments E fh…xt ; 0 †g ˆ 0, t ˆ 0; 1; . . . ; of (18); namely

Empirical likelihood

RT …; ; þ† 

T X tˆ1

131

ln‰1 ‡  0 g!tT …† ‡ þ 0 h!tT …†Š;

…19†

Pm…T† where h!tT …†  sˆÿm…T† !…s; m…T††h…xtÿs ; †. The Lagrange multiplier estimators corresponding to RT …; ; þ† of (19) are given by …~T …†; þ~ T …†† ˆ arg max RT …; ; þ†; ;þ

and the EMELE under the moment conditions (1) and (18) is de®ned by ~T ˆ arg min max RT …; ; þ†: 2 ;þ

De®ne ~T  ~T …~T † and þ~ T  þ~ T …~T †. Similarly to the classical interpretation of the ELR, EL Wald and EL score statistics in section 3.1 for the over-identifying moment conditions (1), we may construct tests of the additional moment conditions E fh…xt ; 0 †g ˆ 0, t ˆ 0; 1; . . . ; of (18), by considering EL-based tests for the parametric hypothesis þ ˆ 0 in the smoothed EL criterion RT …; ; þ† of (19). Firstly, consider the difference of ELR statistics under (1), (18) and (1) respectively, which, as RT …; † ˆ RT …; ; 0†, corresponds to an ELR test for þ ˆ 0; namely  ELRT  ELR;þ T ÿ ELRT

ˆ 2…T†…RT …~T ; ~T ; þ~ T † ÿ RT …^T ; ^T ; 0††:

…20†

Therefore Theorem 4 (Limiting distribution of the ELR statistic for additional moment restrictions.) Under (1) and (18), the ELR statistic ELRT of (20) has a limiting chi-squared distribution with s degrees of freedom. If one were interested in the full vector of moment conditions (1) and  ~ ~ ~ (18), one obtains the statistic ELR;þ T ˆ 2…T†RT …T ; T ; þT † (see (15)), which has a limiting chi-squared distribution with …r ‡ s† ÿ p degrees of freedom; see section 3.1 and Hansen (1982). Secondly, we may also de®ne an EL Wald statistic for þ ˆ 0 or the additional moment restrictions (18) in an obvious way. De®ne the …r ‡ s; s† matrix Sþ to select the elements of þ from the vector … 0 ; þ 0 † 0 ; that is, Sþ0 … 0 ; þ 0 † 0 ˆ þ. The asymptotic variance matrix of the vector ÿ1=2 PT 0 0 0 T tˆ1 …g…xt ; 0 † ; h…xt ; 0 † † is denoted by

132

Richard J. Smith

( V  lim Var T T!1

ÿ1=2

T X ÿ tˆ1

0

g…xt ; 0 † ; h…xt ; 0 †

0


0

) ;

which is assumed positive de®nite. A positive semi-de®nite consistent ^ T of (11) after appro~ T for V may be de®ned similarly to V estimator V ~ T is a ~ ~ priate substitution of the estimators T , T and þ~ T . Similarly, G 0 0 0 consistent estimator for G  E f…r g…xt ; 0 † ; r h…xt ; 0 † † g based on ~T , ~T and þ~ T (see (13)). The EL Wald statistic for the additional moment conditions is then de®ned as6 ÿ1

ÿ1

~ T ~ T ÿ V ~ T G EW T  T…T†2 þ~ T0 …Sþ0 …V 0 ÿ1 ~ 0 ~ ÿ1 ~ 0 ~ ÿ1 ~ T ~ ÿ1 …G V T GT † GT VT †Sþ † þT :

…21†

Therefore Proposition 3 (Limiting distribution of the EL Wald statistic for additional moment restrictions.) Under (1) and (18), the EL Wald statistic EW T of (21) has a limiting chi-squared distribution with s degrees of freedom. The classical-type score test for þ ˆ 0 based on the smoothed log EL criterion RT …; ; þ† is constructed in a standard manner using the score from RT …; ; þ† evaluated at the estimators …^T ; ^T ; 0†. The score vector of the smoothed log EL criterion RT …; ; þ† is 1 0 g!tT …† T X C B C; t …; ; þ†B h!tT …† …22† ST  A @ tˆ1 r g!tT …† 0  ‡ r h!tT …† 0 þ where t …; ; þ†  T ÿ1 ‰1 ‡  0 g!tT …† ‡ þ 0 h!tT …†Šÿ1 ; t ˆ 1; . . . ; T; see t …; †, t ˆ 1; . . . ; T, of (8). Thus, from (22), we see that the score ^ T , has zeros in its ®rst and last vector ST evaluated at …^T ; ^T ; 0†, S blocks from the ®rst-order conditions determining ^T and ^T . Therefore, noting t …; ; 0† ˆ t …; †, t ˆ 1; . . . ; T, the EL score statistic for þ ˆ 0 for the additional moment conditions is given by 6 ÿ1 0 ÿ1 ÿ1 0 ÿ1 The (s; s) matrix Sþ0 …Vÿ1  ÿ V G …G V G † G V †Sþ is positive de®nite as the (r ‡ s; s ‡ p) matrix (Sþ ; G ) is full column rank.

Empirical likelihood

0 ES T 

0 ^ T @ TS

ˆT

T X tˆ1

133

^ T V

^ T ÿG

0 ^ T ÿG

0

1ÿ1 ^ T ; A S

…23†

t …^T ; ^T †h!tT …^T † 0 Sþ0



 ^ 0 ^ ÿ1 ^ ÿ1 ^ ^ ÿ1 ^ ÿ1 ^ ÿ1 ^  V T ÿ VT GT …GT VT GT † GT VT Sþ 

T X tˆ1

t …^T ; ^T †h!tT …^T †;

where ^ T  T ÿ1 G PT

T X …r g…xt ; ^T † 0 ; r h…xt ; ^T † 0 † 0 tˆ1

^ T of (13)) and V ^ T denotes a or T (see G positive semi-de®nite consistent estimator for V based on the estimators ^ T of (11)). ^T and ^T (see V ÿ1

! ^ 0 ! ^ 0 0 tˆ1 …r gtT …T † ; r htT …T † †

Proposition 4 (Limiting distribution of the EL score statistic for additional moment restrictions.) Under (1) and (18), the EL score statistic ES T of (23) has a limiting chi-squared distribution with s degrees of freedom. Other statistics asymptotically equivalent to the above EL-based statistics may be de®ned. For example, a minimum chi-squared statistic is given by ÿ
ÿ
^ T …~ ÿ ^ † 0 þ~ 0 0 : MCT ˆ T…T†2 …~T ÿ ^T † 0 þ~ T0 V …24† T T T The proofs of Theorem 4 and Propositions 3 and 4 show that the ELR, EL Wald, EL score and EL minimum chi-squared statistics of (20), (21), (23) and (24), respectively, are all ®rst-order equivalent. It also immediately follows from the expression of the ELR statistic ELRT of (20) as the  difference of the ELR statistics, ELR;þ T and ELRT , that equivalent statistics may be obtained as the difference of EL Wald and score statistics;  ;þ  namely EW ;þ T ÿ EW T and ES T ÿ ES T , respectively. Furthermore, given the discussion in section 3.1 concerning the equivalence of those EL-based statistics with the GMM statistic GT of (14), the statistics of section 3.2 are equivalent to the difference of estimated GMM criteria

134

Richard J. Smith

and thus the GMM statistic for additional moment restrictions (see Newey (1985a, 1985b)). A ®nal point to note is that, under the moment conditions (1) and (18), all of the statistics of section 3.2 above are asymptotically independent of those of section 3.1, a property also displayed by the classical tests for a sequence of nested hypotheses; see inter alia Aitchison (1962) and Sargan (1980). 3.3

Empirical likelihood speci®cation tests

The limiting distribution results given in section 3.2 may be suitably adapted to allow various classical-type speci®cation (likelihood ratio, Lagrange multiplier and Wald) tests on the parameter vector  to be de®ned. Consider the following parametric null hypothesis expressed in constraint equation form which includes many forms of parametric restrictions: H0 : h…0 † ˆ 0;

…25†

where h…:† is a known s-vector of functions that are continuously differentiable. We assume that the derivative matrix H ˆ r h…0 † 0 is full column rank s.7 We may integrate the parametric restrictions H0 : h…0 † ˆ 0 of (25) into the framework of section 3.2 by recalling that the weights f!…s; m…T††g Pm…T† !…s; m…T†† ˆ 1. Therefore, the corresponding smoothed obey sˆÿm…T† log EL criterion incorporating the parametric restrictions (25) is RT …; ; þ† 

T X tˆ1

ln‰1 ‡  0 g!tT …† ‡ þ 0 h…†Š;

with associated score vector ST 

T X tˆ1

0

B t …; ; þ†B @

g!tT …† h…† r g!tT …† 0  ‡ r h…† 0 þ

…26†

1 C C; A

where, now, we de®ne t …; ; þ†  T ÿ1 ‰1 ‡  0 g!tT …† ‡ þ 0 h…†Š: 7

The methods described below may be adapted for other types of constraints; namely, freedom equation (Seber (1964)), H0 : 0 ˆ …0 †, mixed form (GourieÂroux and Monfort (1989)), H0 : q…0 ; 0 † ˆ 0, and mixed implicit and constraint equation (Szroeter (1983)), H0 : q…0 ; 0 † ˆ 0, r…0 † ˆ 0, restrictions.

Empirical likelihood

135

Denoting the EMELE and Lagrange multiplier estimators based on RT …; ; þ† of (26) byP ~T , ~T and þ~ T , respectively, we see that ~T satis®es ~ h…T † ˆ 0 by recalling Ttˆ1 t …~T ; ~T ; þ~ T † ˆ 1. Therefore, t …~T ; ~T ; þ~ T † ˆ t …~T ; ~T †, t ˆ 1; . . . ; T, and RT …~T ; ~T ; þ~ T † ˆ RT …~T ; ~T †. It is straightforward to demonstrate the consistency of the estimators …~T ; ~T ; þ~ T † for …0 ; 0; 0† under E fg…xt ; 0 †g ˆ 0, t ˆ 0; 1; . . . ; of (1) and H0 : h…0 † ˆ 0 of (25); see Lemma 1. Moreover, the limiting distribution of the EMELE and Lagrange multiplier estimators ~T , ~T and þ~ T may be derived along similar lines to Theorem 1; namely, Theorem 5 (Limiting distribution of the EMELE ~T and Lagrange multiplier estimators ~T and þ~ T .) Under (1) and (25), the EMELE ~T and Lagrange multiplier estimators ~T and þ~ T have limiting distribution given by 0 B B T 1=2 B @

…T†~T þ~ T

1 C C C A

…~T ÿ 0 † 11 00 1 0 ÿ1 0 V ÿ Vÿ1 GNG 0 Vÿ1 P 0 BB C B C  0 0 ÿ1 ÿ1 ÿ1 C CC BB C B ) N BB 0 C; B H …G V G† H 0 CC; P0 AA @@ A @ 0 0 0 0 N 0

where  ÿ1 N  ……G 0 Vÿ1 G†ÿ1 ÿ …G 0 Vÿ1 G†ÿ1 H H 0 …G 0 Vÿ1 G†ÿ1 H  H 0 …G 0 Vÿ1 G†ÿ1 †;  ÿ1 P  Vÿ1 G…G 0 Vÿ1 G†ÿ1 H H 0 …G 0 Vÿ1 G†ÿ1 H : The corresponding ELR and EL Wald statistics become ELRT  2…T†…RT …^T ; ^T † ÿ RT …~T ; ~T ††; EW T 

^ T0 V ^ ÿ1 ^ ~ ^ T0 …G ^ ÿ1 T þ~ T0 H T GT † HT þT ;

…27† …28†

see section 3.2. As the ®rst and last blocks of the score vector ST esti^ T , are zero, the EL score statistic mated at …^T ; ^T ; 0†, S

136

Richard J. Smith

0

^T V

0

^T ÿG

1ÿ1

B C 0 B ^ T ^ T ^T C ES T  T S B 00 C S 0 ÿH @ A ^ T0 ÿH ^ T0 ÿG 0   ÿ1 ÿ1 ^ T0 V ^ ^ T0 G ^ ÿ1 ^T G H ˆ Th…^T † 0 H h…^T †; T T

…29†

^ T are de®ned in (11) and (13), respectively, and ^ T and G where V ^ T  r h…^T † 0 . The forms of the EL Wald (28) and EL score (29) statisH tics are, respectively, those of Lagrange multiplier and Wald statistics in the GMM framework. Proposition 5 (Limiting distribution of EL statistics for parametric restrictions.) Under (1) and (25), the EL statistics ELRT , EW T and ES T statistics of (27), (28) and (29), respectively, each have limiting chi-squared distributions with s degrees of freedom. As the EMELE ^T is ®rst-order equivalent to the ef®cient GMM estimator, not only are all three statistics ELRT , EW T and ES T asymptotically ®rst-order equivalent (see Proof of Proposition 5) but they are equivalent to GMM tests for H0 : h…0 † ˆ 0 of (25); see (29). The Proof of Proposition 5 also shows that the following minimum chi-squared statistics are ®rst-order equivalent to ELRT , EW T and ES T : ^ T …~T ÿ ^T †; MCT ˆ T…T†2 …~T ÿ ^T † 0 V ^ T0 V ^ ~ ^ ÿ1 ^ MCT ˆ T…~T ÿ ^T † 0 G T GT …T ÿ T †; ~ T ~T ÿ ^T0 V ^ T ^T †. Moreover, differences of as is the statistic T…T†2 …~T0 V EL Wald and score statistics for  ˆ 0; þ ˆ 0 and  ˆ 0 are also equivalent to the above statistics from the structure of the statistic ELRT (27); see section 3.2.

4

Empirical likelihood non-nested tests

Denote the model embodied in the moment conditions E fg…xt ; 0 †g ˆ 0, t ˆ 0; 1; . . . ; of (1) by Hg . Consider an alternative model Hh based on the assumed moment conditions E fh…xt ; ÿ0 †g ˆ 0; t ˆ 0; 1; . . . : We denote the smoothed log EL function under Hh of (30) by

…30†

Empirical likelihood

RhT …ÿ; † 

T X tˆ1

137

ln‰1 ‡  0 h!tT …ÿ†Š;

…31†

Pm…T† where h!tT …ÿ†  sˆÿm…T† !…s; m…T††h…xtÿs ; ÿ†. For convenience of exposition, the weights and lead and lag truncation parameter have been chosen identical to those under Hg ; namely f!…s; m…T††g and m…T† of section 2.3. Correspondingly, we de®ne the Lagrange multiplier estimator by ^T …ÿ†  arg max RhT …ÿ; † 

and the EMELE under Hh of (30) by ÿ^T  arg min max RhT …ÿ; †; ÿ2B



…32†

where B is a compact set; we denote ^T  ^T …ÿ^T †. Under Hg of (1), the asymptotic pseudo-true value (PTV) of the Lagrange multiplier estimator ^T is de®ned via  …ÿ† ˆ arg max lim E fln‰1 ‡  0 h!tT …ÿ†Šg T!1



and that for the EMELE ÿ^T as ÿ ˆ arg min max lim E fln‰1 ‡  0 h!tT …ÿ†Šg; ÿ2B



T!1

we write  ˆ  …ÿ †. Therefore, by a similar argument to Lemma 1, under Hg of (1), ÿ^T !P ÿ ;

^T !P  :

In order to avoid the dif®culty of observational equivalence between the smoothed log EL criteria under Hg and Hh , it is assumed that limT!1 E fln‰1 ‡ 0 h!tT …ÿ †Šg > 0. Hence,  6ˆ 0.8 The usual approach to constructing Cox-type (1961, 1962) tests of Hg of (1) against Hh of (30) involves the contrast of consistent estimators under Hg of the probability limit of the requisite criterion function under Hh evaluated at the corresponding estimator PTV; see Smith (1992, equation (2.3), p. 973). That is, for the smoothed EL criterion (31), limT!1 E fln‰1 ‡ 0 h!tT …ÿ †Šg. Clearly, the smoothed log EL criterion (31) evaluated at the estimators …ÿ^T ; ^T †, T ÿ1 RhT …ÿ^T ; ^T † provides one such consistent estimator. In order to provide another consistent estimator to contrast with T ÿ1 RhT …ÿ^T ; ^T †, recall that the empirical measures t …^T ; ^T † ˆ T ÿ1 …1 ‡ oP …1††, t ˆ 1; . . . ; T. Hence, consider the alternative smoothed log EL criterion function 8

As in the Proof of Lemma 1, the minimising PTV  …ÿ† is unique for given ÿ.

138

Richard J. Smith

Rh T …ÿ; † ˆ

T X tˆ1

t …^T ; ^T † ln‰1 ‡  0 h!tT …ÿ†Š;

…33†

and denote the corresponding optimisers by ÿ~T and ~T . Therefore, as …^T ; ^T † are consistent for …0 ; 0† under Hg , …ÿ~T ; ~T † are consistent for …ÿ ;  † and an alternative consistent estimator for limT!1 E fln‰1‡ 0 h!tT …ÿ †Šg is obtained by evaluation of (33) at …ÿ~T ; ~T †: ~ ~ Rh T …ÿT ; T † ˆ

T X tˆ1

t …^T ; ^T † ln‰1 ‡ ~T0 h!tT …ÿ~T †Š:

Consequently, an EL likelihood ratio Cox-type statistic for Hg against Hh is based on the contrast between the optimised smoothed log EL ~ ~ criteria, T ÿ1 RhT …ÿ^T ; ^T † of (31) and Rh T …ÿT ; T † of (33); namely ~ ~ CT …Hg jHh † ˆ T 1=2 …T†…T ÿ1 RhT …ÿ^T ; ^T † ÿ Rh T …ÿT ; T ††:

…34†

Theorem 6 (Limiting distribution of the Cox-type statistic.) Under (1), the Cox statistic CT …Hg jHh † of (34) has a limiting N…0;  2 † distribution where ÿ
 2  0 Vÿ1 ÿ Vÿ1 G…G 0 Vÿ1 G†ÿ1 G 0 Vÿ1  ;   lim E fln‰1 ‡ 0 h!tT …ÿ †Šg!tT …0 †g; T!1

2

and  is assumed non-zero. The asymptotic variance  2 may be consistently estimated under Hg by  ÿ1  ^ ^ 0 ^ ÿ1 ^ ÿ1 ^ 0 ^ ÿ1 ^ ^T ÿ V ^ ÿ1 ^T2  ^T0 V T GT …GT VT GT † GT VT T ; where ^T  T ÿ1

T X tˆ1

ln‰1 ‡ ^T0 h!tT …ÿ^T †Šg!tT …^T †

…35†

^ T are described in (11) and (13) respectively. ^ T and G and V The Proof of Theorem 6 suggests the alternative ®rst-order equivalent linearised Cox-type statistic: LCT …Hg jHh † ˆ ^T0 T 1=2 …T†^T ;

…36†

where ^T is de®ned in (35) (see Smith (1992), section 2.1, pp. 974±976).

Empirical likelihood

139

Corollary 1 (Limiting distribution of the linearised Cox-type statistic.) Under (1), the linearised Cox statistic LCT …Hg jHh † of (36) has a limiting N…0;  2 † distribution where  2 , de®ned in Theorem 6, is assumed non-zero. P Because T 1=2 …T†^T ˆ Vÿ1 T ÿ1=2 Ttˆ1 g…xt ; ^T † ‡ oP …1†, the form of the linearised statistic LCT …Hg jHh † emphasises that non-nested tests of competing hypotheses expressed in moment form are particular linear combinations of the estimated sample moment vector under the null hypothesis as this vector represents the sole information feasible and available for inference purposes (see Smith (1992)). A simpli®ed form of EL likelihood ratio Cox-type statistic evaluates ^ ^ the criterion Rh T …ÿ; † at …ÿT ; T † and is ®rst-order equivalent to the Cox statistics, CT …Hg jHh † of (34) and LCT …Hg jHh † of (36); namely   ^ ^ SCT …Hg jHh † ˆ T 1=2 …T† T ÿ1 RhT …ÿ^T ; ^T † ÿ Rh …37† T …ÿT ; T † : Proposition 6 (Limiting distribution of the simpli®ed Cox-type statistic.) Under (1), the simpli®ed Cox statistic SCT …Hg jHh † of (37) has a limiting N…0;  2 † distribution where  2 , de®ned in Theorem 6, is assumed non-zero.

Appendix Proof of Lemma 1 De®ne MT …; †  E fln‰1 ‡  0 g!tT …†Šg, where, from (1) and (6), E fg!tT …†g ˆ 0 ,  ˆ 0 . Now, E fg!tT …†g!tT …† 0 g is positive de®nite if fg…xt ; †g1 exists no sequence tˆÿ1 is a non-defective process; that Pm is, there 0 such that a g…x of non-zero r-vectors fam;s gm tÿs ; † is a degensˆÿm sˆÿm m;s erate random variable for some m ˆ 0; 1; . . . . By Chesher and Smith (1997, Lemma, p. 643), the matrix of second derivatives of MT …; † with respect to , ÿE f‰1 ‡  0 g!tT …†Šÿ2 g!tT …†g!tT …† 0 g, is negative de®nite. De®ne T …†  arg max MT …; †: 

Hence, T …† satis®es the ®rst-order conditions E f‰1 ‡  0 g!tT …†Šÿ1 g!tT …†g ˆ 0: Therefore, it follows from Chesher and Smith (1997, Lemma and Proof of Theorem 1, p. 643) that the maximiser T …† is unique. Consequently,

140

Richard J. Smith

as 0 uniquely satis®es the moment conditions (1), and, thus, (6), T …0 † ˆ 0 and T …† 6ˆ 0,  6ˆ 0 . Now, 0 ˆ MT …; T …0 ††  MT …; T …††,  6ˆ 0 . Moreover, 0 ˆ MT …; T …0 †† < MT …; T …††,  6ˆ 0 because T …† is a unique maximiser of MT …; †. Therefore, 0 is the unique minimiser of MT …; T …††; namely 0  arg min MT …; T …††; 2

and MT …0 ; T …0 †† ˆ 0. De®ne a -neighbourhood of 0 , N  …0 †, and a sequence of open j -neighbourhoods, N j …j †, j 2 , j ˆ 1; . . . ; J, such that [Jjˆ1 N j …j † covers  ÿ N  …0 †. From continuity and the Dominated Convergence Theorem   E 0 inf ln‰1 ‡ T … 0 † 0 g!tT … 0 †Š ˆ 2j > 0; j ˆ 1; . . . ; J:  2N j …j †

A point-wise double array WLLN gives: ( ) T X ÿ1 0 0 ! 0 inf ln‰1 ‡ T … † gtT … †Š < j < =2J; Pr T 0 tˆ1

 2N j …j †

j ˆ 1; . . . ; J: Hence,

( Pr

inf

 0 2ÿN  …0 †

T

ÿ1

T X tˆ1

) ln‰1 ‡

T … 0 † 0 g!tT … 0 †Š

<  < =2;

where  ˆ minj j . The saddle-point maximisation property of ^T …:† implies ( ) T X ÿ1 0 0 ! 0 ^ Pr 0 inf T ln‰1 ‡ T … † gtT … †Š <  < =2: …A:1†  2ÿN  …0 †

tˆ1

Now, by concavity ln‰1 ‡ ^T …0 † 0 T ÿ1

tˆ1

g!tT …0 †Š  T ÿ1

T X tˆ1

ln‰1 ‡ ^T …0 † 0 g!tT …0 †Š  0:

PT As T ˆT g…xt ; 0 † ‡ oP …1† !P 0, we have PT tˆ1 P ÿ1 0 ! P ^ ^ T …0 † ! 0 and, thus, T tˆ1 ln‰1 ‡ T …0 † gtT …0 †Š ! 0. Therefore ( ) T X ÿ1 0 ! ln‰1 ‡ ^T …0 † gtT …0 †Š > =2 < =2: …A:2† Pr T ÿ1

PT

T X

! tˆ1 gtT …0 †

tˆ1

ÿ1

Empirical likelihood

141

Consider the event ( T X T ÿ1 ln‰1 ‡ ^T …0 † 0 g!tT …0 †Š tˆ1

<

inf

 0 2ÿN  …0 †

T

ÿ1

T X

) ln‰1 ‡

tˆ1

^T … 0 † 0 g!tT … 0 †Š

:

Therefore, ^T 2 N  …0 †. Hence, combining (A.1) and (A.2), we have Prf^T 2 N  …0 †g > 1 ÿ ; which yields the required result.

&

Proof of Theorem 1 Consider the ®rst-order conditions determining ^T and ^T þ ! þ ! T X 0 g!tT …^T † ^ ^ t …T ; T † : …A:3† ˆ ! ^ 0^ 0 r gtT …T † T tˆ1 Pm…T† !…s; m…T†† ˆ 1, !…s; m…T†† ˆ O…m…T†ÿ1 †. Moreover, Now, as sˆÿm…T† T ÿ1

T X tˆ1

r g!tT …0 † 0

ˆ T ÿ1

T X tˆ1

þ

r g…xt ; 0 † 0 ÿ T ÿ1 ÿT

ÿ1

T X

m…T† X m…T† X sˆt

tˆ1

þ

! !…s; m…T†† r g…xt ; 0 † 0 !

tÿT X

!…s; m…T†† r g…xt ; 0 † 0

tˆTÿm…T†‡1 sˆÿm…T†

ˆ T ÿ1

T X tˆ1

r g…xt ; 0 † 0 ‡ OP …m…T†=T†;

Pm…T† Pm…T† !…s; m…T†† ˆ O…1† and m…T†ÿ1 tˆ1  noting that, for example, sˆt 0 r g…xtÿs ; 0 † ˆ OP …1†. Therefore, a ®rst-order Taylor series expansion of (A.3) about 0 and 0 yields þ ! 0 0

þ

ˆ

T ÿ1=2

PT

tˆ1

0

g!tT …0 †

þ

!

‡

ÿV

G

G0

0

!

T

Moreover, by a similar argument to that above,

þ

1=2

…T†^ T ^T ÿ 0

!

‡ oP …1†:

142

Richard J. Smith

T ÿ1=2

T X tˆ1

g!tT …0 † ˆ T ÿ1=2

T X tˆ1

g…xt ; 0 † ‡ OP ……m…T†=T†1=2 †:

Therefore T 1=2 …^T ÿ 0 † ˆ ÿ…G 0 Vÿ1 G†ÿ1 G 0 Vÿ1 T ÿ1=2

T X tˆ1

g…xt ; 0 † ‡ oP …1†; …A:4†

T

1=2

…T†^T ˆ …Vÿ1 ÿ Vÿ1 G…G 0 Vÿ1 G†ÿ1 G 0 Vÿ1 †T ÿ1=2 

T X tˆ1

g…xt ; 0 † ‡ oP …1†:

…A:5†

Hence, fromP(A.4) and (A.5), the result of the theorem follows from the CLT T ÿ1=2 Ttˆ1 g…xt ; 0 † ) N…0; V†. & Proof of Theorem 2 Consider a ®rst-order Taylor series expansion of T 1=2 …^ T …x† ÿ …x†† about 0 and 0: T 1=2 …^ T …x† ÿ …x†† ˆ T 1=2 …T …x† ÿ …x†† ÿ B 0 T 1=2 …T†^T ‡ oP …1†: We have made use of the approximations T ÿ1=2

m…T† T X X tˆ1 rˆÿm…T†

ˆ T ÿ1=2

T X tˆ1

and

þ …T…T††

ÿ1

!…r; m…T††1…xtÿr  x†

1…xt  x† ‡ oP …1†;

m…T† T X X tˆ1 rˆÿm…T†

! !…r; m…T††1…xtÿr 

x†g!tT …0 † 0

see the Proof of Theorem 1. As þþ ! þ 2 þ ! T X 0 1…xt  x† ÿ …x†  ÿ1=2 )N ; T 0 g…xt ; 0 † B tˆ1 Therefore, from (A.5),

ˆ B 0 ‡ oP …1†;

B0 V

!! :

Empirical likelihood

143

T 1=2 …^ T …x† ÿ …x†† ÿB 0 …Vÿ1 ÿ Vÿ1 G…G 0 Vÿ1 G†ÿ1 G 0 Vÿ1 †† þ ! T 1…xt  x† ÿ …x† X ‡ oP …1†; T ÿ1=2 g…xt ; 0 † tˆ1

ˆ …1;

and, hence, the result follows.

&

Proof of Theorem 3 A second-order Taylor series expansion of RT …0 ; 0† ˆ 0 about ^T and ^T gives ÿ
0 ˆ …T†RT …^T ; ^T † ‡ …1=2†T …T†^T0 …^T ÿ 0 † 0 þ 

ÿV G0

1 !0 …T†^T @ A ‡ oP …1†: 0 …^T ÿ 0 †

G

…A:6†

After substitution from (A.4) and (A.5) into (A.6) and noting G 0 T 1=2 …T†^T ˆ oP …1†, we obtain 2…T†RT …^T ; ^T † ˆ T…T†2 ^T0 V^T ‡ oP …1† ˆ T ÿ1=2

T X tˆ1

g…xt ; 0 † 0 ‰Vÿ1 ÿ Vÿ1 G

0

 …G Vÿ1 G†ÿ1 G 0 Vÿ1 ŠT ÿ1=2 

T X tˆ1

yielding the required result.

g…xt ; 0 † ‡ oP …1†;

…A:7†

&

Proof of Proposition 1 Follows directly from the expansion ( A.7). Proof of Proposition 2

&

P From (A.5), T 1=2 …T†^T ˆ Vÿ1 T ÿ1=2 Ttˆ1 g…xt ; ^T † ‡ oP …1†. Moreover, ÿ1=2 PT ! ^ similarly to the Proof of Theorem 1, T tˆ1 gtT …T † ˆ ÿ1=2 PT ^ T tˆ1 g…xt ; T † ‡ oP …1†. The result follows because the ®rst-order

144

Richard J. Smith

conditions determining the optimal GMM estimator imply that ÿ1=2 PT ! ^ ^ T0 V ^ ÿ1 G T T tˆ1 gtT …T † ˆ oP …1†. & Proof of Theorem 4 A Taylor series expansion for RT …~T ; ~T ; þ~ T † similar to that in (A.7) for RT …^T ; ^T † results in 0 1 ~T ÿ
A ‡ oP …1† 2…T†RT …~T ; ~T ; þ~ T † ˆ T…T†2 ~T0 þ~ T0 V @ ~ þT ˆ T ÿ1

T ÿ X tˆ1




g…xt ; 0 † 0 ; h…xt ; 0 † 0 A

þ ! T g…xt ; 0 † X tˆ1

h…xt ; 0 †

‡ oP …1†;

…A:8†

ÿ 0
0 ÿ1 0 ÿ1 ÿ1 0 ÿ1 where A  Vÿ1 H0 , H   ÿ V G …G V G † G VP, G  G E fr0 h…x; 0 †g and V  limT!1 VarfT ÿ1=2 Ttˆ1 … g…xt ; 0 † 0 ; h…xt ; 0 † 0 † 0 g. Hence 2…T†fRT …~T ; ~T ; þ~ T † ÿ RT …^T ; ^T ; 0†g ˆ T ÿ1

T ÿ X tˆ1




g…xt ; 0 † 0 ; h…xt ; 0 † 0 …A ÿ A†

þ ! T g…xt ; 0 † X tˆ1

h…xt ; 0 †

‡ oP …1†;

…A:9†

where A  S‰Vÿ1 ÿ Vÿÿ1 G…G 0 Vÿ1 G†ÿ1 G 0 V
ÿ1 ŠS 0 and S is a selection 0 matrix such that S 0 g…xt ; 0 † 0 ; h…xt ; 0 † 0 ˆ g…xt ; 0 †. Note that V ˆ 0 0 S V S and S G ˆ G. Now V ‰A ÿ AŠV ‰A ÿ AŠV ˆ V ‰A ÿ AŠV as A V A ˆ A and AVA ˆ A. Therefore, Proposition 2 follows from Rao and Mitra (1971, Theorem 9.2.1, p. 171) with degrees of freedom given by trfV ‰A ÿ AŠg ˆ trfV A g ÿ trfV Ag ˆ …r ‡ s ÿ p† ÿ …r ÿ p† ˆ s:

&

Empirical likelihood

145

Proof of Proposition 3 Firstly, we have from (A.5) that ÿ T 1=2 …T† ~T0

þ~ T0


0

ˆ A T ÿ1=2

T ÿ X tˆ1


0 g…xt ; 0 † 0 ; h…xt ; 0 † 0 ‡oP …1†: …A:10†

Combining (A.5) and (A.8) yields the intermediate result that the EL minimum chi-squared statistic MCT of (24) ÿ
ÿ
^  …~ ÿ ^ † 0 þ~ 0 0 T…T†2 …~T ÿ ^T † 0 þ~ T0 V T T T ˆ T ÿ1

T ÿ X tˆ1



T ÿ X tˆ1


0 g…xt ; 0 † 0 ; h…xt ; 0 † 0 …A ÿ A†


g…xt ; 0 † 0 ; h…xt ; 0 † 0 ‡ oP …1†;

…A:11†

which, therefore, from (A.7), is asymptotically equivalent to the difference of ELR statistics (20). Now, þ ! V ÿG ÿ1 ÿG0 0 0 1 ÿ1 0 ÿ1 ÿ1 0 ÿ1 0 ÿ1 ÿ1 ÿVÿ1 Vÿ1  ÿ V G …G V G † G V  G …G V G † A: ˆ@ 0 ÿ1 ÿ1 0 ÿ1 0 ÿ1 ÿ1 ÿ…G V G † G V ÿ…G V G † such that De®ne the …r ‡ s ‡ p; s† selection matrix Sþ 0 Sþ … 0 ; þ 0 ;  0 † 0 ˆ þ. Therefore, the EL Wald statistic EW T of (21) may be expressed as 0 10 …T†…~T ÿ ^T † 0 1ÿ1 þ ! B C V ÿG ÿ1 B C 0 Sþ A EW T ˆ T B C Sþ @Sþ …T†þ~ T @ A ÿG0 0 …~T ÿ ^T † 0

B 0 B  Sþ B @

…T†…~T ÿ ^T † …T†þ~ T …~T ÿ ^T †

1

C C C ‡ oP …1†: A

146

Richard J. Smith

A Taylor series expansion of the score at …~T ; ~T ; þ~ T † about the score at ^ T (see (22)), yields …^T ; ^T ; 0†, S 0 1 …T†…~T ÿ ^T † þ ! B C V ÿG B C ^ T ˆ ÿ ~T T 1=2 B T 1=2 S C ‡ oP …1†; …T† þ 0 @ A ÿG 0 ~ ^ …T ÿ T †

noting Hence,

ÿ

G0 T 1=2 …T† ~T0 0 B T 1=2 B @

þ~ T0

…T†…~T ÿ ^T † …T†þ~ T …~T ÿ ^T †


0

…A:12† ˆ oP …1†

G 0 T 1=2 …T†^T ˆ oP …1†.

and

1

þ V C Cˆÿ A ÿG0

ÿG 0

!ÿ1 ^ T ‡ oP …1†: T 1=2 S

^ T ˆ Sþ PT t …^T ; ^T †h!tT …^T †, Therefore, noting that S tˆ1  ÿ1 0 V ÿG ^ T ‡ oP …1† ^ T S EW T ˆ T S ÿG0 0 ˆ ES T ‡ oP …1†:

…A:13†

^ T from (A.12) and again recalling Substituting into (A.13) for T 1=2 S G0 T 1=2 …T†… ~T 0 þ~ T0 † 0 ˆ oP …1† and G 0 T 1=2 …T†^T ˆ oP …1† yields (A.11) apart from asymptotically negligible terms. & Proof of Proposition 4 This result is immediate from (A.13). & Proof of Theorem 5 Consider the ®rst-order conditions determining …~T ; ~T ; þ~ T †: 1 0 1 0 0 g!tT …~T † T X C B C B C ˆ B 0 C: t …~T ; ~T †B h…~T † A @ A @ tˆ1 ! ~ 0 ~ ~ ~ 0 þ r gtT …T † T ‡ HT T

…A:14†

Because h…0 † ˆ 0, a ®rst-order Taylor series expansion of (A.14) about …0 ; 0; 0† yields

Empirical likelihood

147

1 0 1 0 ÿ1=2 PT ! 0 T tˆ1 gtT …0 † C B C B C B C B C B0C ˆ B 0 A @ A @ 0 0 0 1 0 1 ÿV 0 G …T†~T B C B C B C B C ‡ B 00 0 H 0 CT 1=2 B þ~ T C ‡ oP …1†: @ A @ A G0 H 0 …~T ÿ 0 † Therefore T 1=2 …~T ÿ 0 † ˆ ÿNG 0 Vÿ1 T ÿ1=2

T X tˆ1

g…xt ; 0 † ‡ oP …1†;

…A:15†

T X ÿ
g…xt ; 0 † ‡ oP …1†; T 1=2 …T†~T ˆ Vÿ1 ÿ Vÿ1 GNG 0 Vÿ1 T ÿ1=2 tˆ1

T

1=2

þ~ T ˆ P T 0

ÿ1=2

T X tˆ1

…A:16† g…xt ; 0 † ‡ oP …1†: &

…A:17†

Proof of Proposition 5 A Taylor series expansion for RT …~T ; ~T † about …0 ; 0; 0† results in 2…T†RT …~T ; ~T † ˆ T…T†2 ~T0 V~T ‡ oP …1†:

…A:18†

Combining (A.7) and (A.18): 2…T†…RT …~T ; ~T † ÿ RT …^T ; ^T †† ˆ T…T†2 …~T0 V~T ÿ ^T0 V^T † ‡ oP …1† ˆ T ÿ1=2

T X tˆ1



T X tˆ1

  g…xt ; 0 † 0 P H 0 …G 0 Vÿ1 G†ÿ1 H P 0 T ÿ1=2

g…xt ; 0 † ‡ oP …1†

ˆ T þ~ T0 H 0 …G 0 Vÿ1 G†ÿ1 Hþ~ T ‡ oP …1†; the second equality following from (A.5) and (A.16) and the third from (A.17). From (A.4) and (A.17),

148

Richard J. Smith

T 1=2 h…^T † ˆ ÿH 0 …G 0 Vÿ1 G†ÿ1 HT 1=2 þ~ T ‡ oP …1†: We also note that, from (A.4), (A.15) and (A.17), T 1=2 …~T ÿ ^T † ˆ …G 0 Vÿ1 G†ÿ1 HT 1=2 þ~ T ‡ oP …1†; and, from (A.5), (A.16) and (A.17), T 1=2 …T†…~T ÿ ^T † ˆ Vÿ1 G…G 0 Vÿ1 G†ÿ1 HT 1=2 þ~ T ‡ oP …1†:

&

Proof of Theorem 6 From the ®rst-order conditions determining …ÿ^T ; ^T †, a ®rst-order Taylor series expansion about …0 ; 0† and …ÿ^T ; ^T † for the optimised criterion ~ ~ Rh T …ÿT ; T † yields   ~ ~ … ÿ ;  † T 1=2 …T† T ÿ1 RhT …ÿ^T ; ^T † ÿ Rh T T T þ ˆ T

T X ÿ1 tˆ1

! ln‰1 ‡ ^T0 h!tT …ÿ^T †Šg!tT …0 † 0 T 1=2 …T†^T ‡ oP …1†

ˆ 0 T 1=2 …T†^T ‡ oP …1†: & Proof of Corollary 1 Immediate from that of Theorem 6.

&

Proof of Proposition 6 A ®rst-order Taylor series expansion of the statistic T 1=2 …T† ^ ^ …RhT …ÿ^T ; ^T † ÿ Rh T …ÿT ; T †† about …0 ; 0† yields an identical expansion to that in the Proof of Theorem 6. &

References Aitchison, J. (1962), `Large-Sample Restricted Parametric Tests', Journal of the Royal Statistical Society (Series B), 24: 234±250. Altonji, J.G. and L.M. Segal (1996), `Small Sample Bias in GMM Estimation of Covariance Structures', Journal of Business and Economic Statistics, 14(3): 353±366. Andrews, D.W.K. (1991), `Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation', Econometrica, 59: 817±858.

Empirical likelihood

149

Andrews, D.W.K. and W. Ploberger (1994), `Optimal Tests When a Nuisance Parameter is Present Only under the alternative', Econometrica, 62: 1383± 1414. Back, K. and D.P. Brown (1993), `Implied Probabilities in GMM Estimators', Econometrica, 61: 971±975. Chesher, A. and R.J. Smith (1997), `Likelihood Ratio Speci®cation Tests', Econometrica, 65: 627±646. Cox, D.R. (1961), `Tests of Separate Families of Hypotheses', Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics, 1: 105±123. (1962), `Further Results on Tests of Separate Families of Hypotheses', Journal of the Royal Statistical Society (Series B), 24: 406±424. Davies, R.B. (1977), `Hypothesis Testing When a Parameter is Present Only under the Alternative', Biometrika, 64: 247±254. (1987), `Hypothesis Testing When a Parameter is Present Only under the Alternative', Biometrika, 74: 33±43. DiCiccio, T.J., P. Hall and J.P. Romano (1989), `Comparison of Parametric and Empirical Likelihood Functions', Biometrika, 76: 465±476. GourieÂroux, C. and A. Monfort (1989), `A General Framework for Testing a Null Hypothesis in a ``Mixed Form'' ', Econometric Theory, 5: 63±82. Hansen, L.P. (1982), `Large Sample Properties of Generalized Method of Moments Estimators', Econometrica, 50: 1029±1054. Imbens, G.W. (1997), `One-Step Estimators for Over-Identi®ed Generalized Method of Moments Models', Review of Economic Studies, 64: 359±383. Imbens, G.W., R.H. Spady and P. Johnson (1998), `Information Theoretic Approaches to Inference in Moment Condition Models', Econometrica, 66: 333±357. Kitamura, Y. and M. Stutzer (1997), `An Information-Theoretic Alternative to Generalized Method of Moments Estimation', Econometrica, 65: 861± 874. Newey, W.K. (1985a), `Maximum Likelihood Speci®cation Testing and Conditional Moment Tests', Econometrica, 53: 1047±1070. (1985b), `Generalized Method of Moments Speci®cation Testing', Journal of Econometrics, 29: 229±256. Newey, W.K. and K.D. West (1987), `A Simple, Positive Semi-De®nite Heteroskedasticity and Autocorrelation Consistent Covariance Matrix', Econometrica, 55: 703±708. Neyman, J. (1959), `Optimal Asymptotic Tests of Composite Statistical Hypotheses', in U. Grenander (ed.), Probability and Statistics, New York: Wiley. Owen, A.B. (1988), `Empirical Likelihood Ratio Con®dence Intervals for a Single Functional', Biometrika, 75: 237±249. (1990), `Empirical Likelihood Con®dence Regions', Annals of Statistics, 18: 90± 120. (1991), `Empirical Likelihood for Linear Models', Annals of Statistics, 19: 1725±1747.

150

Richard J. Smith

Qin, J. and J. Lawless (1994), `Empirical Likelihood and General Estimating Equations', Annals of Statistics, 22: 300±325. Rao, C.R. and S.K. Mitra (1971), Generalized Inverse of Matrices and Its Applications, New York: Wiley. Sargan, J.D. (1980), `Some Tests of Dynamic Speci®cation for a Single Equation', Econometrica, 48: 879±897. Seber, G.A.F. (1964), `The Linear Hypothesis and Large Sample Theory', Annals of Mathematical Statistics, 35: 773±779. Smith, R.J. (1987), `Alternative Asymptotically Optimal Tests and Their Application to Dynamic Speci®cation', Review of Economic Studies, 54: 665±680. (1992), `Non-Nested Tests for Competing Models Estimated by Generalized Method of Moments', Econometrica, 60: 973±980. (1997), `Alternative Semi-Parametric Likelihood Approaches to Generalized Method of Moments Estimation', Economic Journal, 107: 503±519. Szroeter, J. (1983), `Generalized Wald Methods for Testing Nonlinear Implicit and Overidentifying Restrictions', Econometrica, 51: 335±353.

5

Ef®ciency and robustness in a geometrical perspective Russell Davidson

1

Introduction

The aim of this chapter is to construct a general geometrical setting, based on Hilbert space, in which one may study various estimation techniques, in particular with respect to ef®ciency and robustness. Given the sort of data one wishes to study, such as continuous, discrete, etc., the set of data-generation processes (DGPs) capable of generating data of that sort is given the structure of a Hilbert manifold. Statistical models will be treated as submanifolds of this underlying manifold. Geometrical methods are frequently used in the study of statistical inference. One important strand of the literature is presented in Amari (1990), whose numerous earlier papers were inspired by some very abstract work of Chentsov (1972) and led to the concept of a statistical manifold. Other review papers and books in this tradition include Barndorff-Nielsen, Cox and Reid (1986), Kass (1989), Murray and Rice (1993), and Barndorff-Nielsen and Cox (1994). Most of this work makes use of ®nite-dimensional differential manifolds, which are usually representations of models in the exponential family. In®nite-dimensional Hilbert space methods are extensively used in another strand of literature, for which the most suitable recent reference is Small and McLeish (1994). This book contains numerous references to the original papers on which it builds. In this work, random variables are represented as elements of Hilbert space, and different probability measures (that is, different DGPs) correspond to different inner products on the Hilbert space. However, no manifold structure is imposed on the set of inner products, so that the set of DGPs, rather than the set of random This research was supported, in part, by grants from the Social Sciences and Humanities Research Council of Canada. It is loosely based on unpublished joint work with Stanley E. Zin, to whom I am grateful for many useful discussions.

151

152

Russell Davidson

variables, is not given a geometrical interpretation. Nevertheless, Small and McLeish's approach provides most of the geometrical elements used in this chapter. Davidson and MacKinnon (1987) introduced in®nite-dimensional statistical manifolds, with Hilbert manifold structure, in a manner similar to that used by Dawid (1975, 1977). In®nite-dimensional differential manifolds are less frequently encountered than ®nite-dimensional ones, but see Lang (1972) for an excellent account. The use of in®nite-dimensional manifolds avoids the need to limit attention to models in the exponential family. In this chapter, that Hilbert space representation is extended, and adapted for use in the context of asymptotic theory. In this chapter, estimators are de®ned in such a way as to correspond to elements of the tangent spaces to the statistical manifold at DGPs belonging to the manifold. In fact, an interpretation of these tangent spaces is given as the space of random variables with zero mean and ®nite variance under the DGP at which the space is tangent. Since a tangent space to a Hilbert manifold is itself a Hilbert space, it can, under this interpretation, be identi®ed with the subspace of Small and McLeish's Hilbert space corresponding to zero-mean random variables. The principal focus in this chapter is on estimators de®ned by the method of estimating functions, as proposed by Godambe (1960). This method is essentially equivalent to the method known in the econometrics literature as the generalised method of moments, introduced by Hansen (1982). With little effort, the results given in this chapter can be extended to Manski's (1983) closest empirical distribution class of estimators. The ef®ciency and/or robustness of an estimator is always treated relative to a statistical model, treated as a Hilbert submanifold. Since estimators estimate parameters, they are de®ned relative to a parameter-de®ning mapping de®ned on the model. A parameterised model is just the pair consisting of the model and the parameter-de®ning mapping. A major result of the chapter is that the tangent space to the underlying statistical Hilbert manifold at a DGP belonging to a parameterised model is the direct sum of three mutually orthogonal subspaces. The model, being a Hilbert submanifold, has its own tangent space at any DGP in it, this being a subspace of the full tangent space. The ®rst of the three subspaces is just the orthogonal complement of the tangent space to the model. The other two are therefore complementary subspaces of the model tangent space. Of these, one is the tangent space to the subset of the model for which the model parameters do not vary, and the other, orthogonal to it, turns out to be the ®nite-dimensional space in which (asymptotically) ef®cient estimators are located.

Efficiency and robustness

153

Robustness of an estimator with respect to a given model is interpreted as meaning that the estimator is root-n consistent for all DGPs in the model. The property of root-n consistency is shown to have a geometrical interpretation according to which the tangents that represent the estimator are orthogonal to the second subspace described above, the one that is tangent to the space over which the parameters do not vary. Quite generally, a root-n consistent estimator can be made ef®cient at any given DGP by projecting it orthogonally in Hilbert space on to the ®nitedimensional third subspace. Such orthogonal projections can be achieved by making use of a particular privileged basis of the third subspace, a basis that is easy to characterise in terms of Godambe's estimating functions. In the next section, the Hilbert manifold of DGPs is constructed, and it is shown how to adapt it for use with asymptotic theory. Then, in section 3, estimators are de®ned in a geometrical context, as also the concepts of ef®ciency and robustness of estimators. The main results pertaining to the three-subspace decomposition of the tangent space are proved in this section. Section 4 is an interlude of examples and illustrations, and in section 5 the results are specialised to estimators de®ned by estimating functions and the generalised method of moments. In section 6, the linear regression model is used as the simplest example in which the results of section 5 can be deployed, and a non-trivial application of orthogonal projection is given. Finally, concluding comments are found in section 7.

2

Data-generation processes in Hilbert space

In Davidson and MacKinnon (1987) a Hilbert space representation was introduced for the set of data-generation processes (DGPs) that could have generated a given data set. The representation used here is a slight generalisation of that presented there, in that we will not restrict ourselves to samples of i.i.d. random variables. First, it is assumed that the DGPs we are concerned with are de®ned on a measure space …ÿ; F †. A DGP corresponds to a probability measure, P say, de®ned on this space. Observed data, yn  fyt gntˆ1 , say, for a sample of size n, are interpreted as realisations of random variables on …ÿ; F †. Thus, if each observation has m components (there are m simultaneously observed dependent variables), then for each t ˆ 1; 2; . . . ; n there exists a mapping Yt : ÿ ! Rm , and for each sample size n ˆ 1; 2; . . . a mapping Y n : ÿ ! Rnm , where Yt andY n are, respectively, the random variable for observation t and the random variable for a complete sample of size n.

154

Russell Davidson

Their stochastic properties are given by the probability measure P, or, equivalently, by the measure that P induces on Rmn by the mapping Y n . A model, for a given sample size n, will be thought of as a set of DGPs, that is, as a set of probability measures on Rmn . We assume that there exists a carrier measure Pn0 on Rmn such that the measures associated with all DGPs in the model are absolutely continuous with respect to it. By the Radon±Nikodym theorem, this ensures for each DGP in the model the existence of a probability density for the random variable Y n . Consider now one single DGP in the model, and denote the density of Y n by Ln : Rmn ! R. Since this is the joint density of the Yt , t ˆ 1; . . . ; n, it can be factorised as follows: Ln …y1 ; . . . ; yn † ˆ

n Y tˆ1

Lt …yt j ytÿ1 ; . . . ; y1 †;

…1†

where Lt denotes the density of the t'th observation, Yt , conditional on all the observations before it in the ordering f1; 2; . . . ; ng, that is, the observations 1 through t ÿ 1. We may now make contact with the representation given in Davidson and MacKinnon (1987), by considering not the density (1) but its square root. Analogously to (1), we write ýn …y1 ; . . . ; yn † ˆ

n Y tˆ1

ýt …yt j ytÿ1 ; . . . ; y1 †;

…2†

where Ln …y1 ; . . . ; yn † ˆ …ýn …y1 ; . . . ; yn ††2 , with a similar relation between Lt …
† and ýt …
†, t ˆ 1; . . . ; n. By construction, ýn belongs to the Hilbert space L2 …Rmn ; Pn0 †, in fact to the unit sphere of that space, since the integral of the square of ýn with respect to dyn  dy1 dy2 . . . dyn equals one. We write Hn for this unit sphere. Usually we choose ýn and the ýt to be the non-negative square roots of n L and the Lt , but this is not necessary. Indeed, in Hilbert space, it is impossible to limit oneself to non-negative square root densities, since the non-negative cone in an in®nite-dimensional Hilbert space has an empty interior, and thus does not have a manifold structure. A consequence of this is that we cannot represent a given DGP uniquely in Hilbert space, but this does not matter for anything in this chapter. Hilbert space, on the other hand, is the natural setting for mean-square convergence, and has the considerable advantage that the information matrix ± to be de®ned later ± is a smooth tensor in this representation. This would not be so if we used, for instance, the log of the density in place of the square root density.

Efficiency and robustness

155

It is clear from (2) that a convenient way to deal with arbitrary sample sizes is to consider in®nite sequences of contributions fýt g1 tˆ1 . For any given sample size n, the joint square root density of the n observations is given by (2). For a given in®nite sequence to de®ne a DGP for each n, it is necessary and suf®cient that … jýt …yt j ytÿ1 ; . . . ; y1 †j2 dyt ˆ 1 …3† for all possible values of the conditioning variables y1 ; . . . ; ytÿ1 . We denote by S the set of sequences satisfying these conditions, and consider S as the space of DGPs for asymptotic theory, since, given any element of S, a proper probability density can be de®ned for arbitrary sample size. A model, for the purposes of asymptotic theory, will thus be a subset of S. Consider ®rst, for a given n, the tangent space to the unit sphere Hn at some DGP ýn 2 Hn . A tangent at ýn is associated with a smooth curve in Hn through ýn . Such a curve is a one-parameter family of DGPs that includes ýn . Let the curve be denoted by ýn …"†, " 2 Š ÿ 1; 1‰, and ýn …0† ˆ ýn . The tangent to this curve at ýn is then represented by the derivative of ýn …"† at " ˆ 0. The appropriate derivative in Hilbert space is a mean-square derivative, …ýn † 0 2 L2 …Rmn ; Pn0 †, say, that satis®es ÿ ÿ ÿ ÿ1 n n n 0ÿ ÿ lim …ý …"† ÿ ý …0†† ÿ …ý † ÿ ˆ 0; …4† "!0ÿ " where k
k is the Hilbert space norm in Hn . Consider next a curve in S through the point ý  fýt g1 tˆ1 . Denote the curve by ý…"†, and, for each n, we have a curve in Hn given by ýn …"† ˆ

n Y tˆ1

ýt …"†:

In order to de®ne the tangent to the curve ý…"†, and for the purposes of asymptotic theory more generally, it is more convenient to consider not the sequence fýn …"†g for a ®xed ", but rather the sequence fýn …nÿ1=2 "†g1 nˆ1 : On differentiating with respect to " at " ˆ 0, this gives the following representation for the tangent to ý…"† at ý: fnÿ1=2 …ýn † 0 g1 nˆ1 ; n 0

where each …ý † is de®ned as in (4).

…5†

156

Russell Davidson

The reason for the factor of nÿ1=2 is that we can now de®ne the norm of the sequence (5), and thus the norm of the tangent ý 0 to ý…"† at ý by the formula ký 0 k ˆ lim knÿ1=2 …ýn † 0 kHn ; n!1

…6†

where the norm of each …ýn † 0 is calculated in the Hilbert space corresponding to sample size n. The limit in (6) will be shown shortly to exist in a wide variety of circumstances. Without the factor of nÿ1=2 , this would not be the case. Another way to see why the factor is useful is to note that its use converts the curve ý…"† into what Davidson and MacKinnon (1993) call a drifting DGP, in the sense of a Pitman drift. Note that there is no obvious way to embed the contributions ýt …"† in a Hilbert space or manifold, and there is therefore no direct way to compute their derivatives with respect to ". An appropriate indirect way is as follows. Recall that ýn with a superscript refers to a product of contributions, while ýt with a subscript refers to a single contribution. Then de®ne derivatives ýt0 recursively by the relations ý10 ˆ …ý1 † 0 ;

ýtÿ1 ýt0 ˆ …ýt † 0 ÿ ýt …ýtÿ1 † 0 :

…7†

For values of …y1 ; . . . ; ytÿ1 † for which ýtÿ1 vanishes, ýt0 is arbitrarily set equal to zero. It should be clear that, whenever the ýt …"† can be differentiated in any useful sense, the derivatives will satisfy (7). With that de®nition, it is clear that the tangent ý 0 can be represented by the in®nite sequence of contributions, fýt0 g1 tˆ1 , such that, for each n, n …ýn † 0 X ýt0 ˆ : ýn ý tˆ1 t

…8†

The construction of the tangent space at the DGP ý 2 S as a Hilbert space is almost complete. Tangents are represented by in®nite sequences of contributions satisfying (8), with the norm (6). The ®nal step, needed so that (6) should be positive de®nite, is to identify tangents of zero norm with the actual zero tangent, de®ned as an in®nite sequence of zero contributions. In this way, the Hilbert space that we consider is the space of equivalence classes of in®nite sequences of contributions satisfying (8), two sequences being equivalent if the difference between them is a sequence of zero norm using the norm (6). It will be clear shortly that the different elements of equivalence classes so de®ned are asymptotically equivalent in the usual sense of asymptotic theory. The Hilbert space thus de®ned, the space of tangents to S at the DGP ý, will be denoted as TS …ý†.

Efficiency and robustness

157

It is now possible to give a statistical interpretation of the space TS …ý†. Consider a curve ý…"† and suppose that, for each n and for all admissible values of yn  …y1 ; . . . ; yn †, ýn …"; yn † is non-zero, so that log jýn …"; yn †j exists everywhere. We remarked above that the curve corresponds to a one-parameter family of DGPs, and it is clear that `n …"; yn †  2 log jýn …"; yn †j is the log-likelihood function corresponding to this oneparameter family. Further, `t …"; yt †  2 log jýt …"; yt †j is just the contribution to `n from observation t, and n X `t …"; yt †: …9† `n …"; yn † ˆ tˆ1 n

Assuming now that ` , `t and ýt can be differentiated with respect to ", we see that, by (8), n n X X @`t @`n …ýn † 0 ýt0 …0† ˆ …0† ˆ 2 n ˆ 2 : …10† @" @" ý ý tˆ1 tˆ1 t The expression second from the left above is the gradient of the loglikelihood of the one-parameter family at " ˆ 0, and, as such, its expectation under the DGP ýn is zero. In Hilbert space, this result corresponds to a simple orthogonality property, as follows. The expectation of each expression in (10), since the square of ýn is the density of yn , can be written as … … n 0 …ý † n 2 n …ý † dy ˆ 2 …ýn † 0 ýn dyn ; …11† 2 ýn and the right-hand side of this can be seen to be zero when the normalisation relation … …ýn †2 …"† dyn ˆ 1; which holds for all admissible ", is differentiated with respect to " and evaluated at " ˆ 0 to yield … …12† …ýn † 0 ýn dyn ˆ 0; which just says that the inner product in L2 …Rmn ; Pn0 † of …ýn † 0 and ýn is zero, so that …ýn † 0 and ýn are orthogonal. Geometrically, this just says that a radius of the unit sphere ± ýn ± is orthogonal to a tangent to that sphere ± …ýn † 0 . From (3), it follows that a result like (12) holds for each contribution: … ýt0 ýt dyt ˆ 0;

158

Russell Davidson

which, in terms of `t , becomes þ   … þ @`t @`t …0; yt † exp…`t …0; yt †† dyt ˆ E …0; yt † þþ ytÿ1 ˆ 0: @" @" The second equation above implies the well-known result that the sequence ( )1 n X @`t …0† @" tˆ1 nˆ1

is a martingale under ý. Now consider the norm of the tangent ý 0 , as given by (6). In order to calculate it, we need the norms, in L2 …Rmn ; Pn0 †, of the tangents …ýn † 0 . These norms are given by the formula … …  n 0 2 …ý † …ýn †2 dyn k…ýn † 0 k2 ˆ ……ýn † 0 †2 dyn ˆ ýn 0" #2 1 n X @` t ˆ Eýn @ …13† …0† A; @" tˆ1 where the last equality follows from (10). The martingale property allows (13) to be simpli®ed to ý 2 ! n X @`t : …14† …0† Eýt @" tˆ1 From (6), the squared norm of ý 0 is ý 2 ! n X @`t ÿ1 lim n …0† ; E ýt n!1 @" tˆ1 where the limit exists under mild regularity conditions allowing a law of large numbers to be applied. This limit can be interpreted as the limiting (asymptotic) variance under ý of the sequence ( )1 n X @`t ÿ1=2 n : …15† …0† @" tˆ1 nˆ1

Although (15) is derived from a triangular martingale array rather than being a martingale, we will refer to sequences like (15) as martingales, by a slight abuse of terminology. Exactly similar considerations allow us to

Efficiency and robustness

159

express the inner product of two tangents …ý1 † 0 and …ý2 † 0 in TS …ý† as the limit of the covariance of the two random variables nÿ1=2

n X @`1t tˆ1

…"1 ˆ 0† and @"1

nÿ1=2

n X @`2t tˆ1

@"2

…"2 ˆ 0†;

in obvious notation. The above considerations lead to an intuitive understanding of the Hilbert space TS …ý† we have constructed. It is the space of equivalence classes of asymptotically equivalent sequences of random variables of the form ( )1 n X ÿ1=2 ht ; hˆ n tˆ1

nˆ1

where Eý …ht j h1 ; . . . ; htÿ1 † ˆ 0;

t ˆ 1; 2; . . . ;

Eý …h2t †

ˆ t < 1; t ˆ 1; 2; . . . ; …16† P and nÿ1 ntˆ1 t converges as n ! 1 to a ®nite limiting variance. The squared norm khk2 of such a sequence is the limiting variance, and the inner product hh1 ; h2 i of two such sequences is the limiting covariance of nÿ1=2

n X tˆ1

h1t

and

nÿ1=2

n X tˆ1

h2t :

…17†

The construction depends heavily on the martingale property. On account of the variety of central-limit theorems applicable to martingales (see for instance McLeish (1974)), this property also justi®es considering limiting normal random variables to which sequences like (17) tend as n ! 1. The choice of the particular Hilbert space structure just constructed so as to de®ne the tangent space at ý confers a Hilbert manifold structure on the set S itself. It is not the aim of the present chapter to conduct a full investigation of this structure, since all the remaining analysis of the chapter will be local, and so just a few remarks will be made. It is clear that it would be necessary to group the elements of S into equivalence classes of DGPs with asymptotically equivalent properties. The regularity conditions (16) implicitly restrict the sorts of DGPs admitted into S. These are not so strong as those imposed by Hansen (1982), who worked in a stationary ergodic framework. Methods of the sort used in White and

160

Russell Davidson

Domowitz (1984) and White (1985) are presumably appropriate for determining just what restrictions are implicit in the present treatment. 3

Ef®ciency and robustness in Hilbert space

All procedures of estimation or inference treated here will be situated in the context of a particular model, that is, a subset of the set S introduced in the preceding section. If M denotes such a model, it is almost always interesting to de®ne some parameters for it. A parameterised model will therefore be a pair, of the form …M; h†, where the mapping h : M !  is termed a parameter-de®ning mapping. The set  is a ®nite-dimensional parameter space, a subset of Rk for some positive integer k. A parameterisation would go the other way, associating a DGP to each parameter vector in . Models that can be estimated by maximum likelihood constitute a very straightforward class. They are special in that a parameterisation does exist for them: for each admissible parameter vector, and for each sample size, the likelihood function gives a probability density for the dependent variables, which is precisely what we mean by a DGP. The image of the parameterisation is the model, and the parameter-de®ning mapping is the inverse of the parameterisation. Note that the inverse will not exist if the parameterisation is not one-to-one. In such cases, the model parameters are not identi®ed. A convenient way to impose identi®cation of all the parameters we consider, not just in the context of maximum likelihood models, is to require the existence of a parameter-de®ning mapping. In more general circumstances, a given parameter vector corresponds to an in®nite number of DGPs. A simple case is that of a linear regression model yt ˆ X t b ‡ ut ;

E…ut † ˆ 0;

E…u2t † ˆ  2 ;

…18†

in which the distribution of the error terms is not speci®ed past the ®rst two moments. Any mean-zero error distribution with ®nite variance can be used in combination with a ®xed parameter vector b and variance  2 . Clearly, there is an in®nite number of such error distributions. In order to bene®t from the Hilbert space structure introduced in the preceding section, it will be desirable to consider only models M that are closed submanifolds of S. Locally, in a neighbourhood of a DGP ý 2 M, this just means that, if we consider the subset of tangents at ý generated by curves that lie entirely in the subset M, this subset, denoted by TM …ý†, should be a closed subspace of the full tangent space TS …ý†. If this condition is satis®ed, then another regularity condition needed for the rest of the development can be imposed on the parameter-de®ning

Efficiency and robustness

161

mappings that may be used with M. It is that such a mapping must be a submersion (see, for instance, Lang (1972)). Among the consequences of this technical condition is that, if h denotes the parameter-de®ning mapping, open neighbourhoods of ý in M are mapped by h into open sets of the parameter space . This avoids redundant parameters: if, for instance, one parameter, 1 say, was always just twice another parameter, 2 , all points in the image of h would satisfy 1 ˆ 22 , and so the image could not be an open set. Another consequence, more important for what follows, is that TM …ý† can be expressed as the direct sum of two orthogonal subspaces, the ®rst, possibly in®nite-dimensional, corresponding to tangents to curves along which the parameters de®ned by h are constant, and the second the orthogonal complement of the ®rst, and necessarily of ®nite dimension k, where k is the number of parameters de®ned by h. A maximum likelihood model is itself of dimension k, and so the ®rst of these orthogonal subspaces contains only the zero element of TM …ý†. In general, the ®rst of the subspaces, that for which the parameters are constant, will be denoted as TM …ý; h†, and the second as E…ý; h†. These two subspaces, together with their orthogonal complement in TM …ý†, comprise the three-subspace decomposition of TM …ý† alluded to in the introduction. An estimator of the parameters of a given parameterised model is a sequence of random k-vectors h^ n which, for each n, are de®ned solely in terms of the random variable Y n of which any data set of size n is a realisation. Thus h^ n maps from Rmn to a parameter space . The estima^ The above tor characterised by the sequence fh^ n g will be written as just h. de®nition clearly contains many useless estimators; usually we will be interested only in consistent estimators. The property of consistency can be expressed as follows. For each DGP ý 2 M, we must have plimý h^ n ˆ h…ý†: n!1

The notation means that the probability limit is calculated under the DGP ý, and that the limit is what is given by the parameter-de®ning mapping h for that DGP. Most root-n consistent estimators correspond to vectors of tangents at each point of the model for which they are de®ned. Consider a DGP ý in a parameterised model …M; h†, and let h…ý† ˆ h0 be the parameter vector ^ construct the vector for ý. Then, for a root-n consistent estimator h, sequence with typical element st  t…h^ t ÿ h0 † ÿ …t ÿ 1†…h^ tÿ1 ÿ h0 †: Clearly

…19†

162

Russell Davidson

n1=2 …h^ n ÿ h0 † ˆ nÿ1=2

n X tˆ1

st :

The components of fst g may not exactly satisfy the conditions (16), but they will usually be asymptotically equivalent to sequences that do. Since such asymptotically equivalent sequences are identi®ed in our Hilbert space structure, the estimator h^ can be associated with the vector of tangents at ý de®ned by the equivalence classes containing the components of fst g. In fact, all estimators that are asymptotically equivalent to h^ are associated with the same vector of tangents. A simple illustration may be helpful here. The OLS estimator of the regression model (18) satis®es the relation ý !ÿ1 N n X X 0 0 1=2 ^ n ÿ1 XtXt nÿ1=2 X t ut …20† n …b ÿ b0 † ˆ n tˆ1

tˆ1

when the P true parameter vector is b0 . Under standard regularity conditions, nÿ1 ntˆ1 X > t X t tends to a non-random, symmetric, positive de®nite limiting matrix A, say. Thus the sequence with typical element …20† is asymptotically equivalent to the sequence ( )1 n X ÿ1=2 ÿ1 0 A X t ut ; s^  n tˆ1

nˆ1

which clearly obeys the requirements of (16). If the parameter space  is k-dimensional, we may denote the k tangents corresponding to h^ at ý by the vector s^, with typical element s^i , i ˆ 1; . . . ; k. It follows from the interpretation of the Hilbert space norm of a tangent as a variance that the k  k matrix with typical element ^ that is, h^si ; s^j i is the asymptotic covariance matrix of h, lim Var…n1=2 …h^ n ÿ h0 ††:

n!1

…21†

The notion of robustness used in this chapter can be de®ned as follows. Suppose we have two parameterised models …M0 ; h0 † and …M1 ; h1 †, where h0 and h1 map into the same parameter space , such that M0  M1 and h0 …ý† ˆ h1 …ý† for all ý 2 M0 . Then a consistent estimator h^ of the parameters of the ®rst model is said to be robust with respect to the second if it is also consistent for the second model. (Note that, since h0 : Rmn ! , it satis®es our de®nition of an estimator of …M1 ; h1 †.) Thus the OLS estimator of the regression model (18) restricted so as to have normal errors is robust with respect to the full model (18) with arbitrary error distribution satisfying the conditions on the ®rst two moments.

Efficiency and robustness

163

It may happen that the `unrestricted' model M1 has more parameters than the `restricted' model M0 . The above de®nition may still be used by limiting the parameter-de®ning mapping h1 to its projection on to those parameters that do appear in …M0 ; h0 †. For instance, the unrestricted regression y t ˆ X t b ‡ Z t c ‡ ut

…22†

contains the restricted regression yt ˆ X t b ‡ ut

…23†

as a special case, but has more parameters. In order to see if an estimator for (23) is robust with respect to (22), one just forgets about the c parameters for model (22). It then follows by standard arguments that the OLS estimator of (23) is robust with respect to (22) if and only if 0 P plimn!1 nÿ1 ntˆ1 Z t X t ˆ 0. Robustness is often thought to entail a cost in terms of the ef®ciency of an estimator. One of the chief aims of this chapter is to make explicit the trade-off between these two desirable features. Before we can do so, we need a geometrical characterisation of ef®ciency. As with robustness, ef®ciency will always be de®ned with respect to a given parameterised model …M; h†. A root-n consistent estimator h^ is ef®cient for …M; h† at a DGP ý 2 M if no other root-n consistent estimator h for …M; h† has smaller asymptotic variance under ý. Speci®cally, the difference between  given by (21), and that for h^ is a the asymptotic covariance matrix of h, positive semi-de®nite matrix. The geometrical characterisation of ef®ciency is given by the following theorem. Theorem 1 Under the regularity assumed so far, the root-n consistent estimator h^ is ef®cient for the parameterised model …M; h† at a DGP ý 2 M if and only if the tangents s^i , i ˆ 1; . . . ; k, associated with h^ belong to the space E…ý; h†. In order to prove this theorem, we will develop in a series of lemmas a number of properties of root-n consistent estimators. First, note that, if the condition of the theorem is true, the s^i , i ˆ 1; . . . ; k, span the kdimensional space E…ý; h†, since any linear dependence of the s^i would imply that the model parameters were not independent, contrary to the assumption that the parameter-de®ning mapping is a submersion. Lemma 1 The tangents si , i ˆ 1; . . . ; k, associated with a root-n consistent estimator h of the parameterised model …M; h† at a DGP ý 2 M are orthogonal to the space TM …ý; h†.

164

Russell Davidson

Proof If TM …ý; h† consists only of the zero tangent, the lemma is trivial. Otherwise, consider a curve in M through ý such that, for all points ý…"† on the curve, h…ý…"†† ˆ h…ý†. The tangent ý 0 to this curve belongs to TM …ý; h† by de®nition, and any element of TM …ý; h† can be generated by such a curve. Then, for all admissible ", the expectation of h n under ý…"† tends to h0  h…ý† as n ! 1. Suppose that the curve is expressed in terms of contributions `t …"†, as in (9), and that the tangents si correspond to components sti satisfying (16). Then we have … 0 ˆ Eý…"† …sti … yt † j ytÿ1 † ˆ exp…`t …"; yt †† sti … yt † dyt : From (19), it is clear that, since h…ý…"†† is independent of ", so too is sti along the curve ý…"†.Thus differentiating with respect to " and evaluating at " ˆ 0 gives þ   … þ @` @`t …0; yt † sti … yt † þþ ytÿ1 exp…`t …0; yt †† t …0; yt † sti … yt † dyt ˆ Eý @" @" ˆ 0: Thus the random variables @`t =@"…0† and sti have zero covariance at ý conditional on ytÿ1 . By the martingale property (compare (14)), this ÿ1=2 Pn @` =@"…0† and implies that the unconditional covariance of n t tˆ1 P nÿ1=2 ntˆ1 sti is zero, and so, letting n ! 1 gives ý ! n X @`t ÿ1=2 n 1=2  n n …24† …0; y † n …h … y † ÿ h0 † ˆ 0: lim Eý n n!1 @" tˆ1 Since the P left-hand side of this is the limiting covariance of n1=2 …h n ÿ h0 † n ÿ1=2 and n tˆ1 …@`t =@"† under ý, the typical element of …24† becomes hý 0 ; si i ˆ 0: Since ý 0 is an arbitrary element of TM …ý; h†, this completes the proof. & Lemma 1 shows that any si can be expressed as the sum of a component in E…ý; h† and a component in the orthogonal complement of TM …ý† in TS …ý†. The two terms of this sum are themselves orthogonal. According to Theorem 1, the second term must vanish for an ef®cient estimator. In fact, the ef®cient estimator will turn out to be asymptotically unique.

Efficiency and robustness

165

For the next lemma, for each j ˆ 1; . . . ; k, consider any curve ýj …"† in M that satis®es the relation h…ýj …"†† ˆ h0 ‡ "ej ;

…25†

where ej is a k-vector all the components of which are zero except for component j, which equals one. The existence of such curves is once more guaranteed by the requirement that h be a submersion. Lemma 2 For any root-n consistent estimator h characterised at the DGP ý in the parameterised model …M; h† by the tangents si , i ˆ 1; . . . ; k, and for any curve ýj …"† satisfying …25†, the inner product hýj0 ; si i ˆ ij . Proof Suppose as in the proof of Lemma 1 that the si correspond to components sti satisfying (16). From (19) and (25) it follows that @sti =@" ˆ ij along ýj …"†. Letting ýj …"† be expressed in terms of contributions …`j †t …"†, then, by exactly the same arguments as in the proof of Lemma 1, for i ˆ 1; . . . ; k, we see that þ   þ @…`j †t t t þ tÿ1 Eý …0; y † sti … y † þ y ˆ ij : @" This implies that hýj0 ; si i ˆ ij ,as required.

&

Lemma 3 At each DGP ý in the parameterised model …M; h†, there exist unique tangents s^i , i ˆ 1; . . . ; k in the space E…ý; h† such that for any root-n consistent estimator h characterised at ý by the tangents si , i ˆ 1; . . . ; k, si ˆ s^i ‡ vi , where vi belongs to the orthogonal complement of TM …ý† in TS …ý†. Similarly, for all j ˆ 1; . . . ; k and for any curve ýj …"† satisfying (25), there exist unique tangents j in E…ý; h† such that ýj0 ˆ j ‡ wj , where wj belongs to TM …ý; h†.  we know from Lemma 1 that si can be Proof For any h, expressed as a sum of a tangent in E…ý; h† and some vi in the orthogonal complement of TM …ý† in TS …ý†. This decomposition is unique, because it is orthogonal. Thus we may choose an arbitrary estimator h and de®ne the s^i by si ˆ s^i ‡ vi . Similarly, for any given set of curves ýj …"† satisfying (25), since the ýj0 lie in TM …ý†, we may de®ne the tangents j by ýj0 ˆ j ‡ wj , j 2 E…ý; h†, wj 2 TM …ý; h†. Clearly the j span E…ý; h†. By Lemma 2, we have ij ˆ hýj0 ; si i ˆ hj ‡ wj ; s^i ‡ vi i ˆ hj ; s^i i;

…26†

166

Russell Davidson

since vi , being orthogonal to TM …ý†, is orthogonal to both j and wj , and wj , being orthogonal to E…ý; h†, is orthogonal to s^i . Consider any other root-n consistent estimator characterised by tangents s~i such that s~i ˆ ti ‡ ui , ti 2 E…ý; h†, ui orthogonal to ti . Then (26) applies to the s~i , and so hj ; ti i ˆ ij : Since the j span E…ý; h†, and the ti and the s^i belong to E…ý; h† and have the same inner products with the basis vectors j , we have ti ˆ s^i , and so the s^i are unique, as claimed. The uniqueness of the j follows by an exactly similar argument starting from any other set of curves satisfying (25). & Since all the tangents in the above lemma can be represented by martingales, the results of the lemma can be expressed in terms of contributions, as follows: Eý …^sti … yt † vti … yt †j ytÿ1 † ˆ 0; and Eý …tj … yt † wtj … yt †j ytÿ1 † ˆ 0: The relations hj ; s^i i ˆ ij

…27†

can be expressed by saying that the j and the s^i constitute a pair of dual bases for E…ý; h†. This property also implies that the k  k matrix with typical element h^si ; s^j i is the inverse of the matrix with typical element hi ; j i. Since the former matrix is the asymptotic covariance matrix of the estimator h, the latter can be thought of as performing the role of the asymptotic information matrix ± in a maximum likelihood model, it would be the asymptotic information matrix in the usual sense. Since the scalar product is a smooth tensor on Hilbert space or a Hilbert manifold, it is seen that the information matrix is smooth in our Hilbert space construction. The proof of Theorem 1 can now be ®nished easily. Any estimator h^ satisfying the condition of the theorem is characterised by tangents lying in E…ý; h†, which, by the uniqueness given by Lemma 3, must be the s^i of that lemma. Any other estimator h has associated tangents of the form s^i ‡ vi . Since all the s^i are orthogonal to all the vi , the asymptotic covariance matrix of h equals the matrix of inner products of the s^i plus the matrix of inner products of the vi . Since all of these matrices are covariance matrices, they are all positive semi-de®nite, and so the difference

Efficiency and robustness

167

between the asymptotic covariance matrix of h and that of h^ is positive semi-de®nite, as required. &

4

Examples and illustrations

As a textbook example, consider the linear regression model (18) with normal errors. Since asymptotic theory is hardly necessary to treat this model, we can consider a ®nite sample size n. The model can be written in matrix notation as follows: y ˆ Xþ ‡ u;

…18†

where y and u are n  1, X is n  k, and b is k  1. We also consider the model (22): y ˆ Xþ ‡ Zc ‡ u:

…22†

As0 we saw, the OLS estimator for (18) is not robust with respect to (22) if Z X is non-zero. However the OLS estimator for (22), restricted to the parameters b, is consistent, but not ef®cient, for (18). The OLS estimator from (18) is 0

0

b^  …X X†ÿ1 X y; and the estimator of b from (22) is 0

0

b  …X M Z X†ÿ1 X M Z y; 0

…28†

0

where M Z  I ÿ Z…Z Z†ÿ1 Z is the orthogonal projection on to the orthogonal complement of the span of the extra regressors Z. It is easy to show that (see, for instance, Davidson and MacKinnon (1993), chapter 11) 0

0

b ÿ b^ ˆ …X M Z X†ÿ1 X M Z M X y;

…29†

with M X de®ned similarly to M Z . When (18) is speci®ed with normal errors, the model is ®nitedimensional, k ‡ 1-dimensional in fact, if  2 is allowed to vary. Since the log-likelihood of the model is n 1 `…b;  2 † ˆ ÿ log 2 2 ÿ 2 ky ÿ Xþk2 ; 2 2 the tangents to the curves along which just one component of b varies are represented by the k-vector of zero-mean random variables r  nÿ1=2

1 0 X u: 2

…30†

168

Russell Davidson

The only way to vary the DGP without changing the parameter vector b is to vary the error variance. Thus the space TM …ý; b† is one-dimensional in this case, and is generated by the tangent represented by ý ! n 2 X @` 1 u nÿ1=2 2 ˆ nÿ1=2 2 ÿ1 ; …31† @ 2 tˆ1  2 which has zero covariance with all the components of (30). This means that these components lie in E…ý; b†, thereby justifying the notation r. The OLS estimator b^ is associated with the tangents 0

0

s^  …nÿ1 X X†ÿ1 nÿ1=2 X u;

…32†

which are seen immediately to be linear combinations of the components of r in (30). The tangents s^ therefore also lie in E…ý; b† and so b^ is seen to be asymptotically ef®cient. Note also that the matrix of inner products of the components of r and s^ is the expectation of nÿ1=2

0 1 0 ÿ1=2 0 X un u X…nÿ1 X X†ÿ1 ˆ I; 2 

con®rming the dual basis property (27). The tangents corresponding to the estimator (28) are seen, from (29), to be 0

0

s ˆ s^ ‡ …nÿ1 X M Z X†ÿ1 nÿ1=2 X M Z M X u: It is simple to check that the covariances of the second term of this with the components of (32) are zero, as also with (31), which represents the tangent that generates TM …ý; b†. Thus this second term represents a tangent orthogonal to all of TM …ý†, as required by the theory of the preceding section. As a slightly less trivial example, consider again the regression model (18), without imposing the normality of the error terms. The OLS estimator is of course robust for any model at all that satis®es the regression equation with zero-mean errors, but it is interesting to enquire under what conditions it is also ef®cient. Consider a parameterised model …M; b† the DGPs of which satisfy (18), but do not necessarily have normal errors. The OLS estimator is still characterised by the tangents (32), and its robustness implies, by Lemma 1, that these tangents are orthogonal to TM …ý; b† for all ý 2 M. Consequently, the estimator is ef®cient at a given ý if M is large enough to contain the tangents (32) in its tangent space TM …ý† at ý, since then, being orthogonal to TM …ý; b†, they must belong to E…ý; b†. Although it is dif®cult to state a precise condition that will guarantee this property,

Efficiency and robustness

169

intuitively it can be seen that the model must include the case of normal errors. It can be checked that, if the model speci®es the error distribution, up to a scale factor, then only normal errors are compatible with the ef®ciency of the OLS estimator. Suppose that the error density, scaled to have unit variance, is denoted by f . Then the log-density of observation t of the model (18) is    1 yt ÿ X t b : f log   The tangent corresponding to a variation of þi is then represented by n X 1 f 0 …et † Xti ÿ nÿ1=2 ;  f …et † tˆ1

where et  …yt ÿ X t b†=. If the tangents s^ given by (32) are linear combinations of those above, for i ˆ 1; . . . ; k, then it is necessary that f 0 …et † ˆ cet ; f …et †

…33†

for some constant c independent of t. The general solution to the differential equation (33) is f …e† ˆ C exp…ce2 =2†; (C another constant) and, since this density must have mean zero and unit variance, it must be the standard normal density, with C ˆ …2†ÿ1=2 , and c ˆ ÿ1. In general, if one wishes to improve the precision of a parameter estimate, more information of some sort is necessary. Such information may take the form of the true value of some other parameter, or the true nature of the error density, or the like. When models are considered as sets of DGPs, this sort of information corresponds to a reduction of the model size, since only DGPs satisfying the constraints imposed by the new information can belong to the model. Then ef®ciency gains are possible because estimators that would not have been robust with respect to the original model may be so for the reduced model. In some circumstances, though, this is not so, in which case the extra information is uninformative concerning the parameters. These general considerations can be illustrated geometrically. Consider ®gure 5.1, which represents the space E…ý; h† for . some parameterised model as a two-dimensional space, with h ˆ ‰þ .. ÿŠ. The origin corresponds to the DGP ý, at which it is supposed that þ ˆ þ0 , ÿ ˆ 0. The

170

Russell Davidson

Figure 5.1 Ef®ciency gain from added information

dual bases f^sþ ; s^ÿ g and fþ ; ÿ g are drawn, and it can be seen that s^þ is orthogonal to ÿ , and s^ÿ to þ . The tangent ÿ gives the direction in which only ÿ varies, and so it is labelled þ ˆ þ0 . Similarly, þ is labelled ÿ ˆ 0. Now suppose that we are provided with the information that ÿ ˆ 0. The model must now be restricted to DGPs that satisfy that property. The two-dimensional E…ý; h† depicted in the ®gure is reduced to the onedimensional line in the direction of þ , that being the direction in which ÿ remains constant at zero. But s^þ does not belong to the one-dimensional E…ý; þ†, and so is no longer ef®cient for the constrained model. The ef®cient estimator for that model is obtained by projecting s^þ orthogonally on to the direction of þ , using the projection denoted by P þ in the ®gure. This gives rise to a new consistent estimator associated with s~þ  Pþ s^þ . Since s~þ is obtained from s^þ by an orthogonal projection, it is of smaller norm or, in statistical terms, of smaller asymptotic variance. In addition, the orthogonal projection means that s~þ has the same inner product with þ as does s^þ , and so it satis®es the condition of Lemma 2 for a consistent estimator. The result of Lemma 1 is also seen to be satis®ed: the inef®cient estimator s^þ for the constrained model equals the ef®cient estimator s~þ plus something orthogonal to the constrained model. If ÿ is a `nuisance' parameter, the value of which is used only to improve the precision of the estimate of þ, then it could have been left out of the parameter-de®ning mapping of the original, unreduced, model. If so, the omission of ÿ once more leads to a one-dimensional E…ý; þ†, but this time in the direction of s^þ . This is because E…ý; þ† must be orthogonal to all directions in which þ does not vary, now including the direction of ÿ . This time, it is þ that is projected orthogonally on to the direction

Efficiency and robustness

171

of s^þ to yield  þ , which replaces þ for the model with ÿ dropped. This orthogonal projection, which means that  þ has smaller norm than þ , corresponds to a reduction in information about þ. Notice that the estimator s^þ is unchanged whether or not ÿ is dropped. The information gain moving from þ to þ is not realised so long as þ and ÿ are estimated jointly, and is realised only when information about ÿ is available. If þ and ÿ were orthogonal, so would be s^þ and s^ÿ , and the directions of þ and s^þ would coincide, as would those of ÿ and s^ÿ . Redrawing ®gure 5.1 to re¯ect this state of affairs shows that information about ÿ no longer leads to any gain in the precision of the estimate of þ. This is perfectly intuitive, since the orthogonality means that the asymptotic covariance matrix of the parameter estimates is diagonal. A simple example of such orthogonality is provided by the linear regression model (18) with normal errors, for which the tangents (30) corresponding to variation of the parameters b of the regression function are orthogonal to the tangent (31), which corresponds to variation of  2 . As is well known, knowledge of the value of the error variance is uninformative about b. In much the same way, it was seen above that if normal errors are not assumed in (18), then, if it were learnt that the errors were in fact normal, this new information would not lead to any gain as regards estimation of b, since the OLS estimator remains ef®cient for normal errors. 5

Estimating functions and GMM

The generalised method of moments (GMM) was proposed by Hansen (1982), apparently without knowledge of a very similar method proposed by Godambe (1960); see also Godambe and Thompson (1989) and Godambe (1991). It is convenient to refer to Godambe's method as the method of estimating functions . Both approaches start from what in the estimating function context are called elementary zero functions , which are functions ± at least one for each observation of the sample ± of both data and parameters. When these functions are evaluated at the correct values for any given DGP, their expectation under that DGP is zero. The simplest example is, as usual, the linear regression model (18), for which the elementary zero functions are the yt ÿ X t b, one for each observation t. Specifying a set of elementary zero functions is very similar to specifying a model and a parameter-de®ning mapping. Suppose that, for each observation t, the elementary zero functions are written as f t … yt ; h†, where, as before, the argument yt 2 Rmt corresponds to the observed data in observations 1 through t, and h is a k-vector of parameters. The p-vector-valued function f t will usually depend on explanatory vari-

172

Russell Davidson

ables (covariates), hence the index t. The natural way to proceed is to specify the model as the set of those DGPs ý for which there exists a unique parameter vector h such that Eý … f t …Y t ; h†† ˆ 0. (Recall that Y t is the random variable of which observations 1 through t are a realisation.) The parameter-de®ning mapping then maps ý to this unique h. The above way of de®ning a parameterised model needs to be quali®ed somewhat, for a number of reasons. The ®rst is that, in order to perform inference, it is necessary to be able to estimate not only the parameter ^ vector h, but also the asymptotic covariance matrix of the estimator h, and, for this, one needs the existence of higher moments. It would therefore be preferable to limit the model to those DGPs for which those higher moments exist. The second reason reveals a dif®culty that arises whenever a model, parameter-de®ning mapping or estimation method makes use of moments, that is expectations. It is that, in any commonly used stochastic topology, including the one used here based on Hilbert space norms (see Billingsley (1968, 1979)), expectations of unbounded random variables are not continuous functions of the DGP under which they are calculated. For instance, even the smallest admixture of the Cauchy distribution with the normal is enough to destroy the existence of the ®rst moment. The unfortunate consequence is that, if a model is de®ned by moments, it will not be a smooth submanifold of the overall set of DGPs, S. The lack of continuity of moments is a problem for establishing appropriate regularity conditions in many contexts, not just the geometrical one. For present purposes, the easiest solution is just to require that the elementary zero functions f t … yt ; h† should be bounded functions of yt . Of course, this assumption excludes most interesting models, even the linear regression model, but, since the emphasis of this chapter is geometrical, it does not seem worthwhile to look further for more suitable regularity conditions. In particular, imposing the existence of moments on a model is not informative about that model's parameters. This can be seen by considering a very simple problem, namely that of estimating the mean of a set of scalar i.i.d. observations. If these observations may take values anywhere in an unbounded set, then the set of DGPs de®ned by requiring that the observations be i.i.d. drawings from a distribution for which the mean exists is not a smooth submanifold of S. However, the set is dense in such a submanifold. To see why, consider the Hilbert space L2 …R† in which the unit sphere represents all univariate densities de®ned on R. Then, for ý in this unit sphere, the mean of the density to which ý corresponds, if it exists, is

Efficiency and robustness

…1 ÿ1

jý…y†j2 y dy:

173

…34†

The integral above de®nes an unbounded quadratic operator on L2 …R†, the domain of which is dense in L2 …R†. In other words, the densities for which the mean exists are dense in the unit sphere of L2 …R†. It is straightforward to extend this univariate result to the asymptotic Hilbert space S. Clearly the model implicitly de®ned by the problem of estimating the mean is just the set of all i.i.d. sequences, and this set does constitute a smooth submanifold because the requirement that all the observations be i.i.d. can be expressed by the relations ýt ˆ ý, for some ý independent of t ˆ 1; . . . ; and these relations are trivially continuously differentiable in the Hilbert space norm of L2 …R†. The set of DGPs in this model for which the mean actually exists is a dense set, so that its closure is the full submanifold. However, any information gain that could lead to increased precision of the estimate of the mean must involve, as we saw above, a reduction in the dimension of the model. Since we have seen that imposing a ®nite mean does not reduce the dimension, no information gain is possible from the knowledge that the mean exists. Any expectation can be expressed as an integral similar to (34), and can therefore be used to de®ne an unbounded operator on the Hilbert spaces for ®nite samples. Thus the argument above generalises to all models de®ned using the expectations of unbounded random variables, and so, for the purposes of geometrical discussions of ef®ciency and robustness, we must limit ourselves to models de®ned in terms of the expectations of bounded random variables, for instance, variables obtained by censoring unbounded variables above some suitably high threshold. Suppose then that we de®ne a parameterised model …M; h† by a set of elementary zero functions given by the components of the p-vector f t … yt ; h†, as above. Suppose further that the parameter-de®ning mapping h thus implicitly de®ned is in fact de®ned for all ý 2 M, so that the identi®cation condition is satis®ed, and that h is a submersion, as we required earlier. Consider any ý 2 M and suppose that h…ý† ˆ h0 . Then, for each component fti , i ˆ 1; . . . ; p, of f t , Eý …fti …Y t ; h0 †† ˆ 0, and so in some circumstances it may be that the sequence nÿ1=2

p n X X tˆ1 iˆ1

ati … ytÿ1 †fti … yt ; h0 †

…35†

represents a vector of tangents at ý, where the ati … ytÿ1 † are predetermined ÿ1 Pn Pp 2 at t, and such that limn!1 n tˆ1 iˆ1 Eý …ati † is ®nite. This may equally well not be so, because, since only the unconditional expectations of the zero functions must vanish, the sequence may not be a martingale,

174

Russell Davidson

as required by the ®rst equation of (16). For the moment, we suppose that the martingale property is satis®ed. Then we have: Lemma 4 For a parameterised model …M; h† de®ned by a set of elementary zero functions f t … yt ; h† obeying the above regularity conditions and such that, for all ý 2 M, the sequence (35) is a martingale, the tangents represented by the components of (35) are orthogonal to TM …ý; h†, the space of tangents at ý that correspond to curves within the model along which the parameters are constant at h0 . Proof As in the proof of Lemma 1, consider a curve ý…"† in TM …ý; h†, represented by the log-density contributions `t …"†. Then, as in Lemma 1, … @` …36† fti … yt ; h0 † t …"; yt † exp…`t …"; yt †† dyt ˆ 0: @" On multiplying by at , the martingale property implies the result.

&

Lemma 4 is the geometrical expression of the fact that, under suitable regularity conditions, the parameters h can be consistently estimated by solving any k linearly and functionally independent equations of the form p n X X tˆ1 iˆ1

^ ˆ 0; ati …Y tÿ1 †fti …Y t ; h†

…37†

where the ati , t ˆ 1; . . . ; n, i ˆ 1; . . . ; p, are predetermined at t. Standard arguments based on a short Taylor expansion (see, for instance, Davidson and MacKinnon (1993), chapter 17) show that, asymptotically, the components of the sequence n1=2 …h^ ÿ h0 † are linear combinations of the tangents represented by nÿ1=2

p n X X tˆ1 iˆ1

ati … ytÿ1 †fti … yt ; h0 †;

…38†

provided that a law of large numbers can be applied to the sequences  1 @fti t …Y ; h0 † : @j tˆ1 Although this point will not be developed here, regularity conditions like these are the algebraic counterparts of the geometrical regularity conditions, involving identi®ability and the submersion property, discussed above. For further discussion, see Newey and McFadden (1994).

Efficiency and robustness

175

Just as with the tangents si used in Lemma 1, the tangents (38) can be expressed as the sum of two orthogonal components, one in the k-dimensional space E…ý; h†, and the other orthogonal to the model M. The ®rst component corresponds to the asymptotically ef®cient estimator, and so, in order to ®nd an ef®cient estimator, we wish to project the tangents (38) orthogonally on to E…ý; h†. Intuitively, this orthogonal projection is on to the model M itself, since the tangents are already orthogonal to TM …ý; h†, the orthogonal complement of E…ý; h† in the tangent space to the model M. We can perform the orthogonal projection by expressing the unique tangents j , j ˆ 1; . . . ; k, de®ned in Lemma 3, in the form (38). As seen in the proof of that lemma, we can compute the inner product of any tangent of the form (38) with j by considering a curve satisfying (25), since the tangent to such a curve equals j plus a component orthogonal to everything like (38). These inner products are given by the following lemma. Lemma 5 For a parameterised model …M; h† de®ned by a set of elementary zero functions f t … yt ; h† obeying the regularity conditions of Lemma 4, the tangent j at DGP ý 2 M corresponding to component j ˆ 1; . . . ; k of h can be represented by the sequence of contributions tj 

p X iˆ1

tij fti …h0 †:

Further, for all i ˆ 1; . . . ; p, j ˆ 1; . . . ; k, þ   þ @f Eý …tj fti …h0 † g j ytÿ1 † ˆ ÿEý ti …h0 † þþ ytÿ1 : @j

…39†

…40†

If the covariance matrix of f t … yt ; h0 † under ý, conditional on ytÿ1 , is Xt … ytÿ1 †, then þ   p X @ftl t þþ tÿ1 tÿ1 tij … ytÿ1 † ˆ ÿ y …Xÿ1 … y †† E … y † : …41† t il ý þ @j lˆ1 Proof The ®rst statement of the lemma, (39), simply requires the equations that de®ne the asymptotically ef®cient estimator to take the general form (37). For this to be false, there would need to exist consistent estimators de®ned by equations that did not take this form. But the model is de®ned by the expectations of the elementary zero functions, and expectations of non-linear functions of these will not in general be

176

Russell Davidson

zero. Thus a consistent estimator cannot be de®ned using non-linear functions of the elementary zero functions, and so (39) is true. For (40), consider, as in Lemma 2, a curve ýj …"† in M satisfying (25). Then we have, for i ˆ 1; . . . ; p, j ˆ 1; . . . ; k, t ˆ 1; . . . ; and all admissible ", … exp……`j †t … yt ; "††fti … yt ; h0 ‡ "ej † dyt ˆ 0; and so, on differentiating with respect to ", and setting " ˆ 0, we get þ þ     þ þ @…`j †t …0† @fti fti …h0 † þþ ytÿ1 ˆ ÿEý Eý …h0 † þþ ytÿ1 : @" @j By Lemma 3 and the remark following it, we may replace @…`j †t …0†=@" in the left-hand side above by the contribution tj , thus yielding (40). Substituting the expression for tj in (39) into (40) gives þ   p X þ @f tlj Eý …ftl …h0 † fti …h0 †j ytÿ1 † ˆ ÿEý ti …h0 † þþ ytÿ1 ; @j lˆ1 from which (41) follows, since by de®nition …Xt †li ˆ Eý …ftl …h0 † fti …h0 † j ytÿ1 †:

&

The following theorem now follows immediately from Lemma 5. Theorem 2 Let the parameterised model …M; h† be de®ned by means of the set of bounded elementary zero functions f t … yt ; h† with the restriction that, for all ý 2 M, the sequences fti …Y t ; h0 †, i ˆ 1; . . . ; p, satisfy the conditions of (16) under ý. De®ne the sequences of random variables ti by (39), with the coef®cients tij … ytÿ1 †, predetermined at t, given by (41). Then the estimator h^ obtained by solving the equations p n X X tˆ1 iˆ1

^ ˆ 0; tij …Y tÿ1 †fti …Y t ; h†

…42†

for j ˆ 1; . . . ; k, is asymptotically ef®cient for …M; h†. Proof As mentioned above, h^ can be expressed asymptotically as a linear combination of the tangents (38) with the ati replaced by tij . By Lemma 5, these tangents belong to E…ý; h† for all ý 2 M. By Theorem 1, h^ is asymptotically ef®cient. &

Efficiency and robustness

177

The conditions (16) are quite essential for Theorem 2, in particular the martingale condition. However, if the elementary zero functions do not satisfy that condition, it is often possible to ®nd linear combinations, gt … yt ; h† of the f s … ys ; h†, s ˆ 1; . . . ; t, that do. The transformation from the f t to the gt is analogous to the transformation used to estimate models by GLS rather than OLS. For instance, suppose that there is just one elementary zero function ft …h† per observation (that is, p ˆ 1), and denote the covariance matrix of the ft , for sample size n, by the n  n matrix V. V may depend on h, and possibly on other parameters as well, such as autocorrelation coef®cients. Let / denote the complete set of parameters, and let /0 be the parameter values for the DGP ý. In addition, since we are interested in the conditional covariance structure, V ts , if t  s, may depend on Y sÿ1 . Then if the lower-triangular matrix P…/† is such that P > …/†P…/† ˆ V ÿ1 …/†, we may form the vector of zero functions g…/† ˆ P…/†f …h†, where f is n  1, with typical element ft . Note that P ts …/† is non-zero only if t  s, and in that case it may depend on ysÿ1 . The covariance matrix of g…/† is then just the identity matrix, and the martingale condition is satis®ed. In order to obtain the optimal estimating equations, we use the relation þ þ   X   t þ þ @gt t @fs s … y ; /0 † þþ ytÿ1 ˆ P ts … ysÿ1 ; /†Eý … y ; h0 † þþ ysÿ1 ; Eý @j @j sˆ1 which holds since þ   þ @P ts sÿ1 s Eý … y ; /†fs … y ; h0 † þþ ysÿ1 @j @P ˆ ts … ysÿ1 ; /†Eý …fs … ys ; h0 † j ysÿ1 † ˆ 0: @j Thus the equations that de®ne the asymptotically ef®cient estimator are obtained from (42) with gt in place of fti (the index i is omitted since we have assumed that p ˆ 1), and t1j is de®ned by (41) with g in place of f . Because p ˆ 1, Xt is just a scalar, equal to one, since the covariance matrix of g is the identity matrix. Putting all this together gives as the estimating equation þ   n X n X @ft ^ þþ tÿ1 ^ s …Y s ; /† ^ ˆ 0: V…/†f Eý^ …/† þ Y …43† @ j tˆ1 sˆ1 The notation is intended to indicate that the conditional expectation of @ft =@j must be estimated in some manner ± we need not specify details,

178

Russell Davidson

since many procedures exist. The result (43) is standard in the estimating functions literature, and can be found, for instance, in Godambe (1960) and Godambe and Thompson (1989). All of the results of this section can be extended quite simply to the sort of model usually found in the context of the generalised method of moments. Such models are still de®ned in terms of elementary zero functions f t … yt ; h†, but the requirement that these have zero mean is strengthened so as to require that their means conditional on some set of random variables be zero. More formally, let F t , t ˆ 1; . . . ; be a nested set of sigma-algebras, with F tÿ1  F t , and such that Y t 2 F t . Then the condition on the zero functions becomes Eý …fti …Y t ; h0 †j F tÿ1 † ˆ 0; where as usual h0 ˆ h…ý†, t ˆ 1; . . . ; and i ˆ 1; . . . ; p. An equivalent way of expressing the condition is to require that, for all random variables htÿ1 2 F tÿ1 , the unconditional expectation of htÿ1 fti …h0 † be zero. Lemma 5 can be applied to all of these new zero functions, and it follows that (40) and (41) are now true with the expectations conditional on F tÿ1 rather than just on Y tÿ1 . In addition, Theorem 2 continues to hold with the tij de®ned by the modi®ed (41). This result is most often expressed in terms of the optimal instruments for GMM estimation (see, for instance, Davidson and MacKinnon (1993), chapter 17).

6

The linear regression model

Despite its simplicity, the linear regression model (18) can be used to illustrate most of the theory of the preceding section. The elementary zero functions, one per observation, are given by ft …yt ; b† ˆ yt ÿ X t b:

…44†

In order to use (41), we compute @ft =@þj ˆ ÿXtj . Thus, if the ft are homoskedastic, and provided that Xtj 2 F tÿ1 , it follows from Theorem 2 that solving the estimating equations n X tˆ1

Xtj …yt ÿ X t b† ˆ 0;

j ˆ 1; . . . ; k;

yields an asymptotically ef®cient estimator, namely the OLS estimator, as required by the Gauss±Markov theorem. In case of heteroskedasticity, if E……yt ÿ X t b†2 † ˆ t2 , the estimating equations are

Efficiency and robustness

179

n X 1 X …y ÿ X t b† ˆ 0; 2 tj t  tˆ1 t

and they yield the Aitken GLS estimator. If the explanatory variables X t are endogenous and do not belong to F tÿ1 , then the estimating equations are, assuming homoskedasticity, n X tˆ1

E…Xtj j F tÿ1 † …yt ÿ X t b† ˆ 0:

In simultaneous equations models, the expectations of endogenous explanatory variables can be expressed in terms of exogenous instrumental variables: the equation above then de®nes an instrumental variables estimator with optimal instruments. It is clear that, if the model M includes heteroskedastic as well as homoskedastic DGPs, then there will be no estimator that is at the same time robust with respect to the whole model and ef®cient at every DGP in the model, unless there is a way of consistently estimating the variances t2 . This will be the case whenever feasible GLS can be used, but not more generally. In section 4, it was seen that, in regression models in which the error density is speci®ed, the OLS estimator is ef®cient only if that density is normal. It is of interest to see if an ef®ciency gain with respect to OLS can be realised when the density is not normal, but is nonetheless of unknown form. We will now derive an estimator that can be used with homoskedastic errors, is robust against non-normal error densities, and is more ef®cient than OLS in some cases of non-normality. This estimator, which was recently proposed by Im (1996) using arguments somewhat different from those here, can be derived directly using the theory of the preceding section. We rename the zero function (44) as ut …yt ; b†, and introduce a new zero function and a new parameter by the relation vt …yt ; b;  2 † ˆ u2t …yt ; b† ÿ  2 ; where the homoskedasticity assumption is made explicit in terms of the error variance  2 . It will also be assumed that the vt are homoskedastic, and that the expectation of ut vt does not depend on t. If this last assumption does not hold, the estimator we are about to derive is still robust, but is no longer ef®cient. Analogously to (39), tangents that span the ef®cient space for the present model can be de®ned by the contributions

180

Russell Davidson

ti ˆ ati ut ‡ bti vt ;

i ˆ 1; . . . ; k; and

…45†

t ˆ at ut ‡ bt vt ;

where ati , bti , at and bt are exogenous or predetermined at t. Now by (40) we have   @ut ˆ Xti ; E…ti ut † ˆ ÿE @þi   @vt ˆ 2E…Xti ut † ˆ 0; E…ti vt † ˆ ÿE @þi   @ut E…t ut † ˆ ÿE ˆ 0; and @ 2   @vt ˆ 1: E…t vt † ˆ ÿE @ 2 Thus, on substituting the de®nitions (45) into the above, we obtain the following equations for the ati , etc.: ati  2 ‡ bti ÿ ˆ Xti ; ati ÿ ‡ bti  ˆ 0; at  2 ‡ bt ÿ ˆ 0; at ÿ ‡ bt  ˆ 1; where ÿ  E…u3t † and   E…u4t † ÿ  4 are independent of t by assumption, and equal 0 and 2 4 under normality. Letting  ˆ  2 ÿ ÿ 2 =, we ®nd that ati ˆ Xti ;

bti ˆ ÿ…ÿ=†Xti ;

at ˆ ÿÿ=;

bt ˆ  2 =:

Thus, according to Theorem 2, the estimating equations for b are n   X ÿ Xti ut ÿ …u2t ÿ  2 † ˆ 0:  tˆ1

…46†

If ÿ is known to be zero, that is, if the error density is not skewed, these equations givePthe OLS estimator. Otherwise, ÿ can be consistently estiP mated by nÿ1 ntˆ1 u^ 3t , and  by nÿ1 ntˆ1 u^ 4t ÿ ^4 , where u^ t and ^2 can be obtained by, for example, OLS. The procedure suggested by Im (1996) makes use of the arti®cial regression yt ˆ X t b ‡ …u^2t ÿ ^2 † ‡ residual;

Efficiency and robustness

181

where  is an auxiliary parameter. A little algebra shows that the OLS estimate of b from this is the solution of (46). Im shows, both theoretically and by Monte Carlo simulation, that his estimator, which he calls RALS (for residuals augmented least squares) is more ef®cient than OLS when the error terms are skewed. In fact, he goes further and, by introducing at third zero function wt …yt ; b;  2 ; ÿ† ˆ u3t …yt ; b† ÿ 3 2 ut ÿ ÿ; in which ÿ, as de®ned above, becomes an explicit parameter, shows that further ef®ciency gains relative to OLS are available if the errors have non-normal kurtosis. The approach of the preceding paragraph can again be used to derive the explicit form of this estimator. As Im points out, estimators of this sort are constructed in the spirit of adaptive estimation (see, for instance, Newey (1988)). 7

Concluding remarks

In this chapter, geometrical characterisations have been given of ef®ciency and robustness for estimators of model parameters, with special reference to estimators de®ned by the method of estimating functions and/or the generalised method of moments. It has been shown that, when a parameterised model is considered as a Hilbert manifold in an underlying space of DGPs, the tangent space at any DGP of the model can be expressed as the direct sum of three mutually orthogonal subspaces. Consistent estimators of the model parameters all have the same component in one of these subspaces, which is ®nite-dimensional, with dimension equal to the number of parameters. This space contains the asymptotically ef®cient estimator. Inef®cient estimators also have a non-vanishing component in the orthogonal complement of the tangent space to the model, and they thus lose ef®ciency by including random variation in directions excluded by the speci®cation of the model. Information about parameters is represented geometrically by tangents that lie within the tangent space to the model. There is a unique tangent in the ®nite-dimensional ef®cient subspace that corresponds to the variation of each parameter, and the tangent to any curve along which that parameter alone varies is the sum of this unique tangent and a component in the tangent subspace in which the model parameters do not vary. These information tangents form a basis of the ef®cient subspace that is dual to that provided by the ef®cient estimators. Ef®cient estimating equations for model parameters can be obtained by projecting arbitrary root-n consistent estimators on to the ef®cient subspace. Lemma 5 provides a simple method of performing this sort

182

Russell Davidson

of projection. As seen in the example of the RALS estimator, the projection can often be implemented by an arti®cial regression. More generally, as shown in Davidson and MacKinnon (1990) in the context of fully parameterised models, arti®cial regressions can be used in many onestep ef®cient estimation procedures that are equivalent to projection on to the ef®cient subspace. The theory of this chapter suggests that arti®cial regressions can be developed to perform such projections in greater generality. One-step estimators of a seemingly different sort have been proposed recently by Imbens (1997), and it is claimed that their ®nite-sample properties are substantially better than those of conventional, asymptotically ef®cient, GMM estimators. Although these estimators are not implemented by arti®cial regression, they are of course the result of implicit projection on to the ef®cient subspace. Another asymptotically ef®cient estimation method with ®nite-sample properties different from those of GMM has been proposed by Kitamura and Stutzer (1997), based on minimisation of the Kullback±Leibler information criterion. It seems probable that this minimisation is another asymptotically equivalent way of projecting on to the ef®cient subspace. It is hoped that the geometrical construction laid out in this chapter will serve as a uni®ed framework for the discussion of asymptotic ef®ciency and robustness. References Amari, S. (1990), Differential-Geometrical Methods in Statistics, 2nd edn, Lecture Notes in Statistics No. 28, Berlin: Springer Verlag. Barndoff-Nielsen, O.E. and D.R. Cox (1994), Inference and Asymptotics, Monographs on Statistics and Applied Probability No. 52, London: Chapman & Hall. Barndorff-Nielsen, O.E., D.R. Cox and N. Reid (1986), `The Role of Differential Geometry in Statistical Theory', International Statistical Review, 54: 83±96. Billingsley, P. (1968), Convergence of Probability Measures, New York: John Wiley. (1979), Probability and Measure, New York: John Wiley. Chentsov, N.N. (1972), Statistical Decision Rules and Optimal Inference (in Russian), Moscow: Nauka. English translation (1982), Translations of Mathematical Monographs, vol. 53, Providence, RI: American Mathematical Society. Davidson, R. and J.G. MacKinnon (1987), `Implicit Alternatives and the Local Power of Test Statistics', Econometrica, 55: 1305±1329. (1990), `Speci®cation Tests Based on Arti®cial Regressions', Journal of the American Statistical Association, 85: 220±227.

Efficiency and robustness

183

(1993), Estimation and Inference in Econometrics, New York: Oxford. Dawid, A.P. (1975), Discussion of B. Efron (1975), `De®ning the Curvature of a Statistical Problem', Annals of Statistics, 3: 1189±1242. (1977), `Further Comments on a Paper by Bradley Efron', Annals of Statistics, 5: 1249. Godambe, V.P. (1960), `An Optimum Property of Regular Maximum Likelihood Estimation', Annals of Mathematical Statistics, 31: 1208±1212. (ed.) (1991), Estimating Functions, Oxford: Clarendon Press. Godambe, V.P. and M.E. Thompson (1989), `An Extension of Quasi-likelihood Estimation' (with discussion), Journal of Statistical Planning and Inference, 22: 137±172. Hansen, L.P. (1982), `Large Sample Properties of Generalized Method of Moments Estimators', Econometrica, 50: 1029±1054. Im, Kyung So (1996), `Least Square Approach to Non-Normal Disturbances', DAE Working Paper No. 9603, University of Cambridge. Imbens, G.W. (1997), `One-Step Estimators for Over-Identi®ed Generalized Method of Moments Models', Review of Economic Studies, 64: 359±383. Kass, R.E. (1989), `The Geometry of Asymptotic Inference (with Discussion)', Statistical Science, 4: 187±263. Kitamura, Y. and M. Stutzer (1997), `An Information-Theoretic Alternative to Generalized Method of Moments Estimation', Econometrica, 65: 861±874. Lang, S. (1972), Differential Manifolds, Reading, Mass.: Addison-Wesley; reprinted 1985. McLeish, D.L. (1974), `Dependent Central Limit Theorems and Invariance Principles', Annals of Probability, 2: 620±628. Manski, C.F. (1983), `Closest Empirical Distribution Estimation', Econometrica, 51: 305±320. Murray, M.K. and J.W. Rice (1993), Differential Geometry and Statistics, Monographs on Statistics and Applied Probability No. 48, London: Chapman & Hall. Newey, W.K. (1988), `Adaptive Estimation of Non-Linear Models', Journal of Econometrics, 38: 301±339. Newey, W.K. and D. McFadden (1994), `Estimation in Large Samples', in D. McFadden and R.F. Engle (eds.), Handbook of Econometrics, vol. 4, Amsterdam: North-Holland. Small, C.G. and D.L. McLeish (1994), Hilbert Space Methods in Probability and Statistical Inference, New York: Wiley-Interscience. White, H. (1985), Asymptotic Theory for Econometricians, New York: Academic Press. White, H. and I. Domowitz (1984), `Nonlinear Regression with Dependent Observations', Econometrica, 52: 143±161.

6

Measuring earnings differentials with frontier functions and Rao distances Uwe Jensen

1

Introduction

In a seminal paper Rao (1945) introduced the concept of `Geodesic Distance' (or `Rao Distance') into statistics. This concept has important theoretical properties and is based on the demanding differentialgeometrical approach to statistics. These mathematical requirements and dif®culties in its application are responsible for the low level of familiarity by econometricians with this generalisation of the wellknown Mahalanobis distance. Econometricians require some detailed knowledge of Riemannian geometry to gain a complete understanding of Rao distances. Section 2 provides a short introduction to the necessary theory on Rao distances, hyperbolic curvature, isocircles and Rao distance tests. Since their original development in the paper by Aigner and Chu (1968), frontier functions have served almost exclusively for estimating production and cost functions consistently with an economic theory of optimising behaviour. In section 3 of this chapter, an extended human capital model is estimated as a stochastic earnings frontier with data from the German socio-economic panel. This provides a deeper interpretation of the deviations of observed income from estimated income. Distinguishing between `potential human capital' and `active human capital ' accounts for the partial failure of human capital models when estimated as average functions. There has been a tendency in Germany for the number of years of education attained to increase, in part owing to the combination of an Thanks for helpful comments to GuÈnter Coenen, Gerd Hansen, Christof Helberger, Norbert Janz, Stefan Mittnik, Marc Paolella, Wolfgang Wetzel and the participants of the Workshop on Applied Econometrics in Berlin and the Workshop on Applications of Differential Geometry to Econometrics in Florence. All remaining errors are my own.

184

Earnings differentials and Rao distances

185

apprenticeship with university studies. BuÈchel and Helberger (1995) call this combination an inef®cient `insurance strategy' pursued mainly by children from low-income and low-education families because of low ef®ciency or high risk-aversion. The authors observe that the strategy is not rewarded by an expected increase in income or a smaller probability of being unemployed, although this result is contentious. The concepts of Rao distance and earnings frontiers are then combined in section 4 and applied to the analysis of the pecuniary consequences of this strategy of education choice. 2

Rao distances

This section gives a short introduction to the Rao distance theory necessary for the economic application in section 4. See Jensen (1995) for more detail and references. 2.1

Derivation of the Rao distance

Let þ be a sample space, A a -algebra of subsets of þ and P : A ! ‰0; 1Š a probability measure. The random variable X has a density function p…x; † with an n-dimensional parameter  ˆ …1 ; 2 ; . . . ; n † and  2 . The parameter space  is an open subset of Rn . Then, S ˆ fp…x; † j  2 g

…1†

is a statistical model. Example 1

If X follows a normal distribution with

…1 ; 2 † ˆ …; †;

 ˆ f…; † j  2 R;  > 0g;

…2†

we identify the family of all normal distributions and the set of all points p on the upper real half-plane. But this identi®cation does not mean that the family of normal distributions is geometrically ¯at. A distance should quantify how dif®cult it is to discriminate between two distributions and, because, for instance, N…1 ; 0:1† and N…2 ; 0:1† can be discriminated more easily than N…1 ; 10† and N…2 ; 10†, this surface should be curved. It is assumed therefore that an n-parametric family of distributions behaves only locally like Rn , so that its potential curvature can be analysed. Such a generalisation of a real space is called an n-dimensional manifold. Then, local coordinates are transferred from   Rn to open neighbourhoods of the points on the n-manifold. In the example of the 2parameter normal distribution, we could take …; † for this purpose.

186

Uwe Jensen

These coordinates can be changed by admissible (smooth) transformations being three times continuously differentiable and having a non-singular functional determinant. In this way, S is equipped with a differentiable structure and S is called a differentiable n-manifold. With the de®nition @i l…x; † :ˆ

@ ln p…x; † @i

…3†

a statistical model is called regular if the following regularity conditions (known from the Cramer±Rao inequality) hold: 1. 8  2  and 8 x 2 þ, p…x; † > 0 2. 8  2 , the n components of the score function …@1 l…x; †; @2 l…x; †; . . . ; @n l…x; †† are linearly independent 3. For r ˆ 3, i ˆ 1; 2; . . . ; n and 8  2 , E‰@i l…x; †Šr < 1 4. For all functions f …x; † used in the following, … … @ @ f …x; † dP ˆ f …x; † dP: @i @i

…4†

…5†

In the sequel, we will inspect only regular statistical models that are differentiable n-dimensional manifolds, which will be called statistical models or n-manifolds for short. The examples used in this chapter are the normal statistical model (see Example 1) and the inverse Gaussian model: Example 2

X  IG…; † with ý ! r  ÿ…x ÿ †2 p…x; ; † ˆ exp I…0;1† …x† 2x3 22 x

…6†

and …1 ; 2 † ˆ …; †;

 ˆ f…; † j ;  > 0g:

…7†

The regularity conditions are partly responsible for the restricted applicability of Rao distances, because distributions outside the exponential family cannot be analysed with this concept. The n linearly independent vectors (see regularity condition 2) @i l…x; † measuring relative density changes are taken as a basis for the ndimensional vector space T…1† , the so-called 1-representation of the tangent space. Then, the inner product

Earnings differentials and Rao distances

187

gij …† ˆ h@i l…x; †; @j l…x; †i ˆ E‰@i l…x; † @j l…x; †Š

…8†

is formed from the basis of E‰@i l…x; †Š ˆ 0

…1† T

for

by averaging out x. Because of i ˆ 1; 2; . . . ; n;

…9†

we have gij …† ˆ Cov‰@i l…x; †; @j l…x; †Š:

…10†

Of course, gij …† is the Fisher information. Since this …n  n†-matrix is de®ned for every point p 2 S, it is a matrix ®eld …gij …††. This matrix ®eld is a Riemannian metric tensor, i.e. a tensor of order two with positive de®nite matrices and the property that the arclength de®ned below (the Rao distance) is an invariant. See, if necessary, an introduction on tensor calculus, e.g. Kay (1988). The main diagonal provides the lengths of the basis vectors: j@i l…x; †j2 ˆ h@i l…x; †; @i l…x; †i ˆ gii …† ˆ Var …@i l…x; ††:

…11†

Beside the main diagonal, we ®nd the necessary information on the angles ij of the basis vectors: cos ij ˆ

h@i l…x; †; @j l…x; †i gij …† ˆ p j@i l…x; †jj@j l…x; †j gii …† gjj …†

ˆ Corr …@i l…x; †; @j l…x; ††: to be

Example 3 1 …gij …†† ˆ 2 

…12†

For X  N…;  2 †, the Fisher information is found ý

1

0

0

2

! :

…13†

Since g12 …† ˆ 0, the coordinate vectors of the coordinate system  ˆ …; † are orthogonal for all p 2 S, but the lengths of the coordinate vectors depend on . Therefore, …; † is not a Cartesian coordinate system. For X  IG…; † 1 0  0 C B 3 C: …14† …gij …†† ˆ B @ 1 A 0 22 Once again, the coordinate vectors of the coordinate system  ˆ …; † are orthogonal for all p 2 S, but their lengths depend on ; with

188

Uwe Jensen

p  :ˆ …E‰XŠ; Std‰XŠ† ˆ …; …3 =††, even the angles of the coordinate vectors depend on . Now let C ˆ fc j  2 g be the set of all curves lying completely in S and connecting two distributions F1 and F2 represented by parameter values 1 and 2 . With t1 < t2 , c …t1 † ˆ 1 and c …t2 † ˆ 2 , the Rao distance (or geodesic distance or Riemannian distance) is the minimum arc-length of all these curves: v … t2 uX n u n X d …t† dj …t† t dt: …15† d…F1 ; F2 † ˆ min gij ……t†† i c 2C t dt dt 1 iˆ1 jˆ1 d…F1 ; F2 † is a mathematical distance and ± since the Fisher information is a Riemannian metric tensor ± it is invariant for all non-singular differentiable transformations of  and þ. 2.2

Calculation of Rao distances

In this chapter, Rao distances will be applied only to two-parameter statistical families. The calculation of Rao distances in the one-parameter case is presented in Jensen (1995). For n-parameter families with n  2, some results on the curvature of manifolds are necessary. Curvature measures are necessary for geometrical intuition and for the classi®cation of statistical distributions. In order to determine the curves with minimum length between two points on S, we ®rst have to `connect' the tangent spaces. An af®ne connection is a bilinear map on a Cartesian product of vector ®elds V…S†  V…S† ! V…S† providing the directional derivation of vector ®elds. There is an in®nite number of possible af®ne connections, but there is only one for which the movement of vectors from one tangent space to another is an isometry (preserving all distances and angles) and the Christoffel symbols (see below) are symmetric. This connection is called the Levi±Civita connection. By choosing a speci®c connection on a manifold, speci®c curves ± called geodesics ± are distinguished to be generalisations of the straight lines in Rn . If the Levi±Civita connection has been chosen, these geodesics locally show the shortest way (in S) between any two points on S. In Rn , the geodesics are the straight lines. If a different connection is selected (e.g. the exponential connection distinguishing one-parametric exponential families as geodesics; see Efron (1975) and Amari (1985)), this leads in general to nonRiemannian geometry and the geodesics are no longer distance minimising.

Earnings differentials and Rao distances

189

An af®ne connection can be de®ned with the help of the n3 Christoffel symbols of the ®rst kind   1 @ @ @ ÿijk ˆ g …† ‡ g …† ÿ g …† : …16† 2 @i jk @j ki @k ij The Christoffel symbol of the second kind ÿkij ˆ

n X mˆ1

ÿijm gmk

for

i; j; k ˆ 1; 2; . . . ; n;

…17†

is the ®rst auxiliary quantity, where gmk …† is the inverse matrix of the metric tensor gmk …†. The use of lower and upper indices has tensorial reasons (see Kay (1988)). The Riemannian tensor of the second kind is a tensor of order 4 with n4 components Rlijk ˆ

n n X X @ l @ l l l ÿik ÿ ÿij ‡ ÿm ÿ ÿ ÿm ij ÿmk ; ik mj @j @k mˆ1 mˆ1

…18†

and the Riemannian tensor of the ®rst kind is also a tensor of order 4: Rijkl ˆ

n X mˆ1

Rm jkl gmi :

…19†

Both tensors consist of second derivatives of the metric tensor (measuring curvature). For n ˆ 2, only one component of the Riemannian tensor of the ®rst kind is independent from the rest and not identically zero: R1212 ˆ R2121 ˆ ÿR1221 ˆ ÿR2112 :

…20†

Finally, for a 2-manifold, the sectional curvature (or mean Gaussian curvature) is Kˆ

R1212 g11 g22 ÿ g212

…21†

and is identical with the Gaussian curvature of surfaces in R3 . K is an invariant. An n-manifold S is said to be of constant curvature if K is constant for all p 2 S. S is said to be ¯at if K vanishes for all p. Two manifolds with constant sectional curvature are locally isometric iff their sectional curvatures are identical. The curvature is called parabolic if K ˆ 0, it is called elliptic if K > 0, and it is called hyperbolic if K < 0. In R3 , the plane is a standard example for a surface with constant K ˆ 0. The sphere is the typical example for a surface with constant elliptic curvature (K ˆ 1=r2 > 0 with radius r). Hyperbolic curvature can be seen on parts of the torus (the tyre of a bicycle). The inner side (facing

190

Uwe Jensen

the spokes) shows hyperbolic curvature, the outer side (facing the road) is elliptically curved. In the neighbourhood of hyperbolic points, tangent planes always cut the surface. It may be appropriate to remark that the curvature measures discussed here are not identical with the `statistical curvature' introduced by Efron (1975), which is the curvature of embedding of a submanifold (known as Euler±Schouten curvature) in non-Riemannian geometry (see Amari (1985) and Lauritzen (1987)). Example 4 The sectional curvature of the 2-manifolds of the normal distributions and the inverse Gaussian distributions is K ˆ ÿ0:5. This means that both manifolds are of the same constant hyperbolic curvature ± independent of the parameterisation (see Amari (1985, pp. 7 and 30) and Lauritzen (1987)). Many standard statistical distributions lead to manifolds with constant negative sectional curvature (see Jensen (1995)). This calls for deeper knowledge of hyperbolic geometry, which is rather unfamiliar for most econometricians. Any Riemannian 2-manifold with positive (not necessarily constant) sectional curvature can be embedded isometrically in R3 . Comparable results are not available for 2-manifolds with hyperbolic curvature, because no embedded complete surface in R3 can have constant negative curvature (a metric space is called complete if any Cauchy sequence converges). Any surface in R3 with constant negative curvature must have singularities. That is why the Poincare model is used in most cases for the distorted representation of 2-manifolds with constant hyperbolic curvature in R2 . For K ˆ ÿ1 (the hyperbolic analogue of the unit sphere, statistically represented by the t…3†-distribution), the set of points is the upper halfplane without the abscissa and the (distorted) geodesics are all semicircles with centre on the abscissa and all straight lines orthogonal to the abscissa (see Millman and Parker (1991)). The derivation of the Rao distance for a 2-manifold with constant hyperbolic curvature now consists of a transformation of its metric to the Poincare metric ds2 ˆ

…d †2 ‡ …d  †2 ; …  †2

…22†

which is the so-called ®rst fundamental form of the 2-manifold with constant K ˆ ÿ1. This manifold is known to have the MoÈbius distance

Earnings differentials and Rao distances

ýs! …1 ÿ 2 †2 ‡ …1 ÿ 2 †2 d…F1 ; F2 † ˆ 2 tanh …1 ÿ 2 †2 ‡ …1 ‡ 2 †2 ÿ1

191

…23†

as Rao distance (see Burbea and Rao (1982)). Example 5 The metric of the 2-manifold of normal distributions with  ˆ …; † is ds2NV ˆ

…d†2 ‡ 2…d†2 : 2

The Rao distance between N…1 ; 1 † and N…2 ; 2 † follows as 0s1 p …1 ÿ 2 †2 ‡ 2…1 ÿ 2 †2 A d…F1 ; F2 † ˆ 2 2 tanhÿ1 @ …1 ÿ 2 †2 ‡ 2…1 ‡ 2 †2

…24†

…25†

(Atkinson and Mitchell (1981, pp. 352ff)), whereas the Rao distance between two inverse Gaussian distributions IG…1 ; 1 † and IG…2 ; 2 † is d…F1 ; F2 †

 0v u ÿp p
2 ÿp p
2 1 u2 1=2 ÿ 1=1 ‡ 1=2 ÿ 1=1 p ˆ 2 2 tanhÿ1 @t ÿp p
2 ÿp p
2 A 2 1=2 ÿ 1=1 ‡ 1=2 ‡ 1=1 …26†

(Villarroya and Oller (1991)). See Jensen (1995) for more results on Rao distances in 2-manifolds and for some results and many problems in multivariate distributions. 2.3

Isocircles

An isocircle on a 2-manifold S is the set of all points p 2 S having constant distance d > 0 from a `centre' p0 2 S. This generalisation of the circle in Euclidean space can be very useful in two-parameter applications of Rao distances (see section 4). Example 6 For the calculation of the isocircle with ®xed `radius' d > 0 and `centre' …1 ; 1 † for the two-parameter normal distribution, (25) has to be solved for 2 and 2 . For improved readability, we write

192

Uwe Jensen

  2 d A ˆ tanh p ; 2 2

m ˆ 1 ;

 ˆ 2 ;

s ˆ 1 ;

 ˆ 2 …27†

with 0  A < 1 and obtain s 1‡A 4As2 …m ÿ †2 1;2 ˆ ÿ : s 2 1ÿA …1 ÿ A†2

…28†

 is maximal (minimal) for m ˆ  and the extreme values of  are p 8A s: …29† max;min ˆ m  1ÿA Figures 6.1 and 6.2 show various isocircles. 1. A shift in -direction does not change the isocircle, because the metric tensor depends only on . 2. A shift in -direction does not change the form (which is always the same within the distribution family) but does change either the Euclidean volume of the isocircle or the relative position of the `centre'. (a) The isocircle for d ˆ 1 and …m; s† ˆ …0; 0:1† has the same form as the isocircle for d ˆ 1 and …m; s† ˆ …0; 1† in ®gure 6.1, but is much smaller because the distances increase with decreasing . Dividing all the coordinates by 10 would restore the original Euclidean volume because the decreasing volume was created by the distorted representation of the hyperbolic surface in R2 . (b) Calculating the isocircle for …m; s† ˆ …0; 0:1† with constant Euclidean volume provides an isocircle of the same form with d ˆ 3:862, where the `centre' has been shifted away from the Euclidean centre (see ®gure 6.2). Note that the distance between any distribution and the -axis is in®nity. 3. Variation of the `radius' d leads to the isocircles in ®gure 6.1: the form is identical but the size varies. Example 7 For the inverse Gaussian distribution with `centre' …1 ; 1 † ˆ …m; s†, we choose the abbreviations   2 d ; A :ˆ tanh p 2 2 and the isocircle follows to be

 :ˆ 2 ;

 :ˆ 2

…30†

Earnings differentials and Rao distances

193

2.4

2.2

2

1.8

sigma

1.6

1.4

1.2

1

0.8

0.6

0.4 -1.5

-1

- 0.5

0 mu

0.5

1

1.5

Figure 6.1 Isocircles, normal distribution, d ˆ 0:25, 0.52, 1 1.6

1.4

1.2

sigma

1

0.8

0.6

0.4

0.2

0

-1.5

-1

- 0.5

0 mu

0.5

Figure 6.2 Isocircle, normal distribution, d ˆ 3:862

1

1.5

194

Uwe Jensen

v1ÿ2 u ýr s!2 u pu 2A B 1‡A 1 1 C t C ; ˆB ÿ @…1 ÿ A†ps  2 2 …1 ÿ A†2 s ÿ m  A 0

1;2

…31† where max;min

for

mˆ

max;min

ýr p !ÿ2 2A 1 p  ˆ : m …1 ÿ A† s …32†

2.4

Rao distance tests

The parameter values de®ning the distributions for which the Rao distances are to be calculated are generally unknown. Hence, the parameter values and the Rao distances must be estimated. If the ndimensional parameters 1 and 2 of an n-parameter family have been estimated by maximum likelihood (ML) from independent random samples, the invariance property of ML estimation provides an estimate of the required Rao distance: d^ML …1 ; 2 † ˆ d…^1;ML ; ^2;ML †:

…33†

Distance measures and tests are closely related. Therefore, it is not surprising to see that it is possible to construct tests from (estimated) distances. For a density function p…x; † with parameter vector , the general test problem H0 : g…† ˆ 0

versus

H1 : g…† 6ˆ 0

…34†

is considered where g is a smooth function. The null hypothesis selects a subset of the parameter space  (a submanifold of the manifold S). H0 ˆ f 2  j g…† ˆ 0g:

…35†

Given a sample X1 ; . . . ; Xm from p…x; †, the Rao distance between ^ and H0 is de®ned as: ^ † j  2 H g: ^ H † ˆ inffd…; d…; 0 0

…36†

If H0 6ˆ ;, the in®mum exists because d…
;
†  0. Then, a critical region ^ H0 † > u g C ˆ f…X1 ; . . . ; Xm † j d……X†;

…37†

Earnings differentials and Rao distances

195

is de®ned with a constant u depending on the signi®cance level of the test. The minimisation in (36) and the derivation of the test statistic may be dif®cult, but the idea behind a (Rao) distance test is extremely simple. The null hypothesis is rejected if the distance (weighted with the sample size) between the estimated distribution and the distribution under H0 is too big. In some standard test problems, Rao distance tests are equivalent to classical t, 2 or F tests (see the survey in Jensen (1995)). In addition, when testing for two or more parameters, the following theorems can provide a useful and easy alternative to the explicit derivation of the individual test statistic. Theorem 1 (Burbea and Oller (1989), pp. 13f) Let S be a regular n-parametric family of density functions. Let X1 ; . . . ; Xm be a random sample of size m from p0 2 S. Let p~ m be a consistent sequence of critical points of the log-likelihood function. Then L

md 2 …p0 ; p~ m † ÿ! Z  2 …n†:

…38†

Theorem 2 (Burbea and Oller (1989), p. 14) Let S be a regular n-parametric family of density functions. Let X1 ; . . . ; Xm and Y1 ; . . . ; Yl be two independent random samples of size m and l, respectively, obtained from p0 2 S. Let p~m and p~ l be two consistent sequences of critical points of the log-likelihood functions. Then ml L d 2 …p~m ; p~ l † ÿ! Z  2 …n†: m‡l

…39†

Example 8 Consider two independent samples of size m1 and m2 from N…1 ; 12 † and N…2 ; 22 † and the test problem H0 : …1 ; 1 † ˆ …2 ; 2 † versus

H1 : …1 ; 1 † 6ˆ …2 ; 2 †:

…40†

A Rao distance test is easily derived with the help of Theorem 2. The critical region is þ   þ m1 m2 2  þ …41† d ……^ 1 ; ^ 1 †; …^ 2 ; ^ 2 †† þ U > u ; Cˆ Uˆ m 1 ‡ m2 where d…
;
† is taken from (25), P…U > u j H0 † ˆ  and U  2 …2† asymptotically under H0 . Testing H0 : …; † ˆ …0 ; 0 †

versus

H1 : …; † 6ˆ …0 ; 0 †

Theorem 1 can be applied to derive the critical region

…42†

196

Uwe Jensen

^ …0 ; 0 †† j U > u g; ^ †; C ˆ fU ˆ md 2 ……; 

…43†

2

where P…U > u j H0 † ˆ  and U   …2† asymptotically under H0 . Villarroya and Oller (1991) extensively analyse Rao distance tests for the inverse Gaussian distribution ± with very satisfactory results. Twoparameter test problems can be treated with Theorems 1 and 2 and the distance (26) as in the previous example. 3

Earnings frontiers

We now turn to consider the estimation of a human capital model as a stochastic earnings frontier. In the ®rst subsection, we look at the basic human capital model for explaining individual income. In subsection 3.2, the data taken from the German socio-economic panel are described. The subsequent subsection provides essential information on stochastic frontier models in general. In subsection 3.4, the earnings frontier function is derived as an extended human capital model with imperfect information on the employees' side. The costs for this imperfection are measured by the inef®ciency terms of the frontier model. Subsection 3.5 presents the estimation results and the economic implications in detail. The ®nal subsection discusses the relative advantages and limitations of the approach. 3.1

Human capital theory

Since the pathbreaking work of Mincer (1958) and Becker (1964), human capital theory has been the most popular approach for explaining individual income. Schooling is seen as an investment in human capital because it means renouncing present consumption for higher future income by increasing the individual's resources. The basic model derived by Mincer (1974) is LI ˆ ÿ0 ‡ ÿ1 S ‡ ÿ2 E ‡ ÿ3 E 2 ‡ v;

ÿ0 ; ÿ1 ; ÿ2 > 0;

ÿ3 < 0; …44†

where LI is the natural logarithm of individual wage income, S is schooling (years of education) and E is experience (age-education-6). For all the success and broad applicability of this approach, it has limitations. Even in extended human capital models the portion of unexplained variance remains persistently near 50 per cent of the total variance and this has led to a debate about the reasons for this partial failure. This discussion has been very fruitful for econometrics but has

Earnings differentials and Rao distances

197

not yet resolved the empirical problems of human capital models (see Wagner (1981)). The number of exogenous variables will be heavily augmented in this chapter, as in many other papers. However, in addition, we apply a frontier function model to explain the deviations between estimated and empirical earnings. The idea behind adopting this approach is simply that human capital models can explain only potential income, whereas actual income achieved depends on the individual's search for an appropriate job. 3.2

The data from the socio-economic panel

The data are taken from the tenth wave of the German socio-economic panel (SOEP). After a detailed data analysis, the twenty-four variables presented below have been selected following the recommendations for the extension of the basic human capital model given by various authors (see e.g. Becker (1964), Brinkmann (1981), Griliches (1977) or HuÈbler (1984)). The 13,179 individuals were reduced to n ˆ 1;334 full-time employees, workers or public servants throughout the year 1992 who provided information on all the required variables. The endogenous variable LI is the natural Logarithm of gross wage Income in 1992 including all extra payments. The exogenous variables are: 1. Schooling: (a) QUUN: Dummy: 1 for QUali®cation for UNiversity entrance ± 0 for lower education 2. Professional TRaining: (a) TRAP: Dummy: 1 for APprenticeship, technical schools, health schools, etc. (b) TRPS: Dummy: 1 for Public Service education Both 0 for no professional training 3. STudies: (a) STTC: Dummy: 1 for Technical College (b) STUN: Dummy: 1 for UNiversity Both 0 for no studies 4. Experience: (a) AGE1: AGE in years divided by 10 (b) AGE2: AGE1 squared 5. On-the-Job-Training: (a) JTNO: Dummy: 1 for `NO interest in on-the-job-training' (b) JTNC: Dummy: 1 for `No interest in on-the-job-training at one's own Cost'

198

Uwe Jensen

6. Health: (a) SAHL: SAtisfaction with one's own Health and Life ± values on an integer scale ranging from 0 (no) to 20 (yes) ± an indicator for physical and emotional health 7. Gifts, predisposition: (a) FATH: Status of the job of the FATHer ± values on an integer scale ranging from 0 to 1.868 ± the Wegener scale provided by the SOEP divided by 1000 8. Ability: (a) PCHA: Probability of CHAnging the occupation in the near future ± values on an integer scale ranging from 1 (yes) to 4 (no) ± an indicator for `being in the right place' (b) PDET: Probability of DETerioration in the same ®rm in the near future ± values on an integer scale ranging from 1 (yes) to 4 (no) ± an indicator for `being on the right level' (c) SATW: SATisfaction with Work ± values on an integer scale ranging from 0 (no) to 10 (yes) 9. Demographic variables: (a) MALE: Dummy: 1 for MALE ± 0 for female (b) MARR: Dummy: Marital status: 1 for MARRied (including those living separated), divorced and widowed ± 0 for unmarried (c) RESI: Size of the RESIdence constructed from the Boustedt type of residence provided by the SOEP ± 0: centre or outskirts of cities with more than 500,000 inhabitants . . . ± 6: towns with fewer than 2,000 inhabitants 10. Job characteristics: (a) SIZE: SIZE of the ®rm: Number of employees divided by 1000 (b) ii. EMPL: Dummy: 1 for salaried EMPLoyees ii. SERV: Dummy: 1 for public SERVants Both 0 for workers (c) STAT: STATus of the job ± values on an integer scale ranging from 0.2 to 1.731 ± the Wegener scale provided by the SOEP divided by 1000 11. Working time: (a) HOUR: Weekly working HOURs including overtime 12. SENIority: (a) SENI: Years of af®liation to the ®rm 13. PROPerty: (a) PROP: Interest and dividend income in 1992 divided by 1000

Earnings differentials and Rao distances

199

Unfortunately, some interesting information is not provided by the SOEP. One has to be satis®ed with proxy variables for health status, predisposition, ability and the region of residence. Furthermore, the variables for health status, ability and the region of residence have been measured on ordinal scales but are treated as quantitative variables. It is to be hoped that the estimation results are not overly biased by these inaccuracies. On the other hand, the choice of the dummy variables for schooling (instead of years of education) and of age (instead of age-education-6) for experience is deliberate and deserves some explanation. I agree with researchers like Helberger (1988) who ®nd it problematic to take the years of education as proxy for education. First, pupils repeating a school year or students changing subject, studying for an excessively long time or breaking off their studies certainly do not augment their human capital according to the theory. Secondly, information on the actual years of education is not available and one has to ®x theoretical values for different types of schooling. Dummies and theoretically ®xed years are not affected by the problems mentioned above, but both are affected in different ways by the `insurance strategy' mentioned in section 1. The decision to use dummy variables is motivated by the fact that years of education cannot discriminate between the yield of one year of apprenticeship and that of one year at university, whereas dummy variable parameters provide interesting information on the yield of different types of education (see subsection 3.5 and section 4). The decision to take age as a proxy for experience (instead of age-education-6) is based on the similarity of the latter with the seniority variable. Becker (1964) mentioned that the search for information is also a very important human capital investment. This suggestion has not been employed here because it is dif®cult to quantify; nevertheless, in section 3.4 we will see a fruitful way to capture it indirectly. 3.3

The stochastic frontier model

A production function gives the maximum possible output that can be produced from given input quantities. Ever since the seminal paper by Aigner and Chu (1968), a range of production functions have been estimated as frontier functions consistent with the economic theory of optimising behaviour. In the following, we will concentrate on the stochastic frontier model of Aigner et al. (1977) and Meeusen and van den Broeck (1977). See Greene (1993) for a discussion of the alternatives. A stochastic Cobb±Douglas production frontier is

200

Uwe Jensen

yi ˆ  ‡

k X jˆ1

ÿj xij ‡ ei ;

ei ˆ vi ÿ ui ;

ui  0;

i ˆ 1; . . . ; n; …45†

where yi is output in logs and xij are inputs in logs. Then, y^i is estimated maximum possible output. The composite error term ei consists of the technical inef®ciency ui of individual number i (in logs), which follows a one-sided distribution, and a symmetric part vi representing statistical noise. vi and ui are assumed to be independent. The vi are assumed to follow a normal distribution. The authors employ the half-normal and exponential distributions for the ui , whereas Stevenson (1980) proposes the truncated normal distribution. Greene (1989) has developed the econometric computer package LIMDEP, which estimates all versions of this model by direct maximisation of the likelihood function. 3.4

The earnings frontier

What do we gain by estimating the earnings function not as an average function but as a frontier function? Why is it sensible? Following Daneshvary et al. (1992), it is assumed that wages LI depend upon personal characteristics H augmenting human capital stock, job characteristics C and information I on labour market conditions, the wage distribution and job search methods. Individuals stop their search when a wage offer exceeds the reservation wage LIr . For any set of H and C and perfect information I  , a potential maximum attainable wage LI  exists. We estimate LI ˆ LI…H; C; I† as a stochastic earnings frontier (45) where y ˆ LI is empirical gross wage income in logs and x is the vector of twenty-four variables presented in subsection 3.2. Then, bi ˆ LIi LI

…46†

is the estimated maximum possible income and the inef®ciency term bi ÿ LIi u^ i ˆ LI

…47†

is interpreted as the cost of imperfect information becoming apparent in the underemployment or overeducation of individual number i. We distinguish between the underemployed `potential human capital' produced through the education process and the `active human capital' used as a factor of production (see MaÈding (1981)).

Earnings differentials and Rao distances

201

The return from estimating an earnings frontier instead of the average function yi ˆ  ‡

k X jˆ1

ÿj xij ‡ vi ;

i ˆ 1; . . . ; n

…48†

with standard assumptions arises from the interpretation of a signi®cant part of the deviations of empirical income from estimated income. Furthermore, the earnings frontier takes account of the important but dif®cult to measure component of human capital consisting of `search for information'. 3.5

Results

Estimation has been carried out using LIMDEP, with all three inef®ciency distributions mentioned in subsection 3.3. In the exponential case, the estimation procedure did not converge. The truncated normal distribution ui  truncN…u ; u2 †

…49†

and its special case for u ˆ 0, the half-normal distribution ui  jN…0; u2 †j;

…50†

provided very similar results and an insigni®cant u in (50). So, the results with speci®cation (51) are presented in the following. LIMDEP takes the OLS regression coef®cients as starting values for an iterative procedure combining the DFP algorithm and the method of steepest descent. The estimation results are given in table 6.1. The OLS regression has an adjusted R2 of 0.6016. One could argue that this is suf®cient for a cross-section data set, especially if compared with the empirical success of extended human capital models in general (see section 3.1). However, 40 per cent unexplained variance leaves some scope for improvement. It can be seen that the step from OLS to the frontier does not turn the results upside down. The coef®cients and t-ratios change marginally. The bene®t of the frontier model lies in the interpretation of the deviations of the observations from the estimated function (see below). To begin with, the frontier coef®cients and t-ratios will be examined carefully. The OLS results can be analysed in a similar way if `maximum possible income' is replaced by `average income'.

202

Uwe Jensen

Table 6.1. Estimation results Estimation method 2

Adjusted R

Variable 1 2 3 4 5 6 7 8 9 10

11 12 13

a a b a b a b a b a a a b c a b c a bi b ii c a a a

OLS

Frontier

0.6016 Coecient

Constant 8.56200 QUUN 0.07614 TRAP 0.05908 TRPS 0.07315 STTC 0.10015 STUN 0.23441 AGE1 0.54392 AGE2 70.05877 JTNO 70.05795 JTNC 70.06603 SAHL 70.00221 FATH 0.01115 PCHA 0.04106 PDET 70.04315 SATW 0.01688 MALE 0.20714 MARR 0.08087 RESI 70.00783 SIZE 0.00989 EMPL 0.09566 SERV 70.12591 STAT 0.35893 HOUR 0.01084 SENI 0.00540 PROP 0.00655 =v  pu   2u ‡ 2v

t-ratio 56.498 2.665 2.709 1.876 2.347 5.978 8.638 78.061 72.979 74.120 70.716 0.432 2.792 72.938 3.337 10.898 3.433 72.058 6.298 4.533 73.557 9.904 9.213 5.363 4.041

Coecient 8.85160 0.07559 0.05502 0.08549 0.09697 0.24188 0.50281 70.05441 70.04884 70.06969 70.00199 0.00310 0.03921 70.04041 0.01621 0.20679 0.07253 70.00714 0.00931 0.09640 70.13828 0.35701 0.01212 0.00514 0.00699 1.52190 0.35832

t-ratio 59.988 2.507 2.594 2.251 2.043 6.213 7.941 77.319 72.580 74.345 70.614 0.119 2.913 72.752 3.375 10.862 2.983 71.888 5.605 4.741 73.825 11.180 13.381 5.063 4.138 12.571 37.601

The step from the average function to the frontier increases the constant, which is (and should be) highly signi®cant. The interpretation of the dummy variable coef®cients is straightforward; if we rewrite equation (45) as INCi ˆ exp…†
ui  0;

k Y jˆ1

exp…ÿj xij †
exp…vi †
exp…ÿui †;

i ˆ 1; . . . ; n

…51†

Earnings differentials and Rao distances

203

with the untransformed gross wage income INC, this implies that studying at the university means, for instance, multiplying maximum possible gross wage income without any study by the factor exp…0:24188† ˆ 1:2736. Being male implies multiplying female maximum possible gross wage income by the factor exp…0:20679† ˆ 1:2297. The health proxy SAHL in 6 and the predisposition proxy FATH in 7 show very insigni®cant coef®cients. Naturally, the insigni®cance could originate in poor suitability of the proxy variables, but employees are perhaps neither healthy nor satis®ed with their health and life when they are working `on the frontier'. The irrelevance of predisposition for explaining income is in agreement with earlier results on this question. The rest of the variables are weakly to highly signi®cant and show expected signs. The maximum possible income is greater * with higher education (1±3b). * with higher experience (ˆ age) (4) with the well-known concavity of the in¯uence of experience on income. * with interest in on-the-job-training (5a,b). * for individuals who will not change their occupation in the near future and who are satis®ed with their work (8a,c). These people seem to be able to accomplish the tasks their job demands. * for individuals who fear ± at the time of estimation ± some deterioration in their prospects in the near future (8b). In the present position, their actual human capital stock may be overcharged. * for men (9a). * for individuals who are or have been married (9b). Part of this effect may be due to the payment system in the public service, but it could also be that being married is highly correlated with hardly measurable parts of human capital stock such as trustworthiness or a sense of responsibility that are appreciated by employers. Many employers hire married persons by preference because of this supposed or actual correlation. * in places with more inhabitants (9c). * in ®rms with more employees (10a). This could be interpreted from an ef®ciency wage viewpoint. Because these ®rms have higher costs for the motivation and monitoring of their employees they give income incentives to increase the ef®ciency of their employees and to prevent shirking because idlers must fear being dismissed even more from this high-income ®rm (see e.g. Franz (1994)). * for salaried employees than for workers, whereas public servants have the lowest maximum possible income (10b). Public servants have to pay the price for the various non-income contributions such as, for example, the greater security of their jobs.

204

Uwe Jensen

for jobs with higher status (10c). with higher working time (11). * with increasing years of af®liation to the ®rm (12). It is possible that some ®rms try to eliminate shirking by rewarding ef®cient workers with delay so that idlers again have to fear being dismissed even more because of the loss of the subsequent payments (seniority hypothesis). * with higher property (13). Individuals with higher income today are likely to have had a higher income in the past, which has led to higher property wealth. Now the frontier speci®c results will be examined. The high signi®cance of the ratio  ˆ u …52† v * *

means that the variation of the inef®ciency terms ui in relation to the variation of the noise vi is reasonably large.  ˆ 0 would lead to the simple OLS model because there are no inef®cient individuals in this case. Note that u is not the standard deviation of u because of de®nition (50). From the estimated coef®cients in the last two lines of table 6.1 we can calculate u ˆ 0:2995 and v ˆ 0:1968. This leads to the estimates for the mean and standard deviation of u r r 2 ÿ2  ˆ 0:2389 and Std…u† ˆ u ˆ 0:1805 E…u† ˆ  u  …53† (see e.g. Aigner et al. (1977)). Combining E…u† with equation (51), we see for example that on average individuals receive only exp…ÿ0:2389†
100 ˆ 78:75% of the maximum possible income. 3.6

Assessment of the approach

Now the bene®t of estimating an earnings frontier instead of an average function mentioned in subsection 3.4 can be seen. The frontier model includes the important but hardly measurable human capital investment `search for information' and it explains an essential part of the deviations of empirical income from estimated income. The extended human capital model explains potential income suf®ciently, but individuals do not obtain their maximum possible income because of imperfect information on labour market conditions, the wage distribution, and job search methods. The more or less underemployed `potential human capital' produced through the education process is distinguished critically from the `active

Earnings differentials and Rao distances

205

human capital' used as a factor of production. In the OLS approach, we simply interpret inef®ciency as misspeci®cation. What are the limits of the extended human capital model underlying both the OLS and the frontier approach? First of all, there are the data problems mentioned in subsection 3.2, including a possible selection bias because of the systematic elimination of many individuals from the data set and a possible bias because of the inclusion of many exogenous variables that could be highly correlated (see Griliches (1977)). The standard assumptions of human capital models, such as the complete entrance of schooling efforts into working productivity, will certainly be violated. Unemployment and other limitations on the labour demand side were excluded. The monetary and social costs of migration and search for information were left out of consideration. Because of these omissions, all conclusions have to be viewed with caution. As for the reasons for income inef®ciency, the frontier analysis concentrated on imperfect information on the employees' side. There are certainly further causes for this sort of inef®ciency. Employers often are not suf®ciently informed about the potential human capital of their employees or applicants, and, if they do know, they are not able to relate individual skills to the tasks available. This is another classic criticism of human capital models (see e.g. Klein (1994)), but this information problem could certainly be included in the frontier model presented above. What arguments could be put forward for applying the standard OLS approach instead of the frontier approach? Many researchers are perhaps interested only in estimating average income, not in estimating maximum possible income. This reservation is meaningless as long as the model lacks a sensible job search variable, which is not available. By simply mixing up inef®ciency and white noise, OLS does not solve the problem.

4

Differentials in earnings and inef®ciency

In this section, Rao distances are applied to the earnings frontier problem presented above. 4.1

Earnings differentials

We now want to consider the issue raised by BuÈchel and Helberger (1995) when criticising the `insurance strategy' discussed in section 1. One ®rst idea would be to isolate this effect by constructing three new dummy variables:

206

Uwe Jensen

TRAPN: Dummy: 1 only for apprenticeship, etc. and No university study * STUNN: Dummy: 1 only for university study and No apprenticeship, etc. * INSU: Dummy: 1 only for the INSUrance strategy Substituting these variables for TRAP and STUN in the earnings frontier of section 3 and re-estimating the function yields minimal changes in the frontier results and an INSU parameter that is smaller than but not signi®cantly different from the STUNN (or STUN) parameter. Rather than analysing point estimates, which focus only on central tendencies, it will be more useful to consider instead the entire income distribution of some sample subgroups constructed from the dummy variables in section 3. The standard approach of applying inequality measures for the comparison of these income distributions will not be adopted in the following, however, since we are interested in earnings differentials rather than in the interpretation of, for instance, Gini coef®cients. In recent years, the kernel density estimation of income distributions has become more and more popular (see e.g. DiNardo et al. (1996)) because it is still an open question which theoretical distribution is best. Unfortunately, this nonparametric approach cannot be applied here because of limited sample sizes in some subgroups. So, some choice has to be made as to a suitable income distribution. The Pearson family is often applicable but is intractable in many respects and this is why the inverse Gaussian distribution for income and the normal distribution for log income were chosen, given that they performed satisfactorily. In these cases, it is possible to examine the subgroup distributions in …; †-plots or …; 1=†-plots, which is much more informative than looking at the densities themselves. Furthermore, Rao distances and isocircles (see section 2) can be applied for an intuitive understanding and for tests in these plots. Finally, a Rao distance analysis with the log-normal distribution for income would not differ from the analysis with the normal distribution for log income because of the invariance property of the distance. Figure 6.3 is the …; †-plot of estimated normal distributions of log income for some subgroups of individuals having passed through certain courses of education. We can see how mean and variance of income increase with increasing the additional quali®cation of apprentices. It can also be seen that university study and the insurance strategy differ in distribution although not so much in mean. The mean difference is in fact insigni®cant, but the Rao distance test (40) shows that these *

Earnings differentials and Rao distances

207

0.55

TRAP+STUN

0.5

0.45 sigma

TRAP+QUUN

0.4

TRAP TRAP+0 STUN STTC

0.35 TRPS

0.3 10.8

10.9

11

11.1

11.2

11.3

11.4

11.5

mu

Figure 6.3 Estimated normal distributions of log income

distributions are signi®cantly different with  ˆ 0:06, mainly due to the difference in variances. Dealing with the mean and variance of (log) income in …; †-plots, an obvious idea is to exploit the analogy with a simple yield±risk analysis in asset theory. We assume * that individuals judge different types of schooling A by their income i distributions represented by expected incomes i and `risk' i , * that the rest of the economic environment is irrelevant for the choice of the optimal A , and * that a preference function þ…; † exists for the selection of A . p Taking þ…; † ˆ  ÿ … 2 =2† with indifference curves  ˆ 2 ÿ c, the decision according to the …; †-rule is consistent with the Bernoulli principle. Of course, there are limitations to the analogy between asset allocation decisions and decisions on courses of education, for instance because ± in the latter problem ± it is dif®cult to include costs and because a `true' portfolio analysis is not applicable given that it is virtually impossible to diversify education risks (Helberger (1997)). Interpreting income variance as risk, we see that the `insured' have the highest risk! Figure 6.3 shows the indifference curve I going through

208

Uwe Jensen

INSU. It is interesting to know if STUN could also lie on I, in other words whether or not it might nevertheless be rational to choose the insurance strategy (if these two courses of education are `true' alternatives for the individual). This can be tested by the Rao distance test (42), where …0 ; 0 † is an unknown point on I. For known sample size and signi®cance level  ˆ 0:05, the critical Rao distance can be determined and the corresponding isocircle can be plotted as in ®gure 6.3. Because this isocircle does not intersect I, the null hypothesis that STUN lies on I can be rejected. This result could also be achieved by drawing a Rao distance con®dence band around I, which can be constructed by moving an appropriate isocircle along I. This con®dence band deviates from I for growing  because of the distorted representation of the hyperbolic surface in R2 . See subsection 2.3 and Jensen (1995) for more details. Finally, ®gure 6.4 shows the …; 1=†-plot of estimated inverse Gaussian distributions of income for the same subgroups. The interpretation is slightly more complicated because the vertical axis no longer represents the standard deviation of the distributions (see Example 3 for the reason). That is why some lines of constant standard deviation are

0.18 std=0.3

std=0.5

std=0.7

std=0.9 TRAP+STUN

0.16 TRAP TRAP+QUUN

TRAP+0

0.14

1/lambda

0.12

0.1 TRPS STTC

0.08

STUN

0.06

0.04

0.02

0 0.8

1

1.2

1.4

1.6

1.8

mu

Figure 6.4 Estimated inverse Gaussian distributions of income

2

2.2

Earnings differentials and Rao distances

209

added to this plot to compensate for this drawback. Apart from that, the results in the inverse Gaussian case do not differ substantially. 4.2

Inef®ciency differentials

The results of subsection 4.1 have made clear ± with the help of …; †plots and Rao distances ± that there are signi®cant differences between the income distribution of university students and that of individuals having followed the insurance strategy. Finally in this subsection, Rao distances and the frontier results from subsection 3.5 on inef®ciency will be combined to get even more insight into the ®nancial implications of these courses of education and of the relationship with other sample subgroups. The inef®ciency distribution of various subgroups will be analysed to answer questions such as: * Are low-educated persons very inef®cient in ®nding suitable jobs? * Are older people more inef®cient than younger people? * Are the `insured' particularly inef®cient? Unfortunately, the individual inef®ciency terms ui are unobservable (see subsection 3.3), but we have ui jei  trunc0 N…i ; 2 †;

i ˆ

ÿu2 ei 2

 ˆ

u v 

…54†

(Jondrow et al. (1982)) and, because the Rao distances for N…i ; 2 † are also Rao distances for trunc0 N…i ; 2 †, the estimated normal distributions of conditional inef®ciency in various sample subgroups can be plotted, as in ®gure 6.5, with the following additional abbreviations (see subsection 3.2): * QUU0: lower education than `quali®cation for university entrance' * 1830, 3140, 4150, 5165: age between 18 and 30; . . . ; 51 and 65 * FEMA: female * UNMA: unmarried * WORK: workers The inef®ciency distributions of the age classes make clear that increasing age leads to higher average inef®ciency and lower standard deviation of inef®ciency. This could be a hint that human capital models need the inclusion of retirement effects. Public servants and those trained for public service show relatively high average inef®ciency and low standard deviation. This had to be expected because of the payment system in the public service, and very plausibly the `insured' are close to these two groups. On the other hand, university students show relatively low average inef®ciency but very high variation. The latter could be due to the

210

Uwe Jensen

0.18

0.17

0.16

STUN sigma

0.15

3140 0.14

EMPL

FEMA

QUUN MARR

1830 UNMA

0.13

QUU0 TRAP

MALE WORK

SERV

4150

0.12 INSU

STTC 0.11 0.1

TRPS 5165

0.12

0.14

0.16

0.18 mu

0.2

0.22

0.24

0.26

Figure 6.5 Estimated truncated normal distributions of conditional inef®ciency

well-known fact that many university graduates do not ®nd suitable jobs. As in the previous subsection, the mean difference between INSU and STUN is clearly insigni®cant, but the Rao distance test (40) indicates that the distributions depending on  and  are signi®cantly different with  ˆ 0:1. Finally, the Rao distance test (42) can be applied to construct a con®dence region around, for instance, STUN. If …0 ; 0 † is an unknown point on the surface, the critical Rao distance can be determined for known sample size and signi®cance level  ˆ 0:05, and the corresponding isocircle can be plotted as in ®gure 6.5.

5

Conclusions

Estimating an extended human capital model as a stochastic earnings frontier has proved to be a fruitful way of explaining potential income and the deviations between empirical and potential income. The partial failure of human capital models when estimated as average functions can thus be explained. The inef®ciency terms of the frontier model capture the important but dif®cult to measure human capital investment in the

Earnings differentials and Rao distances

211

`search for information'. Inef®ciency turned out to be considerable: the individuals in the sample received only 79 per cent of their maximum possible income on average. Human capital theory is able to explain maximum possible income suf®ciently but individuals are inef®cient in their job search. The results of section 4 sustain the judgement of BuÈchel and Helberger (1995) that the combination of an apprenticeship with university study is an inef®cient `insurance strategy'. Comparing the pecuniary implications of university study STUN and the insurance strategy INSU, it turns out that: 1. the income distributions of STUN and INSU differ signi®cantly; 2. interpreting income variance as risk, the insured have the highest risk of all courses of education in the analysis; 3. the inef®ciency distributions of STUN and INSU differ signi®cantly; 4. the name `insurance strategy' is justi®ed in that the inef®ciency distribution of INSU is very similar to that in public service (high average inef®ciency, low variance). At least for the present data set, the second and the fourth items mean that the insurance strategy is poor and costly. It has been shown that the application of Rao distances is straightforward if one is willing to restrict the analysis to the narrow class of distributions allowing explicit analytic calculation of the necessary distance formula. Rao distances and isocircles are very helpful tools in an intuitive comparison of distributions, especially in two-parameter families. Rao distance tests are based on simple geometrical ideas and ± in combination with isocircles ± they make it possible graphically to solve test problems such as the test for a …; †-pair lying on an indifference curve.

References Aigner, D.J. and S.-F. Chu (1968), `On Estimating the Industry Production Function', American Economic Review, 58: 826±839. Aigner, D.J., C.A.K. Lovell and P. Schmidt (1977), `Formulation and Estimation of Stochastic Frontier Production Function Models', Journal of Econometrics, 6: 21±37. Amari, S.I. (1985), Differential-Geometrical Methods in Statistics, New York: Springer Verlag. Atkinson, C. and A.F.S. Mitchell (1981), `Rao's Distance Measure', Sankhya A, 43: 345±365. Becker, G.S. (1964), Human Capital, New York: NBER.

212

Uwe Jensen

Brinkmann, G. (1981), `Die Einkommensfunktion und ihre Testbarkeit', in W. Clement (ed.), Konzept und Kritik des Humankapitalansatzes, Berlin: Duncker & Humblot, pp. 87±116. BuÈchel, F. and C. Helberger (1995), `Bildungsnachfrage als Versicherungsstrategie', Mitteilungen aus der Arbeitsmarkt- und Berufsforschung, 1/95: 32±42. Burbea, J. and J.M. Oller (1989), `On Rao Distance Asymptotic Distribution', Universitat de Barcelona. Burbea, J. and C.R. Rao (1982), `Entropy Differential Metric, Distance and Divergence Measures in Probability Spaces: A Uni®ed Approach', Journal of Multivariate Analysis, 12: 575±596. Daneshvary, N., H.W. Herzog, R.A. Ho¯er and A.M. Schlottmann (1992), `Job Search and Immigrant Assimilation: An Earnings Frontier Approach', Review of Economics and Statistics, 74: 482±492. DiNardo, J., N.M. Fortin and T. Lemieux (1996), `Labour Market Institutions and the Distribution of Wages, 1973±1992: A Semiparametric Approach', Econometrica, 64: 1001±1044. Efron, B. (1975), `De®ning the Curvature of a Statistical Problem (with Applications to Second Order Ef®ciency)', Annals of Statistics, 3: 1189±1242. Franz, W. (1994), ArbeitsmarktoÈkonomik, 2nd edn, Berlin: Springer. Greene, W.H. (1989), LIMDEP, New York: Econometric Software. (1993), `The Econometric Approach to Ef®ciency Analysis', in H.O. Fried, C.A.K. Lovell and S.S. Schmidt (eds.), The Measurement of Productive Ef®ciency, New York: Oxford University Press, pp. 68±119. Griliches, Z. (1977), `Estimating the Returns to Schooling: Some Econometric Problems', Econometrica, 45: 1±22. Helberger, C. (1988), `Eine UÈberpruÈfung der LinearitaÈtsannahme der Humankapitaltheorie', in H.-J. BodenhoÈfer (ed.), Bildung, Beruf, Arbeitsmarkt, Berlin: Duncker & Humblot, pp. 151±170. (1997), personal communication. HuÈbler, O. (1984), `Zur empirischen UÈberpruÈfung alternativer Theorien der Verteilung von Arbeitseinkommen', in L. Bellmann, K. Gerlach and O. HuÈbler, (eds.), Lohnstruktur in der Bundesrepublik Deutschland, Frankfurt: Campus, pp. 17±189. Jensen, U. (1995), `A Review of the Derivation and Calculation of Rao Distances with an Application to Portfolio Theory', in G.S. Maddala, P.C.B. Phillips and T.N. Srinivasan (eds.), Advances in Econometrics and Quantitative Economics, Oxford: Blackwell, pp. 413±462. Jondrow, J., C.A.K. Lovell, I.S. Materov and P. Schmidt (1982), `On the Estimation of Technical Inef®ciency in the Stochastic Frontier Production Function Model', Journal of Econometrics, 19: 233±238. Kay, D.C. (1988), Tensor Calculus, Auckland: Schaum, McGraw-Hill. Klein, T. (1994), `Die Einkommenskarriere von Hochschulabsolventen', Mitteilungen aus der Arbeitsmarkt- und Berufsforschung, 3/94: 205±211.

Earnings differentials and Rao distances

213

Lauritzen, S.L. (1987), `Statistical Manifolds', in S.I. Amari, O.E. BarndorffNielsen, R.E. Kass, S.L. Lauritzen and C.R. Rao (eds.), Differential Geometry in Statistical Inference, Hayward, Calif.: Institute of Mathematical Statistics, pp. 163±216. MaÈding, H. (1981), `Humankapitalbildung zwischen Bildungs- und Arbeitsmarktpolitik', in W. Clement (ed.), Konzept und Kritik des Humankapitalansatzes, Berlin: Duncker & Humblot, pp. 117±140. Meeusen, W. and J. van den Broeck (1977), `Ef®ciency Estimation from Cobb± Douglas Production Functions with Composed Error', International Economic Review, 18: 435±444. Millman, R.S. and G.D. Parker (1991), Geometry ± A Metric Approach with Models, 2nd edn, New York: Springer Verlag. Mincer, J. (1958), `Investment in Human Capital and Personal Income Distribution', Journal of Political Economy, 66: 281±302. (1974), Schooling, Experience and Earnings, New York: NBER. Rao, C.R. (1945), `Information and the Accuracy Attainable in the Estimation of Statistical Parameters', Bulletin of the Calcutta Mathematical Society, 37: 81± 91. Also in S. Kotz and N.L. Johnson, Breakthroughs in Statistics, vol. 1, New York: Springer Verlag (1992), pp. 235±247. Stevenson, R.E. (1980), `Likelihood Functions for Generalized Stochastic Frontier Estimation', Journal of Econometrics, 13: 57±66. Villarroya, A. and J.M. Oller (1991), `Statistical Tests for the Inverse Gaussian Distribution Based on Rao Distance', Department of Statistics, University of Barcelona. Wagner, M. (1981) `UnerklaÈrte Varianz ± Zur Forschungsstrategie der mikrooÈkonomischen Humankapitaltheorie', in W. Clement (ed.), Konzept und Kritik des Humankapitalansatzes, Berlin: Duncker & Humblot, pp. 165±184.

7

First-order optimal predictive densities J.M. Corcuera and F. GiummoleÁ

1

Introduction

Consider a vector of observations x ˆ …x1 ; . . . ; xn †, not necessarily independent, that corresponds to a random variable X having a distribution that depends on an unknown parameter . We suppose that the sample size n is large enough to allow asymptotic approximations. The aim is to predict the value of an as yet unobserved random variable Y, with a distribution related to that of the data. In general, X and Y have different distributions and are dependent. The conditional density of Y given X is assumed to belong to a regular parametric model P x ˆ fp…y; jx†;  2 g; where p…y; jx† are densities with respect to some reference measure , ®nite in Rp , and  is an open set in Rk . If possible, we operate with a prior reduction of the problem. Let z be a minimal predictive suf®cient statistic of the data, that is, z is a minimal suf®cient statistic among those that satisfy p…y; jx† ˆ p…y; jz†: Then, without loss of information, we can write z instead of x. We also ^ a†, where ^ is the maxassume that there exists a decomposition z ˆ …; imum likelihood estimator for  and a is an ancillary statistic, that is, distribution constant, at least up to certain order. According to the conditional principle, the values of a are kept ®xed as the sample varies, so that we can denote the densities in P x as ^ p…y; j†;

 2 :

This work was partially supported by EU HCOM, contract ERB CHRX CT94 0499.

214

Predictive densities

2

215

Different approaches to the problem

It is possible to predict Y by using a point value predictor g…X†. In order to compare different point value predictors, it is necessary to specify a loss function as a measure of the error. In the case of Y and g…X† being square integrable variables, we have ÿ
ÿ
E …Y ÿ g…X††2 ˆ E …Y ÿ E …YjX††2 ÿ
‡ E …E …YjX† ÿ g…X††2 : Thus, if we use the mean square error to measure the goodness of a prediction, the best predictor g…X† is the one that minimises ÿ
E …E …YjX† ÿ g…X††2 : Notice that we cannot choose E …YjX† as the optimal predictor, since it depends on the unknown value of the parameter . If Y and X are independent, the optimal predictor is then the best estimate for E …Y†. Anyway, a point prediction does not give any information about the `likelihood' of values of Y different from the prediction itself. Another way to solve the problem is based on prediction limits or intervals, that is functions c …X† such that  þ P Y < c …X† ˆ : Exact prediction limits can be found by means of pivotal quantities (functions that depend on Y and X, whose distribution is independent of ), or we can look for an approximate solution with an order of approximation that depends on the sample size. One of the problems with this method is that there is not an obvious generalisation to the multivariate case. Another possibility, which we are going to develop throughout the ^ yjx†, that is chapter, consists of ®nding an optimal predictive density, p… as close as possible, in some sense, to the true conditional density of Y given X ˆ x. We need then to specify a loss function between densities.

3

The form of the optimal solutions

~ †, ^ obtained by substituting the We can use an estimative density p… y; j ~ Anyway, there could unknown parameter  with a suitable estimator . be a predictive density outside the model P x that is closer than any ^ estimative density to the true one, p…y; 0 j†.

216

J. M. Corcuera and F. GiummoleÁ

Example 1 Let X1 ; . . . ; Xn and Y be independent random variables with the same normal distribution N…;  2 †,  2 ®xed. It is known that the estimative density  2† …y ÿ x; (…
;  2 † denotes the density function of a Gaussian random variable with zero mean and variance  2 ) does not provide good prediction intervals for Y. Instead,    1 2   1‡  y ÿ x; n gives exact prediction intervals, since    1 Y ÿ X  N 0;  2 1 ‡ : n Notice that this density, for each x, does not belong to the original parametric model of Y. As we shall see in Example 3, this density is closer than the estimative density to the true one, in a precise sense. Let us consider a regular parametric model of dimension r > k that contains P (from now on we will not indicate the dependence of the model on the data): ^ ! 2 þg; M ˆ fp…y; !j†; where þ is an open set in Rr and  a k-dimensional submanifold of þ. In M we choose a coordinate system ! ˆ …; s†, where i , i ˆ 1; . . . ; k, are the old coordinates in P and sI , I ˆ k ‡ 1; . . . ; r, the new ones in M. We also take s ˆ 0 for the points in P and  and s orthogonal in P. We use indices A; B; C . . . for the components of the coordinate system ! ˆ …; s† in the enlarged model M, i; j; k . . . for those in the original family P and I; J; K . . . for the components of s. Every density in M admits the formal expansion ^ ˆ p…y; j† ^ ‡ p…y; j†s ^ I hI …y; j† ^ ‡


; p…y; ; sj† where ^ ˆ @I log p…y; ; sj†j ^ sˆ0 ; hI …y; j†

I ˆ k ‡ 1; . . . ; r;

and we use the repeated index convention. We consider in M predictive densities of the form ^ ˆ p…y; ; ^ s^j†; ^ ^ ^ † p…yjx† ˆ p…y; !j ^ a smooth function of ^ of order nÿ1 , so that with s^…†

Predictive densities

217

^ ˆ 1 s…† ^ ‡ op …nÿ1 †: s^…† n The aim is then to ®nd a predictive density in M,   ^ I …y; j ^ † ^ ‡ op …nÿ1 †; ^ † ^ 1 ‡ 1 sI …†h ^ p…yjx† ˆ p…y; j n

…1†

that minimises, uniformly in , the leading term in the asymptotic expansion of the risk, h  i ^ ; ^ E L p…yjx†; p…y; j† where L is a loss function to be speci®ed and the expectation is calculated keeping the value of the ancillary statistic a ®xed.

4

A measure of the discrepancy between densities

We consider a very general class of loss functions called divergences. Given a ®xed regular parametric model M ˆ fp…y; !†; ! 2 þg; we can use any smooth function ' : þ  þ ! ‰0; 1†; such that '…!; ! 0 † ˆ 0 () ! ˆ ! 0 ; to measure the discrepancy between two points in M. The condition of minimum in the diagonal can be expressed by 6 'i; …!† ˆ 0;

8 ! 2 þ;

…2†

and 6 'ij; …!† is a positive definite matrix,

8 ! 2 þ;

…3†

where 6 ' means that ' is calculated in the diagonal and the semicolon indicates the argument with respect to which the derivative is taken. By repeatedly differentiating expression (2), we obtain the so-called balance equations for ': 6 'ij; …!†‡ 6 'i;j …!† ˆ 0

…4†

6 'ijk; …!†‡ 6 'ij;k …!†‡ 6 'ik;j …!†‡ 6 'i;jk …!† ˆ 0:

…5†

and

218

J. M. Corcuera and F. GiummoleÁ

It is interesting to observe that similar functions have already been studied by different authors. By changing the sign of ' we obtain a normalized yoke (Blñsild (1991)), while ' itself is a smooth contrast function (Eguchi (1992)). Let us now study the geometry that a divergence induces over the manifold where it is de®ned. By (2) and (3), we have a metric tensor associated with ': gij …!† ˆ 6 'ij; …!†:

…6†

Moreover, ' generates a whole family of af®ne connections whose coef®cients are given by ÿ

ÿijk ˆ ÿ

1‡ÿ 1ÿÿ 6 'ij;k ÿ 6 'k;ij ; 2 2

ÿ 2 R:

…7†

In fact, let  be another coordinate system for the model M where ' is de®ned. Using indices i; j; k for ! and a; b; c for , we can write the Christoffel symbols just de®ned in the new parameterisation  as: ý ! ÿ 1‡ÿ @!i @!j @2 !i @!k 6 'ij;k …!† a b ÿ 6 'i;k …!† a b ÿabc …† ˆ ÿ @ @ 2 @ @ @c ý ! 1ÿÿ @!i @!j @2 !i @!k ÿ 6 'k;ij …!† a b ÿ 6 'k;i …!† a b @ @ 2 @ @ @c ý ! ÿ @!i @!j @2 !i @!k ˆ ÿijk …!† a b ‡ gik …!† a b ; @ @ @ @ @c which is exactly the transformation law that characterises the coef®cients of an af®ne connection (Amari (1985), p. 36). Using (5) and (7), it is easy to see that 1

1

ÿ1

6 'ijk; …!† ˆ ÿijk ‡ ÿikj ‡ ÿ jki : Example 2 An -divergence between two points ! and !~ in M is de®ned as follows:  …  ~ p…y; !† ~ !† ˆ f '…!; p…y; !†…dy†; p…y; !† with

Predictive densities

8 > > <

1‡ 4 …1 ÿ t 2 † 2 1ÿ f …t† ˆ t log t > > : ÿ log t

219

 6ˆ 1 ˆ1  ˆ ÿ1:

-divergences include well-known divergences as particular cases. For  ˆ ÿ1 we obtain the Kullback±Leibler divergence and for  ˆ 0 twice the Hellinger distance. ~ l ˆ log p and Suppose  6ˆ 1. If we write p ˆ p…y; !†, p~ ˆ p…y; !†, ~ under regularity conditions, l~ ˆ log p, …  ÿ1 2 p~ 2 ~ !† ˆ @i p~ …dy†; 'i; …!; ÿ1 p …  1‡ p~ 2 ~ !† ˆ ÿ 'i;j …!; p @i l~ @j l …dy†; p …  1‡ 1 ‡  p~ 2 ~ !† ˆ ÿ p @i l~ @j l~ @k l …dy† 'ij;k …!; 2 p …  1‡ p~ 2 ÿ p @i @j l~ @k l …dy† p   …  1‡ p~ 2 1‡ ~ ~ ~ ˆÿ p @k l @i @j l ‡ @i l@j l …dy† p 2 and, by symmetry, ~ !† ˆ ÿ 'i;jk …!;

  …  1ÿ p 2 1ÿ @j l@k l …dy†: p~ @i l~ @j @k l ‡ 2 p~

By (6) and (4), the metric induced by ' on M is gij …!† ˆ6 'ij; …!† ˆ ÿ 6 'i;j …!† ˆ E! ‰@i l@j lŠ; which corresponds to the Fisher metric. Moreover,    1 1‡ @i l@j l @k l ÿijk …!† ˆ ÿ 6 'ij;k …!† ˆ E! @i @j l ‡ 2 and ÿ1 ÿ ijk …!†

 ˆ ÿ 6 'k;ij …!† ˆ E!

@i @j l ‡

  1ÿ @i l@j l @k l : 2

It is immediate to check that the result also holds for  ˆ 1. The family of af®ne connections de®ned by (7) is then given by

220

J. M. Corcuera and F. GiummoleÁ ÿ

ÿijk …!† ˆ E!

   1 ‡ ÿ @i @j l ‡ @i @j l @k l ; 2

…8†

which for ÿ ˆ ÿ1 are the coef®cients of the Amari -connections (Amari (1985), p. 39). Since the conditional model P depends on x, we consider the family of divergences obtained as the data change. This family depends on the data ^ a†, we through z and, since the value of a is considered ®xed in z ˆ …; ÿ^  ^ ^ write ' . Similarly, gij and ÿijk denote the families of metrics and af®ne connections corresponding to '. We can ®nally say that our purpose is to determine the predictive density in M of the form (1) that minimises, uniformly in , the leading term of the asymptotic expansion of the average divergence h ^ i … ^ ^ † d ; ^ ^ !† ˆ ' …!; ^ !†p…; E ' …!; ^ † is the distribution of the maximum likelihood estimator where p…; conditioned on the observed value of the ancillary a. 5

The asymptotic risk

ÿ

From now on, we denote by g and ÿ, respectively, the metric and the family of connections induced by ' in a ®xed M. If !0 ˆ …0 ; 0† is the true value of the parameter ! in M, the following result holds: Proposition 1 Suppose that ^ ÿ 0 ˆ Op …nÿ1=2 †. Then the average ^ to a predictive density divergence from the true distribution p…y; !0 j† ^ s^j†, ^ is given by p…y; ; h ^ i h ^ i ^ 0†; !0 † ‡ 1 gIJ sI sJ E0 ' …!; ^ !0 † ˆ E0 ' ……; 2n2 ÿ

1 ÿ1 ij K ÿ2 ÿ ijK i s ‡ o…n †; 2n2

…9†

where iij ˆ lim nE0 ‰…^ ÿ 0 †i …^ ÿ 0 † j Š and gIJ ˆ

n!1 ÿ1 0 gIJ …!0 †, ÿ ijK

ÿ1

0 ˆ ÿijK …!0 † and s ˆ s…0 †. ^

~ !0 † as a function of !~ Proof The asymptotic expansion of ' …!; in a neighbourhood of !0 , is

Predictive densities ^

221

^

^

~ !0 † ˆ6 ' …!0 †‡ 6 'A; …!0 †…!~ ÿ !0 †A ' …!; ^

‡ 12 6 'AB; …!0 †…!~ ÿ !0 †A …!~ ÿ !0 †B ^

‡ 16 6 'ABC; …!0 †…!~ ÿ !0 †A …!~ ÿ !0 †B …!~ ÿ !0 †C ^

1 ‡ 24 6 'ABCD; …!0 †…!~ ÿ !0 †A …!~ ÿ !0 †B

 …!~ ÿ !0 †C …!~ ÿ !0 †D ‡


: ^ s^†, with s^ ˆ Op …nÿ1 † and ^ ÿ 0 ˆ Op …nÿ1=2 †. Taking into Now, let !^ ˆ …; account (6) and using the fact that ^

giJ …!0 † ˆ 0;

8i ˆ 1; . . . ; k; J ˆ k ‡ 1; . . . ; r;

…10†

we can write ^ ^ ^ ^ !0 † ˆ 12gij …^ ÿ 0 †i …^ ÿ 0 † j ‡ 12gIJ s^I s^J ' …!; ^ ‡ 16 6 'ijk; …^ ÿ 0 †i …^ ÿ 0 † j …^ ÿ 0 †k ^

‡ 12 6 'ijK; …^ ÿ 0 †i …^ ÿ 0 † j s^K ^ 1 ‡ 24 6 'ijkh; …^ ÿ 0 †i …^ ÿ 0 † j …^ ÿ 0 †k …^ ÿ 0 †h ‡ op …nÿ2 †;

where the coef®cients are calculated in !0 . Moreover, by (10) and (4), it holds ^

^

^

^

^

^

0 ˆ @i 6 'jK; ˆ6 'ijK; ‡ 6 'jK;i and 0 ˆ @j 6 'K;i ˆ6 'jK;i ‡ 6 'K;ij : It is then obvious, using also (7), that ^

ÿ1 ^

 6 'ijK; ˆ ÿ ÿijK :

Thus, we can write ^ ^ ^ ^ !0 † ˆ 12gij …^ ÿ 0 †i …^ ÿ 0 † j ‡ 12gIJ s^I s^J ' …!; ^

‡ 16 6 'ijk; …^ ÿ 0 †i …^ ÿ 0 † j …^ ÿ 0 †k ÿ1 ^

 ÿ 12 ÿijK …^ ÿ 0 †i …^ ÿ 0 † j s^K ^ 1 ‡ 24 6 'ijkh; …^ ÿ 0 †i …^ ÿ 0 † j …^ ÿ 0 †k …^ ÿ 0 †h ‡ op …nÿ2 †;

222

J. M. Corcuera and F. GiummoleÁ

so that ^ ^ ^ I J ^ 0†; !0 † ‡ 1gIJ ^ !0 † ˆ ' ……; ' …!; s^ s^ 2 ÿ1

^ ÿ 12 ÿijK …^ ÿ 0 †i …^ ÿ 0 † j s^K ‡ op …nÿ2 †:

By taking the expectations and disregarding terms of higher order, we obtain h ^ i h ^ i   ^ 0†; !0 † ‡ 1g0 …!0 †E s^I s^J ^ !0 † ˆ E0 ' ……; E0 ' …!; 0 2 IJ h i ÿ1 0 …0 †E0 …^ ÿ 0 †i …^ ÿ 0 † j s^K ‡ o…nÿ2 †: ÿ 12 ÿijK Now, since I ^ ˆ s …0 † ‡ op …nÿ1 †; s^I …† n

we have

h ^ i h ^ i ^ 0†; !0 † ‡ 1g0 …!0 †sI …0 †sJ …0 † ^ !0 † ˆ E0 ' ……; E0 ' …!; 2 IJ h i ÿ1 0 …0 †E0 …^ ÿ 0 †i …^ ÿ 0 † j s^K …0 † ‡ o…nÿ2 †: ÿ12 ÿ ijK

Finally, putting

h i iij ˆ lim nE0 …^ ÿ 0 †i …^ ÿ 0 † j n!1

gives the result. 5.1

&

The optimal predictive density in the enlarged model

Expression (9) allows us to split the problem of prediction into one of estimation followed by a correction of the estimative density. In the i.i.d. case, when the model is a curved exponential family, the term corresponding to the estimation can be expanded further and different estimative densities can be compared. Here we concentrate on the problem of ®nding an expression for s that gives the optimal correction to the estimative density obtained with the maximum likelihood estimator. We can easily prove the following result: Proposition 2 sIopt …0 † ˆ

The optimal choice of s is given by

ÿ1 ij 1 0 I 2 ÿij …0 ; 0†i …0 †

‡ o…nÿ1 †:

…11†

Predictive densities

223

By differentiating (9) with respect to s we obtain

Proof ÿ1

ÿ1

sI ˆ 12 ÿ ijK gKI iij ˆ 12 ÿ ijI iij ; which minimises the risk and thus the optimal choice of s (…gKI † is the inverse of …gIK † and we raise and lower indices by multiplying respectively by …gKI † and …gIK †). & ÿ1

Notice that ÿ ijK is in fact a tensor and represents the components of the embedding curvature of P in the enlarged manifold M. This is easily proved since, if we change the parameterisation in M, using (10), we have ý ! ÿ1 ÿ1 @!i @!j @2 !i @!K ÿ abC …† ˆ ÿ ijK …!† a b ‡ giK …!† a b @ @ @ @ @C ÿ1

ˆ ÿ ijK …!†

@!i @!j @!K ; @a @b @C

where we used indices a; b; C for the coordinate system  and i; j; K for !. (1) and (11) allow us to write the asymptotic expression for the optimal predictive density in M:   ÿ1 ^ † ^ 1 ‡ 1 iij ÿij^ I …; ^ 0†…†h ^ I …y; j ^ † ^ ‡ op …nÿ1 †: p^M …yjx† ˆ p…y; j 2n …12† Of course, this expression depends on M, so that, in general, if we further enlarge the model, we can improve the predictive density (12).

6

A global solution

In this section we want to ®nd suf®cient conditions for the existence of an extended manifold M such that the optimal predictive density in M is the global solution to the problem of prediction. In order to do that, we associate with each point  in P, the vector space that includes all the possible directions in which we can extend the original model. First of all, the tangent space T to P in  can be identi®ed with the linear space generated by ^ ^ ˆ @l…y; j† ; @i l…y; j† @i

i ˆ 1; . . . ; k;

224

J. M. Corcuera and F. GiummoleÁ

^ ˆ log p…y; j†. ^ In T , it is possible to de®ne a scalar prowhere l…y; j† duct by means of the Fisher information matrix, and a family of af®ne connections, the Amari -connections de®ned in (8). A vector space containing T ,  … ^ H ˆ h…y† : h…y† p…y; j†…dy† ˆ 0;  … ^ h2 …y† p…y; j†…dy† <1 ; can be associated to each point  in P. H , as a subspace of L2 …p d†, inherits in a natural way the scalar product … ^ hh…y†; g…y†i ˆ h…y†g…y†p…y; j†…dy†; h; g 2 H : It is immediate to see that this scalar product is compatible with the one given in T by the Fisher information matrix. We refer to [ H H…P† ˆ 2

as the vector ®bre bundle of P. Let h 2 H…P† be a vector ®eld such that @a h ‡

 1‡ 1ÿ E…h@a l† ‡ h@a l  r…H† @a l h 2 2

…13†

belongs to H…P†. The previous expression could be seen as a covariant derivative in H…P†; anyway its interest here is due to the fact that …A†



hr…H† @a l @b l; @c li ˆ ÿ abc ;

…14†

…A†

where ÿabc are the coef®cients of the Amari -connections. Notice that the geometry given by the Fisher information matrix and the Amari -connections does not necessarily coincide with that induced by the considered divergence '. The space M , tangent to M in the points ^ sˆ0 , of the original manifold P, is generated by vectors @A l…y; !j†j A ˆ 1; . . . ; r, belonging to H . The divergence induces on M a metric gAB …!† given by (6), which is in general different from the Fisher metric. We also have a family of af®ne connections or covariant derivatives ÿ …M†

r@

ÿ

^ ^ ^  ÿABC ^ …†g CD …†@D l…y; j†

^ @B l…y; j† A l…y;j†

ÿ

^ D ^ …†@D l…y; j†; ˆ ÿAB

Predictive densities

225

ÿ

where ÿABC are given by (7). In general they are not the Christoffel symbols of the Amari -connections. Since T  M , we have an induced scalar product in T given by gij , and an induced covariant derivative obtained by projecting the preceding one: ÿ …P†

ÿ^ 

^ hk

^ ˆ ÿijh …†g r@ l…y;j†^ @j l…y; j† i

ÿ^

^ ˆ ÿij k …†@k l…y; j†: ^ …†@k l…y; j†

Consider now, for i; j ˆ 1; . . . ; k, vectors ÿ1

ÿ1

ÿ1

ÿ1

…M† …P† I I hM ij ˆ ÿ ij hI ˆ ÿ ij @I l ˆ r @i l @j l ÿ r @i l @j l:

They are orthogonal to P and belong to the tangent space to M in P, thus to the vector ®bre bundle H…P†. hM ij 's represent the orthogonal components to P of the ÿ1-covariant derivative induced in M by the divergence '. Using the de®nition of hM ij , we can rewrite expression (12) of the optimal predictive density in M as   ij ^ † ^ 1 ‡ 1 hM p^ M …yjx† ˆ p…y; j i ‡ op …nÿ1 †: …15† 2n ij We now present an example in which there exists a special orthogonal direction to enlarge the original model P, such that, if the model M contains that direction, the preceding predictive density represents the maximal improvement to an estimative density. 6.1

The case of -divergences

In this case, the ÿ1-covariant derivatives induced by -divergences in M and P coincide with Amari -connections, as we have seen in (8). Moreover, as shown in (14), they can be obtained by projecting (13) over the tangent spaces of M and P. Let 

…A†

k hij ˆ r…H† @i l @j l ÿ ÿ ij @k l;

8i; j ˆ 1; . . . ; k:

Notice that, by (14), these vectors are orthogonal to P. We can now prove the following result, which states the existence of an optimal direction to enlarge the model P. Proposition 3 The optimal direction is given by iij hij and the optimal predictive density can be written, up to order nÿ1 , as

226

J. M. Corcuera and F. GiummoleÁ

  ^ † ^ ‡ 1 ÿ  @i l…y; j ^ †@ ^ j l…y; j ^ † ^ ^ † ^ 1 ‡ 1 iij @i @j l…y; j ^ p…yjx† ˆ p…y; j 2n 2  …A† 1 ‡  ^ ^ k ^ ^ ^ ‡ gij …† ÿ ÿ ij …†@k l…y; j† : 2

Proof The gain in average divergence obtained by using p^M instead of the estimative is ^ ^ 0†; !0 †Š ÿ E ‰'^……; ^ s^opt †; !0 †Š E0 ‰' ……; 0

ˆ

ÿ1 1 ÿ2 …ÿgIJ sIopt sJopt ‡ ÿ ijK iij sK opt † ‡ o…n † 2 2n

ˆ

ÿ1 ÿ1 ÿ1 ÿ1 1 …ÿ14gIJ ÿ ijI ÿ khJ iij ikh ‡ 12 ÿ ijI gIJ ÿ khJ iij ikh † ‡ o…nÿ2 † 2 2n

ˆ

1 ÿ1 I ÿ1 J ij kh ÿ2 ÿ ij ÿ kh i i gIJ ‡ o…n † 8n2

ˆ

1 ÿ1 I ij k ÿ ij i @I lk2 ‡ o…nÿ2 † 8n2

ˆ

1 ij 2 ÿ2 khM ij i k ‡ o…n †: 8n2

On the other hand, 

IJ hhij ; @I ligIJ @J l ˆ hr…H† @i l @j l; @I lig @J l ÿ1

IJ M ˆ h r …M† @i l @j l; @I lig @J l ˆ hij ;

8M  P

and we can write 1 1 ij ij 2 khM ki hhij ; @I ligIJ @J lk2 ij i k ˆ 2 8n 8n2 1 ˆ 2 khhij iij ; @I ligIJ @J lk2 ; 8n which depends only on the projection of hij iij on the tangent space to M in P. The gain is then maximum if and only if vector hij iij belongs to that ij ij space and, in this case, hM ij i ˆ hij i , since hij are orthogonal to P. The ®nal expression for the optimal predictive density follows from (13) and the de®nition of vectors hij . &

Predictive densities

7

227

Examples

In the following examples we always take -divergences as loss functions. Example 3 Let x ˆ …x1 ; x2 ; . . . ; xn † be a random sample of size n from X ˆ …X1 ; X2 ; . . . ; Xn † and suppose that we want to predict a value from Y ˆ …Xn‡1 ‡ Xn‡2 ‡


‡ Xn‡m †=m, where Xi  N…;  2 †, i ˆ 1; 2; . . . ; with  2 known. The maximum likelihood estimator for  is given by

^ ˆ X ˆ

n X Xi iˆ1

n

and it is normally distributed with variance  2 =n. Since Y  N…;  2 =m†, we have that @ l ˆ

m…y ÿ † 2

and

@2 l ˆ ÿ

m : 2

It is also easy to see from (8) that 

ÿ ˆ 0;

8:

Then, the optimal predictive density is given by ^ ^  2 =m† p…yjx† ˆ …y ÿ ; " ý !# ^ 2 …1 ÿ †m m…y ÿ †  1‡ ÿ1 ‡ op …nÿ1 †; 4n 2 where …
;  2 † denotes the normal density with mean 0 and variance  2 . p^ can be written in a close form as    1ÿ 2 1 ^ ^  : ‡ p…yjx† ˆ  y ÿ ; m 2n For  ˆ ÿ1 it coincides with the exact solution that we obtain from the pivotal statistics Y ÿ X  : 1 1 1=2 ‡  m n 

Example 4 Let X0 ; X1 ; . . . ; Xn be an autoregressive process of order 1, with X0 ˆ 0 and Xn jXnÿ1  N…Xnÿ1 ;  2 †, where  2 is known and jj < 1. Suppose that x ˆ …x1 ; x2 ; . . . ; xn † and y ˆ xn‡1 .

228

J. M. Corcuera and F. GiummoleÁ

It is well known that n X Xiÿ1 Xi

^ ˆ

iˆ1 n‡1 X iˆ1

2 Xiÿ1

is asymptotically normal with asymptotic variance 1 ÿ 2 . Moreover, @ l ˆ

xn …xn‡1 ÿ xn † 2

and

@2 l ˆ ÿ

x2n : 2

Then the optimal predictive density is, up to order nÿ1 , ^ n; 2† ^ p…yjx† ˆ …xn‡1 ÿ x " ý !# ^ n †2 1 ÿ  x2n …1 ÿ ^ 2 † …xn‡1 ÿ x ÿ1 ;  1‡ 4n 2 2 which can be written in the equivalent form ý ý !! 1 ÿ  …1 ÿ ^ 2 †x2n 2 ^ ^ n;  1 ‡ : p…yjx† ˆ  xn‡1 ÿ x 2n 2 Example 5 Let y ˆ …y1 ; y2 ; . . . ; yn † be an observation from a random vector Y ˆ …Y1 ; Y2 ; . . . ; Yn † such that Yi ˆ ÿ1 xi1 ‡


‡ ÿm xim ‡ i ;

i ˆ 1; . . . ; n:

We can write, using matrix notation, Y ˆ Xÿ ‡ ; where: Y ˆ …Y1 ; Y2 ; . . . ; Yn † 0 ; * X ˆ …x † is a n  m non-stochastic matrix of rank m  n; ij *  ˆ … ;  ; . . . ;  † 0 is distributed as N…0;  2 I †, with  2 > 0 known; 1 2 n n * ÿ ˆ …ÿ ; ÿ ; . . . ; ÿ † 0 is the unknown parameter. 1 2 n *

We also assume that …X 0 X†ÿ1 ˆ

 ‡ O…nÿ2 †; n

with  known. We want to predict the future observation yn‡1 from the random variable

Predictive densities

229

0 Yn‡1 ˆ ÿ1 xn‡1;1 ‡


‡ ÿm xn‡1;m ‡ n‡1 ˆ xn‡1 ÿ ‡ n‡1 ;

with xn‡1 ˆ …xn‡1;1 ; . . . ; xn‡1;m † 0 known and n‡1  N…0;  2 †. The maximum likelihood estimator for the unknown parameter ÿ coincides with the least square estimator: ÿ^ ˆ …X 0 X†ÿ1 X 0 y: The covariance matrix of ÿ^ is ^ ˆ 2 …X 0 X†ÿ1 ; Var…ÿ† 0 ^ ˆ  2 . Moreover, since Yn‡1  N…xn‡1 ÿ;  2 †, we so that limn!1 nVar…ÿ† have that

@i l…yn‡1 ; ÿ† ˆ

0 yn‡1 ÿ xn‡1 ÿ xn‡1;i 2 

and @ij l…yn‡1 ; ÿ† ˆ ÿ

xn‡1;i xn‡1; j : 2

Thus, the optimal predictive density for yn‡1 with respect to an divergence, is " 1ÿ 0 0 2 ^ ^ n‡1 jy† ˆ …yn‡1 ÿ xn‡1 ÿ;  † 1 ‡ p…y x  xn‡1 4n n‡1 ý !# 0 ^2 …yn‡1 ÿ xn‡1 ÿ†  ÿ1 ‡ op …nÿ1 †; 2 which, up to terms of order nÿ1 , can be written as ý  ÿ1 ! 1 ÿ  0 0 ^ 1ÿ 2 : x  xn‡1 N xn‡1 ÿ; 2n n‡1 Again, for  ˆ ÿ1 we obtain a predictive density that gives exact prediction intervals. References Amari, S. (1985), Differential-Geometrical Methods in Statistics, Lecture Notes in Statistics No. 28, New York: Springer Verlag. Blsild, P. (1991), `Yokes and Tensors Derived from Yokes', Annals of the Institute of Statistical Mathematics, 43(1): 95±113. Eguchi, S. (1992), `Geometry of Minimum Contrast', Hiroshima Math. J., 22: 631±647.

8

An alternative comparison of classical tests: assessing the effects of curvature Kees Jan van Garderen

1

Introduction

The last two decades have seen a rapid growth in the application of differential geometry in statistics. Efron (1975) stimulated much of this research with his de®nition of statistical curvature and he showed that curvature has serious consequences for statistical inference. Many papers on the application of differential geometry in statistics go straight into de®ning all the necessary tools, such as the metric and an af®ne connection on a manifold, and show how they can be used in statistical analysis. The emphasis is predominantly on asymptotic theory and applications are mainly in estimation, information loss and higher-order ef®ciency. Barndorff-Nielsen, Cox and Reid (1986) give a very accessible account of the relevant ideas in differential geometry and also provide a historical overview; see also Amari (1985) and Okamoto, Amari and Takeuchi (1991) provide a brief recent summary of main achievements. This chapter is concerned with the effects of curvature on hypothesis testing. However, our approach differs from the differential geometrical approach mentioned above in a number of ways. First, we give a global analysis, i.e. for the whole of the sample and parameter space, not merely in a neighbourhood of a ®xed point such as the true parameter value. Secondly, the analysis is valid for all sample sizes, not just asymptotically. Finally, we are concerned with hypothesis testing, which is less common, and use the partitioning of the sample space into critical region and acceptance region to illustrate our arguments graphically, rather than solely investigating the (analytic) properties of test statistic(s). More speci®cally, this chapter investigates the effects of global curvature on the different classical tests: the Lagrange multiplier (LM), Wald, and likelihood ratio (LR) tests, as well as the point optimal (PO) and geodesic tests. Traditional approaches can be found in most econometrics 230

Curvature and classical tests

231

text books, and see Engle (1984) and Buse (1982) for a comparison between the LM, Wald and LR tests using simple diagrams of the loglikelihood. The LR measures the difference in height of the log-likelihood between restricted and unrestricted models, the LM test measures the slope of the log-likelihood at the null, and the Wald measures the distance between the unrestricted MLE and the restricted value under the null directly. In these approaches all three tests appear equally reasonable, a view reinforced by the fact that they are asymptotically equivalent. We shall see that this can be very misleading and that the tests make very different choices in terms of their critical regions. We also consider the global curvature of the model and show that the one-sided LM and PO tests, for instance, are potentially inconsistent for distant alternatives because of their straight critical regions. Another comparison between the classical tests from a differential geometrical point of view was developed by Kumon and Amari (1983) and Amari (1983); also see Amari (1985, chapter 6). They analysed ®rstorder ef®cient (most powerful) tests, showed that they are automatically second-order ef®cient (most powerful), and compared the third-order power loss, i.e. the asymptotic difference between the power of a test and the power envelope. This power loss depends on the curvature of the model at H0 , common to all tests, and the curvature properties of the boundary of the critical region. The LR is shown to have good performance over a `wide range' of alternatives, the LM test in a range close to H0 , and the Wald test for more distant alternatives, where distance is measured in terms of the geodesic distance to local alternatives. Other articles on hypothesis testing dealing speci®cally with curvature include Efron (1975, section 8), Critchley, Marriott and Salmon (1996), Davidson (1992) and Hillier (1990, 1995). Efron (1975) showed that the (local) curvature at H0 gives a good indication of the performance of the locally most powerful test and concludes that a point optimal test may be preferable if the curvature is larger than 1/8, as a rough guide. Hillier (1990, 1995) considers the global curvature properties of the model and proposes a measure of global curvature. If the measure is small, then curvature does not pose a serious problem and any of the classical tests will give a reasonable test. Davidson (1992) and Critchley et al. (1996) consider the Wald test and propose alternative geometric solutions to the non-invariance problem of the Wald statistic ± a C…† type test and a geodesic test, respectively ± but the two tests must still contend with a number of practical problems. There are two main approaches in analysing testing procedures. The usual approach is to de®ne the test statistic and evaluate its properties, such as its exact, asymptotic or simulated distribution, and ®nd size and

232

Kees Jan van Garderen

power properties. The second approach is in the tradition of Neyman and Pearson and looks at the partition of the sample space into a critical region (CR) and an acceptance region de®ned by the test. The two approaches are equivalent in the sense that every statistic induces a partition of the sample space and vice versa, but in practice they are quite different. This chapter follows the second approach and is therefore closely related to Hillier (1990, 1995); see section 5 below in particular. He uses the change in the PO test CR as a measure of global curvature. We show how curvature also affects the CR of other classical test procedures. This chapter is further related to Kumon and Amari (1983), Amari (1983) and Amari (1985), as mentioned before, who call the boundaries of the CR ancillary manifolds but concentrate on an asymptotic treatment. It is well known that if the model is a one-parameter linear exponential model, then there exists a uniformly most powerful (UMP) test against one-sided alternatives. Conversely, it is also true that if a UMP test exists, then the model must be a one-parameter linear exponential model, as was shown by Pfanzagl (1968). In Theorem 1 in section 2 we give a new and simple proof of a slightly stronger result, assuming the density belongs to an exponential family of arbitrary, but ®xed, dimension. We will assume throughout the chapter that the models of interest are (curved) exponential models. If the density belongs to a curved exponential family, it means that the dimension of the minimal suf®cient statistic is ®nite and ®xed. This allows us to concentrate on the critical region, which is a subset in the sample space of ®xed dimension for all sample sizes, rather than on the test statistic (which concentrates the suf®cient statistic into a one-dimensional statistic). A wide variety of models belong to this class of models. In van Garderen (1997a) we motivate the use of curved exponential models and give a variety of examples in econometrics, such as the seemingly unrelated regressions model, the single structural equation model, demand systems, and time-series models, as well as references to other examples in, for instance, continuous time-series models. Another reason for the assumption is that, since UMP tests do not exist if the model is curved, it seems natural to use a general class of models that includes the one-parameter linear exponential model to determine the effects of curvature. Models close to linear exponential models can then be analysed, as well as models that are very different. Of course, this can be generalised to non-exponential families. Efron (1975) derived a curvature measure based on exponential models and generalised this to arbitrary families of distributions. Many expressions

Curvature and classical tests

233

derived here have analogues in terms of derivatives of the likelihood also. We shall not pursue such a generalisation, but the intuition developed here holds more generally. It should be noted that the dimension in such a generalisation is implicitly restricted because only a ®nite number of derivatives are taken into account and, secondly, these derivatives and their expectations are evaluated at speci®c points.1 This leads quite naturally to asymptotic theory in a neighbourhood of a chosen point. In contrast, we restrict the dimension explicitly from the onset, because we set out to consider the whole of the sample and parameter space and consider ®nite sample behaviour. No test exists that is best for all parameter values if the model is curved. Different test procedures achieve good power properties for certain regions of the parameter space (only), or achieve some overall desirable power properties, and some tests do not achieve either. We want to assess the reasonableness of tests based on the shape of the CR. The LM and PO test give a straight boundary of the critical region regardless of the model curvature, as shown in sections 3 and 4, and thus in effect ignore the curvature of the model. The Wald test does give a curved boundary, but the test statistic is inherently a non-geometric quantity and depends on the parameterisation used (see section 5). A geometric counterpart introduced in section 6, called the geodesic test, also gives a curved boundary, but raises practical and fundamental questions. The analysis of the LR test in section 7 is the most interesting. The LR test gives a curved boundary that trades off the global curvature against the local curvature of the model, and the LR test can be seen as an average of best critical regions in all directions. If a model is not too curved in the sense of both Efron and Hillier, then any of the tests give a reasonable solution to the non-existence of a uniformly most powerful test. The main conclusion of the present analysis is that if the model is seriously curved then the LR test appears the clear favourite, from both a practical and a theoretical point of view. Section 8 provides an example and shows the CRs of the different tests when testing for autocorrelation or for a unit root in the ®rst-order autoregressive (AR(1)) model. Section 9 concludes. 1 Unless we regard the derivative as elements of a function space, e.g. the score which can be regarded as a function of the parameters. If the density belongs to an exponential family then there is a ®nite dimensional basis for the function space of scores. If the density does not belong to an exponential family then this space is in®nite dimensional.

234

Kees Jan van Garderen

2

UMP tests and one-parameter linear exponential models

To test a simple null hypothesis H0 against a simple alternative Ha we can use the Neyman±Pearson lemma to give a test that is at least as powerful as any other test of the same size. Neyman±Pearson Lemma (NPL) Let x 2 X be a random variable of dimension n, with likelihood function L…xjH0 † and L…xjHa † under the null and alternative hypothesis, respectively. When testing H0 versus Ha the critical region ÿ   ÿ L…xjHa † ÿ …1† BCR ˆ x 2 X ÿ  c ; L…xjH0 † c  0, is most powerful among all tests with the same size   Prob‰x 2 CR=H0 Š. The critical value c determines the size . In general, if x is continuous, we can also determine for any ; c such that the size of the test is . The lemma is very general in the sense that it gives the best critical region (BCR) regardless of the distributions under H0 and Ha . The hypotheses may for instance be non-nested. Naturally, the CR (critical region) may change if the alternative changes, or the null for that matter. This implies that a test which is best against one alternative is in general not best against another alternative. Only in special cases will the two tests coincide. If the CR is the same for all elements in the alternative, then this region is called a uniformly most powerful size  region. If the system of CRs has this property for each , the test is called uniformly most powerful (UMP). Finally note that the NPL can be used to determine the power envelope, which is the function describing the maximum attainable power for any test at each point under the alternative. The power envelope is found by constructing, for each point in the alternative, the BCR and calculating the power for this alternative. Since this test is at least as powerful as any other test, this power must be on the power envelope. The BCRs have a particularly simple form in exponential families where the boundary of the CR is orthogonal to the direction from the null to the alternative hypothesis (see Theorem 1 below). An exponential model is a family of distributions with densities that can be written as pdf …x; † ˆ expf 0 t…x† ÿ …†gh…x†:

…2†

This is called the canonical representation with canonical parameter  2 N, a k  1 vector, and canonical statistic t…x† 2 T, a k  1 vector of

Curvature and classical tests

235

random variables. The representation is called minimal if k is the smallest possible integer such that (2) holds. In this case, t…x† is minimal suf®cient and has itself an exponential distribution similar to the distribution of x, but with a different h…
† (see Lehmann (1986), Lemma 8, p. 58). Thus, pdf …t; † ˆ expf 0 t ÿ …†gh…t†: The minimal suf®ciency of t implies that there is no essential loss of information by considering only t, instead of the whole sample x (see, for example, Lehmann (1986), 1.5), and in what follows we shall work directly with the distribution of t itself. If  is a smooth function of a parameter , of dimension d < k, then the model is a …k; d†-curved exponential model. In that case, …† describes a curved d-dimensional manifold (e.g. a surface) in N. The parameter  may originate from the way the model was initially written down, which is not usually in canonical form.2 k is the dimension of t…x†, which is a minimal suf®cient statistic, i.e. a statistic of smallest dimension that still contains all information in the sample. The expectation of t is a function of , which we denote by …†. This function is one-to-one and easily derived using the fact that in exponential models …†  E ‰tŠ ˆ @…†=@:

…3†

In curved exponential models, the expectation of t is a function of , since  ˆ …†, and we write …†  E ‰tŠ. …† is obtained by substitution …† ˆ E ‰tŠ ˆ ……††:

…4†

…† is a d-dimensional manifold and can be thought of as lying in the sample space for t. We shall make regular use of (4) and the following expression obtained by the chain rule  0 ˆ t…†: …5†   The importance of exponential models is that the dimension of t is constant, regardless of the sample size. In van Garderen (1997a) we prove that curved exponential models can be de®ned by the property that the number of parameters is smaller than the dimension of a minimal suf®cient statistic, even for non-i.i.d. observations. This basically means that the dimension of the problem is ®nite and ®xed. The paper motivates curved exponential models more generally and provides econometric examples. 2 The canonical form is not always the most useful form for economic interpretation or even for distributional derivations, but is very instructive here.

236

Kees Jan van Garderen

If the densities belong to a one-parameter linear exponential family, then it is easily shown, as in Theorem 1 below, that the CR is the same for all (one-sided) alternatives. Hence, there is a class of models for which a UMP test exists, but the question is whether or not there are any other classes of models with this property. The short answer, under mild regularity conditions, is no. Borges and Pfanzagl (1963) showed that the existence of a UMP test for all sample sizes and all signi®cance levels  implies a one-parameter exponential family. Pfanzagl (1968) extended this result to one level of signi®cance and all (arbitrarily large) sample sizes. Their assumptions include that the family of distributions is mutually absolutely continuous, a property shared by exponential families. Theorem 1 gives a new simple proof of a slightly more general result, assuming that the density belongs to an exponential family of arbitrary dimension. The dimension of this family is arbitrary but ®xed, which means that the dimension of the problem does not increase with the sample size. In the introduction we have already given a number of arguments why the assumption may not be too restrictive. It is also worth noting that in addition to the regularity conditions, which are considered weak (Lehmann (1986), p. 80), Pfanzagl requires that a UMP test exists for arbitrarily large sample sizes. We restrict the dimension by assuming an exponential model from the start, and the existence of a UMP test for one sample size (and one ) then implies the existence of a UMP test for all sample sizes (and all 's). Theorem 1 Let the density of x (of dimension n) belong to an exponential family of arbitrary, but ®xed dimension k. (a) If the model is a one-parameter linear exponential model then a onesided UMP test exists for all n and . (b) If a UMP test exists for one  and one n  k then the density is a oneparameter linear exponential model. Proof Let  be the …k  1† canonical parameter vector and t…x† the …k  1† canonical statistic. Using the exponential distribution of the minimal suf®cient statistic t, we obtain the BCR for testing H 0 :  ˆ 0 against Ha :  ˆ a directly from the NPL as BCR ˆ ft 2 T j exp……a ÿ 0 † 0 t ÿ …a † ‡ …0 ††  c g

…6†

or simply, by de®ning c…a†  ˆ ln…c † ‡ …a † ÿ …0 †, as BCR ˆ f…a ÿ 0 † 0 t  c…a†  g:

…7†

Curvature and classical tests

237

The boundary of the BCR is a hyperplane of dimension …k ÿ 1† which is orthogonal to …a ÿ 0 †. The BCR is determined by the direction of …a ÿ 0 † only. The length of …a ÿ 0 † is irrelevant because c…a†  is determined by the size condition: Prob‰…a ÿ 0 † 0 t  c…a†  j H0 Š ˆ :

…8†

(a) In a one-parameter linear exponential model the direction of …a ÿ0 † is constant for all a as long as the sign does not change, i.e. a one-sided test in terms of , since the vector is one-dimensional.3 The BCR of size  is constant for all alternatives, and, since this holds for all , the CRs de®ne a UMP test. (b) If the size  BCR is constant for all a , then …a ÿ 0 † must have the same direction for any parameter value a in the alternative, since every such vector is orthogonal to the same …k ÿ 1†-dimensional hyperplane. Hence the vectors are proportional and …a ÿ 0 † ˆ …a1 ÿ0 † for all a and some ®xed a1. But this constitutes a one-parameter exponential family with  as canonical parameter and s…x† ˆ …a1 ÿ 0 † 0 t as canonical statistic. This holds for any  used to construct the BCR. & The statements (a) and (b) follow from simple linear algebra, since a plane determines a normal, and a normal determines a plane, as illustrated in ®gure 8.1. Note that …a ÿ 0 † is a vector in the parameter space, whereas the boundary of CR…a† is a hyperplane in the sample space. The proof therefore makes use of the duality between the parameter space and the sample space.4 This explains why the condition n  k is necessary in part (b) of the proof. If n  k ÿ 1, the boundary of the BCR no longer de®nes a unique normal vector in N (there are fewer than k ÿ 1 restrictions on ), but rather a …k ÿ n ‡ 1†-dimensional hyperplane. Part (b) of the proof shows that the alternative does not need to be a continuous line, but may consist of points as long as these points are on

3  is in general a function of . The necessary and suf®cient condition for (a) in terms of  is that the function …† is one-to-one. Suf®ciency is immediate and necessity follows from the fact that there would otherwise be two values of  with the same value of . We could not sensibly test one value of  against the other because the two models are essentially the same (observationally equivalent), and the NPL does not de®ne a CR. Note that the condition guarantees likelihood ratio monotonicity in terms of . P 4 The duality here is between N and T through the bilinear product h; ti ˆ  0 t ˆ i ti , de®ned on N  T. This is different from the usual convex duality that is used in the theory of exponential models, which is really between the canonical parameter space and the expectation parameter space, and the dual of …† is de®ned through the Legendre transform. See Barndorff-Nielsen (1978).

238

Kees Jan van Garderen

T: Sample Space

N: Parameter Space

Boundary CR: t'(η - η ) = c o α a

η*

(a)

v* ηo

ηa v

η a1

v*

v1

v

Acceptance Region

v1

Critical Region

Reject Ho

Figure 8.1 Best critical regions

the same line. In fact, we can think of the NPL as de®ning straightness in a space of distributions even more general than exponential families, as well as straightness in the sample space, since the NPL is valid for all distributions. The proof also shows a well-known result that if we are testing one linear restriction on the canonical parameter , we can reparameterise the model to obtain a UMP test (see, for instance, Lehmann (1986)). Given the regularity conditions in Pfanzagl (1968) or the assumption made here, a UMP test exists if, and only if, the model is one-parameter linear exponential. This is the ideal situation and most practical situations do not fall into this category. In curved models the NPL cannot be used to give a best test, since no test exists that maximises power against every alternative. We have to use a different approach and compromise power against different alternatives. Different tests compromise in a different way.5 In what follows we will show how different tests choose different critical regions and, in the spirit of Theorem 1, show that they can be thought of as choosing directions for …a ÿ 0 †. The point optimal test chooses an arbitrary `suitable' point  from which a and the direction follow. The LM, locally most powerful and locally best test choose a direction equal to the gradient of …† at 0 . The Wald test uses the gradient of …† at point(s) different from 0 , but this direction is not used to construct a BCR in the 5 Cox and Hinkley (1974, p. 102) offer three suggestions: (a) pick an arbitrary point and maximise power against this alternative; (b) choose this point close to 0 and maximise the power locally near the null hypothesis; and (c) maximise some weighted average of power.

Curvature and classical tests

239

sense that the LM and PO test do. The Wald test introduces a distance concept that is generally not compatible with the model. The geodesic test uses a geometrical notion of distance but there are some theoretical, as well as practical problems, although it does give a test that takes the curvature of the model into account and gives a curved boundary for the CR. The LR test can be viewed as averaging over all possible directions.

3

The point optimal (PO) test

A point optimal test is a direct application of the NPL. It simply chooses a particular parameter point  in the alternative (and a 0 under the null if it is composite) and forms the best CR for testing the simple null versus this simple alternative. If we write a   ˆ … † and 0 ˆ …0 † in (7), we see immediately that the CR for the PO test equals CRPO ˆ f…… † ÿ …0 †† 0 t  c g:

…9†

The CR is determined by the direction of … ÿ 0 † since the boundary of CRPO is a hyperplane orthogonal to this direction. The size condition determines c , and hence how far the boundary is translated from zero. It is worth recalling that c changes with  , depending on the direction and length of ……  † ÿ 0 †, so that c ˆ c …… †† ˆ c … †. (The LR test indirectly uses (9) in all directions with c ®xed, as we shall see below.) Since zero is just an arbitrary point in the sample space, it is more relevant how distant the boundary is from the expectation of t under the null, …0 †. We therefore rewrite the CR by de®ning c~ ˆ c ÿ … ÿ 0 † 0 …0 †, as CRPO ˆ f… ÿ 0 † 0 …t ÿ …0 ††  c~ g;

…10†

which is shown in ®gure 8.2. The left-hand frame shows the canonical parameter space N and the canonical manifold …† with the relevant directions determining the PO test. The right-hand frame shows the sample space T and the CRs for two different choices of  . The expectation manifold, E ‰tŠ ˆ …†, is superimposed on T. The left frame shows how the vector … ÿ 0 † is obtained: choosing  determines a direction and hence a test (choosing  is choosing a different direction and a different test). The direction of … ÿ 0 † depends on  only through …†. This has two consequences: the picture and conclusions drawn from it are the same regardless of the parameterisation used (i.e. whether  is used or some other parameter ), but, on the other hand, a different parameterisation will generally result in a different

240

Kees Jan van Garderen T: Sample Space

N: Parameter Space

CR** o

Acceptance Region

**

w

o

v *

w v

CR*: Reject H o

Figure 8.2 Point optimal tests based on the points  and 

choice of  and hence a different direction. In this sense the test is not invariant under reparameterisations. The PO test is not as widely applied as the LM, Wald and LR tests, but has been successfully exploited in econometrics, mainly in time-series applications. Early examples are Kadiyala (1970) and Berenblutt and Webb (1973), and later examples include King (1987), Brooks (1993), Silvapulle and King (1993) and Saikkonen and Luukkonen (1993). King and Wu (1994) provide a survey and references, including most of King's own and joint work. Evans and King (1985) derive a PO test for heteroskedasticity. The test statistic is often quite easy to derive and the main dif®culty is determining the critical value. c … † can in principle be obtained numerically by, for example, Monte Carlo integration. In time-series models the statistic is frequently a ratio of quadratic forms in normal variates and the critical value can be determined by Imhof's (1961) or Davies's (1980) procedure, similar to obtaining the exact DW critical value. Apart from its simplicity, the test has some other attractions. The power of the test equals the power envelope for at least one parameter point, and we can choose this point based on economic or other external considerations. This freedom, on the other hand, makes the procedure ad hoc, since there is no general principle or optimality consideration to choose the point. One solution to this arbitrariness is to choose the point closest to the null for which the power equals a pre-set level such as 0.8. This gives the

Curvature and classical tests

241

ÿ-optimal test of Davies (1969) but is often computationally dif®cult.6 Other solutions include to choose the maximum or the middle of the  values, but this determines a direction which depends on the parameterisation used. Using  instead of  gives a different middle value and consequently … † 6ˆ … †, resulting in different tests. Another problem is that we can easily think of parameterisations …† that give the same  for two different values of , disturbing the idea that we can sensibly talk about minima, maxima or middle values. Figure 8.2 suggests part of the answer. We should be thinking in terms of choosing a direction, taking into account the global properties of the model, rather than choosing a point in an arbitrary parameterisation. A further question concerning the choice of PO point is whether or not it should be the same regardless of the sample size. The PO test based on  in ®gure 8.2 is very unattractive for large sample sizes and local alternatives because the boundary of the CR is virtually orthogonal to the BCR based on local alternatives (and to the LR, LM and Wald CRs)! It appears more sensible to choose the point optimal direction for large sample sizes based on (a sequence of) local alternatives which achieve maximal power against local alternatives, since, even for these tests, the power will approach 1 for distant alternatives such as  , provided the model is not too curved. If the model is very curved then it is possible, because of the straight boundary, that …† curves back into the acceptance region for large values of . Put differently, the expectation of t in such a case lies in the acceptance region for values of  very distant Pfrom 0 . Asymptotically, with independent sampling and t ˆ …1=n† niˆ1 ti , we have that t ! …† as n ! 1. This means that the test is not globally consistent and, for certain values of the parameter, very different from 0 , the test accepts the null with probability one. Returning to the attractions of the PO test, it follows directly from the NPL that the PO test can be applied to non-nested problems, e.g. testing MA versus AR errors as in Silvapulle and King (1993). Furthermore, if …† is a straight line, then the BCR is constant and the test is UMP, as in Theorem 1. Put differently, if a UMP test exists, then the PO tests will produce it. Even if a UMP test does not exist, the PO test may still give a satisfactory test. In particular if …† is not too curved and therefore ……† ÿ 0 † does not change very much, the BCRs do not change very 6 The ÿ-optimal test may not exist if the power envelope does not reach the pre-speci®ed level such as 0.8.

242

Kees Jan van Garderen

much either. Hence, the power of any of the PO tests will be close to the power envelope. If …† is seriously curved and …… † ÿ 0 † changes a lot for different values of  in the alternative, then the BCRs also change a lot. This means that very different tests are obtained for different choices of  . The test becomes sensitive to the choice of   and this choice gets more important. A small value of  causes a loss of power against alternatives further away from H0 and vice versa, and, because the boundary of the CR is straight, it may be impossible to get desirable power properties against the whole range of alternatives. These ideas were developed in detail by Hillier (1990, 1995). He analysed the global behaviour of the BCRs and suggested a measure of the variability of the BCRs based on the probability of the enveloping region, which he de®ned as the union of all possible ®xed-size PO critical regions. If the probability of the enveloping region under H0 is close to , then the model cannot be very curved, but the model must be globally curved if this probability is much larger than . It is clear from ®gure 8.2 that in the two-dimensional case this relates to the maximum possible angle between any two vectors … ÿ 0 † and … ÿ 0 †, which is a measure of the total curvature of …†. This global concept of curvature is different from Efron's statistical curvature, which is the curvature of the model at a particular point. Efron (1975) related his curvature to the rate of change of the BCR at H0 , whereas Hillier's global measure relates to the total change in all BCRs. The local measure of curvature, evaluated at 0 , does, however, give a good indication of the relative performance of the LMP test, as Efron showed. Large curvature leads to poor performance of the LMP test as compared with a PO test. Efron therefore suggested the use of a PO test as an alternative to the LMP test if the model has curvature larger than 1/8 at H0 , which we shall now discuss.

4

Lagrange multiplier (LM), locally most powerful (LMP) and locally best (LB) test

The locally most powerful (LMP) test is a test that gives maximal power against alternatives in a neighbourhood of the null.7 Although the NPL gives the BCR for all a 6ˆ 0 , it is clear that, if we let a ! 0 , 7

A level- test ' is said to be LMP if, given any other level- test , there exists a number " such that power' …†  power …† for all alternatives  in an "-neighbourhood of 0 , i.e. k ÿ k < ", where k
k is some metric. See Lehmann (1986, p. 537).

Curvature and classical tests

243

…a ÿ 0 † ! 0, and no CR is de®ned. The main point of Theorem 1 is, however, that the direction of …a ÿ 0 † determines the CR, not its length (the critical value c ˆ c …† ˆ c ……†† absorbs the length because of the size condition). The direction does not change if we divide both sides in (7) by ka ÿ 0 k with k
k some measure of distance, and we write the BCR as ……a † ÿ …0 †† 0 ……a †† ÿ ……0 †† c … † tÿ   a : ka ÿ 0 k ka ÿ 0 k ka ÿ 0 k

…11†

We may now let the point a approach 0 arbitrarily closely and take the limit as a ! 0 on both sides to obtain the CR   …0 † 0  t  c ‡ @…0 †=@ : …12† CRLMP ˆ  This shows that the CR of the LMP test is determined by the direction of @…0 †=@ ˆ @…†=@jˆ0 , the gradient of …† at 0 . The CR is bounded by a hyperplane orthogonal to @…0 †=@. We can rewrite the CR as in (10), using (4),   …0 † 0  …t ÿ …0 ††  c : …13† CRLMP ˆ  Figure 8.3 illustrates how the direction and CR are obtained (note that @=@ 6ˆ @=@).8 The equivalence between the tests will be shown below. If …† is a straight line then @=@ has the same direction as … ÿ 0 † for any  and hence the CR of the LMP test is the same as the CR for any PO test. Thus the LMP test also produces the UMP test if it exists. Figure 8.3 also illustrates the parameterisation invariance of the test because the direction is the same regardless of which parameter is used. Analytically this follows because ‰@ 0 @Š ˆ …@=@†‰@ 0 =@Š; and the direction @ 0 =@ determined by  is proportional to @ 0 =@ since …@=@† is 1  1 and the CRs are the same. If …† is curved, the LMP test gives the same CR regardless of how curved the model is. The LMP uses the direction only at 0 , essentially ignoring the curvature. Figure 8.3 shows that this can have serious consequences. Since the boundary is straight, it is possible that …† curves 8 Since …† ˆ @=@, we have by the chain rule @=@ 0 ˆ …@2 =@@ 0 †…@=@ 0 † ˆ …@=@ 0 †, where  ˆ Cov…t†:

244

Kees Jan van Garderen

N: Parameter Space

T: Sample Space CR: Reject H o

o ο

Acceptance Region

Figure 8.3 The one-sided LM, LMP and LB test

back into the acceptance region for large values of  and the testPis not consistent. Again, with independent sampling and t ˆ 1=n niˆ1 ti , t ! …† as n ! 1, and the test will accept the null with probability one asymptotically for values of  very distant from 0 . This is similar to the PO test but, since the LMP test chooses an extreme direction, the problem may be more serious. Usually the test is globally consistent, but even in less extreme cases we would expect poor power performance away from the null if the model has appreciable global curvature. The relation between the curvature and performance of the LMP test was pointed out by Efron (1975, section 8). He concluded that, if the curvature at 0 is larger than 1/8, then the performance of the LMP is seriously affected and a PO test should be considered. His measure describes the local curvature at H0 . Figure 8.3 gives the global picture. The LMP test is derived here as a limiting case of a PO test if we let  ! 0 . The LMP test maximises the slope of the power curve at the null in most cases. A test that maximises the slope of the power is often referred to as a locally best (LB) test. Lehmann (1986, p. 528) gives counter-examples when the two tests are different and suf®cient conditions for the two tests are to be the same: if there is a unique level  LB test, then it is the unique LMP test.9 The LB test can be found directly using a generalised version of the NPL (e.g. Cox and „ Hinkley (1974), p. 126): the region w ˆ ft 2 Tj f …t†  cg…t†g maximises w f …t†dt, subject to „ g…t†dt ˆ a, provided such a c exists. Since the LB test maximises the w slope of the power function, we have

9

See also Cox and Hinkley (1974), pp. 107 and 113, for relevant discussions.

Curvature and classical tests

245

 …  …0 † 0 …t ÿ …0 †† expf00 t ÿ gdt;  w w … … s:t: pdf …t; 0 ††dt ˆ  , expf00 t ÿ gdt ˆ ;

max 

…

pdf …t; ††dtjˆ0 ˆ

w

w

where we have used (3) and the fact that in exponential models we may interchange differentiation and integration. The region w becomes,   …0 † 0 …t ÿ …0 ††  c ; …14† CRLB ˆ  which is equal to (13) with c ˆ c  . The Lagrange multiplier (LM) test, or (ef®cient) score test, is one of the most widely applied tests in econometrics (see Breusch and Pagan (1980), Engle (1984) or Godfrey (1988) and references therein). The general form of the test is to reject for large values of the LM test statistic ^ ÿ1 s…† ^  c ; ^ 0 I…† LM ˆ s…†

…15†

^ is the score statistic @`…; ^ t†=@, a d  1 vector, and I…† ^ is the where s…† ^ d  d information matrix, all evaluated at the restricted MLE…H0 †. A one-sided version of the LM test if d ˆ 1 (ignoring sign) would be to reject for large values of ^  c : ^ 1=2 s…† I…†

…16†

The restricted MLE in the present case is 0 because H0 is simple. Furthermore, using (5), the score in an exponential model can be written as s…† ˆ

1…t; …††  0 ˆ ‰t ÿ …†Š:  

Substitution gives   … † 0 CRLM ˆ I…0 †ÿ1=2 0 ‰t ÿ …0 †Š  c ; 

…17†

…18†

which, after pre-multiplication by I…0 †1=2 , is equal to (13) and (14). Hence, the one-sided version of the LM test in exponential models is equal to the LMP and LB test. Of course, if d > 1 the statistic …@=@† is d  1 and needs to be reduced to a one-dimensional test statistic. The LM statistic does this by weighting the elements by the inverse of the Fisher information matrix, which, incidentally, keeps the test parameterisation invariant even if d > 1. A different matrix could be chosen to give more weight to particular directions.

246

Kees Jan van Garderen

5

The Wald test

One of the main attractions of the Wald test is that it requires estimation of the parameters only under the alternative hypothesis. The Wald test is slightly at odds with the other tests discussed in this chapter because the Wald statistic is inherently a non-geometric quantity. The test statistic depends on the algebraic formulation of the restrictions de®ning the null hypothesis. The main problem with the Wald test is that it uses only the ®rst term in the Taylor expansion of the restriction, and there are uncountably many other restrictions that correspond, up to ®rst order, with the restriction written down. Gregory and Veall (1985) showed that this non-invariance can lead to drastically different conclusions, even in a very simple model. Nevertheless, the Wald test is one of the classic tests and widely applied. The general form of the Wald statistic for testing H0 : r…† ˆ 0 versus

Ha : r…† > 0;

…19†

where r…† describe some non-linear restrictions on , is given by ^ ˆ r…† ^ 0 ‰R…† ^ 0 I…† ^ ÿ1 R…†Š ^ ÿ1 r…†; ^ w…†

^ ˆ R…†

r…† j ^:  ˆ

…20†

In section 8 we use the following example to demonstrate a number of problems set out in this section. Since H0 is simple we may set 0 ˆ 0 without loss of generality. There are many ways of writing this restriction in the form (19), but consider, rb …† ˆ

exp…b† ÿ 1 ˆ 0; b

with Rb …† ˆ exp…b†;

…21†

which for each value of b gives a different way of writing the restriction. In the limiting case b ! 0, (21) reduces to r0 …† ˆ  ˆ 0. The Wald statistics with b ˆ 0 and b ˆ 1 are ^ ˆ ^2 I ; ^ wr0 …†

^ ˆ …1 ÿ exp…†† ^ 2 I ^; wr1 …† 

…22†

and obviously differ, although they agree in a neighbourhood of H0 . The Wald test introduces an arbitrary measure of distance on the manifold (the curve …†), which is determined by the way the restrictions are formulated. This distance is not an intrinsic quantity, meaning that it depends on the parameterisation used. Shortcomings of the Wald statistics are well known; see, for example, Gregory and Veall (1985), Breusch and Schmidt (1988), Lafontaine and White (1986), Phillips and Park (1988) and Nelson and Savin (1990). They all show certain properties of the test, such as dependence on the formulation of the restrictions and the possibility of obtaining any

Curvature and classical tests

247

numerical value, problems with the size and the power function or approximations to them, but offer limited explanations. We provide an alternative view by considering the CR rather than the test statistic, which is quite revealing. We turn to the partition of the sample space into acceptance region and critical region that is induced by the one-sided version of the Wald test. First note that the test statistic is a function of the estimator ^ only, and that it determines a `distance' from ^ to 0 , using the distance measure wr …
†.10 If ^ is further than a particular wr -distance away from 0 we reject the null hypothesis, i.e. the test rejects when11 ^  c ; wr …†

…23†

where the subscript indicates the dependence on r, the formulation of the restrictions. The critical value c is generally based on the asymptotic chi^ under H0 and does not change with r, which square distribution of wr …† has important adverse consequences in ®nite samples as we shall see below. Assume for the moment that wr …
† is monotonically increasing so that ^ Let r be the critical value the CR can be expressed directly in terms of . ^ in terms of , which now does depend on r, precisely because c does not depend on r. The one-sided Wald test rejects for all those observations t 2 T for which ^  r : …t†

…24†

^ to make explicit that ^ is a function of t. There will We have written …t† be many different values of t that give the same value for ^ because the dimension of t is larger than the dimension of . The boundary of the CR is determined by all those t in the sample space that give r as the MLE, i.e. the boundary is the inverse image of r : ^ ˆ r g: boundary CRWald ˆ ^ÿ1 …r † ˆ ft 2 T j …t†

…25†

We can determine the inverse image of r by solving the ®rst-order conditions for the MLE, as a ®rst step. Following Efron (1978) we denote the set of solutions to the ®rst-order conditions (f.o.c.) by 10

The quotation marks are necessary because wr …
† is not necessarily a metric monotonically ^ I ^ depends on ^ and may decrease faster than, for instance, ^2 increasing in terms of :  increases in (22). Of course, if I^ is constant or monotonically increasing then wr …
† is strictly monotonically increasing. 11 The PO, LM and LR tests do not always reject if ^ is larger than a particular value.

248

Kees Jan van Garderen

ÿ ÿ   ÿ1…; t†ÿ ÿ £ ˆ t 2 T ÿÿ ˆ 0 : MLE constant (f.o.c.).  ÿˆ

…26†

Keeping  ®xed at  , we can solve for t such that (26) holds. Thus £ is the pre-image of  , plus all values of t satisfying the ®rst-order conditions only. In curved exponential models we have, using (2) and (5), ÿ ÿ   ÿ ÿ…† 0 ÿ ÿ ˆ 0 : MLE constant (f.o.c.). …t ÿ …††ÿ £  ˆ t 2 T ÿ  ˆ …27† Clearly t ˆ … † satis®es (27) and the complete solution consists of a hyperplane in the sample space through … † orthogonal to the gradients @… †=@i , i ˆ 1; . . . ; d. The hyperplane is thus of dimension …k ÿ d†. The straight part of the Wald test CR is illustrated in ®gure 8.4 for k ˆ 2, d ˆ 1 and critical value r . (The curved part of the boundary is discussed jointly with the geodesic test; see ®gure 8.6 in particular.) The boundary is orthogonal to the gradient of …† at r for the Wald test, versus 0 for the LMP test. Also note that the boundary of the LMP test stretches out in®nitely, whereas the boundary of the Wald test is a hyperplane only until it crosses another line of constant MLE. Points where two constant MLE lines cross and give more than one maximum likelihood solution were de®ned in van Garderen (1995) as critical points, since the MLE is not unique. Beyond such critical points the boundary is curved (see the discussion of the geodesic test below). Using a different formulation of the restriction, e.g. r1 …
† instead of r0 …
†, will result in a different value for r and a different CR. Figure 8.4 would  apply with the boundary shifted to coincide with ^ÿ1 …t† ˆ r1 . Similarly,   changing the critical value c changes r . In principle c a could, and should, be chosen such that the signi®cance level is exactly . In practice this is typically too dif®cult, and asymptotic theory is used to ®nd an approximation. The asymptotic critical value is the same regardless of which rb …
† is used. Note that the asymptotic critical value is based on r close to 0 , the value under the null. This gives a boundary parallel to the LR and LM statistic, and asymptotically the boundaries are equivalent, de®ning the same test statistic. For large r in a curved model, the boundary is at an angle with the asymptotic boundaries and de®nes a different statistic. The asymptotic critical value does not take this into account and size and power calculations are suspect in small samples. It was assumed in ®gure 8.4 that wr …
† was monotonically increasing. If the Fisher information I…† is decreasing faster than ‰r…†=R…†Š2 is increasing, then it is possible that wr …
† is not monotonic, as will be

Curvature and classical tests N: Parameter Space

249

T: Sample Space Acceptance Region

* o

MLE = * r

r

*r

CR: Reject H o

o

Figure 8.4 Critical region of the one-sided Wald test

demonstrated for testing against the null hypothesis of a unit root test in section 8. This means that wr …
† is not a proper distance measure and, as a consequence, there are potentially three (or more) values of ^i for which wr …r …1†† ˆ wr …r …2†† ˆ wr …r …3†† ˆ c :

…28†

The Wald test rejects in that case for values of ^ in two unconnected intervals r …1†  ^  r …2† [ ^  r …3†:

…29†

This means that for values of ^ between r …2† and r …3† we do not reject, even though they are much larger than r …1†, the smallest value for which the test rejects. Instead of the one line in ®gure 8.4, there are now three lines bounding the CR. This is shown in ®gure 8.5. Figure 8.5 shows that the Wald test can lead to very unattractive CRs. It also explains why the power function can be non-monotonic, as reported in Nelson and Savin (1990): the probability of rejection is larger for  ˆ r …2† than for  between r …2† and r …3†. The next section discusses a geometrical version of the Wald test and deals with some further issues not related to the non-invariance of the test. 6

A geodesic distance test

The Wald statistic is a non-geometric quantity which de®nes a `distance' not compatible with the model. Improvements based on differential geometrical arguments have been proposed by Critchley, Marriott and Salmon (1996) and Davidson (1992); see also Vñth (1985). We can de®ne

250

Kees Jan van Garderen

N: Parameter Space

T: Sample Space

*(1) o

Reject Ho CRI

Acceptance Region

*(2)

*(1) *(3)

o

)

*(2)

Acceptance Region *(3)

CRII

Figure 8.5 Disconnected critical region …CRI [ CRII † for non-monotonic Wald statistic

distances on the manifold in a geometric way by de®ning a metric, with coef®cients gij , and an af®ne connection, with coef®cients ÿm ij , and deter12;13 ^ mine the geodesic distance on the manifold from  to 0 . A geodesic on a manifold is the analogue of a straight line in Euclidean space and gives the shortest distance between two points on the manifold, e.g. ^ and …0 †, based on the distance de®ned ^ and …^0 † or …† between …† by the metric, and connection. This distance is the same regardless of the parameterisation. The Fisher information metric is commonly used for gij , together with the Riemannian (Levi±Civita) connection for ÿm ij . Rao (1945, 1949) was the ®rst to explore the idea of a geodesic de®nition of the distance between two distributions (see also Jeffreys (1939), however). We could use this distance for hypothesis testing, as in fact Critchley et al. (1996) do. They use the geodesic distance based on the Fisher information metric between ^ and 0 as test statistic v …1 u uXdX  …x† j …x† ^ 0 † ˆ min t gd…; dx; gij ……x†† i …x† 0 x x i;jˆ1

…30†

s:t: …0† ˆ ^ and …1† ˆ 0 …or …1† ˆ ^ and …0† ˆ 0 †

12 The null hypothesis is simple so that 0 is known. A composite H0 complicates the test even further. 13 ij If gij ˆ gij …† is the metric with inverse Pd 1 gml, then thej Christoffeli symbols for1 the Levi±Civita g ……@gil =@ † ‡ …@glj =@ † ÿ …@gij =@ ††, which is the connection are de®ned by ÿm ij ˆ lˆ1 2 unique Riemannian connection (the conventional superscript on @ denotes a coef®cient and is not a power). gij …† is often the Fisher information metric gij …† ˆ iij ˆ E‰@I=@i @I=@j Š.

Curvature and classical tests

251

where the minimum is over all possible curves …x† in  that connect ^ and 0 . The test rejects for large values of the geodesic distance ^ ˆ gd…; ^ 0 †  c ; gd…†

…31†

which is exactly like (23) for the Wald test, but now the test statistic is by de®nition strictly monotonic and it does not depend on the algebraic formulation of the restrictions. Consequently, the test rejects for all ^   , as in (24), but the critical value does not depend on r. The …t† boundary has the same shape as the (monotonic) Wald test in ®gure 8.4, if the correct critical values are used, and is again determined by the inverse image of  : ^ ˆ  g: boundary CRgeodesic ˆ ^ÿ1 … † ˆ ft 2 T j …t†

…32†

Geodesics have been determined for a limited number of distributions. Rao (1987) summarises a number of these geodesic distances in statistical manifolds. These can be used directly to de®ne a geodesic test, but in general they need to be worked out speci®cally. This poses a serious practical problem for geodesic tests because solving (30) in higher dimensions requires the solution of the following second-order non-linear differential equation d m X i j ˆ 0; ‡ ÿm ij 2 xx x i;jˆ1

2

m ˆ 1; . . . ; d;

…33†

where ÿm ij are the Christoffel symbols for the Levi±Civita connection. Equation (33), together with initial conditions …0† ˆ  and @=@xjxˆ0 ˆ v, de®nes a unique curve …x† that minimises the distance between …0† and other points on …x†. We do need to determine the initial direction v that will assure that …x† passes through the points  and 0 . In the general setting of the Wald test, estimation is carried out only under the (unrestricted) alternative, and hence 0 is unknown. Considering these problems, it seems more natural to use the geodesic distance to construct con®dence regions, since we can start from ^ and ^ These determine in each direction v the points in  of equal distance to . geodesic circles or orbits could be evaluated numerically, and for different distances correspond to different con®dence levels. Moolgavkar and Venzon (1987), for instance, construct Wald-type con®dence regions using a geodesic distance, although based on a different metric and connection. In general we can construct many different metrics and connections. In fact, Chentsov (1982) and Amari (1982) introduced a one-parameter

252

Kees Jan van Garderen MLE = 1

N: Parameter Space

T: Sample Space *

1

tc

Acceptance Region

*

o o

CR: Reject H o

2

MLE = 2

Figure 8.6 The geodesic test

family of so-called -connections, and different values of  are useful in different contexts.14 The Riemannian connection used in (30) and (31) corresponds to  ˆ 0, which may, or may not, be the most relevant for hypothesis testing. The geodesic test also poses a fundamental question, namely: should the shortest distance between two distributions be measured as the distance over the manifold, i.e. as curves in N, or, alternatively, as the shortest distance in the embedding space, i.e. straight lines in N?15 Figure 8.6 shows an extreme case where the manifold curves back to the origin. Regardless of how close, in terms of , we get to the origin, we will always reject if ^   . The test rejects only because it took a long time to ^ not because pdf …t; 0 † is very different from pdf …t; a †. In get to …†, economic applications this would mean that we reject a maintained model even though in economic terms there is hardly any difference between the null and the estimated model. This comparison between models is in contrast with the NPL, where it is the direct distance in the full exponential model that matters. It may therefore be the case that the problem lies, not with the geodesic test in principle, but with the metric and -connection usually chosen. The Fisher information metric, essentially de®ned on the manifold, may be useful in applications other than hypothesis testing, but the NPL and ®gure 8.6 suggest that 14

The -connections determine a non-Riemannian geometry and are de®ned as m …†ÿm ij ˆ ÿij ‡

d 1ÿX gml Tij1 ; 2 lˆ1

where Tij1 ˆ E‰@I=@i @I=@j @I=@1 Š, the skewness tensor. 15 Davidson (1992) raises the same question in his discussion of Critchley et al. (1996).

Curvature and classical tests

253

the geodesics in the embedding density give the relevant distances in hypotheses testing, not the geodesics on the manifold. Decreasing the size of the test does not affect the bottom part of the CR.  will increase, but this changes only the top right-hand side of the CR. Under H0 , the additional points so excluded are less likely than the points at the bottom, which is undesirable. Further note that the boundary of the CR in the top right can be orthogonal to the BCR in that direction. The boundary of the CR consists of two types of points: the inverse image of  , i.e. ^ÿ1 … †, which gives the straight part of the boundary, mainly on the outside of the ellipse; and those points on the inside, such as tc , that have two maximum likelihood solutions, one solution on the left and one solution on the right. Such points were called critical points in van Garderen (1995) and I investigated the sensitivity of the MLE as t ^ The same gets closer to critical points, based on the local curvature at . argument applies here, even though the critical points in ®gure 8.6 are global in nature. The sensitivity for global critical points is only more extreme. Let tc be a point on the boundary on the inside of the curve …†. A small change " in the observation, from tc ÿ "=2 to tc ‡ "=2 leads to a ^ from 1 to 2 . The test statistic suddenly appears highly huge increase in , signi®cant despite the fact that the likelihood of tc ÿ " is virtually the same as the likelihood of tc ‡ " under the null, as well as under the alternative.16 This raises another problem for a geodesic or Wald-type test. Since the test statistic is based on an estimator (whether this is the MLE or some other estimator), there are many points t in the sample space giving the same value of the estimator but lending very different support to the null. Observations on the inside of …† in ®gure 8.6 provide little information about the validity of H0 or Ha , because the likelihood values under the null and alternative will be very similar. In contrast, observations on the right of the curve …† giving rise to the same value of ^ are much more likely under Ha than under H0 and do provide strong evidence against the null. It is unreasonable to treat points on the inside and on the outside the same. This implies that the geodesic and Wald tests should be carried out conditional on the relative position of the observation to the manifold and critical point. The tests could, for example, be conditional on the relative distance auxiliary de®ned in van Garderen (1995). Implementation of such a conditional test is likely to be dif®cult, with little intuition

16 At tc the likelihood is exactly the same under H0 as under Ha , since this equality de®nes critical points.

254

Kees Jan van Garderen

why it should be better than the PO, LM or LR test. Furthermore, the traditional Wald computational advantage of having to estimate only under the alternative seems to have been lost in the process.

7

The likelihood ratio test

The PO, LMP and (one-sided) LM test are all determined by particular directions in the canonical parameter space and have ¯at CR boundaries. As a consequence they have optimal properties in the directions used to construct the test. The LR does not satisfy any such optimality criteria, but performs remarkably well in a range of diverse testing problems. It also tends to perform well against a range of alternatives, whereas the PO and LM test may break down against alternatives that differ from the directions used to construct the test. It is an interesting question why the LR test provides a reasonable solution in such a broad range of testing problems. Part of the explanation is that the boundary of the LR CR mirrors the global curvature of the model. If the model is ¯at, the CR boundary is ¯at, just like the PO and LM boundaries, but if the model is curved the LR boundary re¯ects this curvature. This relation is, however, not straightforward and is quite interesting in its own right. We will therefore spend some time discussing and explaining the relation between three different elements that determine the curvature of the CR. A further explanation for the performance of the LR test is that it implicitly averages over different alternatives, and we give an interpretation of the LR CR as a `weighted average of BCRs'. Because of the issues involved, this section is subdivided as follows. Section 7.1 determines the boundary of the LR test CR, described as a function of the parameter values in the alternative. Section 7.2 discusses how the three aspects referred to above in¯uence the relation between the global curvature of the model and the shape of the CR boundary. Section 7.3 further discusses some of the merits of the trade-off between global and local curvature that the LR makes. This prepares for the interpretation of the LR CR as a `weighted average of BCRs' given in section 7.4. Finally, in section 7.5, we discuss when the LR test may fail. 7.1

The boundary of the LR CR

The LR test rejects for large values of the LR statistic: LR ˆ 2 Sup l…; t† ÿ Sup l…; t†  c :  a

 o

…34†

Curvature and classical tests

255

The second maximisation in (34) is not required if the null hypothesis is simple (as assumed throughout this chapter) and the maximum becomes l…0 ; t†. If, moreover, there was only one point in the alternative then the test would be just like a PO test. If there are more points in the alternative, we ®rst have to maximise the likelihood under the alternative and ^ The LR test can therecalculate the likelihood ratio using this value of . fore be broken up into two steps: ^ (i) maximise the likelihood to obtain the MLE ^ ˆ …t†,   ^ (ii) for this value of  …i:e:  ˆ † calculate the likelihood ratio LR ;0 …t† ˆ 2 ‰l… ; t† ÿ l…0 ; t†Š and compare it with the critical value c . Hence, if t is a point in the sample space on the boundary of the CR, it satis®es two conditions:17 ^  † ˆ  …i† …t

and

…ii†

LR ;0 …t † ˆ c ;

…35†

where  is ®xed at the same value for (i) and (ii). Changing the perspective, we may ask, for each value of  2 , which t 's in the sample space lie on the boundary for that particular value of   . This requires solving for t from (i) and (ii) jointly, for ®xed  . Thus, we are going to determine all t in T satisfying (i) and (ii) as a function of  . We can determine all t that give  as the MLE, just as we did for the Wald and geodesic tests, by solving the ®rst-order conditions (f.o.c.) for the MLE (see (27)). …i†

… † 0 …t ÿ … †† ˆ 0 : MLE constant (f.o.c.); 

…36†

which, for a ®xed value of   , is a hyperplane through … † orthogonal to @=@jˆ . We have further seen from the PO test that the set of t values for which the likelihood ratio is constant is also a hyperplane, but orthogonal to ……  † ÿ 0 †. We have …ii† …… † ÿ 0 † 0 t ˆ 12 c ‡ … † ÿ …0 † : LR constant.

…37†

The `constant MLE' hyperplane is of dimension …k ÿ d† and the `constant LR' hyperplane is of dimension …k ÿ 1†, and they will intersect since the gradient @=@ at  has a direction different from the vector

17

The boundary of the LR CR can in principle also be determined by substitution of ^ and trying to solve LR ^ …t† ˆ 2‰l……t†; ^ t† ÿ l…0 ; t†Š ˆ c directly, but explicit for ˆ …t† ;0 ^ are often unavailable. mulas for …t†

256

Kees Jan van Garderen

T: Sample Space and N: Parameter Space

LR*= c

MLE = * MLE= **

*) o) o

**)

t*

and

t** LR** = c

Figure 8.7 The intersection of constant MLE and constant LR lines

…… † ÿ 0 †.18 The intersection forms part of the boundary of the LR critical region. It is a set in the k-dimensional sample space satisfying d ‡ 1 linear restrictions and is thus a …k ÿ d ÿ 1†-dimensional hyperplane in T. If d ˆ k ÿ 1, this intersection is just a point. The intersection is shown in ®gure 8.7, for k ˆ 2 and d ˆ 1, so (36) and (37) are straight lines intersecting (if at all) in a point. Two values  and  are shown. In ®gure 8.7 we have actually chosen a bivariate normal distribution for t, i.e. t  N……†; I2 †, because the canonical parameter in this case is equal to the expectation parameter …† ˆ …†  …†, and this allows us to depict the parameter and sample space together in one ®gure. To get a general interpretation, all arrows should be interpreted as vectors from the canonical parameter space, as in ®gure 8.1.

18 Different directions in the sense that ‰…… † ÿ 0 †Š cannot be written as a linear combination of the columns of @=@ at  . An exception is a linear exponential model when the gradient is a vector with the same direction as the difference vector for all  . In that case the LR test equals to PO and LM tests and is UMP with a straight boundary of the CR. A second exception is caused by the global curvature of the model. For example, if k ˆ 2, d ˆ 1 and …† curves back such that two  values give the same direction of ……† ÿ 0 †, then there will exist a  value in between such that the MLE constant line and LR constant line are parallel. No point from the PO tests based on the …† furthest from 0 is part of the boundary. Finally, the two lines may fail to cross because ^ÿ1 … † is only a half line, which does not stretch out beyond the critical point. Again, no point from the PO boundary based on this ^ is part of the LR boundary.

Curvature and classical tests

257 T: Sample Space and N: Parameter Space

o

Acceptance Region

CR: Reject H o

t*

t**

Figure 8.8 The boundary of the LR critical region

All points on the inverse image of  (the MLE ˆ  line) to the right of t have LR ;0 …t† > c and therefore lie in the CR. Similarly, all points to the right of t , giving  as the MLE, fall in the CR also. The boundary point t is determined by ®xing  and solving (36) and (37) for this particular value, and likewise t is determined for  . We can repeat this process for all parameter values, generating a curve in the sample space. This curve forms the boundary of the CR, which can thus be described as a function of  . The CR consists of all points in T to the right of this curve. This is shown in ®gure 8.8. 

7.2

Determinants of the shape and location for the LR CR boundary

The boundary of the LR CR clearly re¯ects the global curvature of the model, and it is instructive to examine this relationship in some detail. The shape and position of the boundary depend on three things: the signi®cance level (through c ), the distance from … † to 0 , and the angle between …… † ÿ 0 † and the gradient @=@ at  . First, it is useful to rewrite (36) and (37) by subtracting …@=@† 0 0 and  … ÿ 0 † 0 0 , respectively, to obtain … † 0 … † 0 …t ÿ 0 † ˆ …… † ÿ 0 †;  

…36 0 †

…ii† …… † ÿ 0 † 0 …t ÿ 0 † ˆ 12 c ‡KLI…0 ;  †;

…37 0 †

…i†

258

Kees Jan van Garderen 1

o

2

1

t*( 2) t*( 1)

2

o

t**( 1)

t**( 2)

Figure 8.9 The effect of decreasing the signi®cance level from 1 to 2

where KLI…0 ;  † is the Kullback±Leibler information (directed) distance from pdf …t; 0 † to pdf …t;  †.19 The boundary of the LR CR as a function of  is found by solving …36 0 † and …37 0 † simultaneously for each  2 . It is clear that the signi®cance level and the distance from … † to 0 , directly affect …37 0 † and together they determine the position of the constant LR hyperplane. The constant MLE hyperplane is also determined by  , via …… ††, as well as by the gradient @… †=@. The angle between …… † ÿ 0 † and the gradient determines where the two hyperplanes intersect. We shall now review their in¯uence on the position and the shape of the boundary more closely. 7.2.1

The signi®cance level The signi®cance level in¯uences the position of the constant LR line. Decreasing the size of the test increases c , which in turn shifts the constant LR lines further to the right, because a larger value of t is required for …37 0 † to hold. The shift is illustrated in ®gure 8.9. Figure 8.10 shows the critical regions for two different signi®cance levels. The boundary shifts away from 0 as  decreases and the CR is smaller (reducing the probability of rejecting H0 ). 7.2.2

The distance between the null and the alternative The distance between 0 and … † also determines the position of the constant LR line, as …37 0 † clearly shows. But, unlike increasing c , increasing  does not always shift LR away from 0 , because  enters on both sides of …37 0 †. Increasing  enlarges … ÿ 0 † on the left-hand 19

KLI…o ;  †  E0 ‰lnfL…0 †=L… †gŠ which, for exponential models, equals

KLI…0 ;  † ˆ …0 ÿ  † 0 E0 ‰tŠ ÿ ……0 † ÿ … †† ˆ …0 ÿ  † 0 0 ÿ ……0 † ÿ … ††:

…36 0 † relates to the local curvature of …† through the change in @=@, and to the global curvature of …† through …… † ÿ 0 †. …37 0 † involves the distance between the null and the … † alternative through the terms …… † ÿ 0 † and KLI…0 ;  †, and relates to the global behaviour of …†.

Curvature and classical tests CR(

259

1)

T: Sample Space and N: Parameter Space

CR(α2)

t*( 2)

ο

Acceptance Region

CR: Reject Ho

t*( 1) t**( 1

t**( 2)

Figure 8.10 The boundary of the LR test for two different signi®cance levels 1 > 2

side, pulling LR towards 0 , since t needs to be reduced for …37 0 † to hold. But increasing  also increases the Kullback±Leibler distance on the right-hand side, pushing LR away from 0 , since t needs to be increased for …37 0 † to hold. In order to be more speci®c we need to de®ne the length of … ÿ 0 † and the distance between 0 and LR . This can be done in different ways, but we shall simply use Euclidean distance, which suf®ces for present purposes.20 We de®ne the distance between 0 and LR as … † ˆ ktm ÿ 0 k, where tm is the intersection point of the line through 0 in the direction … ÿ 0 † and the constant LR hyperplane (see ®gure 8.12). Thus tm is the point on LR closest to 0 in Euclidean sense.21 Since …tm ÿ 0 † is a multiple of … ÿ †, we have … ÿ † 0 …tm ÿ 0 † ˆ k ÿ kktm ÿ 0 k, and using …37 0 † we obtain … †  ktm ÿ 0 k ˆ

c KLI…0 ;  † ‡ : 2k ÿ 0 k k ÿ 0 k

…38†

For the normal distribution with identity covariance matrix, as used in ®gure 8.7, we have …† ˆ …† ˆ …†, KLI…;  † ˆ 12 k ÿ  k2 , and (38) reduces to  … †  ktm ÿ 0 k ˆ

c 1 ‡ k ÿ 0 k: ÿ 0 k 2

2k

…39†

20 0 Pÿ1 It is generally more appropriate v in the sample P to use Mahalanobis distance kvk ˆ v space …v 2 T† and kwk ˆ w 0 w in the parameter space …w 2 N†. This would show, for instance, that the distance  between 0 and LR goes to 0 as the number of observations goes to in®nity. The current relations are valid for ®xed sample sizes and in ®xed directions. 21 But not necessarily in the Mahalanobis sense.

260

Kees Jan van Garderen

↑ 5

c =5

4 3

c =2

2 1 0

2.5

5.0

7.5

10.0

|| *

o

|| →

Figure 8.11 The distance of  ˆ  … † from 0 to the constant LR hyperplane

For distant  , the second term in (38) is dominating, and doubling the distance to  approximately doubles …
†. For  close to 0 , the ®rst term in (38) dominates, and now halving the distance approximately doubles the distance …
†. For distant alternatives and nearby ones, more extreme observations are required before the null is rejected. Figure 8.11 shows the distance …
† as a function of the length of … ÿ 0 †. Note that the minimum, and where the minimum is attained, increase with c . The critical value c is constant regardless of the … ÿ 0 † direction, with the result that, as  varies, the distances  from 0 to the CR vary, depending on the distance to the alternative. This turns out to be crucial in the implicit power trade-off between different alternatives that the LR test makes. This concludes the discussion of …37 0 †; the position of the constant LR line is determined by the signi®cance level and length of … ÿ 0 †. The distance from  to 0 also enters …36 0 †, because it determines   and 0 . Increasing  also increases   and, since   lies on the constant MLE hyperplane, shifts this hyperplane away from 0 . Finally, we consider the interaction between …37 0 † and …36 0 †, and what determines their intersection. 7.2.3

The angle between the difference vector and the gradient The angle between … ÿ 0 † and @=@jˆ determines the angle between the LR and the constant MLE hyperplane, and consequently where these planes intersect, i.e. the position of the CR boundary in the sample space. Note that the local behaviour of …† at  only enters in

Curvature and classical tests

261

LR* = c

tm

o

MLE = *

τ*

∝ ( *- o)

b a t* Figure 8.12 The angle between the gradient and the difference vector

…36 0 † and does not affect …37 0 †, which is essentially determined by the global behaviour. Figure 8.12 illustrates the situation. For ®xed  and c , the constant LR line is ®xed and the gradient determines the intersection with the constant MLE line, and therefore the point t on the boundary of the LR CR. The gradient @=@jˆ represents the local direction of the canonical manifold and … ÿ 0 † represents the average direction from the null to the alternative. The closer the direction of the gradient is to the average direction, the further the boundary point moves away from the expectation under the null and the point tm (both a ˆ kt ÿ 0 k and b ˆ kt ÿ tm k increase). The larger the value of b, the larger the proportion of the constant MLE line that falls in the CR. The two extreme cases are when the gradient is orthogonal to the average direction (b ˆ 0 and a ˆ ) and when the gradient is proportional to the average direction and the two lines are parallel; they either coincide or do not cross at all. 7.3 7.3.1

Discussion

The direct effect of decreasing the signi®cance level is to shift LR away from 0 , but there is also an effect on the shape of the boundary, because shifting out LR changes the trade-off between average (global) direction and local direction, but only indirectly through the effects given in subsections 7.3.2 and 7.3.3.

262

Kees Jan van Garderen

7.3.2

We shall now motivate ®gure 8.11 and justify the shifting of LR away from 0 for distant alternatives and alternatives close to H0 , since this is a crucial aspect of the trade-off between different alternatives implicit in the LR test. The increasing value of  for distant alternatives solves a classic problem in hypothesis testing illustrated by the following example. Suppose x  N…; 1† and we are testing  ˆ 0 versus  ˆ 10. If x ˆ 3 is observed, then the null hypothesis is rejected because of the size condition. This is quite unreasonable since the value of 3 is so much more unlikely under the alternative than under the null.22 What the LR test does implicitly is to adjust the critical value for the BCR if the alternative is far away (large power will be attained against distant alternatives regardless; there is little power loss and large size gain), in favour of alternatives closer to H0 (against which higher power is worth the additional size penalty). For distributions very similar to 0 it is also sensible to require observations to be further away from 0 before the null is rejected. This avoids including points in the CR that are of little use in discriminating between Ha and H0 , being almost as likely under the null as under the alternative. They contribute to the size of the test without contributing much in terms of power. 7.3.3

The angle between the gradient and the difference vector helps to explain why the boundary is ¯atter when the model is less curved. First consider a third type of extreme case (in addition to the directions being orthogonal or parallel), when the angle is, in fact, irrelevant, namely at the point on the expectation manifold where the constant LR line and constant MLE line cross. This is the point where the boundary of the CR crosses the expectation manifold. This point depends on the signi®cance level, however, and for a different  the angle does matter. For models close to linear exponential models, this crossing point, call it …x †, helps to show why the LR boundary is less curved. The boundary is a function of  and passes through …x † when  ˆ x , and as a consequence b ˆ 0 in ®gure 8.12. If the model is not very curved the constant MLE and LR lines are close to being parallel and, as  deviates 22

This criticism applies directly to the PO test. The PO test has a critical value that varies with  such that the size of the test is constant. This means that for a ®xed direction the distance to the boundary of the PO CR is constant regardless of the distance to the alternative.

Curvature and classical tests

263

from x , the value of b increases very quickly, despite  being fairly constant, resulting in a very ¯at boundary. If the model is exactly linear exponential, then there is only one value  for which the constant MLE and LR lines coincide and this line forms the straight boundary of the CR. If the local (Efron) curvature at  is large, then the gradient changes quickly relative to the average direction. The distance b changes quickly (depending of course on the distance of  relative to jjt ÿ tm jj) and the boundary is more curved as a result.

7.4

The LR as a `weighted sum of BCRs'

We are now able to give an interpretation of the LR test as a weighted sum of BCRs or PO tests in the following sense. For each possible parameter value  2 Ya , we construct the PO test fLR ;0 …t†  c g 2 T, in the direction  ˆ … † and take the intersection with the inverse image of the MLE  ; ^ÿ1 … † 2 T. The proportion of the inverse image ^ÿ1 … † that forms part of the CR depends on where the constant LR hyperplane using the same  , fLR ;0 …t† ˆ c g, intersects the inverse image. The location of the constant LR hyperplane depends on the signi®cance level  (®gure 8.9), and varies with the distance between the null and alternative (®gure 8.11). For distant (and close) alternatives the hyperplane is more distant from 0 . The …k ÿ d ÿ 1†-dimensional hyperplane of intersection between the inverse image and constant LR plane also depends on the angle between the gradient vector(s) of …† at  (i.e. the tangent space of …† at  ) and the average direction vector … ÿ 0 †. If the angle is small, the constant MLE and LR planes are more parallel and the intersection is further away from 0 , but this also means that a larger proportion of the inverse image of  is part of the CR (®gure 8.12). Each point in the alternative contributes one slice of a BCR, the intersection with the inverse image. The weights refer to the proportion of the slice that is included in the LR CR. An alternative interpretation is thus as a weighted average of constant MLE hyperplanes, but this is simply the other side of the same coin, since we always consider the intersection of the two sets. If the model is a one-parameter linear exponential then the whole BCR in the direction  is taken and the whole inverse image for all    crit .23 23

Thus the test statistic and the estimator determine the same partition of the sample space (de®ne the same orbits in the sample space on which they are constant).

264

Kees Jan van Garderen

In contrast with the LR tests, the PO and LM tests take the whole BCR in one direction, regardless of the value of the MLE, i.e. the PO test uses ^ The Wald and geodesic tests, on only LR ;0 …t† for one  regardless of . the other hand, take the whole inverse image of  , regardless of the value of LR ;0 …t†, as long as  maximises the likelihood and  is larger than crit . 7.5

When does the LR test fail?

The discussion so far gives the impression that the LR test is more than just appealing, but we should be cautious. In curved models the maximum power against all alternatives as an optimality criterion is vacuous since no such test exists. We did not de®ne an alternative optimality criterion, but gave a general appreciation of different tests instead. Optimality requirements are not necessarily desirable and can on occasion be viewed as simply restricting the class of tests to be considered. The LMP and PO tests, for example, uniquely satisfy optimality requirements but potential drawbacks are well known and have been exposed in this chapter. The LR test does not satisfy these optimality requirements, and is inadmissible on these criteria, but seems `reasonable' nevertheless. On the other hand, one would also expect the LR to perform badly in extreme cases, just like the PO, LM and Wald/geodesic tests. In particular, if the model is globally very curved, we can expect the power function of the LR test to differ substantially from the power envelope. The basic reason is that the power envelope is based on the PO test in each and every direction. When the global curvature of the model is large, these PO tests will de®ne very different CRs and the LR CR must differ substantially from the CRs of these tests (which give the power envelope), since all of them are of size . The extent to which they are different is measured by Hillier's (1990) global curvature measure ± the probability of the enveloping region (ER) under the null. If this probability is large, say 2, then the ER is much larger than the LR CR, which is always of size , and we may expect the LR power to be well below the power envelope. Hillier (1995) shows that the LR CR is a proper subset of the ER because the critical value used for the LR test, c in (34), is larger than or equal to the maximum of the critical values c … † in (9), for all PO tests of the same size, i.e. c  max 2Y c … †. Hence, the ER includes every BCR, as well as the LR CR. In fact, Hillier (1995) shows that the ER is the smallest region that contains all sensible CRs (in the sense that it is the smallest region that includes all proper Bayes procedures of size ). Put differently, in determining a solution to the non-existence of a UMP test, we need to consider

Curvature and classical tests

265

only subsets of ER24 The LR, LM and PO CRs are just particular choices. If Prob‰ERŠ ˆ , then ER ˆ CRLR ˆ CRLM ˆ CRPO , for all  , and the three testing principles give the unique optimal test. If Prob‰ERŠ > , no test exists that is UMP and a trade-off is necessary between power against different alternatives. As Prob[ER] gets larger, there is greater (undesirable) freedom to choose, and maximising power against one particular direction has more serious consequences for power against other alternatives. A natural counterpart to ER, which could be used to restrict the choice of CR, is the intersection of all BCRs of size , which we shall call the intersection region (IR). It seems reasonable to require of CRs that they include IR, since it consists of points in T that lie in all possible BCRs, and no BCR would exclude them. By de®nition every PO test includes IR, and the LM test, being a limiting case, essentially includes IR also.25 The LR test turns out to include IR in the examples below, but, although this might hold more broadly, it remains to be proved in general. Note that Prob[IR] could be used as a measure of global curvature also, but since IR can be empty if the model is too curved it provides no additional information on the global curvature beyond this cut-off point. In contrast, Prob[ER] will continue to increase with increasing global curvature and is therefore more useful. Concluding, we need to choose a CR that is a subset of ER and possibly includes IR. This could be based on a desire to optimise power against particular parameter values, which are of economic interest for instance, or to average power over a range of alternatives (both require additional information on the parameter values). The LR averages in an implicit fashion that may not be appropriate for the situation at hand, but in many cases the LR test will be a convenient choice.

8

Example: testing for autocorrelation and unit roots

This section illustrates the critical regions of different tests when testing for autocorrelation and unit roots in an AR(1) model. An illustration of 24

Note that the Wald/geodesic CR is not in general a subset of ER. Related to Roy's union-intersection method, we could think of two tests ± one based on ER and one on IR. The ER tests would be based on the ER of size  (Prob[ER] can be decreased by decreasing the size on which the BCR is based). This test would exclude part of IR in the examples below. The IR test would be based on the IR of size  (Prob[IR] might be increased by increasing the size on which the BCR is based, but if the model is very curved Prob[IR] might be zero for all signi®cance levels used). In the examples below, the IR test is not a subset of ER, i.e. it includes points that are excluded by all BCRs of size .

25

266

Kees Jan van Garderen

the problems of the Wald test, including the possibility of disconnected parts of the CR, is given at the end of this section. Consider the AR(1) model with ®xed initial value: yi ˆ yiÿ1 ‡ "i ;

"i  IIN…0;  2 †;

i ˆ 1; . . . ; T;

…40†

2

and y0 ®xed. Assuming  known reduces the model to a (2,1)-curved exponential model, and we set  2 ˆ 1.26 The canonical parameter  and minimal suf®cient statistics t, as in (2), are easily derived and are 1 0 T X 2 0 1 yiÿ1 C þ ! B 1 C B iˆ1 s ÿ 2 C B ˆB …† ˆ @ 2 A; t…y†  …41† C: C B T d A @X  yi yiÿ1 iˆ1

The expectation of t as a function of , i.e. the expectation manifold …†, equals 1 …† ˆ Ep ‰sŠ ˆ y20 ST …† ‡

1 …T ÿ ST …††; 1 ÿ 2

1 ˆ y20 T ‡ …T ÿ 1†T; 2

jj 6ˆ 1

…42a†

jj ˆ 1

…42b†

all  2 …† ˆ E ‰dŠ ˆ 1 …†; PTÿ1 2i 2T 2 where ST …†  iˆ0  ˆ …1 ÿ  †=…1 ÿ  †, when jj 6ˆ 1. The maximum likelihood estimator is ^ ˆ

d : s

…43†

…44†

It follows trivially from (44) that all d proportional to s by the same factor give the same value for the MLE. Consequently, the lines of constant MLE, (27) and (36), used in the Wald, geodesic and LR test are rays from the origin through …1; †. ^ ˆ 0g : MLE constant (f.o.c.). £^ ˆ fd ÿ s

…45†

^ i.e. ^ ÿ1 …†, ^ does not continue to the left of the The inverse image of , origin, since s is a sum of squares and is always larger than zero. The 26 The model with  2 unknown is a (3, 2)-curved exponential model and is analysed as such more extensively in van Garderen (1995). Note that  2 and  are orthogonal in the sense that the (expected, as well as observed) Fisher information matrix is diagonal, which tends to make the assumption of known  2 less restrictive.

Curvature and classical tests N: Parameter Space

T: Sample Space ↑

1

267

2

↑ 300 d,

2

200

MLE = 2/3 MLE = 1/2

100 1→

0 -0.50

100

0

-100

200

300

s,

1

-200 = -1)

-1 -300

Figure 8.13 Canonical and expectation manifolds for the AR(1) model

origin is the only critical point, because it is the only point in T for which the MLE is not uniquely de®ned. Note, however, that Prob‰…s; d† ˆ 0Š ˆ 0. The canonical and expectation manifolds are shown in ®gure 8.13, as ^ ˆ 1=2 and well as two lines of constant MLE, the inverse image of …t† 2/3, i.e. ^ ÿ1 …1=2† and ^ ÿ1 …2=3†. Thus all the points on each MLE line give the same value for the MLE and the critical point lies on both lines. The expected Fisher information on  is 1 and the observed information equals s. The different classical tests have the following general forms. 1. The PO test for testing H0 :  ˆ 0 against Ha :  ˆ  . 1 The test rejects for large values of … ÿ 0 † 0 t and the CR is CRPO… ;0 † ˆ fÿ 12 …… †2 ÿ 20 †s ‡ … ÿ 0 †d  c g;

…46†

1 regardless of whether  is larger or smaller than 0 . 2. The LM test for testing H0 :  ˆ 0 against one-sided alternatives. 1 When testing against Ha :  > 0 … < 0 †, the test rejects for large (small) values of …@…0 †=@† 0 t and the CR is CRLM ˆ fÿ0 s ‡ d  c g:

…47†

268

1 3. 1

1

4. 1

Kees Jan van Garderen

When testing against Ha :  < 0 , the inequality sign is reversed and a different critical value, obtained from the left tail of the distribution of @…0 †=@ 0 t, is used. The geodesic/Wald test. The standard version of the Wald test is analysed at the end of this section. The geodesic version does not require solving the partial differential equation (33), since the manifold is one-dimensional. Consequently, the geodesic test simply rejects if ^ >  , when Ha :  > 0 (or ^ <  when Ha :  < 0 ). This determines a CR above (or below) the inverse image of ^ ÿ1 … † and  is such that Prob‰^ >  Š ˆ : Note that the inverse image of ^ is a ray through the origin, and the boundary of the CR is a straight line as a result. As a consequence of this straight boundary there exists a PO test that coincides with this geodesic test. The intercept of the PO test changes with  and happens to pass through the origin for particular  's in each of the cases considered here. For this value of  , the PO and geodesic test coincide. Note, however, that  6ˆ  . The LR test for testing H0 :  ˆ 0 against one-sided alternatives. The points on the boundary of the CR are simultaneously on the constant MLE line and on the constant LR line. As a function of  and , they are de®ned by (see (45) and (46)) …i†

d ÿ  s ˆ 0; : MLE constant,

…48†

…ii† … ÿ 0 †d ÿ 12 …2 ÿ 20 †s ˆ 12 c : LR constant:

…49†

1 Solving (i) and (ii) simultaneously for s and d in terms of  and c , we obtain s… ; † ˆ c

1 ;  … ÿ 0 †2

d… ; † ˆ c

 : … ÿ 2 †2

…50†

1 Note that s… ; † and d… ; † do not depend on y0 , other than the value of c .27 The effect of y0 6ˆ 0 is that the expectation manifold shifts to the right. We now focus on testing for serially uncorrelated observations and testing for a unit root, thus specialising the general formulas given above to: 0 ˆ 0 against a > 0, i.e. positive autocorrelation (negative autocorrelation gives the mirror image), and secondly to: 0 ˆ 1 against 27

It is also interesting to note that, if 0 ˆ 0, the boundary points close to the origin are obtained for large values of  in (50).

Curvature and classical tests

269

stationary alternatives, a < 1, and explosive alternatives, a > 1. Critical values were obtained by simulation, using 100,000 replications for testing both serial independence and unit roots.28 To obtain critical values for the one-sided version of the LR test, the signed LR statistic was calcu^ ÿ 0 †LR , where LR ˆ 2……^ ÿ 0 †d ÿ 12 …^ 2 ÿ 20 †s†, lated:  LR ˆ sign… which is larger than 0 for all s and d.

8.1

Testing for serial correlation

Figure 8.14 shows the PO critical region for testing the null of no serial correlation,  ˆ 0, against positive autocorrelation,  > 0, i.e. a one-sided test, using four different values of the PO point. Comments on ®gure 8.14 1. …† is the expectation manifold, i.e. the expectation of …s; d† as a function of . For  ˆ 0 this equals the expectation under H0 , 0 , and for other values describes the expectation of …s; d† under Ha . 2. The tinted area is the intersection region IR of all BCRs considered here, i.e. the part of the sample space that falls in the CR of all the PO tests.29 It is a subset of Hillier's enveloping region ER, which, in addition, includes the `bow tie' shaped area. All the tests, including the LR and geodesic test, have their boundary inside this double conic section, which means that they all have the desirable properties that their CRs are proper subsets of ER, and they include IR. 3. The probabilities on ER and IR were calculated numerically for different nominal values of , and using the PO points 1, 0.75, 0.5 and 0.25 and the LM test. With  ˆ 10% we obtained for Hillier's global curvature measure ˆ0:1 ˆ Prob‰ERŠ ˆ 15:3%, and for the IR region Prob‰IRŠ ˆ 5:1%. This leaves a fair amount of (undesirable) freedom to choose a CR that includes IR and is a proper subset of ER. This freedom is due to the global curvature of the model and means that global curvature poses a notable problem. 28

For the no-serial correlation case a bivariate approximation is available for …s; d† using that under H0 : E‰sŠ ˆ T, Var‰sŠ ˆ 2T, E‰dŠ ˆ 0 and Cov‰s; dŠ ˆ 0. The critical values obtained from this approximation did not seem too far astray, but the union of all BCRs, i.e. the enveloping region, did not include the LR CR, which we know should be a proper subset of the enveloping region. 29   2 …0; 1Š. Stopping at  ˆ 1 might be sensible but is rather arbitrary. Increasing  results in a steeper boundary, smaller IR and larger ER. In the limit, the boundary becomes vertical, implying that the ER includes the CR of a test depending on s only, which may be considered undesirable.

270

30

Kees Jan van Garderen

↑ d

25

20

*=1

Reject Ho

* = 0.75 15 G/Wald & * = 0.5 10

* = 0.25 LR LM

5

Acceptance 0

Region

o

5

10

15

20

25

30

35

40

45

50

s -5

=E ( s ) d

Figure 8.14 Testing for serial correlation. H0 :  ˆ 0 vs. Ha :  > 0. T ˆ 25, y0 ˆ 0,  ˆ 10%. Lines correspond to the CR boundaries of four different PO tests using  ˆ 0:25; 0:5; 0:75; 1; the LM test (horizontal); the one-sided LR (dashed); and the G(eodesic)/Wald test (equal to PO ˆ0:5 ). 0 is the expectation of …s; d† 0 when  ˆ 0, i.e. under H0 , and …† (dashed) is the expectation manifold, i.e. the expectation of …s; d† 0 for values of  under the alternative. The grey area is the intersection region. Prob‰IRŠ ˆ 5%, Prob‰ERŠ ˆ 15%

4. Figure 8.14 also illustrates the LM test as a limiting PO test as  ! 0. The critical region in this case depends only on d, but note that the variance is assumed known … 2 ˆ 1†. 5. The geodesic/Wald test rejects if ^ > 0:25…ˆ  † and coincides with the PO test based on 0.5 (illustrating the obvious point made earlier that  6ˆ  ). 6. The LR boundary lies between the G/Wald and LM boundaries, for large values of s and d below the Wald and above the LM boundaries and vice versa for small s and d. The LR test is actually very close to the PO test with  ˆ 0:25, being slightly above it for central values of s and below it elsewhere. This value for the PO point is the average of the two PO points that generate the Wald test and the limiting value that generates the LM test, but this is coincidental.

Curvature and classical tests 500

271

T: Sample Space

↑

d

LM

τ(ρ)

LR

450

ρ* = 0.75 400

G/Wald

ρ* = 0.5 350

ρ* = 0.25

τo

300

ρ* = 0 250

200

150

100

Reject

H

o

(stationary alternatives

Ha : ρ < 1)

50

0 50

100

150

200

250

300

350

400

450

500

s→ Figure 8.15 Testing for a unit root. H0 :  ˆ 1 vs. Ha :  < 1. T ˆ 25, y0 ˆ 0,  ˆ 10%. Lines correspond to the CR boundaries of four different PO tests using  ˆ 0; 0:25; 0:5; 0:75; the LM test; the one-sided LR (dashed); and the G(eodesic)/Wald test. 0 is the expectation of …s; d† under H0 , and …† (dashed) is the expectation manifold. Prob‰IRŠ ˆ 2%, Prob‰ERŠ ˆ 19%

8.2

Testing for a unit root

Figure 8.15 shows the boundaries of different CRs for testing the null of a random walk,  ˆ 1, against stationary alternatives,  < 1. Figure 8.16 shows the different CR's boundaries for testing the random walk hypothesis versus explosive alternatives. Comments on ®gures 8.15 and 8.16 1. All the CRs in ®gures 8.15 and 8.16 are again proper subsets of the relevant ERs, and include the IRs. 2. The ®gures give the impression that the IR is smaller in the stationary case than in the explosive case, but this is not true in terms of their probability content, which shows, in fact, the reverse. Thus care needs to be taken when making direct comparisons of IR and ER in different

272

Kees Jan van Garderen T: Sample Space

ρ* = 2

G/Wald

ρ* = 1.25 500

↑

ρ* = 1.5

d

LM

450

Reject

400

H

o

(explosive alternatives

Ha: ρ > 1)

350

300

τo

250

200

150

100

:LR

50

τ(ρ)

:

=Eρ( sd)

0 50

100

150

200

250

300

350

400

450

s→

500

Figure 8.16 Testing for a unit root against explosive alternatives. H0 :  ˆ 1 vs. Ha :  > 1. T ˆ 25, y0 ˆ 0,  ˆ 10%. Lines correspond to the CR boundaries of three different PO tests using  ˆ 1:25, 1.5, 2; the LM test; the one-sided LR (dashed); and the G(eodesic)/Wald test. 0 is the expectation of …s; d† under H0 , and …† (dashed) is the expectation manifold. Prob‰IRŠ < 0:1%, Prob‰ERŠ ˆ 25%

testing problems. The IR and ER probabilities serve the purpose of allowing such a comparison. They were calculated for both cases, and can be compared with the serial correlation case discussed earlier. 1 For stationary alternatives we used the LM test and the PO tests based on  ˆ 0:75; 0:5; 0:25 and 0, making it comparable with the serial correlation case. With  ˆ 10% we obtained for Hillier's ˆ0:1 ˆ Prob‰ERŠ ˆ 19:2% and for Prob‰IRŠ ˆ 1:7%. We see that there is much greater freedom for choosing a CR, and global curvature poses a much more serious problem for unit root testing than for testing for serial correlation. For explosive alternatives we used the LM test and the PO tests based on  ˆ 1:25; 1:5 and 2. With

Curvature and classical tests

1

1 3.

4. 1

5.

273

 ˆ 10% we obtained ˆ0:1 ˆ Prob‰ERŠ ˆ 25:5%, and Prob‰IRŠ < 0:1%. It should be noted that  ˆ 2 is much further away from 0 ˆ 1 than  ˆ 0, in terms of, for instance, the Kullback±Leibler or Jeffreys distance.30 The point  ˆ 1:156 on the explosive side is the same Jeffreys distance away from 1 as  ˆ 0 on the stable side. Hence, we further calculated the probabilities using the LM test and the PO tests based on  ˆ 1:05 and  ˆ 1:156, and obtained ˆ0:1 ˆ Prob‰ERŠ ˆ 19:0% and Prob‰IRŠ ˆ 2:2%, which is very similar to the results for stationary alternatives. Irrespective of the question of which points should be included, global curvature poses a much more serious problem for testing unit roots than for testing serial correlation.31 The boundary for the LM test against stationary (one-sided) alternatives is of course parallel to the boundary for the LM test against explosive alternatives, since they are based on the same test statistic. The test against stationary alternatives uses the left tail, and the test against explosive alternatives uses the right tail of the distribution. The geodesic/Wald test rejects for ^ < 0:79, when testing against stationary alternatives, and rejects for ^ > 1:04, when testing against explosive alternatives. One explanation for this asymmetry round 1 is that the distribution of ^ is heavily skewed to the left. Another explanation, related to a point made earlier, is that the distribution of the canonical statistic …s; d† changes much quicker when  > 1 than for  < 1. Considering the mean of s and d, for instance,  needs to be increased by 1 to change expectation from (0,0) to (300,300), whereas an increase by less than 0.04 on the explosive side achieves the same change. Similarly, the Fisher information on , which equals …†, increases much faster for  > 1 than for  < 1. We are better able to distinguish the null distribution from distributions with  > 1 than for  < 1. The LR boundaries for the unit roots test case are much less curved than for the serial correlation case and are almost straight lines. The boundaries again lie between the Wald and LM boundaries, but are much closer to the LM boundary. In fact, it is dif®cult to distinguish between the LM and LR boundaries for large sections.

30 Pertaining to the discussion in section 7, Prob[ER] takes no account of the distance from null to alternative. 31 Van Garderen (1997b) calculates the exact Efron curvature for stable, unit root and explosive values of . The curvature increases dramatically as  approaches the explosive region.

274

Kees Jan van Garderen

8.3

The Wald test

We conclude this section by illustrating a number of problems with the Wald test, in particular when testing for a unit root against stationary alternatives. The Wald test depends on the formulation of the restrictions. We use the class of restrictions (22) and choose different values of b to make a comparison. r…† ˆ …expfb… ÿ 0 †g ÿ 1†=b; R…† ˆ @r…†=@ ˆ expfb… ÿ 0 †g:

…51† …52†

The information matrix …1  1† in this case is equal to 1 …† and the Wald statistic is ^ ˆ wr …†

  expfb…^ ÿ 0 †g ÿ 1 2 ^ 1 …†: b

…53†

The information matrix is increasing in  > 0 since 1 …† is increasing when  > 0. The Wald statistic wr …
† is therefore monotonically increasing when testing H0 :  ˆ 0 against Ha :  > 0 . By the same token, 1 …† is decreasing as  # 0 and the Wald statistic need not be monotonic. This non-monotonicity is shown in ®gure 8.17, when testing for a unit root Ha :  ˆ 1 against Ha :  < 1. The three curves correspond to three different values of b. We know from before that the geodesic version of the Wald test with the correct size, rejects if ^ < 0:79 (at 10% signi®cance level) and we use this value for comparison. Figure 8.17 also illustrates that we can obtain any value for the test statistic we like by varying the choice of b. For ^ ˆ 0:79, the test statistics take the values 0.66, 1.01 and 1.59 for b ˆ 7:5, 5 and 2.5 respectively. This implies that using the same critical value for all three values of b can lead to very different conclusions and is clearly inappropriate (as pointed out by Breusch and Schmidt (1985) and Lafontaine and White (1986)). Instead of using a common critical value, we could choose a critical value such that the value of  closest to 1, for which the tests reject, is 0.79. This is also illustrated in ®gure 8.17. For b ˆ 2:5 this actually gives a size-adjusted monotonic version of the Wald test, and coincides therefore with the geodesic test. This is not the case for b ˆ 5 or 7.5, which illustrate the situation where the test determines two unconnected parts of the CR. Taking b ˆ 5 for example, the test rejects for w5 > 1:01, which corresponds to rejecting if ^ < ÿ0:22

or

0:35 < ^ < 0:79;

…54†

Curvature and classical tests b=5

2

↑ w( )

275 b = 2.5

1.59

b = 7.5 1.01 0.66

0

-0.22

-1

0.35

0

0.79

1

Figure 8.17 The Wald test for three values of b …2:5, 5 and 7). The test with b ˆ 5 and critical value 1.01 rejects if ^ ˆ 0:79

300

200

τρ=1

↑ d

ρ^ = 0.79

Acceptance Region

Critical Region I

^ ρ = 0.35

100

s→

Acceptance Region 0

150

τ(ρ)

-100

200

250

300

350

ρ^ = −0.22 Critical RegionII

Figure 8.18 Disconnected critical region of the Wald test for a unit root based on equation (53) with b ˆ 5. ˆ1 is the expectation of …s; d† under H0 . T ˆ 25, y0 ˆ 0

but does not reject for values of ^ between 70.22 and 0.35. The resulting CR is shown in ®gure 8.18 and is very unattractive indeed. 9

Conclusion

In this chapter we have presented a review of classical tests very different from traditional treatments. Traditional approaches are concerned with explaining and justifying the choice of test statistics and deriving their properties. From this viewpoint questions about correct critical values

276

Kees Jan van Garderen

and appropriate approximations to their distributions are important. Whether we can easily ®nd the correct critical value is also important from a practical point of view but is irrelevant in answering questions about the `optimal' or desirable properties of a test. Optimality considerations apply whether the test is carried out at a 1% or a 10% signi®cance level. The LM, Wald and LR tests are asymptotically equivalent and this gives the impression that choosing a test is merely a question of which test is most convenient. Being aware of the approximate nature of this result, effort is often put into making sure that particular choices are appropriate in particular situations, and determining the right ®nite sample critical values and power functions or higher-order corrections to improve results for ®nite samples more generally. Asymptotic considerations can also go astray because of their local nature. A sequence of local alternatives, for instance, cannot reveal the power properties for alternatives quite different from the null. It may be thought that distinguishing the null from local alternatives is harder than for alternatives further away, and hence that only local properties matter asymptotically. The ®gures presented here illustrate that this can be very misleading because the classical tests behave very differently for distant alternatives. This chapter shows that the choice of test (statistic) is a matter of convenience only if the model is not seriously curved. If the model is appreciably curved then the different tests imply very different critical regions. The PO test and LM test give a straight boundary of the CR and cannot guarantee global consistency. The PO test further depends on the parameterisation of the model and, because of the size condition, ignores the distance to  , which can lead to (inappropriate) high power against (less relevant) distant alternatives. Choosing the same direction regardless of the sample size gives an inferior test in large samples also. The LM test is asymptotically superior to the PO test in most cases and is invariant under reparameterisations. The LM test can be seen, however, as an extreme case of a PO test and the power of the LM test is more easily affected by curvature of the model and is likely to have inferior power properties to the PO test in small samples, especially if the local curvature at H0 is large. The Wald test gives a curved boundary but the numerical value of the test statistic depends on the algebraic formulation of the restrictions. The geometric version of the Wald test, the geodesic test, raises a number of practical problems as well as a number of fundamental questions that apply directly to the Wald test also. The geodesic distance on the mani-

Curvature and classical tests

277

fold is arguably not the right de®nition of distance for hypothesis testing and the geodesic distance in the larger embedding manifold provides a better idea of departure from the null. The fact that the geodesic and Wald test is solely a function of the estimator means that the boundary always consists of the inverse image of the estimator at the critical value, ^ÿ1 … † in the sample space. This raises two problems: ®rst, there are ^ ˆ  , which lends very different support many different t such that …t† to the (non-)validity of H0 so that the test should be carried out conditionally; secondly, the boundary of the CR can be orthogonal to the BCR based on the direction from …0 † to … †, since this (average) direction ÿ1  can be orthogonal to the gradient of …† at  on which ^mle … † is based, if the model is very curved globally. Furthermore, the geodesic/Wald CR is not in general a proper subset of the ER and does not always include the IR. All these problems are a direct consequence of the curvature of the model. No test exists that is best in every direction. Choosing one particular direction, like the PO and LM tests, gives maximum power in that direction, but no guarantee of reasonable properties in different directions. One ®rst needs to investigate the curvature properties of the model. Ideally, we would like a test with a curved boundary that takes into account the curvature of the model and gives reasonable power in all possible directions. One solution would be to take the intersection of all BCRs against all possible alternatives such that the overall size is , i.e. a weighted version of all possible PO tests. The problem is how to choose the weights or critical values associated with each parameter point. If all the signi®cance levels are equal, then this is the enveloping region of Hillier (1990) adjusted for size and is related to Roy's union-intersection method, but this CR does not generally include the IR. We could also specify a weighting function re¯ecting the importance of different parameter values, e.g. based on external or prior information on the parameter values, since the ER ignores the distance of different alternatives from the null. Specifying a weighting function objectively and determining appropriate critical values raise practical as well as ideological problems. The LR test solves many of these problems simultaneously. The LR test uses the BCR in the maximum likelihood direction and a critical value that takes into account the distance from the alternatives on the manifold to the null, penalising alternatives that are too close and too far from the null to contribute reasonable power. It takes only one slice from each BCR boundary, namely the intersection with the inverse image of the MLE that gave rise to the maximum likelihood direction on which the

278

Kees Jan van Garderen

BCR was based. This determines a curved boundary that trades off the local curvature against the average global curvature at every alternative. The LR strikes a convenient compromise in terms of power against alternatives that are too close or too distant from the null, and alternatives that are a moderate distance away, against which reasonable power is desirable and can be attained. The overall conclusion based on this presentation must be that the Wald test should be avoided, not only because of its invariance properties, but also because of the deeper questions relating to the geodesic version; the LM and PO tests can be used if the model is not too curved, preferring a PO if the model is curved locally near the null hypothesis; and, ®nally, the LR test can be used in any case, but when the model is seriously curved then it appears clearly favourite since it implicitly (and automatically) takes the curvature of the model into account.

References Amari, S. (1982), `Differential Geometry of Curved Exponential Familes ± Curvatures and Information Loss', Annals of Statistics, 10: 357±385. (1983), `Comparisons of Asymptotically Ef®cient Tests in Terms of Geometry of Statistical Structures', Bulletin of the International Statistical Institute, Proceedings 44 Session, Book, 2, 1.190±1.206. (1985), Differential-Geometrical Methods in Statistics, Lecture Notes in Statistics No. 28, Heidelberg: Springer. Barndorff-Nielsen, O.E. (1978), Information and Exponential Families in Statistical Theory, New York: John Wiley. Barndorff-Nielsen, O.E., D.R. Cox and N. Reid (1986), `The Role of Differential Geometry in Statistical Theory', International Statistical Review, 54: 83±96. Berenblutt, I.I. and G.I. Webb (1973), `A New Test for Autocorrelated Errors in the Linear Regression Model', Journal of the Royal Statistical Society, Series B, 35: 33±50. Borges, R. and J. Pfanzagl (1963), `A Characterization of the One Parameter Exponential Family of Distributions by Monotonicity of Likelihood Ratios', Zeitschrift der Wahrscheinlichkeitstheorie, 2: 111±117. Breusch, T.S. and A.R. Pagan (1980), `The Lagrange Multiplier Test and Its Applications to Model Speci®cation in Econometrics', Review of Economic Studies, 47: 239±253. Breusch, T.S. and P. Schmidt (1988), `Alternative Forms of the Wald Test: How Long Is a Piece of String', Communications in Statistics, 17: 2789±2795. Brooks, R. (1993), `Alternative Point Optimal Tests for Regression Coef®cient Stability', Journal of Econometrics, 57: 365±376. Buse, A. (1982), `The Likelihood Ratio, Wald, and Lagrange Multiplier Tests ± An Expository Note', American Statistician, 36: 153±157.

Curvature and classical tests

279

Chentsov, N.N. (1982), Statistical Decision Rules and Optimal Inference, Translation of Mathematical Monographs vol. 53, Providence, Rhode Island: American Mathematical Society: ®rst published in Russian, 1972. Cox, D.R. and D.V. Hinkley (1974), Theoretical Statistics, London: Chapman & Hall. Critchley, F., P. Marriott and M. Salmon (1996), `On the Differential Geometry of the Wald Test with Nonlinear Restrictions', Econometrica, 64: 1213±1222. Davidson, R. (1992), `The Geometry of the Wald Test', mimeo, Queen's University. Davies, R.B. (1969), `Beta-optimal Tests and an Application to the Summary Evaluation of Experiments', Journal of the Royal Statistical Society, Series B, 31: 524±538. (1980), `Algorithm AS155: The Distribution of a Linear Combination of 2 Random Variables', Applied Statistics, 29: 323±333. Efron, B. (1975), `De®ning the Curvature of Statistical Problem (with Applications to Second Order Ef®ciency) (with Discussion)', Annals of Statistics, 3: 1189±1242. (1978), `The Geometry of Exponential Families', Annals of Statistics, 6: 362± 376. Engle, R.F. (1984), `Wald, Likelihood Ratio and Lagrange Multiplier Tests in Econometrics', in Z. Griliches and M.D. Intriligator (eds.), Handbook of Econometrics, vol. II, Amsterdam: North-Holland, Ch. 13. Evans, M.A. and M.L. King (1985), `A Point Optimal Test for Heteroskedastic Disturbances, Journal of Econometrics, 27: 163±178. Garderen, K.J. van (1995), `Variance In¯ation in Curved Exponential Models', Discussion Paper in Economics and Econometrics 9521, University of Southampton. (1997a), `Curved Exponential Models in Econometrics', Econometric Theory, 13: 771±790. (1997b), `Exact Geometry of Explosive Autoregressive Models', CORE Discussion Paper 9768. Godfrey, L.G. (1988), Misspeci®cation Tests in Econometrics, Cambridge: Cambridge University Press. Gregory, A. and M. Veall (1985), `On Formulating Wald Tests of Nonlinear Restrictions', Econometrica, 53: 1465±1468. Hillier, G.H. (1990), `On the Variability of Best Critical Regions and the Curvature of Exponential Models', Working Paper, Department of Econometrics, Monash University, Melbourne. (1995), `The Union of Best Critical Regions: Complete Classes, p-Values, and Model Curvature', Discussion Paper in Economics and Econometrics, University of Southampton. Imhof, P.J. (1961), `Computing the Distribution of Quadratic Forms in Normal Variables', Biometrika, 48: 419±426. Jeffreys, H. (1939), Theory of Probability, Oxford: Clarendon Press.

280

Kees Jan van Garderen

Kadiyala, K.R. (1970), `Testing for the Independence of Regression Disturbances', Econometrica, 38: 97±117. King, M.L. (1987), `Towards a Theory of Point Optimal Testing', Econometric Reviews, 6: 169±218. King, M.L. and P.X. Wu (1994), `One-sided Hypothesis Testing in Econometrics: A Survey', Working Paper, Department of Econometrics, Monash University, Melbourne. Kumon, M. and S. Amari (1983), `Geometrical Theory of Higher-Order Asymptotics of Test, Interval Estimator and Conditional Inference', Proceedings of the Royal Society of London, Series A, 387: 429±458. Lafontaine, F. and K.J. White (1986), `Obtaining Any Wald Statistic You Want', Economics Letters, 21: 35±40. Lehmann, E.L. (1986), Testing Statistical Hypotheses, 2nd edn, New York: Wiley. Moolgavkar, S.H. and D.J. Venzon (1987), `Con®dence Regions in Curved Exponential Families: Application to Matched Case-control and Survival Studies with General Relative Risk Function', Annals of Statistics, 15: 346±359. Nelson, F.D. and E. Savin (1990), `On the Dangers of Extrapolating Asymptotic Local Power', Econometrica, 58: 977±981. Okamoto, I., S.-I. Amari and K. Takeuchi (1991), `Asymptotic Theory of Sequential Estimation: Differential Geometrical Approach', Annals of Statistics, 19: 961±981. Pfanzagl, J. (1968), `A Characterization of the One Parameter Exponential Family by the Existence of Uniformly Most Powerful Tests', SankhyaÄ, Series A, 30: 147±156. Phillips, P.C.B. and J.Y. Park (1988), `On the Formulation of Wald Tests of Nonlinear Restrictions', Econometrica, 56: 1065±1083. Rao, C.R. (1945), `Information and Accuracy Attainable in the Estimation of Statistical Parameters', Bulletin of the Calcutta Mathematical Society, 37: 81± 91. (1949), `On the Distance between Two Populations', SankhyaÄ, 9: 246±248. (1987), `Differential Metrics in Probability Spaces', in S.I. Amari, O.E. Barndorff-Nielsen, R.E. Kass, S.L. Lauritzen and C.R. Rao (eds.), Differential Geometry in Statistical Inference, Hayward, Calif.: Institute of Mathematical Statistics, pp. 217±240. Saikkonen, P. and R. Luukkonen (1993), `Point Optimal Tests for Testing the Order of Differencing in ARIMA Models', Econometric Theory, 9: 343±362. Silvapulle, P. and M.L. King (1993), `Non-nested Testing for Autocorrelation in the Linear Regression Model', Journal of Econometrics, 58: 295±314. Vñth, M. (1985), `On the Use of Wald's Tests in Exponential Families', International Statistical Review, 53: 199±214.

9

Testing for unit roots in AR and MA models Thomas J. Rothenberg

In parametric statistical models for weakly dependent data, maximum likelihood estimates are typically root-n consistent and, when standardised, have limiting normal distributions. Likelihood ratio, Wald, and score statistics for testing equality constraints on the parameters have the same limiting distribution under the null hypothesis and lead to tests with the same asymptotic power function for local alternatives. Employing higher-order asymptotic expansions of the distribution functions, one can explain the small-sample departures from the asymptotic results in terms of the statistical curvature of the likelihood function. There exist, however, statistical problems of practical importance where this standard asymptotic theory does not apply. In this chapter, we look at three examples arising in non-stationary time-series analysis: testing for a unit root in a ®rst-order autoregressive model and in two variants of a ®rst-order moving-average model. These three examples have very different likelihood functions, yet have many features in common. In all three cases, power functions resulting from ®rst-order asymptotic analysis are remarkably close to those obtained from second-order asymptotic analysis in the standard case. The examples analysed here involve one-sided, one-parameter tests in models where there are no nuisance parameters. That is, we assume that the distribution of the data depends on a single unknown parameter  and consider tests of the null hypothesis  ˆ  against the composite alternative  < . Since there are no nuisance parameters, the null distribution of any test statistic is known; exact critical values can always be computed by simulation. When there are nuisance parameters not speci®ed under the null hypothesis, ®nding exact (or even approximate) critical values is non-trivial. This added complication is ignored here in order to concentrate on the key issues concerning power. 281

282

Thomas J. Rothenberg

1

The `standard' case

Suppose we have n observations generated according to a probability law that depends on an unknown scalar parameter . To test the null hypothesis that  ˆ  against the alternative that  < , rejection regions of the form Tn < t are commonly employed, where Tn is a test statistic depending on the data and ; t is a constant critical value such that, under the null hypothesis, Pr‰Tn < tŠ is equal to the predetermined size of the test. Tests of the same size can be compared by their power: the probability that Tn < t when the null hypothesis is false. We will denote the power of the test based on Tn when the true parameter is  by T …†; the size of the test is  ˆ T …†: For arbitrary point s <  in the parameter space, let Ln …s† denote the log-likelihood of the sample. By the Neyman±Pearson lemma, the best test of the null hypothesis  ˆ  against the point alternative  ˆ s rejects when the log-likelihood ratio Ln …† ÿ L…s† is small. The power of this size NP(s) test when the true parameter is in fact  will be denoted by NP …; s†. The function  …†  NP …; † is called the power envelope since the power function of the size- NP(s) test is tangent to  at the point s. The Neyman±Pearson lemma asserts that  …) is an upper bound to T …† for all tests of size . In certain exponential models, it turns out that the power function for the NP(s) test is independent of s; there exists a uniformly most powerful test with power function identically equal to the bound. In most testing problems, even in the one-sided, one-parameter case treated here, the Neyman±Pearson bound is not attainable in ®nite samples. But it is often attainable asymptotically. Since only alternatives near the true parameter value are of interest in large samples, it is convenient to reparameterise by writing the true parameter value as  ÿ nÿ1=2 ÿ and a typical point in the parameter space as  ÿ nÿ1=2 c. We will view ÿ as the parameter of interest and, when developing local asymptotic approximations, hold ÿ and c ®xed as n tends to in®nity. Assuming Ln is three times differentiable, we de®ne the standardised score Sn …s†  nÿ1=2 @Ln …s†=@s, the average Hessian Hn …s†  nÿ1 @2 Ln …s†=@s2 , and the average third derivative Rn …s†  nÿ1 @3 Ln …s†=@s3 : By Taylor series we have Ln …† ÿ Ln … ÿ n1=2 c† ˆ Sn …† ÿ 12 Hn …†c ‡ nÿ1=2 Rn …s †c2 ; c

…1†

where s lies between  and  ÿ nÿ1=2 c. In standard cases, Sn …† converges in distribution to a normal random variable; Hn …† and Rn …s † converge in probability to constants. Thus we ®nd that, to a ®rst order of approx-

Testing for unit roots

283

imation, the log-likelihood ratio behaves like a linear function of the asymptotically normal suf®cient statistic Sn …†: For any c > 0, the rejection region based on the likelihood ratio is asymptotically equivalent to rejecting for small values of the score. As n tends to in®nity, the power functions tend to …t ‡ ÿ†, where  is the standard normal distribution function, t is its -quantile, and  2 ˆ Var‰Sn …†Š is the expected information for  (evaluated at ). The local asymptotic power functions are independent of c and are identical to the asymptotic power envelope. To a second order of approximation, the log-likelihood ratio is a linear combination of Sn …† and Hn …†. The loss in power from using only the score depends on the amount of additional information contained in the Hessian. Efron (1975) suggested as a measure of this information the curvature Cov2 ‰Hn ; Sn Š Var‰Hn Š ÿ Var‰Sn Š Cˆn ; …2† Var2 ‰Sn Š where the score and Hessian are evaluating at the true . Typically, Hn and Sn are asymptotically bivariate normal, so the numerator of C is approximately the conditional variance of Hn given Sn . Using Edgeworth expansions of their distribution functions, Pfanzagl and Wefelmeyer (1978) develop asymptotic expansions for the local power functions of various likelihood-based test statistics. The exposition here follows that of Cavanagh (1983) and Pfanzagl (1980). Under regularity conditions, the power envelope for tests of size  has the asymptotic expansion  … ÿ nÿ1=2 ÿ† ˆ ‰t ‡ ÿ ‡ nÿ1=2 P…ÿ† ‡ nÿ1 Q…ÿ†Š ‡ o…nÿ1 †; where P and Q are low-order polynomials depending on the particular likelihood function. The local power function for a typical admissible test has the asymptotic expansion T … ÿ nÿ1=2 ÿ† ˆ  t ‡ ÿ ‡ nÿ1=2 P…ÿ† ‡ nÿ1 Q…ÿ†

 C 2 …3† ÿ …2ÿ T t ‡ ÿ† ‡ o…nÿ1 †; 8n where ÿT is a constant depending on the particular test. For example, the score test has ÿT = 0; its power curve is tangent to the power envelope at the origin where power is equal to the size : The Wald test based on the MLE has ÿT ˆ 1; its power curve is tangent to the power envelope at the point where power is approximately 1 ÿ . The maximised likelihood ratio test has ÿT ˆ 12; its power curve is tangent to the power envelope at the point where power is approximately one-half.

284

Thomas J. Rothenberg

These power formulae have a simple interpretation in terms of power plots on normal paper. That is, suppose we graph ÿ1 (power) against ÿ. In very large samples one should get a straight line with slope . In small samples one will get some curvature from the polynomial terms. The curvature due to P and Q is inherent in the problem and is independent of the particular test statistic. It is captured in the power envelope. The additional curvature due to the particular test statistic is very simple. It is a pure quadratic, tangent to the power envelope, with coef®cient equal to Efron's curvature measure. Although quite elegant, this second-order asymptotic analysis need not be terribly accurate in small samples. Luckily, it can be dispensed with. In our one-parameter problem, the critical value t and the power functions T …† can always be computed to any degree of accuracy by simulation. Indeed, that is the approach we will use in the non-standard unit-root cases where asymptotic analysis does not lead to analytically convenient approximations to the power functions. 2

The AR(1) model

Inference in autoregressive models with a near-unit root has been studied extensively, following the seminal paper by Dickey and Fuller (1979). Stock (1995) and Banerjee et al. (1993) survey some of the literature. The approach employed here is developed in greater detail in Elliott, Rothenberg and Stock (1996) and Rothenberg and Stock (1997). Suppose an observed time-series y1 ; . . . ; yn is generated by the ®rstorder autoregression yt ˆ ytÿ1 ‡ "t

t ˆ 1; . . . ; n;

…4†

where y0 ˆ 0,  is an unknown parameter, and the "'s are unobserved i.i.d. normal errors with mean zero and unit variance. We are interested in testing the null hypothesis that  ˆ 1 against the alternative that  < 1. Except for an additive constant, the log-likelihood function evaluated at the point s is given by n n 1X 1X Ln …s† ˆ ÿ …yt ÿ ytÿ1 †2 ˆ ÿ ‰4yt ‡ …1 ÿ s†ytÿ1 Š2 2 tˆ1 2 tÿ1 " n n X 1 X ˆÿ …4yt †2 ‡ 2…1 ÿ s† ytÿ1 4yt 2 tÿ1 tˆ1 # n X 2 2 ‡ …1 ÿ s† …ytÿ1 † : tˆ1

…5†

Testing for unit roots

285

P P Note that Ln …s† is quadratic in s and that … ytÿ1 4yt ; …ytÿ1 †2 † is a pair of suf®cient statistics. Again it will be convenient to rewrite the model in terms of a local parameterisation around the null hypothesis value  ˆ 1. Writing the true parameter as  ˆ 1 ÿ nÿ1 ÿ and a typical point in the parameter space as s ˆ 1 ÿ nÿ1 c, we ®nd  1 1 L …1† ÿ Ln …1 ÿ nÿ1 c† ˆ An ‡ cBn c n 2

…6†

where An 

n 1X y 4y ; n tˆ1 tÿ1 t

Bn ˆ

n 1X …ytÿ1 †2 : n2 tˆ1

The log-likelihood ratio (6) looks very much like (1) with the remainder term set to zero. But there are three major differences: 1. To obtain ®nite stochastic limits, the score term An must be scaled by nÿ1 and the Hessian term Bn must be scaled by nÿ2 ; hence the reparameterisation in terms of c involves a scaling by n instead of n1=2 . 2. Even after rescaling by nÿ2 , Bn does not converge to a constant under the null hypothesis; the limiting likelihood ratio depends on the pair of suf®cient statistics An and Bn . 3. The joint limiting distribution of An and Bn is not normal. The actual distribution function does not have a simple analytic representation, but the statistics themselves can be expressed as stochastic integrals of normal processes. Despite these differences, the Neyman±Pearson theory discussed in section 1 is still applicable. Each member of the family of test statistics de®ned by (6) is admissible, with power functions tangent to the power envelope. It is only the asymptotic approximations that have to be modi®ed. To describe the limiting behaviour of An and Bn , we note that, when ÿ ˆ n…1 ÿ † is ®xed and n is large, the discrete time-series nÿ1=2 yt can be well approximated by a continuous-time process. Let W…r† represent standard Brownian motion de®ned on [0,1] and let …s Jÿ …s† ˆ eÿÿ…sÿr† dW…r† …7† 0

be the Ornstein±Uhlenbeck process that satis®es the stochastic differential equation dJÿ …s† ˆ ÿÿJÿ …s†ds ‡ dW…s† with initial condition Jÿ …0† ˆ 0. For s in the unit interval, let ‰snŠ be the largest integer less than sn. Then, for parameter values local to unity such that ÿ ˆ n…1 ÿ † remains constant as n tends to in®nity, nÿ1=2 y‰snŠ ) Jÿ …s† and

286

Thomas J. Rothenberg

Figure 9.1 Asymptotic power functions for unit AR root (5% level; normal quantile scale)

 …An ; Bn † )

1 2 2‰Jÿ …1†

ÿ 1Š;

…1 0

Jÿ2 …s† ds



where ) represent weak convergence (see Chan and Wei (1987) and Stock (1995)). Thus, the power functions for our family of NP tests have the form   …1 NP … ÿ nÿ1 ÿ;  ÿ nÿ1 c† ˆ Pr Jÿ2 …1† ‡ c Jÿ2 …s†ds < t…c† ‡ o…1†; 0

where t…c† is the appropriate critical value. Unfortunately, there is no simple analytic expression for the joint dis„ tribution function of the pair …Jÿ …1†; Jÿ2 †. In practice, it is simplest to compute the power function by simulating the joint distribution of …An ; Bn † by Monte Carlo draws from the distribution of the "'s. Power functions for the NP(c) test statistics (6) were obtained from 10,000 replications of samples of size 100. The results are plotted in ®gure 9.1 for c ˆ 0 (the score test) and for c ˆ 7. In addition, the upper-bound power envelope is also drawn. To simplify comparisons, a normal quantile scale is used; that is, ÿ1 (power) is plotted against ÿ. Three facts emerge from these calculations (and others not reported here): 1. With normal quantile scale, the power envelope is essentially a straight line with slope 0.24. (Note: the square root of expected information is 0.7.)

Testing for unit roots

287

2. The Dickey±Fuller t-test (which is the signed maximum likelihood ratio test) behaves almost exactly like the NP(7) test. Their power functions are very close to the power envelope and seem to be tangent to the envelope where power is about one-half. 3. In the family of Neyman±Pearson test statistics (6), the power curves are very similar as long as 2 < c < 30. The relative weight given to the score and the Hessian does not matter much as long as each gets some weight. The members with very small c have particularly poor power characteristics.

3

The stationary MA(1) model

Suppose the observed time-series y1 ; . . . ; yn was generated by yt ˆ "t ÿ "tÿ1

t ˆ 1; . . . ; n

…8†

where the "'s are i.i.d. standard normal variates. To avoid identi®ability issues, we shall assume jj  1. The problem of testing the null hypothesis  ˆ  against the alternative that  <  is standard as long as jj is much less than one. Indeed, the results of section 1 apply whenever jj < 1, although the accuracy of the asymptotic approximations will diminish rapidly as jj tends to one. The problem of testing  ˆ 1, however, is nonstandard. Our analysis builds on the work reported by Tanaka (1990), Saikkonen and Luukkonen (1993), and Davis, Chen and Dunsmuir (1995). Let y be the n-dimensional column vector of the observations. Then, apart from an additive constant, the log-likelihood at parameter value s is Ln …s† ˆ ÿ12 log jÿ…s†j ÿ 12y 0 ÿ…s†ÿ1 y;

…9†

where ÿ…s† ˆ …1 ‡ s2 †I ÿ sD and D ˆ ‰dij Š is a matrix with dij ˆ 1 when ji ÿ jj ˆ 1 and zero otherwise. The characteristic roots of D are 2 cos‰j=…n ‡ 1†Š, j ˆ 1; . . . ; n. Hence the matrix ÿ can be written as Q 0 Q where Q is an orthogonal matrix not depending on s and L is diagonal with typical element j …s† ˆ 1 ‡ s2 ÿ 2s cos‰j=…n ‡ 1†Š. It follows that y 0 ÿ…s†ÿ1 y can be written as x 0 …s†ÿ1 x; where the (observable) transformed variable x ˆ Qy has variance matrix …†. Thus, we can write " #  n n  X X x2j j …† 2  ; ˆ ÿ2Ln …s† ˆ log j …s† ‡ log j …s† ‡ j …s† j …s† j jˆ1 jˆ1 …10† where the (unobserved) elements of  ˆ …† normal.

ÿ1=2

Qy are i.i.d. standard

288

Thomas J. Rothenberg

When jj < 1, L…† ÿ L…s† can be expanded as in (1) where the remainder term is asymptotically negligible. This does not happen when  ˆ 1. We write the true parameter as  ˆ 1 ÿ nÿ1 ÿ and the point alternative as s ˆ 1 ÿ nÿ1 c. Then, noting that    ÿ ÿ ÿ 2  ÿ j ÿ2 1 ÿ cos j 1 ÿ ˆ2ÿ2 ‡ ; n n n n n‡1 we ®nd after considerable algebra that, as n tends to in®nity, ÿ2cÿ2 ‰Ln …1 ÿ nÿ1 c† ÿ Ln …1†Š " # 1 X 2 j 2 ‡ ÿ 2 2 j 2 2 j ‡ ln 2 2 ) : 2 j 2 …2 j 2 ‡ c2 †  j ‡ c2 jˆ1

…11†

This implies that, asymptotically, the score evaluated at  ˆ 1 is zero. Hence, the locally best test is based on the Hessian. It is also apparent that the higher-order derivatives of L are not asymptotically negligible. Unlike the standard case discussed in section 1 and the AR unit-root case discussed in section 2, the log-likelihood ratio is not approximately quadratic in the local parameter c. The limiting likelihood ratio depends not on just one or two suf®cient statistics but on a continuum of suf®cient statistics. There are no convenient analytic expressions for the distribution functions of the limiting NP(c) statistics de®ned by (11). Nevertheless, it is easy to simulate the distributions. (The in®nite series was truncated after 200 terms.) Figure 9.2 presents the power envelope and the power functions for a few values of c. Again, a normal quantile scale is used. We can draw the following conclusions. 1. With a normal quantile scale, the power envelope has noticeable curvature for small values of g but is approximately linear for large values of g. 2. The power curve for the NP(7) test is very close to the envelope. It is tangent to the power envelope at a point where power is about onehalf. 3. In the family of Neyman±Pearson tests of the form (11), the power curves are fairly similar as long as 2 < c < 30. The members with very small c have somewhat poorer power characteristics. 4

A non-stationary MA model

Suppose the observed time-series y1 ; . . . ; yn was generated by yt ˆ "t ÿ "tÿ1 ;

…12†

Testing for unit roots

289

Figure 9.2 Asymptotic power functions for unit MA root (stationary model; 5% level; normal quantile scale)

where "1 ; . . . ; "n are i.i.d. standard normal variates and "0 = 0. This model is identical to the one considered in section 3 except for the ®rst observation. When the hypothesised value  is less than one in absolute value, the modi®cation has no effect to a ®rst order of approximation and has only a trivial effect to second order. When  ˆ 1, the effect is nonnegligible even asymptotically. Note Pthat the joint density for "1 ; . . . ; "n is proportional to expfÿ 12 "2t g. But (12) de®nes a one-to-one mapping of the "'s to the y's with Jacobian equal to one. Recursively solving for the "'s in terms of the y's, we ®nd that, apart from an additive constant, the log-likelihood function evaluated at parameter value s is " #2 n tÿ1 X 1X i sy : …13† Ln …s† ˆ ÿ 2 tˆ1 iˆ0 tÿi Again we transform into local coordinates, writing the true parameter value as  ˆ 1 ÿ nÿ1 ÿ and an arbitrary point is the parameter space as s ˆ 1 ÿ nÿ1 c: It is convenient to de®ne the arti®cial (unobserved) variable xct 

tÿ1 X iˆ0

…1 ÿ nÿ1 c†i "tÿi :

Then we ®nd that

290

Thomas J. Rothenberg tÿ1 tÿ1 X X …1 ÿ nÿ1 c†i ytÿi ˆ …1 ÿ nÿ1 c†i …4"tÿi ‡ ÿnÿ1 "tÿi † iˆ0

iˆ0

ˆ "t ‡ nÿ1 …ÿ ÿ c†xct : Thus we ®nd the log-likelihood ratio for testing  ˆ 1 against the alternative  ˆ 1 ÿ nÿ1 c is given by " #2 " #2 n tÿ1 n tÿ1 X X 1X 1X ÿ1 i Ln …1† ÿ Ln …1 ÿ n c† ˆ sy ÿ y 2 tˆ1 iˆ1 tÿi 2 tˆ1 iˆ1 tÿi ˆ

n  2 1X "t ‡ nÿ1 …ÿ ÿ c†xctÿ1 2 tˆ1

ÿ

n 1X ‰" ‡ nÿ1 ÿx0tÿ1 Š2 : 2 tˆ1 t

…14†

Noting that the process nÿ1=2 xc‰snŠ converges to the continuous time process Jc …s† de®ned in (6), we ®nd that the likelihood ratio converges to the stochastic integral …1 …1 2 2 1 2 1 2 …ÿ ÿ c † J …s† ds ÿ ÿ J02 …s† ds ‡ …ÿ ÿ c† c 2 2 0

0



…1 0

Jc …s† dW…s† ÿ ÿ

…1 0

J0 …s†dW…s†:

…15†

Again, there is a continuum of suf®cient statistics asymptotically. Differentiating and evaluating at c ˆ 0, we ®nd that the score test statistic has the simple asymptotic representation " #" # n tÿ1 tÿ1 X X 1X ÿ1 ÿ1 y iy lim c ‰Ln …1† ÿ Ln …1 ÿ n c†Š ˆ c!0 2 tˆ1 iˆ0 tÿ1 iˆ0 tÿ1  2 …1 1 1 ) ÿ W…1† ‡ ÿ W…s† ds : 2 2 0 …16† Except for the score statistic, which asymptotically is a linear transform of a chi square, there are no convenient analytic expressions for the distribution functions of the limiting NP(c) statistics de®ned by (15). Nevertheless, it is easy to simulate the distributions using the representation (13). The sample size n was set at 200 and 10,000 Monte Carlo replications were used. Using a normal quantile scale, ®gure 9.3 presents

Testing for unit roots

291

Figure 9.3 Asymptotic power functions for unit MA root (non-stationary model; 5% level; normal quantile scale)

the power envelope and the power functions for a few values of c. We can draw the following conclusions. 1. With a normal quantile scale, the power envelope has strong curvature for small values of ÿ but is approximately linear for large values of ÿ. 2. The power curve for the NP(7) test is very close to the envelope. It is tangent to the envelope at a point where power is slightly greater than one-half. 3. In the family of Neyman±Pearson tests of the form (11), the power curves are fairly similar as long as 2 < c < 30. The members with very small c have much poorer power characteristics.

5

Conclusions

The large-sample power curves and power envelopes for the three `nonstandard' examples presented here look very much like the small-sample power curves and envelopes predicted by second-order theory for `standard problems'. Although the test statistics themselves are not even close to being normally distributed, the power curves are not very far from looking like normal power curves. Curiously, the NP(7) test seems to behave similarly in all three examples. For the AR(1) model, van Garderen (1997) shows that Efron curvature, viewed as a function of , increases very sharply when  is near one.

292

Thomas J. Rothenberg

Thus, standard second-order theory seems to explain why the score test of the unit-root hypothesis performs rather badly. The result that the power curve for the maximum likelihood ratio test is tangent to the power envelope at the point where power equals one-half is also consistent with standard second-order theory, although the standard proof is not applicable. For the MA unit-root problem, it is not so clear that standard asymptotic theory provides any explanation for our empirical ®ndings. In particular, Efron curvature is not a very useful measure in these models. When  ˆ 1, all the derivatives of the log-likelihood are the same order of magnitude, so it does not seem useful to concentrate only on the ®rst two. The results presented here cannot be viewed as a solution to the problem of explaining the behaviour of tests in non-standard situations. Rather they are really just a description of some of the facts that have to be explained.

References Banerjee, A., J.J. Dolado, J.W. Galbraith and D. Hendry (1993), Cointegration, Error Correction, and Econometric Analysis of Nonstationary Data, Oxford: University Press. Cavanagh, C.L. (1983), `Hypothesis Testing in Models with Discrete Dependent Variables', Ph.D. thesis, University of California. Chan, N.H. and C.Z. Wei (1987), `Asymptotic Inference for Nearly Nonstationary AR(1) Processes', Annals of Statistics, 15: 1050±1063. Davis, R., M. Chen and W. Dunsmuir (1995), `Inference for MA(1) Process with a Root on or near the Unit Circle', Probability and Mathematical Statistics, 15: 227±242. Dickey, D. and W. Fuller (1979), `Distribution of the Estimators for Autoregressive Time Series with a Unit Root', Journal of the American Statistical Association, 74: 427±431. Efron, B. (1975), `De®ning the Curvature of a Statistical Problem', Annals of Statistics, 3: 1189±1242 Elliott, G., T. Rothenberg and J. Stock (1996), `Ef®cient Tests for an Autoregressive Unit Root', Econometrica, 64: 813±836. Garderen, K. van (1997), `Exact Geometry of Explosive Autoregressive Models', unpublished working paper. Pfanzagl, J. (1980), `Asymptotic Expansions in Parametric Statistical Theory', in P. R. Krishnaiah (ed.), Developments in Statistics, vol. 3, New York: Academic Press. Pfanzagl, J. and W. Wefelmeyer (1978), `An Asymptotically Complete Class of Tests', Zeitschrift fuÈr Wahrscheinlichkeitstheorie, 45: 49±72.

Testing for unit roots

293

Rothenberg, T.J. and J. Stock (1997), `Inference in a Nearly Integrated Autoregressive Model with Nonnormal Innovations', Journal of Econometrics, 80: 269±286. Saikkonen P. and R. Luukkonen (1993), `Point Optimal Tests for Testing the Order of Differencing in ARIMA Models', Econometric Theory, 9: 343±362. Stock, J. (1995), `Unit Roots, Structural Breaks, and Trends', in R. Engle and D. McFadden (eds.), The Handbook of Econometrics, vol. 4, Amsterdam: North-Holland. Tanaka, T. (1990), `Testing for a Moving Average Unit Root', Econometric Theory, 6: 433±444.

10

An elementary account of Amari's expected geometry Frank Critchley, Paul Marriott and Mark Salmon

Differential geometry has found fruitful application in statistical inference. In particular, Amari's (1990) expected geometry is used in higherorder asymptotic analysis and in the study of suf®ciency and ancillarity. However, we can see three drawbacks to the use of a differential geometric approach in econometrics and statistics more generally. First, the mathematics is unfamiliar and the terms involved can be dif®cult for the econometrician to appreciate fully. Secondly, their statistical meaning can be less than completely clear. Finally, the fact that, at its core, geometry is a visual subject can be obscured by the mathematical formalism required for a rigorous analysis, thereby hindering intuition. All three drawbacks apply particularly to the differential geometric concept of a non-metric af®ne connection. The primary objective of this chapter is to attempt to mitigate these drawbacks in the case of Amari's expected geometric structure on a full exponential family. We aim to do this by providing an elementary account of this structure that is clearly based statistically, accessible geometrically and visually presented. Statistically, we use three natural tools: the score function and its ®rst two moments with respect to the true distribution. Geometrically, we are largely able to restrict attention to tensors; in particular, we are able to avoid the need formally to de®ne an af®ne connection. To emphasise the visual foundation of geometric analysis we parallel the mathematical development with graphical illustrations using important examples of full exponential families. Although the analysis is not restricted to this case, we emphasise one-dimensional examples so that simple pictures can

This work has been partially supported by ESRC grant `Geodesic Inference, Encompassing and Preferred Point Geometry in Econometrics' (Grant Number R000232270).

294

Parameterisations and transformations

295

be used to illustrate the underlying geometrical ideas and aid intuition. It turns out that this account also sheds some new light on the choice of parameterisation as discussed by Amari (1990), extending earlier work by Bates and Watts (1980, 1981), Hougaard (1982) and Kass (1984). There are also a number of points of contact between our presentation and Firth (1993). A key feature of our account is that all expectations and induced distributions are taken with respect to one ®xed distribution, namely, that assumed to give rise to the data. This is the so-called preferred point geometrical approach developed in Critchley, Marriott and Salmon (1993, 1994), on whose results we draw where appropriate. Our hope is that the following development will serve to broaden interest in an important and developing area. For a more formal but still readable treatment of differential geometry, see Dodson and Poston (1977). For broader accounts of the application of differential geometry to statistics, see the review chapters or monographs by Barndorff-Nielsen, Cox and Reid (1986), Kass (1987, 1989), Amari (1990) and Murray and Rice (1993). The chapter is organised as follows. The elementary prerequisites are established in section 1. The key elements of Amari's expected geometry of general families of distributions are brie¯y and intuitively reviewed in section 2. In particular, his -connections are discussed in terms of the characteristic statistical properties of their associated af®ne parameterisations. Section 3 contains our account of this geometry in the full exponential family case, as outlined above, and section 4 considers the effect of changing the sample size.

1

Preliminaries

1.1

The general framework

Let M ˆ fp…x; † :  2 g be a p-dimensional parametric family of probability (density) functions. The available data x ˆ …x1 ; . . . ; xn †T is modelled as a random sample from some unknown true distribution p…x; † 2 M. Let the parameter space  be an open connected subset of Rp . The family M is regarded as a manifold, with the parameter  playing the role of a coordinate system on it. Formally, certain regularity conditions are entailed. These are detailed in Amari (1990, p. 16).

296

Frank Critchley, Paul Marriott and Mark Salmon

1.2

The score function

The score function  T @ @ s…; x† ˆ ln p…x; †; . . . ; ln p…x; † @p @1 is very natural to work with statistically as it contains precisely all the relevant information in the likelihood function. Integrating over  recovers the log-likelihood function, l, up to an additive constant which is independent of . This is equivalent to the likelihood up to a multiplicative positive factor which may depend on x but not on . As discussed by Cox and Hinkley (1974, p. 12), two different choices of the constant do not affect the essential likelihood information, which we refer to as the shape of the likelihood. Visually, the graph of the score function displays the shape of the likelihood in a natural and direct way. We use this to advantage later. The score function is also a very natural tool to work with geometrically. An important concept of differential geometry is that of the tangent space. We can avoid the general abstract de®nition here as we have a concrete representation of this space in terms of the score function. Regarding x now as a random vector and following Amari (1990), we identify the tangent space TM at each ®xed p…x; † 2 M with the vector space of random variables spanned by fsi …; x† ˆ

@ ln p…x; † : i ˆ 1; . . . ; pg: @i

Under the regularity conditions referenced in section 2.3 of chapter 1, this vector space has dimension p, the dimension of M. 1.3

Distribution of the score vector

Naturally associated with each ®xed tangent space TM is the joint distribution  of the components of the score vector s…; x†. This may be known analytically but can always, by the central limit theorem, be approximated asymptotically by the multivariate normal distribution Np … …†; g …††, where  …† ˆ Ep…x;† ‰s…; x†Š ˆ nEp…x;† ‰s…; x†Š and g …† ˆ Covp…x;† ‰s…; x†Š ˆ nCovp…x;† ‰s…; x†Š:

Parameterisations and transformations

297

These last two quantities are statistically natural tools that we shall employ in our account of Amari's geometry. The matrix g …† is assumed to be always positive de®nite. Note that, for all ,  …† ˆ 0

g …† ˆ I…† ˆ ni…†;

and

where I and i denote the Fisher information for the sample and for a single observation, respectively. For later use we de®ne the random vector  …; x† by the decomposition s…; x† ˆ  …† ‡  …; x† so that Ep…x;† ‰ …; x†Š vanishes identically in  and . In the one-dimensional case there is a particularly useful graphical representation of the three tools on which our account is based. For a particular realisation of the data x, the plot of the graph of s…; x† against  can give great insight into the shape of the observed likelihood function. We call this graph the observed plot. Together with this we use the expected plot. This is a graph of the true mean score together with an indication of variability. We make extensive use of this graphical method for several important examples below. 1.4

Reparameterisation

So far, we have worked in a single parameterisation . It is important to consider what happens under a reparameterisation. We consider reparameterisations  ! …† that are smooth and invertible. De®ne Bi …† ˆ

@ @i

and

@i B i …† ˆ  ; @

for 1  i;   p. By the chain rule, the components of the score vector transform as 1-tensors. That is: X i X i @l @l B  ……†† i :ˆ B  …†si …; x† ˆ  @ @ iˆ1 iˆ1 p

s ……†; x† :ˆ

p

…1†

for each ®xed . This amounts to a change of basis for the vector space TM . By linearity of expectation, the components of  …† are also 1tensors. That is: …†  ……†† ˆ

p X iˆ1

B i …†i …†:

…2†

298

Frank Critchley, Paul Marriott and Mark Salmon

As covariance is a bilinear form, we see that g …† is a 2-tensor. That is, its components transform according to: g…† ÿ ……†† ˆ

p X p X iˆ1 jˆ1

B i …†B jÿ …†gij …†:

…3†

By symmetry, the assumption of positive de®niteness, and since g …† varies smoothly with , g …† ful®ls the requirements of a metric tensor (see Amari (1990), p. 25). It follows at once, putting  ˆ , that the Fisher information also enjoys this property. In parallel with this tensor analysis, plotting the observed and expected plots for different parameterisations of the model can be extremely useful in conveying the effects of reparameterisation on the shape of the likelihood and the statistical properties of important statistics such as the maximum likelihood estimate (MLE). The question of parameterisation is therefore an important choice that has to be taken in statistical analysis. 2

Some elements of Amari's expected geometry

2.1

Connections

Formally, Amari's expected geometry is a triple …M; I; r‡1 † in which M is a family of probability (density) functions and I the Fisher information metric tensor, as described above. The major dif®culty in understanding revolves around the third component, r ‡1 , which is a particular nonmetric af®ne connection. In section 3 we obtain a simple, statistical interpretation of it in the full exponential family case. Here we note certain facts concerning connections and Amari's geometry, offering intuitive explanations and descriptions where possible. For a formal treatment, see Amari (1990). We emphasise that such a treatment is not required here, as our later argument proceeds in terms of the elementary material already presented. A connection allows us to (covariantly) differentiate tangent vectors and, more generally, tensors (see Dodson and Poston (1977), chapter 7). A connection therefore determines which curves in a manifold shall be called `geodesic' or `straight'. Generalising familiar Euclidean ideas, these are de®ned to be those curves along which the tangent vector does not change. A metric tensor induces in a natural way an associated connection called the Levi±Civita or metric connection. In Amari's structure the Fisher information I induces the af®ne connection denoted by r0 . The Levi±Civita connection has the property that its geodesics are curves of

Parameterisations and transformations

299

minimum length joining their endpoints. No concept of length is associated with the geodesics corresponding to non-metric connections. Amari shows that the two connections r0 and r‡1 can be combined to produce an entire one-parameter family fr :  2 Rg of connections, called the -connections. The most important connections statistically correspond to  ˆ 0; 13; 1, as we now explain.

2.2

Choice of parameterisation

For each of Amari's connections it can happen that a parameterisation  of M exists such that the geodesic joining the points labelled  1 and 2 simply consists of the points labelled f…1 ÿ †1 ‡ 2 : 0    1g. For example, Cartesian coordinates de®ne such a parameterisation in the Euclidean case. When this happens, M is said to be ¯at, such a parameterisation is called af®ne, and the parameters are unique up to af®ne equivalence. That is, any two af®ne parameterisations are related by a non-singular af®ne transformation. In the important special case of a metric connection, M is ¯at if and only if there exists a parameterisation  in which the metric tensor is independent of . For a connection to admit an af®ne parameterisation is a rather special circumstance. When it does, we may expect the af®ne parameterisation to have correspondingly special properties. This is indeed the case with Amari's expected geometry. When an -connection has this property, the manifold is called -¯at and the associated parameterisations are called -af®ne. Amari (1990, Theorem 5.12, p. 152), established the following characteristic features of certain -af®ne parameterisations: 1.  ˆ 1 corresponds to the natural parameter, . 2.  ˆ 13 corresponds to the normal likelihood parameter. 3.  ˆ 0 gives a constant asymptotic covariance of the MLE. 4.  ˆ ÿ13 gives zero asymptotic skewness of the MLE. 5.  ˆ ÿ1 gives zero asymptotic bias of the MLE. These correspond to the  ˆ 0; 13; 12; 23; 1 parameterisations, respectively, of Hougaard (1982), who studied the one-dimensional curved exponential family case. In any one-dimensional family an -af®ne parameter exists for every . A full exponential family, of any dimension, is always ‡1¯at and ÿ1-¯at, with the natural and mean value parameters, respectively, being af®ne. Amari (1990) also established the duality result that M is -¯at if and only if it is ÿ-¯at. This duality between r and rÿ has nice mathematical properties but has not been well understood statistically.

300

Frank Critchley, Paul Marriott and Mark Salmon

3

The expected geometry of the full exponential family

3.1

Introduction

We restrict attention now to the full exponential family. In the natural parameterisation, , we have ( p ) X p…x; † ˆ exp ti …x†i ÿ þ…† : iˆ1

The mean value parameterisation is given by  ˆ …1 ; . . . ; p †, where i …† ˆ Ep…x;† ‰ti …x†Š ˆ

@þ …†: @i

These two parameterisations are therefore af®nely equivalent if and only if þ is a quadratic function of , as with the case of normal distributions with constant covariance. As we shall see, this is a very special circumstance. In natural parameters, the score function is   @þ  si …; x† ˆ n ti …x† ÿ i …† ˆ nfti …x† ÿ i …†g; …4† @ P where nti …x† ˆ nrˆ1 ti …xr †. From (4) we have the useful fact that the ^ ˆ ti . Further, the ®rst two maximum likelihood estimator ^ i :ˆ i …† moments of the score function under p…x; † are given by   @þ @þ  i …† ˆ n …† ÿ …† ˆ nfi …† ÿ i …†g …5† @ i @ i @2 þ g ij …† ˆ n i j …† ˆ Iij …†: @ @ 3.2

…6†

Examples

The following one-dimensional examples are used for illustrative purposes: Poisson, normal with constant (unit) variance, exponential and Bernoulli. Although, of course, the sample size affects the -distribution of t, it enters the above equations for the score and its ®rst two moments only as a multiplicative constant. Therefore our analysis, which is based solely on these quantities, is essentially invariant under independent repeated samples. Our third and fourth examples implicitly cover the gamma and binomial families and together, then, these examples embrace most

Parameterisations and transformations

301

Table 10.1. One-dimensional examples: Poisson, normal, exponential and Bernoulli Poisson ()

Normal (; 1)

Exponential ()

Bernoulli ()

(Figure 10.1)

(Figure 10.2)

(Figure 10.3)

(Figure 10.4)

t…x†

x

x

ÿx

x

ÿ…†

e

1 2  2

ÿ ln 

ln…1 ‡ e † n‰ x ÿ e …1 ‡ e †ÿ1 Š

s…; x†

n… x ÿ e †

n… x ÿ †

n…ÿ x ‡ ÿ1 †

 …†

n…e ÿ e †

n… ÿ †

n…ÿÿ1 ‡ ÿ1 †

g …†

ne

n

nÿ2

…†  B…† s…; x†



…† ˆ e 

ÿ1

n… x ÿ †

1=3



3 ÿ1

…† ˆ ÿ

2

 3

3n… x ÿ  †

2

n

e e ÿn 1 ‡ e 1 ‡ e ne …1 ‡ e †ÿ2

ÿ1

ÿn… x ‡ †

…† ˆ e …1 ‡ e †ÿ1 ‰…1 ÿ †Šÿ1

ÿ2 ÿ2

…† …†

n‰…† ÿ Šÿ1

3n‰3 …† ÿ 3 Š2

n‰…† ÿ Šÿ2

g…† …†

n…†ÿ2

9n4

n…†2 ÿ4

n… x ÿ †‰…1 ÿ †Šÿ1 n

‰…† ÿ Š ‰…1 ÿ †Š

…†‰1 ÿ …†Š n ‰…1 ÿ †Š2



0

0

1

0

n

10

10

10

10

of the distributions widely used in generalised linear models (McCullagh and Nelder, 1989). The examples are summarised algebraically in table 10.1, and are displayed visually in ®gures 10.1 to 10.4, respectively. For each example, for a chosen  and n shown in table 10.1, we give observed and expected plots, both in the natural parameterisation  and in a non-af®nely equivalent parameterisation …†. We take …† to be the mean value parameter …† except in the normal 1 case, where we take …† ˆ 3 . We use this last parameterisation for illustration only, even though it is not invertible at  ˆ 0. In each case,  is an increasing function of . In the expected plots, we illustrate the ®rst two moments of the score function under the true distribution (that is, under p…x; †) by plotting the mean 2 standard deviations. In the observed plots, to give some idea of sampling variability, we plot ®ve observed score functions corresponding to the 5%, 25%, 50%, 75% and 95% points of the true distribution of t for the continuous families and the closest observable points to these in the discrete cases. Recall that these

5 0

Score

-10

-5

0 -10

-5

Score

5

10

Frank Critchley, Paul Marriott and Mark Salmon 10

302

-0.6

-0.4

-0.2

0.0

0.2

0.4

-0.6

-0.2

0.0

0.2

0.4

30 20 -10

0

10

Score

20 10 -10

0

Score

-0.4

Expected Plot: Natural parameters

30

Observed Plot: Natural parameters

0.5

1.0

1.5

2.0

0.5

1.0

Observed Plot: xi-parameters

1.5

2.0

Expected Plot: xi-parameters

10 15 5 0

Score

-15

-5

0 -15

-5

Score

5

10 15

Figure 10.1 One-dimensional Poisson example

-0.4

-0.2

0.0

0.2

0.4

-0.4

0.0

0.2

0.4

20 10 -20

-10

0

Score

10 0 -20

-10

Score

-0.2

Expected Plot: Natural parameters

20

Observed Plot: Natural parameters

-0.5

0.0

0.5

Observed Plot: xi-parameters

Figure 10.2 One-dimensional normal example

-0.5

0.0

0.5

Expected Plot: xi-parameters

15 10 5

Score

-5 -10 0.5

1.0

1.5

2.0

0.5

1.0

1.5

2.0

-20

Score

-60

-40 -60

-40

-20

0

Expected Plot: Natural parameters

0

Observed Plot: Natural parameters

Score

303

0

5 0 -10

-5

Score

10

15

Parameterisations and transformations

-1.6

-1.4

-1.2

-1.0

-0.8

-0.6

-0.4

-1.6

Observed Plot: xi-parameters

-1.4

-1.2

-1.0

-0.8

-0.6

-0.4

Expected Plot: xi-parameters

6 4 2 Score -1.5 -1.0

-0.5 0.0

0.5

-6 -4 -2 0

-6 -4 -2 0

Score

2

4

6

Figure 10.3 One-dimensional exponential example

-1.5 -1.0

1.5

1.0

Score 0.2

0.4

0.6

-0.5 0.0

0.5

1.0

1.5

Expected Plot: Natural parameters

-60 -40 -20 0 20 40 60 80

Score

-60 -40 -20 0 20 40 60 80

Observed Plot: Natural parameters

0.8

Observed Plot: xi-parameters

Figure 10.4 One-dimensional Bernoulli example

0.2

0.4

0.6

Expected Plot: xi-parameters

0.8

304

Frank Critchley, Paul Marriott and Mark Salmon

plots precisely contain the shape of the observed and expected likelihood functions and thus are a direct and visual representation of important statistical information. The observed score graphs do not cross since, for each ®xed parameter value, the observed score function is a non-decreasing af®ne function of t. This holds in all parameterisations, using (1). From (1), (2), (4) and (5) it is clear that, in any parameterisation, the graph of the true mean score function coincides with that of the observed score for data where t…x† equals its true mean …†. In the examples, the true distribution of nt is given by Poisson… ‡ ln n†, normal…n; n†, gamma…; n† and binomial…n; †, respectively. The most striking feature of the plots is the constancy of the variance of the score across the natural parameterisation, and the fact that this property is lost in the alternative parameterisation. Also remarkable is the linearity of the normal plots in the natural parameterisation. A close inspection reveals that for each example, in the natural parameterisation, each of the observed plots differs by only a vertical translation. Again this property will not hold in a general parameterisation. We use these and other features of the plots to better understand Amari's expected geometry. Certain information is evident from the plots straight away. Under standard regularity conditions, the unique maximum likelihood estimate of a parameter for given data occurs when the graph of the corresponding observed score function crosses the horizontal axis from above. Thus, as t ˆ ^ in our examples (even in the degenerate Bernoulli case), these ®ve crossing points are the 5%, 25%, 50%, 75% and 95% points of the true distribution of the maximum likelihood estimate. The position of these ®ve crossing points gives visual information about this distribution, in particular about its location, variance and skewness. Of more direct relevance to our present concern is the fact that, in these one-dimensional cases, there is a straightforward visual representation of the tangent space at each point. TM can be identi®ed with the vertical line through , and  with the distribution of the intersection of this line with the graph of the observed score function. Identical remarks apply in any parameterisation. These tangent spaces are shown in both parameterisations, at the above ®ve percentage points of the maximum likelihood estimator, as lines in the observed plots and as vertical bars in the expected plots. In the observed plot, the ®ve intersection points with any given tangent space TM are the ®ve corresponding percentage points of  . The same is true in any increasing reparameterisation . Thus, comparing the position of these ®ve intersection points at corresponding parameter values in

Parameterisations and transformations

305

the two observed plots gives direct visual information on the difference between  and …† …† ; in particular, on changes in skewness. The observed plots also show very clearly that, as the natural parameter varies, the true distribution of the score changes only in its location, whereas this is not so in a general parameterisation. This brings to light a certain natural duality between the maximum likelihood estimator and the score function. Consider the observed plots in the natural and mean value parameterisations. For any given point consider its corresponding tangent space TM and TM…† in the two plots. In each plot we have ®ve horizontal and ®ve vertical crossing points, as above, giving information about the distribution of the maximum likelihood estimator and the score function respectively in the same parameterisation. Now, these two plots are far from independent. As ^ …x† ˆ …† ‡ nÿ1 s…; x†, the horizontal crossing points in the mean parameter plot are just an af®ne transformation of the vertical crossing points in the natural parameter plot. The converse is true asymptotically. As we discuss below, this simple and natural duality between the maximum likelihood estimator and the score function corresponds with the duality present in Amari's expected geometry. 3.3

Amari's +1-geometry

The above one-dimensional plots have already indicated two senses in which the natural parameterisation is very special. We note here that this is so generally. Our analysis then provides a simple statistical interpretation of Amari's +1-connection. From (4) we see that in the natural parameterisation the score function has the form of a stochastic part, independent of , plus a deterministic part, independent of the data. Recalling (1) and (4) we see that this  property is lost in a non-af®ne reparameterisation , since B…† …:ˆ B 11 …†† is independent of  if and only if  is an af®ne transformation of . An equivalent way to describe this property is that the `error term'  …; x† in the mean value decomposition of s…; x† de®ned at the end of section 1.3 is independent of . Or again, as  …† vanishes, that this decomposition has the form s…; x† ˆ  …† ‡ s…; x†:

…7†

Note that  differs from 0 only by the translation  …† ÿ  … 0 †. In this parameterisation, from one sample to the next, the whole graph of the observed score function just shifts vertically about its -expectation by the same amount s…; x†.

306

Frank Critchley, Paul Marriott and Mark Salmon

As a consequence of (7), the -covariance of the score function is independent of  (and therefore coincides with g …† ˆ I…†). But g …† is a metric tensor (section 1.4) and, in this parameterisation, the metric is constant across all tangent spaces. Recalling section 2.2, we note that if a metric is constant in a parameterisation then the parameterisation is af®ne for the metric connection. All tangent spaces thus have the same geometric structure and differ only by their choice of origin. For more details on this geometric idea of ¯atness, see Dodson and Poston (1977). The metric connection is the natural geometric tool for measuring the variation of a metric tensor in any parameterisation. But Critchley, Marriott and Salmon (1994) prove that, in the full exponential family, the metric connection induced by g …† coincides with Amari's ‡1-connection. Thus we have the simple statistical interpretation that r‡1 is the natural geometric measure of the non-constancy of the covariance of the score function in an arbitrary parameterisation. In the one-dimensional case, the ‡1-connection measures the variability of variance of the observed score across different points of M. Looking again at ®gures 10.1 to 10.4 we see a visual representation of this fact in that the 2 standard deviation bars on the expected plot are of a constant length for the -parameterisation, and this does not hold in the non-af®ne -parameterisation.

3.4

Amari's 0-geometry

The fact that in the natural parameterisation all the observed score functions have the same shape invites interpretation. From (7) we see that the common information conveyed in all of them is that conveyed by their mean. What is it? The answer is precisely the Fisher information for the family. This is clear since  determines I via Iij …† ˆ ÿ

@j @i

…†;

while the converse is true by integration, noting that  …† ˆ 0. Thus, in natural parameters, knowing the Fisher information at all points is equivalent to knowing the true mean of the score function (and hence all the observed score functions up to their stochastic shift term). In particular, in the one-dimensional case, the Fisher information is conveyed visually by minus the slope of the graph of  …† as, for example, in the natural parameter expected plots of ®gures 10.1 to 10.4.

Parameterisations and transformations

307

Amari uses the Fisher information as his metric tensor. It is important to note that, when endowed with the corresponding metric connection, an exponential family is not in general ¯at. That is, there does not, in general, exist any parameterisation in which the Fisher information is constant. The multivariate normal distributions with constant covariance matrix and any one-dimensional family are notable exceptions. In the former case, the natural parameters are af®ne. In the latter case, using (3), the af®ne parameters are obtained as solutions to the equation  2 @ …† þ 00 …† ˆ constant: @ For example, in the Poisson family where þ…† ˆ exp…† one ®nds …† ˆ exp…=2†, as in Hougaard (1982). Thus far we have seen that, in the case of the full exponential family, the fundamental components of Amari's geometry …M; I; r‡1 † can be simply and naturally understood in terms of the ®rst two moments of the score function under the distribution assumed to give rise to the data. I is de®ned by the true mean, and r‡1 by I and the true covariance. Further, they can be understood visually in terms of the expected plots in our one-dimensional examples. We now go on to comment on duality and choice of parameterisation. 3.5

Amari's 71-geometry and duality

The one-dimensional plots above have already indicated a natural duality between the score vector and the maximum likelihood estimator, and that there is a natural statistical curvature, even in the one-dimensional case, unless the manifold is totally ¯at; that is, unless the graph of the true mean score function is linear in the natural parameterisation. We develop these remarks here. Amari (1990) shows that the mean value parameters …† ˆ Ep…x;† ‰t…x†Š ˆ þ 0 …† are ÿ1-af®ne and therefore, by his general theory, duality related to the natural ‡1-af®ne parameters . We offer the following simple and direct statistical interpretation of this duality. We have, ^ ˆ …† ‡ nÿ1 s…; x†: ^ to ®rst order about  gives an asymptotic converse Expanding …†  ^ ˆ _  ‡ nÿ1 B…†s…; x† ˆ  ‡ nÿ1 s…; x†;

308

Frank Critchley, Paul Marriott and Mark Salmon

the right-hand equality following from (1) and where we use ˆ _ to denote  ˆ iÿ1 …†. Thus the dua®rst-order asymptotic equivalence. Note that B…† lity between the ‡1 and ÿ1 connections can be seen as the above strong and natural asymptotic correspondence between the maximum likelihood estimator in one parameterisation and the score function in another. In fact this simple statistical interpretation of Amari's duality is not restricted to the full exponential family (see Critchley, Marriott and Salmon (1994)). It is established formally in a more general case than ‡1 duality here in section 3.7. 3.6

Total ¯atness and choice of parameterisation

The above approximation to ^ is exact when  and  are af®nely equivalent. In this case, ^ and ^ are in the same af®ne relationship and so their distributions have the same shape. In particular, as normality is preserved under af®ne transformations, these distributions are as close to normality as each other whatever the de®nition of closeness that is used. In the case where M is a constant covariance normal family, ^ and ^ are both exactly normally distributed. Af®ne equivalence of  and  is a very strong property. When it holds, much more is true. It is the equivalent in the full exponential family case of the general geometric notion of total ¯atness de®ned and studied in Critchley, Marriott and Salmon (1993). Recall that the natural parameterisation  has already been characterised by the fact that the true covariance of the score function is constant in it. Total ¯atness entails this same parameterisation simultaneously has other nice properties. It is easy to show the following equivalences:  and  are affinely equivalent ()

þ is a quadratic function of 

()

I…† is constant in the natural parameters

()

 …† is an affine function of 

()

9  6ˆ ÿ with r ˆ rÿ

()

8; 8ÿ;

()

the  parameterisation is -affine for all 

r ˆ rÿ

(see Critchley, Marriott and Salmon (1993)). In particular, the maximum likelihood estimators of any -af®ne parameters are all equally close (in any sense) to normality.

Parameterisations and transformations

309

It is exceptional for a family M to be totally ¯at. Constant covariance multivariate normal families are a rare example. In totally ¯at manifolds the graph of  …† is linear in the natural parameterisation, as remarked upon in the one-dimensional normal example of ®gure 10.2. More usually, even in the one-dimensional case, a family M of probability (density) functions will exhibit a form of curvature evidenced by the non-linearity of the graph of  …†. Recall that the graph of  …† enables us to connect the distribution of ^ and . ^ In the natural parameterisation , each observed graph is a vertical shift of the expected graph. This shift is an af®ne function of ^ ^ The intersection of the observed plot with the  axis determines . t ˆ . ^ When the expected plot is linear (the totally ¯at case), then  and ^ are af®nely related and so their distributions have the same shape. When it is non-linear they will not be af®nely related. This opens up the possibility that, in a particular sense of `closeness', one of them will be closer to normality. In all cases, the 0-geometry plays a pivotal role between the 1-geometries. That is, the graph of  …† determines the relationship between the distributions of the maximum likelihood estimators ^ and ^ of the 1af®ne parameters. We illustrate this for our examples in ®gure 10.5. Both distributions are of course exactly normal when the parent distribution is. In the Poisson case, the concavity of  …† means that the positive skewness of ^ is reduced. Indeed, ^ has negative skew, as ®gure 10.5a illustrates. The opposite relationship holds in the exponential case, where  …† is convex (®gure 10.5c). In our Bernoulli example, the form of  …† preserves symmetry while increasing kurtosis so that, in this sense, the distribution of ^ is closer to normality than that of ^ (®gure 10.5d). 3.7

Amari's 13 -geometry and duality

Amari's 13 -connection can be simply interpreted in terms of linearity of the graph of the true mean score function, at least in the one-dimensional situation where the 13 -af®ne parameters are known to exist. If M is totally ¯at, this graph is linear in the natural parameterisation, as in the normal constant covariance family. It is therefore natural to pose the question: Can a parameterisation be found for a general M in which this graph is linear? This question can be viewed in two ways. First, for some given p…x; †, is such a parameterisation possible? However, in this case, any parameterisation found could be a function of the true distribution. In general, there will not be a single parameterisation that works for all . The

310

Frank Critchley, Paul Marriott and Mark Salmon

(a) Poisson 0.12

•

0.10

0.10

0.08

2

•

•

0.06 0.04

•

Score 0

•

-2

• 0.06

• • •

•

•

0.08

•

4

0.12

•• •

•

•

•

•

-4

•

• •

• •

• ••

0.0

0.0

0.02

•

-2.0 -1.5 -1.0 -0.5 0.0 0.5 Probability function of MLE: Natural parameters

•

0.02

0.04

•

-0.6

• •

• • •

-0.4 -0.2 0.0 0.2 0.4 Mean score natural parameters

•

•

•

• •

0.5 1.0 1.5 2.0 Probability function of MLE: Expected parameters

0.4 0.3 0.2 0.1 0.0

0.0

-4

0.1

-2

0.2

Score 0

2

0.3

4

0.4

(b) Normal

-0.3

-0.2 -0.1 0.0 0.1 0.2 0.3 Density of MLE: Natural parameters

-0.4

-0.2 0.0 0.2 0.4 Mean score natural parameters

-0.3

-0.2 -0.1 0.0 0.1 0.2 0.3 Density of MLE: Expected parameters

-3.0

-2.5 -2.0 -1.5 -1.0 -0.5 Density of MLE: Expected parameters

1

2 3 4 5 Density of MLE: Natural parameters

0.10 0.08 0.0

0

-10

2

0.02

-5

4

0.04

0

6

0.06

5

8

10

10

12

0.12

15

(c) Exponential

0.5

1.0 1.5 2.0 Mean score natural parameters

4 •

•

•

• -2

0.15 0.05 0.0

•

•

• -4

0.05

•

•

0.10

•

-2

0.10

•

Score 0

0.15

2

•

•

0.20

0.20

•

0.25

0.25

(d) Bernoulli

-1 0 1 2 Probability function of MLE: Natural parameters

-1 0 1 Mean score natural parameters

•

•

• •

•

0.2 0.4 0.6 0.8 1.0 Probability function of MLE: Expected parameters

Figure 10.5 The distributions of the natural and expected parameter estimates

Parameterisations and transformations

311

second way is to look locally to . This is the more fruitful approach statistically. The question then becomes: Can a single parameterisation  !  be found such that, for all , the graph of the true mean score is linear locally to  ˆ …†? In the one-dimensional case, we seek  such that 8;

@2 …† …† jˆ…† ˆ 0: @2

Such a local approach is suf®cient asymptotically when the observed score function will be close to its expected value and the maximum likelihood estimate will be close to the true parameter. Thus in such a parameterisation, whatever the true value, the observed log-likelihood will asymptotically be close to quadratic near the MLE. Hence the name, normal likelihood parameter. Amari (1990) shows that such parameters always exist for a one-dimensional full exponential family, and that they are the 13 -af®ne parameters. The vanishing of the second derivative of the true expected score function in one parameterisation  ®nds a dual echo in the vanishing of the asymptotic skewness of the true distribution of the maximum likelihood estimator in another parameterisation . This is called the ÿ13 -af®ne parameterisation, because it is induced by Amari's ÿ13 -connection. Note again that the duality is between the score function and the maximum likelihood estimator, as in section 3.5. This can be formalised as follows. Consider any one-dimensional full exponential family, p…x; † ˆ expft…x† ÿ þ…†g: Let  and  be any two reparameterisations. Extending the approach in section 3.5, it is easy to show the following equivalences: @ @ ˆ þ 00 …†: ^ ˆ _  ‡ nÿ1 s…; x† () ^ ˆ _  ‡ nÿ1 s…; x† () @ @ In this case, we say that  and  are þ-dual. Clearly, the natural (‡1af®ne) and mean value (ÿ1-af®ne) parameters are þ-dual. A parameter  is called self þ-dual if it is þ-dual to itself. In this case we ®nd again the differential equation for the 0-af®ne parameters given in section 3.4. More generally, it can be shown that for any  2 R  and  are þ-dual ) ‰ is -affine ()  is ÿ -affineŠ: For a proof see the appendix to this chapter. Thus the duality between the score function and the maximum likelihood estimator coincides quite generally with the duality in Amari's expected geometry. Note that the simple notion of þ-duality gives an easy way to ®nd ÿaf®ne parameters once ‡-af®ne parameters are known. For example,

Frank Critchley, Paul Marriott and Mark Salmon

0

0.0

-150

2

0.5

4

-100

1.0

6

-50

1.5

8

0

10

2.0

50

12

2.5

14

100

312

0.8

1.0 1.2 1.4 1.6 1/3 Parameterisation

0.5

1.0 1.5 2.0 2.5 Expected score 1/3 parameterisation

0.6

0.8

1.0 1.2 -1/3 Parameterisation

1.4

Figure 10.6 The distributions of the 1/3 af®ne parameter estimates: the exponential case 1

given that  ˆ 3 is 13 -af®ne in the exponential family (Hougaard, 1982) where þ…† ˆ ÿ ln…†, one immediately has @ ˆ 3ÿ4=3 ; @ whence ÿ1=3 is ÿ13 -af®ne. Again, in the Poisson family,  ˆ exp…=3† is 13 af®ne gives at once that exp…2=3† is ÿ13 -af®ne. The local linearity of the true score in ‡13 -parameters suggests that asymptotically the distributions of the maximum likelihood estimator of the 13 -af®ne parameters will be relatively close compared, for example, with those of the 1-af®ne parameters. In particular, it suggests that both will show little skewness. Figure 10.6, which may be compared with ®gure 10.5(c), conveys this information for our exponential family example. 4

Sample size effects

In this section we look at the effect of different sample sizes on our plots of the graph of the score vector. For brevity we concentrate on the exponential model. In ®gure 10.7 we plot the observed scores, taken as before at the 5%, 25%, 50%, 75% and 95% points of the distribution of the score vector. We do this in the natural -parameters and the ÿ1-af®ne mean value -parameters, for sample sizes 5, 10, 20 and 50. In the natural parameters we can see that the distribution of ^ approaches its asymptotic normal limit. Its positive skewness visibly decreases as the sample size increases. More strikingly, the non-linearity in each of the graphs of the observed scores reduces quickly as n increases. For the sample size 50 case, we see that each graph is, to a close degree of approximation, linear. This implies that at this sample size

Parameterisations and transformations

313

Score -40 -30 -20 -10 0

Score -10 -5 0 5 10 15

(a) Sample size 5

0.6

0.8

1.0 1.2 1.4 1.6 Observed Plot: Natural parameters

1.8

1.0 1.5 Observed Plot: Natural parameters

2.0

-2.0

-1.5 -1.0 Observed Plot: Expected parameters

-0.5

Score -60 -40 -20 0

Score -10 -5 0 5 10 15

(b) Sample size 10

0.5

-1.6

-1.4

-1.2 -1.0 -0.8 -0.6 Observed Plot: Expected parameters

-0.4

-15

Score -40 -30 -20 -10 0 10

Score -5 0 5 10 15

(c) Sample size 20

0.8

1.0 1.2 1.4 Observed Plot: Natural parameters

-1.4

-1.2

-1.0 -0.8 Observed Plot: Expected parameters

-40

Score -20 -10 0 10 20

Score -20 0 10 20

(d) Sample size 50

0.8

0.9

1.0 1.1 1.2 Observed Plot: Natural parameters

1.3

-1.3

-1.2

-1.1 -1.0 -0.9 Observed Plot: Expected parameters

-0.8

Figure 10.7 The effect of sample size on the relationship between the score vector and the MLE: the exponential case

there will be almost an af®ne relationship between the score in  coordi^ thus demonstrating their nates and the maximum likelihood estimator , well-known asymptotic af®ne equivalence. It also throws light on the familiar asymptotic equivalence of the score test, the Wald test and (given the asymptotic normality of the maximum likelihood estimate) the likelihood ratio test. For any model in any smooth invertible reparameterisation of the natural parameters asymptotically the graphs of the observed score will tend to the natural parameterisation plot of the normal distribution shown in ®gure 10.2. In this limit the graphs become straight and parallel. We can see both these processes in the -parameterisation of ®gure 10.7. In this example, a higher sample size than for the natural parameter case is needed to reach the same degree of asymptotic approximation. The highly non-linear and non-parallel graphs of sample size 5 and 10 have been reduced to a much more moderate degree of non-linearity for sample size 50. However, this sample size is not quite suf®cient to produce the parallel, linear graphs of the -parameterisation, thus there will still not quite be an af®ne relationship between the score and the maximum likelihood estimator.

314

Frank Critchley, Paul Marriott and Mark Salmon

Appendix We give the proof of the equivalence claimed in section 3.7. We assume here familiarity with the use of Christoffel symbols (see Amari (1990), p. 42). Theorem Let M be a one-dimensional full exponential family, and assume the parameterisations  and  are þ-dual. Then  is ‡-af®ne if and only if  is ÿ-af®ne. isation

Proof

From Amari (1990) we have in the natural -parameter-

ÿ …† ˆ

  1ÿ þF…†: 2

Thus in -parameters, by the usual transformation rule, the Christoffel symbols are  3 @ @ @2   ÿ …† ‡ i…† ÿ …† ˆ @ @ @2    3 1ÿ @ @ @2  ˆ ‡þ 00 …† : þF…† 2 @ @ @2 Thus  is -¯at if and only if ÿ !    1ÿ @2  @ 2 00 ˆ 0: þF…† ‡ þ …† 2 @2 @ Similarly in  parameters we have  is ÿ-¯at if and only if ÿ !    1‡ @2  @ 2 00 ˆ 0: þF…† ‡ þ …† 2 @2 @

…A:1†

…A:2†

Since  and  are þ-dual we have @ @ ˆ …þ 00 †ÿ1 …†: @ @ Differentiating both sides with respect to  using the chain rule gives  2 @2  @ @ @2  @ @ 1 …† þF…†; ‡ ˆ ÿ þ 00 @2 @ @ @2 @ @ and multiplying through by …þ 00 †2 and using the þ-duality gives

Parameterisations and transformations

 2  2 @2  @ @2  @ 00 þ …† ‡ 2 þ 00 …† ˆ ÿþF…†: @2 @ @ @

315

…A:3†

Substituting (A.3) into (A.2) gives (A.1), and (A.3) into (A.1) gives (A.2) as required. References Amari, S. (1990), Differential-Geometrical Methods in Statistics, 2nd edn, Lecture Notes in Statistics No. 28, Berlin: Springer-Verlag. Barndorff-Nielsen, O.E., D.R. Cox and N. Reid (1986), `The Role of Differential Geometry in Statistical Theory', International Statistical Review, 54: 83±86. Bates, D.M. and D.G. Watts (1980), `Relative Curvature Measures of Nonlinearity', Journal of the Royal Statistical Society, Series B, 40: 1±25. (1981), `Parametric Transforms for Improving Approximate Con®dence Regions in Non-linear Least Squares', Annals of Statistics, 9: 1152±1167. Cox, D.R. and D.V. Hinkley (1974), Theoretical Statistics, London: Chapman & Hall. Critchley, F., P.K. Marriott and M. Salmon (1993), `Preferred Point Geometry and Statistical Manifolds', Annals of Statistics, 21: 1197±1224. (1994), `On the Local Differential Geometry of the Kullback±Leibler Divergence', Annals of Statistics, 22: 1587±1602. Dodson, C.T.J. and T. Poston (1977), Tensor Geometry, London: Pitman. Firth, D. (1993), `Bias Reduction of Maximum Likelihood Estimates', Biometrika, 80: 27±38. Hougaard, P. (1982), `Parametrisations of Nonlinear Models', Journal of the Royal Statistical Society, Series B, 44: 244±252. Kass, R.E. (1984), `Canonical Parametrisation and Zero Parameter Effects Curvature', Journal of the Royal Statistical Society, Series B, 46: 86±92. (1987), `Introduction', in S.I. Amari, O.E. Barndorff-Nielsen, R.E. Kass, S.L. Lauritzen and C.R. Rao (eds.), Differential Geometry in Statistical Inference, Hayward, Calif.: Institute of Mathematical Statistics. (1989), `The Geometry of Asymptotic Inference', Statistical Sciences, 4: 188± 234. McCullagh, P. and J.A. Nelder (1989), Generalised Linear Models, 2nd edn, London: Chapman & Hall. Murray, M.K. and J.W. Rice (1993), Differential Geometry and Statistics, London: Chapman & Hall.

Index

-divergence, see under divergences acceptance region, 230, 232 af®ne approximation, 24, 28, 59 af®ne connections, 39±40, 188, 218±219, 250, 294 non-metric, 298 symmetric, 40 af®ne embedding, 59 af®ne function, 20 af®ne parameterisation, 48, 51±52, 295 af®ne space, 16±19, 24, 50 curved subspace of, 52 af®ne structure, 19, 40 af®ne subspace, 9, 12, 39±40, 51 Aitken GLS estimator, 179 alternatives, explosive, stationary, 269±270, 272±273 Amari's duality, see duality Amari's expected geometry, 5, 294, 297±298, 304±305 Amari's projection theorem, 49 ancillarity, role of, 1, 294 ancillary statistic, see under statistic arti®cial regression, 4, 182 asymptotic analysis, 281, 294 converse, 307 covariance matrix, 113, 167, 171 critical value, 248 distribution, 112, 116 equivalence, 156, 307 316

expansion, 220, 283 information matrix, 166 theory, 29, 152, 154 variance, 112, 119 autocorrelation, 265 balance equations, 217 Bartlett kernel, 124, 126 basis formulae, 27±28 Bernoulli principle, 207 Cartesian coordinates, 187, 299 Christoffel symbols, 41±45, 46, 188±189, 218, 225, 314 COMFAC model, see under model conditional resolution, 58±59 connection, 41±43, 298 Amari's connections, 220, 224±225, 299 geometric af®ne, 40, 224 large family of, 46 metric, 299, 305 non-metric, 43 Levi±Civita, 43±45, 47±48, 188, 250±251, 298 Riemannian, 252 ÿ1-connection, 45, 48, 52., 307 0-connection, 45, 48, 52, 306, 309 ‡1-connection, 44, 52, 58, 305±306 1/3-connection, 52, 309, 312 -connection, 46, 48, 252, 295, 299 cotangent bundle, 29

Index cotangent ®eld, 29 cotangent vector, 34, 37 covariance, 298 covariant derivatives, 224, 225 Cramer±Rao theorem, 58, 186 critical points, 195, 248, 253 critical region (CR), 195, 230±234, 237, 239, 241, 243, 247, 249, 264±265, 269, 276±277 best (BCR), 234, 236±238, 241±242, 254, 263, 265, 277 critical value, 248, 269, 277 curvature, 40, 47, 50, 56, 188, 230±232, 242±244, 283±284 constant negative sectional, 190 Efron's statistical, 58 elliptic, 189±190 embedding, 54, 58, 60, 223 Euler±Schouten, 190 Gaussian, 189 geodesic, 298 global, 230±233, 254, 257, 265, 269, 272±273, 278 hyperbolic, 184, 189±190 intrinsic, 50, 52, 55 local, 233 model, 233 parabolic, 189 parameter-dependent, 50 sectional, 189, 190 statistical, see statistical curvature `straight', 298 curved boundary, 233, 239 curved statistical spaces, 1 differential geometry, 1, 7, 86, 184. 294, 295 differential operator, 25, 27, 110 directional derivative, 25, 40 distribution, 214 bivariate normal, 256 exponential, 200 families of, 295 ®xed, 295 Gaussian, 190, 192, 196, 208

317 half-normal, 200, 201 income, 206 induced, 295 inef®ciency, 201, 209 limiting chi-squared, 131±133 log-normal, 206 multivariate, 190, 296 normal, 190, 199, 206, 209 true, 294 truncated normal, 200, 201 distribution functions, 19, 127 divergences, 217±220 -divergences, 225±229 average, 226 dual spaces, see under space duality, 5, 7, 47, 48, 307, 308, 309, 311 duality theorem for statistical manifolds, 48

earnings frontiers, 185, 196, 200±201, 204±205, 210 Edgeworth expansions, 283 ef®ciency, 163, 173, 181±182 Efron curvature, 58, 242, 262, 284, 291±292 Einstein convention, 31, 33 Einstein summation, 8, 9, 27 elementary zero function, 171±175, 177, 178 bounded, 176 embedding metric, 31, 36 empirical likelihood (EL), 119±120 ratio test, 130 encompassing, 73, 76 analysis, 79 automatic, 65, 77, 80 mutually, 74 parsimonious, 65, 73±74 properties, 74 relationships, 73, 75, 76, 79 general conditions, 73 enveloping region (ER), 242, 264, 269, 271, 277 equality constraints, testing 281

318

Index

estimation, in econometrics, 119 one-step ef®cient procedures, 182 positive semi-de®nite consistent, 122 semi-parametric, 119 techniques, 151 estimators, 54±55, 152, 171, 181 Aitken GLS, 179 consistent, 55, 161, 170, 175±176 ef®cient, 152, 170, 174, 178 ef®cient maximum empirical likelihood (EMELE), 125±26, 130±131, 135±136 empirical likelihood (EL), 121 empirical likelihood cumulative distribution function (ELCDF), 127 generalised method of moments (GMM), see generalised method of moments estimator inef®cient, 181 instrumental variables (IV), 179 Lagrange multiplier, see Lagrange multiplier maximum likelihood, see maximum likelihood estimator maximum empirical likelihood (MELE), 120, 123 of model parameters, 181 one-step, 182 ordinary least squares (OLS), 50, 162, 168, 171, 178±181 positive semi-de®nite consistent, 126, 132, 133 residuals augmented least squares (RALS), 181, 182 root-n consistent, 4, 153, 161, 163, 165±166, 181 saddle-point, 123 Euclidian distance, 259 Euclidian geometry, 9, 50, 51 Euclidian inner product, 31 Euclidian space, 16, 191, 250 open set, 17 subspaces, 17, 78

exogeneity, role of, 1 exogenous variables, 21 expectations, 172, 175, 179, 266, 295 unconditional, 173, 178 expected geometry, 39, 47, 300 expected Fisher information matrix, see Fisher information expected parameters, 11 families, 8, 9 ARMA, 14 Bernoulli, 9, 300, 309 binomial, 9, 300 constant covariance multivariate normal, 308, 309 continuous, 301 curved exponential, 7, 8, 12±15, 17, 20±21, 36, 39, 55, 57±59, 222, 232, 299 exponential, 21±22, 49, 234, 236, 300, 306 full exponential, 7±12, 20±21, 35±36, 38±39, 46, 48, 52, 56±57, 59±60, 294±295, 299 gamma, 9, 300 geometric structure of, 22 linear exponential, 77 multinomial, 9 non-exponential, 15, 232 normal, 9, 300 one-dimensional, 56, 299, 306±307 one-parameter, 157, 236 one-parametric exponential, 188, 237 p-dimensional parametric, 295 parametric, 8, 12, 15, 28, 44, 46 parametric statistical, 16 Poisson, 9, 300, 309, 312 regular, 9 two-parameter, 188, 211 Fisher information, 23, 35±39, 44±45, 52, 57±58, 60, 87, 112, 224±245, 250, 252, 267, 273, 297, 306±307 ¯atness, 40, 47±48, 306 form,

Index

319

bilinear, 30 component, 31 quadratic, 30 function, average, 184, 200, 201, 202 contrast, 218 distribution, 290 estimating, 181 expected likelihood, 301 exponential mean, 117 frontier, 184, 199, 202, 209 joint distribution, 286 likelihood, 296 log-likelihood, 289, 296 loss, 215 mean, 117 method of estimating, 152 non-decreasing af®ne, 301 non-exponential, 117 non-linear, 175±176 zero, 179 observed likelihood, 297, 301 observed score, 301, 304±305 power, 249, 281±284, 286, 291 probability, 295 production, 199 score, 194, 296, 300±301, 305±306 standard normal distribution, 283 stochastic, 39

geometry of transformations, 5 German socio-economic panel (SOEP), 197 global coordinate chart, 89 Godambe's estimating functions, 153, 171

Gauss±Markov theorem, 178 generalised linear models, 12 generalised method of moments estimator (GMM), 119±121, 123, 125, 130, 133±134, 136, 144, 152, 171, 178, 181 geodesic distance, see Rao distance geodesic orbits, 251 geodesic test, see under tests geodesics, 32, 40±41, 43, 188, 190, 299 geometric af®ne connections, 40 geometric theory, 8, 23±24 of connections, 40

identi®ability, 174 identi®cation condition, 173 inef®ciency, 205, 209±210 inequality restriction, 70±71, 101 inference, conditional, 39 in curved families, 50, 52 encompassing in, 1 likelihood based, 1, 22 method-of-moment based, 1 statistical, see statistical inference information criterion, 66 information loss, 56 information recovery, 58

Hellinger distance, 219 Hessian, 282±283, 285, 288 hetero-, homoskedasticity, 178±179, 240 higher-order asymptotic expansions, 1 Hilbert manifold, 151±153, 156, 159 Hilbert space, 65, 69, 74, 79±80, 151±157, 159±160, 162, 166, 172±173 ef®ciency and robustness in, 160±163 in®nite-dimensional methods, 151, 154 Small and McLeish's, 152 Hilbert submanifold, 152 human capital, 184, 196, 199±201, 203, 205, 210 theory, 196, 199 hyperbolic geometry, 190, 208 hypotheses, nested, 134 parametric null, 134 seniority, 204 hypothesis testing, 230±231, 277

320

Index

information theoretic approach, 120 insurance strategy, 185, 205±206, 208±209, 211 intersection points, 304 intersection region (IR), 265, 269, 270, 271 invariance, 35 invariant de®nition of angle, length, 37 isocircles, 191±192, 206, 208, 210±211 isometry, 188 isomorphism, 34, 81 Jeffreys distance, 273 Kullback±Leibler distance, 259, 273 Kullback±Leibler divergence, 49, 219 Kullback±Leibler information, 49, 182, 258 kurtosis, non-normal, 181 Lagrange multiplier, 5, 122±123, 129, 131, 135±137 Lebesgue measure, 86 Legendre transform, 237 Levi±Civita connection, see under connection likelihood function, semi-parametric 122 likelihood ratio, 281, 290 limiting distribution theory, 79, 285 limiting likelihood ratio, 285, 288 Mahalanobis distance, 184 manifold, 7, 8, 16, 27, 30, 34, 38, 44, 48, 85, 88±89, 218, 250, 295 -¯at, 299 canonical, 239, 267 curved, 24, 29, 188, 252, 298 d-dimensional, 235 direction in, 25 embedded, 16±17, 20, 21, 277 enlarged, 223 expectation, 239, 241, 266±268, 270 expected geometric statistical, 52 extended, 223

®nite-dimensional differential, 151 ¯at, 47 Hilbert, see Hilbert manifold in®nite-dimensional, 152 (n±r)-dimensional, 18 r-dimensional, 24 smooth, 25 statistical, see statistical manifold sub-, 17±18, 20, 160, 172 totally ¯at, 307±308 mapping, 74±76, 81 parameter-de®ning, 160, 163, 170±172 martingale, condition, 177 difference process, 123 property, 158±159, 164, 174 triangular array, 158 maximum likelihood estimator (MLE), 22, 49±50, 56, 61, 85±86, 93, 117, 119, 122, 160, 214, 220, 222, 227, 253, 255, 263, 266, 283, 298, 300, 305, 307±308, 313 method of steepest descent, 201 metric, 31 Fisher, 219, 224 invariance properties of, 32 observed information, 38±39 PoincareÂ, 190 preferred point, 37±39 metric tensor, 30, 32, 35, 38±40, 44±45, 218, 298, 306 Fisher information, 298 Riemannian, 187 minimal suf®ciency, 235 minimization problem, 66 minimum suf®cient statistic, 39, 236 misspeci®cation tests, 120 MoÈbius distance, 190 model, COMFAC, 14±15, 70 conditional, 75 congruent, 64 constrained, 170 curvature properties of, 277, 278

Index curved, 238 curved exponential, 232, 235 encompassing, 64 exponential, 235, 245, 312 exponential regression, 85, 117 ®rst-order autoregressive (AR(1)), 20, 281 ®rst-order moving average (MA(1)), 281, 287±288 frontier, 196±197, 201, 204±205, 210 general, 65, 79 human capital, 196±197, 201, 204±205, 210 linear, 10, 21, 50, 301 linear exponential, 262 linear regression, 171±172, 178 linear simultaneous equation, 9 misspeci®ed, 49 mutually encompassing, 64 nested, 66, 70, 73, 77, 80 non-linear regression, 50 non-nested, 65±67, 71, 73 one-parameter linear exponential, 232, 236±238, 263 ordinary least squares (OLS), 204±205 parameterised, 152, 160, 168, 173±174, 176, 276 parametric probability, 65 parametric statistical, 281 parsimonious encompassing, 64±65, 73 PoincareÂ, 190 regression, 179 regular parametric, 214, 216±217 restricted, 65±66 seemingly unrelated regressions, 232 simple, 80 simultaneous equations, 179 speci®cation, 32 sphere, 17 statistical, 186 stochastic frontier, 199 time-series, 232 model size, 169

321 modelling strategy, general-to-simple, 73, 80 moment conditions, 119, 123±124, 128, 130±132, 136 test for over-identifying, 130 moment indicators, 124 moments, 19, 130, 172 multivariate calculus, 27 nesting, 65, 69±70, 72 Neyman±Pearson Lemma (NPL), 232, 234, 236, 238, 282 Neyman±Pearson theory, 282, 285 non-Riemannian geometry, 190, 252 non-stationary time-series analysis, 281 normal errors, 169, 171, 179 normal quantile scale, 26, 288, 291 numerical integration, 113±114 observed Fisher information, see Fisher information observed geometry, 39, 47 observed information metric, see under metric optimal instruments, 178 optimising behaviour, theory of, 199 Ornstein±Uhlenbeck process, 285 orthogonal projection, 1, 65, 69, 74, 77±80, 82 over-identifying moment conditions, 127 p*-formula, 60±61 parameter, -af®ne, 299 canonical, 234, 238 expectation, 256 explicit, 181 mean value, 301 natural, 312±313 nuisance, 37, 170, 281 parameter effects, 50, 52, 55 parameter orthogonality, 37, 67, 69, 77 parameter space, 9, 20, 76 canonical, 254, 256

322

Index

parameter space (cont.) ®nite-dimensional, 160 natural r-dimensional, 12 normal likelihood, 311 restricted, unrestricted, 67, 70 vector, 160 parameter values, 277 parameterisation, 11, 25, 27, 50±51, 160, 218, 223, 241, 297, 299, 304, 309 alternative, 304 choice of, 7, 295, 308 covariance stabilising, 52 directed likelihood, 52 in econometric models, 5 expectation, 51 formulae, 26±27 general, 304±305 geodesic normal, 51±52 invariance to, 1 mean value, 300, 305 natural, 46, 51, 301, 304±305, 308 non-af®nely equivalent, 301 quadratic log-likelihood, 52 zero asymptotic skewness, 52 parametric families, see under families parametric restrictions, 134 Pitman drift, 156 Poisson regression model, 11, 13 power, 262, 281 power curve, 283, 287±288, 291±292 power envelope, 231, 234, 240, 242, 264, 282±286, 288, 291±292 power formulae, 284 power properties, 233, 262, 288, 291 prediction, goodness of, 223 prediction intervals, 216 prediction limits, 215 predictor, optimal, 215 point value, 215 preferred point geometry, 37, 38, 295 projection, 42, 77±78, 175 projection theorem, 48 pseudo-true value (PTV), 137

Pythagoras, theorem of, 7, 49 Radon±Nikodym theorem, 154 random walk hypothesis, 271 Rao distance, 48, 184, 186±187, 190±191, 194±195, 205±206, 208±211, 231, 250, 277 theory, derivation, 185 tests, 194±196, 206, 210±211 recursive formulae, 92, 109 regularity conditions, 174 reparameterisation, 29, 70, 297±298, 313 isomorphic, 67 non-af®ne, 305 non-linear, 10 repeated index convention, 216 repeated samples, 21 Riemannian curvature tensor, 47, 51 Riemannian distance, see Rao distance Riemannian geometry, 184, 188 Riemannian metric tensor, 188 Riemannian tensor, 189 robustness, 162±163, 168±169, 173, 181±182 Roy's union-intersection method, 277 saddle-point method, 60, 140 sample size effect, 312±313 scalar product, 224±225 score vector, 22±23, 28, 37±38, 132, 296±297 serial correlation, 269, 272±273 sigma-algebras, 178 simultaneous equation model, 10 size condition, 237 skewness, 115 smoothed log EL criterion, 124, 128±129, 132, 134, 137 smoothed moment indicators, 123 space, af®ne, see af®ne space auxiliary, 54±55 canonical parameter, 256 curved sub-, 20

Index dual, 26, 28, 34 embedding, 26, 31, 40, 44, 252 Euclidian, see Euclidian space Hilbert, see Hilbert space isomorphic, 26 linear, 74 parameter, 17 tangent, see tangent space vector, 16±17, 26, 34, speci®cation tests, 134 standard econometric model, geometry of, 16 standard regularity conditions, 304 standard problems, second-order theory for, 291 standard second-order theory, 292 statistic, 7 ancillary, 23, 59, 214, 217 asymptotically normal suf®cient, 283 canonical, 9, 273 Cox-type, 138±139 empirical likelihood (EL) minimum chi-squared, 133, 136, 145 empirical likelihood ratio (ELR), 121, 129, 131±132, 135, 145 empirical likelihood (EL) score, 130±133, 135±136 empirical likelihood (EL) Wald , 129±133, 135±136, 145 general, 50 linearised, 139 minimal suf®cient, 232 minimum chi-squared, 136 predictive suf®cient, 214 score, 23, 24, 129, 133, 136, 281, 290 suf®cient, see suf®cient statistic Wald, 281 statistical curvature, 1, 4, 57±58, 190, 230, 281 statistical inference, 151, 230 geometrical basis of, 1 statistical manifold, 7, 19, 24, 28±29, 44, 46±47, 50, 151, 251 geometrical structure of, 30, 48 mathematical theorems of, 47

323 statistical metrics, 35 statistical noise, 200 statistical theory, 1 geometric aspects of, 21 stochastic differential equation, 285 stochastic limits, 285 straightness, 40, 43, 238 subfamilies, curved, 12 submersion, 161, 174 subspace, 161, ef®cient, 181 ®nite-dimensional ef®cient, 181 suf®ciency, 294 suf®cient conditions, 77 suf®cient statistic, 12, 59±60, 232, 290 r-dimensional, 12 symmetry, 49, 114 tangent, information, 181 zero, 156 tangent bundle, 29 tangent directions, 26 tangent ®eld, 29, 41±42, 53 tangent space, 22, 24±25, 26, 27±28, 30±31, 34, 37±38, 39, 42, 53, 152, 155±156, 159, 168, 181, 225, 296, 304 1-representation of, 186 tangent vector, 24±30, 40, 42, 298 conditional covariance of, 39 Taylor expansion, 23, 29, 141±144, 146±148, 174, 246 Taylor's theorem, 29 tensor, 32, 37 analysis, 298 calculus, 187 components of, 33 contravariant, 33±35 covariant, 33±34 metric, see metric tensor Riemann curvature, see Riemannian curvature tensor rule, 41 skewness, 46

324

Index

tensor (cont.) 1-tensors, 297 2-tensors, 298 terms, 103±104, 106, 109 tests, ÿ-optimal, 241 Dickey±Fuller, 287 ®rst-order ef®cient, 231 geodesic, 5, 43, 230±231, 233, 239, 248±249, 251, 253, 266. 268±269, 273±274, 276 Lagrange multiplier (LM), 230±231, 233, 239±240, 245, 264, 270, 276 likelihood ratio (LR), 230±231, 233, 239±240, 254±255, 263, 270, 313 failure of, 264, 266, 276 locally best (LB), 242, 244±245 locally most powerful (LMP), 242±245, 248, 254, 264 maximum likelihood ratio (MLR), 283 Neyman±Pearson (NP), 286±288, 291 one-sided parameter, 281 optimal properties of, 276 point optimal (PO), 230, 231, 233, 238, 239±241, 243±244, 255, 264, 268±270, 276 score, 37, 132, 245, 292, 312 size of, 262 uniformly most powerful (UMP), 232, 234, 236±238, 241, 243, 264

Wald, 5, 129, 230±231, 233, 238±241, 245±249, 251, 253, 264, 266, 268, 270, 273±274, 283, 312 total ¯atness, 308±309 transformation, 177, 190 rule, 31, 32, 33 2-tensor, 39 triangle inequality, 49 triangular restriction, 72 true expected score function, 311 true parameter value, 289 unit roots, 292 non-standard, 284 testing for 265, 270, 272, 281 vector, coordinate, 188 cotangent, see cotangent vector of coef®cients, 91 difference, 260±262 ®bre bundle, 225 of tangents, 173 tangent, see tangent vector unique normal, 237 yield±risk analysis, 207 yokes, normalized, 218 theory of, 39 þ-duality, 5, 311, 314