Functional Estimation for Density, Regression Models and Processes

Functional Estimation for Density, Regression Models and Processes

8124tp.indd 1

2/11/11 9:29 AM

This page intentionally left blank

Functional Estimation for Density, Regression Models and Processes

Odile Pons INRA, France

World Scientific NEW JERSEY

8124tp.indd 2

•

LONDON

•

SINGAPORE

•

BEIJING

•

SHANGHAI

•

HONG KONG

•

TA I P E I

•

CHENNAI

2/11/11 9:29 AM

Published by World Scientific Publishing Co. Pte. Ltd. 5 Toh Tuck Link, Singapore 596224 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library.

FUNCTIONAL ESTIMATION FOR DENSITY, REGRESSION MODELS AND PROCESSES Copyright © 2011 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher.

For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher.

ISBN-13 978-981-4343-73-2 ISBN-10 981-4343-73-0

Printed in Singapore.

LaiFun - Functional Estimation for Density.pmd 1

2/1/2011, 9:49 AM

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Preface

Nonparametric estimators have been intensively used for the statistical analysis of independent or dependent sequences of random variables and for samples of continuous or discrete processes. The optimization of the procedures is based on the choice of a bandwidth that minimizes an estimation error for functionals of their probability distributions. This book presents new mathematical results about statistical methods for the density and regression functions, widely presented in the mathematical literature. There is no doubt that its origin benefits from earlier publications and from other subjects I worked about in other models for processes. Some questions of great interest for optimizing the methods have motivated much work some years ago, they are mentioned in the introduction and they give rise to new developments of this book. The methods are generalized to estimators with kernel sequences varying on the sample space and to adaptative procedures for estimating the optimal local bandwidth of each model. More complex models are defined by several nonparametric functions or by vector parameters and nonparametric functions, such as the models for the intensity of point processes and the single-index regression models. New estimators are defined and their convergence rates are compared. Odile M.-T. Pons

This page intentionally left blank

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Contents

Preface

v

1.

1

Introduction 1.1 1.2 1.3 1.4

2.

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

Kernel estimator of a density 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11

3.

Estimation of a density . . . . . . . . Estimation of a regression curve . . . Estimation of functionals of processes Content of the book . . . . . . . . . .

23

Introduction . . . . . . . . . . . . . . . . . . . . . . . . Risks and optimal bandwidths for the kernel estimator Weak convergence . . . . . . . . . . . . . . . . . . . . Minimax and histogram estimators . . . . . . . . . . . Estimation of functionals of a density . . . . . . . . . Density of absolutely continuous distributions . . . . . Hellinger distance between a density and its estimator Estimation of the density under right-censoring . . . . Estimation of the density of left-censored variables . . Kernel estimator for the density of a process . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

Kernel estimator of a regression function 3.1 3.2 3.3 3.4

2 10 14 19

Introduction and notation . . . . . Risks and convergence rates for the Optimal bandwidths . . . . . . . . Weak convergence of the estimator vii

23 25 29 33 34 37 39 40 42 44 46 49

. . . . . . estimator . . . . . . . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

49 50 56 60

January 31, 2011

17:17

viii

. . . . . . .

Introduction . . . . . . . . . . . . . Estimation of densities . . . . . . . Estimation of regression functions Estimation for processes . . . . . . Exercises . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

Introduction . . . . . . . . . . . . . . . . . . . . . . Asymptotics for the quantile processes . . . . . . . Bandwidth selection . . . . . . . . . . . . . . . . . Estimation of the conditional density of Y given X Estimation of conditional quantiles for processes . Inverse of a regression function . . . . . . . . . . . Quantile function of right-censored variables . . . . Conditional quantiles with variable bandwidth . . Exercises . . . . . . . . . . . . . . . . . . . . . . .

6.3

6.4 6.5 6.6 6.7

75 76 81 84 85 87

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

Nonparametric estimation of intensities for stochastic processes 6.1 6.2

62 64 68 69 70 71 73 75

Nonparametric estimation of quantiles 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9

6.

Estimation of a regression curve by local polynomials . . Estimation in regression models with functional variance Estimation of the mode of a regression function . . . . . Estimation of a regression function under censoring . . . Proportional odds model . . . . . . . . . . . . . . . . . . Estimation for the regression function of processes . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . .

Limits for the varying bandwidths estimators 4.1 4.2 4.3 4.4 4.5

5.

FunctionalEstimation

Functional estimation for density, regression models and processes

3.5 3.6 3.7 3.8 3.9 3.10 3.11 4.

World Scientific Book - 9in x 6in

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . Risks and convergences for estimators of the intensity . 6.2.1 Kernel estimator of the intensity . . . . . . . . . 6.2.2 Histogram estimator of the intensity . . . . . . . Risks and convergences for multiplicative intensities . . 6.3.1 Models with nonparametric regression functions 6.3.2 Models with parametric regression functions . . Histograms for intensity and regression functions . . . . Estimation of the density of duration excess . . . . . . . Estimators for processes on increasing intervals . . . . . Models with varying intensity or regression coefficients .

87 89 95 98 100 102 104 105 106 107

. . . . . . . . . . .

107 110 111 116 118 119 120 124 126 130 132

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

ix

Contents

6.8 6.9 7.

Estimation in semi-parametric regression models 7.1 7.2 7.3 7.4

8.

137

Introduction . . . . . . . . . . . . . . . . . . . . . . . Convergence of the estimators . . . . . . . . . . . . . Nonparametric regression with a change of variables Exercises . . . . . . . . . . . . . . . . . . . . . . . .

. . . .

. . . .

. . . .

Diffusion processes 8.1 8.2 8.3 8.4 8.5 8.6 8.7

9.

Progressive censoring of a random time sequence . . . . . 135 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

147

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . Estimation for continuous diffusions by discretization . . Estimation for continuous diffusion processes . . . . . . Estimation of discretely observed diffusions with jumps Continuous estimation for diffusions with jumps . . . . . Transformations of a non-stationary Gaussian process . Exercises . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . .

Applications to time series 9.1 9.2 9.3 9.4 9.5 9.6

Nonparametric estimation of the mean . . . . . . . . Periodic models for time series . . . . . . . . . . . . Nonparametric estimation of the covariance function Nonparametric transformations for stationarity . . . Change-points in time series . . . . . . . . . . . . . . Exercises . . . . . . . . . . . . . . . . . . . . . . . .

Appendix Appendix Appendix Appendix

147 149 154 158 162 164 166 167

. . . . . .

. . . . . .

. . . . . .

10. Appendix 10.1 10.2 10.3 10.4

137 139 143 146

168 171 172 174 174 181 183

A B C D

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

183 184 184 187

Notations

189

Bibliography

191

Index

197

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Chapter 1

Introduction

The aim of this book is to present in the same approach estimators for functions defining probability models: density, intensity of point processes, regression curves and diffusion processes. The observations may be continuous for processes or discretized for samples of densites, regressions and time series, with sequential observations over time. The regular sampling scheme of the time series is not common in regression models where stochastic explanatory variables X are recorded together with a response variable Y according to a random sampling of independent and identically distributed observations (Xi , Yi )i≤n . The discretization of a continuous diffusion process yields a regression model and the approximation error can be made sufficiently small to extend the estimators of the regression model to the drift and variance functions of a diffusion process. The functions defining the probability models are not specified by parameters and they are estimated in functional spaces. This chapter is a review of well known estimators for density and regression functions and a presentation of models for continuous or discrete processes where nonparametric estimators are defined. On a probability space (Ω, A, P ), let X be a random variable with distribution function F (x) = Pr(X ≤ x) and Lebesgue density f , the derivative of F . The empirical distribution function and the histogram are the simplest estimators of a distribution function and a density, respectively. With a sample (Xi )i≤n of the variable X, the distribution function F (x) is estimated by Fbn (x), the proportion of observations smaller than x, which converges uniformly to F in probability and almost surely if and only F is continuous. A histogram with bandwidth hn consists in a partition of the range of the observations into disjoint subintervals of length hn where the 1

January 31, 2011

17:17

2

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

density is estimated by the proportion of observations Xi in each subintervals, divided by hn . The bandwidth hn tends to zero as n tends to infinity and nh2n tends to infinity, thus the size of the partition tends to infinity with the sample size. For a variable X defined in a metric space (X, A, µ), the histogram is the local nonparametric estimator defined by a set of neighbourhoods Vh = {Vh (x), x ∈ X}, with Vh (x) = {s; d(x, s) ≤ h} for the metric d of (X, A, µ) Z n X fbn,h (x) = (n dFX )−1 1{Xi ∈Vh (x)} . (1.1) Vh (x)

i=1

The empirical distribution function and the histogram are stepwise estimators and smooth estimators have been later defined for regular functions. 1.1

Estimation of a density

Several kinds of smooth methods have been developed. The first one was the projection of functions onto regular and orthonormal bases of functions (φk )k≥0 . The density of the observations is approximated by a countable P Kn projection on the basis fn (x) = i=1 ak φk (x) where Kn tends to infinity and the coefficients are defined by the scalar product specific to the orthonormality of the basis with Z Z 2 φk (x)µφ (x) dx = 1, φk (x)φl (x)µφ (x) dx = 0, for all k 6= l, R then ak =< f, φk >= f (x)φk (x)µφ (x) dx. The coefficients are estimated by integrating the basis with respect to the empirical distribution of the variable X Z b akn = φk (x)µφ (x) dFbn (x)

P Kn b akn φk (x). The same which yields an estimator of the density fbn (x) = i=1 principle applies to other stepwise estimators of functions. Well known bases of L2 -orthogonal functions are (i) Legendre’s polynomials1 defined on the interval [−1, 1] as solutions of the differential equations 00

(1 − x2 )Pn (x) − 2x Pn0 (x) − n(n + 1)Pn (x) = 0, 1 French

mathematician (1752-1833)

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Introduction

FunctionalEstimation

3

with Pn (1) = 1. Their solutions have an integral form attributed to Hermite and his student Stieltjes Z 2 π sin(n + 21 )φ dφ Pn (cos θ) = . π 0 {2 cos θ − 2 cos φ} The polynom Pn (x) has also been expressed as the coefficient of z −(n+1) in the expansion of (z 2 − 2xz + 1)−1/2 by Stieltjes (1890). They are orthogonal with the scalar product Z 1 < f, g >= f (x)g(x) dx; −1

2

(ii) Hermite’s polynomials of degree n defined by the derivatives Hn (x) = (−1)n ex

2

/2

dn −x2 /2 (e ), n ≥ 1, dxn

they satisfy the recurrence equation Hn+1 (x) = xHn (x) − Hn0 (x), with H0 (x) = 1. They are orthogonal with the scalar product Z +∞ 2 < f, g >= f (x)g(x)e−x dx −∞

√ and their norm is kHn k = n! 2π; (iii) Laguerre’s polynomials3 defined by the derivatives Ln (x) =

ex dn −x n (e x ), n ≥ 1, n! dxn

and L0 (x) = 1. They satisfy the recurrence equation Ln+1 (x) = (2n + 1 − x)Ln (x) − n2 Ln−1 (x) and they are orthogonal with the scalar product Z +∞ < f, g >= f (x)g(x)e−2x dx. −∞

The orthogonal polynomials are normalized by their norm. If the function f is Lipschitz, the polynomial approximations converge to f in L2 and for the pointwise convergence. The corresponding projection estimators also converge in L2 and pointwisely. Though the bases generate functional spaces of smooth integrable functions, the estimation is parametric. The estimator of the approximation function converge to zero in L2 with the 2 French 3 French

mathematician (1822-1901) mathematician (1834-1886)

January 31, 2011

4

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

norm kfbn − fn k2 = { zero, so that

R +∞ −∞

kfbn − fn k22 = =

E(fbn − fn )2 (x)µφ (x) dx}1/2 if n−1 Kn tends to

Z

+∞

E

−∞

Kn X i=1

2

E(b akn − ak ) = E{

Kn X i=1

(b akn − ak )2 φ2k (x)µφ (x) dx

E(b akn − ak )2

Z

= n−1

Z

φk (x)µφ (x) d(Fbn − F )(x)}2

φk (x)φk (y)µφ (x)µφ (y) dC(x, y)

where C(x, y) = F (x ∧ y) − F (x)F (y) is the covariance function of the empirical process n1/2 (Fbn − F ). The convergence rate of the norm the 1/2 density estimator is the sum of the norm kfbn − fn k2 = O(n−1/2 Kn ) and P∞ the approximation error kfn − f k2 = ( i=Kn +1 a2k )1/2 , it is determined by the convergence rate of the sum of the squared coefficients and therefore by the degree of derivability of the function f . Splines are also bases of functions constrained at fixed points or by a condition of derivability of the function f , with an order of integration for its higher derivative. They have been introduced by Whittaker (1923) and developed by Schoenberg (1964), Wold (1975), Wahba and Wold (1975), De Boor (1978), Wahba (1978), Eubank (1988). They allow the approximation of functions having different degrees of smoothness on different intervals which can be fixed. A comparison between splines and kernel estimators of densities may be found in Silverman (1984) who established an uniform asymptotic bound for the difference between the kernel function and the weight function of cubic splines, with a bandwidth kernel λ−1/4 where λ is the smoothing parameters of the splines. Each spline operator corresponds to a kernel operator and the bias and variance of both estimators have the same rate of convergence (Rice and Rosenblatt, 1983, Silverman, 1984). Messer (1991) provides an explicit expression of the kernel corresponding to a cubic sinuso¨ıdal spline, with their rates of convergence. Kernel estimators of densities have first been introduced and studied by Rosenblatt (1956), Whittle (1958), Parzen (1962), Watson and Laedbetter (1963), Bickel and Rosenblatt (1973). Consider a real random variable X defined on (Ω, A, P ) with density fX and distribution function FX . A continuous density fX is estimated by smoothing the empirical distribution

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Introduction

FunctionalEstimation

5

function FbX,n of a sample (Xi )1≤i≤n distributed as X by the means of its convolution with a kernel K, over a bandwidth h = hn tending to zero as n tends to infinity Z n 1 X x − Xi fbX,n,h (x) = Kh (x − s) dFbX,n (s) = K( ), (1.2) nh i=1 h

where Kh (x) = h−1 K(h−1 x) is the kernel of bandwidth h. The weighting kernel is a bounded symmetric density satisfying regularity properties and moment conditions. With a p-variate vector X, the kernel may be defined −1 on Rp and Kh (x) = (h1 . . . , hp )−1 K(h−1 1 x1 , . . . , hp xp ), for p-dimensional vectors x = (x1 , . . . , xp ) and h = (h1 , . . . , hp ). Scott (1992) gives a detailed presentation of the multivariate density estimators with graphical visualizations. Another estimator is based on the topology of the space (X, A, µ), with (1.1) or using a real function K and Kh (x) = h−1 K(h−1 kxkµ ), h > 0. The regularity of the kernel K entails the continuity of the estimator fbX,n,h . All results established for a real valued variable X apply straightforwardly to a variable defined in a metric space. Deheuvels (1977) presented a review of nonparametric methods of estimation for the density and compared the mean squared error of several kernel estimators including the classical polynomial kernels which do not satisfy the above conditions, some of them diverge and their orders differ from those of the density kernels. Classical kernels are the normal density with support R and densities with a compact support such as the BartlettEpanechnikov kernel with support [−1, 1], K(u) = 0.75(1 − u2 )1{|u|≤1} , other kernels are presented in Parzen (1962), Prakasa Rao (1983), etc. With a sequence hn converging to zero at a convenient rate, the estimator fbX,n,h is biased, with an asymptotically negligible bias depending on the regularity properties of the density. Constants depending on moments of the kernel function also appear in the bias function E fbX,n,h − fX and the k moments E fbX,n,h of the estimated density. The variance does not depend on the class of the density. The weak and strong uniform consistency of the kernel density estimator and its derivatives were proved by Silverman (1978) under derivability conditions for the density. Their performances are measured by several error criteria corresponding to the estimation of the density at a single point or over its whole support. The mean squared error criterion is common for that purpose and it splits into a variance and

January 31, 2011

6

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

the square of a bias term M SE(fbX,n,h ; x, h) = E{fbX,n,h (x) − fX (x)}2 = E{fbX,n,h (x) − E fbX,n,h (x)}2 + {E fbX,n,h (x) − fX (x)}2 .

A global random measure of the distance between the estimator fbX,n,h and the density fX is the integrated squared error (ISE) given by Z ISE(fbX,n,h ; h) = {fbX,n,h (x) − fX (x)}2 dx. (1.3)

A global error criterion is the mean integrated squared error introduced by Rosenblatt (1956) Z b b M ISE(fX,n,h ; h) = E{ISE(fX,n,h ; h)} = M SE(fbX,n,h ; x, h) dx. (1.4)

The first order approximations of the MSE and the MISE as the sample size increases are the AMSE and the AMISE. Let (hn )n be a bandwidth sequence converging to zero and and let K R such that nh tends to infinity R be a kernel satisfying m2K = x2 K(x) dx < ∞ and κ2 = K 2 (x) dx < ∞. Consider a variable X such that EX 2 is finite and the density FX is twice continuously differentiable h4 2 002 m f (x). 4 2K They depend on the bandwidth h of the kernel and the AMSE is minimized at a value R 2  K (x) dx 1/5 hAMSE (x) = fX (x) . nm22K f 002 (x) AM SE(fbX,n,h ); x = (nh)−1 fX (x)κ2 +

The global optimum of the AMISE is attained at R 2 1/5  K (x) dx R . hAMISE = 2 002 nm2K f (x) dx

Then the optimal AMSE tends to zero with the order n−4/5 , it depends on the kernel and on the unknown values at x of the functions fX and f 002 , or their integrals for the integrated error (Silverman, 1986). If the bandwidth has a smaller order, the variance of the estimator is predominant in the expression of the errors and the variations of estimator are larger, if the bandwidth is larger than the optimal value, the bias increases and the variance is reduced. The approximation made by suppressing the higher order terms in the expansions of the bias and the variance of the

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Introduction

FunctionalEstimation

7

density estimator is obviously another source of error in the choice of the bandwidth, Hall and Marron (1987) proved that hMISE /hAMISE tends to 1 and hISE /hAMISE tends to 1 in probability as n tends to infinity. Surveys on kernel density estimators and their risk functions were given by Nadaraya (1989), Rosenblatt (1956, 1971), Prakasa Rao (1983), Hall (1984), H¨ ardle (1991), Khasminskii (1992). The smoothness conditions for the density are sometimes replaced by Lipschitz or H¨older conditions and the expansions for the MSE are replaced by expansions for an upper bound. Parzen (1962) also proved the weak convergence of the mode of a kernel density estimator. The derivatives of the density are naturally estimated by those of the kernel estimator and the weak and strong convergence of derivative estimators have been considered by Bhattacharya (1967) and Schuster (1969) among others. The L1 (R) norm of the difference between the kernel estimator and its expectation converges to zero, as a consequence of the properties of the convolution. RDevroye (1983) studied the consistency of the L1 -norm kfbX,n,h − f k1 = |fbX,n,h − f | dx, Gin´e, Mason and Zaitsev (2003) established the weak convergence of the process n1/2 (kfbX,n,h − E fbX,n,h kR1 − EkfbX,n,h − E fbX,n,hk1 ) to a normal variable with variance depending on K(u)K(u + t) du. Bounds for minimax estimators have been established by Beran (1972). A minimax property of the kernel estimator with the optimal convergence rate n2/5 was proved by Bretagnole and Huber (1981). Though the estimator of a monotone function is monotone with probability tending to 1 as the number of observations tends to infinity, the number of observations is not always large enough to preserve this property and a monotone kernel estimator is built for monotone density functions by isotonisation of the classical kernel estimator. Monotone estimators for a distribution function and a density have been first defined by Grenander (1956) as the least concave minorant of the empirical distribution function and its derivative. This estimator has been studied by Barlow, Bartholomew, Bremmer and Brunk (1972), Kiefer and Wolfowitz (1976), Groeneboom (1989), Groeneboom and Wellner (1997). The isotonisation of the kernel estimator fbn,h for a density function is Z v 1 fbSI,n,h (x) = inf sup fbn,h (t) dt (1.5) v≥x u≤x v − u u R and 1{t≤x} fbSI,n,h (t) dt is the greatest convex minorant of the integrated R estimator 1{t≤x} fbn,h (t) dt. Its convergence rate is n1/3 (van der Vaart and van der Laan, 2003). Groeneboom and Wellner studied the weak con-

January 31, 2011

17:17

8

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

vergence of local increments of the isotonic estimator of the distribution function. The estimation of a convex decreasing and twice continuously differentiable density on R+ by a piecewise linear estimator with knots between observations points was studied by Groeneboom, Jonkbloed and Wellner (2001), the estimator is n2/5 -consistent. Dumbgen and Rufibach (2009) proposed a similar estimator for a log-concave density on R+ and established its convergence rate (n(log n)−1 )β/(2β+1) , for H¨older densities of Hβ,M . Stone (1974), De Boor (1975), Bowman (1983), Marron (1987) introduced automatic data driven methods for the choice of the global bandwidth. They minimize theR integrated random risk ISE or the crossPn 2 validation criterion CV (h) = fbX,n,h (x) dx−2n−1 i=1 fbX,n,h,i (Xi ) where fb2 is the kernel estimator based on the data sample without the i-th X,n,h,i

observation, or the Rempirical version of the Kullback-Leibler loss-function K(fbX,n,h , f ) = −E log fbX,n,h dFX (Bowman, 1983). The CV (h) criterion is an unbiased estimator of the MISE and its minimum is the minimum for the estimated ISE using the empirical distribution function. The global bandwidth estimator b hCV minimizing this estimated criterion achieves the bound for the convergence rate of any optimal bandwidth for the ISE, b hCV /hMISE − 1 = Op (n−1/10 ) and b hCV − hMISE has a normal asymptotic distribution (Hall and Marron, 1987). The cross-validation is more variable with the data and often leads to oversmoothing or undersmoothing (Hall and Marron, 1987, Hall and Johnstone, 1992). As noticed by Hall and Marron (1987), the estimation of the density and the mean squared error are different goals and the best bandwidth for the density might not be the optimal for the MSE, hence the bandwidth minimizing the cross-validation induces variability of the density estimator. Other methods for selecting the bandwidth have been proposed such as higher order kernel estimators of the density (Hall and Marron, 1987, 1990) or bootstrap estimations. An uniform weak convergence of the distribution of fbn,h was proved using consecutive approximations of empirical processes by Bickel and Rosenblatt (1973), other approaches for the convergence in distribution rely on the small variations of moments of the sample-paths, as in Billingsley (1968) for continuous processes. The Hellinger distance h(fbX,n,h , f ) between a density and its estimator has been studied by Van de Geer (1993, 2000), here the weak convergence of the process fbX,n,h −fX provides a more precise convergence rate for h(fbX,n,h , f ). All result are extended to the limiting marginal density of a continuous process under ergodicity and mixing conditions.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Introduction

9

Uniform strong consistency of the kernel density estimator requires stronger conditions, results can be found in Silverman (1978), Singh (1979), Prakaso Rao (1983), H¨ ardle, Janssen and Serfling (1988) for the strong consistency and, for its conditional mode, Ould Sa¨ıd (1997). The law of the iterated logarithm has been studied by Hall (1981), Stute (1982), Bosq (1998) Theorem 1.1 (Stute 1982). Let f be a continuous density strictly positive and bounded on a sub-interval [a, b] of its support. Let (hn )n be a bandwidth sequence converging to zero and such that nhn tends to infinity, −1 log h−1 n = o(nhn ) and log hnR /(log log n) tends to infinity. Suppose that K has a compact support and |dK| < ∞, then for every δ > 0 lim sup{ n

nhn 1/2 }1/2 sup |fbX,n,h (x) − E fbX,n,h (x)|f 1/2 (x) = κ2 , a.s. 2 log h−1 Ih n

with Ih = [a + h, b − h].

A periodic density f on an interval [−T, T ] is analyzed in the frequency domain where it is expanded according to the amplitudes and the frequency or period of its components. Let T = 2π/w, the density f is expressed as P+∞ the limit of series due to Fourier4 , f (x) = k=−∞ ck eiwkx with coefficients R T /2 ck = T −1 −T /2 f (x)e−iwkx dx and the Fourier transform of f is defined R T /2 on R by F f (s) = T −1 −T /2 f (x)e−iwsx dx. The inversion formula of the R +∞ Fourier transform is f (x) = −∞ F f (w)eiwsx ds. For a non periodic density, the Fourier transform and its inverse are defined by Z ∞ −1 F f (s) = (2π) f (x)e−isx dx, f (x) =

Z

−∞

+∞

−∞

F f (w)eisx ds.

The Fourier transform is an isometry as expressed by the equality R R |F f (s)|2 ds = |f (s)|2 ds. Let (Xk )k≤n be a stationary time series with mean zero, the spectral density is defined from the autocorrelation coefficients γk = E(X0 Xk ) by P −iwk S(w) = +∞ and the inverse relationship for the autocorrelak=−∞ R ∞ γk e tions is γk = −∞ S(w)eiwsx dx. The periodogram of the series is defined as Pn Ibn (w) = T −1 | k=1 Xk e−2πikw |2 and Rit is smoothed to yield a regular estimator of the spectral density Sbn (s) = Kh (u−s)Ibn (s) ds. Brillinger (1975) 4 French

mathematician (1768–1830)

January 31, 2011

10

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

established that the optimal convergence rate for the bandwidth is hn = O(n−1/5 ) under regularity conditions and he proved the weak convergence of the process n2/5 (Sbn −S) to the process defined as a transformed Brownian motion. Robinson (1986, 1991) studied the consistency of kernel estimators for auto-regression and density functions and for nonparametric models of time series. Cross-validation for the choice of the bandwidth was also introduced by Wold (1975). For time series, Chiu (1991) proposed a stabilized bandwidth criterion having a relative convergence rate n−1/2 instead of n−1/10 for the cross-validation in density estimation. It is defined from the Fourier transform dY of the observation series (Yi )i , using the periodogram of the series IY = d2Y /(2πn) and the Fourier transform Wh (λ) of the kerP nel.The squared sum of errors is equal to 2π i IY (λj ){1 − Wh (λj )}2 with P n λj = 2πj and Wh (λ) = n−1 j=1 exp(−iλj)Kh (j/n). Multivariate kernel estimators are widely used in the analysis of spatial data, in the comparison and the classification of vectors. 1.2

Estimation of a regression curve

Consider a two-dimensional variable (X, Y ) defined on (Ω, A, P ), with values in R2 . Let fX and fX,Y be the continuous densities of X and, respectively, fXY , and let FX and FXY be their distribution functions. In the nonparametric regression setting, the curve of interest is the relationship between two variables, Y a response variable for a predictor X. A continuous curve is estimated by the means of a kernel estimator smoothing the observations of Y for observations of X in the neighborhood of the predictor value. The conditional mean of Y given X = x is the nonparametric regression function defined for every x inside the support of X by Z fX,Y (x, y) m(x) = E(Y |X = x) = y dy, fX (x) it is continuous when the density fX,Y is continuous with respect to its first component. It defines regression models for Y with fixed or varying noises according to the model for its variance Y = m(X) + σε

(1.6)

where E(ε|X) = 0 and V ar(ε|X) = 1 in a model with a constant variance V ar(Y |X) = σ 2 , or Y = m(X) + σ(X)ε

(1.7)

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

11

Introduction

with a varying conditional variance V ar(Y |X) = σ 2 (X). The regression function of (1.6) is estimated by the integral of Y with respect to a smoothed empirical distribution function of Y given X = x Z Kh (x − s)FbXY,n (ds, dy) m b n,h (x) = y fbX,n,h (x) Pn Yi Kh (x − Xi ) = Pi=1 . n i=1 Kh (x − Xi )

This estimator has been introduced by Watson (1964) and Nadaraya (1964) and detailed presentations can bee found in the monographs by Eubank (1977), Nadaraya (1989) and H¨ardle (1990). The performance of the kernel estimator for the regression curve m is measured by error criteria corresponding to the estimation of the curve at a single point or over its whole support, like for the kernel estimator of a continuous density. A global random measure of the distance between the estimator m b n,h and the regression function m is the integrated squared error (ISE) Z ISE(m b n,h ; h) = {m b n,h (x) − m(x)}2 dx, (1.8) its convergence was studied by Hall (1984), H¨ardle (1990). The mean squared error criterion develops as the sum of the variance and the squared bias of the estimator M SE(m b n,h ; x, h) = E{m b n,h (x) − m(x)}2

= E{m b n,h (x) − E m b n,h (x)}2 + {E m b n,h (x) − m(x)}2 .

A global mean squared error is the mean integrated squared error Z M ISE(m b n,h ; h) = E{ISE(m b n,h ; h)} = M SE(m b n,h ; x, h) dx.

(1.9)

Assuming that the curve is twice continuously differentiable, the mean squared error is approximated by the asymptotic MSE (Chapter 3) −1 AM SE(m b n,h ; x) = (nh)−1 κ2 fX (x) V ar(Y |X = x)

+

h4 2 −1 (2) m f (x){µ(2) (x) − m(x)fX (x)}2 . 4 2K X

The AMSE is minimized at a value hm,AMSE which is still of order n−1/5 and depends on the value at x of the functions defining the model and their second order derivatives. Automatic optimal bandwidth selection by cross-validation was developed by H¨ardle, Hall and Marron (1988) similarly to the density. Bootstrap methods were also widely studied. Splines

January 31, 2011

12

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

were generalized to nonparametric regression by Wahba and Wold (1975), Silverman (1985) for cubic splines and the automatic choice of the degree of smoothing is also determined by cross-validation. In model (1.7) with a random conditional variance V ar(Y |X) = σ 2 (X), the estimator of the regression curve m has to be modified and it is defined as a weighted kernel estimator with weighting function w(x) = σ −1 (x) Pn w(Xi )Yi Kh (x − Xi ) m b w,n,h (x) = Pi=1 n i=1 w(Xi )Kh (x − Xi )

or more general function w. In Chapter 3, a kernel estimator of σ −1 (x) is introduced. The bias and variance of the estimator m b w,n,h are developed by the same expansions as the estimator (1.8). The convergence rate of the kernel estimator for σ 2 (x) is nonparametric and its bias depends on the bandwidths used in its definition, on V ar{(Y − m(x))2 |X = x}, on the functions fX , σ 2 , m and their derivatives. Results about the almost sure convergence and the L2 -errors of kernel estimators, their optimal convergence rates and the optimal bandwidth selection were introduced in Hall (1984), Nadaraya (1964). Properties similar to those of the density are developed here with sequences of bandwidths converging with specified rates. The methods for estimating a density and a regression curve by the means of kernel smoothing have been extensively presented in monographs by Nadaraya (1989), H¨ardle (1990, 1992) Wand and Jones (1995), Simonoff (1996), Bowman and Azalini (1997), among others. In this book, the properties of the estimators are extended with exact expansions, as for density desimation, and to variable bandwidth sequences (hn (x))n≥1 converging with a specified rate. Several monotone kernel estimators for a regression function m have been considered, they are built by kernel smoothing after an isotonisation of the data sample, or by an isotonisation of the classical kernel estimator. The isotonisation of the data consists in a transformation of the observation (Yi )i in a monotone set (Yi∗ )i . It is defined by v 1 X Yi , v≥i u≤i v − u j=u P P and i≤k Yi∗ is the greatest convex minorant of i≤k Yi . The kernel estimator for the regression function built with the isotonic sample (Xi∗ , Yi∗ )i is denoted m b IS,n,h . The convergence rate of the isotonic estimator for a monotone density function is n1/3 and the variable n−1/3 (m b IS,n,h − mIS,n,h )(x)

Yi∗ = min max

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Introduction

FunctionalEstimation

13

converges to a Gaussian process for every x in IX . The isotonisation of the kernel estimator m b n,h for a regression function is Z v 1 m b SI,n,h (x) = inf sup m b n,h (t) dt (1.10) v≥x u≤x v − u u R and 1{t≤x} m b SI,n,h (t) dt is the greatest convex minorant of the process R 1{t≤x} m b n,h (t) dt. Its convergence rate is again n1/3 (van der Vaart and van der Laan, 2003). Meyer and Woodroof (2000) generalized the contraints to larger classes and proved that the variance of the maximum likelihood estimator of a monotone regression function attains the optimal convergence rate n1/3 . In the regression models (1.6) or (1.7) with a multidimensional regression vector X, a multidimensional regression function m(X) can be replaced by a semi-parametric single-index model m(x) = g(θT x), where θT denotes the transpose of a vector θ, or by a more general transformation model g ◦ ϕθ (X) with unknown function m and parameter θ. In the single-index model, several estimators for the regression function m(x) have been defined (Ihimura, 1993, H¨ ardle, Hall and Ihimura, 1993, Hristache, Juditski and Spokony, 2001, Delecroix, H¨ardle and Hristache, 2003), the estimators of the function g and the parameter θ are iteratively calculated from approximations. The inverse of the distribution function FX of a variable X, or quantile function, is defined on [0, 1] by −1 Q(t) = FX (t) = inf{x ∈ IX : FX (x) ≥ t},

it is right-continuous with left-hand limits, like the distribution function. −1 For every uniform variable U , FX (U ) has the distribution function FX and, if F is continuous, then F (X) has an uniform distribution function. The −1 inverse of the distribution function satisfies FX ◦ FX (x) = x for every x in −1 the support of X and FX ◦FX = id for every continuity point x of FX . The weak convergence of the empirical uniform process and its functionals have been widely studied (Shorack and Wellner, 1986, van der Vaart and Wellner, 1996). For a differentiable functional ψ(FX ), n1/2 {ψ(FbX,n ) − ψ(FX )} converges weakly to (ψ 0 B) ◦ FX where B is a Brownian motion, limiting distribution of the empirical process n1/2 (FbX,n − FX ). It follows that the −1 −1 process n1/2 (FbX,n − FX ) converges weakly to B ◦ FX (fX ◦ FX )−1 . Kiefer (1972) established a law of iterated logarithms for quantiles of probabilities tending to zero, the same result holds for 1 − pn as pn tend to one.

January 31, 2011

14

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Theorem 1.2 (Kiefer 1972). Let pn tend to zero with npn and hn tend to infinity, and let δ = 1 or −1 then lim sup δ n

Qn (pn ) − npn = 1, a.s. {2npn log log n}1/2

The results were extended to conditional distribution functions and Sheather and Marron (1990) considered kernel quantile estimators. The inverse function for a nonparametric regression curve determines thresholds for X given Y values, it is related to the distribution function of Y conditionally on X. The inverse empirical process for a monotone nonparametric regression function has been studied in Pin¸con and Pons (2006) and Pons (2008), the main results are presented and generalized in Chapb X,n,h and Q b Y,n,h of the ter 5. The behaviour of the threshold estimators Q conditional distribution is studied, with their bias and variance and the mean squared errors which determine the optimal bandwidths specific to the quantile processes. The Bahadur representation for the quantile estimators is an expansion t − FbX,n −1 −1 −1 ◦ FX (t) + Rn (t), t ∈ [0, 1], FbX,n (t) = FX (t) + fX

where the main is a sum of independent and identically distributed random variables and the remainder term Rn (t) is a op (n−1/2 ) (Ghosh, 1971), Bahadur (1966) studied its a.s. convergence. Lo and Singh (1986), Gijbels and Veraverbeke (1988, 1989) extended this approach by differentiation to the Kaplan-Meier estimator of the distribution function of independent and identically distributed right-censored variables. 1.3

Estimation of functionals of processes

Watson and Laedbetter (1964) introduced smooth estimators for the hazard function of a point process. The functional intensity λ(t) of an inhomogeneous Poisson point process N is defined by λ(t) = lim δ −1 P {N (t + δ) − N (t− ) = 1 | N (t−)}, δ→0

it is estimated using a kernel smoothing, from the sample-path of the point process observed on an interval [0, T ]. Let Y (t) = N (T ) − N (t), then Z bh (t) = Kh (t − s)1{Y (s)>0} Y −1 (s) dN (s). λ

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

15

Introduction

For a sample of a time variable T with distribution function F , let F¯ be the survival function of the variable T , F¯ = 1 − F − , the hazard function λ is now defined as λ(t) = f (t)/F¯ (t). The probability of excess is Pt (t + x) = Pr(T > t + x | T > t) = 1 − = exp{−

Z

F (t + x) − F (t) F¯ (t)

t+x

λ(s) ds}.

t

The product-limit estimator has been defined for the estimation of the distribution function of a time variable under an independent rightcensorship by Kaplan and Meier (1957). Breslow and Crowley (1974) studied the asymptotic behaviour of the process Bn = n1/2 (Fbn −F ), they proved its weak convergence to a Gaussian process B with independent increments, mean zero and a finite variance on every compact sub-interval of [0, Tn:n ], where Tn:n = maxi≤n Ti . The weak convergence of n1/2 (F¯n − F¯ ) has been extended by Gill (1983) to the interval [0, Tn:n ] using its expressions as a martingale upR to the stopping time Tn:n . Let τF = sup{t; F (t) < 1}, for τ t < τF and if 0 F F¯ −1 dΛ < ∞, we have Z t∧Tn:n dFbn (s) b n (t) = , Λ 1 − Fbn− (s) 0 Z t∧Tn:n b¯ (s) dΛ b n (s), b Fn (t) = F n 0

Z t∧Tn:n F − Fbn 1 − Fbn (s− ) b (t) = {dΛn (s) − dΛ(s)} 1−F 1 − F (s) 0

as a consequence, the process n1/2 (F − Fbn )F¯ −1 converges weakly on [0, τF [ to a centered RGaussian process BF , with independent increments and varit ance vF¯ (t) = 0 {(1 − F )−1 F¯ }2 dvΛ , where vΛ is the asymptotic variance of b n − Λ). the process n1/2 (Λ

The definition of the intensity is generalized to point processes having a random intensity. For a multiplicative intensity λY , with a predictable process Y , the hazard function λ is estimated by Z bh (t) = Kh (t − s)Y −1 (s)1{Y (s)>0} dN (s). λ Pn For a random time sample (Ti )i≤n , N (t) = i=1 1{Ti ≤t} and the process Pn Y is Y (t) = i=1 1{Ti ≥t} . Under a right-censorship of a time variable T by an independent variable C, only T ∧ C and the indicator δ of the

January 31, 2011

16

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

event {T ≤ C} are observed. Let X = T ∧ C, the counting processes for P P a n-sample of (X, δ) are N (t) = i 1{Ti ≤t∧Ci } and Y (t) = i 1{Xi ≥t} . Martingale techniques are used to expand the estimation errors, providing optimal convergence rates according to the regularity conditions for the hazard function (Pons, 1986) and the weak convergences with fixed or variable bandwidths (Chapter 6). Regression models for the intensity are classical, there have generally the form λ(t; β) = λ(t)rβ (Z(t)) with a regressor process (Z(t))t≥0 and a parametric regression function such as rβ (Z(t)) = r(β T Z(t)), with an exponential function r in the Cox model (1972). The classical estimators of the CoxR model rely on the estimation t of the cumulated hazard function Λ(t) = 0 λ(s) ds by the stepwise prob n (t; β) at fixed β and the parameter β of the exponential regression cess Λ T function rZ (t; β) = eβ Z(t) is estimated by maximization of an expression b n (β) at Ti similar to the likelihood where λ is replaced by the jump of Λ (Cox 1972). The asymptotic properties of the estimators for the cumulated hazard function and the parameters of the Cox model were established by Andersen and Gill (1982), among others. The estimators presented in this chapter are obtained by minimization of partial likelihoods based on kernel estimators of the baseline hazard function λ defined for each model and on histogram estimators. In the multiplicative intensity model, the kernel estimator of λ satifies the same minimax property as the kernel estimator of a density (Pons, 1986) and this property is still satisfied in the multiplicative regression models of the intensity. Pons and Turckheim (1987) proved the asymptotic equivalence of the estimators of an exponential regression model based on the estimated cumulative intensity and a histogram estimator. The comparison is extended to the new estimators defined from kernel estimators of hazard functions in this book. For a spatial stationary process N on Rd , the k-th moment measures defined for k ≥ 2 and for every continuous and bounded function g on (Rd )k by Z νk (g) = E g(x1 , . . . , xk )N (dx1 ) . . . N (dxk ) (Rd )k

have been intensively studied and they are estimated by empirical moments from observations on a subset G of Rd . The centered moments are immePk diatly obtained from the mean measure m and µk = i=1 (−1)i Cki mi νk−i . The stationarity of the process implies that the k-th moment of N

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Introduction

FunctionalEstimation

17

is expressed as the expectation of an integral of a translation of its (k − 1)-th moment Z νk (g) = E g(x1 − xk , . . . , xk−1 − xk , 0)N (dx1 ) . . . N (dxk ) (Rd )k

which develops in the form Z Z νk (g) = E { gk−1 ◦ Tx (x1 , . . . , xk−1 )N (dx1 ) . . . N (dxk−1 )} N (dx) Rd (Rd )k−1 Z =E νk−1 (gk−1 ◦ Tx ) N (dx), Rd

where gk−1 (x1 , . . . , xk−1 ) = g(x1 , . . . , xk−1 , 0). Let λ be the Lebesgue measure on Rd and Tx be the translation operator of x in Rd , then the moment estimators are built iteratively by the relationship Z νbk,G (g) = {λ(G)}−1 νbk−1,Gk−1 (gk−1 ◦ Ty )dN (y). G

The estimator is consistent and its convergence rate is {λ(G)}k/2 . The stationarity of the process and a mixing condition imply that for every function g of Cb ((Rd )k ), the variable {λ(G)}k/2 (b νk,G (g) − νk (g)) converges weakly to a normal variable with variance ν2k (g). The density of the k-th moment measures are defined as the derivatives of νk with respect to the Lebesgue measure on Rd and they are estimated by smoothing the empirical estimator νbk,G using a kernel Kh on Rd and, by iterations, on Rkd . The convergence of the kernel estimator is then hkd/2 {λ(G)}k/2 , as a consequence of the k d-dimensional smoothing. Consider a diffusion model with nonparametric drift function α and variance function, or diffusion, β dXt = α(Xt )dt + β(Xt )dBt , t ≥ 0

(1.11)

where B is the standard Brownian motion. The drift and variance are expressed as limits of variations of X α(Xt ) = lim h−1 E{(Xt+h − Xt ) | Xt }, h→0

β(Xt ) = lim h−1 E{(Xt+h − Xt )2 | Xt }. h→0

The process X can be approximated by nonparametric regression models with regular or variable discrete sampling schemes of the sample-path of the process X. The diffusion equation uniquely defines a continuous

January 31, 2011

18

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Rt process (Xt )t>0 . Assuming that E exp{− 21 0 β 2 (Bs ) ds} is finite, the Girsanov theorem formulates the density of the process X. Parametric diffusion models have been much studied and estimators of the parameters are defined by maximum likelihood from observations at regularly spaced discretization points or at random stopping times. In a discretization scheme with a constant interval of length ∆n between observations, nonparametric estimators are defined like with samples of variables in nonparametric regression models (Pons, 2008). Let (Xti , Yi )i≤1 be discrete observations with Yi = Xti+1 − Xti defined by equation (1.11), the functions α and β 2 are estimated by Pn Y K (x − Xti ) i=1 P n i hn α bn (x) = , ∆n i=1 Khn (x − Xti ) Pn Z 2 K (x − Xti ) 2 i=1 b P n i hn βn (x) = , ∆n i=1 Khn (x − Xti )

where Zi = Yi − ∆n α bn (Xti ) is the variable of the centered variations for the diffusion process. The variance of the variable Yi conditionally on Xti varies with Xti and weighted estimators are also defined here. Varying sampling intervals or random sampling schemes modify the estimators. Functional models of diffusions with discontinuities were also considered in Pons (2008) where the jump size was assumed to be a squared integrable function of the process X and a nonparametric estimator of this function was defined. Here the estimators of the discretized process are compared to those built with the continuously observed diffusion process X defined by (1.11), on an increasing time interval [0, T ]. The kernel bandwidth hT tends to zero as T tends to infinity with the same rate as hn . In Chapter 8, the MISE of each estimator and its optimal bandwidth are determined. The estimators are compared with those defined for the continuously observed diffusion processes.

Nonparametric time and space transformations of a Gaussian process have been first studied by Perrin (1999), Guyon and Perrin (2000) who estimated the function Φ of non-stationary processes Z = X ◦ Φ, with X a stationary Gaussian process, Φ a monotone continuously differentiable function defined in [0, 1] or in [0, 1]3 . The covariance of the process Z is r(x, y) = R(Φ(x) − Φ(y)) where R is the stationary covariance of X, which implies R(u) = r(0, Φ−1 (u)) and R(−u) = R(u), with a singularity at zero. The singularity function of Z is the difference ξ(x) of the left and right derivatives of r(x, x), which implies Φ(x) = v −1 (1)v(x) where v(x) equals

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Introduction

FunctionalEstimation

19

Rx

ξ(u) du. The estimators are based on the covariances of the process 0 Z are built with its quadratic variations. For the time transformation, the P[nx] estimator of Φ(x) is defined by linearisation of Vn (x) = k=1 (∆Zk )2 where the variables Zk = Z(n−1 k) − Z(n−1 (k − 1)) are centered and independent vn (x) = Vn (x) + (nx − [nx])(∆Z[nx]+1 )2 , x ∈ [0, 1[,

vn (1) = Vn (1), b Φn (x) = vn−1 (1)vn (x),

(1.12)

b n − Φ is uniformly consistent and n1/2 (Φ b n − Φ) is asymptotthe process Φ 3 ically Gaussian. The method was extended to [0, 1] . The diffusion processes cannot be reduced to the same model but the method for estimating its variance function relies on similar properties of Gaussian processes. In time series analysis, the models are usually defined by scalar parameters and a wide range of parametric models for stationary series have been intensively studied since many years. Nonparametric spectral densities of the parametric models have been estimated by smoothing the periodogram calculated from T discrete observations of stationary and mixing series (Wold, 1975, Brillinger, 1981, Robinson, 1986, Herrmann, Gasser and Kneip, 1992, Pons, 2008). The spectral density is supposed to be twice continuously differentiable and the bias, variance and moments of its kernel estimator have been expanded like those of a probability density. It converges weakly with the rate T 2/5 to a Gaussian process, as a consequence of the weak convergence of the empirical periodogram. 1.4

Content of the book

In each chapter, the classical estimators for samples of independent and identically distributed variables are presented, with approximations of their bias, variance and Lp -moments, as the sample size n tends to infinity and the bandwidth to zero. In each model, the weak convergence of the whole processes are considered and the limiting distributions are not centered for the optimal bandwidth minimizing the mean integrated squared error. Chapters 2 and 3 focus on the density and the regression models, respectively. In models with a constant variance, the regression estimator defined as a ratio of kernel estimators is approximated by a weighted sum of two kernel estimators and its properties are easily deduced. In models with a

January 31, 2011

20

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

functional variance, a kernel estimator of the variance is also considered and the estimator of the regression function is modified by an empirical weight. The properties of the modified estimator are detailed. The estimators for independent and identically distributed variables are extended to a stationary continuous process (Xt )t≥0 continuously observed on an increasing time interval, for the estimation of the ergodic density of the process. The observations at times s and t are dependent so the methods for independent observations do not adapt immediatly. The estimators are defined with the conditions necessary for their convergences and their approximation properties are proved. The optimal bandwidth minimizing the mean squared error are functional sequences of bandwidths and the properties of the kernel estimators are extended to varying bandwidths for this reason in Chapter 4. The estimators of derivatives of the density, regression function and the other functions are expressed by the means of derivatives of the kernel so that their convergence rate is modified, the k-th derivative of Kh being normalized by h−(k+1) instead of h−1 for Kh . Functionals of the densities and functions in the other models are considered, the asymptotic properties of their estimators are deduced from those of the kernel estimators. The inverse function defined for the increasing distribution function F are generalized in Chapter 5 to conditional distribution functions and to monotone regression functions. The bias, variance, norms, optimal bandwidths and weak convergences of the quantiles of their kernel estimators are established with detailed proofs. Exact Bahadur-type representations are written, with L2 approximations. Chapter 6 provides new kernel estimators in nonparametric models for real point processes which generalize the martingale estimators of the baseline hazard functions already studied. They are compared to new histogram-type estimators built for these functional models. The probability density of excess duration for a point process and its estimator are defined and the properties of the estimator are also studied. The single-index models are nonparametric regression models for linear combinations of the regression variables. The estimators of the parameter vector θ and the nonparametric regression function g of the model are proved to be consistent for independent and identically distributed variables. New estimators of g and θ are considered in Chapter 7, with

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Introduction

FunctionalEstimation

21

direct estimation methods, without numerical iteration procedures. The convergence rate for the estimator θbn,h obtained by minimizing the empirical mean squared estimation error Vbn is (nh3 )1/2 . The estimator m b n,h built with this estimator of θ has the same convergence rate which is not so small as the nonparametric regression estimator with a d-dimensional regression variable. A differential empirical squared error criterion provides an estimator for the parameter which converges more quickly and the estimator of the regression function m has the usual nonparametric convergence rate (nh)1/2 . More generally, the linear combination of the regressors can be replaced by a parametric change of variable, in a regression model Y = g ◦ ϕθ (X) + ε. Replacing the function g by a kernel estimator at fixed θ, the parameter in then estimated by minimizing an empirical version of the error V (θ) = {Y − b gn,h ◦ ϕθ (X)}2 . Its asymptotic properties are similar to those of the single-index model estimators. The optimal bandwidths are precised. The estimators of the drift and variance of continuous diffusion processes depend on the sampling scheme for their discretization and they are compared to the estimators built from the whole sample-path of the diffusion process. New results are presented in Chapter 8 and they are extended to the sum of a diffusion processes and a jump process governed by the diffusion. For nonstationary Gaussian models, a kernel estimator is defined for the singularity function of the covariance of the process. In Chapter 9, classical estimators of covariances and nonparametric regression functions used for stationary time series are generalized to nonstationary models. The expansions of the bias, variance and Lp -errors are detailed and optimal bandwidths are defined. Nonparametric estimators are defined for the stationarization of time series and for their mean function in auto-regressive models, based on the results of the previous chapters.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Chapter 2

Kernel estimator of a density

2.1

Introduction

Let f be the continuous probability density of a real variable X defined on a probability space (Ω, A, P ) and F be its distribution functions. Let IX be the finite or infinite support of the density function f of X with respect to the Lebesgue measure and IX,h = {s ∈ IX ; [s − h, s + h] ∈ IX }. For a sample (Xi )1≤i≤n distributed as X and a kernel K, estimators of F and f are defined on Ω × R as the empirical distribution function FbX,n (x) = n−1

n X i=1

1{Xi ≤x} , x ∈ IX

and the kernel estimator is defined for every x in IX,h as Z n 1X b Kh (x − Xi ), fX,n,h (x) = Kh (x − s) dFbX,n (s) = n i=1

where Kh (x) = h−1 K(h−1 x) and h = hn tends to zero as n tends to infinity and 1A is the indicator of a set A. The empirical probability measure is Pn PbX,n,h (A) = n−1 i=1 δXi (A), with δXi (A) = 1{Xi ∈A} . Let Z b fn,h (x) = E fn,h (x) = Kh (x − s) dF (s), the bias of the kernel estimator fbn,h (x) is Z bn,h (x) = fn,h (x) − f (x) = K(t){f (x + ht) − f (x)} dt.

(2.1)

The Lp -risk of the kernel estimator of the density f of X is its Lp -norm kfbn,h (x) − f (x)kp = {E|fbn,h (x) − f (x)|p }1/p 23

(2.2)

January 31, 2011

24

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

and it is bounded by the sum of a p-moment and a bias term. For every x in IX,h , the pointwise and uniform convergence of the kernel estimator fbn,h are established under the following conditions about the kernel and the density. Condition 2.1. (1) K is a symmetric density such that |x|2 K(x) → 0 as |x| tends to infinity or K has a compact support with value zero on its frontier; (2) The density function f belongs to the class C2 (IX ) of twice continuously differentiable functions defined in IX . (3) The kernel R function satisfies R integrability conditions: Rthe moments m2K = u2 K(u)du, κα = K α (u)du, for α ≥ 0, and |K 0 (u)|α du, for α = 1, 2, are finite. As n → ∞, hn → 0 nhn → ∞. (4) nh5n converges to a finite limit γ. The next conditions are stronger than Conditions 2.1 (2)-(4), with higher degrees of differentiability and integrability. Condition 2.2. (1) The density function f is Cs (IX ), with a continuous and bounded derivative of order s, f (s) , on IX . (2) As n → ∞, nhn → 0 and nh2s+1 to a finite limit γ > 0. The n Rconverges j kernel function satifies mjK = u K(u)du = 0 for j < s, msK and R |K 0 (u)|α du are finite for α ≤ s. The conditions may be strengthened to allow a faster rate of convergence of the bandwidth to zero by replacing the strictly positive limit of nh2s+1 n by nh2s+1 = o(1). That question appears crucial in the relative imporn tance between the bias and the variance in the L2 -risk of fbn,h − f . The choice of the optimal bandwidth minimizing that risk corresponds to an equal rate for the squared bias and the variance and implies the rates of Condition 2.1(4) or 2.2(2) according to the derivability of the density. Considering the normalized estimator, the reduction of the bias requires a faster convergence rate.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a density

2.2

FunctionalEstimation

25

Risks and optimal bandwidths for the kernel estimator

Proposition 2.1. Under Conditions 2.1-1 for a continuous density f , the estimator fbn,h (x) converges in probability to f (x), for every x in IX,h . Moreover, supx |fbn,h (x) − f (x)| tends a.s. to infinity as n tends to infinity if and only if f is uniformly continuous. The first assertion is a consequence of an integration by parts Z 1 x−y sup |fbn,h (x) − fn,h (x)| ≤ sup |Fbn,h (y) − F (y)| |dK( )| h x∈IX,h x∈IX,h h Z 1 ≤ sup |Fbn,h (y) − F (y)| |dK|. h y The Dvoretzky, Kiefer and Wolfowitz (1956) exponential bound implies that for every λ > 0, Pr(supIX n1/2 |Fbn,h − F | > λ) ≤ 58 exp{−2λ2 }, then Z b b Pr(sup |fn,h − fn,h | > ε) ≤ Pr(sup |Fn,h − F | > ( |dK|)−1 hn ε) Proof.

IX,h

with α > 0, and 2.2.

IX

P∞

n=1

≤ 58 exp{−αnh2n }

exp{−nα h2n } tends to zero under Condition 2.1 or 

Proposition 2.2. Assume hn → 0 and nhn → ∞, (a) under Conditions 2.1, the bias of fbn,h (x) is h2 bn,h (x) = m2K f (2) (x) + o(h2 ), 2 denoted h2 bf (x) + o(h2 ), its variance is V ar{fbn,h (x)} = (nh)−1 κ2 f (x) + o((nh)−1 ),

also denoted (nh)−1 σf2 (x) + o((nh)−1 ), where all approximations are uniform. Let K have the compact support [−1, 1], the covariance of fbn,h (x) and fbn,h (y) is zero if |x − y| > 2h, otherwise it is approximated by Z

(nh)−1 {f (x) + f (y)}δx,y K (v − αh K v + αh dv 2 where αh = |x − y|/(2h) and δx,y is the indicator of {x = y}. (b) Under Conditions 2.2, for every s ≥ 2, the bias of fbn,h (x) is hs bn,h (x; s) = msK f (s) (x) + o(hs ), s! and kfbn,h (x) − fn,h (x)kp = 0((nh)−1/p ), for every p ≥ 2, where the approximations are uniform.

January 31, 2011

26

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Proof. The bias as h tends to zero is obtained from a second order expansion of f (x + ht) under Condition 2.1, and from its s-order expansion under Condition 2.2. The variance of fbn,h (x) is Z −1 2 b V ar{fn,h (x)} = n { Kh2 (x − s)f (s) ds − fn,h (x)}. R The first term of the sum is n−1 Kh2 (x − u)f (u)du = (nh)−1 κ2 f (x) + o((nh)−1 ), the second term n−1 f 2 (x) + O(n−1 h) is smaller. R The covariance of fbn,h (x) and fbn,h (y) is written n−1 { I 2 Kh (u − X x)Kh (u − y)f (u) du − fn,h (x)fn,h (y)}, it is zero if |x − y| > 2h. Otherwise let αh = |x−y|/(2h) < 1, changing the variables as h−1 (x−u) = v −αh and h−1 (y − u) = v + αh with v = {(x + y)/2 − u}/h, the covariance develops as Z x+y −1 b b Cov{fn,h (x), fn,h (y)} = (nh) f ( ) K(v − αh )K(v + αh )dv 2 + o((nh)−1 ). If |x − y| ≤ 2h, f ((x + y)/2) = f (x) + o(1) = f (y) + o(1), the covariance is approximated by Z

(nh)−1 {f (x) + f (y)}I{0≤αh <1} K (v − αh K v + αh dv. 2 Due to the compactness of the support of K, is zero if  the covariance αh ≥ 1. For x 6= y, αh tends to infinity and I 0 ≤ αh < 1 tends to zero as n tends to infinity, then the indicator is approximated by the indicator δx,y of {x = y}. R For p = 1, E|fbn,h − fn,h |(x) ≤ IX,h |Kh (x − s)| d|Fbn − F |(s) which converges to zero as n tends to infinity. For p ≥ 3, the Lp -risk of fbn,h (x) is obtained from the expansion of the sum and by recursion on the order of the moment in the expansion of n X  p   p E fbn,h (x) − fn,h (x) = E n−1 Kh (x − Xi ) − fn,h (x) . i=1

The first moments of the centered estimator fbn,h − fn,h are  3 E fbn,h (x) − fn,h (x) = (nh)−1 {−2κ2 f 2 (x) + o(1)},  4 E fbn,h (x) − fn,h (x) = (nh)−1 {κ2 f 3 (x) + o(1)},  5 E fbn,h (x) − fn,h (x) = (nh)−1 {11κ2 f 3 (x) + o(1)}.

By iterations, the product of moments in the expansion of the risk p E fbn,h (x) − fn,h (x) determines its higher order as (nh)−1 . 

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a density

FunctionalEstimation

27

Integrating the R above expansions entails similar bounds for the integrated norms E |fbn,h (x) − fn,h (x)|p dx = 0((nh)−1 ), for every p > 1. For p = 2, E{fbn,h (x) − f (x)}2 = V ar{fbn,h (x)} + {fn,h (x) − f (x)}2 and its first order expansion is n−1 h−1 κ2 f (x) + o(n−1 h−1 ) + 41 m22K h4 f (2)2 (x) + o(h4 ). The asymptotic mean squared error for fbn,h at x is then 1 AM SE(fbn,h ; x) = (nh)−1 κ2 f (x) + m22K h4 f (2)2 (x), 4 it is minimum for the bandwidth function  κ2 f (x) 1/5 . hAMSE (x) = n−1/5 m22K f (2)2 (x)

A smaller order bandwidth increases the variance of the density estimator and reduces its bias, with the order n−1/5 its asymptotic distribution cannot be centered. An estimator of the derivative f (k) is defined by the means of the derivative K (k) of the symmetric kernel, for k ≥ 1. The convergences rates for estimators of a derivative of the density also depend on the order of the derivative. Consider the k-order derivative of Kh (k)

Kh (x) = h−(k+1) K (k) (h−1 x), k ≥ 1. The estimators of the derivatives of the density are (k) fbn,h (x) = n−1

n X i=1

(k)

Kh (x − Xi ).

(2.3)

(k) The next lemma implies the uniform consistency of fbn,h to f (k) , for every order of derivability k ≥ 1 and allows to calculate the variance of the derivative estimators. It is not exhaustive and integrals of higher orders are easily obtained using integrations by parts.

Lemma 2.1. Let K be a symmetric density in class C2 , its R function (j) derivatives satisfy the following properties : K (z) dz = 0, for every R R 2 2 2 j ≥ 1, zK (z) dz = 0, κ22 = z K (z) dz 6= 0 and Z Z Z zK (1) (z) dz = −1, z 2 K (1) (z) dz = 0, z 3 K (1) (z) dz = −3m2K , Z Z Z (2) 2 (2) zK (z) dz = 0, z K (z) dz = 2, z 3 K (2) (z) dz = 0, Z Z z 4 K (2) (z) dz = 12m2K , κ11 = z(K 0 K)(z) dz = −κ2 /2, Z Z K (1) K dz = 0, K (1)2 dz 6= 0.

January 31, 2011

28

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes (1)

The sum fbn,h (x) = n−1 its expectation

Pn

i=1

(1)

Kh (x − Xi ) converges uniformly on IX,h to

Z (1) (1) (1) fn,h (x) = EKh (x − X) = Kh (u − x)fX (u) du Z Z h2 = −f (1) (x) zK (1) (z) dz − f (3) (x) z 3 K (1) (z) dz + o(h2 ) 6 2 h = f (1) (x) + m2K f (3) (x) + o(h2 ), 2 2 (1) then fbn,h converges uniformly to f (1) (x) and its bias is h2 m2K f (3) (x). Its R variance is (nh3 )−1 f (x) K (1)2 (z) dz + o((nh3 )−1 ) and the optimal local bandwidth for estimating f (1) is deduced as R (1)2  (z) dz 1/7 (1) −1/7 f (x) K hAMSE (f ; x) = n , 2 (3)2 m2K f (x) thus the estimator of the first density derivative (2.3) has to be computed with a bandwidth estimating hAMSE (f (1) ; x). For the second derivative, 2 (2) (2) the expectation of fbn,h is fn,h (x) = f (2) (x) + h2 m2K f (3) (x) + o(h2 ), so it 2

converges uniformly f (2) with the bias h2 m2K f (4) (x)+o(h2 ) and the variR to(2)2 5 −1 ance (nh ) f (x) K (z) dz + o((nh4 )−1 ). More generally, Lemma 2.1 generalizes by induction to higher orders and the rate of optimal bandwidths is deduced as follows. (k)

Proposition 2.3. Under Conditions 2.1, the estimator fbn,h of the korder derivative of a density in class C2 has a bias O(h2 ) and a variance O((nh2k+1 )−1 ), its optimal local and global bandwidths are O(n−1/(2k+5) ), for every k ≥ 2. For a density of class Cs and under Conditions 2.2, the bias is a O(hs ) and the variance a O((nh2k+1 )−1 ), its optimal bandwidths are O(n−1/(2k+2s+1) ) and the corresponding L2 -risks are O(n−s/(2k+2s+1) ). (k) As a consequence the L2 -risk of the estimator fb b is a O(n−2s/(2k+2s+1) ) n,hopt for every density in Cs , s ≥ 2. If the k-th derivative of the kernel and the density are lipschitzian with |K (k) (x) − K (k) (y)| ≤ α|x − y| and |f (k) (x) − f (k) (y)| ≤ α|x − y| for some constant α > 0, then there exists a constant C such that for every x and y in IX,h (k) (k) |fbn,h (x) − fbn,h (y)| ≤ Cαh−(k+1) |x − y|. R (k)2 The integral θk = f (x) dx of the quadratic k-th derivative of the density is estimated by Z

θbk,n,h =

(k)2 fbn,h (x) dx,

(2.4)

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

29

Kernel estimator of a density

the variance E(θbk,n,h − θk )2 has the same order as the MISE for the estima(k) tor fbn,h of f (k) , hence it converges to θk with the rate O((n1/2 hk+1/2 ) and the estimator does not achieve the parametric rate of convergence n1/2 . The Lp -risk of the estimator of the density decreases as s increases and, for p ≥ 2, a bound of the Lp -norm is kfbn,h (x) − f (x)kpp ≤ 2p−1

hps {mpsK f (k)p (x) + o(1)} (s!)p
+ (nh)−1 {gp (x) + o(1)} ,

P[p/2] P k P where gp (x) = k=2 1<j1 6=...6=jk ≤p; i ji =p κj1 . . . κjl f (x). The optimal bandwidth is still reached when both terms of this bound are of the same order and minimal. With p = 2 and s = 2, it is 1

hn (x) = O(n− 5 ),

2

kfbn,h (x) − f (x)k2 = O(n− 5 ).

For a density of Cs and the L2 risk, it is hn = O(n−1/(2s+1) ) and kfbn,h (x) − f (x)k2 = O(n−s/(2s+1) ).

The bandwidth and the risk decrease as the order of derivability of the density increases. The derivability condition f ∈ Cs in 2.1 can be replaced by the condition: f belongs to a H¨ older class Hα,M with |f (s) (x)−f (s) (y)| ≤ M |x−y|α−s where s = [α] ≥ 0 is the integer part of α > 0. Proposition 2.4. Assume f is bounded and belongs to a H¨ older class α b Hα,M , then the bias of fn,h is bounded by M m[α]K h /([α]!) + o(hα ), the optimal bandwidth is O(n1/(2α+1) ) and the MISE at the optimal bandwidth is O(nα/(2α+1) ). 2.3

Weak convergence

The Lp -norm of the variations of the process fbn,h − fn,h are bounded by the same arguments as the bias and the variance. Assume that K has the support [−1, 1]. Lemma 2.2. Under Conditions 2.1 and 2.1, there exists a constant C such that for every x and y in IX,h and satisfying |x − y| ≤ 2h E{fbn,h (x) − fbn,h (y)}2 ≤ C(nh3 )−1 |x − y|2 .

January 31, 2011

30

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Proof. Let x and y in IX,h , the variance of fbn,h (x) − fbn,h (y) develops according to their variances given by Proposition 2.2 and the covariance between both terms which has the same bound by the Cauchy-Schwarz inequality. The second order moment E|fbn,h (x) − fbn,h (y)|2 develops as R the sum n−1 {Khn (x − u) − Khn (y − u)}2 f (u) du + (1 − n−1 ){f R n,hn (x) − 2 fn,hn (y)} . For an approximation of the integral I2 (x, y) = {Khn (x − u) − Khn (y − u)}2 f (u) du, the Mean Value Theorem implies Khn (x − u) − (1) Khn (y − u) = R(x − y)ϕn (z − u) where ϕn (x) = Khn (x), and z is between x and y, then {Khn (x − u) − Khn (y − u)}2 f (u) du is approximated by Z Z 2 (1)2 2 −3 (x − y) ϕn (z − u)f (u) du = (x − y) hn {f (x) K (1)2 + o(hn )}. −1 Since h−1 n |x| and hn |y| are bounded by 1, the order of the second moment of fbn,h (x) − fbn,h (y) is a O((x − y)2 (nh3n )−1 ) if |x − y| ≤ 2hn and the covariance is zero otherwise. 

Theorem 2.1. Under Conditions 2.1 and 2.2, for a density f of class Cs (IX ) and with nh2s+1 converging to a constant γ, the process Un,h = (nh)1/2 {fbn,h − f }I{IX,h }

converges weakly to Wf +γ 1/2 bf , where Wf is a continuous Gaussian process on IX with mean zero and covariance E{Wf (x)Wf (x0 )} = δx,x0 σf2 (x), at x and x0 . Proof. The finite dimensional distributions of the process Un,h converge weakly to those of Wf + γ 1/2 bf , as a consequence of Proposition 2.2. The covariance of Wf at x and x0 is Cf,n (x, x0 ) = limn nhCov{fbn,h (x), fbn,h (x0 )}, and Proposition 2.2 implies that Un,h (x) and Un,h (x0 ) are asymptotically independent as n tends to infinity. If the support of X is bounded, let a = inf IX , η > 0 and c >
1/2 1/2 γ |bf (a)| + 2η −1 σf2 (a) , then  Pr{|Un,h (a)| > c} ≤ Pr (nh)1/2 |(fbn,h − fn,h )(a)| + (nh)1/2 |bn,h (a)| > c V ar{(nh)1/2 (fbn,h − fn,h )(a)} ≤ , {c − (nh)1/2 |bn,h (a)|}2

so that for n sufficiently large

Pr{|Un,h (a)| > c} ≤

σf2 (a) {c − γ 1/2 |bf (a)|}2

+ o(1) < η,

the process Un,h (a) is therefore tight. Lemma 2.2 R and the bound {fn,h (x)− f (x) − fn,h (y) + f (y)}2 ≤ |f (x) − f (y)|2 + [ K(z){f (x + hz) − f (y +

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a density

FunctionalEstimation

31

hz)}dz]2 ≤ 2|x − y|2 kf (1) k2∞ imply that the mean of the squared variations of the process Un,h are O(h−2 |x − y|2 ) as |x − y| ≤ 2h < 1, otherwise the estimators fbnh (x) and fbnh (y) are independent. Billingsley’s Theorem 3 implies the tightness of the process Un,h 1[−h,h] and the convergence is extended to any compact subinterval of the support. With an unbounded support for X such that E|X| < ∞, for every η > 0 there exists A such that P (|X| > A) ≤ η, therefore P (|Un,h (A + 1)| > 0) ≤ η and the same result still holds on [−A − 1, A + 1] instead of the support of the process Un,h .  Corollary 2.1. The process sup σf−1 (x)|Un,h (x) − γ 1/2 bf (x)|

x∈IX,h

converges weakly to supIX |W1 |, where W1 is the Gaussian process with mean zero, variance 1 and covariances zero. For every η > 0, there exists a constant cη > 0 such that Pr{ sup |σf−1 (Un,h − γ 1/2 bf ) − W1 | > cη } IX,h

tends to zero as n tends to infinity. Lemma 2.2 concerning second moments does not depend on the smoothness of the density and it is not modified by the condition of a H¨older class instead of a class Cs . The variations of the bias are now bounded by {fn,h (x)−f (x)−fn,h (y)+f (y)}2 ≤ 2M |x−y|2α and the mean of the squared variations of the process Un,h are O(h−2 |x − y|2 ) for |x − y| ≤ 2h < 1. The weak convergence of Theorem 2.1 is therefore fulfilled with every α > 1. With the optimal bandwidth for the global MISE error  1/5 κ R 2 hAMISE = , 2 (2)2 nm2K f (x) dx R (2)2 the limit γR of nh5n is κ2 m−2 (x) dx}−1 . The integral of the second 2K { f derivative f (2)2 (x) dx and the bias term bf = 21 m2K f (2) are estimated using the second derivative of the estimator for f . Furthermore, the variance σf2 = κ2 f is immediatly estimated. More simply, the asymptotic criterion is written Z AM ISEn (h) = {h4 bf (x) + (nh)−1 σf2 (x)}f −1 (x) dF (x)

January 31, 2011

32

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

and it is estimated by the empirical mean n−1

n X i=1

{h4 bf (Xi ) + (nh)−1 σf2 (Xi )}f −1 (Xi ).

This empirical error is estimated by b ISEn (h) = n−1 AM

n X i=1

2 {h4bbf,n,h2 (Xi ) + (nh)−1 σ bf,n,h (Xi )}fbh−1 (Xi ) 2 2

with another bandwidth h2 converging to zero. The global bandwidth hAMISE is then estimated at the value that achieves the minimum of b ISEn (h), i.e. AM n 4n Pn bb2 o−1/5 b−1 i=1 f,n,h2 (Xi )fh2 (Xi ) b hn = . Pn 2 bf,n,h (Xi )fbh−1 (Xi ) i=1 σ 2 2

Bootstrap estimators for the bias and the variance provide another estimation of M ISEn (h) and hAMISE . These consistent estimators are then used for centering and normalizing the process fbn,h −f and provide an estimated process bn = (nb U hn )1/2 σ b−1

b

{fn,bhn f,n,b hn

−f −b γn,bhnbbf,n,bhn }I{IX,bhn }.

An uniform confidence interval with a level α for the density f is deduced from Corollary 2.1, using a quantile of supIX |W1 |. Theorem 2.2. Under Conditions 2.1 and 2.2, for a density f of class Cs (IX ) and with nh2s+2k+1 converging to a constant γ, the process (k) (k) Un,h = (nh2k+1 )1/2 {fbn,h − f (k) }I{IX,h }

converges weakly to a Gaussian process Wf,k + γ 1/2 bf,k , where Wf,k is a continuous Gaussian process on IX with mean and covariances zero. Let X be a vector variable defined in a subset IX of Rd , its density f is estimated by smoothing its distribution function Fbn (x) =

n X

1{X1 ≤x1 ,...,Xd ≤xd } , x = (x1 , . . . , xd ),

i=1

by a multivariate kernel K defined on [−1, 1]d and Kh (x) = h−d K(h−d x), Qd −1 with a single bandwidth or Kh (x) = k=1 h−1 k K(hk xk ) with a vector

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a density

FunctionalEstimation

33

bandwidth, for x in IX,h . The derivatives of the density f (k) are arrays and the rates of their moments depend on the dimension d. If hk = h hs bs,n,h (x) = msK f (s) (x) + o(hs ), s! V ar{fbn,h (x)} = (nhd )−1 κ2 f (x) + o((nhd )−1 ), kfbn,h (x) − fn,h (x)kp = 0((nhd )−1/p ), (2.5) M ISEn (h, x) = O(h2s ) + O((nhd )−1 ).

The optimal bandwidth hn (x) minimizing the M ISEn (h, x) has the order n−1/(2s+d) where the local MISE reaches the minimal order O(n−2s/(2s+d) ). The convergence rate of fbn,h − f is (nhd )1/2 and the results of Theorem 2.1 and its corollary still hold with this rate. 2.4

Minimax and histogram estimators

Consider a class F of densities and a risk R(f, fbn ) for the estimation of a density f of F by an estimator fbn belonging to a space Fb. A minimax estimator fbn∗ is defined as a minimizer of the maximal risk over F fb∗ = arg inf sup R(f, fbn ). n

b f ∈F fbn ∈F

With an optimal bandwidth related to the risk Rpp , the kernel estimator of a density of F = Cs , s ≥ 2, provides a Lp -risk of order hsp n (x; s, p) and this is the minimax risk order in a space Fb determined by the regularity of the kernel, the kernel estimator reaches this bound. R The estimator (2.4) of the integral θk = f (k)2 (x) dx of the quadratic kth derivative of a density of C2 has therefore the optimal rate of convergence for an estimator of θk . The histogram is the older unsmoothed nonparametric estimator of the density. It is defined as the empirical distribution of the observations cumulated on small intervals of equal length hn , divided by hn , with hn and nhn converging to zero as n tends to infinity. Let (Bjh )j=1,...,JX,h be a partition of IX into subintervals of length h and centered at ajh , and let P Kh (x) = h−1 j∈JX,h 1Bjh (x) be the kernel corresponding to the histogram, it is therefore defined as Z fen,h (x) = hKh (x) Kh (s)dFbn (s). P Its bias ebf,h (x) = j∈JX,h 1Bjh (x){f (ajh ) − f (x)} + o(h) = hf (1) (x) + o(h) is larger than the bias of kernel estimators and its variance vef (x) is a

January 31, 2011

34

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

O((nh)−1 ), due to the covariance zero between the empirical distribution on R Bjh and Bj 0 h for j 6= j 0 . As n tends to infinity, h−1 n Bjh dV ar(Fbn − F ) = f (ajh ){1 − 2F (ajh )} + o(1) hence vef,h (x) = (nh)−1 f (x){1 − 2F (x)} + o((nh)−1 ). Let ebf (x) = f (1) (x) and vef (x) = f (x){1 − 2F (x)}. The normalized histogram (nh)1/2 (fen,h − f − hf (1) )(x) converges weakly to a normal variable vef,h (x)N (0, 1) and it is asymptotically unbiased with a bandwidth hn = o(n−1/3 ). Increasing the order of hn reduces the variance of the histogram and increases its bias. The asymptotic mean squared error of the histogram is minimal for the bandwidth 1/3 hn (x) = n−1/3 {2eb2 (x)}−1/3 ve (x) = {2nf (1)2 (x)f −1 (x)}−1/3 f

f

then it is approximated by

2/3

M SEopt (x) = n−2/3 {e vf (x)ebf (x)}2/3 {21/3 + 2−1/3ebf (x)} = n−2/3 {f (x){1 − 2F (x)}f (1) (x)}2/3

×[21/3 + 2−1/3 {f (1) (x)}2/3 ].

These expressions do not depend on the degree of derivability of the density. The optimal bandwidth, the bias ebf (x) and the variance vef (x) of the histogram are estimated by plugging the estimators of the density and its derivative in their formulae. The Lp moments of the histogram are determined by the higher order term in the expansion of |fen,h (x) − fn,h (x)|pp , it is a O((nh)−1 ), for every p ≥ 2. The derivatives of the density are defined by differences of values of the histogram. For x in Bjh , f (1) (x) = h−1 {f (aj+1,h ) − f (aj,h )} + o(1) is estimated by (1) fe (x) = h−1 {fen,h (aj+1,h ) − fen,h (aj,h )} n,h

and the derivatives of higher order are defined in the same way. The bias (1) of fen,h is a O(1) and its variance is a O((nh3 )−1 ). 2.5

Estimation of functionals of a density

The estimation of the integral of a squared density Z Z 2 θ = f (x) dx = f (x) dF (x)

has been considered by many authors and several estimators have been proposed. The plug-in kernel density estimator X 2 θbn,h = Kh (Xi − Xj ) n(n − 1) 1≤i<j≤n

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a density

FunctionalEstimation

35

has been introduced by Hall and Marron (1987). A second plug-in estimator was defined by Bickel and Ritov (1988) as the integral of the square of the estimated density X Z 2 θ¯n,h = Kh (x − Xi )Kh (x − Xj ) dx. n(n − 1) 1≤i<j≤n

Other authors proposed estimators based on projections on orthogonal basis of L2 (X ). Bickel and Ritov (1988) and Gin´e and Nickl (2008) proved the asymptotic equivalence of the estimators and their weak convergence for bounded densities ofR Rthe space H2α of functions satisfying the integrated H¨ older condition |t|−(1+2α) |f (x − t) − f (x)|2 dx dt < ∞ for some 0 < α ≤ 1/2. Then the bias of the estimator θbn,h is (h2α ) and its vari−2/(4α+1) ance is (n2 h ∨ n−1 h2α ), R ).2 Moreover, if α > 1/4 and hn = O(n 1/2 b ¯ n {2θn,h − θn,h − f (x) dx} converges weakly to a centered Gaussian R R variable with variance 4{ f 3 − ( f 2 )2 }. The integrals of the squared k-th derivatives of the density Z θk = f (k)2 (x) dx

are also estimated by the integral of the square of the kernel estimator for the derivative of the density X Z 2 (k) (k) Kh (x − Xi )Kh (x − Xj ) dx, θ¯n,h = n(n − 1) 1≤i<j≤n

which is equivalent to the integral of the squared estimator (2.3). The mode of a real function f on an open set IX is Mf = sup |f (x)|.

(2.6)

IX

It is a norm on the space of real functions defined on IX , and for functions with values in a metric space with the norm | · |. The triangular inequality Pp provides a bound for the mode of mixture density g = k=1 µk fk , where Pp the sum of mixture probabilities µk belonging to ]0, 1[ satisfies k=1 µk = 1 and the densities fk have the same support Mg ≤

p X

µk Mfk .

(2.7)

k=1

If the supports of the densities fk are sufficiently separated, their modes can be identified and the mode of g is identical to the mode of one density

January 31, 2011

36

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

of the mixture. Otherwise mixture densities with overlapping supports are cumulated in g and the mode of g is not always located at the mode of a component. It is the combination of the modes of the component densities of the mixture only if all modes have the same location. For a single density f in class C2 (IX ), the mode Mf is estimated by cf,n,h = M b . M fn,h

Under Conditions 2.1, the density is locally concave in a neighborhood NM of the mode and its estimator has the same property for n sufficiently large. cf,n,h converges to Mf in probability. A Taylor expansion It follows that M for x in NM is written f (1) (x) = (x − Mf )f (2) (Mf ) + o(x − Mf ). At the cf,n,h ) = 0, which entails estimated mode, f (1) (Mf ) = 0 and fbn,h (M cf,n,h − Mf ) = f (2)−1 (Mf ) f (1) (M cf,n,h ){1 + o(1)} (M cf,n,h ) − fb(1) (M cf,n,h )}{1 + o(1)}. = f (2)−1 (Mf ) {f (1) (M n,h

(1) For every x in IX , the variable U(1),n,h (x) = (nh3 )1/2 (fbn,h − f (1) )(x) converges weakly to a Gaussian variable with a non degenerated variance κ2 f (x) and a mean m(1) (x) = limn (nh7n )1/2 m2K f (3) (x)/2. It follows that cf,n,h − Mf ) is (nh3 )1/2 and the convergence rate of (M

cf,n,h − Mf ) = −f (2)−1 (Mf ) U(1),n,h (M cf,n,h ) + o(1). (nh3 )1/2 (M

The following proposition is a consequence of the asymptotic behaviour of the derivative of the estimated density given in Theorem 2.3, the rate of the optimal bandwidth for the density is lower than the rate assumed in Parzen (1962) and the limiting distribution R (1)2 has a non zero mean. 2 (2) −2 Let σM = f (M ){f (M )} K (z) dz. f f f cf,n,h − Mf ) Proposition 2.5. Under Conditions 2.1 and 2.2, (nh3 )1/2 (M (2)−1 2 converges weakly to a variable N (f (Mf )m(1) (Mf ), σMf ).

cf,n,h is n1/3 like Note that if hn = O(n−1/9 ), the convergence rate of M Chernoff’s estimator (1964). The optimal rate for the bandwidth of the first derivative of the density is n1/(2s+3) as established in Proposition 2.3 for every integer s > 1, that is n1/9 for a density in C3 (IX ). For a smaller cf,n,h converges fastly, with the rate bandwidth of order n−1/r , r > 2, M (1−3/r)/2 cf,n,h ) is n . If the density belongs to C3 (IX ), the bias of f (1) (M

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a density

FunctionalEstimation

37

(1) deduced from the bias of the process (fbn,h − f (1) ) and it equals

2 2 cf,n,h ) = − h m2K f (3) (M cf,n,h )+o(h2 ) = − h m2K f (3) (Mf )+o(h2 ), Ef (1) (M 2 2 it does not depend on the degree of derivability of the density f .

The support of a density f can be estimated from its graph defined as Gf = {(x, y); y = f (x), x ∈ IX }. For a continuous function f defined on an open interval IX with compact closure, Gf is an open set of R2 with compact closure. This closed set defines the support of the function f . For every y such that (x, y) belongs to a closed subset A of Gf , there exist x in a closed subinterval of IX such that y = f (x). The graph of a sum of two densities f1 and f2 is the union of their graphs G1 ∪ G2 and by difference G1 = G1 ∪ G2 − G2 \ G1 , with G2 \ G1 = {(x, y); y = f2 (x) 6= f1 (x), x ∈ IX }. Let Gbf,n,h = {(x, y); y = fbn,h (x), x ∈ IX } be the graph of the kernel estimator of an absolutely continuous density f on IX , then Gfbn,h = Gf ∪ Gfbn,h −f = Gf + Gfbn,h −f \ Gf hence Gfbn,h −f = Gbf,n,h − Gf and it converges a.s. to zero as n tends to infinity. The support of the density f is consistently estimated by Gbf,n,h and the extrema of the density are consistently estimated by those of the estimated graph. 2.6

Density of absolutely continuous distributions

Let F0 be a distribution function in a functional space F and Fϕ0 be a distribution function absolutely continuous with respect to F0 , having a density ϕ0 with respect to F0 . The function ϕ0 belongs to a nonparametric space of continuous functions Φ and the distribution functionR Fϕ0 belongs to the ∞ nonparametric model PF ,Φ = {(F, ϕ); F ∈ F, ϕ ∈ Φ, 0 ϕ dF = 1}. The observations are two subsamples X1 , . . . , Xn1 with distribution function F0 and Xn1 +1 , . . . , Xn with distribution function Fϕ0 . The approach extends straightforwardly to a population stratified in K subpopulations. Estimation of the distributions of stratified populations has already been studied, in particular by Anderson (1979) with a specific parametric form for ϕθ , by Gill, Vardi R · and Wellner (1988) in biased sampling models with group distributions 0 wk dF , where the weight functions are known, by Gilbert (2000) in biased sampling models with parametric weight functions, by Cheng and Chu (2004), with the Lebesgue measure and kernel density estimators.

January 31, 2011

38

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

The density with respect to the Lebesgue measure of a distribution function F in F is denoted f and the distributions of both samples are supposed to have the same support. Let n2 = n − n1 increasing with n, such that limn n−1 n1 = π in ]0, 1[, and let ρ be the sample indicator defined by ρ = 1 for individuals of the first sample and ρ = 0 for individuals of the second sample. Let F1 = πF0 and F2 = (1 − π)Fϕ0 be the subdistribution functions of the two subsamples, they are estimated by the corresponding empirical subdistribution functions Fb1,n = n−1

Fb2,n = n−1

n X

i=1 n X i=1

ρi 1{Xi ≤t} = n−1

n1 X

1{Xi1 ≤t} ,

i=1

(1 − ρi )1{Xi ≤t}

and π bn = n−1 n1 . Their densities with respect to the Lebesgue measure are denoted f1 and f2 , and the density of the second sample with respect to the distribution of the first one is ϕ = π(1 − π)−1 f1−1 f2 . The densities f1 and f2 are estimated by smoothing Fb1,n and Fb2,n , then f0 , fϕ and ϕ are estimated by Z −1 b f0,n,h (t) = π bn Kh (t − s) dFb1,n (s), Z fbn,h (t) = (1 − π bn )−1 Kh (t − s) dFb2,n (s), −1 ϕ bn,hn (t) = fb0,n,h (t)fbn,h (t)

on every compact subset of the support of the densities where f0 is strictly positive and kϕ bn,h − ϕ0 k converges in probability to 0. R The expectation of the estimators are approximated by f0;n,h (t) = Kh (t − R 2 (2) s) dF0 (s) + O(n−1/2 ) = f0 + h2 f0 + o(h2 ) and fn,h (t) = Kh (t − 2 (2) s) dFϕ (s) + O(n−1/2 ) = fϕ + h2 fϕ + o(h2 ). The bias of ϕ bn,h is ex2 (2) −1 h (1) (1) 2 panded as bn,h = 2 f0 {ϕf0 + 2ϕ f0 } m2K + o(h ), its variance is vn,h = f0−2 {V arfbn,h + ϕ2 V arfb0,n,h } + o((nh)−1 ) where the variances given in Proposition 2.2, V arfbj,n,h (t) = (nj h)−1 κ2 fj (t) {1 + o(1)} imply similar approximations for the variances of the estimators fb0,n,h and fbn,h . The following approximation with independent subsamples implies the weak convergence of the estimator of ϕ (nh)1/2 (ϕ bn,h − ϕn,h ) = f0−1 (nh)1/2 {(fbn,h − fϕ,n,h ) − ϕ(fb0,n,h − f0,n,h )} + oL (1). 2

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a density

2.7

FunctionalEstimation

39

Hellinger distance between a density and its estimator

Let P and Q be two probability measures and let λ = P + Q be the dominating measure of their sum. Let F and G be the distribution functions of a variable X under the probability measures P and Q, respectively, and let f and g be the densities of P and Q, respectively, with respect to λ. The Hellinger distance between P and Q is Z √ Z p p 1 1 √ 2 2 h (P, Q) = ( dP − dQ) = ( f − g)2 dλ. 2 2 The affinity of P and Q is Z p 2 ρ(P, Q) = 1 − h (P, Q) = f g dλ.

The following inequalities were proved by Lecam and Yang (1990) 1 h2 (P, Q) ≤ kP − Qk1 ≤ {1 − ρ2 (P, Q)}1/2 . 2 Applying this inequality to the probability density f of P , absolutely continuous with respect to the Lebesgues measure λ, and its estimator fbn,h , we obtain s s Z Z bn,h f fbn,h h2 (fbn,h , f ) = (1 − ) dF ≤ {1 − ( dF )2 }1/2 . f f The convergence to zero of the Hellinger distance h2 (fbn,h , f ) is deduced

from the obvious bound s s Z Z bn,h f fbn,h h2 (fbn,h , f ) = (1 − ) dF ≤ ( − 1) d(Fbn − F ) (2.8) f f R q fbn,h b which is consequence of the inequality f dFn ≥ 0. This inequality and the uniform a.s. consistency of the density estimator also imply the a.s. convergence to zero of n1/2 h2 (fbn,h , f ). By differentiation, estimators of functionals of the density converges with the same rate as the estimator of the density, hence h2 (fbn,h , f ) convergences to zero in probability with 1/2 the rate nhn . Applying these results to the probability measures P0 and P = Pϕ0 of the previous section, with distribution functions F0 and F , we get similar formulae Z Z 1 √ √ h2 (P0 , P ) = (1 − ϕ)2 dF0 ≤ {1 − ( ϕ dF0 )2 }1/2 , 2 s Z Z s ϕ bn,h ϕ bn,h 2 h (ϕ bn,h , ϕ) = (1 − ) dF ≤ {1 − ( dF )2 }1/2 . ϕ ϕ

January 31, 2011

40

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

The bound (2.8) is adapted to the density ϕ Z s ϕ bn,h 2 h (ϕ bn,h , ϕ) ≤ ( − 1) d(Fbn − F ), ϕ

1/2

it follows that the convergence rate of h2 (ϕ bn,hn , ϕ) is nhn . 2.8

Estimation of the density under right-censoring

On a probability space (Ω, A, P ), let X and C be two independent positive random variables with densities f and fC and such that P (X < C) is strictly positive, and let T = X ∧ C, δ = 1{X≤C} denote the observed variables when X is right-censored by C. Let X Nn (t) = δi 1{Ti ≤t} 1≤i≤n

be the number of observations before t and X Yn (t) = 1{Ti ≥t} 1≤i≤n

be the number of individuals at risk at t. The survival function F¯ = 1 − F − of X is now estimated by Kaplan-Meier’s product-limit estimator Y Y δi Jn (Ti ) b¯ R (t) = b n (Ti ) , with F 1− = 1 − ∆Λ n Yn (Ti ) Ti ≤t Ti ≤t Z t Jn (s) b n (t) = Λ dNn (s) , (2.9) 0 Yn (s) and Jn (s) = 1{Y (s)>0} . The process FbnR is also written in an additive n

form (Pons, 2007) as a right-continuous increasing process identical to the product-limit estimator Z t dNn (s) FbnR (t) = (2.10) Pn −1 } bR 0 n− j=1 (1 − δj )1{Tj <s} {1 − (Fn (Tj ))

bn which is easily calculated. From the martingale property of the process Λ and Gill’s expression for the Kaplan-Meier estimator Z t FbnR − F 1 − FbnR (s− ) b (t) = − d(Λn − Λ)(s), t ≤ max Ti , (2.11) 1−F 1 − F (s) 0 it follows that Z t 1 − FbnR (s− ) b E {dΛn (s) − dΛ(s)} = 0, 1 − F (s) 0

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

41

Kernel estimator of a density

so the Kaplan-Meier estimator is unbiased and for every t ≤ max Ti , its Rt variance is V arFbnR (t) = (1 − F )2 (t) 0 g −1 dΛ + o(1). For every p ≥ 2, the Burkh¨ older-Davis-Gundy inequality implies the existence of a constant cp such that a bound for the Lp -risk of the Kaplan-Meier estimator is written E{F (t) − FbnR (t)}p ≤ cp {1 − F (t)}p E



Z

0

t

p/2 1 − FbnR (s− ) 2 −2 } Yn (s)dNn (s) 1 − F (s)

therefore kF (t) − FbnR (t)kp = O(n1/2 ). The density of T under right-censoring is estimated by smoothing the Kaplan-Meier estimator FbnR of the distribution function Z R fbn,h (t) = Kh (t − s) dFbnR (s) it is explicitly written using either its multiplicative expression Z R b n (s), (t) = fbn,h Kh (t − s)FbnR (s− ) dΛ IX,n,h

or the additive formula (2.10) R fbn,h (t) =

n X i=1

n−

Pn

j=1 (1

Kh (t − Ti )δi 1{Ti ≤t}

− δj )1{Tj <Ti } {1 − (FbnR (Tj ))−1 }

.

The a.s. uniform consistency of the process FbnR − F , for the Kaplan-Meier R estimator, implies that supIX,h |fbn,h − f | converges in probability to zero, as n tends to the infinity and h to zero. From (2.11), the estimator fbR n,h

satisfies

R fbn,h (t) =

=

Z

Z

Z

1 − FbnR− b d(Λn − Λ)} ds 1−F 0 b n − Λ)(s)] − {1 − FbnR (s− )} d(Λ

Kh (t − s)[f (s){1 +

s

b n − Λ)(s)] Kh (t − s) [dF (s) − {1 − FbnR (s− )} d(Λ

Z Z + {

u

h

Kh (t − s) dF (s)}

1 − FbnR− b n − Λ)(u). (u) d(Λ 1−F

R The bias of the estimated density fbn,h (t) is then the same as in the uncen-

sored case bf,n,h (t) =

h2 (2) (t) 2 f

+ o(h2 ), since the Kaplan-Meier estimator

January 31, 2011

17:17

42

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

b n are unbiased estimators. Its variance develops as and Λ Z Z  s 1 − FbnR− R b n − Λ) 2 ds vf,n,h (t) = E Kh2 (t − s)f 2 (s) d(Λ 1−F 0 Z + E Kh2 (t − s){1 − Fbn (s− )}2 Yn−1 (s) dΛ(s) Z

(1 − FbnR− )2 (s)Yn−1 (s) dΛ(s) Kh2 (t − s) 1−F Z t ≤ (nh)−1 κ2 {f 2 (t) g −1 dΛ + (1 − FC )−1 (t)f (t) − 2E

0

− 2f (t)g −1 (t) + o(1)},

where E

Z 

s

0

2 1 − FbnR− b d(Λn − Λ)} ≤ E 1−F

Z

0

s

1 − FbnR− 2 −1 Yn dΛ. 1−F

R (t) = (nh)−1 vfR (t). It follows that the The variance is then written vf,n,h optimal local and global bandwidths for the estimation of the density under right-censoring are O(n−1/5 ) and the optimal L2 -risks are O(n−2/5 ).

2.9

Estimation of the density of left-censored variables

Let X be left-censored by C, such that P (C < X) is strictly positive, then the observations are T = X ∨ C, δ = 1{C<X} . The notations Nn and Yn for a sample of n independent and identically distributed observations of (Ti , δi )1≤i≤n are unchanged, with this definition of the variables. The cumulative hazard function λ used for right-censoring is replaced by a cumulative retro-hazard function (Pons, 2008) Z ∞ dF ¯ Λ(t) = − F t R ∞ dF P ∆F (s) ¯ or Λ(t) = which uniquely defines the distribution − + − t

F

s>t F (s )

function F of X, for t > mini Ti , as Y ¯ c (t)} {1 + ∆Λ(s)}, ¯ F (t) = exp{Λ s>t

Q ¯ c (t) is the continuous part of Λ(t) ¯ ¯ where Λ and s>t {1 + ∆Λ(s)} its rightcontinuous discrete part. On the interval In = ]mini Ti , maxi Ti ], the func¯ is estimated by tion Λ Z ∞ dNn b ¯ Λn (t) = 1{Yn < n} n − Yn t

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Kernel estimator of a density

43

and a product-limit estimator of the function F is defined on In from the b ¯ n by expression of Λ oδi Yn b ¯ n (Ti ) FbnL (t) = 1 + dΛ . Ti ≥t

On the interval In = ]mini Ti , maxi Ti ] it satisfies Z ∞ bL − F − FbnL Fn (s ) b ¯ n (s) − dΛ(s)}, ¯ (t) = {dΛ (2.12) F F (s) t and n1/2 (F − FbnL )F −1Rconverges weakly to a centered Gaussian process with ¯ t) = ∞ (F −1 F − )2 (H − )−1 dΛ, ¯ with 1 − H = (1 − F )(1 − covariance K(s, s∧t b ¯ n is an unbiased estimator of Λ ¯ FC ). From this expression, it follows that Λ L b and Fn is an unbiased estimator of the distribution function F , moreover kFbnL (t) − F (t)kp = O(n1/2 ), for p ≥ 2. The density of T under left-censoring is estimated by smoothing the Kaplan-Meier estimator FbnL of the Z distribution function L b f (t) = Kh (t − s) dFbL (s). n,h

n

The a.s. uniform consistency of the process FbnL − F implies that L supIX,h |fbn,h − f | converges in probability to zero, as n tends to the inL finity and h to zero. From (2.12), the estimator fbn,h satisfies Z Z ∞ bL− Fn L b ¯ n − Λ)} ¯ ds fbn,h (t) = fn,h (t) + Kh (t − s)[f (s){ d(Λ F s b ¯ n − Λ)(s)] ¯ − FbnL− (s) d(Λ Z ∞ Fb L (s− ) b ¯ n − Λ)(s) ¯ = fn,h (t) + fn,h (s) n d(Λ F (s) t Z b ¯ n − Λ)(s). ¯ − Kh (t − s)FbnL (s− ) d(Λ

As a consequence of the uniform consistency of the estimators FbnL− and b ¯ n , the bias of the estimated density fbL (t) is then the same as in the Λ n,h 2

uncensored case bf,n,h (t) = h2 f (2) (t) + o(h2 ). Its variance is written L vf,n,h (t) = (nh)−1 vfL (t), with the expansion Z ∞  Fb L (s− ) 2 L 2 ¯ vf,n,h (t) = E fn,h (s) n (n − Yn (s))−1 dΛ(s) F (s) t Z ¯ + Kh2 (t − s)FbnL2 (s− )(n − Yn (s))−1 dΛ(s) −2

Z

t

∞

fn,h (s)

FbnL2 (s− ) ¯ Kh (t − s)(n − Yn (s))−1 dΛ(s) F (s)

January 31, 2011

17:17

44

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

where the last two terms are O((nh)−1 ) and the first one is a O(n−1 ). The optimal bandwidths for estimating the density under left-censoring are then also O(n−1/5) and the optimal L2 -risks are O(n−2/5 ). Under Conditions 2.1 or 2.2 and if the support of K is compact, the variance vfL belongs to class C2 (IX ) and for every t and t0 in IX,h , there exists a constant α such that for |t − t0 | ≤ 2h  L 2 L E fbn,h (t) − fbn,h (t0 ) ≤ α(nh3 )−1 |t − t0 |2 .

L L Under the conditions of Theorem 2.1, the process Un,h = (nh)1/2 {fbn,h − L 1/2 L f }I{IX,h } converges weakly to Wf + γ bf , where Wf is a continuous Gaussian process on IX with mean and covariances zero and with variance function vfL .

2.10

Kernel estimator for the density of a process

Consider a continuously observed stationary process (Xt )t∈[0,T ] with values in IX . The stationarity means that the distribution probability of Xt and Xt+s − Xs are identical for every s and t > 0. For a process with independent increments, this implies the ergodicity of the process that is expressed by the convergence of bounded functions of several observations of the process to a mean value: For every x in IX , there exists a measure πx on IX \ {x} such that for every bounded and continuous function ψ on 2 IX Z Z ET −1 ψ(Xs , Xt ) ds dt → ψ(x, y) dπx (dy)dF (x) (2.13) 2 IX

[0,T ]2

as T tends to infinity. The distribution function F in (2.13) is defined as the limit of the expectation of the mean sample-path of the process X Z Z −1 ET ψ(Xt ) dt → ψ(x) dF (x). (2.14) [0,T ]

IX

The mean marginal density f of the process is the density of the distribution function F , it is estimated by replacing the integral of a kernel function with respect to the empirical distribution function of a sample by an integral with respect to the Lebesgue measure over [0, T ] and the bandwidth sequence is indexed by T . For every x in IX,T,h Z 1 T b fT,h (x) = Kh (Xs − x) ds, (2.15) T 0

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Kernel estimator of a density

45

R its expectation is fT,h (x) = IX,n Kh (y − x)f (y) dy so its bias is Z hs bT,h (x) = Kh (y − x){f (y) − f (x)} dy = T msK f (s) (x) + o(hsT ) s! IX,T under Conditions 2.1-2.2. For a density in a H¨older class Hα,M , bT,h (x) [α] Rtends[α]to zero for every α > 0 and it is a O(h ) under the condition |u| K(u) du < ∞. Its variance is expressed through the integral of the covariance between Kh (Xs − x) and Kh (Xt − x). For Xs = Xt , the integral on the diagonal 2 DX of IX,T is a (T hT )−1 κ2 f (x) + o((T hT )−1 ) and the integral outside the diagonal denoted Io (T ) is expanded using the ergodicity property (2.13). Let αh (u, v) = |u − v|/2hT , the integral Io (T ) is written Z Z ds dt Kh (u − x)Kh (v − x)fXs ,Xt (u, v) du dv 2 T T 2 [0,T ] IX,T \DX Z Z Z 1−αh (u,v) = (T hT )−1 { K(z − αh (u, v))K(z + αh (u, v)) dz IX

IX\{u}

−1+αh (u,v)

dπu (v) dF (u)}{1 + o(1)} . R For every fixed u 6= v, the integral K(z −αh (u, v))K(z +αh (u, v)) dz tends to zero since αhT (u, v) tends to infinity as hT tends to zero. If αh (u, v) tends to zero with hT , then πu (v) also tends to zero and the integral Io (T ) is a o((T hT )−1 ) as T tends to infinity. The mean squared error of the estimator at x for a marginal density in Cs is then −2 2 M ISET,h (x) = (T hT )−1 κ2 f (x) + h2s msK f (s)2 (x) T (s!)

+ o((T hT )−1 ) + o(h2s T ) and the optimal local and global bandwidths minimizing the mean squared (integrated) errors are O(T 1/(2s+1) ). If hT has the rate of the optimal bandwidths, the M ISE is a O(T 2s/(2s+1) ). The Lp -norm of the estimator satisfies kfbT,h (x)−fT,h (x)kp = O((T hT )−1/p ) under an ergodicity condition k for (Xt1 , . . . , Xtk ) similar to (2.13) for bounded functions ψ defined on IX Z ψ(Xt1 , . . . , Xtk ) dt1 . . . dtk (2.16) ET −1 [0,T ]k Z Y → ψ(x1 , . . . , xk ) πxj (dxj+1 ) dF (x1 ), k IX

1≤j≤k−1

for every integer k = 2, . . . , p. The property (2.16) implies the weak convergence of the finite dimensional distributions of the process (T hT )1/2 (fbT,h −

January 31, 2011

17:17

46

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

f − bT,h ) to those of a centered Gaussian process with mean zero, covariances zero and variance κ2 f (x) at x. The proof is similar to the proof for a sample of variables, using the above expansions for the variance and covariances of the process. A lipschitzian bound for increments E{fbT,h (x) − fbT,h (y)}2 is obtained by the Mean Value Theorem which imRT plies T −2 0 E{Khn (x − Xt ) − Khn (y − Xt )}2 dt = O(|x − y|2 (T h3T )−1 ) as in Lemma 2.1. Then the process (T hT )1/2 (fbT,h − f − bT,h ) converges weakly to a centered Gaussian process with covariances zero and variance κ2 f . The Hellinger distance h2 (fbT,hT , f ) is bounded like (2.8) h2T (fbT,hT , f ) =

where

Z

(1 −

s

s Z fbT,hT fbT,hT ) dP ≤ ( − 1) d(FbT − F ) f f

FbT (t) = T −1

Z

T

1{Xt ≤s} dt 0

is the empirical probability distribution of the mean marginal distribution function of the process (Xt )t≤T Z −1 FT = T FXt dt, [0,T ]

and F is its limit √ under the ergodicity property (2.14). The convergence rate of FbT −F is T , from the mixing property of the process X. Therefore 1/2 h2 (fbT,hT , f ) convergences to zero in probability with the rate T hT . 2.11

Exercises

R (1) Let f and g be real functions defined on R R and let f ∗ g(x) = f (x − y)g(y) dy be their convolution. Calculate f ∗ g(x) dx and prove that, for 1 ≤ p ≤ ∞, if f belongs to Lp and g to Lq such that p−1 + q −1 = 1, then supx∈R |f ∗ g(x)| ≤ kf kp kgkq . Assume p is finite and prove that f ∗ g belongs to the space of continuous functions on R tending to zero at infinity. (2) Prove the approximation of the bias in (d) of Proposition 2.2 using a Taylor expansion and precise the expansions for the Lp -risk. (3) Prove the results of Equation (2.5).

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a density

FunctionalEstimation

47

(4) Write the variance of the kernel estimator for the marginal density of dependent observations (Xi )i≤n in terms of the auto-covariance coeffiPn cients ρj = n−1 i=1 Cov(Xi , Xi+j ). (5) Consider a hierarchical sample (Xij , Yij )j=(1,...,Ji ),i=1,...,n , with n independent and finite sub-samples of Ji dependent observations. Let Pn P n P Ji −1 N = i=1 Ji and f = limn N i=1 j=1 fXij be the limiting marginal mean density of the observations of X. Define an estimator of the density f and give the first order approximation of the variance of the estimator R x under relevant ergodicity conditions. (6) Let H(x) = −1 k(y) dy be the integrated kernel, F be the distribuPn tion function of X and Fbnh (x) = n−1 i=1 Hh (Xi − x) be a smooth estimator of the distribution function. Prove that the bias of Fbnh (x) is 12 h2 m2K f (1) (x) + o(h2 ) and its variance (nh)−1 κ2 F (x) + o((nh)−1 ). Define the optimal local and global bandwidths for Fbnh .

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Chapter 3

Kernel estimator of a regression function

3.1

Introduction and notation

The kernel estimation of nonparametric regression functions is related to the estimation of the conditional density of a variable and most authors have studied the asymptotic behaviour of weighted risks, using weights proportional to the density estimator so that the random denominator of the regression function disappears. Weighted integrated errors are used for the empirical choice of a bandwidth and for tests about the regression. In this chapter, the bias, variance and norms of the kernel regression estimator are obtained from a linear approximation of the estimator. Let (Xi , Yi )i=1,...,n be a sample of a variable R (X, Y ) with joint density fX,Y . The marginal density of X is fX (x)= fX,Y (x, y)dy and the den−1 sity of Y conditionally on X is fY |X =fX fX,Y . Here, the density fX,Y is supposed to be C2 . Let FXY be the distribution function of (X, Y ) Pn and FbXY,n (x, y)= n−1 i=1 1{Xi ≤ x, Yi ≤ y} be their empirical distribution function. Consider the regression model (1.6) Y = m(X) + σε where m is a bounded function and the error variable ε has the conditional mean E(ε|X) = 0 and a constant conditional variance V ar(ε|X) = σ 2 . Let IX and IXY be respectively subsets of the supports of the distribution functions FX and FXY , and let IX,h = {x ∈ IX ; [x − h, x + h] ∈ IX }, IXY,h = {(x, y) ∈ IXY ; [x − h, x + h] × {y} ∈ IXY } be subsets of the supports. On an interval IXY,h , a continuous and bounded 49

January 31, 2011

50

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

regression function −1 m(x) = E(Y |X = x) = fX

Z

yfXY (x, y) dy

is estimated by the kernel estimator Pn Yi Kh (x − Xi ) m b n,h (x) = Pi=1 . n i=1 Kh (x − Xi )

(3.1)

Its numerator is denoted Z n 1X Yi Kh (x − Xi ) = yKh (x − s) dFbXY,n (s, y) µ bn,h (x) = n i=1

and its denominator is fbX,n,h (x). The mean of µ bn,h (x) and its limit are respectively Z Z µn,h (x) = yKh (x − s) dFXY (s, y), Z µ(x) = yfXY (x, y) dy = fX (x)m(x),

whereas the mean of m b n,h (x) is denoted mn,h (x). The notations for the parameters and estimators of the density f are unchanged. The variance of Y is supposed to be finite and its conditional variance is denoted σ 2 (x) = E(Y 2 |X = x) − m2 (x), Z −1 E(Y 2 |X = x) = fX (x)w2 (x) = y 2 fY |X (y; x) dy, with Z Z 2 w2 (x) = y fXY (x, y) dy = fX (x) y 2 fY |X (y; x) dy.

Let also σ4 (x) = E[{Y −m(x)}4 | X = x], they are supposed to be bounded functions. The Lp -risk of the kernel estimator of the regression function m is defined by its Lp -norm k · kp = {Ek · kp }1/p . 3.2

Risks and convergence rates for the estimator

The following conditions are assumed, in addition to Conditions 2.1 and 2.2 about the kernel and the density. Condition 3.1. (1). The functions fX , m and µ are twice continuously differentiable on IX , with bounded second order derivatives; fX is strictly positive on IX ; (2). The functions fX , m and σ belong to the class Cs (IX ).

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a regression function

FunctionalEstimation

51

Proposition 3.1. Under Conditions 2.1, 2.2 and 3.1(1), (a). supx∈IX,h |b µn,h (x)− µ(x)| and supx∈IX,h |m b n,h (x)− m(x)| converge a.s. to zero if and only if µ and m are uniformly continuous. (b). The following expansions are satisfied µn,h (x) + O((nh)−1 ), fX,n,h (x)
1/2
1/2 −1  nh {m b n,h − mn,h }(x) = nh fX (x) (b µn,h − µn,h )(x) (3.2) b − m(x)(fX,n,h − fX,n,h )(x) + rn,h mn,h (x) =

where rn,h = oL2 (1). (c) For every x in IX and for every integer p > 1, kb µn,h (x) − µ(x)kp and km b n,h (x) − m(x)kp converge to zero, the bias of the estimators µ bn,h (x) and m b n,h (x) is uniformly approximated by bµ,n,h (x) = µn,h (x) − µ(x) = h2 bµ (x) + o(h2 ), Z m2K (2) m2K ∂ 2 fXY (x, y) bµ (x) = µ (x) = y dy, 2 2 ∂x2 bm,n,h (x) = mn,h (x) − m(x) = h2 bm (x) + o(h2 ),

−1 bm (x) = fX (x){bµ (x) − m(x)bf (x)} m2K −1 (2) = f (x){µ(2) (x) − m(x)fX (x)}, 2 X the covariance between µ bn,h (x) and fbX,n,h (x) is

Covµ,fX ,n,h (x) = (nh)−1 {Covµ,fX (x) + o(1)}, Covµ,fX (x) = µ(x)κ2 = m(x)fX (x)κ2

(3.3)

(3.4)

(3.5)

and their variance vµ,n,h (x) = (nh)−1 {σµ2 (x) + o(1)}, σµ2 (x) = w2 (x)κ2 , −1

vm,n,h (x) = (nh) 2 σm (x)

2 (x) {σm 2

(3.6)

+ o(1)},

−2 = {w2 (x) − m (x)f (x)}κ2 fX (x)

−1 = κ2 f X (x)σ 2 (x).

(3.7)

Proof. Note that Condition 3.1 implies that the kernel estimator of fX is bounded away from zero on IX which may be a sub-interval of the support of the variable X. Proposition 2.2 and the almost sure convergence to zero of supx∈IX,h |b µn,h − µn,h |, proved by the same arguments as for the density, imply the assertion (a). The bias and the variance are similar for

January 31, 2011

52

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

µ bn,h (x) and fbX,n,h (x). For µ bn,h (x), they are a consequence of (b). The first approximation of (b) comes from the expansion m b n,h (x) = =

µn,h (x) (b µX,n,h − µX,n,h )(x) + fX,n,h (x) fX,n,h (x) m b n,h (x){fbX,n,h (x) − fX,n,h (x)} − fX,n,h (x)

µn,h (x) (b µX,n,h − µX,n,h )(x) µ bn,h (x)(fbX,n,h − fX,n,h )(x) + − 2 fX,n,h (x) fX,n,h (x) fX,n,h (x) + −

m b n,h (x)(fbX,n,h − fX,n,h )2 (x) , 2 fX,n,h (x)

(b µn,h − µn,h )(x)(fbX,n,h − fX,n,h )(x) , 2 fX,n,h (x)

the expectation of this equality yields mn,h (x) =

µn,h (x) (b µn,h − µn,h )(x)(fbX,n,h − fX,n,h )(x) −E 2 fX,n,h (x) fX,n,h (x) +E

=

m b n,h (x){fbX,n,h (x) − fX,n,h (x)}2 2 fX,n,h (x)

µn,h (x) µn,h (x) + O((nh)−1 ) = + o(h2 ) fX,n,h (x) fX,n,h (x)

(3.8)

uniformly on IX , for any bounded regression function m. The bias of m b n,h (x) is bm,n,h (x) =

 µn,h (x)  µn,h (x) − m(x) + mn,h (x) − , fX,n,h (x) fX,n,h (x)

where the second difference is a o(h2 ), using (3.8). A second order Taylor −1 expansion of fX,n,h (x) as n tends to infinity leads to µn,h (x) −1 = m(x) + {bµ,n,h (x) − m(x)bfX ,n,h (x)}fX (x) + o(h2 ) fX,n,h (x) and the bias of m b n,h (x) follows immediatly. The variance vm,n,h (x) of m b n,h (x) is

 µn,h (x) 2  µn,h (x) 2 vm,n,h (x) = E m b n,h (x) − − mn,h (x) − , fX,n,h (x) fX,n,h (x)

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

53

Kernel estimator of a regression function

where the non random term is a o(h4 ), by (3.8). The first term develops using twice the equality y −1 = x−1 − (y − x)(xy)−1  µn,h (x) fX,n,h (x) m b n,h (x) − =µ bn,h (x) − µn,h (x) fX,n,h (x)  − mn,h (x) fbX,n,h (x) − fX,n,h (x)   µ bn,h (x) − µn,h (x) fbX,n,h (x) − fX,n,h (x) − fX,n,h (x)  2 m b n,h (x) fbX,n,h (x) − fX,n,h (x) , + fX,n,h (x)

so that

(3.9)

 µn,h (x) 2 2 fX,n,h (x)E m b n,h (x) − = V ar{b µn,h (x)} fX,n,h (x) + m 2 (x)V ar{fbX,n,h (x)} − 2mn,h (x)Cov{b µn,h (x), fbX,n,h (x)} n,h

mn,h (x) π0,2,1 (x) π0,2,2 (x) + 2 +2 π0,1,2 (x) − 2 fX,n,h (x) fX,n,h (x) fX,n,h (x)

+

π2,0,4 (x) π1,1,2 (x) π1,0,3 (x) +2 − 2mn,h (x) 2 fX,n,h (x) fX,n,h (x) fX,n,h (x)

−2

π1,1,3 (x) , 2 fX,n,h (x)

where  k 0 00  b n,h (x){b µn,h (x) − µn,h (x)}k {fbX,n,h (x) − fX,n,h (x)}k πk,k0 ,k00 (x) = E m

for k ≥ 0, k 0 ≥ 0 and k 00 ≥ 0. Since m b n,h (x) is bounded, Cauchy-Schwarz inequalities and the order of the moments of µ bn,h (x) and fbn,h (x) imply that 0 00 all terms πk,k0 ,k00 (x) in the above expression are O((nh)−(k +k )/2 so they are o((nh)−1 ) except the covariance term π0,1,1 (x). Using the first order expansions of the means fX,n,h (x) = fX (x) + O(h2 ) the mean develops as mn,h (x) = m(x) + O(h2 ). It follows that  −2  vm,n,h (x) = fX (x) V ar{b µn,h (x)} + m2 (x) V ar{fbX,n,h (x)}  − 2m(x) Cov{b µn,h (x), fbX,n,h (x)} + o(n−1 h−1 ) and the convergence to zero of the last term rn,h in (3.2) is satisfied. The other results are obtained by simple calculus. 

The minimax property of the estimator m b n,h is established by the same method as for density estimation.

January 31, 2011

17:17

54

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

For p ≥ 2, let wp (x) = E(Y p 1{X=x} )

(3.10)

be the p-th moment of Y conditionally on X = x. The Lp risk is calculated from the approximation (3.2) of Proposition 3.1 and the next lemmas. Lemma 3.1. For p ≥ 2 kb µn,h (x) − µn,h (x)kp = O((nh)−1/p ) and −1 −1 kfbn,h (x) − fX,n,h (x)kp = O((nh)−1/p ),

where the approximations are uniform.

Proof. By the expansion (3.9), Proposition 2.2 extends to µ bn,h (x) − µn,h (x) and the moments of order p ≥ 2 of µ bn,h (x) − µn,h (x) and fbX,n,h (x) − fX,n,h are 0((nh)−1/p ) which is decreasing as p increases. Let −1 b an = fn,h {fn,h − fn,h }, then X −p −p −1 −1 p {fbn,h − fn,h } = fn,h {(1 + an )−1 − 1}p = fn,h { (−an )k }p k≥1

and the decreasing order of the moments of the kernel estimator of the density implies X E| (−an )k |p = E|an |p + o(E|an |p ). k≥1



The convergence rate of the bandwidth determines the behaviour of the bias term of the process (nh)1/2 (m b n,h − m), with the following technical results. They generalise Proposition 3.1 to p and s ≥ 2.

Lemma 3.2. Under Conditions 2.1, the bias of µ bn,h (x) and m b n,h (x) are uniformly approximated as Z hs ∂ s fX,Y (x, y) bµ,n,h (x; s) = msK y dy + o(hs ), s! ∂xs hs (s) −1 bm,n,h (x; s) = msK fX (x){µ(s) (x) − m(x)fX (x)} + o(hs ), (3.11) s!

for s ≥ 2, and their variances develop as in Proposition 3.1.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

55

Kernel estimator of a regression function

Proposition 3.2. Under Conditions 2.1 and 3.1 with s = 2, for every x in IXh (nh)1/2 (m b n,h − m) = (nh)1/2 f −1 {(b µn,h − µn,h ) − m(fbX,n,h − fX,n,h )} X

+ (nh5 )1/2 bm + rbn,h ,

(3.12)

and the remainder term of (3.12) satisfies

sup kb rn,h k2 = O((nh)−1/2 ).

x∈IX,h

Proof. rbn,h

Expanding (3.2) yields −1 −1 = (nh)1/2 (fbX,n,h − fX ){(b µn,h − µn,h ) − m(fbX,n,h − fX,n,h )}
−1 −1 + (nh)1/2 fbX,n,h fX,n,h fX,n,h µn,h − m − (nh5 )1/2 bµ
−1 −1 = (nh)1/2 fbX,n,h − fX {(b µn,h − µn,h ) − m(fbX,n,h − fX,n,h )}
−1 + (nh)1/2 µn,h fX,n,h − m − h2 b µ (3.13) −1 µn,h − m + (nh)1/2 fX,n,h


X

k≥1

−

fbX,n,h − fX,n,h
k . fX,n,h

By Lemma 3.1 and Proposition 3.1, the first term is a O((nh)−1/2 ). The second order uniform approximation −1 fX,n,h (x)µn,h (x) − m(x) = h2 bµ (x) + O(h4 ),

(3.14)

implies that the second term in the sum is a O(h4 ) = O((nh)−1 ), as a consequence of Condition 2.1. By Lemma 3.1 and (3.14), the third term is a O((nh)1/2 h2 (nh)−1/2 ), it is therefore a O((nh)−1/2 ).  For a regression function of class Cs , s ≥ 2, the L2 -norm of the remainder term rbn,h is given by the next proposition.

Proposition 3.3. Under Conditions 2.1, 2.2 and 3.1, for every s ≥ 2 the remainder term of (3.12) satisfies the uniform bounds sup kb rn,h k2 = O((nh)−1/2 ).

IX,h

Proof. For functions fX and µ in Cs , the risk of rbn,h is modified by the bias terms of the previous expansion (3.13). The second term in the −1 approximation (3.14) is replaced by fX,n,h (x)µn,h (x) − m(x) = hs bµ (x) + O(hs+1 ) and Conditions 2.2 and 3.2, which implies h2s = O((nh)−1 ). By Lemma 3.2, supx E|b rn,h (x)|2 is bounded by O(nh)[{O(h2s + (nh)−1 )} O((nh)−1 ) + O(h2(2+s) ) + O(h2s (nh)−1 )] which is a O(h2s ) + O(h4 ) + O(h2s ) = O(h4 ).



January 31, 2011

56

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Propositions 2.2, 3.1, Equation (3.2), Propositions 3.2 and 3.3 determine an upper bound for the norm km b n,h − mn,h kp of the estimator of m −1 km b n,h − mn,h kp = kfX {(b µn,h − µn,h ) − m(fbX,n,h − fX,n,h )}kp + O((nh)−1/2 kb rn,h kp ),

−1 ≤ 2p−1 [sup fX {kb µn,h − µn,h kp IX

+ sup |m| kfbX,n,h − fX,n,h kp }] + O((nh)−1/2 kb rn,h kp ), IX

it is therefore a O((nh)−1/2 ). The expression of the Lp -norm km b n,h −mn,h kp is obtained by similar expansions and approximations as in the proof of Proposition 3.1. Under Conditions 2.1, 2.2, 3.1, for a function µ in Cs and a density fX in Cr , the bias of m b n,h is hs hr −1 bm,n,h (x) = fX (x){ bµ (x) − m(x) bf (x)} + o(hs∧r ) s! r! and its variance does not depend on r and s. The derivability conditions fX and µ ∈ Cs of 3.1 can be replaced by the condition: fX and µ belong to a H¨older class Hα,M . Proposition 3.4. Assume fX and µ are bounded and belong to Hα,M then −1 the bias of m b n,h (x) is bounded by M m[α]K hα /([α]!)fX (x){1 + |m(x)|} + α o(h ), by equation (3.2). The optimal bandwidth is O(n1/(2α+1) ) and the MISE at the optimal bandwidth is O(nα/(2α+1) ).

3.3

Optimal bandwidths

The asymptotic mean squared error of m b n,h (x), for p = 2, is −2 −1 2 4 2 −1 (x){w2 (x) − m2 (x)f (x)} (nh) σm (x) + h bm (x) = (nh) κ2 fX

h4 m22K −2 (2) fX (x){µ(2) (x) − m(x)fX (x)}2 4 and its minimum is reached at the optimal bandwidth  κ2 n−1 {w2 (x) − m2 (x)f (x)} 1/5 hAMSE (x) = m22K {µ(2) (x) − m(x)f (2) (x)}2 +

X

where AM SE(x) = O(n−4/5 ). The global mean squared error criterion is the integrated error and it isZapproximated by AM ISE = (nh)−1 κ2 +

h4 m22K 4

Z

−2 fX (x){w2 (x) − m2 (x)f (x)} dx (2)

−2 fX (x){µ(2) (x) − m(x)fX (x)}2 dx

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Kernel estimator of a regression function

57

and the optimal global bandwidth is R −1  κ2 n−1 fX (x)V ar{Y |X = x}f (x)} dx 1/5 hn,AMISE = . R −2 2 m2K f (x){µ(2) (x) − m(x)f (2) (x)}2 dx X

X

For every s ≥ 2, the asymptotic quadratic risk of the estimator for a regression curve of class Cs is 2 AM SE(x) = (nh)−1 σm (x) + hs2 b2m,s (x) −2 = (nh)−1 κ2 fX (x){w2 (x) − m2 (x)f (x)}

+

h2s 2 −2 (s) m f (x){µ(s) (x) − m(x)fX (x)}2 , (s!)2 sK X

its minimum is reached at the optimal bandwidth n (s!)2 κ n−1 {w (x) − m2 (x)f (x)} o1/(2s+1) 2 2 hAMSE (x) = 2sm2sK {µ(s) (x) − m(x)f (s) (x)}2 X where AM SE(x) = O(n−2s/(2s+1) ). The global mean squared error criterion is the integrated error and it is approximated by Z −1 AM ISE(h, s) = (nh)−1 κ2 fX (x)V ar{Y | X = x} dx Z h2s m2sK (s) −2 + fX (x){µ(s) (x) − m(x)fX (x)}2 dx (s!)2 and the optimal global bandwidth is R −1 n (s!)2 κ n−1 fX (x)V ar{Y | X = x} dx o1/(2s+1) 2 hn,AMISE (s) = , R 2sm2sK f −2 (x){µ(s) (x) − m(x)f (s) (x)}2 dx X X

and again AM ISE(hn (s), s) = O(n−2s/(2s+1) ). In order to estimate the constants of the optimal bandwidths, a nonparametric estimator of the conditional variance of Y are defined as Pn Yi2 Kh (x − Xi ) Vb arn,h (Y |X = x) = Pi=1 −m b 2n,h (x). n K (x − X ) h i 2 i=1 More generally, the conditional moment of order p, mp (x) = E(Y p |X = x) is estimated by Pn p i=1 Yi Kh (x − Xi ) m b p,n,h (x) = P n i=1 Kh (x − Xi )

with a bandwidth h = hn such that hn tends to zero and nh2n tends to infinity as n tends to infinity. For every p ≥ 2, the estimator m b p,n,h is −1 also written fbn,h µ bp,n,h , it is a.s. uniformly consistent and approximations

January 31, 2011

17:17

58

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

similar to those of Propositions 3.1 and 3.2 for the regression curve hold for mp (x) −1 (nh)1/2 (m b p,n,h − mp,n,h ) = (nh5 )1/2 bp,m + (nh)1/2 fX {(b µp,n,h − µp,n,h ) b − mp (fX,n,h − fX,n,h )} + rbp,n,h ,

sup kb rp,n,h k2 = O((nh)−1/2 )

x∈IX,h

and for its bias

Z

∂ 2 fXY (·, y) dy + o(h2 ), ∂x2 m2K h2 −1 bmp ,n,h = mp,n,h − mp = fX {bµp ,n,h − mp bf } + o(h2 ). 2 The covariance between µ bp,n,h (x) and fbX,n,h (x) is (nh)−1 mp (x)fX (x)κ2 and the variances of the estimators of µp (x) and mp (x) are bµp ,n,h

m2K h2 = 2

yp

vµp ,n,h (x) = (nh)−1 {wp (x)κ2 + o(1)},

−2 vmp ,n,h (x) = (nh)−1 {κ2 fX (x) {σµ2 p (x) − m2p (x)f (x)} + o(1)}.

The estimators of the derivatives of the regression function m are Pn (1) (1) i=1 Yi Kh (x − Xi ) m b n,h (x) = P n i=1 Kh (x − Xi ) Pn Pn (1) { i=1 Yi Kh (x − Xi )}{ i=1 Kh (x − Xi )} Pn − , { i=1 Kh (x − Xi )}2 (1) (1) −1 = fbX,n,h (x){b µn,h (x) − m b n,h (x)fbX,n,h (x)} (3.15)

and all consecutive derivatives of this expression. The first derivatives (1)

and

(1) fbX,n,h (x)

expectations where

=n

µ bn,h (x) = n−1

Pn −1

(1) µn,h (x)

n X i=1

(1)

Yi Kh (x − Xi )

(1) i=1 Kh (x−Xi ) converge uniformly on (1) (1) = h−1 EY Kh (x − X) and fX,n,h (x),

h2 (1) (1) (3) fX,n,h (x) = fX (x) + m2K fX (x) + o(h2 ), 2 Z (1) −1 µn,h (x) = h yKh0 (u − x)fX,Y (u, y) du dy = (mfX )(1) (x) +

IX,h to their

respectively,

h2 m2K (mfX )(3) (x) + o(h2 ), 2

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

59

Kernel estimator of a regression function (1)

(1)

−1 then m b n,h converges uniformly to fX (x){(mfX )(1) − mfX } = m(1) , as h (1)

tends to zero. The bias of m b n,h (x) is

h2 (3) −1 m2K fX (x){(mfX )(3) − mfX }(x). 2 Its variance is obtained by an application of Proposition 3.1 to equation (3.15), its convergence rate is (nh3 )−1 (see Appendix A) and the optimal global bandwidth for estimating m(1) follows. For the second derivative Pn (2) (2) i=1 Yi Kh (x − Xi ) m b n,h (x) = P n i=1 Kh (x − Xi ) Pn Pn (1) (1) { i=1 Yi Kh (x − Xi )}{ i=1 Kh (x − Xi )} Pn −2 { i=1 Kh (x − Xi )}2 Pn Pn (2) { i=1 Yi Kh (x − Xi )}{ i=1 Kh (x − Xi )} Pn − { i=1 Kh (x − Xi )}2 Pn Pn (1)2 { i=1 Yi Kh (x − Xi )}{ i=1 Kh (x − Xi )} Pn +2 , { i=1 Kh (x − Xi )}3 Pn (2) (2) (2) the estimators fbn,h and µ bn,h (x) = n−1 i=1 Yi Kh (x − Xi ) converge uni2

(4)

formly to f (2) and µ(2) , respectively, with respective biases h2 m2K fX (x)+ 2 o(h2 ) and h2 m2K µ(4) (x) + o(h2 ). The result extends to a general order of derivative k ≥ 1. Proposition 3.5. Under Conditions 2.2 and 3.1 with nh2k+2s+1 = O(1), (k) for k ≥ 1, and functions m and fX in class Cs (IX ), the estimator m b n,h is an uniformly consistent estimator of the k-order derivative of the regression function, its bias is a O(hs ), and its variance a O((nh2k+1 )−1 ), the optimal bandwidth is a O(n−1/(2k+2s+1) ).

The nonparametric estimator (3.1) is often used in nonparametric time series models with correlated errors. The bias is unchanged and the variance of the estimator depends on the covariances between the observation errors E(εi εi+a ) = βa , for a weakly stationary process (Yi )i corresponding to correlated measurements of Y = m(X)+ε. Now the variance σf2 is replaced P by S = σf2 +2 i≥1 βa assumed to be finite (Billingsley, 1968). A consistent Pm estimator of S was defined by Sbm = βbi where the correlation is i=−m

estimated by the mean correlation error with a mean over the lag between the terms of the product and a sum over observations and n−1 m2 tends to zero (Herrmann, Gasser and Kneip, 1992).

January 31, 2011

60

3.4

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Weak convergence of the estimator

The weak convergence of the process Un,h = (nh)1/2 {m b n,h − m}I{IX,h } relies on bounds for the moments of its increments which are first proved, as in Lemma 2.2 for the increments of the centered process defined by the kernel estimator, with a kernel having the compact support [−1, 1]. For a function or a process ϕ defined on IX,h , let ∆ϕ(x, y) = ϕ(x) − ϕ(y). Lemma 3.3. Under Conditions 3.1, there exist positive constants C1 and C2 such that for every x and y in IX,h and satisfying |x − y| ≤ 2h E|∆(b µn,h − µn,h )(x, y)|2 ≤ C1 (nh3 )−1 |x − y|2 ,

E|∆(m b n,h − mn,h )(x, y)|2 ≤ C2 (nh3 )−1 |x − y|2 ,

if |x − y| > 2h, they are O((nh)−1 ) and the estimators at x and y are independent. Proof. Let x and y in RIX,h such that |x − y| ≤ 2h, E|b µn,h (x) − µ bn,h (y)|2 −1 2 develops as the sum n w2 (u){Khn (x − u) − Khn (y − u)} f (u) du + (1 − n−1 ){µn,hn (x)−µn,hn (y)}2 . For an approximation of the integral, the Mean (1) Value Theorem implies Khn (x − u) − Khn (y − u) = (x − y)ϕn (z − u) where z is between x and y, and Z {Khn (x − u) − Khn (y − u)}2 w2 (u)f (u) du Z = (x − y)2 ϕ(1)2 n (z − u)w2 (u)f (u) du Z = (x − y)2 h−3 {w (x)f (x) K (1)2 + o(hn )}. 2 n

Let |x| ≤ hn and |y| ≤ hn , the order of the second moment E|fbn,h (x) − fbn,h (y)|2 is a O((x−y)2 (nh3n )−1 ) if |x−y| ≤ 2hn and it is the sum E µ b2n,h (x) 2 and µ bn,h (y) otherwise. This bound and Lemma 2.2 imply the same orders for the estimator of the regression function m. 

Theorem 3.1. For h > 0, the process Un,h = (nh)1/2 {m b n,h − m}I{IX,h } converges in distribution to σm W1 +γ 1/2 bm where W1 is a centered Gaussian process on IX with variance 1 and covariances zero.

Proof. For any x ∈ IX,h and from the approximation (3.2) of Proposition 3.1 and the weak convergences for µ bn,h − µn,h and fbX,n,h − fX,n,h , the variable Un,h (x) develops as (nh)1/2 {m b n,h (x) − mn,h (x)} + (nh5 )1/2 bm (x)+o((nh5 )1/2 ), and it converges to a non centered distribution {W + γ 1/2 bm }(x) where W (x) is the Gaussian variable with mean zero and

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a regression function

FunctionalEstimation

61

2 variance σm (x). In the same way, the finite dimensional distributions of the process Un,h converge weakly to those of {W + γ 1/2 bm }, where W is a Gaussian process with the same distribution as W (x) at x. The covariance matrix {σ 2 (xk , xl )}k,l=1,...,m between components W (xk ) and W (xl ) of the limiting process is the limit of   nh Cov Un,h (xk ), Un,h (xl ) = Cov{b µn,h (xk ), µ bn,h (xl )} fX (xk )fX (xl ) − m(xk )Cov{fbX,n,h (xk ), µ bn,h (xl )} − m(xl )Cov{b µn,h (xk ), fbX,n,h (xl )}  + m(xk )m(xl )Cov{fbX,n,h (xk ), fbX,n,h (xl )} + o(1) ,

where the o(1) is deduced from Propositions 3.1, 3.2 and 3.3. For every integers k and l, let αh = |xl − xk |/(2h) and v = {(xl + xk )/2 − s}/h be in [0, 1], hence h−1 (xk − s) = v − α and h−1 (xl − s) = v + α. By a Taylor expansion in a neighborhood of (xl + xk )/2, the integral of the first covariance term develops as Cov{b µn,h (xk ), µ bn,h (xl )} Z

x xk + xl
k + xl
= n−1 h−1 w2 fX K v − αh K v + αh dv 2 2 + o(n−1 h−1 )

and zero otherwise, with the notation (3.10). Similar expansions are satisfied for the other terms of the covariance. Using the following approximations for |xk − xl | ≤ 2h : w2 ({xk + xl }/2) = w2 (xk ) + o(1) = w2 (xl ) + o(1) and fX ({xk + xl }/2) = fX (xk ) + o(1) = fX (xl ) + o(1), the covariance of Un,h (xk ) and Un,h (xl ) is approximated by Z

V ar(Y |X = xk ) + V ar(Y |X = xl ) I{0≤αh <1} K (v − αh K v + αh dv. fX (xk ) + fX (xl ) Due to the compactness of the support of K, the covariance iszero if αh ≥ 1. For xk 6= xl , αh tends to infinity as h tends to zero and I 0 ≤ αh < 1 tends to zero as n tends to infinity, therefore the  covariance of Un,h (xk ) and Un,h (xl ) is equal to δk,l + o(1), where V ar Un,h (xk ) is defined in Proposition 3.1. The tightness of the sequence {Un,h } on IX,h will follow from (i) the tightness of {Un,h (a)} and (ii) a bound of the increments E Un,h (x2 ) − 2 Un,h (x1 ) for |x2 − x1 | < 2h. For condition (i), let η > 0 and c >
1/2 γ 1/2 |bm (a)| + 2η −1 σ 2 (a) , then  1/2 Pr{|Un,h (a)| > c} ≤ Pr (nh) |(m b n,h − mn,h )(a)| + (nh)1/2 |bn,h (a)| > c ≤

V ar{(nh)1/2 (m b n,h − mn,h )(a)} {c − (nh)1/2 |bn,h (a)|}2

January 31, 2011

62

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

and for n sufficiently large σ 2 (a) + o(1) < η. {c − γ 1/2 |bm (a)|}2 is written Wn,h + (nh)1/2 bn,h where

Pr{|Un,h (a)| > c} ≤ The process Un,h

(bn,h (x) − bn,h (y))2 ≤ kh2s (x − y)2s = O((nh)−1 )(x1 − x2 )2s

and Wn,h = (nh)1/2 (m b n,h −mn,h ). From Lemma 3.3, there exists a constant CW such that |x−y| ≤ 2h entails E{Wn,h (x)−Wn,h (y)}2 ≤ CW h−2 |x−y|2 , which implies the tightness of the process Un,h and its weak convergence to a continuous Gaussian process defined on IX .  Note that the tightness of the process implies the existence of a constant cη > 0 such that P

−1 Pr{ sup |σm (Un,h − γ 1/2 bm ) − W1 | > cη } → 0. IX,h

The limiting distribution of the process Un,h does not depend on the bandwidth h, so one can state the following corollary. −1 Corollary 3.1. suph>0:nh2s+1 →γ supIX,h σm |Un,h − γ 1/2 bm | converges in distribution to supIX |W1 |.

An uniform confidence interval for the regression curve m is deduced as for the density. Let X be a variable defined in a subset IX of Rd , the regression function m is estimated using a multivariate kernel K defined on [−1, 1]d and Kh (x) = h−d K(h−d x), for x = (x1 , . . . , xd ) in IX,h . The bias is unchanged and the rates of the moments p ≥ 2 are modified by the dimension d

−1 V ar{m b n,h (x)} = (nhd )−1 κ2 V ar{Y |X = x}fX (x) + o((nhd )−1 ), and kfbn,h (x) − fn,h (x)kp = 0((nhd )−1/p ). The local and global errors M ISEn (h) are O(h2s )+O((nhd )−1 ), they are minimal at the optimal bandwidths of order O(n−1/(2s+d) ) where the MISE reaches the minimal order O(n−2s/(2s+d) ). The weak convergence of Theorem 3.1 and its corollary still hold with the rate (nhd )1/2 .

3.5

Estimation of a regression curve by local polynomials

The regression function m is approximated by a Taylor expansion of order k, for every s in a neighborhood Vx,h of a fixed x, with radius h, (s − x)p (p) m + o((s − x)p ). (3.16) m(s) = m(x) + (s − x)m0 (x) + . . . + p!

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a regression function

FunctionalEstimation

63

This expansion is a local polynomial regression where the derivatives at x are considered as parameters. Estimating the derivatives by the derivatives of the estimator m b n,h yields an estimator having a variance sum of terms of different orders, its main term is the variance of m b n,h . Let (Hk,h )k be a square integrable orthonormal basis of real functions with respect to the distribution function of X, with support Vx,h for h converging to zero. Let δk,l be the Dirac indicator δk,l of equality for k and l, k, l ≥ 0. Equation (3.16) is also written m(s) =

p X

k=0

θk (x)Hk (s − x) + o((s − x)p ) = mp (x) + o((s − x)p )

for s in Vx,h , and the properties of the functional basis entail Z E{Hk (X − x)Hl (X − x)} = Hk (s − x)Hl (s − x) dF (s) = δk,l , k, l ≥ 0. In the regression model E(Y |X) = m(X)

θk (x) = E{Y Hk (X − x)} = E{Hk (X − x)m(X)}, k ≥ 1 m(x) = E{Y H0 (X − x)} = E{H0 (X − x)m(X)}.

For fixed x, θk (x) is considered as a constant parameter. This expansion is an extension of the kernel smoothing if the functional basis has regularity properties. The nonparametric regression function is approximated by an expansion R on the first p elements of the basis and its projections satisfy θk (x) = m(s)Hk (x − s) dF (s). The estimation of the parameters is performed by the projection of the observations of Y onto the first p elements of the orthonormal basis. Let (Xi , Yi )i=1,...,n be a sample for the regression variables (X, Y ), so that Yi = m(Xi ) + εi where εi is an observation error having a finite variance σ 2 = E{Y − m(X)} and such that E(ε|X) = 0. An estimator of the parameter is defined as the empirical conditional mean of the projection of Y onto the space generated by the basis. For k ≥ 1, θbk,n (x) = n−1

n X i=1

Yi Hk (Xi − x)

is therefore a consistent estimator of θk . Its conditional variance is n−1 {E(Y 2 |X)Hk2 (X − x) − θk2 }{1 + o(1)}. This approach may be compared to the local polynomials defined by minimizing the local smoothed empirical mean squared error ASE(x) =

n X i=1

{Yi − mp (Xi , θ)}2 Kh (Xi − x).

January 31, 2011

64

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

This provides an estimator of θ with components satisfying n X i=1

{Yi − mp (Xi , θ)}Hk (Xi − x)Kh (Xi − x) = 0.

They are solution of a system of linear equations and θnk is approximated by Pn i Hk (Xi − x)Kh (Xi − x) i=1 YP n i=1 Kh (Xi − x)

if the orthogonality of the basis entails that EHk (X −x)Hl (X −x)Kh (X −x) convergences to zero as h tends to zero, for every k 6= l ≤ p. This estimator is consistent and its behaviour is further studied by the same method as the estimator of the nonparametric regression. A multidimensional regression function m(X1 , . . . , Xd ) can be expanded in sums of univariate regression functions E(Y | Xk = x) and their interactions like a nonparametric analysis of variance if the regression variables (X1 , . . . , Xd ) generate orthogonal spaces generated. The orthogonality is a necessary condition for the estimation of the components of this expansion since Z E{Y Kh (xk − Xk )} = E(Y | X = x) FX (dx1 , . . . , xk−1 , xk+1 , . . . , xd ) + o(1)

= m(xk )fk (xk ) + o(1), E{Y Kh (xk − Xk )Kh (xl − Xl )} = CK m(xk , xl )fXk ,Xl (xk , xl ) + o(1), R where CK = K(u)K(v) du dv, and E{Y Kh (xk − Xk )Kh (xl − Xl )} −1 (x fX , x ) can be factorized or expanded as a sum of regression functions k l k ,Xl only if Xk and Xl belong to orthogonal spaces. The orthogonalisation of the space generated by a vector variable X can be performed by a preliminary principal component analysis providing orthogonal linear combinations of the initial variables.

3.6

Estimation in regression models with functional variance

Consider the nonparametric regression model with an observation error funtion of the regression variable X, Y = m(X) + σ(X)ε defined by (1.7), with E(ε|X) = 0 and V ar(ε|X) = 1. The variance σ(x)2 = E[{(Y − m(X)}2 |X = x] is assumed to be continuous and it is estimated by a

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

65

Kernel estimator of a regression function

localisation of the empirical error in a neighborhood of x Pn {Yi − m b n,h (Xi )}2 1{Xi ∈ Vδ (x)} 2 σ en,h,δ (x) = i=1 Pn i=1 1{Xi ∈ Vδ (x)} or by smoothing it with a kernel density Pn {Yi − m b n,h (Xi )}2 Kδ (x − Xi ) 2 . σ bn,h,δ (x) = i=1 Pn i=1 Kδ (x − Xi )

(3.17)

−1 2 The estimator is denoted σ bn,h,δ (x) = fbX,n,δ (x)Sbn,h,δ (x), with

Sbn,h,δ (x) = n−1 =

Z

n X i=1

{Yi − m b n,h (Xi )}2 Kδ (x − Xi )

{y − m b n,h (s)}2 Kδ (x − s) dFbX,Y,n (s, y).

The mean of Sbn,h,δ (x) is denoted Sn,h,δ (x). By the uniform consistency of m b n,h , Sbn,h,δ converges uniformly to S as n tends to infinity, with h P and δ tending to zero. At Xj , it is written Sbn,h,δ (Xj ) = n−1 i6=j {Yi − m b n,h (Xi )}2n,h Kδ (Xj − Xi ) + o((nh)−1 ). The rate of convergence of δn to zero is governed by the degree of derivability of the variance function σ 2 .

Condition 3.2. For a density fX in Cr (IX ) and a function µ in Cs (IX ) and a variance σ 2 in Ck (IX ), with k, s, r ≥ 2, the bandwidth sequences (δn )n and (hn )n satisfy δn = O(n−1/(2k+1) ),

hn = O(n−1/{2(s∧r)+1} ),

as n tends to infinity. Proposition 3.6. Under Conditions 2.1, 2.2 and 3.1, for every function µ in Cs , density fX in Cr and variance function σ 2 in Ck , E{Y − m b nh (x)}2 = σ 2 (x) + O(h2(s∧r) ) + O((nh)−1 ),

the bias of the estimator Sbn,h,δ (x) of σ 2 (x) defined by (3.17) is 2 βn,h,δ (x) = b2m,n,h (x)fX (x) + σm,n,h (x)fX (x) +

δ 2k (σ 2 (x)fX (x))(2) (k!)2

+ o(δ 2k + h2(s∧r) + (nh)−1 ) and its variance is written (nδ)−1 {vσ2 + o(1)} with vσ2 (x) = κ2 V ar{(Y − 2 m(x))2 |X = x}. The process (nδ)1/2 (b σn,h,δ − σ 2 − βn,h,δ ) converges weakly to a Gaussian process with mean zero, variance vσ2 and covariances zero.

January 31, 2011

66

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Proof. Using Proposition 2.2 and Lemma 3.2, the mean squared error for m b nh at x is E[{Y − m b nh (x)}2 | X = x] and it is expanded as 2 2 2 σ (x) + bm,n,h (x) + σm,n,h (x) + E[{Y − m(x)}{m(x) − m b nh (x)} | X = x] b where the last term is zero. For the variance of Sn,h,δ (x), the fourth conditional moment E[{Y − m b nh (x)}4 (x) | X = x] is the conditional expectation of {(Y − m(x)) + (m − mnh )(x) + (mnh − m b nh )(x)}4 and it is expanded in a 4 sum of σ4 (x) = E{Y −m(x)) | X = x}, a bias term b4m,n,h (x) = O(h8(s∧r) ), E(mnh − m b nh )(x)}4 = O((nh)−1 ) by Proposition 3.1, and products of squared terms the main of which being σ 2 (x)km b nh − mk22 (x) of order −1 4(s∧r) O((nh) ) + O(h ), and the others being smaller. The variance of b Sn,h,δ (x) follows. Moreover, for every i 6= j ≤ n and for every functionR ψ in C2 and integrable with respect to FX , Eψ(Xj )Kδ2 (Xi − Xj ) = ψ(x)Kδ2 (x − x0 ) dFX (x) dFX (x0 ) equals κ2 Eψ(X) + o(δ 2 ) and the main term of the variance does not depend on the bandwidth δ.  The bandwidths hn and δn appear in the bias and the variance, therefore the mean squared error for the variance is minimum under Condition 3.2. Note that the function m which achieves the minimum of the empirical Pn mean squared error for the model Vn,h (x) = n−1 i=1 Kh (x − Xi ){Yi − m(x)}2 is the estimator m b n,h (3.1) and Vn,h (x) converges in probability σ(x). In a parametric regression model with a Gaussian error having a Pn constant variance, Vn (x) = n−1 i=1 {Yi − m(x)}2 is the sufficient statistic for the estimation of the parameters of m. In a Gaussian regression model with a functional variance σ 2 (x), each term of the sum defining the error is normalized by a different variance σ(Xi ) and the sufficient statistic for the estimation of parameters of the function m is the weighted mean square error n X Vw,n (θ) = n−1 σ −1 (Xi ){Yi − mθ (Xi )}2 . i=1

For a nonparametric regression function, an empirical local mean weighted squared error is defined as n X Vw,n,h (x) = n−1 w(Xi ){Yi − m(x)}2 Kh (x − Xi ) i=1

with w(x) = σ −1 (x). A weighted estimator of the nonparametric regression curve m is then defined as Pn w(Xi )Yi Kh (x − Xi ) m b w,n,h (x) = Pi=1 , (3.18) n i=1 w(Xi )Kh (x − Xi )

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a regression function

FunctionalEstimation

67

if the variance is known, it achieves the minimum of Vw,n,h (x). With an unknown variance, minimizing the weighted squared error leads to the es−1 timator built with its estimator w bn = σ bn,h , using (3.17) n ,δn Pn w bn (Xi )Yi Kh (x − Xi ) m b wbn ,n,h (x) = Pi=1 . (3.19) n bn (Xi )Kh (x − Xi ) i=1 w The uniform consistency of w bn,h implies supIn,h |m b wbn ,n,h − mw | tends to zero as n tends to infinity. Assuming that σ belongs to C2 (IX ), the convergence results for m b n,h in Propositions 3.1 or (3.2) adapt to the estimator (3.18), with µw = wµ instead of µ and w(x)fX,Y (x, y) instead of fX,Y (x, y). The approximation (3.2) is unchanged, hence the bias and the variance of the weighted estimator m b w,n,h are hs msK bm,w,n,h(x) = {(mwfX )(s) (x) − m(x)(wfX )(s) (x)} + o(hs ), s!w(x)fX (x) vm,w,n,h (x) = vm,n,h (x). In the approximations of Propositions 3.2 and 3.3, the order of convergence of supx∈IX,h kb rn,h k2 is not modified and the weak convergence of Theorem 3.1 is fulfilled for the process (nh)1/2 {m b w,n,h −m}I{IX,h } , with the modified bias and variance. With an estimated weight, the meanR of the numerator µ bwbn ,n,h (x) is Ew bn (X)m(X)Kh (x − X) and it equals E w bn (y)m(y)Kh (x − y)fX (y) dy 2 since σ bn,h (X ) is equivalent to the estimator of the variance (at Xi ) i ,δ n n calculated from the observations without Xi . With an empirical weight 2 w bn (x) = ψ(b σn,h (x)), the mean of the numerator of the estiman ,δn 2 tor (3.19) is then ENn (x) = Ew(X)m(X)Kh (x − X) + E{(b σn,h − n ,δn 2 0 2 σ )(X)ψ (σ (X))m(X)Kh (x − X)}{1 + o(1)} and the bias of the numerator of (3.19) is modified by adding m(x)fX (x)βn,h,δ (x)ψ 0 (σ 2 (x)) to the bias of the expression with a fixed weight w. In the same way, the expectation of the denominator is EDn (x) = w(X)Kh (x − X) + 2 E{(b σn,h − σ 2 )(X)ψ 0 (σ 2 )(X)Kh (x − X)}{1 + o(1)} and it is approxin ,δn mated by fX (x){w(x) + βn,h,δ (x)ψ 0 (σ 2 (x)). Using the approximation (3.2) of Proposition 3.1, the first order approximation of the bias of (3.19) is identical to bm,w,n,h(x). The variances of each term are κ2 E{w bn2 (x)E(Y 2 | X = x)fX (x)} + o((nh)−1 ), V arNn (x) = nh κ2 V arDn (x) = E{w bn2 (x)fX (x)} + o((nh)−1 ), nh κ2 V arm b wbn ,n,h (x) = V ar(w bn (x)Y | X = x) + o((nh)−1 ). nhw2 (x)fX (x)

January 31, 2011

17:17

68

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

The variance of the estimator with an empirical weight is therefore modified by a random factor in the variance of Y and a normalization by w(x). The convergence rates are not modified.

3.7

Estimation of the mode of a regression function

The mode of a real regression function m on IX is Mm = sup m(x).

(3.20)

IX

The mode Mm of a regular regression function is estimated by the mode cm,n,h = Mm of a regular estimator of the function, M b n,h . Under Conditions 2.1-3.1, the regression function is locally concave in a neighborhood NM of the mode and its estimator has the same property for n sufficiently large, by the uniform consistency of m b n,h , hence m(1) (Mm ) = 0, m(2) (Mm ) < 0, (1) c c m b n,h (M m,n,h ) = 0 and Mm,n,h converges to Mm in probability. A Taylor (1) expansion of m at the estimated mode implies (1)

cm,n,h − Mm ) = {m(2) (Mm )}−1 {m(1) (M cm,n,h ) − m cm,n,h )} + o(1). (M b n,h (M (1)

The weak convergences of the process (nh3 )1/2 (m b n,h − m(1) ) (Proposition cm,n,h − Mm ) as (nh3 )−1/2 and 3.5) determines the convergence rate of (M cm,n,h . it implies the asymptotic behaviour of the estimator M cm,n,h − Proposition 3.7. Under Conditions 2.1, 2.2 and 3.1, (nh3 )1/2 (M Mm ) converges weakly to a centered Gaussian variable with finite variance (1) m(2)−2 (Mm )V arm b n,h (Mm ).

cm,n,h ) is If the regression function belongs to C3 (IX ), the bias of m(1) (M (1) deduced from the bias of the process m b n,h defined by (3.15), it equals 2

cm,n,h ) = − h m2K f −1 (x){(mfX )(3) − mf (3) }(Mm ) + o(h2 ) Em(1) (M X X 2

and does not depend on the degree of derivability of the regression function m. All results are extended for the search of the local maxima and minima of the function m which are local maxima of −m. The maximization of the function on the interval IX is then replaced by sequential maximizations or minimizations.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a regression function

3.8

FunctionalEstimation

69

Estimation of a regression function under censoring

Consider the nonparametric regression (1.6) where the variable Y is rightcensored by a variable C independent of (X, Y ) and the observed variables are (X, Y ∗ , δ) where Y ∗ = Y ∧ C and δ = 1{Y ≤C} . Let FY |X denote the distribution function of Y R conditionally on X. The regression function m(x) = E(Y | X = x) = yFY |X (dy; x) is estimated using an estimator of the conditional density of Y given X under right-censoring. Extending the results of Section 2.8 to the nonparametric regression, the conditional distribution function FY |X defines a cumulative conditional hazard function Z ΛY |X (y; x) = 1{s≤y} {1 − FY |X (s; x)}−1 FY |X (ds; x), conversely the function ΛY |X uniquely defines the conditional distribution function as Y 1 − FY |X (y; x) = exp{−ΛcY |X (y; x)} {1 − ∆ΛY |X (z − ; x)}, z>y

Q where ΛcY |X is the continuous part of ΛY |X and s {1 − ∆Λ(s)} its rightcontinuous discrete part. Let X X Nn (y; x) = Kh (x−Xi )δi 1{Yi ≤y} , Yn (y; x) = Kh (x−Xi )1{Yi∗ ≥y} 1≤i≤n

1≤i≤n

be the counting processes related to the observations of the censored variable Y ∗ , with regressors in a neighborhood Vh (x) of x, and let Jn (y; x) be R y the indicator of Y n(y; x) > 0. The process Mn (y; x) = Nn (y; x) − Y (s; x) dΛY |X (s; x) is a centered martingale with respect to the fil−∞ n tration generated by the observed processes up to y − , conditionally on regressors in Vh (x). The functions ΛY |X and FY |X are estimated by Z b Y |X,n,h (y; x) = 1{s≤y} Jn (s; x)Nn (ds; x) , Λ Yn (s; x) Y b b Y |X,n,h (Yi ; x)}, FY |X,n,h (y; x) = 1 − {1 − ∆Λ Yi ≤y

b Y |X,n,h is unbiased and FbY |X,n,h is the Kaplan-Meier estimathe estimator Λ tor for distribution function of Y conditional on {X = x}. The regression function m is then estimated by Z m b n,h (x) = y FbY |X,n,h (dy; x) =

n X

Jn (Yi ; x) Yi {1 − FbY |X,n,h (Yi− ; x)} . Yn (Yi ; x) i=1

January 31, 2011

70

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

b Y |X,n,h −ΛY |X |, supI The estimators satisfy supIX ×I |Λ |FbY |X,n,h −FY |X | X,Y and supIX |m b n,h − m| converge in probability to zero as n tends to infinity, for every compact subinverval I of IY . For every y ≤ max Yi∗ , the conditional Kaplan-Meier estimator, given x in IX,n,h , still satisfies Z y FY |X − FbY |X,n,h 1 − FbY |X,n,h (s− ; x) b (y; x) = d(ΛY |X,n,h −ΛY |X )(s; x) . 1 − FY |X 1 − FY |X (s; x) −∞ (3.21) The mean of this integral with respect to a centered martingale is zero so the b Y |X,n are unbiased estimators. conditional Kaplan-Meier estimator and Λ The bias of the estimator of the regression function for censored variables Y is then a O(h2 ). 3.9

Proportional odds model

Consider a regression model with a discrete response variable Y corresponding to a categorization of an unobserved continuous real variable Z in a partition (Ik )k≤K of its range, with the probabilities Pr(Z ∈ Ik ) = Pr(Y = k). With a regression variable X and intervals Ik = (ak−1 , ak ), the cumulated conditional probabilities are πk (X) = Pr(Y ≤ k | X) = Pr(Z ≤ ak | X), and EπK (X) = 1. The proportional odds model is defined through the logistic model for the probabilities πk (X) = p(ak − m(X)), with the logistic probability p(y) = exp(y)/{1 − exp(y)} and a regression function m. This model is equivalent to πk (X){1 − πk (X)}−1 = exp{ak − m(X)} for every function πk such that 0 < πk (x) < 1 for every x in IX and for 1 ≤ k < K. This implies that the odds-ratio for the observations (Xi , Yi ) and (Xj , Yj ) with Yi and Yj in the same class does not depend on the class πk (Xi ){1 − πk (Xj )} = exp{m(Xj ) − m(Xi )}, {1 − πk (Xi )}πk (Xj )

for every k = 1, . . . , K, this is the proportional odds model. For k = 1, . . . K, let pk (x) = (πk − πk−1 )(x) = Pr(Y = k | X = x). Assuming that p1 (x) > 0 for every x in IX , the conditional distribution of the discrete variable is also determined by the conditional probabilities αk (x) = P (Y = k|X = x)P −1 (Y = 1|X = x). Equivalently P (Y = k|X = x) =

αk (x) , k = 1, . . . , K, PK 1 + j=1 αj (x)

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a regression function

FunctionalEstimation

71

PK with the constraint This k=1 P (Y = k|X = x) = 1 for every x. reparametrization of the conditional probabilities αk is not restrictive, though it is called the logistic model. Estimating first the support of the regression variable reduces the number of unknown parameters to 2(K − 1), the thresholds of the classes and their probabilities, for k ≤ K − 1, in addition to the nonparametric regression function m. The probability functions πk (x) are estimated by the proportions π bn,k (x) of observations of the variable Y in class k, conditionally on the regressor value x. Let π bn,k (Xi ) Uik = log , i = 1, . . . , n, 1−π bn,k (Xi )

calculated from the observations (Xi , Yi )i=1,...,n such that Yi = k. The variations of the regression function m between two values x and y are estimated by K  Pn X −1 i=1 Uik Kh (Xi − x) P m b n,h (x) − m b n,h (y) = K n i=1 Kh (Xi − x) k=1 Pn  i=1 Uik Kh (Xi − y) P − . n i=1 Kh (Xi − y)

This estimator yields an estimator for the derivative of the regression func(1) tion, m b n,h (x) = lim|x−y|→0 (x − y)−1 {m b n,h (x) − m b n,h (y)} wich is written as the mean over the classes of the derivative estimator (3.15) with response variables Uik . Integrating the mean derivative provides a nonparametric estimator of the regression function m. The bounds of the classes cannot be identified without observations of the underlying continuous variable Z, thus the odds ratio allows to remove the unidentifiable parameters from the model for the observed variables. With a regression multidimensional variable X, the single-index model or a transformation model (Chapter 7) reduce the dimension of the variable and fasten the convergence of the estimators. 3.10

Estimation for the regression function of processes

Consider a continuously observed stationary and ergodic process (Zt )t∈[0,T ] = (Xt , Yt )t∈[0,T ] with values in IXY , and the regression model Yt = m(Xt ) + σ(Xt )εt where (εt )t∈[0,T ] is a conditional Brownian motion

January 31, 2011

72

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

such that E(εt | Xt ) = 0 and E(εt εs | Xt ∧Xs ) = E{(εt ∧εs )2 | Xt ∧Xs ) = 1. The ergodicity property is expressed by (2.13) or (2.16) for the bivariate process Z. The regression function m is estimated on an interval IX,Y,T,h by the kernel estimator RT Ys Kh (x − Xs ) ds m b T,h (x) = 0R T . (3.22) 0 Kh (x − Xs ) ds Its numerator is denoted

1 µ bT,h (x) = T

Z

T

0

Ys Kh (x − Xs ) ds

and its denominator is fbX,T,h (x). The mean of µ bT,h (x) and its limit are respectively Z µT,h (x) = yKh (x − u) dFXY (u, y), I Z XY µ(x) = yfXY (x, y) dy = fX (x)m(x). IXY

Under Conditions 2.1-2.2 and 3.1, the bias of µ bT,h (x) is Z hs bµ,T,h (x) = yKh (x − u) dFXY (u, y) − µ(x) = T msK µ(s) (x) + o(hsT ), s! IXY,T

its variance is expressed through the integral of the covariance between Ys Kh (Xs − x) and Yt Kh (Xt − x). For Xs = Xt , the integral on the diagonal 2 DX of IX,T is a (T hT )−1 )κ2 w2 (x)+o((T hT )−1 ) and the integral outside the diagonal denoted Io (T ) is expanded using the ergodicity property (2.13). Let αh (u, v) = |u − v|/2hT Z Z ds dt y1 y2 Kh (u − x)Kh (v − x)dFZs ,Zt (u, y1 , v, y2 ) Io (T ) = 2 T T \DX [0,T ]2 IXY Z Z Z 1/2 = (T hT )−1 { K(z − αh (u, v))K(z + αh (u, v)) dz IX

IX\{u}

−1/2

µ(u)µ(v)dπu (v) dFX (u)}{1 + o(1)} .

For every fixed u 6= v, αhT (u, v) tends to infinity as hT tends to zero, then R 1/2 the integral −1/2 K(z − αh (u, v))K(z + αh (u, v)) dz tends to zero with hT . If |u − v| = O(hT ), this integral does not tend to zero but the transition probability πu (v) tends to zero as hT tends to zero, therefore the integral Io (T ) is a o((T hT )−1 ) as T tends to infinity. The Lp -norm of the estimator satisfies kb µT,h (x) − µT,h (x)kp = O((T hT )−1/p ) under the ergodicity condition for k-uplets of the process Z (2.16) and the approximation (3.2) is also

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Kernel estimator of a regression function

FunctionalEstimation

73

satisfied for the estimator m b T,h . It follows that its bias, for s ≥ 2, and its variance are approximated by bm,T,h (x; s) = hsT bm (x; s) + o(hsT ),

−1 bm (x; s) = fX (x){bµ (x) − m(x)bf (x)} msK −1 (s) = f (x){µ(s) (x) − m(x)fX (x)}, s! X 2 vm,T,h (x) = (T hT )−1 {σm (x) + o(1)}, −1 2 σm (x) = κ2 fX (x)V ar(Y | X = x)

and the covariance between m b T,h (x) and m b T,h (y) tends to zero. The mean squared error of the estimator at x for a marginal density in Cs is then −1 M ISET,hT (x) = (T hT )−1 )κ2 fX (x)V ar(Y | X = x)

−1 2 ) + o(h2s + h2s T bm (x; s) + o((T hT ) T )

and the optimal local and global bandwidths minimizing the mean squared (integrated) errors are O(T 1/(2s+1) ) n1 o1/(2s+1) 2 σm (x) hAMSE,T (x) = 2 T 2sbm (x; s)m(x)

and, for the asymptotic mean integrated squared error criterion R 2 n1 o1/(2s+1) σm (x) dx R hAMISE,T = . T 2s b2m (x; s)m(x) dx

With the optimal bandwidth rate, the asymptotic mean (integrated) squared errors are O(T 2s/(2s+1) ). The same expansions as for the variance µ bT,h (x) and fbX,T,h (x) in Section 2.10 prove that the finite dimension distributions of the process (T hT )1/2 (fbT,h − f − bT,h ) converge to those of a centered Gaussian process with mean zero, covariances zero and variance κ2 f (x) at x. Lemma 3.3 generalizes and the increments E{fbT,h (x) − fbT,h (y)}2 are approximated as E|∆(m b n,h − mn,h )(x, y)|2 = O(|x− y|2 (T h3T )−1 ) for every x and y in IX,h such that |x− y| ≤ 2hT . Then the process (T hT )1/2 {m b T,h −m}I{IX,T } converges weakly to σm W1 +γ 1/2 bm where W1 is a centered Gaussian process on IX with variance 1 and covariances zero. 3.11

Exercises

(1) Detail the proof for the approximations of the biases and variances of Proposition 3.1.

January 31, 2011

74

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

(2) Suppose Y is a binary variable with P (Y |X = x) = p(x) and express the bias and the variance of the estimator of the nonparametric probability function p. (3) Consider a discrete variable with values in an infinite countable set. Define an estimator of the function m under suitable conditions and give the expression of its bias and variance. (4) Define nonparametric estimators for the bias of the function m and its variance. (5) Define the optimal bandwidths for the estimation of the function µ and its first order derivative. (6) Detail the expression of km b n,h (x) − m(x)kp using the orders of the norms established in Section 3.2. (7) Detail the expressions of the bias and the second order approximation 2 of the variance of σ bn,h,δ (x) in Proposition 3.6. (8) Let FY |X (y; x) = Pr(Y ≤ y | X ≤ x) be the distribution function of Y conditionally on X and FbY |X,n,h (y; x) = n−1

n X i=1

1{Yi ≤y} Hh (Xi − x)

be a smooth estimator of the conditional distribution function (see Exercise 2.11-(6)). Find the expression of the bias and the variance of FbY |X,n,h (x).

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Chapter 4

Limits for the variable bandwidths estimators

4.1

Introduction

The pointwise mean squared error for a density or regression function reaches its minimum at a bandwidth function varying in the domain of the variable X. The question of the behaviour of the estimator of density and regression functions with a varying bandwidth is then settled. All results of Chapters 2 and 3 are modified by this function. Consider a density or a regression function of class Cs (IX ). Let (hn )n be a sequence of functional bandwidths in C1 (IX ), converging uniformly to zero and uniformly bounded away from zero on IX . In order to have an optimal bandwidth for the estimation of functions of class C2 , the functional sequence is assumed to satisfy an uniform convergence condition for the uniform norm khn k. Condition 4.1. There exists a strictly positive function h in C1 (IX ), such that khk is finite and knh2s+1 − hk tends to zero as n tends to infinity. n Under this condition, the bandwidth is uniformly approximated as hn (x) = n−1/(2s+1) h1/(2s+1) (x) + o(n−1/(2s+1) ). The increasing intervals IX,hn are now defined with respect the uniform norm of the function hn by IX,hn = {s ∈ IX ; [s − khn k, s + khn k] ∈ IX }. The main results of the previous chapters are extended to kernel estimators with functional bandwidth sequences satisfying this convergence rate. That is the case of the kernel estimators built with estimated optimal local bandwidths calculated from independent observations. The second point of this chapter is the definition of an adaptative estimator of the bandwidth, when the degree of derivability of the density varies in its domain of definition, and the behaviour of the estimator of 75

January 31, 2011

17:17

76

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

the density with an adaptative estimator. In Chapter 2, the optimal density was obtained under the assumption that the degree of smoothness of the density is known and constant on the interval of the observations. The last assumption flattens the estimated curve by the use of a too large bandwidth in areas with smaller derivability order, the above variable bandwidth hn (x) does not solves that problem. The cross-validation method allows to define a global bandwidth without knowledge of the class of the density. Other adaptative methods are based the maximal variations of the estimator as the bandwidth varies in a grid Dn corresponding to a discretization of the possible domain of the bandwidth according to the order of regularity of the density. It can be performed globally or pointwisely.

4.2

Estimation of densities

Let us consider the random process Un,hn (x) = (nhn (x))1/2 {fbn,hn (x) (x) − f (x)} for x in IX,hn . Under Conditions 2.1 and 4.1, supI |fbn,hn (x) − f (x)| converges a.s. to zero for every compact subinterval I of IX,hn and kfbn,hn (x) − f (x)kp tends to zero, as n tends to infinity. The bias of fbn,hn (x) (x) is bn,hn (x) = 21 h2n (x)m2K f (2) (x) + o(khn k2 ), its variance is b V ar{fbn,hn (x) (x)} = (nhn (x))−1 κ2 f (x) + o((n−1 kh−1 n k) and kfn,hn (x) (x) − −1 −1 1/p fn,hn (x) (x)kp = 0((n khn k) ). Under Conditions 2.1-4.1, for a density of class Cs (IX ) and for every x in IX,h , the moments of order p ≥ 2 are unchanged and the bias of fbn,hn (x) (x) is modified as bn,hn (x; s) =

hsn (x) msK f (s) (x) + o(khn ks ). s!

The MISE and the optimal local bandwidth are similar to those of Chapter 2 using these expressions. For every u in [−1, 1], let αn and v in [−1, 1], |u| in [0, {x + hn (x)} ∧ {y + hn (y)}] be defined by 1 −1 {(u − x)h−1 (4.1) n (x) − (u − y)hn (y)}, 2 1 −1 v = vn (x, y, u) = {(u − x)h−1 n (x) + (u − y)hn (y)} 2 1 = [{hn (x) + hn (y)}u − xhn (y) − yhn (x)], 2hn (x)hn (y) αn (x, y, u) =

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Limits for the varying bandwidths estimators

77

u = un (x, y, v) = {hn (x) + hn (y)}−1 {xhn (y) + yhn (x) + 2vhn (x)hn (y)}, zn (x, y) = {hn (x) + hn (y)}−1 {xhn (y) + yhn (x)},

δn (x, y) = 2hn (x)hn (y){hn (x) + hn (y)}−1 = o(1),

hence αn (x, y, u) is also denoted αn (x, y, v). Lemma 4.1. The covariance of fbn,h (x) and fbn,h (y)} equals Z 2 {f (zn (x, y)) K(v − αn (v))K(v + αn (v)) dv n{hn (x) + hn (y)} Z + δn (x, y)f (1) (zn (x, y)) vK(v − αn (v))K(v + αn (v)) dv + o(khn k)}. Proof.

The integral

EKhn (x) (x − X)Khn (y) (y − X) =

Z

Khn (x) (x − u)Khn (y) (y − u)fX (u) du

is expanded changing the variable u in v and it equals Z 2 K(v − αn (v))K(v + αn (v))f (un (x, y, v)) dv hn (x) + hn (y) Z 2 = {f (zn (x, y)) K(v − αn (v))K(v + αn (v)) dv hn (x) + hn (y) Z (1) + δn (x, y)f (zn (x, y)) vK(v − αn (v))K(v + αn (v)) dv + o(khn k)}.



Lemma 4.2. For functions of class Cs (IX ), s ≥ 1, and under Conditions 3.1 and 4.1, for every x and y in IX,hn the mean variation of fbn,hn between x and y has the order O(|x − y|) and its mean squared variation for −1 2 −1 3 2 b b |xh−1 kh−1 n (x)−yhn (y)| ≤ 1 are E|fn,hn (x)− fn,h (y)| = O(n n k (x−y) ). −1 −1 b b Otherwise, it is a O(n khn k) and the variables fn,h (x) and fn,h (y) are independent. Proof. By the Mean Value Theorem, for every x and y in IX,h there exists s between x and y such that |fn,hn (x) − fn,hn (y)| = |x − y|f (1) (s) and |fn,hn (x) − fn,hn (y)| ≤ |x − y|kf (1) k. Let z = limn zn (x, y) defined in (4.1). The expectation of |fbn,h (x) − R 2 −1 b fn,h (y)| develops as n {Khn (x) (x − u) − Khn (y) (y − u)}2 f (u) du + (1 − −1 2 n ){fn,hn (x) − fn,hn (y)} .

January 31, 2011

78

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Using the notations (4.1), the first term of this sum is expanded as Z 1 S1n = {hn (x)K(v − αn (v)) nhn (x)hn (y){hn (x) + hn (y)} −hn (y)K(v + αn (v))}2 f (zn (v)) dv.

The derivability of the bandwidth functions implies Z 1 {hn (x)K(v − αn ) − hn (y)K(v + αn )}2 f (zn ) dv hn (x)hn (y) Z hn (x) ≤ 2[ {K(v − αn ) − K(v + αn )}2 f (zn ) dv hn (y) Z {hn (x) − hn (y)}2 + K 2 (v − αn )f (zn ) dv], hn (x)hn (y) Z hn (x) 2 S1n ≤ [ f (z) {K(v − αn ) − K(v + αn )}2 dv n{hn (x) + hn (y)} hn (y) Z (1)2 2 hn (η(x − y)) + (x − y) K 2 (v − αn )f (zn ) dv], hn (x)hn (y) (1)2

where η lies in (−1, 1), by the Mean Value Theorem, hn (η) and hn (x)hn (y) have the same order, and Z Z 2 {K(v − αn ) − K(v + αn )}2 dv = 4α2n K (1)2 (v) dv = O(|x − y|2 kh−1 n k ). 2 −1 2 It follows that S1n = O(n−1 kh−1 Since h−1 n k |x − y| khn k ). n (x)|x| −1 −1 −1 1/2 b and hn (y)|y| are bounded by 1, the order of E(nkhn k ) |fn,h (x) − fbn,h (y)|2 = O((x−y)2 ) if |xhn (y)−yhn (x)| ≤ hn (y)hn (x), otherwise fbn,h (x) and fbn,h (y) are independent and it is a sum of variances. 

Theorem 4.1. Under the conditions, for a density f of class Cs (IX ) and a varying bandwidth sequence such that nkhn k2s+1 converges to khk, the process Un,hn (x) = (nhn (x))1/2 {fbn,hn(x) − f (x)}I{x ∈ IX,khn k }

converges weakly to the process defined on IX as Wf (x) + h1/2 (x)bf (x), where Wf is a continuous centered Gaussian process with covariance σf2 (x)δ{x,x0 } between Wf (x) and Wf (x0 ). Proof. The weak convergence of the variable Un,h (x) is a consequence to the L2 -convergence of (nhn (x))1/2 {fbn,hn (x) − f (x) − (nh2s+1 (x))1/2 bf (x)} n to κ2 f (x). In the same way, the finite dimensional distributions of the process Un,h converge weakly to those of a centered Gaussian process. The

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Limits for the varying bandwidths estimators

FunctionalEstimation

79

quadratic variations of the bias {fn,hn (x) (x) − f (x) − fn,hn (y) (y) + f (y)}2 are bounded by Z | K(z){f (x + hn (x)z) − f (x) − f (y + hn (y)z) − f (y)} dz|2 =

msK h(x) 2s (s) h(y) (s) khn k2s [{ } f (x) − { f (y)}2s ]2 s! khk khk

and it is a O(khn k2s |x − y|2 ). This bound and Lemma 4.2 imply that the mean of the squared variations of the process Un,h on small intervals are O(|x − y|2 ), therefore the process Un,h is tight, so it converges weakly to a centered Gaussian process. The covariance of the limiting process at x and y is the limit of the covariance between Un,h (x) and Un,h (y) and it equals 1/2 1/2 limn nhn (x)hn (y)Cov{fbn,h (x),Rfbn,h (y)}. The covariance of fbn,h (x) and fbn,h (y) is approximated by n−1 Khn (x) (x − u)Khn (y) (y − u)f (u)du, for x 6= y it develops as Z 1{0≤αn <1} f (zn (x, y)) K(v − αn )K(v + αn )dv n{hn (x) + hn (y)} + o(n−1 (hn (x) + hn (y))−1 ).

−1 As n tends to infinity, hn (x)h−1 n (y) and hn (y)hn (x) are bounded and 1{0≤αn <1} tends to zero for every x 6= y, hence the covariance of Un,h (x) and Un,h (y)} converges to zero as n tends to infinity. 

The optimal bandwidth for estimating a density has an order between n and n−1/(2α+1) with α > 1/4, under the conditions nhn tends to infinity and f belongs locally to Hα with α > 1/4, or Cα with α ≥ 2. All estimators of the bias of a density depend on its regularity through the constant of the bias and the exponent of h and it cannot be directly estimated without knowledge of α. The bandwidth minimizing the mean squared error of the estimator fbn,h (x) is bounded by −1

M SEn,h (x, α) = V arfbn,h (x){1 + (2α)−1 }

(4.2)

with an order of smoothness α > 1/4, so only the lower bound of the degree is necessary to obtain a bound of the MSE. As the variance of fbn,h(x) (x) does not depend on the class of f , it can be estimated using a bandwidth function h2 such that nkh2 k tends to zero and nkh2 k2 tends to infinity, by \ Vb arn,h2 fbn,h (x) = (nh(x))−1 κ2 fbn,h2 (x) (x). Let M SE n,h,an (x) be the estimator of M SEn,h,an (x) obtained by plugging the estimator of V arfbn,h (x). It can be compared with the bootstrap estimator of the mean squared er∗ ror M SEn,h (x) = V ar∗ fbn,h (x) + B ∗2 (fbn,h )(x) calculated from a bootstrap

January 31, 2011

80

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

sample of independent variables having the distribution Fbn . This estimator and the bootstrap estimator V ar∗ fbn,h (x) yield an estimator of α, by equation (4.2). An optimal local bandwidth can then be estimated from the estimator of α. The choice of the bandwidth function h2 relies on the same procedure and the optimal estimator b hn (x) requires iterations of this procedure, starting to an empirical bandwidth calculated from a discretization of its range. Adaptative estimators of the bandwidth were previously defined using empirical thresholds for the variations of the estimator of the density according to the bandwidth, however constants in the thresholds were chosen by numerical recursive procedures. Another variable bandwidth kernel estimator is defined with a bandwidth function of the variables Xi rather than x n

1X fbX,n,hn (x) = Khn (Xi ) (x − Xi ). n i=1 R Its mean is EfX,n,hn (x) = EKhn (X) (x − X) = Khn (y) (x − y)fX (y) dy and its limit is fx (x), approximating y by x in the integral. Its bias and variance are not expanded as above, the bandwidth at y is now developed (1) as hn (y) = hn (x){1 − zhn (x)} + o(khn k2 ) where x − y = hn (y)z, hence 2 fX (y) = fX (x − hn (x)z + hn (x)h(1) n (x)z + o(khn k ) 1 (2) (1) = fX (x) − hn (x)zfX (x) + hn (x)z 2 {hn (x)fX (x) 2 (1) 2 + 2h(1) n (x)fX (x)} + o(khn k )

and the bias of the estimator is m2K (2) (1) 2 bfbX,n,h (x) = hn (x){hn (x)fX (x) + 2h(1) n (x)fX (x)} + o(khn k ). 2 Its variance is Z V arfbX,n,hn (x) = n−1 { Kh2n (y) (x − y)fX (y) dy − E 2 fbX,n,hn (x)}, Z Z 2 Khn (y) (x − y) dy = K 2 (z)fX (x − hn (x)z − hn (x)h(1) n (x)z) dz = m2K {fX (x) − hn (x)f (1) (x)

Z

+ o(khn k2 )

zK 2 (z) dz + o(khn k)

the first order approximation of the variances are identical and their second order approximation have the opposite sign.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

81

Limits for the varying bandwidths estimators

4.3

Estimation of regression functions

Let us consider the variable bandwidth kernel estimator m b n,hn (x) (x) of the regression function m and the random process related to the estimated regression function Um,n,hn (x) = (nhn (x))1/2 {m b n,hn (x)− m(x)}I{x∈IX,khn k } . Conditions 2.1 and 4.1 for kernel estimators of densities with variable bandwidth are supposed to be satisfied in addition to Conditions 3.1 for kernel estimators of regression functions. Then supx∈IX,khn k |m b n,hn (x) (x) − m(x)| converges a.s. to zero with the uniform approximations µn,hn (x) (x) + O((nkhn k)−1 ), fX,n,hn (x) (x)
1/2  (nhn (x))1/2 {m b n,hn(x) − mn,hn (x) }(x) = nhn (x) (b µn,hn (x) −1 − µn,h (x) )(x) − m(x)(fbX,n,h (x) − fX,n,h (x) )(x) f (x) + rn,h mn,hn (x) (x) =

n

n

X

n

n (x)

,

where rn,hn = oL2 (1), uniformly. For every x in IX,khn k and for every integer p > 1, km b n,hn(x) (x) − m(x)kp converges to zero, the bias of the estimator m b n,hn (x) (x) is uniformly approximated by bm,n,hn(x) (x) = mn,hn (x) (x) − m(x) = hn (x)2 bm (x) + o(khn k2 ), −1 bm (x) = fX (x){bµ (x) − m(x)bf (x)} m2K −1 (2) = f (x){µ(2) (x) − m(x)fX (x)}, 2 X

and its variance is deduced from (3.7) 2 vm,n,hn (x) (x) = (nhn (x))−1 {σm (x) + o(1)},

−2 2 σm (x) = κ2 fX (x){w2 (x) − m2 (x)f (x)}.

For a regression function and a density fX in class Cs (IX ), s ≥ 2, and under Conditons 2.2, the bias of m b n,hn (x) (x) is uniformly approximated by bm,n,hn(x) (x; s) =

hsn (x) (s) −1 msK fX (x){µ(s) (x) − m(x)fX (x)} + o(khn ks ) s!

and its moments are not modified by the degree of derivability. For every x in IX,khn k −1 (nhn (x))1/2 (m b n,hn − m)(x) = (nhn (x))1/2 fX (x){(b µn,hn (x) − µn,hn (x) ) b − m(fX,n,h (x) − fX,n,h (x) )}(x) n

n

+ (nhn (x)2s+1 )1/2 bm (x) + rbn,hn (x) (x),

January 31, 2011

82

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

and supx∈IX,khn k kb rn,hn (x) k2 = O((nkhn k)−1/2 ). The asymptotic mean squared error of m b n,h (x) is 2 (nhn (x))−1 σm (x) + hn (x)4 b2m (x) = (nhn (x))−1 κ2 {w2 (x) − m2 (x)f (x)} 4 h (x)m22K −2 (2) −2 + n fX (x){µ(2) (x) − m(x)fX (x)}2 fX (x) 4 and its minimum is reached at the optimal local bandwidth  κ2 n−1 {w2 (x) − m2 (x)f (x)} 1/5 hn,AMSE (x) = m22K {µ(2) (x) − m(x)f (2) (x)}2 X

where AM SE(x) = O(n−4/5 ). For every s ≥ 2, the asymptotic quadratic risk of the estimator for a regression curve of class Cs is 2 2 AM SE(x) = (nhn (x))−1 σm (x) + h2s n (x)bm,s (x) −2 = (nhn (x))−1 κ2 fX (x){w2 (x) − m2 (x)f (x)}

h2s (s) n (x) 2 m f −2 (x){µ(s) (x) − m(x)fX (x)}2 , (s!)2 sK X its minimum is reached at the optimal bandwidth (s!)2 κ2 n−1 {w2 (x) − m2 (x)f (x)} 1/(2s+1) hn,AMSE (x) = { } 2sm2sK {µ(s) (x) − m(x)f (s) (x)}2 +

X

where AM SE(x) = O(n−2s/(2s+1) ). The covariance of m b n,hn (x) and m b n,hn (y) is calculated as for Theorems 3.1 and 4.1 and it is a o(1) for every x 6= y. b n,hn (x)} equals Lemma 4.3. The covariance of m b n,hn (x) and m Z 2 [σ 2 (zn (x, y))κ−1 K(v − αn (v))K(v + αn (v)) dv 2 n{hn (x) + hn (y)} m (1)

(1)

−2 + δn (x, y)fX (zn (x, y)){w2 − m2 fX }(zn (x, y)) Z × vK(v − αn (v))K(v + αn (v)) dv + o(khn k)].

Proof. The integralREY 2 Khn (x) (x − X)Khn(y) (y − X) = EY 2 Khn (x) (x − X)Khn (y) (y − X) = Khn (x) (x − u)Khn (y) (y − u)w2 (u) du is expanded changing the variable u in v and it equals Z 2 K(v − αn (v))K(v + αn (v))w2 (un (x, y, v)) dv hn (x) + hn (y) Z 2 = {w2 (zn (x, y)) K(v − αn (v))K(v + αn (v)) dv hn (x) + hn (y) Z (1) + δn (x, y)w2 (zn (x, y)) vK(v − αn (v))K(v + αn (v)) dv + o(khn k)}

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Limits for the varying bandwidths estimators

FunctionalEstimation

83

then the L2 -approximation of (nhn (x))1/2 {m b n,hn (x) − mn,hn (x) }(x) and Lemma 4.3 end the proof. 

Lemma 3.3 is extended to µ bn,hn and m b n,hn with functional bandwidths like 4.2 and the weak convergence on IX,khn k of the process with varying bandwidth Un,hn (x) = (nhn (x))1/2 {fbn,hn(x) (x) − f (x)} is proved as for the density estimator. Lemma 4.4. For a regression function m and density fX of class Cs (IX ), s ≥ 2, and under Conditions 3.1 and 4.1, for every x and y in IX,hn the mean of the variation of m b n,hn between x and y has the order O(|x−y|) and 2 3 2 −1 −1 E|m b n,hn (x)−m b n,h (y)| = O(n−1 kh−1 n k (x−y) ) if |xhn (x)−yhn (y)| ≤ 1. −1 −1 Otherwise, it is a O(n khn k).

Theorem 4.2. Under the conditions, for a density f of class Cs (IX ) and a varying bandwidth sequence such that nkhn k2s+1 converges to khk, the process Un,hn converges weakly to the process defined on IX as Wm + h1/2 bm , where Wm is a continuous centered Gaussian process with covariance 2 σm (x)δ{x=x0 } at x and x0 . The estimators of the derivatives of the regression function are modified by the derivatives of the bandwidth and the kernel in each term of the estimators, as detailed in Appendix B, and the first derivative is (1) (1) (1) m b = fb−1 {b µ −m b n,h fb }, like in (3.15), with notations of the apn,h

n,h

n,h

n,h

pendix for d{Khn (x) (x)}/dx. The results of Proposition 3.5 are extended (k)

to the estimator m b n,hn with a varying bandwidth sequence, its bias is a s 2k+1 O(khn k ), and its variance a O((nkh−1 ), hence the optimal bandn k) −1/(2k+2s+1) width is a O(n ) and the optimal mean squared error is a O(n−2s/(2k+2s+1) ). In the regression model with a conditional variance function σ 2 (x), the kernel estimator (3.17) with continuous functional bandwidths hn and δn can be written Pn {Yi − m b n,h (x) (Xi )}2 Kδn (x) (x − Xi ) 2 Pn n σ bn,hn (x),δn (x) (x) = i=1 , i=1 Kδn (x) (x − Xi )

then a new estimator for the regression function is defined using this estima−1 tor as a weighting process w bn = σ bn,h in the estimator of the regression n ,δn function Pn w bn (Xi )Yi Khn (x) (x − Xi ) m b wbn ,n,hn (x) (x) = Pi=1 . n bn (Xi )Khn (x) (x − Xi ) i=1 w

January 31, 2011

84

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

2 The bias and variance of the estimator σ bn,h (x) and the fixed bandn (x),δn (x) 2 width estimator for σ (x) are still similar. The bias of m b wbn ,n,hn (x) (x) and m b w,n,hn (x) (x) have the same approximations, the variance of m b w,n,hn (x) (x) is identical to the variance of m b n,hn (x) (x) whereas the variance of m b wbn ,n,hn (x) (x) is modified like with the fixed bandwidth estimator. The weak convergence theorem 4.2 extends to the weighted regression estimator.

4.4

Estimation for processes

Let (Xt )t∈[0,T ] be a continuously observed stationary and ergodic process satisfying (2.13), with values in IX . The limiting marginal density defined by (2.14) is estimated with an optimal bandwidth of order O(T 1/(2s+1) ) as proved in Section 2.10. For every x in IX,T,khT k Z 1 T fbT,hT (x) (x) = KhT (x) (Xs − x) ds (4.3) T 0

where T 1/(2s+1) khT k = O(1). Conditions 2.1-2.2 are supposed to be satisfied, with a density f in class Cs and assuming that the bandwidth function fulfills Conditions 4.1 with the approximation hT (x) = T −1/(2s+1) {h1/(2s+1) (x) + o(1)}.

(4.4)

The results of the previous sections extends to prove that for every x in hs (x) IX,T,khT k , the bias of fbT,h (x) is bT,hT (x) = Ts! msK f (s) (x) + o(khT ks ), its variance is V ar{fbT,hT (x)} = (T hT (x))−1 κ2 f (x) + o((T −1 kh−1 T k),

b its covariances are o((T −1 kh−1 T k) and the Lp -norms are kfT,hT (x) − −1 1/p −1 fT,hT (x)kp = 0((T khT k) ). The ergodic property (2.16) for kdimensional vectors of values of the process (Xt )t entails the weak convergence of the finite dimensional distributions of the density estimator fbT,h . Lemma 4.2 extends to the ergodic process and entails the weak convergence of (T hT )1/2 (fbT,h − f ) to a Gaussian process with variance κ2 f (x) at x and covariances zero. For a continuously observed stationary and ergodic process (Xt , Yt )t≤T with values in IX,Y , consider the regression model Yt = m(Xt ) + σ(Xt )εt where (εt )t∈[0,T ] is a Brownian motion such that E(εt | Xt ) = 0 and E(εt εs | Xt ∧Xs ) = E{(εt ∧εs )2 | Xt ∧Xs ) = 1. The bivariate process Z is supposed

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Limits for the varying bandwidths estimators

FunctionalEstimation

85

to be ergodic, satisfying the properties (2.13) and (2.16). Under the same conditions as in Chapter 3, the regression function m is estimated on an interval IX,Y,T,khT k by the kernel estimator RT

Ys KhT (x) (x − Xs ) ds . m b T,hT (x) = 0R T KhT (x) (x − Xs ) ds 0

The bias and variances established in Section 3.10 for the functions f and m of class Cs and fixed bandwidth hT are modified, with the notation µ = mf bm,T,hT (x) (x) = hT (x)s bm (x) + o(khT ks ), msK −1 (s) bm (x) = f (x){µ(s) (x) − m(x)fX (x)}, s! X 2 vm,T,h (x) = (T hT (x))−1 σm (x) + o((T khT k)−1 ), −1 2 σm (x) = κ2 fX (x)V ar(Y | X = x)

and the covariance of m b T,hT (x) (x) and m b T,hT (x) (y) is a o((T khT k)−1 ). The weak convergence of the process (T hT (x))1/2 {m b T,hhT (x) (x) − m(x)} is then proved by the same methods, under the ergodicity properties. In a model with a variance function, the regression function is also −1 estimated using a weighting process w bT = σ bT,h in the estimator of the T ,δT regression function RT {Ys − m b T,hT (Xs ) (Xs )}2 KδT (x) (x − Xs ) ds 2 σ bT,hT ,δT (x) = 0 , RT 0 KδT (x) (x − Xs ) ds RT w bT (Xi )Yi KhT (x) (x − Xi ) . m b wbT ,T,hT (x) (x) = R0 T w b (X )K (x − X ) T i i h (x) T 0

The previous modifications of the bias and variance of the estimator extend to the continuously observed process (Xt )t≤T . 4.5

Exercises

(1) Compute the fixed and varying optimal bandwidths for the estimation of a density and compare the respective density estimators. (2) Give the expressions of the first moments of the varying bandwidth estimator of the conditional probability p(x) = P (Y |X = x) for a Y binary variable, conditionally on the value of a continuous variable X (Exercise 3.10-(2)).

January 31, 2011

86

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

(3) For the hierarchical observations of n independent sub-samples of Ji dependent observations of Exercise 2.11-(5), determine a varying bandwidth estimator for the limiting density f and ergodicity conditions for the calculus of its bias and variance, and write their first order approximations. (4) Write the expressions of the bias and the variance of the continuous estimator FbY |X,n,hn (x) for the distribution function of Y ≤ y conditionally on X ≤ x of Exercise 3.10-(8), with a varying bandwidth and prove its weak convergence.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Chapter 5

Nonparametric estimation of quantiles

5.1

Introduction

Let F be a distribution function with density f on R, Fbn its empirical distribution function and νn = n1/2 (Fbn − F ) the normalized empirical process. The process Fbn − F convergences to zero uniformly a.s. and in L2 , and νn converges weakly to B ◦ F , where B is the Brownian motion. The b n is the inverse functional for Fbn , it converges therefore in probquantile Q ability to the inverse QF of F , uniformly on [0, 1]. The quantile estimator is approximated as b n = QF − {νn ◦ QF }{f ◦ QF }−1 + {νn ◦ QF }2 {f 0 ◦ QF }{f ◦ QF }−3 + o(ν 2 ). Q n As a consequence, the quantile process

b n − QF ) = −n1/2 νn ◦ QF + rn n1/2 (Q f

converges weakly to a centered Gaussian process with covariance function {F (s ∧ t) − F (s)F (t)}{f ◦ QF (s)}−1 {f ◦ QF (t)}−1 , for every s and t in [0, 1]. The remainder term is such that supt∈[0,1] krn k is a oL2 (1). Consider the distribution function FY |X of the variable Y conditionally on the regression variable X, in the model Y = m(X) + ε with a continuous regression curve m(x) = E(Y |X = x) and an observation error ε such that E(ε|X) = 0 and V ar(ε|X) = σ 2 (X). It is defined with respect to the distribution function Fε of ε by FY |X (y; x) = P (Y ≤ y|X = x) = Fε (y − m(x)). (5.1) R The marginal distribution function of Y is FY (y) = Fε (y −R m(s)) dFX (s) and the joint distribution function of (X, Y ) is FX,Y (x, y) = 1{s≤x} Fε (y − 87

January 31, 2011

88

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

m(s)) dFX (s). The estimator of FY |X is defined by smoothing the regression variable with a kernel is Pn Kh (x − Xi )1{Yi ≤y} b Pn FY |X,n,h (y; x) = i=1 i=1 Kh (x − Xi )

and an estimator of Fε is deduced from those of FY |X , FX and m as X Fbε,n,h (s) = n−1 FbY |X,n,h (s + m b n,h (Xi ); Xi ). 1≤i≤n

In this expression, the estimator of the regression function can be weighted by the inverse of the square root of the kernel estimator for the variance function σ 2 . Therefore, all functions of the model, m, σ b2 , FY |X and Fε , are easily estimated from the sample (Xi , Yi )i≤n . The quantile of the conditional distribution function of Y given X are first defined with respect to Y , then with respect to X. For every t in [0, 1] and at fixed x in IX , the conditional distribution FY |X (y; x) is increasing with respect to y and its inverse is defined as QY (t; x) = FY−1 |X (t; x) = inf{y ∈ IY : FY |X (y; x) ≥ t}.

(5.2)

It is right-continuous with left-hand limits, like the FY |X . For every x ∈ IX , FY |X ◦ QY (t; x) ≥ t with equality if and only if FY |X (x) is (x, y) belongs to the support of (X, Y ). Assuming that the function m is monotone by intervals, the definition (5.1) implies the monotonicity on the same intervals of the conditional distribution function FY |X with respect to the Y , with the inverse monotonicity. On each interval of monotonicity and for every s in the image of IX , the quantile QX (y; s) is defined by inversion of the conditional distribution FY |X in the domain of the variable X, at fixed y, from equation (5.1)  inf{x ∈ IX : FY |X (y; x) ≥ t}, if m is decreasing, (5.3) QX (y; s) = sup{x ∈ IX : FY |X (y; x) ≤ t}, if m is increasing. For every y ∈ IY , QX ◦FY |X (y; x) = x if and only if m and Fε are continuous on IY and FY |X ◦QX (y; s) = s if and only if m and Fε are strictly monotone, for every (s, y) in DX,Y . The empirical conditional distribution function b X,n,h and Q b Y,n,h , defines in the same way the empirical quantile processes Q b according to (5.3) and (5.2) respectively. If (x, y) belongs to DX,Y,n,h , the bX,n,h and D bY,n,h marginal components x and y belong respectively to D b b which are the domains of QX,n,h and QY,n,h , respectively. Another question of interest for a regression function m monotone on an interval Im is to determine its inverse with its distribution properties.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Nonparametric estimation of quantiles

FunctionalEstimation

89

Consider a continuous regression function m, increasing on a sub-interval Im of the support IX of X, its inverse is defined as m−1 (t) = inf{x ∈ IX : m(x) ≥ t}.

(5.4)

It is increasing and continuous on the image of Im by m and satisfies m−1 ◦ m = m ◦ m−1 = id. 5.2

Asymptotics for the quantile processes

Let IX,Y,h = {(s, y) ∈ IX,Y ; [s − h, s + h] ∈ IX }. Under conditions similar to those of the nonparametric regression,RProposition 3.1 applies considering y as fixed, with fX (x)FY |X (y; x) = 1{ζ≤y} fX,Y (x, ζ) dζ instead of µ(x) and with the conditional function FY |X (y; x), for every (x, y) in IX,Y . The weak convergence of the process defined on IX,h , at fixed y, by (nh)1/2 {FbY |X,n,h (y; ·) − FY |X (y; ·)} is a corollary of Theorem 3.1. The expressions of the bias and the Lp -norms rely on an expansion up to higher order terms of its moments. Proposition 5.1. Let FXY be a distribution function of Cs+1 (IX,Y ). Under Conditions 2.1 for the density fX and 3.1 for the conditional distribution function FY |X (y; x) at fixed y, the variable supIX,Y,h |FbY |X,n,h − FY |X | tends to zero a.s., its bias and its variance are bFY |X ,n,h (y; x) = h2 bF (y; x) + o(h2 )

(5.5)

s+1

1 ∂ FX,Y (x, y) (2) −1 msK fX (x){ − FY |X (x, y)fX (x)}, s! ∂xs+1 vFY |X ,n,h (y; x) = (nh)−1 vF (y; x) + o((nh)−1 ) (5.6) bF (y; x) =

−1 vF (y; x) = κ2 fX (x)FY |X (y; x){1 − FY |X (y; x)}.

At every fixed y in IY , the process (nh)1/2 {FbY |X,n,h (y) − FY |X (y)}1{IX,h } converges weakly to a Gaussian process defined in IX , with mean function limn (nh5 )1/2 bFY |X (y; ·), covariances zero and variance function vFY |X (y; ·). The results for the bias of the estimator FbY |X,n,h extend to a density fX in Cs as in Lemma 3.2. The weak convergence of the bivariate process (nh)1/2 (FbY |X,n,h − FY |X ) defined on IX,Y,h requires an extension of the previous results as for the empirical distribution function of Y . Proposition 5.2. The process

νY |X,n,h = (nh)1/2 {FbY |X,n,h − FY |X )}1{IX,Y,h }

January 31, 2011

90

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

converges weakly to a Gaussian process Wν on IY,X , with mean function limn (nh5 )1/2 bF (y; ·), variance vY |X and covariances at fixed x −1 CovY |X (y, y 0 ; x) = κ2 fX (x){FY |X (y ∧ y 0 ; x) − FY |X (y; x)FY |X (y 0 ; x)},

and zero otherwise. Proof. This is a consequence of the weak convergence of the finite dimensional distributions of νY |X,n,h and of its tightness, due to the bound obtained for the moments of the squared variations between (x, y) and (x0 , y 0 ) of the joint empirical process, νY |X,n,h (y; x) − νY |X,n,h (y 0 ; x) − {νY |X,n,h (y; x0 ) − νY |X,n,h (y 0 ; x0 )} is a O((x0 − x)2 + (y 0 − y)2 ). The bound O((y 0 − y)2 ) is obtained for the empirical process at fixed x and x0 , and O((x0 − x)2 ) as in the proof of Lemma 3.3, at fixed y and y 0 .  Let FY |X (y; x) be monotone with respect to x. If n is sufficiently large, then FbY |X,n,h is monotone, as proved in the following lemma. The means b Y,n,h and, reare denoted FY |X,n,h , QY,n,h and QX,n,h for E FbY |X,n,h , E Q b X,n,h . spectively, E Q

Lemma 5.1. If n ≥ n0 large enough, FY |X,n,h is monotone on IX,Y,h . Moreover, if FY |X is increasing with respect to x in IX then, for every x1 < x2 and ζ > 0, there exists C > 0 such that Pr{FbY |X,n,h (x2 ) − FbY |X,n,h (x1 ) > C} ≥ 1 − ζ.

Proof. Let y be considered as fixed in IY , x1 < x2 be in IX,h and such that FY |X (y; x2 ) − FY |X (y; x1 ) = d > 0. For n large enough the bias of FbY |X,n,h (y; x2 ) − FbY |X,n,h (y; x1 ) is strictly larger than d/2, by Propo-

sition 5.2. The uniform consistency of Proposition 5.1 implies, for every η and ζ > 0, the existence of an integer n0 such that for every n ≥ n0 , Pr{|FbY |X,n,h (y; x1 )−FY |X (y; x1 )|+|FbY |X,n,h (y; x2 )−FY |X (y; x2 )| > η} < ζ. For the monotonicity of the empirical conditional distribution function, let d > η > 0, then Pr{FbY |X,n,h (y; x2 ) − FbY |X,n,h (y; x1 ) > d − η} = 1 − Pr{(FbY |X,n,h − FY |X )(y; x1 ) − (FbY |X,n,h − FY |X )(y; x2 ) ≥ η} ≥ 1 − Pr{|FbY |X,n,h − FY |X |(y; x2 ) + |FbY |X,n,h − FY |X |(y; x1 ) ≥ η}

≥ 1 − ζ.



The asymptotic behaviour of the quantile processes follows the same principles as the distribution functions. We first consider the quantile QY defined

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Nonparametric estimation of quantiles

91

by (5.2) conditionally on fixed X = x, it is always increasing. The empirical quantile function is increasing with probability tending to 1, as in Lemma 5.1 and the functions QX,n,h and QY,n,h are monotone, for n large enough. The results of Section 5.1, are adapted to the empirical quantiles. Another quantile function is defined for n large enough by  e Y,n,h (v; x) = sup y : (x, y) ∈ IX,Y,h , FY |X,n,h (y; x) ≤ v , v ∈ D bY,n,h . Q (5.7) e Y,n,h converges The uniform convergence of FY |X,n,h to FY |X implies that Q uniformly to QY . The derivative with respect to y of FY |X (y; x) belonging to C2 (IY ) is fY |X (y; x), for every x in IX . Let bF and vF be defined by (5.6) and (5.6), respectively. Proposition 5.3. Let FX|Y be a continuous conditional distribution funcb Y,n,h − QY |(u; x) converges in probability tion, the process supDbY,n,h ×IX |Q to zero. If the density fX,Y of (X, Y ) belongs to Cs (IX,Y ), then for every bY,n,h , the bias of Q b Y,n,h equals x in IX and u in D bY (u; x) = −h2

bF

fY |X

◦ QY (u; x) + o(h2 ),

and its variance is vY (u; x) = (nh)−1

vF ◦ QY (u; x) + o((nh)−1 ). {fY |X }2

Proof. By definition of the inverse function, for every x in IX,n,h and u bY,n,h , there exists an unique y in IY,n,h such that u = FbY |X,n,h (y; x), in D then by derivability of the inverse function b Y,n,h (u; x) − QY (u; x) = Q b Y,n,h ◦ FbY |X,n,h (y; x) − QY ◦ FbY |X,n,h (x) Q = QY ◦ FY |X (y; x) − QY ◦ FbY |X,n,h (y; x) =−

FbY |X,n,h (y; x) − FY |X (y; x) fY |X (y; x) + o(FbY |X,n,h (y; x) − FY |X (y; x)).

e Y,n,h satisfy a similar approximation with the functions The functions Q FY |X,n,h . By the uniform convergence in probability of FY |X,n,h to FY |X on IX,n,h and under the condition that the density is bounded away from b Y,n,h and the functions Q e Y,n,h convergence zero on IX , the processes Q uniformly to QY .

January 31, 2011

17:17

92

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

bY,n,h . In order to calculate the bias and variance At fixed x, let u be in D b of QY,n,h (u), we first determine the order of the bias and the variance of the processes b Y,n,h (u) − FY |X ◦ Q e Y,n,h(u) ηbY,n,h (u) = FY |X ◦ Q

(5.8)

which converge in probability to zero. Then the quantile estimator satisfies b Y,n,h = QY ◦ (b e Y,n,h ). Q ηY,n,h + FY |X ◦ Q

(5.9)

e Y,n,h as a function of Q and Q b Y,n,h as Taylor expansions allow to express Q e Y,n,h and of the process ηbY,n,h . Since FbY |X,n,h ◦ Q b Y,n,h and a function of Q e FY |X,n,h ◦ QY,n,h equal identity, (5.8) is also written

e Y,n,h (u) − {FbY |X,n,h − FY |X } ◦ Q b Y,n,h(u). ηbY,n,h (u) = bF,n,h ◦ Q e Y,n,h (u) − h2 E{bF ◦ Q b Y,n,h(u)} + o(h2 ), (5.10) E{b ηY,n,h (u)} = h2 bF ◦ Q

and the variance of ηbY,n,h (u) equals h i b Y,n,h (u)|Q b Y,n,h (u)} V ar{b ηY,n,h (u)} = E V ar{FbY |X,n,h ◦ Q b Y,n,h (u)} + V ar{bF,n,h ◦ Q b Y,n,h (u)} = (nh)−1 E{vF ◦ Q

b Y,n,h (u)} + o(n−1 h−1 + h4 ). (5.11) + h4 V ar{bF ◦ Q

The moments of order l ≥ 3 of ηbY,n,h are bounded using an expansion of the moments of the sum in Equation (5.8) by n o e Y,n,h (u)|l + E|(FbY |X,n,h − FY |X ) ◦ Q b Y,n,h(u)|l , 2l |bY,n,h ◦ Q

b Y,n,h(u)|l } the second right hand term E{|(FbY |X,n,h − FY |X,n,h + bY,n,h) ◦ Q is lower than

thus

b Y,n,h (u)|l |Q b Y,n,h (u)} 2l E[E{|(FbY |X,n,h − FY |X,n,h ) ◦ Q b Y,n,h(u)|l ], + |bY,n,h ◦ Q

h n e Y,n,h (u)|l + 2l E |bY,n,h ◦ Q b Y,n,h (u)|l E|b ηY,n,h (u)|l ≤ 2l |bY,n,h ◦ Q oi b Y,n,h (u)|l |Q b Y,n,h (u)} . + E{|(FbY |X,n,h − FY |X,n,h ) ◦ Q

By Propositions 2.2 and 3.1, the conditional expectation E{|FbY |X,n,h − FY |X,n,h |l is O((nh)−l/2 ), and both terms in bY,n,h are O(h2l ), hence E|b ηn,h (u)|l = o((nh)−1 ) for every l ≥ 3. The expression of the bias of

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

93

Nonparametric estimation of quantiles

e n,h (u) = h2 bF ◦ Q e n,h (u) + o(h2 ), FbY |X,n,h implies {FY |X,n,h − FY |X } ◦ Q therefore e Y,n,h(u) = F −1 (u − h2 bF ◦ Q e Y,n,h (u)) + o(h2 ) Q Y |X = QY (u) − h2

e Y,n,h(u) bF ◦ Q + o(h2 ). fY |X ◦ QY (u)

e Y,n,h (u) = QY (u) + O(h2 ), bF ◦ Q e Y,n,h(u) = bF ◦ QY (u) + O(h2 ), so Since Q that e Y,n,h(u) = QY (u) − h2 bF ◦ QY (u) + o(h2 ). Q (5.12) fY |X Furthermore, by (5.8),

b Y,n,h(u) = F −1 (FY |X ◦ Q e Y,n,h(u) + ηbY,n,h (u)) Q Y |X and, using (5.12),

e Y,n,h (u) + =Q

ηbY,n,h (u) 2 + O(b ηY,n,h (u)), e fY |X ◦ QY,n,h (u)

b Y,n,h(u) = Q e Y,n,h (u) + Q

ηbY,n,h (u) fY |X ◦ QY (u)

2 + O(h2 ηbY,n,h (u)) + O(b ηY,n,h (u)).

(5.13)

e Y,n,h (u) = bF ◦ QY (u) + o(1). With The expansion (5.12) implies bF ◦ Q (5.13) and since E{b ηY,n,h (u)} and V ar{b ηY,n,h (u)} are o(1) b Y,n,h (u) = bF ◦ Q e Y,n,h (u) + bF ◦ Q

ηbY,n,h (u) e Y,n,h(u) bf ◦ Q FY |X ◦ QY (u)

2 + O(h2 ηbY,n,h (u)) + O(b ηY,n,h (u)),

b Y,n,h (u)} = bF ◦ Q e Y,n,h (u) + o(1) = bF ◦ QY (u) + o(1). E{bF ◦ Q

b Y,n,h (u)} = O(h4 + n−1 h−1 ) because of the approxiMoreover, V ar{bF ◦ Q 4 mations V ar{b ηY,n,h (u)} = O(h4 +n−1 h−1 ) and E{b ηY,n,h (u)} = o(n−1 h−1 ). From (5.10), the expectation of ηbY,n,h (u) becomes E{b ηY,n,h (u)} = o(h2 ).

(5.14)

b Y,n,h (u)} = vF ◦ QY (u) + o(1) and In the expansion (5.11), E{vF ◦ Q 4 8 b h V ar{bF ◦ QY,n,h (u)} = O(h + n−1 h3 ) = o(n−1 h−1 ). The variance of ηbY,n,h (u) is then equal to V ar{b ηY,n,h (u)} = (nh)−1 vF ◦ QY (u) + o((nh)−1 ).

(5.15)

January 31, 2011

17:17

World Scientific Book - 9in x 6in

94

FunctionalEstimation

Functional estimation for density, regression models and processes

b Y,n,h (u) is deduced from (5.9), (5.12), (5.13) and (5.14), Finally, the bias of Q which imply b Y,n,h = QY +{(b e Y,n,h −FY |X ◦QY )/(fY |X ◦QY )}{1+o(1)} Q ηY,n,h +FY |X ◦ Q therefore

b Y,n,h(u)} = QY (u) − h2 E{Q

bF + o(h2 ). fY |X ◦ QY (u)

b Y,n,h (u). Equations (5.13) and (5.15) yield the variance of Q



b Y,n,h − QY }1 b Theorem 5.1. The process UY,n,h = (nh)1/2 {Q {DY,n,h } con-

verges weakly to UY =

Wν + γ 1/2 bF ◦ QY where Wν is the Gaussian process fY |X

limit of νY |X,n,h . bY,n,h , there exists an Proof. For every x in IX,n,h and for every u in D b unique y in IY,n,h such that u = FY |X,n,h (y; x) therefore b Y,n,h − QY )(u; x) = (Q b Y,n,h ◦ FbY |X,n,h − QY ◦ FbY |X,n,h )(y; x) (Q = (QY ◦ FY |X − QY ◦ FbY |X,n,h )(y; x).

From the convergence of FbY |X,n,h to FY |X , it follows 1/2

b Y,n,h − QY } = − {νY |X,n,h + γ (nh)1/2 {Q fY |X,n,h

and its limit is deduced from Proposition 5.2.

bY }

b Y,n,h , ◦Q



The representation of the conditional quantile process b Y,n,h = QY + {(b e Y,n,h − FY |X ◦ QY )/(fY |X ◦ Q b Y,n,h)} Q ηY,n,h + FY |X ◦ Q + rY,n,h ,

(5.16)

where ηbY,n,h is defined by (5.8) and where the remainder term rY,n,h is oL2 ((nhn )−1/2 ), was established to prove Proposition 5.3 and Theorem 5.1. b X,n,h An analogous representation holds for the quantile process Q b X,n,h = QX +{(ζbX,n,h +FY |X ◦ Q e X,n,h −FY |X ◦QX )/(fY |X ◦QX )}+rX,n,h Q

b X,n,h −FY |X ◦ Q e X,n,h and rX,n,h = oL2 ((nhn )−1/2 ). where ζbX,n,h = FY |X ◦ Q b X,n,h are The bias bX , the variance vX and the weak convergence of Q (1) deduced. Let FY |X be the derivative of FY |X (y; x) with respect to x.

Proposition 5.4. Let FX|Y be a continuous conditional distribution funcb X,n,h − QX |(u; x) converges in probability tion, the process supDbX,n,h ×IY |Q

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Nonparametric estimation of quantiles

FunctionalEstimation

95

to zero. If the density fX,Y of (X, Y ) belongs to Cs (IX,Y ), then for every bX,n,h , the bias of Q b X,n,h equals x in IX and u in D bX (y; u) = −h2

bF

∂FY |X /∂x

◦ QX (y; u) + o(h2 ),

and its variance is vX (y; u) = (nh)−1

vF ◦ QX (y; u) + o((nh)−1 ). {∂FY |X /∂x}2

b X,n,h − QX }1 b Theorem 5.2. The process UX,n,h = (nh)1/2 {Q {DX,n,h } con-

verges weakly to UX =

5.3

Wν + limn (nh5n )1/2 bF ◦ QX . ∂FY |X /∂x

Bandwidth selection

b Y,n,h (u) as an estimator of The error criteria measuring the accuracy of Q b Y,n,h (u), Q(u) are generally sums of the variance and the squared bias of Q where the variance increases as h tends to zero whereas the bias decreases. Under the assumption that FY |X is twice continuously differentiable with respect to y and using results of Proposition 5.3, the mean squared error n o2 b Y,n,h (u) − Q(u) is asymptotically equivalent to M SEY (h) = E Q vF ◦ QY (u; x) {fY |X ◦ QY (u; x)}2 2  bF 4 +h ◦ QY (u; x) . fY |X

AMSEQY (u; x, h) = (nh)−1

Its minimization in h leads to an optimal local bandwidth, varying with u and x  1/5 vF ◦ QY (u; x) −1/5 hopt,loc (u; x) = n . 4b2F ◦ QY (u; x)

That this also the optimal local bandwidth minimizing the AMSE of FbY |X,n,h (u; x) for the unique value of x such that y = QY (u), that is AMSEF (u; x, h) = (nh)−1 vF (u; x) + h4 b2F (u; x).

If the density fX has a continuous derivative, that is also identical to the b X,n,h (y; x), at fixed y, optimal local bandwidth minimizing the AMSE of Q by Proposition 5.4. Since the optimal rate for the bandwidth has the order b Y,n,h to QY is n−4/5 and the n−1/5 , the optimal rate of convergence of Q

January 31, 2011

96

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

b Y,n,h −QY } converges to a non-centered Gaussian process process (nh)1/2 {Q with an expectation different from zero because nh5 = O(1). Estimating the bias bF and the variance vF by bootstrap allows to estimate the optimal bandwidths for the quantile estimator without knowledge of its order s of derivability. A direct kernel estimator of the variance function of the process −1 νY |X,n,h is vbF,n,h = κ2 fbX,n,h Fb|X,n,h (1−Fb|X,n,h ), according to the expression (5.6) of vF . For a conditional distribution function FY |X (·; x) having derivatives of order s, the bias is modified bs,F,n,h (y; x) = hs bF (y; x) + o(hs ) hs ∂ s+1 FX,Y (x, y) −1 = msK fX (x){ s! ∂xs+1 (s) − FY |X (x, y)fX (x)} + o(h2 ), and the optimal local bandwidth is modified by this s-order bias. The global mean integrated squared error criteria are defined by intebY,n,h the AM SEQY (u; x, h), conditionally grating over all values of u in D on a fixed value of x Z b Y,n,h(u; x) − QY (u; x)}2 du AMISEQY (x, h) = E{Q Z vF ◦ QY (u; x) bY ◦ QY (u; x) 2 = [(nh)−1 + h4 { } ] du {fY |X ◦ QY (u; x)}2 fY |X ◦ QY (u; x) Z AMSEF (y; x, h) = dy, fY |X (y; x) IY,n,h R which differs from the integral AMISEF (x, h) = IY,n,h AMSEF (y; x, h) dy, conditional to X = x. The mean of the conditional random criterion AMISEQY (X, h) is Z AMSEF (y; x, h) dy dFX (x). fY |X (y; x) IX,Y,n,h In the same way, the global mean integrated squared error criteria is bX,n,h , for a fixed value of defined by integratingR AMSEQX (y, h) over D y, AMISEQX (y, h) = IX,n,h AMSEF (y; x, h){fY |X (y; x)}−1 dx, at fixed y. bX,Y,n,h The global AMISE criteria for QX and QY defined as integrals over D are both equal to

AMISEQ (h) =

Z

IX,Y,n,h

AMSEF (y; x, h) dx dy fY |X (y; x)

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

97

Nonparametric estimation of quantiles

and they differ from the global criterion Z AMSEF (Y ; X, h) AMISEF = AMSEF (y; x, h) dx dy = E . fX,Y (X, Y ) IX,Y,n,h Some discretized versions of these criteria are the Asymptotic Mean Average Squared Errors such as the AMASEF corresponding to AMISEF , AMASEQY (x, h) corresponding to AMISEQY (x, h) and EAMISEQY (X, h) are respectively defined by AMASEF = n−1 AMASEQY (x, h) = n−1

n X AMSEF (Yi ; Xi )

i=1 n X i=1

AMASEQY (h) = n−2

fX,Y (Xi , Yi , h)

,

AMSEF (Yi ; x, h) , fY2 |X (Yi ; x, h)

n X n X AMSEF (Yi ; Xj , h) i=1 j=1

fY2 |X (Yi ; Xj , h)

,

which is the empirical mean of AMASEQY (X, h). Similar ones are defined for QX and other means. Note that no computation of the global errors and bandwidths require the computation of integrals of errors for the empirical inverse functions, all are expressed through integrals or empirical means of AMSEF with various weights depending on the density of X and the conditional density of Y given X. The optimal window for AMASEQF (h) is Pn {vF (fX,Y )−1 }(Xi , Yi , h) 1/5 −1/5 n [ Pi=1 ] , n 4 i=1 {b2F (fX,Y )−1 }(Xi , Yi , h) for AMASEQY (h) it is Pn n

−1/5

Pn

i=1 [ Pn 4 i=1

Pj=1 n

{vF (fX,Y )−2 }(Xi , Yj , h)

2 −1 }(X , Y , h) i j j=1 {bF (fX,Y )

]1/5 .

The expressions of other optimal global bandwidths are easily written and all are estimated by plugging estimators of the density, the bias bF and the variance vF with another bandwidth. The derivatives of the conditional distribution function are simply the derivatives of the conditional empirical distribution function, as nonparametric regression curves. The mean b X,n,h squared errors and the optimal bandwidths for the quantile process Q are written in similar forms, with the bias bX and variance vX .

January 31, 2011

98

5.4

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Estimation of the conditional density of Y given X

The conditional density fY |X (y; x) is deduced from the conditional distribution function FY |X (y; x) by derivative with respect to y and it is estimated using the kernel K with another bandwidth h0 Z fbY |X,n,h,h0 (y; x) = Kh0 (v − y) FbY |X,n,h (dv; x) Pn n X Kh (x − Xi )I{Yi ≤Yj } Pn = Kh0 (Yj − y) i=1 . i=1 Kh (x − Xi ) j=1 Proposition 5.5. If the conditional density fY |X belongs to the class Cs (IXY ), s ≥ 2, the process (nhh0 )1/2 (fbY |X,n,h,h0 − fY |X ) converges weakly to a Gaussian process with mean limn (nhn h0n )1/2 (E fbY |X,n,h,h0 −fY |X ), with covariances zero and variance function vf = κ2 fY |X (1 − fY |X ). If h0n = hn and s = 2, the process n1/2 hn (fbY |X,n,hn − fY |X ) converges weakly to a Gaussian process with mean limn (nh6n )1/2 bf where bf =

1 (2) −1 m2K fX {∂ 2 fX,Y /∂y 2 + ∂ 2 fX,Y /∂x2 − fY |X fX }, 2

with covariances zero and variance function vf . The optimal bandwidth is O(n−1/6 ). Proof. By Proposition 5.1, if fY |X belongs to C2 (IXY ), its expectation develops as Z E fbY |X,n,h,h0 (y; x) = Kh0 (v − y) E FbY |X,n,h (dv; x) Z = Kh0 (v − y) (FY |X + bF,n,h )(dv; x) 0

∂ 2 fY |X (y; x) h2 = fY |X (y; x) + m2K 2 ∂y 2 0 ∂bF,n,h (y; x) + + o(h2 ) + o(h 2 ). ∂y

Assuming that h0 = h, its bias bf,n,h (y; x) = h2 bf (y; x) + o(h2 ), with bf =

1 (2) −1 m2K fX {∂ 2 fX,Y /∂y 2 + ∂ 2 fX,Y /∂x2 − fY |X fX }. 2

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

99

Nonparametric estimation of quantiles

Generally, the range of the variables X and Y differs and two distinct kernels have to be used, the bias is then expressed as a sum of two terms 0

(1)

0

bf,n,h,h0 (y; x) = h2 bF (y; x) + h 2 bf (y; x) + o(h2 ) + o(h 2 ), 1 (1) (2) −1 bF = m2K fX {∂ 2 fX,Y /∂x2 − fY |X fX }, 2 1 −1 2 bf = m2K fX ∂ fX,Y /∂y 2 . 2 The variance of the estimator is the limit of V arfbY |X,n,h,h0 (y; x) written Z Kh0 (u − y)Kh0 (v − y)Cov{FbY |X,n,h (du; x), FbY |X,n,h (dv; x)} Z −1 = (nh)−1 κ2 fX (x){ Kh20 (u − y) FY |X (du; x) Z − Kh0 (u − y)Kh0 (v − y) FY |X (du; x)FY |X (dv; x)}. 0

The first integral develops as I1 = h −1 {κ2 fY |X (y; x) + o(1)}, the second R integral I2 = Kh0 (u − y)Kh0 (v − y) FY |X (du; x)FY |X (dv; x) is the sum 2 of the integral outside the diagonal DY = {(u, v) ∈ IX,T ; |u − v| < 2h0n }, which is zero, and an integral restricted to the diagonal which is expanded by changing the variables like in the proof of Proposition 2.2. Let αh0 (u, v) = |u − v|/(2h0 ), u = y + h0 (z + αh0 ), v = y + h0 (z − αh0 ) and z = {(u + v)/2 − y}/(h0 ) Z I2 = Kh0 (u − x)Kh0 (v − x)fY |X (u; x)fY |X (v; x) du dv DY Z 0 −1 =h { K(z − αh0 (u, v))K(z + αh0 (u, v)) dzdufY2 |X (y; x) + o(1)} DY

0

0

and it is equivalent to h −1 κ2 fY2 |X (y; x) + o(h −1 ). The variance of the estimator of the conditional density fY |X is then vfY |X,n,h,h0 = (nhh0 )−1 vf (y; x) + o((nhh0 )−1 ), vf (y; x) = κ2 fY |X (1 − fY |X )

and its covariances at every y 6= y 0 tends to zero.

(5.17) 

The asymptotic mean squared error for the estimator of the conditional 0 (1)2 density is M SEfY |X y; x; hn , h0n ) = h4n bF (y; x)+hn4 b2f (y; x)+(nhn h0n )−1 vf , it is minimal at the optimal bandwidth ( )1/5 1 vf 0 hn,opt,fY |X (y; x) = (y; x) . nhn 4b2f

January 31, 2011

100

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

In this expression, hn can be chosen as the optimal bandwidth for the kernel estimator of the conditional distribution function FY |X . The convergence rate (nhn h0n )1/2 of the estimator for the conditional density is smaller than the convergence rate of a density and than (nh2n )1/2 = O(n2/5 ), at the optimal bandwidth. Assuming that h0n = hn , the optimal bandwidth is now hn,opt,fY |X (y; x) =

(

)1/6 1 vf (y; x) nh 2b2f

and the convergence rate for the estimator of the conditional density fY |X is n1/3 which is larger than the previous rate with two optimal bandwidths. The mode of the conditional density fY |X is estimated by the mode of its estimator and the proof of Proposition 2.5 applies with the modified rates of convergence and limit. The derivative of fbY |X,n,h (y; x) with respect to y 0 converges with the rate (nhn3 hn )1/2 that is n1/2 h2n for identical bandwidths. 5.5

Estimation of conditional quantiles for processes

Let (Zt )t∈[0,T ] = (Xt , Yt )t∈[0,T ] be a continuously observed stationary and ergodic process with values in a metric space IXY and the regression model Yt = m(Xt ) + σ(Xt )εt as in Section 3.10. Under the ergodicity condition (2.13) for (Zt )t>0 , the conditional distribution function of the limiting distribution corresponds to FY |X (y; x) for a sample of variables and it is estimated from the sample-path of the process on [0, T ], similarly to (3.22), with a bandwidth indexed by T FbY |X,T,hT (y; x) =

RT 0

1{Ys ≤y} KhT (x − Xs ) ds . RT 0 KhT (x − Xs ) ds

(5.18)

RT The numerator of (5.18), µ bRF,T,hT (y; x) = T1 0 1{Ys ≤y} KhT (x − Xs ) ds, has the mean µF,T,hT (x) = IX KhT (x − u) FXs ,Ys (du, y) = F (y; x)f (x) + h2T bF (y; x) + o(h2T ) with bµ (y; x) = ∂ 3 {F (x, y)}/∂x3 , for a conditional density of C2 (IXY ). Proposition 5.6. Under the ergodicity conditions and for a conditional density fY |X in class Cs (IXY ), the bias bF,T,hT and the variance vF,T,hT

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

101

Nonparametric estimation of quantiles

of the estimator FbY |X,T,hT are

bF,T,hT (y; x) = hsT bF (y; x) + o(hsT ), msK −1 (s) bF (y; x) = f (x){∂ s+1 F (x, y)/∂xs+1 − F (x)fX (x)}, s! X vF,T,hT (y; x) = (T hT )−1 {σF2 (y; x) + o(1)}, −1 σF2 (x) = κ2 fX (x)FY |X (y; x){1 − FY |X (y; x)}

−1 its covariances are CovY |X (y, y 0 ; x) = κ2 fX (x){FY |X (y ∧ y 0 ; x) − FY |X (y; x)FY |X (y 0 ; x)} and zero for x 6= x0 .

The weak convergence of Proposition 5.2 is still satisfied with the convergence rate (T hT )1/2 and the notations of Proposition 5.6. The quantile processes of Section 5.2 are generalized to the continuous process (Xt , Yt )t>0 and their asymptotic behaviour is deduced by the same arguments from the weak convergence of (T hT )1/2 (FbY |X,T,hT − FY |X ). The conditional density fY |X (y; x) of the ergodic limit of the process is estimated using the kernel KhT , with the same bandwidth as the estimator of the distribution function FY |X (R T ) Z T K (x − X )1 1 h s {Y ≤Y } T s t 0 KhT (Yt − y) dt. fbY |X,T,hT (y; x) = RT T 0 0 KhT (x − Xs ) ds

Its expectation is approximated by Z T −1 fY |X,T,hT (y; x) = T E KhT (Yt − y){FY |X (Yt ; x) + h2T bF (Yt ; x)} dt 0

=E

Z

IY

+ o(h2T )

KhT (v − y)

= fY |X (y; x) + h2T

∂ {FY |X + h2T bF }(v; x) dv + o(h2T ) ∂v

∂ 2 fY |X (y; x) ∂bF (y; x) h2T + m2K ∂y 2 ∂y 2 + o(h2T ),

where bF is the bias (5.6) of the estimator FbY |X,n,h . Let vf be defined by (5.17), the variance of fbY |X,T,hT (y; x) has an expansion similar to the variance of the estimator fbY |X,n,h (y; x) vfY |X ,T,hT = (T h2T )−1 vf (y; x) + o((T h2T )−1 ).

Proposition 5.7. Under the ergodicity conditions and for a conditional density fY |X in class Cs (IXY ), the bias and the variance of the estimator

January 31, 2011

102

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

fbY |X,T,hT are

bfY |X ,T,hT (y; x) = hsT bfY |X (y; x) + o(hsT ), msK −1 (s) bfY |X (y; x) = f (x){∂ s f (x, y)/∂xs − fY |X fX (x)}, s! X vfY |X ,T,hT (y; x) = (T h2T )−1 {vf (y; x) + o(1)}, −1 vf (y; x) = κ2 fX (x)fY |X (y; x){1 − fY |X (y; x)}

its covariances are zero for x 6= x0 or y 6= y 0 . The process T 1/2 hT (fbY |X,T,hT − fY |X ) converges weakly to a Gaussian process with mean limT T 1/2 hs+1 bfY |X , variance vf and covariances are T zero. The optimal bandwidth for fbY |X,T,hT is O(T −1/(2s+2) ) and the convergence rate of fbY |X,T,hT with the optimal bandwidth is T s/(2s+2) , hence it is T 1/3 for s = 2, and the expression of the optimal bandwidth is hT,opt,fY |X defined in the previous section.

5.6

Inverse of a regression function

Consider the inverse function (5.4) for a regression function m of the model (1.6), monotone on a sub-interval Im of the support IX of the regression variable X. The kernel estimator of the function m is monotone on the same interval with a probability converging to 1 as n tends to infinity, by an extension of Lemma 5.1 to an increasing function. The maxima and minima of the estimated regression function, considered in Section 3.7, define empirical intervals for monotonicity where the inverse of the regression function is estimated by the inverse of its estimator. Let t belong to the image Jm by m of an interval Im where m is increasing b m,n,h (t) = m Q b −1 (t) = inf{x ∈ Im : m b n,h (x) ≥ t}. (5.19) n,h

This estimator is continuous like m b n,h , so that m b n,h ◦ m b −1 n,h = id on Jm , and −1 m b n,h ◦m b n,h = id on Im . The results proved in Section 5.2 for the conditional b m,n,h . The bias and the variance of the quantiles adapt to the estimator Q estimator (5.19) on Jm are deduced from those of the estimator m b n,h , as in Proposition 5.3 bm bQm ,n,h (t) = −h2 (1) ◦ Qm (t) + o(h2 ), m 2 σm vQm ,n,h (t) = (nh)−1 (1)2 ◦ Qm (t) + o((nh)−1 ). m

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Nonparametric estimation of quantiles

FunctionalEstimation

103

b m,n,h − Qm ) is a consequence of TheThe weak convergence of (nh)1/2 (Q orem 3.1 and it is proved by the same arguments as Theorem 5.1 and proved by the same arguments. Let W1 be the Gaussian process limit of −1 σm (nh)1/2 (m b n,h − mn,h ) on Im .

b m,n,h − Qm } conTheorem 5.3. On Jm , the process UQm ,n,h = (nh)1/2 {Q 1/2 W1 + γ bm verges weakly to UQm = ◦ Qm . m(1)

The inverse of the estimator (3.22) for a regression function of an ergodic and mixing process (Xt , Yt )t≥0 is (T hT )1/2 -consistent and it satisfies the same approximations and weak convergence, with the notations and conditions of Section 3.10. Under derivability conditions for the kernel, the regression function and the density of the variable X, the estimators m b n,h and its inverse are differentiable and they belong to the same class which is supposed to be sufficiently large to allow expansions of order s for estimator of function m in Ck+s . The derivatives of the quantile are determined (1) b (1) b m,n,h }−1 , by consecutive derivatives of the inverse: Q b n,h ◦ Q m,n,h = {m (2) (1)3 −1 b (2) = {m b m,n,h . Q b {m b } }◦Q m,n,h

n,h

n,h

Consider a partition of the sample in J disjoint groups of size nj , and PJ let Aj be the indicator of a group j, for j = 1, . . . , J. Let Y = j=1 Yj 1Aj PJ and X = j=1 Xj 1Aj where (Xj , Yj ) is the variable set in group j. For j = 1, . . . , J, the regression model for the variables (Xji , Yji )i=1,...,nj is Yji = mj (Xji ) + εji

where mj (x) = E(Y | 1Aj X = x) and the expectation in the whole sample is defined from the probability pj of Aj , the conditional density of X given Aj and the conditional regression functions given the group Aj m(x) =

J X

πj (x)mj (x),

j=1

πj (x) = pj

fj (x) = P (Aj | X = x). f (x)

The density of X in the whole sample is a mixture of J densities condiPJ tionally on the group fX (x) = j=1 pj fj (x) and the ratio f −1 (x)fj (x) is one if the partition is independent of X. The regression functions and the

January 31, 2011

104

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

conditional probability densities are estimated from the sub-samples Pnj i=1 Yji Kh (x − Xji ) m b j,n,h (x) = P , nj i=1 Kh (x − Xji ) PJ Pnj i=1 Yji Kh (x − Xji ) j=1 m b n,h (x) = PJ Pnj , i=1 Kh (x − Xji ) j=1 Pnj Kh (x − Xji ) π bj,n,h (x) = PJ i=1 . Pnj i=1 Kh (x − Xji ) j=1

b j,m,n,h are defined as in Equation (5.19) for each The inverse processes Q group. The inverse of the conditional probability densities πj are estimated using the same arguments. 5.7

Quantile function of right-censored variables

The product-limit estimator Fbn for a differentiable distribution function F on R+ under right-censorship satisfies Equation (2.11) on [0, max Ti [ Z x 1 − FbnR (s− ) b d(Λn − Λ)(s), Fbn (x) = F − {1 − F (x)} 1 − F (s) 0

denoted F − ψn where Eψn = 0 and supt≤τ kψn (t)k2 converges a.s. to b n converges zero for every τ < τF = sup{x > 0; F (x) < 1}. Its quantile Q therefore to QF in probability uniformly on [0, 1[. Let f be the density probability for F and let G be the distribution function of the independent censoring times, the process n1/2 ψn converges weakly to a centered Gaussian process with covariance function Z x∧y {(1 − F )(1 − G− )}−1 dF, CF (x, y) = {1 − F (x)}{1 − F (y)} 0

at every x and y in [0, τ < τ0 ], where τ0 = τF ∧ τG . As a consequence, the quantile process b n − QF ) = −n1/2 n1/2 (Q

ψn ◦ QF + rn f

(5.20)

is unbiased and it converges weakly to a centered Gaussian process with covariance function c(s, t) = CF (QF (s), QF (t)){f ◦QF (s)}−1 {f ◦QF (t)}−1 , for every s and t in [0, F (τ0 )]. The remainder term is such that supt≤F (τ0 ) krn k is a oL2 (1).

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

105

Nonparametric estimation of quantiles

A smoothed quantile process is defined by integrating the smoothed R bn (s) which is an uniformly consistent estimator of the process Kh (t−s) dQ R (1) derivative QF (t) = 1/{f ◦ QF (t)} of QF (t). Its mean is Kh (t − s) dQ(s) 2 (3) and its bias bQF ,n,h = h2 m2K QF (t) if F belongs to C3 . Its variance and covariance functions are deduced from the representation (5.20) of the quantile, for s 6= t and as n tends to infinity b n,h (t), Q b n,h (s)} = n−1 Cov{Q

Z tZ 0

0

s

Z

1

−1

Z

−1 −1

× Kh (v − v 0 ) du dv d2 c(u0 , v 0 ) + (nh)

= (nh)−1 κ2 c(s ∧ t, s ∧ t) + o(n−1/2 ). 5.8

1

1{u6=v} Kh (u − u0 )

κ2 c(s ∧ t, s ∧ t) + o(n−1/2 )

Conditional quantiles with variable bandwidth

The pointwise conditional mean squared errors for the empirical conditional distribution function and its inverses reach their minimum at a varying bandwidth function. So the behaviour of the estimators with such bandwidth is now considered. Conditions 4.1 are supposed to be satisfied in addition to 2.1 or 2.2. The results of Propositions 5.1 and 5.2 still hold with a functional bandwidth sequence hn and approximation orders o(khn k2 ) for the bias and o(nkh−1 n k) for the variance and a functional convergence rate (nhn )1/2 for the process νY |X,n,h . This is an application of Section 4.3 with the following expansion of the covariances. Lemma 5.2. The covariance of FbY |X,n,h (y; x1 ) and FbY |X,n,h (y; x2 ) equals 2[nκ2 {hn (x1 ) + hn (x2 )}]−1

× [vF (y; zn (x1 , x2 ))

Z

K(v − αn (v))K(v + αn (v)) dv (1)

−2 + δn (x1 , x2 )fX (zn (x, y)){(vΛ Y |XfX )(1) − m2 fX }(zn (x, y)) Z × vK(v − αn (v))K(v + αn (v)) dv + o(khn k)].

The mean squared errors the functional bandwidth sequences are similar to the MSE and MISE of Section 5.3. The conditional quantiles are now defined with functional bandwidths satisfying the convergence condition 4.1. The representation of the condi-

January 31, 2011

17:17

106

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

tional quantiles becomes b Y,n,h = QY + (b e Y,n,h − FY |X ◦ QY )(fY |X ◦ QY )−1 Q ηY,n,h + FY |X ◦ Q + oL2 ((nkhn k)−1/2 ),

b X,n,h = QX + (ζbX,n,h + FY |X ◦ Q e X,n,h − FY |X ◦ QX )(fY |X ◦ QX )−1 Q + oL2 ((nkhn k)−1/2 )

b X,n,h − FY |X ◦ Q e X,n,h . with ηbY,n,h defined by (5.8) and ζbX,n,h = FY |X ◦ Q The expansions of their bias and variance are also written with the uniform norms of the bandwidths, generalizing Propositions 5.3 and 5.4 and the weak convergence of the quantile processes is proved as for the kernel regression function with variable bandwidth in Section 4.4. 5.9

Exercises

(1) Consider the quantile process Fbn−1 of a continuous R 1 distribution function F and the smooth quantile estimator Tbn (t) = 0 Kh (t − s)Fbn−1 (s) ds, for t in [0, 1]. Prove its consistency and write expansions for its bias and its variance. (2) Determine the limiting distribution of the quantiles with respect to X and Y for the estimator of the distribution function of Y ≤ y conditionally on X ≤ x. (3) Determine the limiting distribution of smoothed quantiles with respect to X and Y for the estimator of the distribution function of Y ≤ y conditionally on X ≤ x.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Chapter 6

Nonparametric estimation of intensities for stochastic processes

6.1

Introduction

Let Nn = {Nn (t), t ≥ 0} be a sequence of counting processes defined on a probability space (Ω, A, P ) associated to a sequence of random time variables (Ti )1≤i≤n Nn (t) =

n X i=1

1{Ti ≤t} , t ≥ 0,

where Ti = inf{t; Nn (t) = i}, and let Fn = (Fnt )t∈R+ denote the history generated by observations of Nn and other observed processes before t. The predictable compensator of Nn with respect to Fn is the unique en such that Nn − N en is a Fn Fn− -measurable (or predictable) process N martingale on (Ω, A, P ). Consider a counting process Nn with a predictable compensator n Z t X en (t) = N Yi (s)µ(s, Zi (s)) ds i=1

0

where Yi and Zi are predictable processes with values in metric spaces Y and Z and µ(s, z) = λ(s)r(z) is a strictly positive function for s > 0. This model with a random variable or process Z is classical, when the observaP tions are a right-censored counting process Nn and Yn (t) = ni=1 1{Ti ≥t} is the counting process of the random times of Nn after t. The right-censorship is defined by a sequence of random censoring variables (Ci )1≤i≤n so that a Pn censoring process NnC (t) = i=1 1{Ci ≤t} is partially observed with the proP cesses Nn , the observations are the processes Yn (t) = 1≤i≤n 1{Ti ∧Ci ≥t} and the sequences of times (Ti ∧ Ci )1≤i≤n and indicators (δi )i = (1{Ti ∧Ci } )i with values 1 if Ti is observed and 0 otherwise. All processes are observed in an increasing time interval [0, τ ] such that Nn (τ ) = n tends to infinity. 107

January 31, 2011

108

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

With independent and identically distributed variables Ti with distribution function FT and density R t fT , the relationships between the survival function 1 − FT (t) = exp{− 0 λ(s) ds} and the hazard function λT = (1 − FT )−1 fT are equivalent. With independent and identically distributed censoring variables Ci , independent of the time sequence (Ti )1≤i≤n and with distribution function FC , the hazard function of the censored counting process P Nn (t) = 1≤i≤n δi 1{Ti ≤t} is identical to λT . The aim of this chapter is to define smooth estimators for the baseline hazard function and regression function of intensity models and to compare them with histogram-type estimators. Several regression models are considered, with parametric or nonparametric regression functions. Let Jn (t) = 1{Yn (t)>0} be the indicator of censored times occurring after t. The baseline intensity λ of an intensity µn (t) = λ(t)Yn (t) is estimated for t in [h, τ − h] by smoothing R t the Nelson (1972) estimator of the cumulative hazard function Λ(t) = 0 λ(s) ds, which is asymptotically equivalent to Rt −1 e to 1 in probability. The unbiased Nelson 0 Jn (s)Yn (s) dNn (s) as Jn tends R b n (t) = t Jn (s)Yn−1 (s) dNn (s), with the convention estimator is defined as Λ 0 bn 0/0 = 0, and the function λ is estimated by smoothing Λ Z 1 b λn,h (t) = Yn−1 (s)Jn (s)Kh (t − s) dNn (s). (6.1) −1

A stepwise estimator for λ is also defined on an observation time [0, τ ] as the ratio of integrals over the subintervals (Bjh )j≤Jh of a partition of the observation interval into Jh = h−1 τ disjoint intervals with length h tending to zero. For every t belonging to Bjh , the histogram-type estimator of the funtion λ is estimated at t by Z Z e λn,h (t) = { Jn (s) dNn }{ Yn (s) ds}−1 (6.2) Bjh

Bjh

where the normalizing h of the histogram for a density is replaced by an integral. Consider a multiplicative intensity µ(t, Zi (t)) = λ(t)r(Zi (t))Yi (t) for the counting process Ni (t) = 1{Ti ≤t} of the i-th time variable. In the Cox model, the regression function r defining the point process is exp(β T z), with an unknown parameter β belonging to an open bounded set of Rd and z in the metric space (Z, k·, k) of the sample-paths of a regression processes Zi . The intensity conditionally on Ft is then the semi-parametric process X λ(t; β) = 1{Ti ≤t} r(Zi (t))λ(t) 1≤i≤n

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Nonparametric estimation of intensities for stochastic processes

FunctionalEstimation

109 (0)

and the estimators of λ and β are defined by the means of the process Sn defined by weighting each term of the sum in YT by the regression function at the jump time Sn(0) (t; β) =

n X

rZi (t; β)1{Ti ≥t} ,

i=1

with the parametric function rZ (t; β) = exp{β T Z(t)}. For k = 1, 2, let also Pn (0) (0) ⊗k with Sn (t; β) = i=1 rZi (t; β)Zi (t)1{Ti ≥t} be the derivatives of Sn respect to β, let Z ⊗0 = 1, Z ⊗1 = Z and Z ⊗2 be the scalar product. The true regression parameter value is β0 , or r0 for the function r and the predictable compensator of the point process Nn is Z t en (t) = (6.3) N Sn(0) (s; β)λ(s) ds . 0

The function λ is estimated by smoothing the cumulative hazard function of the Cox process, the parameter β by maximizing the partial likelihood Z 1 bn,h (t; β) = λ Jn (s){Sn(0) (s; β)}−1 Kh (t − s) dNn (s), (6.4) −1 Y bn,h (Ti ; β)}δi , βbn,h = arg max {rZi (t; β)λ β

Ti ≤τ

with the convention 00 = 1 and Jn = 1{Yn >0} . The hazard function bn,h = λ bn,h (βbn,h ). The classical estimators of the Cox is estimated by λ model rely on the estimation of the function Λ(t) by the stepwise process R b n (t; β) = τ Jn (s){Sn(0) (s; β)}−1 dNn (s) at fixed β and the parameter β Λ 0 T of the exponential regression function rZ (t; β) = eβ Z(t) is estimated by bn,h (Ti ; β) is replaced maximization of an expression similar to (6.4) where λ b by the jump of Λn (β) at Ti Y βbnC = arg max {rZi (t; β)Sn(0)−1 (Ti ; β)}δi . β

Ti ≤τ

A stepwise estimator for the baseline intensity is now defined as Z Z X en,h (t; β) = λ 1Bjh (t)[ Jn (s) dNn (s)][ Sn(0) (s; β) ds]−1 , (6.5) Bjh

j≤Jh

βen,h = arg max β

Y

Ti ≤τ

Bjh

en,h (Ti ; β)}δi , {rZi (Ti ; β)λ

en,h = λ en,h (βen,h ). More generally, a nonparametric function r is estiand λ mated by a stepwise process rbn,h defined on each set Bjh of the partition

January 31, 2011

110

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

(Bjh )j≤Jh , centered at ajh . Let also (Dlh )l≤Lh be a partition of the values Zi (t), i ≤ n, centered at zlh . The function r is estimated in the form P ren,h (Z(t)) = l≤Lh ren,h (zlh )1Dlh (Z(t)) Z Z X en,h (t; r) = λ 1Bjh (t) Jn (s) dNn (s) [ Sn(0) (s; r) ds]−1 , Bjh

j≤Jh

ren,h (zlh ) = arg max rl

Y

Ti ≤τ

[{

X

l≤Lh

Bjh

en,h (Ti ; rlh )]δi , rl 1Dlh (Zi (Ti ))}λ

(6.6)

P (0) where Sn (t; r) = ni=1 rZi (t)1{Ti ≥t} is now defined for a nonparametric en,h (t, Z) = λ en,h (t, ren (Z(t))). A kernel estimator regression function, then λ for the functions λ is similarly defined by Z 1 b λn,h (t; r) = Jn (s){Sn(0) (s; r)}−1 Kh (t − s) dNn (s) . (6.7) −1

An approximation of the covariates values at jump times by z when they are sufficiently close allows to build a nonparametric estimator of the regression function r like β in the parametric model for r n Z 1 X bn (s; rz )}Kh (z − Zi (s)) dNi (s), rbn,h (z) = arg max {log rz (s) + log λ 2 rz

i=1

−1

where h2 = hn2 is a bandwidth sequence satisfying the same conditions as bn (t, Z) = λ bn (t; rbn (t, Z(t)). h, and λ The L2 -risk of the estimators of the intensity functions splits into a squared bias and a variance term and the minimization of the quadratic risk provides an optimal bandwidth depending on the parameters and functions of the models and having similar rates of convergence, following the same arguments as for the optimal bandwidths for densities.

6.2

Risks and convergences for estimators of the intensity

Conditions for expanding the bias and variance of the estimators are added to Conditions 2.1 and 2.2 of the previous chapters concerning the kernel and the bandwidths. For the intensities, they are regularity and boundedness conditions for the functions of the models and for the processes. Condition 6.1. (1) As n tends to infinity, the process Yn is positive with a probability tending to 1, i.e. P {inf [0,τ ] Yn > 0} tends to 1, and there exists a function g such that sup[0,τ ] |n−1 Yn − g| tends a.s. to zero;

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Nonparametric estimation of intensities for stochastic processes

FunctionalEstimation

111

Rτ (2) 0 g −1 (s)λ(s) ds < ∞ (3) The functions λ and g belong to Cs (R+ ), s ≥ 2. 6.2.1

Kernel estimator of the intensity

Let λn,h (t) =

Z

1

−1

Jn (s)Kh (t − s)λ(s) ds, for t ∈ [h, τ − h],

be the expectation of the kernel estimator (6.1) and let λn (t) = Jn (t)λ(t), defined as λ(t) on the random interval In,τ = 1{Jn =1 }[0, τ ] which may be right-censored. Let also In,h,τ = 1{Jn =1 }[h, τ − h] be the interval where all convergences will be considered. Since Jn (t) − 1 tends uniformly to zero in probability, supt∈[0,τ ] |λ(t) − λn,h (t)| tends to zero in probability. Proposition 6.1. Under Conditions 2.1 and 6.1 with hn converging to zero and nh to infinity bn,h (t)−λ(t)| converges to zero in probability. (b) For every (a) supt∈In,h,τ |λ t in In,h,τ , the bias λn,h (t) − λ(t) of the estimator is hs bλ,n,h (t; s) = msK λ(s) (t) + o(hs ), s! denoted h2 bλ (t) + o(h2 ), its variance is bn,h (t)} = (nh)−1 κ2 g −1 (t)λ(t) + o((nh)−1 ), V ar{λ bn,h (t) and also denoted (nh)−1 σλ2 (t) + o((nh)−1 ) and the covariance of λ b λn,h (s) is Z bn,h (s), λ bn,h (t)} = (nh)−1 { λ ( s + t ) K(v − αh )K(v + αh )dv + o(1)} Cov{λ g 2 if αh = |t − s|/2h ≤ 1 and zero otherwise, with uniform approximations on In,h,τ . The Lp -norms of the estimator are bn,h (t) − λn,h (t)kp = 0((nh)−1/p ) sup kλ t∈In,h,τ

bn,h (t) − λ(t)kp = 0((nh)−1/p + hs ). and supt∈In,h,τ kλ

b n − Λ, its predictable compensator is N fn . By Proof. (a) Let MΛ,n = Λ Lenglart’s inequality applied to the martingale MΛ,n , for every n ≥ 1 Z 1 bn,h − λ| ≥ η) ≤ η −2 E sup P ( sup |λ Kh2 (t − u)(Jn Yn−1 λ)(u) du [h,τ −h]

=η

−2

t∈[h,τ −h]

−1

(nh)

κ2 sup g

−1

−1

(t)λ(t)

t∈[0,τ ]

and the upper bound tends to zero as n tends to infinity.

January 31, 2011

112

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

(b) For every t in In,h,τ , the bias bλ,n,h (t) = λh (t) − λn,h (t) develops as Z 1 bλ,n,h (t) = Kh (t − u)E{Jn (u)λ(u) − Jn (t)λ(t)} du −1 1

=

Z

−1 s

=

E{Jn (t + hz)λ(t + hz) − Jn (t)λ(t)}K(z) dz

h (s) λ (t) + o(hs ), s!

where EYn (s) = P (Yn (s) > 0) = P (maxi≤n Ti > s) = 1 − FTn (s) belongs to ]0, 1[ for every s ≤ τ , for independent times Ti , i ≤ n. Its variance is Z 1 bn,h (t)} = E V ar{λ Kh2 (t − u)Jn (u)Yn−1 (u)λ(u) du −1 Z = (nh)−1 K 2 (z)EJn (t + hz)g −1 (t + hz)λ(t + hz) dz

= (nh)−1 κ2 g −1 (t)λ(t) + o((nh)−1 ). R bn,h (t) and λ bn,h (s) is The covariance of λ Kh (s − u)Kh (t − −1 u)Jn (u)Yn (u)λ(u) du, it is zero if αh = |x − y|/(2h) > 1 and, if αh ≤ 1, it is approximated by a change of variables as in Proposition 2.2 for the density, under Conditions 6.1. R bn,h (t)−λn,h (t) = 1 Kh (t−u)Jn (u)Y −1 (u) d(Nn − The Lp -moment of λ n −1 en )(u) are calculated using the martingale property of Mn = Nn − N en and N its stochastic integrals

bn,h (t) − λn,h (t)|3 = (nh)−1 {κ2 g −1 (t)λ2 (t) + o(1)}, E|λ bn,h (t) − λn,h (t)|4 = (nh)−1 {κ2 g −1 (t)λ3 (t) + o(1)}, E|λ bn,h (t) − λn,h (t)|5 = (nh)−1 {κ2 g −1 (t)λ4 (t) + o(1)}. E|λ

The higher order moments are deduced by iterations. In each case, the main term is expressed as the integral of the product of a squared kernel at a time Ti and other kernel terms at Tjk , where all time variables are independent.  bn,h (t) − λn,h (t)}2 develops on In,h,τ as the sum of For p = 2, E{λ bn,h (t)} a squared bias and a variance terms {λn,h (t) − λ(t)}2 + V ar{λ −2 2 2s (s)2 and its first order expansions are (s!) msK h λ (t) + o(h2s ) + (nh)−1 κ2 g −1 (t)λ(t) + o(n−1 h−1 ). The asymptotic mean squared error AM SE(t; b λn,h ) = (nh)−1 κ2 g −1 (t)λ(t) +

1 m2 h2s λ(s)2 (t) (s!)2 sK

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Nonparametric estimation of intensities for stochastic processes

113

is minimum for the bandwidth function  s!(s − 1)! κ2 λ(t) 1/(2s+1) hAMSE,n (t) = n−1/(2s+1) . 2m2sK g(t)λ(s)2 (t) bn,h at t is The global asymptotic mean integrated squared error for λ Z τ Z τ 1 2 2s bn,h ) = (nh)−1 κ2 m h λ(s)2 (t) dt, AM ISE(λ g −1 (t)λ(t) dt + sK 2 (s!) 0 0 it is minimum for the global bandwidth R τ −1  (t)λ(t) dt 1/(2s+1) −1/(2s+1) s!(s − 1)! κ2 0 g Rτ hAMISE,n = n . 2 (s)2 2msK 0 λ (t) dt It is estimated by R τ −1 b −1/(2s+1) s!(s − 1)! κ2 0 Yn (t) dΛn (t) 1/(2s+1) b hAMISE,n,h = n [ ] . R τ (s) b (t)}2 dt 2m2sK 0 {λ n,h

Another integrated asymptotic mean squaredR error is the average of τ AM SE(T ) for the intensity, E{AM SE(T )} = 0 AM SE dFT also written Z AM ISEn (h; FT ) =

{h2s bλ (t) + (nh)−1 σλ2 (t)}{1 − F (t)} dΛ(t)

and it is estimated by plugging estimators of the intensity, bλ and σλ2 into the empirical mean Z τ −1 b n,h(t)}Y −1 (t) dNn (t). n {h2s b2λ (t) + (nh)−1 σλ2 (t)} exp{−Λ n 0

Its minimum empirical bandwidth is Rτ n −2 o1/(2s+1) b b −1/(2s+1) 0 λn,h exp{−Λn,h }Yn dNn b hλ,n,h = n . R τ (s) b }2 exp{−Λ b n,h}Yn−1 dNn 2m2sK 0 {λ n,h

An estimator of the derivative λ(k) or its integral are defined by the means of the derivatives K (k) of the kernel, for k ≥ 1 Z b(k) (t) = K (k) (t − s)Jn (s)Y −1 (s) dNn (s). λ n n,h h

Proposition 2.3 established for the densities is generalized to the intensity λ. Lemma 2.1 allows to develop the mean of the estimator of the first derivative as Z (1)

(1)

Kh (u − t)λ(u) du Z Z h2 (3) (1) (1) = −λ (t) zK (z) dz − λ (t) z 3 K (1) (z) dz + o(h2 ) 6 2 h = λ(1) (t) + m2K λ(3) (t) + o(h2 ), 2

λn,h (t) =

January 31, 2011

114

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes (1)

s

for an intensity of C3 or λn,h (t) = λ(1) (t) + hs! msK λ(s) (t) + o(hs ) for an R b(1) is (nh3 )−1 g −1 (t)λ(t) K (1)2 (z) dz + intensity of Cs . The variance of λ n,h

o((nh3 )−1 ). The optimal local bandwidth for estimating λ(1) belonging to Cs is therefore R  λ(t) K (1)2 (z) dz 1/(2s+3) hAMSE (λ(1) ; t) = n−1/(2s+3) s!(s − 1)! . 2m2sK g(t)λ(3)2 (t) b(2) is the derivaFor the second derivative of the intensity, the estimator λ n,h

b(1) expressed by the means of the second derivative of the kertive of λ n,h nel. For a function λ in class C4 , the expectation of the estimator b(2) is λ(2) (t) = λ(2) (t) + h2 m2K λ(4) (t) + o(h2 ), so it converges uniλ n,h n,h 2 b(2) is h2 m2K λ(4) (t) + o(h2 ) and its variance formly to λ(2) . The bias of λ n,h 2 R (nh5 )−1 g −1 (t)λ(t) K (2)2 (z) dz + o((nh5 )−1 ).

Proposition 6.2. Under Conditions 2.2, for every integers k ≥ 0 and b(k) of the s ≥ 2 and for intensities belonging to class Cs , the estimator λ n,h k-order derivative λ(k) has a bias O(hs ) and a variance O((nh2k+1 )−1 ) on In,h,τ . Its optimal local and global bandwidths are O(n−s/(2k+2s+1) ) and the optimal L2 -risks are O(n−s/(2k+2s+1) ).

Consider the normalized process bn,h (t) − λ(t)}, t ∈ In,h,τ . Uλ,n,h (t) = (nh)1/2 {λ The tightness and the weak convergence of Uλ,n,h on In,h,τ are proved by studing moments of its variations and the convergence of its finite dimensional distributions. For independent and identically distributed observations of right-censored variables, the intensity of the censored counting process has the same degree of derivability as the density functions for the random times of interest. Lemma 6.1. Under Conditions 6.1, for every intensity of Cs there exists a constant C such that for every t and t0 in In,h,τ satisfying |t0 − t| ≤ 2h  bn,h (t) − λ bn,h (s) 2 ≤ C(nh3 )−1 |t − t0 |2 . V ar λ

bn,h (t0 ) − λ bn,h (t) develops Proof. Let t0 and t in In,h,τ , the variance of λ according to their variances given by Proposition 6.1 and the covariance between both terms which is zero if |t − t0 | > 1 as established in the same bn,h (t) − λ bn,h (t0 )|2 develops as proposition. The second order moment E|λ R {Kh (t−u)−Kh (t0 −u)}2 Jn (u)Yn−1 (u)λ(u) du and it is a O((t−t0 )2 n−1 h−3 n ), by the same approximation as for the proof of Lemma 2.2 and the uniform convergence of Jn Yn−1 . 

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Nonparametric estimation of intensities for stochastic processes

FunctionalEstimation

115

Theorem 6.1. Under Conditions 6.1, for a density λ of class Cs (Iτ ) and with nh2s+1 converging to a constant γ, the process bn,h − λ}1{I Uλ,n,h = (nh)1/2 {λ n,h,τ }

converges weakly to Wλ + γ 1/2 bλ , where Wλ is a continuous Gaussian process on Iτ with mean zero and covariance E{Wλ (t0 )Wλ (t)} = σλ2 (t)δ{t0 ,t} , at t0 and t in Iτ , and σλ2 (t) = g −1 (t)λ(t). Proof. The weak convergence of the finite dimensional distributions of R bn,h − λn,h )(t) = (nh)1/2 1 Kh (t − the process Wλ,n,h (t) = (nh)1/2 (λ −1 u)Jn (u)Yn−1 (u) dMn (u) on In,h,τ is a consequence of the convergence of its variance and of the weak convergence of the martingale n−1/2 Mn to a conR t∧t0 tinuous Gaussian process with mean zero and covariance 0 g(u)λ(u) du at t and t0 . The covariance between Wλ,n,h (t) and Wλ,n,h (t0 ), for t 6= t0 , is approximated by Z n−1 Kh (t − u)Kh (t0 − u)g(u)−1 λ(u) du Z
1{0≤α<1} λ λ { (t) + (t0 )} K(v − α)K(v + α)dv + o((nh)−1 ), = 2nh g g

where αRh = |t0 − t|/(2h) tends to infinity as h tends to zero, then the integral K(v − αh )K(v + αh )dv tends to zero. Lemma 6.1 and the bound E{λn,h (t)−λ(t)−λn,h (t0 )+λ(t0 )}2 ≤ C 0 (nh)−1 |t−t0 |2 imply that the mean of the squared variations of Un,h are O(h−2 |t − t0 |2 ) if |t − t0 | ≤ 2hn and O(|t − t0 |2 ) if |t − t0 | > 2hn . The process Un,h is therefore tight.  Corollary 6.1. The process sup σλ−1 (t)|Uλ,n,h (t) − γ 1/2 bλ (t)|

t∈In,h,τ

converges weakly to supIτ |W1 |, where W1 is the Gaussian process with mean zero, variance 1 and covariances zero. For every η > 0, there exists a constant cη > 0 such that Pr{ sup |σλ−1 (Uλ,n,h − γ 1/2 bλ ) − W1 | > cη } In,h,τ

tends to zero as n tends to infinity. The Hellinger distance between two probability measures P1 and P2 defined by intensity functions λ1 and λ2 is Z λ1 1 − F1 1/2 h2 (P1 , P2 ) = {1 − ( )1/2 ( ) } dF1 λ2 1 − F2

January 31, 2011

17:17

116

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

R and it is also written h2 (P1 , P2 ) = {1 − ( λλ12 )1/2 e−(Λ1 −Λ2 )/2 } dF1 . The estimator of a function λ satisfies Z b b bn,h , λ) = {1 − ( λn,h 1 − Fn )1/2 } dF h2 (λ λ 1−F Z b λn,h 1 − Fbn 1/2 ) − 1} d(Fbn − F ) ≤ {( λ 1−F

bn,h , λ) to zero is nh1/2 and the convergence rate of h2 (λ n .

A varying bandwidth estimator is defined for multiplicative intensities under Condition 4.1, with the optimal convergence rate. The bias and the variance of the estimator are modified as hn (t)s bλ,n,hn (t) (t) = m2K λ(s) (t) + o(khn k2 ), s! its variance is bn,h (t) (t)} = (nhn (t))−1 κ2 g −1 (t)λ(t) + o((n−1 kh−1 k), V ar{λ n n

bn,h (t) (t) − λn,h (t) (t)kp = 0((n−1 kh−1 k)1/p ). The covariance of and Ekλ n n n bn,h (t) (t) and λ bn,h (t) (t0 )} equals λ n n Z E Khn (t) (t − u)Khn (t0 ) (t0 − u)Yn−1 (u) dΛ(u) Z 2 −1 0 = {(g λ)(z (t, t )) K(v − αn (v))K(v + αn (v)) dv n n{hn (t) + hn (t0 )} Z + δn (x, y)(g −1 λ)(1) (zn (x, y)) vK(v − αn (v))K(v + αn (v)) dv + o(khn k)} −1 with αn (x, y, u) = 21 {(u − x)h−1 n (x) − (u − y)hn (y)} and v = {(u − −1 −1 x)hn (x) + (u − y)hn (y)}/2. Lemma 4.2 is fulfilled for the mean bn,h (t) (t) which satisfy E|λ bn,h (t) (t) − squared variations of the process λ n n 0 2 −1 −1 3 0 2 −1 0 −1 0 b λn,h(t0 ) (t )| = O(n khn k (t − t ) ), if |thn (t) − t hn (t )| ≤ 1. Otherbn,h are zero, this implies the weak wise, the mean squared variations of λ 1/2 b convergence of the process (nhn (t)) {λn,hn (t) (t) − λ(t)}I{t ∈ In,khn k,τ } to the process Wf (t) + h1/2 (t)bf (t), where Wf is a continuous centered Gaussian process on Iτ with covariance σλ2 (t)δ{t,t0 } at t and t0 .

6.2.2

Histogram estimator of the intensity

The histogram estimator (6.2) for the intensity λ is a consistent estimator P as h tends to zero and n to infinity. Let Kh (t) = h−1 j∈Jτ,h 1Bjh (t) be

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

117

Nonparametric estimation of intensities for stochastic processes

the kernel corresponding to the histogram, the histogram estimator (6.2) is defined as the ratio of two stochastic integrals on the same subintervals of the partition of [0, τ ]. Its expectation is approximated by the ratio of the expectations of each integral, for t in Bj,h R R R g(s)λ(s) ds (n−1 Bj,h Jn dMn ) Bj,h (n−1 Yn − g)(s) ds B j,h en,h (t) = R R Eλ −E ( Bj,h g(s) ds)2 Bj,h g(s) ds R en,h (t)( λ {n−1 Yn − g)(s) ds}2 Bj,h R +E ( Bj,h g(s) ds)2 R g(s)λ(s) ds B = Rj,h + o(n−1/2 h1/2 ) = λn,h (aj,h ) + o(h) (6.8) g(s) ds Bj,h

en,h (t) can be approximated by an expanuniformly on Iτ,n,h . The bias of λ sion, assuming only that λ belongs to C1 (R+ ), it is written ebλ,h (t) = =

X

j≤Jτ,h

X

j≤Jτ,h

1Bjh (t){λ(ajh ) − λ(t)} + o(h) 1Bjh (t)|t − ajh |λ(1) (t) + o(h) = O(h)

also denoted ebλ,h (t) = hebλ (t) + o(h) and it is larger than the bias of kernel estimator. Assuming that V ar{n−1/2 (Yn − g)(t)} = O(1) for every t, the en,h (aj,h ) equals (nh)−1 V arn−1/2 Yn (ajh ) + variance of the denominator of λ en,h (t) is o((nh)−1 ) = O((nh)−1 . For every t in Bj,h , the variance of λ R R g(s)λ(s) ds 2  g(s)λ(s) ds 2  B j,h en,h (t)− R en,h (t)− BRj,h ven,h (t) = E λ − Eλ , Bj,h g(s) ds Bj,h g(s) ds and the last term is a o(n−1/2 h1/2 ), by (6.8). Following the same calculus as for the variance of the nonparametric estimator of a regression function, Z  −2  ven,h (t) = g(t) + o(h) V ar{(nh)−1 Jn dNn } − 2λ(t)Cov{(nh)−1

+ λ2 (t)V ar{(nh)−1

Z

Z

Bj,h

Bj,h

Bj,h

Jn dNn , (nh)−1

Z

Bj,h

Yn ds} + o((nh)−1 )

Yn ds}



January 31, 2011

118

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

with V ar{(nh)−1 covariance term Z −2 (nh) E{

R

Bj,h

Bj,h

Jn dNn } = (nh)−1 (gλ)(ajh ) + o((nh)−1 ) and the

Jn (dNn − Yn dΛ),

Z

Bj,h

(Yn − g) ds} = O((nh)−1 )

en,h (t) can be written ven,h (t) = (nh)−1 veλ (t) + therefore the variance of λ −1 en,h (t) is o((nh) ). The asymptotic mean squared error of the estimator λ minimal for the bandwidth 1/3

hn (t) = n−1/3 {2eb2λ (t)}−1/3 veλ (t).

This expression and the AMSE do not depend on the degree of derivability of the intensity. 6.3

Risks and convergences for multiplicative intensities

The estimators (6.4) for the exponential regression of the intensity are special cases of those defined by (6.7) in a multiplicative intensity with explanatory predictable processes and an unknown regression function r. For R bn,h (t; r) is still λn,h (t) = 1 Kh (t−s)λ(s) ds every t in In,h,τ , the mean of λ −1 and their degree of derivability is the same as K. With a parametric regression function r, the convergence in the first condition of 6.1 is replaced by the a.s. convergence to zero of (k) (k) (k) (k) supt∈[0,τ ] supkβ−β0 k≤ε |n−1 Sn (t; β) − s0 (t)|, where s0 = s0 (β0 ), for k = 0, 1, 2, and ε > 0 is a small real number. In a nonparametric model, this condition is replaced by the a.s. convergence to zero of (k) (k) (k) supt∈[0,τ ] supkr−r0 k≤ε |n−1 Sn (t; r) − s0 (t)|, where s0 = s(k) (r0 ), for k = 0, 1, 2. The previous conditions 6.1 are modified by the regression function. For expansions of the bias and the variance, they are now written as follows. Condition 6.2. (k)

(k)

(1) As n tends to infinity, the processes n−1 Sn (t; β) and n−1 Sn (t; r) are positive with a probability tending to 1 and the function defined by (k) s(k) (t) = n−1 ESn (t) belongs to class C2 (R+ ); (0) (2) The function pn (s) = Pr(Sn (s; r) > 0) belongs to class C2 (R+ ) and p R nτ(τ, r0 ) converges to 1 in probability; (3) 0 r(z)g −1 (s)λ(s) ds < ∞ (4) The functions λ and g belong to C2 (R+ ) and r belongs to Cs (Z).

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Nonparametric estimation of intensities for stochastic processes

6.3.1

119

Models with nonparametric regression functions

The regression funtion is estimated by rbn,h (z) = arg maxrz Lbn,h (z; r) where n Z X bn (s; rz )} Kh (z − Zi (s)) dNi (s) Lbn,h (z; r) = n−1 {log rz (s) + log λ Iτ,n,h

i=1

for t in In,h,τ . Its expectation Ln (z; r) = E Lbn,h (z; r) is expanded as Z bn (s; rz )} Kh (z − Z(s))S (0) (s; rZ )λ(s) ds Ln (z; r) = E log{rz (s)λ n Iτ,n,h

=E

Z

Iτ,n,h

bn (s; rz )}{S (0) (s; rz ) + log{rz (s)λ n

h2 κ2 Sn(2) (s; rz )} 2

× fZ(s) (z)λ(s) ds + o(h2 ),

where fZ(s) is the marginal density of Z(s). It follows that rbn (z) converges uniformly, in probability, to the value r0 (z) which minimizes the limit of n−1 Lbn (z; rz ) Z 1 L0 (z; rz ) = {log rz (s) + log λ(s; rz )}s(0) (s; rz )fZ(s) (z) λ(s) ds. −1

(k) Lbn

Let be the k-th order derivative of Lbn (z; r) with respect to z, their (k) (k) limits are denoted L0 and their expectations Ln , for k = 1, 2.

Proposition 6.3. Under Condition 6.2, the process (nh)1/2 (b rn,h − r) converges weakly to a Gaussian process with mean zero and variance (1) (L(2) )−1 V(1) {L(2) }−1 (z; r) where V(1) = limn→∞ nhV arLbn .

Proof. The first derivative of Lbn with respect to rz and its expectation bn depend on the derivative of λ Z b(1) (t; rz ) = − λ Kh (t − s)Sn(1) (s; rz )Sn(0)−2 (s; rz ) dΛ(s), n In,h,τ

−1 Lb(1) n = n

L(1) n = E

Z

n Z X i=1 1

−1

{

In,h,τ

{

b(1) λ 1 n (s; rz ) + } Kh (z − Zi (s)) dNi (s), bn (s; rz ) rz (s) λ

b(1) 1 λ n (s; rz ) + } Kh (z − Z(s))Sn(0) (s; rz ) fZ(s) (z)λ(s) ds b rt,z (s) λn (s; rz )

(1) (1) is such that V arLbn (z; r) = O((nh)−1 ), therefore (nh)1/2 (Lbn − L(1) )(z; r) is bounded in probability, and the second derivative is a Op (1). By a Taylor

January 31, 2011

17:17

120

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes (1)

expansion of Lbn (z; r) in a neighbourhood of the true value r0z ≡ r0 (z) of the regression function at z (1) Lb(1) (z; r0 ) = {(rz (0)−r0z (s)}T Lb(2) rn,h −r0 )2 (z(s))) n (z; r)−L n (z; r0 )+Op ((b

and, by an inversion, the centered estimator (b rn,h − r0 )(z) is approxi(2) −1 (1) b mated by the variable {−Ln (z; r0 )} L )(z; r0 ) the variance of which is a O((nh)−1 ). For every z in Zn,h = {z ∈ Z; supz0 ∈∂Z kz − z 0 k ≥ h}, the variable (nh)1/2 (b rn,h − r0 )(z) converges weakly to a Gaussian variable with (1) (2) −1 variance (L ) limn nhn V arLbn {L(2) }−1 (z; r0 ).  bn,h − λn,h )(r0 )1I Proposition 6.4. The processes (nh)1/2 (λ and τ,n,h Z (1) Sn bn,h −λ)+(nh)1/2 (b (nh)1/2 (λ rn,h −r0 ) (s; rbn,h )Kh (·−s) Jn (s) dNn (s)} (0)3 Sn converge weakly to the same continuous and centered Gaussian process on Iτ , with covariances zero and variance function vλ = κ2 s(0)−1 (r0 )λ. b n,h is The bias of Λ Z T bΛ,n,h (t) = − E{Sn(0) (s, r)Sn(0)−1 (s, rbn,h ) − 1} dΛ(s)

Proof.

=

Z

0

0

(1)

T

E{(b rn,h − r)(s)

Sn (s, rbn,h ) + Op (n−1/2 ) (0)

Sn (s, r)S, rbn,h )

} dΛ(s)

and it is also equivalent to the mean of Z T (b rn,h − r)(s)Sn(1) (s, rbn,h ){Sn(0) (t, rbn,h )}−3 dNn (s). 0

bn,h is obtained by smoothing the bias bΛ,n,h (t) The bias of the estimator λ and its first order approximation can be written as the mean of Z 1 (1) Sn (s, rbn,h ) bbλ,n,h (t) = (b rn,h − r)(t) Kh (t − s) (0) dNn (s). −1 {Sn (t, rbn,h )}3 

6.3.2

Models with parametric regression functions T

In the exponential regression model rβ (z) = eβ z with observations (Xi , δi )1≤i≤n , the estimator of the parameter β minimizes Lbn (β) which is written n X bn (Ti ; β)} . Lbn (β) = n−1 δi {β T Zi (Ti ) + log λ i=1

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Nonparametric estimation of intensities for stochastic processes

121

Its mean Ln (β) =

Z

In,h,τ

b {β T s(1) s(0) n (s; β0 ) + E log{λn (s; β)}b n (s; β0 )}λ(s) ds

converges to L(β) =

Rτ 0

{β T s(1) (s; β0 ) + {log λ(s; β)}s(0) (s; β0 )}λ(s) ds.

Proposition 6.5. Under Condition 6.2, n1/2 (βbn,h − β0 ) converges weakly bn,h − to a Gaussian variable with mean zero. The processes (nh)1/2 (λ λn,h )(β0 ) 1Iτ,n,h and 1/2

(nh)

bn,h −λ)(βbn )+n1/2 (βbn,h −β0 )T (λ

Z

(1)

Sn

Iτ,n,h

(0)2

Sn

(s; βbn,h )Kh (·−s) dNn (s)

converge weakly to the same continuous and centered Gaussian process with covariances zero and variance function vλ = κ2 s(0)−1 (β0 )λ. Proof. written

The derivatives with respect to β of the partial likelihood Lbn are −1 Lb(1) n (β) = n

n X i=1

δi {Zi (Ti ) +

b(1) λ n (Ti ; β) }, b λn (Ti ; β)

b(1)⊗2 b(2) λ λ n n
−1 Lb(2) δ (β) = −n − (Ti ; β), i n b2 bn λ λ n

bn,h with respect to β are written where the derivatives of λ Z

b(1) (t; β) = −n−1 λ n

(0)2

Sn

Iτ,n,h

Z

b(2) (t; β) = n−1 λ n

(1)

Sn

(s; β)Kh (t − s) dNn (s),

(1)⊗2

2

Iτ,n,h

Sn

(0)3

Sn

−

(2) Sn
(s; β)Kh (t − s) dNn (s). (0)2 Sn

(k) As h tends to zero, the predictable compensators of Lbn (β), k = 1, 2, develop as −1 L(1) n (β) = n

Z

In,h,τ

−1 L(2) n (β) = −n

Z

{Sn(1) (s; β0 ) +

In,h,τ

b(1) λ n (s; β) (0) S (s; β0 )}λ(s) ds, bn (s; β) n λ

b(1)⊗2 b(2) λ λ n n
− (s; β)Sn(0) (s; β0 )λ(s) ds. b2 bn λ λ n

January 31, 2011

122

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes (1)

bn , k = 1, 2, is deduced from the martingale property The expectation of λ e of Nn − Nn Z (1) Sn (s; β) (0) λ(1) (t; β) = Sn (s; β0 )λ(s)Kh (t − s) ds, n (0)2 In,h,τ Sn (s; β) (1)

Sn (t; β)

=

(0)2

Sn +

λ(2) n (t; β)

Z

=

(t; β)

(1) (2) m2K h2  Sn (t; β) (0) Sn (t; β0 )λ(t) + o(h2 ), (0)2 2 Sn (t; β) (1)⊗2

2

In,h,τ

= 2 + (1)

Sn(0) (t; β0 )λ(t)

Sn

(0)2

Sn

(1)⊗2

Sn

(0)2

Sn

−

−

(2) Sn
(s; β)Kh (t − s)λ(s) ds (0) Sn

(2) Sn
(t; β)λ(t) (0) Sn

(2) (1)⊗2 (2) Sn Sn
m2K h2 + o(h2 ). { 2 (0)2 − (0) (t; β)λ(t) 2 Sn Sn (2)

It follows that Ln (β) and Ln (β) converges to L(1) (t; β) Z τ (1) (1) (0)−2 L (β) = {s(1) (s; β)s(0)2 n (s; β0 ) − sn (s; β)sn n (s; β0 )}λ(s) ds, 0

L(2) (β) = − L(2) (β0 ) = −

Z

τ

Z0 τ

λ(1)⊗2 λ(2)
− (s; β)s(0) n (s; β0 )λ(s) ds, λ2 λ

s(2) (s; β0 )λ(s) ds,

0

where −L(2) (β0 ) is positive definite and L(1) (t; β0 ) = 0 so that the maximum βbn,h of Lbn converges in probability to β0 , the maximum of the (1) limit L of Ln . The rate of convergence of βbn,h − β0 is that of Lbn (β0 ). First n1/2 (Lb(1) )n − L(1) )n )(β0 ) is the sum of stochastic integrals of predictable processes with respect to centered martingales and it Rconvergences weakly to a centered Gaussian variable with variance v(1) = In,h,τ {s(2) − s(1)⊗2 s(0)−1 )(s; β0 )λ(s) ds. Secondly Z (1) − L )(β ) = n1/2 (L(1) [n1/2 {n−1 (Sn(1) − s(1) }(s; β0 ) 0 n In,h,τ

+ n1/2 {

b(1) λ n n−1 Sn(0) − s(1) }(s; β0 )]λ(s) ds, b λn

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Nonparametric estimation of intensities for stochastic processes

123

(1) (1) is continuous and independent of n1/2 (Lbn − Ln )(β0 ), its integrand is a (1) sum of three terms l1n + l2n + l3n where l1n = n1/2 {n−1 (Sn − s(1) }(s; β0 ) convergences weakly to a centered Gaussian variable with a finite variance, (1)

l2n = n1/2 {

λn,h λn,h

l3n = n1/2 [Sn(0) {

n−1 Sn(0) − s(1) }(·; β0 ), (1) b(1) λ λn,h n,h }](·; β0 ). − bn,h λn,h λ

The term l2n convergences weakly to a centered Gaussian variable with a finite variance and l3n has the same asymptotic distribution as b(1) − λ(1) ) − λ(1) λ−1 (λ bn,h − λn,h )}(·; β0 ) n1/2 s(0) λ−1 {(λ n,h

n,h

where the process bn,h − λn,h )(t; β0 ) (nh)1/2 (λ Z τ 1/2 en )(s) =n Sn(0)−1 (s; β0 )Kh (t − s) Yn (s) d(Nn − N 0

has the mean zero and the variance h

R

Iτ,n,h

(0)−1

Sn

(s; β0 )Kh2 (t − s)λ(s) ds

which converges in probability to vλ = κ2 s(0)−1 (t; β0 )λ(t). In the same way, the process Z (1) Sn (1) 1/2 b(1) 1/2 en )(s) n (λn,h − λn,h )(t; β0 ) = n (s; β0 )Kh (t − s) d(Nn − N (0)2 In,h,τ Sn is consistent and it has the finite asymptotic variance vλ,(1) (t) = s(1)⊗2 s(0)−3 (t; β0 )λ(t). The term l3n with asymptotic variance zero converges in probability to zero. The proof of the weak convergence of βbn ends as previously. The process bn,h − λ)(t; βbn ) develops as (nh)1/2 (λ Z en )(s) n1/2 Jn (s)Sn(0)−1 (s; βbn )Kh (t − s) d(Nn − N +

Z

Iτ,n,h

Iτ,n,h

{Sn(0)−1 (s; βbn ) − Sn(0)−1 (s; β0 )}Sn(0) (s; β0 )Kh (t − s)λ(s) ds

the first term of the right-hand side converges weakly to a centered Gaussian process with variance κ2 s(0)−1 (t; β0 )λ(t) and covariances zero, and the second term is expanded into Z − n1/2 (βbn,h − β0 )T Sn(1) (s; β0 )Sn(0)−1 (s; β0 )Kh (t − s)λ(s) ds Iτ,n,h

= −n

1/2

(βbn,h − β0 )T s(1) (t; β0 )s(0)−1 (t; β0 )λ(t) + o(1).



January 31, 2011

124

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

The results are analogous for every parametric regression function rβ Pn (0) of C2 , the processes are then defined by Sn (t; β) = i=1 rZi (t; β)1{Ti ≥t} (k) and Sn is its derivative of order k with respect to β −1 Lb(1) n (β) = n

L(1) n (β) = E

Z

(1)

b(1) (s; β) λ n,h δi { + } dNn (s), b r (Z (T )) β i i λn (s; β) i=1

n X

In,h,τ

−1 Lb(2) n (β) = −n

rβ (Zi (Ti )) (1)

{

n X

rβ (Zi (t) rβ (Zi (t)

+

b(1) λ n (s; β) } b λn (s; β)

× Sn(0) (t; β0 )λ(s) ds,

δi

i=1

b(1)2 b(2) λ λ n n
− (Ti ; β) . b2 bn λ λ n

All results of this section are extended to varying bandwidth estimators as before. 6.4

Histograms for intensity and regression functions

The histogram estimator (6.2) for the intensity λ with a parametric regression or (6.6), for nonparametric regression of the intensity, are consistent estimators as h tends to zero and n to infinity. Their expectations are approximated like (6.8) by a ratio of means. Their variances are calculated as in Section 6.2.2. The nonparametric regression function r is estimated by X ren,h (z) = ren,h (zlh )1Dlh (z) l≤Lh

and the histogram estimator for the intensity defines the estimator ren,h of the regression function by Y X en,h (Ti ; rlh )]δi , ren,h (zlh ) = arg max [{ rl 1Dlh (Zn (Ti ))}λ rl

en,h (t; r) = λ

X

j≤Jh

Ti ≤τ

1Bjh (t)

Z

l≤Lh

Jn (s) dNn (s) [

Bjh

Z

Bjh

Sn(0) (s; r) ds]−1 .

Pn (0) For every t in Bj,h , let Sn (t; r) = i=1 rZi (t)Yi (t), the limit of P −1 (0) (0) n Sn (t; r) is s (t; r) = l≤Lh rzlh Pr(Z(t) ∈ Dlh ) + o(1) and its variance is X v (0) (t; r) = n−1 rz2lh [Pr(Z(t) ∈ Dlh )g(t)−{Pr(Z(t) ∈ Dlh )g(t)}2 ]+o(1) l≤Lh

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Nonparametric estimation of intensities for stochastic processes

125

en,h (t; zlh ) is under Condition 6.1. The mean of λ R s(0) (s; rlh )λ(s) ds Bj,h R λn,h (t; rlh ) = + O((nh)−1 ) = λ(aj,h ) + o(h) (0) (s; r ) ds lh Bj,h s

and its bias is ebλ,h (t; rlh ) = hλ(1) (t)+o(h), uniformly on Iτ,n,h . Its variance is R s(0) (s; rlh )λ(s) ds 2  B e ven,h (t; rlh ) = E λn,h (t) − Rj,h + O((nh)−1 ) (0) (s; r ) ds s lh Bj,h Z  (0) −2  −1 = s (t; rlh ) V ar{(nh) Jn (s) dNn } −1

− 2λ(t)Cov{(nh)

Bj,h

Z

−1

Jn dNn , (nh)

Bj,h

+ λ2 (t)V ar{(nh)−1

Z

Bj,h

Z

Bj,h

Sn(0) (s; rlh ) ds}

Sn(0) (s; rlh ) ds} + o((nh)−1 ) + o(h)



where, for t in Bj,h and Z(t) in Dlh Z −1 V ar{(nh) Jn dNn } = (nh)−1 s(0) (t; rlh )λ(t) + o((nh)−1 ) Bj,h

V ar{(nh)−1 Cov{(nh)−1

Z

Z

Bj,h

Sn(0) (s; rlh ) ds} = (nh)−1 v (0) (t; rlh ) + o((nh)−1 ), Jn dNn , (nh)−1

Bj,h

Z

Bj,h

Sn(0) (s; rlh ) ds}

= (nh)−1 v (0) (t; rlh )λ(t) + o((nh)−1 )

therefore ven,h (t) = (nh)−1 veλ (t) + o((nh)−1 ) with

vλ (t) = s(0)−2 (s; rlh ){s(0) (t; rlh )λ(t) − v (0) (t; rlh )λ2 (t)} e

en,h −λn,h )(t) converges weakly to a centered Gaussian process and (nh)1/2 (λ with variance e vλ (t). en,h (t) is still minThe asymptotic mean squared error of the estimator λ −1/3 e2 −1/3 1/3 imal for the bandwidth hn (t) = n {2bλ (t)} veλ (t). The stepwise constant estimator of the nonparametric regression function r maximizes Z X Ln,h (r) = { log rl 1Dlh (Zn (s))} Jn (s) dNn (s) l≤Lh

+

Z

enh (s; rlh ) Jn (s) dNn (s) λ

January 31, 2011

126

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes (1)

(1)

and it satisfies Ln,h (e r1h , . . . , reLh ,h ) = 0, where Ln,h is a vector with components the derivatives with respect to the components of rh = (rlh )l≤Lh Z X 1 (1) Ln,h,l (rh ) = { 1D (Zn (s))} Jn (s) dNn (s) rlh lh l≤Lh Z e(1) (s; rlh ) Jn (s) dNn (s). + λ n,h,l

The derivatives of the intensity are consistently estimated by differences of values of the histogram, in the same way as the derivatives of a density. e(1) is a O((nh3 )−1 ) and the estimator of the regression The variance of λ n,h function converges with that rate. In the parametric regression model, the histogram estimator for the function λ and the related estimator of the regression parameter have the (0) same form (6.5), where the function r and the process Sn are indexed by the parameter β. Let t in Bj,h Z Z X e λn,h (t; β) = 1Bjh (t)[ Jn (s) dNn (s)][ Sn(0) (s; β) ds]−1 , Bjh

j≤Jh

βen,h = arg max β

Y

Ti ≤τ

Bjh

en,h (Ti ; β)}δi . {rZi (Ti ; β)λ

en,h are obtained by deriving Sn(0) with respect to β The derivative of λ R (1) Z X S (s; β) ds Bjh n (1) e λn,h (s; β) = − 1Bjh (t)[ Jn (s) dNn (s)] R (0) Bjh [ Bjh Sn (s; β) ds]2 j≤Jh

and the derivative of the logarithm of the partial likelihood for β is Z (1) n Z X rZi (1) e(1) (s; β) dNn (s). Ln,h (β) = n−1 (s; β) dNi (s) + n−1 λ nh r Z i i=1 (1)

e . Therefore, It is zero at βen,h and its convergence rate is a O((nh3 )), like λ nh 3 e βn,h has the convergence rate O((nh )) and the estimator of the hazard function has the convergence rate O((nh)). 6.5

Estimation of the density of duration excess

For the indicator NT of a time variable T , the probability of excess is Pt (t + x) = Pr(T > t + x | T > t) = 1 − Pr(t < T ≤ t + x | T > t) = 1 − Pr{NT (t + x) − NT (t) = 1 | NT (t) = 0}.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Nonparametric estimation of intensities for stochastic processes

FunctionalEstimation

127

For a sample of n independent and identically distributed variables (Ti )i≤n , Pn the processes n−1 Nn (t) = n−1 i=1 1Ti ≤t and n−1 Yn (t− ) = 1 − n−1 Nn (t) converge respectively to the functions F (t) and F¯ (t), and Pt (t + x) is estimated by Pbn,t (t + x) = 1 − {Nn (t + x) − Nn (t)}1{Nn (t)
Y

{1 −

1≤i≤n

1{t<Ti ≤t+x} Jn (Ti ) }, Yn (Ti )

(6.9)

with Yn = 1{Jn >0} . The estimator is constant between jump times Ti and the size of the jumps is ∆Pbn,t (Ti ) = 1{t<Ti } Pbn,t (Ti− )Jn (Ti )Yn−1 (Ti ). According to the asymptotic results established for the product-limit estimator, for every interval [a, b] included in the interval ]T1:n , Tn:n [, the product-limit estimator has values in the open interval ]0, 1[ and sup(t,x+t)∈[a,b] |Pbn,t (t + x) − Pt (t + x)| converges a.s. to zero. Let Bn,t (t + x) = n1/2

Pt − Pbn,t (t + x) Pt

(6.10)

also written Pbn,t (t + x) = Pt (t + x){1 − n−1/2 Bn,t (t + x)}.

Theorem 6.2. On every interval [a, b] included in the interval ]T1:n , Tn:n [, the estimator Pbn satisfies Z t+x 1/2 b n − Λ). Bn,t (t + x) = n d(Λ t

It converges weakly to a Gaussian process B with independent increments, R t+x mean zero and a finite variance vB (t + x; t) = t F¯ −1 dF .

b n be the martinProof. Let [a, b] be a sub-interval ofR ]T1:n , Tn:n [ and let Λ t −1 ¯ b gale estimator of Λ = − log F , Λn = 0 Jn Yn dNn is uniformly consistent b n − Λ) converges weakly to a centered Gaussian process on [a, b] and n1/2 (Λ

January 31, 2011

17:17

128

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Rt with independent increments and variance vΛ (t) = 0 F¯ −2 dF . The process Bn,t (t + x) is expanded as Z t+x Z t+x b n } − exp{− − Pt (t + x)Bn,t (t + x) = n1/2 [exp{− dΛ dΛ}] t t Z t+x b n }] + n1/2 [exp{log Pbn,t (t + x)} − exp{− dΛ t Z t+x Z t+x b n − Λ) exp{− d(Λ dΛ∗n } {1 + o(1)} = −n1/2 t

+ n1/2 exp{−

Z

t

t

t+x

dΛ∗∗ n } {log

b¯ (t + x) Z t+x F n b n }, − dΛ b¯ (t) t F n

b n , so it converges uniformly to Λ, and where Λ∗n is between Λ and Λ R t+x ∗∗ b b¯ (t) and R t+x dΛ b n . From − t Λn is between log F¯ n (t + x)} − log F n t b 1/2 b n (t)| Lemma 1.5.1 in Pons (2008), the variable n sup[a,b] |−log{F¯ n (t)}− Λ converges in probability to zero as n tends to infinity then the last term of the expansion of Bn,t (t + x) converges to zero in probability, uniformly on [a, b]. The first term converges weakly to −Pt (t + x) BΛR(t, t + x) where the t+x b n − Λ).  process BΛ (t, t + x) is the limiting distribution of n1/2 t d(Λ By Equation (6.10), the covariance of n1/2 (Pbn,t − Pt )(t + x) and n (Pbn,t − Pt )(t + y) is denoted 1/2

2 (t + x ∧ y) CP (t + x, t + y; t) = Pt (t + x)Pt (t + y) lim EBn,t n

= Pt (t + x)Pt (t + y)vB (t + x ∧ y; t). The weak convergence of n1/2 (F¯n − F¯ ) on the interval [0, Tn:n ] allows to extend theR previous proposition to a weak convergence on [T1:n , Tn:n ]. For τ t < τF , if 0 F F¯ −1 dΛ < ∞ Z (t+x)∧Tn:n Pt − Pbn,t 1 − Fbn (s− ) b (t + x) = d(Λn − Λ)(s). (6.11) Pt 1 − F (s) t∨T1:n

Therefore, the process defined for t and t + x in [T1:n , Tn:n ] by n1/2 {(Pt − Pbn,t )Pt−1 }(t + x) converges weakly on the support of F to a centered Gaussian process BP , with independent increments and variance vF¯ (t + x) − vF¯ (t). By the product definition of the estimator of the probability Pt (t+x), it satisfies Z (t+x)∧Tn:n b b n (s). Pn,t (t + x) = Pbn,t (t + s− ) dΛ t∨T1:n

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Nonparametric estimation of intensities for stochastic processes

129

The estimator is extended without modification to samples of right-censored variables, with an independent censorship by a sequence of independent ¯ Only the and identically distributed variables with survival function G. definitions of Nn and Yn and the expression of the variances are modified ¯ −1 in the integral defining vΛ , as in the variance by a multiplicative term G of the classical product-limit estimator. For p ≥ 2, by Equation (6.11), their exists a constant cp such that ( ) Z b¯ (s− ) 2 dN (s) p/2  t+x F n n p p b E{Pt (t + x) − Pn,t (t + x)} ≤ cp Pt (t + x) E Yn2 (s) F¯ (s) t

therefore kPt (t + x) − Pbn,t (t + x)kp = O(n−1/2 ).

The probability density of excess duration is the derivative of the probability Pt (t + x), pt (t + x) = −f (t + x)/F¯ (t). A kernel estimator of pt (t + x) is now defined, for t < t + x − hn as Z 1 pbn,h (t + x; t) = Kh (t + x − s) dPbn,t (s) −1 1

=

=

Z

−1 n X i=1

=

b n (s) Kh (t + x − s)Pbn,t (s− ) dΛ

Kh (Ti − t − x)Pbn,t (Ti− )Jn (Ti )Yn−1 (Ti )δi 1{t<Ti }

b¯ (T − ) J F n n (Ti ). Kh (Ti − t − x)δi 1{t<Ti } Pt (Ti ) ¯ i Y F (T ) n i i=1

n X

As in Section 2.8, the estimator pbn,h is uniformly consistent. For a density 2 p in class C2 , its bias deduced from (6.11) is bp,n,h (t + x; t) = h2 pt (t + x) + o(h2 ). Its variance has the bound Z Z b¯ −  sF n b n − Λ) 2 ds vp,n,h (t + x; t) ≤ 2[E Kh2 (t + x − s)pt (s)2 d( Λ F¯ 0 2 Z b¯ (s− ) F + E Kh2 (t + x − s) n¯ Y −1 (s) dΛ(s)] = O((nh)−1 ). F (t) n

It follows that for every t and x, (nh)1/2 (b pn,h − pn,h )(t + x; t) converges weakly to a Gaussian variable with mean zero and finite variance vp (t + x; t) = limn nhvp,n,h (t + x; t). The covariance of pbn,h (t + x; t) and pbn,h (t + y; t) is written Z Z Cn,h,p (t+x, t+y; t) = Kh (t+x−u)Kh (t+y−v)Cov{Pbn,t (du) Pbn,t (dy)}.

January 31, 2011

130

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

For x < y, the covariance of Pbn,t (x) and Pbn,t (y) is n−1 Pt (t + x)Pt (t + y)vB (t+ x; t)+ o(n−1 ), hence for |x− y| > 2h, the limit of nCn,h,p converges to Z Z Kh (t + x − u)Kh (t + y − v)pt (t + u)pt (t + v)vB (t + u; t) (1)

+ Pt (t + u)Pt (t + v)vB (t + u; t)} du dv

(1)

= pt (t + y){pt (t + x)vB (t + x; t)}(1) (1)

+ pt (t + y){Pt (t + x)vB (t + x; t)}(1) , if |x − y| ≤ 2h, Cn,h,p = 0((nh)−1 ). Then, the process (nh)1/2 (b pn,h − p) converges weakly to a Gaussian variable with mean zero, covariances zero and variance function vp . 6.6

Estimators for processes on increasing intervals

Let (N (t))t≤T be the counting processes associated to a sequence of random time variables (Ti )i≥1 X NT (t) = 1{Ti ≤t∧T } , t ≥ 0, i≥1

where T is deterministic or a random R stopping time. Its predictable comP eT (t) = N (T ) t Yi (s)µ(s, Zi (s)) ds as in the previous pensator is written N i=1 0 PN (T ) sections. In a model without covariates, YT (t) = i=1 Yi (t) and the baseline intensity λ of the intensity µT (t) = λ(t)YT (t) is estimated for t in [h, T − h]Rby smoothing the estimator of the cumulative hazard function Λ, eT (s), with an indication JT (t) = 1{Y (t)>0} b T (t) = t∧T JT (s)Y −1 (s) dN Λ T T 0 Z 1 bT,hT (t) = λ YT−1 (s)JT (s)Kh (t − s) dNT (s). (6.12) −1

R bT,h is λT,h (t) = 1 E{Y −1 (s)}Kh (t−s) dΛ(s). Conditions The mean of λ T T T −1 for its consistency are an ergodic property for the process (NT , YT ) and Conditions 6.1 written for YsT with limits as T tends to infinity, then its bias deh velops as bλ,T,hT (t) = s!T msK λ(s) (t) + o(hsT ). The variance of the estimator bT,h (t)} = (T hT )−1 κ2 g −1 (t)λ(t) + o((T hT )−1 ) and the Lp -norms is V ar{λ T bT,h (t) − λT,h (t)kp = 0((T hT )−1/p ) of the estimator satisfy supt∈IT ,h,τ kλ T T −1/p b and sup kλT,h (t) − λ(t)kp = 0((T hT ) + hs ). Under a mixt∈IT ,hT ,τ

T

ing condition for the point process (NT , YT ) which ensures the weak conbT,h (t) − λ) vergence of the process T 1/2 (YT − g), the process (T hT )1/2 (λ T

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Nonparametric estimation of intensities for stochastic processes

131

converges weakly to the Gaussian process limit of Theorem 6.1. Now the optimal bandwidth are O(T −1/(2s+1) ) and the minimal mean squared errors are O(T −2s/(2s+1) ). In models with covariates, the process YT is modified by a regression PN (T ) (0) function at the jump times as ST (t; β) = i=1 rZi (t; β)1{Ti ≥t} with a PN (T ) (0) parametric regression function or ST (t; r) = i=1 rZi (t; )1{Ti ≥t} with a nonparametric regression function. The predictable compensator of NT R eT (t) = t∧T S (0) (s; r)λ(s) ds. The function λ is estimated by becomes N T 0 smoothing the estimator of the cumulative hazard function Z 1 (0) bT,h (t; β) = λ {ST (s; β)}−1 Kh (t − s) dNT (s) −1

and the regression function is estimated using the parameter estimator Q bT,h (Ti ; β)}δi or by βbT,hT = arg maxβ Ti ≤T {rZi (t; β)λ T rbT,hT (z) = arg max rz

N (T ) Z T X 0

i=1

Kh2 (z − Zi (s)){log rz (s)

bT,h (s; rz )}Kh (z − Zi (s)) dNi (s) + log λ T 2

in the nonparametric case. Assuming that the process Z((Ti ))i≥1 is bounded, ergodic and has fi(k) (k) nite moments up to order 4, the processes T −1 ST (s; β) and T −1 ST (s; r) converge in probabilty and uniformly on [0, T ] and in a neighbourhood of (k) β0 or r0 to finite limits s(k) and T 1/2 (T −1 ST − s(k) ) satisfies a theorem central limit, for k = 0, 1, 2. All results for the kernel estimators of Section 6.3 are still valid under these conditions replacing n by T . The empirical estimator of the probability of excess duration and a kernel estimator for its density are defined by Y PbT,t (t + x) = {1 − JT (s)YT−1 (s) dNT (s)}, t≤s≤T ∧(t+x)

pbT,hT (t + x; t) =

Z

1

−1

Kh (t + x − s) dPbT,t (s).

The process BT,t (t + x) = T 1/2 {Pt (t + x) − PbT,t (t + x)}Pt−1 (t + x) satisfies BT,t (t + x) =

Z

t

t+x

−

b¯ F T b T − Λ). d(Λ F¯

January 31, 2011

132

6.7

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Models with varying intensity or regression coefficients

More complex models are required to describe the distribution of event times when the conditions may change in time or according to the value of a variable. Pons (1999, 2002) presented results for two extensions of the classical exponential regression model for an intensity involving a nonparametric baseline hazard function and a regression on a p-dimensional process Z • a model for the duration X = T 0 − S of a phenomenon starting at S and ending at T 0 , with a non-stationary baseline hazard depending non-parametrically on the time S at which the observed phenomenon starts λX|S,Z (x | S, Z) = λX|S (x; S) eβ

T

Z(S+x)

,

(6.13)

• a model where the regression coefficients are smooth functions of an observed variable X λ(t | X, Z) = λ(t)eβ(X)

T

Z(t)

.

(6.14)

b n of β and of the The asymptotic properties of the estimators βbn and Λ cumulative baseline hazard function follow the classical lines but the kernel smoothing of the likelihood requires modifications. In model (6.13), the time T 0 may be right-censored at a random time C independent of (S, T 0 ) conditionally on Z and non informative for the parameters β and λX,S , and that S is uncensored. We observe a sample (Si , Ti , δi , Zi )1≤i≤n drawn from the distribution of (S, T, δ, Z), where T = T 0 ∧ C and δ = 1{T 0 ≤C} is the censoring indicator. The data are observed on a finite time interval [0, τ ] strictly included in the support of the distributions of the variables S, T 0 and C, and (S, T 0 ) belongs to the triangle Iτ = {(s, x) ∈ [0, τ ] × [0, τ ]; s + x ≤ τ }. For Si in a neighborhood of s, the baseline hazard λX|S (·; Si ) is approximated by λX|S (·; s), which yields a local log-likelihood at s ∈ [hn , τ − hn ], defined as ln (s) =

X i

−

Khn (s − Si )δi {log λX|S (Xi ; Si ) + β T Zi (Ti )} Z

0

τ

Yi (y) exp{β T Zi (Si + y)}λX|S (y; Si ) dy.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

133

Nonparametric estimation of intensities for stochastic processes

ei = Xi ∧ (Ci − Si ) Let X

In,τ = {(s, x); s ∈ [hn , τ − hn ], x ∈ [0, τ − s]} ,

Yi (x) = 1{T 0 ∧Ci ≥Si +x} = 1{Xei ≥x} , i X (0) −1 Sn (x; s, β) = n Khn (s − Sj )Yj (x) exp{β T Zj (Sj + x)}. j

Rx The estimator of Λ0,X|S (x; s) = 0 λ0,X|S (y; s) dy is defined for (s, x) ∈ In,τ ˆ n,X|S (x; s) = Λ ˆ n,X|S (x; s, βˆn ) with by Λ ˆ n,X|S (x; s, β) = Λ

X Khn (s − Si )1{Si ≤Ci ,Xi ≤x∧(Ci −Si )} (0)

nSn (Xi ; s, β)

i

.

The estimator βbn of the regression coefficient maximizes the following partial likelihood i X h ln (β) = δi β T Zi (Ti0 ) − log{nSn(0) (Xi ; Si , β)} εn (Si ) i

where εn (s) = 1[hn ,τ −hn ] (s). The bandwidth h is supposed to converge to zero, with nh2 tends to infinity and h = o(n−1/4 ) as n tends to infinity, the other conditions are precised by Pons (2002). The variable n1/2 (βbn − β0 ) converges weakly to a Gaussian variable N (0, I −1 (β0 )) where the variance I −1 (β0 ), defined as the inverse of the limit of the second derivative of the partial likelihood ln , is the minimal variance for a regular estimator of β0 . The weak convergence of the estimated cumulative hazard function defined along the current time and the duration elapsed between two events relies on the bivariate empirical processes X b n (s, x) = n−1 H δi 1{Si ≤s} 1{Xi ≤x} , i

¯ (0) (s, x) W n

=n

−1

X i

T

eβ0 Zi (Si +x) 1{Si ≤s} 1{Xei ≥x} ,

bn = n1/2 (W ¯ n(0) − W (0) , H b n − H) 1In,τ , B

bn converges under boundedness and regularity conditions, the process B (0) weakly to a Gaussian limit. With functions λ and s in class C2 , the bias b n,X|S is a O(h2 ), thus the optimal bandwidth minimizing of the estimator Λ the asymptotic mean squared error of Λ is O(n−1/5 ) and it is still written in terms of the squared bias and the variance of the estimator. If the regressor Z is a bounded variable, then there exists a sequence of centered Gaussian

January 31, 2011

134

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes 1/2

bn −Bn kIn,τ = op (hn ). This property improcesses Bn on In,τ such that kB b n,X|S − Λ0,X|S )1{I } plies the weak convergence of the process (nhn )1/2 (Λ n,τ to a centered Gaussian process. b n only involves kernel terms through the regression In model (6.14), Λ b b n have the same non-parametric rate of confunctions but both βn and Λ vergence. In Pons (1999), the estimator βbn,h (x) was defined as the value of β which maximizes X ln,x (β) = δi Khn (x − Xi )[{β(Xi )}T Zi (Ti ) (6.15) i≤n

− log{

X

j≤n

Khn (x − Xj )Yj (Ti )e{β(Xi )}

T

Zj (Ti )

}]

(6.16)

where Yi (t) = 1{Ti ≥t} is the risk indicator for individual i at t. Let P (0) β(Xi )T Zi (t) Sn (t, β) = , an estimator of the integrated baseline i Yi (t)e R b n (t) = t Sn(0)−1 (s, βbn,h ) dNn (s). For every x in IX,h , hazard function is Λ 0 the process n−1 ln,x converges uniformly to Z τ lx (β) = (β − β0 )(x)T s(1) (t, β0 (x), x) 0

− s(0) (t, β0 (x), x) log

s(0) (t, β(x), x) dΛ0 (t) s(0) (t, β0 (x), x)

which is maximum at β0 hence βbn,h (x) = arg max ln,x (x) converges to β0 (x). Let Un,h (·, x) and In,h (·, x) be the first two derivatives of the process ln,x with respect to β at fixed x in IX,h , the estimator of β(x) satisfies Un,h (βbn,h (x), x) = 0 and In,h (x) ≤ 0 converges uniformly to a limit I(x). By a Taylor expansion Un,h (β0 (x), x) = (βbn,h (x) − β0 (x))T {I(β0 , x) + o(1)} and (βbn,h (x) − β0 (x)) = {In,h (β0 , x) + o(1)}}−1 Un,h (β0 (x). Under the assumptions that the bandwidth is a O(n−1/5 ) and the function β belongs to the class C2 (IX ), the bias of βbn,h (x) isRapproximated by I −1 (β0 , x)h2 u(x) τ where u(x) has the form u(x) = m22K 0 φ(t, x) dΛ0 (t) and its variance is −1 (nh)−1 κ2 I −1 (β0 , x) + o((nh) ). The asymptotic mean integrated squared R error AM ISEw (h) = E Xn,h kβbn,h (x) − β0 (x)kw(x) dx for βbn,h (x) is therefore minimal for the bandwidth R κ2 Xn,h kI −1 (β0 , x)kw(x) dx hn,opt = n−1/5 R −1 (β , x)kw(x) dx 0 Xn,h u(x)kI and the error AM ISEw (hn,opt ) has the order n−2/5 .

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Nonparametric estimation of intensities for stochastic processes

135

The limiting distributions of the estimators are now expressed in the (0) (0) following proposition. Let Gn = (n−1 h)1/2 {Sn (βbn,h ) − Sn (β0 )}.

Proposition 6.6. For every x in IX,n,h , the variable (nhn )1/2 (βbn,h −β0 )(x) converges weakly to a Gaussian variable N (0, γ2 (K)I0−1 (x)). b n − Λ0 ) converges weakly to the Gaussian proTheR processR (nhn )1/2 (Λ · (0) cess − 0 G(t){ s (t, y) dy}−2 dΛ0 (t), where the process G is the limiting distribution of Gn . The convergence (nhn )1/2 for the estimator of Λ comes from the variR t (0) rate (0)−2 b n (t) − Λ0 (t) developed by a ance E 0 Sn (s, β0 )Sn (s, βbn,h ) dΛ0 (s) of Λ first order Taylor expansion. 6.8

Progressive censoring of a random time sequence

Let (Ti )i=1,...,n be a sequence of independent random time variables and (Tj )j=1,...,m be an independent sequence of independent random censoring times such that a random number Rj of variables Ti are censored at Tj and Pm j=1 Rj = n. Then the censored variables Xi,j = Ti ∧ Cj are no longer independent, only m sets of variables are independent. Let Nn,m (t) =

Rj m X X

1{Ti ≤t∧Cj } , Yn,m (t) =

j=1 i=1

Rj m X X

1{Xi,j ≥t} .

j=1 i=1

Let F be the common distribution function of the variables Ti and Gj be the distribution function of Cj , the intensity of the point process Nn,m is still written λn,m (t) = λYn,m with λ = F¯ −1 f . Conditionally on the censoring number R = (R1 , . . . , Rm ), the expectations of Nn,m (t) and Yn,m (t) are Z t m X ¯ j dF, E{Nn,m (t) | R} = Rj G E{Yn,m (t) | R} =

j=1 m X

0

¯ j (t)F¯ (t). Rj G

j=1

Let µR = lim Rj for j = 1, . . . , m, and Jn,m (t) = 1{Yn,m (t)>0} , the estimator of the cumulative hazard function Λ and its derivative λ are Z t −1 b n,m (t) = Λ Yn,m Jn,m dNn,m , 0 Z bn,m (t) = Kh (t − s) dΛ b n,m (s). λ

January 31, 2011

136

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Assuming that there exists an uniform limit for the mean survival funcP −1 ¯ = limm→∞ m−1 m G ¯ tion G Yn,m converges uniformly j=1 j , the process n −1 ¯ ¯ to its expectation µ (t) = µ G(t) F (t), and n N Y R n,m converges uniformly Rt bn,m are then unbiased and uni¯ dF . The estimators Λ b n,m and λ to µR 0 G R −1 b n,m −Λ)(t) is t nYn,m formly consistent. The variance of n1/2 (Λ Jn,m dΛ and 0 R t −1 bn,m (t) is it converges to vΛ (t) = 0 µY dΛ. The variance of the estimator λ (1) (nh)−1 κ2 vΛ (t) + o((nh)−1 ) and both estimator processes converge weakly to Gausian processes with zero mean and these variances. The process bn,m − λ) has asympb n,m − Λ) has independent increments and n1/2 (λ n1/2 (Λ totic covariances zero. All results for multiplicative regression models with independent censoring times apply to this progressive random censoring scheme. With nonrandom numbers Rj , the necessary condition for the Pm ¯j . convergence of the processes is the uniform convergence of n−1 j=1 Rj G 6.9

Exercises

(1) Define a kernel estimator for a Poisson process N with a functional intensity λ(t) from the observation of a sample-path on an interval [0, T ], such that T tends to infinity and prove that the process bT,h − λ) converges weakly. (T h)−1/2 (λ (2) Calculate the bias of the estimator of β which maximizes the process ln,x defined by (6.15). (3) Consider a point process with a conditional multiplicative intensity λ(t) r(β T Z(t)) Y (t), where λ and r are nonparametric real functions and β is a vector of unkown parameters. Define kernel estimators for the functions λ and r and an estimator for β by minimization of a partial likelihood. Determine the order of their risks and prove their weak convergence.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Chapter 7

Estimation in semi-parametric regression models

7.1

Introduction

In the single-index regression model, the scalar response variable Y is expressed as a nonparametric transform for a linear combination of the components of a vector X of d regression variables Yi = g(θT Xi ) + σ(η T Xi )εi ,

(7.1)

where X is a vector of regression variables in a bounded subset of Rd and ε is an error variable such that E(ε|X) = 0 and V ar(ε | X = x) = 1, then V ar(Y | X = x) = σ 2 (η T x). The model includes a similar parametrization for the mean and the variance functions. The parameters are vectors η and θ belonging to an open and bounded subset Θ of Rd and g, an unknown function of C2 (R). Several estimators for the semi-parametric regression function m(x) = g(θT x) have been defined from approximations and they are calculated by iterations, without model for the variance. Here the estimators are defined in a two-step procedure from the weighted estimator of the regression function g defined by (3.19). The true parameter value is a vector in R2d (η0T , θ0T )T = arg min V (η, θ) η,θ∈Θ

(7.2)

where V (η, θ) = E[σ −1 (η T Xi ){Y − g(θT X)}2 ] is the mean weighted squared error at a fixed parameters value η and θ. Several empirical criteria can be defined for estimating V , estimators of θ0 satisfying the same property (7.2) are deduced. Let (Xi , Yi )i=1,...,n be a sample of observations in Rd+1 , in the model with known variance function. At fixed θ, let gbn,h (z) be the nonparametric 137

January 31, 2011

138

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

regression estimator defined with regressors values θT Xi in a neighborhood of z, then the parameter θ is estimated by minimizing a mean squared error of estimation, which is a goodness-of-fit criterium for the model. For observations such that θT Xi lies in a neighborhood of z = θT x, the regression function is estimated at fixed η and θ, by Pn −1 bn,h (η T Xi )Yi Kh (z − θT Xi ) i=1 σ gn,h (z; η, θ) = Pn b , −1 bn,h (η T Xi )Kh (z − θT Xi ) i=1 σ

−1 and σ bn,h (η T Xi ) is the estimator (3.19) calculated at fixed η. The global goodness-of-fit error and the estimator of θ minimizing this error are

Vbn,h (η, θ) = n−1

n X i=1

−1 σ bn,h (η T Xi ){Yi − gbn,h (θT Xi ; θ)}2 ,

T T (b ηn,h , θbn,h )T = arg min Vbn,h (η, θ), η,θ∈Θ

(7.3) (7.4)

T m b n,h (x) = gbn,h (θbn,h x; θbn,h ).

We first assume that the variance function is known and denoted σ 2 (x), the error and the estimator of g are then only normalized by σ −1 (x). The global error (7.3) and the estimator (7.4) have been modified by considering the mean of local empirical squared errors. In a neighborhood of z, a local empirical squared error is defined by smoothing (7.3) Vbn,h (z; θ) = n−1

n X i=1

σ −1 (Xi ){Yi − b gn,h (θT Xi ; θ)}2 Kh (z − θT Xi ),

θbn,h,z = arg min Vbn,h (z; θ). θ∈Θ

A global estimator

T θ¯n,h

of θ was defined by an empirical mean of the local bn,i = θbT estimators at the random point Z n,h,Zi Zi . Then an estimator of the regression function m is T m ¯ n,h (x) = gbn,h (θ¯n,h x; θ¯n,h ).

(7.5)

Another estimator is then obtained by minimizing the differential criterion cn,h (θ) = n−2 W

n X

i6=j=1

{σ 2 (Xi ) + σ 2 (Xj )}−1/2 {Yi − Yj − b gn,h (θT Xi ; θ)

cn,h (θ), θen,h = arg min W θ∈Θ

+b gn,h (θT Xj ; θ)}2 Kh (Xi − Xj ),

(7.6)

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

139

Estimation in semi-parametric regression models

with sums on values individuals having close values of the regression variable. Finally, a third estimator of the regression function is T m e n,h (x) = b gn,h (θen,h x; θen,h )

with the estimator (7.6) for the vector parameter θ. The error V (θ0 ) and W (θ0 ) is estimated by plug-in with (7.4) and Vbn,h = Vbn,h (θbn,h ) cn,h = W cn,h (θbn,h ) W

Pn T and V¯n,h = n−1 k=1 Vbn,h (θ¯n,h Xk ; θ¯n,h ) for an empirical mean of local estimators. Various forms of the estimation criteria are weighted mean square errors in order to take into account the unknown variance σ 2 of the variable Y or other weighting functions. Iterative procedures were also defined with alternative estimations of the function g and the parameters. In the next section, the convergence rate of the parameter estimator and the regression function m b n,h are determined for (7.3) and (7.6) with variance 1. In the presence of nonparametric estimator of the function g, the convergence rate of the parameter estimator differs from the parametric rate. The limiting distributions of the parametric and nonparametric estimators are studied cn,h . for the global errors Vbn,h and W More general models are nonparametric regressions with a parametric change of variable. In the nonparametric regression with a change of variables, the linear expression θT X is replaced by a transformation of X using a parametric family of functions defined in Rd , φ = {ϕθ }θ∈Θ subset of the class C2 . The semi-parametric regression model is Y = g ◦ ϕθ (X) + σ(X)ε.

(7.7)

The error is still V (θ, σ) = Eσ −1 (X){Y − m(X)}2 with m = g ◦ ϕθ , at fixed θ and σ, and the parameters can be estimated by minimizing over θ and σ similar empirical estimators of V (θ) as above. 7.2

Convergence of the estimators

Proposition 2.3 and Theorem 2.1 imply the convergence in probability to zero of the empirical error criteria supθ∈Θ |Vbn,h (θ) − V (θ)| and cn,h (θ) − W (θ)|, the local criterium Vbn,h (z; θ) also converges unisupθ∈Θ |W formly on Z × Θ to V (θ; z) = E[{Y − m(X)}2 |θT X = z]. As the minimum

January 31, 2011

17:17

World Scientific Book - 9in x 6in

140

FunctionalEstimation

Functional estimation for density, regression models and processes

of the limits V (θ) and W (θ) is θ0 , all estimators converge to θ0 . The mini(1) mization of Vbn,h provides an estimator θbn solution of Vbn,h (θ) = 0 where (1) Vbn,h (θ) = 2n−1

n X i=1

(1)

{Yi − gbn,h (θT Xi ; θ)} {b gn,h (θT Xi ; θ)} Xi

and for the second derivative, let Zi = θT Xi , at fixed θ n X (2) (2) (1) −1 b Vn,h (θ) = 2n [{Yi − b gn,h (Zi ; θ)} b gn,h (Zi ; θ) − {b gn,h (Zi ; θ)}2 ] Xi⊗2 . i=1

(2) The limit of Vbn,h (θ) is the second derivative of V (θ) = E[{g(θT X) − (2)

g(θ0T X)}2 ] which is minimal at θ0 , therefore −Vn,h (θ) is positive definite in a neighbourhood of θ0 for n large enough since that is true for the limit(1) ing function −V (2) (θ). Expanding Vbn,h (θ), for θ in a neighbourhood of θ0 , (1) (1) (2) implies Vb (θ) = Vb (θ0 ) + (θ − θ0 )T Vb (θ0 ) + OP ((θ − θ0 )2 ). Then the n,h

n,h

n,h

(2) (1) estimators satisfy θbn,h − θ0 = {−Vbn,h (θ0 )}−1 Vbn,h (θ0 ) + OP ((θbn,h − θ0 )2 ) also written (2) (1) θbn,h − θ0 = {−Vbn,h (θ0 )}−1 {Vbn,h (θ0 ) − V (1) (θ0 )} + OP ((θbn,h − θ0 )2 ). (1)

Applying Theorem 2.1, (nh3 )1/2 {b gn,h(θ0 ) − g (1) (θ0 )} converges weakly to a continuous Gaussian process with zero mean and finite variance. The first (1) (1) (1) two moments of Vbn,h are Vn,h (θ) = E Vbn,h (θ), expanded at θ0 as (1)

(1)

Vn,h (θ0 ) = 2E[{m(Xi ) − gn,h (θ0T Xi ; θ0 )} {b gn,h(θ0T Xi ; θ0 )} Xi ] (1)

− 2E[(b gn,h − gn,h )(θ0T Xi ; θ0 )} {b gn,h(θ0T Xi ; θ0 )} Xi ]. (1)

Proposition 10.1 in Appendix A and Appendix C prove that Vn,h (θ0 ) (1) is a O(h2 ) + O((nh2 )−1 ), and the variance of Vb (θT x; θ0 ) equals n,h

(nh3 )−1 ΣV,θ0 + o((nh3 )−1 ), where

0

Z −2 ΣV,θ0 = E[X ⊗2 fX (X){m2g (θ0T X) + σg2 (θ0T X)}w2 (θ0T X)]( K (1)2 ). (1)

(1)

It follows that the variable (nh3 )1/2 {Vbn,h (θ0 ) − Vn,h (θ0 )} converges weakly to a continuous Gaussian variable with mean zero and variance ΣV,θ0 , and (2) (1) (nh3 )1/2 (θbn,h − θ0 ) = 2{Vbn,h (θ0 )}−1 (nh3 )1/2 ){V (1) − Vbn,h }(θ0 ) + op (1). (7.8) (2) Moreover, the second derivative Vbn,h converges uniformly to a bounded (2)

function Vθ

on the parameter space. The result extend to regression

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Estimation in semi-parametric regression models

FunctionalEstimation

141

functions of Cs (IX ), s ≥ 2, and the order nh3 related to the variance of (2)−1 (2)−1 the derivative of gbn,h is unchanged. Let v0 = Vθ0 ΣV,θ0 Vθ0 .

Proposition 7.1. Under Conditions 2.1 and 3.1, and if hn = o(n−1/7 ), the estimators of the parameter θ and the function m in class Cs (IX ) are consistent, (nh3 )1/2 (θbn,h − θ0 ) converges weakly to a Gaussian variable with with mean zero and variance v0 and (nh3 )1/2 (m b n,h − m) converges weakly to a Gaussian process with mean zero and covariance function g (1) (θ0T x)vθ0 g (1) (θ0T x0 ) at (x, x0 ). Proof.

The process m b n,h (x) − m(x) splits as the sum of two terms T T x) − g(θbn,h x), un,h (x) = b gn,h (θbn,h T vn,h (x) = g(θbn,h x) − g(θ0T x).

The convergence rate of vn,h is the same as θbn,h − θ, which is (nh3 )1/2 like (1) (1) (Vbn,h − V (1) )(θ0 ) since the bias of Vbn,h (θ0 ) disappears with hn = o(n−1/7 ). The process (nh)1/2 un,h (x) = (nh)1/2 (b gn,h − g)(θ0T x){1 + op (1)} is a Op (1), then the convergence rate of {m b n,h (x) − m(x)} is (nh3 )1/2 . 

The bandwidth minimizing the sum of the squared bias and the vari(1) (1) ance of Vbn,h (θ0 ) is hV,n = O(n−1/7 ) and the convergence rate of Vbn,h (θ0 ) is n2/7 in that case. Note that with a bandwidth hn = O(n−1/7 ), the limit is (2)−1 (1) a Gaussian variable with the finite mean limn (nh7 )1/2 Vn,h )(θ0 )Vn,h (θ0 ). This rate hn = O(n−1/7 ) is optimal for estimating the first derivative of a function of class C2 and the biases were obtained under this assumption. This is a consequence of the approximation of θbn,h − θ0 in terms of (1) (Vbn,h − V (1) )(θ). The optimal local and global bandwidths for the estimators gbn,h and m b n,h are O(n−1/(2s+3) ) and they are expressed as a ration of their (integrated) bias and variance, the mean squared errors for their estimation are O(n−2s/(2s+3) ).

The estimator minimizing the local error Vbn,h (z; θ) is approximated by linearization of the error for θT Xi in a neighbourhood of z Vbn,h (θ; z) = n−1

n X i=1

Kh (z − θT Xi ){Yi − b gn,h (z) (1)

− (θT Xi − z)b gn,h (z)}2 + O(h2 ),

and the derivatives of the linear approximation are considered with variables Zi = θT Xi in a neighborhood of z. The asymptotic behaviour of its

January 31, 2011

17:17

142

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

derivatives differs from those of the global error due to the kernel smoothing. The mean estimator has a smaller variance than the estimator minimizing the global error Vbn,h .

For independent variables such that Xi 6= Xj and |Xi − Xj | ≤ h, let ∆i,j X = Xi − Xj and ∆i,j ϕ(X) = ϕ(Xi )− ϕ(Xj ), for every function ϕ. Let Z = θT X at fixed θ. With the notations of Proposition 3.1, the differential cn,h defined by (7.6) has the mean criterion W Wn,h (θ) = E[{∆i,j (Y − b gn,h )(θT X)}2 Kh (Xi − Xj )]

= E[{σ 2 (θT Xi ) − σ 2 (θT Xj )} + {g(θ0T Xi ) − g(θ0T Xj ) − g(θT Xi ) + g(θT Xj )}2 + E[(nh)−1 {σg2 (θT Xi ) − σg2 (θT Xj )}

+ h4 {b2g,n,h (θT Xi ) − b2g,n,h (θT Xj )}Kh (Xi − Xj )]{1 + o(1)},

where gbn,h is the regression function estimator. By Lemma 10.1 in Appendix C, it is approximated by O(h2 ) + O(n−1 h). The first derivative of cn,h is W n X c (1) (θ) = −2n−2 W {Yi − Yj − gbn,h (Zi ; θ) + gbn,h (Zj ; θ)} n,h i6=j=1

its mean develops as

(1) {b gn,h (Zi ; θ)Xi

(1)

(1)

−b gn,h (Zj ; θ)Xj } Kh (∆i,j X) (1)

Wn,h (θ) = 2E[{∆i,j (m0 (X) − gn,h (Z))}{∆i,j (Xgn,h (Z))} Kh (∆i,j X)] (1)

− E[{∆i,j (b gn,h − gn,h )(Z)}{∆i,j (Xb gn,h (Z))} Kh (∆i,j X)],

denoted E1 + E2 . The first term is expanded as Z (1) E1 = h2 m2K {m0 (x) − gn,h (θT x)}(1) {xgn,h (θT x)}(1) fX (x) dFX (x),

using a first order approximation for the variations, E2 depends on the co(2) (1) variance of b gn,h (Z) and b gn,h (Z), with a factor h2 , thus the second term is a (1)

0(n−1 h−2 ). The function Wn,h (θ) is then an uniform O(h2 ) + O((nh2 )−1 ) and it tends to zero uniformly on the bounded parameter space. As in Appendix C, the main term of its variance ΣW,n,h (θ0 ) depends on (2) (2) (1) (1) E{b gn,h (Z) − gn,h (Z)}2 {b gn,h (Z) − gn,h (Z)}2 = O(nh4 )−1 , with a factor n−2 h3 , thus it is a 0(n−3 h−1 ). The second order derivative is an empirical mean n X (1) c (2) (θ) = 2n−2 W [{∆i,j {Xb gn,h(θT X)}}⊗2 − {Yi − Yj − ∆i,j gbn,h (θT X)} n,h i6=j=1

(2)

× ∆i,j {X ⊗2 gbn,h (θT X)}]Kh(∆i,j X),

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

143

Estimation in semi-parametric regression models

it is approximated by its expectation (2)

(1)

Wn,h = 2E[∆⊗2 gn,h(θT X)} − {∆i,j {m0 (X) − gbn,h (θT X)} i,j {Xb (2)

× ∆i,j {X ⊗2 b gn,h (θT X)}] Kh (∆i,j X)

(1) T = 2E Kh (∆i,j X) [{∆⊗2 (θ X)} i,j {Xg

(2)

− ∆i,j {m0 (X) − gn,h (θT X)}∆i,j {X ⊗2 gn,h (θT X)} (2)

+ ∆⊗2 gn,h − gn,h )(θT X)}∆i,j {X ⊗2 gbn,h (θT X)}]. i,j {(b (2)

The sequence (Wn,hn )n is therefore an uniform O(h2 + (nh3 )−1 ). Assuming (1)

that nh4n tends to infinity with n, Wn,hn = O(h2 + (nh2 )−1 ) = O(h2 ) and c (1) is that its a necessary condition for the weak convergence of (n3 h)1/2 W n,h

normalized mean converges, i.e. h = O(n−3/5 ). Under this condition, nh5 tends to zero and the convergence rate of (2) Wn,hn is h2 . Arguing as for the estimator of θbn,h related to Vbn,h , θen,h cn,h is such that (n3 h5 )1/2 (θen,h − θn,h ) is approximated by the minimizing W c (2) (θ0 )}−1 n3/2 h1/2 (W c (1) − W (1) )(θ0 ). It converges weakly variable h2 {W n,h

n,h

n,h

to a Gaussian process with variance vθ0 = W (2)−1 (θ0 )ΣW,θ0 W (2)−1 (θ0 ). Following the arguments for the proof of the previous proposition, we obtain the following convergences.

Proposition 7.2. Under Conditions 2.1 and 3.1, and if hn = O(n−3/5 ), the estimators of the parameter θ and of the function m in class C2 are consistent, (n3 h5 )1/2 (θen,h − θ0 ) and (nh)1/2 (m e n,h − m) converge weakly to Gaussian processes with finite variances.

With hn = O(n−3/5 ), the limit of (nh)1/2 (m e n,h −m) is centered, if moreover hn = o(n−3/5 ), both limits are centered. The weighted estimator of the regression function has the same convergence rate as in the model with a constant variance as proved in Section 3.6, the convergence rates of the estimators of θ and g of Propositions 7.1 and 7.2 in the single-index model are therefore unchanged. 7.3

Nonparametric regression with a change of variables

In the semi-parametric regression model (7.7), with m = g ◦ ϕθ , the estimators are built following the same lines as in the single index regression model. For observations such that ϕθ (Xi ) lies in a neighbourhood of z = ϕθ (x),

January 31, 2011

17:17

144

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

the regression function is estimated by Pn −1 bn,h (Xi )Yi Kh (z − ϕθ (Xi )) i=1 σ gn,h (z; θ) = Pn b −1 bn,h (Xi )Kh (z − ϕθ (Xi )) i=1 σ

with a parametric or semi-parametric estimator for the variance function. The global goodness-of-fit error and the estimator of θ minimizing this error are n X −1 Vbn,h (θ) = n−1 σ bn,h (Xi ){Yi − b gn,h ◦ ϕθ (Xi )}2 , i=1

θbn = arg min Vn,h (θ), θ∈Θ

and m b n,h = b gn,h ◦ ϕθbn,h . Assume that the variance is constant and let Zi = ϕθ (Xi ) at fixed θ. The derivatives of Vbn,h are (1) Vbn,h (θ) = −2n−1 (2) Vbn,h (θ) = −2n−1

− n−1

where (2)

(2) −Vbn,h (θ)

n X i=1

n X i=1

n X i=1

(1)

(1)

{Yi − gbn,h (Zi )} b gn,h (Zi ) ϕθ (Xi ) (2)

(1)2

(1)

[{Yi − gbn,h (Zi )}b gn,h (Zi ) − b gn,h (Zi )]{ϕθ (Xi )}⊗2 (1)

(2)

{Yi − gbn,h (Zi )} gbn,h (Zi ) ϕθ (Xi )

is a positive definite matrix converging to a finite limit (1)

ΣV (θ) uniformly on the parameter space. The mean of Vbn,h (θ) and its variance have the same orders as for the derivative of (7.3) in model (7.1) (1)

(1)

(1)

Vn,h (θ) = 2E[{Yi − gbn,h ◦ ϕθ (Xi )} b gn,h ◦ ϕθ (Xi ) ϕθ (Xi )] (1)

(1)

(1)

= 2E{[{(m − gn,h )gn,h − Cov(b gn,h , gbn,h )} ◦ ϕθ ϕθ ](Xi )}

where the first terms in the right-hand side is O(h2 ) and the last term is (1) a O((nh2 )−1 ). The variance of Vbn,h (θ) is a O((nh3 )−1 ). An expansion of Vn,h in a neighborhood of θ0 implies (2)

(1)

(nh3 )1/2 (θbn,h − θ0 ) = {−Vbn,h (θ0 )}−1 (nh3 )1/2 {Vbn,h (θ0 ) − V (1) (θ0 )} + op (1). (1) The variance of Vbn,h (θ0 ) is asymptotically equivalent to (nh3 )−1 ΣV,θ0 + 3 −1 o((nh ) ), with a modified notation Z (1)⊗2 −2 ΣV,θ0 = 4E[ϕθ0 (X)fX (X){g 2 + σg2 } ◦ ϕθ0 (X)w2 (ϕθ0 (X))]( K (1)2 ).

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

145

Estimation in semi-parametric regression models

Proposition 7.3. Under Conditions 2.1 and 3.1, and with a bandwidth hn = o(n−1/7 ) for the estimation of a regression function m in class C2 , the estimators of the parameter θ and the function m are consistent, (nh3 )1/2 (θbn,h −θ0 ) converges weakly to a centered Gaussian with variance vθ and (nh3 )1/2 (m b n,h − mθ0 ) converges weakly to a centered Gaussian process (1) (1) with covariance g (1) ◦ ϕθ0 (x)vθ0 g (1) ◦ ϕθ0 (x0 )ϕθ0 (x) ⊗ ϕθ0 (x0 ) at (x, x0 ). With the optimal bandwidth hn = O(n−1/7 ), the convergence rates of the estimators are n2/7 and the limiting distributions of the estimators have a non zero mean, as in the previous section.

cn,h (7.6) adapted to model (7.7) with a conThe differential criterion W stant variance is written n X cn,h (θ) = n−2 W {Yi − Yj − b gn,h ◦ ϕθ (Xi ) + gbn,h ◦ ϕθ (Xj )}2 Kh (Xi − Xj ), i,j=1

cn,h (θ) and gen,h at θen,h . Let it defines the estimators θen,h = arg minθ∈Θ W cn,h with respect to θ are Zi = ϕθ (Xi ) at fixed θ, the derivatives of W n X c (1) (θ) = −2n−2 W {Yi − Yj − gbn,h (Zi ) + gbn,h (Zj )} n,h i,j=1

c (2) (θ) = W n,h

(1) (1) (1) (1) {b gn,h (Zi )ϕθ (Xi ) − b gn,h (Zj )ϕθ (Xj )}Kh (Xi − Xj ), n X (1) (1) (1) (1) 2n−2 [{b gn,h (Zi )ϕθ (Xi ) − gbn,h (Zj )ϕθ (Xj )}2 i,j=1 (2)

(1)2

− {Yi − Yj − b gn,h (Zi ) + b gn,h (Zj )}{b gn,h (Zi )ϕθ

−

(2) (1)2 gbn,h (Zj )ϕθ (Xj )}2

(1) (2) {b gn,h (Zi )ϕθ (Xi )

(Xi )

+ {Yi − Yj − b gn,h (Zi ) + gbn,h (Zj )}

(1) (2) −b gn,h (Zj )ϕθ (Xj )}]Kh (Xi − Xj ). (1) The mean of the first derivative is Wn,h (θ) = −E E[{g(Zi ) − g(Zj ) − (1) (1) (1) (1) gbn,h
(Zi )+b gn,h (Zj )}{b gn,h (Zi )ϕθ (Xi )−b gn,h (Zj )ϕθ (Xj )} | Xi , Xj ]Kh (Xi −

Xj ) , its order and the order of its variance are the same as in the single index model. The second derivative has the mean (2) (1) (1) (1) (1) Wn,h (θ) = 2E E[{b gn,h (Zi )ϕθ (Xi ) − b gn,h (Zj )ϕθ (Xj )}2 | Xi , Xj ] (2)

(1)2

− E[{g(Zi ) − g(Zj ) − b gn,h (Zi ) + b gn,h (Zj )}{b gn,h (Zi )ϕθ −

(2) (1)2 gbn,h (Zj )ϕθ (Xj )}2

+ E[{g(Zi ) − g(Zj ) − b gn,h (Zi ) +

| Xi , Xj ]

(Xi )

(1) (2) gn,h (Zj )}{b b gn,h (Zi )ϕθ (Xi )


(1) (2) − gbn,h (Zj )ϕθ (Xj )} | Xi , Xj ]Kh (Xi − Xj ) .

January 31, 2011

146

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

The results of Proposition 7.2 are similar with these notations for the asymptotic variances. In a regression model for processes (Xt , Yt )t≥0 , empirical error processes are defined as in Section 3.6, with linear combinations of the components of the d-dimensional regression variable. They are indexed by T , the length of the time interval and the convergence rates are similar replacing n by T . Varying bandwidth estimators of the parameter θ and the function m are defined by a modification of the estimators of the functions g and σ 2 as in Section 4.3. Assuming that the functions g and σ 2 have the same order of derivability, both functions are estimated with bandwidths of the same order, and the modified estimators are introduced in the definition of Vbn,h , where the bandwidths hn (Xi ) and hn (Xj ) differ. Under Condition 4.1, Proposition 7.1 extends to variable bandwidth estimators, with the convergence rate (nkhn k3 )1/2 for the parameter estimator. The differential empirical error uses another weight Kh (Xi − Xj ), for every i 6= j. Its bandwidth was supposed equal to the bandwidth used for the estimation of g in Section 2. Due to the symmetry of the kernel with respect to the variables Xi and Xj , a variable bandwidth in the expression Kh (Xi − Xj ) can be chosen as the mean of the bandwidths at Xi and Xj , with |Xi − Xj | < 2khk. As h tends to zero, h(Xi ) = h(Xj )+(Xi −Xj ){h(1) (Xj )+o(khk)} and the mean bandwidth for Kh (Xi − Xj ) is equivalent to h(Xi ) or h(Xj ). The expansions of Chapter 4 allow to extend Proposition 7.2 for such bandwidths. 7.4

Exercises

(1) Write the mean, the bias and the variance of the local mean squared error Vbn,h (θ; z) and define a sequence of optimal bandwidth functions for this criterium. (2) Write the derivatives of Vbn,h (θ; z) and an approximation for the estimator θbn,h (z) minimizing Vbn,h (θ; z). Determine the orders of its bias and its variance.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Chapter 8

Diffusion processes

8.1

Introduction

Let α and β be two functions of class C2 on a functional metric space (X, k · k), and let B be the standard Brownian motion on R. Their norms on X, kαk1 , kβk2 , Ekα(X(t))k1 and Ekβ(X(t))k2 are supposed to be finite. A diffusion process (Xt )t∈[0,T ] is defined as a stochastic differential equation by dXt = α(Xt )dt + β(Xt )dBt , t ∈ [0, T ]

(8.1)

and its initial value X0 such that E|X0 | < ∞. Equation (8.1) with locally Lipschitz drift and diffusion functions has a unique solution Xt = X0 + Rt Rt α(Xs )ds + 0 β(Xs )dBs , it is a continuous Gaussian process wih mean 0 Rt E(Xt − X0 ) = 0 Eα(Xs ) ds and variance Z t Z t V ar(Xt − X0 ) = V ar{ α(Xs ) ds} + E β 2 (Xs ) ds 0 0 Z t Z t 2 2 ≤ E{α (Xs ) + β (Xs )} ds − { Eα(Xs ) ds}2 . 0

0

The existence and unicity of the process X is proved by construction of a sequence of processes satisfying Equation (8.1) and starting from X0 and satisfying Z t Z t Xn,t −Xn−1,t = {α(Xn,s )−α(Xn−1,s )}ds+ {β(Xn,s )−β(Xn−1,s )}dBt , 0

0

hence Xn,t is the finite sum from X0 to Xn,t −Xn−1,t , where the convergence of the sum is a consequence of the Lischitz property of α and β. By a discretization of the time interval [0, T ] in n sub-intervals of length tending to zero as n tends to infinity, Equation (8.1) is approximated by Yi ≡ Yti+1 = Xti+1 − Xti = (ti+1 − ti )α(Xti ) + β(Xti ){Bti+1 − Bti }, (8.2) 147

January 31, 2011

17:17

World Scientific Book - 9in x 6in

148

FunctionalEstimation

Functional estimation for density, regression models and processes

considering the functions α and β as piecewise constant on the intervals of the partition generated by (ti )i=1,...,n . Let εi = Bti+1 − Bti , it is a random variable with mean zero and variance (ti+1 − ti ) conditionally on the σ-algebra Fti generated by the sample-paths of X up to ti , then Eα(Xti )εi = 0 and V ar(Yi |Xti ) = (ti+1 − ti )β 2 (Xti ). The process Xt solution of (8.1) is a continuous Gaussian process with independent increments. Its increments are approximated by the nonparametric regression model (8.2) with an independent normal error by considering the functions α and β as stepwise constant functions on the partition (ti )1≤i≤n . In the nonparametric regression model (8.2), EYi = (ti+1 − ti )Eα(Xti ) and V arYi = (ti+1 − ti )2 V arα(Xti ) + (ti+1 − ti )Eβ 2 (Xti ). Let t in Ii =]ti , ti+1 ], the approximation errorR of the process Xt − Xti t by the discretized sample-path (8.2) is et;ti = ti {α(Xs ) − α(Xti )}ds + Rt {β(Xs ) − β(Xti )}dBs , it satisfies ti E|et,ti | ≤ (t − ti )2 sup |α(1) (Xs )|,

V ar et,ti = V ar ≤

Z

Z

s∈Ii

t

ti

t

ti

{α(Xs ) − α(Xti )}ds +

V ar{α(Xs ) − α(Xti )}ds +

Z

t

ti Z t ti

E{β(Xs ) − β(Xti )}2 ds E{β(Xs ) − β(Xti )}2 ds

and it bounded by (t − ti )kα(1) k2 kβ (1) k2 E(Xt − Xti )2 = O((t − ti )2 ), with Z t E(Xt − Xti )2 ≤ E{α2 (Xs ) + β 2 (Xs )} ds. ti

The following condition allows to express the moments of increments of the diffusion process as integrals with respect to a mean density. Condition 8.1. There exists a mean density of the variables (Xti )1≤i≤n defined as the limit n X f (x) = lim n−1 fXti (x) = EfXt (x). n→∞

i=1

This condition is satisfied under a mixing property of the process Xt ∞ sup{Pr(B|A) − Pr(B); A ∈ F0t , B ∈ Ft+s , s, t ∈ R+ } ≤ ϕ(s), Z T ϕ(u) du < ∞, (8.3) 0

∞ where the σ-algebras F0t and Ft+s are respectively generated by {Xu , u ∈ [0, t]} and {Xu , u ∈ [t + s, ∞[}. That property is satisfied for the Brownian

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Diffusion processes

FunctionalEstimation

149

motion, its sample paths having independent increments. For the diffusion RT process Xt , it is sufficient that E 0 β 2 (Xs ) ds < ∞. Moments of discontinuous parts of a diffusion process with jumps require another ergodicity condition defining another mean density and it is satisfied under the mixing property (8.3). 8.2

Estimation for continuous diffusions by discretization

The regression model (8.2) with observations at fixed points regularly spaced on a grid (ti )1≤i≤n , of path ∆n = n−1 T , is written Yi = ∆n α(Xti )+ β(Xti )εi , for i = 1, . . . , n. The variables εi have a normal distribution N (0, ∆n ), hence Eε2k+1 = 0 for every integer k, Eε2i = ∆n and i 2(k−1) Eε2k for every k ≥ 1, thus Eε4i = 3∆2n . A nonparai = (2k − 1)∆n Eεi metric estimator of the function α requires a normalization of Yi by the scale ∆−1 n Pn Y K (x − Xti ) i=1 P i h α bn,h (x) = , ∆n ni=1 Kh (x − Xti ) for every x in Xn,h = {y ∈ X ; kx − yk < h}. The approximations and the convergences of Proposition 3.1 are satisfied for the estimator α bn,h . Let −1 αn,h be its mean, it is approximated as αn,h (x) = µα,n,h (x)fX,n,h (x) + −1 O((T h) ), where Z Z −1 µα,n,h (x) = ∆n yKh (x − s) dFXt ,Yt (s, y) µα (x) = EfXt (x)α(x).

The bias of the estimator α bn,h for a function α in class C2 (X ) has a first order expansion bα,n,h (x) = αn,h (x) − α(x) = h2 bα (x) + o(h2 ), m2K −1 (2) bα (x) = f (x){µα (x) − α(x)f (2) (x)} 2

and its variance is −1 vα,n,h (x) = ∆−1 {σα2 (x) + o(1)}, n (nh)

σα2 (x) = κ2 f −1 (x)V ar(Yt | Xt = x) = κ2 f −1 (x)β 2 (x).

For the estimation of the function β defining the variance of X, let Zi = Yi − ∆n α bn,h (Xti ) = ∆n (α − α bn,h )(Xti ) + β(Xti )εi ,

January 31, 2011

17:17

150

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

its mean is E{Zi |Xti = x} = ∆n E(α − α bn,h )(Xti ) = ∆n (α − αn,h )(Xti ) its order is ∆n O(h2 ) and its variance satisfies −1 ∆−1 bn,h )(Xti ) + β(Xti )εi }2 n V ar{Zi |Xti = x} = ∆n E{∆n (αn,h − α

= ∆n V arb αn,h (Xti ) + β 2 (x) = (nh)−1 κ2 f −1 (x) + β 2 (x).

(8.4)

A consistent estimator of the function β 2 (x) is therefore Pn Z 2 K (x − Xti ) 2 i=1 Pn i h βbn,h (x) = . ∆n i=1 Kh (x − Xti )

The approximations and the convergences of Proposition 3.1 are also satisfied for the estimator βbn . Let βn,h be its mean and let 2 µβ,n,h (x) = ∆−1 n E[Zi Kh (x − Xti )]

= E [β 2 (Xt ) + (nh)−1 κ2 f −1 (Xti )


− ∆n (αn,h − α)2 (Xt )}]Kh (x − Xt )

= µβ (x) + o(1), µβ (x) = f (x)β 2 (x).

−1 2 The mean βn,h is approximated as βn,h (x) = µβ,n,h (x)fX,n,h (x) + −1 O((nh) ). Under conditions (2.1) and (3.1) for the functions α and β in class C2 (X ), the bias of the estimator βbn is 2 bβ 2 ,n,h (x) = βn,h (x) − β 2 (x) = h2 bβ (x) + o(h2 ), m2K −1 (2) f (x){µβ (x) − β 2 (x)f (2) (x)} bβ 2 (x) = 2

and its variance is vβ,n,h (x) = (nh)−1 {σβ2 (x)+ o(1)} where σβ2 (x) is the first 2 term in the expansion of κ2 f −1 (x)∆−2 n V ar(Zt | Xt = x), provided by the −2 2 approximation of ∆n V ar(Zt | Xt = x) by 4 E{∆2n (b αn,h − α)4 (Xt ) + β 4 (Xt )∆−2 n ε

2 + 2β 2 (Xt )(b αn,h − α)2 (Xt ) | Xt = x} − E 2 (∆−1 n Zt | Xt = x)

4 2 = β 4 (x)∆−2 αn,h − α)4 (x) + 2β 2 (x){vα,n,h (x) + b2α,n,h (x)} n Eε + ∆n E(b

− {β 2 (x) + vα,n,h (x) + b2α,n,h (x)}2

4 4 = β 4 (x){∆−2 n Eε − 1} = 2β (x) + o(1),

thus σβ2 is written in the form σβ2 (x) = 2κ2 f −1 (x)β 4 (x). Under the condition h = hT = 0(T −1/5 ), let cα = limT →∞ (T h5T )1/2 .

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Diffusion processes

FunctionalEstimation

151

Proposition 8.1. Under Conditions 2.1, 2.2, 3.1 and 3.2 for the functions α and β in class Cs (X ), the estimators α bn,h and βbn,h are uniformly consistent on X , with bias bα,n,h (x) = αn,h (x) − α(x) = hs bα (x) + o(hs ), msK −1 (s) bα (x) = f (x){µ(s) (x)}, α (x) − α(x)f s! 2 bβ 2 ,n,h (x) = βn,h (x) − β 2 (x) = hs bβ (x) + o(hs ), msK −1 (s) f (x){µβ (x) − β 2 (x)f (s) (x)} bβ 2 (x) = s!

and their variances are vα,n,h (x) and vβ,n,h (x). Moreover kb αn,h (x) − αn,h (x)kp = 0((T h)−1/p ) kβbn,h (x) − βn,h (x)kp = 0((nh)−1/p ) ,

for every p ≥ 2, where the approximations are uniform. If h = 0(T −1/5 ), the process (T h)1/2 (b αn,h − α − cα bα ) converges weakly to centered Gaussian 2 process with variance σα2 (x), the process (nh)1/2 (βbn,h − β 2 − γ 1/2 bβ 2 ) converges weakly to centered Gaussian process with variance σβ2 (x) at x, and the limiting covariances are zero. The order for the bandwidths is the order of the optimal bandwidth for the asymptotic mean squared errors of estimation of α. The conditions ensure a Lipschitz property for the second order moment of the increments of the processes, similar to Lemma 2.2 for the density. Moreover, the covariances develop like in the proof of Theorem 2.1. The variance of the variable Y in model (8.2) being a function of X, the regression function α is also estimated by the mean of a weighted kernel as in Section 3.6, with the weighting variables w(X b ti ) = σα−1 (Xti ). As previously, the approximations of the bias and variance of the new estimator (3.19) of the drift function are modified by introducing w bn and its asymptotic distribution is modified. With a partition of [0, T ] in subintervals Ii of unequal length ∆n,i varying with the observation timesti of the process the variable Yi has to be normalized by ∆n,i , 1 ≤ i ≤ n. For every x in Xn,h , the estimators are Pn −1 i=1 ∆n,i Yi Kh (x − Xti ) P α bn,h (x) = , n i=1 Kh (x − Xti ) −1/2

Zn,i = ∆n,i {Yi − ∆n,i α bn,h (Xti )}, Pn 2 i=1 Zn,i Kh (x − Xti ) 2 βbn,h (x) = P . n i=1 Kh (x − Xti )

January 31, 2011

17:17

152

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

The results of Proposition 8.1 are satisfied, replacing the means of sums Pn −1 −1 with terms ∆−1 n,i by means with coefficient n i=1 ∆n,i and assuming −1 that the lengths ∆n,i have the order n T . The optimal bandwidth for the estimation of α is O(T −1/(2s+1) ) and its asymptotic mean squared error is AM SEα (x) = (T h)−1 σα2 (x) + hs2 b2α,s (x), it is minimum for the bandwidth function hα,AMSE (x) =

n (s!)2 κ

2

T −1 V ar(∆−1 n,i Yti )

(s) (x)}2 2sm2sK {µ(s) α (x) − α(x)f

o1/(2s+1)

.

The optimal local bandwidth for estimating the variance function β 2 of the diffusion is a O(n−1/(2s+1) ) and it minimizes AM SEβ (x) = (nh)−1 σβ2 (x) + hs2 b2β,s (x). A diffusion model including several explanatory processes in the coefficients α and β may be written using an indicator process (Jt )t with values in a discrete space {1, . . . , K} as dXt =

K X

αk (Xt )1{Jt = k}dt +

k=1

K X

k=1

βk (Xt )1{Jt = k}dBt , t ∈ [0, T ]. (8.5)

Let Xtk = Xt 1{Jt = k} be the partition of the variable corresponding to the models for the drift and the variance of equation (8.5). The model is equivalent to dXt =

K X

k=1

αk (Xtk )dt +

K X

βk (Xtk )dBt

k=1

and the estimators of the 2K functions αk and βk are defined for every for x in Xn,h by Pn −1 i=1 ∆n,i Yi Kh (x − Xti ,k ) Pn α bk,n,h (x) = , i=1 Kh (x − Xti ,k ) Pn −1 2 i=1 ∆n,i Zi Kh (x − Xti ,k ) 2 Pn βbk,n,h (x) = . i=1 Kh (x − Xti ,k )

Their means are approximated as the estimators for model (8.1) by −1 αk,n,h (x) = µαk ,n,h (x)fX,n,h (x) + O((T h)−1 ), −1 2 βk,n,h (x) = µβk ,n,h (x)fX,n,h (x) + O((nh)−1 ),

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Diffusion processes

FunctionalEstimation

153

where µα,k,n,h (x) = En

−1

n X i=1

∆−1 n,i

Z Z

yKh (x − s) dFXk,ti ,Yti (s, y)

h2 2 m2K µ(2) αk (x) + o(h ) 2 µαk (x) = f (x)αk (x), n X 2 µβk ,n,h (x) = n−1 ∆−1 n,i E[Zi Kh (x − Xk,ti )] = µαk (x) +

i=1

h2 (2) m2K µβk (x) + o(h2 ), 2 µβk (x) = f (x)βk2 (x). = µβk (x) +

The norms and the asymptotic behaviour of the estimators is the same as in Proposition 8.1. The two-dimensional model dXt = αX (Yt )dt + βX (Xt )dBX (t), dYt = αY (Yt )dt + βY (Yt )dBY (t) with independent Brownian processes BX and BY is a special case where all parameters are estimated as before. The process (8.1) is generalized with functions depending of the samplepath and of the current time dXt = α(t, Xt )dt + β(t, Xt )dBt , t ∈ [0, T ]

(8.6)

under similar conditions. A discretization of the time interval [0, T ] leads to Yi = Xti+1 − Xti = (ti+1 − ti )α(ti , Xti ) + β(ti , Xti )(Bti+1 − Bti ). The functions α and β are now defined in (R+ × X) and they are estimated by Pn Y K (x − Xti )Kh2 (t − ti ) i=1 P n i h1 α bn,h (t, x) = , ∆n i=1 Kh1 (x − Xti )Kh2 (t − ti ) Pn Z 2 K (x − Xti )Kh2 (t − ti ) 2 i=1 P n i h1 βbn,h (t, x) = , ∆n i=1 Kh1 (x − Xti )Kh2 (t − ti )

with Zi = Yi − ∆n α bn,h (ti , Xti ) = ∆n (α − α bn,h )(ti , Xti ) + β(ti , Xti )εi . Their variance has now the order (nh1 h2 )−1 , their bias is a O((h1 h2 )2 ) and the convergence rate of the centered processes is (nh1 h2 )1/2 .

January 31, 2011

154

8.3

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Estimation for continuous diffusion processes

The process {Xt , t ∈ [0, 1]} is extended to a time interval [0, T ] by rescaling: Xt = XT s , with s in [0, 1] and t in [0, T ]. Now the Gaussian process B is mapped from [0, 1] onto [0, T ] by the same transform and Bs = T 1/2 Bt/T is the Brownian motion extended from [0, 1] to [0, T ]. The observation of the sample-path of the process {Xt , t ∈ [0, T ]} allows to construct estimators similar those of smooth density and regression function in Sections 2.10 and 3.10, under the ergodic property (2.13). The Brownian process (Bt )t≥0 is a martingale with respect to the filtration generated by the (Bu )u
b X) = Xt − X0 , thus E A(t; b X) = A(t; X) and its variis estimated by A(t; Rt 2 Rt ance equals V ar{ 0 α(Xs ) ds} + E 0 β (Xs ) ds. The drift function α(Xt ) is estimated by smoothing the sample-path of the process X in a neighborhood of Xt = x RT Kh (x − Xs ) dXs α bT,h (x) = R0 T . (8.7) K (x − X ) ds h s 0

The estimators of the density and µα (x) = α(x)fX (x) defining (8.7) are Z T fbX,T,h (Xt ) = T −1 Kh (x − Xs ) ds, 0

µ bα,T,h (x) = T

−1

Z

0

T

Kh (x − Xs ) dXs .

Their limits are expressed with the mean marginal density of the process Z T fX (x) = lim T −1 E fXs (x) ds T →∞

0

and the mixing property of the sample path of the process X implies that Z T −1 fX (x) − lim T E fXs (x) ds = O(T −1/2 ). T →∞

0

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Diffusion processes

FunctionalEstimation

155

Assuming that the kernel satisfies Conditions 2.1-2.2, their moments are approximated using Taylor expansions and the properties of the Brownian motion, with covariance function E(Bs Bt ) = s ∧ t. With a diffusion process X, their expectations are Z E fbX,T,h (x) = Kh (u − x) fX (u) du IX

h2 (2) = fX (x) + m2K fX (x) + o(h2 ), 2 Z T Eµ bα,T,h (x) = T −1 E Kh (x − Xs ){α(Xs ) ds + β(Xs ) dBs } 0 Z = α(u)Kh (u − x)fX (u) du IX

= α(x)fX (x) +

h2 m2K (αfX )(2) (x) + o(h2 ), 2

so the bias of the estimator of µα (x) is h2 bµα (x) = h2 m2K (αfX )(2) (x)/2 + RT o(h2 ). Its variance T −2 E 0 Kh (Xt − x) {dXt − α(Xt ) dt}2 is expanded using the inRSection 2.10, now the covariance R ergodicity property (2.16) as−1 T −1 T of T 0 Kh (Xs −x)β(Xs ) dBs and T 0 Kh (Xt −x)β(Xt ) dBt as a sum R T Id (T ) + Io (T ), where Id (T ) = T −2 E 0 Kh2 (Xt − x)β 2 (Xt ) dt develops as Z Id (T ) = T −1 Kh2 (u − x)β 2 (u)fX (u) du IX

= (T hT )−1 κ2 β 2 (x)fX (x) + o((T hT )−1 )

and expectation using the ergodicity property, with R T Rthe R T Io (T ) is expanded T 2 d(s∧t) = 2 (T −s) ds = T and the notation αh (u, v) = |u−v|/2hT 0 0 0 Z Io (T ) = Kh (u − x)Kh (v − x)xβ(u)β(v) dFXs ,Xt (u, v) du dv 2 \D IX X

=

Z

IX\{u}

Z

1

−1

K(z − αh (u, v))K(z + αh (u, v)) dz β(u)β(v) dπu (v) dFX (u)}{1 + o(1)}.

For every fixed u 6= v, αhT (u, v) tends to infinity as hT tends to zero, R 1/2 then the integral −1/2 K(z − αh (u, v))K(z + αh (u, v)) dz tends to zero. If αh (u, v) = O(hT ), this integral does not disappear but πu (v) tends to zero, therefore the integral Io (T ) is a o((T hT )−1 ) as T tends to infinity. Under the ergodicity condition (2.16) for sets of k finite dimensional distributions of X, the Lp -norm of the centered estimator of µα also satisfies kb µα,T,h (x)−

January 31, 2011

156

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

µα,T,h (x)kp = O((T hT )−1/p ) and the approximation (3.2) is also satisfied for the estimator α bT,h . It follows that the estimator α bT,h (x) of a drift function α in class Cs , for s ≥ 2, has a bias and a variance bα,T,h (x; s) = hsT bα (x; s) + o(hsT ), msK −1 (s) bα (x; s) = f (x){(αfX )(s) (x) − α(x)fX (x)}, s! X vm,T,h (x) = (T hT )−1 {σα2 (x) + o(1)}, −1 σα2 (x) = κ2 fX (x)β 2 (x)

so they have the same expressions as in the discretized regression model (8.2), the covariance of α bT,h (x) and α bT,h (y) tends to zero. Let Z t Z t Z t Z t = Xt − X0 − α bT,h (Xs ) ds = (α − α bT,h )(Xs ) ds + β(Xt ) dBt , 0

0

0

(8.8) its expectation conditionallyRon the filtration generated by the process X t up to t− is E(Zt | Ft ) = − 0 bα,T,h (Xs ) ds = O(h2 ) for every t > 0 and the main term of its conditional variance Z t Z t V ar(Zt | Xt ) = V ar{ α bT,h (Xs )ds} + β 2 (Xs )ds 0 0 Z t Z t − 2Cov{ (b αT,h )(Xs ) ds, β(Xs ) dBs } + O((T hT )−1 ) 0

0

Rt is 0 β 2 (Xs )ds. The variance function β 2 (Xt ) is therefore consistently estimated by RT 2 0 Zs Kh (Xs − x) dZs 2 b βT,h (x) = . (8.9) RT Kh (Xs − x) ds 0

Under conditions (2.1) and (3.1) for the functions α and β in class Cs (X ), 2 the bias of the estimator βbT,h is

bβ,T,h (x) = hs bβ (x; s) + o(hs ), msK −1 bβ (x; s) = f (x){(f β 2 )(s) (x) − β 2 (x)f (s) (x)}. s! Its variance is vβ,T,h (x) = (T h)−1 {σβ2 (x) + o(1)} where σβ2 (x) is the first term in the expansion of κ2 f −1 (x)V ar(Zt2 | Xt = x) calculated like in the discrete model, that is σβ2 (x) = 2κ2 f −1 (x)β 4 (x), as in Proposition 8.1. Under the previous conditions, the processes (T hT )1/2 (b αT,h − α − bα,T,h ) 1/2 b2 2 and (T hT ) (βT,h − β − bβ 2 ,T,h ) converge weakly to a centered Gaussian processes with mean zero, covariances zero and respective variance functions σα2 and σβ2 .

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Diffusion processes

FunctionalEstimation

157

The mean squared error of the estimator at x for a marginal density in Cs is then −1 M ISET,hT (x) = (T hT )−1 κ2 fX (x)V ar(Y | X = x)

−1 + h2s ) + o(h2s T bα (x; s) + o((T hT ) T )

and the optimal local and global bandwidths minimizing the mean squared (integrated) errors are O(T 1/(2s+1) ) n 1 σ 2 (x) o1/(2s+1) α hAMSE,T (x) = T 2sb2α (x; s) and, for the asymptotic mean integrated squared error criterion n 1 R σ 2 (x) dx o1/(2s+1) R α hAMISE,T = . T 2s b2α (x; s) dx With the optimal bandwidth rate, the asymptotic mean (integrated) squared errors are O(T 2s/(2s+1) ). The same expansions as for the variance of µ bT,h (x) and fbX,T,h (x) in Section 2.10 prove that the finite dimension distributions of the process (T hT )1/2 (b αT,h − α − bα,T,h ) and (T hT )1/2 (βbT,h − β − bβ,T,h ) converge to those of a centered Gaussian process with mean zero, covariances zero and variance functions σα2 and σβ2 . Lemma 3.3 generalizes and the increments E{b αT,h (x) − α bT,h (y)}2 and E{βbT,h (x) − βbT,h (y)}2

are approximated by O(|x − y|2 (T h3T )−1 ) for every x and y in IX,h such that |x − y| ≤ 2hT . Then the processes (T hT )1/2 {b αT,h − α}I{IX,T } and 1/2 b (T hT ) {βT,h − β}I{IX,T } converge weakly to σα W1 + γ 1/2 bα and σβ W2 + γ 1/2 bβ , respectively, where W1 and W2 are centered Gaussian processes on IX with variance 1 and covariances zero. The covariance Cα,β,T,h (x, y) of α bT,h (x) and βbT,h (y), with 2|x − y| > hT develops using the approxiRT RT mation (3.2) as {f (x)f (y)T }−2[E{ 0 Kh (Xs − x)β(Xs ) dBs }{ 0 Kh (Xs − RT y)(2Zt dZt − β 2 (Xt ) dt)} − Eα(x){ 0 Kh (Xs − y)β(Xs ) dBs }(fbT,h − RT fT,h )(x) − Eβ(y){ 0 Kh (Xs − x)(2Zt dZt − β 2 (Xt ) dt)}(fbT,h − fT,h )(y) + α(x)β(y)(T hT )−1 Cov(fbT,h (x), fbT,h (y)), it is therefore a o((T hT )−1 ). According to the local optimal bandwidths defined in the previous sections, the estimators α bn,h and βbn,h are calculated with a functional bandwidth sequences (hn (x))n or (hT (x))T . The assumptions for the convergence of these sequences are similar to the assumptions for the nonparametric regression with a functional bandwidth and the results of Chapter 4 apply immediatly for the estimators of the discretized or continuous processes (8.2) and (8.1).

January 31, 2011

158

8.4

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Estimation of discretely observed diffusions with jumps

Let α, β and γ be functions of class C2 on a metric space (X, k · k), let B e be a centered martingale be the standard Brownian motion, M = N − N e (t) = associated to a point process N , with predictable compensator N Rt Y dΛ, and such that M is independent of B. The process Y is predictable 0 and there exists a function g defined on [0, 1] such that sups∈[0,1] |T −1 YT s − g(s)| converges to zero in probability, the function R T g and the hazard function e are finite for every λ is supposed to be in class C2 (R); ET −1 0 γ 2 dN stopping time T . The process Xt solution of the stochastic differential equation dXt = α(Xt )dt + β(Xt )dBt + γ(Xt )dMt , t ∈ [0, T ],

(8.10)

has a discrete and a continuous part. A discretization of this equation into n sub-intervals of length ∆n,i tending to zero as n tends to infinity gives the approximated equation Yi = Xti+1 − Xti = ∆n,i α(Xti ) + β(Xti )∆Bti + γ(Xti )∆Mti .

Let εi = ∆Bti = Bti+1 − Bti , with zero mean and variance ∆n,i conditionally on the σ-algebra Fti generated by the sample-paths of X up to ti , ηi = η(ti+1 ) defined by Mti+1 − Mti , with expectation zero and variance eti+1 − N eti = O(∆n,i ) conditionally on the σ-algebra Fti generated by the N sample-paths of X; E{α(Xti )εi } = 0, E{β(Xti )ηi } = 0, and the martingales (Bt )t≥0 and (Mt )t≥0 have independent increments, by definition. The e are estimated from the functionals of the martingale M and the process N observation of the point process N , as in Chapter 4. The variables XTi are supposed to satisfy an ergodic property for the random stopping times of the counting process N , in addition to Conditions 6.2 and 8.1. Condition 8.2. There exists a mean density of the variables XTi defined as the limit Z T −1 fN (x) = lim T fXs (x) dN (s). T →∞

0

This condition is satisfied if the jump part of Rthe process Xt satisfies the T e (s). The diffuproperty (8.3) and the limit is fN (x) = T −1 E 0 fXs (x) dN sion process Xt defined by (8.10) has the mean Z t µT = EX0 + E α(Xs )ds 0

= EX0 +

Z Z X

0

t

α(x)fXs (x) dt dx = EX0 + t

Z

X

α(x)f (x) dx

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

159

Diffusion processes

and the variance of the normalized variable T −1/2 (XT − µT ) is finite if the integrals Z T Z Sα = ET −1 α2 (Xs ) ds = α2 (x)f (x) dx + o(1), Sβ = ET −1 Sγ = ET −1

Z

Z

0

T

β 2 (Xs ) ds =

0

X

β 2 (x)f (x) dx + o(1),

X

T 0

Z

e (s) = γ 2 (Xs ) dN

Z

γ 2 (x)fN (x) dx + o(1)

X

are finite. Then T 1/2 (T −1 XT −µT ) converges weakly to a centered Gaussian variable with variance SX = Sα +Sβ +Sγ . Let SX (t) be the function defined as above with integrals on [0, t]. The process T 1/2 (T −1 XsT − µsT )0≤s≤1 is a sum of stochastic integrals with respect to the martingales B and M . Proposition 8.2. The process WT,s = T 1/2 (T −1 XsT − µsT )0≤s≤1 is a martingale. If SX < ∞, WT,s converges weakly to a Brownian motion BX with variance function SX (s) on [0, 1]. Let a in R and Ta = inf{s ∈ [0, 1]; BX (s) = a} be a stopping time for the process BX , then for every θ ≥ 0 √ E exp{θSX (Ta )} = exp(−a 2θ). Let a in R and TT,a = inf{s ∈ [0, 1]; WT,s = a} be a stopping time for the process WT,s . Corollary 8.1. For every θ ≥ 0, E exp{θSX (TT,a )} converges to √ exp(−a 2θ) as T tends to infinity. Moments of discontinuous parts of a diffusion process with jumps require another ergodicity condition defining another mean density and it is satisfied under the mixing property (8.3). Conditionally on Fti , the variables Yi have the expectation ∆n,i α(Xti ) and the variance Z ti+1 e (s) V ar(Yi |Xti ) = β 2 (Xti )∆n,i + γ 2 (Xs ) dN ti

e (ti ) + o(∆n,i ) = O(∆n,i ). = β 2 (Xti )∆n,i + γ 2 (Xti ) ∆N

A nonparametric estimator of the function α is the kernel estimator normalized by ∆n,i as in the previous section Pn −1 i=1 ∆n,i Yi Kh (x − Xti ) Pn α bn,h (x) = , i=1 Kh (x − Xti )

January 31, 2011

17:17

160

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Rt Pn −1 −1 e for x in Xn,h . Let ∆−1 n denote n i=1 ∆n,i and E Nt = 0 g(s)λ(s) ds, then the mean of α bn,h (x) is approximated by h2 αn,h (x) = α(x) + m2K {(f α)(2) (x) − α(x)f (2) (x)} + o(h2 ). 2 The variance of α bn,h (x) is a O((T h)−1 ) n X −1 vα,n,h (x) = n−1 ∆−2 {σα2 (x) + o(1)}, n,i (nh) σα2 (x)

= κ2 f

i=1 −1

(x)∆−1 n V ar(Yt | Xt = x)

= κ2 f −1 (x){β 2 (x) + γ 2 (x)g(t)λ(t)} and its covariances tend to zero. The process (T h)1/2 (b αn,h − α) has the −1 asymptotic variance κ2 σα2 (x)fX (x), at x. P The discrete part of X is X d (t) = s≤t γ(Xs )∆Ns and its continuous Rt Rt Rt es , with variations part is X c (t) = 0 α(Xs ) ds + 0 β(Xs ) dBs − 0 γ(Xs ) dN on (ti , ti+1 ) ∆Xic = α(Xti ) ∆n,i + β(Xti ) ∆Bti − γ(Xti ) ∆n,i Y (ti )λ(ti ) = Op (∆n,i ). Rt Then the sum its jumps converges to 0 Eγ(Xs ) g(s)dΛs . Let (Ti )1≤i≤N (T ) be the jumps of the process N . The jumps ∆X d (Ti ) = γ(XTi ) yield a consistent estimator of γ(x), for x in Xn,h P d 1≤i≤N (T ) ∆X (Ti )Kh (x − XTi ) P γ bn,h (x) = 1≤i≤N (T ) Kh (x − XTi ) P 1≤i≤N (T ) γ(XTi )Kh (x − XTi ) P = . 1≤i≤N (T ) Kh (x − XTi )

The expectation of b γn,h (x) is approximated by the ratio of the means of the numerator and the denominator. For the numerator Z T h2 −1 ET γ(Xs )Kh (x−Xs ) dNs = (γfN )(x)+ m2K {(γfN )(x)}(2) +o(h2 ) 2 0 R T and, for the denominator ET −1 0 Kh (x − Xs ) dNs = fN (x) + (2) h2 2 γn,h (x) is then 2 m2K fN (x) + o(h ). The bias of b h2 (2) m2K {fN (x)}−1 [{γ(x)fN (x)}(2) − γ(x)fN (x)] + o(h2 ), 2 also denoted bγ,n,h (x) = h2 bγ . The variance of b γn,h (x) is deduced from the variance of the numerator Z T T −2 E Kh2 (x − Xs ) dNs bγ,n,h(x) =

0

= (T h)−1 {κ2 fN (x) + κ22

h2 (2) f (x)} + o(T −1 hT ) 2 N

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

161

Diffusion processes

the variance of the denominator Z T T −2 E γ 2 (Xs )Kh2 (x − Xs ) dNs 0

= (T h)−1 {κ2 γ 2 (x)fN (x) + h2 κ22 (γ 2 fN )(2) (x)} + o(hT −1 )

and their covariance Z TZ T −2 E 0 −1

= (T h)

0

T

γ(Xs )Kh (x − Xs )Kh (x − Xt ) dNs dNt

{κ2 γ(x)fN (x) + h2 κ22 (γfN )(2) (x)} + o(hT −1 ),

therefore vγ,n,h (x) = T −1 hvγ (x) with

(2)

vγ (x) = κ22 {fN (x)}−1 {(γ 2 fN )(x)(2) − γ 2 (x)fN (x)} + o(hT −1 ).

It follows that the process (T h−1 )1/2 (b γn − γ − cα bγ ) converges weakly to a centered Gaussian process with variance function vγ (x) and covariances zero. For the estimation of the variance function β of model (8.10), let Zi = Yi − ∆n,i α bn,h (Xti ) − b γn,h (Xti )ηi

= ∆n,i (α − α bn,h )(Xti ) + β(Xti )εi + (γ − γ bn,h )(Xti )ηi ,

its conditional expectation E(Zi | Xti = x) = ∆n,i (α − αn,h )(Xti ) tends to zero and its conditional variance satisfies 2 4 −1 ∆−1 ) + o((T h)−1 ). n,i V ar{Zi | Xti } = β (Xti ) + o(h ) + o((nh)

An estimator of the function β is deduced for x in Xn,h P −1 2 1≤i≤n ∆n,i Zi Kh (x − Xti ) 2 b Pn βn,h (x) = . i=1 Kh (x − Xti )

The previous approximations of the estimator βbn,h given in Proposition 8.1 are modified, its expectation is approximated by X −1 2 2 2 βn,h (x) = n−1 ∆−1 n,i EZi Kh (x − Xti )fN (x) + o(h ) 1≤i≤n

2 therefore its bias is E βbn,h − β 2 = bβ,n,h + o(h2 ) with

h2 m2K f −1 (x){(f β 2 )(2) (x) − β 2 (x)f (2) (x)} + o(h2 ). 2 Under conditions (2.1) and (3.1) for the function β in class C2 (X ), the 2 variance of the estimator βbn,h is vβ,n,h (x) = (nh)−1 {σβ2 (x) + o(1)}, with bβ,n,h =

−1 2 σβ2 (x) = κ2 fX (x)∆−2 n V ar(Zt | Xt = x). t

January 31, 2011

17:17

162

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

2 The normalized variance ∆−2 n V ar(Zt | Xt = x) develops as 4 4 E{∆2n (b αn,h − α)4 (x) + β 4 (x)∆−2 bn,h )4 (x)∆−2 n ε + (γ − γ n η

+ O(h4 ) + O((nh)−1 ) + O(hT −1 )

where the Burkh¨ older-Davis-Gundy inequality implies that the order of Eηi4 2 2 is a O((Eηi ) ) = O(∆2n,i ). Then, from the expression of the moments of the 4 4 variable ε, σβ2 (x) = β 4 (x)(∆−2 n Eε −1)+o(1) = 2β (x)+o(1). The variance of βb2 is therefore written vβ,n,h (x) = (nh)−1 vβ (x), it is a O((nh)−1 ) and n,h

the process (nh)1/2 (βbn − β − (nh5 )1/2 bβ ) converges weakly to a centered Gaussian process with variance function vβ and covariances zero. 8.5

Continuous estimation for diffusions with jumps

In model (8.10), the estimator α bT,hT of Section 8.3 is unchanged and new estimators of the functions β and γ must be defined from the continuous observation of the sample path of X. The discrete part of X is also written Rt Xtd = 0 γ(Xs )dNs and the point process N is rescaled as Nt = NT s , with t in [0, T ] and s in [0, 1]. Let NT (s) = T −1 NT s ,

XT (s) = T −1 XT s , t ∈ [0, T ], s ∈ [0, 1]. R eT (t) = T −1 t YT (s)λ(s) ds The predictable compensator of NT is written N 0 on [0, 1] and it is assumed to converge uniformly on [0, 1] to its mean Rt d e E NT (t) = 0 g(s)λ(s) ds, in probability. Then XT (t) converges uniformly Rt in probability to 0 Eγ(XT (s)) g(s)dΛ(s). The continuous part of X is dXtc = α(Xt ) dt + β(Xt ) dBt − γ(Xt )Yt λt dt. A consistent estimator of γ(x), for x in IX,T,h RT Kh (x − Xs ) dX d (s) γ bT,h (x) = R0 T , Kh (x − Xs ) dN (s) 0 RT Kh (x − Xs )γ(Xs ) dN (s) = 0 RT , Kh (x − Xs ) dN (s) 0

it is identical to the estimator previously defined for the discrete diffusion process. Its moments calculated in the continuous model (8.10) are iden1/2 tical to those of Section 8.4 then the process (T h−1 (b γT,hT − γ − cα bγ ) T ) converges weakly to a centered Gaussian process with variance function vγ and covariances zero.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

163

Diffusion processes

The variance function β 2 (Xt ) is now estimated by smoothing the squared variations of the process Z t Z t = Xt − X0 − α bT,h (Xs ) ds (8.11) 0 Z t Z t Z t (γ − γ bT,h )(Xs ) dMs . = (α − α bT,h )(Xs ) ds + β(Xs ) dBs + 0

0

0

For every t in [0, T ], its first two conditional moments are Z t E(Zt | Ft ) = − bα,T,h (Xs ) ds = O(h2 ) 0

and

V ar(Zt | Ft ) = V ar{

=t

Z

Z

0

t

Z t α bT,h (Xs )ds} + E β 2 (Xs )ds 0 Z t es +E (γ − γ bT,h )2 (Xs ) dN 0

2

β (x)fXs (x) dx + O((T hT )−1 ) + O(h4T ).

X

Furthermore, the Burkh¨ older-Davis-Gundy inequality implies the existence of a constant c4 such that Z t Z V arZt2 = E{ β(Xs ) dBs }4 − {t β 2 (x)f (x) dx}2 0 Z t Z ≤ c4 E β 4 (Xs ) ds = c4 t β 2 (x)f (x) dx. 0

The variance function β 2 (x) is then consistently estimated smoothing the process Zt2 RT Kh (Xs − x) Zs dZs 2 b βT,h (x) = 2 0 R T . (8.12) Kh (Xs − x) ds 0

Under conditions (2.1) and (3.1) for the function β in class C2 (X ) and using the ergodicity property (2.13) for the limiting density f of the process (Xt )t∈[0,T ] , the expectation of the denominator of (8.12) is Z T Z T T −1 E Kh (Xs − x) ds = T −1 E fXs (x) ds 0

0

Z T h (2) m2K T −1 E fXs (x) ds + o(h2 ) 2 0 h2 (2) = f (x) + m2K f (x) + o(h2 ) 2 2

+

January 31, 2011

164

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

the expectation of the numerator is Z T 2T −1 E Kh (Xs − x)Zs dZs 0

= 2T −1

Z

T

E

0

Z

2

= β (x) f (x) +

X

Kh (u − x) β 2 (u) fXs (u) du + o(h4 )

h2 m2K (β 2 (x)f (x))(2) + o(h2 ) 2

and its bias is denoted bβ,T,h = h2 bβ + o(h2 ), with bβ =

1 m2K f −1 (x){(f β 2 )(2) (x) − β 2 (x)f (2) (x)}. 2

Under conditions (2.1) and (3.1) for the function β in class C2 , the R variance of the estimator βbT,h is obtained from E(Zt2 | X) = β 2 (Xs ) ds, V ar(Zt2 | X) = O(t) and expanding Z TZ −2 ET Kh2 (x − y)V ar(Zt2 | Xt = y)fXt (y) dy dt = O((hT )−1 ), 0

2 it is therefore written σβ,T,h = (hT )−1 vβ + o((hT )−1 ). Then the process (T hT )1/2 (βbT,h − β − (T h5T )1/2 bβ ) converges weakly to a centered Gaussian process with variance function vβ and covariances zero.

8.6

Transformations of a non-stationary Gaussian process

Consider the non-stationary processes Z = X ◦ Φ, where X is a stationary Gaussian process with covariance R(x, y) = E(Xx Xy ) and Φ is a monotone function C1 ([0, 1]) with Φ(0) = 0 and Φ(1) = 1. The transform is expressed as Φ(x) = v −1 (1)v(x) with respect function of R x to the integrated singularity Rx the covariance r(x, x), v(x) = 0 ξ(u) du. Conversely, 0 ξ(u) du = cξ Φ(x) R1 with cξ = 0 ξ(u) du. A direct estimator of the regularity function ξ is b n (x) defined by (1.12) obtained by smoothing the estimator Φ Z 1 b n (y) ξbn,h (x) = Vn (1) Kh (x − y) dΦ =

Z

0

0

1

Kh (x − y) db vn (y).

R1 The expectation of ξbn,h (x) is ξn,h (x) = 0 Kh (x − y) dv(y) and the process R1 b n − Φ)(y) is uniformly consistent, since (ξbn,h − ξn,h )(x) = 0 Khn (x − y)d(Φ

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

165

Diffusion processes

an integration by parts implies Z 1 Z b b k Khn (s − y) d(Φn − Φ)(y)k ≤ kΦn − Φk 0

0

1

|dKhn (s − y)|

b n − Φk + sup |Khn | kΦ Z b ≤ (sup |K| + |dK(z)|) h−1 n kΦn − Φk

which converges to zero in probability, by the weak convergence of b n − Φk. The process n1/2 (b n1/2 k√Φ vn − v) converges weakly to the proRx cess 2 0 v(y)dW (y) where W is a Gaussian process with mean zero and 1/2 covariances vn − v) R x∧y 2 x ∧ y at (x, y), then the covariance of the limit of n (b b is 2 0 v (y) dy at x 6= y. The limiting variance of ξn,h (x) is E{

Z

0

1

Kh (x − y) d(b vn − v)(y)}2 = E +E

= O(n−1

Z

0

Z

0

1

1

Z

0

1

Z

0

1

Kh2 (x − y) dV ar(b vn − v)(y)

Kh (x − y)Kh (x − u) dCov{(b vn − v)(y), (b vn − v)(u)

Kh2 (x − y)v 2 (y) dy) = O((nh)−1 )

The convergence rate of the process ξbn,h is therefore (nhn )1/2 and the finite dimensional distributions of (nhn )1/2 (ξbn,h − ξn,h ) converge to those of a Gaussian process with mean zero, as normalized sums of the independent variables defined as the weighted quadratic variations of the increments of Z. The covariances of (nhn )1/2 (ξbn,h − ξn,h ) are zero except on the interval [−hn , hn ] where they are bounded, hence the covariance function converges to zero. The quadratic variations of ξbn,h satisfy a Lipschitz property of moments E|(ξbn,h − ξn,h )(x) − (ξbn,h − ξn,h )(y)|2 Z 1 = 2n−1 | {Kh2 (x − u) − Kh2 (y − u)}v 2 (u) du| 0

it is then a O((nh3n )−1 |x − y|2 ) for |x − y| ≤ 2hn . It follows that the process (nhn )1/2 (ξbn,h − ξn,h ) converges weakly to a continuous process with mean zero and variance function 2v 2 and covariances zero. The singularity function of the spatial covariance of a Gaussian process Z is estimated by smoothing the estimator of the integrated spatial transform of Z on [0, 1]3 , the convergence rate of the estimator is then (nh3 )1/2 .

January 31, 2011

166

8.7

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Exercises

(1) Calculate the moments of the estimators for the continuous process (8.6) and write the necessary ergodic conditions for the convergences in this model. (2) Calculate the bias and variance of derivatives of the estimators of functions α, β and γ in the stochastic differential equations model (8.10). (3) Prove Proposition 8.2.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Chapter 9

Applications to time series

Let (X, k·k) be a metric space and (Xt )t∈N be a time series defined on XN by its initial value X0 and a recursive equation Xt = m(Xt−p , . . . , Xt−1 ) + εt where m is a parameric or nonparametric function defined on Xp for some p > 1 and (εt )t is a sequence of independent noise variables with mean zero and variance σ 2 , such that for every t, εt is independent of (Xt−p , . . . , Xt−1 ). The stationarity of a time series is a property of the joint distribution of consecutive observations. The weak stationarity is defined by a constant mean µ and a stationary covariance function ρs,t = Cov(Xs , Xt ) = Cov(X0 , Xt−s ), for every s < t. The series (Xt )t is strong stationary if the distributions of the sequences (Xt1 , . . . , Xtk ) and (Xt1 −s , . . . , Xtk −s ) are identical for every sequence (t1 , . . . , tk , s) in Nk+1 . The nonparametric estimation of the mean and the covariances is therefore useful for modelling the time series. The moving average processes are stationary, they are defined as linear combinations of past and present noise terms such as the MA(q) process Xt = Pq εt + k=1 θk εt−k , with independent variables εj such that Eεj = 0 and
Pq 2 V arεj = σ 2 , for every integer j. The variance of Xt is σq2 = σ 2 k=1 θk +1 and it is supposed to be finite. The covariance of Xs and Xt such that Pq P(t−s+q)∧q 2
0 < t − s < q is Cov(Xs , Xt ) = σ 2 θ + θ , it only k k=t−s k k=t−s+1 depends on the difference t − s. The moving average processes with |θ| < 1 are reversible and the process Xt can be expressed as an auto-regressive process, sum of εt and an infinite combination of its past values. Generally, an AR process is not stationary. In nonstationary series, a nonstationarity may be due to a smooth trend or regular and deterministic seasonal variations, to discontinuities or to a 167

January 31, 2011

168

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

continuous change-points. A transformation such as differencing a stochastic linear trend reduces the nonstationarity of the series, other classical transformations are the square root or power transformations for data with increasing variance. Periodic functions of the mean can be estimated after the identification of the period and nonparametric estimator is proposed in Section 9.2. Change-points of nonparametric regressions in time or at thresholds of the series are stronger causes of non regularity and several phases of the series must be considered separately, with estimation of their change-points. Their estimators are studied in Section 9.5.

9.1

Nonparametric estimation of the mean

The simplest nonparametric estimators for the mean of a stationary process are the moving average estimators k

µ bt,k

1 X = Xt−i , k + 1 i=0

k Xt − for a lag k up to t. The transformed series is Xt − µ bt,k = k+1 P k 1 i=1 Xt−i and it equals (Xt − Xt−1 )/2 for k = 2. A polynomial trend k+1 is estimated by minimizing the empirical mean squared error of the model, then the transformed series Xt − µ bt,k is expressed by the means of moving average of higher order, according to the degree of the polynomial model. Consider the auto-regressive process with nonparametric mean

Xt = µt + αXt−1 + εt , t ∈ N,

(9.1)

with an independent sequence of independent errors (εt )t with mean zero and variance σ 2 . With α 6= 1, its mean µt may be written (1 − α)mt , with an unknown function mt and the solution Xt of Equation (9.1) is Xt =

t−1 X

k=0

µt−k αk + αt X0 +

t X

αk εt−k .

k=1

With a mean and an initial value zero, the covariance of Xs and Xt is Ps∧t ρs,t = σ 2 k=1 α2k and it is not stationary. The asymptotic behaviour of the process X changes as the mean crosses the threshold value 1. For α = 1, the model is the classical nonparametric regression model.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Applications to time series

FunctionalEstimation

169

The parameters of the auto-regressive series AR(1) Xt = µ+αXt−1 +εt , with α 6= 1, are estimated by ¯t + t−1 (b µ bt = (1 − α bt )X αt Xt − X0 ),

m b t = (1 − α bt )−1 µ bt , Pt (Xk−1 − m b t )(Xk − m b t) α bt = k=1Pt , 2 (X − m b ) k−1 t k=1

(9.2)

t

σ bt2

1X ¯t − α = {Xk − (1 − α bt )X bt Xk−1 }2 . t k=1

¯ t + Op (t−1 ) and m ¯ t . For α = 1, the parameFor |α|6= 1, µ bt = (1 − α bt )X b t =X trization µ = (1 − α)m is meaningless and the mean is estimated by µ bt = Pt t−1 k=1 (Xk − Xk−1 ). The estimators are consistent and asymptotially Gaussian, with different normalization sequences for the three domains of Pp α (α < 1, α = 1, α > 1). In the AR(p) model Xt = µt + j=1 αj Xt−j + εt , similar estimators are defined for the regression parameters αj Pt b t )(Xk − m b t) k=j (Xk−j − m α bj,t = Pt b t )2 k=j (Xk−j − m

and the variance is estimated by the mean squared estimation error. In model (9.1) with a nonparametric mean function µt = (1 − α)mt , Xt − mt = α(Xt−1 − mt ) + εt , then the estimator (9.2) of α is modified by ¯ k by a local moving average mean or by a local mean replacing m bk = X Pt j=0 Kh (j − k)Xj m b k = Pt j=1 Kh (j − k) for every k, and the estimator of α becomes Pt (Xk−1 − m b k )(Xk − m b k) α bt = k=1Pt . 2 (X − m b ) k−1 k k=1

Finally, the function µt is estimated by (1 − α bt )m b t or by smoothing Xt − α bt Xt−1 µ bt,h,k = t−1

t X j=0

Kh (j − k)(Xj − α bj Xj−1 )

and the estimator of σ 2 is still defined by (9.2). The asymptotic distributions are modified as a consequence of the asymptotic behaviour of m b k, with mean tending to mk and variance converging to a finite limit. As h tends to zero, the weak convergence to centered Gaussian variables of

January 31, 2011

170

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

t1/2 (m b t − mt ), when |α| < 1, and tα−t (m b t − mt ), when |α| > 1, follows from martingale properties of the time series which imply its ergodicity and a mixing property (Appendix D). If |α| 6= 1 Pt (Xk−1 − m b k−1 )((1 − α)(mk − m b k ) + εk ) α bt − α = k=1 Pt 2 (X − m b ) k−1 k k=1 it is therefore approximated in the same way as in model AR(1) and it converges weakly with the same rate as in this model. When Equation (9.1) is defined by a regular parametrization of the mean µt = (1 − α)mθ (t) for |α| 6= 1, the minimization of squared estimation error Pt Pt ¯t −α kb ε2(t) k2t = k=1 εb2k = k=1 {Xk −(1− α bt )X bt Xk−1 }2 yields estimators of the parameters α and θ for identically distributed error variables εk . If the P variance of εk is σk2 (θ), maximum likelihood estimators minimize k σk−1 ε2k . The robustness and the bias of the estimators in false models have been studied for generalized exponential distributions, the same methods are used in models for time series. In a nonparametric regression model Xt = m(Xt−1 ) + εt

(9.3)

with an initial random value X0 and with independent and identically distributed errors εt with mean zero and variance σ 2 , let F be the continuous distribution function of the variables εt , and f its density. The nonparametric estimator of the function m is still Pt k=1 Kh (x − Xk−1 )Xk . m b t,h (x) = P t k=1 Kh (x − Xk−1 ) It is uniformly consistent under the ergodicity condition Z Z t 1X ϕ(Xk , Xk−1 ) → ϕ(x, y)F (dx − m(y)) dπ(y) t k=1

with the invariant measure π of the process and for every continuous and bounded function ϕ on R2 . Conditions on the function m and the independence of the error variables εi ensure the ergodicity, then the process (th)1/2 (m b t,h − m) converges weakly to a continuous centered Gaussian process with covariances zero and variance κ2 f −1 (x)V ar{Xk | Xk−1 }, where V ar{Xk | Xk−1 } = σ 2 . In model (9.3) with a functional variance, the results of Section 3.6 apply. The observation of series in several groups or in distinct time intervals may introduce a group or time effect similar to population effect in regression samples and sub-regression functions may necessary as in Section 5.6.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

171

Applications to time series

9.2

Periodic models for time series

Let (Xt )t∈N be a periodic auto-regressive time series defined by X0 and Xt = ψ(t) +

p X

αp Xt−p + εt

(9.4)

i=1

where |α| < 1 and ψ is a periodic function defined in N with period τ , ψ(t) = ψ(t + kτ ), for every integers t and k. Let α = (α1 . . . , αp ) and X(p),t = (Xt−1 , . . . , Xt−p ). As ψ(t) = E(Xt − αT X(p),t ), the value of the function ψ at t is estimated by an empirical mean over the periods, with a fixed parameter value α. Assuming that K periods are observed and T = Kτ values of the series are observed, the function ψ is estimated as a mean over the K periods of the remainder term of the auto-regressive process. For every t in {1, . . . , τ } K−1 1 X ψbK,α (t) = (Xt+kτ − αT X(p),t+kτ ) K

(9.5)

k=0

and the parameter vector is estimated by minimizing the mean squared error of the model lK (α) =

T 1 X {Xt − ψbK,α (t) − αT X(p),t }2 . T t=1

The components of the first two derivatives of lK are 2 l˙T,K,t = − T =

T X

∂ ψbK,α {Xt − ψbK,α (t) − αT X(p),t }{ (t) + X(p),t } ∂α t=1

T K−1 2 X 1 X {Xt − ψbK,α (t) − αT X(p),t }{ X(p),t+kτ − X(p),t }, T t=1 K k=0

¨lT,K,t = 2 T

T X t=1

{

1 K

K−1 X k=0

(X(p),t+kτ − X(p),t )}⊗2 .

The vector α is estimated by α bT = arg minα∈]−1,1[d lT,K,t (α). For the first 1/2 ˙ order derivative, T lT,K,t (α0 ) converges weakly to a centered limiting distribution and the second order derivative ¨lT,K,t converges in probability to a positive definite matrix E ¨lT,K,t which does not depend on α. Then −1 the estimator of α satisfies T 1/2 (b αT,K,t − α0 ) = ¨lT,K,t T 1/2 l˙T,K,t (α0 ) + o(1). The estimator α bT is consistent and its weak convergence rate is T 1/2 , if all components of the vector α have a norm smaller than 1. The function ψ is

January 31, 2011

172

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

then consistently estimated by ψbK = ψbK,b αT and, for every t in {1, . . . , τ }, the weak convergence rate of the estimator ψbK (t) is K 1/2 . The true period of the function ψ was supposed to be known. With an unknown period, the estimators ψbK and α bT depend on the parameter τ

and it is consistently estimated by τbT = arg minτ ≤T l[T /τ ] (b αT,τ ). If the function ψ is parametric, its parameters vector θ is estimated by PT minimizing the mean squared error between ψbK and ψθ , T1 t=1 {ψbK (t) − ψθ (t)}2 . As a minimum distance estimator, the estimator θbK is consistent and T 1/2 (θbT − θ) converges weakly to a centered Gaussian variable. The trigonometric series with independent noise are a combination of periodic sinus and cosinus functions r X Xt = M {cos(wj t + Φt ) + sin(wj t + Φt )} + εt =

j=1 r X j=1

{Aj cos(2πwj t) − Bj sin(2πwj t)} + εt

where (wj )j=1,...,r are frequencies wj = jt−1 , Aj = M cos Φt , Bj = M sin Φt such that A2j + Bj2 = M 2 is the magnitude of the series, for j = 1, . . . , r, and Φt its phases. The estimators of the parameters are defined from the Fourier series, for j = 1, . . . , r btj = 2n−1 A

btj = 2n−1 B ct = r−1 M

9.3

t X

k=1 t X

Xt cos(2πkj/t), Xt sin(2πkj/t),

k=1 r X j=1

2 1/2 b2tj + B btj (A ) .

Nonparametric estimation of the covariance function

The classical estimator for estimating the covariances function in a stationary model is similar to the moving average for the mean, with a lag k ≥ 1 between variables Xi and Xi−k , for every i ≥ 1, ρbk,t = (t − k)−1

t X

i=k+1

¯ t )(Xi−k − X ¯ t ). (Xi − X

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

173

Applications to time series

In the auto-regressive model AR(1) with independent errors with mean zero and variance σ 2 , for k ≥ 1, the variable Xk is expressed from the initial value as Xk − m = αk (X0 − m) + Sk,α , where Sk,α =

k X

αk−j εj =

j=1

k−1 X

αj εk−j .

j=0

Let B be the standard Brownian motion, if |α| < 1 the process S[ns],α defined up to the integer part of ns converges weakly to σB{(1 − α2 )1/2 }−1 . If α = 1, the process n−1/2 S[ns],1 converges weakly to σB, and if |α| > 1 the process α−[ns] S[ns],α converges weakly to σB{(α2 − 1)1/2 }−1 . The independence of the error variables εj implies E(Xk − m)(Xk+s − m) = α2k+s V arX0 + Cov(Sk,α , Sk+s,α ), k X

Cov(Sk,α , Sk+s,α ) = E(

j=1

αk−j εj )2 = σ 2

k X

(9.6)

α2(k−j) ,

j=1

2k+s

so E(Xk − m)(Xk+s − m) = α V arX0 + V arSk,α and the covariance function of the series is not stationary. The estimator (9.2) of the variance σ 2 is defined as the empirical variance of the estimator of the noise variables which are identically distributed and independent. In the same way, the covariance is estimated by ρbt,k =

t X 1 {Xi − m b t −α bt (Xi−1 − α bt )}{Xi−k − m b t −α bt (Xi−k−1 − α bt )}, t−k i=k+1

the estimators σ bt2 and ρbt,k are consistent (Pons 2008). The estimators are defined in the same way in an auto-regressive model of order p, with a scalar products α bTt Xi−1 and α bTt Xi−k−1 for p-dimensional variables Xi−1 and Xi−k−1 . In model (9.1), the expansion (9.6) of the variables centered by the mean function is not modified and the covariance E(Xk − mk )(Xk+s − mk+s ) has the same expression depending only on the variances of the initial value and Sk,α , and on α and the rank of the observations. In auto-regressive series with deterministic models of the mean, the covariance estimator is modified by the corresponding estimator of the mean. In model (9.3), the covariance estimator becomes ρbt,k =

t X 1 {Xi − m b t,h (Xi )}{Xi−k − m b t,h (Xi−k )} t−k i=k+1

and the estimators are consistent.

January 31, 2011

174

9.4

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Nonparametric transformations for stationarity

In the nonparametric regression model (9.3), Xt = m(Xt−1 ) + εt with an initial random value X0 and with independent and identically distributed errors εt with mean zero and variance σ 2 , the covariance between Xk and Xk+l is ρt,k,l = E{Xk m∗l (Xk )} − EXk Em∗l (Xk ), with E{Xk m(Xk+l−1 )} = E{Xk m∗l (Xk )}, where m∗l is the composition of l functions m. The nonstationarity of ρt,k,l does not allow to estimate it using empirical means and it is necessary to remove the functional mean µt before studying the covariance of the series. The centered series Yt = Xt − m b t (Xt−1 ) = m(Xt−1 ) − m b t (Xt−1 ) + εt

has a conditional expectation equal to minus the bias of the estimator m bt 2 h (2) −1 E(Yt | Xt−1 ) = − {(mfXt−1 )(2) − mf Xt−1 }(Xt−1 )m2K fX (Xt−1 ) t−1 2 and it is negligeable as t tends to infinity and h to zero. The time series Yt is then asymptotically equivalent to a random walk with a variance parameter σ 2 . The main transformations for nonstationary series (9.3) with a constant variance is therefore its centering. With a varying variance function Eε2i = σi2 = V ar(Xi | Xi−1 ),

the estimator of the mean function of the series has to be weighted by the inverse of the square root of the nonparametric estimator of the variance at Xi , where Pt {Yi − m b t,h (Xi )}2 Kδ (x − Xi ) 2 σ bt,h,δ (x) = i=1 Pn , i=1 Kδ (x − Xi ) as in Section 3.6, the estimator of the regression function is Pt w bt,h,δ (Xi )Yi Kh (x − Xi ) m b w,t,h (x) = Pi=1 n bt,h,δ (Xi )Kh (x − Xi ) i=1 w and the stationary series for (9.3) is Yi = Xi − m b w,t,h (Xi−1 ). A model for non independent stationary terms εt can then be detailed. 9.5

Change-points in time series

A change-point in a time series may occur at an unknown time τ or at an unknown threshold η of the series. In both cases, Xt splits into two processes at the unknown threshold X1,t = Xt It and X2,t = Xt (1 − It )

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Applications to time series

FunctionalEstimation

175

with It = 1{Xt ≤ η} for a model with a change-point at a threshold of the series and It = 1{t ≤ τ } in a model with a time threshold. The pdimensional parameter vector α is replaced by two vectors α and β. Both change-points models are written equivalently, with a time change-point or a series change-points τη = sup{t; Xt ≤ η}, ητ = sup{x; (Xs )s∈[0,τ ] ≤ x}.

(9.7)

Xt = µ1 It + µ2 (1 − It ) + αT X1,t + β T X2,t + εt

(9.8)

With a change-point, the auto-regressive model AR(p) is modified as T

T

where Xt = µ+α X1,t +β X2,t +εt with X1,t = Xt It and X2,t = Xt (1−It ) for a model without change-point in the mean. Considering first that the change-point is known, the parameters are µ, or µ1 and µ2 , α, β and σ 2 . As t tends to infinity, a change-point at an integer time τ is denoted [γt] and sums of variables up to τ are increasing with t. For the auto-regressive process of order 1 with a change-point in time, this equation yields a twophase sample-path t X t Xt,α = mα + α (X0 − mα ) + αt−k εk , t ≤ τ, k=1

Xt,β = mβ + β

t−τ

(Xτ,α − mα ) +

t−τ X

β t−τ −k εk+τ , t > τ,

k=1

Pt or mβ = µ(1 − β)−1 . With α = 1, Xt,α = X0 + (t − 1)µ + k=1 εk and Pt−τ with β = 1 and t > τ , Xt,β = Xτ,α + (t − k − 1)µ + k=1 εk+τ . Let θ be the vector of parameters α, β, mα , mβ , γ. The time τ corresponds either to a change-point of the series or a stopping time defined by (9.7) for a change-point at a threshold of the process X and the indicator Ik relative to an unknown threshold τη of Xt−k is denoted Ik,τ . Pt (Ik−1,τ Xk−1 − m b α,τ )(Ik,τ Xk − m b α,τ ) α bt,τ = k=1 Pτ , 2 b α,τ ) k=1 (Ik−1 Xk−1 − m Pt ((1 − Ik−1,τ )Xk−1 − m b β,t )((1 − Ik,τ )Xk − m b β,t ) βbt,τ = k=1 , Pt 2 ((1 − I )X − m b ) k−1,τ k−1 β,t k=1 where the estimators of mα = (1 − α)−1 µ and mβ = (1 − β)−1 µ are equivalent to ¯τ , m b α,τ = X k X −1 ¯ m b β,τ +k = Xτ,k = k Xτ +j , for t = τ + k ≥ τ, j=1

¯τ − α ¯ τ − βbt,τ X ¯ τ,t if |α| and |β| 6= 1. and µ bt = X bt,τ X

January 31, 2011

176

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

The estimator of the change-point parameter minimizes with respect to τ the mean squared error of estimation. For t > τ , consider the estimation errors εbτ,k = Xk − m b αbt,τ ,τ − α bt,τ Xk−1 if k ≤ τ and εbt,k = Xk − m b βbt,τ ,t − 2 βbt,τ Xk−1 if k > τ . The variance σ and the change-point parameter are estimated by

σt2 (θ) = τ −1

τ X

k=1

εb2τ,k + (t − τ )−1

γ bt = arg min σ bt2 (τ ).

t X

k=τ +1

εb2t,k ,

τ ∈[0,t]

The change-point estimator is approximated by τ 1 X γ bt = arg min t1/2 { (Xk − µα − αXk−1 )2 τ ∈[0,t] τ k=τ0 +1

−

1 t−τ

τ X

(Xk − µβ − βXk−1 )2 − γ0 } + op (1),

k=τ0 +1

γ bt − γ0 is independent of the estimators of the parameter vector ξbt of the regression and all estimators converge weakly to limits bounded in probability. Consider the model (9.8) of order 1 with a change-point at a threshold η of the series, with the equivalence (9.7) between the chronological change-point model and the model for a series crossing the threshold η at consecutive random stopping times τ1 = inf{k ≥ 0 : Ik = 0} and τj = inf{k > τj−1 : Ik = 0}, j ≥ 1. The series have similar asymptotic behaviour starting from the first value of the series which goes across the threshold η at time sj = inf{k > τj−1 : Ik = 1} after τj−1 . The estimators of the parameters in the first phase of the model are restricted to the set of random intervals [sj , τj ] where Xt stands below η, for the second phase the observations are restricted to the set of random intervals ]τj−1 , sj [ where X remains above η. The time τj are stopping times of the series defined for t > sj−1 by  if |α| < 1,   mα + Ssj−1 ,t−sj−1 ,α + op (1), Xt = Xsj−1 + (t − sj−1 − 1)µ + Ssj−1 ,t−sj−1 ,1 , si α = 1,   mα + αt−sj−1 (Xsj−1 −1 − mβ ) + Ssj−1 ,t−sj−1 ,α , if |α| > 1 and the sj are stopping times defined for t > τj−1 by  if |β| < 1,  mβ + Sτj−1 ,t−τj−1 ,β + op (1), Xt = Xτj−1 + (t − τj−1 − 1)µ + Sτj−1 ,t−τj−1 ,1 , if β = 1,  mβ + β t−τj−1 (Xτj−1 − mα ) + Sτj−1 ,t−τj−1 ,β if |β| > 1,

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

177

Applications to time series

The sequences t−1 τj and t−1 sj converge to the corresponding stopping times of the limit of Xt as t tends to infinity. The partial sums are therefore defined as sums over indices belonging to countable union of intervals [sj , τj ] and ]τj , sj+1 [, respectively, for the two phases of the model. Theirs limits are deduced from integrals on the corresponding sub-intervals, instead of sums of the errors on the interval (τ, τ0 ). The estimators of the parameters are still expressions of their partial sums. The results generalize to processes of order p with a possible change-point in each p component. The estimators and their weak convergences are detailed in Pons (2009). Change-points in nonparametric models for time series are estimated by replacing the estimators of the parameters by those of the functions of the models and only the expression of the errors εk determines its estimator. With a change-point at an unknown time τ0 in the nonparametric model (9.3), it is written Xt = Iτ,t m1 (Xt−1 ) + (1 − Iτ,t )m2 (Xt−1 ) + σεt . For every x of IX , the two regression functions are estimated using a kernel estimator with the same bandwidth h for m1 and m2 Pt

m b 1,t,h (x, τ ) = Pi=1 t Pt

Kh (x − Xi )(1 − Iτ,i )Yi

i=1

m b 2,t,h (x, τ ) = Pi=1 t

Kh (x − Xi )(1 − Iτ,i )

Kh (x − Xi )Ii,τ Yi

i=1

Kh (x − Xi )Ii,τ

,

.

The behaviour of the estimators m b 1,t,h and m b 2,t,h is the same as in the model where τ0 is known, and it is the behaviour described in Section 9.1. The variance σ 2 is estimated by 2 σ bτ,t,h = t−1

t X i=1

{Yi − (Iτ,i )m b 1,t,h (Xi , τ ) − (1 − Iτ,i ){m b 2,t,h (Xi , τ )}2

at the estimated τ . The change-point parameter τ is estimated by minimization of the error of the model with a change-point at τ 2 τbt,h = arg min σ bτ,t,h τ ≤t

and the functions m1 and m2 by m b k,t,h (x) = m b k,t,h (x, τbt,h ), for k = 1, 2. Let γ = [T −1 τ ] and the corresponding change-point time τγ = T γ, let

January 31, 2011

17:17

178

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

m = (m1 , m2 ) with true functions m0 , and let σt2 (m, γ) = t−1

t X i=1

{Yi − (Iτγ ,i )m1 (Xi ) − (1 − Iτ,i )m2 (Xi )}2

be the mean squared error for parameters (m, τ ). The difference of the error from its minimal is lt (m, τ ) = σt2 (m, τ ) − σt2 (m0 , τ0 ) t X  = t−1 {Yi − Iτ,i m1 (Xi ) − (1 − Iτ,i )m2 (Xi )}2 i=1

− {Yi − Iτ0 ,i m10 (Xi ) − (1 − Iτ0 ,i )m20 (Xi )}2

=t

−1

t X  i=1

= t−1

{Iτ,i m1 (Xi ) − Iτ0 ,i m10 (Xi )}



(9.9)

2 + {(1 − Iτ,i )m2 (Xi ) − (1 − Iτ0 ,i )m20 (Xi )} ,

t X i=1

[{(m1 − m10 )(Xi )Iτ0 ,i − (m2 − m20 )(Xi )(1 − Iτ0 ,i )}2

+ {(Iτ,i − Iτ0 ,i ))(m1 − m2 )(Xi )}2 ]{1 + o(1)}.

It converges a.s. to l(m, τ ) = Eα (m1 − m10 )2 (X) + Eβ (m2 − m20 )2 (X) + |τ − τ0 |E(m1 − m2 )2 (X) which is minimal for (m0 , τ0 ), and the estimator τbnh minimizes lt (m b nh , τ ). The a.s. consistency of the regression estimators m b nh = (m b 1nh , m b 2nh ) and lt (m, τ ) imply that γ bt,h = [t−1 τbt,h ] is an a.s. consistent estimator of γ0 in ]0, 1[. It follows that the estimator m b nh (x) = m b 1nh (x)Iτbt,h + m b 2nh (x)(1 − Iτbt,h )

of the regression function m0 (x) = m10 (x)Iτ0 + m20 (x)(1 − Iτ0 ) is a.s. uniformly consistent and the process (th)1/2 (m b th − m0 ) converges weakly under Pm0 to a Gaussian process Gm on IX , with mean and covariances zero and with variance function Vm (x) = κ2 V ar(Y |X = x). For the weak convergence of the change-point estimator, let kϕkX be the L2 (FX )-norm of a function ϕ on IX , ρ(θ, θ0 ) = (|γ −γ 0 |+km−m0 k2X )1/2 0 the distance between θ = (mT , γ)T and θ0 = (m T , γ 0 )T and let Vε (θ0 ) be a neighbourhood of θ0 with radius ε for the metric ρ. The quadratic function lt (m, τ ) defined by (9.9) converges to its expectation l(m b th , τbth ) = 0(km b nh − m0 k2X + |b τnh − τ0 |).

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Applications to time series

FunctionalEstimation

179

The process is bounded in the same way lt (m, τ ) = [t−1

t X i=1

+ t−1

{(m1 − m10 )2 (Xi )Iτ0 ,i + (1 − Iτ0 ,i )(m2 − m20 )2 (Xi )}

t X i=1

(Iτ,i − Iτ0 ,i )2 (m1 − m2 )2 (Xi )]{1 + o(1)}.

it is denoted lt = (l1t + l2t ){1 + o(1)}. The process Wt (m, γ) = t1/2 (lt − l)(m, τγ ) is a Op (1). The estimator m b th is a local maximum likelihood estimators of the nonparametric regression functions and the estimator of the change-point is a maximum likelihood estimator. The variable l1t (m b th ) converges to l1 (m0 ) = 0 and l2t (m b th , τbγt ) converges to zero with the same rate if the convergence rate of γ bt is the same as m b th . We obtain the next bounds.

Lemma 9.1. For every ε > 0, there exists a constant κ0 such that E sup(m,γ)∈Vε (τ0 ) lt (m, τγ ) ≤ κ0 ε2 and 0 ≤ l(m, τγ ) ≤ κ0 ρ2 (θ, θ0 ), for every θ in Vε (τ0 ). Lemma 9.2. For every ε > 0, there exist a constant κ1 such that E sup(m,γ)∈Vε (τ0 ) Wt (m, γ) ≤ κ1 ρ(θ, θ0 ). The lemmas imply that for every ε > 0 lim sup P0 (tht |b γtht − γ0 | > A) = 0.

t→∞,A→∞

The proof is similar to Ibragimov and Has’minskii’s (1981) for a changepoint of a density. It implies that lt (θbth ) = (l1t + l2t )(θbth ) + op (1) uniformly. For the weak convergence of (th)1/2 (b γth − γ0 ), let Un = {u = (uTm , uγ )T : um = (th)−1/2 (m − m0 ), uγ = (th)−1 (γ − γ0 )}

A be a bounded set. For every A > 0, let Uth = {u ∈ Ut ; kuk2 ≤ A}. A Then for every u = (um , uγ ) belonging to Uth , θt,u = (mt,u , γt,u ) with −1/2 −1 mt,u = m0 + (th) um and γt,u = γ0 + (th) uγ . The process Wt defines a map u 7→ Wt (θt,u ).

Theorem 9.1. For every A > 0, the process Wt (θ) develops as Wt (θ) = W1t (m) + W2t (γ) + op (1), where the op is uniform on UtA , as t tends to infinity. Then change-point estimator of γ0 is asymptotically independent of the estimators of the regression functions m b 1th and m b 2th .

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

180

Functional estimation for density, regression models and processes

Proof.

For an ergodic process, the continuous part l1t of lt converges to l1 (m) = γ0 km1 − m01 k2F1X + (1 − γ0 )km2 − m02 k2F2X

and the continuous part of Wt is approximated by W1t (m) = t1/2 (l1t − A l1 )(m). On Uth , it is written Z Z W1t (m) = {γ0 (m1 − m01 )2 dν1t + (1 − γ0 ) (m2 − m02 )2 dν2t },

where νkt = t1/2 (Fbkt − Fk0 ) is the empirical processes of the series in phase k = 1, 2, with the ergodic distributions Fk0 of the process. The discrete part of Wt is approximated by W2t (γ) = t1/2 (l2t − l2 )(γ) P where l2t = t−1 ti=1 (Iτ,i − Iτ0 ,i )2 (m10 − m20 )2 (Xi ) + op (|τ − τ0 |) and the sum is developed with the notation ai = (m10 − m20 )2 (Xi ) t−1

t X i=1

=

(Iτ,i − Iτ0 ,i )2 (m10 − m20 )2 (Xi ) = τ0 X

τ0 t X 1 X { (1 − Iτ,i )ai + Iτ,i ai } t i=1 i=1+τ

τth X

0

1 {1{τth <τ0 } ai + 1{τ0 <τth } ai } t i=1+τ i=1+τ

1 = {1{τth <τ0 } t

th

τ0 X

0

τ0 +[h−1 uγ ]

X

ai + 1{τ0 <τth }

i=1+τ0

i=1+τ0 −[h−1 uγ ]

ai }.

Then l2t converges uniformly to Z Z 2 l2 (γ) = |γ − γ0 |{ (m10 − m20 ) dF2X − (m10 − m20 )2 dF1X } X

X

as ht tends to zero. Let ντ,kt , k = 1, 2, be the empirical processes reduced to the variables between τ0 and τ0 + [h−1 uγ ], according to the sign of uγ , and normalized by |τ − τ0 |1/2 . Then the process W2t is approximated by Z Z 2 −1/2 W2t (γ) = |γ − γ0 | { (m10 − m20 ) dν2t − (m10 − m20 )2 dντ,1t }.  X X The limit of the process W1t is a Gaussian distribution Gm . The estimator of the change-point satisfies ubγt = th(b γt − γ0 ) = arg min W2t (uγ ) + op (1). R

Its convergence rate is th and the asymptotic behaviour of the estimator ubγt is deduced from Theorem 9.1 and the continuity of the minimum. Theorem 9.2. The change-point estimator th(b γth − γ0 ) converges weakly to arg minu∈R Q(u) and it is a Op (1).

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Applications to time series

9.6

FunctionalEstimation

181

Exercises

(1) Let Xt be an AR(1) process Xt = αXt−1 + εt , with an initial random variable X0 with mean µ0 and variance σ02 . Write it as a moving average and calculate the mean and the variance of the MA process. (2) Consider the moving average process Xt = εt − θ1 εt−1 − θ2 εt−2 with independent and identically distributed errors with mean zero and variance σ 2 . Calculate the variance of Xt and the covariances between Xt and Xt−k and their empirical estimators. Extend to a model MA(q) of order q. (3) Let Xt = pk,t + θεt−1 + σεt , where pk,t is a polynomial of degree k. Define moving average estimators of pk,t . (4) Let Xt be an ARMA(2,2) process Xt = µt +αXt−1 +βXt−2 +εt −θεt−1 , t in N, with a constant or a varying mean and with noise variables with first moments (0, σ 2 ). Define estimators for the parameters α, β and σ 2 and describe the series as infinite combinations of their past values. (5) Invert the AR(1) model for Xt as a moving average model depending on X0 and the noises and identify the parameters of both model in order to estimate them from Section 9.1. (6) Define maximum likelihood estimators for the parameters α, β and σ 2 of the model Xt = µt (θ) + αXt−1 + βXt−2 + σεt , t in N, with a parametric varying mean, independent identically distributed normal errors εi . For a polynomial trend µt (θ), use differences of the series before the estimation of α and σ 2 .

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Chapter 10

Appendix

10.1

Appendix A

The moments of derivatives of the kernel estimator for the regression function are presented in Chapter 3, here the proofs are detailed. The variance (1) (1) (1) −1 of m b n,h (x) = fbn,h (x){b µn,h (x) − m b n,h (x)fbn,h (x)} is obtained by an approximation similar to (3.2) in Proposition 3.1 (1) (1) (1) V arm b (x) = f −2 [V ar{b µ (x) − m b n,h (x)fb (x)} X

n,h

+

n,h

(1) {µn,h (x) (1)

−

n,h

(1) mn,h (x)fn,h (x)}2 V (1)

− 2{µn,h(x) − mn,h (x)fn,h (x)} (1)

arfbn,h (x) (1)

× Cov{fbn,h (x), µ bn,h (x) − m b n,h (x)fbn,h (x)}] {1 + o(1)},

(1) (1) where the variances V arb µn,h (x) and V arfbn,h (x) are O((nh3 )−1 ), (1) (1) V arfbn,h (x) = O((nh)−1 ), E{fbn,h (x) − fn,h (x)}4 = O((nh3 )−1 ) and 4 −1 E{m b n,h (x) − mn, h(x)} = O((nh) ), (1) (1) (1) V ar{b µn,h (x) − m b n,h (x)fbn,h (x)} = V arb µn,h (x) (1)

(1)

(1)

+ V ar{m b n,h (x)fbn,h (x)} − 2Cov{b µn,h (x), m b n,h (x)fbn,h (x)},

(1) (1) V ar{m b n,h (x)fbn,h (x)} ≤ [E{m b n,h (x)}4 E{fbn,h (x)}4 ]1/2 (1)

(1) − E 2 {m b n,h (x)fbn,h (x)} = O((nh2 )−1 ) (1)

(1)

Therefore V ar{b µn,h (x)− m b n,h (x)fbn,h (x)} and V arm b n,h (x) are O((nh3 )−1 ). Proposition 10.1. (1)

(1)

−2 V arm b n,h (x) = fX V arb µn,h (x) + o((nh3 )−1 ) Z −2 = (nh3 )−1 {fX (x)w2 (x) K (1)2 + o(h)}.

183

January 31, 2011

17:17

184

10.2

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Appendix B

In Chapter 4, the bandwidth is a real function defined on IX and the normalized kernel is ϕn (x) = Khn (x) (x). Its derivative with respect to x is (1)

ϕ(1) n (x) =

1 x hn (x) d K( )− 2 K (x) hn (x) dx hn (x) hn (x) hn (x) (1)

=

(1)

hn (x) 1 x hn (x) K (1) ( ){1 − x }− 2 Khn (x) (x). h2n (x) hn (x) hn (x) hn (x)

As khn k tends to zero, a Taylor expansion of the density in a neighborhood of x and Lemma 2.1 yield Z hn (x) (3) (1) ϕ(1) (x) − hn (x)m2K { f (x) n (x − u)f (u) du = f 2 (2) + 2h(1) (x) + o(khn k + kh(1) n (x)f n k)}.

The expectation of the quadratic variations |fbn,h (x) − fbn,h (y)|2 develops as the sum Z n−1 {Khn (x) (x − u) − Khn (y) (y − u)}2 f (u) du + (1 − n−1 ){fn,hn (x) − fn,hn (y)}2 .

For an approximation of the first term, the Mean Value Theorem implies (1) Khn (x) (x − u) R − Khn (y) (y − u) = (x − y)ϕn (z2 − u) where z is between x and y, then {Khn(x) (x − u) − Khn (y) (y − u)} f (u) du is approximated by Z Z 2 −3 (z − u)f (u) du = (x − y) h (x){f (x) K (1)2 + o(khn k)}. (x − y)2 ϕ(1)2 n n −1 b Since h−1 n (x)|x| and hn (y)|y| are bounded by 1, the order of E|fn,h (x) − 2 2 −1 −1 b fn,h (y)| = O((x − y) n khn k if |xhn (y) − yhn (x)| ≤ 2hn (y)hn (x) and it is a sum of variances otherwise.

10.3

Appendix C

In the single index model studied in Chapter 7, the precise order of the (1) mean and variance of Vbn,h defined Section 7.2 requires expansions. The empirical mean squared error of the estimated function g, at fixed θ has the derivative n X (1) (1) Vbn,h (θ) = n−1 {Yi − gbn,h (θT Xi ; θ)} {b gn,h(θT Xi ; θ)} Xi . i=1

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

185

Appendix (1) Let Z = θT X at fixed θ. The mean of Vbn,h is (1)

(1)

Vn,h (θ) = EE[{g(Zi ) − gbn,h (Zi ; θ)} {b gn,h (Zi ; θ)} Xi | Xi ] (1)

(1)

= E[{bg(1) ,n,h gn,h (Zi ; θ) − Cov(b gn,h , gbn,h )(Zi ; θ)} Xi⊗2 ]. (1)

Its variance is n−1 V ar[{Yi − b gn,h (Zi ; θ)} {b gn,h (Zi ; θ)} Xi ] and, (1)

(1)

V ar[{Yi − gbn,h (Zi ; θ)} gbn,h (Zi ; θ)} = O(V ar{Yi gbn,h (Zi ; θ)}) (1)

+ O(V ar{b gn,h (Zi ; θ) gbn,h (Zi ; θ)})

(1)

(1)

V ar{Yi b gn,h (Zi ; θ)} = O(V ar{b gn,h (Zi ; θ)}) = O((nh3 )−1 ). (1)

The expansions (3.2) for gbn,h and (3.15) for b gn,h are written

 −1 {b gn,h − gn,h }(z) = fX (z) (b µn,h − µn,h )(z) b − g(z)(fX,n,h − fX,n,h )(z) + oL2 ((nh)−1/2 ) (1)

(1)

(1)

(1)

−1 {b gn,h − gn,h }(z) = fX (z)[(b µn,h − µn,h )(z)

(1) (1) − {b gn,h fbX,n,h − E(b gn,h fbX,n,h )}(z)

− g (1) (z) (fbX,n,h − fX,n,h )(z)] + oL2 ((nh3 )−1/2 ),

 (1) (1) (1) −1 gbn,h (z)fbX,n,h (z) = {fbX,n,h − fX,n,h }(z)[gn,h + fX (b µn,h − µn,h ) − g(fbX,n,h − fX,n,h ) ](z)  (1) −1 + fX,n,h (z)[gn,h + fX (b µn,h − µn,h ) − g(fbX,n,h − fX,n,h ) ](z) + oL2 ((nh)−1/2 )  (1) (1) (1) −1 E{b gn,h fbX,n,h }(z) = fX (z) E{fbX,n,h − fX,n,h }(z) (b µn,h − µn,h ) (1) − g(fbX,n,h − fX,n,h ) (z) + fX,n,h (z)gn,h (z) (1)

+ oL2 (nh2 ) = fX,n,h (z)gn,h (z) + O((nh2 )−1 ).

Then (1) (1) (1) (1) {b gn,h fbX,n,h − E(b gn,h fbX,n,h )}(z) = {fbX,n,h − fX,n,h }(z)([gn,h  −1 + fX (b µn,h − µn,h ) − g(fbX,n,h − fX,n,h ) ](z)  (1) −1 + fX,n,h (z)fX (b µn,h − µn,h ) − g(fbX,n,h − fX,n,h ) (z)) + oL2 ((nh)−1/2 )

January 31, 2011

17:17

186

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

(1) (1) (1) (1) (1) (1) −1 {b gn,h − gn,h }(z) = fX (z)[(b µn,h − µn,h )(z) − (fbX,n,h − fX,n,h )(z)  −1 × ([gn,h + fX (b µn,h − µn,h )− g(fbX,n,h − fX,n,h ) ](z)  (1) −1 + [fX,n,h fX (b µn,h − µn,h ) − g(fbX,n,h − fX,n,h ) ](z))

− g (1) (z) (fbX,n,h − fX,n,h )(z)] + oL2 ((nh)−1/2 ) (1) (1) −2 Cov(b gn,h , gbn,h )(z) = −fX (z)[Cov(b µn,h , µ bn,h ) − g (1) Cov(b µn,h , fbX,n,h ) (1) − gCov(fbX,n,h , µ bn,h ) − gg (1) V arfbX,n,h − gn,h

(1) (1) − gnh Cov(fbX,n,h , µ bn,h ) + ggnh Cov(fbX,n,h , fbn,h )

(1) (1) (1) + fn,h V arb gn,h + E{(fbn,h − fn,h )(b gn,h − gn,h ]

+ o((nh2 )−1 ) (1) where the main term has the same order as Cov(b µn,h , µ bn,h ), namely O((nh2 )−1 ) and the variances and covariances for the terms without deriva(1) (1) tives are O((nh)−1 ). The product and E{b gn,h gbn,h }(z) = gn,h (z)gn,h (z) + 2 −1 O((nh ) ), then (1) (1) (1) V ar{b gn,h b gn,h }(z) = E[{b gn,h (z) − gn,h (z)}2 {b gn,h (z) − gn,h (z)}2 (1)

(1)2

2 + gn,h (z)V ar{b gn,h (z)} + gn,h (z)V ar{b gn,h (z)} (1)

(1)

≤ E[{b gn,h (z) − gn,h (z)}4 ]1/2 E[{b gn,h (z) − gn,h (z)}4 ]1/2 (1)

(1)2

2 + gn,h (z)V ar{b gn,h (z)} + gn,h (z)V ar{b gn,h (z)}.

Finally, E[{b gn,h (z) − gn,h (z)}4 ] = O((nh)−1 ), (1)

(1)

E[{b gn,h (z) − gn,h (z)}4 ] = O((nh3 )−1 ), (1)

V ar{b gn,h gbn,h }(z) = O((nh3 )−1 )

(1)

therefore the main term of the variance of the product gbn,h (z)b gn,h (z) is (1)

(1)

2 gn,h (z)V ar{b gn,h (z)} = O((nh3 )−1 ) and the mean Vn,h (θ) is a O(h2 ) + 2 −1 O((nh ) ).

Lemma 10.1. For 1 ≤ i 6= j ≤ n E{ϕ(Xi ) − ϕ(Xj )}Kh (Xi − Xj ) =

h2 m2K (1) E(fX ϕ(2) + fX ϕ(1) )(X) + r1n , 2 Z

E{ϕ(Xi ) − ϕ(Xj )}Kh2 (Xi − Xj ) = E[{fX ϕ(1) }(X)]

zK 2 (z) dz + r2n ,

where r1n = o(h2 ) and Z (1) r2n = h z 2 K 2 (z) dzE[{fX ϕ(2) + 2ϕ(1) fX }(X)] + o(h2 ).

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Appendix

Proof.

FunctionalEstimation

187

Using Lemma 2.1 and an expansion for |Xi − Xj | ≤ h

E{ϕ(Xi ) − ϕ(Xj )}Kh2 (Xi − Xj ) Z Z = {ϕ(x) − ϕ(y)}Kh2 (x − y)fX (x)fX (y) dx dy Z Z z 2 h2 (2) (1) = h−1 {hzϕ(1) + ϕ + o(h)}{fX + hzfX + o(h2 )} 2 × K 2 (z) dFX dz. The higher orders are obtained by further terms in the expansion. 10.4



Appendix D

The ergodicity and mixing conditions for the convergence of functionals of a process (Xt )t≥0 or a sequence of dependent variables (Xi )1≤i≤n are expressed in following conditions A and B. Let (Xi )1≤i∈N be a sequence of dependent random variable with values in a metric space (X, A, µ) and such that EXi2 is finite for every i. Let Mj1 and M∞ j+k be the σ-algebras generated by the sub-samples (Xi )1≤i≤j and, respectively, (Xi )j+k≤i . The sequence (Xi )1≤i is ϕ-mixing if there exists a real sequence (ϕk )k∈N converging to zero as k tends to infinity and such that the conditional probabilities of the sample (Xi )i∈N satisfy sup{|P (B|A) − P (B)|; A ∈ Mj1 , B ∈ M∞ j+k , j, k ∈ N} < ϕk . A1 The sequence (Xi )i≥1 is ergodic if there exists a probability ν on (X, A, µ) such that for every real bounded function φ on X n

1X P φ(Xi ) −−−−→ Eν φ(X1 ). n→∞ n i=1

B1 The sequence (Xi )i≥0 is ϕ-mixing with a sequence (ϕk )k≥1 satisfying P 2 1/2 < ∞. k≥1 (k + 1) ϕk

The ϕ-mixing property and condition B1 are defined in Billingsley (1968), they imply the weak convergence of the normalized variable P n1/2 { n1 ni=1 φ(Xi ) − Eν φ(X1 )} to a normal variable σϕ2 N (0, 1).

Let (Xt )t≥0 be a time indexed process such that for every t > 0, Xt is a random variable in a metric space (X, A, µ) and EXt2 is finite. Let Ms0 and M∞ s+t be the σ-algebras generated by the sample-paths of the process observed on the time intervals [0, s] and [s + t, ∞[ respectively,

January 31, 2011

188

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Ms0 = σ{(Xu )0≤u≤s } and M∞ s+t = σ{(Xu )u≥s+t }, with s and t > 0. The sequence (Xt )t≥0 is ϕ-mixing if there exists a sequence (ϕt )t≥0 converging to zero as t tends to infinity and such that the marginal distributions of the process (Xt )t≥0 satisfy sup{|P (B|A) − P (B)|; A ∈ Ms0 , B ∈ M∞ s+t , s, t ≥ 0} < ϕt . The ergodicity and mixing conditions are modified as follows. A2 The process (Xt )t≥0 is ergodic if there exists a probability ν on (X, A, µ) such that for every real bounded function φ defined on the space X Z Z 1 t P φ(Xs ) ds −−−→ φ(x)dν(x). t→∞ t 0 X B2 The process (Xt )t≥0 is ϕ-mixing with a sequence (ϕt )t≥0 satisfying R∞ 2 1/2 dt < ∞. 0 (t + 1) ϕt The ergodic property is strengthened to allow the convergence of functionals of the joint distributions of the process at several observation times.

A2’ The process (Xt )t≥0 is ergodic if for every integer k there exists a probability νk on (X⊗k , Ak , µ), with the Borel σ-algebra Ak on X⊗k , such that for every real bounded function φ defined on the space X⊗k Z 1 P φ(Xs1 , . . . , Xsk ) ds1 , . . . , dsk −−−→ Eνk φ(X1 , . . . , Xk ). t→∞ tk [0,t]k The expectation withR respect to the limit νk is an integral on X⊗k , Eνk φ(X1 , . . . , Xk ) = X⊗k φ(x1 , . . . , xk ) νk (dx1 , . . . , dxk ). For every integer k > 0, this property is the consistency of the k-th moment of the process X. For k = 2, it implies the convergence in probability of the covariance function of the process R t X. Condition B2 entails the weak convergence of the process t1/2 {t−1 0 φ(Xs ) ds−Eν φ(X1 )} to a normal process σϕ2 W1 . The ϕ-mixing property A1 and condition B1 imply the consistency and the weak convergence of the partial sum of the sequence of variables Zk = X(n−1 k) − X(n−1 (k − 1)), 1 ≤ k ≤ n. Under condition A1, the proP[nx] cess Sn (x) = n−1/2 k=1 Zk , x in [0, 1], converges weakly to the Brownian motion defined on [0, 1] and with covariance function C(s, t) = s ∧ t.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Notations

1A a.s. (Bt )t≥0 Cov(Xi , Xj ) Cs (I) ∆f (x, y) EX FX (x) f (s) FbX,n m−1 Hα,M Kh =R h−1 K(h−1 ·) · dF Λ = 0 (1−F −)

N = (Nt )t≥0 νbn Ln et )t≥0 (N Ω ρ(i, j) V arX Vbn,h

indicator of a set A, almost surely, Brownian motion, covariance of Xi and Xj : E(Xi − EXi )(Xj − EXj ), class of real functions on I, having bounded and continuous derivatives of order s, variation of f : f (y) − f (x), R expectation (or mean) of a variable X: x dFX (x), probability of the random set {X ≤ x}, derivatives of order s for a function f , empirical distribution function, either 1/m or inverse of a monotone function m, class of real functions f such that ∀x and y, |f (s) (x) − f (s) (y)| ≤ M |x − y|α−s , s = [α], normalized kernel with bandwidth h, cumulative hazard function for the distribution function F , point process, empirical process n1/2 (FbX,n − FX ), partial likelihood of N at t such that Nt = n, predictable compensator of a point process N , sample space, correlation of variables Xi and Xj : Cov(Xi , Xj ){V arXi V arXj }−1/2 , variance of a variable X: E(X − EX)2 , empirical mean squared error for a regression,

189

January 31, 2011

190

17:17

World Scientific Book - 9in x 6in

Functional estimation for density, regression models and processes

(Xt )t≥0 (Wt )t≥0 Zh

continuously observed process or time series, Gaussian process, {z ∈ Z; supz0 ∈∂Z kz − z 0 k ≥ h}, with the frontier ∂Z of Z.

FunctionalEstimation

January 31, 2011

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Bibliography

Andersen, P. and Gill, R. D. (1982). Cox’s regression model for counting processes: a large sample study, Ann. Statist. 10, pp. 1100–1120. Bahadur, R. R. (1966). A note on quantiles in large samples, Ann. Math. Statist. 37, pp. 577–580. Barlow, R., Bartholomew, D. J., Bremmer, J. and Brunk, H. D. (1972). Statistical Inference under Order Restrictions (Wiley, New York). Beran, R. J. (1972). Upper and lower risks and minimax procedures, Proceedings of the sixth Berkeley Symposium on Mathematical Statistics, L. Lecam, J. Neyman and E. Scott (eds) , pp. 1–16. Bickel, P. and Rosenblatt, P. (1973). On some global measures of the deviations of density functions estimates, Ann. Statist. 1, pp. 1071–1095. Billingsley, P. (1968). Convergence of probability measures (Wiley, New York). Bosq, D. (1998). Nonparametric Statistics for Stochastic Processes (2nd edition, Springer, New York). Bowman, A. W. (1984). An alternative method of cross-validation for the smoothing of density estimates, Biometrika 71, pp. 353–360. Bowman, A. W. and Azalini, A. (1997). Applied Smoothing Techniques for Data Analysis. The Kernel Approach with S-Plus Illustrations (Oxford Statistical Science Series 18). Breslow, N. and Crowley, J. (1974). A large sample study of the life table and product limit estimates under random censorship, Ann. Statist. 2, pp. 437– 453. Bretagnolle, J. and Huber, C. (1981). Estimation de densit´es : risque minimax, Z. Wahrsch. Verw. Geb. 47, pp. 119–139. Brillinger, D. R. (1981). Time Series Data Analysis and Theory (Holt, Rinehart and Winston, New York). Chaudhuri, P. (1991). Nonparametric estimates of regression quantiles and their local Bahadur representation, Ann. Statist. 19, pp. 760–777. Chernoff, H. (1964). Estimation of the mode, Ann. Inst. Statist. Math. 16, pp. 31–41. Cox, D. R. and Oakes, D. (1984). Analysis of Survival Data (Chapman and Hall, London).

191

January 31, 2011

192

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Cox, R. D. (1972). Regression model and life tables, J. Roy. Statist. Soc. Ser. B 34, pp. 187–220. De Boor, C. (1978). A Practical Guide to Splines (Springer, New York). Deheuvels, P. (1977). Estimation non param´etrique de la densit´e par histogrammes g´en´eralis´es, Rev. Statist. Appl 25, pp. 5–42. Delecroix, M., H¨ ardle, W. and Hristache, M. (2003). Optimal smoothing in singleindex models, J. Multiv. Anal. 286, pp. 213–226. Devroye, L. (1983). The equivalence of weak, strong and complete convergence in l1 for kernel density estimates, Ann. Statist. 11, pp. 896–904. Dumbgen, L. and Rufibach, K. (2009). Maximum likelihood estimatio of a logconcave density and its distribution function: Basic properties and uniform consistency, Bernoulli 15, pp. 40–68. Dvoretski, A., Kiefer, J. and Wolfowitz, J. (1956). Asymptotic minimax character of the sample distribution functions and of the classical multinomial estimator, Ann. Math. Statist. 27, pp. 642–669. Eubank, R. (1977). Spline Smoothing and Nonparametric Regression (Dekker, New York). Fan, J. and Gijbels, I. (1996). Polynomial Modelling and Its Applications (Chapman and Hall CRC). Ghosh, J. K. (1966). A new proof of the Bahadur representation of quantiles and an application, Ann. Math. Statist. 42, pp. 1957–1961. Gijbels, I. and Veraverbeke, N. (1988a). Almost sure asymptotic representation for a class of functionals of the product-limit estimator, Ann. Statist. 19, pp. 1457–1470. Gijbels, I. and Veraverbeke, N. (1988b). Weak asymptotic representations for quantiles of the product-limit estimator, J. Statist. Plann. Inf. 18, pp. 151–160. Groeneboom, P. (1989). Brownian motion with a parabolic dridt and airy functions, Probab. Theory Related Fields 81, pp. 79–109. Groeneboom, P., Jonkbloed, G. and Wellner, J. (2001). Estimation of a convex function: Characterization and asymptotic theory, Ann. Statist. 29, pp. 1653–1698. Groeneboom, P. and Wellner, J. (1990). Empirical processes (Burckhlder, Basel). Guyon, X. and Perrin, O. (2000). Identification of space deformation using linear and superficial quadratic variations, Statist. Prob. Lett. 47, pp. 307–316. Hall, P. (1981). Law of the iterated logarithm for nonparametric density estimators, Stoch. Proc. Appl. 56, pp. 47–61. Hall, P. (1984). Integrated square error properties of kernel estimators of regression functions, Ann. Statist. 12, pp. 241–260. Hall, P. and Huang, L.-S. (2001). Nonparametric kernel regression subjet to monotonicity constraints, Ann. Statist. 29, pp. 624–647. Hall, P. and Johnstone, I. (1992). Empirical functionals and efficient smoothing parameter selection, J. Roy. Statist. Soc. Ser. B 54, pp. 475–530. Hall, P. and Marron, J. M. (1987). Estimation of integrated squared density derivatives, Statist. Probab. Lett. 6, pp. 109–115.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Bibliography

FunctionalEstimation

193

H¨ ardle, W. (1990). Applied Nonparametric Regression (Cambridge University Press). H¨ ardle, W. (1991). Smoothing Methods in Statistics (Cambridge University Press, UK). H¨ ardle, W., Hall, P. and Ihimura, H. (1993). Optimal smoothing in single-index models, Ann. Statist. 21, pp. 157–178. H¨ ardle, W., Hall, P. and Marron, J. M. (1988). How far are asymptotically choosen regression smoothers from their optimum ? (with discussion), J. Amer. Statist. Soc. 6, pp. 109–115. Hirstache, M., Juditsky, A. and Spokoiny, V. (2001). Direct estimation of the index coefficients in a single-index model, Ann. Statist. 29, pp. 595–623. Ibragimov, I. and Has’minskii, R. (1981). Statistical Estimation: Asymptotic Theory (Springer, New York). Ichimura, H. (1993). Semi-parametric least squares and weighted sls estimation of single-index models, J. Econometrics 58, pp. 71–120. Jones, M. C., Marron, J. S. and Park, B. U. (1991). A simple root n bandwidth selector, Ann. Statist. 19, pp. 1919–1932. Kaplan, M. and Meier, P. A. (1958). Nonparametric estimator from incomplete observations, J. Am. Statist. Ass. 53, pp. 457–481. Khasminskii, R. Z. (1992). Topics in nonparametric estimation (American Mathematical Society). Kiefer, J. (1972). Iterated logaritm analogues for samples quantiles when pn ↓ 0, Proceedings of the sixth Berkeley Symposium on Mathematical Statistics, L. Lecam, J. Neyman and E. Scott (eds) , pp. 227–244. Kiefer, J. and Wolfowitz, J. (1976). Asymptotically minimax estimation of concave and convex distribution functions, Z. Wahrsch. Verw. Gebiete 34, pp. 73–85. Kim, J. and Pollard, D. (1990). Cube root asymptotics, Ann. Statist. 18, pp. 191–219. Lecam, L. (1990). On the asymptotic theory of estimation (Springer, New York). Lo, S.-H. and Singh, K. (1986). The product-limit estimator and the bootstrap: some asymptotic representations, Prob. Theor. Rel. Fields 71, pp. 455–465. Mammen, E. (1991). Estimating a smooth monotone regression function, Ann. Statist. 19, pp. 724–740. Marron, J. S. (1988). Improvment of a data based bandwidth selector, Preprint Univ. North Carolina, Chapel Hill , pp. 1–31. Messer, K. (1991). A comparison of a spline estimate to its equivalent kernel estimate, Ann. Statist. 19, pp. 817–829. Meyer, M. and Woodroof, M. (2000). On the degrees of freedom in shapedrestricted regression, Ann. Statist. 28, pp. 1083–1104. Nadaraya, E. A. (1964). On estimating regression, Theor. Probab. Appl. 9, pp. 141–142. Nadaraya, E. A. (1989). Nonparametric estimation of probability densities and regression curves (Kluwer Academic Publisher, Boston). Ould Sa¨ıd, E. (1997). A note on ergodic processes prediction via estimation of the conditional mode, S. J. Statist. 24, pp. 231–239.

January 31, 2011

194

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Parzen, E. A. (1962). On the estimation of probability density and mode, Ann. Math. Statist. 33, pp. 1065–1076. Parzen, E. A. (1979). Nonparametric statistical data modeling, J. Amer. Statist. Assoc. 74, pp. 105–131. Perrin, O. (1999). Quadratic variations for gaussian processes and application to time deformation, Stoch. Proc. Appl. 82, pp. 293–305. Pin¸con, C. and Pons, O. (2006). Nonparametric estimator of a quantile function for the probability of event with repeated data, Dependence in Probability and Statistics, Lect. N. Statist. 17. Springer, New York , pp. 475–489. Plancherel, M. and Rotach, W. (1929). Sur les valeurs asymptotiques des polynˆ omes d’Hermite, Commentarii Mathem. Helvet. 1, pp. 227–254. Pons, O. (1986). Vitesse de convergence des estimateurs ` a noyau pour l’intensit´e d’un processus ponctuel, Statistics 17, pp. 577–584. Pons, O. (2000). Nonparametric estimation in a varying-coefficient Cox model, Mathematical Methods of Statistics 9, pp. 376–398. Pons, O. (2007a). Estimation of absolutely continuous distributions for censored variables in two-samples nonparametric and semi-parametric models, Bernoulli 13, pp. 92–114. Pons, O. (2007b). Estimation of the distribution function of one and two dimensional censored variables or sojourn times of markov renewal processes, Comm. Statist.–Theory Methods 55, pp. 1–18. Pons, O. (2009a). Estimation and tests in distribution mixtures and change-points models (O. Pons, Viroflay, F). Pons, O. (2009b). Nonparametric Estimation for Renewal and Markov Processes (O. Pons, Viroflay, F). Pons, O. and de Turckheim, E. (1987). Estimation in Cox’s periodic model with a histogram-type estimator for the underlying intensity, Scand. J. Statist. 14, pp. 329–345. Prakasa Rao, B. L. S. (1983). Nonparametric Functional Estimation (Academic Press, New York). Rebolledo, R. (1978). Sur les applications de la th´eorie des martingales ` a l’´etude statistique d’une famille de processus ponctuels, Journ´ee de Statistique des Processus Stochastiques, Lecture Notes in Mathematics 636, pp. 27–70. Rice, J. and Rosenblatt, M. (1983). Smoothing splines: regression, derivatives and deconvolution, Ann. Statist. 11, pp. 141–156. Robinson, P. M. (1991). Automatic frequency domain inference on semiparametric and nonparametric models, Econometrika 59, pp. 1329–1363. Rosenblatt, M. (1956). Remarks on some nonparametric estimates of a density function, Ann. Math. Statist. 27, pp. 832–837. Rosenblatt, M. (1975). A quadratic measures of deviation of two-dimensional density estimates, Ann. Statist. 3, pp. 1–14. Rudemo, M. (1982). Empirical choice of histograms and kernel density estimators, Scand. J. Statist. 9, pp. 65–78. Ruppert, D., Wand, M. P. and Carroll, R. J. (2003). Semiparametric Regression (Cambridge University Press, UK).

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Bibliography

FunctionalEstimation

195

Schoenberg, I. (1964). Spline functions and the problem of graduation, Proc. Nat. Acad. Sci. USA 52, pp. 947–950. Schuster, E. F. (1969). Estimation of a probability density function and its derivatives, Ann. Math. Statist. 40, pp. 1187–1195. Scott, D. W. (1992). Multivariate density estimation: theory, practice, and visualization (Wiley, New York). Sheather, S. J. and Marron, J. S. (1990). Kernel quantile estimators, J. Amer. Statist. Assoc. 85, pp. 410–416. Shorack, G. R. and Wellner, J. A. (1986). Empirical processes and applications to statistics (Wiley, New York). Silverman, B. W. (1978a). On a gaussian process related to multivariate probability density estimation, Math. Proc. Cambridge Philos. Soc. 80, pp. 136–144. Silverman, B. W. (1978b). Weak and strong uniform consistency of the kernel estimate of a density and its derivatives, Ann. Statist. 6, pp. 17–184. Silverman, B. W. (1984). Spline smoothing: The equivalent variable kernel method, Ann. Statist. 12, pp. 898–916. Silverman, B. W. (1985). Some aspects of the spline smoothing approach to the nonparametric regression curve fitting, J. Roy. Statist. Soc. Ser. B 47, pp. 1–22. Simonoff, J. S. (1996). Smoothing Methods in Statistics (Springer-Verlag, New York). Singh, R. S. (1979). On necessary and sufficient conditions for uniform strong consistency of estimators of a density and its derivatives, J. Multiv. Anal. 9, pp. 157–164. Stieltjes, T.-J. (1890). Sur les polynˆ omes de Legendre, Ann. Fac. Sci. Toulouse, 1e s´erie 4 G, pp. 1–17. Stone, M. (1974). Cross-validation choice and assessment of statistical prediction (with discussion), J. Roy. Statist. Soc. Ser. B 36, pp. 111–147. Stute, W. (1982). A law of the logarithm for kernel density estimators, Ann. Probab. 10, pp. 414–422. van de Geer, S. (1993). Hellinger consistency of certain nonparametric maximum likelihood estimators, Ann. Statist. 21, pp. 14–44. van de Geer, S. (1996). Applications of empirical process theory (Cambridge university press). van der Vaart, A. and van der Laan, M. (2003). Smooth estimation of a monotone density, Statistics 37, pp. 189–203. van der Vaart, A. and Wellner, J. A. (1996). Weak convergence and Empirical Processes (Springer, New York). Wahba, G. (1977). Optimal smoothing of density estimates, Classification and clustering, (ed.) J. Van Ryzin. Academic Press, New York , pp. 423–458. Wahba, G. and Wold, S. (1975). A completely automatic french curve: Fitting spline functions by cross-validation, Comm. Statist. 4, pp. 1–17. Walker, A. M. (1971). On the estimation of a harmonic component in a time series with stationary independent residuals, Biometrika 58, pp. 26–36. Wand, M. P. and Jones, M. C. (1995). Kernel Smoothing (Chapman and Hall, CRC).

January 31, 2011

196

17:17

World Scientific Book - 9in x 6in

FunctionalEstimation

Functional estimation for density, regression models and processes

Watson, G. S. (1964). Smooth regression analysis, Sankhy˜ a A26, pp. 359–372. Watson, G. S. and Laedbetter, M. (1963). On the estimation of a probability density, Ann. Math. Statist. 34, pp. 480–491. Watson, G. S. and Laedbetter, M. R. (1964). Hazard analysis, Biometrika 51, pp. 175–184. Whittaker, E. T. (1923). On a new method of graduation, Proc. Edinburgh Math. Soc. (2) 41, pp. 63–75. Whittle, P. (1958). On the smoothing of probability density functions, J. Roy. Statist. Soc., Ser. B 20, pp. 334–343. Wold, S. (1975). Periodic splines for the spectral density estimation: The use of cross-validation for determining the degree of smoothness, Comm. Statist. 4, pp. 125–141.

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Index

bias of estimators conditional distribution, 89 conditional density, 98 conditional quantile, 91 diffusion, 150, 161 drift, 149, 160 histogram, 34 histogram for intensity, 117 jump size, 160 kernel density, 80 kernel density estimator, 23 kernel intensity, 111 kernel regression, 51, 81 weighted kernel, 67

duration excess density, 126 probability, 15, 126

change-point estimator, 179 nonparametric time series, 177 consistency kernel density estimator, 25 kernel intensity estimator, 111

hazard function, 14 kernel estimator, 109 Hellinger distance density estimator, 39 intensity estimator, 116 Hermite polynomials, 3 histogram, 33 hazard function, 110 intensity, 108, 117 regression, 124 H¨ olderian density, 29

ergodic density, 44 ergodic measure, 44, 45, 188 diffusion process, 158, 187 regression process, 72, 84 Gaussian process diffusion, 159 singularity function, 19, 164 transformation, 19, 164

diffusion process, 17, 147 approximation, 147, 159 continuous, 154 continuous estimation, 154, 162 drift estimation, 149, 160 ergodic density, 155 estimators, 18 jumps size estimation, 160 variance estimation, 150, 161 with jumps, 158

intensity Cox process, 109 multiplicative, 109 inverse regression function, 102 197

FunctionalEstimation

January 31, 2011

17:17

198

World Scientific Book - 9in x 6in

Functional estimation for density, regression models and processes

isotonic estimator density, 7 regression, 13 Kaplan-Meier estimator, 15 kernel estimator continuous diffusion process, 155 density, 5, 23 density derivatives, 27 density support, 37 ergodic density, 44 product-limit, 41 regression, 11, 50 regression derivatives, 58 regression for processes, 71, 84 varying bandwidth, 75, 80 weighted kernel, 12 kernel function, 24, 27 Laguerre polynomials, 3 left-censoring density, 42 intensity, 43 Legendre polynomials, 3 likelihood approximation, 179 multiplicative intensity, 109 linear representation kernel regression estimator, 51, 55 local polynomial estimator, 62 minimax estimator, 33 MISE density estimator, 6 regression estimator, 56, 73 mode density, 35 regression function, 68 MSE, 27 conditional quantile estimator, 95 density estimator, 5 diffusion process estimator, 157 kernel intensity estimator, 113 regression estimator, 11 norm

density estimator, 26, 29 kernel regression estimator, 54 optimal bandwidth conditional quantile, 95 density, 6, 27, 29, 31 derivatives, 28 diffusion process, 157 kernel regression estimator, 56 orthonormal basis, 2 parametric estimation, 140 periodic density, 9 time series, 9, 171 point process estimation, 108 intensity, 15 moments, 17 progressive censoring, 135 proportional odds model, 70 quantile conditional distribution, 87 quantile estimator, 14 approximation, 87 Bahadur representation, 14 conditional processes, 100 distribution function, 13 regression change of variables, 143 differential mean squares, 138 intensity, 132 kernel estimator, 50 mean squares, 138 model, 49 multiplicative intensity, 119 single-index, 13, 137 retro-hazard function, 42 right-censoring density, 40 point process, 15 quantile, 104 regression, 69

FunctionalEstimation

January 31, 2011

17:17

World Scientific Book - 9in x 6in

Index

stopping time, 15, 128, 159 time series, 167 auto-regressive, 168, 175 change-points, 175 covariance estimation, 173 nonparametric regression, 170 stationarity, 167 transformation, 174 weak convergence, 180 variance of estimators conditional density, 98 conditional distribution, 89 conditional quantile, 91 density, 80 diffusion, 150, 161 drift, 149, 160 histogram for intensity, 117 intensity, 111

FunctionalEstimation

199

jump size, 161 regression, 25, 51, 66, 72, 81 weighted kernel, 65 variation kernel density estimator, 29, 77 kernel regression estimator, 60, 83 weak convergence, 15, 128 conditional density estimator, 98 conditional distribution, 89 density estimator, 32 diffusion estimator, 151, 156, 162 diffusion process, 159 kernel intensity estimator, 114 regression estimator, 60 single-index estimator, 141, 143 varying bandwidth estimator, 78 weighted kernel estimator, 64 regression estimation, 67 time series, 174