A Berry-Esseen Bound For Least Squares Error Variance Estimators Of Regression Parameters

Lithuanian Mathematical Journal, Vol. 39, No. 1, 1999

A BERRY-ESSEEN BOUND FOR LEAST SQUARES ERROR ESTIMATORS OF REGRESSION PARAMETERS

VARIANCE

M. Bloznelis and A. Ra~kauskas Abstract. We construct a precise Berry-Esseen bound for the least squares error variance estimators of regression parameters. Our bound depends explicitly on the sequence of design variables and is of the order O(N -t/2) if this sequence is "regular" enough. Key words: Berry-Esseen bound, least squares estimators, error variance estimators, linear regression.

1. I N T R O D U C T I O N AND RESULTS Consider the linear model

Yi=clq-flXiq-gi,

i = 1 . . . . . N,

(I.1)

where Yl, - . . , Y~' are the observed responses to given variables (design points) X = (Xl, . . . , Xlv), and where ot and fl are unknown regression parameters to be estimated. Here el . . . . . e~v are unobservable errors. We assume that el . . . . . ere are independent identically distributed (i.i.d.) mean zero random variables. The ordinary least squares (OLS) estimators of the parameters ce and fl are defined by f = ' Y - / 3 X" and

/~ ----))-2(xlyl -I---" -4- XNYN)

(see [8]). Here xk = Xk -- X , Yk -~- Yk -- "Y, and "l) = (x 2 -I- . . . q- x 2 ) 1/2. We are interested in the rate of the normal approximation of f - tr and/3 - fl in the case where the variance Ee 2 is unknown. In this c a s e 0-2 is estimated by

0 -2 =

s2 =

(~ +...

+ ~)/(u

-

2),

~k = rk - a - ~ xk

(see [8]) and is used to normalize f - ct and/3 - ft. By the central limit theorem, for large N, distributions of the statistics

a-~ Ol = O I ( X ) =

~ ,

Qs

/~-/~ 02 = 0 2 ( X )

-- - - ,

);s

where

Q=

-~+~]

,

can be approximated by the standard normal distribution. We construct a bound for the accuracy of the normal approximation o f these statistics.

Supported by the Lithuanian State Science and Studies Foundation. Vilnius University, Naugarduko 24; Institute of Mathematics and Informatics, Akademijos 4, 2600 Vilnius, Lithuania. Published in Lietuvos Matematikos Rinkinys, Vol. 39, No. 1, pp. 1-8, January-March, 1999. Original article submitted May 27, 1998. 0363-1672/99/3901-0(~1522.00 9 1999 Kluwer Academic/Plenum Publishers

2

M. Bloznelis and A. Ra&auskas

The Berry-Esseen bound given in Theorem 1.1 below is precise and depends only on the ratio fl3/cr 3, where fl~3 = E[ei 13, and the fraction Ixkl3 1;3

g = gX -- EN--1

If V = 0 , p u t 8 = 1 andOi = 0 . THEOREM 1.1.

Assume that r 2 > O. Then there exists an absolute constant c > 0 such that

suplP{0,. ~<x} x

-

O(x) I ~< cE 3---L3 i = 1, 2, 0"3'

where dO(x ) denotes the standard normal distribution function.

Here and below, c, co, cl . . . . denote generic absolute constants. By c(T1, T2 . . . . ) w e denote a constant which depends only on the quantities indicated in the brackets. If the design points X are regular enough, e.g., they are regularly spaced, Xk = a + k t, k = 1, . . . , N, for some a, t, then Theorem I. 1 yields the Berry-Esseen bound /~3 ' suplP{Oix ~< x} - O(x) l ~< c 4~'cr3

for

i = 1,2.

The rate O ( N -tIE) is achieved also when the design points are taken at random and independently of the errors. Consider the linear model (1.1) in the case where the design points 2' = (2"1,. -., 2"~') are i.i.d, random variables independent of el . . . . . etc. Write F3 = E I2"l --E2"t[ 3 and r 2 = Var2"l. Theorem 1.1 implies the following bound. COROLLARY 1.1.

Assume that cr2 > 0 and r z > O. Then

suplP{~

~ x] - ,t,(x) I ~< c J ~/33Y3 3 r3'

for

i=1,2.

It is easy to recognize the structural similarity between the statistics 0t and 02 and the Student statistics, based on non-identically distributed observations. The Berry-Esseen bound for the Student test was proved by Bentkus and Gttze [1] and extended to the non-i.i.d, case by Bentkus, Bloznelis, and Gttze [2]. In order to obtain the optimal results for error variance estimators of regression parameters we extend the methods developed in these papers and then apply them to the statistics 01 and 02. It seems that the traditional technique, see, e.g., [7], does not allow one to obtain the optimal results: it involves moments of order which is higher than the optimal one. The rest of the paper is divided into two sections. In Section 2, a general Berry-Esseen type bound for statistics such as 01 and 02 without the assumption that the second moment of errors 8i is finite is formulated. Theorem 1. I is a simple consequence of this result. Proofs are given in Section 3. 2. A G E N E R A L R E S U L T Given two sequences of real numbers w = {wl, . . . , WN} and z = {zl . . . . , ZN} such that N

E

N

w/2= 1

i=1

and

E

z 2 = 1,

i=1

we denote T = T ( w , z) = S ( w ) / s ( z ) ,

(2.1)

Berry-Esseen bound

3

w~aere N

S(1o) = E

s2(Z) = ~2 _ N-IS2(z),

wigi'

N

and

= (el + . . . + e r e ) I N

~2 = N - I E ( e i

_ ~2.

i=1

Let E(w) = )--~=1 Iwil 3. Note that by (2.1), N

E(w) >1 N -1/2

and

Elwil~< N 1/2.

(2.2)

i=1

Clearly, the same inequalities hold for ~(z) and ~-~/N=t[Zil as well. Define the number a 2 = a 2 by the truncated second moment equation: a 2 = s u p { b : Eet21{el2 < ~ b N } />b},

a/>0.

It is known (and easy to show) that for any random variable (r.v.) el and any N such a number a exists, and a is the largest solution of the equation a 2 = Ee21{el2 ~< a2N}. Moreover, if r < c~, then a ~< 0". If o" = 00, then aN --+ (30 as N --* 00. In order to formulate a general result we introduce the truncated random variables

ri = a - l N - l / 2 e i l { e 2 i <~a2N},

1 <~i <<.N.

Note that Iri I ~< 1 and E r 2 = 1/N. The result formulated below provides a bound for

8N -- suPlP{T ~< x} - O(x) I x

in the general case, where the existence of the second moment of the r.v. el is not assumed. THEOREM 2.1.

Assume that a > O. Then 8N ~< c N P{e 2 > a2N} + c N IE rtl + c N3/2(E(w) + E(z)) E Iq 13.

(2.3)

3. PROOFS

Proof of Theorem 1.1. A simple calculation shows that ~ = ( 1 - 2 / N ) I / 2 T ( z , ' z ) and 01 = ( 1 - 2 / N ) t / 2 T ( w , z), where z = {zl . . . . . ZN}, Zi = xi/• and where l/) =

{1/)I, . . . ,

ll)N} ,

tOi -~- U i / W ,

13i = N -1 - x i ' - X / V 2,

W 2 =

By Theorem 2.1 and inequalities (2.2), for i = 1, 2,

suplP{0i ~< x) - ~(x) I ~< c ~(E(w) + E(z)). x

Furthermore, by H61der's inequality, ~-2 ~< V2/N" Therefore, N

E i=1

N

Ivi13

~< c N -2 + c(IX-l/Y2) 3 E

Ixi[3 <~c N-3/2(N-'/2 + s

i=1

and we obtain E(w) ~< cC(z). But S(z) = E and the proof is completed.

" X 2 / ) , ) 2 --[-

N -l.

4

M. Bloznelis and A. Ra~kauskas Proof of Corollary 1. I. The result follows from Theorem 1.1 and the inequality Y3

Es

<~c ~/t'~ ~ 3 "

Let us prove this inequality. Without loss of generality, assume that E ,a:'l = 0. Denote Xi = Xi --X, i = 1, . . . , N, where ~ = N-l(,.Yl + . - - + Xtr Let us = E Ix~l ~ and L = v3 v2 3/2. Denote the event A={x~Ex/2},

wherex=x~+...+X~r

Note that gx ~< 1 and thus, E g x <<.P{.A} + 23/2N -1/2 L. By Chebyshev's inequality and Rosenthal's inequality (see, e.g., [9]) P{A} ~< cE Ix - EXI3/2/( N v2) 3/2 ~< c N - l / 2 L. Thus, we obtain E s

<~c N-I/2L. Finally, using the inequalities Ex~=(1-N-i)r2>>.r2/2

and

Elxll3~
we obtain L ~< c y 3 / r 3, thus completing the proof.

Proof of Theorem 2.1. The proof goes along the lines of the proof of Theorem 1.2 in [2]. Therefore, we give a general scheme of the proof and detailed calculations of those steps of the proof which are different from [2]. Denote ~i = w i ( r i - E ri),

x=~xi, i=I

l <<.i <<.N,

rli = "c? - N - 1 ,

N

N

N

- g = ~ l " i - and O = ~ 0 i . i=l i=1

Let ot = (oq, . . . , o~N) stand for a set of i.i.d. Bernoulli random variables such that P{oq = 1} = 1 - P{al = 0} = p, with some 0 < p <~ 1. By ~ we shall denote a standard normal random variable. By (2.1), we can write ~ = ~l + " " + ~tr where ~l . . . . , ~lv are independent centered normal random variables with E ~/2 = w/2, i = 1, 2 , . . . , N. We shall assume that the sets a, (rl . . . . . rN) and (~l . . . . . ~u) are independent. Furthermore, we may and shall assume that for a sufficiently small Co > 0

N IE l"11 < Co and

(e(to) -{--g(z)) N3/2E Irt 13 < co.

(3.1)

If at least one of these inequalities fails, then (2.3) follows from the obvious estimate 8~t ~< 1. Note that the second inequality in (3.1) implies N E Irll 3 < co. Let g: R ~ R denote a function which is infinitely many times differentiable with bounded derivatives and such that 1/8 ~< g(x) ~< 2

for all x 6 R

and

g(x) =

Ix1-1/2

for 1/4 ~< Ixl ~< 7/4.

For a statistic W write 8(W) = sup:, IP{W ~< x} - *(x)l. The proof is divided into two steps. In the first step we replace T by the statistic V = q @ x g ( 1 + r/), which is a smooth function of the random variables ~i and r/i. In the second step we construct estimates for the difference between the characteristic functions f(t) = E exp{i t V} and ~b(t) = E exp{i t ~} = exp{-t2/2} and then give a bound for 8(V) using Esseen's smoothing lemma (see, e.g., [5]). Step I. We prove that 8(T)~<3(V)+cTBI,

whereT~l=NP{eZ>a2N}+NlErll+NElrll

Denote V' = ~/N'SI g(1 + A) and V" = ~/-N~g(1 + A), where N

SI ~ ~

i=1

N

11)iI'i'

$2 -~ S

i=1

zi "ci,

and

A = 1/+ r2/N + S~.

3.

(3.2)

Berry--Esseen bound

5

Let us show that 8(T) ~< 8(V') + cT~l. We have P{T ~ V'} ~< NP{t~ >~a2N} +P{IAI >i 3/4}. By Chebyshev's inequality, Lemma 3.1 (see below), and the inequality N -1/2 ~< N E Irt 13,we get P{Irtl/> 1/4} ~< cE [O[3/z ~
P{r2/N/> 1/4} ~< c N -I <<.c ~ l ,

and P{Szz/> 1/4} ~< cES~. Furthermore, by (2.2) and (3.1), ES~ ~< Er~ + IErll2 ~-~lz~l Izjl ~< N

-l

§ N lEt11"- ~<~l.

We obtain P{IAI > 3/4} ~< cT~l; therefore If(T) - 6(V')I ~< cT~l. Furthermore, I V ' - V"t ~< 2 R1(w), where Rt(w) = 4 ~ ' I E vt[ ~ = l [wi[. It follows from (2.2) that Rl(w) <~ Rl := N IErll and thus S(V') ~< 8 ( V " ) + c ~ l . It remains to show that 8(V") ~< 8(V) + c ~ l . Expanding g in powers of Q = N - l r 2 + S~ we obtain V" = V + R2, where IR21 ~< c 4~'lxQI. By Chebyshev's inequality, P{IR2I/> N -I/2} ~ c N E IxQI <~c (E 1~13)l/3(E 1~]3) 2/3 + c N E

I~IS~.

(3.3)

By Lemma 3.1, the first summand is bounded by c N -l/z. To estimate the second one let us denote u = N y~iffilzi (ri - E r i ) and Rl(z) = 4 ~ I E rll ~ = t Izil. Clearly, EINl/2ul 3 <~c (cf. Lemma 3.1) and Rl(z) <~ Rl. Since NS~ <~2 N u 2 + 2 R~(z), Ixle~Z(z) ~< Ixl 3 + Ig~ 13, and u21xl ~< lul 3 + Ixl 3, the last summand in (3.3) is bounded by cN -1/2 + c R y . We obtain P{IR21 /> N -1/2} <~c ~ t . Therefore, 8(V") ~< 8(V) + C ~ l and (3.2) is proved. Step 2. Here we show that 8(V) <~ c~2, (3.4) where ~2 = NIE rll + N3/Zs

E Irll 3.

It follows from Esseen's smoothing lemma that 8(V) <~

/

Itl-llf(t)

-

I dt +

c

TT'

Zl ~- c2/(N3/2~(to) E Ill 13) 9

O~
This inequality implies (3.4) if we show that I1 :=

f

Itl -t If(t) -

r

Idt

~< c7"r

(3.5)

0~[t[~r

and 12 :~

[t1-1 ]f(t) - q~(t)] dt ~< c~z, f ct <~ltl~
where the constant cl is sufficiently large. Let us prove (3.5). Write H(x) = exp{itx} and denote r IIH'tl + llH"lt + IIH"'II, where IIHII = sup~ IH(x)l. Then (3.5) follows from the inequality [E H(V) - E H(~)] ~< ccH "]-~2-

(3.6) =

(3.7)

6

M. Bloznelis and A. Ra~kauskas

This inequality is a consequence of

IE

(3.8)

- E g( )l

The proof of the first inequality in (3.8) is easy (see [3]). The proof of the second inequality is similar to the proof of the inequality (2.3) in [1]. An inspection of their proof shows that in order to verify the second inequality in (3.8) it suffices to show that

(3.9) N ~"~.i=l N xi rli and U = ,r ~_,l<~i,j<,~r j xirlj. The first two inequalities in (3.9) follow from where D = ~/'N Lemma 3.1. The inequality Er/2 <~ c ~ 2 is trivial, since Ir/il ~< 2, for 1 ~< i ~< N. To prove the last inequality in (3.9) let us write

U n'(v/Nx) =

E Ai,j, l<~i,j<~N,i#j

Ai,j = ~//N~itljnt(~l~:'r

and x = g/'J + xi + ~j, where we denote g/'J = ~t - xi - xj. Expanding H ' ( q ~ " (g/'J + xi + xj)) in powers of ~/N xi and subsequently in powers of ~ x/ we get

[EAe,yl <

cE(4 ,)21 jojl

~< cw~IwjlNI/2E[~'II 3.

Applying (2.2) we complete the proof of (3.9). Thus (3.7) is proved. The proof of (3.6) is more complicated. An additional effort has to be made to ensure the integrability with respect to the measure ~.(dt) = dt/It I in the region Cl ~< Itl of the remainder of expansions. In order to prove (3.6) one uses the approach developed in [4], [6], [10], where a similar integral is estimated in the case where f ( t ) is the characteristic function of U or more general nonlinear symmetric statistics based on i.i.d, observations. An extension of this approach to the situation where the observations are non-identically distributed is given in [2]. To prove (3.6) we expand g in powers of observations and then expand the exponent in much the same way as in the proof of Theorem 1.2 of [2], only now we use the inequalities of l_emma 3.1 to bound the remainders of these expansions. Finally, combining (3.5) and (3.6) we obtain (3.4), thus completing the proof of the theorem. LEMMA 3.1.

Assume that (3.1) holds. Then fors>O, andk=l,2

i

E ~r

N

<~cpNElrtl 3,

i=l E tv/-N ~[ 3 <<.c,

where U

=

~IN EI<~i,j<~N,i~j ~i'~i~

N

. . . . . N,

3/2

E EoliTli I

<<.cpNElrtl 3,

' i=1

EU2<~cp2NEIrl[ 3,

EIUI3/2<~cp7/4NEIrI[ 3,

9

Proof of Lemma 3. I. The proof is easy and routine (cf. the proof of Lemma 2.1 in [2]). REFERENCES 1. V. Bentkus and F. Grtze, The Berry-Essren bound for Student's statistic, Ann. Probab., 24, 491-503 (1996). 2. V. Bentkus, M. Bloznelis, and E G~tze, A Berry-Essren bound for Student's statistic in the non-i.i.d, case, J. Theoret. Probab., 9, 765-796 (1996). 3. V. Bentkus, F. G6tze, V. Paulauskas, and A. Ra~kauskas, The accuracy of Gaussian approximation in Banach spaces, SFB 343. Preprint 90-100, Universit~it Bielefeld (1990). 4. H. Callaert and P. Janssen, The Berry-Esseen theorem for U-statistics, Ann. Statist., 6, 417-421 (1978).

Berry-Esseen bound

7

5. W. Feller, Probability Theory, Vol. 2, Wiley, New York (1971). 6'. R. Helmers and W. R. van Zwet, The Berry-Esseen bound for U-statistics, in: Statistical Decision Theory and Related Topics, HI, S. S. Gupta and J. O. Berger (Eds.), Academic Press, New York (1982), pp. 497-512. 7. S.N. Lahiri, Bootstrapping M-estimators of a multiple linear regression parameter, Ann. Statist., 20, 1548-1570 (1992). 8. E. Malinvaud, Statistical Methods of Econometrics, North-Holland, Amsterdam (1970). 9. Yu. V. Petrov, Sums oflndependent Random Variables, Springer, Berlin (1975). 10. W.R. van Zwet, A Berry-Esseen bound for symmetric statistics, X Wahrsch. verw. Gebiete, 66, 425 449 (1984).