Mathematical Statistics: Basic Ideas and Selected Topics, Vol I (2nd Edition)

where E- 1 = family with statistics (Yj, . . . , Yp, {l'ilJh«<j
1.6.5

•

.

Conjugate Families of Prior Distributions

, • ,

In Section 1 .2 we considered beta prior distributions for the probability of success in n Bernoulli trials. This is a special case of conjugate families of priors, families to which the posterior after sampling also belongs. Suppose X1, . . . , Xn is a sample from the k-parameter exponential family (1.6.10), and, as we always do in the Bayesian context, write p(x I 8) for p(x, 8). Then

n

k

,

•

n

p( x j 9) = liT h(x,)] exp { L 1/; (9) L T;(x,) - nB(9)). i=l i=l j=l

- - --

(1 .6. 1 8)

.....

_ _ _ _ _ _

Section 1.6

63

Exponential Families

() E 8, which is k�dimensional. A conjugate exponential family is obtained from ( 1 .6.18) by letting n and tJ = L� 1 Tj (xi), j = 1, . . . , k, be "parameters" and treating () as the variable of interest. That is, let t = ( t1, . . . , tk+ I ) T and where

k

tk+1B(II))d01 - - - de, exp{ h t,�1(0) 1= 1= j { ( , . . . , tk=+J } 0 w( t1 , . . . , tk )

w(t)

-

=

· ··

f1

'

<

I I

< 00

with integrals replaced by sums in the discrete case. We assume that Problem

( 1 .6.19)

1.6.36), then

rl is nonempty (see

Proposition 1.6.1. The (k + !)�parameter exponential family given by

k (1.6.20) rr,(O) � exp{L �;(O)t; - tk+1B(O) - log w(t)} j=l where t (1 1 , . . . , tk+l) E !1, is a conjugate prior to p(xiO) given by ( 1 .6.18). Proof. If p(xi O) is given by ( 1.6.18) and " by (1 .6.20), then n k rr(Oix) p(xiO)rr,(O) exp{L 1)1 (0) (LT;(x;) + t;) - (tk+l + n)B(ll)} j=l i=l rr,(O), =

rx

rx

rx

( 1 .6.21)

where n

n

t1 + L TI (x;), . . . ,t, + L T•(x;), tk+1 + n i=l i=l and

ex

indicates that the two sides are proportional functions of(). Because two probability

densities that are proportional must family

T

be

equal, 1r(6jx) is the member of the exponential

( 1.6.20) given by the last expression in (I .6.2 1) and our assertion follows.

0

Remark 1.6.1. Note that (1.6.21) is an updating formula in the sense that as data x1, . . . , Xn become available, the parameter t of the prior distribution is updated to s = (t + a), where D a = (I:� 1 T1 (x;), . . . , I: � 1 T,(x;), n)T

It is easy to check that the beta distributions are obtained as conjugate to the binomial in this way.

Example 1.6.12.

Suppose

X1, . . . , Xn

is a

N(B,u5)

is unknown. To choose a prior distribution for model defined by ( 1.6.20).

Por

n

=

8,

sample, where

u6

is known and

8

we consider the conjugate family of the

I

02 Ox p(x l8) exp{ 2 u0 - 2u02 } . oc

( 1 .6.22)

64

Statistical Models, Goals, and Performance Criteria

Chapter 1

This is a one-parameter exponential family with

The conjugate two-parameter exponential family given by ( 1.6.20) has density

( 1 .6.23) Upon completing the square, we obtain

1rt(O)

oc

exp{-

t2

tl 2 2 (0 - - ) ) .

2a0

(1 .6.24)

h

Thus, 1rt (OJ is defined only for t2 > 0 and all t 1 and is the N(tJ/t2, <1� jt2) density. Our conjugate family, therefore, consists of all N(TJo, rJ) distributions where 7Jo varies freely and Tg is positive. If we start with aN(ry0, rJ) prior density, we must have in the (t 1 , t2) parametrization

( 1.6.25) By ( 1.6.21 ), if we observe EX, = s, the posterior has a density ( 1 .6.23) with

ao + n, 2

t2(n) =

2

To

t1(s) =

1Jo(1o2 + s . 2

To

Using (1 .6.24), we find that tr(Bix) is a normal density with mean

o 'l �]

(g

t (s) 1 = Jl(s,n) = , + n)- [s + � t2(n) To To

(1 .6.26)

and variance

1 u5 2 T0 (n) = ( = ( --, + r0 t2 n)

n _1 ) . --, a0

( 1.6.27)

Note that we can rewrite ( 1.6.26) intuitively as

p.(s, n) = W1X + W21]0

( 1.6.28)

where WJ = nT6(n)/u�, w2 = ri!(n)/ril so that w2 = 1 - w 1 .

D

Eo

These formulae can be generalized to the case X; i.i.d. N.(O, E0 ), 1 < i < n, varies over RP, Tg is scalar with known, 6 Np(TJ0, r6I) where > 0 and I is the p x p identity matrix (Problem 1.6.37). Moreover, it can be shown (Problem 1.6.30) that the Np()o,, r) family with )o, E RP and r symmetric positive definite is a conjugate family f'V

7Jo

To

i

Section 1.6

65

Exponential Families

p

p(p

'

.. . ..

'

.

Discussion . .

•

p

to Np(81 Eo), but a richer one than we've defined in (1 .6.20) except for = 1 because + 3)/2 rather than a + I parameter family. In fact, the conditions NP(.\, r) is a of Proposition 1.6.1 are often too restrictive. In the one-dimensional Gaussian case the members of the Gaussian conjugate family are unimodal and symmetric and have the same shape. It is easy to see that one can consuuct conjugate priors for which one gets reasonable formulae for the parameters indexing the model and yet have as great a richness of the shape variable as one wishes by considering finite mixtures of members of the family defined in (1 .6.20). See Problems 1 .6.31 and 1.6.32.

, B}) X , Xn) , T(Xi)·

model of Example 1 .5.3 is not covered by this Note that the uniform U( { I , 2, . theory. The natural sufficient statistic max( 1 ) . which is one-dimensional what­ In fact, the family of distributions ever be the sample size, is not of the form L� 1 in this example and the family U(O, B) are not exponential. Despite the existence of classes of examples such as these, starting with Koopman, Pitman, and Darmois, a theory has been built up that indicates that under suitable regularity conditions families of distributions, which admit k-dimensional sufficient statistics for all sample sizes, must be k-parameter exponential families. Some interesting results and a survey of the literature may be found in Brown ( 1986). Problem 1.6.1 0 is a special result of this type. . .

{Po : 9 E 8}, 8 Rk, 8, 'T } k T1, ... , Tk, h Rq Po k p(x, 9) = h(x) exp[L ry,(11)T;(x) -B(9)] , x E X C Rq. 1.6 .29) j=l = (T1 (X), . . . , Tk (X)) is called the natural sufficient statistic of the family. The T(X) canonical k-parameter exponentialfamily generated by T and h is q(x, 71) = h(x) exp{TT (x)71- A(71)} where A('I) � log J:··· J: h(x)exp{TT(x)71}dx

c Summary. is a k-parameter exponential family of distribu­ and real-valued functions tions if there are real-valued functions ry1, , and B on on such that the density (frequency) function of can be written as •

•

•

(

in the continuous case, with integrals replaced by sums in the discrete case. The set

71{ ERk : -oo < A('I) < oo} is called the natural parameter space. The set E is convex, the E R is convex. A If [ has nonempty interior in R' and 'lo E [, then T(X) has for X � P71, moment­ generating function ,P (s) = exp{A('Io + s) - A('lo)} for all s sucb that 'lo + s is in [. Moreover E71,[T(X)] � A('lo ) and Var71,[T(X)] A( 'lo) where A and A denote the gradient and Hessian of A. [=

map

a

;

---.

the

-

66

Statistical Models, Goals, and Performance Criteria

Chapter 1

Tk An exponential family is said to be of rank k if T is k-dimensional and 1 , T1 , are linearly independent with positive Po probability for some 8 E e. If P is a canonical exponential family with E open, then the following are equivalent: •

•

•

,

(i) P is of rank k,

(ii)

'7

is identifiable,

(iii) Var17(T) is positive definite, (iv) the map '7

�

A(17) is I - I on 1', '

(v) A is strictly convex on f.

A family F of prior distributions for a parameter vector () is called a conjugate family of priors to p(x j 8) if the posterior distribution of 8 given x is a member of F. The ( k + 1 )-parameter exponential family k

?Tt(O) = exp( L 1JJ (O)t, - B(O)tk+1 - logw} j= l where w

and

=

1: 1: ·

exp{l::1J, (O)t; - B(O)}dO,

k t = (t,, . . . , tk+l) E O = {(t, . . . , tk+l ) E R + l : O < w < oo } ,

is conjugate to the exponential family p(x i 9) defined in ( 1.6.29).

1.7

PROBLEMS AND COMPLEMENTS

Problems for Section 1.1

1. Give a formal statement of the following models identifying the probability laws of the data and the parameter space. State whether the model in question is parametric or nonparametric.

,,

(a) A geologist measures the diameters of a large number n of pebbles in an old stream bed. Theoretical considerations lead him to believe that the logarithm of pebble diameter is normally distributed with mean p, and variance a2. He wishes to use his observations to ob­ tain some information about p, and a2 but has in advance no knowledge of the magnitudes of the two parameters.

(b) A measuring instrument is being used to obtain n independent determinations of a physical constant p,. Suppose that the measuring instrument is known to be biased to the positive side by 0.1 units. Assume that the errors are otherwise identically distributed normal random variables with known variance.

Section 1.7

67

Problems and Complements

(c) In part (b) suppose that the amount of bias

is pOsitive but unknown. Can you per­

ceive any difficulties in making statements about f.t for this model ?

(d) The number of eggs laid by an insect follows a Poisson distribution with unknown

mean

A.

Once laid, each egg has an unknown chance p of hatching and the hatching of one

egg is independent of the hatching of the others. An entomologist studies a set of n such

insects observing both the number of eggs laid and the number of eggs hatching for each

nest.

2. Are the following parametrizations identifiable? (Prove or disprove.)

(a) The parametrization of Problem l . l . l(c).

(b) The parametrization of Problem l . l . l(d).

(c) The parametrization of Problem 1 . 1 . l (d) if the entomologist observes only the num­

ber of eggs hatching but not the number of eggs laid in each case.

3. Which of the following parametrizations are identifiable?

(Prove or disprove.)

(a) Xt , · . . , XP are independent with X� ""' N(ai + v, o-2 ) .

and

Po

is the distribution of X

(b) Same as (a) with

= (X1,

•

•

•

, Xp)·

a = (a1, . . . , ap) restricted to

{(a1 , . . . , a. ) : (c) X and Y are independent N( I'1 , a2)

Y - X.

p

I: a, i= l

= 0}.

and N(1'2, a2),

B = (I' 1 , 1'2 ) and we observe

i = 1 , . . . ,p; j = I , . . . , b are independent with X;; � N(!';; , a2) where /Lii = v + ai + Ai• 8 = (at, . . . , ap, At. . . . , Ab, v, o-2 ) and Po is the distribution of Xu , . . . , Xpb· (d) X;; ,

(e)

Same as (d) with

Z:f 1 a; = 0 and L��� >.;

(a1, . . . , ap) =

and

restricted to the sets where

0.

4. (a) Let U be any random variable and Show that

(At , . . . , Ah)

V

be any other nonnegative random variable.

Fu+v(t) < Fu (t) for every t.

(If Fx and Fy are distribution functions such that said to be stochastically larger than Y.)

n2

(b) As in Problem

Fx (t) < Fv (t) for every t, then X is

1.1.1 describe formally the following model.

Two groups of n1 and

individuals, respectively, are sampled at random from a very large population. Each

68

Statistical Models, Goals, and Performane
Chapter 1

member of the second (treatment) group is administered the same dose of a certain drug believed to lower blood pressure and the blood pressure is measured after 1 hour. Each member of the first (control) group is administered an equal dose of a placebo and then has the blood pressure measured after 1 hour. It is known that the drug either has no effect or lowers blood pressure, but the distribution of blood pressure in the population sampled before and after administration of the drug is quite unknown.

' '

5. The number

n of graduate students entering a certain department is recorded. In each of k subsequent years the number of students graduating and of students dropping out is recorded. Let Ni be the number dropping out and Mi the number graduating during year i,

i = 1, . . . , k. The following model is proposed.

Po[Nt nt,Mt = m1, . . . , Nk = nk, Mk mk] n. nt·· . . nk.mt· · · .mk.r. f-lt · ' · f-lk VI · · · Vk p =

=

I

I

I

! !

!

n1

n,.

m1

m,.

r

where ··

f.lt + ' · · + f.lk + Vt + · + Vk + p 1, 0 < /-li < l, 0 < Vi < 1, + · · · + mk + r n n1 + · · · + nk + and () = (f.lt, · . . , f.lk , vl, . . . , vk) is unknown. =

=

mt

I

'

I

i

1
(a) What are the assumptions underlying this model'?

Jl.i

(b) () is very difficult to estimate here if k is large. The simplification = 1r(1 (1 - 7r)(1 v) i - I v for i = 1, . . . , k is proposed where 0 < 1r < 1, 0 < J1. < 1, 0 < v < 1 are unknown. What assumptions underlie the simplification?

Ji.)i-tJl., vi =

�

6. Which of the following models are regular? (Prove or disprove.) (a) Pe is the distribution of X when X is uniform on (0, 0), e

.

=

(0, oo) .

.

(b) Pe is the distribution of X when X is uniform on {0, 1, 2, . . , 0}, e = { 1 , 2, . . }.

(c) Suppose X � N(Ji., 0'2) . Let Y and P8 is the distribution of Y.

=

1 if X < 1 and Y

=

X if X > 1. 0

=

(!', 0'2 )

(d) Suppose the possible control responses in an experiment are 0.1, 0.2, . . . , 0.9 and they occur with frequencies p(0.1), p(0.2), . . . ,p(0.9). Suppose the effect of a treatment be the distribution of a is to increase the control response by a fixed amount 0. Let treatment response.

Po

7. Show that Y - c has the same distribution as -Y + c. if and only if, the density or frequency function p of Y satisfies p(c + t) = p(c - t) fj>r all t. Both Y and p are said to be symmetric about c. Hint: If Y - c has the same distribution as -Y c, then P(Y < t + c) = P( -Y < t - c) = P(Y > c - t) = 1 - P(Y < c - t).

+

8. Consider the two sample models of Examples 1.1.3(2) and 1.1.4(1) .

.

Section 1.7

Problems and Complements

69

(a) Show that if Y � X + o(X). o(x) � 21' + C. - 2x and X � N(J1, "2 ), then G( · ) � F(- - C.). That is, the two cases J(x) - C. and J(x) � 21' + C. - 2x yield the same distribution for the data (X,, . . . , Xn). (Yt . . . . , Yn)· Therefore, G(-) � F(· - C.) does not imply the constant treatment effect assumption. (b) In part (a), suppose X has a distribution F that is not necessarily normal. For what type of F is itpossible to have G(·) � F(·-tl) for both J(x) tl andJ(x) � 2J1+tl-2x? . . .·

(c) Suppose that Y � X + J(X) where X � N(Jl•"') and o(x) is continuous. Show that if we assume that J(x) + x is strictly increasing, then G(-) � F(- - C.) implies that J(x) = C..

.- -

" '"

9. Collinearity: Suppose Yi Let zJ· = (zli· · - - , Znjf-

(a) Show that independent).

(/31 ,

. . . ,

=

L:j== 1 4i/3i + ci, £i

"' N(O, o-2 ) independent, 1 < i < n.

/3p ) are identifiable iff z 1 , . . . , zp, are not collinear (linearly

(b) Deduce that (fit, . . . , {Jp ) are not identifiable if n < p, that is, if the number of pardllleters is larger than the number of observations. 10.

Let X = (min(T, C), l(T < C)) where T, C are independent, P[T = j]

p(j), j = 0, . . . , N,

P[C � j]

r(j), i

�

0, . . , N .

and (p, r) vary freely over F � { (p, r) : p(j) > 0, r(j) > 0, 0 < j < N, L,f 0p(j) � I , L,f 0 r(j) � I } and N is known. Suppose X,, . . . , Xn are observed i.i.d. according to the distribution of X. Show that {p(j) : j � 0, . . , N), {r(j) : j � 0, . . , N) are identifiable. Hint: Consider "hazard rates" for Y = min(T, C), .

.

P[Y

�

j, y � T I y > j].

The Scale Model. Positive random variables X and Y satisfy a scale model with parameter J > 0 if P(Y < t) � P(JX < t) for all t > 0, or equivalently, G(t) F(t/8), J > 0, t > 0. 11.

=

(a) Show that in this case, log X and log Y satisfy a shift model with parameter log J.

(b) Show that if X and Y satisfy a shift model With parameter C., then ex and eY satisfy a scale model with parameter e.o.. (c) Suppose a scale model holds for X, Y. Let c > 0 be a constant. Does X' = Y' yc satisfy a scale model? Does log X', logY' satisfy a shift model? =

12. The Lelunann 1Wo-Sample Model. In Example 1.1.3 let Xt. . . . , Xm and Yi , . .

.

xc, , Yn

denote the survival times of two groups of patients receiving treatments A and B. Sx (t) =

'

' f '

70

Stdtistical Models, Goals, and Performance Criteria

Chapter 1

P(X > t) = 1 - F(t) and Sv(t) = P(Y > t) = 1 - G(t), t > 0, are called the survival functions. Survival beyond time t is modeled to occur if the events T1 > t, . . . , Tk > t all occur, where T1, . , Tk are unobservable and i.i.d. as T with survival function S0. For treatments A and B, k = a and b, respectively.

'

t Ii

.

.

(a) Show that Sv(t)

= s';{"(t). (b) By extending (bja) from the rationals to 5 E (0, oo) , we have the Lehmann model

'

' •

Sv(t) = Si,(t), t > 0.

(1 .7.1)

Equivalently, Sv(t) = sg.(t) with C. = a5, t > 0. Show that if S0 is continuous, then X' = - log S0 (X) and Y' = - log S0(Y) follow an exponential scale model (see Problem 1 . 1 . 11) with scale parameter 5. Hint: By Problem B.2.12, So (T) has a U(O, I) distribution; thus, - log S0(T) has an exponential distribution. Also note that P(X > t) = S0 (t).

•

' I

(c) Suppose that T and Y have densities fo(t) and g(t). Then h,(t) = f0(t)/S0(t) and hy(t) = g(t)jSy(t) are called the hazard rates of To and Y. Moreover, hv(t) = C.ho(t) is called the Cox proportional hazard model. Show that hv(t) = C.h,(t) if and only if

Sv(t) = sg.(t) .

13. A proportional hazard model. Let f(t I z;) denote the density of the survival time Y; of a patient with covariate vector Zi apd define the regression survival and hazard functions of Yi as

Sv(t I z; ) =

!,� f(y I z;)dy, h(t I z; ) = f(t I z; )/Sv (t I z; ).

Let T denote a survival time with density fo(t) and hazard rate ho(t) = The Cox proportional hazanl model is defined as

h(t I z) = h0(t) exp {g(j3, z) )

fo(t)/ P(T > t). (1 .7.2)

where ho(t) is called the baseline hazard function and g is known except for a vector {3 = ({31 , . . . , {3p)T of unknowns. The most common choice of g is the linear form g(/31 z) = zT/3. Set C. = exp{g(j3, z)}. (a) Show that (1. 7.2) is equivalent to Sy (t I

z) = Sf': (t). (b) Assume (1 .7.2) and that Fo(t) = P(T :S t) is known and strictly increasing. Find an increasing function Q(t) such that the regression survival function of Y' = Q(Y) does not depend on ho(t). ' !I ''

"

•

Hint: See Problem 1.1.12.

(c) Under the assumptions of(b) above, show that there is an increasing function Q•(t) such that ifY;' = Q•(Y;), then f'i* = 9 ({31 Zi) + £i for some appropriate €i. Specify the distribution of fi.

-

--

-----

----- -

-

Section 1.7

Problems and

71

Complements

Hint: See Problems 1.1.11 and 1 . 1 . 12.

In Example 1 . 1 .2 with assumptions ( 1)--{4), the parameter of interest can be character­ 1 1 ized as the median v = F- (0.5) or mean I' = f=oo xdF(x) = J0 F-1 (u)du. Generally, J1 and v are regarded as centers of the distribution F. When F is not symmetric, /1 may be very much pulled in the direction of the longer tail of the density, and for this reason, the median is preferred in this case. Examples are the distribution of income and the distri­ bution of wealth. Here is an example in which the mean is extreme and the median is not. Suppose the monthly salaries of state workers in a certain state are modeled by the Pareto distribution with distribution function 14.

I - (xfc)-8, x >

F (x O) ,

c

x
0,

where () > 0 and c = 2, 000 is the minimum monthly salary for state workers. Find the median v and the mean J1 for the values of() where the mean exists. Show how to choose () to make J.L - v arbitrarily large.

Let X1 , . . . , Xm be i.i.d. F, Y1 , . . . , Yn be i.i.d. described by 15.

G,

where the model

{(F, G)}

is

where 1/J is an unknown strictly increasing differentiable map from R to R, '1/J' > 0, '1/J(±oo) = ±oo, and Z1 and Z{ are independent random variables . (a) Suppose Z1 , z; have a N(O, 1) distribution. Show that both ,P and � are identifi­ able. (b) Suppose Z1 and z; have a N(O, (,P (t)) .

Problems for Section 1.2 1.

Merging Opinions. Consider a parameter space consisting of two points fh and Bz, and

suppose that for given B, an experiment leads to a random variable function p(x I 0) is given by 0\x 01 Oz

0 0.8 0.4

l

0.2 0.6

Let 1r be the prior frequency function of 0 defined by rr(111)

(a) Find the posterior frequency function rr(O

X whose frequency

I x).

=

�,

rr

(02 )

=

�.

(b) Suppose X,, . . . , Xn are independent with frequency function p(x posterior 1r(B I x1, , xn ) · Observe that it depends only on 2:� 1 Xi. .

•

.

I

0). Find the

I

72

'

Statistical Models. Goals, and Performance Criterid

(c) Same as (b) except use the prior 1r1 (81)

Chapter 1

.25, 1r1 (82 ) � . 75. = .5n) for the two priors and 1ft when

�

(d) Give the values of P(B = fh I z:::; 1 xi n � 2 and 100.

7r

�

(e)Give the most probable values 8 = arg ma.xo 1r (B I E7 1 Xt = k) for the two priors � 1r and 1r1. Compare these B's for n = 2 and 100. (f) Give the set on which the two B's disagree. Show that the probability of this set tends

to zero as n ---t oo. Assume X rv p(x) = L;=l 1r(8i)p(x I Bi)· For this convergence, does it matter which prior, 7r or 11"1, is used in the formula for p(x )? 2. Consider an experiment in which, for given () = B, the outcome X has density p(x 8) � (2xj82), 0 < X < e. Let 7r denote a prior density for 8.

(a) Find the posterior density of 8 when 1r(8)

(b) Find the posterior density of 8 when 1r (B)

� �

1, 0 < 0 < 1.

302, 0 < 0 < 1.

(c) Find E(8 1 x) for the two priors in (a) and (b).

(d) Suppose X1 , . . . , Xn are independent with the same distribution as X. Find the posterior density of8 given X1 = x1 , . . . , Xn = Xn when 1r(B) = 1, 0 < 8 < 1.

I

I

3. Let X be the number of failures before the first success in a sequence of Bernoulli trials with probability of success 0. Then Po[X � k] � ( 1 - e) • e, k � 0, 1, 2, . . . . This is called the geometric distribution (9(0)). Suppose that for given 8 � 0, X has the geometric distribution

'

(a) Find the posterior distribution of 8 given X un1.,1orm on { 41 , 21 , 43 } .

=

2 when the prior distribution of 8 is

(b) Relative to (a), what is the most probable value of 8 given X �

(c) Find the posterior distribution of 8 given X f3(r, s). 4.

=

2? Given X

•

•

.

, Xn are natural numbers between 1 and 8 and 8 = {1, 2, 3,

(a) Suppose 8 has prior frequency,

' ' "

: '

c(a) . . 7r(J) = ·a , ] = 1 , 2, . . . , J

1 where a > 1 and c(a) � [2:;" r"J- Show that 1

1T (). I XJ, . . . , Xn)

=

c(n + a, m) . ·J n+a , J = m, m + l, . . . ,

--�--

---

k?

k when the prior distribution is beta, i

Let X1, . . . , Xn be distributed as

where x1,

�

.

.

. }.

Section 1.7

73

Problems and Complements

where m = max(x,, . . . , J:n), c(b, t) =

[�;"' , F"J-', b > 1.

(b) Suppose that max (x1 , . . . , Xn) x1 = m for all n. Show that ?T(m 1 as n --+ oo whatever be a. Interpret this result.

=

I

XI,

. . . , Xn) --+

x is not close to 0 or 1 and 5. In Example 1 .2.1 suppose n is large and (1 jn) L� 1 Xi the prior distribution is beta, f3(r, s) . Justify the following approximation to the posterior distribution

=

where q. is the standard normal distribution function and

_

n r J.l = n + r + s x + n + r + s ' _

_2

li(l - li)

= a n +r+s'

Hint: Let /3(a, b) denote the posterior distribution. If a and b are integers, then /3 ( a, b) , W, are in­ is the distribution of (aV fbW)[l + (aVfbW)]- 1 , where V1 , . . . , v., W1 , dependent standard exponential. Next use the central limit theorem and Slutsky's theorem. .

.

•

6. Show that a conjugate family of distributions for the Poisson family is the gamma family.

7, Show rigorously using (1.2.8) that if in Example 1.1.1, D = NO has a B(N, ?To ) distri­ bution, then the posterior distribution of D given X = k is that of k + Z where Z has a

B(N - n, ?To ) distribution.

.

8. Let (X,, . . . , Xn+k) be a sample from a population with density f(x I 8), e E e. Let 9 have prior density 1r. Show that the conditional distribution of (6, Xn + 1 , . . . , Xn+k) given xl = X J , . . . ' Xn = Xn is that of (Y, Zt, . . . Zk) where the marginal distribution of y equals the posterior distribution of 6 given X1 = X t , . . . , Xn = Xn, and the conditional distribution of the zi 's given y t is that of sample from the population with density l

=

f(x I t).

9. Show in Example 1.2.1 that the conditional distribution of 6 given I:; 1 Xi = k agrees with the posterior distribution of 6 given X1 = X t , . . . , Xn = Xn, where I:� 1 Xi = k.

Suppose xl, · · . , Xn is a sample with xi ,....., p(x I 0), a regular model and integrable as a function of e. Assume that A = {X : p(X I 8) > 0} does not involve e. 10.

(a) Show that the family of priors N

11' (8) =

IT p(�, I 8)

i=l

where �i E A and N E {1,2, . . . } is a conjugate family of prior distributions for p(x I 8) and that the posterior distribution of(} given X = x is 1r (8 I x) =

N'

IT vW I 8)

i=l

I

!

74

Statistical Models, Goals, and Performance Criteria

where

N' = N + n and ({I, . . , {hr,) = ( {1 >

• . •

.

(b) Use the result (a) to give .,-(8)

and

12.

Suppose

p(x

.

.

.,-(8 I x) when

U. Let p(x I B) = exp{-(x - 8)}, 0 < B <

•

.

•

Xn ) .

Bexp{ -Ox}, x > 0, B > 0 0 otherwise.

p(x I B)

the posterior density

, {N, X1,

Chapter 1

.,-(B I x).

I 0) is the density of i.i.d.

x and let .,-(8)

= 2 exp{-2B}, B > 0.

Xn. where xi X1 1 of the distribution of Xi. • • • 1

.-v

Find

N (�to , �)' j.lo is

a- 2 is (called) the precision (a) Show that p(x I B) ()( o: n exp (- itB) where t = �� l (X, - l"o) 2 and ()( denotes "proportional to" as a function of B. (b) Let .,-(B) o< ol (>-') exp { -� vB}, v > 0, .\ > 0; B > 0. Find the posterior distri­ bution 1r(O I x) and show that if >. is an integer, given x, O(t + v) has a x�+n distribution. known, and () =

Note that, unconditionally,

v(} has a xX distribution.

(c) Find the posterior distribution of a.

N(iJ., a2) and we formally put 7C(iJ., a) = ,; , then the posterior density !f"(J-t I x, s2) of J1. given (x, s2 ) is such that y'n(p.�.X) "' tn-I· Here s2 = n-1 1 " I...J ( Xt - X )2 • Hinl: Given iJ. and a, X and s2 are independent with X � N(iJ.,a 2 jn) and (n ­ l)s2 ja2 "' X�-l· This leads to p(X, s2 I J.L, a2). Next use Bayes rule. , Xn,Xn+l are i.i.d. f(x I 8), 6 "" 1r, the predictive 14. Ina Bayesian model where X 1 , 13.

Show that if X1 , . . , Xn are i.i.d. .

.

.

•

distribution is the marginal distribution of Xn+I· The posterior predictive distribution is the conditional distribution of Xn+l given X1 , . . . , Xn. (a) If f

and .,- are the

N(B, aJ)

and

posterior predictive distribution.

N(Bo, r.f) densities, compute the predictive and

(b) Discuss the behavior of the two predictive distributions as

n -+ oo.

15. The Dirichlet distribution is a conjugate prior for the multinomial. distribution, D(a:), a = (o:1, , ar)r, O'j > 0, 1 ::; j < r, has density •

Let N =

(N1,

•

•

•

.

.

, Nr ) be multinomial

M(n, 9),

0=

r

(B h . . . , Br)T, 0 < 8; < 1, L B; = 1. j= l

The Dirichlet

Section 1.7

75

Problems and Complements

Show that if the prion r( 0) for 0 is V( a), then the posterion r( 0 f N nr ) where n = ( n t

� n

) is V(a + n),

1 • • • ,

Problems for Section 1.3 1. Suppose the possible states of nature are (}1, (}2, the possible actions are a1, a2, a3, and the loss function l((}, a) is given by a, a,

o 2

I

0

2 I

Let X be a random variable with frequency function p(x, (}) given by

I (1 - p) (I - q) and let d1 , when

.

•

.

,

dg be the decision rules of Table 1.3.3. Compute and plot the risk points

(a) p � q = .1, (b) p � 1 - q � . l. (c) Find the minimax rule among J1 , . . . , b"9 for the preceding case (a). (d) Suppose that (} has prior 1r(BI)

0.5, 1r(B2) � 0.5. Find the Bayes rule for case

�

(a).

2. Suppose that in Example 1.3.5, a new buyer makes a bid and the loss function is changed

to

8\a a,

a,

a,

a3

0 12

7 I

4 6

o,

(a) Compute and plot the risk points in this case for each rule (h, . , dg of Table 1.3.3. .

.

(b) Find the minimax rule among {J 1 . . . , b"g}. ,

(c) Find the minimax rule among the randomized rules. 1'

{d) Suppose 0 has prior 1r(a1) 0.5 and (ii) 1' 0.1.

=

=

� 1'·

1r (a2 )

=

1-

1'·

Find the Bayes rule when (i)

3. The problem of selecting the better of two treatments or of deciding whether the effect of one treatment is beneficial or not often reduces to the pr9blem of deciding whether B < 0. B 0 or 8 > 0 for some parameter 8. See Example 1.1.3. Let the actions corresponding to deciding whether a < 0, (} 0 or a > 0 be penoted by -1, 0, 1, respectively and suppose the loss function is given by (from Lehmann, 1957) =

=

Statistic<�! Models, Goals, and Performance Criteria

76

Chdpter 1

1 -1 0 c b+c 0 b =0 0 b > 0 b+c c 0 where b and c are positive. Suppose X is aN(B, 1) sample and consider the decision rule J,,,(X) = -1 if X < r 0 if r < X < s 1 if X > s. 0\a <0

(a) Show that the risk function is given by

R(O, J,,,)

c( y'n(s - 0)), 0 < 0 b(y'nr), 0=0 c

(y'n(r - 0)), 0 > 0

where =

1 - , and is the N(0, 1) distribution function. (b) Plot the risk function when b = c = 1, n = I and .

(t) r = -s = -1, (n) r = - s = -1. 2 For what values of 8 does the procedure with r = -s = -1 have smaller risk than the procedure with r = -�s = -1? 4. Stratified sampling. We want to estimate the mean J1. = E(X) of a population that has been divided (stratified) into s mutually exclusive parts (strata) (e.g., geographic loca­

'

I

' i

.

i I

'

I

I

"

tions or age groups). Within the jth stratum we have a sample of i.i.d. random variables X1j, · · · , Xnii; j = l, . . . , s, and a stratum sample mean Xi ; j = l, . . . , s. We assume that the s samples from different strata are independent. Suppose that the jth stratum has 100pj% of the population and that the jth stratum population mean and variances are f-£J and o"J. Let N = E;= l nJ and consider the two estimators s

n3

1 /1 1 = N - L LxiJ, where we assume that PJ•

j=l i=l

1 < j < s, are known.

iiz

s

= L PJ XJ

j=l

(a) Compute the biases, variances. and MSEs of ji1 and Jiz. How should nj, 0 be chosen to make /11 unbiased?

I

{b) Neyman allocation. Assume that 0

< J. < s,

< aJ < oo, 1 < j < s, are known (estimates

will be used in a later chapter). Show that the strata sample sizes that minimize MSE(/i2) are given by

( 1 .7.3)

Hint:

77

Problems and Complements

Section 1.7

You may use a Lagrange multiplier.

(c) Show that

MSE(Ji!) with nk � Pk N minus MSE(Ji,) with nk given by ( 1 . 7.3) is

N- 1 2:;= 1 Pj (t7J - a-)2, where a- = 2:;=1 PP7r �

-

Xb and Xb denote the sample mean and the sample median of the sample X1 b . . , Xn - b. lf the parameters of interest are the population mean and median of Xi - b, respectively, show that MSE(X,) and MSE(X,) are the same for all values of b (the

5. Let '

.

�

MSEs of the sample mean and sample median are

6.

Suppose that

that

n is odd.

satisfying

X1 , . . . , Xn are i.i.d.

ac;

X

invariant with respect to shift).

F, that X is the median of the sample, and �

""

We want to estimate "the" median

P(X < v) ?: � ll!ld P(X > ) > �v

�

v ofF,

where

v is defined as a value

(a) Find the MSE of X when (i) F is discrete with

a < b < c. Hint:

�

P(X � c) � p, P(X

Use Problem 1.3.5. The answer is M

where (ii)

P(X � a)

k � .5 (n + ! ) and S � B(n,p).

� b)

�

1 -- 2p, 0 < p

<

1,

SE(X) � [(a-b)2 + (c- b)2]P(S ?: k)

F is uniform, U(O, 1). Hint:

See Problem B.2.9.

(iii) F is normal,

N(O, 1) , n � 1, 5, 25, 75.

Hint: See Problem B.2.13.

Use a numerical integration package.

� MSE(X)/M SE(X) in question (i) when b � 0, a � -,;, b � ,;, p � 20 40 and n � 1, 5, 15. (c) Same as (b) except when n � 15, plot RR for p � .1, .2, .3, 4 .45. (d) Find E[X - b[ for the situation in (i). Also find E[X - bl when n � 1, and 2 and � compare it to E[X - b[. (e) Compute the relative risks MSE(X)fMSE(X) in questions (ii) and (iii). 7. Let X1 , . . . , Xn be a sample from a population with values (b) Compute

the relative risk RR .

,

.

,

.

�

,

-

e - 2,; , e - ,; , e , e + ,; , e + 2,;; ,; > o. Each value has probability

�

-

.2. Let X and X denote the sample mean and median.

Suppose

that n is odd. �

(a) Find MSE(X)

andthe relative risk RR

(b) Evaluate RR when n � 1, 3, 5. Hint: By Problem 1.3.5, set () 0

distribution of

�

=

�

-

� MSE(X)fMSE(X).

without loss of generality. Next note that the

X involves Bernoulli and multinomial trials.

I

I 78

:!

j

'

8. Let X 1 ,

Chapter 1

, Xn be a sample from a population with variance a2, 0 < a2 < oo. (a) Show that s:? = (n - 1) - l E� 1 (Xi - X)2 is an unbiased estimator of u2. Hint: Write (X, - X)2 = ([X, - !'] - [X - l'lJ2, then expand (X, - X)2 keeping the

' ' '

'

Statistical Models, Goals, and Performance Criteria

.

•

.

square brackets intact.

(b) Suppose X,

i :

-

N(!',
(i) Show that MSE(s2 ) =

' '

I' :

(ii) Let &6 = c c=

L�

(n + 1)- 1

l

2(n - 1)-1
(xi - X)2•

Show that the value of c that minimizes

MsE(O'; ) is

Hintfor question (b): Recall (Theorem 8.3.3) that
' ' ' ' '

distribution. You may use the fact that E(Xi - J.L)4

'

I

.I

= 30'2•

9. Let () denote the proportion of people working in a company who have a certain charac­ teristic (e.g., being left-handed). It is known that in the state where the company is located, 10% have the characteristic. A person in charge of ordering equipment needs to estimate 8 and uses where jJ

=

e = (.2J(.1oJ + (.sw

X/n is the proportion with the characteristic in a sample of size n

company. Find

�

from the

MSE(e) and MSE(P). If the true e is eo, for what e0 is MSE(B)jMSE (P) < 1?

Give the answer for n

= 25 and n = 100. 10. In Problem 1.3.3(a) with b = c = 1 and n = 1. suppose 6 is discrete with frequency function 1r(O) = 1r ( -�) = 1r U) = '1· Compute the Bayes risk of dr,s when (a) r = -s -1 (b) r = -�s = -1.

•

=

Which one of the rules is the better one from the Bayes point of view?

11. A decision rule d is said to be unbiased if

E9 (Z(e,o( X) ) ) < E9(Z(e' , o(X))) "

'

II '

•

I

for all e, e'

E 8.

(a) Show that if e is real and l (e, a) definition of an unbiased estimate of 8.

= (e - a ) 2, then this definition coincides with the

(b) Show that if we use the 0 - 1 loss function in testing. then a test function is unbiased in this sense if, and only if, the powerfunction, defined by {J(e, o) = E9(o(X )), satisfies

for all

e' E 81•

, •

1

{J(e',o) > sup({J(e, o ) : e E 8o} , i

-

Section 1.7 Problems and Complements

79

12. In Problem 1.3.3, show that if c ::; b,

z

>

0, and

then Jr,.� is unbiased

13. A (behavioral) randomized test of a hypothesis H is defined as any statistic rp(X) such that 0 <
otherwise. Define the nonrandomized test Ju,

0 < u < 1, by u 0 if
Ou(X)

1

if

Suppose that U - U(O, 1) and is independent of X. Consider the following randomized test J: Observe U. If = u, use the test Ju. Show that J agrees with rp in the sense that,

U

Pe[o(X)

�

1]

�

1 - Pe[o(X)

� OJ = E9 (
14. Convexity ofthe risk set. Suppose that the set of decision procedures is finite. Show that if J1 and J2 are two randomized procedures, then, given 0 < a < 1, there is a randomized procedure 83 such that R(e, J,) = aR(e, o.) + (1 - a)R( e, 62) for all e.

15. Suppose that Pe, (B) = 0 for some event B implies that P9(B) = 0 for all e E 8. Further suppose that l(e0 , ao) � 0. Show that the procedure J(X) au is admissible. 16. In Example 1.3.4, find the set of J1 where MSE(ji) < MSE(X). Your answer should depend on n, 0"2 and J � II' - Jto 1 17. In Example 1.3.4, consider the estimator Mw If n, 0"2 and o �

� WJto +

(1 - w)X.

II' - Jto I are known,

(a) find the value of wo that minimizes MSE(J'w), (b) find the minimum relative risk of P,w0 to X. 18. For Example 1.1.1, consider the loss function ( 1.3.1) and let Jk be the decision rule "reject the shipment iff X 2 k."

(a) Show that the risk is given by (1.3.7). (b) If N � 10, s = r � 1, e0 � .I, and k = 3, plot R(B, Jk) as a function of e.

(c) Same as (b) except k = 2. Compare J2 and 63. 19. Consider a decision problem with the possible states of nature 81 and 82 , and possible actions a1 and a • Suppose the loss function f(B, a ) is 2

80

Statistical Models, Goals, and Performance Criteria

Chapter 1

Let X be a random variable with probability function p(x ! 8) B\x

e, e,

0 o.2 o.4

I o.s o.6

(a) Compute and plot the risk points of the nonrandomized decision rules. Give the minimax rule among the nonrandomized decision rules. (b) Give and plot the risk set S. Give the minimax rule among the randomized decision rules. (c) Suppose B has the prior distribution defined by rr(B,) the Bayes decision rule?

=

0.1, rr(B,)

=

0.9. What is

Problems for Section 1.4

1. An urn contains four red and four black balls. Four balls are drawn at random without replacement. Let Z be the number of red balls obtained in the first two draws and Y the total number of red balls drawn. (a) Find the best predictor of Y given intercept linear predictor.

Z,

the best linear predictor, and the best zero

.

'

(b) CompU!e the MSPEs of the predictors in (a). 2. In Example 1.4.1 calculate explicitly the best zero intercept linear predictor, its MSPE, and the ratio of its MSPE to that of the best and best linear predictors.

3. In Problem B.l.7 find the best predictors ofY given X and of X given Y and calculate their MSPEs. 4. Let U1 , U2 be independent standard normal random variables and set Y = U1 • Is Z of any value in predicting Y?

Z

=

U'f + U?,

5. Give an example in which the best linear predictor of Y given Z is a constant (has no

predictive value) whereas the best predictor Y given Z predicts Y perfectly. 6. Give an example in which

Z can be used to predict Y perfectly, but Y is of no value in predicting Z in the sense that Var(Z I Y) = Var(Z), 7, Let Y be any random variable and let R(c) = E(IY-cl) be the mean absolute prediction error. Show that either R(c) = oo for all c or R(c) is minimized by taking c to be any number such that P[Y 2: c] > !, P[Y < c] > ; . A number satisfying these restrictions is called a median of (the distribution of) Y. The midpoint of the interval of such c is called the conventionally defined median or simply just the median.

I

I

I

Section 1.7

Problems and Complements

81

Hint: If c < eo . .• '·

El Y - col = ElY - cl + (c - co){P[Y > co] - P[Y 8.

< eo]} + 2E[(c - Y)l[c < Y <

c0]].

Let Y have a N(p, <72) distribution.

(a) Show that E(IY - cl) = O"Q[I c - 1"1/<7] where Q(t) = 2[(t)] - t. (b) Show directly that p minimizes E(IY - cl) as a function of c. 9. If Y and Z are any two random variables, exhibit a best predictor of Y given Z for mean absolute prediction error. Suppose that Z has a density p, which is symmetric about all z. Show that c is a median of Z. 10.

c,

p( c + z) = p(c - z) for

11. Show that if ( Z, Y) has a bivariate nonnal distribution the best predictor of Y given z in the sense of MSPE coincides with the best predictor for mean absolute error.

Many observed biological variables such as height and weight can be thought of as the sum of unobservable genetic and environmental variables. Suppose that Z, Y are measure­ ments on such a variable for a randomly selected father and son. Let Z', Z11, Y', Y" be the corresponding genetic and environmental components Z = Z' + Z", Y = Y' + Y", where (Z', Y') have a N(p,, J.L, a2, a2 , p) distribution and Z", Y" are N(v, T2) variables independent of each other and of ( Z', Y'). 12.

(a) Show that the relation between Z and Y is weaker than that between Z' and Y'; that is, [Cor(Z, Y )l < I PI · (b) Show that the error of prediction (for the best predictor) incurred in using Z to predict Y is greater than that incurred in using Z' to predict Y'. 13. Suppose that Z has a density p, which is symmetric about c and which is unimodal; that is, p( z) is nonincreasing for z > c. (a) Show that P[I Z - tl :; s] is maximized as a function of t for each s > 0 by t = c. (b) Suppose (Z, Y) has a bivariate normal distribution. Suppose that if we observe Z = z and predict p ( z ) for Y our loss is 1 unit if II"( z) - Yl > s, and 0 otherwise. Show that the predictor that minimizes our expected loss is again the best MSPE predictor.

14. Let zl and Zz be independent and have exponential distributions with density >..e -Az , z > 0. Define Z = Z2 and Y = Z1 + Z1Z2 • Find

(a) The best MSPE predictor E(Y (b) E(E(Y

I Z)) (c) Var(E(Y I Z)) (d) Var(Y I Z = z) (e) E(Var(Y I Z))

I Z = z) ofY given Z = z

'

'

I

'

'

'

82

Statisf1cal Models, Goals, and Performance Criteria

Chapter 1

(0 The best linear MSPE predictor of Y based on

'

Z = z. Hint: Recall that E( Zt) = E( Z2) = 1/A and Var ( Zt) = Var ( Z2) = 1 /A 2 15. Let J-L(z)

=

E(Y I Z = z). Show that

Var(J-L(Z))/Var( Y)

=

Corr2 (Y, J-L(Z))

= max g

Corr' ( Y, g(Z))

where g(Z) stands for any predictor.

' 1,

'

16. Show that p�y linear predictors.

=

Corr' (Y, J-L L(Z)) =

maxgEC

Corr2 (Y, g(Z)) where £ is the set of

17. One minus the ratio of the smallest possible MSPE to the MSPE of the constant pre­ dictor is called Pearson's correlation ratio 1J� y; that is,

�iY

=

1 - E [Y - J-L(Z)] 2 /Var(Y)

=

Var(J-L(Z) )/Var(Y) .

(See Pearson, 1905, and Doksurn and Samarov, 1995, on estimation of 1]�y .) (a) Show that 1]�y > PiY • where PiY is the population multiple correlation coefficient of Remark 1.4.3. Hint: See Problem 1.4.15. (b) Show that if Z is one-dimensional and h is a 1-1 increasing transformation of 2 ts mvanant un er suc . 2 2 y then ryh(Z)Y atiS, Th d hh = 1Jz . 1J "

Z,

.

.

(c) Let 'L = Y - J-LL(Z) be the linear prediction error. Show that, in the linear model of Remark 1.4.4, €£ is uncorrelated with PL(Z) and 1J� y = p�y· 18. Predicting the past from the present. Consider a subject who walks into a clinic today, at time t, and is diagnosed with a certain disease. At the same time t a diagnostic indicator Zo of the severity of the disease (e.g., a blood cell or viral load measurement) is obtained. Let S be the unknown date in the past when the subject was infected. We are interested in the time Yo = t - S from infection until detection. Assume that the conditional density of Zo (the present) given Y0 = Yo (the past) is

where !L and a2 are the mean and variance of the severity indicator Z0 in the population of people without the disease. Here j3y0 gives the mean increase of Z0 for infected subjects over the time period y0; j3 > 0, y0 > 0. It will be convenient to rescale the problem by introducing Z = (Z0 - J-L)/u and Y = /3Yo/u. (a) Show that the conditional density f(z I y) of Z given Y = y is N(y, !). (b) Suppose that Y has the exponential density

1r(y) = A exp{-.\y}, A > 0, y > 0.

Section 1.7

Problems and Complements

83

Show that the conditional distribution of Y (the past) given Z =

7r(y I z ) � (27r) - l c-1 exp where c

=

{- � [y - (

z

z

}

(the present) has density

- >.) ]2 , y > 0

4>(z - >..) . This density is called the truncated (at zero) normal, N(z - >.. , 1 ) ,

density. Hint: Use Bayes rule.

(c) Find the conditional density rro (Yo I zo) of Yo given Zo (d) Find the best predictor of Yo given Zo E IYo - g( Zo)l. Hint: See Problems 1.4. 7 and 1.4.9.

=

= z0.

zo using mean absolute prediction error

(e) Show that the best MSPE predictor ofY given Z = z is E(Y I Z � z)

�

c-1. - z )

-

(>. - z) .

(In practice, all the unknowns, including the "prior" rr, need to be estimated from cohort studies; see Berman, 1990, and Normand and Doksum, 2000) . 19. Establish 1.4.14 by setting the derivatives of R(a, b) equal to zero, solving for (a, b), and checking convexity.

20. Let Y be the number of heads showing when X fair coins are tossed, where X is the number of spots showing when a fair die is rolled. Find (a) The mean and variance of Y.

(b) The MSPE of the optimal predictor of Y based on X. (c) The optimal predictor of Y given X

= x, x =

1, . . . , 6.

21. Let Y be a vector and let r(Y) and s(Y) be real valued. Write Cov[r(Y), s(Y) I zl for tbe covariance between r(Y) and s(Y) in the conditional distribution of (r(Y), s(Y)) given Z = z. (a) Show that ifCov[r(Y), s(Y)] < oo, then

Cov[r(Y), s(Y)]

�

E{Cov[r(Y), s(Y) I Z]} + Cov{ E[r(Y) I Z], E[s(Y) I Z]}.

(b) Show tbat (a) is equivalent to (1.4.6) when r (c) Show that if Z is real, Cov[r(Y), Z]

�

�

s.

Cov{E[r(Y) I Z], Z}.

(d) Suppose Y1 � a1 + b1Z1 + W and Yz � a2 + b,Z2 + W, where Y and Y2 are 1 responses of subjects 1 and 2 with common influence W and separate influences Zt and Zz, where Z1. Z2 and W are independent with finite variances. Find Corr(Y1 , Yz) using (a).

84

Statistical Models, Goals, and Performance Criteria

Chapter 1

(e) In the preceding model (d), if b1 = b2 and Z1 , Z2 and �V have the same variance cr2• we say that there is a 50% oVerlap between Y1 and Yz. In this case what is Corr(Y1 , Y )? 2 (f) In model (d), suppose that Z1 and z, optimal predictor of Yz given (Yi, Z1, Zz).

are

N(p, u2 ) and

W � N(p0 , u5).

Find the

22. In Example 1.4.3, show that the MSPE of the optimal predictor is u�(l - p�y ). 23. Verify that solving (1.4.15) yields (1.4.14). 24. (a) Let w(y,z) be a positive real-valued function. Then [y - g(z)]2/w(y,z) = 6w(y,g(z)) is called weighted squared prediction error. Show that the mean weighted squared prediction error is minimized by po ( Z ) = EO(Y I Z), where Po (y , z) =

cp(y, z)/w(y, z)

and c is the constant that makes p0 a density. Assume that

Eow (Y, g( Z)) < co for some g and that Po is a density. B(n,z), n > 2 , and suppose that Z has the beta, (b) Suppose that given Z = z, Y ,B(r, s ) density. Find p0(Z) when (i) w(y, z) = 1, and (ii) w(y, z) = z(l - z), 0 < z < 1. �

,

25. Show that EY2 < co if and only if E(Y - c)2 < co for all c.

Hint: Whatever be Y and c, 1

2 - c2 < (Y - c)2 Y 2

=

Y2 - 2cY + c2 < 2 (Y2 + c2). '

i

Problems for Section 1.5 1. Let XI, . . . ' Xn be a sample from a Poisson, P(e), population where e > 0. (a) Show directly that z=:=t Xi is sufficient for 8.

(b) Establish the same result using the factorization theorem. 2. Let n items be drawn in order without replacement from a shipment of N items of which N8 are bad. Let Xi = 1 if the ith item drawn is bad, and = 0 otherwise. Show that L� 1 xi is sufficient for 8 directly and by the factorization theorem. 3. Suppose X1 ,

•

.

•

,

I

'

Xn is a sample from a population with one of the following densities.

8) = 8x9- 1, 0 < X < 1, 8 > 0. This is the beta, ,B(O , !), density. (b) p(x, 8) = 8ax•-1 exp( -8x ) x > 0, 8 > 0, a > 0. (a) p(x

,

"

,

This is known as the Weibull density. (c) p(x, 8)

=

0a9fx(9+1 ), X > a, 8 > 0, a > 0.

1

'

85

Section 1. 7 Problems and Complements

This is known as the Pareto density. In each case, find a real-valued sufficient statistic for (),

a fixed.

4. (a) Show that T1 and T2 are equivalent statistics if, and only if, we can write T2 = H (T1) for some 1-1 transformation H of the range of T1 into the range of T2. Which of the following statistics are equivalent? (Prove or disprove. )

n� 1 Xt and I:� 1 log Xi , Xi > 0 (c) I:� 1 xi and I:� 1 log Xi, Xi > 0 (d) (l:� 1 xi , l:� 1 x;) and (l:� 1 xi, I:� 1 (xi - x?) (e) (I:� 1 Xi, 2:� 1 xl) and (2:� 1 Xi , 2:� 1 (xi X ) 3). 5. Let B = (B" 82) be a bivariate parameter. Suppose that T1 (X) is sufficient for 81 whenever 82 is fixed and known, whereas T2(X) is sufficient for fh whenever 81 is fixed (b)

-

and known. Assume that (}h B2 vary independently, lh E 81, 82 E 82 and that the set S {x : p(x, B) > 0} does not depend on B. =

(a) Show that if T1 and T2 do not depend one, and 81 respectively, then (T1 (X), T2(X))

is sufficient for e.

(T1 (X) , T,(X)) is sufficient for B, T1 (X) is sufficient for 81 whenever 8z is fixed and known, but Tz(X) is not sufficient for 8z, when (}1 is fixed (b) Exhibit an example in which

and known. 6. Let X take on the specified values v1 , vk with probabilities 81, . . . , 8k, respectively. Suppose that X1 , . . . , Xn are independently and identically distributed as X. Suppose that IJ = (81, , Bk) is unknown and may range over the set 8 = { (B, . . . , Bk) : e, > 0, 1 < i < k, L� 1 8i = 1 }. Let Nj be the number of xi which equal Vj· .

.

.

.

.

1

.

(a) What is the distribution of (Nt, . . . , Nk)? (b) Show that N = (N1, . . . , Nk-t) is sufficient for B.

7. Let X1,

. • . 1

Xn be a sample from a population with density p( x , 8) given by -

p(x, B)

0 otherwise. Here e

=

(!",a) with -oo <

11-

< oo, a > 0.

(a) Show that min (X1 , , Xn) is sufficient for fl when a is fixed. (b) Find a one-dimensional sufficient statistic for a when Jl. is fixed. . • .

(c) Exhibit a two-dimensional sufficient statistic for 8.

8. Let X1, . . , Xn be a sample from some continuous distribution Fwith density J, which is unknown. Treating f as a parameter, show that the order statistics X(t ) • . . . , X(n) (cf. Problem B.2.8) are sufficient for f. .

'

''

I !

86

9.

Statistical Models, Goals, and Performance Criteria Let

X1 ,

.

.

•

,

Chapter 1

Xn be a sample from a population with density

fo (x)

a(O) h (x)

if O I

< x < o,

0 othetwise where h(x)

>

0, 0

= (OI , O, ) with -oo

<

OI

< 02 <

oo, and a(O) =

I [J:,' h(x)dx] �

is assumed to exist. Find a two-dimensional sufficient statistic for this problem and apply

02] family of distribution s. 10. Suppose X1, , . . , Xn are U.d. with density f(x, 8) = �e-l.x-S]_ Show that (X{l)• . . . , X(n)). the order statistics, are minimal sufficient. Hint: t, Lx (O) = - L� I sgn(X, - 0), 0 rt {XI, . . . , Xn }, which determines X( I ) , · · · , X(n)• 11. Let X1 , X2 , . . . , Xn be a sample from the unifonn, U(O,B), distribution. Show that X(n) = max{ Xii 1 < i < n} is minimal sufficient for 0. 12. Dynkin, Lehmann, Schejfi 's Theorem. Let P = {Po : () E 8} where Po is discrete concentrated on X = {xi, x,, . . . }. Let p(x, 0) = Pe [X = x] = Lx(O) > 0 on X. Show your result to the U[81,

that

fxx(��) is minimial sufficient.

Hint: 13.

Apply the factorization theorem.

Suppose that X =

bution function

(X1,

• . .

F(x). If F(x)

, Xn) is a sample from a population with continuous distri­ is N(p, u2 ) , T(X) = (X, u2 ), where u2 = n� I l.:(X, ­

X)2, is sufficient, and S(X) = (X(1 1 , . . . , X(n1 ), where X(, 1 = (X(i) - X)ju, is "irrel­ evant" (ancillary) for (J.L, a2 ). However, S(X) is exactly what is needed to estimate the

.

!

'

"shape" of

'

class F = iff

F(x) when F(x) is unknown . The shape ofF is represented by the equivalence {F((· - a)fb) b > 0, a E R). Thus a distribution G has the same shape as F

G E F.

function

:

For instance, one "estimator" of this shape is the scaled empirical distribution

�

F,(x)

jfn,

x(n :S x < x(i+l)• j

=

1, .

..,n-1

x < x(1 ) 1, X > x(n)· � Show that for fixed x, F,((x - x )fu) converges in probability to F(x). Here we are using F to represent F because every member ofF can be obtained from F. 0,

:I

'I

14. Kolmogorov's Theorem.

"

'. •

9,

We are given a regular model with 9 finite.

(a) Suppose that a statistic T(X) has the property that for any prior distribution on the posterior distribution of

9 depends on x only through T(x).

Show that T(X) is

sufficient. (b) Conversely show that if T(X) is sufficient, then, for any prior distribution, the

posterior distribution depends on x only through T

(x) .

i

j l

'

Section 1.7

Problems

87

and Complements

Hint: Apply the factorization theorem. 15. Let Xh . , Xn be a sample from f(x 0), () E R. Show that the order statistics arc minimal sufficient when f is the density Cauchy f(t) � 1 /Jr( l + t2). 16. Let X Xrn; Y1 , . . , Y,l be independently distributed according to N(p,, 0"2) and . .

1,

.

· -

.

•

,

.

N(TJ, r2), respectively. Find minimal sufficient statistics for the following three cases: (i) p,, TJ, (ii)

p,,

17 < oo,

0 < a, T.

a = T and p,, 17, O" arc arbitrary.

(iii) p, =

17.

a, T are arbitrary: -oo < 1] and p,, a, T are arbitrary.

In Example 1.5.4, express t1 as a function of Lx(O,

I)

and

Lx(l, 1).

Problems to Section 1.6 1. Prove the assertions of Table 1 . 6. 1. 2.

Suppose

X1 ,

• • •

, Xn is as in Problem 1.5.3. In each of the cases (a), (b) and (c), show

that the distribution of X forms a one-parameter exponential family. Identify ,.,, B, T, and

h.

3.

Let X be the number of failures before the first success in a sequence of Bernoulli trials

with probability of success 9. Then Pe [X � k] �

the

geometric distribution (9 (B)).

(I - 9)'9, k � 0, I, 2, .

.

. This is called

(a) Show that the family of geometric distributions is a one-parameter exponential fam­

ily with

(b)

T(x)

�

x.

Deduce from Theorem 1.6.1 that if X�o

. . . , Xn is a sample from 9(9), then the

2.:� 1 Xi form a one-parameter exponential family. (c) Show that E� xi in part (b) has a negative binomial distribution with parameters

distributions of

l

(n,9) defined by Pe iL:7 1 X, = k] �

( � ) n

+

- l

( 1 - 9)'9", k = 0 , 1 , 2 , . . . (The

negative binomial distribution is that of the number of failures before the nth success in a sequence of Bernoulli trials with probability of success

0.) Hint: By Theorem 1.6.1, Pe [L� 1 X, = k] = c,(l - 9)'0", 0 < 9 < I. If =

"\" � c,w• =

k=O

I

(1

-w

)n

, 0 < w < I,

then

Ck �

I d' n w ) (I k! fiw k w=O

Which of the following families of distributions are exponential families? (Prove or disprove.)

4.

(a) The U(O, 9) fumily

88

Statistical Models, Goals, and Performance Criteria

Chapter 1

(b)p(." , O) = {exp[- 2 log 0 + log(2x)]}l[x E (0,0)] (c) p(x, O) = �. x E { 0. 1 +0, . . . , 0.9 + 0) (d) The N(O, 02) family, 0 > 0 (e) p( x, O ) = 2(x + 0)/(1 + 20), 0 < x < I, 0 > 0

(f) p(x, 9) is the conditional frequency function of a binomial, B(n, 0), variable X,

given that X > 0.

5. Show that the following families of distributions are two-parameter exponential families and identify the functions 1], B, T, and h.

(a) The beta family.

(b) The gamma family.

6. Let X have the Dirichlet distribution, D( a) , of Problem 1.2.15. Show the distribution of X form an r-parameter exponential family and identify fJ, B, T, and h. 7. Let X = ( (X 1 , Y1 ), . . . , (X., Yn)) be a sample from a bivariate normal population. Show that the distributions of X form a five-parameter exponential family and identify 'TJ, B, T, and h. 8. Show that the family of distributions of Example 1.5.3 is not a one parameter CX(Xlnential family. Hint: If it were, there would be a set A such that p(x, 0) > 0 on A for all 0. 9. Prove the analogue of Theorem 1.6.1 for discrete k-parameter exponential families. 10. Suppose that f(x, B) is a positive density on the real line, which is continuous in x for each 0 and such that if (X1, X2 ) is a sample of size 2 from f (·, 0), then X1 + X2 is sufficient for B. Show that /(·, B) corresponds to a one-arameter exponential family of distributions with T(x) = x. Hint: There exist functions g(t, 0), h(x�, x2) such that log f(x�, 0) + log j(x2, 0) = g(x1 + x2, 0) + h(x1, x2). Fix Oo and let r(x, 0) = log f(x, 0) - log f(x, Oo), q(x, 0) = g(x,O) - g(x,Oo). Then, q(x 1 + x2,0) = r(x, , O) + r(x,, O), and hence, [r(x� , O) ­ r(O, 0)] + [r(x2, 0) - r(O, 0)] = r(x1 + x2, 0) - r(O, 0).

' '

I

11. Use Theorems 1.6.2 and 1.6.3 to obtain moment-generating functions for the sufficient statistics when sampling from the following distributions.

(a) normal, () = (p, a2) (b) gamma, r(p, e = >., p fixed

>.),

(c) binomial (d) Poisson (e) negative binomial (see Problem 1.6.3) () = (p, (0 gamma, r(p,

>.),

.

-

>.) .

�---

------

II

'

Section 1. 7

89

Problems and Complements

12. Show directly using the definition of the rank of an ex}X)nential family that the multi­ nomial distribution, M(n; B,, . . . , B.) , O < B; < 1, 1 < j < k. L:�" 1 B; = 1, is of rank

k - 1.

13. Show that in Theorem 1.6.3, the condition that E has nonempty interior is equivalent to the condition that £ is not contained in any ( k I)-dimensional hyperplane . �

•

14. Construct an exponential family of rank k for which £ is not open and A is not defined on all of t:. Show that if k = 1 and &0 # 0 and A, A are defined on all of t:, then Theorem 1.6.3 continues to hold. 15. Let P = {Po : B E 8} where Po is discrete and concentrated on X = {x 1 , x2, . . . ) , and let p(x, B) = Po [X = x]. Show that if P is a (discrete) canonical exponential family generated b(, (T, h) and &0 # 0, then T is minimal sufficient. Hint: �;,• Lx(TJ) = T; (X) - E'IT;(X). Use Problem 1.5.12. 16. Life testing. Let X1, . . . , Xn be independently distributed with exponential density < < < 2 1 the > ordered X's be denoted by Y1 Y 2 0 for 0, and let · · · e-xl (28)x Y It is assumed that Y1 becomes available first, then Yz, and so on, and that observation is continued until Yr has been observed. This might arise, for example, in life testing where each X measures the length of life of, say, an electron tube, and n tubes are being tested simultaneously. Another application is to the disintegration of radioactive material, where n is the number of atoms, and observation is continued until r a-particles have been emitted. Show that ..

(i) The joint distribution of Y1 ,

1

•

[ p

n!

(28)' (n - r)! ex (ii) The distribution of II: :

l

.

.

, Yr is an exponential family with density

L::� r Yi + (n - r)yr 28

-

]

< < < , 0Yl - . . . - Yr ·

2 + (n r)Yrl/B is x with 2r degrees of freedom. Y;

(iii) Let Yi , Y2 , denote the time required until the first, second, . . . event occurs in a Poisson process with parameter 1/28' (see A.16). Then Zr = Y1/8', Z2 = (Y2 Y1 )/8', z, = (Y3 - Y2)/8', . . . are independently distributed as x2 with 2 degrees , Yr is an exponential family with density of freedom, and the joint density of Y1 , •

.

.

•

1

(28')

r) ( Y ' exp 28'

•

.

'

0<

Yl

< ... <

Yr·

The distribution of Yr/B' is again x2 with 2r degrees of freedom. (iv) The same model arises in the application to life testing if the number n of tubes is held constant by replacing each burned-out tube with a new one, and if Y1 denotes the time at which the first tube bums out, Y2 the time at which the second tube burns out, and so on, measured from some fixed time.

'

. 90

Statistical Models, Goals, and Performance Criteria

Chapter 1

[(ii): The random variables z, � (n - i + l)(Y; - l� � t )/9 (i � 1, . . . . 1· ) are inde­ pendently distributed as x2 with 2 degrees of freedom, and [L� 1 Yi + (n - 1')Yr]/B =

I: :� t Z,.]

17. Suppose that (Tk X l , h) generate a canonical exponential family P with parameter 1J kxl and E = Rk. Let (a) Show that Q is the exponential family generated by IIL T and h exp{ cTT}, where lh is the projection matrix of T onto L � { '1 : '1 � BIJ + c } .

(b) Show that ifP has full rank k and B is of rank l, then Hint: If B is of rank l, you may assume

Q has full rank I.

18. Suppose Yt . . . , Yn are independent with Yi "' N(/31 + /32 Zi, a 2 ), where z1 , . . . , Zn are covariate values not all equal. (See Example 1.6.6.) Show that the family has rank 3. Give the mean vector and the variance matrix of T. ,

19. Logistic Regression. We observe

(z1, Y1 ) , . . . , (zn, Yn) where the Y1, , Yn are inde­ pendent, Yi "-' B(�, .Xi). The success probability .Xi depends on the characteristics zi of .

.

•

the ith subject, for example, on the covariate vector zi = (age, height, blood pressure)T. The function l(u) � log[u/(1 - u)[ is called the logit function. In the logistic linear re­ gression model it is assumed that l ( .\,) � zf(3 where (3 ((31 , . . . , f3d )T and z, is d x 1. Show that Y (Y1, Yn) T follow an exponential model with rank d iff z1, . . . , zd are not collinear (linearly independent) (cf. Examples 1.1.4, 1.6.8 and Problem 1.1.9). =

• . •

=

,

20. (a) In part II of the proof of Theorem 1.6.4, fill in the details of the arguments that Q is generated by (171 - 'lo)TT and that �(ii) =�(i). (b) Fill in the details of part Ill of the proof of Theorem 1.6.4.

21. Find JJ.('I)

I

EryT(X) for the gamma, r(a, .\), distribution, where 9 � (a, ,\) .

�

Xn be a sample from the k-parameter exponential family distribution 22. Let X 1 , (1.6.10). Let T � (I;� 1 T1(X,), . . . , I;� 1 Tk(Xi)) and let •

I

•

•

,

s

�

{(ry1(1J), . . . , ryk(IJ)) , e E 8).

Show that if S contains a subset of k + 1 vectors v0, . . . , Vk+l so that vi - v0, 1 < i < k, are not collinear (linearly independent), then T is minimally sufficient for 8.

I

. i I, '

,

:

23. Using (1.6.20), find a conjugate family of distributions for the gamma and beta fami­ lies. (a) With one parameter fixed.

(b) With both parameters free.

Section 1.7

91

Problems and Complements

24. Using (1 .6.20), find a conjugate family of distributions for the normal family using as parameter 0 (01 , 02) where 01 = Eo (X), Oz = 1/(VaroX) (cf. Problem !.2.12). =

25. Consider the linear Gaussian regression model of Examples 1.5.5 and 1.6.6 except with cr z known. Find a conjugate family of prior distributions for (/31 , /32) T.

26. Using (1 .6.20), find a conjugate family of distributions for the multinomial distribution. See Problem !.2.15.

27. Let 'P denote the canonical exponential family genrated by T and h. For any TJo E £, set h0(x) = q(x, 'lo ) where q is given by (!.6.9). Show that P is also the canonical exponential family generated by T and h0.

28. Exponentialfamilies are maximum entropy distributions. The entropy h(f) of a random variable X with density f is defined by

h(f) = E(- log f(X))

=

- J: [logf(x)Jf(x)dx.

This quantity arises naturally in information in theory; see Section 2.2.2 and Cover and Thomas (1991). Let S = {x : f(x) > 0}. {a) Show that the canonical k-parameter exponential family density

f(x, 'I) = exp

•

1/0 + L 'l;r; (x) - A( 'I) j:=l

, xES

maximizes h(f) subject to the constraints

f(x)

> 0,

Is f(x)dx = 1, Is f(x)r;(x) = a; , 1 < j < k,

where r-,o, . . , TJk are chosen so that f satisfies the constraints. Hint: You may usc Lagrange multipliers. Maximize the integrand. .

(b) Find the maximum entropy densities when r;(x) = xi and (i) S = (0, oo), k I, <>t > 0; (ii) S R, k = 2 , at E R, az > 0; (iii) S R, k 3, a1 E R, o:z > 0, a, E R. =

=

=

=

29. As in Example 1.6.11, suppose that Y 1 , . , Y are i.i.d. Nv(f.L, E) where f.L varies freely in RP and E ranges freely over the class of all p x p symmetric positive definite matrices. Show that the distribution of Y (Yt, . . . Yn ) is the p(p + 3) /2 canonical exponential family generated by h = l and the p(p + 3)/2 statistics .

n

.

=

,

n

n

i=l

i=I

. . . , Y;p). Show that [ is open and that this family is of rank p(p + 3) /2. Hint: Without loss of generality, take n = 1. We want to show that h = 1 and the m = p(p + 3) /2 statistics T; (Y) = Yj, 1 < j < p, and T;1(Y) Y;Yi, 1 < j < l < p,

where Y;

=

(Yil

,

=

92

Statistical Models, Goals, and Performance Criteria

Chapter 1

generate Nv(J.l, E). As E ranges over all p x p symmetric positive definite matrices. so does E-1• Next establish that for symmetric matrices M,

J exp{-uTMu}du < oo iff M is positive definite

by using the spectral decomposition (see B.10. 1 .2)

p

M = L AjejeJ for e 1 , . . . , ep orthogonal, )..J E R.

j=l

To show that the family has full rank m, use induction on p to show that if Z1, . . . , Zv are i.i.d. N(O, and if Bpxp = (b; is symmetric, then

l)

1)

p P I: a;Z; + L b;1 Z;Z1 =

j,l

j., l

c

= P(aTZ + ZTBz = c) = o

a = 0, B = 0, = 0. Next recall (Appendix B.6) that since Y � Np(/', E), then Y = SZ for some nonsingular p x p matrix S. 30. Show that if X 1 , . ,Xn are i.i.d. Nv(8,E0) given 6 where �0 is known, then the Np(A, f) family is conjugate to N (B Eo), where A varies freely in RP and r ranges over c

unless

.

.

.

,

all p x p symmetric positive definite matrices.

31. Conjugate Normal Mixture Distributions. A Hierarchical Bayesian Normal Model. Let {(I';, r; ) : j k} be a given collection of pairs with l'i E R, > 0. Let (J.I., be a random pair with >.; = P((!', u = (!';, r; ) , 0 2::��1 >.; = Let 8 be a random variable whose conditional distribution given (JL, u = (P,J, Tj) is normal, N(p,j, rJ). Consider the model X = 8 + t:, where 8 and e are independent and € N(O, a3) , a� known. Note that 8 has the prior density

1< <

.

•

'

•

' •

I

)

)

< >.; < 1,)

Tj

u)

1.

"'

•

k

1r(B) = L A;'Pr, (B - !'; ) j=l where 'l'r denotes the N(O, uon. •

i� ''' ' ' ': ' •

(1 .7.4)

T2) density. Also note that (X I B) has the N(B, u5) distribu-

(a) Find the posterior k

"

I

•

1r(B I x) = LP((tt,u) (!'j, Tj ) I x)1r(B I (!';,r;),x) j=l =

and write it in the fonn k

L >.;(x)'Pr,(x)(B - !'j(x)) j=l

Section

1.7

93

Problems ;3nd Complements

for appropriate .\J (x ) , TJ (x) and fLJ (x ). This shows that ( 1. 7 .4) defines a conjugate prior for the N(B, 176), distribution.

(b) Let Xi = 8 + Ei, l < i <

where 8 is as previously and E t , . . . , En are i.i.d. N(O, 17�). Find the posterior 7r (B I x , , . . . , Xn ) , and show that it belongs to class (1.7 .4). Hint: Consider the sufficient statistic for p(x I B). n,

32. A Hierarchical Binomial-Beta Model. Let { (r1 , sJ) : 1 < j < k} be a given collection of pairs with r; > 0, s1 > 0, let (R, S) be a random pair with P(R = r1 , S = s;) = >.;,

0 < .\1 < 1, E7= l .\1 = 1, and let 8 be a random variable whose conditional density given R = r, S = s is beta, (J(r, s ) . Consider the model in which (X the binomial, B(n1 e), distribution. Note that e has the prior density

1r(B, r, s)

1r(B) =

I B) has

k

L -\;1r(B, r1 , s1 ) .

( I .7.5)

j=l

Find the posterior

k

1r(B I x) = L P(R = r; , S = s; I x)1r(B I (r; , s;),x) j=l

and show that it can be written in the form L: >-1 (x)1r(B,r;(x),s;(x)) for appropriate >-;(x), r;(x) and s;(x). This shows that (1.7.5) defines a class of conjugate priors for the B( n, B) distribution.

33. Let p(x, ry) be a one parameter canonical exponential family generated by T(x) = x and h(x), x E X C R, and let ,P(x) be a nonconstant, nondecreasing function. Show that E,,P(X) is strictly increasing in

Hint:

ry.

Cov,(,P(X), X)

� E{(X - X')i,P(X) - ,P(X')]) where X and X' are independent identically distributed as 34. Let (Xt 1

•

•

•

, Xn) be a stationary Markov chain with two states 0 and

P[Xi = Ei 1 x1 = c1, . . . , Xi - 1 where (i) (ii)

( P10 Poo

Pol Pn

)

=

1.

That is,

Ei-d = P[Xi = ci 1 xi-1 = Ei-d = p.,._l.,.

is the matrix of transition probabilities. Suppose further that ·

Poo = PII = p, so that, Pw = Pol

P[X1

X (see A.l l.12).

= OJ = P(X1

=

1] = �·

= 1 - p.

94

Statistical

Models,

Goals, and

Performance C riteria

Chapter 1

(a) Show that if 0 < p < 1 is unknown this is a full rank, one-parameter exponential family with T = N00 + N11 where N1j the number of transitions from i to j. For example, 01011 has No1 = 2, N11 = 1, Noo = 0, N10 = I . (b) Show that E(T) = (n - 1)p (by the method of indicators or otherwise). 35. A Conjugate Priorfor the Two-Sample Problem. Suppose that X1 , . . . , Xn and Y1 , . . . , Yn are independent N'(fl1, a2) and N(�J 2 , a2) samples, respectively. Consider the prior 7r for which for some r > 0, k > 0, ro-2 has a x� distribution and given a2, p,1 and fL2 are independent with N(�1 , a2 Ikt) and N(6, a2 I k,) distributions, respectively, where �j E R, kj > 0, j = 1, 2. Show that 1r is a conjugate prior. 36. The inverse Gaussian density, IG(J..t, A), is j(X,Jl., .\)

=

1 [.\I27T] i2x-3i 2 exp{ -.\(x - J1.)2/2J1.2x}, x > 0,

J1.

> 0, .\ > 0.

Show that this is an exponential family generated by T(X) = - ; (X, x- 1 )T and h(x) = (27r) -lf2x- 3/2 (b) Show that the canonical parameters TJt , ry are given by ry1 = fL-2 A, 1J2 = .\, and 2 that A( q1 , 172 ) = (� log(q2 ) + JihijZ) , £ = [O, oo) x (O,oo). (c) Fwd the moment-generating function of T and show that E(X) = Jl., Var(X) Jl.-3 .\, E(x-') = Jl._, + .\_ ,, Var(x-') = (.\Jl.)-' + 2.\ -2. (d) Suppose J1. = Jl.o is known. Show that the gamma family, f(u,,B), is a conjugate pnor. (e) Suppose that .\ = .\0 is known. Show that the conjugate prior formula (1.6.20) produces a function that is not integrable with respect to fL. That is, n defined in (1.6.19) is empty. (f) Suppose that J1. and .\ are both unknown. Show that (1.6.20) produces a function that is not integrable; that is, f! defined in (1.6.19) is empty. 37. Let X1, . . . , Xn be i.i.d. as X � Np(O, �0) where �0 is known. Show that the conjugate prior generated by (1.6.20) is the Afv (q0 , 761) family, where 7Jo varies freely in RP, T6 > 0 and I is the p x p identity matrix. 38, Let X; = (Z;, Y;)T be i.i.d. as X = (Z, Y)T, 1 < i < n, where X has the density of Example 1.6.3. Write the density of X1 , . , Xn as a canonical exponential family and identify T, h, A, and £. Find the expected value and variance of the sufficient statistic. 39. Suppose that Yt , . . . , Yn are independent, Yi N(fLi, a2 ), n > 4. (a) Write the distribution of Y1 , , Yn in canonical exponential family form. Identify T, h, 1), A, and E. (b) Next suppose that fLi depends on the value Zi of some covariate and consider the submodel defined by the map 11 : (11 1 , 112 , II3)T � (p7, a2JT where 11 is determined by /Li = cxp{Bt + Bzzi}, Zt < z2 < · · · < Zn; a 2 = 83 (a)

-

•

'

'P'· ',,' '

.. . ' ..

.

'

H '

......,

, ,, ,.

' ' ,

"

I I' I'

.

.

:

.

•

,I '' '. •

i

I'

··---

------

'

: : •

Section 1.8

95

Notes

where 91 E R, 02 E R, 03 > 0. This model is sometimes used when Jli is restricted to be positive. Show that p(y, 0) as given by ( is a curved exponential family model with l 3.

1. 6.12)

=

40. Suppose Y1 , . . . , Y;.l are independent exponentially, E' ( ,\i), distributed survival times, n > 3. (a) Write the distribution of Y1 , . . . , Yn in canonical exponential family form. Identify T, h, '1. A, and E.

(b) Recall that J..li = E(Yi) = Ai 1 . Suppose /Ji depends on the value Zi of a covariate. Because fti > 0, J.Li is sometimes modeled as

J.Li = cxp{Or + fhzi}, i =

1, . . .

, n

where not all the z's are equal. Show that p(y, fi) as given by nential family model with l =

2.

1.8

(1.6.12) is a curved expo­

NOTES

Note for Section 1.1

(1)all dominated For the measure theoreticaHy minded we can assume more generally that the Po are p(x, 9) denotes df;11 , the Radon Nikodym by a a finite measure and derivative.

ft

that

Notes for Section 1,3

� (I) More natural in the sense of measuring the Euclidean distance�between the estimate (} and the "truth" 0. Squared error gives much more weight to those (} that are far away from (} than those close to (}. (2) We define the lower boundary of a convex set simply to be the set of all boundary points r such that the set lies completely on or above any tangent to the set at r. Note for Section 1.4 (I) Source: Hodges, Jr., J. L., D. Kretch, and R. S. Crutchfield. Statlab: An Empirical Introduction to Statistics. New York: McGraw-Hill, 1975. Notes for Section 1.6 (I) Exponential families arose much earlier in the work of Boltzmann in statistical mechan­ ics as laws for the distribution of the states of systems of particles-see Feynman ( 1 963), for instance. The connection is through the concept of entropy, which also plays a key role in information theory-see Cover and Thomas ( 1991 ).

(2) The restriction that's x E Rq and that these families be discrete or continuous is artifi­ cial. In general if J.L is a a finite measure on the sample space X, p(x, (}) as given by

(1.6.1)

'

'

96

Statistical Models, Goals, and Performance Criteria

can be taken to be the density of

X with respect to J-L-see Lehmann (1997), for instance.

Chapter 1

This permits consideration of data such as images, positions, and spheres (e.g., the Earth), and so on.

Note for Section 1.7 ( I ) uT Mu > 0 for all p x

1.9

1 vectors u # 0.

REFERENCES

BERGER, J. 0., Statistical Decision Theory and Bayesian Analysis New York:

Springer, 1985.

BERMAN, S. M., ''A Stochastic Model for the Distribution ofHIV Latency Time Based on T4 Counts," Biometika. 77, 733-74 1 (1990).

BICKEL, P. J., "Using Residuals Robustly 6, 266--291 (1978).

1: Tests for Heteroscedasticity, Nonlinearity," Ann. Statist.

BLACKWELL, D. AND M. A. GIRSHICK, Theory of Games and Statistical Decisions New York:

Wiley,

1954.

Box, G. E. P., ''Sampling and Bayes Inference in Scientific Modelling and Robustness (with Discus­ sion)," J. Royal Statist. Soc. A 143, 383-430 (1979).

BRowN, L., Fundamentals of Statistical Exponential Families with Applications

in Statistical Deci­

sion Theory, IMS Lecture Notes-Monograph Series, Hayward, 1986.

CARROLL, R. J. AND D. RuPPERT, Transformation and Weighting in Regression New York:

Chapman

and Hall, 1988.

CoVER, T.

M. AND J. A. THOMAS, Elements of Information Theory New York: Wiley,

1991.

DE GROOT, M. H., Optimal Statistical Decisions New York: McGraw-Hill, 1969.

DoKSuM,

K A

AND A. SAMAROV, "Nonparametric Estimation

of Global Functionals and a Measure

of the Explanatory Power of Covariates in Regression,'' Ann. Statist. 23, 1443-1473 ( 1995).

FERGUSON, T. FEYNMAN,

S., Mathematical Statistics New York: Academic Press, 1967.

R. P., The Feynmo.n Lectures on Physics, v. I , R. P. Feynman, R. B. Leighton, and M.

Sands, Eels., Ch. 40 Statistical Mechanics of Physics Reading, MA: Addison-Wesley, 1963.

GRENANDER, U. AND M. ROSENBLATT, Statistical Analysis of Stationary Time Series New York:

Wi­

ley, 1957.

HorXJES, JR., 1. L., D. KRETcH AND R. S. CRUTCHFIELD, Statlab: An Empirical introduction to Statis­ tics New York: McGraw-Hill, 1975.

KENDALL, M. G. AND A. STUART, The Advanced Theory of Statistics, Vols. II, ffi New York:

Hafner

Publishing Co., 1961, 1966.

L6HMANN, E. L., "A Theory of Some Multiple Decision Problems, 1-25, 547-572 (1957).

L6HMANN, E. L., "Model Specification: Statist. Science 5, 160-168 (1990).

LatMANN, E. L., .. . . .' •

'

I

I and II," Ann. Math Statist. 22,

The Views of Fisher and Neyman, and Later Developments,"

Testing Statistical Hypotheses, 2nd ed. New York: Springer, 1997.

Section 1. 9

97

References

D. V., Introduction to Probability and Statistics from a Bayesian Point of View, Part I: Probability; Part II: Inference London: Cambridge University Press, 1965.

LINDLEY,

MANDEL, J., The Statistical Analysis of Experimental Data

New York: J. Wiley & Sons, 1964.

A. DOKSUM, "Empirical Bayes Procedures for a Change Point Problem with Application to HIV/AIDS Data," Empirical Bayes and Likelihood Inference, 67-79, Editors: S. E. Ahmed and N. Reid. New York: Springer, Lecture Notes in Statistics, 2000.

NORMAND, S-L. AND K.

K., "On the General Theory of Skew Correlation and Nonlinear Regression," Proc. Roy. Soc. London 71, 303 (1 905). (Draper's Research Memoirs, Dulan & Co, Biometrics Series II.)

PEARSON,

RAIFFA,

H. AND R. SCHLAIFFER, Applied Statistical Decision Theory, Division of Research, Graduate

School of Business Administration, Harvard University, Boston, 1961. SAVAGE., L. J., The Foundations of Statistics, J. Wiley & Sons,

New York, 1954.

SAVAGE., L. J. ET AL., The Foundation of Statistical Inference London: SNEDECOR, G. W. AND W. G.

Press, 1989. B. and Hall, 1986.

WErHERILL, G.

AND K.

Methuen & Co., 1962.

COCHRAN, Statistical Methods, 8th Ed. Ames, lA: Iowa State University

D. GLAZEBROOK, Sequential Methods in Statistics New York: Chapman

I I

'

'

Chapter 2

METHODS OF ESTIMATION

2.1 2.1.1

BASIC HEURISTICS O F ESTIMATION Minimum Contrast Estimates; Estimating Equations

P E P, usually parametrized as P = Our basic framework is as before, X E X, X {Po : 8 E 8}. In this parametric case, how do we select reasonable estimates for 8 itself? """

-

That is, how do we find a function B(X) of the vector observation X that in some sense "is close" to the unknown 8? The fundamental heuristic is typically the following. We consider a function that we shall call a contrastfunction

p : X x 8 ---> R and define

E8,P(X, 8). As a function of 8, D(80) 9) measures the (population) discrepancy between 8 and the true value 80 of the parameter. In order for p to be a contrast function we require that D(80 1 9) is uniquely minimized for 8 = 60• That is, if P90 were true and we knew D(Bo, 8) as a function of 8, we could obtain 80 as the minimizer. Of course, we don't know the truth so this is inoperable, but in a very weak sense (unbiasedncss), p(X, 8) is an estimate of D(80, 8). So it is natural to consider 8(X) minimizing p(X, 8). This is the most general form of the minimum contrast estimate we shall consider in the next section. Now suppose 8 is Euclidean c Rd, the true 80 is an interior point of 8, and 8 D(Bo, 8) is smooth. Then we expect D(8o, 8)

-----Jo

v8 D(80, 8)

=

o

(2. 1 . 1 )

where V denotes the gradient,

Arguing heuristically again we are led to estimates B that solve

v8p(X, e)

=

o.

(2.1.2)

The equations (2.1.2) define a special form of estimating equations.

99

100

Chapter 2

Methods of Estimation

More generally, suppose we are given a function W and define V(l1o, 11) � Suppose V (80, 8) =

:

XxR d - Rd, W - ('ljl1,

E11, w(X, 11).

.

.

•

, '1/Jd)T

(2. 1 .3)

�

0 has 80 as its unique solution for all 80 E 8. Then we say 8 solving �

w(X,11)

�o

(2.1 .4)

is an estimating equation estimate. Evidently, there is a substantial overlap between the two classes of estimates. Here is an example to be pursued later.

Example 2.1.1. Least Squares. Consider the parametric version of the regression model of Example 1.1.4 with l'(z) = g({3, z), {3 E Rd, where the function g js known. Here the data function n} where Yi , . . . , Yn are independent. A natural are X = {(zi, Yi) : 1 ,$

(I)

i<

p(X, /3) to consider is the squared Euclidean distance between the vector Y of observed , g({3, Zn))T That is, we take Yi and the vector expectation of Y, JL(z) = (g({3, z 1), •

.

.

n

p(X, {3) = IY - �tl2 =

�)Yi - g({3, z;))'.

i=l

(2. 1 .5)

Strictly speaking P is not fully defined here and this is a point we shall explore later. But, for convenience, suppose we postulate that the Ei of Example 1.1.4 are i.i.d. N(O, Then {3 parametrizes the model and we can compute (see Problem 2.1.16),

a5).

E(3 ,P(X, {3)

D (f3o. {3)

n

no-;l + 2)g({30, z,) - g({3 , z, )]2 ,

(2.1.6)

i=l

J

!

which is indeed minimized at {3 = {30 and uniquely so if and only if the parametrization is � identifiable. An estimate {3 that minimizes p(X, {3) exists if g({3, z) is continuous and

' •

lim{)g(fl, z) l : lf31 �

(Problem 2.1.10). The estimate {3 is called the least squares estimate. � If, further, g({3, z) is differentiable in {3, then {3 satisfies the equation alently the system of estimating eq!.lations,

� � 8g � � 8g � ({3, z,)g(/3, z,), ({3, z;)Y; = LL813 i=l 813 i=l J

I'

oo} = oo

3

In the important linear case,

(2.1.2) or equiv(2.1 .7)

I

I

I

d

g({3, zi) = 'L, ziJ,BJ and zi = (Zit, . . . , Zid)T j=l '

• •

.i

•

•

I

•

Section 2.1

Basic Heuristics o f Estimation

101

the system becomes n

d

i=l

=I

I.>'iY; = L k

n

� L z-t;-z-k t

(2.1.8)

i=l

the normal equations. These equations are commonly written in matrix form (2.1 .9)

where Zv = ll ziJIInx d is the design matrix. Least squares, thus, provides a firs t example of both minimum contrast and estimating equation methods. We return to the remark that this estimating method is well defined even if the ci are not i.i.d. a5). In fact, once defined we have a method of computing a statistic {!J from the n}, which can be judged on its merits whatever the true P data X = {(zi 1 Yi)1 governing X is. This very important example is pursued further in Section 2.2 and Chapter 0 6.

N(O,

1
Here is another basic estimating equation example. Example 2.1.2. Method of Moments (MOM). Suppose X1, ,Xn are i.i.d. as X � P8 , 8 E Rd and 8 is identifiable. Suppose that l't ( 8), . . . , I'd (8) are the first moments of the population we are sampling from. Thus, we assume the existence of . • .

d

Define the jth sample moment /11 by,

1 <j

< d.

To apply the method of moments to the problem of estimating 9, we need to be able to express 9 as a continuous function g of the first moments. Thus, suppose

d

8 � (1'1(8), . . , !'d (O)) .

is from Rd to Rd . The method of moments prescribes that we estimate 9 by the solution of

1-1

�

Pi = l'i (O) , 1 :O j


if it exists. The motivation of this simplest estimating equation example is the law of large numbers: For X ,..._, Po, fiJ converges in probability to flJ(fJ). More generally, if we want to estimate a Rk-valued function q(B) of 9, we obtain a MOM estimate of q( 8) by expressing q(8) as a function of any of the first moments 1'1, . . . , I'd of X, say q(O) = h(l'l, . . . , I'd), > k, and then using h(j.t�, . , /id) as the estimate of q(8).

d

d

.

.

Methods of Estimation

102

Chapter 2

For instance, consider a study in which the survival time X is modeled to have a gamma distribution, f(u, A), with density [.\"/r (a)]x"- 1 exp{ -.\x}, x > O; cx > O, .\ > 0. In this case f) � (u, .\), l'l � E(X) � n /.\, and 1'2 � E(X2) � a(l + u) /.\2 Solving

for (} gives

"�

(M•/a)2 ,

.\ � i'•/a2,

=

iii �

(X/<1 )2 ; ), � X !a'

where o-2 p,2 -p,i and &2 = n -lEX[ - X2• In this example, the method of moment es­ timator is not unique. We can, for instance, express (} as a function of J..t t and /-L3 = E(X3 ) D and obtain a method of moment estimator based on /11 and ji3 (Problem

2.1.11).

Algorithmic issues

We note that, in general, neither minimum contrast estimates nor estimating equation solutions can be obtained in closed fonn. There are many algorithms for optimization and root finding that can be employed. An algorithm for estimating equations frequently is quick and M used when computation of M(X, ·) = D'I'(X, ·) - J�$' (X, ·JJ dxd J is nonsingular with high probability is the Newton-Raphson algorithm. It is defined by initializing with 90, then setting (2. 1 . 1 0)

.!

• • •

This algorithm and others will be discussed more extensively in Section 2.4 and in Chap­ ter 6, in particular Problem

6.6.10.

2.1.2

The Plug-In and Extension Principles

We can view the method of moments as an example of what we call the plug-in (or substitu­ tion) and extension principles, two other basic heuristics particularly applicable in the i.i.d. case. We introduce these principles in the context of multinomial trials and then abstract them and relate them to the method of moments. Example 2.1.3. Frequency Plug-in(2) and Extension. Suppose we observe multinomial trials in which the values v1, Vk of the population being sampled are known, but their respective probabilities p1, . . , Pk are completely unknown. If we let X1, , Xn be i.i.d. as X and Ni = number of indices j such that Xi = Vi, then the natural estimate of Pi = P[X = vi] suggested by the law of large numbers is Njn. the proportion of sample values equal to vi. As an illustration consider a population of men whose occupations fall in one of five different job categories, 3, 4, or 5. Here k 5, vi = i, i = . , 5, Pi is the proportion of men in the population in the ith job category and Njn is the sample proportion in this category. Here is some job category data (Mosteller, I 968). .

.

=

1, ..

•

.

,

...

1, 2,

'

I

I

•

'

Section 2.1

103

Basic Heuristics of Estimation

Job Category l

23

2 84

3 289

4 217

5 95

0.03

0.12

0.41

0.31

0.13

I

•

Ni �

Pi

-

n = 2::,�1 Ni

=

Lio)Pi = j

708

for Danish men whose fathers were in category 3, together with the estimates Pi = Next consider the more general problem of estimating a continuous function

Ni fn.

q(p1,

.

•

•

,

Pk) of the population proportions. The frequency plug-in principle simply proposes to re­ place the unknown population frequencies p1, . . . , Pk by the observable sample frequencies

Ntfn, . . . , Nkjn.

That is, use

(2. 1 . 1 1 ) to estimate q(p 1 ,

. . .

,Pk) ·

For instance, suppose that in the previous job category table,

categories 4 and 5 correspond to blue-collar jobs, whereas categories white-collar jobs. We would be interested

q(p,, · · · ,Ps)

=

2 and 3 correspond to

in estimating

(p4 + Ps) - (P2

+ P3) ,

the difference in the proportions of blue-collar and white-collar workers. If we use the frequency substitution principle, the estimate is

0.44 - 0.53

=

-0.09. Equivalently, let P denote p = (p1, . .

which in our case is

,pk) with Pi

=

=

·v,], I < i < k, and think of this model as P = {all probability distributions P on {v1 , . . . , vk} }. Then q ( p ) can be identified with a parameter v : P � R, that is, v(P) = (p4 + p5) - (p, + p,), and the frequency plug-in principle simply says to replace P = (p,, . . . , Pk) in v(P) by D P = ( �� , . . . , �,. ) the multinomial empirical distribution of X1, , Xn. .

P[X

,

.

Now suppose that the proportions

p1, . . . , Pk

functions of some d-dimensional parameter 8 =

a component of 8 or more generally a function

.

•

do not vary freely but are continuous

(fh, .

q(8) .

. , (}d) and that we want to estimate

.

Many of the models arising in the

analysis of discrete data discussed in Chapter 6 are of this type. Example

2.1.4.

Hardy-Weinberg Equilibrium.

Consider a sample from a population in

genetic equilibrium with respect to a single gene with two alleles. If we assume the three different genotypes are identifiable, we are led to suppose that there are three types of individuals whose frequencies

P1 If Ni

=

82, P2

are

=

given by the so-called

28(1 - 8),

p, = ( I

8) 2 , 0 < 8 < L

(2.LI2)

i in the sample of size n, then (Nt , N2 , N3) parameters (n,p1 ,P2, p3) given by (2.1.12). Suppose

is the number of individuals of type

has a multinomial distribution with

-

Hardy-Weinberg proportions

104

Methods of Estimation

Chapter 2

$1. we can use we want to estimate fJ, the frequency of one of the alleles. Because fJ the principle we have introduced and estimate by JN1 jn. Note, however, that we can also 0 write 0 = I ..fii3 and, thus, 1 - .jNJTi is also a plausible estimate of 0. =

-

In general, suppose that we want to estimate a continuous Rl-valued function q of 8. If p1, . ,Pk are continuous functions of 8, we can usually express q(8) as a continuous function of PI , . . . , Pk, that is, .

.

q(IJ) = h(p,(IJ), . . . ,p.(IJ)), with h defined and continuous on P=

.

(p,, . . ,p.) : p; > O,

(2. 1.13)

k

L p; = 1

j

.

'

i= 1

Given h we can apply the extension principle to estimate q( 8) as,

T(X1, . . . , Xn)

=

h ( Nn, , . . , Nn• ) .

.

.

(2 1 . 14)

As we saw in the Hardy-Weinberg case, the representation (2.1.13) and estimate (2.1.14) are not unique. We shall consider in Chapters 3 (Example 3.4.4) and 5 how to choose among such estimates. We can think of the extension principle alternatively as follows. Let

Po = {Po = (p, (IJ), . . . ,p. (IJ)) : e E e} be a submodel of P. Now q(IJ) can be identified if IJ is identifiable by a parameter v : Po � R given by v(Pe ) = q(O). Then (2.1.13) defines an extension of v from Po to P via v : P --> R where v(P) - h(p) and v(P) = v(P) for P E Po . The plug-in and extension principles can be abstractly stated as follows:

T Plug-in principle. If we have an estimate P of P E P such that P E P and v : P is a parameter. then v(P) is the plug-in estimate of v. In particular, in the i.i.d. case if P is the space of all distributions of X and X�, . . . , Xn are i.i.d. as X f'V P. the empirical distribution P of X given by -

-

-

---+

-

1

n

P[X E A] = n L I (X; E A) -

(2.1 . 15)

. t=l

�

is, by the law of large numbers, a natural estimate of P and v(P) is a plug-in estimate of v(P) in this nonparametric context For instance, if X is real and F(x) = P(X :S x) is the distribution function (d.f.), consider va (P) = i [F- 1 (a) + F.J 1 (a)], where a E (0, I) and

F-1 (a) = inf{x : F(x) > a}, Fu 1 (a) = sup{x : F(x) < a},

-

-

�

-

(2. 1.16)

Section 2.1

105

Basic Heuristics of Estimation

v0 ( P) is the ath population quantile X Here x � = v.!. (P) is called the population median. A natural estimate is the ath sample quantile then

'

a.

(2.1.17) where

F is the empirical d.f. Here X!. �

is called the sample median.

'

For a second example, if X is real and P is the class of distributions with EjXj1 < oo, then the plug-in estimate of the jth moment v(P) = f..t1 = in this nonparametric �

context is the jth sample moment v(P) =

E(XJ )

�

J x;dF(x)

=

.

n- 1 I:� 1 Xf. �

Extension principle. Suppose Po is a submod.el of P and P is an element of P but not necessarily Po and suppose v : Po � T is a parameter. If v : P � T is an extension of v � in the sense that v(P) = v(P) on Po. then v(P) is an extension (and plug-in) estimate of v(P). With this general statement we can see precisely how method of moment estimates can be obtained as extension and frequency plug-in estimates for multinomial trials because

1';(8)

k

=

L vfp,(8) = h(p(ll)) i=l

where

h(p)

=

=

v(Po)

k

L v!P< = v(P) ,

i =l

�

and P is the empirical distribution. This reasoning extends to the general i.i.d. case (Prob­ lem 2.1.12) and to more general method of moment estimates (Problem 2.1.13). As stated, these principles are general. However, they are mainly applied in the i.i.d. case--but see Problem 2.1.14.

Remark 2.1.1. The plug-in and extension principles are used when Pe, v, and v are contin­ uous. For instance, in the multinomial examples 2.1.3 and 2.1.4, Pe as given by the Hardy­ Weinberg p(O), is a continuous map from e = [0, 1] to P, v(Pe ) = q(8) = h(p(8)) is a continuous map from 8 to R and v( P) = h(p) is a continuous map from P to R. Remark 2.1.2. The plug-in and extension principles must be calibrated with the target parameter. For instance, let Po be the class of distributions of X = B + �: where B E R

and the distribution of t: ranges over the class of symmetric distributions with mean zero. Let v(P) be the mean of let P be the class of distributions of = (I + < where B E R and the distribution off ranges over the class of distributions with mean zero. In this case both v1(P) = Ep(X) and v2(P) = "median of P" satisfy v(P) = v(P), P E Po,

X and

but only

�

v1(P)

-

=

X is a sensible� estimate of v(P), P

symmetric, the sample median

X

¢ P0, because when P is not

v2(P) does not converge in probability to Ep(X).

106

Methods of Estimation

Chapter 2

Here are three further simple examples illustrating reasonable and unreasonable MOM estimates.

Example 2.1.5. Suppose that X 1 , . . . , Xn is a N(f-1, a2 ) sample as in Example 1 . 1.2 with assumptions ( 1 )--(4) holding. The method of moments estimates of f1 and a2 are X and

-· a .

0

Example 2.1.6.

Suppose

X1 ,

.

•

•

, Xn

are the indicators of a set of Bernoulli trials with

probability of success 8. Because p,1 (8) = () the method of moments leads to the natural

estimate of 8,

X, the frequency of successes. To estimate the population variance 8( 1 - 8)

we are led by the first moment to the estimate,

X(l - X).

Because we are dealing with

(unrestricted) Bernoulli trials, these are the frequency plug-in (substitution) estimates (see Problem

o

2.Ll).

1.5.3 1

Example 2.1.7. Estimating the Size of a Population (continued). In Example where ) . Thus, 0 = 21' X1 , . . c, Xn are i.i.d. U {1, 2, . . . , 0}, we find I' = Eo (X,) = J (0 and 2X - 1 is a method of moments estimate of B. This is clearly a foolish estimate if X(n ) = max Xi > 2X - 1 because in this model B is always at least as large as X(n) . D

+1

As we have seen, there are often several method of moments estimates for the same

q(9).

For example, if we are sampling from a Poisson population with parameter B, then (}

is both the population mean and the population variance. The method of moments can lead to either the sample mean or the sample variance. Moreover, because Po =

exp( -0}, a frequency plug-in estimate of 0 is

- logj)0, where

will make a selection among such procedures in Chapter

Remark 2.1.3.

3.

P(X

Po is n- 1 [#X;

=

0) = 0]. We

=

What are the good points of the method of moments and frequency plug-in?

(a) They generally lead to procedures that are easy to compute and are, therefore, valu­

'

able as preliminary estimates in algorithms that search for more efficient estimates. See

'

Section

2.4.

(b) If the sample size is large, these estimates are likely to be close to the value estimated

(consistency). This minimal property is discussed in Section It does turn out that there are

5.2.

"best'' frequency plug-in estimates, those obtained by

the method of maximum likelihood, a special type of minimum contrast and estimating

2.4, they are often difficult to compute. Algorithms for their computation will be introduced in Section 2.4.

equation method. Unfortunately, as we shall see in Section

Discussion.

When we consider optimality principles, we may arrive at different types of

estimates than those discussed in this section. For instance, as we shall see in Chapter

3,

estimation of (} real with quadratic loss and Bayes priors lead to procedures that are data weighted averages of(} values rather

than minimizers of functions p( 6, X). Plug-in is not

the optimal way to go for the Bayes, minimax, or uniformly minimum variance unbiased (UMVU) principles we discuss briefly in Chapter apparent in Chapters

5 and 6.

3.

However, a saving grace becomes

If the model fits, for large amounts of data, optimality prin­

ciple solutions agree to first order with the best minimum contrast and estimating equation solutions, the plug-in principle is justified, and there are best extensions.

' '

'

'' ' '

Section 2.2

107

Minimum Contrast Estimates and Estimating Equations

Summary. We consider principles that suggest how we can use the outcome X of an experiment to estimate unknown parameters. For the model { Pe : f) E 8} a contrast p is a function from X x H to R such that the discrepancy D (!Jo , IJ)

�

E(;"p(X, IJ), B E 6 C Rd

is uniquely minimized at the true value 8 = 80 of the parameter. A minimum contrast esti­ mator is a minimizer of p(X, 6). and the contrast estimating equations are VeP(X, 6 ) =

0,

For data ( (z;, li) : I < i < n ) with li independent and E(Y;) = g((3, z;), 1 < i < n , where g is a known function and /3 E Rd is a vector of unknown regression coefficients. a least squares estimate of /3 is a minimizer of p(X, (3) = Lfli

-

g((3 , z, )f .

For this contrast. when g (/3 z) = zT{3. the associated estimating equations are called the normal equations and are given by ZbY = ZbZn/3. where Zn = llziJ·IInxd is called the design matrix. Suppose X P. The plug-in estimate (PIE) for a vector parameter v = v(P) is obtained by setting fJ = v( P) where P is an estimate of P. When P is the empirical probability distribution Pe defined by Pe(A) = n- 1 I:� 1 l [X; E Aj, then v is called the empirical PIE. If P = Po. 6 E e. is parametric and a vector q(8) is to be estimated. we find a parameter v such that v(P&) = q(8) and call v(P) a plug-in estimator of q(IJ). Method of moment estimates are empirical PIEs based on v(P) = (f.li. . . . , f.ld )T where In the multinomial case the frequency plug-in estimators f.lJ = E( XJ\ I < j < are empirical PIEs based on v(P) � (PI : . . ,pk ) , where PJ is the probability of the jth category, 1 < j :::; k. Let Po and P be two statistical models for X with Po c P. An extension v of v from Po to P is a parameter satisfying v(P) = v(P), P E P0. If P is an estimate of P with P E P, v(P) is called the extension plug·in estimate of v(P). The general principles are shown to be related to each other. ,

,.....,

�

�

�

�

�

�

d.

.

�

�

2.2 2.2.1

�

M I N I M UM CONTRAST ESTIMATES AND ESTIMATING EQUATIONS Least Squares and Weighted Least Squares

Least squares< 1 ) was advanced early in the nineteenth century by Gauss and Legendre for estimation in problems of astronomical measurement. It is of great importance in many areas of statistics such as the analysis of variance and regression theory. In this section we shall introduce the approach and give a few examples leaving detailed development to Chapter 6.

108

by

Methods of Estimation

In Example 2. 1.1

Chapter 2

we considered the nonlinear (and linear) Gaussian model P0 given

li = g(/3, z;) + <, 1 < i < n where ci are i.i.d.

(2.2.1)

N(O, aJ) and {3 ranges over Rd or an open subset. The contrast n

p(X, i3) = Dli - g(/3, z,Jf i=l -

led to the least squares estimates (LSEs) {3 of {3. Suppose that we enlarge Po to P where we retain the independence of the Yi but only require

IJ.(z; ) = E(Y; ) = g((l,z;),

E(<;) = 0.

(2.2.2)

Are least squares estimates still reasonable? For P in the semiparametric model P, which satisfies (but is not fully specified by) (2.2.2), we can compute still

Dp (/30 , /3)

=

Epp(X, /3)

n

- L Var i=l

p

n

(<; ) + L [g(/30 , z;) i=l

-

(2.2.3)

g(/3, z;)] 2 ,

which is again ntinimized as a function of i3 by i3 = {30 and uniquely so if the map , g(f3,znlf is 1 - 1. f3 � The estimates continue to be reasonable under the Gauss-Markov assumptions,

(g(i3,zt),

.

.

.

E( <;) = O, l < i < n, Var(ci) = a2 > 0, 1 :::; i :::; n,

(2.2.4)

Cov(�:i, Ej) = 0, 1 < i < j ::.; n

(2.2.6)

(2.2.5)

because (2.2.3) continues to be valid. Note that the joint distribution H of (C t , . . , t:n) is any distribution satisfying the Gauss-Markov assumptions. That is, the model is semiparametric with {3, a2 and H un­ known. The least squares method of estimation applies only to the parameters and is often applied in situations in which specification of the model beyond (2.2.4}-{2.2.6) is difficult Sometimes z can be viewed as the realization of a population variable Z, that is, (Zi, Yi). 1 $ i < n, is modeled as a sample from a joint distribution. This is frequently the case for studies in the social and biological sciences. For instance, Z could be edu­ cational level and Y income, or Z could be height and Y log weight Then we can write the conditional model of Y given Zi = Zj , I < j < n, as in (a) of Example 1.1.4 with If we consider this model, (Z,, li) i.i.d. as <j simply defined as Yj E(Yj I Zi = (Z, Y) � P E P = {All joint distributions of(Z,Y) such that E(Y I Z = = g(i3,z), and i3 � (g(/3, z,), . . . ,g(/3, zn)JT is 1 - 1, then f3 has an interpretation as a i3 E parameter on P, that is, f3 = (l(P) is the ntinim12er of E (Y - g(/3, Z))'. This follows .

{31,

-

Rd},

"

I

Zj)·

z)

•

.

•

, !3d

109

Minimum Contrast Estimates and Estimating Equations

Section 2.2

-

from Theorem 1.4.1. In-this case we recognize the LSE {3 as simply being the usual plug-in estimate f3(P), where P is the empirical distribution assigning mass n - 1 to each of the n pairs (Zi, Yi). the most commonly used g in these models is g({3, z) = As we noted in Example is called the linear and zT(3, which, in conjunction with (multiple) regression modeL For the data { (zi, Yi); i = 1, , n } we write this model in matrix form as Y = Zv(3 + <

2.1.1(2.2.1), (2.2.4), (2.2.5) (2.2.6), ...

where Zv = llzij II is the design matrix. We continue our discussion for this important special case for which explicit formulae and theory have been derived. For nonlinear cases we can use numerical methods to solve the estimating equations (2.1.7). See also Problem Sections and and Seber and Wild (1989). The linear model is often the default model for a number of reasons: (I) If the range of the z's is relatively small and p(z) is smooth, we can approximate p(z) by

2.2.41,

6.4.3 6.5,

p(z) = -:::·

d

a p(zol + :L a: (zo)(z - zo), j=l J

for zo an interior point of the domain. We can then treat p(zo) g� (zo)zo as an unknown to give an approximate ( d + I) -dimensional and identify %!; (zo) with linear model with z0 = 1 and as before and

f3o

- E1= l

f3;

Zj

d

p(z) = L f3; z; .

j=O

This type of approximation is the basis for nonlinear regression analysis based on local polynomials, see Ruppert and Wand (1994), Fan and Gijbels (1996), and Volume ll. (2) If as we discussed earlier, we are in a situation in which it is plausible to assume that (Zi, Yi) are a sample from a (d+ 1 )-dimensional distribution and the covariates that are the coordinates of Z are continuous, a further modeling step is often taken and it is assumed . . ) zd , Y)T bas a nondegenerate multivariate Gaussian distribution Nd+I (11, I: that ). In that case, as we have seen in Section 1. , E(Y I Z = z) can be written as •

(Zl )

.

4

d

p(z) = f3o + L f3; z;

j=l

where d

f3o

=

(-((3,, . , {3d), l)�t = i'Y - L f3i !'j ·

..

j= l

(2.2.9)

no

Methods of Estimation

Chapter 2

Furthennore,

< = y - I'(Z) is independent of Z and has a N(O, a 2 ) distribution where

Therefore, given Zi = 1 < i < n.

Zi,

a2 = ayy - Eyz:E z� Ezy.

1
n, we have a Gaussian linear regression model for }i,

Estimation of {3 in the linear regression model. We have already argued in Example � 2. 1. 1 Zn/3 is identifiable, the least squares estimate, {3, exists that, if the parametrization {3 and is unique and satisfies the normal equations (2.1.8). The parametrization is identifiable if and only ifZv is of rank d or equivalently if ZbZv is of full rank d; see Problem 2.2.25. In that case, necessarily, the solution of the normal equations can be given "explicitly" by ------)

(2.2.10)

'

Here are some examples.

Example 2.2.1. In the measurement model in which Yi is the detennination of a constant g(z, il1) = 1 and the normal equation is L:� 1 (y, - il1) = 0, /h, d = 1, g(z, il1J = il" � D whose solution is il1 = ( 1/n) L:� 1 Yi = y.

8�,

Example 2.2.2. We want to find out how increasing the amount z of a certain chemical or fertilizer in the soil increases the amount y of that chemical in the plants grown in that soil. For certain chemicals and plants. the relationship between z and y can be approximated well by a linear equation y = /31 + (32 z provided z is restricted to a reasonably small interval. If we run several experiments with the same z using plants and soils that are as nearly identical as possible, we will find that the values of y will not be the same. For this reason, we assume that for a given z, Y is random with a distribution P(y j z ). Following are the results of an experiment to which a regression model can be applied (Sned.ecor and Cochran, 1967, p. 139). Nine samples of soil were treated with different amounts z of phosphorus. Y is the amount of phosphorus found in com plants grown for 38 days in the different samples of soil. Z;

r;

1

64

4 71

5 54

9 81

11 76

13 93

23 77

23 95

�

28 109

,

' '

The points (zi, Yi) and an estimate of the line !31 + f32z are plotted in Figure 2.2.1. We want to estimate f3I and /32 . The normal equations are n

n

� (y, - il1 - il2 z;) = 0, � z,(y; - il1 - f3zzi) = 0. i=l

(2.2. 1 1 )

When the Zi 's are not all equal, we get the solutions (2.2.12)

'

I

Section 2.2

Minimum Contrast Estimates and Estimating Equations

111

y •

110 90

•

•

.

E6:

•

•

70 50

�

•

•

•

10

0

20

X

30

Figure 22.1. &atter plot { (Zi, y1); i = 1, . . . , n} and sample regression line for the phosphorus data. fs is the residual for (zs, Ys) ·

and fJ1

=

Y - /3zz

(2.2.13)

where z = ( 1 /n) I:� 1 z;, and ii = (1/n) 2::7 1 y;. The line y = {31 + f32z is known as the sample regression line or line of best fit of YI, . . . , Yn on z 1 , . . . , Zn- Geometrically, if we measure the distance between a point (z; , Y;) and a line y = a + bz vertically by d; = IY; - (a + bz;)l. then the regression line minimizes the sum of the squared distances to the n points ( z1, Y1 ), . . . , (zn, Yn). The vertical distances €i = fy,: - (!31 + f32 zi)] are called the residuals of the fit, = 1, . . . , n. The � line y = {31 + (32 z is an estimate of the best linear MSPE predictor a1 + b1 z of Theorem 1 .4.3. This connection to prediction explains the use of vertical distances� in regression. The regression line for the phosphorus data is given in Figure 2.2.1. Here fJ1 = 61.58 and � 0 /3z = 1.42. �

�

�

�

i

�

Remark 2.2.1. The linear regression model is considerably more general than appears at first sight. For instance, suppose we select p real-valued functions of z, 91, . . . , gp. p > d

and postulate that ,u ( z) is a linear combination of 91 (z), . . . , Yv(z) ; that is p

,u(z) = 2:);9; (z). j=l

Then we are still dealing with a linear regression model because we can define Wpxi

112

I

Methods of Estimation

(g1 (z ) ,

Chapter 2

. . . . gp( z)) T as our covariate and consider the linear model Yi

p

L OJWij + ci, 1 < i <

=

j=l

n

where

Wij For instance, if

Yi

=

= 9j( Zi) ·

d = 1 and we take 9i (z) = zi,

0

< j < 2, we arrive at quadratic regression,

Bo + B 1 Zi + (hz'f + £i-see Problem 2.2.24 for more on polynomial regression.

Whether any linear model is appropriate in particular situations is a delicate matter par­

tially explorable through further analysis of the data and knowledge of the subject matter. We return to this in Volume

II.

Weighted least squares. In

Example 2.2.2 and many similar situations it may not be

zi of the However, we may be able to characterize the dependence of Var( fi ) on

reasonable to assume that the variances of the errors covariate variable.

Zi at least up to a multiplicative constant.

fi

are the same for all levels

That is, we can write

(2.2.14) where

are

a2 is unknown as before, but the wi

heteroscedastic

known weights. Such models are called

(as opposed to the equal variance models that are

homoscedastic).

The

method of least squares may not be appropriate because (2.2.5) fails. Note that the variables

y, ,

= _

_

g ((3 , Z; )

Y; - ..;w; ..;w;

+ <; ' ..;w;

are sufficient for the Y; and that Var(<;/ #;) =

g((3, z;) j ..;w;. €, =
and the

=

I < z. < n, - -

ww2 jw;

9(,8, zi ) + ti , 1

=

u2 . Thus, if we setg((3, z;) = (2.2.15)


- satisfy the assumption (2.2.5). The weighted least squares estimate of /3 is now Yi

the value

�

{3. which for given iii

=

yi f ,fWi minimizes

(2.2.16) as

a function of ,8.

Example 2.2.3. Weighted Linear Regression. zi2 = Zi, and g(/3, Zi) = {31 + fhzi. i = 1, . .

{31 and f3z that minimize

I '

n

L v, [y, i=I

-

Consider the case in which .

d

= 2, �

Zii = 1 , �

, n. We need to find the values {31 and /32 of

(/3, + f3zz,W

(2.2.17)

Section 2.2

113

Minimum Contrast Estimates and Estimating Equations

where vi = 1/wi- This problem may be solved by setting up analogues to the normal equations (2.1.8). We can also use the results on prediction in Section 1.4 as follows. Let (Z*, Y*) denote a pair of discrete random variables with possible values (z1, y1) , . . . , (zn , Yn) and probability distribution given by

P[ (Z' , Y') � (z; , y;)J � u;, where

i

�

1, . . , n .

n Ui = Vi/ L vi , i = 1 , . . ,n. i=l + {32Z* denotes a linear predictor of Y* based on Z*, then its MSPE is .

If /1-t (Z*)

given by

=

{31

n

E[Y ' � !lt (Z')f � L u;[y, � ((3, + (32z;)]2 i=l It follows that the problem of minimizing (2.2.17) is equivalent to finding the best linear MSPE predictor of Y"'. Thus, using Theorem 1.4.3, a.

V.l -

and

Cov(Z"', Y"') - L:t UiZi Yi - (L�� t UzYi)(L��J uizi) n ; 2 ( n u;z;)2 Var(Z•) L;�J u z; L;�1

(2.2.18)

�

n � 1" � u;y; - (h - � u;z;. (31 � E(Y') - f32E(Z') � -n1 " n 1 iz.-o-: I �

�

n -"

This computation suggests, as we make precise in Problem 2.2.26, that weighted least D squares estimates are also plug-in estimates.

�

Next consider finding the ,B that minimizes (2.2.16) for g(,B, z; ) = zf,B and for general d. By following the steps of Example 2.2.1 leading to (2.1.7), we find (Problem 2.2.27) � that f3 satisfy the weighted least squares normal equations where W = diag(wt, . . . , Wn) and Zn = llziillnxd is the design matrix. When Zn has n, we can write rank d and Wi > 0, 1


Remark 2.2.2. More generally, we may allow for correlation between the errors

{t:i}· That is, suppose Var(€) = a2W for some invertible matrix Wn x n · Then it can be shown (Problem 2.2.28) that the model Y � Zn,B+• can be transformed to one satisfying (2.2.1) � and (2.2.4H2.2.6). Moreover, when g(,B, z) = zT,B, the ,B minimizing the least squares contrast in this transformed model is given by (2.2.19) and (2.2.20).

Methods of Estimation

1 14

2

Remark 2.2.3. Here are some applications of weighted least squares: When the ith re­ sponse Yi is an average of ni equally variable observations, then Var(Yi) = a2 /ni, and 'Wi = ni 1 . If Yi is the sum of ni equally variable observations, then 'Wi ni. If the vari­ ance of Yi is proportional to some covariate, say z1 . then Var(}i) = Zi 1 a2 and 'Wi Zi l · =

=

I n time series and repeated measures, a covariance structure i s often specified for Problems 2.2.29 and 2.2.42). 2.2.2

e

(see

Maximum Likelihood

The method of maximum likelihood was first proposed by the German mathematician C. F. Gauss in 1 82 1 . However, the approach is usually credited to the English statistician R. A. Fisher ( 1 922) who rediscovered the idea and first investigated the properties of the method. In the form we shall give, this approach makes sense only in regular parametric models. Suppose that p(x, 0) is the frequency or density function of X if 0 is true and that 8 is a subset of d-dimensional space. Recall Lx (0) , the likelihood function of 0, defined in Section 1 .5, which is just p(x, 0) considered as a function of 0 for fixed x. Thus, if X is discrete, then for each 0, Lx ( 0) gives the probability of observing x. If 8 is finite and 1r is the uniform prior distribution on 8, then the posterior probability that 0 0 given X = x satisfies 1r(O I x) ex Lx (O), where the proportionality is up to a function of x. Thus, we can think of Lx ( 0) as a measure of how "likely" 0 is to have produced the observed x. A similar interpretation applies to the continuous case (see A.7. 1 0) . The method of maximum likelihood consists of finding that value B(x) of the parameter that is "most likely" to have produced the data. That is, if X = x, we seek 0(x) that satisfies

Lx( B( x) )

=

p (x, B(x) ) = max{p(x, 0) : 0

E

8}

max{ Lx (O)

:

0 E 8}.

By our previous remarks, if 8 is finite and 1r is uniform, or, more generally, the prior density 1r on 8 is constant, such a 8(x) is a mode of the posterior distribution. If such a 8 exists, we estimate any function q( 0) by q(O(x) ). The estimate q(O(x) ) is called the maximum likelihood estimate (MLE) of q (O) . This definition of q(O) is consistent. That is, suppose q is 1 - 1 from 8 to 0; set w = q( 0) and write the density of X as p0 (x, w) = p(x, q-1 (w)). If w maximizes p0 (x, w) then w = q(O) (Problem 2.2.16(a)). If q is not 1 - 1 , the MLE of w q(O) i s still q({i) (Problem 2.2.16(b)). Here is a simple numerical example. Suppose 0 0 or � and p (x , 0) is given by the following table. 1 x\0 0 2

0

0. 1 0 0.90

2

Then 8( 1 ) = �' 8( 2) = 0. If X 1 the only reasonable estimate of 0 is value 1 could not have been produced when 0 0. =

=

� because the

1 15

Maximum likelihood estimates need neither exist nor be unique (Problems 2.2. 1 4 and 2 .2. 13). In the rest of this section we identify them as of minimum contrast and estimating equation type, relate them to the plug-in and extension principles and some notions in information theory, and compute them in some important special cases in which they exist, are unique, and expressible in closed form. In the rest of this chapter we study more detailed conditions for existence and uniqueness and algorithms for calculation of MLEs when closed forms are not available. When fJ is real, NILEs can often be obtained by inspection as we see in a pair of impor­ tant examples.

Example 2.2.4. The Normal Distribution with Known Variance. Suppose X "" N(fJ, a2 ) , where a2 is known, and let