REGRESSION AND T H E MOORE-P ENROSE PSEU DO1N V ERSE Arthur Albert DEPARTMENT OF MATHEMATICS BOSTON UNIVERSITY BOSTON, MASSACHUSETTS
@
1972
ACADEMIC PRESS New York and London
COPYRIGHT 0 1972, BY ACADEMIC PRESS, INC. ALL RIGHTS RESERVED, NO PART O F THIS PUBLICATION MAY BE REPRODUCED OR TRANSMITTED IN ANY FORM OR BY ANY MEANS, ELECTRONIC OR MECHANICAL, INCLUDING PHOTOCOPY, RECORDING, OR ANY INFORMATION STORAGE AND RETRIEVAL SYSTEM, W I T H O U T PERMISSION I N WRITING FROM T H E PUBLISHER.
ACADEMIC PRESS, INC. 111 Fifth Avenue, New
York, New York 10003
United Kingdom Edirion published b y ACADEMIC PRESS, INC. (LONDON) LTD. 24/28 Oval R o a d . London NW1
LIBRARY OF CONGRESS CATALOG CARDNUMBER:72-77337 AMS (MOS) 1970 Subject Classifications: 62505, 15A09 PRINTED I N THE UNITED STATES
OF AMERICA
To m y Father and Mother on their anniversary
PREFACE
For the past ten years, my professional interests have focused on various aspects of regression. It has been my experience that the pseudoinverse is a great unifying concept. It has helped me to understand, remember, and explain many classical results in statistical theory as well as to discover (and rediscover) some new ones. This book was conceived as a hybrid monograph-textbook. As a text, it would be suitable for the equivalent of a two-quarter course. In teaching such a course, one could fill in the remainder of the year with additional material on (for example) multiple regression, nonlinear regression, large sample theory, and optimal experimental design for regression. For this purpose I have attempted to make the development didactic. On the other hand, most of the material comes from reasonably current periodical literature and a fair amount of the material is my own work (some already published, some not). Virtually all of the material deals with regression either directly (Chapters VI-IX) or as background (Chapters I-V). By restricting the domain of discourse we are able to pursue a leisurely pace and, I hope, to preserve a sense of unity throughout. At the time the manuscript was completed there were, to my knowledge, no textbook treatments of the pseudoinverse. Since that time, two excellent complementary monographs have appeared containing treatments of the Moore-Penrose pseudoinverse in a more general setting. The first (Boullion and Ode11 [2]) appeared in early 1971 and concerns itself mainly with algebraic and structural properties of these pseudoinverses. The second (Rao and Mitra [l]) appeared later in 1971 and is extremely comprehensive in its coverage of the then extant pseudoinverse literature. Both volumes contain large bibliographies. xi
ACKNOWLEDGMENTS
I wish to express my thanks to the Office of Naval Research, Army Research Office, and Air Force Office of Scientific Research for their support during various stages of this book. In particular, the final draft was written while I was a visitor at Stanford during the summer of 1970, at which time I was supported under contract NOOO14-67-A-0112-0053 ; NR-042-267. The index for this book was sorted and alphabetized on our time-share computer terminal. I would like to thank Don Feinberg, who wrote the program, and Nancy Burkey, who fed the necessary information to the computer.
xiii
Part I
THE GENERAL THEORY AND COMPUTATIONAL METHODS
Chapter I ~
INTRODUCTION
In 1920, Moore [l] introduced the notion of a generalized inverse for matrices. The idea apparently lay dormant for about 30 years, whereupon a renaissance of interest occurred. As may be appreciated after a brief look through the bibliography, a large and varied literature has appeared since the early 1950s. At the present time, the theory is elegant, the applications are diverse (e.g., least squares, linear equations, projections, statistical regression analysis, filtering, and linear programming) and most important, a deeper understanding of these topics is achieved when they are studied in the generalized inverse context. The results that we present here are, for the most part, scattered throughout the periodical literature or to be found in out-of-print technical reports. This is the major impetus behind the writing of this monograph. The level of the material presupposes a familiarity with the notion of “limit” and some of the fundamental properties of finite-dimensional Euclidean spaces. That much will suffice for the first half of the book. The second half is devoted to statistical applications and for these it would be helpful to have had a first course in probability (and/or statistics). Many exercises are included. (Solution outlines are provided in a separate booklet.) Some of the exercises are supplementary to the material in the monograph, whereas others are lemmas to subsequent theorems. The reader 3
4
I
INTRODUCTION
is urged to do the exercises. They are the key to a thorough understanding of the material. Sections and equations are numbered according to a decimal system. For example, Eq. (6.3.1) comes after Section (6.2) and before Definition (6.3.1.9). The latter comes before Eq. (6.4). Every effort has been made to maintain the following typographical conventions : Sets-uppercase script Matrices-uppercase Latin Vectors-lowercase Latin Scalars-lowercase Greek Vector random variables-lowercase boldface Latin Real random variables-lowercase boldface Greek. Bibliographic references are enclosed in square brackets; equation and section numbers are written in parentheses at the left. Sometimes a section or an equation number appears in the middle of a line or at the right side. In this case, the assertion preceding this equation number is a direct consequence of that (previously established) result.
Chapter I/
GENERAL BACKGROUND MATERIAL
In this chapter, we will review the key results and definitions from the theory of real linear spaces which are relevant to what follows. This survey is informal and presumes previous exposure of the reader to these notions. (For a more formal review, see Appendix A of Karlin [I]. For a rigorous development, see Halmos [l] or the first half of Bellman [l].) We begin by reminding the reader that linear transformations from one Euciidean space to another can be represented by matrices, once the coordinate systems for the two spaces have been decided upon. Furthermore, vectors can be represented as long, skinny matrices (having one column). If A is any matrix, we denote the transpose of A by AT. In our discussions, all matrices, vectors (and scalars) have real entries. Matrix transposition has the following important properties : (AB)' = P A T ,
(A')'
=
A,
(A+B)' = A'+B'.
If x and y are two vectors having the same number of components, the inner product (or scalar product) of x with y is the scalar x'y (which is the same as y'x). The norm of x is llxli = ( X ' X ) ~ . Two vectors are said to be orthogonal if their inner product vanishes. (The statement, x is orthogonal to y is abbreviated x l y . ) A linear manifold 2 ' is a nonempty subset of a Euclidean space, which is 5
6
II
GENERAL BACKGROUND MATERIAL
closed under addition and scalar multiplication (if x and y are elements of 2’ then for any scalars ct and p, ats+py are members of 2). A vector .Y is orthogonal to the linear manifold 9 if .Y is orthogonal to every vector in 2’ (abbreviated .Y 19). The symbol E will be reserved for set membership (x E 2’ means that x is a member of the set 9). The symbol E denotes set inclusion and c denotes proper inclusion. The following theorem is of fundamental importance in all that follows. We state it without proof:
(2.1) Theorem: Let .Y be a vector in a finite-dimensional Euclidean space and let Y be a linear manifold in that space. Then there is a unique vector 2 E 9 having the property that x-2 I Y . (2.1.1) Comment: An equivalent statement of the theorem is that there is a unique decomposition of x: x=.t+Z
where
~ E Yand
212’.
The vector 2 is called the projection o j x on 2’.It is the vector in 2’ that is “nearest” to x, as we shall now demonstrate. (2.2) Theorem: Let .Y be a vector and 9 a linear manifold. If x = 2 + 2 where 2 E 2’and 2 19, then
Ilx-vll if
’llx-all
~ € 2 ’ and
yf2.
Proof: If y E Y then /Ix-yI1Z = //2+2-y11* = II(.t-y)+2112 =
ll~-vIlZ+ lPIl2
since 2 I Y and 2 - y E 3’.Therefore IIX-YIIZ 2
ll~llz
with strict inequality holding unless !l2--yIl2= 0. Theorems (2. I ) and (2.2) are “existence theorems.” As a consequence of the next theorem, we can show how to reduce the computation of 2 to the solution
7
LINEAR MANIFOLDS IN FINITE-DIMENSIONAL EUCLIDEAN SPACES
of simultaneous linear equations. First though, we remind the reader that a linear manifold Y is spanned by ( y I , y 2...,y,) , if every vector in Y can be expressed as a linear combination of the yj's.
(2.3) Theorem: (a) If x is a vector and Y is a linear manifold, then 9, the projection of x on Y ,is the unique vector in Y satisfying the equations (2.3.1)
for all
2'y = x'y
9.
(b) If Y is spanned by y,,y,, ...,y,, 2 is the unique vector in Y satisfying (2.3.2)
j = 1,2,...,n.
aTyj = x'yj
Proof: Part (a) follows directly from (2.1). That i satisfies (2.3.2) is a consequence of (a). If x* is some other vector in Y satisfying x*'yj
j = 1,2,..., n,
= xTyj
then (2.3.3)
(x*-3)'yj
=
j = 1,2,..., n.
0
Since the yj's span Y , any vector in Y is a linear combination of the yj's, so that x* - 2 is orthogonal to every vector in 9. Since x* - 2 E Y ,it follows that (x* - i)T(x*- 2 ) = //x*- 911 * = 0. Therefore, x* must coincide with 2 if it lies in Y and satisfies (2.3.2). 4 (2.4) Exercise: If Y is spanned by y l , y 2 ,...,y,, then 2 is the unique vector of the form
z
n
=
1ujyj
j= 1
where the 3;s are any scalars satisfying the simultaneous set of linear equations (2."4.1)
C u j ~ ~ y=jyTx ) n
i=
1
i = 1,2,..., n.
(2.5) Exercises (2.5.1) If Y is spanned by y l , y 2 ,...,y, then x L Y if x is orthogonal to each of the y i s . (2.5.2) If x and y are vectors in the same Euclidean space, and Y is a linear manifold in that space, then the projection of ux + py on Y is a2 where 9 and j are the projections of x and y on 9. If y I , y 2 ...,y, , is a set of vectors in the same Euclidean space, we denote
+
8
I1
GENERAL BACKGROUND MATERIAL
the linear manifold spanned by the yj’s by Y ( y , , y 2 ,...,y,). This manifold is the smallest manifold containing all the yj’s and it consists, exactly, of all those vectors which are expressible as a linear combination of the yj’s.
(2.5.3) The projection of x on 2 ( y ) is ( ~ ~ y ) y / I \ ify \y \ # ~ 0. If x , y 1, y 2,...,y, is an arbitrary set of vectors in the same Euclidean space, a particularly simple relationship exists between L,, the projection of x on Y ( y l ,..., y,), and L,-l, the projection of x on 9 ( y 1 ,...,y n d 1 ) , provided that y , is orthogonal to the previous yj’s: (2.6) Theorem: If x , y l ,..., y , are vectors in the same Euclidean space then and if y , IY ( y , , ...,y,-
if y, = 0 otherwise.
Proof: Since 9,- E 2 ( y l , ...,y,- 1 ) , 9,is clearly a member of 9 ( y l , . ..,y,). It is readily verified that the right side of (2.6.1) satisfies (2.3.2), provided y , is orthogonal to Y ( y l , . . . , y n - 1) (and in particular, orthogonal to L,-l). The conclusion follows from (2.3b). As an immediate consequence we can derive the so-called Fourier expansion theorem :
(2.7) Theorem: If u l , u 2 , ..., u, are mutually orthogonal vectors of unit length and .Y is an arbitrary vector in the same Euclidean space, then 2, the projection of x on Y ( u , , ..., u,) is given by (2.7.1)
L
=
(2.7.2)
2
=
c uju; ) c uj.
x,
(j:, n
(UjTX)
j= 1
xy=,
uj ujT Comment: If the uj’s are k-dimensional vectors, the expression is a k x k matrix, so that (2.7.1) is a representation of the matrix which projects s onto Y ( u l ,..., u,). Equation (2.7.2), on the other hand, is an explicit representation of L as a linear combination of the u i s .
If Yl and Y , are linear manifolds and 2,c_ LZ2, define Y 2 - Y l as the set of vectors in 2,which are orthogonal to 9,. , Y2- g1is a linear manifold. (2.7.3) Exercise: If 8, G Y 2 then
(2.7.4) Exercise: Let x be a vector and suppose Yl E Y 2 .Define L2 to be the projection of x on Y 2 ,?21 , to be the projection of A2 on Yl, 2 , to be the projection of x on Yl and Z2 to be the projection of x on Y2- Yl .
LINEAR MANIFOLDS IN FINITE-DIMENSIONAL EUCLIDEAN SPACES
9
Then (The projection of x on 2?l is obtainable by projecting x on (a) 9,,= i1. 9, and then projecting that vector on Yl). (b) A 2 = 21 + 2 2 1 . (c) /lx--.Q1 11 2 ~ ~ x - - . Qwith , ~ ~strict inequality holding if P I is a proper unless x E Pl . subset of 9, (2.8) The Gram-Schmidt Orthogonalization Procedure
This procedure takes an arbitrary collection of vectors h,, h,, . .., h, and generates a set of mutually orthogonal vectors u,, u,, ..., u,, having the properties that
(2.8.1)
P ( u 1 , u 2 , ..., uj) = 9 ( h l ,h,, ...,hi)
for j = 1,..., n
and
(2.8.2)
llujll = 1
if
uj # 0 j = 1,2,..., n.
The Procedure
(2.8.3)
For j = 1,2, ...,n - 1, define (2.8.4) and
The properties (2.8.1) and (2.8.2) are established by induction :The induction hypothesis is that P ( u l , u 2,..., uj) = 9 ( h l,..., hj)
and
uj+l IY ( u , , ..., U j ) . By definition of u l , 9 ( u 1 )= 9 ( h , ) and by (2.5.3), h, is the projection of h, on P ( u , ) . By (2.1), h , - h , I9 ( u 1 ) so that u2 is orthogonal to 9 ( u 1 ) . This establishes the induction hypothesis for j = 1 . If it is assumed that the hypothesis is true for all values o f j up through k , then since u k + l is a linear
10
I1
GENERAL BACKGROUND MATERIAL
combination of h,,, and h k + , [which lies in 9 ( u l , ..., uk) = 9 ( h l , ..., hk)], we see that any vector which is a linear combination of U ~ , . , . , U is ~ +also ~ a This means that 9 ( u l ,..., u k + J c linear combination of h , , ...,h,+ 2’(h, ...,A,+ 1). On the other hand,
,.
f
hk+l
= Ilhk+l-hk+l~~uk+l
+i;k+l
the right side being a linear combination of the nonzero members of { u , , u 2 ,..., uk+,}. Since 9 ( h ,,..., h,) = 2 ( u l ,..., u,) under the induction hypothesis, any vector which is expressible as a linear combination of h , , ... h k + , is also expressible as a linear Combination of u1,..., k + l . Therefore, 2’(h,, ...,A,+ ,) E 9 ( u l ,..., u k +,). This establishes the first half of the induction hypothesis for j = k + 1. The second half follows since h,+, is the projection of h k + , on 2 ( u ,,..., u ~ + ~(2.7.2). ), By (2.1), h k + 2 - h k + 21 2 ’ ( u 1 ,..., u , + , ) and therefore, so is uk+,. (2.8.6) Exercise: uj = 0 if and only if hj is a linear combination of (hl,...,hj-1)*
In what follows, two special linear manifolds will be of particular interest: If H is any matrix, the null space of H , denoted by N ( H ) , is the set of vectors which H maps into zero:
N ( H ) = {x: H X = O}. [The null space of H , M ( H ) ,always has at least one element, namely the null vector 0.1 The range of H , denoted by W ( H ) , is the set of vectors which are afterimages of vectors in the Euclidean space which serves as the domain of H :
W ( H ) = { z :z
= Hx
for some x.}
It is easy to see that N ( H ) and W ( H ) are linear manifolds. (2.9) Exercises (2.9.1) Let the column vectors of H be denoted by h,,hz, ...,h,. Show that 9 ( H ) = 9 ( h , , h Z ,...,A,).
(2.9.2) Show that H T is the adjoint of H . That is to say, if H is an n x m matrix, then for any rn-dimensional vector, x and any n-dimensional vector y , the inner product of x with H y is the same as the inner product o f y with H’x. If 2’ is a linear manifold in a Euclidean space 8,the orthogonafcomplement
RANGE SPACES AND NULL SPACES FOR MATRICES
11
of Y (denoted by 3')is defined to be the set of vectors in d which are (each) orthogonal to 9. It is easy to see that 9 ' is itself a linear manifold.
(2.9.3) ( 9 ' ) '= 2'. (2.9.4) If x is a vector in 6 and x'y
=0
for all y E 6, then x
= 0.
The null space of a matrix is related to the range space of its transpose. In fact, the next theorem shows that the null space of H consists of those vectors which are orthogonal to the column vectors of HT(i.e., the rows of H ) which is just another way of saying (2.10) Theorem:
For any matrix H , N ( H ) = 9 ' ( H T ) ,
) and Proof: x e N ( H ) if and only if H x = O . Therefore, ~ E . N ( H if only if y T H x = 0 for all y (having the correct number of components, of course). [Use (2.9.4).] Since y'Hx = ( H T y ) T ~we, see that Hx = 0 if and only if x is orthogonal to all vectors of the form H T y .These vectors, collectively, make up %?(H'), thus proving the assertion. By applying the projection theorem (2.1), we deduce as an immediate consequence of (2.10), that every vector z (having the correct number of components) has a unique decomposition as the sum of two terms, one lying in 9 ( H ) and one lying in N ( H T ) :
(2.11) Theorem: If H is an n x m matrix and z is an n-dimensional vector, we can uniquely decompose z :
z=i+z where 2 E 9 ( H ) and 2 E N ( H T ) . (2.11.1) Exercise: In (2.11), 2 is the projection of z on g ( H ) and z" is the projection of z on N ( H T ) . Consequently HTz = H T i .
A matrix is said to be symmetric if it is equal to its transpose. Obviously, symmetric matrices are square. (2.11.2) Exercise: Matrices of the form H T H and HH' symmetric. By virtue of (2.10),
(2.11.3)
N ( A ) = %?'(A)
if A is symmetric.
and
%(A) = "(A)
are always
12
I1
GENERAL BACKGROUND MATERIAL
Moreover, if H is any matrix, then
(2.12) Theorem: 9 ( H ) = & ? ( H H T ) , 9?(HT)= 9 ( H T H ) , N ( H T H ) , and N ( H ' ) = N ( H H ' ) .
N ( H )=
Proof: It suffices to prove that &"(ITT) = N ( H H T )and N ( H ) = N ( H ' H ) . Then apply (2.10) and (2.9). To prove that N ( H T )= N ( H H T ) , we note that HHTx = 0 if H T x = 0. On the other hand, if H H T x = 0, then x ' H H ~ x = 0 so that jlHTx1/2= 0 which implies that H T x = 0. Thus H'x = 0 if and only if H H T x = 0. The same proof app!ies to show that N ( H ) = N ( H ' H ) .
A square matrix is nonsingular if its null space consists only of the zero vector. If a square matrix is not nonsingular, it is called singular. (2.12.1) Exercise: If the row vectors of H are linearly independent, then the null space of H' consists of the zero vector. (2.12.2) Exercise: Let h l , h z ,..., h, be a linearly independent set of vectors. Let G be the n x n matrix whose (i-j)th entry is h'hj (G is known as a Grammian). Show that G is nonsingular. [Hint: G = HH' where H is the matrix whose rows are hlT,h2', ..., AnT. Now apply (2.12.1) and (2.12).] If A is a nonsingular matrix, there is a unique matrix, A - ' , which is the left and right inverse of A : A@-')
=
@-')A
=
I
where Z is the identity matrix. (2.13) Theorem: If H is any matrix and 6 is nonzero, then H T H + d 2 f is nonsingular. 7
~ ~ + ~ ~ Proof: I f ( H T H + d 2 1 ) x= 0, then0 = x T ( H T H + d 2 Z ) x* I I H X ~ llx112 which can only occur if x = 0.
w
We close this chapter with a statement of the celebrated diagonalization theorem for symmetric matrices. The proof may be found in Bellman [l]. A (possibly complex) number 1, is called an eigenualue of the (square) matrix A if A - 1.1is singular. (2.13.1) Exercise: If A is real and symmetric, its eigenvalues are real. [Hint: If ( A - 1.1)x = 0, then ( A -21) X = 0, where X and X are the complex conjugates of 1 and x . ]
(2.14) Theorem: If A is real and symmetric with eigenvalues A,,A2, ...,A,,
THE DIAGONALIZATION THEOREM FOR SYMMETRIC MATRICES
13
then there is a matrix T such that TT = T-' and TTAT= diag(A,, A2, ..., A,,). [The term diag(A,, ..., A") refers to a diagonal matrix with entries Al, If TT= T-', T is said to be an orthogonal matrix.]
..., A,,.
(2.14.1) Exercise: If Tis an orthogonal matrix, the rows of Tare mutually orthogonal and have unit length. So too, are the columns.
Chapter 111 ~~
~~
G E O M E T R I C A N D ANALYTIC PROPERTIES OF T H E M O O R E - P E N R O S E P S E U D O I N V E R S E
We begin our treatment of the pseudoinverse by characterizing the minimum norm solution to the classical least squares problem : (3.1) Theorem: Let z be an n-dimensional vector and let H be an n x m matrix. (a) There is always a vector, in fact a unique vector 2 of minimum norm, which minimizes
(Iz-
Hx(('.
(b) 2 is the unique vector in a ( H T )which satisfies the equation
HX = 1 where 1 is the projection of z on R ( H ) . Proof: By (2.1 1) we can write z=1+z"
where 2 is the projection of z on M ( H T ) .Since H x E ~ ( Hfor ) every x , it follows that h - Hx E W ( H )
and since
z" E R'(H), 15
z"
I2 - H x .
16
I11
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
Therefore
This lower bound is attainable since 2, being in the range of H, is the afterimage of some x o : 2=Hx~. Thus, for this x o , the bound is attained:
//z-Hxo/12 =
= 11q2.
IIZ-P/IZ
On the other hand, we just showed that
//z-Hx/I’ = IIP-HXII’
+ (12’112
so that the lower bound is attained at x* only if x* is such that Hx* = 4. For any such x*, we can decompose it via (2.1 1) into two orthogonal vectors:
x* = A*
+ i*
where
A*
E
B(HT)
and
i*E N ( H ) .
Thus
Hx*
=
HA*
so that
and IIx*II2
= 11A*112
~~z-Hx*II*= 11~-HA*11~
+ /jz*1122
/lA*/j2
with strict inequality unless x* = A* [i.e., unless x* coincides with its projection on .g(HT)-which is to say, unless x* E %?(ITT)to begin with]. So far, we have shown that xo minimizes lIz-Hx112 if and only if H x , = 9, and that, among those vectors which minimize Ilz-Hx1I2, any vector of minimum norm must lie in the range of H T . To demonstrate the uniqueness of this vector, suppose that 2 and x* both are in 9 ( H T )and that
HA = Hx* = 2. Then
x* - A E 9 ( ( H T ) . But H(x* - 2 )
=0
so that
x* - 2 EM(H)= B1(HT)as well.
(2.10)
Thus x * - 2 is orthogonal to itself, which means that Ilx*-AIl’ = 0 (i.e., X* = A). H
UNIQUENESS
17
(3.1.1) Comment: An alternate statement of (3.1) which is equivalent, but perhaps more illuminating is this: There is always an n-dimensional vector y such that
~ ~ Z - H H=~infIlz-Hx/12. Y~/~ X
If llz-Hxo~12 = infl(z-Hx112 X
then I/xo/j>, 11 HTyll, with strict inequality holding unless xo = H'y. y satisfies the equation
HH=Y= 2 where 2 is the projection of z on 9 ( H ) . (3.1.2) Exercise: l(z-Hxl12 is minimized by xo if and only if Hx, where 2 is the projection of z on 9 ( H ) .
= 2,
The minimal least squares solution alluded to in (3.1) can be characterized as a solution to the so-called normal equations:
(3.2) Theorem: Among those vectors x, which minimize I1z- Hxl(', 9,the one having minimum norm, is the unique vector of the form
R
(3.2.1)
= HTy
which satisfies (3.2.2)
HTHx = H'z.
Comment: The theorem says that 9 can be obtained by finding any vector yo which satisfies the equation
HTHHTy = HTz and then taking
R
= HTyo.
Proof: By (2.12), 9 ( H T ) = 9 ( H T H ) . Since HTz is in the range of HT, it must therefore be in the range of HTHand so, must be the afterimage of some x under the transformation HTH. In other words, (3.2.2) always has at least one solution in x. If x is a solution to (3.2.2), so then is 2, the projection of x on 9 ( H T ) , since Hx = H2, (2.11.1). Since 2 E B(HT), it is the afterimage of some vector y under HT:
R
= HTy.
I8
I11
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
So far we have shown that there is at least one solution to (3.2.2)of the form (3.2.I). To show uniqueness, suppose and
9, = H T y l
22 = H T y 2
both satisfy (3.2.2).Then
H T H ( H T y y-, H T y 2 ) = 0 so that H T ( y ,- y z ) E A ' ( H T H )
=
M(H)
(2.12)
which implies that
H H ~ ( Y , - ~ , )= 0. Thus
(Yl - Y 2 ) E M ( H H T ) = JWT)
(2.12)
and so =
HTyl
=
HTy2 = t2.
Thus, there is exuctly one solution to (3.2.2)of the form (3.2.1).If we can show that this solution also satisfies the equation
HX
=
2
where i is the projection of z on 2 ( H ) then, by virtue of (3.1b) we will be done. But, in (2.11.1)we showed that
(3.2.3)
HT2
=
HTi.
I n Theorem (3.l),we showed that there is a unique solution in 9 ( H T )to the equation
(3.2.4)
HX
=
2.
This (unique) solution therefore satisfies the equation
HTHx = HT2 as well. Since H T z = H T 9 , (3.2.3),we see that the (unique) solution to (3.2.4) which lies in .%'(HT)must coincide with i, the unique solution to (3.2.2) which lies in :%'(FIT).In summary, the vector P alluded to in the statement of (3.2)coincides exactly with the vector ialluded to in (3.1). We are now in a position to exhibit an explicit representation for the minimum norm solution to a least squares problem. A preliminary lemma is needed which, for the sake of continuity, is stated here and proven later:
H + = lim,,,(HTH+6'I)-'HT
19
(3.3) Lemma: For any real symmetric matrix A , PA = lim(A+dI)-'A = limA(A+H)-' 6- 0
6-0
always exists. For any vector z, h = P*z is the projection of z on W ( A ) .
(3.4) Theorem: For any n x m matrix H,
H+
(3.4.1)
=
lim(HTH+621)-'HT
6-0
= lirnHT(HHT+BZI)-'
(3.4.2)
610
always exists. For any n-vector z,
R
= H'z
is the vector of minimum norm among those which minimize
I~Z-
Hxll'.
Comment: Henceforth, we use the symbol I to represent the identity matrix, whose dimensionality is to be understood from the context. For example, in the expression HTH+I, we refer to the m x m identity, whereas in HHT+ I, we refer to the n x n identity. Proofi
Since (H~HH'+PH~)
=
H ~ ( H H ~ + P I )= ( H ~ ' H + ~ ' Z ) H ~
and since (HHT+ # I ) and (H'H+d21) have inverses when 6' > 0, (2.13), it is clear that the right sides of (3.4.1) and (3.4.2) are equal if either exists. Let z be a given n-dimensional vector, and decompose z into its projections on W(H) and N ( H T )according to (2.1 1); z=h+Z".
Since HTz = HT9
(2.11.1)
and since 9 E W(H) must be the afterimage of some vector xo under H, we see that
(3.4.3)
(H'H
+ 6'1) -
' 'HTHx,.
H T z = (HTH+ d21)- HT2 = ( H T H + 6'1)-
20
111
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
The limit of the last expression always exists and coincides with to,the projection of x, on 9 ? ( H T H ) , by virtue of (3.3). Since .%(IfT) = 9?(HTH), (2.12), and since
2 = Hx, = HAo
(2.1 1.1)
we conclude that .to= lim(HTH+62f)-'HTz 6-0
always exists, is an element of &?(IfT), and satisfies the relation
H2, = 2 where 2 is the projection of z on B ( H ) . The desired conclusion follows directly I from (3.1).
(3.5) Corollary: For any vector z, H H ' z is the projection of z on g ( H ) and ( I - H H + ) z is the projection of z on N ( H T ) .For any vector x, H+Hx is the projection of x on B ( H T ) and ( I - H ' H ) x is the projection of x on J(H>. Proof: By (3.4.2), H H + = lim,,oHHT(HHT+62Z)-' and by (3.4.1), H'H = lim,,,(HTH+62Z)-'HTH. (3.3) tells us that HH'z is therefore the projection of z on &?(HHT), which coincides with the projection of z on B ( H ) , (2.12). Similarly, (3.3) and (2.12) imply that H + H x is the projection of x on g(HTH)= 9?(HT).
Since 2-2 is the projection of z on J ( H T ) if 2 is the projection of z on 9?(H), (2.1 l), it follows that
z- HH'z
is the projection of z on N ( H T ) .
By the same token, (Z-H+H)x
is the projection of x on N ( H ) .
The matrix H + , which we explicitly define in (3.4), is the so-called "MoorePenrose generalized inverse for H." We will usually refer to it more familiarly as "the pseudoinverse of H." Corollary (3.5) is tremendously important and should be noted carefully. It expresses the four most fundamental projection operators in terms of pseudoinverses. The results of (3.4) are attributed (via a slightly different method of proof) to den Broeder and Charnes [l].
(3.5.1) Exercise: H +
=H-'
if H is square and nonsingular.
HH
+
AND
H H +
ARE PROJECTIONS
21
(3.5.2) Exercise: H + = HT(HHT)-' if the rows of H are linearly independent. [Hint: Apply (2.12.2) to show ( H H T ) has an inverse. Then use (3.4.2).] (3.5.3) Exercise: H + independent.
if the columns of H are linearly
= (HTH)-'HT
Before proceeding to the light task of milking these results for all they are worth, we pause to furnish the proof for Lemma (3.3), as promised: Proof of (3.3): If A is any symmetric matrix and So is a nonzero scalar whose magnitude is less than the magnitude of A's smallest nonzero eigenvalue, then for any 6 with
0 < 14 < 1601 ( A + SZ) is nonsingular and hence, for all such S's, ( A + SZ)-' exists. If z is any vector, we write z=2+,? where 2E 9(A)
z" E N ( A )
(2.1 1)
and AZ = AL.
(2.11.1)
Since i E B(A),we can write i = Ax, for some xo and so (A+SZ)-'Az
=
(A+SZ)-'A2
= (A
+ SZ)- ' A (Ax,).
From (2.14), the diagonalization theorem, we conclude that A = TDTT
where D
=
diag(A,,A,, ..., A,)
is the diagonal matrix of A's eigenvalues and T is an orthogonal matrix:
TT = T-I. Thus
+
( A 61) - 'AZ
=
( A + 61)- ' A ~ x = , T(D + SZ)- 'DTx,.
Element-by-element, it is plain to see that lim(D+SZ)-'D2 = D 6-0
22
111
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
so that lim(A+61)-'Az
=
\' 6-0
T D T ~ X=, A x , = 2
the projection of z on 9 ? ( A ) . The same argument works for lim,,,A(A+6Z)-'z.
In (3.5.1)-(3.5.3), formulas for H + in terms of inverses were given for the case where the rows and/or the columns of H are linearly independent. There are cases though where these conditions need not obtain:
In such cases, H does not have a simple formula in terms of inverses. However, a better understanding of H + can be had by treating, in turn, the following cases: H a 1 x 1 matrix, H diagonal, H symmetric, H rectangular: If H is a 1 x 1 (scalar) matrix, then +
H+
=
lim ( H 2 + d 2 Z ) H =
6240
If H is diagonal:
if H = O ( O1/H
if H # 0.
H = diag(Al,A2, ...,A,,,)
then
H
+
=
diag (Al +,&+,...,A,')
where if 1, = 0 if Aj # 0.
l/Aj
In terms of least squares, this makes sense, for if
and x =
is to be chosen to minimize
Iiz-Hx1/2
n
=
1= I ( [ j - A j 5 j ) 2
SPECIALIZATION TO SYMMETRIC
H
23
it is clear that the choice
cj*
=
Cj1J-j
if Aj # 0
arbitrary
if Aj = 0
will minimize the sum of squares, and that
c n
llX*1l2 = j = I g 2 is made smallest when
tj* = 0
if Aj = 0.
Thus, the minimum norm solution for the case of a diagonal H i s
where
tj = A
~ ~ c ~i.e.,;
2 =~ + z .
(3.6) The Special Case of Symmetric Matrices If H is a symmetric m x m matrix, the diagonalization theorem allows us to write
H = TDTT where T is an orthogonal matrix and D is diagonal. BY (3.4),
Ht
=
limT(D2+d21)-'DTT
=
T[lim(D2+d2Z)-1D] TT
6+0
6-0
= TD+T~.
Thus, the pseudoinverse for a symmetric matrix is obtained by pseudoinverting the diagonal matrix of its eigenvalues. Since H i s nonsingular if and only if its eigenvalues are nonzero (in which case D t = W ' ) ,we see that
H t = TD-'TT if H is symmetric and nonsingular: Since TTT= TTT=I, it is easy to see that H H i = H ' H = I i n t h i s c a s e , s o t h a t H + =H-'.
24
111
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
The last result can be expressed in a different notation, and gives rise to various so-called spectral representation theorems [cf. (3.15)] : If the column vectors of T are denoted by ti (i = 1, ..., m), so that we can write T as a partitioned matrix,
T
=
(t,it,i.-.it,,,)
the diagonalization theorem states that (3.6.1)
H
=
=
TDT'
rn
=I
jLjtjtjT.
Furthermore, since
T ~= TI this tells us that t:tj
=
(
1
if i = j
0
otherwise
so that the columns of T are orthonormal (mutually orthogonal with unit length). Furthermore,
HT = TDT'T
=
TD
which can be read column by column as Htj
=
j = 1,2,..., m .
)Litj
Thus, each t j is an eigenvector of H associated with the eigenvalue A,. If the Aj's are not all distinct, the tj's are mutually orthogonal nonetheless. The fact that
H+
=
TD+TT
can be expressed as (3.6.2)
W
H+
= j= 1
jLj+tjt:.
(3.7) Exercises (3.7.1) Let H be an m x m symmetric matrix and suppose the nonzero eigenvalues of H are I,,, R,, ..., R, ( k < m). (a) Show that there is a representation for H of the form H
=
TDT'
EXERCISES
25
-
where
m - k zeros
D
=
diag(A,, I,, ..., A,, 0,. .., 0)
and T is an orthogonal matrix. (Hint: If D is not arranged properly, use a permutation matrix, which is always orthogonal, to do so.) (b) If the columns of T a r e denoted by t , , ? , , ..., t m ,show that
B(H)=
Y(l~,...,fk)
and
N(H)=z(fk+l,
..., lm).
(3.7.2) Without appealing to the diagonalization theorem, show directly that if H is symmetric and if H x = A, x and Hy = A,y then x l y if I l # A,. The representation (3.6.2) is particularly interesting since it clearly shows the radically discontinuous nature of pseudoinversion. Two matrices may be very close to each other element by element. However, if their ranks differ (e.g., if one is singular, while the other is nonsingular) their pseudoinverses usually differ greatly. For example, the diagonal matrices and
D,
=
(
4
0
0
10-l0
)
are close to each other, but
differ greatly. In terms of (3.6.2), it is easy to understand why, since the transformation
I+ = (
l/I
if I # 0
0
if I = O
exhibits an infinite discontinuity at A = 0. This characteristic induces serious computational difficulties which we will discuss at the appropriate time. I n (3.8), we will see that the pseudoinverse of an arbitrary rectangular matrix is expressible in terms of the pseudoinverse of symmetric matrices: H+
=
(HTH)+HT
= HT(HHT)+
so that one can, in theory, diagonalize symmetric matrices (for which wellknown algorithms are available) and proceed directly to pseudoinversion from there. However, there are other, less tedious methods for computing H (Chapter V). The problem of round-off is quite serious, as the reader may have already come to appreciate. +
26
1x1
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
(3.7.3) Let H be an n x m matrix and let xl, x2, ..., x , r normal set of n-dimensional vectors such that
< m be an ortho-
9 ( H ) = L?(xI,x2,..., x,). Then
H H + = i]xjx;.7. j= 1
[Hint: Use (3.5) and (2.7).] A symmetric matrix P is called a projection matrix if it is idempotent (i.e., P 2 = P).
(3.7.4) The eigenvalues of a projection matrix are either zero or unity. .Y
(3.7.5) If P is a projection matrix, P on & ? ( P ) If . x E W ( P ) , then P x = x.
+
=P
and P x is the projection of
(3.7.6) HH', H + H , I - HH', and I - H + H are projection matrices. (3.7.7) If P , , P,, ..., P , are projection matrices having the property that Pi Pj = 0 if i # j and if P is another projection such that then
P
x:=
n
= j= I
Pi
[Hint: First show that Q E Pj is a projection matrix. Then Q' = Q and so QQ'x = Qx which is the projection of x on W ( Q ) . On the other hand, P P ' s = P s is the projection of x on the range of P. Since 9 ( P ) = 9 ( Q ) , we have P x = Qx for all x.] (3.7.8) Let h , , h 2 , . . . . h nbe a set of vectors and let H be the matrix whose column vectors are h , , h,, ..., hn. Then for any x, HH+x is the projection of .Y on Y ( h 1 h,, , ...,An). (3.7.9) If H is any matrix, then H x for some y . (3.7.10) If H is any matrix, then z for some u.
=0
E
if and only if x
= (I-
% ( H ) if and only if z
H+H)y =
HH+u
We now turn our attention to an exploration ofthe most important properties of H'. (3.8) Theorem: For any matrix H. (3.8.1)
H'
=
(HTH)+HT
PROPERTIES OF
(3.8 2 )
(HT)'
(3.8.3)
H+
H+, H'H, HH+, ETC.
=
(H+)T
=
HT(HHT)+.
27
Proof (HTH)' H T = {lim[(HTH)2+6ZZ]-1(HTH)} H' S+O
and
H + = lim[HTH+621]-' HT 6-0
(3.4)
By (2.1 l), any vector z can be written as where
z
=
Hx,
+ z"
for some xo
HTz"= 0.
Thus (3.8.4)
(HTH)' HTz = lim[(HTH)Z+62Z]-1( H T H ) 2 ~ o 6-0
and
(33.5)
H'z
lim[HTH+6'Z]-'
=
6-0
HTHx,.
Using the diagonalization theorem (2.14), we write H ~ H = TDT~
where D is diagonal and T is orthogonal. Equation (3.8.4) takes the form (HTH)'HTz = T{lim[D2+S21]-1D2} TTxO 6-0
while (3.8.5) takes the form HTz = T{lim[D+621]-'D} 6+0
TTx,.
For diagonal D's, the matrices in braces are the same, which proves (3.8.1). To prove (3.8.2), notice that (HHT+SZI)-' is symmetric so by (3.4.1),
(HT)+ = lim(HHT+ S2Z)-' H 6-0
= =
lim [HT(HHT+ 6'1) - 'IT
6-0
(H')T.
(3.4.2)
To prove (3.8.3), we use (3.8.2) to establish that (3.8.6)
[(HT)'JT
=
H+
28
Ill
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
and (3.8.1) to establish that (3.8.7)
( H T ) + = (HHT)' H .
Since (HH')' is symmetric, (3.8.3) follows from (3.8.6) after taking transposes in (3.8.7).
In his paper of 1955, which was probably responsible for the rebirth of interest in the topic of generalized inverses, Penrose [I], characterized the pseudoinverse as the (unique) solution to a set of matrix equations. The pseudoinverse H that we have been investigating satisfies the Penrose conditions: +
(3.9) Theorem: For any matrix H , B = H + if and only if
H B and B H are symmetric
(3.9. I ) (3.9.2)
HBH
=
H
(3.9.3)
BHB
=
B.
Proof of necessity
H H + = lim H H T ( H H T + d 2 Z ) - ' 6-0
and H'H
=
lim(HTH+d2Z)-' H T H .
6-0
(3.4)
Both are symmetric. This shows that H + satisfies (3.9.1). By (3.5), H H + is the projection on 9 ( H ) . Since H x ~ 9 2 ( H for ) all x,(3.7.5) assures that ( H H + ) ( H x )= Hx.This shows that H f satisfies (3.9.2). By (3.8.1), (3.9.4)
H+H
=
(H~H)+(H~H).
By (3.8.1) and (3.9.2) Hf
=
(HTH)'HT
=
(H~H)+H~(HH+)~.
=
(HTH)'[H(HfH)JT
Since H H + is symmetric, (3.7.6), H+
=
( H T H ) + H T ( H H t )= ( H T H ) + ( H T H ) H +
and by (3.9.4), the last is equal to ( H + H )H'. This establishes (3.9.3).
PENROSE'S CHARACTERIZATION OF
Proof of suficiency: Since
H+
29
Suppose B satisfies (3.9.1)-(3.9.3): and
BH
=
(BH)T
H
=
HBH = HHTBT.
H
=
HBH
Since HH'H H+H
=
H
=
H + ( H H T B T )= [ H ( H + H ) I T B T
and so H + H = H'BT
(3.9.5) Since B
= BHB
=
BH.
and since HB is symmetric,
(3.9.6)
BT = HBB'.
Premultiplying (3.9.6) by HH', we find that H H + B T = HH'HBB'
=
HBBT
(3.9.2)
and by (3.9.6) the last is equal to B'. Thus (3.9.7)
BT
=
(HHf)BT.
Taking transposes in (3.9.7), we find that B
=
B(HH+)' = ( B H ) H +
and by (3.9.5) we finally conclude that B
=
H+HH+.
Since H + H H + = H + , we see that
B=H+. The Penrose characterization for pseudoinverses is extremely useful as a method for proving identities. For instance, if one thinks that a certain expression coincides with the pseudoinverse of a certain matrix H , a handy way of deciding is to run the expression through conditions (3.9.1)-(3.9.3) and observe whether or not they hold. (3.10) Exercise: If A and B are nonsingular, it is well known that ( A B ) - ' = B - ' A - ' . Use (3.9) to show that it is not generally true that (AB)' = B + A + . Where do the conditions break down? Exhibit a counterexample. [See (4.10)-(4.16) for a detailed study of this problem.]
30
111
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
(3.11) Exercise: Prove the following:
H.
(3.11.1)
(H')'
(3.11.2)
( H T H ) += H + ( H T ) + and ( H H T ) + = ( H T ) + H C .
(3.11.3)
If A is symmetric and c i > 0, then (A")' and Az(A')+ = ( A z ) + A x= A A + .
(3.1 1.4)
(HTH)'
(3.11.5)
d ( H + ) = W ( H + H )= 9'?(HT); J " ( H ) = N ( H + H ) = A' [ ( H T H ) +1.
(3.11.6)
If A is symmetric, A A +
(3.11.7)
H H + = ( H H ' ) ( H H ~ ) += ( H H ~ ) + ( H H ~ ) and H ' H = ( H T H ) ( H T H ) ' = ( H T H ) + ( H T H ) .
(3.1 1.8)
If A is symmetric and
(3.11.9)
If H is a nonzero n x 1 matrix (a column vector) H f and H H + = HHT/11H//2.
=
=
= (A')"
H + ( H H T ) + H= H T ( H H T ) ' ( H T ) + .
s(
=
A'A.
> 0, A'A"
= A'A'. =HT/HTH
The properties of pseudoinverses and projections which we have developed can be readily applied to the theory of least squares subject to constraints. But first, we summarize the general results for unconstrained least squares and the related theory of linear equations, in the language of pseudoinverses: (3.12) Theorem: (a)
.yo
minimizes
(3.12.1)
~IZ-HXI/~
if and only if xo is of the form (3.12.2)
so
= H+Z
+(I-H+H)y
for some y . (b) The value of x which minimizes (3.12.1) is unique if and only if H + H = I . The last is true if and only if zero is the only null vector of H . (c) The equation (3.12.3)
HX
=z
has a solution if and only if
H H ' z = Z. The last is true if and only if z E B?(H).xo is a solution to (3.12.3) if and only if it is of the form (3.12.2). Equation (3.12.3) has a unique solution ( = H + z ) if and only if H H ' z = z and H f H = I.
CHARACTERIZATION OF ALL SOLUTIONS TO LEAST SQUARES MINIMIZATION
31
Proof: (a) Ilz-Hxll' is minimized by x, if and only if H x , = 9, where 2 is the projection of z on %'(H), (3.12). By (3.4), H + z minimizes Ilz-HxlI' so that H x , = H ( H + z ) . This means that x , - H + z is a null vector of H if xo minimizes (3.12.1). The last is true if and only if
xo - H'z = ( I - H + H ) y
for some y .
(3.7.9)
Conversely, if x, has the form (3.12.2), then H x , = H ( H + z ) since H(Z- H ' H ) = 0, (3.9.2). This proves part (a). (b) The value of x which minimizes (3.12.1) is unique if and only if ( I - H + H ) y vanishes for all y . This can only happen if the projection of all vectors on N ( H ) is zero, which means that N ( H ) consists only of the zero vector. ( I - H + H ) y = 0 for all y , by the way, if and only if H + H = I. This proves (b). (c) Equation (3.12.3) has a solution if and only if z is the afterimage of some x under H . This is, by definition, the same as saying z € % ' ( H ) . By virtue of (3.7.10), the last holds true if and only if z = HH+U
for some u. Since H H + is a projection, (3.7.6), it follows that HH'z
=
( H H + ) 2=~ HH'u
= Z.
When (3.12.3) has a solution x , , this solution must minimize IIz-HxI12 (the minimal value in this case being zero) and so x, must be of the form (3.12.2). The solution is unique if and only if H ' H = I [part (b)] and these conclusions collectively serve to establish (c).
(3.12.4) Corollary: Let G be a rectangular matrix and suppose u is a vector in %'(G). Then (a) Y = { x : Gx = u } is nonempty and x, minimizes llz- Hxll' over if and only if x,
= G+U
Y
+ A + z+ (I-G+G)(z-R+A)y
for some y , where
I
=z
- HG+u
and
R
= H(Z-G+G).
(b) The vector of minimum norm among those which minimize llz- Hx//' over Y is
G+u+R+z. Proof:
(a) If u E B ( G ) then by (3.12), Y is nonempty and
9 = { x : x = ~ + + u( z - G + G ) ~ for some u } .
32
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
111
Therefore min IIz-HxII = min IIZ-Wull. v
X € Y
The latter minimum occurs at uo if and only if forsome y
u,,=R+z+(I-R+R)y
(3.12a)
so that xo minimizes llz-Hxll over 9 'if and only if
+
+
+ u( I - G + G ) [ R+ z( I - B + R ) y ]
xo = ~
for some y . (3.8.3)
and since (I-G'G)'
= (I-G'G)
=
(3.7.6)
(I-G+G)T
it follows that (3.12.4.1)
(I-G+G)B+
=
(I-G+G)'HT(RRT)+
=
R'
and so it is clear that any value of x which minimizes I/z-HxII over 9 'is of the form XO
= G+u
+ R + z+ ( I - G + G ) ( Z - R + B ) y
for some y . (b) [ ( I - C + C ) ( I - 17 +B)yITG+u= yT(Z- R + R ) ( I - G+G)G+u since (Z-G'G) and I-R'H are symmetric. The last is zero since (I-G+G)G+ = Gi - G+GG+ = 0. Thus (3.12.4.2)
( Z - G + G ) ( I - R + R ) y I G'u.
On the other hand,
B R ) y ] T R+z= yT(1- R +B) ( I - G +G ) R +z = yT(Z-R+R)R+z. (3.12.4.1)
[ ( I - G+G)( I -
+
Since R - R ' R B = 0, we see that ( I - G+G)(I- B + R ) yI 17 + Z as well, so that if xo minimizes llz-Hxlj2 over 9, then +
+
11x0
I/ ' =
IlG+u+R '211'
+ i/(I- G+C)(I- W +R)yII'
3 IIG+u+R+Zll'
with strict inequality holding unless x,, = G+u
+ B'5.
LINEAR PROGRAMMING
33
(3.12.5) Exercise: (a) The equation H x = z has a solution for all z if the rows of H are linearly independent. (b) If the equation Hx = z has a solution, the solution is unique if and only if the columns of H are linearly independent. (3.12.6) Exercise: Let H be an n x m matrix with linearly independent columns. For any k x m matrix G, let R = H(I-G+G). Show that (Z-G+G)(Z-B'R) = 0. [Hint: If w = (Z-G+G)(Z-~+R)u,thenHw = 0. Apply (2.12. l).] Comment: If the columns of H are linearly independent and u is in the range of G, then (3.12.6) and (3.12.4) together, imply that there is a unique vector which minimizes Ilz-Hxl12 over 9'.In general, though, the minimizing vector is not unique. However, the vector, 2o = G + u + R + Z , has minimum norm among those which minimize ( ( z - H x ( j 2over Y . The vector % = R'.? is the vector of minimum norm among those which minimize IlZ-Rx((2subject to no constraints. E and 2o differ by G'u, the minimum norm vector in 9.
(3.12.7) Exercise: [Refer to (3.12.4).] If Y is empty, then x*
= G+u+
R'Z is the vector of minimum norm which minimizes ( / z - H x ( (over Y * where Y* = {x: IIGx-uIl' is minimized}. [Hint: Y *= { x : x = G+u-(ZG + G ) ufor some v . } The proof of (3.12.4) carries over word for word.]
(3.12.8) Application to Linear Programming (Ben-Israel and Charnes [2], also Ben-Israel et al. [l] ) Let a, 6 , and c be specified n-vectors and let A be a given m x n matrix. Consider the problem of minimizing cTx with respect to x, subject to the constraints
a
a
34
111
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
which can be made arbitrarily small and negative unless c is orthogonal to every null vector of A . Thus, the problem is bounded if and only if c E "(A)
(2.10)
= &(AT).
Since A + A c is the projection of c on &(AT)
(3.5)
we see that boundedness implies A + A c = c. Conversely, if A'Ac = c, then c E W ( A T )so that c = ATz for some z. Thus cT.y = zTAx. Each component of A x is bounded below on the constraint set so that zTAx is bounded below as x ranges over the constraint set. Henceforth, we assume that c E 9?(AT). We now make the additional assumption that the rows of A are linearly independent. In this case, the equation A x = z has a solution for every m-dimensional vector z , (3.12.5), and so min cTx = min min cTx.
a
The set of x's for which z
= Ax
a
r=Ax
is of the form
A+Z
+(z-A+A)y
where y is free to vary unrestricted over n space, (3.12). Since c E &(AT) = N 1 ( A ) ,we see that cis orthogonal to all null vectors of A [of which ( I - A + A ) y is one] so that cTx = c T A + z
if A x
= z.
Consequently, rnin cTx = rnin cTA+z
a
a
=
min ( A + T ~ ) T ~
a
and if z" minimizes the right side, any x of the form 2 = A+P
+ (I-A+A)y
will minimize the left side over the constraint set. The minimization of the right side is trivial: Denote the components of a, b, A + T ~and , z by x i , pi, y i , and Ci,respectively. Then
SOLUTION OF MATRIX EQUATIONS
35
and the components of the solution vector are clearly
ei
=
{
if yi > 0 :nything
between
( a i and
Pi
(inclusive)
if yi
=
0.
The present solution depends crucially upon the assumption that the rows of A are linearly independent. If this assumption is violated, a modified procedure (more complicated) must be resorted to in order to solve the problem (Ben-Israel and Robers [I]). (3.13) Exercises
(3.13.1) If A and B are matrices with the same number of rows, then .%?(A)E 9 ( B )
if and only if BB'A
=
A.
(3.13.2) If A and Bare matrices with the same number of rows, the matrix equation
BX= A has a solution if and only if W ( A ) c W ( B ) . In this case, any matrix of the form
x = B+A + (I-B+B)Y
(where Y has the same number of rows and columns as X ) is a solution. The solution is unique if and only if B'B = Z [i.e., N ( B ) = { O } ] .
(3.13.3) If M is nonsingular and A is rectangular with the same number of rows then ( M A ) + ( M A ) = A'A. If N is nonsingular and A is rectangular with the same number of columns,
(AN)(AN)+ = AA+. (3.13.4) The matrix equation AXB = C has a solution in X if and only if
A A i C B f B = C.
36
I11
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
In this case, the general solution is
X = A + C B + + M - A'AMBB'
where M is arbitrary (same number of rows and columns as X ) . (3.13.5)
For any matrix A,
A+
=
WAY
where W and Yare, respectively, any solutions to
WAAT = AT and
ATAY = A' (Decell, [l]). (3.13.6) A matrix is said to be normal if it commutes with its transpose. In this case, show that
A+A = AA+. (3.13.7) If T is orthogonal, (AT)'
= TTA+.
(3.13.8) If 9 ( B ) c B(A),then among those matrices X that satisfy
A X = B, the one which minimizes the trace of X T X is If A 2 = B, then trace(ZTZ) > t r a c e ( P 2 )
2 = A'B. if
z z 2.
(3.13.9) The following conditions are equivalent: (a) XH+ (b) XHT (c) X H + H
= = =
0, 0, 0.
(3.13.10) If P is a projection, and if H is any rectangular matrix (having the right number of columns) then (RTR)'
=
P(RTR)+
=
(RTB)+P,
and
B+ = (RTjJ)+HT where
i7
= HP.
pR+ = R + ,
STATIONARY PROBABILITY FOR A MARKOV CHAIN
37
(3.14) Exercises
(3.14.1)
Let h,, h 2 , ... h,, ... be any collection of vectors and define A, = I
[A,-
,
if h, is a linear combination of h,, h,, ...)h,-
Show that for each n (a) A , is the projection on P ( h , , ..., A,). (b) A , h,, is the (unnormalized) (n+ 1)st vector in the Gramm-Schmidt orthogonalization of h l , hz, ...,An+ 1 . (3.14.2) Stationary Probability for a Markov Chain
A matrix P is called a stochastic matrix if its entries are nonnegative and its row sums are unity. Such matrices can be used to represent the one step transition probabilities for homogeneous Markov chains. If the process is ergodic (a condition which can be deduced by visual inspection of the transition matrix, Feller [l, Chap. XV]) then lim (P*)” = (xi x i
n+ w
i x)
where x is the unique probability vector (a vector having nonnegative components which sum to unity) satisfying
PTX
=
x.
This probability vector is the steady state probability distribution for the Markov chain. The ith component of x represents the (steady state) probability that the process is in state i at any given instant. Using these facts as a starting point, show that (a) y = PTy only if y is a multiple of x. (b) The row vectors of I - P T are linearly dependent. Denoting the columns of I - P by q l , q2,..., q N , show that (c) q l , q2,..., qN- are linearly independent. (d) If A , = I , An = An-, - ( A n - , q n ) ( A n - , q , ) T / q ~ A , - l q n
n = 1,2,...,N- 1
38
III
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
and u is the N-dimensional vector whose components are all ones, then = AN- 1 U/UTAN-1 U.
X
cf. Decell and Ode11 [Hint: A N - l is the projection on -YL(q,,q2,...,qN-l);
c11.1
In (3.6.1) and (3.6.2) we showed how a symmetric matrix and its pseudoinverse can be represented in terms of its eigenvalues and eigenvectors. Using the theory which we have developed thus far, we can deduce an analogous result for an arbitrary rectangular matrix A in terms of the eigenvalues and eigenvectors of A’A and AA’ (see Good [l]). We begin by reminding the reader that an y matrix of the form ATA has a unique symmetric square root, which has the explicit representation (ATA)”
= TD”TT
where Tis the orthogonal matrix which reduces A’A to the diagonal matrix D , A ~ = A TDT~
and D” is obtained by taking the (positive) square root of D’s (nonnegative) diagonal entries. [D’s entries are clearly nonnegative since D = TTATAT
so that for any x X ~ D= X IIATxl/’ 2 0. In particular, if D = diag(d,, ..., dn) and x has all zero components except for a 1 in t h e j t h place, X’DX = dj 3 0.1 (3.15) Theorem: Let A be an n x rn matrix and let L be the r x r diagonal matrix of (AAT)’s nonzero eigenvalues arranged in arbitrary order. Then there is an n x r matrix P and an r x rn matrix Q such that the following conditions hold:
(3.15.1)
A
=
PL”Q.
(3.15.2)
AAT = PLPT,
(3.15.3)
AA’
(3.15.4)
P T P = I.
(3.15.5)
A ~ = A Q~LQ.
(3.15.6)
A’A = QTQ.
(3.15.7)
QQ’
=
=
PPT.
I.
SINGULAR VALUE DECOMPOSITION THEOREMS
39
Comment: By (3.15.4) and (3.15.7), the columns of P are orthonormal as are the rows of Q. By (3.15.2) and (3.15.4), AA‘P = P L which shows that the columns of P are eigenvectors of AAT, the j t h column being an eigenvector associated with t h ej t h diagonal entry Aj of L. By (3.15.5) and (3.15.7), the same goes for t h e j t h column of Q’ (which is t he jth row of Q). This result is often referred to as the singular decomposition theorem. Proof:
By (3.11.3) and (3.11.7) AA’
=
(AAT)%[(AAT)%]+
A
(AAT)” [(AAT)”]’A.
so by (3.9.2) (3.15.8)
=
By virtue of (3.7.1) (3.15.9)
A A= ~ TDT~
where
n-r
D = diag(A,,I, ,...,A,,O,O ,..., 0),
Aj > 0, and T is an orthogonal matrix. Let us express T and D as partitioned matrices : r n-r
n
nT
=
n[Pi P o ] r n-r
Li 0 ~ = n ~ r....[....o_.__. 0 i1
where
L
=
diag(A,,I,, ..., A,).
Then (3.15.9) can be rewritten as (3.15.10)
AA’
=
PLP~
which proves (3.15.2). Define (3.15.1 1)
Q = PT[(AAT)”]’A.
Since (AAT)”
and since
= TD”TT
40
111
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
we can write (3.1 5.12)
(AAT)"
= p~"pT.
Therefore, by (3.15.1 1) PL"Q = (PL"PT) [(AAT)"]+A
(3.15.12) and (3.15.8)
= A.
This proves (3.1 5.1). AA+
(3.1 1.7)
=
(AA~)(AA~)+
=
( T D T ~ ) ( T D + T=~ T) D D + T ~
= PPT
which proves (3.15.3). Since T is an orthogonal matrix, its columns are orthonormal, so P T P = Z which establishes (3.15.4). QTLQ = A T [ ( A A T ) 1 / 2 ] + P L P T [ ( A A T ) " ] + A = AT[(AAT)"]+(AAT)[(AAT)"]'A
(3.1 5.12)
(3.11.3) and (3.11.7)
AT[(AA+)]A
=
(3.15.1 1)
=A ~ A
which proves (3.15.5). QTQ = AT[(AAT)"]+PPT[(AAT)']+A.
(3.15.11)
By virtue of (3.15.9) (AAT)+ = T D f T T = pL-'pT
and so by (3.11.3) [(AAT)']+
= [(AAT)+]"
=
pL-"pT.
Thus Q ~ Q = A ~PL( %p T )(PP T, (PL- " p T )A = ATPL-'PTA =
AT(AAT)+A
= A'A.
(3.8.3)
SINGULAR VALUE DECOMPOSITION THEOREMS
41
This proves (3.15.6). Finally, by (3.15.11) QQT = PT[(AAT)"]'AAT[(AAT)%]+P
(3.11.3) and (3.11.7)
=P~(AA+)P = P'(PPT) P
(3.15.3)
= z.
(3.1 5.4)
(3.15.13) Exercise: The nonzero eigenvalues of ATA and AAT are identical. (3.15.14) Exercise: A + = QTL-"PT, Q'
= QT, P + = P T .
(3.16) Exercise: There exists a representation for A of the form (3.16.1) A = C Aj%pjqjT
(3.16.1)
i
where the Aj's are the nonzero eigenvalues of ATA(or AAT)repeated according to their multiplicity, (3.16.2)
AATpj = Ajpj;
ATAqj = Ajqj
and (3.16.3)
p.'p. = I J
0
if i # j
1
if i = j
= 4i 4 j .
The representation is not necessarily unique, but not all such representations [with pi and qi satisfying (3.16.2)-(3.16.3)] are valid. (3.17) Exercise: A + = xjA,:l/Zqjp: if A satisfies (3.16.1)-(3.16.3). (3.18) Exercise: Any matrix A can be represented as a linear combination of partial isometries: A = C Aj"U(Aj)
where the Aj's are the distinct nonzero eigenvalues of ATA and
U(A) = A-"A(Z-
(ATA-AZ)+(ATA-AZ)}
42
111
PROPERTIES OF THE MOORE-PENROSE PSEUDOINVERSE
satisfy
u(Ij)UT(&) = O
if k # j
O
if k # j
~'(1~)u ( A k ) A'
=
=
1I , ~ ~ U T ( 3 L j ) . i
[Hint: I - ( A T A - A Z ) + ( A T A - A I ) is the projection on the null space of
A'A -AI, which is spanned by the eigenvectors (if any) of ATA associated with 1.. Then use (3.16.1).] See Ben-Israel and Charnes [l], Penrose [I],
and Golub and Kahan [l] for related work.
(3.19) Exercise Iterative method for finding the dominant eigenvalue and associated eigenvector for AAT and A'A. Let Y o = Axo//lAxoII x,+1 = AT~,//lATY,II
n = 0,1,-...
yn+l = Axn+1/IIAxn+lll
Then lim x,
n+ m
=
q
and
limy, = p
n-
w
exist
provided that xo is not orthogonal to N ( A T A-1, I ) where Al is the largest eigenvalue of A'A. Furthermore, and
AATp = A l p
ATAq = I , q
so that
=
lim / / ~ x , j / ~ .
n- m
Chapter IV
P S E U D O I N V E R S E S OF P A R T I T I O N E D M A T R I C E S
AND S U M S AND P R O D U C T S OF M A T R I C E S
If cI,cz,..., c, are a collection of vectors in an n-dimensional space, we can write the n x rn matrix
c, = ( c l : c 2 : * - . : c m ) whosejth column is c j , in terms of Cm-l and c,: (4.1)
C, = (C,-, i ),c
(rn
=
2,3, ...).
The pseudoinverse of C , is easy to compute: (4.2)
c,+= C1=/CITC1
(3.1 1.9)
,
and so, if we can develop a convenient relationship between C,’ and C:- , we will have a nice computational procedure for pseudoinverting a rectangular matrix “a column at a time”: (4.3) Theorem (Greville [2]):
(4.3.1)
If
44
IV
PSEUDOINVERSES OF PARTITIONED MATRICES
then (4.3.2) where
Comment: ( I - C, C)’, c, ., is zero if and only if C, Cm+c,+ = c,+ [i.e., if and only if c , + ~ is in the range of C, (3.7.5)]. The range of C, is spanned by c,, . .., c, (why?) and so k,, is defined by the first part of (4.3.3) if and only if c,+ is not a linear combination of cl, ..., c,. Proof: The proof is a straightforward, though tedious verification that the right side of (4.3.2) satisfies the conditions of (3.9). We leave the details to the reader. (Greville’s original proof was constructive. The interested reader will find it instructive, as well.)
(4.4) Application to Stepwise Regression It is very common for an experimentalist t o observe samples of a function of an independent variable (e.g., time) whose functional form is not known apriori and for him to wish to model the behavior of this function (e.g., for the purpose of prediction). Typically, the experimenter has in mind a family of functions, (ppl( ~ ) , ( j f ~ ( ~ )..., , cp,(~) and the data is modeled by choosing an appropriate set of weights and representing (approximating) the observed data as a linear combination of the cp’s. To be more explicit, if the observations denoted by ll,[2,...,Cn are made at times T , , T ~ ..., , T , and if the points ( T ~ , C ~ are ) plotted on Cartesian coordinates, the problem boils down to choosing a set of scalars tl, t2,...,t, so that the graph of the function
plotted as a function of T , comes as close as possible to the given data points. The most popular method for choosing the weights is (for good cause) the method of least squares, in which the t j ’ s are chosen to minimize the sums of the squares of the (vertical) distances from the data points to the curve.
APPLICATION TO STEPWISE REGRESSION, I
In mathematical terms, the
ti's
45
are chosen to minimize
(4.4.1)
x=(y 1
If a vector notation is used, so that z=
((')
C j = (
Cn
i
qj(T1)
V j (Zn)
tm
and
cm= (cl i c2 ... ; c,)
(an n x m matrix)
then the problem of choosing tl, ..., 5, to minimize (4.4.1) is the same as choosing x to minimize (4.4.2)
112-
cnlx I12.
In (3.4) we showed that (4.4.3)
g(m) =
cm+z
always minimizes (4.4.2) and is in fact the vector of minimum norm which does the job. Suppose i(m) is computed and that the residual sum of squares [/z-CrnPq2
(which measures the degree of fit between the data and the model) is unacceptably large. The standard response to such a situation is to augment the family of by adding an (m+1)st function ~p,,,+~(.) functions q l ( . )(,p 2 ( - ) , ...,cp,(.) and then choosing the m + l weights <1,(2,...,(,+1 to minimize the new residual sum of squares
2 (1.
m+ 1
-
j = 1 < j Vj('k)>.
k= 1
which can be expressed in vector-matrix notation as IIz-Cm+lxI12 where z is as defined before,
46
IV
PSEUDOINVERSES OF PARTITIONED MATRICES
and s is now an m f I-dimensional vector, x = m+ 1
The minimum norm solution is
(4.4.4)
?(,+I)
=
p+,.
and the results of (4.3) show how 2 ( " ' + ' )is related to 2(,):
(4.5) Exercise: Let i(,) = C,2("' and
d m= ) I I Z - ~ ( ~ ) I1
a
(a) e("+" < e("') with strict inequality holding unless c , + ~ is a linear combination of c l , ...,c, or unless cm+ I(z-2("')). (b)
e'"")
=
(1-P:,rn+1lm)
Pz,m+llm
=
C ( Z - ~ ~ ~ C/IZ-Z^'"" ) ) ~ ~ ~II .+I I z~m +Il I I I +
e(m)3
and Fm+ I
= (1-
C m
C m +I ern+ I
is called the partial correlation coefficient between z and given cl, ..., c,. Interpret the partial correlation coefficient geometrically. (c) e'm' = [ f l ? = I ( l - ~ : , j l j - 1 ) ] //z/12. (d) The multiple correlation coefficient of z on cl, ..., c, is defined to be pZ,,+,
c,+
Im
I ,
r z l m=
lp(q).
ZTP)/(//Z/I.
Interpret the multiple correlation coefficient geometrically and show that
n rn
J =
1
(l-P:,jlj-l)
= 1 - r,21rn.
A word of caution is in order with regard to the interpretation of (4.5a). The object of modeling data is generally predictive in nature. Therefore, the fact that the residual sum of squares can be reduced by the addition of another regressor is not sufficient cause to add regressors ad infiniturn. A fine line must be tread between parsimonious modeling (using few regressors) and
RELATIONSHIP BETWEEN
(zj”=cj CjT)+
AND
(z?::cj cjT)+
47
getting a good fit. In many statistical contexts, underspecifying the number of regressors results in biased estimates. Over specification (too many regressors) results in loss of accuracy. The question, “How many regressors is enough?” is partially answered by the theory of the analysis of variance and covariance, under certain assumptions. The question, “How should I choose my family of regressors to begin with?” is much more difficult and not easily answered in quantitative terms. The results of (4.3) lead directly to an important representation for pseudoinverses of certain types of matrix sums: (4.6) Theorem: Let e l ,c 2 , ... be a collection of n-dimensional vectors
and let
m
s, = 11 CjCj’ j=
=
cmcm=(m = 1 , 2 , ...)
where C,,,is as defined in (4.1). Then
- (sm+crn+,)(Amcm+l)T+
(4.6.1) S,’+l = .
(Amcm+l)(sm+cm+l)T
c;f+ 1 A m c m +
1
if c , + ~is not a linear combination of c l r . . . , c m
where
(4.6.2)
A,
=
I - S,S,’.
Comment: A , is the projection on S, = C, C,’ and
9( C m Cm’)
= 9’(Sm),(3.5).
=
9(Cm)
=
p(C1,C2,.-.,Cm)
Since
(2.12)
,
(4.3.4)
we see that A,,, is the projection on 2 ’ ( c , , ..., c,), and so c,+ E 9 ( c l , ..., c,) if and only if Amc,+ = 0. Thus, c,+ is not a linear combination of cl, ..., cm if and only if A , cm+ # 0. Furthermore, by virtue of (3.14.1), Am satisfies a recursion of its own [replace h, by c, in (3.14.1)].
48
IV
PSEUDOINVERSES OF PARTITIONED MATRICES
Proof of (4.6) S,'+
1
(4.6.3)
= (Cm+1 C,'+ 1)' =
(C,'+ d'
(3.11.2) and (3.8.2)
1).
(4.6.1) results when (4.3.2) and (4.3.3) are combined in (4.6.3) and note is taken of the fact that
I - C,C,'
=
I - S,S,'
= A,.
(4.6.4) Exercise [Extension of (3.14. l)] : Let
c, = (cl ; c2 ; ... i c,) and
D,
=
D , - Do C,(CmTD0C,)+CmTD0
where Do is an arbitrary matrix of the form D, = RTR. Then (a>
D m = RTQm R
where Q,l,is the projection on ~ ' ( R c ,..., , Rc,) (b)
&+I
=
I
I D m -
(D,C , + 1 ) ( D , C , T Cm+
+
= B'(RC,,,).
if Rcm + 1 is not a 1.C. (linear combination) of Rc, ,..,,Rc, ; otherwise.
1)'
1DmCm+I
Dm
(c) If RTR is nonsingular, R C , + ~is a linear combination of R c , , ..., Rc, if and only if D , c,~,+ = 0, so that D,+ is defined by the first half of the recursion if D,,,c+, # 0, otherwise by the second half.
,
Cominmt:
If A,
=I-
C,,, C,'
then A,
=I
and
If RTR is nonsingular, D, and A, satisfy the same recursion. A, and D, difTer because of the different initial conditions. (4.6.5) Exercise:
If A is symmetric, nonsingular and h T A - ' h #
(A+hhT)-' = A - ' -
-
I , then
( A - h)( A - h)'
1 +hTA-'h
.
The results of (4.3) and (4.6) can be extended to the case of higher-order partitioning. The proofs are computational in nature, but are basically the
(UUTI- Y P ) +
49
same as (4.3) and (4.6). We state the results without proofs and refer the interested reader to the papers by Cline [2, 31 for details.
(4.7) Theorem (4.7.1) where (4.7.2) (4.7.3) and (4.7.4)
J = C+
+ (Z-C'C)
K V T U + T U + ( Z -VC')
c = (I-UU+)v
K = { I + [U + V(Z- C +C)IT[U 'V(Z- C 'C)]}
-
'.
Comment: Any matrix of the form f + D T D is nonsingular, (2.13), so K always exists. The dimension of K is the same as the dimension of CTC which is the same as the dimension of Y'Y. If U + is known, ( U ! Y)' can be computed at the expense of inverting a square matrix the size of VTV, and finding I - C'C, the projection on N ( C ) .
The extension of (4.6) to higher-order partitions is
(4.8) Theorem (4.8.1) (UUT+VVT)'
=
(CCT)'+ [f-(VC')T] x [(UUT)+- (UUT)+V(f-C'C)KV'(ULIT)i]
x
[I-vc']
where C and K are as defined in (4.7). Comment:
(4.8.2)
We can also write
c = [I-(UUT)(UUT)'] v
and (4.8.3)
K = ( I +C ( f - C ' C ) Y ' ( t l U ' ) + V ( I - C + C ) ] ~ - ' .
Here, U and Vcan be interchanged throughout the right side of (4.8.1) without altering the validity of the statement, owing to the symmetry of the left side. (4.8.4) Exercise:
( U i 0)'
)+:(
=
------.
.
(4.8.5) Exercise: In the special case where U = C,,, and V = e m + ,, show that (4.7) and (4.8) yield formulas which coincide with (4.3) and (4.6).
50
IV
PSEUDOINVERSES OF PARTITIONED MATRICES
Theorem (4.8) is extremely important if for no other reason than that it allows the development of an explicit perturbation theory for the pseudoinverse of matrices of the form A'A : (4.9) Theorem
(4.9. I ) [H'H
+I
-
'GTG]
+
=
+ 1.' ( I -
(A'R)'
A H ) (GTG)+( I - A+ +
- A"(Z-R+H)[H(GTG)+-JT x QM(A)Q[ H ( C T G ) + (]I - R ' H ) T
where (4.9.2)
A = H(I-G+G)
(4.9.3)
Q = 1-
=
HII-(GTG)+(GTG)]
RR+
and (4.9.4)
M ( i & )= [Z+I.2QH(GTG)+H T Q ] - ' .
Proof: Let CJ = GT/E.,V = H' and apply (4.8):
[H'H+I-2GTG]+
=
(CC')'
+
[Z-(HTC')]T
x [(GTC)+ - A2(GTG)+HT(Z- C ' C ) K H ( G T G ) + ] x
(I- HTC +)
where C = (I-G+G)HT =
AT,
I - C + C= Q
and
K
=
[I+~.~QN(G~G H )~+Q ] - '= ~ ( 1 ) .
But, (I-HTC')T
= (I-H'RT+)T
=
I - R+H
and
GT+HT= (HG+)T. Finally, if I + A is nonsingular, then
(I+ A ) -
=
I - ( I + A)- ' A
so that (I- C' C )K and (4.9.1) follows directly.
QM(1.) = QM(3,)Q
51
PERTURBATION THEOREMS
A scalar function cp(.), of a real variable A, is said to be O(A") as 1. 0 if cp(A)/A'' is bounded as A -+ 0. A matrix valued function is O(A") if each entry of the matrix is O(;ln). --f
(4.9.5) Corollary
o(A2)as A + 0.
(a)
M(I) = I +
(b)
[H'H+(A2)-'C'G]+
=
(HTA)' + A2(Z-A+H)(GTG)' x
( z - R + H )+~ 0(A4) as A -+ 0.
Proof: (a) M(A) is obviously O ( l ) , since limA+oM(A)= I, Since (Z+A)-' = Z - A ( Z + A ) - ' , we see that M ( A ) - I = -A2QH(GTG)+HTQM(A) so that [M(A)-Z]/A2 equals a constant matrix times M(A), which is bounded. Thus, M(A)-Z = O(A2),as asserted. (b) Follows directly.
(4.9.6) Exercise: Show that limA-,mM(1.) exists, and calculate the limit. [Hint: Let E = l/A and apply (4.9.5b) to [ Z + E - ' A ~ A ] - ' , where A = GT+ H'Q .] (4.9.7) Exercise: Under what conditions will limE-,o( H T H + ~ 2 G T G ) + exist? Comment: Exercise (4.9.7) again illustrates the extreme discontinuous nature of pseudoinverses. (See the discussion preceding (3.7.3). Also see Stewart [l] and Ben-Israel [ l ] ) .
(4.9.8) Exercise: (a) If U'U is nonsingular and A ( ] . ) = O(1) as /I + 0, then U T U + A A (A) is nonsingular for all suitably small A, and (b) [UTU+AA(A)]-'
=
(UTU)-'
+ O(A)
as y
+ 0.
(4.9.9) Exercise: Let C be an arbitrary symmetric matrix. Then (a) (C+A21)-' and (Z+A2C+)-' exist whenever A2 is suitably large or suitably small and so (b) [Z+A-2C]-1 = (I-CCf)+A2Cf(Z+A2C+)-' if A2 is suitably large or suitably small, and (c) (Z+A2C)-' = (I-CC+)+C+(C++A2Z)-' if A2 is suitably large or suitably small. (d) [Z+1-2C]-' = ( Z - C C + ) - C j " = , ( - A 2 ) j ( C + ) j + O ( A 2 n + 2as) E.+O. [Hint: Write C = TTDT where T is orthogonal and D is diagonal, and use (3.6).]
52
IV
PSEUDOINVERSES OF PARTITIONED MATRICES
(4.10) The Concept of Rank If 9 is a linear manifold in a Euclidean n-space, the dimension of 9 [abbreviated dim(Y)] is defined to be the maximum number of vectors in 2' that can be chosen linearly independent of one another. A fundamental fact that we shall take for granted is that any basis for 2 has exactly r linearly independent vectors, where r = d i m ( 9 ) .
c 9,, then (4.10. I ) Exercise: If 2, dim(P2-Y1) = dim(P2) - dim(9,). If A is any matrix, the rank o f A [abbreviated rk(A)] is defined to be the dimension of A's range:
rk(A) = dim[.*(A)]. Several properties of rank play a n important role in the theory of pseudoinverses for products and in the statistical applications of Chapter VI. They are straightforward consequences of already established results and are left as exercises :
(4.10.2) Exercise (a)
For any matrix A rk(A) = rk(ATA) = rk(AT)
=
rk(AAT).
(b) For any matrices A and B (of the right size) rk(AB) = rk(A+AB) = rk(ABB+).
(c) rk(AB) < min[rk(A),rk(B)] (d) If rk(A)
=
rk(AB) then & ( A )
=
% ( A B ) and the equation
ABX = A has a solution in X . (4.10.3) Exercise: (a) If P is a projection, r k ( P ) = trace(P). (b) IfP,, P, , andP, ar epr ojections wtthd( P, ) E 2'(Po),%'(P2) G %?(Po) and P , P, = 0 then
P,
=
P,
+ P,
if and o n l y if rk(P,) = r k ( P , ) + u k ( P , ) (c.f. Cochran's theorem, ScheffC [I]). We now turn our attention to the question of pseudoinverses for products of matrixes. If A and B are nonsingular, ( A B ) - ' = B - ' A - ' , but it is not
PSEUDOINVERSE OF PRODUCTS
generally true that (AB)'
= B+A+
53
as evidenced by the example
Greville [ 3 ] has found necessary and sufficient conditions for (AB)' = B + A ' . These we state as the following. (4.11) Theorem: (AB)' = B + A + if and only if
(4.1 1.1)
9 ( B B T A T )E ,@(AT)
and 9 ( A T A B )c 9 ( B ) .
(4.11.2)
Proof: Since A + A = ATAT', (4.11.1) holds if and only if the equation (4.11.3)
(3.13.1)
A + A B B T AT = BB'A'.
By the same token, (4.1 1.2) holds if and only if
BB A ~ A B= A'AB.
(4.1 1.4)
+
We will show that (4.11.3) and (4.11.4) are necessary and sufficient for (AB)' = B + A + : Suppose (4.11.3) and (4.11.4) hold. Multiply (4.11.3) on the left by B + and on the right by ( A B ) T f:
B [ A A ( B BT AT )]( B'A') +
+
+
=
B + A + ( A B ) [ ( A B ) +( A B ) I T
=
B+A+(AB)
while B +[BBTAT](AB)T+ =
B+BBTAT(AB)T+
=
(AB)T(AB)'+
=
(AB)+(AB)
so that if (4.11.3) holds
(4.11.5)
B + A + ( A B )= ( A B ) + ( A B ) .
By the same token, if both sides of (4.1 I .4) are premultiplied by (All)'+ and postmultiplied by A + , we find that
(4.1 1.6)
( A B )B + A = ( AB ) ( A B ) +
+
.
54
IV
PSEUDOINVERSES OF PARTITIONED MATRICES
The right-hand sides of (4.11.5) and (4.1 1.6) are symmetric and so B + A+ satisfies (3.9.1). We are done if we can show that B + A + satisfies (3.9.2) and (3.9.3) as well : If (4.11.5) is premultiplied on both sides by A B , we see that B + A + does indeed satisfy (3.9.2). (3.9.3) is a bit more subtle: Since B+A+
=
(B+BB+)(A+AA+)
=
( B B +T)(BTAT)(A+-'A+) +
(4.10.2) tells us that
r k ( B + A + ) < rk(BTAT)= rk(AB).
(4.11.7)
On the other hand, (4.10.2b) asserts that (4.1 1.8)
rk(AB) = rk[(AB)+(AB)] =
rk[(B+A+)(AB)]
< rk(B+A+).
(4.11.5) (4.10.2c)
I f (4.1 1.7) and (4.1 1.8) are combined, we find that (4.11.9)
rk(B+A+) = r k ( B + A + A B )
so that the equation (4.1 1.10)
B+A+ABX= B+Ai (4.10.2d)
has a solution in X . Premultiply (4.11.10) by A B and apply (4.11.6) to deduce (4.1 1.1 I )
( A B )( B + A + ) ( A l l )X
=
( A B ) ( A B ) + ( A B ) X= (AB)X
=
(AB)(B+A+).
Substituting the last expression for ABX in (4.11.10), we see that ( B + A + ) ( A B ) ( B + A += ) B+A+ which establishes (3.9.3) and shows that B + A + = (AB)' if (4.11.3) and (4.1 1.4) hold. To prove the converse, suppose that (AB)' = B + A f : Then (AB)T
=
[(AB)(AB)+(AB)]T
=
( A B ) +(AB)(AB)T
=
B+A+(AB)(AB)T.
PSEUDOINVERSE OF PRODUCTS
55
If the left and right sides are premultiplied by ABBTB and use is made of the identity BT = (BB+B)T =
B=BB+
we find that ABB'B(AB)= = A B B ~ B B + A + ( A B ) ( A B ) = or equivalently A B B ~ ( ZA - + A )B B ~ = A 0. ~
(4.11.12)
Since H T H = 0 implies H
= 0,
(4.1 1.2) tells u s that
[take H
(Z-A+A)BBTAT = 0
= (I-A+A)BBTAT]
which is the same as (4. I 1.3). Equation (4.11.4) is proved the same way, interchanging AT and B throughout in the preceding proof. 1 A general representation for (AB)' was derived by Cline [ l ] .
(4.12) Theorem: (AB)'
= B,+A1+
where
B, = A+AB
(4.12.1)
and (4.12.2)
A , = ABIBl+
Proofi
Clearly
(4.12.3)
A B = A , B,.
Furthermore,
B,B,+
(4.12.4)
=
A+A1
since A + A , = A + ( A B , B , + ) = ( A + A ) ( A + A ) B B ~= + A+ABB~+
Similarly (4.12.5)
A 1 + A 1 = A'A,
because ( A , + A , ) ( B ~ B , +=) A , + ( A B , B , + ) B ,B ~ = + A , + ( A B , B , + )= A ~ + A , from which follows (4.12.5) after transposes are taken and note is made of (4.12.4).
56
PSEUDOINVERSES OF PARTITIONED MATRICES
IV
It is now easy to show that A , and B , satisfy (4.1 1.3) and (4.1 1.4): A l + 4 Bl(A, BJT = ( A + A I ) B l ( A I
WT
= A + ( A B ) ( A 1B,)T =
Bl ( A , BAT
(4.12.5) (4.12.3) (4.12.1)
so (4.1 1.3) holds.
Similarly,
B,B,+AIT(AI B , )
=
(A+A,)AIT(A1 B,)
(4.12.4)
=
( A 1 + A , ) ( A 1') ( A 1 B,)
(4.12.5)
=
AITAl B,
which proves (4.11.4), so that ( A , B1)+ = B 1 + A 1 + . The desired conclusion follows from (4.12.3). (4.13) Exercise: If A is n x r a n d B is r x m then
(AB)' if rk(A)
=
rk(B) = r.
(4.14) Exercise: (AB)'
(a)
AT^
(b) BB' (c)
= BfA+
=
z or
=
I or
=
B + A + if
B = AT or
(d) B = A + . (4.15) Exercise: If H is rectangular and S is symmetric and nonsingular,
(SN)'
=
H+S-'[/-(QS-')+(QS-')]
where
p
=
(l-HH+).
Chapter V
COMPUTATIONAL METHODS
In recent years, a sizable literature relating to the computation of pseudoinverses has accompanied the rebirth of interest in the theory. I n this chapter, we will describe four distinct approaches to the problem. The first method is based upon the Gramm-Schmidt orthogonalization (hereafter abbreviated GSO), the second is a modification of the “old faithful,” Gauss-Jordan elimination, the third is based upon the ideas of gradient projection and the last is a n exotic procedure derived from the Cayley-Hamilton theorem. (5.1) Method I
(GSO, Rust, Burrus, and Schneeburger [l])
Let A be a n n x m matrix of rank k < min(n, m). It is always possible to rearrange the columns of A so that the first k columns are linearly independent while the remaining columns are linear combinations of the first k. This is the same as saying that for some permutation matrix, P (a square matrix of zeros and ones with exactly one nonzero entry in each row and
column) (5.1.1)
AP = (R/ 5’)
where R is n x k and has rank k and the columns of S are linear combinations 57
58
V
COMPUTATIONAL METHODS
of the columns of R : (5.1.2)
S
=
for some
RU
(3.13.2)
U.
P is an orthogonal matrix so that
A
=
(Rj R U ) P T ,
A+
=
P[R(ZI U ) ] ' .
and (4.14)
The rank of (Zi U ) is the same as the rank of ( I ! U)(Z i U)' = I + U U T , (4.10.2a) which is k . Therefore the rows of (Z!U ) are linearly independent so that
[R(Zj U)]'
= (Zi
(4.13)
U)+R+
(3.5.2)
= ( I ; U)T(Z+UU')-'R+,
hence (5.1.3)
A+
=
P(Zi U)'(Z+ U U T ) - ' R + .
The last equation is the starting point for the computational procedure based on GSO; GSO is used to evaluate P, R + , U , and ( I + LIUT)-' : (a) Evaluation of P Perform a GSO on the columns of A , but do not normalize: i.e., denote the columns of A by a , , a,, ..., a,,,, let cl* = a ,
where Sj
=
{i: i< j - 1 and ci* # O}.
The vectors cj* are mutually orthogonal and 9 ( c l * , c2*, ..., ci*) = 9 ( a l , a , ,..., ai)
for each
i.
(2.8.1)
If the vectors cI*,c2*,... ,c,* are permuted so that the nonzero vectors (of which there will be k ) come first, the same permutation matrix applied to the vectors a,,a,, ..., am will rearrange them so that the first k are linearly independent, while the last m - k are linear combinations of the first k , since cj* = 0 if and only if aj is a linear combination of the preceding a's. So, if P
is any matrix for which
(5.1.4)
(c1*jc2*j ... jc,*)P
=
(cl ;c2j
... ! c,,,)
GRAMM-SCHMIDT METHOD OF RUST, BURRUS, AND SCHNEEBURGER
59
where Ilcjll
j = 1,2, ..., k
>0 =
j =k+l,
0
...,m
then
AP
(5.1.5)
= (a,ja,j-..ja,)P
= (R/S)
where R is n x k of rank k and the columns of S are linear combinations of the columns of R. (b) Computation of R+ The (nonzero) vectors, cl, c2,..., ck defined above, represent a GSO of the columns of R. If we let
(5.1.6) then
g(Q> = g(R) so by (3.13.2) there is a k x k matrix B such that
(5.1.7)
RB
=
Q.
'
Indeed since R has rank k, B = (RTR)- R'Q. We will derive an algorithm for B in (5.1.16). Since the columns of Q are orthonormal, QTQ = Z so that B is nonsingular (Q'RB = Z), and hence
(5.1.8)
R
=
QB-'.
Exercise (4.14) applies again and we find that
(5.1.9)
R+
=
BQ'
=
B(QTQ)'QT
=
BQ'.
It remains to evaluate B, U, and ( I + U U T ) - ' . (c) Computation of B and U Denote the columns of R by r l , r 2 ,..., rk and the columns of S by k. The vectors (el, c 2 , ..., ck,ck+ ',..., c,) defined in (5.1.4) represent a nonnormalized GSO of ( r l ,r 2 , .. ., rk,sl,..., s,-~). Indeed
sl, s2,...,s,-
c1 = r l
(5.1.10)
c j = r j i-- lx rj mTci c i i= 1
j=2,
...,k
60
v
COMPUTATIONAL METHODS
and
From (5.1.10) it is easy to deduce (by induction on j ) that (5.1.12) where
(5.1.13)
y EJ. . =
i>j i=j
I -2
y(r;Ca) y i a
IIca /I
i <j .
On the other hand, (5.1.11) shows that (5.1.14)
k
s. =
C m..r.
i= 1
IJ
I
where oijis obtained by substituting (5.1.12) into (5.1.1 1):
(5.1.1 5) From (5.1.12) and (5.1.6), we see that
where B is the k x k matrix whose (i,j)th entry is (5.1.16)
Pij
= Yijlllcjll
while from (5.1.14) we conclude that (5.1.17)
S = RU
where U is the k x m - k matrix whose (i,j)th entry is w i j .
GRAMM-SCHMIDT METHOD OF RUST, BURRUS, AND SCHNEEBURGER
61
Notice that (5.1.13) defines yij in terms of y i j - l , y i j - 2 , . . . , y i i . Computations are conveniently carried out in the following order : 711;
722,712;
733,723,713;
?44?Y34,Y24,714;
etc.
( d ) Evaluation of (It UUT)This inversion is achieved via one more GSO: (5.1.18) Theorem: If U is a k x r matrix and a GSO is performed on the columns of
the resulting matrix of orthonormal vectors is
where and
z - v, VIT = (Z+ UUT)-'.
(y)
(7-) (y)
(y)T(y)
Comment: The rank of is r since has the same rank as = I+ U T U (r x r ) which is nonsingular, (2.13), and hence has rank r. Therefore, will generate only nonzero the GSO, when performed on the columns of vectors.
Proof of theorem: Let H = (F). H has rank r [so that H'H = Z, (3.5.3)] and g ( V ) = 9 ( H ) , so the equation
HZ = v has a solution 2 = H'V.
(3.13.2)
Since the columns of V are orthonormal, VT V = Z, hence V + = (VTV)' VT = VT. Since 9 ( V ) = B ( H ) ,
H H + = YV' = VVT. Therefore
zzT= H + ( V V T ) H f T = (HTH)' =
(HTH)-'
since H has rank r.
(4.10.2a) and (3.5.1)
v
62
COMPUTATIONAL METHODS
Since
we have
uz = v,
and
z = vz.
Therefore V, V2T = ZZT = ( H T H ) - '
=
(I+ UTU)-'
The second part of the theorem follows from the identity ( I + UlJT)-'
=
I - U(uTu+I)-1u~
and the fact that
v, = uz = uv,. Therefore ( I + UUT)-
=
I - U(V, V 2 T ) U T = I - v,V , T .
Summary of Method I
To find the pseudoinverse of the n x m matrix A : I . Perform a GSO on the columns of A . Do not normalize. Call this set of vectors (cl * j ... j c,*). 2. Permute the cj*'s so that ( c , ! c2 j ... ! c,,) = (cl* ! c2* ! - - - i c,*)P where P is a permutation matrix chosen so that
cj # 0
j = 1,2,..., k
cj
j = k+l, ...,m.
=
0
3. Compute yij for j = i + l , ..., k ; i = 1 ,..., k , according t o (5.1.13) ( y i i = 1 , yij = 0 if i < j ) . ~ ~the , k xm-k 4. B is the k x k matrix whose (i,j)th entry is y i j / ~ ~ cUj is matrix whose (i,.j)th entry is wij [given by (5.1.15)], (I+ U U T ) - ' is obtained and by performing a (normalized) GSO on the columns of
(7.)
5.
A + = P ( I i U ) T ( / +U U T ) - ' B Q T .
GRAMM-SCHMIDT METHOD OF RUST, BURRUS, AND SCHNEEBURGER
Comment: A
FORTRAN listing for
this procedure is given in Rust et al. [l].
(5.1.19) Example
.-(; ; -; ; j. 1
0
1
1
1
k=2,
P=Z,
R = ( :
:), 0
1
S=(-i
711
= 1
Y22
= 1
Y12
0 1 1
= 1
012
= 1
021
= -1
022
= 0.
811
= I/&
822
=
I- V,V,T
214
=-
5
63
812
[ ] 1 3
=
-3
= -1IJ6.
=
(I+ UU=)-'.
1
1.
64
V
COMPUTATIONAL METHODS
(5.1.20) Exercise Alternate method .for inverting ( I + U U T ) : Let the columns of U be denoted by u I ,u2,..., u,. Let
w, = I Then for each n,
(
w,= If hence [Hint:
CUjU;
)
-I
w,= ( z + u u T ) - ' . Use (4.6.5) or prove directly by induction on n.]
(5.1.21) Exercise Alternate method for (Tewarson [2]>
the pseudoinversion of
(a) If P is square and P T ( I j U ) = STwhere S'S (b) If the GSO is applied to the columns of
= I,
(Zi U )
then ( I : U)' = S P T .
k
and the resulting set of orthonormal vectors are denoted by s1,s2,..., sk, then there is a k x k matrix P such that
(c) Let S
= (sIjs2 i ... isk)be partitioned: k
Then
P
=
S,
and
(Zj U)' = SSIT.
GAUSS-JORDAN
ELIMINATION METHOD OF BEN-ISRAEL, WERSAN, AND NOBLE
65
(5.2) Method I1 (Based on Gauss-Jordan elimination, Ben-Israel and Wersan [l], Noble El])
If A is an n x m matrix of rank k , there always exists a nonsingular matrix E and an orthogonal matrix P such that k m-k
(5.2.1)
E A ~ A P=
Indeed, there are many such E and P . For instance, if P diagonalizes A T A in such a way that the nonzero eigenvalues of ATA appear in the upper lefthand corner: k m-k k D' 0 P ~ A ~ A= Pm - k (-;-(j 0
then
-)
will do the trick:
which is a stronger version of (5.2.1). However, the method to be described here only requires that Eand P be chosen to satisfy (5.2.1). The present method is based upon the following identity: (5.2.2) Theorem: satisfy (5.2. I), then
If E is nonsingular and P is orthogonal, chosen to A+
=
P ( E A T A P ) +EAT.
Proof: The equation ATAX = AT
always has a solution in X , (3.13.1), (3.13.2), hence the equation
(5.2.3)
ATAPY = AT
always has a solution in Y since P is nonsingular, hence
(5.2.4)
E A ~ A P Y= E A ~
always has a solution in Y. Among all solutions to (5.2.3), the unique matrix which minimizes tr(YTY) is (5.2.5)
P = ( E A ~ A P ) E+ A ~ .
(3.13.8)
66
v
COMPUTATIONAL METHODS
The set of matrices Y satisfying (5.2.4) is the same as the set of matrices satisfying (5.2.3) since E is nonsingular, so P also minimizes
( P is orthogonal)
tr(YTY) = ~ I - [ ( P Y ) ~ ( P Y ) ] over that class. By (3.13.8),
PP
(5.2.6)
=
( A T A ) + A T= A S
and the theorem follows when (5.2.6) and (5.2.5) are combined.
(5.2.7) Corollary: If E is nonsingular and P is orthogonal and E A T A P = ($-) then A+
=
P ( H + iO)(EAT).
It remains to show how to compute EAT, H t , and P. (a) Computation of H , EAT, and P
m
n
Write down the augmented matrix, m ( A T Aj AT), and perform row operations on this matrix until the left block is reduced to the point where a permutation of its columns will result in the row echelon form: m
n
nr(ATA !AT)
-row operations only (step 1)
permute first m columns (step 2)
n
m(EATA j EAT) n
~ E A ~ AE PA I~ ) '
k m-k
The elementary row operations are reversible so E is nonsingular, as required. The permutation matrix P is orthogonal and is recorded as the columns of EATA are permuted to achieve row echelon form. The matrix EAT occupies the right-hand block of the augmented matrix just after Step 1. The matrix H is k x ni and is given by ( I [ L ) after Step 2. (b) Computation of H t
Evaluate (Z+LLT)-' either by (5.1.18) or (5.1.20), or evaluate (ZiL)' by (5.1.21).
GAUSS-JORDAN ELIMINATIONMETHOD OF BEN-ISRAEL, WERSAN, AND NOBLE
(5.2.8) Example
--I
0
1
-1
I
0 -1 1
0 - 1
A =
1 -1
0
-2
4
( P Aj AT)
=
-2
4
-2
@ -8
-2
-2
4
10
-2
-8
10
28
1 -2
:-1
-1
0
:
0
1
I
1
0
I
-2
-3
-1
0
’
3
0
1 - 1 1
2
67
-1
-1
2
1
0
1
1 -1
0
1 -1
0
-1
3
1
-2
-3
Pivot on the 4th element of the first row (which means “reduce the 4th element of all the other rows to zero by subtracting an appropriate multiple of the first row from each”):
-2
4
-18 4
-2
12
18 -12 -36
54
:
-2
@
0
-6
0
-18
0
-1
-1
5
4 -4
-5
:-12
-15
I
0
1
0
-1
-5
1
-3
-4
5
4
15
12
1 -1 3
1
Next, pivot on the third element of the second row:
0 - 2 : 3
-2
2
-18
12
6
0
0
0
0
0
0
0
0
0
I
3 - 3
3 -f
3
-1
1 -5
4
4
5
:
o
0
0
0
0
0
:
o
0
0
0
0
0
I
Next, divide the first row by -2, the second by 6:
1 -3
0 0
-1
:-g
0
1
2
1
0
:
8
0
0
0
:
o
0
0
0
0
0
0
:
o
0
0
0
0
0
0
= (EATA i EAT).
0
I
I
68
v
COMPUTATIONAL METHODS
This completes Step 1 ; EAT is the right-hand 4 x 6 block. The left-hand block is reduced to row-echelon form by postmultiplication by
0 0 1 0 p=[ 0 O 1 O 0 01 1 1 0 0 0
which puts column 4 into column 1, column 3 into column 2, column 1 into column 3, and column 2 into column 4. The result:
The matrix L is the upper right-hand block:
-1s
P(H+ ~OIEA= T 102
8 7
-6
-18
3
13 -5
5 -3
-3
18
5 -13
2
-2
9
-9
15 -8
- 5 -7 3 -6
=
A'.
GRADIENT PROJECTION METHOD OF PYLE
69
(5.2.9) Exercise: Let E, A , and P be as defined in (5.2.1). Show that the last m - k rows of EAT are always zero. [Hint: XATA = 0 if and only if XAT = 0. Specialize to matrices of the form k in-k
x = n(Oi
Yl ).]
Comment: A GATE 20 program for this procedure is given in Ben-Israel and Ijiri [l]. Additional refinements are contained in Tewarson [l].
(5.3) Method I11 (Based on gradient projection method, Pyle [l]) If A is an n x m matrix and b E 2 ( A ) , the equation
(5.3.1)
AX = b
has at least one solution and
i = A+b
(5.3.2)
is the only solution which lies in 9 ( A T ) ,(3.lb). Let the columns of AT be denoted by a,, a,, .. ., a, and denote by Ak (resp. bk) the k x m matrix (resp. k-vector) obtained by deleting the last n- k rows (resp. components) from A (resp. 6). Since (5.3.1) has a solution, so does
(5.3.3)
AkXk =
bk
k
=
1,2,..., n,
since the set of simultaneous equations which (5.3.3) represents is a subset of those represented by (5.3.1). Furthermore, Ak = Ak+bk
(5.3.4)
is the only solution to (5.3.3) which lies in 9 ( A k T )= 9 ( a l ,..., ak),(3.4, 3.lb). The present procedure develops a simple recursion which relates .Qk+ to B k . Since 2 = A +b is the same as t,, the recursion can be carried out n times and the desired solution to (5.3.1) results, provided b is in %?(A).Extensions to the general case follow readily as we will show: By definition, i l is the unique vector in 9 ( a l ) satisfying alTXl = PI
where Pj is the j t h component of b ( j = 1, .. .,n). Obviously
(5.3.5)
$1
=
@lT)+P1
= P1al(lla,112)+.
If 2 k - I is known, we construct i k as follows: h,) be the nonnormalized set of vectors which result from a Let (Al, h,,
...,
70
v
COMPUTATIONAL METHODS
Gramm-Schmidt orthogonalization of ( a l , . .,a,): (5.3.6)
hi = hk
where
= ak -
jtSx
(akThj)hj/llhj1/2
Sk = { j : j< k - 1 and
llhjii # 0}
k
=
1, ..., n.
We have shown that h, IL ? ( U ~ , . . . , ~ ~ -(2.8), ~ ) , and since 2 k - 1 Satisfies (5.3.7)
Ak-ltk-1
=
bk-1
it follows that (5.3.8)
= pj
ajT(2k-1+rkkk)
j = 1,23...,k-1
for any scalar xk. [Just write (5.3.7) in componentwise form.] In particular, if
(5.3.9)
8,
=
[ (pk
if
0
hk
=
0
otherwise
- aKTtk_ ,)/(hkTak)
then (5.3.10)
9 k
=
tk-1
+
Bkhk
satisfies (5.3.1 I )
aj’~= Pj
j E
&+
t
and (5.3.12)
Pk
y(hl>...>hk).
By construct i on , y ( / ? j ; j E s k + l )
=
y(hl,...,hk)
= 9( a l , . ..,a k ) = 2 (aj ;j E
Ask+ 1 )
sirice hj = 0 if and only if uj is a linear combination of its predecessors, (2.8.6). Therefore, 9, is the unique vector in Y ( a j ;j E sk+,) satisfying (5.3.1 1). O n the other hand,
P,
=
A , + ~ , E ~ ( U..., , ,a,) = 2 ’ ( a j ; j E S k + l )
satisfies (5.3.13)
ajTx =
Pj
j = 1, ..., k
andhenceit akosatisfies(5.3.1 I ) . S i n c e ~ , i s t h e u n i q u e v e c t o r i n ~ ( a j ; j&+ E ,) satisfying (5.3.1 I), Pk = i k .
GRADIENT PROJECTION METHOD OF PYLE
In summary:
(5.3.14) Theorem: If AT = (a,/a2j...ja,) and
b=
then
A+b
=
2,
71
L:1 42
E~?(A)
where 2, = 0
2,
=
8,
=
k
2k-1 +Bkhk
[
=
1,2,..., n
0
if hk = 0
(/3k-akTgk- l)/(hkTak)
otherwise
and ( h , , h , , ..., h,,} are the nonnormalized vectors resulting from a GSO of
{ a1
9
a2 2
.. ., 0,).
In general, if b # W(A) [or if you are not able to decide whether b E 9 ( A ) or not, conveniently] resort to the following trick: Let d = ATb and c=A~A. Then d E 9 ( A T ) = 9 ( C ) so that (5.3.14) can be used to compute C'd. Since C'd = ( A T A ) + A T b ,this additional step generalizes the procedure to the case of arbitrary b. If, instead of A'b, it is desired to compute A', proceed in the following manner: First, perform a GSO on the columns of A . Denote the normalized set of nonzero (orthonormal) vectors by d,, d,, .. ., d,. These vectors span the same linear manifold that the columns of A span, namely %?(A). By (3.7.3), it therefore follows that
AA'
=
1 did:.
j= 1
Denote the columns of AA' by bl,bz, ..., b,. Since 9 ( A ) = W ( A A f ) = 2 ( b l , b z ,...,b,), it must be that bj E W(A) for e a c h j so that (5.3.14) can be used n times to compute A + b j ( j = 1 , ...,n). Since
( A + b , i A + b 2 / * * - / A + b=, )A'(b1ib2i-*-/b,)= A + ( A A + ) = A + , this procedure generates the desired result.
72
V
COMPUTATIONAL METHODS
Summary of Method I11 1. If it is known that b E B(A) then (5.3.14) can be used directly t o construct A + b . 2. If it is not known whether or not b E %?(A),computed = ATb, C = ATA and use (5.3.14) to compute C'd which coincides with A'b. 3. To compute A + , first perform a GSO on the columns of A . Denote the resulting orthonormal vectors by d,, d2,..., d r . Construct the matrix El=, d,d,T and denote the columns of this matrix by b , , b 2,..., 6,. Then for each j , b, E & ( A ) and (5.3.14) can be used t o construct A'b,, j = 1, ..., n.
A
+
=
( A +b, / A +b2 j .. . j A +b").
(5.3.15) Exercise: In (5.3.14) we defined 8, =
Show that hkTa, # 0 if
[
0
if h,
(Pk
hk
- akT?k- ,)/(h,Ta,)
=
0
otherwise.
# 0.
(5.3.16) Example: Compute A + where A is given by (5.1.19):
The columns of A are
The GSO generates the orthonormal set:
A A + = d,dIT+dzdzT =3
(
2 -1
1.
GRADIENT PROJECTION METHOD OF PYLE
The columns of A A + are
6, =
(1
b2=(-!
3
3
j
73
b 3 = ( i j .
To construct A t b j , we perform a GSO on the columns of A T : The columns of AT are
The nonnormalized GSO produces
To compute A ' b , , use (5.3.14):
6,
A
u2
=
=
2 JS 9
-JE 45
To compute A + b , :
J?
8, =-
9
74
V
COMPUTATIONAL METHODS
A
c(2
Jis
=-
9
To compute A'b,:
Thus
which agrees with (5.1.19). (5.4) Method IV (Based on the Cayley-Hamilton theorem, Decell [2], BenIsrael and Charnes [l]) The present method is based upon two theorems. The first uses the classical Cayley-Hamilton relation (which says that any square matrix satisfies its own characteristic equation) to deduce an expression for the generalized inverse of a matrix in terms of its characteristic polynomial. The second theorem is a result of Fadeev and Fadeeva [I], which generates the coefficients of said characteristic polynomial in an efficient manner.
(5.4.1) Theorem: (a) Let A be an n x n symmetric matrix and let n(A) = det ( A - ) . I ) , which can always be factored as follows: X(A) =
where n - k is the rank of A .
aAk(l-Aq(A))
CAYLEY-HAMILTON METHOD OF DECELL, BEN-ISRAEL, A N D CHARNES
75
Then if A is nonsingular, A-' = q ( A ) and generally A + = P(A)
+ 40(O)CA4o(A)-Il.
(b) If H is an arbitrary m x n matrix and if the characteristic polynomial of H T H is given by aAk(1 - Acp (A)), then
H + = q(HTH)HT. Comment: The matrix cp(A) is obtained by raising A to the appropriate powers and summing these powers using the coefficients of q(-)as weights.
Proof: (a) The Cayley-Hamilton theorem (cf., Bellman [l]) states that A satisfies the matrix equation z ( A ) = 0. Therefore aAk(Z-Aq(A)) = 0. If A is nonsingular, k = 0 and hence Aq(A)
=
so that
I
q(A) = A-'.
Generally, Ak = A k + ' q ( A ) .
Since (A+)k+'Ak = A+(A+kAk) =
A+(A+A)
(3.1 1.3)
=
A+(AA+)
(3.11.6)
= A+
(3.9.3)
it follows that A+
=
( A +)k + l A k + l q ( A ) .
Again, using (3.11.3) and (3.11.6), we see that (5.4.1.1)
A + = AA+cp(A).
Since q ( A ) - q ( 0 ) involves only positive powers of A and since AA+A' = A'
if a > O
(3.1 1.3)
76
v
COMPUTATIONAL METHODS
it follows from (5.4.1.1) that
A+ (5.4.1.2)
+
= AA+ [ q ( ~ ) - c p ( ~ ) ]A A + ~ ~ ( o )
= c p ( 4 - Cp(0)
and so AA'
= Aq(A)
+ AA+cp(O)
+ A(I-AA+)cp(O).
Since A A + = A + A for symmetric A , (3.1 1.6), the second term vanishes and (5.4.1.3)
AA+ = Aq(A).
Since ~ ( 0 is) a multiple of the identity, it commutes with A q ( A ) and so, if (5.4.1.3) is inserted into the right side of (5.4.1.2) we find
+ (Acp(A)-I)cp(O) = v ( A ) + cP(O)(Acp(A)-I).
A+ = c p ( 4
(b) From ( 4 ,
(HTH)+ =
'p(HTH)
+ cp(O)[HTHq(HTH)-I].
From (5.4.1.3) and (3.11.7)
HTHp(HTH) = H'H so that
H+
=
+ = q ( H T H ) H T+ q ( 0 ) ( H + H - I ) H T
( H T H ) + H T= c p ( H T H ) H T q(0)[HTHq(HTH)-I]H1 =
(since H + H H ~ = H').
'p(HTH)HT
Superficially, the last result appears to be of theoretical interest only, owing to the fact that q ( A ) is defined in terms of the characteristic polynomial of A which, in turn, is defined i n terms of a determinant which requires a prohibitively large amount of computation for large matrices. The following result of Fadeev and Fadeeva [ l ] reduces that problem t o manageable proportions: (5.4.2) Theorem:
(5.4.2.1)
I
I
I
Let A be an n x n symmetric matrix and define A, = A yk =
trAk/k
Bk = A, =
ABk
-
k
(tr& is the sum of Ak's diagonal elements.)
=
1,2)...)n - I .
CAYLEY-HAMILTON METHOD OF DECELL, BEN-ISRAEL, A N D CHARNES
Let (5.4.2.2) M
=
the first value of k
n
77
< n for which ABk = 0 if such exists,
otherwise,
and let (5.4.2.3)
r be the largest k
< M for which y k # 0.
Then, (5.4.2.4)
~(1.)= det(A-21)
= (-l>"y,Afl-r(l
-cp(A))
where (5.4.2.5) Proof: The proof consists of two parts: First we show that coefficient of
Yk
is the
(-lY+lp-k in the expansion of det(A-1.1) for k = 1, ...,n. Then we use that result to show that y k = 0 fork 3 r + 1 . The desired conclusion follows easily thereafter.
Proof that (5.4.2.6)
det(A-AI)
(-1)"
=
Here, y, = tr A which is the coefficient of ( - 1)"" A"-' in the expansion of det ( A - 1.1). We proceed by induction, supposing that y l , y 2 , . . .,y k - are, respectively, the coefficients of( - I)"" A"- ,( - 1y+'1 Y 2 , ...) ( - 1 ) n + l l 2 n - k + l in the expansion of det (A - 2.1). Define '
uk = tr(Ah).
(5.4.2.7) From (5.4.2. I ) ,
Ak+ 1
=
- YkA
AAk
which can be iterated backward to obtain (5.4.2.8)
k- 1
A k -- Ak - . C - y j A k - j
k = 1,2, ..., n
j=I
and hence from (5.4.2.7) (5.4.2.9)
fr(Ak)
k- I
=
(Tk
-
1Yj'Tk-j.
j =I
78
COMPUTATIONAL METHODS
V
It is well known that for any A , tr(A) is the sum of A’s eigenvalues, and if L,,i,,,..., ;I-,, are the eigenvalues of the symmetric matrix A , then Alk ,...,1 ; are the eigenvalues of A k , so that
c s”. n
=
0,
j =I
Furthermore, Newton’s formula states that for any polynomial,
1” -
c n
pjA”-j
j = 1
having roots at
&, ..., I,,,, the relations k- I
(5.4.2.10) hold for k
=
kpk
=
sk-
j= 1
pjsk-j
1,2, ...,n, where k
s, =
1 A;
(BBcher [l]).
j= I
Writing det(A-3.1)
=
(-1)”
j= I
we see that under the induction hypothesis, by (5.4.2.10),
pi
1
=yj, ( j=
1,2, ..., k - 1) so
(5.4.2.11) By (5.4.2.9) and (5.4.2.1)
kpk = tr(Ak) = ky, which shows that y k = p k and hence establishes the induction hypothesis for all k . The recursion (5.4.2.1) guarantees that A, = 0, Bk = 0, and Y k = 0 for all k > M if A B , = 0 for some M < n. The definition of r therefore implies that yk = Y for k > r . From this and (5.4.2.6), det(A-AZ)
=
(-1)”
r
j= 1
This proves (5.4.2.4) and (5.4.2.5).
1
CAYLEY-HAMILTON METHOD OF DECELL, BEN-ISRAEL, A N D CHARNES
79
The application of (5.4.1) and (5.4.2) to the pseudoinversion of an n x m matrix is particularly amusing: (5.4.3) Theorem: Let A = H T H (where H is an arbitrary m x n matrix), and define A,, B,, Y k , M , and r as in (5.4.2). Then
H+
(5.4.3.1)
= (yr)-l
B r - l H"
and the rank of H is r.
Proof: The rank of H is the same as the rank of A. The rank of A is n - (the multiplicity of the characteristic root at 2. = 0)
since A is symmetric. By (5.4.2.4) this multiplicity is n-r, which proves that the rank of H and A is r.
Summary of Method IV To find H + where H is an m x n matrix:
1 . Let A , = H ~ H , 2. 7, = tr (AJk), Bk = A, - Y k I , A, + 1 M
=
1
first value of k for which AB, less than n
n
H+
=0
if such exists and is
otherwise,
r = largest k 3.
= AB,,
< M for which yk # 0,
=(Y~)-'B~ H-T. .~
Comment: Although it is true that A - ' = q ( A ) for nonsingular symmetric A , it is not generally true that A + = q ( A ) as evidenced by the example A = diag(0, O,+, 4).
80
v
COMPUTATIONAL METHODS
For this example q(i) = 5
61.
-
so that q ( A ) = diag(5,5,2,3)
whereas
+
A + = (0,0,2,3) = v(A) q ( O ) [ A q ( A ) - I ] .
(5.4.4) Example
y , = 21
(-: -6
B, = A , - 2 1 1 =
6
-18 0
yz = -54
B2=A2+541=9(-2 1 1
M = r = 2
- 24 -2
-3 0 -18
)
-:1 1
CAYLEY-HAMILTON METHOD OF DECELL, BEN-ISRAEL, AND CHARNES
81
(5.4.5) Exercise: Prove the Cayley-Hamilton theorem for symmetric matrices. (Hint: Use the diagonalization theorem.) (5.4.6) Exercise: Establish the result analogous to (5.4.3), using the characteristic polynomial for H H T instead of H T H .
(5.4.1) General comments In Methods I-IV, the option always exists between working with A or A’. For example, in Method 1, A + can be found by performing a GSO on the columns of A or one can compute (A’)’ by performing a GSO on the columns of AT. Similarly, ( I f U U T ) - ’ can be evaluated by performing a GSO on the columns of or on the columns of (-“,’). The proper choice is governed mainly by the size of the matrices involved. All four methods depend crucially upon the computational determination of the rank of the matrix being pseudoinverted. In some of these cases, this is done during the course of a GSO, at that point where one decides whether a vector which is numerically close to zero, is actually zero. In Method 11, a decision must be made as to whether the Gauss-Jordan elimination has reduced a row to zero or not. If an error (usually due to roundoff) is made at such stages, like as not, the error will be propagated through the remainder of the procedure in a discontinuous manner and cause large errors in the final result. A minor modification significantly improves the GSO in this regard. Whereas the “naive” GSO of (2.8) orthogonalizes the h-vectors in the order they are given (so that for eachj, 2 ( u , , ..., u j ) = 9 ( h l ,..., hj), the u’s being the orthogonal set induced by the h’s), the so called “modified GSO” (Osborne [I]) derives each u j + from that h-vector whose projection on 9 ’ ( u l , ..., uj) is longest. Thus, the borderline cases (is the present h-vector linearly independent of the previously used ones?) are put off until last. Additional algorithms for pseudoinversion are described in the papers by Pereyre and Rosen [II], Golub and Kahan [I], and Golub [ I , 23, all of which pay particular attention to the question of numerical stability. The recursions of (4.3) and (4.6) can be interpreted as computational algorithms for C,,,+ and (C,,,CmT)+but the comments above, apply quite strongly in these cases. Extreme care must be exercised in making the decision whether or not a vector c,,,+’ is a linear combination of its predecessors. We will return to these recursions in Chapters VIII and IX.
(7-)
Port
STATIST1CA L AP PLICAT1ONS
I/
Chapter VI
T H E GENERAL L I N E A R HYPOTHESIS
In this chapter, we will apply the results obtained so far to some “classical” statistical problems and thereby produce streamlined derivations (and generalizations) for many of the most important results in linear regression analysis. In particular, we will deal with the Gauss-Markov theorem, the distribution theory for best linear unbiased estimator residuals under the normality hypothesis, confidence ellipsoids for linear regression parameters under the normality assumption, the distribution of the likelihood ratio statistic for testing the general linear hypothesis and its relation to confidence ellipsoids, the relationship between Gauss-Markov estimates and “naive” least squares estimates, and the theory of orthogonal designs. In fairness to the reader, we admit that this chapter presupposes a familiarity with elementary statistical concepts (as exemplified by the usual introductory, but post calculus, course in statistics). Such notions as independence, expectation, normal distributions, tests of hypotheses, variance, etc. are used freely without much in the way of preliminaries. If things get too heavy, the reader can skip to Chapters VII and VIII which are essentially nonstatistical. Chapter IX is probabilistic in spirit. 85
86
VI
THE GENERAL LINEAR HYPOTHESIS
(6.1) Best Linear Unbiased Estimation; The Gauss-Markov Theorem
In many applications, observations are made on a random process, and it is felt that a reasonable "model" for the observations is of the form p
'(-r) =
L IXi epiCr) + a residual
;=1
where T j is the value ofthejth independent variable (e.g., the time at which the jth observation was made, or the concentration of a catalyst in some chemical experiment, or the amount of drug injected into the jth laboratory animal in a drug trial experiment); '(T) is the observation (or datum, or dependent rariable) corresponding to the level Tj; the ep/s are a specified family of functions and the ex's are parameters whose value is to be estimated from the data. (6.1.1) Example: The voltage drop across a legnth of wire is proportional to the current flowing through the wire, the constant of proportionality being the resistance of that wire. It is further known that the resistance of a wire (of uniform thickness) is proportional to its length. The constant of proportionality is the resistance per unit length of the wire and is to be measured as follows. A long length of wire is connected in series to a resistor and a voltage source. See Figure I. A voltmeter is used to measure the difference X
-<E(------ T - - - - - _ )
A
V \------------+-------,Mf\NJIN\N\/'----.
Wire
Resistor
Figure 1.
in voltage between the fixed point A and the variable point X. As X moves, the distance, T, between X and A changes. The voltage drop, «T), is recorded at various values of T (say T t, T2' ... ' Tn)' Since the resistance between A and X is a linear function of T, a sensible model for this data is (r) = prj
+ R + residual error
so long as X is to the left of the resistor (which is used to prevent "short circuiting" the voltage source). Here, p is the resistance per unit length, R is the resistance between A and the left end of the resistor. For this model, (Pt (r) = r, (P2(r) = I, ex l = p, and ex 2 = R. The residual errors are due to inaccuracy in measurement and in the physical model for the phenomenon being studied.
BEST LINEAR UNBIASED ESTIMATION; THE GAUSS-MARKOV THEOREM
87
(6.1.2) Example: The mean daily temperature at a given location fluctuates in a quasi-periodic manner with the period of the fluctuation corresponding to the change of seasons. If mean temperatures are recorded daily, then a reasonable model for the temperature ((T~) on the rjth day is
where the period 365.25 takes into account that a calendar year is slightly out of phase with a solar year. The number of harmonics ( k ) should be chosen reasonably small to keep the model simple. The u's and p's are not known and must be estimated from the data. The residual error now is attributable mainly to the nondeterministic nature of temperature variation rather than errors in measurement. Both examples are special cases of a so-called general linear model in which the vector of observations
(6.1.3) is assumed to take the form
(6.1.4)
Z =
Hxfv
where H is an n x p matrix whose j t h row is
(6.1.5)
hjT = ( C P ( T i~ ) , C P ~ ( T ~ .*.> ),
VpiTj))
x is thep-dimensional vector of unknown coefficients which is to be estimated from the data:
(6.1.6) and v is a vector of residuals which is assumed to be a vector random variable with mean value 0:
(6.1.7)
bv = 0.
Following Scheffk [I] we define a parametric function $ to be a scalarvalued linear function of the unknown vector parameter x:
*
= CTX
where c is a specified p-dimensional vector. (For instance, each component of x is a parametric function.)
88
VI
THE GENERAL LINEAR HYPOTHESIS
A parametric function II/ is said to be estirnabIe if it has a n unbiased linear estimate. (This means that there exists a vector of constant coefficients a, such that 8 2 2 =
*
no matter what the true value ofx.) The question ofestimability for parametric functions is easily decided : (6.1.8) Theorem: The parametric function $ = cTx is estimable if and only if H ’ H c = c [i.e., if and only if c E .%‘(HT)].
Proof: Since dnTz = aT8z= a T [ 8 H x + b v ] and since Bv = 0 a n d H x is nonrandom, we see that II/ = cTx is estimable if and only if there exists a vector a such that (6.1.8.1)
aTHx = cTx
for all x.
But (6.1.8.1) holds for all s if and only if there exists a n a such that H T a - c is orthogonal to all s and the last is true if and only if there is a vector a such that
HTa = c.
(6.1.8.2)
(2.9.4)
The conclusion now follows from (3.12~).
The rnuiance of an unbiased estimator for rl/ is a hallowed measure of its efficacy. Suppose that the residual covariance matrix is denoted by V 2 : (6.1.9)
GVVT =
v2
where V 2 is an n x IZ nonnegative definite matrix. [Since uTV2u= uT(&vvT)u= 2 0 for all u and is therefore nonnegative as aserted.] If = uTz is an unbiased estimator for $, its variance is
C;(vTir)’, it follows that u’V2u
4
(6.1 .10)
8($-II/)Z
= r;(aTv)(vTa) =
aTVZa.
The best linear unbiased estimator (BLUE) for $ is (defined to be) the one with minimum variance. It is a quick matter to exhibit a formula for the BLUE of a parametric function: let
(6.1.11) Theorem:
(6.1.11.1)
Let
It/ = cTx be an estimable parametric function and ii
=
(I-
v+
V)(H+)TC
where (6.1.1 1.2)
-
I/ =
v(z-HH+).
BEST LINEAR UNBIASED ESTIMATION; THE GAUSS-MARKOV THEOREM
89
Then
4
(a) = dTz is a BLUE for Ic/ and (b) if \Ir* is any other linear unbiased estimator for Ic/
mk*-$I2
unless
'a@-$>'
4= \Ir* with probability
1.
Proof: If is estimable, then aTz is unbiased if and only if a satisfies (6.1.8.2). By (3.12c), all such a's are of the form
(6.1.1 1.3)
a, =
H ~ +C
for some u.
(I-HH+)u
(We used the identity H H + = H T f H T .) The variance of the estimator \Ir, = a,'. is
- $)2
€(+,
= € (auTz- C'xy
= ba,T(z - Hx)2
[since a satisfies (6.1.8.2)] = €(a,'v)2 =
a;V2uU
=
//va,112
(V is the symmetric square root of V 2 . )Since IIVa,112 =
J I V H T f C - Vu112
we see that the variance of the linear unbiased estimator u is of the form
a
=
\Ir, is minimized if
V + VHT+C+ (Z- V + V )w
where u' is free to vary. Thus by (6.1.11.3) and (3.13.10), a, =
(z-V+V)(H+)~C-(z-HH+)(z-V+V)w
=d
-(z-HH+)(I-V+ V ) w
where w is free to vary. But for such an a, &[(a;-b)Tz]Z
= U,T(Z-
V +V ) ( ~ - H H + ) V ~ ( I - H H + ) ( Z - V + V ) w
= 11 V(1-
Vf V )w112
=o
(3.9.1)
so that ( U , - ~ ) ~ Z=
Thus, all BLUE'S for
0
with probability 1.
are of the form bTz with probability 1.
90
THE GENERAL LINEAR HYPOTHESIS
VI
Convnent: The results of (6.1.1 1) hold whether the residual covariance is singular or not. This result generalizes the Gauss-Markov theorem, which restricts its attention to nonsingular residual covariances (cf. Zyskind and Martin [I], Zyskind [I]).
Theorem (6. I . 1 1) yields the classical result as a special case: (6. I . 13) Theorem (Gauss-Markov): -
Let
v = v(I-HH+),
(6.1.12.1)
G
=
H+[I-(V+v)T]
P
=
Gz.
and (6.1.12.2) Then (a) 8%= H + H . u for all x . ( b ) If +b = c T y is an estimable parametric function, there is a unique B L U E for 4, namely cTP. (c)
If V 2 is nonsingular
s
=
(HTV-ZH)+HTV-2z.
P ~ o o f : (a) 6% = GH.Y = H + H x - H + V T ( V + ) ' H . ~ .
But (V+)TH =
and
-
VH
(b) The BLUE for
(VVT)+
=
VH
(3.8.1)
(6.11.2)
0.
+b is unique by (6.1.1 I), and is expressible in the form $ = aTz
where
ii = [ H + ( I - ( V + V ) T ) ] T C = Thus
4 = cTli
GTC.
as asserted.
(c) By (6.l.I2.1), G if V is nonsingular.
=
H + V I I - ( V + T V ) ]V - '
BEST LINEAR UNBIASED ESTIMATION; THE GAUSS-MARKOV THEOREM
91
By (3.13.10)
V+T = [ ( Z - H H + ) V + ] T so V+Tv= v + T P
hence G = H + V[Z-
P+VT] V - ' .
By (4.15) the last is exactly ( V - ' H ) + v-' since
-
where Q = I - H H +
VT=QV
so that
G
= ( V - ' H ) + V - ' = (HTV-'H)+HTV-'
(3.8.1)
and ft = GZ = ( H T V - 2 H ) f H T V - 2 ~ .
Comment:
When V 2 is nonsingular, then 2 = ( V - ' H ) + V - ' z minimizes
11 V - 'z- V - 'Hxll
= (z - H x ) ~ V - '(Z
- Hx)
which is a weighted sum of squares. In the special case where V = diag(v,, v2, ..., v,)
d minimizes n
1 =1
(Tj-hTX)Ivj2
cj,
which is the sum of squares of the deviations between each observation, and its expectation, each term being weighted according to that observation's accuracy (vj2 = variance of Cj). Accurate observations (small vj) are weighted more heavily than inaccurate ones. This interpretation is explored in greater depth in Chapter VII, where we examine the relationship between constrained least squares estimators and weighted least squares.
92
VI
THE GENERAL LINEAR HYPOTHESIS
(6.1.13) Counterexample: If V 2 is singular, the BLUE for cTx is not cT(HTV + 2H ) H' Y+' z : Take +
Componentwise,
5 + v1 = 5.
where v, has unit variance
= T2
The second observation gives an error-free look at the unknown parameter and the best estimator would be On the other hand,
c2.
(H'V+2H)+
=
1
H T V + 2 = (1 0)
so if c = 1,
C ~ ( H ~ V + ~ H ) + H '= V GI +~Z which is not correct. The correct result is obtained if (6.1.12.2) is used:
0 1
G
=
H + [ I - ( f 7 f V ) T ] = (0 1)
GZ =
c2.
(6.1.14) Exercise: When do least squares estimators coincide with GaussMarkov estimators? If z = H x + v where €vvT = V 2 , the naive least squares estimate is defined to be
f
=
H'z.
IfI+!I = cTxis an estimable parametric function, the naive least squares estimate for I+!I is defined to be c T f . (6.1.14.1) Show that the naive least squares estimate for every estimable parametric function is a BLUE if and only if
v+ v = ( V + V )
DISTRIBUTION FOR QUADRATIC FORMS IN NORMAL RANDOM VARIABLES
where
93
-
v = V(I--HH+).
(6.1.14.2) If V is nonsingular, the condition in (6.1.14.1) reduces to
(V)+ = ( I - H H + ) v - ’ . (6.1.14.3) In general, the condition (6.1.14.1) is equivalent to 9(V2H) E 9 ( H )
(i.e., the range of H is invariant under V 2 ) . Comment (Jocular):
Naive least squares estimates & are chosen so that
llz- Hx1I2 has “tiniest residual.” If cTxis estimable, cT&is unbiased. There-
fore, cT4is a TRUE (tiniest residual unbiased estimator) for cTx.(6.7.3) says that f is TRUE-BLUE if and only if 9 ( V 2 H ) E 9 ( H ) . Other conditions equivalent to (6.7) are given in Zyskind [l], Krusbal [l], Mitra et al. [l], and Watson [I]. For instance, it is shown by Zyskind that (6.7.1) is equivalent to 9 ( H ) = U ( y , , . . . , y r ) where the yj’s are a subset of V’s eigenvectors. Mitra et al. show that another necessary and sufficient condition is that V 2 should be of the form
V 2 = HSHT + ( I - H H ’ ) T(I-HH’)
+ kZ1
where S, Tare nonnegative definite and k is a scalar. (6.1.15) Exercise Generalized least squares (Price [1)) It has been shown that 9 = H + z is the unique vector which minimizes I(x(1 = xTx, among those which also minimize ( z - H x ) ~ (z Hx). Let V 2 and W 2 be nonnegative-definite matrices. Show that among those x’s which minimize ( Z - H X ) ~ V ~ ( Z - H Xone ), which minimizes IIX~/,,~- = x T W 2 x is 2 = [ I - (PW2P)+PW2](HTV2H)+HTVZZ
where
P
=
z - (VH)+(VH).
Are there others? (6.2) Distribution for Quadratic Forms in Normal Random Variables
In this section, we will investigate the distribution theory for certain types of quadratic forms in random variables which have normal distributions and
94
VI
THE GENERAL LINEAR HYPOTHESIS
apply these results to regression analysis (tests of the general linear hypothesis and construction of confidence ellipsoids). We first review some well-known facts about normal random variables (Scheffe' [l]; Anderson [l]).
...,c,,
(6.2.1) If G,,G2, have a joint normal distribution, they are mutually independent if and only if their covariance matrix is diagonal. (6.2.2) If k,, t2,..., 5, are independent and normally distributed with means f t l , p 2 , ...,p, and unit variances, then the distribution of
depends on the pj's only through
6
=
c p j =)
(j:,
and this distribution is a noncentral chi-squared distribution with n degrees of freedom and noncentrality parameter 6. If 6 = 0, the distribution is a chi-square distribution (central), with reference to centrality usually omitted. (6.2.3) If x,,' has a noncentral chi-square distribution with n degrees of freedom and noncentrality parameter 6 and if xm2 is independent of xf12 and has a (central) chi-square distribution with m degrees of freedom, then n - xn2/m- xm2 has a noncentral F distribution with n and m degrees of freedom and noncentrality parameter 6. Reference to centrality is usually omitted when 6 = 0. (6.2.4) The random vector z has a multivariate normal distribution with mean m and covariance R if and only if there exists a matrix H such that z = H x + m , where the components of x are independent normal (a) random variables with mean zero and unit variance, and R = HHT. (b)
If (-:-) has a zero mean multivariate normal distribution, then u is independent of v if and only if &UVT =
0.
If z is a vector random variable which has a multivariate normal distribution with mean m and nonsingular covariance R, the density function of z has the form const x exp - +(z-m)TR-'(z-m).
DISTRIBUTION FOR QUADRATIC FORMS IN NORMAL RANDOM VARIABLES
95
There is a well known proverb in statistical lore that asserts “the exponent of a normal distribution with nonsingular covariance has a chi-square distribution.” We will now prove a generalized version of this statement. (6.2.5) Theorem: If z is a vector random variable having a normal distribution with mean m and covariance R and if the rank of R is p , then (a) z T R + z has a noncentral chi-square distribution with p degrees of freedom (df) and noncentrality parameter
6
=
(mTR+m)”.
(b) (z-m)’R+(z-m) has a chi-square distribution with p df.
Proof: R is nonnegative definite since U’RU = ~ ( u ’ x ) ~0
where x
=
Z-m,
so R has a symmetric square root R”. Let m1 = (R”)’m and m2 = (I- R”R”+)m. Then m = R%m, + m , .
If y is a normally distributed random vector with mean zero and covariance I, then Z = R ” ( y + m l ) m,
+
has the same distribution as z, hence ZTR+Z has the same distribution as z T R + zand it will suffice to establish (a) for Z’R’L: ZTR+Z = ( y + m , ) T R % R f R % ( y + m l )
+ 2(y+m,)’R%R’m2
+ m,’R+m2.
Since R”R”+ = RR’ = R’R, it is easy to see that R’m, diagonalization theorem, it is easy to see that
= 0.
Using the
i= 1
where the ri’s are the orthonormal eigenvectors associated with the nonzero eigenvalues of R :
[cf. (3.6)]
RR+ = R + R =
C rjrjT. P
j= 1
96
VI
THE GENERAL LINEAR HYPOTHESIS
Thus ZTR+Z = ( y + m , ) ' R R + ( y + m , )
c CrjYy+m,)12. P
=
j= 1
The vector random variable P(y+m,) where
has a multivariate normal distribution with mean P m , and covariance since &yyT
P ( € y y T ) P T = PP'
=
I.
The rows of P are orthonormal, so PPT = 1 (of order p x p ) , hence by (6.2.1), the components of P ( y + m , ) are independent normal with mean zero and unit variance, so by (6.2.2)
II P(Y + m1)ll has a noncentral chi-square distribution with p degrees of freedom and noncentrality parameter lIPm, 11. But IlP(Y +n71)Il
=
P'P
=
(Y + mY'P'P(y + m >
and P
rjrjT = R'R.
j= I
Therefore IIP(y+rn,)ii2
=
ZTR+ZT
and I(Pm,/12= m , T R R + m l = rn'R+m
=
d2
since ni,'RR+m,
=
11 RR+tn, 11'
RR'rn,
=
RR'(R%)'ni
and =
R+%m.
Part (b) follows as a special case of part (a) when n7 mean zero. (See Rao [I], for related results.)
= 0,
since Z - m has
ESTIMABLE VECTOR PARAMETRIC FUNCTIONS AND CONFIDENCE ELLIPSOIDS
97
(6.2.6) Exercise: If z has a multivariate distribution with mean m and covariance 0'1, and if R is a projection matrix with rank p , then o-' llRzllZ has a noncentral chi-square distribution with noncentrality parameter and
o-'(mTRm)%
p df.
(6.3) Estimable Vector Parametric Functions and Confidence Ellipsoids in the Case of Normal Residuals
In (6.1) we introduced the notion of an estimable parametric function (a real-valued linear function of the unknown vector parameter). We now extend this notion to vector-valued linear functions: If z = Hx+v
where v is a zero mean vector random variable, H is a known matrix and x is an unknown vector parameter, we define an estimable vector parametric function (evpf) to be a n y vector-valued linear function of x whose components are estimable parametric functions. Thus, an evpf is any vector of the form y = Gx where G is a matrix whose rows are gl', gzT,..., gkT,where, for eachj, gjTxis an estimable parametric function. I t is a trivial consequence of (6.1.8) that (6.3.1) Gx is an evpf if and only if 9 ( G T ) c 9 ( H T ) .
By (3.13. l), the last statement is equivalent to (6.3.2) Gx is an evpf if and only if G H ' H If G =
= G.
( "': ) gk
is an evpf, then each component of Gx has a BLUE, namely g j T S where S is the Gauss-Markov estimator defined in (6.1.12). Henceforth, we will call GS the BLUE for the evpf G.Y. For the sake of the following discussion, let us focus our attention on the case where the residuals v are normally distributed and have covariance V 2 proportional to the identity matrix: (6.3.3)
vz
=
b W T
= 021.
98
VI
THE GENERAL LINEAR HYPOTHESIS
According to (6. I . 12), the BLUE for an evpf, y
Q
= Gx,
is Q
= GQ, where
=H+z
happens to be the least squares estimate for x. The vector random variable Q - y has zero mean (since GQ is unbiased), and the covariance of 9 - y is
8 (9 -Y>
(9 -y)'.
But y = GX = G H + H x
(6.3.2)
and
Q
=
GQ = G H ' z
=
GH'Hx
9-y
=
GH+v
+ GH'v
so that and
(6.3.4)
Cov(9-y)
E
&($-y)(Q-y)T = & " ( G H + ) w ~ ( G H ' ) ~ ]
=
U~G(H'H)+G~.
Therefore, 9 - y has a multivariate normal distribution with mean zero and G ' by virtue of (6.2.5), covariance C J ~ G ( H ~ H ) +so, 6 ( y - 9)' [G(HTH)' GT] (y - 9 )
(6.3.5)
+
has a (central) chi-square distribution with k , (=rankGI#+) degrees of freedom, since rk(A) = rk(AAT)
for any matrix
A.
(4.10.2)
It is always true that
(6.3.6)
B ( G H + H ) E R ( G H + ) c R(G)
and if G is an evpf R(G)
=
R ( G H + H ) as well
(6.3.2)
so in this case,
(6.3.6.1)
W ( G H + )= W(G).
Therefore
k , = rk(G). The last result [concerning the distribution of (6.3.5)] can be used to construct a confidence set for y provided a2 is known. For if o,- a is the upper
ESTIMABLE VECTOR PARAMETRIC FUNCTIONS AND CONFIDENCE ELLIPSOIDS
99
lOO(1- LY)percentile of said chi-square distribution, then
Pr{(y-f)T[G(HTH)+GT]+(y-fr)< oZml-,)
= 1 - CI
which means that the ellipsoid
Y ( 0 2 w ,-,)
=
( u : ( u - ~ ) ~ [ G ( H ~ H ) +( G u -~9) ] d +
CT'CO~ -,}
covers the true value of y with probability 1 -ci. If o2 is not known, its maximum likelihood estimate is proportional to (6.3.7)
8' = (n-kZ)-' llz-H21lZ
where n is the number of observations (=number of rows in H ) and k , is the rank of H . We will now show that (6.3.8)
(
/ Z - H ~ has / ~ a chi-square distribution with n - k , df and is independent of y-9.
Once (6.3.8) is established, then from (6.2.3) ( k , 02)- ( y - 9) [ G ( H T H ) ' G T ] ( y - 9) +
(6.3.9)
o-262
=
( k , a2)- ( y - 9)T[G(HTH)'GT]' ( y - 9 )
has an F distribution with k , and n - k , df. Because of this, the ellipsoid 9 ( k 16 2 5 ,-,a ) covers the true value of y with probability 1 - a if 5,-,is the upper 100( 1 - LY)percentile of said F distribution, since Pr{y E Y ( k , it2&, -,)} =
Pr{(k,62)-1(y-9)T[G(HTH)+GT]+(y-9)< G I - , }
= 1 - IX.
To prove (6.3.8), we notice that o-'(z-HS) (6.3.10)
=
o-'(I-HH+)z
=
o-'(I-HH+)(Hx+v)
=
a-'(I-HH+)v
where o - l v has a multivariate normal distribution with mean zero and covariance I. Furthermore, 9 - y = CH'V
(6.3.4)
hence
o - ' € ( ~ - ~ ) ( z - H P )= ~ GH+(I-HH+)
=
0
100
VI
THE GENERAL LINEAR HYPOTHESIS
so that ( 9 - y ) and z - H S are jointly normally distributed (both are linear functions of v) and are uncorrelated. By (6.2.4) they must be independent. So, then, are
a2 = ( n - k 2 ) - '
and
jlz-HS/lz
(y-9).
(6.3.10) shows that 0 - l ( z - H S ) is of the form C ' P v where P i s a projection, so by (6.2.6) (a2)-' / / z - H j i l i 2 has a chi-square distribution with as many degrees of freedom as the rank of I - H H +. (6.3.11) Exercise
rk(I-HH')
=
rk(I)- rk(H).
[Hint: I is a projection and so is H H ' . Use (4.10.1)-(4.10.3).] (6.3.12) Exercise: Generalize the results of (6.3) to the case where the residuals have a normal distribution with zero mean and covariance c 2 Y 2 where d is unknown and V 2 is a known nonnegative-definite matrix.
(6.4) Tests of the General Linear Hypothesis Consider the observation model described in the previous section. Instead of a confidence ellipsoid for y = Gx, we seek a test of the hypothesis
Ye,: G x
=
0.
In the case where c2 is known, it turns out that the appropriate likelihood ratio test is to reject Xo0 for large values of
(6.4.1)
p1 =
o-2(l/z-HX]12
-
llZ-Hji112)
where 2 is the least squares estimator for x: (6.4.2)
S = H+z
and X is the least squares estimator for x, computed subject to the constraint that Gx = 0:
x = R+z
(6.4.3) where
R If
(i2
(6.4.4)
=
H(I-G'G).
(3.12.4)
is not known, the likelihood ratio test rejects Xo for large values of p2 =
llz-HKIlZ/IIZ-HS/y.
TESTS OF THE GENERAL LINEAR HYPOTHESIS
101
(6.4.5) Exercise: If %* and ji* are any other least squares estimate and constrained least squares estimate (computed subject to G x = 0), show that and
H%* = H%
Hi*
=
H i
so that the same test statistics, p1 and p2, result. [Hint: (3.12.4).]
Use (3.12.2) and
We will now prove that (6.4.6) pl hasanoncentralchi-squaredistributionwith7 [=rk(HH+ - R R + ) ] degrees of freedom and noncentrality parameter 6 where
s = 0-l \I(HH+
(6.4.7)
-RR+)HxII
and (6.4.8)
n-k, -(p2-1) T
[
=n - Tk 2
l\z-Hi1/2 - IlZ-Hftll, ll~-Hk//~
has a noncentral F distribution with T and n - k , [ k z = rk(H)] and noncentrality parameter 6.
1
degrees of freedom
Furthermore, (6.4.9)
6=0
if G x = O
so that the distributions of p1 and p2 are central chi-square and F when the null hypothesis is true. To prove (6.4.7), we note that (6.4.10)
W(R)c W ( H )
so that ( H H + ) ( R R + )= R R + . After a slight amount of algebra, it follows that (6.4.11)
( ( ( H H + - R R + ) Z= ~ ~~~\ ( z - R R + z )( z - H H + z ) / I 2 = ~ ~ ( z - H Z) (z= llz-HX//, -
H%)//'
llz-Hk~~2.
Thus p1 = Ilo-'(HH+ - R a + ) z l l 2 .
The random variable K 1 z is normally distributed with mean C ' H x and covariance I . Since W(R)G W ( H ) , H H + - H H + is the projection o n W ( H ) - B ( R ) , (2.7.4b), and so (6.2.6) guarantees that
11 0 - l ( H H + - R R + ) z 1
102
VI
THE GENERAL LINEAR HYPOTHESIS
has a noncentral chi-square distribution with noncentrality parameter 6 and 7 df. This establishes (6.4.7). To prove (6.4.8), it suffices to point out that z - H 9 = ( I - H H + ) z is is uncorrelated with (HH' - R R + ) z because & ( Z - H 9 ) [ ( H H + - RR') =
(Z
-H x ) ] ~
(I-HH+)&vv~(HH+-RR+)
=o because GvvT = 0'1 and H H + ( R R + )= RH+, (6.4.10). Therefore Il~-HEt11~ is independent of I ( H H f - H R + ) z I I 2 = //z-RTZ//~/ / z - H S / j 2 ,(6.4.11), and we showed in (6.3) that (a2)-'l/z-H91i2 has a central chi-square distribution with n - k , df, where k 2 = rk(H). The result now follows from (6.2.3). To prove (6.4.9), we note that x = ( I - G + G ) x if G x = 0 so that
( H H + - R R f ) HX
=
HH'Hx - RR'Hx
=
H X - RR'H(I- G'G) x
= H ( I - G + G ) X - RR+&
=o which implies
6 = 0. If 6 > 0, the values of p1 and p2 tend to be larger than if 6 = 0. To test .F0at the significance level LY, reject H0 if p1 > wl-o [the upper 100 (1 - a ) percentile of the central chi-square distribution with T df] provided a is known. Otherwise, reject Ho if T - ( m - k 2 ) ( p 2- 1) > G I - a , the upper lOO(1 - a ) percentile of the central F distribution with T and n - k , df. The power of both tests against various alternatives is a function of 6 only, and may be computed using the appropriate noncentral F and chi-square tables.
'
(6.5) The Relationship between Confidence Ellipsoids for Gx and Tests of the General Linear Hypothesis We will now show that the likelihood ratio tests which we developed in (6.4) (a known and a unknown) are equivalent to the test procedures which reject the hypothesis Gx = 0 whenever the 1 -a-level confidence ellipsoid (a known, a unknown) for G x fails to cover the point 0. (Recall, we developed two ellipsoids; one for known sigma, one for unknown sigma.) This result, plus an alternate representation for 6, the noncentrality parameter, and 5, the rank of H H + - R R + follow easily from
RELATIONSHIP BETWEEN CONFIDENCE ELLIPSOIDS AND TESTS
If W(GT) E 9 ( H T )and
(6.5.1) Theorem:
(a) and (b)
rk[H(Z-G'G)]
=
R = H(Z-G'G)
103
then
r k ( H ) - rk(G)
(GH+)+(GH') = HH' - RR' (a)
Proof:
rk [H(Z- G'G)]
= rk [ H + H ( Z - G'G)] =
rk [ H ' H - G'G]
since H + H G + G = G'G whenever W(G') rk[H(Z-G'G)]
(4.10.2b)
s 9 ( H T ) ,(3.5), so that
=
rk(H+H-G+G)
=
r k ( H + H ) - rk(G'G)
(4.10.3)
=
r k ( H ) - rk(G).
(4.10.2)
(b) Let
PI = ( G H ' ) + ( G H + ) and
P2 = RR'. %'(PI) = W(H+TGT)E 9 ( H + T )= W ( H ) = W ( H H + ) (3.5) and (3.11.5) and
9(P,)
=
W(R)E W ( H ) = B ( H H + ) .
By virtue of the fact that B(GT) E 92(HT),it follows that (6.5.1.1)
G
Hence
=
GH'H.
(6.3.1) and (6.3.2)
CH+R = GH+H(Z-G+G)
=
0.
Therefore P,P, = 0 and by (4.10.3), it suffices to show that rk(P,) + rk(P,) rk(HH'). It will then follow immediately that HH' = P,+ P , . But rk(P,)
=
rk(GH')
=
rk(GH+H)
=
rk[H(Z-G'G)]
=
r k ( H ) - rk(G)
=
rk(G) (6.5.1.1) and (4.10.2)
=
and rk(P,)
=
rk(R)
Thus, rk(P,)
+ rk(P,)
and this proves (b), (4.10.3b).
=
rk(H)
=
[part (a)].
rk(HH+)
(4.10.2)
104
VI
THE GENERAL LINEAR HYPOTHESIS
(6.5.2) Corollary: Let 6 and
7
be as defined in (6.4.6) and (6.4.7). If
Gx is an evpf, then 7 = rk(G) and 6 = o-' I\(GH+)+Gx/l.
Proof T =
rk(HH+-HH+).
If G is an evpf, then 9?(GT) E 2 ( H T ) ,(6.3.1), and by (6.5.lb) and (6.3.6.1), 7 =
rk[(GH+)+(GH+)] = rk(GH+) = rk(C).
By the same token,
6
= o-1 ll(HH+-RR+)HxlI = 0-l
~~(GH+)+(GH+H~)/I
= 0-l
ll(GH+)+Gxll
if G is an evpf.
(6.3.2)
Referring to section (6.3), we now see that the confidence ellipsoid Y ( 0 2 w l -,) covers the point y = 0 if and only if (6.5.3)
9'[G(HTH)+GT]+9
< 02w1-,
where w1 -,is the upper lOO(1 -a) percentile of the chi-square distribution with z df. Since 9 = G H ' z , we can write (6.5.3) as II(GH+)+GH+zl12< a2c01-a
(6.5.4)
which, by virtue of (6.5.lb), is the same as (6.5.5)
(o2)-'
II(HH+- R R + ) z l l 2
< ol-a.
Thus, the likelihood ratio test for the case of known o,rejects Zoat the level a if and only if the (1 -a) level confidence ellipsoid for Gx fails to cover the point y = 0. (6.5.6) Exercise: Show that the likelihood ration test (for unknown a ) at the level a rejects Xoif and only if
9 ( k 1a2b1-,) fails to cover y
[defined in (6.3)]
= 0.
(6.5.7) Exercise: If G is an evpf then 6
=0
if and only if G x = 0.
(6.6) Orthogonal Designs In statistical investigations, it is often desired to obtain information about the value of one or more parametm and in an effort to do so, the researcher designs an experiment which will yield data of the form z=Hx+v
ORTHOGONAL DESIGNS
105
where z is the “observation vector,” x is an unknown vector whose components include the parameters of interest (and possibly other so-called “nuisance parameters”) v is the vector of residuals (or “observation errors”). H i s a known matrix, often referred to as the “experimental design.” The art of experimental design occupies an important niche in the lore of applied statistics and one of its basic concepts is the notion of “orthogonal design.” Suppose y j = G j x are evpf’s for j = 1,2, ...,p (i.e., each component of each y j is an estimable parametric function). The design H is said to be orthogonal with respect to the evpf’s G I x, G, x, ..., G,x if and only if the BLUE’S for y i and y j are uncorrelated when i $ j . One reason that orthogonal designs are desirable is to be explained in Theorem (6.6.9). The BLUE for each y j can be computed as though all the other yi’s are known to be zero. This fact often reduces the computational complexity of a least squares problem by several orders of magnitude. For the sake of clarity, we will confine our attention to the case where the residual vector v has covariance proportional to the identity matrix. In that case, the BLUE for y j is
9J. = GJ . H + z
(6.6.1)
( j = 1, . . . , p )
(6.1.12)
and
Cov(Qi,9 j )
&(9i-yi)(9j-yjIT = EG, H
+(z - HX)(Z - H X ) ~ ( GH~ +)T
=
ciH + & V V ~ H +T~;
=
0 2 ~ i ~ + ~ + T ~ j T
=~ G ~ ( H ~ H ) + G ~ ~ .
Thus,
9i is uncorrelated
(6.6.2)
with
fj
if and only if
G ~ ( H ~ H )= +G o~~
which is the same as saying the row vectors of Gi must be “orthogonal” to the row vectors of G j with respect to the inner product induced by (HTH)’ (which, by the way, is proportional to the covariance of A = H’z, the BLUE for x). Equation (6.6.2) can be written in the alternate form
(6.6.3)
( G & + ) ( G ~ H + )=~O
i
+j
so that H is orthogonal with respect to G,x, G,x, ..., G,x if and only if
W [(GjH’)T] c
N(Gi H’)
when i # j .
106
VI
THE GENERAL LINEAR HYPOTHESIS
Since g [ ( G j H + ) T ]= W [ ( G jH + ) +(GjH + ) ] and Jlr(GiH+)= Jlr[(GiH+)+(GiH+)]
(3.1 1.5)
we see that H is orthogonal with respect to GI x, ..., G,x if and only if Pipj = 0
(6.6.4)
when i # j
where (6.6.5)
Pj = ( G j H + ) + ( G j H + ) j = 1,2,..., p .
In the proof of (6.5.lb) we showed that w [ ( G H + ) + ( G H + )c ] 9(HH+) if Gx is an evpf, so it follows that (6.6.6)
1
"
w C ( G ~ H + ) + ( G ~ HE+ ~) ( H H + ) . j= 1
If equality holds in (6.6.6), then by (3.7.7), H H + is expressible as the sum of orthogonalprojections, a fact which we will see to be of more than passing interest. [Two projections P I and P, are said to be orthogona! if P , P, = 0. Clearly P,P, = 0 if and only if P, P I = 0 and the last is true if and only if %!(PI)_c Jlr(P,).] If strict inclusion holds in (6.6.6) (as it may) we can, by suitably defining one more evpf (which we call y o = C o x ) , decompose H H + into a sum of orthogonal projections: HH+
=
2 (GjHf)+(GjH+).
j=O
In the next theorem, we show how to define Go: (6.6.7) Theorem: G , x , ..., G,x and if (6.6.7.1)
If H is an orthogonal design with respect to
T
=
HH+ -
f ( G ~ H + ) + ( G ~ Hz +o)
j= 1
then H is also an orthogonal design with respect to G,x, G , x, ..., G , x where Go = TH.
ORTHOGONAL DESIGNS
107
Furthermore,
(6.6.7.2)
HH+ =
P
C(G~H+)+(G~H+)
j=O
and the projections in the sum are mutually orthogonal. Proof
GoH'H
=
THH'H
T H = Go
=
so Go is an evpf, (6.3.2). To show that H is orthogonal with respect to C o x , G 1 x , ..., G,x, we must show that
k = l,...,~.
( G k H + ) ( G o H + ) T= 0 The left side is
( G k H + ) ( T H H + ) T= G k H + T (since H + H H + = H + and T is symmetric). If G, H + is applied to each tcrm on the right side of (6.6.7.1), we see that
Gk H T +
=
Gk H [ H H - (Gk H +) (G, H ')] +
= GkH+
+
+
- GkH'
= 0.
This shows that His orthogonal with respect to Gox,..., G , x. Since Go = T H ,
[GOH+lT = [ T H H + I T = HH'T and since W ( T ) E 9 ( H H + ) , (6.6.6),
HH'T = T.
(3.13.1)
Thus,
T = TT = (GoH+)T and since T is the projection on 9 ( H H + ) - W ( P , + P 2 + . . - + P , ) ,
(GoH+)+ = T + = T
so that T = T2 = (GoH+)+(GoH+) which is another way of expressing (6.6.7.2). Comment: The zeroth term of the sum (6.6.7.2) has an interpretation: Recall that the set of all estimable parametric functions consists of all parameters of the form gTx where g is free to range over 9 ( H + H ) . The
108 VI
THE GENEUAL LINEAR HYPOTHESIS
components of the evpf’s GI x, G2x, ..., G,x collectively correspond to the class of estimabie parametric functions of the form gTx where g is free to range over 92 [(GIT! GZTj ... ! GPT)]. Since H’HG? = GjT, this manifold is a subspace of 9 ( H t H ) . Now, Go is defined in such a way that
W[(GoT/GITj...jGpT)] = W(H+H) and so that H remains an orthogonal design. (6.6.8) Exercise: Show that 92 [(GoT i G,T j ...i GpT)] = W [ H ’ H ] . The concept of “orthogonal design” was of great importance in the days before the widespread availability of digital computers. The task of solving the least squares “normal equations” and evaluating the residual sum of squares on a desk calculator, was tedious at best and overwhelming if the dimensionality of the unknowa vector x was greater than 6 or 7 unless the design was orthogonal. In thai case, as we will see in the next theorem, each evpf could be estimated separately as though the rest were known to be zero. The general effect was to reduce the estimation of a k-dimensional parameter (with associated computations equivalent in complexity to that of inverting a k x k matrix) to the estimation of p + 1 parameters, each of dimension pi (each requiring the equivalent of a pi x pi matrix inversion) where x i p i = k. Generally speaking, the inversion of a k x k matrix requires on the order of k 3 arithmetic operations. Therefore
so that considerable computational savings would be realized were one to capitalize on the orthogonality of the design. Finally, an intuitively informative decomposition for the residual sum of squares, I/z- H . U ] is / ~ another dividend associated with orthogonal designs. ‘To be precise :
(6.6.9) Theorem (Orthogonal decomposition f o r analysis of variance) (a) If z = H x + v where v is a zero-mean vector random variable with covariance 021,if y i = G i x (i = 1, . . . , p )are evpf’s, and if H is an orthogonal design with respect to these evpf’s, then (6.6.9.1) / / z - H . u / / ‘ = //z-HEa1I2 +
f (fj-yj)T[Gj(HTH)+G~]+(~,-yj)
J:o
where Go is the (possibly zero) matrix defined in (6.6.7), j i = H ’ z is the least squares estimate for .Y and ji, = G j 2 is the BLUE for y j ( j = 0, 1, . . . , p ) .
ORTHOGONAL DESIGNS
109
(b) 9j can be computed in the following (alternative) fashion: Let be any vector minimizing Ilz-HxII’ subject to the constraints
Gix = 0
for i = 1,2,...,j - l , j + l ,
mi
...,p .
Then
fj = G j g j .
(6.6.9.2)
(c) If it is assumed, in addition, that the residual vector v is normally distributed, then the first term on the right side of (6.6.9.1) (when divided by a’) has a chi-square distribution with n-k df, where n is the dimension of z and k is the rank of H . T h e j t h term of the sum (when divided by 0’) has a chi-square distribution with p j df, where p j is the rank of Gj . All p + 2 terms are mutually independent and pi = k .
Comment: The sum of squares decomposition (6.6.9.l), “explains” the “total” sum of squares Ilz-HxII’ in terms of the sum of squares associated with each of the evpf’s y j and the so-called residual sum of squares 112- Hftlj*. These quantities are traditionally computed with the yj’s set equal to zero and they are exhibited under the title “analysis of variance table” (Scheffk [ l , Chap. 41). Proof: (a) Since (6.6.9.3)
+ HH+(z-Hx) = (I-HH+)z + HH+(z-Hx)
z - HX = z - H H ’ z
we deduce from (6.6.7.2) that (6.6.9.4) z - HX = ( Z - H H + ) z
+ 1(GjH+)+(GjH+)(z-Hx). P
j=O
The Gj’s are associated with evpf’s, hence G j H f H x= Gjx = yj and since H ’ z
y j , we see that
(6.6.9.5)
=f
is the BLUE for H ’ H x and G j f
z - Hx = ( z - H f )
(6.3.2) = $+j
is the BLUE for
+ f(GjH+)+(fj-yj). j=O
The first term on the right side of (6.6.9.4) is uncorrelated with and orthogonal to each term in the sum since (a) Z-HH’ projects onto W(H)’ [ = N ( H T ) ] , whereas each term of the sum projects onto a subspace of W ( H ) , (6.6.6), and (b) the covariance of z - H x is proportional to the identity. Since H is an orthogonal design with respect to the Gjx’s, the
110
VI
THE GENERAL LINEAR HYPOTHESIS
projections in the sum are mutually orthogonal, (6.6.4). So then are each of the vectors in the sum, and since the covariance of z - H x is proportional to the identity, the terms are also mutually uncorrelated. The mutual orthogonality of all terms allows for the sum of squares decomposition for (6.6.9.5):
H2112
= llz-
+ f(~j-yj)T[Gj(HTH)+GjT]f($j-yj), j=O
which establishes (a). (b) If minimizes Ilz- Hx/I2subject to G i x = 0 for all i # j , then m. = H g .
minimizes l ~ z - m ~ subject ~2 to G i H t m = 0 for all i#j and m E B ( H ) , since Gi H +m = Gi H H x and G, H ' H x = Gi x when Gi is an evpf, (6.3.2). Therefore, mj minimizes llz - m /Iz subject to the constraint +
(6.6.9.6)
(I-HH+)m
+ f ( G , H + ) + ( G i H + ) m= 0, i=O i#j
since GiH+m= 0 if and only if ( G i H + ) + ( G i H + ) m= 0 and since the terms in (6.6.9.6) are mutually orthogonal. By (6.6.7.2),
(6.6.9.7) ( Z - H H + )
P
+ 1 ( G i H t ) + ( G i H f )= I - ( G j H + ) + ( G J H + )= Q j i=O i#j
which is a projection, so that liz-mli2
=
I/z - Q j m - (Z-Qj)m(12
is minimized subject to the constraint Q j m = 0 if and only if
(6.6.9.8)
mj = ( z - Q j ) Z .
(3.12.4)
G1. H + A . = G . H + H f t . = G1. 91 .
(6.3.2)
Since mj = H e j , we see that
ORTHOGONAL DESIGNS
1 11
whereas (6.6.9.8) implies that
Gj H + e?, = Gj H + [(GjH + ) + ( G jH’)] z = G j H + 2 = Gjm.
Hence
GJ - 9 = Ginj = fj which proves (6.6.9.2). (c) In part (a), we showed that all terms on the right side of (6.6.9.5) are uncorrelated. If a normality assumption is added, they are independent as well, (6.2.4), which implies that all terms on the right side of (6.6.9.1) are mutually independent. The distribution of o - llz~ H91I2 has been shown to be chi-square with n - k df [(6.3.10), (6.3.11)]. The distribution of 0 - ’ ( 9 ~--yj)=[Gj(HTH)’G;] ( f j - y j ) was shown to be chi-square with pi df in (6.3.5). The left side of (6.6.9.1), when divided by o*, is of the form
2 n
j= 1
(vj/O)’
where the vj’s are independent with mean zero and variance o’, so that the left side of (6.6.9.l), when divided by ts’ is chi-square with n df. Thus,
n=n-k+fpj j=O
so that P
C pi
j=O
= k.
Comment: By definition, H is always orthogonal with respect to a single evpf G , x . This means that a decomposition of the form (6.6.9.1) can be exhibited for the case p = 1, provided Go is properly defined [according to (6.6.7)].
(6.6.10) Exercise: (a) H i s orthogonal with respect to G i x (i = 1,2, . . . , p ) if and only if ( H H + - A i W i + ) ( H H + - A j A j + )= 0
nj
when i # j , where = H(I- C j Cj’). [Hint: Use (6.5.lb).] (b) Let xj* be any x which minimizes I ~ Z - H Xsubject ~ ~ ~ to XE
N ( G , ) n ... n J V ” ( G ~ -n ~ )N ( G j + l )n ... n N ( G p ) .
112
VI
THE GENERAL LINEAR HYPOTHESIS
Let yj* = Gjxj* and let pi = G j 9 , the BLUE for y j = C j x . H is orthogonal with respect t o G i x (i = 1,2, ... , p ) if and only if yj*
=
for j = 1,2 ,...,p .
jj
(6.6.1 1) Example (Straight line regression): Observations are taken, of the form
ci =
c(
+
+ vi
i = l,...,n.
In the vector-matrix notation, Z =
HX+V
where
If we let GI = (1 0 ) and G, = (0 l), theny, = GI x = tl andy, = G , x = B. H has rank k = 2 if the T'S are not all the same, in which case H + H = I so y , and y , are evpf's.
Rj
H(Z-GjGj+) =
(Ofr)
j = 1
(eIO)
j =2
so that
The quickest way to find H H is to perform a Gramm-Schmidt orthogonalization on the columns of H and then apply (3.7.3): +
HH+
eeT
=-
llel12 +
uuT
m
WWT - -itT 1It/l2+
Thus
/lw112
where
u = t--e
where
(tTe) w = e --t. II t ll
llellZ
ORTHOGONAL DESIGNS
113
and so
( H H + - R , R 1 + ) ( H H +-R2 IT,+) =
wTu I~wl~’(wuT)*
112.4112
By virtue of (6.6.10) H is an orthogonal design with respect to GI x, G, x if and only if the last expression vanishes. Since
and since t is not a multiple of e if the z’s are not all the same, Schwarz’s inequality tells us that the term in square brackets is strictly negative so w’u = 0 if and only if (e’t) = 0. In summary, H i s orthogonal with respect to a and p if and only if n-l 7j 3 5 = 0. So, H is an orthogonal design if and only if the observations are chosen at values of zj whose average value is zero. In that case, we can compute the BLUE for tl as though p = 0: “Pretend” that
xy=
Then the BLUE for
tl
is
ci = tl + v i d
i = I, ...,n.
c ci. n
=
i= 1
Similarly, we can compute the BLUE for /3 as though tl = 0: “Pretend” that The BLUE for j? is
ci = p7i + vi
s
=
t+z =
i = l,...,n.
fcjrj/
j= 1
.:T j= 1
In this special two-dimensional problem, the BLUE for x can be exhibited explicitly in the general (nonorthogonal) case :
where
and
114
VI
THE GENERAL LINEAR HYPOTHESIS
It is apparent that the orthogonality condition i = 0, reduces these formulas to the estimates for CI and p given previously. The sum of squares decomposition for the orthogonal case is
c n
6-2
n
= 6-2
(qj-.-pTj)2
2
(&j-a-pTj)2
j= I
j = 1
+
(S-p)2/a@’
+
(&-CI)’/a,2
where B =
n
n-l
1 i&
a, 2 = 02/n
i= 1
If the design were not orthogonal with respect to CI and p, it would still be orthogonal to either one, taken separately. Suppose p were the parameter of interest. Let us redefine
G,
=
(0 1)
so that G,x = B.
Using (6.6.7), we define Go
=
[ H H + - (G, H + ) + ( G ,H ’ ) ] H
=
e(njn7).
Thus, Cox = n ( a + f p ) e
and H is orthogonal with respect to Cox, G , x. We can compute S by minimizing IIz-HxI/’ subject to C o x = 0 (i.e., CI = -ip). In this case, the sum of squares n
becomes
which is minimized when
as it should.
ORTHOGONAL DESIGNS
115
(6.6.12) Exercise (Two-way layout): Suppose observations are made of the form i = 1..... I j = 1..... J
~ij=ai+jlj+vij
where the a’s and p’s are not known and the residuals, v i j ,have zero mean and are uncorrelated with common variance, 0’.
z =
Cll
v11
Cl2
v12
clJ
v1 J
sz1
VZ 1
x =
52J
VZ J
61 1
VI1
61J
vIJ
and
I
.......................................... I
J
1
o...o
----
---
0 1...0
Let G,x=
Ii I ] 6 1
and
1
0 0 . ..1
I------I
I
1
o...o
Gzx=
1[?I.. DJ
Show that the design is not orthogonal with respect to G , x and G z x . (Hint: Show that the 1.s.e. for G , x is correlated with the 1.s.e. for G z x . )
116
VI
THE GENERAL LINEAR HYPOTHESIS
(6.6.13) (Continued): Let E =
I-'
J
I i= 1
ai
+ J-' 1 Bj j= 1
and let z and v be as in (6.6.12). (a)
Find H so that
Z=HX+V (Hinr: & j = ~ + b i + y j + v i j , i = 1,..., Z, j = I ,..., J.) (b) Let
Show that H is orthogonal with respect to GI x, G, x,and G , x. fj. Then the orthogonal de(c) Define 8, = -Cf= 8 j , f J = -Z;: composition for the sum of squares for an orthogonal two-way layout is
c (~ij-&-di-yj)2 c ( ~ i j - e - 8 i - q j ) z + J c (8, dJ2 + 1c =
ij
ij
-
i
+ zJ(a -
i
(qj
-Y j ) ,
E)2.
I f the residuals v i j are independent and normally distributed with mean 0 and variance 1, the terms on the right have independent chi-square distributions with ZJ- ( I fJ- I), I - 1, J - 1 and 1 df, respectively.
ORTHOGONAL DESIGNS
1 17
(6.6.14) Exercise: Let P
4
H = n(H1 j H2)
be an n x ( p + q) matrix of rank p
+q, and suppose
Z=
HX+V
where the residual v has covariance I. The least squares estimate (= BLUE) for x is unique and can be obtained via the pseudoinversion of the n x ( p + q) matrix H : ft = H'z.
An alternative method which may involve less work (and which has applications to stepwise regression in Chapter VIII) goes like this : Let
Go = H I H , + H
X = x2
and P 4
G.= q(0jI). Then (a) H is an orthogonal design with respect to Go x and Gx. (b) The BLUE for x 2 is unique and is given by
9, = Gx* where x* minimizes I ~ z - H xsubject ~ ~ ~ to Gox = 0. (c) fr,
= (Q,H,)+z
= (H2TQlH2)+HZTQ,z, where
Q, = I - H, HIf
E
I - H,(HITH,)-'H,T.
(d) The BLUE for x1 is unique and is given by
9, = H,+(z-H2S2)
(H1TH1)-1H,T(~-H292).
[Comment: 9 , and ft2 and hence,
can be obtained by way of a q x q inversion and a p x p inversion. Ifp z q and both are large, a sizable computational saving can result.] (e) Let x, * = H , z. +
118
VI
THE GENERAL LINEAR HYPOTHESIS
Then llz-H%/I2 = [ [ Z - H , X1*1l2 - %2T(H2TQ1
H2)%2.
(Thus, the residual sum of squares associated with the BLUE for x under the “full” model z = H, x,
+ H2 x2 + v
differs from the residual sum of squares associated with the BLUE for x1 in the restricted model z = H,x,
+v
by the “correction factor”
9 2 HzTQ1H2 92
/IQi ffz 9 2 II’..)
Chapter VII ~
C O N S T R A I N E D LEAST SQUARES, P E N A L T Y F U N C T I O N S , A N D BLUE’S
(7.1) Penalty Functions
In many applications, we have seen that it is necessary to compute a weighted least squares estimator subject to linear equality constraints. That is, it is necessary to find a value of x which minimizes (7.1.1)
( z - H X ) V - 2 (z - Hx)
subject to the constraints (7.1.2)
GX = u
where V is a known positive-definite matrix, z and u are given vectors, and H and G are given rectangular matrices. A very general result in the theory of minimization which is associated with the “penalty function method,” asserts that the value of x which minimizes (7.1.3)
h(x)
+P g 2 ( x )
[call it x(A)], converges (as L + 0) to (7.1.4)
120
VII
CONSTRAINED LEAST SQUARES, PENALTY FUNCTIONS, AND
BLUE’S
if certain mild continuity restrictions are met, and that xo minimizes h ( x ) subject to the constraint g ( x ) = 0 (Butler and Martin [l]). Furthermore,
+ A-2g2(x(A))] = h(xo)
Iim[h(x(li))
(7.1.5)
1-0
so that the minimal value of (7.1.3) converges to the minimal value of h on the constraint set. The term ( i L - 2 ) g 2 ( xis ) called a “penalty function” because the minimization of h ( x ) A - 2 g 2 (x) suffers when x lies outside the constraint set [ g ( x ) = 01 if A is small. In the case at hand, if we let
+
(7.1.6)
h(x) =
(Z-Hx)TV-2(Z-Hx)
and
(7.1.7)
g2(x)= (u-Gx)~(u-Gx)
and if x(L) is a value of x which minimizes (7.1.8)
(Z
- H x ) V~ -
(Z
-H x )
+ 1-’ (U - G x ) ~ (-uGx)
then it is reasonable to expect that xo = limx(A)
(7.1.9)
1-0
exists and minimizes (7.1.6) subject to g 2 ( x ) = 0 (i-e., Gx = u). Instead of invoking the general theorem, we will produce a self-contained proof based upon the material developed so far: (7.1.10) Theorem: Let H be an n x m matrix, G be a k x m matrix and V be an n x n positive-definite matrix. Let [order ( n + k ) x (n+k)] and
where z is a given n-vector and u is a given k-vector. Let f(A) =
Then (a)
[ P - ’ ( L ) A ] +P-l@)2.
no.) is the vector of minimum norm among those which minimize
(2 - fix)’
P - 2 (A)(2 - fix) = ( 2 - Hx)T v - 2 ( z - H x ) + 1 - 2 (u- G X ) T ( U - Gx).
CONSTRAINED LEAST SQUARES, PENALTY FUNCTIONS, AND
BLUE'S
121
(b) limA+o:(I) = xo always exists. (c) Among all vectors which minimize IIu- Gxlj', xois the one of minimum norm among those which minimize ( 2 - H x ) ~ V - '(2- H x ) . (d) If u E B(G), then the set of x's which minimize IIu- Gx1I2 is identical with the set of x's which satisfy the constraint Gx = u. In this case, xo minimizes (z- H x ) V~ P 2( z - H x ) subject to the constraint Gx = u. Furthermore, Proof of theorem: (a) ( Z " - f i ~ ) ~ p - ~ ( A ) ( Z - f= i xlI8-'z-8-'Axll2 ) and part (a) follows from (3.4). (b) and (c): Let
(7.1.10.1)
F = V-'H
(7.1.10.2)
w
=
v-'z.
Then by (3.8.1) Z(A)
=
[V-qA)R]+V - l z
= [FTF+ A-2GTG]+ [FTw
BY (4.9), [ F T F + K 2 G T C ] + = (FTF)'
+A-2GTu].
+ A2(Z-F+F)[(G+G)++ J(A)](I-F+F)T,
where
F
=
F(Z-G+G)
and
J(A)
=
O(A')
as
A
-+ 0.
Thus
(7.1.10.3) Z ( I ) = (FTF)+FTw+ I-'(PTF)+GTu
+ (I-F+F)[(G'G)+ + O ( A Z ) ] ( I - F + F ) T C T u as
A
--*
0.
But
FGT = F ( I - G + G ) G T
(7.1.10.4)
=
F[G(Z-G+G)IT
=
0
so B [ G T ] c N(F) = N(F+T) = N ( F + F ) = N[FTF)+] (3.11.5)
and so (7.1.10.5)
(FTF)+GT= 0
F+TGT = 0.
and
Therefore (7.1.10.6) ( I - F f F ) ( G T G ) +( I - F+F)TGT= (I- F+F)(GTG)+CT =
( I - F'F) G +
(3.8.1)
122
VII
CONSTRAINED LEAST SQUARES, PENALTY FUNCTIONS, AND
BLUE’S
and by (3.13.10) (7.1.10.7)
(FTF)+FT= F+
Combining (7.1.10.3)-(7.1.10.6), (7.1.10.8)
2(A) = F + w
+ (Z-F+F)[G+u+O(A2)]
as
A
+0
so
I(A)
(7.1.10.9)
= xo
+ o(A’)
as
A -+ 0
where (7.1.10.10)
x0 = F’w =
+ (Z-F+F)G+u
F + ( w - FG’u)
+ G+u.
By (3.12.7), the latter is exactly the vector of minimum norm which minimizes Ilw-Fxll’ among those vectors which also minimize Iju- Gx11’. This proves (b) and (c), since
/Iw - Fxll ’ = (2- H X ) ~ I / - ’ ( Z Hx). -
(d)
lim [Z - AI(A)ITV-’
=
(A)[Z - W?((n)]
+
lim { 11 w- Fxo O(A’)II
1+0
+ A-’
IIu- Gxo
+ O(A’)>l’}.
Ifu E g ( G ) , thenx’must satisfy theequationGx = uifitminimizes IIGx-ull’, so that the last term tends to zero as A 40 while the first tends to / I w - Fxo]j2. (7.2) Constrained Least Squares Estimators as Limiting Cases of BLUE’S
By (6.1.12c), I (A), defined in (7.1. lo), coincides with the BLUE for x when observations of the form (7.2.1)
Z=Rx+f
are used to estimate x, where B is a vector random variable with mean zero and covariance P2(E.), where P(A)is defined in (7.1.10). Thus, (7.1.10) shows that any constrained weighted least squares estimator can be viewed as the limiting case of a BLUE, some of whose observations are extremely reliable (i.e., have extremely low residual variances). To put it another way, constrained, weighted least squares estimators can be approximated arbitrarily well, by treating the constraints as fictitious “observations” which are
CONSTRAINED LEAST SQUARES AS LIMITING CASES OF
BLUE’S
123
extremely accurate, and computing the BLUE for x using both the “real” observations (pretending they have covariance V ) and the “fictitious” ones (pretending they have covariance A2Z, with A’ small). (7.2.2) Exercise: If V has covariance
the BLUE for x is given by (7.2.2.1)
Z(0) = R+[Z--
qo)(Q“v(O))+]z
where Q“ = (Z-fifi’), (6.1.12). Is it true that Z(0) coincides with xo = limL+o%(A)? (This would mean that constrained least squares estimates are obtainable as BLUE’S by treating the constraints as “perfectly noiseless” observations. See Zyskind and Martin [I] ; Goldman and Zelen [I].)
Chapter Vlll
RECURSIVE C O M P U T A T I O N OF LEAST SQUARES E S T I M A T O R S
(8.1) Unconstrained Least Squares In many contemporary applications of the least squares technique, the data arrives in a stream (i.e., in temporal succession) and in such cases it is desirable to carry out the computation of the least squares estimate so that at each instant of time it fully reflects all the data that have been collected so far (cf. American Statistical Association [11 for notable applications of this philosophy). In Chapter V, several algoiithms were described for computing H ' z , but none of these were ideally suited for a situation where the least squares estimate is to be updated every time a new datum is obtained. For the sake of concreteness, we will initially view this question in the framework of linear regression analysis, but this interpretation is not essential. We are really just relating the least squares estimate for x, based on n data points, to the least squares estimate for x based on n + 1 data points. Imagine a stream of scalar observations, 12,... arriving in temporal succession, and assume that the jt h observation is of the form
cl,
(8.1.1)
cj = hjTx + vj 125
126
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
where each hj is a known p-dimensional vector, the vj's are uncorrelated zero mean residuals with common variance and x is an unknownp-dimensional vector. In (6.1.12), we showed that the BLUE for x based on the data
is (8.1.2)
2,
=
H,+ Z ,
where
If a new observation of the form L + 1
=
h;f+lx+vn+l
is taken, then the BLUE for x is (8.1.3)
%+I
=
H,++lzn+,
where
and
It turns out that Bn+ is nicely related to 2,: (8.1.4)
H,++l
=
By (4.3.2)
C(Z-Kn+lhT+l)Hn+!Kn+lI
where if ( I - H , + H , ) h , + , # 0 (8.1.5) K , + , =
. otherwise.
UNCONSTRAINEDLEAST SQUARES
127
Thus,
(8.1.6)
where
Po
=
0.
Notice that the recursion (8.1.6) takes the so-called “differential-correction” form: If 8, is used as an estimate for x, a “predictor” for (the next datum) is h;, 2,. The prediction error is &,,+ -h;f+ 9,. From (8.1.6) the new estimate f , + , is obtained from 2, by adding on a term proportional to the prediction error. The “vector of proportionality” K,+ is sometimes referred to as the “smoothing vector.” It does not depend on the data, (8.1.5). The recursion need not be restricted in its interpretation to the present statistical domain of discourse. It is a fully general result which relates H + z to
c,+l
where hT is an arbitrary row vector (of the correct size) and [ is an arbitrary scalar.
(8.1.7) Special case: Suppose H,, is of full rank (= the number of columns of H,). This means that H,’s columns are linearly independent. So then are H n + l ’ s . Consequently, HmTH, has an inverse for all m 3 n. This allows an alternative derivation: B,
=
(HmTH,)-’
Let for m >, n.
We have already shown, (4.6. I), that (8.1.7.1)
where h i + is the row vector that is adjoined to H , to produce H,+ Since am+, =
H,++lzrn+l = Bm+IH,T+lZm+l
= ~rn+~(HrnT~~m m ++ 11 ) ( < ~ - )
1 .
128
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
we can apply (8.1.7.1) and obtain (using the symmetry of B,) 9m+
=
1
Jm+ 1
+ Am+ 1
B m CHrnTzm
Cm+ 1 3
where
Since B,,, HmTzm= H,+z,
= 9,
and
we find that
Since the columns of H,,, are linearly independent by assumption, Jlr(H,) = {0}, hence the projection on . N ( H , ) is zero: I - H,,,+H,, = 0. It follows that (Z-J7,,,+H,.,)hm+, = 0, so that K,,+ I , defined in (8.1.5), is given by the second half of the right side which coincides with B, h,+ 1 / 1 + h i + B, h,+ as it should. (8.1.8) Exercise (a) is a linear combination (1.c.) of h l , ..., h, if and only if ( I - H,,+H,) h,,+ = 0. (b) Let A , = I - H , + H , and B,, = (H;H,,)+. Then -Aflhrl, 1
(8.1.8.1) K , , ,
=
G+
1
if hfl+ is not a 1.c. of h , , ..., h,
Aflhfl+1
Bflhfl+1 1 +AT+ 1 B, A,+
otherwise 1
where (8.1.8.2) A , + 1
=
(A,h,,+ l ) ( A , ,h,+ A$+ 1 A h , , +1
if h,+ is not a 1.c. of hi > .. ., h, otherwise
A,=I
B,=O
RECURSIVE COMPUTATION OF RESIDUALS
129
and
+A:+ Bnhn+ + 1(h;f+ Anhn+ 1
(8.1.8.3) B,+, = .
11,
1
Bn
-
1
(Anhn+1)(Anhn+1IT
,
if h,+ is not a I.c. of h,, ..., h,
(Bnh"+I)(Bnhn+1IT
(8.2) Recursive Computation of Residual Errors Denote the residual error associated with 2, = H,+z,, by (8.2.1)
En
=
IIZn-Hn%II'*
The results of (8.1) can be used to relate (8.2.2) Theorem:
E~
E,+,
= (HnTH,)'
and
E,:
=0
if A,+
where B,
E,:
is not a 1.c. of h,, ...,h,
satisfies (8.1.8.3).
Comment: (8.2.1) is the same as ~ . ; = , ( i j - h ~ i , JsoZ ,(8.2.2) asserts the truth of the following identity:
where
J,
= {j:j
and hj is a 1.c. of h , ,..., h i_ , ) .
The identity holds for all scalars
cl, ..., i n ,all vectors, h,, h,, ..., h,, and all n.
Proof: Since
we see from (8.1.6) that (8.2.2.1) zn+
- H, + 2,+
=
(t,+,- h;f+,A,).
130
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
Case 1: hn+,is not a 1.c. of h, ,..., h,,: Then (Z-H,,+H,,)hn+, # 0, and Kn+, is defined by the first part of (8.1.8.1). Therefore
Hn Kn + 1 = Hn An hn + ,/(h;f+1 An hn + 1 )-
Since A,, = I - H,,+H,,
it follows that H,,A,, = 0
so and
H,,Kn+, = 0
h;f+lKn+l= 1.
Hence by (8.2.2.1), = IIZn+l-Hn+lgn+lIIZ
En+1
= llz,-Hn55,1/2 = E n .
Case 2: h,,+l is a I.c. of h l , ..., hn: In this case, K,, is defined by the second half of (8.1.8.1). Then by (8.2.2.1),
(8.2.2.2) /Izn+1 -Hn+ 1 %+ =
111’
IIzn-Hn~nII’+(in+~-h;f+~~n)z
- 2 {(zn- Hn 9JTHnKn+1
x
(in+ 1
-A:+
+ ( i n + 1 -A:+
1an)
+ {KT+1 H,THn Kn+1 + (A$+ 1 Kn+1)’> =
Ilzn-
H n
1 Rn)h;f+ 1 Kn+ 1 1
’
2nlI - ’ZJ(I-Hn
(in+ 1
-C+1 9,)’
Hn+)HnKn+1 ( C n +
+ C(K,T,1 HJHn Kn+ 1 ) + (1 -A:+
1
-A$+
1
2,)
1 Kn+ 11’1 ( i n + 1 -h$+ 1 gnY*
[We have used the fact that (z,,- H,, QT= zT(Z- H, Hn+)to generate the second term in (8.2.2.2.).] Since (Z-HnH,+) H,, = 0, the second term vanishes. Since K,,+ is defined by the second half of (8.1.8.1) and since BnT(HnrH,,)B,, = B,,
[because B,, = (HnTH,,)+,which is symmetric] it follows that
WEIGHTED LEAST SQUARES
131
and
- h;f+ Q2/( 1+h;f+ B,,h,+
The third term in (8.2.2.2), reduces t o (C,+ and this proves the theorem. (8.3) Weighted Least Squares
The recursion (8.1.6) and (8.1.8) can be viewed as a relationship between the value of x which minimizes n
C Ctj-h;x)2
j= 1
and the value of x which minimizes
C
n+ 1
- hjTx)'.
((j
j= 1
Not surprisingly, a similar relationship holds between the values of x which respectively minimize
2 (ij-h:x)2/aj2
j= 1
n+ 1
and
j= 1
(Cj -h;x)/oj2.
(8.3.1) Exercise: Let g I 2 ,cZ2, ... be a sequence of positive scalars and let T,, be the vector of minimum norm among those which minimize cy= (Cj -hjTx)/cj2.Then I, = 0, (8.3.1.1)
I n + l
=
I n
+ Kn+I(Cn+1-hT+lzn)
where (8.3.1.2)
K,+l
&An+
=
,/hT+ 1 Anhn+ 1
8,, h,+ ,/(a:+
+h;f+ 8, h,+ A,
(8.3.1.3) A , + , =
IA ,
=
1)
if h , + , is not a I.c. of hl?...>hn otherwise,
I,
(An/zn+l)(Anhn+ h;5+1 A h , + 1
if h,, is not a 1.c. of hl,...,h" otherwise,
132
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
B”,
=
0,
Denoting the associated residual error by En( =C;= =
(Cj- hjT2,J2/aj2),we have
0,
if h,+l is not a I.c. of h , ,..., h,
h
For each n, (8.3.1.6)
A,
=
I - H,+H,
and
8, =
(8.3.1.7)
)+
C hjhT/aj2
(j:,
=
(HnTV,-2H,)+
where
[Hint:
Let
tj = Cj/aj, hj = hj/crj and notice that 2 ’ ( h l ,h,, ...,h,)
=
9 ( h l , h2,..., h,)
and n
n
j= 1
j= 1
1 (Cj-hjTx)2/aj2 = 1 (rj-L;x)’.
Apply (8.1.6)-(8.1.8) to the hi’s.]
rj and hj’s and then translate back to Cj’s and
RECURSIVE CONSTRAINED LEAST SQUARES, I
133
(8.4) Recursive Constrained Leaqt Squares, I
In (3.12.4b) we showed that (8.4.1)
x0 = G'u
+ [H(f-G+G)]+(z-HG+u)
is the vector of minimum norm among those which minimize (8.4.2)
112-
Hxll
subject to the constraints (8.4.3)
Gx = u
provided the constraint set is nonempty.
el, cy=l
Imagine a "stream" of data, c2, ..., en, ..., and denote the vector of n observations by z, and let x,," be the minimum norm vector among those (cj-h?x)', subject to Gx = 0: which minimize I ~ z , - H , , x ~=/ ~
xno = G+U + R,+Z,
(8.4.4)
where R,, = H,(I-G'G)
is the matrix whose j t h row is
hjT = [ ( f - G + G ) h j l T
( j = 1,2, ..., n)
and Z, = z, - H, G + u is the n vector whose j t h component is
c. = c. - hTG+u J
J
The recursion for 12, computation of
( j = 1,2,..., n).
J
= Hn+z,, can
xn0 = G + u (8.4.5) Theorem:
where
be applied almost verbatim to the
+ R'Z,.
Let xoo = G+u, x,"+ = x,'
+K,+
(c,+
- h;f+ x,,")
where
J,,= f- R,,+R,
and
B,
=
(RnTEn)+
can be computed recursively by (8.1.8.2) and (8.1.8.3) with h's replaced by h's throughout.
134
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
Proof: Let yn0 satisfy the same recursion as above except that yoo = 0. Then by (8.1.6), yn0 = R,'.Z,, for every n . If we set d,, = xn0-yno, then d,, satisfies
But do = zoo= G'u and hjTdo= hjT(I-G'G)G ' u
d,,,, = do
=0
so that
for all n
which proves that x,"+ = y,"+ +do : Thus, x:+~ = G'u
+ R,,'Z,.
If the previously mentioned recursion is used to process data in "real time," one precomputes G'u and I - G'G. ASeach datum C j is obtained, it is transformed to = C j - hjTG+u and the corresponding regression vector is transformed to hi = ( I - G f G ) h j . The 2,'s and B,'s are computed by the ancillary recursions (8. I .8.2) and (8.1.8.3). The vector G'u can be computed by any of the methods of Chapter V or recursively: Let the rows of G be denoted by g I T...,gtT. , Define
rj
G,
rn = 1,2,..., k .
=
Use (8.1.6), (8.1.8.2), and (8.1.8.3) to compute G,'u, where urn is the vector consisting of u's first rn component, for rn = 1,2, ..., k . As a by-product, the A recursion will generate I-G'G. (8.4.6) Exercise: Let
and let h , , ..., h,, ... be a sequence of vectors having the same dimensionality as the g's. Let
hj
=
(I-C'C)h,
( j = 1,2, ...).
Show that for any n, h, is a linear combination of h , , h,, ..., 6,~(gl,...,g~,h~,...,~n-l)
(i.e., h, is a linear combination of h,, .. ., h,-
and the g's).
,if and only if
RECURSIVE CONSTRAINED LEAST SQUARES, II
135
(8.4.7) Exercise: Supposefe Y ( g ,,...,gk, h, ,...,A,,): Then hjhT
+ A - 2 C gjgj k
j= 1
if and only i f f € Y ( g l ,...gk). (8.4.8) Exercise: Extend the results of (8.4.5) and develop a recursion for Z,’, the value of x which minimizes
subject to Gx = u.
(8.5) Recursive Constrained Least Squares, II In (8.4) we showed how constraints can be incorporated into a least squares recursion by altering the h-vectors and the observations, and by suitably initializing the x-vector. An essential feature of the technique is the necessity to specify the constraints in aduance of the data acquisition. Under certain circumstances, this is a drawback. Often, one wishes to collect all the data and afterward compute the least squares estimate subject to a succession of progressively more restrictive linear side conditions. By examining the residual errors [cf. (6.4)] one can judge the reasonableness of the constraints in the light of the available data. Or, one may wish to incorporate a few side conditions into the computation at the outset and then, after the data acquisition (or perhaps, even during) experiment with constraints. In this section we shall show that constraints can be incorporated into the recursion (8.3.1) by treating them as fictitious observations with zero variance. (8.5.1) Theorem: Let h , , h 2 , ... be a sequence of vectors, let 4‘1,52, ... be a sequence of scalars, and let 6 , c 2 2.., . be another sequence of nonnegative (but not necessarily positive) scalars. Let I,, K“,, B,, A , , and F,, be defined recursively as in (8.3.1).
’,
(a) If the aj2 are all positive, then for each n, 5, minimizes the weighted sum of squares
C (ti- h:x)2/cj2 n
j= I
B,
=
>+
C hjhjTlojZ
(j:,
136
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
and E,, =
C ([j-h:Z,,)2/aj2. n
j= 1
(b) If some of the oj2are zero, say if j~ S
=0
aj2(
if j $ S
>O
where S is a finite set of integers, then for each n, %,, minimizes
subject to the constraints hjTx = and En =
I
J
cj
j E S,,
C (cj - hjT%,,)'/aj2 €2,"
provided (hi;j E S } is a linearly independent set. (Here we define S,, = { 1 , 2,..., n} n S and T,, = S-S,,.) Proof: (a) We have merely restated the results of (8.3.1) for easy reference. (b) If some of the aj's are zero, (8.3.1) does not apply directly, since those results were derived, under the assumption of positive aj's. For this reason, we define oj2 if j # S Tj'(2)
=
'A
if j E S
and consider the recursions of (8.3.1) with aj2 replaced throughout by rji2(I). The A recursion does not depend on A, but the x , B, K, and E recursions do, so we display the relationship by denoting the outputs of these recursions by ,?,,().), Bn(L),Kn(l.),and .?,,(I.). We will show that 2,(A) and &(A) converge to the desired quantities' as 3, -+ 0 and that limA+o%,,(A) and limA40&(A) satisfy the stated recursions with B,(A)replaced by B,,= limA+o&,(A),E?,,(A) replaced by l i m A + o ~ , , ( land v ) z j 2 ( A ) replaced throughout by aj2 or 0 depending on whether j E Tior j E Sj. (1) Convergence of Z,,(l.) and &,(Ib)
By (8.3.1), ,?,,(A) is the vector of minimum norm among those which minimize n
([jj= 1
hjTx)2/.tj2( I ) .
* The constrained least squares estimate and the associated residual error.
RECURSIVE CONSTRAINED LEAST SQUARES, I1
137
Therefore (8.5.1.1)
I,(A) = A,'(A)?,(A)
=
[A,'(n)A,(~)]'A,T(n)?,(A)
where
In (7.1.10.2)-(7.1.10.10) we showed that if the equation Gx = u has a solution, and I@)
=
(F~F+A-~G~G)+(F~w+A-~G~u)
then limI(A) 1+0
=
xo
the vector of minimum norm among those which minimize I I w - F x ~ /subject ~ to Gx = u, and that Iim[/jw-FI(A)/j2
a+o
+ A - 2 //u-GI(A)[/~]= j j ~ - F x ~ I / ~
In the case at hand we let (8.5.1.3)
F be the matrix whose rows are {hjT/oj;j E T,}
and (8.5.1.4) (8.5.1.5)
G be the matrix whose rows are {h:;j
E S,}.
w and u are the vectors whose respective components are {Cj/oj;jE
Tn}
and
{Cj;jE S,}.
Then (8.5.1.2) takes the form I, (A)
=
[FTF+ A- 2GTG]+ [FTw+A- 2GTu].
The rows of G are linearly independent by assumption, hence the equation Gx = u has a solution, (3.12.5), hence lim Z,(A) = I, A+O
138
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
which minimizes C j e T (, ( j - h j T ~ ) Z / asubject j2 to the constraints hjTx = j E S, . Furthermore
(2) Convergence of
cj;
B, (A)
B,(jL)= [ f i n T ( 2 ) f i n ( 2 ) ] +(8.3.1.7), , which is of the form [FTF+A-’GTG]+ where F and G were defined earlier. By (4.9), limL-o (FTF+A-’GTG)+ always exists. So then does (8.5.1.6) B,, = IimB,,(A). a-o
B,, , (A) is related to 8,(1)through the recursion (8.3.1.4) with C J : + ~ replaced ,(A). If n + 1 E T,+, , then limL+,,zf+ ,(A) = a:+ ,and so B,,+, is related by ti+ to B, through (8.3.1.4). If ( n + 1) E S,+ ,, then limL-o z+: ,(I.) = 0. 8,+, will be related to B, through (8.3.1 -4) - with o,’+, set equal to zero if we can
rule out the possibility that AT,, B,,+lh,+l = 0 when h,+, is a I.c. of h , ,... Ih,. But if n + l ES,+, and h,+, is a 1.c. of h , ,..., A,,, then h:, I B,+ h,, = 0 if and only if
,
,
and by (8.4.7) this can only happen if h,+ is a 1.c. of { h j ; jE S,}. The last possibility is excluded because { h j ; , j €S } is required to be a linearly independent set by assumption. Thus
B,
=
limB,(I)
1-0
exists for each n and satisfies (8.3.1.4). (3) Recursions for in and En Since (8.5.1.7)
2, + 1 0.1 = 2, (1.1 + Rn + 1 (1.1CCn + 1 - A+:
1i n
and since
(]&)I
if h,,+t is not a 1.c. of h l , ..., h, otherwise (where o:+
, = 0 if n + I E S ) , we see that Pn+ 1
= 2,
+ Kn+ 1 I
CCn+
1-h$+
1 XnI
since all terms on the right side of (8.5.1.7) have limits.
RECURSIVE CONSTRAINED LEAST SQUARES, II
Similarly
lo
139
if A,+, is not a 1.c. of h, ,... h,
and since both sides tend to limits,
lo
if h n f l is not a 1.c. of h17
.**7
hn
Notice that h $ + 1 8 , h , + 1 > O if h,+l is a 1.c. of h l , ..., h,, and n + l E S , , + ~ (by previous argument).
(8.5.2) Exercise: Let 8, be as defined earlier, and let P,,be the projection on the linear manifold spanned by { h j ; j € S,}. Assuming the last to be a linearly independent set, show that
(8.5.3) Exercise: If the rows of H are linearly independent of the rows of G , then min I l ~ - H x 1 1 ~ = min I ~ Z - H X ~ ~ ~ Cx=u
X
(i.e., the minimal residual error for the unconstrained 1.s.e. is the same as the residual error for the constrained 1.s.e. in this case.)
(8.5.4) Exercise: Suppose the set { A j ; j Show that
-
B,+
E
S } is not linearly independent.
I
=
lim B,, (A)
A-0
=
-
B,
if h,+l is a 1.c. of { h j ; j € S,} and o,”+ = 0.
(8.5.5) (Continued): Let V,,
=
{x:h;x
= Cj;j E
S,}.
(a) Show that lim [ C n + 1 -h;f+1 x ~ ( A ) I ~ / c ~ , Z +
LAO
if n+ 1 E S,+
I(a)
+C+B n ( A ) 1
h,,, is a 1.c. of { h j ; jE S,,}, and V,+
An+
11 = 0
is not empty.
140
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
Hence, if the restriction in (8.5.1) that {hi; j E S } be a linearly independent set is dropped, and if we define where
Kn+1 =
1
zn+l
+ f?n+1(In+l-h%+lZn)
Anhn+ I A$+ 1 Anhn+ 1
= En
if h,+, is not a I.c. of h,, ...,h,
Bnhn+ 1
2 on+ 1
E”.+,
= gn
+A:+
otherwise
B n An+ 1
if is not a I.c. of h,, ...,h, or if n + 1 E S,+, and hn+ is a 1.c. of (hi;j E Sn}
0
+
1
[in+
(a?+I
1-A:+ +A:+
1 1
%,I2
Bnhn+
otherwise,
1)
ifh,.,
isa1.c. o f h , ,..., hn
if hn+, is not a 1.c. of h, ,..., h, E
S,+, and h,+
,is a 1.c. of {hi; j E S,}
otherwise (b) 2, is the vector of minimum norm among those which minimize T, subject to x E W,, provided %,‘ is nonempty. Furthermore, En is the associated residual error.
zjE(lj-Ajrx)’/oj’
(8.5.6) Example: In (4.3) we showed how to relate (HTjh)+ to HT+. Using the results of (8.5), we can go in the opposite direction: Given ( H T !h)’, we can compute HTf without having to “start from scratch.” The key to this procedure resides in the fact that the vector of minimum norm among those which minimize Ily-CvIj2 subject to the constraint gTu= 0 (where C = ( H T i h) [anm x (n+ 1) matrix] and gT = (O,O, ...,O, 1) [a 1 x (n+ 1)
RECURSIVE CONSTRAINED LEAST SQUARES, I1
141
row vector] is (8.5.6.1) This is a direct consequence of (3.12.4): i? =
C’y
where C = C(Z-gg+).
Since n l
C = ( H T i 0) and so
But the results of (8.5) tell us that the constraint gTv = 0
can be thought of as a “new” observation and that
cmfl = 0 with zero variance,
8 = c+y
can be updated in the light of the new observation according to (8.3.1.1)(8.3.1.4) :
B
=
G + Kc({,,,+ 1 -gT8)
=
8 - Kc(gT8)
where (I-CfC)g gT(I- c C)g
if g is not a 1.c. of the rows of C
(CTC)+g ST(CTC)+ 9
otherwise.
+
But g is a I.c. of the rows of C if and only if g is a I.c. of the columns of CT which is the same as g E %?(CT) so that g is a 1.c. of the rows of C if and only if c’cg
= g.
Since
C
= (HTih)
g = (0, ...,0 , l )
and
it follows that Cg
=
h
(3.7.10)
142
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
so that
C'Cg = C'h. Thus,
so that
(FiT) = ( I - K cg T ) ( H Ti h)+
(8.5.6.2) where
(Z-C+C)g gT(Z- c C ) g
if C + h # g
(CTC)+g ST(CTC)+ 9
otherwise
+
where C = (HT h ) and g = (O,O, ..., 0, 1). (8.5.6.2) is the desired result which relates H T +to (HT! A)+. Exercise [Continuation of (8.5.6)]
C'h = g if and only if ( I - H ' H ) h # 0. Kc =
[
if ( z - H ' H ) ~ = O
( g - C + h ) / ( l -hT(ccT)+h)
C +(CCT)+h/ll(CCT)+hII *
otherwise.
(8.5.8) Example Straight line regression performed recursively Suppose observations are collected which are of the form (8.5.8.1)
i,= cr+pn+ v,
n
=
1,2,3
where the v's are uncorrelated, zero mean random variables with common variance. CL and are not known and are to be estimated. Denote the least squares estimate (=BLUE) for x = (i),based on the first k observations by
%=( If we let
2)
k
=
1,2,3.
RECURSIVE CONSTRAINED LEAST SQUARES, I1
then 2k can be computed recursively via (8.3.1): Ao=Z
Bo=O
Since hz is not a multiple of h , ,
2,=0
143
144
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
and since h3 is a linear combination of h , and h, (h3 = 2h,-h,),
(8.5.8.2)
In Exercise (6.6.1 l), B3 and f13 are given in closed form:
and
61,
=
[-p 3 i
where
[= In the present case,
‘sj
=j
3 j= 1
cj
and
T
3
=
$1T
~ .
j= 1
so .5 = 2. Thus
and
= -45, + - - r-2 3 3 which agrees with (8.5.8.2). The covariance for ,t3is given by a 2 B 3 where and
o2 is
( A 3 = 0 since h , and h, span two-dimensional space.)
213 3 the variance of the v’s
RECURSIVE CONSTRAINED LEAST SQUARES, I1
145
(8.5.9) Adding a Constraint Suppose we now wish to compute the least squares estimate for x = )(; based on the first three observations, subject to the constraint p = 0. According to (8.5. I), this may be achieved by adding the fictitious observation c4 = 0 “pretending” at the same time that
where v4 has zero variance and
We apply (8.3.1) with c4’
(8.5.9.1)
24 = 2,
=0
and find that
+ K4(O-hqTg3)
which is what we should get if there is any justice.
(8.5.10) Incorporating the Constraint Using (8.4), we could carry the constraint right along and compute the least squares estimate for x based on one, two, and three observations, all subject to p = 0. The constraint /3 = 0 is the same as Gx = 0 where G = (0 I). Then
146
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
Therefore, h2 is a I.c. of hi as is h3 and
(~ ) (Ao h1)(A o hl)T (h?A oh l )2
which agrees with (8.5.9.1).
(8.6) Additional Regressors, II (Stepwise Regression) In (4.4) we discussed the problem of enriching a linear model by adding additional regressors and we derived a recursive relationship between the best m-dimensional regression and the best (m + 1)-dimensional regression in (4.4.4). That result is limited to the situation where regressors are added one at a time. This restriction is not essential. Indeed, the results of (6.6.14) can be applied to yield immediate results for the general case:
ADDITIONAL REGRESSORS, 11 (STEPWISE REGRESSION)
147
Suppose then, that a “preliminary” model of the form z = H1xl
(8.6.1)
+v
is analyzed, where H , is an n x p matrix, and v’s covariance is proportional to the identity matrix. The BLUE for x , would then be (8.6.2)
xl* = H,+z
with the associated residual error (8.6.3)
El
= IIZ-H1X1*112.
If E , were judged to be too large, an enriched model of the form
(3+
z = (H, i H2)
(8.6.4)
v
might be investigated, where H 2 is some suitably chosen n x q matrix. The BLUE for
would now be
(i;) i
(8.6.5)
=
(H,
H2)+Z
and the residual error would be (8.6.6)
El2 =
((z-H,sfl-H2a2((2.
If ( H , H , ) has rank p + q , we can apply the results of (6.6.14) to these computations: Letting Q , = I - H , H 1 + we find that Hz)+z
(8.6.7)
sfz =
(8.6.8)
sf, = H,+(z-H2S2)
(8.6.9)
EIZ
=
(Qi
~1
-
llQi H z f z l 1 2 .
(8.6.10) Exercise (a) If ( H , i H,) has full column rank then H2’Ql H 2 is nonsingular (b) In this case H2’Qi H2 sf2 =
and (c)
EIZ
=
~i
- (H2 92)’(Qiz).
HzTQiz
148
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
(8.7) Example Relationship between analysis of variance and analysis of covariance (Scheff6 [ 1, Section 6.31) : Let
(8.7.1)
and suppose a model for the observations of the form
(8.7.2)
j = 1 ,..., J i ; i = 1, ...,k,
6 . . = P. + vij I
I,
were assumed (one-way layout), where the Jij’s were uncorrelated zero mean random variables with common variance. The BLUE (=least squares estimate) for the vector of pi’s is unique and easily seen to be given by
pi* = ( J i ) - l 1cij 3 Q.
(8.7.3)
j
which may be verified by differentiating
C ij
(Cij
-
Pi)’
with respect to the pi’s, setting the results equal to zero and solving the resulting normal equations. The associated residual sum of squares is
(8.7.4)
=
~1
1 ij
(tj-5i.>2.
In vector notation, we write z = H,x, + V
where
The BLUE for x1 is
(8.7.5)
XI* =
( ’!* ) Pk*
=
H 1 + z=
( ‘!’) 5k
9
RELATIONSHIP BETWEEN ANALYSIS OF VARIANCE AND OF COVARIANCE
149
and (8.7.6)
El =
=
/ / z - H , x1*112 = l1(Z-H1 llQ1Z1l2
=
C ij
H,+)Zll2
(6j-Q*)2*
Equation (8.7.5) tells us that the operator H , transforms any vector z which is written in the double subscript form (8.7.1), into the vector H , + z whose components are obtained by averaging the components of z having the same initial subscript. Symbolically, +
(8.7.7) By the same token, (8.7.6) tells us that form (8.7.1) into a vector
Q1
transforms any vector z of the
(8.7.8) where (8.7.9)
[.. 1J
= [.. If - 5. 1 ' .
If c l is judged to be too large and the model is enriched by the addition of a regression term of the form y q i j (where y is unknown and the qij's are known), the model takes the form (8.7.10)
Gij
=
pi
+ yqij +
In the vector notation (8.7.1 1 )
z = H ~ X+,H
vij.
+v
2 ~ 2
[ :::1.
where H , is a matrix with one column (Le., a vector) and x2 has one component : (8.7.12) (8.7.13)
x2
H,
=
=
Crl
VkJk
150
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
Applying (8.6.7), we see that the BLUE for x2 is (8.7.14)
92 =
(Qi
Hz)+z.
By virtue of (8.7.8), (8.7.15) where q.. = q.. - q 1. ' 1J IJ
(8.7.16)
We know that for any matrix with one column, (8.7.17)
H2
+
=
(R2TR2>-* H 2 T
=
R2T//jR2112,
and so 92
= (~2Tz)/liH2112
or equivalently (8.7.18) The BLUE for the p's in the enriched model are found using (8.6.8): (8.7.19)
9, = H , + ( Z - H ~ ~ ~ ~ ) .
But (8.7.20) where
<*. = IJ
(8.7.21)
r.. - qq... IJ
IJ
By virtue of (8.7.7), the ith component of H , + z *is 6:. , so (8.7.22) Finally, the residual sum of squares E ~ 1 2
~ is ~obtained ,
li~-H1ti,-H,92I]~=
via (8.6.9):
~1 - T 2 ( l Q 1 H 2 ( ) 2 .
By virtue of (8.7.15), the last is equal to (8.7.23)
E,2
= El - 9'
c j;. c ij
=
ij
(%$-q2ij$).
MISSING OBSERVATIONS
151
(8.8) Missing Observations
In (6.6) we discussed the computational advantages of orthogonal designs when computations are carried out on a desk calculator. But the best laid plans of men for mice often go haywire and it is painfully common for something to foul up the experimental design during the course of the experiment (e.g., an experimental animal dies) thereby causing an experimental observation to be ruined, and destroying the orthogonality of the design. For example, suppose the experiment were planned in such a way that the observation vector z is of the form Z=HX+V where x is the (k-dimensional vector) parameter of interest. Suppose further, that the experiment had been carefully designed so that the columns of H would be mutually orthogonal and have unit lengths. Then the least squares estimate would be marvelously easy to compute : 2 = H'z = ( H T H ) - ' H T z = H T z
(since H T H = I if the columns of H are orthonormal). If a rat dies before the experiment is over, an observation is ruined. The actual vector of observation is now Z*
= H*x
+v
where H * is a maimed version of H : One row is missing so that H*'s columns are no longer orthogonal. It looks as though we are stuck with the job of psuedoinverting H * or something equally messy, right? Wrong! For the sake of concreteness, assume that the missing observation corresponds to the last one, so that if H i s an n x k matrix, H* is an (n - 1) x k matrix and
(where hT is the last row vector of H , the one that is missing from H * ) . By (8.5.6.2) [ ( H * ) + 01 where
=
HT(Z-gKT)
152
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
and ( I - H H +>g K = / gT(I- HH ' )g ( H H T )+ 9 g'(HH') + 9
if g is not a I.c. of the columns of H otherwise.
Since the columns of H (call them g,,g2,. . .,gk) are orthonormal, the question of whether g is a I.c. o f g , , ...,gk is easily settled: HH'g = HH'g
k
=
1 (gTg)gj
j= I
so that one need only compare g to the right side of the last equation to see whether or not (Z-HH+)g = 0. Notice also that (HH')' = HH', (3.7.3). Thus [(H*)+
01 = H'(Z-gK')
where K=(
(Z-HH')g/(l - IIH'gll')
if HHTg # g
HHTgI /IHH'g
if HH'g = g
II
( I - H H T ) g / ( l - IIHTgI12)
if HHTg # g otherwise.
= (Y
In this case, a very simple relationship exists between the pseudoinverse of the matrix H (whose columns are orthonormal) and the pseudoinverse of H*, the matrix obtained by deleting H's last row. The least squares estimate for x and the corresponding residual sum of squares for the "censored" data, z*, is thereby simply related to that which would have been obtained if the rat had not died:
II*
=
H*+z* = [(H*)'iO]z
(since
where [ is the missing observation) so where 2 =
2
- g(K'z)
MISSING OBSERVATIONS
153
is a modified observation vector which agrees with the uncensored sample z , in the first n components, but "pretends" that the missing observation was actually observed and had the value 4'- K'z. Since K'g = 1, it follows that
(Z-gKT)g = g(Z-KTg) = 0 so that
(Z-gKT)(z, -z2)
=
0
if z1 and z2 agree in their first n - 1 components. Therefore, Z does not depend explicitly on the choice of z's last component, (, and we can choose 4' = 0. This analysis can be generalized to the case of a missing observation for an arbitrary design.
(8.8.1) Theorem: Let
R
=
H'z
E =
I]Z-HR//~ X
=
-
and
H+Z
E = jjZ-uX1/2.
Then
X
(8.8.1 . I )
= (Z-BhhTM)R
where M = (
(8.8.1.2)
E" =
--Z/(l -hTBh)
if h is a I.c. of the rows of
B/ I1Bh I1
otherwise
E - (hT2)2/(l-hTBh)
i?
if h is a 1.c. of the rows of
i?
otherwise
Comment: 2 is the least squares estimate for x based on the full data model z = Hx+v,
and E is the associated residual sum of squares. If an observation is missing (say the last), then the observations are of the form
z"=ilx+i; where i? is obtained from H by deleting its last row. The new least squares estimate is 2 = fi+z and its residual sum of squares is E" = 1 z" -
11 '.
154
VIII
RECURSIVE COMPUTATION OF LEAST SQUARES ESTIMATORS
If more than one observation is missing, the recursion (8.8.1) can be iterated as many times as necessary. However, care must be exercised in defining = (A'R)' for all cases [cf. (8.8.2)], not just in the case where h is a 1.c. of A's rows. ?=
Proof of theorem: k = (
(a'
0)" = H+(Z-gkT)z where if h is a 1.c. of the rows of
( g - H T + h ) / l -hT(HTH)+h
HT+(WTH)'h/II(HTH)+h))' otherwise.
Since z's last component is zero, g'z
1
kTz =
=0
fi
(8.5.6.2) and (8.5.7)
so that
- hT2/l- hTBh
if h is a I.c. of
hTB15/1 Bh I/
otherwise.
A's rows
(8.8.1.1) follows once we observe that H'g = BHTg = B ( a T i h ) g = Bh since g has all zero components except the last. The formula for E' follows from (8.2.2). (8.8.2) Exercise: Let
lB-
(a)
=
1
B = (RTR)'.
Then
(Bh)(B2h)T+(B2h)(Bh)T
I1Bh II
+ hrB3h -(Bh)(Bh)T I/Bh I/
if h is not a I.c. of otherwise.
(b)
hTBh < 1
if h is a I.c. of A's rows.
(8.8.3) Example: Suppose
Let and
z=(:-).
G's rows
MISSING OBSERVATIONS
Then h = (l), B = ( H T H ) - '
= n-',
and
But 2
= [n/(n- l)]
2, so n2t2 n-1
-- -
hence
as it should.
(n-1):'
155
Chapter IX _____
~
N0 NNEGATIVE DEFl NITE MATRICES, CONDITIONAL EXPECTATION, A N D KALMAN FILTERING
(9.1) Nonnegative Definiteness It is well known that the following conditions are equivalent definitions for nonnegative definiteness of a symmetric matrix S, (Bellman [l]): (9.1.1) S (9.1.2)
= H H T for
some matrix H ;
xTSx2 0 for all vectors x;
(9.1.3) The eigenvalues of S are nonnegative; (9.1.4) There is a symmetric matrix R such that R2 = S. ( R is called the square root of S, and is denoted by the symbol, S”.) If S is nonnegative definite and nonsingular, it is said to be positive definite. In this case, S” is also positive definite and the inequality in (9.1.2) is strong for all nonzero x. The statement “ S is nonnegative definite” is abbreviated “ S 2 0.” Similarly, “ S > 0” means “ S is positive definite.” (9.1.5) Exercise: If S 2 0 and T 2 0 then S + T 2 0 with strict inequality holding if and only if N ( S ) nN ( T )= (0). One of the classical results concerning nonnegative definiteness, states that 157
1 58
IX NONNEGATIVE DEFINITE MATRICES, CONDITIONAL EXPECTATION
S is nonnegative definite (positive definite) if and only if the principal subdeterminants of S are all nonnegative (positive), (Bellman [l]). This result reduces the decision about S's nonnegativeness to the evaluation of a number of determinants. The material which we have developed in earlier chapters concerning pseudoinverses, can be used to prove some new results about nonnegative (positive) definiteness. (9.1.6) Theorem (Albert [l]):
Let S be a square matrix:
,
where S , is a symmetric n x n matrix and S,, is a symmetric m x m matrix. Then
,
(a) S 3 0 if and only if S , 3 0, S , S:, S12= SI2and
s,, - s T 2 s:, s 1 2 >, 0. (b) S > O i f a n d o n l y i f S , , > 0 , S , 2 > 0 , S l l - S 1 2 S Z ; I S T 2 > O a n d
s,, - s:,S,'sl2> 0. Proof: (a) (Sufficiency): If S 3 0 then S = HH', (9.1.1), where H has n + m rows. Write H a s a partitioned matrix:
H = "(X). m Y Then,
so that
S,, By (3.1 1.9),
=
XXT 3 0
and
s,,s:, = ( X X T ) ( X X T ) +
S,, =
=
XYT.
xx+
so that (S,, S;,)s,, = ( X X + ) X Y T = X Y T =
Finally, if we let
u = Y-s:,s:,x
then 0
< uuT= s,, - ST2s:, s,, .
s,,
NONNEGATIVE DEFINITENESS
159
(Necessity): Let n m
n m
v=rn(Oi~) x = S F l U
u=n(IiO) rn
Y = sT2s:, sl(;u
+ (S,, -s:, s:, S12)% v.
Since UVT = n o , we see that
(b) (Sufficiency): If S > 0 then by part (a), S , , 2 0. Furthermore, det S1 > 0 since all principal subdeterminants of S are positive. Hence S , is nonsingular and therefore S1 > 0. Similarly, S,, - ST, S;,' S,, 3 0 so that S,, >, 0 and by the same argument S,, > 0. The eigenvalues of S-' are the reciprocals of S's and so, S-' > 0 if S > 0, (9.1.3). We can therefore write n m
where the condition
ss-'
zio
= (0-)
dictates that
(S11and
5'12
S;; ST2)A
=
I
- s:, s;; Sl,) c = I.
(s22
A > 0 and C > 0 since S - '0. Hence A - ' > 0 and C - ' > 0 by the same argument as above. This proves sufficiency. (Necessity): From part (a), S 2 0. Let
(S1 - S1, S ',
A
=
B
= - S',
S12 (S22
ST,)-
- S T 2 5';; S12)-'
and
c = (S22- s;2s;;s12)-1. It is easy to show that
A - i B C - 1 = A-1 c-P,,
-~12~;21~12)-1s12sz;11~-1
160
IX
NONNEGATIVE DEFINITE MATRICES, CONDITIONAL EXPECTATION
so that B
=
-(S11
-S12S;;S1J1S12Sz;L.
Routine calculations then verify that
so S is nonsingular. Comment: In earlier chapters we pointed out that the class of covariance matrices coincides with the class of nonnegative-definite matrices. The preceding theorem tells us that the partitioned submatrices of covariance matrices have particular properties. This has a statistical interpretation which we will exploit in the section on conditional expectations. (9.1.7) Exercise: Let S , be an N x N symmetric matrix whose (i,j)th element is o ( i , j ) .Let Sk be the k x k submatrix occupying the upper left corner of S,, and let sk be the k-dimensional column vector (just to the right of &) with components o ( k + 1, l), o(k+ 1,2), ..., o(k+ 1,k). Define So = 0 and so = 0. Then
(a) S , > O i f a n d o n l y i f S k ~ , S , + _ , s k ~ , = s ~ ~ , and a ( k , k ) 3 s l - ] S t - , s k- I
for
k = 1,2,..., N .
, ,
(b) S, > 0 if and only if o ( k , k ) > s l - , Sc- sk- for k = 1,2, ..., N . [ H i n t : Use (9.1.6) and proceed by induction. Notice that if s k > 0, then [Sk'.(I-xxT)Sk%] > 0 if and only if Z - X X ~ > 0 which is true if and only if xTx < 1. Apply to
x
= [G(k+
1, k + I)Sk]-"S,.
See Albert [l] for details.] (9.1 .S) Exercise (Recursionsfor k
Let
& + ) : Suppose Sk 1
O and
CONDITIONAL EXPECTATIONS FOR NORMAL RANDOM VARIABLES
161
Then (a) s k + > 0 if and only if (b) In this case
S k tk
= s k and
c(k
3 0.
(see Albert [l]). (9.2) Conditional Expectations for Normal Random Variables
A vector valued, normally distributed random variable x has a distribution which is completely specified by its mean, m,
E
&x
and its covariance S,,
=
8(x-mm,)(x-m,)T
Anderson [l].
It is known, Anderson [I], that if y = Gx and z = Hx, then the conditional distribution of y, given that z = z is, for each possible value of z [i.e., each z E B(H)] normal, and therefore specified completely by its conditional mean and covariance. We shall use the results of the last section to derive the key properties of conditional means and covariances in a streamlined fashion. (9.2.1) Theorem: Suppose
has a multivariate normal distribution with mean zero and covariance
Then (a) x has the same distribution as
162
IX
NONNEGATIVE DEFINITE MATRICES, CONDITIONAL EXPECTATION
where
s" x2* = s:, s:, xl* + (S,, -sT2s:, S12)%W2 xl* =
?I w1
and
"3) 4 w2
has a normal distribution with mean zero and covariance I . (b) The conditional distribution of x2 given that xI = x is normal with conditional mean m 2 1 1 ( .= ~ ) B(x21x, = x ) = S:,S:,x,
provided .Y E 3 ( S 1 The conditional covariance of x2 given that x1 = x is s2211
= G[(X2-m2,1(X))(X2-m211(X))TIX1 = s22 -
s:, s:1
=x]
s12
provided x E d ( S , ,). (Notice that S2,, does not depend explicitly on x.) (c) If x = "(x!) 4 x2
has mean value
which is different from zero, then the conditional covariance of x 2 given that x l = .Y is the same as in (b) and 1??21*(.Y) 3
R ( x , l x , =.u)
= I??,
+ST2S:l(s-m,),
provided .Y
- '77,
E
9(SIl).
Proojl (a) Since S is a covariance, it is nonnegative definite; so from (9.1.6), S , 2 0, S,, - S:, S:, S , 3 0, hence both of the aforementioned matrices have square roots. Part (a) follows since x* has a multivariate normal distribution with mean zero and (after a little algebra), covariance S. Notice that x,* (hence x , ) must lie in .H(SFl) = %(SI,) with probability 1. ( b) Without loss of generality, part (a) permits us t o assume that x 1 = xl* and x, = x 2 * when we are exploring any properties of x's distribution.
,
CONDITIONALEXPECTATIONS FOR NORMAL RANDOM VARIABLES
163
Since w1 and w2 are independent, so then are x 1 and w2 so that & (w2 1 x 1 = x) = & (w2) = 0. Thus, from part (a), if
x € w ( s p l )= B(Sl1) mzI1(x) = €(x21xl = x) = S:,S:,&(x,
Ixl
= x) =
ST2s:, x
and s2211
= &E(X2-m2,1(X))(X2-m2,1(x))TIXI =
=.I
€[(Tw~)(Tw~)~ =IX] x~
where T
=
( S 2 ,- ST, S:, S , 2)1/2.
Again, using the independence of w2 and x l , we see that ~ [ ( T W , ) ( T W , )= ~ )X]X ~= ~ “ ( T w , w , ~ T ~ ) =
T T= ~
( s , -s;, ~ s:, s12)
since &(W,W,T)
=
I.
(c) Apply (b) to x2- m 2 and x1-m1 : & ( x 2 - m 2 ~ x 1 - m 1= x - r n 1 > = ~ T ~ s : ~ ( x - m ~ )
so &(x, I x
= x) - m, =
ST2S:, ( x - m l ) .
The formula for the conditional covariance is unchanged if a constant vector is added to x Z - m 2 . Again, we must tack on the restriction that x - m , be in g ( S l in order that the formulas hold. (See Marsaglia 111.) The results of (9.2. lc) establish a relationship between the conditional mean and covariance of u given v = u and the unconditional mean and covariance of u for jointly normally distributed u and v. The same argument can be applied to establish a similar relationship between the conditional mean (and covariance) of u given v = u and y = y , and the conditional mean (and covariance) of u given v = u, for any jointly normally distributed vector random variables, v, y, and u: (9.2.2) Theorem: Suppose
164
IX
NONNEGATIVE DEFINITE MATRICES, CONDITIONAL EXPECTATION
has a multivariate normal distribution with mean m and covariance
and suppose
Let
(9.2.2.1)
mu = bu
m,
mzl,(u)
(9.2.2.2)
=
=
my = €y
&v
€(zlu
= u)
mVl,(u) = €(v[ u = u) my,&)
=
€(Y
lu = 4
and
(9.2.2.3)
S"V1U
szz,,
and
SYYlll
be the respective conditional covariances of z, v, and y given u = u. Then
(a) If u- m uE .B(S,,,,), the conditional distribution of z, given that u = u, is normal with mean
and covariance
(9.2.2.5)
=
~
z
z
~
~
z
~
~
~
(b) If u-mu E B(S,,,,) and Y - m y l u ( 4 E =wSyyIu)
then the conditional distribution of v given that u = u and y with mean
(9.2.2.6)
m v l u , y ( w )= m v l , ( 4
S",,,,,
is normal
+~ v y ( u ~ y + y y l u ( Y - ~ y , u ~ ~ ~ ~
and covariance
(9.2.2.7)
=Y ,
= SV",, - Svy,,
which does not depend explicitly on u and y.
S,',lU
Sv",I,
u
~
~
CONDITIONAL EXPECTATIONS FOR NORMAL RANDOM VARIABLES
165
Comment: ,Notice the formal similarity between (9.2.2.6) and (9.2.2.4) on the one hand and between (9.2.2.7) and (9.2.2.5), on the other. Proof: Part (a) follows from (9.2.1) and from the fact that z can be partitioned in the form
(b) Let u be fixed and a possible value for the random variable u [i.e., u--171, E 9?(Snn)].The conditional joint distribution of v and y given that u = u is normal with mean (9.2.2.8) and covariance (9.2.2.9) where the elements of mz,,(u) and S,,,, are given in part (a). For fixed u, let (9.2.2.10) be a random variable which has a normal distribution with (unconditional) mean (9.2.2. I 1)
mz* = mzln(U>
and (unconditional) covariance (9.2.2.12)
&*z*
= SZ,,,.
Then for fixed u, the joint (unconditional) distribution of v* and y* is the same as the joint (conditional) distribution of v and y, given that u = u. Therefore, the conditional distribution of v*, given that y* = y is the same as the conditional distribution of v given that y = y and u = u provided that y is an allowable realization of y* [i.e., y - m y , E 9?(Sy,y*)]. So, for each y with y - m y . E 9?(SYey*), the means and covariances of the two conditional distributions must coincide: (9.2.2.13) (9.2.2.14)
mvln,y(U,Y) = m v * l y * ( ~ ) Svvln,y = Sv*v*ly*.
From part (a) (9.2.2.15)
mv*ly8dy) = mv8
+ SyTIV.Sy+ryl(y-myt)
166
IX
NONNEGATIVE DEFINITE MATRICES, CONDITIONAL EXPECTATION
and (9.2.2.16)
s,*,*/,* = s,,,.
-
SV.,’SY”,’
s:*,*
Since (9.2.2.11)
(9.2.2.17) and
(9.2.2.12) (9.2.2.6) follows from (9.2.2.13), (9.2.2.15), and (9.2.2.17) while (9.2.2.7) follows from (9.2.2.14), (9.2.2.16), and (9.2.2.18). 1 (9.2.2.19) Exercise: If
have a joint normal distribution and z is independent of
then Sxzly = 0. (9.2.3) Exercise: (a) For any matrices of the right size, tr(AB)
=
tr(BA).
(b) If x is a random variable with covariance S,,, then &JIxJJ2 = tr(Sxx). (9.2.4. I ) Exercise (Optimum properties of conditional expectation and rvide-sense conditional expectation) If
is a vector random variable with covariance matrix
we say that m, + S,, S:, ( y - my) is the wide-sense conditional expectation of x given y = y . (If x and y have a joint normal distribution, the wide-sense conditional expectation coincides with the usual conditional expectation.)
CONDITIONAL EXPECTATIONS FOR NORMAL RANDOM VARIABLES
167
(a) Assume that m, and my are both zero. Show that &ljx-Ayl12 2 6 l ~ ~ - S , , S ~ , yfor 1/~ any matrix A of the right size, with strict inequality holding unless AS,, = S,, . (b) If
has a joint normal distribution and g ( - ) is a measurable mapping from qdimensional Euclidean space to p-dimensional Euclidean space, then
6llg -9 (Y) I1 3 8 IIX - s,, s;, Y I\ with strict inequality holding unless g ( y ) = S,, Slyy for almost all y . Comment: (a) Shows that the wide sense conditional expectation is the best linear predictor for x, based on y when mean square error is to be minimized. (b) Shows that the wide sense conditional expectation is the best predictor of all (linear or not), in the normal case. (9.2.4.2) Exercise: Suppose
Then u E S,, and y - ST, S,'.u
E
&?(Syyl,)if and only if
z = (l-)
E
9(Sz,).
Comment: The last result explains away the apparently asymmetric conditions on u - m u andy-m,,,,(u) which are necessary in order that (9.2.2b) should hold. In short, (9.2.2.6) and (9.2.2.7) hold provided
(9.2.5) Exercise: Suppose
I68
IX
NONNEGATIVE DEFINITE MATRICES, CONDITIONAL EXPECTATION
has a multivariate normal distribution. Let
%*ly(Y)
= &(X*lY
=y)
m, = #(w)
Sx*x*ly = d[(x*-m,*(,(y))(x*-mx*,y(y))TIY = Y l
and
s,
= Q (w- m,)
(w- r n , ) T .
If (9.2.5.1)
x = Qx*
+w
(where @ is a deterministic square matrix)
and if w is independent of
then (9.2.5.2) (9.2.5.3)
m,ly(Y> = @mx*Iy(Y) + mw
Sxxry=
@s,*x*]yQT + sww.
(9.2.6) Exercise: If v has a normal distribution with mean 0 and covariances S,, and is independent of
which has a normal distribution, and if (9.2.6. I )
z=Hx+v
where H is a deterministic matrix, then (9.2.6.2)
+
n7,1y,(~,z) = n ? , l Y ( ~ ) SxxlrHT(HSx,IyH T + S y v ) +(z-Hm,l,(y))
and where n i x , y , z ( y , zis ) the conditional expectation of x given that y
=y
and
z = z , Sxxly,z is the conditional covariance of x given y = y and z = Z , etc.
Comnirtit: (9.2.6) shows how the acquisition of new data changes the posterior distribution. l f y represents all the data u p to the last sampling period, then the posterior distribution of x (i.e., conditional distribution) given the
data y = y, contains all of the available information about x that can be learned from analyzing z. This distribution, in the normal case, is characterized
KALMAN FILTERING
169
by its mean and covariance. If a new datum of the form (9.2.6.1) is observed, then the posterior distribution of x changes and is now characterized by its conditional mean and covariance given y and z. (9.2.6.2) and (9.2.6.3) show how the new mean and covariance are related to the old.
(9.3) Kalman Filtering Suppose {x,; n = 0,1,2, ...} is a vector valued stochastic process whose evolution is governed by a so-called “state equation” of the form (9.3.1)
Xn+1 = @ n X n + W n
where {w,; n = 0,1,2, ...>is a sequence of independent vector valued normal random variables with means 0 and covariances (9.3.2)
T,, = bw,,wnT.
The so-called “state variable” can be partially observed in the presence of noise, the observations being of the form (9.3.3)
2,
=
H,x,+ v,
n
=
1,2, ...
where {v,; n = 1,2, ...} is a sequence of independent vector valued normal random variables with zero means and covariances (9.3.4)
R, = &v,,v,,~.
The {v,} and {w,} processes are assumed to be independent,’ and the initial state xo is assumed to be a normally distributed zero mean random vector which is independent of the v,’s and w,’s. Under these assumptions, it is not hard to show that the joint distribution of x o , x , ,..., x,, 1, z l , ...,z, is normal. In (9.2.4.1), it was established that the best mean square predictor for x,, based upon the observations 2,
=
z1, 2 2
=
z2,
... ,z,
=
z,
is (9.3.5)
x,,,
= Q(x,lz,
= Z],
..., z,
= z,)
where we adopt the convention that XO l 0 =
0.
This assumption can be dispensed with at the expense of slightly more complicated formulas.
170
IX
NONNEGATIVE DEFINITE MATRICES, CONDITIONAL EXPECTATION
Similarly the best predictor for x,+ (9.3.6) Z,
.Y,+
= 8(x,+
11,
given the first n observations, is 1
I z1 = Z ] , ...)z,
= z,).
I f we denote the conditional covariances of x, and x,+], given z1 = ,..., Z, = z, by S,,, and S,+Iln respectively, then by virtue of (9.2.5) and
(9.2.6) (9.3.7)
xn+l,n=
(9.3.8)
r n + 1In+ 1
=
n = 0,1,2, ..
@,.Y,,, xn+ 1 ~n
+ K +1 (zn + 1 - H n +
1
xn+ 1 I n )
where (9.3.9)
~
(9.3.10)
n 1+=
sn + 1I n
1 (Hn+1 s n + 1 In
H,’+ 1 + Rn+ I ) +
Sn+lln= @,S,l,,@/-t T,
and (9.3.111
s,.,,,.,
=
Sn+l(,s,+,I,H:+I
x
( H n + 1 sn+ 1In
H,’+
1
+ Rn+ l)+Hn+1 sn+ 11.,
These relations are initialized by .Yolo
(9.3.12)
=0
so,,= &xoxo‘.
[These recursions follow instantly from (9.2.5) and (9.2.6) if we identify
y with the “super vector”
,
,
z with z,+ , x* with x,, and x with x,+ .] The relations (9.3.7)-(9.3.12) are the famous Kalman “filtering equations,”
Kalman 113, about which an enormous literature has grown up in the systems engineering journals. The practical importance of these recursions is best exemplified by Battin [I], Battin and Levine [I], and American Statistical Association [ I]. C O ~ W J K WThe ~ S :restrictions that xo and the state equation noise have zero means is not essential. If xo has mean /no, then the recursion (9.3.7) is reinitialized by xol0= m 0 . I f the state equation noise, w,, has mean m,, then (Y.3.7) becomes
LSE‘S
THE RELATIONSHIP BETWEEN
AND CONDITIONAL EXPECTATIONS
171
The key feature of the Kalman equations are their recursive nature. The stream of data z1,z2,... is incorporated “in real time” to produce a stream of estimates for the current values of the state variable. The normality assumption can be dropped, in which case, the recursions generate wide-sense conditional expectations.
(9.4) The Relationship between Least Squares Estimates and Conditional Expectations Suppose
z = HX+V
(9.4.1)
where
has a normal distribution with mean 0 and covariance
The conditional expectation of x given z = z is IM, I ( z ) = 1.-
(9.4.2)
’C ’HT(1- HC ’IfT+I )- z .
By (4.9.5), (9.4.3) (i-+I) HC’H~
+
=
+
I - (HC2HT)(HC2HT)+ A 2 ( H C 2 H T ) +
+ O(A4) =
as
A -+ O
I - ( H C ) ( H C ) ++ i.’(HCZNT)+ + 0(1-4)(3.11.7)
so that m,,,(z)
+ C ( H C ) T ( H C 2 H T ) + Z+ o(i.2)
= (Cji.2) [ ( I - ( H C ) ( H C ) + ) ( H C ) ] T z =
C ( H C ) + z+ o@’)
as i. 0. --f
But C ( H C ) + zminimizes l ( z - H x l J 2 subject to the constraint x E 2 ( C ) since min Ilz-H,ul12 = min l ~ z - ~ C y ( I *
xt..A(C)
Y
occurs when x = C j and j = ( H C ) + z .Thus, for small values of I-, 8(x 1 z = z ) is close to the constrained least squares estimator for x [subject to the constraint ( I - CC+)x = 01.
172
IX
NONNEGATIVE DEFINITE MATRICES, CONDITIONAL EXPECTATION
If C is nonsingular, Z-CC' = 0 and so in this case, to the naive least squares estimate for x.
&(XI
z = z ) is close
(9.4.4) Example: Suppose {vn; n = I , 2, .. . ] is a sequence of independent normally distributed random variables with mean zero and variances a,,'. Suppose, x is a zero mean p-dimensional vector random variable having a normal distribution with mean 0 and covariance (l/I.')C. Assume that x is independent of the v,'s. Suppose observations of the form
6,
= h,TX+
vn
n
=
1,2,..
are made in order to estimate x . Let 2,,(1.) be the conditional expectation of x given Cl = [t,...,C,, = [,, and let &,(EL) be the conditional covariance of x, given the data up through the nth. Then the Kalman recursions reduce to
&(/I) =0
&I+ 10.)
s,(n) = K 2 C ,
= Sfl(4 -
hn+ 1) ( s n (1,) An+ 1)' 2 a"+1 +A$+ 1 sfl(4hfl+1
(sn
O L )
'
Compare this with the recursions (8.3.1) and (8.4.5). [Read B, for S,,(I.).] When i. is close to zero, the present recursions generate a sequence which is close to 4, the least squares estimate for x based on l I ,...,in computed subject to the constraints ( I - C C ' ) x = 0. (See Albert and Sittler [l].)
REFERENCES
Albert, A. [ I ] Conditions for positive and nonnegative definiteness i n terms of pseudoinverses. S I A M J . Appl. Math. 17 (l969), 434440. Albert, A., and Sittler, R. [ I ] A method for computing least squares estimators that keep up with the data. SIAM J . , Control 3 (1965), 394417. American Statistical Association [ I ] Regression procedures for missile trajectory estimation. Proc. of the 105th Regional Meeting, Florida State Univ. (1965). Anderson, T. W. [ I ] “An Introduction to Multivariate Statistical Analysis.” Wiley, New York, 1958. Battin, R. H. [ I ] “Astronautical Guidance.” McGraw-Hill, New York, 1964. Battin, R. H., and Levine, G . [I] Application of Kalman filtering techniques to the Apollo program. MIT Inst. Lab. Tech. Rep. E2401, April 1969. Bellrnan, R. [I] “Introduction to Matrix Analysis.” McGraw-Hill, New York 1960. Ben-Israel, A. [ I ] On error bounds for generalized inverses. SIAM J. Numer. Anal. 3 (l966), 585-592. Ben-Israel, A., and Charnes, A. [ I ] Contribution to the theory of generalized inverses. S I A M J. 11 (1963), 667-699. [2] An explicit solution of a special class of linear programming problems. Operutions Res. 16 (1968), 1166-1175. 113
174
REFERENCES
Ben-Israel, A,, Charnes, A., and Robers, P. D. [ I ] On gene/-alized inverses and interval linear programming. In “Theory and Application of Generalized Inverses” (T. Boullion and P. Odell, eds.). Proceedings of a symposium at Texas Technological College, March, 1968. Ben-lsracl, A. and Ijiri. Y . [ I ] A report on the machine calculation of the generalized inverse of an arbitrary matrix. O N K Rc.vc,crr-ch .Mcj/mr N o . 110, Carnegie Inst. Of Tech.. March 1963. Ben-lsr-acl, A., and Robers. 1’. D. [ I ] A suboptimization method for interval linear programming. S~.stetn.~ Res. Memo No. 204. Northwestern Univ., June 1968. Ben-lsracl, A,. and Wersan, S. J . [ I ] An elimination method for computing the generalized inverse for arbitrary complex . c r t . / i . 10 (1963). 532-537. matrix. J . Assoc. C o t ~ t p i i t M Bocher, M. [ I ] ”Introduction to Higher Algebra.” Macmillan, New York, 1907. Boullion, T. and Odell. P. (eds.) [ I ] “Theory and Application of Generalized Inverses.” Proceedings of symposium at Texas Technological College, March 1968. (21 “Generalized Inverse Matrices.“ Wiley (Interscience), New York, 1971. den Broeder. G. G . , and Charnes, A. [ I ] Contributions to the theory of generalized inverses for matrices. O N R Res. Memo Xo. 39. Northwestern Univ., 1962. Butler. T.. and Martin. A. V. [ I ] On a method of Courant for minimizing functionals. J . Math. Ph!.s. 41 (1962), 291-299. Cline, R. E. [ I ] Note on the generalized inverse of the product of matrices. S l A M Rev. 6 (1964), 57-58. [I] Representations for the generalized inverse of a partitioned matrix. S I A M J . Appl. :!.lLiI//. 12 (1964), 588-600. [3] Representations for the generalized inverse of sums of matrices. S l A M J . Nunier. Anal. 2 (1965). 99-1 14. Deccll, H. P. [ I ] An alternate form of the generalized inverse of an arbitrary complex matrix. S I A M R r i . 7 (1965). 356-358. [2] An application of the Cayley-Hamilton Theorem to generalized matrix inversion. SIAM Rcr. 7 ( 1965), 526-528. Decell. H. P., and Odell, P. L. [ I ] On the fixed point probability vector of regular or ergodic transition matrices. J . Am. Stcit. A s ~ o c 62 . (1967), 600-602. Fadecv, D. K., and Fadeeva. V. N. [ I ] “Computational Methods of Linear Algebra.” Freeman, San Francisco, 1963. Feller, W. [ I ] “An lntroduction to Probability Theory and Its Applications,” 3rd ed., Vol. 1. Wiley, New York, 1952. Goldnian, A. J., and Zelen. J. [ I ] Weak generalired inverses and minimum variance linear unbiased estimation. J . Kes. Not. Bur. Standards Sect. B, 68B (1964j, 151-172.
REFERENCES
175
Golub, G. [ l ] Numerical methods for solving linear least squares problems. Nunier. Muth. 7 ( 1963, 206-2 16. [2] Least squares singular values and matrix approximations. Apl. Mat. (Prague) 13 (1 968), 44-5 1. Golub, G., and Kahan, W. [ I ] Calculating the singular values and pseudo-inverse of a matrix. SIAM J. Nunier. Anal. 2 (1965), 205-224. Good, I. J. [ I ] Some applications of the singular decomposition of a matrix. Terhnonwtrirs 11 ( I 969). 823-83 1. Greville, T. N. E. [ I ] The pseudoinverse of a rectangular matrix and its applications to the solution of systems of linear equations. SIAM Reu. 1 (1959), 38-43. 121 Some applications of the pseudoinverse of a matrix. SIAM Rev. 2 (1960), 15-22. [3] Note on the generalized inverse of a matrix product. S I A M Rev. 8 (1966), 518-521. [Erratum 9 (1967).] Halmos, Paul R. [ I ] Finite dimensional vector spaces. Ann. ( $ M a t h . Stirdies 7 (1955). Kalman, R. E. [I] A new approach to linear filtering and prediction problems. J . Basic Engrg. 82 (1960), 35-45. Karlin, S. [I] “Mathematical Methods and Theory in Games Programming and Economics.” Addison-Wesley, Reading, Massachusetts, 1959. Kruskal, W. [ I ] When are the Gauss-Markov and least squares estimators identical? A coordinate free approach. Ann. Math. Statist. 39 (1968), 70-75. Marsaglia, G . [ I ] Conditional means and covariances of normal variables with singular covariance matrices. J. A m . S/nt. Assoc. 59 (1965), 1203-1204. Mitra, S. K., and Rao, C. R. [ I ] Conditions for optimality and validity of simple least squares theory. Ann. Math. Statist. 40 (1968), 1617-1624. Moore, E. H. [ I ] Abstract. Bull. Amer. Math. Sor. 26 (1920), 394-395. 121 General analysis, Part I. Memoirs Amer. Philos. Sac. 1 (l935), 1-23 1. Noble, B. [ I ] A method for computing the generalized inverse of a matrix. S I A M J . Numer. Anal. 3 (1966), 582-584. Osborne, E. E. [ I ] Smallest least squares solutions of linear equations. SIAM J. Numer. Anal. 2 ( I 965), 300-307. Penrose, R. [I] A generalized inverse for matrices. Proc. Cambridge Philus. Soc. 51 (1955), 406-4 1 3. [2] On best approximate solutions of linear matrix equations. Proc. Cambridge Philos. SOC.52 (1956), 17-19.
176
REFERENCES
Pereyre, V., and Rosen, J . B. [ I ] Computation of the pseudoinverse of a matrix of unknown rank. Computer Sciences Division, Stanford Univ., Tech. Rep. CS 13, Sept. (1964). Price, C. M. [ I ] The matrix pseudoinverse and minimal variance estimates. SIAM Rev. 6 (1964), 115-120.
Pyle, L. D. [ I ] Generalized inverse computations using the gradient projection method. J . Assoc. Co/qm/. Mach. 11 ( I 964), 422429. Rao, C. R. [ I ] A note on a generalized inverse of a matrix with applications to problems in mathematical statistics. J . Roy. Sturist. Soc. Ser. B, 24 (1962), 152-158. Rao, C . R. and Mitra, S . K. [ I ] “Generalized Inverse of Matrices and Its Application.” Wiley, New York, 1971. Rust, R., Burrus, W . R., and Schneeberger, C. [ I ] A simple algorithm for computing the generalized inverse o f a matrix. Comm. A C M 9 (1966), 381-386. Scheffe, H. [ I ] “The Analysis of Variance.” Wiley, New York, 1959. Stewart, G. W. [ I ] On the continuity of the generalized inverse. SIAM J . Appl. Math. 17 (1969), 33-45. Tewarson, R. P. [ I ] A direct method for generalized matrix inversion. SIAM J . Numer. Anal. 4 (1967), 499-507. [2] On two direct methods for computing generalized inverses. Computing, 7 (1971), 236-239. Watson, G. S. [ I ] Linear least squares regression. Ann. Math. Statist. 38 (1967), 1679-1699. Zyskind, G. [ I ] On canonical forms, nonnegative covariance matrices and best and simple least squares linear estimators in linear models. Ann. Math. Statist. 38 (l967), 1092-1 109. Zyskind, G., and Martin, F. B. [ I ] On best linear estimation and a general Gauss-Markov theorem in linear models with arbitrary nonnegative covariance structure. S l A M 1. Appl. Math. 17 (1969), 1190-1202.
Numbers in italics refer to the pages on which the complete references are listed. A Adjoint, 10 Albert, A., 158, 161, 172, 173 Analysis of variance, 108 table, 109 Anderson, T. W., 94, 161, 173
B Battin, R. H., 170, 173 Bellman, R., 5, 12, 75, 158, 173 Ben-Israel, A,, 33, 42, 65, 69, 74, 173, 174 BLUE (best linear unbiased estimator), 88 ff., 105, 126, 148ff. and constraints, 122 for estimable vector parametric functions, 97 ff. and naive least squares, 92 and stepwise regression, I 1 7 and straight line regression, 11 3 and two-way layout, 116 Bocher, M., 78, 174 Boullion, T., 174 Bounded linear program, 33 den Broeder, G. G., 20, 174 Burrus, W. R., 57, 176 Butler, T., 120, 174
C
Cayley-Hamilton theorem, 75 Characteristic polynomial, 76 computation of coefficients, 76-77 Charnes, A., 20, 33,42, 74, 173, 174 Chi-square distribution, 94 ff., 98 ff. 109 and projections, 97, 100 Cline, R. E., 49, 55, 174 Cochran’s theorem, 52 Computational methods, based on Cayley-Hamilton theorem,74 ff. based on Gauss-Jordan elimination, 65 ff. based on gradient projection, 69 ff. based on Gramm-Schmidt orthogonalization, 57 ff. Conditional covariance, normal case, 161 recursion for, 164 Conditional distribution, normal case, 161 Conditional expectation, and least squares, 171 as optimal mean square predictor, 166 for normal case, 161 recursion for, 164 Confidence ellipsoids, 97 ff. and tests of general linear hypothesis, 102 Constrained least squares, 31, 121 recursive computation of, 133 177
178
INDEX
Constraint set for linear program, 33 Constraints, as fictitious observations, 135 Covariance of BLUE, recursive computation of,127
D Decell, H. P., 38, 74, 174 Degrees of freedom, 94 ff., 98 ff,, 109 Dependent variable (in regression), 86 Diagonalization theorem, 12, 27 Differential correction, 127 Dimension of linear manifold, 52 Dominant eigenvalue algorithm, 42 E Eigenvalue, 12, 23, 24, 38 ff. Eigenvector, 24, 38 ff. Estimable, 88 Estimable parametric function, 88 ff. Estimable vector parametric functions, 97 ff. EVPF (estimable vector parametric function), 97 ff.
F
F distrihution, 94 ff., 99 ff., 101- I 02 Fadeev, D. K . , 74, 76, 174 Fadeeva, V. N., 74, 76, 174 Feasible linear program, 33 Feller, W.. 37, 174 Fourier expansion, 8 G Gauss-Jordan elimination, 65 Gauss-Markov estimator, 97 Gauss-Markov theorem, generalized, 90 General linear hypothesis, tests of, 100 ff. General linear model, 87 Goldman, A. J., 123, 174 Golub, G., 42, 81, 175 Good, I . J . , 38, 175 Gramm-Schmidt orthogonalization, 9, 57, 58 ff., 70 ff., 112 modified, 81 transformation expressed as a matrix, 60 Grammiam, 12 Greville, T. N . E., 43, 53, 175
H Halmos, P. R., 5, 175
I Ijiri, Y., 69, 174 Independent variable (in regression), 86 Inner product, 5 Inverse, 12 K Kahan, W., 42, 81, 175 Kalman, filtering, 169 ff. Kalman, R. E., 170, 175 Karlin, S . , 5, 175 Kruskal, W., 93, 175
L Least squares estimator, recursive computation of, 125 ff. Least squares, and conditional expectation, 171 generalized, 93 minimum norm solution to, 15, 17 Levine, G., 170, 173 Likelihood ratio test, 100, 104 Linear equations, 30 Linear manifold, 6 ff. definition of, 5 Linear programming, 33 Linearly independent, 12, 21 M Markov chain, 37 Marsaglia, G., 163, 175 Martin, A. V., 120, 174 Martin, F. B., 90, 123, 176 Matrix equations, 35-36 Missing observations, general case, 153 in orthogonal designs, 151 Mitra, S., 93, 175, 176 Modified Gramm-Schmidt orthogonalization, 81 Moore, E. H., 3, 20, 175 Moore-Penrose generalized inverse, definition of, 20 Multiple correlation coefficient, 46 N Naive least squares and BLUE, 92 Noble, B., 65, 175 Noncentrality parameter, 94 ff., 97 Nonnegative definiteness, 157 ff.
INDEX
Nonsingular, 12 . Norm, 5 Normal equations, 17 Normal matrix, 36 Normal random variables, 94 Null space, 10 ff., 15
0 Odell, P. L., 38, 174 Orthogonal, 5 to a linear manifold, 6 Orthogonal complement of a linear manifold, definition of, 1I Orthogonal design, 104 ff. and constrained least squares, 105 and projections, 106 and sum of squares decomposition, 108 Orthogonal matrix, 12, 36 Orthonormal, 24 Osborne, E. E., I75 P Parametric function, 87, 88 Partial correlation coefficient, 46 Partial isometry, 41 Penalty functions, 119 ff. Penrose, R., 20, 42, 175 Penrose conditions, 28 Permutation matrix, 57 Perturbation theorems, 50 ff. Pereyre, 8 1, I76 Positive definiteness, 157 ff. Price, C. M., 93, 176 Probability vector, 37 Projection, 8, 16 ff., 18 ff. on null space of matrix, 20, 42, 47 on range of matrix, 20 recursion for, 37 Projection matrix, definition of, 26 eigenvalues of, 26 Projection theorem, 6, 11 Projections and pseudoinverse, 20 Pseudoinverse, and constrained least squares, 31 definition of, 19 discontinuous nature of, 25 and linear programming, 33 and matrix equations, 35
179
Penrose conditions, 28-29 and perturbation theorems, 50 ff. of products, 29, 52 ff. and projections, 20 properties, 26 ff., 30 recursion for nonnegative definite matrix, 47,49 recursion for partitioned matrix, 43-44, 49 recursive computation for nonnegative definite matrix, 160 and roundoff, 25 for special case, 22 ff. and theory of linear equations, 30 Pyle, L. D., 69, 176
Q
Quadratic forms, distribution theory for normal case. 93
R
Range space, 10, 11, 12, 15, 17 Rank of a matrix, 52 Rao, C. R., 93, 96, 175, I76 Recursion, 48 for conditional covariance, 164 for conditional expectation, 164 for constrained least squares estimator, 133 for covariance of BLUE, 127 for least squares estimator, observations missing, 153 ff. for projections, 37 for pseudoinverse of nonnegative definite matrix, 47, 49, 160 for pseudoinverse of partitioned matrix, 43-44,49 for residual error, 129 in weighted least squares, 132 for steady state probability of Markov chain, 37 for straight line regression, 142 for unconstrained least squares, 125 ff. for weighted least squares estimator, 131 ff. Regression multiple, 46 stepwise, 44 ff. straight line, 112
180
INDEX
Residual covariance, 88 Residual error, recursive computation, for weighted least squares, I32 recursive computation of, 129 Residual sum of squares, for analysis of covariance, 150 Robers, P. D., 33, 174 Rosen, J . B., 81, 176 Rust, B., 57, 63, 176 S Scalar product, 5 Scheffk, H., 52, 87, 94, 109, 148, 176 Schneeburger, C., 57,176 Singular, 12 Singular value decomposition theorem, 3839 Sittler, R., 172, 173 Smoothing vector, 127 Spanning vectors, 7 for linear manifold, 8 Spectral representation, 24 Steady state probability vector of Markov chain, recursion for, 37 Stepwise regression, 44 ff., 117, 146 and analysis of covariance, 148 Stewart, G. W., 51, 176 Stochastic matrix, 37 Straight line regression, 112 recursive computation of, 142
Sum of squares decomposition, 108, 110 for straight line regression, 114 for two-way layout, 116 Symmetric matrix, 1 1 ff. T Tests of general linear hypothesis, and confidence ellipsoids, 102 Tewarson, R. P., 69, 176 Transpose, 5 TRUE-BLUE, 93 Two-way layout, 115 ff. Typographical conventions, 4
Unbiased, 88 Variance, 88
U V
W Watson, G. S., 93, 176 Weighted least squares, recursive computation of, 131 ff. Wersan, S. J., 65, 174 Wide-sense conditional expectation, as optimal linear predictor, 166
Z Zeien, J., 123, 174 Zyskind, G., 90, 132, 176
