This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
JOHN WILEY & SONS Chichester· New York ·Brisbane· Toronlo Singapore ·
Copyright(!) 1996 by John Wil�y & Sons Ltd,
Baftins l�� ne , Chichester, West Sussex POI9 I U D. Nation al
lntemational
Engla nd
01243 779777
( +44) 1243
779777
e-mail (lor orders and customer service emtuiries): �-s-books(f�wiley.co.uk. Visit our l lome Page
All
on
http://www.wiley.co.uk
or http://www. wiley.com
Rights Reserved.No part
retrieval system,
or
of this hook may he reproduced. stored in a
tmnsmitted, in any fi>nn or hy any means, eledronic,
mechanical, photocopying. recording or otherwis�. except under the tenns of the C opyright.
D�signs and Patent.' Act 1988 or under the tenns of a lic�-nce issued hy the Copyright Licensing Agency , 90 Tottenham Court Road, London, UK WI P 91IE. without the pennis.,ion in "Tiling of the puhlisher.
Other Wiley EJ1tonal Offices
Jolm Wiley & Son.'. Inc.. 605 Third Awnue, New York, NY I 0 158·00 12. liSA
Jacaranda Wiky Ltd,
:n
Park Road. Milhlll,
Q ueensland 4064. Australia
John Wile�· & Sons (Canada) Ltd, 22 W o rcester Road, Rexdalc. ( lntario 1\19W 11.1. Cana
Jolm Wiley & Sons (A.,ia)
Jin
X ing Distripark.
Pte
l.td, 2 C kme nti l..<><>p 1102-0 I.
Singapore 0512
British Library Ciltalo�:r�iu�: iu l'r�blicatiorr Data A catalogue r�cord t<•r this hook is ava ila bl e from the British Lihmry
ISBN
0 471 9661!8 6; 0 471 97015!! (phk.)
Produced from camcra-rcmly
copy supplkd hy the ••uthors using l.�TeX
Printed ;uld hound in ( ircat Britain hy Biddies Ltd. (iuildt<>rd and King's Lynn
·n1is h<><•k is printed on acid-free paper re,ponsihly manul<•ctnrcd trom ;ustainablc c>r whi ch at least l\\o trees :�rc planted tl>r c·;�ch <•nc used t<•r pap�r produc tion.
for�st:�tion.
Contents
Preface . . . . .
.
XI
.
X Ill
List of Symbols
1 Definitions, Notation, Terminology . 1.1
1.2 1.3 1.4 1.5 1.6
Basic Notation and Terminology . . Operations Relating M atrices . . . Inequality Relations Between M atrices Operations Related to Individual M atrices Some Special M atrices . . . . . . . . . . Some Terms and Quantities Related to Mat rices .
.
2 Rules for Matrix Operations . . . . . . . . . . . 2.1 Rules Related to M atrix Sums and Differences 2.2 Rules Related to M atrix M u ltiplication . . . 2.3 Rules Related to M u ltiplication by a Scal ar 2.4 Rules for t he Kronecker Product 2.5 Rules for t he H adamard Product 2.6 Rules for Direct Sums . . . . . .
3 Matrix Valued Functions of a Matrix 3.1 The Transpose . . . . . . 3.2 The Conjugate . . . . . . . . . 3.3 The Conjugate Transpose . . . 3.4 The Adj oint of a Square M atrix 3.5 The I nverse of a Square M atrix 3.5.1 General Results . . . . . 3.5.2 Inverses Involving Sums and Differen ces 3.5.3 Part i tioned Inverses . . . . . . . . . . . 3.5.4 Inverses Involving Commut ation, Du plication and Elimination M at rices . . . . . . . . . . . . . . . . . .
l 1 3 4
4 9 II 1;j 15 16 18 1\) 20 22 n
'2 ;) '24 2J 27 '27 '27
'28 '2V 31
VI
CONTENTS Generalized Inverses . . . . . . . . 3.6.1 G eneral Results . .. . .. . 3.6.2 The Moore - Penrose Inverse 3.7 M atrix Powers . . . 3.8 The Absolute Value
3.6
4 Trace, Determinant and Rank of a Matrix 4.1 The Trace . . . ... . . .. . . . . . . 4.1.1 G eneral Results . .. . . . . .. . . . 4.1.2 Inequalities Involving the Trace .. . 4 .1.3 Optimization of Functions I nvol ving the Trace 4.2 The Determinant . . . . . .. . . .. . . . . . 4.2.1 G eneral Results . . . . . . .......... . 4.2.2 Determinants of Partitioned M atrices . . . . 4.2.3 Determinants Involving Duplication M atrices 4.2.4 Determinants Involving Elimination M atrices 4.2.5 Determinants Involving Both Duplication and Elimination M at rices . . . . . . . . . . . . . . . . . . . . . . 4 .2.6 Inequalities Related to Determinants .... . ... . 4.2.7 Optimization of Functions I nvolving a Determinant. 4.3 The Rank of a M atrix . . . . . . . . . . . . . . . . . 4.3.1 G eneral Results . . . . .. . . . . ... . . . . 4.3.2 M atrix Decompositions Related to the Rank 4.3.3 Inequalities Related to the Rank
63 63 5.1 Defini tions . . . . . . ..... . . . . . 5 . 2 Properties of Eigenvalues and Eigenvectors 64 5.2.1 G eneral Results . . . . . . . . . . . . 64 5.2.2 Optimization Properties of Eigenvalues 67 69 5.2.3 M atrix Decompositions Invol ving Eigenvalues 72 5 . 3 Eigenvalue Inequalities . . . . . . . . . . . . . . . . . 72 5.3.1 Inequalities for the Eigenvalues of a Single M atrix 5.3.2 Relations Between Eigenvalues of More Than One M atrix 74 76 5 . 4 Results for the S pectral Radius 78 5 . 5 Singul ar Values . . . . . 5.5.1 General Results . 78 5.5.2 Inequ alities . . . 80
6 Matrix Decompositions and Canonical Forms 6.1 Complex M atrix Decompositions . . 6.1.1 Jordan Type Decompositions 6.1.2 Diagon al Decompositions . .
83 83 83 85
..
VII
CONTENTS 601 .3 Other Triangular Decompositions and Factorizations 601.4 Miscellaneous Decompositions 0 602 Real M atrix Decomposition s 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 60201 Jordan Decompositions 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 60202 Other Real B lock Diagonal and Diagonal Decompositions 60203 Other Triangular and M iscellaneous Reductions 0
0
7 Vectorization O perators 0 0 0 701 Defi nitions 0 0 0 0 0 0 0 0 0 0 7 0 2 Rules for the vee Operator 0 703 Rules for the vech Operator
95 95 97 99
8 Vector and Matrix Norms 0 801 General Defi n itions 0 0 0 0 0 802 Specific Norms and I n ner Products 803 Results for General Norms and Inner Products 8.4 Results for M atrix Norms 0 0 0 8.401 General M atrix Norms 0 8.402 Induced M atrix Norms 0 805 Properties of Special Norms 80501 G eneral Results 0 80502 Inequalities 0 0 0 0 0
9 Properties of Special Matrices
86 88 89 89 90 92
101 101 103 1 04 106 106 108 109 109 111
1 13 901 Circulant M atrices 0 0 0 0 113 115 902 Com mutation M atrices 0 0 116 90 201 General Properties 0 117 90202 Kronecker Products 118 90203 Relations With Duplication and Elimination M atrices 903 Convergent M atrices 119 1 20 904 Diagonal M atrices 1 22 905 Duplication M atrices 1 22 90501 General Properties 90502 Relations With Commutation and Elimination M atrices 123 90503 Expressions With vee and vech Operators 0 0 0 0 0 0 0 123 905.4 Duplication M atrices and Kronecker Products 0 0 0 0 0 0 124 90505 Duplication M atrices, Eliminat ion M atrices and Kronecker Products 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 126 906 Elim i n ation Matrices 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 127 90601 General Properties 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 27 90602 Relations With Commutation and Duplication M atrices 127 128 90603 Expressions With vee and vech Operators 0 0 128 906.4 Elimin ation Matrices and Kronecker Products 0 0 0 0
0
0
0
0
.
0
Vlll
CONTE NTS
Elim i nation M atrices, Duplication M atrices and Kro. . . . . necker Products . Hermitian M atrices . . . . . . . . . . . . . 9.7. 1 General Results . . . . . . . . . . 9.7.2 Eigenvalues of H ermitian M atrices 9.7.3 Eigenvalue Inequalities . . . . . . 9.7.4 Decompositions of Hermitian Mat rices Idempotent M atrices . . . . . . . . . . . . . \'onHegative, Positivt> and Stochas t ic Matrices . 9.9.1 Definitions . . . . . . . . . . . . . . . . 9.9.2 Geueral Results . . . . . . . . . . . . . 9.9.3 Results Rel ated to the Spectral Radius . Orthogonal M atrices . . . . . . . . . . . . . . 9. 10.1 General Results . . . . . . . . . . . . . 9 .I0.2 Deco111positious of Orthogonal M atrices Partitioned Matrices . . . . . . . . . . . . . 9.11.1 General Resul ts . . . . . . . . . . . . 9.11.2 Determ inants of Partitioned M atrices 9.1 1.3 Partitioned Inverses . . . . . . . . . 9.11.4 Partitioned Generalized Inverses . . 9. 11.5 Partitioned Matrices Related to Duplic ation M atrices Positive Definite , :"Jegative Defini te and Semidefinite M atrices 9.12.1 General Propnties . . . . . . . . . . . . . . . 9.12.2 Eigenvalue Rt>s u l t s . . . . . . . . . . . . . . . 9.1 '2 .:3 Decomposit ion Tht'orerns for Defini.te M at rices Symmetric M atrices . . . . . . . . . . . . 9. 13.1 General Propert ies . . . . . . . . . 9.13.2 Symmetry and Duplication M atrices 9. 13.:3 Eigenval ues of Symmetric M atrices . 9.1:3.4 Eigenvalue Inequalities . . �).1:).5 Decompositious of Symmetric and Skew-Symmetric Matrices . . . . . . . . . . . . . . . . . . . . . . . . Triangular M atrices . . . . . . . . . . . . . . . . . . . . . . 9. 14.1 Properties of General Tri angul ar M atrices . . . . . . 9. 14.2 Triangularity, Elimination and D u plication M atrices 9. 14.3 Properties of S trictly Triangu lar M atrices · 11 \.'Hilary 'M atnces . . . . . . .
Preface Nowadays m atrices are used in m any fields of science. A ccordingly they h ave become standard tools in statistics, econometrics, m at hematics, engineering and natural sciences textbooks. I n fact, many textbooks from these fields h ave chapters , appendices or sections on matrices. Often collections of those results used in a book are included . Of course, there are also numerous books and even journals on m atrices. For my own work I h ave found i t useful , however, to have a collection of important matrix results h andy for quick reference in one source. Therefore I h ave started collecting matrix results which are important for my own work . Over the years, this collection has grown to an extent that it may now b e a valuable sou rce of results for other researchers and students as wel l . To m ake it useful for less advanced students I h ave also included m any elementary resul ts. The idea is to provide a collection where special matrix results are easy to locate. T herefore there is some repetition because some results fit u nder different headings and are consequently listed more than once. For example, for suitable matrices A , B and C, A ( B + C) A B + AC is a result on m atrix multiplication as well as on matrix sums . I t is therefore listed u nder both headings. Although reducing search costs has been an i mportant objective in putting together this collection of results, it is stil l possible that a specific result is listed in a place where I would look for i t but where not everyone else would, b ecause too much repetition turned out to be counterproductive. Of course, this collection very much reflects my own personal preferences in this respect. A lso, it is, of course, not a complete collec tion of results. In fact , in this respect it is again very subjective. Therefore, to make the vol u me more useful to others in the future, I would like to hear of any readers ' pet results that I h ave l eft out. Moreover, I wou l d be grateful to learn about errors that readers discover. I t seems unavoidable that flaws sneak in somewhere. In this book, defi nitions and results are given only and no proofs. At the end of most sections there is a note regarding proofs. Often a reference is made to one or more textbooks where proofs can be found . No attempt has been made to reference the origins of the results. A lso, computational algorithms =
..
Xll
arC' not given . These m ay again be found in the references. As mentioned earlier, it is hoped t h at this book is useful as a reference for researchers and st udents. I should perhaps give a warning, however, that it requires some basic knowledge and understanding of m atrix algebra. It is not meant to be a tool for self teaching at an introd uctory level . Before searching for specific matrix results readers m ay want to go over Chapter l t o familiarize themselves with my notation and terminology. Definitions and brief explanations of m any terms related to m atrices are given in an appendix. Generally the resul ts are stated for complex m atrices ( m atrices of complex muubers) . Of cou rse, they also hold for real matrices because the l atter may bt> regarded as special complex m atrices. In some sections and chapters many results are formulated for real m atrices only. In those instances a note to t hat effect appears at the beginning of the section or chapter. To make sure t hat a specific result holds indeed for complex matrices it is t herefore recommended t o check the beginning of the section and chapter where the result is found to determine possible qualifications . Finally, I would l ike to acknowledge the help of many people who com mented on t his volume and helped me avoid ing errors. In partic u lar, I ant grateful to A lexander Benkw itz , Jorg Breitung, M aike B u rda, Helmu t Her wartz , Kirsti n H ubrich , Martin Moryson , and Rolf Tschernig for their care fu l scrutiny. Of course , none of them bears responsibility for any remaining errors. Financial support was provided by the Deutsche Forsch ungsgeruein srhaft . Sonderforschungsbt>rt>ich 373. Bt>rlin. April 1 996
Helmut Lutkepohl
List of Symbols General Notation
L n
s u m m ation product equivalent to, by definition equal to implies that, only if if and only if 3 0 1 4 1 59 v'-1, imaginary unit complex conjugate of a complex n umber n atural logarith m exponential function sine function cosine function m axrmum mm1mum supremum , least upper bound infimu m , greatest lower bound 0
I 0
-
c
In exp 0 Sill
cos max mrn sup inf df dx of ox n! 0
0 0
derivative of the function f(x) with respect to
=1°2
.
.
0
11
bin omial coefficient
Sets and Spaces ffi
.
x
partial derivative of the function f(-) with respect to x
n! k!(n - k)!
{0:
c
.
o}. { 0} .
.
set real numbers complex numbers
.
XIV
IN 7L.
IR
Q
positive i ntegers integers m-dimensional Euclidean space, space of real (m x I ) vectors real ( m x n ) matrices m-d imensional complex space, space of complex (m x I ) vectors com plex ( m x n ) matrices
m
_'nl. X n.
General M at rices A= [a;j] x n) A A= [A1 : A2] A = [A 1 : . . : An ]
matrix matrix m atrix matrix
A=
matrix consisting of submatrices A1 and A2
(m
.
with typical element a;j with m rows and n columns consisting of submatrices A1 and A2 consisting of sub matrices A 1, ... An ,
AI
matrix consisting of submatrices A 1, .
A=
A=
[
An A II A21
.412 An
]
A=
. . ,
p artitioned matrix consisting of submatrices A11,A12,A2I,An
partitioned m atrix consisting of submatrices A;j
Special Matrices
Drn
( m 2 �m(m + I ) ) duplication m atrix , 9 ( x m) identity matrix, IO (mn x ) commutation matri x , 9 ( �m(m + I ) x m 2 ) elimination matrix , 9 (m x ) zero or null matrix, 2 x
lm 1\mn or l\m n Lm Omxn 0 '
m
mn
n
zero, null matrix or zero vector
Matrix O p erations A+B .4-B AB
An
sum of matrices A and B, 3 d ifference between matrices A and B, 3 product of matrices A and B, 3
XV
cA, Ac A0B A0B
product of a scalar c and a matrix A, 3 Kronecker product of matrices A and B. 3 H ad amard or elementwise product of mat rices A and B, 3 direct sum of matrices A and B . 4
Matrix Transformations and Functions det A , det ( A ) t r A , tr(A) II A l l
IIA\h
rk A , rk( A ) l A Iabs A' A AH
A a dj A-1 A
A+
A' A 1 /2
determ inant of a matrix A, 6 trace of a matrix A , 4 norm of a matrix A , 10 1 Euclidean norm of a m atrix A, 1 03 rank of a matrix A , 12 absolute value or modulus of a mat rix A. 5 transpose of a m atrix A, 5 conj ugate of a matrix A, 5 conj ugate transpose or H ermitian adjoint of a matrix A , 5 adjoint of a matrix A , 6 inverse of a matrix A , 7 generalized i nverse of a mat rix A, 7 Moore - Penrose inverse of a m at rix A, 7 ith power of a matrix A, 7 square root of a matrix A , 8 0 a11
dg( [a;j ] ) 0
amm
a1 1
0
0
a mm
diag( a l l , . . . , amm) vee , col rvec row vech
col u m n stacking operator , 8 row stacking operator , 8 operator that stacks the rows of a m atrix in a column vector, 8 half- vectorization operator, 8
Matrix Inequalities A >0
all elements of A are posit ive real numbers. 4
.
XVI
A> 0 A< 0 A< 0 A> B A> B
all elements of A are nonnegative real numbers, 4 all elements of A are negative real numbers, 4 all elements of A are nonpositive real numbers, 4 each element of A is greater than the corresponding element of B, 4 each element of A is greater than or equal to the correspondi ng element of B, 4
O ther Symbols Related to Mat rices minor ( a;1 ) cof( a,j )
,\(A) Ama.r or Ama.r(A) Amin or Amin(A) cr( A ) CT mar or CTma.r (A) CTmin or CTmin(A) p( A ) PA ( · ) QA ( )
minor of the element a;j of a matrix A , 6 cofactor of the element a;j of a matrix A, 6 eigenvalue of a matrix A, 63 maximu m eigenvalue of a matrix A, 63 mi nimum eigenvalue of a matrix A, 63 singular value of a matrix A, 64 maximu m singular value of a matrix A, 64 minimum singular value of a matrix A, 64 spectral radius of the m atrix A, 64 characteristic polynomial of a matrix A, 63 minimal polynomial of a matrix A, 215
1 Definitions, Notation, Terminology 1.1
Basic Notat ion and Terminology
A n ( m x n) matrix A is an array of numbers : ...
A=
A lternative notations are: A(mx n ),
A ( m x n)
A = [a ;j ] ( m x n ) .
and n are posit i ve integers denoting the row dimension and t h (' column dimension, respectively. ( m x n) is the dimension or order of A. Thf' a;j, i = 1 , .. . , m , j 1 , . . . , n, are real (elements of IR) or complex numbers (elements of
=
•
A=
A11 .421
AI2 .422
A1q A2q
= [.4;1 ] •='·· }= 1'
A p1
A p2
••
·•
= [A;J].
A pq
Here the A;j are (m; x n j ) submatrices of A with 2'..:�'=1 m; = m aud 2'..:)=1 n1 = n. A matrix written in terms of submatrices rather than individual
) -
H A N D BOOK O F M ATRICES
'
<:>lements is often called a partitioned matrix or a block matrix. S pecial cases are A= [At . . . . , An] =(At: . . . : An], where the A; are blocks of colum ns, and A=
where the A; are blocks of rows of A . A matrix A ( m xm )
having the same number of rows and columns i s a square or quadratic matrix . The elements a11,a22, ... ,amm (a;;, i l , . . . , m) const itute its principal diagonal. =
Im ( m x m)
1
0
0
1
with a;; = 1 for i = 1, . . . , m , and a;j = 0 for i -:f. j is an (m x m) identity or unit matrix and 0 0 Omxn =
= [a;j] 0 0 ( m x n)
with a;1 = 0 for all i, j is an ( m x n) zero or null matrix. It is someti mes simply denoted by 0 . A ( 1 x n ) matrix a=[at, ... ,an] is an n-dimensional row vector or ( 1 x n ) vector. An ( m x 1 ) matrix
b= is an m-vector, m-dimensional vector or m-dimensional colun1n vector. The set of all real m-vectors is often denoted by IRm and t he set of all complex m-vectors i s denoted by a:;m. M any more special matrices are listed in Section 1.5 and the Appendix.
3
D E F I N ITIONS, N OTATION, TER M I NOLOGY
1.2
Operat ions Relating Matrices
Addition: A= [a;i] ( m x n), B= [b;i] (m x n)
A+ B:::: [a;j+ b;j] ( m x n)
(for the r ules see Section 2 . 1) .
Subtraction: A= [a;j] ( m x n), B = [b;j] ( m x n)
A - B= [a;i - b;i] ( m x n)
(for the rules see Section 2 . 1) .
Multiplication by a scalar: A= [a;j] (m x n), c a number ( a scalar) cA:::: [ca;j] (m x n)
Ac
=
( for the r ules see Section 2 . 3 ) .
[ca;j] ( m x n)
Matrix multiplication or matrix product : A= [a;iJ (m x n), B= [b;j] (n x p) AB = [ta;kbkil ( m x p) k=l (for the r u les see Section 2 . 2 ) . Kronecker product, tensor product or direct product: A= [a;j] ( m x n), B = [b;j] (p x q) A®B=
a11B
am!B
a1nB
" ·
amnB
(mp x nq)
( for t h e rules see Section 2 . 4 ) . Hadamard product, Schur product or elementwise product: A= [a;j] ( m x n) , B = [b;i] (m x n) A 0 B = [a;ibij] (m x n)
( for the rules see Section 2 . 5 ) .
4
H A N D BOOK OF M ATRICES
Direct sum: A (m x m), B ( n x n ) A q, B
=
[ � � ] ((
+ n) x (m + n ))
m
( for the rules see Section 2.6).
Hierarchy of operations: If more than two matrices are related by the foregoing operations. they are performed in the following order :
(i) operations in parentheses or brackets, ( i i ) m atrix multiplication and multiplication by a scalar , ( i i i ) Kronecker product and H adamard product, ( i v ) addition and subtraction, ( v ) d i rect s u m . Op erations of the same hierarchical level are p erformed from left to righ t .
1.3
Inequality Relations Between Mat rices
A = (a;j] , B = (b;j] ( m x A A A A
>0 > - 0 > B > B
n
-¢=:::}
-¢=:::}
¢::::::;> ¢::::::;>
) real :
Qjj > a;j -> a;i > a;i >-
0, l 0, l b;j, b;j,
- 1, . . - 1, . . l - 1, . l - I, . .
.
- 1, . , n , , m , J - 1, . . , n, . . . , m, J - 1 , . , n, . - I, . . . , n . . , m, J J
.
.
, m,
.
.
.
.
.
.
.
.
Warning: In ot her literature i nequality signs between matrices are sometimes used to denote different rel ations between posi tive definit<:> and st'midefinite matrices.
1.4
Operat ions Related to Individual Matrices
Trace: A =
[a,iJ
(m x m) m
tr A
=
tr( A )
_
a11 + · · · + amm =La;;
( for its properties see Section 4 . I) .
i= I
5
D E F I N ITIONS, NOTATION, T E R M I N O LOGY
Absolute value or modulus: A= [a,j] (m x n)
IAiabs = [ la;jlabs] =
Ia I I Iabs la21labs
la12labs lanlabs
lainlabs la2nlabs
lam I labs lam2labs
lamn labs
(m x n )
where lclabs denotes the modulus of a complex number r = c1 + ir" which is defined as lclabs = Jci + c� = JCC. Here c is the complex conj ugate of c. Properties of the absolu te value of a m atrix are given in Section 3 . 8 . Warning: In other literature IAI sometimes denotes the determinant of the mat rix A. Transpose: A = [a,;] ( m x n ) I
A' =
that is, the rows of A are the columns of A' (see Sec tion 3.1 for its properties ) . In some other l iterature the transpose of a matrix A is denoted by AT. Conjugate: A = [a;j] ( m x n)
where ii;i is t he complex conj ugate of a;j (see Sect ion p ropert ies ) .
3.2
Conjugate t ranspose o r Hermitian adjoint: A= [a;j] (m x n ) AH _A'= [ii;;]' (n x m) (see Section 3 . 3 for its properties ) . Diagonal matrices: A= [a;jJ ( m x m) 0
dg (A) =
for i t s
6
H A N D BOOK O F M ATRICES
0
diag( a 1 1, . . . , amm) = d i ag
0
( see Section 9 . 4 for further details ) . Determinant: A =
[a;j] (
m
x
det A = det ( A )
m
=
)
l:)-1)Pat;1a2i,
X ·· · X
am;m,
where the sum is t aken over all products consisting of precisely onf' element from each row and each column of A multiplied by -1 or 1, if t he permutation it , . . . , im is odd or even , respectively ( see Section 4 . 2 for the p roperties of determinants) . Minor of a matrix: The determi nant of a submatrix of an ( m x matrix A is a m inor of A.
m
)
Minor of an element of a matrix: A = [a;j] ( m x m ) . The m i nor of a,i is the determi nant of the matrix obtained by deleting the ith row and jth column from A ,
min or( a;i)
=
I
a;- ,ja; j
det
+t , - t
am,jPrincipal minor: A =
I
[a;j]
(m x
m
I
a;- ,j a; + j
+I
ai-l m '
t , +t
1
)
det is a principal minor of A for k = 1 ,
. . . , m-
1.
Cofactor of an element of a matrix: A= [a;j] ( m x of a;j is cof( a ; j ) 8 ( - 1 / +i mi nor(a ;j) . Adjoint: A =
[a;1]
(m x
m
), m
>
m
) . The cofactor
1, I
cof( ami)
cof( amm)
= [cof( a;i )]'.
I
�
DEFI NITIONS, NOTATIO N , T E R M I NOLOGY AadJ = 1 if m
=
1 (see Section 3 . 4 for the properties of the adjoint ) .
Inverse: A = [a;j J ( m x m ) with det ( A) -::j:. 0 . The inverse of A is t he u nique ( m x m) matri x A-1 satisfying AA-1 =A-1A = Im (see Sect ion 3.5 for its propert ies ) . Generalized inverse: An ( n x m ) matrix A- is a generalized inverse of the ( m x n) matrix A i f it satisfies AA-A= A (see Section 3 . 6 for i t s properties ) . Moore -Penrose ( generalized) inverse: The ( n x m) matrix A+ is the Moore - Pen rose ( generalized ) inverse of the ( m x n) mat rix A. if it satisfies ( i ) AA+A= A, ( i i ) A+AA+ =A+, ( iii ) (AA+)H = AA+, ( i v ) (A+A)11 =A+A (see Section 3 . 6 . 2 for its properties) . Power of a matrix: A ( m x m )
n;=I A= A X . .. X A i t imes
A'=
for positive integers i for i = 0 for negative integers i, if det ( A. ) -::j:. 0
I f A can be wri tten
as
0 A=U
0 for some unitary matrix U (see Section 1.5 and Chap ter 6 ) t hen the power of A is defined for any a E IR, a > 0, as follows: Aa
=
U
,\"I
0
0
,\0' m
This definition applies for instance for Hermitian and real symmet ric matri ces (see Section 1.5 for the defini tions of Hermi tian and sym metric m atrices and Section 3 . 7 for the properties of powers of matrices ) .
8
H A N DBOOK OF M ATRICES
Square root of a matrix: The ( m x m) matrix A 1 12 is a square root of t he (m x m ) m atrix A if A112A112 A Elsewhere in the l iterature a matrix B satisfying B' B A or B B' A is sometimes regarded as a square root of A. =
=
.
==
Vectorization: A = [a;j ] ( m x n )
vee A = vec ( A )
=
( mn
col( A ) =
x 1)
t hat ts, ve e stacks the columns of A in a column vector. rvec(A.)
=
[vec( A' )]'
that is, rvec stacks the rows of A in a row vector and row( A ) = vec( A ' ) = rvec ( A)'
t hat is, row stacks the rows of A in a column vector. (See Chapter 7 for the properties of vectorization operators) . Half-vectorization: A= [a ;j] (m x m )
vech A= vech( A )
n m( m + 1 )
=
X 1)
Urnm
that is, vech stacks the columns of A from the principal di agonal downwards in a column vec t or ( see Chapter 7 for the propertirs of the half-vectorization operator and more details) .
9
D E F INITIO NS, NOTATION, T E R M I N O LOGY 1.5
Some S pecial Mat rices
Commutation matrix: The (mn x mn) matrix 1\,n is a commutation matrix if vec ( A ' ) = 1\mn vec ( A ) for any (m x n) matrix A. I t is sometimes denoted by f-(m,n . For example,
/\32 = /\3 2 '
=
1 0 0 0 0 0
0 0 1 0 0 0
0 0 0 1 0 0 0 0 1 0 0 0
0 0 0 1 0 0
0 0 0 0 0 1
is a com mutation matrix . ( For t he properties of commutation matrices see Section 9. 2 . ) D iagonal matrix: An ( m x m) matrix UJ!
0
with a;1 = 0 for i #- j is a diagonal matrix . ( For the propert ies of diagonal matrices see Section 9 . 4 . ) Duplication matrix: An (m 2 x �m(m + 1 ) ) matrix Dm is a duplication matrix if vec ( A ) Dm vech ( A ) for any symmetric ( m x m) matrix .4 . For example, 1 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 =
is a duplication matrix. (For the properties of duplication matrices see Section 9 . 5 . ) Elimination matrix: A ( �m( m + 1 ) x m 2) elimination matrix L , is defined such that vech ( A ) L,vec( A ) for any (m x m) matrix A. =
10
H A N DBOOK O F MATRICES
For exampl e, 1 0 0 0 0 0
L3 =
0 l
0 0 0 0
0 0
0 0 1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0
0 0 0
0 0 0 0 0 1
is an elimination matrix. ( For the properties of elim ination m atrices see Section 9 . 6 . ) Hermitian matrix: A n ( m x m) matrix A i s Hermitian if A' = AH =A . ( For its properties see Section 9. 7.) Idempotent matrix: An ( m x m) m at rix A is idempotent if A2 = thP properties see Section 9 . 8 . )
A.
( For
Identity matrix: Au ( m x m ) m atrix 1
0
0
I
lm =
=
l for with a;; unit m atrix.
1
=
I,
. . . , m and a;j = 0 for i f. j is an identity or
Nonnegative matrix: A real ( m x n) m atrix A= [a;j] is nonnegatiw if a;j > 0 fori = 1 , . . . , m, j = 1 , . . . , n. ( For the properties see Section 9.9. ) Nons in gular matrix: An ( m x m) matrix A is said to be nonsingular or i uverti ble or regular if det (A) f. 0 and thus A- 1 exists. ( For the rules for mat rix inversion see Section 3 .5 . ) Normal matrix: An ( m x m ) m atrix A i s normal i f AH A= AAH .
Null matrix: Au ( m x n ) matrix is a null m atrix or zero matrix, deno!t'd by Omxn or simp ly by 0, if all its elements are zero. Orthogonal mat rix: A n ( m x m) m atrix A is orthogonal if A nonsingular and A'= A-1• ( For the properties see Section 9 . 1 0 . )
Is
Positive an d negative definite and semidefinite matrices: A Hermitian or real symmetric ( m x m) m atrix A is positive definite if xH Ax > 0 for all ( m x 1 ) vectors x f. 0; it is positive semidefinite
D E F I N ITIO N S, NOTATIO N, TERM I N O LOGY
II
if x H A x > 0 for all ( m x 1 ) vectors x; it is negative definite if xH Ax < 0 for all ( m x 1 ) vectors x :f. 0; it is negative semidefinite if xH Ax < 0 for all (m x 1 ) vectors x; it is indefinite if ( m x 1) vectors x and y exist such that xH Ax > 0 and yH Ay < 0 (see Section 9. 12). Positive matrix: A real ( m x n ) m atrix A = [a;j] is positive if a;j > 0 for i = 1 , . . . , m, j = 1, . . . , n ( see Section 9.9 ) . Symmetric matrix: A n ( m x m ) m atrix A = [a;i] with a;j = aj;, !, J = 1 , . . . , m is symmetric. I n other words, A is symmetric if A' = A (see Section 9 . 1 3 ) . Triangular matrices: A n ( m
x
m ) m atrix
0
0
0 with a,i = 0 for j > i is lower t riangular. An ( m
x m
) m at rix
0 0
0
amm
with a;j = 0 for i > j is upper t riangular (see Section 9. 1 4 ) . Unitary matrix: A n ( m x m ) m atrix A is unitary if it is n onsingular and A H = A - 1 (see Section 9. 1 5 ) . Note: M any more special m atrices are listed i n the Appendix. 1.6
Some Terms and Quantities Related to Mat rices
Linear independence of vectors: The m-dimensional row or column , Xk are linearly independent if, for complex numbers vectors x1, Ct, . . . , Ck, CtX1 + · · + CkXk = 0 implies c1 = · · = ck = 0. They are linearly dependent if c1x1 + · · + CkXk = 0 holds with at least one c; :f. 0. I n other words, x 1 , . . . , Xk are linearly dependent if aj E If exist such that for some i E {1 , . .. , k } ,x; = o:1 x 1 + · · · + o:;_ 1 x;_1 + Cl:i + tXi+ l + · · · + Cl:kXk. .
.
•
·
·
·
H A N D BOOK O F M ATRIC ES
Rank: A = [aij] ( m x
n)
=
rk A rk( A ) = maximum number of linearly independent rows o r colum ns of A row rk A row r k ( A ) = maximum number of linearly i ndependent rows of A =
=
col r k ( A ) col rk A maximum nu mber of l inearly independent columns of A _
( for rules related to t he rank of a m atrix see Section 4 . 3 ) . Elementary operations: The following changes to a matrix are cal led t>lenwntary operations: ( i ) interchanging two rows or two columns, ( ii )
mu
l t ip l y i n g any row or column by a nonzero number,
( iii) add ing a multiple of one row to another row , ( i v ) adding a multiple of one colum n to another colu m n . Quadratic form: Given a real symmetric (m x m) matrix A , the funct ion IR defined by Q( .r ) = .r' Ax is called a quadratic form. The Q : IRm quadratic form is called positive ( semi ) definite if A is posit ive (semi ) definite. I t is callf'd negative ( semi) definite if A is negativP ( sem i ) defi nite. I t is indefinite if A is indefinite. �
Hermitian form: Given a Hermitian ( m x m) matrix A , the function Q : If"' IR defined by Q ( .r ) = .rH Ax is called a HPrmitian form. The Hermit ian form is called positive ( sem i) definite if A is p os i t i ve (se m i ) definite. I t is cal l ed negative ( semi) definite if A is rwgatiw (st rl l i ) definite. I t is indefinite if A is indefi n ite . �
'
Characteristic p olynomial: The polynomial in A given by det(U m - A) is the charact erist ic poly nomial of the (m x m) matrix A ( see Section 5. l ) . Characteristic determinant: The determinant det(U m characteristic determi nant of the (m x m) matrix A . Characteristic equation: The equation det(Alm - A ) characteristic equation of the ( m x m) matrix A .
-
A ) is t he
0
IS
t he
Eigenvalue, characteristic value, characteristic root or latent root: The roots of t he characteristic polynomial of an ( 111 x m ) matrix A
DEFINITION S . N OTATIO N, TERMINO L OGY
are the eigenvalues, the characteristic values, t.he charact eristic roots or the l atent roots of A (see Chapter 5 ) . Eigenvector or characteristic vector: A n ( mx 1 ) vector v f. 0 satisfy i ng Av = A v, w here A is an eigenvalue of the ( m x m ) matrix A, is an eigenvector or characteristic vector of A corresponding to or associated with the eigenvalue A (see Chapter 5 ) . Singular value: The singular val ues o f an ( m x n ) matrix A are the nonnegative square roots of the eigenval ues of AAH if m< n , and of AH A, if m > n (see Chapter 5 ) .
13
2
Rules for Matrix Operations In the following all m atrices are assumed to be complex matrices u nless otherwise stated. All rules for complex matrices also hold for real mat rices because the latter may be regarded as special complex matrices. 2.1
Rules Related to Matrix S u ms and D ifferences
(1) A , B ( m xn),c E ([;: ( a ) A ±B = B ±A. ( b ) c(A ±B) = ( A ± B)c = cA ±cB . ( 2 ) A,B,C ( mxn): ( A±B)±C = A±( B±C) = A±B±C. ( 3 ) A ( m xn) , B ,C (nxr ) : A ( B ±C) = A B ± AC. ( 4 ) A, B ( m xn), C ( nxr ) : (A± B)C = AC±BC. ( 5) A ( m xn) , B , C (P xq) : ( a ) A 0( B ±C) = A0B±A0C. ( b ) ( B±C) 0A = B0A±C0A . ( 6 ) A, B, C ( mxn): (a ) A 8( B ±C) = A 8B±A8C. ( b ) ( A ±B )I:;)C = A8C±B8C. ( 7) A, C ( mxm ) , B, D (nxn): (AEBB)±(CEBD) = ( A±C) $( B±D ) . ( 8 ) A, B ( m xn) : ( a ) l A + B labs < I A iabs + ! B l abs· ( b ) ( A ±B)'== A'±B' . ( c ) ( A ± B )H = AH ± BH . - ( d ) A ± B = A± B .
HANDHOOK OF MATRWES
16 (9) A, B (m x m) : ( a ) tr(A ± B) = tr( A ) ± tr( B ) . ( b)
(c) ( 10)
(11)
.4,
A
d g (A ±B)= dg(A) ± dg(B). vech ( A ±B)= vech ( A ) ± vech( B ) .
B ( x n): ( a) rk(A ±B)< rk ( A ) + rk ( B) . (b) ( A ± B)= vec ( A ) ± vec(B). m
vee
(m x m):
(a)
.-1.
+ A' is a symmetric ( m x m)
matrix .
(b) A - .-1.' is a skew-symmetric ( m x m ) matrix . ( c ) A+ A H
is
a Hermit i an ( m x
m) matrix . (d ) A- AH is a skew- Hermitian ( m x m ) matrix . Note: All t he rules of this section fol low easily from basic principles by writing down the typical elements of the matrices invol ved. For further details consult introductory matrix books such as Bronson ( 1 989), Barnett ( Hl90). Horn & Johnson ( 1 98.5) and Searle ( 1 982). 2.2
Rules Related to Matrix Multiplication
( 1 ) A (m x
n
) , B ( n x p),C
( 2 ) A, H ( m x m ) (3) (4)
:l
.-1.,
(m x
11
x
11
n ) ,C
(b)
(9)
(c)
.4,
(AB)C =
x
(n
=
=
p) :
) , B (p x q) , C ( 11
11
lJ
x
(n x n) :
) R (m X n) :
X m .
A( BC) = ABC.
.-l.(B ±C') AR ± AC. (.4 ± B)C AC ± BC. ) D (q x s ) :
1· ,
(.-I.:-;;. B)(C 0 D)=
( 8 ) A (m x
( a)
q) :
AB i= RA in general.
:
( 6 ) A, C (m x m ) , B,
( 7) A (m
x
) , B, C (n x p):
B (m x
( 5) A ( m
(p
(.4 <£! B)(C r:p D)= AC tB RD.
l ABiabs<
) , B ( n x p) :
(AB)' = B'A'. ( AB ) H BHA11. AB = A B .
AC 0 B D.
IAiabsiBiabs·
=
B( 1
1 1
x 111) nonsingular :
(AR)-1 = B-1 A-1.
17
Rl'LES FOR M ATRIX O P E RATIO N S ( 10 ) A = [A;j) (m x B ;i ( n, x Pi) :
n)
with A;j ( m; x
AB = ( 1 1 ) A (m x
n), B
( 12) A, B (m x
m
(n
)
x m):
:
(13) A (rn x n ) , B (n
[�
nj). B
A;kBki
[B,1) ( n
-
x
p) with
]·
tr( A B) = tr( B A).
det(AB) = d et( A ) det(B).
x p):
rk( A B )< min{rk( A), rk( B)}.
n) : rk( A A') = rk( A' A)= rk(A). A (m x n), B ( n x p) :
(14) A (m x (15)
vec( A B ) = (Jp 0 A)vec( B) = ( B' (16) A (m x 11), B (1 1 x p) , C (p x q) : (17) A (m x n), B ( n x m) :
lm)vec(A) = (B' C A)vec(ln).
vee( ABC) = (C' 0 A )vec(B).
tr( A B)= vec(A')'vec(B) = vec(B')'vec(.-\).
(18) A (m x n ), B ( n x p),C (p x m) :
(19) A ( m x n ) . B ( n
0
tr(AB C) = vec(A')'(C':�'J)vec(H).
x p ) . C (p x q) , D (q x m) :
tr(ABCD) = vec(D')'(C' 0 A)vec(B) =we ( D)'(..\::' C')vec(B'). (20) A (m x m)./3 (m x n): (21) A (m x n ) . B (11
x
n ):
A n onsing u l ar
=>
rk( t B)
.4
(m
x
m) nonsingu)a;:
A A-1 = A-1.4 =
(24) A ( m x n ) :
(a ) A ln = lm A = A .
( b ) AOnxp = Omxp. (25) A (m x n ) real:
=
rk( B ).
B nonsingular => rk( A B) = rk(.4)
(22) A ( m x m ) , B ( m x n ) , C ( n x n ) : A, C nonsingular => rk(A BC) = rk(B).
(23)
..
!111.
Opxm A = Opxn·
(a) .4A' and A' A ar e symmetric positive semidefinite.
( b ) rk( A ) = m => A A ' is p ositive definite. (c) rk(A) = n => .-\'A is positive definite.
( 26) A ( 111
X
1!)
:
(a) AAH and Alf A are He r m i t i an positive sPmidPfinitP.
( b ) rk(A) = m => AA H is positive definite. (c) rk( A.)
= n
=> A H A is positive definite.
18
H A N D BOOK OF M ATRICES
Note: M ost of the results of this section can be proven by considering typical elements of the matrices involved or follow d irectly from the definitions (see also introductory books such as Bronson ( 1 989 ) , Barnett ( 1 990 ) , Horn & J ohnson ( 1 985 ) , L ancaster & Tismenetsky ( 1 985) and Searle ( 1 982 ) ) . The rules involving the vee operator can be found, e.g. , in M agnus & Neudecker (I988 ) .
2.3
Rules Related to M u ltip lication by a S calar
( 1 ) A ( m x n ), c 1 , c2 E ([;:
( a) c 1 ( c2 A ) = c.( Ac 2 ) = (c 1 c 2 )A = A ( c 1 c2 ) . ( b ) ( c1 ± c2)A c 1 A ±c2 A . =
A, B ( m x n ) , c E ([; c( A ± B ) = c A ± c B . (3) A ( m x n ) , B ( n x p ) , c1, c 2 E ([;: ( c1A ) ( c2 B ) = c 1 c2 A B . ( 4) A ( m x n ) , B ( p x q ) , c 1, c2 E ([; : ( 2)
:
( a) c.( A 0 B) = (c1 A ) ® B = A 0 ( c1B). ( b ) c 1 A 0 c2 B = ( c1c2 ) ( A 0 B) .
( 5 ) .4, B ( m x n ) , c1, c2 E
( a ) c 1 ( A 8 B) = ( c1A ) 8 B = A
8
( c 1 B).
( b ) c 1 A ,:;:: c2B = ( c1c2) ( A \-)B).
(6) A (m x m ) , B ( n x n ) , c E ([;: (7) A ( m X n ) :
c( A EB B ) = cA EB cB.
OA = A O = Dmxn·
(8) .4 ( m x n ) , c E ([':
(a) / cA/abs = / c / abs / A / abs · ( b ) ( c A )' = cA'. (c ) ( cA ) H = cA H .
(9) A (m x m), c E ([;:
(a) tr( cA) = c tr( A ) . ( b ) dg( c A ) = c dg( A ) . ( c ) A is nonsingu l ar, c i= 0 ( d ) det ( c A ) = c m det( A ) .
=>
( cA ) - 1
=
� A- 1 •
(e) vech(cA ) = c vech ( A ) . ( f ) >. is eigenvalue of A
=>
c>. is eigenvalue of c A .
19
RU L ES FOR M ATRIX O P ERATIO N S
( 10 ) A ( m x n ) , c E
( b ) c :j:. 0
=? =;.
( c ) vec( c A )
=
rk( cA ) = rk(A ) . (cA) + = � A + . c vec(A ) .
Note: A l l these rules are elementary and can be found in introductory matrix books such as Bronson ( 1989 ) , Barnett ( 1 990) , Horn & J oh nson ( 1 985 ) , Lancaster & Tismenetsky ( 1 985) or follow directly from definitions. 2.4
Rules for t he K ronecker P roduct
( 1 ) A ( m x n ) , B (p x q) :
A @ B :j:. B @ A in general.
( 2 ) A (m x n) , B, C ( p x q) :
A @ ( B ± C) = A @ B ± A @ C.
( 3 ) A ( m x n ) , B (p x q ) , C ( r x s)
:
A @ ( B @ C ) = ( A @ B) @ C = A @ B 0 C.
( 4) A , B ( m x n ) , C, D ( p x q ) : ( A + B ) @ ( C + D) = A @ C + A @ D + B @ C + B 0 D. ( 5) A ( m x n ) , B (p x q ) , C ( n x
r ,
) D (q x s)
:
( A @ B ) ( C @ D) = A C @ BD. (6) A (m x n), B
(P x q ) , C ( r x s ) , D ( n x k ) , E (q x 1), F ( s x t ) :
( A @ B @ C ) ( D @ E @ F) = A D @ B E @ C F. (7) A ( m x n ) , c E
c @ A = cA = A @ c.
( 8) A ( m x n ) , B ( p x q ) , c, d E
cA @ dB = cd( A @ B ) = ( cdA) @ B = A @ ( cdB) . (9) A ( m x m) , B ( n x n) , C (p x p ) :
( 1 0 ) A ( m x n) , B (p x q ) : ( a) ( A @ B)' = A' @ B1 • ( b ) ( A @ B ) H = A H @ BH . -
-
( c ) A @ B = A @ B. ( d ) ( A @ B )+ = A+ @ B+ .
( A EB B) @ C = ( A @ C) EB ( B 0 C ) .
20
H A N D BOOK OF M AT R IC ES
(e) l A
0
(f) rk( A
B l abs = I A i abs
c:;
0
I H i abs
B ) = rk( A ) rk( B ) .
( 1 1 ) A ( m x m ). B ( 11 X 11 ) : ( a) A, B nonsingular ( b ) tr( A
0
=?
(A
0
B)- 1 = A- 1 0
B ) = t r ( A ) tr( B ) .
B- 1 .
( c ) dg( A 0 B ) = dg( A ) 0 dg( B) .
( d ) det ( A 0 B ) = ( det A t (det B )m . (e) >. ( A ) and >.( B ) are eigenvalues of A and B . respectively, w ith associated eigenvectors v ( A ) and v ( B ) =? >. ( A ) · >. ( B ) is eigenvalue of A 0 B with eigenvector v ( A ) 0 v( B ) . ( 1 2 ) .r . y ( m x 1 ) : ( 1 3)
.r
.r ' 0 y = yx' = y 0 x' .
(m x 1 ) , y ( n x 1 ) :
vec(.ry' ) = y 0 x .
( 1 4) A (m x 11 ) , B ( 11 x p). C ( p x q ) :
( C' 0 A ) vec ( B) = vee( A BC ) .
( 1 5 ) A ( m x 11 ) , B ( n x p ) , C ( p x q ) , D ( q x m ) : tr( A BCD)
vee ( D' )' ( C'
:�!
A ) vee ( B )
vec ( D' ) ' (C' 0 A ) vec ( B ) vee( D ) ' ( A 0 C' )vec ( B' ) , vec ( A ' ) '( D' 0 B ) vec ( C ) vee( H' ) ( A' 0 C)vec( D) vee( C' )' ( B' 0 D)vec ( A ) . '
Note: A substantial collection of results on K ronecker products incl uding m any of those given here can be fou nd in M agnus ( 1988 ) . Some resu lts are also given in M agnus & Neudecker ( 1 988) and Lancaster & Tismenetsky ( 1 985 ) . There are m any more rules for Kronecker products related to spt>cial matrices, in particular to com mutation , duplicat ion and el iminat ion matrices (see Sections 9.2. 9.5 and 9.6 ) .
2.5
Rules for t he Hadamard P rod uct
( 1 ) A . B ( 1 1! X
11
)
•
c
E
Q�
:
(a) A ,_;:. B = B '�' A . ( h ) c( A :· B ) = ( c A ) ·?• B = A ,_;:, (cB).
(2) A. B. C ( m x 11 ) : ( a ) A ·.·: ( B , : C ) = ( A ; B ) ,_;:· C = A · � B c ; . c_
21
RULES FOR M ATRIX O PERATI O N S
( b ) ( A ± B ) 0 C = A 0 C ± B 0 C. ( 3 ) A , B, C, D ( m x
n.
)
:
( A + B ) 0 ( C + D) = A 0 C + A 0 D + B 0 C + B 0 D . ( 4 ) A , C ( m x m ) , B , D ( n. x n ) :
( A EB B) 0 (C EiJ D) = ( A �1 C) 6.1 ( B C:: 1 D).
(5) A, B ( m x n ) :
( a) ( A C:, B)' = A' 0 B' . ( b ) ( A '::' B ) H = A H 8 B H . (c ) A
0
-
-
B = A 0 B.
( d ) l A (:) B l abs = l A I a bs 8 I B i abs ·
( 6 ) A ( m X n) :
A
8 Omxn
= O m xn 0 A = Omxn ·
A 0 Im = Im 8 A = dg( A ) . 1 1 ( m x m) : ( 8) A ( m x n ) , J = 1 1 (7) A ( m x m ) :
(9) A , B , C ( m x n ) :
tr[A'( B 0 C)] = tr[(A' 1::1 B' )C'] .
( 10 ) A , B ( m x m ) , j = ( 1 , . . . , 1 )' ( m x 1 ) :
tr ( A B ' ) = j'(A
��
B)j.
( 1 1 ) A, B, D ( m x m), j = ( 1 , . . . , 1 )' (m x 1 ) : D d i agonal ( 12 ) A , B , D ( m x m ) : ( 13) A. B (m x
n
( a ) vec( A
)
r:;'
=>
tr( AD B' D) = j ' D( A
D diagonal
=>
8
B ) Dj.
( DA ) C:' ( BD) = D( A ,::; B ) D.
:
B ) = diag( vec A )vec ( B ) = diag( vec B )vec ( A ) .
( b ) vec( A :;; B ) = ( vee A )
r::J
vec( B ) .
( 14 ) A , B ( m x m ) : ( a) vech ( A 8 B ) = diag( vech A )vech( B ) = diag( vech B ) vech ( A ) .
( b ) vech ( A 1::1 B ) = ( vech A )
0
vech ( B) .
( 1 5 ) ( Schur product theorem) A , B ( rn x m ) : A , B positive (semi ) definite (semi) definite.
=>
A ·�.· B posi t i ve
Note: Most results of this section fol low directly fro m definitions. ThP remaining ones can be found in M agnus & Neudecker ( 1988) . The Schur product theorem is, for example, given in Horn & Johnson ( 1 985 ) .
22 2 .6
H A N DBOOK O F M ATRICES Rules for D irect S ums
( 1 ) A ( m x m ) , B ( n x n ) , c E
( a ) c ( A EB B) = cA EB cB. ( b ) A :j:. B <==> A EB B :j:. B EB A .
( 2 ) A , B ( m x m ) , C, D ( n x n ) : ( a ) ( A ± B) EB (C ± D) = ( A EB C ) ± ( B $ D) . ( b ) ( A EB C) ( B EB D) = A B EB C D . (3) A ( m x m ) , B ( n x n ) , C ( p x p) :
(a) A EB ( B $ C) = ( A EB B) EB C = A EB B EB C. ( b ) ( A $ B) 0 C = ( A ® C) EB ( B® C) . (4) A , D ( m x m ) , B , E ( n x n ) , C, F ( p x p) :
( A EB B EB C ) ( D EB E EB F) = A D EB B E EB C F. (5) A ( m x m ) , B ( n x n ) :
( a ) l A EB Blabs = l A Iabs EB ! B labs · ( b ) ( A EB B)' = A' EB B' . ( c ) ( A EB B) H = A H EB B H . (d) A
87
-
-
B = A EB B . ( e ) ( A EB B ) - 1 = A - 1 EB B - 1 , if A and B are nonsingul ar . ( f ) ( A EB B) + = A + EB B + .
( 6) A ( m x m ) , B ( n x n ) :
( a) t r ( A EB B) = tr(A)
+ tr( B) .
( b) dg( A EB B) = dg( A ) EB dg( B ) . ( c ) rk( A EB B) = rk ( A ) + r k ( B ) .
( d ) det( A EB B) = det( A )det ( B ) .
( e ) ..\ ( A ) and ..\ ( B) are eigenvalues of A and B , respectively, with associated eigenvectors v ( A ) and v( B ) => ..\ ( A ) and ..\ ( B) are eigenvalues of A EB B with eigenvectors
respecti vely.
[
v(A ) O
]
and
[ 0 ] v
( B)
'
Note: All results of this section fol low easily from the defi n itions (see also Horn & J oh nson ( 1 985) ) .
3
Matrix Valued Functions of a Matrix In this chapter all matrices are assu med to
be complex matrices unless
otherwise stated . A l l rules for complex matrices also hold for real matrict>s as the latter may be regarded as special complex matrices. 3.1
T he Transpose
Definition: The ( n x
m) matrix
I
A' =
(n x
is the transpose of the ( m x n ) matrix A = (a ;j] . ( 1 ) A , B (m x n ) :
( A ± B)' = A' ± B' .
( m x n ) , c E
( 5) A, B (m x n) :
( A 8 B )' = A' 1::J B' .
( m x m) , B ( n x n ) : ( 7) A ( m x n ) : ( a ) I A' I abs = I A I�bs · (b) rk(A') = rk( A ) . (c) (A')' = A . (6) A
(A
E£l
B)' = A' E£l B' .
m)
24
H A N DBOOK O F M ATRIC eS
(d) (A')H
( A H )'
=
=
(e) A ' = A' = A H .
(8) A ( m x m)
A
.
: =
(a) dg(A' )
dg( A ) .
( b ) t r ( A' ) = tr( A ) . ( c ) d et ( A ' ) = det ( A ) . ( d ) ( A' ) - 1 = ( A - l ) 1 , if A is nonsingular. ( A adi ) ' . (e) ( A' ) ad; =
(f) >. is eigenvalue of A
(9) A ( m x
>. i s eigenvalues of A' . ( A ' ) + = ( A+ )'.
) rea l : n ) , 1\mn ( m n n
x ( 1 0 ) A (m x vee ( A' ) = 1\mn vee( A ) .
(11) A, B (m x n) : ( 12 ) A ( m x
rnn
) B ( m x p) , C
:
=>
( b ) A is syrn metric
x
n
) , D (q
:
x p) :
A' B'
C' D'
].
A' = A .
¢::::::}
( 1 4 ) A ( m x m ) nonsingular:
)
(q
= � r [� [
( a ) A is d iagonal
n
) commutation matrix:
vec ( B' )' vec ( A ) = tr( A B ) .
n ,
( 1 3 ) A ( m x m)
( 15) A (m x
=>
A' = A . A orthogonal
¢::::::}
A' = A - l .
A'A and AA' are symmetric m atrices .
Note: All these results are elementary. They fol low either directly from definitions or can be obtained by considering the individual elements of I he 111atrices involved (see, e.g., Lancaster & Tismenetsky ( 1 985) ) . 3.2
The Conj ugate
Definition: The ( m x n ) matrix A = [a;1] is the conjugate of the ( m x n ) matrix A = [ aij ] - Here aij denotes the complex conjugate of a; j . ( 1) A , B ( m x
(2) A (m x
n
)
-
:
A ± B = A ± B.
) B ( 11
n ,
( 3 ) A ( m x n ), c
E
-
x p) :
([' :
c
AB = AB.
-
A = cA. ·-
-
25
M ATRIX VA L U E D F U N CTIONS OF A M ATRIX
(4) (5) (6) ( 7)
B (P q) : A @- B =- A @ B. ) : A B = A B. - A EB B = A EB B. m), B ( ) A (m ) ( a ) I A i abs = I A i abs· ( b ) rk( A ) = rk( A ) . (c) A = A. ( d ) A' = A' = A H . (e) AH = A H = A'(f) vec ( A ) = vec ( A ) . A = A. (g) A is real
A (m A, B ( A (m
(8) A
(m
x
m
)
x
n ,
x
8
n
x
x
x
-
n
n
x
n
8
:
:
m) :
=;.
= det A .
(a) tr(A) = tr(A).
-
=A A = A - 1 , if A is nonsingular.
( b ) d et ( A )
adj
(C) A adj (d) - 1
.
(e) vech ( A ) = vech ( A) . ( f ) dg( A ) = dg( A) .
(g)
(9}
A(
m
A
A
is eigen value of with eigenvector with eigenvector v . x
n
),
B (m
( 1 0) a 1 , . . . , a m E Q:: :
x p) , C ( q x
n
) , D (q
-
v =? A
is eigenvalue of
A -
x p) :
[� �] =[� �]-
diag(a l , - - - , a m ) = diag( a l , - - - , a m ) -
Note: The resu lts in this section are easily obtained from basic definitions or element wise considerations and the fact that for c 1 , c2 E Q:: , c 1 c2 = i'1 c2 .
3.3
The Conj ugate Transp ose
AH _ A' = [a;jJ' ) ( A ± B)H = AH BH .
Definition: The conjugate t ranspose of the ( n x m ) matrix (1)
A, B ( m
x
n
:
±
(m
x
n
) matrix
A = [a;j ] is the
26
H A N D BOOK OF M ATRICES
( 2 ) A (m x 11 ) , B ( 11 x p) : ( 3 ) A ( m x n ) , c E
( A B) H = B H A H .
( cA ) H = cA H .
( 4 ) A ( m x n ), B (p x q) : (5) A, B (m x n) :
(A
�� '
(A 0 B ) H = A H 0 B H . B) H
=
A H (-;-) B H .
(.4 EB B ) H
(6) A ( m x m ) , B ( 11 x n ) :
(7) A (m x n ) : ( a) / A H /abs = / A /�bs · ( b ) rk ( A H ) = rk( A ) .
= AH
tfi
BH.
( c ) ( A H ) H = A. .
( d ) ( A.' ) H = A. = ( A H )1 • (e) :� H = A' = ( A ll ) . ( f) ( A. H ) + = ( A + ) H . (g) vec ( A 11 ) = vee( . ' ) A
( h ) A is real
=>
.
All =
.4 1 .
(8) A ( m x m) : ( a) t r ( A 11 ) = t. r(A: ) = tr( A ) . ( b ) det( A H ) = det ( "·l ) = det A . ( c ) ( A H ) a dj = ( JlOdi ) H .
( d ) ( A H ) - 1 = ( A - 1 ) H , if A is nonsingular. _H (e) dg( A ) = dg( .4 ) = ( dg A ) H
(9) A (m x n), B
( m x p) , C ( q x
[ (' D ] [ A A
( 10 )
G] ,
B
H
H BH
. . . , am E
(11) A (m x
m) :
[diag( a 1 , . . . , am )] H = diag( a 1 , . . . , ilm ) · .4 Hermitian ¢:=:;> A H = A .
( 12 ) A ( m x m ) nonsingular: (13) A
n ) . D ( q x p) :
(m x n ) :
A unitary
¢:=:;>
AH = A-1
(a) A H A and A A H are Hermitian positive semidefinite. ( b) rk( A ) = m ( c ) rk( A ) = n
=>
=>
A A H is Hermitian positive definite.
A H A is Hermitian positive definite.
Note: The results in this section are easily obtained from basic definit ions or clementwise considerations (see, e.g., Barnett ( 1 990 ) ) .
27
M ATRIX VA L U E D F U N CTIONS O F A M ATRIX
3.4
The Adjoint of a Sq uare Mat rix
Definition: For m > 1 , the ( m x m) m atrix A adj = (cof( a;1 )]' is the adjoint of the ( m x m) matrix A = [ a ;j ] . Here cof( a ;j ) is the cofactor of a ;j . For m- 1 A a dj = 1 . For i nstance, for m = 3 , '
[ aa3n2 a, - det [ 2 a3 2 al det [ 2 an det
A a dj =
( 1 ) A , B (m
X
] a!J ] a 33 a13 ] a 23
a23 a33
[ aa32 11 a1 det [ 1 a3 1 al l - det [ a2 1 - det
] a 13 ] a 33 a,J ]
a 23 a33
a 23
[ aa32 I1 [ aa3l i -det 1 a det [ 1 1 a2 1 det
an ] a3 2 a12 ] a 32 a12 ]
I
an
( A B) a dj = a adj A a dj . m ( 2 ) A ( m X m ) , c E
x
m) : ( a) A a dj = det( A ) A - 1 , if A is nonsingular. ( b ) A A a di = A a di A = det( A ) lm .
(3) A ( m
( c ) ( A ' ) a dj ( d ) ( A H ) a dj
=
( A adi ) ' . ( A a dj ) H .
=
(e) det(A a dj ) = ( det A) m - 1 . ( f) rk( A ) < m - 1 =:} A a dj = 0. ( 4) A ( m
x
m), m > 2 : m 1 0
if if if
rk( A ) = m rk( A ) = m - 1 rk( A ) < m - 1
Note: These results follow easily from basic principles and definitions ( see, e.g . , L ancaster & Tismenetsky ( 1 985) and M agnus & Neudecker ( 1 988 ) ) . 3.5
The I nverse of a Sq uare Mat rix
Definition: An ( m x m ) matrix A - 1 is the inverse of the ( m x m) matrix A i f A A - 1 = A - 1 A Im . =
3.5.1
General Result s
( 1 ) A , B ( m x m) n onsingular :
(AB)- 1 = s- 1 A- 1 .
28
H A N DBOOK O F M AT RI C ES c
( 2 ) A ( m x m) nonsingular,
E
( 3 ) A ( m x m ) nonsingular, B ( n x
n
(a) ( A 0 B ) - 1 = A- 1 0 8 - 1 •
( b ) ( A ifl B ) - 1 = A - 1
ifl
(cA ) - 1 = c- 1 A - 1 .
) nonsingular:
s- 1 .
(4) A ( m x m ) nonsingular: l
(a) .4 - 1 =
A adj .
det ( A ) ( b ) ( A ' ) - I = ( A - I )' . ( c) ( A ) - 1 = A - 1 . ( d ) ( AH )- 1 = ( A - I )H . (e) ( A.adj ) - I =
l
det ( A )
A.
l
( f ) det ( A - I ) =
det ( A )
(g) >. is eigenvalue of A with eigenvector .4 - 1 with eigenvector v . ( 5 ) .4 ( m x m ) :
¢:=:;>
( a) rk( A ) = m
(b) rk(A) =
( c ) A is diagonal dominant
(6) a1 ,
.
.
•
, am
"4
E Q' . a;
. d = J ag ( a , , . . . , am )
( a) A ort hogonal A unitary
� is eigenvalue of
=?
A is nonsingular.
=j:. 0 for i = 1 , . . . m :
( 7 ) Irn ( rn x m ) identity matrix: ( 8 ) A ( m x m ) nonsingular:
(b)
=?
A - 1 exists .
A+ = A-1.
=?
m
v
{=::::::>
¢:=:;>
=:} .4 - 1
= d'Jag ( a -1 1 ,
- 1 ).
. . . , am
Im ' = Im .
A - 1 = A'.
A - 1 = AH .
( 9 ) A ( m x m) positive definite:
( .4 1 1 2 ) - 1 is a square root of A - 1 .
Note: M any of these results are standard rules which can be found 1 1 1 introductory text books such as Barnett ( 1 990) and Horn & J ohnson ( 1 985 ) or follow immediately from definitions. 3.5.2
Inverses Involving Sums and D ifferences
( 1 ) .4 ( m
x
m ) with eigenvalues >. , , . . . A ,
m ,
j .>.; /abs < 1 , i =
l,
... m ,
:
M ATRIX VA L U E D F U N CTIONS OF A M ATRIX
(a) Um + A ) - 1 =
29
00
:2:) - A ( i=O 00
i=O
00
i=O 00
{2) A ( m x m), B ( m x n) , C (n x m ) , D (n x n) :
if all involved i nverses exist . (3) A (m x n) :
(4) A ( m x m ) nonsingular, B (m x m) : ( A + B B H ) . Um + B H A - 1 8 ) nonsingular => ( A + B B H ) - 1 B = A - 1 B ( /m + B H A - 1 B ) - 1 . m
(5) A, B ( m x
) nonsingular:
(a) A - 1 + B - 1 = A- 1 ( A + B ) B - 1 . ( b ) A - 1 + B - 1 nonsingular => ( A - 1 + B - ' ) - ' = A( A + B ) - 1 B = B ( A + B ) - 1 A . ( c ) A - 1 + B - 1 = ( A + B ) - 1 => A B - 1 A = B A - 1 B .
(6) A . B ( m x m ) :
(a) Im + A B nonsingular => ( /m + A B ) - 1 A = A ( Im + B A ) - 1 . ( b ) A + B nonsingular => A - A ( A + B ) - 1 A = B - B ( A + B ) - 1 B . Note: Most resul ts of this subsection may be found in Searl ( 1 982, Chapter 5 ) or follow from results given there. For ( 1 ) see Section 5.4. 3.5.3
Partitioned Inverses
(1) A ( m x
)
m ,
H
(m x
n
), C (n x m). D (n x n) :
30
H A N D BO O K O F !>J AT R ! C E S
[ (_� � ] A and ( D - CA - I B) nonsingular � [ � � rl ( D - CA - 1 B ) - 1 CA- 1 - A - 1 B ( D - CA- 1 B ) - 1 ] [ A - 1 +-A( D- 1-B CA. 1 B ) - 1 CA- 1 ( D - CA - 1 8 ) - 1 ,
( 2 ) A ( m x m) , B ( m x n ) . C ( n x m ) , D ( n x n ) :
- ( A - BD- ' C ) - ' B D - 1 D- 1 + D- 1 C( A - BD- 1 C ) - 1 B D - 1
].
( 3 ) A ( m x m ) symmetric, B ( m x n ) , C ( m x p ) , D ( n x n ) symmetric. f�' (p x p) symmetric : A B' C'
B D 0
C 0 E
-I
-FBD- 1 - FCE- 1 D- 1 B' FC E - 1 D- I + D - I B' F B D- 1 E - 1 C' F B D - 1 E - 1 + E - 1 C' FCE- 1 i f all inverses exist and F = ( A - B D- 1 B' - C E- 1 C') - 1 . F - D - 1 B' F - E - 1 C' F
( 4) A; ( m; x m; ) nonsingular, i
=
1, .
-I
0
m,
0
n E IN , m > n , A ( m x n ) , B ( m x ( m - n ) ) : rk( A ) n , rk( B ) = m - n , A H B =
� [A : B) - 1 (6)
m,
, r :
0
0 (5)
. .
=
[ ((BAHH AB )) -- !! ABHH ] . =
0
n E IN , m < n , A ( m x n ) , B ( ( n - m) x n ) : rk ( A) = m, rk( B ) n - m, A B H = 0
�
[ � r'
=
=
[A H ( A A H ) - ' = B H ( B B H ) - ' ] .
31
M ATRIX VA L U E D F U N CTIONS OF A M ATRIX
Note: The inverses of the partitioned matrices in ( 1 ) - ( 3 ) are given in M agnus & N eudecker ( 1 988, Chapter 1 ) . The other result s are straightforward consequences of the definition of t he inverse. 3.5.4
Inverses Involving Commutation, Duplication and Elimination Matrices
Reminder: •
•
•
l\mn or l\m,n denotes an ( mn x mn ) commutation matrix ( for det ails, see Section 9 . 2 ) . 2 Dm denotes the ( m x � m ( m + 1 ) ) duplication matrix (see Section 9.5). Lm denotes the ( � m ( m + 1 ) x m 2 ) elimination matrix (see Sectiou 9.6 ) .
(6) A ( m x m ) , lm 0 A + A ® lm nonsingular: (a ) [ D;t', ( A ® lm ) Dm J - 1 = 2D;t', ( lm 0 A + A 0 lm ) - 1 Dm .
( b ) [ D;t', ( lm 0 A + A 0 lm ) DmJ - 1
=
D;t', ( lm
(7) A , B ( m x m ) , A @ B + B 0 A nonsi ngular:
[D;!; ( A 0 B ) Dm t '
=
0
A + A 0 lm ) - 1 Dm .
2D;!; ( A 0 B + B 0 A ) - 1 Dm .
(8) A ( m x m ) real symmetric nonsingular ,
c
E 1R :
{ D;t', [A 0 A + c vec ( A ) vec ( A )']Dm } - 1
=
[
D;!; A - 1 0 A - 1 -
c
1+
]
ern vec ( A - 1 ) vec ( A - 1 )' Drn.
( 9 ) A , B ( m x m ) real , lower triangular, i n vertible : [Lrn ( A' 0 B ) L'rr, t '
=
Lm ( ( A' ) - 1
( 1 0 ) A ( m x m ) real , lower triangular, invertible,
0
c
B- 1 ) L '.r, .
E 1R :
{ L m [A ' 0 A + c vec ( A ) vec(A' )'] L;n } - 1
,'}•))
H A N D BOOK O F
M AT R I C ES
Note: The results of this subsection may be found in M agnus ( 1 988 ) .
3.6
Generalized Inverses
Definition: A n ( 71 x m ) matrix A- is a generalized inverse of the ( m x n ) matrix A if i t satisfies A A - A = A . 3.6.1
General Result s
(1) A ( m x m ) :
rk ( A )
=
m � A- = A- t .
(2) A ( m x n) : ( a) A - is not unique in general. ( b ) A A - and A - A are idempotent. ( c ) !,. - A A - and In - A - A are idempotent.
( d ) A ( A H .·\ ) - A_H is idempotent.
(3) A ( m x n ) :
(a) rk ( A ) = rk( A - A ) = rk( A A - ) . ( b ) rk ( A ) = tr ( A - A ) = tr(A A - ) . ( c ) r k ( A A - ) = tr( AA - ) .
( d ) rk(A - ) > rk( A ) . ( e ) rk( A - ) = rk ( A ) � A - A A - = A - . (f) rk ( A ) = n ¢:::=> :1 - A = ln . ( g ) rk ( A )
=
m � A A - = lm .
(4) A ( m x n) : (a) A ( AH A ) - A H A = A . ( b ) A H A (A /{ A ) - A H = AH . ( c ) A (A H A ) - A H is Hermitian. ( d ) ( A_ - ) H is a generalized inverse of A H . ( 5 ) On x m is a generalized inverse of Om x n · ( 6) A ( m x m )
:
A is i dem potent
=>
A is a generalized inverse of itself.
( 7 ) A ( m x n ) , B ( n x m) : A - is a generalized inverse of A :1 - + B - A - A R A A - is a generalized inverse of A .
=>
33
M ATRL : VA L U E D F U N CTIONS OF A M AT RI X A - is a generalized inverse of ( 8) A ( m x n ) , B , C ( n x m) : A => A - + B ( lm - A A - ) + ( In - A - A )C is a general ized i nverse of A.
( 9 ) A (m x n ) , B (m x m) , C ( n x n ) : B , C nonsi ngular => c- 1 A - s - 1 is a generalized inverse of BAC.
( 10) A (m
X
n ) , B (P X q ) :
A - ® s - is a generalized inverse of A l:) B .
( 1 1 ) A (m x n ) , B ( m x r ) , C ( m x r ) : Generalized inverse matrices A - and c- of A and C, respectively, exist such that .4 .4 - Be- C = B can be solved for X and B => the system of equations AXC X = A - sc- + y - A - A Ycc- is a solution for any ( n X m ) matrix =
Y.
Partitioned Matrices ( 12 ) A ( m x m ) nonsingular, B ( m x n) , C ( r x m ) , D ( r x n ) :
= m ' D = CA - 1 B ] D ] a genera 1zed .mverse of [ C [ On �]. B
=>
4-I xm
Dm x r On x r
( 13 ) A ( m x n ) , rk( A ) = r : such that BAC =
=>
.
IS
1.
B (m x m ) , C (n x
[6 �] C[ � � ]
A
11
) are nonsingular and
B is a generalized inverse of .4 ,
for any D ( r x ( n - r )) , E ( ( m - r ) x r ) , F ( ( m - r ) x ( n - r ) ) .
( 1 4 ) A ( m; x n, ) , i = 1 , . . . , r :
0
0 is a general ized inverse of
0
0
Note: A number of books on generalized inverses exist which contain the foregoing results and more on generalized inverses ( e .g . , Rao & �I i tra ( 1 91 1 ) . Pringle & Rayner ( 1 97 1 ) , Boullion & Odell ( 1 97 1 ) , Ben- Israel & GreYille ( 1 974 ) ) . Some of these books contain also extensive lists of references .
34
H A N D BOOK OF M AT R I C ES
3.6.2
The Moore -Penrose Inverse
Definition: The ( n x m ) matrix A+ is t he Moore -Penrose (generalized ) i nverse of the ( m x n ) matrix A i f it satisfies the following four conditions: (i) AA+A = A ,
( ii ) A + A A + = A+ , ( ii i ) ( A A + ) H = A A + , (iv)
(A + A ) H = A + A .
Properties ( 1 ) A (m x n ) :
A + exists and is unique.
( 2 ) .-1 ( m x n ) : A = cH
[ g � ] v is the singular value decomposi tion of A => [ � ] U. A+ = VH
( 3) 1 ( m x .
-
11
),
U
( m x m), U,
F
�·
o- 1 0
(n x n) :
unitary => ( U A V ) + = V H A + U H .
( 4 ) A ( m x n ) , rk( A ) = r :
B
(m X r ) , c ( r X n ) such that A = BC => A+ = c+ B + .
if c # 0 if c = 0
( 5 ) c E Q' : ( 6 ) A ( m x n ) , c E
( cA ) +
( 7 ) A ( m x n ) , B ( t• x s ) :
( 8)
A ( m x m) :
=
c+ A + .
(A 0 B)+ = A+
0
B+
(a) A is nonsingular => A + = A - 1 .
( b ) A is Hermit ian => A + is Hermitian .
(c )
A
is Hermit ian and idempotent => A+ = A .
(9) A ( m x n ) :
(a) ( A + ) + = A . (b) ( A H )+ = ( A + ) H .
35
M ATRIX VA L U ED F U N CTIONS O F A M ATRIX
(c) A H A A + = A H .
(d) A + AAH = A H . (e) A H ( A + ) H A + = A + . (f) A + ( A + ) H A H = A + . (g) ( A H A ) + = A + ( A + ) H . (h) (AAH )+
==
(A+)HA+.
{i) A ( A H A ) + A H A == A . (j ) A A H ( A A H ) + A == A . (k) A + = ( A H A ) + A H = A H ( A A H ) + . { 10) A ( m
x
n) :
(a) rk( A ) =
m
( b ) rk( A ) = n
{=:::::}
A A + = lm .
{=:::::}
A + A = ln .
( c ) rk( A ) = n => A + = ( A H A ) - 1 A H .
( d ) rk( A ) = m (e) r k ( A ) = n
=>
A+ = A H (AAH )- 1
=>
(AAH )+ = A (AHA ) -2A H . (f) rk( A ) = 1 => A + = [tr( A A H ) ] - 1 A H .
(g) A = Om x n { 1 1 ) A (m
x
{=:::::}
A + = On x m ·
n) :
(a) rk( A + ) = rk ( A ) .
( b ) rk(A A + ) = rk(A+ A ) == rk ( A ) . ( c ) t r ( A A + ) == rk( A ) .
{ 1 2 ) A (m
x
n) :
(a) A A + and A + A are idempoten t .
(b) lm - A A + and In - A + A are idempotent. ( 13 ) A ( m
{ 14 ) A ( m
X
n) , B (n
X
n) , B
( 15 ) A , B ( m
x
(m
X r
)
X r
:
)
A B = Om x r :
{=:::::}
A H B == On x r
n ) , A BH = 0 :
B + A + = Or x m ·
{=:::::}
A + B = On x r ·
( A + B )+ = A + + ( in - A+ B ) [C+ + Un - c+c ) M B H ( A + ) H A + ( lm - BC + ) j ,
where C == ( /m - A A + ) B and
:l6 ( 16 )
H A N D BOOK O F M AT R I C ES A
(m x
n
), B ( n x
r
), C ( m x
r
AH A B = A H C
(17)
A
(m x
B (n x
)
n ,
r
)
:
)
:
<===>
AB = AA+c.
det( B B H ) # 0
=>
A B ( A B )+ = A A + .
(18 ) .4 ( m x m ) Hermitian idempotent, B ( m x n) :
=> A - B B+ is Hermitian idempotent with rk ( A - B B+ ) = rk( A ) - rk( B ) .
(a) A B = B
( b ) A B = 0 and r k ( A ) + rk( B ) = m => A = lm - B B + . (19) n ( m
X
m ) Hermitian
p os i t i ve
defi n ite, A ( m X n ) :
AHn- ' A(AH n- ' A ) + AH
= AH.
( 20 ) :1 ( m x m ) Hermitian : ,\ # 0 is eigenvalue of A with associated I is Pigenvalue of A+ w i th associated eigenwct or e i gen v ec t or ! ' => ,\ !• .
Partitioned Matrices
( 2 1 ) A; ( m; x n, ) ,
i
=
1, . . . . r
:
+
0 0 ( 22 ) A ( m x
n
)
Ar
:
[� r [� 0
,
·
Bj+ -
_
A r+
. � [ [ ]' � �r � ] [ + D) ] D
A+
( 2 3 ) A. ( m x n ) , B ( m x p) :
( 4. .
0
0
=
A
0
+
=
A + - A + B( c+ c+ +
•
where C = Um - A A + ) B an d D ( 24)
:1
= (m x
( lp - c + C ) [ lp + ( lp - c + C) B H ( A + ) H A + B ( lp - c+C) J - 1 x BH ( A + ) H A + ( lm - BC+ ) .
n
). B
(p x
n):
[�r
= [A + - TBA+ : T] ,
where T = £+ + ( !,.. - £ + B ) A + ( A + ) H BH I\ ( lp - E E + ) with E = B( l,. - A + A ) and I\ = [lr + ( lp - E E+ ) B A + ( A+ ) H BH ( lp - E E + )J - 1 •
37
M ATRIX VA L U ED F U NCTIO NS OF A M ATRIX
Note: These results on Moore - Penrose generalized inverses are also contained i n books on generalized inverses such as Rao & M itra ( 1 97 1 ) , Pri ngle & Rayner ( 1 97 1 ) Boullion & Odell ( 1 97 1 ) , Ben-Israel & Greville ( 1974 ) . A good collecti on of resul ts is also contai ned in M agnus & N eudecker ( 1 988), including many of t hose given here. M any of the results fol low easily by verifying the defining properties of a Moore -Penrose inverse. ,
3.7
Matrix Powers
Definition: For i E ll. , the ith power of the A' , is defined as follows:
(m
x m
) m at rix A, denoted by
1
for positive integers i
A
'
=
for i = 0
-I
-I
IT A
for negative integers i , if det( A ) # 0.
j=l If
A can
be written
as
0
A=U
0
for some unitary m atrix U (see Chapter 6 ) then the power of A is defined for any n E IR, o > 0 , as fol lows: 0
A" = U 0 This definition app l ies, for instance, for Hermitian matrices . The definit ions are equivalent for integer values of o . Properties
(1)
A(
m x m
), c E
(C , i E IN :
( c A ); = c1 A i .
( 2 ) (Binomial formul a) A, B ( m x m ) , i E IN , i > 1 :
( A + B) i
I
=
L L A k , BA k 2 B . . . A k, B AkJ+l .
j =O
38
H A N DBOOK OF M ATRICES
where the second sum is taken over all k 1 , . . . , kj + l E { 0 , . . . , i } with kl + · · · + ki + l = i - j . ( 3 ) ( Binomial formula) A , B ( m x m ) : ( A + B) 2 = A 2 + A B + BA + B 2 . ( 4 ) ( Binomial formula for commuting matrices) A , B ( m x m ) , i E IN , i > 1 : AB = BA => ( A + B) ;
::::
t (�) Ai s i-i J
.
; =0
.
( 5 ) A , B ( m x m ) , i E IN , i > 1 :
i- 1 j Aj - Bj = LA ( A - B ) B i - 1 - j . j= O (6) A (m
x
m ) , B ( n x n ) , i E IN :
(a) ( A 0 B) ; = A ; 0 B ; . (b) ( A
ffi
B) ; =
[ � � r :::: [ �;
�;
] :::: A;
ffi
Bi .
(7) A ( m x m) , i E IN :
(a) J Ai l abs < J A I �b · ( b) ( A ; )' :::: (A'( ( c ) ( A i ) H :::: ( A H ( ( d ) Ai :::: A ; . (e) ( A i ) - 1 = ( A - I ) ; . (f) rk(A' ) < rk (A). (g) det ( A ' ) :::: ( det A ) ' . s
.
.
tr( A i ) = vec ( Ai f2 1 ) 1 vec( A i f2 ) . lm . m ) identity matrix , i E IN : r:,
( 8 ) A ( m x m ) , i E IN even : ( 9 ) lm ( m
X
( 10 ) i E IN , i -f; O :
=
O ;, x m = Om x m ·
39
M ATRIX VA L U E D F U N CTIONS OF A M ATRIX
( 1 1 ) i E IN : A;
>. 0
>.
0 0
0 0
1
0 1
'
0 0
>.
1
>.
(m x n )
-1 ) >.i (�
0
>. i
0
0
0
0
m+l ( m i- 1 ) >. • m+ 2 ( i ) >.i m- 2
( 12 ) A ( m x m ) : ( a) A is idempotent => Ai = A for i = 1 , 2 , . . . ( b ) A is nilpotent ( c ) A is symmetric ( d ) A is Hermitian ( 13 ) A ( m x m ) , i E IN : vec ( A' )
( 14 ) A ( m x m) : A ' - i -oo 0
<==>
A' = Om x m for some i > 0.
=>
=>
Ai is symmetric for i = 1 , 2 , . . . Ai is H ermitian for i = 1 , 2 , . . .
( /m 0 A' ) vec ( lm ) ( ( A i ) ' 0 lm ) vec( Im ) ( ( A i f 2 ) 0 Ai f 2 ) vec ( Im ) , if A is posit i vi' definite ( ( A i/2 ) ' 0 Ai f 2 ) vec( Im ) , if i i s even ( A' 0 A )vec( A i - 2 ) , if i > 2 .
<==>
all eigenvalues of A h ave modulus less than 1 .
Note: M ost results of this section are basic and follow immediately from definitions. The binomial formulae m ay be found in Johansen ( 1 995. Sec t ion A .2) and ( 1 3 ) is a consequence of an important rel ation bet ween Kroneckt>r products and the vee operator ( see Section 2 . 4 ) .
3.8
T he A bsolute Value
Definition: G iven an ( m x n ) matrix A = (a;j] its absolute value or modulus . IS
! a l l labs
la2 1labs
Ia dabs Ian labs
la1n labs la2n labs
(m x n )
H A N DBOOK OF M AT RICES
40
where t he modulus of a complex number c = c1 + i c 2 is defined as lclabs Jci + c� = .jCC . Here c is the complex conj ugate of c.
=
( 1 ) A (m x n ) : ( a) I A iabs > Om xn · ( b ) I A i abs = Om xn {::::::} A = Om xn · ( 2 ) A ( m X n ) , c E ([, : lcAiabs = l c l abs I Ai abs · ( 3 ) A , B ( m X n ) : l A + B labs < I A i abs + ! B l abs · ( 4 ) A ( m X n ) , B ( n X p) : I AB i abs < I A i abs I B labs · ( 5 ) .4 ( m X m ) , i E IN : lA; Iabs < I A I� bs · l A 0 B l abs = I A i abs ® ! B l abs · (6) A (m X n) , B (p X q) (7) A ( m X m ) , B ( n X n ) : l A 81 B l abs = I A i abs 81 ! B l abs · (8) A ( m x n) : ( a) I A' I abs = I A I�bs · (b) I Alabs = I A l abs · ( c ) I A H iabs = I A I�bs · ( 9 ) A ( m x n ) , B ( m x p ) , C ( q x n ) , D (q x p) : :
[ CA DB ]
_
abs -
[ I CAiabs
! B labs I i abs I D iabs
].
These results may be found in Horn & Johnson ( 1985, Chapter 8 ) or follow easily from the definition of the absolute value. Note:
4
Trace, Determinant and Rank of a Matrix 4.1
The Trace
Definition: The trace of an ( m x m) matrix A = [a ;j ] is defined
trA = t r ( A ) 4. 1 . 1
(3) (4)
=
(6)
(7) (8)
=
L a;; . i= l
tr( A ± B ) = tr(A ) ± tr( B ) . A ( m x m), c E tr(A) = rk( A ) . A ( m x m) with eigenvalues A 1 , . . . , Am : tr( A ) = A 1 + . . . + Am A (m x n ) , B ( n x m) : (a) tr( A B ) = tr( BA). ( b ) tr(AB) = vec (A')'vec(B) = vec(B') ' vec(A ) . =
(5)
a 1 1 + · · · + amm
m
General Results
( 1 ) A , B ( m x m) : (2)
=
as
42
H A N DBOOK O F M AT R IC ES
( 9 ) A ( m x n ) , B (n x p) , C (p x q ) , D ( q x m) :
vec( D' )' (C' 0 A ) vec( B ) vec(A' )' ( D ' 0 B)vec(C) vec( B' )' ( A' 0 C)vec( D) vec(C' )' ( B ' 0 D)vec ( A ) .
t r( A BC D)
B nonsingular => tr ( BAB- 1 ) = tr( A ) . ( 1 1 ) A , B ( m x n ) , j k = ( 1 , . . . , 1 )' (k x 1 ) : tr( AB') = j:r, ( A 0 B)jn . ( 1 2 ) A , B, D ( m x m) , j = ( 1 , . . . , 1 )' ( m x 1 ) :
( 10 ) A , B ( m x m) :
tr( A D B' D) = j ' D' ( A :�.' B ) Dj.
D diagonal ( 1 3 ) A , B, C ( m x n ) :
tr[A' ( B (-:) C)] = tr((A' >-:1 B' )C] . ( 14 ) A ( m x m) : tr(A ��� lm ) = tr(A) . ( 15 ) A (m x m ) , B ( n x n) : ( a) tr(A 0 B) = tr( A )tr( B). ( b ) tr(A oil B ) = tr( A ) + tr( B). ( 16 ) A (m x m ). B ( m x n ) , C ( n x m ) , D ( n x n ) : tr
[ � � ] = tr( A)
+
tr( D ) .
tr( /\mm ) = m. ( 18 ) Drn (m2 x � m ( m x 1 ) ) duplication matri x : ( a) tr( D:r, Dm ) = tr( Dm o:r, ) = m2. ( b ) tr( D:,. Dm ) - 1 = m( m + 3 )/4. ( 19) Lrn ( � m( m + 1 ) x m2 ) elimination matrix :
( 1 7 ) l\.mm ( m2 x m 2 ) commu tation matri x :
tr( Lm L:r, ) = tr( L:r, L m ) = � m( m + I ) .
( 20 ) A =
[a;1], B ( rn x m) real positive semidefinite, tr( A 0 ) =
m
L a� i= I
<===>
a
E Dl, a > 0 , a f. I :
A is diagonaL
( 2 1 ) A , B f. 0 ( m x m) real positive semidefinite, a E Dl, 0 < a < I :
tr( A 0 B 1 - o )
=
(tr A t (tr B ) 1 - o
<===>
B = cA for some c E
Dl, c > 0 .
43
TRACE, D ET ER M I N A NT A N D R A N K OF A M ATRIX
( 22 )
A, B (m
x m
) real positive semidefinite, a E IR, a > 1 :
[tr (A + B) a ] i f a = (tr A " ) i f a + (tr B a ) i f a {::::::} B = cA for some c E IR , c > 0.
The rules involving the vee operator, the commutation , duplication and elimination matrices are given in Magnus & Neudecker ( 1 988) and Magnus ( 1 98 8 ) . (20) - (22) are given in Magnus & Neudecker ( 1 988, Chapter 1 1 ) . The other rules follow from basic principles. Note:
4.1.2
Inequalities Involving the Trace
In this subsection all matrices are real unless otherwise stated. A ( m x n) complex : (a) l tr A labs < tr lA I abs · (b) tr(A H A ) = tr(AA H ) > 0 . (2) A, B (m x n) : (a) (Cauchy - Schwarz inequ ality) tr(A' B ) 2 < tr(A' A)tr(B' B ) . ( b ) tr(A' B ) 2 < tr(A' A B' B ) . (c) tr(A' B ) 2 < tr ( A A' BB' ) . ( 3 ) (Schur's inequal ity) A ( m x m ) : tr(A 2 ) < tr(A' A ) . ( 4 ) A ( m x m ) positive semidefinite: (det A ) 1 f m < ,!. tr(A ) . (5) A ( m x m ) : All eigenvalues of A are real => I ,!. tr( A) Iabs < [,!. tr( A 2 )] 1 12 . (6) A ( m x m ) symmetric with eigenvalues ) q < · · < Am , X ( m with X'X = In : n n LA; < tr( X' AX) < LAm-n+i . i=l (1)
·
(7) A = [a;j ] ( m
x m
) positive semidefinite:
tr( Aa ) (8) A, B =f:. 0 ( m
x m
{
>
- 'C""' L.... ':l a =l a� for (} > 1 < 2:: ;" a � for O < a < l . 1 u
) positive semidefinite, a E IR :
x
n)
H A N DBOOK OF M ATRICES
44
( a) ( Holder's inequality ) 0 < a < 1 => tr(A0 B 1 - 0 ) < (tr A)" ( tr B ) 1 - <> . ( b ) a > 1 , /3 a/( a - 1 ) => tr( AB) < ( tr A0 ) 1 1 ° ( tr BP ) 1 /i3 . ( c ) ( Minkowski 's inequality) a > 1 => [tr( A + B) 0 j l /<> < (tr A 0 ) L/ o + (tr B 0 ) 1 1 <> . (9) A , B ( m x m ) positive semidefinite: ( a) E IR. 0 < a < 1 => tr( A0 B 1 - 0 ) < tr( a A + ( 1 - a ) B ) . ( b) ,� tr( A B ) > ( det A ) 1 1"' ( det B ) 1 f m . ( 10 ) A ( m x m ) positive definite: ln det( A) < tr( A ) ( 1 1 ) A ( m x m) posit ive definite, B ( n x m), C ( m x n ) : BC = In => tr(C ' AC) > t r( BA - 1 B' ) - 1 . =
n
m.
( 1 2 ) A ( m x m ) posit ive semidefinite with maximum eigenvalue Ama .r ( A ) . B (n x m) : ( 13) A ( m x m)
tr( BAB ' ) < A m ax ( A)tr( B B ' ) . positive semidefinite with maximum eigenvalue A m ax ( A )
:
?
tr( k ) < A m a x (A)tr( A). ( 14) ..\ . B ( m x m ) : t r[( A + R ) ( A + B)'] < 2 [tr( AA' ) + tr( BB')]. ( 1 5 ) A ( 1 11 x m ) posit ive definite, B ( m x m) positive semidefinite: ex p (tr( A - 1 B)] > ( 16)
(17)
( 18 ) ( 19) ( 20 )
det( A + B) . det( A )
A,
B ( m x m ) posit i ve semidefinite: ( a) tr( A B ) < tr( A c_:::; B ) . ( b ) t r( A B ) < � ( tr A + tr B) 2 . (c) t r( A ·2 B ) < � ( t.r A + tr B j'l . ( d ) tr( A , ;, B) > 0 . ( e ) tr( A • ·.· B ) < tr(A 0 B ) . A ( m x m ) , B ( 11 x n ) positive semidefinite : tr( A. 0 B) > 0 . A , lJ , C, D ( m x m ) positive semidefinite: C - A , 0 - B positive semidefin ite => tr( A B ) < tr(C D). A . B ( m x m ) symmetric: tr( A B ) < � tr( A2 + B2 ) . A, B ( m x m ) : tr( A 0 B) < � tr( A 0 A + B 1;;) B ) , -
-
45
TRACE, D ET ERM I N A N T A N D R A N K OF A M ATRIX
tr(A �� B) < � tr(A C:' A + B r:;:: B). Note: Inequalities ( 1 6 ) - (20) are from Neudecker & Shuangzhe ( 1 993) (see also Neudecker & Shuangzhe ( 1 995) ) . The other inequalities can be found in Chapter 1 1 of Magnus & Neudecker ( 1 988 ) . 4. 1 . 3
Optimization of Functions Involving the Trace
In this subsection again all matrices are real if not otherwise stated . ( 1 ) n ( m X m ) real sym metric with eigenvalues ) q < . . . < A m and associated orthonormal ( m x 1 ) eigenvectors v1 , . . . , 1'm , n E { l , . . . , m} :
min{ tr(B'fl B) : B ( m x n ) real , B ' B = /, } = A 1 + · · · + A, . The m inimizing matrix is B
=
[v1 , . . . , v, ] .
max{tr( B'flB) : B ( m x n ) real, B ' B = /, } = A m + · · · + A m - n + I · The maximizing matrix is B [ vm , . . . , Vm - n + d · ( 2 ) n ( m X m ) complex Hermitian with eigenvalues A t < . . . and associated orthonormal ( m x I ) eigenvectors r 1 , . . . { 1 , . . . , m} : =
< Am
, l'm .
min{tr( B H Q B ) : B ( m x n ) complex, B H B The m inimizing matrix is B
=
=
/, }
=
n E
A 1 + · · · + A, .
( v1 , . . . , v,] .
max{ tr( B H flB) : B ( m x n ) complex, R H H = /11 } =
Am + · · · + A m - n + l ·
The maximizing matrix is B [vm , . . . , t'm - n + t l · ( 3 ) n ( m X m ) positive definite with eigenvalues A t < . . . < Anl and associated orthono.-mal eigenvectors v1 , . . . , Vm . 0 < r < m : =
min{t r(fl - A ) 2
:
A ( m x m) positive semidefinite, rk( A ) = r } A T + · · + A �, -r · =
The m inimum is attained for A
=
m
L
i= - r+ l m
A, v, v: .
H A N D BOO K OF M ATRICES
46
( 4 ) X (m x n ) ,
)q
· · ·
eigenvalues of X' X with associated orthonormal eigenvectors V t , . . . , v, , 0 < r < n : <
< >.,
min{ tr[( X - A B ) ( X - A B )'] : A ( m x r ), B ( r x n ) with BB' = Ir } = A t + A 2 + · · · + A m -r . The minimum is attained for B (5)
Y
(m
=
[v, ,
.
.
. , Vn - r + d ' and A = X B'.
x n ) , X (p x n) with rk( X ) = p, Q ( m x m) positive definite, A t < · · · < A m eigenvalues of Q t f 2 Y X'(X X')- t X Y'nt / 2 ' with associated orthonormal eigenvectors V t , . . . , Vm , I < r < m < p : min{ tr[( Y - A B X )' fl(Y - ABX )] : A ( m x r ) , B ( r x p), rk( A ) = rk( B ) = r} = tr[( Y - A B X )'fl(Y - A BX )] , where
A = Q - t / 2 [v m , . . . , V m - r + l ]
and (6)
y
(m
X
B = [v m , . . . , V m - r + ! ]'Q t f 2YX'(X X ' ) - 1 . n ) , X (p X n ) , rk( X ) = p, n ( m X m ) positive definite:
min{ t r[( Y - AX )'fl(Y - AX )] : A ( m x p ) } = tr(QYY' - flY X'(X X' )- t XY' ) . The minimum is attained for A = YX'(XX' ) - t . ( 7 ) y ( m X n ) , X ( P X n ) with rk ( X ) = p, n ( m X m ) positive definite, R (q x m ) with rk( R) = q , C ( q x p ): rnin{ tr[(Y - AX )'fl(Y - A X )] : A ( m x p), RA = C} = tr[(Y - AX )'fl(Y - A X )] , -
.
where ( 8 ) B , C ( m x m ) pos itive definite, B diagonal, A t < · · · < Am eigenvalues of s - t f2C s - t / 2 with associated orthonormal eigenvec tors V t , . . . , V m , I < r < m : min{ln det(AA' + B) + tr [(AA' + B ) - t C] : A ( m x r ) } = m + In det (C) + L:;r'� r ( .A; - In A ; - I ) .
47
T RACE, D ET ERM I N A NT A N D R A N K OF A M ATRIX
The m inimum is attained for 0
1 /2
0 (9) B ( m x m)
positive definite: m
in { tr( B - 1 A) - In ldet( B - 1 A) l abs : A ( m x m) positive semidefinite} = m.
The m inimum is attained for A = B . The results of this subsection are partly given in Magnus & Neudecker ( 1 988, Chapter 1 7), Horn & Johnson ( 1 985, Chapter 4 ) and Liitkepohl ( 199 1 , Section A . 1 4 ) . Result (9) is from Johansen ( 1995, Section Note:
A. ! ).
4.2
The Determinant
Definition:
The determinant of the ( m x m ) matrix A = (a ;j ] is defined
as
where the sum is taken over all products consisting of precisely one element from each row and each column of A multiplied by - I or 1 , if the permutation i 1 , . . . , i m is odd or even , respectively. 4 .2. 1
General Results
(1) ( 2 ) A = [a;j ] ( m x m) , m > 2 :
det ( A )
a ; 1 cof(ai i ) + a 1 j cof(a 1 j ) +
· · · · ·
+ a ; m cof( a i m ) · + a mj cof(a mj )
for any i, j E { 1 , . . . , m } . Here cof( a;i ) denotes the cofacor of a ;j (see Section 1 . 4). ( 3 ) A (m x m), c E
18
H A N DBOOK O F M ATRICES
( a) det ( A B ) = det ( A )det ( B ) .
( h ) B is n o n si ng u l ar
=>
( 5 ) !, ( m x m ) i d e n t i ty matrix: (6) A (m
x m
det ( /111 )
x m
=
)
det. ( A ) . l.
) with eigenvalues ) q , . . . , Am : det ( A ) =
(7) A (m
=
det ( B A B- 1 )
m
>.1 · · · Am = II >.; . z
=I
:
(a) det ( A ' )
=
det.( A ) .
( b) det ( A H ) = det ( A ) .
( c ) det( A ) = det( A ) . ( d ) det ( .-\ - 1 ) = ( det A ) - 1 . if A is nonsingu lar . ( e ) d et ( Aadj ) = ( d e t .-\ ) m - l . ( f ) det ( A ) lm
( 8 ) .-1 ( m
=
.4 ad1 A = A A a dj .
) B (n x n) :
x m .
( a ) det ( A C B ) = ( d e t A )" ( det B )"' .
(b)
d t ( A ·I· H ) = det ( A )det ( B ) . e
( 9 ) .·\ = [a;1 ] ( m x m ) :
( a ) A = diag( a l l •
(b)
( 1 0 ) :1 ( m
A
. . . , a711,
is t ria n g u l a r
x n
) real :
=>
)
=>
det( A )
det ( A ) =
de t ( lm + .4 .41 )
a1 1
=
=
a11 · · · a
· · · Um m
, ,.
m
= II i= I
=
m
II
i= I
a; ; .
a;; .
d et( l + A' A ) . ,.
( 1 1 ) .-\ ( m x m ) : ( a ) rk( A )
<
( b ) rk ( .-1 ) =
m -¢:::::::} det ( A )
m -¢:::::::} det( A )
(c ) A is singular
-¢:::::::}
= 0.
f. 0 .
det( A ) = 0 .
( d ) The rows of .·\ are linearly i ndependent
(e)
The
( f) :! (g) .· l
-¢:::::::}
d r t ( A ) f.
0.
col u mns of .-\ are linearly independent. -¢:::::::} d e t ( A ) f. 0 .
has
has
a
row or col umn of zeros
=>
det ( A )
t wo ident ical rows o r columns
( 1 2 ) A . IJ ( r n x m ) :
=>
=
0.
d et ( A )
=
0.
49
TRACE, DETERM I N A NT A N D R A N K O F A M ATRIX
(a) B is obtained from A by adding to one row (column) a scalar multiple of another row (column) => det(A) = det ( B ) . (b) B is obtained from A by interchanging two rows or columns => det ( B ) = -det ( A ) . ( 1 3) ( B ine t Cauchy formula) A , B ( m x n ) , m < n : det ( A B' ) = 2::: det(A, )det ( B�1 ) , where the su m is taken owr all ( m x m) subm at.rices A m of A and B n are the corresponding submatrices of B . ( 14 ) A (m x m ) p osi t ive definite, B ( m x m ) positive semidefinite: r
det( A + B) = det ( A ) ( 15)
B = 0.
A = [ a ;J ] (m x m ) positive definite: m
d('t( A ) = II a;; i =I
( 16 )
{::::::}
{::::::}
A = diag( a I I ·
.
.
. , a,rn ) .
( Vandcrmondc determinant)
AI
,
.
.
.
•
Am E
=
det
0 0
0 0
1 0
0 1
II(,\,
)
-
Aj ) .
= ( - 1 ) m po .
Note: Most results of this subsection are straightforward implications of
th(' defi nitions and elementary matrix rules. They can be found in t ext books su h as Lancaster & T ismenet s k y ( 1 985) and Barnett ( 1 990) . c
4.2.2 (1) A
Determinant s of Partitioned Matrices
( m x n ) , B ( m x p) , C = [A : B) (m x ( n + p) ) : (a) det (CH C ) = det ( A H A )det[B H ( lm - AA+ )B] = det( BH B)det [A H ( l, - BH + )A] .
50
H A N D B O O K O F M ATRICES
(b) rk( A) = n :::} det(CH C ) = det( A H A)det [BH B - BH A(AH A) - 1 AH B] . ( c ) rk( B) = p :::} det(CH C) = det( B H B)det[A H A - A H B(BH B) - 1 BH A] . ( 2 ) det
[ 1�
10n
]
( - l )m .
=
( 3 ) A ( m x m), B (m x n ) , C ( n x n ) :
det (4)
A
det ( A )det(C).
=
det(A)det(C').
B, C ( m x m ) : det
:1
=
(m x m), B ( n x m), C ( n x n ) :
( 5 ) A.
(6)
[ �nxm � ]
[
A C
B
Orn x tn
]
=
( - l ) m det( B)det(C).
(m x m). B ( m x n ) , C ( n x m), D ( n x n ) : A nonsingu lar
:::}
det
D nonsingular
:::}
det
[ � � ] = det ( A )det( D - CA - 1 B ) . [ �· � ]
(7) A ( m x m ) , a, b ( m x 1 ) : A nonsingular ( 8 ) A ( m x m), a , b ( m x 1 ) ,
:::} c
det
E
=
det(D)det ( A - BD - 1 C).
[ 6't � ]
==
-b H A a d1 a .
A nonsingular
Note: Some of these results on partitioned matrices are given in M agnus ( 1988). Most results are straightforward implications of the definitions and
t'lementary matrix rules, however.
51
TRACE, D ETERM I N A NT A N D RA N K O F A M ATRIX
4.2.3
Determinants Involving Duplication M atrices
Reminder: Dm denotes the ( m 2 x � m ( m + 1 ) ) duplication matrix (see Section
9.5 for more details).
det( D:r, Dm ) = 2 m ( m - l ) / 2 . ( 2 ) det(Dm D:r, ) = 0. ( 3 ) A (m x m ) : ( a) det ( D:r, (A ® A)Dm ) = 2 m ( m - l )/2 det ( A ) m + I . (b ) det ( D;t", ( A ® A) D;t", ' ) = 2 - m ( m - l )/2 det ( A ) m + l . (c ) det ( D;t", ( A ® A ) Dm ) = (det A ) m + l . (4) A, B ( m x m ) : (a) det ( B ) 0 => det(D;t", ( A ® B)Dm ) = 0. (b ) det(B) :;f 0 and A 1 , . . . , A m are the eigenvalues of AB - 1 => det[D;t", ( A ® B )Dm ] = T m ( m - l )/2 det ( A ) (de t B) m IT ( A , + Aj ) · (1)
=
i >j
( c ) det( A ) '# 0 and At , . . . , Am are the eigenvalues of B A - l => det[D;t", (A ® A ± B ® B)Dm ] = (det A) m + t IT ( 1 ± A; Ai ) .
i>j
(5) A (
) with eigenvalues )q , . . . A m det[D� ( A ® lm )Dm ] = T m ( m - l ) f2 det( A ) IT (.\; + Aj ) ,
m x m
,
:
i >j
det[D;t", ( /m ® A + A ® lm )Dm] = 2m det ( A ) IT (A; + A1 ) . (6) A =
[ a ;j] , B = [b;i J (
m x m
) lower triangular:
i >j
det[D� (A ® A ± B ® B)Dm ] = IT(a ;; ai i ± b;;bjj ) .
i>j
( 7 ) A (m
x m
) real , symmetric, nonsingular, c E IR :
( 8 ) A (m
x m
) with eigenvalues At , . . . A m , i E IN , i > 1 :
det ( D! [A ® A + c vec (A)vec (A)']Dm ) = ( 1 + cm)det (A) m +l . ,
i- 1
det D� L ) A i - t - i ® Ai )Dm i =O
= im (det A ) i - l IT /l k l
k >1
I I A � D B O O K O F M AT R I C E S
where �k1 =
{
·\i I
l/lk-
Note: The rules p resented in this ( l 988).
if .\k =F .\1 if .\k = .\1 �u bsection may be fo und
i n M agnus
Determinants Involving Elimination Matrices
4.2.4
Reminder:
Lm
f3
=
[b ; j ] ( m x m )
upper ( lower) t ri angu l ar :
,· . B ) L d et ( L rn ( .'"11 \'� rn ) = I
(4 )
:l
= [a 1 ] , B ,
=
[ b; j ] ( m x m ) det ( L, ( A'
det ( J.111 ( .4
whert' Al i i I
(6) A
-
0
·2
rn
II
i
=I
m biiiaii - ' + I .
lower t riangular: m
B ) L�, ) =
II
i= I
b:; a :7- ' + 1 .
rn
B ) L:n )
aII
a1;
a; I
a ;;
= II det ( A ( • I )det ( B1 i 1 ) , i= I
B( ' I -
b; m
b; i bm;
...
l . . . . , m. ( m x rn ) real . lower t ri angu l ar n o u s i ngu l ar , c E
b mm
-
,
det( L, [.-1' : ; :1 + r vec( A)vec ( A' )'] L�, ) ( 7) A . B ( m x m ) lower triangu lar: :
IR :
= ( 1 + r m )d t> t ( .4 )"' + 1 .
det( Lm ( A B' 0 A ' B ) L :ll ) d et ( L m ( A B
c:;
B' A ) U,. )
det( L m ( A B ' 0 A ' B' ) Dm ) ( det .4 ) "' + 1 ( det B ) "' + 1 .
53
TRACE, D ET E RM I N A NT A N D R A N K O F A M ATRIX
(8)
[b;i ] , C
A = [a,j ] , B =
= [ cij ] , D
=
[ d;j ] ( m
d et ( L ( AB ® C' D ) L', ) = '
m
x m
) lower triangular:
m m II (c; ; d;, ) i ( a ; ; b;; ) - i + l , i= I
d t ( L m ( A' ® B + C' ® D ) L ', ] = II ( b;; aii + d;; cii ) . e
(9) A = [a,j] (
i>j
m x m
) lower triangular, n E IN :
n-1
l:) A' )" - I - i @ Ai
det L m
j =O
L�,
= n m ( det A )" - 1 II JLkf · k>l
where if akk
= au
Note: The rules presented in t hi s subsection may be found in Magnus
( 1 988). 4.2.5 (1)
Determinants Involving Both Duplication and Elimination M at rices
A = [a ;j] , B =
[ b ;i J ( m
x m
) lower triangular:
d et( L m ( A' 0 B' ) Dm ) d et ( L m ( A 0 B' ) Dm )
m II i= I
(2)
A, B ( d et (
m x m
Lm(
where
)
:
A® B ) Dm ) =
0
det ( A ) ( det B) m
m- 1 II n=l
C1 1
L]n
Cn t
·.·
if det ( B) = 0 det(C(n l ) if det ( B ) :f; 0
Cnn
denotes the nth principal submatrix of AB - 1 = [c;1 ] .
54
H A N D BOOK OF M AT R I C ES
(3)
A =
[a; j ] , B = [b ;j ] , C = [c;j] ( m x m) det [L m ( A ® B' C)Dm] det [ L m ( A B' ® C' ) D m]
=
lower triangular :
m
II (b; ; c;d a;:' - i + l ,
i= 1 m
= II ci ; ( a;; b;; ) m - i + 1 .
(4) A , B ( m x m) :
i= 1
det( L m ( A ® B) D! ) '
2- m ( m - 1 ll 2d et ( A )( det
where
if det(B)
=
0
B) m II det (C(n) ) if det ( B ) /:.
0
0
m- 1 n=1
C1 1
C1n
are the principal submatrices of C = hi]
= AB- 1 .
Note: The rules presented in this subsection are given in Magnus ( 1 98 8 ) . 4.2.6
Inequalities Related to Determinants
( 1 ) ( Cauchy-Schwarz inequality) A , B (m x n ) : ( det A 8 B) 2 < det ( A8 A )det ( B8 B).
( 2 ) ( H adamard 's inequality) A = [a ij] ( m x m) :
(3)
( det A ) 2 <
m
m
i=1
j=1
II L i a;i l;bs
(Hadamard's inequality) A = [a ;j] ( m x m) positive semidefinite:
det ( A ) <
m
II a;; .
•=1
( 4 ) A = [a;j] ( m x m) : ( 5)
( Fischer's i nequality) A (m x m), B (m x n), C (n D
=
x n) :
[ ,4BH B ] pos1t1ve. d fi mte. C
.
e
=>
det( D)
<
det ( A )det ( C ) .
55
TRACE, DET E RM I N A NT A N D R A N K O F A M ATRIX
( 6 ) A = [a;j ] (m x m ) positive definite:
det ( 7 ) A = [a;j ]
a; 1
(m x m)
>
a;;
0,
i = 1 , . . . , m.
positive semidefinite:
det .
.
.
>
a;;
0,
i
= 1 , . . . , m.
( 8 ) A = [a;j] (m x m) positive definite with eigenvalues k
II .\ ;
i == I
<
)q
<
·
·
· < .\111
k
det
<
II .\m - k + • ·
i =I
In det ( A ) < tr(A ) - m positive semidefinite: ( det A) 1 f m < � tr(A). positive defin i te, B ( m x m ) positive semidefinite:
( 9 ) A (m x m) positive definite:
A (11) A
(10)
(m x m) ( m x m)
det ( A + B ) > det ( A ) . ( 1 2 ) A , B ( m x m ) positive semidefinite:
det ( A + B) > det ( A ) + det ( B ) . ( 13 ) A ( m x m ) positive definite, B ( m x m ) negative semidefinite:
det(A + B )
<
det ( A ) .
( 14 ) ( M inkowski 's inequality ) A '# 0 , B '# 0 ( m x m) positive semidefinite:
[det ( A + B)p! m > ( det A) 1 f m + ( det B) l /m .
( 1 5 ) A , B ( m x m) positive semidefinite , et E IR , 0 < et < 1 :
( det A) " (det B) 1 - o
<
det(etA + ( 1 - et)B).
( 1 6 ) A, B (m x m ) positive semidefinite:
(det A ) l /m (det B) 1 f m
<
1 m
-t r(A B ) .
H A i'<' DBOOK OF � ATRICES
( 1 7) A ( m
x m
)
posi t ive definite, B ( m x m) positive semidefinite:
�
de ( A + B )
< e x p [tr ( A- 1 B )] .
et( A )
( 1 8 ) ( Oppenhe i m 's i nequal i ty ) A . B = [b;j] ( rn x m ) p ositive se m i d efi n i te : det ( A
( 1 9) .-l . B ( rn
x m
A
)
)
m
B ) > det( A ) IT b;; .
8
i=1
p ositive definite:
det( A
:;: ,
,
B ) > det ( A )det( B ) .
( 20 ) ( Ost rowsk i Taussky inequal ity)
(m
x m
:
det � ( A + A 11 ) < l det ( A ) I aLs · det ( A B ) < � [( det A ) 2 + ( det B ) 2 ] .
� ( A + Aff ) p osi tive de fi n i t e
(21 ) .·l . B
(22 ) .·l .
(m B (m
x
m ) re a l :
x m
)
re a l :
d et ( A
:::}
0
B ) < Hdet( A
8
A ) + dt>t( H :.:; B ) ] .
Note: Inequalit ies ( 2 1 ) and ( 2 2 ) are from Neudecker & S huangzlw ( 1 99:� .
1 99 5 ) . The other i nequalit ies m ay b e found in H orn & J oh nson ( 1 98 fi ) an d \hg nus & !\ e u d ecke r ( 1 98 8 ) .
Optimizat ion of Functions Involving a Determinant
4 .2. 7
I n t h is subsection all matrices are real if not stated oth er w i sf' .
(1)
S1
( m x m)
positive de fi ni te with e i ge n val u es ) q < A2 < · · · < A,, an d associated ort honon H al ( m x l ) e i ge n vec t ors u 1 , . . . , l'm :
min{ det ( B'S1 B )
:
max { det( B' n B ) : B ( m m
A 1 · A 2 · · · A11 .
= [v1 , . . . , v ,., ] .
The m i nimizing matrix is B
The
) B' B = /,., } =
x n ,
B (m x
axi m iz i n g matrix is B
n ) , B' B = In } = Am · Am - 1 · · · Am - n + 1 · =
[ vm , . . . , Vm - n + d ·
(2 ) n ( m X 111 ) complex Hermit ian positive defi n i t t• w i t h e i gP n \· alu es ,\ 1 < .\ 2 < · · · < Am and associated orthonormal ( m x l ) eig em·e c t o r s 1' 1 · · · · · 1'm :
m i n { det( B H D B ) : B ( m
x n
Tlw m in imizing matrix is B m
ax { d e t ( B H S1 B )
:
)
com p l e x , B H B
= h , . . . , vn]
B (m
The n t axi rnizing m a t r i x is B
=
=
x
/,. } = A 1 · A2 · · · A, .
n ) co m pl e x B H B
=
..\,.
.
/,., }
. A ,, 1 . . . Am - + 1 . -
[ um , . . . t"rn - n + d .
=
rl
57
TRACE, DETE R M I N A NT A N D RA N K OF A M ATRIX
(3)
Y(
m
x n ) , X (p x n ) , rk( X )
=
p:
A X ) ( Y - AX )'] : A ( x p) } = det [Y(/n - X' (X X' )- 1 X )Y'J .
m i n { det [( Y -
m
A = Y X'(X X' ) - 1 . rk ( X ) = p, R (q x )
The minimum is attained for
(4) Y ( m x n ) , X (p x n ) with
C (q x p) :
min { det[(Y -
m
A X ) ( Y - AX)'] : A (
m
with
x p), RA
= det[( Y - A X ) ( Y
rk( R ) = q ,
C} - A X ) '] , =
YX'(XX' )- 1 + R'( RR' ) - 1 (C - RYX' ( X X ' ) - 1 ). )q < x n ) , rk( Y ) = rk( X ) = < .\ m t he eigenvalues of
w here A =
(5)
Y, X (
m
· · ·
m,
( X X' ) - 1 /2 X Y ' ( Y Y') - 1 Y X' ( X X' ) - 1 1 2 '
with associated ort honormal eigen vectors
v1 ,
. . . , Vm :
BCX ) ( Y - BCX )'] : B ( x r ) , C ( r x ) rk(B ) rk(C) = r } = det( YY ' ) ( l - A m ) · · · ( 1 - A m -r+ I ) .
m i n { det[(Y m
m
=
,
The m i n imum is attained for
- [V m , · · · Cand
,
v vl ) - 1 / 2
Vm - r + l J ' ( ...-\
./\
YX'C' (CXX ' C' ) - 1 . x ) p osi t i ve definite, B diago n al , .\ 1 ( 6 ) B, C ( .\m the eigenval ues of B - 1 1 2 CB - 1 1 2 with associated eigenvectors v 1 , Vm , 1 < r < B
m
=
m
•
.
.
m :
.
m i n { l n det(AA'
< < ort honormal
+ B) + tr[( AA' + B)- 1 CJ =
m
+ I n det ( C) +
m -r
L ( .\ i= I
:
;
A (m x r ) }
- I n .\; -
1 ).
The min imum is attained for
0
1 /2
0
Note: The resu l ts of this subsection m ay be obtained from :\I agnus & Neudecker ( 1 988 ) and Liitkepohl ( 1 991 , Section A . 1 4 ) .
H A N D B O O K O F M AT R I C ES
'i 8
4.3
The Rank of a Matrix
Definitions :
The
rank of a m atrix A , denoted by rk A or rk( A ) , is t he maximum number of l inearly i ndependent rows or col u m ns of A . The col u m n rank of a matrix A , denoted by col rk A or col rk( A ) , is the n u mber of l i rlf'arly independent columns of A . The row rank of a matrix A , denoted by row rk A or row rk( A ) . is t lw nu mber of linearly independent rows of A .
General Result s
4.3.1
( 1 ) :1 ( m x n ) :
row rk( A ) = col rk( A ) = r k( A ) .
(2) :1 ( m x n ) . r E <1' :
r
(3) A (m x n) :
=>
-::j:. 0
rk(rA ) = rk( A ) .
( a ) rk( A' ) = r k ( A ) .
( h ) r k( A ) = r k ( ..l ) .
( c ) r k ( :\ 11 ) = rk( A ) .
( d ) r k ( A + ) = rk( .4 ) .
(4)
:1
( e ) rk ( A - ) = r k ( A. ) (m x m) :
{:=::}
A- AA- = A - .
m 1 0
rk( :l a dJ ) = (5) A (m x n) :
( a) rk( A A ' ) = rk( A ' A )
=
if if if
rk( A ) = m rk( A ) = m - 1 rk( A ) < m - I
rk( A ) .
( b ) rk( A .-\11 ) = rk( A 11 A ) = rk( A ) .
( 6 ) a . b ( nr x l ) : (7 )
:1
(m
a , b -::j:. O => rk( a b' ) = l
11 ) : rk( A ) = r· => ( ( r· x n ) such t hat ..1 = BC.
.
t h ere exist
x
rna
t rices B ( m x 1' ) a n d
'
( 8 ) :l ( m x n ) . B ( n x n ) :
( 9 ) .-\ ( m x n ) . B ( m x
m
)
:
H
nonsi ngu l ar
=>
rk( A B ) = rk( .·1 ) .
B nonsingular => rk ( BA )
=
( 1 0 ) A ( m x n ) , lJ ( m x m ) , C ( n x n ) : H, C nonsi ngular => rk( B A C ) ( 1 1 ) A ( m x n ) , B (p x q) :
( 1 2 ) :1 ( m x m ) . B ( n x n ) :
rk( A
·:.Y
rk( A
=
rk( A ) .
B ) = rk( A )rk ( B ) .
tf'
B ) = rk( A ) + rk( B ) .
rk ( .4 ) .
TRACE, D ET E RM I N A N T A N D R A N K
(m A (m
B (r
( 13 } A
x
n),
( 14 }
x
n) :
x
OF
m) : rk(B) =
r,
59
A M ATRIX
rk(A) = m
=>
rk( BA) =
r.
(a) rk(A) = m => A + = A H ( AA H ) - 1 . (b) rk(A) n => A+ = ( A H A ) - 1 A H . =
( 15 ) A
(m m) : x
( a) rk( A ) < m - 1 => A a dj = 0 . (b) rk( A ) = m {::::::} det(A) ::j:. 0 . (c) rk(A) = {::::::} A is nonsingular. A (m n) : rk(A) = 0 {::::::} A = 0 . A ( m m) i dempotent : (a) rk(A) = m => A = Im . (b) rk(A) = tr(A ) . (c) rk(/m - A) = m - rk(A ) . rk(/m ) = m. Kmn (mn mn) commutation matrix: rk ( Kmn ) = m n . Dm ( m 2 x � m (m + 1 ) ) duplication matrix: rk(Dm ) = � m ( m + 1 ) . L m ( � m(m + 1 ) m 2 ) elimination matrix: rk( Lm ) = � m(m + 1 ) . A (m x m) : ( a) rk( . Um - A) < m {::::::} � is an eigenvalue of A . ( b ) 0 is a simple eigenvalue of A => rk(A) = m - 1 . A ( m n) : rk (A) = r => rk( I<mn (A' 0 A)) = r 2 . A (m m), A;j ( n n ) submatrix of A : rk(A) = r, n > r => det( A;j ) = 0 . A ( m m) : ( a) rk(A) = r => there exist (m m) matrices B; , i = l , . . . , r, with rk( B; ) = 1 and A = B 1 + + Br . (b) rk(A i ) = rk( A i + l ) for some i E IN => rk(A ; ) = rk ( Ai ) for all j > i. (c) There exists an i E IN such that rk(A i ) rk( A i +l ) . ) A (m m) nonsingular, B (m n ) , C ( n x m), D ( n m
( 16 } ( 17}
( 18} ( 19 ) (20) (21) (22)
(23) ( 24} ( 25)
x
x
x
x
x
x
x
x
x ·
·
·
=
(26)
x
x
x n
:
60
H A N DBOOK O F M ATRICES
Note: M any of these results are elementary and fol low immediately from ddi nit ions ( see. e.g . , text books such as Horn & Johnson ( 1 985) and Lancaster i< Tismenetsky ( I 985 ) ) . See also 1\I agnus ( I 988 ) for some of t he results. Matrix Decompositions Related to the Rank
4.3.2
( 1 ) (Singular value decomposition) A ( m x n ) , rk( A ) = 7', a 1 , . , 0"r "# 0 singular values of A : exist u ni tary m at rices U ( m x m ) , V ( n x n ) such that .
.
o
0
J
vn
There
'
where D = d iag(a 1 ar ) and some or all of t he zero submatrices d isappear when 7' = m and/or 7' = n . •
(2)
.4
.
.
.
•
( m x m ) symllletric :
(m x
111
) mat rix
T
rk( A ) = such that A =
T
7'
¢::::::>
[6 �]
there exis t s a nonsingular T' . 7'
where the zero submat rices d isappear when
= m.
( 3) ( Di agonal reduction ) .-1 ( m x 11 ) : rk( A ) = 7' ¢::::::> there exists a nonsingular ( m x m ) Juat rix S and a nonsingular ( n x n ) m atrix T such that A=S
[6 �]
T,
w lwrt> some or all of the zero submatrices disappear when 7' 111 and/or r = 11 . ( 4 ) A ( 111 x m ) : rk( A ) = rk( A 2 ) ¢::::::> there exist nonsingular matricPs P and D such that
( 5 ) ( Rank fac torization ) A ( m x n ) : rk( A ) = 7' ¢::::::> there exists an ( m x r ) matrix B and an ( r x u ) matrix C w i t h rk( B ) = rk(C') = 7' such t h at A = BC. ( 6 ) ( Triangular factorization) A = [a,1) ( m x m ) . rk( A ) = det
7' :
=f. O ,
i = l,
.
.
.
.
7'
TRAC E, D ET ER M I N A N T A N D R A N K
OF
A M ATRIX
61
t here exists a lower triangular ( m x m) m atrix L and an upper tri angular ( m x m ) m atrix U one of w hich is nons i ngular such that A = LU. =>
( 7 ) ( Polar decompositio n ) A ( m x m ) : rk( A ) = r ¢::::::> there exists a positive semidefin ite ( m x m ) m atrix P with rk( P ) = r and a u n i t ary ( m x m ) m atrix U such that A = PU . Note: For these and further decomposition theorems see Chapter 6. 4.3.3
Inequalities Related to the Rank
( 1 ) A ( m x n) :
( a) r k ( A ) < m i n ( m , n ) . ( b ) r k ( A - ) > rk( A ) .
( 2 ) A ( m x n) , B ( n x r) :
( a) r k ( A B ) < m i n { rk( A ) , rk(B) } .
( b ) r k ( A ) + r k ( B ) < rk( A B ) + n . ( c ) A B = 0 => rk(A) + rk( B ) < n . (3) A, B ( m x n) :
rk( A + B ) < rk ( A ) + rk( B ) .
(4) A (m x n) , B ( m x r) : (5) A (m x n) :
rk( [A : B] ) < rk( A ) + rk( B ) .
Ax = 0 for some ( n x 1 ) vector
( 6 ) ( Sy lvester's law) A, B (m x m) :
x
f. 0 => rk( A ) < n - 1 .
rk( A ) = r. rk( B ) = s => r k ( A B ) > r + s - m .
( 7 ) A , B ( m x m ) H ermitian : ( 8 ) A ( m x m ) Hermitian :
rk( A 8 B ) < rk( A ) rk ( B ) . rk( A ) > [tr( A )] 2 /tr( A 2 ) .
( 9 ) ( Frobenius inequality ) A ( m x n) , B ( n x r), C (r x s) : rk( A B ) + rk( BC) < r k ( B ) + rk(ABC). Note: These results c an be found i n m any m atrix textbooks ( see, e .g., Horn & Johnson ( 1 985 ) or Rao ( I 973 ) ) or fol low eas i ly from rules given t here.
5
Eigenvalues and Singular Values In this chapter all m atrices are complex unless otherwise stated .
5.1
Defi n itions
For an ( m x m) matrix A the polynomial in � . PA ( � ) det ( Mm - A ) . is c alled the characteristic polynomial of A. The roots of PA ( � ) are said to be the eigenvalues , characteristic values, characteristic roots or latent roots of A. Let � 1 , . . . , � n be the distinct eigenvalues of the ( m x m ) matrix A . Then the characteristic polynomial can be represented as =
where the m; are positive integers with L � 1 m , = m . The number m ; is the multiplicity or algebraic multiplicity of the eigenvalue �i , i = 1 , . . . . 11 . A n eigenvalue is called simple, if its multiplicity is 1 . The geometric multiplicity of an eigenvalue � of an ( m x m ) m atrix A is t he number of blocks 0 � 1 0 0
�
0
0
0
0
0
1 �
with � on the principal d i agonal, i n the Jordan decomposition of A which is given in the next section (see also Section 6. 1 ) . For an ( m x m ) m atrix A . p ( A ) = max { l � l abs : � is an eigenvalue of A }
H A N D BO O I\ OF �t. �'I' R i f.ES
64
is t he spectral radius of A . The set { � : A is eigenvalue of ..4 } is t hP spectrtiiJl of .A . For an ( 111 x 1n ) n1atrix ..4 with eigenvalue A any ( 1 n x 1 ) vector r' f. 0 satisfying ..4 r' = A r is said to be an eigenvector or characteristic vector of .·\ corresponding to or associated with the eigenvalue A. For t\vo ( 1 n x 1 n ) 1natrices ..4 and B a root of the polynon1ial PA .n(A) = det ( A B ..4) is sotu eti tnes called eigea1value or characteristic value of .4 i11 tl1e 111etric of B. A ccordingly for an eigenvalue A of A in the n1etric of lJ an ( 111 x 1 ) vector r :j:. 0 sati sfy ing ..4t' = ABv, is called an eigenvector or el1aracteristic vector of .4 in the 1netri(� of B corresponding to or associated with A. For a n ( 111 x 11 ) 1nat rix .4 thP nonnegat i ve squarP roots of t he eigenvalut�s of 1 11 1 if 111 > 11 , and of .4 ..4 H , if 111 < 11 , are said to be t h e singtiiar val11es of :\ . -
·
-
.
.
.
,
.
P roperties of Eigenvalues a nd Eigenvectors
5.2
Notatio11:
denotes an eigenvalue of tht., tnatrix .4 .
A( .4 ) A
( .A )
nu n
An,a.r ( ..4 ) ..
is t.he sJnallest eigenvalue of thP rnatrix .4 if all eigen values are real . is t he l argest eigenvalue of t he n1atrix ..4 if all eigenvalues are real .
General Results
5.2 1
( 1 ) .4 ( 11l x 17l ) : 1 has ex ac tly 11l eigenvalues if the ntult.iplicit.it.,s of t h{"' roots are taken i nto account. .-
( 2 ) .4 . B ( 111 x 1n ) : .4 , B h ave the san1e eigenvalues � tr( ..4 k ) t r( B k ) k = 1 . . , 1n . =
( 3)
(4)
.-1
4
..
( 1u
x 111
.
.
) \V it h t")igenvalue
( a ) ,\ is eigenvalue of
.4
A
.
and associated eigeHV<.,ctor
\vith eigenvector
r- :
v.
( b ) ,\ i is eigt>nval ue of .4 r with eigenvect or 1 ' for i E Lr\J . ( c ) A - 1 is eigenvalue of ..4 - 1 wit.h eigenvt,.ctor r: , if ..4 is nonsingular. ( 11l
X Ill
)
:
( a ) A is eigenvalue of 4 � ( b ) ,\ is Pigenval ue of
( c ) ,\ is f'igenvalue of
..
.4
A
is eigenvalue of A'.
� ,\ is eigenvalu(;) of ..4 H .
.A =>
A+
T
is eigenvalue of
.4 +
.
T
l,n .
EIG ENVA L U ES A N D S I N G U LA R VA L U ES
( 5 ) A = [a;j ] ( m x m) : ( a ) A = diag( a , , , . . . , a mm ) => a l l , · · · · a mm are the eigenvalues of A. (b) A is triangular => a 1 1 , . . . , am m are the eigenvalues of A . ( c ) A is H ermitian => all eigenvalues of A are real numbers. ( d ) A is real symmetric => al l eigenvalues of A are real numbers . (e) A is idempotent => all eigenvalues of A are 0 or 1 . (f) A is nilpotent => all eigenvalues of A are 0 . (g) A is real orthogo n al => all eigenvalues of A are 1 or - 1 . ( h ) A is singular {::::::} 0 is an eigenvalue of A . ( i ) A is nonsingular {::::::} all eigenvalues o f A are nonzero . ( 6 ) A (m x m ) Herm i ti an : ( a ) A is pos i tive definite greater t h an 0 .
{::::::}
all eigenvalues of A are real and
( b ) A is positive semidefinite {::::::} all eigenvalues of A are real and greater than or equal to 0 .
( 7 ) A ( m x m) real sy mmetric : ( a) A is pos i tive definite greater t h an 0 .
{::::::}
all eigenvalues of A are real and
( b ) A is positive sem idefi nite {::::::} all eigenvalues o f A arc real and greater t h an or equal to 0 .
( 8 ) A ( m x m ) , B ( m x m) nonsingular: >. is eigenvalue of A {::::::} >. is eigenvalue of s- 1 A B .
( 9 ) A ( m x m ) , B ( m x m) u nitary : >. is eigenvalue of A {::::::} A is eigenvalue of BH AB.
( 10 ) A ( m x m ) , B ( m x m) orthogonal : >. is eigenvalue of A {::::::} >. is eigenvalue of B' A B . ( 1 1 ) A ( m x m ) , B ( m x m ) positive definite: >. is eigenvalue of BA {::::::} A is eigenvalue of B 1 1 2 A B 1 1 2 . (12) A (m x n), B (n x m), n > m : ( a) A is eigenvalue of A B => A is eigenvalue of BA.
( b ) >. -:f 0 is eigenvalue of BA => >. is eigenvalue o f A B . ( c ) >. 1 . . . . , Am are the eigenvalues of A B => .>. , . . . . . A 111 • 0 . . . . . 0 are the f.'igenvalues of BA .
65
H A N DBOOK O F M AT R I C E S
66 ( 13 ) A ( m x m) ,
c E
( a) >. is eigenvalue of A with eigenvector c A with eigenvector v.
(b)
v
::::}
c>.
is eigenvalue of
1'
is eigenvector of A corresponding to eigenvalue >. eigenvector of A corresponding to eigenvalue >. .
( 14 ) A ( m
x m ) , c 1 , c2 E
c2 # 0
correspondi ng to an eigenvalue >. ::::} A corresponding to eigenvalue >. .
::::}
111
) . ') o . ') 1
.
v 1 , v2 eigenvectors of A c1 t•1 + c2 t'2 is eigenn"C'tor of
( 1 5 ) A (m x m ) w i t h eigenvalues A; , Aj and associated eigenvectors 0 : A; -:f Aj ::::} v; and Vj are linearly independent.
( 1 6 ) ..\ ( 111 x
cr IS
r, . r-1
#
'/p
E . is eigenvalue of A with eigenvt•ctor r ::::} 1o + ') 1 A + · · · + ')p AP i:; eigenvalue of 1 o lm + 1 1 A + · · · + 1 r ··V' w i t h eigenvect or v . •
.
.
.
.
(m x m ) ,
B ( n x n ) , >. ( A ) , >. ( B ) eigenvalues of A and B , respectively, with assoc iated eigenvect ors t' ( A ) and r( H ) . respectively:
( 17) A
( a ) A( A ) · A( B ) is eigenvalue of A 0 B with eigenvector v ( A ) C r( H ) .
( b ) A ( .-1 ) aud A ( B ) are eigenvalues of A
[ t'\)4) ]
and
,
m
( 19)
=
>. 1 · · · A
m
=
B with eigenvectors
[ v(�) ] , respectively .
( 1 8 ) A ( 111 x m ) wit h eigenvalues >. 1 , . . . A m ( a) det ( A )
t3
:
IJ A; .
i=I
m
( b ) t r( A ) = >. 1 + · · · + A m = L A; . i= I
A (m x m ) :
nonzero eigenvalues ::::} rk( A ) > r. ( b ) rk( >. lm - .4 ) < m - 1 ¢::::::> >. is eigenvalue of A. . ( a) A h as
1'
( c ) 0 is a simple eigenvalue of A
::::}
rk(A. ) = m - 1 .
( 20 ) A ( m x m ) with eigenvalues >. 1 , . . . , Am : ( a) I A; I abs < 1 , i = 1 , . . . , m ¢::::::> det ( lm - A z ) # 0 for l z l abs < I , z E
(c)
i
= 1, .
I A ; Iabs < l . i = l ,
.
. .
. .
,m
::::}
Im ± A is nonsingular.
, m ::::} Im, ± A 0 A is nonsingu l ar .
67
EIGENVA L U ES A N D S I N G U L A R VA L U E S
( 2 1 ) (Cayley - Hamilton t heorem) A (m x m)
:
)q, . . .
, A m are eigenvalues of A
m
=>
IJ (>. Im - A ) ;
=
O.
i= I ( 22 ) A ( m x m ) : The geometric multiplicity of an eigenvalue >. of A , that is, the number of Jordan blocks with >. on the principal diagonal, is not greater than the algebraic multiplicity of .>.. (See Section 5.2.3 for the Jordan decomposition . ) Note: Most of the results of this section can be found in Horn & Johnson ( 1 985 ). The others are easy to derive from results given there . Chatelin ( 1993) is a reference for further results on eigenvalues and vectors including computational algorithms. 5.2.2
Optimization P roperties of Eigenvalues
( 1 ) ( Rayleigh - Ritz theorem) A ( m x m ) Hermitian :
{ X H Ax XH X { x H Ax A m ar ( A ) = max A m i n ( A ) = min
XH X
: x (m
( 2 ) ( Rayleigh - Ritz theorem for real matrices) A ( m x m ) real symmetric:
x'Ax { A m i n (A) = min x'x x'Ax Am ax (A ) = max { xx
} }
x ( m x 1 ) real, x f. 0 , x ( m x 1 ) real , x f. 0 .
'
(3)
} x 1 ) , x =f. O } .
x ( m x 1), x =f. O ,
( Courant - F ischer theorem) A ( m x m ) Hermitian with eigenvalues A1 < ···< Am 1 < i < m : .>. ; =
y,,
min
, y ., _ , ( m x I )
max
{ xHAx
'
:
H
X X
X ( m X 1), X f. 0 , X H Yj = O , j = 1 , . . . , m - i
and A; =
max min l y 1 . . . . , y , _ 1 (m x )
{ x H Ax H
X X
:
}
}
X ( m X 1 ) , X f. 0 , X H Yi = 0 , j = 1 , . . . , i - 1 .
68
H A N DBOOK O F M ATRIC ES
( 4 ) ( Courant - F ischer t heorem for real m atrices ) .-\ ( 111
m :
x
m ) real >;yrumetric with eigenvalues A 1 < · · · < Am . 1 <
>.; =
Yl ·
(m x
.
ll1lll
x
· · Y111 - 1
l)
real
m ax
{ x' Ax,
max
!!,q ,
(m J.'
x
t)
. Yt - 1
real
Jll \ll
{ x' Ax x
'x
<
:
X X
( m x 1 ) real. x -:f 0, x'yi = O, j = 1 , . . . . m
and
i
-
i}
:
}
( 1 11 x 1 ) real . x -:f 0 , x'Yi = 0 , j = 1 , . . . . i - 1 .
m ) H e r m i t i a n with e i ge n valu es >. 1 < · · · < Am and associated ort honormal e i ge n vect o rs t ' t , . . . , t>m . n E { 1 . . . . . Ill } :
( 5 ) .-\ ( m
x
rnin{ tr( X H A X ) : X The
m
( m X n) ,
x H X = In } = A t + . . . + >., .
i n i m iz i ng m atrix is X = [vt , . . . . t•nl ·
axi m i z i n g m at rix is X = [vm . . . . . t 'm - n + d · ( 6 ) .-\ ( m x m ) real symmetric with eigenvalues A t < · · · < >.,, and as so c i a t e d ort honormal eigenvec tors l' t , . . . . l'm · 1 1 E { 1 . . . . . m } : The
n
m
t i n { t r ( X ' A X ) : X ( m x n ) real, X ' X = In } = A 1 + · · · + >." .
The m i n i m iz i n g matrix
is X = (
11 1 ,
m ax{ t r ( X ' .-\ X ) : X ( Ill x n ) real ,
( 7)
T he !ll aximizing matrix is
. . . , l'n ] . X ' X = I11 } = >.,, + · · · + Am - n + t ·
X = (1•,. , . . . , l'm - n + d ·
A ( m x m ) Hermitian with eigenvalues 0 < )' 1 < { 1 . . . . . m} : >. t . >. .,- . . . >.
< >., .
"
min{ ( J.· f1 A x t ) · · (x� Axn ) ( m X 1 ) . X = [J.' t . . . . , J.'n ] ( m ·
J.',
X n),
x /{ x
!, } .
11
E
69
EIGENVA LU E S A N D S I N G U LAR VA L U ES
( 8 ) A (m x m) Hermitian positive definite with eigenvalues ) q < A 2 < · · · < A m and associated orthonormal ( m x l ) eigenvect ors v 1 , Vm : .
.
.
•
min{det( B H A B ) : B ( m x n) , B H B = ln } = A t · A 2 · · · A, . The minimizing matrix is B = [ v, , . . . , Vn ] · max{det ( B H A B ) : B (m x n) , B H B = ln } = Am · A m - l · · · A m - n + l · {9)
The m aximizing matrix is B = [v m , . . . , V m - n + d · A ( m x m) real symmetric positive definite with eigenvalues A 1 < A 2 < · · · < Am and associated orthonormal ( m x l ) eigenvectors min{det( B' AB ) : B ( m x n ) real, B ' B = ln } = A , · A2 · · · An . The minimizing matrix is B = [v, , . . . , v n ] . max{ det ( B' A B ) : B ( m x n ) real, B ' B = /, } = Am ·A rn - 1 · · · Arn - , + 1 · The maximizing m atrix is B = [vrn , . . . , Vm- n + d ·
Note: The results of this section may be found in Horn & Johnson ( l 98.') ) and partly i n M agnus & Neudecker ( 1 988 ) . Matrix Decompositions Involving Eigenvalues
5.2.3
( 1 ) (Schur decomposition)
( m x m) : There exists a unitary ( m x m) matrix U and an upper triangular matrix A with the eigenvalues of A on t he principal diagonal such that A = U AU H . (2) (Schur decomposition of a real matrix) A ( m x m) real with real eigenvalues : There exists a real orthogonal ( m x m) matrix U and a real upper triangular matrix A with the eigenvalues of A on the principal diagonal such t hat A = F A C ' . ( 3 ) ( Spectral decomposition of a normal matrix) A ( m x m) normal with eigenvalues A 1 , . . . , A m A = U AU H , w here A = diag( A 1 , . . . , A m ) and U is the unitary ( m x m) matrix whose columns are the orthonormal eigenvectors v1 , . . . , Vm of A associated with A 1 , . . . , A m . In other words, m A = L A; vfl . A
:
i =I
v,
70
H A N D BOOK OF M AT R IC E S
( 4) ( Spectral decomposition of a Hermitian m atrix) A = U AC H . .4 ( m x m) Hermitian with eigenvalues A 1 , . . . , Am where A = diag{ A 1 , . . . , Am ) and U is the unitary ( m x m ) matrix whose columns are the orthonormal eigenvectors V t , . . . , V m of A associated with A t , . . . , Am . In other words,
m
A=
L A; v; vfl . i= t
( 5) ( Spectral decomposi tion of a real symmetric matrix) A ( m x m ) real sy mmetric with eigenval ues A t , . . . , Am : A = { ' A [ 0 where A = diag( A t , . . . , Am ) and U i s the real orthogonal ( m x m ) matrix whose colu m ns are the orthonormal eigenvectors v 1 , . . . , t:m of A associated with A t , . . . , Am . In other words,
m
A=
L A; v, v; . i= t
J ordan Decompositions ( 6 ) A ( m x m ) with distinct eigenvalues A t , . . . , A n : There exists a nonsi ngular ( m x m ) matrix T such that A = TAr - t , where
A=
is block diagonal with blocks 0 1 A; =
0
0
0
0
0 0
i = l , . . . , k > n.
on the diagonal and { n t , . . . , nk } = { l , . . . , n } , that is, the same eigenvalue m ay appear on the diagonal of more than one :\. , . ( For more details see Chapter 6 . ) ( 7 ) A ( m x m ) with m distinct eigenvalues A t , . . . , Am : A = TAT - t , where A = diag( A t , . . . , Am ) and T is a nonsingular ( m x m ) matrix whose columns are eigenvectors of A associated with A 1 , . . . , Am .
il
EIGENVA L U ES A N D S I N G U L A R VA LUES
( 8 ) ( Near-diagonal Jordan decomposition ) A ( m x m) with distinct eigenvalues A1 , . . . , An , t > 0 a given real number: There exists a nonsingular ( m x m) matrix T such that A = TAT- • , w here
A = 0 is block diagonal with An , 0
An ,
0
0
A; ( t ) =
0
.
{
i = I, . . . , k > n, {
0
and { n 1 , . . . , nk } = { I ,
0 0
0
{
.
.
.
.
An ,
. , n} .
( 9 ) ( Real Jordan decomposition) A ( m x m) real with distinct real eigenvalues A1 , . . . , Ap and dist inct There exists a real complex eigenvalues a 1 ± i/11 , . . . , a q ± i/1q : nonsingular ( m x m) matrix T such that A = TAr- • , where 0 A =
Note: The results of this section can be found in Horn & J ohnson ( 1 9Ki> ) .
Set•
Chaptt•r
5.3
for furt her matrix decomposition results.
Eigenvalue Inequalities
Inequalit ies for t he Eigenvalues of a S ingle Matrix
5.3.1
(1)
6
(
.-l
m x m
)
H � · r mit ian,
.l'
:f. 0
(
m x
.
A 7l
)
x
{ �
( 111 x
111
:
\· II \"
'
'
( ..\ m ( A ) ,
(4) A
X
}
m x
{
}.
x
X
X
I ' . . . . 17!.
I =
{3)
:
11 A x x < Ama.r ( A ) . Amin ( A ) < H m ) Hermitian with eigenvalues ..\ 1 ( A ) , . . . , .Arn ( A ) : xH x : x ( I) , x :f. 0 uin t H x < ..\, ( A ) < max x11Ax 11 : x ( 111 l ) . x :f- 0 X
(2) 4 (m
I)
m
x
)
Hermit ian with eigenvalues ..\ I ( A ) < · - - < .Am ( A ) .
- I
"
m
)
X (m
X ' X = l,
i=I
i= I
..\ J ( A )
real symmetric with eigenvalues n ) real:
x
n
i=I
n
<
X (m x
<
73
EIG ENVALU ES A N D S I N G U L A R VA L U E S
( 5 ) A = [a ; j] ( m
x m
) Hermitian: i=
Amin ( A) < a;; < ..\max ( A ) , ( 6 ) A = [ a;j] ( m x
m
[a;j ] (
m x
Am (A) :
.
.
. , m.
) Hermitian with eigenvalues ..\ 1 ( A ) <
n n n L ..\; ( A ) < L a;; < L ..\m -n+i ( A ) , i= I i= 1 i= I
( 7) A =
1,
m
n
=
·
1, .
·
·
< ..\m ( A ) :
. . , m.
) Hermitian with eigenvalues 0 < ..\ 1 ( A) <
n n II ..\; ( A ) < II a;; , i=l i=l
n = 1,
<
. . . , m.
( 8 ) A ( m x m ) Hermitian positive definite with eigenvalues ..\ J ( A) < · · · < Am ( A) , X ( m x n ) :
(9) A (m
n
n
i= 1
i= I
) real pos 1 t 1 ve definite with eigenvalues ..\! ( A ) < Am ( A ) , X ( m x n ) real : x
m
<
n n II ..\ ; ( A) < det(X ' A X ) < II Am - n + i ( A ) . i=l i=l
X' X = In
( 1 0 ) A = [ a;j] ( m x m ) Hermitian positive definite with eigenvahws ..\ 1 ( A ) < · · · < Am ( A ) :
A(n) =
(11)
n
n
i= I
i=l
(Incl usion principle) ) Hermitian with eigenvalues ..\ 1 ( A ) < < A ( ..\m (A) , A( n ) ( n n ) a principal submatrix of A with eigenvalues ..\ 1 ( A ( n J ) < · · · < ..\n (A(n) ) : m
x
m
x
..\ ; ( A ) < ..\ ; ( A( n ) ) < Am -n+i ( A ) ,
i
=
1,
.
. .
, n,
14
H A N DBOOK OF M AT R I C ES
Relations Between Eigenvalues of More Than One Matrix
5.3.2
( 1 ) A ( m x 111 ) Hermi t i an. B ( 1 1 1 x m ) positive semidefinite:
R) > Amin ( A ) . Amar ( A + R ) > Amar ( A ) .
( a ) Am•n ( A +
(b)
( 2 ) A . R ( m x m ) Herm itian:
( a) ( b)
{ 3 ) ..1 .
l3
Amin ( A + B ) > Amin ( A )
+
Ami n { B ) .
Amar ( A + B ) < Ama.r ( A ) + Amax ( B ) .
( rn x m ) H ermitian , 0 < a , b E IR :
( a ) ..\"'"' ( a A
( b ) Am a .r ( a A
+
+
b B ) > a ..\ m;,. ( A ) + 6 ..\m; ,. { B ) . b R ) < a A mar ( A ) + b..\mar ( B ) .
( 4 ) A ( m x m ) Hermitiau w ith eigenvalues A t ( A ) < · · · < A m ( A ) . B ( m x 111 ) Ilermitiau positive sem idefinite . ..\ d A + B ) < · · · < Am ( A + H ) eigeuvalues of A + B : ..\; ( A + B ) > ..\; ( A ) ,
(5)
i
=
:\
1 , . . . , m.
( 1n x m ) Hermitian with eigenvalues ..\ 1 ( A ) .::; · · < A m ( A ) . J' ( 111 x I ) . A , ( A ± J'J' 11 ) < · · · < ..\m ( A ± J.' .I' 11 ) eigenvalues of A ± .u11 : ·
A; ( A. ±
n:11 )
< A;+ I ( A ) < A; + 2 ( A ±
.r.IJ1 ) .
..\, ( A ) < A , + I ( A ± x.r 11 ) < A;+2 ( A ) ,
i
i
=
1 , 2, . . . .
1 , 2, . . . , m
=
m -
-
:! :
:! .
< Am ( A ) { 6 ) .4 . B ( m x 111 ) Hermitian , r k ( B ) < r , A I ( A ) < · · eigenval ues of A . A I ( A + B ) < · · < Am ( A + B ) eigenval ues of A + B : ·
·
A; ( A + B ) < A;+,. ( A ) < A, + 2r ( A + B ) . A, ( A ) < A; + r ( A + B ) < ..\ ; + 2r ( A ) ,
i i
=
( 7) ( Poincare's separation theorem)
=
1 . 2 , . . . . m - 'b· :
1 , 2 , . . . . m - 2r.
A ( m x 1 1 1 ) Hermitian with eigenvalues ..\ ! ( A ) < · · · < ..\,. ( A ) , X ( m x n ) such t hat X 11 X = /,. , 11 < m, A 1 ( X 11 A X ) < . . . < A, ( X H A X ) eigenvalues of X 11 A X : A; ( A ) < A; ( X 11 A X ) < Arn- n +i ( A ) ,
i
=
1 . . . . . n.
( 8 ) A . B ( 1 1 1 x m ) Hermitian with eigenvalues A d A ) < · · · < ..\m ( A ) aud A I ( B ) < · · · < Am ( B ) . eigenvalues of A + B :
( a)
respectively ; n
..\ ! ( A + B ) <
L [ A i ( A ) + ..\; ( B ) ] < L ..\ i ( A + B ) , i= l
·
· · < ..\m ( :l + B )
n = 1, . . . . m.
75
EIG ENVA L U ES A N D S I N G U LAR VA L U ES
n = 1, . . . , m.
( b ) ) q ( B ) < ..\n ( A + B) - ..\n ( A ) < ..\m ( B ) ,
( c) I .An ( A + B ) - An ( A ) I abs (d ) (9)
< max { l..\j ( B ) I abs : j = 1 , . . . , m } = p( B) , n = l , ..\n ( A + B) < min{ ..\;( A ) + ..\i ( B) : i + j = n + m } , n = 1 , . . . , m.
. .
. , m.
(Weyl 's theorem ) A, B ( x ) Hermitian with eigenvalues ..\ 1 ( A ) < · · < ..\m ( A ) and ..\ 1 (B) < · · · < ..\m ( B ) , respectively ; ..\ ! ( A + B) < · · < ..\m ( A + B ) eigenvalues of A + B : m
m
·
·
..\; ( A + B) >
and ..\; ( A + B) <
i= 1
..\; ( A ) + ..\! ( B) ..\; - 1 ( A ) + ..\ 2 ( B)
..\; ( A ) + .Am ( B) ..\;+ I ( A) + ..\m - ! ( B)
, . . . , m.
(m x
) Hermitian with eigenvalues ..\ 1 ( A ) < · · · < .Am ( A) and ..\ 1 ( B ) < · · · < ..\m ( B), respectively ; ..\ ! (A - B) < · · · < Am ( A - B ) eigenvalues of A - B :
( 10 ) A , B
m
..\ 1 ( A - B) > 0
=>
..\; ( A ) > ..\; ( B) , i
=
1, . . .,
m.
( 1 1 ) A ( m x m ) Hermitian with eigenvalues ..\ I ( A) < · · · < ..\m ( A ) , 1 ) , c E IR, A B=
x [ xH ]
x(
m
x
c
with eigenvalues ..\ 1 ( B) < · · · < ..\m +l ( B) : ..\; ( B) < ..\; (A) < ..\;+ • ( B ) ,
i = 1,
. . . , m.
( 1 2 ) A ( n x n ) Hermitian with eigenvalues ..\ I ( A) < · · · < ..\n ( A ) , D (p x p) Hermitian, C (n x p) and B
=
[ �H � ] (
m X m
)
with eigenvalues ..\ ! ( B) < · · · < ..\m (B) : ..\; ( B ) < ..\ ; ( A ) < Am - n+i ( B)
for i = 1 , . . . , n.
i'6
H A N D B O O K O F M AT R I C ES
The results of this section are given in Horn & Johnson ( 1 98 5 . C 'hapt ers 1 and 7 ) or follow easily from results given there. Some results. in part icular t hose on real sy nunetric matrices, may also be found in �Iagnus l'i.: :'ieudecker ( 1 988 ) . :\-I any of these and further results are also rontained in C 'hatelin ( I 993 ) . Note:
5 .4
Results for t he Spectral Radius
aij
.4 > ( > ) 0 A > (>) B
a,j
.
( 1 ) .4 ( 111 X 111 ) . i E N (2)
> ( >) 0. i = 1, . . > ( > ) b;1 , i = 1 , .
.
.
, m.
1 , . . . . 11 . j = 1, . ..
j =
. , m,
.
n .
.
: p( A ) l = p( A l ) . A . R ( 111 x m ) Hermitian with eigenvalues .\ ! { A ) < · · · < A m ( A ) and .\ t ( H ) < · · · < A, ( B ) . respectively: A t ( A. + B) < · · < Arn ( --1 + B ) ••igemalu•·s of :l + B : ·
f .\ , ( .4 + H ) -
(3)
A
( 4 ) :1
( 111 x m ) :
( a)
p( A )
<
1 ¢::=:} A" � 0 :::>
( h ) p( A )
<1
( C ) p( A )
<
( d ) p( .·1 )
=>
(e) p( A ) < l
=>
= [a,1] ( 111
( a)
1 :::>
x m)
i
-X i ( A. ) l abs < p( B ) .
L>-\' t =O
�
•X ,
i.e
.
. A
1
=
( Im
+ A)- 1 .
L Ai 0 A i = { lm2 - .4 0 A. )- 1 . i=O L( -A)i 0 A i = ( lm2 + A C A )- 1 . i=O
re-al :
.4 > 0 => p( A ) > O .
( b ) A > 0.
L a;1 j=1
> 0.
i = l, . .,m .
( c ) A stochast ic => p( A ) > 0. ( d ) .·l dou bly stochil.Stic => p( A )
=> p( A ) > 0 .
> 0.
m.
is ronvergent .
( lm - A ) - 1
=
L( -A )
as n
= I.. . ..
77
EIG E N VA L U ES A N D S I N G U L A R VA L U E S
(e) A > 0, A i > 0 for some i > 1 => p( A) > 0. ( 5 ) A, B ( m x m ) real: (a) 0 < A < B => p( A ) < p( B) . ( b ) 0 < A < B => p( A) < p( B) . ( 6 ) A = [ a;j] ( m x m ) real,
principal submatrix of A : A > 0 => p ( A ( n ) ) < p( A ) , n = 1 , .
( 7 ) A = [ a;j] > 0 ( m x m ) real : (a) . max a ;; < p( A ) . 1 = I , . . . ,m m
m
; =I
;=1
. . , m.
(b) _min L aij < p( A ) < 1. _-max L a•j · 1 , . ,m . 1 - l , . . . ,m .
(d)
0
( 8 ) A = [ a;j] >
(a)
(m
x m
m
) real, x = ( x 1 , . . . , x, )' > 0 m
1 """' 1 """' min a; ) x) < < . max - L.., a I ) x) . - p( A ) t = 1 , , m X i L.., t = l , . . . , m Xi . . ·
·
; =1
(b) (9) A (m
(m x 1 )
m
min Xj L 1 m
. ;= , x
,
rPal :
.
;=1
a;i
X; . z=l
< p( A ) < . max x1 = i , . . . ,m
m ) real, A > O , x ( m
;
x
1)
m
a ;i L . . Xj •=I
real , x > O . a , b E IR, a , b > 0 :
(a) ax < Ax < bx => a < p( A ) < b. ( b ) ax < Ax < bx => a < p( A ) < b. ( c ) Ax = a x => a = p( A ) . ( 1 0) A ( m x m ) real, ..\ eigenvalue of A with real eigenvector x :f. 0 : A > 0, I .A I? bs = p( A) => I .A i abs is eigenvalue of A with eigenvector l x labs ·
78
H A N DBOOK O F �1 AT R I C ' E S .-1
( m x m) real , A > 0 : (a) p( A. ) is a simple eigenvalue of A . ( b ) p( A) is eigenvalue of A with eigenvector x > 0 . ( 1 2 ) A ( m x m ) real , A > 0 : p( A ) is eigenvalue of A with eigenvector x > 0 , x :f. 0 . ( 1 3 ) A ( m x m ) real , A > 0 , ..\ eigenvalue of A : ..\ :f. p( A ) => I ..\ I abs < p( A) · { 14) ( Hopf's theorem) .·1 [a;j ] ( m x m ) real , A > 0 , A m - I eigenvalue of A with second largest modulus, Af = rnax{ a;j : i , j = 1 , . , m } , Jl min{ a;1 ! , J = 1. . . . . m} : I .Am - dabs < M - jl < 1 . (11)
..
=
{ 15) A (m
x m ) real ,
wrresponding corresponding
A
- M
:
+ 11
> 0 , x > 0 ( m x 1 ) real eigenvec tor of A to eigenvalue p( A ) , y > 0 ( m x 1 ) real eigenvector of A' to eigenvalue p( A' ) , x'y 1 : Iirn [p( A ) - 1 Ar = x y' . =
( 1 6 ) \ ( m x m ) rea l A > O .
p(A)
=
·
,
a - oc
:
Note: Most results of t his subsection are, for instance, given in Horn 8.: Johnson ( 1 9 85 ) . 5.5
S ingular Values
Notation: O'( A ) O'mi n ( A ) O'max ( A ) 5.5.1
denotes a singular value of the matrix A . denotes the smallest singular value of the matrix A . denotes t he largest singular val ue of the matrix A.
General Results
( 1 ) A (m x n), m > H :
. {( mm
xH AH Ax XH X
)
l/2
X
( 11 X
1),
X
:j:. 0
}
79
EIGENVA L U ES A N D S I N G U L A R VA L U ES
(2) A ( m x n)
( { max
:
O"max ( A )
xH A H Ax
) 1 12
X(
n X
1),
X -::j; 0
}
XH X 1 m ax { ( x H A H A x ) 12 : x ( x 1 ) , x H x = 1 } . n
(3) A (m x ) r A, 1 < i < r : n
O";
,
=
min{ m, n } ,
.mm
y1 , . . , y , _ 1 (m x l )
u1 >
· · · > O"r
singular values of
( x H A H Ax ) 1 12 { max X
HX
x ( m x 1 ) , x -::j; O, x H yj = O, j = 1 , . . , i - 1
and u; = max y , , . . . ,Y m - • (m x l ) X (m X 1), (4) A (m
x
m)
X
.mm { (
-::j; 0,
xH A H Ax
XHX
X H Yj = 0, j =
}
) /2 l
1, . . . , m
-
i}.
nonsingular: : x ( m x 1 ) , x -:j; O
( 5 ) A ( m x n ) : u 1 , . . . , 0"r singular values of A values of A H . (6) A ( m x
n
<==:::>
.
u 1 , . . . , 0"r singular
) , r = min{ m, n } , O" J , . . . , O"r singular values of A : r tr(A H A) = L u? , i= I
.
( 7) (Singular value decomposition) A ( m x ) rk( A ) = r, u1 , . . , O"r -::f 0 singular values of A : exist unitary matrices U ( m x m ) , V ( x ) such that n
}
,
n
There
n
where D = diag( u1 , . . . , O"r ) and some or all of the zero sub mat ricE's disappear if r = m and/or r = n.
80
H A l\' D BO O K OF \1 AT R I C E S
( 8 ) ( Singular va]llt> decomposition of a real matrix ) ..! ( 111 x n ) rea l . r k ( A ) = r , t7 1 , . . . , t1"r =f 0 s i n gula r values of A : There exist real orthogonal matri c es [ ! ( m x 111 ) , V ( 11 x 11 ) such t hat
C'AV
[� �].
=
where D = d iag( t7 1 , . . . , t7, ) and sonw or all of t he zero sub 1 uat.rict's d isappear if r = m aud/or r = 11 .
(9) A ( m x n) : :\
=
('
] V 11 is t he singular value decomposit ion o [ �: ()) ] 0
0
( 1 0 ) A ( 111 x 11 ) r· = •
1 1 1 ill { m . n } ,
singular val ues of A -¢::::::} Zl'ros arc t lw eigeuvalues of
5.5.2 (1)
:\ . ·
·
(
0 < t7 t , . . . , t7r E ffi : t1" 1 ,
.
.
.
•
t1"r , - t7 1
•
.
.
.
•
.
t7 t , . . . . t7r i HP t lw - G"r an d !m - n ! ,�,,
Inequalities ·
fJ ( 1 11 x n ) . r = t u i u { m , n } . t7! { .4 ) > · · · > t7r ( .4 ) > 0 , t7! { B ) > > t7r ( H ) > 0 . t7! { A + fJ ) > · · · > t7r ( A + B ) > 0 singtllar values
of .4. .
H and A + B . rt>spectivrly :
(7'• +; - d A + fJ ) < t7, ( A ) + t7j ( B ) , ( 2 ) . \ . fJ ( m X n ) : (3)
{·H
..
f \
I < i, j <
r
with i + j < r + ! .
G"mar ( A + B ) < G"max ( A ) + G"mar ( B ) .
:1 . }] ·
·
.4. .
·
( m x n ) . r = m i n { m , n } . t7l { A ) > · · · > t7r ( A ) > 0 . t7 t ( B ) > > t7r ( H ) > 0 , t7 1 ( A B11 ) > · · · > t7r ( .4 B 11 ) > 0 singular val ues of lJ and A IJ11 • respectively :
[a 1 . . . . . a11,] ( 111 x 111 ) . Uj ( 111 x l ) , t7 t ( A ) values o f A :
( 4 ) :\
=
"
f1
"
<
> · · · > am ( .·\ ) si t tgular
L t7J ( A ) " .
j=l
11 = 1 . 2 . .
.
.
.
m.
81
EIG E N VA L U ES A N D S I N G U L A R VA L U ES x n ,
) B ( m x ( n - 1 ) ) is obtained from A by deleting any one column of A , r = min { m, n } , · · · >
· · · > O" min( m ,n - l ) ( B ) singular values of A and B, respectively :
(5) A (m
if m
>
n
and
if m < ( 6 ) A ( m x ) B ( ( m - 1 ) x ) is obtained from A by deleting any one row of A . r = m in { m , } , · · · > O"r ( A ) and 0"1 ( B) > · · · > O"m1n ( m - l , n ) ( B) singular values of A and B , respectively : n.
n
,
n
n
· · · > 0
if m < n and if m >
n.
(7) A (m X n ) :
1
O"m a .r ( A ) > C t r ( A H .4 ) ) 12 .
Note: The results of this section can be found in Horn & Johnson ( 1 985 ) .
Other books discussing singular values include Barnet t ( I 990 ) , Lancaster & Tismenetsky ( 1 985 ) and , in particular, Chatelin ( 1 993) where further results may be found.
6
Matrix Decompositions and Canonical Forms All matrices in this chapter are complex unless otherwise specified. 6.1
Complex Matrix Decompositions
6. 1 . 1
Jordan Type Decompositions
( 1 ) (Jordan decomposition) A
There exists a ( m x m ) with distinct eigenvalues A 1 , . . . , A n nonsingular ( m x m ) matrix T such that A = TA T- 1 , where A is a Jordan fo rm, that is, :
A =
is block diagonal with blocks
A; =
An , 0 0 0
1 An ,
0 1
0 0
...
...
0 0
i = l,
. . . , k > n,
1 An ,
on the diagonal and { n 1 , . . . , nk } { 1 , . . . , n} , that is, the same eigenvalue may appear on the principal diagonal of more than one =
H A N D B O O K O F !\L\T R IC ES A; .
For example, 2 0 0
:\ =
I
0
2
I
()
0
2 [2]
0
is a Jordan form of a matrix with the two distinct eigenvalu('s A 1 = 2 , A 2 = 3 . 2 and 2 0 0
I
0
2 0
I
t\
-
, 3 -
A z = [2] .
2
[
3.2 0
( 2 ) ( J ordan decomposition of a matrix whose eigenvalues are all distinct ) A ( m x m ) with m distinct eigenval ues .\ 1 , . . . , .\m : A = T.t\1'- 1 , where A = diag( A 1 , . . . , A m ) and T is a nonsingular ( m x m ) mat rix whose colu mns are eigenvectors of A corresponding to A 1 , . . . , A711 •
( 3 ) ( Near-d iagonal J ordan decomposition) A
( m x m ) wit h d istinct eigenval ues A 1 , . . . , A,.. , f > 0 a given real nu mber: There exists a nonsingular ( m x m ) matrix T such t h at 1 A = TAT- • w here A I (f)
0
0
Ak(f)
A =
is block diagonal with An , A;(f)
=
(
0
An '
0
0
0
0
0 (
0 0
i = l , . . . , k > n.
and { n 1 , . . . nk } = { l , . . . , n } , t h at is, the same eigenvalue appear on t he principal d iagonal of more t h an one A ; ( f ) . .
n 1 a \'
.
For proofs see, e.g. , Horn & Johnson ( 1 985, Chapter 3 ) or Bamet t ( I 990 , Chapter 8 ) . Note:
85
M ATRIX DECO M POSITIONS A N D C A N O NICAL FO R MS 6.1.2
Diagonal Decompositions
( 1 ) ( Spectral decomposition of a normal m atri x ) A ( m x m) normal w i t h eigenvalues A1 , . . . , Am : A = U AU11 , where A = d i ag( A 1 , . . . , Am ) and U is the unitary ( m x m ) m atrix whose columns are the orthonormal eigenvectors v 1 , . . . , Vm of A a...<;sociatrd with A 1 , . . , Am . I n other words, .
m A = L A;V;V� . i= I
( 2 ) ( Spectral decomposition of a H ermitian m atrix) A ( m x m ) H ermitian with eigenvalues A 1 . . . , Am
A = U A U 11 .
where A = d iag( A 1 , . . . , Am ) and U is the unitary ( m x m ) matrix whose columns are the orthonormal eigenvectors v1 , . . . , t'm of A associated with A 1 , . . . , Am . In other words, ,
m A = L A; ";vf . i= I
( 3 ) ( Singular value decomposition) A ( m x n ) , rk( A ) = r , <11 , . . . ,
A-u
There
[ D0 0 ] V II , 0
where D = diag( <11 , . . . ,
( 4 ) ( Diagon al reduction) A ( m x n ) . rk( A ) = r : 5 and a nonsingular ( n
There exists a nonsingular ( m x m ) mat rix x n ) matrix T such t h at
where some or all of the zero su b m at rices disappear when and/or r = n .
r·
m
( 5 ) (Congruent canonical form) A ( m x m) H ermi tian : There exists a nonsingular ( m x m) mat rix T such that A = TAT11 , w here A = diag(d1 , , dm ) anci tlw d, are + 1 , - 1 or 0 corresponding to the positive, negative and zero eigenvalues of A , respectively. .
•
.
86
H A N DBOOK O F M ATRICES
( 6 ) A ( m x m) symmetric, rk( A ) ( m x m ) matrix T such that
r
There exists a nonsingular 0 0
] T'
where the zero submatrices disappear when
' r
= m.
( 7 ) A ( m x m) : There exists a nonsingular ( m x m) matrix T, a unitary ( m x m ) matrix U and an ( m x m) d iagonal matrix A with nonnegative elements on the principal diagonal such that A = TU AU'T - 1 . S imultaneous D iagonalization of Two Matrices ( 8 ) ( Simultaneous d i agonalization of two Hermitian matrices) A , B ( m x m ) Hermitian , A B = BA : There exists a unitary ( m x m) matrix U such that A = U DU H and B = U AU H , where D and A are d i agonal matri ces. ( 9 ) ( Simultaneous d i agonalization of a positi ve definite and a H ermitian
matrix ) A ( m x m) Hermitian positive definite, B ( m x m) Hermitian : There exists a nonsingular ( m x m) matrix T such that A = TTH and B = T ATH , where A is a diagonal matrix. ( 1 0 ) ( Simul tan eo us d i agonalization of a positi ve definite and a symmetric matri x ) A ( m x m ) positive definite, B ( m x m ) sym metric: There exists a nonsingular ( m x m ) m atrix T such that A = TTH and B = TAT' , w here A is a real diagonal matrix with nonnegative diagonal elements. Note: The results of this subsection and further diagonal decompositions may be found in Horn & J oh nson ( 1985 , Chap ters 3 , 4 and 7 ) and Rao ( 1 973, Chapter 1 ) . 6.1.3
O t her Triangular Decompositions and Factorizations
( 1 ) ( Schur decomposition) There exists a unitary ( m x m ) matrix U and an A (m x m) : upper triangular matrix A with the eigenvalues of A on the principal diagonal such that A = U AU H . ( 2 ) ( Choleski decomposition) A ( m x m) positive definite: There exists a unique lower ( upper) triangular ( m x m) matrix B with real positive principal diagonal elements such that A = B B H .
87
M AT RIX DECOM POSITI O N S A N D C A N O N ICA L FO R M S
( 3 ) (Triangular factorization) A = [a;j] ( m x m ) , rk( A ) =
r,
# 0,
det
n = l, . . . , r :
There exists a lower triangular ( m x m ) matrix L and an upper triangular ( m x m) m atrix U , one of which is nonsingular, such that A = LU. ( 4 ) (Triangular factorization of a nonsingular matrix) A = [a;j] ( m x m) nonsingular, det
# 0,
n = l,
. . .
,m:
There exists a unique lower triangular ( m x m ) matrix L and a unique upper t ri angul ar ( m x m) m atrix U both having unit principal dm ) with diagonal such that A = L DU, where D = diag( d 1 , .
a1 1
n
II dj
i= l
•
•
,
aln n = 1 , . . . , m.
= det an i
. .
.
an n
( 5 ) (G ram - Schmidt triangular redu ction) There exists an ( m x n ) m atrix U with A ( m x n) , m > n : orthogonal colu m ns and an upper triangu lar ( n x n) matrix R such t h at A = UR. ( 6 ) ( G ram- Schmidt triangular reduction of a square matrix)
A ( m x m) : There exists a unitary ( m x m) m atrix [T and an upper triangular ( m x m ) m atrix R such that A = U R.
( 7 ) ( Gram - Schmidt triangul ar reduction of a nonsingular matrix) A ( m x m) nonsingular : There exists a unique u nitary ( m x m ) m atrix U and a unique upper triangular ( m x m) matrix R with positive principal diagonal such that A = U R. (8) A (m x m) : There exist (m x m) permutation m at rices P and Q, a lower triangular ( m x m) matrix L and an upper triangu lar ( m x m ) m atrix [T so that A = P LUQ. ( 9 ) A (m x m ) nonsingul ar :
There exists an (m x m ) permutation m atrix P , a lower triangular ( m x m) matrix L and an upper triangular (m x m) matrix U such that A = P L U .
88
H A N DBOOK O F M ATRICES
Note: For proofs and further details see, e.g. , Horn & J ohnson ( 1 985, Chapters 2 and 3 ) . 6.1.4
Miscellaneous Decompositions
( 1 ) ( Rank factorization) A ( m x n ) , rk( A ) = r : There exists an ( m x r) matrix B and an ( r x n ) m at rix C with rk( B) = rk(C) = r such that A = BC. ( 2 ) ( Polar decomposition )
A ( rn x m ) , rk( A ) r : There exists a p ositive semi definite ( m x m ) matrix P with rk( P) = r and a unitary ( m x m ) mat rix U such t h at A = PU. =
(3) A (
) nonsingular : There exists a symmetric ( m x m ) matrix ( complex) orthogonal ( m x m ) m atrix Q such that A = PQ.
m x m
P and
a
( 4 ) ( Square root decomposition ) A ( m x m ) p ositive (semi ) definite: There exists a positive ( sem i ) definite ( m x m ) matrix B such that A = B B , that is, B i s a square root of A . ( 5 ) A ( m x m ) unitary : There exists a real orthogonal ( m x m ) matrix Q and a real sy mmet ric ( m x m ) matrix S such that A = Qexp( iS). ( 6 ) A ( m x m ) orthogonal : There exists a real orthogonal ( m x m ) matrix Q and a real skew�symmetric ( m x m ) m atrix S such t hat A = Qexp( iS).
( 7 ) ( Echelon for m ) A ( m x m ) : There exists a nonsingular ( m x m ) matrix B such that BA = [c;j] is in echelon form, that is , for each row i , either c;j O , j = l , . . . , m , or there exists a k E { 1 , . . , m } such t h at Cik I and Cij = 0 for j < k and Cf k = 0 for I ,P i . ( For more details on the echelon form see the Append ix . ) =
.
=
( 8 ) ( Hermite canonical for m ) A ( m x m ) : There exists a nonsingular ( m x m ) m atrix B such that BA is an idempotent H ermite canonical form. ( For the definition of a Hermite canonical form see the Append i x . ) ( 9 ) A ( m x m ) : There exists a d i agonalizable ( m x m ) matrix D and a nilpotent ( m x m ) m at ri x N such that A = D + N and DN = N D. Note: For results ( 1 ) , ( 7 ) and (8) see Rao ( 1 973, Chapter 1 ) . The other results may be found i n Horn & Johnson ( 1 985, Chapter 3 ) . Both referellet'S contain a number of further decomposition results.
89
M ATRIX D EC O M POSITIONS A N D C A N O N ICAL FO RMS
6.2 6.2. 1
Real Matrix Decompositions
Jordan Decompositions
( 1 ) (Jordan decomposition of a real matrix with real eigenvalues) A ( m x m ) real with distinct eigenvalues A 1 , . . . , An which are all real : There exists a real nonsingular ( m x m ) matrix T such that A = TAr- • , w here
At
0
0
Ak
A= is block diagonal w i th
A; =
An, 0
1 An ,
0
0
0
0 1
. . .
0 0 .
i = 1, . . , k > n, 1 An ,
0
{ 1 . . . , n } , that is, the same eigenvalue may and { n 1 , . . . , n k } appear on the principal d iagonal of more than one A ; . ,
( 2 ) ( Real Jordan decomposition) A ( m x m ) real with distinct real eigenval ues A 1 . . . , Ap and distin c t There exists a real complex eigenval ues o 1 ± i/31 , . . , O q ± i /3q : nonsingular ( m x m ) matrix T such that A = TAT- 1 , where ,
.
0
A =
r. r,
0 is block d iagonal with
A; =
A p, 0
1 .\p ,
0
0
0
0
0 1
. .
.
0 0 .
i = 1 , . . , k > p, 1 A p,
H A N D BOOK OF M AT R ICES
90
{P I · . . , pk } = { I , . . . , p} , and .
[
/3q, O: q,
O: q , - i3q ,
f; =
0 0
]
[
/2
]
0
[2
0
0
0
i = 1' . . . ' I
/3q, O: q,
O:q, - /3q,
0
[
0
> q ' { q I ' . . . ' ql } = { 1 ' . . . ' q } .
[2
O: q , -/3q,
i3q , O: q,
]
Note: For proofs and further details see Horn & J ohnson ( 1985, Chapter 3) or Barnett ( 1 990. Chapter 8 ) . 6.2.2
O ther Real B lock Diagonal and Diagonal Decompositions
( 1 ) A ( m x m) real normal ( A' A = AA') : ( m x m) m atrix Q such that
There exists a real orthogonal
0 Q' .
A=Q 0
where the A, are real numbers or real ( 2 x 2 ) matrices of the form A; =
[ �: !: ] . -
( 2 ) (Spectral decomposition of a real symmetric matrix ) A ( m x m ) real sym metric with eigenvalues A 1 , . . . , Am : A = Q AQ' , where A = di ag( .\ 1 , . . . , .\m ) and Q is the real orthogonal ( m x m ) m atrix whose columns are the orthonormal eigenvectors v 1 , Vm of A associated with A 1 , . . . , Am . In other words, .
m
A=
L A;v; v: . I=
I
.
.
•
91
M AT RIX D ECOM POSITI ONS A N D CANONICAL FO R M S (3) A
(m
(m
x
x m
m
) real skew -symmetric:
There exists a real orthogonal
) m atrix Q such that 0
0 0
A =Q
Q' ,
0
where t he A ; are real ( 2
x
2 ) m atrices of the form b; 0
(4) A (m
x m
) real orthogonal :
m atrix Q such t h at
]
.
There exists a real orthogonal ( m x m ) 0
,\ I
Q' ,
A = Q 0
where t he
,\ ;
= ± 1 and t he A ; are real ( 2 _ A t. -
[ -Sincos B; •
()i
2 ) matrices of the form
x
sin B; cos B;
].
( 5 ) ( Singu l ar value decomposition of a real m atrix) A ( m x n ) real , rk( A ) = r, O"J , . . . , O'r ::j:. 0 singular values of A There exist real orthogonal m atrices U ( m x m ) , V ( n x n ) such that A=U
[D ] O
0 O
1
V ,
where D = diag( u1 , . . . , O'r ) and some or all of the zero submatrices disappear when r = m and/or r = n . Note: The results ( 1 ) - (4) are taken from Horn & Johnson ( 1 985, Chapter 2 ) and for ( 5 ) see Chapter 7 of the same reference.
H A N DBOOK O F M AT RICES
92
6.2.3
O t her Triangular and Miscellaneous Reductions
( 1 ) ( Schur decomposition of a real m atrix w ith real eigenvalues) A ( m x m) real with real eigenvalues : There exists a real orthogonal ( m x m ) matrix Q and a real upper triangular matrix A with the eigenvalues of A o n the principal d iagonal such that A = Q A Q' . ( 2 ) ( Choleski decomposition of a real matrix) A ( m x m) real positive definite : There exists a u n ique real lower ( upper) triangular ( m x m) m atrix B with positive diagonal elements such that A = B B' . ( 3 ) ( G ram-Sch m idt t riangular reductio n of a real matri x ) There exists a real ( m x n) matrix Q A (m x n) real , m > n : with orthogonal columns and a real upper triangular ( n x n ) matrix R such that A = Q R .
(4)
.4 ( m x n ) real , m >
: There exists a real orthogonal (m x m) matrix Q and a real upper triangular ( n x n ) m atrix R such that n
QA -
[
R O(m - n ) x n
where the zero m atrix disappears i f n
=
]'
m.
( 5) ( G ram -Schmidt triangular reduction of a real square matri x ) .4 ( m x m) real : There exists a real orthogonal (m x m) matrix Q and a real upper triangular ( m x m) mat rix R such that A = Q R . ( 6 ) ( G ram - Schmidt triangular reduction o f a real nonsingular m atri x ) A ( m x m ) real nonsingular : There exists a unique real orthogonal ( m x m) matrix Q and a unique u pper triangular real ( m x m) m atrix R with positive principal d i agonal such that A = Q R. (7)
A
(m
x m ) real :
Therr exists a real orthogonal ( m x m) matrix Q and au upper H esseuberg matrix R such that A = QRQ' .
( 8 ) (Square root decomposition) A ( m x m) real p ositive ( se m i ) defi n i te: There exists a real positive ( semi ) definite ( m x m ) matrix B such that A = B B , that is, B is a square root of 4 .
.
( 9 ) ( Simul taneous d iagonalization of a real positive defin ite and a real
sy mmetric matrix ) There A ( m x m) rral positive definite, B (m x m) real symmetric: exists a real nonsingular ( m x m) matrix T such that A = TT' and B = TAT' , where l\. is a real d iagonal matri x . Note: These decomposition and factorization theorems can be fou n d , for instance, in Horn & Johnson ( 1 98 5 ) and Rao ( 1 973 ) A lgorithms for comput i ng .
M AT RIX D ECO M POSITIONS A N D CANONICAL FO R M S
93
m any of the m atrix factors d iscussed in this chapter are described in Golub & Van Loan ( 1 989) .
7 Vectorization Operators All matrices i n this chapter are complex unless otherwise specified.
7.1
Defi nit ions
vee denotes the col u m n vectorizi ng operator which stacks t he columns of matrix i n a colu m n vector , that is, for an (m x n) matrix A = [a;j ] ,
vee A = vee( A) = col( A ) _
Related operators are
rvec( A) [vee( A')]'
( mn x 1 ) .
a
H A N DBOOK OF M ATRICES
96
which stacks the rows of A i n a row vector and
row( A ) = vec ( A ' )
=
( mn x 1 )
rvec ( A ) ' =
which stacks the rows of A in a column vector. For example,
vee
aII a2 1 a3 1
a12 a 22 a32
aI I a2 1 a31 a12 a 22 a32
and
aII a12 aI I a12 a2 1 row a 2 1 a 22 a 22 a3 1 a 32 a31 a32 The half- vectorization operator, vech , stacks only the columns from the principal diagonal of a square m atrix downwards in a column vector, that is, for an ( m x m) matrix A = (a ;j ] ,
vech A = vech( A )
=
q m(m + 1 ) x 1 ) .
Umrn
97
V ECTO RI Z ATION O P E RATO RS
For example,
vech
aI I a2 1 a3 1
a12 an a32
a13 a 23 a 33
a1 1 a2 1 a3 1 an a 32 a 33
I n the fol lowing sections results are given for vee and vech only. Similar results can be obtained for other vectorization operators from the results given. The following matrices are closely related to the vee and vech operators : •
•
•
7.2
Kmn or Km.n is an ( m n x m n ) commutation matrix defined such that for any ( m x n ) m atrix A , Kmn vec ( A ) = vec ( A' ) ( see Section 9 . 2 for its pro perties) . Dm denotes the ( m 2 x b m( m + 1 ) ) duplication m at rix defined such that vec ( A ) = Dm vech ( A ) for any symmetric ( m x m ) matrix A (see Section 9 . 5 ) . Lm denotes the ( b m ( m + 1 ) x m 2 ) elimination matrix defined such that vech( A ) = L m vec ( A ) for any (m x m) m at rix A (see Section 9.6). Rules for t he vee O perator
( m x n) : (2) A (m x n), c E (3) A (m x n) : ( 1 ) A, B
vec ( A ± B ) = vec ( A ) ± vee( B ) .
vec (cA) = c vec ( A ) .
(a) vec ( A ' ) = A"mn vee( A ) .
( b ) vec ( A ) = /\"n m vec ( A ' ) .
x m) : Lm vec ( A ) = vech ( A ) . ( 5 ) A ( m x n ) , B ( n x p) : (4) A (m
vee( A B) = ( /p (6) a ( m
0
A )vec ( B ) = ( B ' 0 Im )vec ( A ) = ( B ' 0 A ) \·ec ( ln ) .
x 1 ) , b ( n x 1 ) : vee ( ab' ) = b 0 a. ( 7) A ( m x n ) , B ( n x r ) , C ( r x s ) : vec( A BC) = (C' ® A )vec ( H ) . ( 8 ) A ( m x n ) , B ( r x s) : ( a) vee( A G B ) = Un 0 1\,m 0 lr ) ( vee( A ) 0 vee( B ) ) . ( b ) vec ( A ' ·Sl B ) = ( /\m• . n 0 lr ) ( vec( A ) 0 vec ( B ) ) . ( c ) vec( A 0 B' ) = ( /n 0 Kr, m • ) ( vec ( A ) 0 vee( B ) ).
H A N D BOOK OF M ATRICES
98
(d) vee( A' 0 B ) = ( Km , , n ( In 0 Km • ) 0 lr )vec(A 0 B ) . (e) vec( A 0 B') = ( In 0 Kr, m • ( Km • 0 lr ))vec( A 0 B ) . (f) vee( A ) 0 vee( B ) = ( /n 0 Km • 0 lr )vec(A 0 B ) . ( 9 ) A , B ( m x n) : ( a) vee ( A 8 B) = diag(vec A)vec(B) = diag(vec B)vec ( A ) . ( b ) vee( A 8 B ) = (vee A ) 8 vec( B) . (c) vec( A )'vec ( B ) = vec( B)' vec ( A ) . ( 10 ) A ( m x n ), B ( n x m) ( a) vec (A')'vec( B ) = tr(A B ) = tr( BA) . ( b ) vec(A' )'vec( B) = vec( B' ) ' vec( A ) . ( 1 1 ) A ( m x n ) , B ( n x p ) , C (P x q ) , D ( q x m ) :
:
vec( D')'(C' 0 A)vec ( B ) - tr(A BC D) - tr(DABC) - tr(CDA B ) - tr( BC DA).
( 12 ) A ( m x n) , B ( n x p) , C ( P x q ) , D (q x m ) :
vec( D' ) ' ( C' 0 A )vec( B) - vec(A' )'( D' 0 B)vec (C) vec ( B' )' ( A' 0 C)vec(D) vee( C' )' ( B' 0 D)vec( A) vec(D)'( A 0 C')vec ( B' ) .
( 13 ) A ( m x n) , B ( n x p) , C (p x q ) , D ( q x m ) :
vec( D' ) ' ( C' 0 A)vec(B) - vec( D' )'vec( ABC) - vee( A' D' )'vec ( BC) - vee( A' D'C' )'vec ( B ) .
( 14 ) A (m x m) :
( a) A lower triangular => vee( A ) = L;, vech ( A ) . ( b) A symmetric => Dt vec(A) = vech(A ) . (c) A sym metric => vec(A) = Dm vech(A). ( 15 ) A ( m x m ) : ( a) n;, vec ( A ) = vech( A + A ' - dg( A ) ) . ( b ) Dt vee( A ) = ! vech ( A + A').
99
VECTO RIZATION OPE RATO RS
( 16 )
(c) Dm v;, vee( A) == vee( A + A' - dg( A ) ) . ( d ) vec(dg( A ) ) = L;, L m Kmm L;, L m vec(A). A ( m x m ) , a, b (m x 1 ) , c E
[a A v;,+1 vee c b' ] = + Dm + l vee
( 1 7)
[ ac Ab' ] --
c a+b v;, vee( A) c � ( a + b) D;!; vec ( A )
A , S, V ( m x m) : S = ASA' + V and lm > - A 0 A nonsingular vec(S) = Urn> - A 0 A) - 1 vec( V ) .
Note: A very rich source for results on the vee operator is Magnus ( 1 988 ) . M agnus & Neudecker ( 1 988) also contains many of the results given here. 7.3 (1)
(2) (3)
(4)
(5)
Rules for t he vech Operator
A , B ( m x m) : vech(A ± B) = vech ( A ) ± vech ( B) . A ( m x m), c E vech(cA) = c vech(A). A, B ( m x m) : ( a) vech (A 8 B ) = diag(vech A)vech ( B) = diag(vech B)vech ( A ) . ( b ) vech ( A 8 B) = (vech A ) 8 vech( B). A ( m x m) : ( a) vech ( A ) = L m vec(A). (b) vech( A + A') = 2 D;!; vec(A). (c) vech ( A + A' - dg(A)) = D;, vee( A). (d) D;, Dm vech( A ) = 2vech(A) - vech(dg( A ) ) . (e) vech (dg(A ) ) = Lm Km m L� vech(A). A ( m x m) : (a) A lower triangular => L;, vech( A ) = vee( A ) . ( b ) A symmetric => vech( A ) = D;!; vec( A ) .
H A N D BOOK O F M AT RI C ES
1 00
(c) A symmetric => Dm vech ( A ) vec ( A ) . ( 6 ) A ( m x n ) , B ( n x p ) , C ( p x m ) : vech ( A BC) = L m (C' 0 A )vec ( B ) . ( 7 ) A , B ( m x m ) : vech( A ) ' vech ( B ) = vech( B)' vech ( A ) . =
M any results for the vech operator including the nontrivial ones given here are col lected in Magnus ( 1 988). Note:
8
Vector and Matrix Norms 8.1
General Defi nitions
function I I · / I attaching a nonnegative real number / l A / I to an ( m x n ) matrix is a norm if the fol lowing three conditions are satisfied for all complex ( m x ) matrices A , B and complex numbers c : A A
n
(i) (ii) (iii)
/ IA II
>
0
if A ::f 0 ,
/ /c A / / = / c / abs / I A / 1 ,
( triangle inequality). + Bll < IIAII + IIBII Here /c/abs denotes the modulus of c , that is, / c/ abs = y'CC with complex conjugate of c. If (i) is replaced by I IA
c
being the
( i )' /\A\\ > 0 . is said to be a seminorm. I n this case 1 / A / 1 = 0 is possible even if A ::f 0 . Instead of defining a norm for all complex ( m x n ) matrices, it may be defined for real matrices only. If (i), ( ii) and (iii) hold for all real ( m x n ) mat rices and real numbers c, 1 / · I I is called a norm over the field of real numbers ( ffi ) . In that case / c / abs is, of course, simply the absol ute value of c. Since vectors are ( m x 1 ) or ( 1 x ) matrices, norms are defined for vectors as well. Let l l · l l a and 1 / · 1 / b be norms for ( m x 1 ) and ( x 1 ) vectors , respect ively. The norm I I · I I for ( m x ) matrices is compatible wit h /1 · / I a and I I · l i b if II · II
n
n
n
1/A x//a < I IA II I I x/l b
for all ( m x ) matrices A and ( x 1 ) vectors x . A norm \ \ · / I a for ( m vectors is compatible with a norm for ( m x m ) matrices I I · I I if n
n
x
1)
1 / Ax l l a < I I A / I l l x \ / a
for all ( m x m) matrices A and (m x 1 ) vectors x . An (m x m) matrix A is called an isometry for a norm 1 1 · 1 1 for ( m x 1 ) vectors if 1 / A x / 1 = 1/ x /1 for all ( m x 1 ) vectors x .
H A N DBOOK O F M ATRICES
1 02
A norm I I · II for ( m x m ) matrices is called multiplicative or submultiplicative if for all ( m x m ) norm .
I I AB I I < I I A II I I B I I matrices A , B . A multiplicative norm is said to be a matrix
{
Ax l l : II sup A x II IIIub = l l x ll
(m x 1 ) ,
x =f.
0
}
is t he matrix norm induced by a norm II · II for ( m x 1 ) vectors .r . Alternatively i t is called sup norm o r operator norm or lub ( least upper bound) norm. A norm for ( m x n) matrices A is unitarily invariant if !IU A V II = II A ll for any two unitary matrices U (m x m) and V (n x n) . A norm 1 1 · 1 1 for ( m x n ) matrices is monotone if for any two (m x n ) matrices A = [aij ] . B = [bij] the fol lowing holds: l a ij l abs < lbij labs , i = 1 , . . , m , j = 1 , . . , n ·
.
::}
I I A II < I I B I! .
A norm for ( m x n) matrices is absolute if for all (m x n ) matrices A = [a,1 ] ,
II A ll = II [ l a ii l ab J I I · s
A matrix norm I I · II is said to be a minimal matrix norm , if for any matrix norm II · !I a the fol lowing holds:
I I A I! a < II A II for all
(m x m ) matrices
A
::}
II A II = !l A l l a for all A .
A m at rix no rm 1 1 · 1 1 is self-a djoint if I I AH I I = II A I I for all ( m x m ) matrices
A.
Related to norms are inner products. A function < · , · > attaching a complex number < A , B > to any two (m x n) matrices A , B is called an inner product if for any arbi trary (m x n ) matrices A , B, C and c E
< A , A > > 0 if A =f. 0, < A + B, C > = < A, C > < cA, B > = c < A , B > , < A , B > = < B, A > .
+
< B, C > ,
function < · , > is an inner product over the field of real numbers if i t attaches a real number to a pair of real matrices in such a way that for any real (m x n) matrices A , B, C and c E lR the fol lowi ng conditions are satisfied: ·
( i) < A, A > > 0 if A =f. 0,
103
VECTO R A N D M ATRIX N O R M S
(ii) < A + B , C > = < A , C > + < B, C > , (iii) < cA , B > = c < A , B > , (iv) < A , B > = < B, A > .
If < · > is an inner product for ( ) matrices, IIAII = < A , A > 1 / 2 is a norm for ( ) matrices. II I I is said to be a norm derived from the ·,
m x n
m x n
inner product
<
·
·
,
>.
·
Note: More information on vector and matrix norms can be obtained from many books on matrices and li near algebra. Almost all results of this <:hapter can be found in H orn & Johnson ( 1985). 8.2
S pecific Norms and Inner P roducts
Special norms for
(m
x n
) matrices A
(a ,i ) arP :
=
Euclidean norm, /2-norm, Frobenius norm, Hilbert-Schmidt norm or Schur norm: 1/2 m
n
for complex matrices.
L L l aij l ;bs i=l j = l m
n
I I A I I 2 = )t r(A ' A ) = L L l aij l �bs
1/ 2
i=l j=1
for real matrices.
lp-norm or M inkowski p-norm:
I I A II P Mahalanobis norm:
definite (
m x m
Maximum norm:
=
1/p
n
for 1 < p <
L L l aij ��bs i=l j=1
I IAI I n
=
[ tr( A H QA) ] 1 1 2 , where
) matrix. I I A II oo = max { l a ij l abs : i = 1, . . , m , .
Column-sum norm:
Row-sum norm:
m
II A I I col _ max
{f ·= 1
n
j a;j l abs : j
I I A II row - max L j a;j labs j=1
:i
==
=
1,
=·
r2
j
is a positive ==
1, . .
. .
. . . , m
1 , . . , n} .
n
}
.
.
1 04
H A N DBOOK O F M ATRICES
Spectral norm or max1n1um singular value norm or operator norm: •
II A II•pec = O"max ( A ) = max{ J:\ : ..\ is eigenvalue of A H A} , that is, ma x ( A) is the maximum singular value of A . Special inner products for ( m x n) matrices A : < A , B > _ tr(AH B ) for complex matrices A , B . < A , B > = tr( A' B ) for real matrices A, B. u
8.3
Res u lts for General Norms and Inner P roducts
( 1 ) (Cauchy-Schwarz inequality)
< · , · > an inner product for (m x n) matrices (over A, B (m x n) :
(2) (3)
ffi
or <[;) ,
I < A, B > 1 ;bs < < A, A >< B, B > . < · , · > an inner product for (m x n ) matrices, A , B ( m x n ) : I < A, B > 1;bs = < A , A > < B, B > -<===> A = cB for some c E an inner product over ffi for real (m x n ) matrices, A, B (m x n ) real:
I < A , B > l � bs = < A , A >< B, B >
-<===>
A = cB for some c E ffi .
(4) ( Parallelogram i dentity) II · II a norm for ( m x n ) mat rices derived from an inner product, A, B (m x n ) :
II A + B ll 2
+
I I A - B ll 2 = 2 ( II A II 2 + II B II2 l·
( 5 ) ( Pythagoras theorem ) 1 1 · 1 1 a norm for ( m x n ) matrices derived from an inner product < · . > , A, B (m x n ) : ·
< A , B > = 0 => II A + B l l2 = II A II 2 + II B W . ( 6 ) ( Polarization identity) 1 1 · 1 1 a norm for ( m x n ) matrices derived from an inner product < · , · A, B (m x n) : Re < A. B > ! ( I I A + B ll 2 - II A - B W ) . Re < A . B > � ( II A + B ll 2 - IIAII 2 - II B II 2 ) . where Re denotes the real part of a complex number.
>.
105
V ECTO R A N D M AT R I X N O R M S
( 7 } I I · I I a norm for (m x n) matrices derived from an inner product, A , B (m x n) :
I I A + BII I IA - Bll
< I I A I I2 + I I B I I2 -
( 8 ) 1 1 · 1 1 a seminorm for (m x n) matrices, A , B (m x n ) : I I I A I I - II B I I I abs
< I!A - Bll.
( 9 } l l · l l a , I I · l i b norms for ( m x n) matrices A , c E IR, c > 0 : (a) (b) (c)
I I A I I = c j J A I I a is a norm for (m x n ) matrices. I I A I I - I I A I I a + I I A I Ib defines a norm for ( m x n ) matrices. I I A l l = max( JI A IIa , I I A I I b ) defines a norm for ( m x n ) matrices.
( 10) 1 1 · 1 1 a norm for (m x 1 ) vectors x, T ( m x m) nonsingular: l lxiiT - I JTxJ I is a norm for ( m x 1 ) vectors . ( 1 1 ) 1 1 · 1 1 a seminorm for ( m x 1 ) vectors x, T (m x m ) : l lxiiT = I I Txl l is a seminorm for ( m x 1 ) vectors. ( 12) 1 1 · 1 1 a norm for ( m x m ) matrices, y (m x 1 ) , y :f. 0 : l lxJjy = I JxyH J I is a norm for ( m x 1 ) vectors. ( 13} 1 1 · 1 1 a norm for ( m x m) matrices , S, T ( m x m) nonsingular: I I A I Is,T = I I SATI I is a norm for (m x m) matrices. ( 14 ) 1 1 · 1 1 a norm for ( m x m) matrices, S = [s;j] (m x m ) , s;J :f. 0, i , j = 1 , . . . , m : I I A I I 0 = l i S �> A l l is a norm for ( m x m ) matrices . ( 15 ) l l · l l a , I I · l i b norms for (m x n ) matrices : There exist positive constants Ct , c2 E IR s uch that c t J I A I I a < I I A I I b < c2 I I A I I a for all ( m x n ) matrices A.
( 16 ) 1 1 · 1 1 a norm for (m x n ) matrices: II · I I is monotone {::::::} I I · I I is absolute. ( 1 7) 1 1 · 1 1 an absolute norm for (m x n ) matrices, A , B (m x n ) : I A i abs < ! B la bs ::} II A I I < I I B I I There exists a norm for ( 18 ) I I · I I a norm for (m x m ) matrices: ( m x 1 ) vecto rs which is comp atible with I I · I I ::} I I A t l l · · · I J A ; I I > p(A t · · · A ; ) = max{ J.A i abs ..\ is an eigenvalue of A 1 · · · A ; } for all , A ; and i = 1 , 2, . . . (m x m) matrices A 1 , :
•
•
•
Note: The results of this section may be found in Horn & Johnson ( 1985, Chapter 5 ) .
H A N DBOOK O F M ATRICES
1 06
8.4 8 .4. 1 (1)
(2)
Results for Mat rix Norms
General Matrix Norms
1 1 · 1 1 a matrix norm for (m x m) matrices, c E IR, c > 1 : II A II c = ciiAI I is a matrix norm for (m x m) matrices. I I · I I a matrix norm for (m x m) matrices, S (m x m) nonsingular: I I A I I s = I I S 1 A SI I is a matrix norm for (m x m) matrices . 11 · 11 ( 1 ) • . . . . l l · ll(n) matrix norms for (m x m) matrices: I I A II = max{ I I A I\( l ) • · . . , II A \\(n ) } is a matrix norm for (m x m ) matrices. 1 1 · \\ a matrix norm for (m x m) matrices: I I A I \ H = II A H \ I is a matri x norm for (m x m) matrices . \ \ · \\ a norm for (m x m) matrices: There exists c E IR, c > 0 , such that 1 \ A \ I c = c\\A\\ is a matrix norm for (m x m) matrices. \\ · \ \ a norm for (m x m) matrices: -
(3)
(4)
(5) (6)
\\ A\\m ax 1
=
max{ I I AB\\ : B (m x m) , 1\B\ 1 = 1 }
and
1 1 A IImax 2 = max{ I I BA I I : B ( m X m) , II B II = 1 } are matrix norms for (m x m) matrices. ( 7 ) 1 1 · 11 a matrix norm for (m x m) matrices, S, T ( m x m) nonsingular: II A I I s,T = I I S AT II is a norm for (m x m) matrices which may not be a matrix norm , that is, II · ll s,T may not be submultiplicative. ( 8 ) 11 · 11 a matrix norm for (m x m) matrices , S = [s;j ] (m x m) , sij -# 0. i, j 1 , . . . , m : I I A I I 0 = I I S 0 A I I is a norm for (m x m) matrices which may not be a matrix norm . ( 9 ) 11 · 1 1 a matrix norm for (m x m) matrices, A (m x m) : ( a) l l lm l l > 1 . ( b ) 1 \ A k l l < I I A I \ k for k = 1 , 2 , . . . (c ) I I A I I > p( A ) = max { \ .A iabs : ..\ is eigenv alue of A } . (d) lim I I A i l l 1 / i = p ( A ) = max { \.Aiabs : ..\ is eigenvalue of A } . (e) A -# 0 idempotent => II A II > 1 . (f) A singular => l l lm - A l l > 1 . (g) I I Im - A\1 < 1 => A nonsingular. =
• - oo
(h) A non sing ular => I I A - 1 1 1 >- \ I Im l l . 1\A\1
107
V ECTO R A N D M ATRIX N O R M S
(i) ! l im ! ! = 1 , ! ! A l l < 1 ::}
1
1 < + I!A \1
II ( Im -
A) - 1 , I <
1 1 - I! AI I
(j ) I I A I I < 1
l l lm \ 1 < l l lm l l + I I A I I -
::}
I I ( !m
_
A} - 1 1 1 < -
lllm l l - (lllm ll 1 - IIAII
( 10} 1 1 · 11 a m atrix norm for ( ) matrices , A, B ( (a) I I A B - Im l l < 1 ::} A , B nonsingular. m x m
(b) A nonsingular, B singular ::} ! l A - B ll ( 1 1 ) 1 1 · 1 1 a m atrix norm for ( A + B nonsingular:
m x m
1 )11AII
>
m x m
)
.
:
�
' !lA I I!
) matrices, A , B ( m x m ) , A nonsingular,
(a) II A - 1 - ( A + B) - 1 1 1 < ! I A - 1 I I I I ( A + B) - 1 I I II BII · (b) (c)
I I A - 1 B ll I I A - 1 - ( A + B) - 1 11 1 < I I A - B l l < 1 ::} 1 IIA- 11 1 - I I A - 1 Bll 1 I I A - 1 B ll < 1 and I ! BII < IIA- 1 1\ I I A - 1 - ( A + B) - 1 1 1 < ::} 1 I!A- 11
1
IIA- 1II IIBII - /IA - 1/I I I B/ 1 .
) matrices: 1 1 · 1 1 is unitarily invariant { 12} 1 1 · 1 1 a m atrix norm for ( ) matr ices A . ::} ! I AI \ .pec < ! ! A ll for all ( ) : There exists a matrix norm 1 1 · 1 1 such that I I A I I < 1 ::} ( 13 ) A ( l im;_00 A i 0, i.e., A is convergent. ) p(x ) = :Lr' 0 p; xi a power series: There exists a matrix ( 14 ) A ( norm 1 1 · 1 1 such that :L 7 0 IPdabs i ! A W converges for ::} p( A) = :L:"' 0 p; Ai exists. ) f E IR, f > 0, p(A) = max{ ! .A \abs : ..\ is eigenvalue of A } : ( 15 ) A ( There exists a matrix norm 1 1 · 1 1 such that p( A) < I I A I I < p(A ) + f . ) ( 16} A ( m x m
m x m
m x m
=
m x m
,
n --+ oo
m x m
m x m
,
:
inf { I I A l l : I I · I I is a matrix norm for ( ) matrices} = p( A) - max{ I .Aiabs : ..\ is eigenvalue of A } . m x m
Note: The foregoing results may be found in Horn & Johnson ( 1 985,
Chapter 5).
H A N DBOOK O F
1 08
8.4.2
M AT RI C ES
Induced Matrix Norms
( 1 ) : 1 · 1 1 a norm for ( m x 1 ) vectors: I I A I I tu b = m ax _
{
I I A x l l : x ( m x 1 ) , x 'f. O
l l .r l l
}
is a matrix norm for ( m x m) ma t ri ces . (2)
11 · 11 a norm for ( m x 1 ) vectors with induced matrix norm
A (m x m) :
max { I I A .rl l : .r ( m x 1 ) , J l x l l = 1 } max { I I A. x l l : .r ( m x 1 ) , ! l x l l < 1 } ma x
{
I I Axll
l l · l l tu b .
}
I x (m x 1 ) . I '1 .r I :I a l l l .r l fo r any no rm I I · I I for ( m x 1 ) vectors . =
:
a
( 3 ) ! l · l l lub an indu ced matrix norm for ( m x m) ( 4 ) il · IIL�i . l l · lll�� ind ll(' ed ma trix nor ms :
{
m
at r ices :
I I) :i A I I (b) : A (m x m ) , max J I A I I ,u b (b) .4 = max I I J I Iu b : .4 ( m x m ) . A 'f. 0 I I A I I\: b)
{
l l lm l l t ub =
}
1
.
( 5 ) 11 · 11)�� . 11 · 11).� � mat rix norms induced by norms l l · l l a . l l · l l b . respectively. I I A I I ��� for every ( m x m ) matrix J A I I i :i for ( m x 1 ) vect ors: .-l. � t here exists a c E IR . c > 0, such that l l x l l a = r l l x l l b for a l l ( m x 1 ) vectors x . ( 6 ) I1 · !1L�i . l 1 · ll\,� � ind u ce d matrix norms : JIAiiL�i < liAi l )�� for all ( m x m ) matrices .4 � I I A I I i :i 1 I A I I \ �� for all ( m x m ) m a t ri c es .4 . ( 7 ) I J · I I a matrix norm for ( m x m) matrices: There exists an induced mat rix norm 1: · l i 1 u b such that : J A i l > l i A : II ub for every ( m x m ) matrix =
=
.4 .
( 8 ) 1 1 · i l a mat rix norm for ( m x m) matrices . I I · l l tub an induced mat rix norm for ( m x m ) matrices: 1 1 .4 1 1 < J j .4 i J iub for all ( m x m ) matrices A � 1 1 .4 1 1 I I A I I I ub for all ( m x m ) matric es A. =
( 9 ) I I · I I a matrix norm for ( m x m) mat r ices : 1 1 · 11 is a minimal matrix norm � i l · l l is an induced matrix norm .
109
V ECTO R A N D M ATRIX N O R M S
( 10} II · II a norm for ( m x 1 ) vectors with induced matrix norm l l · l l 1ub : 1 1 · 11 is an absolute norm -<==> I I D I I Iub = max, = t , . . . , m l dd ab for every diagonal matrix D = diag(dt , . . . , dm ) · ( 1 1 } I I · I I a norm for ( m x 1) vectors with induced matrix norm II · l i M . s ( m m) nonsin gular, llxlls- • � n s- t x l \ , II A II \ �l = IlS- I A SIIM : 1 1 · 11\ �l is the matrix norm induced by l l · lls - t . s
X
Note: The foregoing results on induced matrix norms may be found in
Chapter 5 of Horn 8.5
8.5.1
&
Johnson ( 1 985).
P roperties of Special Norms
General Results
It -norm
(1) (2} (3) (4)
11 · 1 \ t is a matrix norm. I I · l i t is not derived from an inner product. x (m x 1 ) : ! l x l l t = max{ l x H Y\abs : Y (m x 1 ) , \I Y\ I oo = 1 } . A ( m x m) : I I A H \ \ t = II A I\ t ·
Euclidean norm
(5} l l · ll 2 is a matrix norm for ( m x m) matrices. ( 6 ) l l · l l 2 is derived from the inner product tr( A H B) for complex ( m x n ) matrices A , B or tr(A' B) for real (m x n ) matrices A , B. (7) A (m x n) : I I A H II 2 = I I A II 2 · ( 8 ) U ( m x m) unitary, A ( m x m) : I I U A ll = 11A I I 2 · 2 (9} U ( m x m) unitary, V ( n x n) unitary, A (m x n) II U A V\ \ 2 = II A \ \ 2 . ( 10 ) A ( m x m) with eigenvalues A t , . . . , Am : :
m A normal ::} I I A I I � = L \ A ; I;bs· •=
I
Maximum norm
( 1 1 ) II · l l oo is not a matrix norm. ( 12} 1 \AII = m i i A I I x; is a matrix norm for ( m x m) matrices.
H A N DBOOK O F M ATRICES
1 10
( 13} A ( m
x
m) :
limoo IIAIIr · I I A I I oo = p-
S pectral norm
( 1 4 ) l l · l l . pe c is a matrix norm for ( m x m) matrices induced by the vector norm ll · l l 2 · { 15 ) U ( m x m ) unitary, V ( n x n ) unitary, A ( m x n ) : II U A Vl!,pec = I I A II•pec· ( 16) I I · l l•re c is the only induced unitarily invariant matrix norm. ( 1 7 ) 11 · 11 a matrix norm for ( m x m ) matrices: 1 1 · 1 1 is unitarily invariant => I I A II•p ec < I I A I I for every ( m x m ) matr ix A . ( 1 8 ) A ( m X m ) : I I A H II•rec = I I A I I•pec · ( 1 9 } II · ll •p e c is the only induced , self-adjoint matrix norm . {20) A ( m x m ) : \ I AI \ p 6
ec
max { \ \Ax\1 2 : x ( m 1 ) , \\x\! 2 1 } max { I I Axlb : x ( m 1 ) , \ \ x l b < 1 } x m ax IIA l l 2 : x ( m 1 ) , x :f. 0 l l xl l 2 max { IYH Ax l abs : x , y ( m 1 ) , l l x l\2 = II YII 2 = 1 } ma x { IYH Ax\abs : x , y ( m 1 ) , l\ x l\2 = \\Y\\2 < 1 } · I IAA H I I • p ec = 1\A H A l \ • p ec = I! A \ I;pe c · x
{
=
x
}
x
x
x
( 21 ) A ( m
x
m) :
Column-sum norm
(23) \ l · l l c o l is a matrix norm for ( m m ) matrices. (24) l l · l lcol is induced by 1 1 · \\ t , that is, I I Ax l l t x ( m 1 ) , x :f. 0 . = max c ol 1\AII l lx \ l t x
{
:
x
}
Row-sum norm
( 25 ) I I · l l row is a matrix norm for ( m m ) matrices. ( 26 ) \ 1 · \ \ r ow is induced by 1\ · l l oo , that is, \ I A x lloo m ax ow X ( m 1 ) , x :f. 0 . I I A \ Ir : l \ x l loo x
=
{
X
}
Ill
V ECTO R A N D M ATRIX N O R M S
Note: The foregoing properties of special norms may be found in Horn & Johnson ( 1985, Chapter 5) . 8.5.2
Inequalities
( 1 ) A (m x n) :
(a) I I A l i i < jnin II A II2· (b) 1 \A\\t < m n \ \ A\\oo . ( c ) !l A IIt < n I I A I Icol · (d) 1 \ A \ \ 1 < m \ \ A I \ row A ( m x m ) : !l AI I t < m3 1 2 1\ Ail.pec · ( Holder's inequality) A , B ( m m) , 1 < p, q < oo, ! + � = 1 : A (m x n ) : (a) \ I A \ 1 2 < \ l A I I t . (b) \I A\1 2 < jnin i\AIIoo · (c) 1 \ A\ b < vn \ I A\Icol · (d ) \ I A \ 1 2 < vm ii AIIrow . ( e) 1 \ A I \ 2 < jmin(m, ) 1 \ A I \. pec · (Cauchy�Schwarz inequality) A , B ( m n) : (a) \ I A H B \ 1 2 < IIA I\ 2 \ I B \ b(b) \ tr( A H B ) \abs < II A II2 \ I B \ 1 2 · A, B (m m) : ( a) 1 \ A B \ b < \ I A I \ . p ec \ I B \ 1 2 · ( b ) \ I A B I \ 2 < \ I A\ 1 2 \I B \ I,pec· ·
( 2)
(3)
x
(4 )
IIA B \ 1 1 < 1 \ A I\p I I B \Iq .
n
( 5)
x
( 6)
x
( 7 ) A ( m x m)
(8} A (m x n) :
(a) (b) (c) (d) (e)
\ I A \ Ioo 1 \ A IIoo 1 \ A \ \ oo 1 \ A \Ioo 1 \ A \ \ oo
with eigenvalues < \ lA I I t · < IIAII2 · < 1 \ A I\col · < IIAII row
·
< 1 \ A I \. p ec·
)q ,
. . . , Am :
\ IA\1� >
m
L \ >.i \ �bs· i= 1
H A N DBOOK O F M ATRICES
112
(9) A ( n) : (a) I I A IIcol < I I A i h · ( b ) II AI leo/ < JTi I I A I I 2 · ( c ) I I AIIcol < m iiAIIoo · { d) II AIIcol < m IIAIIrow · ( 10 ) A ( m ) I I A I Icol < Jffi i i AII•pec· ( 1 1 ) A (m n ) : (a ) II A I Irow < I I A I I l · ( b) I I A IIrow < Jffi !I A I I 2 · ( c } I I A IIrow < T! I I A I I = · ( d ) I I A IIrow < n I I A IIcol · ( 1 2) A ( m X m ) : I I A IIrow < Jm iiAII.pec· ( 13) A ( m n ) : (a) II A II.pec < IIAII l · ( b ) I I A I I ,pec < I I A I I 2 · ( c ) II A II,pec < Jmi1 11AII = · ( d ) II AII,pec < JTi ii A IIcol · (e) II A II.pec < rm I I A IIrow . ( 14 ) A ( m) : I I A II;p ec < I I A IIcoi i i A I I r ow • m x
X m
:
x
x
m X
Note: The foregoing inequalities can be derived from results presented in Horn & J ohnson ( 1985, Chapter 5), where also further references are givf'n .
9
Properties of Special Matrices 9.1
Circulant Mat rices
Definition: A n ( m x m) matrix
am - l
a2 a1 am
a3 a2 a!
a rn - l Um - 2 U rn - 3
U rn am - l a fn - 2
a3 a2
a4 a3
a5 a4
a1
a2 at
bm c 1 c2
E
a1 am eire( a !
1
•
•
•
,
am) =
am
is a circulant matrix .
Properties
(1)
a1
1
•
•
•
,
am
1
b1
1
•
•
•
,
1
1
r 1 e i r e ( a I • . . . , am )
= circ ( c l a l ( 2 ) A = circ(a J , . . . a m )
1
•
•
•
1
b, )
c2b1 . . . . , r1 am ± c2bm ) .
:
,
( a)
±
± c2 c i r r ( b1
( b ) A = c i r r ( a 1 , . . , ilm ) . -
.
( 3) A = circ( a J , . . , am ) , a( .r ) = a 1 + a 2 .r + 0 = exp( 27ri/ 1 1 1 ) : .
· · · +
a
m
'n l.
J r
( a ) tr( A ) = ma 1 . ( b ) det( A )
(c) Aj
=
111 -
1
= 0 a ( Oi ) . j =D
a(Bi ) .
j = 0,
.
.
.,
m -
I , are the eigenvalues of .4 .
H A N DBOOK O F M ATRIC ES
1 14
(4) A ( m
x
m ) circulant:
(a) A H is circulant. ( b ) A ; is circulant for i = 0, 1 , 2, . . . (c) A - 1 is circulant if A is nonsingular. (d) A+ is circulant. (e) rk(Ai ) = rk( A ) for i = 1 , 2, . . . ( f ) A is normal, that is, A H A = A A H . (g) A is simple. ( h ) A. is a Toeplitz matrix . ( i ) A. is unitarily similar to a diagonal matrix. ( 5 ) A., B (
m
x
m
) c irculant:
(a) A B is circulant . (b) AB
(c) (6)
A. �)
=
B
BA.
is circulant.
A,
B ( m x m) circulant with eigenvalues A 1 ( A ) , . . . , Am ( A ) . . . , Am ( B ) , respectively:
and A t ( B ) ,
(a) A t ( A ) + A t ( B) , . . . , Am ( A ) + Am ( B ) are the eigenvalues of A + B . ( b ) A 1 ( A ) - A 1 ( B ) , . . . , Am ( A ) - Am ( B ) are the eigenvalues of A - B . ( c ) A t ( A) A t ( B ) , . . . , Am ( A ) Am ( B ) are the eigenvalues of A B . (7) A (m x
m ) circulant with eigenvalues A 1 , . . . , Am :
(a) A is H ermitian {::::::} A; E IR, i = l , . . . ( b ) . is Hermitian positive definite {::::::} A; E IR, A; > 0, i I , . . . , m. ( c ) A is unitary {::::::} [ A; \abs = I , i = I , . . . ( d ) A = F H diag(A 1 , . . . , Am ) F , where F is a Fourier matrix (see the A ppendix for a definition) . (e) nl-imoo A n exists {::::::} A1 = I or [A; \ abs < 1 , i = I , . . . , m. , m.
A
, m.
l (f) nlim ( lm + A + · · · + A n - l ) exists {::::::} I A; I abs < I , i n -oo I , . . . , m.
115
PROPERTIES O F S PECIAL M ATRICES
0 0
( 8 ) C = c i rc ( 0 , I , 0 , . . . , 0 )
(b) (c)
(d) (e) (f) (9)
0 0
0 1
(m
= 0 I
( a)
1 0 0 0
0 0
1 0
x m
)
:
em = lm · C' = c m - l _ c - 1 = cm 1 c- l = C' . A = circ( a 1 , . . . , am ) -¢::::::> A = a 1 lm + a 2C + · · · + am em ) is circulant -¢::::::> AC = CA. A( -
_
I _
m x m
lm = eire( ! , 0 , . . . , 0 ) .
Note: For proofs and more on circulant matrices see Davis ( 1 979).
9.2
Commutation Matrices
Definition: The ( m n x m n ) commutation matrix Kmn or Km . n is defined such that, for any ( m x n ) matrix A , Kmn vec( A ) = vec ( A ' ) . For instance,
1{32 =
so that, for a ( 3
/{32 vee
x
2) m atrix
a1 1 a2 1 a3 1
a12 an a32
I 0 0 0 0 0
0 0 1 0 0 0
0 0 0 0 I 0
0 I 0 0 0 0
0 0 0 1 0 0
0 0 0 0 0 I
A = [ aij ] ,
= 1{32
al l a2 1 a 31 a1 2 an a 32
a1 1 a12 a2 1 an a 31 a32
= vec ( A ') .
H A N D BO O K O F M ATRIC ES
1 16
9.2.1
General P roperties
( 1 ) Hkt = [h;j ] ( m x
n
) with h k t
= 1 and h;j = 0 if i i: k or j =f. l : m
L L H kl c:> H�� ·
1\·,, =
(2) (3) (4) (5)
n
k = 1 1= 1
Kmn is a real m atrix. Km n is a permutation matrix. 1\m l = Em . f\ 1 n = !,. . ==
( 6 ) 1\:r, n l\n m · (7) I\·,-;.. ;. = l\nm ·
( 8 ) f\rn n .p
= f\m,n p l\n ,p m
= l\·n ,p · m l\m , n
p
= ( Em 0 !\,. ,p ){ f\m,p ® In ) = ( fn 0 f\m,p ) { f\n ,p
(9) (10) (11) (12)
0
fm ) .
1\m rl ,p f\,. p ,rn f\p m , n = Em n p · r k { l\mn ) =
mn.
t r { I\,.11 ) = 1 + greatest comm on divisor of m t r( Km
m
)=
m.
-
1 and
n
- 1.
( 1 3) de t( l\m n ) ( - l ) m n ( m - 1 ) ( n - 1 )/4 . ( 1 4 ) de t ( Km m ) = ( - l ) m ( m - l )/ 2_ ( 15 ) The eigenvalues of 1\m m are 1 and - 1 with multiplicit ies 1 m ( m + I ) and � m( m - 1 ) , respectively. =
( 16 ) ( !,.2 + Kmm )1 = fm 2 + f\mm · ( 1 7) � ( 1m 2 + 1\m m ) is idem potent. 111 2
( 1 8 ) rk { l
(19) ( 20) (21) ( 22 ) (23) ( 24 )
+ [\.mrn ) =
�m(m + 1 ) .
t r ( f,., + J\,,1 ) = m( m + I ) .
1\rnm { lrn2 + Kmm ) = f,., + K mm · ( lm 2 + f\mm ) l\mm = Iml + K mm · { lm 2 - f\rnm )1 = fm2 - Kmm · � ( 1m2 - A' mm ) is idempotent . rk{ lm > - 1\mrn ) = � m(m - 1 ) .
117
P RO P E RTIES O F SPECIAL M ATRICES
( 25) t r ( Im2 - A"mm ) = m ( m - 1 ) . 9.2.2
(1)
Kronecker P roducts A (m x n) :
(a) 1\mn ( A '
(b)
0
[ /\mn ( A '
0
A ) is symmetri c .
A J ] 2 = AA' 0 A' A .
( c ) rk ( A ) == r ::::} r k [ l\mn (A' 0 A )] = r 2 .
(2) A ( m x m) :
(a ) ( 1m2 + A'mm ) ( A 0 A ) ( /m> + Kmm ) = 2( fm 2 + 1\mm ) ( A 0 A ) = 2 ( A 0 A ) ( !m2
(b)
+
1\mm ) .
( /m2 - 1\mm ) ( A 0 A ) ( fm2 - 1\mm ) = 2( fm 2 - 1\mm } ( A 0 A ) = 2( A 0 A ) ( fm2 - 1\mm ) .
( 3 ) A ( m x n ) , a (p x l ) , b ( 1 x p) :
1\p m ( A ® a ) = a @ A , 1\mp (a ® A ) = A ® a , ( A 0 b ) Knp = b 0 A . ( b 0 A ) 1\pn = A r:Y b.
(4) A ( m x m ) . a (m
x
1) :
(a )
( 1m 2 + Km m ) ( A 0 a ) = ( Im2 + 1\mm ) ( a C A ) = A 0 a + a 0 A.
(b)
( 1m2 - A'm m ) ( A. 0 a ) = - ( 1m2 - 1\"mm ) ( a 0 A ) = A 0 a - a O A.
( 5 ) A ( m x n ) , B (p x q ) : (a ) 1\pm ( A 0 B ) = ( B 0 A ) 1\qn · ( b ) 1\pm ( A 0 8 ) 1\nq B 0 A . =
( c ) vec ( A 0 B ) = Un 0 1\qm 0 lp ) [vee( A ) 0 vee{ B )} .
(6) A, B (m x n) :
tr [1\mn ( A' 0 B)} = tr(A' B ) .
(7) A, B (m x m) :
( a) ( Jm 2 + A"mm ) ( A ® B ) ( fm 2 + Kmm ) =
( Jm 2 + A"mm ) ( B 0 A ) ( !m2 + A"mm ) ·
{ b ) ( 1m2 - 1\·mm ) ( A 0 B ) { lm 2 - h' m m ) ::::: ( 1m2 - l\.mm ) ( B 8 A ) ( Im2 - 1\ , , ) .
( c ) ( 1m 2 + l\.mm ) ( A 0 B + B 0 .4 )( 1"1 2 + 1\.m m ) ::::: 2( !,2 + 1\mm ) ( A 0 B ) ( !m2 + 1\.mrn ) = 2 ( /m 2 + 1\mm ) ( A 0 B + B 0 A )
H A N D BOOK O F M ATRICES
1 18
= 2(A 0 B + B 0 A)(lm> + Kmm ) . ( d ) Um• - Kmm ) ( A 0 B + B 0 A)( lm• - Kmm ) = 2( /m> - Kmm ) ( A 0 B ) ( lm> - Kmm ) = 2(/m> - Kmm )(A 0 B + B 0 A) 2(A 0 B + B 0 A ) ( lm • - l\mm ) . ( 8 ) A ( m x n ) , B (p x q ) , C ( q x s ) , D ( n x r ) : =
1\.mp ( BC 0 AD) = ( A 0 B ) Kn q (C 0 D). (9) A (m
x
n) ,
B (p
x
q), C (r
x
s) : Km p,r (C 0 A 0 B ) K, , q n Km ,p r ( B 0 C 0 A ) K, q,n ·
( 10 ) A ( m
n ) real , rk( A )
= r : A ) , . . . , Ar > 0 are eigenvalues of A' A ::::} -\ 1 , . . , Ar and ± �, i < j, are the nonzero eigenvalues of Km n ( A' 0 A). x
.
9.2.3
Relations With Duplication and Elimination Matrices
( 4 ) lm> + Kmm = Dm Lm Um> + Kmm ) . ( 5 ) D'!;, = � Lm ( /m> + Kmm ) . ( 6 ) Lm Km m L'm = 2/ ! m( m + I ) - D'm Dm . ( 7) Dm = ( /m> + Kmm ) L'm ( Lm ( lm > + Kmm ) L'm ) - 1 . (8) L m A. mm L 'm is idempotent diagonal. ( 9 ) rk( Lm l\.mm L'm ) = m . ( 10 ) tr( Lm Kmm L'm ) = m. ( 1 1 } L'm Lm K mm L'm Lm is idempotent diagonal. ( 1 2 ) rk(L'm L m Kmm L'm Lm ) = m. ( 1 3 ) tr( L'm Lm Kmm L'm Lm ) = m. ( 1 4 ) L'm Lm Kmm L'm = Kmm L'm Lm Kmm L'm . ( 15 ) Kmm = Dm D'm + L'm Lm Kmm L'm Lm - lm> . ( 1 6 ) A ( m X m ) : L'm Lm Kmm L'mLm vec (A) = vec(dg ( A ) ) .
PROPERTIES O F S P ECIAL M ATRICES
1 19
Note: A substantial collection of results for commutation matrices including the ones given here may b e fou nd in M agnus ( 1 98 8 ) .
9.3
Convergent Matrices
Definition : A (m x m) is said to be convergent or stable if A n ___. 0 as n ( 1 ) A (m x m), B ( n x n ) , ( a) A is convergent,
c E (C : l c l abs <
I
=> cA is convergent .
( b ) A , B are convergent => A 0 B is convergent .
(2)
( c ) A , B are convergent => A EB B is convergent.
A
(m x m) with eigenvalues
.\ 1 , . . . , Am : <==>
A is convergent
(3)
A ::: diag(a , . . 1 A is convergent .
,
am )
<==>
:
max I .\ ; l abs < 1 . m
i= l �
I a ; I abs < 1 , i = 1 ,
,
. . . , m.
( 4 ) A ::: [a;j ] ( m x m) triangular: A is convergent <==> l a;; l abs < I , i = 1 , . . . , m.
(5)
A
(m x m) : (a) A is convergent
<==>
n
L Ai i= I
converges for n ___.
oo .
( b ) I I A I I < I for some matrix norm 1 1 · 1 1 => A is convergent. ( c ) A is convergent => Im ± A is nonsingul ar.
( 6 ) A (m x m) convergent: 00
(a) Urn + A ) - 1 =
L(-A( 00
( b ) ( /m - A ) - 1 =
L A; . i=O
( c ) Um � + A ® A ) - l =
00
L::: ( - A ) i 0 A' . 00
( d ) ( /m� - A 0 A ) - 1 =
L Ai 0 Ai . i=O
___. DC .
H A N DBOOK OF M ATRICES
1 20
( 7 ) (Stein 's theorem )
A ( m x m) : A is convergent {::::=} given a positive defin ite ( m x m ) matrix V , there exists a positive definite matrix Q such that Q - A H Q A = V.
Note: These results fol low straightforwardly from definitions or m ay be found in Lancaster & Tismenetsky ( 1 985 ) or H orn & J ohnson ( 1 985) . 9.4
Diagonal Matrices
Definit ion: A = ( a ij ] ( m x m ) is a d iagonal matrix or , briefly, A is d iagonal . if aij = 0 for i f- j, that is, .•1 -
Properties
( 1 ) A B ( m x m) d iagonal, c E ( a ) cA is d iagonal.
(f, :
( b ) A ± B is d iagonal . ( c ) A B is d iagonal.
(d) AB = BA.
( e ) A '�' B is diagon al .
( 2 ) A ( m x m ) diagonal . B (a) A
(b)
C::·
A .fl
( n x n ) d iagonal :
B is d iagonal. B is d iagonal.
( 3 ) A ( m x m ) d iagonal:
( a) r k ( A ) = number of nonzero d iagonal e lements.
( b ) A' = A .
(c ) AY = A . if A is real.
(4) A
= d iag( a l l • . . . , a ,nm ) :
( a) t r ( A ) = a 1 1 + m
( b ) det ( A. ) =
· · ·
IT a ii ·
i= l
+ amm ·
121
PROP ERTIES O F S PECI A L M ATRICES
( c ) a;;
r -t
0,
. = 1 , . . . , m => A - 1 = d t. ag( a - 1 , amm 1 1 . . . , - 1 ). -, 1 if a;; ::j; 0 a . where a:!: = 0 if a;; 0
2
II
.
d.
a
{
=
for i # j
0
m II Unn for l. = J.
(e) Aa dJ = [a�/ ] , where a I]. . =
n=l n ;t '
( 5 ) A ( m x m) diagonal :
(a) A is idempotent => all diagonal elements of A are either 0 or 1 . ( b ) A is n ilpotent => A = Om x m · (c) A is orthogonal => A = Im . (d ) A is unitary => A = Im . ( 6 ) A = diag( a . . . , amm ) : ( a) A is positive definite => 0 < a;, E IR, i = 1 , . . . , m. ( b ) A is positive semidefinite => 0 < a;; E IR. i = 1 , . . . , m . ( c ) A is negative definite => 0 > a;; E IR , i = 1 . . . , m. (d) A is negative semidefinite => 0 > a;; E IR , i = 1 , . . . , m. a;; is an eigenvalue of A and the it h ( 7) A = d iag( a . . . , amm ) : col umn of Im is a corresponding eigenvector for i = 1 , . . . . m. , bmm ) ( 8 ) A = diag(a l 1 • . . . , amm ) , B = diag(b t t 1 (a) D� ( A 0 B)Dm is a n m( m + l ) x �m(m + l ) ) diagonal matrix with diagonal elements � (a;;bjj + ajj b;; ), 1 < j < i < m . ( b ) Lm ( A 0 B)L;.,.. is a ( � m( m + 1 ) x � m( m + 1 ) ) diagonal matrix with diagonal elements a;; bjj , i < j. (c) L m ( A 0 B)Dm is a ( � m(m + 1 ) x � m( m + 1 ) ) diagonal matrix with diagonal elements a;; bjj , i < j. 1 1 ,
.
11,
•
•
:
•
. . . , amm ), B = diag(b t l , bmm) : m (a) d et[Lm( A 0 B)L:nJ = II b: ; a;:' - i + l . m + ( b ) d et[Lm ( A 0 B)Dm ] = II b:;a7,' - i 1 . 1
( 9 ) A = diag( a ! l ,
•
i= I
i=
.
.
.
H A N DBOOK O F M AT R I C E S
1 22
Many of the results of this section follow easily by considering the individual elements of the matrices involved. The others may be found in \lagnus ( 1988 ) or in Horn & Johnson ( 1985 ) . Note that a diagonal matrix is a special case of a triangular matrix. For transforming matrices to diagonal form see Chapter 6 . Note:
9.5
D uplication Mat rices
Definition: The ( m 2 x �m( m + I ) ) duplication matrix Dm is defin ed su'h m) matrix A . For example. that vec ( A ) = Dm vech( A ) for any symnwtric ( •
m x
D2 =
1
0
0 0
1 1
0 0 0
0
0
I
so t hat
[ vee a 1 1 a
a1 1 a12 a12 an
12
9.5.1
= D2
General Properties
( 1 ) Dm is a real matrix. ( 2 ) rk ( D, ) = � m ( m + 1 ) . ( 3 ) Dt, = ( D:n Dm ) - 1 D:n . (4 ) (5) (6) (7) (8) (9)
-1 + + Dm Dm - ( D' D ) det ( D:n Dm ) 2 m ( m - 1 ) / 2 . I
n1
n1
·
=
d et ( Dm D:n ) = 0 .
tr ( D:n Dm ) = m 2 . tr( D�, Dm ) - 1 = t m ( m + 3 ) . 1 0
0
0
0 0
0
D:n Dm
2 /m
all a12 a 22
= D2 vech
[
123
P RO P E RTIES O F S PECI A L M AT RICES
1 0 0
0 0 (D:n Dm ) - 1
0 � lm 0
( 12 ) D:n Dm is a n m ( m+ 1 ) x � m(m+ l ) ) diagonal matrix with 1 (m times) and
2 n m( m - 1 ) tim
e
s) on the diagonal .
( 1 3} Dm D:n is (m2 x m2 ) w i th eigenvalues 0 ( � m( m - 1 ) times ) , 1 (m times) and
9.5.2 (1} (2} (3) (4}
2
n m( m - 1 ) times) .
Relations With Commutation and Elimination Matrices Dm D"/;. = � ( 1m 2 + Kmm ) . Kmm Dm = Dm . + l\'mm -- Dm+ Dm · Lm Dm = f �m ( m +l ) ·
(5} Dm Lm ( fm2 + Kmm ) = 1m2 + Kmm · ( 6 } D"/;. = � Lm ( /m2 + Kmm ) · ( 7 } D:n Dm = 2 /�m(m+l ) - Lm f{m m L:n . { 8 } Dm = ( 1m2 + Kmm ) L:n [Lm ( lm> + Kmm ) L:n] - 1 . 9.5.3 Expressions With vee and vech Operators ( 1 ) A ( m x m) : D"/;. vec ( A ) = � vech(A + A' ) , D"/;.. vee( A ) = Lm vec (A) = vech( A ) , if A is symmetric D:n Dm vech( A ) = 2vech ( A ) - vech(dg( A)) , D:n vee( A ) = vech (A + A' - dg( A))
,
Dm D:n vec(A) = vec (A + A' - dg( A )] .
(2 ) A ( m x m ) , a, b ( m x 1 ) ,
c
E
] Dm +l vee [ C 1
a
�
b' ] [ D� + vec 1 a A = c
( 3 ) A ( m x m ) sym m et r ic nonsingular , c E
c
a+b D:n vee( A ) c
�(a + b) D"/;.. vee ( A )
(a) det(D"/;. [A 0 A + c vec(A)vec ( A )') Dm )
=
( 1 + em)( det A) m +l .
,
H A N D BOOK O F M ATRICES
1 24
( b ) ( D;!", [A 0 A + c vec(A )vec ( A )'] Dm ) - 1
[
= D! A- 1
0
A- 1
]
cc vec(A - 1 )vec( A - 1 ) ' Dm . l + m
-
Duplication Matrices and Kronecker Products
9.5.4
(1} A
( m x m) :
det[D;, ( A 0 A ) Dm] = 2m ( m - l )/ 2 ( det A) m + l . [D� ( A 0 A ) Dm] - 1 = D;!", ( A - 1 0 A- 1 )D;!",' , if det ( A ) f- 0. det[D;!", ( A 0 A )Dm] (det A) m + l . Dm D;t", ( A 0 A ) Dm = (A 0 A ) D m . D;t", ( A 0 A ) Dm D;!", = D;t", ( A 0 A ). [ D;!", ( A 0 A ) Dm ] ' = D! ( A ' 0 A ' )Dm , s = 0, 1 , 2, . . . [D;t", ( A 0 A ) Dm] - • = D! ( A - • 0 A ' )Dm , s = 1 , 2, . , if dt>t ( A ) f- 0 . ( h ) [D;!", ( A C A ) Dm j l / 2 D;!", ( A 1 12 0 A 1 1 2 ) Dm , if A 1 1 2 exists. A ( m x m ) with eigenvalues .\ 1 , . . , A m : .\ ; .\i , 1 < j < i < m , art> the eigenvalues of D!( A 0 A ) Dm . A , B ( m x m) : D! ( A 0 B)Dm = D;t", ( B 0 A )Dm . D;t; ( A 0 B )Dm = � D;t", ( A 0 B + B 0 A)D m tr [D;!", ( A 0 B)Dm ] = � [tr(A )tr( B) + tr(A B)], det ( B) = 0 => det[D;!", ( A 0 B)Dm ] = 0. diag (b J J , . . . , bmm) : D;t; ( A 0 B ) Dm A = diag (a! J , . . . , am m ) , B is a diagonal matrix with diagonal elements � ( a ; ; bii + aii b;; ) , 1 < j < i < m. A = [a;j] . B = [b; i] ( m x m ) upper (lower) triangular: D;!", ( A ·2 B)Dm is upper ( lower ) triangular with diagonal elements � ( a;, b11 + aj 1 b;; ) , 1 < j < i < m . A ( m x m). H ( m x m ) nonsingular, .\ 1 , . . . , .\ m eigenvalues of A H - 1 : m det[D! ( A •0 B) Dm] = T m( m- l )/2det(A)(det B) II(.\ ; + Aj ) . (a) (b) (c) (d) (e) ( f) (g)
=
-
.
=
(2}
(3)
.
,
(4) (5}
(6)
=
i >j
(7)
A,
H ( m x m)
with
A
0
B + B 0 A nonsingular:
[D! ( A 2· H )Dm ] - 1 = 2 D;t", ( A 0 B + B 0 A ) - 1 Dm .
(8} A (m x m)
with eigenvalues .\ 1
•
.
.
.
, Am
:
.
125
P RO P E RTIES O F S PECIAL M AT RICES
(a) Dm Dt, ( Im 0 A + A 0 lm )Dm = ( lm 0 A + A 0 lm )Dm . ( b ) tr[D;t ( Im 0 A + A 0 lm )Dm] ( + l )tr(A). ( ) [ D;t ( lm 0 A + A @ lm ) Dm]-1 = Dt, (Im 0 A + A 0 lm ) - 1 Dm , if Im 0 A + A 0 Im is nonsingular. (d) det[D;t ( Im 0 A + A 0 lm )Dm] = 2 m det(A) II (A; + Aj ). i>i ( e) A , + Aj , 1 < i < j < are the eigenvalues of D;t ( Im 0 A + A 0 lm )Dm . ( 9 ) A (m x m ) with eigenvalues AJ , . . . , A m , i E IN , i > 1 : =
m
c
m,
i- 1 det D;;. L(Ai- 1 -j 0 Ai )Dm
=
j =O
im (det Ar - 1 IT il k l k>l
where
Jlkl = ( 10 )
A = [ a ;1 ] ,
B
==
{
[b;j] ( m x
m
) lower triangular :
det[Dt, (A @ A ± B 0 B)Dm] = IT ( a aj j ± b;; bj j ) · i> j
(11)
;;
A (m x m) nonsingular, B ( m x m), A 1 , . . . . A m eigenvaluPs of BA - 1 : det[Dt, ( A 0 A ± B 0 B)Dm]
( 12 ) A, B ( m x
m
,
(a) D m + 1
)
=
(det A) m + l II( l ± A;Ai ). •>i
symmetric, a , b ( m x 1 ) , o , /3 E
] [
([
])
ob' + f]a' o/3 ( a' 0 b1 )Dm ob + f]a o B + f]A + ab' + ba' ( a' 0 B + b' 0 A ) Dm D:n ( a 0 b) D:n (a 0 B + b 0 A ) D:n (A @ B)Dm (b)
D! + 1
( [ o a'A ] 0 [ b b'B ] ) D!+ ' = a
(3
'
� (ob' + ;3a' ) o/3 ( a ' 0 b' ) D;t � (ob + Ba) �(o B + f]A + ab' + ba' ) �- ( a1 0 B + b' 0 A)D;t,' D;t", ( a 0 b) � D;t ( a 0 B + b @ A ) Dm+ ( A c;. B)Dm+ ' '
H A N DBOOK O F M ATRICES
1 26
9 .5.5
Duplication Matrices, Elimination Matrices and Kronecker Products
(1) A (m
x m
)
(a) L m ( A
:
0
A ) Dm = D;!; ( A 0 A ) Dm
0
( b ) L m Um 0 A + A 0 lm ) Dm = D;!; ( lm 0 A + A 0 lm ) Dm o
( 2 ) A = diag(a l l , o . . , am m ) , B = diag( b u , . . o , b m m ) :
L m ( A 0 B ) Dm
is a diagonal matrix with diagonal elements a;;bjj , 1 < i < j < m o
( 3 ) A = [a;1 ] , B
=
[b;j ] ( m
x m
) upper ( lower) t riangular:
(a) L,. ( A 0 B ) Dm is upper ( lower) triangular with diagonal elements a ; ; bi i , 1 < i < j < m o m ( b ) det [Lm ( A 0 B ) D m ] = II bt a;'; - i + l o i=l ( 4 ) A = [ a ;j ] , B = [ b;j ] , C = [ c;J] ( m x m ) lower triangular: ( a ) L m ( A 0 B' )Dm Lm ( A 0 B' ) L'r,. o =
( b ) det [Lm ( A B'
0
B' A ) L ; ] n
det[Lm ( A B 0 B' A ) Dm ] = det [Lm ( A B' 0 A' B' )Dm ] m 1 = ( det A ) + (d et B ) m+ l o m
( d ) det [ Lm ( A B ' 0 C ' ) Dm ] =
(5) A . B ( m
) det ( B ) f- 0 :
x m ,
=
i=I m
II C� ; ( a;;b;; ) m -i + l
0
i=I
det[Lm ( A 0 B ) Dm ] = det (A)(det B)
m
m- 1
II det(C(n))
n=l
and mm m det [ L m ( A 0 B ) D:, ' J = T ( - l ) / 2 det ( A)(det B) w here
C11
Cln
Cn 1
Cnn
are the principal su bmatrices of C = [cij]
=
AB- 1
0
m- 1
II det(C(n))•
n=l
127
PROPERTIES O F SPEC I A L M ATRICES
Note: M agnus ( 1 988) provides a very extensive collection of resul t s on duplication matrices. The results of this section are taken from that source.
Elimination Matrices
9.6
Definition: The ( � m ( m + 1 ) x m 2 ) elimination matrix Lm is defi ned such that vech( A ) = Lm vec ( A ) for any ( m x m ) matrix A. For example, L2 =
1 0 0 0 0 1 0 0 0 0 0 1
so that vech
9.6.1
[
a1 1
a2 1
a,2
an
a1 1
]
a21
an
a1 1
= L? •
a2 1
a12 an
=
L 2 vee
[
al l
a21
a12
an
].
General Properties
( 1 ) Lm is a real matrix. { 2 ) rk( Lm )
( 3 ) L �,
=
=
L�,
tm(m + 1 ) .
( 4 ) L m L 'm = /! m ( m + l ) · ( 5 ) L 'm Lm is idempotent diagonal with diagonal elements 1 and 0. ( 6 ) r k ( L 'r,. L m ) = � m( m + 1 ).
Dt, vec ( A ) = vech ( A ) , if A is sy m m etr i c . ( b ) vec ( A ) = L;11vech( A ) , if A is lower triangular. ( c ) L m l\.mm L'mvech( A ) vech[dg( A )] . (d) L;n Lm l\mm L'm L,.vec( A ) vec [dg( A)] . ,
( a) L vec ( A )
=
=
=
(2) A ( m x m ) nonsingular, lower triangular, E {[' : c
(a) dct ( L .,. [A' 8 A + c vec(A)vec( A' ) '] L'm ) = ( I + cm)( det. 4 ) + 1 . ( b ) { L m [A' C:) A + vec( A )vec( A' )') L'm } - 1 , .C vec ( A - 1 )vec ( A · L m . - Lm A - 1 0 A - 1 I + em _
9.6. 4
[
r
-
1 - l \1
]
.
"'
I
Elimination Matrices and Kronecker Products
A = diag( a l l · . . . , Omm ) . B = diag(bl l , . . . , bmm ) : L.,. ( A 0 B) L:n d i agon a l with d i agonal elements a;;b1 j . i < j. ( 2 ) A = [a;j ] , lJ [b;i J ( m x m ) upper (lower) triangular: (1)
IS
=
( a) L m ( A ·8 B ) L'm is upper ( lower) triangular w i t h elements a;;bi1 , 1 < i < j < m. m ( b ) det[Lm ( A C! B ) L ;11 ] = II bj; a;:' - i + l . 1
(3)
A =
[a;j ] .
=I
B = [b;j ] ( m x m ) lower triangular :
diagonal
P RO P E RTIES O F S P EC I A L �f ATRICES
(a ) d et [L ( 4' rn
·
l
"' 0
B )L 'm
1 29
m _ J II b i a m - i+ l -
ii u
i= I
·
m . are the eigenvalues of L m ( A' 2 B) L�,. . ( c ) [L m ( .4 ' � B)L ;,, ]' ::::: Lm[(A') ' :S· B']L �, for s 0. 1 . 2 . . . . s . . . - 2 , - l , if .4 . B nonsingular � . if lower t riangular .4 1 1 2 • B 1 12 exist s ( d ) det [Lrr. ( A B ' 7:· B' A )L;n J det[Lm ( A B' C A' B)L ;,] = (d et A ) m + l (det B ) m + l . ( e) L;, L m ( A ' 2 B ) L;, = ( A' 8 B ) L ;, . ( 4 ) A . B (m x m ) : A . B are strictly lower triangular =? L m ( A ' Z B) L;,. is nilpotent. ( 5 ) A ::::: [ a;j ]. B ::::: [b ;j ] . C = [c;j] . D = [d;j ] ( m x m ) lower triangular : m det[Lm ( A B' :& C' D)L ;,] = IT( r; , d;; )' ( a1 1 b1 1 )m - • - : , (b)
a ; ; bj j ·
< i <j <
:::=
i= I
det [Lm ( A ' ·2 B
�
C'
•7:
D)L � J =
IT ( b " aj j + d;; r1 ; )
• ?. J
( 6 ) A. B, C. D ( m x m ) nonsingular, lower triangular : -:- .- ( C ' ) - 1 j1 L' · r ( ·1 B' -.'. · C' D) L 'm .� - 1 = L m [( B' ) - 1 '7'· D - 1 ]L' L [ .. • - 1 .... ( 7 ) A = � a ;j ] ( m x m ) lower t riangular. n E [\ : l det Lm L (A' t - l - j 8 AJ L'm = n rr. ( det A t - 1 IT Jlu . L
m
•
......_.
-·
f'l1
rn
""1
··
Tl -
J=C
where Jl k l
=
{
( 8 ) A = [aij J · B = [ b ;J ]
k>I
( a )\ - a /1 )/ ( au - au ) n a nk k- 1
(m
x
m) :
m n_ ,
-.
where a1 n := l . . . . . m .
if a H if a H
a ,.. ..
'1
Un n
b,., ,
f.
a1 1
= a11
n:
·
H A N D BOOK O F M ATRICES
130
9.6.5
Elimination Matrices, Duplication Matrices and Kronecker Products
( 1 } A (m x m) :
(a) L m ( A ·Sl A ) Dm = D� ( A 0 A)Dm . ( b ) Lm ( /m 0 A + A 0 lm )Dm = D� ( /m 0 A + A 0 lm )Dm . {2} .4 = diag(a1 1 , . . . , a mm ) , B = diag (bl l , . . . , bmm ) : Lm ( A 0 B ) Drn is diagonal matrix with diagonal elements a ;;bjj . 1 < i < j < (3} A = [a;j] . B = [b ,j] ( m x ) upper (lower) triangular: (a) L m ( A 0 B ) Dm is upper ( lower) triangular . m (b) det[Lm ( A 0 B ) Dm ] = II b:;a;'.' - i + l . a
m.
m
[ c;j ] ( m x m ) lower triangular: (a ) Lm ( A c;;, Lm ( A 0 B ' ) L 'rr, . ( b ) det[Lm ( A B' ;-:;. B' A ) L:n J = det[L m ( A B 0 B' A ) Dm ] = det[Lm ( A B' 0 A' B' ) Dm] = ( de t A ) m + 1 ( de t B) m + 1 . m ( c ) det[Lm( A 8 B' C ) Dm ] = II (b;; c;; ); a;'.'- i+ t .
(4} A = [a ;j] , B
=
[b;j ] , C = B ' ) Dm =
i=l
m
( d ) det[Lm ( A B' 0 C' ) Dm ] = II c;; ( a;; b;; )m - i + l . i=l ( 5 ) A. B ( m x m ) , det ( B) ::f. O :
det[L111 ( A 0 B )D, ] and
=
m- 1
det ( A )(det B ) m II det(q n )) n=l m- 1
det [ L m ( A 8 B ) /J�, ' ] = 2- m ( m - 1 J/2det( A ) ( det B ) m II det( Cr " ) ) , n= where l
are the principal su bmatrices of c = [ Cjj l = A s - l . Note: The results of this section are taken from Magnus ( 1 988) where also further results on elimination matrices may be found.
131
PROPERTIES O F SPECI A L M ATRIC ES 9.7
Hermitian Mat rices
Definitions: An ( m x m ) matrix A is Hermitian if AH = A, that is, the ijth element a;j is the complex conjugate of the jith element., iij i = a;i , t h at is,
w here the a;; , i I , . . . , m , are real . An ( m x m ) m atrix A is skew-Hermitian if A H = - A , that is , the ijth element a;j is - 1 times the complex conjugate of the jith element, - iii ; = a;i , that is, =
9. 7 . 1
General Results
( 1 } A (m
) real : A is Hermitian x m ) Hermitian , c E IR :
x m
( 2 } A, B ( m
<==>
A is symmetric.
(a) cA is Hermitian.
( b ) A ± B is Hermitian . (c) A B
=
BA
=?
A B is H ermitian.
( d ) A 8 B is H ermitian.
(3) A (m
x m
) Hermitian , B ( n
x n
(a) A 0 B is H ermitian .
) Hermitian:
( b ) A EB B is H ermitian . ( 4 ) A = [a;j) ( m
x m
) Hermi tian :
(a) A' is Hermitian . ( b ) A H is Hermitian .
( c ) A- 1 is Hermitian , if A is nonsingular.
( d ) Ai is Hermitian for i = 1 , 2, .
-
(e) A is Hermitian .
..
1 32
H A N DBOOK O F M ATRICES
(f)
I A i abs
is Hermitian . is Hermitian for i
(g ) a; 1
a;;
( h ) .4 11 A = (5) A (m x m) :
A A 11 ,
that is,
A
=
1 , 2, . . . , m .
is normal .
(a) A + A 11 is Hermitian . ( b ) A - A 11 is skew- Hermitian . ( c ) AAlf is Hermitian . ( d ) .4. 11 A is Hermitian . ( ) There exist unique Hermitian ( m x m ) matrices B , C such that A = B + iC. ( 6 ) A (m x m) : e
(a) A is Herm itian (b) A is Hermitian ( c ) A is Hermitian B.
<:=:}
<:=:}
¢::::::}
is skew- Hermitian. x 11 Ax E IR. for all ( m x 1 ) vee tors x . B 11 A B Hermitian for all ( m x m ) matrices iA
( d ) A is Hermitian {::::::} A 11 A = A A 11 and all eigenvalues of A are real numbers. (e) A is Hermitian {::::::} there exists a unitary ( m x m ) matrix l ' and a real diagonal ( m x m) matrix A such that A U !I.. U 11 . ( Householder transformation ) 2 x x 11 x ( m x 1 ) : lm - 11 is Hermitian . X J: A is Hermitian => B11 A B is Hermitian. A ( m x m ) . H ( 111 x ) A, B ( m x m ) Hermitian: A H HA {::::::} there exists a unitary ( m x m ) matrix C such that [.' A U 11 and U BU 11 are diagonal matrices. A is similar to a real ( m x m) matrix B {::::::} A ( m x m) there exist Hermitian ( m x m ) matrices H and C', one of which is nonsingular, such that A II C. .4. , B ( m x m) Hermitian : =
( 7) (8) (9)
( 10)
n
:
=
:
=
(11)
(a) [tr( AB }f < t r( A B 2 ) . ( tr A )2 . ( b ) rk( A ) > , f tr( A 2 ) :f 0 . t r( ) 1 2
�� /t�
1 33
P RO P E RTIES O F SPECIAL M ATRICES
( ) rk ( A 8 B) < rk ( A ) rk ( B ) . ( 12 ) A (m x m) , b (m x 1 ) , c E IR : . . c bH t. H B= ermtttan => det ( B) s b A c
]
[
=
c det( A) - bH Aadj b.
Note: M any of these results are elementary. The others can be found 1 n Horn & Johnson ( 1985 ) . 9. 7.2
Eigenvalues of Hermitian Matrices
Notation:
-\( A )
denotes an eigenvalue of the matrix A .
Amin ( A )
is the smallest eigenvalue of t he matrix A .
Amax ( .4 )
is the largest eigenvalue of t he matrix A .
( 1 ) A ( m x m) H ermitian : ( a) A l l eigenvalues of A are real numbers. <:::=?
( b ) A is positive definite greater than 0 .
all eigenvalues of A are real and
( c ) A is posit i ve semidefinite <:::=? al l eigenval ues of A are real and greater than or equal to 0 . ( 2 ) ( Rayleigh-Ritz theorem ) A ( m x m) Hermitian : A
min
xH Ax { ( A ) = min : X (m X 1 ) , :f. 0 } , XH X { xH Ax x (m x 1 ) , x :f. O } . = max X
Am ax ( A )
XH X
:
( 3 ) ( Courant-Fischer theorem ) A ( m x m) Hermitian with eigenvalues -\ 1 < · · · < A m , 1 < i < xH A x -\; min max : !J J . · · · Ym - • x X H (m X I)
{
=
X ( m X 1 ) , X :f. 0 , X H Yj and -\;
=
YJ
m ax
, y, _ l
(mX 1) . . . .
min
= 0, j =
1, . . . , m
{ xHAx :
- i}
XHX
X ( m X 1 ) , X :f. 0 , XH Yj = 0 ,
j = l, . .
.,i- 1
}·
m :
H A N D BO O K O F M AT RI C ES
1 34
( 4 ) A ( m x m) Hermitian with eigenvalues A 1 < · orthonormal eigenvectors VJ , . . . , Vm n E { 1 1
· · < Am and associated 1 m} : min{tr(XH AX) : X ( m x n ) XH X = In } = ) q + · · · + An . The minimizi ng matrix is X = [V 1 1 1 vn] · max{tr(XH AX) : X ( m x n ) 1 XH X = In } = Am + · · · + Am - n+l · Vm -n+d · The maximizing matrix is X = [vm1 1
•
•
•
1
•
•
•
9. 7.3
(1)
(2)
A ( m x m) Hermitian,
A (m
•
•
•
x
(6)
1
:f. 0 ( m x 1 ) :
:
1 m.
x
m ) Hermitian with eigenvalues X ( m x n) : xn x
(5)
•
x HAx Ami n ( A) < X H X < Am ax( A). A ( m x m ) Hermitian with eigenvalues A 1 (A), . . . , Am (A) : { xHAx : x (m x 1 ) , x :f 0 min xHx } { x H Ax x ( m x 1 ) 1 x :f 0 < A; ( A ) < max i = 11
(4)
•
Eigenvalue Inequalities
xHx
(3)
•
= In
}
1
A 1 (A) < · · · < Am(A),
n n :�::::.x i (A) < tr(XH AX) < :�:::> m - n+; ( A). i= I i=I
A = [a;j] ( m x m ) Hermitian: Am;n (A) < a;; < Amax (A), i = 1 , . . . , m . A = [a;j ] ( m x m ) Hermitian with eigenvalues A 1 (A) < · · · < Am (A) : n n n i=l i= l A = [a;j ] ( m x m ) Hermitian with eigenvalues 0 < A 1 (A) < · · · < Am (A) : n II A ; (A) < II a ii, n = 1 , . . . , m. i=l i= l n
1 35
P RO P E RTIES OF S P EC I A L M ATRICES
( 7 ) ( Inclusion principle ) < A ( m x m ) Hermitian with eigenvalues .X I ( A) < Am ( A ) , A( n J ( n x n ) a principal submatrix of A with eigenvalues
A J ( A (n) ) < · · · < A n ( A ( n ) ) :
< Am- n + i ( A ) ,
.X ; ( A ) < .X ; ( A c n J )
i = 1 , . . . , n,
Aman ( A ) < Amin (A (n) ) < Amax ( A (n) ) < Amax (A) . (8) A (m
x m
(9) A, B (m
) Hermitian , B ( m
x m
x m
Am;n (A + B)
>
Amin ( A ) ,
Amax ( A + B)
>
Amax (A) .
) H ermitian :
Amin ( A + B) ( 10 )
(11)
A, B (
m x m
) p ositive semidefi nite:
>
Amin(A) + Am;n (B).
Amax (A + B ) < Amax( A ) + Amax ( B ) . ) H ermitian , 0 < a, b E IR. : Am ; n ( aA + bB)
>
Amax ( aA + bB)
<
a.Xmin ( A ) + b.Xmin ( B) ,
aAmax ( A) + b.Xmar ( B ) . A( ) H ermitian with eigenval ues .X 1 ( A ) < · · · < .X, ( A ) , B ( ) H ermitian posit i ve semidefinite, .X I ( A + B ) < · · · < Am ( A + B ) m x m
m x
m
eigenvalues of A + B :
.X ; ( A + B ) > .X ; ( A ) ,
i = 1 , . . . , m.
( 12 ) A ( m x m ) Hermitian with eigenvalues .X 1 ( A ) < · · · < Am (A), x (m x 1 ) , A 1 ( A ± xxH ) < · · · < A m ( A ± xxH ) eigenvalues of A ± xxH : .X ; ( A
±
x xH ) < .X ; + 1 ( A ) < A; + 2 ( A ± xxH ) ,
.X ; ( A ) < .X; + 1 ( A ± xx H )
< .X;+ 2 ( A ) ,
i = 1 , 2, . . . , m - 2;
i = 1 , 2,
...
, m -
2.
( 13 ) A , B ( m x m ) Hermitian , rk ( B ) < r, .X 1 ( A) < · · · < Am ( A ) eigenvalues of A , .X 1 ( A + B) < · · · < A m ( A + B ) eigenvalues of A + B : .X ; ( A + B ) <
.X; (A)
<
Ai+ r ( A)
<
Ai+ r (A + B )
A;+ 2 r ( A + B ) , <
.X;+ 2 r(A),
i
= 1 . 2,
. . . , m - 2r;
i = 1 . 2 , . . . , m - 2r·.
1 36
H A N DBOOK OF M ATRICES
( 14) ( Poincare's separation theorem )
A(
m x m
.X 1 ( A) < · < A m(A), X ( .X , ( XHA X ) < . . . < An (XH AX )
) Hermitian with eigenvalues
n ) such that X H X = eigenvalues of XH AX
·
m x
·
ln , n < : .X ; (A) < A;( X H AX) < Am- n+ i (A), i = 1 , , n. ) H er mitian with eigenvalues .X 1 (A) < · · :S Am(A) and ( 15 ) A, B ( .Xt (B) < · · · < Am ( B ) , respectively ; .X , ( A + B ) < · · · < Am ( A + B) eigenvalues of A + B : n n ( a) 2:: [ -X; (A) + .X ; ( B ) ] < 2:: .X ; ( A + B), n = 1 , m,
. . .
m x m
·
.
i=l
. . , m.
.X , ( B ) < An ( A + B) - An ( A ) < Arn ( B), = I , (c ) I .Xn (A + B ) - An ( A ) Iabs } n = 1, < max{ I .Xj ( B)Iabs : j = 1 , ( d ) An(A + B ) < min{.x ; ( A ) + Aj (B) i + j = n + } = 1, ( b)
. . . , m
n
. .
.
m
,
, m.
·
and
.X; ( A + B) <
.X t (A) < · · · < Am( A) and + B ) < · · · :S A171 ( A + B )
.X; (A) + .A t ( B) .X ;_ 1 (A) + .X 2 ( B)
.X;(A + B ) >
(17)
. . . , m.
,
:
( 16 ) ( Weyl 's theorem ) A , B ( m x m ) H ermitian with eigenval ues .X , ( B) < · · < Am(B), respectively ; .X , ( A t>igenvalues of A + B :
i
. . . , m.
n
.X; (A) + A m ( B ) .X i + t ( A) + Am- t ( B)
= 1, .4 , B ( m ) Hermitian with eigenval ues .X 1 ( A ) < · · < Am ( A ) an d .X , ( B) < · · < Am( B), respectively : .X , ( A - B) < · · · :S Am ( A - B ) eigenval ues of A - B : .X i ( A - B) > 0 => .X; (A ) > .X ; ( B), i 1 , A( ) Hermitian with eigenvalues .Xt (A) < · · · < Am (A), ( .
. . , m.
x m
·
·
=
(18)
m x m
1 ) , c c IR,
[
A B = XH
x] C
. . . , m.
x
m x
1 37
P RO P ERTI ES O F S P EC I A L M ATRICES
with eigenvalues .A 1 ( B ) <
·
·
·
< Am+ I ( B ) :
.A ; ( B ) < .A ; ( A ) < Ai+ I ( B ) ,
i
= 1, . . . , m.
( 19 ) A ( m x m ) Hermitian w ith eigenvalues .A 1 ( A ) < p) Hermitian , C ( m x p) and B
·
·
·
·
< Am(A),
D (p x
< An ( B ) :
.A ; ( B ) < .A; ( A ) < An -m+i ( B ) , 9. 7.4
·
[ �H � ] ( n x n )
=
with eigenvalues .A 1 ( B ) <
·
i
= 1 , . . . , m.
Decompositions of Hermitian Matrices
( 1 ) (Spectral decomposition) A = U A UH . A ( m x m ) Hermitian with eigenvalues .A 1 , . . , .Am w here A = diag( .A 1 , . . . , A m ) an d U is the unitary (m x m) matrix w hose columns are the orthonormal eigenvectors v 1 , . . . , Vm of A associated with .A 1 , . . . , A m . In other words, m A = L .A ; v; v f . i= I .
( 2 ) ( Congruent canonical form) A ( m x m ) Hermitian : There exists a nonsingular ( m x m ) matrix T such that A = TATH , w here A = diag( d 1 , . . . , dm ) and t he d; are + 1 , - 1 or 0 corresponding to the positive, negative and zero eigenvalues of A , respectively. ( 3 ) ( Si multaneous diagonalization of two Hermitian matrices) A , B ( m x m ) H ermitian , AB = BA : There exists a unitary ( m x m ) matrix U such t hat A = U D U H and B = U A U H , w here D and A are d iagonal m atrices. ( 4 ) ( Simultaneous diagonalization of a p ositive definite and a Herm itian matrix) A ( m x m ) positive definite , B ( m x m) Hermitian : There exist s a nonsingular ( m x m ) matrix T such that A = TTH and B = TATH , w here A is a diagonal matri x . Note: M ost results of this section are given i n Horn & J ohnson ( 1 985. Chapter 4 ) or follow easily from results given there. For the results on eigenvalues see also Chap ter 5 and for the decomposition theorems see C'haptt>r 6.
H A N DBOOK OF M ATRICES
1 38
9.8
Idempotent Matrices
Definition: An ( m x m ) matrix A is idempotent if A2 = A . ( 1 ) A ( m x m ) idempotent , B ( n x n ) idempoten t : ( a ) A 0 B is idempotent.
( b ) A EB B is idem potent. (2)
.4 ( m x m ) idempotent : ( a) .4' = A ,
(b)
.4 '
i
= 1,2 . . . .
is idem potent . ( c ) .4 H is idempotent.
( d ) Im - A is idempotent. (e) A l l eigenvalues of A are 0 or 1 . (f) r k ( .4 ) = t r( A ) .
( g) A is simple.
( 3 ) A ( m x m ) idempotent:
( a ) A is uonsingular => A = lm .
( b ) A is Hermitian => A+ = A . ( c ) A is Hermitian
=>
A is positive semidefinite.
( d ) A is diagonal => t he diagonal elements of A are 0 or 1 .
( e ) rk( A ) = r· => t here exists a nonsingular matrix Q such t hat
w here the zero submatrices disappear if r = m. (4) A ( m x n) : (a) A A+ and A+ A are idempotent. ( b ) Im - AA+ and In - A+ A are idempotent . ( c ) A ( A H A ) + A H is idempotent. ( d ) A H ( AA H ) + A is idempotent . ( e ) Im - A ( A H A )+ A H is idempotent. (f)
In -
A H ( A A H )+ A is idempotent.
( 5 ) A ( m x m) :
A idempotent
{::::::}
rk( A ) + rk( Im - A ) = m .
( 6 ) A; ( m x m ) idempotent , i = 1 , . . . , n :
139
P RO P E RTIES O F S P E C I A L M ATRICES
(a) A ; Aj = O m x m . i :f j => A 1 + · · · + A n is idempotent. (b) A 1 + · · · + A n is idem potent ::::> rk ( A 1 + . . . + An ) = rk(A l ) + · · · + rk( A n ) .
( 7 ) A ; ( m; x mi ) idempotent for i = 1 , . . , r : .
is idem potent.
Note: Most of t hese results follow immediately from t he defining property of idempotency. A number of results is given in Lancaster & Tismenetsky ( 1 985) and other matrix books.
9.9
Non negative, Positive and Stochastic Matrices
In this section all matrices are real unless otherwise noted .
9. 9 . 1
Definitions
A = [a;j ] . B = [b;j ] ( m x
n
) real :
A is nonnegative ( A > 0 or 0 < A) if a;j > 0, i = 1 , . . . , m, j = 1 , . . . , n . A is positive ( A > 0 or 0 < A ) if a;j > 0, i = 1 , o o . , m , j = 1 , o o . , n . A<
B (or B > A ) if a;j < b;i , i = 1 ,
A<
B (or B > A ) i f a;i < b;i , i = 1
A = [a;j ] ( m x m) real : A is stochastic if 0 < a;j < 1 , 1, . . . , m.
l,J
,
0
0
•
0
0
•
, m, j = 1 , ,
= 1,
m, j = 1 ,
0
0
.
0
0
•
0
0
•
, m, and
, n. ,
n.
r:; I a;j
A is doubly stochastic if 0 < a;j < 1 , i , j = 1 , . . . , m , 1 , i = 1 , o o . , m , and I:7' 1 a;j = 1 , j = 1 , o o . , m. p ( A) - max { I .X I abs :
A
= 1, i =
I:;'
1
a;j
is eigenvalue of A } is the spectral radius of A.
140
H A N DBOOK OF M ATRICES
9.9.2
General Results
A l l results for non negative matrices are also valid for stochastic and doubly
stochastic matrices because they are special nonnegati ve matrices. ( 1 ) A , B ( m x n) , a , b E lR : ( a) A > 0 , B > 0 , a , b > 0 ::::} a A + bB > 0.
( b ) A > O ::::} A' > O. ( c ) A > 0 ::::} A' > 0. ( 2 ) A, B ( m x
m
)
:
( a ) A > 0 ::::} A' > 0 , i = 0 , 1 , 2 . . . ( b ) A > O ::::} Ai > O , i = 1 , 2 . . . (c) 0 < A < B
::::}
.
.
0 < A' < B' , i = 0 , 1 , 2 . . .
( 3 ) A, B ( m x m ) , C . D ( m x n ) : 0 < A < B, 0 < C < D
::::}
0 < AC < B D .
( 4 ) A ( m x n ) , x ( 11 x 1 ) :
(a) A > 0, x > 0, x :f. 0 ::::} Ax > 0 .
( b ) A > 0 , x > 0 , Ax = 0 ::::} A = 0 . ( 5 ) A ( m x m ) , U = diag( dJ , . . . , dm ) : A > 0 , d; > 0 . i 1 , . . . , m ::::} D- 1 A D > 0. 1 ( 6 ) .-1 ( m x m ) nonsingular, A > 0 : A - > 0 {::::::} A is a general ized permutation matrix . =
(7) A (m x m), A > 0 :
( a ) A is t ridiagonal ::::} all eigenvalues of A are real .
(b) (8) A, B
(a)
(b) (9)
A
A
is stochastic
{::::::}
1
1
1
1
A
( m x m) : A,
A.
H are s tochastic ::::} A B is stochastic.
B are doubly stochastic
::::}
A B is doubly stochasti c .
( 111 x 1T1 ) : A is doubly stochastic <:==:} for some 11 E IN , t here exist ( m x 111 ) permut ation 11 1atrices P1 . . . , Pn and c 1 , c,. E U{ with C J + · · · + c,. = 1 such that A = c 1 P1 + + cn Pn . ,
.
·
·
•
.
,
·
Note: More 0 11 non 1 1egati ve watrices can be found in Mine ( 1 988 ) , Sen eta ( 1 97 3 ) . Berman , �euman n & Stern ( 1 989) , Berman & Plemmons ( 1 97 9 ) .
P RO P E RTIES O F S P ECI A L M ATRICES
141
M any of the results of t his section can also be found in Horn & Johnson ( 1985, C hapter 8 ) . 9.9.3 (1 )
Results Related to the Spectral Radius
A = [a;j ] ( m x ) (a) A > 0 ::::} p(A) > 0 . m
:
m
(b) (c) (d) (e) (2)
(3)
A > 0 , 2::::a ii > 0 , i = 1 , . . . , m ::::} p(A) > 0 . j =I A is stochastic ::::} p( A) > 0 . A i s doubly stochastic ::::} p(A) > 0 . A > 0 , A' > 0 for some i > 1 ::::} p(A) > 0 .
A, B ( m x m ) (a) 0 < A < B ::::} p( A) < p(B). ( b ) 0 < A < B ::::} p(A) < p(B) . A = [a;j] ( m ) , A > 0 : p(A (n) ) < p(A ) , :
m x
are principal submatrices of (4)
n = 1,
.
..
, m,
A.
A = [a;j] ( m x ) A > 0 : ( a) :=max ,l . . . , m a;; < p(A) . m ,
(b) (c )
m
m
m
m
· L: a;i < p(A) < . max L a • i m t = l , . . . , z =min , l J=l J=l . . .
,
rn
_ -l, m i n L a ,J < p(A) < _max L a ;j . _ . J - 1, . , m •=I J . . . , m •=I . .
(d) (5)
A = [a ;j] > O ( m . min, m
•=1,
x m), m
-1 '""
X;
x (x 1 , . . . , .rm ) ' > 0 ( m x =
1
1) :
m
· ; a . maxm - '"" J � a · xJ· .J = l xJ· -< p(A) -< •=1, , J=l �
Xj
.
I)
l
where
142
H A N DBOOK O F M ATRIC ES
. min J=l, , .
rn
Xj
m
a ·
L . _!_!... Xi
•=I
m
< p( A ) <
a··
. =ml a.x, m Xj L . _!_!..., i
J
,
.
•=I
X
( 6 ) A ( m x m ) , A > O , x ( m x l ) , x > O , a, b E IR, a , b > O : ( a ) ax < Ax < bx ( b ) ax < A x < bx ( c ) A x = ax
=}
=}
=}
a < p( A ) < b. a < p( A ) < b.
a = p( A ) .
( 7 ) A ( m x m ) , A eigenvalue of A with eigenvector x :f- 0 : A > 0 . I A i abs = p( A ) =} I A i abs is eigenvalue of A with eigenvector l x l abs · ( 8 ) A ( m x m ) , .4 > 0 : p( A) is eigenvalue of A with eigenvector x > 0 , x :f- 0 . (9) A (m x m), A > 0 :
( a) p( A ) is a simple eigenvalue of A . ( b ) p( A ) is eigenval ue of A with eigenvector x > 0 . ( c ) A is eigenvalue o f A , A :f- p( A )
=}
I A i abs < p( A ) .
( 1 0 ) ( H opf's theore m ) A [a,j J ( m x m ) , A > 0 , A m - 1 eigenvalue o f A w i t h second largest max { a;J : i, j = l , . . . , m } , J.l. = min{ a;i : i , j = modulus, M l , . . . , m} : l Am - I labs < M - J.1. < 1. p( A ) - M + J.1. =
( 1 1 ) A ( m x m ) , A > O , x > 0 ( m x 1 ) eigenvector of A corresponding to eigenvalue p( A ) , y > 0 ( m x 1 ) eigenvector of A' corresponding to eigenvalue p( A ) , x'y = 1 : lim [p(A ) - 1 A ]; = xy' . • - oo
( 12 ) A ( m x m) , A > 0 : ( a) l im [p( A ) - 1 A]i > 0 .
Note: The results o f this subsection are given i n Horn & J oh nson ( 1 985, Chapter 8).
9.10
Ort hogonal Matrices
Definition: A n ( m x m) m at rix A is orthogonal if it is nonsingul ar and A - 1 = A' .
143
P RO P E RTIES OF S P ECIAL M ATRICES
9.10.1
General Results
( 1 ) A (m x m) : ( a) A is orthogonal
-¢::::::>
AA' = Im
-¢::::::>
A' is orthogonal . A H is orthogonal .
( b) A is orthogonal (c ) A is orthogonal
(d ) A is orthogonal (e)
A
is orthogonal
-¢::::::> -¢::::::> -¢::::::>
-¢::::::>
A' A = Im .
-
A is orthogonal . A - I is orthogo nal .
( 2 ) A ( m x m ) orthogonal : ( a)
Ai
is orthogonal for i = 1 , 2 , . . .
( b ) jdet ( A ) I abs
=
1.
( c ) rk( A ) = m . ( 3 ) A ( m x m ) real : (a)
A
is orthogonal => A is normal .
( b ) A is orthogonal (c)
A
-¢::::::>
A is u nitary.
is orthogonal -¢::::::> x' A' Ay = x'y for all real ( m x 1 ) vectors x , y. (d) A is skew-symmetric => ( Im - A ) ( Im + A ) - 1 is orthogonal. (4)
( a ) The ( m x m ) u n i t m atrix Im is orthogonal . ( b ) The ( m n x mn) commutation matrix l\m n is orthogonal .
[ cosSln. B
] 1. s orthogonal.
( c ) A permutation m atrix is orthogonal .
(d) For B E IR ,
8
-sin B COS 8
( 5 ) A; ( m x m) real i = 1 , 2, . . . : => l i m A; is real orthogonal . ( 6 ) .4 , B ( m x m ) :
A ; is orthogon al and lim A; exists
A , B are orthogonal => AB is orthogonal .
( 7 ) A ( m x m ) orthogonal , B ( n x n ) orthogonal : ( a) A 0 B is orthogonal. ( b ) A EB B is orthogonal.
( 8 ) x (m x 1 ) , A (m x m ) : (9) A (m x n), B (n x n) : ( 10 ) A ( m x m ) , B ( m x n ) :
A is rea l ort hog on al => I ! Ax ll2 = l l x l h ·
B is real orth ogo nal => I I A B I I 2 = I I A i h .
A is real ort hog ona l => I I A B I I 2 = I I B i h ·
( 1 1 ) A ( m x m ) real orthogonal:
H A N DBOOK OF M ATRICES
1 44
(a) A is eigenvalue of A => l -\ l abs = 1 . ( b ) ,\ is eigenvalue of A => 1 / ,\ is eigenvalue of A .
( 1 2 ) A ( m x m ) real with eigenvalues -\ 1 , . . . , Am : A is normal . I -\; l a bs = 1 , i = 1 , . . . , m -¢::::::> A is orthogonal. ( 1 3 ) A ( m x m ) real : is sim ilar to A ' .
A is similar to a real orthogonal matrix
=>
A-1
Note: The results of this subsection may b e found i n Horn & Joh nson ( 1 98,5 ) or follow easily from (:� ) . 9. 10.2
Decompositions of Orthogonal Matrices There exists a real orthogonal ( m x m )
( 1 ) ..1 ( m x m ) real ort hogonal: matrix Q such t hat
0
Q ''
A = Q
0
where t he -\; = ± 1 and the A; are real ( 2 x 2) matrices of the form A; =
cos 0, [ -sin B;
sin B; cos B;
].
( 2) A ( 111 x 111 ) ort hogonal : There exists a real ort hogonal ( m x m ) matrix Q and a rPal skew-symmetric ( m x m ) matrix 5 such that A = Qexp ( i S ) . Note: For t h e n·sn l ts of t his subsect ion see Chapter 6 .
9.11
Partitioned Mat rices
Definition:
An
( m x 11 ) matrix
145
P RO P E RTIES OF S P E C I A L M ATRICES
consisting of ( m ; x n1 ) sub matrices A;j , i = 1 , . . . , p, j 1 , . . . , q , is said to be a part itioned matrix or a block matrix. Special cases are =
A = [A 1 , . . . , An] = [A t : . . . : An] ,
where the A; are blocks of columns, and A=
where the A1 are blocks of rows of A . 9.11.1
General Results
( 1 ) A ( m x n ) , B ( m x p) , C (q x n ) , D ( q x p ) :
(a)
[ �· � r [ �: g: ] . -
[ � � ] [ 2. � ] . A B ]H A ll CH ] [ (c) = [ BH DH C D [ (d) [ A B I A i abs ! B l abs ] C D ] abs - I C iabs I D iabs .
(b)
=
·
_
( 2 ) A ( m x n ) . B ( m x p ) , C ( q x n ) , D (q x p) . E ( n x r ) . F ( n x s ) , G (p x r ) . H (p x s ) :
[ AC DB ] [ GE HF ] [ A EE -
( 3 ) A (m x m) , B ( m
x
n ),
tr
] . DH
A F + BH
CF +
x m), D (n x n) :
[� �]
( 4 ) A ( m x n) , B (p x q) :
rk
C (n
C
+ BG + DC
[� �]
=
=
tr ( A ) + tr( D).
rk ( A ) + rk( B ) .
-\(A ) and -\ ( B ) are eigenvalues of A and B. respectively, with associated eigenvectors t•( A ) and t• ( B) => -\ ( A )
( 5 ) .4 ( m x m ) , B ( n x n ) :
146
H A N D BOOK OF M ATRICES
and }.. ( B ) are eigenvalues of
with eigenvectors
[
v( A ) O
]
[ v (0B ) ] '
an d
respectively. ( 6 ) A ( n x n ) Hermitian with eigenvalues A J ( A ) < · · · < A ( A ) D ( p x p ) Hermitian , C ( n x p) and n
,
with eigenvalues A t ( B) < · · < Am ( B ) : ·
A ; ( B ) < A ; ( A ) < Am - n +; ( B )
( 7 ) A ; ( m; x m; ) , i = l , At
. . . , r :
0
I
-
(a) 0
for i = 1 , . . . , n .
Ar
A't
0 i = 0, 1 , 2, . . .
A'r
0
( b ) A; is idempotent , i = l , . . . , r
is idempotent.
9 . 1 1.2 ( 1 ) det
Determinants of Partitioned Matrices
[ I� 10 ]
=(
-
1 )m .
( 2 ) A ( m x m) , B ( m x n) , C ( n x n ) :
[� �] det [ � g ]
det
nxm
mxn
=
det ( A )det ( C ) ,
= det ( A )det( C ) .
147
P RO P E RTI ES O F SPECIAL M ATRI CES
( 3 ) A , B, C ( m
x
m) :
[ CA OBm x m ] ( - l )m det( B)det (C), mxm det [ g ! ] = ( - l )m det( B)det ( C ) . det
( 4 ) A ( m x m) , B ( m
=
) C (n
x n ,
(a) A nonsingular
=>
( b ) D nonsingular
=>
x
m), D (n
[ :,
�]
9. 1 1 .3
:
[ � � ] = det ( A )det ( D - CA - 1 B ) . det( D)det ( A - B D - 1 C). det [ � � ] =
1),
x
E (C :
c
= c det( A ) - b' Aa
( 6 ) ( Fischer's inequality ) A ( m x m) , B ( m x n ) , C ( n D=
)
det
(5) A (m x m ) nonsingular, a , b ( m det
x n
x n
)
:
[ �H � ] positive definite
=>
det ( D) < det(A )det (C) .
Partitioned Inverses
( 1 ) A; ( m; X m; ) nonsingular, i = l , . 0
-
0 (2) A (m x m), B (m
( a)
.
.
, r :
I
0 0
) C (n
x n ,
x
m) , D ( n
x n
)
:
[ � � ] , A and ( D - CA- 1 B) nonsingular =>
[
[ � � rl A - 1 + A - 1 B ( D - C A - 1 B) - 1 CA - 1 - ( D - CA - 1 B) - 1 C A - 1
- A - 1 B( D - CA - 1 B) - 1 ( D - CA- 1 B) - 1
]
.
148
(b)
[� �] r� �
�
[
(3)
H A N D BOOK O F M ATRICES
,
D and (A - BD- 1 C) nonsingular
l r
( A - BD- 1 C) - 1 - ( A - BD- 1 C') - 1 BD- 1 - D- 1 C(A - BD- 1 C)- 1 D- 1 + D - 1 C( A - BD - 1 C) - 1 BD - 1 A (m x m) sym metric, B ( m x n), C ( m x p) , D (n x n ) symmetric, E ( J > x p) sym met ric : A B C -I B' D 0
C'
0
E
-FBD- 1 - FCE- 1 F D- 1 B' FCE - 1 - D- 1 B'F D - 1 + D- 1 B' F BD- 1 E- 1 + E- 1 C' FCE-1 E- 1 C' FBD- 1 - E- 1 C'F if all inverses exist and
F = (A - BD- 1 B' - CE- 1 C' ) - 1 . n , A ( m x n), B ( m x ( m - n)) :
(4)
m, n E
(5)
m. n E IN , m < n , A ( m x n ) , B ( (n - m) x n ) :
IN , m >
rk( A )
�
9 . 1 1.4 (1) A
rk( B ) = n - m, AB11 = 0 � [A H ( A A H ) - I : BH ( B B H ) - 1 ] .
= m,
[ rl
Partitioned Generalized Inverses
(m x
m
) nonsingular .
B ( m x n), C ( r x m ) . D ( x n ) : r·
rk [ � � ] = m. D = cA - 1 B [ OAn-x 1m OOnmxxrr ] t·s a genera1.tzed .mverse or [ CA � ] . __..... -.r
(2) A
=
( m, x u, ), i =
1,
.
...
r :
]
.
PROPERTIES O F SPECIAL M ATRICES
AI
( a)
149
0
0
is a generalized inverse of 0
0
AI
+
0
0
(b) 0
(3)
(4)
A ( m x n) :
A ( m x n) , B ( m x p) :
[
A 0
0 0
] + = [ A+ ]
_[
D A+ A+ ) B (C+ + . ] [A B + c+ + D ·
w here D
(5)
0
]
0 0
.
·
C = ( /m - AA+ ) B and
= ( /p - C + C )[Ip + ( /p - C+ C )BH (A + )H A+B(lp - C+c)t 1 x BH ( A + )H A + ( /m - BC + ) .
A ( m x n) , B ( p x n) :
[ � r = [A + - TsA + : T] ,
= E+ + ( In - E+ B)A + ( A + )H BH 1\(lp - EE+ ) with E = B(ln - A+ A) and /\. = (Ip + (lp - EE+ ) BA + ( A + )H BH ( lp - EE+ W 1 .
w here T
9. 1 1 .5 ( 1 ) Dn
Partitioned Matrices Related to Duplication M atrices
(n2 x �n(n + 1 ) )
duplication m atrix : 1 0 0 1 0 0
0
2/m 0
0
� fm 0
0 0 D:r. Dm
H A N DBOOK OF M ATRICES
! 50
(2)
A(
)
m x m ,
a, b (
m x
1 ) , c E (C :
[
1
C
Dm+ l vee a
[
�
]
c
]-
+ l vee c b' Dm+ a A (3)
A, B ( (a)
m x m
D� + l
) symmetri c , a, b ( m
x
a+b D:n vee( A) c
;(a + b) D;!:. vee( A) 1 ) , a, {3 E (C :
( [ : � ] 0 [ � � ] ) Dm+ l =
ab' + (3a' (a' ® b')Dm a B + {3A + ab' + ba' (a' 0 B + b' ® A)Dm D:n (A ® B)Dm D� (a 0 B + b ® A)
(a' 0 b')D;!:. a(3 ; ( ab' + (3a') ; (ab + (3a) haB + {3A + ab' + ba' ) ; (a' 0 B + b' 0 A ) D;t.' D;!:. (a ® b) ; D;t. (a 0 B + b 0 A) D;t", ( A 0 B)D;t.' '
Note: Results on partitioned matrices including a number of the foregoing ones are given , for i nstance, in M agnus ( 1 988) and Magnus & Neudecker ( 1 98 8 ) . Many of the foregoing results are immediate consequences of definitions.
9.12
Positive Defi nite, Negative Definite and Semidefinite Matrices
Definitions: A Hermitian ( m • • •
•
x m
) matrix
A is
positive defin ite if xH Ax > 0 for all ( m x 1 ) vectors x # 0; positive semidefinite if xH Ax > 0 for all ( m x 1 ) vectors x; negative definite i f xH Ax < 0 for all ( m x 1 ) vectors x # 0 ; negative semidefinite i f xH Ax < 0 for all ( m x 1 ) vectors x .
Note that a definite m atrix is always Hermitian accord ing to t his definition. A real sym metric ( m x m ) matrix A is •
positive definite if x' Ax
> 0 for all real (
m x
1 ) vectors
x # 0;
151
P RO P E RTIES OF S P E C I A L M ATRICES •
• •
positive sem idefinite if x' Ax > 0 for all real ( m x 1 ) vectors x ; negative definite if x' A x < 0 for all real ( m x 1 ) vectors x :f- 0; negative sem idefinite if x'Ax < 0 for all real ( m x 1 ) vectors x .
Without special notice, a real definite matrix is always understood to be symmetric. 9.12.1
General Properties
( 1 } A , B ( m x m) :
( a) A positive definite, B positive semidefinite => A + definite.
(b) A negative definite, B negative sem idefinite definite. (c) A , B positive sem i definite (d) A, B negative sem idefinite ( 2 } A ( m x m ) , c E ffi :
=>
=>
A + B negative semidefinite. cA is positive (sem i )
( b ) A negative (semi ) defi nite, c > 0 definite.
A positive (sem i ) definite, c < 0
=>
=>
( d ) A negative (semi) definite, c < 0 -¢:::::::>
(e) A positive (semi) definite (f) A =
[a;j] -¢:::::::>
(g) A =
[a;j]
A + B negative
A + B positive semidefinite.
( a) A posi tive (semi) definite, c > 0 definite.
(c)
=>
B positive
cA negative (semi ) cA negative (sem i ) definite. cA positive (sem i ) definite.
-A negative (sem i ) definite.
positive (semi) definite d et
> ( > ) 0 for i = 1 , . . . , m.
a;; negat ive (sem i ) definite
-¢:::::::>
a
II
.
..
a li
det
a;
1
for i = 1 , . . . , m. ( h ) A positive (sem i ) definite i = 1 , 2, . . .
a;; :::}
{
> ( > ) 0,
z
< (<) 0,
i odd
.
even
A' positive ( semi ) definite for
1 52
H A N DBOOK O F M ATRICES
( 3 ) A . B ( m x m)
:
( a) (Schur product theorem) A . B positive (semi ) defi n ite ( b ) A , B positive sem idefinite (4) A (m x m), B (n x n) : positive (sem i ) definite.
'*
'*
A (� ' B positive (sem i ) definite.
rk ( A C::> B ) < rk( A ) rk ( B ) .
A , B positive ( semi ) definite
'*
A
4
B
( 5 ) A ( m x m ) positive (sem i ) definite: ( a) det ( A ) > ( > ) 0 . ( b ) t r ( A ) > ( > ) 0. (6) A ( m x m) negative ( semi) definite :
tr( A) < ( < ) 0.
(7) A (m x m ) positive ( negative) definite: (a) r k ( A )
= m.
( b ) A is nonsingular. ( c ) .4 - t is positive ( negative) definite. (8) A ( m x m) Hermitian : A is idempotent '* A is positive semidefinite. ( 9 ) A = d i ag( a t J , . . . , amm ) : ( a) A positive (semi ) definite ( b ) A negative (sem i) definite ( 10 ) A ( m x m ) , B ( m
x
<::::=:}
<::::=:}
aii > ( > ) 0, i aii < ( < ) 0 , i '*
( b ) A positive ( negative) definite, rk( B ) ( negative) definite. ( 1 1 ) ( Fischer 's inequality) A (m x m). B (m x n), C (n
[ ABH B ] c
( 12 ) ( A itken's integral )
=
I , . . . , m.
1 , . . . , m.
n) :
( a) A positive ( negative) semidefini te ( n egative) semidefinite.
D=
=
x
n
=
B H A B positive '*
B H A B positive
n) :
positive definite
.4 ( m x m ) real positive definite, x
=
'*
det ( D ) < det ( A )det ( C ) .
(x 1 , . . . x m ) (m ,
'
x
1 ) real:
1 53
PROPERTIES O F S P EC I A L M ATRICES
Eigenvalue Results
9 . 1 2. 2
( 1 ) A ( m x m ) Hermitian (or real symmetri c ) : ( a ) A is positive ( semi ) definite positive ( nonnegative) .
¢::::::>
all eigenvalues of A are
( b ) A is negative (semi ) definite negative ( nonpositive ) .
¢::::::>
all eigenvalues of
( 2 ) A = [a;j ) ( m x m ) positive definite with eigenvalues A 1 n
n
i=l
i=l
<
···
A
are m
:
n = 1 , . . . , m.
II A ; < II a ;; ,
( 3 ) A ( m x m ) positive definite with eigenvalues A 1 < · · · < Am , X (m x n): n
n
II A ; < det( X H AX ) < II Am - n + i · i= I
( 4 ) A ( m x m ) real positive definite with eigenvalues ) q < Am , X ( m x n ) real : X' X = In
=>
<
n
n
II A ; < det ( X ' A X ) < II Am -n +i · i=l
( 5 ) A = [a;J) ( m x m ) positive definite with eigenvalues A 1 < · · · < Am n
''
i= I
i= I
II A; < det ( A ( n ) ) < II Am - n + i ·
(6)
m ) Hermitian (or real sym metric ) , B ( m x m ) posit ive semidefinite: Am in ( A + B) > Amin ( A ) ,
A (m
x
Am ax ( A + B) > Am a x ( A ) . ( 7 ) A ( m x m ) Hermitian (or real symmetric) with eigenvalues A 1 ( A ) < < · · < A m ( A ) , B ( m x m ) positive semidefinite, A 1 ( A + B ) < A m ( A + B ) eigenvalues of A + B : ·
>., ( A + B) > >.; ( A ) ,
i = l, .
. . , m.
! 54
(8)
H A N DBOOK O F M ATRICES A
( m x m ) positive definite with eigenvalues ) q { 1 , . . . , m} : ..\ 1 · ..\ 2 · · A ·
<
···
<
..\ 771 ,
n
E
n=
min { ( .r{f A.r l ) · · · ( x :; A xn ) : Xj ( m X 1 ) , X = [ x i · · . . , xn ] (m X ) x H x n
,
( 9 ) :1 (m x m ) positive definite with eigenvalues ..\ 1 < ..\ 2 < · · assoc iated orthonormal ( m x 1 ) eigenvectors t> 1 , . . . , Vm : ·
=
In } -
<
>.rn
aud
Illin{ det ( B H A B ) : B ( m X ) B H B = In } = ..\ 1 · ..\ 2 · · · An . n
,
The minimizing matrix is B = [ v1 , . . . , vn ) -
The maximizing matrix is B = [ v m . . . . , Vm - n + d · ( 10 ) A ( m x m ) real positive definite with eigenvalues ..\ 1 < ..\ 2 < · · · < and associated orthonormal ( m x 1 ) eigenvectors v 1 , . . . , Vm : min{ det ( B ' A B ) : B ( m x The minimizing matrix is B
=
n
)
real, B ' B
= In }
= ..\ 1 · ..\ 2 · · · A n .
[ v 1 , . . . , vn ] -
max{ det ( B ' A B ) : B ( m x ) real , B ' B = In } n
>. ,.
=
>. , . · Am - I · A m - n + 1 . ·
·
The Illaximizing matrix is B = [ vm , . . . , Vm - n + J 9. 12.3 (1)
Decomposition Theorems for Definite Matrices
A (m x m ) ( a) A is positive definite '* A is similar to Im . ( b ) A is positive semidefinite, rk ( A) -'* A IS. similar to :
[ �n 00 ] .
n
A is negative definite '* A is similar to - I771 • ( d ) A is negative semidefinite, rk (A ) = A I. S similar to '* (c)
(2)
[ -�n
A (m x m) :
�].
n
( a) ( Choleski decom position) A is positive definite ¢::::::> there ex ists a lower triangular ( m x m ) matrix B with real positive elements on the principal diagonal such that A = B B H .
P RO P E RTIES O F SPECIAL M ATRICES
1 55
( b ) ( Choleski decomposition of a real m atrix) A is real positive definite ¢::::::> there exists a real lower triangular ( m x m ) matrix B w i th positive elements on the p rincipal diagonal such that A = B B' . (c) (Square root decomposition) A is positive (semi ) definite ¢::::::> there exists a positive (semi) definite ( m x m) matrix B such that A = BB, that is , B is a square root of A . ( d ) ( Square root decomposition of a real matrix) A is real positive (semi) definite ¢::::::> there exists a real positive (semi) definite ( m x m ) matrix B such that A = B B , that is, B is a real square root of A . (e) A is positive definite ¢::::::> there exists a nonsingular ( m x m ) m atrix B s uch that A = B H B .
( f) A is real positive definite ¢::::::> there exists a real nonsingular ( m x m ) m atrix B such that A = B' B.
( 3 ) A ( m x m) : A is positive semidefin i te, rk ( A ) = r =} there exists an ( m x r) matrix B w i th rk ( B ) = r such that A = BB H and A+ = B ( B H B ) - 2 B H .
( 4 ) (Simultaneous diagonalization of a positive definite and a Hermitian matri x ) A ( m x m) positive definite, B ( m x m ) Hermitian : There exists a nonsingular ( m x m ) matrix T such that A = TT H and B = TAT H , w here A is a diagonal matrix. ( 5 ) (Simultaneous diagonalization of a positive definite and a symmetric matrix ) A ( m x m ) positive definite, B ( m x m ) symmetric: There exists a nonsingular ( m x m ) matrix T such that A = TT H and B = TAT' , where A is a real diagonal matrix with nonnegative diagonal elenwnts.
Note: M any results on positive and negative (semi ) definite matrices including proofs can be found in Horn & Johnson ( I 985, Chapter 7 ) and other books on matrices. See also Chap ter 4 for results related to traces and determinants of defin i te matrices. For Aitken 's integral see Searle ( 1 982, p . 340) and for t h e decomposition results see Chapter 6. Since positive ( negati ve) definite and semidefini te matrices are Hermitian , all results for the l atter also hold for the former matrices (see Section 9 . 7 ) .
H A N DBOOK O F M ATRICES
1 56
9.13
Symmetric Matrices
Definitions: An (m x m ) m atrix A is symmetric if A' = A , that is, the ijt h element a ;j i s equal to the j ith element, aji = a ;j , so that
a1 1 a1 2
A=
a1m a2m
a12 a 22
An ( m x m) m atrix A is skew-symmetric if A' = - A , that is, the ijth element a;j is - I times the jith element, a ;j = - a ;j , so that 0 -a12
A=
a 1m a 2m
a12 0
- a i m - a 2m
9.13.1
0
General P roperties
( 1 ) A ( m x m) real:
A is symmetric
( 2 ) A , B ( m x m) sym metric, c E (I; :
¢::::::>
A is Hermitian .
(a) cA is sym metric.
(b) A ± B is symmetric. ( c ) .4 8 = BA ��
:::}
A B is symmetric .
B is symmetric. ( 3 ) A ( m x m ) sym metri c , B ( n x n ) symmetric: (d) A
(a) A 0 B is symmetric.
(b)
A if' B is symmetric.
( 4) A = [ a ;j) (m x m) sy m metric: ( a ) A' is sym metric.
(b)
A H is sym metric . ( c ) A - ! is sym metric , if A is nonsingular.
( d ) A a dj is sym metri c . ( e ) A' i s sym n 1 etric for i = 1 , 2 , . . . (f)
A ..
is symmetric .
PROP ERTIES O F S P E C I A L M ATRICES
157
(g) I A iats is sym metric .
( 5 ) A (m x m ) :
...
1 , 2 , . . . , m.
=
is symmetric for i
(h) a;;
(a) A + A ' is sym metric . ( b ) A - A ' i s skew-symmetric. (c) A A' is sym metric. ( d ) A' A is symmetric. ( 6 ) A (m x m) : {:::=}
( a) A is symmetric m atrices B .
B'AB is sym metric for all (m x m)
( b ) A is symmetric {:::=} there exists a real orthogonal ( m x m) matrix Q and a real diagonal ( m x m) m atrix A such that A = QAQ' . ( 7) A ( m x m ) , B ( m x n ) :
A is sym metric
( 8 ) A, B ( rn x m) real sym metric: (a) [tr( A B )F < tr ( A 2 B 2 ) .
( b ) rk(A) >
B' A B is symmetric.
( tr A )2 if tr( A2 ) =f. 0 . , tr ( A 2 )
( 9 ) A ( m x m ) real , b ( m x I ) real ,
B=
'*
[ b b'A ] 1. s symmetnc.
c
c
E IR : '*
det( B)
=
c
det ( A ) - b'Aa dj b
( 10) A ( m x m) , B ( n x n ) , C ( m x n) :
A , B are symmetric
'*
[ �� � ]
is symmetric.
Note: M any of these resu lts are elementary. Most others can be found in Horn & Johnson ( 1985 ) . More rules for sym metric matrices are given in t he section on positive defi nite and semidefinite matrices which are symmetric by definition if they are real (see Section 9 . 1 2 ) . 9.13.2
Symmetry and Duplication Matrices
( 1 ) A ( m x m) sym metric ,
Dm ( m 2 x � m( m + 1 ) ) duplication matrix:
1 58
H A N D BOOK O F M ATRICES
(a) vec(A) = Dm vech ( A ) . ( b ) D;t",vec(A) = vech ( A ) . ( 2 ) A ( m x m) sym metric nonsingular, c E (I; : ( a) det ( D;t". [A 0 A + c vec(A)vec( A )'] Dm ) = ( 1 + cm)det ( A ) m + l . ( b ) ( D;t", [A 0 A + c vec (A )vec ( A )'] Dm ) - 1 = D + A - 1 0 A - 1 - c vec(A - 1 ) vec( A - 1 )' Dm · m
[
1 + em
( 3 ) A , B ( m x m) symmetric, a , b (m x 1 ) , a , (3 E (I; :
( [ a a' ] [ /3 b' ] ) Dm + l = (a) Dm + l a A b B ,
]
0
ab' + f]a' a;J (a' 0 b' ) Dm ab + ;Ja a B + f]A + ab' + ba' (a' 0 B + b' 0 A ) Dm D:-r, ( A 0 B)Dm D:-r, (a 0 b) D:n (a 0 B + b 0 A )
'
hab' + f]a' ) (a' 0 b') D;t", a/3 � (ab + f]a ) � (aB + f]A + ab' + ba' ) � ( a' 0 B + b' 0 A ) D;t' D;t", ( A 0 B) D;t' D;t". ( a 0 b) � D;t", (a 0 B + b 0 A ) Note: These results are given in Magnus ( 1988) (see also Section 9 . 5 ) . 9.13.3
Eigenvalues of Symmetric Matrices
denotes an eigenvalue of the matrix A . ..\(A) >. m in ( A ) is the smallest eigenvalue of the matrix A . >. m a.r ( A ) is the largest eigenvalue of the matrix A . ( 1 ) A ( m x m ) real symmetric: (a) All eigenvalues of A are real numbers. (b) A is positive definite ¢::::::> all eigenvalues of A are real and greater than 0 . (c) A positive semidefinite ¢::::::> all eigenvalues of A are real and greater than or equal to 0 . ( 2 ) A ( m x m) real symmetric with eigenvalues ..\ 1 , . . . , >.m and associated orthonormal eigenvectors V 1 , . . . , Vm : m A = 2: ..\; v; v: .
Notation:
i=I
159
P RO P E RTIES O F S P ECI AL M ATRICES
( 3 ) ( Rayleigh-Ritz theorem) A ( m x m) real symmetri c : Am;,. ( A ) Amax ( A)
Ax { = min x' x { x'xAxx = m ax x
'
( 4 ) (Courant-Fischer theorem) A (m x m ) real symmetric w ith eigenvalues ..\ 1
A; =
Yt
(m X
x
1 I
Ym-
I
1)
and ..\ ; =
11 1
(m X
l
.
··
<
Am ,
real, x # 0, x'Yj = 0 , j
=
1 . . ., m ,
-
x'Ax { min : xx
max
. . .
·
#0 .
'X
)real
(m x
<
} }
#0 ,
x' Ax { : m ax x
m in
•
x
: x ( m x 1 ) real,
'
1
x
x ( m x 1 ) real,
Y, - t
'
)rea)
x ( m x 1 ) real, x # 0, x' YJ = 0 , j = 1 , .
..,i
-
1
i} }.
( 5 ) A (m x m) real symmetric with eigenvalues ..\ 1 < · · · < A m and associated orthonormal eigenvectors v 1 , . . , Vm , n E { 1 , . , m } : .
min{ tr( X ' A X ) : X
(m x n )
(m x n)
. . , v,. .
]
real , X' X = I,. } = A m + · · · + A m - n + l ·
The maximizing matrix is X = [ vm , . . . , Vm - n + d · Note: For these results see Chapter 5 . 9 . 1 3 .4 (1) A
Eigenvalue Inequalities
(m x m ) real symmetric, x # 0 ( m Am;n ( A )
.
real , X ' X = I,. } = ..\ 1 + · · · + ..\,. .
The m i n imizing matrix is X = [ v 1 , . max{ tr(X ' A X ) : X
.
<
x' Ax x' x
--
x
1 ) real:
< Amax ( A ) .
1 60
H A N DBOOK O F M ATRICES
( 2 } A ( m x m ) real symmetric with eigenvalues ) q ( A ) , . . . , >. m ( A ) :
. { x' A x
mm
:
'
x x
x ( m x 1 ) real ,
{ x'Ax,
< ..\; ( .4 ) < max i
=
X X
x
( m x 1 ) real ,
1 , . . . , m.
(m
x m ) real symmetric with eigenvalues ..\ 1 ( A ) Am ( A ) , X ( m x n ) real:
(3) A
n
L ..\; ( A ) < tr ( X' A X )
X ' X = In (4) A =
[a;j] ( m
i=l
[a ;j] ( m
Am ( A )
:
n
L ..\; ( A ) < L a;; i= I
i=I
[a;j] ( m x m )
>.rn ( A ) :
· <
n
L Am-n+i( A ) . i=I
i
=
1 , . . . , m.
x m ) real symmetric with eigenvalues ..\ 1 ( A ) <
n
(6) A =
· .
x m ) real symmetric:
Amin ( A ) < a;; < >.max ( A ) , (5) A =
<
<
n
L Am - n +; ( A )
<
i= 1
<
n = 1 , . . , m. .
,
real symmetric with eigenvalues 0 < ..\ 1 ( A ) <
n
n
II >., ( A ) < II a; ; , •= I
i=l
n
=
1,
.. .
.
·
·
·
<
m.
( 7 ) ( Inclusion princip le ) A ( m x m ) real symmetric with eigenvalues ..\ 1 ( A) < · < >.m ( A ) , A( n l ( n x n ) principal submatrix of A with eigenvalues ..\ i ( A r n d < · · · < A n ( A ( n ) ) : ·
·
Amin ( A ) < Amin ( A( n j ) < Ama.r ( A ( n ) ) < Amax ( A ) . (8) A (m x
rn
) real symmetric,
B ( m x m)
>.m i n ( A + B) Ama.r ( A
real p ositive semidefinite:
> Ami n ( A ) ,
+ B) > Amax ( A ) .
161
P RO P E RTIES O F S PECIAL M ATRICES ( 9 ) A, B ( m x m) real symmetric:
Amon ( A + B ) > Amin ( A ) + Amin ( B ) , Amax ( A + B) < Amax ( A ) + Amar ( B ) . ( 10 ) A, B ( m x m) real symmetric, 0 < a, b E IR : >. min ( a A + b B ) > a>. min ( A ) + b >.m;n ( B ) , Amax ( a A + bB) < a>. max ( A ) + b >. max ( B ) . ( 1 1 ) A ( m x m) real symmetric with eigenvalues >. ! ( A ) < · · · < < >.m ( A ) , B ( m x m) real positive semidefinite, >. ! ( A + B ) < Am ( A + B ) eigenvalues of A + B : >.; ( A + B ) > >.; ( A ) ,
i = 1 . . . , m. ,
< ( 12 ) A ( m x m) real symmetric with eigenvalues >. ! ( A ) < >.m ( A ) , x ( m x 1 ) real, .>.1 ( A ± xx' ) < · · · < >.m ( A ± x.r ' ) eigenvalues of A ± x x ' : ·
>.; ( A ± x x ' ) < >.;+ I ( A) < >.; + 2 ( A ±
.r .r
'
),
>.; ( A ) < >. i + ! (A ± xx ' ) < >.;+ 2 ( A ),
·
·
i = 1 , 2, . . . , m - 2;
i = l , 2, . . , m - 2.
( 13 ) A , B ( m x m ) real symmetric, rk ( B ) < r·, >.! ( A) < · · < >.m ( A ) eigenvalues of A , >.! ( A + B ) < · · · < >.m ( A + B ) eigenvalues of A + B : ·
>.; ( A + B ) < >.;+ r ( A ) < >.;+ 2r ( A + B ) , >., ( A ) < >. i +r ( A + B ) < .>., + 2 r ( A ) ,
i i
=
= 1 , 2 . . . . , m - 2r; 1 , 2,
. . . , m -
2r.
( 1 4 ) ( Poincare's separation theorem ) A ( m x m) real symmetric w ith eigenvalues .>.! ( .4 ) < · · · < Am ( A ) , X ( m x n ) real such that X ' X = In , Tl < m , >. I ( X ' A X ) < · · · < An ( X' A X ) eigenvalues of X' AX :
>.; ( A ) < A;( X ' A X ) < Am - n + ; ( A ) .
i = 1 , . . . , n.
( 1 5 ) A , B ( m x m ) real symmetric with eigenvalues >. ! ( A ) < · · · < >.m ( A ) and >. 1 (B ) < · · < Am( B ) , respectively; >. ! ( A + B ) < · · · < >. ( A + B ) eigenvalues of A + B : n n ( a ) �:_) >.; ( A ) + >.; ( B)] < L >.; ( A + B ) , n = 1 , . . . , m . m
·
i= I
(b) ..\ 1 ( B ) <
i= I
An ( .4 + B ) - An ( A ) < Am ( B ) ,
n = 1 , . . . , m.
162
H A N DB OOK O F M ATRICES
( c) I .:X n ( A + B) - An ( A ) Iabs < max{ l .:\j ( B ) I abs n = 1,
.
.
:
. , m.
j = 1, .
( d ) A n ( A + B ) < min{.:\; ( A ) + Aj ( B ) : i + j = n + n = 1 , . . , m.
.
.
, m
},
},
m
.
( 16 ) ( Weyl 's theorem ) A , B ( m x m ) real symmetric with eigenvalues .X 1 ( A ) < · < Am ( A ) and A 1 ( B ) < · · < Am ( B ) , respectively; .:\ J ( A + B) < · < Am ( A + B ) eigenvalues of A + B : ·
·
·
.:\; ( A + B) >
and
·
.X; ( A ) + .:\ 1 ( B ) Ai - I ( A) + .:\ 2 ( B )
.X i ( A ) + .Xm ( B ) A i + I ( A ) + Am _ I ( B )
.:\; ( A + B) < i
·
Am ( A ) + .X; ( B )
1 , . , m. ( 1 7 ) A, B ( m x m ) real sym metric with eigenvalues .:\ 1 ( A ) < · · < Am ( A ) =
.
.
·
and .:\ I ( B ) < · · · < Am ( B ) , respectively; A 1 ( A - B) < eigenvalues of A - B :
.X J ( A - B) > O
=>
·
.:\, ( A ) > .:\ ; ( B ) , i = l ,
·
.
·
.
< Am ( A - B )
.
, m.
( 1 8 ) A ( m x m ) real symmetric with eigenvalues .:\ J ( A ) < Am ( A) , .r ( m x 1 ) real , c E IR. ,
B=
<
X ] [ x' A
c
with eigenvalues .X I ( B) < · < Am+ ! ( B) : ·
·
.X; ( B ) < .X; ( A ) < A;+ 1 ( B) ,
i = 1, .
. . , m.
( 19 ) A ( m x m ) real sy mmetric with eigenvalues .:\ 1 (A ) < A m ( A ) , D (p x p ) real sym metric , C ( m x p) real and
B=
[ �1 � ] ( n X n)
with eigenvalues .:\ 1 ( B ) < · · · < An ( B) :
.:\; ( B ) < .:\; ( A ) < An-m+i ( B ) Note: For these results see Chapter 5 .
for i = 1 , .
.
. , m.
<
163
PROPERTIES O F SPECIAL M ATRICES
9.13.5
Decompositions of Symmetric and Skew-Symmetric M atrices
( 1 ) (Spectral decomp osition ) A ( m x m ) real symmetric with eigenvalues A 1 , . . . , Am : A = Q AQ', w here A = d i ag(A 1 , . . . , Am ) and Q is the real orthogonal ( m x m ) matrix w hose columns are the orthonormal eigenvectors v 1 , . . . , Vm of A associated with A 1 , . . . , Am . I n other words, A =
m
L A ; v; v; . i=l
( 2 ) A ( m x m ) symmetric, rk ( A ) = ( m x m ) matrix T such that A =T
[
r
lr
O
There exists a nonsingular
� ] T' ,
where the zero submatrices disappear if r = m. ( 3 ) A ( m x m ) real skew-symmetric: ( m x m ) matrix Q such t hat
There exists a real orthogonal
0
0 0
A=Q
Q',
0
where the A; are real (2 x 2 ) matrices of the form b; 0
]
.
( 4 ) ( Simultaneous d i agonalization of a real positive definite and a real symmetric matrix) A ( m x m ) real p ositive definite, B ( m x m ) real symmetric: There exists a real nonsingular ( m x m ) matrix T such that A = TT' and B = TAT' , w here A is a real diagonal matrix . Note: For more details o n matrix decomposition results see Chapter 6 .
H A N DBOOK OF M ATRICES
164
9.14
Triangular Matrices
Definitions: A n ( m x m) matrix aI I
0
0
= [aij ] ,
A= ami with Uij
=
0 amm
...
0 for j > i , is a lower triangular matrix . A n ( m x m) matrix ... .
. .t
-
0 0
0
Umm
w ith a;j = 0 for i > j, is upper triangular . A is triangular if it is upper t riangular or lower triangular. A is strictly upper ( lower) triangular if a;j = 0 for i > j ( i < j ) , that is , all elements on the principal diagonal in addition to the elements below or above that diagonal are zero. A is strictly triangular if it is strictly upper or strictly lower triangular. 9.14.1
Properties of General Triangular Matrices
( 1 ) A , B ( m x m) upper ( lower) triangular, c E
(b)
c
A is upper ( lower) triangular.
( c ) A B is upper ( lower) triangular. ( d ) A (�) B is upper ( lower) triangul ar. ( 2 ) A (m x m ) upper ( lower) triangular, B ( n x n ) upper ( l ower ) triangul ar:
( a) A (b) A
r:9
cl•
B is upper ( lower) triangular. B is upper ( lower) triangular.
( 3 ) A ( m x m) upper (lower) triangular: ( a) A' is lower ( upper) triangul ar. ( b ) A H is lower (upper) triangular . ( c ) A is upper (lower) triangular. ( d ) A - 1 is upper ( lower) triangular, if A is nonsingular.
1 65
P RO P E RTIES O F SPECI A L M ATRICES
(e) A a d] is upper ( lower) triangular. (4)
A = [ a;; ] ( (a) (b)
m x m
) triangular:
m t r ( A ) = a 1 1 + · · · + a mm = L a;; . i=l m d et ( A ) = a 1 1 · · · amm = II a;; . i=l
( c ) rk( A ) > num ber of nonzero diagonal elements. (d) a 1 1 , . .
.
, am m
are the eigenvalues of A.
(e) A is symmetric => (f) A is Hermitian =>
A = d iag( a 1 1 , . . . , amm ) · A is real diagonal .
Note: Most of these results are simple consequences of t he definitions. See Magnus ( 1 988) for more details. 9.14.2 (1) (2)
Triangularity, Elimination and Duplication Matrices
A( ): A = [a 1 ] , B (a) D;!-, (A
A lower triangular => L'm vech( A )
m x m
= vec ( A ) .
[b;; ] ( m x m ) upper ( lower) triangular: 0 B ) Dm is upper ( lower) triangular with diagonal elements � ( a ;;bj] + aj jb;; ) , 1 < j < i < m . ;
(b)
=
det [ D;!", ( A 0 B ) Dm ] = � IT (a; ; bjj + ajj b;; ) i>j
(c) L m ( A 0 B)L'm is upper (lower) triangular wit h diagonal elements a ;,bjj , 1 < i < j < m .
m
i= 1
(e) L m ( A 0 B ) Dm is upper ( lower) triangular w i t h diagonal elements a;; b]J , 1 < i < j < m .
m
(3)
A = [a ; 1 ]
i= I ,
B = [b ;;] ( m
x m
) lower triangular:
m
i= I
( b ) a ;;b;i • 1 < i < j < ( c ) [L ( A m
'
0
m,
are the eigenvalues of
B ) L �, ] ' = Lm [( A ' ) ' 0 B ' ] L 'r, for
L m ( A' 0 B) L:71 •
H A N DBOOK O F M ATRICF:S
166 s
0 , 1 , 2, . . . . . - 2, - 1 , if A , B nonsingular
s
.
.!.
if lower triangular A 1 12 B112 exist , 2 (d) det[D;!; ( A ® A ± B ® B ) Dm ] = II (a;;aii ± b;;bj j ) . i>j (e) det [Lm ( A B' 0 B' A ) L :r,] = det[Lm ( A B' 0 A' B ) L:r,] = (de t A ) m + 1 ( de t B ) m + 1 . ( f ) L :r, Lm ( A' ® B ) L :r, = (A' ® B ) L :r, . s
,
( 4 ) A = [a;i ] , B = [b,i ] , C = [c;j ) , D = [d;i J ( m x m) lower triangular : m m det [Lm ( A B' @ C' D) L:n J = II (c;;d;d (a;;b; ; ) -i+ l , i= I
det[Lm ( A' @ B + C ' @ D)L:nJ =
II (b ;;aj j + d;; cj j ) ·
i>j ( 5 ) A, B , C, D (m x m) nonsingular, lower triangular:
[L m ( A B' 0 C' D)L:n J - 1 = Lm [ ( B' ) - 1 ® D- 1 ) L:n L m [A - I ® ( C ' ) - 1 ] L:r, . ( 6 ) A = [a ;j ] ( m x m ) lower triangular, n E det
Lm
n-1
L ( A' t - l - j ® Aj
j =O
where
J.l k l =
{
IN :
L'm
(ak k - a;; ) / ( a u - a l l ) n a nk k- 1
= n m ( det A t - l
II J.l k l ,
k>l
if a u =/:- a ll
if a u = a ll ( 7) A = [a;i ] , B = [b ;j ] , C = [c;j ] ( m x m) lower triangular: ( a) Lm ( A 0 B ' ) Dm Lm ( A ® B' ) L :r, . ( b ) det[Lm ( A B 0 B' A ) Dm ] = det [L m ( AB' 0 A' B' ) Dm ] = det[Lm ( A B' ® B' A ) L:r, ] = ( de t A ) m + l ( de t B ) m + 1 . m ( c ) det [Lm ( A ® B'C) Dm ] = II (b;;c; d a7,' - i + 1 • i=l m m (d ) det [L m (A B' ® C' ) Dm ] = II c:; (a;;b;; ) - i + l . =
i=l
Note: The foregoing results may b e found in M agnus ( 1 988) which is a rich source of results for t ri angular matrices and their relationship to d uplication and elimination matrices.
P RO P E RTIES O F SPECI A L M AT RICES
9.14.3
167
Properties of S trictly Triangular Matrices
( 1 ) A ( m x m) strictly triangular: ( a) A is si ngular. ( b ) A m = Om x m · ( c ) tr( A ) = 0 . ( d ) det ( A ) = 0 . ( 2 ) A , B ( m x m ) strictly upper ( lower) triangular, c E
( b ) cA is strictly upper ( lower) triangul ar. (c) A B is strictly upper ( lower) triangular. (d) A
0
B is strictly upper ( lower) triangular.
(e) Lm ( A' 0 B ) L'm is nilpotent.
( 3 ) A ( m x m ) strictly upper ( lower) triangular, B (n x n ) strictly upper ( lower) triangular: (a) A 0 B is strictly upper ( lower) triangular. (b) A
EEl
B is stri ctly upper (lower) triangular.
( 4 ) A ( m x m ) strictly upper (lower) triangul ar:
( a) A' is strictly lower ( upper) triangular. ( b ) A H is strictly lower ( upper) triangular.
( c ) A is strictly upper ( lower) triangular. ( d ) A a dJ is strictly upper ( lower) triangular. Note: These results are easy implications of t he defi nitions. 9.15
Unitary Matrices
Definition: A n ( m x m ) matrix A is unitary if it IS nonsingular and A - I = AH . ( 1 ) A ( m x m) :
( a) A is unitary -<==> the colu m ns of A are orthonormal vectors.
( b ) A is unitary -<==> t he rows of A are orthonormal vectors. ( c ) A is u n itary -<==> x H A H Ay = x H y for all ( m x l ) vectors x , y. ( 2 ) A ( m x m ) : A is unitary -<==> A A H = A H A = Im .
H A N DBOOK OF M AT RICES
1 68 -¢::=>
¢::::::}
¢::::::}
¢::::::}
¢::::::}
( 3 ) A ( m x m) real :
A' is unitary. A H is unitary. A is unitary. A - l is unitary. A' is unitary for i = I , 2, . . . ¢::::::}
A is orthogonal
( 4 ) A ( m x m) unitary: ( a) jdet( A ) Iabs = ( b ) r k ( A ) = m.
A is unitary.
l.
( c ) A is norm al , that is, A H A = A A H . ( d ) A is eigenvalue of A => j .A ! ab = l . s
1 ) => I ! A x l ! 2 = ! ! x l l 2 · (f) B ( n x m ) => I ! B A I ! 2 = I ! B I I 2 ·
(e)
x
(m x
(g) B ( m x n ) => ! I A B I ! 2 = I ! B I I 2 ·
( 5 ) A; ( m x m ) , i = 1 , 2 , . . . : A; is u n itary for i = 1 , 2 , . . . and Em A; exists => Em A ; is unitary. s - oo
( 6 ) A , B ( m x m) :
A , B are unitary
1 - oo
=> A B is unitary.
( 7) A ( m x m) unitary, B ( n x n) unitary : (a) A ® B is unitary. (b) A
ffi
B is unitary.
( 8 ) A ( m x m) with eigenvalues A J , . . . , A m : A is normal, j .A; l ab s = 1 , i = 1 , . . . , m ¢::::::} A is unitary.
(9)
(a) The (m x m ) unit m atrix Im is unitary. ( b ) The ( mn x mn ) commutation matrix l\mn is unitary.
[ cossin BB
].
( c ) A permutation m atrix is unitary. ( d ) For () E IR ,
-sin B cos B
IS
. umtary.
(e) ( Householder transformatio n ) 2xx H . . s umtary. For x ( m x 1 ) , lm xHx 1 ( 10) A , B ( m x m) : ( a) A is similar to a unitary matrix => A - 1 is similar to A H .
( b ) A , B are unitarily equivalent values.
-¢::=>
A , B have the same singular
1 69
P ROPERTIES O F S PECI A L M ATRICES
( c ) A , B are unitarily equivalent
{::=:}
A H A and B H B are similar.
( 1 1 ) A ( m x m) unitary : There exists a real orthogonal ( m x m) m atrix Q and a real sym metric ( m x m ) matrix S such that A = Qexp(i5). Note: The results of this subsection follow either directly from the definitions or c an be found in Horn & J ohnson ( 1 985 ) . A number of results on unitary m atrices are also given in Lancaster & Tismenetsky ( 1 985).
10 Vector and Matrix Derivatives In this chapter all vectors and matrices are assumed to be real unless otherwise stated . Differentiable always means continuously differentiable of sufficiently high order so that all expressions are well-defined .
10.1
Notation
f( ( x)
(
Let be a differentiable real valued function of the real m x 1 ) vector X = X1 , . . . , X m )' .
fO X)( x)
of
8
of ax of OX
OX )
-
or
af(x) ax fOX( X ) {)
m
(m
x 1)
m
g;, ( :� . . . ' :�) or a���) ( a£;�) , . . . , a;;:) ) ( 1 m ) are vectors o f first order partial derivatives. f( x)/ ax is sometimes called the gradient vector of f ( x). and
=
=
'
x
{)
oQf X and
of ax
Xo
of' ax
xo
x=x o
of'l f' x
u
- afa(xxo )
x =x o
=
=
(m
aff'l(x'o ) - [af(f'l xo )] ' ( 1 x x =
u
u
x 1)
x m)
H A N DBOOK O F M ATRICES
1 7"2
are vectors of first order partial derivatives evaluated at the ( m x 1 ) vector xa .
[)2 f
{)x{)x '
=
[) 2 f( X ) {)x{)x'
=
[
[)2 f( X ) ax; aXj
[) 2 f
]
82 !
ax l ax )
.
[) 2 f OX m OX t
.
.
.
8x t 8Xm
.
.
-
(m x m)
[)2 J OXmOXm
is the Hessian matrix of second order partial derivativeo. of /( x ) and
.r o
=
82/( xo ) axox'
=
[
02/
ax 1· ox].
x=xo
]
(m x m)
is the Hessian matrix of second order partial derivatives of f(x) evaluated at :r = xa . Let f( X ) be a differentiable real valued function o f the real ( m x n ) matrix X = [ :r ,i l ·
af ax
af( X ) ax
(m x n)
at
of
is t he II I atrix of first order partial derivatives of /( X ) and a!
(m x n)
{)X
is the matri x of first order partial derivati ves of f( X ) evaluated at the ( m x n ) mat ri x X o . avec( X ) avec ( X ) '
avec( X )avec ( X )'
( mn x mn )
is the Hessian matrix of second order partial derivatives of f( X ) . Here it is i n 1 p ortant to note the order in which the partial derivatives are arranged . They have the same order as for the function f( vee( X ) ) . Accordingly, avec ( X ) a vec ( X ) I X = X o
is the Hessian mat rix evaluated at X = Xo .
avec ( X )avec ( X )'
1 73
V ECTOR A N D M ATRIX DERIVATI VES
Let y( x ) = [y1 ( x ) , . . . , Yn ( x )] ' be a real ( n x 1 ) vector of differentiable functions of the real ( m x 1 ) vector x ( x 1 , . . . , .r m ) that is, y is a function mapping a subset of m.m on a subset of IRn . =
and
,
OY 1
oy1 oy o.r '
'
oy( x ) ox'
( n x m)
( )
, oy (m x n) ox' OX OX are matrices of first order partial derivatives of y(x ) . oyfox' is sometimes called the Jacobian matrix of y and oy' /ox is sometimes called the gradient of y. oy'
oy ox '
r = .r o
oy ox '
oy( x ) '
-
oy(xo ) ox'
xo
..
.
is the J acobian m atrix evaluated at x = x0 and its transpose is the gradient of y evaluated at .r o . For m n , =
det
( 0���) )
= det
( ::, )
is the Jacobian or J acobian determinant of y( x ) . The Hessian matrix of y( x ) is ovec(oyfox ' ) ox ' m.mxn , For a matrix valued function of a single variable, A : S C IR x ...... A(x) (a ;j (x ) ] , the matrix of first order derivatives is �
=
dA(x) d .r
dA dx
-
d a mI d .r and the corresponding matrix of first order derivatives evaluated at .ro is d A (xo ) dx
dA dx
x = xo
H A N DB O O K O F M ATRICES
1 74
In the remainder of this chapter all functions are assumed to be differentiable.
Gradients and Hessian Matrices of Real Valued Functio ns wit h Vector Arguments
10.2
10.2. 1
Gradients 1R
( 1 ) x (m x 1 ), c E
oc
constant:
OX
= Om x l ·
( 2 ) ( Linearity ) x ( m x 1 ) , l ( x ) , g(x) real valued functions, c 1 , c2 E IR :
o [c. J( x ) + c2 g(x)] c ol( x ) = 1 ox ox
og( x) + c2 ox ·
( 3 ) ( P roduct rule) x ( m x 1 ) , 1 ( x ) , g ( x ) real valued functions:
a1 ( x ) g ( x)
= I( X )
OX
og ( x ) ax
+ g(X )
o1 ( x ) ox
( 4 ) x ( m x 1 ) , 1( x ) , g( x) , h ( x) real valued functions:
ol( x ) g( x)h ( x ) ox
=
o h ( x)
l(x )g( x)
OX
+ l( x)h ( x )
og ( x ) ( x) ol( x ) . x)h + g ( ax ox
( 5 ) ( Ratio rule) x ( m x 1 ) , l( x ), g ( x ) :/; 0 real valued functions:
[
]
o [l( x )f g( x) ] 1 x) . ol( x) _ o ( g = x ) I( x ox g( x) 2 g ( ) o x ox ( 6 ) (Chai n rule) x ( m x 1 ) , y( x ) ( n x 1 ) , I( Y) a real valued function :
ol ( y(x )) ax (7) x, a (m x 1) :
oy( x )' oi (Y) o x oy ·
Oa1 X
a. = x o
( 8 ) X ( m X 1 ) , a , y( X ) ( n X 1 ) (9) x (m x 1 ) , A (m x m) :
:
Oa1 y( X ) OX
=
o y( X)1 a. OX
ox ' Ax A + A )x . ( = ox I
1 75
V ECTOR A N D M ATRIX D E RI VATIV ES
( 10 ) x ( m x 1 ) , a ( n x 1 ) , A ( n x m ) , B ( n x n ) symmetric:
�
( a - Ax) (a - Ax)
= -2A ' B( a - Ax).
( 1 1 ) X ( m X 1 ) , y( X ) ( n X 1 ) , A ( n X n ) :
ay( x )' Ay(x) ay(x)' (A + A' ) y( X ) . ax ax x 1 ) , y(x) ( n x 1 ) , z(x) (p x 1 ) , A ( n x p) : ay( x )' A z(x) ay( x)' az( x ) ' , AZ( X ) + A y( X ) . = ax ax ax _
( 1 2) x ( m
Note: ( 1 ) - ( 6 ) are standard rules of calculus for functions with vector arguments. For the chain rule see, e.g., M agnus & Neudecker ( 1 988, Chapter 5 , Sec. 1 2 ) . ( 7 ) and ( 8 ) follow from ( 2 ) . ( 9 ) is obtained from stan dard rules for derivatives by writing x ' Ax i n summation notation and computing the derivative for each individual component of x . ( 1 0 ) and ( 1 1 ) follow from ( 9 ) and the chain rule ( 6 ) . ( 1 2) i s a consequence of the product and chain rules. 10.2.2 (1) ( 2) (3)
Hessian Matrices
a2a' x = Om x m · X, a ( m X 1 ) : axax ' 2 x ' Ax a X ( m X 1 ) , A ( m X m) : = A + A' . ax ax' x ( m x 1 ) , a ( n x 1 ) , A ( n x m ) , B ( n x n ) symmetric : a2 (a - Ax)' B(a - Ax) = 2 A' B .4 . uxux ' x ( m x 1 ) , y( x) ( n x 1 ) , A ( n x n ) : ay( x ) a2 y(x )' Ay( x ) ay(x)' ' (A + A ) axax ' ax ax' a + [y (x ) ' ( A + A' ) 0 Im ] a ' ve e n
(4)
n
( 5 ) X ( m X 1 ) , y( X ) ( n X 1 ) , z ( X ) (p X 1 ) , A ( n X p)
a2 y(x) ' Az(x) ax ax '
� [ ( y�; ) ] . :
ay(x) ' az( x ) az( x )' ' ay( x ) A A + ax ax' ax' ax a ay( X ) + [z(x) A 0 Im l vee ax ax' a )' + [y(x )' A 0 Im ] a ' vee 1
1
)'
[ ( 1)] � [ ( �; ) ] .
H A N DBOOK O F M ATRICES
1 76
Note: A l l resu lts are simple consequences of the resu lts in the previous subsection and the rules for vector valued functions with vector arguments given in the fol lowing sections.
Derivatives of Real Valued Functions with Mat rix Arguments
10.3
1 0.3. 1 (1)
\
.
General and Miscellaneous Rules
ac = Om x n · aX
( m x n ) , c E IR constant :
( 2 ) ( Linearity ) X ( m x n ) , f( X ) , g ( X ) real val ued fu nctions, c 1 , cz E IR : oh f( X ) + c2 g ( X )] {)X
.\ ) = CJ af( aX
o g(X ) + c2 aX .
( 3 ) ( Product rule) X (m
x
n ) , f( X ) , g ( X ) real valued functions: af( X )g ( X l ax
1
=
( .Y )
af( X ) ag(X ) + ( Y) Y ax ax ·
( 4 ) .\ ( m x n ) , f( X ) , g ( .\ ) , h ( X ) real valued functions: 8/( X ) g ( X ) h ( X ) ax
( 5 ) ( Ratio rule) X ( m x n ) , f( X ) , g ( X ) -:/; 0 real valued functions: a [ f( X ) / g ( X ) ] ax
=
1
g( X )2
[
9
(X)
af( X ) ax
_
ag( X ) f( X ) ax
( 6 ) ( C h ai n rule) .\ ( m x 11 ) , y = /( X ) , g ( y ) real valued functions: og ( f ( X ) ) ax
_
dg( y ) af( X ) ax · dy
( 7 ) .\ ( m x n ) , f( X ) a real valued function : vee
(
af( X ) aX
)
=
af( X ) avec ( X ) .
]
.
V ECTOR A N D M ATRIX DERIVATI V ES
1 77
(8) X (m x n), a (m x 1 ) , b (n x 1 ) : aa' X b
ax
( 9 ) X ( m x m) , a , b ( m x 1 ) : 0�:�;
( 10 )
X
b
= ab' .
= ) : ( Xi ) ' ab ' ( x i - 1 -i ) ' ,
i
) =0
= 1 , 2, . . .
( m x m ) P.onsingular, a , b ( m x 1 ) :
aa
X 1b
�
:
.\
= - ( X - 1 ) ' ab' ( X - 1 ) ' .
( 1 1 ) X (m x n). a, b (n x 1) :
aa' X ' X b
(12) X (m x n), a, b ( m x 1 ) :
{)X
aa' XX ' b {)X
=
V ( ba l + a b' ) .
....�
b
=( a +a I
b' ) .
.\ .
( 1 3 ) X ( m x m ) symmetric with simple eigenvalue _\ and corresponding eigenvec tor v satisfying v ' v = 1 : {)
_\ -= aX
I
vv .
Note: ( 1 ) - (7) are standard results from calculus. ( 9 ) and ( 1 0 ) are given in M agnus & N eudecker ( 1 988, Chapter 9, Sec . 1 3 ) and ( 8 ) is a special c asr of ( 9 ) . ( 1 1 ) and ( 1 2) follow from results given in M agnus & Neudecker ( 1 98 8 . Chapter 9, Sec. 9) by noting t hat a' Ab = tr ( Aba ) . Result ( 1 3 ) i s from :vl agnus & Neudecker ( 1988, C hapter 9 , Sec . 1 1 ) . '
10.3.2
Derivatives of the Trace
First Order Derivatives atr( X) a tr( X ' ) ( 1 ) X ( m x m) : = = lm . {) X aX (2) X ( m x n ), A (n x m) :
(3) X (m x n). A (m x n ) :
atr(AX) atr( X A ) = ' . = A aX aX atr ( X ' A ) atr( AX' ) . = =A aX aX
H A N DBOOK O F M AT RICES
1 78
(4)
X ( m x n), A (p x m ) , B ( n x p) :
(5)
X (m x n) , A ( p x n), B (m x p) : otr(X 2 ) = 2X , X (m x m) : ox otr(X; ) = ; i - t (X' ) ox
(6)
(7) X,
(11) ( 12) (13)
·
,
i = l , 2, . . .
A , B (m x m) :
ot (X'X v otr(XX' ) ) r 2 = = ( x n) : 0X 0X ot r ( X ) = ( A + A' ) X . X ( m n), A ( m X m ) : otr( X) = 2 AX. X (m n), A ( m X m) symmetric : otr( x' ) = X(A + A' X (m n), A (n X n ) : ). otr( x' ) = 2 XA. X ( m n). A ( n X n) symmet r ic : X ot ) = X ' A' + A' X' . r X , A (m x m) :
(8) X
( 10 )
·
·
•
(9)
otr( AX B) = A' B' ox otr(AX' B ) = BA ox
,'\. .
m
:�A
X
:�A
X
::
X
X
(:/
::
:
x' A' otr(A ) = 2 A' AX. (14) '( ( 1 5 ) X ( m x n ) , A (p x n ) : otr(AX ' X A' ) 2 XA'A. ox Y. X ' B ) otr(A ) ( X ) A B 1 6 : m x n , (p x m) , ( ( BA + A' B' ) -'-" . m x p) ( �u X otr(A X B ) = X(BA + A ( 1 7 ) X (m x n), A (p x n), B (n x p) : ' B' ). o otr( ' B ) = B'XA ' + BXA. ( 18 ) X (m x n), A (n x n), B (m x m) : otr( X, AX B ) -- B' X ' A' + A' X' B ( 1 9 ) X (m x n), A , B (n x ) . uX X (m n ) , A (p X m ) : X
_
_
_
_
� ��X
m
:
I
V ECTOR A N D M ATRIX DERIVATI V ES
( 20) X ( m
x
n) ,
A (p x
m
1 79
) , B ( n x n ) , C ( m x p) :
atr(AX BX ' C ) = ' ' ' A C X B + C AX B . ax (21) X (m
x
n),
A (P x at r(
m
), B (n x
m
) , C ( n x p) :
A:X: C ) = A' C' X' B'
( 22) X ( m
x m
) nonsingular :
( 23 ) X ( m
x m
) nonsingular, A , B ( m x
( 24 ) X ( m
X
(25) X ( m
x
m
)
+
B ' X ' A'C ' .
:
, avec F(X) . atr[F(X)] = vee( !p ) avec(X)' avec(X)' n ) , F ( X ) ( p x q) , G( X ) ( r x s ) , A (q x r ) , B ( s x p) :
n),
F(X) (P
X
P) :
atr[F(X)AG(X ) B] _ avec( X)'
[
avec F(X) vee(!P ) ' [B'G(X) ' A ' Q? IP ] avec( X)'
+
[B' Q? F(X)A]
]
avec G( X ) . avec( X) '
Hessian Matrices
)
(1) X (m
x m
(2) X (m
x
n) ,
A (P x
(3) X (m
X
n) ,
A (P
(4) X (m
X m
)
:
:
m
) , B ( n x p) :
X n),
B (m
X p) :
a2 tr ( AXB) ----'------'---avec(X)avec (X)' = Omn x mn a2 tr(AX' B) · avec(X )avec (X)' = Om n x mn ·
a2tr(X A X' ) = ( A + A ) 0 Im . avec ( X ) avec ( X )1 I
n) :
(P
) . A (p
x
m)
a2 tr(X A X ' ) = 2 ( A 0 1, ) . avec( X )avec ( X ) '
a2tr( AX X ' A ' ) -2 ( In 0 A' A ) . avec ( X )Ovec ( X ) '
:
a2 t r ( A X ' X A' ) = 2 ( A ' A 0 Im ) · avec ( X ) avec (X )' m ) , B ( m x p) :
X n) : x
a2 tr( A X X ' B ) avec ( X )avec ( X ) ' ( 14 ) X ( m
°
a2 tr(X A X ) I • l\ ) l A ( mm . m + I A m 0 0 = a ve c( X ) a vec ( X ) 1
x
X
x
:
( 11) X (m
(13)
x m
)
n ), A (P
x
n), B (n
x
-
-
In
°
( B A + AI BI ) .
p) :
a2t r( AX 1 X B) I I B ) 0 Im . ( BA A + av ec ( X )Ovec ( X ) ' = ( 15)
X
(m
x
n ), A (n
n), B (m
x
x m
)
:
a2 t r ( X A X' B ) _ . - A !I "' !I uvec ( ·"- )uvec ( X )' ( 16 ) X ( m
x
n) , A , B (n
x m
)
0
B' + A'
0
R.
:
a2tr(X AX B) _ ' -'-:.._ - ( A 0 B1 + B 0 A ) l\.mn avec( X )Ovec ( X ) '
-
( 1 7) X ( m
x
n ). A (p
-
x m
), B (n
x
n ), C (m
x
·
p) :
a 2 t r ( A X BX' C ) _ 'C'' + B 1 0 C' A . __,_ .:.._ - B 0 A avec ( X )avec ( X )'
_
( 18) X (m
x
n ), A (p
_ _ _
) B (n
x m ,
x m
), C (n
x
p) :
a2 tr( AX BXC) = ( B 0 A'C 1 + C A 0 B' ) l\'m n . X c av ec ( X )a ve ( ) '
181
V ECTO R A N D M AT RI X DERIVATI V ES
( 19) X (m
x m
) nonsingular :
iP tr(X - 1 ) = Kmm (X ' - 2 Q? x - t + x '- t Q? x - 2 ) . ovec ( X ) 8vec ( X ) ' (20) X ( m
x m
) nonsingular, A , B ( m x
m
)
:
o2 tr( A X - 1 B ) ovec( X )ovec ( X )' . = l\ mm ( X ' - 1 A' B' X' - 1 Q? X - 1 + X' - 1 @ X - 1 B A X - 1 ) . Note: The results of this subsection are partly given i n M agnus & Neudecker ( 1 988, C h apter 9 , Sec. 9 and 1 3 ) and partly fol low from their results by straightforward application of the rules for the trace and matrix derivatives, notably the chain rule and the product rules of Section 1 0 . 5 . 10.3.3
D erivatives of Determinants ad ) ( 1 ) X ( m X m) nonsingular :
(2) X (m
x
n), A (p
x
m
), B (n
�iX
=
det ( X ) ( X ' ) - 1
=
( X a dj ) ' .
x p) , det( A X B ) =f. 0 :
��X B ) = det ( A X B ) A' ( B' X ' A' ) - 1 B' . odei: X ' ) ) rk( X = = 2det ( X X' ) ( XX' ) - 1 X . ode�;' X ) 2det ( X ' X ) X ( X ' rk( X ) = a de t
( 3 ) X (m X n) , (4) X ( m
(5) X (m
x
n) ,
x m
m :
=
n :
X )- 1 .
) nonsingu l ar:
2) �1\ 2( det X ) 2 ( X ' ) - 1 . ( a) ode�,� - t ) - ( de t X ) - 1 ( X ' ) - 1 . ( b) ') od ��\ = i ( det X ); ( X ' ) - 1 , i = ± l , ± 2 , . . . (c) od
=
=
( 6 ) X, A , B (m x
m
) , det ( A X i B ) =f. 0 :
i- t . 8det ( AX ; B ) . . = det ( A X ' B ) 2:) (x' - 1 - 1 ) B ( A X ' B ) - 1 A X J ] ' . ax =l
J
H A ND BOOK OF M ATRICES
182
(7)
X
(m
x
n ) , A (p x
m
B ( n x n) , C
),
(m
x p), det ( A X B X ' C )
:j; 0 :
odet ( A X B X ' C ) ox det ( A X B X ' C ) {C[AX BX'G] - 1 A X B + A ' [C' X B' X ' A'] - 1 C' X B' } .
(8) X (m
x
(9) X (m
x
n ) , A (p
x
n), B
(m x
m
),
C (n
x
)
C (n
x
p), det ( AX' BXC) :j; 0 :
odet ( A X' B X C ) ox det(AX' BXC) { BXC[A X ' B X G] - 1 A + B' X A' [C' X' B' X A't 1 C' } .
n ) , A (p x
m
B (n x
),
m ,
p) , det ( A X B X C) :j; 0 :
odet ( A X BXC) _ ox det( A X B X C ) { C[AX BXC]- 1 AX B + B X C[AX BXG] - 1 A } ' .
( 10 )
X
(m
(11) X (m ( 12 ) X ( m
)
x m , x
x
n), A
(m x
o ln d
)
e��
m
=
(X' ) _ 1
·
ox positive definite, det ( X ' A X ) > 0 : 'AX)
=
2 A X ( X ' AX ) - 1 •
n ) , A ( n x n ) positive definite, det ( X A X ' ) > 0 : o l n det ( X A X ' )
=
ox
n ) , A ( n x p) , B det ( A ' X ' B X A ) > 0 :
( 13) X (m
o ln det( X )
det ( X ) > 0 :
x
o l n det(
:�X
( m x m)
' BXA)
=
2(XAX') - I XA
·
positive defi nite,
2 B X A ( A' X' BXA) - I A' .
n ) , A (p x m ) , B ( n x n ) positive definite, det ( A X BX' A') > 0 :
( 14 ) X ( m
x
o l n det(
( 15 ) X ( m
)
x m ,
::
BX'A')
= 2A'( A X B X ' A' ) - 1 A X B .
det ( X ) > 0 : o2 l n det ( X )
ovec ( X )ovec ( X ) '
-
-
Kmm
( ( X ' ) - I 0 x- I ] .
Note: ( 1 ) - (5) and ( 7) - ( 9 ) are either given in or follow straightforwardly from Magnus & N eudecker ( 1 988, Chapter 9, Sec. 10). Rules ( 10) - ( 15 ) follow from the chain rule and t he derivative of the logarithm . I n ( 1 5 ) , the derivat i ve of vec ( X - 1 ) from Sec. 1 0 . 6 is used i n addition . ( 6 ) results from ( 2 ) , a chain rule and a prod uct rule.
183
V ECTOR A N D M ATRIX D ERIVATI V ES 1 0 .4
10.4.1
Jacobian Matrices of Linear Functions
Linear Functions with General Matrix Arguments
( 1 ) X (m x n) :
avec( X) avec(X) '
=
lmn ,
avec(X') _ 1\. - mn ' avec( X)'
( 2 ) X (m x
( 3) X (m
x
(4) X (m x ( 5 ) X (m x
avec( X ) = Knm · avec( X ' ) ' avec( eX ) c = n ) , E IR : avec( X) ' clmn · avec(AX B ) __ "' B' A. n ) , A ( p x m ) , B ( n x q) : avec( X ) ' avec( AX' B ) n ) , A (p x n ) , B ( m x q) : = ( B 0 A)l\ mn . avec(X) ' n ) , A ( p x m ) , B ( n x q ) , C (p x q) : v
I
avec( AX B + C) = B' 0 A. avec(X)' ( 6) X ( m
x n) ,
A, C (p
x m) ,
B, D (n
x q) :
avec(AX B ± CX D) = B' 0 A ± D' 0 C · avec( X ) ' ( 7) X ( m x n ) , A, C (p x n ) , B , D ( m x q) :
avec( AX' B ± CX' D) = ( B 0 A ± D 0 C) l\ mn · avec(X )' I
I
•
( 8 ) X ( m x n ) , A (p x m ) , B ( n x q ) , C (p x n ) , D ( m x q) :
avec( AX B + CX' D) avec ( X ) '
avec (CX' D + AX B) avec ( X ) ' - B ' 0 A + ( D' 0 C) I\mn ·
_
( 9 ) X ( m x n ) , A ; (p x m ) , B; ( n x q),
i=
1,
.
.
.,r
:
.
H A N DBOOK O F M ATRICES
1 84
( 1 0 ) X ( m x n) , Ai (p x n ) , Bi ( m x q ) ,
8vee ( L �- 1 A ; X 1 B; ) = 8vee ( X ) 1
i = l, . .
.
,r:
(� B'1 0 A'- ) \ mn · y
�
•=1
Kronecker and Hadamard Products (11)
X
( m x n ) , A (P x q) :
8vec (A) 0 vee ( X ) - vee( A ) 0 Imn · Bvee( X )1 ( b ) 8vec( X ) 0 vee( A) = Imn 0 vee( A ) . 1 ovec( X ) 8vee(A 0 X ) . = 0 { l (c) I\ q np 0 Im ) [vec(A) 0 Imn ] . 8vee( X ) 1 8vec(X 0 A ) . (d) = ( I 0 I\ qm 0 Ip )(Imn 0 vec(A)]. n 8vee ( X ) 1 X ( m x n ) . A (p x q ) , B ( r x m ) , C ( n x s ) (a)
( 12)
_
:
.
avec( A 0 BXC) = ( Iq 0 I\ •p 0 Ir ) [vee( A ) 0 C 0 B] , Clvec ( X )1 1
avec ( BXC 0 A ) = ( 1, 0 I\ qr 0 Ip )[C 0 B 0 vec(A)] . ovee( X )1 x n), A (P x q ) , B (r x s ) : I
"
( 13 ) X (m
8vee( A 0 X 0 B ) 8vec( X ) ' = ( Iq 0 I\n , ,p (?) Imr ) [vee ( A ) 0 ( In @ 1\,m 0 Ir )(Imn
( 1 4 ) X ( m x n) , A (p x q ) , B ( 1· x s ) , C (k x m) ,
D (n
x l)
(2)
vee( B ) ) ] .
:
8vec( A 0 CX D 0 B ) 8vee( X )' = ( Iq 0 I\, l,p 0 h r ) [ vec(A) 0 ( 11 0 K,k 0 Ir ) ( D1 0 C 0 ver( B))]. ( 15 ) X ( m x n ) , A (p x q ) , B ( mp x nq ) :
.
8vee [(A 0 X ) 8 B] = d 1. ag( vee B)( lq 0 I\ np 0 Im )[vee(A) 0 Imn J , 8 vee ( X ) 1 8vec [( X ·S> A" ) J
V ECTO R A N D M ATRIX DERIVATIV ES
1 85
( 1 6 } X, A ( m x n ) : avec( X 0 A ) avec( A 0 X ) = = diag(vec A ) . avec( X ) ' avec( X ) ' ( 17} X ( m x n ) , A ( P x q ) , B ( p x
) C (n x q) :
m ,
avec( A 0 BXC) _ _ avec( B X C 0 A) _ ' - d t ag( vee A ) ( C' 0 B ) . -----::-avec( X ) ' avec ( X ) ' �
( 1 8 ) X ( m x n ) , A ( p x q ) , B ( p x n ) , C ( m x q) : avec ( B X ' C 0 A ) ' avec( A 0 B X ' C) = = d t ag( vee A )(C' 0 B ) J(mn . 8vec( X ) ' 8vec( X )' ( 19 } X, A , B ( m x n ) : avec( A 0 X 0 B ) = diag[vec( A 0 B ) ) . avec ( X ) ' ( 2 0 } X ( m x n) , A , B ( n x
m
)
:
avec( A 0 X ' 0 B ) . . = dt ag[vec ( A 0 B )] A mn · 0 vee ( X ) ' ( 2 1 } X , A ( m x n) , B ( P x q) : avec [ ( A 0 X ) 0 B) . . = ( In 0 1\ q m 0 Jp ) [d tag(vec A ) 0 vee( B ) ) , avec( X )' avec[B 0 ( A 0 X )) . . = ( /q 0 l\ np 0 Im ) [vec ( B ) 0 d t ag(vec A ) ) . avec ( X )' Note: ( 1 ) - ( 10) fol low from basic properties of the derivative and t he rule vee( AX B) = ( B' 0 A ) vec ( X ) for the vee operator (see Section 7.2). ( 1 1 ) ( 2 1 ) m ay be derived using vec( A 0 B ) = ( I 0 1\ 0 / ) [vee( A ) 0 vee( B )) and vee ( A 8 B) = diag(vec A ) vec ( B) and other standard rules for Kronecker and H adamard p roducts (see Chapter 2 and M agnus & Neudecker ( 1 988, Chapter 9, Sec. 14 ) ) . 10.4.2
Linear Functions with Symmetric Matrix Arguments
Reminder: Dm denotes a duplication m atrix and D1;, its M oore -Penrose inverse (see Section 9 . 5 ) . Lm is an elimination matrix ( see Section 9.6) .
H A N DBOOK O F M ATRICES
1 86
( 1 ) X ( m x m)
(2) X (m x (3} X (m x
sym metric :
8vech( X ) avech( X ) ' = Im ( m + l l /2 '
avec(X ) = Dm . avech( X ) ' avec(cX ) m ) symmetric, c E IR : = cDm . avech( X )' m ) symmetric, A ( m x n ) : avech( A'X A) L = n ( A' ° A' ) D"' avech( X ) '
(4) X ( m x m)
symmetric, A ( n
x m),
B (m
avec ( AX B ) = (B avech( X )'
I
0
D + ( A' ° A' ) D"' .
=
n
x p) :
A )D, .
( 5 ) X ( m x m ) symmetric , A, B ( m x m ) :
avech (AXA' ± BX B' ) = D+ (A 0 A ± B 0 B ) Dm . avech( X ) ' "'
(6)
X (m x m)
symmetric , A, B ( m
x m) :
avech(AX B' + BX A' ) = D + ( B 0 A + A 0 B ) Dm · avech( X) ' "'
( 7) X ( m x m )
symmetric, A, C ( n
B, D ( m
x m),
x p) :
avec ( AX B ± CX D) = ( B ' 0 A ± D' 0 C ) Dm · avech( X ) ' (8}
X (m
x m)
symmetric, A; ( n
x m) ,
B; ( m
x p) ,
i = 1, . . .,r :
avec ( L:�= l A; X B; ) - � ( B ; 0 A. ) Dm · � avech( X ) ' I
( 9 } X ( m x m)
symmetric, A; ( m
x n) , i = 1 ,
avech( L ;- 1 A ; X A i ) � = � L n ( A'I 0Vech ( X ) ' •=I
. . . , r :
� 0 A' ) Dm = � D + ( A ' 0 A ' )D m . I
•= 1
n
I
I
Note: These results fol low from basic properties of the derivative and the
vee and vech operators. Notably, the rule vec( A X B ) ( B' 0 A)vec( X ) is useful (see Chapter 7 ) . For proofs see also Magnus ( 1 988, Capter 8, Sec . 8 . 2 ) . =
187
V ECTO R A N D M AT RI X DERIVATIV ES
L inear Functions with Triangular Matrix Arguments
10.4.3
Reminder: Lm denotes an elimination m atrix (see Section 9 . 6 ) .
( 1 ) X ( m x m ) lower triangular:
avech( X ) = J/2 ' 8vech(X)1 lm ( m+ I
avec(X) 1 = Lm 8vech(X)1 avec( eX) cL:., . m ) lower triangular, c E IR. : 8vech( X )1 m ) lower triangular, A ( n x m ) , B ( m x p) •
(2)
X (m
x
(3)
X (m
x
=
:
avec( AX B) 8vech (X)1 (4} X ( m
x m)
1 ( B 1 0 A)L m
lower triangular, A, e ( n avec ( AX B ± ex D) 8vech(X)1
( 5 ) X ( m x m)
=
=
lower triangular, Ai ( n
•
B, D ( m
x m) ,
( B 1 0 A ± D1 0 e) L1m · x m) ,
Bi ( m
x p) , i
avech( AXB) avech( X) I m ) lower triangular:
(6) X, A, B ( m x m ) lower triangular: ( 7 ) X, A, B , e, D ( m x
avech(AX B ± ex D) 8vech (X}1 (8)
X, A i , Bi ( m
x m)
=
=
=
=
1,. . .,r
:
I I L m ( B 0 A) L m .
Lm ( B1 0 A ± D 1 0 e)L1m ·
lower triangular, i
avech (L�- 1 A i X B i ) 8vech( X)1
x p) :
Lm
=
1,
.
.
.,r:
(�I B'� 0 A'- ) Lm1 · � •=
Note: The results of this subsection fol low from basic properties of
derivatives and the vee and vech operators. In particular, the rules vec ( A) L:., vech( A) for lower triangular matrices A (see Section 9 . 1 4 ) and vee( AX B ) = ( B1 0 A)vec ( X ) ( see Section 7.2) are useful. For proofs see also Magnus ( 1 988, Chapter 8, Sec. 8.3 ) . =
H A N DBOOK O F M ATRICES
1 88
10.4.4
Linear Functions of Vector and Matrix Valued Functions with Vector Arguments
ac = On x m · ax' ( 2 ) X ( m X 1 ) , y( X ) , Z ( X ) ( n X 1 ) , C I , C2 E ffi : ( 1 ) x (m x 1),
c
(n x 1)
constant :
az( x ) ay( x ) a[cl y (x) ± c2 z(x)] C C ± 2 £l = 1 £l £l 1 ux' ux' ux
( 3 ) x ( m x 1 ) , yi ( x ) ( n
x 1 ) , ci E
ffi,
•
i = 1, . . ., r :
a [L: �- 1 c;yi (x)] � ay; ( x ) c; . = ax' x' a L.I ..., i= ( 4 ) x ( m x 1 ) . y( x ) ( n x 1 ) , z ( x ) ( p x 1 ) , A ( q x n ) , B ( q x p ) :
a [ A y(x) ± B z ( x)] az ( x ) ay( x ) ± B !'l 1 . =A ax ' ux ax' avec( c Y ) avec ( Y ) =c ax' ax ' ( 6 ) x ( m x 1 ) , Y ( x) ( n x p ) . A ( q x n ) , B (p x r ) : ( 5 ) x ( m x 1 ) , Y( x ) ( n x p ) , c E IR :
avec( A Y B) ax'
=
I
(B 0 A)
avec ( Y ) . ax'
( 7 ) x ( m x 1 ) , Y; ( x ) ( n ; x pi ) , A ; ( q x n; ) , B; (p; x r ) , i
=
avec ( }i ) avec (L: := I A ; l( B; ) - � I L... ., A ) ( Bl 0 !) I £l I l!X X u I= I I
1,
. . . , s :
•
.
( 8 ) x ( m x 1 ) . Y ( x ) ( n x p ) , A ( q x n ) , B (p x r ) ,
C (q
avec ( A Y B + C) _ avec ( Y ) ( - B ® .4 ) £l 1 £l 1 ux ux I
( 9 ) x ( m x 1 ) , Y ( x ) ( n x p ) , A ( r x s) :
x r)
:
.
avec( Y ) ) ( vec ( A ) ( !, , ax' ( avec ( Y ) ) avec ( Y ® A ) . c(A) . = ( lp ® l\ • n 0 lr ) 0 vc ' avec( A 0 Y ) ax' ax
=
0
. l\p r 0 In )
0
ax'
VECTOR A N D M ATRIX D E RI VATI V E S ( 10}
x (m
x 1),
Y(x ) ( n
x p) ,
A (r
x
1 89
s ,
) B ( k x /) :
avec( A ® Y ® B) ox' = ( I, ® f\p l,r ® Ink )
(11)
x (m
x
[vec( A )
0 ( lp 0 K 1n
x 1),
Y(x) (n
x p) ,
A (r
x
( ov��('Y ) 0 vee( B) )] .
0 h) s ,
) B ( k x n ) , C ( p x /) :
( [(c' B ) ovecox('Y ) ] ® vee ( A) ) ovec ( Y ) ' ( c ( [ ) ) . ] 0 Ik ) vee (A 0 0 B) ox'
ovec ( BYC ® A) _ y ® ( I - 1 \ ,k ® I ) ox '
0
r
ovec( A ® BYC) _ ( I 0 1\. l r ox' x ( m x 1 ) , Y(x ) ( n x p ) , A ( n -
( 12}
•
'
x p) :
ovec(Y 8 A ) _ ovec(A 8 Y ) _ . ovec( Y ) ( A ag vee ;::, ) 0 - dt ;::, ' ux' ux x' Note: The results of this subsection fol low from the linearity of the functions considered, the rules for the vee operator and basic matrix operations. .
10.5
P rod uct R u les
10.5. 1 (1)
Matrix Products
X (m
x m) :
avec( X 2 ) (a) = X 0 Im + Im 0 X . avec ( X )' . i- 1 ve X ) e( ( b) = " (X' ) i - t - j ® xi , L... vec ( X ; =0
�
r/
•
�
1
i = 1 , 2, . . .
8vec ( X ' 2 ) (c) ( X ® Im + Im 0 X ) I\m m · = X c( ) ve o ' X (m x n) : avec ( X ' X ) = ( In > + A nn ) ( ln ® X ) . (a) ;::, uvec( X ) avec ( X X') = Urn > + A.m m )( X ® Im ) · (b) o v ec ( X ) ' •1
( 2)
1
•
•
I
H A N DBOOK O F M AT RICES
1 90
(3) X (m x n), A (n x m) :
avec( X AX) A 0 Im = X c( )' fJ ve X I
(4) X ( m
x n) ,
A (m
I
x m) :
avec( X' AX) = ( X A 0 In ) A m n ) X c( ' a ve ,
( 5 ) X ( m x n),
A (m
,
+ In 0 X A .
x m)
1
•
+ ( In
0
,
X A).
symmetric:
avec( X' AX) a ve c( X )' =
.
0 )\
Un > + l\ n n ) U
,
n
,
A).
(6) X (m x n) , A ( n x n) :
avec( X AX' ) = ( X A 0 Im ) + ( Im 0 X A) l\ mn . f)vee ( X ) ' .
I
( 7 ) X ( m x n ) , A ( n x n ) symmetric:
avec( X AX') avec( X ) '
,
,
= ( 1m 2 + l\ mm )( X A
0
Im ) ·
( 8 ) X ( m x n ) , A ( P x m ) , B ( n x m ) , C ( n x q) :
avec(AX BXC) = C' X' B' 0 A + C' 0 AXB . ) avec ( X ' (9) X (m
( 10 )
x n),
A (p
x n) ,
avec( AX ' BXC) fJvec ( X )'
B (m =
x m ) , C ( n x q) :
( C' X ' B' 0 A ) F \ mn + (C' 0 AX' B) .
X ( m x n ) , A ( p x n ) , B ( m x m ) symmetric:
avec( AX' BXA' ) = ( lp > avec( X ) '
( 1 1 ) X ( m x n ) , A (p x m ) , B ( n x n ) ,
+
I
I\pp ) (A 0 AX B). '
C ( m x q) :
avec ( A X B X ' C ) = ( C ' X B ' 0 A ) + ( C ' 0 A X B ) Km n . avec( X ) '
( 12) X ( m x n ) , A ( p x m ) , B ( n x n ) symmetric:
avec( AX BX' A ' ) = Ur > avec( X ) '
, + I'-rr H A X B 0 A ) .
191
V ECTO R A N D M AT RIX DERIVATIVES
(13)
X ( m x m ) , A ( n x m ) , B ( m x p) : .
i- 1
avec ( AX' B ) = L B '(X ' ) i- 1 - j 0 AX i ' avec(X)' = J O
(14)
.
X, A ( m x m ) : avec( A + X ) 2 avec(X)' avec( A + X ) ; avec( X)'
( 16)
= 1, 2, . . .
X ( m x n ) , A (p x m ) , B ( n x p) : avec (A X B )' L ' ' A ') i - 1 -i ' 0 (A )i A (B X XB , i = B avec(X)' j O = =
( 15)
i
i- 1
1 , 2, . . .
(A ' + X ' ) 0 lm + lm 0 (A + X ), i- 1
L(A ' + X' ) i - 1 -j 0 ( A + ; qi '
j =O
i = 1, 2. . . .
X, A ( m x n ) : avec[( A + X ) '( A + X ) ] = ( I F\ nn )[ /n 0 ( A' + -\ ' ) ] , + n (X avec ) ' avec [(A + X ) ( A + X ) ' ] ( I m2 + 1\.mm )[( 4 + "' ) "' Im ] . avec (X.) ' X ( m x n ) , A (P x p) , B (p x m ) , C ( n x p) avec( A + BX C )2 = (A ' + C ' X ' B' ) C' 0 B + C' 0 (A + BXC)B, avec(X)' avec (A + BX C ) ; ' + C' X ' B ' ) i - 1 -i C' 0 ( A + B X C )J B , (A = avec( X )' j =O 2
_
•
(17)
-�
v
:
�
i = 1, 2, . . .
A ( p x q) , B ( p x m ) , C ( n x q) : avec[(A + B XC) ' (A + BXC) ] -_ Iq, + 1\.. qq )[C. ' (A ' + c�' '\'' B' )B 0 ]. ( avec(X)' avec [(A + B X C )( A + B X C )' ] = /p > 1\pp )[( A + B X C )C' 8 B ] . ( + avec( X )' Note: ( 1 ) - ( 2 ) are gi ve n in Magnus & Neudecker ( 1 988, Chapter 9, Sec .
( 18) X (m x n),
The remaining results follow via the product and chain rules of different ial calculus. 13).
H A N DBOOK OF M ATRICES
1 92
Kronecker and Hadamard Products
10.5.2
( 1 } X (m x n) :
avec ( X ) 0 vec( X ) £l = lm n 0 vec( X ) + vec ( X ) 0 lmn · 1 uvec ( X )
( a)
avec( X 0 X ) £l (b) \" ) = ( ln 0 l\.nm 0 lm ) [ lmn 0 vec(,\. ) + vec ( ,\. ) 0 lmn ] . uvec ( , 1 avec( X 0 X') (c) avec ( X ) I = ( In \5 1\mm 0 ln ) [ lmn 0 vec ( X ' ) + vee( X ) 0 Kmn ] · avec(X' 0 X ) ( d ) avec ( X )' = Um 0 1\nn 0 lm ) [ l\.mn 0 vec ( X ) + vec( X ' ) 0 lmn l · avec( X' 0 X ' ) (e) avec ( X )' = Urn 0 1\mn 0 In ) [ 1\mn 0 vec ( X ' ) + vec ( X ' ) 0 Km n ] ·
(2}
X (m x n ) , A ( P x m ) , B ( n x q ) , C ( r x
avec ( A X B 0 C X D) avec ( X )' = ( lq 0 r.;, P c� lr ) [B'
(3)
X
0
m
) , D (n x s) :
A •:V vec ( CX D) + vec ( A X B ) 0 D'
( m X n ) , A (p X m ) , B ( n X q ) , avec( A X B 0 CX' D) avec ( X ) ' = { lq 0 1\", p 0 lr )
r (r
X n), D ( m X s)
0
C] .
:
X [B' 0 A S vec ( CX' D) + vec ( A X B ) 0 ( D' 0 C ) l\m n ] .
( 4 ) X ( m x n ) , A ( P x n ) , B (m x q ) , C ( r x m ) , D ( n x s ) :
avec ( A X ' B 0 CX D) avec ( X )' = ( fq (� f\, p 0 fr ) x [( B'
(5) X ( m
0
A ) l\mn
x n ), A (p x n),
avec( AX' B 0 C X ' D) avec ( X )'
= ( lq
•2 1\",P (:)
l )[( B' r
0
vec ( CX D) + vec ( A X ' B ) 0 D'
B (m x q),
0
A ) l\mn
0
C ( r x n) , D ( m x s) :
ve c ( CX' D)+
8
C] .
193
V ECTO R A N D M ATRIX DERIVATIV ES
+ vec(AX ' B) 0 (D' 0 C ) Kmn ] ·
X ( m x n) : 8 X) 2 " ( ag vec .'\. d ). ( a) avec(X 1 avec( X ) X 8 X 8 X ) 3 d " ag [vec (X 8. X )] . ( b ) avec(avec(X)' I ' X 8 X' ) avec( . 2 d (c) uvec( X) ' 1 ag(vec X )h mn · ( 7 ) X (m x m) : avec( X 8 X') = avec(X 1 8 X) d . ag(vec ):, )1\. n + d . ag(vec ,\. 1 ) . m 1 1 uvec(X)1 uvec(X ) 1 ( 8 ) X ( m n ) , A, C ( P x m ) , B, D ( n x q) : avec( AX B C::1 CX D) avec( X )' diag[vec(AX B)](D' 0 C) + diag[vec(CX D)](B' 0 A). (9) X ( m x n ) , A (p x n ) , B ( m x q) , C (p x m ) , D ( n x q ) : avec( C X D 0 AX ' B) avec(AX' B 0 CX D) avec( X)' avec( X)' diag[vec(AX' B)](D1 0 C) + diag[vec(CX D)](B' 0 A ) h. m . ( 10) X ( m x n ) , A , C (p x n ) , B , D ( m x q ) : avec(AX1 B (.:_) CX ' D) ' 0 C) {diag[vec(AX' B)](D avec( X )1 + diag[vec( CX ' D) ] ( B' 0 A)}/\·,, . Note: The results of this subsection follow from the b asic product rule for differentiation and the rules for Kronecker and Hadamard products . See also Magnus & Neudecker ( 1 988, Chapter 9, Sec. 1 4 ) . (6)
I
v
=
=
!.l
!.l
!.l
I
=
=
"
,
x
=
n
10.5.3
Functions with Symmetric Matrix Arguments
Dm denotes a duplication matrix and D�, its Moore -Penrose inverse (see Section 9.5) . L m is an elimination matrix (see Section 9.6) . (1) X (m ) symmetric: Reminder:
x m
H A N DBOOK O F M ATRIC ES
1 94
8vec(X2) = ( X 0 Im + lm 0 X ) Dm · ( a ) a v ech (X )' ' -1 i ' av ec ) (X b ( ) = """ ( \" i - 1 -j 0 Xi ) Dm ' i = l , 2, . . . .. avech(X)' JL... =O . i- 1 av (X ) ec h ' (c ) avech(X)' = D� L x i - 1 -j 0 xi Dm , i = I , 2 , i =D X , A ( m m ) symmetric: avech(X AX) - D;!; (X A rg; Im + Im 0 X A) Dm avech(X)' Lrn(XA 0 Im + lm 0 XA)Dm . X ( m m) symmetric, A ( p m ) , B ( m x m ) , C ( m q ) : avec ( A X B X C ) = (C' X B' A + C' 0 A \" B )Dm . 0 a vech(X) ' X ( m m ) symmetric, A (p x m ) , B ( m q ) : . i- 1 avec(AX' B ) = L ' x i j 0 AXi )Dm , i = 1 , 2 , . . . (B avech(X)' i = D m ) symmetric, A (p m ) , B ( m p ) : X (m iavec( AX B)' ' X A ' ) i - 1 - j B' 0 (AX B)j A B L ( avech(X) ' i=O '
.
(2)
(3)
.
.
x
x
x
x
'
(4)
x
x
-
(5)
x
1
-
x
x
1
i = l , 2, . . . (6)
X ( m m ) symmetric, A ( p x m ) , B ( m p ) , C (p p) : avec( AX B + C)i avech(X) ' iL ( B' XA' + C' ) i - 1 -j B' C?; (AXB + C)i A Dm , x
x
x
1
j =O
i = l,2, . . . ( 7)
X (m m ) symmetric, A (p m), B ( q), C (p q) : avec [(A X B + � ) (A X B + C) ] (/ ) + K q ) [B' 0( B' X A ' + C' ) A] Dm . q q avec ( X )' m x
x
x
'
=
x
195
V ECTOR A N D M ATRIX DERIVATI V ES
X (m x m) symmetric, A (p x ) B (m x q ), C ( p x q) : avec[(AX B + C)( AX B + C) '] ( /p > + K )[(AX B + C)B' 0 A]Dm . pp avech(X)' ( 9 ) X ( m x m) symmetric: avec(X 0 X) = U 0 mm 0 Im )[lm> 0 vec(X) + vec(X) ® lm>]Dm . rn l\ avech(X) ' ( 1 0 ) X (m x m) symmetric, A (p x m ) , B (m x q), C ( r x m), D (m x ) avec( AX B 0 C X D) avech(X )' ( /q 0 K$p 0 lr )[B' 0 A 0 vec(C X D) + vec(AX B) 0 D' 0 C]Dm . ( 1 1 ) X (m x m) symmetric: X) 8 avec(X ( a) fJvech( X )' -_ 2 d'1 ag ( vee X)Dm . av 8 X X X e ) c l 8 · 3 di ag [vec (X 8 X )] Dm b ( ) . uvech ( X ) 1 ( 1 2 ) X (m x m) symmetric, A ( p x m), B (m x q), C (p x m), D (m x q) : avec( AX B 8 CX D) ' 0 C) { diag[vec(AX B)]( D fJvech(X)' + diag[vec ( CX D)](B' 0 A)}Dm . Note: These results follow from those of the previous su b section and the chain rule for matrix differentiation. See also Magnus ( 1 988, Chapter 8 , Sec. (8)
m
,
=
..:. __ .. _:: ____:..._ ..._ .::.�
•
s
:
=
=
8.2) .
1 0.5.4
Functions with Lower Triangular M atrix Arguments
D;t; is the Moore -Penrose inverse of the duplication matrix Dm (see Section 9 .5 ) and Lm denotes an elimination matrix (see Section 9.6). ( 1 ) X (m x m) lower triangular: avech(X i ) Lm L (X')i-1-i 0 xi L'm , i 1 , 2, . . . fJvech(X)' i=O ( 2 ) X, A (m x m) lower triangular: fJvech(X AX) Lm (X ' A ' 0 Im + Im 0 X A)L ' . m fJvech (X)'
Reminder:
=
i- 1
=
=
H A N DBOOK O F M AT RICES
1 96
(m
x
lower triangular: avech(X ' X ) = 2 D+m ( /m ® X ' )L'm • avech(X)' avech ( XX') = 2D+ ( X 0 Im )L' m m· avech(X)' Note: For proofs see Magnus ( 1 988, Chapter 8 , Sec. 8.3 - 8.4).
( 3)
X
10.5.5
m)
P ro ducts of M at rix Valued Functions with Vector A rguments
( 1 ) x (m x 1 ) , Y ( .r ) ( n x n) :
avec(Y i ) - � [( ) i - 1 -j i ] avec(Y) ®Y .t Y uX a X ' - L... . O x ( m x 1 ) , Y( x) ( n x p) : avec(Y ' Y) _ ( I + r\ ) ( I Y ' ) avec(Y) , ® ax' ax ' avec(YY ' ) = ( /n2 + A.n n ) ( Y ® In ) avec( Y) ux' ux' X (m J ) , Y ( X ) ( n p), Z ( X ) (p q) avec ( Y Z ) - ( I 0 Y) avec ( Z) + ( Z ' I avec ( Y) . ® n ) ax' ax' ax' x ( m x 1 ) , Y ( x ) ( n x p) , Z ( x) ( q x r) , A ( s x n ) , B (p x q), C ( r x k ) : avec ( AYBZC) = ( C' 0 A Y B) avec(Z) + ( C' Z' B 0 A ) avec ( Y) ax' ax' ax' . x ( m x 1 ) , Y( x ) ( n x p) , A (q x n) , B (p x q ) , C ( q x q) : avec(AY B + C)' ax' i-1 L( B ' Y ' A ' + C')i - l - j B' 0 (AY B + C)i A avec(Y) ax' j =O I
!.l
)=
(2)
-
p>
pp
p
_
(4)
X
X
X
.
!.l
!.l
(3)
.
t
:
q
I
(5)
i = 1 , 2, . . .
( 6 ) x ( m x 1 ) , Y ( ) ( n x p), .r
A ( q x n ) , B (p x r ) , C ( q x r)
:
V E CTO R A N D M AT R I X D E R I VATI V ES
1 97
iJvec [(A Y 13 + C)'( A Y R + C)] ( a) ax' avec ( Y ) . = ( /r> + l\rr ) [H ' 0 ( H' Y' A ' + C' ) A ] 0 x' avec[( A Y lJ + C ) ( A Y lJ + C)'] (b) ax' OVPc ( Y) . = ( lq> + l\ qq ) [( M R + () B G) A] ' c') x ( 7 ) x ( m x I ) , Y ( x ) ( n x p) , Z ( x ) ( q x r· ) : , cJVPr ( Z ) 8vec ( Y) 8(ver ( Y ) 0 vec ( Z )] . 0 + ver 0 vec ( Z ) (} ) = ( a) >l ' u:r ax ' ox' CJVt'C ( Y (;) Z ) ,
(h)
.
,
I
[
(h:'
]
.
CJvPc ( Z ) ovec ( Y) . O ve r ( X ) + ve c ( } ) O lh:' = ( lp (·) l\ r n O lq ) . i.Jx' ( 8 ) x ( m x 1 ) , Y ( .r ) ( n x p) , Z (.r ) (q x r ) , A ( s x n ) , lJ ( p x k ) . (' ( / X q) , ]) ( r X lz ) :
CJvec ( A Y 13 c;;J (.'ZD) o;r'
( h 16> X
1\'h, 0
[( H'
0
A)
/J)
ow r ( y ) ax'
()
VPr ( ( 'ZD)
+ VPc( A Y H ) @ ( D' @ C)
owe ( Z ) (,1J' '
(9) x ( m x 1 ) , Y ( x ) , Z(x) ( n x p) :
].
avec ( Z 0 Y) ' ;r ox ' c} . , avec ( Y ) Hve r ( Z ) . . dtag( vee }' ) dtag( vee Z) + .1 ']r ;r ' ( .r' ( 1 0 ) J' ( m x 1 ) , Y ( .r ) ( n x p) , Z(x) ( q x r· ) , A ( s x n ) , U ( p x /; ) , Dve c ( Y
<·1
Z)
.
('
( .'i X
q),
[)
( r· X
CJve<· ( i\ Y R
( )
ox'
.
k) :
( 'ZD)
ovec ( (.' Z lJ (�) ax' .
'
A Y H)
dtag[w r( ( Z IJ ) ] ( H
I
C:)
A)
aw r ( Y )
().r 1 . . , , ()vpr ( Z) . + dtag[ ver ( A } R ) ] ( f) C:l ( ) ) ' '
'
( J' Not(�: The rPsult.s of t h is sn bsertion follow frotu t.lw ch a i n r u l e and t lw r u ks for basic matrix operations. Sf'(' also the rPsttl!.s of the p revious su bsed.ioll S , i n particular Sections 1 0 . 5 . 1 and 1 0 .5 . 2 .
H A N DBOOK O F M AT RICES
1 98
Jacobian Matrices of Functions I nvolving I nverse Matrices
10.6
X (m
x
m)
nonsingular: avec(X- 1 ) x' - l ® x - � . 8vec(X)' 8vec(X'- 1 ) 8vec(X) ' ( 2 ) X ( m m ) symmetric nonsingular: a vech(X- 1 ) Dm+ (X _ 1 ® X _ 1 )Dm , 8vech(X)' - where Dm denotes a duplication matrix and D;t. its Moore - Penrose inverse (see Section 9.5) . Note: For proofs of these results see Magnus & Neudecker ( 1 988, Chapter 9, Sec . 1 3 ) and Magnus ( 1 988, Chapter 8, Sec. 8.2 ) . (1)
x
_
1 0 . 6. 1
Matrix Products
(1) X (m
x
nonsingular, A ( m ) , B ( m p) : avec(AX- 1 B) - B' X t - 1 0 AX- I 8vec(X)' m ) nonsingular : i- 1 a vec [ (X - l ) '' j = - "' i - i 0 x- i - 1 ' i = 1 , 2, . . . ' ) (X L... 8vec (X)' ; =0 m)
n x
x
=
X (m
x
(3) X (m
x
(4)
x
(2)
·
n),
rk(X) = n : 8vec[(X ' X )- 1 ] = - (In, + Knn)[(X' X) - 1 0 (X' X) - 1 X ' ] . ovec(X)' X ( m n ) , rk(X) = m : ave;�� ��( 'r 1 l = - ( 1m, + Kmm )[(X X' ) - 1 x 0 (X X' ) - 1 ] .
(5) X ( m
x
m)
nonsingular, A ( n
x
m) ,
B (m
x
m) ,
C (m
x
p) :
1 99
V ECTOR A N D M ATRI X D ERIVATIVES
' BXC) = C' 0 A x- ' B - C' X' B ' X ' - ' 0 A X - ' . avec(Ax(a) avec(X)' ' C) = C' X ' - ' B' 0 A - C' X' - ' 0 AX E X - ' . avec(AX BXb ( ) avec(X)' 1 BX1 C) avec(AX(c) avec(X)' = - C ' x' - ' B ' x'- ' 0 A x - ' - C' x' - ' 1:9 A x - ' Ex - ' . Note: The results of this su b section follow from the chain rule and the results of the previous sections . 10.6.2 (1}
(2}
Kronecker and Hadamard Products
X (m m ) nonsingular, A ( p x q ) : avec(A 0 X - 1 ) - ( / .mp 0 Im ) [vec (A ) 0 X·t- J @ X· - I ] , avec( X ) ' = q <9 A avec(X- 1 0 A) - ( 0 . 0 Ip)[X ,_ 1 0 X, _ 1 0 vee( A)] . ovec( X )' = /m A qm X ( m m ) nonsingular: avec(.\ ... - I ) 0 vee (X ) I) I >I " '61vec ( " ) . (X vee (a) m uvec( v ) X ) 0 vec(X - 1 ) = I , 0vec(X - 1 ) - vec(X )0X '- 1 0X- 1 . avec( (b) m a vee ( ,\' )' ' avec( Xvec(X 1 ) 0 ) (c) avec(X)' = - [X ' - 1 0 x-' 0 vec(X - 1 ) + vec(X - 1 ) 0 X ' - 1 0 X - ' ] . 1 avec (X(d) avec(X)'0 X ) = ( /m 0 J\mm 0 Im )[vec(X- 1 ) 0 Im, - X ' - 1 1:9 x- 1 0 vee( X)]. avec( X 0 x- 1 ) (e) 8vec( X )' = ( /m 0 f{m m 0 Im) x [Im 0 vee (X- 1 ) - vee (X) 0 X , _ 1 0 X- 1 ] . X
x
.. \
-'l
2
1
KA '61
v 1-�
KA '61 ,1.
'
- I ,0,
,1.
'
H A N D BOOK OF M AT RICES
200
(3)
(4)
l 0 x- l ) x fJvec( (f) fJvec(X)' = -( Im 0 f\mm 0 Im ) x [X ' - 1 0 X - 1 0 vec(X-1 ) + vec(X- 1 ) 0 X ' - 1 0 X - 1 ] . X (m m) nonsingular, A (m x m) : fJvec(A 0 X- 1 ) _- fJ vec(X - 1 0 A) -_ _ d mg · ( vee A)(X ' - 1 0 x - 1 ) . 8vec(X )' 8vec(X )' X ( m x m ) nonsingular: 1 fJvec(X0 X ) _ ovec(X 8 x - 1 ) (a) 8vec(X)' 8vec(X)' x
diag(vec X - 1 ) - diag(vec X)(X ' - 1 0 X- 1 ) . I I X ()vee ) - 2 di ag(vec x - I )( X ' - 1 0 x - 1 ). 0 � \"" b ( ) vec(X )' Note: The results of this su b sec tion follow from the chain rule for derivatives and the results of the previous sections . =
=
10.6.3 (1)
Matrix Valued Functions with Vector A rguments
x (m x
Y(.r) ( n x n ) nonsingular: fJvec(Y - 1 ) _ ( yt - l 0 y- 1 /vec(Y) . fJ x' fJ x' ( x 1 ) , Y(x) ( n x n ) nonsingular, Z(x) ( n x p) : fJvec(Y - 1 Z) = ( I 0 y - l ) fJ vec(Z) _ ( z' y ' - l 0 y - l ) fJvec(Y) P fJ x' fJ x' fJx' _ ( m x 1 ) , Y(.r) ( n x n ) nonsingular, Z(.r) (p x n ) : fJvec( Z }" - 1 ) - (Y ' _ 1 0 I fJ vec( Z ) ( y ' - l 0 zy - 1 ) fJvec(Y ) P) fJ x' fJ x' fJ x' (m x 1 ) , Y(.r) , Z(x) ( n x n ) nonsingular: fJvec(Y- 1 z - 1 ) fJ x' ( y ) - ( zt - I 0 y - I Z- I ) fJvec ( Z) . ( zt - y 1 - I 0 y - ) fJvec fJ x' fJ x' 1),
=
(2)
x
(3)
x
m
_
_
·
( 4)
x
=
-
I
I
20 1
V ECTO R A N D M AT RI X DERIVATI V ES (5)
x (x)( ( 1 ) , Y(x)(x) (( U q )V avec( U Y-1 V z-1 W ) ax' m x
x n ,
n x n) n x p) ,
nonsingular,
W (x) (p
Z(x) (p
x r) :
x
p)
nonsingular,
Y(x) ( q) , Z(x) ( p p) nonsingular: a[vec(Y) Q9 vec(Z-1 )] avec (Y1 ) vee( z - l) ax' ux ) (Z ec av ' - vee( y) 0 ( z - l z - l ) ux' ( 7) x ( Y(x) ( ) nonsingular, Z(x) (p q) : a[vec(Y-1) 0 vec(Z)] ax' (6)
x(
m x 1),
x
n x
(9
£l
(9
m x 1),
( 8)
---!l :-: -' -'-
x
n xn
x ( 1 ) , Y(x) ( ) nonsingular, Z(x) (p p) nonsingular: a[vec(Y-1) Q9 vec(Z-1 )] - vee ( y - I ) 0 [( z'- I 0 Z - I ) _a__ve,-_, c -'-(- Z_:--_)] ax' - [(Y '- I C9 y- 1) 8vecax'(Y) ] (S: vec (z -1 ). x ( 1 ) , Y(x) ( q), Z(x) (p p) nonsingular: avec(Y 0 z-1) 0 0 ax' avec(Y) [ vec(z - l ) X ) v . �:�Z 0 ) ] ' ( 1 1 0 Q9 c( - ve Y ) (z Z ) m
x
n x n
x
O J:'
(9)
mx
n x
x
-
( fq
I\pn
OX '
fp )
(9
H A N DBOOK OF M ATRICES
202
( 10) x
( m x 1 ) , Y ( x ) ( n x n ) nonsingular, Z ( x ) ( p x q ) : ovec(Y- 1 <9 Z) ( /n @ I\qn @ fp ) ox'
ovec(Z) _ 1 (Y ) vee @ [ 0 x' - ( (Y ' - t @ Y - t ) ov��( Y ) ) @ vec (Z ) ] . ' nonsingular, Z(x ) (p x p) nonsingular:
x
(11)
x ( m x 1 ) , Y( x ) ( n x n) ovec(Y- 1 @ z - t ) ox'
- ( /n @ 1\pn @ fp )
[vec( Y - 1 ) <9 (( z'- 1 @ Z - 1 /v��Z� ) ) ov Y ) ( l I ' ) @ vec( z- t )] . ��� y Y ) + ( @
x
( 12)
x ( m x 1 ) , Y (x) ( n x n) , Z(x) ( n x n) nonsingular: ovec ( Y 0 Z- 1 ) ovec( z - t 8 Y) ox' ox' . ( vee z _ 1 ) ovec ( Y ) d tag ::::. x' u - diag(vec Y ) (Z' - 1 <9 z- t ) ov�c(Z) . x'
( 13)
x (m x 1 ) , Y ( x ) ( n x n ) nonsingular, Z ( x ) ( n x n ) nonsingular: ovec(Y) ovec(Y- 1 <� z - t ) t t ' 1 <9 z[diag(vec )(Y y - ) ox' ox' . ] + dtag(vec Y - 1 )(Z'- 1 <9 Z - 1 ) ovec(Z) . ox'
The results of this sub section follow from the chain rule for derivatives and the rules of the previous sections. Note:
10.7
C hain Rules and M iscellaneous Jacobian Matrices
denotes a duplication matrix and D:;, its Moore -Penrose inverse (see Section 9 . 5 ) . Lm is an elimination matrix (see Section 9.6).
Reminder: Dm
V ECTO R A N D M ATRIX DERIVATIV ES
203
( 1 ) x ( m x 1 ) , y(x ) ( n x 1 ) , z ( y) (p x 1 ) : oz(y(x)) ox'
(2) X ( m
x n),
Y(X) ( p
x
q) ,
oz(y) oy(x) oy' ox'
Z(Y) ( r x s )
:
8vec Z(Y(X)) 8vec Z(Y) 8vec Y(X) . ovec(Y)' ovec(X) ' 8vec(X)' (3) X ( m x n ) , S open subset of IRm x n : rk (X) = r for all X E S => x + is a differentiable fun ction of X on S and + x
+ x + 0 Un - x + X )] Kmn . (4) X ( m x n ) , Y ( X ) (p x q), S open subset of IRm x n : rk[Y ( X )) r for all X E S => Y ( X ) + is differentiable on S and =
(- y + ' 0 y +
8vec Y(X) + 8vec( X )'
+
+ y+' y + 0 ( I q
(5) X ( m
x m)
:c --::----.:.--::-:,-:...
X (m
x
m)
[(Ip - yy + ) 0 y + y + ' _
y + Y)]Kp
8vec(Y) . )q ovec(X)'
nonsingular:
ovec( X a q; ) ovec(X)' (6}
'
=
det(X) [(vec x - 1 )(ve e X ' - 1 ) ' - x' - 1 0 x - 1 ) .
symmetric nonsingular:
8vech( X a
+ [(vee X - 1 ) (vee X - 1 ) , - X - 1 0 X - 1 ] Dm . ) t(X de Dm ovech ( X )' _
( 7 ) X ( m x m ) lower triangular, nonsingular:
0;::�;���; )
(8}
X
(m x n ) ,
=
det(X)L m [(vec X - 1 )(vec X' - 1 ) ' - X '- 1 0 X - 1 ] L :., .
Y(X) (p
8vec[Y ( X ) a
=
x
p) nonsingular:
8vec( Y ) 1 1 I I 'y )[( Y c ' )'c Y ) ve )( ve Y Y 0 det( . 8vec( X )'
H A N DBOOK OF M ATRIC ES
204
(9)
X
( m x m ) positive definite, Y ( X ) = [Yii ] ( m x m ) lower triangular with Yii > 0 , i = 1 , . . , m , such that X = Y Y ' , that is, Y is obtained by a C holeski decomposition of X : 8vech ( Y ) 8vech ( X )' .
� [D;t; ( Y 0 lm ) L:., t 1 . Note: ( 1 ) and ( 2 ) are just chain rules. (3) is given in �hgnus & Neudecker ( 1 988 , Chapter 8, Sec . 5 ) . ( 4 ) follows from (3) and a chain rule. (5) is established in Magnus & Neudecker ( 1 988, Chapter 8, Sec. 6 ) . (6), ( 7 ) and ( 8 ) follow from ( 5 ) and a chain rule ( see also Magnus ( 1988, Chapter 8, Sec. 8 . 2 , 8 . 3 ) ) . ( 9 ) is, e.g., given in Liit kepohl ( 199 1 , Appendix A , Sec. A . 1 3 ) . 1 0 .8
10.8 . 1
Jacobian Determinant s
Linear Transformations
8vec ( X ) ) ( ( a) det = 1. 8vec( X )' ' (b) det ( 8vec( .": ) ) = ( - l ) mn ( m - 1 )( n - 1 )/4 . 8vec ( ) 8ve c( ,: ) ) ( = ( - l ) mn ( m - l ) ( n - 1 ) /4 . (c) det 8vec( .\ ' ) ' 8vech(X) ) ( ( ( 1. et d 8vech( X )' mn · IR : det ( 8vec(cX ) ) (3) X 8vec(X) ' 8vech(cX ) ) det ( ( ) cm( m + 1 ) / 2 . 8vech( X )
(1) X (m x n) :
,\
2) X m x m)
(m x n),
(4) X m x m ,
'
=
:
c
E
_
-
c
(5) X (m x n), A
E IR :
( 111
x
'
m),
c
=
B (n x n) : o ec ( - B) (de t A) "(det B) "' . de t vee ,\ ) ' (6) X (m x n), A (n x n), B (m x m) : 8vec ( A ,":' B) ( - l ) m n ( m - l )( n - 1 ) / 4 ( de t A ) m ( de t B t . det 8ve c( .\ )'
(
( � �: ) )
=
=
205
V ECTO R A N D M ATRIX DERIVATIVES
( 7 ) X, A =
[a ;j ), B = [b;j ) ( m x m ) lower triangular : ( 8vec h( A X B ) ) m ; rn- i + l = rr a . b d et · 8vech(X)' 1::1 , "
.
(8)
X (m
x
m)
symmetric, A ( m
x m) :
( 8vech ( A' X A) ) = (det A ) m + l . det 8vech( X )' (9)
X ( m x m ) symmetric, A , B ( m eigenvalues of BA - 1 : det
( 10) X ( m
x
x m) ,
det(A)
:/= 0, A 1 ,
.
•
( 8vech (AX A' ± BX B' ) ) 1 = (det A ) m + l rr( ± A; A1 ) > 8vech( X )' I. _ }. m)
symmetric, A =
[a;j ), B = [b;i] (
m
x m)
.
, Am
.
lower triangular:
( 8vech(AX A' ± BX B') ) = IT det > (a;;ajj ± b ii bjj ). 8vech( X ) ' J _} ( 1 1 ) X ( m x m ) symmetric, A , B ( m x eigenvalues of A B - 1 :
m) ,
det(B)
:/= 0, A 1 , . . . , Am
( ov ec h ( A X B ' + B X A ') ) = 2 m d et ( A )( d et B ) m ii d et ( + 8vech( X ) ' •. > "• "; ) ; ( 12 } X , A = [a;j], B = [b;j], C = [c;j], D = [d;j) ( m x m ) lower triangular: \ .
(8vech(AX B + C X D) det 0 vech ( X ) '
( 13 ) X ( m
x
m)
\ .
.
) = II ( aii bii + c;; d11 ) . i?. i
lower triangular, A , B ( m x m) nonsingular:
mII1 ( 8vech( B' X A + A' X' B) ) det(C(il ), det = 2 m det(A)(det B ) m ovech( X ) ' I :: I -
where i
c( i) =
Cj1
...
=1
'
...'m
Cii
are the principal submatrices of C =
[c;j ) = A B - 1 .
-
1'
H A N D BOOK O F M ATRICES
206 ( 14 )
X,
A=
[a ;j ] ( m x n ) , B ( m x m ) , C ( n x n ) :
m 8vec ( A 8 BXC) ) ( n aij · det = ( det B) n (det C) rn ii II o vec ( X )'
( 1 5 ) X ( m x n) ,
det
( 16)
A=
i=l j=l
[a;j] ( n x m ) , B ( n x n ) , C ( m x m ) :
( 8vec(A 8 BX' C) ) 8vec ( X ) '
n m l = ( ) mn( m - 1 )( n - 1 )/ 4 ( de t B ) m ( de t C t II II a;j . i= I j =I X , A = [a ;j], B = [b;j] ( m x n ) : m n 8vec ( A 8 X 8 B) ·b· · · d et II II - i= a , 8vec(X) ' i=J l ;
)
(
( 17) X (m x n),
A=
8vec ( A ( det
_
•; .
[a;j], B = [bij] ( n x m ) :
0
X' 8 B) 8vec( X)'
) = ( - 1 )mn( m- l )(n - 1 )/4 IIn IIrn a · · b · · . i=l i=l
•;
•;
Note: The results of this subsection follow from the rules of the previous
sections of this chapter and the rules for determinants (see also Magnus ( 1988, Chapter 8)). 10.8.2 (1)
Nonlinear Transformations
( Chain rule)
x,
y( x ) , z (y) ( m x
(
1) :
x) d et 8z (y( ) (2)
( Chain rule) X, Y ( X ) , Z ( Y ) ( m
>�
ux'
) = det ( oz (y) ) det ( oy( x ) ) >�
>�
uy'
ux'
.
x n) :
( ( 8vec Y(X) ) ( 8vec Z( Y ) ) 8vec Z ( Y ( X ) ) ) . det det = det 8vec ( X )'
( 3 ) X (m
x m)
8vec( Y )'
symmetric with eigenvalues .\ 1 , .
8vec( X)'
. , A rn :
( 8vech(X i ) ) ·m det = z (det X ) i - 1 II ovech( X ) '
.
"'L > I
J.l k t ,
207
V ECTO R A N D M ATRIX D E RI VATIV ES
wherP
{ -
/-l k l
( 4 ) X = [x1:1]
A�: # At A�:
=
At
(m x m) lower triangular:
( 8vech( X i ) ) det 8vech( X) '
where /-l k l
(5) X, A
if if
( A � - A I }/( A �: - At ) i A �- 1
_
=
{
=
m
· z ( det X ) i - l
( x �k - x i1 ) / ( xu - x u ) . i 1 z x u-
if if
II /-l k l ,
k>l
XU
= X I/
(m x m) symmetric, A 1 , . . . , A m eigenvalues of XA :
( 8vech(XAX) ) det = 2 m det(A )det (,\ ) II ( A; + Aj ) . 8vech( X )'
(6) X = [x;i ] , A = [a;j]
,
.
• >J
(m x m) lower triangular:
8vech(X AX) ) ( det = 2 m det(A)det(X) II (a; ; X;; + ajjXjj ) . 8 vech ( X ) ' .. • >J
(7)
X = [x;j ]
(m x m) lower triangular:
( 8vech (X'X) ) det 8vech( X ) '
( 8vech(X X' ) ) det 8vech(X)'
(8) X (m
X
2 m II xi; , i=l
m
- .
2 m II x 7;' i + l i=
1
x m) nonsingular: det
(9)
m
(8;�:��;):)) = ( - l ) m (det X) - 2m .
(m x m) symmetric nonsingular:
8vech ( X - l ) ) ( det = ( - l ) m ( m + l )/2 (de t X ) - ( m +l ) . 8vech( X )'
( 10)
X
( m x m) nonsingul ar, lower triangular:
8vech ( X - l ) ) ( = ( - l ) m ( m + l ) /2 (det X ) - ( m + l ) . de t 8vech( X)'
H A N D BOO K O F M ATRICES
208
(11) X
(m
X
(m
x
m) symmetric nonsingular: 8vech( X a dj ) = - m ( m + 1 ) /2 ( 1 - m)( det X ) ( m + 1 )( m - 2)/2 . ( 1) det 8vech(X)'
(12)
(
)
x
m) nonsingular, lower triangular: 8vech( X a � ) ( - 1 ) m (m + 1 ) f 2 ( 1 - m)(d et X ) ( m + l ) ( m - 2)/2 . det 8vech( X ) '
(
)
=
Note: Most results of this subsection are given in Magnus ( 1988, Chapter 8 ) . They fol low from rules of the previous sections and t he results for
determinants .
Matrix Val ued Functions of a S calar Variable
10.9
( 1 ) x E IR., A ( x ) = A (2)
(m x
n)
dA = dx
constant:
'( Linearity) x E IR., A ( x ) , B ( x ) ( m x n ) , c 1 , c 2 E
Om x n ·
IR. :
+
d [c 1 A ( x ) c2 B( x )] dA(x) =c 1 dx dx
+ c2
dB(x) dx ·
( 3 ) ( Product rule) .r
E
IR. ,
A ( .r ) (m x n ) , B( x ) ( n x p ) : dA(x) B(x) dX
( 4 ) .r E IR ,
_ 4 ( ) d B( x )
dA(x) B( + dx dx X). A ( x ) ( m x n ) , B( .r ) ( n x p) , C ( x ) (p x q ) : -
'
X
d A ( x ) B( .t· )C( x ) dx d C( x ) dB(x) = A ( x ) H(x) C( x ) + A( x ) dX dX
(5) x E
IR., A ( .r ) ( m x m )
+
dA(x) B ( x )C( x ) . dX
nonsingular:
d.4( x ) - 1 dx (6)
( Ratio rule)
.r E IR.. A ( x ) (m x d B ( x ) A ( .r ) - 1 d .r
m) nonsingular, =
B ( x ) (n
x
m) :
dA(x) dB(x) _ 1 _ _ 1 A(x)- l . A(x ) B( x ) A ( x ) dx dx
V ECTOR A N D M AT R I X D E RI VATIV E S
209
( 7 ) (Generalized i nverse rule) x E IR, A ( x ) ( m x n ) , A ( x ) - some generalized inverse of A ( x ) : A(x)
dA(x)A(x) dx
=
-A(x)A ( x ) -
dA ( x ) x )- A ( x ) . A( dx
Note: The first fou r ru les follow from t he corresponding rules for real valued fu nctions by considering typical elements of t h e matrices i nvolved. Rule ( 5 ) is obtained by apply ing t he product ru le to A A - 1 lm , ( 6 ) follows from ( 5 ) and the pro duct rule and the resu l t in ( 7 ) follows by applying t llf' product rule to A A - A = A and multiplying by A A - from the left . =
11 Polynomials, Power Series and Matrices 1 1. 1
Definitions and Notations
11.1.1
Definitions and Notation Related to Polynomials
Polynomial: For given Po , PI , . . . , Pn E
·
r given P o , PI , . . . , Pn E IR, a function p : lR defined by p( x ) = Po + PI x + · · + Pn x n is a real polynomial .
Real polynomial:
F
o
-
IR
·
Degree of a polynomial: A polynomial p(x ) Pn =/= 0 is said to have degree n .
=
Po +P 1 X + · · +Pn X n with ·
+ p 1 ,\ + · · · +
Root of a polynomial: A number ,\ for which p(,\ ) = po Pn ,\ n = 0 is called a root of the polynomial p( · ) .
Multiplicity of t he root of a polynomial: A polynomial p( x ) = Po + P I X + · · · + Pn X n with distinct roots .\ 1 . . . , A r can be written in the ,
form
p(x )
=
Pn ( X - .\ 1 ) 1: 1 · · · ( x - A r ) k •
=
r
Pn
IJ ( x - .\ ; ) 1: ,
i= I
where the k; are positive integers. ki is called the multiplicity of the root .\; , i = 1 , . . . , r.
Convergent polynomial: A polynomial p( x) = po + p 1 x + · · + pn x n with roots ,\ 1 , . . . , Ar is said to be convergent if j.\; labs < 1 , i = 1 , . . . , r . Divergent polynomial: A polynomial p(x) = po + P I X + · · · + PnX n with ·
roots .\ 1 , . . . , A r is said to be divergent if j .\; labs > i E { l , . . . , r} .
1
for at least
one
·) l "l �
H A N DBOOK O F M ATRJ< ' ES
�
Asymptotically stable polynomial: A real polynomial p( x ) = p0 + PI X + · + p71 x" with roots o 1 + i/31 , . . . , O r + iBr is said to he asymptotically stable if o , < 0, i = 1 , . , r. ·
·
. .
Unstable polynomial: A real polynomial p( x ) = po + p1 x + with roots o 1 + i8 1 , . . . , O r + i/3r is said to be u nstable if at least one i E { 1 . . . r} . .
.
11
11
·+p X o; > 0 for
·
·
L inear polynomial: A polynomial of degree 1 is linear. Monic polynomial: A polynom ial p( .r ) = Po + PI x + Pn = 1 is called monic.
· ·
· + p11x" with
Factor or divisor of a polynomial: If the polynomial p(x) can be written as the product of two polynomials q ( x ) and r(x ) , where q ( x ) has degree > 0 . p( .r ) = q ( x ) r( .r ) , the polynomial q ( x ) is said to be a factor or a divisor of the polynomial p( x ) . Common factor or divisor of polynomials: I f the polynomials p( x ) and q ( x ) both have a factor r ( x ) , that is, t here exist polynomials p l ( x ) and ql ( x ) such that p( x ) = r( x )pl ( x ) and q ( .r ) r( x )q1 ( x ) , r ( x ) is said to be a common factor or divisor of the polynomials p(x ) an d q( x ) . =
Greatest common divisor of two polynomials: Among all monic polynom ials which are common divisors of t he polynomials p( x ) and q( .r ) t he one with the largest degree is said to be the greatest common divisor or factor of p( x ) and q( x ) . ,
Relatively prime o r coprime polynomials: The polynomials p( x ) and q ( x ) are relat i vely prime or coprime if they do not have a common
factor.
Dividing polynomial: A polynomial q ( x ) divides a polynomial p( x ) if q ( x ) is a factor of p( x ) . Least common multiple of two polynomials: The least common mul tiple of the polynomials p(x) and q ( x ) is the polynomial r( x ) with smallest degree such that p(x) and q( x ) divide r·( x ) . Resultant: A function f : (f,m + n + l (f, is called a resultant if f (po , p J , · · . Pm - 1 • Qo , ql , · · . , qn ) i= 0 ¢::::::> p(x ) = Po + P i X + · · · + l + x m and q( x ) = qo + x + · · + q,. x are relatively prime m x 1 Q1 " Pm--+
,
·
polynomials.
POLY N O M I A LS , POWER S E R I ES A N D M ATRICES
11.1.2
213
Matrices Related to Polynomials
Companion matrix: The companion matrix of a monic polynomial p ( x ) = po + PI X + · · · + Pm - I X m - l + x m is the (m x m) matrix 1
or
or -Pm - 1 -Pm - 2
- po
-P I
- Pm - 2
-Pm - 1
0
0
1
0 0
0
0
1
.
.
.
0 0 0
0
1
0
0
0 0
0 0
1
0
0
1
-po
-pi
-Pm - 2
-Pm - 1
1
0
0
1
.
.
.
0 0
0
- po -pl
0 0 0 0
1
or -pi -po
1
0 0 0 0
1
0 0
0
0 -Pm - 2
0 1
-Pm - 1
Sylvester matrix: For the polynomials p ( x ) = po +PI x+ · · · + Pm - l x m- 1 + x m and q(x) = qo + Q1 x + · · · + Qn - t X n - l + Qn X n , the corresponding
Sylvester matrix is 1
0 5=
Pm - 1 1
0
0
Qn
Qn - 1 Qn
0
Pm - 2 Pm- 1
Po PI 1
Qn - 2 qn - 1
0 Po
Pm - 1 Qo Ql
0 0 PI
0
Po
0 0
qo
( (m + n ) x ( m + n ) )
where the upper part consists of n rows and t h e low r p art consi s t s of m r ows For example, for p ( x ) = p0 + p 1 .r + p2 .r 2 + .r 3 and e
where Pi = 0 for i < 0. For example, for m = 4 , H=
P3 P4
0 0
Pt P2 P3 P4
0 Po Pt P2
0 0 0 Po
is the Hurwitz matrix corresponding to p( x ) . Derogatory matrix: An ( m x m) matrix A is derogatory if a polynomial p( x ) = Po + p1 x + · + Pr .rr of degree r < m exists such that ·
·
p( A ) = Pa lm + Pt A + · · · + Pr Ar 1 1 . 1.3
=
Om x m ·
Polynomials and Power Series Related to Matrices
Characteristic polynomial: The polynomial defined by p(x ) = det ( x lm - A )
is t he characteristic polynomial of the ( m x m) matrix A . It is denoted by PA C ) . Annihilating polynomial: A polynomial p(x ) = Po + Pt X + · · · + Pn x" is said to annihilate the ( m x m) matrix A if p( A ) = pa lm + Pt A + · · · + PnA"
=
Om x m ·
POLYN O M I A LS, POW ER SERIES A N D M ATRICES
•) �
Minimal polynomial of a matrix: The monic polynomial with minimal degree that annihilates an ( m x m ) matrix A is called the mini mal polynomial of A . It is denoted by QA ( · ) . Matrix polynomial: Given a polynomial p(.r ) = Po + PI X + · · · + Pn X" . th e fu nc ti o n p : (fm x m --> IRm x m ) defined by
p(A ) = Pa lm + P I A + · · · + Pn A " for
(m x m )
matrices A, is a matrix polynomial .
Matrix power series: Given a power series p(x) = 2:::::: � 0 pn .r " . the function p : ([' m x m (or p : IRm x m --> IRrn x m ) defined by p(A )
=
L Pn A"
n=O
for ( m x m) matrices A , is a matrix power series. provided t he infinite sum exists. Exponential function of a matrix: A ( m x m)
exp ( A )
=
1 L -1 A" . co
n=O
Tl .
Sine of a matrix: A ( m x m )
sin(A) = L ( - 1 )" co
n=O
(2n + 1 ) !
A2 n + l .
Cosine of a matrix: A ( m x m)
cos( A )
1 1 .2
=
co
( - 1 )"
n=O
2n.
L
1
A 2" .
Resu lt s Relating Polynomials and Matrices
Reminder: For an ( m x m) matrix A , • •
PA ( x ) = det(xlm - A ) denotes the characteristic polynomial . Q A ( x ) denotes the minimal polynomial.
10 "
216
(1)
H A N D BO O K O F M ATRICES
A ( x ) ,\ E (f : (a) QA (x) is unique . (b) degree[qA (.r)] < (c) p(x) is a polynomial with p(A) = 0 => QA (x) divides p(x). (d) PA (A) = 0 (Cayley-Hamilton theorem). (e) QA (x) divides PA (x). (f) PA ( ,\ ) 0 {::::=> ,\ is eigenvalue of A. (g) QA (,\) 0 {::::=> ,\ is eigenvalue of A . p(.z: ) a polynomial with companion matrix C: The eigenvalues of C are the roots of p( x) . A ( m x m) with distinct eigenvalues .\ 1 , . . . , ,\" having multiplicities m1 , respectively: m
m
,
m.
=
=
(2)
(3)
. . . , mn ,
(4)
and r; is the order of the largest block of the Jordan decomposition of A corresponding to the eigenvalue .\, (see Chapter 6 for the Jordan decomposition) . (.z: A ( m x m ) with distinc t eigenvalues .\ 1 , . . . , .\, , q(x) (.r-.\ 1 ) .\71 ) : A is diagoualizable {::::=> q(A) = 0. =
( 5 ) A ( rn x rn :
) ( a) A is diagonalizable {::::=> all roots of QA(x) have multiplicity 1 . (b) A is diagonalizable {::::=> all linear factors of QA (x) are distinct . (c) A is diagonalizable {::::=> [qA(x) 0 => dqA (x)/dx -:f. 0] ( where dqA(J· )/dx denotes the derivative of QA(.r)). p(x) a monic polynomial: C is a companion matrix of p(x) => p(x) qc(x) pc(x). A ( m x m ) . C companion matrix of PA (x) : (a) A is similar to C {::::=> QA(x) = PA (x). (b) A is similar to C' {::::=> every eigenvalue of A has geometric multiplicity 1 . A , B (rn x rn ) : (a) A , B similar => QA (x) QB(x). ( b) A , B similar => PA (x) = PB (x). (c) P .4. (x) = P B (x) QA (x) = QB (x) => A, B similar . =
(6) (7)
(8)
· · ·
=
=
=
=
P OLYN O M I A LS, POW E R SERIES A N D M ATRICES
217
similar, p(x ) a polynomial: p( A ) = 0 ¢:::::} p ( B ) = 0. QA$B ( x ) is the least common mu ltiple of ( 10 ) A ( m x m ) , B ( n x n ) q A ( X) and q B ( X ) . ( 1 1 ) ( Hurwitz theorem) A real polynomial p( x ) is asymptotically stable if and only if all principal minors of its Hurwitz m atrix are positive. ( 9 ) A, B (m x m )
Note: Most results of this section can be found in Horn & Johnson ( 1 985 ) . The last resu l t is given in Barnett { 1 990) . 1 1 .3
Polynomial Matrices
1 1. 3 . 1
Definitions
Polynomial matrix: An ( m x n ) matrix P ( x ) = [p,j ( .r )] whose typical elements are polynomials Pij ( x ) = Pij ,O + Pij , ! X + · · · + Pij, r ,1 X r '1 , is a
polynomial matrix. A lternative notation :
where r
=
max; , j r;j and
with Pij , k = 0 for k
P1 1 ,k
Pl n , k
Pm l ,k
Pm n ,l:
k = 0, . . . , r,
> r;j .
Real polynomial matrix: A polynomial matrix whose elements are real
polynomials is sai d to be a real polynomial matrix.
Degree of a polynomial matrix: The degree of an ( m x n ) polynomial matrix P ( x ) [p;j ( x)] with typical elements p;; ( x ) = Pij ,O + PlJ. . . 1 x + · . · + p IJ· · ,r,1 x r • is max &. ,J. rIJ· ' where rlJ· is the degree of Pii ( x ) , i = 1 , . . . , m , j = 1 , . . . , n . Equivalently, the degree of P(x) = Po + P1 x + + Pr x r is r if Pr 'f. 0. =
1
·
·
!
·
Polynomial matrix operations: All matrix operations specified in Chapter 1 are defined analogously for polynomial matrices except
where noted otherwise in the following.
Rank of a polynomial matrix: The rank of a polynomial matrix is the
number of columns of the largest submatrix whose determi nant is not identically zero.
H A N DBOOK OF M AT R I C ' ES
218
Eigenvalues or latent roots of a polynomial matrix: The eigenvalues or latent roots of the polynomial matrix P( x ) are the roots of t.lw polynomial det P ( x ) . Latent vector of a polynomial matrix: Let P ( x ) be a polynom ial m atrix w i t h eigenvalue .\ . A vector u such t hat P( .\ ) v = 0 is called a l atent vector of P ( x ) . Regular polynomial matrix: The ( rn x rn) polynomial matrix P ( x ) Po + P1 .1· + · · + P, .I. r is c alled regular i f Pr is nonsingu lar.
=
·
Monic polynomial matrix: The ( m x m) polynomial matrix P( x ) Po + P1 x + · · · + P xr is monic if Pr = lm . r
Unimodular or invertible polynomial
matrix: The
( m x m)
polynomial mat rix P ( .1· ) i s said to be u n i mod u l ar or invertible if det P ( 1.· ) = const ant :f. 0. t h at is. det P ( .l' ) is a const ant fu nc tion .
Left an d right multiples of polynomial matrices: The ( m x r1 ) polynom ial mat rix P ( J' ) sat isfy i ng P ( x ) = Q ( x )T( x ) for polynomial
matrices Q ( J.' ) ( 111 x h ) and T(.l' ) ( h x n ) is called a left m u l t i ple of T( .l' ) and a right m u l tiple of Q ( x ) .
Left quotient and left divisor: Consi der polynomial matri ces P ( .I' )
=
Po + P1 x
+
· · ·
+
Pr .l'r
(m X n).
P, ::j::. 0.
and Q ( x ) = Qo
+
Q1x
+
·
· ·
+
Q.x•
(m x h),
det( Q. ) :f. 0 and
s
< r.
That is, t he latter polynomial is regular. An (h x n) polynomial matrix T( x ) is a left quotient of P ( x ) and Q ( x ) if an ( m x n ) poly nomial matrix R ( x ) w i t h degree less than s exists such t hat P ( x ) = Q ( J.· )T ( J: ) + R ( x ) . The polynomial matrix R(x) is said to be a left remainder . Q ( J.: ) is called a left divisor of P( x ) if R( .l' ) 0 and i n t his c a.se P ( x ) is said to be left divisible by Q ( x ) . _
Right quotient and right divisor: Consider polynomial matrices
and Q(x)
=
Q0 + Q 1 x
+
· · ·
+ Q . x ·'
(m x h),
d e t(Q. ) :f.
0 and
s
< r.
T hat is, the l atter polynomial is regular. An ( m x h ) polynom ial matrix T(.1· ) is a right quot ient of P ( x ) and Q ( x ) if an ( m x n )
219
POLYN O M I A LS, POWER S E RI ES A N D M ATRICES
R( x ) with degree less than s exists such that P ( x ) = T ( x ) Q ( x ) + R( x ) . The polynomial matrix R(x) is said to be a right remainder. Q ( x ) is called a right divisor of P ( x ) if R ( x ) El 0 and in this case P ( x ) is said to be right divisible by Q ( x ) .
polynomial matrix
Common left divisor: A polynomial matrix D(x) is a common left divisor of the polynomial matrices P ( x ) and Q ( x ) if D ( x ) is a left
divisor of both P ( x ) an d Q ( x ) , that is, if polynomial matrices P1 ( x ) and Q 1 ( x ) exist such that P ( x ) = D ( x ) P1 ( x ) and Q ( x ) = D( x ) Q I ( x ) . D(x) is called a greatest common left divisor of P ( x ) and Q ( x ) if for any other common left divisor F( x ) there exists a polynomial matrix B ( x ) such that D ( x ) = F ( x ) B ( x ) . The polynomial mat rices P ( x ) and Q ( x ) are said to be relatively left prime or left coprime if their greatest common left divisors are unimodular.
Common right divisor: A polynomi al matrix D(x) is a common right divisor of the polynom ial matrices P ( x ) and Q ( x ) if D( x ) is a right
divisor of both P ( x ) and Q ( x ) , that is, if polynomial matrices PI ( x ) and Q ! ( x ) exist such that P ( x ) P1 ( x ) D( x ) and Q(x) Q 1 ( x ) D( x ) . D(x) is cal led a greatest common right divisor of P ( x ) and Q ( x ) if for any other common right divisor F ( x ) there exists a polynomial matrix B ( x ) such that D(x) = B ( x ) F ( x ) . The polynomial matrices P ( x ) and Q ( x ) are said to be relatively right prime or right coprime if thei r greatest common right divisors are unimodular. =
=
Skew prime polynomial matrices: The polynomial matrices P ( x ) and
are said to be skew prime if det P ( x ) and det Q(x) are relatively prime polynomials. Q(x)
Elementary operations for polynomial matrices: The following
modifications of a polynomial matrix are called elementary operations: ( i ) i nterchangi ng two rows or two columns, ( ii ) multiplying any row or column by a nonzero number, ( iii ) adding to one row another row multiplied by an arbitrary polynomial , ( iv) adding to one column another column multiplied by an arbitrary polynomial.
Elementary polynomial matrix: An (m X m) polynomial matrix is
elementary if it may be obtained by applying a single elementary operation to lm .
H A N DBOOK OF M ATRICES
220
Equivalent polynomial matrices: Two polynom ial matrices P ( x ) and Q(x) U(x)
are said to be equivalent if unimodular polynom ial matrices and V ( x ) exist such that P ( x ) = U (x ) Q (x ) V (x ) .
Characteristic matrix: The ( m x m) polynomial matrix P ( x ) = x lm - A is the characteristic m atrix of an (m x m) matrix A . Invariant factors or invariant polynomials of a polynomial mat rix: The invariant factors or polynomials of an ( m x m) polynomi al matrix P( x ) of rank r are defined as . lk ( x )
=
dk ( X ) , dk - L ( X )
k = 1 , . . . , r,
w here d0 ( x ) = 1 and dk ( x ) is the monic greatest common div isor of all minors of order k of P ( x ) for k = 1 , . . . , r. Elementary divisors of a polynomial matrix: Let ,\ be a root of an
i nvariant factor of a polynomial matrix P(x ) . Then the linear polynomial p( x ) x - ,\ is an elementary divisor of P(x ) . =
Block companion matrix of a polynomial matrix: The (block ) companion matrix of an ( m x rn ) polynomial matrix P( x ) Po + P1 .r + + Pr _ 1 .r r - L + lm .r r is the ( rrn x tm ) matrix =
·
·
·
...
0
or
0 0 - Pt
or - Pr - L - Pr - 2
fm 0
0 lm
- Po
0 0
0 0
0
0
0 0 0 - Po
- PL
... ...
0 0
0
0
lm 0 - Pr - 2
0 fm - Pr - L
0 lm
0 0
0 0
- Po - PL
0 0
lm 0
0 lm
- Pr - 2 - Pr L
or - Pt - Po
0 0
0 0
lm 0
) ·) 1 ·--
POLYNO M I A LS, POWER SERIES A N D M ATRICES 1 1.3.2
Results for Polynomial Matrices
. valid for The results for matrix operations listed in Chapter 2 rem am polynomial matrices. Here a few special results are given. ( 1 ) P( x) ( m x m )
( a) (b) (c) (d)
(e)
polynomial matrix of degree r : The degree of det P ( x ) < mr. Elementary operations on P ( x ) do not alter rk P ( x ) . Elementary operations on P ( x ) do not alter its invariant factors. P(x) is an elementary polynomial mat-r ix => P(x) is unimodular. P ( x ) is unimodular 1
1 P ( x ) - = --- P(x)a d'� det P (x)
is a polynomial matrix of finite degree. ( 2 ) r > 1 , P(x) Po + P1 x + · · · + Pr xr ( m x m ) polynomial matrix. Pr :f. 0 : P(x) is unimodular :::} det Pr = 0. ( 3 ) P(x) (m x m ) monic polynomial matrix: det P ( x ) iJ ( x ) · i111 ( x ) . where the ik ( x ) are the invariant fac tors of P(x ) . ( 4 ) P ( x ) ( m x m ) monic polynomial matrix of degree r and corresponding block companion matrix C: det ( x lr m - C) = det P ( x ) . (5) P(x) = Po + P1 x + · · · + Pr xr , Q ( x ) = Q o + Q ! x + · · · + Q , x ' ( m x m ) polynomial matrices, r > s : ( a) Q ( x ) is regular => the left ( right) quotients and remainders of P ( x ) and Q(x ) are unique. ( b) A greatest common left (right) divisor of P ( x ) and Q ( x ) is unique up to postmultiplication (premultiplication) by an arbitrary unimodular polynomial matrix. ( 6 ) P ( x ) , Q(x) ( m x m ) polynomial matrices: ( a) P ( x ) , Q ( x ) are unimodular => P(x ) Q ( x ) and Q ( x ) P ( x ) are unimodular. (b) P ( x ). , Q ( x ) are left coprime => P(x)' and Q ( x)' are right. copr i me. (c) P(x ). , Q ( x ) are right coprime => P(x)' and Q ( x ) ' are left copnme. (d) P(x ) , Q ( x ) are left coprime => there exist ( m x m ) polynomial m atrices T(x ) , S(x) such that P(x)T( x ) + Q ( x) S( x) = Im . =
=
·
·
·)•)•) -�-
H A N DBOOK O F M ATRICES
coprime =} there exist ( m x m ) polynomial such that T(x)P(x) + S( x ) Q ( x ) = !., . are left coprime =} rk [ P ( x ) Q ( x )] = m . are right coprime
(e)
P ( .r ) , Q ( .r ) are right matrices T( .r ) , S(.r)
(f) (g)
P ( x ) , Q ( .r ) P ( .r ) . Q ( .r )
:
=}
rk
[ ���� ]
=
m.
are left coprime =} there exists a unimodular polynomial matrix T(x) such that [ P ( x ) : Q ( x )] T( x ) = [Im 0] . ( i ) P ( .r ) , Q ( x ) are right coprime =} there exists a unimodular polynomial matrix T(x) such that
(h)
P(.r ) , Q ( .r )
:
•
T(x )
[
P(x) Q(x) =}
(j )
P ( .r ) , Q ( x ) [ Im : OJ .
are left coprime
(k)
P ( .r ) . Q ( x )
are right coprime =}
(I)
P(x), Q(x) .
copnme.
[ ���\ ]
] = [ lm0 ] .
[P(x) : Q ( x )]
is equi valent to
are skew prime
=}
is equivalent to
[ I� ] .
P(x), Q(x)
are left and right
( 7 ) P(.r ) , Q ( x ) ( m x m ) monic polynomial matrices of degrees r and s. respectively: P(.r ) , Q ( .r ) are skew prime -¢:::::::> the equation X ( .r ) P( x ) + Q ( x ) Y ( x ) Im has a unique solution for polynomial matrices X (.r ) and Y (.z: ) with degree X ( x ) < s and degree Y ( x ) < r . =
( 8 ) P ( .r ) , Q ( x ) ( m x m ) monic polynom ial matrices of respecti vely, C block companion matrix of P ( x ) :
skew prime Qo
-¢:::::::>
0
lm r + Q 1 0 C +
· · ·
+ Q, - 1
0
degrees r and s. P ( x ) , Q ( .r ) are
c• - 1 + Q , 0
C'
is nonsingular. ( 9 ) P(x) = Po + P1 .r , Q ( x ) = Q 0 + Q 1 x ( m x m ) polynomial matrices: ?1 and Q 1 are nonsi ngular =} P ( x ) and Q ( x ) are equivalent. ( 1 0 ) P ( x ) , Q ( x ) ( m x m ) polynomial matrices: P ( x ) , Q ( x ) are equ ivalent -¢:::::::> P( x ) and Q ( x ) have the same invariant factors. ( 1 1 ) (Smith normal ( canonical ) form) P(x) ( m x m ) polynomial matrix, rk P(.r) = r : There exist
POLYNO M I AL S , POW E R SERIES A N D MATRICES
unimodular polynomial mat.rices
U(x)
and
V(x)
223
such that. 0
P(x) = U(x) 0
ir ( X ) 0
where the
i!: ( x )
V(x)
are the invariant factors of
0 P(x).
Note: The results listed in this subsection can be found, e.g. , in Barnet t ( 1 990 ) . For the Smith normal form see also Gantmacher ( 1 959a ) .
App endix A Dictionary of Matrices and Related Terms !!!
Warning
" ' • • •
Man y t erms gzven in t h e following a re defined differe n tly in some of th e m a t rix lzt e rature. The preci5e mean ing of a given t erm in other litera t u re sh ould be checked ca refu lly in e a ch individual case .
Absolute value of a matrix: The absolute value of an ( m x n ) mat rix A = [ a ,1] is defi ned as
l A Iabs
=
[ la;j labs ] =
l a 1 1 l abs la2 1 l abs
l a 1 2 labs l a22 \abs
la in labs la2n labs
l am I l abs
\am2 l abs
\amn \abs
(m xn)
where \ c labs denotes the modulus of the complex number c = c 1 + ic2 defined as \c\ abs = Jci + c� = vee. Here c is the complex conj ugat e of c . ( For the properties of the absolut e value of a matrix see Sf'rt.ion 3.8. ) Addition of matrices: A = [a ;j] ( m x n ) , B = [b;j] ( m x n )
A+B
=
[a ;1 + b;j ] ( m x n ) .
( For the rulf's see Section 2 . 1 ) . Adjoint of a matrix: For m > 2 , the ( m x m ) matri x A a dj = [cof( a,1 )]' is t he adjoint of the ( m x m) matrix A = [a ;1 ] . Here cof( a ;j ) is t he
H A N D B O O K O F M ATRICES
'226
cofactor of a;j . For i nstance, for
[ - det a 1 2 [ a32 det a 1 2 [ an det
j .4 a d
an a32
111
= 3,
a2 1 [ a31 ] a1 3 det [ a 1 1 a 31 a 33 ] a1 3 -det [ a 1 1 a2 1 a 23 ]
a23 a33
-det
a2 1 [ ] a 31 a1 1 a13 [ -det a 31 a 33 ] a a13 [ det a2 1 a 23 ]
a 23 a33
det
II
] a12 a 32 ] a1 2 a 22 ]
a 22 a 32
For m = 1 , A a dj is defined to be 1 . ( See Section 3.4 for the properties of t he adjoint . ) Adj ugate of a matr ix: The a djoint of a manx Is somet i mes called its
adjugate.
Algebraic multiplicity of an eigenvalue: Let A 1 , . . . , An be the dis tinct eigenvalues of the ( m x m ) matrix A. Then the characteristic poly n o m z a l of
A can be represented
as
where t he rn ; are positive integer s with 2::: 7 1 m; = m. The numhf' r alue m , is the mult i plicity or algebr aic multip licity of the eigenv ( For more on eigenva lues see Chapte r 5 . ) A, , i = 1 , . . . , n.
Annihilating polynomial: A polynomial p( x ) = Po + p 1 x + · · · + Pn X " is said to annihilate the ( m x m ) matrix A if p ( A ) = p0 1m + p1 A + · · · + p, A" = Om x m · ( See Section 1 1 . 1 . 1 . ) Antimetric matrix: An ( m x m ) matrix A is antimetric or skew symmetric if A = - A ' . ( See skew-sy mmetric matrix. ) Arithmetic matrix: An ( m
m ) matrix A = [a;j ] is called ari thmetic rnat.rix if complex numbers a and d exist such that a;j = a + ( i+ j - 2 )d. For example, for m = 3,
A=
is an arithmetic matrix .
x
a + d a + 2d a a + d a + 2d a + 3d a + 2d a + 3d a + 4d
I
')•J7
DICTI O N A RY O F M AT RICES A N D RELAT E D TERMS m,
Band matrix or banded matrix: For r , s <
'
I
an ( m x m ) matrix 0 0
0 0
0
a,
��
a, 2 '
0
a• + l ,2
0
0
0
am - r + l ,m
0 0
0 0
0
a m,m - • + 1
...
with ai,j = 0 for i - j > s and for j - i > r is a band matr ix or band ed matr ix with band widt h r + s + 1 . Bidiagonal matrix: An ( m x m ) matrix A = [aij ] is bidiagonal if a;j = 0 for i > j and j > i + 1 or for i > j + 1 and j > i, that is, if aII 0
a12 an
0 0
0 a 23
A= am - l m a,n m '
0
0 or an a 21 0
0 an a 32
...
0 0 a 33
0 0 0
A = 0
0
. . '
B inary matrix: An ( m x n ) matrix is said to be binary if all elements are 0 or 1 . For instance,
[�
is a binary matrix.
1 0
1 1
�]
H A N DBOOK OF M ATRICES
228
Block band matrix: An ( m x n) matrix 0
0
0
A q - p +l . q 0 0
0
consisti ng of ( m; x nj ) submatrices A;j , i , j = l , . . . , q , is a block band matrix i f A.;j = 0 fo r i - j > s and for j - i > p. Here s , p < q. Block circulant matrix: A n ( n x n) matrix of the for m
where the A; are ( k x k ) matrices, is a block circulant matrix. Block companion mat rix of a polynomial matrix: The ( block) com panion matrix of an ( m x m) polynomial matrix P ( x ) = Po + P1 x + r - l + lm xr is t he (rm x rm) matrix · · · + Pr - I X - Pr - 2 0 lm 0
or
- PI 0 0
- Po 0 0
0
0 0 0 - P0
,..
0 0 - P1
0 0
0
lm 0 - Pr - 2
0 lm - Pr - 1
229
DICTIO N A RY OF M ATRICES A N D RELATED TERMS
or - Pr - t - Pr- 2
lm 0
-P I - Po
0 0
0 lm
...
0 0
- Po - Pt
0 lm
0 0
0 0
0 0
lm 0
0 - Pr - 2 lm - Pr - t
or lm 0
0 0
(See Section 1 1 . 3 for more details.) Block diagonal matrix: An ( x n ) matrix m
AI
0
m,
with ( x nj ) su bmatrices A; , i = 1 , . . . , p, along the main diagonal and zeros elsewhere, is a block diagonal matrix. (See Section 9 . 1 1 for properties.) Block Hankel mat rix: For given ( n x n ) matrices B; , i = 1 , . . . , 2m - 1 , the matrix
with A;j = Bi +j - t is called a block Hankel matrix. Block matrix: An ( m x n) matrix A = [Aij] , consisting of ( x j) su bmatrices A;i , i 1 , . . . , p, j = 1 , . . . , q , is a partitioned or block matrix. (See Section 9 . 1 1 for the properties.) Block Toeplitz matrix: A matrix of the form m;
=
At A2 .43
A q+ t At A2
Aq +2 Aq + t At
A 2q - 2 A 2 q- 3 A 2 q- 4
A 2q - I A 2q - 2 A 2q - 3
Aq - 1 Aq
Aq - 2 A q- t
A q-3 A q- 2
At A2
Aq + t At
=
n
= [A ;i ]
with A;J A.; + k ,J + k for all i. j, k , is a block Toeplitz matrix. Here the A ;j are ( p x p) matrices.
H A N D B O O K O F M ATRICES
230
Block triangular matrix: An ( m x m) matrix A = [A;i ] , consisting of ( m; x ni ) submatrices A;j is a block triangular matrix if A;j = 0 for i > j or if A;j = 0 for i < j . ( See also lo wer block t riangula r m a trz1·
and
uppe r block tri a ngu lar m atrix. )
Block tridiagonal matrix: An ( m x m) matri x A = [A;j ] . consisting of ( m; x nj ) submatrices A;j is a block tri diagonal matrix if A;j = 0 for I i j l abs > 1 , that is, -
A1 1 .4 2 1
A 12
0
An A. 32
0 0
0 0
A =
0 A 23 A 33
0 0 0
0 0 0
...
A p - I ,p - 1 Ap,p- 1
Ap - 1 ,p Ap,p
Bordered G ramian matrix: An ( ( m + n ) x ( m + n )) matrix
with A being a positive semidefinite ( m x m ) matrix and B being an ( m x n ) matrix , is a bordered Gramian matrix. Bordered matrix: An ( ( m + n ) x (m + p ) ) matrix
[� �]'
where A is ( m x
m), B
is ( m x p), C is
(n x
m ) , is a bordered matrix.
Brownian matrix: An ( m x m ) matrix A = [a;j ] is a Brownian matrix if
j> i>
for i, j
= 1, .
.
i
j
. , m . For i nstance, for m = 4 , A=
is a Brownian matrix.
a1 a3 a3 a3
a2 a4 a6 a6
a2 a5 a7 ag
a2 a5 as a1o
DICTI O N A RY OF M ATRICES A N D R E L AT E D TER M S
231
Centro-Hermitian matrix: An ( m x m ) matrix A = [ a ;j ] is called centro Hermitian if a ;j = a m + l - i , m + l - i • z. J = 1 , . . . , m . For exam ple, t he ( 4 x 4 ) matrix a 1 a 2 a3 a4 a5 a 6 a7 a s A= a s a7 a 6 a5 a4 a3 a 2 a t -
is centro-Hermitian. Centro-symmetric matrix: An ( m x m ) matrix A = [ a ;j ] is centrosymmetric if a;j a m+ I - i , m + l - j . I , J = 1 , . . . , m . For example. t he ( 4 x 4 ) m atrix a t a 2 a 3 a4 a5 a 6 a7 as A= a s a7 a 6 a5 a4 a 3 a 2 a t is centro-symmetri c . Characteristic determinant: The polynomial in A defi ned by t he determinant det( Alm - A ) is the characteristic determ inant or characteristic polynomial of the ( m x m ) matrix A. ( See characlerisl z c polynom ial. )
Characteristic equation: The equation det( Aim charac teristic equation of the ( m x m ) matrix for more details . )
-
.4 .
4)
.
0 is tht:> (See Sec t ion 5 . 1
Characteristic mat rix: The ( m x m) polynomial matrix P ( x ) = x Im - A is the ch aracteristic m atri x of an ( m x m) matrix A . ( See Section 5 . 1 for more details. ) Characteristic polynomial: The polynomial in A given by det ( M,., - .4 ) is the characteristic polynomial of the ( m x m ) matrix A . ( See Sect ion 5 . 1 for more details . ) Characteristic root or value: The roots of the characteristic polynomial of an ( m x m ) m atrix A are the eigenvalues , the characterist ic values. the ch aracteristic roots or the latent roots of A . ( See also ezgon•alu c of a m at rix and lat e n t roo t and refer to Ch apter 5 for det ai ls . ) Characteristic root of a matrix in the metric of another matrix: For ( m x m) matrices A and B , the roots of the polynomial p( A) = det( A B - A ) are called eigenvalues or characteristic roots of A in t he metric of B . ( See also e igenvalue of a matrzx z n the melrz c of a noth e r matrix and refer to Section 5 . 1 for further details . )
?J•)
- -
H A N DBOOK O F M ATRICES
Charac teristic vector: A n ( m x 1 ) vector v f:. 0 satisfying A v = A V , for an ( m x m) matrix A and a complex number A , is a characteristic vector or eigenvector of A corresponding to or associ ated with the characteristic value A . ( See also ezgenveclor and refer to Chapter 5 for properties . ) Charac teristic vector of a matrix i n the metric of another matrix: Given ( m x m) m atrices A and B , an ( m x 1 ) vector v f:. 0 satisfying Av = A Bv for some com p lex number A , is called a characteristic ( See also e igenvector of vector or eigenvector of A in the metri c of a matrix in the me t r·ic of another· matrix and refer to Section 5 . 1 for further details . )
B.
Check matrix: For n > m , an ( m x n ) matrix A = [Im : B] is a check matrix or parity check matrix if B is a b inary matrix, that is, all elements of B are 0 or 1 . For instance, A -
[ 01
0 1
!]
is a check matri x . C irculant matrix: For a 1 , . . . , a m E
circ( a 1 , . . . , a m ) -
al am am - I
a2 al am
a3 a2 al
a3 a2
a4 a3
a5 a4
...
am - I am - 2 am - 3
am am - I am - 2
al am
a2 a1
is a ci rculant matri x . ( See Section 9 . 1 for its properties . ) Cofactor matrix: The ( m x m ) matrix [cof( a ij ) ] is t h e cofactor matrix of the ( m x m ) matrix A = [a ;j ] . ( See also Section 3 .4 . ) Cofactor of an element of a mat rix: For an ( m x m ) matrix A = [a ,j ] . the cofactor of the ijth element a;j 1s
233
DICTI O N A RY OF M ATRICES A N D R E L ATED TERMS
where minor( a 1j ) is the determinant of the m atrix obtained by deleting the ith row and jth column from A , ...
minor( a ;j )
=
... ...
det
a ; - t ,j - 1
ai - I ,j + I
a i + l .i - 1
a i + I .J + I
a m ,j - 1
a, _ I
... .
..
anl.
•
•
m
m
( See also Section 3.4.) Cogredient matrices: The ( m x m ) matrices A and B are a perm u tation m a t rix P exists such that A = P' B P . Column dimension of a matrix:
its column dimension.
cogredient if
The number of columns of a matri x is
The column rank of a matrix A ( col rk A or col rk( A ) ) is the maximum number of linearly independent columns of A. ( See Section 4.3 for further details . )
Column rank:
Column regular matrix:
rk( A )
=
An (
m
x
n
) matrix A is column regular if
n.
) m atrices A = [a ;j ] is said to be column stochastic if A > 0 , that is, a ;i > 0, i, j = 1 , . . . , m , and I:;" 1 a ;i = 1 , j 1 , . . . , m . (See also Section 9.9.)
Column stochastic matrix:
An (
m
x m
=
Column vector:
An (
m x
1 ) matrix is a column vector.
Column vectorization of a matrix: An ( m x m ) matrix A = [a ,i ] ( m x n ) can be vectorized by stacking its columns in one vector. Notation:
vee A = vec ( A ) = col( A ) =
( mn
x
1).
H A N D BOOK OF M ATRICES
234
(See Section 7 . 2 for the rules . ) Common left divisor of polynomial matrices: A polynomial matrix D ( x ) is a common left divisor of the polynomial matri ces P(x ) and Q ( x ) if D(x ) is a left divisor of both P ( x ) and Q(x ) , that is, if polynomial matrices P1 ( x ) and Q 1 ( x ) exist such that P( x ) = D( x ) PJ ( x ) and Q ( x ) = D( x ) Q 1 ( x ) . (See Section 1 1 .3 for further
det ails. )
Common right divisor of polynomial matrices: A polynomial matrix D(x) is a common right divisor of the polynomial matrices P( x ) and Q ( x ) if D ( x ) is a right divisor of both P ( x ) and Q ( x ) , that is, if polynomial matrices P! ( x ) and Q 1 ( x ) exist such that P( x ) = Pl ( x ) D ( x ) and Q ( x ) = Q 1 ( x ) D ( x ) . ( See Section 1 1 . 3 for further
details. )
. The ( ) commutation matrix l\ mn (or n) l\m , n ) is defined such that vec( A' ) = l\. rnnvec( A ) for any ( matrix A. For example,
Commu tation matrix:
mn
x mn
m
1 0 0 0 0 0
/\32 =
0 0 I
0 0 0
0 0 0 1 0 0 0 0 1 0 0 0
0 0 0 1 0 0
x
0 0 0 0 0 1
( See Section 9 . 2 for the properties of com mutation matrices . ) ) matrices A and B commute if Commuting matrices: The ( A B = BA . m
x
m
Companion mat rix: The companion matrix p ( x ) = Po + P 1 X + + Pm - ! X m - l + x m is ·
or
·
·
of a monic polynomial the ( X ) matrix m
- Pm - 1 1 0
-Pm - 2 0 1
- p. 0 0
-po 0 0
0
0
1
0
0
1
0
0
0 0 -po
0 0 -p 1
1 0
0 1
m
23 5
TE D TE RM S DI CTI O N A RY O F M AT RI CE S A N D RE LA
or - Pm - 1 - Pm - 2
1 0
0 1
0 0
0 0
0 0
1 0
0 1
0 0 0 0
0 0
1 0 0 1
or -Pm- 2 -Pm - 1
(See also Se cti on 1 1 . 1 . )
{m x 1 ) an d ( n s for rm no be d an t · Le s: ll rm l l no a b le ll · l l tib pa Co m
vectors , res pe cti vel y. Th e no rm 1 1 · 11 for ( wit h l l · l la an d l l · l lb if
m x n
x
1)
) ma trices is com patib le
1 ) vectors x . A nor m I I · II for ) matric es A and ( for all ( ) squ are ma tric es is com pa tib le wit h II · I I a if ( m
n x
x n
m x m
) matrices A and ( for all ( detail s on no rm s.) m x m
m x
1 ) vectors
x.
(See Ch apt er 8 for
matrix A = [a;j ] , the Co mp lex co nju ga te ma tri x: F�r an ( m x n ) all ele me nts com ple x conjug ate ma trix A is obt ain ed by rep lac ing (Se Sec tio n e ]. [ii;j = A is, t tha , ii;j s ate jug con x ple com ir the by a;j 3 . 2 for the pro per ties.)
x matrix if ple com a is m x n ) matrix A = [a;j ] ( e Th x: tri ma lex Co mp the a;j are com ple x num ber s. of A if b;j is a mi nor Co mp ou nd ma tri x: B = [b;j ] is a com pou nd ma trix ], of a given size of A . For examp le, for a {3 x 3) ma trix A = [a;j
[ al l [ det a 31 a det [ a 2 1 31
det B=
aI I a21
[ ] al l al2 [ det a 31 a3 2 ] a an det [ a 21 31 a 32 ]
al2 an
is a com po und matrix of A .
det
aI I a2 1
a12 [ an ] a a13 det [ a 1 2 32 a 33 ] a a 23 det [ a n 32 a 33 ]
a13 a 23
d et
] a 13 a 33 ]
a 13 a 23
a 23 a 33
]
H A N DBOOK O F M ATRICES
236
Condition number: The condition number of an (m x n ) matrix A is the ratio 0'm ax / 0'min , where 0'm a r and 0'min denote the largest and smallest singular values of A , respectively. {See Chapter 5 for details
on singular values .)
Conditional inverse matrix: A generalized inverse matrix is sometimes
called conditional inverse matrix . (See
generalized inverse m a t rix. )
Conformable matrices: The matrices A and B are conformable if their product A B is defined , that is, if the number of columns of A is equal to the number of rows of B. Congruent matrices: The ( m x m) matrices A and B are congruent if a nonsingular ( rn x m) matrix C exists such that B = CH A C. {See
Chapter 6 for some congruent matrices.)
Conjugate matrix: The ( m x n ) matrix A = [a;1] is the conjugate or complex cony u g a t e of the { m x n ) matrix A = [a;j ] , where a;j is the complex conj ugate of aij · (See Section 3 . 2 for the properties. ) Conjugate transpose matrix: The conj ugate transpose of the { m x n ) matrix A = [a;j ] is the ( n x m) matrix A H - A' = [a;j ] ' . ( See Section 3 . 3 for the properties and furt her details .) Conjunctive matrices: The ( m x m ) matrices A and B are said to be conjunctive if a unitary ( m x m) matrix U exists such that A = U H B U . (See Chapter 6 for some conjunctive matrices.) Controllability matrix: For an ( m x m) matrix A and an ( m x n ) matrix B the corresponding ( 111 x mn) controllability matrix is defi n ed as C
=
[B : A B :
·
·
·
:
A m I B] . -
Convergent matrix: A n { m x m) square matrix A is convergent or stable if A n Om x m for n __, x . (See Sect ion 9 . 3 for the properties. ) -
Cosine of a matrix: The cosine of an ( m x m ) square matrix A is defined as
ros( A )
( See Section
=
f: (-2n.1 t A 2n . n=O
1 1 . 1 for more details . )
D-inverse: See Dra::zn m t•erse.
DICTIONARY O F M AT RICES A N D RELATED TERMS
�
Decomposable matrix: An ( m x m ) m atrix A is said to be decomp osable or reducible i f t here exists a permutation matrix P such that P' A P
=
[ AA 21 11
where A 1 1 is ( k x k ) , A 22 is ( ( m - k ) x ( m - k) ) and A 2 1 is ( ( m - k ) x k ) . Defective matrix: A n ( m x m ) m atrix A is called defective if i t h as fewer t han m linearly independent eigenvectors . Deficient matrix: Alternative expression for defectwe m a t rix. Definite matrix: A matrix is definite if i t is p ositive definite or negative definite. ( See also positive defin ite m a t rzx and negat ire defin z t e m a trzx and refer to Sec tion 9 . 1 2 for the p roperties . ) Degree of a polynomial matrix: The degree of a n ( m x n ) p olynomial m atrix P ( x ) = [p;j {x) ] with typical elements Pij ( x ) Pi; . o + Pij, ! X + · · · + Pij ,r ,1 X r '1 , is m ax;,j rij , w here �"ij is the degree of Pij ( x ) , i = l , . . . , m, j = l , . . . , n . Equivalen t ly, t he degree of P ( x ) = Po + P1 x + · · + Pr x r , is r if Pr :f. 0 . ( See Section I I .3 for details on polynomial m atrices . ) =
·
Dependent vectors : See linea rly depe ndent vectors. Derivative matrix: Let A ( x ) = [a;j ( x )] be an ( m x n ) matrix w hose elements are differentiable functions a;j : IR ----+ IR. The mat rix of first order derivatives d a ;j dA = dx dx
[ ]
is called t he derivative m atrix of A ( x ) . ( See Sections 1 0 . 1 and 1 0 . 9 for details.) Derived norm: If < > is an inner product for ( m x n ) m atrices, 1 I I A I I = < A , A > / 2 is a norm for ( m x n ) m atrices. I I · I I is said to be a n orm derived from the i nner product < · > . (See C'hapter 8 for details. ) ·
,
·
·.
Derogatory matrix: An { m x m) m atrix A is derogatory if a p olynomial p(r) = Po + PI X + · · · + Pr X r of degree r < m exists such that
p( A ) = Pa lm + P I A + · · · + Pr Ar = 0 . (See Sect ion 1 1 . 1 . )
'
') 3-
H A N DB OOK O F M ATRICES
238
Determinant of a matrix: The determinant of the (m
A = [a ;j ] is defined
as
x
m
)
matrix
where the sum is taken over all products consisting of precisely one element from each row and each column of A multiplied by - 1 or 1 , if the permutation i1 , . . . , im is odd or even, respectively. ( See Section 4 . 2 for the properties. ) Diagonal dominant matrix: An ( m
dominant if
x m
) matrix A
=
[a ij] is diagonal
m
l a ;;lats > L i a ;j l ats for i = 1 , 2 , . . . , m . J= l
} o# •
For example, the matrix -5 1 5
1 3 -5
2 1 11
is diagonal dominant. Diagonal matrix: An ( m x m ) matrix
with a ;j = 0 for i :f. j is a diagonal matrix. ( See Section 9.4 for the properties . ) Diagonal of a matrix: A diagonal of an ( m x m ) matrix A = [a;j] consists of all elements a;j for which i - j equals a given integer . Diagonal similar matrix: An ( m x m) matrix A is diagonal similar if it is similar to a diagonal matrix, that is, if there exists a nonsingular ( m x m) matrix P such that P A p - I is diagonal . {See also diagonalizable m atnx
and Chapter 6 . )
Diagonalizable matrix: A n ( m x m) matrix A is diagonalizable if a nonsingular ( m x m ) m atrix P exists such that PAp- 1 is a diagonal
matrix. (See Chapter 6 for conditions. )
239
DICTION A RY O F M ATRICES A N D RELATED TERMS
Dimension of a matrix: ( m
n) is the dimension or with m rows and n columns.
order
x
of a matrix
Direct product : The direct product or Kronecke r p rodu c t or tensor p rodu ct of two matrices A = [a;i] ( m x n) and B = [b;i ] (p x q )
.
IS
( mp
x
nq ) .
( See Section 2 . 4 for the rules . ) Direct sum: The direct sum of two matrices
B [b;i] ( n =
x
n ) is
] �
[
A EB B = Ao
(See Section 2 . 6 for rules . )
((m + n)
A
[a ij ] (
x
(m + n)).
m
x
m) and
m ) matrix B = [b;j ]. a nonnegative ( m m ) matrix A = [ a;j ] is said to dominate B if lbij lats < a;; , i, j = 1 , . . . , m, that is, IBi ats < A. (See Section 9 . 9 for further details. )
Dominating nonnegative matrix: Given an ( m
x
x
Dominating principal diagonal: An ( m x m ) matrix A. = dominating principal diagonal if d; E ffi, d; > 0 , i = 1 , .
such that
d; l ajj labs >
For instance,
m
L d; l a j lat s. ;
A=
[�
j
=
1, . .
[a;;] has a . . exist , m,
. , m.
-� ]
has a dominating principal diagonal . Doubly centered matrix: An ( m x m) matrix
centered if
and
m
L a;; i= l m
=
0
for
.= J
A = [ a;i] is called doubly
1 , . . . , 1n
. = 1, . . . for t = 0 a i L ; i=l
, m.
240
HAN DBOOK O F M ATRICES For ex a m pl e , the matrix 2
2
I
1
I
1 I
I
2
I
1
2 I
2
2
is doubly centered. Doubly quasi-stochastic matrix: A real ( m x m) m atrix A = [a,;] is : 1 a;j 1 for i = 1 , . . . , m and called doubly quasi-stochastic if 2:; 2::: 7' 1 a;j = I fo r j = I . . . , m . ( See also doubly stochastzc matrix and refer t o Sect ion 9 . 9 for further details. ) =
.
Doubly stochast ic matrix: A n ( m x m) matrix .4 [a;j] is said to be 1 , . . . , m and doubly stochastic if A > 0 , t h at is, a;; > 0 , i. j I:j" 1 a , j = I . i l , . . . , m and 2::: 7' 1 a;j = 1 , j = I , . . . , m . For example, t he ( 1 1 1 x m ) matrix =
=
I
I
m
m
I
m
l
m
is doubly stochast i c . ( See Sec tion 9 . 9 for further details. ) Drazin inverse: Let. A b e an ( m x m ) mat r i x w ith index r , that is, r i s t he smallest nu mber such t hat r k ( A r ) = r k ( Ar + l ) . The Drazin inverse o r D-ln t·o·se A 0 of A is t.he u nique solut ion of the set of equations ,4r + 1 A D A D A. A D AAD
( See also ge n e ral1:ed in t•erse. ) �
D uplicat ion matrix: An ( 1W X � m( m + 1 ) ) vee( A ) = Dm vech ( A ) . for any sym metric ( m a d u pli cat ion ma t r i x . Fo r ('Xample,
x
I 0 0 0 0 0
D:�
=
0 0 0
()
0 0 0 0
I 0
0
0 0 0 0 0
0 0
1
I 0 I
0 0
0 0 0 1
0 0 0 0
0 0 0 0
I
0 0 0 0 0 0 0
0
1
I
0
atr i x Dm satisfying m ) matrix A , is call('d
m
DICTI O N A RY OF M ATR ICES A N D R E L ATED TER�1 S
9.5
is a d u p l i cat ion m a t r i x . (See Sec t ion
Dyadic produc t:
24 1
for t h t> propert i t>s . )
For vect ors
(m x 1 ) ,
X =
Yl
y =
X rn
(n
x
1)
Yn
t he p ro d u c t xy
I
=
Xm Y l
is c a l l e d t h e dya d i c pro d u c t .
) m a t ri x A = [ a ;j] i s i n echelon form i f for any or t h ere exist s a A· E { 1 . . . . , n } row i . eit her a ;j = O . j = 1 , . 71 such t h a t a , �.. = I and a ;j = 0 for j < J.· and a t k = 0 for I =/:- i . For exaniplt>.
Echelon form:
A n ( 111
x 1l
.
0 0 0 0
0 0 0 0
is
in
ec h e lon
.
,
,
0 1
0 0
0 0
0 a25
1
a35
0
0
form . ( See a l so Section 6 . 1 . 4 . )
Effect ive inverse matrix: A
genera l i z e d i n verse m a t rix is some t i l l les
c a l lt>d an effe c t i v e i n verse m a t r i x . (St>e
g e n f ralt�ed i n t• c r.sf m a t ru. )
Eigenvalue of a matrix: The roots of t he c h a rac t e r i s t ic pol y no m i al of an ( m x m ) m a t ri x A . t h at i s . the roots of p( ,\ ) = d e t ( Um - A ) . are tlw eigenval ues o f
refer
A.
( See a l so
to C' h a p t er 5 for det a i l s . )
rh a ra c 1 E n8tic raha or late n t roo t a n d
Eigenvalue of a mat rix in the metric of another matrix: (; i w n two ( m x m ) m a t rices A a n d B , t h e roots o f t he poly u omial p( ,\ ) = A. ) are c a l led Pigenval ut>s or ch aracterist i c root s of A i n t he m etric of H . ( See Sec t ion 5 . 1 for fu r t lwr det a i ls . )
det( -X B -
Eigenval ue of a polynomial matrix:
tlw roots of t lw p ol y llOlllial dt>t P ( l' ) .
a polynomial 1 n a t rix P ( x ) are
Eigenvector matrix:
Let
A
i n dependent eigenvectors .
be
t· 1
. m . T h e n V = [ t• 1 o r m odal m a ln:r o f A .
i
= 1, .
.
•
an
.
.
.
.
The eigenva l u es or l a t Pn t root s of
.
.
(m
I'm
m ) m a t rix w i t h l i n e a r l y of length I . t h at i s . t-{1 t·, = I for x
. . , l'm] is c al led a n rigenvrc t or m a t r i x
H A N DBOOK O F M ATRICES
242
Eigenvector of a matrix: An ( m x 1 ) vector v # 0 satisfying A v = ,\ t• for an ( m x m ) matrix A and a complex number ,\ is an eigenvector or c h a ra ct e ri s t i c vector of A corresponding to or associated with the eigenvalue value ,\ . (See Chapter 5 for details . ) Eigenvector of a matrix in the metric of another matrix: Given two ( m x m ) matrices A and B, an ( m x 1 ) vector t ' # 0 satisfying
,\ Bv for some complex number ,\, is called an eigenvector or characteristic vector of A in the metric of B. ( See Section 5 . 1 for further details. ) Av
=
Elementary divisors of a polynomial matrix: Let ,\ be a root of an
of a polynomial matrix P ( x ) . Then the linear polynomial p( x ) = x - ,\ is an elementary divisor of P(x ) . ( See Section 1 1 . 3 for more details.) z n v a ri a n t
fa c t o r
Elementary matrix: An ( m
)
matrix is called elementary if it is obtained by applying a single e le m e n t a ry ma trix opera t w n to the ( m x ) identity matrix Im . x
m
m
Elementary matrix operations: The following modifications of a matrix
are called elementary operations: ( i) (ii) (iii ) ( iv )
interchanging two rows or two columns, multiplying any row or column by a nonzero number, adding a multiple of one row to another row , adding a multiple of one column to another column.
Elementary operations for polynomial matrices: The following
modifications of a polynomial matrix are called elementary operations: ( i) interchanging two rows or two columns, (ii) multiplying any row or column by a nonzero number, (iii ) adding to one row another row multiplied by an arbitrary polynomial, (iv) adding to one column another column multiplied by an arbitrary polynomial . (See Section
1 1 .3
for details on polynomial matrices.)
Elementary polynomial matrix: An (m x m) polynomial matrix is
elemen tary if it may be obtained by applying a single
e le m e n t a ry
243
DICTI O N A RY O F M ATRICES A N D REL ATED TERMS polynomial m a t rix ope ration
polynomial matrices .)
to Im . (See Section 1 1 . 3 for details on
Elementwise product: See Hada mard p roduct . Elimination matrix: A ( ; m ( m + 1 ) x m 2 ) matrix L m satisfying vech( A ) = L m vec { A ) , for any ( m x m ) matrix A , is called an elimination
matrix . For example,
L3 =
1 0 0 0 0 0
0 1 0 0 0 0
0 0 1 0 0 0
0 0 0 0 0 0
0 0 0 1 0 0
0 0 0 0 1 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 1
is an elimi nation matrix . (See Section 9 . 6 for the properties.) EP-matrix: An ( m x m ) matrix A is said to be an EP-matrix if it commutes with its Moore - Penrose inverse A + , that is, A A + = A+ A . (See Section 3 . 6 . 2 for details on the Moore - Penrose inverse. ) Equality of matrices: Two ( m x n ) matrices A = [a;j ] and B = [b;1 ] are equal if a;j = b;j for i = 1 , . . . , m , j = 1 , . . . , n . Equivalent matrices: The ( m x n) matrices A and B are equivalent if nonsingular matrices C ( m x m) and D ( n x n ) exist such that B = C A D. ( For examples see Chapter 6 . ) Equivalent polynomial matrices: Two polynomial matrices P ( x ) and Q( x) are said to be equivalent if u n imodular polynomial matrices U ( x ) and V ( x ) exist such that P ( x ) = U ( x) Q ( x ) V ( x ) . {See Section 1 1 .3 . ) Ergodic matrix: An ( m x m ) matrix A = [a;j ] is said to be ergodic i f it is stoch astic { that is, a;j > 0 , i, j = 1 , . . . , m, and I: m 1 a;i 1 , i 1 is the only eigenvalue with modui us 1 . Moreover . 1 , . . . , m ) and ,\ if the multiplicity of ,\ = 1 is k , then there exist k linearly independent eigenvectors associated with ,\. (See also Section 9.9. ) =
=
Exchange matrix: The ( m x m) matrix 0 0 1
0 1 0
1 0 0
=
244
H A N DBOOK OF M ATRICES w
i th
for i + j = m + otherwise
is called an exc hange matrix,
1
re verse u n z t matrix or fizp ma trix.
Exponential function of a matrix: The t>xponential function of au
( m x m ) matrix A is defined
as
=
exp( A)
00
1
n. A Ln .1
n =O
Alternatively, t his expression is called the ( See Section l l . l . )
exponen tial m atrix
of A .
Exponential matrix: See expo n e ntial fu n ction of a ma trix. Flip matrix: A u e;uh a nge m a trix is sometimes called a flip matrix. Fourier matrix: An ( m x m) matrix A = [a;j ] is a Fourier matrix if
l w l i - l )l j - 1 ) Fn with w =
exp( - 2 7r i/ m ) . For PXarnple. for m = l J3
l
1
1
3.
1
1
is a Fourier matrix. Frobenius matrix: A u (m x m) matrix A
[a;j ] is a Frobenius matrix
=
if one column only cont ains elements which are neither zero nor OllP and an identity matrix may be obtained by eliminating one row and one column. For inst auce , l
(12 l
0
l
0 0
l arc Frobenius 111atrices.
and
1
0 1
a 2rn
0
0
a mtn
0
245
DICTI O N A RY O F M ATRICES A N D RELAT E D TERMS
g-circulant matrix: An (m x m) matrix a2 am - g + 2 a m - 2g +2
where the last g elements of a row are the first elements of the next row , is a g-circulant matrix . For example, a ( 4 x 4 ) 2-circulant matrix is of the form a1 a3 a1 a3
(See also
a3 a1 a3 a1
a2 a4 a2 a4
a4 a? a4 a2 •
c irc ula n t m a trix. )
g-inverse matrix: Short for g e n e rali::ed in l;ersc
m
atrir.
Generalized characteristic root or value: An e igon•alue of a m a t n2·
is sometimes cal led a generalized characteristic root or generalized characteristic value. in th e m e t rir of a n ot h e r m a t rix
Generalized eigenvalue: An e igen value of a m atrix in th e m e t ric of
is sometimes called a generalized eigenvalue. Generalized eigenvector of a matrix: Let A be an ( m x m ) matrix and k E IN . k > 1 . An ( m x 1 ) vector x is a generalized eigenvector of grade k or a prw cipal 1·ector of gra de k corresponding to an eigenvalue ,\ if ( Um - A ) k x 0 and ( Mm - A ) k - l .r =/: 0 . Alternatively, an e igen vector of a m atrix i n t h e m e trir of a n oth f r m a tnx is sometimes called a generalized eigenvector. Generalized inverse matrix: An ( n x m ) mat rix A - is a general ized inverse of the ( m x n ) matrix A if it satisfies A A - A = A. ( See S<'ct ion 3 . 6 for the properties. ) an other m a t rix
=
Generalized permutation matrix: An ( m x m ) mat rix wit h at most one
nonzero element. in each row and each column and zeros elsewhere said to be a generalized permutation matrix. For examplt', 0 b 0
a 0 0
1s
0 0 c
is a generalized permutation matrix. (Set:' also
pe rmu tation m a t rix. )
246
H A N DBOOK O F M AT RI C ES
Geometric matrix: An ( m x m ) matrix A = [ ;j) i s cal led a geometric matrix if a 1 1 = 1 and a ;j = >, i +i - 2 for i + j a> 2. Here >. is a complex number. For example, for m = 3 ,
>.
1
>.
A=
>, 2 >, 3
>,2
>, 2 >, 3 >, 4
is a geometric matrix. Geometric multiplicity of an eigenvalue: The geometric multiplicity of an eigenvalue >. of an ( m x m) matrix A is the number of blocks of
the form
1
>. 0
>.
0
0
0
0 0
1
>. with >. on the principal diagonal , in the Jordan decomposition of A . (See Section 6. 1 . ) 0
Givens matrix: For
E IN ,
r, s
0
r
<
s
< m , an ( m X m ) matrix Gr• =
[Yij]
is said to be a Gi vens matrix or rotation matrzx if Yrr = g,. = cos (), Yii = 1 for i = 1 , . . . , m, i =/: r, s , Y•r = - s i n 8 , Yr• = s in 8 and all other elements are zero. For example, cos 0
(J
- sin
(}
0 I
0
sin ()
0 cos
()
cos (J
- sin ()
sin
8
cos (J
]
are Givens matrices. Gradient matrix: The ( m x n ) matrix of first order partial derivati ves of the vector valued function y : U C IRm ---. IR" , oy' OX
is the gradient matrix of y( x ) = ( y i ( x ) , . . . , Yn ( x ) ) ' , where x ( x 1 , . . . , X m ) ' is an ( m x 1 ) vector. (See Chapter 1 0 for more details.) =
247
DICTI O N A RY O F M ATRICES A N D R EL ATED TERMS
Gradient vector: The ( m x 1 ) vector of first order partial derivatives of the scalar valued function f : U C IRm - IR, of ax
,
of OXm
is the gradient vector of f ( x ) , where x ( x 1 , vector. (See Chapter 1 0 for more details. ) =
•
•
.
, x m )'
is an ( m x 1 )
Gram matrix: For an ( m x m ) matrix A , G = A H A is the corresponding
G ram matrix. ( See also
Gramian m atrix. )
Gramian matrix: An ( m x m ) matrix A is a Gramian or positive semidefinite matrix if it is Herm itian and xH Ax > 0 for all ( m x 1 ) vectors x . (See also positive s e m idefinite m atrzx and refer to Section 9 . 1 2 for the properties . ) Greatest common left divisor: A polynomial matrix G(x) is a greatest common left divisor of the poly nomial mat rices P( x ) and Q ( x ) if G(x) is a common left di v isor of b ot h P ( x ) and Q ( x ) (that is, P ( x ) = G(x)P1 ( x ) and Q ( x ) = G(x ) Q 1 ( x ) for suitable mat rix polynomials P1 ( x ) and Q 1 ( x ) ) and for any other common left divisor D ( x ) t here exists a polynomial matrix B ( x ) such that G ( x ) = D ( x ) B(x ) . ( See
Section 1 1 .3 . )
Greatest common right divisor: A polynomial matrix G(x ) is a greatest common right divisor of the polynomial matrices P ( x ) and Q(x) if G(x) is a com m on right divisor of both P ( x ) and Q(x) (that is, P ( x ) = P1 ( x )G( x ) and Q( x ) Q 1 (x ) G ( x ) for suitable matrix polynomials P1 ( x ) and Q 1 ( x ) ) and for any other common right divisor D( x ) there exists a polynomial matrix B ( x ) such that G (x ) = B ( x ) D ( x ) . (See Section 1 1 .3 . ) =
Group inverse of a matrix: I f the ( m x m ) matrix A satisfies rk(A) rk ( A 2 ) . the matrix A # satisfying A 2 A # = A , A # AA# = .4 # and AA# = A # A is called the group inverse of A . (See also g e n e rali::: e d =
inverse m a t rix. )
0, 1 , . . . , is a ( 2 k x 2 k ) matrix which has only elements 1 and - 1 and which is obtained recursively as
Hadamard matrix: A Hadamard matrix H k , k
] [�
=
k = I , 2, . . .
,
H A N D BO O K O F M AT RICES
248
starting with
Ho
= l.
For instance, 1 1 1 1 1 -1 1 -1 1 1 -1 -1 I 1 -1 -1
H2 =
Alternatively, a matrix with dominating princi pal diagonal is Hadamard matrix. ( See dominating pnncipal sometimes called a
diago n a l m a t rix. )
Hadamard product : The Hadamard product or Sch u r product or e le m e n twise product of the two matrices A = [a;j ] ( m x n ) and B = [b;1 ] ( m x n ) is defined as
A (:}
B
=
[a;j b,j ] ( m x n ) .
(See Section 2 . 5 for the rules.) Half-vectorization operator: For a symmetric ( m x m ) matrix A = [aij )
the half-vec torization operator vech stacks all columns from the princi pal diagonal downwards, t hat is,
vech A
( See Section
7.3
=
vech( A ) =
( � m( m + 1 ) x 1 ).
for the rules . )
Hamiltonian matrix: A ( 2 m x 2 m ) matrix H is a Hamiltonian matrix if - Im 0
For instance, for matrices
] [ Im0 H
'
A ( m x m ) , B (m x - B B'
A'
is a Hamiltonian matrix .
- fm 0
]
].
n) and C (r x m ) ,
DICTIONA RY O F M ATRICES A N D R E L ATED TERMS
Hankel matrix: For
C1 1 ,
. . . , Cl2m- l E
an (
m
x
m
249
) matrix of the form
with a ;1 = Cli +j - l is a Hankel matrix. Harmonic matrix: An ( x ) matrix A = [a; j ) is called a harmonic m at rix i f a ;j = a +(i +IJ _ 2)d for complex numbers a and d such that the denominators are nonzero. For example, for = 3 , m
m
m
I
I
a
a+d
a+2d
a+d
a+2d
a+3d
a +2d
a+3d
a+4d
I
I
A=
I
is a harmonic matrix. (See
I
I
I
I
for a special case.)
Hilbert matrix
Helmert mat rix: An ( m x m) matrix of the form m_
.l. 2
m
') - .l. 2
-
�
2
_ .l.
m
2
') _ .l. �
2
_ .l.
0 -2 . 6- t
6- t
6- t
1 [m ( m - 1 )] !
1 [m(m - 1 )] !
is called a Helmert matrix. For instance, for I
7j I
72 I
is a Helmert matrix.
76
I
a I]·
I
2
76
canonical form if
0 or 1 0 0 0
7j
- 72
Hermite canonical form: An ( m x
I
7j
I
m
2
_ .l.
m
2
_ .l.
0 0
0 0
1 [m ( m - 1 )] !
-(m - 1 ) [m(m - 1 )] !
m
=
3,
0
- 76
) matrix A
[ a;j] IS Ill Her Illite
if i = j if i > j for j = 1 , . . . , m , i =j: j, if a ;; = O for i 1 , . . . . i =/: j. if aJ J = 1 =
m
m,
H A N D BOOK O F M ATRICES
250
For exam ple,
1 0 0 0 0
a12 0 0 0 0
0 0 1 0 0
a14 0 a 34 0 0
a15 0 a 35 0 0
is in Hermite canonical form . (See also Section 6 1 4 ) .
.
.
Hermitian adjoint : Alternative expression for conjugate transpos e . Hermitian form: G iven a Herm itian ( m x m ) matrix A , the function Q :
matrix A = [a;j J with a;j = 6.1 ; is a Hermit ian 11 1at rix. Here a1 ; denotes the complex conjugate of aii · In other words, A is Hermitian if .4' = A H = A . (See Section 9 . 7 for the properties and more details .)
Hermitian matrix: An ( m x
m)
Hessenberg matrix: An ( m x m ) matrix A = [a;j ] is a Hessenberg matrix or npper Ilessenherg matrix if a;j = 0 for i > j + 1 , that is, a1 1 a21 0
a12 a ?? a 32
a13 a 23 a 33
a 1 rn - 1 a 2 rn - 1 a3 m - 1
a 1 .rn a 2 rn a3, m
0 0
0 0
0 0
a m - l ,m - 1 a rn,rn - 1
am - l m
- ·
'
'
'
'
A= '
(See also Section 6 .2 . 3 . ) Hessian matrix: The
IS
(m x m ) matrix of second order partial derivatives
the Hessian matrix of the scalar valued function f( x ) , where .r: = ( .r: 1 , . . . ,.r:,. ) 1 is an ( m x I ) vector. (See Chapter I 0 for more details.)
DICTIONARY OF M ATRICES A N D R E L AT E D TERMS
251
Hilbert matrix: The ( m x m ) matrix A = [ a ,j ) is a Hilbert matrix if a ij = 1 / ( i + j - 1 ) , that is, if I
1 I
2 I 3
I
m+l
2
A =
I m I m+l
m
I
I
2m- !
Householder t ransformation: The Householder transformation of real ( m x 1 ) vector x =/: 0 is the ( m x m) matrix
a
2xx' H = lm x'x
The Householder transformation of a complex ( m . IS
x 1)
vector
x
=j: 0
2xx H H = lm - H . X X
Hurwitz matrix: The Hurwitz matrix corresponding to the polynomial P ( x ) = Po + PI x + · · · + Pm x m is
H=
where p;
=0
Pm - 1 Pm 0 0
Pm -3 Pm - 2 Pm- 1 Pm 0
0
0
0
for i
< 0.
Pm- 5 Pm -4 Pm-3 Pm- 2 Pm - 1
P-m+l P-m+2 P- m+3 P-m+4 P-m+5
(m x m)
Po
For example,
H=
P3 P4 0 0
Pt P2 P3 P4
0 Po Pt P2
0 0 0 Po
is the H urwitz matrix corresponding to p ( x ) p3 x 3 + P4 x 4 . (See also Section 1 1 . 1 . )
= Po + P t X + p2 x 2 +
Idempotent matrix: An ( m x m) matrix A is idempotent if A2 = A . For
example,
is idempotent. (See Section 9 . 8 for the properties. )
•) ; ·) -v
H A N DBOOK OF M ATRICES
Identity matrix: A n ( m x m) matrix 1 lm
0 =
=
0
with a ; ; = 1 for i = 1 , . . . , m and a,j unit matrix. ( See also u n z i matrix. )
[a;j ]
1 0
for
i =/: j
is an identity or
Imaginary matrix: A ( m x n ) is an imaginary matrix if it can be
represented as A = iB, where B is a real matrix and i = ..;=T. that is, all elements of A are imaginary numbers or zero.
1mprimitive matrix: A n irreduc zble ( m x m ) matrix A is said to be
imprimitive if it has more t han one eigenvalue whose modulus is equal to the spectral radiu8 of A .
Indefinite matrix: A Hermitian ( m x m) matrix A is indefinite if it is not definite. I n other words, A is indefinite if ( m x 1 ) vectors .r 1 and .r2 exist with .rf A .r 1 > 0 and .r� A.r2 < 0 . Independent vectors: See linearly independent vectors. Index of a square matrix: The sm allest non uegative integer i such that rk( A' ) = rk( A i + 1 ) is t he index of the square matrix A . Induced matrix norm: Given a vector n o rm I I · I I for ( m x 1 ) vectors, I I A I I 1ub
= sup
_
{
I I A .r l l .. x ( m x 1 ) , .r =/: 0 llxll
}
is the matrix norm for ( m x m ) matrices induced by I I · I I · Alternat ively it is called the sup norm or ope rator u orm or lub ( least upper bou u d) u orm . ( For more details O i l norms see Chapter 8 . ) Inertia of a square matrix: The inertia of an ( m x m ) matrix A. is t he t riple of integers ( a , b, c ) , where a , b , c are t he nu mbers of eigenvalues
( counted with their algebraic m ultiplicz iies) with positive real part . negative real part. and zero real part, respectively. ( See also Sert io l l
5. 1 . )
Inner product: A fu nc t ion < > attaching a complex number < A , B > to any two ( m x n ) matrices A , B is called an inner product if for any arbitrary ( m x n ) matrices A , B , C and c E ([' the following conditions · ,
·
are satisfied : ( i)
<
A,
A> >
0
if
A =/: 0 ,
253
D I CTION A RY OF M AT RICES A N D RELATED T E R M S
(ii) < A + B . C > = < A , C > + < B. C > . (iii) < cA , B > = c < A , B > , (iv) < A , B > = < B, A > . ( See Chapter 8 for more details. ) Inner product over the real numbers: A function <
· ,
·
> is an inrwr
product over the field of real numbers if it attaches a real number to a pair of real matrices in such a way that for any real ( m x n ) matrices A , B, C and c E IR the following conditions are satisfied: (i) (ii) (iii) (iv)
< A, A > > 0 < A + B,
C>
< cA , B > < A, B >
(See Chapter
8
=
=
if A =/: 0,
=
< A , C > + < B, C > ,
c < A, B > , < B, A > .
for further details.)
Integer matrix: A = [a;1 ) ( m x n ) i s an integer matrix if all elements are integers, that is, a;j E 7L . Integral matrix: Let .4 ( 1 ) = [a;j ( l )) be an ( m x n ) matrix of int egrable functions. Then J A dt - [f a;j ( t )dt] is the integral mat rix of A( f ) . Invariant factors of a polynomial matrix: The invariant fact ors or invariant polynomials of an (m x m ) polynomial mat rix P( .r: ) of rank r
are defined
as
ik (.r:) =
/k -k (tx( x) ) ,
k = 1,
. . . , r,
where do ( x ) = 1 and dk ( x) is the monic greatest common divisor of all minors of order k of P( x ) for k = 1 , . . . , r. (See Section I I . I . I for more details . ) Invariant polynomials of a polynomial matrix: See invana tJ i fa ctors of a polynomial m a t rix.
Inverse matrix: A n (m x m ) matrix A - 1 is t he inwrst" of the ( m x m ) matrix A if A A - 1 = A - I A = Im . (See Sect ion 3 . 5 for the propert ies
and further details. )
Invertible matrix: A n ( m x m ) matrix A i s invert ible or n on singula r or regular if det(A ) =/: 0 and t hus .4-I exists. (See Sect ion 3 . 5 for the
proper! ies . )
254
H A N DBOOK O F M ATRICES
Invertible polynomial matrix: The ( m x m ) poly nomial matrix P(x) is said to be u n i m odular or inverti ble if det P ( x ) = constant =/: 0. that is, det P ( x ) is a constant function. For instance, for a E IR, P ( x ) = /2 +
[� �]
has determinant det P( x ) = 1 and, hence , Section 1 1 .3 for more details.)
x
P(x)
is unimodular . (See
Involutory matrix: An ( m x m ) matrix A = [a;1] is called involutory or un 1po l e n t if A. 2 = Im . For exam ple,
[ cossin ()()
() ]
sin - cos ()
is involutory. Irreducible matrix: An ( m x m) matrix A which is not reducible is called i rreducible, that is, A is irreducible if there does not exist a perm u t ation m a t rix P such that P' A P =
where A 1 1 is (k x k),
A22
[
A11 A2 1
Ok x ( m - k ) A22
]
is ( ( m - k) x ( m - k ) ) and A2 1 is ( ( m - k ) x k ) .
Isometry for a norm: An ( m x m ) matrix A i s called an isometry for a norm 11 · 1 1 for (m x 1 ) vectors if I I A x l l = l l x l l for all ( m x 1 ) vectors .r . (See Chapter 8 . ) Jacobian matrix: The ( n x m ) matrix of first order partial derivatives {)yl
{)yl
{) x 1
{) xm
{) y {)x '
is the J acobian matrix of the ( n x 1 ) vector function y ( x) = ( yi ( x ) . . . , Yn ( .r ) ) ' , where .r = ( .r 1 . . . , x m )' is an (m x 1 ) vector. ( See Chapter ,
,
10.)
Jordan canonical form or Jordan normal form: A block diagonal
matrix
255
DICTION A RY O F M ATRICES A N D R E L AT E D T E R M S
with
0 1
0 0
...
1 -\;
A; = ...
0 i
=
L
0
. . . , p , is a Jordan canonical form or Jordan normal form or
Jordan matrzx.
For example, 2 0 0
A=
1 2 0
0 1 2
0 [2]
0
[ 30.2
1 3.2
]
is a Jordan canonical form with 2 0 0
1 2 0
0 1 2
A3 -
[ 30. 2
(See Chapter 6 for more details.) Jordan matrix: A Jordan ca n o nical form is sometimes called a Jordan
matrix.
Kronecker matrix: An ( m x m ) matrix A = [a;j] with 0 c 1
a;j =
for i :j:. j for i = j = k for i = j :j:. k
for some k E { 1 . . . . , m } , is a Kronecker matrix. For instance,
is a Kronecker matrix.
1 0 0 0
0
0 0 1
0 0 0
0
0
1
0
c
H A N DBOOK OF M ATRICES
256
Kronecker product : The Kronecker p roduct of two mat rices [aij ] ( m x 11 ) and B = [b,j ] ( p x q ) is defined as -
4 "" B �
, .
-
( mp
.4.
=
x nq).
Um ) B ( See also dzrect p rodu c t or t e n s o r p roduct and refer to Sect ion 2.4 for t he rules . )
Kronecker sum: Let A an d B b e ( m x m ) an d ( n x n ) m at rices, resp ect ively. The m atrix D A iZ• In + Im 0 B is sometimes c al led K ronec ker sum of A and B . ==
Latent root: The roots of t he characteristic polynom ial of an ( m x 111 ) mat rix .tl , t h at is, the roots of p( ,\ ) = det ( Um .4 ) , are t he latent roots of .4 . ( S('(' rh a m c t e rist z c r•alue or ezgem•alu e ) . -
Latent vector of a polynomial matrix: Let P ( x ) b e a poly nomial m atrix with eigen val ue A . A vec t or I ' such t hat P ( ,\ ) r· = 0 is c al led a l atent vector of P ( .r ) . ( See Sec t ion 1 1 .3 . ) Least upper bound norm: G iven a norm I I · I I for ( m x 1 ) vect ors,
II A I I lub
= sup _
{ I I A .r l l . .
.
ll.rll
.1:
(m
x 1 ),
x
::f 0
}
is a norm for ( m x m ) m at rices cal led the least upper bound norm i n d uced by I I · I I · A lternat ively it is cal led t lw sup norm or ope ra t o r n o rm o r z n duced n o rm . ( See Chapter R for more details . )
Left coprime polynomial matrices: The polynom i al m atrices P ( x ) and Q ( J.' ) are said to be rr la l l l'ely ltft p rz rn e or left coprime if t heir greatest co m m o n left d i r•zsors are u n z m odula r. ( See Sect ion I 1 . 3 . ) Left divisibility of a polynomial matrix: A polynomial mat rix P ( .r ) is divisi ble on t he left hy a polynom i al matrix Q ( x ) if Q ( .r ) is a left d iv isor of P ( .r ) , t hat is, if for some pol y nom ial matrix T( .r ) w i t h degree great e r t h an 0 , P ( x ) = Q ( .r )T( x ) . ( See Sect ion 1 1 . 3 . )
Left divisor of a polynomial matrix: A polynom i al matrix Q( J.' ) is
a
left d i v isor of the polynom ial matrix P ( x ) if a polynomial mat r i x T(J.' ) ex ists such that P( .r ) Q ( .r )T( x ) . ( See Sect ion 1 1 . 3 ) ==
Left eigenvector: A ( 1 x m ) vector v H ::f 0 satisfy i ng v H A = At• H for an ( m x m ) m atri x A and a com p lex uumber ,\ is a left eigenvector of A correspon d i ng to or associated w i t h the eigenvalue ,\ .
DICTION A RY OF M ATRICES A N D R E L ATED T E R M S
'-) 5-I
Left inverse matrix: Let A be an ( m x n ) matrix. An ( n x m ) matrix B satisfying BA In is called a left inverse of A . (Ser also g e n c rali::fd =
inverse matrz1·
and refer to Section 3.6 for fu rther details . )
Left multiple of a polynomial matrix: The ( m x n ) polynomial mat rix P(x) is a left multiple of an ( h x n ) polynomial matrix T( x ) if t hen'
exists an ( m x h ) polynom ial mat rix Q(x) such t hat P( x ) = Q(x)T( x ) . ( See Section 1 1 .3 . )
Left quotient of polynomial matrices: Let P ( .r ) = ?0 + P1 .r + · · · + Pr x r ( m x n ) , Pr ::f 0 , and Q ( .r ) = Q o + Q l x + · · · + Q, r' ( m x h ) regular, s < r . A n ( h x n ) polynomial matrix T( .r ) is a left quot ient
of P(x) and Q(x) if an ( m x n ) polynom ial matrix R(x) with degrt>e less than s exists such that P(x) Q( x )T(x ) + R( x ) . ( See Sect ion 1 1 .3 . ) =
Left remainder: Let P(x) and Q ( .r ) b e polynom ial matrices of dt>grrt>s r and s , respectively. s < r . If T( x ) is a left q u o t i e n t of P ( J' ) and Q(x) w i t h P ( .r ) = Q(x)T( x ) + R( x ) , t hen R( .r ) is said to be a h>ft
remai nder. ( See Sect ion 1 1 .3. )
Length of a vector: The length or modulus of an ( m x 1 ) vector x - ) 1 1 2 . (S ee a I so C'll aptt>r - ( X t X J + · · · + Xm Xm ( X J , . . . , Xm ) lS V� X .. X 8.) I
=
'
Leontief matrix: A n ( m x m ) matrix of the form
- am l
1
- a m2
with unit main diagonal and non posit ive rlements elst>where, is called a Leontief matrix. Leslie matrix: An ( m x m ) matrix 0 0 0 b1
where
a;
::f 0 .
i
=
0 b2
0 b3
···
Um - l bm
1 , . . . , m - 1 , is called a Leslif' matrix .
258
H A N DBOOK OF M ATRICES
Liapunov matrix equation: Let A be a real ( m x m) matrix and X , Q
be
real sym m e t ric positwe definite
matrices . The matrix equation
A'X + XA
= -Q
is called a Liapunov (matrix) equation . Linear combination of matrices: Let A; , i = 1 , . . . , k , be ( rn x n ) mat rices and c; E
L inearly dependent vectors: The ( m x 1 ) vectors .r 1 , . . . , x, are linearly dependent if for some i E { 1 , . . . n } scalars C t , . . . , Ci- l , Ci+ l , . . . . en exist such t hat .r ; = C t .J: t + · + c;- t .J:i - 1 + c;+ t .J: i + l + · · + Cn .L'n · (See also Section 4 . 3 . ) .
·
·
·
Linearly independent vectors: The ( m x 1 ) vectors .r 1 , . . . , X n are
linearly independent if they are not linearly dependent. In other words , .r 1 , . . . , .I: n are linearly independent if, for C t , . . . , Cn E 0 implies that C t = · · · = c,
=
=
Loewner matrix: An ( m x m ) matrix A
if its elements are of t he form
. . . . C'rn ,
[ a;j ] is called a Loewner matrix
i, j = l , . . . , m,
Uij =
where C t , are distinct.
=
d 1 , . . . , drn , /1 , . . . , fm , h t , . . . , h m E
Lower block triangular matrix: An ( m
0
x
n) matrix
0
where the A ;j are ( m; x nj ) matrices with I:: f= 1 m; = n and A;j = 0 for j > i , is lower block triangular.
m , I::j' = 1
ni
DICTI O N A RY O F M ATRICES A N D R E L ATED TERMS
259
[a;j ] is a low er A x tri ma m) x m ( An : rix at m rg be Lower He ss en if He sse nb erg ma tri x if a;j = 0 for j > i + 1 , th at is, 0 a23 a33
a12 a 22 a32
a11 a21 a31
'
0 0 0
0 0 0
..
A=
am- I m Um ,m
a m- I m - 1
'
'
am 1 '
Lower t riangular matrix: An ( m x m) matrix 0
a11
'
0
..
= [a ;j]
am i
'
'
.
'
.
'
0 a mm
with a;j 0 for j > i, is a lower triangular matrix. (See Section 9 . 1 4 for the properties . ) =
M-matrix: A real ( m x m) matrix A = [a;j] is said t o be an M-mat rix or Minkowski matrix if A is nonsingu lar, a ;j < 0 for i :f. j and A - l > 0 . Main diagonal of a matrix: See principal diagonal. Matrix: An (m x n ) matrix A is an array of numbers :
A=
a1 1 a21 am l
ar2 an am 2
a1n a 2n
= [a;j ] •=l J=l
.
. , .
m
. , n
= [a ;i ] ·
am n
Alternative notations : A (m x n),
A (m x n)
A = [a;j ] ( m x n ) .
m x n --+ ffi.) , A ,__.. f( A ) rn, (or M at rix fu nc tio n: A fun cti on s are t he ple am Ex x. tri ma a of on cti fun or on cti fun x tri ma a led is cal det erm in a n t an d the tra ce, where n = m. m n f :
H A N DBOOK O F M ATRICES
260
Matrix multiplication: The product of two matrices A = [a;j ] ( m x n ) and B = [ b,i ] ( 11 x p) is defined as AB
=
[t l k= l
a;kbkj
( m x p) .
(See Section 2 . 2 for the rules . ) Matrix norm: A fu n c tio n l l l l attaching a nonnegative real number I I A I I to an ( m x m ) mat rix A is a matrix norm if the following four cond itions are sat isfied for all complex ( m x m ) matrices A , B and c E
(i) ( ii) ( iii) ( i v)
I I A II > 0
if
A i= 0 ,
l l c A I I = l c l abs I I A I I . IIA + Bl l < IIA I I + I I B I I
( triangle inequality ) ,
I I A B II < I I A I I I I B I I ·
Here l c l abs denotes the modulus of c, that is, l c l abs = .,fCii with c being t he complex conjugate of c. Instead of defin ing a matrix norm for all complex ( 111 x m) mat rices, it may be defined for real matrices only. If ( i ) . ( i i ) , ( iii) and ( i v ) hold for all real ( m x m ) matrices and real numbers c , 1 1 · 1 1 is called a matrix norm over the field of real numbers ( IR ) . In that case l c l abs is the absolute value of c . ( See Chapter 8 . ) Matrix polynomial: Given a polynomial p(.t• ) = Po + P t X + · · + p, .L·" . ( or p : IR"' x m IR"' x m ) defiJwd by Q�rn th e fu nc tio n p : ([' "' x "'
�
·
x rn
�
p( A ) = pa lm + p , A +
for ( m x m ) matrices
.4 ,
·
·
·
+ Pn A "
is a matrix polynomial . ( See Section 1 1 . 1 . )
Matrix power series: Given a power series p( x ) = I:;: o Pn X n . the fu nc ti o n p :
�
p( A ) = :L:>n A n n=O
for
(m x 111) matrices A , is a matrix power series , prov ided the infinite sum exists . ( See Sec tion 1 1 . 1 . ) Mat rix product: The product of two or more matrices is a matrix product. ( See m a t rix multiplication. ) Metzler matrix: A real ( m x m ) matrix A = [a;j] is a Metzler matrix i f a ; , < 0 and a;j > 0 for i i= j.
261
DICTIO N A RY O F M ATRICES A N D R E L AT E D TERMS
Minimal polynomial of a matrix: The monic polynomial q ( x ) wit h minimu m degree that annihilates an ( m x m ) matrix A , that is, for which q ( A ) = 0, is called the mi nimal polynomial of A . It is denoted by qA ( ) . (See Section 1 1 . 1 . 3.) -
Minkowski matrix: A real ( m x m ) matrix A = [a ;i ] is said to be a M inkowski matrix or M- matrix if A is nonsingular, a ;i < 0 for i :f. j and A 1 > 0 . -
.
M inkowski-Metzler matrix: A real ( m x m ) matrix A = [a ;1 ] IS a M inkowski - \1et zler matrix if a ;; > 0 for i = 1 , . . . , 111 and a ;j < 0
for i :f. j. (See
Leontief matrix for
a special case.)
M inor of a matrix: The determinant of a square su bmatrix of an ( 111 x m ) matrix A is a m i nor of A . Minor of an element of a matrix: For an ( m x m ) matrix A = [a ;j ] the
minor of an element a;1 is the determ inant of t he matrix obi ained by delet ing t he ith row and jth column from A . a 1 ,j -
m inor( a ;j )
=
det
.
.
.
1
a ; - l ,j - 1
a , - I ,j + I
ai - l
a i + l ,j - 1
a i + I ,j + l
a i + l .n>
m
U n l , nl
(See also Section 3 .4 . ) Modal matrix: Let A be an ( m x m ) matrix with linearly independent eigenvectors V t , . . . , Vm of length 1 , that is, vfl v; = 1 for i = 1 . . . . , m . Then V = [v 1 , , , , v m ] is called a modal mat rix or eigen r:ector m a t ru of A . ( See Chapter 5 for results on eigenvectors. ) •
Modulus of a vector: The modulus or length of an ( m - Xm ) 1/2 . - ( X [ , . . . , X m )' 'lS V/'H _ ( X- t X t + · · · + Xm X.. X X-
x
1 ) vector
Monic polynomial matrix: The ( m x m ) polynomial matrix P( .L' ) = Po + Pt .r + · + Pr x r is called monic if Pr = lm . (See Sect ion 1 1 .3 . ) ·
·
Monomial matrix: An ( m x m ) matrix A is said to be monomial i f there exists a permutation matrix P and a diagonal m a trix D such t hat A = PD.
262
H A N DBOOK O F M ATRICES
Moore -Penrose ( generalized) inverse matrix: The ( n x m ) matrix .4+ is the Moore -Penrose (generalized) i nverse of the ( m x n ) matrix A if it satisfies the following four condi tions: (i) AA+ A = A , (ii) A + A A + = A + , ( iii) ( AA+ )H = A A + , (iv) ( A + A ) H = A+ A . (See
Section 3.6.2 for the properties. )
M P-inverse of a matrix: Short for Moore -Penrose inverse matrix. Multiplication by a scalar: The product of an ( m x n ) matrix A = [a,j ]
and a number c ( a scalar) is defined cA
and
Ac
=
_
as
[ca;j ] ( m x n ) [ca;j ]
(See Section 2.3 for the rules . )
(m
x n).
Multiplication of matrices: See matrix multiplica tion. Multiplicative norm: A norm I I · I I for ( m x m ) matrices is called a
multiplicative or submultiplicative or matrix norm if for any ( m x m ) matrices A , B , I I A B I I < I I A I I I I BI I · (See Chapter 8 for more details on norms. )
Multiplicity of an eigenvalue: The multiplicity of an eigenvalue of an ( m x m ) matrix A is the multiplicity of the corresponding root of the ch aract e ristic equation of A. (See Section 5 . 1 . ) Negative definite Hermitian form: A Hermitian form xH Ax, defined by a Hermitian ( m x m ) matrix A , is negative definite if A is negative
definite. (See Section 9 . 1 2 for more on definite matrices . )
Negative definite matrix: A n ( m x m ) matrix A is negative definite if it is Hermitian (or real symmetric ) and x H Ax < 0 (or x' Ax < 0) for any (real) ( m x 1 ) vector x f. 0. (See Section 9 . 1 2 for the properties. ) Negative definite quadratic form: A quadra tic form x' A x , defined by a real symmetric ( m x m ) matrix A , is negative definite if A is negative
definite. (See Section 9 . 1 2 for more on definite matrices . )
Negative matrix: A real ( m x n ) matrix A = [a;j ] is negative if a;j < 0 for i = 1 , . . . , m , j = 1 , . . . , n . (See also Section 9 .9 . ) Negative semidefinite Hermitian form: A Hermitian form x H A x , defined by a Hermitian ( m x m ) matrix A , is negative semidefinite if A is negative semidefinite . (See Section 9. 12 for more on definite
matrices . )
DICTION A RY O F M AT RI CES A N D RE LATED TERMS
263
) matrix A is negat ive semidefinite if it is Hermitian (or real symmetric) and :rH A.t· < 0 ( or :r'Ax < 0 ) for any ( real) ( m x 1 ) vector :r . ( See Section 9 . 1 2 for t he properties. )
Negative semidefinite matrix: An ( m
x
m
Negative semidefinite quadratic form: A quadratic form x' A x , de
fined by a real symmetric ( ) matrix A , is negative semidefinite if A is negati ve semidefinite. (See Section 9. 1 2 for more on defin i t e matrices . ) m x m
) matrix A is nilpotent i f there exists positive integer i such that A i = 0 .
Nilpotent matrix: An ( m
x
m
a
Nonderogatory matrix: An ( m x m ) matrix A is nonderogatory if every eigenvalue of A has geome tric multiplicity 1 . ( See Chapter 5 for more
on eigenvalues.)
Nonnegative definite matrix: An ( m
) matrix A is nonnegat ive definite or positive semidefinite if it is Hermitian (or real sy mmetric) and x H Ax > 0 (or x'Ax > 0) for any ( real) ( m x 1 ) vector (See pos z t n•e s e m idefinite ma trix and refer to Sect ion 9 . 1 2 for the properties.) x
m
:r .
Nonnegative matrix: A real ( m x n ) matrix A = [a;j ] is nonnegative if a;i > 0 for i = 1 , . . . , m , j = 1 , . . . , n . (See Section 9 . 9 . )
) matrix A is nonpos 1 t1ve definite or negati ve semidefinite if it is Hermitian (or real symmetric) and x H Ax < 0 (or x'A:r < 0) for any ( real) ( m x 1 ) vector :r . (See n egaiit>e semidefinite matrix and refer to Section 9. 1 2 for tlw properties. )
Nonpositive definite matrix: An ( m
x
m
Non positive matrix: A real ( m x n ) matrix A = [a;i ] is non positive if a;i < 0 for i = I , . . . , m , j = I , . . . , n . (See also Section 9.9.) Nonsingular matrix: An ( m
or regu lar if det( A ) properties . )
:f. 0
) matrix A is nonsingular or invertible and thus A - 1 exists. ( See Section 3.5 for the x m
Norm: A function II · I I attaching a nonnegative real number II A I I to an
( ) matrix A is a norm if t he following three conditions are satisfied for al l complex ( ) matrices A, B and complex numbers m
x
n
m x n
c:
(i) I I A I I > 0 if A :f. 0 , (ii) l l c A I I = l c l abs I I A I I ,
H A N DBOOK O F M ATRICES
264
(iii) I I A
( triangle inequality).
+ B l l < II A I I + I I B I I
Here l c l abs denotes the modulus of c , that is, lclabs = ..jCC with c being the complex conjugate of c. I nstead of defining a norm for all complex ( m x n ) matrices , it may be defined for real matrices only. I f ( i ) , (ii) and ( ii i ) hold for all real ( m x n ) matrices and real numbers c , 11 · 1 1 is called a norm over the field of real numbers ( IR) . In that case l c l abs is the absolute value of c . (See Chapter 8 for details.) Normal form of a matrix: An ( m x m) matrix A can be reduced by elementary column and row operations to a matrix N = [d;j] with d., 1 or 0 . d;j = 0 for i ::/- j . The matrix N is said to be the normal form of A . For instance, =
[�
0
1
0 0
�]
and
1
0
0 0 0
1 0 0
0 0 0 0
are normal forms of matrices. Normal matrix: An ( m x m ) matrix A is normal if A 11 A = A A 11 . Null matrix: An ( m x n ) matrix is a null matrix, denoted by
simply by
0,
if all its elements are zero. (See also
O m x n or
zero m atrix. )
Null vector: An ( m x 1 ) or ( 1 x m) vector is a null or zero vector if all
its elements are zero.
Numerical range of a matrix: For an ( m x m) matrix A , the set { x11 A.r : .r ( m x l ) , .r 11 .r 1 } is sometimes called the numerical range of A . =
Operator norm: G iveu a norm I I I I for ( m x 1 ) vectors,
{ ��il ·
ll sup I I A I I 1ub -
�
:
x ( m X 1 ) , x ::/- 0
}
defines a norm for ( m x m) matrices called operator norm induced by 1 1 · 1 1 - A lternatively it is called sup norm or least upper bou n d n orm or in duced n orm. (See Chapter 8 for details on norms. ) Order of a matrix: ( m x n ) is the order or dim ension of a matrix A with m
rows and <"olurnns. Sometimes the degree of the polynomial of A is called its order. n
characteristir
265
DICTI O N A RY O F M AT RI C ES A N D R ELATED TERMS
Orthogonal complement matrix: For m > n , the (m x ( m - n ) ) matrix A.L is an orthogonal complement of the ( m x n ) matrix A if [A : A .L ] is invertible and A H A .L = 0. Orthogonal matrix: An ( m x m ) matrix A is orthogonal if A' (See Section 9. 1 0 for its properties.)
=
A- J .
Orthogonal projector: A Hermitian idempo t e n t matrix is said to be an orthogonal projector, that is, A ( m x m ) is an orthogonal projector if A = A H = A 2 . For instance, given an ( m x 11 ) matrix B with rk( B ) = m, the matrix B H ( B BH ) - 1 B is an orthogonal projector. (See Section 9. 7 for more on Hermitian matrices and Sec t ion 9.8 for
the p roperties of idempotent matrices. )
Orthogonal vectors: The ( m x 1 ) vectors x and y are orthogonal if XH y = 0 . Orthogonally equivalent matrices: The ( m x n ) matrices A and B are orthogonally equivalent if there exists a real orthogonal ( m x m ) matrix U and a real orthogonal ( n x n ) matrix V , such that A U ' B 'V . (See also Section 6 . 2 . ) =
Orthogonally similar matrices: The real ( m x m ) matrices A and B are orthogonally similar if an orthogonal (m x m ) matrix U exists such that B = U A U ' . (See also Section 6 . 2 . ) Orthonormal vectors: The ( m x 1 ) vectors X H y = 0 and X H X = yH y = 1 .
x
and y are orthonormal
if
Orthostochastic matrix: An ( m x m) matrix A = [a;j ] is said to be orthostochastic if there exists a real orthogonal matrix Q = [%] such that a;j = qli for i, j = 1 , . . , m. .
Orthosymmetric matrix: A Hankel m at rzx is sometimes called an
orthosymmetric matrix .
Oscillatory matrix: A real ( m x m) matrix A is called oscillatory if all
its minors of all possible orders are nonnegative and there exists a positive integer k such that all minors of Ak are positive.
Parity check matrix: For n > m an (m x n ) matrix A = [Im
or parity check matrix if B is a all elements of B are 0 or 1 . ch eck m at rix
:
b in ary m at ru,
B] is a th a t is. .
Partitioned matrix: An ( m x n ) matrix A = [A;j ] consisting of (m ; X Tij ) sub matrices A,1 , i = 1 , . . . , p , j = 1 , . . . , q, is a partitioned mat rix. (See Section 9. 1 1 for the properties. )
266
H A N DBOOK
OF
M ATRICES
Permanent of a matrix: The permanent of an ( m x n ) matrix A = [a;1 ] with m < n is the quantity defined by m
� IT a . . ""' i=
lJ ,
1
where the sum is taken over all one-to-one functions from { 1 , 2, . . . , m } to { 1 , 2 , . . . } , n
.
Permutation equivalent matrices: The ( m x m) matrices A and B are
said to be permutation equivalent if there exist P and Q such that A = P BQ.
permutation m atnces
Permutation matrix: An ( m x m) matrix with exactly one I in each
row and each column and zeros elsewhere is a permu tation matrix. For instance, 0
I
0
0 0
1
0 0
1
is a permutation matrix. (See Section 9 . 2 for the properties of special permutation matrices.) =
Persymmetric matrix: An ( m x m ) matrix A
[a ;j] is persymmetric
if it is symmet ri<" with respect to the diagonal from the upper right hand corner to the lower left hand corner, that is, aI I a2 1
A=
. . . a l ,m- 1
a,2 a 22
a m- 1 . 1 am i
a1m a , m- 1 •
am - 1 , 1
..
.
a 22 a2 1
a12 al l
is persymmetric. Polynomial matrix: An ( m x n ) matrix P(x ) = (p ;j ( x )] whose typical elements are polynomials Pii ( x ) = PiJ,O + PiJ, 1 x + + PiJ, r ,, x r ,, , is a polynomial matrix. Alternative notation : P ( x ) = Po+ ?1 x + + Pr x r , where 1' = max;,j r;j and ·
·
·
·
P 1 1 .k
P1 n ,k k
Pm 1 , k
with Pi j ,k
= 0
for k
>
r ;i .
=
0 , I , . . . , r,
Pm n ,k
(See Section 1 1 .3 for details . )
·
DICTIO N A RY O F M AT RICES A N D RE LATED T E R M S
267
Positive definite Hermitian form: A Hermitian fo rm xH A x , defined by a Hermitian ( m x m ) mat rix A , is positive definite if A is positive definite. ( See Section 9 . 1 2 for the properties of positive definite matrices. ) Positive definite matrix: A n ( m x m ) matrix A is positive definite if it is Hermitian (or real symmetric) and xH Ax > 0 (or x' Ax > 0) for any ( real ) ( m x 1 ) vector x f. 0 . (See Section 9 . 1 2 for the properties. ) Positive definite quadratic form: A quadratic fo rm x' Ax , defined by a real symmetric ( m x m ) m atrix A , is positive definite if A is positive definite. ( See Section 9. 1 2 for the properties of positive definite mat rices. ) Positive matrix: A real ( m x n ) mat rix A = [a;j) is positive if a;j > 0 for i = 1 , . . . , m , j = 1 , . . . , n . ( See Section 9 . 9 for the properties . ) Positive semidefinite Hermitian form: A Hermitian form x H A x , de fined by a Hermitian ( m x m ) matrix A , is positive semidefinite if A is positive semidefinite. ( See Section 9. 1 2 for the properties of positive semidefinite mat rices . ) Positive semidefinite matrix: A n ( m x m ) matrix A is positive semidefinite if it is Hermitian (or real symmetric) and xH Ax > 0 (or x'Ax > 0 ) for any (real ) ( m x 1 ) vector x . ( See Section 9 . 1 2 for the properties. ) Positive semidefinite quadratic form: A qu adratic fo rm x' A x , defi ned by a real symmetric ( m x m ) m atrix A , is positive semidefinite if A is positive semidefinite. ( See Section 9 . 1 2 for the properties of positive semidefinite mat rices. ) Postmultiplication: Let A , B b e ( m x n ) and ( n x p ) mat rices , respectively. In the mat rix product A B , the matrix A is said to be postmultiplied by B . Power of a matrix: The ith power of the ( m Ai , is defined as follows:
'
A =
x m
) matrix A , denoted by
n;=l A
for positive integers i
cnj I I A )- 1
for negative integers i, if det ( A ) f. 0
for i = 0
H A N DBOOK O F M ATRICES
268
If A can be written as 0 A = U 0
for some ll1l l t a ry matriz· U ( see Chapter 6 for conditions) , then tlw power of A is defined for any n E IR, n > 0 , as follows: _.\ <> I
0
0
This defi n i tion applies, for instance, for Hermitian matnces. ( See Sec tion 3 . 7 for some properties of powers of matrices ) . Prcmultiplication: Let A . B be ( m x n ) and ( n x p) mat rices respectively. In t he matrix product A B , t he m atrix B is said to be premultiplied by A. . Primitive matrix: A n ( m x m ) matrix A is said to be primi tive i f i t is i rredu e � b le and i t h as only one eigenval ue with modulus equal to its spectra l ra dius.
Prineipal diagonal of a matrix: The elements a, ; , i = I , . . . , m, const i t u te the princi pal diagonal or m a in diago n al of an ( m x m) matrix A = [a;j ] . Principal minor: The determinant
det of a p ri n C ipal s u b m a trix of the ( m x m ) matrix A = [a;j ] is a princi pal minor of A for k I. . . . , m - I. =
Principal submatrix: The matrices k = l, . . . , m - 1,
are the princi pal submat rices of the ( m x m ) matrix A = [a ;j] .
DICTIO N A RY OF M ATRICES A N D RELATED TERMS
269
Principal vector of a matrix: A lternative expression for ge n e rali::ed e igenvector of a m a t rix.
P roduct matrix: The product A B of the ( m x n ) matrix A and the ( n x p) matrix B is the product matrix of A and B . ( See Section 2 . 2 for properties.) Product of matrices: See ma trix mult ip l! cation. Projection matrix: An ( m x m) matrix PA is a projection m atrix for t he ( m x n ) matrix A if PA = P1 = Pf , PA A = A and rk( PA ) = rk( A ) . For instance, if rk ( A) n , PA = A ( A H A ) - 1 A H is a projection matrix for A . ( See Sections 9. 7 and 9 . 8 for properties . ) =
Pseudo-inverse matrix: A generalized inverse matrix is somet im es called a pseudo- inverse matrix. ( See Section 3 . 6 for some properties . ) Quadratic form: G iven a real symmetric ( m x m) m atrix A , t he funct ion Q : IR"' IR defined by Q(x) = x ' Ax is called a quadrat ic form. ( See Section 9 . 1 3 for the properties of symmetri c m atrices. ) �
Quadratic matrix: An ( m x n ) mat rix w ith (See also squ a re matrix. )
m
= n is a quadratic matrix .
Rank of a matrix: The rank of a matrix A , denoted by rk A or rk( .-\ ) , is the maximum number of linearly independent rows or columns of A . ( S ee Section 4 . 3 . ) Rank of a polynomial matrix: The rank of a polynomial mat rix is the number of columns of the largest submatrix whose determinant is not identically zero ( See Section 1 1 . 3 . ) Rational matrix: A n ( m x n ) matrix R( .r ) = [r;j ( .r )] whose elements r,j ( x ) are rational functions, that is, rat ios of polynomials . is a rational matrix. Rayleigh ratio: The Rayleigh ratio for an ( m x m) square matrix A and an ( m x 1 ) vee tor x # 0 is
( See Sect i ons 5 . 2 and 5 . 3 for some of its propert ies. ) Real matrix: A
ma
tri x is called real if all its elements are real numbers.
Real polynomial matrix: A polynomial matrix is called real if all elements are real polyno m ials. ( See Section 1 1 .3 . )
its
H A N DBOOK O F M ATRICES
270
Rectangular mat rix: An ( m x n ) matrix with m # n is sometimes called a rectangular matrix. Reducible matrix: An (m x m) matrix A is said to be reducible or decomposable if there exists a perm u t ation m a trix P such that P'AP
[ AA l 1l
=
2
]
Ok x ( m - k ) An
where A 1 1 is ( k x k ) , An is ( ( m - k ) x ( m - k ) ) and A2 1 is ( ( m - k ) x k ) .
Reflexive generalized inverse matrix: The ( n x m) matrix A ; is a reflexive generalized inverse of the ( m x n ) matrix A if AA;:- A A and A; AA;:- = A; . ( See Section 3 . 6 for properties of generalized inverse matrices . ) ==
Regular matrix: A n ( m x m ) matrix A i s regu lar or nonsingular or invertible if det ( A ) # 0 and thus A - 1 exists. ( See Section 3 . 5 for details. ) Regular polynomial matrix: The ( m x m ) polynomial mat rix P ( x ) = Po + P1 .r + + Pr .rr is regular if Pr is a nonsingular matrix . ( See Section 1 1 .3 . ) ·
·
·
Relatively left prime polynomial matrices: The polynomial matrices P ( x ) and Q ( x ) are said to be relatively left prime or left cop rim e if their g reatest common left divisors are unim odular. ( See Section 1 1 .3 . ) Relatively right prime polynomial matrices: The polynom ial matrices P( .r ) and Q( x ) are said to be relatively right prime or rzght copri m e if their greatest com m o n right divzsors are unim odular. ( See Section 1 1 .3 . ) Resultant matrix: An ( m x m) matrix A is a resultant m atrix if det( A ) is a resultant. ( See Section 1 1 . 1 . 1 for the definition of a resultant. ) Reverse unit mat rix: The ( m x m) matrix
with
-{ 1
.. -
a I)
0
0 0
... ...
0
1
1
0
1
0
...
0
=
for i + j m + 1 otherwise
is called an exch ange m a trix, reverse unit matrix or flip m atrix.
271
DICTION A RY O F M ATRICES A N D RELATED TERMS
Riccati equation: Let A, B be ( m x m ) m atrices, P ( m x m ) real symmetric, R ( m x m ) real sym metric , positive definite and Q real symmetric, positive semidefinite. The equation P B R- 1 B' P
-
A' P - P A - Q
=
0
is called a Riccati equation. Right coprime polynomial matrices: The polynomial matrices P ( .r ) and Q ( x ) are said to b e relatively right prime or right coprime if their great est co m m o n right divisors are unim odular. (See Section 1 1 .3 . ) Right divisibility of a polynomial matrix: A polynomial matrix P ( .r ) is d ivisible on the right by a polynomial matrix Q(x) if Q(.r) is a right diviso r of P ( x ) , that is, if for some polynom ial matrix T( .r ) with degree greater than 0 , P (x ) = T( x )Q(x ) . (See Section 1 1 .3 . ) Right divisor of a polynomial matrix: A polynomial matrix Q(.r) is a right d ivisor of the polynom ial matrix P(x) if a polynom ial mat rix T(x) exists such that P ( x ) T(.r)Q(x). (See Section 1 1 .3 . ) =
Right eigenvector: An eigenvector or characteristic vector is someti mes called a right eigenvector. ( See e igenvector of a m a trix.) Right inverse matrix: Let A be an ( m x n ) matrix. An ( n x m ) matrix B satisfying A B = Im is c alled a right i nverse of A . (See also Sections 3 . 5 and 3.6. ) Right multiple of a polynomial matrix: The ( m x n ) polynomial matrix P(x) is a right multiple of an ( m x h) polynomial matrix T(x) if there exists an ( h x n ) polynom ial matrix Q ( .r ) such that P(x) = T(x)Q ( x ) . (See Section 1 1 .3 . ) Right quotient of polynomial matrices: Let P(x) = Po + P1 .r + - - - + Pr xr ( m x n ) , Pr =/: 0 , and Q(x) = Qo + Q 1 x + + Q,x• ( h x n ) regu lar , with s < r. A n ( m x h) polynomial matrix T(x) is a right quotient of P(x) and Q(x) if an ( m x n ) polynomial matrix R(.r) with degree less than s exists such that P(x) = T(x)Q(x) + R( x ) . (See Section 1 1 . 3 . ) ·
·
·
Right remainder: Let P(x) and Q(.r) b e polynomial matrices of degrees r and s , respectively, s < r . If T( x ) is a right qu o t i e n t of P(.r) and Q(x) with P(x) T(.r)Q(x) + R ( x ) , then R(x ) is said to be a right rem ai nder. ( See Sec t ion 1 1 .3 . ) =
H A N DBOOK O F M AT R IC ES
Rotation matrix: For r, s E IN , r < s < m , an ( m x m ) mat rix Gr. = [g;1] is said t o be a rot ation matrix or Givens m a trix if Yrr = g., = cos 0 . g;; = 1 for i = 1 , . . . , m , i # r, s , Y • r = - si n 0 . Yr• = sin 0 an d a l l ot her elements are zero. For exam ple , cos 0 0 - s in 0
0
1 0
sin 0 0 cos 0
cos f) - sin 0
sin 0 cos 0
]
are rotation m atrices .
Routh mat rix: A tndwgo n a l ( m x m ) m atrix m at rix if
a l l = -b l , a , _ I . , = .Jb:,
where b 1 , . . . . b"'
an·
a;; = 0 , a ; ,, _ I = - Viii ,
A
[ a;1 ] i s c al led Rou t h
.
t = 2 � ·)') . . . . , 111 ,
uon zero real n nmbers. For example,
0 0 0
0 _
_;s
J5 0
is a Routh mat rix.
Row dimension of a matrix: The num ber of rows of a matrix is i t s row d imension .
Row rank of a matrix: The row rank of a matrix A , denoted by row rk
.-1 or row rk( A ) . is t he maximum number of l inearly i n dependent rows of A. . ( See Sec t ion 1 . :3 . )
Row regular matrix: A n ( 111 x n ) matrix A is row regu lar if rk( A ) =
111 .
Row stochastic matrix: Al ternative expression for storhasttc m a trix. Row vector: A ( 1 x n ) m atrix is a row vector. Row vectorization: Stac k i n g the rows of a mat rix i n a row vector is c allt>d row n•d orizat ion of t he m at rix. �otation : For A = [a;1 ] ( m x n )
( See SPc t ion
7. 1 . )
273
DICTION A RY O F M AT RI C ES A N D RELATED TERMS
Scalar matrix: An ( m x m) mat r i x A = [a;i] is so m e t i m es said t o be a scalar m atrix if, for some c E ([' , if i = j if i # j , t h at i s A = d i ag(r, . . . , c ) = e lm . ,
Scalar multiplication: See m u ltiplica tion by a scalar. Scalar product: The scalar product of two vec t ors Y1
y=
X = X rn
i s d efin e d
where
i;
as
Ym
(m
x
1)
m
- "" - , XH y = L.., x;y; i= I
is t he complex c onj u gat e of x, . ( See al so w n e r prod u c t . )
Schur complement of a matrix: For matrices A ( m x m ) nonsingular. B ( m x n ) , C ( n x m ) , D ( n x n ) . t he ( n x n ) matrix S = D - CA - 1 B i s called t llf' Schur c o m p lemen t of A i n
Schur product : See Hada m a rd product. Schur-stochastic matrix: An ( m x m ) matrix A [a;1 ] is said to bt> S c hu r- s t oc h as t ic or u n ita ry stochastic if there exist s a unitary ( m x m ) matrix U = [It;j ] such that a;j = l u;j l �bs for i, j 1 . . . . . m . =
=
Secondary diagonal: The el enw nts a 1 +un - i . i = 0 . . . . . m - 1 , of an ( m x m ) mat rix A = [a,j ] con st i t u t e the secondary diagon al of A . I n other words. t he secon dary d i ago n al i s t h e d i ago na l from t h e u p per right hand to the lower left hand corner of A . Selfadjoint matrix: A Hermitian ( m x m ) m at rix A i s sometimes called selfadj o i nt that is, A is selfadjoint if A H = A . (See Sec t ion 9 . 7 for t h e p ro per t i f's . ) ,
274
H A N DBOOK OF M ATRICES
Semi circulant matrix: An ( m x m) matrix of the form am - I am - 2 a m-3 0 0
0 0
am a m- I am - 2
0
is said t o b e semicirculant . ( See also circu lant m atrix. ) Semidefinite matrix: A matrix is semidefinite if it is positive semz definite or n eg ative s e m i defin t l e . ( See Section 9. 1 2 for some propert ies . ) Seminegative matrix: A real ( m x m ) m atrix A = [a;1 ] is serninegative if a ,J < 0 for all i , j = 1 , . . . , m and at least one element is strictly negati ve. ( S ee also Section 9 . 9 . ) Seminorm: A fun('! ion // · // attaching a nonnegative real number 1 / A / 1 to an ( m x 11 ) matrix A is a sem i norm if the followi ng three conditions are satisfied for all complex ( m x n ) matrices A , B and complex nu mbers c :
(i) //A/1 >
0,
( i i ) / l eA l / = / c / ab s 1 / A / 1 , ( i i i ) / / A + B / 1 < 1 / A / 1 + 1/ B/1
( triangle inequality ) .
Here / c / abs denotes the modulus of c , that is, / c / ab s = .;;:c with c being the complex conj ugate of c . I nstead of defini ng a seminorm for all complex ( m x n ) matrices, it m ay b e defined for real m at rices only. If ( i ) , ( i i ) and ( i i i ) hold for all real (m x n ) matrices and real numbers c, 1 / 1 / is called a seminorm over the field of real numbers ( IR ) . I n that case / c / abs is the absolute val ue of c . ( See Chapter 8 for further details . ) ·
Semiorthogonal matrix: A n ( m x n ) matrix A is semiorthogonal if A A ' = Im or A'A = In . (See also orthogonal matrix. ) Semipositive matrix: A real ( m x m ) matrix A = [a; j ] is semipositive if a;j > 0 for all i , j = 1 , . . . , m and at least one element is stric t ly positive. ( See Section 9 . 9 . )
275
D ICTI O N A RY O F M ATRICES A N D RE LAT E D TERMS
Shift matrix : The ( m x m) circ u lant matrix 0 0
1 0
0 1
0 0
0 1
0 0
0 0
1 0
is called a shift m atrix . (See Section 9 . 1 for some properties. ) Similar matrices: The (m x m) matrices A and B are similar if an invertible matrix P exists such that B = P A P- 1 • ( See also Chapter 6.) Simple eigenvalue: A root of the characteristic polynomial of an ( m x m ) matrix A with multiplicity 1 is a simple eigenvalue of A . ( See Section 5. 1 . ) Simple matrix: A n ( m x m ) matrix A is said to be simple if it is similar to a diagonal matrix , that is, if there exists an i nvertible matrix P such that PAP- 1 is a diagonal matri x . ( See Chapter 6 for examples . ) Simultaneously diagonalizable matrices: The ( m x m ) matrices A 1 , . . . , A,. are said to be simultaneously diagonalizable if a nonsingular ( m x m ) matrix P exists such that P A; p - 1 is diagonal for i = 1 , . . . , n . (See Sections 6. 1 .2 and 6 . 2 . 3 . ) Sine of a matrix: The sine of an ( m x m) matrix A i s defined sin ( A )
=
f (2n +
n =O
as
( - 1 )n A2n + l . 1 )!
( See Section 1 1 . 1 . 3 . ) Singular matrix: A n ( m x m ) matrix A is singular i f det( A )
=
0.
Singular value: The singular values of an ( m x n ) matrix A are the nonnegative square roots of the eigenvalues of A A H , if m < n , and of A H A , if m > n . ( See Sections 5 . 1 and 5 . 5 . ) Skew-circulant matrix: For form a1 - am
- am - 1
a 1 , . . . , am
a2
at -a m
E
aa
am- I
am
0[
am - 2
Om - 1
am- 3
am - 2
a2
H A N DBOOK O F M ATRICES
276
is said to be a skew-circulant matrix. ( See also circula n t ma trix. ) Skew-Hermitian matrix: An ( m 0 -a12 -
A =
-
-aim
x m
) m atrix a lm a 2m
a12 0
= (a ;j ] ,
0
-
- a 2m
w ith a ;1 = - a1 ; , is a skew- Hermitian matrix. Here ll;j denotes the complex conj ugate of a ;1 . In other words, A is skew- Hermitian if AH = - A . ( See Section 9 . 7 . ) Skew prime polynomial matrices: The polynomial matrices P ( .r ) and Q ( x ) are said to be skew prime if det P ( .r ) and det Q ( x ) are relatively prime polynomials . ( See Section 1 1 . 1 . 1 for the defi nition of rela t ively prime polynomials and Section 1 1 .3 for more on skew prime polynom ial matrices . ) Skew-symmetric matrix: An ( m 0
x m
) matrix
a12 0
a1m a2rn
with a ;; = 0 and a ;j = - a1 ; , is skew-sym metric. I n other words. an ( m x m ) matrix A is skew-symmetric if A' = - A . ( See Section 9 . 1 :3 . ) Skew-symmetric part of a matrix: The matrix � ( A - A' ) is the skew symmetric part of an ( m x m ) matrix A . Sparse mat rix: A matrix is sparse if only a small fract ion of its element s are nonzero. Spectral radius: p( A. ) = max; l>.dabs is t he spectral radius of an ( m matrix A with eigenvalues ..\ 1 , . . . , A m . ( See Section 5.4 . )
x
m)
Spectrum of a matrix: The set of eigenvalues of a matrix A is called t lw spectrum of A . ( See Chapter 5 . ) Square matrix: A n ( m q u a dra tic m a t rix.
x
n
) matrix with
m
n
ts a square matrix or
Square root matrix: An ( m x m ) matrix B is a square root of the ( m x m ) mat rix A if B B = A . It is denoted by A 1 1 2 ( See Sect ion 6 . 1 .4 . )
277
DICTI O N A RY OF M ATRICES A N D RELATED TERMS
Stable matrix: An (m x m) square matrix A is stable or co nvergent if A n _. 0 for n oo . ( See Section 9 . 3 . ) ----.
Stieltjes mat rix: A symmetric nonsingular M- matrix is called a Stieltjes matrix . Stochastic matrix: An ( m x m ) matrix A = [a ;j ] is said to be a stochastic or row stochastic or transition matrix if 0 < a ;j < 1 , i, j = 1 , . . , m . and L::j 1 a;j = 1 , i = 1 , . . . , m . For i nstance. .
I
3
0 I
2
I
I
3
3
0
2
0
1 I
is a stochastic matrix . ( See Section 9 . 9 . ) Strictly lower triangular matrix: A n ( m x m ) matrix 0
0 0
...
0 0
am m- 1
0
'
=
with a ;j 0 for j > i is a strictly lower triangular matrix. ( See Section 9 . 1 4 for the properties . ) Strictly triangular matrix: A matrix is strictly triangular i f i t is strictly lower t ria ngula r or strictly upper triangula r. ( See Section 9 . 1 4 for t.he properties . ) Strictly upper t riangular matrix: An ( m x m ) matrix
Um- l m '
0
0
=
with a;j 0 for i > j is strictly upper t riangular. ( SPe Sec-tion 9 1 4 for the properties. ) S triped matrix: A m at rix is called striped if it is either Toeplitz ma trz:r.
a
Hankel or
a
278
H A N DBOOK O F M ATRICES
Submatrix: A (p x q ) matrix B [a;k j , ] with i 1 < < ip and j1 < < jq is a submatrix of the ( m x n ) matrix A = [a;j ] · =
·
·
·
·
·
·
Submultiplicative norm: See multip licative n orm or m atrix n orm. Subtraction of matrices: The difference between two matrices A [a;j ] ( m x n ) and B = [b;j ] ( m x n ) is defined as A - B = [ a ,j - b;j J ( m x n ) .
( See Section 2. 1 for the rules. )
Sylveste r matrix: Let p ( x ) = Po + P 1 X + + Pm - t X m - l + x m , q ( x ) = qo + q t .r + · + qn - t X n - l + q xn be polynomials. The Sylvester matrix n corresponding to p( ) and q ( ) is ·
·
·
·
·
·
1 0
·
Pm - t I
Pm- 2 Pm - t
Po Pt
0 Po
0 0
( ( m + n) x ( m + n) )
where the upper part consists of n rows and the lower part consists of m rows. For example, for p( x ) = Po + p 1 x + p2 x 2 + x 3 and q ( x ) = qo + q 1 x + q2 x 2 ,
1 5=
0 q2 0 0
P1 P2 qo ql q2
P2 1 ql q2 0
Po Pl 0 qo ql
0 Po 0 0 qo
( See Section 1 1 . 1 . 2 . ) Symmetric matrix: An ( m x m ) matrix a1 1 Ut2
a 12 a22
Utm
a 2m
..
.
Utm a 2m U mm
= [aij ]
279
DICTIO N A RY O F M ATRICES A N D RELATED TERMS
aj ; , i , j = l , . . , m , is symmetric . In other words, an ( m x m ) matrix A is symmetric if A' = A . ( See Section 9 . 1 3 for the properties . )
with a;j
=
.
Symmetric part of a matrix: The matrix � ( A + A') is the symmetric part of an ( m x m ) matrix A . Symmetric r- Toeplitz matrix: For r < ( m x m ) matrix of the form 0
0
0
0
and
m
0
0
0
0 0
a
1, . . .
, Gr
E ([' ,
an
0 0
O'r- 1
0
a;+k , j + k and a;j = aj i for all i , j , k and a;j = 0 for l i - 1 l ats > r , is a symmetric r-Toeplitz matrix. ( See also Toeplit::
with a;1
=
matnx. )
Symmetric Toeplitz matrix: For a 1 , . . . , n m E
0: 2 0: 1 0: 2
O'J 0: 2 0' 1
O'm - 1 O' m - 2 O' m - 3
with a;j = a;+k . i +k and a;j = aj i for all matrix. ( See also Toep/itz matrix. )
i , j,
O'm O'm- 1 O'm - 2
x
m ) matrix
= [ a;i ]
k , is a symmetric Toeplitz
Tensor product: See Kron ecker product or direct product.
H A N DBOOK O F M AT R I C ES
280
Toeplitz matrix: For o: 1 , . . . , o: 2m- l E If , an ( m
Cl:m- 1 Cl: m
w i t h a ,J
=
Cl:m+ l Cl: t 0: 2
Cl:m+2 Cl:m+ l Cl: t
Cl: m - 2 Cl:m- 1
Cl: m - 3 Cl: m - 2
x m
Cl:2m - 2 Cl:2m - 3 Cl: 2 m - 4
) matrix of the form
Cl: 2m - l Cl: 2m - 2 Cl: 2 m - 3
[a ;j ]
a ; + k ,j + k for all i , j , k , is a Toeplitz matrix.
Totally nonnegative matrix: A real square matrix is called totally nonnegative if all i t s m i nors (of all possible or ders) are nonnegat ive. Totally positive matrix: A real square matrix is called totally posit ive if all its minors ( of all possible orders) are positive. Trace of a matrix: The t race of an ( m t r A = t r( A )
=
x m
) matrix A = [a ;j ] is m
a11 +
·
·
L a,; .
· + amm =
i=
l
( See Section 4 . 1 for the properties . ) Trans ition matrix ( of a M arkov chai n ) : A nonnegative ( m x m ) matrix A = [a ;1 ] is a t ransition matrix or a stochastic ma trix if 0 S a ;j < 1 , i, j = 1 , . , m , and L_'; 1 a ;j = 1 for i = 1 , , m . For example, .
.
.
I
1
.
I
3 0
3 0
3
I
0
2
2
.
1 I
is a transition matrix. ( See Section 9.9. ) Transpose matrix: The transpose of the ( m ( n x m ) matrix
A' =
all a2 1
al2 an
aln a2n
'
x n
) matrix A = [a ;j] is t he
a1 1 al2
a21 a22
am i a m2
(see Section 3 . 1 for its properties ) . Note that the t ranspose of a tnatrix T A is sometimes denoted by A .
28 1
DICTI O N A RY OF M ATRICES A N D RELAT E D TERMS
Transposition matrix: An ( m x m ) matrix is a transposition matrix if it is obtained from the unit matrix Im by interchanging two rows or two columns. For example, 1 0 0 0 0
0 0 1 0 0
0 1 0 0 0
0 0 0 1 0
0 0 0 0
1
is a transposi tion matrix . Triangular mat rix: A lower o r upper t ria ngular matrix is a triangular matri x . (See Section 9. 1 4 for the properties . ) Tridiagonal matrix: An ( m x m ) matrix A = (a ;j ] is tridiagonal i f a ;j = 0 for l i - j l abs > 1 , that is, a11 a21 0
a12 a 22 a 32
0 0
0 0
A=
0 0 0
0 0 0
a m - l .m- 1 am m - 1
am - l m anl m
0 a 23 a 33
'
'
'
(See also band m atrix. ) Tripotent mat rix: An ( m
x m
) matrix A is tripotent if A 3 = A . ( See also
idempo t e n t m a t rix. )
Unimodular integer matrix: An ( m ular if det ( A ) = ± 1 .
x m
) integer m a t rix A is unimod
Unimodular polynomial matrix: The ( m x m ) polynomial matrix P( x ) is sai d to be unimodular or i nvertible if det P ( x ) = constant :f. 0 , that is, det P( x) is a constant function . (See Section 1 1 . 3 . ) Unipotent matrix: A n ( m x m ) matrix A is sai d to be unipotent or involut ory if A2 lm . For example, for an ( n x n ) matrix B , =
is unipotent..
)
· -
s
)
H A N DBOOK O F M ATRICES
· �
Unit lower triangular mat rix: An ( m x m ) matrix 1
0 0
0
1
0 for i < j , is a u n it lower ith a ; ; = 1 for i = 1 , . . . , m , and a;j t riangular lllatrix. ( See also lowe r tria ngula r m a t rzx. )
w
=
Unit matrix: A n ( m x 1 11 ) matrix A = [a;j] is a unit ma t rix or zdent ity 0 for i # j and a;; = 1 for i = 1 , . . . , m . m a tri x . denoted by !., . if a;j =
Unit triangular matrix: A u n i t lower or unit upper tria ngular m a t rix i s a unit triangular mat rix. Unit upper triangular matrix: An ( m x m ) matrix
0
1
0
with a ; ; = 1 for i = 1 , . . . . m, and a;j = 0 for i > j , is u nit upper t riangular. ( Sef' also upper· t riaugular m a t rix. ) Unit vector: A n ( m x 1 ) vector a is a unit vector if l l a l / 2 = � = 1 . Unitarily equivalent matrices: The ( m x 11 ) matrices A and B are unitarily P qu i v al e nt if there exists a u n itary ( m x m ) matrix [ : and a u u i t ary ( n x n ) m at r i x � . such that A = [ ! H B � · . ( See also Sect ion 9.15.) ·
Unitarily similar mat rices: The ( m x rn ) matrices A and B are u nitarily sim ilar if a u ni t ary ( m x m ) m atrix U exists such that B = U A [: H (See also Section 9 . 1 5 . ) Unitary matrix: A n ( m x m ) matrix A Section 9. 1 5 for t he p roperties. )
IS
unitary if A H
Unitary stochastic matrix: A n ( m x m ) matrix A = [a;j] is sai d to be unitary stochastic or Sch ur-stoch astzc if there exists a unitary ( m x m ) matrix [! = [u; j ] such t hat a ;j l u ;j l ;ts for i , j = l , . . . , m . =
DICTION A RY OF M ATRICES A N D REL ATED TERMS Upper block t riangular matrix: An (m
x
283
n) matrix
A11
0
= [A ;i ] ' App
0
0
where the Aij are (m; x nj ) m atrices with L:: f= 1 m ; = m , L:j= l ni = n and A;1 = 0 for i > j , is upper block triangular. ( See also upper triangula r matrzx and partitioned matrix. ) Upper Hessenberg matrix: An ( m x m ) matrix A = [a ;j ] is an upper Hessenberg matrix or simply a Hesse nberg m atnx if a ;j = 0 for i > j + 1 , that is,
ll
A=
a 1 ,m
a33
a3'm - 1
a3 ' m
0 0
a m - I 'm - 1
a 13
0
an
a 23
a3 2
0 0
0 0
a2 1
...
a 1 'm - 1
a12
a
a 2 'm - 1
a m 'm - 1
a2,m
a m - 1 'm am,m
( See also Section 6 . 2 . 3 . ) Upper triangular matrix: A n ( m
x
m ) matrix a 1m
0 0
0 with a ;j = 0 for properties . )
i
amm
> j is upper triangular. ( See Section 9 . 1 4 for the
Vandermonde determinant: For ) q , . . . , A m E
1
1
1
At
A2
Am
m-1 A1
m- 1 A2
m- 1 Am
= II ( A j j
- Aj ) .
is c alled a Vandermonde determinant . I n other words, the determinant of a Van de rmo nde matrzx is a Vandermonde determinant.
H A N DBOOK O F M ATRICES
284
Vandermonde matrix: An ( m x
),m - 1 1
m
) matrix
),m - 1
.>,m - 1 m
2
" ,i- 1 wr comp l ex numbers Aj , w .1 t h aij = "'i Vandermonde matrix .
J
l, . . . ,
m,
.IS
a
vee operator: For an ( rn x n ) matrix A = [a ;i ] , the vee operator is defined as
vee A = vec ( A ) =
( rnn x 1 ) ,
that is, vee stacks the columns of A in a column vector. ( See Section 7 . 2 for the rules. ) vech operator: For an ( m x defined as
m
vech A = vech ( A )
) matrix A
=
[a ij] , the vech operator ts
( � rn(rn +
1) x 1 ),
anlnl that is, vech stacks the columns of A from the principal diagonal downwards in a col u m n vector. ( See Section 7 . 3 for the rules . )
DICTI O N A RY OF M ATRICES A N D R ELAT E D TERMS
285
Vector norm: A function I I · I I attaching a nonnegative real number l l x l l t o a n ( m x 1 ) vector x is a norm or a vector norm i f the fol lowing three conditions are satisfied for all complex ( m x 1 ) vectors x , y and complex numbers c :
if x #- 0 ,
( i ) l l x ll > 0
( ii ) l l cx II = l c l abs l l x l l , ( i i i ) l l x + Y l l < l l x l l + I I YI I
(triangle inequality ) .
Here l c labs denotes the modulus of c, that is, l c l abs = .JCC w ith c being the complex conj ugate of c . I nstead of defining a norm for all complex ( m x 1 ) vectors, it may be defined for real vectors only. If ( i ) , ( ii ) and ( ii i ) hold for all real ( m x 1 ) vectors and real numbers c , 1 1 · 1 1 is called a norm or vector norm over the field of real numbers ( IR ) . In that case lc l abs is t he absolute value of c . (See Chapter 8 . ) Vector polynomial: The function
p( X I ,
·
.
·
,
Xm )
p :
nl
----+
IR ) defined by
rn
Po + L P, ,; x ; + L L P2,ijXiX1 i=l i= l i = l m
m
. L Pd,i . j Xi . . . Xj +...+L . . i= l i = l is a vector poly nom i al of degree d. For example,
is a vector polynom ial of degree 2 .
Vector produc t : The vector product of t he 3-dimensional vectors ( x , , x 2 , x 3 )' and y = ( y , , Y2 · Y3)' is
r
H A N DBOOK O F M ATRICES
286
Vectorization of a rnatrix: Stacki ng the rows or columns of a m atrix A = [a;j ] in a vector is c al led vectorizat ion of A . Notation:
vee
A=
vee
( A ) = col( A )
rvec( A )
=
=
( mn x l ) ,
(vec( A ' )] ' ,
row ( A ) = vec ( A' ) = rvec( A ) ' . ( See C h apter 7 for details. ) Zero matrix: An ( m x n ) m atrix is a zero or null matrix, denoted by Om or simply by 0 , if all its elements are zero. x n
Zero vector: A 1 1 ( m x 1 ) or ( I its elements are zero.
x
m ) vector is a zero or
n
u/1
vector if all
References
Note: There exists a t remendous number of books on mat rices. A lso a substant ial nu mber of books on linear algebra and other topics such as econometrics and mul tivariate statistical methods contain chapters, sections or appendices on matrices. The number of ar ticles with special m at rix results is even greater. In t he following only those books and articles cited in the text are listed in addi tion t o a number of general and some more special matrix books.
Agaian , S.S. ( 1 975 ) . Hadamard Matrices and Their· A pplications. Springer- Verlag. Berlin . Ait ken , A . C. ( 1 956 ) . Determina nts and Matrices. 9th ed . Oliver and Boyd. Edinburgh . Anderson T . W . ( 1 984 ) . A n Introduc tion to Multivariate Stat istical A naly.i is. 2nd ed n , J ohn Wiley. :--l e w '{ark. Barnett . S. ( 1 979 ) . Mat rix Methods for Engineers and Scimtists. M cGraw- Hill. London. Barnett, S. ( 1 983 ). Polynomials and Linear r'ontrol Systems. M a reel Dekker. :\' ew York. Barnett , S. ( 1 984 ) . Afatrices in Control Theory, ( 2nd edn ) . K rieger, Florid a . Barnet t , S. ( 1 990). Afatrices: Methods and A pplications. Clarendon Press, Oxford . Barnett, S . & C . S torey ( 1 970 ) . Matrix Methods in Stabil1 ty Theory. :\ elson, Londo n . Basilevsky, A . ( 1 98 3 ) . A pplied Matrix A lgebra i n the Statistical Sciences . Nort hH olland, New York. Beckenbach E . F . & R . Bellman ( 1 961 ) . Inequ alities. Springer-Verlag. Berlin . Bellman . R . ( 1 970 ) . Introduction to Matrix A nalysis. 2d. ed . McGraw- H ill. :\ew York. Ben-Israel, A . & T . N . E. G reville ( 1 974 ) . Generalized lnt•erses: Th eory and A pplications. Wiley, New York. Berman, A . , M . Neumann & R. J . Stern ( 1 989 ) . Non-negative Matrices m Systems Theory. Wiley, New York. Berman , A . &. R. J. Plemmons ( 1 97 9 ) . Non-negative Matrices in the Jt.f athematical Scie nces. Academic P ress, New York. Bjorck, A . , R. J . Plem mons & H . Schneider ( eds) ( 1 981 ) . Large Scale Matnr Problems. North H olland, A msterdam. Boullion, T . L. & P. L . Odell ( 1 97 1 ) . Generalized Im•erse Matrices . Wiley. New York.
288
H A N D BOOK OF M AT R I C ES
Bradley, I . & R . L . Meek ( 1 986 ) . Matrices and Soc iety. Pengu in, H armondswor t h . M iddlesex. Bronson, R . ( 1 98 9 ) . Schaum 's O utline of Theory and Pro blems of Mat rix Operations. M cG raw- Hill, New York. Bunch , J . R . & D . J . Rose ( eds) ( 1 976 ). Sparse Matrix Computations. Academic P ress. New York. Campbell , S . L. & C . D. Meyer J r . ( 1 979 ) . Grnerallzed Inverses of Lmwr Transforma tions. Pitman, London . Chatelin, F. ( 1 993 ) . Eigenvalues of Matrices. Wiley, New York. Collar, A . R . & A . Sim pson ( 1 987). Matrices and Engineering Dynamics. El lis H orwood, Chichester. Cullen, C . G . ( 1 966 ) . Matrices and Linear Transformations. Add ison-Wesley. Reading, M ass. Davis, P. J . ( 1 9 7 9 ) . Circ ula nt Matrices. Wiley, New York. Donoghue, \\' . F . J r . ( 1 974 ) . Monotone Matr1x Fun ctwns and A nalytic Conlinualwn. S pringer- Verlag, Berli n . D u ff, l . S . , A . M . Erisman & J . K . Reid ( 1 987). Direct Methods fo r· Spa rse Matrices. Clarendon P ress, Oxford. Fiedler, M . ( 1986 ) . Spec ial Matrices and The1r A pplications in Sumcncal .\falhematics. M artinus N ijhoff, Dord recht. Fran klin, J . ( 1 968 ) . Matrix Theo ry. P rentice- Hall, E nglewood Cliffs, N . J . Gantmacher, F . R . ( 1 959a) . The Theory of Mat rices. 2 vols. Chelsea, New York. Gautmacher, F . R . ( 1 959b). A pplica tions of the Theory of Matrices. l n terscience, :\ ew York. Gohberg, 1 . . P. Lancaster & L . Rodman ( 1 98 2 ) . A-latrix Polynomials. Academic P ress, New York. Gohberg, ! . , P. Lancaster & L . Rodman ( 1 983 ) . Matrices and Indefinite Scolrw Pmducts. Birkh ii.user- Verlag, Boston . Golub, G . H . & C. F . Van Loan ( 1989). Matr·ix Computatio ns. ( 2 nd ed n ) . J ohns Hopkins U n iversity P ress, Baltimore. Graham, A . ( 1 98 1 ) . Kronecker Products and Matrix Calculus with A pplica t ions. Horwood. Chichester, U . K . G raybill, F . A . ( 1983). Matrices with A pplica tions to Statistics. ( 2 nd ed n ) . Wadsworth, Belmon t , California. H einig. G. & R. Rost ( 1 984 ) . A lgebra1c Methods fo r Toeplitz-l1ke Afatnces ond Opera tors . A kademi<"- Verlag, Berlin. llorn, R . A . & C' . R . Joh nson ( 1 98 5 ) . J-fatr·ix A nalysis. Cambridge l" niversit y Press , C ambridge. Horn . R . A . & C . R . Johnson ( 1989 ) . Topic'! in Matrix A nalysis. Cambridge U n iversity Press, Cambridge. Householder, A . S . ( 1 96 4 ) . Th e Theory of ,lyfa/rices in Numer·ical A nalysis. Blaisdell . New York. J ohansen, S. ( 1 995 ). Likelihood Based Inference in Coinlegrated Vector A uloregres -� ive .\Iodds . Oxford U niversity Press, Oxford. Kim, K . H . ( 1 982 ) . Boolean .\latrix Theory and A pplica tions. M arcel Dekker, :"'ew York. Kowalsky, H . ( 1 97 2 ) . L i nmre A lgebra. 6th ed. de Gruy ter, Berlin. l\rishnamu r t hy, E . V . ( 1 98 5 ) . Er-ror-free Polyno mial Matrix Co mputations. Springer Verlag, New York. Lancaster, P. ( 1 96 6 ) . La m bda-ma tr·ices and Vibrating Systems. Pergamon , Oxford . L ancaster, P. ( 1 969 ) . Theory of .Ha tnces. Academic P ress, New York.
R E F E RENCES
289
Lancaster, P. & M . Tismenetsky ( 1 985 ) . The Theory of Afatnces W1th A pplzcalions. 2d ed . Academic P ress, New York . L ii tkepohl, H . ( 1 99 1 ) . Introduction to Multiple T1me Series A n alysis. Springer. Berlin. M acDuffee, C.C. ( 1 946) . The Th eory of Matrices. Chelsea, New York. M agnus, R . J . ( 1 98 8 ) . L inear Structures. Griffin, London . M agnus, J . R . & H . Neudecker ( 1 988 ) - Matrix Differential Calculus W1th A pplicat ions in Statistic.� and Eco no metrics. Wiley, Chichester. M arcus, M . & H . M ine ( 1 964 ) - A Survey of Matrix Theory and Matrix Inequalities. A llyn and Bacon, Boston . Miller, K . S . ( 1 9 8 7 ) . Som e Eclectic Matrix Th eory. K rieger, Florida. Mine. H . ( 1 98 8 ) . Nonnegative Matrices. Wiley, New York. Namboodiri, K . ( 1 984 ) . Matrix A lgebra : A n Introduc tion. Sage Pu blicat ions, BPvPrly Hills. :--; ashed , M . Z . ( ed . ) ( 1 976). Generalized Inverse.� and A pplications. Academic Press, New York. l'< ering, E. ( 1 9 6 3 ) . L mear Algebra and Matrix Theory. 2d ed. \.Viley, New '{ork. Neudecker, H . & L. S huangzhe ( 1 99 3 ) . " M atrix Trace I nequ alities I nvolving SimpiP. K ronecker, and H adamard P roducts." Econo metric Th eory, 9. 690. Neudecker, H . &: L. Shu angz he ( 1 995 ) . " M at rix Trace I nequalit ies I n volving Simple . K ronecker, and H ad amard P roducts - Solut ion ." Econometric Theory, 1 1 . 669 670. N ewman, M . ( 1 972). Integral Matrices. Academic P ress, New York. O rtega, J . M . ( 1 987). Matrix Theory, a Second Co urse. Plenum P ress, New York. Perlis, S . ( 1 95 2 ) . Theo ry of Matrices. Addison-Wesley, Reading, �1ass . Pissanetsky, S . ( 1981 ) . Sparse Matrix Technology. Academic Press, London . Pollock , D . S . G . ( 1 9 7 9 ) . The A lgebra of Econometrics. \\Iiley, Chichester. Pringle, R. M . & A . A . Rayner ( 1 97 1 ) . Generalized lnt!t'r'SF :\1at rices u·ith A pplications to Statistics. G riffin, L ondon. Rao , C . R . ( 1 97 3 ) . Linear Statistical Inference and its A pplications. 2nd ed n , John Wilev, New York. Rao, C . R . &: S . K . Mitra ( 1 97 1 ) . Generalized Inverse of Matnces and its A pplzca twns. Wiley. New York. Rogers, G.S. ( 1 980 ) . Mc1trix Derivatives. M arcel Dekker, N ew York. Searle, S . R . ( 1 982 ) . .\la t rix A lgebra Useful for Stat1sticB. Wiley, New York. Seber, G . A . F . ( 1 981 ) . Multivariate Observations. Wiley, .'lew York. Senet a, E . ( 1 973 ) . Nonnega tive Matrices. Wiley, New York. Stewart, G . W . ( 1 973 ). Introduc tion to Matrix Computa twns. Academic P ress. New York. Su prenenko, D . A . & R . I . Tyshkevich ( 1968). Comm utalit'e Matrices. Academic P ress, New 't'ork. Tu rnbull , H . \\' . ( 1 9.1 0 ) . The Theo ry of Determinants, Ma trices and !nt•a ria n t s . Blackie, London. Tu rnbull, H . W. & A . C. Aitken ( 1 932). A n lntroductwn to the Theory of Ca u o u i m l .\latrices. Blackie, London. Varga, R.S. ( 1 962 ) . Jfa trix Itera tive A nalysis. P rentice- H all , Englewood Cliffs. \ . J . Wilkinson, J . H . ( 1 965 ) . The A lgebra ic Eigen value Problem. Cl are ndon P ress , Oxford . Zurmiihl, R . & S . Falk ( 1 98 4 ) . Matrizen und ihre A n wendungw, Te1l l : Gnmd/agfrJ . Springer, Berlin . Zurmiihl, R. & S . Falk ( 1 986 ) . Ma tnzen und ihrf A nwendungm. Tf'il 2: Nunurischr Jfcthodcn. S pringer, Berlin.
Index absolute norm , 102 absolute value of a matrix, 5, 39, 225 addition of m atrices, 3, 225 adjoint of a matrix, 6, 27, 2 25 adj ugate of a matrix. 226 Aitken's integral, 1 .5 2 an nihilating polynomial of a matrix, 2 1 4 , 226 antimetric m atrix, 226 arithmetic m atrix, 226 asymptotically stable polynomial , 2 1 2 band matrix, 227 block, 227 banded matrix, 227 bidiagonal m atrix, 227 binary matrix, 227 Binet-Cauchy formula, 49 binomial for mula, 37, 38 block matrix, 2, 229 band , 227 circulant , 228 companion , 220, 228 diagonal, 229 H an kel. 229 properties of, 1 44 Toeplitz, 229 triangular, 230 tridiagon al, 230 bordered G ramian mat rix, 230 bordered matrix, 230 Brownian matrix, 230 canonical forms Choleski decomposi tion, 86 congruent, 85 diagonal reduction, 85 echelon, 88 Gram-Schmidt triangular red uction , 87, 92
H ermite, 88, 249 J ordan , 83, 84, 254 polar decomposition , 88 rank factorization , 88 real Choleski decomposition. 92 real J ordan, 89 real Schur decomposi tion , 92 Schur decomposition. 86 singular value decomposition , 85. 9 1 Smith, 222 spectral d ecomposition , 85, 90 t riangular factorization, 87 Cauchy-Schwarz ineq uality, 4 3 , 54, 104. 111 Cayley-H amilton theorem, 67, 2 1 6 centro- Hermitian matrix, 23 1 centro-symmetric matrix. 23 1 chain r ul(' for derivatives. 1 74 , 176, 202, 206 characteristic determinant of a matrix, 1 2 , 23 1 characteristic equation of a matrix, 1 2 , 23 1 characteristic matrix, 220, 2 3 1 characteristic polynomial of a matrix. 1 2 , 63, 2 1 4 , 2 3 1 characteristic root, 1 2 , 6 3 , 231 generalized, 245 in the metric of another mat rix. 23 1 characteristic value, 12, 63 , 231 generalized , 245 in the metric of another matrix, 64 characteristic vector, 1 3 , 64, 23 2, 271 i n the metric of another matrix, 64, 232 check matrix, 232, 265 Choleski decomposition , 86, 1 54 of a real matrix, 92, 155 circulant matrix, 1 1 3 , 232 block, 228
')9') -
-
properties of, 1 1 3 cofact or matrix, 232 cofactor of an element of a m atrix, 6, 232 cogredient m atrices, 233 col umn d imension of a matrix, I , 233 column rank of a matrix, 12, 233 colum n regular matrix, 233 col umn stochastic matrix, 233 col u m n vector , 2, 233 column vectorization of a matrix, 233 column vectorization operator, 95 colum n-sum norm, ! 0 3 properties of, I I 0 common divisor of polynomials, 2 1 2 common factor of poly nomials, 2 1 2 common left divisor of polynomial mat rices, 2 1 9, 234 greatest, 24 7 common right divisor of poly nomial matrices. 2 1 9, 231 greatest, 247 com mutation m atrix, 9 , 3 1 , 97, 1 1 5 , 1 23 , 1 2 7 , 234 properties of, 1 1 5 commuting m atrices, 2 34 companion m atrix, 2 1 3 , 234 block, 228 compatible norms, I 0 I , 235 complex conjugate m atrix, 235 complex mat rix, I . 235 com pou nd m at rix, 235 condition number of a m atrix, 2 36 conditional inverse matrix, 236 conformable m atrices, 236 congruent canonical for m of a matrix, 85, 137 congruent matrices, 236 conjugate of a m atrix. 5, 24, 2 3 6 conj ugate t ranspose of a matrix, 5, 25, 236 conjunctive matrices, 236 cont rollability matrix, 236 convergent m atrix, 1 1 9, 236, 277 properties of, 1 1 9 convergent polynomial , 2 1 1 coprime polynomials, 2 1 2 cosine of a m atrix, 2 1 5. 236 Courant-Fischer t heorem, 67, 68, 1 3 3, 1 59
I N DEX D-inverse, 236 decomposable matrix, 237, 2 7 0 decomposition of orthogonal matrix . 144 defective matrix. 237 deficient matrix, 237 definite matrix, 237 degree of a poly nomial, 2 1 1 of a polynomial matrix, 2 1 7 , 237 dependent vectors, 23 7 derivative matrix, 237 derivatives of H ad amard products, 1 84 , 1 92. 1 99 of inverse m atrix, 1 98 of K ronecker products, 1 84 , 1 92 , 1 99 of the deter minant, 1 8 1 of the trace, 177 derived norm, 237 derogatory matrix, 2 1 4 , 237 determinant, 6, 47, 238 Binet-Cauchy formula for, 49 derivatives of, 1 8 1 inequalities, 54 J acobian, 1 73 , 204 of duplication m atrix terms, 5 1 , 53 of elimination matrix terms, 52. 53 of partitioned matrix, 49 optimization of, 56 Vandermonde, 49, 283 diagonal dominant matrix, 238 diagonal m atrix, 5. 9, 1 20 , 238 block, 229 proper ties of, 1 20 diagonal of a matrix. 23 8, 259 dominating, 239 principal , 2 secondary, 273 diagonal reduction of a matrix , 60, 85 diagonal similar matrix, 238 diagonalizable matrix, 2 1 6 , 238 difference of matrices. 3, 15 dimension of a m atrix, I , 239, 264 colu mn, 233 row, 272 direct product, 3, 239, 279 direct sum, 4 , 22, 239 divergent polynomial , 2 1 1 dividing polynomial , 2 1 2 di visibility of a poly nomial matrix left, 256
I N DEX right , 271 divisor greatest com mon left, 247 greatest common right, 247 of a polynomial , 2 1 2 of a polynomial matrix left, 256 right, 271 dominating non negative matrix, 239 dominating pri ncipal diagonal, 239 dou bly centered matrix, 239 doubly quasi-stoch astic m atrix, 240 doubly stochastic mat rix, 1 39 , 240 Drazin i nverse, 240 du plication m atrix, 9 , 3 1 , 5 1 , 53, 97, 1 1 8, 122, 127, 1 30, 1 57, 1 65 , 240 properties of, 122, 149 dyadic prod uct, 24 1 echelon form, 88, 24 1 effective inverse mat rix, 24 1 eigenvalue, 1 2 , 63, 64 algebraic multiplicity, 226 generalized , 245 geometric multiplicity, 246 H opf's theorem for, 1 4 2 in the met ric of anot her matrix, 64, 24 1 i nclusion principle, 7 3 , 135, 1 60 inequalities , 72, 1 34 , 1 59 multiplicity of, 262 of a H ermitian matrix, 1 33 of a matrix, 24 1 of a poly nomial m at rix, 2 1 8 , 24 1 of a positive definite matrix. 1 53 of a sym met ric matrix, 1 58 Poincare's separation theorem, 74, 1 36, 1 6 1 simple, 63, 275 Weyl's theorem, 75, 1 36 , 1 62 eigenvector, 1 3 , 64, 242, 271 generalized , 245 in the metric of another matrix, 64, 242 left, 256 right, 271 eigenvector matrix, 24 1 , 261 elementary d ivisors of a polynomial matrix, 220, 242 elementary m atrix, 242 elementary operations, 1 2
293 for a matrix, 242 for a polynomial matrix, 2 1 9, 242 elementary polynomial matrix, 2 1 9 , 242 elements of a matrix, 1 elementwise product, 3, 243 elimination matrix, 9, 3 1 , 52 , 53, 97. 1 1 8, 1 23 , 126, 1 27, 1 65, 243 properties of, 127 entries of a matrix, 1 EP-matrix, 243 equality of matrices, 243 equivalent matrices, 243 equivalent poly nomial m at rices, 220. 243 ergodic matrix, 243 Euclidean norm , 103 properties of, 1 0 9 exchange matrix, 243, 270 exponential fu nction of a matrix, 2 1 5 . 24 4 exponential matrix, 244 factor of a poly nomial, 2 1 2 Fischer's inequality for determinants, 54, 1 47, 1 5 2 flip matrix, 244, 270 Fourier matrix, 1 1 4 , 244 Frobenius inequality, 6 1 Frobenius matrix, 244 Frobenius norm, 1 03 fu nction of a matrix, 259 g-circu]ant matrix, 245 g- inverse matrix, 24 5 generalized characteristic root, 245 generalized charac teristic value, 245 generalized eigenvalue, 245 generalized eigenvector of a mat rix, 245 generalized inverse of a matrix, 7, 32, 245, 269 generalized inverse rule for derivatives. 209 generalized permu tation mat rix . 140. 245 geometric matrix, 246 geometric multiplicity of an eigenvalue, 246 Givens matrix, 246 gradient matrix, 246 gradient vector, 1 7 1 , 1 74 , 247 G ram matrix, 247
294
I N D EX
Gram Schmid t triangular red uction of a m atrix, 87 of a nonsi ngu lar matrix, 87 of a real m atrix, 92 of a real nonsingular matrix, 92 of a real square matrix, 92 of a squ are matrix, 8 7 Gramian matrix, 247 bordered , 230 greatest common divisor of poly nomi als, 2 1 2 greatest common factor of polynomials, ) } 'J
'
-
greatest common left d ivisor of polynomial matrices, 2 1 9 , 2 4 7 greatest com mon right divisor of poly nomial m atrices. 2 1 9, 247 group inverse of a m atrix, 247 Holder's i nequality, 4 4 , I l l H adamard m atrix, 247 H ad amard prod uct, 3, 20, 248 derivatives of, 1 8 4 , 1 92, ! 99 H adamard 's i nequ ali ty, 54 half- vectorization of a matrix, 8, 96, 248 half- vectorization operator, 96, 248 H amiltonian matrix, 248 H ankel matrix, 24 9, 265 block, 229 harmonic matrix, 24 9 H elmert matrix, 249 Hermite canonical form , 88, 249 H ermitian adj oi nt of a matrix, 5, 250 H ermitian form, 1 2 , 250 negative defi nite, 262 negative semidefi nite, 262 positive defi ni te, 267 positive semidefinite, 267 Hermitian matrix, 1 0 , 1 3 1 , 250 decomposition of, 1 37 eigenvalues of, 1 33 properties of, 1 3 1 H cssenberg matrix, 250, 283 lower, 258 Hessian matrix, 172, 1 75 , 250 of the log d eterminant, 182 of t he t race, 1 79 Hilbert matrix, 251 H ilbert-Schmidt norm , 103 Hopf's theorem for eigenvalues, 78, 1 42
H ouseholder transformation, 1 3 2, 1 68 . 251 H u rwitz matrix, 2 1 4 , 251 H u rwitz theorem , 2 1 7 idempotent matrix, 1 0 , 1 38, 2 5 1 properties of, 1 38 iden tity matrix, 2, 1 0 , 252 imaginary matrix, 252 imprimitive matrix, 252 inclusion principle for eigenvalues, 7 3 , 1 35 , 1 60 indefi nite matrix, 252 independent vectors, 252 index of a square matrix, 252 induced matrix norm , 1 02, 252, 264 properties of, I 08 inequalities Cauchy-Schwarz, 43, 54, 1 04 , I l l Fischer's, 54, 147, 1 52 for eigenvalues, 72, 1 34 , 1 59 for norms, I l l Frobenius, 6 1 Holder 's, 4 4 , I l l H adamard 's, 54 Minkowski's, 4 4 , 55 Oppenheim's, 56 Ostrowski-Taussky, 56 Schu r's, 43 singular value, 80 Sylvester's, 6 1 triangle, 1 0 1 inequality relations for matrices, 4 inertia of a square matrix, 252 in ner prod uct, 1 02 , 252 over the complex numbers, I 02 over the real numbers, I 02, 253 special, I 04 integer matrix, 253 u nimodular, 2 8 1 integral matrix, 253 i nvariant factors of a poly nomial ma trix, 220, 253 i nvariant poly nomials of a poly nomial matrix, 220, 253 inverse matrix, 7, 27, 253 conditional , 236 derivatives of, 1 98 Drazin, 240 effective, 24 1 g-, 245
I N DEX matrix sum, 1 5 m aximum norm , 1 0 3 properties of, I 09 m aximum singular value norm, I 04 Metzler matrix, 260 minimal matrix norm, 102 mini mal poly nomial of a matrix, 2 1 5, 261 Minkowski m atrix, 259, 261 Minkowski norm, 103 Minkowski's inequality, 4 4 , 55 Minkowski-M etzler mat rix, 261 minor of a m atrix, 6, 26 1 principal , 268 minor of an element of a matrix, 6, 261 modal mat rix, 261 modulus of a matrix, 5, 39 modulus of a vector. 257, 261 monic poly nomial , 2 1 2 monic polynomial matrix, 2 1 8 , 26 1 monomial matrix, 261 monotone nor m , 1 02 Moore -Penrose inverse, 7, 34, 262 M P-inverse of a matrix, 262 multiple of a polynomial matrix left, 257 right, 271 multiplication by a scalar, 3 , 18, 262 of matrices, 3, 1 6 , 260 multiplicative norm, 1 02 , 262 multiplicity of an eigenvalue, 63 , 262 algebraic, 63 geometric, 63 negative defi nite Her mitian form , 262 negative definite matrix, 10, 262 properties of, 150 negative definite quadratic form, 262 negative matrix, 262 negative semidefi nite Her mitian form, 262 negative semidefi nite m at rix, 1 0 , 263 properties of, 1 50 negative semidefi nite quadratic form , 263 nilpotent mat rix, 1 2 1 , 263 nonderogatory matrix, 263 non negative defi nite matrix, 263 non negative m atrix, I 0, 1 39, 263 dominating, 239
properties of, 139 nonpositive definite matrix, 263 nonpositive matrix, 263 nonsingular matrix, I 0, 253, 263, 270 norm, 1 0 1 , 263 absolute, 102 column-su m , I 03, 1 1 0 compatible, 1 0 1 , 235 derived from an inner produc t , 1 03 , 237 Euclidean, 1 03, 1 09 Frobenius, 1 03 Hilbert-Schmidt, 1 03 induced, 1 0 2 isometry for, 1 0 1 , 254 l1 , I 09 12 ' 1 03 lp, 1 03 least upper bound, 1 02, 256 lub, 1 02 M ah alanobis, 1 03 matrix, 1 0 1 , 1 02 maximum, 1 03, 1 09 maximum singular value, 1 04 minim al, I 02 Minkowski, 1 0 3 monotone , I 02 multiplicative, 102, 262 of a matrix, 260 operator, 1 02, 1 04 , :264 over the complex numbers, 1 0 1 over the real numbers, 1 0 I row-su m, 1 03, 1 1 0 Schur, 1 03 self-adjoint , 102 spectral, 1 04 , 1 1 0 submultiplicative, 102, :278 sup, 102 unitarily invariant, 1 02 vector, 1 0 1 , 285 norm i nequalities, I l l normal form of a matrix, 264 normal matrix, 10, 1 1 4 , 1 3 2, 1 4 3 , 264 null matrix, I 0, 264 null vector, 264 nu merical range of a matrix, 264 operator norm, 1 02, 1 04 , 264 Oppenheim's inequality, 56 order of a matrix, I , 264 orthogonal complement matrix, 265
I N D EX common left divisor of, 2 I 9 , 234 common right divisor of, 2 1 9, 234 equivalent, 220, 243 greatest common left divisor of, 2 1 9 greatest common right divisor of, 2 1 9 left coprime, 2 1 9, 256, 270 left quotient of, 2 I 8, 257 left remai nder of, 2 1 8 , 257 relatively left prime, 2 I 9 , 256, 270 relatively right prime, 2 1 9 , 270, 2 7 1 right coprime, 2 1 9, 270, 2 7 I right q uotient of, 2 I 8 , 2 7 1 right rem ainder of, 2 I 9 , 2 7 1 skew prime, 2 I 9, 276 polynomial mat rix, 2 I 7, 266 block companion mat rix of, 220 com plex, 2 1 7 degree of, 217, 237 eigenvalue of, 2 1 8, 24 1 elementary, 2 I 9, 242 elementary divisors of, 220, 242 elem entary operations for , 2 1 9, 242 invariant factors of, 220, 253 invariant polynomials of, 220, 253 invertible, 2 1 8 , 254 latent root of, 2 1 8 late nt vector of, 2 I 8 , 256 left divisibility of, 2 1 8, 256 left divisor of, 2 1 8 , 256 left multiple of, 2 I 8 , 257 monic, 2 I 8, 2 6 1 operations for, 2 1 7 rank of, 2 I 7, 269 real , 2 1 7, 269 regular, 2 1 8 , 270 right divisibility of, 2 1 9, 2 7 1 right divisor of, 2 I 9, 271 right m ultiple of, 2 I 8 , 2 7 1 unimodular, 2 1 8 , 281 polynomial mat rix operations, 2 I 7 polynomial of a matrix characteristic, 231 minimal , 261 polynomials common divisor of, 2 1 2 common factor of, 2 1 2 copri me, 2 I 2 greatest common divisor of, 2 1 2 greatest common factor of, 2 1 2 least common multiple of, 2 I 2 relatively prime, 2 1 2
INDEX
301
positive defi nite Hermitian form, 267 positive definite m at rix, 1 0 , 267 properties of, 1 50 positive defi nite quadratic form , 267 positive m at rix, 1 1 . 1 39, 267 properties of, 1 3 9 positive semidefi nite H ermitian form, 267 positive semidefi nite m at rix, 1 0 , 267 properties of, 150 positive semidefi n ite q u ad ratic form, 267 postmultiplication by a matrix, 267 power of a m atrix, 7, 37, 267 power senes matrix, 2 1 5 premultiplication by a matrix, 268 primitive m at rix, 268 principal diagonal of a mat rix, 2, 268 principal minor of a m atrix, 6, 268 principal su bmat rix, 268 p rincipal vector of a m atrix, 269 product direct, 3 , 239 dyadic, 24 1 elementwise, 3 , 243 H ad am ard, 3, 20, 248 K ronecker, 3, 1 9 , 256 matrix, 3, 1 6 , 260, 269 scalar , 273 Schur, 3, 273 tensor, 3 , 279 product matrix, 269 product rule for derivatives, 1 74 , 1 76, 1 89, 208 projection m atrix, 269 projector orthogonal, 265 pseudo-inverse mat rix, 269 Pythagoras t heorem , 1 04 quadratic form , 1 2 , 269 negative defi nite, 262 negative semidefinite, 263 positive defi nite, 267 positive semidefinite, 267 quadratic mat rix, 2, 269. 276 quotient of poly nomial m at rices left , 257 right, 2 7 1 rank factorization of
a
m at rix. 60, 8 8
rank inequalities, 6 1 rank of a m at rix, 12, 58, 269 colu m n , 1 2 . 233 row , 1 2 , 272 . ·) 1 -1 , ·)69 . I rn at nx, ran k of a poIy nom1a ratio rule for derivati ves, 1 74 , 1 76 . 208 rational matrix, 269 Rayleigh r atio, 269 Rayleigh- Ritz theorem , 67, 133, 1 5 9 real m atrix, I , 269 negative definite, 1 5 1 negative semidefi nite, 1 5 1 positive defi nite, 150 positive semidefinite, 1 5 1 real poly nomial matrix, 2 1 7 . 269 rectangular m atrix, 2 70 reducible m at rix, 270 reflexive generalized inverse m at rix, 270 regular m atrix, 10, 253, 263, 270 regular polynomial mat rix. 2 1 8 , 270 relatively left prim e polynomial matrices, 2 1 9, 256, 270 relatively prime polynomials, 2 1 2 relatively right prime poly nomial mat rices, 2 1 9. 270, 2 7 1 resu ltant, 2 1 2 resultant m atrix, 270 reverse unit m atrix. 270 Riccati eq uatio n , 2 7 1 right coprime polynomial matrices. 2 1 9. 270. 2 7 1 right divisi bilit y of a polynomial mat rix . 2 1 9 , 271 right divisor of a polynomial mat rix. 2 1 9, 271 right eigenvector, 271 right inverse m atrix, 271 right multiple of a polynomial m at rix, 2 1 8 , 271 right quotient of polynomial m at rices . 2 1 8, 271 right remainder of poly nomial mat rices. 2 1 9 , 271 root of a poly nomial, 2 1 1 multiplicity of, 2 1 1 rotation matrix, 272 Routh matrix, 272 row dimension of a matrix, I , 272 row r ank of a mat rix, 1 2 . 272 row regular mat rix, 272 row stochastic mat rix, 272, 277 _
_
302 row vector, 2 , 272 row vectorization of a m atrix, 272 row vectorization operator, 96 row-sum norm, 1 0 3 properties of, 1 1 0 scalar matrix , 273 scalar multiplication , 3, 1 8 , 273 scalar product, 273 Schu r complement of a matrix, 273 Schu r decom position , 6 9 , 86 of real m atrix, 92 Schu r norm, 1 03 Schu r product, 3 , 273 Schur product theorem , 2 1 , 1 52 Schu r's inequality, 43 Sch u r-stochastic matrix, 273, 282 secondary diagon al of a matrix, 273 self- adjoint m at rix norm, 1 0 2 selfadjoint mat rix, 273 semicirculant mat rix, 274 semidefinite m atrix, 274 seminegative matrix, 274 seminorm, 1 0 1 . 274 �emiort hogon al mat rix , :274 semipositive m atrix, 274 shift matrix, 27 4 similar m atrices, 2 1 6, 275 simple eigenvalue, 63, 275 simple matrix , 1 1 4 , 1 38, 275 simultaneous diagonaliz ation of H ermitian matrices, 86, 1 37 of mat rices . 86, 275 of positive definite and H ermitian matrices, 86, 1 3 7 , ! .5 5 of positive defi nite and symmetric matrices, 86, 92, 1 5 5, 1 63 sine of a matrix, 2 1 5 , 275 singular matrix, 275 singular value, 1 3 , 63, 64, 78, 275 i nequalities, 80 singular value decomposition, 60, 79, 80, 85 of real mat rix, 9 1 skew prime polynomial mat rices , 2 1 9, 276 skew-circulant matrix, 275 skew- Hermitian matrix, 1 3 1 , 276 properties of, 1 3 1 skew-sy mmetric matrix, 1 5 6 , 276 properties of, ! 56
I N DEX
skew-sy mmetric part of a matrix, 276 Smith canonical form, 222 Smith normal form, 222 sparse mat rix, 276 spectral decomposition, 69, 70, 85, 90, 1 37, 1 63 of a H ermitian matrix, 70, 85, 1 37 of a normal matrix, 69, 85 of a real sy mmetric matrix, 70, 90 of a sy mmetric mat rix, 1 63 spectral norm, 1 04 properties of, I I 0 spectral radius, 64, 76, 1 3 9 , 276 properties of, 1 4 1 spectrum of a m at rix, 64, 276 square matrix, 2, 269, 276 index of, 252 inertia of, 252 squ are root decom position, 1 55 of a real mat rix, ! 55 sq u are root of a mat rix , 8, 276 stable mat rix, 1 1 9, 277 properties of, 1 1 9 Stein's theorem , 1 20 Stieltjes matrix, 277 stochastic matrix, 1 39, 272, 277 doubly, 240 properties of, 1 3 9 st rictly lower triangular mat rix, 277 properties of, 164 strictly triangular mat rix, 277 properties of, 164, 1 6 7 strictly u pper triangu lar matrix, 277 properties of, ! 64 striped matrix, 277 su bmatrix, 1 , 278 principal, 268 submultiplicative norm, 1 0 2 , 262, 278 su btraction of mat rices, 3, 278 sum direct, 4 , 22, 239 K ronecker, 256 of mat rices, 3, 1 5 sup norm, I 02, 264 Sylvester matrix, 2 1 3, 278 Sylvester's law , 6 ! symmetric r- Toeplitz matrix, 279 sy mmetric matrix, I I , 1 56, 278 decomposition of, 163 derivatives of, 1 85, 193 eigenvalues of, 1 58
I N DEX
3 03 ,
pr o pe r t i es o f 1 5 6 symmetric part o f a m a t r i x , 27 9 sy mm t> t r i c To ep li t z matrix. 279 tensor prod u c t ,
3 . 279
theorems Cayley·-Hamilton, 67, '.! 1 6 Courant Fisrher. 67. 68 , 1 33 , 1 5 9 ll op f ' s , 78, 142 H u rwi t z . 2 1 7 i n cl usion principle, 73, 1 35, 160 Poincare's separation, 74, 1 36, 1 6 1 Py t hag or as, 1 04 Rayleig h - Ri t z 67, 1 33, 1 59 Schur prod u c t , 1 52 Stein's. 1 20 Wevl - 's. 75.. 136 . 1 62 To e pl i t z mat rix, 1 1 4 , 2 7 9 block, 2 2 9 ,
sy m m e t ric. 279
t o t al ly non nega t i v e m a t rix. t o t a ll.v p osit i ve matri x . 280 t r ac !' of
280
a matri x , 4 . 4 1 . 280
d e rivat i ves of. 1 7 7 i n eq u al i t i es , 43 o p t i mi z at i o n
of, 4.1 transition m at rix, 277, 280 t ranspose of a m a t ri x , .'l , 2 3 . 280 t r a n s p os i t i on matrix. 28 1
t riangle inequality, 1 0 1 t r i a n gu l a r
fact oriz at ion of a m a t r i x , 60, 87 of a n o n si n g u lar mat r i x , 8 7 t r i an g ula r mat rix. I I . 28 1 b l o ck , 230 deriva tives of, 1 87, 1 95 l owe r , 1 64 . 2 5 9 lower hlock . 2.58 pro p e r t i e � of, 164 s t rict ly, 1 64 , 277 str i c t ly lower. 277 s t r ictl y u pp e r , 277 unit , 282 u n i t lower, 282 unit upper, 282 u p p!' r . 1 64 , 283 u p per b l o c k , 2 83 t r id iag o nal m a t r i x , 1 4 0 , 28 1 block, 230 t r ip ot e n t m a t r i x , 2 8 1 uni mod ular in t ege r
mat rix, 281
u nim odu lar
m at r ix ,
po l y n o m ial
218,
254, 28 1 u ni p o t e nt m a t r i x, 28 1 unit lower t riangu lar matrix. 282 unit mat rix, 2, 282 unit triangular ma t ri x , 2 8 2 unit u ppe r t ria n gu la r matr i x . 2 8 2 unit vector. 282 unitarily eq u ivale n t mat r in s, 28 2 u nita r il y invariant nor m . 1 0'.! u n i t arily sim ilar ma t r i c e s , 282 unitary m at rix. I I . 282 p r o p e rt i e s o f, 16 7 unitary stochastic matrix, 27.3. ·2 82 u n s t a b l e polynomial. 2 1 2 u pper block t r ian g u lar m at rix. 28.1 u pper H e s e n b e r g mat rix, 283 u p p e r t r i a n g u la r m a t r i x , I I . 28 1 . 28.1 prop!'rt i!'s of. 1 64 ·
•
·
·
s
Vandcrmond!' d!' t e rm i n a n t , 4 9 ,
Vandermo nd!' m a t r i x . 2 8 4 vee operator,
8,
9S, 97, I n ,
28:1
1 2 8 284
vech operator, 8 , 9 6 , 9 9 . I D . 1 28 . 28 4 •
vector, 2 ch aract!'rist ic , 64 colum n . 2, 233 g r ad i e n t . 1 7 1 . 247 le n gt h of, 257. 261 modulus of, 2.) 7 . 261 null. 264 row , 2 , 2 7 2
unit, '28'l zero, 286
171 vector norm, 1 0 I , 28.) ve c t or poly nom ial , 285 ve ct o r p r od u c t , 285 vectorization of a matrix, 8. column, 233 half, 248 row , 272 vect o r derivatives,
vec torizat ion operato r . 9 .')
col u m n , 95
row, 96 v e ct o rs
237 in d e pe n d e n t , 252 linearly d e pen den t , 258 li n ea r l y i nd epen d en t , 258 o r t h og o n a l , 26.5 depe n d e n t .
9.5.
23.1, 285
:304 orthonorm al , 265 Weyl's theorem for eigenvalues, 75, 1:36, 162 zero matrix, 1 0 , 264, 286 zero vector, 264, 286
