SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications
This Page Intentionally Left Blank
SVD AND SIGNAI~ PROCESSING, III Algorithms, Architectures and Applications
Edited by
MARC MOONEN BART DE MOOR ESAT-SISTA Department of Electrical Engineering Katholieke Universiteit Leuven Leuven, Belgium
1995 ELSEVIER Amsterdam
Lausanne
New York
Oxford
Shannon - Tokyo
ELSEVIER SCIENCE B.V. Sara Burgerhartstraat 25 P.O. Box 211, 1000 AE Amsterdam, The Netherlands
Library of Congress C a t a ] o g l n g - l n - P u b l l c a t t o n
Data
SVD and s i g n a l p r o c e s s i n g I I I : a l g o r i t h m s , a r c h i t e c t u r e s , and a p p l i c a t i o n s / e d i t e d by Marc Moonen, B a r t De Moor. p. cm. Includes b l b l l o g r a p h l c a l references and index. ISBN 0-444-82107-4 (a]k. paper) I. Signal p r o c e s s i n g - - D l g i t a ] techniques--Congresses. 2. Decomposition (Mathematlcs)--Congresses. I . Moonen, Marc S., 1963. I I . Moor, Bart L. R. de., 1960. I I I . T i t l e : SVD and signal processing three. IV. T i t l e : SVD and signal processing 3. TK5102.9.$83 1995 621.382'2'015194--dc20 95-3193 CIP
ISBN: 0 444 82107 4 9 1995 Elsevier Science B.V. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the publisher, Elsevier Science B.V., Copyright & Permissions Department, P.O. Box 521, 1000 AM Amsterdam, The Netherlands. Special regulations for readers in the U . S . A . - This publication has been registered with Center Inc. (CCC), 222 Rosewood Drive, Danvers, MA, 01923. Information can be about conditions under which photocopies of parts of this publication may be made copyright questions, including photocopying outside of the U.S.A., should be referred Elsevier Science B.V., unless otherwise specified.
the Copyright Clearance obtained from the CCC in the U.S.A. All other to the copyright owner,
No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. pp. 375-382, 415-422, 467-474: Copyright not transferred. This book is printed on acid-free paper. Printed in The Netherlands.
PREFACE This book is a compilation of papers dealing with matrix singular value decomposition and its application to problems in signal processing. Algorithms and implementation architectures for computing the SVD are discussed, as well as a variety of applications such as systems and signal modeling and detection. The book opens with a translation of a historical paper by Eugenio Beltrami, containing one of the earliest published discussions of the singular value decomposition. It also contains a number of keynote papers, highlighting recent developments in the field. The material in the book is based on the author contributions presented at the 3rd International Workshop on SVD and Signal Processing, held in Leuven, August 22-25, 1994. This workshop was partly sponsored by EURASIP and the Belgian NFWO (National Fund for Scientific Research), and organized in co-operation with the IEEE Benelux Signal Processing Chapter, the IEEE Benelux Circuits and Systems Chapter. It was a continuation of two previous workshops of the same name which were held at the Les Houches Summer School for Physics, Les Houches, France, September 1987, and the University of Rhode Island, Kingston, Rhode Island, U.S.A., June 1990. The results of these workshops were also published by Elsevier Science Publishers B.V. (SVD and Signal Processing, Algorithms, Applications and Architectures, edited by E.F. Deprettere, and SVD and Signal Processing, Algorithms, Analysis and Applications, edited by R. Vaccaro). It has been a pleasure for us to organize the workshop and to work together with the authors to assemble this book. We feel amply rewarded with the result of this co-operation, and we want to thank all the authors here for their effort. We would be remiss not to thank the other members of the workshop organizing committee, Prof. Ed Deprettere, Delft University of Technology, Delft, The Netherlands, Prof. Gene Golub, Stanford University, Stanford CA, U.S.A., Dr. Sven Hammarling, The Numerical Algorithms Group Ltd, Oxford, U.K., Prof. Franklin Luk, Rensselaer Polytechnic Institute, Troy NY, U.S.A., and Prof. Paul Van Dooren, Universit@ Catholique de Louvain, Louvain-la-Neuve, Belgium. We also gratefully acknowledge the support of L. Delathauwer, who managed all of the workshop administration.
Marc Moonen Bart De Moor
Leuven, November 199~
This Page Intentionally Left Blank
vii
CONTENTS
A short introduction to Beltrami's paper
On bilinear functions E. Beltrami, 1873 English Translation" D. Boley
P A R T 1. K E Y N O T E P A P E R S
1. Implicitly restarted Arnoldi/Lanczos methods and large scale SVD applications D. C. Sorensen
21
2. Isospectral matrix flows for numerical analysis U. Helmke
33
3. The Riemannian singular value decomposition B.L.R. De Moor
61
4. Consistent signal reconstruction and convex coding N. T. Thao, M. VetterIi
79
P A R T 2. A L G O R I T H M S A N D T H E O R E T I C A L C O N C E P T S
5. The orthogonal qd-algorithm U. von Matt
6. Accurate singular value computation with the Jacobi method Z. Drmad
99
107
viii
Contents
7. Note on the accuracy of the eigensolution of matrices generated by finite elements Z. DrmaS, K. Veselid
115
8. Transpose-free Arnoldi iterations for approximating extremal singular values and vectors M.W. Berry, S. Varadhan
123
9. A Lanczos algorithm for computing the largest quotient singular values in regularization problems P.C. Hansen, M. Hanke
131
10. A QR-like SVD algorithm for a product/quotient of several matrices G.H. Golub, K. SoMa, P. Van Dooren
139
11. Approximating the PSVD and QSVD S. Qiao
149
12. Bounds on singular values revealed by QR factorizations C.- T. Pan, P. T.P. Tang
157
13. A stable algorithm for downdating the ULV decomposition J.L. Barlow, H. Zha, P.A. Yoon
167
14. The importance of a good condition estimator in the URV and ULV algorithms 175 R.D. Fierro, J.R. Bunch
15. L-ULV(A), a low-rank revealing ULV algorithm R.D. Fierro, P.C. Hansen
183
16. Fast algorithms for signal subspace fitting with Toeplitz matrices and applications to exponential data modeling S. Van Huffel, P. Lemmerling, L. Vanhamme
191
17. A block Toeplitz look-ahead Schur algorithm K. Gallivan, S. Thirumalai, P. Van Dooren
199
Contents 18. The set of 2-by-3 matrix pencils - Kronecker structures and their transitions under perturbations - And versal deformations of matrix pencils B. KdgstrSm
ix
207
19. J-Unitary matrices for algebraic approximation and interpolation -The singular case P. Dewilde 209
P A R T 3. A R C H I T E C T U R E S
AND REAL TIME IMPLEMENTATION
20. Sphericalized SVD Updating for Subspace Tracking E.M. Dowling, R.D. DeGroat, D.A. Linebarger, H. Ye
227
21. Real-time architectures for sphericalized SVD updating E.M. Dowling, R.D. DeGroat, D.A. Linebarger, Z. Fu
235
22. Systolic Arrays for SVD Downdating F. Lorenzelli, K. Yao
243
23. Subspace separation by discretizations of double bracket flows K. Hiiper, J. GStze, S. Paul
251
24. A continuous time approach to the analysis and design of parallel algorithms for subspace tracking J. Dehaene, M. Moonen, J. Vandewalle
259
25. Stable Jacobi SVD updating by factorization of the orthogonal matrix F. Vanpoucke, M. Moonen, E.F. Deprettere
267
26. Transformational reasoning on time-adaptive Jacobi type algorithms H.W. van Dijk, E.F. Deprettere
277
27. Adaptive direction-of-arrival estimation based on rank and subspace tracking B. Yang, F. Gersemsky
287
28. Multiple subspace ULV algorithm and LMS tracking S. Hosur, A.H. Tewfik, D. Boley
295
Conmn~ P A R T 4. A P P L I C A T I O N S
29. SVD-based analysis of image boundary distortion F.T. L uk, D. Vandevoorde
305
30. The SVD in image restoration D.P. O'Leary
315
31. Two dimensional zero error modeling for image compression J. Skowronski, L Dologlou
323
32. Robust image processing for remote sensing data L.P. Ammann
333
33. SVD for linear inverse problems M. Bertero, C. De Mol
341
34. Fitting of circles and ellipses, least squares solution W. Gander, R. Strebel, G.H. Golub
349
35. The use of SVD for the study of multivariate noise and vibration problems D. Otte
357
36. On applications of SVD and SEVD for NURBS identification W. Ma, J.P. Kruth
367
37. A tetradic decomposition of 4th-order tensors : Application to the source separation problem J.-F. Cardoso
375
38. The application of higher order singular value decomposition to independent component analysis L. De Lathauwer, B. De Moor, J. Vandewalle
383
39. Bandpass filtering for the HTLS estimation algorithm: design, evaluation and SVD analysis H. Chen, S. Van Huffel, J. Vandewalle
391
Con~n~ 40. Structure preserving total least squares method and its application to parameter estimation H. Park, J. B. Rosen, S. Van HuffeI
399
41. Parameter estimation and order determination in the low-rank linear statistical model R.J. Vaccaro, D.W. Tufts, A.A. Shah
407
42. Adaptive detection using low rank approximation to a data matrix L P. Kirsteins, D.W. Tufts
415
43. Realization of discrete-time periodic systems from input-output data E.L Verriest, J.A. Kullstam
423
44. Canonical correlation analysis of the deterministic realization problem J.A. Ramos, E.L Verriest
433
45. An updating algorithm for on-line MIMO system identification M. Stewart, P. Van Dooren
441
46. Subspace techniques in blind mobile radio channel identification and equalization using fractional spacing and/or multiple antennas D.T.M. Slock
449
47. Reduction of general broad-band noise in speech by truncated QSVD: implementation aspects S.H. Jensen, P.C. Hansen, S.D. Hansen, J.A. Sorensen
459
48. SVD-based modelling of medical NMR signals R. de Beer, D. van Ormondt, F.T.A.W. Wayer, S. Cavassila, D. Graveron-Demilly, S. Van Huffel
467
49. Inversion of bremsstrahlung spectra emitted by solar plasma M. Piana
475
Authors index
485
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
A SHORT
INTRODUCTION
TO BELTRAMI~S
PAPER
The last 30 years the singular value decomposition has become a popular numerical tool in statistical data analysis, signal processing, system identification and control system analysis and design. This book certainly is one of the convincing illustrations of this claim. It may come as a surprise that its existence was already established more than 120 years ago, in 1873, by the Italian geometer Beltrami. This is only 20 years after the conception of a matrix as a multiple quantity by Cayley in his 'Memoir on Matrices' of 1858. Says Lanczos in [8, p.100]: "The concept of a matrix has become so universal in the meantime that we often forget its great philosophical significance. To call an array of letters by the letter A was much more than a matter of notation. It had the significance that we are no longer interested in the numerical values of the coe.O~cients aij. The matrix A was thus divested of its arithmetic significance and became an algebraic operator." And Sylvester himself says: "The idea of subjecting matrices to the additive process and of their consequent amenability to the laws of functional operation was not taken from it (Cayley's ' M e m o i r ' ) but occured to me independently before I had seen the memoir or was acquainted with its contents." Approximately twenty years thereafter, there was a sequence of papers on the singular value decomposition, starting with the one by Beltrami [3] that is translated here. Others were by Jordan [7], Sylvester [11] [12] [13] (who wrote some of his papers in French and called the singular values, les multiplicateurs canoniques.), and by Autonne [1] [2]. After that, the singular value decomposition was rediscovered a couple of times and received more and more attention. For the further history and a more detailed explanation of these historical papers, we refer to the paper by Stewart [9] and a collection of historical references in the book by Horn and Johnson [6, Chapter 3]. We would like to thank Prof. Dan Boley of the University of Minnesota, Minneapolis, U.S.A., for his translation of the Beltrami paper.
E. Beltrami References
[1] Autonne L. Sur les matrices hypohermitiennes et les unitaires. Comptes Rendus de l'Acadhmie des Sciences, Paris, Vo1.156, 1913, pp.858-860. [2] Autonne L. Sur les matrices hypohermitiennes et sur les matrices unitaires. Ann. Univ. Lyon, Nouvelle S6rie I, Fasc. 38, 1915, pp.177. [3] Beltrami E. Sutle Funzioni Bilineari, Giornale di Mathematiche, Battagline G., Fergola E. (editors), Vol. 11, pp. 98-106, 1873. [4] Cayley A. Trans. London Phil. Soc., Vo1.148, pp.17-37, 1858. Collected works: Vol. II, pp.475-496.
[5] Dieudonn4 J.
Oeuvres de Camille Jordan. Gauthier-Villars & Cie, Editeur-Imprimeur-Libraire, Paris, 1961, Tome I-II-III.
[6] Horn R.A., Johnson C.R.
Topics in matrix analysis. Cambridge
University Press, 1991.
[7] Jordan C. Mdmoire
sur les formes bilindaires. J. Math. Pures Appl. II, Vol.19, pp.35-54, 1874. (see reprint in [5, Vol.III, pp.23-42]).
[8] Lanczos C.
Linear Differential Operators. D. Van Nostrand Com-
pany Limited London, 1961. [9] Stewart G.W. On the early history of the singular value decomposition. SIAM Review, December 1993, Vol.35, no.4, pp.551-566. [10] Sylvester J.J. C.R. Acad. Sci. Paris Vo1.108, 1889 pp.651-653Mess. of Math. Vol.19, 1890, pp.42-46.
[ii] Sylvester J.J.
Sur la rdduction biorthogonale d'une forme lindolin'eaire h sa forme canonique. Comptes Rendus, CVIII., pp.651-
653, 1889 (see also in [10, p.638]).
[12] Sylvester J.J.
A new proof that a general quadric may be reduced to its canonical form (that is, a linear function of squares) by means of a real orthogonal substitution. Messenger of Mathematics, XIX,
pp.l-5, 1890 (see also [10, p.650])
A Short Introduction to Beltrami's Paper
[13] Sylvester J.J. On the reduction of a bilinear quantic of the
n TH
order to the form of a sum of n products by a double orthogonal substitution. Messenger of Mathematics, XIX, pp.42-46, 1890 (see
also [10, p.654]).
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 1995 Elsevier Science B.V.
ON BILINEAR
FUNCTIONS
(SULLE FUNZIONIBILINEARI) by E. B E L T R A M I
English Translation- Copyright @1990 Daniel Boley
A Translation from the Original Italian of one of the Earliest Published Discussions of the Singular Value Decomposition.
Eugenio Beltrami (Cremona, 1835 - Rome, 1900) was professor of Physics and Mathemat-
ics at the University of Pavia from 1876 and was named a senator in 1899. His writings were on the geometry of bilinear forms, on the foundations of geometry, on the theory of elasticity, electricity, and hydrodynamics, on the kinematics of fluids, on the attractions of ellipsoids, on potential functions, and even some in experimental physics.
E. Beltrami
GIORNALE
DI MATEMATICHE AD USO DEGLI S T U D E N T I
DELLE
UNIVERSITA
ITALIANE
PUBBLICATO PER CURA DEI PROFESSORI G. B A T T A G L I N I ,
E. F E R G O L A
IN UNIONE DEI PROFESSORI E. D ' O V I D I O , G. T O R E L L I e C. S A R D I
Volume XI.-1873
NAPOLI BENEDETTO PELLERANO EDITORE LIBRERIA SCIENTIFICA E INDUSTRIALE Strada di Chiaia, 60
On Bilinear Functions
JOURNAL
OF MATHEMATICS FOR USE OF THE STUDENTS
OF THE ITALIAN UNIVERSITIES PUBLISHED UNDER THE DIRECTION OF THE PROFESSORS G. B A T T A G L I N I , E. F E R G O L A TOGETHER WITH THE PROFESSORS E. D ' O V I D I O , G. T O R E L L I and C. S A R D I
Volume XI.-1873
NAPLES BENEDETTO PELLERANO PUBLISHER SCIENTIFIC AND INDUSTRIAL BOOKSTORE 50 Chiaia Road
English Translation- Copyright @1990 Daniel Boley
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
ON BILINEAR FUNCTIONS by
E. B E L T R A M I
The theory of bilinear functions, already the subject of subtle and advanced research on ~he part of the eminent geometers KttONECKER and CHttISTOFFEL (Journal of BOttCHAttDT t. 68), gives occasion for elegant and simple problems, if one removes from it the restriction, almost always assumed until now, that the two series of variables be subjected to identical substitutions, or to inverse substitutions. I think it not entirely unuseful to treat briefly a few of these problems, in order to encourage the young readers of this Journal to become familiar ever more with these algebraic processes that form the fundamental subject matter of the new analytic geometry, and without which this most beautiful branch of mathematical science would remain confined within a symbolic geometry, which for a long time the perspicuity and the power of pure synthesis has dominated. Let f - ~rsCrsXrYs be a bilinear function formed with the two groups of independent variables Xl~ x2, ...Xn; Yl , Y2 , .. . Yn.
Transforming these variables simultaneously with two d i s t i n c t linear substitutions (1)
xr = ~ r a r p ~ p ,
Ys = F~sbsq~?q,
(whose determinants one supposes to be always different from zero) one obtains a transformed form = 2pqTpq~p?Tq, whose coefficients 7pq axe related to the coefficients crs of the original function by the n 2 equations that have the following form: (2)
~ / p q - ~rsCrsarpbsq.
Setting for brevity 2mc,~rams = hrs,
Emc~mb,~ = k~,
10
E. B e l t r a m i
this typical equation can be written in the two equivalent forms (3)
Eshspbsq - 7pq,
~ r k r q a r p - 7pq.
Indicating with A, B, H, K, I" the determinants formed respectively with the elements a, b, c, h, k, 7, one has, from these last equations, I' = HB = KA. But, by the definition of the quantities h, k, one has as well H = CA, K = CB; hence F = ABC, that is, the determinant of the transformed function is equal to that of the original one multiplied by the products of the moduli of the two substitutions. Let us suppose initially that the linear substitutions (1) are both orthogonal. In such a case, their 2n 2 coefficients depend, as is known, on n 9 - n independent parameters, and on the other hand, the transformed function ~ can be, generally speaking, subjected to as many conditions. Now the coefficients 7pq whose indices p, q are mutually unequal are exactly n 2 - n in number: one can therefore seek if it is possible to annihilate all these coefficients, and to reduce the bilinear function f to the canonical form
To resolve this question, it suffices to observe that if, after having set in equations (3) >
7vq = 0 for p < q and 7pp = 7p,
one multiplies the first by brq and one carries out on the result the summation Eq; then one multiplies the second by asp and one carries out the summation Ep, one obtains h~p = 7vbrp,
ksq - %asq.
These two typical equations are mutually equivalent to the corresponding ones of equations (3), and, as a consequence of these latter ones, one could in this way recover the equations (3) in the process. In them are contained the entire resolution of the problem posed, that one obtains in this way: Writing out the last two equations in the following way
(4)
c l r a l s + c2ra2s + ... + Cnrans = 7sbrs, crlbls + cr2b2s + ... + crnbns = 7sars,
then setting, for brevity, CrlCsl + Cr2Cs2 + ... + CrnCsn • ~rs,
ClrCls ~- C2rC2s ~" ... -~ Cnrgns -- Vrs , (so that #rs = #st, vrs = r'sr), the substitution into the second equation (4) of the values of the quantities b obtained from the first one yields
(5)1
# r q a l s + Pr2a2s + ...-4- ~rnans = 72ars;
On Bilinear Functions
11
likewise, the substitution onto the first equation (4) of the values of the quantities a recovered from the second one yields (5)2
Vrlbls
+ v r 2 b 2 s + ... +
vrnbns
= "/'2brs.
The elimination of the quantities a from the n equations that one deduces from equations (5)x setting r = 1, 2, ... n in succession, leads one to the equation
A1
=
#11 - 7 2 #12
...
#1,~
#21 # 2 2 -- ,,[2 .oo ~ . . . . . . . . . . . . . . . . . . . .
...
#2n
#nl
...
"- 0 ~176.~
#n2
#nn
- - "~
2
which the n values 3,12, 72, ... 72 of 3, 2 must satisfy. Likewise, the elimination of the quantities b from the n equations that one deduces from equation (5)2 setting r = 1, 2, ...n in succession, leads one to the equation V l l -- ,,/2 A 2 _.
/212
/221 /222 _ ,)12 .............................
...
Vln
..
/'/2n
- - 0~
9
9
12nl
12n2
...
Vnn -- 72
possessing the same properties as the preceding equation. It follows t h a t the two determinants A1, and A 2 are mutually identical for any value of g a m m a (*). In fact, they are entire functions of degree n with respect to 9,2, which become identical for n -t- 1 values of 7 2, that is for the values 72, 722, ... 72 that simultaneously make both determinants zero, and for the value h' = 0 that makes them both equal to C 2. The n roots 7 2, 72, ... 72 of the equation A = 0 (likewise indicating indifferently A1 = 0 or A2 = 0) are all real, by virtue of a very well known theorem; to convince oneself that they are also positive, it suffices to observe that the coefficients of 7 0 , - - 7 2, 7 4 ,
--7 6 , etc.
are sums of squares. But one can also consider that, by the elementary theory of ordinary quadratic forms and by virtue of the preceding equations, one has F = Z~r~#~,z~z, = ")'I2 2~I + G -- ~r,VrsYrYs
22
... +
22
22 22 22. ---- ~'17"]1 ~- "/2T]2 "~ "'" ~- ~'nT]n,
on the other hand, one has as well f
-- ~m(glmX
1 -4- C 2 m Z 2 J I - . . . ~- C n r n X n ) 2 '
G = Zm(cmlyl + c,~2Y2 + ... + c,~,y~)2; hence the two quadratic functions F, and G are essentially positive, and the coefficients 72, 72, ... 7 2 of the transformed expressions, i.e. the roots of the equation A = 0, are necessarily all positive. (*) This theorem one finds demonstrated in a different way in w
of Determinants by BRIOSCHI
12
E. Beltrami
The proposed problem is therefore susceptible of a real solution, and here is the procedure: Find first the roots 7 2, 7 2, ... 7 2 of the equation A = 0, (which is equivalent to reducing one or the other of the quadratic functions F, G to the canonical form); then with the help of the equations of the form (5)1 and of those of the form
a~, + a~, + ... + a,,2 = i , one determines the coefficients a of the first substitution (coefficients that admit an ambiguity of sign common to all those items in the same column). This done, the equations that have the form (4) supply the values of the coefficients b of the second substitution (coeffidents that also admit an ambiguity of sign common to all those items in the same column, so that each of the quantities 7, are determined only by its square 72). Having done all these operations, one has two orthogonal substitutions that yield, exactly as desired by the problem, the identity ErsCrsZrYs = E m T m f m ~ m , in which everyone of the coefficients7m must be taken with the same sign that it is assigned in the calculation of the coefficientsb. It is worth observing that the quadratic functions denominated F and G can be derived from the bilinear function f setting in the latter on the one hand ys :
C l s X l "4" C2s;r,2 "Jr ... "~ CnsZn~
and on the other Z r = C r l Y l + cr2Y2 + ... + C r n y n .
Now if in these two relations one applies the substitutions (1), one then sees immediately that they are respectively converted into the following relations in the new variables ~ and ~?: ~,~ = 7,~f,~,
f,~ = %~-~,
which transform the canonical bilinear function
into the respective quadratic functions 22
22
22
7:f: + 72~2 + ...+ 7 ~ , 2 + "y22~2 + ... + "Y.~.. 22 "~:2 ~1
And, in fact, we have already noted that these two last functions are equivalent to the quadratics F, G. We ask of what form must be the bilinear function f so that the two orthogonal substitutions that reduce it to the canonical form turn out substantially mutually identical. To this end, we observe that setting b:s = =t=als,
b2, - -4-a2,,
...
bns = =t=a,~s
13
On B i l i n e a r F u n c t i o n s
the equations (4) are converted to the following: Clrals "4- c2ra2s + ... ~- cnra,~s = 4"Tsars, c r l a l s "4" cr2a2s -}-... + Crnans -- +7oa,.~,
which, since they must hold for every value of r and s, give crs = Csr. Reciprocally this hypothesis implies the equivalence of the two linear substitutions. So every bilinear form of the desired form is a s s o c i a t e d h a r m o n i c a l l y with an ordinary quadratic form: that is to say that designating this quadratic form with
the bilinear function is 1 de
f = r~,g~y,. In this case, the equation zX = 0 can be decomposed in this way:
A
--
(:11 - - '~
s
...
Cln
(:11 "~- "~
C12
...
Cln
c21
(:22 - - "}I
...
C2n
C21
C22 "~- "[
...
C2n
Cnl
On2
...
Cnn -- "~
Cnl
On2
...
C n n "4-'~
.
The first factor of the second member, set to zero, gives the well known equation that serves to reduce the function r to its canonical form. If the two substitutions are absolutely mutually identical, then coefficients 7 are the same for the quadratic function and for the bilinear one. But if, as we have already supposed, one concedes the possibility of an opposition of sign between the coefficients a and b belonging to two columns of equal index, the coefficient 7s of the corresponding index in the bilinear function can have sign opposite to that in the quadratic. From this, the presence in the equation A = 0 of a factor having for roots the quantities 7 taken negatively. In the particular case just now considered, the quadratic function denoted F is
and, on the other hand, one finds that there always exists an orthogonal substitution which makes simultaneously identical the two equations
r
2, 2
~
=2~2~ 2.
This is a consequence of the fact that, as is well known,
is a symbol invariant with respect to any orthogonal substitution.
14
E. Beltrami
We translate into geometrical language the results of the preceding analysis, assuming (as is generally useful to do) that the signs of the coefficients 7 are chosen to make AB = 1 and hence F = C. Let Sn, S~ be two spaces of n dimensions with null curvature, referred to, respectively, by the two systems of orthogonal linear coordinates z and y, for which we will call O and O ~ the origins. To a straight line $1 drawn through the origin 0 in the space Sn, there corresponds a specific set of ratios zz : x2 : ... : x,~; and on the other hand, the equation f = 0, homogeneous and of first degree in the xl, x2, ..., zn and in the Yl, Y2, ..., yn defines a correlation of figures in which to each line through the point O in the space Sn corresponds a locus of first order in n - 1 dimensions that we call S ~ n--1 within the space S~; and vice-versa. By virtue of the demonstrated theorem, it is always possible to substitute for the original coordinate axes in the z's and y's new axes in the ~'s and r/'s, respectively, with the same origins O and O ~, so that the correlation rule assumes the simpler form ~1~IT}1 "~" ~2~2~2 "~" ... "~- "~n~nT}r~ : 0.
Said this, one may think of the axis system r/, together with its figure, moved so that its origin O ~ falls on O, and that each axis ~r falls on its homologous axis ~r (*). In such a hypothesis, the last equation expresses evidently that the two figures are found to be in polar or involutory correlation with respect to the quadric cone (in n - 1 dimensions)
~
+~
+ ... + ~
= 0
that has its vertex on O. Hence, one can always convert a correlation of first degree of the above type, through a motion of one of the figures, into a polar or involutory correlation with respect to a quadric cone (in n - 1 dimensions) having its vertex on the common center of the two figures overlaid. In the case of n = 2, this general proposition yields the very well known theorem that two homographic bundles of rays can always be overlaid in such a way that they constitute a quadratic involution of rays. In the case that n = 3, one has the theorem, also known, that two correlative stars (i.e., such that to every ray in one corresponds a plane in the other, and vice-versa) can always be overlaid in such a way that they become reciprocal polar with respect to a quadric cone having its vertex on the common center. One can interpret the analytic theorem in another way, and recover other geometric properties in the cases n = 2, and n = 3. If with YlYx + Y2Y2 -[- ... + YnYn "b 1 = 0 one represents a locus of first order (S~_l) lying in the space S~, to every orthogonal substitution of the form Ys -"
~qbsq~q
(*) Something that is possible by having AB = 1.
On Bilinear Functions
15
applied on the local coordinates y, corresponds an identical orthogonal substitution Y8 = 2CqbsqEq applied to the tangential coordinates Y. Given this, suppose that between the x's and y's one institutes the n - 1 relations that result from setting equal the n ratios ClrXl +
Y" C2rX2
+
... +
(, = I, 2,
CnrXn
...'0.
This is equivalent to considering two homographic stars with centers on the points 0 and 0 ~. With such hypotheses, the equation ylY1 + y2Y2 + ... + ynY,~ = 0, which corresponds to this other equation in the ~?-axis system ~?IE1 + rl2E: + ... + ~?~E,~ = 0, is equivalent to the following relation between the x's and the y's ~rsCrsXrYs = O,
and this is in turn reducible, with two simultaneous orthogonal substitutions, to the canonical form 71fiE1 + 72~2E2 + ... + %f,~En = 0. Now the relation that this establishes between the new tangential coordinates E cannot differ from that contained in the third to last equation; hence it must be ~. 'TI~I
-"T2~2
...
--
.
"Tn~n
From these equations, which are nothing else than the relations of homography, expressed in the new coordinates ~ and ~?, it emerges evidently that the axis ~1 and r/l, ~2 and ~72, ... ~n and r/n are pairs of corresponding straight lines in the two stars. Since it is thus possible to move one of the stars in such a way that the ~ axes and the ~7 axes coincide one for one, one concludes that, given two homographic stars in an n dimensional space, one can always overlay one upon the other so that the n double rays acted upon by the overlaying, constitute a system of n orthogonal cartesian rays. From this, setting n = 2 and n = 3, one deduces that two homographic groups of rays can always be overlaid so that the two double rays are orthogonal; and that the two homographic stars can always be overlaid so that the three double rays form an orthogonal cartesian triple. Returning now to the hypothesis of two arbitrary linear substitutions, but always acting to give to the transformed function ~ the canonical form r = EmTm~m~?m, one solve the equations (3) with respect to the quantities h and k, respectively, with which one finds the two typical equations, mutually equivalent, (6)
Bhrs = Brs%,
Akr, = Ars%,
16
E. Beltrami
in which At,, B,s are the algebraic complements of the elements ara, bra in the respective determinants A, B. The equations (4) are nothing else than particularizations of these. Let ff be a second bilinear function, given by the expression f = Z~clrszrys,
and suppose we want to transform simultaneously, with the very same linear substitutions (1), the function f into the canonical form ~o and the function f into the canonical form = Z,~7,~,~y,~. Indicating with h ~ and k ~ quantities analogous to the h and k for the second function, together with the equations (6), these further equations must hold (6)'
Bh~,~ = B ~ 7 : ,
Akin, = A~,7:.
Dividing the first two equations of each of the pairs (6) and (6)' one by the other, one obtains hra - Aah~rs = 0 where As = "Y---ts 7;' or
(7)
( Clr -- A s C t l r ) a l s Jr" (C2r -- AsCt2r)g2s Jr" ... Jr" (Cnr -- A s C t n r ) a n .
= O.
Setting r = 1, 2, ...n in this last equation and eliminating the quantities als, a2s, ... arts from the n equations obtained in this way, one arrives at the equation
O --
C l l -- AC~l
c12 -
~c~2
...
s
c21 -
c22 -
AC~2
""
c2n - Act2n
Ac~l
..o . o . . ~
..o ..~ . . . . . . .
Cnl -
cn2 -
ACtnl
Actn2
...
~c~n
-- 0
~ ,~ ,~176 Cnn-
~C:r ,
which must be satisfiedby the n values AI, A2, ... A n of A. One can convince oneself of this by another way, by observing that | is the determinant of the bilinear function f - A f, and that hence one has by virtue of the general theorem on the transformation of this determinant (*) O = AB('),I- A ' y ~ ) ( " [ 2 - A " [ 2 ) . . . ( ' ~ ' r ~ - A']/~), whence emerges just as expected, for A and B to be different from zero, that the equation | = 0 has for its roots the n ratios 71
~'
72
~'
Vn
...~,,.
Of the rest, since the two series of quantities V and V~ are not determined except by these ratios, it is useful to assume for more simplicity that 7~ = V~ . . . . . V~ = I, and hence % = A~. (*) f r o m t h i s one sees t h a t t h e d e t e r m i n a n t of a b i l i n e a r f u n c t i o n is zero o n l y w h e n t h e f u n c t i o n i t s e l f c a n b e r e d u c e d to c o n t a i n t w o f e w e r v a r i a b l e s , a p r o p e r t y t h a t one c a n easily s h o w d i r e c t l y .
On Bilinear Functions
17
Here then is the procedure that leads to the solution of the problem: Find the n roots (real or imaginary) A1, A2, ..., A,~ of the equation | = 0; then substitute them successively into the equations of the form (7). One obtains in this way, for each value of s, n linear and homogeneous equations, one of which is a consequence of the other n - 1, so that one can recover only the values of the ratios az, : a2s :... : arts. Having chosen arbitrarily the values of the quantities al,, a2s, ..., an, so that they have these mutual ratios, the equations that are of the form of the second part of (6) ~ yield (8)
I A I b,, - Czr z, + C2rA2s + ... + CIA
I Cnr An,.
and in this way all the unknown quantities can be determined. One can observe that if for the coefficients at, one substitutes the products paara, something that is legitimate by a previous observation, the coefficients bra, determined by the last equation, are converted into ~ . This is the same as saying that if for the variables ~, one substitutes p,~,, the variables ~?s are converted to ~ ; but this change leaves unaltered the transformed functions 7~ and 7~~. If one denotes by | the algebraic complement of the element cr, - Adr, in the determinant $, and by Sr,(A,) what results by setting A = A, in this complement, it is easy to see that the equations (7) are satisfied by setting a,., = c~,O,.,(,L).
In this way, if one forms the equations analogous to (7) and containing the coefficients b in place of the coefficients a, they can be satisfied by setting
b,, =/~,o,,(~,,). The quantities aa and ~a are not determined completely: but the equations analogous to (2) determine the product aa~a. In any case, these factors are not essential, since by writing ~a in place of aa~a and ~?, in place of ~,~7,, they can be removed. From the expressions a.. = O . . ( A . ) .
b. = O..(A.).
that result from this supposition, it is evident that this important property emerges, that ~ i.e. when the two bilinear functions are associated harmonically when Cr, = C,r, %, = C,r, to tWO quadratic functions, the substitutions that reduce them both simultaneously to canonical form are s u b s t a n t i a l l y m u t u a l l y identical. If we suppose that the second bilinear function ft has the form f l : XlYl + x2Y2 + ... + XnYn,
the general problem just treated assumes the following form: Reduce the bilinear function f to the canonical form
18
E. Beltrami
with two simultaneous linear substitutions, so that the function XlYl
+ x2Y2
q" ... q" Z n Y n
is transformed into itself. In this case, the general formula (8) becomes Abe, = A~,, whence Ay~
=
Arlr/1
q- A r 2 r / 2
+
... q-
Arnrh~,
and hence ~Ta = a l s Y l + a28Y2 + ... + anaYn.
This last formula shows that the two substitutions (a) and (b) are inverses of each other, something that follows necessarily from the nature of the function that is transformed into itself. The relations among the coefficients a and b that follow from this ~rarsbr, = 1, ~,arsbrs = 1 Y~rarsbr,, = 0, ~,arsbr,s = 0, AB=I make a perfect contrast to those that hold among the coefficients of one orthogonal substitution, and they reduce to them when ara = br,, such a case arising (by what we have recently seen) when the form f is associated to a quadratic form. In the special problem which we have mentioned (and which has already been treated by Mr. CHRISTOrFEL at the end of his Memoirs), the equation 0 = 0 takes on the form C l l - - )~
s
...
Cln
C21
C22 - - '~
""
C'2n
........,
, ....................
Cnl
Cn2
...
- - O~
Cnn - -
and under the hypothesis cr, = csr (that we have just alluded to), it is identified, as is natural, with what the analogous problem leads to via a true quadratic form. We will not add any word on the geometric interpretation of the preceding results, since their intimate connection with the whole theory of homogeneous coordinates is evident. Likewise, we will not discuss for now the transformations of bilinear functions into themselves, an important argument but less easy to handle than the preceding ones, for which Mr. CHItISTOFFEL has already presented the treatment in the case of a single function and a single substitution.
PART
1
KEYNOTE PAPERS
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
IMPLICITLY RESTARTED ARNOLDI/LANCZOS AND LARGE SCALE SVD APPLICATIONS
21
METHODS
D.C. SORENSEN Department of Computational and Applied Mathematics Rice University P. O. Box 1892 Houston, TX 77251 sorensen @rice.edu ABSTRACT. Implicit restarting is a technique for combining the implicitly shifted QR mechanism with a k-step Arnoldi or Lanczos factorization to obtain a truncated form of the implicitly shifted QR-iteration for eigenvalue problems. The software package ARPACK that is based upon this technique has been successfully used to solve large scale symmetric and nonsymmetric (generalized) eigenvalue problems arising from a variety of applications. The method only requires a pre-determined limited storage proportional to n times the desired number of eigenvalues. Numerical difficulties and storage problems normally assodated with Arnoldi and Lanczos processes are avoided. This technique has also proven to be very useful for computing a few of the largest singular values and corresponding singular vectors of a very large matrix. Biological 3-D Image Reconstruction and Molecular Dynamics are two interesting applications. The SVD plays a key roll in a classification procedure that is the computationally intensive portion of the 3-D image reconstruction of biological macromolecules. A relatively new application is to analyze the motions of the proteins using the SVD instead of normal mode analysis. The primary research goal is to pick out the non-harmonic phenomena that usually contains most of the interesting behavior of the protein's motion. This paper reviews the Implicitly Restarted Arnoldi/Lanczos Method and briefly discusses the biological applications of the SVD. KEYWORDS. Lanczos/Arnoldi methods, implicit restarting, image reconstruction, molecular dynamics, computational biology.
1
INTRODUCTION
Large scale eigenvalue problems arise in a variety of applications. In many of these, relatively few eigenvalues and corresponding eigenvectors are required. The Lanczos method
22
t
D. C. Sorensen
for symmetric problems and the Arnoldi method, which is a generalization to nonsymmetric problems, are two important techniques used to compute a few eigenvalues and corresponding vectors of a large matrix. These methods are computationally attractive since they only require matrix-vector products. They avoid the cost and storage of the dense factorizations required by the more traditional Q R-methods that are used on small dense problems. A number of well documented numerical difficulties are associated with the Lanczos process [11]. These include maintenance of numerically orthogonal basis vectors or related techniques to avoid the introduction of spurious approximations to eigenvalues; required storage of basis vectors on peripheral devices if eigenvectors are desired; no a-priori bound on the number of Lanczos steps (hence basis vectors) required before sufficiently accurate approximations to desired eigenvalues appear. Since the Arnoldi technique generalizes the Lanczos process, these difficulties as well as the additional difficulties associated with computing eigenvalues of nonsymmetric matrices are present [13]. The Arnoldi method does not lead to a three term recurrence in the basis vectors as does Lanczos. Hence considerable additional arithmetic and storage is associated with the more general technique. Implicit restarting is a technique for combining the implicitly shifted QtL mechanism with a k-step Arnoldi or Lanczos factorization to obtain a truncated form of the implicitly shifted Qtt-iteration for eigenvalue problems. The numerical difficulties and storage problems normally associated with Arnoldi and Lanczos processes are avoided. The technique has been developed into a high quality public domain software package ARPACK that has been useful in a variety of applications. This software has proven to be very effective for computing a few of the largest singular values and corresponding singular vectors of a very large matrix. Two applications of the SVD occur in Biological 3-D Image Reconstruction and in Molecular Dynamics. The SVD plays a key roll in a classification procedure that is the computationally intensive portion of the 3-D image reconstruction of biological macromolecules. A relatively new application is to analyze the motions of the proteins using the SVD instead of normal mode analysis. The primary research goal is to pick out the non-harmonic phenomena that usually contains most of the interesting behavior of the protein's motion. This paper will review the implicit restarting technique for eigenvalue calculations and will discuss several applications in computing the SVD and related problems.
2
S O F T W A R E F O R L A R G E SCALE E I G E N A N A L Y S I S
The Arnoldi process is a well known technique for approximating a few eigenvalues and corresponding eigenvectors of a general square matrix. A new variant of this process has been developed in [14] that employs an implicit restarting scheme. Implicit restarting may be viewed as a truncation of the standard implicitly shifted QR-iteration for dense problems. Numerical difficulties and storage problems normally associated with Arnoldi and Lanczos processes are avoided. The algorithm is capable of computing a few (k) eigenvalues with user specified features such as largest real part or largest magnitude using 2nk + O(k2)storage. No auxiliary storage is required. Schur basis vectors for the k-dimensional eigen-space are computed which are numerically orthogonal to working precision.
Implicitly Restarted Arnoldi/Lanczos Methods
23
This method is well suited to the development of mathematical software. A software package ARPACK [10] based upon this algorithm has been developed. It has been designed to be efficient on a variety of high performance computers. Parallelism within the scheme is obtained primarily through the matrix-vector operations that comprise the majority of the work in the algorithm. The software is capable of solving large scale symmetric, nonsymmetric, and generalized eigenproblems from significant application areas. This software is designed to compute approximations to selected eigenvalues and eigenvectors of the generalized eigenproblem (1)
Ax = A M x ,
where both A and M are real n x n matrices. It is assumed that M is symmetric and positive semi-definite but A may be either symmetric or nonsymmetric. Arnoldi's method is a Krylov subspace projection method which obtains approximations to eigenvalues and corresponding eigenvectors of a large matrix A by constructing the orthogonal projection of this matrix onto the Krylov subspace S p a n { v , Av, ..., A k - l v } . The Arnoldi process begins with the specification of a starting vector v and in k steps produces the decomposition of an n x n matrix A into the form (2)
A V = V H + f e T,
where v is the first column of the matrix V E l:t nxk, v T v = Ik; H E R, kxk is upper Hessenberg, f E R n with 0 = v T f and ek E R k the kth standard basis vector. The vector f is called the residual. This factorization may be advanced one step at the cost of a (sparse) matrix-vector product involving A and two dense matrix vector products involving V T and V. The explicit steps are:
1 . / ~ = Ilfll; v + - / 1 ~ ;
v +- ( v , v); H +-
Zekr
.
2. z ~ Av; 3. h ~ VTz; f ~-- z -
Vh;
4. H .-- (H, h); The dense matrix-vector products may be accomplished using level 2 BLAS. In exact arithmetic, the columns of V form an orthonormal basis for the Krylov subspace and H is the orthogonal projection of A onto this space. In finite precision arithmetic, care must be taken to assure that the computed vectors are orthogonal to working precision. The method developed in [4] may be used to accomplish this. Eigenvalues and corresponding eigenvectors of H provide approximate eigenvalues and eigenvectors for A. If Hy=yO,
and we put x = V y .
D. C. Sorensen
24
Then x, 0 is an approximate eigenpair for A with
liAr, - xS[I = []f[lleTy[, and this provides a means for estimating the quality of the approximation. The information obtained through this process is completely determined by the choice of the starting vector. Eigen-information of interest may not appear until k gets very large. In this case it becomes intractable to maintain numerical orthogonality of the basis vectors V and it also will require extensive storage. Failure to maintain orthogonality leads to a number of numerical difficulties. Implicit restarting provides a means to extract interesting information from very large Krylov subspaces while avoiding the storage and numerical difficulties associated with the standard approach. It does this by continually compressing the interesting information into a fixed size k dimensional subspace. This is accomplished through the implicitly shifted QR mechanism. An Arnoldi factorization of length k + p is compressed to a factorization of length k by applying p implicit shifts resulting in
AVk++p = Yk+pHk+ + + p + fk+pek+pQ T ,
(3)
where Vk~p = Vk+pQ, Hk+ p+ = QTHk+pQ, and Q = Q1Q2"" "Qp, with Qj the orthogonal matrix associated with the shift #j. It may be shown that the first k - 1 entries of the T vector ek+pQ are zero. Equating the first k columns on both sides yields an updated k - s t e p Arnoldi factorization
Av: = V:H+
+
(4)
with an updated residual of the form f+ = Vk+_4.pek+l~k+ fk+po'. Using this as a starting point it is possible to use p additional steps of the Arnoldi process to return to the original form. Each of these applications implicitly applies a polynomial in A of degree p to the starting vector. The roots of this polynomial are the shifts used in the Q R process and these may be selected to filter unwanted information from the starting vector and hence from the Arnoldi factorization. Full details may be found in [14]. The software ARPACK that is based upon this mechanism provides several features which are not present in other (single vector) codes to our knowledge 9 Reverse communication interface 9 Ability to return k eigenvalues which satisfy a user specified criterion such as largest real part, largest absolute value, largest algebraic value (symmetric case), etc. 9 A fixed pre-determined storage requirement suffices throughout the computation. Usually this is n , O ( 2 k ) + O ( k 2) where k is the number of eigenvalues to be computed and n is the order of the matrix. No auxiliary storage or interaction with such devices is required during the course of the computation. 9 Eigenvectors may be computed on request. The Arnoldi basis of dimension k is always computed. The Arnoldi basis consists of vectors which are numerically orthogonal to working accuracy. 9 Accuracy: The numerical accuracy of the computed eigenvalues and vectors is user specified and may be set to the level of working precision. At working precision,
Implicitly Restarted A r n o l d i / L a n c z o s Methods
25
the accuracy of the computed eigenvalues and vectors is consistent with the accuracy expected of a dense method such as the implicitly shifted QR iteration. 9 Multiple eigenvalues offer no theoretical or computational difficulty other than additional matrix vector products required to expose the multiple instances. This cost is commensurate with the cost of a block version of appropriate blocksize. 3
COMPUTING
THE SVD WITH ARPACK
In later sections we shall discuss large scale applications of the Singular Value Decomposition (SVD). To avoid confusion with the basis vectors V computed by the Arnoldi factorization, we shall denote the singular value decomposition of a real m x n matrix M by M = U SW T
and we consider only the short form with U E R mx'~ , W E l:t"~x'~ and S = d i a g ( a l , a2, ..., an) with o'j ~_ aj+l. We assume m > n and note that u T u = w T w = In. The applications we have in mind focus upon computing the largest few singular values and corresponding singular vectors of M. In this case (with m > n ) it is advantageous to compute the desired right singular vectors W directly using ARPACK z ~ Av
where A = M T M
If Wk is the matrix comprised of the first k columns of W and Sk is the corresponding leading principal submatrix of S then A W k -- W k ( S k 2)
The corresponding left singular vectors are obtained through the relation
vk = M W k S [ ~. Although this mechanism would ordinarily be numerically suspect, it appears to be safe in our applications where the singular values of interest are large and well separated. Finally, it should be noted that the primary feature that would distinguish the implicit restarting method from other Lanczos style iterative approaches [1] to computing the SVD is the limited storage and the ability to focus upon the desired singular values. An important feature of this arrangement is that space on the order of 2 n k is needed to compute the desired singular values and corresponding right singular vectors and then only m k storage locations are needed to compute the corresponding left singular vectors. The detail of the matrix vector product z ~ A v calculation is also important. The matrix M can be partitioned into blocks M T = (MT, M T, ..., M T) and then matrix-vector product z ~ A v may be computed in the form
26
D. C. Sorensen
9 for j = 1, 2, ..., b, 1. z ~ z + M T ( M j v ) ;
end. Moreover, on a distributed memory parallel machine or on a network or cluster of workstations these blocks may be physically distributed to the local memories and the memory traffic associated with reading the matrix M every time a matrix vector product is required could be completely avoided. 4
LARGE SCALE APPLICATIONS
OF THE SVD
In this section we discuss two important large scale applications of the SVD. The first of these is the 3-D image reconstruction of biological macromolecules from 2-D projections obtained through electron micrographs. The second is an application to molecular dynamical simulation of the motions of proteins. The SVD may be used to compress the data required to represent the simulation and more importantly to provide an analytical tool to help in understanding the function of the protean. 4.1
ELECTRON
MICROGRAPHS
We are currently working with Dr. Wah Chiu and his colleagues at the Baylor College of Medicine to increase the efficiency of their 3-dimensional image reconstruction software. Their project involves the identification and classification of 2-dimensional electron micrograph images of biological macromolecules and the subsequent generation of the corresponding high resolution 3-D images. The underlying algorithm [16] is based upon the statistical technique of principal component analysis [17]. In this algorithm, a singular value decomposition (SVD) of the data set is performed to extract the largest singular vectors which are then used in a classification procedure. Our initial effort has been to replace the existing algorithm for computing the SVD with ARPACK which has increased the speed of the analysis by a factor of 7 on an Iris workstation. The accuracy of the results were also increased dramatically. Details are reported in [6]. The next phase of this project will introduce the parallel aspects discussed previously. There are typically several thousand particle images in a working set, and in theory, each one characterizes a different viewing angle of the molecule. Once each image is aligned and processed to eliminate as much noise as possible the viewing angle must be determined. The set is then Fourier transformed, and each transformed image is added to a Fourier composite in a position appropriate to the this angle. The composite is then transformed back to yield a high resolution 3-D image. If we consider each 2-D image to be a vector in N dimensional space, where N is the number of pixels, we know that we can describe each image as a weighted sum of the singular vectors of the space defined by the set. The distribution of the images in this N dimensional coordinate system is then used to break up the set into classes of viewing angles, and the images in a class are averaged. Determining the left singular vectors of
Implicitly Restarted Arnoldi/Lanczos Methods
27
the space of images is presently the computational bottleneck of the process. As is true in many applications, the classification can be accomplished with relatively few of the singular vectors corresponding to the largest singular values. In a test example (using real data) the largest 24 singular triplets uiaiw T were extracted from a data set of size 4096x1200, corresponding to 1200 64x64 images. The decomposition was done with an existing software package as well as with a newer algorithm with the hopes that the computational bottleneck would be resolved. The existing package ran with 24 iterations of its method in roughly 29 minutes of CPU time. The new routines performed 7 iterations in just under 4 minutes, an improvement in speed by a factor of 7. In this test, the data set fits into memory. The ultimate goal is to process data sets consisting of 100,000 images each 128x128 pixels and this will require 6.4Gbytes of storage. I/O will then be a severe limitation unless a parallel I/O mechanism as described above can be implemented.
4.2
MOLECULAR DYNAMICS
Knowledge of the motions of proteins is important to understanding their function. Proteins are not rigid static structures, but exist as a complex dynamical ensemble of closely related substates[7]. It is the dynamics of the system that permit transitions between these conformational substates and there is evidence that this dynamic behavior is also responsible for the kinetics of ligand entry and ligand binding in systems such as myoglobin. These transitions are believed to involve anharmonic and collective motions within the protein. X-ray crystallography is able to provide high-resolution maps of the average structure of a protein, but this structure only represents a static conformation. Molecular dynamics, which solves the Newtonian equations of motion for each atom, is not subject to such approximations and can be used to model the range of motions available to the protein on short (nanosecond) time-scales. Time-averaged refinement extends the molecular dynamics concept, restraining the simulation by requiring the time-averaged structure to fit the observed X-ray data[2]. With this methodology, an ensemble of structures that together are consistent with the dynamics force-fields and the observed data is generated. This ensemble represents a range of conformations that are accessible to proteins in the crystalline state. One difficulty with time-averaged refinements is the plethora of generated conformations. No longer is one structure given as the model, rather, thousands of structures collectively describe the model. Although it is possible to analyze this data directly by manually examining movies of each residue in the simulation until some pattern emerges, this is not a feasible approach. Nor is it feasible to watch an animation of an entire protein and discern the more subtle global conformation states that may exist. The SVD can provide am important tool for both compressing the data required to represent the simulation and for locating interesting functional behavior. In order to employ the SVD we construct a matrix C from the simulation data. A column of C represents the displacement of the atoms from their mean positions at a given instant of time in the simulation. Each group of three components if this column vector make up the spatial displacement of a particular atom from its mean position. As discussed in Section 2 above, we construct an approximation to C using the leading k singular values
28
D. C. Sorensen
and vectors. C ~ Ck = U k & V T.
For a visually realistic reconstruction of the simulation (ie. run the graphics with Ck in place of C one can typically set k around 15 to 20). This representation not only compresses the data but classifies the fundamental spatial modes of motion of the atoms in the protein. The SVD also includes temporal information in a convenient form. It is the temporal information that sets it apart from the more traditional eigendecomposition analysis. This transform can also be thought of as a quasinormal mode calculation based on the dynamics data rather than the standard normal mode analysis obtained by an approximation to the system's potential energy function. This standard approach only admits harmonic motions. Using the SVD, it is possible to extract information about the distribution of motion in the protein, and to classify the conformational states of the protein and determine which state the system is in at any point in the simulation. Projections of the entire trajectory onto the SVD basis can also be used for understanding the extent of configuration space sampling of dynamics simulations and the topography of the system's energy hypersurface[3]. In [12], we have computed a time-averaged refinement of a mutant myoglobin. Using the SVD, we have characterized the motions within the protein and begun to automate this analysis. We have also used the SVD to distinguish between different conformational states within the dynamics ensemble. This study exposes anharmonic motions associated with the ligand binding sites through analysis of the distribution of the components of the right singular vectors. It also documents the efficiency of computing the SVD through ARPACK in this application. 5
G E N E R A L A P P L I C A T I O N S OF A R P A C K
AR.PACK has been used in a variety of challenging applications, and has proven to be useful both in symmetric and nonsymmetric problems. It is of particular interest when there is no opportunity to factor the matrix and employ a "shift and invert" form of spectral transformation, fi ~ ( A -
a I ) -1 .
(5)
Existing codes often rely upon this transformation to enhance convergence. Extreme eigenvalues {#} of the matrix A are found very rapidly with the Arnoldi/Lanczos process and the corresponding eigenvalues {A} of the original matrix A are recovered from the relation A = 1/# + a. Implementation of this transformation generally requires a matrix factorization. In many important applications this is not possible due to storage requirements and computational costs. The implicit restarting technique used in ARPACK is often successful without this spectra/transformation. One of the most important classes of application arise in computational fluid dynamics. Here the matrices are obtained through discretization of the Navier-Stokes equations. A typical application involves linear stability analysis of steady state solutions. Here one linearizes the nonlinear equation about a steady state and studies the stability of this state
Implicitly Restarted Arnoldi/Lanczos Methods
29
through the examination of the spectrum. Usually this amounts to determining if the eigenvalues of the discrete operator lie in the left halfplane. Typically these are parametrically dependent problems and the analysis consists of determining phenomena such as simple bifurcation, Hopf bifurcation (an imaginary complex pair of eigenvalues cross the imaginary axis), turbulence, and vortex shedding as this parameter is varied. ARPACK is well suited to this setting as it is able to track a specified set of eigenvalues while they vary as functions of the parameter. Our software has been used to find the leading eigenvalues in a Couette-Taylor wavy vortex instability problem involving matrices of order 4000. One interesting facet of this application is that the matrices are not available explicitly and are logically dense. The particular discretization provides efficient matrix-vector products through Fourier transform. Details may be found in [5]. Very large symmetric generalized eigenproblems arise in structural analysis. One example that we have worked with at Cray Research through the courtesy of Ford motor company involves an automobile engine model constructed from 3D solid elements. Here the interest is in a set of modes to allow solution of a forced frequency response problem ( K - AM)x = f(t), where f ( t ) is a cycnc forcing function which is used to simulate expanding gas loads in the engine cylinder as well as bearing loads from the piston connecting rods. This model has over 250,000 degrees of freedom. The smallest eigenvalues are of interest and the ARPACK code appears to be very competitive with the best commercially available codes on problems of this size. For details see [15]. Another source of problems arise in magnetohydrodynamics (MHD) involving the study of the interaction of a plasma and a magnetic field. The MHD equations describe the macroscopic behavior of the plasma in the magnetic field. These equations form a system of coupled nonlinear PDE. Linear stability analysis of the linearized MHD equations leads to a complex eigenvalue problem. Researchers at the Institute for Plasma Physics and Utrecht University in the Netherlands have modified the codes in ARPACK to work in complex arithmetic and are using the resulting code to obtain very accurate approximations to the eigenvalues lying on the Alfven curve. The code is not only computes extremely accurate solutions, it does so very efficiently in comparison to other methods that have been tried. See [9] for details. There are many other applications. It is hoped that the examples that have been briefly discussed here will provide an indication of the versatility of the ARPACK software as well a the wide variety of eigenvalue problems that arise. Acknowledgements The author is grateful to Ms. Laurie Feinswog and to Mr. Tod Romo for working with ARPACK in the 3-D image reconstruction and molecular dynamics applications mentioned above. They provided most of the information conveyed in Section 3.
References [1] M. W. Berry, "Large scale singular value computations", IJSA ,6,13-49,(1992).
D. C. Sorensen
30
[2] J.B. Clarage, G.N. Phillips, Jr. "Cross-validation Tests of Time-averaged Molecular Dynamics Refinements for Determination of Protein Structures by X-ray Crystallography," Acta Cryst., D50,24-36,(1994). [3] J.B. Clarage, T.D. Romo, B.M. Pettit, G.N. Phillips, Jr., "A Sampling Problem in Molecular Dynamics Simulations of Macromolecules, Keck Center Rept., Rice University, (1994). [4] J. Daniel, W.B. Gragg, L. Kaufman, G.W. Stewart, "Reorthogonalization and stable algorithms for updating the Gram-Schmidt QR factorization," Math. Comp.,30, 772795,(1976). [5] W.S. Edwards, L.S. Tuckerman, R.A. Friesner and D.C. Sorensen, "Krylov Methods for the Incompressible Navier-Stokes Equations," Journal of Computational Physics,
~0,82-102 (1994). [6] L. Feinswog, M. Sherman, W. Chiu, D.C. Sorensen, "Improved Computational Methods for 3-Dimensional Image Reconstruction" , CRPC Tech. Rept., PAce University (in preparation). [7] H. Frauenfelder, S.G. Sligar, P.G. Wolynes, " The Energy Landscapes and Motions of Proteins, Science, 254,1598-1603, (1991). [8] G.H. Golub, C.F. Van Loan. Matrix computations. North Oxford Academic Publishing Co., Johns Hopkins Press,(1988). [9] M.N. Kooper, H.A. van der Vorst, S. Poedts, and J.P. Goedbloed, "Application of the Implicitly Updated Arnoldi Method with a Complex Shift and Invert Strategy in MHD," Tech. Rept., Institute for Plasmaphysics, FOM Rijnhuizen, Nieuwegin, The Netherlands,(Sep. 1993) ( submitted to Journal of Computational Physics). [10] R. Lehoucq, D.C. Sorensen, P.A. Vu, ARPACK: Fortran subroutines for solving large scale eigenvalue problems, Release 2.1, available from
[email protected], (1994). [11] B.N. Parlett, The Symmetric Eigenvalue Problem , Prentice-Hall, Englewood Cliffs, NJ. (1980). [12] T.D. Romo, J.B. Clarage, D.C. Sorensen, and G.N. Phillips, Jr., "Automatic Identification of Discrete Substates in Proteins: Singular Value Decomposition Analysis of Time Averaged Crystallographic Refinements," CRPC-TR 94481, Rice University, (Oct. 1994). [13] Y. Saad, Numerical Methods for Large Eigenvalue Problems, Halsted Press-John Wiley & Sons Inc., New York (1992). [14] D. C. Sorensen, "Implicit application of polynomial filters in a k-Step Arnoldi Method," SIAM J. Matt. Anal. Apps.,13 , 357-385, (1992).
Implicitly Restarted Arnoldi/Lanczos Methods
31
[15] D.C. Sorensen, P.A. Vu, Z. Tomasic, "Algorithms and Software for Large Scale Eigenproblems on High Performance Computers," High Performance Computing 1993Grand Challenges in Computer Simulation,Adrian Tentner ed., Proceedings 1993 Simulation Multiconference, Society for Computer Simulation, 149-154, (1993). [16] M. Van Heel, J. Frank, "Use of Multivariate Statistics in Analysing the Images of Biological Macromolecules," Ultramicroscopy, 6 187-194, (1981). [17] S. Van Huffel and J. Vandewalle, The Total Least Squares Provblem: Computational Aspects and Analysis, Frontiers in Applied Mathematics 9, SIAM Press, Philadelphia,(1991).
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
ISOSPECTRAL
MATRIX
FLOWS FOR NUMERICAL
33
ANALYSIS
U. HELMKE
Department of Mathematics University of Regensburg 93040 Regensburg Germany
[email protected] ABSTRACT. Current interest in analog computation and neural networks has motivated the investigation of matrix eigenvalue problems via eigenvalue preserving differential equations. An example is Brockett's recent work on a double Lie bracket quadratic matrix differential equation with applications to matrix diagonalization, sorting and linear programming. Another example is the Toda flow as well as many other classical completely integrable Hamiltonian systems. Such isospectral flows appear to be a useful tool for solving matrix eigenvahe problems. Moreover, generalizations of such flows are able to compute the singular value decomposition of arbitrary rectangular matrices. In neural network theory similar flows have appeared in investigations on learning dynamics for networks achieving the principal component analysis. In this lecture we attempt to survey these recent developments, with special emphasis on flows solving matrix eigenvalue problems. Discretizations of the flows based on geodesic approximations lead to new, although slowly convergent, algorithms. A modification of such schemes leads to a new Jacobi type algorithm, which is shown to be quadratically convergent. KEYWORDS. Dynamical systems, Riemannian geometry, SVD, Jacobi method.
1
INTRODUCTION
Isospectral matrix flows are differential equations evolving on spaces of matrices, which have the defining property that their solutions have constant eigenvalues. Such equations have been studied over the past 25 years, appearing in diverse areas such as classical and statistical mechanics, nonlinear partial differential equations, numerical analysis, optimization, neural networks, signal processing and control theory. The early development of isospectral flows goes back to the pioneering work of Rutishauser on the infinitesimal analogue of
U. Helmke
34
the quotient - difference algorithm. However Rutishauser's work in the 1950s has stayed isolated and it took more than 20 years that interest into such flows did renew through the discovery of Flaschka, Symes and Kostant that the Toda flow from statistical mechanics is a continuous - time analogue of the QlZ - algorithm on tridiagonal matrices. More precisely the QI~ - algorithm was seen as a discretization of an associated isospectral differential equation, the discretization being at integer time. This work on the Toda flow has spurred further investigations of Chu, Driessel, Watkins, Elsner and others on continuous time flows solving matrix eigenvalue problems. More recently, work by Brockett and others has considerably broadened the scope of applications of isospectral flows. In this paper I will focus on the applications within numerical linear algebra. More specifically the focus will be on matrix eigenvalue methods. After a brief outline on the historical development of isospectral flows the double bracket equation of Roger Brockett is studied. This is a quadratic matrix differential equation which is derived as a gradient flow for a least squares cost function. Approximations of the flow based on short length geodesics are introduced. These lead to linearly convergent algorithms involving calculations of matrix exponentials. A variant of the classical Jacobi algorithm is introduced which is based on optimization of the least squares cost function along several geodesics. This algorithm, termed Sort-Jacobi algorithm, has attractive properties such as quadratic convergence and incorporates sorting of the eigenvalues. The extension to the SVD case is straightforward. This is a survey paper. Accordingly most proofs are omitted. I have tried to preserve the structure of my talk at the workshop in as much as possible.
2
ISOSPECTlZAL FLOWS
A differential equation on matrix space ~nxn
A(*) = F(A(t)),
A e I~'~•
(1)
is called isospectral, if the eigenvalues of the solutions A(t) are constant in time, i.e. if spectrum(A(t)) = spectrum(A(0))
(2)
holds for all t and all initial conditions A(0) E I~nxn. A more restrictive class of isospectral matrix flows are the self-similar flows on I~nxn. These are defined by the property that
A(t) = S(t)A(O)S(t) -1
(3)
holds for all initial conditions A(0) and times t, and suitable invertible transformations S(t) E GL(n, n~). Thus the Jordan canonical form of the solutions of a self-similar flow does not change in time. It is easily seen that every self-similar flow (1) has the Lie-bracket form =
[A, f(A)]
= Af(A)- f(A)A
(4)
Isospectral Matrix Flows f o r Numerical Analysis
35
for a suitable matrix-valued function of the coefficients of A. Conversely, every differential equation on I~nxn of the form (4)is isospectral; see e.g. Chu [7], Helmke and Moore [19]. Of course, instead of considering.time-invariant differential equations (1) we might as well consider time-varying systems A = F ( t , A ( t ) ) . More generally we might even consider isospectral flows on spaces of linear, possibly infinite-dimensional, operators, but we will not do so in the sequel. E x a m p l e 1 ( L i n e a r Flows) The simplest example of an isospectral flow is the linear self-similar flow (5)
A=[A,B]
for a constant matrix B E Ii~nxn. The solutions are A ( t ) = e - t s A(O)e t s ,
which immediately exhibits the self-equivalent nature of the flow. E x a m p l e 2 ( B r o c k e t t ' s Flow) The second example is Brockett's double Lie bracket equation
(6)
= [A, [A, B]] = A 2 B + B A 2 - 2 A B A .
For this equation there is no general solution formula known; we will study this flow in section 3.
Example 3 (QR-Flow) This is a rather nontrivial looking example. Here = [A, (log A)_]
(7)
where B_ denotes the unique skew-symmetric part of B with respect to the orthogonal decomposition
B=
0
-b21
. . . .
b21
"'.
"'.
b,n
...
bn,,~-i
bnl
+ 0
*
......
*
0
"'.
:
0
...
0
,
into skew-symmetric and upper triangular matrices. This differential equation implements exactly the QR-algorithm in the sense that for all n E N: A(n) = n-
th step of Qtt-algorithm applied to A(0).
Let us now briefly describe some of the earlier developments on isospectral flows.
U. Helmke
36 2.1
KORTEWEG-DE VRIES EQUATION
An important part of the theory of isospectral matrix flows is linked to comparatively recent developments in the theory of nonlinear partial differential equations and, in particular, with the theory of solitons. This is explained as follows. Let us consider the Korteweg-de Vries (KdeV) ~
= 6~
- ~.
,
~ = ~ ( ~ , t).
(8)
This nonlinear partial differential equation models the dynamics of water flowing in a shallow one-dimensional channel. By a beautiful observation of P. Lax the KdeV equation has the appealing form Lt = [P, L]
(9)
where 92
/-, = -~-'~'z2 -F u(x,t)
(I0)
denotes the SchrSdinger operator and
o3
P = -4~--dx3 + 6u
+ 3ux.
(11)
Therefore the KdeV equation is equivalent to the isospectral flow (9) on a space of differential operators. In particular, the spectrum of the SchrSdinger operator (10) yields invariants for the solutions u(z,t) of (t). It is thus obvious that the isospectral nature of (9) may lead, and indeed has led, to further insights into the solution structure of the KdeV equation. Following this fundamental observation of Lax several other important nonlinear partial differential equations (such as, e.g., the nonlinear SchrSdinger equation or the LandauLifshitz equation) have also been recognized as isospectral flows. These developments then have culminated into what is nowadays called the theory of solitons, with important contributions by Gelfand-Dikii, Krichever, Novikov, Lax, Adler and van Moerbeke and many others. For further information we refer to [1], [2].
2.2
EULElZ EQUATION
One of the many mechanical systems which do admit an interpretation as an isospectral flow is the classical Euler equation
Isospectral Matrix Flows for Numerical Analysis
I=~=
=
(Ia - h)~'l~oa
Ia~aa
=
( h -/2)~01~,=.
37
(12)
Using the notation
B'=
0 0
I~"1 0 0 I~-1
,
A: =
r/3 -72
0 771
-r/1 0
(13)
where (~h, ~12,*/3) = (/1 r
I2w2, I3w3 )
a simple calculation shows that /[ = [A2, B] = [A, AB -i- BA].
(14)
Thus the Euler equation (12) is equivalent to the quadratic isospectral matrix flow (14) on the Lie algebra so(3) of 3 x 3 skew-symmetric matrices. In particular we conclude that IIAII2 = Ii2wl2 + I~02 2 2 + i3w32 is an invariant of the Euler equation. Moreover, from the identity tr(A[B, C]) = tr([A, B]C) it follows that
2tr(BAfi) = tr(BA + AB),;i = tr((AB + BA)[A, AB + BA]) = O. Thus
tr(BA 2) = / 1 ( / 2 + I3)w 2 + I2(I1 + I3)w~ + I3(I~ + / 2 ) w 2 is another invariant of the Euler flow. Thus for c _> 0, d >_ 0 the ellipsoids
Mc = {A e so(3) [ tr(A2B) = -c} as well as the spheres {A e so(3) lllAII 2 = d} are invariant submanifolds of (14). From this the familiar phase-portrait description of the Euler equation on II~3 is easily obtained; see e.g. Arnol'd [2]. Thus the isospectral form (14) of the Euler equation leads to a complete integration of the flow. For a further analysis of generalized Euler equations defined on arbitrary Lie algebras we refer to Arnol'd [2], Fomenko [16].
38 2.3
U. H e l m k e
TODA FLOW
The Toda flow is one of the earliest examples of an isospectral flow which yields a continuous analogue of a numerical matrix eigenvalue problem. The original motivation for the flow stems from statistical mechanics; cf. Flaschka (1974, 1975), Moser (1975), Kostant (1979). Consider the one-dimensional lattice of n idealized mass points Zl < ... < xn, located on the real axis I~. We assume that the potential energy of the system is described by
Y(xl,..., z~) = ~ e ~ - ~ + ~ k= 92 ?%
X o ' = - o o , x n + l " = +oo; while the kinetic energy has the usual form ] ~] x~. k=l
Thus, if the particles are sufficiently far apart, then the total potential energy is very small and the system behaves roughly like a gas. The Hamilton function of the system is
(~ = ( ~ , . . . , ~ ) , v = (v~,..., y~))
H(~,v)=~
lfiv~+ fi~ - ~ + ~ k=l
(15)
k=l
where Yk = :~k denotes the velocity of the k-th mass point. The associated Hamiltonian system is
'xk
=
~]k
=
OH
= Yk Oyk OH OXk = e X k - l - x k -- eXk-xk+l
(16)
for k = 1 , . . . , n. These are the Toda lattice equations on 11~2n. Equivalently, the Toda lattice is described by the second order differential equations Xk
-- s
_
exk--xk+l
,
k =
1,...,
(17)
n.
To relate the Toda system to an isospectral flow we make the following (not invertible) change of variables (this observation is due to Flaschka). (xl,...,xn,
Yl,...,Yn)'
)(al,...,an-l,bl,...,bn)
with ak --
1 e (xk-xk+l)/2
'
bk --
-1 T
Yk
~
k - 1,
"'''
n.
Then, in the new variables, the Toda system becomes (ao = 0, bn+l = 0).
(18)
Isospectral Matrix Flows for Numerical Analysis hk = ak(bk+l -- bk) bk = 2 ( a ~ - a ~ _ l ) ,
k= 1,...,n.
39
(19)
This is called the Toda flow. Thus we have associated to every solution ( x ( t ) , y ( t ) ) o f (16) a corresponding solution (a(t),b(t)) of (19). Conversely, if (a(t),b(t))is a solution of (19) with initial conditions satisfying al(0) > 0 , . . . , an-l(0) > 0~ then one can uniquely reconstruct from (a(t),b(t)) via (18) the velocity vector (yl(t),...,yn(t)) as well as the pairwise differences x l ( t ) x 2 ( t ) , . . . , X n _ l ( t ) - xn(t) of the positions of the mass points. Moreover, if (x(0), y(0))is an initial condition of (16), then by conservation of momentum we have
9~(t) = t. ~ w(0)+ ~ ~ ( 0 ) k=l
k=l
k=l
Thus, given any initial condition (x(0), y(0))of (16) and a corresponding solution (a(t), b(t)) of (19), one can explicitly determine the solution (x(t),y(t)) of (16). In this sense the systems (16) and (19) are equivalent. In order to exhibit the isospectral nature of the flow (19) we store the state variables as components of a tridiagonal matrix 9 Let
A:=
bl
al
al
b2
0 "'.
9
0
A
9
(20)
an_ 1 bn
9
at,-1 0
al
--al
0 9 9
0
0 "'. " 9
an-1
(21) an-1 0
With these notations the flow (19) is easily verified to be equivalent to the isospectral Toda flow on tridiagonal matrices ,~i = [A, A_]
(22)
The convergence properties of the Toda flow (22) are summarized in the following theorem, whose proof is due to Moser [23], Flaschka [15], Delft, Nanda and Tomei [11]. T h e o r e m 1.1
(a) The solutions of the Toda flow (22)
40
U. Helmke A = [A,A_]
exist and are tridiagonal for all time.
(b) Every tridiagonal solution of (22) converges to a diagonal matrix lim
A(t) =
diag(A1,..., An),
where ) u , . . . , A n are the eigenvalues of A(O). For almost every initial condition A(O) in the set of tridiagonal Jacobi matrices the solutions converges to diag(Ax,..., An) with )u < ...
(c) The Toda flow interpolates the QR-algorithm in the sense that for all k E N" e A(k) - k - th QR step, applied to e A(O).
(d) Let N = d i a g ( 1 , 2 , . . . , n ) and A1 < ... < An. The Toda flow is the gradient flow of n
the function t r ( N A ) = ~ iaii the isospectral manifold of tridiagonal matrices with fixed i=1
eigenvalues A1, . . . , An.
It follows immediately from this theorem that the solutions of the Toda lattice equation behave as follows: The distances I x i - xi+ll between subsequent mass points xx(t) < ... < xn(t) go to infinity while the velocities ~] ( t ) , . . . , kn(t) converge to their limiting values given by the eigenvalues of A(0). Moreover, for generic initial conditions the limiting velocities are sorted according to their magnitude. This sorting property of isospectral flows such as the Toda flow is an important aspect of the theory and has been further analyzed by Brockett [5], Bloch, Brockett and Ratiu [4]. It is instructive to consider an example. The simplest nontrivial case is n = 3. Here the set of 3 x 3 real tridiagonal matrices with fixed distinct eigenvalues Ax < A2 < A3 can be shown to be a Riemann surface of genus 2; compare e.g. Tomei [29]. There are 3! = 6 equilibrium points corresponding to the isospectral diagonal matrices and every solution converges to one of them. The unique local attractor of the flow is diag(A1, A2, A3) with A1 < A2 < A3. For a complete phase portrait analysis of the Toda flow we refer to Faybusovich (1992). 2.4
RUTISHAUSEI~'S FLOW
The perhaps earliest example of a continuous-time isospectral matrix flow is that of Rutishauser (1954). Rutishauser constructs an infinitesimal analogue of the well-known QDalgorithm. Rutishauser's infinitesimal QD-algorithm is described as follows. Let F(t) denote a function of a real variable t. For any h > 0 let F(
h)
,
o,1,2,
.,
denote the sequence of sampled values of F. difference scheme:
Consider the following modified quotient
41
Isospectral Matrix Flows f o r Numerical Analysis
Q u o t i e n t Difference S c h e m e r
_ S~-)
O~,,)
:=
~k
D~')
.=
'~k
E(~) k
: =
Ek-1
k+l
: ~
~,(~') h i r ~"Jk
h
-(,,+1)
D~)
nt.
Rutishauser then considers the limit process where h ~ 0, hu --. t. He thus proposes the following Rutishauser's infinitesimal Q D - s c h e m e
Sl(t)
:=
F(t)
Qk(t)
"=
S'k(t) = (log Sk(t))' Sk(t)
Dk(t)
=
Qtk(t )
Ek(t)
=
D k ( t ) + Ek+l (t)
Sk+l(t)
-
-
Ek(t)Sk(t)
,
k EN
(23)
The computational capabilities of the algorithm are demonstrated by the easily estaboo
lished fact, that for a convergent Dirichlet series F ( t ) = ~ cke ~kt then lim Qk(t) = ,kk k=l
t--,oo
holds for all k E N; see [24] for a proof. A simple reformulation of the Rutishauser system (23), using ak(t): = E k ( t )
,
bk(t):-- Qk(t),
(24)
yields the following system of quadratic differential equations. R u t i s h a u s e r ' s Flow hk
=
ak(bk+x -- bk)
bk
=
ak -- ak-1
9
(25)
Note the striking similarity with the Toda flow! The isospectral nature of Rutishauser's flow is seen as follows: Consider the non-symmetric tridiagonal matrices
42
U. Helmke
bl al
1 b2
0
a2
b3
"'.
9
o
9
A"
9 9
9149
0
0 0
1
9149
""
~
0
1
an-t
br~
0
A
*
-
-
-
0
al
0
0
a2
9
0 "'.
9
, 9
9
0
, ,
9
, ,
'''
0
,
an-1
0
Then Rutishauser's flow (25) is equivalent to the isospectral flow (26)
= [A,A_],
evolving on the set of tridiagonal matrices of the above form. Again it can be shown that (26) interpolates the LR-algorithm in the sense that for any solution A ( k ) of (26) th LR iteration for e A(O).
e A(k) "- k -
For an elementary proof of this fact, as well as for further results on the LR-flow (26) we refer to [30], Watkins and Elsner [31]. 2.5
Q1R,-FLOW
There has been extensive work on studying a continuous-time analogue of the QR-algorithm. As an incomplete list of references let us mention here Symes (1982), Deift, Nanda and Tomei (1983), Chu (1984), Watkins (1984), Watkins and Elsner (1989). For any A E ~nx,~ let 0
Am
~"
a21
--a21 9 9"
anl
. . . .
" " 9
anl
9
9
" 9
--an,n-1
an,n-1
0
denote the skew-symmetric part. Similarly we can compute (log A)_ for any choice of a logarithm of A. The result then is that the QR-algorithm is obtained via constant time sampling of the isospectral flow
Isospectral Matrix Flows for Numerical Analysis
43
.,4 = [A, (log A)_]. That is, for any k E N0,
A(k)
= k-
th step in the qR-algorithm applied to A(0).
Similar results also exist for versions of the QR-algorithm incorporating shifts; see Watkins and Elsner [32]. S
THE DOUBLE BRACKET
FLOW
Brockett [5] has introduced a new class of isospectral flows on the set of symmetric matrices. He considers the double Lie bracket equation )(
= =
[X,[X,A]] AX 2 + X2A-
2XAX
(27)
where [A, B] = A B - B A denote the Lie bracket and A, X are real symmetric n x n matrices. Here the matrix A plays the role of a parameter to guide the system to a desired equilibrium state. The matrix differential equation (27) has remarkable properties" it can be used to diagonalize symmetric matrices, solve linear programming problems and for sorting lists of real numbers. It generalizes both the Toda flow as well as matrix Riccati equations. Moreover related differential equations serve as learning equations for neural networks and can model arbitrary finite automata. In this paper we focus on applications to numerical linear algebra, viz. symmetric matrix diagonalization and singular value decomposition. As Brockett's equation (27) is a gradient flow which minimizes a matrix least squares distance problem with spectral constraints we first discuss such minimization tasks. 3.1
A MATRIX LEAST SQUARES MINIMIZATION PROBLEM
Let A'=diag(A~,...,An)
,
A~_>...>A~.
(28)
The set of real symmetric n x n matrices with eigenvalues A1,..., An is the isospectral set M(A) =
{OAOT I ooT = I).
(29)
M(A) is an orbit of the Lie group action ,7: SO(n) x ~"• (e,X)
~
~nxn
~
exe T
(30)
44
U. Helmke
and is therefore a smooth, compact and connected submanifold of IRnx'~. Here SO(n) denotes the Lie group of real orthogonal n x n matrices with determinant one. In fact, M(A) is a homogeneous space (a generalized flag manifold) and is thus a well known space from topology. The fact that M(A) is a homogeneous space of a Lie group action simplifies the subsequent geometric analysis considerably and enables us to compute in a rather straightforward way tangent spaces and Riemannian metrics on M(A). Given any symmetric n x n matrix A = A T E n~n•
we consider the smooth least squares distance function fA" M(A)
, I~
fA(X)--IIA- X l l 2 --IIAll 2 + IIXll 2 - 2tr(AX) for the Frobenius norm []Xll2 - t r ( X X T ) .
(31) From (31) and orthogonal invariance of the
Frobenius norm IIXll2 = IIA[I2 = ~ A~ we deduce that minimization of f A ( X ) i s equivalent i=1 to maximization of the trace function t r ( A X ) on M(A). Our first rcsult is slight extension of a result of Brockett [5]; see also Helmke and Moore (1994). T h e o r e m 2.1
(a) A matrix X e M(A) is a critical point of fA: M(A) --* IR if and only if A X = X A . (b) The local minima coincide with the global minima. (c) The set of global minima is connected. (d) f A : M ( A ) ~
I~ is a Morse-Bott function.
Proof For a proof of (a), (b) we refer to Brockett [5], Helmke and Moore [19]. Part (d) is shown in Duistermaat, Kolk and Varadarajan (1983), while (c) follows from the explicit characterization of the global minima, given in Brockett [5]. D We specialize the above result to two important cases C a s e 1: D i a g o n a l i z a t i o n For matrix diagonalization we choose A as A= diag(#l,...,#~)
,
#1 > ... > #~.
Then Theorem 1 specializes to C o r o l l a r y 2.2 Let A be as above. Then
Isospectral Matrix Flows for Numerical Analysis
45
(a) fA" M(A) ~ tt~ has ezactly n! critical points. These are diagonal matrices X = diag(A~(1),..., A~(n))
where r: { 1 , . . . , n} ---, { 1 , . . . , n} is any permutation. (b) There is a unique local = global minimum X = diag(A1,...,A,~)
,
A~ >_ ... >_ A~.
All other critical points diag(A,r(1),..., A~(n)) for r # id are saddle points. (c) The Hessian of fA" M(A) ~ I~ at each critical point is nondegenerate, for A1 > . . . > An. El
Case 2: Block Diagonalization This case is of obvious interest for principal component analysis. Here we choose A = diag(1,..., 1,0,...,0), the eigenvalue 1 appearing k times. Theorem 1 then specializes to C o r o l l a r y 2.3 Let A be as above. Then (a) The critical points of fA" M(A) ---, [~ are all block diagonal matrices X = diag(Xll,X22) E M(A)
with Xlx E IRkxk. (b) The set of all local = global minima consists of all block diagonal matrices X = diag(Xll,X22)
where Xxl has eigenvalues ~1 >_ ... >__Ak
and X22 has eigenvalues Ak+l >_ ... _> A,~.
3.2
THE DOUBLE BRACKET FLOW
The above results show that we can characterize the tasks of symmetric matrix diagonalization or block diagonalization as a constrained optimization problem for the cost function f A : M ( A ) ~ IR. A standard approach to tackle optimization problems on manifolds is by a steepest descent approach, following the solution of a suitable gradient flow. What is then the gradient flow of fA: M(A) ---, IR? To compute the gradient of fA one have to specify a Riemannian metric on M(A). As the answer depends crucially on the proper choice of a Riemannian metric it is important to make a good choice.
46
U. Helmke
It turns out that there is indeed (in some sense) such a best possible metric, the so-called normal Riemannian metric. With respect to this normal metric the double bracket flow (27) is then seen as the gradient flow of fA: M(A) ~ II~; see Bloch, Brockett, and Ratiu (1990). All other choices of Riemannian metrics lead to much more complicated differential equations. T h e o r e m 2.4 ( B r o c k e t t (1988), Bloch, B r o c k e t t a n d l:tatiu (1990)). Let A = A T E I~ nXn. (a) The gradient flow of fA: M(A)---, R is the double bracket flow )( = IX, [X, A]] on symmetric matrices X = X T E I~nx'~.
(b) Th~
diff~,'~ntial~quatio,~ (27) d ~ n ~
matrices X E ]~nxn.
a~ i~osp~ctratflow on th~ ~t o/,'~al~ymm~t,'i~
(c) Every solution X ( t ) of (27) exists/or all t e It~ and converges to an equilibrium point X ~ . The equilibrium points of (27) are characterized by [A, X~] = 0, i.e. by A X ~ = X ~ A . (d) Zet A = d i a g ( m , . . . , ~ ) with #1 > . . . > #,~. Every solution X ( t ) of (27) converges to lim X ( t ) = diag()~TrO),... )~r(n))
t----~oo
'
for a suitable permutation 7r: { 1 , . . . , n} ~ { 1 , . . . , n}. There exists an open and dense subset U C M(A) such that for every initial condition X(O) E U then lim Z ( t ) = diag(,\l,...An)
,
~1 >_ ..._> ~ .
t--*oo
Moreover, convergence of X ( t ) to this unique attractor of the flow is exponentially fast.
Thus solving the double bracket flow leads us to a diagonalization method for symmetric matrices. Of course, this is a rather naive method which is not recommended in practice! D o u b l e B r a c k e t Flow D i a g o n a l i z a t i o n M e t h o d 9 The to be diagonalized matrix X0 is the initial condition of (27). 9 Choose, e.g., A = d i a g ( n , n - 1 , . . . , 1 ) . 9 Solve the differential equation (27)! Then X ( ~ ) = diag(A1,..., An)
with ) q , . . . , An the eigenvalues of X0.
Isospectral Matrix Flows for Numerical Analysis
3.3
47
DISCRETIZATIONS
A major drawback of the double bracket flow diagonalization method is that the solutions of a nonlinear ODE have to be computed. For this it would be most helpful if explicit solutions of the double bracket flow were known. Unfortunately this appears to be possible only in one simple case.
3.3.1 Exact Solutions Suppose that X is a projection operator, that is X 2 = X holds. Then the double bracket equation (27) is equivalent to the matrix Riccati equation = AX + XA-
2XAX
(32)
For this Riccati equation explicit solutions are well-known. We obtain the formula
x ( t ) = r177
Xo + d'AXo)-~r 'A
(33)
for the solution of the double bracket flow (27) with X02 = X0, X0T = X0. Note that (33) defines indeed a projection operator for each t E I~. We use this formula to show that discretizing (27) at integer time t = k E N is equivalent to the power method for the matrix eA. For simplicity let us restrict to the case where rkXo = 1. Connection with the Power Method Suppose X(0) is a rank 1 projection operator. Then the solution formula (33) simplifies to
X(t)-
etAX(O)e tA tr(e2tAX(O)).
Thus
eAX(k)e A
x(k + 1)= t~(~2AX(k))"
(34)
Since X is a rank 1 projection operator we have X = ~ T for a unit vector ~ E I~n. The above recursion (34) is then obtained by squaring up the power iterations for the matrix eA:
eA~k O
3.3.2 Geodesic Approximation As we have mentioned above, it is in general not possible to find explicit solution formulas for the double bracket equation. Thus one is forced to look for approximate solutions. A straightforward although naive approach would be to discretize numerically the ODE on the space of symmetric matrices, using Euler type
48
U. Helmke
or Runge-Kutta methods. Such a standard discretization method is however not recommended, as it may lead to large errors in the eigenvalues. The difficulty with such an approach is that the fixed eigenvalue constraints are not satisfied during the iterations. Thus our goal is to find suitable approximations which preserve the isospectral nature of the flow. Such approximate solutions are given by the geodesics of M(A). The geodesics of the homogeneous space M(A) can be explicitly computed. They are of the form e t ~ X e -t~
,
X E M(A)
where f~ is any skew-symmetric matrix in the tangent space of M(A) at X. We are thus led to consider the following Geodesic Approximation Scheme Xk+l = e~k[Xk'A]Xke-~k[Xk'A]
(35)
for a still to be determined choice of a step-size factor ak. We assume, that there exists a function aA:M(A) ~ It~ such that ~k = a A ( X k ) for all k. Moreover we require that the step-size factor satisfies the following condition. Boundedness Condition There exist 0 < a < b < oc, with C~A:M(A)---, [a,b] continuous everywhere, except possibly at those points X E M(A) where A X = X A.
We then have the following general convergence result of (35). P r o p o s i t i o n ([22]) Let A = diag(#l,. . ., I.tn) with #1 > . . . > #n and let aA" M(A) ~ [a, b] satisfy the Boundedness Condition. For Xk E M(A) let ~k = a A ( X k ) and define A f A ( X k , ak) = t r ( A ( X k - Xk+l )) where Xk+l is given by (35). Suppose A f A ( X k , ak) < 0
whenever [A, Xk] # O.
Then
(a) The fixed points X e M(A) of the discrete-time iteration (35) are characterized by AX = XA. (b) Every solution Xk, for k e N, of (35) converges as k ---, oc to a diagonal matrix Xoo = diag()~r(1), . " , )~r(n)), :r a permutation. []
Note that the above result does not claim that Xk converges to the optimal fixed point A, which minimizes fA" M(A) ~ ~. In fact, the algorithm will converge to this local attractor A = diag(A1,..., A~), with A1 _> ... >_ A,~, only for generic initial conditions. For
Isospectral Matrix Flows for Numerical Analysis
49
a nongeneric initial condition the algorithm may well get stuck in a saddle point. This is in sharp contrast to the globally convergent Sort-Jacobi algorithm which will be introduced later. The following examples yield step-size selections ak which satisfy the assumption of the Proposition. E x a m p l e 1: C o n s t a n t Step-Size ([22]) 1 o~ : 4IIAll " ilXoll
(36)
E x a m p l e 2" Variable Step-Size I ([22])
1 I[A, Xdll = ) ~k--II[A, Xk]ll log(1 + IlXo : [i[-~:~.:x~]]ll
(a7)
E x a m p l e 3: Variable Step-Size II ([6])
211[A, Xk]ll 2
~k -- II[A, [A, Xk]]ll" II[[A, Xk], Xk]ll
(as)
In all three cases the isospectral algorithm (35) converges to a diagonalization. While the last variable step-size (38) is always larger than the (36) or (37), the on-line evaluation of the norms of Lie brackets, in comparison with the constant step size, causes additional computational problems, which may thus slow down overall convergence speed. What are the advantages and disadvantages of the proposed geodesic approximation method? 1. The algorithm (35) is isospectral and thus preserves the eigenvalues. 2. Linear convergence rates hold in a neighbourhood of the global attractor. 3. By combining the method with known quadratically convergent optimization schemes, such as the Newton method or a conjugate gradient method, local quadratic convergence can be guaranteed. However, the domains in M(A) where quadratic convergence holds can be rather small, in addition to the hugh computational complexity one faces by implementing e.g. Newton's method. Thus these quadratically convergent modifications are not suitable in practice. See the recent Ph. D. thesis of R. Mahony [21] and S. Smith [26] for coordinate free description of such methods. 4. Despite of all such possible advantages of the above geodesic approximation algorithm, the bottleneck of the method really is the need to compute on-line matrix exponentials. This makes such gradient-flow based schemes hardly practical. In section 5 we thus describe a way how to circumvent such difficulties.
U. Helmke
50 4
GENERALIZATIONS
The previous theory of double bracket flows can be extended in a straightforward way to compute eigenvectors or the singular value decomposition. Moreover, the whole theory is easily generalized to flows on Lie algebras of compact semisimple Lie groups.
4.1
FLOWS ON ORTHOGONAL MATRICES
Let SO(n) denote the Lie group of n x n real orthogonal matrices of determinant one and let A, X0 be n x n real symmetric matrices. We consider the task of optimizing the smooth function CA,Xo: SO(n) ----* I~
CA,Xo = [[A- OXoOTII2;
,
(39)
this time the optimization taking place on the Lie group SO(n) rather than on the homogeneous space M(Xo). We have the following analogous result of Theorem 2.1 by Brockett. Let us assume for simplicity that A = d i a g ( # l , . . . , #n) with #1 > ... > #~.
Theorem 3.1 (Brockett (1988)) (a) A matrix O e SO(n) is a critical point of CA,x0 : SO(n) ---, I~ if and only if[A, OXo| T] = O. Equivalently, for A = d i a g ( # l , . . . , # n ) , with #1 > ... > #n, Q E SO(n) is a critical point if and only if the row vectors of | form an orthonormal basis of eigenvectors of Xo. (b) The local minima of CA,Xo : S O ( n ) ~ (c) CA,X0: S O ( n ) ~
~ coincide with the global minima.
I~ is a Morse-Bott function.
D
The gradient flow of CA,X0: SO(n) ---, It~ with respect to the (normalized) Killing form is easily computed to be the cubic matrix differential equation
(9 = [|174
,
|
E SO(n).
(40)
Moreover, the solutions | E SO(n) exist for all t E It( and converge, as t ~ =t=~, to an orthogonal basis of eigenvectors of X0; see Brockett [5]. The associated recursions on orthogonal matrices which discretize the gradient flow (40) are
Ok+l = eak[Okx~174
(41)
where ak is any step-size selection as in section 3.3. In particular, if ak is chosen as in (36) - (37) (with Xk = QkXoQT), then the convergence properties of the gradient flow (2) remain in force for the recursive system (40); see [22] for details. Thus
Ok+~ ' e~[O~X~
,
o~ = 1/(411Xoll
IIAII),
(42)
Isospectral Matrix Flows for Numerical Analysis
51
converges to a basis of eigenvectors of X0. The rate of convergence in a neighbourhood of the stable equilibrium point is linear.
4.2
SINGULAR VALUE D E C O M P O S I T I O N
One can parallel the analysis of double bracket flows for symmetric matrix diagonalization with a corresponding theory for flows computing the singular value decomposition of rectangular matrices 9 For a full analysis we refer to [18], [25]. Let A, E E IR'~xm, m >_ n, be rectangular matrices of the form
A:=
E'=
#1
"'"
0
.
:
"..
:
9
0
...
#,~
o'1
.."
0
:
"..
:
0
"''
(7" n
O~,x(~_,-,)
(43)
: 0~x(.~_~)
(44)
.
with al _> 9149149 > a,~ >_ 0 and #1 > . . . > #n > 0. The Lie group O(n) x O(m) acts on the matrix space I~ n x m by ~: o ( ~ ) • o ( m ) • ~ •
((u, v ) , x )
--~ ~-~
~• uxy T
(45)
Thus the set M ( E ) of n x m real matrices X with singular values a l , . . . , an M ( E ) = { u E v T I u e O(n), V E O(m)}
(46)
is an orbit of a and therefore M ( E ) is a smooth, compact manifold. Consider the task of optimizing the smooth least squares distance function
FA: M ( E ) ---* II~ FA(X) =
IIA- XII 2
= E (o? + ~ ) - 2t~(AX~). i-1
For proofs of the following results we refer to [18], [25]; see also [9].
(47)
52
U. Helmke
T h e o r e m 3.2
(a) A matrix X E M ( 2 ) is a critical point of FA: M ( 2 ) - - , IR if and only if A X T = X A T and A T x = X T A holds. Equivalently, for A as in (~3) and al > ... > an > O, X is a critical point if an only if X = (diag(exa~(1),...,ena~(,~)), O,~•
(48)
with ei E {q-l} and rr: { 1 , . . . n } --~ { 1 , . . . , n} is a permutation.
(b) The local minima coincide with the global minima. T h e o r e m 3.3
(a) The gradient flow of the least squares distance function FA: M ( E ) ---, IR, F A ( X ) =
IIA- Xll 2,
is
2 = (AX T- xAT)X
- x(ATX
- xTA).
(49)
This differential equation defines a self-equivalent, i.e. singular value preserving, flow on I~ r~• m .
(b) The solutions X ( t ) of (~9) exist for all t e I~ and converge to a critical point of FA: M ( E ) ~ I~. In particular, every solution X ( t ) converges to a matrix Xr
= (diag(Ax,...,)~n), Onx(m-,~))
where ])ql, i = 1 , . . . , n, are the singular values of X(O) (up to a possible re-ordering).
(50) []
Similarly also recursive versions of (49) and corresponding flows for the singular vectors exist. We refer to [22] for details.
4.3
FLOWS ON LIE ALGEBRAS
Brockett's double bracket equation generalizes in a straightforward way to flows on Lie algebras. The importance of such generalizations lies in the ability to cope in a systematic way with various structural constraints, such as for skew-symmetric or Hamiltonian matrices. The standard example of a Lie algebra is the Lie algebra of skew-symmetric matrices. 1 More generally let us consider an arbitrary compact semi-simple Lie group G with Lie algebra ~. Then G acts on ~ via the adjoint representation Ad: G x ~
----* r
(g, ~) ~
Ad(g) .
1The set of symmetric matrices is not a Lie algebra, but a Jordan algebra!
(51)
Isospectral Matrix Flows for Numerical Analysis
53
where Ad(g). ~ is defined as d.
t~
Ad(g) . ~ = --~(ge g-1)lt=o. By differentiating Ad(g).~ with respect to the variable g we obtain the Lie bracket operation
ad: ~ x ~
(~,,7)
~
---,
[~, hi.
(52)
We also use the notation ad~(~)= [~, 7/]. Then the orbit
M(~) = {Ad(g). ~ l g e G)
(53)
is a smooth compact submanifold of the Lie algebra qS. If G is semi-simple, then the Killing form (~, r/): = -tr(ad~ o adn)
(54)
defines a positive definite inner product on @. Consider for a fixed element 7/E q5 the trace function Cn:M(~)
~~
,
Cn(~) = ((,77).
(55)
We now have the following natural generalization of Theorems 2.1, 2.4 to arbitrary compact semi-simple Lie algebras.
Theorem 3.4 (Bloch, Brockett, Ratiu (1990)) (a) There exists a Riemannian metric on M(~) such that the double Lie bracket flow
= [~, [~, ~]]
(56)
is the gradient flow of Cn: M(~) ~ ~ . (b) Every solution ((t) of (17) converges to an equilibrium point ~oo, characterized by [~oo, ~] = o. Therefore the double bracket flow (17) converges to the intersection of tow sets, one being defined by the a spectral constraints ~oo E M(~) while the other set is defined by the linear constraint [~oo,r/] = 0. Thus such flows are capable of solving certain structured inverse eigenvalue problems. This can in fact be done in more generality; cf. [10] for further examples and ideas.
54
U. Helmke
Remarks
(a) It is not quite proved (nor even stated!) in [4] that every solution r of the gradient flow (56) really converges to a single equilibrium point, rather than a set of equilibria. While this would be a trivial consequence of standard properties of gradient flows if (56) were known to have only finitely many equilibrium points, it is not so obvious in case (16) has a continuum of equilibria. However, as r M(~) ~ I~ can be shown to be a MorseBott function, convergence to a single equilibrium point then it follows from appropriate generalizations of standard convergence results of gradient flows; see e.g. Helmke and Moore (1994), Prop. 1.3.6. (b) There is a step-size selection ak similar to (38), such that the recursion ~k+l -- Ad(e-~
(57)
9~k
has the same convergence properties as (56); see Brockett (1993).
5
SORT-JACOBI
ALGORITHM
In the previous sections we have seen that double Lie bracket isospectral flows and their respective discretizations lead to convergent algorithms for matrix diagonalization. Although such algorithms are attractive from a theoretical viewpoint, they do lead only to highly unpractical, slowly convergent algorithms. Our goal in this section is to show how modifications of such algorithms yield efficient, quadratically convergent algorithms for matrix diagonalization. The algorithm which we derive is actually a modification of the classical Jacobi algorithm, which also incorporates sorting of the eigenvalues. This part is forthcoming joint work with K. Hiiper, where full details will appear.
5.1
MATRIX DIAGONALIZATION
We use the notation of section 3. Let ei E I~n denote the i-th standard basis vector of I~n. For 1 _< i < j _ n and t E I~ let Gij(t) = cos(t). (eie T + eje T) - sin(t). (eie T - eje T)
(58)
denote the Jacobi-rotation in the (i,j)-plane. Using any fixed ordering of {(i,j) E 1~21 1 _< i < j _< n} we denote by
G~(t),
. . ., GN(t)
,
lV =
ln(n-
1)
(59)
the Jacobi rotations of I~n. The Sort-Jacobi algorithm is build up similarly to standard numerical eigenvalue methods via the iteration of sweeps.
Isospectral Matrix Flows for Numerical Analysis
55
k-th Sort-Jacobi Sweep Define
X 0)
.=
G l ( t O ) ) X k G l ( - t (1))
X~ 2) .= G2(t!2))X~I)G2(-t!2) ) X~ N) "---- GN(t!N)~ y(N-1)GN(
)
(60)
where t! 0 9= arg re.in .{fA(Gi(t J"k ~ ( i - 1 )Gi(-t)}. tE[0,2~rJ for/=
(61)
1,...,N.
Thus X~ i) is recursively defined as the minimum of the least squares distance function
fA: M(Zo) ~ ~, when restricted to the i-th Jacobi geodesic {Gi(t)X~i-1)Gi(-t) l t e I~} r(~-~)
containing ~ k
9
The Sort-Jacobi algorithm then consists of the iteration of sweeps. Sort-Jacobi Algorithm 9 Let X o , . . . , X k
E M(Xo) be given for k E H0.
9 Define the recursive sequence
Xk(1) , X~2) , . . . ,
x~N)
as above (sweep). 9 Set Xk+l"= X~ N). Proceed with the next sweep.
The convergence properties of the algorithm are established by the following result. T h e o r e m 4.1 Let Xo = XTo E Xn•
be given and A = diag(#l,. . .,#n) with #1 > ... > #n. Let Xk, k = O, 1, 2,..., denote the sequence generated by the Sort-Jacobi algorithm. Then
(a) Global convergence holds, that is lim Xk = diag(11,... k---,oo
1,~)
(62)
with A1 >_ ... _> A~.
(63)
U. Helmke
56
Here hx,...,)~n are the eigenvalues of Xo. (b) The algorithm is quadratically convergent. Remarks
(a) The above algorithm automatically sorts the eigenvalues of X0 in a given order. If the diagonal entries #i in A were, e.g., chosen in an increasing way, than the eigenvalues A1,...,An of X0 would also appear in an increasing order. Similarly for any other permutation. This property does not hold for standard versions of the aacobi algorithm. (b) The theorem remains in force for any a priori chosen order, in which the sweep steps are applied. (c) The basic difference to standard textbook Jacobi algorithms is that we are maximizing the linear trace function tr(AX) in each step, rather than minimizing the quadratic Jacobi cost function Iloffdiag(X)ll ~. This is important for the proof, as the critical point structure of the trace function on M(A) is much simpler that that of I]offdiag(X)ll 2. This difference in the choice of cost functions is the main reason why our theory appears to be simpler than the classical theory of aacobi algorithms. P r o o f of T h e o r e m 4.1 ( S k e t c h ) For details we refer to our forthcoming work Hiiper and Helmke (1994). First we establish global convergence. By construction
/A(Xk+,) < /A(Zk). It is easily seen that equality holds if and only if Xk+l = Xk is a critical point of fA: M(Xo) Ii~. By a Lyapunov type argument we deduce that the sequence (Xk) converges to a critical point of fA: M(Xo) ~ Ii~; i.e. to a diagonal matrix X ~ . Suppose Xr162were a saddle point. Then a single sweep involving permutations, applied to Xk for k sufficiently large, would bring us arbitrarily close to the unique global minimum. Contradiction. Thus (Xk) converges to the global minimum of the cost function. To prove quadratic convergence we consider the algorithm in a neighbourhood of the global minimum A. Using the implicit function theorem one shows that there exists a smooth map ~p: U + U, defined on an open neighbourhood U C M(A) of A, such that for all sufficiently large k
Xk+~ = ~(Xk) holds. Thus the algorithm consists in iterating a locally defined smooth map of M(A). A calculation shows that the first derivative of ~: U ~ U at the fixed point A is zero! Thus the result follows from a standard Taylor expansion argument. Q.E.D. []
Isospectral Matrix Flows for Numerical Analysis 5.2
57
SVD
In an analogous way a Sort-Jacobi algorithm which computes the singular values of a matrix is obtained. Let A" = ( d i a g ( # : , . . . , #n),
0nx(m-~))
(64)
E : = (diag(a:,...,a,~),
0n•
(65)
with #: > ... > #n > 0 and a: _> .. >_ an _> 0. Using the well-known transformation
f~'--
[0A AT
0
'
0
we can reduce the problem to the symmetric case. Thus for any matrix Xo E I~n• with singular values a l , . . . , an we can apply the Sort-Jacobi algorithm to the symmetric (m + n) • (m + n) matrix )(0. The only difference now is that we only perform those Jacobi iterations which do preserve the structure of )(0. That is we are applying, in each sweep, only Jacobi-iterations of the form G:XG2 E M ( X ) . We then have an analogous convergence theory for the SVD Sort-Jacobi algorithm. T h e o r e m 4.2 Let Xo E I~n• and let A be defined by (64). W i t h # : >_ ...>_ #n >_ 0 let Xk, k = O, 1, 2,..., denote the sequence generated by the Sort-Jacobi SVD algorithm. Then (a) Global convergence holds, i.e. lira Xk = (diag(a:,... k-,,,,+ ~
an)
On•
(66)
'
with a: > _ . . . > a n > _ O .
(67)
Here a : , . . . , an are the singular values of Xo. (b) The algorithm converges quadratically fast.
Acknowledgement This work was partially supported by the German-Israeli-Foundation for scientific research and development, under grant 1-0184-078.06/91.
58
U. Helmke
References
[1] Ablowitz, M.J. and Clarkson, P.A., Solitons, Nonlinear Evolution Equations and Inverse Scattering, London Math. Soc. Lecture Notes Series 149, Cambridge Univ. Press, 1991 [2] Arnold, V.I., Mathematical Methods of Classical Mechanics, Second Edition, Graduate Texts in Mathematics 60, Springer, New York, 1989 [3] Bloch, A.M., Steepest descent, linear programming and Hamiltonian flows, Contemporary Math. 114, 77-88, 1990 [4] Bloch, A.M., Brockett, R.W. and Ratiu, T., A new formulation of the generalized Toda lattice equations and their fixed point analysis via the moment map, Bull. Amer. Math. Soc., 23,447-456, 1990 [5] Brockett, I~.W., Dynamical systems that sort lists and solve linear programming problems, Proc. 27th Conference on Decision and Control, Austin, TS, 779-803. See also Linear Algebra Appl. 146 (1991), 79-91, 1988 [6] Brockett, I~.W., Differential geometry and the design of gradient algorithms, Proceedings of Symposia in Pure Mathematics, 54, 69-91, 1993 [7] Chu, M.T., The generalized Toda flow, the QR algorithm and the center manifold theory, SIAM J. Disc. Meth., 5,187-201, 1984b [8] Chu, M.T., A differential equation approach to the singular value decomposition of bidiagonal matrices, Linear Algebra and its Appl., 80, 71-80, 1986 [9] Chu, M.T. and Driessel, K.R., The projected gradient method for least squares matrix approximations with spectral constraints, SIAM J. Numer. Anal. 27, 1050-1060, 1990 [10] Chu, M.T., Matrix differential equations: A continuous realization process for linear algebra problems, Nonlinear Analysis, TMA 18, 1125-1146, 1992a [11] Deift, P., Nanda, T. and Tomei, C., Ordinary differential equations for the symmetric eigenvalue problem, SIAM J. Numer. Anal. 20, 1-22, 1983 [12] Duistermaat, J.J., Kolk, J.A.C. and Varadarajan, V.S., Functions, flow and oscillatory integrals on flag manifolds and conjugacy classes in real semisimple Lie groups, Compositio Math. 49,309-398, 1983 [13] Faybusovich, L., Toda flows and isospectral manifolds, Proc. American Mathematical Society 115,837-847, 1992 [14] Flaschka, H., The Toda lattice, I, Phys. Rev. B 9, 1924-1925, 1974 [15] Flaschka, H., Discrete and periodic illustrations of some aspects of the inverse methods, in dynamical system, theory and applications, J. Moser, ed., Lecture Notes in Physics, 38, Springer-Verlag, Berlin, 1975
Isospectral Matrix Flows for Numerical Analysis
59
[16] Fomenko, A.T., Symplectic Geometry, Advanced Studies in Contemporary Mathematics, Vol. 5, Gordon and Breach Publ., New York, 1988 [17] Golub, G.H. and Van Loan, C.F., Matrix Computations, Second Edition, The John Hopkins University Press, Baltimore, 1989 [18] Helmke, U. and Moore, J.B., Singular value decomposition via gradient and self equivalent flows, Linear Algebra and its Appl. 69,223-248, 1992 [19] Helmke, U. and Moore, J.B., Optimization and Dynamical Systems, CCES, SpringerVerlag, London, 1994 [20] Kostant, B., The Solution to a generalized Toda lattice and representation theory, Advances in Mathematics 34, 195-338, 1979 [21] Mahony, R., Optimization Algorithms on Homogeneous @aces, Ph.D. Thesis, Canberra, ANU, 1994 [22] Moore, J.B., Mahony, R.E. and Helmke, U., Numerical gradient algorithms for eigenvalue and singular value decomposition, SIAM J. of Matrix Analysis, Appl., Vol. 15, 881-902, 1994 [23] Moser, J., Finitely many mass points on the line under the influence of an exponential potential- An integrable system, in J. Moser, Ed., Dynamical Systems Theory and Applications, Springer-Verlag, Berlin - New York, 467-497, 1975 [24] Rutishauser, H., Ein InfinitesimaIes Analogon zum Algorithmus, Arch. Math., (Basel), 5, 132-137, 1954
Quo tienten- Diff erenzen-
[25] Smith, S.T., Dynamical systems that perform the singular value decomposition, Systems and Control Letters 16,319-328, 1991 [26] Smith, S.T., Geometric optimization methods for adaptive filtering, PhD Thesis, Harvard University, 1993 [27] Symes, W.W., Systems of the Toda type, inverse spectral problems and representation theory, Inventiones Mathematicae 59, 13-51, 1980 [28] Symes, W.W., The QR algorithm and scattering for the finite nonperiodic Toda lattice, Physica 4D, 275-280, 1982 [29] Tomei, C., The topology of isospectral manifolds of tri-diagonal matrices, Duke Math. Journal 51,981-996, 1984 [30] Watkins, D.S., Isospectral Flows, SIAM Rev., 26,379-391, 1984 [31] Watkins, D.S. and Elsner, L., Self-similar flows, Linear Algebra and its Appl. 110, 213-242, 1988 [32] Watkins, D.S. and Eisner, L., Self-equivalent flows associated with the singular value decomposition, SIAM J. Matrix Anal. Appl. 10, 244-258, 1989
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
THE RIEMANNIAN
SINGULAR
61
VALUE DECOMPOSITION
B.L.R. DE MOOR ESAT-SISTA, K a t h o l i e k e Universiteit L e u v e n K a r d i n a a l M e r c i e r l a a n 9~ 3001 L e u v e n , Belgium bart. demoor@esat, kuleu ven. ac. be
ABSTRACT. We define a nonlinear generalization of the SVD, which can be interpreted as a restricted SVD with Riemannian metrics in the column and row space. This so-called Riemannian SVD occurs in structured and weighted total least squares problems, for instance in the least squares approximation of a given matrix A by a rank deficient Hankel matrix B. Several algorithms to find the 'minimizing' singular triplet are suggested. This paper reveals interesting and sometimes unexplored connections between linear algebra (structured matrix problems), numerical analysis (algorithms), optimization theory, (differential) geometry and system theory (differential equations, stability, Lyapunov functions). We also point out some open problems. KEYWORDS. (Restricted) singular value decomposition, gradient flows, differential geometry, continuous algorithms, total least squares problems, power method.
1
THE RIEMANNIAN
SVD: M O T I V A T I O N
Since the work by Eckart-Young [14], we know how to obtain the best rank deficient least squares approximation of a given matrix A E Itr215 of full column rank q. This approximation follows from the SVD of A by subtracting from A the rank one matrix u . a . v T, where (u, a, v) is the singular triplet corresponding to the smallest singular value a, which satisfies A v = uo" ~ ATu : va ,
uTu = 1 , v T v : 1.
(1)
Here u E li~p and v E ~q are the corresponding left resp. right singular vector. When formulated as an optimization problem, we obtain B
min
E It~T M
y EIt~q
I]A-BI] ~
subject to
By=0, yTy = 1 ,
(2)
62
B.L.R. De Moor
the solution of which follows from (1) as B = A - u a v T, y - v. Here, a is the smallest singular value of A and it is easy to prove that minB I I A - BII ~ = a 2 if S is required to be rank deficient. This problem is also known as the Total (Linear) Least Squares (TLS) problem and has a long history [1] [2] [14] [16] [17] [20] [25] [27] [28]. The vector y describes a linear relation between the columns of the approximating matrix B, which therefore is rank deficient as required. In this paper we consider two generalizations of the TLS problem. First of all, in the remainder of this Section, we describe how the introduction of user-given weights and/or the requirement that the approximant B is to be structured (e,g. Hankel or Toeplitz), leads to a (nonlinear) generalization of the SVD, which we call the R i e m a n n i a n S V D . In Section 2, we show that the analysis and design of continuous time algorithms provides useful insight in many problems and how they also provide a unifying framework in which several mathematical engineering disciplines meet. In Section 3, we discuss the power method in more or less detail to illustrate this claim. We also derive some new continuous-time algorithms, that are not yet completely understood. In Section 4, we discuss some ideas to generate algorithms for the Riemannian SVD. Conclusions are presented in Section 5. There are at least two important variations on the Total Least Squares problem (2) 1. The first one is called the Weighted TLS problem: p q min ~_~ ~ ( a i j - bij)2wij subject to B 6 n~pxq i=lj=l y6N q
By.o..-_ 0,1 tJ~u '
(3)
where the scalars wij E 1~+ are user-defined weights. An example is given by chosing wii - 1/a~j in which case one minimizes (a first order approximation to) the sum of relative errors squared (instead of the sum of absolute errors squared as in (2)). Another example corresponds to the choice wij E {0, 1) in which case some of the elements of B will be the same as corresponding elements of A (namely the ones that correspond to a weight wij - 1, see also the Example in Section 4.2). Yet other examples of weighted TLS problems are given in [10] [11]. The second extension consists of adding a so-called structural constraint to the optimization problem (2), in which case we have a Structured TLS problem. For instance, we could require the matrix B to be a Hankel matrix. The problem we are solving then is to approximate a given (possible structured) full column rank matrix A by a rank deficient Hankel matrix B. Rank deficient Hankel matrices in particular have important applications in systems and control theory. But in general there are many other applications where it is required to find rank deficient structured approximants [10] [11] [12] [13]. The results of this paper apply to affine matrix structures, i.e. matrices that can be written as a linear combination of a given set of fixed basis matrices. Examples are (block) Hankel, Toeplitz, 1The list of constraints discussed in this paper is not exhaustive. For instance, one could impose constraints on the vectors y in the null space, an example that is not treated here (see e.g. [9]).
63
The Riemannian SVD
Brownian, circulant matrices, matrices with a given zero pattern or with fixed entries, etc... The main result, which we have derived in [10], applies to both Weighted and/or Structured TLS problems and states that these problems can be solved by obtaining the singular triplet that corresponds to the smallest singular value that satisfies: Av-
u T D v u = 1,
D~ua ,
ATu---- D u v a ,
vTDuv-
1.
(4)
Notice the similarity with the SVD expressions in (1). Here A is the structured data matrix that one wants to approximate by a rank deficient one. Du and Dv are nonnegative or positive definite matrix functions of the components of the left and right singular vectors u and v. Their precise structure depends on the weights in (3) and/or the required affine structure of the rank deficient approximant B. To give one example (which happens to be a combination of a Weighted and Structured TLS problem), let us consider the approximation of a full column rank Hankel matrix A 9 ~pxq, p _> q, rank(A) = q, by a rank deficient Hankel matrix B such that [[A- S[[~ is minimized. In this case, the matrix D~ has the form Dv - T v W - 1 T T where W=diag[123
...
qq ...q
...321],
y-
(p-q+l)
times
iv v v oool
and Tv is a banded Toeplitz matrix (illustrated here for the case p - 4, q - 3) of the form"
T~ =
0
V1
V2
V3
0
0
0 0
0 0
Vl 0
v2 vx
va v2
0 v3
"
The matrix Du is constructed similarly as D~, - T u W - 1 T T. Obviously, in this example, both Du and D~ are positive definite matrices. Observe that B has disappeared from the picture, but it can be reconstructed as B = A - multilinear function of (u, a, v), (see the constructive proof in [11] for details). Observe that the modification to A is no longer a rank one matrix as is the case with the 'unstructured' TLS problem (2). Instead, the modification is a multilinear function of the 'smallest' singular triplet, the detailed formulas of which can be found in [10] [11]. We are interested in finding the smallest singular value in (4) because it can be shown that its square is precisely equal to the object function. For more details and properties and other examples and expressions for Du and Dv for weighted and structured total least squares problems we refer to [10] [11] [12] [13]. In the special case that D~ = Ip and Dv = Iq we obtain the SVD expressions (1). In the case that D~ and Dv are fixed positive definite matrices that are independent of u and v, one obtains the so-called Restricted SVD, which is extensively studied in [7] together with some structured/weighted TLS problems for which it provides a solution. In the Restricted
B.L.R. De Moor
54
SVD, D~ and D~ are positive (or nonnegative) definite matrices which can be associated to a certain inner product in the column and row space of A. Here in (4), D~ and D~ are also positive (nonnegative) definite, but instead of being constant, their elements are a function of the components of u and v. It turns out that we can interprete these matrices as Riemannian metrics, an interpretation which might be useful when developping new (continuous-time) algorithms. For this reason, we propose to call the equations in (4), the
Riemannian SVD 2. 2
CONTINUOUS-TIME
ALGORITHMS
As discussed in the previous section, in order to solve a (weighted a n d / o r structured) TLS problem, we need the 'smallest' singular triplet of the (Riemannian) SVD. The calculation of the complete SVD is obviously unnecessary (and in the case of the Riemannian SVD there is even no 'complete' decomposition). For the calculation of the smallest singular value and corresponding singular vectors of a given matrix A, one could apply the power method or inverse iteration to the matrix AT A (see e.g. [16] [24] [29]), which will be discussed in some more detail in the next Section. One of the goals of this paper is to point out several interesting connections between linear algebra, optimization theory, numerical analysis, differential geometry and system theory. We also would like to summarize some recent developments by which continuous time algorithms for solving and analyzing numerical problems have gained considerable interest the last decade or so. Roughly speaking, a continuous time method involves a system of differential equations. The idea that a computation can be thought as a flow t h a t starts at a certain initial state and evolves until it reaches an equilibrium point (which then is the desired result of the computation) is a natural one when one thinks about iterative algorithms and even more, about recent developments in natural information processing systems related to artificial neural networks 3. There are several reasons why the study of continuous time algorithms is important. Continuous time methods can provide new additional insights with respect to and shed light upon existing discrete time iterative or recursive algorithms (such as e.g. in [22], where convergence properties of recursive algorithms are analysed via an associated differential equation). In contrast to the local properties of some discrete time methods, the continuous time approach often offers a global picture. Even if they are not particularly competitive with discrete time algorithms, the combination of parallellism and analog implementation does seem promising for some continuous-time algorithms. In many cases, continuous-time algorithms provide an alternative or sometimes even better understanding of discrete-time versions (e.g. in optimization, see the continuous-time version of interior point techniques in [15, p.126], or in numerical analysis, the self-similar 2This name is slightly misleading in the sense that we do NOT want to suggest that there is a complete decomposition with rain(p, q) different singular triplets, which are mutual 'independent' (they are 'orthogonal') and which can for instance be added together to give an additive decomposition of the matrix A (the dyadic decomposition). There might be several solutions to (4) (for some examples, there is only one), but since each of these solutions goes with a different matrix D~ and D~, it is not exaclty clear how these solutions relate to each other, let alone that they would add together in one way or another to obtain the matrix A. SSpecifically for neural nets and SVD, we refer to e.g. [3] [4] [21] [23] (and the references in these papers).
The Riemannian SVD
65
iso-spectral (calculating the eigenvalue decomposition) or self-equivalent (singular value decomposition) matrix differential flows, see the article by Helmke [19] in this Volume and its rather complete list of references). The reader should realize that it is not our intention to claim that these continuous-time algorithms are in any sense competitive with classical algorithms from e.g. numerical analysis. Yet, there are examples in which we have only continuous-time solutions and for which the discrete-time iterative counterpart has not yet been derived. To give one example of a continuous-time unconstrained minimization algorithm, let us derive the continuous-time steepest descent method and prove its convergence to a local minimum. Consider the minimization of a scalar object function f ( z ) in p variables z E I~p. Let its gradient with respect to the elements of z be Vzf(z). It is straightforward to prove that the system of differential equations ~(t) = - V ~ f ( z ) , z ( 0 ) = z0,
(5)
where z0 is a given initial state, will converge to a local minimum. This follows directly from the chain rule: df(z) = (Vzf(z))TL " = dt
_llVzf(z)ll2
(6)
which implies that the time derivative of f ( z ) is always negative, hence, as a function of time, f ( z ) is non-increasing. This really means that the norm of the gradient is a Lyapunov function for the differential equation, proving convergence to a local minimum. In Section 3.8 we will show how to derive continuous-time algorithms for constrained minimization problems, using ideas from differential geometry. 3
CONTINUOUS-TIME PROBLEM
ALGORITHMS
FOR THE EIGENVALUE
The symmetric eigenvalue problem is the following: Given a matrix C - C T E IRpxp, find at least one pair (x, A) with x E I~p and A E I~ such that C x - " x ~ , x T x -- 1 .
(7)
Of course, this problem is of central importance in most engineering fields and we refer to [16] [24] [29] for references and bibliography. Here we will concentrate on power methods, both in continuous and discrete time. The continuous-time power methods are basically systems of vector differential equations. Some of these have been treated in the literature (see e.g. [6] [18] [26]). Others presented here are new. The difference with the matrix differential equations referred to in the previous section is that typically, these vector differential equations compute only one eigenvector or singular vector while the matrix differential equations upon convergence deliver all of them (i.e. the complete decomposition). Obviously, when solving (structured/weighted) TLS problems, we only need the 'minimal' triplet in (1) or (4). For the TLS problem (2) one might also (in principle) calculate the smallest eigenvalue of A T A and its corresponding eigenvector.
B.L.R. De Moor
56
3.1
G E O M E T R I C INTERPRETATION
Consider the two quadratic surfaces xTcx = 1 and xTx = 1. While the second surface is the unit sphere in p dimensions, the first surface can come in many disguises. For instance, when C is positive definite, it is an ellipsoid. In three dimensions, depending on its inertia, it can be a one-sheeted or two-sheeted hyperboloid or a (hyperbolic) cilinder. In higher dimensions, there are many possibilities, the enumeration of which is not relevant right now. In each of these cases, the vectors Cx and x are the normal vectors at x to the two surfaces. Hence, when trying to solve the symmetric eigenvalue problem, we are looking for a vector x such that the normal at x to the surface x T c x = 1 is proportional to the normal at x to the unit sphere xTx -- 1. The constant of proportionality is precisely the eigenvalue A.
3.2
THE EXTREME EIGENVALUES AS AN UNCONSTRAINED OPTIMIZATION PROBLEM
Consider the unconstrained optimization problem: min f(z)
zE~P
with f ( z ) -
~(zTCz)/(zTz) .
(8)
It is straightforward to see that Vzf(z) = ( C z ( z T z ) - z(zTCz))/(zTz) 2 ,
(9)
from which it follows that the stationary points are given by
C(z/
zVq~z)= ( z / ~ ) ( z T C z ) / ( z T z )
.
This can only be satisfied if x = z/[[z[[ is an eigenvector of C with corresponding eigenvalue x T c x . Obviously, the minimum will correspond to the minimal eigenvalue of C (which can be negative, zero or positive). The maximum of f(z) will correspond to the maximum eigenvalue of C.
3.3
E X T R E M E EIGENVALUES AS A CONSTRAINED OPTIMIZATION PROBLEM
We can also formulate a constrained optimization problem as: min f(x) subject to x ElmP
xTx--1
where
f(x)----xTcx.
(10)
The Lagrangian for this constrained problem is L(x, A ) = x T c x -1- A ( 1 - x T z ) , where A E R is a scalar Lagrange multiplier. The necessary conditions for a stationary point now follow from VxL(x, A) = 0 and V~L(x, A) = 0, and will correspond exactly to the two equations in (7). Observe that these equations have p solutions (x, A) while we are only interested in the minimizing one.
The Riemannian SVD
3.4
67
THE CONTINUOUS-TIME P O W E R M E T H O D
Let us now apply the continuous-time minimization idea of (6) to the unconstrained minimization problem (8). We then find the 'steepest descent' nonlinear differential equation ~. -- - ( C z ( z T z) -- z ( z T C z ) ) / ( z T z )
2.
(11)
An important property of this flow is that it is isonormal, i.e. the norm of the state vector z is constant over time: []z(t)[[ = [Iz(O)[[, Yt > O. This is easy to see since d[[z[[2/dt = 2 zT2; = O, hence [Iz(t)[[ is constant over time. This means that the state of the nonlinear system (11) evolves on a sphere with radius given by [[z(0)[ I. We can assume without loss of generality that [Iz(0)[]- 1. Hence (11)evolves on the unit sphere. Replacing z by x with Ilxll 2 - x T x -- 1 we can rewrite (11) = -cz
+ z
xTCx xTx
.
(12)
We know about this flow that it is isonormal and that it will converge to a local minimum. Hence, (12) is a system of nonlinear differential equations that will converge to the eigenvector corresponding to the minimal eigenvalue of the matrix C. This flow can also be interpreted as a special case of Brockett's [5] double bracket flow (see [19] for this interpretation).
3.5
THE CONTINUOUS-TIME P O W E R M E T H O D F O R THE LARGEST SINGULAR VALUE
The preceeding observations can also be used to derive a system of nonlinear vector differential equations that converges to the largest singular value of a matrix A E I~TM, p _> q. It suffices to choose in (12) C=
() 0 Ar
A 0
and
z =
v
.
(13)
This matrix C will have q eigenvalues given by a~(A), q eigenvalues equal to -a~(A) and p - q eigenvalues equal to 0. Hence the smallest eigenvalue is -amax(A). The first q components of z will go to the corresponding left singular vector u while its last q components converge to the right singular vector v.
3.6
THE D I S C R E T E - T I M E P O W E R M E T H O D
The surprising fact about the nonlinear differential equation (12) is that it can be solved analytically. Its solution is 9 (t) - -
e-c~(o)/ll~-C~x(O)ll,
(14)
which can be verified by direct substitution. If we consider the analytic solution at integer times t = k = 0, 1 , 2 , . . . , we see that x(k + 1) = e-Cx(k)/lIe-Cx(k)[I,
B.L.R. De Moor
68
i o.8
...............i................ ~ ..........
i
..... : .... :. i............. i .............. i................ : .................................
0.6
"~):-... ........
0.4I\,
0 2~-\..x 9
0
i-:~ ...........
i ..............
:.................................
................ 9
!.x.~.:-~ .........
; ................
:................
:................
...... i.. ::-,..:. k ..... i ................ ~ : .....%,:. \ : '...'..'.~: ~ ~ ~
i ............... : :
~ ..........
..........
~...i ~. . . . . . . . . . . . ".'.'::~;;. ;f,T:.~ ,.,,,',~,=.7.:.: ~. . . . . .
: ................
i ................ : : .-_.~~.-~-.~..~ - _
i ............... : : ~
.
,,~_,..
-o.2
..............::..".......:..i................!...............::................i...............
-0.4
..............
-o.~
-o.8
::...............
i ................
:: ...............
i ................
:: ...............
...............
" ...............
i ................
...............
i ...............
i ................
i. ...............
i ................
i................
::................
i ................
i
................
-1
0
0.5
1
1.5
2
2.5
3
Figure 1: As a function of time, we show the convergence of the elements of the vectors u(t) and v(t) of the differential equation (12) with C given by (13), where A is a 4 x 3 'diagonal' matrix with on its diagonal the numbers (5, 3, 1). The initial vector z0 = (u(O) T v(O)T) T is random. This differential system converges to the eigenvector corresponding to the smallest eigenvalue of C in (13), which is -amax(A) = - 5 . Hence, this flow upon convergence delivers the 'largest' singular triplet of the matrix A. The picture was generated using the Matlab numerical integration function 'ode45'.
which shows that the continuous time equation (12) interpolates the discrete time power method 4 for the matrix e -c'. This implies that we now have the clue to understand the global convergence behavior of the flow (12). Obviously, the stationary points (points where = 0) are the eigenvectors of C, but there is only one stationary point that is stable (as could be shown by linearizing (12) around all the stationary points and calculating the eigenvalues of the linearized system). It will always convergence to the eigenvector corresponding to the smallest eigenvalue of C, except when the initial vector x(0) is orthogonal to the smallest eigenvector. These observations are not too surprising since the solution for the linear part in (12) (the first term of the right hand side) is x(t) = e x p ( - V t ) x ( O ) . The second term is a normalization term which at each time instant projects the solution of the linear part back to the unit sphere. We can rewrite equation (12) as xx T . 5~ = ( I - ~-~x ) C x . This clearly shows how ~ is obtained from the orthogonal projection of the vector C x (which is the gradient of the unconstrained object function 0 . h z T C x ) onto the hyperplane which is tangent to the unit sphere at z.
4...which is exactly the reason why (12) is called the continuous-time power method.
The Riemannian SVD 3.7
69
A D I F F E R E N T I A L EQUATION DRIVEN BY T H E RESIDUAL
Another interesting interpretation is the following: Define the residual vector r(t) - Cx x(xTCx/xTx). Then the differential equation (12) reads = -~(t).
Hence, the differential equation is driven by a residual error and when r(t) = 0, we also have ~ = 0 so that a zero residual results in a stationary point. Moreover, using a little known fact in numerical anaylsis [24, p.69] we can give a backward error interpretation as follows. Define the rank one matrix M(t) - r(t).x(t) T. Then we easily see t h a t
( C - M(t))x(t) = x(t))~(t) with A(t)
= (x(t))Tcx(t)/(x(t)Tx(t))
.
The interpretation is that, at any time t, the real number )~(t) is the exact eigenvalue of a modified matrix, namely C - M(t). The norm of the modification is given by IIM(t)l] = IIr(t)ll, which is the norm of the residual vector and from the convergence, we know that
3.8
DERIVATION AS A G R A D I E N T F L O W
Gradient flows on manifolds can be used to solve constrained optimization problems. The idea is to consider the feasible set (i.e. the set of vectors that satisfies the constraints) as a manifold, and then by picking out an appropriate Riemannian metric, determining a gradient flow. Let us illustrate this by deriving a gradient flow to solve the constrained minimization problem (10). The set of vectors that satisfies the constraints is the unit sphere, which is known to be a manifold:
M= { z e~Pl xrz= l }. The tangent space at x is given by the vectors z that belong to
T x M = { z e I~Pl z T x = O }. The directional derivative Dxg(z) is the amount by which the object function changes when moving in the direction z of a vector in the tangent space:
Dxg(z) -~ xTCz
9
Next, we can choose a Riemannian metric represented by a smooth matrix function W(x) which is positive definite for all x E M. It is well known (see e.g. [18] that, given the metric W(x), the gradient Vg can be uniquely determined from two conditions:
1 Compatibility: Dzg(z)= z T C x -
2 Tangency: Vg E T~M
r
x TVg
zTW(x)Vg. = O.
70
B.L.R. De Moor
The unique solution to these two equations applied to the constrained optimization problem (10) results in the gradient given by
Vg(~) =
W(~)-~(C~
-
9
xrW(z)-lCz). ~rW(~)-~
~,From this we obtain the gradient flow: =
-w(~)-~(c~
_ ~
~rw(~)-1C~ ~rw(~)-~ )"
(15)
It is easily seen that the stationary points of this system must be eigenvector-eigenvalues of the matrix C. Convergence is guaranteed because one can easily find a Lyapunov function (essentially the norm of the gradient) using the chain rule (see e.g. [18] for details). Observe that the norm IIx(t)ll is constant for all t. This can be seen from 1 dllzll 2 = xT ~ = _ z T w ( z ) _ X C x + z T w ( z ) _ X C x = 0 2 dt
Hence, if IIx(0)ll = 1, we have IIx(t)l I = 1,Vt > 0. If we choose the Euclidean metric, W ( x ) = I~, we obtain the continuous-time power method (12). An interesting open problem is how to chose the metric W ( x ) such that for instance the convergence speed could be increased and whether a choice for W ( x ) other than Ip leads to new iterative discrete-time algorithms.
3.9
NON-SYMMETRIC CONTINUOUS TIME POWER METHOD
So far we have only considered symmetric matrices. However most of the results still hold true mutatis mutandis if C is a non-symmetric matrix. For instance, the analytic solution to (12) is still given by (14) even if C is nonsymmetric. For the convergence proof, we can find a Lyapunov function using the left and right eigenvectors of the non-symmetric matrix C. Let X E ]tip• be the matrix of right eigenvectors of C while Y is the matrix of left eigenvectors, normalized such that AX-XA, yTx--Ip , ATy = YA , X Y T - Ip . For simplicity we assume that all the eigenvalues of C are real (although this is not really a restriction). The vector y~n E ~P is the left eigenvector corresponding to the smallest eigenvalue. It can be shown that the scalar function L(z) = (xTymin) 2 / ( x T y y T x )
is a Lyapunov function for the differential equations (12) because L > 0, Yr. Note that x - x ( y T x ) , which says that the vector y T x contains the components of x with respect to the basis generated by the column vectors in X (which are the right eigenvectors). The denominator is just the norm squared of this vector of components. The numerator is the component of x along the last eigenvector (last column of X), which corresponds to the smallest eigenvalue. Since g > 0, this component grows larger and larger relative to all other ones, which proves convergence to the 'smallest' eigenvector.
The Riemannian SVD 3.10
71
N O N - S Y M M E T R I C C O N T I N U O U S - T I M E P O W E R M E T H O D F O R T H E SMALLEST SINGULAR VALUE
We can exploit these insights to calculate the smallest singular triplet of a matrix A 6 i~p• as follows. Consider the 'asymmetric' continuous-time power method:
i)
=
-
AT
-~Iq
v
v
~
'
(16)
with = ~(ur~-
v rv) I (~r~ + ~rv) "
Here c~ is a user-defined real scalar. When it is chosen such t h a t a > amin(A), u and v will converge to the left resp. right singular vector corresponding to the smallest singular value of A. This can be understood by realizing that the 2q (we assume t h a t p _> q) eigenvalues of the matrix
AT
-a Iq
are Ai = 4-~/a2 _ a2(A) and that there are p - q eigenvalues equal to c~. Hence, if c~ > hi, the corresponding eigenvalues Ai are real, else they are pure imaginary. So if a > amin, the smallest eigenvalue is A _ _ ~/c~2 _ O'mi 2 n to which (16) will converge. This is illustrated in Figure 2.
3.11
CHRISTIAAN'S F L O W F O R THE SMALLEST SINGULAR VALUE
The problem with the asymmetric continuous time power method (16) is t h a t we have to know a scalar ~ that is an upper bound to the smallest singular value. Here we propose a new continuous-time algorithm for which we have strong indications t h a t it always converges to the 'smallest' singular value, but for which there is no formal proof of convergence yet. Let A 6 n~T M (p >_ q) and r 6 I~ be a given (user-defined) scalar satisfying 0 < r < 1. Consider the following system of differential equations, which we call Christiaan's flow 5:
(it)(aIPi) = -
CA T
-ar
(17)
It is our experience that (u(t), a(t), v(t)) converges to the smallest singular triplet of A. The convergence behavior can be influenced by r (for instance, when r --* 1, there are many oscillations (bad for numerical integration) but convergence is quite fast in time; when r --* 0, there are no oscillations but convergence is slow in time). There are many similarities with the continuous-time algorithms discussed so far. When we rewrite these equations as
it = iJ -
Av-ua, -r
va) ,
(18)
5... after one of our PhD students Christiaan Moons who one day just tried it out and to his surpise found out that it always converges.
72
B.L.R. De Moor
1
!
!
i
1
!
0.8
0.8 "i"
0.6
0.6
0.4
0.4
o~o
~ -0.2 -
-0.4
-0.4
:
:
:
:
!
!
!
!
!
i 2
2.5
3
!
i 3.5
i
-0.2
-0.6
!
L
..... 1
-0.6-
:
-0.8 -0.8 -1 ................................
0
0.5
.
":.................................................................
, 1
1.5
2
2.5
3
.11 0
.
i
!
o 0.5
, 1
.
I 1.5
Figure 2: Convergence as a function of time of the components of u(t) and v(t) of the asymmetric continuous-time power method (16) for the same matrix A and the same random initial vectors u(0) and v(0) as in the previous Figure. The left picture shows the behavior for a = 2, which is larger than the smallest singular value and therefore converges. The right picture shows the dynamic behavior for a - 0.5. Now, all the eigenvalues of the system matrix are pure imaginary. Therefore, there is no convergence but instead, there is an oscillatory regime.
we easily see from (1) that both equations are 'driven' by the (scaled) residual error (see
Section 3.7). Another intriguing connection is seen by rewriting (17) as
(0) (0 01), -I
0
0
A
u
u
which compares very well to (15), except that the metric here is indefinite! As for a formal convergence proof, there are the following facts: It is readily verified that the singular triplets of A are the stationary points of the flow. When we linearize the system around these stationary points, it is readily verified that they are all unstable, except for the stationary point corresponding to the 'smallest' singular triplet of A. However, at this moment of writing, we do not have a formal proof of convergence (for instance a Lyapunov function that proves stability) ~.
4
ALGORITHMS
FOR THE RIEMANNIAN
SVD
In this section, we'll try to use the ideas presented in the previous section to come up with continuous time algorithms for the Riemannian SVD (4) and hence for structured/weighted total least squares problems. The ideas presented here may be premature but on the other 6...and as we have done before, we offer a chique dinner in the most exquis restaurant of Leuven if somebody solves our problem.
The Riemannian SVD
!
I
73
!
0.8
o.6
~. .....
'
~
:
0.4
~: 9 ...... ~i . . . . . . . . . . .
02
:
: .........
~
i .........
i..........
!.........
:
:
:
:
:
i !.........
i ~.........
i i .........
i
i
i
7
8
9
i i ~ ......... i ..........
:
i
9
-0,
2
3
4
i
5
i .........
i .........
. i
6
i
10
Figure 3: Example of the convergence behavior of Christiaan's flow (17) for the same matrix A and initial vectors u(0) and v(0) as in the first Figure. The full lines are the components of u, the dashed ones those of v. Both vectors converge asymptotically to the left and right singular vectors of A corresponding to the smallest singular value.
hand, they offer intriguing and challenging perspectives, which is the reason why we mention them. It should be noted that we have derived a heuristic iterative discrete-time algorithm for the Riemannian SVD in [12], the basic inspiration of which is the classical power method. While this algorithm works well in most cases, there is no formal proof of convergence nor is there any guarantee that it will convergence to (an at least local) minimum. Therefore, we try to go via these continuous-time algorithms to find an approach which would be guaranteed to converge to a local minimum. So far we have not succeeded in doing so, but we hope that the elements presented in this section provide enough new material to convince the reader of the usefulness of the presented approach.
4.1
AN OPTIMIZATION PROBLEM
Let us try to solve (see the equations (4):
uTAv vTATu a = min -- min . u,v uTDvu u,v vTDuv A nice property which can often that for every vector u and v, we The fact that D= is independent continous-time algorithm (5) and da 1 it -du = uTDv"-'--"~( - A v
iJ =
da 1 -- ~ ( - A T u dv vT Du v
be used in manipulating formulas in this framework, is always have uTDvu = vTDuv (see [12] for a proof). of v and Dv is independent of u allows us to apply the derive the system of differential equations uTAv + Dvu u r D v u ) ,
-~" Duv
vTATu ) 9 vT Du v
74
B.L.R. De Moor
While this system will converge to a local minimum, it is not this local minimum that solves our structured/weighted TLS problem. Indeed, to see why, it suffices to consider the special case where Dv = Ip and Du - Iq in which case we recover the system (12)-(13) which converges to -amax(A) and the corresponding singular vectors (and NOT amin(A)!).
4.2
CHRISTIAAN'S FLOW FOR THE RIEMANNIAN SVD
Although we are completely driving on heuristics here, it turns out that the following generalization of Christiaan's flow (17) works very well to find the minimal singular triplet of the Riemannian SVD (4).
with r E II~ a user-defined number satisfying 0 < r < 1. The only difference between (17) and (19) is the introduction of the positive definite metric matrices D,~ and D,~. We have no idea whatsoever about possible convergence properties (that are formally provable), except for the fact that in a stationary point, the necessary conditions (4) are satisfied. Yet, our numerical experience is that this system of nonlinear differential equations converges to a local minimum of the object function. As an example, let A 6 n~6x5 be a given matrix (we took a random matrix), that will be approximated in Frobeniusnorm by a rank deficient matrix B, by not modifying all of its elements but only those elements that are 'flagged' by a '1' in the following matrix
V=
oo
0 1 000 1 0 1 0 1 O 0 1 1 0
1 1 1 0 1
; Let W =
oo
oo
oo
oo
i
i
oo
1
I
c~
I
I
oo oo
0 1 1 1 1 0 0 1 1 1
1
1
oo
i
I
i
oo
1
i
I
oo
oo
i i i
be the elementwise inverse of V. The matrix W contains the weights as in (3) and the element oo means that we impose an infinite weight on the modification of the corresponding element in A (which implies that it will not be modified and that the corresponding element in B will be equal to that in A). It can be shown [11] that the metric matrices D~ and Dv for this flow are diagonal matrices given by
D~=diag(V
v~
),
OH = d i a g ( Y T
u~
)"
Here vi and ui denote the i-th component of v, resp. u and u 2 and v2 are their squares. As an initial vector for Christiaan's flow (19) we took [u(0) T v(0) T] = leT/v/6 eT/v/5] where ek E irk is a vector with all ones as its components. The resulting behavior as a function of
75
The Riemannian SVD
x~xxxx ~
4).
0
2
4
6
8
10
1
14
16
1
20
Figure 4: Vectors u(t) and v(t) as a function of time for Christiaan's flow (19), which solves the problem of least squares approximation of a given matrix A by a rank deficient one, while not all of its elements can be modified as specified by the elements in the matrix V which belong to {0, 1}. The vector differential equation converges to the same solution as the one provided by the discrete-time algorithm of [11], which on its turn is inspired by the discrete-time power method. time is shown in Figure 4. 5
CONCLUSIONS
In this paper, we have discussed how weighted and/or structured total least squares problems lead to a nonlinear generalization of the SVD, which we have called the Riemannian SVD. Next, we have derived several interesting interpretations of the continuous-time power method (geometric, unconstrained and constrained optimization, gradient flow, residual driven differential equation). We have also discussed Christiaan's flow, which is a set of nonlinear differential equations that seems to converge to the 'smallest' singular triplet, both of the SVD and the Riemannian SVD. As of now, there is however no formal proof of convergence. There are interesting analogies between the SVD and the Riemannian SVD, which are enumerated in the following table: There are still several open problems, which all together form a complete research program: Proofs of convergence of some of the algorithms we have introduced here, discretization (e.g. based on geodesic approximations (see e.g. [26])), decreasing the rank with more than 1 while preserving the structure, etc . . . . Also in the applications there are several interesting sideways that could be followed. For instance, in [8] we have shown how our approach leads to an H2-model reduction algorithm, while in [12] [13] we show how the structured total least squares problem with a 'double' Hankel matrix of noisy inputs and outputs is the
B.L.R. De Moor
76
Problem Applications
Decomposition Approximation B Algorithms
'Euclidean' TLS (Eckart-Young) Linear relations, noisy data
SVD of A 'Identity' metric Rank 1 modification Power method Christiaan's flow
'Riemannian' Weighted/Structured TLS Hankel matrices Model reduction, Dynamic TLS Relative error TLS Maximum likelihood Riemannian SVD of A Positive definite metric Du, Dv Multilinear matrix function Power method Christiaan's flow
same as the L2-optimal so-called errors-in-variables problem from system identification. Acknowledgements The author is a Senior Research Associate of the Belgian National Fund for Scientific Research. This work was supported by grants from the Federal Ministry of Scientific Policy (DWTC, with grants IUAP/PAI-17 (Modelling and Control of Dynamical Systems), IUAP/PAI-50 (Automation in Design and Production)), the Flemish NFWO-project no. G.0292.95 (Matrix algorithms for adaptive signal processing systems, identification and control) and the SIMONET (System Identification and Modelling Network) supported by the Human Capital and Mobility Program of the European Commision. I would sincerely like to thank Christiaan Moons, Jeroen Dehaene and Johan Suykens for many lively discussions as well as Thomas De Moor for letting me use his balloon in my attempts to illustrate the equivalence principle of general relativity in Gene Golub's car. References
[1] Adcock R.J. Note on the method of least squares. The Analyst, Vol IV, no.6, Nov. 1877, pp.183-184. [2] Adcock R.J. A problem in least squares. The Analyst, March 1878, Vol V, no.2, pp.5354. [3] Baldi P., Hornik K. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, Vol.2, pp.53-58, 1989. [4] Bourlard H., Kamp Y. Auto-association by multilayer perceptrons and singular value decomposition. Biol. Cybern., 59, pp.291-294, 1988. [5] Brockett R.W. Dynamical systems that sort lists and solve linear programming problems. Linear Algebra and its Applications, 146, pp.79-91, 1991.
The Riemannian SVD
77
[6] M.T. Chu, On the continuous realization of iterative processes, SIAM Review, 30, 3, pp.375-387, September 1988. [7] De Moor B., Golub G.H. The restricted singular value decomposition: properties and applications. Siam Journal on Matrix Analysis and Applications, Vol.12, no.3, July 1991, pp.401-425. [8] De Moor B., Van Overschee P., Schelfhout G. H2-model reduction for SISO systems. Proc. of the 12th World Congress International Federation of Automatic Control, Sydney, Australia, July 18-23 1993, Vol. II pp.227-230.7 [9] De Moor B., David J. Total linear least squares and the algebraic Riccati equation. System ~z Control Letters, Volume 18, 5, pp. 329-337, May 1992 s. [10] De Moor B. Structured total least squares and L2 approximation problems. Special issue of Linear Algebra and its Applications, on Numerical Linear Algebra Methods in Control, Signals and Systems (eds: Van Dooren, Ammar, Nichols, Mehrmann), Volume 188-189, July 1993, pp.163-207. [11] De Moor B. Total least squares for aJi~nely structured matrices and the noisy realization problem . IEEE Transactions on Signal Processing, Vol.42, no.11, November 1994. [12] De Moor B. Dynamic Total Linear Least Squares. SYSID '94, Proc. of the 10th IFAC Symposium of System Identification, 4-6 July 1994, Copenhagen, Denmark. [13] De Moor B., Roorda B. L2-optimal linear system identification: Dynamic total least squares for SISO systems. ESAT-SISTA TR 1994-53, Department of Electrical Engineering, Katholieke Universiteit Leuven, Belgium. Accepted for publication in the Proc. of 33rd IEEE CDC, Florida, December 1994. [14] Eckart C., Young G. The approximation of one matrix by another of lower rank. Psychometrika, 1, pp.211-218, 1936. [15] Fiacco A.V., McCormick G.P. Nonlinear programming; Sequential unconstrained minimization techniques. SIAM Classics in Applied Mathematics, 1990. [16] Golub G.H., Van Loan C. Matrix Computations. Johns Hopkins University Press, Baltimore 1989 (2nd edition). [17] Golub G.H., Van Loan C.F. An analysis of the total least squares problem. Siam J. of Numer. Anal., Vol. 17, no.6, December 1980. [18] Helmke U., Moore J.B. Optimization and dynamical systems. CCES, Springer Verlag, London, 1994. [19] Helmke U. Isospectral matrix flows for numerical analysis. This volume. 7Reprinted in the Automatic Control World Congress 1993 5-Volume Set, Volume 1: Theory. 8Forms also the basic material of a Chapter in "Peter Lancaster, Leiba Rodman. Algebraic Riccati Equations. Oxford University Press, 1994.
78
B.L.R. De Moor
[20] Householder A.S., Young G. Matrix approximation and latent roots. Americ. Math. Monthly, 45, pp.165-171, 1938. [21] Kung S.Y., Diamantaras K.I., Taur J.S. Neural networks for extracting pure/constrained/oriented principal components. Proc. of the 2nd International Workshop on SVD and Signal Processing, June 1990.
[22] Ljung L. Analysis of recursive stochastic algorithms. IEEE Transactions on Automatic Control, Vol.AC-22, no.4., August 1977, pp.551-575. [23] Oja E. A simplified neuron model as a principal component analyzer. J. Math. Biology, 15, pp.267-273, 1982. [24] Parlett B. The symmetric eigenvalue problem. Prentice Hall, Englewood Cliffs, NJ, 1980. [25] Pearson K. On lines and planes of closest fit to systems of points in space. Phil. Mag., 2, 6-th series, pp.559-572. [26] S.T. Smith, Geometric Optimization Methods for Adaptive Filtering, PhD Thesis, Harvard University, May 1993 Cambridge, Massachusetts, USA. [27] Van Huffel S., Vandewalle J. The Total Least Squares Problem: Computational Aspects and Analysis. Frontiers in Applied Mathematics 9, SIAM, Philadelphia, 300 pp., 1991. [28] Young G. Matrix approximation and subspace fitting. Psychometrika, vol.2, no.l, March 1937, pp.21-25. [29] Wilkinson J. The algebraic eigenvalue problem. New York, Oxford University Press, 1965.
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
CONSISTENT
SIGNAL
RECONSTRUCTION
79
AND
CONVEX
CODING
N.T. THAO
Department of Electrical and Electronic Engineering Hong Kong University of Science and Technology Clear Water Bay, Kowloon Hong Kong M. VETTERLI
Department of Electrical Engineering and Computer Science University of California, Berkeley Berkeley, CA 9~720 U.S.A. ABSTRACT. The field of signal processing has known tremendous progress with the development of digital signal processing. The first foundation of digital signal processing is due to Shannon's sampling theorem which shows that any bandlimited analog signal can be reduced to a discrete-time signal. However, digital signals assume a second digitization operation in amplitude. While this operation, called quantization, is as deterministic as time sampling, it appears from the literature that no strong theory supports its analysis. By tradition, quantization is only approximately modeled as an additive source of uniformly distributed and independent white noise. We propose a theoretical framework which genuinely treats quantization as a deterministic process, is based on Hilbert space analysis and overcomes some of the limitations of Fourier analysis. While, by tradition, a digital signal is considered as the representation of an approximate signal (the quantized signal), we show that it is in fact the representation of a deterministic convex set of analog signals in a Hilbert space. We call the elements of the set the analog estimates consistent with the digital signal. This view leads to a new framework of signal processing which is non-linear and based on convex projections in Hilbert spaces. This approach has already proved effective in the field of high resolution A/D conversion (oversampling, Sigma-Delta modulation), by showing that the traditional approach only extracts partial information from the digital signal (3dB of SNR are "missed" for every octave of oversampling).
80
N. Z Thao and M. Vertterli
The more general motivation of this paper is to show that any discretization operation, including A/D conversion but also signal compression, amounts to encoding sets of signals, that is, associating digital signals with sets of analog signals. With this view and the framework presented in this paper, directions of research for the design of new types of high resolution A/D converters and new signal compression schemes can be proposed. KEYWORDS. A/D conversion, digital representation, oversampling, quantization, SigmaDelta modulation, consistent estimates, convex projections, set theoretic estimation, coding.
1
INTRODUCTION
Although numbers are usually thought of real continuous numbers in theory, signal processing is nowadays mostly performed digitally. Traditionally, digital signals are considered as the encoded version of an approximated analog signal. In many cases, the approximation error is considered negligible and digital numbers are thought of quasi-continuous. However, this assumption starts to be critical in more and more emerging fields such as high resolution data conversion (oversampled A/D conversion) and signal compression. In this paper, we ask the basic question of the exact correspondence which exists between analog signals and there encoded digital signals. This starts by reviewing the existing foundations of analog-to-digital (A/D) conversion. It is known that A/D conversion consists of two discretization operations, that is, one in time and one in amplitude. While a strong theory (Shannon's sampling theorem) describes the operation of time discretization, we will see in Section 2 that the analysis of the amplitude discretization, or quantization, is only approximate and statistical. This approach turns out to be insufficient in fields such as oversampled A/D conversion. To find out what the exact analog information contained in a digital signal is, it is necessary to have a more precise description of the whole A/D conversion chain. In Section 3 we define a theoretical framework which permits a more precise description of A/D conversion. To do this, we go back to the basic description of an analog signal as an element of a Hilbert space (or Euclidean space in finite dimension), and wedescribe any signal transformation geometrically in this space, instead of using the traditional Fourier analysis which is limited to time-invariant and linear transformations. In this framework, we show that the precise meaning of a digital signal is the representation of a deterministic convex set of analog signals. The elements of the set are called the analog estimates consistent with the digital signal. Because of the convexity property, we show that, given a digital signal, a consistent estimate must be picked as a necessary condition for optimal reconstruction. With this new interpretation, digital signal processing implies a new framework of (nonlinear) signal processing based on convex projections in Hilbert spaces and presented in Section 4. In fact, the case of A/D conversion which is thoroughly considered in this paper is only a particular case of digitization system. The more general motivation of this paper is to
81
Consistent Signal Reconstruction and Convex Coding
show that the basic function of any digitization system, including high resolution data acquisition systems (Section 5) but also signal compression systems, is to associate digital representations with sets of analog signals, or, to encode sets of analog signals. Not only does this view give a genuine description of their functions, but it indicates new directions of research for the design of A/D converters and signal compression systems. 2
CLASSICAL PRESENTATION
OF A / D C O N V E R S I O N
The term of "digital signal processing" often designates what should be actually be called "discrete-time signal processing" [1]. Thanks to Shannon's sampling theorem, it is known that any bandlimited analog signal can be reduced to a discrete-time signal, provided that the sampling rate is larger than or equal to the Nyquist rate, that is, twice the maximum frequency of the input signal. Mathematically speaking, there exists a invertible mapping between bandlimited continuous-time signals x(t) and sequences (Zk)keZ such that 1 xk = x ( k T s ) , provided that T, = ]'8 >_ 2fro, where fm is the maximum frequency of x(t). Therefore, any processing of the continuous-time signal x(t) can be performed in the discrete-time domain. This constitutes the foundation of discrete-time processing. However, digital signal processing assumes that a second discretization in amplitude, or quantization, is performed on the samples, as indicated by Figure 1. The digital output bandlimited
x(t)
real
-Isamplerl
-I
real
xk _! quantizer l
I
time discretization
ampfitude discretization
c~
integer
_1 coder I
-I
linear scaling
dk
I
Figure 1: Analog-to-digital (A/D) conversion sample dk of an A/D converter is an integer representation of ck which is a quantized version of the continuous-amplitude sample xk. The transformation from Xk to Ck is known to introduce an error ek " - C k - Xk, called the quantization error. While the time discretization process is supported by a solid theory, it appears from the literature that there only exists an approximate analysis of the quantization process. Either the quantization error is neglected and the quantization operation is considered as "transparent", or, when some close analysis is needed, it is commonly modeled as a uniformly distributed and independent white noise [2, 1] 9 This leads to the classical mean squared quantization error of ~where q is the 12 quantization step size. However, this model, which is in fact only accurate under certain conditions [3, 4], does not take into account the deterministic nature of the quantization operation. This is particularly critical when dealing with oversampled A/D conversion. Oversampiing is commonly used in modern data conversion systems to increase the resolution of conversion while using coarse quantization. While the independent white noise model validity conditions become less and less valid with oversampling [4], it is still used as basic model to recover a high resolution estimate of the source signal from the oversampled and
82
N.T. Thao and M. Vertterli
coarsely quantized signal. With this model, a frequency analysis of the quantized signal shows in the frequency domain that only a portion of the quantization error energy lies in the input baseband region. Thus, the total energy of quantization noise can be reduced by the oversampling factor R = 2f-~, by using a linear lowpass filter at cut off frequency f,n (see Figure 2). In practice, the lowpass filtering is performed digitally on the encoded lowpass filter cut off =fm
=Isampler
quantizerI (~)
I
v
c(o)) X (co)~ in-band E ((o),~
Wj
-rr,
2~/R
error rr,
(b) Figure 2: Oversampled A/D conversion. (a) Principle: the sampling is performed at the frequency fs > 2fro. (b) Power spectrum of the quantized signal (Ck)keZ with the white quantization noise model. version (dk)keZ of represented.
(Ck)keZ. On
Figure 2(a), only the equivalent discrete-time operation is
Although this noise reduction can be observed in practice under certain conditions, this does not tell us how much exactly we know about the analog source signal from the oversampled and quantized signal. We can already give a certain number of hints which tell us that a linear and statistical approach of the quantization process is not sufficient to give a full analysis of the signal content process. First, it is not clear whether the in-band noise which cannot be canceled by the lowpass filter is definitely irreversible. Because quantization is a deterministic process, there does exist some correlation between the input signal and the quantization error signal, even after filtering. Second, with the linear filtering approach, it appears that the quantization mean squared error (MSE) has a non-homogeneous dependence with the time resolution and the amplitude resolution. Indeed, the MSE is divided by 4 when the amplitude resolution is multiplied by 2 (that is, q is divided by 2), whereas it is divided by 2 only when the time resolution is multiplied by 2 (that is, R is multiplied by 2). This is a little disappointing when thinking of A/D conversion as the two dimensional discretization of a continuous graph. In fact, an example can already be given which shows by some straightforward mechanisms that the in-band noise is indeed not irreversible. Figure 3 shows a numerical example
Consistent Signal Reconstruction and Convex Coding
2q
! ,
.
.
! .
:
:
quantization threshold
' ,.
~ ~ ,
~
analog input signal : x(t)
3q/2 ................. :-- * -* '--:- :- *- ". z~* -~ i
'
:
"
'
.
i
X
',
i
'
.
i
i
83
'
,
9 quantized signal
ck
,, estimate c k obtained from linearfiltering of
ck
~, projection of the estimate
ckonC
.... i .... i .... ) .... i 1 :2 3 4
5 6
'7 '8 9 fO 1.1 1~2 13 1'4 1.5 1'6 17
I remaining error
Figure 3: Example of oversampling and quantization of a bandlimited signal with reconstruction by linear filtering. of a bandlimited signal x(t), shown by a solid line, which is oversampled by 4 and quantized, giving a sequence of values (ck)keZ represented by black dots. The classical discrete-time reconstruction (~k)keZ obtained by lowpass filtering (ck)keZ is shown by the sequence of crosses. Some error represented by grey shades can be observed between the signal reconstruction ((3k)keZ and the samples of the input signal. We know that this error forms a signal located in the frequency domain in the baseband region. However, some anomalies can be observed in the time domain. At instants 11 and 12, it can be seen that the values of (dk)kez are larger than q, while the given values of the quantized signal (ck)keZ tell us that the input signal's samples necessarily belong to the interval [0, q]. Not only is the sequence (ck)keZ not consistent with the knowledge we actually have about the source input signal, but this knowledge also gives us a deterministic way to improve the reconstruction estimate (ck)keZ. Indeed, although we don't know where exactly the samples of the input signal are located within the interval [0, q] at instants 11 and 12, we know that projecting the two respective samples of (ck)keZ on the level q leads to a necessary reduction of the error (see Figure 3). This shows that the in-band error is not irreversible. These hints show that a new framework of analysis is necessary.
3 3.1
NEW ANALYSIS OF A/D CONVERSION SIGNAL
ANALYSIS
FRAMEWORK
The goal is to define a framework where quantization can be analyzed in a deterministic way with the given definition of an error measure. Bandlimited analog signals are usually formalized as elements of the space /~2(R) of square summable functions, where Fourier decomposition is applicable. Thanks to Shannon's sampling theorem, the analog signals x(t) bandlimited by a maximum frequency fm can be studied as elements of the space s of square summable sequences, thanks to the
84
N. T Thao and M. Vertterli
invertible mapping
z(t)
, (xk)k~Z E L2(Z) where xk = x(kT,),
under the condition that fa = ~1 _ 2fro. Errors between the bandlimited analog signals are measured using the canonical norm of s and can also be evaluated in the discrete-time space/:2(Z) using its own canonical norm, thanks to the relation:
• f~ I~(t)l~t = ~ I~kl~ T8
R
kEZ
Unfortunately, this framework cannot be used to study quantization because the quantized version (ck)keZ of an element (Zk)keZ of s is not necessarily an element of s (or, is not necessarily square summable). For example, using the quantization configuration of Figure 3, although a sequence (Xk)kE Z may be decaying towards 0 when k goes to infinity, its quantized version (ck)keZ never goes below ~ in absolute value. On the other hand, while the MSE type of error measure can be applied for the analysis of quantized signals, it cannot be applied to the elements of s since it would systematically lead to the value 0. Therefore, we propose to confine ourselves to another space of bandlimited signals which can be entirely defined on a finite time window [0, T0]. Precisely, we assume that the sinusoidal components of the Fourier series expansion of x(t) on [0, T0] are zero as soon as the corresponding frequencies are larger than fro. This is equivalent to saying that the T0-periodized version of z(t) defined on [0, To] is bandlimited by the maximum frequency f,n. Under this assumption, we have a finite time version of Shannon's sampling theorem. It can be easily shown that, under the condition ~N-1 _ 2fro equivalent to the Nyquist condition, there is an invertible mapping between z(t) and its discrete-time version X = (Xk)l
MSE(x(t),x'(t))
= ~
=o
This error can be in fact evaluated in the discrete-time domain using the mean squared sum
MSE(X,X')
= IIX' - Xll 2 = -~ 1 ~N I~'~k-1 thanks to the relation, easy to show [5, 6]"
__1 To
TO lz,(t) _ z(t)[2dt= _~ -o
~1 ~,
[Ztk _ zk[2.
Now, the quantized version of the discrete-time signal X = (Zk)l
Consistent Signal Reconstruction and Convex Coding 3.2
85
QUANTIZATION ANALYSIS
The quantization operation can be deterministically defined as a mapping Q from R, N t o it. N. But unlike the sampling operation, this is a many-to-one mapping. While, under the Nyquist condition, a discrete-time signal is characteristic of a unique analog bandlimited signal, a quantized signal C is characteristic of a whole set of possible continuous-amplitude and discrete-time signals. Mathematically, this set is simply the inverse image of C through the mapping Q, usually denoted by Q-I[C] C R g. If a sequence X is only known by its quantized version C, the exact knowledge about X available from C is that X belongs to the set of signals Q - I [ C ] . We call the elements of the set C = Q - I [ C ] the estimates of l~ N consistent with the quantized signal C. Figures 4(a) and (b) show the form of the set C of consistent estimates in the cases N = 1 x3
Xt C
i X (x 1...... N
(a)
C
l
j j2
J coder
c 1...... Ni I
=
d I ..... dN)
(b)
Figure 4: Quantization as a many-to-one mapping of R N. (a) Case N = 1. (b) Case N > 1. and N > 1 respectively. In the case N = 1, C is equal to the whole quantization interval which contains the given quantized value c. Using the classical configuration of uniform quantization, the value c appears to be the particular consistent estimate located at the mid-point of the interval C. In the traditional view point, the digital output d of a quantizer is a binary encoded version of the quantized value c. In our approach, we consider that d is a digital representation of the complete set C. In the case N > 1, C is obviously the N dimensional cross-product of real intervals and therefore forms geometrically a hypercube of R N parallel to the canonical axes. As a generalization of the case N = 1, the quantized signal C appears to be the particular consistent estimate located at the geometric center of C. As in the case N = 1, we consider that the digital output D is the encoded version of the whole set C, not of the signal C. In the case of oversampled A/D conversion, the quantization operation is performed, not on any element of R N, but on the sampled version of bandlimited signals only. Indeed, it is easy to see from the assumption of Section 3.1 that the bandlimited signals have a finite Fourier series expansion containing not more than 2f,.,To + 1 components. As a
86
N. I". Thao and M. Vertterli
consequence, they belong to a space of finite dimension equal to W = [2freT0 + 1], where [y] designates the smallest integer greater than or equal to y. As a second consequence, their sampled version also belongs to a W dimensional space, since the sampling operation applied on bandlimited signals is a linear and invertible mapping. Because of the Nyquist rate condition ~ > 2fro, note that we necessarily have W < N. Therefore, the sampled versions of the bandlimited signals belong to a W dimensional subspace S of l:t N. By abuse of language, we call 5` the space of bandlimited discrete-time signals. It can be shown that the dimensional ratio N coincides approximately with the oversampling ratio R. To recapitulate, in the oversampling context, the inputs to the quantizer are elements of the subspace S C R g. Once X E 5' is quantized into C, the complete knowledge which is available about X is that X belongs to the set S N 17 where 17 =bf Q-I[C]. We will say that S g117 is the set of estimates consistent with C. This set is geometrically represented in Figure 5.
..........1
Figure 5: Geometric representation of oversampled A/D conversion.
3.3
NECESSITY FOR CONSISTENT RECONSTRUCTION
In the previous section, it was shown that when an input signal is quantized, the exact information which remains available to us is that it belongs to the set of consistent estimates. However, nothing tells us until now that we must pick a consistent estimate if we want to estimate the input signal from its quantized version. We show in this section that this is in fact the case in a certain sense. It can be easily shown from the previous section that sets of consistent estimates are convex. We recall that .,4 is a convex set if and only if for any couple of elements X, Y E `4, the segment [X, Y] is entirely included in ,4. Because the considered norm I1" II in R g is a euclidean norm, the convexity property will appear to play an important role thanks to the following lemmas: L e m m a 3.1 [7] Let X be an element of I~ N and A C I~ N be a convex set. There exists a unique element X ' of the closure -~ of A such that for all Y E `4, I l Z ' - Y[[ < [Iz - Yli. The transformation from X to X ~ is then a mapping of R N called the convez projection on
Consistent Signal Reconstruction and Convex Coding
87
A~ L e m m a 3.2 [8] If X I is the convex projection of X on a convex set A C R N and X q~ "A, then for all Xo e ,4, [ I X ' - Xo[[ < [ I X - Zo[[. These lemmas are illustrated by Figure 6. They lead to the following proposition.
Figure 6: Geometric representation of convex projection. P r o p o s i t i o n 3.3 Let Xo E I~ N be the input of a quantizer, Co the output of the quantizer and X an element of R y which does not belong to the set A of estimates consistent with Co. Then, although we don't know where the input Xo is located within the set A, the distance between X and Xo can be deterministically reduced 1 by a convex projection of X on .A. We recall that .A is equal to Q-I[C0] without oversampling, and SNQ-I[C0] with oversampiing. In any case, the operation of convex projection on .A is uniquely determined by the knowledge of Co. As a conclusion, when reconstructing a discrete-time signal from its quantized version Co, any non-consistent estimate is by necessity non-optimal and can be deterministically improved using the knowledge of Co.
3.4
DETERMINISTIC ANALYSIS OF OVERSAMPLED A/D CONVERSION
We have already seen from Figure 3 an example where the reconstruction estimate C = (ck)l
IV. Z Thao and M. Vertterli
88
The second important question is now to know what the performance yielded by consistent estimates is in terms of MSE. The following result was recently shown in [5, 9]: T h e o r e m 3.4 Let x(t) be a bandlimited and To-periodic signal which has a time density of quantization threshold crossings larger than or equal to the Nyquist rate. At the oversampling ratio R = N let X E R N be the sampled version of x(t), C the quantized version of X and X ~ E ~t 1~ any estimate consistent with C. Then, there exists a constant a > 0 which only depends on x(t) and the definition of the quantizer, such that
MSE(X,X')
<_ - ~ .
Qualitatively speaking, this theorem implies that under a certain condition on the input's quantization threshold crossings, signals chosen in the set of consistent estimates yield an MSE which asymptotically decreases with R in (.9(R-2), instead of (.9(R -1) as it is the case with the classical linear reconstruction. This represents a faster decrease of MSE over the classical method by 3dB per octave of R. With this new result, the symmetry of the MSE dependence with the amplitude and the time resolutions is recovered.
4
CONVEX
PROJECTION
BASED SIGNAL PROCESSING
We have seen that a digital signal is not the representation of a single estimate, but of a complete set of estimates called the consistent estimates. Although the convex set corresponding to a given digital signal is deterministically known, the problem of using this knowledge to retrieve a consistent estimate or at least partially improve a non-consistent estimate, is not trivial. Although the space of analysis R g is of finite dimension, N may be "infinitely" large compared to the finite time window of operation of the working processor. For this reason, the existing algorithms derived from the field of non-linear and linear programming [10] to retrieve an estimate satisfying convex constraints, may be not feasible. More feasible algorithms may be derived from the field of set theoretic estimation [11] in euclidean spaces or, for the case of infinite dimension, in Hilbert spaces. The basic idea is that a convex set .A can be often decomposed as intersection of a certain number p of convex sets Ci, i = 1, ...,p with simple structure and on which the convex projections are implement able. For example, in oversampled A/D conversion, the convex set of consistent estimates is the intersection S I"1C, where 5" and g are two convex sets of relatively simple structure. While it is difficult to find directly an estimate in 8 M g, the convex projections on S and g respectively are easily defined. The hypercube g of R y c a n be itself seen as the intersection of 2N convex sets which are half-spaces of R N. The projection on each of these convex sets is trivial and only implies local operations in time. Figure 7 shows in general how polygonal sets can be decomposed as intersection of half-spaces. Assuming that the set of consistent estimates has the following decomposition ,4 P 1Ci CIi= and that the convex projection on each set r is implement able , we already have a way to partially improve any non-consistent estimate. Indeed, if X ~ .A, there exists by necessity i E {1, ..., p} such that X ~ gi. The projection of X on gi will reduce the distance of X with any element of Ci, and therefore any element of A, since ,4 C gi. This is exactly what was performed in the example of Figure 3. After noticing that (~ does not belong to Q-I[C], it - -
Consistent Signal Reconstruction and Convex Coding
89
Figure 7: Decomposition of a polygon into half-spaces and representation of the parallel projection algorithm. can be easily shown that the time domain operation indicated in Figure 3 is the projection of C on the convex set C = Q - I [ C ] and leads to a necessary improvement since C includes S N C. This process can be in general reiterated as long as the current estimate X does not belong to .A (see Figure 8). Xn
Figure 8: Geometric representation of the alternating projection algorithm. It was in fact proved in [8] that by applying convex projections onto C1,C2, ...,Cp alternately and periodically, one converges to an element of the intersection 2. We formalize this property as follows: T h e o r e m 4.1 Let CI,...,Cp be p convex sets in a Hilbert space 7-l, P1,.-.,Pp be the convex projections on Ca, ...,Cp respectively, and (Xn)neN be a sequence in 7-l such that X,,+I = Pn mod n+l [Xn], for n e N. p
Then the sequence (Xn)nEN converges to an element of A = Ni=IC, in the sense of the Hilbert norm of 7-[.
This is often called the algorithm of alternating projections or the POCS algorithm (Projection Onto Convex Sets). This algorithm became popular in signal processing with the work by Youla [12]. 2Rigorously, one converges to an element of the intersection of Cl, C2, ...,C"pp,inthe case where Cl, e2, ..., Cp have not been specified as closed sets.
N.T. Thao and M. Vertterli
90
There exists a more general version of this algorithm including relaxation coefficients (a,~),~N and based on the following operation: X,~+I = a,~. P,~ mod p+l[ X,~] + (1 - a,~). X,~. Note that the choice a,~ = 1 brings us back to the simple case of alternating projections. For the general case an ~ 1, it is shown that this single operation reduces the distance 3 of the current estimate X,~ with any element of ,4. It is also shown that the infinite iteration converges to an element of A if there exists e > 0 such that Vn E N, a,~ E [e, 2 - e]. In practice, the speed of convergence can be often accelerated by empirical adjustments of the coefficients an in ]1, 2[. One drawback of the alternating projection algorithm is that it does not permit parallel processing. A new algorithm involving parallel projections was recently introduced by Combettes [13] and based on the following operation
Xn+l -- ~
'Wi,n" Pi[Zn], where Wi,n >_ 0 and ~
ieln
wi,,~ = 1,
ieI,,,
and where In is a subset of indices of {1,...,p}. Qualitatively speaking, at each step n, a certain number of sets among C1, ..., Cv is selected (the set of the indices of the selected sets is called In) and the convex projections of Xn on these selected sets are applied. This forms a set of points {Pi[X,~] / i E In) and Xn+l is chosen in the convex envelop of this set. These operations are illustrated in Figure 7. The distance of the estimate X,~ with any element of ,4 is shown to be reduced by this transformation [14] and the infinite iteration is proved to converge to an element of ,4 under certain conditions on the sequence (In)heN [13]. The admissible choices of (I~)~eN include two particular cases: (i) I,~ = {1, ...,p}: This is the case where all convex projections are performed in parallel at each step. (ii) I,~ = {n mod p + 1}: This fails back to the case of alternating projections. A version with relaxation coefficients is also introduced in [13] as: X,,,+I = an" ~ wi,,~Vi[X,-,] + (1 - an)" X~,. iE I,., The convergence to an element of ,4 is shown to be guaranteed if 3e > 0, gn E N, an E [e, 2Lr, - e] where _
E ~ . ~,~
- It- , 7o 5
APPLICATION
IIP~[X~]-
X~ll ~
itl TO HIGH RESOLUTION
DATA CONVERSION
Although the deterministic approach was introduced on the simple version of oversampled A / D conversion in Section 3, it is also applicable to modern techniques of high resolution data conversion such as oversampled ~ A [15, 16]. The conversion scheme is similar to that of Figure 2, but the quantizer is replaced by a more sophisticated circuit called a ~ A 3The reduction of distance is strict when Xn ~t S rood v+l and ~n E [0, 2].
Consistent Signal Reconstruction and Convex Coding
91
modulator, including an integrator, a quantizer and a feedback loop (see Figure 9). This
ck
I
ZA I
sampler
,
(a)
I
-I
I
TM
V% (b) Figure 9: 2 A modulation. (a) Overall principle. (b) Detail of the ~A modulator. type of data conversion allows the use of very coarse quantization (down to one bit), and thus simple circuitry, while reproducing a high resolution estimate after lowpass filtering. Although the conditions of validity of the white quantization noise model are not really applicable here [4], ~ A modulation is still classically analyzed using this model [16]. In this context, it is shown that a ~A behaves like an additive source of independent noise whose spectrum is "shaped" as shown in Figure 10. Then, it is easy to show that the ~,. i ~
in-band
error
C(o))
f
/-1
I
2rdR Figure 10: Power spectrum of the output of a ~A modulator with the assumption of white quantization noise. portion of noise energy contained in the baseband of the quantized signal decreases with the oversampling ratio R in R -3, which represents a decrease of 9dB per octave of R. In spite of the limited validity of the assumed model, this result is observed in practice. More sophisticated architectures of ~A modulation exist, which include a higher number of integrators [17]. In general, for an n th order ~ A modulator, the noise energy remaining in the baseband of the quantized signal depends on R in R -(2n+1). Now, the same kind of question as in Section 3 can be raised here. What do we know exactly about a bandlimited signal after it is oversampled and processed through a ~A modulator ? Like a single quantizer, a F~A modulator can also be studied as a many-to-one mapping of R g. The set C of estimates consistent with the output of a ~A modulator can be obtained
92
N.T. Thao and M. Vertterli
by inversion of this mapping. It is shown in [5, 6] that the set is no longer a hypercube, but a parallelepiped (the edges are no longer perpendicular). However, this is still a convex set, and it is shown that the quantized signal C = (r is still located at its geometric center. As in Section 3, the set of consistent estimates is S N C. This is geometrically represented in Figure 11..Mthough the distance between X and C, due to the in-band C
X
s
~
.............~....
l
I-
-of-band
...................
Figure 11: Geometric represetation of oversampled lEA modulation. error remaining in the quantized signal C, decreases with R faster than in the case of simple quantization, it appears that C is still not necessarily a consistent estimate. In fact, numerical experiments performed on bandlimited and T0-periodic signals [5, 6] show that the MSE yielded by consistent estimates decreases in average w i t h / / i n O(//-4) instead of (.0(//-3). In general, for an n th order EA modulator~ it was shown that the average MSE of consistent estimates behaves in O(//-(2~+2)) instead of 0(R-(2~+1)), implying, as in the case of simple quantization, a faster decrease of MSE by 3dB per octave of R, regardless of the order n. With a deterministic approach, these experiments show that the output of a lEA modulator contains more information about the input signal than that recovered with the classical approach of A/D conversion. 6
CONCLUSION
AND RELATED RESEARCH
The full meaning of a digital signal is obtained by a deterministic analysis of the digitization process as a many-to-one mapping. Thus, a digital signal is the representation of, not a single estimate, but a whole set of analog signals, called the set of consistent estimates. This set plays two roles: (i) it gives the exact knowledge of the possible locations of the original analog signal, (ii) it is the set where a signal should be picked when estimating the original signal from its digital version. The second item is due to the convexity of the set, as observed on classical quantization schemes. With this approach, not only is a more precise analysis of the A/D conversion process given, but, in the context of oversampling and ~A modulation, it also leads to the conclusion that a digital output signal contains more analog information about the input
93
Consistent Signal Reconstruction and Convex Coding
signal than that traditionally recovered by the classical analysis of A/D conversion. Namely, the MSE of consistent estimates decreases with the oversampling ratio R fastest than that of the classical linear reconstruction estimate by 3dB per octave. This new approach of digital signals implies a new framework of signal processing based on convex projections in Hilbert spaces, derived from the field of set theoretic estimation. This past research leads to the new idea that the intrinsic function of an A/D converter is to split the space of analog input signals into convex sets, and assign a digital representation to each of them. For this reason, we say that an A / D converter is a convex coder. The intrinsic performance of a convex coder can be evaluated by its ability to split the input space into small sets with respect to the considered error measure. Recent research has been done to measure the intrinsic performance of an oversampled A/D converter or a EA modulator [18, 19, 20]. Figures 12 and 13 show that the evolution of the intrinsic performance of a ]CA modulator with the oversampling ratio R or with the order n of the modulator can be graphically observed by the set partition it defines in the input space. The intrinsic performance of the encoder can be measured by the average MSE of optimal SINGLE-LOOP, R=6
SINGLE-LOOP, R=4
/
~ 3 ~
\\
\'
\
' 0.6 0.4
0.4
iX
0.2
,
'~
0.2
/
/
~
8
o
~
o~ o -0.2 o -0.4
-0.2 -0.4
-0.6
..
-1
-0.5
-0.6
0 sin component
(a)
0.5
'-,,
,/
1
-1
-0.5
0
0.5
sin component
(b)
Figure 12: Partition defined by a first order EA modulator in the 2 dimensional space of To- periodic sinusoids with arbitrary amplitude and phase: (a) Case of oversampling ratio R = 4 . (b) C a s e R = 6 . reconstruction which consists of picking for each cell of the input space partition its centroid. It was shown in [19, 20] that optimal reconstruction yields the same MSE behavior in R as consistent reconstruction. This input space view can be a new direction for the design of high resolution data converters, traditionally designed using the noise shaping approach. The convex coding approach can also be applied to signal compression [21]. Although this field implies a digital to digital transformation, the input signal is usually considered as quasi-continuous in amplitude. In this context, it is shown in [21] that classical signal compression schemes such as block DCT coding can be analyzed as convex coding schemes. An example of new signal compression scheme is proposed by a direct and active control of the encoded sets.
94
N. T Thao and M. Vertterli SINGLE-LOOP, R=4
.
.
.
DOUBLE-LOOP, R=4
.
.
.
i
I
oo ~o-0.
(~ -0.5
-,/ -2
"
~
-1.5
-~
-0'.5 0 0.5 sin component
(~)
1
1.5
2
-2
-1.5
-1
-0.5
0
0.5
1
.5
sin component
(b)
Figure 13: Partition defined by a ~A modulator in the 2 dimensional space of T0-periodic sinusoids with arbitrary amplitude and phase at oversampling ratio R = 4. (a) Single-loop case. (b) Double-loop case. References [1] A.V.Oppenheim and R.W.Shafer, Discrete-Time Signal Processing. 1989.
Prentice Hall,
[2] N.S.Jayant and P.Noll, Digital Coding of Waveforms. Prentice-Hall, 1984. [3] W.R.Bennett, "Spectra of quantized signals," Bell System Technical Journal, vol. 27, pp. 446-472, July 1948. [4] R.M.Gray, "Quantization noise spectra," IEEE Trans. Information Theory, vol. IT-36, pp. 1220-1244, Nov. 1990. [5] T.T.Nguyen, "Deterministic analysis of oversampled A/D conversion and ~A modulation, and decoding improvements using consistent estimates," PhD. dissertation, Dept. of Elect. Eng., Columbia Univ., Feb. 1993. [6] N.T.Thax) and M.Vetterli, "Deterministic analysis of oversampled A/D conversion and decoding improvement based on consistent estimates," IEEE Trans. on Signal Proc., vol. 42, pp. 519-531, Mar. 1994. [7] D.G.Luenberger, Optimization by vector space methods. Wiley, 1969. [8] L.M.Bregman, "The method of successive projection for finding a common point of convex sets," Soviet Mathematics - Doklady, vol. 6, no.3, pp. 688-692, May 1965. [9] N.T.Thao and M.Vetterli, "Reduction of the MSE in R-times oversampled A/D conversion from O(1/R) to O(1/R2)," IEEE Trans. on Signal Proc., vol. 42, pp. 200-203, Jan. 1994.
2
Consistent Signal Reconstruction and Convex Coding
95
[10] D.G.Luenberger, Linear and nonlinear programming. Wiley, 1984. [11] P.L.Combettes, "The foundations of set theoretic estimation," Proc. IEEE, vol. 81, no. 2, pp. 1175-1186, Feb. 1993. [12] D.C.Youla and H.Webb, "Image restoration by the method of convex projections: part 1 - theory," IEEE Trans. Medical Imaging, 1(2), pp. 81-94, Oct. 1982. [13] P.L.Combettes and H.Puh, "A fast parallel projection algorithm for set theoretic image recovery," Proc. IEEE Int. Conf. ASSP, vol. V, pp. 473-476, Apr. 1994. [14] P.L.Combettes and H.Puh , Personal communication, May 1994. [15] J.C.Candy, "A use of limit cycle oscillations to obtain robust analog-to-digital converters," IEEE Trans. Commun., vol. COM-22, pp. 298-305, Mar. 1974. [16] J.C.Candy and G.C.Temes, eds., Oversampling delta-sigma data converters. Theory, design and simulation. IEEE Press, 1992. [17] S.K.Tewksbury and R.W.Hallock, "Oversampled, linear predictive and noise shaping coders of order N > 1," IEEE Trans. Circuits and Systems, vol. CAS-25, pp. 436-447, July 1978. [18] S.Hein, K.Ibraham, and A.Zakhor, "New properties of sigma-delta modulators with dc inputs," IEEE Trans. Commun., vol. COM-40, pp. 1375-1387, Aug. 1992. [19] N.T.Thao and M.Vetterli, "Lower bound on the mean squared error in multi-loop 2EZX modulation with periodic bandlimited signals," Proc. 27th Asilomar Conf. on Signals, Systems and Computers, Pacific Grove, CA, Nov. 1993. [20] N.T.Thao and M.Vetterli, "Lower bound on the mean squared error in oversampled quantization of periodic signals," IEEE Trans. Information Theory. Submitted in June 1993, revised in Sept. 1994. [21] K.Asai, N.T.Thao, and M.Vetterli, "A study of convex coders with an application to image coding," Proc. IEEE Int. Conf. ASSP, vol. V, pp. 581-584, Apr. 1994.
This Page Intentionally Left Blank
PART
2
ALGORITHMS AND THEORETICAL
CONCEPTS
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved. :t.
THE
ORTHOGONAL
99
QD-ALGORITHM
U. VON MATT Institute for Advanced Computer Studies University of Maryland College Park, MD 207~2 U.S.A. na. von matt @na- ne t. ornl. gov
ABSTRACT. We present the orthogonal qd-algorithm to compute the singular value decomposition of a bidiagonal matrix. This algorithm represents a modification of Rutishauser's qd-algorithm, and it is capable of determining all the singular values to high relative precision. We also introduce a generalization of the Givens transformation, which has applications besides the orthogonal qd-algorithm. KEYWORDS. Generalized Givens transformation, Newton's method, Laguerre's method, orthogonal qd-algorithm, singular value decomposition.
1
INTRODUCTION
In 1990 J. Demmel and W. Kahan showed that all the singular values of a bidiagonal matrix are determined to high relative accuracy by the entries of the matrix [2]. They modified the SVD-algorithm by Golub and Reinsch [6] as implemented in the LINPACK library [3] by introducing special zero shifts to compute the small singular values to high relative accuracy. Their code is now a part of the LAPACK library [1]. A different approach is proposed by K. V. Fernando and B. N. Parlett [4]. They use a modification of Rutishauser's qd-algorithm (cf. [5, 9, 10, 11, 12, 13]) to compute the singular values of a bidiagonal matrix. However, their algorithm can no longer be expressed as a sequence of orthogonal transformations applied to the bidiagonal matrix. Consequently, it is not possible to compute the singular vectors simultaneously with the singular values. Our approach is based on Rutishauser's qd-algorithm, too. But the orthogonal qdalgorithm can be expressed as a sequence of Givens rotations applied from the left and the right to the bidiagonal matrix. This enables us to compute the singular vectors along with the singular values.
1O0
2
U. von Matt
GENERALIZED
GIVENS
TRANSFORMATION
Usually the Givens transformation --8
C
with c 2 + s 2 = 1 is determined such that z2
0
"
The matrix G is therefore used to selectively annihilate elements in a vector or a matrix. But it is also possible to introduce another value a different from zero: ~
9
X2
(7
The v~ue of~ is given b y e ' - +~/~ + ~ -
~2.
Obviously, this transformation is only possible if lal ___ ~/x~ +
z~.
It is easily verified that
1 8
Xl2 -{- X 2
X2
--Xl
0.
We will also need a variant of the generalized Givens transformation, its so-called differential form 9 In this case we determine the matrix G such that
G[xl] = rl X2
r2
'
where ~1 : = ~ i g n ( ~ l ) ~ / ~ ~ =
2 -
sign(~2)J~2 +
~, 0"2.
Of course, this is only possible if ]0"[ _< [Xl[. It is easily verified that the values of c and s are given by ~
3
- ~ + ~
~2
ORTHOGONAL
-~1
r2
"
QUOTIENT-DIFFERENCE
In what follows, we will refer to the following matrices L and U: C~l i
:-
]~1
STEPS
n-by-n lower bidiagonal 71
'"
U :~ /~n--I
" 9 O/n
~n-1
7n
and upper bidiagonal
The Orthogonal QD-algorithm
101
|
81
81
O/2
8~
0/3
82
0
O/2
|
82
0/3
"..
~3 ~
|
P ~r
|
ff
~
81
*)'1
~1
0
3'2
82
0
%
9
~
.9
~
9
9149 o 9
P P
P
O"
O" 9 9149J
~149149
Figure 1: Orthogonal Left lu-Step with Shift s. D e f i n i t i o n 3.1 Let Q be an orthogonal (2n)-by-(2n) matrix. We call the transformation
(2
aI
=
~/a 2 + s2I
an orthogonal left lu-step with shift s. This transformation exists if and only if the condition Isl _< a ~ n ( L ) holds. The singular values of L are diminished by the amount of the shift s, i.e. =
The matrix Q can be constructed by the sequence of Givens rotations depicted in Figure 1. The quantity p is an abbreviation for V'a2+ s 2. Note that every other step consists of a generalized Givens transformation. In order to avoid numerical problems for small shifts s it is necessary to use the differential form of the generalized Givens transformation. D e f i n i t i o n 3.2 Let Q denote an orthogonal (2n)-by-(2n) matrix. We call the transformation
Q
:
v/a2 -~- S2I
an orthogonal left ul-step with shift s. This transformation represents the dual version of the orthogonal left lu-step. It can be carried out if and only if Isl < a ~ n ( U ) . The singular vMues of U are reduced by the amount of the shift s, i.e. :
U. von Matt
102
|174
al
'~
~1
0
"~
9176 ~
0 0~2
0
#2
~3
9
~
~
". ~
,~
Figure 2: 0rthogonal Right ul-Step. An orthogonal left ul-step, too, can be executed by a sequence of Givens rotations. The mechanism is the same as in Figure 1, with the only exception that the transformations are applied from bottom to top. D e f i n i t i o n 3.3 Let Q denote an orthogonal n-by-n matrix. We call the transformation I
QT
Q=
aI
an orthogonal right ul-step. This transformation can always be executed, and it leaves the singular values of U unchanged. If 7n = 0 we have an = fin-1 = 0 after the transformation. This property is useful to deflate a matrix U with Vn = 0. The sequence of Givens rotations necessary for an orthogonal right ul-step is depicted as Figure 2. D e f i n i t i o n 3.4 Let Q denote an orthogonal n-by-n matrix. We call the transformation
an orthogonal right lu-step. This transformation represents the dual version of the orthogonal right ul-step. It can also be executed unconditionally, and it preserves the singular values of the matrix L. If the first row of L is zero, i.e. if a l = 0, we get a matrix U with 3'1 = 61 = 0. We can therefore use this transformation to deflate a matrix L with a l = 0. An orthogonal right lu-step can also be carried out by a sequence of Givens rotations. The same technique is used as shown in Figure 2, except that the transformations are applied from bottom to top. We will refer to the four transformations that have been introduced in this section by the generic term of orthogonal qd-steps. 4
ORTHOGONAL
QUOTIENT-DIFFERENCE
ALGORITHM
Let B denote a given n-by-n lower bidiagonal matrix whose singular value decomposition is desired. The orthogonal qd-algorithm transforms the (2n)-by-n matrix
A:=
B] 0
The Orthogonal QD-algorithm
103
by a sequence of orthogonal qd-steps into the (2n)-by-n matrix 0] E ' where IE denotes an n-by-n diagonal matrix with the singular values of B. In matrix terms this process can be described by the equation
where P denotes an orthogonal (2n)-by-(2n) matrix, and Q denotes an orthogonal n-byn matrix.
5
DEFLATION
In this section we analyse the conditions that allow us to set to zero an entry of the lower bidiagonal matrix L. This reduced matrix will be called L. We compare the singular values of the original matrix A :=
aI
with those of the modified matrix A.:=
aI
"
We require that all the singular values of A and A agree within the precision of the computer arithmetic. The analysis in [14, pp. 143-149] shows that we can set the diagonal entry ak to zero if the condition
applies numerically. It should be noted that/30 =/3,~ = 0. The numerical criterion for an off-diagonal element ~k reads
~2 + I~kl(l~kl + min(l~kl, I~k+~l))= ~2. Similar criterions can be derived for an upper bidiagonal matrix U.
6
SINGULAR VECTORS
If the orthogonal transformations are accumulated the orthogonal qd-algorithm delivers the decomposition (1). In most cases, however, we would like to get the singular value decomposition B = Vr~V T
(2)
of the n-by-n lower bidiagonal matrix B, where U and V denote orthogonal n-by-n matrices. It is straightforward to identify the matrix Q from (1) with the matrix V in (2). The
104
U. von M a t t
matrix U can then be obtained from the QR-decomposition (without pivoting) of B V : BV
= UR.
Care must be taken that the diagonal elements of R remain nonnegative. We get an alternative way of computing the left singular vectors by partitioning the orthogonal matrix P in (1) into n - b y - n submatrices: P=
[ Pll /~
P12 ] /~ "
If B is nonsingular then we must have Pll = P22 = 0, and P12 is an orthogonal matrix consisting of the left singular vectors of B. Even in the presence of rounding errors we can force Pll and P22 to go to zero by an appropriate convergence criterion.
7
SHIFTS
The performance of the orthogonal qd-algorithm mainly depends on the choice of the shift s in each step. In this section we describe two different shift strategies based on Newton's and Laguerre's method to compute the zeros of a polynomial. The zeros of the characteristic polynomial p(A) := det(U wU - AI) are the eigenvalues of u T u , which are equal to the squares of the singular values of U. Thus if we use Newton's or Laguerre's method to approximate the smallest zero of p(A) we can also get an approximation for the smallest singular value of U. Newton's method can be described by the iteration
?(~k)
)~k+l = )~k - p,()~k) ,
and in the case of Laguerre's method we have
Ak+l
p(~k)
F(~k)
~
.
1 + i(n - 1)(nP'C~)~-P( ~)p''(~)p,(~)~ - 1)
We choose A0 = 0 as our initial value. It is well-known (cf. [7, 8] and [15, pp. 443-445])that both methods will then converge monotonically to the smallest zero of p(A). In particular we have
0 - ~0 _~ )~1 _~ Amin(UTU) 9 Laguerre's method enjoys cubic convergence (cf. [8, pp. 353-362] and [15, pp. 443-445]). On the other hand Newton's method will converge only quadratically [15, p. 441]. We will use v / ~ as the shift in an orthogonal qd-step. Thus we define Newton's shift and Laguerre's shift as follows:
3Newton
./
p(O)
V p,(o)'
The OrthogonalQD-algorithm
8Laguerre
"-'-i -p,(o) p(o)l[
105
n
p'(0)~ - 1) 1 Both Newton's and Laguerre's shift can be computed by recurrence formulas. Special attention is necessary to avoid overflow and underflow problems. More details may be found in [14, pp. 149-151]. 8
NUMERICAL RESULTS
In this section we compare the performance of the orthogonal qd-algorithm with the subroutine sbdsqr from the LAPACK library [1] (see also [14, pp. 152-155]). This subroutine represents an implementation of the work of Demmel and Kahan [2]. Both algorithms compute all the singular values to high relative accuracy. We also observe that Laguerre's shift is quite expensive to evaluate. If n is the size of the matrix, the calculation of Laguerre's shift needs on the order of O(n) operations. On the other hand, Demmel and Kahan use Wilkinson's shift which only needs 0(1) operations. Wilkinson's shift also requires fewer iterations per singular value to converge. Unfortunately it can only be applied in conjunction with the QR-algorithm since it does not compute a lower bound for the smallest singular value. Laguerre's shift, on the other hand, always gives us a lower bound on the smallest singular value of the bidiagonal matrix, which is essential for the orthogonal qd-algorithm. If only the k smallest singular values are desired the orthogonal qd-algorithm can compute them in O(kn) operations. If the singular vectors are also needed the operation count increases to O(kn2). This represents a significant savings compared to O(n 3) which is the computational complexity of a complete singular value decomposition. 9
CONCLUSIONS
We have presented the orthogonal qd-algorithm to compute the singular values of a bidiagonal matrix to high relative accuracy. Our approach differs from the qd-algorithm by Fernando and Parlett as we do not transpose the bidiagonal matrix in each step. This enables us to accumulate the orthogonal transformations and thus obtain the singular vectors. We use Newton's and Laguerre's method to compute the shifts for the orthogonal qdsteps. Although Laguerre's shift does not quite attain the efficiency of Wilkinson's shift it has the advantage that it always computes a lower bound on the smallest singular value of the bidiagonal matrix. We have also presented two generalizations of the Givens transformation. They come in very handy in the context of the orthogonal qd-algorithm, but they should also be useful in other applications. Acknowledgments Our thanks go to W. Gander, G. H. Golub, and J. Waldvogel for their support. The author also thanks G. W. Stewart for his helpful comments.
106
U. von Matt
References [1] E. Anderson, Z. Bal, C. Bischof, J. Demmel, J. Dongarra, J. Du Croz, A. Greenbaum, S. Hammarling, A. McKenney, S. Ostrouchov and D. Sorensen. LAPA CK Users' Guide. SIAM Publications, Philadelphia, 1992. [2] J. Demmel and W. Kahan. Accurate Singular Values of Bidiagonal Matrices. SIAM J. Sci. Stat. Comput. 11, pp. 873-912, 1990. [3] J. J. Dongarra, C. B. Moler, J. R. Bunch and G. W. Stewart. LINPA CK Users' Guide. SIAM Publications, Philadelphia, 1979. [4] K. V. Fernando and B. N. Parlett. Accurate singular values and differential qd algorithms. Numer. Math. 67, pp. 191-229, 1994. [5] W. Gander, L. Molinari and H. Svecovs Numerische Prozeduren aus Nachlass und Lehre yon Prof. Heinz Rutishauser. Internat. Ser. Numer. Math., Vol. 33, Birkhguser, Basel, 1977. [6] G. H. Golub and C. Reinsch. Singular value decomposition and least squares solutions. Numer. Math. 14, pp. 403-420, 1970. [7] E. Hansen and M. Patrick. A Family of Root Finding Methods. Numer. Math. 27, pp. 257-269, 1977. [8] A. M. Ostrowski. Solution of Equations in Euclidean and Banach Spaces. Third Edition of Solution of Equations and Systems of Equations, Academic Press, New York, 1973. [9] H. Rutishauser. Der Quotienten-Differenzen-Algorithmus. ZAMP 5, pp. 233-251, 1954. [10] H. Rutishauser. Der Quotienten-Differenzen-Algorithmus. Mitteilungen aus dem Institut fiir angewandte Mathematik Nr. 7, Birkh~iuser, Basel, 1957. [11] H. Rutishauser. Uber eine kubisch konvergente Variante der LR-Transformation. ZAMM 40, pp. 49-54, 1960. [12] H. Rutishauser. Les propridtds numdriques de l'algorithme quotient-diffdrence. Rapport EUR 4083f, Communaut~ Europ~enne de l'Energie Atomique - EURATOM, Luxembourg, 1968. [13] H. Rutishauser. Lectures on Numerical Mathematics. Birkh~iuser, Boston, 1990. [14] U. von Matt. Large Constrained Quadratic Problems. Verlag der Fachvereine, Ziirich, 1993. [15] J. H. Wilkinson. The Algebraic Eigenvalue Problem. Clarendon Press, Oxford, 1965.
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
ACCURATE SINGULAR JACOBI METHOD
VALUE
107
COMPUTATION
WITH
THE
Z. DRMA(~ Fachbereich Mathematik, Fern UniversitSt Hagen, Postfach 9~0, D-5808~ Hagen, Germany. zlatko, drmac @fern uni-hagen, de
ABSTRACT. The main interest in this work is implementation of the Jacobi method as reliable software for computing the singular values of a general real matrix. The modifications of formulas for computing the rotation angle and modified Jacobi transformation are given in order to ensure reliable computation of singular values in the full range of floating point numbers. If the gradual underflow is used, then the effect of underflow on the accuracy properties of the modified Jacobi SVD method is not worse than the uncertainty caused by roundoff errors. KEYWORDS. Jacobi method, relative accuracy, singular values.
1
INTRODUCTION
The following is a very short summary of recent results in the field of singular value computation:
The singular values of general matrices are not perfectly conditioned ( in the sense of Weyl's theorem ), if we consider relative perturbation estimates. In other words, if a > 0 is singular value of A E R mxn, then small perturbation 5A = A o E, 5Aij = AijEij,
IEijl <_ 9 << 1
can cause ~a with ]3al/a > e. The estimate based on Weyl's theorem implies for a r 0 ]~a[ _~ [[~A[]2 _ V/-~g2(A)e, ~2(A)= [IA[12[[A~[]2. o"
o"
This can be improved for full column rank A by A + ~A = (Q + ~ A R - 1 ) R ==~
I~1 (7
<_ [15AR-1112 <_ v / ~ 2 ( A s ) e ,
(1)
Z. Drrna~'
108
where A = QR is the QR factorization of A and As = Adiag (llAe~llz)-~. The estimate (1) is valid/or IlaAe~llz _< ellAe~ll2, 1 _< i _< n, 9 < 1/(v/'ff~2(As)). This is the essence of the perturbation theory of Barlow and Demmel [1] and Demmel and Veselid [~] which introduced ~2(As) as a condition number for relative perturbations. Since ~2(As) < v'ff~z(A) and since it is possible that ~;2(As) << ~;2(A), the bounds yrom [I], [4] are better than those derived from the Weyl's theorem. The use of orthogonal transformations in the singular and eigenvalue computation is not per se the guarantee for the relative accuracy of the computed output. For example, the singular values of A can be insensitive to small pointwise perturbations Aij ~ Aij h- r and small pointwise perturbations of A ~ = AV, V some orthogonal matrix, can cause loss of information on the smallest singular values of A. The nonorthogonal transformations, if properly used, are useful tool in the matrix computation, even in solving the singular value and the symmetric eigenvalue problems. The new development introduced the following approach" Describe a set of matrices for which the singular values can be computed with relative accuracy and find a method capable of achieving that accuracy. For full column rank A with moderate ~2(As) the algorithms of choice are the implicit Jacobi SVD of Demmel and Veselid [4] and the implicit Cholesky SVD of Fernando and Parlett [9]. In this work we discuss reliable floating point implementation of the Jacobi SVD method. The presented material is short summary of [7], [5], [8], [6].
2
JACOBI SVD
Let A E t t m•
have full column rank. The Jacobi SVD algorithm generates the sequence
A (~ = A, A (k+l) - A(k)u (k), k - 0 , 1 , 2 , . . . where U (k) is Jacobi rotation in the (p(k),q(k)) plane. Demmel and Veselid have shown that the relative errors in the computed singular values depend on maxk>o ~2(A (k)) (A (k) A(k)diag (llA(k)eill2) -1) and not on ~2(A). If the elements of A already have uncertainties, then even exact computation will not improve on Jacobi. This claim of Demmel and Veselid is based on well behaviour of maxk>0 ~2(A(k))/~2(As). However, the usual floating point implementation of the algorithm is not reliable software in the sense of [3]. We want to recommend modification of the Jacobi SVD method which satisfies the criterion of reliability [3].
2.1
TO ROTATE OR NOT TO ROTATE
Let ti = A[ep, eq]
= [ap, aq] e
R rex2 and
r = [ (a~,~) ,l~,,~ (a~,~)] ,,~,,~ '
j=
cos r - sin r
sin r cos r
, j ~ r J = diag
(2)
Singular Value Computation with the Jacobi Method
109
where
cot 2r -- Ilaqll2 - Ilapll2 tan r = sign cot 2 r 2(av, aq) I cot 2r + X/1 + cot2 2r The standard approach in the implicit Jacobi-like methods reads 1. Compute the Gram matrix of pivot cohmns; 2. Decide whether to rotate or not; 3. If to rotate then 3.1. Compute J using (3) ( or some other symmetric 2 x 2 eigensolver ); 3.2. Perform ft. := A J using some library routine.
(3)
First observation is that, because of squaring ( 11%112 2, Ilaqll~ ), the computation is possible for (roughly) a(A) C (v/'5, x / ~ , where a(A) denotes the set of singular values of A and u, co axe the underflow and the overflow treshold, respectively. Next, the dot product (av, %) can underflow and that additionally reduces the range of safe computation to the interval (roughly) ( x / ' ~ , V/w), where e denotes the roundoff unit. This restriction can be removed by rewritting the formula for cot 2r to
Ila.ll~ Ila.l12 cot 2r = [lapl]2 - Ilaqll2 2
(ap, aq) Ilapll2[laqll2
where for Ilapll211aqll2 ~ ( v / 6 , a , ) , []ap[]21]aql]2
(ap, a~)lll%ll2111aqll2 is
[]ap[]2' l]aq[]2 ) --
._
[]ap]]2
c o m p u t e d as
Ilaq]]2 "
Let t o l be the tresh for performing the Jacobi rotation and let
l(ap, aq)l > toi. llapll211aqll2
(4)
Usually t o l E {me, v/'me}. It is reasonably to assume 1;ol << 1. The importancy of the criterion (4) for the relative accuracy is explained in [1], [4]. The bound (4)implies that cot 2r overflows ( [cot 2r > co ) only if
max
11%112'11ap112 > 2cotol.
Assume for simplicity [lapll2 > I]aqll2. We shah analyse the situation in the case fl(cot 2r e {Inf,--Inf}, i.e. 11%112 >> Ilaq[12 9 Note, in that case t a n r is computed as zero and the transformation is identity, thus useless and the whole process may not converge. This means that for ~2(A) > co (roughly) the floating point implementation depends on tc2(A). We have to simulate the acting of the Jacobi rotation with I tanr < v. The insight into the geometric structure of the transformation in such situation is possible for
Ilaqll~ Ila~l12 >
Z. Drmat'
110 In that case
Ilaqli2 11~ll2 Ilapll2- Ilaqll2 ~
2r
cot
Ila~ll2 -Ilaqll~
,
Icot2r
~
1
and
Ilapll~
tanr ~ ~ c o t 2 r ~
hold with relative error e. Thus, I tan Note that !
aq
aq "F ap tan r ~, aq
=
r -< v~ and the cosine will be computed as one.
ap, aq)
performs the Gram-Schmidt orthogonalization of aq against ap and that apI = ap - a q t a n r changes ap by
I tanr
~
I(ap, aq)l Ilaqll~ < Ilaq I~ Ila~l1211a~l12 Ila~l12 - I1% ~llapll2 _< ellapll2.
In the case of overflow due to II~pll2/llaqll2 > ~ t o l , the relative change of ap is negligible. On the other side, the change of aq is bounded by I cos | aq)lllaqll2 and it cannot be neglected. In this situation we perform the Jacobi rotation in the following way: !
ap
=
(7)
ap,
Thus, although there is no machine representation for cot 2r and tan r because of overflow and underflow, respectively, performing Jacobi rotation is possible thanks to the connection between the Jacobi rotation and the Gram-Schmidt orthogonalization. It is important to note that large cot 2r reflects relative relations between geometric terms and cannot be removed by scaling cA, c 6 R. Note, the transformation (7, 8) in matrix notation reads
[a~ a~q] = lap aq] ,
,
E 1 0 ][ Ilapll2
o
1
II~qll2
1
o
apaq 1
Ilapll211aqll2 1
Ilapll2
o
o
Ilaqll2
1
"
Three nonorthogonal transformations are used to simulate the orthogonal one.
2.2
HOW TO ROTATE
Our next modification is motivated by the -qmaf option I of the X L - F O K T K A N 2.0 compiler on IBM RISC System/6000. The -qmaf option means using the mar ( multiply-and-add ) 1This option is default.
Singular Value Computation with the Jacobi Method
111
function which computes a. x + y with one rounding, i.e. for floating point reals a, x, y
Let us shortly discuss application of the Jacobi rotation in the floating point arithmetic providing the mar function. The Jacobi SVD algorithm does not explicitly introduce zeros into the matrix, thus inaccurate tangents affect the convergence ( and, therefore, the overall perturbation analysis ) but not the one-step floating point error analysis. Let t be the computed value of the tangent of the Jacobi angle. The computed value of the cosine is estimated as follows: = f z ( t~= )1,1+
=
x/ll~ l +2 e 2l+ei +t
(1 + ~=), I~,1 _ 2~ + ~=
I~=1, I~=1 < ~.
The transformation is implemented in the factorized form .q
t
C
0
1
0
c
1
[x,y]
-t
] = [(~ - ty)~, (y + t~)~]
'
~, y e R ~
(9)
9
This can be treated as exactly orthogonal transformation 1 1
[~, y]
~
[(~
-
ty) v'l + t2' (y + t~) v'l + t2],
followed by the multiplicative perturbation I + Ec, IIEcll2 ___ec, which causes ec relative error in the singular values. Note that the condition number of the current matrix does not impact relative errors in singular values caused by I + E~. This means that r is a small part of the perturbation, as compared with relative perturbations caused by performing the transformation. Thus, in the case of forward stable rotation technique, the condition number takes place after maximal possible number of nonoverlapping rotations. This suggests a sort of optimal preconditioning in computing the singular values of full column rank A E R m• - performing the maximal number of nonoverlapping rotations between pivot pairs with largest cosines of the angles between corresponding columns. An additional advantage of (9) is that it can exploit small tangent t for faster computation. Indeed, for Itl <_ v/~ only the "fast part" of the transformation is performed. Consider now de Rijk's [2] linked triads rotation technique. Our main interest is forward stable implementation of the linked triads technique in the floating point environment providing the mar function. We give short forward error analysis which supports the claim: The linked triads implementation of the Jacobi rotation should determine the smaller vector in the first triad. Consider the two possible linked triads factorization of the rotation,
[ap, a'q] =
[
alp,a
;]
=
[ap, aq]
1
_ tan r
0 1
i ~ 0
cosr 0
1
0 cosr
i 1 tanr li 1 1 0 [ os0 01 ]
[ap,aq] 0
1
-~sin2r
1
0
COS ~)
(10) '
"
(11)
112
Z. Drmag
Let t be the computed value of the tangent and let ~ = f l ( t / ( 1 + t2)). Then the first triad in (10) can in the floating point computation be represented by 2 a~ := (r + Z ) ( a ~ - ta~), IIEIl~ < r The next one reads aq := ( I + F)(aq + a ( I + Z)(ap - taq))
= (x + F ) ( ~ + , , ( ~ - ta~))+ ~(I + F ) E ( ~ - t ~ ) , IIFII~< ~, and we easily see the forward manifestation of the wrong choice. 3 In the case of grading ( [lapl]2 > I]aqll2 ) the first triad in (11)is replaced by the modified transformation, and the next one is skipped.
2.3
OVERALL ERROR BOUND. RELIABILITY
In some applications of the SVD procedure is the input matrix quadratic. For example A E R mx'~ is factorized ATr = Q R by the QR factorization with optional column pivoting and the SVD of R is computed. This is reasonable e.g. for m >> n. (In the eigenvalue computation positive definite H is factorized, ~r*HTr = LI, ~, by Cholesky factorization with optional pivoting and the SVD of/_, is computed.) Due to the unitary invariance of the spectral norm, ~2(As) = ~2(Rs), where A s = Adiag ([]Aei[[2) -1, R s = Rdiag ([[Rei[]2) -1. The accelerated Jacobi method [4] runs on R r (on L in the eigenvalue computation of r~Hlr = LLr). The following two theorems show that the perturbations of singular values caused by the Jacobi procedure are of the same order as those caused by the QR (Cholesky) factorization. T h e o r e m 2.1 Let A E l:tmxn and let al >_ ".>_ an > 0 be its singular values. Let R be the upper triangular matrix computed by Givens QR factorization in the I E E E arithmetic with relative precision r and let 6A be the backward perturbation (A + 6A = QR). If the overall proces can be performed in p parallel steps, then the singular values al ~ "'" ~ an of R satisfy
I~
O'il
_< v~llAts[12((1 + 6r p - 1),
1 ___i _< n,
(12)
where q < n denotes the maximal number of nonzero entries in any row of 6A and provided that the right hand side in (12) is less than one. For m >> n and suitably chosen pivot strategy p ~ log 2 m + (n - 1) log 2 log 2 m. T h e o r e m 2.2 If one sweep of the Jacobi SVD method with some pivot strategy can be performed in p parallel steps on A E R nxn, then after l sweeps
l a i - ail _ v/-ff{id~sR{]2((1 + 10r _ 1), 1 <_ i <_ n, AsR = diag ({le~d[12)-lA,(13) ryi where al >_ ... >_ an > 0 and ~r1 >_ ... >_ ~,~ are the singular values of A and the matrix obtained at the end of the ith sweep, respectively, and provided that the bound in (13) is less than one. ~For the sake of technical simplicity we do not consider underflow exceptions. STake e.g. Ilapl[2 ~ HaqH2and ap and aq nearly parallel.
Singular Value Computation with the Jacobi Method
113
The main difference between our and the analysis of Demmel and Veselid is that we estimate relative errors in matrix rows while transforming its columns. The point is that the condition number of row-scaled matrices remains unchanged during the process because of the unitary invariance. 4 The conclusion of the error analysis is: If the gradual underflow is used and the singular values of the stored matrix belong to (v, r then the effect of underflow on the accuracy of the modified Jacobi SVD method is not worse than the uncertainties caused by roundoff errors.
2.4
FAST C O N V E R G E N C E ON GRADED MATRICES. WHY?
As we know, if the pivot columns are differently scaled, then the Jacobi rotation behaves like the Gram-Schmidt orthogonalization. On the other side, the modified Gram-Schmidt procedure follows the pattern (1, 2), ( 1 , 3 ) , . . . , ( n - 1, n), just as the row-cyclic 3acobi process does. With proper pivoting, introduced by de Rijk [2], the Jacobi SVD algorithm is forced to behave on graded matrices like the modified Gram-Schmidt orthogonalization, s (Initial sorting of matrix columns in nonincreasing sequence additionally supports this similarity.) This explains fast convergence of Jacobi SVD method on graded matrices. 2.5
CONDITION BEHAVIOUR. AN EXAMPLE
Explaining the excellent behaviour of maxk tc2(A(sk))/tc2(As) is stated in [4] as an important open problem. For the sake of compatibility with [4], [10] we consider hereafter the symmetric two-sided Jacobi method. Set H = ArA, Hs = ArsAs, H (k) = (A(k))*A (k), k = 1 , 2 , . . . Mascarenhas [10] found examples where maxk>ox2(H(k))/~2(Hs) can be as large as n/4. Although this is not bad because of factors of n already in the floating point errors, the condition behaviour in the Jacobi method remains interesting problem. Mascarenhas used S(n,a) = ( 1 - a)I + aee ~, e = ( 1 , . . . , 1 ) ~, 0 < a < 1, with n = 2 t and pivot strategy that produced recursively matrices of the same type. In the first step he obtained S (1) = (1 - a)In/2 • (1 + a)S(n/2, (1 - a ) / ( 1 + a)); in the next step the same strategy was applied to the lower right 2 z-1 x 2 l-1 submatrix of S (1) etc. To the growth of the condition of scaled matrices shown in [10], we added in [8] the fonowing observation:
The matrix S(n,a) is already optimally scaled, i.e. ~2(S(n,a)) <_ ~2(DS(n,a)D) for any diagonal nonsingular D, and the same property have all matrices S (i) generated by Jacobi. Similar analysis holds for the row-cyclic strategy. Thus, the matrix S(n, a) is actually an example of perfect behaviour of Jacobi rotations. Acknowledgements The material presented in this note is contained in the authors PhD Thesis, made at the Department of Mathematics, University of Hagen, and under the supervision of Prof. K. 4A similar result is obtained independently by Mathias [11]. 5The same analysis holds for the J-symmetric Jacobi method [12].
114
Z. Drma~'
Veseli6. The author wishes to express his appreciation and thanks for many useful comments to Prof. K. Veseli6. References
[1] J. Barlow and J. Demmel. Computing accurate eigensystems of scaled diagonally dominant matrices. SIAM J. Num. Anal., 27(3):762-791, 1990. [2] P. P. M. de Rijk. A one-sided Jacobi algorithm for computing the singular value decomposition on a vector computer. SIAM J. Sci. Stat. Comp., 10(2):359-371, 1989. [3] J. Demmel. Underflow and the reliability of numerical software. SIAM J. Sci. Stat. Comp., 5:887-919, 1984. [4] J. Demmel and K. Veseli~. Jacobi's method is more accurate than QR. SIAM J. Matrix Anal. Appl., 13(4):1204-1245, 1992. [5] Z. Drma~.. On relative errors caused by Jacobi and Givens rotations. Submitted to Numer. Math., October 1993. [6] Z. Drma~. Accurate singular value computation by Jacobi method. SIAM J. Sci. Comp., March 1994.
Submitted to
[7] Z. Drma~. Computing the Singular and the Generalized Singular Values. PhD thesis, Lehrgebiet Mathematische Physik, Fernuniversit~t Hagen, 1994. [8] Z. Drma~. On the condition behaviour in Jacobi method. Matrix Anal. Appl., January 1994.
Submitted to SIAM J.
[9] K. V. Fernando and B. N. Parlett. Implicit Cholesky algorithms for singular values and vectors. Technical report, Center for Pure and Applied Mathematics, University of California, Berkeley, August 1993. [10] W. F. Mascarenhas. A note on Jacobi being more accurate than QR. SIAM J. Matrix Anal. Appl., 15:215-218, 1993. [11] R. Mathias. Fast accurate eigensystem computations by Jacobi methods. Technical report, University of Minnesota, Minneapolis, May 1993. [12] K. Veseli6. A Jacobi eigenreduction algorithm for definite matrix pairs. Numer. Math., 64:241-269, 1990.
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
NOTE ON THE ACCURACY OF THE MATRICES GENERATED BY FINITE
Z. DRMAr
115
EIGENSOLUTION ELEMENTS
OF
K. VESELIC
Fachbereich Mathematik, Fern Universit~t Hagen, Postfach 9~0, D-58084 Hagen, Germany. [email protected], kresimir, [email protected]
ABSTRACT. Matrices generated by finite elements usually have high condition numbers which may spoil the accuracy of their lowest eigenvalues. This note considers some common classes of such matrices. These matrices are determined by some parameters which are not merely their non-zero elements. These parameters determine well the eigenvalues and the eigenvectors and the Cholesky factor. The latter is shown to be a perfect condition estimator and rank revealer. Accurate computation of this factor is still an open problem, we have succeeded to perform it only for some very simple cases: the simple chain a ad the simple loop with an arbitrary number of earth connections. KEYWORDS. Finite elements, relative accuracy, singular values.
1
INTRODUCTION
AND
SUMMARY
To find the singular values ai of a matrix with a high relative accuracy means to get the relative errors 6ai/ai small even then when ai differ greatly in their magnitude:
I~1 ~ c ~
(1)
for the corresponding singular values ai, ai + ~ai, respectively. Here C is a mild constant ([1]) and ~ is the machine precision. Algorithms of QR type with this kind of accuracy exist for bidiagonal matrices ([3, 6]). The same error bound holds for totally acyclic matrices (and for no other sparsity pattern) treated by a bisection algorithm ([2]). On the other side there exist broad classes of dense matrices for which (1) holds. These matrices are characterised not by a sparsity pattern but by a "scaled" condition number. In fact, if
G = DB , B has full column rank ,
(2)
where D is diagonal and nonsingular then
C .~ condB,
(3)
Z. Drma~ and K. Veseli~
116
([1, 4, 8]). A nearly best D has the diagonals equal to the Euclidean norms of the rows of G. The matrix G for which g2(DG) with some diagonal nonsingular D is a moderately growing function of n is called well-scaled from left. Since the transposed matrix G T has the same singular values on a square matrix G the right scaling G = B1D1 has the same consequence. On these matrices Jacobi methods are accurate ([4, 5, 7]). We are interested here in matrices arising e.g. in finite element formulation of vibrational systems. A vibrational system is governed by the differential equation
K s + M x = 0, where K (stiffness matrix) and M (mass matrix) are n x n real symmetric and positive definite. The eigenfrequencies ai of this system are the square roots of the eigenvalues of the problem Kx = AMz , or, equivalently, the generalized singular values of any pair (G1, G2) with K = GTG1, M = GTG2. We consider here the matrices of the form
K= ~
kiziz T , M = diag ( m t , . . . , m ~ ) , (4) i where ki, mi are positive parameters and zi are some incidence vectors. This again leads to the singular value problem for the matrix G = diag (gi)A0diag (#i) with A~ = [zlz2...], gi = v/~s, #i = mTsl/2. Quite generally it is advisable to compute the eigenvalues of a positive definite matrix K by the singular values of its factor, especially if that factor can be assembled directly. The reason is that in this way we avoid squaring which may spoil small eigenvalues. Our situation is even more special: here the singular values depend on the parameters ai, #i "with the condition one". Indeed, the perturbation ki + ~ki, mi -F ~mi with I~kil <_ ~ki, [~mil_ Emi implies for all x E R n ( 1 - ~)xT g x <_ x T ( g + ~ g ) x < (1 + ~)xT g x , (5) ( 1 - - e ) x T M x <_ x T ( M + ,SM)x <_ ( I + e ) x T M x . (6) As an immediate consequence of the minimax principle this implies (1) with C replaced by approximately 1 t. It would be ideal to have an algorithm, based on the input data ai, #i, which computes the singular values with that accuracy. In want of genuine direct algorithms we may try to transform "safely " the initial matrix e.g. into a bidiagonal one. No way of doing this seems to be known yet. Another possibility would be to arrive at a well-scaled matrix by a "safe" QR decomposition G M - 1 / 2 P = Q[R.TO]T, where Q is orthogonal and R is upper triangular. In other words, R is the Cholesky factor of H = p T M - 1 / 2 K M - 1 / 2 P , K = GrG. Here P is a permutation. Our first result (w 2) is" Let K be an irreducible diagonally dominant symmetric M-matrix; take the permutation P such that the (diagonal) matrix P M P T is increasingly ordered. Then the scaled condition of R from the left is at most nx/(n + 1)/2 independently of K, M. Here two points are remarkable: (i) the pivoting used is not the one of the standard "optimal" column pivoting, putting at every step the largest column to the first position, (ii) the factor R is a perfect condition estimator and rank-revealer in the sense that the singular values ai satisfy the aNote that the estimate for a general bidiagonal matrix is obtained just in transforming it to a scaled bidiagonal form, the larger constant obtained there (it is linear in n) is due to the loss upon this transformation ([1, 3]).
117
The Accuracy o f the Eigensolution o f Matrices
following estimate
J
n(n + 1)IR.I < ~i < J'~IR.I ,
(7)
independently of K, M. The other problem: to transform K, or better G, safely to this R seems to be much tougher. We have succeeded to do this only for two types of oscillation nets: the chain and the simple loop with an arbitrary number of earth connections; see w 3. In both cases the QR factorization of G is even forward stable.
2
WEAKLY
DIAGONALLY
DOMINANT
M-MATRICES
We consider here symmetric matrices with the properties: 1. the diagonal is positive, 2. the off-diagonal is non-positive, 3. the sum of each column is non-negative. LFrom this it follows that K is positive semidefinite. Without loss of generality we can assume that K is irreducible. An example of such K is (8)
g = ~ kijwijw T , i<j
with kij >_ 0 and ei, ei - ej,
wij =
for i = j, for i ~t j
(ei canonical basis vector).
Conversely, it is immediately seen that all symmetric matrices with the properties 1, 2, 3 above are of the form (8) 2. Obviously, for an irreducible K the positive definiteness means that one of kil is positive. Note that the Cholesky factor of K coincides with the QR factor of the matrix G for which K = G ' G . For instance, taking in (8) n = 3 and the coefficients k11, k12, k23, k13 > 0 we obtain
K =
kll + k12 q- k13 -k12 -kx2 k12 + k23
[
-k13
-k23
-k13 -k23 k23 + k13
J
G = '
K;12 -~12 ~13 0
0 -~13
0 /~11
--/~23
/~23
0
. (9)
0
In spite of the mentioned fact that the parameters kij determine perfectly the singular values of G and their squares i.e. the eigenvalues of K, by taking above k12 = k13 = k23 1, kll = 10 -16 the MATLAB function svd(G) gave here the singular values, rounded to 8
2One should distignuish between the matrices (kij) and K, the latter having elements Kij. Of course, ki.i = - K i j for i ~r j.
118
Z. Drma~ and K. Veseli~
places:
[
+000]
1.7320508e + 000 O.O000000e + 000
.
Thus, no correct digit is obtained in the smallest singular value 10-16/~/3 + (.9(10 -32) ,,~ 5.773502691896258.10 -17 although the machine precision of MATLAB is 10 -16. Also, the triangular factor computed by MATLAB's qr(G) had large relative error in the (3, 3) entry. We now come to the central result of this section. Any given upper triangular R with the properties 1, 2, 3 above can be scaled as R = D(I-
S~) , D = diag ( R l l , . . . , R n n )
9
(10)
Here S,~ is a strictly upper triangular non-negative matrix of order n with the row sums not exceeding one. T h e o r e m 2.1 Let Sn be as above 9 Then
O <_ A,~ = ( I -
1 0 .
S,~) - I <_ T,~ =
1 1 ..
9
0
... ... .. 9
...
1 1 . .
0
.
,
1
If, in addition, all nonvanishing row sums of S,~ are one then the last column of A,~ has all ones.
Since the spectral norm is a pointwise monotone norm on positive matrices we have [[A,~[12 _< [IT,~[12 _< ~ n ( n 2 + 1) .
(11)
Here the first inequality is attainable, while the second is very sharp since 1 2n 2 sin ~ r 4n+2 Thus, we have the following estimates for the extremal singular values of I -
~.(I-
s~) >
2 ~(~ + 1)' ~ ~
S,~
(i - s~) < v ~
The matrix R from (10) is well-scaled from the left; in particular, the inequalities (7) follow from Ostrovsky's theorem. The same estimates follow for matrices of type n x m: R =D[
(I-S,~)
, ] , D=diag(R11,...,R,~)
]
.
(12)
C o r o l l a r y 2.2 A n y K of the form (8) has a Cholesky factor or, equivalently, any corresponding a = diag ( k l l , k 1 2 , . . . )
[ wll w12
The Accuracy of the Eigensolution of Matrices
119
has the QR-factor which is well scaled. Note also that all assertions above are absolutely independent of the permutation of rows and columns in K or, equivalently, of columns in G. If the matrix K is substituted by M - 1 / 2 K M -1/2, M diagonal then instead of R above we obtain R M -1/2 which will inherit the well-scaledness of R, if the diagonals of M are increasing. Otherwise the left scaled condition of R M -1/2 may be spoiled. This necessitates the coresponding prepermutation of K, M or equivalently, G, M which, however, is different from the standard column pivoting OI1 e .
3
FORWARD
STABLE QR FACTORIZATION
In this section we describe the floating point implementation of the QR factorization of matrices from Corollary 2.2. Let ~, e, w be the underflow treshold, the roundoff unit and the overflow treshold, respectively. We use the standard model of the floating point arithmetic, i.e. for floating point reals a, b, c in the absence of underflow and overflow f l ( a o b) = (a o b)(l + eab), leabl _< e, o e { + , - - , ' , / } ,
ft(fi)
=
f i ( ~ + ~),
I~ _< ~.
We first consider quadratic matrices with the following two properties: P1. Each nonzero column contains exactly two nonzero elements. P2. Each nonzero row contains exactly two nonzero elements and their sum is zero. Let 7~n C R n• denote the set of matrices that have the properties P1 and P2. For A E Pn the QR factorization can obviously be divided into n - 1 ( at most ) steps where each step consists of one Givens rotation and eventually one row interchange. T h e o r e m 3.1 Let A E 7),~ and let A = Q R be the QR factorization of A constructed by a sequence of Givens rotations in some prescribed order. Let R = i:l + 5R be the triangular factor computed by the floating point implementation of the same process. Then in the absence of underflow and overflow
15Rij] <_
Lx_ kx _/
- 1 IRijl, 1 < i < j < n,
(13)
hold, where X+ = 1 + e and Pij <_ i - 1 depend on the number of changes of relevant elements in the past of the position (i, j). This analysis immediately implies the following perturbation estimate. T h e o r e m 3.2 Let A E 79n be given by incidence matrix Ao and n parameters h i , . . . , h n . Let A' E 79n have the same incidence matrix Ao and perturbed parameters h~ = hi + ~hi, 15hi] <_ elhil, I <_ i <_ n. Let R, R' be the triangular factors of A, A', determined by the same order of Givens rotations. Then
' R ~ j - RiJ' < [ (1+ 1-e
1] .Rij., l < i _< j _<
The forward stability property in the QR factorization of A E P~ remain if the structure is fixed against rigid-body motion. We give another interesting physical example as introduction to the next step in the algorithm and perturbation theory development.
Z. Drmal and K. Veseli~
120
In the nodal analysis of e.g. electrical resistance network the node conductance matrix of the Julie bridge is T A B ~" T A D nc TEA --TAB 0 --TAD
K =
--TAB TAB q- 7 B C -[- 7BE --TBC 0
--TAD
0 --TBC
0
7 B C @- 7 C D "4-7 C E --TCD
--TCD T A D 2r 7 C D 2r ")'DE
"
Then K = A~A with
~V~7~ - ~V'q~ o o o o ~ - ~Vq-~ o o ~ o o o ~ o o o ~ o o o
A
o - ~ o - ~ o o o
The upper 4 x 4 part of A is an element from 7)4 and can be transformed into the triangular form in a forward stable way. To get the triangular factor of K the update by the lower 4 x 4 part of A is needed 9 This leads us to the following problem: Given M - m a t r i x g = ~ k i j w i j = R~VR with upper triangular M - m a t r i x R and [( = K § ~_,~=1wieie~, wi >__ O, .find the Cholesky factor of [( in a forward stable way, using R. and w l , . . . , w n . Without loss of generality we take wl > 0, w2 . . . . . wn = O. The sign paterns are as follows: ~
D
9
m .
.
.
.
.
.
.
.
.
.
. m
+
.
.
.
.
.
. . . .
A-
R~I,
~
021 el
ArA=I(
=
+
.
:
:
:
9
9
.
.
.
". 9
.
+
Rotating at (1, n § 1) in A produces o
.
A
9
.
9
.
.
:=
+
+
...
+ +
and this transformation is obviously forward stable 9 Now, rotate at (2, n § 1). The sign patern of the transformation of pivot rows is -
+
+
+
+
...
+
0
+
+
...
+
"
All 'pluses' can be computed in a forward stable way. The minus signs ( @ ) are secure because /~ is an M-matrix, but in the application of the rotation they are computed by
The Accuracy of the Eigensolution of Matrices
121
proper substraction which does not obey the forward stability property. We can easily handle this problem by using formulas from the Cholesky factorization of K. Indeed,
A2j := ~ 1 ( / ( 2 j - A12Alj), 3 <_j <_n, /'122 ~
<0
>o
are forward stable formulas for the rest of the second row. Generally, Aii is stable function during the QR factorization and
Aij "=
(Kij-
AkiAkj), i + 1 <_j <_n, k=l
are also forward stable formulas. In simple words, the diagonals of R are computed .from the QR factorization and the off-diagonals from the Cholesky factorization, if needed. The single precision QR factorization followed by one sided single precision Jacobi algorithm gave 53 = 0.57735030.10 -18 as smallest singular value of the matrix (9), with relative error less than 6.10 -s. References
[1] J. Barlow and J. Demmel, Computing accurate eigensystems of scaled diagonally dominant matrices, SIAM J. Num. Anal., 27 (1990), pp. 762-791. [2] J. Demmel and W. Gragg, On computing accurate singular values and eigenvalues of acyclic matrices, Linear Algebra Appl., 185 (1993), pp. 456-467. [3] J. Demmel and W. Kahan, Accurate singular values of bidiagonal matrices, SIAM J. Sci. Stat. Comp., 11 (1990), pp. 873-912. [4] J. Demmel and K. Veselid, Jacobi's method is more accurate than QR, SIAM J. Matrix Anal. Appl., 13 (1992), pp. 1204-1245. [5] Z. Drma~, Computing the Singular and the Generalized Singular Values, PhD thesis, Lehrgebiet Mathematische Physik, Fernuniversitgt Hagen, 1994. [6] K. V. Fernando and B. N. Parlett, Accurate singular values and differential qd algorithms, Numer. Math., 67 (1994), pp. 191-229. [7] R. Mathias, Fast accurate eigensystem computations by Jacobi methods, technical report, University of Minnesota, Minneapolis, May 1993. [8] K. Veselid and I. Slapni~ar, Floating-point perturbations of Hermitian matrices, Linear Algebra Appl., 195 (1993), pp. 81-116.
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
123
TRANSPOSE-FREE ARNOLDI ITERATIONS FOR APPROXIMATING EXTREMAL SINGULAR VALUES AND VECTORS 1 M.W. BERRY Department of Computer Science University of Tennessee i07 A yres Hall Knoxville, TN 37##6-I301 USA berry @cs.utk. edu S. VARADHAN Department of Computer Science University of Tennessee 107 A yres Hall Knoxville, TN 37996-1301 USA varadhan @cs.utk. edu ABSTRACT. Block Lanczos algorithms are commonly used to compute extremal eigenpairs or singular triplets of large sparse matrices. For well-conditioned matrices A, the block Lanczos algorithm can be used to compute extremal singular values and singular vectors by solving the equivalent symmetric eigensystem of A TA. Each iteration requires multiplication by both A and A T. An alternative algorithm based on the block Arnoldi method for the nonsymmetric eigensystem [A]0 ] is presented. This method does not require multiplication by A T up to restarting. The performance of both methods in the number and type of sparse matrix multiplications is presented for a few selected test matrices. For diagonally-dominant matrices, Inverse Iteration (INVIT) is one acceleration alternative that can be used to compute higher-accurate restart vectors for a single-vector Arnoldi-based SVD method. Both single-vector and block Arnoldi SVD methods may be more suitable for architectures in which multiplication by A T does not equal that by A subject to data structure constraints. KEYWORDS. Arnoldi, extremal, singular, sparse, transpose-free, values, vectors. 1This research was supported by the National Science Foundation under grant Nos. NSF-CDA-9115428 and NSF-ASC-92-03004.
124 1
M.W. Berry and S. Varadhan INTRODUCTION
In applications such as information retrieval and seismic reflection tomography [3], the singular value decomposition (SVD) of large sparse matrices is needed in the shortest possible time. Given the growing availability of high-performance computer systems, there has been great interest in the development of efficient implementations of the singular value decomposition. SVDPACKC [4] is an ANSI C software library specifically designed to compute the SVD of large unstructured sparse matrices. Without loss of generality, suppose A is a sparse m by n (m _> n) matrix with rank(A) = r. The singular value decomposition (SVD) of A can be defined as A = U E V T,
(1)
where u T U = v T v = In and E = diag(al, ..., an), ai > 0 for 1 __ i < r, ai = 0 for i > r + 1. The first r columns of the orthogonal matrices U and V define the orthonormalized eigenvectors associated with the r nonzero eigenvalues of A A T and A TA, respectively. The singular values of A are defined as the diagonal elements of E which are the nonnegative square roots of the n eigenvalues of A A T. The set {ui, ai, vi} is called the i-th singular triplet. The singular vectors (triplets) corresponding to large (small) singular values are called large (small) singular vectors (triplets). SVDPACKC uses Lanczos, block-Lanczos, subspace iteration, and trace minimization methods to approximate extremal singular values and corresponding singular vectors. Two canonical sparse symmetric eigenvalue problems are solved by SVDPACKC routines to (indirectly) compute the sparse SVD. For well-conditioned matrices A, the n x n eigensystem of AT A is the desired eigenvalue problem. For SVDPACKC methods such as B[.52, which implements a block Lanczos iteration similar to [5], each step of the Lanczos recursion requires multiplication by A and A T. If memory is not sufficient to store both A and A T, the matrix A is normally stored in an appropriate compressed sparse matrix format [2]. Subsequently, the multiplication by A T may perform poorly as compared to multiplication by A simply due to excessive indirect addressing enforced by the data structure in A [3]. Can the multiplication by A T be avoided altogether? Probably not, but as demonstrated in the next section, the block Arnoldi method applied to the m • m matrix [AI0 ] is an alternative scheme which can suppress the multiplication by A T up to restarts. 2
AI:tNOLDI SVD METHOD
Sand and Schultz [8] [9] originally proposed the GMI:tES method for solving large sparse nonsymmetric linear systems based on the Arnoldi process [1] for computing the eigenvalues of a matrix. Can Arnoldi's method, which is essentially a Gram-Schmidt method for computing an orthonormal basis for a particular Krylov subspace, be used to compute the SVD in (1) without having to reference A T explicitly? Consider the block Arnoldi iteration below.
Transpose-free Arnoldi Iterations
125
A l g o r i t h m 2.1 Transpose-Free ARnoldi (T[=AR) S V D iteration. 1. Choose V1 (m • b) such that v T v 1 = /b, and define B = [AlO], so that B is an m • m nonsymmetric matrix, and A is a given m • n (m _> n) sparse matrix. 2. For j = 1, 2, ...k W
=
B•
Hid
=
viTw,
i=l,2,...,j J
W
=
W-y~Vi•
W
=
QR, QTQ = Ibm and R is upper triangular,
-
R.
i=1 Hj+I,j
Let V = IV1, V2,..., Vk], and assemble the block upper Hessenberg matrix H from the b • b submatrices, Hi,j,i = 1 , . . . , k + 1;j = 1 , . . . , k , generated in Step 2 of the Arnoldi iteration above. Define the n • bk matrix V = [In l0] • V so that
is an orthogonal factorization of 1)" where Q k is n x bk and has orthonormal columns. It can be shown that AQ, k = VCk + Vk+l HTk+l,k • GT]~-I,
(2)
where Ck = H k R -1 , Hk is the bk• block upper Hessenberg matrix constructed by deleting the last b • b submatrix o f / t k , and G T is the b • bk matrix defined by G T = [0, ... , 0 I/b]. Via (2), the extremal singular triplets of the m • n matrix A are approximated by the k singular triplets of the k • k Hessenberg matrix Ck. If the SVD of Ck = (]kEkV T , where uT~rk = ~'TVk = Ik and ~k = diag[~l, a2, . . . , ak], then A(Ok~/k) = (V~fk)Ek +
(Vk+~HT+~,k)(aTk-~fZk),
(3)
and the i-th diagonal element of Ek is an approximation to the exact singular value, hi, of A in (1). Now, define Yk = QkVk = [Yl, Y2, . . . , Yk] and Zk = YUk = [zl, z2, . . . , Zk], where Yk and Zk are n x k and m x k, respectively. Then, (3) becomes AYk -- Z k ~ k + (Vk+l H Tk+l,k )(GT/~-IVk)
(4)
where the columns of Zk and Yk are approximations to the left and right singular vectors of A, respectively. If the columns of Yk are not suitably accurate approximations of the right singular vectors of A, Algorithm 2.1 can be restarted by setting
vxr = [(ATzk~;~ ) r 10~_~,b], r
(5)
126
M. W. Berry and S. Varadhan
as an improved approximation to b-largest right singular vectors of A. Hence, b multiplications of A T are required upon each restart of Algorithm 2.1. If one can easily solve systems of the form
(AT A - b2I)y k+l = 0,
(6)
then Inverse Iteration (INVIT) can be used to greatly improve the accuracy (see [7]) of the restart vectors stored as columns of V1. The use of INVIT for restarting Algorithm 2.1 is referred to as Arnoldi with INVerse iteration or AINV. 3
PRELIMINARY
PERFORMANCE
The performance of Algorithm 2.1 or TFAR and BL52 from SVDPACKC [4] is compared with respect to the number of required multiplications by A and A T. All experiments were conducted using MATLAB (Version 4.1) on a Sun SPARCstation 10. For the four test matrices mentioned below, the p-largest singular triplets, where p = 2q, q = 1, 2, 3, 4, are approximated to residual accuracies [Iri[[2 = []A~i- aifii[]~ of order (..9(10-3) or less. Deflation was used in both methods to avoid duplicate singular value approximations. In Figure 1, the well-separated spectrum of the 374 x 82 term-document matrix from [3] is approximated. Initial block sizes b = p and Krylov subspace dimension (bk) bounds of 2p were used in both methods. The TFAR scheme required, on average, 60% fewer multiplications by A T (represented by y=A'x) with a 40% increase in multiplications by A only for the cases p = 6, 8. In Figure 2, a well-separated portion of the spectrum of the 457 x 331 constraint matrix from the collection of linear programming problems assembled in [6] was approximated. These experiments used initial block sizes of 2, 4, 6, 4 and corresponding Krylov subspace dimension bounds of 12, 24, 24, 20 for p = 2, 4, 6, 8 triplets, respectively. On average, TFAR required 65% fewer multiplications by A T but unfortunately at the cost of over 200% more multiplications by A. Figures 3 and 4 compare the performance of AINV, BLS2, and TFAR on two 50 x 50 synthetic matrices with clustered spectra. The spectra of these test matrices (see table below) are specified by the function r fl, 6, k,n,~x, ima=) = (ak + t3) + 6i, where k = 0 , 1 , . . . , k m a x is the index over the clusters, i = 1 , 2 , . . . , irnax is the index within a duster, 6 is the uniform separation between singular values within a duster, and a, fl are scalars.
Matrix CLUS1 CLUS2
Number of Clusters 13 5
Cluster Separation (6) 10 -3 10-6
Maximum Cluster Size 4
10
~.(a,13,6, kmax, ima=)
r r
12,4)
In approximating the p-largest triplets of CLUS1, both TFAR and BL52 used initial block sizes of 2, 4, 6, 4 and corresponding Krylov subspace dimension bounds of 12, 24, 24, 20 for p = 2, 4, 6, 8, respectively. For CLUS2, initial block sizes of 2, 4, 6, 8 and corresponding Krylov subspace dimension bounds of 4, 8, 12, 16 were used. An implementation of AINV using a Krylov subspace dimension bound of 16 and deflation for a static block size b = 1
Transpose-free Arnoldi Iterations
127
470
300
= --m" -,a-
BLS2 (y=Ax) BLS2 (y=A'x) TFAR (y=Ax) TFAR (y=A'x)
/ / /
,,4~4" ,,,4"," /("
cg~ 100
Number of Singular Triplets
Figure 1: Performance of BLS2 and TFAR on the 374 • 82 matrix ADI for 10 -3 residual accuracy.
470
3oo
= BLS2 (y=Ax) --i- BLS2 (y=A'x) -*-4 TFAR (y=Ax)
.~
~2oo
100
9
1
"
~
"
8
Number of Singular Triplets
Figure 2: Performance of BLS2 and TFAR on the 457 x 331 matrix SCFXM1 for 10 -3 residual accuracy.
M.W. Berry and S. Varadhan
128
: - 4-
:
- g-
AINV (y=Ax) (y=A'x) BLS2 (y=Ax) BLS2 (y=A'x)
AINV
,; ,"
s
9 TFAR(y=Ax)
..o
--~- TFAR (y=A'x)
~ 100I I
1
2
---
"
----O-
1
~
8
Number of Singular Triplets
Figure 3: Performance of AINV, BLS2, and TFAR on the 50 • 50 matrix CLUS1 for 10 -3 residual accuracy.
| | / .2~2oo.~
:
-4-
mNV
sI
(y=Ax)
AINV (y=A'x) BLS2 ( y = A x ) - - - BLS2 (y=A'x) 9 TFAR (y:Ax)
/."
=
~ "
/." /~'"
sssssSSSS
2
4 6 Number of Singular Triplets
8
Figure 4: Performance of AINV, BLS2, and TFAR on the 50 x 50 matrix CLUS2 for 10 -3 residual accuracy.
129
Transpose-free Arnoldi Iterations
(i.e., a single-vector iteration) was also applied to these synthetic matrices. In Figure 3, about 85% fewer multiplications (on average) by A T and 35% fewer multiplications (on average) by A are required by TFAR. In Figure 4, the reduction factors in multiplications by A and A T for TFAR are about 50% and 80%, respectively. At the cost of solving the systems in (6), both Figures 3 and 4 demonstrate that AINV requires approximately 42% and 96% fewer multiplications by A and A T, respectively, than that of BIS2.
20
40 60 Subspace Dimension
•
Mults by A
l
Mults by A'
80
Figure 5: Distribution of sparse matrix-vector multiplications for TFAR as the Krylov subspace dimension increases. The largest singular triplet of the 374 x 82 term-document matrix ADI is approximated to a residual accuracy no larger than 10 -3. Figure 5 illustrates how the distribution of sparse matrix-vector multiplications by A or A T varies as the Krylov subspace dimension (k from Algorithm 2.1) increases. Here, k ranges from 10 to 75 as we approximate the largest singular triplet { u l , a l , Vl} of the 374 x 82 ADI matrix. Clearly, the need for restarting (see (5)) diminishes as the subspace dimension increases and thereby requires fewer and fewer multiplications by A T at the cost of requiring more storage for the Arnoldi vectors (columns of V in Algorithm 2.1). 4
FUTURE WORK
Although extensive testing of TFAR and AINV is certainly warranted, the results obtained thus far for selected matrices is promising. Future research will include comparisons with recent Arnoldi methods based on implicitly shifted QR-iterations [10]. Although there does not appear to be a way to completely remove the multiplication by A T in computing the SVD of A, there can be substantial gains for methods such as TFAR (or AINV) when multiplication by A T is more expensive than multiplication by A. Alternative restarting
130
M.W. Berry and S. Varadhan
procedures may improve the global convergence rate of the TI=AR method and further reduce the need for multiplications by A T . References
[1] W. E. Arnoldi, The Principle of Minimized Iteration in the Solution of the Matrix Eigenvalue Problem, Quart. Appl. Math., 9 (1951), pp. 17-29. [2] R. Barrett et al., Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, SIAM, Philadelphia, 1994. [3] M. W. Berry, Large Scale Singular Value Computations, International Journal of Supercomputer Applications, 6(1992), pp. 13-49. [4] M. W. Berry et al., SVDPACKC: Version 1.0 User's Guide, Tech. Rep. CS-93-194, University of Tennessee, Knoxville, TN, October 1993. [5] G. Golub, F. Luk, and M. Overton, A Block Lanczos Method for Computing the Singular Values and Corresponding Singular Vectors of a Matrix, ACM Transactions on Mathematical Software, 7 (1981), pp. 149-169. [6] I. J. Justig, An Analysis of an Available Set of Linear Programming Test Problems, Tech. Rep. SOL 87-11, Systems Optimization Laboratory, Stanford University, Stanford, CA, August 1987. [7] B. N. Parlett, The Symmetric Eigenvalue Problem, Prentice Hall, Englewood Cliffs, NJ, 1980. [8] Y. Saad, Krylov Subspace Methods for Solving Large Unsymmetric Linear Systems, Mathematics of Computation, 37 (1981), pp. 105-126. [9] Y. Saad and M. H. Schultz, GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems, SIAM Journal of Statistical and Scientific Computing, 7 (1986), pp. 856-869. [10] D. C. Sorensen, Implicit Application of Polynomial Filters in a K-Step Arnoldi Method, SIAM Journal of Matrix Analysis and Applications, 13 (1992), pp. 357-385.
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
131
A LANCZOS ALGORITHM FOR COMPUTING THE LARGEST QUOTIENT SINGULAR VALUES IN REGULARIZATION PROBLEMS
P.C. HANSEN
UNI. C, Building 304 Technical University of Denmark DK-2800 Lyngby, Denmark Per. [email protected] M. HANKE
Institut fiir Praktische Mathematik Universitgt Karlsruhe, Englerstrasse 2 D-76128 Karlsruhe, Federal Republic of Germany hanke @ipmsun l. mathematik, uni-karlsruhe, de
ABSTRACT. In linear regularization problems of the form min{llA x - bll 2 + )~21IBxll2}, the quotient singular value decomposition (QSVD) of the matrix pair (A, B) plays an essential role as analysis tool. In particular, the largest quotient singular values and the associated singular vectors are important. We describe an algorithm based on Lanczos bidiagonalization that approximates these quantities. KEYWORDS. Lanczos bidiagonalization, QSVD, regularization.
1
INTRODUCTION
If the coefficient matrix of a (possibly overdetermined) system of linear equations is very ill conditioned, then the solution is usually very sensitive to errors (measurement errors, approximation errors, rounding errors, etc.), and regularization is therefore needed in order to compute a stabilized solution which is less sensitive to the errors. Perhaps the most popular regularization method is the method due to Tikhonov and Phillips, see [7]. In this method, the regularized solution is computed as the solution to the following problem
rnJn {llAx - bl]2 + )~211Bzl]~).
(1)
P.C. Hansen and M. Hanke
132
Here, A is the coefficient matrix and b is the right-hand side of the linear system to be solved. In many regularization problems in signal processing, such as image restoration and other deconvolution problems, the coefficient matrix A is either (block) Toeplitz or (block) Hankel. This structure of A should be utilized whenever possible. The matrix B in (1) defines a seminorm, and typically B is a discrete approximation to loss of generality we can assume that B has full row rank. We will also assume that B is reasonably well conditioned--otherwise it will not have the desired regularizing effect.
a derivative operator. Without
The parameter A is the regularization parameter, and it controls how much emphasis is put on minimizing the smoothness of the solution x, measured by lib zl12 , relative to minimization of the residual norm IIA x - b[12. How to choose )~ is an important topic which we shall not consider here; see, e.g., [9] for details. In addition to the Tikhonov/Phillips method, many other regularization methods can be defined. Surveys of some of these methods can be found in, e.g., [4], [7] and [9]. Common for all these methods is the appearance of the matrix pair (A, B), where A is the coefficient matrix, and B is the regularization matrix. For all these regularization methods involving the pair (A,B), it turns out that the a very valuable tool for analysis of the problem as well as for computing regularized sohtions. Let A and B have dimensions m x n and p x n, respectively, satisfying m __ n _> p. Then the QSVD takes the form
Quotient Singular Value Decomposition1 (QSVD) is
A = U ( ~0 In-,O) X-I'
B = V ( M , O ) X -1,
(2)
where U and V have orthonormal columns, X is nonsingular, and E and M are p x p diagonal matrices, = diag(ai),
M = diag(#i).
(3)
Then the quotient singular values are defined as the ratios "r~ = ,~lu~,
i = 1,...,p.
(4)
We note that if B is nonsingular, then the quotient singular values are the ordinary singular values of the matrix quotient A B -1. The reason why the QSVD is so important in the analysis of regularization problems is that the regularized solution can often be expressed in terms of the QSVD components. For example, for Tikhonov regularization (1) the regularized solution is given by
~'r
"/? + '~2 ffi
i=p+l
Here, ui and xi are the columns of U and X, respectively. Other regularized solutions can be obtained from (5) by replacing 72/(72 + ,~2) with the appropriate filter factors for the particular regularization method. Thus, we see that information about the QSVD components gives important insight into the regularization problem. 1The QSVDwas previously known as the
GeneralizedSVD.
A Lanczos Algorithm
133
In particular, the QSVD components corresponding to the largest quotient singular values 7i are important because they constitute the essentially "unregularized" component of the regularized solution, i.e., the component for which the associated filter factors are close to one. In fact, one can define a regularization method (called "truncated GSVD" in [10]) in which the corresponding filter factors are simply ones and zeros. Another important use of these QSVD components is to judge the choice of the regularization matrix B. Only limited theoretical results are available regarding this choice, and often the user wants to experiment numerically with different B-matrices and monitor the influence of B on the basis vectors--the columns of X ~ f o r the regularized solution. The largest quotient singular value is also needed as a scaling factor in connection with the v-method for iterative regularization, cf. [9, w Hence, there is a need for efficient algorithms that compute the largest QSVD components when A is large and structured or sparse. A Lanczos-based algorithm for computing the largest QSVD components of a general matrix pair was proposed recently by Zha [14]. In this paper we present an alternative algorithm specifically designed for the case where B is a banded matrix with full row rank~which is the typical situation in regularization problems. By making use of this property of B, we obtain an algorithm which is faster than the more general algorithm. A related direct algorithm which involves a full SVD computation can be found in [11].
2
THE
QSVD L A N C Z O S B I D I A G O N A L I Z A T I O N
ALGORITHM
The key idea in our algorithm is to apply the Lanczos bidiagonalization algorithmmfor computing the largest singular values, see, e.g., [1, w [2, w and [6, w the matrix A, but "preconditioned" from the right in a specialized way such that the quotient singular values and associated vectors are obtained. The "preconditioner" is BtA, the A-weighted generalized inverse of B, see [3], which in terms of the QSVD can simply be written as
BtA = X ( M-1) 0 vT '
(6)
and we see that A BtA is then given by
0 Hence, the ordinary
(7)
SVD of A BtA is identical to part of the QSVD of (A, B).
The use of this "preconditioning" for iterative methods was initially advocated in [8]. We emphasize that the purpose of the "preconditioner" is not to improve the spectrum of the matrix, but rather to transform the problem such that the desired quotient singular values are computed. More details can be found in [9]. In the Lanczos bidiagonalization algorithm we need to multiply a vector with A BtA and
(A BtA)T. Hence, two operations with BtA are required, namely, matrix-vector multiplication
P.C. Hansen and M. Hanke
134
with BtA and (BtA) T. With the aid of some limited preprocessing, both operations can be implemented in ( . 9 ( ( n - p + l)n) operations, where I is the band width of B. The preprocessing stage takes the following form. PREPROCESSING STAGE W .-- basis for null space of B A W = Q R ( QR factorization) S ~ R-1QTA /3 ~ augmented B (see text below) /} = ],/)" (L V factorization of/}). Usually, the n x (n - p) matrix W can be constructed directly from information about B; see, e.g., [11] and (19) below. The QR decomposition of the "skinny" m x ( n - p) matrix A W is needed to compute its pseudoinverse (A W) t = R -1 QT. The computation of S requires O ( m ( n - p)2) operations plus n - p multiplications with A and with A T. Next, the p x n matrix B is augmented with n - p rows such that the resulting m a t r i x / 3 is square and nonsingular, followed by L U factorization of J3. If B is a discrete approximation to the (n - p)th derivative operator of the form B=
(1_1) "..
then one can set
".. 1
or
B=
-1
(1_21) "..
".. 1
0),
".. -2
,
(s)
1
(~)
and no L U factorization is actually required because /} is lower triangular. If B is a discrete approximation to the Laplacian on a square N x N grid (in which case n = N 2 and p = n - 4), then one particular choice s of B has the block form B(~ B(1)
0 B(2) 9..
B=
B(1) ..
..
B(i) B(~) BO) 0
where B (~ is ( N -
B (o) B (1)
= =
(~0)
B (~
2) • N, B (1) and B (~) are N x N, and these matrices have the form
"..
"..
"..
2
-4
(11) 2
d i a g ( 2 , 1 , 1 , . . . , 1,2)
2This particular choice of B corresponds to the MATLABfunction del2.
(12)
A Lanczos Algorithm
-4 1
0 -4 "..
B (2) =
1 ".. 1
".. . -4 1 0 -4 Then one can obtain/} by augmenting B (~ as follows ~(o) ,_
(40...0) (o)
,
135
(la)
(14)
0...04 and since B has bandwidth 2N + 1, an L U factorization without pivoting requires O ( n N 2) = O(N 4) operations, see [6, w For this choice of/}, pivoting is not required because/~ is column diagonally dominant (see [6, w Given the matrices W and 5', and possibly an LUfactorization of/}, the two algorithms for multiplication with B?A and (B?A) T take the following form. COMPUTE ,-y ~ y ~
y = BtA z x (pad x with zeros according to B ~ / ~ ) /}-12 y-Wb'y.
COMP,TE V = ( B ~ ) r 9 X ~-- X-- s T w T x
fl y
,*-
(i~-~ )r ~ ~ (extract elements of ~) according to B ~ B).
Both algorithms require (.9((n- p + t)n) operations, where l is the band width of B. In particular, if B is given by (8) and (10), then 3 ( n - p)n and 2(N + 4)N 2 multiplications axe required, respectively.
3
HOW TO EXTRACT
THE NECESSARY QUANTITIES
The Lanczos algorithm immediately yields approximations of the largest quotient singular values 7i and the associated singular vectors ui and v~ (the columns of U and V in (2)). From 7i, the quantities ai and #i in (3) can easily be computed from 7~ 1 "'
= J1 +-~'
~'
: J1 +-y?
(15)
which follow from (4). Finally, we need to be able to compute the columns xi of X corresponding to 7i. Using the Q S V D (2) and the definition (6) of B?A, we get
B~v=x
0
'
(16)
136
P.C. Hansen and M. Hanke
from which we obtain the relation zi = ~i BOAvi.
(17)
Hence, xi can be computed from the corresponding vi by multiplication with B?A, which can be performed by means of the algorithm from the previous section. Notice that the vectors ui and xi for i = p + 1 , . . . , n cannot be determined by our Lanczos algorithm. However, the spaces span{xp+i,..., xn} and span{up+i,..., un}--which may be more important than the vectors--are spanned by the columns of the two matrices W and A W, respectively. Moreover, the vector
9o = ~ i..---p+l
~Tb ~i,
which is the component of the regularized solution x~ (5) in the null space of B, can be computed directly by means of the relation
xo = W (A W ) t b = W R -1QT b,
(iS)
where Q and R form the QR factorization of A W, cf. the preprocessing stage. We emphasize that our single-vector Lanczos algorithm cannot compute the multiplicity of the quotient singular values. If this is important, one should instead switch to a block Lanczos algorithm, cf. [5]. 4
NUMERICAL
EXAMPLES
We illustrate the use of our Lanczos algorithm with a de-blurring example from image reconstruction. All our experiments 3 are carried out in MATLAB using the REGULARIZATION TOOLS package [12]. The images have size N x N. The matrix A is N ~ x N 2, it represents Gaussian blurring of the image, and it is doubly Toeplitz, i.e., it consists of N x N Toeplitz blocks which are the same along each block diagonal; see [9, w167 and 8.2] for details. The matrix B is the discrete approximation to the Laplacian given by (10)-(13), and its size is ( g 2 - 4) x g 2. The starting vector is Ae/llAell2 with e = ( 1 , . . . , 1) T, and no reorthogonalization is applied. The matrix W representing the null space of B is computed by applying the modified Gram-Schmidt process to a matrix W0 of the following form, shown for N = 4: wT =
Ii
1 1 1
1 1 1 1 1 1 1 1 1 1 1
1 2 2
1 1 3 4 3 4
2 2 1 2 2 4
2 3 6
2 4 8
3 1 3
3 2 6
3 3 9
3 4 12
1 1
4 4 1 2 4 8
1
4 3 12
i\
4 4 16
"
(19)
Our first experiment involves a small problem, N = 16, for which we computed all the 252 quotient singular values 7i explicitly by means of routine gsvd from [12]. The first five 7i axe very well separated in the range 175-46, the next 15 7i are less well separated in the range 38-6.3, and the next 30 7i are even closer separated in the range 5.5-0.67. 3The MATLABroutines used in these tests are available from the authors.
137
A Lanczos Algorithm
number of iterations k 10 20 30 40 50
1 1 3 4 5 7
2 1 2 3 4 5
3 4 1 2 3 4 5
index i 6 5 7 1 1 1 2 2 3 2 3 3
of exact quotient singular value 3'~ 10 14 8 9 11 12 13 15 16 17 1 1 2 2
1 1 2 2
1 1 2 2
1 1 1 2
1 1 2
1 1 1
1 1 1
1 1 1
18
19 20
1 1
Table 1: The convergence history of the quotient singular values in the small example with N = 16. The table shows the number of approximations to the ith exact quotient singular value 3'i, after k Lanczos iterations, whose absolute accuracy is better than 10 -l~ Multiple Vi, such as V3 and 74, are grouped. O
k 10 20 30 40 50
u(k) 5 11 14 21 28
~
smallest converged 1.19.103 2.56 9102 2.56 9102 8.98.101 3.21.101
Table 2" The convergence of the quotient singular values for the large example with N = 64. The table shows the number u(k) of converged distinct quotient singular values after k Lanczos iterations, together with the smallest converged value.
The convergence of the approximate quotient singular values reflects this distribution, see Table 1 next page where we list the computed quotient singular values whose absolute accuracy is better than 10 -l~ We see that the first, well separated 7i are captured fast by the algorithm, while more iterations are required to capture the 7{ which are less well separated. We also observe the appearance of multiple (or "spurious") quotient singular values which is a typical phenomenon of the Lanczos process in finite precision and which is closely related to the convergence of the approximate 7i, cf. [13, w Our next example involves a larger problem, N = 64, which is too large to allow computation of the exact quotient singular singular values by means of routine gsvd. Table 2 shows the number of converged distinct quotient singular values after k = 10, 20, 30, 40, and 50 Lanczos iterations, using a loose convergence criterion (10 -3) and identifying spurious values using the test by Cullum & Willoughby [2, w The largest quotient singular value is 71 = 7.66.103, and the table also lists the size of the smallest converged quotient singular value. The numbers u(k) are for N = 64 are not inconsistent with the results for
N=16.
138
P.C. Hansen and M. Hanke
References [1] /~. BjSrck. Least Squares Methods. In 9P.G. Ciarlet and J.L. Lions (Eds.), Handbook of Numerical Analysis, Vol. I~ North-Holland, Amsterdam, 1990. [2] J.K. Cullum and R.A. Willoughby. Lanczos Algorithms for Large Symmetric Eigenvalue Computations. Vol.I Theory. Birkh~user, Boston, 1985. [3] L. Eld~n. A weighted pseudoinverse, generalized singular values, and constrained least squares problems. BIT 22, pp 487-502, 1982. [4] H.W. Engl. Regularization methods for the stable solution of inverse problems. Surv. Math. Ind. 3, pp 71-143, 1993. [5] G.H. Golub, F.T. Luk and M.L. Overton. A block Lanczos algorithm for computing the singular values and corresponding singular vectors of a matrix. A CM Trans. Math. Soft. 7, pp 149-169, 1981. [6] G.H. Golub and C.F. Van Loan. Matrix Computations. 2. Ed. Johns Hopkins University Press, Baltimore, 1989. [7] C.W. Groetsch. Inverse Problems in the Mathematical Sciences. Vieweg, Wiesbaden, 1993. [8] M. Hanke. Iterative solution of underdetermined linear systems by transformation to standard form. In : Proceedings Numerical Mathematics in Theory and Practice, Dept. of Mathematics, University of West Bohemia, Plzefi, pp 55-63 (1993) [9] M. Hanke and P.C. Hansen. Regularization methods for large-scale problems. Surv. Math. Ind. 3, pp 253-315, 1993. [10] P.C. Hansen. Regularization, GSVD and truncated GSVD. BIT 29, pp 491-504, 1989. [11] P.C. Hansen. Relations between SVD and GSVD of discrete regularization problems in standard and general form. Lin. Alg. Appl. 141, pp 165-176, 1990. [12] P.C. Hansen. Regularization Tools: A Matlab package for analysis and solution of discrete ill-posed problems. Numerical Algorithms 6, pp 1-35, 1994. [13] B.N. Parlett. The Symmetric Eigenvalue Problem. Prentice-Hall, Englewood Cliffs, N.J., 1980. [14] H. Zha. Computing the Generalized Singular Values/Vectors of Large Sparse or Structured Matrix Pairs. Report CSE-94-022, Dept. of Computer Science and Engineering, Pennsylvania State University, January 1994; submitted to Numer. Math. A condensed version appears in these proceedings.
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
A QR-LIKE SVD ALGORITHM SEVERAL MATRICES
FOR
139
A PRODUCT/QUOTIENT
OF
G.H. GOLUB
Computer Science Department Stanford University Stanford, CA U.S.A. golub @sccm.stanford, edu K. SOLNA
Computer Science Department Stanford University Stanford, CA U.S.A. solna @sccm.stanford, edu P. VAN DOOREN Cesame
Universitd Catholique de Louvain Louvain-la-Neuve Belgium vandooren @anma. ucl. ac. be
ABSTRACT. In this paper we derive a new algorithm for constructing unitary decomposition of a sequence of matrices in product or quotient form. The unitary decomposition requires only unitary left and right transformations on the individual matrices and amounts to computing the generalized singular value decomposition of the sequence. The proposed algorithm is related to the classical Golub-Kahan procedure for computing the singular value decomposition of a single matrix in that it constructs a bidiagonal form of the sequence as an intermediate result. When applied to two matrices this new method is an alternative way of computing the quotient and product SVD and is more economical than current methods. KEYWORDS. Numerical methods, generalized singular values, products of matrices, quotients of matrices.
140
1
G.H. Golub et al.
INTRODUCTION
The two basic unitary decompositions of a matrix A yielding some spectral information are the Schur form A = U T U * - where U is unitary and T is upper triangular - and the singular value decomposition A = UF~V* - where U and V are unitary and ]E is diagonal - (for the latter A does not need to be square). It is interesting to notice that both these forms are computed by a QR-like iteration [4]. The SVD algorithm of Golub-Kahan [3] is indeed an implicit QR-algorithm applied to the Hermitian matrix A * A . When looking at unitary decompositions involving two matrices, say A and B, a similar implicit algorithm was given in [6] and is known as the QZ-algorithm. It computes A = QTaZ* and B = QTbZ* where Q and Z are unitary and T~ and Tb are upper triangular. This algorithm is in fact the QR-algorithm again performed implicitly on the quotient B - 1 A . The corresponding decomposition is therefore also known as the generalized Schur form. This is not the case, though, when considering the generalized singular value decomposition of two matrices, appearing as a quotient B - 1 A or a product B A . In this case the currently used algorithm is not of QR type but of a Jacobi type. The reason for this choice is that Jacobi methods extend to products and quotient without too much problems. The bad news is that the Jacobi algorithm typically has a (moderately) higher complexity than the Q R algorithm. Yet, so far, nobody proposed an implicit QR-like method for the SVD of a product or quotient of two matrices. In this paper we show that, in fact, such an implicit algorithm is easy to derive and that it even extends straightforwardly to sequences of products/quotients of several matrices. Moreover, the complexity will be shown to be lower than for the corresponding Jacobi like methods.
2
IMPLICIT
SINGULAR
VALUE DECOMPOSITION
Consider the problem of computing the singular value decomposition of a matrix A that is an expression of the following type : A = A~.....
A~2. A~1,
(1)
where si = + l , i.e. a sequence of products of quotients of matrices. For simplicity we assume that the Ai matrices are square n x n and invertible, but as was pointed out in [2], this does not affect the generality of what follows. While it is clear that one has to perform left and right transformations on A to get U * A V = E, these transformations will only affect A K and A1 Yet, one can insert an expression Q*Qi = In in between every pair "asi+a A~i *i+1 in (1). If we also define Q K - U and Qo - V, we arrive at the following expression : 9
U * A V = ( Q K9A K,K QK-I)
"
9. .
(f),asl 9(f),as2 k ~ 2 ~ - ~ 2 Q1). \ ~41-'-~1 Q0)
(2)
9
With the degrees of freedom present in these K + 1 unitary transformations Qi at hand, one can now choose each expression -O~*iA- isi Qi-1 to be upper triangular. Notice that the expression .~,O*.AS.iQi_l._,= T~ ~ with Ti upper triangular can be rewritten as 9 Qi 9 A i Q i - l = Ti
forsi
---
1,
Q i*_ l A i Q i
Ti
for si = - 1 .
(3)
A QR-like SVD Algorithm
141
/,From the construction of a normal Q R decompostion, it is clear t h a t , while making the matrix A upper triangular, this "freezes" only one m a t r i x Qi per m a t r i x Ai. The remaining unitary m a t r i x leaves enough freedom to finally diagonalize the matriz A as well. Since meanwhile we computed the singular values of (1), it is clear t h a t such a result can only be obtained by an iterative procedure. On the other hand, one intermediate form t h a t is used in the G o l u b - K a h a n SVD algorithm [3] is the bidiagonalization of A and this can be obtained in a finite recurrence. We show in the next section t h a t the matrices Qi in (2) can be constructed in a finite number of steps in order to obtain a bidiagonal Q*KAQo in (2). In carrying out this task one should try to do as much as possible implicitly. Moreover, one would like the t o t a l complexity of the algorithm to be comparable to - or less t h a n the cost of K singular value decompositions. This means t h a t the complexity should be O ( K n 3) for the whole process.
3
IMPLICIT
BIDIAGONALIZATION
We now derive such an implicit reduction to bidiagonal form. Below ~ / ( i , j ) denotes the group of Householder transformations having ( i , j ) as the range of rows/columns they operate on. Similarly ~(i, i + 1) denotes the group of Givens transformations operating on rows/columns i and i + 1. We first consider the case where all si = 1. We thus only have a product of matrices Ai and in order to illustrate the procedure we show its evolution operating on a product of 3 matrices only, i.e. A3A2At. Below is a sequence of "snapshots" of the evolution of the bidiagonal reduction. Each snapshot indicates the p a t t e r n of zeros ('0') and nonzeros ( ' x ' ) i n the three matrices. First perform a Householder transformation Q~X) 6 ~ ( 1 , n) on the rows of A1 and the columns of A2. Choose Q~I)" to annihilate all but one element in the first column of A1 " X
X
X
X
X
2;
X
2;
2;
X
X
2;
2;
2;
2;
2;
2;
X
2;
2;
2;
X
2;
2;
2;
0
2;
2;
2;
2;
X
X
2;
2;
2;
2;
2;
2;
2;
2;
0
2;
2;
2;
2;
2;
2;
2;
2;
2;
2;
2;
2;
X
2;
0
2;
2;
2;
2;
2;
2;
2;
2;
2;
2;
2;
2;
X
X
0
2;
2;
X
X
.
Then perform a Householder transformation Q~I) 6 T/(1, n) on the rows of A2 and the columns of A3. Choose Q t21)' to annihilate all but one element in the first column of A2 " X
X
2;
X
2;
X
2;
2;
2;
2;
X
2;
2;
X
2;
2; x x x
x x x x
2; x x 2;
2; x x x
x x 2; 2;
0 0 0 0
2; x 2; 2;
2; x x 2;
2; x x x
x x 2; 2;
0 0 0 0
x x x x
x x 2; 2;
x x 2; 2;
x x 2; 2;
.
Then perform a Householder transformation Q0) E ~/(1, n) on the rows of A3. Choose
142
G.H. Golub et al.
Q i 1) t o a n n i h i l a t e Z
Z
At
Z
in the first column
Z
Z
Z
Z
Z
Z
Z
of A3
Z
Z
Z
Z
0
x
x
x
x
0
x
x
x
z
0
x
x
x
x
0
x
x
x
x
0
x
x
x
z
0
x
x
x
x
0
x
x
x
x
0
x
x
x
x
0
x
x
x
x
0
x
z
x
z
0
x
z
x
z
0
z
z
z
x
Notice that three matrices
this third 9
transformation
9
.
yields the same form
also for the product
Z
Z
Z
Z
Z
Z
Z
X
Z
Z
z
z
z
z
z
x
X
x
x
X
0
z
z
x
z
0
z
z
z
x
0
z
z
x
z
0
z
z
z
z
0
z
z
z
z
0
z
z
x
z
0
z
z
z
z
0
x
z
z
z
0
x
z
z
x
0
x
z
x
z
0
x
z
z
z
0
z
z
x
z
0
z
z
z
z
0
z
z
z
z
0
x
x
x
x
0
z
x
x
x
this stage
above).
all b u t o n e e l e m e n t
Z
we are interested
in t h e
This row can be constructed
first row
as t h e p r o d u c t
=
of this product
(indicated
which annihilates
transformation
Q(1) e 7-/(2, n ) o p e r a t i n g
all b u t t w o e l e m e n t s
.
in boldface
of t h e first r o w of A3 w i t h t h e m a t r i c e s
t o t h e r i g h t o f i t , a n d t h i s r e q u i r e s o n l y O ( K n 2) f l o p s . O n c e t h i s r o w is c o n s t r u c t e d find a Householder
of the
on the last (n-
we can
1) e l e m e n t s
'
= o o o]. This transformation is t h e bidiagonalization since
Q(~)'AQ~)=
Now perform columns
Then columns
applied
x
x
0
0
0
0
Z
X
X
X
0
9
9
9
9
0
x
x
x
x
0
X
X
X
X
a Householder
(4)
t o A1 o n l y
transformation
o f A 2 . C h o o s e Q~2) t o a n n i h i l a t e
and
this
completes
Z
X
X
X
X
X
X
27
X
X
X
X
X
X
0
z
z
x
z
0
z
z
z
z
0
z
x
z
x
0
z
x
x
x
0
x
z
x
z
0
0
x
x
z
0
z
x
z
z
0
z
z
z
z
0
0
z
z
z
0
x
x
z
x
0
z
x
x
x
0
0
z
x
z
a Householder
transformation
o f A 3 . C h o o s e Q~2) t o a n n i h i l a t e
first stage
of the
Q~2) E 7 / ( 2 , n ) o n t h e r o w s o f A1 a n d
all b u t t w o e l e m e n t s
X
perform
the
in the second column
.
Q~2) e 7~(2, n ) o n
t h e r o w s o f A2 a n d t h e
all b u t t w o e l e m e n t s in t h e s e c o n d c o l u m n
X
X
X
X
X
X
X
X
X
X
X
X
X
Z
X
0
z
x
x
z
0
z
x
z
x
0
x
z
z
x
0
x
x
x
z
0
0
z
z
x
0
0
x
z
x
0
x
x
x
z
0
0
x
z
z
0
0
z
z
z
0
x
x
z
z
0
0
x
x
x
0
0
z
z
x
the
o f A1
.
o f A2
A QR-like SVD Algorithm
143
Then perform a Householder transformation Q(2) 6 7-/(2, n) on the rows of A3 and choose it to annihilate all but two elements in the second column of A3 " x
2;
z
x
z
2;
x
2;
2~
x
x
x
2;
x
0
z
z
x
x
0
x
x
x
x
0
z
x
z
x
0
0
z
z
z
0
0
z
z
z
0
0
z
z
x
0
0
x
x
x
0
0
z
x
x
0
0
x
z
x
0
0
x
z
z
0
0
x
z
z
0
0
x
x
z
.
For the product we know that 9 2;
X
2;
2;
2;
2;
2;
X
X
2;
z
z
z
z
z
z
z
0
0
0
0
2;
Z
2;
Z
0
Z
2;
Z
Z
0
z
z
z
z
0
x
x
x
x
0
0
2;
2;
2;
0
0
2;
Z
2;
0
0
z
z
z
0
0
z
x
x
0
0
2;
Z
2;
0
0
Z
Z
Z
0
0
z
z
z
0
0
z
z
x
0
0
Z
2;
2;
0
0
Z
Z
Z
0
0
z
z
z
0
0
z
z
x
--
.
At this stage we are interested in the second row of this product (indicated in boldface above). This row can be constructed as the product of the second row of A3 with the matrices to the right of it, and this again requires only O(K(n-1) 2) flops. Once constructed we can find a Householder transformation Q(2) 6 ~ ( 3 , n ) operating on the last ( n - 2) elements which annihilates all but two elements :
[o
9 9 o o].
(5)
This transformation is then applied to A1 only, completing the second step of the bidiagonalization of A :
Q~)'Q~)'AQ(0~)Q~)=
x
x
0
0
0
x
x
0
0 0
0
0
~
~
~
0
0
z
z
x
0
0
z
z
x
It is now clear from the context how to proceed further with this algorithm to obtain after n - 1 stages 9 X
2;
2;
X
X
x
x
x
x
x
x
z
x
x
x
x
z
0
0
0
X
X
X
X
0
z
z
x
z
0
z
z
z
x
0
z
z
0
0
0
0
z
z
z
0
0
z
z
z
0
0
z
x
x
0
0
x
x
0
0
0
0
z
x
0
0
0
z
z
0
0
0
x
x
0
0
0
z
z
0
0
0
0
z
0
0
0
0
z
0
0
0
0
z
0
0
0
0
z
--
0
.
Notice that we never construct the whole product A = A3A2A1, but rather compute one its rows when needed for constructing the transformations Q(00. The only matrices that are kept in memory and updated are the Ai matrices and possibly ~ K and Q0 if we require the singular vectors of A afterwards. of
The complexity of this bidiagonalization step is easy to evaluate. Each matrix Ai gets pre and post multiplied with essentially n Householder transformations of decreasing range. For updating all Ai we therefore need 5Kn3/3 flops, and for updating QK and Q0 we need 2n 3 flops. For constructing the required row vectors of A we need ( K - 1)n3/3 flops. Overall
G.H. Golub et al.
144
we thus need 2 K n 3 flops for the construction of the triangular Ti and 2n 3 for the outer transformations Q K and Qo. Essentially this is 2n 3 flops per updated matrix. If we now have some of the si = - 1 we can not use Householder transformations anymore. Indeed in order to construct the rows of A when needed, the matrices A~"1 have to be trangularized first, say with a QR factorization. The Q R factorization is performed in an initial step. From there on the same procedure is followed, but using Givens rotations instead of Householder transformations 9 The use of Givens rotations allow us to update the triangularized matrices A~"1 while keeping them upper triangular 9 Each time a Givens rotation detroys this triangular form, another Givens rotation is applied to the other side of that matrix in order to restore its triangular form. The same technique is e.g. used in keeping the B matrix upper triangular in the Q Z algorithm applied to B -1A. The bookkeeping of this algorithm is a little more involved and so are the operation counts, which is why we do not develop this here. One shows that when there are inverses involved, the complexity of the bidiagonalization step amounts to less than 4n 3 flops per updated matrix 9
4
COMPUTING
THE SINGULAR
VALUES
The use of Householder and Givens transformations for all operations in the bidiagonalization step guarantees that the obtained matrices 7"/in fact correspond to slightly perturbed data as follows : * Ti = Qi(Ai + ,SAi)Qi-1, si
-
1,
Tj = Q j* - I (
A
j + ~Aj)Qj, sj = - 1 ,
(6)
where
II~Ai[I <_ ecnllAi[I ,
IIQ~Qi - I,~11 <- ed~,
(7)
with e the machine precision and cn, dn moderate constants depending on the problem size n. This is obvious since each element transformed to zero can indeed be put equal to zero without affecting the e bound (see [7], [4]). Things are different with the elements of A since they are not stored in the computer. How does one proceed further to compute the generalized singular values of A ? Once the triangular matrices Ti axe obtained, it is easy and cheap to reconstruct the bidiagonal : ql
s
q2 T~'...'T~
~'T; ~ = B =
01,3
e3
9 9 9
01 ,n
9 On_2) n
9
(8)
en
qn The diagonal elements qi axe indeed just a product of the corresponding diagonal elements of the Tj matrices, possibly inverted 9
qi
8K 9 = tKi i ..."
82 t2ii
81 9t l i i ,
A QR-like SVD Algorithm
145
and the off diagonal elements ei can be computed from the corresponding 2 x 2 diagonal blocks (with index i - 1 and i) of the Tj matrices. It is clear that the qi can be computed in a backward stable way : all errors can e.g. be superposed on the diagonal elements of the first matrix T1. For the errors performed when computing then ei one needs a little more analysis, but it can be shown that the backward errors can be superposed on the corresponding 2 x 2 blocks of Tj, without creating any conflicts or error build up. In other words, the computed bidiagonal corresponds exactly to the bidiagonal of the product of slightly perturbed triangular matrices Tj, who in t u r n s a t i s f y the bounds (6,7). Unfortunately, nothing of the kind can be quaranteed for the elements oid in (8), who are supposed to be zero in exact arithmetic. The best bound we can obtain from the construction of the rows (4,5) is that :
Ioi,jl <- ec,~l]Tg(i, :)11" ]ITK-11] " ..." 117"111, which is a much weaker bound than asking the off diagonal elements of A to be e smaller that the ones on the bidiagonal. This would be the case e.g. if instead we had :
Ioi,j] <_ ecnllTg(i, :)TK-~ . . .Till. Yet, this is the kind of result one would hope for if some singular values of A are small and still have to be computed to a high relative accuracy. These two bounds can in fact be very different when significant cancellations occur between the individual matrices, e.g. if IIAII < < I ] A ~ I I ' . . . " IIA~211" IIA~111. One way to test the performance of this algorithm in cases with very small singular values is to generate powers of a symmetric matrix A = S K. The singular values will be the powers of the absolute values of the eigenvalues of S :
ai(A) = I)q(S)l K, and hence will have a large dynamic range. The same should be true for the bidiagonal of A and the size of the oi,j will then become critical for the accuracy of the singular values when computed from the bidiagonal elements qi, ei. We ran several tests with matrices 5' of which we know the exact eigenvalues and observed a very high relative accuracy even for the smallest singular values. The only explanation we can give for this is that as the bidiagonalization proceeds, it progressively finds the largest singular values first and creates submatrices that are of smaller norm. These then do not really have cancellation between them, but instead the decreasing size of the bidiagonal elements is the result of decreasing elements in each transformed matrix Ai. In other words, a grading is created in each of the transformed matrices. We believe this could be explained by the fact that the bidiagonalization is a Lanczos procedure and that such grading is often observed there when the matrix has a large dynamic range of eigenvalues. The consequence of all this is that the singular values of such sequences can be computed (or better, "estimated") at high relative accuracy from the bidiagonal only ! Notice that the bidiagonalization requires 2 K n 3 flops but that the subsequent SVD of the bidiagonal is essentially free since it is 0(n2).
146
5
G.H. Golub et al.
SINGULAR VECTORS
AND ITERATIVE REFINEMENT
If one wants the singular vectors as well as the singular values at a guaranteed accuracy, one can start from the bidiagonal B as follows. First compute the bidiagonal : B = Q*K A Q o = T~:K . . . . . T~2 . T~ 1,
and then the SVD of B : B = UEV*,
where we choose the diagonal elements of l] to be ordered in decreasing order. We then proceed by propagating the transformation U (or V) and updating each Ti so that they remain upper triangular. Since the neglected elements oi,j were small, the new form : ,,$
^ =
will be upper triangular, and nearly diagonal. This is the ideal situation to apply one sweep of Kogbetliantz's algorithm. Since this algorithm is qudratically convergent when the diagonal is ordered [5], one sweep should be enough to obtain e-small off diagonal dements. The complexity of this procedure is as follows. If we use only Givens transformations we can keep all matrices upper triangular by a subsequent Givens correction. Such a pair takes 4n flops per matrix and we need to propagate n 2 / 2 of those. That means 2n 3 per matrix. The cost of one Kogbetliantz sweep is exactly the same since we propagate the same amount of Givens rotations. We arrive thus at the following total count for our algorithm : 2 K n 3 for triangularizing Ai ~ Ti
2n 3 for constructing Q K and Q0 8n 3 for computing U and V 2 K n 3 for updating Ti ---* ~'i 2 K n 3 for one last Kogbetliantz sweep.
The total amount of flops after the bidiagonalization is thus comparable to 2 Kogbetliantz sweeps, whereas the Jacobi like methods typically require 5 to 10 of those sweeps ! Moreover, this method allows to select a few singular values and only compute the corresponding singular vectors. The matrices Q K and Q0 can e.g. be stored in factored form and inverse iteration can be performed on B to find its selected singular vector pairs, and then transformed back to pairs of A using Q g and Q0.
6
CONCLUDING
REMARKS
The algorithm presented in this paper nicely complements the unitary decompositions for sequences of matrices defined for the generalized QR [2] and Schur decompositions [1]. These decompositions find applications in sequences of matrices defined from discretizations of ordinary differential equations occurring e.g. in two point boundary value problems [8] or control problems [1]. We expect they will lead to powerful tools for analyzing as well as
A QR-like SVD Algorithm
147
solving problems in these application areas. Acknowledgements G. Golub was partially supported by the National Science Foundation under Grants DMS9105192 and DMS-9403899. K. Solna was partially supported by the Norwegian Council for Research. P. Van Dooren was partially supported by the National Science Foundation under Grant CCR-9209349. References
[1] A. Bojanczyk, G. Golub and P. Van Dooren, The periodic Schur form. Algorithms and Applications, Proceedings SPIE Conference, pp. 31-42, San Diego, July 1992. [2] B. De Moor and P. Van Dooren, Generalizations of the singular value and QR decomposition, SIAM Matt. Anal. ~ Applic. 13, pp 993-1014, 1992. [3] G. Golub and V. Kahan, Calculating the singular values and pseudo-inverse of a matrix. SIAM Numer. Anal. 2, pp 205-224, 1965. [4] G. Golub and C. Van Loan, Matrix Computations 2nd edition, The Johns Hopkins University Press, Baltimore, Maryland, 1989. [5] J.P. Charlier and P. Van Dooren, On Kogbetliantz's SVD algorithm in the presence of clusters. Linear Algebra ~ Applications 95, pp 135-160, 1987. [6] C. Moler and G. Stewart, An algorithm for the generalized matrix eigenvalue problem, SIAM Numer. Anal. 10, pp 241-256, 1973. [7] J. H. Wilkinson, The algebraic eigenvalue problem, Clarendon press, Oxford, 1965. [8] R. Mattheij and S. Wright, Parallel stable compactification for ODE with parameters and multipoint conditions, Int. Rept. Argonne Nat. Lab., IL, 1994.
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
,APPROXIMATING
THE
PSVD
149
AND QSVD
S. QIAO
Department of Computer Science and Systems McMaster University Hamilton, Ontario Canada LSS ~K1 qiao@maccs, dcss. mcmaster, ca
ABSTRACT. This paper presents an adaptive method for approximating the SVD of the product AB or the quotient AB -1 of two matrices A and B. Specifically, it computes an approximation of the complete orthogonal decomposition of AB or AB -1. The algorithm is an extension of the ULV decomposition algorithm. KEYWORDS. PSVD, QSVD, complete orthogonal decomposition, ULV decomposition.
1
INTRODUCTION
In this paper we consider the problem of the SVD of the product AB (PSVD) and the quotient AB -1 (QSVD) of two matrices A and B. We present an adaptive method for approximating the PSVD or the QSVD. Specifically, this method computes an approximation of the complete orthogonal decomposition [1] of the product AB or the quotient AB -1 without explicit matrix multiplication or inversion. The algorithm is a generalization of the ULV decomposition algorithm [2]. For example, it computes the decomposition
AB=U(
G E)vH 0 F
(1)
where U and V are orthonormal, G and F are upper triangular, and I[E[[~ + [IF[[~ is approximately the sum of the squares of the "small" singular values of AB. This algorithm can be generalized to any finite number of matrices. This paper is organized as follows. In section 2, we present a generalization of the estimator in [3]. It gives an estimated right singular vector associated with the smallest singular value of AB or AB -1. In sections 3 and 4, we extend the deflation and refinement
150
S. Qiao
schemes in [2] to the cases of A B and A B -1 . Finally, in section 5, we describe the updating procedures. In the following sections, we use 2-norm for all vector and matrix norms and adopt MATLAB notations, e.g., Ai.j,k:l denotes the submatrix of A consisting the entries in rows from i to j and columns from k to I.
2
THE ESTIMATOR
To begin with, we describe the estimator for the right singular vector corresponding to the smallest singular value of the product A B . Without the loss of generality, we assume both A and B are upper triangular and of order n. The estimator consists of two stages: first, it finds a vector d of unit norm such that the norm of the solution z to the system BHAHx = d
is large; second, it solves for v in A B v = x. Then v is an approximation of the right singular vector associated with the smallest singular value of A B and iixJi/ilvJJ is an approximation of the smallest singular value of A B . We only describe the first stage for the second one is straightforward. The procedure of finding d is incremental in that d is determined one element at a time. Let A a n d / } be the order k - 1 leading principal submatrices of A and B respectively, then the kth order leading principal submatrices are:
0
Ak,k
0
Bk,k
where a = Al.k-l,k and b = Bx:k-l,k. Suppose that we have found a ( k - 1)-vector d of unit norm so that the norm of the solution ~ to the system/~H.~iH~ = d is large. Our goal is to increment d to a k-vector of unit norm and, at the same time, make the norm of the solution x to B ( k ) H A ( k ) H x = d large. Because of the unit norm of d, we can assume that d= (sd~
\ s ]
where
Jcj2+lsJ2=l.
(3)
Let ~ be the solution to the triangular system/~H~ = d, then, from (2) and (3), the solution y to B ( k ) H y = d is given by
where c* denotes the complex conjugate of a complex number c and def H ~'--b y.
Similarly, if ~ is the solution to the triangular system ~ H ~ = ~, then the solution to A ( k ) H x = y is
Approximating the PSVD and QSVD
151
where def = aHfc.
In solving a triangular system, we often keep a partial sum vector, for example, in this case /~ de__fAl:k_l,k:nHfc. Consequently, the first component of t3 is ~ and we have the partition
~= where ~ = Al:k_l,k+l:n H fc. Having solved for x, we can update the partial sum vector using def H X , . P -- Al:k,k+l:n - - s p q- ~ A k k + l : n H
Since the partial sum p will affect the solutions in the subsequent steps, we choose c and s so that the norm
I1(;)[I is maximized. After some algebraic manipulation, we obtain x-W1
8
where
Wl = (A;,kB;,k) -~
8
( 01
""
Bk,kAk,kz -7r-B~,k,5
)
'
and W2 = (A~:,kB;,k)-l(Ak,k+l:,~ H
A~c,kB~,k~-(Tr + B;,k6)Ak,k+l:nH).
Thus (c, s) T is the normalized right singular vector associated with the larger singular value of the n • 2 matrix
W=(
wl
Now we derive the estimator for the right singular vector corresponding to the smallest singular value of the quotient A B -1 where, again, both A and B are upper triangular. Similarly, the crucial step is to incremently find a vector d of unit norm such that the norm of the solution z to the system
B -HAHx - d is large 9 Using the partitions in (2) and (3), let ~)= BHd, then
Y=
sb H d + cB~, k
152
S. Qiao
The solution x and its corresponding partial sum p are
and p -- s~ nt- (sbHd nt- cB~, k - s~)Ak,k+l:n H/A*k,k
recalling that i5
=
where
~ =
a g~,.
Denote
we have Ak'k)
B*
k,k
bHd
and W 2 - ( A k ,9k ) -1 ( B k ,9k A k , k + l ' n H
A k 9, k P- nt- ( b g d -
~)Ak,k+l:ng).
Thus we choose c and s so that (c,s) T is the normalized right singular vector associated with the larger singular value of the matrix
w2 3
"
THE DEFLATION
Suppose v is the approximation of the singular vector computed by the estimator described in the previous section. To deflate A B , we first find a sequence of row rotations transforming v into the n t h unit vector en = (0, ...,0,1) T. We then apply the rotations to the columns of B and, at the same time, restore the triangular structure of B by applying a sequence of rotations to the rows of B. These row rotations are propagated to A in order to maintain the product A B . Finally, we apply another sequence of row rotations to restore the triangular structure of A. We can show that AB = URV H
where both U and V are orthonormal and R is nearly block diagonal:
where Ilrll 2 + lal 2 approximates the square of the smallest singular value of A B . If we denote QH as the sequence of rotations applied to the rows of B, then we can write A = URAQ H
and
B = QRBV H
where RA and RB axe resultant triangular matrices and R A R B = R.
(5)
Approximating the PSVD and QSVD
153
The deflation of the quotient A B -1 can be obtained by noticing that B - 1 V = (V HB) -1. Specifically, assuming v is the estimated right singular vector associated with the smallest singular value of A B -1. we first apply the sequence of rotations which transforms v into the nth unit vector to the rows of B and simultaneously restore the triangular structure of B by column rotations. Then, as in the case of the product A B , we propagate these rotations to A and restore the triangular structure of A. Similarly, if we denote Q as the sequence of rotations applied to the columns of B (or the columns of A), then we have
A = URAQ H
and
B = VRBQ H
(6)
where R A R B 1 = R. The deflation procedure can be repeated until the decomposition (1) is achieved depending on a tolerance for the numerical rank. If v is the exact singular vector associated with the smallest singular value amin of, say A B , then [O'min[2-- [[ABv[[ 2 = []r[[2 + [a[ 2 which implies that
On the other hand, since R and A B have the same singular values and a = e nT Re,., , it follows from the minimax theorem for singular values that
This shows that lal = lamJnl and r = 0, when v is the exact singular vector associated with the smallest singular value. Thus an accurate estimate of the singular vector can bring R close to block diagonal. To improve the accuracy of the singular vector approximation v computed by the estimator described in the previous section, we can apply the inverse power method using the estimate v as an initial vector. Alternatively, we can bring R arbitrary close to block diagonal and a arbitrary close to a ~ n using a refinement scheme presented in the following section.
4
THE REFINEMENT
Let C = A B , we first find a rotation on the ( n - 1, n)-plane to eliminate Cn-l,n using Cn-l,n-1. This rotation is determined by the five elements in A and B: A,~-1,,~-1, A,~-I,,~, An,n, Bn-l,r,, and Bn,n, since Cn-l,n = A n - l , n - l B n - l , n + An-l,nBn,n and Cn-l,n-1 = A n - l , n - l B n - l , n - 1 . Then, after applying this rotation to the nth and the n - 1st columns of B, we annihilate the ( n - 1, n)-entry of B using a row rotation on the ( n - 1, n)-plane.
154
S. Qiao
The following figure illustrates the procedure: X
B:
X
X
X
X
X
X
X
X X
X
X
X
X
X
X
X
~
X
|
X
~
X
X
X
~
X ~
X
X
X
X
X
X
X
The above row rotation must be propagated to A in order to maintain the product A B as shown in the following figure: x
A:
x
x
x
X
X
X
X
x
x
x
X
X
X
X
X
X
X
X
X
~
x
Now that both the resultant product A B and the resultant B have the structure: x
x
x
x
x
x
x
(7)
x x
x
it can be verified that the resultant A must have the same structure. This procedure can be repeated until A and B and their product have the following structure: x
x
x
x
x x
x
x
x
x
To restore the triangular structure, we start with applying a sequence of row rotations to triangularize B. These rotations are propagated to A to maintain the product A B . Finally we apply another sequence of left rotations to A to restore its triangular structure. We can prove that if we iterate the above procedure, the off diagonal vector r in (4) converges to zero and a to the smallest singular value of A B . Similarly, in the case of the quotient A B -1 , let C = A B -1 , then the rotation eliminating C,~-l,n is determined by C,~-l,n = -A,~-I,,~-IB,~-I,,~/(B,~-I,,~-IBn,,~)-t- A,~-l,n/Bn,,~ and Cn-l,r,-1 = A , ~ - l , n - 1 / B n - x , n - 1 . Then we apply this rotation to the rows of B and annihilate the ( n - 1, n)-entry of B using a column rotation. Note that if B has the structure (7), then so does B -1. The rest of the refinement procedure for the quotient A B -1 is similar to that for the product A B . Note that all the rotations applied to the columns of A can be accumulated in the matrix Q in (5) or (6).
Approximating the PSVD and QSVD 5
155
UPDATING
In updating the decomposition, we assume the form (5) or (6). In other words, the matrix Q is the product of all the rotations applied to the columns of A. 5.1
ADDING A ROW TO A
When a new row
aT is added to A, updating the decomposition can be achieved by writing
A
0 0
aTQ ) QH.
Then RA is updated by a sequence of rotations eliminating gated to the left and absorbed by U. 5.2
aTQ. The rotations are propa-
ADDING A ROW OR COLUMN TO B
When a row is added to B in the case of the quotient AB -1, the updating is symmetric to the procedure described above. In the case of the product AB, when a column b is added to B, we write
(B b) = Q(RB QHb) ( V H O) 0 1 " Then RB is updated by a sequence of column rotations which is absorbed by V. After updating, we need to apply deflation and possible refinement since the numerical rank may change. A ck n o w l e d g e m e nt s This work is supported in part by an NSERC grant OGP0046301. The author is grateful to C.T. Pan for valuable discussions. References [1] G.H. Golub and C.F. Van Loan. Press, 1989.
Matrix computations. Second Edition, Johns Hopkins
[2] G.W. Stewart. Updating a rank-revealing ULV decomposition. Appl. 46, pp 479-492, 1993.
SIAM J. Matrix Anal.
[3] C. Van Loan. On estimating the condition of eigenvalues and eigenvectors. Algebra Appl. 8 8 / 8 9 , pp 715-732, 1987.
Linear
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
IBOUNDS ON SINGULAR QR FACTORIZATIONS
VALUES REVEALED
157
BY
C.-T. PAN
Department of Mathematical Sciences Northern Illinois University De Kalb, IL 60115-2888 U.S.A. [email protected] P.T.P. TANG
Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Ave. Argonne, IL 60439-4801 U.S.A. [email protected]
ABSTRACT. We introduce a pair of dual concepts: pivoted blocks and reverse pivoted blocks. These blocks are the outcome of a special column pivoting strategy in Q R factorization. Our main result is that under such a column pivoting strategy, the Q R factorization of a given matrix can give tight estimates on any two a priori-chosen consecutive singular values of that matrix. In particular, a rank-revealing Q R factorization is guaranteed when the two chosen consecutive singular values straddle a gap in the singular value spectrum that gives rise to the rank degeneracy of the given matrix. The pivoting strategy, called cyclic pivoting, can be viewed as a generalization of Golub's column pivoting and Stewart's reverse column pivoting. Numerical experiments confirm the tight estimates that our theory asserts. KEYWORDS. Singular value decomposition, rank-revealing QR factorization, cyclic column pivoting. 1
INTRODUCTION
Recently, Hong and Pan [5] showed that for any matrix A E ]Rmxn, m _> n and given k,
C. -T. Pan and P. T.P. Tang
158
1 < k < n, there exists a column permutation 1I such that
k AII = QR = Q ( Rll 0
n-k
(i)
R12 ) R22
'
and an~n(Rll) ~
ak(A),
O'max(R22) ~
ak+l(A),
(2)
where Q E p mxn, QTQ = I, R E pj~x,~ is upper triangular, and
al(A) >_a2(A) _>... >_ an(A) are singular values of A. In particular, if k is the numerical rank of A, i.e., at(A) ::~ ar+l(A) ~ 0. ( [4, 5]), the factorization is rank revealing ([1]). Unfortunately, since the choice of the permutation II in (1) relies upon partially on information of the SVD of A, the proof does not render a practical algorithm to substitute the SVD in determining rank. In this paper, we give an algorithm which identifies the permutation II without using any information on the SVD. We introduce a pair of dual concepts: pivoted blocks and reverse pivoted blocks. They are the natural generalization of what we called pivoted magnitudes and reverse pivoted magnitudes, which are the results of the well-known Golub's column pivoting [3] and a less well-known column pivoting strategy proposed by Stewart [7]. The main result is that a pivoted block Rll (or equivalently a reverse pivoted block R22) ensures (2). This result allows us to devise a column pivoting strategy, which we call cyclic pivoting, that produces the pivoted and reverse pivoted blocks. In particular, if k is the numerical rank of A, the cyclic pivoting strategy guarantees a rank-revealing QR factorization. It should be pointed out that our pivoting strategy is similar to that of Hybrid-III in [2], without using the SVD, but the two methods also are fundamentally different. The rest of the paper is organized as follows. Section 2 establishes bounds on singular values derived from the properties of pivoted and reverse pivoted magnitudes. Section 3 presents our main theorems on pivoted blocks and reverse pivoted blocks. Section 4 presents the cyclic column pivoting algorithm used to find the pivoted blocks. Section 5 presents some numerical experiments used to confirm the theoretical results.
2
PIVOTED
MAGNITUDES
Given a matrix A, there are two well-known pivoting strategies for Q R factorization that produce tight bounds on amax(A) and anon(A) from particular entries in R. Those particular entries of R are important to our subsequent discussion. We make two definitions. Given an m-by-n matrix A, m _ n, let IIi,j be the permutation such that AIIi,j interchanges columns i and j of A. The pivoted magnitude of A, 7/(A), is defined to be the maximum
Singular Values Revealed by QR Factorizations
159
magnitude of the (1, 1) entry of the R factors of AIIl,t, 1 = 1 , 2 , . . . , n, thus: rll
~(A) aef =max
r12
9. 9
tin
r22
999
r2n
r~ 13
Irnl'AIIl,t=Q
.
,l=l,2,...,n
.
i
rnn
Algorithmically, one can think of applying QR factorization with column pivoting [3] to A. Then, the magnitude of r n of the resulting R factor is ~?(A). Clearly,
~?(A) = maxllAejll2,
j = 1,2,...,n,
where ej is the standard basis vector in ~'~. We now define the reverse pivoted magnitude of A, r(A), to be the minimum magnitude of the (n,n) entry of the R factors of AIIt,,~, 1 = 1 , 2 , . . . , n , thus: rll
r12
9. .
rln
9
r2n
r22
r(A)d=dmin
Irnnl'AIIt'n=Q
O
"..
"
,I=1,2,...,n
.
rr~r~
If A is nonsingular, we also have
~(A) = ~ / m ~ ll~}"A-~ll~, 3
as shown in [7], where Stewart calls a related column pivoting strategy the reverse pivoting strategy. The following lemma is not new. The result for r ( A ) i s proved in [7] and [1]. The result for r/(A) is rather straightforward. We therefore only state the results" L e m m a 1 Let A be an m-by-n matrix, m >_n. Then, r/(A) < amax(A) _< V~r/(A),
(3)
(1/~/-Qr(A) < amin(A)_
(4)
and r(A).
Now we consider the Q R factorization
k A=QR=Q
(Rn 0
n-k R12) R22 "
Two related submatrices are important in subsequent discussions: /~11, the (k + 1)-by-(k + 1) leading principal submatrix of R, and R22, the ( n - k + 1)-by-(n- k + 1) trailing principal submatrix of R. The next two lemmas facilitate the discussions to follow.
C. - T. Pan and P. T P. Tang
160
L e m m a 2 If T(Rll) ~ f ~(/~22) for 80me 0 < f ~ 1, then
amin(Rll) <_ak(A) <_ ( q k ( n - k + 1)/f). amin(R11)
(5)
(1/V~). r(R11) _ ak(A) <_ ( q k ( n
(6)
and -
k
+ 1)/f). r(R11).
Proof. From the interlacing properties of singular values, we have amin(R~) _< ak(A) and amax(/~22) ~ ak(A). Furthermore,
~n~n(Rl~) > ~(R~1)/4~ ~> f ?'}(-R22)/ V/~ > f ~n~x(k22)/~/k(~- k + 1) >_ f ak(A)/qk(n- k + 1). Thus (5)is proved. From (5) and Lemma 1, (6) follows easily.
[[
Lemma 3 If T(A~ll) ~ f Tl(R22) for 80me 0 <~ f ~ 1, then (f / q ( n - k)(k + 1)). amax(R22) _< ak+l(A) <_amax(R22),
(7)
(f / q ( k + 1)(n - k)). r/(R22) _ ak+l (A) <_~/n - k . r/(R22).
(8)
and Proof. From the interlacing properties of singular values, we have ak+l(A) <_ amax(R22) and amin(/~ll) _< ak+l(A). Furthermore,
~n~x(R2~) < 4~ - k ~(R~)
< (~/f)~/(~- k),(kl~) _< ( l / f ) 7 ( n -
k)(k + 1)a~n(/~ll)
_< ( l / f ) x / ( ~ - k)(k + 1)~k+~(A). Thus (7)is proved. From (7) and Lemma 1, (8) follows easily.
|
Roughly speaking, Lemma 2 says that if T(.Rll) ~ fT](/~22) with f not too small, then r(Rll) ~ amin(Rll) ~ ak(A);
(9)
and Lemma 3 says that if r(/~11) >_ fr/(R22) with f not too small, then ~(R22) ~ amax(R22) ~ ak+l(A).
(10)
Same results (without f) can be found in [2], where the algorithm "Hybrid-III" is based on Lemmas 2 and 3.
Singular Values Revealed by QR Factorizations 3
PIVOTED
161
BLOCKS
Let
k
n-k
(Rll A = QR = Q 0
R12 ) R22 "
Our main result is that if Rll is a pivoted block, or equivalently, if R22 is a reverse pivoted block (both to be defined momentarily), then (9) and (10) hold simultaneously. By a pivoted block, we mean the following. l = 1 , 2 , . . . , k , and
AIIt k = Q(t)R(t) = Q(l)
k o(0 ~11
'
n
0
kkl
r/(/~
Consider the column permutations IIl,k, r(l)
-- k
r(0
11
12
~(t) ~ "~12 ) = O(t) p(t) ""22
r(t) 22 0
..
r(0
..
r(l) 2~
"" 9
"
"
"
in
for l = 1, 2 , . . . , k,
then we call Rll a pivoted block. Note that, although R i ~ ) = R~21 = 9.. = R~k2), the .```22 fi(') are different in general. Therefore, by itself, the condition r(R~{ )) = ~(k~J2)) for some j, 1 _ j _ k, cannot guarantee a pivoted block Rll. We note here that when k = 1 the pivoted block is r 1,1 (z) and -r/~ (l) = ~(A). Thus one 'lk 22 can view pivoted block as a generalization of the concept of pivoted magnitude defined in Section 3. We also consider pivoted blocks relaxed by a factor f (we call them f-pivoted blocks), 0 < f _ 1, where the equalities above are replaced by the inequalities
rkkl _> f
for 1 = 1 , 2 , . . . , k .
Among other benefits, the f-factor provides us with flexibility in algorithm implementation without sacrificing theoretic rigor. T h e o r e m 1 Let k
A =QR=Q
Rll
0
n-k R12 ) R22 "
If Rll is a pivoted block, then drain(R11) _< ak(A) <_ ~ / k ( n - k + 1) amin(Rll),
and ~ / ( n - k)(k + 1)amax(R22) _< ak+l(A) <_amax(R22).
C. - T. Pan and P. T. P. Tang
162
Moreover, ak(A) and ak+l(A) can be estimated by the quantities r(R11) and y(R22): <_
<_
+
_<
_<
and
+
k))
k
In other words, the conclusions (5), (6), (7) and (8) of Lemmas 2 and 3 are all satisfied with f = 1. More generally, if Rlx is an .f-pivoted block, 0 < .f < 1, then (5), (6), (7) and (8) are all satisfied with the relaxation factor f. In particular, if A has numerical rank k, we have a rank-revealing QR factorization with O'min(.~ll )
~> ( f / ~ / k ( n - k + 1))ak(A)
and
O'max(R22)
<: (~/(n- k)(k + 1 ) / f ) a k + l ( A ) .
(See [6] for a proof). Now we define what we call a reverse pivoted block. Consider the column permutations IIl,k+l, l = k + 1, k + 2 , . . . , n , and k
AIIt,k+I = Q (')R(')
__
II Q(') ( R(t)
0
i k+l,k+l[
r(0 11
n- k "•(0 "12
)
R 22 (0
r(0 12
r 22 (0
(I)
= Q
O
.. 9
r(0 1~
..
r 2(')n
"
"'.
"
for l = k + 1, k + 2 , . . . , n ,
we call R22 a reverse pivoted block. Note that when k = n - 1 the reverse pivoted block is r (0n,n and v ( / ~ ) = r(A). Thus one can also view reverse pivoted blocks as a generalization of the concept of reverse pivoted magnitude defined in last section. We also consider reverse pivoted blocks relaxed by a factor f (we call them reverse fpivoted blocks), 0 < f _ 1, where the equalities above are replaced by the inequalities r r~-`'11 ~ ( 0 ) >- f It(0 k+l,k+l[
for l = k + 1, k + 2 , . . . , n.
It turns out that pivoted blocks and reverse pivoted blocks are dual concepts. T h e o r e m 2 Let
k (Rll A = QR = Q 0
n-k R12 ) R22 "
Rll is a pivoted block if and only if R22 is a reverse pivoted block. More generally, Rll is an f-pivoted block if only if R22 is an reverse f-pivoted block, where 0 < f < 1. (See [6] for a proof).
Singular Values Revealed by QR Factorizations
4
163
ALGORITHM
In the previous section, the main properties of pivoted and reverse pivoted blocks are discussed. Do these pivoted blocks always exist? The following algorithm answers this question positively. We use the notation 7Z(M) to denote the R factor of a matrix M. A l g o r i t h m 1 Cyclic P i v o t i n g . Given k, 1 <_ k < n, and a threshold f, 0 < f <_ 1, this algorithm produces a column permutation II such that AII = QR and Rll is a f-pivoted block. As a result,
~=n(R~l) < ~k(A) < (X/k(~- k + 1)//)~=n(R~), and ( f /~/(n - k)(k + 1)) amax(R22) _< ak+l(A) <_ amax(R22). Step 0. Initialization: R := ~(AII) with (Golub) column pivoting, where II is the column permutation. Set i := k - 1. S t e p 1. Iteration; cyclic pivoting Step 1.1. If i = 0, exit algorithm. S t e p 1.2. Set R := 7~(R. Hi,k); II := II. IIi,k. Step 1.3. If ]rkkl >_ f ~7(//22), then set i := i - 1. Otherwise, perform exchange as follows: Find an l, k + 1 _< l _< n, such that ?](-i~22) ---- II[rk,l, r k + l , l , . . ., r n , l ] T l l 2
Set R "= TC(RIIkj), II := 1I. IIk,l, and i := k - 1. S t e p 1.4. Go back to Step 1.1. The iteration will terminate because whenever an exchange takes place in Step 1.3, the value I det(Rll)l strictly increases by a factor of at least 1/f. Therefore this exchange can happen only a finite number of times. Then, at most k - 1 iterations can take place after the final exchange. Clearly, at termination, Rll is an f-pivoted block. Algorithm 1 produces an f-pivoted block Rll. Likewise we can also propose an algorithm to make R22 a reverse f-pivoted block. For this kind algorithms, see [6] Algorithm 2 and 3. 5
NUMERICAL
EXAMPLES
In this section we present our preliminary numerical results illustrating the theory and algorithms proposed. All computations were done on a SUN SPARC IPC, using MATLAB 3.5i. We only present results of Algorithm 1 here as the results from the other algorithms are similar. Our first experiment tests the ability of the algorithms proposed in detecting the gaps between two consecutive singular values. We applied Algorithm 1 with f = .99 to the following example.
164
C. - T Pan and P. T P. Tang
Table 1: Gap Detection Using Algorithm 1 k
a n n 9n kc~(k) ~ll )
1 2 3 4 5 6 7 8 9 10
80.0453 7.5904 4.3518 1.3744 0.4588 0.1016 0.0763 0.0450 0.0097 *
ak(A) 100.0000 10.0000 8.0000 4.0000 1.0000 0.2000 0.1000 0.0500 0.0100 0.0001
am
[ D(k-1) aXk-'~22 )
10.3392 8.5638 5.0602 2.1754 0.2887 0.2517 0.0804 0.0137 0.0001
E x a m p l e 1 Let A E ]R 12x10 be A=//12
0
(11)
ttlo,
where H,~ = I -
L--eeT , n
with e T = (1,1, ..., 1)., and ]E = diag(100, 10, 8, 4,1, .2, .1, .05, .01, .0001). The numerical results are summarized in Table 1. In Table 1, the superscript of R!~) indicates the corresponding Rll E pJxt. From the results we know that the gaps (100, 10), (1, .2), (0.05, 0.01), and (0.01, 0.0001) are well detected. Acknowledgements The authors wish to thank Pete Stewart for his encouragement in this work. References
[1] T. F. Chan, Rank revealing QR factorizations, Linear Algebra and Its Applications, 88/89:67-82, 1987. [2] S. Chandrasekaran and I. Ipsen, On rank-revealing QR factorizations, Research Report YALEU/DCS/RR-880, 1991. [3] G. H. Golub, Numerical methods for solving least squares problems, Numerische Mathematik 7:206-216, 1965. [4] G. H. Golub, V. Klema, and G. W. Stewart, Rank degeneracy and least squares problems, Report TR-456, Dept. of Computer Science, University of Maryland, 1976.
Singular Values Revealed by QR Factorizations
165
[5] Y. P. Hong and C.-T. Pan, Rank-revealing QR factorizations and the singular value decomposition, Mathematics of Computation 58:213-232, 1992. [6] P. T. P. Tang and C.-T. Pan, Bounds on singular values revealed by QR factorizations, Tech. Report, MCS Argonne National Lab., MCS-P332-1092. [7] G. W. Stewart, Rank degeneracy, SIAM Journal on Scientific and Statistical Computi'ng, 5:403-413, 1984. [8] G. W. Stewart, Determining rank in the presence of error, Tech. Report TR-92-I08, Dept. of Computer Science, University of Maryland, 1993.
This Page Intentionally Left Blank
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
A STABLE ALGORITHM DECOMPOSITION
FOR DOWNDATING
167
THE
ULV
J.L. BARLOW, H. ZHA
Department of Computer Science and Engineering The Pennsylvania State University University Park, PA 16802-6106 U.S.A. [email protected],edu,[email protected] P.A. YOON
Department of Computer Science and Engineering and Applied Research Laboratory The Pennsylvania State University University Park, PA 16802 U.S.A. [email protected],edu ABSTRACT. A common alternative to performing the singular value decomposition is to
fact~176
u ( C )VTwhereUandVare~176176
is a lower triangular triangular matrix which indicates a separation between two subspaces by the size of its columns. These subspaces are denoted by V = (V1,1/2) where the columns of C are partitioned conformally into C = (C1,C2) with II c T IIF~ • ' Here e is some tolerance. If the matrix A results from statistical observations, it is often desired to remove old observations, thus deleting a row from C. In matrix terms, this is called a downdate. A downdating algorithm is proposed that preserve the above block structure in the downdated matrix C. KEYWORDS. Rank revealing decompositions, downdating, chasing algorithms.
1
INTRODUCTION
We give a method for downdating the ULV decomposition. For m x n matrix A where m >__n, the decomposition can be characterized by writing A in the form
A + S A = u ( C V T) O
(1)
168
J.L. Barlow et al.
where ~A is an m x n matrix of errors, U E ~mxm and V E ~nxn are orthogonal, and C E ~nxn is lower triangular and has the form L C=
0
O)
FGO 0 0
L E Nkxk, G E
~(p-k)x(p-k)
lower triangular
(2)
II (F C)liE< e.
0
Here k is the numerical rank of A and e < v / p - k . tol where II L-~ IIq> tol, q = 1, 2 or c~ according to some condition estimator. We use ]]. I]2 to denote Euclidean norm, I]" [IF to denote Frobenius norm, and ]]. ]I2,F to denote expressions that hold in either norm. It is assumed that I] C ]]~ 1 which can be assured by setting I] C liE = 1. The ULV decomposition was described by Stewart [8] who also gives a method for updating it. It is a particular case of what Lawson and Hanson [5] called HRK decompositions. A difference here is that we separate out blocks that are exactly zero.
(wT)
The downdating problem is that of obtaining the ULV decomposition of A where A=
ft.
(3)
and the ULV decomposition of A is known. The opposite computation, updating, consists of finding the ULV decomposition of A from that of ft.. The problem can be transformed into the problem of finding a lower triangular matrix C', orthogonal matrices Cr E ~(n+l)x(n+l), ~" E ~nxn such that
(0)z
There is never a need to explicitly compute b = riTz. The algorithm is posed in terms of C and z. Like the updating routine of Stewart [8] the matrices C , V , and U can be produced using O(n) Givens rotations, thus updating the factorization in O(n2).
The following are the main results of this paper: 1. A blockwise procedure for downdating the ULV decomposition where
g'=
/, o o) P O o 0
0
0
L, G lower triangular
II (F C)II~,r
(5)
where the blocks are conformal with (2). 2. We show that our procedure works when L TL - x z T is positive definite. That is, we can perform a downdate when the full rank portion of C can be downdated. An error analysis showing the favorable stability properties of this algorithm is presented in an expanded version of this paper [2]. The procedures we need to develop our algorithm is described in section two. We then describe our ULV downdating algorithm in detail in section three.
Downdating
2
A CHASING
ROUTINE
169
the ULV Decomposition
FOR A LOWER
TRIANGULAR
MATRIX
In this section we give a simple routine that finds orthogonal matrices 0 , V such that uTcv
?Tz
= C'
lower triangular,
p~, p
=
=11 z I1=.
We call this an l c h a s e operation. We need routine for the formation of a plane rotation ( f o r m r o t ) and the application of one ( a p p l y r o t ) . For instance f o r m r o t ( a , b, c n , s n , c a b s ) produces the plane rotation that maps the vector ( a l , a 2 ) T into ( c a b s , O) T with cabs(a1, a 2 ) -
cabs -
C a ~ + a~; c n -
al/cabs;
sn -
as/cabs;
a p p l y r o t ( v , w , c n , s n , n ) applies a plane defined by f o r m r o t thus modifying the 2 x n matrix wT
according to
wT
*--
--871,
WT
C7~
9
We now describe a simple chasing routine for a lower triangular matrix. Consider the 5 x 5 case below. The right arrows
denote rotations from the left on two particular
rows, whereas the downarrows I I denote rotations from the right on two particular columns. Here z denotes elements in the vector z, c denotes elements in C, and ~ or denotes an element about to be zeroed out.
11 z
z
z
z
11 ~
z
c
z
z
z
0
z
r
c
c
C
C
C
~
C
C
C
C
C
C
C
C
C
z
z
~
0
c
c
c
C
C
C
~
~
C
C
C
C
~
C
C
C
~
c
c
C
C
C
C
C
C
C
C
C
C
C
C
C
C
0
0
C
$$ z
z
z
0
0
z
C
z
z
C
c
c
~
C
C
C
C
C
C
C
C
0
C
C
C
C
C
z
0
0
0
C
c
c
~
C
C
C
C
C
C
C
C
C
C
C
~ ~
C
c
c
=:~
C
C
C
C
C
C
C
C
C
C
C
C
J.L. Barlow et al.
170
z ,~ 0 0 0 c C
C
~
C
C
C
C
C
C
C
C
C
C
C
z 0 0 0 0 c ~:
z 0 0 0 0
~ ~
C
C
c
c
C
C
C
c
c
c
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
C
a
C
We state the procedure lchase formally below.
procedure lchase(z,C,n); (* This procedure performs a chasing operation on C *) for i +-- n - 1 , . . . , 1 formrot (z(i),z(i + 1),cn, sn, z(i));
~ -~,
~(i, i); ~(i, i)
~- ~ ,
~(i,
i);
a p p l y r o t ( c ( i + 1: n , i ) , c ( i + 1: n,i + 1),cn, sn, n - i); formrot (c(i-F 1, i+ 1),e, cn, sn, c(iH- 1, i + 1)); a p p l y r o t ( c ( i + 1,1: i), c(i, 1: i), cn, 8n, i); endfor endlchase Stewart[8] points out that if the matrix C is from a rank revealing decomposition with k "large" rows and n - k "small" rows, this algorithm can yield k + 1 "large" rows, thus the rank revealing nature of C may be lost. 3
A PROCEDURE
F O R ULV D O W N D A T I N G
The following algorithm is introduced for the ULV decomposition. It has the advantage that for C in (2) we obtain II (F G)112,F-----II (F G)112,F for the corresponding blocks of the downdated matrix C'. Our procedure produces a downdated matrix C' if L TL - xx T remains positive definite. A similar assumption is made implicitly by Park and Elden [7] for their URV downdating routine. A l g o r i t h m 3.1 (Procedure for ULV d o w n d a t i n g ) Given a lower triangular matrix C of the form (2) and a vector z = (x T, yT, yo)T x E ~k,y E ~p-k, YO E ~n-p, this algorithm finds a lower triangular matrix C of the form ( 5 ) and orthogonal matrices ~] and V satisfying (3). The components in yo are ignored. This is justified by Proposition 3.2 at the end of this section. Throughout the description of this algorithm L(k) and G (k) denote lower triangular matrices, f~k)= [F(k)]Tel, and g~k)= eT[G(k)]el. 1. ( R e d u c e G a n d y ) Use the procedure lchase in section 2.1 on G and y to produce ~]1, V1 such that
G (1) = ~]TGv"1 lower triangular and so that
Downdating the ULV Decomposition
171
Also, compute
F (1) = ~rTF. 2. ( M a k e glaeT+l a s i n g l e t o n r o w ) Find an orthogonal matrix U'2 such that
0
h) ( =o[
g~
Define F (2) = ( I -
0)
{i~,l} ~ ~
(6)
.
exeT)F(1),that is, as F (1) with its first row zeroed out.
3. ( P e r f o r m D o w n d a t e on L) Use the downdating algorithm in [4] (or alternatives [6]) find a vector a E ~k, scalar c~, and orthogonal matrix ~]3 such that
_
_
_
a
-- e 1
Then do I
xT
4. ( P e r f o r m d o w n d a t e
0
on gll ) Also do
- a -1 (p - h Ta)
if t5 _ g~2) t h e n
else gll 5 -6p
(7)
where 6p = p - (, gll(2) + aTh). Define ~]4 = J ( 1 , k + 1,0) as the plane rotation with cn = cos tg, sn = sin O such that cn = gll /gll or cn = O if g = 0 . Then ~] = diag(Ik+l , ~]l , In_p)diag(1, ~r2, In-p-1)~f 4diag( ~f3, I n - k )
J.L. Barlow et al.
172
5. ( R e s t o r e lower t r i a n g u l a r f o r m ) Find an orthogonal matrix ~d2 such that
I s ~ ) r o ther~ g'=
( 0o) (L3 0 0) P
d~ o
0
0
=
F(3)
0
0
G(4) 0 0
.
0
~r = diag(Ik, ~ , In-p)diag(V2, In-p) Go to step 7 else go to step 6. a.
(Reduce G fu~the~ if ~ingul~) II g~)= 0
th~
S~)= 0
at~o ,~.c~ it ~a~ I o ~ m ~
from g~) using V2. Thus k p-k
(L30) F(3)
G(4)
=
k
1 p- k - 1
k p-k
k
0 0 p ~(4) 1 p-k-1
(~(4) is a lower Hessenberg matrix. We then find an orthogonal matrix fr3 such that
n-k-1
1
If we place the zero row at the end, C has the form
k
C=
p-k-1 n-p+1
( 00)
fl' G 0 0 0 0 k p-k-1 n-p+1
We note here that ~r = diag(Ik, ~ , In_p)diag(V2, In_p)diag(Ik, V3, In-p). 7. (ULV d e c o m p o s i t i o n of L) Perform a ULV decomposition of L to determine its numerical rank. If we have determined that rank of L correctly, the rank of L should be k or k - 1. Make appropriate adjustments to P and G.
Downdating the ULV Decomposition
173
R e m a r k 3.1 We note that (7) in Step 4 is equivalent to computing ~'T (g~21)+(SP)O~ + (~p)a )
(~)=
5P=P--(ctg~21)+aTh)
since ~]T maps the additive noise vector (Sp)(a, aT) T into (So)el 9 Only h is of interest in the computation. satisfies (5) as is stated the following proposition proven in [2]. P r o p o s i t i o n 3.1 For the matrix C resulting from Algorithm 3.1 we have (5). In the absence of rounding error, we can show that the "additive noise" in step 4 and the act of ignoring Y0 actually make the matrix C closer to being orthogonally equivalent to A than C is to A. Again the proof of this result is simple and we give it below. P r o p o s i t i o n 3.2 Assume that Algorithm 3.1 is performed in exact arithmetic, that U and
V are exactly orthogonal, that ~] = U[7 satisfies Uel = el , and that z = VTw and pax = ~Ty are computed exactly. Also let V = Vf" and b = vTw. If
(bT)
then A + 8Ao =
ft. + 8ft.
(9)
0
Thus II 5Ao [12,F=II ,Sfi 112,F<_II5A 112,F. Proof: Let 171 = Vdiag(Ik, Vl,I~_p). Then OT Af/i =
(
A1
Vo)
A2
A3
~]T( 0 )diag(ik ~1 i,~_p)= C
'
'
L (2) F(2) 0
/re T G(a) 0
0 0 0
Thus from (8)II 5w 112=118x 1122+5p2+ II yo [1~. Similarly note that
~'
?T, i,~_p)diag(?T,
Using the fact that fTel = el , we have t/Ao =
F(21 a(al
(0) 5fi~
0
0
0 0
. The result follows. [::l
The downdating algorithm presented here coupled with updating algorithm of Stewart [8] show that the ULV decompositions can be updated and downdated in O(n 2) flops (including the time for updating invariant subspaces) in a manner that preserves their structure. In a
174
J.L. Barlow et al.
report that is an expanded version of this paper we show that the above algorithm satisfies a blockwise stability criterion [2], similar to that in [1]. Acknowledgements The research of Jesse L. Barlow and Hongyuan Zha was supported by the National Science Foundation under grant no. CCR-9201612 (Barlow) and grant no. CCR-9308399(Zha). The research of Peter A. Yoon was supported by the Office of Naval Research under the Fundamental Research Initiatives Program. References
[1] J.L. Barlow, H. Zha, and P.A. Yoon. Efficient and stable algorithms for modifying the singular value decomposition and partial singular value decomposition. Technical Report CSE-93-19, Department of Computer Science and Engineering, The Pennsylvania State University, University Park, PA, September 1993. [2] J.L. Barlow, P.A. Yoon, and H. Zha. A stable algorithm for downdating the ULV decomposition. Technical Report, in preparation. [3] L. l~lden and H. Park. Perturbation analysis of block downdating of a Cholesky decomposition. Numerische Mathematik, to appear, 1994. [4] P.E. Gill, G.H. Golub, W. Murray, and M.A. Saunders. Methods for modifying matrix factorizations. Math. Comp., 28:505-535, 1974. [5] C.L. Lawson and R.J. Hanson. Solving Least Squares Problems. Prentice-Hall, Englewood Cliff, NJ, 1974. [6] C.-T. Pan. A modification to the LINPACK downdating algorithm. BIT, 30:707-722, 1990. [7] H. Park and L. l~lden. Downdating the rank-revealing URV decomposition. Technical Report LiTH-MAT-1992-47, Department of Mathematics, LinkSping University, LinkSping, Sweden, December 1992. [8] G.W. Stewart. Updating a rank-revealing ULV decomposition. SIAM J. Matrix Anal. Appl., 14"494-499, 1993.
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
'THE I M P O R T A N C E O F A G O O D C O N D I T I O N URV A N D ULV A L G O R I T H M S
175
ESTIMATOR
IN THE
R.D. FIERRO Department of Mathematics California State University San Marcos, CA 92096, U.S.A. fierro@th under, csusm, edu
J.R. BUNCH Department of Mathematics University of California, San Diego La Jolla, CA 92093, U.S.A. [email protected] ABSTRACT. The URV and ULV decompositions are promising alternatives to the singular value decomposition for determining the numerical rank k of an m x n matrix and approximating its fundamental numerical subspaces whenever k ~ rain(m, n). In this paper we prove general a posteriori bounds for assessing the quality of the subspaces obtained by two-sided orthogonal decompositions (such as the ULV and URV decompositions). We show that the quality of the subspaces obtained by the URV or ULV algorithm depends on the quality of the condition estimator and not on a gap condition. KEYWORDS. Two-sided orthogonal decompositions, condition estimator.
1
INTRODUCTION
The singular value decomposition (SVD)is a widely used computational tool and is the most reliable tool for detecting near rank-deficiency in a matrix [4, p. 246]. It has important applications, for instance, in matrix approximation, subset selection, spectral estimation, direction of arrival estimation, optimization, rank-deficient least squares, and total least squares. The SVD of A (see [4, w denoted A = UEV T (1)
R.D. Fierro and J.R. Bunch
176 where, for m _> n,
kn-k U=[UkUoU •
'
V=[VkVo]
and
E=
[Ek
0 ] k
0
E0
n-
0
0
m-n
k.
The nonnegative diagonal elements of E, denoted ai, are the singular values of A and are arranged in decreasing order. The numerical rank of A is k. Also, rl = ak+l/O'k, and the "gap" in the singular values of A is large when 1 - '1 is close to 1. For recursive problems where a "noise" subspace must be adaptively estimated, the SVD is viewed as computationally demanding or difficult to update. Therefore, alternative decompositions have been considered which yield the numerical rank, subspace information, or matrix approximations that are nearly as reliable as the powerful but computationally demanding SVD. G.W. Stewart [9, 10] introduced rank-revealing "two-sided" orthogonal (or complete) decompositions, so-called URV and ULV decompositions, as alternatives. While complete orthogonal decompositions have been around for some time (e.g., see [5]), Stewart's technique is quite promising because it is guaranteed to reveal the numerical rank. In this algorithm the rectangular matrix A is preprocessed by a QR factorization, and then condition estimation and plane rotations on both sides are employed to produce a rank revealing decomposition. The URV, and ULV algorithms are designed for the case k ~ min(m, n), where k is the numerical rank of the m • n matrix. For the low-rank case k << min(m, n), more efficient algorithms are available, see [3]. We also mention that a perturbation analysis for two-sided orthogonal (or complete) decompositions is given in [1]. For a URV decomposition of A, there exist orthogonal matrices UR 6 ~mxm and VR 6 ~nxn such that A = URR VT = [Unk URo U~] R [Vm: Vno]T, (2) where kn-k 0 0
G 0
n-k m-n
is upper triangular and k _< n. For a ULV decomposition of A, there exist orthogonal matrices UL 6 ~,n• and VL 6 ~n• such that A = ULL VT = [ULk ULO U~] L [VLk VLo]T (3) where kn-k H 0
E 0
n-k m-n
A Good Condition Estimator in the URV and ULV Algorithms
177
is lower triangular. In [9] it is shown how so-called "left" and "right" iterations may be used to iteratively refine the decompositions (in the sense that the norm of the off-diagonal block decreases). Based on this refinement strategy, error bounds for estimating the singular values of the matrix A are also provided in [6, 9]. Ideally, it would be useful for the user to have some kind of a diagnostic measure to assess the quality of subspaces obtained by a two-sided orthogonal decomposition as compared to the reliable SVD. This has important applications in areas where the SVD might be used. The objectives of this paper are to 1. provide error bounds for the subspaces determined by any two-sided orthogonal decomposition 2. show the importance of a good condition estimator in the high-rank revealing URV and ULV algorithms. In [2] we show how these decompositions may be more accurate alternatives to the SVD than RRQR. The paper is organized as follows. In w we derive a posteriori error bounds for decompositions of the form (2) and (3). These bounds, which are independent of the numerical rank or the algorithm used to compute the decomposition, suggest that if IIHII ~ IIFII, a ULV decomposition may yield a more accurate estimate of the numerical nullspace than a URV decomposition, while the URV decomposition may yield a better estimate of the numerical range. In w our theoretical results show that the quality of the subspaces obtained by Stewart's high-rank revealing URV and ULV algorithms depend on the condition estimator, not on the gap in the singular values. This is demonstrated in numerical simulations in [2]. Finally, I1" II = I1" 112 unless otherwise indicated, C ( I : i, 1: i) denotes the leading submatrix of C of order i, and superscripts t and T denote the pseudoinverse and transpose, respectively. 2
SUBSPACE BOUNDS
It is well known that a singular vector corresponding to a singular value in a cluster is extremely sensitive to small perturbations, but that the span of the singular vectors corresponding to the cluster is well determined, i.e., relatively insensitive to small perturbations. Thus we provide bounds for the error in approximating the span. The following definition defines subspaces associated with the SVD. Definition [7] Let A E Nmxn and let X C Nn and Y C Nm be subspaces of dimension 1. Then X and Y form a pair of singular subspaces for A if
(i) (ii)
AX C Y A T y C X.
T/(Vk) and 7~(Uk) are subspace pairs of dimension k, and Ta,,(Vo) and 7~(Uo) are subspace pairs of dimension n - k, where 7~(C) denote the range of matrix C. 7~(Vo) is termed the
R.D. Fierro and J.R. Bunch
178
numerical nullspace of A, sometimes referred to as the "noise" subspace. T~(Uk) is termed the numerical range of A. For the (nonunique) URV decomposition in (2), there exist matrices Q and P (see [7]) such that Ti(URk + URoQ) and Tt(VRk + VRoP) form a pair of singular subspaces for A with
II[Q P]IIF ---
211FIIF
~(Rk)-
Ilall'
where I1" liE denotes the Frobenius norm and a~n(C) denotes the smallest singular value of the matrix C. If F = 0 then n(URk) and n(VRk) form a pair of singular subspaces for A as well as 7e(URo) and Ti(VRo). If liFII is small then n(Ui) and ~(V~) (i = Rk, RO) nearly form a pair of singular subspaces for A (similar statements can be made for the ULV). However, we wish to determine the "distance" between the subspaces n(V0) and 7~(Va0) (and 7~(Vo) and T2~(VLo)), as well as Ti(Uk) and Ti(URk) (and T~(Uk) and "]~(VLk)). We will need the following definition for the distance between two subspaces. D e f i n i t i o n [4, p.76] Let W = [W1 W2] and Z = [Z1 Z2] be orthogonal matrices where W~, Z~ e Npx(p-q) and W2, Z2 E NTM. If See = 7~()/Yoo)and Se = 7~(Zoo)then
mst(&o, Be)--IIW/Z~ll. 2.1
SUBSPACE BOUNDS FOR THE URVAND SVD
If sin 0 = dist(Soo, Se) then O is the (largest) subspace angle between See and Se. Let s i n 0 v R . - dist(Te(Yo),Te(VRo)) and let sin CuRv - d i s t ( n ( V k ) , n ( V R k ) ) . Based on the definition, it easily follows that
~i~O~Rv =
IIVIVRoil
=
IIVRrkVoli ~ a
si~ r
•
= IIV~rURkii = IIV~k Vkii,
where U~- =- [Uo U • and U~k =_ [UR0 U• However, for our analysis in w a more useful expression for sin CvRv is needed and requires some preliminary work. We will use the following result throughout the paper. L e m m a 1 Given the usual SVD and URV factorizations of A, then sin CuRv = [[uT URo[] = [[UTkUo[[. P r o o f : See [2]. A corresponding lemma can be proven for the ULV decomposition. Now we are ready for the main result of this section. T h e o r e m 1 ( U R V E r r o r B o u n d s ) Let A E R "x'~ have the usual SVD and URV
decomposition, and define ~ = ak+l/ak. Then the distance between the numerical nullspace ze(Vo) ~ d th~ UaV avpro~imat~ .~U~pac~ Ze(VRo). a.d th~ di~ta~ b ~ t ~ . th~ .um~ical range T~(Uk) and its URV estimate Ti(URk) are bounded by: (a)
ilnllll+f~llall < dist(7~(V0), T~(VRo)) <_ -~~ lR l f l' l ,a 2
(b)
dist(7~(Uk)7~(URk)) <
~
,
_
Ilfiillall ~-ilall 2"
k)-- karl
179
A Good Condition Estimator in the URV and ULV Algorithms
P r o o f i See [2]. It can also be shown dist(TC(V0), 7r
_<
IIFII O'min(R/c)- ?llall
and dist(n(Uk), n(Unk)) <_
IIFII ~rk+l
~(n,)
-
~+~
Note that it is possible to find a posteriori upper bounds by using Theorem 1 together with the facts a~n(Rk) <_ ak, ~? <_ 1, and ak+l _ [[G[I. Given the above, the following corollary is immediate. C o r o l l a r y 1 (A Posteriori B o u n d s for the following a posteriori bounds hold:
(~)
tlFtL~,,_ IIRII+I < dist(7C(Vo),
7r
UP.V) Under the assumptions of Theorem 1,
IIFII~mi.(R*)
< .L.ia~)_l;vll~
(b)
dist(7~(Uk) 7~(Unk)) < , L ] IFIIIlall , _ . (Rk)-Ilall 2' These bounds show explicitly that when IIFII is small then the subspaces nearly coincide. In w we discuss a way to achieve a small IIFII for the high-rank case k ~ min(m, n) so that high quality subspaces are obtained. 2.2
SUBSPACE BOUNDS FOR THE ULVAND SVD
In this section we are concerned with the ULVdecomposition. Let A have the usual SVD and ULV as in w Let sin OULV =-- dist(TC(V0), Ti(VLo)) and sin CULV = dist(TC(Uk), Ti(ULk)). After the ULV factorization is complete, we wish to determine upper bounds on the errors in the approximate subspaces. T h e o r e m 2 ( U L V E r r o r B o u n d s ) Let A E R m• have the usual SVD and ULV decomposition, and define ~7= O'k+l/ak. Then the distance between the numerical nullspace Ti(Vo) and the (fLY approximate nullspace ~(VLo), and the distance between the numerical range Ti(Uk) and its ULV estimate 7"r are bounded by: (a)
dist(7~(Vo) ~'~(Vno)) < crk+lllHII , -- a~ia(Lk)-a~+l
(b) ,l~ll~ltj~,l < dist(TC(U~) 7r
< L~'t'~=~o(~) rain L ~ k ) -
k+l
P r o o f : See [2]. As before, it is possible to generate a posteriori upper bounds for the subspace angles in terms of computed ULV factors. The necessary facts are amin(Lk) <_ ak, r/ < 1, and ak+l _< IIEII, as well as Theorem 2. C o r o l l a r y 2 (A Posteriori B o u n d s for the following a posteriori bounds hold:
U L V ) Under the assumptions of Theorem 2,
(a)
IIHIIIIEI] dist(7~(V0), Ti(VLo)) <_ ~,~(Lk)_IIEll2
(b)
IIHIJ t,Ltl+JtEtt
< -- dist(TC(G) ' Ti(ULk)) <_ r ~-i~(L~)tlHlt 2 9
R.D. Fierro and J.R. Bunch
180
This shows a large [IH[I translates to a large sin CULY. In the next section we discuss a way to produce a small IIH[] for the high-rank case k ~ min(m, n). Comparing the a posteriori bounds for the URV and ULV, we may conclude that the ULV can be expected to yield a higher quality estimate of the numerical nullspace than the URV. Tables 1-4 summarize the results of some typical experiments that verify this conclusion.
3
THE IMPORTANCE
OF A GOOD CONDITION
ESTIMATOR
As mentioned earlier, any URV or ULV decomposition may be refined iteratively using orthogonal transformations, cf. [9]. The purpose of refinement procedures is to concentrate less "energy" of R (or L) in the 1,2 (or 2,1) position so as to decouple the matrix as much as possible. Recall from w that when the triangular matrix is decoupled (i.e., the off-diagonal block is a zero matrix) then we have obtained singular subspaces for the matrix, and when the off-diagonal block is small then we have obtained good singular subspace approximations. In [9] it is shown how so-called "left" and "right" ("shiftless" QR) iterations may be used to iteratively reduce the norm of the off-diagonal block, and hence refine the subspaces. Based on this particular refinement strategy, error bounds for estimating the singular values of the matrix A are provided [6, 9]. In this section we show that a small off-diagonal block is achieved in the high-rank revealing URV and ULV algorithms by using a good condition estimator. Now we turn our attention to a brief but important discussion of the algorithms by Stewart. At the i th step of the URV algorithm, we work with the upper triangular matrix
in-i 0
Gi
n-i
and a unit estimate v~.ti of the exact i x 1 right singular vector v i~ corresponding to amin(Ri). Using plane rotations, find an i x i orthogonal matrix Qi such that (Qi)Tvieat = ( 0 , . . . , 0,1) T and RiQ i is upper Hessenberg. Then determine an i • i orthogonal matrix pi such that (Pi)T(RiQi) is upper triangular. Partition the updated triangular matrix by i-1
0
1 n-i
0
Gi
n - i.
Analogously, at the i th step of the ULV algorithm we work with the lower triangular matrix
in-i Li 0 Hi Ei
i n- i
i n corresponding to an~n(Li). and a unit estimate u~s t i of the exact i • 1 left singular vector umi i t = ( 0 , . . . 0,1) T Using plane rotations, find an i x i orthogonal matrix pi such that P i uea and PiLi is lower Hessenberg. Then use plane rotations to determine an i x i orthogonal matrix Qi such that (PiLi)(Qi)T is lower triangular. Partition the updated triangular
A Good Condition Estimator in the URV and ULV Algorithms
181
matrix by
i-1 Hi(Qi) T
Ei
=
1 n-i
hi H~
eli ei
0 Ei
1 n - i.
The following result shows how the accuracy of the estimate ves t i is related to the norm of the subcolumn fi, the approximation of amin(Ri) 2 by r~i, and [[F[[, as well as how the accuracy of the estimate ues t i is related to the size of the subrow hiT, the approximation of ff2n(Li) by li~, and IIH[I. T h e o r e m 3 Using the notation above, let v.ti with unit e-norm denote an estimate of i , the right singular vector of Ri corresponding to a ~ n ( R i ) . If Oun i v denotes the angle VmJn between V eis t and VmJn i , then
Ilfi[I < sin 0/uRv
IIR~II
and
~/r2i
Analogously, using
-
a ~2 n (Ri) < sin 8iUnV.
IIa~ll
-
-
the notation above, let ues with unit 2-norm denote an estimate of Urr~n i ,
the left singular vector of Li corresponding to am_in(Li) 9 If CULY i denotes the angle between and U mi J n , then
U iest
IIh~ll IIL~ll
_< sin r 9
and
~/12i - a ~ 2n ( L i ) i]/il] _< sin r
Moreover,
IIFll IIRII
< _ ~ n - k sin Dmax run v
and
~ <
--
v / n - k sin
where vuR v~j ~ max{O~rRv,..., vURVJak+l~ and w~max U L V -- max{r
./,max
~'ULV
,r "t v'ULVJ"
P r o o f : See [2]. This means good estimates of the right (left) singular vectors of Ri (Li) for i = n , . . . , k + 1 lead to a small IIFII (IIHII). By Corollaries 1 and 2 this means the quality of the subspaces depends on the quality of the estimated singular vectors. Theorem 3 generalizes [11, Theorem 1] , where it is proven that if all the estimates ves i t (or Uest) i are indeed singular vectors, then F = 0 (or H = 0) and the relevant subspaces coincide.
References [1] R.D. Fierro, Perturbation analysis for two-sided (or complete) orthogonal decompositions, PAM Technical Report 94-02, Department of Mathematics, California State University, San Marcos, CA 92096. [2] R.D. Fierro and J.R. Bunch, Bounding the subspaces from rank revealing Two-Sided Orthogonal Decompositions, to appear in SIAM J. Matrix Anal. Appl.
182
R.D. Fierro and J.R. Bunch
[3] R.D. Fierro and P.C. Hansen, Low-rank revealing two-Sided orthogonaldecompositions, PAM Technical Report 94-06, Department of Mathematics, California State University, San Marcos, CA 92096. [4] G.H. Golub and C.F. Van Loan, Matrix computations (Second Edition), John Hopkins University Press, 1989. [5] C.L. Lawson and R.J. Hansen, Solving least squares problems, Prentice Hall, Engelwood, Cliffs, N.J., 1974. [6] R. Mathias and G.W. Stewart, A Block QR algorithm and the singular value decomposition, Lin. Alg. Appl. 182 (1993), pp. 91-100. [7] G.W. Stewart, Error and Perturbation Bounds for Subspaces Associated with Certain Eigenvalue Problems, SIAM Review, 15 (1973), pp. 727-64. [8] G.W. Stewart, An Updating algorithm for subspace tracking, IEEE Transactions on Signal Processing, 40, (1992), pp. 1535-1541. [9] G.W. Stewart, On an algorithm for refining a rank-revealing URV decomposition and a perturbation theorem for singular values, Technical Report CS-TR 2626, University of Maryland, Dept of Computer Science, 1991. [10] G.W. Stewart, Updating a rank revealing ULV decomposition, SIAM J. Matrix Anal. Appl., 4 (1993), pp. 494-499. Lin. Alg. Appl., 28, (1979), pp. 257-278. [11] S. Van Huffel and H. Zha, An e]j%ient total least squares algorithm based on a rank revealing two-sided orthogonal decomposition, Numerical Algorithms, 4, (1993), pp. 101133.
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
L-ULV(A), A L O W - R A N K
REVEALING
183
ULV A L G O R I T H M
R.D. FIERRO
Department of Mathematics California State University San Marcos, CA 92096, U.S.A. fierro @thu nder. csusm, edu P.C. HANSEN
UNI . C, Building 304 Technical University of Denmark DK-2800 Lyngby, Denmark Per. [email protected]
ABSTRACT. We present and analyze an algorithm L-ULV(A)for computing a rankrevealing ULV decomposition. Our algorithm is efficient whenever the numerical rank of the matrix A is much less than its dimensions. We also show that good estimates of the singular vectors, needed in the algorithm, produce a rank revealing form that yields good approximations to the singular subspaces of A. KEYWORDS. ULV decomposition, low-rank problems.
1
INTRODUCTION
Most of the current algorithms for computing rank revealing decompositions are tailored to the high-rank case. However, the are also applications for which the matrix A E ~mxn has low rank, i.e., where the numerical rank is much smaller than min(m,n). Matrices with low rank arise when a relatively small number of parameters suffice to describe a system, and "heavy" oversampling is used to reduce the influence of noise or errors in the data. Examples of such problems are linear prediction problems in spectroscopy [1], [13] and noise reduction in speech processing [6]. The high-rank revealing algorithms are not suited for such problems because they "peel off" all the small singular values one by one. Instead, one needs algorithms that are specially designed for the low-rank case. The main purpose of this paper is to present and analyze
R.D. Fierro and P.C. Hansen
184
an algorithm for computing a low-rank revealing ULV decomposition that produces good estimates of the fundamental numerical subspaces associated with a matrix. If k denotes the numerical rank of A, then we want to compute approximations to the k largest singular values as well as approximate basis vectors for the two k-dimensional subspaces spanned by the first k left and right singular vectors of A. The approximations we seek are obtained directly from a rank revealing ULV decomposition of A, denoted by
A = U L VT '
L = ( LkH E0)"
(1)
Here, UTU = In, v T v = I,~, Lk is a k • k lower triangular matrix whose singular values approximate al through ak, while II(H, E)I I is of the order ak+l. Then the columns of ULk -- (Ul,...,uk) and VLk = (Vl,...,Vk) are the desired approximate basis vectors, and the smaller the norm IIHI], the better the approximations [3], [8]. Here, and throughout the paper, I]" I] denotes matrix and vector 2-norms. 2
THE LOW-RANK
ALGORITHM:
L-ULV(A)
A careful analysis of two different low-rank ULV algorithms, plus some variants of these, was carried out by the authors in [4]. The first algorithm, L-ULV(L), requires an initial ULV factorization, and it is very similar to Stewart's original high-rank algorithm [12] except that it is based on estimates of the largest singular values (instead of the smallest). The second algorithm, L-ULV(A), avoids the initial ULV decomposition and starts off directly with the matrix A. The latter algorithm was found to be the most efficient algorithm in most cases (essentially when the numerical rank is very small compared to m and n), and it takes the following form. ALGORITHM L-ULV(A). Input an m x n data matrix A and rank tolerance r. 1. 2. 3. 4.
Initialize i ~ 1, U ~ Ira, and V ~ In. Compute estimates a est (1) and ~ ~ e(1) s t of the largest singular value and corresponding left singular vector of A. (,,(i) ) While \yes t > r and i < n do Determine a Householder matrix P(0 such that
= (1, o, ..., o) eo v A and update U .-\ o (o/ \ o
o P(o/"
5.
Form Ai ~ (Ii-1
6.
Determine a Householder transformation Q~,i) that annihilates elements i + 1 through n of the ith row of Ai.
7.
Update A ~ , ( / , _ l
8.
0 Qci) 0 Q(0 " Deflate: set i ~ i + 1 and compute estimates Ves t'(0 and U~s) of the largest singular value and left singular vector of A(i: m, i: n).
End of ALGORITHM L-ULV(A).
0 ) andV~_V(',_l
0 )
L-ULV(A), A Low-rank Revealing ULV Algorithm
185
Table 1" Flop counts for computing various factors by means of the L-ULV(A) algorithm and the two SVD algorithms applied to a general m x n matrix with m >_ n. The number Wd(A) is the average work to compute the singular value and vector estimate. Required L
L,V L,U
L, U~ L, U, V L, U1, V
L-ULV(A) 8mnk + 3(m + n)k + (k + 1)Wd(A) 8mnk + 3(m + n)k + 4n2k + (k + 1)Wd(A) 4m2k + 8mnk + 3(m + n)k + (k + 1)Wd(A) 12mnk + 3(m + n)k + (k + 1)Wd(A) 4m2k + 4(2m + n)nk + 3(m + n)k + (k + 1)Wd(A) 12mnk + 3(m + n)k + 4n2k + (k + 1)Wd(A) R-SVD Golub-Reinsch SVD
Required
(m _ 5n/3)
E E, V
2mn 2 + 2n 3 2mn 2 + l l n 3 4m2n + 13n 3 6mn 2 + l l n 3 4m2n + 22n 3 6mn 2 + 20n 3
E,O Z, ~ , ?
(5n/3 _ m >__n) 4ran 2 - 4n3/3 4mn 2 + 8n 3
4m2n + 8mn 2 14mn 2 - 2n 3
4m2n + 8mn 2 + 9n 3 14mn 2 + 8n 3
In Table 1 we summarize the flop counts for our L-ULV(A) algorithm. We compare the flop counts with those of two SVD algorithms, cf. [5, w the Golub-Reinsch SVD algorithm and the R-SVD algorithm. An important consideration is "how much" of the ULV decomposition is required. For example, in many total least squares problems, only k and V are required. Therefore, we give flop counts for various combinations of the factors U, U1, L and V, and compare with the two SVD algorithms. In the table, Wd(A) denotes the average work involved in a deflation step to compute the required singular value and vector estimate. We refer to w for more details about Wd(A), and to w for an evaluation of the algorithm. As mentioned earlier, in [4] we present and analyze ALGORITHM ULV(L),which requires an initial "skinny" QR factorization of A. We note that if the matrix A has Toeplitz structure, then the initial skinny QR factorization of A can be computed much faster by means of specialized algorithms: R and U1 can be computed in m n + 6n 2 and 12mn flops, respectively; see, e.g., [9] (a more accurate R can be computed in 2mn + 6.5n 2 flops by new algorithm by Park & Eld~n [10]). The Toeplitz structure is lost in the R factor. Altogether, the flop count for ALGORITHM L-ULV(L)is mn(6k + 13)+ n2(12k +6) + (k + 1)Wd(L) when A has Toeplitz structure and U1 and V are required explicitly with L. See also [4] for a discussion of an implicit ULV algorithm in which the L matrix is never explicitly computed but kept in factored form.
R.D. Fierro and P.C. Hansen
186
3
THE IMPORTANCE
OF A GOOD SINGULAR VECTOR
ESTIMATE
In this section we prove that a very important part of the algorithms given above is the estimation of the largest singular value and associated singular vector in the ith deflation step. The following theorem reveals the importance of a good singular vector estimate during the ith stage of ALGORITHM L-ULV(A), and readily applies to ALGOttlTHM LULV(L) (see [4] for the proof.) T h e o r e m 1 Let Vest'(i)and west" (i) denote the estimate of the largest singular value a~i) and
associated left singular vector u~i) of A(i: m, i: n) 9 Let Oi denote the angle between.' * e(i) and at u~i). Let the orthogonal matrices P(i) and Q(i) come from ALGORITHM L-ULV(A), and partition the updated P(i)A(i: m, i" n)Q(i ) according to P(i)A(i:m,i'n)Q(i)=
1 n-i (li 0 ) 1 hi L (i) m - i.
(2)
Then aq)~
< sin Oi -
and
-~ <
- 1 - sin 0i
Further,
Ilhill
2 sin Oi
~('j(ij -< 1 - sin Oi It follows that if we based our algorithm on the exact singular vectors of the submatrices A(i: m, i: n) in each step, then we would get H = 0, and Lk would be diagonal and simply consist of the largest k singular values of A. This would give a low-rank "partial" SVD. Further, if we set k = n, Theorem 1 simplifies to an existence proof [5, Theorem 2.5.1] for the SVD. In practice, however, we can only supply an estimate of the singular vector, and the norm of the off-diagonal subcolumn hi in the ith stage directly depends on the accuracy of this estimate--as measured by Oi. When Oi is small, then Ilhil] is correspondingly small and ]lil is a good estimate of a~i). If we assume good singular vector estimates are obtainable, then Theorem 1 serves as a proof for the existence of a rank revealing two-sided orthogonal decomposition of a matrix. The following important theorem for ALGORITHM L-ULV(A) shows that good singular vector estimates yield a small [IHII (the theorem readily applies to ALGORITHM L-ULV(L); see [4] for the proof). T h e o r e m 2 Let Oi denote the angle between U(ie2tand u~i) at the i-th stage of the algorithm, where 1 < i < k. Define 0m~x = max {01,..., Ok). When the algorithm terminates, then []L[[ -
1 - sin0m~x'
L-ULV(A), A Low-rank Revealing ULV Algorithm
187
Theorem 2 proves that the better the singular vector estimates, the smaller the IIHII. Thus, the effort one spends in computing good estimates is paid off in a small off-diagonal block. On the other hand, the norm of this block can always be reduced by block QR iterations (also known as VLYrefinement)as pointed out in [8]. 4
ESTIMATION
OF THE LARGEST SINGULAR VECTOR
We now discuss two algorithms for estimation of the largest singular value Crest and the corresponding left singular vector Uest of an m X n matrix A: the simple power method and the more sophisticated Lanczos bidiagonalization algorithm. The same methods were also proposed in [2] in connection with the low-rank revealing QR algorithm, but here we give a more detailed analysis of these algorithms. Both methods need a good starting vector which has a nonzero component along the largest singular vector, and we propose to use the vector
Ae u0 = i]A el I ,
where
e = ( 1 , . . . , 1) T.
In the rare case that e is a null vector of A, try random vectors until IIAell ~ 0. Our stopping criterion is based on the current singular value estimate. Due to the importance of the singular vector estimate, as proved in Theorem 2, one could also monitor the angle between successive approximate singular vector estimates, but this is computationally more expensive. For convenience, let p and ~ denote the required number of iterations in the POWER and LANCZOS algorithms, respectively. Notice that if p = g = 1 then the two algorithms produce the same estimates. The POWER METHOD only needs multiplications with A and A T plus computations of vectors norms, and a systolic implementation of this is described in [7]. Thus, the algorithm requires pm(4n + 3) flops for a full A, which reduces to pn(2n + 3) if A is triangular. The LANCZOS METHOD requires additional vector operations as well as the computation of the SVD of a bidiagonal matrix. Hence, the algorithm is more expensive than the POWER METHOD and it does not map onto a systolic array. Since we are only interested in computing the largest singular value of A, the LANCZOS METHOD does not need any reorthogonalization! If, on the average, 1.5 iterations are required per singular value, then computation of the singular values in the j t h step requires 22j 2 flops, while accumulation of the left singular vectors (once the algorithm has terminated) requires 3/3 + 4.5/2 flops. The whole algorithm requires about 4gmn + 8gm + 6in + 10g3 flops for a full A, which reduces to 2in 2 + 14in + 10s 3 flops if A is triangular. The average amount of work Wd(') in the various combinations of algorithms is summarized in Table 2, where we have neglected all lower-order terms, including those coming from the fact that the dimensions of L and A are actually reduced in each step (these terms are negligible as long as k << n). In both the POWER METHOD and the LANCZOS METHOD for the symmetric eigenvalue problem, the convergence of the eigenvectors is governed by the square root of the corre-
R.D. Fierro and P.C. Hansen
188
Table 2: The average amount of work Wd(X) in the algorithms for estimating the largest singular value and corresponding vector of the matrix X. Here, p and I denote the average number of power and Lanczos iterations. POWER METHOD
Wd(L) Wd(A)
2p,~2 4pmn
LANCZOS METHOD 2In 2 + 14in + 10/3 4lmn + 81m + 6In + 1013
sponding factor for the convergence of the eigenvalues, cf. [5, w and [11, p. 62 and (12-4-1)]. Hence, the comparison of the POWER and LANCZOS methods from [5, w for computing eigenvalues essentially carries over to the computation of singular vectors. As long as al is well separated from a2, the LANCZOS METHOD converges much faster than the POWER METHOD. If al ~ ... ~ aq, then one should compare the angle between the estimated singular vector and the space s p a n { v l , . . . , %}, and again the LANCZOS METHOD outperforms the POWER METHOD when aq is well separated from aq+l. Hence, we advocate using the LANCZOS METHOD in our low-rank algorithms.
5
EVALUATION OF THE ALGORITHM
When we combine the information in Tables 1, [4, Table 2], and Table 2 we can evaluate for which combinations of m, n and k ALGORITHM L-ULV(A)is more efficient than L-ULV(L) (see [4] for details about L-ULV(L)). The flop counts for L-ULV(L) differ for m _> 1.5 n and n _< m < 1.5 n, and we therefore analyze both cases. We see from Table 2 that for m and n large, the dominating amount of work in both the POWER and LANCZOS METHODS is Wd(L) = 2pn2 and Wd(A) = 2pmn, where p is the number of either power or Lanczos iterations. Then we can derive the approximate dominating terms for L-ULV(L) and L-ULV(A) shown in Table 3, from which we easily find the break-even ratios. We conclude that if the numerical rank k of A is very low, i.e., k << n, then L-ULV(A)is usually more efficient than L-ULV(L). For example, if p = 3 (which is realistic) and if we compute either L or the pair L, V, then L-ULV(A) using the first approach is more efficient whenever n > 10k. In [4, Table 2] we see by comparing dominating terms that the L-ULV(L) algorithm is faster than the two SVD algorithms. In Table 3 we compare the L-ULV(A) algorithm to the SVD algorithms" to the Golub-Reinsch SVD algorithm for 5n/3 >_m >_ n, and to the R-SVD algorithm for m >_ 5n/3. Table 3 shows that in low-rank problems ALGORITHM L-ULV(A) is the faster algorithm. This substantiates the claim that L-ULV decompositions are computationally attractive alternatives to the SVD. Finally, we conclude that approximations to the largest k singular values and vectors can also be computed by means of applying Lanczos bidiagonalization directly to A. This is particularly attractive when A is a structured matrix, and good results are reported in [13]. Unlike our algorithms, however, Lanczos bidiagonalization may experience loss of orthogonality in the orthogonal factors and is not guaranteed to detect the numerical rank.
L-ULV(A), A Low-rank Revealing ULV Algorithm
189
Table 3: Comparison of approximate dominating terms of the flop counts for the low-rank ULV algorithms, and ALGORITHM L-ULV(A) versus the SVD algorithms. The integer p is the number of either power or Lanczos iterations.
Required L L,V L,U L, U~ L, U,V L, U1, V Required L L,V L,U L, U~ L, U, V L, U1,V
Est. Requ.
~,U
:~, Crl :~, (r, ? Est. Requ.
y~, fz r,, i] Y~,G F~,U,V
r~,fh,#
Approx. dom. term (m >_ 1.5 n) L-ULV(L) L-ULV(A) 2ran 2 4mnkp + 8mnk 2mn 2 4mnkp + 8mnk 4m2n - 2ran 2 4m2k + 4mnkp + 8rank 4mn 2 4mnkp + 12mnk 4 m 2 n - 2mn 2 4m2k + 4mnkp + 8rank 4mn 2 4mnkp + 12rank Approx. dom. term (1.5 n > m > n) L-ULV(L) L-ULV(A) 4mn 2 - -v~ n 3 4mnkp + 8mnk 4ran 2 - n 3 4mnkp + 8mnk 4 m 2 n - n3 4ra2k + 4mnkp + 8rank 8mn 2 - ll n3 4mnkp + 12mnk 4m2k + 4mnkp + 4(2m + n)nk 8mn 2 -4mnkp + 12mnk Approx. dom. term (m >_ 5 n/3) R-SVD L-ULV(A) 2mn 2 4mnkp + 8mnk 2mn 2 4mnkp + 8mnk 4m2n 4m2k + 4mnkp + 8rank 6mn 2 4mnkp + 12mnk 4m2n 4m2k + 4mnkp + 8mnk 6mn 2 4mnkp + 12mnk Approx. dom. term (5n/3 >_ m >_ n) Golub-Reinsch SVD L-ULV(A) 4mn 2 - 4n3/3 4mnkp + 8rank 4ran 2 + 8n 3 4mnkp + 8rank + 4n2k 4m2n + 8ran 2 4m2k + 4mnkp + 8rank 14mn 2 - 2n 3 4mnkp + 12mnk 4m2n + +Smn 2 + 9n 3 4m2k + 4mnkp + 8mnk 4mn 2 + 87~3 4mnkp + 12mnk + 4n 2k
L- ULV(A) is faster than L-ULV(L) when n/k > 2p + 4 n/k > 2p+ 4 n/k > 3 ( p + 3)/2 n/k>p+3
~/k > 3(p+ 3)/2 n/k > p + 3 L-ULV(A) is faster than L- ULV(L) when
,~/k > ~(p + 2)/2 n/k n/k n/k n/k
> 4 ( p + 3)/3 > 4(p + 3)/3 >p+3 >p+4
,~/k > 3(p + 3)/4 L-ULV(A) is faster than R-SVD when n/k > 2 p + 4 n/k > 2p + 4 n/k > p + 3
~/k > 2(p+ 3)/2 n/k > p + 3
~/k > 2(p + 3)/3 L-ULV(A) is faster than Golub-Reinsch SVD when n/k > 3 ( p + 2)/2
~/k > (p + 3)/2 =/k > (p + 3)/2 n/k > 1 + p/3
~/k > (p + 4)/3 ~/k > (p+ 4)/2
190
R.D. Fierro and P.C. Hansen
References
[1] H. Barkhuijsen, R. De Beer and D. Van Ormondt. Improved algorithm for noniterative time-domain model fitting to exponentially damped magnetic resonance signals. J. Magnetic Resonance 30, pp 553-557, 1987. [2] T.F. Chan and P.C. Hansen. Low-rank revealing QR factorizations. Num. I, in. AI9. Appl. 1, 33-44, 1994. [3] R.D. Fierro and J.R. Bunch. Bounding the subspaces from rank-revealing two-sided orthogonal decompositions. To appear in SIAM J. Matrix Anal. Appl. [4] R.D. Fierro and P.C. Hansen. Low-rank revealing two-sided orthogonal decompositions. Report UNIC-94-07, UNIoC, September 1994. [5] G.H. Golub and C.F. Van Loan. Matrix Computations (Second Edition), John Hopkins University Press, Baltimore, 1989. [6] S.H. Jensen, P.C. Hansen, S.D. Hansen and J.Aa. SCrensen. Reduction of broad-band noise in speech by truncated QSVD. Report ESAT-SISTA/TR 1994-16J, Dept. of Electrical Engineering, Katholieke Universiteit Leuven, March 1994; submitted to IEEE Trans. Speech Proc. A short version of this paper appears in these proceedings. [7] F. Lorenzelii, P.C. Hansen, T.F. Chan and K. Yao. A systolic implementation of the Chan/Foster RRQR algorithm. IEEE Trans. Signal Processing 42, 2205-2208, 1994. [8] R. Mathias and G.W. Stewart. A block QR algorithm and the singular value decomposition. Lin. Alg. Appl. 182, 91-100, 1993. [9] J.G. Nagy. Fast inverse QR factorization for Toeplitz matrices. SIAM J. Sci. Comput. 14, 1174-1193, 1993. [10] H. Park and L. Eld~n. Fast and accurate triangularization of Toeplitz matrices. Technical Report LiTH-MAT-R-1993-17, University of LinkSping, Dept. of Mathematics, April 1993. [11] B.N. Parlett. The Symmetric Eigenvalue Problem. Prentice-Hall, Englewood Cliffs, N.J., 1980. [12] G.W. Stewart. Updating a rank revealing ULV decomposition. SIAM J. Matrix Anal. Appl. 14, 494-499, 1993. [13] S. Van Huffel, P. Lemmerling and L. Vanhamme. Fast algorithms for signal subspace fitting with Toeplitz matrices and applications to exponential data modeling. These proceedings.
SVD AND SIGNAL PROCESSING, III Algorithms, Architectures and Applications M. Moonen and B. De Moor (Editors) 9 1995 Elsevier Science B.V. All rights reserved.
191
FAST ALGORITHMS FOR SIGNAL SUBSPACE FITTING WITH TOEPLITZ MATRICES AND APPLICATIONS TO EXPONENTIAL DATA MODELING S. VAN HUFFEL, P. LEMMERLING, L. VANHAMME Katholieke Universiteit Leuven Kardinaal Mercierlaan 9~ 3001 Leuven Belgium sabine, vanh uf f el @esat. kule u yen. ac. be
ABSTRACT. Fast methods are presented for solving the following key problem: "Given an L x M Toeplitz matrix T of effective rank K << min{L,M}, find an estimate of its K-dimensional signal subspace". This arises in signal processing and system identification problems, such as exponential data modeling, direction-of-arrival estimation and subspace tracking. The SVD, proven to be a valuable and reliable tool for solving this problem, is computationally too expensive. Replacing this SVD by a Low Rank-Revealing (LRR) two-sided orthogonal decomposition is shown to speed up the computations, up to 10 times. Properties of this factorization are presented, showing that the obtained subspace estimate is exactly the same as the space generated from the start vectors used. I.e. the LRR factorization does not further improve the subspace accuracy and, therefore, iterative methods are more efficient when the rank is fixed. An iterative method, based on Lanczos recursions and block power iterations, is presented and shown to be up to 30 times faster than the SVD computation in an exponential modeling problem encountered in NMR data quantification.
KEYWORDS. Rank-revealing two-sided orthogonal decomposition, orthogonal iteration, Lanczos method, exponentially damped sinusoids, nuclear magnetic resonance data.
1
INTRODUCTION
It is well known that the Singular Value Decomposition (SVD) of a rank K matrix: T
=
U E V * = [ U1 K
with E1
=
U2
U•
M-K
L-M
]
0
E2
0
0
[Vl
V2
K
M-K
]*
d i a g ( a l , . . . , o'g) , E2 = d i a g ( o ' g + l , . . . , O'M), and al _> ... >_ aM
(1)
192
S. Van Huffel et al.
yields a basis V1 of the signal subspace 7Z(V1). L _> M is assumed and * denotes the conjugated transpose. However, since O ( L M 2 T M 3) operations are needed, faster factorizations axe considered here, such as the rank-revealing (Rtt) URV decomposition: T = U T ( K ) v * = [ U1 U2 K M-K
Ux
]
L-M
0
G
[ ~'1
0
0
K
V2
]*
(2)
M-K
where U and V axe unitary, D 6 C KxK and G axe upper triangular. The UI%V decomposition is rank revealing in the sense that the numerical rank K of T is exhibited in (2) from
IIFIl + Ilall
+... +
where ak denotes the kth singular value of T. In the noise-free case, all the singular values a K + l , . . . , a M axe equal to zero in exact arithmetic. However, in a noisy environment, the singular values a g + l , . . . , aM correspond to the noise part of the data, and the numerical rank K has to be estimated. The 1~1~UP~V decomposition was first introduced by Stewart [11] for matrices of which the dimension of the signal subspace is large, i.e. when K is only slightly smaller than M. Recently, it was adapted to the computation of a basis for the signal subspace when K << M, which is referred to as the Low RI~ (LRR) URV decomposition [4]. Alternatively, the Kl~ ULV decomposition can be computed in which T (K) is lower triangular. Here, we restrict ourselves to the 1~1~UI~V decomposition.
2
S I G N A L S U B S P A C E E S T I M A T I O N B A S E D O N A L O W R R U R V DECOMPOSITION
We improve the low RI~ URV algorithm, presented in [4], in the following way. First of all, the initial QI~ factorization of the matrix, T = QR, requiring O(LM 2) operations, is replaced by a fast triangularization method that fully exploits the Toeplitz structure of T and requires only O ( L M + M 2) operations. In the fast algorithms introduced by Bojanczyk et al. [1] and Chun et al. [2], each row of the triangular factor R is computed recursively via updating and downdating steps. However, when the problem is ill-conditioned, these algorithms may produce an inaccurate R factor due to the difficult downdating steps. A method that monitors the conditioning of the downdating problems and chooses a more stable alternative in computing the downdating transformation, and also an algorithm where the computed 1~ factor is refined using the method of corrected semi-normal equations are presented in [9]. These methods can be used for the triangularization of the Toeplitz matrix with a little extra cost for better accuracy when the problem is ill-conditioned. Next, a ttl~ UttV decomposition of the triangular M x M factor R is computed. Instead of estimating the left singular vector Umax of R corresponding to its largest singular value al, as initially proposed by Fierro and applied in [10], we start from estimates Wl of the corresponding right singular vector Vmax. Based on the relation IIRwlll2 ~ al
and
Rwl = tlQ*Qwl,
Signal Subspace Fitting with Toeplitz Matrices
193
for any unitary Q, we choose Q as a product of Givens rotations which reduces wl to the first unit vector el, up to a factor e~r . The application of Q* from the right to R gives an upper Hessenberg form, and by another sequence of rotations, applied to R from the left, the upper triangular form is restored, giving a matrix R0). All the right transformations are accumulated into V which is initialized as an identity matrix. If wl would estimate the largest singular vector Vmax of R exactly, then, as a result of these transformations in exact arithmetic, the first column of the new triangular matrix would become the scaled unit vector ale~e ~r . The same procedure is then applied to the lower principal ( M - 1) • ( M - 1) submatrix of R (1) and after K such steps we have R(K) and ~'. Given R, the computation of R (g) and ~" requires O ( K M 2) operations, and is outlined below. A l g o r i t h m : TURVV
Given. TL•
Toeplitz, its numerical rank K, L >_ M > > K, or a tolerance tol.
Compute T = Q R, using a fast Toeplitz triangularization method [9] Initialize R (1) ~ R ( I ' M , I ' M ) , / ~ 1 ~ R , V (1) ~ I , i ~ l Compute an estimate wl of Vmax(R), [[Wl[[ = 1 While []Riwi[[ > tol or i _ g do compute plane rotations Q(i) such that Q(i).
0 wi
= e~r
1 0
1 M-
i
compute plane rotations p(0 such that R (i+1) = P(O(R(OQ (0) is upper triangular update V: V (i+1) .-- V(i)Q (i) set" Ri+l ~ R(i+l)(i + 1 9M , i + 1 9M) compute an estimate wi+l of Vmax(Ri+l), [[wi+l[] = 1 i*--i+l end while Z ~ i - 1 and V1 = v(K+I)( ", l ' g ) is a basis estimate of the signal subspace T~(V1). The total complexity of the TURVV algorithm is O ( L M + M 2 + K M 2 ) , which is clearly more efficient than the SVD computation provided K < < M. In this paper, the estimates wi are computed by a few iterations, say n, of the power method [5] with start vector s/[[s[[ where sj = ~k(Ri)jk. Alternatively, a Lanczos method could be used as done in section 3. This TURVV algorithm, based on estimates of Vn~x(Ri), ensures a more accurate signal subspace estimate ~ t u r v v than the so-called TURVU procedure, based on estimates of Um~x(Ri) and outlined in [10], as proven below (see [7, Theorem 1] for a proof): T h e o r e m 1. Given a matrix T E C LxM with SVD (1) and LRR URV decomposition (2). Superscripts turvv (resp., turvu) are added to indicate that the LRR URV decomposition is obtained by means of the T U R V V (resp., TURVU) algorithm. I/dist(n(0~u~u), u(U~)) = d i s t ( n ( v ~ r ~ ) , ~(V~)) then dist(T~(V~turvv) , ~(V1)) < --
~+1t u r v u ) dist(n(ff~urvu), T~(V~)) ~ ~K+I dist(n(~turvu), n(V1)) aK
amin(D
The larger the gap a g + l / a g , the closer the TURVV signal subspace estimate V1turvv approaches the desired singular subspace T~(V1), compared to the TURVU estimate, as illustrated in Table l(a). Using the RR ULV decomposition reverse conclusions hold.
S. Van Huffel et al.
194
In practice, the condition dist(7~(uturvu),n(U1)) = dJst(n(~rturvv),T~(V1))is roughly satisfied when the largest left and right singular vectors Umax(Ri) and Vmax(Ri) of the Ri, i = 1 , . . . , K, are equally well estimated. This latter condition is used in Table l(b). This is also clear from the next theorem which proves that the accuracy of the TURVV signal subspace estimate solely depends on the accuracy of the start vectors wi, i = 1 , . . . , K: T h e o r e m 2. Given a matrix T E C L x M with SVD (1) and L R R URV decomposition (2). Using the notations of the T U R V V algorithm, we have:
Moreover, if wi = Vmax(Ri) exactly, i = 1 , . . . , K ,
then (I . I denotes the absolute value)"
] D l= El, F = 0, G = RK+x; V1 = V1 and Ux = U1, up to a factor ~. See [7, Theorems 2 and 3] for proofs. The appearance of (I) in Theorem 2 is due to the fact that the singular vectors of a complex matrix T are only defined up to a scaling factor of the form ear A similar theorem can be proven for TURVU. Theorem 2 clearly shows that the LRR URV decomposition is not able to improve the initial signal subspace estimate based on the computed start vectors wi. This is even more clear when one estimates the K first right singular vectors of T corresponding to a l , . . . , aK at once and then uses the columns of this start matrix, say Wstart, one by one as estimates of Vmax(Ri), i = 1 , . . . , K. In this case, the transformations Q(i), applied to the consecutive R (i) during formation of the LRR URV decomposition, must also be applied simultaneously to Wstar t. Under these conditions, it is easy to prove [7, Theorem 4] that
~ ? ~ v = W~t~rta = W~t~rtdi~g(~-~,...,
~-~')
i.e., the LRR URV subspace estimate is exactly the same as the space generated by the start vectors used for the LRR URV computation. Moreover, this decomposition is also costly and at most one order of magnitude faster than the SVD. Further refining the obtained LRR URV decomposition, as described in [8], improves the signal subspace accuracy but this refinement is also costly, especially when K is not very low. Therefore, the use of the TURVV method for signal subspace fitting is only recommended in those applications where the rank of T must be revealed (e.g. when K is not fixed) or whert this subspace needs to be up- or downdated continuously. Note that all theorems presented here apply for general (not necessarily Toeplitz) matrices. If K is fixed, which is often the case in NMR data quantification, then an iterative method is mostly a more efficient alternative. 3
ITEttATIVE
SIGNAL SUBSPACE ESTIMATION
A basis of the signal subspace 7~(V1) can be estimated efficiently by first performing K iterations of the Lanczos bi- (or tridiagonalization) method [5] on T and then further refining this estimate by applying n more iterations of the block power method [5], as outlined below.
Signal Subspace Fitting with Toeplitz Matrices
195
A l g o r i t h m : ITER G i v e n . T E C L x M Toeplitz of numerical rank K, L > M > > K and the number n of desired block power iterations 9 al
Z2
0
1. Apply K Lanczos recursions: Bi = U(O*TV(O =
, i=l,...,Kto 0
CXi
obtain the first K right Lanczos vectors V(K) (making use of the F F T when computing matrix-matrix products of the form T X or T ' X ) . 2. Refine V (g) by n iterations of the block power method. For i = K + 1 , . . . , K + n do V (i) ~ T*(TV6-1)) making use of the F F T twice. orthonormalize V (i) using a modifed Gram-Schmidt method end for V (g+n) is a basis estimate for the signal subspace T~(V1). Orthogonality of the generated Lanczos vectors in step 1 is only guaranteed in exact arithmetic while it fails to hold in inexact arithmetic when spurious singular values of T start to appear in Bi. The cure is to use selective or complete reorthogonalization or to identify the spurious singular values [5]. This algorithm performs best if N = 2p is a power of 2. If not then 2p - N zeros should be appropriately inserted in the original data sequence. The benefits of the use of the F F T are already apparent from N = 32 on. The total complexity of the algorithm ITER (including complete reorthogonalization in step 1 as done in this paper) is O(nKp2 p + n M K 2) flops. 4
APPLICATIONS
The efficiency and accuracy of the presented methods TURVV and ITER are compared with the classical SVD in the following exponential data modeling problem encountered in NMR spectroscopy 9 Suppose N uniformly sampled data points Yn are given and are to be fitted to the following model function: K
K
Yn = ~ CkZ~n+6) = Z(ake~r k=l
(-dk+~2~rfk)(n+6)/xt, n = 0 , . . . , N -
1
(3)
k=l
where K is the model order, At is the constant sampling interval, and ~At is the time lapse between the effective time origin and the first sample. The objective is to estimate the frequencies fk, damping factors dk, amplitudes ak, and phases Ck, k = 1 , . . . , K. To take advantage of the presented fast Toeplitz algorithms, the parameters are estimated as follows. First arrange the N data samples y,~ in an L x M Toeplitz matrix T, N = L + M-.. 1, with first column [YL-1,..., Yo]T and first row [YL-1,..., YN-1], then estimate a basis V1 of the K-dimensional right singular subspace 7~(V1) of T. From V1, a set of linear
196
S. Van Huffel et al.
400 350
4
3
30O
l
25O
5
50
1500
O'i/
0.1 0.2 0.8
a K.t.1
21.3119 9.9642 2.2053
(b)
000
ratio 24.55 12.45 10.87
~00
0
k 4
5
- S O0
method HSVD TURVU TUttVV HSVD TURVU TURVV
- 1000
- XSO0
fk(Hz) -2.534-9.3 -2.374-10 -2.654-9.8 7.864-19 8.154-21 7.824-20
k
A
d~
1 2 3 4 5
-1379 -685 -271 353 478
208 256 197 117 808
-2000
dk(rad/s) 38.94-69 57.94-110 48.14-90 31.94-119 62.84-240 40.54-144
ak Ck 6.1 9.9 6.0 2.8 17.0
15 15 15 15 15
(a) ak(a.u.) 0.744-1.21 1.104-2.62 0.894-1.55 -0.434-1.87 -0.534-2.03 -0.494-1.92
Ck (deg.) 7.414-20 8.684-21 8.154-21 -2.394-6.7 -3.68:J::10 -2.864-7.8
(c) Table 1: (a) FFT and exact parameter values of a simulated 31p NMR signal modeled as in (3). Ck=r (b) Numerical illustration of Theorem 1 using the first 128 data points of (a) with added white Gaussian noise of variance 2a~ and arranged in a 65 • 64 matrix. The start vectors wi used in the algorithms TURVV and TUI~VU are such that the stated condition is satisfied, ratio=dist(T~(~'lturvu), T~(V1))/dist(T~(v~urvv), T~(V1)). (c) Bias-t-standard deviation values of the parameter estimates of peaks 4 & 5 when noise with au = 1.5 is added, a.u.=arbitrary units. Start vectors wi for TUttVV and TUttVU are equally well computed via 5 power iterations. HSVD, TUItVV and TUttVU failed to resolve peaks 4 and 5 in respectively 4%, 7% and 7% of the 1000 runs performed.
equations VITX: ~, V1~ is derived where ~-T (resp., ~1) consists of the top (resp., bottom) M - 1 rows of V1. From the least squares (LS) solution X*ls of this set, estimates of the nonlinear parameters are derived from the eigenvalues "zk = e(--dk+32~r'fk)A~ of Xls. Then, the linear parameters Ck, which contain the amplitudes ak and phases Ck, are estimated as the LS solution of a set A c ,,~ b obtained by fitting the N model equations (3) to the data samples y,~ (and replacing each zk by its estimate zk). It is easy to show that this method is equivalent to Kung et al.'s state-space based method [6], called HSVD in NMR spectroscopy [3], which arranges the data in a Hankel matrix H and uses the K left singular vectors U1 of H. An improved variant, based on total least squares, is presented in [12]. Numerical tests performed with Matlab show that the increased accuracy of the computed TURVV signal subspace estimates -as proven in Theorem 1- also leads to a similar increase in the accuracy of the derived signal parameter estimates (in terms of bias and variance), as compared to TURVU (see Table l(b)). At comparable parameter accuracy, it is shown that TURVV (resp., ITEP~) computes the estimate T~(V1) of the signal subspace typically 8 to 9 (resp., 20 to 30) times faster than HSVD for reasonable signal-to-noise ratios. For smaller ratios (around 0 dB) the gain
Signal Subspace Fitting with Toeplitz Matrices
5000 4500
4000
.
.
.
.
original signal HTLS residual HSVD residual
.
.
.
-:
.
.
] I
3500
K 9 10 11 15 16
300O
1500
~'r NAA
1000
500
197
400
300
200
100
0
-100
-200
-300
-400
Rturvv
niter
1 1 1 4 6
2 2 2 4 5
9.3 8.3 7.6 4.2 3.4
30.3 26.0 22.8 10.7 8.7
-500
Frequency(Hz)
Figure 1: Ratio of the number of flops (in Matlab) needed by HSVD to that required by the algorithms TURVV and ITER (including parameter estimation) for fitting, reconstructing and removing the water peak (> - 70 Hz) of the 1H NMR signal (full line) prior to estimating the peaks of interest in the residual signal (dashed line). K denotes the chosen model order and n denotes the number of power iterations used in TURVV and ITER to achieve comparable parameter accuracy. The water peak (with modulus up to 40.000) has been cut off in order to visualize the small peaks of interest. N = 256 and L = M + 1 = 129. in efficiency is less pronounced. This subspace computation is the computationally most intensive part of the estimation procedure. As an example, consider again the simulated alp NMR signal, as described in Table 1. For 0.5 _ a~ _< 2, this subspace calculation represents 98% of the computational work performed by the HSVD algorithm, while the execution of TURVV (using 2 power iterations) represents 88% and that of ITER (with n = 2 to 4) only 70 to 80% of the work involved in estimating the parameters. Efficiency is of primary importance in medical MR spectroscopic imaging when disturbing water/fat contributions are to be removed from the signals in order to produce high-quality metabolite images [3]. Typically, these peaks can be well fitted by very few exponentially damped sinusoids (i.e. K << M). A single metabolite image may require as many as 32 • 32 = 1024 times removal of unwanted signals. In these applications, the fast algorithms presented here yield significant time savings, as shown in Figure 1. Acknowledgements We wish to thank Rick Fierro for introducing us the LRR URV/ULV factorizations and providing us Matlab code of the LRR URV algorithm based on estimates of Um~x(Ri). Beyond this, the work described here was performed independently of the results reported in [4], since that report was not available to us. We are also grateful to Haesun Park, Lars Eld~n and Jim Nagy for providing us part of the Matlab code. The first author is a Research Associate with the N.F.W.O. (Belgian National Fund for
198
S. Van Huffel et al.
Scientific Research). This paper presents research results of the Belgian Programme on Interuniversity