LINEAR ALGEBRA AND LINEAR OPERATORS IN ENGINEERING with Applications in Mathematica
This is Volume 3 of PROCESS SYSTEMS ENGINEERING A Series edited by George Stephanopoulos and John Perkins
LINEAR ALGEBRA AND LINEAR OPERATORS IN ENGINEERING
with Applications in Mathematica H. Ted Davis Department of Chemical Engineering and Materials Science University of Minnesota Minneapolis, Minnesota
Kendall T. Thomson School of Chemical Engineering Purdue University West Lafayette, Indiana
ACADEMIC PRESS An Imprint ofEbevier
San Diego
San Francisco
New York
Boston
London
Sydney
Tokyo
This book is printed on acid-free paper,
fe/
Copyright © 2000 by ACADEMIC PRESS All Rights Reserved, No part of this publication may be reproduced or transmitted in any fonn or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without pennission in writing from the publisher. Permissions may be sought directly frotn Elsevier's Science and Technology Rights Department in Oxtbrd, UK. Phone: (44) 1865 843830, Fax: (44) 1865 853333, e-mail:
[email protected]. You may also complete your request on-line via the Elsevier homepage: hllp://www.elsevier.com by selecting "Customer Support" and then "Obtaining Permissions".
Academic Press An Imprint of Elsevier 525 B Street, Suite 1900, San Diego, CaUfomia 92101-4495, USA http://www.academicpres$.com
Academic Press Harcouit Place, 32 Jamestown Road, London NWl 7BY, UK http;//www.hbuk.co.uk/ap/ Library of Congress Catalog Card Number: 00-100019 ISBN-13:978-0-12-206349-7 ISBN-10:0-12-206349-X PRINTED IN THE UNITED STATES OF AMERICA 05 QW 9 8 7 6 5 4 3 2
CONTEKTS
PREFACE
xi
I Determinants 1.1. Synopsis 1 1.2. Matrices 2 1.3. Definition of a Determinant 3 1.4. Elementary Properties of Determinants 6 1.5. Cofactor Expansions 9 1.6. Cramer's Rule for Linear Equations 14 1.7. Minors and Rank of Matrices 16 Problems 18 Further Reading 22
2 Vectors and Matrices 2.1. 2.2. 2.3. 2.4.
Synopsis 25 Addition and Multiplication The Inverse Matrix 28 Transpose and Adjoint 33
26
VI
CONTENTS
2.5. Partitioning Matrices 35 2.6. Linear Vector Spaces 38 Problems 43 Further Reading 46
3 Solution of Linear and Nonlinear Systems 3.1. Synopsis 47 3.2. Simple Gauss Elimination 48 3.3. Gauss Elimination with Pivoting 55 3.4. Computing the Inverse of a Matrix 58 3.5. LU-Decomposition 61 3.6. Band Matrices 66 3.7. Iterative Methods for Solving Ax = b 78 3.8. Nonhnear Equations 85 Problems 108 Further Reading 121
4 General Theory of Solvability of Linear Algebraic Equations 4.1. Synopsis 123 4.2. Sylvester's Theorem and the Determinants of Matrix Products 124 4.3. Gauss-Jordan Transformation of a Matrix 129 4.4. General Solvability Theorem for Ax = b 133 4.5! Linear Dependence of a Vector Set and the Rank of Its Matrix 150 4.6. The Fredholm Alternative Theorem 155 Problems 159 Further Reading 161
5 The Eigenproblem 5.1. Synopsis 163 5.2. Linear Operators in a Normed Linear Vector Space 5.3. Basis Sets in a Normed Linear Vector Space 170 5.4. Eigenvalue Analysis 179 5.5. Some Special Properties of Eigenvalues 184 5.6. Calculation of Eigenvalues 189 Problems 196 Further Reading 203
165
CONTENTS
VII
6 Perfect Matrices 6.1. Synopsis 205 6.2. Implications of the Spectral Resolution Theorem 206 6.3. Diagonalization by a Similarity Transformation 213 6.4. Matrices with Distinct Eigenvalues 219 6.5. Unitary and Orthogonal Matrices 220 6.6. Semidiagonalization Theorem 225 6.7. Self-Adjoint Matrices 227 6.8. Normal Matrices 245 6.9. Miscellanea 249 6.10. The Initial Value Problem 254 6.11. Perturbation Theory 259 Problems 261 Further Reading 278
7 Imperfect or Defective Matrices 7.1. Synopsis 279 7.2. Rank of the Characteristic Matrix 280 7.3. Jordan Block Diagonal Matrices 282 7.4. The Jordan Canonical Form 288 7.5. Determination of Generalized Eigenvectors 294 7.6. Dyadic Form of an Imperfect Matrix 303 7.7. Schmidt's Normal Form of an Arbitrary Square Matrix 7.8. The Initial Value Problem 308 Problems 310 Further Reading 314
8 Infinite-Dimensional Linear Vector Spaces 8.1. Synopsis 315 8.2. Infinite-Dimensional Spaces 316 8.3. Riemann and Lebesgue Integration 319 8.4. Inner Product Spaces 322 8.5. Hilbert Spaces 324 8.6. Basis Vectors 326 8.7. Linear Operators 330 8.8. Solutions to Problems Involving fe-term Dyadics 8.9. Perfect Operators 343 Problems 351 Further Reading 353
336
304
VIII
CONTENTS
9 Linear Integral Operators in a Hilbert Space 9.1. Synopsis 355 9.2. Solvability Theorems 356 9.3. Completely Continuous and Hilbert-Schmidt Operators 9.4. Volterra Equations 375 9.5. Spectral Theory of Integral Operators 387 Problems 406 Further Reading 411
366
10 Linear Differential Operators in a Hilbert Space 10.1. Synopsis 413 10.2. The Differential Operator 416 10.3. The Adjoint of a Differential Operator 420 10.4. Solution to the General Inhomogeneous Problem 426 10.5. Green's Function: Inverse of a Differential Operator 439 10.6. Spectral Theory of Differential Operators 452 10.7. Spectral Theory of Regular Sturm-Liouville Operators 459 10.8. Spectral Theory of Singular Sturm-Liouville Operators 477 10.9. Partial Differential Equations 493 Problems 502 Further Reading 509 APPENDIX
A.l. Section 3.2: Gauss Elimination and the Solution to the Linear System Ax = b 511 A,2. Example 3.6.1: Mass Separation with a Staged Absorber 514 A.3. Section 3.7: Iterative Methods for Solving the Linear System Ax = b 515 A.4. Exercise 3.7.2: Iterative Solution to Ax = b—Conjugate Gradient Method 518 A.5. Example 3.8.1: Convergence of the Picard and Newton-Raphson Methods 519 A.6. Example 3.8.2: Steady-State Solutions for a Continuously Stirred Tank Reactor 521 A.7. Example 3.8.3: The Density Profile in a Liquid-Vapor Interface (Iterative Solution of an Integral Equation) 523 A.8. Example 3.8.4: Phase Diagram of a Polymer Solution 526 A.9. Section 4.3: Gauss-Jordan Elimination and the Solution to the Linear System Ax = b 529 A. 10. Section 5.4: Characteristic Polynomials and the Traces of a Square Matrix 531 A. 11. Section 5.6: Iterative Method for Calculating the Eigenvalues of Tridiagonal Matrices 533 A. 12. Example 5.6.1: Power Method for Iterative Calculation of Eigenvalues 534
CONTENTS
IX
A. 13. Example 6.2.1: Implementation of the Spectral Resolution Theorem—Matrix Functions 535 A.M. Example 9.4.2: Numerical Solution of a Volterra Equation (Saturation in Porous Media) 537 A.15. Example 10.5.3: Numerical Green's Function Solution to a Second-Order Inhomogeneous Equation 540 A.16. Example 10.8.2: Series Solution to the Spherical Diffusion Equation (Carbon in a Cannonball) 542 INDEX
543
This Page Intentionally Left Blank
PREFACE
This textbook is aimed at first-year graduate students in engineering or the physical sciences. It is based on a course that one of us (H.T.D.) has given over the past several years to chemical engineering and materials science students. The emphasis of the text is on the use of algebraic and operator techniques to solve engineering and scientific problems. Where the proof of a theorem can be given without too much tedious detail, it is included. Otherwise, the theorem is quoted along with an indication of a source for the proof. Numerical techniques for solving both nonlinear and linear systems of equations are emphasized. Eigenvector and eigenvalue theory, that is, the eigenproblem and its relationship to the operator theory of matrices, is developed in considerable detail. Homework problems, drawn from chemical, mechanical, and electrical engineering as well as from physics and chemistry, are collected at the end of each chapter—the book contains over 250 homework problems. Exercises are sprinkled throughout the text. Some 15 examples are solved using Mathematica, with the Mathematica codes presented in an appendix. Partially solved examples are given in the text as illustrations to be completed by the student. The book is largely self-contained. The first two chapters cover elementary principles. Chapter 3 is devoted to techniques for solving linear and nonlinear algebraic systems of equations. The theory of the solvability of linear systems is presented in Chapter 4. Matrices as linear operators in linear vector spaces are studied in Chapters 5 through 7. The last three chapters of the text use analogies between finite and infinite dimensional vector spaces to introduce the functional theory of linear differential and integral equations. These three chapters could serve as an introduction to a more advanced course on functional analysis. H. Ted Davis Kendall T. Thomson
Xi
This Page Intentionally Left Blank
I
DETERMINANTS
I . I . SYNOPSIS For any square array of numbers, i.e., a square matrix, we can define a determinant—a scalar number, real or complex. In this chapter we will give the fundamental definition of a determinant and use it to prove several elementary properties. These properties include: determinant addition, scalar multiplication, row and column addition or subtraction, and row and column interchange. As we will see, the elementary properties often enable easy evaluation of a determinant, which otherwise could require an exceedingly large number of multiplication and addition operations. Every determinant has cofactors, which are also determinants but of lower order (if the determinant corresponds to an n x n array, its cofactors correspond to (n — 1) X (n — 1) arrays). We will show how determinants can be evaluated as linear expansions of cofactors. We will then use these cofactor expansions to prove that a system of linear equations has a unique solution if the determinant of the coefficients in the linear equations is not 0. This result is known as Cramer's rule, which gives the analytic solution to the linear equations in terms of ratios of determinants. The properties of determinants established in this chapter will play (in the chapters to follow) a big role in the theory of linear and nonlinear systems and in the theory of matrices as linear operators in vector spaces.
CHAPTER I DETERMINANTS
1.2. MATRICES
A matrix A is an array of numbers, complex or real. We say A is an m x ndimensional matrix if it has m rows and n columns, i.e..
-'22
^23
-*32
'*33
^m2
•*m3
^3n
(1.2.1)
The numbers a,y (/ = 1 , . . . , m, 7 = I , . . . , w) are called the elements of A with the element a^j belonging to the ith row and 7th column of A. An abbreviated notation for A is
Ki-
ll.2.2)
By interchanging the rows and columns of A, the transpose matrix A^ is generated. Namely,
A^ =
(1.2.3)
The rows of A^ are the columns of A and the iji\\ element of A^ is a^,, i.e., {^)ij = ciji' If A is an m X n matrix, then A^ is an n x m matrix. When m = n, we say A is a square matrix. Square matrices figure importantly in applications of linear algebra, but non-square matrices are also encountered in common physical problems, e.g., in least squares data analysis. The m x 1 matrix
•^2
X =
(1.2.4)
and the 1 x m matrix y'^ = [ j i , - . . > J n ]
(1.2.5)
are also important cases. They are called vectors. We say that x is an m-dimensional column vector containing m elements, and y^ is an n-dimensional row vector containing n elements. Note that y^ is the transpose of the n x 1 matrix y—the ndimensional column vector y.
DEFINITION OF A DETERMINANT
If A and B have the same dimensions, then they can be added. The rule of matrix addition is that corresponding elements are added, i.e., A + B = [a^j] + [bij] = [a^j -f b^j]
^21 + ^21
^22 + ^22
, ^ml + ^ m l
^ml '^ ^ml
«2« + K
(1.2.6)
Consistent with the definition of m x « matrix addition, the multiplication of the matrix A by a complex number a (scalar multiplication) is defined by a A = [aa,.^.];
(1.2.7)
i.e., a A is formed by replacing every element a^j of A by aa,y.
1.3. DEFINITION OF A DETERMINANT A determinant is defined specifically for a square matrix. The various notations for the determinant of A are
D = D ^ = D e t A = |A|=:K.y|
(1.3.1)
We define the determinant of A as follows:
D^ Yl (-l)''«./,«2,,
(1.3.2)
where the summation is taken over all possible products of a^^ in which each product contains n elements and one and only one element from each row and each column. The indices / , , . . . , / „ are permutations of the integers 1 , . . . , n. We will use the symbol Y! to denote summation over all permutations. For a given set {/i,..., /„}, the quantity P denotes the number of transpositions required to transform the sequence /j, / 2 , . . . , /„ into the ordered sequence 1, 2 , . . . , w. A transposition is defined as an interchange of two numbers /, and Ij. Note that there are n\ terms in the sum defining D since there are exactly n\ ways to reorder the set of numbers { 1 , 2 , . . . , n) into distinct sets {IxJi^ • • •»h)As an example of a determinant, consider aji
^12
ci\
^21
^22
^23
^31
^32
^33
— ^n«22%3 ~ ^12^21^33 ~ ^11^23^32
«13«22«31 + «I3«2I<'32 + «I2«23«31 •
(1.3.3)
CHAPTER I DETERMINANTS
The sign of the second term is negative because the indices {2, 1,3} are transposed to {1, 2, 3} with the one transposition 2, l , 3 - > 1,2,3, and so P = 1 and (—1)^ = —1. However, the transposition also could have been accomplished with the three transpositions 2 , 1 , 3 - > 2,3,1 -> 1,3,2-^ 1,2,3, in which case P = 3 and (—1)^ = — 1. We see that the number of transpositions P needed to reorder a given sequence / j , . . . , /„ is not unique. However, the evenness or oddness of P is unique and thus (—1)^ is unique for a given sequence. • • • EXERCISE 1.3.1. Verify the signs in Eq. (1.3.3). Also, verify that the number I I • of transpositions required for <^ii«25^33'^42^54 is even. A definition equivalent to that in Eq. (1.3.2) is (1.3.4) If the product ai^iai^2'' '%n *s reordered so that the first indices of the «/, are ordered in the sequence 1 , . . . , « , the second indices will be in a sequence requiring P transpositions to reorder as 1 , . . . , M. Thus, the n! n-tuples in Eqs. (1.3.2) and (1.3.4) are the same and have the same signs. The determinant in Eq. (1.3.3) can be expanded according to the defining equation (1.3.4) as
'*22
^23
— ^11^22^32 ~ ^21^12^33 ~ ^11^32^23
(1.3.5)
'*32
« 3 , a 2 2 ^ j 3 4 - « 2 l « 3 2 « l 3 + «3l<^12«23-
It is obvious by inspection that the right-hand sides of Eqs. (1.3.3) and (1.3.5) are identical since the various terms differ only by the order in which the multiplication of each 3-tuple is carried out. In the case of second- and third-order determinants, there is an easy way to generate the distinct n-tuples. For the second-order case,
21
"22
the product of the main diagonal, an^22» is one of the 2-tuples and the product of the reverse main diagonal, ai2«2i» is the other. The sign of «i2<^2i is negative since {2,1) requires one transposition to reorder to {1, 2}. Thus, — diiCl'j') Ml "22 ^21
^22
di'yCl' 12"21
(1.3.6)
DEFINITION OF A DETERMINANT
since there are no other 2-tuples containing exactly one element from each row and column. In the case of the third-order determinant, the six 3-tuples can be generated by multiplying the elements shown below by solid and dashed curves / /Tl
/« f l /C^Al /013 ' /
/
/
qi\ a^2 «^3 [ OKi^ ail ai^ ^
The products associated with solid curves require an even number of transpositions P and those associated with the dashed curves require an odd P. Thus, the determinant is given by a,3 ^22
«23
= «11^22^33 + «12^23«31 + ^13<^2I«32
(1.3.7)
«33 «I3«22^31 -
« l l « 2 3 « 3 2 " ^12^21^33^
in agreement with Eq. (1.3.3), the defining expression. For example, the following determinant is 0: 2 5 2
= 1(5)1 + 2(6)3 + 3(2)4 - 3(5)3 - 2(6)1 - 1(2)4 (1.3.8) = 0.
The evaluation of a determinant by calculation of the n\ n-tuples requires {n — l)(n!) multiplications. For a fourth-order determinant, this requires 72 multiplications, not many in the age of computers. However, if n = 100, the number of required multiplications would be
<"-""-<"-'>(7)"=K^)'
(1.3.9)
~ 3.7 X 10*^^
where Stirling's approximation, n\ ^ {n/eY, has been used. If the time for one multiplication is 10~^ sec, then the required time to do the multiplications would be 3.7 X lO^'*^ sec,
or
1.2 x 10^^^ years!
(1.3.10)
Obviously, large determinants cannot be evaluated by direct calculation of the defining n-tuples. Fortunately, the method of Gauss elimination, which we will describe in Chapter 3, reduces the number of multiplications to n^. For n = 100, this is 10*^ multiplications, as compared to 3.7 x 10^^^ by direct n-tuple evaluation. The Gauss elimination method depends on the application of some of the elementary properties of determinants given in the next section.
CHAPTER i DETERMINANTS
1.4. ELEMENTARY PROPERTIES OF DETERMINANTS If the determinant of A is given by Eq. (1.3.2), then—because the elements of the transpose A^ are Uj^—it follows that (1.4.1) However, according to Eq. (1.3.4), the right-hand side of Eq. (1.4.1) is also equal to the determinant D^ of A. This establishes the property that 1. A determinant is invariant to the interchange of rows and columns; i.e., the determinant of A is equal to the determinant of A^. For example, 1 3
8 = 7 - 2 4 = -17 7
1 8
3 = 7 - 2 4 = -17. 7
Another elementary property of a determinant is that 2. If two rows (columns) of a determinant are interchanged, then the determinant changes sign. For example. D =
=
flii^^??
— floiflf 21"12»
m
D' =
^11
a^xQ i\^\i
^w^n = - ' ^ •
From the definition of D in Eq. (1.3.2),
^=EVi)%«2/,
hir'
^nL^
it follows that the determinant D' formed by the interchange of rows i and j in D is ^ ' = E (~l)^ «l/,«2/2 • • • ^Jh • ' ' ^ilj ' ' ' ^nl„ '
(1.4.2)
Each term in D' corresponds to one in D if one transposition is carried out. Thus, P and P' differ by 1, and so (-1)^' = (-1)^^^ = - ( - 1 ) ^ . From this it follows that D' = — D. A similar proof that the interchange of two columns changes the sign of the determinant can be given using the definition of D in Eq. (1.3.4). Alternatively, from the fact that D^ = D^T, it follows that if the interchange of two rows changes the sign of the determinant, then the interchange of two columns does the same thing because the columns of A^ are the rows of A.
ELEMENTARY PROPERTIES OF DETERMINANTS
/
The preceding property implies: 3. If any two rows (columns) of a matrix are the same, its determinant is 0. If two rows (columns) are interchanged, D ~ —D'. However, if the rows (columns) interchanged are identical, then D = D\ The two equalities, D = —D^ and D = D\ are possible only if D = D' = 0. Next, we note that 4. Multiplication of the determinant D by a constant k is the same as multiplying any row (column) by k. This property follows from the commutative law of scalar multiplication, i.e., kab — {ka)b = a(kb), or kD=
E'(-l)^^i/,«2/2 • • - H / , • "(ini„
«2l
"2«
(1.4.3)
kai„ ci„„
a,.
«12
ka^
«22
kG'j
*ln
ka nj
^nl
Multiplication of the determinant
D
1 2 1
2 4 2
3 6 8
by \ gives 1 2 2^ = 1
1 2 1
3 6 8
from which we can conclude that D/2 = 0 and D = 0, since D/2 has two identical columns. Stated differently, the multiplication rule says that if a row (column) of D has a common factor k, then D = kD\ where D' is formed from D by replacing the row (column) with the common factor by th6 row (column) divided by the
CHAPTER I DETERMINANTS
common factor. Thus, in the previous example, 1 D=4 I 1
1 1 1
3 3 8
The fact that a determinant is 0 if two rows (columns) are the same yields the property: 5. The addition of a row (column) multiplied by a constant to any other row (column) does not change the value of D. To prove this, note that
(1.4.4)
The second determinant on the right-hand-side of Eq. (1.4.4) is 0 since the elements of the iih and jth rows are the same. Thus, D' = D. The equality D^ = D^T establishes the property for column addition. As an example, 1 1
H-2^ 1+3A:
2 =3-2=1= 3
2 = 3 + 6A:-2-6A:=l. 3
Elementary properties can be used to simplify a determinant. For example. 1 2 1
2 4 2
3 6 = 8
1[ 2 1
-2(2)
1 = 4 1 1
2 4 2
3 -2 1 2 1 6-4 = 2 4 2 8-2 1 1 2 6
1 1 1 0 0 0
1 1 1
1 1 1 = 4 1 6 1
0 1 0 = 41 0 5 0
0 0 0
1-1 1-1 1-1
1 -1 1-1 6-1
(1.4.5)
Ol o| 5|
The sequence of application of the elementary properties in Eq. (1.4.5) is, of course, not unique. Another useful property of determinants is: 6. If two determinants differ only by one row (column), their sum differs only in that the differing rows (colunms) are summed.
COFACTOR EXPANSIONS
That is. «ir--
«„.•• ay-\-by-
«ii
a,„
+ ^m
«nl • • • «m + Ki • • • «n
^nn
(1.4.6) This property follows from the definition of determinants and the distributive law (ca + cb = c(a -f b)) of scalar multiplication. As the last elementary property of determinants to be given in this section, consider differentiation of D by the variable /: dD 'dt
= E'(-, + •••
, ^
-an,
• • '^nl„
+ E'(- !)''«„.^2/2
+ E'(--Ifa,,
day, dt
da„i^ dt
(IA7)
or dD dt
= E'(-
dt
-%2 ' • • «/„«
+ E'(-- 1 ) % .
%2-
dt
(1.4.8)
=j:^i' The determinant D'- is evaluated by replacing in D the elements of the ith row by the derivatives of the elements of the ith row. Similarly, for Dl\ replace in D the elements of the ith column by the derivatives of the elements of the ith column. For example. dOii
d
dt
It
dci^2
dt
+
«22
da2\
da22
IT
dt
da II
dai2
dt dan dt
(1.4.9)
dt
-f
d^22 ^21
IT
.5. COFACTOR EXPANSIONS We define the cofactor A,^ as the quantity (—1)'^^ multiplied by the determinant of the matrix generated when the ith row and jth column of A are removed. For example, some of the cofactors of the matrix
A =
^21
^22
^23
•^32
^33
(1.5.1)
10
CHAPTER I DETERMINANTS
include Au =
^22
^23
^32
,
«12
A21 — —
«13 '
^33
«32
a,2
fl,3
^22
^23
^31 —
«33
(1.5.2)
In general, M.7+1
*i,;-i
A,^^{-\r^
^i-l,n
H-iA
(1.5.3)
^f + l,n
^nl
'*«.7-l
Note that an /i x n matrix has n^ cofactors. Cofactors are important because they enable us to evaluate an nth-order determinant as a linear combination of n {n — l)th-order determinants. The evaluation makes use of the following theorem: CoFACTOR EXPANSION
THEOREM.
The determinant D of K can be computed
from where i is an arbitrary row,
(1.5.4)
where j is an arbitrary column.
(1.5.5)
D — XI ^u ^'7' 7=1
or D = ^ciij^ijy 1=1
Equation (1.5.4) is called a cofactor expansion by the fth row and Eq. (1.5.5) is called a cofactor expansion by the 7th column. Before presenting the proof of the cofactor expansion, we will give an example. Let
A =
4 -1 0
-1 4 -1
0 -1 4
(1.5.6)
By the expression given in Eq. (1.3.7), it follows that D = . 6 4 + 0 + 0 + ( - 0 ) - 4 - 4 = 56. The cofactor expansion by row 1 yields D=4
4 -1
-1 4
(-1)
= 60 - 4 + 0 = 56,
1
-1
0
4
+0
-1 0
4 -1
II
COFACTOR EXPANSIONS
and the cofactor expansion by column 2 yields
-1 0
D^^-H^l)
+4
4 0
0 4
= 56.
1(-1)
To prove the cofactor expansion theorem, we start with the definition of the determinant given in Eq. (1.3.4). Choosing an arbitrary column j , we can rewrite this equation as (1.5.7) '= 1
/l.....'•
In
where the primed sum now refers to the sum over all permutations in which Ij — i. For a given value of / in the first sum, we would like now to isolate the ijth cofactor of A. To accomplish this, we must examine the factor (—1)^ closely. First, we note that the permutations defined by P can be redefined in terms of permutations in which all elements except element / are in proper order plus the permutations required to put / in its place in the sequence 1, 2 , . . . , w. For this new definition, the proper sequence, in general, would be 1,2,3,
, / - 1, / + 1, / + 2 , . . . , ; - 1, 7 + 1, ; + 2 , . . . , n - 1, n.
(1.5.8)
We now define P/^ as the number of permutations required to bring a sequence back to the proper sequence defined in Eq. (1.5.8). We now note that \j — i\ permutations are required to transform this new proper sequence back to the original proper sequence 1, 2 , . . . , n. Thus, we can write (-1)'' = (-1)^^(-1)'+^ and Eq. (1.5.7) becomes
D = Ei-lY^^l i=l
E'
(-l)''^«/,.«(,2 • • •a,._,;-,%,;+, • • •«/„„)«o,
\/,,....i,...,/„
(1-5-9)
/
which we recognize using the definition of a cofactor as
1=1
A similar proof exists for Eq. (1.5.4). With the aid of the cofactor expansion theorem, we see that the determinant of an upper triangular matrix, i.e.,
0 U = 0
u 22
*23
*2/i
0
II 33
*3n
0
0
(1.5.11)
where u^j = 0, when / > j , is the product of the main diagonal elements of U, i.e., \V\ = f]u,. 1=1
(1.5.12)
12
CHAPTER I DETERMINANTS
To derive Eq. (1.5.12), we use the cofactor expansion theorem with the first column of U to obtain ^22
^^23
*2n
0
u 33
*3n
|U| = M„
1=2
0
0
"22
0
"23
*2n
U'^^
*3/i
0
0
where Un is the /I cofactor of U. Repeat the process on the (n — l)th-order upper triangular determinant, then the (n — 2)th one, etc., until Eq. (1.5.13) results. Similarly, the row cofactor expansion theorem can be used to prove that the determinant of the lower triangular matrix.
L =
0
\lu In
'22
IL
l„2
•
•
0
•
0 (1.5.13)
••
hin
IS
(1.5.14)
\u = Uhi'^
i.e., it is again the product of the main diagonal elements. In L, l^j ~ 0 when j > i. The property of the row cofactor expansion is that the sum n
replaces the ith row of D^ with the elements a^j of the ith row; i.e., the sum puts in the ith row of D^ the elements a^i, a,2' • • •»^/n- Thus, the quantity n
puts the elements a j , fl2» • •»^n in the /th row of D^, i.e.,
"2n ^i-1,1
(1,5.15) ^i+i, 1
a
13
COFACTOR EXPANSIONS
Similarly, for the column expansion. Of,
^l,j-\
a 1../+1
(1.5.16) i=\
^nj-l
^n
^«,y+l
EXAMPLE 1.5.1.
1 3 1
A =
A21 —
An =
2 2 1 2 1
*31
£)^ = 1 X A l l + 3 X A21 -f 1 X ^31
= l ( 3 ) ^ 3 ( - l ) + l(-4) = - 4 A'-
2
3
2
1
D ^ , = « ! X A , i + a2 X A21 + 0^3 X A3J = 30?! + 0 ' 2 — 4Qf3.
The cofactor expansion of D^ by the first-column cofactors involves the same cofactors, Ay, A21, and A31, as the cofactor expansion of D^' by the first column. The difference between the two expansions is simply that multipliers of A^, A21, and A31 differ since the elements of the first column differ. Consider next the expansions n
ky^j,
(1.5.17)
and
E%A-r
k^i.
(1.5.18)
The determinant represented by Eq. (1.5.17) is the same as D^, except that the 7th column is replaced by the elements of the /cth column of A, i.e., column j T.^ik^ij
column k (1.5.19)
= ^rti
^nk
14
CHAPTER I DETERMINANTS
The determinant in Eq. (1.5.19) is 0 because columns j and k are identical. Similarly, the determinant represented by Eq. (1.5.18) is the same as D^, except that the ith row is replaced by the elements of the A;th row of A, i.e.,
^kn
row I (1.5.20) row k
^ni
The determinant in Eq. (1.5.20) is 0 because rows / and k are identical. Equations (1.5.19) and (1.5.20) embody the alien cofactor expansion theorem: ALIEN COFACTOR EXPANSION THEOREM.
The alien cofactor expansions are
0, Le., Yl^ik^ij=^^
^i'h
i^\
(1.5.21)
E«ityA7=0,
k + i.
7=1
The cofactor expansion theorem and the alien cofactor expansion theorem can be summarized as (1.5.22) where ^^^ is the Kronecker delta function with the property hi = 1»
^= h
= 0,
k +j.
(1.5.23)
1.6. CRAMER'S RULE FOR LINEAR EQUATIONS Frequently, in a practical situation, one wishes to know what values of the variables jCi, ^ 2 , . . . , ^„ satisfy the n linear equations
«21^1 + «22-^2 + • • • + ^2n^n = ^2
^nl^l+««2^2 + " - + « « n - ^ n = ^ .
(1.6.1)
15
CRAMER'S RULE FOR LINEAR EQUATIONS
These equations can be summarized as Y^aijXj =bi,
/ = 1,2,... ,w.
(1.6.2)
7=1
Equation (1.6.2) is suggestive of a solution to the set of equations. Let us multiply Eq. (1.6.2) by the cofactor A,^ and sum over /. By interchanging the order of summation over / and j on the left-hand side of the resulting equation, we obtain (1.6.3) j
f
i
By the alien cofactor expansion, it follows that (1.6.4)
E«oAit=0 unless j = k, whereas, when j = k, the cofactor expansion yields ^11
«12
^21
^22
«nl
^«2
^2fi
J2^ik^ik = ^ =
(1.6.5)
Also, it follows from Eq. (1.5.19) that Ml
'^l.Jt-l
"12
^1
^l.it+1
D,^j:b,A,,=
(1.6.6) ««1
««2
^n,it-l
^/i
^n,k+l
where D^ is the same as the determinant D except that the /:th column of D has been replaced by the elements ^ j , ^2' • • • ^ ^w According to Eqs. (1.6.4)-( 1.6.6), Eq. (1.6.3) becomes (1.6.7)
Dx. = D.. Cramer's rule follows from the preceding result:
If the determinant D is not 0, then the sohition to the linear
CRAMER'S RULE.
system, Eq. (L6J), is
£1 D
k = I,.., ,n.
(1.6.8)
and the solution is unique. To prove uniqueness, suppose x^ and y^ for / = 1 , . . . , w are two solutions to Eq. (1.6.2). Then the difference between J2j a^jXj = b^ and J^j a^jyj = bf yields E « i 7 ^ -3^;) = ^'
' =^
^'
(1.6.9)
16
CHAPTER I DETERMINANTS
Multiplication of Eq. (1.6.9) by A^^ and summation over / yields D(x, - y,) = 0,
(1.6.10)
or jc^ = j ^ , k = 1 , . . . , n, since D 7^ 0. Incidentally, even if D = 0, the linear equations sometimes have a solution, but not a unique one. The full theory of the solution of linear systems will be presented in Chapter 4. EXAMPLE 1.6.1. Use Cramer's rule to solve 2Xi
-\-X2=l
Solution. D =
2 1
i =3 2
7 3
1 = 11 2
2 1
7 = -1 3
11
_D^_
1
y
D Even if the determinant found no other role, its utility in mathematics is assured by Cramer's rule. When D / 0, a unique solution exists for the linear equations in Eq. (1.6.1). We shall see later that, in the theory and applications of linear algebra, the determinant is important in a myriad of circumstances. .7. MINORS AND RANK OF MATRICES Consider the m x n matrix
^21
A =
^22
^2n
(1.7.1)
^w2
If m — r rows and n — r columns are struck from A, the remaining elements form an r X r matrix whose determinant M^ is said to be an rth-order minor of A. For example, striking the third row and the second and fourth columns of
A = ^31 L"4l
-^22
^23
•^24
'*25
^32
^33
'*34
^35
^43
^44
^45 J
(1.7.2)
17
MINORS A N D RANK OF MATRICES
generates the minor
M\ =
«ii
«J3
«15
«2l
«23
«25
*43
^45
(1.7.3)
We can now make the following important definition: DEFINITION. The rank r {or r^) of a matrix A is the order of the largest nonzero minor of A.
For example, for
(1.7.4)
|A| = 3
A =
and so r . = 2. On the other hand, all of the minors of
A =
1 1 1 1 1 1 1 1 1 1 1 1
(1.7.5)
except M\, are 0. Thus, r^ = 1 for this 3 x 4 matrix. For an m x n matrix A, it follows from the definition of rank that r^ < min(m, n). Let us end this chapter by mentioning the principal minors and traces of a matrix. They are important in the analysis of the time dependence of systems of equations. The jth-order trace of a matrix A, tr^ A, is defined as the sum of the yth-order minors generated by striking n — j rows and columns intersecting on the main diagonal of A. These minors are called the principal minors of A. Thus, for a 3 X 3 matrix, «ii
^12
^13
«21
^22
^23
«31
^32
^33
tr2A = «n
«12
«21
^22
tr3A =
+
= DetA,
(1.7.6)
«11
^13
-'23
a31
+ 1'*22
^33
^32
*33
trt A = ail + ari + a-^-i.
(1.7.7) (1.7.8)
For an n X n matrix A, the nth-order trace is just the determinant of A and tr, A is the sum of the diagonal elements of A. These are the most common traces encountered in practical situations. However, all the traces figure importantly in the theory of eigenvalues of A. In some texts, the term trace of A is reserved for trj A = Yl]=\ <^jj* ^^^ the objects tr^ A are called the invariants of A.
18
CHAPTER I DETERMINANTS
EXERCISE 1.7.1. Show that all of the traces of the matrix 4 1 1 0
A =
1 -4 1 1
1 1 -4 1
0 1 1 -4
(1.7.9)
are positive. We will show in Chapter 6 that this implies that A has only negative eigenvalues. It also implies that, independently of initial conditions, the solution x to the equation dx = Ax dt I I I
(1.7.10)
always vanishes with increasing time t.
PROBLEMS 1. Evaluate the determinant of the matrix given by Eq. (1.7.9) using the formulas
^ =
E h
(-i)^^i/,%/2• • '^^ = E ^ o ^ /«
'=1
2. Solve the following determinants by elementary operations 6 5 4
(a)
3 3 4
14 3
(b)
a-{-b a b
2 2 8 2
9| 9 9 9
c c b -\- c a = ^Aabc. b c -{- a
(c) Solve the following set of equations: ~r X2
+ 2JC3 = 1
—JCi 4 - 4JC2
+ 5JC3 = 1
JX^
— iXl
+ 2Xr^, + J C 3 =
. Evaluate the determinants 0 0
0 0
0
^1
^1
«2
0
Ci
h
«3
di
C2
&3
a4
-l.
19
PROBLEMS
and
h
^3
0 0 0
Cj
^3
e^
h
^5
0 d. 0
0 ds e^
4. Using Cramer's rule, find jc, y, and z for the following system of equations:
3x~4y + 2z=l 2x-^3y-3z
= -l
5x -5y-\-4z = 7. 5. Using Cramer's rule, find jc, y, and z for the following system of equations:
6/x-2/y-^\/z=4 2/x + 5/y-2/z 5/x-l/y
= 3/4
+ 3/z = 63/4.
6. Show that 0 a b c
a 0 c b
b c 0 a
c b a 0
0 1 1 1
1 c^ 0 a^
1 0 c^ b^
I b^ a^ 0
7. Use the determinant properties to evaluate .2
a"
^2
8. Using Cramer's rule, find x, y, and z, where 3JC
+ 43; - 2z = 3
2;c 4- 2}^ - 3z = 1 -jc + >' - 2z = - 2 . 9. Using Cramer's rule, find x, y, and z for the system of equations 4JC + 3JC
7y - z = 7
+ 2y 4- 2z = 9
jc + 5 j — 3z = 3.
20
CHAPTER I DETERMINANTS
10. Using Cramer's rule, find x, y, and z for the system of equations x + 3y = 0 4- 6y + 4z = 0
2JC
-x + 2z = 0, 11. Using Cramer's rule, find x, y, and z for the system of equations X + y -\-z = 1 x + l.000ly-{-2z
=2
x + 2y-{-2z = 1. 12. Let D,=
^11
^n
^21
^22
and
D2 =
^1 ^21
hi
Show that
D = D,D2 =
«ll
«12
0
^21
^22
^
0 0
-1
0
fc„
^12
0
-1
bn.
^22
1 1 3
3 -2 -1
4 -1 2
5 1 3 1
2 -1 5 4
4 6 1 8
13. What is the rank of 2 1 0 14. Evaluate the determinant 3 1 2 2
D
by first generating zero entries where you can and then using a cofactor expansion. 15. Show that 1+c, 1 1 1
1 I+C2 1 1
1 1 I+C3 1
1 1 1 I+C4
A
1
1
1
1\
V
Ci
C2
C3
C4/
= c,c-,c3C41+ - + - + - +
- .
21
PROBLEMS
16. What is the rank of
A =
1 2 1 4 5 7
3 6 9
4 8 1
17. Give all of the minors of A=: ^21
^22
^23
18. Give all of the traces of the matrix whose determinant is shown in Problem 15. 19. Solve the equation l-x
7 -h 2JC
0 -h 3A:
4 + 2x
10-4JC
6-6A:
2
4
5
20. Without expanding the determinant, show that x^ y2
X y
ix
~y)(y~z)ix-z).
21. Consider the set of matrices
a b A =
b
b
(a) Defining the determinants D„ as D„ = |A„| find a recursion relation for D„ (i.e., D„ = f(D„_i, D„-2,...; a, /?)). (b) Letting a = 0.5 and b = I, write a computer program to evaluate Dgg.
22
CHAPTER I DETERMINANTS
22. Consider the n-dimensional matrix jc a
a X a
a X a a X a
0 For the case where a = I and jc = 2cos^, prove that the determinant is given by D = sin{n 4-1)0/ sin 6 as long as 9 is restricted to 0 < ^ < n. 23. Find the determinant of the n x n matrix whose diagonal elements are 0 and whose off-diagonal elements are a, i.e., 0 a a
a 0 a
a a 0
a a
a a
a a
•• .
a a a
a a a
0 a
a 0
24. Find the following determinant:
l+«, 1+fl, a
a.
1+a,
^1
^2
«3
• * • ! + « «
25. Prove the following relation for the Vandermonde determinant'. 1
1 X2 ^2
1
= n fi ^-^/)(=1 7=« + l
^n-l
FURTHER READING Aitken, A. C. (1948). "Determinants and Matrices." Oliver and Boyd, Edinburgh. Aitken, A. C. (1964). "Determinants and Matrices." Interscience, New York. Amundson, A. R. (1964). "Mathematical Methods in Chemical Engineering." Prentice-Hall, New Jersey. Bronson, R. (1995). "Linear Algebra: an Introduction." Academic Press, San Diego,
FURTHER READING
23
Muir, T. (1960). "A Treatise on the Theory of Determinants." Dover, New York. Muir, T. (1930). "Contributions to the History of Determinants, 1900-1920." Blackie & Son, London/ Glasgow. Nomizu, K. (1966). "Fundamentals of Linear Algebra." McGraw-Hill, New York. Stigant, S. A. (1959). "The Elements of Determinants, Matrices and Tensors for Engineers." Macdonald, London. Tumbull, H. W. (1928). "The Theory of Determinants, Matrices and Invariants." Blackie, London/Glasgow. Vein, R. "Determinants and Their Applications in Mathematical Physics," Springer, New York.
This Page Intentionally Left Blank
VECTORS AND MATRICES
2 . 1 . SYNOPSIS In this chapter we will define the properties of matrix addition and multiplication for the general mxn matrix containing m rows and n columns. We will show that a vector is simply a special class of matrices: a column vector is an m x I matrix and a row vector is a 1 x n matrix. Thus, vector addition, scalar or inner products, and vector dyadics are defined by matrix addition and multiplication. The inverse A~^ of the square matrix A is the matrix such that AA~^ = A~'A = I, where I is the unit matrix. We will show that when the inverse exists it can be evaluated in terms of the cofactors of A through the adjugate matrix ^n\
adjA =
^12
^22
A^n
^In
Hi
Specifically, by using the cofactor expansion theorems of Chapter 1, we will prove that the inverse can be evaluated as A-^ = - a d j A . D ^
25
26
CHAPTER 2
VECTORS A N D MATRICES
We will also derive relations for evaluating the inverse, transpose, and adjoint of the product of matrices. The inverse of a product of matrices AB can be computed from the product of the inverses B~^ and A~^ Similar expressions hold for the transpose and adjoint of a product. The concept of matrix partitioning and its utility in computing the inverse of a matrix will be discussed. Finally, we will introduce linear vector spaces and the important concept of linear independence of vectors sets. We will also expand upon the concept of vector norms, which are required in defining normed linear vector spaces. Matrix norms based on the length or norm of a vector are then defined and several very general properties of norms are derived. The utility of matrix norms will be demonstrated in analyzing the solvability of linear equations.
2.2. ADDITION AND MULTIPLICATION The rules of matrix addition were given in Eq. (1.2.6). To be conformable for addition (i.e., for addition to be defined), the matrices A and B must be of the same dimension m x n. The elements of A + B are then a^ + b^j; i.e., corresponding elements are added to make the matrix sum. Using this rule for addition, the product of a matrix A with a scalar (complex or real number) a was defined as aai
aa, (2.2.1)
Of A
aa„
aa„
Using the properties of addition and scalar multiplication, and the definition of the derivative of A,
dk ,. A(r + AO-A(0 -— = hm , dt
A/^o
(2.2.2)
Ar
we find that
^A -— = lim dt A/^o
a,^{t ^ ^t) ~ a^^(t) At
a^ijt-\- At) ~ a^^jt) At da^ da. dt dt
. dt
a,„(f + AO -- «i«(0 At a,„„{t + AO -a^nit) At
(2.2.3)
da„ dt
We can therefore conclude that the derivative of a matrix dkjdt elements are the derivatives of the elements of A, i.e.,
dA __ [^^,7] dt " \_ dt J'
is a matrix whose
(2.2.4)
27
ADDITION AND MULTIPLICATION
Note that \dA/dt\ ^ d\A\/dt, The determinant of d\/dt is a nonlinear function of the derivatives of a,y, whereas the derivative of the determinant |A| is linear. If A and B are conformable for matrix multiplication, i.e., if A is an m x n matrix and B is an n x /? matrix, then the product (2.2.5)
C = AB is defined. C is then anm x p matrix with elements
(2.2.6)
^0k=\
Thus, the ijWx element of C is the product of the /th row of A and the yth column of B, and so A and B are conformable for the product AB if the number of columns of A equals the number of rows of B. For example, if
A==
[l
l1
2
1
[3
ij
and
B =
then AB =
(2.2.7)
whereas BA is not defined. EXERCISE 2.2.1. Solve the linear system of equations dX
BA,
It
(2.2.8)
where Mt = 0) =
(2.2.9)
and B =
-2 1
1 -2
(2.2.10)
If X and y are «-dimensional vectors, then the product x^y is defined since the transpose x^ of x is a 1 x n matrix and y is an n x 1 matrix. The product is a 1 x 1 matrix (a scalar) given by (2.2.11) 1=1
x^y is sometimes called the scalar or inner product of x and y. The scalar product is only defined if x and y have the same dimension. If the vector x is real, then
28
CHAPTER! VECTORS AND MATRICES
the scalar product x^x = X!"=i ^f ^^ positive as long as x is not 0. We define the "length" ||x|| of x as ||x|| = \ / x ^
(2.2.12)
for real vectors. For a complex vector, J2i ^f is not necessarily positive—or even real. In this case, we define the inner product by
xV = E<>'i
(2-2.13)
and the length of x by ilx..
= yxtx = J f : K l ' ,
(2.2.14)
=1
where the quantity |jc,p (= x*x^) is the square of the modulus of x^. Here, again, x^ denotes the adjoint of x, namely, the complex conjugate of the transpose of x, x^ = [xj*,x*,,..,jc:],
(2.2.15)
where x* is the complex conjugate of x^. The length ||x|| has the desired property that it is 0 if and only if every component of x is 0 (i.e., if x = 0). By definition, the matrix product AB exists only if A is an m x p matrix and B is a /? X n matrix. The product AB is then an m x n matrix. Thus, if x is an mdimensional vector (m x 1 matrix) and y is an n-dimensional vector (n x 1 matrix), the vector y^ is a 1 x n matrix. The product xy^ therefore exists as an m x « matrix given by •^1^1
'"^lyi
•••
-^iJn
T
(2.2.16)
xy' ^my\
^myi
••'
^myn
In the language of vectors and tensors, in Euclidean vector space the product xy^ is known as a dyadic. We will use this term in later chapters. 2.3. THE INVERSE MATRIX We note that the product Ax is defined for an m x n matrix A and an «-dimensional vector X (i.e., an n x 1 matrix). With the matrix product so defined, the set of linear equations ^11-^1+ «12-^2+ • • • + ^Xn^n = ^1
:
:
:
:
(2.3.1)
can be summarized by the single equation Ax = b.
(2.3.2)
We can see that the multiplication of A by x transforms the n-dimensional vector X into the m-dimensional vector b.
29
THE INVERSE MATRIX
When n = m in Eq. (2.3.1) and D^ ^ 0, we know from Cramer's rule that Eq. (2.3.1) can be solved for x uniquely for any b. In this case, it seems natural to hunt for an n x n matrix A~^ that is the inverse of the n x n matrix A, i.e., a matrix such that A-^A = AA-' = 1 ,
(2.3.3)
where I is the identity matrix. The identity matrix is defined by the property that ly = y;
(2.3.4)
i.e., the product of the n x n identity matrix I and an n-dimensional vector y produces the vector y again. With this property, Eq. (2.3.2) can be solved by multiplying it by A~^ to obtain A ^Ax = Ix = x = A ^b.
(2.3.5)
If A~^ can be found, then the solution x to Ax = b is simply A~^b. The identity matrix I has the simple form 1 0
0 1 [5,7]'
0
(2.3.6)
0
i.e., I is unity on the main diagonal and is 0 elsewhere. We can use the cofactor expansion theorems proved in Chapter 1 to construct the inverse of a square, nonsingular (i.e., |A| / 0) matrix A. We define the adjugate of a matrix A by AnX ^22
adjA =
(2.3.7)
A^n where A,y is the cofactor of a^j. Note that the ijih element of adj A is the cofactor of the 7/th element of A. The elements of the matrix product A adj A
(2.3.8)
are then Jl^ikA
jk-
(2.3.9)
jt=i
But, according to the cofactor theorems (Eq. (1.5.18)), (2.3.10) k=l
30
CHAPTER 2
VECTORS A N D MATRICES
where D is the determinant of A. This means that A adj A is a diagonal matrix having the determinant D on the main diagonal (and, of course, O's elsewhere), i.e.,
A adj A =
D 0 0
0 D 0
0 0 D
0' 0 0 = Dl.
_0
0
0
D
(2.3.11)
Likewise, the ijih element of the product (2.3.12)
(adj A) A is
(2.3.13) k=l
which again, by the cofactor expansion theorems, obeys n
(2.3.14) k=l
and implies (2.3.15)
(adj A) A = DI. In summary, the properties of the adj A are that AadjA = (adjA)A = DI.
(2.3.16)
If D 7^ 0, then Eqs. (2.3.11) and (2.3.15) can be rearranged to give A(^-iadjA^ = (|-iadjA^A = I,
(2.3.17)
which, when compared with Eq. (2.3.3), shows that the inverse of A can be computed as A-' = - a d j A . D
(2.3.18)
Since the elements of (adj A) b are J^Ajibj
= Df,
r = l,...,n,
(2.3.19)
where D, is defined in Eq. (1.6.6), it follows that the elements of x = A 'b are Xj =
—'-,
D
i =
I,...
,n.
(2.3.20)
31
THE INVERSE MATRIX
Not surprisingly, the solution x = A"'b to Ax = b is exactly the one given by Cramer's rule. What is important, however, is that Eq. (2.3.8) gives the inverse of A once and for all. We can therefore generate a solution for a particular b by taking the matrix product A~^b. It should be noted that true inverses are only defined for square matrices. Otherwise, A~'A and AA"' cannot be equal to the same square matrix. We can, however, define left and right inverses or pseudo-inverses. If there exists a matrix G such that GA = I,
(2.3.21)
then we say G is a left inverse of A. Accordingly, if there exists a matrix H such that (2.3.22)
AH = I,
then we say H is a right inverse of A. Note that left and right inverses need not be unique. EXAMPLE 2.3.1. Find the inverse of A =
2 2
1 2
(2.3.23)
Since |A| = 2 , the adjugate formula gives A-^ =
(2.3.24)
such that ~'A =
• • •
1 '2 2 0
0" "1 — 2 0
0] = ij
I = AA
EXAMPLE 2.3.2. Find the left or right inverse of (2.3.25)
A=
Since |A| = 0, the adjugate formula will not give an inverse. To find a left inverse, we must find G =
^11
^12
^21
^22
(2.3.26)
such that GA =
^11-^12 ^21 -
^22
-^11+^12 - ^ 2 1 + ^22
1 0
0 1
(2.3.27)
32
CHAPTER 2
VECTORS A N D MATRICES
or ^11 - ^ 1 2 = 1 -^11+^12=0
(2.3.28)
^21 - ^22 = 0 ~82\ + ^ 2 2 = 1-
These equations imply that gn — 812^ Six — 822^ ^^^ 0 = 1 , which is impossible. Thus, G does not exist. Similarly, we can show that H does not exist. Examples 2.3.1 and 2.3.2 illustrate two general results for a square matrix. Namely, if a square matrix is nonsingular (|A| / 0), its inverse always exists. We have shown this by the above adjugate constructions. Also, if a square matrix is singular (|A| = 0), it has neither a right nor a left inverse. The proof of this property requires a bit more than we have learned so far. We need to know that the determinant of a product of square matrices obeys the formula |AB| = |A||B|.
(2.3,29)
From this property, which is proved in Chapter 4, and the property |I| = 1, it follows that |GA| = 0 is always true if |A| = 0. Therefore, the equations GA = I and AH = I, upon taking the determinants, imply that 0 = 1 , which is impossible. EXAMPLE 2.3.3. Find the right inverse of (2.3.30)
A = The pertinent equation is AH = I, _ 1 1
AH
1 2
hu11
hx2
^21
^22
Jhi
^32_
_ 1 = 0 ^
_. 0
ij
(2.3.31)
The corresponding set of equations is ^11 - ^21 + ^31 = 1' ^U + ^21 + 2^31 = ^'
^12 - ^22 + ^3 *32
0
(2.3.32)
^^12 + ^22 + 2/I32 = 1.
For arbitrary /131 and /132, these equations yield
H =
1
-I-/131
l - 3 / i 32 \-h 32
2/131
2h 32
1 - 3/131
(2.3.33)
• • I Thus, the right inverse of A exists, but is not unique. Wmm EXERCISE 2.3.1. Show that the matrix defined by Eq. (2.3.30) does not have • I • a left inverse.
33
TRANSPOSE AND ADJOINT
wmam
EXERCISE
2.3.2. show that the matrix 1 1
-1 -1
1 1
(2.3.34)
has neither a left nor a right inverse. An important property of the inverses of square matrices is that (AB)-^ = B-^A-\
(ABC)-^ = C-^B-^A-\
etc.;
(2.3.35)
that is, the inverse of a string of products of matrices is the product of the inverse of the matrices written in the opposite sequence. To prove Eq. (2.3.35), let D denote the inverse of AB, i.e., D = (AB)-^ Then DAB = ABD = I. However, B 'A" (ABD) = B
HA^A)BD
(2.3.36)
= B ^IBD = (B^B)D = ID = D.
Since ABD = I and B-^A"^! = B-^A-^ it follows that D = B-^A-^ or (AB)-^ = B - ^ A - ^
(2.3.37)
To prove the property for longer strings, set B' = BC.
(2.3.38)
(ABC)-* = (AB')-* = (B')-*A-* = C-'B-*A-^
(2.3.39)
Then
where the result in Eq. (2.3.37) is used. 2.4. TRANSPOSE AND ADJOINT We saw in Chapter 1 that the transpose A^ of the m x n matrix A is the n x m matrix formed by interchanging the rows and columns of A, i.e..
A^ =
«11
«21
'^m\
^12
^22
^ml
.^In
^2n
(2.4.1)
Accordingly, the adjoint A^ of A is the complex conjugate of the transpose of A, i.e..
Lt
_
(2.4.2)
34
CHAPTER! VECTORS AND MATRICES
where a*j is the complex conjugate of the element a^j. In the shorter notation, A = [a^jl
A" = [aj,l
A^ = [a*,].
(2.4.3)
Since a scalar a is a 1 x 1 matrix, the adjoint of a scalar is just its complex conjugate, i.e., a^ = a*. Of course, if A is real the adjoint and the transpose are the same since (A^)* = A^. In many cases, one needs the transpose or adjoint of matrix products such as AB, ABC, etc., for which the following property is useful: (AB)'^ = B^A'^,
(ABC)'^ = C^B'^A'^,
etc.,
(2.4.4a)
(AB)^ = B^A^
(ABC)^ = C^B^A^
etc.;
(2.4.4b)
that is, the transpose (adjoint) of a product string of matrices is the product of the transposes (adjoints) of the matrices taken in reverse order. The proof of Eq. (2.4.4a) is straightforward. The ijih element of B'^A'^ is [B^A'^],^ = Y,^ bjj^ajj = Y.k ^jkK^ proving that (AB)'^ = B'^A'^. If we define D = BC, then we have just proved that (AD)'^ = D^^A^ and D'^ = C^B, so that (ABC)'^ = C'^B'^A'^. A similar proof holds for a product stream of any length. Moreover, since the adjoint is simply the complex conjugate of the transpose, Eq. (2.4.4b) follows immediately from Eq. (2.4.4a). The matrices in Eq. (2.4.4) must be conformable to multiplication but, in general, do not have to be square. • • •
EXAMPLE 2.4.1. If A is an m x n matrix, x an m-dimensional vector (m x 1 matrix), and y an n-dimensional vector (n x 1 matrix), then the product x^Ay is a 1 x 1 matrix or a scalar. The adjoint of x^Ay, namely, (x^Ay)^ = y^A^x, is simply • • • its complex conjugate, (x^Ay)*. In many practical situations (such as reactor stability analysis or process control), one would like to know whether the equation — = -Ax (2.4.5) dt predicts that an arbitrary perturbation x will decay to 0 or diverge in time. In such a problem, the matrix A is square, say nxn. The answer to the question is: if the scalar z^(A + A'^)z is positive for every nonzero n-dimensional vector z, then x -> 0 as r = oo for any initial condition. To prove this, note that d\\xf dt
d
^X
(dx^\ /^x^\
.dx +dx
If X obeys Eq. (2.4.5), then ^
= -xt(A + A V dt
(2.4.7)
35
PARTITIONING MATRICES
If x^(A 4- A^)x is positive for every vector x, then Eq. (2.4.7) implies that d\\x\\' < 0 dt
(2.4.8)
for all time,
or the magnitude of all the components of x must approach 0. If, for any x, the product x^(A -f- A^)x is negative, then there will be some initial conditions for which the length of x, and hence the magnitude of its components, grow in time. In such a case, the reactor or control system would be unstable. Let us close this section by noting that the inverse of a transpose (adjoint) is the transpose (adjoint) of the inverse, i.e., KT
iAT=(A-')
and
(Atr' = (A-')t.
(2.4.9)
The properties T = I and I = (AA-')^ = ( A - ' ) T A T show that (A"')'^ = (A'^)-'. Similarly, it follows from 1+ == I that (A'^V = (A^)-'.
2.5. PARTITIONING MATRICES Any matrix can be partitioned into an array of matrices and/or vectors. As long as the matrices A and B are conformable for addition or multiplication, they can be partitioned into conformable matrices of matrices. Consider, for example, the partition of the 3 x 6 matrix A denoted by the lines: «11
A =
a^2
«21
a,3
«14
^15
«16
«23
«24
«25
«26
(2.5.1) ^21
.^31
«32
^33
«34
«35
*^23
«36
where An = A21 =
1^11
^12
^13
[_'^21
^22
^^23
[«31»«32i<^33]»
A19
«14
—
A,3 =
L«24 ^22 =
^25
"26
(2.5.2)
A23 = [ « 3 5 ' « 3 6 ] -
^34'
When A and B are conformable for addition, and are conformably partitioned, then their sum can be given by M?
A+ B =
Bu
B,
B,
B> i
Bp2
Bp « j
+ p«j
A„+B„
(2.5.3)
A„ + B„
A.,+B,,. By "conformably partitioned" we mean the elements A,^ and B^j have to have the same dimension for addition.
36
CHAPTER 2
VECTORS A N D MATRICES
When A and B are conformable for multiplication (i.e., the number of columns of A equals the number of rows of B), the product of the partitioned matrices A and B is M/;
B,
Bi.
^pl
B^r.
AB
••• A,QPJ
Ki
L ^
(2.5.4)
ZJ ^qk"k\
-k=]
'''
J2 ^qk^kr
k=\
Note that the product A^j^B^^^y has to be conformable for each k and for all ij pairs in question. A partitioning that will often be useful in later chapters is the partitioning of a matrix x into its column vectors, i.e., —Xii
—1
•^12
•^In
x=
[Xj, X2, . . . , X„J,
_^ml
(2.5.5)
X
^m2
where x^ is an m-dimensional vector with components x^j. To see how this is useful, consider the equation AX = I
(2.5.6)
for the n X n square matrices A, X, and I. The unit matrix I can be partitioned as I = [ei,e2,...,eJ,
(2.5.7)
where e, is a vector with 1 in the ith row and O's elsewhere, i.e., '0^
e. =
(2.5.8)
The vectors e, are orthonormal, which means e,^. = 5,,.
(2.5.9)
37
PARTITIONING MATRICES
If in Eq. (2.5.6) X is partitioned as shown in Eq. (2.5.5) and I as in Eq. (2.5.7), then multiplication of A and X using the rule shown in Eq. (2.5.4) yields [Axi, A x 2 , . . . , AxJ = [e,, 6 2 , . . . , e„, ],
(2.5.10)
or, equating matrix components. Ax, = e,,
/ = 1 , . . . , n.
(2.5.11)
When A is not singular (|A| 7^ 0), Cramer's rule assures the solution to the equations of (2.5.11). In this case, the matrix X is the inverse A~^ of A. Thus, by using matrix partitioning, we find yet another way to obtain the inverse of A, namely, to solve the linear equations Ax, = e, for each unit vector e,. The efficient solution of such equations is the topic of the next chapter. EXAMPLE 2.5.1. Find the inverse of A =
2 8
3 7
(2.5.12)
Axj =01 yields the equations 1 (2.5.13) Sxii +7JC21 = ^
or X,
=
10 4 5
(2.5.14)
Ax2 = 62 yields "^12 '
•^22 —
(2.5.15)
•^12 •" ^"^22 ~~
or X.
=
3_ 10
(2.5.16)
_1 5
and so 2_ A-^=[Xi,X2]
• • • It is easy to verify that AA IHIH
10 4 5
10
_i 5
= I.
EXERCISE 2.5.1. (a) Give the solution to Ax = b
• • •
(2.5.17)
for arbitrary b for the matrix A in the preceding example. (b) Calculate the transpose of the inverse of A.
(2.5.18)
38
CHAPTER 2
VECTORS A N D MATRICES
2.6. LINEAR VECTOR SPACES In the analysis of linear systems, it is frequently useful to define a vector space S and linear operators in that space. Generalization of the concept of vector length to the concept of operator norm can lead to some very strong statements about the existence of solutions and the convergence of some solution techniques. We say 5 is a linear vector space if it is a collection of elements x, y , . . . for which addition is defined and which obey the following properties: 1. If X and y belong to S, then x -f y belongs to S; i.e., if x, y e S, then X + y G 5'. 2. x + y = y-f X. 3. There exists the zero element 0 such that
x+0
X.
4. For every element x, there exists an element —x such that X -f ( - X ) = 0.
5. If Of and p are complex numbers, then a{p\) = {aP)x (a -f p)x = ax + jSx a(x + y) = a x + ay. The collection £„ of all n-dimensional vectors obeying the previously given rules of addition and multiplication by a scalar obviously has the properties 1-5. Thus, E,^ is a linear vector space. In much of this book, E^ will be the linear vector space of interest. However, the properties of a linear vector space admit much more general objects than n-dimensional vectors. A pertinent example is a collection of all functions sharing common properties, such as continuity, differentiability, integrability, and the like. Another example is the collection of all n x m matrices. An important vector concept is linear independence. If Xj and X2 are linearly dependent, then there exists a complex (or real) number c such that x, + CX2 = 0. For example, if X,
=
(2.6.1)
and
then Xi = 2X2 or c = —2. If x^ and X2 are linearly independent, then there is no number c such that x^ + CX2 = 0. For example, the vectors and
X.
=
(2.6.2)
are linearly independent because there is no multiplier of X2 that will place 1 in the second component of X2. The general definition of linear independence is that the set of p n-dimensional vectors {x,, X 2 , . . . , x^} are linearly independent if there exists no nonzero set of numbers {c^, C 2 , . . . , c^} (not all of which are 0) such that
Ec,x,=0.
(2.6.3)
39
LINEAR VECTOR SPACES
EXAMPLE 2.6.1. Prove that the vectors
3 X,
=
2 ' 1
1 \2
=
2 ' _3_
and
X,
=
(2.6.4)
are linearly dependent. We seek a solution to (2.6.5) /=1
or 3ci 4- ^2 + 5c3 = 0 2c, + 2c2 + 6C3 = 0 c, + 3c2 + Vc^
(2.6.6)
0.
Assume c^ ^ 0 and solve the first two equations for c, and C2. Muhiply the first equation by 2 and subtract the second equation 2(3ci + C2 = -5C3) 2c, -f 2c2 = —6C3 4c, = -4c3
(2.6.7) or c, = —C3 and C2 = —2C3.
The values c, = —C3 and C2 = —2C3 satisfy the third equation in Eq. (2.6.6). Therefore, C3 = 1, c, = — 1, and C2 = —2 are a solution to Eq. (2.6.5), and so the • • • vectors x,, X2, X3 are linearly dependent. We create what is known as a normed linear vector space by assigning to every vector x in 5 the norm (or length) ||x||. By definition, the norm is required to satisfy the properties: 1. ||x|| > 0 and ||x|| = 0 if and only if x = 0. 2. ||Qfx|| = \a\ ||x|| for any complex number a and any x.
(2.6.8)
3. I|x + y | | < | | x | | + ||y||. Here, \a\ denotes the magnitude of the complex number a. Property 3 is known as the "triangle inequality" after the well-known property of Euclidean vectors as illustrated in Figure 2.6.1. In ordinary Euclidean space.
FIGURE 2.6.1
40
CHAPTER 2
VECTORS A N D MATRICES
this inequality represents the property that the shortest distance between points a and ft is a straight line (which lies along x 4- y and is of length ||x + y|| = [Y^i=\{^i + yd^]^^^^ where x, and y^ denote the Cartesian coordinates of x and y). Clearly, if the norm ||x|| is defined as the square root of the inner product x^x, then properties 1-3 follow. However, this definition of the nomi is only one among many, and it is not always the most convenient one. For example, if x € £„, the quantity (2.6.9)
l|x|l
acceptat) norm. The special choice of for any positive real number p is also an acceptable i p = 2 corresponds to the familiar length \v/ V ? xx introduced earlier. For the case p = oo, it follows that \lp
ltx||^=
l\m\\xj''f:{\x,\/\xjy
= \xj\im{v)'/''
(2.6.10)
= \xjv°
p-^oo
= \x„ where |x^| is the magnitude of jc, of maximum value, i.e., \xj = m a x | x j ,
!>/>«,
(2.6.11)
and V is the number of components jc, having the same maximum magnitude. For example, if
(2.6.12)
X =
where 0 is real and i = y—T, then |jCi| =
l/2,
|JC2| =
1,
and
kal = 1.
and so |jc^| = 1 and v = 2. There are still other norms that we can define. For example, consider a selfadjoint matrix A (i.e., A^ = A) with the additional property that x^Ax > 0
for all X 7^ 0 in E^.
Such a matrix is called positive definite. The quantity x^Ax is real since (x^Ax)* = x^A^x = x^Ax, and thus the quantity ||x|| = Vx^Ax
(2.6.13)
41
LINEAR VECTOR SPACES
obeys the requisite properties of a norm. Similarly, if A is positive definite, the quantity
M„A = { E [ ( E 4 ^ ; ) ( E « - 7 ^ . ) ] ' " Y ' '
(2-6.14)
obeys the conditions for a norm for any positive real number p. Next, we want to define linear operators in linear vector spaces. An operator A is a linear operator in S if it obeys the properties: 1. If X € 5, then Ax € 5.
(2.6.15)
2. If X and y € 5, then A (ax + ^y) = a Ax -f ^Ay e S,
(2.6.16)
where a and fi are complex numbers. If 5 = £„, an n-dimensional vector space, then all n x n matrices are linear operators in £"„ when the product Ax obeys the rules of matrix muhiplication. Square matrices as operators in vector spaces will preoccupy us in much of this book. However, if 5 is a function space, then the most common linear operators are differential or integral operators (more on this in Chapters 8-10). We define the norm of a linear operator as ||A||=max||Ax||/||x||.
(2.6.17)
Thus, IIAII is the largest value that the ratio on the right-hand side of Eq. (2.6.17) achieves for any nonzero vector in S. From properties 1~3 of the norm of a vector, the following relations hold for the norm of a linear operator: 1. ||A|| > 0 and ||A|| = 0 if and only if A = 0.
(2.6.18)
2. ||aA|| = |a| ||A|| for any complex number a.
(2.6.19)
3. ||A + B||<||A|( + ||B||.
(2.6.20)
4. IIABII < IIAII ||B||.
(2.6.21)
Conditions 1 and 2 are easy to establish from the properties of ||x||. To prove condition 3, note that II(A + B)x|| = IIAx + Bxll < IIAxil + ||Bx||,
(2.6.22)
and so n.A_LitMi ll(A + B)x|| (A + B) = max -— x#o ||x|| IIAxll IIBxIl < max -—— + max ——x^O
Ijxll
x^O ||X||
< IIAII + IIBJI.
(2.6.23)
IIAyll < IIAII llyll
(2.6.24)
For condition 4, note that
42
CHAPTER! VECTORS AND MATRICES
follows from the definition of ||A||. But if we set y = Bx and use again the property ||y|| = ||Bx||<||B||||x||,
(2.6.25)
IIABxIl < IIAII ||B|| llxll
(2.6.26)
IIABxIl IIABII = max \ - ^ < ||A|| ||B||, x#o ||x||
(2.6.27)
we find
or
which proves condition 4. •^•1
EXERCISE 2.6.1. Show that, for the space E^ and the norm ||x||^, ||A|U = maxX:K7l \
.^ 7=1
(2.6.28)
^
and
IIAII, = max X : K I '
(2.6.29)
i.e., that the /? = cx) norm of the matrix A is the maximum of the sum of the magnitudes of the row elements and the /? = 1 norm is the maximum of the sum of the magnitudes of the column elements. Let us close this chapter with an example of how operator norms can be useful. Suppose we want to solve the linear equation X - Ax = b.
(2.6.30)
The above equation can be rearranged to x = Ax + b.
(2.6.31)
Letting x^*^^ denote an initial guess of the solution, the next estimate to the solution can be obtained from x(i) ^ Ax^^^ + b.
(2.6.32)
Continuing this process, known as Picard iteration, gives for the A:th estimate to the solution x(^) = Ax^*-^^ -f- b.
(2.6.33)
The question is: will this solution converge? Suppose x is the true solution to Eq. (2.6.30). Then subtraction of Eq. (2.6.33) from the successive iteration equations yields X - x^^^ z= A(x - x^^^)
(2.6.34)
X - x^*^ = A(x - x^^-^^) = A*(x - x^^^).
(2.6.35)
43
PROBLEMS
Taking the norm of Eq. (2.6.35) and using the properties of the norm, we obtain ||x - x^^^ll < ||A|h|x - x^^^ll.
(2.6.36)
Thus, if IIAll < 1, Eq. (2.6.36) guarantees that ||x —x^'^^H -> 0 as A; -> oo and so in Picard iteration x^*^ converges to the solution of Eq. (2.6.30). If ||A|| > 1, Picard iteration may or may not converge to a solution. In the next chapter, we will introduce iterative solution schemes whose convergence can be analyzed in terms of vector and matrix norms.
PROBLEMS 1. Show that the unit vectors e, in an n-dimensional space are similar to Cartesian unit vectors in three-dimensional Euclidean space. 2. Find the inverse of -1 1 0
A =
1 -2 1
0 1 -2
Calculate the determinant of the inverse of A. 3. Find the maximum value of the function / = x^Ax - 2b^x, where A is the matrix defined in Problem 2 and
b =
Recall that 9//3JC, = 0, / = 1, 2, and 3, at the maximum. 4. Calculate the scalar or inner product y^x, the lengths ||x|| and ||y||, and the dyadic xy^ where
X
and
=
y =
and / = v ^ ^ . 5. Compute the adjugate and inverse of the matrix defined in Problem 2. 6. Find the right inverse of A =
1 3
2 1
2 1
44
CHAPTER 2 VECTORS AND MATRICES
7. Prove that a two-dimensional space in which the quantity ||x|| is defined by ||x|| = x^Ax is a normed linear vector space if 2 -1 Hint: Show that the properties in Eq. (2.6.8) are obeyed. 8. Prove that the vectors
X,
=
1 2 3
5 1 2
X2 =
and
X.
=
are linearly independent. 9. Consider the equation X = Ax + b,
(1)
where 1 2
1 4
1 5
1
1
6
3
7
5
4
4
and
b =
Prove that Eq. (1) can be solved by Picard iteration, i.e., if x<^+^)=Ax^'^+b,
it = 1 , 2 , . . . ,
then x^^^ -^ X, the solution of Eq. (1), as A: -> oo. 10. Calculate the adjugate and inverse of the matrix
A =
1 2 3
5 1 2
3 7 1
11. Show that if A(^) =
cos^ sin^
— sin^ — cosO
then (1) Consider the two-dimensional Euclidean vector represented as X =
Examine A(6) \ and give a geometric interpretation of the result in Eq. (1).
PROBLEMS
45
12. Suppose that A = [a,y], where
^
( n - J - / + 1)!0 - 1)!
Compute A^. 13. Find all 2 x 2 solutions of the equation A" = A, where n is a positive integer. Consider first the case n = 2 and then solve for the general case. 14. Prove that, in general.
Under what conditions would e^^+®^^ = e^U^^ be true? 15. Suppose — = A(Ox,
x{t = 0) = Xo,
(1)
at where the elements a^j of A depend on t, (a) Use the result in Problem 14 to prove that = expl / A(T)dT IXQ is not a solution to Eq. (1). (b) If A is independent of time, prove that X = ^^'Xo
is the solution to Eq. (1). 16. Prove that the Frobenius norm, 1/2
IIAII obeys the requisite conditions of a norm, namely, Eqs. (2.6.18)~(2.6.21). 17. (a) Define a normed linear vector space for the linear vector space which consists of all the « x m matrices, (b) What are the linear operators in the space?
46
CHAPTER 2 VECTORS AND MATRICES
FURTHER READING Bellman, R. (1970). "Introduction to Matrix Analysis." McGraw-Hill, New York. Bronson, R. (1995). "Linear Algebra: an Introduction." Academic Press, San Diego. Hoffman, K. and Kunze, R. (1971). "Linear Algebra." 2nd Ed., Prentice Hall International, Englewood Cliffs, NJ. Householder, A. S. (1965). "The Theory of Matrices in Numerical Analysis." Blaisdell, New York. Nomizu, K. (1966). "Fundamentals of Linear Algebra." McGraw-Hill, New York. Noble B, and Daniel, J. W. (1977). "Applied Linear Algebra." Prentice Hall International, Englewood Cliffs, NJ. Smiley, M. E (1951). "Algebra of Matrices." Allyn & Bacon, Needham Heights, MA. Wade, T. L. (1951). "The Algebra of Vectors and Matrices." Addison-Wesley, Reading, MA.
SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
3 J . SYNOPSIS In this chapter we will deal exclusively with problems involving square matrices. We know from Cramer's rule that if |A| ^^ the linear system Ax = b has a unique solution. However, we learned in Chapter 1 that a straightforward application of Cramer's rule for a large system can be prohibitively expensive. In the next section we shall discover that a method due to Gauss (Gauss elimination) greatly reduces the cost of solving linear equations. We will explain how Gauss elimination can be used to find the inverse of matrices and the LU-decomposition when it exists. The LU-decomposition of a matrix can lead to reduced costs of solving sequential linear equations involving the same matrix. It also can be the most economical way to store the matrix A for eventual computer solutions to linear equations. In each row of a band matrix, there are nonzero elements only in p columns to the left and q columns to the right of the main diagonal element. This structure leads to very efficient LU-decomposition ii p -{-q is small compared to the dimension of the vector space. Despite the power of Gauss elimination, it is often more efficient to use iterative methods for solving linear equations. We will consider three such methods—the Jacobi, the Gauss-Seidel, and the successive overrelaxation (SOR) methods—and explore the conditions for the convergence of each. We will close the chapter by introducing the Picard and Newton-Raphson methods for solving nonlinear algebraic systems. The Newton-Raphson method is the more powerful, albeit more complicated, of the two methods. Getting a good first guess is essential
47
48
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
to the success of both methods and thus continuation methods are discussed. We will see that, when the Newton-Raphson method converges, it does so quadratically with the number of iterations. This quadratic convergence is not only very efficient, but it also provides a fingerprint whereby programming errors can be detected.
3.2. SIMPLE GAUSS ELIMINATION Consider the equation Ux = b, where U is an upper triangular matrix; i.e., U has nonzero elements only on and above the main diagonal. The equations corresponding to Ux = b are
= b.
W 22-^2 + "
(3.2.1)
Symbolically, these equations can be represented in Fig. 3.2.1, where the triangle represents the matrix U and the vertical lines represent the vectors x and b. Expansion by cofactors shows that the determinant of U is given by |U| = H-Li ^^r Thus, if none of the diagonal elements of U is 0, then |U| ^ 0 and Eq. (3.2.1) has a unique solution. The solution can be determined by back substitution, namely. x„ = n-I
^n-\,n-^n
^n-l
*«-!.«-
FIGURE 3.2.1
49
SIMPLE GAUSS ELIMINATION
or "
u (3.2.2)
^^_^-I..=.>i^.-.^.^
/=«-l,n-2,...,l.
From x„ we compute jc„_i, from which we compute x„,2^ etc. until {jf„,..., jCj} have been determined. To estimate the time it takes to evaluate the x^, we ignore the faster addition and subtraction operations and count only the slower multiplication and division operations. We note that at the iih step there are {n — /) multiplications and one division. Thus, the total number of multiplications and divisions can be determined from the series 1 n J ^ ( ^ - / + !) = Y^(n - / + !) = -n{n + 1).
(3.2.3)
The way to evaluate a sum such as that in Eq. (3.2.3) is to note that in analogy with the integral ^ ( n — / + 1) Ai <-> / (n - / -f 1) di '=* ^
(3.2.4)
= n(n-l)-^(n^-l)+/T-l, we expect a sum of first-order terms in n and summed to n terms to be a quadratic function of n. Thus, we try n
Y^(n - / + ! ) = an^ +fo/z+ c.
(3.2.5)
i=\
By taking the three special cases, « = 1, 2, and 3, we generate three equations for a, b, and c. The solution to these equations is a = b = ^ and c = 0, thus yielding Eq. (3.2.3). By blind computation with Cramer's rule, we saw in Eq. (1.3.9) that the number of multiplications and divisions of an nth-order system is (n — \)(n/e)" or 3.7 X 10^^^ if w = 100. By back substitution, Eq. (3.2.1) requires 50(101) = 5.05 X 10^ such operations if n = 100. Quite a savings! Similarly, if L is a lower triangular matrix, the linear system Lx = b is
^21-^1 " ^ '22-^2
~
^2
(3.2.6)
represented symbolically in Fig. 3.2.2. The determinant of L is |L| = Vl"=\hiThus, if no diagonal elements are 0, |L| ^ 0 and Eq. (3.2.6) has a unique solution,
50
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
FIGURE 3.2.2
which can be computed by forward substitution, namely, b, Ml
bj
l2[Xi
I22
or ^1 Xi
=
(3.2.1) X:
/ = 2 , . . . , n.
=
I.
We note that at the ith step there are / — 1 multiplications and one division. Thus, solution of Eq. (3.2.5) by forward substitution requires 1 J2i = •;^n(n-\- 1) multiplications and divisions, which is the same as solution of Eq. (3.2.1) by back substitution. Clearly, if the matrix A is an upper or lower triangular matrix, the cost of solving Ax = b is much smaller than what is expected by Cramer's rule. The method of Gauss takes advantage of such savings by transforming the problem to an upper triangular problem and solving it by back substitution. One could, equally easily, transform the problem to a lower triangular matrix and solve it by forward substitution. Consider the set of equations 2JCI + JC2 + 3^:3 =
1
3:^1 + 2:^2 + ^3 = 2 4xi + ^2 4- 2JC3 = 3.
(3.2.8)
51
SIMPLE GAUSS ELIMINATION
We can multiply the first equation by — | and add it to the second equation and then multiply the first equation by — | and add it to the third equation. The result is
ij,, -lx, = l2
2 "^
(3.2.9)
2
—X2 — 4JC3 = 1.
Now we multiply the second equation by 2 and add it to the third one to obtain 2Xy -\- X2 + 3JC3 = 1
1 7 1 2^2-2^3 = 2
^^-^-^^^
-11JC3=:2.
Note that the equations have been rearranged into upper triangular form. By back substitution, Xj
2 —"IT
X2
=
JC,
=
3 )
(3.2.11)
"
0 +^+ 2
sl =10
IT
The preceding steps constitute an example of solution of an algebraic system by Gauss elimination. In the general case, one wants to solve the system Ax = b, or a^Xi
+ ^12^:2 -f- ai3^3 + • • • + ^In^n = ^1
«21-^1 + «22-^2 + «23-^3 + ' ' ' + «2n-^« = ^2
(3.2.12)
As the first step in the elimination process, we multiply the first equation by —ail/an and add it to the /th equation for / = 2 , . . . , n. The result is a\\^x, + al^'^2 + «l3 ^3 + • • • + <^n
= ^1
«22^^2+«23^^3 + - - - + « 2 « ^ ^ « = ^ 2
(3.2.13)
^:^2+^:^3+---+«;i^n=e The coefficients in the first equation are unchanged and are denoted as a^^j^ to distinguish them from the coefficients in the equations / = 2 , . . . , « , which have
52
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
changed. In fact, (2) _{\) % """'7
"/I ^(1) (1) 1; ' a II
I, J —L, , . . ,n.
(3.2.14)
bf>=Z,.">-V/'-
'=2,...,n.
Ml
In the next step, the second equation is multiplied by —a^a l^ri ^^^ ^^^ result is added to the /th equation for / = 3 , . . . , n. This eliminates the terms in X2 in equations / = 3 , . . . , n and generates the set of equations
^22 ^2 "^ ^23 ^^3 + ' ' ' + ^2n ^n ^ ^2
af^x, + --- + 4„^x„=bf^
(3.2.15)
Continuation of this process eventually transforms Eq. (3.2.12) into the upper triangular problem a\\^x, + a|^^X2 + a[\\, + • • • + a['^x„ = b'^^ «2?^2 + «2?^3 + • • • + «2n^^'n = -,(3)^
,
^f
I ^(3)^ _ 1,(3)
<3% + " - + < ^ « = ^ 3
(3.2.16)
^r, which can be solved by back substitution. This procedure is Gauss elimination in its simplest form. Since the operations used in transforming Eq. (3.2.12) to Eq. (3.2.16) involve only the additions to or subtractions from various equations by multiples of other equations, the determinant of the matrix in Eq. (3.2.16) is the same as the determinant of A, i.e., |A| = n 4 f .
(3.2.17)
k=l
Thus, a by-product of Gauss elimination is the evaluation of the determinant of A. Let us now calculate the number of multiplications and divisions needed for Gauss elimination. At the A;th elimination step, there are / = fc + 1 , . . . , n ratios af^/af^ to compute and 7 = /: -f- 1 , . . . , n + 1 products {af^^ la^kk)^tj to compute for each /. For the purpose of counting, we call bf^ the {n + l)th element. Thus,
53
SIMPLE GAUSS ELIMINATION
it takes n-l
E ( ^ - ^) + E ( « - ^)('^ + l-k) k=\
= ln(n - 1) -f \n(n^ - 1)
k=\
^
(3.2.18)
^
multiplications and divisions to carry out the upper triangularization of the system of equations. To this we must add ^n{n + 1) multiplications and divisions to carry out the back substitution to get a final solution. Thus, the total number of operations i.,/^^2 -f_L 3n Q^ _ j ^ With A7 = 100 and a computer taking for Gauss elimination is ^n{n^ 10"^ sec for an arithmetic operation, a Gauss elimination solution costs about |(100)^ X 10"^ sec = 3.3x lO"'* sec compared to 3.7x 10^"^^ years for Cramer's rule. In carrying out the steps of Gauss elimination, the variables jr^, JC2,... play only a passive role. What is happening is that the augmented matrix augA = [A,b]
(3.2.19)
is being rearranged by addition and subtraction of multiples of rows. In the case of the problem in Eq. (3.2.8), the augmented matrix is
augA
2 3 4
1 3 1 2 1 2 1 2 3
(3.2.20)
The first step in Gauss elimination is the subtraction of | times the first row from the second row and twice the first row from the third row to obtain 2 0 0
1 1
3 7
1 1
2 -1
~2 -4
2 1
(3.2.21)
In the next step, twice the second row is added to the third row to obtain
2
1
«
\
0
0
3 _7 ~2 -11
1 1 2 2
= [A„,bJ.
(3.2.22)
Once Gauss elimination has transformed the augmented matrix to the form [Atf, b,J, where A^. is an upper triangular matrix, the solution to Ax = b can be computed from A„x = b^ by backward substitution.
(3.2.23)
54
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
Gauss elimination as just carried out above was based on the assumption that the coefficients afj^ were always nonzero. This is not always the case, even when |A| ^ 0 and a solution is assured. Consider the equations ^ 1 ~\~ ^X'2 +
3^3 = 1 (3.2.24)
2JCI -h Ax2 -f 2JC3 == 1 X^
1
-^2
+
A:3 =
3.
the first step in Gauss elimination,, the set becomes vVj -|- 2wV2 "f 3JC3 = 1
-4JC3 = ~X2-
(3.2.25)
-1
- 2;c3 = 2.
In this case, af^ = 0 and so the next step would fail since ^23/^22' = 00. On the other hand, if the second and third equations are interchanged, we obtain ^ j + 2JC2 -f 3JC3 =
1
-JC2 - 2x3 = 2
(3.2.26)
-4^3 = - 1 ,
which can be solved by backward substitution. As long as |A| ^ 0, at any step k in Gauss elimination there will exist at least one equation / > k such that the coefficient a^f ^ 0. Otherwise, the determinant of A would be 0, in contradiction to the nonsingular case we are considering in this chapter. A strategy of implementation of Gauss elimination that lends itself to an easy computer program can now be described. We want to solve the nxn problem Ax = b. First, we define the components of bi as h^ = a, „_,_i and the equations as rows /?,, i.e., il-
^11-^1 + ^12^2 + • —h«i„A:„ — «i „+i
ii-
^21-^1 + «22-^2 + • • • -^ ^2n^n
— %,n+l
(3.2.27)
n •
^n\^\
+ a^2^2 + ' '• *
' ^nn^n
^
^n,n-fl
The elements of the augmented matrix, aug A, are fl,y, / = 1 , . . . , n, 7 = 1 , . . . , n + I. In a computer code to execute Gauss elimination, we set the dimension of the augmented matrix to be /i x (n + 1), input the elements ay, and set DET = I. Then we carry out the following sequence of operations: Elimination Step 1. For / = 1 , . . . , n — 1, do steps 2-5. Step 2. Find the first-row p, i < p < n, with nonzero entry in the iih column (a^^i / 0). If none is found, then NO SOLUTION, |A| = 0, and STOP. Otherwise,
GAUSS ELIMINATION WITH PIVOTING
5 5
Step 3. If p ^ i, then perform a row swap Rp ^ Rf
(3.2.28)
and set DET = {-\)DET.
(3.2.29)
Step 4. For 7 = / + 1 , . . . , « , do step 5. Step 5. Set /?,->
R:-^R:,
(3.2.30)
Step 6. If a„„ = 0, then NO SOLUTION, |A| = 0, and STOP. Backward Substitution Step 1. Set x^ =
^S!l±l.,
(3.2.31)
5^^p 8. For / = n — 1 , . . . , 1, set ^i = («i,n+i -
E ^ij^j)/^ii'
(3.2.32)
j=i+i
5rep 9. \A\ = DETl\a,,.
(3.2.33)
Mathematica routines are provided in the Appendix for the Gauss elimination algorithm described above and in Section 3.3. In the computer program just outlined, we assume that the elements of the augmented matrix are written over at every elimination step. Thus, at the end of the elimination process, the augmented matrix is composed of the elements Oy\ i = 1 , . . . , w, j = / , . . . , n 4- 1. Note that an interchange of rows changes the sign of the determinant, and so the role of Eq. (3.2.29) is to keep track of the net sign change. Subtracting a multiple of a row from other rows does not change the sign of a determinant, and so there is no effect on the determinant in Eq. (3.2.30). The determinant of A is then the product Hi ^ti tidies the net sign change from row swaps; hence Eq. (3.2.33) follows.
3.3. GAUSS ELIMINATION WITH PIVOTING If all our arithmetic is done with infinite precision, Gauss elimination with row swaps as described in the preceding section will always yield the correct solution
56
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
to Ax = b whenever |A| ^0. However, with finite-figure floating-point arithmetic, round-off errors can cause problems. For example, consider the equations
jc, + I.OOOOIJC2 + 2JC3 = 2
(3.3.1)
^1 -1-2X2 +2-^3 = 1-
Gauss elimination reduces these equations to X^+X2+X2 = 1 0.00001jC2-fJC3 = 1 99,999JC3
(3.3.2)
= 100,000.
Let us now perform back substitution with a computer (admittedly a little fellow, but it makes the point) that does four-figure floating-point arithmetic. We find JC3 = 1.0000,
X2 = 0,
jci = 0.
(3.3.3)
The actual solution is, of course, ^3 = - ^ ^ ^ = - ^ 2 ,
^1 = 1.
(3.3.4)
One way to try to avoid round-off errors is to introduce so-called "pivoting" strategies in executing Gauss elimination. Consider the general elimination problem at the /cth step. The equations are of the form <^n -^1 ~^ ^12-^2 + 4.
^22 ^2 "^
% - ]
^(2)
_
, (2)
+
(3.3.5)
+ •<
^
^kn ^n
—
^k
These equations can be represented schematically as in Fig. 3.3.1. When the row swap as described in the previous section is not numerically stable, partial pivoting is frequently used to increase stability. In this pivoting strategy, we determine the largest magnitude of the elements aj^, j = k, k -h \, . . . , n. Suppose this is the element a^^^, i.e., l4',^| = max|af;|,
(3.3.6)
i.e., a^pl is the largest magnitude element of thefcthcolumn lying below the (fc—l)th row. In the partial pivoting scheme, rows k and p are swapped and the next Gauss elimination step is carried out. The interchange process of partial pivoting is shown in Fig. 3.3.2.
57
GAUSS ELIMINATION WITH PIVOTING
FIGURE 3.3.1
EXERCISE 3.3.1. Show that partial pivoting eliminates the round-off problem encountered in solving Eq. (3.3.1). Note that the only difference between partial pivoting and the Gauss elimination process outlined in Eqs. (3.2.27)-(3.2.33) is that in the former one hunts the first nonzero ajl\ i > k, whereas in the latter one seeks the largest of |a,-f |, / > k. In either case, once the row of the object element has been determined, say row p, one interchanges rows k and p and continues the Gauss ehmination process. Thus, in partial pivoting the only change in the process given by Eqs. (3.2.27)-(3.2.33) is to replace the search for the first-row p, k < p < n, with nonzero a^f^ by the search for the row p with the maximum of |fl,-f |, / > k. Although not as often used, complete pivoting gives even further numerical stability. In this scheme, one hunts the row p and column q containing the largest value of \ay^\i = k,... ,n, j = k,... ,n, i.e., 7^*^ I = ^^
max k
(fe)|
(3.3.7)
•'
The rows k and p and columns k and q are then interchanged so that element a^^^^ becomes the pivot element for the next step of Gauss elimination. Schematically, the interchange is shown in Fig. 3.3.3. Complete pivoting costs substantially more to compute and is more complicated to program, and so it is not usually employed in canned computer programs. The column interchange amounts to renaming the elements of the vector x, and so this feature needs to be added to a computer program executing complete pivoting.
row k
row p
FIGURE 3.3.2
58
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
column k
column q
row k
row p
FIGURE 3.3.3
In closing, we note that the simplest Gauss elimination procedure, without row interchange or pivoting, frequently works and is guaranteed to be successful in two cases: (1) when A is strictly diagonally dominant, i.e.. (3.3.8) 7=1
and (2) when A is positive definite, i.e., x^Ax > 0 for all x 7^ 0.
3.4. COMPUTING THE INVERSE OF A MATRIX In Chapter 1 we saw that the inverse of a nonsingular matrix (|A| 7*^ 0) is given by A"* = TTT^^JA
(3.4.1)
where the ij element of adj A is the ji cofactor of A. In retrospect, this would be an expensive way to compute A~^ A cheaper method can be devised using Gauss elimination. Consider the p problems Ax, = b , ,
/ = l,...,p.
(3.4.2)
These can be summarized in partitioned form as [Axi, A x 2 , . . . , AXp] = [bj, b 2 , . . . , b^]
or A[Xi,...,Xp] = [ b i , . . . , b p ] .
(3.4.3)
59
COMPUTING THE INVERSE OF A MATRIX
The augmented matrix corresponding to Eq. (3.4.3) is [A,bi,b2,.
(3.4.4)
,bp].
If Gauss elimination is carried out on this augmented matrix, the number of division and multiplication operations is n—\
t
n—l
J2(n-k)-^Y.(n-k)(n
+
1
p-k)=^-n(n^-l)-\--pn{n-l).
(3.4.5)
Programming Gauss elimination for Eq. (3.4.4) is the same as that which was oudined in Eqs. (3.2.26)-(3.2.32), or its variation with pivoting, except that the augmented matrix is of dimension n x (n + p) and it has elements a,j, /, ] — ! , . . . , « , and a, „^^ = 6,^, / = 1 , . . . , n, ; = 1 , . . . , /?. Gauss elimination transforms Eq. (3.4.4) into [Ajy, bj t r , . . . , b^ j j ,
(3.4.6)
where A^^ is an upper triangular matrix. The solutions to the equations in Eq. (3.4.2) can now be computed by backward substitution from / = 1,. . . , p .
A X — b
(3.4.7)
The number of division and multiplication operations required to solve these equations is - « ( n - 1),
(3.4.8)
and so the total number of operations needed to solve Eq. (3.4.2) for x, is the sum of Eqs. (3.4.5) and (3.4.8), i.e.. 1
-n(n - \)^-pn(n-
1).
(3.4.9)
Let us now consider the problem of finding the inverse of A. Define x, to be the solution of Ax, = e,,
/ = 1 , . . . ,n,
(3.4.10)
where e, is the unit vector defined in Chapter 2:
o'
(3.4.11)
60
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
where only the /th element of e, is nonzero. Defining the matrix X = [Xi,...,x,],
(3.4.12)
AX = [ A x , , . . . , AxJ = [e„ . . . , e j = I,
(3.4.13)
we see that
where I = [5,^] is the unit matrix. We now assume that A is nonsingular; i.e., A~^ exists. Multiplying Eq. (3.4.13) by A"^ we find (3,4.14)
X = [ x i , . . . , x j = A ^;
i.e., the inverse of A is a matrix whose column vectors are the solution of the set of equations in Eq. (3.4.10). We have just shown that the number of divisions and multiplications for solving n such equations is nin^~
(3.4.15)
l)+n^(n-l).
Thus, for large n, the inverse can be computed with |n^ operations. EXAMPLE 3.4.1. Find the inverse of 2 1 0
A =
1 -2 1
0 1 -2
(3.4.16)
The appropriate augmented matrix is 2 1 0
1 -2 1
0 1 -2
1 0 0
0 1 0
0 0 1
(3.4.17)
After the first elimination step, the matrix is 2 0 0
1 3 ~2 1
0
1 0
1 1
0
1 0
- 2 0 0 1
After the second step, -]
l ^ t r ' ^ 1 , tr' ^ 2 , tr» ^ 3 , trl —
2
1
0
0
3 ~2
1
0
0
4 ~3
1 1 2 1 3
0
0
1
0
2 3
1
(3.4.18)
61
LU-DECOMPOSITION
Solving AtrXj = e, ^r by backward substitution, we obtain
i-i
[ X | , X2, X3] —
3 4 1 ~2 1 - 4
1 2 -1 1 2
1 4 1 ~2 3
(3.4.19)
4J
3.5. LU-DECONPOSmON Under certain conditions, a matrix A can be factored into a product LU in which L is a lower triangular matrix and U is an upper triangular matrix. The following theorem identifies a class of matrices that can be decomposed into a product LU. LU-DECOMPOSITION THEOREM.
lAJ^O,
IfAisannxn
matrix such that
/» = 1 , . . . ,n — 1,
(3.5.1)
where A is the matrix formed by the elements at the intersections of the first p rows and cohimns of A, then there exists a unique lower triangular matrix L = [/,y], /„ = 1, and a unique upper triangular matrix U = [u^j] such that (3.5.2)
A = LU.
The determinants |A^|, /? = 1 , . . . , n, are known as principal minors of A. Note that since |A| = |A„| = 0 is allowed by the theorem, an LU-decomposition can exist for a singular matrix (i.e., for a matrix for which A"^ does not exist), but not too singular since |A„_,| ^ 0 implies the rank of A is greater than or equal io n — \. An example of a matrix obeying the hypothesis of the theorem is
A =
-2 1 0
1 -2 0
0 1 0
1
0
0
-2 0
' 0
(3.5.3)
In this case,
L =
(3.5.4)
and
U
0
1
0
-5
1
0
1
(3.5.5)
62
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
The reader can readily verify that LU = A and that |A| = 0 . Of course, A need not be singular, as the example -3 1 1
A =
1 -3 1
1 1 -3
(3.5.6)
illustrates. In this case. 0
0
1
0
L = 3 1 L ~3
-2
'
-3
1
0
8 ~3
1 4 3
0
0
-2
(3.5.7)
and
U =
(3.5.8)
It is worthwhile to note that the hypothesis of the theorem only constitutes a sufficient condition for LU-decomposition. For example, if A =
0 0
1 1
(3.5.9)
then |A[| = 0 and IA2I = |A| = 0. Nevertheless, there exist L and U such that (3.5.10)
LU = A. In particular, if 1
0
1
1
and
U =
0
1
0
0
(3.5.11)
then LU
0 0
1 1
= A.
(3.5.12)
If A = LU, it follows from the rules of matrix multiplication that min(/, j)
(3.5.13) k=l
LU-DECOMPOSITION
63
The upper limit on the summation over k comes from the fact that L has only zero elements above the main diagonal and U has only zero elements below the main diagonal. When a matrix can be upper triangularized by Gauss elimination without interchange of rows (which is true, for example, for positive-definite matrices), LUdecomposition is generated as a by-product. Recall that at the ^th step of Gauss elimination we eliminate the coefficients aj^K i > k, by replacing the elements a^f, /, 7 = / : + l , . . . , n , by a^+^> = al^' - m|f 4 \
/ = fc + 1 , . . . , /I, 7 = ^ , . . . , n,
(3.5.14)
where
^^-i)^
/=^,...,«.
(3.5.15)
Let us define a lower triangular matrix L = [/,y] by
/o = { 5 ' '•-'' [0,
<3.5.16)
I < J.
RecaUing that ajj^ = a,^, the original elements of A, and expressing Eq. (3.4.14) for the sequence of k's ranging from 1 to r, we obtain
-/n4\ ij "ij
/ = 2 , . . . ,/i, 7 = 1 , . .. , / i ,
ij "iJ
(3.5.17)
h3^3j
or «.7=«.v
+ E U y -
(3.5.18)
k=\
Ji) Gauss elimination generates an upper triangular matrix with elements ajj on and above the main diagonal and 0 below the main diagonal. With r = / — 1, Eq. (3.5.18) becomes I
aij=a^
(k)
+
j:hk4 *"' Jk)
k=i
(3.5.19)
64
CHAPTER 3
SOLUTION OF LINEAR A N D NONLINEAR SYSTEMS
since /,, = 1. Since af-' = 0 if A: > 7, Eq. (3.5.19) becomes min(/,y)
(3.5.20) Comparison of Eqs. (3.5.13) and (3.5.20) yields the conclusion that the LUdecomposition of A is 1
0 0
0
m-y
0 0 (3.5.21)
L = L ^ni
m„
^ni
and ,(0
"11
"12
0
(2) a22
0
0
U =
(3.5.22) ^{n)
where m,-, = a-j/ajj and the elements afj of U are the elements resulting from the upper triangularization of A by Gauss elimination without row interchange. As an example of LU-decomposition by Gauss elimination, consider -2 1 0
A =
1 -2 1
0 1 -2
(3.5.23)
Multiplying the first row by mji = —1| and subtracting from the second row, and multiplying the first row by 1^3, = 0 and subtracting from the third row, we obtain -2 0
I •
0
1
Multiplying the second row by m32 yields
-2
-1 and subtracting from the third row then
-2 U =
(3.5.24)
0 0
1 3 "2 0
0 1 4 — '3
(3.5.25)
65
LU-DECOMPOSmON
With nijj = l,L = [mjj] becomes
L =
1 1 2 0
0
0
1
0
2
(3.5.26)
1
The reader can verify that A = LU. This example serves to illustrate how the LUdecomposition of A can be obtained by simply storing the multipliers m,y when the simplest Gauss elimination process (no row interchanges) works. As an example of how LU-decomposition might be useful, consider the equations Ax, = b,
(3.5.27)
Ax2 = b2(Xi)
(3.5.28)
in which b2 depends on the solution of Eq. (3.5.27). Performing Gauss elimination twice to solve these equations requires twice n(n^ — l)/3 + n(n -f- l)/2 divisions and multiplications, whereas carrying out Gauss elimination to solve Eq. (3.5.27) costs n(n^ — \)/3-\-n(n + l)/2 and yields L and U. Equation (3.5.28) can then be written as LUX2 = b2,
(3.5.29)
and setting y = Ux2, the equation Ly = b2 can be solved by forward substitution for y. Subsequently, the equation Ux2 = y
(3.5.30)
can be solved by backward substitution for X2 at a cost of «(n + 1). Thus, without using LU-decomposition, the solution for x, and X2 costs }n(n^ - l)-\-n{n + 1),
(3.5.31)
whereas by using LU-decomposition it only costs -n(n^ - 1) + -n{n -f 1),
(3.5.32)
a savings on the order of n^/3 for large n. As pointed out at the end of Section 3.3, the simplest Gauss elimination process is guaranteed to work for stricdy diagonally dominant matrices and for positivedefinite matrices. Thus, LU-decomposition is assured for these two classes.
66
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
3.6. BAND MATRICES If the matrix A is such that a^j = 0 for j > i -{- p and / > 7 + ^, we say A is a band matrix of bandwidth w = p -\-q + 1. Examples are
1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1
«,7 = 0 , 7 > I 4- 1, / > 7 + 1, w; = 1,
(3.6.1)
2 5 0 0
au = 0 , 7 > / + 2, / > 7 + 2, w = 3.
(3.6.2)
and 2 2 7 0
0 4 2 6
0 0 9 2
Symbolically, we represent band matrices as in Fig. 3.6.1. In the part of the matrix labeled A, there are nonzero elements. In the parts labeled 0, all elements are 0. When A is banded, n large, and p and q small, the number of operations in the Gauss elimination solution to Ax = b is on the order of npq, which is much smaller than the n^ required for a full matrix. One of the most important properties of a band matrix is that its LUdecomposition, when it exists, decomposes A into a product of more narrowly banded matrices. In particular, if the LU-decomposition can be achieved with Gauss elimination (without row interchange), it follows that the structure of L and U is as shown in Eq. (3.6.3) (see Fig. 3.6.2). Only in the parts of L and U labeled IJ and U' are there nonzero elements. As an example of the simplification resulting
FIGURE 3.6.1
67
BAND MATRICES
(3.6.3)
FIGURE 3.6.2
Equation (3.6.3).
from Eq. (3.6.3), consider the tridiagonal matrix and its LU-decomposition: a, h
q «2 b,
^2 a-.
K-i
a,n-\
^n-\
(3.6.4) 1 Pi
«!
1
ft
5i 0^2
^2
1
0
Of.
ft
1
Carrying out the matrix multiplications on the right-hand side of Eq. (3.6.4) and comparing with the corresponding elements in A, we find 5; = C,
(3.6.5)
and aj = flj, p^ =
, a^ = ajt — Pk^k-i^ fe = 2 , . . . , n.
(3.6.6)
The number of operations needed to find the coefficients a^, )S,, and 5, are only 3(n — I) additions, multiplications, and divisions. As a special case of a tridiagonal matrix, consider 1 - 1 0 0 - 1 2 - 1 0 0 - 1 2 - 1 0 0 - 1 2
(3.6.7)
68
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
With the rules given in Eq. (3.6.6), we find
L =
1 -1 0 0
U
1 0 0 0
0 1 -1 0
0 0 1 -1
0 0 0 1
(3.6.8)
0 0 -1 1
(3.6.9)
and -1 1 0 0
0 -1 1 0
Note that U = L^ for this case. The inverse of L is easy to find. Consider the augmented matrix, [L, I], corresponding to Lx, = e,, / = 1 , . , . , 4, 1 1 0 0
0 1 -1 0
0 0 1 -1
0 0 0 1
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
(3.6.10)
In this case, Gauss elimination gives 1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
1 1 1 1
0 1 1 1
0 0 1 1
0 0 0 1
(3.6.11)
The solutions to Lx^ = e,, / = 1 , . . . , 4, are
rr1 x, =
1
Li_
,
X2 =
"0" 1 1 _1_
,
X3 =
"0" 0 1 _1_
,
X4 —
"01 0 0 __1 J
(3.6.12)
and so the inverse of L, which is just the matrix [x,, X2, X3, X4], is the lower triangular matrix
1 1 1 1
0 0 0 1 0 0 1 1 0 1 1 1
(3.6.13)
69
BAND MATRICES
Since U = L^, it follows that U"' = (L"')^, or 1 1 1 1 0 1 1 1 0 0 1 1 0 0 0 1
U-'
(3.6.14)
By the same procedure, it is easy to show that the LU-decomposition of the n X n tridiagonal matrix 1 -1
-1 2
0 -1 (3.6.15) -1
0
2 -1
-1 2
IS
(3.6.16)
U = L^ 1
1
and 1
1
KT U-' = (L-')
L' = 1
1
(3.6.17)
1
Since the inverse of A is U ^ L ', we find
A-i =
n n — I n — 2 n — 3 •• n — \ n — \ n — 2 n — 3 •• n — 2 n — 2 n — 2 n — 3 ••
2 2 2
1]
••
2 1
1 1
2 1
2 1
2 1
2 1
becomes '5 4 A-' =
3 2 1
4 4 3 2 1
3 3 3 2 1
2 2 2 2 1
1 1 1 1 1
1 1
70
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
An important point for band matrices can be made by comparing A~^ to L and U. A~* is a full matrix, requiring storage of n^ elements. L and U require storage of only 4n — 2 elements. To solve LUx = b, we could solve Ly = b and Ux = y in n(n — 1) multiplications and divisions, and so evaluation of A~^b requires only n^ divisions and multiplications. Thus, when storage is a problem, it is better to find the LU-decomposition of a banded matrix, store L and U, and solve LUx = b as needed. As another example, consider the n x n tridiagonal matrix
2 1
-1 2
0 -1 (3.6.18)
A =
-1
2 -1
0
-1 2
The main diagonal elements are 2, whereas the elements on the diagonals just above and just below the main diagonal are —1. For this case, we find 1
(3.6.19)
L = j - i
1 «-l
and
-1 3 2
0 -1
U =
(3.6.20)
0
1+1
-1
J
n+1
The inverses of L and U are not as easy to compute as in the previous example. However, the solution of LUx = b only requires n(n — 1) divisions and multiplications. Matrix theory plays an important and often critical role in the analysis of differential equations. Although we will explore the general theory and formal solutions to specific differential equations in Chapter 10, we are nonetheless confronted
71
BAND MATRICES
with the cold hard fact that most practical applications in engineering and applied science offer no analytical solutions. We must, therefore, rely on numerical procedures in solving such problems. These procedures, as with the finite-difference method, usually involve discretizing coordinates into some finite grid in which differential quantities can be approximated. As a physical example, consider the change in concentration c of a singlecomponent solution under diffusion. The equation obeyed by c in time t and direction y is dc d^c - = D-,.
(3A2„
where D is the diffusion coefficient. At r = 0, we are given the initial condition c(yj)
= fiy),
(3.6.22)
and at the boundaries of the system, y = 0 and y = L, the concentration is fixed at the values c(y = 0, 0 = a
(3.6.23)
c(y^Lj)^p,
Using the finite-difference approximation, we estimate c only at a set of nodal points y, = / Ay, / = 1 , . . ., /t, where (n -[-1) Ay = L. In other words, we divide the interval (0, L) into n -f- 1 identical intervals as shown in Fig. 3.6.3. If we let Ci denote c(}?,, t) (the value of c at j , ) , then the finite-difference approximation of d^c/dy^ at y, is given by ^^c{yi,t) q_i - Icj -h c^^x —E"l— = 7T~u •
.^ . ^ . . (3.6.24)
With this approximation, Eq. (3.6.21), at the nodal points, reduces to the set of equations ^^\ dx
•—- = a - 2ci -h ^2
^ = c,_, - 2c, -^ q ^ „ ax
/ = 2 , . . ., n - 1,
where x = t Ay^/D.
yo
yi
y2
I
I
I
0
I Ay I
FIGURE 3.6.3
yn-i I
I
j
I
yn Vn+i \
j
(3.6.25)
72
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
With the definitions a 0
^1
and
(3.6.26)
b = 0
the equations in Eq. (3.6.25) can be summarized as dc = -Ac + b, dx
(3.6.27)
where A is the tridiagonal matrix defined by Eq. (3.6.18). The initial condition corresponding to Eq. (3.6.22) is then given by
c(r = 0) =
fiyx) fiyi)
(3.6.28)
L/wJ If we replace the boundary condition c(y = O^t) = a by the boundary condition dc/dy = 0 at J = 0 (barrier to diffusion), it gives for the finite-element approximation Ci = c{y = 0, t)
(3.6.29)
and changes the first equation in Eq. (3.6.25) to dc. dx
= - C i + C2
(3.6.30)
Once again, we obtain Eq. (3.6.27), but the matrix A in this case is the one given by Eq. (3.6.15). We note that, at steady state, dc/dt = 0, and the concentration at the nodal points is given by Ac = b. The formal solution to Eq. (3.6.27) can, therefore, be written as c(r) = c(0)exp(-rA) -h
f exp(-(r - ^)A) J q b .
(3.6.31)
To compute c(r) from this formula, we need to know how to calculate the exponential of a matrix. Fortunately, the eigenvalue analysis discussed in Chapters 5 and 6 will make this easy. As a further example, consider a one-dimensional problem in which the concentration c{x, t) of some species is fixed at jc = 0 and x = a, i.e., c(0, t) = a,
c(a, t) = p.
(3.6.32)
73
BAND MATRICES
Further, the concentration initially has the distribution c(jc, r = 0) = /r(jc),
(3.6.33)
and in the interval 0 < JC < a is described by the equation (3.6.34)
J^T-i^m.
where D is the diffusion coefficient and / ( c ) represents the production or consumption rate of the species by chemical reaction. In general, / ( c ) will not be linear so that analytical solutions to Eq. (3.6.34) are not available. So, again, we solve this problem by introducing the finitedifference approximation. Again, the first and second spatial derivatives of c at the nodal points x^ are approximated by the difference formulas: dc{xj, t) ^ c(xj^^J) dx
c{xj_^,t)
2AJC
(3.6.35)
and d^c(xj,t)
_ c(xj^i,t)~-2c(xj,t)
dx^
+ c{Xj_^J)
i^xf
(3.6.36)
These are the so-called centered difference formulas, which approximate dc(Xj)/dx and d^cixj)/dx^ to order (Axf. With the notation Cj
(3.6.37)
=c(Xj,t),
the finite-difference approximation to Eq. (3.6.34) at the nodal points 7 = 1 , . . . , n is given by - 2c^ + c^-_i dt
+ /(C;).
(AJC)2
(3.6,38)
With the boundary conditions CQ = a and c„_^i = ^, Eq. (3.6.38) can be expressed as the matrix equation — = Ac 4- ——-f (c) -h b, dx D where x ^
(3.6.39)
tD/{Axf
b =
c =
fie,)
a "1 0
fci'
(3.6.40)
f = 0
Lc„.
L^J
I f(c„) J
74
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
and 2 1 0
-1 2 -1
0 -1 2
0 0 0 (3.6.41)
A = 0 0
0
2 -1
-1 2
With the initial condition
h(xO (3.6.42)
c(0) = h(Xn)
Eq. (3.6.39) can be solved to determine the time evolution of the concentration at the nodal points. If the nodal points are sufficiently close together, c{t) provides a good approximation to c(x, t). For a two-dimensional problem with square boundaries, the above problem becomes dc
d^c' dx 2+^)dy^
+ fie).
(3.6.43)
The initial condition is given by c(x,y,t
=0)
=h(x,y),
(3.6.44)
and the boundary conditions are indicated in Fig. 3.6.4. The finite-difference approximation for the square is generated by dividing the square into (n +1)^ identical cells of area (A.jc)^. Numbering the nodal points, indicated by solid circles in the figure, from 1 to n^ as indicated above, we can express the finite-difference approximations of the second derivatives at the ith node as d%
df
'•.+1
- 2c, + c,._, (Ax)'
(3.6.45)
^i+n
- 2c, + c;_„ {Axf
(3.6.46)
With the boundary conditions given above, the matrix form of the finite-difference approximation to Eq. (3.6.43) is — = - A c + ——-f (c) H- b. dr D
(3.6.47)
c and f (c) are as defined in Eq. (3.6.40), but A in this case has five diagonals with nonzero elements and b is more complicated than before. For the special
75
BAND MATRICES
c{x,a,t)
=j{x)
^—-v.v-^—-
—-T
i-l --•:
?+ l
c{0,y,t) = l3{y)
c(a,y,t) = % )
1—--t--—tr--n i___i
___l____4
y
A c{x,0,t) = a{x) -^x FIGURE 3.6.4
case n = 3, the formulas for A and b are 4 1 0 1 0 0 0 0 0
-1 4 -1 0 -1 0 0 0 0
0 -1 4 0 0 -1 0 0 0
-1 0 0 0 0 - 1 0 0 () 0 -1 0 1^-1 0 - 1 -1 4 - 1 0 () - 1 4 0 -1 0 0 4 (3 - 1 0 -1 (3 0 - 1 0
0 0 0 0 -1 0 -1 4 -1
0 0 0 0 0 -1 0 -1 4
(3.6.48)
and " a(x,) + )8(y,) ' a(x2) a{Xj) + S{y^)
Piyd b =
0
yixg)
(3.6.49)
76
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
Although the two-dimensional problem is more complicated to reduce to matrix form, the resulting matrix equation is about as simple as that of the one-dimensional problem. EXAMPLE 3.6.1 (Mass Separation with a Staged Absorber). It is common in engineering practice to strip some solute from a fluid by contacting it through countercurrent flow with another fluid that absorbs the solute. Schematically, the device for this can be represented as a sequence of stages, at each stage of which the solute reaches equilibrium between the fluid streams. This equilibrium is frequently characterized by a constant K. Thus, at stage /, the concentration x^ in one fluid is related to the concentration >?, in the other fluid by the equation (3.6.50)
yi = KXi.
A schematic diagram of the absorber is shown in Fig. 3.6.5, L is the flow rate of inert carrier fluid in stream 1 and G is the corresponding quantity in stream 2. Lx^ is the rate at which solute leaves stage j with stream 1 and Gjj is the rate at which it leaves stage j with stream 2. Thus, if hx^ and gy^ represent the amount of solute in stage j in streams 1 and 2, h and g being the "hold-up" capacities of each stream, then a mass balance on stage j yields the equation of change dXj
h^^s''^ dt dt
--Lxj_i-hGyj^^-Lxj-Gyj,
Input of Stream 1
7 = 1,...,n.
Output of Stream 2
Stage 1 iI
L, xi
G , J/2 Stage 2 k
L, X2
G, yz
' Stage 3
L , X3
I J
G, J/n-l
Stage n — 1 L,
G, Vn
Xn^l
Stage n J, Xn
I
Output of Stream 1 FIGURE 3.6.5
Input of Stream 2
(3.6.51)
77
BAND MATRICES
Combined with Eq. (3.6.50), this expression becomes dx —^ = Lx:^ - (L + KG)Xj + KGx:^^, ax where x ^t/{h-\-gK).\n
(3.6.52)
matrix form, this set of equations can be summarized as dx = Ax + b, dx
(3.6.53)
where LXr,
b =
X =
(3.6.54)
0 and
-{L + GK)
GK
0
L 0
-iL + GK) L
0
GK 0 -(L + GK) GK (L-^GK) (3.6.55)
A is referred to as a tridiagonal matrix because it has nonzero elements only along the main diagonal and along two other diagonals parallel to the main diagonal. Such matrix problems are especially easy to handle. In a typical problem L, G, K, XQ, and y„^i are the givens. Questions asked might be: (1) How many stages n are needed for a given separation jc„? (2) What are the steady-state values of jc, and 3^,? (3) Is the steady state stable? (4) What is the start-up behavior of the absorber? (5) Can cyclic behavior occur? The countercurrent separation process described above leads to an equation like Eq. (3.6.27) with a tridiagonal matrix A in which a^ — —(L + GK), b^ = L, and Ci — GK. At steady state, Ax = —b, where LXQ
0 b = 0 Gyn+\ Consider a process where a liquid stream of 35001b^/hr water containing 1% nicotine is contacted with pure kerosene (40001b^/hr) in a countercurrent
78
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
•
H
TABLE 3.6.1 Existing Nicotine Concentration versus N u m b e r of Theoretical Trays
7 8 9 10 11
0.00125 0.00111 0.00099 0.00090 0.00083
liquid-liquid extractor at room temperature. The dilute partition coefficient for nicotine is estimated as AT = 0.876 (weight fraction nicotine in kerosene)/(weight fraction nicotine in H2O). If the target nicotine concentration in water is jc„ < 0.001, how many theoretical trays are required? To solve this problem, the matrix equation Ax = —b must be solved at different values of n (the number of stages). The matrix A and the vectors x and b are defined in Eqs. (3.6.54) and (3.6.55). Given L = 3500, G = 4000, K = 0.876, X Q = : 0 . 0 1 , and y,^_^_^ =0, the Gauss elimination procedure can be applied to find x(/i) as a function of n. The correct number of theoretical stages is then the minimum value of n such that x„ < 0.001. Table 3.6.1 shows the values of x„ for different values of n. For this system, the required number of theoretical trays is 9. A Mathematica program to solve this problem is given in the Appendix.
3.7. ITERATIVE METHODS FOR SOLVING Ax = b
Cramer's rule and Gauss elimination are direct methods for solving linear equations. Gauss elimination is the preferred method for large problems when sufficient computer memory is available for matrix storage. However, iterative methods are sometimes used because they are easier to program or require less computer memory for storage. Jacobi's method is perhaps the simplest iterative method. Consider the equations J2^ij^j = ^/'
/ = 1 , . . . , /I,
(3.7.1)
for the case a,, / 0, / = 1 , . . . , n. Note that Eq. (3.7.1) can be rewritten as ^i = (-i^u^j
+ ^,j/%^
/ = 1,...,n.
(3.7.2)
ITERATIVE METHODS FOR SOLVING Ax = b
79
If we guess the solution {x^^\ ... ,xl^^} and insert it on the right-hand side of Eq. (3.7.2), we obtain the new estimate [x{^\ . . . , jc^^^} of the solution from
^r^ = ( - £ " " y ^ T + ^ . ) / « n -
r = 1 , . . . , n.
(3.7.3)
Continuing the process, we find that at the (^-|-l)th iteration the estimate is given by
^(M) = (^- X:a,jxf^ + b^/a,,,
/ = 1 , . . . , n.
(3.7.4)
J¥^i
If x^*^ converges to a limit x as ^ -> oo, then x is the solution to the linear equation set in Eq. (3.7.4); i.e., x is the solution to Ax = b. This is called the Jacobi iteration method. If |A| = 0, the method, of course, fails. However, even if |A| =^ 0, the method does not necessarily converge to a solution. We will discuss the convergence problem further, later in this section. In computing x\^^^\ ^2*^^\ • • • from Eq. (3.7.4), we note that when calculating ^(A^+i) fj>Q^ jj^g /ji^ equation the estimates xf^^\ . . . , Jc/i|^^ are already known. Thus, we could hope to accelerate convergence by updating the quantities already known on the right-hand side of Eq. (3.7.4). In other words, we can estimate jc/*^^^ from i-i
"~
'^
' '
/ = l,...,n.
(3.7.5)
This iteration scheme is called the Gauss-Seidel method and it usually converges faster than the Jacobi method. Still another method, which is a variation of the Gauss-Seidel method, is the successive overrelaxation (SOR) method. If we define the "residual" r/^^ as ^ik) ^
(k+l) _
(k)
(3.7.6) ^
7= 1
7=/
^
where xf^^^ and xf^ are Gauss-Seidel estimates, then rf^ is a measure of how fast the iterations are converging. For example, if successive estimates oscillate a lot, then rf^ will be rather large. The SOR method "relaxes" the {k + l)th oscillation somewhat by averaging the estimate xf^ and the residual. Namely, the {k -f l)th SOR estimate is defined by
xf^''=xf'+corf'
0.1.1)
80
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
or Jk+l)
(3.7.8) .(^+1) ; Thus, the SOR estimate of jc,is a weighted average of the A;th estimate x;ik) (weighted by 1 - CD) and the estimate of Jcf "^^^ computed by the Gauss-Seidel method (the coefficient of co in Eq. (3.7.8)). Note that co is an arbitrary weighting parameter—although, as we shall claim shortly, it cannot lie outside the range 0 < ft)< 2 if the method is to converge. For every matrix A, there is a "best value" CO of the SOR parameter for which the method converges the most rapidly. With 0) = 1, the SOR method is the same as the Gauss-Seidel method. The SOR method will converge faster than the Gauss-Seidel method if the optimal o) is used. To illustrate the three iterative methods just described, consider the problem Ax = b, where
A =
4 1 1 0 0
-1 4 0 -1 0
-1 0 4 -1 -1
0 -1 -1 4 0
0 0 -1 0 4
(3.7.9a)
and
b =
(3.7.9b)
Table 3.7.1 shows jcf^ versus iteration k for the three methods using as the zeroth-order estimate x^^^ = 0. The solution found by Gauss elimination is given at the bottom of the table and we will use this as the correct solution x and define the error of the iterative result as error = ||x'(k)
(3.7.10)
We see that, in order to find a solution with error less than 10"'*, it takes 15 iterations for the Jacobi method, 8 for the Gauss-Seidel method, and 5 for the SOR method. The value of co (1.1035) used is optimal for the matrix A. For this matrix, SOR is the best method, Gauss-Seidel the next best, and Jacobi the poorest. Mathematica programs have been included in the Appendix for all three of these methods.
81
ITERATIVE METHODS FOR SOLVING Ax = b
TABLE 3.7.1 C o m p a r i s o n of Rates o f Convergence o f T h r e e I t e r a t i v e Solutions o f A x = b f o r A and b Given in Eqs. (3.7.9a) and (3.7.9b)
x^
k
^2
^3
x>
^5
Jacob! method 1 2 3 4 5 6 7 8 9 10 11 12
0.25 0.375 0.46875
0.5 0.527344 0.536133 0.543945 0.546448 0.548676 0.549389 0.550024 0.550227
0.5
0
0.625 0.6875 0.734375 0.75 0.763672 0.768066 0.771973 0.773224 0.774338 0.774694 0.775012
0.25 0.3125 0.375 0.394531 0.412109 0.417725 0.422729 0.424332 0.425758 0.426215 0.426622
0.25 0.375 0.46875?
0.5 0.5
0.527394 0.536133 0.543945 0.546448 0.548676 0.549389 0.550024 0.550227
0.5625 0.578125 0.59375 0.598633 0.603027 0.604431 0.605682 0.606083 0.60644 0.606554
0.40625 0.508789 0.538635 0.547161 0.549592 0.550285 0.550483
0.515625 0.583008 0.600037 0.604832 0.606197 0.606586 0.606697
0.470081 0.541685 0.550745 0.550605 0.550575
0.572746 0.603268 0.606434 0.606824 0.606743
0.550562
0.606742
0.5
Gauss-Seidel method
1 2 3 4 5 6 7
0.25 0.40625 0.508789 0.538635 0.547161 0.549592 0.550285
0.0625 0.332031 0.400146 0.419327 0.424788 0.426345 0.426789
0.5625 0.703125 0.754395 0.769318 0.773581 0.774796 0.775143
S O R method
1 2 3 4 5
0.275875 0.441528 0.54464? 0.550439 0.550636
0.627857 0.738257 0.77503? 0.775323 0.775309
0.076107 0.401619 0.424549 0.427148 0.427003
Gauss elimination method 0.550562
0.775281
0.426966
The defining equations for Jacobi iteration, Eq. (3.7.2), can be written in vector form as X = BjX-fCj,
(3.7.11)
where Bj = - D
H L + U)
(3.7.12)
and c, = D 'b.
(3.7.13)
82
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
The quantities D, L, and U are given by au11
0 (3.7.14)
D = 0
L =
«31
L^nl
(3.7.15)
«32
^n2
-*n,n—1
and
fo
«12
«13
0
«23
0
U =
*3«
(3.7.16)
0 0
L
D is a diagonal matrix with the diagonal elements of A along the main diagonal. L is a lower diagonal matrix having O's on the main diagonal and the elements of A below the main diagonal. U is an upper diagonal matrix having O's on the main diagonal and the elements of A above the main diagonal. Thus, it follows that A = D + L + U.
(3.7.17)
Clearly, Eq. (3.7.11) is a rearrangement of Ax = b, which can be accomplished only if none of the diagonal elements is 0. Note that L and U defined here have nothing to do with the L and U in our discussion of LU-decomposition. The iterative solution of Eq. (3.7.11) is given by x^^+^> = BjX^^^ + Cj,
it = 0, 1
(3.7.18)
The vector form of the defining equations of the Gauss-Seidel iteration is (I + D L)x = - D ^ U x + D ^ b or ^ "~
"GS^
"^ ^GS»
(3.7.19)
where BGS = - ( D + L ) - ^ U
(3.7.20)
ITERATIVE METHODS FOR SOLVING Ax = b
,
83
and Cos^CD-fLr^b.
(3.7.21)
Equation (3.7.19) is again just a formal rearrangement of Ax = b, which is allowed if A has no zero elements on the main diagonal. D + L is a lower diagonal matrix, which is nonsingular if |D| i^ 0. The corresponding iterative solution of Eq. (3.7.19) becomes x^*+i> = Bosx^*^ + Cos,
^ = 0, 1 , . . . .
(3.7.22)
Finally, the SOR iteration begins with the vector equation X = BSORX + CSOR,
(3.7.23)
BsoR = (D + oA.)~\(\ - ^)D - oy\i\
(3.7.24)
CsoR = (I> + ^ L ) ' ' b .
(3.7.25)
where
and
Equation (3.7.23) is again a formal rearrangement of Ax = b, which is allowed if |D| 7^ 0 or a^^ 7«^ 0, / = 1 , . . . , /2, and the iterative solution to Eq. (3.7.23) is x^*-''^ =
BSORX^'>
+ CsoR,
^ = 0, 1 , . . . .
(3.7.26)
The components of Eqs. (3.7.18), (3.7.22), and (3.7.26) can be shown to correspond to Eqs. (3.7.4), (3.7.5), and (3.7.7), which are the Jacobi, Gauss-Seidel, and SOR equations, respectively. We now need some mechanism to predict the convergence properties of iterative methods. It turns out that a sufficient condition for convergence of the iteration process to the solution x of Ax = b is provided by the following theorem: THEOREM.
//
IIBII < 1, then the process ^(k+i) ^ g^(^) _^^^
it = 0, 1 , . . . ,
(3.7.27)
generates the solution x of X = Bx + c.
(3.7.28)
84
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
To prove the theorem, suppose x is the solution of Eq. (3.7.28). Subtracting Eq. (3.7.28) from Eq. (3.7.27), we obtain x(fc+i) _ X = B(x^*^ - x).
(3.7.29)
Note that x^*^ - X = B(x^^^ - X) x^'> - X = B(x<'> - X) =
B2(X^^^
- X) (3.7.30)
x^^^-x = B*(x^^^-x). From the properties of matrix norms, it follows that l|x^^^ - x|| < IIB^II ||x^^^ - x|| < ||Bf ||x^^^ - x||.
(3.7.31)
Thus, if the norm ||B|| of B is less than 1, Eq. (3.7.31) implies that lim ||x^^^-x|| :=0;
(3.7.32)
i.e., the sequence x^^^ generated by Eq. (3.7.27) converges to the solution of Eq. (3,7.28). We know that if B = Bj, B^g, or BSQR, Eq. (3.7.27) is equivalent to Ax = b, and so the Jacobi, Gauss-Seidel, and SOR processes will generate the solution to Ax = b if the norm of the corresponding matrix B is less than 1. Interestingly, the value of c, and therefore of b, does not affect convergence of the iteration process. •
EXERCISE 3.7.1. Use the /? = oo norm to prove that Jacobi's method con• verges when A is a diagonally dominant matrix. Using the eigenvalue techniques that will be developed in Chapters 5 and 6, we can prove that the SOR method converges if and only ifB^Q^ exists (|D| / 0) and the parameter co lies in the range 0 < a>< 2.
•
(3.7.33)
EXERCISE 3.7.2. If iA| ^ 0 and A is a real n x n matrix, then the problem Ax = b can be solved by the conjugate gradient method. According to this method, we start with the initial guess XQ, compute the vector FQ = b—AXQ, and set Po = FQ. We then iterate llr l|2 _ llr,-IP <*1
p/^Ap,
(3.7.34a)
"i+l
= x,+«,P,-
(3.7.34b)
r.+i
= r, - a/Ap,
(3.7.34c)
l|r,+ilP llr, IP
(3.7.34d)
A
p,+i = r,-+i + AP,-
(3.7.34e)
85
NONLINEAR EQUATIONS
for / = 1, 2 , . . . , « — 1. A Mathematica program implementing the above conjugate gradient method is included in the Appendix. Show that the vector x„ converges to • • • the solution to Ax = b for a variety of matrices A.
3.8. NONLINEAR EQUATIONS Most "real-world" problems are nonlinear. Given this fact, it might appear a bit surprising that linear equations receive so much attention in numerical analysis and applied mathematics. The reason is simple: the most common method for solving nonlinear systems is to do so iteratively through linearized equations. The NewtonRaphson method is frequently the method of choice for such iterative solutions. Another iterative approach is the Picard method, which is easier than, but not as powerful as, the Newton-Raphson method. Both methods require an initial guess of the solution and, as we will see, their success depends on the "goodness" of the initial guess. Although linear equations occasionally admit multiple solutions, they very often have unique solutions. Nonlinear equations, on the other hand, normally admit multiple solutions. For example, the equation jc^-5x2+ 4 = 0
(3gj^
has four solutions, namely, jc = 1, — 1, 2, and —2. Similarly, the equation cos^x-l^O
(3.8.2)
has an infinite number of solutions, namely, x = nn, n = 0,±\, ± 2 , . . . . There are, of course, nonlinear equations that have unique solutions. For example, e~^ - X = 0
(3.8.3)
has the unique solution JC = 0.567.. .. Furthermore, the equation e'""^ —x=0 has a unique solution for any positive real value of a and no solution at all for any negative real value of a. When there are multiple solutions, the particular solution to which an iterative scheme converges will depend on the initial guess. In this section we will concern ourselves with a system of n equations and n unknowns, i.e., /i(xi,...,xj = 0 :
(3.8.4)
/„(xi,...,xj=0, which we can express in vector form as f (X) = 0,
(3.8.5)
where f is a column vector whose components /) are nonlinear functions of the components x^ of the column vector x.
86
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
For Picard iteration, we rewrite f as f(x) = x — g(x), and Eq. (3.8.5) is rearranged to get x = g(x).
(3.8.6)
If we choose the initial estimate of the solution to be x^^\ subsequent estimates are then computed from x(^4-i) ^ g(x(^))^
j^ = l , 2 , . . . .
(3.8.7)
If this iteration converges, i.e., if lim ||x^^+^^-x^*>||=0,
(3.8.8)
then lim^_^^ x^^^ = x, where x is a solution of Eq. (3.8.5). In practical applications, the criterion for convergence is usually that one accepts x^^"^^^ as a solution when ||x(^+i) _ x^^^ll < €, where the tolerance 6 is a small number chosen by physical considerations. The Newton-Raphson method also begins with an initial guess x^^K However, this method is derived by expanding the equations in Eq. (3.8.4) in a multivariable Taylor series about x^^\ obtaining
4-0(x-x^^^)^
(3,8.9)
or, in vector notation, f (X) = f (x^^^) + J(x^^^)(x - x^^^) -f 0(x - x^^^)\
(3.8.10)
The matrix J(x) is called the Jacobian matrix (or just the Jacobian) and has the elements Ju = ^(x,..^.,x,).
(3.8.11)
In the Newton-Raphson method, we assume the higher order terms in the Taylor expansion are small and we obtain the next guess x^^^ for the solution to f (x) = 0 from the expression 0 = f (x^^^) + J(x^2^ - x^^>),
(3.8.12)
which can be solved using Gauss elimination or any of the iterative methods discussed in the previous section. Iteration of this process yields 0 = f (x^^^) + J(x^^^)(x<^+^^ - x^^^),
(3.8.13)
where k = 1 , 2 , . . . . When the process converges, the result x = linif^^^ x^*^ is a solution to f (x) = 0.
NONLINEAR EQUATIONS
87
In practice, the quantity x^*+^^ is accepted as a solution when ||x(^+i) _ x^ii < ^^
and
||f(x^^-^^^)|| < €f,
(3.8.14)
where the tolerances 6^ and €f are small quantities chosen by physical considerations. For example, if |x^^"*^^^|| = 10^, then €^ = 1 could be sufficiently small for practical purposes. On the other hand, if Hx^^'+^^H = 10"^, then e^ should be considerably smaller than 10"^. One way to choose 6^ would be to require €, < 10-^11x^^^11, and likewise €f < 10-^||f(x^'))||. Of course, 10"^ could be replaced by a smaller or larger number, depending on the tolerance one is willing to accept. The Newton-Raphson method will always converge to a solution when the guess x^^^ is sufficiently close to a root of the equation f (x) = 0. This is not so for the Picard method. For instance, for the equation X = - V 3 J C -e'
+ 2,
(3.8.15a)
there is no guess for which the Picard method will converge to the solution, which is jc = —0.390272. Interestingly, if Eq. (3.8.15a) is squared and rearranged to get X=
x^ -\re'
-2
,
(3.8.15b)
then the guess x^^^ = 1 will lead to the convergence of a Picard iteration to the root X = —0.390272. In general, for x = g{x), when \dg/dx\ < 1 for x^^^ in the vicinity of the root x, the Picard method will converge. Otherwise, it is likely to fail. EXERCISE 3.8.1. Show that the value of dg/dx for x near -0.390272 guarantees convergence of the Picard method for Eq. (3.8.15b) but not for Eq. (3.8.15a). Explore how robust the Picard method is for Eq. (3.8.15b) by trying different initial guesses. An important feature of the Newton-Raphson method is that, when it starts to converge, the convergence is quadratic. In other words, if convergence begins and ||x(^+i)_x(*)||^10-^C,
(3.8.16)
where C is some scale factor, then after v more iterations
Thus, at three iterations beyond the ^th iteration, the difference ||x^*+^+''^ -x^^+^'^H is a factor of 10"^ smaller than at the A:th iteration. This remarkable result actually can be used as a test of the correctness of the evaluation of the Jacobian in a problem. Any error in J will reduce the rate of convergence below the theoretical quadratic rate. Picard iteration usually converges at a much slower rate (usually arithmetically instead of quadratically) than the Newton-Raphson method, as illustrated in the following example.
88
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
• • •
EXAMPLE 3.8.1 (Illustration of Quadratic Convergence of the NewtonRaphson Method). Consider the system of equations 0 = Xj tan(xj) — Jx\ — x\ / ^-
tan(x,+7r/4)
:
n
(3.8.18)
V ' 1 ^ 4 ; *
Starting with the Picard method, the residual expressions in Eq. (3.8,18) are rewritten in the proper form as Xx = 8i{xy,X2) =Xx-Xx
tan(xi) + y x | - xj (3.8.19)
«,=«,(„.,,,=,,+,
(''+i)
It is now just a matter of choosing an initial guess and iterating as in Eq. (3.8.7). It turns out that the only choice of initial values that yields a convergent solution using the Picard method is the exact solution (within the requested tolerance). Here is another example of the inadequacy of the Picard method in certain applications. Using the Newton-Raphson method, we define the residual expressions f (X) = 0 from Eq. (3.8.18) fx(xy,X2) =Xx tan(xi) - y x | - xj (3.8.20)
The Jacobian elements are then 9-^1
y/xl - X
Vl
X2
Vz _
-— =
dx,
,
+ Xi sec (jCi) + tan(A:i)
_{x,+iT/4)
^jcl-ix,-^n/4f
- coti X, H — I + (Jc, H — I IX, H — )
V'
4/
V*
4J\'
4/
a/2
a^2
I l l
y/xl-(xx+n/4f
With a tolerance of IQ-^^ (i.e., |x^^+^> - x^^^H < 10"^'^ and ||f || < IQ-^^), the Newton-Raphson method yields the solutions x^ = 0.99279 and X2 = 1.81713. Table 3.8.1 illustrates the convergence properties of the Newton-Raphson procedure for the final five iterations. A Mathematica program for solving this problem is given in the Appendix.
89
NONLINEAR EQUATIONS
TABLE 3.8.1
Residuals and J;acobiam Elements Jacobian
Residuals ||x(fc*')_x(fc)||
||f||
ill
0.6185 X 10 ' 0.6540 X 10-2 0.5087 X 10~^ 0.3951 X 10-» 0.1513 X 10-'5
0.2130 X 10« 0.1696 X 1 0 ' 0.1518 X 1 0 ^ 0.1013 X 10-^ 0.1391 X 10-^''
6.38067 5.58271 5.51184 5.51125 5.51125
/l2
-1.20243 -1.19462 -1.19396 -1.19395 -1.19395
/2I
hi
6.64350 6.77445 6.81964 6.81972 6.81972
-4.55258 -4.80056 -4.85622 -4.85644 -4.85644
EXAMPLE 3.8.2. A common model of a chemical reactor is the continuously stirred tank reactor (CSTR). A feed stream with volumetric flow rate q pours into a well-stirred tank holding a liquid volume V. The concentration of the reactant (denoted here as A) and the temperature in the feed stream are c^^ (moles/volume) and TQ, respectively. We define the concentration and temperature in the tank as Cj^ and r , respectively, and note that the product stream will—in the ideal limit considered here—^have the same values. The exiting volumetric flow rate is q. The situation is illustrated by the diagram in Fig. 3.8.1. We assume that the reaction of A to its products is first order and that the rate of consumption of A in the tank is
-^0 e x p ( ^ ) CAV.
(3.8.21)
where R is the gas constant (Avogadro's number times k^) and k^ exp(—E/RT) is the Arrhenius formula for the rate constant—kQ and E being constants characteristic of the chemical species. Mass and energy balances on the contents of the tank yield, under appropriate conditions, the rates of change of c^ and T with time t: dt
= qi^A, - (^A) - ^oexpf — j c ^ V = / i ( r , c^)
(3.8.22)
and ypCp^
= qpC,{T^ -T)-
A:oexp(
-E c^V{AH)-AUiT-T,) RT) (3.8.23)
The symbol p represents the density and Cp the specific heat per unit mass of the contents of the tank. AH denotes the molar heat of reaction and the term —AU(T ~ T^) accounts for heat lost from the tank to its surroundings (which is at temperature T^), U is the overall heat transfer coefficient and A is the area through which heat is lost. The first question to ask regarding the stirred tank reactor is: What are the steady states of the reactor under conditions of constant q, c^^, TQ, and T^l The answer is the roots of the equations
o=/,(r,c^)
(3.8.24)
90
CHAPTER 3
SOLUTION OF LINEAR A N D NONLINEAR SYSTEMS
Feed Stream
y.cA^T
Q^CA^T
FIGURE 3.8.1
The Newton-Raphson technique reduces finding these roots to a matrix problem. The next question is whether a given steady state (r% c\) is stable to small fluctuations. For small deviations of T and c^ from T^ and c ^ , / i ( r , c^) and /2(r, c^) can be expanded in a Taylor series and truncated after linear terms. For such a situation, Eqs. (3.8.22) and (3.8.23) become dxi dt
O-i^Xi -f- flj2-^2
— a2iXi
dt
+ ^22-^2
or dx ~dt =
j(ci,nx,
(3.8.25)
where X
=
"-A 'T
^A rrs
(3.8.26)
and
-94 Js
9c^
9/, dT
(3.8.27) 9/2
L dc^
97
The steady state is stable if, for an arbitrary value of x at f = 0, Eq. (3.8.24) predicts that x -> 0 as r -> 00. As we shall learn later, the asymptotic behavior of X can be deduced from the properties of the matrix J. Indeed, if all the eigenvalues of J have negative real parts, then x will go to 0 as f goes to 00. Otherwise, x will not approach 0. It is worth noting that the matrix generated in solving Eq. (3.8.27) by the Newton-Raphson technique is the matrix A needed for stability analysis. EXAMPLE 3.8.3 (The Density Profile in a Liquid-Vapor Interface). According to an approximate molecular theory of a planar interface, the density n{x) between the liquid and vapor phase obeys the nonlinear integral equation CO
/
K(\x' -x\)[nix') -OO
- n{x)]dx' = fio{nix)) - n,
(3.8.28)
NONLINEAR EQUATIONS
9 I
where A'CIA:!) is a function of order unity for x on the order of a molecular interaction distance a and which is vanishingly small when \x\ ^ a. fi is the chemical potential of the system and /XQC'^) is the chemical potential of a homogeneous fluid at density n. For the van der Waals model, often used in qualitatively modeling fluid behavior, — -l] + ^ — ^ - 2na, (3.8.29) \nb ) \ —no where a and h (h ^ a^) are parameters of the model. The van der Waals equation of state for the pressure of a homogeneous fluid is fi^in) =
IXHT)
-k^T\nl
p.{n) = — ? — - n^a, (3.8.30) 1 — nb and the value of the chemical potential at the liquid and vapor coexistence densities, «, and Wg, are determined by the thermodynamic equilibrium conditions
Po(«,)=/'o('g (3.8.31) Once /x is fixed, the density versus positions x in a liquid-vapor interface can be determined by solving Eq. (3.8.28). Of course, K must be specified for actual computations. The formula ^:(l^l) = ^ e x p ( ^ )
(3.8.32)
yields qualitatively correct results for a planar interface. To solve Eq. (3.8.28), we assume that, for x > L, n(x) = n^ and, for x < —L, «(jc) = Wj so that the equation becomes f K(\x' -x\)n(x')dx'
- \ r
K(\x'\)dx'\n(x)
- fJio{n(x)) (3.8.33)
= -fj,-n^
K{\x'-x\)dx'-nJ
K(\x' - x\)dx'=
g(x).
Dividing the interval (—L, L) into A^ intervals of width Ax and introducing the approximation (the trapezoidal rule) j
K{W-x,\Mx')dx'
= Y^
AxKi\Xj-x,\)n(Xj)
-h - A ^ [ ^ ( | ^ o - XilMxo) + K{\x^ - Xi\)n(Xf^)], (3.8.34) we obtain for Eq. (3.8.33) Y, KijHj - arii + Pi - ii^ini) = g,-,
(3.8.35)
92
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
Where K,j ^ Ax K(\xj - x^, a ^ / ! ^ KiW\) dx\ A ^ '^{K^^n, + Kj,,n^), n, = n{Xi), and g^ = g(Xi). The matrix formula for Eq. (3.8.35) is Kn - a n + jS - iiQ{n) = g.
I I I
(3.8.36)
Since fio(n) is a nonhnear function of n, an iterative method must be used. The Appendix has a Mathematica program to solve Eq. (3.8.36). Frequently, one is interested in the solution x to the problem f (a, x) = 0
(3.8.37)
for various values of a parameter a. Suppose one has found the solution x(a) for the parameter value a. If or' is another value that is not far removed from a, we can take Xj = x{a) as the first guess in a Newton-Raphson scheme and obtain the solution x(aO by iteration of the equation f (a^ x^*>) + ]{a\ x^'^){x^'^'^ - x^'^) = 0.
(3.8.38)
This is called zeroth-order continuation of the Newton-Raphson technique. First-order continuation is an improvement over zeroth-order continuation. In this case, suppose we have found the solution x{a) from the Newton-Raphson method. Then consider a new value a' = a-f-5a, where 8a is a small displacement from a. We then estimate an initial value of x^^^ of x{a') from the equation f(a, X) + J(a, x{a)){x^'\a') af
Hot'-a)
- x(a))
.
(3.8.39)
— {a\x) = 0. da a'=a, x=x(a)
Since x(a) is a solution for the parameter value a, f (a, x{a)) = 0, and since J(a, x(a)) has been evaluated already in finding x(a), the only work to do in Eq. (3.8.39) to obtain x^^^ is to take the derivatives 3f/3Qr and solve the resulting linear equation for x^^\a'). From here the solution x(a') is obtained by the Newton-Raphson procedure in Eq. (3.8.38). If, instead of a single parameter a, there is a collection of parameters denoted by a, Eq. (3.8.37) is replaced by = 0,
J(a, x(a)){x^'\cc') - x(a)) + X^K' - « / ) ^
(3.8.40)
a'=a, x=x(a)
where the components of a^ are of, + 5a,. Again the solution x(a') is found by iteration of Eq. (3.8.38). Continuation strategy is often used when there is no good first guess x^^^ of the solution at the parameter value(s) a of interest. If, however, at a, a good guess exists, then the solution x(a) is found and zeroth- or first-order continuation is used repeatedly by advancing the parameter(s) by some small value 8ot until the desired value a is reached. The following example illustrates the value of using continuation with the Newton-Raphson technique.
NONLINEAR EQUATIONS
• • •
93
EXAMPLE 3.8.4 (Phase Diagram of a Polymer Solution). In the Flory-Huggins theory of polymeric solutions, the chemical potentials of solvent molecules and polymer molecules obey the equations /x, = kTInx^ -\-kT(\ - - ) X 2 + 6JC^
(3.8.41)
fji2'=kT\nx2 + kT(l -v)x^
(3.8.42)
+ v€JCp
where x^ is the volume fraction of solvent, jCj is the volume fraction of polymer, and V is the molecular weight of the polymer measured in units of the solvent molecular weight. 6 is a polymer-solvent interaction parameter, k is Boltzmann's constant, and T is the absolute temperature. Note that the volume fractions sum to unity, i.e., jc,-fjc2 = l .
(3.8.43)
Solution properties are determined by the values of the dimensionless parameters V and X = kT/€. At certain overall concentrations and temperatures, the solution splits into two coexisting phases at different concentrations. From thermodynamics, the conditions of phase coexistence are that
l^2\-^\
' -^2 ^ — M2V^1 ' ^ 2 ) '
where x" and jcf are the volume fractions of component / in phases a and p. As the parameter x increases (temperature increases), the compositions of the phases become more and more similar until, at the critical point, 1 ^2c = 7 — 7 =
1 and
-
=.
1/ (1 +
1 \2 ) ,
(3.8.45)
and the phases are indistinguishable. Above jCjc, there are no coexisting phases. For the polymer scientist, the problem is to solve Eq. (3.8.44) to determine the two-phase coexistence curve as a function of x- If the variables x = x^, y = xf, and fi = ^ h ( ^ ' 1 - ^) - i^i(y^ 1 - >')]
(^•8-46)
are defined, the problem becomes f = 0,
(3.8.47)
where the components of f are /i and /2. The range of values of molecular weights V of interest to the polymer scientist are typically v > 100. Solving Eq. (3.8.44) requires a good first guess, which is difficult to obtain for y > 100. On the other hand, for v = 1, Eq. (3.8.44) admits the analytical solution 1-2JC ^ " ln[(l-jc)/jc] y — l—x.
(3.8.48)
94
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
This suggests using the Newton-Raphson technique with continuation from v = 1 to find the coexistence curve for values of v of interest to the polymer scientist. The following routine solves for the coexistence curve at some given v (in this example, the goal will be to find the phase diagram for v = 500). The routine can be broken down into two parts, each of which uses first-order continuation. The first part uses the solution at y = 1 (at a particular choice of y) to determine the first set of points (x,y,x) foi* the coexistence curve at v = 500. This is accomplished by a succession of first-order continuation steps, followed by the Newton-Raphson technique at incrementing values of v. The incrementation of v must be chosen small enough to assure convergence of the Newton-Raphson steps. Once a solution is found at v = 500, the second part of the program uses the Newton-Raphson technique to solve (3.8.44) at several values of y, again using first-order continuation to construct the initial guesses x and x for ^^^h value of y. Alternatively, we could have chosen values of x and then computed values of x and y. However, since the y values sometimes approach unity, where the Jacobian is singular, the problem is better behaved when y is fixed instead. The derivatives needed for the calculation are Vi dx
1 X
/. \
1\ v/
+ (l-v) +
^ = _dx 9/l ^
2(1-X) X
I—X X (1-^)2 _ ( l _ y ) 2
9X
X'
9/2
v(x^ - y') 2
'' dy
' y
(3.8.49) \
vJ
X
fi = l-y J _ - ( l - v ) -X^ dy dv
v^
- — = -(x-J)-fdv
. X
The solution is shown in Fig. 3.8.2, where we present a plot of x versus solvent volume fraction in the coexisting phases for y = 1, 50, and 500. At a given value of X below the critical point, there are two values of jcj, corresponding to the left and right branches of the curve in Fig. 3.8.2. These values of JCJ are the volume fractions of solvent in the coexisting phases a and p. It is interesting to note that for V = 500 not far below the critical point are two distinct coexisting phases having very little polymer. A Mathematica program to solve this problem is given in the Appendix. Here we outline the program logic in case the reader wishes to program the problem in some other language.
95
NONLINEAR EQUATIONS
0.6
0.2
0.4
0.6
0.8
1.0
Solvent concentration (.r,i/) T"
X
1.4
0.6
0.7
0.8
0.9
1.0
Solvent concentration {x^y) 1.850 i/ = 500
critical point,
1.825 1.800 1.775 1.750 1.725
0,85
0.90
0.95
1.00
Solvent concentration (x,y) FIGURE 3.8.2
Two-phase coexistence curves for polymer solutions.
We begin the program by defining the user input parameters. These are the variables of the program that both define the problem and establish the accuracy desired. The variable nucalc is the target value of v used for the calculation of the coexistence curve and deltanu is the step size used in the first continuation sequence from v = 1 to v = nucalc. The value of ymax defines the maximum desired value of y on the coexistence curve and ydelta is the step size used for the incrementation of y (in the second continuation sequence): nucalc — 500 deltanu — 0.005 yvnax = 0.999 2/de/^a = 0.001.
96
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
Next, the parameters specific to the Newton-Raphson routine are defined. TOLERANCE is used in checking for the convergence of the solution and MAXITER sets the maximum number of iterations allowed in the calculation: CONVERGE = 10-^ MAXITER
= 100.
The solution arrays are then initialized to their values for v = 1. Here we are defining y{\] as ymax and when the coexistence curve is eventually calculated, y{i^ will take on values of y as it is deincremented by ydelta. The values of x[i] and XU] are the corresponding solution values for y[i]. Initially, the array need only be defined for / = 1: yU] = ymax x[l]=l-y[l] X[l] = (1 - 2^[l])/ln[(l - x[l])A[l]]. Before beginning the main program body, the initial value of v must be set to 1 and the number of v steps (nustep) must be calculated for control of the loop: v=1 nustep = (nucalc — 1.0)/deltanu. The Jacobian elements for the first continuation step must also be calculated (at V = 1): jacobian[l, 1] = —dx jacobian[\, 2] = —jacobian[2, 1] = —dx a/2
jacobian[2, 2] = — . The loop for the first continuation sequence begins. Here we let k range from 1 to nustep: For k = I, nustep. We now compute the residual vector for the next continuation step: residual[\] =
-deltanu dv
residual[2] =
a/2
dv
deltanu,
NONLINEAR EQUATIONS
97
and then compute the solution to the linear equations in Eq. (3.8.40) using a Gaussian elimination routine, xdel is the solution array containing (x^*"^^^ — X^*^) and (jc^''"^*^ — jc^*^) as elements: xdel = LinearSolveijacobian,
residual).
The values of x[l] and jc[l] are then updated and the value of v is incremented in preparation for the Newton-Raphson loop: xW = xm + xdel[l] x[l] = x[l] + xdel[2]. Before beginning the Newton-Raphson loop, some internal variables need to be initialized. The variable normdel will be used to store the value of the modulus of xdel and normres will be the modulus of the residual vector residual. Both will be compared to TOLERANCE in order to determine if convergence has been obtained. normdel = 0 normres — 0. Also, the value of v must be updated: V = V -f deltanu. The Newton-Raphson routine first computes the updated components of the Jacobian matrix and the residual vector. Then it solves for the vector xdel using the Gaussian elimination subroutine. The respective moduli normdel and normres are then computed and the solutions x[l] and x{\] are updated. The if-statement checks for convergence using the criterion that both normdel and normres must be less than TOLERANCE. The loop is repeated until convergence is obtained or the maximum number of iterations is reached: For i = 2,
MAXITER
jacobian[l, 1] =
dx
jacobian[l, 2] =
dx jacobian[2, I] = 9A dx
jacobian[2, 2] =
residual[l] =
9X
-Mx[l],y[llxUlv)
residual[2] = - / , ( ; c [ l ] , ^[1], x[l], v) xdel = LinearSolveiJacobian, residual) normdel = y/ixdel[\]'^ + xdel[2f)
98
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
normdel = y/{residual[l]^ + residualllf-) xm = xU] + xdel[l] x[l]=x[l]-hxdel[2] If (normdel < TOLERANCE) (normres < TOLERANCE),
and then {exit loop)
end loop End loop over k: end loop At this point, a solution has been found at the target value of v = nucalc. The second continuation sequence now begins for the calculation of the coexistence curve. The first step is to find the minimum value of y for our deincrementation {ymin). This value corresponds to the critical point values given in Eq. (3.8.45). ymin = 1 — 1/(1 4- -/v). Now we need to find the number of deincrementation steps in y to take (nmax). The function Floor is used to find the maximum integer less than its argument and subtracting the small number 10~^^ is required to ensure that the critical point is not included in our count: nmax = F\oor((ymax ~ ymin)/ydelta — 10"^^) + 1. Next, the calculation of the coexistence curve proceeds by utilizing successive firstorder continuation steps with respect to y. This is done by looping over the nmax values of y[i]. For / = 1, nmax. Calculate the current value of y: y[i] = ymax — (i — \)y delta. Conduct first-order continuation and update of x and x * residual[l] = residual[2] =
—y delta 9/2
y delta
^yi-i
xdel = LinearSolveiJacobian, residual) X[i]^Xli-l]-\-xdel[\] x[i]=^x[i - I]-h xdel[2].
NONLINEAR EQUATIONS
99
Begin the Newton-Raphson loop: normdel = 0 normres = 0 For
i=j,MAXITER dU jacobian[l, 1] = -^— dx a/i jacobian[l, 2] = — a/2
jacobian[2,1] = — dx jacobian[2, 2] = — residual[\] = - / i ( x [ / ] , y[/], xLH, v) residual[2] = —/i(jc[/], j[/], / [ / ] , v) xdel = LinearSolveijacobian,
residual)
normdel = y (a:de/[l]^ + irrfe/[2]^) normdel — yf {residual[\Y + residwtt/[2]^) X\i] = X{i] + xdel[\] x[i] = A^[/] + xdel[2] If (normdel < TOLERANCE) (normres < TOLERANCE),
and then(exit loop)
end loop end loop Finally, calculate the values of jc, y, and x at the critical point: x[nm,ax + 1] = ymin y[nmax + 1] = ymin • • • • • •
xlnmax + 1] = (l + l/\/(v + deltanu)f/2. ILLUSTRATION 3.8.1 (Liquid-Vapor Phase Diagram). In units, the van der Waals equation of state is 8j where
P
3
T
dimensionless
(3.8.50) V
100
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
P, r , and V are pressure, temperature, and molar volume, respectively; and P^, T^, and Vc are the respective values of these quantities at the liquid-vapor critical point (the point at which liquid and vapor become indistinguishable). The conditions for the critical point are dx
dx^
(3.8.51)
= 0.
(i) Verify that these equations imply that jc = j = z = 1 at the critical point. The dimensional van der Waals equation is given by
P = JL.^^
(3.8.52)
V-b V^ where R is the universal gas constant and b and a are molecular constants characteristic of a given fluid. (ii) Find P^, T^, and V^ in terms of R, b, and a. The chemical potential /JL of a van der Waals fluid is given by ix = V~b
\b
)
V
(3.8.53)
At liquid-vapor equilibrium, the pressure and chemical potential of each phase are equal, i.e..
p(y„r)-P(K,r) = o
M(v„r)-M(K,r) = o,
(3.8.54)
where V^ and V^ are the molar volumes of each phase. (iii) From these equilibrium conditions, find Vj and V^ as a function of temperature for temperatures less than T^. The problem is easier to solve in dimensionless variables. First, define w = lil\x^ and X =
•^1
(3.8.55)
where x, = Vi/V^ and x^ = Vy/K' ^^^ /i == ^(^1, y) - zix^. y)
(3.8.56)
/2 = u;(^i, y) - w{x^, y). Now let f =
/i
h
(3.8.57)
and thus the equiUbrium equation to solve is f (X) = 0.
(3.8.58)
101
NONLINEAR EQUATIONS
PIPc
F I G U R E 3.8.3
Two-phase coexistence curves for van der Waals fluids.
Use the Newton-Raphson method to solve this equation for values of 0.50
PIPc T
V/Vc FIGURE 3.8.4
P-V isotherms for van der Waals fluids.
102
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
water cooler
ambient air at To
hot furnace
graphite rod
d
^""""•^ cladding i
•
>
T
'
JL
FIGURE 3.8.5
(v) In the van der Waals model, h denotes molecular volume and a/b denotes the strength of pair intermolecular attraction. Explain qualitatively, from your dimensionless phase diagram, how an increase or decrease in molecular size or intermolecular interaction affects the phase diagram of a van der Waals fluid. (vi) The fact that all phase diagrams of all fluids obeying the van der Waals equation of state reduce to the same dimensionless phase diagram is known as obeying the law of corresponding states. State this law physically and explain how • I one would use it. ••I
ILLUSTRATION 3.8.2 (Cooling a Graphite Electrode). Consider the graphite electrode illustrated in Fig. 3.8.5. The electrode, of diameter d, passes from a hot furnace into a water cooler to help control the electrode temperature. The temperature t where the electrode leaves the furnace (at position jc = 0) is r(0) = 1600''C. The rod resides for a distance L in ambient air at temperature TQ = 20°C. In this space, the electrode is covered with an insulating cladding, which gives an overall effective heat transfer coefficient of U (between the air and the electrode). It is desired that the water cooler keep the temperature of the rod at 160°C at the point where it enters the cooler (at x = L). What is the rate that heat must be absorbed by the water cooler to hold T{L) = 160°C. Assuming that the temperature of the electrode varies only along its axis, it can be shown (see p. 41 of V. G. Jenson and G. V. Jeffreys, Mathematical Methods in Chemical Engineering, Academic Press, 1977) that it obeys the heat equation
ih-otT)
d^T
{dT\ dx )
PiT-T,)=0,
(3.8.59)
where IcQ — aT is the temperature-dependent thermal conductivity of graphite, and P = 4U/d. The parameters of the problem are f/=:1.7WrCm^ ito = 152.6 W/°C m^ a = 0.056 W/(X)^ m J = 10 cm L = 25 cm 20°C
NONLINEAR EQUATIONS
103
7(0) = 1600°C T(L) = 160"C. The rate of heat transfer to the water cooler is
/jtdVdT
e = -(A:o-ar)f — j —
atJc = L.
(3.8.60)
The heat equation can be solved by two different methods. The first is accomplished by transforming the differential equation to an integral equation. Define u by dx so that, by using the relation d{dT/dx)
= {du/dT)dT,
d^T dx^
we obtain
du dx
Then the heat equation becomes 1 diii^) ^ — (/ C o ccT)-^ -u^~ 2a ^ " ' dT which can be rearranged first to d{ii^) dT
la KQ
—
2 u —
aT
6 ^{T - To) = 0, a
(3.8.61)
ofl ' ^ " ^ 0 ^^ kQ — aT
(3.8.62)
and then to 1
iK
d
IT
:((*o
-aTfu ')
T -- 7 - 0 A-o
(3.8.63)
•aT
This expression integrates to (*o - ctTfu^ = P{T - T^fiko - ocTo) - ^{T
- T„)' + C,
(3.8.64)
where C is a constant of integration. Recall that u = dT /dx. With this substitution, the above expression rearranges to dx = f(x)dT, which can be integrated to obtain jc = /
7-pj dT = 0.
•^no) (c +
(3.8.65a)
p(k^-aTy-laP{T-T^)y^
The constant of integration, C, is determined from
/(c, = t - r
•^no) i^c +
fc^^:^
5, .r = 0.
Piko-aTf-lap{T-T^)Y
(3.8.65b) (i) Calculate C by first plotting / ( C ) versus C and finding the interval of C in which / ( c ) switches from negative to positive. Then use the Picard method or the Newton-Raphson method to precisely determine C such that / ( C ) = 0. In evaluating / ( C ) , use the trapezoidal rule for computing an integral.
I 04
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
(ii) Once C is determined, calculate and plot x versus T, again using the trapezoidal rule. Determine the required cooling rate for the water cooler. Another way to solve the heat equation is to discretize it using the finitedifference approximation. Let J C , = / A J C , where Ax = L/{N + 1) and / = 0, 1 , . . . , A^ -M. Next, let 7; = r(x,), / = 0 , . . . , A^ + 1. The midpoint approximations to the first and second derivatives of 7 , evaluated at jc,, are dT dx
2^x
(3.8.66)
and d^T dx^
Ti+y - 27;. + 7;._,
i^xf
(3.8.67)
With these approximations, the heat equation becomes
<*. - "T.) ( ' " ' ( l y ^ " ' ) - ^ C - , . . - T,.,f - m - r.) = 0 (3.8.68) for i = 1 , . . . , A^. The boundary points are given by
(3.8.69) and
,^ - »^.)(""-'"(iy''-) ^ ^ ( n « - r,.,,' - «r. - r.) = o. (3.8.70) (iii) Solve this nonlinear algebraic system for 7] using the Newton-Raphson • • • method and the above parameters. Compare the computation time for each method. • • •
ILLUSTRATION 3.8.3 (Multireaction CSTR at Constant Temperature). Consider the CSTR analyzed in Example 3.8.2. We saw how the steady-state concentration could be determined by using the Newton-Raphson scheme. In the following problem we will solve for the steady-state output concentrations of a multicomponent mixture of reactants and products in a constant-temperature CSTR. The chemical species (A, B, C, and D) are assumed to be in dilute aqueous solution with input concentrations c^^, c^^, CQ, and Cjy^ (mol/L). The total volumetric flow rate is Q (L/s) and the reactor volume is V^ (L). The following reaction set applies: A + B-^C
(1)
B-\-C -^ D
(2)
D -^ B -h C,
(3)
105
NONLINEAR EQUATIONS
with corresponding rate equations
'*2 = V B ^ C
(3.8.71)
where r, is the rate in moles/second per volume of reaction /. For an ideal CSTR, the output concentrations are assumed to be equal to the mixture concentrations within the reactor (perfect mixing). Mole balances for each of the species can then be approximated as Component A Component B Component C
0 = Q(Cc^ - Cc) + (k^c^Cji - k^c^Cc + ^3^0)K
Component D
0 = Q{Co^ - Co) + {k^c^Cc - k^Co)Vr' (3.8.72)
The problem can be recast into dimensionless form by introducing the following quantities: Q Q* ^ TTf—
(3.8.73)
k; ^ ^
(3.8.74)
c>-^.
(3.8.75)
Only A and B are supplied to the reactor. Thus, the mole balance equations can be rewritten in dimensionless form as /, = 0 = e * ( l - c * ) - f ( - c * c * )
/2 = 0 = Q*{cl^ - c*) + i-c\cl - ^*4c* + k^cl)
(3.8.76)
/3 = 0 = Q*(-4) + (c^cl - klclcl + k^cl) /4 = 0 = (3*(-4) + {klc^cl - klcl). The next step is to define the problem in vector notation. By relabeling the dimensionless reactor concentrations as {A:,, ^2, X3, x^], i.e.,
(3.8.77)
X =
D J
we can recast the problem as f (X) = 0, where f is defined in Eq. (3.8.76).
(3.8.78)
106
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
Feed Stream V ? ^i^o
:n
/
^" Recycle Stream
/
Vr,T U
~
-
Qout') ^U 1
(l-a)Q»«<
Precipitator
C
FIGURE 3.8.6
(i) Use the Newton-Raphson method to solve for the output concentrations of the reactor when Q* = l,^ = 5,k^ = 10, and c^ = 1. (ii) The desired product is C, which is removed through precipitation from the reactor output stream. Note that D may be recycled to the reactor, possibly increasing the yield of C. Figure 3.8.6 illustrates the flow diagram. The precipitator is assumed to remove component C from the output stream without changing the total volumetric flow rate. We define the dimensionless output flow rate by Ql^^ = Gout/(K^i<^/io) ^"^ ^^^ recycle fraction a as the fraction of the reactor output flow rate that is recycled. Reformulate the problem in dimensionless parameters and assume the same input values of 2* and Cg . Solve the new system of equations for various values of 0 < a < 1 using zeroth- or first-order continuation. Plot the dimensionless production rate of C (GoutCj) as a function of a. ILLUSTRATION 3.8.4 (Temperature Profile in a Plug Flow Reactor). It is desired to calculate the adiabatic temperature profile of a plug flow reactor for the gaseous phase reaction A+B
(3.8,79)
The reactor, illustrated in Fig. 3.8.7, is loaded with catalyst beads and has an effective cross-sectional area of S^. We define the total molar flow rate through the reactor as A^ and the individual component flow rates as A^, such that
The initial flow rates are designated as {N^ , Ng , N(^ = 0 } . We define the steadystate conversion rate, which is a function of position along the catalyst bed, as $(jc). Mole balances on each species yield
(3.8.80)
Ncix)=^ix) and (3.8.81) where the total initial molar flow rate is NQ = Nj^^ + Ng^.
107
NONLINEAR EQUATIONS
input stream
output stream
catalyst bed Se
h
0
x= L
FIGURE 3.8.7
The overall rate of reaction at a point x in the catalyst bed is ,(;,) = 1 ^ =
^_NA(X)__NBM_
S^dx
Vix)N(x)V(x)N{xy
(3.8.82)
where V{x) is the molar volume of the gas mixture given by (3.8.83)
P is the pressure (assumed constant), R is the universal gas constant, and T(x) is the reactor temperature. The differential equation describing § is, therefore.
dx
^^^^{RT(X))
(3.8.84)
with the initial condition that ^(0) = 0. If the reactor were operating isothermally, we would simply integrate Eq. (3.8.84) to generate an expression for ^(x). However, the temperature dependence must be determined through an enthalpy balance. Defining the molar heat capacity of component / as C,, the initial mixture temperature as 7], and the molar heat of reaction as H^ determined from the reference temperature T^, the enthalpy balance can be expressed as 0=
{T,-T,){C^N^^-\-C,N,^) (3.8.85)
We can recast these equations in dimensionless form by introducing (3.8.86)
r= a = N A(i
kS^L [ P V
(3.8.87) (3.8.88)
(3.8.89)
108
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
^4
(3.8.90)
c
(3.8.91)
r* — —H
H.
H* =
(3.8.92)
where L is the total length of the catalyst bed. Equation (3.8.84) is then
(i-r)(«-n *\2
(3.8.93)
(l+a-n
(i) For the isothermal case, ^ is a constant and equal to 1. Equation (3.8.93) can be integrated directly to yield the following implicit expression for ^*: (v2
/
^3
\
c* = r + -; ln|l - n - -; + (I + a ) ' + a In a - r a 1 —a \ l —« /
(3.8.94)
where a ^ I. Plot ^* versus KX* on the interval 0 < JC* < 1 for several values of a. (ii) For the nonisothermal case, Eq. (3.8.85) can be written in dimensionless form as 0 = l + a C ; - ^ - ^ ( l - r + C;(a-n)
i-o.
(3.8.95)
9-0, C * r - H*H\
• •
where 0, = TJT^i^\, Solve the coupled equations (3.8.93) and (3.8.95) by either the trapezoidal rule or the finite-difference approximation for the dimensionless parameters AC = O.l, a = 0.5, C% — 1.2, CJ = 0.85, 0, = 0.25, and H* = 2.8. Plot the dimensionless conversion ^* and reactor temperature profile ^ for 0 < JC* < 1. Compare the output conversion to the isothermal case, (iii) Derive a solution for Eq. (3.8.93) for the isothermal case when or = 1. Plot §* versus KX* for this case.
PROBLEMS 1. Use Gauss elimination to solve Ax = b, where A =
1 2 3
2 3 1
3 4 2
and
-4 -5 0
109
PROBLEMS
2. Find the LU-decomposition of the matrix A in Problem 1. 3. Consider the matrix
A =
3 -1 0 0 0
-2 3 -1 0 0
0 -2 3 -1 0
0 0 -2 3 -1
0 0 0 -2 3
Without the aid of a computer: (a) Find the inverse A"^ of A. (b) Evaluate the determinants of A and A~^ (c) Solve Ax = b, where ^, = 1, / = 1 , . . . , 5. 4. Write a computer program to carry out Gauss elimination with partial pivoting. 5. Consider heat transfer in a square slab of length 10 in. on a side. At steady state, the temperature obeys the equation 0. Assume that the temperatures of the boundaries of the plates are controlled so that r(jc,0) = 370K(jc/10in.) T{0,y) = lOOKCv/lOin.) r(jc, 10 in.) = 100 K(l - jc/10 in.)^ r(10 in., y) = 370 K(l - y/10 in.)^
(a) Verify the finite-difference forms given for A and b in Eqs. (3.6.40) and (3.6.41). Generalize the results to an arbitrary subdivision of the square into n cells. (b) Using your Gauss elimination program, compute the temperature distribution by the method of finite differences. Estimate the discretization n for which the computed temperature obeys the accuracy criterion -10 AO
/
/
AO -10
T^"\x, y)dxdy-
JQ JQ
/
r^"-^^(jc, >^) dx dy
JQ JQ AO -10
< 10"^ /
/
T^"-^\x,y)dxdy,
where T^"^ and T^" ^^ are temperatures computed from successive discretization.
I I0
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
6. Solve the problem Ax = b, where A is a 100 x 100 tridiagonal matrix with -|-5 on the main diagonal, —3 on the diagonal just above the main diagonal, and —2 on the diagonal just below the main diagonal, and b is a vector with all elements equal to 1. Calculate x using the following methods: (a) (b) (c) (d)
Simple Gauss elimination. The Jacobi method. The Gauss-Seidel method. The SOR method.
Compare convergence properties and total evaluation times for each method. 7. Consider the system Ax = b, where 10"^jci + 10~^jC2 + JC3 = 2 lO'^jCi -
1 0 - % 4- X3 = - 2 X 10-^
jc, + JC2 + 2:^3 = 1.
(a) Does the system have a solution? Why? Using three-digit floating-point arithmetic, solve the system by: (b) Simple Gauss elimination without pivoting. (c) Gauss elimination with complete pivoting. (d) What is the exact solution to six-digit floating-point arithmetic? 8. (Least Squares Method). It is frequently known that some quantity p depends linearly on a set of variables, say 5 ^ , . . . , s„, i.e., n
P = Y,XjSj,
(1)
where the x^ are constant coefficients. In the language of data fitting (statistics), Eq. (1) is referred to as a linear regression formula. Suppose that in an experiment run m times (where m does not necessarily equal n), the values of /?, ^ i , . . . , 5„ are measured. We would like to use these data to estimate the values of the coefficients [xi,... .x^}. For this, we define m /
n
\ 2
i=\ \
;=1
/
(2) If the measurements could be done with infinite accuracy and if the regression formula (1) were indeed obeyed by p and the s^, then for each run / the equation n
II
PROBLEMS
would hold and the quantity L would be identically 0. Any error in measurement would give rise to a violation of Eq. (3) and yield a positive value of L. L provides a collective measure of the error or scatter of data from all the runs. To determine the "best" set {.t,,..., x„} of coefficients from m experimental runs, we choose the coefficients so that L is a minimum. This procedure is known as the method of least squares for fitting a linear regression formula. (a) To minimize L, the coefficients must be chosen such that dL
0,
k = \,.
,«,
(4)
By defining the m x n matrix C such that c^j is the value of s^ measured in the ith experiment, prove that the condition in Eq. (4) leads to the matrix expression
where P\
(b) Use your Gauss elimination program to solve the equation when n = 5, m = 7, and the experimental data are as listed in Table 3.8. Indicate the degree of scattering of the data. 9. Consider the following three equations of state for single-component fluids: Redlich-Kwong: P =
RT y-b
Soave: P =
RT V-h
s/TV{V+b)
V(V-f^)
Peng-Robinson:
RT V -b
aa
V^ + 2Vb-b^'
Using the least squares method and the Newton-Raphson method, generate the "best fit" parameters for methane for each of the above equations of state using the experimental data given in Table 3.R9. Use your calculated parameters to plot P versus V for methane at T = 350 K for each equation of state. Include the above experimental values in your plot. 10. Consider the structural framework shown in Fig. 3.R10, consisting of n rods attached at a common pin joint at A. The rods are free swiveling at their attachment points and at the pin joint. We assume an external force f is applied at the pin joint, and denote {t[ ,tj as the tension of rods 1 , . . . , n and {^j,..., ^„} as their orientations with respect to the x axis.
12
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
TABLE 3.R9 Data
T(K)
1 2 3 4 5 6 7 8 9
200 200 200 300 300 300 400 400 400
P (bar)
1.0
V (Ug mol) 16.52 1.553 0.7149 24.90 2.452 1.206 33.24 3.310 1.648
10.0 20.0
1.0 10.0 20.0
1.0 10.0 20.0
Letting /j and /2 denote the x and y components of f, a force balance at A yields St = f,
(1)
where S is the 2 x n structure matrix S =
cos 01 sin ^1
COS O2 COS ^3 sin O2 sin 0^
cosO„ sin^„
(2)
and t is an n-dimensional vector with components ?,,...,?„. With no applied force (f = 0), when none of the rods is under any tension, each of the rods is of some equilibrium length {/p /2, • • •, In) ^nd equilibrium orientation [0^, 0^,..., 0^}, (a) Defining the two-dimensional displacement vector d as the x- and 3;-coordinate values of the displacement of the pin joint (A) from its equilibrium position, and the ^-dimensional strain vector e as the
FIGURE 3.RI0
PROBLEMS
I I3
elongation of the rods under tension (e^ = /, — /?), prove that e = S^d in the "small strain" approximation (i.e., 0^ ^ 09). (b) If the rods are assumed to be Hookean substances, then the strain 6, = /, — /? obeys the linear law e, = A;,r,, where k^ is the flexibility constant (k^^ is the elastic modulus) of the /th rod. In matrix notation, e = Kt, where K is the matrix of materials constants
K =
ki
0
0 0
k2
0 0
0
^3
0
0
0
•• • • •
^1 ^
0 • •
KJ
Show that the strain d can be related to the external force f by Ad = f, where A = SR-^S'^. (c) Consider the case of five rods all made of steel of elastic modulus 207 X 10^ N/m^ (note the units). The above elastic modulus is defined by the relation cr = Ee, where a has units of stress (N/m^), e is the strain (€, = (/, — l^)/l^) in dimensionless form, and E is the elastic modulus. Assume that all of the rods have 0.1 cm^ cross-sectional area, and the attachment pins are separated by 2 cm (i.e., 2 cm apart) along the wall. When / = 0, the point A is at (8 cm, 7 cm) where the origin is located on the wall at the point of attachment of the lowest rod and the y axis is along the wall. If each of the components of the displacement vector d undergoes a 1% increase, find the force required and the tensions in each of the rods. (d) If the force applied at point A is given, set up the algorithm to find the displacement vector. Hint: Note that SK~^S^ is set up in terms of the ^,'s, which are unknown quantities. (e) Assume that a force of 50 kN is applied at A at 45° to the horizontal in the downward direction. Use the algorithm set up in part (d) to obtain the displacement vector d. Also find the new lengths of the rods and the tensions in each of them. Use the Newton-Raphson method to solve the system. 11. Solve the following equation: X = I -{- exp(—flx/jc) by (a) Picard iteration (error criterion: (|A:^*+" - jc^*'| < 10"')). (b) The Newton-Raphson method (error criteria: |x**+" — JC***| < 10"'* and |/(;c<*'|< 10-*). (Let fl = i )
14
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
12. Use the Newton-Raphson method with first-order continuation to find x as a function of a from the equation X = I -{• exp(—aVx). 13. Consider the system of equations 0 = -jc + h^{exp(yy/x) + 3x^) 0 = ->' -h 0.5 + h^ tan(exp(jc) + J^). Use the Newton-Raphson method to solve these equations for /i = 0.1 and 0.2. 14. Consider the concentric cylinders shown in Fig. 3,R14. Two fluids, labeled A and B, are separated by a meniscus, which is pinned on the tops of the inner (z = a,r = R^) and outer (z = b,r = R2) cylinders. At equilibrium, the meniscus position z = z{r) is determined by the Young-Laplace equation
-AP-\-Apgz = 2Hy,
(1)
where y is the surface tension, AP is the capillary pressure (pressure of phase A at z = 0 minus pressure that would exist at z = 0 if phase A were replaced by phase B), Ap is the difference between the density of fluids A and B, g is the acceleration of gravity, and H is the mean curvature, 2// =
1
[1 + (dz/dr^f^
dh
1
dr^ + r[\ + {dz/drff^
dz
dr '
(2)
One often wants to find the shape of the meniscus, i.e., the function z(r), by solving Eq. (1) subject to the boundary conditions
zir = RO
z=b
2 = 0
FIGURE3.P.I4
and
z(r = R2) = b.
PROBLEMS
115
(a) By introducing the finite-difference approximation, show that Eq. (1) takes the form
+ - ^ [ ( A r ) 2 + (Zj^, - z,_i)V4]'^'[AP - Apgzj] = 0 (3) for 7 = 1 , . . . ,/i. (b) Use the Newton-Raphson method to solve for the meniscus shape when fluid A is liquid water (at 293 K), fluid B is air at atmospheric pressure, R^ = \ cm, /?2 = 15 cm, a = I cm, and b = 1.2 cm. At 293 K, p^^ter = 0.998 g/cm\ p^^ = 0.0012 g/cm^ and y = 72.75 dyn/cm. Plot the resulting curve z(r). 15. In dimensionless units, the continuous stirred tank reactor equations can be expressed as a expi
dt
I.*
—~ = 1 — ^2 + K exp( — — JJK*! — hX2. Assume that or = 100, p = 5,y = 149.39, and 8 = 0.553. (a) Find the steady-state solutions. (b) Which steady-state solutions are stable to small perturbations? Why? (c) With the initial value XQ = (1 ±0.1)x% compute and plot x(t) versus time for each of the steady states x\ Use the Runge-Kutta method to solve the transient problem. The Runge-Kutta method to use for solving dx — =f(x), at
Xo = x ( r = 0 ) ,
is as follows: X„+, = X„ + -[k, + 2k2 + 2k3 + k4], with k, = hUx„)
k, = ht(x„ + Ik,) k3 = M(X„ + i k , ) k4 = hf(x„ + kj), where x„ = x(/ == nh).
I I6
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
16. Consider the set of equations /j = 1 + h^{Gxp(y^/x) + 3JC^)
-kz-x=0
/2 = 0.5 + h^tan(exp(jc) + y^)-}-ksmz-y
/3 = 1 4- h^xyz + kz^ -\-k^co^y-z
=0
= 0.
When h = k = 0, the solution to these equations is x = I, y = 0.5, and z = I. Use the Newton-Raphson method and continuation to explore solutions to the set as a function of the parameters h and k. In particular, find the solutions as a function of h when k = 0 and of k when h =0. Then look along the lines of constant values of h -\-k. 17. (a) Use Newton-Raphson method to determine the nonzero root of X = I — exp(—Qf^) to four correct decimal places (i.e., |6„_^il = \x,^_^_l — xj < 10~^) for a = 2. Choose J^Q = ^ to show that the solution converges quadratically. (b) Given a new a, set up the first-order continuation scheme for the above problem. Give a stepwise complete algorithm to obtain the new solution. Give all necessary equations. (c) Starting with a = 2 and the solution obtained from part (a), perform the first-order continuation and one Newton-Raphson iteration for a = 3.0. 18. The time-dependent reaction-diffusion equation in Eq. (3.6.34) can be solved iteratively by noting that the time derivative in the finite-difference approximation can be written as dc
Jt ~
c{x,t + At) — c(x,t)
Xt
'
By defining cf ^ as the value of c at the rth nodal point and the kth time cj^^ c{i AJC, k At)), the matrix form of the time-dependent step (i.e., cf ^ = c( problem becomes
c'*+» - c<*> = ^ [ A C « + /(c<*>)] + A th,
(1)
where A is defined in Eq. (3.6.41) (this is often referred to as the Euler method). Consider the reaction-diffusion problem for the second-order reaction f(c) = —kc^ over the spatial interval (0 < x < 1). Using the dimensionless values D = 0.001 and k = 0.25, plot the dimensionless concentration profile at r = 10 for the following initial and boundary conditions: (a) c(0, 0 = 0 c(l,r) = l
c(x,0)=x^.
PROBLEMS
117
(b) c(0, 0 = 0
a
—c(i,o = o dx
c(jc,0)=JC^
(c) An important improvement to the Euler method is to replace the right-hand side of Eq. (1) with its future (A: + 1) values, giving ^iM) _ ^(k) ^ ^^\xc^M)
+ /(c<^+0)l + ^^ b
(2)
(referred to as the implicit or backward Euler method). Write a program to solve Eq. (2) and repeat parts (a) and (b) above using the implicit Euler method. 19. Find the inverse of the n x n Hilbert matrix A whose components are 1 for the case n = 3 and 10. 20. For n = 10, solve by Gauss-Seidel iteration the equation Ax = b for ^j = 1, ^^ = 3, and h^ = 0, i 7^ 1 or 6, where A is the Hilbert matrix defined in Problem 19. 21. Find the LU-decomposition of the 10 x 10 matrix whose elements are «n = 2/,
«,,, + ! = - / ,
«/./_! =
-/
and a^j — 0 otherwise. 22. The pressure and chemical potential of a van der Waals fluid are given by nkT -, P = --n^a I — no (I -\ \1 nl :ihkT fi = -kT ln( — - 1 ) + -; 7 - 2na, where n and T are the density and the temperature, respectively, k is Boltzmann's constant, and a and b are parameters characteristic of the fluid. A vapor of density n^ is in equilibrium with a liquid at density n, if P{n,) = P{n,) /x(n^) = /x(n,).
(1)
Find the densities n^ and n^ of coexisting vapor and liquid densities as a function of temperature T. The problem should be solved in terms of the dimensionless variables n*=:nb
and
^* ^Tb T* = . a
I 8
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
In terms of these dimensionless variables, the critical point is n* = | and T* = Yj' Above T*, liquid and vapor do not coexist (i.e., AT* = n* is the only solution to Eq. (1)), but for T* below T^ the equations in Eq. (1) admit the solutions ^?* / n* appropriate for coexistence. In terms of the dimensionless variables, the equations in Eq. (1) become
Use the Newton-Raphson method to solve these equations for n* and n\ as a function of temperature. A good first guess is needed. 23. Consider the van der Waals equation of state in Example 3.8.3: nkT
r, - na
l-nb
fji = -kT Inl — - 1 I + ' I —nb
'(;s-')
2na,
(a) Using the Newton-Raphson method, calculate the boiling-point temperature (T) and the equilibrium liquid and vapor densities (^i and «g) for ethanol (a = 12.016 atm-L^g-moF, b = 0.08405 L/g-mol) at atmospheric pressure (P = I atm). (b) Calculate and plot the planar interfacial density for ethanol using values of L/^ = 1 , 5 , and 10. 24. (Reversible Anionic Polymerization). A reaction scheme sometimes used to model polymerization is ki
I + M -> Py
initiation step
Pj + M^P2 polymerization steps
where / denotes the concentration of the initiator, M the monomer, and P„ the polymer containing n monomers. The rate equations corresponding to this scheme are % = -kJM
(1)
dM °° -7-=-klIM-k^Y,Pn+k2EPn "'
/!=1
dP, —^ = kJM - k^MPy 4- i^2^2 at
°° (2) n=2
(3)
119
PROBLEMS
dt
= A:,MP„_j - {kyM + k2)P„ + hPn^^.
n>2.
(4)
In matrix form, the rate equations for P|, P2' • • become dV = AP + b, dt
(5)
where
-kJM b =
0
(6)
and A is the tridiagonal matrix
A =
-k^M k2 0 0 . 0 k^M -(itiM + itj) k2 0 •0 0 k^M -(k^M + k2) k2 0
(7) This problem is special in that the vectors and matrices have an infinite number of entries and that A and b depend on time (through / and M, which must satisfy Eqs. (1) and (2)). (a) Solve for the steady-state solution P* in terms of k^, ^2* ^O' ^^^ ?*' where MQ is the initial monomer concentration and f * is the steady-state conversion defined by ^* = (MQ — M*)/MQ ( M * is the steady-state monomer concentration). (b) Describe how you would go about solving for the steady-state conversion §* for a given set of initial monomer and initiator concentrations. (c) In terms of the rate constants, what is the maximum possible value for the conversion §* for a given initial monomer concentration MQ? What initial conditions would be needed to achieve this conversion? 25. (Heat Exchanger Performance). As a design engineer, you have been asked to evaluate the performance of a heat exchanger and offer your advice. Two liquid streams (labeled "cold" and "hot") are contacted in a single-pass heat exchanger as shown in Fig. 3.P.25. Currently, the cold stream enters the exchanger at 95°F and exits at 110°F, while the hot stream enters at 150°F and exits at 115T. However, recent changes in the reactor temperature specification requires the cold stream to exit at 120°F (10° hotter than the current specs). Furthermore, the flow rates of both streams {Q^ and Qh) ^^^ fi^^d by other process constraints and cannot be changed.
120
CHAPTER 3 SOLUTION OF LINEAR AND NONLINEAR SYSTEMS
Qh Th,o
"cold"
^\
Qc
>c n
t- Qh ThA
"hot" FIGURE 3.P.25
Rather than design and install a new heat exchanger (a costly and time-consuming job), you observe that the two streams in the current configuration are contacted "co-currently." Thus, you suspect that simply swapping the input and output points of one of the streams (i.e., changing to a "countercurrent" configuration) may be all that is needed to meet the new specifications. Your goal is to estimate what the output temperatures of both streams would be (t^^ and r^.o) ^^ y^^ }^^^ rearranged the exchanger to be "countercurrent." (Assume that the heat transfer coefficient is unaffected.) (a) Use the following design equations to derive two independent equations for the two unknowns {t^^ and t^ J. Hint: Define new variables a^ = Q^C^^^/iUA) and a^ = GhQ,h/(^^) ^^^ ehminate them from the system of equations. "Cocurrent" configuration:
GcC„,c(7;,„-r,,i)-t/A
[(i'h,i-7;,i)-(7;,o-7;,o)] in[(rh,i-r,,i)/(7;. Tc,o)y
"Countercurrent" configuration:
c,o)]
HiKo-T,,i)/(T^,i-tc,o)Y Variables: Cc = flow rate of "cold" stream Qh = flow rate of "hot" stream Cy c = heat capacity of "cold" stream Q h = heat capacity of "hot" stream U = overall heat transfer coefficient A = effective contact area of exchanger r^ i = inlet temperature of "cold" stream
FURTHER READING
121
T^ i = inlet temperature of "hot" stream T^^ = outlet temperature of "cold" stream (original configuration) 7]^^ z= outlet temperature of "hot" stream (original configuration) t^ ^ = outlet temperature of "cold" stream (new configuration) ^h, o = outlet temperature of "hot" stream (new configuration) (b) Calculate the Jacobian matrix elements for the Newton-Raphson procedure in terms of the current guesses (f^ ^ and ^^,0)(c) Solve this problem using the Newton-Raphson method. (d) The process engineers inform you that the "cold" stream inlet temperature occasionally drops from 95 to 85°F. Using first-order continuation, calculate new initial guesses for a Newton-Raphson calculation at this new inlet temperature (T^ j = 85°F) based on the converged solutions (t*^ and t^ J from the old inlet temperature (r^ i = 95''F). Use the Newton-Raphson method to solve for the new outlet temperature.
FURTHER READING Adby, P. R. (1980). "Applied Circuit Theory: Matrix and Computer Methods/' Halsted, New York. Chapra, S. C, and Canale, R. R (1988). "Numerical Methods for Engineers." McGraw-Hill, New York. Dahlquist, G., and Bjorck, A. (1974). "Numerical Methods." Prentice Hall International, Englewood Cliffs, NJ. Faddeeva, V. N. (1959). "Computational Methods of Linear Algebra." Dover, New York. Gere, J. M., and Weaver, W. (1965). "Analysis of Framed Structures." Van Nostrand, Princeton, NJ. Golub, G. H., and Van Loan, C. F. (1989). "Matrix Computations," 2nd Ed. Johns Hopkins Press, Baltimore. Hackbush, W. (1994). "Iterative Solution of Large Sparse Systems of Equations." Springer-Verlag, New York. Hoffman, J. (1992). "Numerical Methods for Engineers and Scientists." McGraw-Hill, New York. Householder, A. S. (1965). "The Theory of Matrices in Numerical Analysis." Blaisdell, Boston. Lewis, W. E., and Pryce, D. G. (1966). "Application of Matrix Theory to Electrical Engineering." Spon, London. Magid, A. R. (1985). "Applied Matrix Models: a Second Course in Linear Algebra with Computer Applications," Wiley, New York. Press, W. H., Teukolsky, S. A., Vetterling. W. T, and Flannery, B. P (1988). "Numerical Recipes in Fortran," 2nd Ed. Cambridge University Press, Cambridge, UK. Saad, Y (1996). "Iterative Methods for Sparse Linear Systems." PWS, Boston. Strang, G. (1988). "Linear Algebra and Its Applications." Academic Press, New York. Varga, R. S. (1962). "Matrix Iterative Analysis." Prentice Hall International, Englewood Cliffs, NJ. Watkins, S. W. (1991). "Fundamentals of Matrix Computations," Wiley, New York. Wolfram, S. (1996). "The Mathematica Book." 3rd Ed. Cambridge University Press, Cambridge, UK.
This Page Intentionally Left Blank
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
4 . 1 . SYNOPSIS If in the linear system Ax = b the matrix A is square and nonsingular, then Cramer's rule establishes the existence of a unique solution for any vector b. The only problem remaining in this case is to find the most efficient way to solve for X. This was the topic of the previous chapter. On the other hand, if A is singular or non-square, the linear system may or may not have a solution. The goals of this chapter are to find the conditions under which the linear system has a solution and to determine what kind of solution (or solutions) there is. In pursuit of these goals, we will first prove Sylvester's theorem, namely, that the rank r^ of C = AB obeys the inequality r^ < min(r^, r^), where r^ and r^ are the ranks of A and B, respectively. This theorem can be used to establish that r^ = rj^{oT rg) if A (or B) is a square nonsingular matrix. A by-product of our proof of Sylvester's theorem is that |C| = |A| |B| if A and B are square matrices of the same dimension. Thus, the determinant of the product of square matrices equals the product of their determinants. We will then prove the solvability theorem, which states that Ax = h if and only if the rank of the augmented matrix [A, b] equals the rank of A. When a solution does exist for an m x « matrix, the general solution will be of the form X = Xp -I- Yl'i=[ ^i^h^^ where Xp is a particular solution, i.e.. Ax = b, and the vectors x^^\ . . . , x^""'^^ are «—r linearly independent solutions of the homogeneous equation Ax^ = 0 (the c, are arbitrary numbers).
123
124
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
As a by-product of the solvability theorem, we will also prove that if A is an m xn matrix, of rank r^, then A contains r linearly independent row vectors. The remaining n —r column vectors and m — r row vectors are linear combinations of the r linearly independent ones. It follows from these properties that the number of linearly independent column vectors in the set { a , , . . . , a,J is equal to the rank of the matrix A = [ a ^ , . . . , a„]. Gauss or Gauss-Jordan transformation of A provides an efficient way to determine the rank of A or the number of linearly independent vectors in the set { a j , . . . , a„}. In the last section, we will restate the solvability theorem as the Fredholm alternative theorem. That is, Ax = b if and only if b is orthogonal to z (i.e., b^z = 0) for any solution of the homogeneous equation A^z = 0, where A^ is the adjoint of A. The advantage of the Fredholm alternative form of the solvability theorem is that it is generalizable to abstract linear vector spaces (e.g., function spaces in which integral or differential operations replace matrices) where the concept of the rank of a matrix does not arise.
4.2. SYLVESTER'S THEOREM AND THE DETERMINANTS OF MATRIX PRODUCTS fir)
Recall from Chapter 1 that an rth-order determinant M\ of the matrix formed by striking m—r rows and n — r columns of an m x n matrix A is called an rth-order minor of A. For example, consider the matrix
A =
^22
^23
-*24
^32
''33
^34
(4.2.1)
If we strike the second row and the second and third columns «11
«14
L «31
(4.2.2)
«34 J
we obtain the second-order minor
Mf
=
«11
^14
^31
^34
(4.2.3)
n, we obtain the third-c ^12
«13
«21
^22
^23
«3 1
«32
« 33
r(3) _
(4.2.4)
We previously defined the rank r^ of a matrix A as the highest order minor of A that has a nonzero determinant. For instance, the rank of 1
2 1
1 1
(4.2.5)
125
SYLVESTER'S THEOREM A N D THE DETERMINANTS OF MATRIX PRODUCTS
is 2 since striking the third column gives the minor 1 1
M<^> =
2 1
(4.2.6)
-1.
The rank of 1 1
1 1
1 1
(4.2.7)
on the other hand, is 1 since striking one column gives 1 1
M^^> =
1 1
0,
(4.2.8)
and striking two columns and one row gives M^^^ = | l | =
(4.2.9)
One of the objectives of this section will be to relate the rank of the product of two matrices to the ranks of the matrices; i.e., if C = AB, what is the relationship among r^ and r^ and r^? If A is a /? x n matrix with elements a^j and B is an n X q matrix with elements b^j, then C is a /? x ^ matrix with elements ^ij = H^ikhj^
/ = 1 , . . . , /7, y = 1,. . , , ^.
(4.2.10)
k=i
Since only a square matrix has a determinant, we know from the outset that r^ < min(/7, n), TQ < min(n, q), and r^ < min(/?, q). There are, of course, cases where
2 0
0 1
and
1 0
B
0 2
then C =
2 0
0 2
(4.2.11)
and so r^ = r^ = r^ = 2. However, there are also cases where r^^ < r^ and r^; e.g., if A =
1
1
0
0
and
B =
1
0
-1
0
,
then C =
0
0
0
0
(4.2.12)
and so r^ = 0, r^ = rj5 = 1. Similarly, if A =
11 00
and
B =
1 1 1 -1 0 - 1
then C =
010 000 (4.2.13)
and so r^ = 1, r^ = 1, and r^ = 2. The question is whether there are other possibilities. The answer is provided by
126
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
SYLVESTER'S THEOREM. If the rank of A is r^ and the rank o/B is r^, then the rank r^^ ofC = AB obeys the inequality
(4.2.14)
re <min(r^,rg).
The proof of the theorem is straightforward though somewhat tedious. If A and B are /? X n and n x q matrices, then Cii
Ci
(4.2.15)
C = Si
^2
PQ J
where the elements c^j are given by Eq. (4.2.10). Let the indices h^,... ,h^ and A:,,..., /:,. denote the rows and columns of M^'\ an rth-order minor of C given as ^h,k,
^h^k.
(^h,k2
(4.2.16)
M^^' = ^h,ki
Since C;,.^, = ^",=1 ^j.bj^kr
^c
^h.kj
' ••
^Kk,
can be rewritten as
2Z%h^hk,
^h,k2
^h^K
Jl^Kh^hk,
%k2
^Kk,
%k2
^h.kr
(r)
M,
^hJx^j.k,
(4.2.17)
E J]
^hX ^h,K
h
^Kh
^Kh
^hrK
In accomplishing the succession of equations in Eq. (4.2.17), we have utilized the elementary properties 6 and 4 of determinants discussed in Chapter 1. We can continue the process by setting c^^i^^ - I ] " =1 ^hj2^hh ^^^ carrying out the same elementary operations to obtain %h
%J2
^h^k.
^h,kr
(4.2.18) %h
%J2
^Kh
^h,k.
127
SYLVESTER'S THEOREM A N D THE DETERMINANTS OF MATRIX PRODUCTS
Continuation of the process finally yields
%h
%J2
•"hxjr
(4.2.19) ^Kh
=E---
^Kh
E
' ' •
^Kjr
b,,^...b,,Mr,
where we have identified that the remaining determinant is an rth-order minor M (r) of A. Note that if r > r^, then the minor Mf(r)^ is 0 by the definition of r^. Thus, Eq. (4.2.19) proves that rc<^^.
(4.2.20)
Beginning again with Eq. (4.2.16), we can express the elements of the first row of M^c^ as (4.2.21) 71 = 1
and then use elementary properties to obtain
(42,22)
M[^^ = E^-
hiJi
^h,kx
^hJi
^h,kr
Continuation of this process with the other rows eventually yields
<-E--J\
E «*,;.•--"^.y,
h¥J\-Jr-
^hh
'''
^J,K
(4.2.23)
Again, the rth-order minor Mg of B will be 0 if r > r^, and so we conclude rr < rn.
(4.2.24)
The combination of Eqs. (4.2,20) and (4.2.24) implies Sylvester's theorem, Eq. (4.2.14).
128
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
In the special case that A and B are square n x n matrices
(4.2.25)
c= E^«;i^-,l
E%2^22
and elementary operations similar to those yielding Eq. (4.2.23) give
hr
ici=i:---
E
b; i 2
b; (4.2.26)
'l7l
Jr1^h-Jr-I ^Jnl
^;«2
Jnn
The determinant above involving the b^j elements will be nonzero only if the integers Jx, ji^ • - - ^ Jn ^^e a permutation of 1, 2 , . . . , n. Otherwise, two rows of the determinant would be the same. Moreover, the determinant in Eq. (4.2.26) is equal to (—1)''|B|, where P is the number of transpositions necessary to reorder the integers 7i, 72' • •» Jn to the sequence 1, 2 , . . . , n (thereby rearranging the columns to the proper order to get |B|). Thus, Eq. (4.2.26) can be rewritten as
ici = iBix:...i:(-i)%. ...fl,njn'
(4.2.27)
Noting that the factor to the right of |B| is, by definition, the determinant |A|, we have proved the following: THEOREM. If A and B are square matrices, the determinant of their product C is the product of their determinants, i.e.,
|C| = |A||B|.
(4.2.28)
This result is not only useful in evaluating products of determinants, but it is also frequently employed in proving theorems concerning matrices and linear equations. For example, if A is nonsingular, i.e., |A| ^ 0 and A~^ exists, then AA-* = I and Eq. (4.2.28) implies lAA'^l = |A| |A-^| = |I| = 1, or |A-*| = 1/1 A|. This greatly simplifies finding the determinant of the inverse of A. Before leaving this section, let us establish a corollary to Sylvester's theorem that is very useful in the theory of solvability of Ax = b. COROLLARY. The multiplication of a matrix B by a square nonsingular matrix A does not change the rank ofB; i.e., ifC = AB, and if A is square and nonsingular, then
rc=^r,.
(4.2.29)
The proof is simple. If A is a nonsingular n x n matrix, then its rank r^ is n since |A| :J^ 0, and so is the rank of A"^ since |A~^| ^ 0. According to Sylvester's theorem, re < min(r^, r^) = min(n, r^).
(4.2.30)
129
GAUSS-JORDAN TRANSFORMATION OF A MATRIX
But (4.2.31)
B = : A 'C for which Sylvester's theorem requires Tg < min(r^-i, r^^) = min(n, r^).
(4.2.32)
Equations (4.2.30) and (4.2.32) combine to imply that r^^ < Vj^ and r^ < r^^, which can only be true if r^ —r^, thus proving the corollary. A similar proof establishes that the rank of BA is the same as that of B if A is a square nonsingular matrix. EXAMPLE
4.2.1. Consider the matrices A =
-2 1
1 -1
B
-2 2
(4.2.33)
and (4.2.34)
AB
We note that since |A| 3, r^ = 2 , and since the order of the only nonzero minor of B is 1, r^ = 1. The same is true of AB and so r^^ 1. This is admittedly a rather trivial example, but in the next sections we will flex the true muscle of the corollary.
4.3. GAUSS-JORDAN TRANSFORMATION OF A MATRIX Consider the nonsingular matrix ^11
a,2
«2l
«22
^31
«32
(4.3.1)
Assuming a^^ 7^ 0, the first step in Gauss elimination yields ^12
0 0
^22
^23
hi
^33
(4.3.2)
where fc,y = a^^ (^n/^ii)^U» ' = 2» ^' J = 1' 2' ^- If ^22 = ^» ^^ interchange rows 2 and 3 (one of ^22 ^^^ ^32 ^^^^ to be nonzero since r^ = 3) to obtain
0 0
b'^r. ^22
b^ 23
b'.. ^32
b'33
(4.3.3)
I 30
CHAPTER 4 GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
Next, we multiply row 2 by 012/^22 ^^^ subtract it from the first row to obtain «ii
0
c,3
0
b',^ b',,
0
b',2 ^33
(4.3.4)
We then cany out Gauss elimination on the third row to get
0
b'22
^23
0
0
C33
(4.3.5)
Multiplying the third row by C13/C33 and subtracting it from the first row, and multiplying the third row by ^23/^33 ^^^ subtracting it from the second row, we finally arrive at
^22
0 0
0
Cr
0
«u 0 0
(4.3.6)
The process we just described leading to the result in Eq. (4.3.6) is known as Gauss-Jordan elimination. If we wish to solve the system of equations
^21^1 + <^22-^2 + ^23-^3 ~ ^2
(4.3.7)
a^iXi + ^732^2 + ^33X3 = ^3
represented by (4.3.8)
Ax = b,
where r^ = 3, then the Gauss-Jordan transformation of the augmented matrix [A, b] yields «n
0 0
^22
0 0
0
•^33
0
«l1 «2
(4.3.9)
«3 J
where a^ are the values of b^ obtained through the transformation process. The linear equations corresponding to Eq. (4.3.9) are then a,ix,
= a, (4.3.10)
^22^2 ^33-^3 — ^ 3 '
which leads to the simple solution x^ = cci/a^^, X2 = ^2/^22' ^^^ -^3 = ^3/<^33-
131
GAUSS-JORDAN TRANSFORMATION OF A MATRIX
For the general nonsingular problem Ax = b, the Gauss-Jordan transformation converts the equations anXi+ai2X2-}-'"+a^„x^
= b^ (4.3.11)
(^n\Xi + a„2^2 + • • • + a„,x„ = b. into the system =
«ii^i
ai
Cl'yyjC'y
(4.3.12)
for which the solution is simply jc, = oii/a-f. In Eq. (4.3.12) the quantities a'.- and a, are values of the original aij and b^ obtained through the transformation process. We see that for nonsingular problems the advantage of Gauss-Jordan elimination over Gauss elimination is that the backward substitution step is eliminated (or rendered trivial since jc, = «//«-,). A Mathematica routine is provided in the Appendix for the Gauss-Jordan algorithm. If we use a complete pivoting strategy for numerical stability, then column interchange, as well as row interchange, could occur. Suppose a column exchange occurs at the second step. Then the matrix in Eq. (4.3.2) would be
0 0
fo^2 fei23 bL b'..
(4.3.13)
with a[2 = <3i3, ajj = a^2^ b'22 = ^23* ^^c- The corresponding system of equations can be expressed as «llJl+«l23'2+^i3>'3 = « 1 ^223^2 + ^23)^3 = ^2
(4.3.14)
'^32>'2 + fc33}^3=«3' where y^ = jc,, }^2 = •^3» ^^d y'^ = X2. Thus, Gauss-Jordan elimination with row and column interchange transforms Eq. (4.3.7) into the system Atry = a
(4.3.15)
132
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
or ^uy\ (4.3.16)
^iiyi ^333^3 =
«3
The components }^i, ^2' ^3 of y are simply a reordering of the components x^,X2,x^ of X. The conespondence between the components of x and y is easily determined by keeping track of the column interchanges. To do this symbolically, let us define a matrix 1,^, which, when multiplied by x, will interchange components i and j . For a four-dimensional problem, an example is
l23
=
1 0 0 0 0 1 0 1 0 0 0 0
0 0 0 1
(4.3.17)
from which it is easy to show that
(4.3.18)
L,x L-^4 J
For this four-dimensional case, it is easy to see that the matrix I^y, which interchanges components jc- and Xj through the product I,^x, is obtained by interchanging columns / and j of the unit matrix I. This turns out to be generally true. That is, for an «-dimensional case, the matrix 1,^, obtained by interchanging columns / and j of the n X n unit matrix I, interchanges rows / and j of x through the multiplication I^^x. To see how the 1,^ might be useful, imagine a Gauss-Jordan elimination process in which columns 1 and 3 are interchanged followed by columns 2 and 5 and then columns 3 and 6. In the first interchange x is transformed into I13X, in the second into I25I13X, and in the third into I36I25I13X, so that, in the equation Aj^y = a, y = Qx.
(4.3.19)
We can see that the determinant of I,^ is equal to —1, since |Ij^| differs from |I| only by the interchange of two columns. This implies that the matrix Q is also nonsingular. For the Q defined in Eq. (4.3.19), |Q| = II36III25III13I = — 1, and since Q is nonsingular, the solution x to Ax = b is uniquely determined as X = Q~^y from the solution to A^j = a.
GENERAL SOLVABILITY THEOREM FOR Ax = b
133
4.4. GENERAL SOLVABILITY THEOREM FOR Ax = b Although up to this point we have concentrated mostly on solving Ax = b for nonsingular matrices A, the more general problem is of substantial practical and theoretical interest. Consider the system of equations
^21^1 + %2-^2 + ' " + C^ln^n = ^2
(4.4.1)
or Ax = b,
(4.4.2)
where A is an m x n matrix with elements a,^, x is an n-dimensional vector, and b is an m-dimensional vector. We will consider the general situation in which m and n need not be the same and the rank r of A will be less than or equal to min(m, n). Square nonsingular and singular matrices constitute special cases of the general theory. A non-square problem may or may not have a solution and when it does have a solution it is may not be unique. For instance, the system (4.4.3) 2xi H- A:2 + 2x3 = 2 has the solution jCi = - l - X 3 ,
JC2=0
(4.4.4)
for arbitrary JC3, and thus a solution exists but is not unique. On the other hand, the system JC, -f- 2X2 -f- JC3 = 1
(4.4.5) 2JCI + 4JC2 + 2JC3 =
1
has no solution since multiplication of the first equation by 2 and subtraction from the second equation yields the contradiction 0=-l.
(4.4.6)
Thus, the equations in Eq. (4.4.3) are compatible, whereas those in Eq. (4.4.5) are not. The theory of solvability of Ax = b must address this compatability issue for the general case.
134
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
As an aid in examining the general case, we will first discuss Gauss-Jordan elimination for the 3 x 4 matrix
A =
^21
-^22
^23
^24
'*32
^33
''34
(4.4.7)
At this point, we know nothing about A except the values of a^j. For the elimination process, we note that the matrix I^y defined in the previous section has the useful properties that pre-multiplication of A by I,y (i.e., 1,^ A) interchanges rows / and j and post-multiplication of A by I^^ (i.e., AI,y) interchanges columns / and j . For example, if 0 1 0
1 0 0
0 0 1
(4.4.8)
then it is easy to see that
I,2A =
0 1 0 a21
1 0 0
0 0 1 a22
'*23
'*22
'*23
^24
^32
^33
'*34
(4.4.9)
«24 ^14
^33
*32
«34
And if 0 1 0 0
Ii. =
1 0 0 0
0 0 1 0
0 0 0 1
(4.4.10)
then
AI,2 =
^22
^23
^24
^31
^32
^33
^34
^22
^21
^23
^24
^33
-*34
^32
0 1 0 0
1 0 0 0
0 0 1 0
0 0 0 1
(4.4.11)
Note that, for an m x « matrix A, the matrix I,y has to be an m x m matrix for pre-multiplication and an n x n matrix for post-multiplication.
135
GENERAL SOLVABILITY THEOREM FOR A x = 6
For Gauss-Jordan elimination, one other matrix is needed. This is the square matrix J^jik), i ^ j , which has elements 7*;, = 1 for / = 1 , . . . , n, j^j = k, and ji^ = 0 if/ 7«^ m and Im ^ ij. To generate Jij(k), we can start with the unit matrix I and add k as the [ij ]ih element. For example, the 3 x 3 version of J23(fc) is 1 0 0
J23W =
0 1 0
0 k 1
(4.4.12)
Notice that the product of J23(^) with the matrix defined by Eq. (4.4.7) is ^14
J23(^)A =
«21 H- ^«31
«22 + ^«32
^23 + ^^33
«3l
^32
^33
^24
-hka 34
(4.4.13)
a34
Thus, pre-multiplying A by Jij(k) produces a matrix in which the product of k and the elements of the jth row are added to the elements of the /th row. According to the elementary properties of determinants, the determinants of A and }fj{k)A are the same. Since Jfjik) can be generated by multiplying the /th column of I by ^ and adding this to the yth column, it follows that the determinant of Jij(k) is 1. We also note that the inverse of J,y(A:) is J,^(—A;) because lj(-k)lj(k)
= l,
(4.4.14)
by inspection. Since I,yl,y = I, the matrix I,y is its own inverse. The matrices I,^ and 3jj{k) are the tools we need to establish the solvability of any linear system represented by Ax = b. In Gauss-Jordan elimination, the matrix A is transformed into A^^ by multiplication of the square nonsingular matrices 1,^ and J,^. It is important to remember that this will not change the rank of A. Consider the matrix defined by Eq. (4.4.7). If the rank r of A is not 0, a nonzero element can be put at the {11} position by the interchange of rows and/or columns. As was shown above, this can be accomplished by pre- and/or post-multiplying A by the appropriate 1,^ matrices. For simplicity of discussion, suppose that r > 0 (otherwise, A = [0]) and that fl|, ^ 0. Then the first step of Gauss elimination is accomplished by the matrix operation
' < - ' & -
«n 0
a,2
a,3
«14
^22
^23
&24
«31
«32
«33
«34
(4.4.15)
where 0
0
1
0
0
1
(4.4.16)
136
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
a n d b22 = <^22 ~ ^12^21/^11' ^23 — ^23 ~ ^13^21/^ii» ^ ^ ^ ^24 ~ ^24 ~ ^u^ii/^ii'
^^
the next step, Eq. (4.4.15) is multiplied by J3i(—%i/an) to obtain «11
«12
^13
«14 1
^"{-"tM-'&-
^22
^23
^24
^32
^33
^34 J
0 0
(4.4.17)
If the rank r^ = 1, then all the elements b^j in Eq. (4.4.17) are 0 and the transformed matrix is simply
Ir —
«11
«12
«13
«14
0 0
0 0
0 0
0 0
(4.4.18)
If r^ > 1, then at least one of the elements b^j has to be nonzero. By pre- and/or post-multiplication of Eq. (4.4.17) by the appropriate 1,^ matrices, row and/or column interchanges can be carried out to place a nonzero element at the {22} position. For illustration, assume that ^22 ^^ ^32 ^^ ^23 ^^ ^ ^nd Z733 ^ 0. Then ^11
IM J 23J31
{-'tM-'t>"-
«12
^13
0 0
0 0
^33
0
«14 ^34
(4.4.19)
boA
Pre-multiplication of the above matrix by Ji2(—^13/^33) yields 0 0
^^33
0
^34
0
0
0
^24
(4.4.20)
If the rank r^ of A is 2, then ^24 = ^^ i^., the transformed matrix is 0 Atr =
If, however, r^ yields
0
^33
0
^34
0
0
0
0
(4.4.21)
3, then ^^24 7^ 0, and post-multiplication of Eq. (4.4.20) by I34 0 0
^33
^34
0
0
0
-^24
0
(4.4.22)
Pre-multiphcation by Ji3(—^14/^24) followed by J23(—^34/^24) gives the transformed matrix
0 0
0
0
-^33
0
0
^34
0
(4.4.23)
137
GENERAL SOLVABILITY THEOREM FOR A x = b
Equations (4.4.18), (4.4.21), and (4.4.23) are the Gauss-Jordan transformations of A for the cases r^ = 1, 2, and 3, respectively (with the special situation conditions ^722 = ^32 = ^23 = ^ introduccd for illustration purposes). The relationship between Ajr, say in the case of Eq. (4.4.23), is (4.4.24)
PAQ, where
i-m-^M-^M-'tM-'t) Gi^a 13"31
1+
« i i ^ :33
21*-14 a^xc
«]3 -1
+ aub
^34
^33
^21^24
^24
1
^11^34
^34
11*^34
(4.4.25)
0 and 1 0 0 0
23*34 —
0 0 1 0
0 0 0 1
0 1 0 0
(4.4.26)
P is a product of nonsingular 3 x 3 matrices, and so it is a nonsingular 3 x 3 matrix itself. Likewise, Q is a nonsingular 4 x 4 matrix. For the particular cases resulting in Eqs. (4.4.25) and (4.4.26), we find |P| = - 1 , |Q| = -<1. From this 3 x 4 example, the matrix transformation in the Gauss-Jordan elimination method becomes clear for the general matrix
^21
A=
^22
'*2n
(4.4.27)
If the rank of A is r, then, through a sequence of pre- and post-multiplications by the I,y and J^y matrices, the matrix can be transformed into
A„ = PAQ =
0
Yu 0
K22
0 0
0 0
0
0
•
0
Ki, r+1
••
Yxn
•
•
0
¥2. /-fl
"
Yin
•
•
Yrr
Yr.r+1
Yrn
0
0
0
(4.4.28)
I 38
CHAPTER 4 GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
in which P is a product of the I,y and J,y matrices and Q is a product of the I,^ matrices, and where y^i = 0 for / 7^ j and /, 7 = 1 , , . . , r and y^j = 0 for / > r. The values of y^j fori < r, j > r may or may not be 0. Thus, the matrix A^^ can be partitioned as
A,
Aj A2 O3 O4
(4.4.29)
where A| is an r x r matrix having nonzero elements y,, only on the main diagonal; A2 is an r X (n—r) matrix with the elements y,^, / = 1 , . . . , r and 7 = r + 1 , . . . , /i; O3 is an (m —r)xr matrix with only zero elements; and O4 is an im — r)x(n — r) matrix with only zero elements. Since the determinants |P| and |Q| must be equal to ± 1 , the rank of A^^ is the same as the rank of A. Thus, the Gauss-Jordan elimination must lead to exactly r nonzero diagonal elements y^,. If there were fewer than r, then the rank of A would be greater than r. Either case would contradict the hypothesis that the rank of A is r. The objective of this section is to determine the solvability of the system of equations aji^l +'3j2^2 + ---+^ln-^;i
=^1
(4.4.30)
represented by Ax = b.
(4.4.31)
If P and Q are the matrices in Eq. (4.4.28), it follows that Eq. (4.4.31) can be transformed into PAQQ X = Pb
(4.4.32)
At,y = a,
(4.4.33)
or
where y = Q-*x
and
a = Pb.
(4.4.34)
139
GENERAL SOLVABILITY THEOREM FOR A x = b
In tableau form Eq. (4.4.33) reads
(4.4.35) 0 =«,+ ,
0
=a^.
If the vector b is such that not all or,, where i > r, are 0, then the equations in Eq. (4.4.35) are inconsistent and so no solution exists. If a, = 0 for all / > r, then the solution to Eq. (4.4.35) is 3^1 = -Pl,r+iyr+l
Puyn
+
(^i/Yn
yi = -ft.r-hlJr+l
^yn
+ «2/K22
(4.4.36) yr = -A-,r+l3^r+l
Prnyn + «r/Kr,
for arbitrary values of y^_^i,..., y„ and where a
_ ^U
(4.4.37)
Of course, if Eq. (4.4.35) or Eq. (4.4.33) has a solution y, then the vector x = Qy is a solution to Ax = b (i.e., to Eq. (4.4.31)). Let us explore further the conditions of solvability of Eq. (4.4.33). The augmented matrix corresponding to this equation is Yn 0 [At„a] =
0 0
0
0
0
K22
0
•• • •• •
0 0
n,r+l
••
Yin
«!
Yi r+1
"
Yin
«2
Yrr
Yr,r + l
Yrn
0
0
0
0
0
a. a,.
(4.4.38)
As indicated above, Eq. (4.4.33) has a solution if and only if or, = 0, i > r. This is equivalent to the condition that the rank of the augmented matrix [Aj^, a ] is
140
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
the same as the rank of the matrix A^^—which is the same as the rank r of A. If Af^y = a has a solution, so does Ax = b. When the rank of [Ajj., a ] is r, what is the rank of the augmented matrix [A, b]? From the corollary of the previous section, it follows that the rank of C, where C = P[A, b] = [PA, Pb] = [PA, a ] ,
(4.4.39)
is the same as the rank of [A, b] since P is a square nonsingular matrix. Note also that the augmented matrix (4.4.40)
[A,„a] = [PAQ,a]
has the same rank as C since PAQ and PA only differ in the interchange of columns—an elementary operation that does not change the rank of a matrix. Thus, the rank of [A, b] is identical to the rank of [Atj., a ] . Collectively, what we have shown above implies the solvability theorem: THEOREM. The equation Ax = b has a solution if and only if the rank of the augmented matrix [A, b] is the same as the rank of A..
Since the solution is not unique when A is a singular square matrix or a nonsquare matrix, we need to explore further the nature of the solutions in these cases. First, we assume that the solvability condition is obeyed (a, = 0, / > r) and consider again Eq. (4.4.36). A particular solution to this set of equations is Ji = — ,
/ = 1,. . . ,r,
Yii
}\=0,
(4.4.41) r = r + l , . . . , n,
or yii
(4.4.42)
Yrr
0
0 However, if yp is a solution to Eq. (4.4.36), then so is yp + y^, where y^^ is any solution to the homogeneous equations y\ = - ^ l , r + i y r + l
AnJ/t
(4.4.43) yr = -Pr,r+\yr^\
Pmyn'
141
GENERAL SOLVABILITY THEOREM FOR A x = b
One simple solution to this set of equations can be obtained by letting y^_^^ = \ and y, = 0, / > r + 1. The solution is
~A,/-fl
(4.4.44)
1 0
yi"
Similarly, choosing y^^^ = 0, y^^2 = 1' ^"^ yi = 0, / > r + 2, gives the solution
~Pr,r+2
0 1 0
yf
(1)
(4.4.45)
^An-r)
Using this method, the set of solutions yj, , • . . , y},"
can be generated in which
-A, r+j ~Pr,r+j
0
yH'
(4.4.46) (r -f 7)th row
0 for 7 = 1 , . . . , n — r. There are two important aspects of these homogeneous solutions. First, they are linearly independent since the (r -f 7)th element of y^^^ is 1, whereas the (r + 7)th element of all other y^^\ k j^ 7, is 0, and so any linear combination of y^^\ k ^ 7, will still have a zero (r + j)th element. The second aspect is that any solution to Eq. (4.4.43) can be expressed as a linear combination of the vectors yl^\ To see this, suppose that y ^ ^ , , . . . , j„ are given arbitrary values
I 42
CHAPTER 4 GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
in Eq. (4.4.43). The solution can be written as
yh
0
+
0
-
Prnyn
+ +
0 0
+
yn (4.4.47)
which, in turn, can be expressed as (4.4.48) 7= 1
The essence of this result is that the equation A^^y = ct has exactly n — r linearly independent homogeneous solutions. Any other homogeneous solution will be a linear combination of these linearly independent solutions. In general, the solution to A^^y = ot can be expressed as 0)
y = yp + E ^ i y h
(4.4.49)
;=i
0) where yp is a particular solution obeying Aj^yp = a, yl; are the linearly independent solutions obeying At^y^^^ = 0, and Cj are arbitrary complex numbers. Because y^^^ is a solution to the homogeneous equation Aj^yh = 0, the vectors xj,'^ = Qy^^^ are solutions to the homogeneous equation Ax^^^ = 0. The set {x^^^}, 7 == 1,. . . , « — r, is also linearly independent. To prove this, assume that the set is linearly dependent. Then there exist numbers a j , . . ., a„_^, not all of which are 0, such that
0.
(4.4.50)
But multiplying Eq. (4.4.50) by Q"^ and recalling that yj,^^ = Q~^x;^^ yields .0')
E^^yh
0.
(4.4.51)
7= 1
JD Since the vectors y\^^ are linearly independent, the only set [aj] obeying (4.4.51) is a^ = = . . . = a,^_^ = 0, which is a contradiction to the hypothesis that the x^^^'s are linearly dependent. Thus, the vectors xj^^ 7 = 1 , . . , , n - r, must be linearly independent.
143
GENERAL SOLVABILITY THEOREM FOR A x = b
We summarize the findings of this section with the complete form of the solvability theorem: SOLVABILITY THEOREM.
The equation
(4.4.52)
Ax = b
has a solution if and only if the rank of the augmented matrix [A, b] is equal to the rank r of the matrix A. The general solution has the form x = Xp + Xlc^- .0)
(4.4.53)
where Xp is a particular solution satisfying the inhomogeneous equation AXp = b, the set x^^ consists ofn — r linearly independent vectors satisfying the homogeneous equation Axj^ = 0, and the coefficients Cj are arbitrary complex numbers. For those who find proofs tedious, this theorem is the "take-home lesson" of this section. Its beauty is its completeness and generality. A is an m x n matrix, m need not equal n, whose rank r can be equal to or less than the smallest of the number of rows m or the number of columns n. If the rank of [A, b] equals that of A, we know exactly how many solutions to look for; if not, we know not to look for any; or if not, we know to find where the problem lies if we thought we had posed a solvable physical problem. EXAMPLE 4.4.1. Consider the electric circuit shown in Fig. 4.4.1. The V's denote voltages at the conductor junctions indicated by solid circles and the conductance c of each of the conductors (straight lines) is given. The current / across a conductor is given by Ohm's law, / = cAV, where AV is the voltage drop between conductor junctions. A current i enters the circuit where the voltage is V, and leaves where it is VQ. The values of Vj and VQ are set by external conditions.
V/
(1)
Ki
FIGURE 4.4.1
Ye
(2)
Vo
144
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
From the conservation of current at each junction and from the values of the conductances, we can determine the voltages V , , . . . , Vg. The conservation conditions at each junction are 2(V2 -VO + {V,-
Vi) + 5(V3 - y,) + 4{V, - V,) + (V, - V,) = 0 2(V, - V2) + ( n - V2) = 0 3(^6 - V3) + 5(V,
-V,)=0
4(Vi - V4) + 2(Vs - F4) = 0 2(V4 -V,)
(4.4.54)
=0
(V, - V,) + (V2 - Ve) + 3(^3 - V,) + 2(Vo - V,) = 0 3(^8
-Vj)=0
or - 1 3 ^ 1 + 2 ^ 2 + 5^3+4^4 +V6 2V,-3V2 +V, 5V, -8V3 +3V^ 4y, - 6y4 + 2y5 2y4 - 2y5 y, + y , + 3y3 -7K
•Vi
= = = =
0 0 0 0
(4.4.55)
-2yn
3y7 + 3yg = 0, which, in matrix form, is written as
AV = b,
(4.4.56)
where
13 2 5 4 0 1 0
2 -3 0 0 0 1 0
5 0 -8 0 0 3 0
4 0 0 -6 2 0 0
0 0 0 2 -2 0 0
1 1 3 0 0 -7 0
0 0 0 0 0 0 -3
0 0 0 0 0 0 3
(4.4.57)
145
GENERAL SOLVABILITY THEOREM FOR A x = b
and
b =
0 0 0 0 0 -2Vn
(4.4.58)
We will assume that Vj = 1 and VQ = 0 and use Gauss-Jordan elimination to find the voltages V j , . . . , Vg. Note that since A is a 7 x 8 matrix, its rank is less than or equal to 7, and so if the system has a solution it is not unique. The reason is that the conductor between junctions 7 and 8 is "floating." Our analysis will tell us what we know physically, namely, that voltages Vj and Vg are equal to each other but are otherwise unknown from the context of the problem. Solution. We perform Gauss-Jordan elimination on A, transforming the problem into the form of Eq. (4.4.32). The resulting transformation matrices P and Q are
507 101
1495
1729
1729
1105
303
303
303
303
303
0
105 101
2450 1313
1260 1313
105 101
105 101
1085 1313
0
4715 2121
1476
707
5945 2121
4715 2121
4715 2121
3854 2121
0
37506 20705
32994 20705
6486 4141
23547 8282
23547 8282
4794 4141
0
57988 128169
5668 14241
50140 128169
91015 128169
157069 128169
37060 128169
0
85 109
93 109
94 109
85 109
85 109
1
0
1729
P =
1
and Q = l8.
(4.4.59)
146
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
the n = 8 identity matrix. The matrix A then becomes
3
0
0
0
0
0
0 0
0
35 13
0
0
0
0
0 0
0
0
0
0 0
0
0
0 0
0
0 0
303 109
0 0
0 PAQ
41 7
0
846 205
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
436 423
0
-3
3 (4.4.60)
and the vector b is transformed into ./22iOT/ V 303 *^0
T/\ + 1229 303 1/
4- 101 ^i)
V1313 *^C
17 \ + 1715 2121 '^1/
V2121 *^0
a = Pb
-(
/ 9588 1/ V4141 ^ O
37506 y \ + 20705 ^i)
74120 y 128169 ^O
57988 y \ + 128169 ^l)
(4.4.61)
-(2Vo -h 109 *^I/
0 Since for this example y = x, the voltages can be obtained by inspection. Substituting Vj = 1 and VQ = 0, we find Vi = 0.439 V2 = 0.386 V^ = 0.380
(4.4.62)
V4 = 0.439 V5 = 0.439 V6 = 0.281. The last equation in A^^y = ct reduces to V7 = Vg. Thus, the values of Vj and Vg cannot be determined uniquely from the equation system. By Gauss-Jordan elimination, we found that the rank of A is equal to the number of rows (7) and by augmenting A with b the rank does not change. Therefore, although a solution does exist, it is not a unique one since A is a non-square matrix (and m < n). EXAMPLE 4.4.2 (Chemical Reaction Equilibria). We, of course, know that molecules are made up of atoms. For example, water is composed of two hydrogen atoms, H, and one oxygen atom, O. We say that the chemical formula for
147
GENERAL SOLVABILITY THEOREM FOR A x = b
water is H2O. Likewise, the formula for methane is CH4, indicating that methane is composed of one carbon atom, C, and four hydrogen atoms. We also know that chemical reactions can interconvert molecules; e.g., in the reaction 2Ho + O, = 2H,0,
(4.4.63)
two hydrogen molecules combine with one molecule of oxygen to form two molecules of water. And in the reaction 2CH4 + O2 = 2CH3OH,
(4.4.64)
two molecules of methane combine with one molecule of oxygen to form two molecules of methanol. Suppose that there are m atoms, labeled a , , . . . , a,„, some or all of which are contained in molecule Mj. The chemical formula for Mj is then M^ = (a, )„,,(a2)„,,..., («,„)„„.
(4.4.65)
Mj is thus totally specified by the column vector
'2j
(4.4.66)
For example, if H, O, and C are the atoms 1, 2, and 3, and methane is designated as molecule 1, then the corresponding vector
(4.4.67)
a, = tells us that the formula for methane is (H)4(0)o(C)j or CH4.
If we are interested in reactions involving the n molecules M^ .. ., M„, the vectors specifying the atomic compositions of the molecules form the atomic matrix
A = [ai,...,aJ
'*22
(4.4.68)
^7n2
We know from the solvability theory developed above that if the rank of A is r, then only r of the vectors of A are linearly independent. The remaining n — r vectors are linear combinations of these r vectors; i.e., if {a^,..., a^} denotes the set of linearly independent vectors, then there exist numbers p^j such that ^k = T.Pkj^j^ j=i
k = r -\- \,.. . ,n.
(4.4.69)
I 48
CHAPTER 4 GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
These equations represent chemical reactions among the molecular species. Since each vector represents a different molecule, Eq. (4.4.69) implies that the number of independent molecular components is r and a minimum of n — r reactions exists for all of the different molecular species since each equation in Eq. (4.4.69) contains a species not present in the other equations. As an application, consider the atoms H, O, and C and the molecules H2, O2, H2O, CH4 and CH3OH. The atomic matrix is given by
H 0 C
H, 2 0 0
O2 0 2 0
H2O 2 1 0
CH4 4 0 1
CH3OH 4 1 1
The rank of this matrix is 3, and so there are three independent molecular components and there have to be at least 5 — 3 = 2 reactions to account for all the molecular species. Hj, O2, and CH4 can be chosen to be the independent components (because aj, 82, and Si^ are linearly independent) and the equilibrium of the two reactions in Eqs. (4.4.63) and (4.4.64) suffice to account theimodynamically for reactions among the independent species. H2, O2, and H2O cannot be chosen since 33 = BJ + a2/2, reflecting the physical fact that carbon is missing from these three molecules. ILLUSTRATION 4.4.1 (Virial Coefficients of a Gas Mixture). Statistical mechanics provides a rigorous set of mixing rules when describing gas mixtures with the virial equation of state. The compressibility of a mixture at low density can be written as z = l + B^,p + C^.p' + . . . ,
(4.4.70)
where p is the molar density of the fluid and the virial coefficients, B^^i^, C^^^, etc., are functions only of temperature and composition. At low enough density, we can truncate the series after the second term. From statistical mechanics, we can define pair coefficients that are only functions of temperature. The second virial coefficient for an A^-component mixture is then given by Br^. = T.Y.yiyjBijiT).
(4.4.71)
where y, refers to the mole fraction of component /. We desire to find the virial coefficients for a three-component gas mixture from the experimental values of B^^^^^ given in Table 4.4.1. For a three-component system, the relevant coefficients are B^, ^22, ^33, B12, ^ B , and B23. The mixing rule is given by ^.ix =y]Bu
-f- 3^2^22 + J3 ^33
^
149
GENERAL SOLVABILITY THEOREM FOR A x = b
lABLE 4.4.1 Gas Mixture
Second Virial Coefficient at 200 K for Ternary
^mlx 10.3 17.8 26.2 13.7 7.64
Yi 0.25 0.50 0.75 0.25 0
Y\ 0 0 0 0.50 0.75
¥3 0.75 0.50 0.25 0.25 0.25
We can recast this problem in matrix form as follows. We define the vectors mix, 1
bs
i5^ B„
X
(4.4.73)
=
By
B.mix, 5
B23
and the 5 X 6 matrix A by
A =
yh
yh
yii
yii
yh
>'l,l>'2,l
yi.iXi.i
y2,iy3.i
y\.i
yi.iyi,!
>'l,2}'3,2
yz.iy^.i
3'l,3>'2.3
3'l,3>'3.3
3'2.3>'3,3
V|,4^2.4
>'l,4>'3,4
.V2,4>'3,4
>'l,53'2.5
>'l,53'3,5
3'2,53'3,5
yh yh yh y'U yU yh yU yh yh
(4.4.74)
where }\j refers to the ith component and the jth measurement in Table 4.4.1. Similarly, the subscripts in the components of b refer to the measurements. Solving for the virial coefficients has then been reduced to solving the linear system Ax = b.
(4.4.75)
(i) Using the solvability theorems of this chapter, determine if a solution exists for Eq. (4.4.75) using the data in Table 4.4.1. If a solution exists, is it unique? Find the most general solution to the equation if one exists. If a solution does not exist, explain why. We can find the "best fit" solution to Eq. (4.4.75) by applying the least squares analysis from Chapter 3. We define the quantity 2
(4.4.76) 1=1 \
j=i
/
which represents a measure of the relative error in the parameters x,. The vector x that minimizes L can be found from the requirement that each of the derivatives of L with respect to x^ be equal to 0. The solution (see Chapter 3, Problem 8) is A^Ax = A^b.
(4.4.77)
ISO
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
Here x contains the best fit parameters B^j for the given data.
I I I
(ii) Show that Eq. (4.4.77) has a unique solution for the data given in Table 4.4.1. Find the best fit virial coefficients. How does this best fit solution compare to the general solution (if it exists) found in (i)?
4.5. LINEAR DEPENDENCE OF A VECTOR SET AND THE RANK OF ITS MATRIX Consider the set {ap . . . , a„} of m-dimensional vectors. Whether this set is linearly dependent depends on whether a set of numbers {jc,,..., jc„}, not all of which are 0, can be found such that (4.5.1) 7= 1
We can pose the problem in a slightly different manner by defining a matrix A whose column vectors are a ^ , . . . , a„. In partition form, this is L^l» ^2' • • • » ^«J-
(4.5.2)
Next, we define the vector x by
^2
(4.5.3)
X =
Now we ask the question: Does the equation Ax = 0
(4.5.4)
have a nontrivial (x ^ 0) solution? This equation can be expressed as
[a^, a 2 , . . . , a„J
= 0
(4.5.5)
or, carrying out the vector product, (4.5.6) which is the same as Eq. (4.5.1) anXi + --- + ai„x„ = 0 (4.5.7) «ml^l+---+flmn-«„=0-
151
LINEAR DEPENDENCE OF A VECTOR SET AND THE RANK OF ITS MATRIX
The question of whether the vectors a j , . . . , a„ are linearly dependent is thus seen to be the question of whether the homogeneous equation Ax = 0 has a solution, and the answer to this latter question depends on the rank of A. From what we have learned from the solvability theory of Ax = b, we can immediately draw several conclusions: (1) If A is a square nonsingular matrix, then X = 0 is the only solution to Eq. (4.5.4). Thus, a set of n n-dimensional vectors a^ are linearly independent if and only if their matrix A has a rank n, i.e., |A| ^ 0. (2) This conclusion is a subcase of a more general one: a set of n m-dimensional vectors a^ are linearly independent if and only if their matrix A has a rank n. This follows from the fact that Ax = 0 admits n — r solutions. (3) A set of n m-dimensional vectors a^ are linearly dependent if and only if the rank r of their matrix A is less than n. This also follows from the fact that Ax = 0 admits n — r solutions. (4) From this it follows that if n > m, then the set of n m-dimensional vectors a^ will always be linearly dependent since r < (m,n). We are familiar with these properties from Euclidean vector space. We know that no three coplanar vectors can be used as a basis set for an arbitrary threedimensional vector. What is useful here is the fact that analysis of the rank of the matrix whose Cartesian vector components form the column vectors of the matrix will establish whether a given set of three vectors is coplanar. Coplanar means that one of the vectors is a linear combination of the other two (linear dependence). From the solvability theory developed in this chapter, we can use the rank of , a„] not only to determine whether the set {a, , a„} is linearly [a. dependent but also to find how many of the a^ are linearly dependent on a subset of the vectors in the set. The rank of A is equal to the number of linearly independent column vectors and to the number of linearly independent row vectors that A contains. To prove this, consider first an m x n matrix A of rank r in which the upper left comer of A contains an rth-order nonzero minor. Proof of this special case will be shown to suffice in establishing the general case. We can rearrange the homogeneous equation Ax = 0 into the form flll^l
4 - • • • + ^ir-^r —
^l.r+l-^r+l
a\„x,, (4.5.8)
^ml^l
+ ' * * I (^mr^r —
^ m , r + l'^r+l
^MIM-^M •
Since the minor
^21
^22
(4.5.9)
is nonzero and the rank of A is r, the equations in Eq. (4.5.8) have a solution {jCi,..., jc^} for arbitrary jc^_j.,,..., jc„. Note that Ax = 0 always has a nontrivial solution for r < min(m, n) because the rank of [A, 0] is the same as the rank of A.
I 52
CHAPTER 4 GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
One solution to Eq. (4.5.8) is obtained by setting x^^i = 1 and Xj = 0 and solving for x[^\ . . . , jc^^^^ With this solution, Eq. (4.5.8) can be rearranged to get *l,r+l
= - ( a „ x r ' + f l , , x f + . . . + a.,^^'>) (4.5.10)
or, in vector notation. a.+i = - E ^ j % '
(4.5.11)
This proves that the (r + l)th column vector of A is a linear combination of the set {ap . . . , a^}. In general, if x^ = 1 for r > r and Xj =0 for j > r and j =^ /, then the solution {xf\ . . . , xj!^] of Eq. (4.5.8) can be found, and so
:
:
(4.5.12)
or r
a, = - X ] 4 S '
^>^'
(4.5.13)
Thus, we have proven that all the column vectors a ^ ^ j , . . . , a„ are linear combinations of the first r column vectors a j , . . . , a^. The vectors a j , . . . , a^ are linearly independent because, otherwise, there would exist a set of numbers { c j , . . . , c^}, not all 0, such that
Eqa,=0 1 =1
or
:
:
(4.5.14)
If this set of equations has a nontrivial solution, then the rank of the matrix [ a j , . . . , a,.] has to be less than r, which contradicts our hypothesis. In summary, for a matrix of rank r and of the form considered here, the last n — r column vectors are linearly dependent on the first r column vectors, which themselves are linearly independent. Since the rank of the transpose A^ of A is also r, it follows that the last m — r column vectors of A^ are linearly dependent on the first r column vectors, which themselves are linearly dependent. But the column vectors of A^ are simply the row vectors of A, and so we conclude that
153
LINEAR DEPENDENCE OF A VECTOR SET AND THE RANK OF ITS MATRIX
the last m — r row vectors of A are linearly dependent on the first r row vectors, which are themselves linearly independent. Next, consider the general case, i.e., an m x n matrix A of rank r, but in which the upper left comer does not contain a nonzero minor. By the interchange of columns and rows, however, a matrix A' can be obtained that does have a nonzero rth-order minor in the upper left comer. We have already shown that the rank of A' is also r, and so if A'=:[a;,a^,...,a;],
(4.5.15)
then the r m-dimensional column vectors a'^ . . . , a^ are linearly independent and the n — r column vectors a|.,.p . . . , a„ are linear combinations of the first r column vectors. Also, from what we presented above, the r /i-dimensional row vectors [a,^p . . . , i^f^y, / = 1 , . . . , r, are linearly independent and the remaining m — r row vectors are linear combinations of the first r row vectors. To prove what was just stated, note first that the relationship between A and A'is ,(2) A' = Q^^^AQ^^\
(4.5.16)
where the square matrices Q^^^ and Q^^^ are products of the 1,^ matrices that accomplish the appropriate row interchanges and column interchanges, respectively. Since the determinants of Q^^^ and Q^^^ are ibl, it follows that the ranks of A' and A are uic same, riuiii uic piupciiy *^ij*-ij = *> ^^ IIUWS A^
Q(.)Q(/) ^ 1
(4.5.17)
or that Q^" equals its own inverse. Consequently, A = Q"'A'Q<2'
= Q">[a;,...,a:]Q"' Hn
Hhx
(4.5.18)
=:[Q<'V„...,Q<"<] ,(2)
U=i
k=\
J
and, therefore, (4.5.19) k=\
We proved already that (4.5.20) 7=1
I 54
CHAPTER 4 GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
which, when inserted into Eq. (4.5.19), yields
a/ = EA;Q^^X-
(4-5-21)
where Pij = Y!k=\ ^u^^kj- "^he vector a^ is related to one of the set { a j , . . . , a„}, say a^ , by the row interchange operation Q^^\ i.e.,
a;. = Q^^^a;.,
(4.5.22)
Q^^^a;. = Q^^^Q^'^a^, = a^..
(4.5.23)
and it follows that
Thus, Eq. (4.5.21) reads
where [Ij} indicates the indices of the r column vectors of A that were moved to the columns 1 , . . . , r in A to put a nonzero rth-order minor in its upper left comer. This proves that any column vector in A is a linear combination of the r column vectors a;^,a;^,...,a^. These r vectors are linearly independent. To prove this, assume the contrary; i.e., assume that there exists a set of numbers {cp . . . , c^}, not all 0, such that T.^j^ij=0.
(4.5.25)
10) By multiplying Eq. (4.5.24) by Q^^\ it follows that Eq. (4.5.24) implies
or that the set {a'p . . . , a^} is linearly dependent. However, this is a contradiction, and so the vectors a^^,..., a/^ must be linearly independent. Similarly, by considering the transpose of A', we can prove that r of the row vectors of A are linearly independent and that the other m — r row vectors are linear combinations of these r row vectors.
155
THE FREDHOLM ALTERNATIVE THEOREM
The "take-home" lesson of this section is as follows: THEOREM. If the rank of the m x n matrix A is r, then (a) there are r mdimensional column vectors {and r n-dimensional row vectors) that are linearly independent and (b) the remaining n — r column vectors {and m — r row vectors) are linear combinations of the r linearly independent vectors.
EXAMPLE 4.5.1. How many of the vectors
fl' M
-
2
[3_
,
82
=
"l" 1 1
,
a3 =
'2~ 3 _4_
"3l ,
34
5
=
(4.5.26)
_1 \
are linearly independent? Since the rank r of A,
A — 13], 3 2 ' 33,34J —
1 1 2 2 1 3 3 1 4
3 5 7
(4.5.27)
is less than or equal to 3, we know at most three vectors are linearly independent. By Gauss elimination, we transform A to
K=
1 0 0
1 -1 0
2 -1 0
3 -1 0
(4.5.28)
Thus, the rank of A is 2. Therefore, only two of the vectors are linearly independent. Indeed, the pair aj and a2 are linearly independent and
• • •
83 - - a^ + 82
and
VLA — z«a-5
a^.
(4.5.29)
4.6. THE FREDHOLM ALTERNATIVE THEOREM For algebraic equations, the condition of solvability, stated as the equality of the ranks of the augmented matrix [A, b] and the matrix A, is especially attractive. Straightforward Gauss-Jordan elimination establishes the ranks of both [A, b] and A and results in a final set of equations, which, when solvable, require very little further work to obtain the solution or solutions to Ax = b. There is, however, another way to state the solvability theorem, known as the Fredholm alternative theorem. While not suggestive of a method of solution, it is powerful because its form carries over to much more general vector spaces (e.g., function spaces whose operators are differential or integral operators instead of matrices) where the concept of the rank of a determinant is not defined. Before stating the theorem, some additional properties of the matrix A need to be established. Recall that the adjoint A^ is the complex conjugate of the transpose
156
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
A^ of A; i.e., if
«2n
A =
(4.6.1)
then ^ml •*22
A^ =
•*m2
(4.6.2)
Since the interchange of rows and columns does not change the value of a determinant, it follows that the rank of the adjoint A^ is the same as the rank of A. This can be seen explicitly by recalling the form of A^^ in Eq. (4.4.39). The adjoint of At, is •
0 0 0
••
0
0 0 0
0
0
y*
Vl.rM
rlr+t
0 0
••• •••
Yin
Yin
K.i
0 0 A"^ -
0 Yii
frr
•'•
n!.+i
... ...
0 0 0
0 0
(4.6.3)
Since only r columns of A^^, have nonzero elements and since ni=i Yu ¥" 0^ it follows that the rank of Af, is r, the rank of A. Recall, however, that A^^. = PAQ, where |P| and |Q| are ± 1 . Thus, Af, = Q^A^P^ from which it follows that the rank of AJ, is the same as the rank of A^ which proves our claim that the rank of A^ is the same as the rank r of A. The matrix A^ has m column vectors (which are the complex conjugates of the transpose of the row vectors of A), and so, according to the solvability theorem in Eqs. (4.4.52) and (4.4.53), the homogeneous equation A+z = 0
(4.6.4)
has m - r linearly independent solutions; i.e., there exist m — r n-dimensional vectors z. Z 2 , . . . , z„,_r satisfying Eq. (4.6.4). Recall that the homogeneous equation Ax = 0
(4.6.5)
has n — r linearly independent m-dimensional vector solutions x , , . . . , x„_,.. Thus, only if A is a square matrix do A and A^ have the same number of solutions to their homogeneous equations.
157
THE FREDHOLM ALTERNATIVE THEOREM
FREDHOLM ALTERNATIVE THEOREM.
The equation
Ax = b
(4.6.6)
has a solution if and only ifb is orthogonal to the solutions of Eq. (4.6.4) b^z^. = 0,
(4.6.7)
where Zj is any of the m — r linearly independent solutions of the homogeneous adjoint equation (4.6.4). The solvability condition required by the Fredholm alternative theorem places m — r conditions on b, namely, b^Zy = Y!i=\ ^t^tj = 0, 7 = 1 , . . . , m — r. The conditions required to ensure that the rank of [A, b] is the same as the rank of A are that (4.6.8)
«m = 0,
Of,r+2
^r+l
where a, = Yl^=i Ptk^k ^^^ Pik ^^^ elements of the matrix P in the transformation Aj^ = PAQ. Thus, the solvability conditions in Eq. (4.6.8) also place m — r conditions on b and must, of course, be equivalent to the conditions of the Fredholm alternative theorem. The proof of the necessity ("only i f ) on the conditions of the Fredholm alternative theorem is quite simple. Suppose the solution x to Eq. (4.6.6) exists and take the inner product of z, and Eq. (4.6.6) to obtain ZJAX =
(4.6.9)
ztb,
where z, is any solution to Eq. (4.6.4). Taking the adjoint of each side of Eq. (4.6.9)—using the rule (Eq. (2.4.4b)) for forming adjoints of products of matrices—we obtain (4.6.10)
X^A'^'Z; = b'^'Z;.
But A^Zj = 0, or b^z, = 0, proving that Eq. (4.6.7) is a necessary condition for Ax = b to have a solution. To prove the sufficiency condition, we must assume that the conditions in Eq. (4.5.7) are true and prove that this implies the existence of X. This part of the proof is somewhat tedious and will not be given here. EXAMPLE 4.6.1. Under what conditions does
[2 1
ll 2
1
'^1
-,
P Xt
1
=
[3 4J 1 ^2
bi
or
Ax = b
(4.6.11)
JA
have a solution? The homogeneous equation A^z = 0 is given by 2 1
1 2
3 4
= 0 ^3
(4.6.12)
158
CHAPTER 4
GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
or 2zi + ^2 + 3z3 = 0 (4.6.13) and has the solution ^2 (4.6.14) 3
where we have set Z3 equal to 1 (but a solution exists for arbitrary ^3). The solvability condition, b^z = 0, is then (4,6,15) If, for example, h^ = 1, ^2 = U and b^ = | , then Eq. (4.6,11) has a solution. In this case. Ax = b is IXy 4- JC2 = 1 Xi
H- 2JC2 =
3x,
+4JC2 =
1
(4.6.16)
- .
Gauss-Jordan elimination yields
3
_ 2 ~ 3 _ 1
~X'}
— —•
2JCI
2 ^
(4.6.17)
2
0 = 0, or X, = I and X2 = \. Since the rank of A is 2 and n = 2, n — r = 0, there are no homogeneous solutions to Ax = b, and so
X =
1 3 1 3J
• is a unique solution to Ax = b, even though A is not a square matrix.
(4.6.18)
159
PROBLEMS
EXERCISE 4.6.1. Show the solvability conditions that the Fredholm alternative theorem requires for b when
[l
11 1 1J
1
[l
(4.6.19)
Pick a b satisfying the conditions and find the most general solution to Ax = b for this case. *
PROBLEMS 1. A and B are defined as A=:
' -2
and
-2 1
B
1 1
(a) Compute the determinants |A|, |B|, and |AB| and verify that |AB| = |A||B|. (b) Compute A"' and B"' and verify that |A-'| = 1/|A| and |B-M = 1/|B|. 2. If B and C are of ranks rg and r^, show that the rank of A, where B 0
A
0 C
is r^ + re. 3. Find the general solutions to Ax = b for the following: (a) A =
1 6 -2 2
-1 -10 4 -2
2 10 -3 4
b =
(b) 3 0 6 3 3
A =
-1 2 -4 1 -- 3
b =
(c) A =
2 0 2
0 1 1
1 1 2
0 1 1
2 0 1
b =
6 4 8 10 2
I 60
CHAPTER 4 GENERAL THEORY OF SOLVABILITY OF LINEAR ALGEBRAIC EQUATIONS
4. Consider the system of equations 2x -h y + az = P 2x-ay
+ 2z = P
X — 2y -\-2az = 1. (a) For what values of a and p does the system have a unique solution? (b) Use Cramer's rule to obtain the unique solution in terms of a and p. (c) Are there any other nonunique solutions for other values of a and pi If so, give a single example (i.e., choose specific values for a and p and give the general form of the solution.) 5. Consider the set of equations X — 3y = —2 2JC +
J = 3
3x — 2y = a. (a) Are there values of a for which this set has no solution? If so, what are they? (b) Are there values of a for which this set has a solution? If so, give an example. 6. For what values of k will the system 2x+ky
+z=0
(k-\)x-y-2z 4x-hy-\-4z
=0 =0
have nontrivial solutions? 7. Prove that the equations X 4- (cos y)y + (cos P)z = 0 (cos y)x 4- y + (cos a)z = 0 (cos P)x -f (cos a)y + z = 0 have a nontrivial solution if a + P -{- y = 0. 8. Consider the equations ax + by + cz ^ fci a^x + b'^y + c^z = ^:2 a^x+by-^c^z
= b^.
Give the solution to these equations when a,b, and c are different. Give the conditions for a solution when a = b ^ c and give the most general solution in this case.
161
FURTHER READING
9. Consider the augmented matrix
[A, b] =
1 1 -1
3 -9 3
-8 -10 9
2 -3 0
(a) Use simple Gauss elimination to find the rank of [A, b] and A. (b) How many of the column vectors of [A, b] are linearly independent? Why? How many row vectors are linearly independent? Why? (c) If the problem Ax = b (1) has a solution, find the most general one. If there is no solution, why not? (d) Is there a solution to A^z = 0? What are the implications of the answer to this question to the solvability of Eq. (1)? 10. Repeat parts (a)-(d) in Problem 9 using the following augmented matrix:
[A,b] =
3 - 1 2 6 0 2 0 2 2 -9 3 0 3 2 3
-1 -2 0 3 -1
11. Prove that (I + B)x = b has a solution for arbitrary real b if B is a real skew symmetric matrix (B^ = - B ) 12. In the pyrolysis of a low-molecular-weight hydrocarbon, the following species are present: C2H6, H, C2H5, CH3, CH4, H2, C2H4, CgHg, and C4H10. Determine the number of independent reactions among these species. 13. A reaction mixture is found to consist of O2, H2, CO, CO2, H2CO, CH3OH, C2H5OH, (CH3)2CO, CH3CHO, CH4, and H2O. How many independent components are there in the mixture? Which ones can they be? What is the minimum number of reactions possible to produce this mixture?
FURTHER READING Amundson, A. R. (1964). "Mathematical Methods in Chemical Engineering." Prentice Hall, Englewood Cliifs, NJ. Bellman, R. (1970). "Introduction to Matrix Analysis." McGraw-Hill, New York. Noble B., and Daniel, J. W. (1977). "Applied Linear Algebra." Prentice Hall, Englewood Cliffs, NJ.
This Page Intentionally Left Blank
THE EIGENPROBLEM
5. L SYNOPSIS In this chapter we will present the concept of an n x n matrix as a linear operator in an n-dimensional linear vector space £„. We can, as shown in Chapter 2, define an inner product from which we can create an inner product space. The inner product provides us with convenient vector and matrix norms. We will show that the inner product (x, y> obeys the Schwarz inequality, |{x, y>| < ||x|| ||y||, where ||x|| denotes the norm or length of the vector x. The Schwarz inequality, in turn, implies the triangle inequality ||x4-y|| < ||x|| + ||y|| for the norms (lengths) of the vectors, and ||A + B|| < ||A|| + ||B|| for the norms of the matrices. The inequality also implies that ||AB|| < ||A|| ||B||. Any set of n linearly independent vectors {x^,..., x„} forms a basis set in £*„ such that any vector x belonging to E„ can be expanded in the form
- = Eof.x.. where a, are complex scalar numbers. We can subsequently define the reciprocal basis set {z^,..., z„} by the properties xjz^ = S^j. We will show that the vectors {z,} can be formed from the column vectors of (X~*)^ where X = [ x , , . . . , x , J . We say that {x,} and {z,}, satisfying the orthonormality conditions xjzy = 5,^, form biorthogonal sets.
163
I 64
CHAPTER 5 THE EIGENPROBLEM
The concept of basis and reciprocal basis sets is immensely important in the analysis of perfect matrices. We say that the n x n matrix A is a perfect matrix if A has n eigenvectors, where an eigenvector x^ of A obeys the eigenequation Ax,- = A.,x,-,
and A, is a scalar (real or complex number) called the eigenvalue. Specifically, we say X, is an eigenvector of A with eigenvalue X^. We will show that the n x n identity matrix I„ can be expressed by the equation n
for any linearly independent basis set {x,} and its corresponding reciprocal basis set {z^}. Since A = AI, it follows that a perfect matrix A obeys the spectral resolution theorem n
From this we will show that the reciprocal vectors {z,} are the eigenvectors of the adjoint matrix A^ and that the eigenvalues of the adjoint matrix are A.* (the complex conjugate of A,,). The spectral resolution theorem enables us to express the function / ( A ) as / ( A ) = Yll=i /(^j)x,zj for any function f{t) that can be expanded as a Taylor or Laurent series in / near the eigenvalues A-. As we will show, the spectral resolution theorem for exp(aA) provides a simple solution to the differential equation dx/dt = Ax, x(/ = 0) = XQ. We will see that the eigenvalues of A obey the characteristic equation |A — Xl\ = 0, which is an /ith-degree polynomial P„(A.) = Jl%o^j(~^'^^~^^ where aj = lYj A. Note that tr^ A is the 7th trace of A, which is the sum of the yth-order principal minors of A. The traces of A are invariant to a similarity transformation; i.e., try A = try(S~^AS), where S is any nonsingular matrix. This invariance implies that
i.e., the yth trace of A is the sum of all distinct y-tuples of the eigenvalues A i , . . . , A,,,. A is a diagonal matrix with the eigenvalues of A on the main diagonal, i.e., A = [X^Sij]. We will also prove that the traces obey the property try(AB) = try(BA). This implies that the eigenvalues of AB are the same as the eigenvalues of BA. We can determine the degree of degeneracy of an eigenvalue without actually solving for the eigenvectors. The number «, of linearly independent eigenvectors of A that correspond to a given eigenvalue A., (called the degeneracy of A.,) is equal to n — T;^., where r^^, is the rank of the characteristic determinant |A — X,I|. If A has n, eigenvectors {x,} corresponding to the eigenvalue A.,, then the adjoint A^ has n, eigenvectors {z,} corresponding to the eigenvalue A.*. If A,, 7«^ A,^, then the eigenvectors of A corresponding to Ay are orthogonal to the eigenvectors of A^ corresponding to A*.
LINEAR OPERATORS IN A NORMED LINEAR VECTOR SPACE
I 65
Solving the characteristic polynomial equation P„{X) — 0 can be numerically tricky, and so other methods are sometimes sought for finding the eigenvalues. For instance, tridiagonal matrices are often handled via a recurrence method. The power method is an iterative method that is especially easy to implement for computing the eigenvalue of maximum magnitude, but the method can also be used to compute any eigenvalue. We v^^ill end the chapter by presenting Gerschgorin's theorem, which gives a domain in complex space containing all the eigenvalues of a given matrix. As we will see, this is sometimes useful for estimating eigenvalues.
5.2. LINEAR OPERATORS IN A NORMED LINEAR VECTOR SPACE We asserted in Section 2.6 that 5 is a linear vector space if it has the properties: 1. If x,y € 5, then x4-y€5.
(5.2.1)
x + y = y + x.
(5.2.2)
2.
3. There exists a zero vector 0 such that x + 0 = x.
(5.2.3)
4. For every x € 5, there exists —x such that X + (-X) = 0.
(5.2.4)
5. If a and j6 are complex numbers, then
{a 4- i8)x = ax -h i^x
(5.2.5)
a(x + y) = ofx + ay. To produce the inner product space S, we add to the above properties of S an inner producty (x, y>, which is a scalar object defined by the following properties: if X, y G S, then (x,ay) = a ( x , y ) , (y,x) = (x,y)*, (x, x) > 0
if X 7^ 0,
and {x, X) = 0
if and only if x = 0,
where (x, y)* denotes the complex conjugate of (x, y).
(5.2.6)
I 66
CHAPTER 5 THE EIGENPROBLEM
We define the norm or length ||x|| of a vector x in an inner product space by ||x|| ^ /(^),
(5.2.7)
The properties of the inner product given in Eq. (5.2.6) yield a vector norm (or length) obeying the defining conditions of a normed linear vector space (Eq. (2,6.8)), which are the physical conditions commonly associated with the concept of length in the three-dimensional Euclidean vector space we occupy. Namely, 1. ||x|| > 0 and ||x|| = 0 only if x = 0. 2. ||ax|| = \a\ ||x|| for any complex number.
(5.2.8)
3. ||x4-y||<||x||-M|y||. Here \a\ denotes the absolute value of the complex number a, i.e., \a\ = \/ot\ + af, where a^ and a^ are the real and imaginary parts of a: (a = a^ 4- /orj). The proof of item (3), called the triangle inequality, proceeds by first proving the Schwarz inequality. To prove this, we note that 7 > 0, where / ^ (x + ay, X -I- ay) = llxf + a*(x, y)* + a{x, y) + \a\'\\y\\\
(5.2.9)
and a{= a^ + ia^) is an arbitrary complex number. Since 7 > 0, we seek the value of a that minimizes the value of J for a given x and y. At the minimum, a^ and «! obey the equations ^
= (X, y)* + (X, y) + 2a,,\\yf = 0
(5.2.10)
^
= -(x,y)*/ + (x,y>/ +2a,||y|p = 0
(5.2.11)
whose solutions yield Re{x,y)
llyr
«i' =
Ini(x,y) ., .,2 llyll
/colo^ (5.2.12)
or {x,y)
llyf
(5.2.13)
The notation Re{x, y) and Im{x, y) denotes the real and imaginary parts of (x, y). Insertion of Eq. (5.2.13) into Eq. (5.2.9) yields
or, upon rearranging, l(x,y>l
(5.2.15)
167
LINEAR OPERATORS IN A NORMED LINEAR VECTOR SPACE
In the ordinary Euclidean vector space, (a, b) = a b = ||a|| ||b|| cos 0, where 6 is the angle between the vectors a and b. If a and b are colinear, then | cos ^| = 1 and |a • b| = ||a|| ||b||. Otherwise, |cos 0| < I since the projection of a onto the direction of b is shorter than the length of a, as shown in Fig. 5.2.1. In an abstract linear vector space, the Schwarz inequality is geometrically equivalent to the projection of the vector x onto the direction y of y, where y is the unit vector
Ilyll
(5.2.16)
In terms of this unit vector, Eq. (5.2.15) reads l<x,y)l
(5.2.17)
which is analogous to |a • b| = ||a|| | cos6\ < ||a||. We see that only if x and y are colinear will the equality hold in Eq. (5.2.15). To prove the triangle inequality, note that
l|x + yf = ||x|p + 2|(x,y)| + ||yf,
(5.2.18)
which, when combined with Eq. (5.2.15), yields l|x + y f < \\xf + 2||x|| Ilyll + \\yf = (||x|| + ||y||)^ or l|x + y||<||x|| +Ilyll.
(5.2.19)
Linear operators in a vector space were also defined in Section 2.6. We say that A is a linear operator in 5 if it has the properties 1. If X € 5, then Ax e 5.
(5.2.20)
2. If X, y e 5, then A(ax + py) = aAx + ;8Ay.
(5.2.21)
We saw in Section 2.6 that if the norm of a linear operator is defined by II Axil IIAII = max x#o llxll '
FIGURE 5.2.1
(5.2.22)
168
CHAPTER 5
THE EIGENPROBLEM
then the vector norm properties listed in Eq. (5.2.8) imply the following properties for the norms of linear operators: 1. ||A|| > 0 and ||A|| = 0 if and only if A = 0.
(5.2.23)
2. ||aA|| = |a| ||A||, a for any complex number.
(5.2.24)
3. ||A + B | | < | | A | | + ||B||.
(5.2.25)
4. IIABII < IIAll ||B||.
(5.2.26)
Any nxn matrix A obeying the previously defined properties of multiplication and addition is a linear operator in the n-dimensional linear vector space £"„. If we define the inner product by (5.2.27) then A becomes a linear operator in a normed linear vector space with the norm Vx^x. As an operator, A transforms a vector x in E^ into another vector in E^, Geometrically, this transformation can involve rotation, stretching (or shrinking), or both. The possibilities are illustrated in Fig. 5.2.2 for vectors in E3. Note that the vector y is the same length as x, i.e., ||x|| = ||y||, and so in this case the action of A on X is purely a rotation since x and y are not colinear. The vector z is neither colinear with x nor of the same length, and so in this case the action of A is to rotate and stretch (if ||z|| > ||x||) or shrink (if |{z|{ < ||x||) the vector x. In the third case. Ax = A.x, where X is a scalar (real or complex). Thus, A.x is colinear with x and so A merely stretches (|X| > 1) or shrinks (|X| < 1) the vector x. In this last case, i.e., when Ax = Ax,
(5.2.28)
we say that x is an eigenvector of A and X is an eigenvalue of A. Determining what vectors x and numbers k satisfy Eq. (5.2.28) is referred to as the eigenproblem.
FIGURE 5.2.2
169
LINEAR OPERATORS IN A NORMED LINEAR VECTOR SPACE
We will soon learn that a matrix can be usefully classified in terms of how many linearly independent eigenvectors it has. If an n xn matrix has n eigenvectors, it is called 2L perfect matrix. Otherwise, it is called an imperfect or defective matrix. The value of this classification will become clear later. The infinite-dimensional vector space E^, whose vectors x have an infinite number of components, occurs in physical problems such as those encountered in Chapters 9 and 10. With the inner product defined by (5.2.29)
xV = E<>'i'
the requirement that vectors x and y belong to a normed linear vector space is that and
llyf = E l x l ' < ^ -
(5.2.30)
1=1
Thus, the vector
(5.2.31)
with the norm llxll
=[l^]'"^
1.26
(5.2.32)
belongs to a normed linear vector space, whereas the vector 1
7!
(5.2.33)
1
7! with the norm llyll
ST]
= oo
does not belong to a normed linear vector space.
(5.2.34)
170
CHAPTER 5
THE EIGENPROBLEM
We can define inner products other than x^y, which are sometimes useful. For example, if A is a positive-definite matrix (i.e., if A = A^ and x^Ax > 0 for all X ^ 0 in E„), then the inner products (y, x> = y"^Ax
(5.2.35)
(y, x) = y^A^Ax
(5.2.36)
and
satisfy the defining criteria in Eq. (5.2.6). The inner product in either Eq. (5.2.35) or (5.2.36) with the norm defined by ||x|| = ^/(x\ x) will yield a normed linear vector space obeying the conditions in Eq. (5.2.8). We can, of course, define a normed linear vector space in terms of the conditions in Eq. (5.2.8) without defining an inner product. For example, the norm ll^ll = [ !]"=! 1^1 K] ^^ for any positive real /?, defines an acceptable normed linear vector space in E^. However, \\x\\p is not necessarily related to an inner product. In the remainder of this and the next two chapters, we will concentrate primarily on n X n matrices as linear operators in the finite-dimensional linear vector space £•„ in which we will usually define the inner product as y^x. We will, however, demonstrate the utility of alternative inner products such as Eq. (5.2.35) or (5.2.36) for symmetrizing certain problems and for expediting certain proofs. In the last three chapters, we will explore linear operators in infinite-dimensional spaces. These include integral or differential operators in function spaces. EXERCISE 5.2.1. Consider the matrix
A =
Show that the inner product {y. x) = y^Ax I I I
obeys all the conditions in Eq. (5.2.6).
5 3 . BASIS SETS IN A NORMED LINEAR VECTOR SPACE If X is a vector in £„, it can always be expressed in the form
X = E-^re/'
(5.3.1)
171
BASIS SETS IN A NORMED LINEAR VECTOR SPACE
where
0 e, = I 1 I ith row. 0
(5.3.2)
0 The set {e^ . . . , e„} is thus a basis set in E„ in the sense that any vector x can be written as a linear combination of {e,}. Since the set obeys the condition eje^=5/;,
(5.3.3)
where S^j = 1 for / = j and 5,y = 0 for i ^ j , we say that the set {e,} forms an orthonormal basis set. That is to say, the vectors e, are mutually orthogonal (ejcy = 0, / 7^ j) and of unit length (We^f = eje, = 1). Although the set {e,} is perhaps the simplest basis set, it is by no means the only one. Suppose {Xj, X 2 , . . . , x„} is any linearly independent set in E„. Then this set provides a basis set in E„, The requirement for a basis set is that for any vector X there exists a unique set of numbers { « ! , . . . , « „ } such that (5.3.4) This equation can be rewritten as X = Xa,
(5.3.5)
where X is a square matrix whose column vectors are x,, i.e., X = [Xi,...,x„]
(5.3.6)
and a is the vector
(5.3.7)
a— a.
Since the vectors { x , , . . . , x„} are linearly independent, X is nonsingular (|X| ^ 0) and Eq. (5.3.5) has the unique solution (5.3.8) proving that the set { x , , . . . , x„} forms a basis set in E„, We say {Xj,...,x„} is an orthonormal basis set if xjx^. =5,^..
(5.3.9)
172
CHAPTERS THE EIGENPROBLEM
As we have seen, one example of such a set is {e,, 6 2 , . . . , e,,}. However, there are numerous other orthonormal sets. In fact, we can use the Gram-Schmidt procedure to construct an orthonormal set from any linearly independent set { z , , . . . , z„). The Gram-Schmidt procedure constructs an orthogonal set {yi, • • •, y„} from the set {Z;} as follows: yi = z i y2--2
(yizz)
||y_||,y.
y3 = Z3 - - ; ; — j y ,
--T-nryi
Ilyif
Ily2
(5.3.10) S^ (y>/)
_"yl(yK) Note that ylyz = 0 ylya = ylya = o (5.3.11) yjy, = 0 ,
i^ j,j = 1
i - 1, i = 1 , . . . , n,
i.e., each vector y, is orthogonal to every other vector y^, j ^ /. There are exactly n orthogonal vectors y, corresponding to the n linearly independent vectors z,. If there were fewer y,, say n — 1, then y„ = 0, which implies that a linear combination of z, is 0. This, of course, is a contradiction of our hypothesis that the z, are linearly independent. If we now define - - ^' ' lly.ll'
i = l,...,n,
(5.3.12)
then the vectors x^ obey ^hj=S-j,
(5.3.13)
and so they form an orthonormal set. The Gram-Schmidt procedure is the same for a more general inner product—simply replace yjzy in Eq. (5.3.10) by {y, z).
173
BASIS SETS IN A NORMED LINEAR VECTOR SPACE
EXAMPLE 5.3.1. Consider the set 2 1 1
1 z-^ = 2 1
1 1 2
Z3 =
(5.3.14)
Since the determinant of 2 I 1
[ Z | , Z2, Z3J
1 1 2 1 1 2
(5.3.15)
is nonzero (Z? equals 4), the z, are linearly independent. By the Gram-Schmidt procedure, we obtain
-
rr y2 =
2
5 ~6
[i_
21 ~3
2 1 1
7 6
=
1 6 -
rr y3 =
1 L2
5 ~6
2 1 1
5
~TT
(5.3.16)
2~3 7 6 1 6
-
4 n
"IT 4 —
~iT 12 1^
Next, defining x, = y,/||y,||, we find the orthonormal set 1
iT X,
=
X.
yi • • for which xjxy = 5,^.
=
'•'66
X,
=
iT iT
(5.3.17)
174
CHAPTER 5
THE EIGENPROBLEM
When ( x , , . . . , x„} is an orthonormal set, the coefficients a, in the expansion X = X^a,x,
(5.3.18)
1=1
are especially simple to calculate. If we multiply Eq. (5.3.18) by x], we find x]x = f2a,x]x,^f:a,8,j /=i
(5.3.19)
i=l
or (5.3.20)
a J = x)x.
Thus, for an orthonormal basis set, the coefficients a, in Eq. (5.3.18) are just the inner products of x, with x. Since the product x^a^ is the same as a,x,, insertion of Eq. (5.3.20) into Eq. (5.3.18) allows the rearrangement (5.3.21) V/=i
i=i
/
This relationship is valid for any vector x in £„. Therefore, the matrix X]"=i X/xJ has the property that it maps any vector x in E,^ onto itself. This is, by definition, the unit