Kronecker Products and Matrix Calculus: With Applications (Mathematics and Its Applications)

X == XlJ

so that

.r' == lJ'X'

we associate the transition matrix

4>.

= B'(p.


such that

.

(3.14)

111e problem is to find the transition matrix associated with the equation

X=:: AX+XB

(3.15)

given the transition matrices epl and (D 2 defined above. SO/U/iol1

We can write (3.15) as

x ==

Gx

where x and G were defined ill the previous problem. We define a matrix'" as

!J; (t, r) =

'I>. (t, r) ®
(3.16)

We obtain by (3.4) ~

::;; (i12 ® 'PI + 1/.12 ® (~I + '112 ® (A(ll t ) == (B'q,2) ® (1epl) + (1ep2) ® (Aepl) ::;; (lJ'

I

== [B'®I+I®A}[ep2®q,d I Ience

by (3.13) and (3.14)

.,

by (2.11)

~ = GifJ . if;(t, t) =
=1.

(3.18)

The two equations (3.17) and (3.18) prove that if; is the transition matrix for

(3.15) Example 3.5 Find the transition matrix for the equation

. ~l° -~ 2

X+X

1 OJ

G

0 -1

'J

r !

Also

X ==

"

.

44

Some Applications of the Kronecker Product

[eh.3

Solution In this case both A and B are Constant matrices. From Example 3.4.

[~t :~r- e 2


that

ret

[" La

J

J

0 e- t

a

- ," a a ," e3t

I/J ;:;: .
o 0 o a

1

a

L,j

et

For this equation

G

I~ -~ ~ _~l

l~

0

0

J

and it is easily verified that

~ = GI/J and

1/1(0) = I. 3.7 PROBLEM 5 Solve the equation

AXB;:;: C where all matrices are of order n X n. Solution Using (2.13) we can write (3.19) In the form

Hx ;:;: c

(3.20)

where H = B' ® A, x = vee X and c ;:;: vee C. The criteria for the existence and the uniqueness ofa solution to (3.20) are well known (see for example [18]). The above method of solving the problem is easily generalised to the linear equation of the form

A I XB I +A 2 XB 2 + ... +ArXBr ;:;: C

(3.21)

Sec.3.!l1

Problem 6

4S

Equation (3.21) can be written as for exampJe (3.20) where this time

n; ® AI + /32® A2 + '" + B; ® Ar

II = Example 3.6

Find the matrix X, given where

AIXB I

=

AI

+ A 2 XB2

G-J'

B2 = [ 0 -J

iJ, ~

= C

B\

=

and

~~. ~J, C =

A2 =

[-~ --~J '

~ -6l .

Lo 8J

Solution For this example it is found that 2 2 1 -1 [

-21

022

-~2 S

-4 -2 -5 -4

and c' = [4 0 --ti 8J It follows that

x

= W', =

so that

X =

[=i

rL-l -2loj J

3.8 PROBLEM 6 This problem is to determine a constant output feedback matrix K so that the closed loop matrix of a system has preassigned eigenvalues. A multi variable system is defined by the equations

x=

Ax +Bu

y = Cx

(3.22)

where A(n X n), BCn X m) and CCr X n) are constant matrices. u, x and yare r.olumn vectors of order m, nand r respectively.

3S0A Some Applications of the Kronecker Product

[Ch.3

We are concerned with a system having an output feedback law of the form u

= Ky

(3.23)

.. where K(rn X r) is the constant control matrix to be determined. On substituting (3.23) into (3.22), we obtain the equations of the closed loop system = (A+BKC)x (3.24)

x y

=

ex .

The problem can now be restated as follows: Given the matrices A, B, and C, determine a matrix K such that (3.25)

= 0 for preassigned values A = AI' Al' ... ,A/I . Solution Various solutions exist to this problem. We are interested in the application of the Kronecker product and will follow u method suggested in [24]. We consider a matrix H(n X n) whose eigenvalues are the desired values fq, Al' ... , An, that is

1t.J-HI:::; 0 for and Let

so that

IAI - HI

A:::; AltAl'''',}.n

= ao + alA + '" + an_lAn-I + "An

(3.26) (3.27)

A+BKC = H BKe = H-A = Q (say)

(3.28)

Using (2.13) we can write (3.28) as (e' @ B) vec K = vec Q

(3.29)

or more simply as Pk = q

(3.30)

whereP= e'@B,k=vecK and q = vec Q. Notice that P is of order (n 2 X mr) and k and q are column vectors of order mr and n2 respectively. The system of equations (3.30) is overdetermined unless of course m = n = r, in which case can be solved in the usual manner - assuming a solution does exist! In general, to solve the system for k we must consider the subsystem of linearly independent equations, the remaining equations being linearly dependent

Sec. 3.8.1

47

Problem 6

on this subsystem. In other words we determine a nonsingular JlIatrix T(nl X n7.) such that

(3.31 ) where PI is the matrix of the coefficients of the linearly independent equations of the system (3.30) and Pl is a null matrix. Premultiplying both sides of (3.30) by T and making use of (3.31), we obtain

TPk:; Tq

or (3.32)

If the rank of P is mr, then PI is of order (IIlr X fIlr), P2 is of order «(n 2 - II1r1X mr) and u and v are of order mr and (/1 2 - mr) respectively. A sufficien t condition for the I!xbtencc of a solu tion to (3.32) Of equivalently to (3.30) is that

v :; 0

(3.33)

in (3.32). If the condition (3.33) holds and rank PI:; mr, then

k = Wlu .

(3.34 )

The condition (3.33) depends on an appropriate choice of H. The underlying assumption being made is that a matrix H satisfying this condition does exist. This in turn depends on the system under consideration, for example whether it is controllable. Some obvious choices for the foo11 of matrix 11 are: (a) diagonal, (b) upper or lower triangular, (c) companion form or (d) certain combinations of the above forms. Although forms (a) and (b) are well known, the companion form is less well documented. Very brief1y, the matrix 0

11=

1

0

0

0

I

0

0

0

-al

-a2

-aD

-fJ

is said to be in 'companion' form, it has the associated characteristic equation

(3.35)

"

'j

Some Applications of the Kronecker Produ.ct

(eh.3

Example 3. 7 Determine the feedback matrix K so that the two input - two output system

x

~ ~ r~ ~ G ~J ~ J 3

x+

u

-3

has closed loop eigenvalues (-I, -2, -3).

Solution We must first decide on the form of the matrix H. Since (see (3.28»

H-A = BKC and the first row of B is zero, it follows that the first row of H-A must be zerO. We must therefore choose H in the compan ion form. Since the characteristic equation of His

(:\+1) (:\+2) (:\+3) = :\3+6:\ 2+11:\ +6

H =

I ~ ~ ~J L-6-1 1-6

O.

(see (3.35»)

and hence (see (3.28)) Q =

I_~ _~ ~l

L-8 -8 -~J

o 0 0 0 I

0

1

0

o o

1 0

1

p = C'®B

0 0 0

o o o o

0

0 0 0 0 0

I

000

0

Sec. 3.8)

Problem 6

An appropriate matrix

T is the following

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

T=

i

49

0

0

0

0

0

0

1 0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

1 0

0

0

0

I

0

0-1

0

0 0-1

0

0

0

0 0

0

0 0

It follows that

0

0

0

0

0

0

I

0

I

0

1 0

0

TP = ---------0 0 0 0 0 0 0 0 0

0

0

0

0

0

0

0

0

0

0

0

and

0

-8 -3

Tq =

-8 0 0 0 0

0

t;j

{~j ,',

I

23

so

Some Applications of the Kronecker Product

Since

so that (see (3.34»

k

~3J

[

~ If'. ~ _:

Hence

K =

f-3 ol Lo-sJ

[eh.3]

CHAPTER 4

Introduction to Matrix Calculus 4.1 INTRODUCTION

It is becoming ever increasingly clear that there is a real need for matrix calculus in fields such as multivariate analysis. There is a strong analogy here with matrix algebra which is such a powerful and elegant tool in the study of linear systems and elsewhere. Expressions in multivariate analysis can be written in terms of scalar calculUS, but the compactness of the equivalent relations in terms of matrices not only leads to a better understanding of the problems involved, but also encourages the consideration of problems which may be too complex to tackle by scalar calculus. We have already defined the derivative of a matrix with respect to a scalar (see (3.1)), we now generalise this concept. The process is frequently referred to as fonnal or symbolic matrix differentiation. The basic definitions involve the partial differentiation of scalar matrix functions with respect to all the elements of a matrix. These derivatives are the elements of a matrix, of the same order as the original matrix, which is defined as the derived matrix. The words 'formal' and 'symbolic' refer to the fact that the matrix derivatives are defined without the rigorous mathematical justification which we expect for the corresponding scalar derivatives. This is not to say that such justification cannot be made, rather the fact is that this topic is still in its infancy and that appropriate mathcmatical basis is being laid as thc subject develops. With this in mind we make the following observations about the notation used. In general the elements of the matrices A, B, C, ... will be constant scalars. On the other hand the elements of the matrices X, Y, Z, ... are scalar variables and we exclude the possibility that any element can be a constant or zero. In general we will also demand that these elements are independent. When this is not the case, for example when the matrix X is symme tric, is considered as a special case. The reader will appreciate the necessity for these restrictions when he considers the partial derivatives of (say) a matrix X with respect to one of its elements x rs . Obviously the derivative is undefined if x rs is a constant. The derivative is Ers if x rs is independent of all the other elements of X, but is Ers + Esr if X is symmetric.

",

38'01\ [Ch.4

Introduction to Matrix Calculus

52

There have been attempts to define the derivative when xrs is a constant (or zero) but. as far as this author knows. no rigorous mathematical theory for the general case has been proposed and successfully applied. 4.2 THE DERIVATIVES OF VECTORS Let x and y be vectors of orders nand m respectively. W~ can define various deriva tives in the following way [lS]:

(1) The derivative of the vector y with respect to vector x is the matrix

OXI

3Y2 OXI

3Ym OXI

oYI ax?,

oll ox?,

3Ym OXl

OYI 3Y2 OXn OXn

8Ym aXn

3YI

oY

ax

==

of order (n X m) where YI' Y?" ... ,Ym and y and x respectively.

(4.1)

XI, X2 • •• ' ,Xn

are the components of

(2) The derivatives of a scalar with respect to a vector. If Y is a scalar

oY aXI

oy ax

(4.2)

(3) The derivative of a vector y with respect to a scalar x

y .,. a m]

ax

Example 4.1 Given y

(4.3)

Sec. 4.2]

The Derivatives of Vectors

53

and YI = xl -Xl Y2 = xj + 3x. Obtain iJy/iJx. Solutiofl

aYI 3y

iJx

==

aXI

ah aXI

iJYI

ay.

2XI\ 0

ax. ax. aYI

aYl

aX3

aX3

-1

3

0

2X3

In multivariate analysis, if x and yare of the same order, the abwlutc value of the ueterminan t of 3x/3y, that is of

I :; i

t

is called the Jacobian of the transformation determined by

t

y = y(x) .

t

Example 4.2

The transformation from spherical to cartesian co-ordinates is defined by x = r sin e cos ..p, y = r sin e sin ..p, and z = r cos () where r > 0, 0 < () < 1f and o ~..p <21f. Obtain the Jacobian of the transformation.

t

Solution

I

Let

f

"

.,

i I

t f. f,.

and

r

= Yl,

J =

ax oy

()

= Yl,

1/1

= Y3

,

sin h cos Y3

sin h sin Y3

COSh

Yl cos h cos Y3

Yl cos h sin Y3

-Yl siny.

-Yl sin Y. sin Y3 Yl sin h cos Y3

o

=Y? siny •.

f

. t ~ !f

1 l

Definitions (4.1), (4.2) and (4.3) can be used to obtain derivatives to many frequently used expressions, including quatratic and bilinear forms.

It t.

f

i,

Introduction to Matrix Calculus

[Ch.4

For example consider y == x'Ax

Using (4.2) it Is not difficult to show that

iJy

iJx

+A'x

:=

Ax

:=

2Ax if A is symmetric.

We can of course differentiate the vector 2Ax with respect to x, by deflJ1ition (4.1).

(ay) == -(2Ax) a.

-a -

ax ax

ax

= 2A' = 2A (if A is symmetric).

The following table summarises a number of vector derivative formulae. y

ay

scalar or a vector

ax

Ax x)t , xx

A' A 2x Ax +A'x

x~x

(4.4)

4.3 THE CHAIN RULE FOR VECTORS Let

~[~' [}j y

x

and

m

Using the definition (4.1), we can write

G:)'

z

aZ l

aZ l

aXl

aX2

aXil

aZ 2

aZ 2

aZ2

aXj

aX2

aXn

aZm

aZm

aZm

aX l

aX2

aXil

aZ l

(4.5)

Sec. 4.3]

The Chain Rule for Vectors

55

Assume that

z

= y(x)

so that OZt

iJx,

=

i

, :=

1,2, ... ,m

j

:=

1,2, ... , II.

2: 2:

0Z1

OZt oYq

q=l oYq ox,

Then (4.5) becomes iJZ1

2: Yq

(::)' 2:

3Yq OXI

0Z2 oYq iJYq oXl

iJZm iJYq

2. ilYq

=

aXI

oYq

0Z2 iJYq oYq OX2

'2.0Zm OYq aYq 0.\:2

...

~ ax"

0Z2

aYq

oYq

ox"

0Zm

aYq

iJZ1

2: aYq

aZI aZI

aZI

aYl ay!

°YI

aYI aYl

ay,

aXl i)x 2

aXn

aZ 2 aZ2 aYI aYl

aZ2

aYl ah aXl aX2

ah

ay,

oYq

"J. 2:

oYq iJX2

aXn

OX'I ,

aZm aZm

aZm

ay, oy,

ay,

OYI OYl

oy,

aXI aX2

OXn

G:)' G~) =

,

(by (4.1))

(a y az)' ax oy

on transporting both sides, we finally obtain

oz oy az -oy = ax oy

(4.6)

. ;

[Ch.4

Introduction to Matrix Calculus

56

4.4 THE DERNATlVE OF SCALAR FUNCTIONS OF A MATRIX WITH RESPECT TO THE MATRIX Let X = [xII) be a matrix of order (m X n) and let

y

= f(X)

be a scalar function of X. The derivative of y with respect to X, denoted by ay

ax is defined as the following matrix of order (m X n) ay ay ax

3y

3y

aXil aX 12 Cly Cly

aX In ay

aXll aX22

ax1n

ay ~ aXml aXm 2

ay

=

=

[:,J

=

L 1.1

ay

EII aXil

(4.7)

aXmn

where Eii is an elementary matrix of order (m X n). Definition When X = [Xi;} is a matrix of order (m X 11) and y X, then af(X)/aX is known as a gradient matrix.

= f(X) is a scalar function of

Example 4.3 Given the matrix X = [xli] of order (11 X n) ob tain ay lax when y = tr X. Solution

Y = tr X = Xll

+ X22 + ... + xnn ay

-

ax =

= tr X' (see 1.33) hence by (4.7)

1

n'

An important family of derivatives with respect to a matrix involves functions of the determinant of a matrix, for example y = IXI

or

y

= lAX I .

We will consider a general case, say we have a matrix Y = [Yi/] whose components are functions of a matrix X = [Xii]' that is YI/ = Ii/(x)

where x = [xu xu··· xmnJ'.

Sec.4.4J

The Derivative of Scalar Functions of a Matrix

57

We will determine

which will allow us to build up the matrix

a/YI

ax Using the chain rule we can write

olYI - - ==

where Yi/15 the cofactor of the elementYl/ln IY!. Since the cofactors are Independent of the element YII' we have

Yilt Yi., ...

olYI aYi!

- - == It f ol1O\'/s that

alYI (4.8) Although we have achieved our objective in determining the above formula, it can be written in an alternate and useful form. With and

bil

oxr.\'

we can write (4.8) as

L "2.I ali b

lj

I

= LLal/ejbl/el I

I

"f,A1.'Bi .

(by (1.23) and (1.24»

I

tr (AB')

==

where A == [ali) and B == [bl;l.

tr (B'A)

(by (1.43»

,i ,"

123

380A 58

Introduction to Matrix Calculus

[Ch.4 .

Assuming that Y is of order (k X k) let

Yll Yll

Ylk (4.9)

and since

we can write

oIYi ax,..

= tr(OY'z) .

(4.10)

Xn

We use (4.10) to evaluate use (4.7) to construct

olYl/a1:l1. alYl/axl2, ... , a IYI/oxmll

and then

olYl

ax EXample 4.4 Given the matrix X = [xli) of order (2 X 2) evaluate

aIXI/aX.

([) when all the components xII of X are independent (ii) when X is a symmetric matrix.

Solution (i) In the notation of (4.1 0), we have

so that ay/axn

= En (for notation see (1.4».

As

Z = rXll LX21

Xl~.

X2~

we use the result of Example (1.4) to write (4.10) as

a/YI ax/:\,

-

= (vee

,

E,..) vee Z

I

• J

to

y i

Scc.4.4J

The Derivative of Scalar Functions of a Matrix

S9

So that, for example

ant!

= XI2 and so on. Helice

alYI

.(\\~

= alXI = fXlI

ax

X2J

LX21

ax

IXI(X- I )'

(Sec [18] p. 124).

(ii) This time

Y=

XI~ ~II x I2 X 2 2

hence

alYI

alYI

= Ell , -

Xu

Xll

=

EI2 +E 21

and so on.

(See the introduction to Chapter 4 fur explanantion of the notation.)

It

fOllO:;::'~ ~ alYl

JX12

[0

I I

01

JX21

;,:J = Xu

X22

hence

21

alXI

2[X·j-diag{X·}. I} /I

2X

I2

I2

(Since X12 = X21 )

The above results can be generalised to a matrix X of order We obtain, in the symmetric matrix case

-ax =

=

X +X

en x IJ).

'. , I

Introduction to Matrix Calculus

60

[Ch.4

We defer the discussion of differentiating other scalar matrix functions to Chapter 5. 4.5 THE DERNATNE OF A MATRIX WITH RESPECT TO ONE OF ITS ELEMENTS AND CONVERSELY

In this section we will generalise the concepts discussed in the previous section. We again consider a matrix

X ::: [Xi,] or order (m X /I) . The derivative of the matrix X relative to one of its elements Xrs (say), is obviously (see (3.1))

ax

= Ers axrs

(4.11)

where Ers is the elementary matrix of ord~r (m X n) (the order of X) defined in section 1. 2. It follows Immediately that

ax'

-

3xr.r

=E~.

(4.12)

A more complicated situation arises when we consider a product of the form

Y = AXB where

x

(4.13)

= [xii] is of order (m X n)

A ::: [aii] is or order (l X m)

and

B = (h i/ ] is of order (n X q) Y = (Yij] is of ordeF (I X q) .

A and B are assumed independent of X.

Our aim is to find the rule for obtaining the derivatives

ay and where X rs Is a typical elellll:nt of X and YI/Is a typicul element of Y. We will first obtain the (I,})th element Ylj In (4.13) as a function of the elements of X. We can achieve this objective in a number of different ways. For example, we can use (2.13) to write vecY = (B'<8lA)vecX.

Sec. 4.5]

The Derivative of a Matrix

61

From this expression we see that Yij is the (scalar) product of the ith row of [bljA: b2j A: ... : bTl; AJ and vec X.

so that Yij =

Ii

p=1

(4.14)

ail bp;Xlp

1=1

From (4.14) we immediately obtain OYi; = airbsj

( 4.15)

axrs

We can now write the expression for OYi// aX • °Yi/

°Yil

aXil aXJ2 aJ"'j

-=

ax

?!!L

aXln

oYI/

'OYII

aX'll

aX22

ilX 2n

°Yii

aYii

OYii

ilXml ilXm2

i

°Yii

(4.16)

aXmn

Using (4.15), we obtain

°Yii -=

ax

ail b Ij

ail b2j

ail bnj

ai2 b lj

a/2 b 21

ai2 bnj

(4.17)

We note that the matrix on the right hand side of (4.17) can be expressed as (for notation see (1. 5)(1.13) (1.16) and (1.17))

=

Aj.n./

= A'e/ei B'.

I

I I I 1

123

38'0:"

62

Introduction to Matrix Calculus

[Ch.4

So that

OYi/ oX

= A'E n'

(4.18)

II

where Ell is an elementary matrix of order (l X q) tlte order of the matrix Y. We also use (4.14) to obtain an expression for aYjox,s.

oY ~YI/J ax,s = U3xrs

(r, s fixed, i, / variable 1 .;; i '" I, 1 .;; /..; q)

that is

aYIl

ay --:;:

ax,s

ox's OY21 ax,s

aYII ax,s

oYI2 ax,s

aYlq ax,s

aY22 ax,s

aYlq ax,s

0Y12

OYlq axrs

ox's

=

,LE11aYil i.1 ax,s

(4.19)

where E/j is an elementary matrix of order (l X q). We again use (4.15) to write

Ut

aJ,bsl a2,bsl

al,bSl a2,bs2

al,bsq a2,bsq

am,bsl

am,bSl

amrbsq

..

ax,s

:;:

ai' al,

[bsl bs2 ... bsq ]

a,n, :::: A·rBs.' :::: Ae,e~B .

So that

a(AXB) ax,s

:::: AErsB

(4.20)

where Ers is an elementary matrix of order (m X n), the order of the matrix X

Sec.4.5J

The Derivative of a Matrix

63

Example 4.5

Find the derivative

aY/ox rs , given

Y == AX'B where the order of the matrices A, X and B is such thallhc product on the right hand side is defined.

Solution l3y the method used above to obtain the derivative il/ilxr$ (AXB), we find

a

- - (AX'B) == AE~sB .

axrs

'.

Before continuing with further examples we need a rule for determining the derivative of a product of matrices. Consider

Y == UV

(4.21)

where U == [uij] is of order (m X Ii) and V == [ttl] is of order (/1 X l) and both U and Vare [unctions of a matrix X. We wish to determine

The (i,j)th element of (4.21) is n

YI/ ==

L

u/ p !7>j

(4.22)

p~1

hence (4.23)

For fixed rand s, (4.23) is the (i,j)th element of the matrix ay/axrs of order (m X /) the same as the order of the matrix Y. On comparing both the terms on the right hand side of (4.23) with (4.22), we can write

o(UV)

as one would expect.

au

av

-v+uaXrs axrs

(4.24)

64

bltroduction to Matrix Calculus

[Ch.4

of the 011 the other hand, when fixing (i,j), (4.23) is the (r,3)th element matrix aYij/ax, which is of the same order as the matrix X, that is

*

au/pv + ~ u aVpl aY/1 = L.., ax pi L.., /p ax ax p=1

(4.25)

p=1

s. We will make use of the result (4.24) in some of the subsequ ent example

Example 4.6 Let X = lX,sJ be a non-singular matrix. Find the derivative (i) y = AX-IB, and (ii) y= X)tX

ay/ax,s, given

Solution (i) Using (4.24) to differentiate yy-\ = I, we obtain

ay-I ay -y-I+ y _ = 0, ax,s ax,s hence

ay-I -y-y -. ax,s

ay ax,s But by (4.20)

ay-l so that

ay

Oi) Using (4.24), we obtain

ay --r-

ax,s

, a(AX) ax' = -AX+X_ rs

ax

ax,s

(by (4.12) and (4.20» . for all i, j Both (4.18) and (4.20) were derived from (4.15) which is valid . involved matrices the of orders the r, $, defined by

Sec.4.5J

The Derivative of a Matrix

65

TIle First Transformatioll Principle It follows tilat (4.18) is a transformation of (4.20) and conversely. To obtain (4.18) from (4.20) we replace A by A', B by B' and Ers by Eii (careful,E,s and

Ell may be of different orders).

The interesting point is that although (4.18) and (4.20) were derived for constant matrices A and B, the above transformation is indepcndcnt of the status of the matrices and Is valid even when A and n arc functions of X.

Example 4. 7 Find the derivative of

oYii/ax, given

(i) Y=AX'n, (ij) Y=AX-1n, and (iii) Y = x'Ax where X = [xli] is a nonsingular matrix.

Sollllion

(i) Let W = X', tlien

.'

ay

Y = AWn so that by (4.20) =AE,sB awrs hence

aYil aw

.,

= Alg.B ' . IJ

But

hence

;

aYij ax

=

aYlj aw'

=

(ayi/)' aw

r~ .;

ay··

II

-.2!. =

ax

BE'A IJ

l j.

~,

~.

(ii) From Example 4.6(i)

ay -

axrs

so that

.

;

I,.

= -AX-1t."',sx-1n.

letA! =AX- 1 a11dB l =X-1B, then

ilY

~

F'. t.

Introduction to Matrix Calculus

66

(eh.4

(iii) From Example 4.6(ii)

ay -

ax,s

/

=:

letA, == I,B, ::: AX,A 2

ax

- - =:

ilxrs

/ ,

E,sAX + X AE,s

= x'A and B2 = I, then /

A1ErsBI

+ A 2 Ers ih .

'fhe seal,)l\u tcrlll on the right haml side is in st~lH.lard form. The first term is in the form of the solution to Example 4.5 for which the derivative ilYj)ilX was found in (i) above, hence

= AXEij

+ A'XEjj •

It is interesting to compare this last result with the example in section 4.2 when we considered the scalar Y == x/Ax. In tlus special case when the matrix X has only one column, the elementary matrix which is of the same order as Y, becomes

Ejj ::: E:j = I . Hence

ily / aYij == - == Ax +Ax

ax

ax

which is the result obtained in section 4.2 (see (4.4». Conversely using the above techniques we can also obtain the derivatives of the matrix equivalents of the other equations in the table (4.4).

Example 4.8 Find

when (i) Y == AX, and (li) Y == X/X.

Solution (i) With B == I, apply (4.20)

aY -

ax,s

= AE'3'

Sec,4.6J

The Derivatives of the Powers of a MatrLx

67

The transformation principle results in iJYlj

ax

, = A Eij'

Oi) This is a speciul case of Example 4.6 (ii) in which A ::: I, We IlUve found the solution

, -(lxilY = E,.X + XE" rs

rs and (Solution to Example 4.7 (iii))

aYi, ax

+ x£..

= XE/ J

1/'

4.6 THE DERIVATIVES OF THE POWERS OF A MATRIX

"

,

:~

Our aim ill this section is to obtain the rules for determining

ay when

and

aXrs Y = X

n

ax

,

Using (4.24) when U = V = X so that Y = Xl we immediately obtain

-

ar

axrs

I I

and, applying the first transformation principle, ay·· -.-!!. = E .. X' + X'E ...

ax

1/

1/

It is instructive to repea t this exercise wi th

U = X2 so that We obtain

and

\

::: ErsX + Xn~s

and

V = X

,: t·;1, fL

~;

Ii I

~ ~l

\., ~ .. ~.

123

3S'OA (Ch.4

Introduction to. Matrix Calculus

68

Marc generally, it can be proved by induClioil, that [or

Y == X"

"-I

ay

_

ax,s

(4.26)

== "" L..., XkE,s X"-k-I k=O

O

where by defmition X == I, and aYi/' .

-

==

ax

2:

>1-1

(4.27)

(X')"Eij(X,)"-k-1

k=1

Example 4.9

Using the result (4.26), obtain

ay/ax,s when

y "" X-n

Solution Using (4.24) on both sides of X-fiX"

=I

we find

acx-") ax,s acx-") _ ax,s

___ XII

so that

aeX") + X- II _ ax,s

= 0

a(X")

== -X-" --X-".

ax,s

Now making use of (4.26), we conclude that

acX-") _ _ ==

ax,s

~I-I

L...,

k=O

Problems for Chapter 4

(1) Given

X

= IXl!

Xl1

Xl~,

[.:11 X11 X1J

ay ax

J

-X-II" XkE,sX"-k-1 X-II.

ay and

ax

y ==

~

e2X

J

1

X-I

2Xl

sin x

The Derivatives of the Power of a Matrix

Sec.4.6 J (2) Given·

xl

GillX

x

and

Lcosx ej

evaluate

alXI

ax by

(a) a direct method (b) use of a derivative formula. (3) Given

X =

XII

XI2

XI~

[ XZI

X22

X2J

and

y

x'X,

use a direct me thad to evaluate

IlY (a)-

and

(b)

ilx.!

oy 13

ax

(4) Obtain expressi ons for

ay axrs

aYij and

when (a) Y = XAX

ax and

(b) Y = X~X'.

ular. (5) Obtain an expressi on for a\AXEI/ax rs . It is assumed AXE is non-sing

(6) Evaluate ily/axrs when

(a) Y = X(X')2

and

(b) Y = (X')2X.

69

CHAPTERS

Fllrther Development of Matrix

Calculus including an Application of Kronecker Products 5.1 INTRODUCTION In Chapter 4 we discussed rules for determining the derivatives of a vector and then the derivatives of a matrix. But it will be remembered that when Y is a matrix, then vec Y is a vector. This fact, together with the closely related Kronecker product techniques discussed in Chapter 2 will now be exploited to derive some interesting results. Also we explore further the derivatives of some scalar functions with respect to a matrL'< first considered in the previous chapter.

5.2 DERIVATIVES OF MATRICES AND KRONECKER PRODUCTS In the previous chapter we have found aYijlaX when

Y == AXB

(5.1 )

where Y == [Ylil. A == [alj], X == [XII) and B == [bill. We now obtain (a vec y)/(a vec X) for (5.1). We can write (5.1) as y == Px

(5.2)

where y == vec Y, x == vec X and P == B' ® A . By (4.1), (4.4) and (2.10)

-ay == p' == ('rO.)' B ~A == B ®A , .

ax

(5.3)

TIle corresponding result for the equation

Y == AXE is not so simple.

(S.4)

[Sec. 5.2]

Derivatives of Matrices and Kronecker Products

71

The problell1 is that when we write (5.4) in the form of(S.2),wehave this time y

>::

pz

(5.5)

where z =: vee X' We can lind (see (2.25» a permutation matrix U such that vecX' = UvecX

(5.6)

in which case (5.5) becomes y == PVx

so that

~

ax

=:

(PU)'

U'{E ®A').

(5.7)

It is convenient to write

V'(E ® A')

=

(B ® A')(n)

(5.8)

v'

is seen to premultiply the matrix (ll ® A'). Its effect is therefore to rearrange the rows of (ll ®A'). In fact the first and every subsequent 11th row of (B ® A') Corm the first consecutive m rows of (ll ® A')(n)' The second and every subsequent Ilth row form the next m consecutive rows of (B ® A')(Il) and so on. A special case of this notation is for II=: J, then

(B ® A ')(1)

=:

B ® A' .

(5.9)

Now, returning to (5.5), we obtain, by comparison with (5.3)

ay

-

ax

==

Example 5.1

Obtain (iJ vec Y)/(a vee X), given X == [xij] of order (m X n), when

(i) Y == AX, (ii) Y::= XA, (iii) Y = AX' and (iv) Y = XA . Solution

Let y == vec Y and x = vec X. (i) Use (5.3)withB==I

ily

ax

= I®A'.

(5.10)

3S'OA Further Development of Matrix Calculus

[eh.5

(ii) Use (5.3)

ay -

ax

= A ®J .

(iii) Use (5.10)

ay

I

ax = (I ® A )(n)

(iv) Use (5.10)

ay

- = (A ®/)(n) ax

.

5.3 THE DETERMINATION OF (a vee X)/(a vee Y) FOR MORE COMPLICATED EQUATIONS In this section we wish to determine the derivative (il vee Y)/(il vee X) when, for example, Y = x'Ax (5.11) when) X is of order (m X n). Since Y is a matrix of order (/I X n), it follows that vec Y and vec X are vectors of order nn and nm respectively. With the usual notation

Y = [Yij] ,

X = [Xij]

we have, by definition (4.1),

a vee Y ovecX

--=

ayu

aY21

aYnn

axu

aXil

aXil

aY11

aY21

aYnn

OXIJ

aX21

aX21

aY11

aY21

aYnll

OXmn

OXmn

OXmn

Bu t by definition (4.19),

ay),

(

tile first rOW oflhc matrix (5.12) is vec- , aXil

(

ilY)' ,etc.

the second row of the matrix (5.12) is vee -

aX 21

(5.12)

The Determination of (0 vee x)/(o vee Y)

Sec. 5.3]

73

We can therefor!: wrlte (5.12) us oveeY [: oy.: vee -oy.: ... :. vee -ay]' - - ::: vee oveeX aXil QX21 aXmn

(5.13)

We now use the solution to Example (4.6) where we had established that

when Y = x'Ax,

then

oY

axrs

= E:sAX + X'AErs .

(5.14)

It follows that

oY " , vee _.- = vee ErsAX + vee X'AEts ax rs = (x'A' ® I) vee E:s

+ (/ ® x'A) vee Ers

(5.15)

(using (2.13» .

Substituting (5.15) into (5.13) we obtain

avee Y il veeX

+

[(J®X'A)[vecE l1 : veeEzI : ••. : vecEmnJ]'

+

[vec Ell: vec E21 : vee Emn]' (1 ® A'X)

[vec E{l: vec £;1:

... :vec E~n]' (AX ® I) (5.16)

(by (2.1 D». The matrix

is the unit matrix 1 of order (mn X mil). Using (2.23) we can write (5.16) as

ovee Y

U'(AX ®I) + (1 ®A'X).

aveeX 111 a t is

dRCY --

ilvccX

,

:=

(AX ® 1)(11) + (I ® A X) .

(5.17)

In thc above calculations we have u$cd the derivative af/ax rs to ob tain ca vee f)1 (0 vee X).

123

74

Further Development of Matrix Calculus

[Ch.5

The Second Trans/onnation PrincipleOnly slight modifications are needed to generalise the above calculations and show that whenever

where A, B, C and D may be functions of X, then avec Y I - - ;:: B ® A ovecX

I

+ (D ® C )(11)

(S.18)

We will refer to the above result as the second transformation principle.

Example 5.2 Find avec Y

ovecX

when (i) Y = X'X

(ii) Y = AX -I B .

Solution Let y = vec Y and x

= vec X

.

(i) From Example 4.8 OY I I - - = ErsX + XErs

ax rs

Now use the second transformation principle, to obtain ay

ax = I ® X + (X ® /)(n)

(li) From Example 4.6

ay

-

axrs

= -AX-I ErsX- 1B

hence

Hopefully, using the above results for matrices, we should be able to rediscover results for the derivatives of vectors considered in Chapter 4.

Sec. SA]

7S

More on Derivatives of Scalar Functions

For example iet X be a column vector x then Y = XiX

y ==

becomes

XiX

(y is a scalar).

ay lax becomes

The above result for

av

..:.... == (I®x)+(x®J)(l)'

ax

But the unit vectors involved are of order (n X 1) which, fur the one column vector X is (I X I). Hence

ay

l®,x+x®l

ax

x

+ x ==

(use (5.9»)

2x

which is the result found in (4.4). 5.4 MORE ON DERIVATIVES OF SCALAR FUNCTIONS WITH RESPECT TO A MATRIX

In section 4.4 we derived a formula, (4.10), which is useful when evaluating

aIY I/ax for a large class of scalar matrix functions defined by Example 5.3 Evaluate the derivatives log IXI and (i) ax

a

illXl r

(ii) - -

ax

Solution 0) We have () -

axrs

I al)(1 (log 1)(1) == - - -

1)( I axrs

From Example 4.4,

alXI Hence

ax

=

IXI(X-I),

a log IXI ax (ii)

(X- t )'

(non-symmetric case).

.

alXI' = rlXl r- 1 alXI

axrs

axrs

Y.

123

3S'OA Further Development of Matrix Calculus

76 Hence

[Ch.5

illXl' --= ax

function s Traces of matrices form an importa nt class of scalar matrix formuthe in statistics in rly particula ions, applicat of covering a wide range . lation of least squares and various optimisa tion problem s. product s Having discussed the evaluation of the derivative ay/ax,s for various e derivativ the of on evaluati the to results these apply of matrices , we can now

a(tr Y)

ax We first note that

(5.19) a matrix where the bracket on the right hand side of (5.19) denotes , (as usual) . of the same order as X, defined by its (r,s)th element definitio n As a consequ ence of (5 .19) or perhaps more clearly seen from the have we X, ing (4.7), we note that on transpos

Y))' a(tr Y) _ (a- (tr --ax ax'

(5.20)

considering Another , and possibly an obvious property of a trace is found when (4.19)). (see the definitio n of aY/ Assuming that Y = [Yij] is of order (n X n)

ax,s

a y _ aYII aY22 tr- = -+-+

ax,s

ax,.s ax,s -

a

ax,s

Hence,

(Yll

a(tr Y) ay tr- = ---

ax,s

Example 5.4 Evaluate

a (r(AX) ax

ax,s

aYnn

... + -

ax,s

+ Y22 + ... + Ynn)

. (5.21)

More

Sec. 5.4]

011

Derivatives of Scalar Functions

77

Solutioll

u tr (AX)

::

tf

u(AX)

by (5.21)

ax,s

ox's

tr (AEr.)

by Example (4.8)

tr(E:3 A')

since tr Y = tr Y'

(vee Er.)' (vec A ') by Example (1.4). Hence,

a tr (AX)

ax

= A'.

trace of As we found in the previous chapter we can use the derivative of the . product t differen a of trace the of e derivativ the one product to obtain

Example 5.5 Evaluate

a tf (AX')

ax Solution

From the previous rcsul t

a t1' (BX) _ ---ax

_ ' utr (X'B') -B

.

ax

Let A' = B in the above equation, it follows that atr(X'A )

atr(A'X )

ax

ax

---=

=A.

be found The derivatives of traces of more complicated matrix products can similarly.

Example 5.6 Evaluate

a(tr Y) ay

when (i) Y=XA X (li) Y=XA XB SO/utIUI/

It is obvious that (i) follows from Oi) when B = I.

Further Development of Matrix. Calculus

78

[Ch.5

(ii) Y = XIB where XI = X~x.

aY ax,s

ax! ax,s

-- B

= E;sAXB +X~E,.rB Hence,

(by Example 4.6)

Oy), , (OX,S = tr (E,sAXB) + tr (X 'AE,sD)

tr -

= tr (E;sAXB)

+ tr (E;sA'XB')

= (vec Era)' vec (AXB)

+ (vee E,a)' vee (A'XB') .

It follows that

a(tr Y)

- - == AXB+A'xn'.

ax

(i) Let B == I in the above equation, we obtain

3(tr Y) - - = AX+A'X

ax

=

(A+A')X.

5.5 THE MATRIX DIFFERENTIAL For a scalar functionf(x) where x = as =

x n ]', the differential dfis defined

of

n

df

[XI X2 '"

L-dXI.

(5.23)

1=1 aXI

Corresponding to this definition we define the matrix differential dX for the matrix X = [xii] of order (m X n) to be

(5.24)

The following two results follow immediately: d(aX) = a(dX)

(5.25)

(where a is a scalar)

d(X+Y) = dX+dY.

Consider now X

= [Xij] of order (m X /1) and Y = [Ylj]

XY = [~XijYjk] , J

(5.26) of order (n X p).

ff{1

Sec. 5.5]

The Matrix Differential

79

hence

It follows that d(XY) = (dX)Y + X(dY) .

(5.27)

Example 5.7

Given X = [xlf] a nonsingular matrix, evaluate

(0 dlXI, (il) d(X'-I)

-;-

Solution (i) By (5.23) dlXI =

=

"" -alXI (dx,,) L.... /,/ aXij

Li,j Xjj(dXij)

since (aIXI)/caXij) = Xii' the cofactor OfXij in IXI. By an argument similar to the one used in section 4.4, we can write

dlXI = tr {Z'(dX)}

'.

(compare with (4.10))

where Z = [X,/). Since Z I = IX IX -I , we can write dlXI = IXI tr {X-I (dX)} .

(ii) Since X-IX = I

we use (5.27) to write d(X-I)X + X-I(dX) = O.

Hence d(X-I) = -X-I(dX)X- 1

(compare with Example 4.6). Notice that if X is a symmetric matrix, then

X = X' and (dX)' = dX

(5.28)

Further Development of Matrix Calculus

80

[Ch. 5]

Problems for Chapter 5 (J) Consider

A =

[all a12] , x a21 ~2

=

[XU X12] Xli

and Y == AX'.

Xi2

Use a direct method to evaluate avec Y

a vee X and verify (5.1 0). (2) Obtain avec Y i'lvecX ' when (i) Y = AX'B and (ii) Y = llMi Xl . (3) Find expressions for

3 tr Y

3X when (a) Y

= AXB,

(b) Y = X 2

and

(c) Y == XX'.

(4) Evaluate

a tr Y

ax when (a) Y == X-I, (b) Y = AX-1B, (c) Y == Xn and (d) Y == eX.

(5) (a) Use the direct method to obtain expressions for the matrix differential dY when (i) Y == AX, (ii) Y == X'X and (iii) Y == X2 . (b) Find dY when Y = AXBX.

ClIAPTER 6

The Derivative of a Matrix with respect to a Matrix 6.1 INTRODUCTION In the previous two chapters we have defined the derivative of a matrix with respect to a scalar and the aerivative of a scalar with respect to a matrix. We will now generalise the dcfinitiftlls to include the derivative of a matrix with respect to a matrix. The author 1.1u:Cadoptcd tile definition suggested by Vetter [31J, although other definitions also'give rise to some useful results, 6.2 THE DEFINITIONS AND SOME RESULTS

Let Y == [Ylj] be a matrix of order (p X q). We have defined (see (4.19» the derivative of Y with respect to a scalar Xm it is the matfix [aYlllaxrs ] of order (p X q). Let X == [xrs] be a matrix of order (m X 11) we generalise (4.19) and defme the derivative of Y with respect to X, denoted by

ay

ax as the partitioned matrix whose (r,s)th partition is

ay

aXrs in other words ay

ay

ay

aXil

aXln

ilY

ilY

aXl2 ()Y

ax

aX21

aX22

aX2n

ay

ay

ay

aXII1 1

aXm 2

aXmn

-=

()Y

L r,

S

ilY Ers®axrs

(6.1)

123 82

[Ch.6

The Derivative of a Matrix with Respect to a Matrix

TIle right hand side of (6.1) following from the definitions (1.4) and (2.1) where Ers is of order (m X n), the order of the matrix X. l! is seen that aY/ax is a matrix of order (ml' X IIq).

Example 6.1 Consider y=

[;u

sin (Xli

J

XIi

X1:2 X22

+ X12)

e x" log (XII

:t- X21)

and

X=

[Xli XI2J X21 X22

I

Evaluate

oY

ax Solution

ay aX21

~ x,LJ

XI2 X2 2

ay

-=

ax

cos (XII

aY aX22

Xllx

X 22

e

Xu

+ X21

+ X12)

0

"

0

~"x"

~II e

XIIX

0

XI 1 X22

cos (XI1

']

+ XI2)

Xu XI2

0 XI1

I

0 XII

+ X21

0

Example 6.2 Given the matrix X = [Xjj] of order (m X n), evaluate aX(dX when (i) All elements of X are independent

(li) X is a symmetric matrix (of course in this case m = n).

0

eXIl Xu

Sec. 6.2}

The Definitions and Some Results

83

Solution (i)

By (6.1)

ax ax

-=

(ii)

2: Ers ® Ers

D

=

(see (2.26))

r, s

ax

- = 1.:...... + l;'sr oXrs

ax

-=1:-"',.,. axrs

for

r,-ps

for

r=s

We cun write the above as;

Hence,

ax

-=

oX

2: Ers ® b....s + L EN ® b'sr ,,s

'IS

D+

U-"EErr®Err

ax' lax given

Solution

By (6.1) we have

ax' ax

- = Ers ® E;s

u.

Hence

ax' ax

==

0

0

Q

0

0

0

0

1

0

0

0

0

0

0

0

1

0

0

0

0

0

0

0

0 0

0

0

0

0

0

0

L

Esr ® Err

,,s

(see (2.24) and (2.26))

Example 6.3

Evaluate and write out in full

Ors

0

3801. The Derivative of a Matrix wilh

84

Re~\lect

to

11

Matrix

[Ch.6

From the definition (6.1) we obtain

by (2.10)

from (4.19).

It follows that 3Y' (:;)' =

(6.2)

ax'

'6.3 PRODUCT RULES FOR MATRICES We shall first obtain a rule for the derivative of a product of matrices with respect to a matrix, that is to find an expression for

il(XY)

az where the order of the matrices arc as indicated

X(m X n),

Yen X v), Z(p X q) .

By (4.24) we write a(xY)

ax ay -y+xilz,s az,s

where Z = [z,sJ. If E,s is an elementary matrix of order (p X q), we make use of (6.1) to write

a(xY) ax - - = 2E,s® Y + XilY] [ ilz,s ilZ ilz,s " S

Sec.6.3J

Product Rules for Matrices

!IS

(where 1q ano lp arc unit matrices of order (q X q) and (p X p) respectively)

2: (t~s ® az,. ax )(Jq ® y) + 2: (Ip ® (£r$ ® a~y)

=

X)

~s

~s

a~r:

(by 2.11)

finally, by (6.1)

a(XY)

az

ax (Iq

=

az

® Y)

+ (lp

® X)

ay

(6.3)

az

Example 6.4 Find an expression for

ax- I

ax Solution Using (6.3) on XX- I =1, we obtain

a(xx- 1) _ _-c.

I .j hence

ax ax-I = ax

ax ax

ax- 1

= -(1®x- I )+ (J®X) -

ax

= 0

ax ax

-(1®xt l -(1 (8) X-I)

= -(1(8) X-I) 0(1 (8) X-I)

(by Example 6.2 and (2.12». Next we determine a rule for the derivative of a Kronecker product of matrices with respect to a matrix, that is an expression for o(X® Y)

az TIle order of the matrix Y is not now restricted, we will consider that it is (u X v). On representing X ® Y by it (i,j)th partition [xijYj (i = 1, 2, ... , m, k = 1,2, .. " n), we can write

380A The Derivative of a Matrix with Respect to a Matrix

86

[ell. 6

where (r, s) are fixed =

[_axllyl+

az,s

=

r

_i.lYJ

J t:

X1J

az,:

ax aY -®y +x® -

az,:

azrs

Hence by (6.1)

a(x®

y)

az

=

2: E r,

ax

rs ® -

az rs

S

® Y+

LE

ay

rs ® X ® -

T,S

az

TS

where Ers is of order (p X q)

= -ax ® y + az

L. E

rs

To·

® (X ® -ay) •

azr :

The summat ion on the right hand side is not X ® ay/az as may appear at first sight. nevertheless it can be put into a more conveni ent form, as a product of matrices. To achieve this aim we make repeated use of(2.8) and (2.ll)

by (2.11)

Hence (6.4) where U and U. are permuta tion matrices of orders (mu X mu) and (nv X nv) 1 repectively. We illustrate the usc of equation (6.4) with a simple example.

Example 6.5 A = [aii] and X = [Xij] are matrices, each of order (2 X 2). Use (i) Equatio n (6.4), and (ii) a direct method to evaluate

a(A ® X)

ax

Sec.6.3J

Product Rules for Matrices

87

Solution (i) In this example (6.4) becomes

il(A eEl X)

--ax- =

[/ eEl

,l

rax

Ud Lax eEl A

J

[/ eEl U.J

where / is the unit matrix of order (2 X 2) and

~ ~ l~ ~ 0

U1

U2

'2:.Ers eEl h-';s

Since

ax ax

0=

0

0

0

0

0

0

0

0

0

0

1

1

0

0

0

',.

..

~

only a simple calculation is necessary to obtain the result. It is found that

aeA eEl X) ax

all

0

al2

0

0

all

0

al2

0

0

0

0

0

0

0

0

a21

0

all

0

0

a21

0

all

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

all

0

an

0

0

all

0

all

0

0

0

0

0

0

0

0

a21

0

al2

0

0

a21

0

a22

(il) We evaluate

Y = A @X =

~BXB

all Xl2

al2 x U

all X21

all X22

al2 X21

a2l xli

allx I 2

a22 XII

an XI2

a21 x 21

a21 x n

a2l x 21

a22 x 22

and then make use of (6.1) to obtain the above result.

'''X'j al2 X22

,;('.

L • i

I

! .,.

t

I,.

~. ·

38'0" 88

The Derivative of a Matrix with Respect to a Matrix

[Ch.6

6.4 THE CHAIN RULE FOR THE DERIVATIVE OF A MATRIX WITH RESPECT TO A MATRIX We wish to obtain an expression for

az ax where the matrix Z is a matrix function of a matrix X, that Is

z = y(x) where

X == [xli] is of order (m X n)

Y == [Yij] is of order (u X v) Z

== [zi/l is of order (p X q)

By definition in (6.1)

az

-

ax

=

az ax

2: E ',s

r = 1,2, ... ,m

rs ® rs

s = 1,2, ... , n

where Ers is an elementary matrix of order (m X n),

i = 1,1, ... , u j = 1,2, ... , q

where Ell is of order (p X q) As in section 4,3, we use the chain rule to write aZ11 ax,s

-=

L aZ11 . aYail

0:

a, il aY<>jl ax,s

{1 = 1, 2, ... , v

= 1,2, ... , u

Hence

az ax

-=

2:" s

E,s ®

2 "',il

y a ail ]

.. til

22 ',S "',il =

[2: Eii 2 -aZil- . ax,s aY"'il

a,

E,s- ® ax,s

aY"'il

az

ax

aY"il

-0-

il aY"'il

2E

aZij

I,J'

aY"'il

lj - -

(by 2.5)

(by (4.7) and (4.19» .

Sec. 6.4]

The Chain Rule for the Derivative of a Mattix

89

If fro and fp are unit matrices of orders (n X n) and (p X p) respectively. we can write the above as ()Z

-=

ax

Hence, by (2.11)

az ax

(6.5)

Equation (6.5) can be written in a more convenient form, avoiding the summation, if we define an appropriate notation, a generalisation of the previous one. Since

Y =

[~~: ~~: Yul

'"

YIV

..•

Y2"

J

Yuv

Yu2

than (vee Y)' = [YlI Y21 '" Yuv]' We will write the partitioned matrix

[

aYlI ® 1 :- aYll 01 : ... ayuv ®

ax

P -

ax

ax

p.

jlpJ

as

a[YlI

hi'"

Yuvl

--'--'-'--a-x----.:...;..c. or as

® Ip

a(vee Y)' ®Ip ax

Similarly. we write the partitioned matrix

az

j n

j

0aYII

az

0n aY21

as

~

J

1 0az -n

avec Y

90

[Ch.6

The Derivative of a Matrix with Respect to a Matrix

We can write the sum (6.5) in the following order

az = [aYII CiS> Ipl [In 0 az ]+ [aY21 0 I) ax ax j. aYll ax j

y

+ [a

u

"

ax

0

JpJ

rIn 0 Yll az J+ ...

L

r1nLoy"" 0 ~] .

We can write this as a (partitioned) matrix product az

10n

J

y az = [a_ll®I oy ay -. ax ax p.:--2.!®! ax P :• ... •:~®1 ax P

aYll

az aY21

! ®n .

Finally, using the notations defined above, we have

r

az = [a [vec Y)' 01J 1 0 ~J ax az P L n avec Y

(6.6)

We consider a simple example to illustrate the application of the above formula. The example can also be solved by evaluating the matrix Z in terms of the components of the matrix X and then applying the definition in (6.1).

Example 6.6 Given the matrix A = [aij] and X

= [xii]

both of order (2 X 2), evaluate

aZjaX where Z = Y'Y and Y:= AX.

(i) Using (6.6)

(ii) Using a direct method.

Solution (I) For convenience write (6.6) as

az - = QR ax

where

Q

Y]' J = [a(vec ax 01p '

and

R= [In®~J. avec Y

The Chain Rule for the Derivative of a Matrix

Sec. 6.4]

91

From Example 4.8 we know that

3Yli = A'E.

ax

1/

so that Q can now be easily evaluated,

["o 0 00i '" 0 0

01 0 0 all 0

Q=

all 0 0 1 0

1

a1'2

0

0 0 1 a22

o

all 0 0 1 0

1

1

1 a21 0 01 0 0 0 all 1 0 0 01 0 0 all 0 1 a22 0 01 0 0 0 a l2 1

Also in Example 4.8 we have found

az = ErsY+ Y E -rs I

I

aYrs

we can now evaluate R

0

0

Y12

0

o

o

o o

o

2Yll Y12

o

Y12

0

2Yl1 Y12

R

2121 121

0

0

Y22

0

0

0

o o o

o Yl1

0

0

Yll

2Y12 0

0

o o

o o

o

Yl1

Yll

2Y12

-0- -

o

--Y;; --0- - - -0--

121

2Y22 0

0

o o

0

121

o

0

0 0

a'll

0 0 0 0 0 all

0 0 0

0] a:ll

~21

.

3 3S'OA 92

The Derivative of a Matrix with Respect to a Matrix

[Ch.

The product of Q and R is the derivative we have been asked to evaluate

o

allYn

+ allYll

o a12Yl! + a22YlI

+ a2lY2l + 2a 21Y2l allYn + a22Y:U aUYlI

2allYI2

2a 12Y12 + 2anY22

(ii) 13y a simple .extension of the result of Example 4. 6(b) we find that when Z = X/A/AX

az ax rs

= E:sA'AX + X'A'AErs

where Y=AX. By (6.1) and (2.11)

az ax

2: (Ers ® E:s) (/ ® A'y) + 2:

==

',3

(I ® Y'Z) (Ers ® Ey,) .

r,s

Since the matrices involved are all of order (2 X 2)

[ ~l 0

' LErs®Ers = and

LErs®Ers =

0

~

0

0

1

0

0

0

[~ ~l 0

0

0

0

0

0

0

0

On substitution and multiplying out in the above expression for az/ax, we obtain the same matrix as in (i). Problems for Chapter 6

(1) Evaluate

ay/ax given

Y=

COS (X12 [

e XII XI)

-.t- X22) X ll X 2 1] Xl2 X 22

and

1

6]

93

Problems

(2) Xli XI2

The elements of the matrix X = [

X21] X22

Xu X23

are all independent. Use a direct method to evaluate (3) Given a non-singular matrix X

ax/ax.

= [XII X12] X:21

."(2;2

use a direct method to obtain

ax-I

ax and verify the solution to Example 6.4. (4) The matrices A = [alf] and X = [xi;1 arc both of order (2 X 2), X is non· singular. Use a direct method to evaluate

123

CHAPTER 7

Some Applications of Matrix Calculus 7.1 INTRODUCTION As in Chapter 3, where a number of applications of the Kronecker product were considered, in this chapter a number of applications of matrix calculus are discussed. The applications have been selected from a number considered in the published literature, as indicated in the Bibliography at the end of this book. These problems were originally intended for the expert, but by expansion and simplification it is hoped that they will now be appreciated by the general reader.

7.2 THE PROBLEMS OF LEAST SQUARES AND CONSTRAINED OPTIMISATION IN SCALAR VARIABLES In this section we consider, very briefly, the Method of Least Squares to obtain a curve or a line of 'best fit', and the Method of Lagrange Multipliers to obtain an extremum of a function subject to constraints. For the least squares method we consider a set of data (Xi,Yi)

i = 1,2, ... , n

(7.1)

and a relationship, usually a polynomial function

(7.2)

Y = [(x).

For each Xi, we evaluate [(Xt) and the residual or the deviation

(7.3)

et = Yt - [(Xt) .

The method depends on choosing the unknown parameters, the polynomial coefficients when [(x) is a polynomial, so that the sum of the squares of the residuals is a minimum, that is

S

=

*

Lei2 1=1

is a minimum.

= L~ (YI 1=1

[(XI)) 2

(7.4)

[Sec. 7.2]

The Problems of Least Square and Constrained Optimisation

9S

In particular. when/ex) Is a linear function y =

ao + alx

S(ao.al) is a minimum when

as

as

= 0 =aao aal

(7.5)

These two equations. known as normal equations. determine the two unknown parameters ao and al which specify the line of 'best fit' according to the principle of least squares. For the second method we wish',to determine the extremum of a continuously differentiable function [(XI, Xl.

whose

Ii

"',

(7,6)

xn )

variables are contraincd by

g,(Xi> Xl,

.... x n )

= O.

1/1

equations of the form

1= 1.2 .... ,TIl

(7.7)

The method of Lagrange Multipliers depends on defining an augmented function ttl

f'"

= /

+

2:

/Jigi

(7.8)

1;1

where the III are known as Lagrange multipliers. The extreme of lex) is determined by solving the system of the (m equations

af* ax,

= 0

gj = 0

r

+ 11)

= 1, 2, ... , 11

(7.9) i = 1,2, '" ,m

for the m parameters Ill, 1l2, ... , Ilm and the n variables x determining the extremum. Example

7.1

Given a matrix A = [ali] of order (2 X 2) determine a symmetric matrix X = [xli] which is a best approximation to A by the criterion of least squares.

Solution Corresponding to (7.3) we have

E = A-X where E = [eli] and elj = ali -

Xii'

3S'Oi\ 96

Some Applications of Matrix Calculus

[Ch.7

The criterion of least squares for this example is to minimise

S =

Le~

= L(ali -XI/)2

1,/

which is the equivalent of (7.6) above. The constraint equation is xll -Xli

= 0

and the augmented function Is

[of'. = r.(al/ -x,,)2

ar

-

aXil

ar

-

+ P(Xl'l -X21)

=:

0

= -2(J u - Xli) = 0

= -2 (a22 -X22) = 0

aX22

This system of 5 equations (including the constraint) leads to the solution

Hence

x=

[a.:: a" a" : a"J 2

a22

i(A + A')

7.3 PROBLEM 1- MATRIX CALCULUS APPROACH TO THE PROBLEMS OF LEAST SQUARES AND CONSTRAINED OPTIMISATION

If we can express the residuals in the form of a matrix E, as in Example 7.1, then the sum of the residuals squared is

S=trE'E.

(7.10)

Problem 1

Sec. 7.3]

97

The criterion of the least squares method is to minimise (7.1 0) with respect to the parameters involved. The constrained optimisation problem then takes the form of fmding the matrix X such that the scalar matrix function

s

= I(X)

is minimised subject to conlraints on X in the form of

. G(X) = 0

(7.11)

where G = [elf] is a matrix of order (s X t) where sand t are dependent on the number of constraints gil inv~lvcd. As for the scalar case, we usc Lagrange multipliers to forlll an augmented matrix function ["(X). Each constraint gil is associated with a parameter (Lagrange multiplier) Since where

kJ..lijgij = tr U'G U = [J..II/]

we can write the augmented scalar matrix function as

f*(X) = tr E'E + tr U'G

(7.12)

which is the equivalent to (7.8). To find the optimal X, we must solve the system of equations

a/*

-

ax

=

o.

(7.13)

Problem Given a non-singular matrix A = [all] of order (n X n) determine a matrix X = [xII] which is a least squares approximation to A

(i) when X is a symmetric matrix (ii) when X is an orthogonal matrix.

Solution (i) The problem was solved in Example 7.1 when A and X are of order (2 X 2). With the terminology defined above, we write E = A-X G(X)

=

X-X'

=

0

so that G and hence U are both of order (n X n).

123

98

Some Applications of Matrix Calculus

[Ch.7

Equation (7.12) becomes

l'" =

tr [A'-·

= tr A'A -

X'] (A -Xl + tf V'(X -X'] tr A'X - tr X'A + tr X'X + tr V'X - tr V'X' .

We now make use of the results, in modified form if necessary, of Examples 5.4 and 5.5, we obtain

or _._. = -2A + 2X + V ax

= 0

Then

X = A

for

v' u-u'

+-2

V'-u

X' = A ' + - 2

and since X "" X', we finally obtain

X=HA+A'). (ii) This time

G(X) = X'X-I

Hence so that

t'"

= O.

= tr[A'- X'J[A - X]

ar =

-

ax

+ tr V'[XX'- I]

-2A +2X+X[V+ V']

= 0

for

X

v +' V '] = A-X r[-2-

Pre multiplying by X' and using the condition X'X = I we obtain

X'A =

V+V'

1+-2

and on transposing A'X = Hence A'X

V+V'

1+--

= X'A.

2

(7.14)

If a solution to (7.14) exists, there are various ways of solving this matrix equation.

Sec.7.3J

99

Problem 1

For example with the help of (2.13) lind Example (2.7) we can write it as

l(/ ® A') .- (II' ®

I)UJ x

=0

(7.15)

where U is a permutation matrix (see (2.24» and

x == vecX. We have now reduced the matrix equation into a system of homogeneous equations wh.ich cart be solved by a standard method. If a non-trivial solution to (7.15) does exist, it is not unique. We must scale it appropriately for X to be orthogonal. There may, of course, be more than one linearly indepentlent solution to (7.15). We must choose the solution corresponding to X being an orthogonal matrix.

Example 7.2 Given

find the othogonal matrix X which is the least squares best approximation to A.

Solution

[I0A'] =

1 r l

0

2

1

0

o o

0 0

o

-1

2

,

and [A 0I] U =

1

Equation (7.15) can now be written as

[-~

0

[,-I 0

0

0

2

I

0

0

0

2

l

0 -1

-1

1

0

0

x = 0 .

-IJ

There are 3 non-trivial (linearly independent) solutions, (see [18] p. 131). They are x = [1 1 2 -1]' and x = [2 -3 3 2]' . x = [1 -2 1 1]', Only the last solu lion leads to an orthogonal matrix X, it is

I [2-3 2J31 .

X = yTI

380A 100

[Ch.7

Some Applications of Matrix Calculus

7.4 PROBLEM 2 - THE GENERAL LEAST SQUARES PROBLEM

The linear regression problem presents itself in the following form: N samples from a population are considered. The ith sample consists of an observation from a variable Y and observations from variables Xl> X 2 , ••• , X" (say). We assume a linear relationship between the variables. If the variables are measured from zero, the relationship is of the form

(7.16) If the observations are measured from their means over the N samples, then

(7.17) bOt b l , b2 • ••• , bn are estimated parameters and et Is the corresponding residual. In matrix notation we Can write the above equations as y = Xb+ e

where y

=

(7.18)

[~:J ' [:] [i] , e

b =

=

and x12

X=

X22

Xln X2I1

or

X=

XN2··· xNn

Xu X12

Xl n

X21 x22

x2n

XNI xN2 '"

xNn

As already indicated, the 'goodness of fit' criterion is the minimisation with respect to the parameters b of the sum of the squares of the residuals, which in this case is S = e/e = (y/-b'X')(y-Xb). Making use of the results in table (4.4), we obtain

a(e/e) - - = -(y/X)' - X/y + (X'Xb + X'Xb) ab = -2X'y + 2X'Xb = 0 for X'Xb = X'y .

(7.19)

where "b is the least squares estimate of b. If (X/X) is non-Singular, we obtain from (7.l9)

b=

(X/xt 1 X/y .

(7.20)

101

Problem 2

Sec. 7.4] We can write (7.19) as

X'(y -Xii) or

0

X'e = 0

(7.21)

which is the matrix. form of the normal equations defiend in section 7.2.

Example 7.3 Obtain the normal equations for a least squares approximation when each sample consists of one observation from Yand one observation from

(i) a random variable X (ii) two random variables X and Z. Solution (i) b

hence

X'[y-Xh]

So that the normal equations are

LY, =

and

LX/Y,

biN + h1, LX, =: hi LX, + b1, LX?

.

(ii) In this case

X=

XI

ZI

X2

Z2

XN ZN

YI

Y

Y1,

YN

The normal equations are

LY, = LX,Yj and

LXjZ/

biN + b1, LX; + b3 LZ, 1,=: b l LX, + b1, LX; + b3 LX/Zj • 1, =: b l LZ; + b1, LXjZ; + b3 LZ/ .

b

=

~:J

123

102

[eh.7

Some Applications of Matrix Calculus

7.S PROBLEM 3 - MAXIMUM LIKELIHOOD ESTIMATE OF THE MULTIVARIATE NORMAL

=

Let X/(i 1,2, .•. , n) be /I random variables each having a normal distribution with mean IJ./ and standard deviation 0/, that Is

= n (1J.1o all .

XI

(7.22)

The joint probability density function (p.d.f.) of the n random variables is

=

!(xl>X2," .,xn)

1 (27T)n/21VII/2 exp

(-!ex -

p.)' V-I(X 2

-

p.))

(7.23)

where

(i=1,2, ... ,II)

-oo<X/
and

is the covariance matrix.

,,' = [IJ.IoI-ll,,,.,Pn],

I

.'I

and

=

01/

x' =

[XI,X2,''''Xn ]

(I '1= j)

Plj a/ OJ

all = a?

are the covariances of the random variables. Plj is the correlation coefficient between XI and Xj. The covariance matrix V is symmetric and positive defmite. Equation (7.23) is called a multivariate normal p.d.f. Maximum likelihood estimates have certain properties (for example, they are asymptotically efficient) which makes them very useful in estimation and hypothesis testing problems. For a sample of N observations from the multivariate normal distribution (7.23) the likelihood function is

_

L -

1

(27T)

nN/2

( I..!!:: L...

IN/'l exp - -

IV I

.

2

(XI-

I

p.) V

-I

(XI - P.

)}

/=1

so that 10gL

N I ~ = C--IogiVl-L... (x,-,,), V- 1 (x,-,,)

2 where C is a constant.

2

/=1

(7.24)

Sec. 7.5]

Problem 3

103

(u) The muximum likelihood estimate of Il

On expanding the last term of (7. 24), we obtain 1~

--2 2: {x/ V-I xI - Il' V-I

XI -

x/ V-I Il + Il' V-I Il}.

1=1

With the help of table (4.4) and using the result (x/ V-I)' = V-I XI

(since Vis symmetric)

we differentiate with respect to Il, to obtain

when

= 0

jJ.

};XI

N

x.

=

Hence the maximum likelihood estimate of" Is " = X, the sample mean.

(b) The maximum likelihood estimate of V·

We note the following results:

(1)

N

L y/V-IYI = tr(Y'V-Iy) 1=1

where

Y = [YI Y2'" YNl

and

YI = XI-I!

(i=1,2, ... ,N.

V-I is a symmetric matrix. (2) By Example 5.3, but taking account of the symmetry of V-I (see Example 4.4) a log IV-II - - - 1 - = 2V-diag {V}.

av-

(3) If X is a symmetric matrix

a tr(AX) ---'----' = A

ax

+A

I

- diag {A} .

Let A = yy' and X = V-I, then atr(yy'V- I )

aV-I

I

== 2Yy' - diag {yy } .

23

30'01. 104

Some Applications of Matrix Calculus

[Ch.7

We now write (7.24) as

N

1

10gL = C+ -log V- I - - tr(YY'V- I ). 2 2 ' Differentiating log L with respect to V-I, u~ng the esHmate p = i1., and the results (2) and (3) above, we obtain

a 10gL N i l , -I = - [2 V - diag {V}] - YY + - diag {YY } .

av-

2

2

Let Q = NV - YY', then

alog L aV-I

1

"" Q -

=0

2" diag {Q} 2Q

when

= dla!! {Q}

.

Since Q is symmetric, the only solution to the above equation is

Q = O. It follows that the maximum likelihood estimate of V is

7.6 PROBLEM 4 - EVALUATION OF THE JACOBlANS OF SOME TRANSFORMATIONS The interest in Jacobians arises from their importance particularly with reference to a change of variables in multiple integration. In terms of scalars, the problem presents itself in the following way. We consider a mUltiple integral of a subset R of an n-dimensional space

Lf(XhXl,'" ,xn ) d'1:1 dx l

... ,

dxn .

(7.25)

where f is a ph:ccwise continuous function in R. We consider a one to one transformation which maps R onto a subset T (7.26) and the inverse transformation XI

=

X'

= [XIoXZ, ... ,x n]

where

WI(Y), Xl

=

wz(y), ... , Xn and

= Wn(y)

y' = [YhYz,·.· ,Yn]

(7.27)

Sec. 7.6]

Problem 4

105

Assuming the first partial derivations of the inverse transformation (7.27) to be continuous. (7.25) can be expressed as

J![W\(Y).

WI (y) •...•

wlI(y)]IJI dYl dY2'" dYII

(7.28)

where IJ I can be wrl tten as

III

subject to

ax.

aXil

aYI

aYI

aYl

aXI

ax.

ax"

ay' aYl

aY2

aXI

ax.

aXn

aYII

aYn

aYII

I:.; ,.

(7.29)

IJI not vanishing identically in T.

Example 7.4 Let

aXI

L

I = 2

exp {-2xI

o<XI <

00.

+ 3X2} dXI dx.

0 <X2 < 00.

Consider the transformation YI = 2xI -X2 Y2 = X2 .

Write down the integral

corre~pondillg

to (7.28).

Solution We are given

R ;::: {(XI, X2); 0 <XI< "", 0 <X2 < co} . The above transformation (corresponding to (7.26)) results in the following inverse transforma tion (7.27) XI = 1(YI +h) X2 = Y2

which defines

123

Some Applications of Matrix Calculus

106 and by (7:29)

Hence

!

=

IJI

0 =

!

J:[!

/=

VI

[Ch.7

! .

+Y2),12] dYI d12

= {ex p (-YI

+ 2Yi)dYi dYl'

Our main interest in this section is to evaluate Jacobians when the transformation corresponding to (7.26) is expressed in matrix form, for example as

= AXB

Y

(7.30)

where A, X and B are all assumed to be of order (n X n). . As in section 5.2 (see (5.1) and (5.2» we can write (7.6) as

(7.31)

y = Px

where y = vee Y, x In this case

= vee X and P = 8' ® A .

ay = B®A' ax ax = [B®A'tl ay

and

= B-1®(A'r l

by(2.12).

It follows that

IJI

YI-

avec a vecX

I

1

181-n IAI- n

(by Property X, p. 27)

Example 7.5 Consider the transformation

Y = AXB where A

=

rL-l2-4] 3

and B = [ 21

Find the Jacobian of this transformation

(i) By a direct method (ii) Using (7.32).

11]'

(7.32)

Sec. 7.6 J

Problem 4

107

Solution (i) We have

X

= A-IYB- I = 1~ [3Y I + 4)/2 -3Y3 -

4Y4

YI + 2Y2 -Y3- 2Y4

3 =

(if

IAI =

2,

IB\ =

I

hence

-1 -2

4

2

4

0

3

-2 \JI= i.

8

-4 (ii)

-3

0

+ 6YJ + 8Y4J

-YI - 2yl + 2Y3

so that

ax ay

-3YI - 4Y2

==

+ 4)'4

4

4

Similarly, we can use the theory developed in this book to evaluate the lacobians of many other transformations.

Example 7. 6 Evaluate the Jacobian associated with the following transformation

(i) Y=X- 1 tii) Y = X2.

Solution

(i) From Example 5.2

so that

ay ax ax ay

= -X-I ® (X-I)'

= -X®X'.

Hence J = mod I : : 1 = \X ® x' \

= \X I-n IX I-n

(ii) From section 4.6

so that by the 2nd transformation principle (see section 5.3)

ay

- = X®I+I®X' ax and

J = IX®I+I®X'I-I

123 [Ch. 7

Some Applications of Matrix Calculus

108

7.7 PROBLEM 5 - TO FIND THE DERIVATIVE OF AN EXPONENTIAL MATRIX WITH RESPECT TO A MATRIX Since We make use of the spectral decomposition of an exponential matrix, we now discuss this technique briefly. Assume that the matrix Q = [qjj] of order (n X n) has eigenvalues

AI, Al,',', A" (n6t necessarily distinct) and corresponding linearly independent eigenvectors

The eigenvectors of Q' are

Y\, Yi,"" Yn • ThClse two sets of eigenvectors have the property xj Y/ =

a

Y/ xI

or (equivalen tly)

=0

(i "*' j)

(7.33)

and can be normalised so that xI Yi

=

1 or

Y; Xi = 1 (i = 1, 2, ... ,n) .

(7.34)

Sets of eigenvectors {Xi} and {Yj} having the properties (7.33) and (7.34) are said to be properly nomlalised. It is well known (see [18] p. 227) that exp (Qt) = P diag {eAlt, e A•t, ... , e A" t} p- I where P is the modal matrix of Q, that is the matrix

(7.35)

P = [XI:X2:.":X,,].

It follows from (7.33), (7.34) and (7.35) that

,

p-l

(7.36)

Y2

,

Yn Hence exp (Qt)

= [XI X2 ... x,,]

0

o

o

,

o

Yl

o

y~

,

Yn

Sec. 7.7]

Problem 5

that is

i

=

exp (Qt)

109

XjY! exp (Ajt)

(7.37)

/=1

The right hand side of (7.37) is kIlOwn as the spectral representation (or spectral decomposition) of the exponential matrix exp (Qt). We consider a very siniple Illustrative example. Exumple 7. 7 Find the spectral representation of the matrix exp (Qt), where

Solution

I, A2 = .-1;

x;

[2 -1],

x;

= [1 -1]

[1 1], y~ = [-1 -2] .

I

YI By (7.37)

[_~][1

exp(Qt) =

[ 2 2]

=

-1 -1

1] exp(t)+[_~][-l -2] exp (t) +

l-1 -2J 1

3

exp(-t)

exp (-t) .

Although we have considered matrices having real eigenvalues, and eigenvectors having real elements, the spectral decomposition (7.37) is also valid for complex elements as can be shown by a slight modification of the above exposition. By the use of (2.17), that is of the result exp (1 ® Q) = 1 ® exp (Q) we generalise the result (7.37) to exp (1 ® Q)t

= 'L(I ® x/y/) exp (A/t) .

(7.38)

We now consider the main problem, to obtain an expression for

a(ll/aZ where so that

cl>(t) = exp(Qt),

(7.39)

<1>(0) = 1,

(7.40)

d -

(1) =

Q,

dt and

Z = [zij]

is a matrix of order (r X s).

(7,41 )

123

110

Some Applications of Matrix Calculus

[Ch.7]

The matrix Q is assumed to be a function of Z, that Is Q(Z). Making use of the result (6.5), we can write

d a~ dt az

=

az = aQaz (/®~)+(/®Q) aclaz> a(Q~)

(7.42)

and from (7.40)

a

az (0)

(7.43)

= O.

We next make use of a generalisation of a well known result (see [19] p.68); Given d

-x = RX+BU dt

'and

X(o)

then

= 0, (7.44)

X = itexp {R(t -r)1BU(r)dr.

For

x ;: :;

04>

az'

R;:::; / ® Q

B

= aQ

az

I

and

= / ® (ll

U

the solution to (7.42) subject to (7.43) becomes

a4>

-

az

=

It

aQ

exp {I ® Q(t -r)} -

az

0

;:::;

t

Jo

""

[/ ®
~ (/ ® XiYi) exp (Ai(t -r» I

a , az [/Q ® XjY, 1exp O,;r) dr

1,/

(by 7.37 and 7.38) =

.2 (/ ® XIY;) aQaz-;:- (/ ®

Xjyj) exp (Alt)

I,i

It

exp «AI

-

A/)r) dr.

0

Hence,

a
where

az

""

aQ (i ® x/yj) exp (Alt) flj (t) az

= L, I ® XIY;) -

1,1

= t if AI = Ai ii/t);:::; (1/(AI-A/))[exp(A/-A/)t)-I] fu(t)

and

if A/*A/'

(7.45)

Solution to Problems

CHAPTER 1 (1)

(2) (a) The kth column of AElk is the ith column of At all other columns are zero. (b) The ith row of EikA is the kth row of A, all other rows are zero. AEik

=

Aeie~

=

A'ie~

EikA = eie~A = ejA'k

(3)

tf ABC = L

e;ABCei = L(eiA)B(Cei)

L A /. BC.,

=

(4)

tf AE/j

=

L e~ Eljek = L e~a,sErsEljek k

=

k",S

2: a,se~ e,e~eie;ek 2: ==

kiT,S

==

a,sflkrOslOjk

Ic, r,s

a/I'

123 380A

112

Solution to Problems

.L 2: = 2: ekBEilel<~1 L

A =

(5)

ail Ail =

tr(BEI/liil)Eil

1.1

/.1

=

l.i,I<

=

.L

ekBelfill<Ei/

I,i.k

2:

eIBe/EIi =

/

bllEIi = diag {B},

I

CHAPTER 2 ( 1) Since U is an orthogonal ma trix. the resul t follows. More formally,

L [Ers(m X n) r.s

® Esr(n X m)] [Esr(n X m) ® Erim X n)]

=L [Erim X n)Esr(n X m)] ® [Esr(n X m)Ers(m X n)] r,s

L [Oss Err Cm X m)] ® [orrEssCn X n)] r,s =

L Err(m

X m)] ®

[.2 Ess(n

= 1m ® In

(3) (a)

A®B =

::;=

Imn. the result follows.

r ~ r i

2 -1

0

2

o -1

0

(b)

X n)]

s

r

2

B®A =

~

-1

2

I

0

2

0

2

0

1 0 0 0

UI

= U2 =

0 0 0

0 0 0

0 0 0 (4)

See [18] p. 228 for methods of calculating matrix exponentials. (a) [2e -e- I 2(e -e- I )] exp(A) = -I e -e 2e- 1 - e

J

113

Chapter 2

[2e -e-1

(b)

.

,

0 -(e-e - I )

exp(A ®I) =

0

2e -e-

I

o

{'~'-' 2e

exp(A) ®I

c- I -e

2(e-e- l )

0

-e- I

0

2c- 1

0

e-I-e

0

-c 0

Hence exp (A) ® I = exp (A ® I) . (5) (a) A

_, [I IJ _, '[-4 -~J ' =

-1. -2

A-I ®B- I

B

'

['

~~ ~

=-

2

2 -1

-4

-2

3

(b)

As A®B =

~~

-3

(A ®B)-I =

4

1

8

3

-2

-1

-4

-3

['

3

-6

-3

1

3/2

-1/2

~3/2

so that

3

-J -1

2

~-2 I ]

, it follows that

3/2

-1/2

-1

4

-2

1/2

-3

-4

This verifies (2.12) (6) (a) ForA; Al = -1, 1..2 = 2, x; == [14] For B; III = 1, 112 == 4, y; = [I -1]

(b)

A ®B =

~6 ~ =~ =~J 8

4 -4

-2

8

12-4

--{)

==

and

x; == [11].

and

y~ == [1 2] .

E (say).

23 380A

114

Solution to Problems

C(A) = /A/-E/ = A4-SAJ- 30A2 +40A-30 = (A + I)(A + 4)(A - 2)(A - 8) . Hence the eigenvalues of E are

{-I, -4, 2, 8}

= {AIJ.lb A,II.l2, A,2J.lI, A2J.l2}'

The corrcsponuing eigcllv(!ctors of H arC:

[~ .[il

tJ

'nd

[iJ

(c) This verifies Property IX

(7) For some non-singular P and Q A = p-Icp

and

B = Q-IDQ.

Hence A ®B = p-1CP®Q-1DQ l

= (r ® Q-I)(CP®DQ) l

= (P ® Qr (C ® D)(P ® Q) =R-1(C®D)R

where

by (2.11) by (2.12) and (2.11)

R = P®Q.

The result follows. CHAPTER 4

(1)

(2)(a)

:;

IXI =

~ [~:

ay

o

ax

x sin x -exp (2,)

~= ax

[ex

-cos

-x

xl

sinxJ.

IXI = exp (x) sin x -x cos x, alxl

ax

=

Ix L-2e

l J.

-2e 2x 2x

sinx

~

o

2e2X] -x-2

4x cos x

115

Chapter 4 (b)

a~:1 = [;~: ;~:J alxl

ax

= 2[X/j]-diag {Xii}

= (3)

Y =

[~X ~::SXJ.

=

[~~ex ~~i::J -[~ ~nxJ

xli + X~I

XII XI2

Xl2Xli +X22X21

X!2

[XI3 X II

+x2J X 2i

+ x21 X22 + X12

XI3 X l2 +X23 X 22

hence

(a)

aX21

" .

~2~

x22

2X21

ay -- =

r

X22

0

X2J

0

oJ.

From Example 4.8

~~ ~J ~" 0

o

(b) Since Y13 = aYI3 ax

xU x 13

21

XI2

x22 X2)

0

X23

0

X21

+

[" X'] ~ ~J x12 X22

XI3 x23

+ X21x23

l" "J

XI~

=

which is the result in Example 4.8.

0

1 0

123

116

Solution to Problems

(4)(a)

aYi,

= E-jX'A'

ax'

+ A'X'E... 'I

(b)

°Yil = AX'Eii

ax

+ E1,X'A' .

(5) By (4.10)

alYI

= tr {IYI(y-I)'B'E;sA'}

= WI tr {A'(y-I)'B'E;s} =

IY I (vee Ers)' vee

[A'(y-I)'B']

IAXBlzrs where

[zrs] = Z = A'[(AXBtl]'B' .

(6) (a) Since

3 (XX') , , - - = ErsX +XErs ,

ax,s

(b)

CHAPTER 5

(l) Since Y2l

+ al2 x l2 a21 x 11 + 012 X I2

Y12

a11 x 2l

Y22

a2l x 2l

Yll

allx ll

+ al2 x 22 + a22 x 22

117

Chapter 5

0]

all

o

a21

o

0

all

a22

•

(2) (a)

oveeY _ (B®A')(n)

a vee X (b)

(3)(a)

o vee Y a veeX

= X ®I

+I

by(s.i8)

® X'

a tr Y ax,s

(vee E,.)' (vee A'il') ,

hence

a tr Y ax (b)

a tr

Y

ax,s hence aIr Y

ax (c)

a tr Y ax"

= A'lJ'. 2 tr E;sX' ,

2X' . 21rE;sX,

hence

a tr Y - - - 2X. (4)(a)

ax a tr Y axrs

-Ir E;s (X- 2 )' ,

-Ir X-I /.:.~. X-I

hence

a tr Y ax

(b)

(x-2)' .

a tf Y = -tf AX-IErS x-In , ax,s hence

a tf Y ax

-(X-lBAx- 1),

•

123 380A

118

Solution to Problems

(c)

atrY

ax"

hence a tr Y

- - = n(X ax"

(d)

exp (X) = I

II

-

1 )'

I

+X +-

I

+-

Xl

21 hence by the result (c) above

31

Xl + ...

a tr Y , ax= exp(X). (5) (a) (i)

X l1 dx l2 +Xl2 dx ll

+X22dX21]

+ 2X22dxn

2X12dxI2

=

+~21dx22

[:::~ :::~] [:~: :~:] + [::~ ::~] [::~: ::~:] (dX)'X

+ X'(dX)

.

(iii) dY =

2X l1 dX l1 +X12dx21 +X21dx12 [

xlldx21 +X21 dx ll +X22 dx 21 +X21 dx22 Xli

Xll dXll [ X21 dX ll

+ X12 dX 21 + xn dX21

dx 12

+ X 12 dx II + X 12 dx 22 + X 22 dx 12 ] X21 Ux 12 +X12 Ux 21 + 2X'22 Ux 22

+ X12dX22] X21 dX12 + x22 dX22

Xll dX12

XlldXll +X2Idx12 [ Xll dX21

= X(dX)

+ (dX)X .

+ x21 dX21

x 12 dx ll +X22dX12]

X12 dX21

+ X22 dX22

119

Chapter 6

(b)

Write Y== UVwhere U=AXand V=BX then dY== U(dV) + (dU)V == AXB(dX) + A (dX)BX .

I

CHAPTER 6

~-'In(x" + x,,)

(I)

ay -

ax

0

xli

XI2exllx,.

Xllx

0

Xu e

0

Xli

-sin (x 12

0

0

==

0

0]

X22

..

+ X 22)

~12

(2) and so on

hence by (6.1) 1 0 0

I

0 0 0 0

0 0 0 0

ax ax

0 0 0 0

-

O.

1 0 0 I 0 0 0 0 0 0 0 0 0 0 0 0

0 0 (3) Since X-I /::,.

where /::,. == XIIX22

L"

-X21

- Xl2X2l>

-x"J

XII

we can calculate aX-ljaxrs , for example

I

123

380A 120

Solution to Problems

Hence ilX- 1

~

oX

A'

~

~2

-x 12X22

X

-X.2 X 21

22

J

-x21 X 22

X I2X21

X~I

XlI

-x12xn

xI.

X 12 X21

-XlI X 1.

XU X 22

~XI2XU

-XU X .1

xii

-XlIX.I

o

o

=

= -(I®X- I ) O(I®X- 1). (4)

A ®X- 1 == -1 A

a12 x ,2

allxll

-allx:!l

a12x ll

a21 x 22

-a2l x l2

a22 x 22

-a21 x 12

-a2l x 21

a21 x ll

-a2Z x "1l

a22x ll

where A = XllX22 - Xl2x,1 We can now calculate

a(A

l2 Xlj

-au Xu [a\lX22 -aUx,l

-a

•

® X-l)/axrs

axrs and form

o

o

0

o

ax

=A

-all

0

o

0

o

o

o o

o

o o o

o

o

a21

o o

a22

0

o

0

o

0

all

-all 0

-a12

0

0

o

o

0

all

o o o o

o o o

o

0

o

I

·1

'.i '.j

,I

I

i

.1

I

Tables of Formulae and Derivatives

'j

':1

"I - i

· I

·. ;

i j

Table 1 Notation used: A = [ailLE = [bill

,

Eii = e/ei Bij :: e/ ci = e/ el E;jer =

Bjre;

E;jErs = BjrEis EjjEjsEsm = E,m

if j =l=r A = l::Z;aijE;j

EjjErs = 0 I

l

.1

J

,

A. j = Aej Aj • = A'el

E/jAErs = ajrEij trAE = 'I.'£a/lbl/ / I

tr AS' = tr A'B.

trAB = (vecA')'vecB.

; ·

\

123

3S'OA 122

Tables of Formulae and Derivatives Table 2 A ®B A ® (aB) (A + B) ® C A ® (B + C) A ® (B ® C) (A ®il)' (A ®B)(C®D) (A ®Br l vec(AYB) IA ® ill

A ®B tr (A ® B)

A C±> B U

= laljBj

= a(A @ B) = A ® C + il ® C = A ® B +A ® C = (A ® B) ® C = A' ®B' = AC®BC = A-I ®B- I = (B'®A)vec Y = IAlm IBln when A and B are of ordt!r (/1 X n) and (m Xm) respectively == UI(B ®A)U21 UI and U2 are permutation matrices = tr A tr B = A QlI 1m + In ® B = 'L'LErs®E~s r s

Table 3

a(Ax)

:=:

A'

:=:

A

ax a(x'A)

ax a(x'x) ax

--:=:

a(x'Ax) ---:=:

ax

a7. ax

2x

,

Ax+Ax

ay az ax ay

Tables of Fonnulae and Derivatives Table 4

alex) ax alxl ax

IXI(X- 1), when elements of X are independent

2 [X,/l - uiag {XII}, when X is sYlllmetric.

ax axr .!' ax' -

oXrs

= Ers

= E;s

a(AXB) - - - = AErsB axrs

il(AX'B)

, = AErsB

oXrs

a(X'A'AX) axrs

E,.'A'AX

+ X'A'AErs

3(X'AX)

ax,s

I

a(x n ) = ax,s Ic=O

XlcErsXn-k-l

123

123

1;

124

Tables of Formulae and Derivatives Table S

a vee (AXB) a veeX

----=---'-

I,

a vee (X'A,X) a vecX J a vee (AX- B) a veeX

,

= B ® A

= U'(AX ® I)

+ (/ ® A 'X)

= _(X-I B)<® (x-I)'A'

I'

II ii

Table 6

I

I I

a log IXI

ax

= (X-I)'

aIXlr

- - = rIXlr(X-I)'

ax

a tr (AX)

ax a tr (A'X) -....:----=-

ax

a tr (X'AXB)

-~-~

ax

a tr (XX')

ax a tr (X")

ax

a tr(e x )

ax a tr (AX-I B)

=A =

,

A ,

= AXB+AXB

,

= 2X = nX"-1

=e

x

= -(X ~I BArl)'

Tables of Fonnula e Ilnd Derivatives Table 7

ax ax ax -

ax

(X symmet ric)

0+ U - L,Err ® Err = (]

ax' ax a(xy)

(clemen ts of X indepen dent)

-=U

ax (I ® Y) + (I ® az

-- = -

az

ax- 1 ax

a ex ® Y) az

ay az

X) -

= -(1 ® X-I)O( / ® X-l)

ax =

az

ra

Y

®Y

~

+ [/ ® Uil Laz ® X [1 ® U2 ]

J

125

123

38'01.

Bibliography

[1] Anderso n, T. W., (1958), An Introduction to Multivariate Statistical Analysis, John Wiley. [2] Athans, M., (1968), Tile Matrix Minimum Principle, Informa tion and Control, 11,592- 606. (3] Athans, M., and Tse, E., (1967), A Direct Derivati on of the Optimal Linear Filter Using the Maximu m Principle, IEEE Trans. Auto. Control, AC-12, No.6, 690-698 . [4] Athans M., and Schwep pe, F. C., (1965), Gradient Matrices and Matrix Calculations, MIT Lincoln Lab. Tech., Note 1965-53 , Leming ton, Mess. [5J Barnett, S., (1973), Matrix Differential Equatio ns and Kroneck er Product s, SIAM, 1. App/. Ma tho , 24, No.1. [6] Bellman, R., (1960), Introdu ction to Matrix Analysis, McGraw -Hill. (7] Bodewig, E., (I 959),Matrix Calculus, Amsterd am: North Holland Publishing Co. [8] Brewer, J. W. (1978), Kroneck er Product s and Matrix Calculus in System Theory, IEEE Trans. on Circuits and Systems, 25, No.9, 772-781 . (9J Brewer, 1. W., (1977), The Derivatives of the Exponen tial Matrix with respect to a Matrix, IEEE Trans. Auto. Control, 22, 656-657 . [lOJ Brewer, 1. W., (1979), Derivatives of the Characte ristic Polynom ial Trace and Determi nant with respect to a Matrix, IEEE Trans. Auto. Control, 24,787- 790. [11 J Brewer, J. W., (1977), The Gradien t with respect to a Symmet ric Matrix, IEEE Trans. Auto. Control, 22, 265-267 . [12J Brewer, 1. (J 977), The Derivative of the Riccati Matrix with respect to a Matrix, IEEE Trans. Auto. Control, 22, No.6, 980-983 . [13J Conlisk, 1. (1969), The Equilibr ium Covariance Matrix of Dynami c Econometric Models, American Stat. Ass. Journal, No. 64,277- 279. [14J Deemer, W. L. and Oikin, I., (1951), The Jacobian s of certain Matrix Transfo rmation s, Biometrika, 30, 345-367 .

w.,

~

Bibliography

I I! i.'.

I

127

[15] Dwyer, P. S. and Macphail, M. S., (1948), Symbolic Matrix Derivatives, Ann. Math. Statist., 19,517-537. [16] Dwyer, P. S., (1967), Some Applications of Matrix Derivatives in Multi· variate Analysis, American Statistical Ass. Journal, June, pt 62; 607-625. [17] Geering, ll. P., (1976), On Calculating Gradient Matrices, IEEE Trans. Auto. Control, August, 615-616. [18J Graham, A., (1979), Matrix Theory and Applications for Engineers and Mathematicians, Eills Horwood. [J 9] Graham, A., and Butglles, D., (1980), Introduction to Control Theory Including Optimal Control, Ellis Horwood. [20] Lancaster, P., (1970), Explicit Solutions of Unear MatriA Equations, SIAM Rev., 12, No.4, 544-566. : [21] MacDuffee, C. C. (1956), The Theory of Matrices, Chelsea, New York. [221 Neudecker, H. (1969), Some Theorems on Matrix Differentiation with special reference to Kronecker Matrix Products, 1. Arner. Statist. Assoc., 64,953-963. [23] Neudecker, II., A NOle alKronecker Matri.;'( Products alUi Matrix Equation

Systems. [24] Paraskevpoulos, P. N. and King. R. E., (1976), A Kronecker Product approach to Pole assignment by output feedback, Int. 1. Contr.. 24. No.3, 325-334. [25J Roth, W. E .• (1944), On Direct Product Matrices. Bull. Amer. Math. Soc., No. 40, 461-468. [26] Schonemann. P. H., (1965), On the Formal Differentiation of Traces and Detenninants, Research Memorandum No. 27, University of North Carolina. [27] Schweppe, F. C., (I973), Uncertain Dynamic Systems, Englewood Cliffs, Pren lice I-lall. [28J Tracy, D. S. and Dwyer, P. S., (1969), Multivariate Maxima and Minima with Matrix Derivatives, 1. Arner. Statist. Assoc., 64, 1576-1594. [29] Turnbull, H. W., (1927), On Differentiating a Matrix,Proc. Edinburgh Math. Soc., 11, ser. 2, 111-128. [30] Turnbull, H. W., (1930/31), 'A Matrix Form of Taylor's Theorem', Proc. Edi/lburgh Math. Soc., Ser. 2, 33-54. [31 J Vetter, W. 1., (1970), Derivative Operations on Matrices, IEEE Trans. Auto. Control, AC-15, 241-244. [32] Vetter, W. 1., (1971), Correction to 'Derivative Operations on Matrices', IEEE Trans. Auto. Control, AC-16, 113. [33] Vetter, W. 1., (1971), An Extension to Gradient Matrices,JEEE Trans. SysL Man. Cybernetics, SMC-l, 184-186. [34J Vetter, W. 1., (1973), Matrix Calculus Operations and Taylor Expansions, SIAM Rev., 2, 352-369. [35] Vetter, W. 1., (1975), Vector Structures and Solutions of Linear Matrix Equations, Lillear Algebra and its Applicatiol1s, 10, 181-188.

12

1

Bibliography

[W. 1., (1971), On Linear Estimates, Minimum Variance and LeastWeighting Matrices, IEEE Trans. Auto. Control, AC-16, 265[:-, R.I. and Mulholland, R. J., (1980), Kronecker Product Represenor the Solution of the General Linear Matrix Equation,JE.EE Trans. 'antral, AC-25, No.3, 563-564.

jex

c Rule ~rix, 88 tor,54 :teristic equatio n, 47 or,57 11 vector, 14 mion form, 47 'ained optimis ation, 94, 96

D

nposition of a matrix, 13 product , 21 ltive onecker product , 70 itriX, 60, 62, 64, 67,70, 75,81 alar function , 56,75 ctor,52 minant, 27,56 tion,94 E lValues, 27, 30 Ivectors, 27, 30 entary matrix, 12, 19 'anspose, 19 mential matrix, 29, 31, 42,108 G

ient matrix, 56 J

lbian, 53,109

K Kronecker delta, 13 produc t,21,23 ,33,70 ,85 sum, 30 L

Langrange multipli ers, 95 least squares, 94, 96, 100 M

Matrix calculus, 51, 94 compan ion, 47 decomp osition, 13 derivative, 37,60, 62,67, 70,75, 81,84, 88 differential, 78 elemen tary, 12, 19 expone ntial, 29, 31,42, 108 gradien t, 56 integral, 37 orthogo nal, 97 permut ation, 23, 28, 32 product rule, 84 symmet ric, 58, 95, 97 transitio n, 42 maximu m likeliho od, 102 mixed product rule, 24 multivariable system, 45 multivariate normal, 102 N

normal equatio ns, 95,101