Multiscale optimization methods and applications

(49)

where, for consistency, both sides are expressed as column vectors. Let Aj denote an eigenvalue of the matrix A~^B, and let r, denote the corresponding eigenvector. By equation (46), this eigenvector satisfies Qrj = 0, therefore we have

284

Ken Chan V^U. =0

Let the point P(xi, to ri(xi, X2,..., x„,l). lies on the surface Q. of <25 and obtain from

.

(50)

xg,..., x„) in nonhomogeneous coordinates correspond Since P lies on the surfaces A and B, therefore it also At this point, we evaluate the n-dimensional gradient equations (48) and (50)

(51) r Br)|^^^^ = AiV(r^Ar)|^^^^ . That is, the two quadratic surfaces A and B are tangent at point P. This proves the theorem. • Remarks: Note that this theorem is very general. If we are dealing with two n-diraensional surfaces other than hyper ellipsoids, then the singular quadratic surface may not be a point (which is a degenerate hyper elhpsoid). In general, we have a singular quadratic surface which may be a line, a plane, or any other surface Q as long as | Q |= 0. What this theorem says is that at the common point P the two surfaces A and B are tangent. This theorem does not say that this tangent point is necessarily an isolated point; it may be a point on a locus at which A and B are tangent. One example of this latter case is a sphere A inside a prolate ellipsoid of revolution B so that the locus of tangent points is a circle which is also the intersection of the plane Q with A (or B). Another example is an elhpsoid A inside a hyperboloid of one sheet B (both being concentric and having the same principal axes) so that the locus of tangent points is an ellipse which is also the intersection of the plane Q with A (or B). Corollary: If two quadratic surfaces A and B intersect (tangency excluded), then no point in the set of points comprising the intersection can correspond to an eigenvector of A~^B. Theorem 4. Converse Tangency. If two quadratic surfaces A and B are tangent at point P, then that point yields the homogeneous coordinates of an eigenvector Vi of A^^B. Proof. Let A and B be tangent at a point P(xj, X2,..., x^) whose homogeneous coordinates are denoted by r^(xj, X2,..., x^,l). Associated with A and B are two corresponding quadratic polynomials r-'^Ar and r-^Br. It follows that their gradients are proportional at point P V(r^Br) 1^^^,^ = A^ V ( / ^ A r ) |^^^,^

(52)

where AJ is some number. Let another quadratic surface Q be defined by r^Qr = r^Br - A-r^Ar = 0

.

(53)

Since A and B are symmetric, therefore Q is also symmetric. Associated with Q is the quadratic polynomial

Determining Intersection of Quadratic Surfaces ^ = r'^Qr = r ^ B r - A^r^Ar

.

285 (54)

Prom equations (52) and (54), we obtain V*lr=r', = 0

(55)

where V ^ is an n-dimensional row vector. Equation (55) states that it has n zero components. Since Q is symmetric, it may be shown that V ^ = 2[I„xnOnxi]Qr

.

(56)

If we evaluate the n-dimensional gradient V<5 at r^, it foUows from equations (55) and (56) that the column vector Qr^ has zero for the first n components, i.e., Qr'i = (0,0,...,0,a)

.

(57)

Since P lies on the surfaces A and B, therefore it also lies on the surface Q. Moreover, this point has nonhomogeneous coordinates obtained from v'^. Hence, the following equation holds

r'fQr', = 0 .

(58)

This yields 0 + l a = 0 so that a = 0

.

(59)

Consequently, Qr'i-0

.

(60)

That is, v[ is an eigenvector of A~^B. This proves the converse theorem. D Remarks: Let two quadratic surfaces A and B have tangent points Pi and P2. Then, the two vectors OPi and OP2 can be colhnear or non-colhnear. The first case can be easily constructed by using the example of a circle concentric with an ellipse whose minor axis is equal to the radius of the circle. This second case can be seen from the Illustrative Example for the case of a
286

Ken Chan p' + Po

.

(61)

However, this is not a convenient form for use in the extended system. Rather, let r and r ' respectively denote the homogeneous coordinates of the same point P. Then, they are related by the following matrix multiplication equation r = Tr'

(62)

where the (n+1) x (n+1) translation matrix T is given by T = T{po)

^nxn Po Olxn 1

(63)

By expanding the determinant of this matrix using the last row of elements and their cofactors, we obtain for all translations \T{Po)\ = l

•

(64)

Prom equation (62), it follows that r' = T^i(p„)r

.

(65)

v' = T{-po)v

,

(66)

Since we also have

it follows directly (without having to perform any matrix inversion) that T-i(po)=T(-p„)

.

(67)

T-\p,)^T^{p,) .

(68)

Note that

That is, the inverse of T is not equal to its transpose. However, by obtaining the inverse of the transpose of T given by equation (63), we obtain

[T'^{po)Y'=^[T-\p,)Y^l

.

(69)

That is, these two operations commute. The above discussion leads us to state the following: Theorem 5. Translation. The (n+1) x (n+1) translation matrix T given by equation (63) has the properties given by equations (64), (67) and (69). Let there be a rotation of axes from (e'^, 63,. • •, e^) to (e", e2,. • •, e^) where e^ and e" respectively denote the unit vectors associated with the positive x[ and x" axes. Let •jij denote the direction cosines of the "angle" between e[ and e'j

Determining Intersection of Quadratic Surfaces

lij=<-e';

.

287

(70)

Let S denote the n x n rotation matrix comprising these direction cosines S = [7d

•

(71)

Let p' = (x[,X2, • • •,x'^) and p" = {x'l,x'2, • • •,x'„) respectively denote the position of a point P in the ' system and the " system, both with the origin at O'. Then, we obtain p' = Sp"

.

(72)

Let r' and r" respectively denote the homogeneous coordinates of the same point P. Then, they are related by the following equation r' = R r "

(73)

where the (n+1) x (n+1) rotation matrix R is given by R = R(S)

^nxn

"nxl

Olxn 1

(74)

By expanding the determinant of this matrix using the last row or column of elements and noting that | S |= 1, we obtain for all rotations |R(S)|=:1

.

(75)

Since S is an orthogonal transformation (in the general sense), it follows from equation (68) that the extended rotation matrix R is also an orthogonal transformation. Consequently, we have R - i =: R^

.

(76)

That is, the inverse of the extended rotation matrix is precisely equal to the transpose. This is in direct contrast to the case of the extended translation matrix T. The above discussion leads us to state the following: Theorem 6. Rotation. The (n+1) x (n+1) rotation matrix R given by equation (74) has the properties given by equations (75) and (76). Remarks: For rotations, the usual and extended equations respectively given by (72) and (73) have the same form involving matrix multiplication. On the other hand, for translations, the usual and extended equations respectively given by (61) and (62) do not have the same form. One has vector sum while the other has matrix multiphcation. The older approaches in three dimensional analytic geometry (as in [Dre64]) have always used nonhomogeneous coordinates. As a result, they had to deal with translations and rotations in differing formulations. Thus, they encountered extensive algebraic

288

Ken Chan

manipulations which can be judiciously circumvented by using homogeneous coordinates. The advantages of this point are obvious in proving the following theorems on invariants. Theorem 7. Invariants. Let C denote the extended matrix whose elements are coefficients of a second degree polynomial with n variables. We shall associate these n variables with the coordinates in a rectangular system of the same dimension. We shall perform any sequence of translations and rotations taken in any order, even remembering that these operations do not commute. After such a sequence of transformations, we can replace the entire process by just a translation followed by a rotation or a rotation followed by a translation. However, in conformance with previously introduced notation in equations (62) and (73), we shall illustrate with a translation T and then a rotation R of axes taking the original system from the unprimed to the double primed variables. Then, the determinant of C is invariant under these transformations. Proof. Thus, we have r^Cr = (TRr")^C(TRr") = r"' R^T^CTRr" = r"' C"r"

(77)

so that the matrix C transforms according to C" = R'^T^CTR

.

(78)

If we take the determinant on both sides, it follows from Theorems 5 and 6 that |C"| = |C|

.

(79)

This proves the theorem. D Remarks: For the case of three dimensions. Theorem 7 is proved through much laborious pains by Dresden [Dre64] who used nonhomogeneous coordinates. If such an approach is taken for higher dimensions, it is conceivable that the methodology would soon break down. Suppose we wish to reduce C to its canonical form containing only diagonal second degree terms, only first degree terms that do not have a second degree term associated with them, and a constant term. This may be accomplished by first performing a rotation R* to eliminate the second degree off-diagonal terms, and then a translation T* to eliminate those first degree terms having a diagonal term associated with them. (Remember that the original coefficients of the linear terms have changed after performing the rotation R*.) Henceforth, we shall let C" denote the canonical form of C. This problem of reducing C to its canonical form is one of two very important transformations involving quadratic surfaces. The other is almost the inverse process: Given a canonical form C", transform it to another set of

Determining Intersection of Quadratic Surfaces

289

axes with orientation and displacement specified in the new coordinate system. A Httle consideration will reveal that, again, we first perform a rotation using the knowledge of direction cosines and then a translation to accomplish the required displacement in the new system. Interchanging these two steps will not yield the desired result. Corollary 1. It follows from the above theorem that (1) For the case of hyper ellipsoids with semi-axes ji, we have 1

ICI = |C"|

(80)

Til! (2) For the case of hyper hyperboloids, we have

ICI

1

|C"| = ±-

(81)

Uif Let us now return to the general case of two quadratic surfaces A and B defined in n-dimensional space, A being non-singular so that its inverse exists. Let D be defined as D =: A - ^ B

.

(82)

Even though A and B are symmetric (so that A~^ is also symmetric), it does not necessarily follow that A^^B is also symmetric. As an example, see the matrix in equation (18). From equation (82), we obtain |D|

(83)

IBI

Let (A ^)" and B " respectively denote the canonical forms of A ^ and B. This leads us to Theorem 8. Determinant. The determinant of the canonical form of the inverse of matrix A is equal to the determinant of the inverse of its canonical form, i.e., (84)

( A - i ) " = (A")-^ Proof. By Theorem 7, equation (79), we have |A"| = lAI (A-) Since | A

-1

=|(A-)|

(85) .

1, it follows from equations (85) and (86) that

(86)

290

Ken Chan IA"|

(87)

(A-^)

However, we also have = 1

(A")"

Consequently, we obtain equation (84). This proves the theorem. D

Remarks: Note that we proved the relation (84) relating determinants of (A~^) and (A )"•' even though it is not true in general that (A"^) = (A )"^. This can be seen from a simple example hke the matrix B given by equation (15) for which we observe that (B~^) ^ (B ) " ^ It is easy to prove that (B-^)" = ( B " ) - I if and only if T = I (i.e., Po = 0). Theorem 9. Combined Invariants. Let A and B be two hyper ellipsoids in n-dimensions. In this case, the inverse exists for both of them. Let D be defined by equation (82). Let ai and Pi respectively denote the semi-axes associated with A and B. Then, we have "

IDI

n\^f

2

(89)

Proof By Theorem 7, equation (79), and equation (83), we obtain (A-)

IDI

(90)

•IB''

Note that in deriving this equation, we do not assume that the transformations which take A to A" and B to B " are necessarily the same. Prom equation (80), we have |B"|

1

(91)

From equations (80) and (84), it follows that

(A-i)" = (A")-^ = - 1 1 " ?

(92)

i=l

By substituting equations (91) and (92) into (90), we obtain equation (89). This proves the theorem. D We can now rewrite equation (43) as |D - All - (-1)"+! [A"+i + d„A" + ... + diA + do]=0

(93)

where, in terms of the determinant | D |, we have according to equation (89)

Determining Intersection of Quadratic Surfaces

291

2

n

do-(-ir+MD| = (-l)"+^n?i ..:/?^

•

(94)

However, we can also write do in terms of the eigenvalues Aj as follows n+l

do = {-ir^'l[Xi

•

(95)

i=i

Prom these two equations, we obtain a relation between the product of the eigenvalues of A~^B and the product of semi-axes of the hyper ellipsoids A and B . This leads us to the following: Theorem 10. Eigenvalues Product. The (n+l) eigenvalues Aj of A~^B and the n semi-axes a, and /3j respectively of the hyper ellipsoids A and B are related by

n^'-nsA •

(96)

Remarks: If the two elhpsoids are translated or rotated, the matrix elements of A and B wiU change. Thus, the eigenvalues of A~^B wiU change, i.e., they are not invariant under rotation and translation. However, the semi-axes will not change because they are the diagonal elements of the canonical forms. This theorem states that do which is the product of the (n-)-l) eigenvalues is invariant under any combination of rotation and translation of A and B independently. This is not the case with the other coefficients dj in equation (93) which depend on the relative position and orientation of the two ellipsoids. This is evident even in the simple case studied in the Illustrative Example, as seen in equation (21). In these statements, it should be borne in mind that we are dealing with the eigenvalues {Aj ; i = 1,2,... ,(n-|-l)} of the extended matrix A~-^B in contrast to the eigenvalues {^,i ; i = 1,2,... ,n} associated with a quadratic form which are invariant under rotation. Needless to say, {ii, }
4 Application to ellipsoids The general theorems previously obtained for n-dimensional quadratic surfaces will now be apphed specifically to the three dimensional physical space so familiar to us. Moreover, we shall first restrict discussion to the case of two ellipsoids. (It helps to refer to Figure 3 which is really for elhpses in the two dimensional case.) Since n = 3, we have four eigenvalues of the matrix A^^B. Let us first consider a tangent point P when the ellipsoids A and B are exterior to each other (referred to as exterior tangency). We note that Vr-^Br is of opposite sign to Vr'^Ar. Hence, one eigenvalue (say Ai) is negative. Moreover, by Theorem 10, equation (96), the product of the four eigenvalues is positive.

292

Ken Chan

Consequently, there must be a second eigenvalue (say A2) which must also be negative. (The other two eigenvalues A3 and A4 must have a product which is positive. It does not matter presently what they are.) We note that when any negative A (not necessarily an eigenvalue) is substituted into equation (41), we obtain a quadratic surface Q which is an ellipsoid. However, when the eigenvalue Ai is substituted into this equation, Q becomes a singular ellipsoid which must therefore be a point. Let the eigenvalue Ai correspond to the eigenvector r i . Then, by Theorem 4, this eigenvector yields the nonhomogeneous coordinates of the tangent point P. Similarly, the eigenvalue A2 corresponds to the eigenvector r2 which yields the same tangent point P since there can be only one exterior tangent point between two ellipsoids. It follows that Ai = A2. Hence, the characteristic polynomial equation (43) has a repeated negative root when the surfaces A and B are exteriorly tangent. We note that the coefficients of this characteristic polynomial are functions of the elements of the matrices A and B. These are alternative ways of expressing the size, shape, orientation and relative position of the ellipsoids A and B. Hence, by subjecting these eUipsoids to undergo continuous variations in any fashion, the eigenvalues of equation (43) will also undergo continuous variations. This forms the basis of our approach in proving the ensuing general results without having to deal with the details of extremely complicated equations. If we move the two elhpsoids closer together or push them apart, then one of these configurations must correspond to the case in which the characteristic polynomial equation has two complex conjugate roots while the other configuration must correspond to the case in which it has two distinct negative roots. Let us first consider the case of two intersecting ellipsoids. For this, we have a locus of points in common between the ellipsoids A and B. Suppose this case corresponds to two negative eigenvalues Ai and A2. The eigenvalue Ai corresponds to an eigenvector ri which yields the nonhomogeneous coordinates of a point Pi on the singular surface Q which is a point. However, by Theorem 1, the set of points comprising the intersection between A and B must also lie on Q. This is impossible. Hence, the case of two intersecting ellipsoids must correspond to the characteristic polynomial equation having two complex conjugate roots. Consequently, the case of two ellipsoids exterior to each other must correspond to the characteristic polynomial equation having two distinct negative roots. Let us push the eUipsoids until they become tangent once again (referred to as interior tangency). At this tangent point, we note that Vr-^Br is of the same sign as Vr'^Ar. Hence, one eigenvalue (say Ai) is positive. Consequently, there must be a second eigenvalue (say A2) which must also be positive. In the configuration with interior tangency, there will be common volume between A and B. Depending on their size, shape, orientation and relative position, there are three possibilities: (i) They intersect again somewhere else; (ii) They become tangent again at another point; and (iii) They do not intersect again (one is contained entirely within the other). If we proceed as for exterior

Determining Intersection of Quadratic Surfaces

293

tangency, we can easily prove that for Cases (i) and (iii) we must have Ai = A2. To prove that Case (ii) also has two equal positive eigenvalues, let us consider Case (i) continuously deforming into Case (iii). We know that the eigenvalues also undergo continuous variations. It follows that at the transition (for which we have two interior tangent points), we stiU have Ai = A2. Note that this result is true for any two interior points in general. (In this process just discussed, the singular quadratic surface Q degenerates from a cone in Case (i) to a straight line or a plane in Case (ii) and then to a point in Case (iii).) We are able to prove this equahty of eigenvalues without having to deal with the details of the ehipsoidal surfaces. All the preceding theorems and discussion lead us to the following extremely general theorem on relative configuration based solely on eigenvalues. Theorem 11. Characteristic Equation. Let A and B denote two ellipsoids. If the extended 4^4 inatrix A~^B (or B~^A) has • • • • •

two distinct negative eigenvalues, then A and B are exterior to each other; two equal negative eigenvalues, then A and B are exteriorly tangent; two complex conjugate eigenvalues, then A and B intersect; two equal positive eigenvalues, then A and B are interiorly tangent at one or two points; two distinct positive eigenvalues, then A and B intersect or one is entirely within the other with no points of tangency.

Corollary 2. In view of Cases (2) and (4), it is also impossible for the matrix A~^B to have two pairs of equal eigenvalues of opposite signs (say Ai == A2 > 0 and A3 = A4 < 0). We will now apply this theorem to study the three cases above. For simplicity, we shall consider the two dimensional case as already exemplified by the Illustrative Example. We wish to observe how the eigenvalues change as the elUpse B (with semi-axes b and c) passes through the circle A (with radius a). Case (i) is illustrated by Stage 4 in Figure 4 for which c< a
For Case (ii), we consider a
294

Ken Chan

^r^f^

<==

(J) Two complex corrugate roots

(2) Two equal negative roots

-co

CD (4) Two equal positive roots

(1) Two distinct negative roots

(5) Two tlistinctpositivB roots

<=>

(9) Two distinct negative roots

c=0 (8) Two eqxjal negative roots

(e^ Two equal poatiye rooU

<:="

(7) Two complex conjugate roots

Fig. 4. Transition of Eigenvalues for the Case of c < a < b

(3) Two complex conjugate roots

(2) Two equal negative roots

(1) Two distinct negative roots

^

(4) Two equal positive roots

(9) Two distinctne^tive roots

(5) Two distinct positive roots

(8) Two equal ne^tive roots

(6) Two equal positive roots

(7) Two complex conjugate roots

Fig. 5. Transition of Eigenvalues for the Case of c < 6 < a

Determining Intersection of Quadratic Surfaces

295

stages are the same as before. If c is much larger than a, then Stages 4 through 6 (as well as Stages 8 through 10) collapse into one stage with only one tangent point. This critical condition depends on the curvature of the two conies. For example, see the Illustrative Example, equation (30). All these illustrations are included to clarify what could be an extremely difficult situation to describe analytically. They do not constitute a proof of the above theorem whose proof has been given already.

(J) Two complex corpugite roots

(2) Two equal regative roob

(1) Two distinct negative roob

(h ' CD ' (4) Two equal positive loots

(5) Two distinct positive mots


(9) Two distind positive roob

(6) Two eqoal positive loots

<^

(8) Two equal positive loots

(7) Two distinct pos itive roots

4

C10) Two eqaal positive roots

(ll)'Iwocon^lexcor5ugate loob

(12) Two equal negative roob

4

(i;^ Two distinct negative roob

Fig. 6. Transition of Eigenvalues for the Case of a < c < b

Ken Chan

296

Finally, consider Case (4) of Theorem 4 for which we have two equal positive eigenvalues such that A and B are interiorly tangent at two points Pi and P2. This is illustrated in Figure 7 for two ellipses. Note that Figure 7a depicts the case of two colhnear vectors OPi and OP2, each of which corresponds to two equal positive eigenvalues Ai = A2. Also note that these two colhnear vectors are not coUinear in the extended form since they both have 1 in the last component. What is not so obvious is that in Figures 7b through ??d it is also true that each of the two vectors OPi and OP2 has the same two equal positive eigenvalues Ai = A2 even though OPi and OP2 are not cohinear. This property follows from the fact that because Pi and P2 are common to A and B, they also lie on the singular quadratic surface Q defined by Q = B - AiA.

(a) - Collinear Vectors Figure

(b) - Non-collinear Vectors

0'

2 (c) - Non-coIIinear Vectors Figure

^2 (d) - Non-collinear Vectors

Fig. 7. Illustrative Cases of Two Vectors with Equal Eigenvalues

Theorem 12. Multiple Interior Tangent Points. Let A and B denote two ellipsoids. Suppose the extended 4 '>^ 4 matrix A~^B (or B~^A) has two equal positive eigenvalues (say Ai = A2) such that A and B are interiorly tangent at two points Pi and P2. Then, the two vectors OPi and OP2 are each associated with the same two equal positive eigenvalues Ai and A2. These

Determining Intersection of Quadratic Surfaces

297

tangent points may not be isolated points; they may be on a locus of points at which A and B are tangent. Corollary 3. / / the singular quadratic surface Q defined by Q = B - XiA is a straight line, then there can be at most only two interior tangent points common to A and B. Corollary 4. / / the singular quadratic surface Q defined by Q = B - XiA is a plane intersecting A (or B), then there are infinitely many interior tangent points common to A and B. These tangent points form a locus which is given by the intersection of Q and A (or B). Remarks: A singular conic can only be a point, a pair of intersecting lines, a pair of parallel lines or a coincident line. Therefore, there can be only one or two tangent points between two conies in general. In contrast, a singular degenerate quadric (which is of rank less than 3) can only be a straight line, a pair of intersecting planes, a pair of parallel planes or a coincident plane. If we also include a point (which is actually a singular non-degenerate quadric of rank 3), then for these singular quadrics Q there can be only one, two or infinitely many tangent points between two quadrics A and B in general. However, the author has not analyzed the case of other singular non-degenerate quadrics (a double elliptic cone and a cyhnder of the elliptic, hyperbolic or parabolic type) to arrive at any conclusion. It is possible that the above statement is still valid.

5 Discussion It is obvious that the foregoing theorems are applicable to the case of n dimensions. All the results we obtained in the simple Illustrative Example on conies become so obvious now. If we had attempted a more difficult two dimensional or a three dimensional problem, we would have been mired in excessive comphcated mathematical details without being able to obtain the answer. We achieved our objective by using abstract symbolism and invariant properties of the extended A~^B matrix which circumvented substantial unnecessary algebraic manipulations. Moreover, we may apply the above process to analyze other configurations such as an ellipsoid and a hyperboloid of one or two sheets, an ellipsoid and an elliptic paraboloid, two hyperboloids of two sheets, etc. At a tangent point for each of these cases, we have to cautiously determine whether Vr-^Br is of the same sign as Vr'^Ar or otherwise. This is especially true when we deal with open surfaces such as hyperboloids of one or two sheets, elliptic paraboloids and hyperbolic paraboloids. The convention for this sign determination is intrinsically given by the polynomials associated with the quadratic surfaces. Rather than delve into these more difficult topics, it is instructive to note that all the classical analyses of lines and planes intersecting or tangential

298

Ken Chan

to conies and quadrics can now be easily and compactly treated in a unified way. That is, all the equations previously derived for secants, tangents, normals and contact points can now be obtained and studied more qualitatively. Moreover, we can also extend this approach to study the intersection of other improper quadrics such as cyhnders of the elliptic, hyperbolic or parabolic types with any of the five proper quadrics. Even these simpler topics will provide substantial material for further investigations.

6 Conclusion The foregoing analysis presents a bridge between two seemingly disparate topics in mathematics. One deals with the intersection or tangency of surfaces in solid analytical geometry while the other reveals the relationship of the eigenvalues of an associated problem in linear algebra.

References [AGOl] S. Alfano and M. L. Greer, "Determining If Two Ellipsoids Share the Same Volume," AAS/AIAA Astrodynamics Specialists Conference, Quebec, Canada, July 2001. [ChaOl] K. Chan, "A Simple Mathematical Approach for Determining Intersection of Quadratic Surfaces," AAS/AIAA Astrodynamics Specialists Conference, Quebec, Canada, July 2001. [Pet66] A. J. Pettofrezzo, Matrices and Transformations, Dover Publications, New York, pp 103-110 (1966). [Dre64] A. Dresden, Solid Analytical Geometry and Determinants, Dover PubUcations. New York, pp 197-205, 230 (1964). [GKT68] Granino A. Korn and T. M. Korn, Mathematical Handbook for Scientists and Engineers, McGraw-Hill, New York, p 75 (1968).

Applications of Shape-Distance Metric to Clustering Shape-Databases Shantanu H. Joshi^ and Anuj Srivastava^ ' Department of Electrical Engineering, Florida State University, Tallahassee, PL 32310 USA. [email protected] ^ Department of Statistics, Florida State University, Tallahassee, FL 32306 USA. anujOstat.fsu.edu

Summary. Based upon the geometric approach to clustering outlined in [KSMJ04], this paper presents an application to hierarchical clustering of imaged objects according to the shapes of their boundaries. The shape-distance used in clustering is an intrinsic metric on a nonlinear, infinite-dimensional shape space, obtained using geodesic lengths defined on the manifold. This analysis is landmark free, does not require embedding shapes in R^, and uses ODEs for flows (as opposed to PDEs). Clustering is performed in a hierarchical fashion. At any level of hierarchy clusters are generated using a minimum dispersion criterion and an MCMC-type search algorithm is employed to ensure near-optimal configurations. The Hierarchical clustering potentially forms an efficient (O(logn) searches) tool for retrieval from shape databases.

1 Introduction Numerous problems in pattern recognition deal with classifying and labeling objects of interest present in the observed images. Discriminating features of the objects can be characterized according to their location, pose, textures, colors and shapes. Shape is a feature that is receiving widespread attention in analysis of images or improving understanding of objects that are changing in a scene. Analysis of shapes, especially those of complex objects, is a challenging task and requires sophisticated mathematical tools. Various applications of shape analysis include biomedical image analysis, automatic surveillance, biometrics, military target recognition and general computer vision. There have been various breakthroughs in understanding of shape. A majority of this research has been restricted to landmark-based analysis [DM98], where shapes are represented by a course, discrete sampling of the object contours. Shape clustering and learning using the resulting Procrustean analysis are discussed in [DJDOl, DSJ99]. Klassen et. al. [KSMJ04, SMKJ03], propose a new metric on the space of continuous, closed curves in IR'^, without a need for defining landmarks. The computation of this metric obviates the use of

300

Shantanu H. Joshi and Anuj Srivastava

computationally expensive tools such as Euclidean embeddings and nonlinear PDEs and uses ODEs instead. The idea involves identifying a space S of allowable shapes and imposing a Riemannian structure and further utilizing its geometry to solve optimization and inference problems. For planar curves in M?, of length 2TT and parameterized by the arc length, the coordinate function a{s) relates to the direction function 6{s) according to a{s) = e*^^*\ i = A/—1. The planar closed curves are represented by their angle functions. Considering closed curves, and making them invariant under rigid motions (rotations, translations), and uniform scaling, one obtains: C=ie£

L ^ l ^ f ^ e{s)ds = TT, f ^ e^^'Wds = o | .

(1)

The paper [KSMJ04] describes algorithms for computing geodesic paths between any given shapes. It also defines an intrinsic mean on the shape space and describes an algorithm of computing the mean for a set of shapes. The length of the geodesic d{9i,9j) for any 6i,9j S S can be used to quantify shape differences. This metric once computed, can be used to solve a host of challenging problems in image analysis, segmentation and feature extraction. In this paper in particular we focus on the clustering algorithm used to partition a group of shapes. Clustering can be further extended in a hierarchical fashion leading to the organization of a large database of objects according their shapes in a manner that allows for efficient searches and retrievals.

2 Shape Clustering An important need in shape studies is to classify and cluster previously observed shapes. In this section, we develop an algorithm for clustering of objects according to shapes of their boundaries. It is assumed that the task of boundary extraction or segmentation of the shape from the image is already completed. Thus we are presented with a set of 2D planar closed contours in the form of their co-ordinate functions. The shape-distance metric is computed pair-wise between all the set of shapes. In Euclidean spaces with the usual metric, there are several algorithms for clustering that can be adapted to the shape space S using its Riemannian structure. However, to keep the algorithm simple and efficient, we restrict to a simple modification of the kmean clustering idea. This modification is applied since the computation of means in S is an iterative and a computationally costly procedure, and we try to minimize that calculation. 2.1 Minimum Pair-wise dispersion Clustering We view clustering as a problem of partitioning n shapes (in S) into k clusters. To motivate our algorithm, we begin with a discussion of a classical clustering

Applications of Shape-Distance Metric to Clustering Shape-Databases

301

procedure for points in Euclidean spaces, which uses the minimization of the total variance of clusters as a clustering criterion. More precisely, consider a data set with n points {yi,y2, • • • ,yn} with each y, G R'^. If a collection C = {Ci, 1 < i < fc} of subsets of W^ partitions the data into k clusters, the total variance of C is defined by Q{C) = S j = i Ylyed 11^ ~ MilPi where n IS the mean of data points in d. The term Y^^Q. \\y — MilP can be interpreted as the total variance of the cluster Q . The total variance is used instead of the (average) variance to avoid placing a bias on large clusters, but when the data is fairly uniformly scattered, the difference is not significant and either term can be used. The widely used k-Means Clustering Algorithm is based on a similar clustering criterion (see e.g. [JD98]). The soft k-Means Algorithm is a variant that uses ideas of simulated annealing to improve convergence [BH91, R98]. These ideas can be extended to shape clustering using d{9, iXiY instead of \\y — HiW^, where d{-, •) is the geodesic length and /Xj is the Karcher mean of a cluster Ci on the shape space. The Karcher mean is an intrinsic mean on the shape space that is obtained after an iterative procedure that minimizes the average variance for all the set of shapes. However, the computation of Karcher means of large shape clusters is a computationally expensive operation. Thus, we propose a variation that replaces d{9,jii)'^ with the average distance-square Vi{9) from 9 to elements of Cj. If Ui is the size of Ci, then Vi(0) = :^ X^g'gc, d{9,9')'^. The cost Q associated with a partition C can be expressed as

Q(^) = E ^ ( E i=\

E

* \BaeCib
rf(^-^6)M.

(2)

J

The scale factor in the above equation of Q{C) ensures there is no additional preference to those clusters having a large number of shapes, while moving shapes from one cluster to other. The algorithm is outUned next. Algorithm 2.1 For n shapes and k clusters initialize by randomly distributing n shapes among k clusters. Set a high initial temperature T. 1. Compute pairwise geodesic distances between all n shapes. This requires n{n — l ) / 2 geodesic computations. 2. Pick a shape 9j randomly. If it is not a singleton in its cluster then compute Qf for alii = 1,2,...,k. 3. Compute the probability PT{J, i) for all i = 1,... ,k and re-assign 9j to a cluster chosen according to the probability PT{j,i). 4- Update temperature using T = T/0 and return to Step 2. We have used (3 = 1.0001 in our experiments. The set of all configurations of n shapes into k clusters is finite, and the stochastic nature of Algorithm 2.1 guarantees the convergence to the optimal

302

Shantanu H. Joshi and Anuj Srivastava

configuration as long as T is reduced slowly. Instead of analyzing convergence theoretically, we present some experimental results. The Algorithm 2.1 is apphed to a collection of 25 shapes from the Kimia database [SKK03] shown in Figure 1. Left panel of Figure 1 shows the 25 shapes. The Right panel of Figure 1 shows the cluster configuration for fc = 6 clusters. The success of Algorithm 2.1 in clustering these diverse shapes is visible in these results; similar shapes have been clustered together.

3J532:s-=21 iL^J^^JI^/.)} !>== i i = i > ^ 1== ZZSO-Y 3EaH E E H B E YYYY X fco-[>E

zzzz

Fig. 1. Left panel shows 25 objects whose shapes are analyzed here. They are numbered from top left to bottom right in increasing order. Right panel shows a clustering arrangement (row-wise) for the 25 objects into 6 clusters.

3 Hierarchical Organization An important goal of this paper is to organize large databases of shapes in a fashion that allows for efficient searches. One way of accomplishing this is to organize shapes in a tree structure, such that shapes are refined regularly as we move down the tree. In other words, objects are organized (clustered) according to coarser differences (in their shapes) at top levels and finer differences at lower levels. This is accomplished in a bottom to top construction as follows: start with all the shapes at the bottom level and cluster them according to Algorithm 2.1 for a pre-determined k. Then, compute mean shape for each cluster and at the second level cluster these means according to Algorithm 2.1. Applying this idea repeatedly, one obtains a tree organization of shapes in which shapes change from coarse to fine as we move down the tree. Critical to this organization is the notion of the intrinsic mean of shapes as discussed earlier. Shown in Figure 2 is an example of a tree structure obtained for clustering result for 25 shapes shown in Figure 1. At the bottom level, these 25 shapes are clustered in fc = 6 clusters, with the clusters denoted by the indices of their element shapes. Computing the means of each these six clusters, we obtain the

Applications of Shape-Distance Metric to Clustering Shape-Databases

303

last to bottom row of shapes. Repeating the clustering for fc = 4 clusters we obtain the next level and subsequently their means. In this example, we have chosen to organize shapes in four levels with a single shape at the top level. The choice of parameters such as the number of levels, and the number of clusters at each level, depends on the required search speed and performance.

19 22 10 13 20 21

3 5 6 7 24

9 15 16 17

4

11 12 18 23

2 14 25

Fig. 2. Hierarchical Organization of 25 shapes

4 Conclusion We have presented a hierarchical organization of shapes based upon the shapedistance metric which utilizes the Riemannian structure of the shape space. Clustering is performed efficiently by minimizing the pair-wise average variance within the clusters and can be used in clustering of shape databases of objects. Hierarchical clustering reduces the search and test times for shape

304

Shantanu H. Joshi and Anuj Srivastava

queries against large databases. This has enormous potential for systems which use shape based object retrieval.

References [BH91] Brown, D.E. and Huntley, C.L. (1991), "A practical application of simulated annealing to clustering," Technical Report IPC-TR-91-003, Institute for Parallel Computing, University of Virginia, Charlotesville, VA. [DM98] Dryden, I.L. and Mardia, K.V. (1998), Statistical Shape Analysis, John Wiley & Son. [DJDOl] Duta, N., Jain, A.K. and Dubuisson-Jolly, M.-P. (2001), "Automatic construction of 2D shape models," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 5. [DSJ99] Duta, N., Sonka, M., and Jain, A.K. (1999), "Learning shape models from examples using automatic shape clustering and Procrustes analysis," Proceedings of Information in Medical Image Processing. Vol. 1613 of Springer Lecture Notes in Computer Science, pp. 370-375. [JD98] Jain, A.K. and Dubes, R.C. (1998), Algorithms for Clustering Data, Prentice-Hall. [KSMJ04] Klassen, E., Srivastava, A., Mio, W., and Joshi, S.H. (2004), "Analysis of planar shapes using geodesic paths on shape spaces," IEEE Pattern Analysis and Machine Intelligence,Yol. 26 No. 3, pp. 372-383. [R98] Rose, K. (1998), "Deterministic annealing for clustering, compression, classification, regression, and related optimization problems," Proceedings of IEEE, 86(ll):2210-2239. [SKK03] Sebastian, T.B., Klein, P.N. and Kimia, B.B. (2003), "On aligning curves," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 25, No.l. [SMKJ03] Srivastava, A., Mio, W., Klassen, E. and Joshi, S.H. (2003), "Geometric Analysis of Continuous, Planar Shapes," Proceedings of Energy Minimization Methods in Computer Vision and Pattern Recognition., Vol 2683 of Springer Lecture Notes in Computer Science, pp. 341-356.

Accurately Computing the Shape of Sandpiles"* Christopher M. Kuster and Pierre A. Gremaud Department of Mathematics and Center for Research in Scientific Computation, Raleigh, NC 27695, USA. {cmkuster,gremaud}9math.ncsu.edu

Summary. Using an Eikonal formulation, we model the surface of sandpiles formed in regions containing obstacles. The fast marching method is adapted to have the optimal rate of convergence. We also apply the fast marching method to an industrial problem. K e y w o r d s : Fast Marching, upwind methods, Eikonal equation.

1 Introduction There are many examples of granular materials in industrial applications. These range from ore in mining operations to fine powders in the pharmaceutical industry. It is common to store these materials in piles, either in bins or in the open. Often, the area in which these piles are formed contains soHd obstacles which the grains must flow around in the forming of the pile. Our goal is to accurately measure the volume of such piles using only observations of the surface. In [ABG03], we created a mathematical model of the system, discretized the corresponding equations, and used the Fast Marching method to solve for the surface profile. This gave a method t h a t converged to the proper solution upon refinement, b u t it did so more slowly t h a n expected. Here, we modify the procedure to improve accuracy. T h e only physical parameter t h a t we consider for our model is the angle of repose. This is the maximum angle of inclination the surface of the grain can have and still be at rest. It roughly corresponds to the angle of internal friction of the material. For the purposes of our model, we assume t h a t the angle of inclination of the surface from vertical is equal to the angle of repose at every point. In one dimension, our assumption leads to the equation This research was supported by the National Science Foundation (NSF) through grant DMS-0244488.

306

Christopher M. Kuster and Pierre A. Gremaud |/'(x)|=tan5

where f{x) defines the height of the material at a point x, and S is the angle of repose. Requiring this equation to be satisfied nearly everywhere leads to an infinite number of solutions (See figure 1 (left)). In order to obtain a unique solution, we add the condition that the surface decrease with distance from the source point (See figure 1 (right)). This is commonly referred to as the viscosity solution. Generalizing this to multiple dimensions results in the Eikonal equation |V/(a:)| = t a n ^

Fig. 1. Possible solutions to the 1-D equation Obstacles can be modeled by modifying the right hand side in an appropriate way, namely

\yf{x,y)\

=

S{x,yJ{x,y))

where S, the slowness function, is defined by f tan ^ if {x, y, f{x, y)) ^ Obstacle, loo if (a;,y,/(x,y)) S Obstacle Travel time problems are closely related to the above formulation. In this paper, we look at prismatic obstacles. That is, obstacles with a cross sectional area that does not change with height. In this case, the slowness function is only a function of x and y. It is possible that our methods can be adapted to non-prismatic obstacles.

2 Numerical M e t h o d s One standard way of solving Eikonal equations is the Fast Marching Method (FMM) ([Tsi95, Set99]). The FMM is a wave propagation method based on upwind formulations. It starts with known values on a region and propagates the solution outward from there. The FMM is described by the following algorithm

Accurately Computing the Shape of Sandpiles f = ( o o , . . . ,00) "front" = "source" while "front" ^ 0 fmin = m i n f G " f r o n t " remove tmm from " f r o n t " add n e i g h b o r s of fmin t o " f r o n t " f o r ie G " f r o n t " solve F(fi,,..,f^_i, ff,f^+i,...,fN) = 0 end end

for

307

f^

In order to find fmin, Fast Marching uses a heap sort [Wil64]. This is a d a t a structure t h a t requires O ( l o g n ) operations in order to add, remove, or u p d a t e a node, where n in the number of nodes in the d a t a structure. T h e Fast Marching Method malces the assumption t h a t the slowness function is continuous. This is not the case in our problem, and in fact, our slowness function has a singularity inside every obstacle. When applying the F M M with a uniform mesh (see figure 2 (right))to a region with a cyhndrical obstacle, a loss of accuracy is observed. By adding a small number of extra mesh points around the b o u n d a r y of the obstacle, the F M M can be made to work weU in a region with singularities. In particular, new mesh points are added at the intersection of the hnes in the uniform mesh with the obstacle boundary (see figure 2 (left)).

41

()

4

•

't

•

41

(I

(

' t ^

II ^1

•

m

i

f

p

• >—

•

• •

0

•


s

;,

• >—

A

^

——^

i

• •

h-*—s

«

Fig. 2. Uniform Grid (left), Modified Grid (right)

This addition of nodes creates a non-Cartesian mesh near the obstacle. We use the process described in [SVOO, SV03], and consider a node XQ at which the solution / is t o be computed and two neighboring nodes xi and X2 at which the values of / {fe, i = 1,2) and its derivatives {dxfe,dyfe, i = 1,2) are known or have already been computed (See figure 3). We define

Ne

XQ

— X(

\Xo — X(

1,2

and

Af •

No

308

Christopher M. Kuster and Pierre A. Gremaud

where A/" is a 2x2 nonsingular matrix, assuming XQ, xi and 0:2 are not co-linear. The directional derivatives in the directions A''i and A^2 are approximated by Def = ,

\

\xo

£=1,2

(to 1st order),

Def = 2/° 2^!\ -Nf\dJt,dyfe\ £=1,2 (to 2nd order). I Xo Xe\ Those approximate directional derivatives are linked to the gradient by D,J = Ne-Vf

(1)

-Xi\

+ 0{\xo-Xin

or

1 5 / = A ^ V / + 0(/i«),

(2)

(3)

where Df = [Di/, D2/], h = max{|xo — xi|,|a;o — X2I}, and a = 1 or 2 depending on whether (1) or (2) is respectively used. Solving for V / and plugging into (1) results in the quadratic equation defining the unknown /o Df\UM'r'Df

= {S{x^))\

(4)

Fig. 3. Direction vectors for non-Cartesian grid.

3 Results 3.1 Without Obstacle With no obstacle, the solution is a cone, see figure 4 (left). Since there is no obstacle, the slowness function is continuous and the results are identical to those obtained by the Fast Marching Method on a uniform grid. The order of convergence is first order or second order depending on the approximation used for the directional derivatives (1) or (2).

Accurately Computing the Shape of Sandpiles

309

Fig. 4. Contours of surface ( f ) with and without obstacles

3.2 With Cylindrical Obstacle The exact solution is the generalized distance function from the source. This is consistent with observation (see figure 4 (right)). Table 1 is a grid refinement study looking a t the convergence rate of the L~ and L" errors. With a uniform grid, accuracy is lost due t o the singularity at the obstacle. By adding in nodes around the obstacle boundary, the second order accuracy expected from our second order discretization is preserved in the L1 norm. The first order results obtained for the Lm norm are expected due t o the "shock" behind the obstacle. Figure 5 shows the absolute error in the solutions produced using the uniform grid (Left), and the modified grid (Right) with M = 1600.

Fig. 5. Error plots for: Uniform Grid (left), Modified Grid (right)

3.3 Application

An application of this process is the determination of the volume of material in a bin with inserts, see figure 6. From this picture, it can be seen that the insert is not a prismatic obstacle. Preserving optimal accuracy is much more challenging here as the (x, y)-projection of the line of contact between

310

Christopher M. Kuster and Pierre A. Gremaud

M 25 50 100 200 400 800 1600 3200

L' 2.91(0) 1.71(0) 1.07(0) 6.42(-l) 3.76(-l) 2.21(-1) 1.31(-1) 7.80(-2)

M 25 50 100 200 400 800 1600 3200

L' 2.52(0) 1.45(0) 6.36(-l) 3.02(-l) 1.52(-1) 6.91(-2) 3.42(-2) 1.48(-2)

first order methods standard method rate L' rate L°° rate L' 9.00(-l) 6.11(-1) 1.13(0) .77 5.24(-l) .78 3.52(-l) .79 7.58(-l) .68 3.27(-l) .68 2.23(-l) .66 4.41(-1) .74 2.00(-l) .71 1.46(-1) .61 2.39(-l) .77 1.20(-1) .74 9.36(-2) .64 1.27(-1) .77 7.24(-2) .73 6.07(-2) .62 6.38(-2) .76 4.40(-2) .72 3.82(-2) .67 3.07(-2) .75 2.68(-2) .72 2.44(-2) .65 1.49(-2) second order methods standard method rate L ^ rate L°° rate L' 1.07(0) 8.99(-l) 1.17(0) .80 5.94(-l) .85 7.39(-l) .28 2.83(-l) 1.2 2.45(-l) 1.3 3.10(-1) 1.3 1.23(-1) 1.1 1.20(-1) 1.0 1.73(-1) .84 3.29(-2) .99 6.16(-2) .97 8.49(-2) 1.0 1.52(-2) 1.1 2.91(-2) 1.1 4.41(-2) .95 2.06(-3) 1.0 1.45(-2) 1.0 2.31(-2) .93 4.82(-4) 1.2 6.36(-3) 1.2 1.12(-2) 1.0 1.15(-4)

modified method rate L^ rate L°° 3.04(-l) 1.44(-1) .58 2.00(-l) .60 9.18(-2) .78 1.13(-1) .82 4.92(-2) .88 6.10(-2) .89 2.65(-2) .92 3.24(-2) .91 2.02(-2) .99 1.64(-2) .99 1.54(-2) 1.1 7.91(-3) 1.0 1.14(-2) 1.0 3.88(-3) 1.0 6.77(-3) modified method rate L°° rate 1/ 2.84(-l) 1.16(-1) 2.0 9.59(-2) 1.6 4.54(-2) 1.2 4.26(-2) 1.2 1.97(-2) 1.9 1.14(-2) 1.9 1.22(-2) 1.1 5.32(-3) 1.1 5.93(-3) 2.9 8.22(-4) 2.7 3.11(-3) 2.1 2.47(-4) 1.7 1.68(-3) 2.1 7.64(-5) 1.7 8.83(-4)

rate .65 .88 .91 .39 .39 .43 .75

rate 1.4 1.2 .70 1.0 .94 .89 .93

Table 1. Convergence study for formally first and second order methods in the presence of a circular obstacle (Example 2); M measures the number of nodes on one edge of the domain, i.e., total number of nodes N = 0{M'^).

material and obstacle is not known in advance. The results shown below were computed on a uniform grid and are thus not optimally accurate. By applying the fast marching process to several different source heights, a chart can be made of the volume of material in the bin as a function of the height of the surface at the source. Figure 7 shows such a chart for materials with several different angles of repose.

4 Conclusions An Eikonal formulation of general sandpile formation is proposed. The problem is discretized and solved. It is shown how to maintain optimal accuracy with fast marching when obstacles are present. The feasibility of the approach is illustrated by the solution of an industrial problem. Future work will address how to maintain optimal accuracy for non-prismatic obstacles.

Accurately Computing the Shape of Sandpiles

311

Fig. 6. Example of a bin with an insert

Hopper wIBINSERT, Materials wldiffering angle of repose

0

I

I

I

I

2

4

6 Height (feet)

8

1

30 degrees 40 degrees ------50 degrees --.-----

10

Fig. 7. Volume as a function of height for various materials

12

312

Christopher M. Kuster and Pierre A. Gremaud

References [Abg96] R. ABGRALL, Numerical discretization of the first order Hamilton-Jacobi equations on triangular meshes, Comm. Pure Appl. Math., 49 (1996), pp. 1339-1377. [ABG03] S.A. AHMED, R . BUCKINGHAM, P.A. GREMAUD, C . D . HAUCK, C.M. KUSTER, M . PRODANOVIC, T . A . ROYAL, V. SILANTYEV, Volume

[HS99] [0S91] [QS02] [Set99] [SVOO]

[SV03]

[Tsi95] [Wil64]

determination for bulk materials in bunkers, submitted to Int. J. for Num. Math, in Eng., Center for Research in Scientific Computation, NCSU, Technical Report CRSC-TR03-24. C. Hu, AND C.-W. SHU, A discontinuous Galerkin method for HamiltonJacobi equations, SIAM J. Sci. Comput., 21 (1999), pp. 666-690. S. OSHER, AND C.-W. SHU, High-order essentially nonoscillatory schemes for Hamilton-Jacobi, SIAM J. Numer. Anal., 28 (1991), pp. 907-922. J. QiAN, AND W.W SYMES, An adaptive finite difference method for traveltimes and amplitudes. Geophysics, 67 (2002), pp. 167-176. J.A SETHIAN, Fast marching methods, SIAM Review, 41 (1999), pp. 199235. J. A. SETHIAN, AND A. VLADIMIRSKY, Fast methods for the Eikonal and related Hamilton-Jacobi equations on unstructured meshes, Proc. Natl. Acad. Sci. USA, 97 (2000), pp. 5699-5703. J. A. SETHIAN, AND A. VLADIMIRSKY, Ordered upwind methods for static Hamilton-Jacobi equations: theory and algorithms, SIAM J. Numer. Anal., 41 (2003), pp. 325-363. J.N. TsiTSiKLiS, Efficient algorithms for globally optimal trajectories, IEEE Trans. Automat. Control, 40 (1995), pp. 1528-1538. J.W.J WILLIAMS, Heapsort, Com. ACM, 7 (1964), pp. 347-348.

Shape Optimization of Transfer Functions^ Jiawang Nie and James W. Demmel Department of Mathematics, University of California, Berkeley, CA 94710, USA. {nj w,demmel}9math.berkeley.edu

Summary. We show how to optimize the shape of the transfer function of a linear time invariant (LTI) single-input-single-output (SISO) system. Since any transfer function is rational, this can be formulated as an optimization problem for the coefficients of polynomials. After characterizing the cone of polynomials which are nonnegative on intervals, we formulate this problem using semidefinite programming (SDP), which can be solved efficiently. This work extends prior results for discrete LTI SISO systems to continuous LTI SISO systems. K e y w o r d s : Linear system, transfer function, shape optimization, nonnegative polynomials, convex cone, semidefinite programming.

1 Introduction Consider the following linear time invariant (LTI) single-input-single-output (SISO) system i = Ax + bu

(1)

y = c^x + du

(2)

where yl € K " ^ " , 6, c G R " , d £ R. T h e transfer function is H{s) = d + A)~'^b, which can also be written as the rational function

c^{sl-

ELpQ'fc's'' = gi(g) Note t h a t deg[qi) < deg(g2) < n. Conversely any function H{s) of this kind is the transfer function of some LTI system. Any such a LTI system is called a realization of H{s). There are many such (algebraically equivalent) LTI systems [CD91, chap. 9]. * This research was supported by the National Science Foundation Grant No. EIA0122599.

314

Jiawang Nie and James W. Demmel

In many engineering applications, we want the transfer function to have certain attractive properties. For example, we may want the Bode plot (the graph of |i?(s)| along the pure imaginary axis s = j • UJ) to have a certain shape corresponding to some kind of filtering. In this paper we study the shape optimization problem of choosing the coefficients of the rational function H{s) so its Bode plot has some desired shape. Now consider a discrete LTI system, i.e. the governing differential equation (l)-(2) is replaced by the difference equation 52fc=i C(kz{n—k) = X]^=i Pe.v[n— t) where {u(fc)}^j is a sequence of discrete inputs and {z{k)}^^^ is the sequence of state variables. In this case there are several nice papers [AV02, GHNOO, WBV97] that show how to formulate the filter design problem as the solution of the feasibility problem for certain convex sets. The main idea is to apply the spectral factorization of trigonometric polynomials, a characterization of nonnegative univariate polynomials, and semi-infinite programming. This approach can be used to design the transfer function to be a bandpass filter, piecewise constant or polynomial, or even have an arbitrary shape. Our contribution is to extend these results to continuous time LTI SISO systems (l)-(2). In this case the transfer function is not a trigonometric polynomial and hence we cannot directly apply spectral factorization. Fortunately our transfer function is a univariate rational function, which lets us apply certain characterizations of nonnegative univariate polynomials over the whole axis (—00,00), semi-axis (0,oo), or some finite interval [a,5]; see section 2. Using these characterizations, we show how to solve the shape optimization problem for the following shapes: 1. 2. 3. 4.

standard bandpass filter design; arbitrary piecewise constant shape; arbitrary piecewise polynomial shape; general nonnegative function.

We will show that the first three shape optimization problems can be solved by testing the feasibility of certain convex sets, which are the intersections of certain hyperplanes and the cone of semidefinite matrices. This feasibility testing can be done efficiently using semidefinite programming (SDP) [VB96]. The fourth shape optimization problem can be solved by semi-infinite programming (SIP) [P0I97, WBV97]. We introduce some notation. For any m G N, denote by 5"* the vector space of m — by — m symmetric matrices, and let S^ be the intersection of S^ and the positive semidefinite matrices. A y B{A y B resp.) means that A — B is positive definite (semidefinite resp.). [r\ denotes the largest integer no greater than r. deg{p) is the degree of the polynomial p(-). Given a cone K C K^, y ^K 0 means that y G mtK, the interior of K. K* denotes the dual cone of K, i.e., /f* = {u e M^ : u^y > 0, Vy G K). The rest of this paper is organized as follows. In Section 2 we give a characterization of the cone of polynomials which are nonnegative on certain intervals. In Section 3, we reformulate the shape optimization problem for

Shape Optimization of Transfer Functions

315

transfer functions to be convex optimization, and also discuss related work. In Section 4 we show how to recover the transfer function from its absolute value. Section 5 draws conclusions.

2 Cone of nonnegative polynomials on intervals We characterize univariate polynomials which are nonnegative on certain intervals. For a survey paper see [PROO]. First, we characterize the nonnegative polynomials on the positive semiaxis [0, oo). The following result is due to Markov and Lukacs about one century ago. Theorem 1 (Markov, Lukacs [Lukl8, Mar48, PS76]). Let q{t) G be a real polynomial of degree n. Let rii = [^J and n2 = L^^J • //'?(*) ^ 0 for all f > 0, then q{t) = qiit^ + ^92(0^ where deg{qi) < n\ and deg{q2) < n2Now we apply this theorem to characterize the transfer function, which is similar to the spectral factorization for trigonometric polynomials. Observe that

l92(jt^)P

\q2,even{j(^) + q2,odd{joj)\'^

921(^2)2-I-w2g22(w2)2

___ Pljw) 2 = —-,—r where w = u P2[w) Here qi^even and qi^odd denotes the even and odd parts of the polynomial gj, and qij,i,j = 1,2 are defined accordingly. Note that pi{w) and P2{w) are nonnegative polynomials on w £ [0, oo). Conversely, by Theorem 1, given any such nonnegative pi{w) and P2{w), it is possible to reconstruct the qij{w), and so qi{ju)) and H{jio). In other words, p\{w) and P2{w) with deg{pi) < deg{p2) satisfy \H{ju)\'^ = Pi{w)/p2{w) where w = u'^ for some transfer function H{juj) if and only if they are nonnegative on [0, oo). The characterization of polynomials nonnegative on some finite interval [a, h] is analogous: Theorem 2 (Markov, Lukacs [Lukl8, Mar48, PS76]). Letq{t) € R[t] be a real polynomial. Suppose q{t) > 0 for all t G [o, b], then one of the following holds. 1. If deg[q) = n = 2m is even, then q{t) = gi(t)^ + (* •" o-){b — 092(0^ where deg{qi) < m and deg{q2) < m — 1. 2. If deg{q) = n = 2TO + 1 is odd, then q{t) = {t - a)qi{t)'^ + (6 - t)q2{tf where deg{q{) < m and deg{q2) < m.

316

Jiawang Nie and James W. Demmel

In our algorithms we will need to compute the polynomials qi{jio) from Pi{w), i.e. we need computationally effective versions of Theorems 1 and 2. These are given in section 4. To make the connection to semidefinite programming, we next characterize the polynomials nonnegative on an interval (either [0, oo) or [a,b]) by using certain convex cones. As introduced in [NesOO], define the vector of monomials v{t) = [1 t i^ • • • t""]"^ and the two convex cones of polynomials Ko,co = {P G K"+^ : P^v{t) > 0 V i > 0} Ka,b = {pe ]R"+i : p'^v{t) > 0 V ^ e [a, b]}. Let Hn,i G 5"+i be the i-th Hankel matrix, i.e., H^\=\^' ' lO,

if /e + / = i + 1, otherwise.

As introduced in [NesOO], define linear operators

by the following 2rn + l

^l{v) = Yl

2n2 + l

'^i^rii,i:

^2(-y) = Yl

i=l

^i+l^n2,i-

i=l

Another two operators A^, and A/^ are defined according to whether n is even or odd. When n = 2m, yl3 : ]R"+i-> S'™+\ Ai-.W+^^S"^ are defined as 2m+l M{v)

=

2m-1

Yl '"i^rn.i, i=l

Ai{v)

=

^ [(a + b)ViJ^i - Vi+2 " i=l

aK]^m-l,i-

When n = 2m + 1, yl3 :M»+i ^5-"^+!,

yl4 : ]R"+i-^ 5""+i

are defined as 2m+l

^3(w) = YZ [^»+l ~ "^il-^n^.i' i=l

2m + l

^4(^) = Yl, [^^» ~ •'^i+ll'f^m.ii=l

Let yli,yl2,yl3,^4 be their adjoint operators respectively, with respect to the inner product < A,B >= trace{A^B) for symmetric matrices of the same size. The following theorem is a compact characterization of cones ii'o.ooi -^o,b and their dual cones.

Shape Optimization of Transfer Functions

317

Theorem 3 (Nesterov [NesOO]). The cones Ko^(X), Ka^b can be chcLrcictevized as follows 1- -f^o.oo f^fid its dual KQ ^ are characterized as follows: i^o.oo = {P G R"+i : p = AUYi) + A*^iY2), Yi e 5!f>+\F2 G

Sl'^'},

^o.oo = {c G lR"+i : yli(c) h 0, A2{c) h 0}; 2. when n = 2m is even, Ka,b = {P G R"+i : p = AliYs) + yi:(F4),1^3 G 5!f'+\y4 G S!^}, Kl,

= {c G ]R"+i : yl3(c) ^ 0,yl4(c) >r 0};

when n — 2m + 1 is odd, K,,b = {P G R"+i : p = AUYs) + A^Y^), Y3 G 5!^+\y4 G 5!^+i}, i r ; ^ = {c G R"+i : .13(c) h 0 , ^ ( c ) ^ 0}; 3. Both Ko^aoiKa,b) and KQ^{K*I^) with non-empty interiors.

are convex, closed, and pointed cones

Now suppose we have L subintervals of [0,00): {[ai,bi]}i=,i- Let K = i^o.oo X Ka,,bi X • ••Ka^,bj^. Then its dual K* = K^^ x K*^f,^ x • • • K*^,,^. Given a matrix A of (L + l)(n + 1) rows and 2(n + 1) columns, consider the following problem: find a vector ( if it exists ) p e K^Cn+i) s.t. Ap G K. This can be done by solving a SDP feasibility problem by Theorem 3, say, using the SDP solver in [Stu99]. However it will introduce 2{L+ 1) symmetric matrices of size [n/2] or [n/2] + 1. In order to use interior-point methods to solve it, the complexity of one iteration will be at least 0(2(L + ^)n^) arithmetic operations. Fortunately, the dual cone K* (^ does not involve two symmetric matrices. A natural barrier function [NN94] for K* is given by L

F{c) = -lndetyli(co) -lndetyl2(co) - ^ ( l n d e t 4 ' ^ ( c i ) + Indetyl^'^(ci)), where 713(714) is the operator A3{A4) corresponding to Kai,bi in Theorem 3. Here the vector c = (CQ, • • • , cz,) G IR""*"-^ x • • • R""'"^ Now solve the following L+i times analytic center problem: min F{c) s.t. A'^c = 0,ceintK*.

(3) (4)

318

Jiawang Nie and James W. Demmel

The barrier function F{c) will tend to infinity as c approaches dK*. Hence the minimum will be attained in the interior of K*, which is not empty as guaranteed by Theorem 3. The optimality condition is that

VF{c)=AX,

ceintK*;

A'^c = 0. The optimal solution c* and its Lagrange multiplier A* can be found very efficiently using Newton's method. For any c G intK*, it can be shown [NN94] that VF(c) -
3 Shape optimization In this section, we will show how to design the transfer function of a LTI SISO system so that it has a desired Bode plot. Four kinds of shapes will be discussed: standard bandpass filter, piecewise constant, piecewise polynomial, and general shapes. 3.1 B a n d p a s s filter design The goal is to design a transfer function \H{ju)\'^ = ^ 4 ^ which is close to one on some squared frequency {w = w^) interval [u)^,u;''] and tiny in a neighborhood just outside this interval. The design rules can be formulated as Pi{w),P2{w) > 0, y w >0 P2{w) Pijw) <S, Piiw)

VwG[wlw'2]U[w{,wl2]

where the interval [wfjtOg] is to the left of \w^,w^], and [101,102] ^^ t° ^^'^ right. Here a and f3 are tiny tolerance parameters (say around .05). Let pi and p2 be the vectors of coefficients of pi{w) and P2{w) respectively. Then the constraints above can we restated as

Shape Optimization ofTransfer Functions

319

Pi,P2 e Xo.oo pi - (1 - a)p2 e K^e^^r

Using Theorem 3, we see that the above cone constraints can be expressed as Ap £ K where • In+i

0

0 In+l In+1 [a - l ) / „ + l A = -In+l (l+/3)/„+l —In+1 SIn+1 _—In+l ^In+1

P

and K ^ KQ^OO X ^O,OO X -f'^^^iu'- x -^-u^'.-iu x-f^w«,^ xii'^.^^r. Given (Q;,/3, 5), solve the analytic center problem (3)-(4) and then recover the coefficients p. As introduced in [GHNOO] for the discrete case, we can also consider the following objectives: • • •

minimize a + /9 for fixed S and n minimize S for fixed a, /3, and n minimize the degree n of pi and p2 for fixed a, /?, and 6.

These optimization problems with objectives are no longer convex, but quasiconvex. This means that we can use bisection to find the solution by solving a sequence of analytic center problems. A design example is given in figure 1. For the simplicity of programming, we used SeDuMi[Stu99] to solve the primal feasibility problem.

Fig. 1. The design filter shape for [w' w''] = [2 3], [w' w'2] = [0 1.8], [wl w^] = [3.2 5], a = /3 = 0.05, 5 = 0.05, n = 10.

320

Jiawang Nie and James W. Demmel

3.2 Piecewise constant shape design Here we extend the shape design technique from the last section to piecewise constant shapes. In other words we want the transfer function to be close to given constant values ci, ...,c„ in a set of m disjoint intervals ui"^ = w G \ak,bk], where ai < foi < a2 < 62 < • • • < Om < ^m- More precisely we want the transfer function to lie in the interval [{l—a)ck, (1 + /3)cfc] for w G [0^,6^]. By picking enough intervals (picking m large enough) we can approximate any continuous function as closely as we like. These constraints may be written Pii'w),P2{w) > 0, y w >0 pi{w) (1 - a)ck < < (1 + I3)ck, y w e [ak,bk], k = !,••• P2{w)

,m.

Using Theorem 3 as before, these constraints can be rewritten as the cone constraints Pliw),P2{w)

pi - (1 -a)ckP2,

e /tTo.oo

(l+/3)cfcP2 - P i e Ka^,bk,k = !,•

, m.

As before, find vector p such that Ap G K where 0 0 'n+1 /n+1 (a - l)ci/„+i (l+/?)ci/„+i -/„+i l)Cm-^n+l

(1 + f3)CmIn+l •X K? {, . Solve the analytic center problem and K ^ 0 , 0 0 ^ -^ai.bi ^ (3)-(4) again and recover the vector p. As in the preceding subsection, various design objectives can be considered by applying bisection. A design example for a step function with 3 steps is given in figure 2. 3.3 Piecewise polynomial shape design Here we extend the techniques of the last section to piecewise polynomials. Thus, on each interval [afc,&fc] we ask that Pi{w)/p2{w) be close to a given polynomial (j>k{w)-, in particular that it lie in an interval [(1 — a)<j)k{w)^ (1 + (5)(j)k{w)]. This leads to the constraints Pi{w),P2{w) > 0, V 10 > 0 {l-a)<j>k{w)k{w), P2\W)

Vu;G[afc,5fc],fc = l,--. ,m.

Shape Optimization of Transfer Functions

321

Fig. 2. The step function shape design for [ai bi] = [0 1.8], [02 62] = [2 3], [as 63] [3.2 5], ci = 1,C2 = 3,C3 = 2, a = /3 = 0.Q5, n = 10.

Using Theorem 3 once again, we transform these to the following cone constraints

Pi - (1 - a)4>k{w)p2{'w) e Kaf,,bk,k = !,••• ,m; (1 + (3)p2{w)(f)k{w) - p i G Ka^,bt,k = !,•••

,m.

which are again a set of linear equations and LMI's. As before, pi and P2 can be obtained by solving some appropriate analytic center problem (3)-(4), and bisection can be used to achieve certain design goals. 3.4 General shape design So far we have considered bandpass filter design, piecewise constant shape design, and piecewise polynomial shape design. Here we discuss general shape design. The goal is to design a transfer function \H{ju>)\'^ = ^^\^l so that it behaves like some general nonnegative function f{w) for w G [a, b] where 0 < a < 6. In other words we want: Piiw),p2{w) >0,yw& [0,00) Pijw) (l-a)/H< <{l+p)f{w), \/w€[a,b]. P2{w)

(5) (6)

Now we can not apply Theorem 3 directly, and must instead apply approximation methods. One obvious approach is to partition [a,b] into subintervals {[afc,&fc]}fcLi and approximate f{w) by a constant or more general polynomial in each subinterval. Then we can apply the method from the preceding sections. Another approach is to apply semi-infinite programming(SIP), as described in [WBV97]. The idea is to choose N sample points

322

Jiawang Nie and James W. Demmel a < Wi < W2 < • • • < wj^ < b

and replace the semi-infinite inequality constraints (5)-(6) by N simple inequality constraints. A standard rule of thumb is to choose A'' = 15n in practice [WBV97]. Then the approximate optimization problem is solved iteratively [Pol97]. 3.5 Related work There are several related papers [AV02, GHNOO, WBV97] on various filter design techniques. Most of them are for discrete systems, and some techniques can be applied to continuous systems with some modifications. Note that the mapping s = j ^ maps the pure imaginary axis onto the unit circle except (—1,0). Then our transfer function H{s) becomes a rational trigonometric function R{z). Each interval j[ai,bi] is mapped onto some arc {e^^ : u G [w/, wf ]} on the unit circle. The methods described in [AV02, GHNOO, WBV97] all can be applied. However, all of them will involve the constraints of the form that some trigonometric polynomial is nonnegative on some interval. The characterization of (trigonometric) polynomials nonnegative on some intervals will eventually need Theorem 3 or its equivalent form to transform to a LMI. In this paper, we transform our design problems using constraints of real polynomial nonnegativity on some intervals in the positive semi-axis. Then we may apply Theorem 3 directly to characterize these constraints using LMIs. As described at the end of Section 2, we solve an appropriate analytic center problem, instead of solving these LMIs directly. The structure of this problem can be exploited to use Newton's method to efficiently find the analytic center. There are also several good papers [Fab02, NesOO, GHY03] on polynomials on the real axis, unit circle, pure imaginary axis, and other curves. [NesOO] is the classical paper that characterizes polynomial nonnegativity constraints by LMIs; our paper is based mostly on it. In [GHY03] the authors characterized the cone of positive pseudopolynomial matrices and discussed optimization over this cone. The authors also discussed the conditioning of such optimizations, and proposed using the basis of Chebyshev polynomials to improve conditioning. [Fab02] gives an abstract version of [NesOO, GHY03], characterizing polynomials which are nonnegative on the disjoint union of several intervals.

4 Recovery of t h e transfer function In this section, we show how to use Theorems 1 and 2 effectively. First, given polynomials pi{w) and P2{w) {w = u"^) such that ^ desired shape, we need to find real polynomials qi and q2 so that Pijw) P2{w)

2

92 (jw)

Shape Optimization of Transfer Functions

323

To this end, given a polynomial p{w) that is nonnegative on [0, oo), we will provide an algorithm to find two polynomials qe{w) and qo{w) such that p{w) = q1{w) + w • Qoiw). Then qe contains the even coefficients and qo the odd coefficients (modulo signs) of the desired polynomial q as described in Section 2. Lemma 1. The following polynomial identities hold: 1. iff + wgJXfl + wgl) = {hh + wgxg2? +w{hg2 - /25i)^' 2. {w - r)2 + b^^{wv 9 2 T F ) 2 + w • 2{Vr^+lP - r). Proof. Verify directly. D Lemma 2. If a polynomial p{'w) is nonnegative on [0, oo), then its factorization must have the form

(

111

\

n2

J

i=l

nC-^ + Ci) i=l

Y[{.{w-rif

"3

+ b^)l[{w

+ ai)

i=l

where a >0, bi > 0, n\ + 2n2 + na = n, 0 < ai < • • • < a„3, Ci < 0. Proof. Write the factorization p(tf) = a YYk=i{'^~'Pk)• First consider the constant term a; it must clearly satisfy a > 0 for piw) to be nonnegative over [0, oo). Next consider the three classes of roots pk- real positive, complex, and real nonpositive. The positive roots must all have even multiplicity for p{w) to be nonnegative, so we can write the product of all their factors w—pk as 0^=1 (^"1" Cj)^ where Cj < 0. Next, the complex roots come in complex conjugate pairs Pk = Tk + j • bk and p^ = rk - j • bk, so we can write {w - Pk)iw - pk) = {w — rkY + b\ for all n2 complex conjugate pairs. Finally consider the na nonpositive real roots 0 > —ai > • • • > ~a„3 of p{w). Their corresponding factors u; — (—Oj) = u; + ttj are all nonnegative on [0, oo). D

Using the above two lemmas, we get the following algorithm. Algorithm 4.1 This algorithm will find qe{w) and qo{w) such that p{w) = q1{w) + « ; • q1{w) if p{w) is nonnegative on [0, oo). Let ge = l>9o = 0. Step 1 Find the factorization of where hi > 0, ni + 2n2 + ns = n, 0 < ai < • • • < a„3,0 > Cj £ K. Step 2 Find the {-f + w{-f form ofXXZ^iw + a^) for fc = 1 : n3 qe = y/alqe

+ W • qo

qo = \fa'iqo - qe qe •=qe,qo -^^qoend

324

Jiawang Nie and James W. Demmel

Step 3 Find the {-f + w{-f form ofY^^l^{w'^ + 6f) for k = 1 : n2 qe = qejw - ^rf + b'i) + w • g „ ^ 2 ( ^ K ' + 6f - r^) 9o = qeyj2{^rt^+yf^)qo{w - ^rf + b'j) Qe ••=qe,qo ••^'qoend Step 4 qe := aqe ]Xi=ii'^ + ^i), 9o := aQo n " = i ( ^ + ^i). Now apply algorithm 4.1 topi{w){i = 1,2) respectively, i.e., Und qij{w){i,j = 1, 2) such that Pi{w) = qfiiw) + w • ql2{'^) fo^ i = 1, 2. Then we obtain the desired transfer function H{s) 92,l(-'S^) + S 9 2 , 2 ( - S ^ ) '

Remark: If a polynomial p{w) is nonnegative on a finite interval [—1,1] (another finite interval can be changed to this one by a linear transformation), then we can also apply the above algorithm to find two polynomials pi and P2 such that p{w) = Pi{w) + (1 — w){w + l)p'2{w). ActuaUy, we only need to do the Goursat transform (see [PROO]) for piw), i.e.,

p(«;) = (^ + l ) " p ( i ^ ) , and then apply the above algorithm to find qe,qo such that

p{w)

=ql{w)+wqliw),

and then apply the inverse Goursat transform to get back p{w): p{w) = 2-"(w + l)'*''ff(P)p(i-^^).

5 Conclusions and discussion This paper discusses shape optimization for a transfer function for a LTI SISO system by formulating it via semidefinite programming. Given the shape (absolute value) of the transfer function, we show how to extract the transfer function itself. Since the optimization process uses semidefinite programming, it may be done efficiently. We do not consider any constraints on the components A, b, c, d of the LTI system. However in practice, these components may not be arbitrary, but instead have special structure and depend on certain design parameters. Thus an interesting question is finding those parameters to optimize the shape of the transfer function as we did in Section 3. This is in general not a convex

Shape Optimization of Transfer Functions

325

problem, and can be very hard to solve. But it is still a feasibility/optimization problem about polynomials, if {A, b, c, d) are polynomials in those parameters. Therefore, we may formulate t h e m using polynomial optimization, and then solve t h e m by techniques such as the sum of squares and positivstellensatz (see [ParOl]). B u t this is more difficult, and future work. A c k n o w l e d g e m e n t s : The authors would like to t h a n k Prof. El Ghaoui and the Referee for many useful comments and suggestions.

References [AV02] B. Alkire and L. Vandenberghe, Convex optimization problems involving finite autocorrelation sequences. Mathematical Programming Series A 93 (2002), 331-359. [CD91] Prank M. Callier, Charles A. Desoer, Linear System Theory, SpringerVerlag, New York, 1991. [Fab02] L. Faybusovich, On Nesterov's approach to semi-infinite programming. Acta Applicandae Mathematicae 74 (2002), 195-215. [GHNOO] Y. Genin, Y. Hachez, Yu. Nesterov, P. Van Dooren, " Convex Optimization over Positive Polynomials and filter design", Proceedings UKA CC Int. Conf. Control 2000, page SS41, 2000. [GHY03] Y. Genin, Y. Hachez, Yu. Nesterov, P. Van Dooren, Optimization problems over positive pseudopolynomial matrices, SIAM Journal on Matrix Analysis and Applications 25 (2003), 57-79. [KS95] T. Kailath and A.H. Sayed, "Displacement Structure: theory and applications", SIAM Rev. 37(1995), 297-386. [LuklS] Lukacs, "Verscharfung der ersten Mittelwersatzes der Integralrechnung fur rationale Polynome", Math. Zeitschrift, 2, 229-305, 1918. [Mar48] A.A. Markov, "Lecture notes on functions with the least deviation from zero", 1906. Reprinted in Markov A.A. Selected Papers (ed. N. Achiezer), GosTechlzdat, 244-291, 1948, Moscow(in Russian). [NN94] Yu. Nesterov and A. Nemirovsky, "interior-point polynomial algorithms in convex programming", SIAM Studies in Applied Mathematics, vol. 13, Society of Industrial and Applied Mathematics(SIAM), Philadelphia, PA, 1994. [NesOO] Yu. Nesterov, "Squared functional systems and optimization problems". High Performance Optimization(H.Frenk et al., eds), Kluwer Academic Publishers, 2000, pp.405-440. [Pol97] E. Polak, "Optimization: Algorithms and Consistent Approximations". Applied Mathematical Sciences, Vol. 124, Springer, New York, 1997. [PS76] G. Polya and G. Szego, Problems and Theorems in Analysis II, SpringerVerlag, New York, 1976 [PROO] V. Powers and B. Reznick, "Polynomials That are Positive on an Interval", Transactions of the American Mathematical Society, vol. 352, No. 10, pp. 4677-4692, 2000. [WBV97] S.-P. Wu, S. Boyd, and L. Vandenberghe, "FIR filter design via spectral factorization and convex optimization". Applied and Computational Control, Signals and Circuits, B. Datta, ed., Birkhauser, 1997, ch.2, pp.51-81.

326

Jiawang Nie and James W. Demmel

[ParOl] P.A. Parrilo. Semidefinite Programming relaxations for semialgebraic problems. Math. Prog., No. 2, Ser. B, 293-320, 96 (2003). [Stu99] J.F. Sturm, "SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones". Optimization Methods and Software, ll&£:12(1999)625-653. [VB96] L. Vandenberghe and S. Boyd, "Semidefinite Programming", SIAM Review, 38(l):49-95, 1996.

Achieving Wide Field of View Using Double-Mirror Catadioptric Sensors Ronald Perline^ and Emek Kose^ ^ Department of Mathematics, Drexel University, Philadelphia, PA 19103, USA. rperlineOmcs.drexel.edu ^ Department of Mathematics, Drexel University, Philadelphia, PA 19103, USA. ek58®drexel.edu

Summary. For many applications such as surveillance, medical imaging, photography and robot navigation, it is required that the camera have a wide field of view. Traditional approaches to solve this problem include using a rotating camera, stitching images, complex lenses or multiple cameras. We are proposing a catadioptric sensor with a camera-mirror pair for enhancing the field of view. Devices consisting of a reflective surface (catoptrics) and a camera (dioptrics) are called catadioptric sensors. Our single viewpoint double-mirror system formed of a conical mirror coupled with a proper secondary mirror arises as a solution to a nonlinear first order ordinary differential equation. The ordinary diS'erential equations are obtained as geometric solutions to the problem. Our system is designed to image a plane at infinity, without distortion, requiring no digital unwarping. In this work, we are analyzing the family of cones and corresponding secondary mirrors to obtain a correct image and a large field of view.

1 Introduction Using mirrors combined with lenses is an effective solution to the general problem of enhancing the field of view, which is necessary for various apphcations, such as surveillance, medical imaging, photography and robot navigation. In robotics, especially in autonomous mobile robots, omnidirectional sensors are the best for navigation, and in dynamic environments, catadioptric sensors work the best. T h e images produced by the sensor can easily be transformed by various software. Presenting such advantages, interest in catadioptric sensors has increased recently. Catadioptric sensors are oriented such t h a t the camera is fixed, with the rotationally symmetric mirror suspended above it. A camera, the dioptric component, realizes a mapping between the 3D world and 2D image plane. We employed perspective and orthographic camera models in our designs. In perspective projection, the world points are projected onto the image plane with perspective rays originating at the center of

328

Ronald Perline and Emek Kose

projection (COP), which would lie in physical camera. The camera reahzing perspective projection is often referred to as pinhole camera. Orthographic projection can be viewed as a special case of perspective projection, it has a large focal distance, in other words, the center of projection is at infinity. In cameras realizing orthographic projection, objects close to the optical axis are imaged, and parallel hnes are mapped to parallel lines. So the difference of orthographic projection from perspective projection is that light rays travel parallel to optical axis, as opposed to passing through the center of projection. Traditionally, imaging systems have been designed to maintain a single effective viewpoint. In other words, all the rays of light entering the camera intersect at a single point, called the effective pinhole, so single effective viewpoint acts as a virtual center of projection. By placing the center of perspective camera lens at the outer focus of the hyperbohc mirror, the inner focus then becomes the single effective viewpoint. Single viewpoint allows the unwarping of accurate perspective images. The single viewpoint has been studied and recommended by Baker and Nayar [BN98]. Our system is designed to image a plane at infinity, compared to the diameter of the system, without distortion, requiring no digital unwarping. The catoptric component of the system is a double-mirror consisting of a primary mirror and its dual, as we call them, and both mirrors employed are rotationally symmetric. This allows us to picture the problem in 2D. The primary mirror has been chosen a concave up cone of different slopes under perspective and orthographic projections. The corresponding secondary mirror, which is also called its "dual" was designed according to the transformation reahzed by the primary mirror. For the special cone whose slope is 1, we can easily obtain the analytical solution to the ordinary differential equation for its dual whereas for cones with slopes different than 1, we have to solve the resulting ordinary differential equation by numerical methods. The scheme we used in designing our catadioptric sensors, as suggested by Hicks and Perhne [HPOl] consists of: 1) Differential formulation: Primary mirror is fixed, the desired design characteristics of the dual mirror are restated in terms of a set of ordinary differential equations. 2) RealizabiUty: A systematic check is performed to determine whether it is theoretically possible to construct a catoptric which exactly realizes the specified design characteristics. 3) Optimization: If the mirrors are not realizable, then we perform optimization methods, to find the 'best' possible catoptric component.

2 Related Work Here, we review past work on catadioptric sensors, with focus on design rather than applications. There has been an increasing interest in investigating the design and appfications of catadioptric sensors. Much of this work focuses on

Achieving Wide Field of View

329

designing panoramic or wide field of view sensors (see [BN98, Hic99, Nay97, Ree70, YK90, CM99]). An early use of employing mirrors for panoramic imaging is a patent by Rees [Ree70], who proposes the use of a hyperbolic mirror for television viewing. In [HPOl], Hicks and Perline describe a general method for catadioptric sensor design for a prescribed projections, using geometric distributions in 3-dimensional space, generalizing vector fields. Nayar presents a new single viewpoint, double-mirror, double-camera catadioptric design which is truly an omnidirectional sensor. His design consists of two parabolic mirrors, paired with two orthographic camera [Nay97]. In 1996, Nalwa [Nal96] proposed a panoramic sensor that consists of 4 planar mirrors coupled with 4 imaging systems. The system possesses single viewpoint and it has a large field of view, approximately 360° x 50°. In [HBOO], Hicks and Bajcsy obtain a mirror shape that is wide-angle but approximates perspective views. This design, which is not single effective viewpoint, distorts the view plane minimally. The polar sensor derived by Hicks, Perline and CoUetta [HPCOl] is very close to our Right Angle Mirror design, as we call it. They suggest a sensor design under cylindrical projection, such that the sensor images the world linearly.

3 Conical Mirrors 3.1 Right Angle Mirror In any of our designs, we start with a choice of primary mirror and the projection. Depicting the problem for the chosen parameters, we derive ordinary differential equations that describe the secondary mirror. Solving the ordinary differential equations obtained, either analytically, which is only valid for Right Angle Mirror under Orthographic Projection, or numerically, gives us the graph of the cross section of the secondary mirror. 3.1.1 Right Angle Mirror U n d e r O r t h o g r a p h i c P r o j e c t i o n This is the most basic design in the family of cones. The conical primary mirror we chose has slope equal to 1 and our camera realizes orthograpliic projection. The profile curve of the system can be seen in Figure 1. Assumptions made are: the view plane is infinitely far from the camera, compared to the diameter of the system and is in +y direction. Our camera is located on —y axis, which at the same time is the optical axis of the entire system. So, to set the problem from an inverse direction, all light rays emanating from the image plane travel parallel to each other to shoot the primary mirror,reflecting off parallel to a-axis, they intersect the dual mirror and they reflect off to shoot a point in the view plane, which is y = K. a is a. scaling

330

Ronald Perline and Emek Kose

-0.

Incoming L i g h t Ray

Fig. 1. Cross-section of the Right Angle Mirror under Orthographic Projection. constant that lets us adjust the amount of scaling of the view plane onto the image plane. The primary mirror, which is a cone has equation y=x. Let us define the vectors u and v; u = < - l , a ; i > ; v ==< Kaxi,K

> - < x',f{x')

>

(1)

Normalizing u and v, we get: u=<

-1 l+xi'l

Xi

X

,

Xi

+ xj

Since the view plane y = K is infinitely far, the limit X

lim

O/i

< axi - -r;,l —77 > = < axi, 1 >

K—>oo

K

(2)

K

The vector sum nd = u + v is parallel to the normal vector of the secondary mirror. By making use of the relation between the normal to the surface and the tangent to the surface, we can say that slope of the tangent line to the surface at the point x = x' equals the negative reciprocal of the slope of the normal to the surface at the same point.

That is, - S i = fix') We then have: f'{x)

= 2x+\/4x2 + l

(3)

Integrating both sides, we obtain the analytical solution to our problem:

Achieving Wide Field of View 1

331

1

fix) = x^ + -xV4x2 + 1 + - sinh-^(2x)

(4)

The graph of /(x) is the cross section for our secondary mirror. In Figure 1 you can see how this system images the checker board, far above. We use checkerboards for our simulations, because we are interested in how the system distorts lines. 3.1.2 Right Angle Mirror Under Perspective Projection Our primary mirror is the same cone with slope equal to 1 and we are considering a pinhole camera. The assumptions made are; view plane is at an infinite distance, the optical axis is the y-axis and the center of projection of the camera is at the point ( 0 , - 1 ) . All hght rays that pass through the pinhole are going to reflect off the conic mirror and then hit the secondary mirror, yet unknown. Reflecting off the secondary mirror, they shoot the plane y = Kxi is the intersection point of the incoming ray with the x-axis. We approach this problem with the vector-method discussed above. Let us define the vectors, u , v and w; u = < xi, 1 >, V = < —1, —xi >, w = < axi, 1 > . Let us also define m^ = slope of the incoming light ray, mr= slope of the reflected light ray, di= the angle that incoming light ray makes with positive X-axis, 9r=the angle that reflected light ray makes with positive x-axis and 9m= angle between positive x-axis and the mirror. It is obvious from the Figure 2 that rrii = ^ . Prom optics, we know that 0j. + Oi = 26^ , since we have 0m = f i this imphes; 61^ = I - 6li =» tan(6l^) = tan(f - Oi) =^ rtir = tan(f - di) = cot(6ii). Hence, this gives us ; m^ = xi and m, = —. The point of intersection of the conic mirror with the incoming hght ray can be found simply by simultaneously solving both equations. This point of intersection is: ( j ^ ^ , i r ^ ) Any point (x2,y2) on the secondary mirror can be written as: {X2,y2) =

{--,——,z——)+tv. 1 - xi 1 - xi This implies xi = —^TT . Also we know that irTr + jrhi is in the normal ^

^

X2 + ^

||V||

||W||

direction of the secondary mirror. After necessary substitutions and simplifications we obtain a nonlinear implicit ordinary differential equation which is not possible to solve analytically. However, numerical solution gives us the cross section of the dual mirror, seen in the Figure 2. 3.2 30-Degree Conical Primary Mirror One of most efficient systems we obtained is the 30-degree conic primary mirror system. Efficiency, in the sense that, the system can view a large portion

332

Ronald Perline and Emek Kose

Fig. 2. Cross-section of Riglit Angle Mirror under Perspective Projection. of the scene without having a large diameter. Keeping the diameter of the entire system small is important. We could sacrifice having a longer system for a narrow one. The reason for this is the future applications may include medical imaging. The primary mirror is a cone with slope equal to t a n ( | ) . The camera is orthographic.

0.5

Fig. 3. Cross-section of a 30-degree Primary Mirror under Orthographic Projection.

Achieving Wide Field of View

333

Fig. 4. POV-Ray simulation of the view from 30-degree system in the test room.

We are going t o employ the vector method t o derive the secondary mirror. The primary mirror has equation: y = tan(;)x. The vectors v and w after normalization are then;

So the vector addition of v and w is parallel to normal vector of the dual mirror at (x, f (x)). If n d = v w then the slope of n d equals the negative reciprocal of the slope of the tangent line of the secondary mirror at the same point. That is:

+

ndl nd2

--

=

f I(.)

Expressing tan(!) geometrically, using the figure helps us engage f (x) in the slope equation. That is

Solution of equation (8) for

gives us this relation;

334

Ronald Perline and Emek Kose

By equations (7) and (9) we derive the nonlinear, implicit ordinary differential equation for secondary mirror corresponding to 30-degree conic primary mirror. The cross section of this mirror can be seen in Figure 3. The simulation of this system after it has been revolved about the y-axis is as in Figure 5. The 30-degree conical mirror system inside the test room with checkerboard walls views close t o 180°, Figure 4.

Fig. 5. Sideview of a 30-degree Primary Mirror under Orthographic Projection.

4 Conclusion In this paper, we have exhibited a catadioptric sensor design which enables a normal camera an ultra wide field of view with minimal distortion. These sensors are based on a family of double-mirrors with conical primary mirrors, derived as numerical or analytical solutions of non-linear ordinary differential equations which describe how a plane perpendicular to the optical axis of the system is imaged on the film. The images obtained require no further processing. In the future work, we would like t o investigate the general conics as primary mirrors, such as parabola, ellipse and hyperbola. We also want to implement an actual system and experiment on it, perform full error analysis.

References [BN98] Baker, S, and Nayar, S.K. (1998), "A Theory of Catadioptric Image Formation," Proc. International Conference on Computer Vzsion, pp. 35-42.

Achieving Wide Field of View

335

[CM99J Conroy, T. and Moore, J. (1999), "Resolution Invariant Surfaces for Panoramic Vision Systems," Proc. International Conference on Computer Vision, 392-397. [Hic99] Hicks, R.A. (1999), "Reflective Surfaces as Catadioptric sensors ," Proceedings of the 2nd Workshop on Perception ofr Mobile Agents, CVPR 99 , 82-86. [HBOO] Hicks, R.A. and Bajcsy, R. (2000), "Catadioptric Sensors that Approximate Wide-Angle Perspective Projections," Proc. Computer Vision and Pattern Recognition, 545-551. [HPOl] Hicks, R.A. and Perline, R. (2001), "Geometric Distributions for Catadioptric Sensor Design," Proc. Computer Vision and Pattern Recognition 2001. [HPCOl] Hicks, R.A., Perline, R. and Coletta, M.L. (2001), Catadioptric Sensors For Panoramic Viewing, World Scientific. [Nal96] Nalwa, V. (1996), "A True Omnidirectional Viewer," Technical Report, Bell Laboratories, Homdel, NJ 07733, USA. [Nay97] Nayar, S.K. (1997), "Catadioptric Omnidirectional Camera," Proc. Computer Vision and Pattern Recognition, 482-488. [Ree70] Rees, D. (1970), "Panoramic Television Viewing System," United States Patent, (3,505,465) [YK90] Yagi, Y. and Kawato, S. (1990), "Panoramic Scene Analysis with Conic Projection ," Proceedings of International Conference on Robots and Systems , 82-86.

Darcy Flow, Multigrid, and Upscaling James M. R a t h Institute for Computational Engineering and Sciences, University of Texas at Austin, TX 78712, USA. organismQices.utexas.edu

Summary. Simulation of flow through a heterogeneous porous medium with finescale features can be computationally expensive if the flow is fully resolved. Multigrid algorithms are often used to compute approximations to such problems because of storage considerations, the high condition number of the problem, and multigrid's resolution independent convergence rate. The proposed algorithm takes an ordinary multigrid method and replaces the coarsening procedure with an upscaling one. Upscaling aims for performing computations on a coarser scale but still retaining information about the fine-scale flow and problem data. That is, an upscaled computation is just as cheap as a coarse one, but much more accurate. By replacing coarsening with upscaling in multigrid, faster convergence ensues. Numerous upscaling techniques have been proposed; variational multiscale subgrid upscaling is used here. By keeping within the usual finite element variational framework, known multigrid analysis techniques can be applied. Like multigrid, the proposed scheme has a convergence factor dependent on both the permeability field (the ratio of maximum to minimum values) and the relative resolution of the fine and coarse grids.

1 Introduction Darcy's law describes fluid flow through a porous medium. It is an empirical law t h a t asserts bulk flow of a fluid through the medium is proportional to the gradient of the pressure across the medium (accounting for hydrostatic differences from gravity) [Dar56, Bea72, Sch74, FC79]: u = — (Vp-pg) M Darcy's law has found wide applicability in modeling subsurface flows, and has been generalized to model multicomponent and multiphase flows. (The above differential form is itself a generalization of the relation Darcy formulated.) Our primary interest is in using Darcy's law to model oil reservoir and groundwater contaminant flow.

338

James M. Rath

Darcy's law alone is insufficient to describe the physics: conservation of mass (the continuity equation) and equations of state (relating density, viscosity, and permeability to phase fraction and temperature) are necessary. In the appUcations being considered, there is often a need for the velocity to be very accurate and to strictly (locally) observe mass conservation; hence our focus on mixed methods [RW83, DEW83, DEW84, ERW84]. Although our presentation ignores aspects of multiphase flow, the proposed ideas and software can readily be adapted to model such flows. 1.1 Heterogeneity and why it is a problem Geostatistical modeling is used to generate the necessary data (porosity and permeability) to specify the problem to be approximated [DJ97]. This data is typically given at a very fine resolution [Dur02], but the goal is to predict long-range flow behavior; it is tempting, then, to approximate the problem at a very coarse scale. However, fine-scale features of the problem data can have very large effects on the coarse-scale flow behavior [Dur02]. Therein lies one big difficulty: it is necessary to resolve the flow at very fine scales resulting in computationally poor conditioned problems to solve [G+85]. Moreover, the resolution cannot be reduced to shrink the size of the system: (1) heterogeneity in the permeabihty (irregular, short spatial-scale jumps) means p-refinements (high-order approximations) will not help, and (2) spatially-limited resolution and spatially-uniform heterogeneity means /i-refinements (coarse scaling) will not help either. The fine-scale resolution necessary in simulations makes for poor conditioning, yet there is still another difficulty: the jumps in the permeability can sometimes be quite severe (several orders of magnitude changes) between nearby locations (see for example Figure 4). This makes our computation of an approximation even more poorly conditioned. The more heterogeneous and fine-scale the problem, the more computationally expensive it becomes as all known direct/iterative linear solvers have behavior which worsens with greater condition number [GV96]. We seek to broaden the range of problems that are computationally feasible. 1.2 Proposed solution technique One approach used to overcome the poor-conditioning in modeling multiscale phenomena is upscaling [Dur02]. In upscaling techniques, an averaging process is used to determine the influence of fine-scales on the simulation and to adjust the coarse-scale computations accordingly. Typically an upscaled model will more accurately predict the coarse flow, and will also correctly display flne-scale features of the flow. With mixed variational multiscale subgrid upscahng [Arb, HFMQ98], the scales are spht into subgrid and coarse parts before approximating, thereby keeping all the fine-scale information in the model. The mixed framework keeps strict conservation of mass.

Darcy Flow, Multigrid, and Upscaling

339

Any upscaling technique must ignore some fine-scale information, though. As noted above, fine-scale features of the problem data can have very large effects on the coarse-scale flow behavior — sometimes a full resolution of the problem is inescapable to correctly predict flow features like production rates or breakthrough times. Not all the benefit of upscaling need be lost: we can use upscaling to aid the computation of the fine-scale flow. Since one goal of upscahng is to obtain a better coarse-scale approximation of the flow, it is a natural idea to try substituting the upscaled approximation for an ordinary coarse-scale approximation in a multigrid scheme. The upscaHng substitution is also natural in that it is about as computationally expensive as the ordinary coarsening. This substitution forms the basis for the proposed algorithm. Although we cannot hope to reduce the computational complexity of multigrid, we can dramatically reduce the time necessary to accurately model a fine-scale flow. As repeated single-phase flow simulations are at the heart of multi-phase simulations, geostatistical sampling, and flow optimization studies, this is a significant achievement.

2 Mixed Variational Multiscale Subgrid Upscaling Because of the complexity of the proposed algorithm, it is useful to first describe the underlying upscaling method and introduce some terminology and notation. Further details of this technique can be found in [Arb, Arb02, AB02]. Use /? to denote the spatial extent of the porous medium and assume that it is a connected, convex, polygonal domain in E" (where n is 1, 2, or 3). Let p and u denote the unknown pressure and macroscopic velocity. Our model problem in the interior of the domain consists of Darcy's Law u = --(Vp-pg), M

(1)

V • u = /,

(2)

and the continuity equation where K, /i, p, and g are the (spatially-varying) permeability, viscosity, density, and gravity, and / is a source term. On one piece of the boundary rV, specify the flow u-v = gN, (3) and on another, r b , specify the pressure P = 9D-

(4)

The pieces of the boundary add up to the whole dQ = PIM U PD , the pieces are disjoint F^ C] To = 0, and v is the unit outer normal vector to dSl. A cell-centered finite difference method (CCFD) is used to approximate fiow equations for the pressure.

340

James M. Rath

The proposed methods, however, apply to the problem (l)-(4) in mixed variational form. Let V = {v G iJ(div, /?) I V • 1/ = 0 on FN} , W = L'^{Q), and Vg„ be an element of if (div, Q) such that Vg^ • v = g^ on Fff and V(,„ • v = 0 on FD. (An important special case occurs if F^ = dQ. In that case, use W = L^/K instead; a compatibility condition on / and g^ is also required.) The problem is then to find u G V and p &W such that (V • n,w) = {b,w) (ku, v) - (p, V • v) = (c, v) -(£(£,,. VV' i •^ )iy)ro rD

Ww G W, Vv G V.

(5) (6)

The substitutions k = f^K.~^, b = f — V rg^, and c == pg — kvg„ have been made to simphfy the notation. Note that the velocity u = u + Vg„ is the solution to the original problem (l)-(4). In order for either formulation of the problem to have a solution, the data needs to meet some regularity constraints. In the variational formulation, one of interest in this proposal is that k must be a symmetric rank-two tensor in (L°°(i7))"^" (denote its essential supremum by k*), and must be uniformly elliptic (denote its essential infimum by fc*). The ratio of these two values fc*/fc* will have an impact on the performance of the proposed preconditioners. 2.1 A mixed finite element approximation For a fine-scale problem, assume the permeability data k is given as piecewise constant on a grid with spacing h, and that measurement uncertainties prevent specifying it at any finer resolution. It then makes sense to find solutions at the same resolution, and no finer. That is, find u/j G Vh C V and ph G Wh C W such that (V • Uh, Wh) = {b, Wh) (ku,i,v^)-(p/j,V-v/i) = (c,v/i)-(to,v/,-J/)rD

Wwh G Wh, Vv/j G V^,

(7) (8)

where Wh^Vh are Raviart-Thomas-Nedelec elements of order zero (RTO) [RT77, BF91, RT91] on the given grid. See Figure 1 for an illustration of the degrees of freedom of these elements. Using quadrature — the trapezoidal rule — to compute the term (ku/j, Yh) is equivalent to solving the original problem (5)-(6) using cell-centered finite differences [RW83, AWY97]. If a coarse-scale approximation is desired instead, then find u/f G VH C V and PH G WH C W such that {S7 •UH,WH)

= {b,WH)

(ku/f,v/f) - (p//,V-Vi^) = (c,Vfl') - (firo,v/f • j^jro

WwHeWn, Vv//G Vf/.

(9) (10)

Darcy Flow, Multigrid, and Upscaling —**—

^ »

: •

,

'

.i

.

•

:

•»

:

: * '

•

;

#:

•

•

•ii

•:

'

*

ill

;;

'!!!(•

,

r

<$ :

•

a- , ;

* ,

;

•

9

:

• ::

:•

:; »

:^

• ,:

: «:

: •"

: » .

:'i

•

• ;:

• : • ; : « ^; . :

•It

:

1!

.

:

.ill

:

:

•

II

;

;

# ::

• )

:

» : : • ;:

•

<•

::

f

;:

—fi—

III

:

,

.; • : :

s -

»

;

5

:

:

•

;

.

* ;

:

« ,

; . :

^

*

• :: .:

• :

•

. :
• : • ;

'::

• :: • ; : • ::

• : • :.

: • : : • ;: • ;: • :

m :: :

:

:

.

•

:;
.•

: •^ -

)

;

:,

« :: • ^ ,

m ^ ::

: • :

'

••

:

'-

^ . : ffl

;

$i

;

; « :

:

"

:

:

: ^

341

„: —

« :; »

•

• ,'

e .:

• ;:

« :• ,. ?

«

ti'

:i

®

:

11

;

:'. @ ;

• -

—

I

—

•

Fig. 1. Degrees of freedom for 2-D RTO elements on a fine rectangular grid: normal velocity degrees of freedom, and • pressure degrees of freedom.

•

•

*

'!•

•

a

•

•

^1^

Fig. 2. Degrees of freedom for 2-D BDMl [BDM85] elements on a coarse rectangular grid: :'. (linear) normal velocity degrees of freedom, and « pressure degrees of freedom.

342

James M. Rath

where WH X V^ are RTO elements or, perhaps, Brezzi-Douglas-Duran-Fortin order one (BDDFl) [BDDF87] elements) on a grid with spacing H, an integer multiple of h. See Figure 2 for an illustration. For the RTO elements, some error estimates [BF91, DR85, RT91] for a grid size h are ||u-u„||o
^0{h),

(11)

\\P - Phh < C\\p\\2h

=0{h).

(12)

For BDMl/BDDFl elements the velocity is an order of h more accurate ||u-u,,||o
=0{h').

(13)

2.2 The upscaling technique The subgrid upscaling technique uses the mixed finite element method (MFEM) with a variational multiscale [HFMQ98] technique. Before any approximation, decompose V and W (using a chosen coarse grid) so that (i) no information is lost, V = Vc ® SV, W = Wc® 5W; (ii) mass is conserved, V • Vg = Wc, V • 5Y = SW; and (iii) the fine-scale velocities are locally supported over the coarse grid, 6Y • i^ = 0 on the boundary of each coarse cell. (Section 3.1 — in particular Theorem 3.3 — of [Arb] show that this decomposition is always possible and satisfies some additional necessary properties. There is also more than one way to choose the decomposition.) Note that Condition (iii) above allows us to disconnect the subgrid problems from one another, but not from the coarse grid problem. A Green's or influence function approach is employed to effect a forward elimination of the subgrid problems into the coarse problem, and — once the coarse problem has been solved — a back substitution to recover the subgrid solutions. The two can then be recombined to obtain a fine scale solution. Since V and W have been decomposed into direct sums, the problem (5)~ (6) can be described as follows. Find Ug e Vc and pc G Wc, and Su G SV and Sp € 6W such that on the coarse scale {V •{uc + 5u),Wc)^{b,Wc)

Vu;c G Wc, (14)

(k(uc4-(5u),Vc) - (Pc + (5p, V • Vc) = (c,Vc) - igD,yc-'^)rD

Vvc e Vc, (15)

and on the subgrid scale {V-iuc + 5u),Sw) = {b,Sw) (k(uc-F(5u),(5v)-(pc + <5p,V-5v) = (c,(5v)

ySw&SW, VSveSY.

(16) (17)

Darcy Flow, Multigrid, and Upscaling

343

Then u = Uc + (5u and p = Pc + Sp solve the original variational problem (5)(6). To define the (5-subgrid operator and perform the forward elimination of the coarse components, rewrite the subgrid scale equations with coarse components on the right-hand-side. That is, consider the coarse components as sums of basis elements Pc

yjajW*

and

u^ = 2_.Pj'^i-

(18)

The parts of the 5-operator are obtained by substituting these expressions in (16) and (17) and solving for the a^- and /3j-influence function coefficients. The constant part of the 5 operator is given by solving (V • 5u, 5w) = {b, 5w)

\/Sw € 5W,

(19)

(k^u, Sv) - {5p, V • (5v) = (c, 5v)

V(5v e 5Y.

(20)

\/6w€ 5W, VJv G 5\.

(21) (22)

The V1/c-linear part of the S operator is given by solving (V • (5ui, Sw) = 0 {kSui, 6v) - {Spi, V • (5v) = {wi, V • (5v) The Vc-linear part of the 6 operator is given by solving (V • 5uj,5w) = -{V-vlSw) (k(5u^-,<5v)-(<5pj,V-(5v) = - ( k v ^ c H

\/6w e 5W,

(23)

ydv G 5Y.

(24)

Then 5p = Y^ UiSpi + ^ = 5p{p,)

pjSpj + Sp

+5p{n,)

+5p

(25)

and (5u = V ] ctiSui + 2_] P-jSvLj + 5vL = 5VL{PC)

+(5u(uc)

+5u

(26)

are implicit expressions for 5p and 6u in terms of Pc and Uc. Note that 5p{w].) = 5pi, 5p{vl) = Spj, 6u{wl) = Sui, and 5u(v^) = 6uj. Return to the coarse scale equations (14)-(15). Rewrite them substituting in the influence-function expressions for 5p and Su, and gathering the coarse coefficients ccj and /3j out front. Equations for the coarse components' coefficients result

344

James M. Rath ^ a i ( V • 5ui,wc) + X!'^i(V • (vi + Suj),We) = (6 - V • Su,Wc) i

i

Vu;eGVFc, (27) ^ a i ( k < 5 a i , V,) + ^ / 3 , ( k ( v ^ , + 5u,),Ve) i

3

- Y^ ai{wi + 5pi, V • Ve) - ^ i

pj{5pj, V • Ve)

J

= (c-k<5u,v,) + (<5p,V-Ve)

VVCGVC,

(28)

or using the operator notation (V • (ue + 5u{pc) + <5u(ue)), We) = {b-V( k ( U c + Su{Pc)

5u, Wc)

'iw, G Wc,

(29)

+ (5u(Uc)),Vc)

-(Pc + Sp{pc) + Sp{uc), V • Vc) = (c - k(5u, Vc) + {5p, V • v^) -igD,^c-'^)ro

VVeGVe,

(30)

Solve these equations for aj and /3j (that is, Uc and Pc), and write the solution as p = Pc

+Sp

= ^ atwl + Y2 (^i^Pi + Yl ^^^Pi + ^^ i

i

("^^)

j

and u = Uc

+ (5u

= E ^i^c + E "'"^^^ + E ^i'^"i + "^^^^ j

»

(32)

i

Lastly, (29)-(30) can be written in symmetric form by rearranging terms and using some identities not identified here. See [Arb] for details, as well as proofs that each of the above problems are well-posed. Also note that the i5-subgrid operator has a natural definition. No information has been lost in the reformulation of the problem, and no ad-hoc assumptions have been reintroduced to simplify the action of the 5 operator. 2.3 Approximating the upscaled problem Only after deciding on the decomposition is the approximation then made. We are free to pick discretizations for each subgrid independent of each of the others. The discretization of the coarse spaces is only constrained by the already chosen coarse grid. Standard mixed finite element spaces such as Raviart-Thomas or BDDF elements will do.

Darcy Flow, Multigrid, and Upscaling

345

The subgrid upscaling technique has a number of advantages. Each subgrid problem is independent from the others because of the Neumann boundary conditions along coarse cell edges; the subgrid problems can be solved in paraUel with no communication of boundary data needed across coarse cell edges. Each subgrid problem also has many fewer degrees of freedom than the whole problem (by about a factor of the number of coarse grid cells); these problems are far better conditioned, and may be amenable to a direct solver. Each subgrid problem has the same left-hand side and many different righthand sides creating an opportunity for computational efficiency. The coarse problem has many fewer degrees of freedom than the whole problem (by about a factor of the number of fine cells chosen per coarse ceU) and so is also much better conditioned. If the coarse grid spacing is a small multiple of the fine grid spacing, the work in solving the upscaled problem is nearly all done in solving the coarse problem. Lastly, there is only one coarse grid (not many as in multigrid). Further, the number of degrees of freedom in the upscaling space is nearly as many as that in the fine space (compare Figure 1 with Figure 3 below). However, for small upscaling factors the upscaled solution is only about as computationally expensive as a coarse solution. This would be only mildly interesting if the upscaling solution was only as accurate as a coarse solution, but in fact it captures much more (see example below in Subsection 2.5; also see Figures 10-17 in Section 3 on the multigrid-hke preconditioner). There are some algorithm implementation issues to be considered. First, the elements of SWk may not be easy to compute with. For instance, suppose RTO elements are used for Wh and WH, and SWh = {WH) with the complement taken in M^/,. Then elements of SWh do not have local support on fine cells — they must have zero average on coarse cells. A computational trick explained in [Arb02] and [AB02] avoids this comphcation. Second, there is obvious parallelism in the subgrid problems since each one is independent of all the others, and the coarse-scale influence functions allow disconnecting the subgrid from the coarse scale. However, for coarsescale problem there is not any obvious parallelism. To overcome this lack of paraUelism, one could use a domain decomposition method [GW88] or another technique. Lastly, all of the problems being considered are saddle point problems because of the mixed formulation. For small upscaling factors, the subgrid problems may be solved directly. In other cases and for the coarse problem, iterative solvers may be necessary. The Uzawa algorithm [BF91, BraOl] may be used to solve the indefinite system. Also, the hybrid mixed method [AB85, BF91] or a Schur complement could be used to transform the indefinite problem into a positive definite one.

James M. Rath

346

2.4 S a m p l e a p p r o x i m a t i o n of t h e u p s c a l e d p r o b l e m As an example, use RTO elements to approximate 5W and (5V, denoted SWh and 5Yh, and B D D F l elements for Wc and Vc, denoted WH and V ^ . Let WH,h = WH (B SWh and VHJI = ^H © SVh- Elements of WH are functions which are piecewise constant on coarse cells. Elements of SWh are functions which are piecewise constant on fine cells, but must have a zero average over each coarse cell. Elements of V/f are vector-valued functions where the components normal to coarse cell faces are continuous across those faces, and vary linearly along those faces. Lastly, elements of SVh, are vector-valued functions where the components normal to fine cell faces are continuous across those faces, and are constant along those faces; the normal components across coarse cell faces are zero. See Figure 3 for an illustration.

-»•

-«-

-»

T8^

S)

(5

U

r

J

(

a

ii

r

() if::

;: • (J

•

-ii-

(

o

(

o

)

»

o d)

z •

X

•

i'

()
Fig. 3. Degrees of freedom for 2-D RTO/BDDFl elements on an "upscaled" rectangular grid: ® coarse velocity (linear) degrees of freedom, x subgrid velocity degrees of freedom, and » pressure degrees of freedom. [Arb02]

No closure assumption is made regarding the permeability. Accuracy is simply a question of approximation theory — how well does VHJI approxim a t e V ? — and not a question of how much the physics is changed. Defining effective coarse permeabilities is not necessary. T h e closure assumption can be said to be t h a t ah flux across a coarse element face is due to the coarse-scale functions. (The 5-subgrid operator is approximated using well-worn techniques; it is not altered in an ad-hoc fashion to facilitate computation.) In this example, to compensate for the coarseness

Darcy Flow, Multigrid, and Upscaling

347

of the restriction of velocities along coarse edges, higher order accurate basis functions have been employed. For the BDDFl/RTO elements, some error estimates are [Arb] ||u - UHAO

< C{\\phh^

+ (||u||2 + ||u • 42,ro)H^}

= 0[H^),

(33)

lb - PH,hh < C{\\p\\ih + 1|V • u||i/i2 + (||u||2 + ||u • ^ i | 2 , r j ^ ' } = 0{h + H') (34) However, the simplicity of the above expressions belies some error analysis complications; for instance, the 'irH,h operator for the upscaled elements is obtained not just from gluing together the operators from the coarse elements TTH and each of the subgrid elements STTh,,Ec: but also solving an auxiliary problem (except under additional assumptions). The error estimates above also do not capture all the subtleties of the subgrid upscaled solution (see, for instance, the graphs of the eigenstructure of the subgrid upscahng preconditioner in Section 3). One thing to note, though, is that if H/h is small, then 0{h + H^) = 0{h) for the pressure and 0{H'^) = OHH/hfK^) for the velocity. A comparison between the upscaling error estimates with those for BDMl/BDDFl elements on fine grid, (12) and (13), shows that they are the

2.5 Some practical results To show the ability of the subgrid upscaling to approximate problems almost as well as a full fine-scale solution (and far better than a coarse-scale one), some results from a simulated quarter five-spot oil reservoir water flood are presented. The above ideas were adapted to two-phase immiscible displacements, and run on the following example [ArbOO]. The domain is a 40 m by 40 m square with a uniform, rectangular, 40 x 40 grid. The base-10 logarithm of the permeability field is shown in Figure 4; the porosity is 25%. The field initially has a 20% saturation of water. There is an injection well in the lower-left corner, and a production well in the top-right corner. Each has a rate of 0.2m'^/day, and water is being injected. Figures 5, 6, and 7 speak for themselves. The upscaled and fine-scale solutions are certainly close in the eyeball norm; the coarsened solution is not close to either and fails qualitatively to capture some behavior. The upscaled solution requires only about as much time to compute as the coarse solution, but has nearly as many degrees of freedom as the fine solution; hence the "success." Throughout most of the rest of the paper, only RTO/RTO elements will be used. Although the ideas are applicable to others, the description, analysis, and implementation of higher order combinations gets much more complicated.

348

James M. Rath

Fig. 4. Logarithm of the permeability field (geostatistically generated).

Fig. 5. Water saturation contours at 100 and 500 days for the fine 40 x 40 solution.

2.6 Other upscaling techniques

Upscaling techniques are numerous; surveys of various ones currently under study can be found in [Dur02, Far02, RdM97, WGH961. A few descriptions follow. Chen et al. [CDGWOS] describe their technique as a local/global one: use a global coarse simulation t o determine appropriate boundary conditions for local simulations that then determine an effective coarse permeability. Other techniques are local; for instance, periodic boundary conditions can be imposed on the subgrids when computing effective permeabilities as in [BLP78, SP801. Oversampling is another possibility where a local prob-

Darcy Flow, Multigrid, and Upscaling

349

Fig. 6. Water saturation contours at 100 and 500 days for the upscaled 5 x 5 solution.

Fig. 7. Water saturation contours at 100 and 500 days for the coarse 5 x 5 solution.

lem (with various boundary conditions) that is slightly larger than a subgrid is used to determine the effective permeability for that subgrid; Chen and Hou [CH02] and Hou and Wu [HW97] use a multiscale finite element technique to implement this idea. Wu, Efendiev, and Hou also use this idea [EHWOO], and in [WEH02] apply a flow-based gridding dynamic approach to limit the necessary upscaling. Holden and Nielsen [HNOO] have developed a global upscaling method that uses a least-squares problem to determine effective coarse permeabilities. Some distinguishing features of the upscaling technique used here are the use of mixed methods (and the variational setting), the lack of ad-hoc assumptions about the 6-subgrid operator, and the lack of computed effective coarse permeabilities.

350

James M. Rath

3 A Two-level Multigrid-like Scheme In traditional multigrid, one pairs smoothings on fine and coarse grids together with the intent of each smoother reducing the error in different parts of the spectrum. The fine-grid smoother reduces the error on short scales, and the coarse-grid smoother reduces it on larger scales [BHMOO]. The action of each smoother can be computed using a number of operations and storage units proportional to the number of unknowns. The tolerance for the stopping criterion (accuracy) can be chosen independently of the number of unknowns (but not the ratio of largest to smallest permeability). For problems with a great number of unknowns, multigrid is therefore preferable to direct methods. In our modified multigrid, upscaling is substituted for coarsening. Upscaling produces a better approximation than coarsening (at both fine and coarse scales), and is just as cheap. By substituting upscaling for coarsening we keep all of multigrid's benefits, and an even faster scheme results. To diagram the multigrid algorithm and our variant, let us introduce some notation for the matrices involved. If we cast equations (5)-(6) in operator form, we get " 0 B'^' P 'b (35) -B D _ u = c Forming the Schur complement of the velocities gives Ap={B'^D-^B)p = f = b-B'^D--^c.

(36)

For the discrete versions of these equations, add a subscript h for the fine scale, H for the coarse scale, and H, h for the upscaled. A schematic of a twolevel V-cycle multigrid is shown in Figure 8. Our modified version is shown in Figure 9 — the only change is the substitution of upscaled for coarse scale. Multigrid works so well partly because smoothing the solution iterates on different levels reduces the error in different parts of the spectrum. Towards that end, we have developed code to compute and plot the eigenvectors (and corresponding eigenvalues) for the action of each smoother on the error. The matrices analyzed have the form I — M A from the error update equation Gj+i = (/ — M A)ei that describes the action of the preconditioner M on the error e. The plots are of eigenvalue/eigenvector pairs. The horizontal axis shows the norm of the gradient of the eigenvectors (considered as functions) — a proxy for "eigenfrequency." The absolute value of the corresponding eigenvalue is used as the vertical axis. The results of applying the eigenanalysis to several two dimensional examples can be seen in Figures 10-17; some convergence histories are shown in Figures 18-19. Each of the examples has the unit square as the domain with uniform porosity and Dirichlet boundary conditions. The first example has a very simple permeability: it takes one value on the left half of the domain, and another value on the right half (shown in Figure 10). In Figure 1 1 a "typical" result for the eigenstructure analysis as applied to the simple two-level permeability field is shown. On the horizontal axis

Darcy Flow, Wlultigrid, and Upscaling

351

S k100'1'FI c, =

$ (cliagAhf- 1 r,

P I + I= p, -+ CI ~ + i + l

rs = fh START

- Ahpi A T>OSE'! Is llrt11 siritxll?

i=O

po = 0 ro = ftL

- ALP{) COAItSEY 13, = A;' Rr, P ~ + I= P I

+ PEI

i+-i+l

FISISII Ph

Pa

rl = f i t - Ahpv

Fig. 8. Schematic of two-level V-cycle multigrid as used to solve the linear system

Ahph = fh. The sequence of iterates for the pressure pi are computed using the ith-stage residual r , = f h - Ahp,, the diagonal elements of Ah (diagAh used in the Jacobi smoothing step), arid the coarse corrector from solving a reduced problem AH. The restriction operator R : Wh --' W I ~x 0, and the prolongation operator P : W H x V H + Wh are the natural projection and inclusion operators. Also note that there may be many ( u ) applications of the smoother between upscaling steps. is measured the norm of the gradient of the eigenvector when interpreted as a 2-D pressure field (this measure is a stand-in for frequency). The vertical axis shows on a logarithmic scale the absolute value of the corresponding eigenvalue. The blue dots show the check-mark-like structure for a weighted Jacobi smoother. This picture is similar to ones usually used in multigrid analysis [BHMOO]. The cyan dots show the structure one obtains for the corresponding weighted Jacobi smoother on the coarse level. I t has the same shape, but the small-eigenvalue dip comes at lower frequencies (coarser modes) as expected. The magenta dots show the structure for the preconditioner that exactly solves the coarse problem. It has eigenvalues greater than one in the coarser modes because the upscaling factor H l h is equal t o three, and not two (the more usual result with eigenvalues bounded by one can be seen in the bottom right of Figure 13). Lastly, the red dots show the structure for the upscaling preconditioner. Some notable features are its complicated organization, that coarse modes (over a large range) have quite small eigenvalues, that there is a steady rise in eigenvalues versus frequency, and that the eigenvalues top out at greater than one. However, this maximum occurs near where the weighted Jacobi smoother is performing at its best.

352

James M. Rath

i t i + l rs = f,,

-

.At,p,

A

1.- l'SC1,2 121;

E; = : 4 ~ !Rr+ ,~ pi+l = PI $* PEi

FTSISlI

i-i+l

PI,

" PI

r, = f f , - A f , p (

Fig. 9. Schematic of the modified two-level V-cycle multigrid as used to solve the

linear system Ahph = fh. The sequence of iterates for the pressure pi are computed using the ith-stage residual ri = fh - Ahpi, the diagonal elements of Ah (diagAh used in the Jacobi smoothing step), and the upscale corrector from solving a reduced problem AH,^. The restriction operator R : Wh --t WH,hx 0, and the prolongation ~ Wh are the natural projection and inclusion operaoperator P : W H , x~ V H , --t tors. Also note that there may be many (v) applications of the smoother between upscaling steps. Figure 12 shows the effect of varying h, the grid size, while leaving the upscaling factor H l h and the permeability field alone. As can be seen, the overall structure remains the same independent of h, but the level of detail one sees in the diagrams increases, as does the scale of the frequencies plotted. Figure 13 shows the effect of varying H l h , the upscaling factor, while leaving the fine grid size h and the permeability field alone. As H l h increases, there is a shift t o lower frequencies for the coarse smoother and the upscaling preconditioner. (As noted above, the eigenvalues of coarse modes for the exact coarse preconditioner get larger with increasing H l h . ) There is a general worsening of performance with increasing H l h as would be expected. Figure 14 shows the Arcoperm permeability field [Arc97], and Figure 16 shows the Brent permeability field [Lin92]. Both show considerably more variation than the simple two-level permability field. They both also have a greater ratio between the greatest and least values of the permeability: the Arcoperm field about a half an order of magnitude, and the Brent field more than four orders of magnitude. For the Arcoperm field, Figure 15 shows the same rich structure as seen for the two-level permeability field, but surprisingly the up-

Darcy Flow, Multigrid, and Upscaling

353

Fig. 10. A simple permeability field.

scaling preconditioner has a similar performance: small eigenvalues for coarse modes with a general trend towards larger (but not too large) eigenvalues for fine modes. At the greatest upscaling factor Hlh of six, the middle modes begin to suffer, too. The results in Figure 17 show the same general trends. However, the middle modes become much more of a problem (note the vertical scale has changed from a high of 10' before to 10' now). One obvious conclusion to draw from the eigenstructure diagrams is that the performance of the upscaling level suffers in the mid-frequency eigenvectors (see also the convergence histories below). Perhaps Krylov space methods (a CG-like iteration) [Bru95, GV96, Dem971, a third level with an upscaler based on a grid staggered relative to the first upscaler, semi-coarsening in different levels in different directions [CWW92], or perhaps a traditional multigrid coarse smoother might be used to improve the method. The diagrams also lead to the reasonable inference that the behavior of the proposed two-level scheme depends on Hlh and k*/k,, but not the absolute level of h. This mirrors the behavior of multigrid, but we have not proven this yet for this scheme. It also seems a reasonable inference that upscaling modified multigrid will always perform better than traditional multigrid. In this proposal, we have been assuming that a single coarsening of the grid is enough to result in a problem that is computationally inexpensive to use. Sometimes this "coarse" grid just is not coarse enough, and the associated problem is still too computationally expensive to work with. In this case, a

James M. Rath

354

amplify attenuate

8"

0

0.5

1

1.5

2

2.5

3

3.5

4

lo4 IUTX!

Eigen-Frequency

h igl I

Fig. 11. A "typical" eigenstructure result for the smoothers and upscaler (RTO elements used at all levels). The fine grid used was 24 x 24, the coarse grid 6 x 6 (with H / h = 4), and Icz = 0.1. The colored dots are eigenpairs for the * coarse

smoothing preconditioner, fine smoothing preconditioner, and preconditioner.

subgrid upscaling

true variational multi-scale (and not just two-scale) method for the upscaling level would be appropriate. This might be accomplished say by decomposing V = V , $ S V = V , , @ A V @ S V and W = W , @ S W = W , , $ A W @ S W into three levels (or more) with a correspoding approximation with multiple levels. The details of this possibility have yet to be worked out. It might also be possible to use traditional algebraic multigrid on the coarsened system that comes from the upscaled problem (29)-(30). The eigenstructure diagrams also give us a way to estimate how often the Jacobi smoother is applied relative t o the number of times the upscaling preconditioner is applied. It is also possible to then estimate the overall error reduction factor for an iteration of our two-level method. This is accomplished by examining frequency by frequency the product of the error reduction factors (eigenvalues) of the smoother and upscaling preconditioner, and by computing the maximum such product. This allows us also to estimate the relative performance of multigrid with the two-level upscaling/smoother method for a given problem. Of course, this is assuming that eigenvectors with

Darcy Flow, Multigrid, and Upscaling

Vary h ; F i x H/h= 3 ; Fix ka = 0.1 . . . .

to-

.""...t..:-...'..".

1'

,,

.

.

.... .

,:a'

A

. ..

..

, I

1

.

.

I-.---.

A

1

''0

. .

10'

355

d.

&

- 9 n r E g W d b V 1

+

*

I

.

,A

m

,

. .. . '

.I

I

..

E ,L & &

K& wMqlPa,.4-#IY1

An A Lm

mm

12 x 12

6x6 i

ID'

...

..

-. .

.. **

-

L

--ma-(*.-*-

24 x 24

_ .

& I

em.

;

.

'

ID.Y--Y-*-

A

-

.

.I$

48 x 48

Fig. 12. Eigenstructure results for the preconditioners for problems with various fine grid sizes h, but fixed upscaling ratio H l h and fixed permeability. The colored dots are eigenpairs for the * coarse smoothing preconditioner, e coarse exact preconditioner, fine smoothing preconditioner, and subgrid upscaling preconditioner.

like "frequency" are alike; in reality, the eigenbases differ and the comparison for a given frequency is not apples-to-apples. Some examples of convergence histories using this analysis are shown in Figure 18. The logarithm of the norm of the residuals/errors is shown on the vertical axis; the number of V-cycle iterations is shown on the horizontal axis. The red and blue pairs show the history for the upscaling modified multigrid; the pink and cyan pairs for an ordinary two-level multigrid. Significantly faster convergence is obtained with the upscaling than with multigrid (although both display linear convergence and dependence on k*/k,). A zero initial guess for the solution is used along with a random source term. T h e diagrams in the top row show the eigenstructure diagram and convergence history for the simple permeability field (shown in Figure 10) on a 24x 24 grid with H l h = 4 and k2 = 0.1. An estimation from the eigenstructure diagram indicates that a single Jacobi smoothing step pre- and post-upscaling in a V-cycle will ensure convergence. Indeed this is the case: the upscaling scheme has a convergence factor of about 0.6 per V-cycle step. On the other hand,

James M . Rath

356

Fix h = 1/24 ; Vary H / h ; Fix k2 = 0.1 w/RTO .... . .i>.. .,. .

.

,.

.

,d

."'

.

.

.

.

.

.

*..4.. 5 _.-.-.

..

r.

.......

.. ...

Lo-.

' ?.

.. .. .

.

7m-.

.

I

d

*- U . . - - y . p n C -

Ph

;'u

a

n

.

.

c

.

IS'

H/h =6 Id

d

.* R . * . r '. m C . W n r * . n , ~k.W

.

.

H/h =4 .

.

;'u

.

.

4

.I#

.

. . . .. +

d

*-.

... ..-. 5 :

... ?

,ma.

-

...

-

I

8

I

'

d"

;

*;

;

-4.mme*.e*..n*-

H/h = 3

0;

; '&

4

td

0

0;

;

'

.~l)*mnr~.w-~-

;r

i

*L

4

I,.'

H/h=2

Fig. 13. Eigenstructure results for the preconditioners for problems with various upscaling ratios H l h , but fixed fine grid size h and fixed permeability. The colored

dots are eigenpairs for the .r coarse smoothing preconditioner, o coarse exact preconditioner, fine smoothing preconditioner, and * subgrid upscaling preconditioner. multigrid has a factor of about 0.94 for the same V-cycle with a smoother on the coarse level. The diagrams in the middle row show the the eigenstructure diagram and convergence history for the Arcoperm permeability field shown in Figure 14 for H l h = 5. An estimation from the eigenstructure diagram indicates that two Jacobi smoothing steps pre- and post-upscaling in a V-cycle again will ensure convergence. Again, the upscaling scheme clearly outperforms ordinary multigrid. The diagrams in the bottom row show the the eigenstructure diagram and convergence history for the Brent permeability field shown in Figure 16 also for H l h = 5. This time estimating the number of necessary smoothings per iteration is more difficult. Forty smoothings were used; note the most smoothings comes from the coarsest eigenmode - the red dot furthest to the left. This is most likely an overestimate of the number of smoothings necessary

Darcy Flow, Multigrid, and Upscaling

357

Fig. 14. The Arcoperm permeability field. The minimum base-10 logarithm of the

permeability is -0.2916 and the maximum 0.1477. for a convergent scheme, but smoothings are inexpensive relative t o upscalings (and probably have a beneficial effect - see below). Figure 19 shows some effects of underestimating the number of necessary smoothings. In the case of the Arcoperm permeability field, changing the number of smoothings from two t o one only slows convergence (and both ordinary and modified multigrid perform comparably). In the case of the Brent permeability field, the iterations diverge. One feature t o note in the top diagram for the Arcoperm permeability field is that there is a steep initial decline in the size of the error. This is probably attributable t o the initial guess for the solution (all zeros) having an error that has large components in the direction of the coarse eigenmodes of the upscaling preconditioner (this probably also explains the large jump in the first or first few steps in each of the other convergence histories - note that the blue/cyan and red/pink dots coincide for step "zero"). The "stalling" of the convergence is probably attributable t o the poor performance of the preconditioner on middle eigenmodes. Another feature t o note is that the doubling of smoothings roughly doubles the rate of convergence of ordinary multigrid, but the modified multigrid experiences a dramatic improvement in performance. It would seem that overestimating the number of necessary fine smoothings has a synergistic effect (and is not very computationally expensive).

James M. Rath

358

Fix h = 1/40 ; Vary H / h ; Fix k w/RTO ,d -

d -

-

to4*

L

A

;

i*

i s .a' 7:. rn!m*rnC.C.ldlcN

A ;

A

x

'd

0

A

b

&

H/h =2 ,d 1

. ... . .-.

..,I

.

; *;

; ,

rnW.LM'IIY-I*

a. -

A

:.

*

r ID'

H/h=5

,.--.

.rr.. I.

..... 1 ! & A d a ; a ;

~ L - C - ~ Y P A

I;;;;I:I n w b L L b . L ;

H/h= 8

r d

; z ~ c I z. I ;

ryr-yawru-asw

H

d

H / h = 20

Fig. 15. Eigenstructure results for the preconditioners for problems with various upscaling ratios H l h with the Arcoperm permeability field in Figure 14. The colored dots are eigenpairs for the .- coarse smoothing preconditioner, * coarse exact preconditioner, fine smoothing preconditioner, and subgrid upscaling preconditioner.

One last feature t o note is t h a t for all the convergent schemes, the error is monotonically decreasing (as is expected for a multigrid-like scheme). Contrast this with (preconditioned) conjugate gradients where the convergence is typically non-monotone (although theoretical bounds are monotone). Conjugate gradients is also more costly asymptotically. In the future we would like t o produce plots of the size of spectrum of the error versus iteration number. A break-down of inter-step frequency reduction (the spectrum before and after smoothing, and before and after upscaling) would also be useful. The ability t o visualize a given eigenmode (that is, make an n-dimensional plot of the pressure that corresponds t o an eigenvector) by clicking on the eigenstructure plot would be a handy tool. In future research, we hope t o be able t o prove t h a t our two-level scheme converges linearly (independent of h, but dependent on H l h and k*/k,), and does so faster than ordinary multigrid. Toward that end, the methods used in standard multigrid will certainly be useful as the upscaling scheme is

Darcy Flow, Multigrid, and Upscaling

359

Fig. 16. The Brent permeability field. The minimum base-10 logarithm of the

permeability is -0.6909 and the maximum 3.5230. posed in a variational setting. There are several monographs [Hac85, McC87, Wes92, Bra93, Hac93, Bru95, Saa95, Sha95, BHMOO, Bra011 and review articles [Xu92, Yse93, Xu97a, Xu97bj that provide useful information (several of the monographs on iterative methods generally). As well, Bramble, Pasciak, and Xu have developed a general framework for subspace correction methods [BPX91, BPWXgla, BPWXglb, Xu92, Bra93, Xu97a, Xu97bl. Their work will be useful in having developed a unified approach to multigrid-like methods, and also specifically because they treat the possibility of non-nested spaces for corrections ( V H , is~ not necessarily a subset of V h ) , and perturbed linear forms (e.g., using quadrature on the term (kuh,v h ) ) . Lastly, we might try to modify the upscaling preconditioner so that it itself is a smoother. Also, the upscaling preconditioner might be modified for use in discontinous Galerkin (DG) methods, or the expanded mixed method [AWY97].

360

M,Rath

James

. .

d

Fix h = 1/40 ; Vary H / h ; Fix k w/RTO .

,

,

.

,

,

,

.

lf

.

.

.

.

.

.

.

\.

.

d.

.

........

.

.

- ..

a .

"

I@',

A

.

.

.

.

.

.

..>

'

"

U

"

U

.

.

.

*;

.

;A

.

t

mid

'4 &

A

A

; *;

A

h

;A

.IIIC-.*cIp.dCI*

2

I lb

H/h =5

.

'*..,, ..-,

.

-

.

.

.

.

.

.

ih.

..

'

.

U

,:,

&

Hjh = 2

.. ..... 3:: . :. O

;

A &

mulbl-.Ips-dLW*

.

..

.Z

. U

'

!

%

1

.

< 1

II*(.IL-C~eLLUI

H/h=8

I

4

l

d

I

r d

A

I

~

U

D

I

U

I

I

B

rnlllnn..,.r*l.l...-N

U

t

A

. U

1

1

I

r d

H / h = 20

Fig. 17. Eigenstructure results for the preconditioners for problems with various upscaling ratios H l h with the Brent permeability field in Figure 16. The colored dots are eigenpairs for the coarse smoothing preconditioner, e coarse exact preconditioner, fine smoothing preconditioner, and 0 subgrid upscaling preconditioner.

Darcy Flow, Multigrid, and Upscaling

361

Fig. 18. Convergence histories (right column) of the two-level upscaling and multigrid schemes for the three sample permeability fields in Figures 10, 14, and 16. The vertical axis shows the logarithm of the residuals/errors; the horizontal axis the number of V-cycle iterations. The light and dark blue colored dots mark the sizes of the residual for the multigrid and upscaling schemes, respectively. The light and dark red dots mark the sizes of the error. For reference, the left column shows the eigenstructure diagrams for the corresponding permeability fields.

362

James M. Rath

0

10

20

90

40

-1

50

Brent (v = 5 )

I

80

70

..

10''

lo'*

BO

numbex

.

-

Resklual(UP) Ermr (UP) Residual (MQ) Ermr(MG)

.....

10'~-

......... ............ .-aL

-

...

........ . .....*..........-.

Fig. 19. More convergence histories of the two-level upscaling and multigrid schemes that illustrate the effect of and underestimating the number of necessary smoothings. The coloring scheme is the same as used in Figure 18. The diagram on the top is comparable to the one in the center-right of Figure 18; one smoothing per step are used instead of two. The diagram on the bottom is comparable to the one in the bottom-right of Figure 18; five smoothings per step are used instead of forty.

Darcy Flow, Multigrid, and Upscaling

363

References [AB85]

D. N. Arnold and F. Brezzi. Mixed and nonconforming finite element methods: implementation, postprocessing and error estimates. RAIRO Model. Math. Anal. Numer., 19:7-32, 1985. [AB02] T. Arbogast and S. L. Bryant. A two-scale numerical subgrid technique for waterflood simulations. SPE J., pages 446-457, Dec. 2002. [Arb] T. Arbogast. Analysis of a two-scale, locally conservative subgrid upscaling for elliptic problems. [ArbOO] T. Arbogast. Numerical subgrid upscaling of two-phase flow in porous media. In Z. Chen, R. E. Ewing, and Z.-C. Shi, editors. Numerical treatment of multiphase flows in porous media, volume 552 of Lecture Notes in Physics, pages 35-49. Springer, Berlin, 2000. [Arb02] T. Arbogast. Implementation of a locally conservative numerical subgrid upscaling scheme for two-phase Darcy flow. Computational Geosciences, 6:453-481, 2002. [Arc97] Arcoperm permeability field data. Personal correspondence, 1997. [AWY97] T. Arbogast, M. F. Wheeler, and I. Yotov. Mixed finite elements for elliptic problems with tensor coefficients as cell-centered finite differences. SIAM J. Numer. Anal., 34:828-852, 1997. [BDDF87] F. Brezzi, J. Douglas, Jr., R. Duran, and M. Fortin. Mixed finite elements for second order elliptic problems in three variables. Numer. Math., 51:237-250, 1987. [BDM85] F. Brezzi, J. Douglas, Jr., and L. D. Marini. Two families of mixed elements for second order elliptic problems. Numer. Math., 47:217-235, 1985. [Bea72] J. Bear. Dynamics of Fluids in Porous Media. Dover, New York, 1972. [BF91] F. Brezzi and M. Fortin. Mixed and hybrid finite element methods. Springer-Verlag, New York, 1991. [BHMOO] William L. Briggs, Van Emden Henson, and Stephen Fahrney McCormick. A Multigrid Tutorial. SIAM, 2000. [BLP78] A. Bensoussan, J. L. Lions, and G. Papanicolaou. Asymptotic Analysis for Periodic Structure. North Holland, Amsterdam, 1978. [BPWX91a] James H. Bramble, Joseph E. Pasciak, J. Wang, and Jinchao Xu. Convergence estimates for multigrid algorithms without regularity assumptions. Mathematics of Computation, 1991. [BPWXQlb] James H. Bramble, Joseph E. Pasciak, J. Wang, and Jinchao Xu. Convergence estimates for product iterative methods with applications to domain decomposition. Mathematics of Computation, 57:1-21, 1991. [BPX91] James H. Bramble, Joseph E. Pasciak, and Jinchao Xu. The analysis of multigrid algorithms with nonnested spaces or noninherited quadratic forms. Mathematics of Computation, 56:1-34, 1991. [Bra93] James H. Bramble. Multigrid Methods. Number 294 in Pitman Research Notes in Mathematics. Longman Scientific and Technical, New York, 1993. [BraOl] Dietrich Braess. Finite elements : theory, fast solvers, and applications in solid mechanics. Cambridge University Press, New York, second edition, 2001.

364 [Bru95]

James M. Rath

Are Magnus Bruaset. A Survey of Preconditioned Iterative Methods. Number 328 in Pitman Research Notes in Mathematics. Longman Scientific and Technical (John Wiley), New York, 1995. [CDGW03] Y. Chen, Louis J. Durlofsky, M. Gerritsen, and Xian-Huan Wen. A coupled local-global upscaling approach for simulating flow in highly heterogeneous formations. Advances in Water Resources, 2003. Submitted. [CH02] Z. Chen and T. Y. Hou. A mixed multiscale finite element method for elliptic problems with oscillating coefficients. Math. Camp., 2002. [CWW92] L. C. Cowsar, A. Weiser, and M. F. Wheeler. Parallel multigrid and domain decomposition algorithms for elliptic equations. In D. Keyes et al., editors. Fifth International Symposium on Domain Decomposition Methods for Partial Differential Equations, pages 376-385. SIAM, Philadelphia, 1992. [Dar56] Henry Darcy. The Public Fountains of the City of Dijon, chapter Appendix D. Victor Dalmont, Paris, 1856. [Dem97] James W. Demmel. Applied numerical linear algebra. SIAM, Philadelphia, 1997. [DEW83] J. Douglas, Jr., R. E. Ewing, and M. F. Wheeler. Approximation of the pressure by a mixed method in the simulation of miscible displacement. R.A.I.R.O. Model. Math. Anal Numer., 17:17-33, 1983. [DEW84] B. L. Darlow, R. E. Ewing, and M. F. Wheeler. Mixed finite element methods for miscible displacement problems in porous media, SPE 10501. Soc. Petrol Eng. J., 24:391-398, 1984. [DJ97] Clayton V. Deutsch and Andre G. Journel. GSLIB geostatistical software library and user's guide. Oxford University Press, New York, second edition, 1997. [DR85] J. Douglas, Jr. and J. E. Roberts. Global estimates for mixed methods for second order elliptic equations. Math. Comp., 44:39-52, 1985. [Dur02] Louis J. Durlofsky. Upscaling of geological models for reservoir simulation: Issues and approaches. Computational Geosciences, 6:1-4, 2002. [EHWOO] Y. R. Efendiev, T. Y. Hou, and X.-H. Wu. Convergence of a nonconforming multiscale finite element method. SIAM J. Numer. Anal, 37:888-910, 2000. [ERW84] R. E. Ewing, T. F. Russell, and M. F. Wheeler. Convergence analysis of an approximation of miscible displacement in porous media by mixed finite elements and a modified method of characteristics. Comp. Meth. in Appl. Mech. and Engng., 47:73-92, 1984. [Far02] C. L. Farmer. Upscaling: a review. International Journal for Numerical Methods in Fluids, 40(l-2):63-78, 2002. [FC79] R. A. Freeze and J. A. Cherry. Groundwater. Prentice-Hall, Englewood Cliff's, New Jersey, 1979. [G'*'85] James Glimm et al. Sharp and diffuse fronts in oil reservoirs: Front tracking and capillarity. In William E. Fitzgibbon, editor, Mathematical and Computational Methods in Seismic Exploration and Reservoir Modeling, pages 54-76, Philadelphia, 1985. SIAM. [GV96] Gene H. Golub and Charles F. Van Loan. Matrix Computations. The Johns Hopkins University Press, Baltimore, third edition, 1996.

Darcy Flow, Multigrid, and Upscaling [GW88]

365

R, Glowinski and M. F. Wheeler. Domain decomposition and mixed finite element methods for elliptic problems. In R. Glowinski et al., editors, First International Symposium on Domain Decomposition Methods for Partial Differential Equations, pages 144-172. SIAM, Philadelphia, 1988. [Hac85] Wolfgang Hackbusch. Multi-grid methods and applications. Number 4 in Computational mathematics. Springer-Verlag, New York, 1985. [Hac93] Wolfgang Hackbusch. Iterative Solution of Large Sparse Systems of Equations. Number 95 in Applied Mathematical Sciences. SpringerVerlag, New York, 1993. [HFMQ98] T. J. R. Hughes, G. R. Feijoo, L. Mazzei, and J.-B, Quincy. The variational multiscale method—a paradigm for computational mechanics. Comp. Meth. in Appl. Mech. and Engng., 166:3-24, 1998. [HNOO] L. Holden and B. F. Nielsen. Global upscahng of permeability in heterogeneous reservoirs; the output least squares (OLS) method. Transport Porous Media, 40:115-143, 2000. [HW97] T. Y. Hou and X. H. Wu. A multiscale finite element method for elliptic problems in composite materials and porous media. J. Comput. Phys., 134:169-189, 1997. [Lin92] Brent Lindquist. Geostatistically generated permeability field data. Personal correspondence, October 1992. [McC87] Stephen Fahrney McCormick, editor. Multigrid methods. SIAM, 1987. [RdM97] Ph. Renard and G. de Marsily. Calculating equivalent permeability: a review. Advances in Water Resources, 20:253-278, 1997. [RT77] R. A. Raviart and J. M. Thomas. A mixed finite element method for 2nd order elliptic problems. In Mathematical Aspects of Finite Element Methods, number 606 in Lecture Notes in Mathematics, pages 292-315. Springer-Verlag, New York, 1977. [RT91] J. E. Roberts and Jean-Marie Thomas. Mixed and hybrid methods. In P. G. Ciarlet and J. L. Lions, editors, Handbook of Numerical Analysis, volume 2, pages 523-639. Elsevier Science Publishers B.V. (NorthHolland), Amsterdam, 1991. [RW83] T. F. Russell and M. F. Wheeler. Finite element and finite difference methods for continuous flows in porous media. In R. E. Ewing, editor, The Mathematics of Reservoir Simulation, number 1 in Frontiers in Applied Mathematics, pages 35-106, Chapter II. Society for Industrial and Applied Mathematics, Philadelphia, 1983. [Saa95] Yousef Saad. Iterative Methods for Sparse Linear Systems. Brooks/Cole, 1995. [Sch74] A. E. Scheidegger. The Physics of Flow Through Porous Media, 5rd ed. University of Toronto Press, Toronto, 1974. [Sha95] V. V. Shaidurov. Multigrid methods for finite elements, volume 318 of Mathematics and its applications. Kluwer Academic Publishers, Boston, 1995. [SP80] E. Sanchez-Palencia. Non-homogeneous Media and Vibration TheoryNumber 127 in Lecture Notes in Physics. Springer-Verlag, New York, 1980. [WEH02] X. H. Wu, Y. Efendiev, and T. Y. Hou. Analysis of upscaling absolute permeability. Discrete Contin. Dyn. Syst. Ser. B, 2:185-204, 2002.

366

James M. Rath

[Wes92] [WGH96]

[Xu92] [Xu97a]

[Xu97b]

[Yse93]

Pieter Wesseling. An Introduction to Multigrid Methods. John Wiley, New York, 1992. Xian-Huan Wen and Jaime J. Gomez-Hernandez. Upscaling hydrauhc conductivities in heterogeneous media: An overview. Journal of Hydrology, 1996. Jinchao Xu. Iterative methods by space decomposition and subspace correction: A unifying approach. SIAM Review, 34:581-613, 1992. Jinchao Xu. An introduction to multigrid convergence theory. In Raymond H. Chan, Tony F. Chan, and Gene H. Golub, editors, Iterative methods in scientific computing, Singapore, 1997. Springer. Jinchao Xu. An introduction to multilevel methods. In M. Ainsworth, J. Levesley, M. Marietta, and W. A. Light, editors. Wavelets, Multilevel Methods and Elliptic PDEs, Numerical Mathematics and Scientific Computation (Oxford science publications), pages 213-302, Oxford, 1997. Clarendon Press. Harry Yserentant. Old and new convergence proofs for multigrid methods. Acta Numerica, 3:285-326, 1993.

Iterated Adaptive Regularization for the Operator Equations of the First Kind * Yanfei Warig^ and Qinghua Ma^ ^ State Key Laboratory of Remote Sensing Science, Jointly Sponsored by the Institute of Remote Sensing Applications of Chinese Academy of Sciences and Beijing Normal University, P.O. BOX 9718, Beijing 100101, P.R.China, yf wang.ucf Qyahoo . com Department of Mathematics, University of Central Florida, P.O.BOX 161364, Orlando, FL 32816-1364. yfwaiig9math.ucf.edu ^ Department of Information Sciences, College of Arts and Science of Beijing Union University, Beijing, 100038, P.R.China, qiiighua9ygi.edu.cn S u m m a r y . The adaptive regularization method is first proposed by Ryzhikov et al in [RB98] for the deconvolution in elimination of multiples which appears in geoscience and remote sensing. They have done experiments to show this method is very effective. This method is stronger than the Tikhonov regularization in the sense that it is adaptive, i.e., it automatically eliminates the small singular values of the operator when which is nearly singular. In this paper, we will show that the adaptive regularization can be implemented in an iterated way. Particularly, we show that if some priori knowledge-based information (i.e., a priori strategy for choosing the regularization parameter) is known in advance, the order of the convergence rate can approach 0{5'>''+^) for some i/ > 0. A posteriori strategy for choosing the regularization parameter is also introduced, the regularity is proved.

1 Introduction Many inverse problems of mathematical physics and problems of remote sensing lead to the solution of the first kind operator equations Tx = y,

(1)

where T is a bounded linear operator from Hilbert space X into the Hilbert space Y and T is often known as a convolution operator. Assume t h a t we are interested in the MA'^^'-soIution (minimum-norm least-squares solution) x"*" of equation (1), then it is well-known t h a t The work is partly supported by Chinese national 973 research project G20000779.

368

Yanfei Wang and Qinghua Ma x+=T+2/,

(2)

where T+ denotes the Moore-Penrose generalized inverse of T'^. For nonclosed range R{T) of T, the MA'^S'-solution x'^ exists only for D{T+) = R{T) + B,{T)^

CY

and depends discontinuously on the right-hand side. A prototype for such ill-posed problems is the Predholm integral equations of the first kind {Tx){t):=

/ k{s,t)xit)dt Jo

= y{s),

SG[0,1],

where A; is a non-degenerate L^-kernel and X = Y = L^[0,1] (see [EHN96, Tik63]). In many applications, the right-hand side can not be obtained exactly, that is to say, only some ys GY is available, \\y-ys\\<5,

(3)

where 5 is a priori known error level. Since T+ is unbounded, T'^ys is not a reasonable approximation to T+y, even if it exists. Because of this problem, one has to use the regularization method for approximating T'^y. A well-known method for the solution of ill-posed problems is the Tikhonov regularization, which is considered by Tikhonov, 1963 (see [Tik63]) and independently by Philhps, 1962 (see [Phi62]). For a > 0, we denote x" as the unique solution of (T*T + aI)x = T*ys, (4) where a is called the regularization parameter. For the theory of Tikhonov regularization, we refer the well written book by Groetsch, 1984 (see [Gro84]). It is well-known that if the regularization parameter a is chosen in dependence of 5 such that (see [Tik63]) (52 hm —T— = 0, and lim aid) = 0,

5—^0 a{S)

5^0

^ '

then lim | | x f ) - T + y | | = 0 . 0

>U

If the exact solution fulfills the smoothness property T+y e R{{T*Ty) for some i/ £ (0,1], then for an priori choice of a such as a{5) = c 5 ^ , c > 0, one obtains the convergence rate (see [Gro84, Neu97])

(5)

Iterated Adaptive Regularization for the Operator Equations

\\xf^^

369

-T+y\\=0{5^^).

The standard Tikhonov regularization can be implemented in an iterated way, we refer [EHN96, HG98, KC79, Gfr87] for details. Schock in [Sch84] considered the regularization with adjoint operators. Recently Ryzhikov et al [RB98] and Wang [Wan02] have considered the adaptive regularization method (simply denote by AR) for solving ill-posed inverse problems (1) with adjoint operator. Ryzhikov et al have utihzed this kind of method for solving deconvolution problems in elimination of multiples successfully. Denote H = T*T, which is an adjoint operator, Ryzhikov et al's adaptive regularization is based on the minimization problem min J[a,a;] = \\Tx - y^f + a\\x\\l,

(6)

where \\X\\D := ^/{Dx,x), i? is a definite or semi-definite operator. Choosing D = H"^ and denoting x" the solution of (6), then for a > 0, Ryzhikov et al obtain (if 2 + al)x1 = Hzs, (7) where zs = T*ys. The filtering function of the AR is defined as i?'^^(A) = A(A^ + a)~^. The remarkable difference between the adaptive regularization and the Tikhonov regularization lies in that the former can simultaneously ehminate the null space elements of the operator when it approaches singular while the later needs the regularization parameter to suppress the singularity. The adaptive regularization (7) can be implemented in an iterated way: (F^ + al)xf, = axi_i

+Hzs,

a > 0.

(8)

We will analyze the convergence properties of the iterated AR in the section 2. In section 3, we will prove that the convergence rate of iterated AR can be up to 0(5'^"+^) for any z^ > 0 if a priori strategy is satisfied. In section 4, we show that if the discrepancy principle is chosen as a posteriori strategy, then we can only obtain the optimal convergence rate 0((52"'+i) for some v > ^•

2 Convergence of t h e Iterated AR

(lAR)

For noise-free data, the iterated version of the AR is generated as {H'^+ aI)xn = aXn-i+Hz

(9)

or Xn = a{H'^ + aI)-^Xn^\

+ {H'^ + aI)-^Hz,

(10)

where z = T*y. For simplicity, we choose the initial guess value XQ = 0. Note that the operators [H"^ -t- al)~^ and {H'^ + aI)~^H

are both every-

where defined and bounded with 11(^2-fa/)-i|| < - and WiH"^+ aI)-'^H\\ <

370

Yanfei Wang and Qinghua Ma

1 . Therefore, for each fixed n, the sequence {a;„} generated by (9) is stable with respect to perturbations in y. By induction, (10) is transformed to n-l

J2 a^{H^

+ ocI)-^-^Hz

:= RI'^^{H)Z,

(11)

fc=0

where XQ := 0 and R(/\^{X) is defined as

«r(A)-i:ix4^4(i-(^)"). fc=0 ^

(12)

'

We also have that \\R'''''{H)\\ < p | (13) It is easy to see that i?^-^^(A) —> A^^ as a —> 0 and Rn^^{X) —> 0 as A —> 0. This is a major difference from traditional Tikhonov regularization method. We have known that the Tikhonov filtering function F{F'^'^^{\) = (A + a)^^ —> a~i as A —> 0. This indicates that if the operator T*T is degenerated and has an eigenvalue being null, the /Ai?-inverse operator eliminates null-space components at all. In the following, we analyze some of the convergence properties of the lAR when the data are noise-free. Lemma 1. Let x he any solution of Tx = Qy, where Q is the orthogonal projection ofY onto R{T). If {xn} is given by (9) or (10) then for alln> 1, we have x~Xn==a''{H'^ + aI)-''x. (14) Proof Since Tx = Qy, hence Hx = z. Prom (10) we know x-xi

= X- {H'^ + aiy^Hz = [I~~{H^+aI)-^H^]x

= a{H'^ +aiy^x; X ~ Xn = X — a{H'^ + aI)~^Xn-i = a{H'^ + aiy^{x

— {H + al)"

Hz

~ x„_i).

By induction and notice that XQ := 0, we obtain the results immediately. D Denote r„(A) = ( A ^ + O ) " ' ^'^'^ "•°^® ^^^^ •"''^ ~ '^^V^ '^^ h&ve Lemma 2. x+ — x„ =

rn{H)x'^.

Clearly, a;„ —> x'^ for any a > 0 as n —> oo. If in addition some "smoothness" conditions are given, i.e., x+ = Wtu for some w S D{H'^) and u > Q, then we have the following results:

Iterated Adaptive Regularization for the Operator Equations

371

Lemma 3. a;+ — a;„ = Sn,u{H)u), where Sn,v{X) := A'^r„(A). The error analysis will hinge on an investigation of the function A

-T ex

where u > 0, a > 0 are given parameters. As we are interested in fixed f > 0 and n —> oo, we shall assume that n > i^/2 (note that for v > 2n, Sn,(^(A) is increasing in A). An easy calculation shows that s^ ^^(A) = 0 if and only if A* =

( ^ ) * , 2n - w

thus max s„,^(A) = s„,i/(A*) A€[0,oo) 2n-L' =

( ,2n

) — 1/.

<

. 2 (1

V )"

V

)—'

av
(15)

Theorem 1. If x'^ = H'^uj for some u > Q and lo € D{H'^), then

\\x+-xJ
Theorem 1 indicates that the approximation error can be expressed as 0{n~'^), thus the behavior of the iteration index k also serves as a regularization parameter.

3 Regularity under a Priori Strategy Assume that ys is an approximation to the data y with error level 5 such that \\y — ysW < S- Let {x^} be the sequence generated by (10) using the data ys, i.e., xi = a{H^ + aiy^xi_^ + {H^ + aiy^Hzs (16) or for simplicity <

= R'n'^aWzs,

(17)

where zs = T*ysWe call the sequence {.T^} is stable, if \\xf^ — Xn\\ —» 0 as 5 —» 0. From (12) one can easily calculate that

372

Yanfei Wang and Qinghua Ma r„(A) + Afl^^„«(A) = l.

Since 0 < r„(A) < 1, hence 0 < 1 - Ai?^^„^(A) < 1. We now derive a stability estimate for the sequence xf^. Since x„, x^ G D{T), and note that (13), we have by simple calculation that

=

{K'^^{H){z-zs),Ri^^{H){z~zs))

<5'^\\T\\-'^. Furthermore, by (12) and using triangular inequahty, we have ,iARn^

^ 1- .

"


^

A2 + 2a A(A2+a)'

The maximum value of the function ._ A^ + 2a ^^^' - A(A2 + a) is /max(A) — y ^ . Hence the error ||.T„ — a:;^||^ can be further estimated as 11 -^n ~ 2;„ 11 = [Xn ~ X^, Xn — X^ )

= ( r < ^ „ « ( F X „ « ( F ) r * ( y - ys),y~

ys)

<\\Ri^J^iH)Ri^J^iH)H\\\\y~ysf <\\R'^^^{H)\\\\y-ysf

< 2v^ and hence

x.'ll<

^ '

\/2^'

By (17), we also have \\Txr,-Txi\\

=

\\TRi'^J'{H){z-zs)\\

<\\T\\5\\T\\-'=S. With the above analysis, we write a lemma as follows: Lemma 4. For the sequence {xn}, {xf^}, we have the following stability results:

5„ . V^s and \\Tx„~-Tx'J<5.

Iterated Adaptive Regularization for the Operator Equations

373

Theorem 2. If a{5,ys) —> 0 and -—-z—r- —> 0 as 5 —> 0 and n —> oo, a{5,y5) then xi := R'„^^{H)zs -^ x+ = T+y. Proof. By Lemma 2 and Lemma 4, the proof of the result is straightforward. D Compared with the Tikhonov regularization, this is a much better result, which provides more stronger regularization. As long as n —> oo and 5 —> 0, then for any regularization parameter a{6) = 6'', k < 4, the iterated AR converges, while for Tikhonov regularization, a{6) cannot be exceeded to 6^. Lemma 5. Let x'^ ^ 0. Then for all 5 > 0 there exists a unique a{S) satisfies the equation \\xn-x+\\=6/^.

(18)

Proof. Since Tx^ = Qy, from (11) we have a:„ - a;+ = -a"(fl-2 ^

al)-''x+.

Denoting the spectral family of the operator H = T*T as E\, we have

Hence if we define

« ) - /o/

7i(^^d\\E,x+f-5'^

(A2 + a)2»

then by definition x"*" G N{T)'^ and since x'^ ^ 0, we know that 0(a) is continuous and strictly monotonically increasing with lim (/»(«) = —S'^ and a—^0

lim

(/)(Q;)

= oo,

a—>oo

this proves the lemma. D Now we prove the rate of convergence for the iterative method (10). Theorem 3. Let x+ G R{H") and let a[5) he as in Lemma 5, then \\xf^,g-. — x+\\ =

0{5^).

Proof. Let a = a{5) be as in Lemma 5 and define \\Xn C5

••=

X ;;

a" then by (18) we obtain and a can be expressed as

,

374

Yanfei Wang and Qinghua Ma

Now by Lemma 4, ||4(5) - ^'^ II < W^niS) -X+\\ + \\xi(5) - ^niS) \\

This completes the proof. D

4 A Posteriori Choice of the Iteration Index Prom the former section we know that the optimal order of convergence is obtained if the choice of a{5) is in an apriori way, i.e., [|x-„ — x+H = bj\foi. However this is not applicable in applications. Note that by Bernoulli's inequality, we also have

^A2 + a '

'

A2 + a '

"

\^ A-a

hence

<«^(^)=^A^^25S-

(^^)

In this way,

= {Ri^^{H)T*{ys = {Ri^^{TT*)TT*{y,

- y), i?^^„«(ff)T*(y, - y)) - y),Rl^^{H){ys

- y))

2V^ Thus X rj,

X,

5 I

n

,1

Therefore, if the natural numbers n = n[5) is chosen such that n{5) —> oo and (5(n(5))2 —^ 0 as (5 -^ 0, then we also have ||a;^ ^ 3;+|| ^ 0 as 5 —> 0. Numerically, we can not expect too many iteration steps. This means that the iteration index must be controUed in some way such that n{5) is a finite number. In practice, a posteriori way will be better. A popular posteriori way is the discrepancy principle: the iteration process should be stopped at the first occurrence of the index n{5) such that \\Tx'„^s)-y5\\
(21)

Iterated Adaptive Regularization for the Operator Equations

375

with T > 1 another parameter. Note that r„(A) = 1 — Ai?^^^(A) —> 0 as n —> oo, hence the discrepancy ^^n(5) ~ ys satisfies \\ys-Txi^s)\\

=

\\ys~TRi'^''{H)T*ys\\

<e, where e is an arbitrarily small number. This shows that the discrepancy principle (21) will terminate the iteration after n{S) < oo iterations. In the following we will analyze that the iteration with the discrepancy principle as the stopping rule is a regularization. Theorem 4. If n{5) is chosen by the above stopping rule, then l b - T x „ ( 5 ) ! | < ( r + l)(5, ||y-Ta;„(5)_i||>(r-l)5.

(22) (23)

Proof. Note that 1 - r„(A) < 1 and y - Txn-i

- !/5 - Txi^i

- Rltl^{TT*)TT*{ys

- y)

and y - Tx„ = 2/5 - Txi + Rl,^^{TT*)TT*{y

^ y,),

so let n = n{5) and by triangular inequalities, we have \\y - rx„(5)_i|| = \\ys - Txi^s)-i

" (-^ " ^n(5)^i(rr*))(y5 - y)||

> ys - Tx'n(s)-i\\ - W - r„(5)„i(TT*))(j/5 - 2^)11 >TS-5 = (r-l)(5; \\y - Tx„^s)\\ = \\y6 - Tx'^nis) + {!- r„^s){TT*)){y - 2/5)|| < \\ys - Txi^5)\\ + \\{I- rn(6){TT*)){y - ys) < T5 + (5 = (r + 1)5.

Theorem 5. Assume that 0;+ G R{H''), v > ^, x^,^. is the solution of (1) when ys instead of y is given and n{S) is chosen according to (21), then for fixed a, ||x^(^) - x+1| = 0((5 w ) . Proof. Suppose a;"*" = //"w, where LO is normahzed and to S R{H''), then Xn{,5) -X'^

=

rn{H)H''uj.

v > \,

376

Yanfei Wang and Qinghua Ma

By Holder's inequality and (22) of Theorem 4, we obtain Xn(S)-X+\\

<

\\rn{H)L0\\^^\\H-^rn{H)H''ij\\^^ 1

1

< ||w|h-Vl||ff2(a;'Cn(5) , . _a;+)||2'+l =

0{S^STT).

Now assume that n{S) > [ ^ 1 , by (23) of Theorem 4, (r-l)5<||y-Ta;„(5)_i|| = ||Tx+ -ra;„(5)_i|| = ||Tr„(a)_i(ff)ff-a;||. By (15), we find \\Tr^i5)-iiHWtof

= (l/r„(5)_i(F)/f'^a;,r„(5)_i(F)/f^a;) n(o)

'n(d)

n(d) where -Ei_jy and i?2,i/ are two constants with respect to v. Thus

where Ei, = ^JE^.

This shows n((5)
(25)

where Fj, = (:;^)^J^^. Now by (20), we obtain

= C)(55I:TT) + 0((5217TT)

(for fixed a)

= 0((5^). D

Remark. Theorem 5 shows that, using the discrepancy principle as a posteriori strategy, we can not expect to obtain the optimal convergence rate 0((5''''+i) as in section 3. In fact, we even can not obtain the optimal convergence rate Oib"^"^^). This indicates that, sometimes, if some priori knowledgebased information is known in advance, then we can get good results. This phenomenon is particular useful in remote sensing (see [LGWSOl]). But for the posteriori strategy, recalhng that in section 3, ||x„ — x* || can be bounded by f^Ar- and noting that (20), we conclude that even for the posteriori strategy, it does not need too many iterations. If we choose only n = 3 iterations to generate convergence.

> 0, then it needs

Iterated Adaptive Regularization for the Operator Equations

377

References [EHN96] Engl, H.W., Hanke, M. and Neubauer, A. (1996), Regularization of Inverse Problems, Kluwer, Dordrecht. [Gfr87] Gfrerer, H. (1987), "An A-Posteriori Parameter Choice for Ordinary and Iterated Tikhonov Regularization of Ill-posed Problems Leading to Optimal Convergence Rates," Mathematics of Computation, Vol. 49, 507-522. [Gro84] Groetsch, C.W. (1984), The Theory of Tikhonov Regularization for Fredholm Equations of The First Kind, MA:Pitman, Boston. [HG98] Hanke, M. and Groetsch, C.W. (1998), "Nonstationary Iterated Tikhonov Regularization," Journal of Optimization Theory and Applications, Vol. 98, 37-53. [KC79] King, J.T. and Chillingworth, D. (1979), "Approximation of Generalized Inverses by Iterated Regularization," Numer. Fund. Anal. Optim., Vol. 1, 499-513. [LGWSOl] Li, X.W., Gao, F., Wang, J.D. and Strahler, A. (2001), "A priori knowledge accumulation and its application to linear BRDF model inversion". Journal of Geophysical Research, Vol. 106, 11925-11935. [Neu97] Neubauer, A. (1997), "On Converse and Saturation Results for Tikhonov Regularization of Linear 111-Posed Problems," SIAM.J.Numer. Anal, Vol. 34, No.S, 517-527. [Phi62] Phillips, D.L. (1962), "A Technique for the Numerical Solution of Certain Integral Equations of the First Kind," J. Assoc. Comput. Mach., Vol. 9, 84-97. [RB98] Ryzhikov, G.A. and Biryulina, M.S. (1998), "Sharp Deconvolution in Elimination of Multiples," 68th Ann. Internat. Mtg. Soc. Expl. Geophys., 1983-1986. [Sch84] Schock, E. (1984), "Regularization of Ill-posed Equations with Selfadjoint Operators," in: German-Italian Symposium on the Applications of Mathematics in Technology (Eds: Boffi, V. and Neunzert, H.), Teubner, Stuttgart, 340-351. [Tik63] Tikhonov, A.N. (1963), Regularization of Incorrectly Posed Problems, Sov.Math.Doklady, Vol. ^ 1624-27. (translation from Dokl.Akad.Nauk SSSR 153, 49-52, 1963) [Wan02] Wang, Y.F., On the Optimization and Regularization Methods for Inverse Problems, Ph.D Thesis, Academy of Mathematics and System Sciences, Chinese Academy of Sciences, June, 2002.

Recover Multi-tensor Structure from H A R D MRI Under Bi-Gaussian Assumption* Qingguo Z e n g \ Yunmei C h e n \ Weihong Guo^, and Yijun Liu^ ^ Department of Mathematics, University of Florida, Gainesville, FL 32611, USA. {qingguo, yun, guo }9niath. uf 1. edu ^ Department of Psychiatry, University of Florida, Gainesville, FL 32611,-USA. yijunliuQpsychiatry.uf1.edu Summary. We present a variational framework for determination of intra-voxel fiber orientations from High Angular Resolution Diffusion-Weighted (HARD) MRI under the assumption of biGaussian diffusion. The approach is simultaneously estimating and regularizing the two tensor fields and the field of the proportionality corresponding to the mixture of two Gaussians. The prior information on location of the voxels with strong isotropic or one-fiber diffusion is incorporated into the energy functional in order to increase the accuracy of the estimations. The prior information on voxel classification is obtained from the spherical harmonic (SH) representation of the apparent diffusion coefficient (ADC) profiles estimated and regularized simultaneously from noisy HARD data. The performance of the proposed model has been evaluated on the human HARD MR images, and the experimental results indicate the effectiveness of the model in recovering intra-voxel multi-fiber diffusion. K e y w o r d s : Multi-fiber, biGaussian diffusion, spherical harmonics, regularization.

1 Introduction Diffusion-weighted (DW) imaging adds to conventional MRI the capability of measuring the random motion of water molecules, referred as diffusion. T h e diffusion of water molecules in tissue over a time interval t can be described by a probability density function pt on the displacement r. Since Pt(r) is largest in the directions of least hindrance to diffusion and smaller in other directions, the information about Pf (r) reveals fiber orientations and leads to meaningful inferences about the microstructure of the tissues. Since pt(r) is related with D W signal s(q) through the equation: s{q) =^ so f Pt{r)e-'^''dT, * Thanks to NIH/DA016221 for funding.

(1.1)

380

Qingguo Zeng, Yunmei Chen, Weihong Guo, and Yijun Liu

where q is the direction of the diiTusion sensitizing gradients, and SQ is the signal in the absence of any gradient, it can be estimated from inverse Fourier transform (FT) of s(q)/so- However, this requires a large rmmber of measurements of s(q) over a wide range of q in order to perform a stable inverse FT. In ([TWBW99]) Tuch et al. introduced the method of high angular resolution diffusion-weighted (HARD) imaging. In [WRTOO] Wedeen et al. succeed in acquiring 500 measurements of s(q) in each scan to perform a fast FT inversion. One of the alternative for recovering the intra-voxel structure is using the information of ADC profiles d, that is defined for DW MRI by

d(q) = -hog'M, 0

(1.2)

So

If pt is a Gaussian, d{u) = bu^Du and s(u) = SQC^^ '-'" where D is the diffusion tensor, b = t | q p is the diffusion-weighting factor, and u = q/|q|, This model is useful for detecting single fiber diffusion. ([BML94a, BML94b, CBP90]) However, it has been recognized that the Gaussian model is inappropriate for assessing multiple fiber tract orientations([BML94b, AHLOl, FVaOlb, TWBW99, WRTOO, FraOla]). A simple alternative is assuming that pt is a mixture of n Gaussians. Then the diffusion is modeled by n

s(u) = s o E / i e - ' ' " " ^ ' " ,

(1.3)

where /j > 0 and Yl^=i fi = ^- The /j is considered as the apparent volume fraction of the voxel with diffusion tensor Di. Recently, Parker et al [PA03] and Tuch et al. ([TRW02]) used a mixture of two Gaussian densities to model the diffusion for the voxels where the Gaussian model fits the data poorly. Such voxels were identified by using the SH representation of d{u) in [PA], and the feature of multiple max/min of d{u) in u in [TRW02]. In this note we present a new variational method for recovering the intravoxel structure under the assumption that pt is a mixture of two Gaussians. Our approach differs from the existing methods in the following aspects. First, we recover each field Di{x) or /i(x) globaUy by simultaneous smoothing and data fitting, rather than estimating them from (1.3) in each isolated voxel, which leads to an ill-posed problem. Second, we recover the ADC profile d{x, 9,4>) in SH representation from the noisy HARD data before estimating -Di(x) and /i(x). The recovered d and the voxel classification on diffusion anisotropy from d are incorporated in our energy function to raise the accuracy of the estimations. Third, we applied the biGaussian model to all the voxels in the field, rather then the voxels where the Gaussian model fits poorly only. Since both the constraint of / i w 1 on the region of strong one-fiber diffusion, and the regularization for /j and Di are built in the model, the single fiber and multi-fiber diffusions can be separated automatically by the model solution. This approach will be less sensitive to the errors in the voxel classification.

Recover Multi-tensor Structure from HARD MRI

381

2 Recovery of A D C Profiles One of promising approaches for identifying voxels containing crossing fibers is using SH representation of the ADC profiles estimated from HARD data, that was initiated by Prank [EraOlb], and also studied in [ABA02]. In their work (i(x, u) was computed from HARD raw data via (1.2), and represented by a truncated SH series: ''max

•>

O!(X,0,0)=^

J2

1=0

Am{^)yi,m{0,
(2.1)

m=-l

where Yi^jn{9,(t>) ^tre the spherical harmonics. The odd-order terms in the SH series are zero, since d is real and antipodal symmetry. If Imax = 2, (2.1) provides a model equivalent to the Gaussian diffusion. Their experimental results showed that the profiles with significant 4th-order components arise consistently in various regions to human brain where two fiber crossing is known to exist. The ^;,m's were computed from the inverse SH transform of d in [FraOlb], and as the least-squares solutions of (2.1) in [ABA02]. Recently, Chen et al. ([CGZ04a]) proposed a simultaneous estimation and smoothing model for recovering d{x, 9,0) from the noisy HARD measurements s(q). In [CGZ04a] d was represented by (2.1) with / = 0, 2,4, and the A;,m(x)'s were estimated by solving the following constrained minimization problem:

™" / { E E iv^i,^(x)r('') + ivso(x)r('') •^^

+Xi I /o

( = 0,2,4 m = - ;

I |s(x,6l,0)-so(x)e-'"'('''"'^)psm0d6ld(/) + A 2 | s o ( x ) - s o ( x ) n d x , Jo (2.2)

with the constraint; I

d{x,0,cp)= J2

E

^'.™W^'.™(^''^)>0-

(2.3)

(=0,2,4m=-;

In (2.2) p(x) and q{x) were chosen such that the smoothing for v4;,m(x) and So(x) is isotropic in the homogeneous regions and anisotropic along their edges to preserve the relevant features. Their experiments demonstrated the effectiveness of this method in recovering d and in enhancement of diffusion anisotropy. The characterization of non-Gaussian diffusion based on the recovered d showed a consistency of the results from their human HARD MRI data with the known neuroanatomy.

3 Determination of Fiber N u m b e r s In our approach of determining fiber directions, the first step is to recover the ADC profiles d from noisy HARD data by using model (2.2)-(2.3). Then,

382

Qingguo Zeng, Yunmei Chen, Weihong Guo, and Yijun Liu

from the SH representation of tfie recovered d we define |Ao,o(x)|

^°W = - A ( ^ ' ^^W =

E::-2l-42.m(x)|

^(^)

wtiere ^ ( x ) = J2i=o 2 4 X^m=-i l^i,jn|(x). Tire voxels witfi significant RQ and i?2 are identified as strong isotropic diffusion and one-fiber diffusion, respectively. The union of all these voxels are denoted by J?i. On J7i /lin (1.3) should be close to 1. Under the assumption of pt being a mixture of two Gaussians, the diffusion is modelled by (1.3) with n = 2. To estimate D, and fi in (1.3) we minimize the following function; 2

min / ( V | V L i | ^ ' ( ' ' ) + |V/|^^W)dx + Ai /

+A2 /

/

JnJo

/

{f-lfdx

|so(x)V/ie-''"^^'^^^"-s(x,^,0)|2sm0d0d0rfx,

Jo

(3.1)

^^1

with the constraint L^'"* > 0. In (3.1) for i = 1, 2 Aj > 0 is a parameter, Pii^) = 1 + i+k\VG\*VLi[^' P/(^) = 1 + i+fc|VG.*v/|^' ^i ^^ ^ ^^^^^ triangular matrix such that D, = LiLj, that is the Cholesky facterization for Di to achieve the positive definite constraint on Dj (see [WVCM03]). jVLjl*' = Z ^ l < m , n < 3 \^ -^i

I i

The first two terms in (3.1) are the regularization terms By the choice of Pi(x) (similarly for p / ) , in the homogeneous region image gradients are close to zero and Pi(x) ~ 2, the smoothing is isotropic. Along the edges, image gradients makes Pi{x) ~ 1, the smoothing is the total variation based and only along the edges. At all other locations, the image gradient forces 1 < p < 2, and the diffusion is between isotropic and total variation based, and varies depending on the local properties of the image. Therefore, the smoothing governed by this model well preserves relevant features in these images. The third term in (3.1) is forcing / ~ 1 on i?i. The last term is the nonlinear data fidelity term based on (1.3). The fiber orientations at each voxel are determined by the directions of the principle eigenvectors of Di and I?2. For the voxels where / (or 1 — / ) is significantly large, we consider 1 — / (or / ) as zero, and (1.3) reduce to the Gaussian diffusion model.

4 Experiment Results We apphed model (3.1) to a set of HARD MRI human data. The raw HARD MR images were obtained using a single shot spin-echo EPI sequence.

Recover Multi-tensor Structure from HARD MRI

383

The imaging parameters for the DW-MRI acquisition are repetition time (TR) = 1000ms, echotime{TE) = 85ms. Diffusion-sensitizing gradient encoding is appUed in fifty-five directions with b = lOOOs/mm^. Thus, a total of fifty-six DW images with the matrix size =256 x 256 were obtained for each slice, and images through the entire brain are obtained by 24 shces. The slice is transversally oriented and the thickness is 3.8mm, and intersection gap between two contiguous shces wiU be 1.2mm. The field of view (FOV) =220mm x 220mm. The figures below show our experimental results from a particular subject in a brain shce through the external capsule. In this experiment we first recovered the ADC profiles d using (2.2)-(2.3), and defined J7I as the set of the voxels, where RQ > 0.8416 or i?2 > 0.1823. These thresholds were selected using the histograms of RQ and i?2- Then, we solved the minimization problem (3.1) by the energy decent method. The information of / w 1 on i7i was also incorporated into the selection of the initial / . By solving (3.1) we obtained the solutions Lj and / , and consequently, Di = LiLf {i = 1,2). Fig.la-b represents the image of / before and after smoothing. / ?» 1 on the dark red regions. The voxels in these regions are identified as isotropic or one-fiber diffusion. This is consistent to the known neuroanatomy. Fig.lc Compare shape of d at three particular voxels(first, second, third column) got by two models, first row: model (3.1); second row: model (3.2) in [CGZ2]. The blue and red arrows indicate the first and second orientations respectively. Fig. 2 shows the color representation of the directions of the principle eigenvectors for I?i(x) and 1^2(x), respectively. By comparing the color in Figs.2a,2b with that in the color pie shown in Fig.2c, the fiber directions are uniquely determined. The representation in Fig.2a,2b are implemented by relating the azimuthal angle {(j)) of the vector to color hue (H) and the polar angle [9 > 7r/2) to the color saturation (S). Shghtly different from [SC], we define H = 0/27r, S = 2{TT- e)/TT, and Value{V) = 1 in SHV. If the direction of the principle eigenvector is represented by (0, d), the fiber orientation can be described by either {9, cp) or {n — 9,(f> + TT). We express the vectors in the lower hemisphere, i.e. 9 > TT/2. The upper hemisphere is just an antipodally symmetric copy of the lower one. The xy plane is the plane of discontinuity. To examine the accuracy of the model in recovering fiber directions, we selected a region inside the corpus callosum, where the diffusion is known as one-fiber diffusion. For each voxel in this region we computed the direction in which d is maximized. This direction vector field is shown in Fig.3a. On the other hand we solved (3.1) and found that the model solution / ~ 1 on this region. Moreover, the direction field generated from the principle eigenvector of Di, as shown in Fig.3b, not only preserves the vector field in Fig.3a, but also more regularized due to the regularization terms in the model.

384

Qingguo Zeng, Yunmei Chen, Weihong Guo, and Yijiln Liu

5 Conclusion A novel model for determination of fiber directions for biGaussion diffusion from HARD MRI is presented. In this model two tensor fields are recovered by simultaneous smoothing and data fitting, as well as incorporating the information on voxel classification of diffusion anisotropy. The numerical results indicates the effectiveness of the model in recovering intra-voxel structure. The choice of the parameters in (3.1) and the determination of the region R1would influence the results. Our choice is made based on the principle that the one-fiber direction from the model agrees with the direction in which d is maximized. We will study in the future how to increase the accuracy in the estimation of the orientations of crossing fibers within a voxel.

Fig.1 (a)-(b) Images of f before and after smoothing respectively. The red region in l ( a ) represents R1.In (b), function f % 1 on the dark red regions. The voxels in these regions are identified as isotropic or one-fiber diffusion, which is consistent to the known neuroanatomy. The comparison of them indicates that the smoothing for f achieved by the constraint f = 1 on Rl is preserved. (c). Compare shape of d at three particular voxels(first, second, third column) got by two models, first row: model (4); second row: model (5) in [CGZ04b]. The blue and red arrows indicate the first and second orientations respectively.

Recover Multi-tensor Structure from HARD MRI

(4

(b)

385

(c)

Fig.2 (a)-(b) Color representation of the directions of the principle eigenvectors for D l ( x ) and D z ( x ) , respectively. By comparing the color-coding in (a) and (b) with the color pie shown in (c), the fiber directions are uniquely determined. We also can see the fiber direction field is smooth.

Fig.3 Vectors in field (a) are the directions corresponding t o maxima of d, and vectors in field (b) are the principle eigenvectors of D's (solution of (3.1))

References [ABA02] D.C.Alexander, G.J.Barker and S.R.Arridge. 2002, " Dection and Modeling of Non-Gaussian Apparent Diffusion Coefficient Profiles in Human Brain Data," Magn.Reson. Med., Vo1.48, 3317. [AHLOl] A.L. Alexander, K.M.Hasan, M.Lazar, D.L.Parker. 2001, "Analysis of partial volume effects in diffusion-tensor MRI," Magn. Reson. Med. V01.45,770-780. [BML94a] P.J.Basser, J.Matliello and D. Lebihan. 1994, "Estimation of the Effective Self-Diffusion Tensor from the NMR Spin Echo," Magn.Reson, Vol. 103(B), 247-254.

386

Qingguo Zeng, Yunmei Chen, Weihong Guo, and Yijun Liu

[BML94b] P.J.Basser, J. Mattiello.D. LeBihan. 1994, "MR diffusion tensor spectroscopy and imaging," Biophys,Vol.66(C267), 255. [CBP90] T.L.Chenevert, J.A.Brunberg, J.GPipe. 1990, "Anisotropic diffusion in human white matter: demonstration with MR techniques in vivo," Radiology, Vol. 177(C405), 401. [CGZ04a] Y. Chen, W. Guo and Q. Zeng. 2004, "Estimation, smoothing and characterization of apparent diffusion coefficient profiles from high angular resolutin DWl", Computer Vision and Pattern Recognition, to appear [CGZ04b] Y. Chen, W. Guo and Q. Zeng. 2004,"Recovery of intra-voxel structure from HARD MRI", ISBI to appear [PraOla] L.R.FYank. 2001, "Characterization of anisotropy in high angular resolution diffusion weighted MRI," Proc. of the 9th Annual Meeting oflSMRM, 1531, [PraOlb] L.R.Pranlc. 2001, "Anisotropy in high angular resolution diffusionweighted MRI," Magn.Reson.Med ,Vol.45,935-939. [PA03] G.J.M.Parker and D.C.Alexander. 2003 "Probabilistic Monte Carlo based mapping of cerebral connections utilising whole-brain crossing fiber information," Proc. of Information Processing in Medical Imaging,684-696. [TWBW99] D.S.Tuch, R.M. Weisskoff, J.W.Belliveau, V.J.Wedeen. 1999, "High angular resolution diffusion imaging of the human brain," Proc. of the 7th Annual Meeting of ISMRM,32l. [TRW02] D.S.Tuch,T.G.Reese,M.R.Wiegell,N.Makris,J.W.Belliveau, V.J.Wedeen. 2002, "High angular resolution diffusion imaging reveals intravoxel white matter fiber heterogeneity," Magn Reson Afed,Vol.48,577-582. [WRTOO] V.J.Wedeen, T.G.Reese, D.S.Tuch, M.R.Weigel, J.G.Dou, R.M.Weisskoff, D.Ghesler. 2000 "Mapping fiber orientation spectra in cerebral white matter with Fourier transform diffusion MRI," Proc. of the 8th Annual Meeting of ISMRM, 82. [WVCM03] Z.Wang, B.C.Vemuri, Y.Chen and T.Mareci. 2003 "A Constrained Variational Principle for Direct Estimation and Smoothing of the Diffusion Tensor Field from DWI," Proc. of IP MI,660-671, [PP] S.Pajevic and C.Pierpaoli, "Color schemes to represent the orientation of anisotropic tissues from diffusion tensor data: Application to white matter fiber tract mapping in the human brain," Magn. Reson. Med. Vol. 42,526-540

PACBB: A Projected Adaptive Cyclic Barzilai-Borwein Method for Box Constrained Optimization* Hongchao Zhang and William W. Hager Department of Mathematics, University of Florida, Gainesville, FL 32611, USA. {hzhang,hager}®ufl.edu Summary. The adaptive cyclic Barzilai-Borwein (BB) method [DZ05] for unconstrained optimization is extended to bound constrained optimization. Using test problems from the CUTE library [BCGT95], performance is compared with SPG2 (a BB method), GENCAN (a BB/conjugate gradient scheme), and L-BFGS-B (limited BFGS for bound constrained problems). K e y w o r d s : box constrained optimization, cyclic Barzilai-Borwein stepsize method, nonmonotone line search

1 Introduction Recently, we developed an adaptive cyclic Barzilai-Borwein (ACBB) method [DZ05] for solving unconstrained optimization problems. In this paper, we explain how the line search can be modified so as to solve bound constrained optimization problems of the form: min f(x),

(1)

where / is a smooth function, B= {x G 5R"|L < x < U}, and L and U are upper and lower bounds, possibly infinite. For the bound constrained problem, the ACBB search direction are projected onto the feasible set. Hence, the new algorithm is denoted PACBB [projected adaptive cyclic Barzilai-Borwein method). A step in the BB method [BB88] is given by

4-iSk-i Xk+\=Xk-ak9k,

.„.

ctk =-y •, (2) Sk-iVk-i * This material is based upon work supported by the National Science Foundation under Grant No. 0203270.

388

Hongchao Zhang and William W. Hager

Fig. 1. The projected line search.

where g^ = Vf{xk) is the gradient, viewed as a column vector, Sfc_i = Xk — Xk-i, and yu-i = Qk — Qk-i' Advantages of BB type methods are their low memory requirements and their simphcity - in a neighborhood of a local minimizer, no Une search is needed since the convergence is hnear (see [DZ05]) when the Hessian is strictly convex at the solution. In the cyclic BB method, the same stepsize au is used repeatedly for several iterations - we observe in [DZ05] that by reusing a step for several iterations, convergence can be accelerated. In the adaptive cyclic BB method, we adaptively adjust the cycle length as the iteration progress. In the projected cyclic Barzilai-Borwein method, we project the ACBB iterates onto the feasible set B and perform a nonmonotone hne search between the current iterate and the projection point using the scheme in [DZ05].

2 Algorithm The line search is illustrated in Figure 1. We first take a step along the negative gradient to a point Xk = Xk — ctkgk, where the initial stepsize ak is a safeguarded version of either the previous stepsize or the newly computed stepsize if the cycle length has been reached. If the point Xk lies outside B, we compute

PACBB for Box Constrained Optimization

389

the projection Ps^Xk) of Xk onto B. The search direction dk = Pei^k) ^ ^k is a descent direction since B is convex. A nonmonotone line search is performed along the line segment connecting Xk and PB(xfc). If possible, we accept the point Ptsixk). Otherwise, we backtrack towards Xk- When Xk lies outside of B, the next initial stepsize a^+i is given by the BB formula found in (2). In a forthcoming paper, we prove that when B is replaced by a closed, convex set n, we have liminf ||Pr2(a;fc ~ gk) -XfcHoo = 0 fc—>oo

for a gradient projection/nonmonotone hne search scheme with the structure depicted in Figure 1

3 The numerical results In this section, we compare the performance of the projected adaptive cychc BB algorithm (PACBB) to the SPG2 algorithm developed in [BMROO, BMROl], to the GENCAN algorithm developed in [BM02], and to the LBFGS-B version of the limited BFGS method for box constrained optimization developed in [BLN95, ZBN97]. All codes were written in Fortran and compiled with f77 (default compiler settings) on a Sun workstation. The GENCAN codes were obtained from Jose Martinez's web page: http://www.ime.unicamp.br/~martinez/software.htm and the L-BFGS-B codes were obtained from Jorge Nocedal's web page: http://www.ece.northwestern.edu/~nocedal/software.html For each code, we stopped the iterations if either \\PB{Xk-gk)-Xk\\oo<W~'^

(3)

or \fixk)-fixk.^)\/{l

+ \fixk)\)<10-''.

(4)

We also terminated a code if the number of function evaluations was more than 10^. The test set consisted of all bound constrained problems from the (2002) CUTE hbrary [BCGT95] with more than 50 variables. For all problems where more than one choice of the dimension is given, we use the largest dimension. The numerical results are posted at the following web page: http://www.math.ufl.edu/~hager/papers/CBB Relative to the CPU time, the numerical comparison of PACBB with the other three routines can be summerized as follows: •

PACBB is faster than SPG2 in 36 problems, while SPG2 is faster in 6 problems.

390 • •

Hongchao Zhang and William W. Hager PACBB is faster than GENCAN in 33 problems, while GENCAN is faster in 9 problems. PACBB is faster than L-BFGS-B in 34 problems, while L-BFGS-B is faster in 11 problems.

Excluding the problems where the difference in CPU time was less than 10%, the numerical results can be summerized as follows: • • •

PACBB is faster than SPG2 in 33 problems, while SPG2 is faster in 4 problems. PACBB is faster than GENCAN in 30 problems, while GENCAN is faster in 8 problems. PACBB is faster than L-BFGS-B in 32 problems, while L-BFGS-B is faster in 9 problems.

Figure 2 shows the performance profiles, proposed by Dolan and More [DM02], for the four codes. That is, for the methods analyzed, we plot the

Fig. 2. Performance profiles

fraction P of problems for which any given method is within a factor r of the best time. In a performance profile plot, the top curve is the method that solved the most problems in a time that was within a factor r of the best time. The percentage of the test problems for which a method is the fastest is given on the left axis of the plot. The right side of the plot gives the percentage of the test problems that were successfully solved by each of the methods. In essence, the right side is a measure of an algorithm's robustness.

PACBB for Box Constrained Optimization

391

Since the top curve in Figure 2 corresponds to P A C B B , this method yielded the best C P U time performance for this set of 48 test problems with dimensions ranging from 50 to 15,625. Similar to SPG2, the algorithm PACBB is suitable for large dimensional problems due to its low memory requirements. It is pointed out in [BMROO] t h a t for ill-conditioned problems, SPG2 may converge slowly. Although PACBB seems to deal with ill-conditioning better t h a n SPG2, b o t h G E N C A N and L-BFGS-B are more efficient for the very ill-conditioned problems. Finally, the PACBB algorithm is very easy to implement and it has many promising applications (see [BCM99, GHR93]).

References [BB88] J. Barzilai and J. M. Borwein, Two point step size gradient methods, IMA J. Numer. Anal, 8 (1988), pp. 141-148. [BCM99] E. G. Birgin, I. Chambouleyron, and J. M. Martinez, Estimation of the optical constants and the thickness of thin films using unconstrained optimization, J. Comput. Phys., 151 (1999), pp. 862-880. [BM02] E. G. Birgin and J. M. Martinez, Large-scale active-set box-constrained optimization method with spectral projected gradients, Comput. Optim. Appl., 23 (2002), pp. 101-125. [BMROO] E. G. Birgin, J. M. Martinez, and M. Raydan, Nonmonotone Spectral Projected Gradient Methods for convex sets, SIAM J. Optim., 10 (2000), pp. 1196-1211. [BMROl] E. G. Birgin, J. M. Martinez and M. Raydan, Algorithm 813: SPG software for convex-constrained optimization, ACM Trans. Math. Software, 27 (2001), pp. 340-349. [BLN95] R. H. Byrd, P. Lu and J. Nocedal, A Limited Memory Algorithm for Bound Constrained Optimization, SIAM J. Sci. Comput., 16, (1995), pp. 11901208. [BCGT95] I. Bongartz, A. R. Conn, N. I. M. Gould, and P. L. Toint, CUTE: constrained and unconstrained testing environments, ACM Trans. Math. Software, 21 (1995), pp. 123-160. [DZOl] Y. H. Dai and H. Zhang, An Adaptive Two-point Stepsize Gradient Algorithm, Numer. Algorithms, 27 (2001), pp. 377-385 [DZ05] Y. H. Dai, W. W. Hager, K. Schittkowski and H. Zhang, Cyclic BarzilaiBorwein Stepsize Method for Unconstrained Optimization, March, 2005 (see www.math.ufl.edu/~hager/papers/CBB) [DM02] E. D. Dolan and J. J. More, Benchmarking optimization software with performance profiles. Math. Prog., 91 (2002), pp. 201-213. [GHR93] W. Glunt, T. L. Hayden, and M. Raydan, Molecular conformations from distance matrices, J. Comput. Chem., 14 (1993), pp. 114-120. [GLL86] L. Grippo, F. Lampariello, and S. Lucidi, A nonmonotone line search technique for Newton's method, SIAM J. Numer. Anal., 23 (1986), pp. 707-716. [RayOl] M. Raydan, Nonmonotone spectral methods for large-scale nonlinear systems Report in the International Workshop on "Optimization and Control with Applications", Erice, Italy, July 9-17, 2001

392

Hongchao Zhang and William W. Hager

[Toi97] Ph. L. Toint, A non-monotone trust region algorithm for nonlinear optimization subject to convex constraints, Math. Prog., 77 (1997), pp. 69-94. [ZBN97] C. Zhu, R. H. Byrd and J. Nocedal, L-BFGS-B: Algorithm 778: L-BFGS-B, FORTRAN routines for large scale bound constrained optimization, ACM Trans. Math. Software, 23 (1997), pp. 550-560.

Nonrigid Correspondence and Classification of Curves Based on More Desirable Properties Xiqiang Zheng, Yunmei Chen, David Groisser, and David Wilson Department of Mathematics, University of Florida, Gainesville, FL 32611, USA. {xzheng,yun,groisser,dcw}9math.uf1.edu

S u m m a r y . We present a few more desirable properties to find correspondence and dissimilarity of two plane curves in nonrigid sense. A crossed scheme is used to define dissimilarity metric, which ensures an actual bi-morphism between two curves to be aligned. The optimal correspondence is found by a modified dynamic-programming method. From the optimal correspondence between two curves, we compute their dissimilarity by the overall difference of curvature, local dissimilarity, and local scale of stretching of their corresponding segments. Based on dissimilarity, we can do pattern recognition in nonrigid sense among curves which is very important in many areas such as recognition of hand written letters and cardiac curves where rigid transformations and scaling do not work well. Finally, the effect of correspondence and the classification is shown in application to cardiac curves. K e y w o r d s : Curve alignment, recognition, dynamic programming, correspondence.

1 Introduction This paper employs a few more desirable properties to find correspondence and dissimilarity of two plane curves in nonrigid sense. First, we review current approaches for curve alignment, discuss how we extend current techniques, and present an overview of our approach. Current curve alignment methods can be classified into two categories: methods based on rigid transformations [AF86], [Ume93], and those based on nonrigid deformations [CAS92], [Tag99], [BCGJ98], [You98], [TSG02], [SKK03], and [BMPOl]. Methods based on rigid transformations rely on matching feature points by finding the optimal rotation, translation, and scaling parameters. Since these methods assume t h a t the outlines can be aligned by a rigid transformation, they are sensitive to articulations, deformations of parts, occlusion, and other variations in the object form. Methods based on nonrigid deformations model articulation and other deformations by finding the mapping from one curve to another t h a t minimizes a performance functional consisting of stretching and bending energies.

394

Xiqiang Zheng, Yunraei Chen, David Groisser, and David Wilson

The minimization problem in the discrete domain is transformed into one of matching shape signatures with curvature, bending angle, or absolute orientation as attributes. These methods suffer from one or more of the following drawbacks: asymmetric treatment of the two curves, sensitivity to sampling of the curves, lack of rotation and scaling invariance, and sensitivity to occlusion and articulations. As pointed in [SKK03], Cohen et al. [CAS92] pioneered the deformation-based approach to curve matching. The basic premise of their approach was to match high curvature points along the curves, while maintaining a smooth displacement field. Tagare, Shea, and Groisser in [Tag99] and [TSG02] pointed out the inherent asymmetry in the treatment of the two curves and proposed a bi-morphism that ensures a symmetric formulation. They search in the space of pairs of functions. Sebastian, Klein, and Kimia in [SKK03] defined a similarity metric based on the alignment curve using two intrinsic properties of the curve, namely, length and curvature. The optimal correspondence is found by an efficient dynamic-programming method both for aligning pairs of curve segments and pairs of closed curves, and is effective in the presence of a variety of transformations of the curve. Belongie, Malik, and Puzicha in [BMPOl] introduced the shape context, a rich shape descriptor that aids in judging shape similarity between similar shape points. Frenkel and Basri in [FB03] defined a similarity metric similar to Kimia in [SKK03] and employed Sethian's Fast Marching Method to find the solution with sub-resolution accuracy and in consistence with the underlying continuous problem. Based on those previous work, we formulate a few more desirable properties, which are local dissimilarity, and local scale of stretching, and use crossed scheme to find correspondence and dissimilarity of two plane curves in nonrigid sense. What we implemented are the actual bi-morphisms in the discrete formulation, and the optimal solution is found by a modified dynamic-programming method.

2 The Mathematical Formulation This section discusses the mathematical formulation of the problem of aligning two curves. We first review some previous approaches of aligning two curve segments and then formulate some more desirable properties. 2.1 Some Previous Approaches Denote the two curves to be matched by Ci(si) = (.TI(SI), j/i(si)), Si £ [0, Lj] and C2{s2) = (3^2(52)12/2(52)),S2 6 [0,i^2] , where Sj is the arc length, Xi and j/j are coordinates of each point, Li is the arc length of Cj for i = 1 and 2. Cohen et al. [CAS92] compare the displacement velocities and bending energies. They search for a function / : [0,Li] -^ [0,L2] which minimizes

Eif) = j{KcAf{s.))

- KcJ'dsr + R J ll^i^^ili^lRz^lM^^^,^,

(1)

Nonrigid Correspondence and Classification of Curves

395

where Kc^ and Kc2 are the curvatures along the curves Ci and C2 respectively, and i? is a parameter. Tagare [Tag99] handles this issue by introducing a bi-morphism and searching in the space of pairs of functions. A bi-morphism treats two curves Ci and C2 symmetrically and can be parameterized as ij{s) : [0,L] —> Ci x C2 such that it is regular and its projections on Ci and C2 are the onto, where L is the arc length of the product space. So a bi-morphism consists of a pair of functions lii{s) : [0, Li] -^ Ci and ^^{s) : [0,1/2] —> C2 such that ^i(s) =pi(/u(s)) and /U2(s) = P2{l^{s)) , where pi is a projection from Ci x C2 to Ci for i = 1,2 He defines J ( C I , C 2 ; M ) = / ( ^ ^ i M f M _ d02{P2{Ks)))y^^ /

(A/O

(2)

(J/O

which measures the overall dissimilarity between curves Ci and C2 , and H{CuC2;f^)=

l i - ^ r + i^-d^'ds 'Li ds'^ ' ^Li ds

(3)

which measures the closeness of the bi-morphism to a uniform mapping. Then he searches for the bi-morphism jj, which minimizes the following functional Q{CuC2•,^i) = J{Ci,C2;fi)

+ XH{Ci,C2;i^)

(4)

where A is a parameter. Younes in [You98] considers the correspondence between two curves Ci and C2 as a group action. Actually we just need to consider all ordered permutations of Ci, and hence the search space is a lattice-ordered group consisting of all the ordered permutations of Ci. He got the solution for the case of polygons. If two curves represented by ci(s) and C2{t), where s G [0,Li],t G [OL2], and Li is the arc length of curve d for i = 1 and 2, then a bi-morphism can be represented as a path c{p) : [0, L] —> [0, Li] x [0, L2] where L can be taken as the arc length in the product space. Prenkel and Basri in [BMPOl] search for a path c(p) which solves min / F{c{p))\c'{p)\dp

(5)

where F{s, t) = \Ki{s) - K2{t)\-\-X for some parameter A > 0. Then they use shape contexts to rescue. They show that the ODE: c'(p) = VT satisfies the Euler-Lagrange equation of (5), where T is a function satisfies | VT| = F{s, t). Hence we can use Fast Marching Method to solve for T first, then get c{p) from the ODE: c'{p) = VT. 2.2 Some Observations about the Energy Functions We have the following example which shows that there is a bi-morphism between two curves such that the bending energy is zero but two curves are not

396

Xiqiang Zheng, Yunmei Chen, David Groisser, and David Wilson

similar. Similarly we can show that for any two convex smooth curves, there is a bi-morphism such that the bending energy is zero. E x a m p l e 1 Let 0 < e < 0.1 be a fixed constant; Ci : y = ^,x G [e, 1] and C2 '• y = ^,x E [^/e,l]. Define /i : [e, 1] —» Ci x C2 with projections pi(/i(i)) = {t, ^ ) and p2(/uW) - {Vt, iti) . Then C[{t) = (l,i) and C^{t) = (^,iti).Then ei{t) = a r c t a n ( ^ i ^ ) = arctan(i)

(6)

and ,

v'(t)

-ti

e2{t) = arctan(^^f)-() = a r c t a n ( - ^ ) = arctan(t) Now ^

= Vix[{t))^

+ {y[{t))^ = x / I T F > 0 for any t G [e, 1], and ^

(7) =

y R W F T M W = \/ii + -4 = "W- > ° ^°' ^^y * ^ 1^' 1]' ^ y Equation (2) on page 573 of [Tag99], f = ,J{^^i§Y^{^^2 > Q for any t e [e,l]. Hence si(i), S2(i), and s{t) axe all strictly increasing functions for t G [e, 1], and hence their inverse functions exist. Then by Equations (6) and (7), we have OiipMs))) ds

dSiM^m

diipMsit)))) ds

_ ^

ds dt

,,etan^ ds dt

(8)

ds dt

for i = 1 and 2. Thus dOi{pi{ix{s))) _ d02{p2{l^{s)))

ds

ds

arctani

4^

(9)

at

for any s. Therefore

J(C:,C2;M)= [(^^^MIMA J

ds

^ ^^^MlMAf ds = ^ ds

However in the following, we can show that curve C\ is not similar to curve C2. First their graphs are shown in fig. 1. There let C{a, b) be a point on the curve y — x'^/2 with 0 < a < 1 and b = 0? 12. The equation of Une OA is a; — 2y = 0. Then the distance of the

Nonrigid Correspondence and Classification of Curves

397

Matching of y=x /2 (The rod dots) and y=x73 (The bluo dots) CE=sqrt(5)/20 is the maximai height of the red curve over OA DF=2'sqrt(30)/90 is the maximai height of the blue curve over OB CE/OA=1/10; DF/OB=sqrt(3)/15> CE/OA

0.7

Fig. 1. Matching between curve y = \

0.8

0.9

1

(red dots) and curve y = \- (blue dots)

point C to the line OA is d = ^^^ = ^^^ since 0 < a < 1. The v^ V5 V5 maximum is reached at a = ^ and ^ = | which result in d = - ^ . Hence CE = - ^ . Similarly we can get DF = ^ ^ ^ . The key point between Ci and C2 is that the total change of the angles of tangent lines are the same when ( changes from 0 to 1, and these angles are monotonically changing. Although we have the second term which measures the uniformity of arc length, it may be still difficult to find the global minimum.

3 Formulation of a Few More Desirable P r o p e r t i e s For a bi-morphism /x(s) : [0, L] —> Ci x C2, if we divide the interval into equal subintervals [0,ai], [ai,a2]v t^nd [a„_i,a„] , and if we don't assume any curve segment can be compressed to one point, then the curve Ci is divided into n segments Q j for i = 1,2 and j = 1,2,..., n. Actually in elastic model, no points disappear and hence we can assume it is one to one but a big segment can be compressed to a small segment. If we assume Ci and C2 are smooth, then Q is completely determined by its corresponding n segments. If one segment dj^ of curve d is deformed, then the whole curve is deformed. How much is the whole curve deformed depends on how much those segments are deformed and how well those segments are connected.

398

Xiqiang Zheng, Yunmei Chen, David Groisser, and David Wilson

Two hexagons in fig. 2 are very similar in nonrigid sense b u t differ much by rotation, translation, and scaling.

-15

-10

-5

-15

5

-10

(a)

10

15

(b)

Fig. 2. In fig. 2(b), the x coordinates of the right 3 points are added 3 units

Hence we do need to compare figures in nonrigid sense in many situations such as handwritten recognition and cardiac curves. In fig. 3(b), both angels and edges are changed a bit which results in a much different figures.

-15

5

-10

(a)

10

15

-15

-10

-5

5

to

15

(b)

Fig. 3. In fig. 3(b), 3 angles and 3 edges are changed

In many applications, each corresponding segment does not deform much. A desirable correspondence should match the corresponding curve segments as much as possible.For convenience, let us make the following definitions for curve segments.

Nonrigid Correspondence and Classification of Curves

399

Definition 1. For any curve segment Cij, call the line segment connecting the beginning and the ending points of a curve segment as the base of the curve segment. Definition 2. The maximal distance of the points of a curve segment to its base is called the height of the curve segment. For two curve segments Cij and C2J, let us denote the length of their bases as dj for i = 1,2, and assume that rfi < d2 temporarily. If we enlarge Cij and compress C2J by a factor r so that their bases can match after rotation and translation, then rdi = — and hence r = \/^. After this ahgnment, we can compute the area Ai which is enclosed by the curve segment C,j and its base for i = 1,2. Also we can define the symmetric difference AA12 between Ai and A2, which is the area of {Ai — A2) U(^2 — ^ i ) - Those will be used in the following definitions. Definition 3. Define the scaling factor r between curve segments Cij andC2j flu Definition 4. Define the dissimilarity between curve segments Cij and C2J as p{Cij,C2j) = ^jp-, where AA12 is the symmetric difference between Ai and A2 and d is their common base after scaling. For example, the dissimilarity between an arc of degree 2a of a unit circle and its base is f{a) = ^^7in"a" • ^^'^^easy to show that / ( a ) approaches zero when a approaches zero. Since f'{a) — 2stn^'atana' / ( ^ ) '^ increasing when a G [0,7r]. So the arc segment of degree a behaves like a straight line when a approaches zero. Also when a is bigger in the range [0, 27r], it is more different from straight line. Obviously /(2a) ^ 2/(a), hence this dissimilarity is not additive. Usually / ( 2 a ) > 2/(a) and hence the energy function wiU decrease when two curves are divided into more pairs. However if the number of pairs is fixed, this effect will disappear because one scaling becomes bigger will force some other scaling becomes smaller. Then it is additive based on the method of computation. Definition 5. Define the angle difference between curve segments Cij andC2j as Ad{Cij,C2j) = \AAi — AA2\, where AAi is the angle between beginning and ending tangent line of the curve segment Cij for i = 1,2. Now we can formulate our energy function. Let us assume curve Ci is well sampled into Uj segments Cy for i = 1,2 and j = 1,2,...,Ui. Up to a scaling, we can assume that curves Ci and C2 have the same arc length. Let the interval [0,1] is divided into n subintervals /, = [^^, ^] for i = 1,2,..., n. Then we need to find a bi-morphism ii{s) : [0,1] ^ Ci x C2 such that Pi° ^ is an ordered correspondence between [0,1] and Q for i = 1, 2 such that

400

Xiqiang Zheng, Yunmei Chen, David Groisser, and David Wilson n

minY,p{Pi)

+ r{Qi) + kiAe{Pi) + fc2|r(P.) - 1|

(10)

where Pi = {pi{n{Ii)),p2{fJ'{Ii))) is the ith pair in the matching, Qi = ^(Pj_i U Pj) is the union of the {i — l)th pair and the ith pair in the matching scaled by a factor ^, p{Pi) the scahng of the pair Pi, T{Qi) the scahng of the pair Qi, A9{Pi) the difference between the change of angles of pair Pi as defined in Definition 3.5, and r(P,) the scaling of pair Pi as in Definition 3.3. In equation (10), the term T{Qi) is optional and it can measure the local similarity of connecting points between the {i — l)th pair and ith pair.

4 The Algorithms For Finding the Optimal Bi-Morphism and Correspondence Let 5{{ii,i2], [ji, J2]) denote the cost of matching the iith segment through the i2th segment of curve Ci to the jith segment through the j2th segment of curve C2', let d{i,j) = 6{[l,i], [l,i]). If Ci consists of Ui segments for i = 1,2, then d{ni,n2) is the total cost of matching curves Ci and C2. In many applications, one segment cannot be matched to too much segments. Usually we can only allow one segment be matched to less than or equal to 3 segments. Then we have d{i,j) = min{d{i - k, j - I) + 5{[i - k,i],[j - I, j])\l
< 3|11)

which gives a recipe for computing the total cost d{ni,n2) by a dynamic programming approach. Because of being non additive of local dissimilarity, we need to fix our number of pairs between two curves. But based on the usual approach using equation (12), we just know the upper bound min{ni,n2} of the number of pairs. So we need to modify it a bit to satisfy our purpose. Without lose of generality, by a common refinement we may assume that both curves have the same number of segments. Actually for a lot of data such as the Cardiac data, the number of points representing curves already have the same size. Suppose each curve Cj is almost uniformly divided into n segments Cy for j = 1, 2,..., n. Since we can only allow the scaling of one segment is between g and 3, to find the matching cost between two curves Ci and C2 the second curve C2 should be interpolated into 3n segments d^j for j = l,2,...,3n. Then Ci„ can be matched to 9 possible choices which are Si = C2^„, S2 — C 2 , 3 n U C ' 2 , 3 n - l i •••' '^^ ^9 = ^^2,371 U ^"2,371-1 U C'2,,3n-2 U ••• U C'2,3n-8- T h e n

we have d{n,3ny = min{d{n~

l,3n - fc)'+ (5([n - l,n], [3n -/c,3n])'|l < A: < 9^12)

where d{n, 2>n)' means the cost of matching the first i segments of Ci of to the first j segments of C2, and 5{[ii,i2],[ji,J2])' is defined similarly. Apply equation (13) once, we can get a pair. Hence finally we can get n pairs. Before

Nonrigid Correspondence and Claissification of Curves

401

we compute the total cost, we also need to estimate the two parameters ki and ^2 in (13). For a certain apphcation we can figure out the suitable parameters. Now we can formulate the following algorithms: Step 1 Smooth each set of data a little bit. Step 2 Normalize each curve so that all curves have the same length. Step 3 Apply parametric cubic sphne to fit each data set to get a continuously differential curve. Also we need it to compute the overlapping area to get the local dissimilarity. Step 4 Use cubic polynomial to fit each curve and refine the second curve to be matched into 3n points. Step 5 Compute the tangent line and the corresponding angle at each point. Step 6 Estimate the parameter fci and k2Step 7 Start from curve 1 and apply equation (13) recursively to get the total cost (i(n,3n)'. Step 8* Start from curve 2 and do similarly as Step 6 to get the total cost d(3n,n)". Step 9 Compute the total cost d(Ci, C2) = d{n, 3n)' + d{3n, n)". Note*; Step 8 is optional. By applying Step 8, two curves are treated symmetrically and might improve the cost a little bit, but we need more computation to achieve it. We applied our algorithms to some data of cardiac curves, the results of correspondence are shown in Figure 4.

5 Curve Classification Based on t h e Cost of Nonrigid Correspondence Let us consider the set of all smooth curves and denote this set as F and fix the number of pairs when we search for the best bimorphism for any two curves. Proposition 1. The matching cost d{Ci,C2) between two curves Ci and C2 satisfies the following: l).diCuCi) = 0; 1'). d(Ci, C2) > 0 if and only if Ci is not similar to C2; 2).d{Cx,C2)^d{C2,Ci) Proof. 1) and 2) follows directly from the definitions and algorithms to compute the distance. To prove 1'), if Ci is similar to C2,then after a scaling, they are the same in rigid sense and hence ii(Ci, C2) = 0. On the other hand, if d(Ci, C2) = 0, then after a common scaling, for the corresponding n segments, all local similarities are the same in rigid sense and aU angle changes are the same. Hence Ci and C2 are the same in rigid sense. D Remark: In many previous papers, just 1),2) are shown. One direction of V) has been shown in one previous paper.

402

Xiqiang Zheng, Yunmei Chen, David Groisser, and David Wilson Local Simila'ily am

CortBSpwv

\

•V

:: 0.45

0.5

[ •-^ ^ , (b)

Q.55

(a)

.

|2ll«a;100lKMj4H0.l«l|

.

|3]lila:lO0OO4^4H1.ncl

,

,A-^ „ 075

(c)

O.a

0B5

(d)

Fig. 4. Matching based on Local Similarity and Uniformity of Arc Length Now suppose we have a set of m curves and we want to classify them into several groups based on their similarity which is in nonrigid sense. Based on the algorithm in the last section, we can compute the cost of the matching between any two curves of them which turned out to be a m x m matrix. Then this matrix can be used to pattern recognition among those curves. We tried 30 cardiac curves. Among them 15 curves are normal. For contrast of effect of different terms, we tested the classification effect by different setting of parameters. The results are shown by figures 5 through 10.

6 Conclusion We have shown a new promising algorithm for nonrigid correspondence and curve classification. We compute local similarity based on overlapping area between two corresponding curve segments which is very stable. Actually local similarity can catch the features of ordered curve segments very well. Our computation is based on an efficient modified dynamic programming approach. The experimental results show that this algorithm is robust. Figure 4

Nonrigid Correspondence and Classification of Curves Seed: 3

Seed: 2 •••.

{

403

{

•

1

•

2

J)

\

D

(

\

GIE

.../

((

„;:::- .•••

(b)

•"(a)

(d)

(c)

Seed;?

Seed: 10

(

I ::

-

L^

'"'T-^M.^'^

(e)

•••••• D

(f)

/

JH

•

-p-iol

m

V

,.•••'

'(h)-

(g)

\ \ r~Tn

<.

•

^ ^ /

L

J'

•:

C

13 23

)

_^_!4j

/

:

(k)

(1)

Fig. 5. Angle CoefRcient=5, Uniformity Coefficient=5, Local Similarity not counted Seed: is

Seed: 16

/

\, )

1 •

' 5 |

(

•}

••;,

. / • • • • • • ' •

[

• 16]

]

V- re

)

/ ;

I.

"(a)'

(b)

••

""'"(c)

••

•"••(I)

•

"

Seed: 21

f n~2oi 'v

••'''

/

/

)

/

! • 2'l

1 V

(e)

./•••

1 • 24

).

(f)

"

(8)

)) r^ '

'

•

27

'U.z:y

. . . ' • •

•

•:

(h)

••

Seed; 30

) 1-301

./

/

('(••• (

...,

^^....

....•••••••••••••

(j)

Fig. 6. Angle Coefficient=5, Uniformity Coefficient=5, Local Similarity not counted

404

Xiqiang Zheng, Yunmei Chen, David Groisser, and David Wilson leedi'

Seed: 2

\

(

rf )J V„,...''*"

(a)

)

:

! ..

/

i

••

/

/

G3]

^ v.....^:--(b)

/•••'

nm

L

•---'

(e)-

•)

C—

(f)

mZEi

1\

(h)4

•'1

L

../

/

y

. V..,.-

"TF]

(g) Seed: 13

j

) ;

JT

••

c

c

Seed: 12

;

(d)

(c) /

X

Z3

i

L...

C.

:••• ••(k)

(i)

\ )

1 . 14

13 1

(1)

Fig. 7. Angle CoefRcient=0, Uniformity Coefficient=10 Seed: 2D

Seed; 16

Seed: 1G

\

'\^

16 1

c

I (a)

• •

\\

^ ^ '

^^ (b)

n

18 19

\

^ ''•'v,.^

"(df

(c)

") : ID

i

(e)

L•

24l

/

\CZ1]

^.1L.>/ /

(f)

i,-

(g) Seed: 30

29

(.'"

1 1-291

/

./

( (

(i)

20 1

•

„

•

•

.^.^.,

•

"

..

Fig. 8. Angle Coefficient=0, Uniformity Coefficient=10

•(h)"

)

Nonrigid Correspondence and Classification of Curves

•j .•/

• •

16 17

V

f" (

L^ (a)

•

.-•••]

\

I ( \

10 1

...••••'

(c)

(b)

(d)' s».

Seed: 24

Seed

(

( 1

.

/ •;

-)

15|

/•

p ; 241

/'

I

(„

Si

(f)

ft

/

(X

I

••S..V-

)

// rtt-"';.

1

•

9|

\

! • "1 )

•••••-

(

I

•••••••••

(0

rtr-

Seed: 4

,/

/ C 31]

•••

i

11 12

(h)

Seed: 9

/

• •

X*'"'".'.

(g)

( .•' .) L-^

12

v.

('

r~^

.,...••

(e)

1 • 5|

y.

Seed: 10

405

(k)

)

'(1)

Fig. 9. Angle CoefRcient=10, Uniformity Coefficient=10 Seed: 20

Seed: 6

{

f

f

rt'( A] I

, . ' •

^

.,>•

= -: (a)

('(•

\

/ . / / •• 1

26 27

/ 20 1

• ^

(c)

(b)

S«23

Seed: 30

("

, [Z

Seed: 25

/ y

1

•

)

30 1

/ (

• " • • •

..•••••••••

(e)

1 • 23|

( \ ••

/•

.•••"

;

,

.

,

'"(g)"

(f)

C

V ••(h)'

Seed: 26

)1 EU]

) ::

?

( ( • • •

(i)

//

.••'

...

:••'

• .

25]

28 29

(J)

Fig. 10. Angle Coefficient=10, Uniformity Co6fRcient=10

406

Xiqiang Zheng, Yunmei Chen, David Groisser, and David Wilson

shows t h a t the effect of corresponding is already very good if we just use local similarity and uniformity of arc length. Figures 7 and 8 show t h a t the effect of classification is also very good if we just use two terms just mentioned. We have shown some different results of the classification among 30 cardiac curves for contrast with some previous method. For different kinds of data, we can get better results if we choose suitable parameters to get the appropriate combined effect among local similarity, angle change and uniformity of arc length. If we just use angle change and uniformity of arc length, we may get some undesirable classification such as figure 5(k). T h e computation of this algorithm is efficient. To classify 30 cardiac curves with 193 points representing a curve, we set the scahng between ^ and 2, then the running time is less t h a n 3 hours on a 2.8 GHz machine. Experimental results show t h a t scaling between i and 2 is sufficient for the classification of this data.

7 Acknowledgments T h e authors would like to t h a n k Dr. Hemant for his talk given last year about nonrigid correspondence, and Dr. Frenkel for generously providing some help.

References [AF86] Ayache, N.J. and Faugeras, O.D. (1986), HYPER: A New Approach for the Recognition and Positioning of Two-Dimensional Objects, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 8, no. 1, 44-54. [Ume93] Umeyama, S. (1993), Parameterized Point Pattern Matching and its Application to Recognition of Object Families, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 2, 136-144. [CAS92] Cohen, I., Ayache, N., and Sulger, P. (1992), Tracking Points on Deformable Objects Using Curvature Information, Proc. European Conf. Computer Vision, 458-466. [Tag99] Tagare, H.D. (1999), Shape-Based Nonrigid Correspondence with Application to Heart Motion Analysis, IEEE Trans. Medical Imaging, vol. 18, no. 7, 570- 578. [BCGJ98] Basri, R., Costa, L., Geiger, D., and Jacobs, D. (1998), Determining the Similarity of Deformable Shapes, Vision Research, vol. 38, pp. 2365-2385. [You98] Younes, L. (1998), Computable Elastic Distance between Shapes, SIAM J. Applied Math., V ol. 58, 565-586, . [TSG02] Tagare, H.D., Shea, D.O., and Groisser, D. (2002), Non-Rigid Shape Comparison of Plane Curves in Images, J. Mathematical Imaging and Vision, Vol. 16, no. 1, pp. 57-68, . [SKK03] Sebastian, T.B., Klein, P.N., and Kimia, B.B. (2003), On Aligning Curves, IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 25, no. 1, pp. 116-124. [BMPOl] Belongie, S., Malikl, J.,and Puzicha, J.(2001),"Shape Contexts: A New Descriptor for Shape Matching and Object Recognition," NIPS, Vol. 13,831837

Nonrigid Correspondence and Classification of Curves

407

[FB03] Frenkel, M. and Basri, R. (2003), Curve Matching Using the Fast Marching Method, Lecture Notes in Computer Science , Vol. 2683, pp. 35-51. [SS87] Schwartz, J.T. and Sharir, M. (1987), Identification of Partially Obscured Objects in Two and Three Dimensions by Matching Noisy Characteristic Curves, Intl J. Robotics Research, Vol. 6, no. Z, pp. 29-44. [JWGOO] Jacobs, D.W., Weinshall, D,, and Gdalyahu, Y. (2000), Classification with Nonmetric Distances: Image Retrieval and Class Representation, IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 22, no. 6, pp. 583600. [Uhl91] Uhlmann, J. (1991), Satisfying General Proximity/Similarity Queries with Metric Trees, Information Processing Letters, Vol. 40, pp. 175-179. [MV93] Marzal, A. and Vidal, E. (1993), Computation of Normalized Edit Distances and Applications, IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 15, pp. 926-932. [LS90] Liu, H.C. and Srinath, M.D. (1990), Partial Shape Classification Using Contour Matching in Distance Transformation, IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 12, no. 11, pp. 1072-1079. [TY85] Tsai, W.H. and Yu, S.S. (1985), Attributed String Matching with Merging for Shape Recognition, IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 7, no. 4, pp. 453-462.