Acta Numerica 1999 (Volume 8)

— sin 0 cos 6 u 2v

Inverting the Laplace transform we reach our final form:

3u dp

Ui . 2v

„

n

^

(du ydt

U? \ 4v J

{

^

1 R = 0,

(2.46)

where

P{t) = (nvQ-We-Q,

§i(R,t) = C

Due to the presence of the irrational function of s in the argument of §1, (2.46) cannot be directly expressed as a local operator. We note that the special case of the heat equation is recovered by setting U = 0. For the heat equation, however, fast methods for evaluating the solution operator are available (Greengard and Lin 1998), which may lead to more efficient methods of solution in most cases. The Schrodinger equation The Schrodinger equation is denned by .du = -,2W -i— dt

74

T. HAGSTROM

Clearly, representations of exact boundary conditions in transform space may be obtained from those above by setting U = 0, replacing s by —is and choosing appropriate branches. For the case of a planar boundary, define ^J—is + \k\* so that, for Re s > 0,

Re U-is

+ \k\2) > 0 .

(2.47)

Then, on the planar boundary x = 0 we have, in analogy with (2.45),

Tu

= _ i ^ _ ot

v

v

y

w = e-*ifcia*+W4(7rt)-i/2_

On the spherical boundary, choosing \/—is according to (2.47), we adapt (2.46) and find

where

T/ie linearized incompressible Navier-Stokes equations As our final example of an equation of mixed order, we consider the construction of exact boundary conditions at a planar boundary for the NavierStokes equations linearized about a uniform flow. Again we shall see that our general techniques apply, and that the temporally nonlocal operators that arise are the same which are needed in the case of the advection-diffusion equation. (For alternative constructions in each case see Halpern (1986), Halpern and Schatzman (1989).) We thus consider

du o — + U u + Vp = vV2u, in the tail H C M3 denned by x > 0. We make no restrictions on U\ so that there may be either inflow or outflow at the boundary. Following our standard construction, we perform a Fourier-Laplace transformation and make the system first order in x. Here we use the divergence constraint to

75

RADIATION BOUNDARY CONDITIONS

eliminate dui/dx, leading to a system of six equations: / dw —— = Mw, ox

w=

P \

U3

9a;

U

—s

0 0 0 i&2

0 0 0 0

\ik3 where s = s + iUtan

-ik3 0 0 0

0 0 s

0 0

s

0 k + v\k\2.

The eigenvalues of M are 2v

with 7 satisfying Re 7 > 0 when Re s > 0 is defined by 7=

It is natural to define exact boundary conditions by the left eigenvectors associated with A i ^ which may be given by

12

=

13

=

ik2, - 2

C/1+7 ik3,~2 ik3vs

,s-vkl,-vk2k3,

Ui+1

, s - vk%, 0, ^

,0 , 7

However, when s = u\k\2 — Ui\k\, Ai = A2,3, and we find that q\ is in the span of <72,3- Therefore we replace it by a linear combination of the three eigenvectors which remains independent of (72,3:

- (l, -2(1 + v\k\W+),ik2v{l where z = Ifc^s and =

+ zW-),ik3u(l

+ 2W-),0,0) ,

76

T. HAGSTROM

Introducing the operators

H = F-\\k\-lFu),

N=(^

+ Utan-V- ^Vt2an ; ,

the kernels — e

which we recognize from our study of the advection-diffusion equation, and the temporally nonlocal operator = Jr~l * (Tu)), the exact conditions in the time domain are expressed by

^

dy

+^ )

dz J

= 0, (2.48)

-

"

Construction of exact conditions on a spherical boundary has not, to our knowledge, been carried out. A closer study of the exact conditions in the planar case is quite suggestive of how these would look. Note, in particular, that (2.48) is related to the exact condition for the Laplace equation satisfied by p, namely px + H~lp = 0. The other two conditions are related to the advection-diffusion equation for the vorticity. As the exact conditions on a sphere for the Poisson equation and the advection-diffusion equation are easily formulated, it is reasonable to believe that an exact condition for the Navier-Stokes equations can be similarly found and will involve the same nonlocal operators. Exact boundary conditions for the compressible Navier-Stokes equations can be found using the same techniques as discussed by Halpern (1991) and Hagstrom and Lorenz (1994). In that case, the number of boundary conditions is different at inflow and outflow boundaries. In addition, nonlocal operators associated with the wave equation are involved. 3. Approximations and implementations Having now completed an exhaustive study of exact boundary conditions for a wide class of problems of physical interest, we turn to the problem of efficient implementation of or approximation to the nonlocal operators appearing in our formulations. I will restrict attention to the scalar wave equation. As we have seen, exact conditions for most other important hyperbolic systems involve the same pseudodifferential operators, so the techniques we develop will be applicable in all these cases.

RADIATION BOUNDARY CONDITIONS

77

As mentioned earlier, it was generally believed that the direct implementation of the exact conditions is prohibitively expensive, a belief which discouraged their study. Considering flop counts, this belief is false. Hairer, Lubich and Schlichte (1985) present an algorithm for the fast solution of convolutional Volterra equations. Its application to the exact boundary conditions in integral form yields a method for which the computational effort is smaller than that required by the interior solver, except for unusually long times. However, this approach does require the storage of full time histories at the boundary, which is excessive for moderately long time simulations. The primary alternative to direct implementation of the temporal integrals is the use of approximations to (or in the spherical case representations of) the kernels by sums of complex exponentials. For such kernels, convolution is equivalent to the solution of differential equations, so that the necessary work per time-step and storage is proportional to the number of exponentials used. In the next few sections I will develop the basic error estimates for such approximations and consider some examples. The same theory can be used to analyse the error associated with sponge layers, and I will do so for the so-called perfectly matched layer (PML), a reflectionless sponge layer recently introduced in computational electromagnetics. 3.1. Convolution with sums of complex exponentials Consider the problem of computing Ou = H-1 (Ei * (Hu^), where now H is any of the spatial harmonic transforms from the preceding sections, I is the harmonic index, * is temporal convolution, and E\ takes the form j

<0.

(3.1)

3=1

Note that Ei(s) is a rational function of s of degree (n/ — 1, n{). There are two distinct and useful ways to represent E\. The first is as a sum of poles, which is directly derivable from (3.1):

Then, for any function w(t),

'£lj(t),

(3.2)

78

T. HAGSTROM

where the y satisfy the differential equations dt

4>ii(0) = 0.

"

The second representation is as a finite continued fraction,

which is terminated by the condition %m+i = 0. Then, following Xu and Hagstrom (1999), Hagstrom and Hariharan (1998), we may evaluate E\ * w in recursive form. In particular, set >

J

i

k

I

l

k

lk

where En = E\ and E^ni+\

= 0. Set

Wk = Elk * wk-li

w

0

=

w

,

Wni + l

=

0.

Then we have dwk Ot

= l,...,ni.

(3.4)

Hence, for each representation, we must introduce rii auxiliary functions for the Zth harmonic for a total of N 1=0

auxiliary functions where N is the number of harmonics used to represent the solution. The work per step and storage associated with the convolution is thus proportional (with a small constant) to 7V"a. Note that we must also apply the harmonic transform, 7i, and its inverse at each step. In some instances this simply involves fast Fourier transforms, while in others we require spherical harmonic transforms using, for instance, the methods of Mohlenkamp (1997) and Driscoll, Healy and Rockmore (1997). In the former case the work is O(N In N), but in the latter we have 0(iVln 2 N). In special cases, the harmonic transform phase of the application of the boundary condition can be avoided. This occurs when the constants Q/J, fyj or jij, 6ij are eigenvalues of differential operators with eigenfunctions given by the Zth harmonic. The most important example of this is the case of the exact boundary condition at a spherical boundary. Then (2.19) has the

RADIATION BOUNDARY CONDITIONS

79

form (3.3) with s replaced by Rs/c and la

8

lj

1(1 + 1) = —g '

=

3-

Recalling that we see that for A; = 1 , . . . , N (3.4) takes the form

with w;o = u/2, u>./v+i = 0. Then, if we assume

we have

that is, w\ is precisely the nonlocal part of (2.18). (Note: Wj as defined here differs from that in Hagstrom and Hariharan (1998) by a factor of (—I)-7.) The recursive form above can also be modified for use in approximating the nonlocal terms in (2.22), as discussed by Hagstrom and Hariharan (1998), though the approximation is no longer exact. For different approaches to implementing (2.18), see Sofronov (1999) and Grote and Keller (1995, 1996). As these involve spherical harmonic transforms, I expect the formulation given here will be somewhat more efficient, and certainly much easier to implement. The derivation of the continued fraction form was inspired by the reformulation by Barry, Bielak and MacCamy (1988) of asymptotic boundary conditions based on progressive wave expansions first suggested by Bayliss and Turkel (1980). See Hagstrom and Hariharan (1996) for more details. A different approach to applying approximate boundary conditions, based on localizable, homogeneous, rational approximations to the transformed representation of the exact boundary condition on a planar boundary (2.12), is developed in Higdon (1987, 1986). In particular, suppose that we approximate IM2

9

n

.

80

T. HAGSTROM

which is a general form for a homogeneous approximation that is localizable in time and space. Set

A 2 = s2 + \k\2, and note that, if u is a solution of the wave equation, 2

.

d2ti

Replacing |fc|2 by A2 — s 2 leads to a boundary condition of the form Q(s,X)u

=

0,

Multiplying through by the denominator and factoring the result we find that J[

rjjS + X)u = 0,

which is equivalent to

Stably implementing boundary conditions in this form is a reasonably simple matter. However, it may not be feasible to choose q very large (as is my intent) due to the growth of the difference stencil into the interior domain, so that I generally use auxiliary functions as in (3.2) or (3.4). Starting from (3.7), the parameters rjj, which typically correspond to cosines of incidence angles of perfect absorption, may be adjusted directly. This formulation has been applied in a number of more complex settings by Higdon (1991, 1992, 1994).

3.2. Stability and consistency Error estimates are derived, as always, by establishing the stability and consistency of the approximate boundary conditions. Let us begin with the simplest case of a planar boundary and an approximate boundary condition defined by

s + Vs + Fr Assume further, for simplicity, that T is the half-space x < 0. Then the Fourier-Laplace transform of the exact solution, u, and the error, e, take

RADIATION BOUNDARY CONDITIONS

81

the form ft = Ae-^+^x,

e=

with the amplitudes related by (3.8)

(Generally, for more realistic choices of T, e must satisfy homogeneous boundary conditions on S, and so has a more complicated form. However, similar error estimates can be derived.) By Parseval's relation we find, on bounded subsets T ' c T : INIL3(O,T;L2(T'))


( r 2T)(fc)r / e

\J

2

s u p \e\ \\u(0,k,-)\\l2i0!T)dk)

Res=t)(fc)

V/2

.

/

To bound |e|, we must derive an upper bound on its numerator (consistency), and a lower bound on its denominator (stability). It is interesting to note that if we replace the approximate boundary condition by the exact Of course boundary condition, the denominator has zeroes at s = the numerator is identically zero in this case, so the error is indeed zero. However, we expect that accurate approximate conditions will have small denominators near these points. A simple sufficient condition for stability is Rei?>0, Res>0. (3.9) This can be relaxed somewhat, as will be seen in one of the examples. Ideally, we would take r\ = 0 so that our estimates are uniform in time. However, stable, homogeneous spatially localizable conditions generally have poles on the imaginary s axis. (See Trefethen and Halpern (1986, 1988).) At such points |e| > 1, so that no useful estimate holds. Therefore, for spatially local conditions we generally must settle for finite time estimates. In numerical calculations we can only treat functions with wave numbers in some bounded set, |fc| < M. Our error analysis simplifies somewhat if we restrict our attention to this set. The accuracy of any method is then characterized by 6(T,M;R)=

sup inf sup eT/r|e|. 7

(3.10)

?S0

Alternatively, we can seek estimates involving derivative norms of the solution, leading to error estimates in terms of sup inf sup In this exposition I will follow the former, simpler approach. For examples of the latter see Hagstrom (1995, 1996) and Xu and Hagstrom (1999).

82

T. HAGSTROM

The analysis outlined above is easily extended to our other special boundaries. For example, suppose T is the sphere of radius R and the approximate boundary condition is defined by

Then =

ei(s,R)ul{s,R)JV PP h+\/2{Rs)

where //+1/2 i s the modified Bessel function of the first kind (Abramowitz and Stegun 1972, Ch. 9), and e; is given by C.

T/.

r-

(3.H)

The accuracy of the method may then be characterized by the obvious analogue of (3.10). In the following sections I will estimate 8 for various approximations. 3.3. Pade approximants and

generalizations

For planar boundaries the function R(s, \k\) approximates the function

with z = s/\k\. Let Rp(z) be some approximation to K. Noting that J\

==

2z + K

it is reasonable to set

Approximations based on (3.12) are extensively analysed in Xu and Hagstrom (1999). It is clear that they are in continued fraction form, and hence implementable via the recursion (3.4). Precisely, \k\2 71 = - g - .

6j = 0,

\k\2

.

7 j = -^-1

j = 1,...,

k =

raj_i,

Note that Rn+1 ~K =

(2z + K)(2z

j,...,n

h

£„, = |/c|^o.

RADIATION BOUNDARY CONDITIONS

83

Therefore, if we assume that Rp is at least bounded as z —> oo, we gain two orders in the large z approximation at each step. That is,

k - k \ = o(z-2)\Rp-k\. Consider the initialization RQ

= a > 0.

Then an easy induction argument shows that KeRp > 0 when Re 2 > 0 and that all poles are in the closed left half-plane. For a > 0, we can further conclude that the poles and zeroes of the real part lie in the open left half-plane. The choice a = 0, however, leads to the Pade approximants introduced by Engquist and Majda (1977) and Lindman (1975). These are spatially localizable, with poles on the imaginary z axis between . Xu and Hagstrom (1999) show that Rp is given by the following explicit formula: S^/WTW ~ b2n+1 - (-l)nb + a(b2n + (-1)"6 2 ) which leads to the remarkably simple formula for e: (3.13) From this we derive the following theorem. Theorem 1 Let a > 0 and ry, M > 0 be given. Then, sup

|e| < (1 + a)

Res=T],\k\<M

The proof of this theorem follows immediately from the inequality inf |6| > Jl + 2fi2. v

Re z=f]

From the estimate it might be concluded that the Pade approximants, o = 0, are optimal in this class. However, we have proceeded crudely. The minimum of |6| leading to the inequality above occurs at Imz = 0 where we can force e = 0 by a proper choice of a. A more detailed analysis is given in Xu and Hagstrom (1999). Imposing a tolerance r and a computation time T we require 6(T, M) <

T,

which implies _

1

In

^+VT

84

T. HAGSTROM

Optimizing this over r\ yields (for cMT on the order of In 1/r) the following result. Theorem 2 For any 0 < a < 1 there exists a constant, C, such that for any tolerance 0 < r < 1, wave number bound M, and time T > 0, the approximation Rp with p > C In - + V T satisfies 6(T, M; Rp) < r. We see that the number of terms required is weakly dependent on the tolerance, but strongly dependent on the time and the tangential wave numbers. A completely different convergence analysis for the Pade approximants was given in Hagstrom (1995), with the same conclusions. Many other space-time localizable conditions are proposed in Trefethen and Halpern (1988), whose accuracy has not, to my knowledge, been estimated.

3.4- Truncations of (2.19) and asymptotic boundary conditions A second approach to the construction of local approximate boundary conditions has been through the use of the progressive wave expansion, given in the cylindrical case by r

~3~*fj(ct-r,9),

(3.14)

j=o

and in the spherical by oo

u~^2r~j~1fj(ct-r,0,
(3.15)

j=o

where for notational convenience we now use r instead of p to denote the spherical radius. (For a mathematical discussion of expansions of this type see Ludwig (I960).) These are used by Bayliss and Turkel (1980) to construct a hierarchy of boundary conditions satisfied by truncations of the expansion. This is easily accomplished using normal derivatives: (Biu))) = 0, 1 d

d

dt^dr^

R

where a = 1/2 for (3.14) and a = 1 for (3.15). However, the product form limits the order, p, which can be practically used. An alternative proposed by Hagstrom and Hariharan (1998) is to use the continued fraction form

RADIATION BOUNDARY CONDITIONS

85

(2.19) or its cylindrical analogue, which has the form (3.3) with s replaced by Rs/c and I2 - 1/4 ^ ,

7a

=

llj

~

Sij

= j.

4

Here, I is the dual Fourier variable to 6. Truncating the expansions after p — 1 terms, that is, setting wp = 0, leads to an approximate boundary condition whose accuracy may be assessed, in the spherical case, via (3.11). We have not carried through this analysis in full detail, as in the preceding section. However, we can make some conclusions. Assume I S> p. We then rely on the uniform large index asymptotic expansions for Bessel functions developed by Olver (1954). In particular, we find, to leading order, that away from the transition zones Rs « SkjjRs) ki(RS) which obviously corresponds to the planar case. By direct computation we see that the approximate boundary condition poorly approximates the exact condition when ^ » |5|.

(3.16)

For \s\ » l/R, on the other hand, the approximation is good. Here, instead of increasing the order of the boundary condition, we can expand the domain, moving the boundary to -yR. Evaluating the error on the original domain only, we see that our expression for e picks up an extra factor of Kl+l/2{R~s)

'

Il+1/2{-yRSy

Assuming (3.16), this factor is approximately 7~ 2 '. Therefore, we may hope that the error is small if 7~2? is small, which requires

i Clearly, this argument is far from a proof. In particular we have completely ignored the transition regions. Their analysis might introduce some time dependence in the estimates, as in the similar case of the Pade approximants. We find that, with these favourable assumptions, some improve-

86

T. HAGSTROM

ments can be made by combining domain extension with increases in p. It is therefore of some interest to carry through the convergence analysis.

3.5. Uniform rational

approximants

The analysis above points to a defect of the continued fraction approximations, namely, poor approximation properties for tangential wave numbers which are large in comparison to s. This suggests that substantial improvements can be made by uniformly approximating the transforms of the exact boundary kernels along lines Re s = rj > 0. This program is carried out in Alpert et al. (19996). It consists of two parts: proofs using multipole theory that good approximations exist, followed by the numerical construction of the poles and coefficients via nonlinear least squares. The fundamental approximation theorems used, which follow from the methods of Anderson (1992) and are proven in Alpert et al. (1999&), can be summarized in the following form. Theorem 3 Suppose Di, i = 1,... ,p are disks in the complex plane of radius ri and centre Cj. Suppose the complex numbers Zj, j = 1,..., n, and curve, C, lie within the union of the disks and that the function f(z) is defined by

Then there exists a rational function gm(z) with mp poles, all lying on the boundaries of the D{, such that for any z satisfying Re (z — Cj) > ari > rj,

where

The proof follows from the direct construction of gm. One simply places poles symmetrically around the boundary of the disks with coefficients chosen to match the large z expansion of each disk's contribution to / . The application of Theorem 3 to the approximation of Si (z) is quite direct. As Si is a rational function, it can be written as a sum of poles. These poles are zeroes of Ki+i/2{z), and uniform large I expansions of their locations are given in Olver (1954). In particular, they lie near a curve in the left half £-plane connecting the points z = with the poles nearest the imaginary axis separated from it by an O(l1/3) distance. We cover these poles by O(lnZ) disks such that all points in the closed right half z-plane satisfy the

RADIATION BOUNDARY CONDITIONS

87

Table 1. Number of poles, p, required to approximate Si, I < M, with S(T, M)
10" 6 10" 8 lO"15

M = 128 M = 256 M = 512 M = 1024 12 16 26

14 17 29

15 19 33

16 21 36

inequality Re (z — a) > 2r\. Using m O(lnZ) we thus achieve an error that scales like 2~m. Taking into account the behaviour of the denominator of (3.11) near z = we deduce that, for some /-independent constant C, \ei\ < Cl^^T™. Hence we have the following theorem. Theorem 4 There exists a constant C such that, for any tolerance 0 < r < 1, wave number bound M > 1, radius R > 0, and time T > 0, there exists a p-pole rational approximation P; (z) with p
such that 6(T,M;Pi) < r. Note that the number of poles required is bounded independent of T. We have numerically constructed approximations satisfying Theorem 4. The number of poles required as M and r are varied are listed in Table 1. One cannot but be impressed by the efficiency of these approximations, which allow the evaluation of the exact condition for harmonics of index up to 1024 to double precision accuracy with no more than 36 poles per harmonic. It is also possible to apply Theorem 3 to the approximation of the transform of the cylindrical kernel, Cj, and the planar kernel, K. In each case we use integral representations of the form given in the theorem. Details in the cylindrical case for kz = 0 are given in Alpert et al. (19996), while the planar case will be discussed elsewhere. Both CQ and K have branch points on the imaginary axis. This forces us to settle for approximations with Re s > rj > 0. The number of poles required will thus depend on the time, T, as well as on M and r. For the planar case we have the following theorem. Theorem 5 There exists a constant C such that, for any tolerance 0 < r < 1, wave number bound M, and time T such that cMT > 2, there exists

88

T. HAGSTROM

a j?-pole rational approximation R(s, k) with p
\ncMT

\

-(]n-

r

such that S(T, M; R) < r. The planar approximations are computed by specifying r and r\ = {MT)~l. Choosing r = 1CP3 and r\ = 10~4 leads to a 21-pole approximation which is used in the numerical experiments below. 3.6. Reflectionless sponge layers An alternative to the imposition of radiation boundary conditions at the artificial boundary is to surround the computational domain T with a sponge layer or absorbing region, within which propagating waves are damped. Though the construction of layers that absorb wave energy is reasonably simple, additional errors are typically introduced by the interaction of waves with the interface between the computational domain and the layer. Recently this approach was revitalized by the construction in Berenger (1994) of a sponge layer for Maxwell's equations with a reflectionless interface: the so-called perfectly matched layer, or PML. As shown in Abarbanel and Gottlieb (1997), the original formulation is only weakly well-posed. Petropoulos (1999) gives a clear mathematical derivation of reflectionless sponge layers in Cartesian, cylindrical and spherical coordinates, and derives strongly well-posed formulations. Surprisingly, much less has been written about reflectionless sponge layers for the wave equation. Here I adapt the construction of Chew and Weedon (1994) and Petropoulos (1999) to the wave equation and analyse the error, restricting myself to the planar case. The error estimates thus derived coincide with those derived for Maxwell's equations by the same techniques. Take x to be the coordinate normal to the layer interface and suppose that T corresponds to x < 0. The simplest starting point for the analysis is at the level of the solutions in T, described after our usual Fourier-Laplace transformation by Clearly, these solutions are not damped with increasing or decreasing x for imaginary s satisfying |s| > |A;|, that is, for propagating modes. Damping may be achieved by modifying the exponent so that it has a real part that decreases with increasing x for the right-propagating (—) mode, and increases with increasing x for the left-propagating mode. As the sign of the imaginary part of y/s2 + \k\2 coincides with that of s in the propagating mode regime, this is accomplished by adding to x an increasing in x imaginary function whose imaginary part has the opposite sign from that of s. A

RADIATION BOUNDARY CONDITIONS

89

simple function with this property is

So *(w) dw

~>n

s Thus we seek solutions in the sponge layer in the form *( The interface x = 0 will be reflectionless if the solutions and their xderivatives coincide there. This imposes the additional constraint cr(O) = 0. This constraint is not present in applications to hyperbolic systems, but is almost always imposed in computations. No other conditions on a are needed to make the interface reflectionless. From the layer solutions it is straightforward to derive a pseudodifferential equation. For u we have o, s2u =

s d ( s du\ ,, ,o— —- - \k\2u. s + a ox \s + b ox J

Inverting the transforms, introducing a = co, and assuming zero initial data in the layer, we finally obtain

where

=f

I *w =

Jo Local implementations involve the introduction of auxiliary variables to eliminate the convolutions. It is here that strong well-posedness can be lost. In our numerical experiments we replace (3.17) by dv -v>, ax

1

i

2


~di

+

i

av = a

du dx

(3.19)

dv d2u (3.20) 2 dx dx These are strongly well-posed, but possibly not asymptotically stable, so that other reformulations may be better. To complete the layer description we must specify its length, d, and impose a boundary condition at its edge. Here I make the simplest choice, + crw = a

90

T. HAGSTROM

u = 0, though of course one could lower the error with more sophisticated conditions. The general solution within the layer is then given by U = At (e-V^ + lWttx-V+s-1 f* a(w)dw) Matching this to a solution in T of the form

we find that E is related to A as in (3.8) with e(5,|*|) = We see that, as in the case of rational approximants to K, good error estimates do not hold along the imaginary s axis, particularly near s = Taking Re s = r\ > 0 and introducing a = d-1 Jo a(w) dw, we must estimate

min Re hdVz2 + l(|fc| + z^a)) .

(3.21)

Consider the case of f] = 4h
+ C 2 )),

Re

|CI < 1,

C~

-

Hence, for 77 sufficiently small, the minimum is achieved near |£| = 1. Restricting 77 to be no greater than one, we have, for some constant C, g < eiic\k\T-Cdy/rj(\k\+ir) _

Minimizing over fj and maximizing over \k\ we prove the following. Theorem 6 There exists a constant C such that, for any tolerance 0 < r < 1, wave number bound, M, and time, T, the ideal reflectionless sponge layer (3.17) with average absorption a > 0 and width, d, satisfying

da>C\ \fcEf + Win -

in -, T

will have S(T, M) < r. A remarkable feature of this bound is its independence of the maximum wave number, M.

RADIATION BOUNDARY CONDITIONS

91

3.7. Complexity Armed with these error estimates, we are in a position to assess the relative efficiency of the various accurate approaches. We consider two distinct idealized problems governed by the wave equation, with units chosen so that c = 1. The first problem assumes a three-dimensional computational domain T that is 1-periodic in two coordinate directions, has length 1 in the third direction, and is truncated by an artificial boundary at each end. The second problem assumes that T is contained within a sphere of radius 1, which serves as the artificial boundary. The first problem is used to test conditions for periodic or waveguide problems and the second to test conditions for exterior problems. In practice, planar boundaries are used to solve exterior problems, enclosing the computational domain in a box. Unfortunately, we do not as yet have any hard error estimates for either the Pade approximants or reflectionless sponge layers used in this way, and so can only make conjectures concerning their efficiency. In each case we assume that wave numbers up to M must be resolved on the boundary and that we are interested in the solution up to time T. We also suppose that the error tolerance is r. For purposes of comparison it is useful to note the work, Wj, and storage, Si, required by the interior solver. Assuming an explicit method with reasonable stability constraint, these are Wj oc aAM4T,

Si OC a3M3,

where a is the number of points per wavelength. Appropriate values for a are strongly dependent on the order of the method and may also depend on T. It will also be proportional to r~llp for a pth order method. In what follows I will suppress the a dependence, but it should then be kept in mind that in some instances a can be fairly large, and some methods will allow coarser representations when evaluating the boundary conditions. Similarly, we will treat logarithmic dependences on the tolerance as 0(1) constants. I have tried to keep the analysis as simple as possible, ignoring possible improvements in the complexity estimates that might be achievable by better implementations. Of course I hope that in the future this analysis will be supplemented by serious computational experimentation. Domain extension By far the simplest way to achieve an accurate solution is to exploit the finite signal speed and extend the domain so that the boundary cannot influence the solution in T for times less than T. Clearly this requires an extension of width O(T). For the first problem the volume of the extended region is proportional to T so that the extra work and storage, which we shall always denote by WB and SB, is given by WB OC M 4 T 2 ,

SB OC M3T.

92

T. HAGSTROM

We see that the work and storage exceeds that required by the interior scheme by a factor of T, which is clearly unacceptable for moderate to large times. The results for the second problem are even worse, as the volume of the extension is proportional to T 3 , that is, WB oc M 4 r 4 ,

SB oc M 3 T 3 ,

a factor of T 3 above the interior scheme. Kirchoff 's formula Here we assume two spherical boundaries separated by a small distance. To compute the solution at each point on the outer boundary requires the computation of an integral over a sphere. This involves O(M4) work per time-step. Although it is reasonable to assume that the integration can be carried out on a coarser grid than required by the solution of the wave equation, so that the constant of proportionality may be small, the order estimates are SB oc M 3 . WB oc M5T, We see that the storage required is comparable to (probably less than) Si but that the work is greater by a factor of M. Direct implementation of the planar exact condition Here we require direct and inverse Fourier transforms at each time step as well as the solution of the convolutional Volterra equation for each mode. Making use of the FFT, the work associated with the transformations is seen to be O(M2 lnM) per time-step. For the convolution we may use the algorithm presented in Hairer et al. (1985), which requires O(MT In2 MT) operations per mode. As for storage, the direct implementation requires full storage of the time histories of the Fourier coefficients. This could probably be reduced somewhat using the t~3!2 decay of the convolution kernel, K(t), but I have not quantified the effect. Therefore we have WB oc M3T In2 MT,

SB oc M3T.

Except for extraordinarily long times, WB compares favourably with Wj. However, we generally have SB > Si, possibly much greater for large T. Direct implementation of the spherical exact condition Here I will consider the completely local version (3.5). The implementations of Grote and Keller (1995, 1996) and Sofronov (1999) require an additional direct and inverse spherical harmonic transform per time-step. Using the fast algorithms of Driscoll et al. (1997) and Mohlenkamp (1997), this requires

RADIATION BOUNDARY CONDITIONS

93

O(M2 In2 M) work per step and does not increase the order of the complexity estimate. We require, then, the solution of M additional equations on the boundary, associated with M auxiliary variables. This costs WB

OC M4T,

SBOCM3.

Here we have, taking account of the probable smaller proportionality constants due to the effect of a, WB < Wi, SB < 5/: the first method we have seen that meets our goals of arbitrary accuracy without increase in cost! Pade approximants

Suppressing the weak dependence on the error tolerance, we require O(MT) auxiliary functions and equations on the boundary. The cost then is WB OC M4T2,

SB OC M3T.

This is more by a factor of T than what is required by the interior solver, and so is unacceptable for long time computations. We note that in our numerical experiments the growth in the number of terms required as T increases was fairly mild, so that we expect the proportionality constants to be small. High-order conditions based on the Pade approximants have been implemented on rectangular domains. This depends on remarkable constructions of corner compatibility conditions given by Collino (1993) and Vacus (1996). It would be of interest to extend the error estimates to this case. It is conceivable that they will be somewhat better, as glancing modes which reflect off one boundary may be effectively absorbed at near normal incidence by another. Asymptotic boundary conditions In this case we have no proven error estimates. Accepting the optimistic assumption that we must expand the domain by a factor of T~l^2p\ we find WB OC ( T ~ £ - 1)M 4 T + pM3T,

SB oc ( r " i - 1)M 3 +pM2.

These estimates are optimized by poc which leads to work and storage estimates that are better than those obtained for the exact condition, p = M, in some cases. Clearly it would be of interest to make the error estimates precise.

94

T. HAGSTROM

Uniform rational approximants These have been constructed for both planar and spherical boundaries, and hence are directly applicable to both problems. In the first case we require O (In2 MT) auxiliary functions per mode on the boundary, and direct and inverse Fourier transforms each time-step. Therefore the work and storage required are WB oc M3T In2 MT,

SB oc M 2 In2 MT.

Except for extraordinarily long times, we have WB <S WJ and SB
SB oc M 2 In2 M.

Theoretically, then, the uniform approximants represent a completely acceptable solution to the boundary condition problem in the constant coefficient case for exterior problems, as they provide essentially arbitrary accuracy for arbitrary times with WB -C WJ and SB ^ Si. There are some practical issues concerning the efficiency of the fast spherical harmonic transforms and the necessity of using an aspect ratio one computational domain. Reflectionless sponge layers Here there are two parameters, the layer width d and the average absorption, a. Clearly, the number of mesh points in the layer will scale like dM3 with d in the planar case. Also, following the analysis in Collino and Monk (1998), the mesh spacing must scale inversely with a. Hence we will take a fixed and reduce the error by increasing d. As d oc y/T this implies WB OC M^T3'2,

SB OC M3Tll2,

which is unacceptable for large T. Most practical applications of this technique involve exterior problems with a computational domain which is a box. Therefore, I believe it would be of great interest to extend the error estimates to this case. If the time dependence of the errors were as bad as in the planar case, then the long time behaviour would be even worse, as the volume of the layer would grow like d3 oc T 3 / 2 . However, there is some reason to believe that far better estimates hold. This is due to the fact that glancing waves at one boundary are nearly normally incident to another, and hence should be effectively absorbed.

RADIATION BOUNDARY CONDITIONS

95

An alternative for exterior problems is the spherical layer developed in Petropoulos (1999). I believe that this layer could be directly analysed as a straightforward extension of the analysis given here. However, for problems of near unit aspect ratio, it is doubtful that a layer technique could be more efficient than the uniform rational approximants. 3.8. A numerical example We now consider a simply described yet, as we shall see, difficult-to-solve concrete problem to illustrate some of our results. In particular, we solve the initial value problem for the wave equation in the planar region (x, y) 6 (—1,1) x (0,1), assuming periodicity in y. An exact solution, u(x,y,t), is constructed by setting off periodic arrays of pulses at various negative times. This leads to 11 ( T* II ~t\ U/\*JUy y ^ V I

where

00

\ 7

7/ - i T* ?/ 1*2 V"^5 M i

rt-n

i(x,y,t)= Yl /I m(x,y,t)= Y, , fc=-c

'/"I /5

ds

'

J—oo

and

rl = (x- Xi)2 + (y-yi-

kf,

*i(s) =

Aie-^s-^\

Here, Tj < 0 is chosen so that A{e~iHTi is negligibly small. At t = 0 the solution is made negligibly small (to more than 11 digits) outside T. A program that accurately evaluates u using high-order Gaussian quadrature and high-order end-point corrected trapezoidal formulas for singular integrals was generously provided by Leslie Greengard, and is used to produce the error tables listed below. I am confident that the accuracy of the evaluation is on the order of ten decimal digits and, hence, far exceeds that of the numerical solutions. These solutions were used to test Pade approximants of K in numerical solutions of the linearized Euler equations (Hagstrom and Goodrich 1998) and will be used in the extensive numerical experiments to be presented in Alpert et al. (1999a). Here we consider a single pulse with parameters: Hi = 150,

n = --,

xi = 0,

yx = - ,

Ai = 1.

Three types of approximate boundary conditions were considered: the Pade sequence, using (3.12) with RQ = 0, and its generalization, (3.12) with RQ — 1; a strongly well-posed local implementation of the PML; and the planar uniform approximant computed with e = 10~3 and 77 = 10~4. The latter . We employs 21 poles. The approximate conditions are imposed at x —

96

T. HAGSTROM

Table 2. ,Errors using (3. 12) Ro

0 0 0 0 0 0 1 1 1 1 1 1

P

0 5 10 15 20 25 30 5 10 15 20 25 30

Max. err. t < 5 Max. err . t<25 1.2

9.9 1.2 2.6 1.5 1.5 1.5 8.9 6.6 1.6 1.5 1.5 1.5

x 10- 1 X X X X X X X X X X X X

3

10" 10" 3 10" 4 10" 4 lO- 4 10" 4 10" 3 10" 4 10" 4 lO- 4 10" 4 10" 4

2.1 x 10"1 7.0 x 10"2 3.3 x 10"2 1.9 x 10"2 1.0 x 10"2 6.4 x lO- 3 3.5 x 10"3 6.9 x io- 2 3.2 x 10"2 1.7 x 10"2 8.8 x lO- 3 4.9 x 10"3 2.5 x lO- 3

Max. err. t < 50 2.1 7.6 5.5 3.7 2.9 1.9 1.4 7.7 5.5 3.5

x 10"1 xlO- 2 xlO- 2 xlO" 2 xlO" 2

X IO- 2 X 10" 2 XlO-2 X 10" 2 X 10" 2 2 2.6 X 10" 2 1.8 X 10" 1.1 x 10"2

use a fourth-order explicit two-step method as our basic solver on a 200 x 100 uniform mesh. This provides a relative error less than 10~3 for 0 < t < 50. The boundary conditions and/or layer equations were also approximated to fourth order. In all cases we compute the relative L2-errors on a uniform 50 x 25 mesh. Complete details on our discretization techniques will be given in Alpert et al. (1999a). Pade approximants and generalizations Results of these experiments are summarized in Table 2. I ran experiments with p = 0 — 30 and the initializations RQ = 0 and Ro = 1. The results are consistent with the error estimates. Generally, the largest errors occurred near t = 50, and we are unable to achieve an error at the level of the discretization error at this late time with p < 30. We are, on the other hand, fairly close to this goal at t = 25. The errors for the second sequence, .Ro = 1, are generally slightly smaller than those obtained using the Pade sequence. This sequence also has the advantage of being exact at steady state. Finally we note that the short time error of 1.5 x 10" 4 is the best one can do for the problem at hand and the mesh I have used. Indeed, it is the same error found if one uses the exact solution as a Dirichlet condition at the artificial boundary.

97

RADIATION BOUNDARY CONDITIONS

Table 3. Uniform rational approximation coefficients, K « YLj r = 10~3, 77 = 10~4. Complex numbers are written in the form (real, imaginary) I 2431987763837349E-6)

(-4998142304334231E-4, ±

.9999998607359947)

(-.1617695923999794E-5, ± . 1638622585172068E-5)

(-.2501648855535112E-3, ±

.9999990907954994)

(-.7723476507531262E-5, ± .7878743138182415E-5)

(-.8021925048752190E-3, ±

.9999958082358295)

( -.34003O4516975200E-4, ± 3510673092397324E-4)

(-2263515963206483E-2, ±

.9999820162287431)

( -.1454893381589074E-3, ± .1535469093409158E-3)

(-.6112737916O31916E-2, ±

.9999224860282032)

( -.6104572904148162E-3, ± .6733883694898616E-3)

(-.1625071664643320E-1, ±

.9996497460330479)

13,14

(-2473202929583869E-2, ± 3011442350813045E-2)

(-.4295328074381198E-1, ±

.9982864080633248)

15,16

( - 8964957513027O30E-2, ± .1398751873403249E-1)

(-.1129636068874967, ± 9907617913485537)

17,18

( -.1846252520037211E-1, ± .6565858806543060E-1)

( -.2902222956062986, ± .9462036470847180)

19,20

( 9181095934161065E-1, ± .2076825633238755)

(-.6548034445533449, ± .7077228221122372)

(.3787484004895032,0)

(-.9345542777004186,0)

1,2 3,4 5,6 7,8 9,10 11,12

(-.2410467618025768E-6, ±

21

Table 4. Errors usmg uniform rational approximant V

Max. err. t < 5

Max. err . t<25

Max. err. t < 50

21

1.5 x 10"-4

4.1 x 10-4

5.3 x 10"4

Uniform approximants For all Fourier modes we use the approximant determined by a tolerance of r = 10~3 and an offset parameter of r) = 10~4. This yields a 21-pole approximation with parameters listed in Table 3. Note that for our mesh we certainly have M < 100 so that 77 < (MT)~l. Hence the error due to the boundary condition is smaller than r. The results, summarized in Table 4, are very encouraging. For all time intervals considered, the accuracy is the best that can be obtained on the given mesh, as determined by using exact Dirichlet data at the boundaries. Given the modest number of poles used, the efficiency of this approach is unparallelled, as predicted by the complexity analysis. We note that the data in this case was generously provided by Brad Alpert. More comprehensive experiments and detailed discussions of the numerical algorithms will be given in Alpert et al. (1999 a).

98

T. HAGSTROM

Reflectionless sponge layer Results of our experiments with the reflectionless sponge layer or PML are summarized in Table 5. I must admit from the outset that the great variety of possible layers, determined by different absorption profiles, widths, and local realizations, makes it difficult to claim that any particular experiment is definitive or comprehensive. Of interest in this regard is the paper of Turkel and Yefet (1998), where a variety of different formulations are compared for the same problem. Equations (3.18)-(3.20) are discretized to fourth order, with an implicit treatment of the absorption terms. We use a quartic rather than the usual quadratic variation of a. Periodic boundary conditions are used at the terminating point. Listed in the table are the maximum value of a, which we varied between 10 and 80, and the total number of points in each layer, ni, which we varied between 25 and 75. For values of crmax too large in comparison with the layer width, in particular for the 25 point layer and 0max = 40, 80, the solution grew and the errors became large. I do not list these results here. The results of this experiment are in basic agreement with the error estimates. This can be verified by checking the ratio da In absolute terms, however, the performance is somewhat disappointing. The best results are achieved for the thickest layer and amax = 40, and almost reach the minimum possible error for t < 5. However, the long time results are worse than those obtained with the other methods. Improvements might be achieved by changing the discretization. I was unable to take a very large, for fixed mesh spacing, without encountering asymptotic instabilities. 3.9. Approximations

in the dissipative case

In the preceding sections we studied many well-developed techniques for approximating radiation boundary conditions in the hyperbolic case. In contrast, much less has been done for equations of mixed type. It is an interesting issue if the highly efficient rational approximants we shall consider can be extended to these cases. As the kernels to be approximated often involve the same analytic functions with arguments restricted to subdomains of the domains of interest in the hyperbolic case, one is tempted to conjecture that it is possible. However, there is the complicating factor that the transforms do not behave like rational functions at infinity. For problems with small diffusion coefficients or viscosities, rational approximants in the spirit above have been proposed and analysed by Halpern (1986) for the scalar advection-diffusion equation, and by Halpern and

99

RADIATION BOUNDARY CONDITIONS

Table 5. Errors using (3.17) ^max

ni

10

25 25 50 50 50 50 75 75 75 75

20 10 20 40 80 10 20 40 80

Max. err. t < 5 3.0 6.2 5.0 5.3 7.7 2.9 8.0 2.8 2.0 4.8

x 10"1 2

Max. err. t < 25

Max. err .

1 3.5 x 10"

4.3 6.3 1.0 8.7 1.7 5.8 7.4 5.7

X lOX 10"2

1.9 1.0

x 10"3 x 10"4 x 10" 3 xlO" 3 x 10"4 x 10"4 x 10"4

6.1 4.8 1.4

6.9 3.2 1.1 2.9

x 10"i x 10"i xlO" 2 x lO- 2 x 10"1 X 10"2 X 10"2 XlO- 2 x 10"2

4.3 1.1

t<50

X

10"1 lo- 1 lo-1

X

lO- 2

X X

X X X

lo- 1 10"2 10"2

X

io- 2 io- 2

X

io-i

X

Schatzman (1989) for the linearized incompressible Navier-Stokes equations. Error estimates in the small parameters are given. It is also possible to make use of the special properties of dissipative problems to derive simple boundary conditions with some provable accuracy. One can exploit the differing decay rates of different modes to identify 'dominant' wave groups in the far field, and use boundary conditions that interpolate the exact conditions at these locations in wave number space. This procedure is developed in Hagstrom (1991a). Asymptotic error estimates are thereby derived, leading to an error, for planar boundaries, which decays in the general case like L" 1 where L is domain length. In Hagstrom (19916) these ideas are applied to the incompressible Navier-Stokes equations in a channel, producing conditions which seem reasonably accurate at moderate Reynolds numbers. An interesting class of problems that can be accurately solved using simple boundary conditions are singularly perturbed hyperbolic systems at boundaries with no incoming characteristics. The simplest model of such a problem is the scalar advection-diffusion equation (2.43) at outflow {U\ > 0) under the assumption ! / < l . We then notice that the 'incoming' solution u = Eex+x,

X =

is of boundary layer type, as

100

T. HAGSTROM

Therefore, any error we make at the boundary very rapidly decays into the interior. From a numerical perspective, however, the boundary layer may lead to large errors for an unrefined mesh - and one certainly does not want to refine an unphysical boundary layer at an artificial boundary. The amplitude of the layer can be decreased through the use of extrapolation conditions. Namely, we impose dru The error associated with this boundary condition can again be explicitly analysed. Using the exact solution near the boundary given in (2.44), we find that the error satisfies

Clearly, for s + iUta.n k + v\k\2 not too large, the error is O(vr) and r xderivatives are bounded independent of v. In the large s, \k\ regime the prefactor is not small, but generally the solution amplitude, A, is exponentially small as its exponent has an O{y~l) negative real part. Precise arguments and error estimates, including general boundaries and variable coefficients, are given in Loheac (1991). Nordstrom (1995),Nordstrom (1997) established the accuracy of extrapolation boundary conditions at supersonic outflow for the compressible Navier-Stokes system. Surprisingly these results are extended to the subsonic case, where there is an incoming characteristic, if there are large transverse solution gradients. This result is applicable when physical boundary layers intersect the artificial boundary. Extrapolation conditions can also be used for the incompressible Navier-Stokes equations as in Johansson (1993). 4. Conclusions and open problems Our fundamental conclusion, amply demonstrated by the theory and fully supported by our as yet sparse numerical experimentation, is that the basic constant coefficient equations of wave theory on unbounded domains with sufficiently simple tails can be accurately solved at essentially the same cost as solving a standard problem on the bounded subdomain of interest. Excepting the case of computational domains of high aspect ratio, which I will discuss further below, the uniform rational approximants seem to provide an ideal solution. In particular, the work associated with their application is generally less than required by the interior solver. However, for spherical boundaries, direct implementations of the exact condition are only slightly less efficient, and, using the local form, extremely easy to implement. For short to moderate time calculations, local conditions based on high

RADIATION BOUNDARY CONDITIONS

101

degree Pade approximants or the use of refiectionless sponge layers are also acceptable. Despite the remarkable progress made over the past few years, there are still many important problems which remain unsolved. Below I will mention those which I think are most important, along with some speculations concerning their possible solution. High aspect ratio domains. For exterior problems, our best techniques require the use of a spherical artificial boundary. In the most extreme cases, for example, scattering from a body with two small dimensions such as a wire, we are required to use a computational domain whose volume is greater by a factor of the aspect ratio squared than the potential domain of interest. This is clearly undesirable. One possible solution for short to moderate times is to use a long cylindrical domain with simple boundary conditions at the ends, but such a procedure becomes inefficient as the time becomes large. More desirable would be the development of efficient approximations to the exact boundary condition on a family of domains that include domains of high aspect ratio. The obvious candidates here are prolate and oblate spheroids, as the wave equation is separable in the associated coordinate system. Prom our point of view, the primary difference between this case and those we have treated is the lack of scale invariance of the boundary. This leads to the dependence on the Laplace transform parameter, s, of the angular eigenfunctions of the exact boundary operator. If we choose to expand in a fixed basis, such as the spherical harmonics, the temporally nonlocal part of the operator is no longer a diagonal matrix. Naturally, this complicates its approximation, but I do not believe that it precludes the existence of effective uniform approximants. Therefore, the detailed study of this operator seems worthwhile. A distinct approach to its elucidation and approximation may be via multipole expansions of the solution. We have seen that these may be used to express the exact condition on a sphere. They have recently been studied for frequency domain problems in the spheroidal case by Holford (1999). Planar boundary conditions on a rectangular box. Another possibility for constructing high aspect ratio domains is the use of rectangular boxes and either the Pade approximants or a refiectionless sponge layer. In light of the bad long time behaviour of our error estimates and computational experiments on periodic domains, this would seem an unlikely solution. However, there are suggestive arguments that better estimates might hold on box domains. Therefore, I believe that a rigorous error analysis for these boundary conditions on boxes should be developed. Anisotropic systems in exterior domains. As mentioned in our discussion of the linearized compressible Euler equations, we have no general construction of exact boundary conditions for anisotropic problems on bound-

102

T. HAGSTROM

aries which include characteristic points. Of course a general theory, which might be based on the Riemann function, would be most desirable. Failing that, a particular solution in the gas dynamics case would have many applications. Such a solution would probably follow easily from the construction of exact conditions for the convective wave equation.

Approximation of exact conditions for problems of mixed order. As we have seen, it is straightforward to develop expressions for exact boundary conditions in these cases. However, the associated theory of rational approximations is essentially undeveloped. Progress on this front could be very useful, particularly for nondissipative problems such as equations of Schrodinger or Boussinesq type. Variable coefficients. Many important problems in wave theory involve propagation in inhomogeneous media. Examples include aeroacoustics, underwater acoustics, and seismics. Unfortunately, the theory as now developed says little about such problems, except in some very special cases mentioned above. A natural starting point is the wave equation in a stratified medium. A reasonable program is to characterize the exact boundary condition and attempt to construct approximations. However, this will again be a case where the eigenfunctions of the exact operator will depend on the Laplace transform parameter, so it is not clear how far one can go with our most successful approach. It may prove easier to construct reflectionless sponge layers, but this is again an open question. It is worth remembering that some of the earliest work on numerical radiation boundary conditions, namely Engquist and Majda (1977, 1979), made use of geometrical optics and, hence, was naturally extensible to variable coefficients. The existence of a limiting operator could even be proven. However, the use of this theory to characterize the accuracy, as carried out by Halpern and Rauch (1987), depends on the assumption that the wave field is dominated by high frequencies. An intriguing alternative, suggested by the results of Radvogin and Zaitsev (1998), is to use a coordinate system in the exterior which is characteristics-based in the hope it will allow for a significant coarsening of the mesh and, hence, extension of the computational domain. Geometrical optics again seems the logical tool for assessing this approach. In comparison with the problems mentioned above, I believe that a satisfactory general treatment of problems with variable coefficients will prove to be the most difficult. Nonetheless, good results just for some special cases, such as stratified media or perturbations thereof, would be quite useful.

RADIATION BOUNDARY CONDITIONS

103

REFERENCES S. Abarbanel and D. Gottlieb (1997), 'A mathematical analysis of the PML method', J. Comput. Phys. 134, 357-363. M. Abramowitz and I. Stegun, eds (1972), Handbook of Mathematical Functions, Dover, New York. B. Alpert, L. Greengard and T. Hagstrom (1999a), 'Accurate solution of the wave equation on unbounded domains'. In preparation. B. Alpert, L. Greengard and T. Hagstrom (19996), 'Rapid evaluation of nonrefiecting boundary kernels for time-domain wave propagation', SIAM J. Numer. Anal. To appear. C. R. Anderson (1992), 'An implementation of the fast multipole method without multiples', SIAM J. Sci. Statist. Comput. 13, 923-947. A. Barry, J. Bielak and R. MacCamy (1988), 'On absorbing boundary conditions for wave propagation', J. Comput. Phys. 79, 449-468. A. Bayliss and E. Turkel (1980), 'Radiation boundary conditions for wave-like equations', Comm. Pure Appl. Math. 33, 707-725. J.-P. Berenger (1994), 'A perfectly matched layer for the absorption of electromagnetic waves', J. Comput. Phys. 114, 185-200. P. Bettess (1992), Infinite Elements, Penshaw Press, Sunderland, UK. W. Chew and W. Weedon (1994), 'A 3-D perfectly matched medium from modified Maxwell's equations with stretched coordinates', Microwave Optical Technol. Lett. 7, 599-604. F. Collino (1993), Conditions d'ordre eleve pour des modeles de propagation d'ondes dans des domaines rectangulaires, Technical Report 1790, INRIA. F. Collino and P. Monk (1998), 'Optimizing the perfectly matched layer'. Preprint. L. Demkowicz and K. Gerdes (1999), 'Convergence of the infinite element methods for the Helmholtz equation in separable domains', Numer. Math. To appear. G. Doetsch (1974), Introduction to the Theory and Application of the Laplace Transformation, Springer, New York. J. Driscoll, D. Healy and D. Rockmore (1997), 'Fast discrete polynomial transforms with applications to data analysis for distance transitive graphs', SIAM J. Comput. 26, 1066-1099. B. Engquist and A. Majda (1977), 'Absorbing boundary conditions for the numerical simulation of waves', Math. Comput. 31, 629-651. B. Engquist and A. Majda (1979), 'Radiation boundary conditions for acoustic and elastic wave calculations', Comm. Pure Appl. Math. 32, 313-357. A. Eringen and E. §uhubi (1975), Elastodynamics, Vol. 2, Academic Press, New York. T. Geers (1998), Benchmark problems, in Computational Methods for Unbounded Domains (T. Geers, ed.), Kluwer Academic Publishers, Dordrecht, Netherlands, pp. 1-10. M. Giles (1990), 'Nonreflecting boundary conditions for Euler equation calculations', AIAA Journal 28, 2050-2058. D. Givoli (1991), 'Non-reflecting boundary conditions', J. Comput. Phys. 94, 1-29. D. Givoli (1992), Numerical Methods for Problems in Infinite Domains, Vol. 33 of Studies in Applied Mechanics, Elsevier, Amsterdam.

104

T. HAGSTROM

D. Givoli and D. Kohen (1995), 'Non-reflecting boundary conditions based on Kirchoff-type formulae', J. Comput. Phys. 117, 102-113. J. Goodrich and T. Hagstrom (1999), 'High-order radiation boundary conditions for computational aeroacoustics'. In preparation. L. Greengard and P. Lin (1998), 'On the numerical solution of the heat equation on unbounded domains (Part I)'. Preprint. L. Greengard and V. Rokhlin (1997), A new version of the fast multipole method for the Laplace equation in three dimensions, in Ada Numerica, Vol. 6, Cambridge University Press, pp. 229-269. M. Grote and J. Keller (1995), 'Exact nonreflecting boundary conditions for the time dependent wave equation', SIAM J. Appl. Math. 55, 280-297. M. Grote and J. Keller (1996), 'Nonreflecting boundary conditions for time dependent scattering', J. Comput. Phys. 127, 52-81. M. Grote and J. Keller (1998), Exact nonreflecting boundary conditions for elastic waves, Technical Report 1998-08, ETH, Zurich. M. Grote and J. Keller (1999), 'Nonreflecting boundary conditions for Maxwell's equations', J. Comput. Phys. To appear. B. Gustafsson and H.-O. Kreiss (1979), 'Boundary conditions for time-dependent problems with an artificial boundary', J. Comput. Phys. 30, 333-351. T. Hagstrom (1983), Reduction of Unbounded Domains to Bounded Domains for Partial Differential Equation Problems, PhD thesis, California Institute of Technology. T. Hagstrom (1991a), 'Asymptotic boundary conditions for dissipative waves: General theory', Math. Comput. 56, 589-606. T. Hagstrom (19916), 'Conditions at the downstream boundary for simulations of viscous, incompressible flow', SIAM J. Sci. Statist. Comput. 12, 843-858. T. Hagstrom (1995), On the convergence of local approximations to pseudodifferential operators with applications, in Proc. 3rd Int. Conf. on Math, and Num. Aspects of Wave Prop. Phen. (E. Becache, G. Cohen, P. Joly and J. Roberts, eds), SIAM, pp. 474-482. T. Hagstrom (1996), On high-order radiation boundary conditions, in IMA Volume on Computational Wave Propagation (B. Engquist and G. Kriegsmann, eds), Springer, New York, pp. 1-22. T. Hagstrom and J. Goodrich (1998), 'Experiments with approximate radiation boundary conditions for computational aeroacoustics', Appl. Numer. Math. 27, 385-402. T. Hagstrom and S. Hariharan (1996), Progressive wave expansions and open boundary problems, in IMA Volume on Computational Wave Propagation (B. Engquist and G. Kriegsmann, eds), Springer, New York, pp. 23-43. T. Hagstrom and S. Hariharan (1998), 'A formulation of asymptotic and exact boundary conditions using local operators', Appl. Numer. Math. 27, 403-416. T. Hagstrom and H. B. Keller (1986), 'Exact boundary conditions at an artificial boundary for partial differential equations in cylinders', SIAM J. Math. Anal. 17, 322-341. T. Hagstrom and J. Lorenz (1994), Boundary conditions and the simulation of low Mach number flows, in Proceedings of the First International Conference on

RADIATION BOUNDARY CONDITIONS

105

Theoretical and Computational Acoustics (D. Lee and M. Schultz, eds), World Scientific, Singapore, pp. 657-668. E. Hairer, C. Lubich and M. Schlichte (1985), 'Fast numerical solution of nonlinear Volterra convolutional equations', SIAM J. Sci. Statist. Comput. 6, 532-541. L. Halpern (1986), 'Artificial boundary conditions for the linear advection diffusion equation', Math. Comput. 46, 425-438. L. Halpern (1991), 'Artificial boundary conditions for incompletely parabolic perturbations of hyperbolic systems', SIAM J. Math. Anal. 22, 1256-1283. L. Halpern and J. Rauch (1987), 'Error analysis for absorbing boundary conditions', Numer. Math. 51, 459-467. L. Halpern and J. Rauch (1995), 'Absorbing boundary conditions for diffusion equations', Numer. Math. 71, 185-224. L. Halpern and M. Schatzman (1989), 'Artificial boundary conditions for viscous incompressible flows', SIAM J. Math. Anal. 20, 308-353. S. He and V. Weston (1996), Wave-splitting and absorbing boundary conditions for Maxwell's equations on a curved surface, Technical Report TRITA-TET96-14, KTH, Stockholm. R. Higdon (1986), 'Absorbing boundary conditions for difference approximations to the multidimensional wave equation', Math. Comput. 47, 437-459. R. Higdon (1987), 'Numerical absorbing boundary conditions for the wave equation', Math. Comput. 49, 65-90. R. Higdon (1991), 'Absorbing boundary conditions for elastic waves', Geophysics 56, 231-254. R. Higdon (1992), 'Absorbing boundary conditions for acoustic and elastic waves in stratified media', J. Comput. Phys. 101, 386-418. R. Higdon (1994), 'Radiation boundary conditions for dispersive waves', SIAM J. Numer. Anal. 31, 64-100. R. Holford (1999), 'A multipole expansion for the acoustic field exterior to a prolate or oblate spheroid'. Submitted to J. Acoust. Soc. Amer. C. Johansson (1993), 'Boundary conditions for open boundaries for the incompressible Navier-Stokes equations', J. Comput. Phys. 105, 233-251. T. Kato (1976), Perturbation Theory for Linear Operators, Springer, New York. H.-O. Kreiss and J. Lorenz (1989), Initial-Boundary Value Problems and the Navier-Stokes Equations, Academic Press, New York. E. Lindman (1975), 'Free space boundary conditions for the time dependent wave equation', J. Comput. Phys. 18, 66-78. J.-P. Loheac (1991), 'An artificial boundary condition for an advection-diffusion equation', Math. Meth. Appl. Sci. 14, 155-175. D. Ludwig (1960), 'Exact and asymptotic solutions of the Cauchy problem', Comm. Pure Appl. Math. 13, 473-508. M. Mohlenkamp (1997), 'A fast transform for spherical harmonics'. Preprint. R. Newton (1966), Scattering Theory of Waves and Particles, McGraw-Hill, New York. J. Nordstrom (1995), 'Accurate solutions of the Navier-Stokes equations despite unknown outflow boundary data', J. Comput. Phys 120, 184-205.

106

T. HAGSTROM

J. Nordstrom (1997), 'On extrapolation procedures at artificial outflow boundaries for the time-dependent Navier-Stokes equations', Appl. Numer. Math. 23, 457-468. F. Oberhettinger and L. Badii (1970), Tables of Laplace Transforms, Springer, New York. F. Olver (1954), 'The asymptotic expansion of Bessel functions of large order', Philos. Trans. Royal Soc. London A247, 328-368. P. Petropoulos (1999), 'Reflectionless sponge layers as absorbing boundary conditions for the numerical solution of Maxwell's equations in rectangular, cylindrical and spherical coordinates'. Submitted to SIAM J. Appl. Math. Y. Radvogin and N. Zaitsev (1998), Absolutely transparent boundary conditions for time-dependent wave problems, in Seventh International Conference on Hyperbolic Problems. A. Ramm (1986), Scattering by Obstacles, D. Reidel, Dordrecht, Netherlands. V. Rokhlin (1990), 'Rapid solution of integral equations of scattering theory in two dimensions', J. Comput. Phys. 86, 414-439. V. Ryaberikii (1985), 'Boundary equations with projections', Russian Math. Surveys 40, 147-183. M. Schwartz (1987), Principles of Electrodynamics, Dover, New York. I. Sofronov (1993), 'Conditions for complete transparency on the sphere for the three-dimensional wave equation', Russian Acad. Sci. Dokl. Math. 46, 397401. I. Sofronov (1999), 'Artificial boundary conditions of absolute transparency for twoand three-dimensional external time-dependent scattering problems', Euro. J. Appl. Math. To appear. L. Ting and M. Miksis (1986), 'Exact boundary conditions for scattering problems', J. Acoust. Soc. Amer. 80, 1825-1827. L. Trefethen and L. Halpern (1986), 'Well-posedness of one-way wave equations and absorbing boundary conditions', Math. Comput. 47, 421-435. L. Trefethen and L. Halpern (1988), 'Wide-angle one-way wave equations', J. Acoust. Soc. Amer. 84, 1397-1404. S. Tsynkov (1998), 'Numerical solution of problems on unbounded domains. A review', Appl. Numer. Math. 27, 465-532. E. Turkel and A. Yefet (1998), 'Absorbing PML boundary layers for wave-like equations', Appl. Numer. Math. 27, 533-557. O. Vacus (1996), Singularites de frontiere et conditions limites absorbantes: le probleme du coin, Technical Report 2851, INRIA. L. Xu and T. Hagstrom (1999), 'On convergent sequences of approximate radiation boundary conditions and reflect ionless sponge layers'. In preparation.

Ada Numerica (1999), pp. 107-141

© Cambridge University Press, 1999

Numerical methods in tomography Frank Natterer Institut fur Numerische und Instrumentelle Mathematik, Universitdt Miinster, Einsteinstrasse 62, D-48149 Miinster, Germany E-mail: nattereOmath. uni-muenster. de

In this article we review the image reconstruction algorithms used in tomography. We restrict ourselves to the standard problems in the reconstruction of function from line or plane integrals as they occur in X-ray tomography, nuclear medicine, magnetic resonance imaging, and electron microscopy. Nonstandard situations, such as incomplete data, unknown orientations, local tomography, and discrete tomography are not dealt with. Nor do we treat nonlinear tomographic techniques such as impedance, ultrasound, and nearinfrared imaging.

CONTENTS 1 Introduction 2 The filtered backprojection algorithm 3 3D reconstruction formulas 4 Iterative methods 5 Circular harmonic algorithms 6 Fourier reconstruction 7 Conclusions References

107 109 121 127 131 134 138 139

1. Introduction By 'tomography' we mean a technique for imaging 2D cross-sections of 3D objects. It is derived from the Greek word TO^LO<; = slice. Tomographic techniques are used in radiology and in many branches of science and technology.

108

F. NATTERER

1.1. The basic example In the simplest case, let us consider an object whose attenuation coefficient with respect to X-rays at the point x is f(x). We scan the cross-section by a thin X-ray beam L of unit intensity. The intensity past the object is e This intensity is measured, providing us with the line integral

g(L) = Jf(x)dx.

(1.1)

L

The problem is to compute / from g. In principle this problem has been solved by Radon (1917). Let L be the straight line x 0 = s where 6 = (cos
g{0,s)= J f(x)dx = (Rf)(6,s).

(1.2)

x-e=s

R is known as the Radon transform. Radon's inversion formula reads 2?r

where g' is the derivative of g with respect to s, and 6 = (cos
NUMERICAL METHODS IN TOMOGRAPHY

109

1.3. Outline of the paper In Section 2 we describe the most important algorithm in tomography, namely the filtered backprojection algorithm. Not only is it the workhorse for today's tomography, but it also serves as the model for the algorithms in future imaging devices, such as the 3D algorithms described in Section 3. Since many imaging problems can be described by large linear sparse systems of equations, iterative methods suggest themselves (see Section 4). Algorithms exploiting rotational symmetry of the imaging devices are described in Section 5. In Section 6 we deal with algorithms which work exclusively in Fourier space and which have the potential to outperform the filtered backprojection algorithm in speed. 1.4-

Prerequisites

The reader needs only a rudimentary knowledge of numerical analysis (interpolation, quadrature, linear systems), functional analysis (linear operators, distributions), Fourier analysis (Fourier transform, Fourier series, inversion, convolution, fast Fourier transform (FFT)) and sampling theory (Shannon's sampling theorem for band-limited functions). 2. The filtered backprojection algorithm In this section we give a detailed description of the most important algorithm in 2D tomography. The discrete implementation depends on the scanning geometry, that is, the way the data are sampled. This algorithm is essentially a numerical implementation of the Radon inversion formula (1.3). However, a different approach avoiding singular integrals is simpler. We describe this approach for the n-dimensional Radon transform

9(0,8) = J f(x)dx = (Rf)(0,s) x-6-s n

where / is a function in R and 0 € S n ~ \ s e t 1 . Let (R*g)(x)=

I

g(6,0-x)d6

gn-l

be the backprojection operator and let V, v be functions such that V = R*v. Mathematically, R* is simply the Hilbert space adjoint of the Radon transform R. It is easy to see that V*f = R*(v*g),

(2.1) n

where the convolution on the left-hand side is in R , while the convolution on

110

F. NATTERER

the right-hand side is a ID convolution with respect to the second variable: fv(x-y)f(y)dy

=

f

f v(x

6 - s)g(8, s) dsd9.

(2.2)

The idea is to choose V as an approximation to Dirac's <5-function. Then, V -k f is close to / . The interrelation between V, v is easily described in terms of the Fourier transform. Denoting with the same symbol 'A' the n-dimensional Fourier transform = (27r)-"/2 f

e-ix
and the ID Fourier transform v{6, a) = (27T)"1/2 / e~isav{6, s) ds,

a G R1,

R1

we have

see Natterer (1986). By |£| we mean the Euclidean length of £ G W1. The choice of V determines the spatial resolution of the reconstruction algorithm. We use the notion of resolution from sampling theory: see Jerry (1977). We give a short account of some basic facts of sampling theory. A function / in M.n is said to be band-limited with bandwidth 17, or simply Jl-band-limited, if /(£) = 0 for |£| > fi. An example for n = 1 is the sine function sin x smc (x) = , x which has bandwidth 1. Obviously, sine (fix) has bandwidth Q. J7-bandlimited functions are capable of representing details of size 27r/f2 but no smaller ones. This becomes clear simply by looking at the graph of sine. In tomography the functions we are dealing with are usually of compact support. Such functions cannot be strictly band-limited, unless identically zero. Hence we require the functions only to be essentially Q-band-limited, meaning that /(£) is approximately 0 for |£| > Q in some sense: see Natterer (1986). A reconstruction method in tomography is said to have resolution 2vr/0 if it reconstructs essentially fi-band-limited functions reliably. For strictly fi-band-limited functions we have the following propositions, which also hold, with very good accuracy, for essentially fi-band-limited functions.

NUMERICAL METHODS IN TOMOGRAPHY

111

1. If / is f2-band-limited and h < TT/Q (the Nyquist condition), then / is uniquely determined by the values f(hk), k € Z, and f[x) =

- hk).

2. If / is fi-band-limited and h < ir/Cl, then

3. If /i, J2 are fi-band-limited and h < vr/fi, then

f fi(x)f2(x)dx

=

Returning to the creation of a reconstruction algorithm with resolution 2vr/f2, we have to determine V such that

where 0 is a filter with the property

This follows from the formula 6 = (2-7r)"n/'2. This means that for the filter function v we must have in—1,

(2.3)

Examples are the ideal low pass 1, \a 0, \a the cos filter _ J cos (air/2), 0,

\a \a

and the filter __ J sine (air/2), which has been introduced in tomography by Shepp and Logan (1974). The corresponding functions v for n — 2 are vn{s) =

^M(QS),

U(S)

= sine (s) - - (sine (-J J

112

F. NATTERER

for the ideal low pass,

with the same function u, and /

/2

- s sin s,

for the Shepp-Logan filter. More filters can be found in Chang and Herman (1980). The integral on the right-hand side of (2.2) has to be approximated by a quadrature rule. We have to distinguish between several ways of sampling 9 = Rf2.1. Parallel geometry in the plane In this case the 2D Radon transform g(0,s) = (Rf)(0,s) is sampled for 6 = 0j = (cospj, sinipj)T, ifj = irj/p, j = 0,... ,p - 1 and s = se = £p/q, £ = —q,..., q. Here p is the radius of the reconstruction region, that is, we assume f(x) = 0 for x G M2, \x\ > p. This means that the measured rays come in p parallel bundles with directions evenly distributed over 180°, each bundle consisting of 2q + 1 equispaced lines. This was the scanning geometry of the first commercial scanner for which Hounsfield received the Nobel prize in 1979. This geometry has been replaced by more efficient ones in today's scanners (see below), but it is still used in scientific and technical imaging. We evaluate the integral in (2.2) by the trapezoidal rule

(V * f)(x) ^2-^Y,Y. Pq

"«(*

9

i - 8i)9(6j> se)-

(2-4)

j=Q£=-q

The accuracy of this approximation can be assessed by sampling theory, according to which the trapezoidal rule for an inner product is exact provided the step-size h satisfies the Nyquist criterion, that is, h < ir/Cl where Q is the bandwidth of the factors in the inner product. In our case the first factor is VQ(X 0j — s) (as a function of s) which has bandwidth $1. The second factor is g(0j, s) (again as a function of s). This is given and does not, in general, have finite bandwidth. At this point we have to make an assumption. We assume / to be essentially band-limited with essential bandwidth 0. The n-dimensional Fourier transform of / and the ID Fourier transform Rf (with respect to the second variable) are interrelated by (Rf)A(0, a) = (2 7 r)("- 1 )/ 2 /(^).

(2.5)

This is the famous (and easy to prove) 'projection' or 'central slice' theorem

NUMERICAL METHODS IN TOMOGRAPHY

113

of computerized tomography. In the present context we need it only to deduce that / and g = Rf have the same (essential) bandwidth. Thus the sintegral in (2.2) is accurately represented by the l-sum in (2.4) provided that the step-size h = p/q in that sum satisfies the Nyquist criterion h < 7r/S7. In other words, q > -pQ.

(2.6)

The condition for the number p of directions which makes the j'-sum in (2.4) a good approximation for the ^-integral in (2.2) is less obvious. Based on Debye's asymptotic relation for the Bessel functions one can show that the essential bandwidth of Rf as a function of ip, 8 = (cos ip, sin nP.

(2.7)

Inequalities (2.6), (2.7) are the conditions for high accuracy in (2.4), assuming / to be zero outside the ball of radius p and essentially band-limited with bandwidth Q,. The double sum in (2.4) has to be evaluated for each reconstruction point x. This leads to unacceptable complexity. This complexity can be reduced by introducing the function 9

Then, (2.4) reads

%=0

This requires only a simple sum for each reconstruction point x, at the expense of an additional interpolation in the second argument of h. In most cases linear interpolation suffices (but nearest neighbour does not). This leads us to the filtered backprojection algorithm for standard parallel geometry (see Herman (1980, p. 133)).

Algorithm 1 Filtered backprojection algorithm for standard parallel geometry Data: The values {gj/ = g(9j, se) : j = 0,... ,p — 1, £ = —q,..., q}, where g is the 2D Radon transform of / . Step 1: For j = 0,... ,p — 1 carry out the discrete convolution p Y~» q

e=-q

114

F. NATTERER

Step 2: For each reconstruction point x, find the discrete backprojection f

( \ — — V^ ((1 — 9U

-I- 9/?

1

^ 3=0

where k = k(j, x) and i? = -d(j, x) are determined by + — L

2. ,

£ — l+l

i9

t

h

A,

U

(/

ft,.

It

I ,

Result: fpB is an approximation to f(x). The algorithm depends on the parameters p, q and on the choice of VQ. It is designed to reconstruct a function / with support in \x\ < p and with essential bandwidth fi, that is, the spatial resolution of the algorithm is 2TT/Q. Conditions (2.6), (2.7) should be satisfied. A few remarks are in order. 1. Condition (2.6) has to be strictly satisfied. Otherwise the s integral in (2.2) is not even approximately equal to the £ sum in (2.4), leading to unacceptable errors. 2. If (2.7) is not satisfied, the reconstruction is still accurate for \x\ < p/9, < p. 3. Filter functions VQ whose 'kernel sum'

does not vanish should not be used: see Natterer and Faridani (1990). 4. Usually, linear interpolation in Step 2 is sufficient. However, for difficult functions / (e.g., functions containing large objects at the boundary of the reconstruction region) linear interpolation generates visible artefacts. In that case an oversampling procedure similar to the one of Algorithm 2 below is advisable. Alternatively one may use the circular harmonic algorithm from Section 5. 5. The algorithm needs O(p) operations for each reconstruction point. Algorithms with lower complexity (such as O(logp)) can be obtained either by Fourier reconstruction (see Section 6) or by the fast backprojection algorithm in Section 2.5. 6. Conditions (2.6), (2.7) suggest taking p — irq. This much debated condition is usually not complied with in radiological applications, where p is chosen to be considerably smaller. This is due to the special requirements in radiological imaging.

NUMERICAL METHODS IN TOMOGRAPHY

115

2.2. The interlaced parallel geometry It is well known (see, for instance, Kruse (1989)) that the data in the standard parallel geometry are redundant: if p is even, then one can omit each g(6j,se) with £ + j odd without impairing the resolution. Deriving algorithms that use only the remaining 'interlaced' data (i.e., those g(Oj, Si) for which j + £ even) is fairly subtle. What happens is the following. If in the £ sum in (2.4) every second term is dropped, the sum no longer approximates the corresponding s integral in (2.2). Miraculously, the large quadrature error cancels when the j sum in (2.4) is computed. This means that success depends entirely on a subtle interplay between different directions. This interplay is disrupted by the interpolation procedure in Step 2 of Algorithm 1. There are two ways out. The first one is to avoid interpolation altogether by using circular harmonic algorithms: see Section 5. The second one is to make the interpolation more accurate, for instance by oversampling. This leads to an algorithm that has the structure of a filtered backprojection algorithm. Algorithm 2 Filtered backprojection algorithm for parallel interlaced geometry Data: The values {g(9j, se) : j = 0,... ,p — 1, £ — —q,..., q, £ + j even}, where g is the 2D Radon transform of / , and p has to be even. Step 1: Choose a sufficiently large integer M > 0 (M = 16 will do) and compute, for j — 0,... ,p - 1,

E ^(g^)s(^)

k = -Mq,...,Mq.

t=-q £+j even

Step 2: For each reconstruction point x, compute 2 p"1 fFB(x) = — Y, ((1 - #)hj,k + %,fc+i), P

3=0

where k = k(j,x), 1? = $(j, x) are determined by

t = Ma—p-, Result:

/FB(^)

k=\t\,

#= t-k.

p/q is an approximation to f(x).

Note that the difference between this algorithm and Algorithm 1 is that it needs only half the data but produces the same image quality. We study the various assumptions underlying this algorithm.

116

F. NATTERER

1. The algorithm is designed to reconstruct a function / supported in \x\ < p with essential bandwidth fi. The sampling conditions (2.6), (2.7) have to be satisfied. In contrast to Algorithm 1, oversatisfying these conditions may lead to artefacts. Thus the algorithm should be used only if (2.6), (2.7) are satisfied with equality, that is, for p = ir q. 2. Only filters v with a smooth transition from nonzero to zero values should be used. The reason is that the additional filtering of the interpolation step is not present in Algorithm 2. 2.3. Standard fan beam geometry This is the most widely used scanning geometry. It is generated by a source moving on a concentric circle of radius r > p around the reconstruction region \x\ < p, with opposite detectors being read out in small time intervals (third generation scanner). Equivalently we may have a fixed detector ring with only the source moving around (fourth generation scanner). Denoting the angular position of the source by 0 and the angle between a measured ray and the central ray by a (a > 0 if the ray, viewed from the source, is left of the central ray), then fan beam scanning amounts to sampling the function = (Rf)(9,r sin a),

g{0,a)

9 = (COf + a\) \ sm(/j + a) J

(2.8) x

'

at the points 0 = 0j=jA0,

A0 = 2n/p, ,

j = 0,... £=-q,...,q.

Here, q is chosen so as to cover the whole reconstruction region \x\ < p with rays, and d is the detector offset which is either 0 or . First we derive the fan beam analogue of (2.1). We only have to put

(V * f)(x) = r

//

/ v(x 0

-TT/2

9 — r sina)g(0, a) cosccdad/3

NUMERICAL METHODS IN TOMOGRAPHY

117

with 9 as in (2.8). Discretizing the integral by the trapezoidal rule yields p-l

(V*f)(x)

q

~ r A a A / 3 ^ ^ va{x j=o e=-q

6(/3j + ae) - rsmae)g(f3j,ae) cosae.

(2.9) This is the fan beam analogue of (2.4) and defines a reconstruction algorithm for fan beam data. One can show that for this algorithm to have resolution 2?r/f2 one has to satisfy

see Natterer (1993). As in the parallel case, an algorithm based on (2.9) needs O(pq) operations for each reconstruction point. Reducing this to O(p) is possible here, too, but this is not as obvious as in the parallel case. We first establish a relation for the expression x 0(ip) — s in (2.2). Let b = rd((3) be the source position, and let 7 be the angle between x — b and —6. We take 7 positive if x, viewed from the source b, lies to the left of the central ray, that is, we have

(b-x) -b |6-xl-|6|'

CO67=

where 0(
(2.11)

Thus, 6{(p) - s) = \b - x|- 2 u n | 6 _ x |(sin(7 - a)).

vQ(x

Using this in (2.2) we obtain 2TT

(V*f)(x) = r / \b-x\~ 0

T/2

2

/ vn\b_xi(sm(j-a))g((3,a)cosadad/3. -TT/2

Here, b = r9((3), and 7 is independent of a. Unfortunately, the a integral has to be evaluated for each x since the subscript ft\b — x\ depends on x. In order to avoid this we make an approximation: we replace Cl\b — x\ by fir. This is not critical as long as \b — x\ ~ |6|, that is, as long as p
118

F. NATTERER

in most scanners r ~ 3p, and this is sufficient for the approximation to be satisfactory. However, if p is only slightly smaller than r, problems arise. Upon the replacement of i>n|b_x| by VQr we obtain 2TT

x

(V * f)( ) —

r

T/2

2

I \b — x\~ 0

I ^nr(sin(7 — a))g((3, a) cosadad/3.

-TT/2

The a integral can now be precomputed as a function of 7 and (3, yielding an algorithm with the structure of a filtered backprojection algorithm. Algorithm 3 Filtered backprojection

algorithm for parallel standard fan beam

D a t a : T h e values {gj
geometry

: j = 0 , . . . ,p - 1, £ = -q,...,q},

where

g is the function in (2.8). Step 1: For j = 0,... ,p — 1 carry out the discrete convolutions vur(sm(ak - ae))gje cos at,

k = -q,...,

q.

l=-q

Step 2: For each reconstruction point x, compute the discrete weighted backprojection p-i b

fFB(x) = rApJ2\ J .

~ x\~2 (C1 - #)h3,i° + #

3=0

where k = k(j, x) and d = ~d(j, x) are determined by bj - x)

7 = it arccos \bj-x\\bj\'

bj

the sign being the one of —x bj and bj = r9((3j), t = -?-, Aa

k=\t\, tf =

t-k.

Result: / F B ( ^ ) is an approximation to f(x). The algorithm as it stands is designed to reconstruct a function / with support in \x\ < p which essentially band-limited with bandwidth fl from fan beam data with the source on a circle of radius r > p. The remarks following Algorithm 1 apply by analogy. In particular, conditions (2.10) have to be satisfied. For r ~ p and with dense parts of the object close to the boundary of the reconstruction region, problems are likely to occur.

NUMERICAL METHODS IN TOMOGRAPHY

119

2.4. Linear fan beam geometry Here, the detector positions within a fan with vertex b are evenly spaced on the line perpendicular to b. We need the explicit form of the inversion formula mainly for the derivation of the FDK algorithm in 3D cone beam tomography in the next section. With g the function in (2.8), the sampled data are 9j,e

=

g{/3j,ae),

Pj

=

—j, P

j = 0,...,p-l,

£=

ae = t&-n(ye/r),

-q,...,q

ye = (£ +

d)Ay.

The coordinates (f3, y) are related to the parallel coordinates (ip, s) in the representation x 9(ip) = s of the rays by V

<^ = /3 + arctani, r

TV y

s=

1

(2.12)

ylyil

[r +

Hence, r3

d { ) 2

d{0,y) (r + Substituting (/?, y) for (ip, s) in (2.2) leads to 2TT

3

(V*f)(x) = J Jv(x.9(
where (2.12) has to be inserted for (
6(/3) )

_ rx

9(0 + TT/2) r-x

z

Prom (2.11) it follows that vn(c(z - y)) = c~2vcn(z - y), yielding 2TT

3 r dyd/3

2

(V*f) = I Jc- vcU(z-y)g((3,y)-{( 2 r

0 Rl 2TT

f

= r

1 f n/aw> I J [r — x u(p)Y J

v

cn(z-y)g((3,y)i

120

F. NATTERER

As in the standard fan beam case we make the approximation c ~ 1. Again this is justified if p 3p. Then,

(V*f)(x)^

f

^

J (r — x

f vn (^^ ~ y) 9(8, V),\r*-\2 \ \r — x t) )

vy J

S1

where

K1

— 6(ip + IT/2). Defining h(0, z) = j vQ(z - y)g(O, y)

^

u/2,

(2-13)

this can be written as

(v*f)(x)~

fh(e, ™_'^e) de-

( 2 - 14 )

The implementation of (2.13), (2.14) can now be done exactly as in the standard case, leading to a filtered backprojection algorithm which needs O(p) operations for each reconstruction point x.

2.5. Fast backprojection The backprojection (Step 2 in Algorithms 1-3) is the most time-consuming part of the filtered backprojection algorithm. The filtering or convolution step (Step 1 in Algorithms 1-3) requires in principle the same number of operations, but this can easily be reduced drastically either by cutting off the filter VQ or by using the fast Fourier transform (FFT). The backprojection consists of the evaluation of the sums

o n a p x p grid. This is the simplest case of Algorithm 1, the resolution of the image being adjusted to the number of views p according to the sampling theorem. Nilsson (1997) suggested a divide and conquer strategy for doing this with O(p2logp) operations, as opposed to the O(p 3 ) operations of a direct evaluation. Suppose p = 2m. Step 1: For j = 0,1, 2 , . . . ,p — 1, compute

f}(x)=g(9j,x-ej). Since fj is constant along the lines x 6j = s it suffices to compute fj (x) at 2p points x. We need p 2p operations. Step 2: For j = 0 , 2 , 4 , . . . ,p - 2, compute

NUMERICAL METHODS IN TOMOGRAPHY

121

Since fj, fj+l are constant along the lines x-Oj = s, x-Oj+i — s, respectively, fj is almost constant along the lines x 9j = s where 9j — (0j + 0j+i)/\/2. Hence it suffices to compute fj only for a few, say 2, points on each of those lines. This means that we have to evaluate fj at 4p points, requiring | 4p operations. Step 3: For j = 0,4,8,..., p - 4, compute

With the same reasoning as in Steps 1 and 2 we find that it suffices to compute ff(x) for only for 8p points, requiring | 8p operations. Proceeding in this fashion we arrive in Step m at the approximation /j™ to /, which has to be evaluated at 2mp points. Hence the number of operations in Steps 1 to m is p 2p + |

4p + |

8p +

+^

2 > = mp2

and this is O(p2logp). Of course this derivation is heuristic, and we have simply ignored the necessary interpolations and approximations. However, practical experience demonstrates that such an algorithm can be made to work. 3. 3D reconstruction formulas Algorithms for 3D tomography are still under development. The main problem is that the data entering explicit inversion formulas are usually not available. Thus the main task in 3D tomography is the derivation of inversion formulas that use only the data measured by a specific imaging device. It is clear that these formulas are tailored to the imaging device. In this section we restrict ourselves to the derivation of exact or approximate inversion formulas. The implementation in a discrete setting can be done along the lines of the 2D algorithms. 3.1. Inversion of the 3D Radon

transform

Let g = Rf, R the 3D Radon transform, be given on S2 xR 1 . Using (2.1) for n = 3 leads directly to a filtered backprojection algorithm, exactly as in the 2D case. Introducing spherical coordinates
(

sini/) cos (/A sinV' sin ip J , cos?/> J

0 < ip < 2TT,

0 < ip < IT,

122

F. NATTERER

(2.1) reads (V*f)(x)=

f f h(8,x-d)sinipdipdp,

(3.1)

0 0

where 8 = 8((p, ip) and

h(8,t) = I g(6,s)v(t-s)ds. Once h is computed, the evaluation of (3.1) requires the computation of a 2D integral for each reconstruction point. This is prohibitive in real world applications. Fortunately we can exploit the structure of the 3D Radon transform as the composition of two 2D Radon transforms. Putting n

kv(s,t)

— / h(6() sinipdip, o we can rewrite (3.1) as (V*f)(.x)=

(3.2) / k^xa, xi cos ip + x2 sin ip) dip. o The last two formulas are essentially 2D backprojections. They can be evaluated exactly as described in the previous section. After having precomputed h and k the final reconstruction step (3.2) requires only a ID integral for each reconstruction point. This algorithm is reminiscent of the two-stage algorithm of Marr, Chen and Lauterbur (1981), developed for magnetic resonance imaging (MIR), except that the convolution steps are not present. 3.2. The FDK approximate formula This is the most widely used algorithm for cone beam tomography with the source running on a circle. It is well known that this inversion problem is highly unstable. However, practical experience with the FDK formula is nevertheless quite encouraging. The function sampled in cone beam tomography with the source on a circle is

g(0,y) = Jf(rO + ty)dt, where 9 is a direction vector in the X\-X2 plane, 0 = (cos ip, sirup, 0)T, (see below) is the vector 01- is the subspace orthogonal to 9, while

NUMERICAL METHODS IN TOMOGRAPHY

123

(— simp, cosip, 0)T perpendicular to 9. As usual we assume / = 0 outside x\ < p where p < r.

The FDK formula is an ingenious adaptation of the 2D inversion formula of Section 2.4 to 3D. Consider the plane ir(9, x) through r9 and x which intersects 91- in a line parallel to the x\-X2 plane. Compute in this plane for each 9 the contribution to (2.14). Finally, integrate all these contributions over Sl, disregarding the fact that the contributions come from different planes. The necessary computations are unpleasant, but the result is fairly simple. Based on (2.14),

(V*f)(x)* I -r^-r 2 fvQ(u-u')g(9,u',z) x J v~ ')

S1

J

V

R1

rdti

+u

'f

=,

+ zA

(3.3)

where r r —x

9

'

r r—x

9

and (u',z) are coordinates in ^ , that is, g(9,u',z) stands for g(9,y) with y = (—u'sinip, u'cosip, z)T. The implementation of (3.3) leads to a reconstruction algorithm of the filtered backprojection type. The reconstructions computed with the FDK formula (3.3) are - understandably - quite good for flat objects, that is, if / is nonzero only close to the x\~X2 plane in which the source runs. If this is not the case then exact formulas using more data such as Grangeat's formula (see below) have to be used. 3.3. Grangeat's

formula

Grangeat's formula requires sources on a curve C with the following property: each plane meeting supp (/) contains at least one source. This condition is obviously not satisfied when C is a circle for which the FDK approximation has been derived. The data for Grangeat's formula are generated by the function

g(c,9)= f f{c + t9)dt,

ceC,

9eS2.

The condition on the source curve means that for each x with f(x) ^ 0 and each 9 € S2 there exists a source c = c(x, 9) G C such that x 9 = c(x, 0) 9. The gist of Grangeat's inversion is a relation between g and the 3D Radon transform Rf of / . This relation reads (Grangeat 1991) g-s(Rf)(9,s)\s=x.e=

j

—g{c(x,0),cj)du>,

(3.4)

where ^ stands for the derivative in the direction 9 € 5 2 , acting on the

124

F. NATTERER

second argument. For this to make sense we have to extend g to all of C x E3 by using the above definition not only for 9 € S2, but for all of R 3 . This is equivalent to extending g by homogeneity of degree —1 in the second argument. With the help of the 3D inversion formula

S2

for the 3D Radon transform, Grangeat's formula leads immediately to an inversion procedure for the data g. Related inversion formulas for cone beam tomography have been derived by Tuy (1983) and Gelfand and Goncharov (1987). For details of the implementation see Defrise and Clack (1995).

3.4- Orlov's inversion

formula

This formula inverts the ray transform

(Pf)(9,y) = J f(y + t9)dt, ye9x,

9 e S2

(3.5)

R1

which arises, for instance, in 3D emission tomography (PET, Defrise, Townsend and Clac (1989)). If 9 is restricted to a plane, then we simply have the Radon transform in this plane, and we can reconstruct / in that plane by any of the methods in the previous section. In practice g = Pf is measured for 9 e S2, where SfiCS2. In Orlov's formula (Orlov 1976), S$ is a spherical zone around the equator, that is, S2 = {0() :^-(v)
if>+{
using spherical coordinates 9((p,tp) — (cos (p cos ip, simp cos I/J, sin ip)T and -7r/2 < ip- < V+ < TT/2. Then, f{x)

= A f h(0, x - (x 9)9) d0, g(9,x-y)

dy,

(3.6)

where A is the Laplacian acting on x and £(9, y) is the length of the intersection of S2 with the plane spanned by 0,9,y € M3. The first formula of (3.6) is - up to A - a backprojection, while the second one is a convolution in #-*-. Thus an implementation of (3.6) is again a filtered backprojection algorithm. P can also be inverted by the Fourier transform. We have £G0\

(3.7)

NUMERICAL METHODS IN TOMOGRAPHY

125

where 'A' denotes the (n — l)-dimensional Fourier transform in 91- on the left-hand side and the Fourier transform in W1 on the right-hand side. Assume that S2, C S2 satisfies the Orlov condition: every equatorial circle of S2 meets SQ. Note that the set S2. (the spherical zone) we used above in Orlov's formula satisfies this condition. From (3.7) it follows that / is uniquely determined by (P/)(0, ) for 9 G SQ under the Orlov condition. Namely, if £ G R n is arbitrary, then Orlov's condition says that there exists S e ^ n ^ 1 , and /(£) is determined from (3.7). 3.5. Colsher's inversion formula Assume again that g = Pf is known for 9 G S2, C S2. We want to derive an inversion procedure similar to the one in Section 2.1. With the backprojection

(P*g)(x) =

Jg(9,x-(x-9)9)d9,

2

So

we again have V*f

= P*(v*g)

provided that V = P*v. Again the convolutions on each side have different meanings. Explicitly this reads

(V * f)(x) = J Jv(9,x-

(9

- y)g{9,y) dyd0,

(3.8)

si which corresponds to (2.2). As in (2.2) we express the relationship V = P*v in Fourier space, obtaining

see Colsher (1980). In order to get an inversion formula for P we have to determine v such that V = 6 or V = (2n)~3/2, that is,

J A solution v independent of 9 is

, O d 0 = (27r)- 2 K|.

(3.9)

126

F. NATTERER

where \SQ D ^ l is the length of SQ C\ £-*-. For the spherical zone SQ from Section 3.4 with ip_ = ipQ, ip+ = ipQ, ipo a constant with 0 < tpo < | , Colsher computed h explicitly. Setting £3 = |£J cos?/' we obtain (2vr)2 \\(4arcsin(sinVo/sinV) (4

\

Filters such as the Colsher filter (3.10) do not have small support. This means that g in (3.8) has to be known in all of 91-. Often g is only available in part of 91- (truncated projections). Let us choose

where 9 = 9(
/ where |£3| = |^'| tan V, l^'l = VW+M-

^ T h i s is close

' to F = (27r)"3/2 if Vo

is small. In this case reconstruction from truncated projections is possible, at least approximately.

3.6. Conical tilt geometry In electron microscopy (Frank 1992) one is faced with the case

for some ipo where 0((p,ip) = (cos(psin.i/j,sm(psinip, cos^O^- SQ does not satisfy Orlov's condition, and (3.9) cannot be satisfied since SQH^ = for some £. In that case we put

^ 0,

otherwise.

With this choice of v, (3.8) is the minimal norm solution of Pf = g. A proper discretization along the lines of Section 2.1 leads to the weighted backprojection algorithm of electron microscopy: see Frank (1992).

NUMERICAL METHODS IN TOMOGRAPHY

127

4. Iterative methods If exact inversion formulas are not available, iterative methods are the algorithms of choice. However, even if exact inversion is possible, iterative methods may be preferable due to their simplicity, versatility and ability to handle constraints and noise. Iterative methods are usually applied to discrete versions of the reconstruction problem. These discrete versions are obtained either by starting out from discrete models, as in the EM algorithm below, or by a projection method, known as a 'series expansion method' in the tomographic community (Censor 1981). This means that the unknown function / is written as N

for certain basis functions B^. With g\ the ith measurement, the measurement process being linear, we obtain the linear system N

for the expansion coefficients /^, the matrix element an being the ith measurement for the basis function B^. In tomography we always have an > 0. Also, the matrix (an) is typically sparse. Often B( is the characteristic function of pixels or voxels. Recently, smooth radially symmetric functions with small support (the 'blobs' of Lewitt (1992) and Marabini, Herman and Carazo (1998)) have been used. Blobs have several advantages over pixel- or voxel-based functions. Due to the radial symmetry it is easier to apply the Radon transform (or any of the other integral transforms) to B^, making it easier to set up the linear system (4.1). The smoothness of the B^ prevents a 'checkerboard' effect (i.e., the visual appearance of the pixels or voxels in the reconstruction) and does part of the necessary filtering and smoothing. The linear system (4.1) may be overdetermined (M > N) or underdetermined (M < N), consistent or inconsistent. Useful iterative methods must be able to handle all these cases. 4-1- ART (algebraic reconstruction

technique)

This is an extension of the Kaczmarz (1937) method for solving linear systems. It has been introduced in imaging by Gordon, Bender and Herman (1970). We describe it in a more general context. We consider the linear system Ajf

= gj,

j = 0,...,p-l1

(4.2)

128

F. NATTERER

where Ao : H j—* Hj are bounded linear operators from the Hilbert p space H p o ilbert space space Hj. With C,- : ifj —> H,- a positive definite operator into the Hilbert an iteration as follows: we define an iteration step fk —> / ffc,0

_ *fc j = O,...,p-l _

(4.3)

fk,p

If Cj = A*-Aj (provided Aj is surjective) and u — 1, then / f c j is the orthogonal projection of fk^~1 onto the afflne subspace Ajf — gj. For dim(Hj) = 1 this is the original Kaczmarz method. Other special cases are the Landweber method (p = 1, C\ = I) and fixed-block ART (dim(Hj) finite, Cj diagonal: see Censor and Zenios (1997)). It is clear that there are many ways to apply (4.3) to (4.1), and we will make use of this freedom to our advantage. One can show (Censor, Eggermont and Gordon 1983, Natterer 1986) that (4.3) converges provided that p-i

J ^ y.range i=o

{Aj)

and 0 < to < 2. This is reminiscent of the SOR theory of numerical analysis. In fact we have fk = Auk where uk is the A;th SOR iterative for the linear system AA*u = g with

;

/ 50

H

\5P-I.

If (4.2) is consistent, ART converges to the solution of (4.2) with minimal norm in H. Plain convergence is useful, but we can say much more about the qualitative behaviour and the speed of convergence by exploiting the special structure of the image reconstruction problems at hand. With R the Radon transform in Rn we can put H = L 2 (|x| < 1),

Hj = L 2 ( - l , +1; wl~n),

(Ajf)(s) = (Rf)(0j,s) where w is the weight function (1 — s 2 ) 1 / 2 and 9j E S1""1. One can show that the subspaces

CmJ(x)

=

C^2{x-63),

the Gegenbauer polynomials of degree m (Abramowitz and Stegun 1970)

NUMERICAL METHODS IN TOMOGRAPHY

129

are invariant subspaces of the iteration (4.3). This has been discovered by Hamaker and Solmon (1978). Thus it suffices to study the convergence on each subspace Cm separately. The speed of convergence depends drastically on UJ and - surprisingly - on the way the directions Oj are ordered. We summarize the findings of Hamaker and Solmon (1978) and Natterer (1986) for the 2D case where 6j = (cos ipj, sinipj). 1. Let the tpj be ordered consecutively, that is, ipj = jir/p, j = 0 , . . . ,p— 1. For u large (e.g., u> = 1) convergence on Cm is fast for m < p large and slow for m small. This means that the high-frequency parts of / (such as noise) are recovered first, while the overall features of / become visible only in the later iterations. For u small (e.g., u = 0.05) the opposite is the case. This explains why the first ART iterates for u = 1 look terrible and why surprisingly small relaxation factors (e.g. u = 0.05) are used in tomography. 2. Let { p are irrelevant since they describe those details in / that cannot be recovered from p projections because they are below the resolution limit. This is a result of the resolution analysis in Natterer (1986).

4-2. EM (expectation maximization) The EM algorithm for solving the linear system Af = g reads k

>0

(4 4)

Multiplications and divisions in this formula are understood componentwise. It is derived from a statistical model of image reconstruction: see Shepp and Vardi (1982). The purpose is to compute a minimizer of the log likelihood function

t(f) = J>* MAf); - (Af)i).

(4.5)

130

F. NATTERER

The convergence of (4.4) to a minimizer of (4.5) follows from the general EM theory in Dempster, Laird and Rubin (1977). Note that (4.4) preserves positivity if an > 0. The pure EM method (4.4) produces unpleasant images which contain too much detail. Several remedies have been suggested. An obvious one is to stop the iteration early, typically after 12 steps. One can also perform a smoothing step after each iteration (EMS algorithm of Silverman, Jones, Nychka and Wilson (1990)). A theoretically more satisfying method is to add a penalty term —B(f) to (4.5), that is, to minimize *(f)-B(f),

(4.6)

—B(f) may be interpreted either in a Bayesian setting or simply as a smoothing term. Typically,

= (f-7fB(f-J) where B is a positive definite matrix and / is a reference picture. Unfortunately, minimizing (4.6) is more difficult than minimizing (4.5) and cannot be done with a simple iteration such as (4.4). For partial solutions to this problem see Levitan and Herman (1987), Green (1990), Setzepfandt (1992). As in ART, a judicious arrangement of the equations can speed up the convergence significantly. The directions have to be arranged in 'ordered subsets': see Hudson, Hutton and Larkin (1992).

4-3. MART (multiplicative

algebraic reconstruction

technique)

While ART converges in the consistent case to a minimal norm solution of (4.1), MART is designed to converge to a solution of (4.1) which minimizes the entropy M

A log A-

(4.7)

For this to make sense we assume that (4.1) has a positive solution, and we seek the minimizer of (4.7) among those / that have only positive components. This is reasonable in many tomographic problems. The step fk -> fk+1 of the MART algorithm for (4.1) is as follows:

fk'°

= fk, aTfk,i-l

fk+1

_

fk,M

MART is an example of a multiplicative algorithm (see Pierro (1990)); another example is the EM algorithm.

NUMERICAL METHODS IN TOMOGRAPHY

131

5. Circular harmonic algorithms Circular harmonic algorithms can be derived from the inversion formula of Cormack (1963). For the 2D problem Rf = g it is obtained by Fourier expansions

f(x) = g(9,s)

= ^gt(s)e^,

0 = (cos ^, sir

I

One can show that oo 1

fjr\

r

/

,

(a2

—

jf.\ ) — — / \

s

r

r^\~l2T,

~

)

J

I

S N

\n'la\Aa

|£| I - )5^v s J c l s -

(£,!}

I53-1;

r

Tt is the Chebyshev polynomial of the first kind of order L In principle this defines an inversion formula for R: the Fourier coefficients gn of g — Rf determine the Fourier coefficients /^ of / via (5.1), and hence / is determined by g. The formula (5.1) is useless for practical calculations since Te increases exponentially with £ outside [—1, +1]. Cormack (1964) also derived a stable version of his inversion formula. It reads

-m

where Ug is the Chebyshev polynomial of the second kind of order £. This formula does not suffer from exponential increase. It is the starting point of the circular harmonic reconstruction algorithms of Hansen (1981), Hawkins and Barrett (1986) and Chapman and Cary (1986). We take a different route and start out from (2.1) again. We consider only the case n = 2. Putting x — x^ = Si9((fk), 0(ip) = (cos<£>,sin
{V*f)(xik) =

^2 5Zvn(sjCos(^_fc)

Pq

-se)g(0j,st).

e=-qj=o

The j sum is a convolution. In order to make this convolution cyclic we extend g(9j,S() by putting g(9j+p,se) = g(@j,—se), in accordance with the evenness property of the Radon transform. Then, {V*f){xik)

=^ ^ pq

j=o

vn(sico&(
132

F. NATTERER

Defining 2p-l

huk

=

- } P

vn(sicos{
j

3=0

£ = -q,...,q,

i = O,...,q,

we have

k = 0 , . . . , 2p - 1,

g

P V^

q hm

"~ h ' This defines the circular harmonic algorithm.

Algorithm 4 Circular harmonic algorithm for standard parallel geometry Data: The values {gjtt = g(9j, S() : j = 0 , . . . ,p — 1, £ = —q,..., q}, where g is the 2D Radon transform of / . Step 1: Precompute the number V£tij = VQ(S{ cos(ipj) — S£j and extend g^£ to all j = 0 , . . . , 2p — 1 by gj+p,£ = gj,-e-

Step 2: For i = 0 , . . . , q, £ = —q,..., q carry out the discrete cyclic convolutions 2p-l 7T ^ - - \

P '^"^

''

Step 3: For i = 0 , . . . , q and k = 0 , . . . , 2p — 1, compute x

P

Result: fcH(%ik) is a n approximation to

f(xik).

Step 2 of the algorithm has to be done with a fast Fourier transform (FFT) in order to make the algorithm competitive with filtered backprojection. In that case Step 2 requires O(q2p log p) operations. This is slightly more (by the factor logp) than what is needed in the filtered backprojection algorithm. Step 3 needs Apq2 additions. Algorithm 4 can be used almost without any changes for the interlaced parallel geometry, that is, for c/^ with £ + j odd missing (p even). One simply puts gjj = 0 for j + £ odd and doubles hgn- in Step 2. Circular harmonic algorithms are also available for standard fan beam data. Setting x = Xjfc = £i#(/3fc), U = iAt, At = p/q in (2.9) gives q

(V*f)(xik) ~rAaA/? ^

p-l

]Pvn{Ucos(/3k_j-ae)-rsinae)g{i3j,ae)cosae,

NUMERICAL METHODS IN TOMOGRAPHY

133

where g is now the fan beam data function from (2.8).

Algorithm 5 Circular harmonic algorithm for standard fan beam geometry Data: The values {gjte = g((3j, at) : j = 0 , . . . , p - 1, £ = -q,..., given by (2.8).

q}, g being

Step 1: Precompute the numbers V£tij = vn(ti cos((3j — at) —r sinai) cosa^. Step 2: For i — 0 , . . . , q, £ = —q,..., q carry out the discrete cyclic convolutions p-l j=0

Step 3: For i = 0 , . . . , q and k = 0 , . . . ,p — 1 compute the sums q

fcft(xik) = rAa ^2 l=-q

Result: fcHfaik) is an approximation to The complexity of Algorithm 5 is again A few remarks are in order.

f(xik). O(q2p\ogp).

1. Circular harmonic algorithms compute the reconstruction on a grid in polar coordinates. Interpolation to a Cartesian grid (for instance for the purpose of display) is not critical and can be done by linear interpolation. 2. The resolution of the circular harmonic algorithms is the same as for the corresponding filtered backprojection algorithms in Section 2.1. 3. Even though circular harmonic algorithms are asymptotically a little slower (by a factor of logp) than filtered backprojection, they usually run faster due to their simplicity. This is true in particular for fan beam data because in that case the backprojection is quite time-consuming. 4. Circular harmonic algorithms tend to be more accurate than filtered backprojection because no additional approximations, such as interpolations or homogeneity approximations (in the fan beam case), are used. 5. The disadvantage of circular harmonic algorithms lies in the fact that they start with angular convolutions. This is considered impractical in radiological applications.

134

F. NATTERER

6. Fourier reconstruction We have already made use of the relation for the Radon transform in R n in Section 2.1, and of the corresponding formula for the n-dimensional ray transform

in Section 3.4. In this section we use these formulas to derive reconstruction algorithms. To fix ideas, we consider the case of the 2D Radon transform, sampled as in the standard parallel geometry. This means that g = Rf is g i v e n f o r 0 = 8j = (cos ipj, s i n ipj)T, ifj = irj/p, j = 0, . . . , p — 1 a n d s = s^ = £p/q, £ = — q,..., q, as in Section 2.1. Here, / is assumed to vanish outside |x| < p. The idea of Fourier reconstruction is very simple: do a ID Fourier transform on g with respect to the second variable for each 6. According to (6.1) this yields / in all of R 2 . Do a 2D inverse Fourier transform to obtain / . Even though this seems fairly obvious, the numerical implementation in a discrete setting is quite intricate. In fact, good Fourier algorithms have been found only quite recently. To begin with we describe the simplest possible implementation. We warn the reader that this algorithm is quite useless since it is not sufficiently accurate.

Algorithm 6 Standard Fourier reconstruction Data: The numbers {gj^ — g(Oj, S() : j = 0 , . . . ,p— 1, £ = —q,..., q}, where g is the 2D Radon transform of / . Step 1: For j = 0 , . . . ,p — 1 carry out the discrete Fourier transform

gjtr = ( 2 7 r ) V ^ J2 einr£/%,t, q e=-q

r=-q,...,q.

Step 2: For each k G Z 2 , \k\ < q, find (j,r) such that r6j is as close as possible to k and put fk = (27T)- 1 / 2 ^, r . Step 3: Do the 2D discrete inverse Fourier transform

|fc|<9

Result: fm is an approximation to

NUMERICAL METHODS IN TOMOGRAPHY

135

The algorithm is designed to reconstruct a function / with support in |re| < p which is essentially Jl-band-limited. Inequalities (2.6), (2.7) have to be satisfied. We stress again that the algorithm as it stands is not to be recommended because of poor accuracy. Better versions are described below. A few comments are in order. In Step 1 we compute an approximation 9j,r to g

s) ds,

j e^g^,

^

Under assumption (2.6) this approximation is reliable since Fourier transforms are evaluated exactly by the trapezoidal rule if the Nyquist condition, in our case (2.6), is satisfied. According to (6.1), Step 1 provides us with the values

In Step 2 we compute / on the Cartesian grid interpolation: 7T , \

f^-k)~fk,

;

,

-2

keZz,

(TT/P)Z2

by nearest neighbour

\k\
Since / vanishes outside \x\ < p, f has bandwidth p. Thus sampling of / on a 2D grid with step-size ir/p is adequate by the sampling theorem. Step 3 is the trapezoidal rule for the 2D inverse Fourier transform, properly discretized and complying with the Nyquist condition. Hence fm is an approximation to f(pm/q). Steps 1 and 3 of Algorithm 6 are justified by the sampling theorem. Thus the failure of the algorithm must be caused by the interpolation in Step 2. This is in fact the case. Of course we can replace the interpolation by a more accurate one, such as linear interpolation. However, this does not help much. In spite of its poor accuracy, Fourier reconstruction is attractive because of its favourable complexity, which is due to the fast Fourier transform (FFT). We have used FFT in the circular harmonic algorithm already, but FFT is so essential for Fourier reconstruction that we say a few words here; for a thorough treatment we refer to Nussbaumer (1982). The discrete Fourier transform of length q is defined to be q-1

Any algorithm that computes yo,

jjq-i from yo,

, yq-\ in less then

136

F. NATTERER

typically O(qlogq), operations is a called an FFT. In the circular harmonic algorithm we have used FFT just for the evaluation of (6.3). In Fourier reconstruction we employ FFT for the evaluation of Fourier integrals

for the functions / in R1 with support in (—p, p). Sampling theory tells us that / has to be discretized with step-size ir/p (/ has bandwidth p). With h — p/q the step-size for / , the trapezoidal rule provides the approximation k = -q,...,q-l.

(6.4)

The range of k is in agreement with the sampling theorem: the step-size h = p/q corresponds to the bandwidth Q = n/h = (ir/p)q; hence |fc| < q in (6.4) suffices. Of course (6.4) is a discrete Fourier transform of length 2q. Sometimes one wants to adjust the step-sizes of / and / differently. Then one has to evaluate 9-1

yk = J2e~iC'k/qyt'

k= 0,...,q-l

(6.5)

£=0

with an arbitrary real parameter c. This can be done by the chirp-zalgorithm (see Nussbaumer (1982)), again using typically O(q\ogq) operations. Assuming that we have a fast Fourier transform (FFT) algorithm that does a discrete Fourier transform of length q with O(q log q) operations (this may restrict q to 'FFT-friendly numbers'), the complexity of Algorithm 6 is as follows. Step 1 does p Fourier transforms of length 2q, requiring O(pqlogq) operations. Assuming that the interpolation in Step 2 can be done in 0(1) operations per point we get 0(q2) as the complexity of Step 2. The 2D Fourier transform in Step 3 can be done with 0(q2 log q) operations. Hence the complexity of Algorithm 6 is O((pq + q2) logq). This is much better than the filtered backprojection algorithm (Algorithm 1), which needs O(q2p) operations for a reconstruction on a q x q grid. Presently there exist two Fourier methods with satisfactory accuracy and favourable complexity. 1. The linogram algorithm (Edholm and Herman 1987) Here, interpolation in Fourier space is avoided altogether by a clever choice of the directions

NUMERICAL METHODS IN TOMOGRAPHY

137

where j , £ = —q,..., q. Doing a ID Fourier transform on g^e results in

q

e=-Q

where
we get from (6.1)

e=-q Note that this can be done efficiently by the chirp-z-algorithm. The key observation is that the points

k-K

Q 1

__ kir ( 1 \ _ k-K ( 1 ~ p \cot tpj) ~ p \j/q

form a grid lying on vertical lines with distance ir/p, being evenly spaced within each vertical line (though with different step-sizes in different lines). On such a grid we can do a ID FFT in the horizontal direction in the usual way. In the horizontal direction the step-size is not what we need for a direct application of the FFT, but the chirp-z-algorithm is still applicable. This takes care of the 2D inverse Fourier transform in | ^ | > |£i|- For |£i| < l&l w e proceed analogously with the data gj^, evaluating f((T0j) for a = kir/p cos ipj. We remark that the linogram data in Edholm and Herman (1987) is a little different from ours, namely S£ = hisimpj, sg = hi cos
2. The gridding algorithm (O'Sullivan 1985, Kaveh and Soumekh 1987, Schomberg and Timmer 1995) This algorithm works on the standard parallel data used in Algorithm 6. It does the interpolation in Step 2 of Algorithm 6 in the following way. Let w be a smooth function in R2 with w = 1 on \x\ < p which is decaying exponentially at infinity. Put fw = wf. Then,

oo

1

= (27T)" fa 0

Iw(Z-a9)f(a6)deda S

1

oo

= (2vr)-3/2 fafwfc-

aO)g(O, a) d0 da

138

F. NATTERER

by (6.1). Using a quadrature rule with nodes {ar,9j} obtain the approximation p

fw,k =

3 2

(2TT)- /

^

q

\

a

Yl ^rW ( -k - ar03 J gjr

j=Qr=-q

to fw(-k).

/

and weights Oj>, we

^

(6.7)

'

The method relies on the following assumptions.

1. w is decaying at infinity so fast that only a few terms of the r sum in (6.7) have to be retained. 2. The dependence on the angle is not critical, so that it suffices to retain only a few terms in the j sum of (6.7). If these conditions are met then fw^ of (6.3) is a good approximation to fw{^k) which can be evaluated essentially in 0(1) operations for each k. This takes care of Step 2. Of course, when using (6.7) we have to divide / by w after Step 3 to make up for the previous multiplication with w. It is needless to say that our derivation of the gridding algorithm is purely heuristic. It seems that at present there exists no convincing theoretical analysis of the gridding algorithm. 7. Conclusions In the preceding sections we have given the fundamentals of the most widely used algorithms in tomography. In many ways these fundamentals are quite different from traditional numerical analysis, the main difference being the consistent use of sampling theory and Fourier analysis. The development of algorithms is still very lively, particularly in 3D and in Fourier-based algorithms. We have dealt only with the most simple problems and with standard situations. Practical problems deviate in many ways from the simple ones we considered. Often the data is incomplete (see Louis (1980)), leading to nonuniqueness and instability. Sometimes the integral equations to be solved are not completely specified, for instance the weight function (as in emission tomography (Welch, Clack, Natterer and Gullberg 1997)) or the directions (as in electron microscopy (Wuschke 1990, Gelfand and Goncharov 1990)) are unknown. In technical applications in particular, the number of data is often so small (see, for instance, Sielschott and Derichs (1995)) that full reconstruction is impossible and special algorithms have to be developed, usually tailored to the specific application. Sometimes only certain features of the object, such as boundaries between regions of different densities, are sought (Faridani, Finch, Ritman and Smith 1997, Ramm and Katsevich 1996), calling for special algorithms.

NUMERICAL METHODS IN TOMOGRAPHY

139

At present we have an adequate understanding of the fundamentals of tomographic reconstruction algorithms. However, new applications of tomography are arising almost daily, each presenting new challenges to the numerical analyst. So I guess that research in this field will go on for ever!

REFERENCES M. Abramowitz and I. A. Stegun, eds (1970), Handbook of Mathematical Functions, Dover. Y. Censor (1981), 'Row-action methods for huge and sparse systems and their applications', SIAM Review 23, 444-466. Y. Censor, P. B. Eggermont and D. Gordon (1983), 'Strong underrelaxation in Kaczmarz's method for inconsistent systems', Numer. Math. 41, 83-92. Y. Censor and S. A. Zenios (1997), Parallel Optimization, Oxford University Press. L. T. Chang and G. T. Herman (1980), 'A scientific study of filter selection for a fan-beam convolution algorithm', SIAM J. Appl. Math. 39, 83-105. C. H. Chapman and P. W. Cary (1986), 'The circular harmonic Radon transform', Inverse Problems 2, 23-49. J. G. Colsher (1980), 'Fully three-dimensional emission tomography', Phys. Med. Biol. 25, 103-115. A. M. Cormack (1963), 'Representation of a function by its line integrals, with some radiological applications I', J. Appl. Phys. 34, 2722-2727'. A. M. Cormack (1964), 'Representation of a function by its line integrals, with some radiological applications IF, J. Appl. Phys. 35, 195-207. S. R. Deans (1983), The Radon Transform and some of its Applications, Wiley. M. Defrise, D. W. Townsend and R. Clack (1989), 'Three-dimensional image reconstruction from complete projections', Phys. Med. Biol. 34, 573-587. M. Defrise and R. Clack (1995), 'A cone-beam reconstruction algorithm using shiftvariant filtering and cone-beam backprojection', IEEE Trans. Med. Irnag. 13, 186-195. A. P. Dempster, N. M. Laird and D. B. Rubin (1977), 'Maximum likelihood from incomplete data via the EM algorithm', J. R. Statist. Soc. B 39, 1-38. P. Edholm and G. T. Herman (1987), 'Linograms in image reconstruction from projections', IEEE Trans. Med. Imag. 6, 301-307. A. Faridani, D. V. Finch, E. L. Ritman and K. T. Smith (1997), 'Local tomography IF, SIAM J. Appl. Math. 57, 1095-1127. J. Frank, ed. (1992), Electron Tomography, Plenum Press. I. M. Gelfand and A. B. Goncharov (1987), 'Recovery of a compactly supported function starting from its integrals over lines intersecting a given set of points in space', Doklady 290 (1986); English Translation in Soviet Math. Dokl. 34, 373-376. I. M. Gelfand and A. B. Goncharov (1990), 'Spatial rotational alignment of identical particles given their projections: Theory and practice', Translation of Mathematical Monographs 81, 97-122. R. Gordon, R. Bender and G. T. Herman (1970), 'Algebra reconstruction techniques (ART) for three-dimensional electron microscopy and X-ray photography', J. Theor. Biol. 29, 471-481.

140

F. NATTERER

P. Grangeat (1991), 'Mathematical framework of cone-beam reconstruction via the first derivative of the Radon transform', in Mathematical Methods in Tomography, Vol. 1497 of Lecture Notes in Mathematics (G. T. Herman, A. K. Louis and F. Natterer, eds), Springer, pp. 66-97. P. J. Green (1990), 'Bayesian reconstruction from emission tomography data using a modified EM algorithm', IEEE Trans. Med. Imag. 9, 84-93. C. Hamaker and D. C. Solmon (1978), 'The angles between the null spaces of X-rays', J. Math. Anal. Appl. 62, 1-23. E. W. Hansen (1981), 'Circular harmonic image reconstruction', Applied Optics 20, 2266-2274. W. G. Hawkins and H. H. Barrett (1986), 'A numerically stable circular harmonic reconstruction algorithm', SIAM J. Numer. Anal. 23, 873-890. G. T. Herman (1980), Image Reconstruction from Projection: The Fundamentals of Computerized Tomography, Academic Press. G. T. Herman and L. Meyer (1993), 'Algebraic reconstruction techniques can be made computationally efficient', IEEE Trans. Med. Imag. 12, 600-609. H. M. Hudson, B. F. Hutton and R. Larkin (1992), 'Accelerated EM reconstruction using ordered subsets', J. Nucl. Med. 33, 960-968. A. J. Jerry (1977), 'The Shannon sampling theorem - its various extensions and applications: a tutorial review', Proc. IEEE 65, 1565-1596. S. Kaczmarz (1937), 'Angenaherte Auflosung von Systemen linearer Gleichungen', Bulletin de I'Academie Polonaise des Sciences et des Lettres A35, 355-357. A. C. Kak and M. Slaney (1987), Principles of Computerized Tomography Imaging, IEEE Press, New York. M. Kaveh and M. Soumekh (1987), 'Computer assisted diffraction tomography', in Image Recovery: Theory and Application (H. Stark, ed.), Academic Press, pp. 369-413. H. Kruse (1989), 'Resolution of reconstruction methods in computerized tomography', SIAM J. Sci. Statist. Comput. 10, 447-474. E. Levitan and G. T. Herman (1987), 'A maximum a posteriori probability expectation maximization algorithm for image reconstruction in emission tomography', IEEE Trans. Med. Imag. 6, 185-192. R. M. Lewitt (1992), 'Alternatives to voxels for image representations in iterative reconstruction algorithms', Phys. Med. Biol. 37, 705-716. A. K. Louis (1980), 'Picture reconstruction from projections in restricted range', Math. Meth. Appl. Sci. 2, 109-220. R. Marabini, G. T. Herman and J. M. Carazo (1998), 'Fully three-dimensional reconstruction in electron microscopy', in Computational Radiology and Imaging: Therapy and Diagnostics (C. Borgers and F. Natterer, eds), Vol. 110 of IMA Volumes in Mathematics and its Applications, Springer. R. B. Marr, C. N. Chen and P. C. Lauterbur (1981), 'On two approaches to 3D reconstruction in NMR zeugmatography', in Mathematical Aspects of Computerized Tomography (G. T. Herman and F. Natterer, eds), Proceedings, Oberwolfach 1980, Springer, pp. 225-240. F. Natterer (1986), The Mathematics of Computerized Tomography, Wiley and Teubner.

NUMERICAL METHODS IN TOMOGRAPHY

141

F. Natterer (1993), 'Sampling in fan beam tomography', SI AM J. Appl. Math. 53, 358-380. F. Natterer and A. Faridani (1990), 'Basic algorithms in tomography', in Signal Processing Part II: Control Theory and Applications (F. A. Griinbaum et al., eds), Springer, pp. 321-334. S. Nilsson (1997), 'Application of fast backprojection techniques for some inverse problems of integral geometry', Linkb'ping Studies in Science and Technology, Dissertation No. 499, Department of Mathematics, Linkoping University, Linkoping, Sweden. H. J. Nussbaumer (1982), Fast Fourier Transform and Convolution Algorithm, Springer. S. S. Orlov (1976), 'Theory of three dimensional reconstruction, II: The recovery operator', Sov. Phys. Crystallogr. 20, 429-433. J. D. O'Sullivan (1985), 'A fast sine function gridding algorithm for Fourier inversion in computer tomography', IEEE Trans. Med. Imag. 4, 200-207. A. R. De Pierro (1990), 'Multiplicative interative methods in computed tomography', in Mathematical Methods in Tomography (G. T. Herman, A. K. Louis and F. Natterer, eds), Springer, pp. 167-186. J. Radon (1917), 'Uber die Bestimmung von Funktionen durch ihre Integralwerte langs gewisser Mannigfaltigkeiten', Berichte Sdchsische Akademie der Wissenschaften, Math.-Phys. KL, 69, 262-267, Leipzig. A. Ramm and A. Katsevich (1996), The Radon Transform and Local Tomography, CRC Press. H. Schomberg and J. Timmer (1995), 'The gridding method for image reconstruction bx Fourier transformation', IEEE Trans. Med. Imag. 14, 596-607. L. A. Shepp and B. F. Logan (1974), 'The Fourier reconstruction of a head section', IEEE Trans. Trans. Nucl. Sci. NS-21, 21-43. L. A. Shepp and Y. Vardi (1982), 'Maximum likelihood reconstruction for emission tomography', IEEE Trans. Med. Imag. 1, 113-122. B. Setzepfandt (1992), 'ESNM: Ein rauschunterdriickendes EM-Verfahren fur die Emissionstomographie', Dissertation, Fachbereich Mathematik, Universitat Minister. H. Sielschott and W. Derichs (1995), 'Use of collocation methods under inclusion of a priori information in acoustic pyrometry', Proc. European Concerted Action on Process Tomography, Bergen, Norway, pp. 110-117. B. W. Silverman, M. C. Jones, D. W. Nychka and J. D. Wilson (1990), 'A smoothed EM approach to indirect estimation problems, with particular reference to stereology and emission tomography', J. R. Statist. Soc. B 52, 271-324. H. K. Tuy (1983), 'An inversion formula for cone-beam reconstruction', SIAM J. Appl. Math. 43, 546-552. A. Welch, R. Clack, F. Natterer and G. T. Gullberg (1997), 'Towards accurate attenuation correction in SPECT', IEEE Trans. Med. Imag. 16, 532-541. K. Wuschke (1990), 'Die Rekonstruktion von Orientierungen aus Projektionen', Diplomarbeit, Institut fur Numerische und Instrumentelle Mathematik, Universitat Miinster.

ActaNumerica (1999), pp. 143-195

© Cambridge University Press, 1999

Approximation theory of the MLP model in neural networks Allan Pinkus Department of Mathematics, Technion - Israel Institute of Technology, Haifa 32000, Israel E-mail: [email protected]

In this survey we discuss various approximation-theoretic problems that arise in the multilayer feedforward perceptron (MLP) model in neural networks. The MLP model is one of the more popular and practical of the many neural network models. Mathematically it is also one of the simpler models. Nonetheless the mathematics of this model is not well understood, and many of these problems are approximation-theoretic in character. Most of the research we will discuss is of very recent vintage. We will report on what has been done and on various unanswered questions. We will not be presenting practical (algorithmic) methods. We will, however, be exploring the capabilities and limitations of this model. In the first two sections we present a brief introduction and overview of neural networks and the multilayer feedforward perceptron model. In Section 3 we discuss in great detail the question of density. When does this model have the theoretical ability to approximate any reasonable function arbritrarily well? In Section 4 we present conditions for simultaneously approximating a function and its derivatives. Section 5 considers the interpolation capability of this model. In Section 6 we study upper and lower bounds on the order of approximation of this model. The material presented in Sections 3-6 treats the single hidden layer MLP model. In Section 7 we discuss some of the differences that arise when considering more than one hidden layer. The lengthy list of references includes many papers not cited in the text, but relevant to the subject matter of this survey.

144

A. PlNKUS

CONTENTS 1 On neural networks 2 The MLP model 3 Density 4 Derivative approximation 5 Interpolation 6 Degree of approximation 7 Two hidden layers References

144 146 150 162 165 167 182 187

1. On neural networks It will be assumed that most readers are pure and/or applied mathematicians who are less than conversant with the theory of neural networks. As such we begin this survey with a very brief, and thus inadequate, introduction. The question 'What is a neural network?' is ill-posed. From a quick glance through the literature one quickly realizes that there is no universally accepted definition of what the theory of neural networks is, or what it should be. It is generally agreed that neural network theory is a collection of models of computation very, very loosely based on biological motivations. According to Haykin (1994, p. 2): 'A neural network is a massively parallel distributed processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: 1. Knowledge is acquired by the network through a learning process. 2. Interneuron connection strengths known as synaptic weights are used to store the knowledge.' This is a highly nonmathematical formulation. Let us try to be a bit less heuristic. Neural network models have certain common characteristics. In all these models we are given a set of inputs x = {x\,..., xn) £ W1 and some process that results in a corresponding set of outputs y = ( y i , . . . , ym) £ M.m. The basic underlying assumption of our models is that the process is given by some mathematical function, that is, y = G(x) for some function G. The function G may be very complicated. More importantly, we cannot expect to be able to compute exactly the unknown G. What we do is choose our 'candidate' F (for G) from some parametrized set of functions using a given set of examples, that is, some inputs x and associated 'correct' outputs y = G(x), which we assume will help us to choose the parameters. This is a very general framework. In fact it is

APPROXIMATION THEORY OF THE MLP

MODEL IN NEURAL NETWORKS

145

still too general. Neural network models may be considered as particular choices of classes of functions F(x, w) where the w are the parameters, together with various rules and regulations as well as specific procedures for optimizing the choice of parameters. Most people would also agree that a neural network is an input/output system with many simple processors, each having a small amount of local memory. These units are connected by communication channels carrying data. Most neural network models have some sort of training rule, that is, they learn or are trained from a set of examples. There are many, many different models of neural network. (Sarle (1998) lists over 40 different recognized neural network models, and there are a plethora of additional candidates.) Neural networks have emerged, or are emerging, as a practical technology, that is, they are being successfully applied to real world problems. Many of their applications have to do with pattern recognition, pattern classification, or function approximation, which are all based on a large set of available examples (training set). According to Bishop (1995, p. 5): 'The importance of neural networks in this context is that they offer a very powerful and very general framework for representing non-linear mappings from several input variables to several output variables, where the form of the mapping is governed by a number of adjustable parameters.' The nonlinearity of the neural network models presents advantages and disadvantages. The price (and there always is a cost) is that the procedure for determining the values of the parameters is now a problem in nonlinear optimization which tends to be computationally intensive and complicated. The problem of finding efficient algorithms is of vital importance and the true utility of any model crucially depends upon its efficiency. (However, this is not an issue we will consider in this survey.) The theory of neural nets has become increasing popular in the fields of computer science, statistics, engineering (especially electrical engineering), physics, and many more directly applicable areas. There are now four major journals in the field, as well as numerous more minor journals. These leading journals are IEEE Transactions on Neural Networks, Neural Computation, Neural Networks and Neurocomputing. Similarly, there are now dozens of textbooks on the theory. In the references of this paper are listed only five books, namely Haykin (1994), Bishop (1995), Ripley (1996), Devroye, Gyorfi and Lugosi (1996), and Ellacott and Bos (1996), all of which have appeared in the last five years. The IEEE has generally sponsored (since 1987) two annual conferences on neural networks. Their proceedings run to over 2000 pages and each contains a few hundred articles and abstracts. A quick search of Mathematical Reviews (MathSciNet) turned up a mere 1800 entries when the phrase 'neural network' was entered (and you should realize that much of the neural network literature, including all the above-mentioned journals, is

146

A. PlNKUS

not written for or by mathematicians and is not reviewed by Mathematical Reviews). In other words, this is an explosively active research area and deserves the attention of the readership of Ada Numerica. Initially there was a definite lack of mathematical sophistication to the theory. It tended to be more a collection of ad hoc techniques with debatable justifications. To a pure mathematician, such as the author, reading through some of the early literature in the field was an alien experience. In recent years the professionals (especially statisticians) have established a more organized framework for the theory. The reader who would like to acquire a more balanced and enlarged view of the theory of neural networks is urged to peruse a few of the abovementioned texts. An additional excellent source of information about neural networks and its literature is the 'frequently asked questions' (FAQs) of the Usenet newsgroup comp.ai.neural-nets: see Sarle (1998). This survey is not about neural networks per se, but about the approximation theory of the multilayer feedforward perceptron (MLP) model in neural networks. We will consider certain mathematical, rather than computational or statistical, problems associated with this widely used neural net model. More explicitly, we shall concern ourselves with problems of density (when the models have at least the theoretical capability of providing good approximations), degree of approximation (the extent to which they can approximate, as a function of the number of parameters), interpolation, and related issues. Theoretical results, such as those we will survey, do not usually have direct applications. In fact they are often far removed from practical considerations. Rather they are meant to tell us what is possible and, sometimes equally importantly, what is not. They are also meant to explain why certain things can or cannot occur, by highlighting their salient characteristics, and this can be very useful. As such we have tried to provide proofs of many of the results surveyed. The 1994 issue of Ada Numerica contained a detailed survey: 'Aspects of the numerical analysis of neural networks' by S. W. Ellacott (1994). Only five years have since elapsed, but the editors have again opted to solicit a survey (this time albeit with a slightly altered emphasis) related to neural networks. This is not unwarranted. While almost half of that survey was devoted to approximation-theoretic results in neural networks, almost every one of those results has been superseded. It is to be hoped that the same will be said about this paper five years hence. 2. The MLP model One of the more conceptually attractive of the neural network models is the multilayer feedforward perceptron (MLP) model. In its most basic form this is a model consisting of a finite number of successive layers. Each layer

APPROXIMATION THEORY OF THE MLP MODEL IN NEURAL NETWORKS

147

consists of a finite number of units (often called neurons). Each unit of each layer is connected to each unit of the subsequent (and thus previous) layer. These connections are generally called links or synapses. Information flows from one layer to the subsequent layer (thus the term feedforward). The first layer, called the input layer, consists of the input. There are then intermediate layers, called hidden layers. The resulting output is obtained in the last layer, not surprisingly called the output layer. The rules and regulations governing this model are the following. 1. The input layer has as output of its jth unit the (input) value XQJ. 2. The kth unit of the ith layer receives the output Xij from each jth unit of the (i — l)st layer. The values x^ are then multiplied by some constants (called weights) w^ and these products are summed. 3. A shift 6ik (called a threshold or bias) and then a fixed mapping a (called an activation function) are applied to the above sum and the resulting value represents the output Xi+\^ of this kth. unit of the ith layer, that is,

A priori one typically fixes, for whatever reasons, the activation function, the number of layers and the number of units in each layer. The next step is to choose, in some way, the values of the weights w^ and thresholds Oik- These latter values are generally chosen so that the model behaves well on some given set of inputs and associated outputs. (These are called the training set.) The process of determining the weights and thresholds is called learning or training. In the multilayer feedforward perceptron model, the basic learning algorithm is called backpropagation. Backpropagation is a gradient descent method. It is extremely important in this model and in neural network theory. We shall not detail this algorithm nor the numerous numerical difficulties involved. We will classify multilayer feedforward perceptron models not by their number of layers, but by their number of hidden layers, that is, the number of layers excluding the input and output layer. As is evident, neural network theory has its own terminology. Unfortunately it is also true that this terminology is not always consistent or logical. For example, the term multilayer perceptron is generically applied to the above model with at least one hidden layer. On the other hand the word perceptron was coined by F. Rosenblatt for the no hidden layer model with the specific activation function given by the Heaviside function

^ ^ { ,

t
148

A. PlNKUS

Thus